2021-07-31T01:17:51.680 INFO:root:teuthology version: 1.1.0-73aa7e3b 2021-07-31T01:17:51.682 DEBUG:teuthology.run:Teuthology command: teuthology --description rados/upgrade/parallel/{0-distro$/{rhel_8.3_kubic_stable} 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} --verbose --owner scheduled_kchai@teuthology --archive /home/teuthworker/archive/kchai-2021-07-31_00:20:26-rados-wip-kefu-testing-2021-07-31-0023-distro-basic-smithi/6303840 --name kchai-2021-07-31_00:20:26-rados-wip-kefu-testing-2021-07-31-0023-distro-basic-smithi -- /home/teuthworker/archive/kchai-2021-07-31_00:20:26-rados-wip-kefu-testing-2021-07-31-0023-distro-basic-smithi/6303840/orig.config.yaml 2021-07-31T01:17:51.749 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2021-07-31T01:17:51.822 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/kchai-2021-07-31_00:20:26-rados-wip-kefu-testing-2021-07-31-0023-distro-basic-smithi/6303840 branch: wip-kefu-testing-2021-07-31-0023 description: rados/upgrade/parallel/{0-distro$/{rhel_8.3_kubic_stable} 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} email: tchaikov@gmail.com first_in_suite: false job_id: '6303840' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'run run randomized correctness test for rados operations on an erasure-coded pool ' - desc: 'object class functional tests ' - desc: 'generate read/write load with rados objects ranging from 1MB to 25MB ' - desc: 'run basic import/export cli tests for rbd ' - desc: 'librbd C and C++ api tests ' - desc: 'librbd python api tests ' name: kchai-2021-07-31_00:20:26-rados-wip-kefu-testing-2021-07-31-0023-distro-basic-smithi nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: rhel os_version: '8.3' overrides: admin_socket: branch: wip-kefu-testing-2021-07-31-0023 ceph: conf: global: mon election default strategy: 1 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd shutdown pgref assert: true create_rbd_pool: true log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: f25860a2cae30438a8a8d15f1f74f6043cc73d75 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: f25860a2cae30438a8a8d15f1f74f6043cc73d75 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-kefu-testing-2021-07-31-0023 sha1: f25860a2cae30438a8a8d15f1f74f6043cc73d75 owner: scheduled_kchai@teuthology priority: 70 repo: git://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b sha1: f25860a2cae30438a8a8d15f1f74f6043cc73d75 sleep_before_teardown: 0 suite: rados suite_branch: wip-kefu-testing-2021-07-31-0023 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_f25860a2cae30438a8a8d15f1f74f6043cc73d75/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: f25860a2cae30438a8a8d15f1f74f6043cc73d75 targets: smithi045.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDCR9AQrzxSWscI8OVzteUDkQPPgaJVUGUbHLgCI9njS7lpF/BMJxOtjlcKXZ3YtJXYrLbbv2tlD/tnFOcPkLM2lN6T7lShV2OQZ1WOmsDsXylL0+x4/SenqtXzAoLOdY5vOduw2Wh9KJQ21wH3jcOWCNH7Ze04Sl6Hf4mHHlTrbVI4h3PaNBGTeNWCbTYWZvCuNImEiahgEOimXPoMqV/U1Z1lGYcK5rT9qmNyJ6RXdwUYAsGjnQOuLGjje9MXo0n0f4Zd0Jpt34aHTyTzkkSrc1Qh3AJwlKMX0LCSQ572HRUmispi6ocGk8LWMuwnoWJ/S65gtdF5fFjCTwdd4BDqj+1F1ZLVeHp5sxR0+ZmPdIN9f1AzUQnG2iXM8RO0VQLxs6D+ge0yEAUMYiVdBO8X6jGCA5he6WL2q7lK5BeWlkzCDS5RJraF+4hcUuTlJpJYVI1qVoLtsEVc3EoKm73dhXpBVyoBscD+0dV5UoUwpKdKdytmN9T6kCHItu6csZM= smithi154.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMHllBo8MSxfRYXoLrl4tjeDJ/VzYdlxOLeB7HzeZcIzkWU4hYgZAnogxOu36VbcXqLnd7jaYW9H/iDjI/fUWX4vH2A7z4dBrpYaEBYBb0VRQHRtQRiFMz/H+oyuGyZkiaZCvY24rk/BsHSU8pnVtSgZGtScsW6JkXn5iOXSwPjR1aYtDiYYOtrupPVf0BJ6dmjC0aePtMk/jExY7+4xHLIvYSSsyNvE26ARhwXyFzoyZfrMlyfLA2iM2Pi2VXhEZPpvCWJ/wXMfANV4ChPArNIwKZX2qRwtRQ4dfXfXm6M4gAPYOZW32MpADacfN8n9blOqMuuIVGIRpiUjZlOU0lIOy0ta/2MtA2bnj6PrqkQXbIETzWTA55V8WOVJVbUPnbb2aVEsM2pgzXvddJ6u3nWmCRi3pyaYGVVsPLBjlR1Z43WI1p6WQHNVxzjBwM1tXj+DPjhckTXvHibDYJzHIwOk2JxYhMaCBh39Kg4w+vx4SNtnX+8eySi0W75C/Eqjk= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module disable container-tools - sudo dnf -y install 'dnf-command(copr)' - sudo dnf -y copr enable rhcontainerbot/container-selinux - sudo curl -L -o /etc/yum.repos.d/devel:kubic:libcontainers:stable.repo https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/CentOS_8/devel:kubic:libcontainers:stable.repo - sudo dnf remove -y podman - sudo dnf -y install podman - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - install: branch: pacific - print: '**** done install task...' - print: '**** done start installing pacific cephadm ...' - cephadm: cephadm_branch: pacific cephadm_git_url: https://github.com/ceph/ceph conf: osd: osd_class_default_list: '*' osd_class_load_list: '*' image: docker.io/ceph/daemon-base:latest-pacific - print: '**** done end installing pacific cephadm ...' - print: '**** done start cephadm.shell ceph config set mgr...' - cephadm.shell: mon.a: - ceph config set mgr mgr/cephadm/use_repo_digest true --force - print: '**** done cephadm.shell ceph config set mgr...' - print: '**** done start parallel' - parallel: - workload - upgrade-sequence - print: '**** done end parallel' teuthology_branch: master teuthology_sha1: 73aa7e3b960c7ffac669297b6aa86606265edd1b timestamp: 2021-07-31_00:20:26 tube: smithi upgrade-sequence: sequential: - print: '**** done start upgrade, wait' - cephadm.shell: env: - sha1 mon.a: - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - while ceph orch upgrade status | jq '.in_progress' | grep true ; do ceph orch ps ; ceph versions ; sleep 30 ; done - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 - print: '**** done end upgrade, wait...' user: kchai verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.30581 workload: full_sequential: - print: '**** done start ec-rados-default.yaml' - rados: clients: - client.0 ec_pool: true objects: 50 op_weights: append: 100 copy_from: 50 delete: 50 read: 100 rmattr: 25 rollback: 50 setattr: 25 snap_create: 50 snap_remove: 50 write: 0 ops: 4000 write_append_excl: false - print: '**** done end ec-rados-default.yaml' - print: '**** done start rados_api.yaml' - workunit: branch: pacific clients: client.0: - cls - print: '**** done end rados_api.yaml' - print: '**** done start rados_loadgenbig.yaml' - workunit: branch: pacific clients: client.0: - rados/load-gen-big.sh - print: '**** done end rados_loadgenbig.yaml' - print: '**** done start rbd_import_export.yaml' - workunit: branch: pacific clients: client.1: - rbd/import_export.sh env: RBD_CREATE_ARGS: --new-format - print: '**** done end rbd_import_export.yaml' - print: '**** done start test_rbd_api.yaml' - workunit: branch: pacific clients: client.0: - rbd/test_librbd.sh - print: '**** done end test_rbd_api.yaml' - print: '**** done start test_rbd_python.yaml' - workunit: branch: pacific clients: client.0: - rbd/test_librbd_python.sh - print: '**** done end test_rbd_python.yaml' 2021-07-31T01:17:51.823 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_f25860a2cae30438a8a8d15f1f74f6043cc73d75/qa; will attempt to use it 2021-07-31T01:17:51.824 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_f25860a2cae30438a8a8d15f1f74f6043cc73d75/qa/tasks 2021-07-31T01:17:51.825 INFO:teuthology.run_tasks:Running task internal.check_packages... 2021-07-31T01:17:51.827 INFO:teuthology.task.internal:Checking packages... 2021-07-31T01:17:51.847 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'basic' and ceph hash 'f25860a2cae30438a8a8d15f1f74f6043cc73d75' 2021-07-31T01:17:51.848 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2021-07-31T01:17:51.848 INFO:teuthology.packaging:ref: None 2021-07-31T01:17:51.849 INFO:teuthology.packaging:tag: None 2021-07-31T01:17:51.849 INFO:teuthology.packaging:branch: wip-kefu-testing-2021-07-31-0023 2021-07-31T01:17:51.849 INFO:teuthology.packaging:sha1: f25860a2cae30438a8a8d15f1f74f6043cc73d75 2021-07-31T01:17:51.850 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-kefu-testing-2021-07-31-0023 2021-07-31T01:17:52.014 INFO:teuthology.task.internal:Found packages for ceph version 17.0.0-6562.gf25860a2 2021-07-31T01:17:52.014 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2021-07-31T01:17:52.028 INFO:teuthology.task.internal:no buildpackages task found 2021-07-31T01:17:52.028 INFO:teuthology.run_tasks:Running task internal.save_config... 2021-07-31T01:17:52.055 INFO:teuthology.task.internal:Saving configuration 2021-07-31T01:17:52.083 INFO:teuthology.run_tasks:Running task internal.check_lock... 2021-07-31T01:17:52.129 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-07-31T01:17:52.155 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-07-31 01:12:33.271171', 'description': '/home/teuthworker/archive/kchai-2021-07-31_00:20:26-rados-wip-kefu-testing-2021-07-31-0023-distro-basic-smithi/6303840', 'locked_by': 'scheduled_kchai@teuthology', 'up': True, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDTK96k32gtr+d+0TTjdMUHvAO7ygYOPNZuKByIJjepMZdWGy6+cbx9dVVV9vSlLqK0LFpjoRGuO78nQolqetOY=', 'os_version': '8.3', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'rhel', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi045.front.sepia.ceph.com'} 2021-07-31T01:17:52.173 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-07-31 01:12:33.271149', 'description': '/home/teuthworker/archive/kchai-2021-07-31_00:20:26-rados-wip-kefu-testing-2021-07-31-0023-distro-basic-smithi/6303840', 'locked_by': 'scheduled_kchai@teuthology', 'up': True, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIr5tHnLtm0JGRBniBGKzWe3qr61dT8blzoLNRY0odUyAQN7yekjp/F3CdTpWv5EegKhoJdnCPo0Bps80vAHLmI=', 'os_version': '8.3', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'rhel', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi154.front.sepia.ceph.com'} 2021-07-31T01:17:52.173 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2021-07-31T01:17:52.199 INFO:teuthology.task.internal:roles: ubuntu@smithi045.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a'] 2021-07-31T01:17:52.199 INFO:teuthology.task.internal:roles: ubuntu@smithi154.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2021-07-31T01:17:52.199 INFO:teuthology.run_tasks:Running task console_log... 2021-07-31T01:17:52.281 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fef4d2e0e18>, signals=[15]) 2021-07-31T01:17:52.283 INFO:teuthology.run_tasks:Running task internal.connect... 2021-07-31T01:17:52.296 INFO:teuthology.task.internal:Opening connections... 2021-07-31T01:17:52.297 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi045.front.sepia.ceph.com 2021-07-31T01:17:52.298 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-07-31T01:17:52.374 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi154.front.sepia.ceph.com 2021-07-31T01:17:52.375 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-07-31T01:17:52.444 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2021-07-31T01:17:52.455 DEBUG:teuthology.orchestra.run.smithi045:> uname -m 2021-07-31T01:17:52.501 INFO:teuthology.orchestra.run.smithi045.stdout:x86_64 2021-07-31T01:17:52.501 DEBUG:teuthology.orchestra.run.smithi045:> cat /etc/os-release 2021-07-31T01:17:52.555 INFO:teuthology.orchestra.run.smithi045.stdout:NAME="Red Hat Enterprise Linux" 2021-07-31T01:17:52.556 INFO:teuthology.orchestra.run.smithi045.stdout:VERSION="8.3 (Ootpa)" 2021-07-31T01:17:52.557 INFO:teuthology.orchestra.run.smithi045.stdout:ID="rhel" 2021-07-31T01:17:52.557 INFO:teuthology.orchestra.run.smithi045.stdout:ID_LIKE="fedora" 2021-07-31T01:17:52.557 INFO:teuthology.orchestra.run.smithi045.stdout:VERSION_ID="8.3" 2021-07-31T01:17:52.557 INFO:teuthology.orchestra.run.smithi045.stdout:PLATFORM_ID="platform:el8" 2021-07-31T01:17:52.558 INFO:teuthology.orchestra.run.smithi045.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.3 (Ootpa)" 2021-07-31T01:17:52.558 INFO:teuthology.orchestra.run.smithi045.stdout:ANSI_COLOR="0;31" 2021-07-31T01:17:52.558 INFO:teuthology.orchestra.run.smithi045.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.3:GA" 2021-07-31T01:17:52.558 INFO:teuthology.orchestra.run.smithi045.stdout:HOME_URL="https://www.redhat.com/" 2021-07-31T01:17:52.559 INFO:teuthology.orchestra.run.smithi045.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-07-31T01:17:52.559 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:17:52.559 INFO:teuthology.orchestra.run.smithi045.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2021-07-31T01:17:52.559 INFO:teuthology.orchestra.run.smithi045.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.3 2021-07-31T01:17:52.559 INFO:teuthology.orchestra.run.smithi045.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2021-07-31T01:17:52.560 INFO:teuthology.orchestra.run.smithi045.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.3" 2021-07-31T01:17:52.561 INFO:teuthology.lock.ops:Updating smithi045.front.sepia.ceph.com on lock server 2021-07-31T01:17:52.578 DEBUG:teuthology.orchestra.run.smithi154:> uname -m 2021-07-31T01:17:52.597 INFO:teuthology.orchestra.run.smithi154.stdout:x86_64 2021-07-31T01:17:52.598 DEBUG:teuthology.orchestra.run.smithi154:> cat /etc/os-release 2021-07-31T01:17:52.651 INFO:teuthology.orchestra.run.smithi154.stdout:NAME="Red Hat Enterprise Linux" 2021-07-31T01:17:52.652 INFO:teuthology.orchestra.run.smithi154.stdout:VERSION="8.3 (Ootpa)" 2021-07-31T01:17:52.652 INFO:teuthology.orchestra.run.smithi154.stdout:ID="rhel" 2021-07-31T01:17:52.652 INFO:teuthology.orchestra.run.smithi154.stdout:ID_LIKE="fedora" 2021-07-31T01:17:52.652 INFO:teuthology.orchestra.run.smithi154.stdout:VERSION_ID="8.3" 2021-07-31T01:17:52.653 INFO:teuthology.orchestra.run.smithi154.stdout:PLATFORM_ID="platform:el8" 2021-07-31T01:17:52.653 INFO:teuthology.orchestra.run.smithi154.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.3 (Ootpa)" 2021-07-31T01:17:52.653 INFO:teuthology.orchestra.run.smithi154.stdout:ANSI_COLOR="0;31" 2021-07-31T01:17:52.653 INFO:teuthology.orchestra.run.smithi154.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.3:GA" 2021-07-31T01:17:52.653 INFO:teuthology.orchestra.run.smithi154.stdout:HOME_URL="https://www.redhat.com/" 2021-07-31T01:17:52.654 INFO:teuthology.orchestra.run.smithi154.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-07-31T01:17:52.654 INFO:teuthology.orchestra.run.smithi154.stdout: 2021-07-31T01:17:52.654 INFO:teuthology.orchestra.run.smithi154.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2021-07-31T01:17:52.654 INFO:teuthology.orchestra.run.smithi154.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.3 2021-07-31T01:17:52.655 INFO:teuthology.orchestra.run.smithi154.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2021-07-31T01:17:52.655 INFO:teuthology.orchestra.run.smithi154.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.3" 2021-07-31T01:17:52.655 INFO:teuthology.lock.ops:Updating smithi154.front.sepia.ceph.com on lock server 2021-07-31T01:17:52.670 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2021-07-31T01:17:52.680 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2021-07-31T01:17:52.720 INFO:teuthology.task.internal:Checking for old test directory... 2021-07-31T01:17:52.721 DEBUG:teuthology.orchestra.run.smithi045:> test '!' -e /home/ubuntu/cephtest 2021-07-31T01:17:52.724 DEBUG:teuthology.orchestra.run.smithi154:> test '!' -e /home/ubuntu/cephtest 2021-07-31T01:17:52.738 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2021-07-31T01:17:52.793 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2021-07-31T01:17:52.794 DEBUG:teuthology.orchestra.run.smithi045:> test -z $(ls -A /var/lib/ceph) 2021-07-31T01:17:52.798 DEBUG:teuthology.orchestra.run.smithi154:> test -z $(ls -A /var/lib/ceph) 2021-07-31T01:17:52.820 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2021-07-31T01:17:52.874 INFO:teuthology.run_tasks:Running task kernel... 2021-07-31T01:17:52.890 DEBUG:teuthology.task.kernel:normalized config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.c': {'kdb': True, 'sha1': 'distro'}, 'mgr.y': {'kdb': True, 'sha1': 'distro'}, 'osd.0': {'kdb': True, 'sha1': 'distro'}, 'osd.1': {'kdb': True, 'sha1': 'distro'}, 'osd.2': {'kdb': True, 'sha1': 'distro'}, 'osd.3': {'kdb': True, 'sha1': 'distro'}, 'client.0': {'kdb': True, 'sha1': 'distro'}, 'node-exporter.a': {'kdb': True, 'sha1': 'distro'}, 'alertmanager.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'mgr.x': {'kdb': True, 'sha1': 'distro'}, 'osd.4': {'kdb': True, 'sha1': 'distro'}, 'osd.5': {'kdb': True, 'sha1': 'distro'}, 'osd.6': {'kdb': True, 'sha1': 'distro'}, 'osd.7': {'kdb': True, 'sha1': 'distro'}, 'client.1': {'kdb': True, 'sha1': 'distro'}, 'prometheus.a': {'kdb': True, 'sha1': 'distro'}, 'grafana.a': {'kdb': True, 'sha1': 'distro'}, 'node-exporter.b': {'kdb': True, 'sha1': 'distro'}} 2021-07-31T01:17:52.891 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2021-07-31T01:17:52.891 DEBUG:teuthology.orchestra.run.smithi045:> uname -r 2021-07-31T01:17:52.910 INFO:teuthology.orchestra.run.smithi045.stdout:4.18.0-240.1.1.el8_3.x86_64 2021-07-31T01:17:52.911 INFO:teuthology.task.kernel:Running kernel on smithi045: 4.18.0-240.1.1.el8_3.x86_64 2021-07-31T01:17:52.911 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum install -y kernel 2021-07-31T01:17:54.438 INFO:teuthology.orchestra.run.smithi045.stdout:Updating Subscription Management repositories. 2021-07-31T01:17:54.439 INFO:teuthology.orchestra.run.smithi045.stdout:Unable to read consumer identity 2021-07-31T01:17:54.526 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:17:54.526 INFO:teuthology.orchestra.run.smithi045.stdout:This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. 2021-07-31T01:17:54.526 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:17:57.038 INFO:teuthology.orchestra.run.smithi045.stdout:Extra Packages for Enterprise Linux 11 MB/s | 10 MB 00:00 2021-07-31T01:17:59.200 INFO:teuthology.orchestra.run.smithi045.stdout:lab-extras 4.6 MB/s | 24 kB 00:00 2021-07-31T01:17:59.801 INFO:teuthology.orchestra.run.smithi045.stdout:Package kernel-4.18.0-240.1.1.el8_3.x86_64 is already installed. 2021-07-31T01:17:59.801 INFO:teuthology.orchestra.run.smithi045.stdout:Package kernel-4.18.0-240.el8.x86_64 is already installed. 2021-07-31T01:17:59.920 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2021-07-31T01:17:59.929 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2021-07-31T01:17:59.930 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2021-07-31T01:18:00.059 DEBUG:teuthology.orchestra.run.smithi045:> echo no | sudo yum reinstall kernel || true 2021-07-31T01:18:00.340 INFO:teuthology.orchestra.run.smithi045.stdout:Updating Subscription Management repositories. 2021-07-31T01:18:00.340 INFO:teuthology.orchestra.run.smithi045.stdout:Unable to read consumer identity 2021-07-31T01:18:00.393 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:18:00.394 INFO:teuthology.orchestra.run.smithi045.stdout:This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. 2021-07-31T01:18:00.394 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:18:00.575 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:01 ago on Sat 31 Jul 2021 01:17:59 AM UTC. 2021-07-31T01:18:00.732 INFO:teuthology.orchestra.run.smithi045.stdout:Installed package kernel-4.18.0-240.1.1.el8_3.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2021-07-31T01:18:00.733 INFO:teuthology.orchestra.run.smithi045.stdout:Installed package kernel-4.18.0-240.el8.x86_64 (from anaconda) not available. 2021-07-31T01:18:00.766 INFO:teuthology.orchestra.run.smithi045.stderr:Error: No packages marked for reinstall. 2021-07-31T01:18:00.803 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum reinstall -y kernel || true 2021-07-31T01:18:01.081 INFO:teuthology.orchestra.run.smithi045.stdout:Updating Subscription Management repositories. 2021-07-31T01:18:01.082 INFO:teuthology.orchestra.run.smithi045.stdout:Unable to read consumer identity 2021-07-31T01:18:01.122 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:18:01.122 INFO:teuthology.orchestra.run.smithi045.stdout:This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. 2021-07-31T01:18:01.122 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:18:01.306 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:02 ago on Sat 31 Jul 2021 01:17:59 AM UTC. 2021-07-31T01:18:01.474 INFO:teuthology.orchestra.run.smithi045.stdout:Installed package kernel-4.18.0-240.1.1.el8_3.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2021-07-31T01:18:01.474 INFO:teuthology.orchestra.run.smithi045.stdout:Installed package kernel-4.18.0-240.el8.x86_64 (from anaconda) not available. 2021-07-31T01:18:01.505 INFO:teuthology.orchestra.run.smithi045.stderr:Error: No packages marked for reinstall. 2021-07-31T01:18:01.546 DEBUG:teuthology.orchestra.run.smithi045:> rpm -q kernel --last | head -n 1 2021-07-31T01:18:01.632 INFO:teuthology.orchestra.run.smithi045.stdout:kernel-4.18.0-240.1.1.el8_3.x86_64 Tue 10 Nov 2020 09:16:49 PM UTC 2021-07-31T01:18:01.634 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-240.1.1.el8_3.x86_64 2021-07-31T01:18:01.634 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2021-07-31T01:18:01.634 DEBUG:teuthology.orchestra.run.smithi154:> uname -r 2021-07-31T01:18:01.652 INFO:teuthology.orchestra.run.smithi154.stdout:4.18.0-240.1.1.el8_3.x86_64 2021-07-31T01:18:01.653 INFO:teuthology.task.kernel:Running kernel on smithi154: 4.18.0-240.1.1.el8_3.x86_64 2021-07-31T01:18:01.653 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum install -y kernel 2021-07-31T01:18:03.189 INFO:teuthology.orchestra.run.smithi154.stdout:Updating Subscription Management repositories. 2021-07-31T01:18:03.192 INFO:teuthology.orchestra.run.smithi154.stdout:Unable to read consumer identity 2021-07-31T01:18:03.359 INFO:teuthology.orchestra.run.smithi154.stdout: 2021-07-31T01:18:03.360 INFO:teuthology.orchestra.run.smithi154.stdout:This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. 2021-07-31T01:18:03.360 INFO:teuthology.orchestra.run.smithi154.stdout: 2021-07-31T01:18:05.658 INFO:teuthology.orchestra.run.smithi154.stdout:Extra Packages for Enterprise Linux 12 MB/s | 10 MB 00:00 2021-07-31T01:18:07.649 INFO:teuthology.orchestra.run.smithi154.stdout:lab-extras 4.2 MB/s | 24 kB 00:00 2021-07-31T01:18:08.193 INFO:teuthology.orchestra.run.smithi154.stdout:Package kernel-4.18.0-240.1.1.el8_3.x86_64 is already installed. 2021-07-31T01:18:08.194 INFO:teuthology.orchestra.run.smithi154.stdout:Package kernel-4.18.0-240.el8.x86_64 is already installed. 2021-07-31T01:18:08.319 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2021-07-31T01:18:08.325 INFO:teuthology.orchestra.run.smithi154.stdout:Nothing to do. 2021-07-31T01:18:08.325 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2021-07-31T01:18:08.424 DEBUG:teuthology.orchestra.run.smithi154:> echo no | sudo yum reinstall kernel || true 2021-07-31T01:18:08.687 INFO:teuthology.orchestra.run.smithi154.stdout:Updating Subscription Management repositories. 2021-07-31T01:18:08.688 INFO:teuthology.orchestra.run.smithi154.stdout:Unable to read consumer identity 2021-07-31T01:18:08.733 INFO:teuthology.orchestra.run.smithi154.stdout: 2021-07-31T01:18:08.733 INFO:teuthology.orchestra.run.smithi154.stdout:This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. 2021-07-31T01:18:08.733 INFO:teuthology.orchestra.run.smithi154.stdout: 2021-07-31T01:18:08.911 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:00:01 ago on Sat 31 Jul 2021 01:18:07 AM UTC. 2021-07-31T01:18:09.049 INFO:teuthology.orchestra.run.smithi154.stdout:Installed package kernel-4.18.0-240.1.1.el8_3.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2021-07-31T01:18:09.049 INFO:teuthology.orchestra.run.smithi154.stdout:Installed package kernel-4.18.0-240.el8.x86_64 (from anaconda) not available. 2021-07-31T01:18:09.087 INFO:teuthology.orchestra.run.smithi154.stderr:Error: No packages marked for reinstall. 2021-07-31T01:18:09.125 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum reinstall -y kernel || true 2021-07-31T01:18:09.391 INFO:teuthology.orchestra.run.smithi154.stdout:Updating Subscription Management repositories. 2021-07-31T01:18:09.391 INFO:teuthology.orchestra.run.smithi154.stdout:Unable to read consumer identity 2021-07-31T01:18:09.431 INFO:teuthology.orchestra.run.smithi154.stdout: 2021-07-31T01:18:09.431 INFO:teuthology.orchestra.run.smithi154.stdout:This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. 2021-07-31T01:18:09.431 INFO:teuthology.orchestra.run.smithi154.stdout: 2021-07-31T01:18:09.604 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:00:02 ago on Sat 31 Jul 2021 01:18:07 AM UTC. 2021-07-31T01:18:09.738 INFO:teuthology.orchestra.run.smithi154.stdout:Installed package kernel-4.18.0-240.1.1.el8_3.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2021-07-31T01:18:09.739 INFO:teuthology.orchestra.run.smithi154.stdout:Installed package kernel-4.18.0-240.el8.x86_64 (from anaconda) not available. 2021-07-31T01:18:09.773 INFO:teuthology.orchestra.run.smithi154.stderr:Error: No packages marked for reinstall. 2021-07-31T01:18:09.809 DEBUG:teuthology.orchestra.run.smithi154:> rpm -q kernel --last | head -n 1 2021-07-31T01:18:09.907 INFO:teuthology.orchestra.run.smithi154.stdout:kernel-4.18.0-240.1.1.el8_3.x86_64 Tue 10 Nov 2020 09:16:49 PM UTC 2021-07-31T01:18:09.909 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-240.1.1.el8_3.x86_64 2021-07-31T01:18:09.909 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2021-07-31T01:18:09.910 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2021-07-31T01:18:09.910 DEBUG:teuthology.orchestra.run.smithi045:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2021-07-31T01:18:09.943 INFO:teuthology.orchestra.run.smithi045.stdout:ttyS1 2021-07-31T01:18:09.960 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2021-07-31T01:18:09.960 DEBUG:teuthology.orchestra.run.smithi154:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2021-07-31T01:18:09.996 INFO:teuthology.orchestra.run.smithi154.stdout:ttyS1 2021-07-31T01:18:10.011 INFO:teuthology.run_tasks:Running task internal.base... 2021-07-31T01:18:10.023 INFO:teuthology.task.internal:Creating test directory... 2021-07-31T01:18:10.024 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2021-07-31T01:18:10.027 DEBUG:teuthology.orchestra.run.smithi154:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2021-07-31T01:18:10.066 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2021-07-31T01:18:10.075 INFO:teuthology.run_tasks:Running task internal.archive... 2021-07-31T01:18:10.133 INFO:teuthology.task.internal:Creating archive directory... 2021-07-31T01:18:10.134 DEBUG:teuthology.orchestra.run.smithi045:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2021-07-31T01:18:10.136 DEBUG:teuthology.orchestra.run.smithi154:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2021-07-31T01:18:10.171 INFO:teuthology.run_tasks:Running task internal.coredump... 2021-07-31T01:18:10.211 INFO:teuthology.task.internal:Enabling coredump saving... 2021-07-31T01:18:10.212 DEBUG:teuthology.orchestra.run.smithi045:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2021-07-31T01:18:10.217 DEBUG:teuthology.orchestra.run.smithi154:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2021-07-31T01:18:10.248 INFO:teuthology.orchestra.run.smithi154.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-31T01:18:10.251 INFO:teuthology.orchestra.run.smithi045.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-31T01:18:10.257 INFO:teuthology.orchestra.run.smithi154.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-31T01:18:10.263 INFO:teuthology.orchestra.run.smithi045.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-31T01:18:10.265 INFO:teuthology.run_tasks:Running task internal.sudo... 2021-07-31T01:18:10.274 INFO:teuthology.task.internal:Configuring sudo... 2021-07-31T01:18:10.275 DEBUG:teuthology.orchestra.run.smithi045:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2021-07-31T01:18:10.308 DEBUG:teuthology.orchestra.run.smithi154:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2021-07-31T01:18:10.333 INFO:teuthology.run_tasks:Running task internal.syslog... 2021-07-31T01:18:10.347 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2021-07-31T01:18:10.347 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2021-07-31T01:18:10.376 DEBUG:teuthology.orchestra.run.smithi154:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2021-07-31T01:18:10.391 DEBUG:teuthology.orchestra.run.smithi045:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2021-07-31T01:18:10.469 DEBUG:teuthology.orchestra.run.smithi045:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2021-07-31T01:18:10.526 DEBUG:teuthology.orchestra.run.smithi045:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2021-07-31T01:18:10.602 DEBUG:teuthology.orchestra.run.smithi045:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2021-07-31T01:18:10.652 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2021-07-31T01:18:10.652 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2021-07-31T01:18:10.717 DEBUG:teuthology.orchestra.run.smithi154:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2021-07-31T01:18:10.753 DEBUG:teuthology.orchestra.run.smithi154:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2021-07-31T01:18:10.839 DEBUG:teuthology.orchestra.run.smithi154:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2021-07-31T01:18:10.875 DEBUG:teuthology.orchestra.run.smithi154:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2021-07-31T01:18:10.928 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2021-07-31T01:18:10.929 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2021-07-31T01:18:10.993 DEBUG:teuthology.orchestra.run.smithi045:> sudo service rsyslog restart 2021-07-31T01:18:10.996 DEBUG:teuthology.orchestra.run.smithi154:> sudo service rsyslog restart 2021-07-31T01:18:11.071 INFO:teuthology.orchestra.run.smithi045.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2021-07-31T01:18:11.100 INFO:teuthology.orchestra.run.smithi154.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2021-07-31T01:18:11.515 INFO:teuthology.run_tasks:Running task internal.timer... 2021-07-31T01:18:11.523 INFO:teuthology.task.internal:Starting timer... 2021-07-31T01:18:11.523 INFO:teuthology.run_tasks:Running task pcp... 2021-07-31T01:18:11.561 INFO:teuthology.run_tasks:Running task selinux... 2021-07-31T01:18:11.622 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2021-07-31T01:18:11.624 DEBUG:teuthology.orchestra.run.smithi045:> sudo service auditd rotate 2021-07-31T01:18:11.703 INFO:teuthology.orchestra.run.smithi045.stdout:Rotating logs: [ OK ] 2021-07-31T01:18:11.705 DEBUG:teuthology.orchestra.run.smithi154:> sudo service auditd rotate 2021-07-31T01:18:11.833 INFO:teuthology.orchestra.run.smithi154.stdout:Rotating logs: [ OK ] 2021-07-31T01:18:11.835 DEBUG:teuthology.task.selinux:Getting current SELinux state 2021-07-31T01:18:11.835 DEBUG:teuthology.orchestra.run.smithi045:> /usr/sbin/getenforce 2021-07-31T01:18:11.855 INFO:teuthology.orchestra.run.smithi045.stdout:Permissive 2021-07-31T01:18:11.856 DEBUG:teuthology.orchestra.run.smithi154:> /usr/sbin/getenforce 2021-07-31T01:18:11.879 INFO:teuthology.orchestra.run.smithi154.stdout:Permissive 2021-07-31T01:18:11.879 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi045.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi154.front.sepia.ceph.com': 'permissive'} 2021-07-31T01:18:11.880 DEBUG:teuthology.orchestra.run.smithi045:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|scontext=system_u:system_r:logrotate_t:s0\)' 2021-07-31T01:18:11.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-07-31T01:18:11.922 DEBUG:teuthology.orchestra.run.smithi154:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|scontext=system_u:system_r:logrotate_t:s0\)' 2021-07-31T01:18:11.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-07-31T01:18:11.948 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2021-07-31T01:18:11.948 DEBUG:teuthology.orchestra.run.smithi045:> sudo /usr/sbin/setenforce permissive 2021-07-31T01:18:11.990 DEBUG:teuthology.orchestra.run.smithi154:> sudo /usr/sbin/setenforce permissive 2021-07-31T01:18:12.020 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2021-07-31T01:18:12.040 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master was just updated or references a specific commit; assuming it is current 2021-07-31T01:18:12.040 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to origin/master 2021-07-31T01:18:12.070 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2021-07-31T01:18:12.072 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi045.front.sepia.ceph.com,smithi154.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2021-07-31T01:18:12.436 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2021-07-31T01:18:13.231 INFO:teuthology.task.ansible.out:[WARNING]: file /home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/roles/testnode/tasks/setup-opensuse.yml is empty and had no tasks to include 2021-07-31T01:18:16.035 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2021-07-31T01:18:16.058 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:18:16 +0000 (0:00:00.038) 0:00:00.038 ********* 2021-07-31T01:18:16.070 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:18:16 +0000 (0:00:00.012) 0:00:00.050 ********* 2021-07-31T01:18:17.498 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi154.front.sepia.ceph.com] 2021-07-31T01:18:17.547 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:18:17 +0000 (0:00:01.476) 0:00:01.527 ********* 2021-07-31T01:18:17.989 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** 2021-07-31T01:18:17.991 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-07-31T01:18:18.010 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:18:18 +0000 (0:00:00.463) 0:00:01.990 ********* 2021-07-31T01:18:18.550 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi045.front.sepia.ceph.com] 2021-07-31T01:18:18.598 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:18:18 +0000 (0:00:00.588) 0:00:02.578 ********* 2021-07-31T01:18:18.636 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** 2021-07-31T01:18:18.638 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2021-07-31T01:18:18.778 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:18:18 +0000 (0:00:00.179) 0:00:02.758 ********* 2021-07-31T01:18:18.795 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** 2021-07-31T01:18:18.797 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-07-31T01:18:18.828 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:18:18 +0000 (0:00:00.050) 0:00:02.809 ********* 2021-07-31T01:18:18.845 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** 2021-07-31T01:18:18.849 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:18:18.870 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:18:18 +0000 (0:00:00.041) 0:00:02.850 ********* 2021-07-31T01:18:19.118 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** 2021-07-31T01:18:19.120 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2021-07-31T01:18:19.141 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:18:19 +0000 (0:00:00.271) 0:00:03.121 ********* 2021-07-31T01:18:19.185 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** ok: [smithi154.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-07-31T01:18:19.211 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:18:19 +0000 (0:00:00.069) 0:00:03.191 ********* 2021-07-31T01:18:19.218 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** 2021-07-31T01:18:19.219 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:18:19.246 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:18:19 +0000 (0:00:00.034) 0:00:03.225 ********* 2021-07-31T01:18:19.433 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** 2021-07-31T01:18:19.435 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-07-31T01:18:19.455 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:18:19 +0000 (0:00:00.209) 0:00:03.435 ********* 2021-07-31T01:18:20.236 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** 2021-07-31T01:18:20.239 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1627694299.2558072-16208-216169401749255/source", "state": "file", "uid": 0} 2021-07-31T01:18:20.259 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:18:20 +0000 (0:00:00.803) 0:00:04.239 ********* 2021-07-31T01:18:20.267 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1627694299.5084317-16253-197011195687518/source", "state": "file", "uid": 0} 2021-07-31T01:18:20.300 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:18:20 +0000 (0:00:00.037) 0:00:04.277 ********* 2021-07-31T01:18:20.836 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Add authorized keys for the ansible user.] ************* 2021-07-31T01:18:20.838 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2021-07-31T01:18:20.839 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2021-07-31T01:18:20.864 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2021-07-31T01:18:20.866 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2021-07-31T01:18:20.940 INFO:teuthology.task.ansible.out: PLAY [teuthology] ************************************************************** skipping: no hosts matched 2021-07-31T01:18:20.943 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2021-07-31T01:18:20.955 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* 2021-07-31T01:18:20.959 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:18:20 +0000 (0:00:00.659) 0:00:04.936 ********* 2021-07-31T01:18:22.546 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] 2021-07-31T01:18:22.554 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] 2021-07-31T01:18:22.571 INFO:teuthology.task.ansible.out: TASK [set_fact] **************************************************************** 2021-07-31T01:18:22.573 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:18:22 +0000 (0:00:01.615) 0:00:06.552 ********* 2021-07-31T01:18:22.600 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"ansible_facts": {"ran_from_cephlab_playbook": true}, "changed": false} 2021-07-31T01:18:22.614 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"ansible_facts": {"ran_from_cephlab_playbook": true}, "changed": false} 2021-07-31T01:18:22.618 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2021-07-31T01:18:22.795 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:18:22 +0000 (0:00:00.223) 0:00:06.775 ********* 2021-07-31T01:18:22.824 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:18:22 +0000 (0:00:00.022) 0:00:06.798 ********* 2021-07-31T01:18:23.450 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi154.front.sepia.ceph.com] 2021-07-31T01:18:23.509 INFO:teuthology.task.ansible.out:[WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. Saturday 31 July 2021 01:18:23 +0000 (0:00:00.690) 0:00:07.488 ********* 2021-07-31T01:18:24.057 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** changed: [smithi154.front.sepia.ceph.com] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.178069", "end": "2021-07-31 01:18:24.025072", "rc": 0, "start": "2021-07-31 01:18:23.847003", "stderr": "", "stderr_lines": [], "stdout": "fb244dd758144cf455d1a3240c7f765bdfc7340a", "stdout_lines": ["fb244dd758144cf455d1a3240c7f765bdfc7340a"]} 2021-07-31T01:18:24.079 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:18:24 +0000 (0:00:00.571) 0:00:08.060 ********* 2021-07-31T01:18:24.264 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** 2021-07-31T01:18:24.266 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.009538", "end": "2021-07-31 01:18:24.247322", "failed_when_result": false, "rc": 0, "start": "2021-07-31 01:18:24.237784", "stderr": "", "stderr_lines": [], "stdout": "46fd2a03b54c2ce7520f33746fa0586027b558c3", "stdout_lines": ["46fd2a03b54c2ce7520f33746fa0586027b558c3"]} 2021-07-31T01:18:24.293 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:18:24 +0000 (0:00:00.208) 0:00:08.268 ********* 2021-07-31T01:18:24.361 INFO:teuthology.task.ansible.out: TASK [Determine if we can skip users and pubkeys updates] ********************** skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:18:24.383 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:18:24 +0000 (0:00:00.095) 0:00:08.363 ********* 2021-07-31T01:18:24.473 INFO:teuthology.task.ansible.out: TASK [Merge extra_admin_users into managed_admin_users] ************************ skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:18:24.500 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:18:24 +0000 (0:00:00.116) 0:00:08.480 ********* 2021-07-31T01:18:24.510 INFO:teuthology.task.ansible.out: 2021-07-31T01:18:24.512 INFO:teuthology.task.ansible.out:TASK [Gathering Facts] ********************************************************* ok: [smithi045.front.sepia.ceph.com] 2021-07-31T01:18:24.565 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:18:24 +0000 (0:00:00.065) 0:00:08.545 ********* 2021-07-31T01:18:24.585 INFO:teuthology.task.ansible.out: TASK [Remove managed_admin_users from managed_users] *************************** skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:18:24.614 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:18:24 +0000 (0:00:00.048) 0:00:08.594 ********* 2021-07-31T01:18:24.705 INFO:teuthology.task.ansible.out: TASK [Filter the managed_users list] ******************************************* 2021-07-31T01:18:24.707 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:18:24.729 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:18:24 +0000 (0:00:00.115) 0:00:08.710 ********* 2021-07-31T01:18:24.812 INFO:teuthology.task.ansible.out: TASK [Filter the managed_admin_users list] ************************************* 2021-07-31T01:18:24.814 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:18:24.836 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:18:24 +0000 (0:00:00.106) 0:00:08.817 ********* 2021-07-31T01:18:24.987 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** changed: [smithi045.front.sepia.ceph.com] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.175308", "end": "2021-07-31 01:18:24.964414", "rc": 0, "start": "2021-07-31 01:18:24.789106", "stderr": "", "stderr_lines": [], "stdout": "fb244dd758144cf455d1a3240c7f765bdfc7340a", "stdout_lines": ["fb244dd758144cf455d1a3240c7f765bdfc7340a"]} 2021-07-31T01:18:25.011 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:18:25 +0000 (0:00:00.174) 0:00:08.991 ********* 2021-07-31T01:18:25.073 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* ok: [smithi154.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-07-31T01:18:25.102 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:18:25 +0000 (0:00:00.087) 0:00:09.079 ********* 2021-07-31T01:18:25.224 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** 2021-07-31T01:18:25.225 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.010071", "end": "2021-07-31 01:18:25.203537", "failed_when_result": false, "rc": 0, "start": "2021-07-31 01:18:25.193466", "stderr": "", "stderr_lines": [], "stdout": "46fd2a03b54c2ce7520f33746fa0586027b558c3", "stdout_lines": ["46fd2a03b54c2ce7520f33746fa0586027b558c3"]} 2021-07-31T01:18:25.247 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:18:25 +0000 (0:00:00.148) 0:00:09.227 ********* 2021-07-31T01:18:25.315 INFO:teuthology.task.ansible.out: TASK [Determine if we can skip users and pubkeys updates] ********************** 2021-07-31T01:18:25.317 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:18:25.337 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:18:25 +0000 (0:00:00.090) 0:00:09.318 ********* 2021-07-31T01:18:25.407 INFO:teuthology.task.ansible.out: TASK [Merge extra_admin_users into managed_admin_users] ************************ skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:18:25.438 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:18:25 +0000 (0:00:00.100) 0:00:09.418 ********* 2021-07-31T01:18:25.449 INFO:teuthology.task.ansible.out: TASK [Create all admin users with sudo access.] ******************************** 2021-07-31T01:18:25.450 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2021-07-31T01:18:25.514 INFO:teuthology.task.ansible.out: TASK [Remove managed_admin_users from managed_users] *************************** skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:18:25.536 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:18:25 +0000 (0:00:00.097) 0:00:09.516 ********* 2021-07-31T01:18:25.613 INFO:teuthology.task.ansible.out: TASK [Filter the managed_users list] ******************************************* skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [Create all admin users with sudo access.] ******************************** 2021-07-31T01:18:25.615 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2021-07-31T01:18:25.635 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:18:25 +0000 (0:00:00.099) 0:00:09.615 ********* 2021-07-31T01:18:25.708 INFO:teuthology.task.ansible.out: TASK [Filter the managed_admin_users list] ************************************* 2021-07-31T01:18:25.710 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:18:25.730 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:18:25 +0000 (0:00:00.095) 0:00:09.711 ********* 2021-07-31T01:18:25.801 INFO:teuthology.task.ansible.out: TASK [Create all admin users with sudo access.] ******************************** 2021-07-31T01:18:25.803 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2021-07-31T01:18:25.972 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2021-07-31T01:18:25.974 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-07-31T01:18:25.996 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:18:25 +0000 (0:00:00.263) 0:00:09.974 ********* 2021-07-31T01:18:26.014 INFO:teuthology.task.ansible.out: TASK [Create all admin users with sudo access.] ******************************** ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2021-07-31T01:18:26.195 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2021-07-31T01:18:26.318 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2021-07-31T01:18:26.398 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2021-07-31T01:18:26.539 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2021-07-31T01:18:26.593 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2021-07-31T01:18:26.731 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2021-07-31T01:18:26.785 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2021-07-31T01:18:26.936 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2021-07-31T01:18:26.980 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2021-07-31T01:18:27.139 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2021-07-31T01:18:27.173 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2021-07-31T01:18:27.333 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2021-07-31T01:18:27.366 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2021-07-31T01:18:27.538 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2021-07-31T01:18:27.561 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2021-07-31T01:18:27.761 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2021-07-31T01:18:27.944 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2021-07-31T01:18:27.966 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2021-07-31T01:18:28.156 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2021-07-31T01:18:28.167 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2021-07-31T01:18:28.338 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2021-07-31T01:18:28.385 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2021-07-31T01:18:28.537 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2021-07-31T01:18:28.591 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2021-07-31T01:18:28.724 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1118} 2021-07-31T01:18:28.799 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2021-07-31T01:18:28.922 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2021-07-31T01:18:29.000 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2021-07-31T01:18:29.128 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2021-07-31T01:18:29.217 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2021-07-31T01:18:29.325 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1121} 2021-07-31T01:18:29.423 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2021-07-31T01:18:29.530 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1123} 2021-07-31T01:18:29.626 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2021-07-31T01:18:29.711 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1124} 2021-07-31T01:18:29.828 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1118} 2021-07-31T01:18:29.913 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1125} 2021-07-31T01:18:30.031 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2021-07-31T01:18:30.108 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1126} 2021-07-31T01:18:30.226 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2021-07-31T01:18:30.301 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1127} 2021-07-31T01:18:30.439 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1121} 2021-07-31T01:18:30.492 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1128} 2021-07-31T01:18:30.631 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1123} 2021-07-31T01:18:30.696 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1129} 2021-07-31T01:18:30.834 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1124} 2021-07-31T01:18:30.888 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1130} 2021-07-31T01:18:31.037 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1125} 2021-07-31T01:18:31.090 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1131} 2021-07-31T01:18:31.239 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1126} 2021-07-31T01:18:31.282 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1132} 2021-07-31T01:18:31.442 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1127} 2021-07-31T01:18:31.475 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1133} 2021-07-31T01:18:31.646 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1128} 2021-07-31T01:18:31.668 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1134} 2021-07-31T01:18:31.854 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1129} 2021-07-31T01:18:31.855 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1135} 2021-07-31T01:18:32.054 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1136} 2021-07-31T01:18:32.055 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1130} 2021-07-31T01:18:32.246 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1137} 2021-07-31T01:18:32.269 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1131} 2021-07-31T01:18:32.448 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-07-31T01:18:32.470 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1132} 2021-07-31T01:18:32.631 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-07-31T01:18:32.677 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1133} 2021-07-31T01:18:32.826 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1141} 2021-07-31T01:18:32.880 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1134} 2021-07-31T01:18:33.021 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1142} 2021-07-31T01:18:33.089 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1135} 2021-07-31T01:18:33.226 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1143} 2021-07-31T01:18:33.291 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1136} 2021-07-31T01:18:33.411 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1144} 2021-07-31T01:18:33.508 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1137} 2021-07-31T01:18:33.613 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1145} 2021-07-31T01:18:33.698 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-07-31T01:18:33.804 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-07-31T01:18:33.919 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-07-31T01:18:34.001 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-07-31T01:18:34.128 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1141} 2021-07-31T01:18:34.227 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-07-31T01:18:34.354 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1142} 2021-07-31T01:18:34.408 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1149} 2021-07-31T01:18:34.563 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1143} 2021-07-31T01:18:34.617 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1150} 2021-07-31T01:18:34.771 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1144} 2021-07-31T01:18:34.826 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1151} 2021-07-31T01:18:34.971 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1145} 2021-07-31T01:18:35.024 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1152} 2021-07-31T01:18:35.183 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-07-31T01:18:35.212 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1153} 2021-07-31T01:18:35.387 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-07-31T01:18:35.421 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1154} 2021-07-31T01:18:35.586 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-07-31T01:18:35.608 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1155} 2021-07-31T01:18:35.798 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1149} 2021-07-31T01:18:35.810 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1156} 2021-07-31T01:18:36.002 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1150} ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1157} 2021-07-31T01:18:36.193 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1151} 2021-07-31T01:18:36.226 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1158} 2021-07-31T01:18:36.414 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1152} 2021-07-31T01:18:36.427 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1159} 2021-07-31T01:18:36.605 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1153} 2021-07-31T01:18:36.630 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1160} 2021-07-31T01:18:36.811 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1154} 2021-07-31T01:18:36.833 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1161} 2021-07-31T01:18:37.022 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1155} 2021-07-31T01:18:37.034 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1162} 2021-07-31T01:18:37.219 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1156} ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1163} 2021-07-31T01:18:37.419 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1164} 2021-07-31T01:18:37.420 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1157} 2021-07-31T01:18:37.614 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-07-31T01:18:37.625 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1158} 2021-07-31T01:18:37.804 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1165} 2021-07-31T01:18:37.837 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1159} 2021-07-31T01:18:38.009 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1166} 2021-07-31T01:18:38.042 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1160} 2021-07-31T01:18:38.194 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1167} 2021-07-31T01:18:38.258 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1161} 2021-07-31T01:18:38.397 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1168} 2021-07-31T01:18:38.467 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1162} 2021-07-31T01:18:38.594 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1169} 2021-07-31T01:18:38.667 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1163} 2021-07-31T01:18:38.789 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1170} 2021-07-31T01:18:38.890 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1164} 2021-07-31T01:18:38.992 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1171} 2021-07-31T01:18:39.103 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-07-31T01:18:39.193 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1172} 2021-07-31T01:18:39.308 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1165} 2021-07-31T01:18:39.410 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1173} 2021-07-31T01:18:39.540 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1166} 2021-07-31T01:18:39.598 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1174} 2021-07-31T01:18:39.734 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1167} 2021-07-31T01:18:39.800 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1175} 2021-07-31T01:18:39.944 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1168} 2021-07-31T01:18:39.998 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1176} 2021-07-31T01:18:40.171 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1169} 2021-07-31T01:18:40.227 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1177} 2021-07-31T01:18:40.406 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1170} 2021-07-31T01:18:40.455 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1178} 2021-07-31T01:18:40.602 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1171} 2021-07-31T01:18:40.669 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1179} 2021-07-31T01:18:40.819 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1172} 2021-07-31T01:18:40.870 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1180} 2021-07-31T01:18:41.034 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1173} 2021-07-31T01:18:41.070 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1181} 2021-07-31T01:18:41.227 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1174} 2021-07-31T01:18:41.275 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1182} 2021-07-31T01:18:41.435 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1175} 2021-07-31T01:18:41.481 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1183} 2021-07-31T01:18:41.643 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1176} 2021-07-31T01:18:41.679 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1184} 2021-07-31T01:18:41.865 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1177} 2021-07-31T01:18:41.893 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1185} 2021-07-31T01:18:42.065 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1178} 2021-07-31T01:18:42.077 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1186} 2021-07-31T01:18:42.282 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1187} 2021-07-31T01:18:42.283 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1179} 2021-07-31T01:18:42.479 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1188} 2021-07-31T01:18:42.480 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1180} 2021-07-31T01:18:42.684 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1189} 2021-07-31T01:18:42.694 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1181} 2021-07-31T01:18:42.895 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1190} 2021-07-31T01:18:42.914 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1182} 2021-07-31T01:18:43.086 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1191} 2021-07-31T01:18:43.134 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1183} 2021-07-31T01:18:43.286 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1192} 2021-07-31T01:18:43.342 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1184} 2021-07-31T01:18:43.493 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1193} 2021-07-31T01:18:43.546 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1185} 2021-07-31T01:18:43.690 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1194} 2021-07-31T01:18:43.754 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1186} 2021-07-31T01:18:43.893 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1195} 2021-07-31T01:18:43.982 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1187} 2021-07-31T01:18:44.089 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1196} 2021-07-31T01:18:44.195 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1188} 2021-07-31T01:18:44.293 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1197} 2021-07-31T01:18:44.400 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1189} 2021-07-31T01:18:44.495 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1198} 2021-07-31T01:18:44.606 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1190} 2021-07-31T01:18:44.695 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1199} 2021-07-31T01:18:44.824 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1191} 2021-07-31T01:18:44.900 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1200} 2021-07-31T01:18:45.040 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1192} 2021-07-31T01:18:45.103 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1201} 2021-07-31T01:18:45.250 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1193} 2021-07-31T01:18:45.316 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-07-31T01:18:45.456 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1194} 2021-07-31T01:18:45.523 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1202} 2021-07-31T01:18:45.677 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1195} 2021-07-31T01:18:45.741 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1203} 2021-07-31T01:18:45.899 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1196} 2021-07-31T01:18:45.966 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1204} 2021-07-31T01:18:46.129 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1197} 2021-07-31T01:18:46.170 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1205} 2021-07-31T01:18:46.321 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1198} 2021-07-31T01:18:46.376 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1206} 2021-07-31T01:18:46.525 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1199} 2021-07-31T01:18:46.580 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1207} 2021-07-31T01:18:46.732 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1200} 2021-07-31T01:18:46.779 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1208} 2021-07-31T01:18:46.935 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1201} 2021-07-31T01:18:46.991 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1209} 2021-07-31T01:18:47.143 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-07-31T01:18:47.185 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1210} 2021-07-31T01:18:47.344 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1202} 2021-07-31T01:18:47.377 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1211} 2021-07-31T01:18:47.564 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1203} 2021-07-31T01:18:47.575 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1212} 2021-07-31T01:18:47.775 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1204} 2021-07-31T01:18:47.788 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1213} 2021-07-31T01:18:47.982 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1205} 2021-07-31T01:18:47.994 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1214} 2021-07-31T01:18:48.197 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1206} 2021-07-31T01:18:48.209 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1215} 2021-07-31T01:18:48.211 INFO:teuthology.task.ansible.out: 2021-07-31T01:18:48.417 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1207} 2021-07-31T01:18:48.418 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1216} 2021-07-31T01:18:48.618 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1217} 2021-07-31T01:18:48.619 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1208} 2021-07-31T01:18:48.811 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1218} 2021-07-31T01:18:48.833 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1209} 2021-07-31T01:18:49.041 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1219, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "move_home": false, "name": "sunny", "shell": "/bin/bash", "state": "present", "uid": 1219} 2021-07-31T01:18:49.069 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1210} 2021-07-31T01:18:49.229 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1220, "groups": "sudo", "home": "/home/xuxuehan", "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "move_home": false, "name": "xuxuehan", "shell": "/bin/bash", "state": "present", "uid": 1220} 2021-07-31T01:18:49.264 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1211} 2021-07-31T01:18:49.426 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1221, "groups": "sudo", "home": "/home/aclewett", "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "move_home": false, "name": "aclewett", "shell": "/bin/bash", "state": "present", "uid": 1221} 2021-07-31T01:18:49.491 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1212} 2021-07-31T01:18:49.617 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1222, "groups": "sudo", "home": "/home/cblum", "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "move_home": false, "name": "cblum", "shell": "/bin/bash", "state": "present", "uid": 1222} 2021-07-31T01:18:49.684 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1213} 2021-07-31T01:18:49.822 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1223, "groups": "sudo", "home": "/home/mlatimer", "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "move_home": false, "name": "mlatimer", "shell": "/bin/bash", "state": "present", "uid": 1223} 2021-07-31T01:18:49.897 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1214} 2021-07-31T01:18:50.024 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1224, "groups": "sudo", "home": "/home/shyamsundarr", "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "move_home": false, "name": "shyamsundarr", "shell": "/bin/bash", "state": "present", "uid": 1224} 2021-07-31T01:18:50.102 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1215} 2021-07-31T01:18:50.230 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1225, "groups": "sudo", "home": "/home/shuzhenyi", "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "move_home": false, "name": "shuzhenyi", "shell": "/bin/bash", "state": "present", "uid": 1225} 2021-07-31T01:18:50.315 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1216} 2021-07-31T01:18:50.416 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1226, "groups": "sudo", "home": "/home/mdipalma", "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "move_home": false, "name": "mdipalma", "shell": "/bin/bash", "state": "present", "uid": 1226} 2021-07-31T01:18:50.518 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1217} 2021-07-31T01:18:50.612 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1227, "groups": "sudo", "home": "/home/kchang", "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "move_home": false, "name": "kchang", "shell": "/bin/bash", "state": "present", "uid": 1227} 2021-07-31T01:18:50.719 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1218} 2021-07-31T01:18:50.815 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1228, "groups": "sudo", "home": "/home/vereddy", "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "move_home": false, "name": "vereddy", "shell": "/bin/bash", "state": "present", "uid": 1228} 2021-07-31T01:18:50.920 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1219, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "move_home": false, "name": "sunny", "shell": "/bin/bash", "state": "present", "uid": 1219} 2021-07-31T01:18:51.007 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1229, "groups": "sudo", "home": "/home/benhanokh", "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "move_home": false, "name": "benhanokh", "shell": "/bin/bash", "state": "present", "uid": 1229} 2021-07-31T01:18:51.134 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1220, "groups": "sudo", "home": "/home/xuxuehan", "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "move_home": false, "name": "xuxuehan", "shell": "/bin/bash", "state": "present", "uid": 1220} 2021-07-31T01:18:51.220 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1230, "groups": "sudo", "home": "/home/jsalomon", "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "move_home": false, "name": "jsalomon", "shell": "/bin/bash", "state": "present", "uid": 1230} 2021-07-31T01:18:51.377 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1221, "groups": "sudo", "home": "/home/aclewett", "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "move_home": false, "name": "aclewett", "shell": "/bin/bash", "state": "present", "uid": 1221} 2021-07-31T01:18:51.424 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1231, "groups": "sudo", "home": "/home/jolmomar", "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "move_home": false, "name": "jolmomar", "shell": "/bin/bash", "state": "present", "uid": 1231} 2021-07-31T01:18:51.600 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1222, "groups": "sudo", "home": "/home/cblum", "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "move_home": false, "name": "cblum", "shell": "/bin/bash", "state": "present", "uid": 1222} 2021-07-31T01:18:51.656 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1232, "groups": "sudo", "home": "/home/hmunjulu", "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "move_home": false, "name": "hmunjulu", "shell": "/bin/bash", "state": "present", "uid": 1232} 2021-07-31T01:18:51.819 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1223, "groups": "sudo", "home": "/home/mlatimer", "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "move_home": false, "name": "mlatimer", "shell": "/bin/bash", "state": "present", "uid": 1223} 2021-07-31T01:18:51.855 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1233, "groups": "sudo", "home": "/home/julpark", "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "move_home": false, "name": "julpark", "shell": "/bin/bash", "state": "present", "uid": 1233} 2021-07-31T01:18:52.037 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1224, "groups": "sudo", "home": "/home/shyamsundarr", "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "move_home": false, "name": "shyamsundarr", "shell": "/bin/bash", "state": "present", "uid": 1224} 2021-07-31T01:18:52.084 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1235, "groups": "sudo", "home": "/home/pdhange", "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "move_home": false, "name": "pdhange", "shell": "/bin/bash", "state": "present", "uid": 1235} 2021-07-31T01:18:52.086 INFO:teuthology.task.ansible.out: 2021-07-31T01:18:52.249 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1225, "groups": "sudo", "home": "/home/shuzhenyi", "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "move_home": false, "name": "shuzhenyi", "shell": "/bin/bash", "state": "present", "uid": 1225} 2021-07-31T01:18:52.471 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1226, "groups": "sudo", "home": "/home/mdipalma", "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "move_home": false, "name": "mdipalma", "shell": "/bin/bash", "state": "present", "uid": 1226} 2021-07-31T01:18:52.681 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1227, "groups": "sudo", "home": "/home/kchang", "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "move_home": false, "name": "kchang", "shell": "/bin/bash", "state": "present", "uid": 1227} 2021-07-31T01:18:52.886 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1228, "groups": "sudo", "home": "/home/vereddy", "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "move_home": false, "name": "vereddy", "shell": "/bin/bash", "state": "present", "uid": 1228} 2021-07-31T01:18:53.086 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1229, "groups": "sudo", "home": "/home/benhanokh", "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "move_home": false, "name": "benhanokh", "shell": "/bin/bash", "state": "present", "uid": 1229} 2021-07-31T01:18:53.306 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1230, "groups": "sudo", "home": "/home/jsalomon", "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "move_home": false, "name": "jsalomon", "shell": "/bin/bash", "state": "present", "uid": 1230} 2021-07-31T01:18:53.510 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1231, "groups": "sudo", "home": "/home/jolmomar", "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "move_home": false, "name": "jolmomar", "shell": "/bin/bash", "state": "present", "uid": 1231} 2021-07-31T01:18:53.727 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1232, "groups": "sudo", "home": "/home/hmunjulu", "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "move_home": false, "name": "hmunjulu", "shell": "/bin/bash", "state": "present", "uid": 1232} 2021-07-31T01:18:53.732 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10102, "groups": "sudo", "home": "/home/dpivonka", "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "name": "dpivonka", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10102} 2021-07-31T01:18:53.935 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1233, "groups": "sudo", "home": "/home/julpark", "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "move_home": false, "name": "julpark", "shell": "/bin/bash", "state": "present", "uid": 1233} 2021-07-31T01:18:54.137 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1235, "groups": "sudo", "home": "/home/pdhange", "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "move_home": false, "name": "pdhange", "shell": "/bin/bash", "state": "present", "uid": 1235} 2021-07-31T01:18:55.424 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10103, "groups": "sudo", "home": "/home/adking", "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "name": "adking", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10103} 2021-07-31T01:18:55.780 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10102, "groups": "sudo", "home": "/home/dpivonka", "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "name": "dpivonka", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10102} 2021-07-31T01:18:57.011 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10104, "groups": "sudo", "home": "/home/amathuri", "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "name": "amathuri", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10104} 2021-07-31T01:18:57.578 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10103, "groups": "sudo", "home": "/home/adking", "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "name": "adking", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10103} 2021-07-31T01:18:58.565 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10105, "groups": "sudo", "home": "/home/nizamial09", "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "name": "nizamial09", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10105} 2021-07-31T01:18:59.167 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10104, "groups": "sudo", "home": "/home/amathuri", "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "name": "amathuri", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10104} 2021-07-31T01:19:00.079 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10106, "groups": "sudo", "home": "/home/dachary", "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "name": "dachary", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10106} 2021-07-31T01:19:00.758 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10105, "groups": "sudo", "home": "/home/nizamial09", "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "name": "nizamial09", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10105} 2021-07-31T01:19:01.605 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10107, "groups": "sudo", "home": "/home/amctagga", "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "name": "amctagga", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10107} 2021-07-31T01:19:02.332 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10106, "groups": "sudo", "home": "/home/dachary", "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "name": "dachary", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10106} 2021-07-31T01:19:03.138 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10108, "groups": "sudo", "home": "/home/kevinz", "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "name": "kevinz", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10108} 2021-07-31T01:19:03.919 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10107, "groups": "sudo", "home": "/home/amctagga", "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "name": "amctagga", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10107} 2021-07-31T01:19:04.745 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10109, "groups": "sudo", "home": "/home/liu-chunmei", "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "name": "liu-chunmei", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10109} 2021-07-31T01:19:05.482 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10108, "groups": "sudo", "home": "/home/kevinz", "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "name": "kevinz", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10108} 2021-07-31T01:19:06.334 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10110, "groups": "sudo", "home": "/home/cyx1231st", "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "name": "cyx1231st", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10110} 2021-07-31T01:19:07.120 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10109, "groups": "sudo", "home": "/home/liu-chunmei", "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "name": "liu-chunmei", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10109} 2021-07-31T01:19:07.868 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10111, "groups": "sudo", "home": "/home/sunnagar", "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "name": "sunnagar", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10111} 2021-07-31T01:19:08.678 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10110, "groups": "sudo", "home": "/home/cyx1231st", "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "name": "cyx1231st", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10110} 2021-07-31T01:19:09.439 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10112, "groups": "sudo", "home": "/home/lflores", "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "name": "lflores", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10112} 2021-07-31T01:19:10.369 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10111, "groups": "sudo", "home": "/home/sunnagar", "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "name": "sunnagar", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10111} 2021-07-31T01:19:11.072 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10113, "groups": "sudo", "home": "/home/jsawaya", "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "name": "jsawaya", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10113} 2021-07-31T01:19:12.113 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10112, "groups": "sudo", "home": "/home/lflores", "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "name": "lflores", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10112} 2021-07-31T01:19:12.670 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10114, "groups": "sudo", "home": "/home/markw8973240", "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "name": "markw8973240", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10114} 2021-07-31T01:19:13.765 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10113, "groups": "sudo", "home": "/home/jsawaya", "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "name": "jsawaya", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10113} 2021-07-31T01:19:14.249 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10115, "groups": "sudo", "home": "/home/luisrico", "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "name": "luisrico", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10115} 2021-07-31T01:19:15.390 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10114, "groups": "sudo", "home": "/home/markw8973240", "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "name": "markw8973240", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10114} 2021-07-31T01:19:15.824 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10116, "groups": "sudo", "home": "/home/matan", "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e"}, "name": "matan", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10116} 2021-07-31T01:19:17.103 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10115, "groups": "sudo", "home": "/home/luisrico", "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "name": "luisrico", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10115} 2021-07-31T01:19:17.439 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10117, "groups": "sudo", "home": "/home/ozeneva", "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "name": "ozeneva", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10117} 2021-07-31T01:19:18.736 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10116, "groups": "sudo", "home": "/home/matan", "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e"}, "name": "matan", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10116} 2021-07-31T01:19:19.104 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10118, "groups": "sudo", "home": "/home/sudhanaruna", "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "name": "sudhanaruna", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10118} 2021-07-31T01:19:20.379 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10117, "groups": "sudo", "home": "/home/ozeneva", "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "name": "ozeneva", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10117} 2021-07-31T01:19:20.723 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10119, "groups": "sudo", "home": "/home/zl31wang", "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "name": "zl31wang", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10119} 2021-07-31T01:19:21.994 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10118, "groups": "sudo", "home": "/home/sudhanaruna", "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "name": "sudhanaruna", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10118} 2021-07-31T01:19:22.405 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10120, "groups": "sudo", "home": "/home/mayagilad", "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "name": "mayagilad", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10120} 2021-07-31T01:19:22.568 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:19:22 +0000 (0:00:56.568) 0:01:06.543 ********* 2021-07-31T01:19:22.636 INFO:teuthology.task.ansible.out: TASK [Create all users without sudo access.] *********************************** 2021-07-31T01:19:22.657 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:19:22 +0000 (0:00:00.093) 0:01:06.637 ********* 2021-07-31T01:19:22.752 INFO:teuthology.task.ansible.out: TASK [Merge managed_users and managed_admin users] ***************************** 2021-07-31T01:19:22.753 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratza@fedora 2021-07-31T01:19:22.755 INFO:teuthology.task.ansible.out:rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265 2021-07-31T01:19:22.757 INFO:teuthology.task.ansible.out:c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop K 2021-07-31T01:19:22.758 INFO:teuthology.task.ansible.out:D0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9 2021-07-31T01:19:22.760 INFO:teuthology.task.ansible.out:df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9 2021-07-31T01:19:22.761 INFO:teuthology.task.ansible.out:b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", " 2021-07-31T01:19:22.763 INFO:teuthology.task.ansible.out:ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn" 2021-07-31T01:19:22.764 INFO:teuthology.task.ansible.out:: "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 2021-07-31T01:19:22.766 INFO:teuthology.task.ansible.out:f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": 2021-07-31T01:19:22.769 INFO:teuthology.task.ansible.out:"alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, {"name": "gsalomon 2021-07-31T01:19:22.773 INFO:teuthology.task.ansible.out:", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"nam 2021-07-31T01:19:22.775 INFO:teuthology.task.ansible.out:e": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e104 2021-07-31T01:19:22.776 INFO:teuthology.task.ansible.out:6655420d4d0a79ce5667bec1b469a0100"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449 2021-07-31T01:19:22.778 INFO:teuthology.task.ansible.out:ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, {"name": "jsalomon", "ovpn 2021-07-31T01:19:22.781 INFO:teuthology.task.ansible.out:": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185 2021-07-31T01:19:22.783 INFO:teuthology.task.ansible.out:052887b82ba5c343a42eb271747a"}, {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e"}, {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}]}, "changed": false} 2021-07-31T01:19:22.805 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:19:22 +0000 (0:00:00.144) 0:01:06.782 ********* 2021-07-31T01:19:23.747 INFO:teuthology.task.ansible.out: TASK [Create all admin users with sudo access.] ******************************** 2021-07-31T01:19:23.748 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10119, "groups": "sudo", "home": "/home/zl31wang", "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "name": "zl31wang", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10119} 2021-07-31T01:19:23.821 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2021-07-31T01:19:23.823 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"after": "fb244dd758144cf455d1a3240c7f765bdfc7340a", "attempts": 1, "before": "fb244dd758144cf455d1a3240c7f765bdfc7340a", "changed": false, "remote_url_changed": false} 2021-07-31T01:19:23.844 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:19:23 +0000 (0:00:01.041) 0:01:07.824 ********* 2021-07-31T01:19:24.215 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-07-31T01:19:24.217 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDdkebJGXKQWBG/7wZG334P+EAdC+5VNMrjLjXhaZZbt3V9vKDQCodcU3zSOjMa/GDGg2WHQdO/vioCc6O/m61KTbgLDaaPSzHH16j/e1SaU8BIgKKU42AUBJzpPeaZ2wCC1i12Hq8ULwjEW29IDsnMortzlCSQmaTKc3DN2nCXZWQXA1rSiQ2AhIq5WNJY9S36lsg5tkex4xUajyNW4vPubLtkSv+WrzNK7ZJ8zfzP6eJEJ+Uj7CLdanBCgwoLGSPfGA5c/gyUT7mHibOUwvFB5sFVsozMPp2PXzUUIv70adJpOskLYfKY7TUSuaB4AUOPu6NVauzVIl3Tsn+uvHqvtn8KC3BPpyH7+JiFQWEt+aO+/ynprfoqCjktuf07SXo0ewoVYT2S++JEBa/N2hKRkzIoSPJ+Rvoazc8GZIjXpQRyhMdTV/LCSeyUTJ7lDQVrQVhUIHKnMm+hZmf4tLQKZvNAThO984CpY93ST5Ww07fjRLMXNZGuaz+zHKNexhWSPKr+ggChnZCOLl/F6zbDnBZoCaHR6hH4+QLz0K/FAgxN2Nn00RiOFkmIlzEioxMcwC/peTl3WzRfqPINdTftiTzcFi/c2QmlHhnxsueIwNhHOn4frkGkFmexdDrXRStERzSBJ13X4kWduPmQsn+bx5/7xwqC6M4bXpkYFOefOw== aschoen@redhat.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2021-07-31T01:19:24.449 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2021-07-31T01:19:24.689 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2021-07-31T01:19:24.927 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZ 2021-07-31T01:19:24.928 INFO:teuthology.task.ansible.out:tIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2021-07-31T01:19:25.169 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2021-07-31T01:19:25.411 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+ 2021-07-31T01:19:25.413 INFO:teuthology.task.ansible.out:o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2021-07-31T01:19:25.498 INFO:teuthology.task.ansible.out: TASK [Create all admin users with sudo access.] ******************************** 2021-07-31T01:19:25.500 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10120, "groups": "sudo", "home": "/home/mayagilad", "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "name": "mayagilad", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10120} 2021-07-31T01:19:25.641 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:19:25 +0000 (0:00:01.796) 0:01:09.621 ********* 2021-07-31T01:19:25.648 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-07-31T01:19:25.649 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2021-07-31T01:19:25.705 INFO:teuthology.task.ansible.out: TASK [Create all users without sudo access.] *********************************** 2021-07-31T01:19:25.731 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:19:25 +0000 (0:00:00.089) 0:01:09.711 ********* 2021-07-31T01:19:25.824 INFO:teuthology.task.ansible.out: TASK [Merge managed_users and managed_admin users] ***************************** 2021-07-31T01:19:25.825 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratza@fedora 2021-07-31T01:19:25.827 INFO:teuthology.task.ansible.out:rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265 2021-07-31T01:19:25.828 INFO:teuthology.task.ansible.out:c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop K 2021-07-31T01:19:25.830 INFO:teuthology.task.ansible.out:D0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\ 2021-07-31T01:19:25.830 INFO:teuthology.task.ansible.out:nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9 2021-07-31T01:19:25.833 INFO:teuthology.task.ansible.out:df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9 2021-07-31T01:19:25.834 INFO:teuthology.task.ansible.out:b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", " 2021-07-31T01:19:25.836 INFO:teuthology.task.ansible.out:ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 2021-07-31T01:19:25.838 INFO:teuthology.task.ansible.out:f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": 2021-07-31T01:19:25.839 INFO:teuthology.task.ansible.out:"alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc99 2021-07-31T01:19:25.840 INFO:teuthology.task.ansible.out:3327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, {"name": "gsalomon 2021-07-31T01:19:25.842 INFO:teuthology.task.ansible.out:", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"nam 2021-07-31T01:19:25.844 INFO:teuthology.task.ansible.out:e": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449 2021-07-31T01:19:25.846 INFO:teuthology.task.ansible.out:ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, {"name": "jsalomon", "ovpn 2021-07-31T01:19:25.848 INFO:teuthology.task.ansible.out:": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8yt 2021-07-31T01:19:25.848 INFO:teuthology.task.ansible.out:ILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185 2021-07-31T01:19:25.850 INFO:teuthology.task.ansible.out:052887b82ba5c343a42eb271747a"}, {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e"}, {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}]}, "changed": false} 2021-07-31T01:19:25.875 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:19:25 +0000 (0:00:00.144) 0:01:09.855 ********* 2021-07-31T01:19:25.892 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-07-31T01:19:25.893 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCgaz5AsLB+hPwnHjlf6vktlUQ8iWsKA+hViok+j+uCm0U/iho7O7Qd8E/3tp9yBx10Wkvuz1mmf6v2AO7Qa8+SE1QQPmA+4xLp4xXCqlUGFjnzm/7uQcQZtlWdfBHzQ9jVyPAgUpnGVUtW8PhW0txe8MEtecOO2C3Snx0yuTOx6O11jNvT938QjOeRQWw3X2YFUtMIOiwEhKtqstBIGXlMdaQKfvdlbA9MjyYy6VfK1x8lVcu8umUf/z56YmMkC2iO117/obhdu1XsU9SmjM51wHMOUCXSVqH8E3BWu0yohy4QmZDkdFihyYa+hoS9dOqGC91dFaQc46xrdmV4wqg1 jcaratza@MacBook-Pro\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAV78mqEy/VkB2BthIE/m7iE5sbTjTRS0KuH7AB0KPvZBsDIw5cL8jKyxiaPC85+aOkBa4LrP0TUFBzN87Snt+PLMz2QHBOvlSTfUuRkHVo3vWJFzVuUSzzddkYds3hAxUriAHKv6CRSy1yFBxPQQHTSATUHKWqihMElbUCZrrrfH8A7J9vwixzCSLg4KbINjfa77rxefcZwugUqiYiVUWUSOb023f+QEaDm7OP+/uzz3SjF1Y0LMwa75yctttIfkRBKjY7uFCuZdKc3l2A/fdPxbFwlZmIox3LLngdTzHMVXGbVHqq9Ixo8FSAXiI1Buy2LTMoXMJwQYSvzsv7xNwujKVlUjoE9cOOIdzCC5AziwSiJB4+tNh73c5tRUzMMC98HLOhPJriSPvOt1pMEdWDX3/evcyxvRgJUKgYnmQG7n8v8O/CLyG0sAJDmGC3U9S/3fUDUjZIKWSsc85Q3xRaJwZJIS2VZwH0zG1Gmynih8PPq7VeDx7tKuNm4uTYT8= bigjust@desktop-jsho3l9.lan", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2021-07-31T01:19:26.128 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2021-07-31T01:19:26.139 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:19:26.372 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMor 2021-07-31T01:19:26.374 INFO:teuthology.task.ansible.out:bnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2021-07-31T01:19:26.519 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2021-07-31T01:19:26.521 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"after": "fb244dd758144cf455d1a3240c7f765bdfc7340a", "attempts": 1, "before": "fb244dd758144cf455d1a3240c7f765bdfc7340a", "changed": false, "remote_url_changed": false} 2021-07-31T01:19:26.544 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:19:26 +0000 (0:00:00.667) 0:01:10.523 ********* 2021-07-31T01:19:26.614 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-07-31T01:19:26.615 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvG6+jM8aulc5CID/yE6AXPXczxq8tMgY+2sdo/ihWpgDZuFwGL4hwj8nyh5cTqZ2LEa4kRkUUIoH9avLCFN5pDBVvcj1/O06ZGnkq/hV6791UiPF5S5RGmz3MGxv0RDfjA0OLFav7u1s+ZJ6z2W1Tpy/W+g/fMYqbPc6rV+pm3Uo16+BWz/diZH8smqcty4chnnCy6gqOrcAxCZLpeyjX6Ydn3fnTLTAmhoQ9eUz7LjP4ZnFD5zL6jrSBlJ5WNTLevUzvS8eL7RuP8D3Qfn8ukIzPXG2vs594FhSc/HnvubKricvKMm0P0ki4Z0vM9VruQn7Tbeqdwpi1IfgyiAMX aemerson@work-yubi\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8/FJn7hz2WieoQFUGXS0bhwSwL1LCI9kkjWYNkWb8IJPrBetlkr5TsvHEKDPZGF5Sv8LsYA6njPnbOGyAHpYlnv/Gz4Hwq3hLgmXCs6JGWBQv8NS+pjwnG6Nh6JfFI0ANeg652Rk9Qt9uLYeKVKmHXXrVZAD4NMvJGfEvneQzuwwpylUmV80WAdaQw/H0EP4NScZhNhknwAs/+KPFZH+RFoUj/Cb3poIFjwyviIgyal5hjAEsoczCW3hAla/9IyC2VDS027WPiPFmBFaYK5lYYLStBQdRx+6amH76XScLFy0FEPLL5qiN0ClAptvh1oIURgipdXZwaxpf06AXG8qw== aemerson@electric", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemer 2021-07-31T01:19:26.617 INFO:teuthology.task.ansible.out:son", "validate_certs": true} 2021-07-31T01:19:26.625 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:19:26.627 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:19:26.874 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2021-07-31T01:19:26.919 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDdkebJGXKQWBG/7wZG334P+EAdC+5VNMrjLjXhaZZbt3V9vKDQCodcU3zSOjMa/GDGg2WHQdO/vioCc6O/m61KTbgLDaaPSzHH16j/e1SaU8BIgKKU42AUBJzpPeaZ2wCC1i12Hq8ULwjEW29IDsnMortzlCSQmaTKc3DN2nCXZWQXA1rSiQ2AhIq5WNJY9S36lsg5tkex4xUajyNW4vPubLtkSv+WrzNK7ZJ8zfzP6eJEJ+Uj7CLdanBCgwoLGSPfGA5c/gyUT7mHibOUwvFB5sFVsozMPp2PXzUUIv70adJpOskLYfKY7TUSuaB4AUOPu6NVauzVIl3Tsn+uvHqvtn8KC3BPpyH7+JiFQWEt+aO+/ynprfoqCjktuf07SXo0ewoVYT2S++JEBa/N2hKRkzIoSPJ+Rvoazc8GZIjXpQRyhMdTV/LCSeyUTJ7lDQVrQVhUIHKnMm+hZmf4tLQKZvNAThO984CpY93ST5Ww07fjRLMXNZGuaz+zHKNexhWSPKr+ggChnZCOLl/F6zbDnBZoCaHR6hH4+QLz0K/FAgxN2Nn00RiOFkmIlzEioxMcwC/peTl3WzRfqPINdTftiTzcFi/c2QmlHhnxsueIwNhHOn4frkGkFmexdDrXRStERzSBJ13X4kWduPmQsn+bx5/7xwqC6M4bXpkYFOefOw== aschoen@redhat.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2021-07-31T01:19:27.109 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtS 2021-07-31T01:19:27.111 INFO:teuthology.task.ansible.out:oAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2021-07-31T01:19:27.162 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2021-07-31T01:19:27.341 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2021-07-31T01:19:27.415 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2021-07-31T01:19:27.565 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": 2021-07-31T01:19:27.567 INFO:teuthology.task.ansible.out:null, "state": "present", "user": "davidz", "validate_certs": true} 2021-07-31T01:19:27.659 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZ 2021-07-31T01:19:27.661 INFO:teuthology.task.ansible.out:tIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2021-07-31T01:19:27.805 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2021-07-31T01:19:27.912 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2021-07-31T01:19:28.038 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2021-07-31T01:19:28.156 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+ 2021-07-31T01:19:28.158 INFO:teuthology.task.ansible.out:o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2021-07-31T01:19:28.272 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2021-07-31T01:19:28.409 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2021-07-31T01:19:28.504 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9 2021-07-31T01:19:28.506 INFO:teuthology.task.ansible.out:CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2021-07-31T01:19:28.662 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCgaz5AsLB+hPwnHjlf6vktlUQ8iWsKA+hViok+j+uCm0U/iho7O7Qd8E/3tp9yBx10Wkvuz1mmf6v2AO7Qa8+SE1QQPmA+4xLp4xXCqlUGFjnzm/7uQcQZtlWdfBHzQ9jVyPAgUpnGVUtW8PhW0txe8MEtecOO2C3Snx0yuTOx6O11jNvT938QjOeRQWw3X2YFUtMIOiwEhKtqstBIGXlMdaQKfvdlbA9MjyYy6VfK1x8lVcu8umUf/z56YmMkC2iO117/obhdu1XsU9SmjM51wHMOUCXSVqH8E3BWu0yohy4QmZDkdFihyYa+hoS9dOqGC91dFaQc46xrdmV4wqg1 jcaratza@MacBook-Pro\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAV78mqEy/VkB2BthIE/m7iE5sbTjTRS0KuH7AB0KPvZBsDIw5cL8jKyxiaPC85+aOkBa4LrP0TUFBzN87Snt+PLMz2QHBOvlSTfUuRkHVo3vWJFzVuUSzzddkYds3hAxUriAHKv6CRSy1yFBxPQQHTSATUHKWqihMElbUCZrrrfH8A7J9vwixzCSLg4KbINjfa77rxefcZwugUqiYiVUWUSOb023f+QEaDm7OP+/uzz3SjF1Y0LMwa75yctttIfkRBKjY7uFCuZdKc3l2A/fdPxbFwlZmIox3LLngdTzHMVXGbVHqq9Ixo8FSAXiI1Buy2LTMoXMJwQYSvzsv7xNwujKVlUjoE9cOOIdzCC5AziwSiJB4+tNh73c5tRUzMMC98HLOhPJriSPvOt1pMEdWDX3/evcyxvRgJUKgYnmQG7n8v8O/CLyG0sAJDmGC3U9S/3fUDUjZIKWSsc85Q3xRaJwZJIS2VZwH0zG1Gmynih8PPq7VeDx7tKuNm4uTYT8= bigjust@desktop-jsho3l9.lan", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2021-07-31T01:19:28.736 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBY 2021-07-31T01:19:28.738 INFO:teuthology.task.ansible.out:B0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2021-07-31T01:19:28.914 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2021-07-31T01:19:28.924 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:19:28.967 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2021-07-31T01:19:29.167 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMor 2021-07-31T01:19:29.169 INFO:teuthology.task.ansible.out:bnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2021-07-31T01:19:29.210 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2021-07-31T01:19:29.411 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvG6+jM8aulc5CID/yE6AXPXczxq8tMgY+2sdo/ihWpgDZuFwGL4hwj8nyh5cTqZ2LEa4kRkUUIoH9avLCFN5pDBVvcj1/O06ZGnkq/hV6791UiPF5S5RGmz3MGxv0RDfjA0OLFav7u1s+ZJ6z2W1Tpy/W+g/fMYqbPc6rV+pm3Uo16+BWz/diZH8smqcty4chnnCy6gqOrcAxCZLpeyjX6Ydn3fnTLTAmhoQ9eUz7LjP4ZnFD5zL6jrSBlJ5WNTLevUzvS8eL7RuP8D3Qfn8ukIzPXG2vs594FhSc/HnvubKricvKMm0P0ki4Z0vM9VruQn7Tbeqdwpi1IfgyiAMX aemerson@work-yubi\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8/FJn7hz2WieoQFUGXS0bhwSwL1LCI9kkjWYNkWb8IJPrBetlkr5TsvHEKDPZGF5Sv8LsYA6njPnbOGyAHpYlnv/Gz4Hwq3hLgmXCs6JGWBQv8NS+pjwnG6Nh6JfFI0ANeg652Rk9Qt9uLYeKVKmHXXrVZAD4NMvJGfEvneQzuwwpylUmV80WAdaQw/H0EP4NScZhNhknwAs/+KPFZH+RFoUj/Cb3poIFjwyviIgyal5hjAEsoczCW3hAla/9IyC2VDS027WPiPFmBFaYK5lYYLStBQdRx+6amH76XScLFy0FEPLL5qiN0ClAptvh1oIURgipdXZwaxpf06AXG8qw== aemerson@electric", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemer 2021-07-31T01:19:29.413 INFO:teuthology.task.ansible.out:son", "validate_certs": true} 2021-07-31T01:19:29.422 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:19:29.423 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:19:29.457 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7 2021-07-31T01:19:29.459 INFO:teuthology.task.ansible.out:gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrSX8nDL2WaHe0b/rjE3TZhj5VwbrG6nn/hyWmm2dWcUqotSp973yD2G4tQL9UygeSkDGmjb6whrz2mcAFJ5P4oLtIcCLyZC550dcpwV3BtP/cMAIGE6xeEARKiP8KlVGFNDRg9/l1aAd08Bepe+rJyAYD7IxOOBfgKJMKbeOWnT40RTfwiD5CXNwjzQuhzZDEbm+8zjTChFM4P1EkfR3c+0X3MYpA1AsRQY0ElvFtcn6G0yPKR+obGdIs+ydGupKh1+peGWLemHMA5e62Bs4/Lwtyebyp7AOguC3ZBJ3TZWL4Vn1q6z21cQe6AuowjPrneZWC4ifFB6OUQJqM+R00pHBcajcvsfpYOblZvTJHM726Ek1vahOYfcbEMpqWlVYo+WPRUilzl5cI3ReyAcwphebqKPbhmzCuyJw3lWBKIVDhWjEFzGkxV/iFU5rIXFm/Z0JGNFFxmY1U4GlA4lFXtL7OvEdMsiZo1x0cNmFWLu0k9fQBtiQI19ZBiA7FPKc= yehuda@crestview", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2021-07-31T01:19:29.679 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2021-07-31T01:19:29.703 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2021-07-31T01:19:29.917 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtS 2021-07-31T01:19:29.919 INFO:teuthology.task.ansible.out:oAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2021-07-31T01:19:29.939 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2021-07-31T01:19:30.172 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2021-07-31T01:19:30.184 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2021-07-31T01:19:30.409 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": 2021-07-31T01:19:30.411 INFO:teuthology.task.ansible.out:null, "state": "present", "user": "davidz", "validate_certs": true} ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2021-07-31T01:19:30.656 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2021-07-31T01:19:30.659 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2021-07-31T01:19:30.891 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQv 2021-07-31T01:19:30.900 INFO:teuthology.task.ansible.out:UxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2021-07-31T01:19:30.923 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2021-07-31T01:19:31.136 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2021-07-31T01:19:31.180 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2021-07-31T01:19:31.377 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2021-07-31T01:19:31.418 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9 2021-07-31T01:19:31.420 INFO:teuthology.task.ansible.out:CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2021-07-31T01:19:31.615 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2021-07-31T01:19:31.659 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBY 2021-07-31T01:19:31.661 INFO:teuthology.task.ansible.out:B0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2021-07-31T01:19:31.850 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2021-07-31T01:19:31.903 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2021-07-31T01:19:32.085 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2021-07-31T01:19:32.149 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2021-07-31T01:19:32.318 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-07-31T01:19:32.407 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7 2021-07-31T01:19:32.410 INFO:teuthology.task.ansible.out:gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrSX8nDL2WaHe0b/rjE3TZhj5VwbrG6nn/hyWmm2dWcUqotSp973yD2G4tQL9UygeSkDGmjb6whrz2mcAFJ5P4oLtIcCLyZC550dcpwV3BtP/cMAIGE6xeEARKiP8KlVGFNDRg9/l1aAd08Bepe+rJyAYD7IxOOBfgKJMKbeOWnT40RTfwiD5CXNwjzQuhzZDEbm+8zjTChFM4P1EkfR3c+0X3MYpA1AsRQY0ElvFtcn6G0yPKR+obGdIs+ydGupKh1+peGWLemHMA5e62Bs4/Lwtyebyp7AOguC3ZBJ3TZWL4Vn1q6z21cQe6AuowjPrneZWC4ifFB6OUQJqM+R00pHBcajcvsfpYOblZvTJHM726Ek1vahOYfcbEMpqWlVYo+WPRUilzl5cI3ReyAcwphebqKPbhmzCuyJw3lWBKIVDhWjEFzGkxV/iFU5rIXFm/Z0JGNFFxmY1U4GlA4lFXtL7OvEdMsiZo1x0cNmFWLu0k9fQBtiQI19ZBiA7FPKc= yehuda@crestview", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2021-07-31T01:19:32.563 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2021-07-31T01:19:32.648 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2021-07-31T01:19:32.795 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIKepJjIovBTCkdVyvAdptljhw9bYcwtIqV/ojdbpeuJtxDo9UyrEkNoLeVpZgOcrbgrkOE7AEixo3qB9PbD8z5NRUleV7wipSh3QwrDmSylSVReJKOwYdxUKkzesZYBGIGYzAbTKqcVzPUWCeW314+xmv90XkO/+1vgbC5uoXfbfnaiDHQimBom8QV5Egr+EhhcuCqMEqaVSvjMr1T0e6IL9gUens6Mys6czI37go6QvmgEzkBbDJJ64PsiOOFHVNk4DJVJ1c7TRtsnpIHM1YVFjsH1AnAL6UufbgNSFf5dziUB/Bz815P1/jsOGpHkUKE37DbHU5xtLGl3MmimMt4ndcG6EggBGEYBwfEWPRAsZs3qngssrNH5W0CzbysLaoovzq5VqS8SdLMu5XrRLBEB0uWWzLq2bobYymk3t/oby5zrHJV78xAuLmldqjvlPZSdkgwWNv9YBLkteBVr2HhjtuashJ6amNPbA4kCGhXMpeRhfvfqs31CBEKbTV+A8ODWbWAoQxLCcqS24x0yMi73N0UzuqDTaJMcE8ODJvOxcLNaV688QDbrC/UehpX/Bp/DM7fLHj4YhkRpZxkkN5w2kb/bBbWya4DrffhHKqn9CH1z5ynj0237PhTwCXjtRn0JzpeRgN8OraFgR2H3uPc9XNqlkFxxOBK0lRngbxzw== owasserm@redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2021-07-31T01:19:32.880 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2021-07-31T01:19:33.024 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1 2021-07-31T01:19:33.026 INFO:teuthology.task.ansible.out:OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2021-07-31T01:19:33.119 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2021-07-31T01:19:33.256 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyK 2021-07-31T01:19:33.258 INFO:teuthology.task.ansible.out:z2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2021-07-31T01:19:33.363 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2021-07-31T01:19:33.488 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2021-07-31T01:19:33.603 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2021-07-31T01:19:33.719 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2021-07-31T01:19:33.845 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQv 2021-07-31T01:19:33.847 INFO:teuthology.task.ansible.out:UxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2021-07-31T01:19:33.950 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2021-07-31T01:19:34.086 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2021-07-31T01:19:34.180 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtV 2021-07-31T01:19:34.182 INFO:teuthology.task.ansible.out:TFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDISEXvGI2oJLTRR15ePZLkmdYjaQJxt9V/yBwKdGqYvnPzmLOpdmVpdQJo7ZOWwdp5tJsCy4q5uX5vIUv6Gyi3uVSGmZnrKUbsqoDNwAX5vx/gFBINIs/ZXXyD4fpUT8aoZeq1XmXdIaWG0nImsrtRnRxKP9NGZEcW8z0WPwgF0Ck+RH0qZUix6Bfbgy7vMXnivuDv+NdjEwyszcrPWvf6Krr0am9Q0+9nj/mjvk6CZLl+CH3HNdkhZuNs+qMG4XW8KhLvQB2p790XVlTHuVU4jWrFmjymggZxkhoCY/ukHH6wi2Pbx6yjjKOj8iIw1xKabJzJ7S8UKT23A41d918DUCUuEG66pN6Z4dnv+fZJxj5iEo1GdycSWIc4SpLTK3z4zDZypVKIzjI4o8VwQaN48HkVjUdTICTvGQfytFi3jX1I5nxIZiiPcxMQLR5hQggP8pn2XtvXpomn0aBc8tVuuZJLVq4zCVnhlVB6IACmb71evjFGOsoSuEsaTufUr48= prsrivas@t480.lab.blr.redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-07-31T01:19:34.326 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2021-07-31T01:19:34.410 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjE 2021-07-31T01:19:34.412 INFO:teuthology.task.ansible.out:fdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+d 2021-07-31T01:19:34.414 INFO:teuthology.task.ansible.out:TzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2021-07-31T01:19:34.577 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2021-07-31T01:19:34.651 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2021-07-31T01:19:34.808 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2021-07-31T01:19:34.884 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2021-07-31T01:19:35.051 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2021-07-31T01:19:35.105 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2021-07-31T01:19:35.292 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-07-31T01:19:35.334 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2021-07-31T01:19:35.534 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2021-07-31T01:19:35.567 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2021-07-31T01:19:35.777 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIKepJjIovBTCkdVyvAdptljhw9bYcwtIqV/ojdbpeuJtxDo9UyrEkNoLeVpZgOcrbgrkOE7AEixo3qB9PbD8z5NRUleV7wipSh3QwrDmSylSVReJKOwYdxUKkzesZYBGIGYzAbTKqcVzPUWCeW314+xmv90XkO/+1vgbC5uoXfbfnaiDHQimBom8QV5Egr+EhhcuCqMEqaVSvjMr1T0e6IL9gUens6Mys6czI37go6QvmgEzkBbDJJ64PsiOOFHVNk4DJVJ1c7TRtsnpIHM1YVFjsH1AnAL6UufbgNSFf5dziUB/Bz815P1/jsOGpHkUKE37DbHU5xtLGl3MmimMt4ndcG6EggBGEYBwfEWPRAsZs3qngssrNH5W0CzbysLaoovzq5VqS8SdLMu5XrRLBEB0uWWzLq2bobYymk3t/oby5zrHJV78xAuLmldqjvlPZSdkgwWNv9YBLkteBVr2HhjtuashJ6amNPbA4kCGhXMpeRhfvfqs31CBEKbTV+A8ODWbWAoQxLCcqS24x0yMi73N0UzuqDTaJMcE8ODJvOxcLNaV688QDbrC/UehpX/Bp/DM7fLHj4YhkRpZxkkN5w2kb/bBbWya4DrffhHKqn9CH1z5ynj0237PhTwCXjtRn0JzpeRgN8OraFgR2H3uPc9XNqlkFxxOBK0lRngbxzw== owasserm@redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2021-07-31T01:19:35.811 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home 2021-07-31T01:19:35.813 INFO:teuthology.task.ansible.out:/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2021-07-31T01:19:36.021 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1 2021-07-31T01:19:36.023 INFO:teuthology.task.ansible.out:OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2021-07-31T01:19:36.032 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2021-07-31T01:19:36.252 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnOq6OHTRJuv7n0GZygBue/AE2q47pFM3KyrvmZxPRNmwtXn2r3ngHQQyc6DLZdFIv9Yn0VWu1KB/ryCdN3Xy7HAxBfHdKjtQ4vvfsWUbQF2JjhEjrsl+usgpqK1Tz6yuL4uAtFgLTN9o6gcH2bsQ15w+rEHCiYVzaGp7Cj+y+/VPBRIEHETSUZxwqiDH57uFTHxJrBIn6yhCorbn5QOjfmHUGmkxl3Geph++LbuQeha5F/DHaylMulsYZZBDEhIG8G2coDmVz8Cp1gw0kBWv63JHKnXaJboGXty2Zj36O/wdZKTE+7cV6JhxfJczhZBTb2cImIhspGwdHfnFgBoXz myoungwon@fg2server", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2021-07-31T01:19:36.276 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyK 2021-07-31T01:19:36.277 INFO:teuthology.task.ansible.out:z2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2021-07-31T01:19:36.487 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": " 2021-07-31T01:19:36.489 INFO:teuthology.task.ansible.out:gabrioux", "validate_certs": true} 2021-07-31T01:19:36.520 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2021-07-31T01:19:36.725 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2021-07-31T01:19:36.765 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2021-07-31T01:19:36.949 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2021-07-31T01:19:37.003 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2021-07-31T01:19:37.192 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kk 2021-07-31T01:19:37.194 INFO:teuthology.task.ansible.out:ke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2021-07-31T01:19:37.246 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtV 2021-07-31T01:19:37.248 INFO:teuthology.task.ansible.out:TFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDISEXvGI2oJLTRR15ePZLkmdYjaQJxt9V/yBwKdGqYvnPzmLOpdmVpdQJo7ZOWwdp5tJsCy4q5uX5vIUv6Gyi3uVSGmZnrKUbsqoDNwAX5vx/gFBINIs/ZXXyD4fpUT8aoZeq1XmXdIaWG0nImsrtRnRxKP9NGZEcW8z0WPwgF0Ck+RH0qZUix6Bfbgy7vMXnivuDv+NdjEwyszcrPWvf6Krr0am9Q0+9nj/mjvk6CZLl+CH3HNdkhZuNs+qMG4XW8KhLvQB2p790XVlTHuVU4jWrFmjymggZxkhoCY/ukHH6wi2Pbx6yjjKOj8iIw1xKabJzJ7S8UKT23A41d918DUCUuEG66pN6Z4dnv+fZJxj5iEo1GdycSWIc4SpLTK3z4zDZypVKIzjI4o8VwQaN48HkVjUdTICTvGQfytFi3jX1I5nxIZiiPcxMQLR5hQggP8pn2XtvXpomn0aBc8tVuuZJLVq4zCVnhlVB6IACmb71evjFGOsoSuEsaTufUr48= prsrivas@t480.lab.blr.redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-07-31T01:19:37.414 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2021-07-31T01:19:37.498 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjE 2021-07-31T01:19:37.500 INFO:teuthology.task.ansible.out:fdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+d 2021-07-31T01:19:37.502 INFO:teuthology.task.ansible.out:TzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2021-07-31T01:19:37.648 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2021-07-31T01:19:37.735 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2021-07-31T01:19:37.892 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2021-07-31T01:19:37.978 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2021-07-31T01:19:38.127 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-07-31T01:19:38.214 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2021-07-31T01:19:38.360 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2021-07-31T01:19:38.455 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2021-07-31T01:19:38.591 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2021-07-31T01:19:38.698 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2021-07-31T01:19:38.825 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2021-07-31T01:19:38.942 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home 2021-07-31T01:19:38.943 INFO:teuthology.task.ansible.out:/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2021-07-31T01:19:39.057 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2021-07-31T01:19:39.185 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2021-07-31T01:19:39.294 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2021-07-31T01:19:39.435 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnOq6OHTRJuv7n0GZygBue/AE2q47pFM3KyrvmZxPRNmwtXn2r3ngHQQyc6DLZdFIv9Yn0VWu1KB/ryCdN3Xy7HAxBfHdKjtQ4vvfsWUbQF2JjhEjrsl+usgpqK1Tz6yuL4uAtFgLTN9o6gcH2bsQ15w+rEHCiYVzaGp7Cj+y+/VPBRIEHETSUZxwqiDH57uFTHxJrBIn6yhCorbn5QOjfmHUGmkxl3Geph++LbuQeha5F/DHaylMulsYZZBDEhIG8G2coDmVz8Cp1gw0kBWv63JHKnXaJboGXty2Zj36O/wdZKTE+7cV6JhxfJczhZBTb2cImIhspGwdHfnFgBoXz myoungwon@fg2server", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2021-07-31T01:19:39.527 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2021-07-31T01:19:39.684 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": " 2021-07-31T01:19:39.686 INFO:teuthology.task.ansible.out:gabrioux", "validate_certs": true} 2021-07-31T01:19:39.749 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2021-07-31T01:19:39.919 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2021-07-31T01:19:39.983 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2021-07-31T01:19:40.164 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2021-07-31T01:19:40.208 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2021-07-31T01:19:40.408 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kk 2021-07-31T01:19:40.410 INFO:teuthology.task.ansible.out:ke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2021-07-31T01:19:40.441 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2021-07-31T01:19:40.648 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2021-07-31T01:19:40.681 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDklgWHrL9TMaMxIyN9MxVR6nR3xh9zfj0iR63/yOzaslX0WoVBk7CYTsRlH2GFVctvixVwrWbQgMaKe49utal/3FA6sMl8S2DTrtgC1a/exUTvU0U/+ZhyUlkyjQfRgtYOUwodNupL5NsC8VZqZshgmgXC34uye3bplx3VwvpUt9r6QxJN3/9Dbi6HxudhDCLfmvROhwQYHaFF+GglcbyjAt3uvsuwXVKBd1/vMQwbcUo9/zQlrnmWBRuqTNkXGrZauWigoi/hyuWYQ3MUbRc3LVpJncMv7BEtcj8fCVvPB09SFee3nSeTtFkm01GvRcYd3HBz/YL4Rz92wXl9VIxqs8aaJBzy+MD0NO496Rfqtz6YuZ1m7AxDIccDW+OpcPgYwaUKFXNTuVLWZXhkS0p4swefcKWRlQFF3BLOebN/Kr9jq1MFa+5pX7Gj4WF5uCIX0/fx40KVZNhfl3qywBy/d5MKwXO7PCHMiGcu2FIXdZ334MxdgiIVoX54PpRP4hs= smanjara@localhost.localdomain", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2021-07-31T01:19:40.914 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2021-07-31T01:19:41.056 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2021-07-31T01:19:41.142 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2021-07-31T01:19:41.300 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2021-07-31T01:19:41.374 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDQu/CuhCIFVGCLbA0cjn+qqcwX12cRcFA1sHyJtkj6Ltfmge8FnfgAsex/LaPJwbtbBiVl9xRcazHNfBaDw5DxMRSJF+e7WTydZC2Jm4qIj4D3Lr+imGPrJTK+kjN0QXRXknGsNn36oRF/I41cGqYRGwawFGSPZBHuav+ehDk1doTrbDlll4zMUWm2iXTcBIcJo4NR1LoE2c9/MPfbH7jKm5ke7aS0ets+EANEplkeClfCVAashtIF+Bz6a9qk4F2n0baLT/s3IQkXCwwO2fxxY03x8gWiB1n4E7sousEB+J1Skl/z0bwmaiY0C+raPoV/FyZxQuh0ocvdU7fnQbKuNoGE/Usfm9Y3g8eH81b/k4fMJL2lqXHkPft6zUV9bJsp3v1iGEkIf2+ER/QLczteNz6kw/uTPQctMEDXM17GEnBXISVrQcPSW3i93eFwWacs8iT3G4bjEQvpxSBQrhDdCCoxy8bVWqkVeDWw5u2q/3WL9Ov1NrY+G4/DYb2qONNQds0vTHCGvemU5GNjtaeCbojkZ7Z3KOeAY8080kKrsO7fve+ZMACCIFD+se8kF7NN4ZJASSqq9lJpBWvtbc0O8jCcVGNuN7exXjrcbiN1L/9HkARV+KEJU3jbXObAhAnfAaeRWBwmTfZodCu+fDd/Jor6l36tjD4kC4/CPrDtyQ== alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2021-07-31T01:19:41.535 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-07-31T01:19:41.613 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2021-07-31T01:19:41.792 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2021-07-31T01:19:41.834 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2021-07-31T01:19:42.035 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2021-07-31T01:19:42.067 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDp0xz0ml2XuoQe70TxVeTo3Ib5xMinDrYNJ5eq5XYNa8ZLGcEByeDK4myzixTloAo31Rb/yDyGqv6kQ6nFChihO2aab3Ni/bsMOaJat1scrznaIYXGvLvw5KYx/qBDUxb3qW0vO0N/bB3dUPS1 2021-07-31T01:19:42.069 INFO:teuthology.task.ansible.out:lXTJaVf+KuljUk7v38ov2gsPWlofRYZNkPGQfUKXFEXc39OVj+Ex5LkehH/Z1LwjRYJa7jAdSm8hE5m8m6WborrzAA4/iFuLbhtM/Bqm4boSqa7koBVux422QX/HblSQ2KAeT1GFOS7xdcO6ndyp6mIpCZWunTjZwSLyK20NvIV/AzdAro4GZWi0D34PPtbSG344ZT1dSLeDRhYVkDjS6uutrmw9C+Ns1bo8s1ydhxL2XEsU14SWRWlbyrMF9PXyJOsaNiBNWGwc9lTly0ZlmPnN3l0AC3RIKOstkEYmI+TMj21r1tltz48e+6dH1an51wwnhNRYiODd3hbS67tmIg05BuSU5+whJ93wzXw2rC4FLE= ylifshit@localhost.localdomain", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2021-07-31T01:19:42.276 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2021-07-31T01:19:42.298 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2021-07-31T01:19:42.518 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2021-07-31T01:19:42.530 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2021-07-31T01:19:42.761 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2021-07-31T01:19:42.763 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2021-07-31T01:19:43.010 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2021-07-31T01:19:43.012 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2021-07-31T01:19:43.243 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.locald 2021-07-31T01:19:43.246 INFO:teuthology.task.ansible.out:omain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2021-07-31T01:19:43.256 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2021-07-31T01:19:43.477 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq 2021-07-31T01:19:43.478 INFO:teuthology.task.ansible.out:2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2021-07-31T01:19:43.488 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2021-07-31T01:19:43.709 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdqbYcbJYtgCrAbLo7PcS82caXxcT8cpX0RkULaFXfaiUDs8Foy6Yf/XoEHIdpgw8kX6qBdydYXxti6HFhYDNsL5KleKu/yA+bVKlvri39CRNkNxFHAUL2CRBT5Hefj1AJQBm/BKVEN4/eb05OlJJ808Qb/aNycot/YwgpNvUIaO2yTMA0meq82wNdZmUcmyLB9QXz4QInTcoAsxq70KxX95YC938rTWpbg1hM8gzLkJJt/OD/Z7TMYv/Sc5KGOFqo930AmObQ5KPmffkWGYbLT65TxlCULnvIr63lWHdGTj5mYjib/v9ncu24az9GYRWYcicV0B/def+KVrrIp3Mz ideepika@teuthology", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2021-07-31T01:19:43.732 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2021-07-31T01:19:43.930 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2021-07-31T01:19:43.973 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2021-07-31T01:19:44.163 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrGYBGPDQqn+2jnxKhDurSrAJvTT8dZMYGVCt6NbjpFhDFYtyvrF8Jv5NKBQynG18LwscN7a57L0Oq0e4jOEPRz//CDcAfP+L6p45L4BdUzTysN4+rQ7waMnIiYp7VjSswhGH/BnYIVKwmDqzM2fo1w/oLIT6mgwUYqAdceeswAkXBt7jgagL8ox3E4oHBqER7f60AHdTviuW953t5luVQiMjyF2fOwZuo2ycufIzBee0CwS+xyRuJQwdgD3HL4Qs2vtBCX+H8n3wC6A9JrMA5tzCDr5VZKBJZURIniwvaXfJodjO4SvBL5jAfXoMqriv8ZWHbap2fk82YE8/uPqNvSISjixJEBgQinklZ+iAOYm5tfPbhPM/8ErpN/yJ0EoGRn+nj6X6nAjXAIk3AENopeqtM7mLZ26RFIuEV5ld7eqUvAfiy6Hk2LH9Q1Z4CXaxD0gDGzUxgdAaaAgLAR4btWi2n7Tb7ZNDdOfctg2ya9ftzZBOHvCaVXSpJCry/GMU= skoduri@localhost.localdomain", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2021-07-31T01:19:44.205 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDklgWHrL9TMaMxIyN9MxVR6nR3xh9zfj0iR63/yOzaslX0WoVBk7CYTsRlH2GFVctvixVwrWbQgMaKe49utal/3FA6sMl8S2DTrtgC1a/exUTvU0U/+ZhyUlkyjQfRgtYOUwodNupL5NsC8VZqZshgmgXC34uye3bplx3VwvpUt9r6QxJN3/9Dbi6HxudhDCLfmvROhwQYHaFF+GglcbyjAt3uvsuwXVKBd1/vMQwbcUo9/zQlrnmWBRuqTNkXGrZauWigoi/hyuWYQ3MUbRc3LVpJncMv7BEtcj8fCVvPB09SFee3nSeTtFkm01GvRcYd3HBz/YL4Rz92wXl9VIxqs8aaJBzy+MD0NO496Rfqtz6YuZ1m7AxDIccDW+OpcPgYwaUKFXNTuVLWZXhkS0p4swefcKWRlQFF3BLOebN/Kr9jq1MFa+5pX7Gj4WF5uCIX0/fx40KVZNhfl3qywBy/d5MKwXO7PCHMiGcu2FIXdZ334MxdgiIVoX54PpRP4hs= smanjara@localhost.localdomain", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2021-07-31T01:19:44.394 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2021-07-31T01:19:44.447 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2021-07-31T01:19:44.625 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2021-07-31T01:19:44.688 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2021-07-31T01:19:44.847 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2021-07-31T01:19:44.944 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDQu/CuhCIFVGCLbA0cjn+qqcwX12cRcFA1sHyJtkj6Ltfmge8FnfgAsex/LaPJwbtbBiVl9xRcazHNfBaDw5DxMRSJF+e7WTydZC2Jm4qIj4D3Lr+imGPrJTK+kjN0QXRXknGsNn36oRF/I41cGqYRGwawFGSPZBHuav+ehDk1doTrbDlll4zMUWm2iXTcBIcJo4NR1LoE2c9/MPfbH7jKm5ke7aS0ets+EANEplkeClfCVAashtIF+Bz6a9qk4F2n0baLT/s3IQkXCwwO2fxxY03x8gWiB1n4E7sousEB+J1Skl/z0bwmaiY0C+raPoV/FyZxQuh0ocvdU7fnQbKuNoGE/Usfm9Y3g8eH81b/k4fMJL2lqXHkPft6zUV9bJsp3v1iGEkIf2+ER/QLczteNz6kw/uTPQctMEDXM17GEnBXISVrQcPSW3i93eFwWacs8iT3G4bjEQvpxSBQrhDdCCoxy8bVWqkVeDWw5u2q/3WL9Ov1NrY+G4/DYb2qONNQds0vTHCGvemU5GNjtaeCbojkZ7Z3KOeAY8080kKrsO7fve+ZMACCIFD+se8kF7NN4ZJASSqq9lJpBWvtbc0O8jCcVGNuN7exXjrcbiN1L/9HkARV+KEJU3jbXObAhAnfAaeRWBwmTfZodCu+fDd/Jor6l36tjD4kC4/CPrDtyQ== alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2021-07-31T01:19:45.097 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2021-07-31T01:19:45.193 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2021-07-31T01:19:45.329 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDS3HkgHb6z+gi4tN5rBL/N+WLJgN8RtUokt0YnRjpor2epRbmqgmEFRbRFqjLK9LiTuLxmkwol1/G56AJms9ixue33M6jVs73nc259LDxn2qj/NCtxeU7xdjSDuhKyItgM0yuiFjWOKSYTghNtwoR26PX7SQVbycD4NN1ZuwUbdMQA31VrLvQR7d0lYKlis2auH2Xb6WifotoEyszZz1KYRwzJvOHtHku/jAyNmTD/8kAsuCjEgUPSyyehAfdWn59tY2gM7aFyj/dtRoAghekv5gkov9zx+zjN6D2LSN7e+DRti05iJMifA3nfmdj7f8aHzXDtsDL0XNkmmT3g9N2OQ8WL4ESOFASXPPj9JGUROealgnaQ+Y64evDClhbt2L7FLbgj5sgKb+XYhpGnAiC+v3t1Rq3HZrstOYf2qSYguIo+zW9gHwzJDqH5YNCpSF4ZDCfkJ94ngl5ptQmZ/FcrH2hBs2cMoyK9EyDpcI9hBeACTzu19kaB6DrOC+UTvBgYL2sNbCvi+caIqhENJICZ21rI4u4EZqu7PVb+Y3CLgSjTEgclGYBZoxlrT/X2QHrIfZF7lLGeeh0bureedCrBg4b61SL7ZU3Me9x1iWOZMSJgVvg+qPSrzZiHbA46k8i0af5Q+geL1BuuGy76zbYd/KR0KiNZnXjdlHyRz76rpQ== kamoltatsirivadhna@Ksirivad-MacBook-Pro.loc 2021-07-31T01:19:45.331 INFO:teuthology.task.ansible.out:al", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2021-07-31T01:19:45.435 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2021-07-31T01:19:45.551 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2021-07-31T01:19:45.692 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDp0xz0ml2XuoQe70TxVeTo3Ib5xMinDrYNJ5eq5XYNa8ZLGcEByeDK4myzixTloAo31Rb/yDyGqv6kQ6nFChihO2aab3Ni/bsMOaJat1scrznaIYXGvLvw5KYx/qBDUxb3qW0vO0N/bB3dUPS1 2021-07-31T01:19:45.694 INFO:teuthology.task.ansible.out:lXTJaVf+KuljUk7v38ov2gsPWlofRYZNkPGQfUKXFEXc39OVj+Ex5LkehH/Z1LwjRYJa7jAdSm8hE5m8m6WborrzAA4/iFuLbhtM/Bqm4boSqa7koBVux422QX/HblSQ2KAeT1GFOS7xdcO6ndyp6mIpCZWunTjZwSLyK20NvIV/AzdAro4GZWi0D34PPtbSG344ZT1dSLeDRhYVkDjS6uutrmw9C+Ns1bo8s1ydhxL2XEsU14SWRWlbyrMF9PXyJOsaNiBNWGwc9lTly0ZlmPnN3l0AC3RIKOstkEYmI+TMj21r1tltz48e+6dH1an51wwnhNRYiODd3hbS67tmIg05BuSU5+whJ93wzXw2rC4FLE= ylifshit@localhost.localdomain", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2021-07-31T01:19:45.790 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2021-07-31T01:19:45.936 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2021-07-31T01:19:46.021 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2021-07-31T01:19:46.169 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2021-07-31T01:19:46.255 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2021-07-31T01:19:46.417 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2021-07-31T01:19:46.481 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2021-07-31T01:19:46.650 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2021-07-31T01:19:46.713 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2021-07-31T01:19:46.904 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.locald 2021-07-31T01:19:46.905 INFO:teuthology.task.ansible.out:omain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2021-07-31T01:19:46.946 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDISEXvGI2oJLTRR15ePZLkmdYjaQJxt9V/ 2021-07-31T01:19:46.948 INFO:teuthology.task.ansible.out:yBwKdGqYvnPzmLOpdmVpdQJo7ZOWwdp5tJsCy4q5uX5vIUv6Gyi3uVSGmZnrKUbsqoDNwAX5vx/gFBINIs/ZXXyD4fpUT8aoZeq1XmXdIaWG0nImsrtRnRxKP9NGZEcW8z0WPwgF0Ck+RH0qZUix6Bfbgy7vMXnivuDv+NdjEwyszcrPWvf6Krr0am9Q0+9nj/mjvk6CZLl+CH3HNdkhZuNs+qMG4XW8KhLvQB2p790XVlTHuVU4jWrFmjymggZxkhoCY/ukHH6wi2Pbx6yjjKOj8iIw1xKabJzJ7S8UKT23A41d918DUCUuEG66pN6Z4dnv+fZJxj5iEo1GdycSWIc4SpLTK3z4zDZypVKIzjI4o8VwQaN48HkVjUdTICTvGQfytFi3jX1I5nxIZiiPcxMQLR5hQggP8pn2XtvXpomn0aBc8tVuuZJLVq4zCVnhlVB6IACmb71evjFGOsoSuEsaTufUr48= prsrivas@t480.lab.blr.redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-07-31T01:19:47.157 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq 2021-07-31T01:19:47.158 INFO:teuthology.task.ansible.out:2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2021-07-31T01:19:47.168 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOJg56HsyNa5vHHb52G6O5vtv9I2tdlNMjFlG3SVCCu/bVZwVeGzp1ekuNGU10owmnIvhlSldMjLn2wwTwiDjbvYJVVCPvzK63vCUcMevRsvr9EyU16JdAgrmrGMyuYYEb+wh8XWI9Dc72o84fMaBYFSSer1S9nUBL+KVLVhIe7culgADimrDgDVh2t7u2JHBI0m0nKFGJZNcIyYf+GKOM+n7SaQXIYIytvSteU5BvUIKPsDd4z5lVlWiKO3kG6jF2RnKFV6KWBWI9m8YGItdGu25mXELXN9MOlApZF3AoGTmCE2Web7uDVGvO6rz6b+dc7RXTOlbWIhtMoRqQf6mz zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2021-07-31T01:19:47.398 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdqbYcbJYtgCrAbLo7PcS82caXxcT8cpX0RkULaFXfaiUDs8Foy6Yf/XoEHIdpgw8kX6qBdydYXxti6HFhYDNsL5KleKu/yA+bVKlvri39CRNkNxFHAUL2CRBT5Hefj1AJQBm/BKVEN4/eb05OlJJ808Qb/aNycot/YwgpNvUIaO2yTMA0meq82wNdZmUcmyLB9QXz4QInTcoAsxq70KxX95YC938rTWpbg1hM8gzLkJJt/OD/Z7TMYv/Sc5KGOFqo930AmObQ5KPmffkWGYbLT65TxlCULnvIr63lWHdGTj5mYjib/v9ncu24az9GYRWYcicV0B/def+KVrrIp3Mz ideepika@teuthology", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2021-07-31T01:19:47.411 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2021-07-31T01:19:47.633 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2021-07-31T01:19:47.644 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2021-07-31T01:19:47.864 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2021-07-31T01:19:47.896 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrGYBGPDQqn+2jnxKhDurSrAJvTT8dZMYGVCt6NbjpFhDFYtyvrF8Jv5NKBQynG18LwscN7a57L0Oq0e4jOEPRz//CDcAfP+L6p45L4BdUzTysN4+rQ7waMnIiYp7VjSswhGH/BnYIVKwmDqzM2fo1w/oLIT6mgwUYqAdceeswAkXBt7jgagL8ox3E4oHBqER7f60AHdTviuW953t5luVQiMjyF2fOwZuo2ycufIzBee0CwS+xyRuJQwdgD3HL4Qs2vtBCX+H8n3wC6A9JrMA5tzCDr5VZKBJZURIniwvaXfJodjO4SvBL5jAfXoMqriv8ZWHbap2fk82YE8/uPqNvSISjixJEBgQinklZ+iAOYm5tfPbhPM/8ErpN/yJ0EoGRn+nj6X6nAjXAIk3AENopeqtM7mLZ26RFIuEV5ld7eqUvAfiy6Hk2LH9Q1Z4CXaxD0gDGzUxgdAaaAgLAR4btWi2n7Tb7ZNDdOfctg2ya9ftzZBOHvCaVXSpJCry/GMU= skoduri@localhost.localdomain", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2021-07-31T01:19:48.098 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2021-07-31T01:19:48.141 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2021-07-31T01:19:48.334 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2021-07-31T01:19:48.398 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2021-07-31T01:19:48.566 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2021-07-31T01:19:48.650 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2021-07-31T01:19:48.796 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2021-07-31T01:19:48.890 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2021-07-31T01:19:49.028 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2021-07-31T01:19:49.137 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDS3HkgHb6z+gi4tN5rBL/N+WLJgN8RtUokt0YnRjpor2epRbmqgmEFRbRFqjLK9LiTuLxmkwol1/G56AJms9ixue33M6jVs73nc259LDxn2qj/NCtxeU7xdjSDuhKyItgM0yuiFjWOKSYTghNtwoR26PX7SQVbycD4NN1ZuwUbdMQA31VrLvQR7d0lYKlis2auH2Xb6WifotoEyszZz1KYRwzJvOHtHku/jAyNmTD/8kAsuCjEgUPSyyehAfdWn59tY2gM7aFyj/dtRoAghekv5gkov9zx+zjN6D2LSN7e+DRti05iJMifA3nfmdj7f8aHzXDtsDL0XNkmmT3g9N2OQ8WL4ESOFASXPPj9JGUROealgnaQ+Y64evDClhbt2L7FLbgj5sgKb+XYhpGnAiC+v3t1Rq3HZrstOYf2qSYguIo+zW9gHwzJDqH5YNCpSF4ZDCfkJ94ngl5ptQmZ/FcrH2hBs2cMoyK9EyDpcI9hBeACTzu19kaB6DrOC+UTvBgYL2sNbCvi+caIqhENJICZ21rI4u4EZqu7PVb+Y3CLgSjTEgclGYBZoxlrT/X2QHrIfZF7lLGeeh0bureedCrBg4b61SL7ZU3Me9x1iWOZMSJgVvg+qPSrzZiHbA46k8i0af5Q+geL1BuuGy76zbYd/KR0KiNZnXjdlHyRz76rpQ== kamoltatsirivadhna@Ksirivad-MacBook-Pro.loc 2021-07-31T01:19:49.139 INFO:teuthology.task.ansible.out:al", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2021-07-31T01:19:49.262 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2021-07-31T01:19:49.369 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2021-07-31T01:19:49.508 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} 2021-07-31T01:19:49.614 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2021-07-31T01:19:49.741 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} 2021-07-31T01:19:49.858 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2021-07-31T01:19:49.976 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} 2021-07-31T01:19:50.102 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2021-07-31T01:19:50.208 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} 2021-07-31T01:19:50.338 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2021-07-31T01:19:50.433 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} 2021-07-31T01:19:50.582 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2021-07-31T01:19:50.666 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} 2021-07-31T01:19:50.824 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDISEXvGI2oJLTRR15ePZLkmdYjaQJxt9V/ 2021-07-31T01:19:50.825 INFO:teuthology.task.ansible.out:yBwKdGqYvnPzmLOpdmVpdQJo7ZOWwdp5tJsCy4q5uX5vIUv6Gyi3uVSGmZnrKUbsqoDNwAX5vx/gFBINIs/ZXXyD4fpUT8aoZeq1XmXdIaWG0nImsrtRnRxKP9NGZEcW8z0WPwgF0Ck+RH0qZUix6Bfbgy7vMXnivuDv+NdjEwyszcrPWvf6Krr0am9Q0+9nj/mjvk6CZLl+CH3HNdkhZuNs+qMG4XW8KhLvQB2p790XVlTHuVU4jWrFmjymggZxkhoCY/ukHH6wi2Pbx6yjjKOj8iIw1xKabJzJ7S8UKT23A41d918DUCUuEG66pN6Z4dnv+fZJxj5iEo1GdycSWIc4SpLTK3z4zDZypVKIzjI4o8VwQaN48HkVjUdTICTvGQfytFi3jX1I5nxIZiiPcxMQLR5hQggP8pn2XtvXpomn0aBc8tVuuZJLVq4zCVnhlVB6IACmb71evjFGOsoSuEsaTufUr48= prsrivas@t480.lab.blr.redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-07-31T01:19:50.908 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnuL8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} 2021-07-31T01:19:51.076 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOJg56HsyNa5vHHb52G6O5vtv9I2tdlNMjFlG3SVCCu/bVZwVeGzp1ekuNGU10owmnIvhlSldMjLn2wwTwiDjbvYJVVCPvzK63vCUcMevRsvr9EyU16JdAgrmrGMyuYYEb+wh8XWI9Dc72o84fMaBYFSSer1S9nUBL+KVLVhIe7culgADimrDgDVh2t7u2JHBI0m0nKFGJZNcIyYf+GKOM+n7SaQXIYIytvSteU5BvUIKPsDd4z5lVlWiKO3kG6jF2RnKFV6KWBWI9m8YGItdGu25mXELXN9MOlApZF3AoGTmCE2Web7uDVGvO6rz6b+dc7RXTOlbWIhtMoRqQf6mz zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2021-07-31T01:19:51.140 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devBox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} 2021-07-31T01:19:51.318 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2021-07-31T01:19:51.361 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4pIcNiZKVIfT9GxyTe/B8G2GyHKxrAle3iExEzLH5bBCKtBrottX1y66KBzQSsrAb0z9h+A+uxlswlY6VM/gNos9VRB6x4NRB4ADY1Bku+JsKs5aXwRB1zz2ueZhrFN9hJwMl1heLICx85ksHBdm0q+JPdUAV+/LBdr0H/IjWveV34WL3Awf7QikzyKhzUciMB2gN9PJXTW0bl8PP56y60ys++L3nMSRJ+15uPcWl8MS3D2vABII0Tu+3PsxzrTTNodElyQ6G/fwyFfWhdp7fkidjR7mDDx8NyWfCl2rKejXv9LnVC0wb3tFoQqvwdz6shN4E51EIEGXHcLUeRpBhL2pFOqIZ0qfnmLIQVZ88RKMTgo42CgtmS5UzxCLOyKOh9GxF7DDEajTinJFJJVbNJ5xF494H2geSnmP1o1AecGsGbCZ4ndhS9unSfIZLAyv1J3wZlAlMsvxXgY7yrh1srUiASVyKK8jba1IT+v3TXR8kQUCNsLKKbvuX/l0IXME= xuxuehan@bogon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBd4eSizc2zAkWDy3mFZUhELO2wXDgkVxDNM/Nw2zT1li3L+6YOjFfdT4NOxpkg5laLd54bdKb5GVRA6iKqAO+aqo53xHHJGC8OY5eJaUPjtpypgIWO4Qzbos2ZzsC+nX5jvi8O9nM+JkabSW5iFv3gGPTqHixFYkxK2NIBa/UtXFOhmb1wZ6CG5Q+WObCosBUEMWokC8ntCADfhi6qDsO2fol6mYUhYEE/0nhQ5uLRrVIM5QM5dhrQVEeAs4KcYpmPDw6q6AKgx/f5zObh5OSJFTVsgx2cqzdQVJVDDx2OD2QVRtZ9fgvI2EGmKSwe1TUQOdqa5bmsVIp2Jfi8MC1 xuxuehan@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwIokXSvPYP3TRlSw0SBgmMFW5jVltGf4c6CqjIW4KKCgsL4c4Vs0FztRyDGc9p0krZUoGJCZHErVToXm6/lSq8JlfrlLX4X4yh37fUifWqc2UTJGeH8kLaP3WEt9vT8eQ9h8t2GqlMlGQU8KzVAdC+c5IwrbGRsS6S7D09/y0rklkhdJ6nuj8cssN/2TzLF9YIvVcFZHdyznk84eB7TgpPKD88MOpTnEgWFlD4vAWEuHpkuxgj1KI9LscgSFcW9/RxLeRxcV0GypZ8jPmczoCnin/ZOstCTpbcmjL 2021-07-31T01:19:51.363 INFO:teuthology.task.ansible.out:d+LW1kgGXXQSkf09iHeJ/cyCw6ooZOkynVwwmI9Uz2ZaMedCUBpG0bKclIerJ9jn+CoTs3U3Orhd9+6MVtCHX9VNJbVa4q0KCS1sp5S3HDrYkgUyawUxdYorrLZcqfRHWWo1luiridmuQdr8lQ6dvNpz4rhsCrAG6S1eeaC/s+o0Rzpv1YxU64JzquZ6WNR+W3U6Bg3iYkSoDCK9byf0rsptyL8= luqixi@luqixideMacBook-Air.local", "key_options": null, "keyfile": "/home/xuxuehan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuxuehan", "validate_certs": true} 2021-07-31T01:19:51.551 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2021-07-31T01:19:51.594 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMOuxxTZyxz/TEVA+ODgtLrNcSsUhKE64BZg/Ub96LLffQXQc9SYT/0YI4wxu2vSsky05PnVSON7jqGxunCt0+AdRqxXhW4ttmRIOz8rq8SiMnnLayV34nIF1G0aYTkgY4ej4+VmSVIHU3bb5mp/Q3D0hxQwDPdgOJWNyW9oYerHbnx6lVgchvsZHB9VO09GNHcAsyUz4p+xBf9iR7eS74p4dZmUbth/oBBqpOa4NgNr69iyZ9a/pNDLxaVQoIpDFrevk5UU5nUHQq2frFNxF/Y2Ja1EBXCavvPF/m5v3J3xNEu3SI/GbLFCPKCKeBDpXJbe1QqTBKJjMl9qGmKEJ3 aclewett@mac", "key_options": null, "keyfile": "/home/aclewett/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aclewett", "validate_certs": true} 2021-07-31T01:19:51.792 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2021-07-31T01:19:51.824 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdTLDprr3DXtsgiqrB/miglJwSkY7BAv+OW9sC0ZHQf8Y4dFoUtr7q9fd6W7ehKL6aMjya/8eQSMPRoEIZFD8f2KPCuE74Wgwgv2cpAkFmLjcPtjixqLo3LGFCyEJWGLQxjlthdzx59mpCWgBsEJQmlKP36JIYjILIMtXtsETIUGFa8QPIn9sMdIgutPPf3yB/6DHcR7C6yIAuvbJactEsiCt4C8+uJinvCii3PzYOMAEd+mictPGk8nOCQFZrI+ZmwTz4UbJylxFqnX8lhhoPIUEfNAXlb3lkZWH1LaFOxarbmupuPYcg2rT0+kFtyNr3/mAx0FO01jADYooqcW1Z cardno:000604936806", "key_options": null, "keyfile": "/home/cblum/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cblum", "validate_certs": true} 2021-07-31T01:19:52.033 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2021-07-31T01:19:52.055 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDPUCIVfNQOOG+uAeKWo7tLAdgGO/oQu4exTE3XLfyTVdd8u6pbQCmIDFEnbGgMtlYuWJlCJtxJIzEZJ5HIwf+po3y7kb7HxEeWnEb9DsqJB4nCRE8fHnNLgF3bh9gvE7sOZdNXEKqtsnGcaaS81oXQ6QgkFGQzlOBOqxj7jIvad7AtNWD3WbJi/uTVDsWcukU51tn/aEW+hHMD7Y5QJkLH7O0PrCeEZCUCWoiMP7sNwACBym0DsqX+POkGJOaGx9EeNhGVA4B6o3xYXJPqmFbgMWMfG3/Y3SiheZjxzySMQ7FKW4Q817NaWqbjtLZQsSZw7JNqNFkcdTzVZwwCOJ1AAecajpniKeQLnyiWQ3Z/EkSGGwtDj67wo7uqT85tpOyLfYhQm6Zp+xSh+I6s0YJ0xO/huLKzeT5D1r5vVMk2VdcLD3CCxXUwZgaMUxYAC2Wq6mpIkxVZFQLbGuA/dWkQbFsrpbf7ES+kh+uzp4R6dQ29OjXbVcSfF2vyIuA+VUKoyj370YOZQhs50yuAJ4BSHlEgoJdRzi3XUX48h45XHiZcCY3X8LTX6a+0KAIgqeAWdFhRgQtAxoRYjhOxC3k/boBc8sEQ3o4pkxnhtE5Y/tnM2CPkPMX9HWYirNf9GyZTYcR/5rDf4ETQ18G68WLaNK758OuUPjSkwttcyLd4dw== mlatimer@suse.com", "key_options": null, "keyfile": "/home/mlatimer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mlatimer", "validate_certs": true} 2021-07-31T01:19:52.286 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2021-07-31T01:19:52.288 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYh28a9R3a7jpRfPqVAL9EcD+S6KkZzKTfCmRn4s8Fg39kUZsgnWUZQ3jUxKSeBA00te7WMR8CsIEzulm7f+/egNuPcvzNI/SaaCDtVMaxqlEuW85ll8uBQi3s4SLmOu9oS+5oid4cqTrGTcG6uhl29oWS/53SxoGh+T+y6GesO2cYRuTpcUHxGMvIgZe/bKtVGdP1y7rQd62e8emb9UupMBxaHpkYj9AaMuLjovHbBgecWbe4PlzPWNzl+eITVk2d6CxxIem4/bw9VHkRjQZlhL2nbwRgtczLYhlaZAd4ePEtsqA+7nTDjTDhXXHVIf1pDAsJbZEdgQCSGH3FzxtrQ2hZJrH/NIKdHk/OtI06e7POSGj7PcHzTvMjQSFnTpJXinxwzn0lvFSlmABN/00lIE0p2ZXfPs6dgWJ17JnTq6dMUoTiGhfKphPHyfSXJCzfa2VB+lQQ5eAaa9cRXy9pLmWGHXVGz2hsBMtyYmTjR/wr8kdTzeK+Jk4opBtGgcxDustg59b9x/jvgoc3HCdtnkqLTug2KmutB6kb+4KnVO07GsSp0RmNtDmCRcXs9YN/eia9ICtIGDWPc3YDRppsw1Vktza/+ZW5vGzIKILZTo4SNGcg19zQw4Qxd9eLkXfB66vE2CQvA90CGo7mcye+UWjjXLr7i0cDEszAloQ9aQ== srangana@redhat.com", "key_options": null, "keyfile": "/home/shyamsundarr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyamsundarr", "validate_certs": true} 2021-07-31T01:19:52.516 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRZyOnj6D9tCw5o4+ujpos5Uj1O0XVvea798DsXPRh8bEE1pLkARQJLv4iW6LsG3x/CXruahmwT+6C/VHppIsbgtpadMWdCTPYrsjctPK0/N9nr4kQEL6XeK3/GHFnwqYcc+LySpNNfbKrgN4qWOwdZBhQjSV7K7FJd4ZYSm4wp6TVbvKKp7o+JdeM2zhstfC9AcCtBv/TpxixFFq/+y8/Z0PwqN1YyHduDOr1dr1DWn4ZzrkhiTYfPOltx0jweD6zTJoTAJDqae7kBA+3XH3dUGX3n7Tchsa3bnIKVz9sLQvH/vPro61I+Xig6MQh77JsylNvpwb1LLOEzDf7dxl bigwhite@wrjj37deMacBook-Pro-2.local", "key_options": null, "keyfile": "/home/shuzhenyi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shuzhenyi", "validate_certs": true} 2021-07-31T01:19:52.527 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2021-07-31T01:19:52.747 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcI3X9mNxtHRSvi2gVFavu21kRCWDd1erXW3d/1zWnoUvRxe9ft2xA1c3Bmo067GqhFudPcXoYJvz3atAGR9k3n1UTtTJe95XRZ1Zmn/boi6mn7I2oRVFpW/KjxaiM3iN2SHKknMtcs6qRTioapYCU7u924oc0izqPBvkrC73cfVF4emH1ERWEcQBTxWub6N6Mn3xaDmJ5xnGlNvDY70CC9osoa2VRNJM7GU9eUQClHirPbSooDvvVKfTGD7YfMH76j6PdDR/a/cgNeiMevtMbeFGhu1SOEHb1ihCNoGq2vQFPPWwwTCzMI05XuUEc4iaxw/kbGBMpWbmIvswIMQrRwgwLZk67GGpf/dEeutI6WseeNKNuHnTzzx43AW9G2coGq0RKE7OuA5Cqj0CO7Kcf5UA4AqIIroLETVBrrEV6qzpkD74QBZXPl7gbSZIyxLpvtQoMfSSzN+Mj184LmQoURCqIT+yM5gddF5/S5I+rpw81vBj5M9ZskyzVQAo0H8= michelledipalma@mdipalma-mac", "key_options": null, "keyfile": "/home/mdipalma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mdipalma", "validate_certs": true} 2021-07-31T01:19:52.759 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2021-07-31T01:19:52.980 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+tqgRGdlbb7NMefZVPCtbzVZBapxmmPcKaUtqT7E/9AKRoHuroZQ7KJQfO15POpIfEja7BugBt5F95aW2Bd9/xhulzC4/JjiZYc6ST5Na03BrcudI3YcktoGR8Mm7ZVzSojvm0SKtLLHM5Su17JpCC7BitMdao1mETlUOklEXc6pgbr9XHOayo1zeuos8DB90iA8TQ2soR/POD1Rix4GKniCHXz10+iXQFYPsif1HsQ3rh8Vi1M0cTq98bMSmfsKODil8KIxf+lvLCL4GAgDYEyxnstiTFh5GAcIMOOo2dl1fDb/Oa981CxWU+xxJD+M7AEJVeyAqqAnShvZU6y1L kiefer@kiefer-uchi", "key_options": null, "keyfile": "/home/kchang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchang", "validate_certs": true} 2021-07-31T01:19:53.003 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2021-07-31T01:19:53.224 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjj3bbbLvd4IHeFrSJyM9N6uKXA7ElQrzfcisKRGjUPON8IsAWkXQeAnHoBsIQv7tRGudXOrK+tbfv7crLI0/CxL+o0cCVW8bxuTEtdz07ZFsND9cppCc7Yq+wHsg7nQ/n0QfxnlYc0T9pATXUoT+aW4c3NQoDbsupLSkNs1C+G/MEBzz6iH6tgrv43OAOUccfQbC+IYx4RIKL5MteEjVrZanNELIG+jsYe+CZLdsONk1nR8+xKQw19Sm/jkKdl1K9NXoVPEBQmFXrYl6G+7tCdWeU3Vq24lM1X5p1mVspcG6FyyctcuP/8+zhL2L208DrXlOVPcqHR/4FEf3Wmk89 vereddy@vereddy.remote.csb", "key_options": null, "keyfile": "/home/vereddy/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vereddy", "validate_certs": true} 2021-07-31T01:19:53.246 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2021-07-31T01:19:53.447 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+EkUbnaeC7Dqtq2N4/dX9v+WGlrgRS2ygHhlbZy4KHbd6+Q9T5NAWtqnFwqcbFpYYUBwMvz8zLo6HYN6gNskRN0qTiFKPQdtnydF0Tx1KHrdGGcX3G2lxyGH5P8nrqReXAPPvRiVc2/Mj7APCTmY2JftkSS1qYcTleeueivVyho6z6uD1AyC7OXANhtn7kGeIY/NvVKBhDr+3TuPmpPwPCQMcJUyZWCHYyp47v2vzNBy2Etp7WjGwOnfZ5ULiJGjH6PCXx0YfpmTUOPVkNbV6Om+0YJ3fcLlmQ/2nsdsWfOKvoAr+1uzO81h7ZBX9LVEitj8YU3WeSCV2vqS2VvOpq5NyHXWGq+ZkOCfhy4Z9MvMG+6hqp4p4z1JMpKsZ191kP7eAQua41GkTSQPlEfF+um1UV6ZjNzhB68utOXyGDoDcxct9oOdnhhjwu5W4/0t9iBHuMn02iHRYVrUWOH2FXVysR0p5x/L2xdvNPiTgFAZ2eN+EFWmdan4e9mAAw3M= gbenhano@localhost.localdomain", "key_options": null, "keyfile": "/home/benhanokh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "benhanokh", "validate_certs": true} 2021-07-31T01:19:53.500 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} 2021-07-31T01:19:53.683 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBYtdHKBXQcXQFkEW8fSwlFK/xZM16VcFY288ABI/DPfejTStHhxiRvmlsWzMy0Ar5WadZ8FU1ptjA3MKBEY1Do1r1Ul369k3zHCvxXyg9QGGMNPYfHmnNeoF4GEgoJpuounPs0mkLmrPTR+cR+/XuaZZt38nsad8M/zJetnolvqCWpcV87D7+Rt8F32J6kGDknC4ikEe8I0n7F+goOmJhlgkqegbKEeJ/EMnOItwVZHFwaC+sxOq4IlYp2f1aZMKGAtr5+ICwYH1SMCcvnp3qWL8O//t2tvguZgQxzIs6Nf5GaxwdQkh4CvjbWiCkIbGFJ57YICS6k++KMXMbEnwh jsalomon@Josh-laptop.tlv.redhat.com", "key_options": null, "keyfile": "/home/jsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsalomon", "validate_certs": true} 2021-07-31T01:19:53.747 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} 2021-07-31T01:19:53.916 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9ltUNitnn0FNwMeiGkBnozpBN5ziVR2MreaqkfqTQY2TgI24559xuILyy+TnAKumhemZ8i8nIQ876Iso85OzIXbUbRM45hpL/bMQX7pWyMIRNeFKrCDwf18OT0DPc8aeDOAFQ6YoN0j4awUENjdZuuAW80cfcjoQO3m7ORF8wBF0IvMWwPDIQQrz+Li31ECgXg6VC7K3e+BNHlfNQ/2/ahS0+V4J6JYFsylZL0r2tydhDMd6rR8QA9c9BgeBz7+KWobGtiZVe7KReTzJ0O38PxNd8br3aS50j5BiN63gch8eFxPAROaUMnuiavu6OTDy4/aeL3Zc5WpUy3VyyumWf jolmomar@localhost.localdomain", "key_options": null, "keyfile": "/home/jolmomar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jolmomar", "validate_certs": true} 2021-07-31T01:19:53.990 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} 2021-07-31T01:19:54.157 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQKiUAlB8dxudAY60M7jnIJvs9yb2pNaT+BVv1Vk1kMs0XjPjcw9cyT3DtQ1xGLF367phRi/cfQbXOYbHG+uyBzQ4EEnBFvziktWRx4mfeWYxKjkjbfkVfpKPsxztqCbwm6+NYzMxGK2cExN7jUHnghHAjfA0IUpVVg9peBD3kUveLwS+UxuJAnuj+G4480LS6GDyLrhFEXU7uOIIlW4S9oLnU6vJL6GAzwyLE4ijTA8Ek03SJ01Ey/gpU9AqnQNp9u12l/7sJely6sI9pAuIzmS8Ht324hexxSWN5sFjXgLvLfFK83ziDBdrFBr0WXnWsFdRpHs2CwSq1JTkByqbN hmunjulu@hmunjulu.remote.csb", "key_options": null, "keyfile": "/home/hmunjulu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hmunjulu", "validate_certs": true} 2021-07-31T01:19:54.230 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} 2021-07-31T01:19:54.387 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoefxOfeArToGucfWG7y1tyLF2I1J/IkBSlmxxEJyHyDUB6ElXjYIJReoiPfmB0xLEmINpmt8QA4XN/8UbR1r9LFOexlZZbX1LSbihi11LXCvfIJWqkHVwIyKHRTu/duMrntBFVENJ0LVf0GSWVTtllXaDA+fMkp6/ra/hlRHQbx19UXy/9+Pogj8Xl+SntnMGiHlX8FFq2A2tUqhdQSf/1GW1HfUIa5WtwdEqIpKpAgMM7sOW9i3QAmLAMo8HiKrArSiy7jrRUODCqsTJsfifxEbmnOS0ZtmDh0eeWL/Dyqbz99Rg4WbFNeK8oyZYr7D6bEAhd2az2A+wycPx4Nq7 julpark@julpark.remote.csb", "key_options": null, "keyfile": "/home/julpark/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "julpark", "validate_certs": true} 2021-07-31T01:19:54.470 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} 2021-07-31T01:19:54.629 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD03otezqmbR7v9dq62mWH5Kkg692jdmDZk6Ve4vux0A0YhxBuXDCyBWLE1Z4lLNxQahvK0pGsgokp+Uk+Q6aU+ZnF/E2jLp6YWerFXIHY3M7byObJx9+QuVBfNI0bZBkV3+vSjhSCGOYU+TrTHsKZqhsTcejZjmL3gFLmWfU6dTh0sX05qz97cdJl3CMPhCIR3KjtkiPygj417/N7wet/Efv1t79TvyS7dtK1oz+zRLEpKi/fZjvpXb8xec7as72Y0XexsWMDyZJ/N8PijTQ5L6EfLMV7ae/bWGzNAKsoC1dHVY7q/Q4pzktC6zU0Rc9sDRMtrgFg+q+04BfveSLfxTb9tqf7SbSvh7IAG1eKiEuDT/uy3T6Uiw+CiZ0eIqYzZscO/aDjUArGbS8eyaycybdYCBK9Jz38i0mYKkXg855yoVuIExr6MXieYwyVoVW8gbx1bAslaafrzra2dkAFByOJ0sbosPFRgR2+boUSxD6IaE+ALgno4T/0CI2RMiH0= prashant@cephdev", "key_options": null, "keyfile": "/home/pdhange/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdhange", "validate_certs": true} 2021-07-31T01:19:54.723 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} 2021-07-31T01:19:54.849 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDg7xW5+Mk7pG5MjLxOc7GTvDTCPBJj2L3MEQZIapmmo6t8uPODwDewXncSzrKDH6zFVE4TTTD9fJdu+BhyT+jCFgXrEOMMabj7vJsg0wO0SWqr+4NECFPhD9y4ewfYiewBeXisclA5Bto91v93AtYpZBCdmPYSi0BEcPVc6S3iBUD0PzIrL62I/kBAtm0f/bScRiT7XvQJXvkbEObW2yS2XhUv2qgq1VlFvO1jl3B85KMTHVb8+LrKpQWOKfRfxJ74c51y6tnEjZLFO6Xn0JevqOhd8gPQIXueFZOM0g0pNp1mbtGaaBlEIibbrzo2p9kzsdvJeHMq1XuMIi1/Qyj9 dpivonka@dhcp-41-144.bos.redhat.com", "key_options": null, "keyfile": "/home/dpivonka/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dpivonka", "validate_certs": true} 2021-07-31T01:19:54.975 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnuL8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} 2021-07-31T01:19:55.080 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0m+8wna4RnrOcuCUj971EBe4cFiTXVkEWkIsUX3yeCUwMaS/ZjaoOjcdYX7cYpCp1ilo4FSO5BsochU0RiMbQEwLGcE0zB2q1ac8+yYKCuWWyknig2t4bOReu5OrXvl6mj/9R+p7clFlYhwIoNG5ZZ/2eTbo3nH7gWgaOKaPbTtvEGg/5R/qjQKg8OTC8oix6kLsaNTisQJ78Zd7b4ukg9bO0jlq/2UhQOBM072sU1/kJpWTkcA0xxm3nsC4rWPwapgz7UwBmMdyx116GZgWdfQecTLKO7b8pyJw21npk2ntUmeMi7WPZGX6VI3IpaAhBjW3KiATAb8JbAQh8F9bmejkw6x7zrP3hYKQOg3A+U8DKRRo/Ad+f0zz9gjC42DUvMEOr09RBdxZcEAPY7H/YRWEqbA2zPX3Gfth4b7/letzM3m9ToUvVfSt1CdjaZD+gIia9pCgDMb+j1j2w9reG4F7Us+W/1f5pYCyFyrm4XO3AdoYM9/FOAOR3vffTIQU= adking@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJOefD/VJfS15LRyuVDLluKv93Zgocsl8BhiDMimvBNbrGWc795v2+eevWuwaL/SlL44SnsNZ9Mz8FVn9YuFoQiHLg7cHZNLg9EJh5SMXzZzwLZHnF02MdI2CU6U++9nmh8FtBrdBWKzZ7o+01wOw+52dYOxvP7IJ7C/vjwOLgfigEN8Nd1PrxALVNIQz7fsiet0n3odk9PJfoqTu3IloXm9qbQiPFyfMA6b8a20RdLZCJDGGG5rd6gtsyeFHoAPZEz8FEUlNkow7NM7ZhdxOz/fWXyMZ27KmY8WSAgflbxysMzexyifkGzypiE3NQ10hdPGBgoYsHrAN4Qmte1/Z4aI5nxrNzLy/uNc+L/Bp/0XkLj9zfoiFFOReXvlz/laAV6LpbHyacr+iCwv9rlI7echXmvC6mSJXfBPuyR1uromUXpYRhEkn5uiOsCIQCuYpx2jeil7JV+LJlF62lLSndOv4g12Gq2TEZ4Bhsr2902kftPkLFenyfDTh3A2qGGNM= adking@localhost.localdomain", "key_options": null, "keyfile": "/home/adking/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adking", "validate_certs": true} 2021-07-31T01:19:55.216 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devBox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} 2021-07-31T01:19:55.321 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCs6q2b5NcUsNXl+w1Mrc7idNvGdwWkL0XRX5zTw22UNpQyqPkSrRTFKi4NxoY8ucv8JoLnaJL7hYSRHKTh7SqH78uPLZX0FLtoHvpYLgUgJzzJ5AOW5uekIhCMCm+3egmFnEBwFcNxtGnU2es02ByXNEXRxt4/LQ0f8K6lMqxVRSKYcltvWzlA7WFFdTnYMunbCYOdpw88RXm2pYjGIHRBnie7+pFiFSS6K/mkzQcfLiJmahgLSaYslI0ho1vQd8xfNLmZ700C5F4VTUlT/mJK/dI2LF3OfaPTsyqKNzOchrjLFx5b6dhZ4WZyqu7VKsHNm+xHZwT8VP3LPGSVVBdmEsiCsmV4g4QTI5ssZLi7siZDpaFp72nVAIO1i6kBGuiF3or+J4LmIh8ZRKzTc0DRDOw7S+IkWu1fv2UhYg83z6wrz4HeczI+bnolFFWW6HgGV3MFaUeRTRvrbxw2DPoEGQ+cjP0wsjyEhxlScwdh8W8IbVpNl+7mBe45YGUH2hg83Lst5QQxD4PHelriu40moWRHFKCnKfNesb4L1uqu29o8f+/PgpLdrWE/P5zIKe2ReWJyScYAI7jTMlFw+xgf+0TKryk44NMy90l3ei0umlx4qsFbbEv+lhzAALrUJ/da7fsQw7eIVykmcscBeYr99/YSYk4MGdk4jVCiHsQkdw== amathuri@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJ3ch+pU88/tX5TVVFl/1pPief8drYLRkK4BtfJk+xVNxX9qkDtOtiLAU+OugYF1/xk9hE9zTbm3UBL23unrzhlR80t8f2FiF1YFhHVzeZ/p/OunmkU0RFQzkMVuwKVOUNulw9FwpEZ/0Bjfxvaims0SBWaCzBgE5g0R+Ofdq/bVF82K8vU5Laa3KZ/cCdGzFgW00cBvPAKIpQ93g8TRqsCsFvR37nY9v5SU1hJ9araEp6LRA86tUt3EFtxD2tRcmqKGtCb0kOr+rRXWVGiaUBiedHxFudIy0+TNj4fNrJwpoC0hcBO9ek4Bh6hM7Zqzum9HpkPGlz84jkm+r+xMThaZCh8iFkgxyK+bw8B8smaVJD04ob9Wb4RYoFYfQ7TEkPkO9gr+UsSesbP9fWfucemUqwK0htOOtpc1YRwdOPcdPJWAfx9J5+YVL8BTWYBaTdmHslOKeUGM5fCob6QxC6Er9xweys9jh7MXttSpe013bY7P4ZDRF8uWvygpsytPE= amathuri@redhat.com", " 2021-07-31T01:19:55.322 INFO:teuthology.task.ansible.out:key_options": null, "keyfile": "/home/amathuri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amathuri", "validate_certs": true} 2021-07-31T01:19:55.458 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4pIcNiZKVIfT9GxyTe/B8G2GyHKxrAle3iExEzLH5bBCKtBrottX1y66KBzQSsrAb0z9h+A+uxlswlY6VM/gNos9VRB6x4NRB4ADY1Bku+JsKs5aXwRB1zz2ueZhrFN9hJwMl1heLICx85ksHBdm0q+JPdUAV+/LBdr0H/IjWveV34WL3Awf7QikzyKhzUciMB2gN9PJXTW0bl8PP56y60ys++L3nMSRJ+15uPcWl8MS3D2vABII0Tu+3PsxzrTTNodElyQ6G/fwyFfWhdp7fkidjR7mDDx8NyWfCl2rKejXv9LnVC0wb3tFoQqvwdz6shN4E51EIEGXHcLUeRpBhL2pFOqIZ0qfnmLIQVZ88RKMTgo42CgtmS5UzxCLOyKOh9GxF7DDEajTinJFJJVbNJ5xF494H2geSnmP1o1AecGsGbCZ4ndhS9unSfIZLAyv1J3wZlAlMsvxXgY7yrh1srUiASVyKK8jba1IT+v3TXR8kQUCNsLKKbvuX/l0IXME= xuxuehan@bogon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBd4eSizc2zAkWDy3mFZUhELO2wXDgkVxDNM/Nw2zT1li3L+6YOjFfdT4NOxpkg5laLd54bdKb5GVRA6iKqAO+aqo53xHHJGC8OY5eJaUPjtpypgIWO4Qzbos2ZzsC+nX5jvi8O9nM+JkabSW5iFv3gGPTqHixFYkxK2NIBa/UtXFOhmb1wZ6CG5Q+WObCosBUEMWokC8ntCADfhi6qDsO2fol6mYUhYEE/0nhQ5uLRrVIM5QM5dhrQVEeAs4KcYpmPDw6q6AKgx/f5zObh5OSJFTVsgx2cqzdQVJVDDx2OD2QVRtZ9fgvI2EGmKSwe1TUQOdqa5bmsVIp2Jfi8MC1 xuxuehan@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwIokXSvPYP3TRlSw0SBgmMFW5jVltGf4c6CqjIW4KKCgsL4c4Vs0FztRyDGc9p0krZUoGJCZHErVToXm6/lSq8JlfrlLX4X4yh37fUifWqc2UTJGeH8kLaP3WEt9vT8eQ9h8t2GqlMlGQU8KzVAdC+c5IwrbGRsS6S7D09/y0rklkhdJ6nuj8cssN/2TzLF9YIvVcFZHdyznk84eB7TgpPKD88MOpTnEgWFlD4vAWEuHpkuxgj1KI9LscgSFcW9/RxLeRxcV0GypZ8jPmczoCnin/ZOstCTpbcmjL 2021-07-31T01:19:55.460 INFO:teuthology.task.ansible.out:d+LW1kgGXXQSkf09iHeJ/cyCw6ooZOkynVwwmI9Uz2ZaMedCUBpG0bKclIerJ9jn+CoTs3U3Orhd9+6MVtCHX9VNJbVa4q0KCS1sp5S3HDrYkgUyawUxdYorrLZcqfRHWWo1luiridmuQdr8lQ6dvNpz4rhsCrAG6S1eeaC/s+o0Rzpv1YxU64JzquZ6WNR+W3U6Bg3iYkSoDCK9byf0rsptyL8= luqixi@luqixideMacBook-Air.local", "key_options": null, "keyfile": "/home/xuxuehan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuxuehan", "validate_certs": true} 2021-07-31T01:19:55.554 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDr6H7hmtGCUrO3GppgMHwbmA4XT/Vipz/uLG/N8h6ZxtaLQKpCefRpdKcFlOdDwxkTmT5q9vmXQ8nl95lTGq+eeMxl4ho104ZD9e9AIYmgYqFy7/qNP8gkqduIPlypLL9m5rElTcGd7zqfMqgljMvAmnXLtZNJlIuFtszVU5WPI1QCwfQ3t5aHF6j66bpVoN9uWItVGB1LApQz9TtViWsQ9Hl8X4YauJToO+TkuSX+H/OGH7ZC6CPOzNZb3b3vF/jLMnTzXl86PJs1se4DjKkRGLVCVbr2WhLwQeZ8TEn3Hg1Gn5X8zNM5cTo6U8SnXBAe9ZYbuyfvghrNLdlZpmN+8l0/l5zC3z+N+sEzklXwHw4MQLF+KSloVFd9NN/kyenufTZAyp8VxllZ9TgdC+CPORIWyzmu1elOtYo7LNuE6CIh/Zm9B54SO6hVHhVoOCqe/dIyjeHJA0WKIZblK1XZEvWuu0E2T3jhxfe1GICyTukeB8WoBQH9DZ1yzMPt59E= nia@redhat.com", "key_options": null, "keyfile": "/home/nizamial09/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nizamial09", "validate_certs": true} 2021-07-31T01:19:55.694 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMOuxxTZyxz/TEVA+ODgtLrNcSsUhKE64BZg/Ub96LLffQXQc9SYT/0YI4wxu2vSsky05PnVSON7jqGxunCt0+AdRqxXhW4ttmRIOz8rq8SiMnnLayV34nIF1G0aYTkgY4ej4+VmSVIHU3bb5mp/Q3D0hxQwDPdgOJWNyW9oYerHbnx6lVgchvsZHB9VO09GNHcAsyUz4p+xBf9iR7eS74p4dZmUbth/oBBqpOa4NgNr69iyZ9a/pNDLxaVQoIpDFrevk5UU5nUHQq2frFNxF/Y2Ja1EBXCavvPF/m5v3J3xNEu3SI/GbLFCPKCKeBDpXJbe1QqTBKJjMl9qGmKEJ3 aclewett@mac", "key_options": null, "keyfile": "/home/aclewett/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aclewett", "validate_certs": true} 2021-07-31T01:19:55.779 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC//ZCaEeVN/J09FtsvuZ4eEs4rTBtYoQyG+ZpuWkJLL90VyJeYrZzRsgvveKW2W9oFxCFcMy7IBhLes0TZ/X/4Ox9eUmP/eyqK8fBqT3K1F1W/uOmKLFhlQhkxxj+p9BOMBD5trfES/QJQ6UFSYIS2WhWaitDgONjoLMR6wXZ9r7yZeQBxbmG4wGI0Cwx4Wgg+I59UbE/5iUfnX++gN3rXrPnv9V0yFeciS4HvXRFXdrIzvQFHcZMmpOOmGruHXN/ME2dZCG+kaZhZdI6kICSlOwpU7TU1lv99zjBCeEzGfFkjv041ykKdNVR3qE0k5u0AOMqxJBORL/woSCdGvrBXqju2G/2ohl3B4zD6TAve+JLfWOV5C20NFRPPbTd53JaNpV9OQqwOfVniowzSYYBdedjm9sq6YwhJt8iBXze3OA8ZO0GPXo7PC2MyrrUtu6uu3DZqU4x0cl8iJxqOlx5pH0Gydj8nuUG0+X6wHZzPXZRe5M7/RzmvXch9koG6FQMiikNjUkK8Guz4NMBdAYUpxYkdPGBhiuELoaAcgIVzSGoZTuICpNYwAxBHk6q18G8Qd8vaeUdGdim4YvwycsSOwFE47wg/XmpCHrNJUZzzB9UD7Kn84TG82LAmCuAzpzcp0uPjsBckbXzj9Zt0hRgF80yDgUazsQkbgtd2p8KxIQ== loic", "key_options": null, "keyfile": "/home/dachary/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dachary", "validate_certs": true} 2021-07-31T01:19:55.932 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdTLDprr3DXtsgiqrB/miglJwSkY7BAv+OW9sC0ZHQf8Y4dFoUtr7q9fd6W7ehKL6aMjya/8eQSMPRoEIZFD8f2KPCuE74Wgwgv2cpAkFmLjcPtjixqLo3LGFCyEJWGLQxjlthdzx59mpCWgBsEJQmlKP36JIYjILIMtXtsETIUGFa8QPIn9sMdIgutPPf3yB/6DHcR7C6yIAuvbJactEsiCt4C8+uJinvCii3PzYOMAEd+mictPGk8nOCQFZrI+ZmwTz4UbJylxFqnX8lhhoPIUEfNAXlb3lkZWH1LaFOxarbmupuPYcg2rT0+kFtyNr3/mAx0FO01jADYooqcW1Z cardno:000604936806", "key_options": null, "keyfile": "/home/cblum/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cblum", "validate_certs": true} 2021-07-31T01:19:56.029 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDwVh9KqjV0b+65aBxQV66IUwNo3D5zTbN8U6Kh6eE4QtaIVe5Mmvi2SBbGMSWofE0zIIw2JmsI5LzdOoCyZl/3gNDPHQd6D9o0xRHQv94giZ2a5K+mW79foAxsuppTjV/Fbg8EZ/19W91+BqMWotlMR66ka0RPT/6U8V5mzXoydSminViO/B6q7ZYzIIgGZqaTsSxCMvz5BAzzeEaGfSlGxw5MOye9sF/tDppuRKG4rMUZULEWGuxgXAtabtXCduocAArqAY3BRJEgj7JetPdPNjipGrULyYQmT7GeknvP/waC0sPm5o8t7taJca7Vu9qKKMKPAqiK4KhlrW/yVBFliT2CjAdVuYvGj87LCcEn6NMeF+4ONwDz++YwjrAWQT2zHUJPvLnYOZPiD1LUm3F34cEHD1y4Uf+oeiYgriUMauskfFn+ShbTmWWooFRFyNobzweg9aMBZSkA+kFFifqOV1FwHTOYg342NumRGB8svK0vvTUID7AU1Ev8YXYqCEGbb0UHth/1pEbAF103enLTtPZ6qKua74rE0qJ89OF9dZXv4JY9+8Juyps7bi2XEyHNm1nHZOY5tqdY0eKFmSGJMoWBDzGgdMjmXlWp4jnY2fC619nZqxMrCQpeT70CqnBhtBemUuu6jHYUbbrPjnXIkH3FZ7/qvZt0jF0IMP9FGw== amctagga@redhat.com", "key_options": null, "keyfile": "/home/amctagga/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amctagga", "validate_certs": true} 2021-07-31T01:19:56.180 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDPUCIVfNQOOG+uAeKWo7tLAdgGO/oQu4exTE3XLfyTVdd8u6pbQCmIDFEnbGgMtlYuWJlCJtxJIzEZJ5HIwf+po3y7kb7HxEeWnEb9DsqJB4nCRE8fHnNLgF3bh9gvE7sOZdNXEKqtsnGcaaS81oXQ6QgkFGQzlOBOqxj7jIvad7AtNWD3WbJi/uTVDsWcukU51tn/aEW+hHMD7Y5QJkLH7O0PrCeEZCUCWoiMP7sNwACBym0DsqX+POkGJOaGx9EeNhGVA4B6o3xYXJPqmFbgMWMfG3/Y3SiheZjxzySMQ7FKW4Q817NaWqbjtLZQsSZw7JNqNFkcdTzVZwwCOJ1AAecajpniKeQLnyiWQ3Z/EkSGGwtDj67wo7uqT85tpOyLfYhQm6Zp+xSh+I6s0YJ0xO/huLKzeT5D1r5vVMk2VdcLD3CCxXUwZgaMUxYAC2Wq6mpIkxVZFQLbGuA/dWkQbFsrpbf7ES+kh+uzp4R6dQ29OjXbVcSfF2vyIuA+VUKoyj370YOZQhs50yuAJ4BSHlEgoJdRzi3XUX48h45XHiZcCY3X8LTX6a+0KAIgqeAWdFhRgQtAxoRYjhOxC3k/boBc8sEQ3o4pkxnhtE5Y/tnM2CPkPMX9HWYirNf9GyZTYcR/5rDf4ETQ18G68WLaNK758OuUPjSkwttcyLd4dw== mlatimer@suse.com", "key_options": null, "keyfile": "/home/mlatimer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mlatimer", "validate_certs": true} 2021-07-31T01:19:56.265 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDtiHTDuQk8EDH5R+JfDMucI4ZpOzKN5SLD+j35QLLySgGXholoDxtqjHnmRjM1rpj5SIaXjHBZdJi7iax+1MRywHYBi5BfTtzYPgA+NH8mt+XkcdkEuFPcYxGh6fMQDQWb2g9Euze/6qi0W9mbYXvvb1/Q3bhVCXg1UgKD42BndWEpbMYFfzJLZWPtOD/lg81pwUmf/4BpPFv52sy87lQVfkkvcUliLObR4S0TI5hPLYxBS+RXmHYKgvPMB2QJ9wum62lJ+YM1MxPygS0OWbIB6UQGXmvxRXA4osXkxJ8ksPPCupOgFNiiYro91cQh3mmi6iHc9kMz5R1aNGGOfDdlnLx9DRSIiNcacjXEVNFaoM0x441f4GeU2ZR8/7uq2dfk7QjPlh1eJGxG9pzRiQ8JCDF1car0eYwXdMTsQoQqpIi5imqcELdcqGGp9FhzFElgUwpf15JYlkouVhS6hq4LDjhLNk7dDT8fmCEnHOPcbxaTgN7J3idnJK6zeIniuuk= root@iZj6ce071s2zz3reioxn93Z", "key_options": null, "keyfile": "/home/kevinz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kevinz", "validate_certs": true} 2021-07-31T01:19:56.427 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYh28a9R3a7jpRfPqVAL9EcD+S6KkZzKTfCmRn4s8Fg39kUZsgnWUZQ3jUxKSeBA00te7WMR8CsIEzulm7f+/egNuPcvzNI/SaaCDtVMaxqlEuW85ll8uBQi3s4SLmOu9oS+5oid4cqTrGTcG6uhl29oWS/53SxoGh+T+y6GesO2cYRuTpcUHxGMvIgZe/bKtVGdP1y7rQd62e8emb9UupMBxaHpkYj9AaMuLjovHbBgecWbe4PlzPWNzl+eITVk2d6CxxIem4/bw9VHkRjQZlhL2nbwRgtczLYhlaZAd4ePEtsqA+7nTDjTDhXXHVIf1pDAsJbZEdgQCSGH3FzxtrQ2hZJrH/NIKdHk/OtI06e7POSGj7PcHzTvMjQSFnTpJXinxwzn0lvFSlmABN/00lIE0p2ZXfPs6dgWJ17JnTq6dMUoTiGhfKphPHyfSXJCzfa2VB+lQQ5eAaa9cRXy9pLmWGHXVGz2hsBMtyYmTjR/wr8kdTzeK+Jk4opBtGgcxDustg59b9x/jvgoc3HCdtnkqLTug2KmutB6kb+4KnVO07GsSp0RmNtDmCRcXs9YN/eia9ICtIGDWPc3YDRppsw1Vktza/+ZW5vGzIKILZTo4SNGcg19zQw4Qxd9eLkXfB66vE2CQvA90CGo7mcye+UWjjXLr7i0cDEszAloQ9aQ== srangana@redhat.com", "key_options": null, "keyfile": "/home/shyamsundarr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyamsundarr", "validate_certs": true} 2021-07-31T01:19:56.495 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1YAaxAvlRgu72huAggHpoDKSgtF/b00ACfUX4Wk5NyLFbYI3PAQapHNcNL4mIsH8pc6YkdjcJ5N/F12aEupJGQ0Q06OS324xapnaBoUy1hQl9IvhtPqTLEwR0X5xLIAhNBLIEJDNVg13MKwr0u5YVUXdrQRu7Kj9RXAPBahNZFo/Ed+cnR3LICQWliHeHFrk/DU2DUHol3a5wAPxrowAG0AxAecP6/7rASMmitvLQwdQcfIJAhWYgfyXzudg+gOmAEgv3O3U4QOikgmkm/EabUXLJnlTj6yuhtuuctdJr73UT3wLGAGmphLKCkug27kb1nTNoDa+PW/+grP63zhU1 chunmei@otccldstore05", "key_options": null, "keyfile": "/home/liu-chunmei/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "liu-chunmei", "validate_certs": true} 2021-07-31T01:19:56.667 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRZyOnj6D9tCw5o4+ujpos5Uj1O0XVvea798DsXPRh8bEE1pLkARQJLv4iW6LsG3x/CXruahmwT+6C/VHppIsbgtpadMWdCTPYrsjctPK0/N9nr4kQEL6XeK3/GHFnwqYcc+LySpNNfbKrgN4qWOwdZBhQjSV7K7FJd4ZYSm4wp6TVbvKKp7o+JdeM2zhstfC9AcCtBv/TpxixFFq/+y8/Z0PwqN1YyHduDOr1dr1DWn4ZzrkhiTYfPOltx0jweD6zTJoTAJDqae7kBA+3XH3dUGX3n7Tchsa3bnIKVz9sLQvH/vPro61I+Xig6MQh77JsylNvpwb1LLOEzDf7dxl bigwhite@wrjj37deMacBook-Pro-2.local", "key_options": null, "keyfile": "/home/shuzhenyi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shuzhenyi", "validate_certs": true} 2021-07-31T01:19:56.763 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDLbPdDb+XHOD2fo8g3RacATj0cG07uELwUvqQP35y+JmhNRSb/ttaysFLA5ss0y77TiXx22saP3ECEm5SnuPg5IERkPQmXiw84ATTNfLnzx2G5H223SLaTQiGfSgniBCJHVPCJfNahJiurlZoOAlwDz6SKXTUiivVrFJ22iTBDCQUO6HFQKjE7aM3RAo5mZXKVlTXzM6WOd8vlzk3HMzV0/91JR6H86Lt9MEzXT4dOzqmtxd8gS6U+vA+50zOIspdK+orTRJ0PO2Fn8c1RSYM8g7uqQtpDcOEHderW8TRsqBbDxpyslK6hRX5urUNBX506XE6JNoViLNvCofgseCGj yingxin@yingxin-server", "key_options": null, "keyfile": "/home/cyx1231st/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cyx1231st", "validate_certs": true} 2021-07-31T01:19:56.912 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcI3X9mNxtHRSvi2gVFavu21kRCWDd1erXW3d/1zWnoUvRxe9ft2xA1c3Bmo067GqhFudPcXoYJvz3atAGR9k3n1UTtTJe95XRZ1Zmn/boi6mn7I2oRVFpW/KjxaiM3iN2SHKknMtcs6qRTioapYCU7u924oc0izqPBvkrC73cfVF4emH1ERWEcQBTxWub6N6Mn3xaDmJ5xnGlNvDY70CC9osoa2VRNJM7GU9eUQClHirPbSooDvvVKfTGD7YfMH76j6PdDR/a/cgNeiMevtMbeFGhu1SOEHb1ihCNoGq2vQFPPWwwTCzMI05XuUEc4iaxw/kbGBMpWbmIvswIMQrRwgwLZk67GGpf/dEeutI6WseeNKNuHnTzzx43AW9G2coGq0RKE7OuA5Cqj0CO7Kcf5UA4AqIIroLETVBrrEV6qzpkD74QBZXPl7gbSZIyxLpvtQoMfSSzN+Mj184LmQoURCqIT+yM5gddF5/S5I+rpw81vBj5M9ZskyzVQAo0H8= michelledipalma@mdipalma-mac", "key_options": null, "keyfile": "/home/mdipalma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mdipalma", "validate_certs": true} 2021-07-31T01:19:56.999 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdqw+8an6CHqsagFqXfuzQi1zCFHCh0e9pDtYz+ylA7L+7vnJLjTR7km0bb4Z6VTUh7h0/nUfF6faPzD2KGmTKvS+mTpeOUaN/H8O8j8qqh90f8z4T8F8L3DwqQbPf/R0izXmTgz0FZObFHmj+G8a+D4ndxF80EPzKkML/G0Hg2GEVdYzazSY4imSY2f56ul6qByBknhFshaTo2DUDcdbDDokF6lO6WgXhY0MJrT24CUSMyZmyi+3hF7vowW2dfbJ+SuDft4XmPDnIxRhDWWvi12jZb8Le7SWYzUlu7fAGbWe63VJqNh992myI61aZshApDD24+1xqIoo+Z/XkQD3l sunnagar@localhost.localdomain", "key_options": null, "keyfile": "/home/sunnagar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunnagar", "validate_certs": true} 2021-07-31T01:19:57.160 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+tqgRGdlbb7NMefZVPCtbzVZBapxmmPcKaUtqT7E/9AKRoHuroZQ7KJQfO15POpIfEja7BugBt5F95aW2Bd9/xhulzC4/JjiZYc6ST5Na03BrcudI3YcktoGR8Mm7ZVzSojvm0SKtLLHM5Su17JpCC7BitMdao1mETlUOklEXc6pgbr9XHOayo1zeuos8DB90iA8TQ2soR/POD1Rix4GKniCHXz10+iXQFYPsif1HsQ3rh8Vi1M0cTq98bMSmfsKODil8KIxf+lvLCL4GAgDYEyxnstiTFh5GAcIMOOo2dl1fDb/Oa981CxWU+xxJD+M7AEJVeyAqqAnShvZU6y1L kiefer@kiefer-uchi", "key_options": null, "keyfile": "/home/kchang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchang", "validate_certs": true} 2021-07-31T01:19:57.234 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDTlsh9TFwLCeeD2RR+56VhRZ0hp9CHj72U3Z1gbMpxpcg6PYEDzb+0qPVQp1/cIhcwwy0sm13yFl/A5i0iCRpyqB2MbyaXjj0HgkNinxLBFkPSiW2C83lfAyAhG/gLk+FcmWE4+xVrE29aoSM0JtcNtp+q83UGdnFcvhFe0pgJbt5vITo3ZMJND/UZrzeewyK8Vy4Rvpfy8aHYi2Eq+7WQrRHwUwxn0CwsxldnUfmZtu58d+Rtt78LRF0mvcScNlC7q5D+8htmYwkWkK639w5IB3xPv+Z5cV49Psi24Q+9BIr9Zo9/OnnCTauAi5ivDlh+kk/iBksAE8rQpMhvfuSFnNoq2O87aPQvNQHdRCcOa7FVIDcuocu6AB8UP4cGcJreT4+BW5Zbk6vQeJZhmj9yExpRj9MuONy5793nKuAMPn9lOjuXRfK7A5Tb3OOhpEJK1/MjtdZz4mNmFFfFIZ24hgTZO0pOiMwDf/tQ6BG7H8NS9g3QmXPyZajUocw6Szs= lflores@redhat.com", "key_options": null, "keyfile": "/home/lflores/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lflores", "validate_certs": true} 2021-07-31T01:19:57.405 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjj3bbbLvd4IHeFrSJyM9N6uKXA7ElQrzfcisKRGjUPON8IsAWkXQeAnHoBsIQv7tRGudXOrK+tbfv7crLI0/CxL+o0cCVW8bxuTEtdz07ZFsND9cppCc7Yq+wHsg7nQ/n0QfxnlYc0T9pATXUoT+aW4c3NQoDbsupLSkNs1C+G/MEBzz6iH6tgrv43OAOUccfQbC+IYx4RIKL5MteEjVrZanNELIG+jsYe+CZLdsONk1nR8+xKQw19Sm/jkKdl1K9NXoVPEBQmFXrYl6G+7tCdWeU3Vq24lM1X5p1mVspcG6FyyctcuP/8+zhL2L208DrXlOVPcqHR/4FEf3Wmk89 vereddy@vereddy.remote.csb", "key_options": null, "keyfile": "/home/vereddy/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vereddy", "validate_certs": true} 2021-07-31T01:19:57.476 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGq2g8OFN5msjIn5CNB2BRss9eIyJRk3r+7RQcnBOQid jsawaya@redhat.com", "key_options": null, "keyfile": "/home/jsawaya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsawaya", "validate_certs": true} 2021-07-31T01:19:57.648 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+EkUbnaeC7Dqtq2N4/dX9v+WGlrgRS2ygHhlbZy4KHbd6+Q9T5NAWtqnFwqcbFpYYUBwMvz8zLo6HYN6gNskRN0qTiFKPQdtnydF0Tx1KHrdGGcX3G2lxyGH5P8nrqReXAPPvRiVc2/Mj7APCTmY2JftkSS1qYcTleeueivVyho6z6uD1AyC7OXANhtn7kGeIY/NvVKBhDr+3TuPmpPwPCQMcJUyZWCHYyp47v2vzNBy2Etp7WjGwOnfZ5ULiJGjH6PCXx0YfpmTUOPVkNbV6Om+0YJ3fcLlmQ/2nsdsWfOKvoAr+1uzO81h7ZBX9LVEitj8YU3WeSCV2vqS2VvOpq5NyHXWGq+ZkOCfhy4Z9MvMG+6hqp4p4z1JMpKsZ191kP7eAQua41GkTSQPlEfF+um1UV6ZjNzhB68utOXyGDoDcxct9oOdnhhjwu5W4/0t9iBHuMn02iHRYVrUWOH2FXVysR0p5x/L2xdvNPiTgFAZ2eN+EFWmdan4e9mAAw3M= gbenhano@localhost.localdomain", "key_options": null, "keyfile": "/home/benhanokh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "benhanokh", "validate_certs": true} 2021-07-31T01:19:57.713 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDNIdsy+yA0QjwkvjzTiH4lKB+txlh4oGkgC0r1l9swmFvD+MuzNNkD+oB/YscT51tJXsP5/wZDZMuk60xds4Os2ejoIBvzdGBgZ1G2B33etHPmzgTFE7ZegBkMlvWYmrJ9YS+iaMvXn0MMtqfE4zq0kemVAA4NBfFZMhHPJMKCg7Gj31MR0S0b7dcPx7Jnpg4A45oHr0qnufeqZYQaHGgo+LO2xdoV4V2EswWr7YG+b1wI6DzFZBbNsGyQd+cE1BzQ8NmETPNNpDPk5dCyBS7U+4ZTf/WbWk3+Cn+Ky47iWfxcscupQRkCjmTYI/uCMhnrpGU0zEHkbFwLTS6wqDtDmVHm++xof3xSsGJ2cE/vZySZGHqdxqVXfT8gt17h35rME8eX3TU392rjfK6kjEUD3UgwJJp4NFzQIEXi2HTmDYx/Vi9nuId3V1NrDz98QLPUd1ijiy1DuzI0BJPE/DTDmsb7hNXNIcAV8VRc1243W3eaee/Pdg/wmwfkrgjTU3s= shuaiwang@Shuais-MacBook-Pro.local", "key_options": null, "keyfile": "/home/markw8973240/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "markw8973240", "validate_certs": true} 2021-07-31T01:19:57.892 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBYtdHKBXQcXQFkEW8fSwlFK/xZM16VcFY288ABI/DPfejTStHhxiRvmlsWzMy0Ar5WadZ8FU1ptjA3MKBEY1Do1r1Ul369k3zHCvxXyg9QGGMNPYfHmnNeoF4GEgoJpuounPs0mkLmrPTR+cR+/XuaZZt38nsad8M/zJetnolvqCWpcV87D7+Rt8F32J6kGDknC4ikEe8I0n7F+goOmJhlgkqegbKEeJ/EMnOItwVZHFwaC+sxOq4IlYp2f1aZMKGAtr5+ICwYH1SMCcvnp3qWL8O//t2tvguZgQxzIs6Nf5GaxwdQkh4CvjbWiCkIbGFJ57YICS6k++KMXMbEnwh jsalomon@Josh-laptop.tlv.redhat.com", "key_options": null, "keyfile": "/home/jsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsalomon", "validate_certs": true} 2021-07-31T01:19:57.934 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+ZvZuogU+WfyGj4BR2xYllNoQqAoxQ6AM0IYddBV2pL1xgTM/SopuAxDRP0q2stpozMtjVjlvuiIgr/caqyTuOgYtYCvB09ppefqdnCDrJJBCOKpjUEnCpAKcBMMuz9V32Zqt5FgpIi0Mu7Gp/QCvB97imxEkoUunpdX/8Ri/xz0nJO+QqoJX5wOn91jG49XnkMCLzx+MuafhT90WFO9CV1sIVREZS+X2C7ACQuhx9Qwk7R4BXt2f3M4dA0K2nLwaAnVW50M1y6Gs50Wl+ojDR1itySOxv8YpwyRlRTOzQTyVMnG0Qsaxxwp+hy4b/vQW3NIUvHYdTyvqALTQdGHT lricomor@lricomor-mac", "key_options": null, "keyfile": "/home/luisrico/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "luisrico", "validate_certs": true} 2021-07-31T01:19:58.136 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9ltUNitnn0FNwMeiGkBnozpBN5ziVR2MreaqkfqTQY2TgI24559xuILyy+TnAKumhemZ8i8nIQ876Iso85OzIXbUbRM45hpL/bMQX7pWyMIRNeFKrCDwf18OT0DPc8aeDOAFQ6YoN0j4awUENjdZuuAW80cfcjoQO3m7ORF8wBF0IvMWwPDIQQrz+Li31ECgXg6VC7K3e+BNHlfNQ/2/ahS0+V4J6JYFsylZL0r2tydhDMd6rR8QA9c9BgeBz7+KWobGtiZVe7KReTzJ0O38PxNd8br3aS50j5BiN63gch8eFxPAROaUMnuiavu6OTDy4/aeL3Zc5WpUy3VyyumWf jolmomar@localhost.localdomain", "key_options": null, "keyfile": "/home/jolmomar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jolmomar", "validate_certs": true} 2021-07-31T01:19:58.170 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKjxTgyhL1yFriyNvvuhUc6SE8+6f6lT6FTF8cZL9uiz matan.brz@gmail.com", "key_options": null, "keyfile": "/home/matan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "matan", "validate_certs": true} 2021-07-31T01:19:58.384 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQKiUAlB8dxudAY60M7jnIJvs9yb2pNaT+BVv1Vk1kMs0XjPjcw9cyT3DtQ1xGLF367phRi/cfQbXOYbHG+uyBzQ4EEnBFvziktWRx4mfeWYxKjkjbfkVfpKPsxztqCbwm6+NYzMxGK2cExN7jUHnghHAjfA0IUpVVg9peBD3kUveLwS+UxuJAnuj+G4480LS6GDyLrhFEXU7uOIIlW4S9oLnU6vJL6GAzwyLE4ijTA8Ek03SJ01Ey/gpU9AqnQNp9u12l/7sJely6sI9pAuIzmS8Ht324hexxSWN5sFjXgLvLfFK83ziDBdrFBr0WXnWsFdRpHs2CwSq1JTkByqbN hmunjulu@hmunjulu.remote.csb", "key_options": null, "keyfile": "/home/hmunjulu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hmunjulu", "validate_certs": true} 2021-07-31T01:19:58.406 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkFTEUxQkQ1cplOLOf86Teu4qTYGkMqWtjIwzsU/AcimjFolSI5fJ3B72PH7bhQKkSnOlTjiOos2LrD5i/XgxkJD6icVWwHO11lczsq/8K5x0ZMJbJaiT3QphBB4yItoge2Y4/7g4o/78Ne+U7eZ1L5fgeO1fExtIaEXK1lS8pVlpZQXyproTIzp1WLHoaXb38Mxc5jAtuvEKfEycdX3c8k/9ti2qBCZnmjPH05WOxkqCRyNBT+bSJZyFMYt9RDUuycn3BYVSXTAzPW0qwgvKY5vij+qiFoxij7nKNMH7w6C0arolCcJxhiMyEF3UbTuOxgVONWyqFk/vScGGSYj9ytQYjv0L1cFyOVhlpbu+tnptulfQ3faMpjROE/eYxTIoY7OPDgWbFqEUJ4eEhKwAC7aYIYUhvoI/8k+4u64nsyEJQdHXmB+OY3WE5TegYUCuqSsdPY74eO1o8dDgxF363eggDSBmYqSSs+Tpu9TNhJG7iJFX1tiAPwmKvjVrJOac= ozeneva@omri", "key_options": null, "keyfile": "/home/ozeneva/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ozeneva", "validate_certs": true} 2021-07-31T01:19:58.617 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoefxOfeArToGucfWG7y1tyLF2I1J/IkBSlmxxEJyHyDUB6ElXjYIJReoiPfmB0xLEmINpmt8QA4XN/8UbR1r9LFOexlZZbX1LSbihi11LXCvfIJWqkHVwIyKHRTu/duMrntBFVENJ0LVf0GSWVTtllXaDA+fMkp6/ra/hlRHQbx19UXy/9+Pogj8Xl+SntnMGiHlX8FFq2A2tUqhdQSf/1GW1HfUIa5WtwdEqIpKpAgMM7sOW9i3QAmLAMo8HiKrArSiy7jrRUODCqsTJsfifxEbmnOS0ZtmDh0eeWL/Dyqbz99Rg4WbFNeK8oyZYr7D6bEAhd2az2A+wycPx4Nq7 julpark@julpark.remote.csb", "key_options": null, "keyfile": "/home/julpark/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "julpark", "validate_certs": true} 2021-07-31T01:19:58.639 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDepyzA+MeQT5oRM52SxmUUKVAzkcrH/NqFxoCpBAjUSioTMaB1h0PhHE00nvHnKRvqWdPl8LCBWS3Ncjh30r9lwHaP+oPy4iLQEc7D2Mwn7WyPh5cq9N3UF25C0AlMG744k1tBNb97qy73OJI+oc5pAj4vUpCC1Ut9pMP5E5pQCx/kwgSOpC7mk0dYr/H5FxjPb0ca55eisFTLXraHPSChdrHjlYC5XpOZ9csilTwFdF0fJdu0nMDwIxGvwTOaUcIcfNLkpGB2Sj/pAM4L9pVsIbeE3C8d978MZ8Ff+ksQ0T8eNzew2ZjSkSuV+oR1sBCJYZ7mbDQuYLKAs5DanO1VFJvgjBJm1zP2ONu+g0fn/j04qIosuudziwoG/dOTkFo+6dBvF5hQi7TbZOIUVFr5dBlGS+O4eoK0gi/wpzun227E4dadY9uB1wiZ/PTD/5og+gxWG32RkHqqX3ALlK4m8JKzrq214nZ7Dt9CNewiUkR15uRkDasGIdiu6qb4Ez8= ArunaSudhan@sudhan-linux", "key_options": null, "keyfile": "/home/sudhanaruna/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sudhanaruna", "validate_certs": true} 2021-07-31T01:19:58.861 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD03otezqmbR7v9dq62mWH5Kkg692jdmDZk6Ve4vux0A0YhxBuXDCyBWLE1Z4lLNxQahvK0pGsgokp+Uk+Q6aU+ZnF/E2jLp6YWerFXIHY3M7byObJx9+QuVBfNI0bZBkV3+vSjhSCGOYU+TrTHsKZqhsTcejZjmL3gFLmWfU6dTh0sX05qz97cdJl3CMPhCIR3KjtkiPygj417/N7wet/Efv1t79TvyS7dtK1oz+zRLEpKi/fZjvpXb8xec7as72Y0XexsWMDyZJ/N8PijTQ5L6EfLMV7ae/bWGzNAKsoC1dHVY7q/Q4pzktC6zU0Rc9sDRMtrgFg+q+04BfveSLfxTb9tqf7SbSvh7IAG1eKiEuDT/uy3T6Uiw+CiZ0eIqYzZscO/aDjUArGbS8eyaycybdYCBK9Jz38i0mYKkXg855yoVuIExr6MXieYwyVoVW8gbx1bAslaafrzra2dkAFByOJ0sbosPFRgR2+boUSxD6IaE+ALgno4T/0CI2RMiH0= prashant@cephdev", "key_options": null, "keyfile": "/home/pdhange/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdhange", "validate_certs": true} 2021-07-31T01:19:58.874 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDuHIpWAz6Sk3ATLvTG4KS24JUzgqQBEm4EesPgf1cEG+xK4gISUixaVC9OSNS/KR+8/yoWAhmw992p8cp7mWXxqrRC9kIrthLWvD6Jjz3HTXgarO2oNibeVu27kK/Fm2vvUNACARBkoC5YXd6IuoZOEg1bJu06DLw5Y7pGMqZRVDOrE8WUDwr27roZSgU2/N+18Y1LYn4gwSvWpuC/IotLyGSpW7/dgrZAIreRoYboMdTSlIjjuzeWvYx/AnZmkeEo0IhfyuBRl7ZQwxSFYHn9Z1+iQkt+O9hEZYSZuYRuyFPRqHNMKCEj1iYVRMcwbf/x2/dhjfmbWmjOMJoNNwm+DpixHHbId6Idx82GlAtCgQSTe9aG783oCeZYuVUr1plIRjeEsdjY9UBRYKNknEAChKjMSbxysvO3Dy4yrU2r8qo1iBS1fi4HUg12asCLrJX7vSCvOhOn7ltqDTmt5UiLSn+eCzXpOTBDWz7McB28FoYyR+Rwz5GOH3ITFhPyhXc= wzl@manjaro-wzl", "key_options": null, "keyfile": "/home/zl31wang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zl31wang", "validate_certs": true} 2021-07-31T01:19:59.113 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIi8JMOVD91QR7EXBkhxeAsB2UO2gyWqYPiP6HjJO/TzzMKf7QtO928XzdORwQ1/jm2hDF1kaNPaU/XIHD0hHSvfICS4ELQcBPa5uPya6d5Kn5fVlpGqBcXBxKGbHB01PGjn0214eupt2LebWxkm7TP/BGDHKXXaoR748AlbyyG2L3TaNoGkwQiEoZe3CWgH4qkrf2CEpyfOa7BM7U0h+lceJxIGe+Yo6Ll14zyngDBa059E+ndFTpkQ6vECII7FdB1+A4Orhu40XHZgyVnlRudD1l80pIVnZves8N2zHZhQo8hp0sB5oY8HENiB09j7Xa6YQ/c1c2wm9E2xJHSNo7muFiHz48vQEgvNmu0lZdbVwXIy3IgLEBHjs6Kyqjtg618V4UaoP/LBOqtWtLd6TvGJ3qBlNyvhO5gZ2DGeLMN2WWoHxfl3VM+QGCGI2dnzUyHhfG1VBDYj8fzWXOuUVpVCMUw8fEg8GWmYLDwm3SbdMACWU87J0kAmaDPKcwwL6AwtWJZr/p89sSypl6y50JlOoDrbDbxrIbHQgbf1tcEgPOuLUN+O06qpi+OxKMtFdbvz+NQ4V4CpXpvMMQZj5AbuHp0EJ5Vme/Kcmpl/xoYGyFDzOEGR8lw6LMz+CpQ4GtnK5XXe1Xn3TJh6Tn2tVyS20p7artemGQQquCN8iQOw== mayag@gett.com", "key_options": null, "keyfile": "/home/mayagilad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mayagilad", "validate_certs": true} changed: [smithi045.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDg7xW5+Mk7pG5MjLxOc7GTvDTCPBJj2L3MEQZIapmmo6t8uPODwDewXncSzrKDH6zFVE4TTTD9fJdu+BhyT+jCFgXrEOMMabj7vJsg0wO0SWqr+4NECFPh 2021-07-31T01:19:59.126 INFO:teuthology.task.ansible.out:D9y4ewfYiewBeXisclA5Bto91v93AtYpZBCdmPYSi0BEcPVc6S3iBUD0PzIrL62I/kBAtm0f/bScRiT7XvQJXvkbEObW2yS2XhUv2qgq1VlFvO1jl3B85KMTHVb8+LrKpQWOKfRfxJ74c51y6tnEjZLFO6Xn0JevqOhd8gPQIXueFZOM0g0pNp1mbtGaaBlEIibbrzo2p9kzsdvJeHMq1XuMIi1/Qyj9 dpivonka@dhcp-41-144.bos.redhat.com", "key_options": null, "keyfile": "/home/dpivonka/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dpivonka", "validate_certs": true} 2021-07-31T01:19:59.229 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:19:59 +0000 (0:00:32.686) 0:01:43.209 ********* 2021-07-31T01:19:59.366 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0m+8wna4RnrOcuCUj971EBe4cFiTXVkEWkIsUX3yeCUwMaS/ZjaoOjcdYX7cYpCp1ilo4FSO5BsochU0RiMbQEwLGcE0zB2q1ac8+yYKCuWWyknig2t4bOReu5OrXvl6mj/9R+p7clFlYhwIoNG5ZZ/2eTbo3nH7gWgaOKaPbTtvEGg/5R/qjQKg8OTC8oix6kLsaNTisQJ78Zd7b4ukg9bO0jlq/2UhQOBM072sU1/kJpWTkcA0xxm3nsC4rWPwapgz7UwBmMdyx116GZgWdfQecTLKO7b8pyJw21npk2ntUmeMi7WPZGX6VI3IpaAhBjW3KiATAb8JbAQh8F9bmejkw6x7zrP3hYKQOg3A+U8DKRRo/Ad+f0zz9gjC42DUvMEOr09RBdxZcEAPY7H/YRWEqbA2zPX3Gfth4b7/letzM3m9ToUvVfSt1CdjaZD+gIia9pCgDMb+j1j2w9reG4F7Us+W/1f5pYCyFyrm4XO3AdoYM9/FOAOR3vffTIQU= adking@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJOefD/VJfS15LRyuVDLluKv93Zgocsl8BhiDMimvBNbrGWc795v2+eevWuwaL/SlL44SnsNZ9Mz8FVn9YuFoQiHLg7cHZNLg9EJh5SMXzZzwLZHnF02MdI2CU6U++9nmh8FtBrdBWKzZ7o+01wOw+52dYOxvP7IJ7C/vjwOLgfigEN8Nd1PrxALVNIQz7fsiet0n3odk9PJfoqTu3IloXm9qbQiPFyfMA6b8a20RdLZCJDGGG5rd6gtsyeFHoAPZEz8FEUlNkow7NM7ZhdxOz/fWXyMZ27KmY8WSAgflbxysMzexyifkGzypiE3NQ10hdPGBgoYsHrAN4Qmte1/Z4aI5nxrNzLy/uNc+L/Bp/0XkLj9zfoiFFOReXvlz/laAV6LpbHyacr+iCwv9rlI7echXmvC6mSJXfBPuyR1uromUXpYRhEkn5uiOsCIQCuYpx2jeil7JV+LJlF62lLSndOv4g12Gq2TEZ4Bhsr2902kftPkLFenyfDTh3A2qGGNM= adking@localhost.localdomain", "key_options": null, "keyfile": "/home/adking/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adking", "validate_certs": true} 2021-07-31T01:19:59.462 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:19:59.482 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:19:59.490 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:19:59.496 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:19:59.498 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:19:59.509 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:19:59.511 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:19:59.521 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:19:59.607 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** changed: [smithi045.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCs6q2b5NcUsNXl+w1Mrc7idNvGdwWkL0XRX5zTw22UNpQyqPkSrRTFKi4NxoY8ucv8JoLnaJL7hYSRHKTh7SqH78uPLZX0FLtoHvpYLgUgJzzJ5AOW5uekIhCMCm+3egmFnEBwFcNxtGnU2es02ByXNEXRxt4/LQ0f8K6lMqxVRSKYcltvWzlA7WFFdTnYMunbCYOdpw88RXm2pYjGIHRBnie7+pFiFSS6K/mkzQcfLiJmahgLSaYslI0ho1vQd8xfNLmZ700C5F4VTUlT/mJK/dI2LF3OfaPTsyqKNzOchrjLFx5b6dhZ4WZyqu7VKsHNm+xHZwT8VP3LPGSVVBdmEsiCsmV4g4QTI5ssZLi7siZDpaFp72nVAIO1i6kBGuiF3or+J4LmIh8ZRKzTc0DRDOw7S+IkWu1fv2UhYg83z6wrz4HeczI+bnolFFWW6HgGV3MFaUeRTRvrbxw2DPoEGQ+cjP0wsjyEhxlScwdh8W8IbVpNl+7mBe45YGUH2hg83Lst5QQxD4PHelriu40moWRHFKCnKfNesb4L1uqu29o8f+/PgpLdrWE/P5zIKe2ReWJyScYAI7jTMlFw+xgf+0TKryk44NMy90l3ei0umlx4qsFbbEv+lhzAALrUJ/da7fsQw7eIVykmcscBeYr99/YSYk4MGdk4jVCiHsQkdw== amathuri@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJ3ch+pU88/tX5TVVFl/1pPief8drYLRkK4BtfJk+xVNxX9qkDtOtiLAU+OugYF1/xk9hE9zTbm3UBL23unrzhlR80t8f2FiF1YFhHVzeZ/p/OunmkU0RFQzkMVuwKVOUNulw9FwpEZ/0Bjfxvaims0SBWaCzBgE5g0R+Ofdq/bVF82K8vU5Laa3KZ/cCdGzFgW00cBvPAKIpQ93g8TRqsCsFvR37nY9v5SU1hJ9araEp6LRA86tUt3EFtxD2tRcmqKGtCb0kOr+rRXWVGiaUBiedHxFudIy0+TNj4fNrJwpoC0hcBO9ek4Bh6hM7Zqzum9HpkPGlz84jkm+r+xMThaZCh8iFkgxyK+bw8B8smaVJD04ob9Wb4RYoFYfQ7TEkPkO9gr+UsSesbP9fWfucemUqwK0htOOtpc1YRwdOPcdPJWAfx9J5+YVL8BTWYBaTdmHsl 2021-07-31T01:19:59.609 INFO:teuthology.task.ansible.out:OKeUGM5fCob6QxC6Er9xweys9jh7MXttSpe013bY7P4ZDRF8uWvygpsytPE= amathuri@redhat.com", "key_options": null, "keyfile": "/home/amathuri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amathuri", "validate_certs": true} 2021-07-31T01:19:59.770 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-07-31T01:19:59.772 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2021-07-31T01:19:59.782 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:19:59.784 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:19:59.845 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** changed: [smithi045.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDr6H7hmtGCUrO3GppgMHwbmA4XT/Vipz/uLG/N8h6ZxtaLQKpCefRpdKcFlOdDwxkTmT5q9vmXQ8nl95lTGq+eeMxl4ho104ZD9e9AIYmgYqFy7/qNP8gkqduIPlypLL9m5rElTcGd7zqfMqgljMvAmnXLtZNJlIuFtszVU5WPI1QCwfQ3t5aHF6j66bpVoN9uWItVGB1LApQz9TtViWsQ9Hl8X4YauJToO+TkuSX+H/OGH7ZC6CPOzNZb3b3vF/jLMnTzXl86PJs1se4DjKkRGLVCVbr2WhLwQeZ8TEn3Hg1Gn5X8zNM5cTo6U8SnXBAe9ZYbuyfvghrNLdlZpmN+8l0/l5zC3z+N+sEzklXwHw4MQLF+KSloVFd9NN/kyenufTZAyp8VxllZ9TgdC+CPORIWyzmu1elOtYo7LNuE6CIh/Zm9B54SO6hVHhVoOCqe/dIyjeHJA0WKIZblK1XZEvWuu0E2T3jhxfe1GICyTukeB8WoBQH9DZ1yzMPt59E= nia@redhat.com", "key_options": null, "keyfile": "/home/nizamial09/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nizamial09", "validate_certs": true} 2021-07-31T01:20:00.014 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2021-07-31T01:20:00.094 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** changed: [smithi045.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC//ZCaEeVN/J09FtsvuZ4eEs4rTBtYoQyG+ZpuWkJLL90VyJeYrZzRsgvveKW2W9oFxCFcMy7IBhLes0TZ/X/4Ox9eUmP/eyqK8fBqT3K1F1W/uOmKLFhlQhkxxj+p9BOMBD5trfES/QJQ6UFSYIS2WhWaitDgONjoLMR6wXZ9r7yZeQBxbmG4wGI0Cwx4Wgg+I59UbE/5iUfnX++gN3rXrPnv9V0yFeciS4HvXRFXdrIzvQFHcZMmpOOmGruHXN/ME2dZCG+kaZhZdI6kICSlOwpU7TU1lv99zjBCeEzGfFkjv041ykKdNVR3qE0k5u0AOMqxJBORL/woSCdGvrBXqju2G/2ohl3B4zD6TAve+JLfWOV5C20NFRPPbTd53JaNpV9OQqwOfVniowzSYYBdedjm9sq6YwhJt8iBXze3OA8ZO0GPXo7PC2MyrrUtu6uu3DZqU4x0cl8iJxqOlx5pH0Gydj8nuUG0+X6wHZzPXZRe5M7/RzmvXch9koG6FQMiikNjUkK8Guz4NMBdAYUpxYkdPGBhiuELoaAcgIVzSGoZTuICpNYwAxBHk6q18G8Qd8vaeUdGdim4YvwycsSOwFE47wg/XmpCHrNJUZzzB9UD7Kn84TG82LAmCuAzpzcp0uPjsBckbXzj9Zt0hRgF80yDgUazsQkbgtd2p8KxIQ== loic", "key_options": null, "keyfile": "/home/dachary/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dachary", "validate_certs": true} 2021-07-31T01:20:00.263 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** ok: [smithi154.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2021-07-31T01:20:00.270 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:00.281 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:00.291 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:00.292 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:00.303 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:00.314 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:00.316 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:00.326 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:00.327 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:00.338 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} TASK [users : Update authorized_keys using the keys repo] ********************** changed: [smithi045.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDwVh9KqjV0b+65aBxQV66IUwNo3D5zTbN8U6Kh6eE4QtaIVe5Mmvi2SBbGMSWofE0zIIw2JmsI5LzdOoCyZl/3gNDPHQd6D9o0xRHQv94giZ2a5K+mW79foAxsuppTjV/Fbg8EZ/19W91+BqMWotlMR66ka0RPT/6U8V5mzXoydSminViO/B6q7ZYzIIgGZqaTsSxCMvz5BAzzeEaGfSlGxw5MOye9sF/tDppuRKG4rMUZULEWGuxgXAtabtXCduocAArqAY3BRJEgj7JetPdPNjipGrULyYQmT7GeknvP/waC0sPm5o8t7taJca7Vu9qKKMKPAqiK4KhlrW/yVBFliT2CjAdVuYvGj87LCcEn6NMeF+4ONwDz++YwjrAWQT2zHUJPvLnYOZPiD1LUm3F34cEHD1y4Uf+oeiYgriUMauskfFn+ShbTmWWooFRFyNobzweg9aMBZSkA+kFFifqOV1FwHTOYg342NumRGB8svK0vvTUID7AU1Ev8YXYqCEGbb0UHth/1pEbAF103enLTtPZ6qKua74rE0qJ89OF9dZXv4JY9+8Juyps7bi2 2021-07-31T01:20:00.342 INFO:teuthology.task.ansible.out:XEyHNm1nHZOY5tqdY0eKFmSGJMoWBDzGgdMjmXlWp4jnY2fC619nZqxMrCQpeT70CqnBhtBemUuu6jHYUbbrPjnXIkH3FZ7/qvZt0jF0IMP9FGw== amctagga@redhat.com", "key_options": null, "keyfile": "/home/amctagga/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amctagga", "validate_certs": true} 2021-07-31T01:20:00.349 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-07-31T01:20:00.351 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:00.360 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:00.362 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:00.373 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:00.385 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:00.396 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:00.408 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:00.411 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:00.420 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:00.432 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:00.434 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:00.445 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:00.460 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:00.469 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:00.479 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:00.483 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:00.491 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:00.493 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:00.503 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:00.514 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:00.524 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:00.544 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:00.546 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:00.555 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:00.566 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:00.568 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:00.578 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-07-31T01:20:00.579 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDtiHTDuQk8EDH5R+JfDMucI4ZpOzKN5SLD+j35QLLySgGXholoDxtqjHnmRjM1rpj5SIaXjHBZdJi7iax+1MRywHYBi5BfTtzYPgA+NH8mt+XkcdkEuFPcYxGh6fMQDQWb2g9Euze/6qi0W9mbYXvvb1/Q3bhVCXg1UgKD42BndWEpbMYFfzJLZWPtOD/lg81pwUmf/4BpPFv52sy87lQVfkkvcUliLObR4S0TI5hPLYxBS+RXmHYKgvPMB2QJ9wum62lJ+YM1MxPygS0OWbIB6UQGXmvxRXA4osXkxJ8ksPPCupOgFNiiYro91cQh3mmi6iHc9kMz5R1aNGGOfDdlnLx9DRSIiNcacjXEVNFaoM0x441f4GeU2ZR8/7uq2dfk7QjPlh1eJGxG9pzRiQ8JCDF1car0eYwXdMTsQoQqpIi5imqcELdcqGGp9FhzFElgUwpf15JYlkouVhS6hq4LDjhLNk7dDT8fmCEnHOPcbxaTgN7J3idnJK6zeIniuuk= root@iZj6ce071s2zz3reioxn93Z", "key_options": null, "keyfile": "/home/kevinz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kevinz", "validate_certs": true} 2021-07-31T01:20:00.592 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:00.602 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:00.614 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:00.625 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:00.627 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:00.637 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:00.649 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:00.660 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:00.661 INFO:teuthology.task.ansible.out: skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:00.672 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:00.687 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:00.695 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:00.697 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:00.706 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:00.707 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:00.717 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:00.719 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:00.729 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:00.730 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:00.740 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:00.752 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:00.867 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:00.871 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", " 2021-07-31T01:20:00.872 INFO:teuthology.task.ansible.out:ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c365945 2021-07-31T01:20:00.874 INFO:teuthology.task.ansible.out:85e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:00.875 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:00.875 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-07-31T01:20:00.875 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunm 2021-07-31T01:20:00.877 INFO:teuthology.task.ansible.out:ei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1YAaxAvlRgu72huAggHpoDKSgtF/b00ACfUX4Wk5NyLFbYI3PAQapHNcNL4mIsH8pc6YkdjcJ5N/F12aEupJGQ0Q06OS324xapnaBoUy1hQl9IvhtPqTLEwR0X5xLIAhNBLIEJDNVg13MKwr0u5YVUXdrQRu7Kj9RXAPBahNZFo/Ed+cnR3LICQWliHeHFrk/DU2DUHol3a5wAPxrowAG0AxAecP6/7rASMmitvLQwdQcfIJAhWYgfyXzudg+gOmAEgv3O3U4QOikgmkm/EabUXLJnlTj6yuhtuuctdJr73UT3wLGAGmphLKCkug27kb1nTNoDa+PW/+grP63zhU1 chunmei@otccldstore05", "key_options": null, "keyfile": "/home/liu-chunmei/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "liu-chunmei", "validate_certs": true} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:00.888 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:00.891 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:00.901 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:00.903 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:00.913 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:00.915 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:00.925 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:00.927 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:00.936 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:00.939 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:00.949 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:00.960 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:00.971 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:00.974 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:00.984 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:00.997 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:00.997 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:01.009 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:01.011 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:01.022 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:01.032 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:01.033 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:01.044 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:01.055 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:01.057 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:01.067 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:01.078 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:01.080 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:01.092 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:01.094 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** changed: [smithi045.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDLbPdDb+XHOD2fo8g3RacATj0cG07uELwUvqQP35y+JmhNRSb/ttaysFLA5ss0y77TiXx22saP3ECEm5SnuPg5IERkPQmXiw84ATTNfLnzx2G5H223SLaTQiGfSgniBCJHVPCJfNahJiurlZoOAlwDz6SKXTUiivVrFJ22iTBDCQUO6HFQKjE7aM3RAo5mZXKVlTXzM6WOd8vlzk3HMzV0/91JR6H86Lt9MEzXT4dOzqmtxd8gS6U+vA+50zOIspdK+orTRJ0PO2Fn8c1RSYM8g7uqQtpDcOEHderW8TRsqBbDxpyslK6hRX5urUNBX506XE6JNoViLNvCofgseCGj yingxin@yingxin-server", "key_options": null, "keyfile": "/home/cyx1231st/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cyx1231st", "validate_certs": true} 2021-07-31T01:20:01.104 INFO:teuthology.task.ansible.out: 2021-07-31T01:20:01.106 INFO:teuthology.task.ansible.out:TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:01.108 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:01.120 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:01.134 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:01.142 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:01.154 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:01.165 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:01.176 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:01.187 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:01.189 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:01.198 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:01.200 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:01.211 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:01.256 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:01 +0000 (0:00:02.021) 0:01:45.231 ********* 2021-07-31T01:20:01.321 INFO:teuthology.task.ansible.out: TASK [Filter the revoked_users list] ******************************************* 2021-07-31T01:20:01.323 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:01.351 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:01 +0000 (0:00:00.099) 0:01:45.331 ********* 2021-07-31T01:20:01.357 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-07-31T01:20:01.358 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdqw+8an6CHqsagFqXfuzQi1zCFHCh0e9pDtYz+ylA7L+7vnJLjTR7km0bb4Z6VTUh7h0/nUfF6faPzD2KGmTKvS+mTpeOUaN/H8O8j8qqh90f8z4T8F8L3DwqQbPf/R0izXmTgz0FZObFHmj+G8a+D4ndxF80EPzKkML/G0Hg2GEVdYzazSY4imSY2f56ul6qByBknhFshaTo2DUDcdbDDokF6lO6WgXhY0MJrT24CUSMyZmyi+3hF7vowW2dfbJ+SuDft4XmPDnIxRhDWWvi12jZb8Le7SWYzUlu7fAGbWe63VJqNh992myI61aZshApDD24+1xqIoo+Z/XkQD3l sunnagar@localhost.localdomain", "key_options": null, "keyfile": "/home/sunnagar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunnagar", "validate_certs": true} 2021-07-31T01:20:01.585 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDTlsh9TFwLCeeD2RR+56VhRZ0hp9CHj72U3Z1gbMpxpcg6PYEDzb+0qPVQp1/cIhcwwy0sm13yFl/A5i0iCRpyqB2MbyaXjj0HgkNinxLBFkPSiW2C83lfAyAhG/gLk+FcmWE4+xVrE29aoSM0JtcNtp+q83UGdnFcvhFe0pgJbt5vITo3ZMJND/UZrzeewyK8Vy4Rvpfy8aHYi2Eq+7WQrRHwUwxn0CwsxldnUfmZtu58d+Rtt78LRF0mvcScNlC7q5D+8htmYwkWkK639w5IB3xPv+Z5cV49Psi24Q+9BIr9Zo9/OnnCTauAi5ivDlh+kk/iBksAE8rQpMhvfuSFnNoq2O87aPQvNQHdRCcOa7FVIDcuocu6AB8UP4cGcJreT4+BW5Zbk6vQeJZhmj9yExpRj9MuONy5793nKuAMPn9lOjuXRfK7A5Tb3OOhpEJK1/MjtdZz4mNmFFfFIZ24hgTZO0pOiMwDf/tQ6BG7H8NS9g3QmXPyZajUocw6Szs= lflores@redhat.com", "key_options": null, "keyfile": "/home/lflores/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lflores", "validate_certs": true} 2021-07-31T01:20:01.637 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-07-31T01:20:01.640 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2021-07-31T01:20:01.817 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-07-31T01:20:01.819 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGq2g8OFN5msjIn5CNB2BRss9eIyJRk3r+7RQcnBOQid jsawaya@redhat.com", "key_options": null, "keyfile": "/home/jsawaya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsawaya", "validate_certs": true} 2021-07-31T01:20:01.832 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** ok: [smithi154.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2021-07-31T01:20:02.022 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2021-07-31T01:20:02.076 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-07-31T01:20:02.078 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDNIdsy+yA0QjwkvjzTiH4lKB+txlh4oGkgC0r1l9swmFvD+MuzNNkD+oB/YscT51tJXsP5/wZDZMuk60xds4Os2ejoIBvzdGBgZ1G2B33etHPmzgTFE7ZegBkMlvWYmrJ9YS+iaMvXn0MMtqfE4zq0kemVAA4NBfFZMhHPJMKCg7Gj31MR0S0b7dcPx7Jnpg4A45oHr0qnufeqZYQaHGgo+LO2xdoV4V2EswWr7YG+b1wI6DzFZBbNsGyQd+cE1BzQ8NmETPNNpDPk5dCyBS7U+4ZTf/WbWk3+Cn+Ky47iWfxcscupQRkCjmTYI/uCMhnrpGU0zEHkbFwLTS6wqDtDmVHm++xof3xSsGJ2cE/vZySZGHqdxqVXfT8gt17h35rME8eX3TU392rjfK6kjEUD3UgwJJp4NFzQIEXi2HTmDYx/Vi9nuId3V1NrDz98QLPUd1ijiy1DuzI0BJPE/DTDmsb7hNXNIcAV8VRc1243W3eaee/Pdg/wmwfkrgjTU3s= shuaiwang@Shuais-MacBook-Pro.local", "key_options": null, "keyfile": "/home/markw8973240/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "markw8973240", "validate_certs": true} 2021-07-31T01:20:02.217 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** ok: [smithi154.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2021-07-31T01:20:02.312 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-07-31T01:20:02.314 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+ZvZuogU+WfyGj4BR2xYllNoQqAoxQ6AM0IYddBV2pL1xgTM/SopuAxDRP0q2stpozMtjVjlvuiIgr/caqyTuOgYtYCvB09ppefqdnCDrJJBCOKpjUEnCpAKcBMMuz9V32Zqt5FgpIi0Mu7Gp/QCvB97imxEkoUunpdX/8Ri/xz0nJO+QqoJX5wOn91jG49XnkMCLzx+MuafhT90WFO9CV1sIVREZS+X2C7ACQuhx9Qwk7R4BXt2f3M4dA0K2nLwaAnVW50M1y6Gs50Wl+ojDR1itySOxv8YpwyRlRTOzQTyVMnG0Qsaxxwp+hy4b/vQW3NIUvHYdTyvqALTQdGHT lricomor@lricomor-mac", "key_options": null, "keyfile": "/home/luisrico/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "luisrico", "validate_certs": true} 2021-07-31T01:20:02.407 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-07-31T01:20:02.409 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2021-07-31T01:20:02.545 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-07-31T01:20:02.546 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKjxTgyhL1yFriyNvvuhUc6SE8+6f6lT6FTF8cZL9uiz matan.brz@gmail.com", "key_options": null, "keyfile": "/home/matan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "matan", "validate_certs": true} 2021-07-31T01:20:02.605 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** ok: [smithi154.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2021-07-31T01:20:02.792 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2021-07-31T01:20:02.803 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-07-31T01:20:02.805 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkFTEUxQkQ1cplOLOf86Teu4qTYGkMqWtjIwzsU/AcimjFolSI5fJ3B72PH7bhQKkSnOlTjiOos2LrD5i/XgxkJD6icVWwHO11lczsq/8K5x0ZMJbJaiT3QphBB4yItoge2Y4/7g4o/78Ne+U7eZ1L5fgeO1fExtIaEXK1lS8pVlpZQXyproTIzp1WLHoaXb38Mxc5jAtuvEKfEycdX3c8k/9ti2qBCZnmjPH05WOxkqCRyNBT+bSJZyFMYt9RDUuycn3BYVSXTAzPW0qwgvKY5vij+qiFoxij7nKNMH7w6C0arolCcJxhiMyEF3UbTuOxgVONWyqFk/vScGGSYj9ytQYjv0L1cFyOVhlpbu+tnptulfQ3faMpjROE/eYxTIoY7OPDgWbFqEUJ4eEhKwAC7aYIYUhvoI/8k+4u64nsyEJQdHXmB+OY3WE5TegYUCuqSsdPY74eO1o8dDgxF363eggDSBmYqSSs+Tpu9TNhJG7iJFX1tiAPwmKvjVrJOac= ozeneva@omri", "key_options": null, "keyfile": "/home/ozeneva/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ozeneva", "validate_certs": true} 2021-07-31T01:20:02.984 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** ok: [smithi154.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} 2021-07-31T01:20:03.049 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** changed: [smithi045.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDepyzA+MeQT5oRM52SxmUUKVAzkcrH/NqFxoCpBAjUSioTMaB1h0PhHE00nvHnKRvqWdPl8LCBWS3Ncjh30r9lwHaP+oPy4iLQEc7D2Mwn7WyPh5cq9N3UF25C0AlMG744k1tBNb97qy73OJI+oc5pAj4vUpCC1Ut9pMP5E5pQCx/kwgSOpC7mk0dYr/H5FxjPb0ca55eisFTLXraHPSChdrHjlYC5XpOZ9csilTwFdF0fJdu0nMDwIxGvwTOaUcIcfNLkpGB2Sj/pAM4L9pVsIbeE3C8d978MZ8Ff+ksQ0T8eNzew2ZjSkSuV+oR1sBCJYZ7mbDQuYLKAs5DanO1VFJvgjBJm1zP2ONu+g0fn/j04qIosuudziwoG/dOTkFo+6dBvF5hQi7TbZOIUVFr5dBlGS+O4eoK0gi/wpzun227E4dadY9uB1wiZ/PTD/5og+gxWG32RkHqqX3ALlK4m8JKzrq214nZ7Dt9CNewiUkR15uRkDasGIdiu6qb4Ez8= ArunaSudhan@sudhan-linux", "key_options": null, "keyfile": "/home/sudhanaruna/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sudhanaruna", "validate_certs": true} 2021-07-31T01:20:03.177 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** ok: [smithi154.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "name": "alfredodeza", "state": "absent"} 2021-07-31T01:20:03.283 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-07-31T01:20:03.285 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDuHIpWAz6Sk3ATLvTG4KS24JUzgqQBEm4EesPgf1cEG+xK4gISUixaVC9OSNS/KR+8/yoWAhmw992p8cp7mWXxqrRC9kIrthLWvD6Jjz3HTXgarO2oNibeVu27kK/Fm2vvUNACARBkoC5YXd6IuoZOEg1bJu06DLw5Y7pGMqZRVDOrE8WUDwr27roZSgU2/N+18Y1LYn4gwSvWpuC/IotLyGSpW7/dgrZAIreRoYboMdTSlIjjuzeWvYx/AnZmkeEo0IhfyuBRl7ZQwxSFYHn9Z1+iQkt+O9hEZYSZuYRuyFPRqHNMKCEj1iYVRMcwbf/x2/dhjfmbWmjOMJoNNwm+DpixHHbId6Idx82GlAtCgQSTe9aG783oCeZYuVUr1plIRjeEsdjY9UBRYKNknEAChKjMSbxysvO3Dy4yrU2r8qo1iBS1fi4HUg12asCLrJX7vSCvOhOn7ltqDTmt5UiLSn+eCzXpOTBDWz7McB28FoYyR+Rwz5GOH3ITFhPyhXc= wzl@manjaro-wzl", "key_options": null, "keyfile": "/home/zl31wang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zl31wang", "validate_certs": true} 2021-07-31T01:20:03.368 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-07-31T01:20:03.370 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2021-07-31T01:20:03.540 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-07-31T01:20:03.542 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIi8JMOVD91QR7EXBkhxeAsB2UO2gyWqYPiP6HjJO/TzzMKf7QtO928XzdORwQ1/jm2hDF1kaNPaU/XIHD0hHSvfICS4ELQcBPa5uPya6d5Kn5fVlpGqBcXBxKGbHB01PGjn0214eupt2LebWxkm7TP/BGDHKXXaoR748AlbyyG2L3TaNoGkwQiEoZe3CWgH4qkrf2CEpyfOa7BM7U0h+lceJxIGe+Yo6Ll14zyngDBa059E+ndFTpkQ6vECII7FdB1+A4Orhu40XHZgyVnlRudD1l80pIVnZves8N2zHZhQo8hp0sB5oY8HENiB09j7Xa6YQ/c1c2wm9E2xJHSNo7muFiHz48vQEgvNmu0lZdbVwXIy3IgLEBHjs6Kyqjtg618V4UaoP/LBOqtWtLd6TvGJ3qBlNyvhO5gZ2DGeLMN2WWoHxfl3VM+QGCGI2dnzUyHhfG1VBDYj8fzWXOuUVpVCMUw8fEg8GWmYLDwm3SbdMACWU87J0kAmaDPKcwwL6AwtWJZr/p89sSypl6y50JlOoDrbDbxrIbHQgbf1tcEgPOuLUN+O06qpi+OxKMtFdbvz+NQ4V4CpXpvMMQZj5AbuHp0EJ5Vme/Kcmpl/xoYGyFDzOEGR8lw6LMz+CpQ4GtnK5XXe1Xn3TJh6Tn2tVyS20p7artemGQQquCN8iQOw== mayag@gett.com", "key_options": null, "keyfile": "/home/mayagilad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mayagilad", "validate_certs": true} 2021-07-31T01:20:03.616 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** ok: [smithi154.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2021-07-31T01:20:03.639 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:03 +0000 (0:00:02.287) 0:01:47.619 ********* 2021-07-31T01:20:03.734 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-07-31T01:20:03.736 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:03.748 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:03.761 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} TASK [Remove revoked users] **************************************************** ok: [smithi154.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 4 2021-07-31T01:20:03.763 INFO:teuthology.task.ansible.out:50919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:03.772 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:03.784 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:03.794 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:03.952 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** ok: [smithi154.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2021-07-31T01:20:04.048 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2021-07-31T01:20:04.059 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:04.070 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:04.144 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-07-31T01:20:04.146 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2021-07-31T01:20:04.301 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-07-31T01:20:04.303 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2021-07-31T01:20:04.335 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-07-31T01:20:04.336 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2021-07-31T01:20:04.525 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2021-07-31T01:20:04.536 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-07-31T01:20:04.540 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2021-07-31T01:20:04.558 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:04.562 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:04.571 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:04.582 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:04.584 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:04.593 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:04.604 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:04.606 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:04.616 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:04.618 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:04.628 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:04.639 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:04.641 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:04.652 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:04.665 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:04.668 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:04.675 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:04.677 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:04.686 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:04.698 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:04.699 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:04.711 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:04.721 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-07-31T01:20:04.726 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:04.735 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:04.738 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:04.749 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:04.759 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:04.770 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:04.772 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:04.784 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:04.786 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:04.796 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:04.808 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:04.819 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:04.831 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:04.841 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:04.843 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:04.853 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:04.865 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:04.876 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:04.878 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:04.887 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:04.889 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:04.900 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:04.902 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:04.911 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-07-31T01:20:04.913 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:04.923 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:04.935 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:04.946 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:04.957 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:04.959 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:04.969 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:04.970 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:04.981 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:04.992 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:04.996 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:05.004 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:05.005 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:05.016 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:05.030 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:05.031 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:05.041 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:05.043 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:05.053 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:05.064 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:05.066 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:05.076 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:05.078 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:05.088 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:05.099 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:05.101 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** ok: [smithi154.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:05.112 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:05.114 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:05.124 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:05.135 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:05.137 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:05.147 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:05.148 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:05.158 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:05.169 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:05.170 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:05.182 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:05.192 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:05.194 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:05.204 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:05.206 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:05.216 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:05.217 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:05.229 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:05.230 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:05.240 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:05.252 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:05.254 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:05.263 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:05.265 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:05.274 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:05.276 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:05.286 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:05.288 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} TASK [Remove revoked users] **************************************************** ok: [smithi154.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2021-07-31T01:20:05.298 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-07-31T01:20:05.300 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:05.310 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:05.321 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:05.323 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:05.333 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:05.344 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:05.346 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:05.355 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:05.357 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:05.367 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:05.368 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:05.379 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:05.390 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:05.401 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:05.403 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:05.413 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:05.424 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:05.426 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:05.436 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:05.447 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:05.449 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:05.458 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:05.460 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:05.469 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:05.477 INFO:teuthology.task.ansible.out: 2021-07-31T01:20:05.484 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:05.500 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-07-31T01:20:05.502 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2021-07-31T01:20:05.525 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:05 +0000 (0:00:01.886) 0:01:49.505 ********* 2021-07-31T01:20:05.599 INFO:teuthology.task.ansible.out: TASK [Filter the revoked_users list] ******************************************* skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:05.623 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:05 +0000 (0:00:00.097) 0:01:49.603 ********* 2021-07-31T01:20:05.674 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-07-31T01:20:05.676 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2021-07-31T01:20:05.879 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2021-07-31T01:20:05.922 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2021-07-31T01:20:06.061 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2021-07-31T01:20:06.125 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2021-07-31T01:20:06.251 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2021-07-31T01:20:06.325 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2021-07-31T01:20:06.446 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2021-07-31T01:20:06.520 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2021-07-31T01:20:06.636 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2021-07-31T01:20:06.721 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2021-07-31T01:20:06.834 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2021-07-31T01:20:06.929 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2021-07-31T01:20:07.036 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2021-07-31T01:20:07.120 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2021-07-31T01:20:07.230 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2021-07-31T01:20:07.327 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} 2021-07-31T01:20:07.422 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2021-07-31T01:20:07.528 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "name": "alfredodeza", "state": "absent"} 2021-07-31T01:20:07.615 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2021-07-31T01:20:07.732 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2021-07-31T01:20:07.811 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2021-07-31T01:20:07.938 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2021-07-31T01:20:08.002 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2021-07-31T01:20:08.138 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2021-07-31T01:20:08.192 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2021-07-31T01:20:08.339 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2021-07-31T01:20:08.382 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "name": "pranith", "state": "absent"} 2021-07-31T01:20:08.529 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2021-07-31T01:20:08.738 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2021-07-31T01:20:08.936 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2021-07-31T01:20:09.126 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2021-07-31T01:20:09.329 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2021-07-31T01:20:09.530 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2021-07-31T01:20:09.729 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2021-07-31T01:20:09.929 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "dorinda", "name": "dorinda", "remove": false, "state": "absent"} 2021-07-31T01:20:09.930 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2021-07-31T01:20:10.127 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2021-07-31T01:20:10.326 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2021-07-31T01:20:10.530 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2021-07-31T01:20:10.720 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2021-07-31T01:20:10.922 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2021-07-31T01:20:11.120 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2021-07-31T01:20:11.321 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2021-07-31T01:20:11.460 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item=zyan) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "zyan", "name": "zyan", "remove": false, "state": "absent"} 2021-07-31T01:20:11.513 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2021-07-31T01:20:11.713 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2021-07-31T01:20:11.916 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2021-07-31T01:20:12.114 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2021-07-31T01:20:12.314 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2021-07-31T01:20:12.506 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2021-07-31T01:20:12.705 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2021-07-31T01:20:12.903 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "name": "pranith", "state": "absent"} 2021-07-31T01:20:13.020 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item=jdillaman) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "jdillaman", "name": "jdillaman", "remove": false, "state": "absent"} 2021-07-31T01:20:13.219 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2021-07-31T01:20:13.432 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2021-07-31T01:20:13.620 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2021-07-31T01:20:13.808 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2021-07-31T01:20:13.996 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2021-07-31T01:20:14.195 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2021-07-31T01:20:14.384 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2021-07-31T01:20:14.509 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "dorinda", "name": "dorinda", "remove": false, "state": "absent"} 2021-07-31T01:20:14.573 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2021-07-31T01:20:14.754 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "name": "jbainbri", "state": "absent"} 2021-07-31T01:20:14.953 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "name": "kdhananj", "state": "absent"} 2021-07-31T01:20:14.996 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:14 +0000 (0:00:09.373) 0:01:58.976 ********* 2021-07-31T01:20:15.590 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2021-07-31T01:20:15.592 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => {"changed": true, "checksum": "41f0b9973307e6d58781b948bc481539828e2674", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "2504cfe296043f68ff0fc227b511d52e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1627694415.1021852-4578-130331649839341/source", "state": "file", "uid": 0} 2021-07-31T01:20:15.613 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:15 +0000 (0:00:00.616) 0:01:59.593 ********* 2021-07-31T01:20:15.694 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** ok: [smithi154.front.sepia.ceph.com] => { "msg": "Host smithi154.front.sepia.ceph.com is running RedHat 8.3 (Ootpa)" } 2021-07-31T01:20:15.718 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:15 +0000 (0:00:00.103) 0:01:59.697 ********* 2021-07-31T01:20:15.798 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** ok: [smithi154.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/redhat_8.yml) => {"ansible_facts": {"nrpe_selinux_packages": ["python3-libsemanage", "python3-policycoreutils"], "rhsm_repos": ["rhel-8-for-x86_64-baseos-rpms", "rhel-8-for-x86_64-appstream-rpms"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/redhat_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/redhat_8.yml"} 2021-07-31T01:20:15.820 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:15 +0000 (0:00:00.103) 0:01:59.800 ********* 2021-07-31T01:20:15.899 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:15.921 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:15 +0000 (0:00:00.100) 0:01:59.901 ********* 2021-07-31T01:20:16.135 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-07-31T01:20:16.137 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item=zyan) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "zyan", "name": "zyan", "remove": false, "state": "absent"} 2021-07-31T01:20:16.523 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** ok: [smithi154.front.sepia.ceph.com] => {"changed": false, "cmd": "timedatectl | grep -E \"Time ?zone\" | sed -e \"s/.*: \\(.*\\) (.*/\\1/\"", "delta": "0:00:00.362081", "end": "2021-07-31 01:20:16.498159", "rc": 0, "start": "2021-07-31 01:20:16.136078", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2021-07-31T01:20:16.546 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:16 +0000 (0:00:00.625) 0:02:00.526 ********* 2021-07-31T01:20:16.628 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2021-07-31T01:20:16.629 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:16.651 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:16 +0000 (0:00:00.104) 0:02:00.631 ********* 2021-07-31T01:20:16.742 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2021-07-31T01:20:16.744 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:16.765 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:16 +0000 (0:00:00.114) 0:02:00.745 ********* 2021-07-31T01:20:16.846 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2021-07-31T01:20:16.849 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:16.869 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:16 +0000 (0:00:00.102) 0:02:00.848 ********* 2021-07-31T01:20:16.948 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2021-07-31T01:20:16.949 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:16.970 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:16 +0000 (0:00:00.101) 0:02:00.950 ********* 2021-07-31T01:20:17.050 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2021-07-31T01:20:17.052 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:17.074 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:17 +0000 (0:00:00.103) 0:02:01.054 ********* 2021-07-31T01:20:17.164 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2021-07-31T01:20:17.166 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"ansible_facts": {"entitlements_path": "/etc/ansible/secrets/entitlements.yml"}, "changed": false} 2021-07-31T01:20:17.186 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:17 +0000 (0:00:00.112) 0:02:01.167 ********* 2021-07-31T01:20:17.289 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2021-07-31T01:20:17.291 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} ok: [smithi154.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-07-31T01:20:17.315 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:17 +0000 (0:00:00.128) 0:02:01.295 ********* 2021-07-31T01:20:17.397 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** ok: [smithi154.front.sepia.ceph.com] => {"ansible_facts": {"have_entitlements": true}, "changed": false} 2021-07-31T01:20:17.419 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:17 +0000 (0:00:00.104) 0:02:01.400 ********* 2021-07-31T01:20:17.719 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-07-31T01:20:17.721 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item=jdillaman) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "jdillaman", "name": "jdillaman", "remove": false, "state": "absent"} 2021-07-31T01:20:17.929 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2021-07-31T01:20:18.130 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2021-07-31T01:20:18.329 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2021-07-31T01:20:18.528 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2021-07-31T01:20:18.697 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2021-07-31T01:20:18.711 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed /root/.ansible/tmp/ansible-moduletmp-1627694418.5535085-b8l1nhpp/katello-ca-consumer-latest.noarch_hniwuq2.rpm"]} 2021-07-31T01:20:18.721 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:18 +0000 (0:00:01.301) 0:02:02.701 ********* 2021-07-31T01:20:18.729 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-07-31T01:20:18.730 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2021-07-31T01:20:18.931 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2021-07-31T01:20:19.121 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2021-07-31T01:20:19.269 INFO:teuthology.task.ansible.out: TASK [common : Set the releasever] ********************************************* 2021-07-31T01:20:19.271 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => {"changed": true, "checksum": "a626a25b7b500f9214acf310b1376506a11f0de8", "dest": "/etc/yum/vars/releasever", "gid": 0, "group": "root", "md5sum": "5e20d6ed7744f334ad7deb3cea4d5905", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 3, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1627694418.830418-4917-266831518939137/source", "state": "file", "uid": 0} 2021-07-31T01:20:19.296 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:19 +0000 (0:00:00.575) 0:02:03.276 ********* 2021-07-31T01:20:19.325 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-07-31T01:20:19.327 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2021-07-31T01:20:19.518 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "name": "jbainbri", "state": "absent"} 2021-07-31T01:20:19.731 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "name": "kdhananj", "state": "absent"} 2021-07-31T01:20:19.778 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:19 +0000 (0:00:00.481) 0:02:03.758 ********* 2021-07-31T01:20:19.913 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2021-07-31T01:20:19.915 INFO:teuthology.task.ansible.out:fatal: [smithi154.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} ...ignoring 2021-07-31T01:20:19.938 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:19 +0000 (0:00:00.159) 0:02:03.918 ********* 2021-07-31T01:20:20.038 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** ok: [smithi154.front.sepia.ceph.com] => {"ansible_facts": {"rhsm_registered": false}, "changed": false} 2021-07-31T01:20:20.061 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:20 +0000 (0:00:00.123) 0:02:04.041 ********* 2021-07-31T01:20:20.381 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2021-07-31T01:20:20.383 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => {"changed": true, "checksum": "41f0b9973307e6d58781b948bc481539828e2674", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "2504cfe296043f68ff0fc227b511d52e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1627694419.8986578-5035-48704728356613/source", "state": "file", "uid": 0} 2021-07-31T01:20:20.404 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:20 +0000 (0:00:00.343) 0:02:04.385 ********* 2021-07-31T01:20:20.473 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** ok: [smithi045.front.sepia.ceph.com] => { "msg": "Host smithi045.front.sepia.ceph.com is running RedHat 8.3 (Ootpa)" } 2021-07-31T01:20:20.495 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:20 +0000 (0:00:00.090) 0:02:04.475 ********* 2021-07-31T01:20:20.575 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2021-07-31T01:20:20.576 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/redhat_8.yml) => {"ansible_facts": {"nrpe_selinux_packages": ["python3-libsemanage", "python3-policycoreutils"], "rhsm_repos": ["rhel-8-for-x86_64-baseos-rpms", "rhel-8-for-x86_64-appstream-rpms"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/redhat_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/redhat_8.yml"} 2021-07-31T01:20:20.597 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:20 +0000 (0:00:00.102) 0:02:04.577 ********* 2021-07-31T01:20:20.665 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2021-07-31T01:20:20.667 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:20.688 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:20 +0000 (0:00:00.090) 0:02:04.668 ********* 2021-07-31T01:20:21.279 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2021-07-31T01:20:21.281 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "cmd": "timedatectl | grep -E \"Time ?zone\" | sed -e \"s/.*: \\(.*\\) (.*/\\1/\"", "delta": "0:00:00.347253", "end": "2021-07-31 01:20:21.254131", "rc": 0, "start": "2021-07-31 01:20:20.906878", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2021-07-31T01:20:21.307 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:21 +0000 (0:00:00.619) 0:02:05.288 ********* 2021-07-31T01:20:21.379 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:21.401 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:21 +0000 (0:00:00.093) 0:02:05.381 ********* 2021-07-31T01:20:21.471 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2021-07-31T01:20:21.473 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:21.495 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:21 +0000 (0:00:00.093) 0:02:05.475 ********* 2021-07-31T01:20:21.576 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2021-07-31T01:20:21.578 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:21.601 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:21 +0000 (0:00:00.105) 0:02:05.581 ********* 2021-07-31T01:20:21.679 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2021-07-31T01:20:21.681 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:21.702 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:21 +0000 (0:00:00.101) 0:02:05.683 ********* 2021-07-31T01:20:21.772 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:21.793 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:21 +0000 (0:00:00.090) 0:02:05.774 ********* 2021-07-31T01:20:21.873 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2021-07-31T01:20:21.874 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"ansible_facts": {"entitlements_path": "/etc/ansible/secrets/entitlements.yml"}, "changed": false} 2021-07-31T01:20:21.895 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:21 +0000 (0:00:00.101) 0:02:05.875 ********* 2021-07-31T01:20:21.986 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2021-07-31T01:20:21.988 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} ok: [smithi045.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-07-31T01:20:22.010 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:22 +0000 (0:00:00.115) 0:02:05.990 ********* 2021-07-31T01:20:22.092 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2021-07-31T01:20:22.093 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"ansible_facts": {"have_entitlements": true}, "changed": false} 2021-07-31T01:20:22.114 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:22 +0000 (0:00:00.103) 0:02:06.094 ********* 2021-07-31T01:20:23.127 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2021-07-31T01:20:23.129 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed /root/.ansible/tmp/ansible-moduletmp-1627694422.9759655-qqw34g1z/katello-ca-consumer-latest.noarch7n4yxpet.rpm"]} 2021-07-31T01:20:23.149 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:23 +0000 (0:00:01.034) 0:02:07.129 ********* 2021-07-31T01:20:23.714 INFO:teuthology.task.ansible.out: TASK [common : Set the releasever] ********************************************* 2021-07-31T01:20:23.716 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => {"changed": true, "checksum": "a626a25b7b500f9214acf310b1376506a11f0de8", "dest": "/etc/yum/vars/releasever", "gid": 0, "group": "root", "md5sum": "5e20d6ed7744f334ad7deb3cea4d5905", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 3, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1627694423.25781-5448-182299877505423/source", "state": "file", "uid": 0} 2021-07-31T01:20:23.737 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:23 +0000 (0:00:00.588) 0:02:07.717 ********* 2021-07-31T01:20:24.335 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2021-07-31T01:20:24.338 INFO:teuthology.task.ansible.out:fatal: [smithi045.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} ...ignoring 2021-07-31T01:20:24.358 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:24 +0000 (0:00:00.621) 0:02:08.338 ********* 2021-07-31T01:20:24.451 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** ok: [smithi045.front.sepia.ceph.com] => {"ansible_facts": {"rhsm_registered": false}, "changed": false} 2021-07-31T01:20:24.476 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:24 +0000 (0:00:00.118) 0:02:08.456 ********* 2021-07-31T01:20:37.753 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2021-07-31T01:20:37.755 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => {"attempts": 1, "censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2021-07-31T01:20:37.776 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:37 +0000 (0:00:13.300) 0:02:21.757 ********* 2021-07-31T01:20:37.857 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2021-07-31T01:20:37.859 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"ansible_facts": {"rhsm_registered": true}, "changed": false} 2021-07-31T01:20:37.880 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:37 +0000 (0:00:00.103) 0:02:21.860 ********* 2021-07-31T01:20:38.471 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2021-07-31T01:20:38.473 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"changed": false, "cmd": "subscription-manager release --list | grep -E '[0-9]'", "delta": "0:00:00.364349", "end": "2021-07-31 01:20:38.451295", "failed_when_result": false, "rc": 0, "start": "2021-07-31 01:20:38.086946", "stderr": "", "stderr_lines": [], "stdout": "7.8\n7.9\n7Server\n8\n8.2\n8.3\n8.4", "stdout_lines": ["7.8", "7.9", "7Server", "8", "8.2", "8.3", "8.4"]} 2021-07-31T01:20:38.493 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:38 +0000 (0:00:00.613) 0:02:22.474 ********* 2021-07-31T01:20:38.572 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2021-07-31T01:20:38.574 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:38.595 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:38 +0000 (0:00:00.101) 0:02:22.575 ********* 2021-07-31T01:20:38.675 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2021-07-31T01:20:38.677 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:38.697 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:38 +0000 (0:00:00.102) 0:02:22.678 ********* 2021-07-31T01:20:38.765 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2021-07-31T01:20:38.767 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-07-31T01:20:38.788 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:38 +0000 (0:00:00.090) 0:02:22.768 ********* 2021-07-31T01:20:38.867 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-07-31T01:20:38.868 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:38.889 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:38 +0000 (0:00:00.101) 0:02:22.869 ********* 2021-07-31T01:20:41.646 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* ok: [smithi154.front.sepia.ceph.com] => {"changed": false, "cmd": "subscription-manager repos --list | grep -B4 'Enabled:.*1' | grep 'Repo ID:' | sed -e 's/Repo ID:\\s*\\(.*\\)/\\1/' | sort", "delta": "0:00:02.518512", "end": "2021-07-31 01:20:41.622527", "rc": 0, "start": "2021-07-31 01:20:39.104015", "stderr": "", "stderr_lines": [], "stdout": "rhel-8-for-x86_64-appstream-rpms\nrhel-8-for-x86_64-baseos-rpms", "stdout_lines": ["rhel-8-for-x86_64-appstream-rpms", "rhel-8-for-x86_64-baseos-rpms"]} 2021-07-31T01:20:41.668 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:41 +0000 (0:00:02.778) 0:02:25.648 ********* 2021-07-31T01:20:41.758 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2021-07-31T01:20:41.760 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"ansible_facts": {"repo_list": ["rhel-8-for-x86_64-appstream-rpms", "rhel-8-for-x86_64-baseos-rpms"]}, "changed": false} 2021-07-31T01:20:41.780 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:41 +0000 (0:00:00.112) 0:02:25.760 ********* 2021-07-31T01:20:41.881 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2021-07-31T01:20:41.883 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:41.909 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:41 +0000 (0:00:00.128) 0:02:25.889 ********* 2021-07-31T01:20:41.994 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** ok: [smithi154.front.sepia.ceph.com] => {"ansible_facts": {"replace_repos": false}, "changed": false} 2021-07-31T01:20:42.024 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:42 +0000 (0:00:00.115) 0:02:26.005 ********* 2021-07-31T01:20:42.109 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2021-07-31T01:20:42.111 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"ansible_facts": {"replace_repos": true}, "changed": false} 2021-07-31T01:20:42.133 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:42 +0000 (0:00:00.108) 0:02:26.114 ********* 2021-07-31T01:20:42.655 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2021-07-31T01:20:42.657 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => {"attempts": 1, "censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2021-07-31T01:20:42.679 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:42 +0000 (0:00:00.545) 0:02:26.659 ********* 2021-07-31T01:20:42.758 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2021-07-31T01:20:42.760 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"ansible_facts": {"rhsm_registered": true}, "changed": false} 2021-07-31T01:20:42.782 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:42 +0000 (0:00:00.103) 0:02:26.762 ********* 2021-07-31T01:20:43.424 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2021-07-31T01:20:43.426 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "cmd": "subscription-manager release --list | grep -E '[0-9]'", "delta": "0:00:00.400975", "end": "2021-07-31 01:20:43.395872", "failed_when_result": false, "rc": 0, "start": "2021-07-31 01:20:42.994897", "stderr": "", "stderr_lines": [], "stdout": "7.8\n7.9\n7Server\n8\n8.2\n8.3\n8.4", "stdout_lines": ["7.8", "7.9", "7Server", "8", "8.2", "8.3", "8.4"]} 2021-07-31T01:20:43.448 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:43 +0000 (0:00:00.665) 0:02:27.428 ********* 2021-07-31T01:20:43.528 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2021-07-31T01:20:43.530 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:43.550 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:43 +0000 (0:00:00.102) 0:02:27.530 ********* 2021-07-31T01:20:43.618 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:43.640 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:43 +0000 (0:00:00.090) 0:02:27.621 ********* 2021-07-31T01:20:43.719 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2021-07-31T01:20:43.721 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-07-31T01:20:43.742 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:43 +0000 (0:00:00.101) 0:02:27.722 ********* 2021-07-31T01:20:43.821 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-07-31T01:20:43.823 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:43.842 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:43 +0000 (0:00:00.100) 0:02:27.823 ********* 2021-07-31T01:20:44.264 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2021-07-31T01:20:44.266 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2021-07-31T01:20:44.287 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:44 +0000 (0:00:00.444) 0:02:28.268 ********* 2021-07-31T01:20:46.816 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "cmd": "subscription-manager repos --list | grep -B4 'Enabled:.*1' | grep 'Repo ID:' | sed -e 's/Repo ID:\\s*\\(.*\\)/\\1/' | sort", "delta": "0:00:02.727551", "end": "2021-07-31 01:20:46.795790", "rc": 0, "start": "2021-07-31 01:20:44.068239", "stderr": "", "stderr_lines": [], "stdout": "rhel-8-for-x86_64-appstream-rpms\nrhel-8-for-x86_64-baseos-rpms", "stdout_lines": ["rhel-8-for-x86_64-appstream-rpms", "rhel-8-for-x86_64-baseos-rpms"]} 2021-07-31T01:20:46.838 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:46 +0000 (0:00:02.550) 0:02:30.818 ********* 2021-07-31T01:20:46.918 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2021-07-31T01:20:46.920 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"ansible_facts": {"repo_list": ["rhel-8-for-x86_64-appstream-rpms", "rhel-8-for-x86_64-baseos-rpms"]}, "changed": false} 2021-07-31T01:20:46.940 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:46 +0000 (0:00:00.102) 0:02:30.921 ********* 2021-07-31T01:20:47.009 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2021-07-31T01:20:47.011 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:47.031 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:47 +0000 (0:00:00.091) 0:02:31.012 ********* 2021-07-31T01:20:47.112 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2021-07-31T01:20:47.114 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"ansible_facts": {"replace_repos": false}, "changed": false} 2021-07-31T01:20:47.134 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:47 +0000 (0:00:00.102) 0:02:31.115 ********* 2021-07-31T01:20:47.214 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2021-07-31T01:20:47.216 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"ansible_facts": {"replace_repos": true}, "changed": false} 2021-07-31T01:20:47.236 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:47 +0000 (0:00:00.101) 0:02:31.216 ********* 2021-07-31T01:20:47.716 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2021-07-31T01:20:47.718 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => {"changed": true, "cmd": ["subscription-manager", "repos", "--enable=rhel-8-for-x86_64-baseos-rpms", "--enable=rhel-8-for-x86_64-appstream-rpms"], "delta": "0:00:03.196671", "end": "2021-07-31 01:20:47.700007", "rc": 0, "start": "2021-07-31 01:20:44.503336", "stderr": "", "stderr_lines": [], "stdout": "Repository 'rhel-8-for-x86_64-baseos-rpms' is enabled for this system.\nRepository 'rhel-8-for-x86_64-appstream-rpms' is enabled for this system.", "stdout_lines": ["Repository 'rhel-8-for-x86_64-baseos-rpms' is enabled for this system.", "Repository 'rhel-8-for-x86_64-appstream-rpms' is enabled for this system."]} 2021-07-31T01:20:47.742 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:47 +0000 (0:00:00.505) 0:02:31.722 ********* 2021-07-31T01:20:48.271 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2021-07-31T01:20:48.272 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"changed": false, "path": "/etc/yum.repos.d/cd.repo", "state": "absent"} 2021-07-31T01:20:48.293 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:48 +0000 (0:00:00.550) 0:02:32.273 ********* 2021-07-31T01:20:48.518 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2021-07-31T01:20:48.520 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-07-31T01:20:48.541 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:48 +0000 (0:00:00.248) 0:02:32.521 ********* 2021-07-31T01:20:49.048 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2021-07-31T01:20:49.050 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "86c0d58bde7ad4cfaafe592d2fefd92c920c42ca", "dest": "/etc/yum.repos.d/epel.repo", "gid": 0, "group": "root", "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 236, "state": "file", "uid": 0} 2021-07-31T01:20:49.479 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2021-07-31T01:20:49.481 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** ok: [smithi154.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a1bad5509ed35b895c2061841236b1cfe10619ae", "dest": "/etc/yum.repos.d/epel-testing.repo", "gid": 0, "group": "root", "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 262, "state": "file", "uid": 0} 2021-07-31T01:20:49.505 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:49 +0000 (0:00:00.964) 0:02:33.485 ********* 2021-07-31T01:20:49.532 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:49 +0000 (0:00:00.026) 0:02:33.512 ********* 2021-07-31T01:20:50.043 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2021-07-31T01:20:50.045 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "3c9ba7bacd1e30bed014cefd1fb4eab0f0f8fa10", "dest": "/etc/yum.repos.d/epel-mirrorlist", "gid": 0, "group": "root", "item": "epel-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 484, "state": "file", "uid": 0} 2021-07-31T01:20:50.494 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "d542c3e2c30a0d3eb7d454775342c302c0ded15b", "dest": "/etc/yum.repos.d/epel-testing-mirrorlist", "gid": 0, "group": "root", "item": "epel-testing-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 540, "state": "file", "uid": 0} 2021-07-31T01:20:50.517 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:50 +0000 (0:00:00.984) 0:02:34.497 ********* 2021-07-31T01:20:50.599 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-07-31T01:20:50.600 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:50.620 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:50 +0000 (0:00:00.103) 0:02:34.601 ********* 2021-07-31T01:20:50.689 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-07-31T01:20:50.691 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:50.713 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:50 +0000 (0:00:00.093) 0:02:34.694 ********* 2021-07-31T01:20:50.783 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2021-07-31T01:20:50.785 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:50.810 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:50 +0000 (0:00:00.096) 0:02:34.790 ********* 2021-07-31T01:20:50.884 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2021-07-31T01:20:50.885 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:50.907 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:50 +0000 (0:00:00.095) 0:02:34.886 ********* 2021-07-31T01:20:50.985 INFO:teuthology.task.ansible.out: TASK [common : Mask sleep units] *********************************************** 2021-07-31T01:20:50.987 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item=sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "sleep.target", "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item=suspend.target) => {"ansible_loop_var": "item", "changed": false, "item": "suspend.target", "skip_reason": "Conditional result was False"} 2021-07-31T01:20:50.997 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item=hibernate.target) => {"ansible_loop_var": "item", "changed": false, "item": "hibernate.target", "skip_reason": "Conditional result was False"} 2021-07-31T01:20:50.999 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item=hybrid-sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "hybrid-sleep.target", "skip_reason": "Conditional result was False"} 2021-07-31T01:20:51.021 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:51 +0000 (0:00:00.115) 0:02:35.001 ********* 2021-07-31T01:20:51.091 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:51.112 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:51 +0000 (0:00:00.091) 0:02:35.093 ********* 2021-07-31T01:20:51.183 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:51.208 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:51 +0000 (0:00:00.095) 0:02:35.188 ********* 2021-07-31T01:20:51.280 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:51.302 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:51 +0000 (0:00:00.094) 0:02:35.283 ********* 2021-07-31T01:20:51.371 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2021-07-31T01:20:51.373 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:51.395 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:51 +0000 (0:00:00.091) 0:02:35.374 ********* 2021-07-31T01:20:51.475 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2021-07-31T01:20:51.477 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:51.497 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:51 +0000 (0:00:00.102) 0:02:35.477 ********* 2021-07-31T01:20:52.518 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2021-07-31T01:20:52.520 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.835733", "end": "2021-07-31 01:20:52.500694", "rc": 0, "start": "2021-07-31 01:20:51.664961", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-07-31T01:20:52.543 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:52 +0000 (0:00:01.046) 0:02:36.523 ********* 2021-07-31T01:20:53.113 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2021-07-31T01:20:53.115 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => {"changed": true, "cmd": ["subscription-manager", "repos", "--enable=rhel-8-for-x86_64-baseos-rpms", "--enable=rhel-8-for-x86_64-appstream-rpms"], "delta": "0:00:03.365984", "end": "2021-07-31 01:20:53.095880", "rc": 0, "start": "2021-07-31 01:20:49.729896", "stderr": "", "stderr_lines": [], "stdout": "Repository 'rhel-8-for-x86_64-baseos-rpms' is enabled for this system.\nRepository 'rhel-8-for-x86_64-appstream-rpms' is enabled for this system.", "stdout_lines": ["Repository 'rhel-8-for-x86_64-baseos-rpms' is enabled for this system.", "Repository 'rhel-8-for-x86_64-appstream-rpms' is enabled for this system."]} 2021-07-31T01:20:53.136 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:53 +0000 (0:00:00.592) 0:02:37.116 ********* 2021-07-31T01:20:53.375 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2021-07-31T01:20:53.377 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "path": "/etc/yum.repos.d/cd.repo", "state": "absent"} 2021-07-31T01:20:53.398 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:53 +0000 (0:00:00.262) 0:02:37.379 ********* 2021-07-31T01:20:53.638 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2021-07-31T01:20:53.639 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-07-31T01:20:53.660 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:53 +0000 (0:00:00.261) 0:02:37.640 ********* 2021-07-31T01:20:54.192 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2021-07-31T01:20:54.194 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "86c0d58bde7ad4cfaafe592d2fefd92c920c42ca", "dest": "/etc/yum.repos.d/epel.repo", "gid": 0, "group": "root", "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 236, "state": "file", "uid": 0} 2021-07-31T01:20:54.644 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a1bad5509ed35b895c2061841236b1cfe10619ae", "dest": "/etc/yum.repos.d/epel-testing.repo", "gid": 0, "group": "root", "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 262, "state": "file", "uid": 0} 2021-07-31T01:20:54.669 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:54 +0000 (0:00:01.009) 0:02:38.649 ********* 2021-07-31T01:20:55.209 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2021-07-31T01:20:55.211 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "3c9ba7bacd1e30bed014cefd1fb4eab0f0f8fa10", "dest": "/etc/yum.repos.d/epel-mirrorlist", "gid": 0, "group": "root", "item": "epel-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 484, "state": "file", "uid": 0} 2021-07-31T01:20:55.668 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "d542c3e2c30a0d3eb7d454775342c302c0ded15b", "dest": "/etc/yum.repos.d/epel-testing-mirrorlist", "gid": 0, "group": "root", "item": "epel-testing-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 540, "state": "file", "uid": 0} 2021-07-31T01:20:55.691 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:55 +0000 (0:00:01.022) 0:02:39.671 ********* 2021-07-31T01:20:55.783 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:55.813 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:55 +0000 (0:00:00.121) 0:02:39.793 ********* 2021-07-31T01:20:55.886 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-07-31T01:20:55.890 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:55.909 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:55 +0000 (0:00:00.096) 0:02:39.889 ********* 2021-07-31T01:20:55.978 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2021-07-31T01:20:55.980 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:56.000 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:55 +0000 (0:00:00.091) 0:02:39.980 ********* 2021-07-31T01:20:56.068 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2021-07-31T01:20:56.070 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:56.090 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:56 +0000 (0:00:00.090) 0:02:40.071 ********* 2021-07-31T01:20:56.163 INFO:teuthology.task.ansible.out: TASK [common : Mask sleep units] *********************************************** skipping: [smithi045.front.sepia.ceph.com] => (item=sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "sleep.target", "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item=suspend.target) => {"ansible_loop_var": "item", "changed": false, "item": "suspend.target", "skip_reason": "Conditional result was False"} 2021-07-31T01:20:56.174 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=hibernate.target) => {"ansible_loop_var": "item", "changed": false, "item": "hibernate.target", "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item=hybrid-sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "hybrid-sleep.target", "skip_reason": "Conditional result was False"} 2021-07-31T01:20:56.196 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:56 +0000 (0:00:00.105) 0:02:40.176 ********* 2021-07-31T01:20:56.265 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-07-31T01:20:56.267 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:56.287 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:56 +0000 (0:00:00.090) 0:02:40.267 ********* 2021-07-31T01:20:56.356 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2021-07-31T01:20:56.357 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:56.382 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:56 +0000 (0:00:00.094) 0:02:40.362 ********* 2021-07-31T01:20:56.452 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2021-07-31T01:20:56.454 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:56.477 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:56 +0000 (0:00:00.094) 0:02:40.457 ********* 2021-07-31T01:20:56.546 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:56.569 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:56 +0000 (0:00:00.091) 0:02:40.549 ********* 2021-07-31T01:20:56.637 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2021-07-31T01:20:56.638 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:20:56.670 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:56 +0000 (0:00:00.101) 0:02:40.650 ********* 2021-07-31T01:20:57.524 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2021-07-31T01:20:57.526 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.652628", "end": "2021-07-31 01:20:57.500344", "rc": 0, "start": "2021-07-31 01:20:56.847716", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-07-31T01:20:57.550 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:20:57 +0000 (0:00:00.879) 0:02:41.529 ********* 2021-07-31T01:21:04.156 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** ok: [smithi154.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-07-31T01:21:04.179 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:21:04 +0000 (0:00:06.629) 0:02:48.159 ********* 2021-07-31T01:21:04.251 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2021-07-31T01:21:04.253 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:21:04.277 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:21:04 +0000 (0:00:00.098) 0:02:48.257 ********* 2021-07-31T01:21:04.369 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2021-07-31T01:21:04.371 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:21:04.404 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:21:04 +0000 (0:00:00.126) 0:02:48.384 ********* 2021-07-31T01:21:04.477 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2021-07-31T01:21:04.478 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:21:04.499 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:21:04 +0000 (0:00:00.094) 0:02:48.479 ********* 2021-07-31T01:21:04.568 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2021-07-31T01:21:04.570 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:21:04.593 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:21:04 +0000 (0:00:00.094) 0:02:48.574 ********* 2021-07-31T01:21:05.087 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2021-07-31T01:21:05.089 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "secontext": "system_u:object_r:krb5_conf_t:s0", "size": 131, "state": "file", "uid": 0} 2021-07-31T01:21:05.109 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:21:05 +0000 (0:00:00.515) 0:02:49.089 ********* 2021-07-31T01:21:07.817 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** ok: [smithi154.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-07-31T01:21:07.839 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:21:07 +0000 (0:00:02.729) 0:02:51.819 ********* 2021-07-31T01:21:08.356 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** ok: [smithi154.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "secontext": "system_u:object_r:bin_t:s0", "size": 2716736, "state": "file", "uid": 0} 2021-07-31T01:21:08.790 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "secontext": "system_u:object_r:bin_t:s0", "size": 1952038, "state": "file", "uid": 0} 2021-07-31T01:21:09.223 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "secontext": "system_u:object_r:bin_t:s0", "size": 580432, "state": "file", "uid": 0} 2021-07-31T01:21:09.247 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:21:09 +0000 (0:00:01.408) 0:02:53.227 ********* 2021-07-31T01:21:09.446 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2021-07-31T01:21:09.449 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "secontext": "system_u:object_r:bin_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-07-31T01:21:09.475 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:21:09 +0000 (0:00:00.228) 0:02:53.455 ********* 2021-07-31T01:21:10.032 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-07-31T01:21:10.055 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:21:10 +0000 (0:00:00.579) 0:02:54.035 ********* 2021-07-31T01:21:10.063 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-07-31T01:21:10.067 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": true, "checksum": "00d0404b545463be4a65f475e0870e07a48f35b6", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "md5sum": "14041bf2666d8f8d277567958c7e9f69", "mode": "0755", "owner": "root", "secontext": "system_u:object_r:bin_t:s0", "size": 13355, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1627694469.5745368-11475-99789346919533/source", "state": "file", "uid": 0} 2021-07-31T01:21:10.124 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:21:10.146 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:21:10 +0000 (0:00:00.091) 0:02:54.126 ********* 2021-07-31T01:21:10.226 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2021-07-31T01:21:10.228 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:21:10.248 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:21:10 +0000 (0:00:00.102) 0:02:54.229 ********* 2021-07-31T01:21:10.319 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2021-07-31T01:21:10.321 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:21:10.344 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:21:10 +0000 (0:00:00.094) 0:02:54.324 ********* 2021-07-31T01:21:10.415 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:21:10.438 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:21:10 +0000 (0:00:00.094) 0:02:54.419 ********* 2021-07-31T01:21:10.456 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-07-31T01:21:10.458 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 8602, "state": "file", "uid": 0} 2021-07-31T01:21:10.957 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": true, "checksum": "2fbc67b36d59f66fc2621f3ca2dcf60dea2167a5", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "md5sum": "fae049f2bb3feda9c19a2dc30bce838a", "mode": "0755", "owner": "root", "secontext": "system_u:object_r:bin_t:s0", "size": 3612, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1627694470.4828653-11475-201773646268808/source", "state": "file", "uid": 0} 2021-07-31T01:21:10.982 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:21:10 +0000 (0:00:00.543) 0:02:54.962 ********* 2021-07-31T01:21:11.024 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "secontext": "system_u:object_r:krb5_conf_t:s0", "size": 131, "state": "file", "uid": 0} 2021-07-31T01:21:11.057 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:21:11 +0000 (0:00:00.069) 0:02:55.031 ********* 2021-07-31T01:21:11.074 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:21:11.096 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:21:11 +0000 (0:00:00.045) 0:02:55.076 ********* 2021-07-31T01:21:11.180 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2021-07-31T01:21:11.182 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib64/nagios/plugins", "nrpe_group": "nrpe", "nrpe_packages": ["nagios-common", "nrpe", "nagios-plugins", "nagios-plugins-load"], "nrpe_service_name": "nrpe", "nrpe_user": "nrpe"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/yum_systems.yml"], "changed": false} 2021-07-31T01:21:11.204 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:21:11 +0000 (0:00:00.107) 0:02:55.184 ********* 2021-07-31T01:21:11.451 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* changed: [smithi154.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2021-07-31T01:21:11.479 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:21:11 +0000 (0:00:00.275) 0:02:55.459 ********* 2021-07-31T01:21:11.574 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** skipping: [smithi154.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-07-31T01:21:11.585 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-07-31T01:21:11.597 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-07-31T01:21:11.608 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-07-31T01:21:11.637 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:21:11 +0000 (0:00:00.152) 0:02:55.615 ********* 2021-07-31T01:21:13.852 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2021-07-31T01:21:13.854 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-07-31T01:21:13.881 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:21:13 +0000 (0:00:02.245) 0:02:57.861 ********* 2021-07-31T01:21:14.422 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-07-31T01:21:14.423 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "secontext": "system_u:object_r:bin_t:s0", "size": 2716736, "state": "file", "uid": 0} 2021-07-31T01:21:14.867 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "secontext": "system_u:object_r:bin_t:s0", "size": 1952038, "state": "file", "uid": 0} 2021-07-31T01:21:15.311 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "secontext": "system_u:object_r:bin_t:s0", "size": 580432, "state": "file", "uid": 0} 2021-07-31T01:21:15.334 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:21:15 +0000 (0:00:01.453) 0:02:59.315 ********* 2021-07-31T01:21:15.538 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2021-07-31T01:21:15.540 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "secontext": "system_u:object_r:bin_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-07-31T01:21:15.561 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:21:15 +0000 (0:00:00.226) 0:02:59.541 ********* 2021-07-31T01:21:16.157 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-07-31T01:21:16.159 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": true, "checksum": "00d0404b545463be4a65f475e0870e07a48f35b6", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "md5sum": "14041bf2666d8f8d277567958c7e9f69", "mode": "0755", "owner": "root", "secontext": "system_u:object_r:bin_t:s0", "size": 13355, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1627694475.6673026-12461-132238393295338/source", "state": "file", "uid": 0} 2021-07-31T01:21:16.591 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 8602, "state": "file", "uid": 0} 2021-07-31T01:21:17.085 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": true, "checksum": "2fbc67b36d59f66fc2621f3ca2dcf60dea2167a5", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "md5sum": "fae049f2bb3feda9c19a2dc30bce838a", "mode": "0755", "owner": "root", "secontext": "system_u:object_r:bin_t:s0", "size": 3612, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1627694476.615528-12461-40023977462546/source", "state": "file", "uid": 0} 2021-07-31T01:21:17.109 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:21:17 +0000 (0:00:01.547) 0:03:01.088 ********* 2021-07-31T01:21:17.180 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:21:17.202 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:21:17 +0000 (0:00:00.094) 0:03:01.183 ********* 2021-07-31T01:21:17.278 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** ok: [smithi045.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib64/nagios/plugins", "nrpe_group": "nrpe", "nrpe_packages": ["nagios-common", "nrpe", "nagios-plugins", "nagios-plugins-load"], "nrpe_service_name": "nrpe", "nrpe_user": "nrpe"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/yum_systems.yml"], "changed": false} 2021-07-31T01:21:17.311 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:21:17 +0000 (0:00:00.108) 0:03:01.291 ********* 2021-07-31T01:21:17.551 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2021-07-31T01:21:17.553 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2021-07-31T01:21:17.578 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:21:17 +0000 (0:00:00.267) 0:03:01.558 ********* 2021-07-31T01:21:17.681 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2021-07-31T01:21:17.683 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-07-31T01:21:17.692 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-07-31T01:21:17.714 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-07-31T01:21:17.726 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-07-31T01:21:17.754 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:21:17 +0000 (0:00:00.175) 0:03:01.734 ********* 2021-07-31T01:21:30.954 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2021-07-31T01:21:30.956 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: nagios-plugins-load-2.3.3-5.el8.x86_64", "Installed: nagios-common-4.4.6-4.el8.x86_64", "Installed: nagios-plugins-2.3.3-5.el8.x86_64", "Installed: nrpe-4.0.3-6.el8.x86_64", "Removed: nrpe-4.0.3-1.el8.x86_64", "Removed: nagios-common-4.4.5-1.el8.x86_64", "Removed: nagios-plugins-2.3.3-4.el8.x86_64", "Removed: nagios-plugins-load-2.3.3-4.el8.x86_64"]} 2021-07-31T01:21:30.977 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:21:30 +0000 (0:00:13.223) 0:03:14.958 ********* 2021-07-31T01:21:31.058 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** 2021-07-31T01:21:31.060 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:21:31.080 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:21:31 +0000 (0:00:00.102) 0:03:15.061 ********* 2021-07-31T01:21:31.590 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2021-07-31T01:21:31.592 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"changed": false, "checksum": "b423b985ccbb69b65a711d372645a5f704b30e8b", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "secontext": "system_u:object_r:etc_t:s0", "size": 178, "state": "file", "uid": 0} 2021-07-31T01:21:31.613 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:21:31 +0000 (0:00:00.532) 0:03:15.593 ********* 2021-07-31T01:21:32.132 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2021-07-31T01:21:32.133 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib64/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib64/nagios/plugins/check_mem.sh", "secontext": "system_u:object_r:nagios_unconfined_plugin_exec_t:s0", "size": 1964, "state": "file", "uid": 0} 2021-07-31T01:21:32.155 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:21:32 +0000 (0:00:00.542) 0:03:16.135 ********* 2021-07-31T01:21:32.250 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2021-07-31T01:21:32.252 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "DAEMON_OPTS", "skip_reason": "Conditional result was False"} 2021-07-31T01:21:32.262 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "NRPE_OPTS", "skip_reason": "Conditional result was False"} 2021-07-31T01:21:32.284 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:21:32 +0000 (0:00:00.129) 0:03:16.264 ********* 2021-07-31T01:21:32.522 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2021-07-31T01:21:32.524 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-07-31T01:21:32.545 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:21:32 +0000 (0:00:00.260) 0:03:16.525 ********* 2021-07-31T01:21:32.833 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2021-07-31T01:21:32.834 INFO:teuthology.task.ansible.out:fatal: [smithi154.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} ...ignoring 2021-07-31T01:21:32.856 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:21:32 +0000 (0:00:00.311) 0:03:16.836 ********* 2021-07-31T01:21:32.937 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2021-07-31T01:21:32.938 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:21:32.969 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:21:32 +0000 (0:00:00.112) 0:03:16.949 ********* 2021-07-31T01:21:33.476 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2021-07-31T01:21:33.478 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"changed": false, "checksum": "aa9150d4582f919d7a8504c01c8d86f05e3eeb5d", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "secontext": "system_u:object_r:nrpe_etc_t:s0", "size": 1184, "state": "file", "uid": 0} 2021-07-31T01:21:33.500 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:21:33 +0000 (0:00:00.531) 0:03:17.481 ********* 2021-07-31T01:21:34.267 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2021-07-31T01:21:34.268 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nrpe", "state": "started", "status": {"ActiveEnterTimestamp": "Sat 2021-07-31 01:21:17 UTC", "ActiveEnterTimestampMonotonic": "259531857", "ActiveExitTimestamp": "Sat 2021-07-31 01:21:17 UTC", "ActiveExitTimestampMonotonic": "259508165", "ActiveState": "active", "After": "tmp.mount var-run.mount time-sync.target local-fs.target sysinit.target systemd-tmpfiles-setup.service network-online.target system.slice systemd-journald.socket basic.target nss-lookup.target -.mount", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Sat 2021-07-31 01:21:17 UTC", "AssertTimestampMonotonic": "259531097", "Before": "multi-user.target plymouth-quit.service xdm.service getty@tty1.service shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Sat 2021-07-31 01:21:17 UTC", "Co 2021-07-31T01:21:34.269 INFO:teuthology.task.ansible.out:nditionTimestampMonotonic": "259531097", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target nrpe.socket", "ControlGroup": "/system.slice/nrpe.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/nrpe (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "16917", "ExecMainStartTimestamp": "Sat 2021-07-31 01:21:17 UTC", "ExecMainStartTimestampMonotonic": "259531834", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_SSL_OPT ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /run/nrpe/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nrpe.service", "FreezerState": "thawing", "GID": "984", "Group": "nrpe", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nrpe.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestamp": "Sat 2021-07-31 01:21:17 UTC", "InactiveEnterTimestampMonotonic": "259530415", "InactiveExitTimestamp": "Sat 2021-07-31 01:21:17 UTC", 2021-07-31T01:21:34.271 INFO:teuthology.task.ansible.out:"InactiveExitTimestampMonotonic": "259531857", "InvocationID": "b9ca645d147949ca97d6d6a514660297", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127237", "LimitNPROCSoft": "127237", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127237", "LimitSIGPENDINGSoft": "127237", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "16917", "MemoryAccounting": "yes", "MemoryCurrent": "872448", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "nrpe.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/run/nrpe/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateT 2021-07-31T01:21:34.273 INFO:teuthology.task.ansible.out:mp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice -.mount sysinit.target", "RequiresMountsFor": "/var/tmp /run/nrpe", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectory": "nrpe", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Sat 2021-07-31 01:21:17 UTC", "StateChangeTimestampMonotonic": "259531857", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203580", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "988", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "User": "nrpe", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Sat 2021-07-31 01:21:17 UTC", "WatchdogTimestampMonotonic": "259531856", "WatchdogUSec": "0"}} 2021-07-31T01:21:34.291 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:21:34 +0000 (0:00:00.790) 0:03:18.271 ********* 2021-07-31T01:21:34.526 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2021-07-31T01:21:34.528 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => {"changed": true, "cmd": ["getenforce"], "delta": "0:00:00.001674", "end": "2021-07-31 01:21:34.506669", "rc": 0, "start": "2021-07-31 01:21:34.504995", "stderr": "", "stderr_lines": [], "stdout": "Permissive", "stdout_lines": ["Permissive"]} 2021-07-31T01:21:34.551 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:21:34 +0000 (0:00:00.260) 0:03:18.531 ********* 2021-07-31T01:21:37.809 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2021-07-31T01:21:37.811 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: nagios-plugins-load-2.3.3-5.el8.x86_64", "Installed: nagios-common-4.4.6-4.el8.x86_64", "Installed: nagios-plugins-2.3.3-5.el8.x86_64", "Installed: nrpe-4.0.3-6.el8.x86_64", "Removed: nrpe-4.0.3-1.el8.x86_64", "Removed: nagios-common-4.4.5-1.el8.x86_64", "Removed: nagios-plugins-2.3.3-4.el8.x86_64", "Removed: nagios-plugins-load-2.3.3-4.el8.x86_64"]} 2021-07-31T01:21:37.834 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:21:37 +0000 (0:00:03.283) 0:03:21.814 ********* 2021-07-31T01:21:37.917 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:21:37.938 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:21:37 +0000 (0:00:00.104) 0:03:21.919 ********* 2021-07-31T01:21:38.468 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2021-07-31T01:21:38.470 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "checksum": "b423b985ccbb69b65a711d372645a5f704b30e8b", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "secontext": "system_u:object_r:etc_t:s0", "size": 178, "state": "file", "uid": 0} 2021-07-31T01:21:38.491 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:21:38 +0000 (0:00:00.552) 0:03:22.471 ********* 2021-07-31T01:21:39.001 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2021-07-31T01:21:39.003 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib64/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib64/nagios/plugins/check_mem.sh", "secontext": "system_u:object_r:nagios_unconfined_plugin_exec_t:s0", "size": 1964, "state": "file", "uid": 0} 2021-07-31T01:21:39.023 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:21:39 +0000 (0:00:00.532) 0:03:23.004 ********* 2021-07-31T01:21:39.104 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2021-07-31T01:21:39.106 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "DAEMON_OPTS", "skip_reason": "Conditional result was False"} 2021-07-31T01:21:39.115 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "NRPE_OPTS", "skip_reason": "Conditional result was False"} 2021-07-31T01:21:39.139 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:21:39 +0000 (0:00:00.115) 0:03:23.119 ********* 2021-07-31T01:21:39.377 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* ok: [smithi045.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-07-31T01:21:39.400 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:21:39 +0000 (0:00:00.260) 0:03:23.380 ********* 2021-07-31T01:21:39.724 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2021-07-31T01:21:39.725 INFO:teuthology.task.ansible.out:fatal: [smithi045.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} ...ignoring 2021-07-31T01:21:39.746 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:21:39 +0000 (0:00:00.345) 0:03:23.726 ********* 2021-07-31T01:21:39.816 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:21:39.839 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:21:39 +0000 (0:00:00.093) 0:03:23.820 ********* 2021-07-31T01:21:40.373 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "checksum": "aa9150d4582f919d7a8504c01c8d86f05e3eeb5d", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "secontext": "system_u:object_r:nrpe_etc_t:s0", "size": 1184, "state": "file", "uid": 0} 2021-07-31T01:21:40.395 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:21:40 +0000 (0:00:00.555) 0:03:24.375 ********* 2021-07-31T01:21:40.758 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2021-07-31T01:21:40.761 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nrpe", "state": "started", "status": {"ActiveEnterTimestamp": "Sat 2021-07-31 01:21:23 UTC", "ActiveEnterTimestampMonotonic": "312737816", "ActiveExitTimestamp": "Sat 2021-07-31 01:21:23 UTC", "ActiveExitTimestampMonotonic": "312715216", "ActiveState": "active", "After": "sysinit.target -.mount system.slice time-sync.target nss-lookup.target var-run.mount systemd-tmpfiles-setup.service network-online.target local-fs.target basic.target tmp.mount systemd-journald.socket", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Sat 2021-07-31 01:21:23 UTC", "AssertTimestampMonotonic": "312737057", "Before": "plymouth-quit.service xdm.service shutdown.target multi-user.target getty@tty1.service", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Sat 2021-07-31 01:21:23 UTC", "Co 2021-07-31T01:21:40.763 INFO:teuthology.task.ansible.out:nditionTimestampMonotonic": "312737057", "ConfigurationDirectoryMode": "0755", "Conflicts": "nrpe.socket shutdown.target", "ControlGroup": "/system.slice/nrpe.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/nrpe (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "16914", "ExecMainStartTimestamp": "Sat 2021-07-31 01:21:23 UTC", "ExecMainStartTimestampMonotonic": "312737791", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_SSL_OPT ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /run/nrpe/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nrpe.service", "FreezerState": "thawing", "GID": "984", "Group": "nrpe", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nrpe.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestamp": "Sat 2021-07-31 01:21:23 UTC", "InactiveEnterTimestampMonotonic": "312736398", "InactiveExitTimestamp": "Sat 2021-07-31 01:21:23 UTC", 2021-07-31T01:21:40.765 INFO:teuthology.task.ansible.out:"InactiveExitTimestampMonotonic": "312737816", "InvocationID": "634a5842ee99458495d3f381037341d9", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127262", "LimitNPROCSoft": "127262", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127262", "LimitSIGPENDINGSoft": "127262", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "16914", "MemoryAccounting": "yes", "MemoryCurrent": "876544", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "nrpe.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/run/nrpe/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateT 2021-07-31T01:21:40.767 INFO:teuthology.task.ansible.out:mp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice -.mount sysinit.target", "RequiresMountsFor": "/run/nrpe /var/tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectory": "nrpe", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Sat 2021-07-31 01:21:23 UTC", "StateChangeTimestampMonotonic": "312737816", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203619", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "988", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "User": "nrpe", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Sat 2021-07-31 01:21:23 UTC", "WatchdogTimestampMonotonic": "312737815", "WatchdogUSec": "0"}} 2021-07-31T01:21:40.782 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:21:40 +0000 (0:00:00.387) 0:03:24.762 ********* 2021-07-31T01:21:41.011 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* changed: [smithi045.front.sepia.ceph.com] => {"changed": true, "cmd": ["getenforce"], "delta": "0:00:00.001788", "end": "2021-07-31 01:21:40.993499", "rc": 0, "start": "2021-07-31 01:21:40.991711", "stderr": "", "stderr_lines": [], "stdout": "Permissive", "stdout_lines": ["Permissive"]} 2021-07-31T01:21:41.033 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:21:41 +0000 (0:00:00.250) 0:03:25.013 ********* 2021-07-31T01:21:45.888 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2021-07-31T01:21:45.890 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-07-31T01:21:45.912 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:21:45 +0000 (0:00:04.878) 0:03:29.892 ********* 2021-07-31T01:21:53.119 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-07-31T01:21:53.142 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:21:53 +0000 (0:00:07.229) 0:03:37.122 ********* 2021-07-31T01:22:36.913 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2021-07-31T01:22:36.915 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: selinux-policy-3.14.3-54.el8_3.4.noarch", "Installed: selinux-policy-targeted-3.14.3-54.el8_3.4.noarch", "Removed: selinux-policy-3.14.3-54.el8.noarch", "Removed: selinux-policy-targeted-3.14.3-54.el8.noarch"]} 2021-07-31T01:22:36.938 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:22:36 +0000 (0:00:43.795) 0:04:20.918 ********* 2021-07-31T01:22:37.733 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2021-07-31T01:22:37.735 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"changed": false, "name": "nagios_run_sudo", "persistent": true, "state": true} 2021-07-31T01:22:37.755 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:22:37 +0000 (0:00:00.817) 0:04:21.735 ********* 2021-07-31T01:22:47.639 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* changed: [smithi045.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: selinux-policy-3.14.3-54.el8_3.4.noarch", "Installed: selinux-policy-targeted-3.14.3-54.el8_3.4.noarch", "Removed: selinux-policy-3.14.3-54.el8.noarch", "Removed: selinux-policy-targeted-3.14.3-54.el8.noarch"]} 2021-07-31T01:22:47.663 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:22:47 +0000 (0:00:09.908) 0:04:31.644 ********* 2021-07-31T01:22:48.183 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2021-07-31T01:22:48.184 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "name": "nagios_run_sudo", "persistent": true, "state": true} 2021-07-31T01:22:48.204 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:22:48 +0000 (0:00:00.540) 0:04:32.184 ********* 2021-07-31T01:22:54.182 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2021-07-31T01:22:54.184 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-r", "nrpe"], "delta": "0:00:16.203591", "end": "2021-07-31 01:22:54.160777", "failed_when_result": false, "rc": 0, "start": "2021-07-31 01:22:37.957186", "stderr": "", "stderr_lines": [], "stdout": "libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority).", "stdout_lines": ["libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority)."]} 2021-07-31T01:22:54.206 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:22:54 +0000 (0:00:06.001) 0:04:38.186 ********* 2021-07-31T01:22:54.692 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2021-07-31T01:22:54.694 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"changed": false, "checksum": "99d9b2ea41fb2d6613898129718064833d2de428", "dest": "/tmp/nrpe.te", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/tmp/nrpe.te", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 934, "state": "file", "uid": 0} 2021-07-31T01:22:54.714 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:22:54 +0000 (0:00:00.508) 0:04:38.694 ********* 2021-07-31T01:22:54.953 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2021-07-31T01:22:54.954 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => {"changed": true, "cmd": ["checkmodule", "-M", "-m", "-o", "/tmp/nrpe.mod", "/tmp/nrpe.te"], "delta": "0:00:00.010845", "end": "2021-07-31 01:22:54.929640", "rc": 0, "start": "2021-07-31 01:22:54.918795", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-07-31T01:22:54.975 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:22:54 +0000 (0:00:00.261) 0:04:38.955 ********* 2021-07-31T01:22:55.212 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2021-07-31T01:22:55.214 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule_package", "-o", "/tmp/nrpe.pp", "-m", "/tmp/nrpe.mod"], "delta": "0:00:00.015054", "end": "2021-07-31 01:22:55.192841", "rc": 0, "start": "2021-07-31 01:22:55.177787", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-07-31T01:22:55.234 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:22:55 +0000 (0:00:00.258) 0:04:39.214 ********* 2021-07-31T01:23:05.340 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2021-07-31T01:23:05.342 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-r", "nrpe"], "delta": "0:00:16.878976", "end": "2021-07-31 01:23:05.320250", "failed_when_result": false, "rc": 0, "start": "2021-07-31 01:22:48.441274", "stderr": "", "stderr_lines": [], "stdout": "libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority).", "stdout_lines": ["libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority)."]} 2021-07-31T01:23:05.366 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:23:05 +0000 (0:00:10.132) 0:04:49.347 ********* 2021-07-31T01:23:05.908 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2021-07-31T01:23:05.910 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "checksum": "99d9b2ea41fb2d6613898129718064833d2de428", "dest": "/tmp/nrpe.te", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/tmp/nrpe.te", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 934, "state": "file", "uid": 0} 2021-07-31T01:23:05.930 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:23:05 +0000 (0:00:00.563) 0:04:49.910 ********* 2021-07-31T01:23:06.201 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2021-07-31T01:23:06.203 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => {"changed": true, "cmd": ["checkmodule", "-M", "-m", "-o", "/tmp/nrpe.mod", "/tmp/nrpe.te"], "delta": "0:00:00.015533", "end": "2021-07-31 01:23:06.176969", "rc": 0, "start": "2021-07-31 01:23:06.161436", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-07-31T01:23:06.233 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:23:06 +0000 (0:00:00.303) 0:04:50.214 ********* 2021-07-31T01:23:06.483 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2021-07-31T01:23:06.485 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule_package", "-o", "/tmp/nrpe.pp", "-m", "/tmp/nrpe.mod"], "delta": "0:00:00.013880", "end": "2021-07-31 01:23:06.463828", "rc": 0, "start": "2021-07-31 01:23:06.449948", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-07-31T01:23:06.507 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:23:06 +0000 (0:00:00.271) 0:04:50.485 ********* 2021-07-31T01:23:12.448 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2021-07-31T01:23:12.450 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-i", "/tmp/nrpe.pp"], "delta": "0:00:16.980374", "end": "2021-07-31 01:23:12.421590", "rc": 0, "start": "2021-07-31 01:22:55.441216", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-07-31T01:23:12.471 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:23:12 +0000 (0:00:05.966) 0:04:56.451 ********* 2021-07-31T01:23:12.707 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2021-07-31T01:23:12.709 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"changed": false, "path": "/tmp/nrpe.*", "state": "absent"} 2021-07-31T01:23:12.729 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:23:12 +0000 (0:00:00.258) 0:04:56.710 ********* 2021-07-31T01:23:12.799 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2021-07-31T01:23:12.801 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"ansible_facts": {"ceph_dependency_packages_to_remove": ["boost-random", "boost-program-options", "leveldb", "xmlstarlet", "python-jinja2", "python-ceph", "python-flask", "python-requests", "boost-random", "python-urllib3", "python-babel", "hdparm", "python-markupsafe", "python-werkzeug", "python-itsdangerous"], "ceph_packages_to_remove": ["ceph", "ceph-base", "ceph-selinux", "ceph-common", "ceph-debuginfo", "ceph-release", "libcephfs1", "ceph-radosgw", "python-ceph", "python-rados", "python-rbd", "python-cephfs", "librbd1", "librados2", "mod_fastcgi"], "dash_stream": "", "ntp_service_name": "ntpd", "packages_to_remove": ["device-mapper-multipath"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "sshd"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/dnf_systems.yml"], "changed": false} 2021-07-31T01:23:12.823 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:23:12 +0000 (0:00:00.093) 0:04:56.803 ********* 2021-07-31T01:23:12.904 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2021-07-31T01:23:12.906 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-07-31T01:23:12.928 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:23:12 +0000 (0:00:00.105) 0:04:56.908 ********* 2021-07-31T01:23:13.007 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2021-07-31T01:23:13.009 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/redhat_8.yml) => {"ansible_facts": {"common_yum_repos": {"lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "copr_repos": ["ceph/python3-asyncssh"], "epel_packages": ["dbench"], "nfs_service": "nfs-server", "ntp_service_name": "chronyd", "packages": ["dnf-utils", "git-all", "sysstat", "libedit", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse-libs", "openssl", "libuuid", "attr", "ant", "lsof", "gettext", "bc", "xfsdump", "blktrace", "usbredir", "podman", "redhat-lsb", "firewalld", "wget", "libev-devel", "valgrind", "nfs-utils", "ncurses-devel", "gcc", "git", "make", "python3-nose", "python3-virtualenv", "genisoimage", "qemu-img", "qemu-kvm-core", "qemu-kvm-block-rbd", "libacl-devel", "autoconf", "gdb", "iozone", "lvm2"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/redhat_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/redhat_8.yml"} 2021-07-31T01:23:13.031 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:23:13 +0000 (0:00:00.102) 0:04:57.011 ********* 2021-07-31T01:23:13.111 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ ok: [smithi154.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-07-31T01:23:13.135 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:23:13 +0000 (0:00:00.104) 0:04:57.116 ********* 2021-07-31T01:23:13.320 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2021-07-31T01:23:13.321 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-07-31T01:23:13.341 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:23:13 +0000 (0:00:00.206) 0:04:57.322 ********* 2021-07-31T01:23:13.565 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* ok: [smithi154.front.sepia.ceph.com] => {"changed": false, "gid": 1236, "name": "ubuntu", "state": "present", "system": false} 2021-07-31T01:23:13.588 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:23:13 +0000 (0:00:00.245) 0:04:57.567 ********* 2021-07-31T01:23:13.646 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:23:13.668 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:23:13 +0000 (0:00:00.081) 0:04:57.648 ********* 2021-07-31T01:23:13.737 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2021-07-31T01:23:13.738 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:23:13.758 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:23:13 +0000 (0:00:00.090) 0:04:57.739 ********* 2021-07-31T01:23:14.025 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** ok: [smithi154.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/bash", "state": "present", "uid": 10101} 2021-07-31T01:23:14.046 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:23:14 +0000 (0:00:00.287) 0:04:58.027 ********* 2021-07-31T01:23:14.596 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2021-07-31T01:23:14.598 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 112, "state": "file", "uid": 0} 2021-07-31T01:23:14.618 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:23:14 +0000 (0:00:00.572) 0:04:58.599 ********* 2021-07-31T01:23:14.697 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2021-07-31T01:23:14.698 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi154"}, "changed": false} 2021-07-31T01:23:14.719 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:23:14 +0000 (0:00:00.100) 0:04:58.699 ********* 2021-07-31T01:23:15.763 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2021-07-31T01:23:15.765 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "", "ansible_fqdn": "smithi154", "ansible_hostname": "smithi154", "ansible_nodename": "smithi154"}, "changed": false, "name": "smithi154"} 2021-07-31T01:23:15.785 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:23:15 +0000 (0:00:01.066) 0:04:59.766 ********* 2021-07-31T01:23:16.280 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2021-07-31T01:23:16.282 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"changed": false, "checksum": "a5e550b2aa74f6355e926d70d73a50ad9ab4b4be", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "secontext": "system_u:object_r:etc_t:s0", "size": 1043, "state": "file", "uid": 0} 2021-07-31T01:23:16.302 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:23:16 +0000 (0:00:00.516) 0:05:00.283 ********* 2021-07-31T01:23:16.802 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2021-07-31T01:23:16.804 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "secontext": "system_u:object_r:etc_t:s0", "size": 570, "state": "file", "uid": 0} 2021-07-31T01:23:16.824 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:23:16 +0000 (0:00:00.521) 0:05:00.804 ********* 2021-07-31T01:23:17.406 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2021-07-31T01:23:17.408 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2021-07-31T01:23:17.428 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:23:17 +0000 (0:00:00.603) 0:05:01.408 ********* 2021-07-31T01:23:17.935 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2021-07-31T01:23:17.937 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"changed": false, "checksum": "68d0b8ee2f88a77895f257aa69f32a42e47f455b", "dest": "/etc/security/limits.d/remote.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.d/remote.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 116, "state": "file", "uid": 0} 2021-07-31T01:23:17.957 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:23:17 +0000 (0:00:00.529) 0:05:01.938 ********* 2021-07-31T01:23:18.195 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-07-31T01:23:18.279 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"changed": false, "dest": "/etc/fuse.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 0, "state": "file", "uid": 0} Saturday 31 July 2021 01:23:18 +0000 (0:00:00.260) 0:05:02.198 ********* 2021-07-31T01:23:18.452 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-07-31T01:23:18.454 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"changed": false, "gid": 36, "name": "kvm", "state": "present", "system": false} 2021-07-31T01:23:18.473 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:23:18 +0000 (0:00:00.255) 0:05:02.453 ********* 2021-07-31T01:23:18.739 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-07-31T01:23:18.741 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1236, "groups": "kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2021-07-31T01:23:18.763 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:23:18 +0000 (0:00:00.290) 0:05:02.744 ********* 2021-07-31T01:23:19.269 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-07-31T01:23:19.270 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"changed": false, "checksum": "be3223c8b0917bde02657a5f9ddd95277cc3f812", "dest": "/etc/sudoers", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers", "secontext": "system_u:object_r:etc_t:s0", "size": 1808, "state": "file", "uid": 0} 2021-07-31T01:23:19.301 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:23:19 +0000 (0:00:00.537) 0:05:03.281 ********* 2021-07-31T01:23:19.823 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-07-31T01:23:19.825 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"changed": false, "checksum": "12a2e17405b9e00ea2061e25929905d08e5fe19f", "dest": "/etc/security/limits.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 854, "state": "file", "uid": 0} 2021-07-31T01:23:19.845 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:23:19 +0000 (0:00:00.544) 0:05:03.825 ********* 2021-07-31T01:23:22.183 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-07-31T01:23:22.185 INFO:teuthology.task.ansible.out:[WARNING]: Consider using the yum, dnf or zypper module rather than running 'rpm'. If you need to use command because yum, dnf or zypper is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi154.front.sepia.ceph.com] => {"changed": true, "cmd": ["rpm", "--rebuilddb"], "delta": "0:00:02.113597", "end": "2021-07-31 01:23:22.161289", "rc": 0, "start": "2021-07-31 01:23:20.047692", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-07-31T01:23:22.206 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:23:22 +0000 (0:00:02.360) 0:05:06.186 ********* 2021-07-31T01:23:22.276 INFO:teuthology.task.ansible.out: TASK [testnode : Clean up Stream distro-provided repos] ************************ 2021-07-31T01:23:22.277 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:23:22.299 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:23:22 +0000 (0:00:00.092) 0:05:06.279 ********* 2021-07-31T01:23:22.368 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-07-31T01:23:22.370 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:23:22.390 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:23:22 +0000 (0:00:00.090) 0:05:06.370 ********* 2021-07-31T01:23:22.460 INFO:teuthology.task.ansible.out: TASK [testnode : Include CentOS Stream specific variables] ********************* 2021-07-31T01:23:22.461 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml", "skip_reason": "Conditional result was False"} 2021-07-31T01:23:22.483 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:23:22 +0000 (0:00:00.093) 0:05:06.463 ********* 2021-07-31T01:23:22.554 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:23:22.576 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:23:22 +0000 (0:00:00.092) 0:05:06.556 ********* 2021-07-31T01:23:23.107 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2021-07-31T01:23:23.109 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'key': 'lab-extras', 'value': {'name': 'lab-extras', 'baseurl': 'http://apt-mirror.front.sepia.ceph.com/lab-extras/8/', 'enabled': 1, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a0e2ac0628a264e14b96b1c57627a3aa344a6c52", "dest": "/etc/yum.repos.d/lab-extras.repo", "gid": 0, "group": "root", "item": {"key": "lab-extras", "value": {"baseurl": "http://apt-mirror.front.sepia.ceph.com/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/lab-extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 203, "state": "file", "uid": 0} 2021-07-31T01:23:23.133 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:23:23 +0000 (0:00:00.555) 0:05:07.111 ********* 2021-07-31T01:23:23.200 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2021-07-31T01:23:23.222 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:23:23 +0000 (0:00:00.091) 0:05:07.202 ********* 2021-07-31T01:23:23.674 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** changed: [smithi045.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-i", "/tmp/nrpe.pp"], "delta": "0:00:16.929084", "end": "2021-07-31 01:23:23.645042", "rc": 0, "start": "2021-07-31 01:23:06.715958", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-07-31T01:23:23.696 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:23:23 +0000 (0:00:00.474) 0:05:07.677 ********* 2021-07-31T01:23:23.946 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2021-07-31T01:23:23.947 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "path": "/tmp/nrpe.*", "state": "absent"} 2021-07-31T01:23:23.968 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:23:23 +0000 (0:00:00.271) 0:05:07.948 ********* 2021-07-31T01:23:24.037 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2021-07-31T01:23:24.039 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"ansible_facts": {"ceph_dependency_packages_to_remove": ["boost-random", "boost-program-options", "leveldb", "xmlstarlet", "python-jinja2", "python-ceph", "python-flask", "python-requests", "boost-random", "python-urllib3", "python-babel", "hdparm", "python-markupsafe", "python-werkzeug", "python-itsdangerous"], "ceph_packages_to_remove": ["ceph", "ceph-base", "ceph-selinux", "ceph-common", "ceph-debuginfo", "ceph-release", "libcephfs1", "ceph-radosgw", "python-ceph", "python-rados", "python-rbd", "python-cephfs", "librbd1", "librados2", "mod_fastcgi"], "dash_stream": "", "ntp_service_name": "ntpd", "packages_to_remove": ["device-mapper-multipath"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "sshd"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/dnf_systems.yml"], "changed": false} 2021-07-31T01:23:24.062 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:23:24 +0000 (0:00:00.093) 0:05:08.042 ********* 2021-07-31T01:23:24.089 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2021-07-31T01:23:24.091 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item=ceph/python3-asyncssh) => {"ansible_loop_var": "item", "changed": true, "cmd": ["dnf", "-y", "copr", "enable", "ceph/python3-asyncssh"], "delta": "0:00:00.592619", "end": "2021-07-31 01:23:24.066789", "item": "ceph/python3-asyncssh", "rc": 0, "start": "2021-07-31 01:23:23.474170", "stderr": "Enabling a Copr repository. Please note that this repository is not part\nof the main distribution, and quality may vary.\n\nThe Fedora Project does not exercise any power over the contents of\nthis repository beyond the rules outlined in the Copr FAQ at\n,\nand packages are not held to any quality or security level.\n\nPlease do not file bug reports about these packages in Fedora\nBugzilla. In case of problems, contact the owner of this repository.", "stderr_lines": ["Enabling a Copr repository. Please note that this repository is not part", "of the main distribution, and quality may vary.", "", "The Fedora Project does not exercise any power over the contents of", "this repository beyond the rules outlined in the Copr FAQ at", ",", "and packages are not held to any quality or security level.", "", "Please do not file bug reports about these packages in Fedora", "Bugzilla. In case of problems, contact the owner of this repository."], "stdout": "Updating Subscription Management repositories.\nRepository successfully enabled.", "stdout_lines": ["Updating Subscription Management repositories.", "Repository successfully enabled."], "warnings": ["Consider using the dnf module rather than running 'dnf'. If you need to use command because dnf is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message."]} [WARNING]: Consider using the dnf module rather than running 'dnf'. If you need to use 2021-07-31T01:23:24.093 INFO:teuthology.task.ansible.out: command because dnf is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. 2021-07-31T01:23:24.115 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:23:24 +0000 (0:00:00.053) 0:05:08.095 ********* 2021-07-31T01:23:24.132 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2021-07-31T01:23:24.134 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-07-31T01:23:24.159 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:23:24 +0000 (0:00:00.044) 0:05:08.139 ********* 2021-07-31T01:23:24.177 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:23:24.201 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:23:24 +0000 (0:00:00.041) 0:05:08.181 ********* 2021-07-31T01:23:24.242 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2021-07-31T01:23:24.244 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/redhat_8.yml) => {"ansible_facts": {"common_yum_repos": {"lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "copr_repos": ["ceph/python3-asyncssh"], "epel_packages": ["dbench"], "nfs_service": "nfs-server", "ntp_service_name": "chronyd", "packages": ["dnf-utils", "git-all", "sysstat", "libedit", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse-libs", "openssl", "libuuid", "attr", "ant", "lsof", "gettext", "bc", "xfsdump", "blktrace", "usbredir", "podman", "redhat-lsb", "firewalld", "wget", "libev-devel", "valgrind", "nfs-utils", "ncurses-devel", "gcc", "git", "make", "python3-nose", "python3-virtualenv", "genisoimage", "qemu-img", "qemu-kvm-core", "qemu-kvm-block-rbd", "libacl-devel", "autoconf", "gdb", "iozone", "lvm2"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/redhat_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/redhat_8.yml"} 2021-07-31T01:23:24.270 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:23:24 +0000 (0:00:00.068) 0:05:08.250 ********* 2021-07-31T01:23:24.289 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2021-07-31T01:23:24.291 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-07-31T01:23:24.316 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:23:24 +0000 (0:00:00.045) 0:05:08.296 ********* 2021-07-31T01:23:24.355 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2021-07-31T01:23:24.357 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-07-31T01:23:24.381 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:23:24 +0000 (0:00:00.064) 0:05:08.361 ********* 2021-07-31T01:23:24.387 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2021-07-31T01:23:24.388 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:23:24.413 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:23:24 +0000 (0:00:00.032) 0:05:08.393 ********* 2021-07-31T01:23:24.506 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-07-31T01:23:24.508 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:23:24.528 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:23:24 +0000 (0:00:00.115) 0:05:08.509 ********* 2021-07-31T01:23:24.577 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2021-07-31T01:23:24.579 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-07-31T01:23:24.601 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:23:24 +0000 (0:00:00.072) 0:05:08.581 ********* 2021-07-31T01:23:24.608 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2021-07-31T01:23:24.609 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:23:24.632 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:23:24 +0000 (0:00:00.030) 0:05:08.612 ********* 2021-07-31T01:23:24.710 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2021-07-31T01:23:24.712 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:23:24.732 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:23:24 +0000 (0:00:00.100) 0:05:08.713 ********* 2021-07-31T01:23:24.838 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "gid": 1236, "name": "ubuntu", "state": "present", "system": false} 2021-07-31T01:23:24.861 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:23:24 +0000 (0:00:00.128) 0:05:08.841 ********* 2021-07-31T01:23:24.933 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2021-07-31T01:23:24.935 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:23:24.956 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:23:24 +0000 (0:00:00.094) 0:05:08.936 ********* 2021-07-31T01:23:24.985 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2021-07-31T01:23:24.987 INFO:teuthology.task.ansible.out:fatal: [smithi154.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": ["rpm", "-q", "ceph-debuginfo"], "delta": "0:00:00.007188", "end": "2021-07-31 01:23:24.962868", "msg": "non-zero return code", "rc": 1, "start": "2021-07-31 01:23:24.955680", "stderr": "", "stderr_lines": [], "stdout": "package ceph-debuginfo is not installed", "stdout_lines": ["package ceph-debuginfo is not installed"]} ...ignoring 2021-07-31T01:23:25.010 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:23:25 +0000 (0:00:00.054) 0:05:08.990 ********* 2021-07-31T01:23:25.028 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2021-07-31T01:23:25.029 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:23:25.051 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:23:25 +0000 (0:00:00.041) 0:05:09.031 ********* 2021-07-31T01:23:25.089 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2021-07-31T01:23:25.091 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:23:25.117 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:23:25 +0000 (0:00:00.065) 0:05:09.097 ********* 2021-07-31T01:23:25.315 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2021-07-31T01:23:25.317 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/bash", "state": "present", "uid": 10101} 2021-07-31T01:23:25.338 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:23:25 +0000 (0:00:00.221) 0:05:09.318 ********* 2021-07-31T01:23:25.921 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2021-07-31T01:23:25.923 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 112, "state": "file", "uid": 0} 2021-07-31T01:23:25.943 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:23:25 +0000 (0:00:00.605) 0:05:09.924 ********* 2021-07-31T01:23:26.013 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* ok: [smithi045.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi045"}, "changed": false} 2021-07-31T01:23:26.035 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:23:26 +0000 (0:00:00.091) 0:05:10.015 ********* 2021-07-31T01:23:26.697 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2021-07-31T01:23:26.699 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "", "ansible_fqdn": "smithi045", "ansible_hostname": "smithi045", "ansible_nodename": "smithi045"}, "changed": false, "name": "smithi045"} 2021-07-31T01:23:26.719 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:23:26 +0000 (0:00:00.683) 0:05:10.699 ********* 2021-07-31T01:23:27.224 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2021-07-31T01:23:27.226 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "checksum": "a5e550b2aa74f6355e926d70d73a50ad9ab4b4be", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "secontext": "system_u:object_r:etc_t:s0", "size": 1043, "state": "file", "uid": 0} 2021-07-31T01:23:27.250 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:23:27 +0000 (0:00:00.530) 0:05:11.230 ********* 2021-07-31T01:23:27.759 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2021-07-31T01:23:27.761 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "secontext": "system_u:object_r:etc_t:s0", "size": 570, "state": "file", "uid": 0} 2021-07-31T01:23:27.781 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:23:27 +0000 (0:00:00.531) 0:05:11.762 ********* 2021-07-31T01:23:28.407 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** changed: [smithi045.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2021-07-31T01:23:28.434 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:23:28 +0000 (0:00:00.651) 0:05:12.413 ********* 2021-07-31T01:23:29.043 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "checksum": "68d0b8ee2f88a77895f257aa69f32a42e47f455b", "dest": "/etc/security/limits.d/remote.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.d/remote.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 116, "state": "file", "uid": 0} 2021-07-31T01:23:29.067 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:23:29 +0000 (0:00:00.634) 0:05:13.047 ********* 2021-07-31T01:23:29.305 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-07-31T01:23:29.307 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "dest": "/etc/fuse.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 0, "state": "file", "uid": 0} 2021-07-31T01:23:29.326 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:23:29 +0000 (0:00:00.259) 0:05:13.307 ********* 2021-07-31T01:23:29.574 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "gid": 36, "name": "kvm", "state": "present", "system": false} 2021-07-31T01:23:29.596 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:23:29 +0000 (0:00:00.269) 0:05:13.576 ********* 2021-07-31T01:23:29.863 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-07-31T01:23:29.864 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1236, "groups": "kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2021-07-31T01:23:29.886 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:23:29 +0000 (0:00:00.289) 0:05:13.866 ********* 2021-07-31T01:23:30.395 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-07-31T01:23:30.396 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "checksum": "be3223c8b0917bde02657a5f9ddd95277cc3f812", "dest": "/etc/sudoers", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers", "secontext": "system_u:object_r:etc_t:s0", "size": 1808, "state": "file", "uid": 0} 2021-07-31T01:23:30.419 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:23:30 +0000 (0:00:00.533) 0:05:14.400 ********* 2021-07-31T01:23:30.939 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-07-31T01:23:30.941 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "checksum": "12a2e17405b9e00ea2061e25929905d08e5fe19f", "dest": "/etc/security/limits.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 854, "state": "file", "uid": 0} 2021-07-31T01:23:30.962 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:23:30 +0000 (0:00:00.541) 0:05:14.941 ********* 2021-07-31T01:23:33.523 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-07-31T01:23:33.525 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => {"changed": true, "cmd": ["rpm", "--rebuilddb"], "delta": "0:00:02.330153", "end": "2021-07-31 01:23:33.500130", "rc": 0, "start": "2021-07-31 01:23:31.169977", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-07-31T01:23:33.546 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:23:33 +0000 (0:00:02.584) 0:05:17.526 ********* 2021-07-31T01:23:33.615 INFO:teuthology.task.ansible.out: TASK [testnode : Clean up Stream distro-provided repos] ************************ 2021-07-31T01:23:33.617 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:23:33.637 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:23:33 +0000 (0:00:00.091) 0:05:17.617 ********* 2021-07-31T01:23:33.718 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-07-31T01:23:33.720 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:23:33.741 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:23:33 +0000 (0:00:00.103) 0:05:17.721 ********* 2021-07-31T01:23:33.823 INFO:teuthology.task.ansible.out: TASK [testnode : Include CentOS Stream specific variables] ********************* 2021-07-31T01:23:33.825 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml", "skip_reason": "Conditional result was False"} 2021-07-31T01:23:33.845 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:23:33 +0000 (0:00:00.104) 0:05:17.826 ********* 2021-07-31T01:23:33.915 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2021-07-31T01:23:33.916 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:23:33.938 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:23:33 +0000 (0:00:00.092) 0:05:17.919 ********* 2021-07-31T01:23:34.473 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2021-07-31T01:23:34.474 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'key': 'lab-extras', 'value': {'name': 'lab-extras', 'baseurl': 'http://apt-mirror.front.sepia.ceph.com/lab-extras/8/', 'enabled': 1, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a0e2ac0628a264e14b96b1c57627a3aa344a6c52", "dest": "/etc/yum.repos.d/lab-extras.repo", "gid": 0, "group": "root", "item": {"key": "lab-extras", "value": {"baseurl": "http://apt-mirror.front.sepia.ceph.com/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/lab-extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 203, "state": "file", "uid": 0} 2021-07-31T01:23:34.497 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:23:34 +0000 (0:00:00.558) 0:05:18.477 ********* 2021-07-31T01:23:34.556 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2021-07-31T01:23:34.577 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:23:34 +0000 (0:00:00.080) 0:05:18.558 ********* 2021-07-31T01:23:35.470 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2021-07-31T01:23:35.472 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item=ceph/python3-asyncssh) => {"ansible_loop_var": "item", "changed": true, "cmd": ["dnf", "-y", "copr", "enable", "ceph/python3-asyncssh"], "delta": "0:00:00.642377", "end": "2021-07-31 01:23:35.442579", "item": "ceph/python3-asyncssh", "rc": 0, "start": "2021-07-31 01:23:34.800202", "stderr": "Enabling a Copr repository. Please note that this repository is not part\nof the main distribution, and quality may vary.\n\nThe Fedora Project does not exercise any power over the contents of\nthis repository beyond the rules outlined in the Copr FAQ at\n,\nand packages are not held to any quality or security level.\n\nPlease do not file bug reports about these packages in Fedora\nBugzilla. In case of problems, contact the owner of this repository.", "stderr_lines": ["Enabling a Copr repository. Please note that this repository is not part", "of the main distribution, and quality may vary.", "", "The Fedora Project does not exercise any power over the contents of", "this repository beyond the rules outlined in the Copr FAQ at", ",", "and packages are not held to any quality or security level.", "", "Please do not file bug reports about these packages in Fedora", "Bugzilla. In case of problems, contact the owner of this repository."], "stdout": "Updating Subscription Management repositories.\nRepository successfully enabled.", "stdout_lines": ["Updating Subscription Management repositories.", "Repository successfully enabled."], "warnings": ["Consider using the dnf module rather than running 'dnf'. If you need to use command because dnf is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message."]} 2021-07-31T01:23:35.494 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:23:35 +0000 (0:00:00.916) 0:05:19.474 ********* 2021-07-31T01:23:35.563 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2021-07-31T01:23:35.564 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:23:35.585 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:23:35 +0000 (0:00:00.090) 0:05:19.565 ********* 2021-07-31T01:23:35.665 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2021-07-31T01:23:35.667 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-07-31T01:23:35.688 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:23:35 +0000 (0:00:00.103) 0:05:19.668 ********* 2021-07-31T01:23:35.757 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2021-07-31T01:23:35.759 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:23:35.779 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:23:35 +0000 (0:00:00.091) 0:05:19.759 ********* 2021-07-31T01:23:35.849 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-07-31T01:23:35.851 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:23:35.873 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:23:35 +0000 (0:00:00.094) 0:05:19.854 ********* 2021-07-31T01:23:35.953 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2021-07-31T01:23:35.955 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:23:35.976 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:23:35 +0000 (0:00:00.102) 0:05:19.956 ********* 2021-07-31T01:23:36.045 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2021-07-31T01:23:36.047 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:23:36.067 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:23:36 +0000 (0:00:00.091) 0:05:20.047 ********* 2021-07-31T01:23:36.324 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2021-07-31T01:23:36.326 INFO:teuthology.task.ansible.out:fatal: [smithi045.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": ["rpm", "-q", "ceph-debuginfo"], "delta": "0:00:00.007462", "end": "2021-07-31 01:23:36.303711", "msg": "non-zero return code", "rc": 1, "start": "2021-07-31 01:23:36.296249", "stderr": "", "stderr_lines": [], "stdout": "package ceph-debuginfo is not installed", "stdout_lines": ["package ceph-debuginfo is not installed"]} ...ignoring 2021-07-31T01:23:36.349 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:23:36 +0000 (0:00:00.281) 0:05:20.329 ********* 2021-07-31T01:23:36.419 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:23:36.452 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:23:36 +0000 (0:00:00.102) 0:05:20.432 ********* 2021-07-31T01:23:52.571 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-07-31T01:23:52.573 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: librados2-1:12.2.7-9.el8.x86_64", "Removed: librbd1-1:12.2.7-9.el8.x86_64", "Removed: qemu-kvm-block-rbd-15:4.2.0-34.module+el8.3.0+7976+077be4ec.x86_64"]} 2021-07-31T01:23:52.597 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:23:52 +0000 (0:00:16.145) 0:05:36.577 ********* 2021-07-31T01:24:05.587 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: librados2-1:12.2.7-9.el8.x86_64", "Removed: librbd1-1:12.2.7-9.el8.x86_64", "Removed: qemu-kvm-block-rbd-15:4.2.0-34.module+el8.3.0+7976+077be4ec.x86_64"]} 2021-07-31T01:24:05.611 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:24:05 +0000 (0:00:13.013) 0:05:49.591 ********* 2021-07-31T01:24:17.885 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-07-31T01:24:17.887 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: boost-program-options-1.66.0-10.el8.x86_64", "Removed: boost-random-1.66.0-10.el8.x86_64"]} 2021-07-31T01:24:17.911 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:24:17 +0000 (0:00:12.300) 0:06:01.891 ********* 2021-07-31T01:24:31.797 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: boost-program-options-1.66.0-10.el8.x86_64", "Removed: boost-random-1.66.0-10.el8.x86_64"]} 2021-07-31T01:24:31.833 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:24:31 +0000 (0:00:13.921) 0:06:15.813 ********* 2021-07-31T01:25:12.098 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* changed: [smithi154.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: poppler-0.66.0-27.el8.x86_64", "Installed: perl-NTLM-1.09-17.module+el8.3.0+6498+9eecfe51.noarch", "Installed: redhat-lsb-desktop-4.1-47.el8.x86_64", "Installed: poppler-utils-0.66.0-27.el8.x86_64", "Installed: perl-Module-Runtime-0.016-2.el8.noarch", "Installed: cups-filters-libs-1.20.0-20.el8.x86_64", "Installed: ghostscript-9.25-7.el8.x86_64", "Installed: qemu-kvm-common-15:4.2.0-34.module+el8.3.0+10437+1ca0c2ba.5.x86_64", "Installed: libXxf86misc-1.0.4-1.el8.x86_64", "Installed: urw-base35-bookman-fonts-20170801-10.el8.noarch", "Installed: boost-program-options-1.66.0-10.el8.x86_64", "Installed: perl-XML-SAX-Base-1.09-4.el8.noarch", "Installed: perl-Module-Pluggable-2:5.2-7.el8.noarch", "Installed: urw-base35-p052-fonts-20170801-10.el8.noarch", "Installed: openjpeg2-2.3.1-6.el8.x86_64", "Installed: liberation-fonts-common-1:2.00.3-7.el8.noarch", "Installed: perl-Net-HTTP-6.17-2.module+el8.3.0+6498+9eecfe51.noarch", "Installed: cups-1:2.2.6-38.el8.x86_64", "Installed: liberation-mono-fonts-1:2.00.3-7.el8.noarch", "Installed: libmcpp-2.7.2-20.el8.x86_64", "Installed: urw-base35-nimbus-mono-ps-fonts-20170801-10.el8.noarch", "Installed: cups-filesystem-1:2.2.6-38.el8.noarch", "Installed: urw-base35-nimbus-roman-fonts-20170801-10.el8.noarch", "Installed: perl-XML-NamespaceSupport-1.12-4.el8.noarch", "Installed: perl-WWW-RobotRules-6.02-18.module+el8.3.0+6498+9eecfe51.noarch", "Installed: libXmu-1.1.3-1.el8.x86_64", "Installed: urw-base35-z003-fonts-20170801-10.el8.noarch", "Installed: xorg-x11-server-utils-7.7-27.el8.x86_64", "Installed: mcpp-2.7.2-20.el8.x86_64", "Installed: google-droid-sans-fonts-20120715-13.el8.noarch", "Installed: urw-base35-fonts-common-20170801-10.el8.noarch", "Installed: librados2-1:12.2.7-9.el8.x86_64", "Installed: lcms2-2.9-2.el8.x86_64", "Installed: po 2021-07-31T01:25:12.100 INFO:teuthology.task.ansible.out:ppler-data-0.4.9-1.el8.noarch", "Installed: perl-File-CheckTree-4.42-303.el8.noarch", "Installed: qemu-img-15:4.2.0-34.module+el8.3.0+10437+1ca0c2ba.5.x86_64", "Installed: libpaper-1.1.24-22.el8.x86_64", "Installed: adobe-mappings-cmap-deprecated-20171205-3.el8.noarch", "Installed: perl-HTTP-Negotiate-6.01-19.module+el8.3.0+6498+9eecfe51.noarch", "Installed: perl-XML-LibXML-1:2.0132-2.el8.x86_64", "Installed: libgs-9.25-7.el8.x86_64", "Installed: libSM-1.2.3-1.el8.x86_64", "Installed: urw-base35-nimbus-sans-fonts-20170801-10.el8.noarch", "Installed: libidn-1.34-5.el8.x86_64", "Installed: perl-Try-Tiny-0.30-7.module+el8.3.0+6498+9eecfe51.noarch", "Installed: redhat-lsb-cxx-4.1-47.el8.x86_64", "Installed: cups-filters-1.20.0-20.el8.x86_64", "Installed: qemu-kvm-block-rbd-15:4.2.0-34.module+el8.3.0+10437+1ca0c2ba.5.x86_64", "Installed: avahi-0.7-19.el8.x86_64", "Installed: qpdf-libs-7.1.1-10.el8.x86_64", "Installed: urw-base35-gothic-fonts-20170801-10.el8.noarch", "Installed: redhat-lsb-languages-4.1-47.el8.x86_64", "Installed: urw-base35-d050000l-fonts-20170801-10.el8.noarch", "Installed: libXt-1.1.5-12.el8.x86_64", "Installed: perl-Data-Dump-1.23-7.module+el8.3.0+6498+9eecfe51.noarch", "Installed: urw-base35-c059-fonts-20170801-10.el8.noarch", "Installed: redhat-lsb-4.1-47.el8.x86_64", "Installed: librbd1-1:12.2.7-9.el8.x86_64", "Installed: redhat-lsb-submod-multimedia-4.1-47.el8.x86_64", "Installed: mesa-libGLU-9.0.0-15.el8.x86_64", "Installed: urw-base35-fonts-20170801-10.el8.noarch", "Installed: perl-XML-SAX-1.00-1.el8.noarch", "Installed: perl-File-Listing-6.04-17.module+el8.3.0+6498+9eecfe51.noarch", "Installed: perl-Pod-Plainer-1.04-7.el8.noarch", "Installed: libpng12-1.2.57-5.el8.x86_64", "Installed: perl-Class-ISA-0.36-1022.el8.noarch", "Installed: jbig2dec-libs-0.14-4.el8_2.x86_64", "Installed: qemu-kvm-core-15:4.2.0-34.module+el8.3.0+10437+1ca0c2ba.5.x86_64", "Installed: adobe-mappings-cmap-20171205-3.el8.noarch", "Installed: perl-HTTP-Cookies-6.04-2.module 2021-07-31T01:25:12.102 INFO:teuthology.task.ansible.out:+el8.3.0+6498+9eecfe51.noarch", "Installed: cups-ipptool-1:2.2.6-38.el8.x86_64", "Installed: perl-B-Lint-1.20-11.el8.noarch", "Installed: redhat-lsb-printing-4.1-47.el8.x86_64", "Installed: perl-Text-Unidecode-1.30-5.el8.noarch", "Installed: libijs-0.35-5.el8.x86_64", "Installed: adobe-mappings-pdf-20180407-1.el8.noarch", "Installed: urw-base35-standard-symbols-ps-fonts-20170801-10.el8.noarch", "Installed: boost-random-1.66.0-10.el8.x86_64", "Installed: libICE-1.0.9-15.el8.x86_64", "Installed: perl-Text-Soundex-3.05-8.el8.x86_64", "Installed: avahi-glib-0.7-19.el8.x86_64", "Installed: perl-libwww-perl-6.34-1.module+el8.3.0+6498+9eecfe51.noarch", "Installed: perl-Pod-LaTeX-0.61-302.el8.noarch", "Removed: qemu-img-15:4.2.0-34.module+el8.3.0+7976+077be4ec.x86_64", "Removed: qemu-kvm-common-15:4.2.0-34.module+el8.3.0+7976+077be4ec.x86_64", "Removed: qemu-kvm-core-15:4.2.0-34.module+el8.3.0+7976+077be4ec.x86_64"]} 2021-07-31T01:25:12.120 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:12 +0000 (0:00:40.287) 0:06:56.101 ********* 2021-07-31T01:25:23.358 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** ok: [smithi154.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-07-31T01:25:23.381 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:23 +0000 (0:00:11.260) 0:07:07.361 ********* 2021-07-31T01:25:26.168 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** ok: [smithi154.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-07-31T01:25:26.190 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:26 +0000 (0:00:02.809) 0:07:10.170 ********* 2021-07-31T01:25:26.282 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-07-31T01:25:26.284 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:26.306 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:26 +0000 (0:00:00.115) 0:07:10.286 ********* 2021-07-31T01:25:26.655 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2021-07-31T01:25:26.656 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"changed": false, "enabled": false, "name": "firewalld", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "polkit.service sysinit.target basic.target dbus.socket dbus.service system.slice", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "network-pre.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "BusName": "org.fedoraproject.FirewallD1", "CPUAccounting": "no", "CPUAffinity": "", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "ip6tables.service ipset.service iptables.service shutdown.target nftables.service ebtables.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "firewalld - dynamic firewall daemon", "D 2021-07-31T01:25:26.658 INFO:teuthology.task.ansible.out:evicePolicy": "auto", "Documentation": "man:firewalld(1)", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/firewalld (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/firewalld ; argv[]=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/firewalld.service", "FreezerState": "running", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "firewalld.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "mixed", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "Limi 2021-07-31T01:25:26.660 INFO:teuthology.task.ansible.out:tNOFILESoft": "1024", "LimitNPROC": "127237", "LimitNPROCSoft": "127237", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127237", "LimitSIGPENDINGSoft": "127237", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "0", "MemoryAccounting": "yes", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "firewalld.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "dbus.socket sysinit.target system.slice", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "null", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "null", "S 2021-07-31T01:25:26.662 INFO:teuthology.task.ansible.out:tartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestampMonotonic": "0", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "203580", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "dbus", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "UtmpMode": "init", "Wants": "network-pre.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-07-31T01:25:26.678 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:26 +0000 (0:00:00.372) 0:07:10.659 ********* 2021-07-31T01:25:26.998 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ fatal: [smithi154.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service iptables: host"} ...ignoring 2021-07-31T01:25:27.021 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:27 +0000 (0:00:00.341) 0:07:11.001 ********* 2021-07-31T01:25:27.792 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2021-07-31T01:25:27.794 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"changed": false, "configfile": "/etc/selinux/config", "msg": "", "policy": "targeted", "reboot_required": false, "state": "permissive"} 2021-07-31T01:25:27.814 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:27 +0000 (0:00:00.793) 0:07:11.794 ********* 2021-07-31T01:25:27.884 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:27.905 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:27 +0000 (0:00:00.091) 0:07:11.886 ********* 2021-07-31T01:25:27.975 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2021-07-31T01:25:27.976 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:27.996 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:27 +0000 (0:00:00.091) 0:07:11.977 ********* 2021-07-31T01:25:28.075 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2021-07-31T01:25:28.077 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:28.099 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:28 +0000 (0:00:00.102) 0:07:12.079 ********* 2021-07-31T01:25:28.188 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:28.210 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:28 +0000 (0:00:00.111) 0:07:12.191 ********* 2021-07-31T01:25:28.280 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ 2021-07-31T01:25:28.281 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:28.303 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:28 +0000 (0:00:00.092) 0:07:12.283 ********* 2021-07-31T01:25:28.373 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2021-07-31T01:25:28.375 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:28.396 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:28 +0000 (0:00:00.093) 0:07:12.376 ********* 2021-07-31T01:25:28.455 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2021-07-31T01:25:28.478 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:28 +0000 (0:00:00.082) 0:07:12.459 ********* 2021-07-31T01:25:28.550 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2021-07-31T01:25:28.552 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:28.585 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:28 +0000 (0:00:00.106) 0:07:12.565 ********* 2021-07-31T01:25:28.659 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2021-07-31T01:25:28.661 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:28.684 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:28 +0000 (0:00:00.099) 0:07:12.665 ********* 2021-07-31T01:25:28.768 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* skipping: [smithi154.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-07-31T01:25:28.770 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2021-07-31T01:25:28.791 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:28 +0000 (0:00:00.106) 0:07:12.771 ********* 2021-07-31T01:25:28.862 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:28.884 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:28 +0000 (0:00:00.093) 0:07:12.864 ********* 2021-07-31T01:25:28.955 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2021-07-31T01:25:28.976 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:28 +0000 (0:00:00.092) 0:07:12.956 ********* 2021-07-31T01:25:29.056 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:29.078 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:29 +0000 (0:00:00.101) 0:07:13.058 ********* 2021-07-31T01:25:29.146 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:29.169 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:29 +0000 (0:00:00.089) 0:07:13.148 ********* 2021-07-31T01:25:29.237 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:29.261 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:29 +0000 (0:00:00.093) 0:07:13.241 ********* 2021-07-31T01:25:29.341 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-07-31T01:25:29.343 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:29.363 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:29 +0000 (0:00:00.102) 0:07:13.343 ********* 2021-07-31T01:25:29.432 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2021-07-31T01:25:29.434 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:29.454 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:29 +0000 (0:00:00.090) 0:07:13.434 ********* 2021-07-31T01:25:29.522 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-07-31T01:25:29.524 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:29.544 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:29 +0000 (0:00:00.090) 0:07:13.524 ********* 2021-07-31T01:25:29.614 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:29.636 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:29 +0000 (0:00:00.091) 0:07:13.616 ********* 2021-07-31T01:25:29.705 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages with --no-install-recommends] **************** skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:29.726 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:29 +0000 (0:00:00.090) 0:07:13.706 ********* 2021-07-31T01:25:29.795 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2021-07-31T01:25:29.797 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:29.817 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:29 +0000 (0:00:00.090) 0:07:13.797 ********* 2021-07-31T01:25:29.886 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2021-07-31T01:25:29.888 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:29.908 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:29 +0000 (0:00:00.090) 0:07:13.888 ********* 2021-07-31T01:25:29.977 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2021-07-31T01:25:29.978 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:29.998 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:29 +0000 (0:00:00.090) 0:07:13.979 ********* 2021-07-31T01:25:30.057 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-07-31T01:25:30.059 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: poppler-0.66.0-27.el8.x86_64", "Installed: perl-NTLM-1.09-17.module+el8.3.0+6498+9eecfe51.noarch", "Installed: redhat-lsb-desktop-4.1-47.el8.x86_64", "Installed: poppler-utils-0.66.0-27.el8.x86_64", "Installed: perl-Module-Runtime-0.016-2.el8.noarch", "Installed: cups-filters-libs-1.20.0-20.el8.x86_64", "Installed: ghostscript-9.25-7.el8.x86_64", "Installed: qemu-kvm-common-15:4.2.0-34.module+el8.3.0+10437+1ca0c2ba.5.x86_64", "Installed: libXxf86misc-1.0.4-1.el8.x86_64", "Installed: urw-base35-bookman-fonts-20170801-10.el8.noarch", "Installed: boost-program-options-1.66.0-10.el8.x86_64", "Installed: perl-XML-SAX-Base-1.09-4.el8.noarch", "Installed: perl-Module-Pluggable-2:5.2-7.el8.noarch", "Installed: urw-base35-p052-fonts-20170801-10.el8.noarch", "Installed: openjpeg2-2.3.1-6.el8.x86_64", "Installed: liberation-fonts-common-1:2.00.3-7.el8.noarch", "Installed: perl-Net-HTTP-6.17-2.module+el8.3.0+6498+9eecfe51.noarch", "Installed: cups-1:2.2.6-38.el8.x86_64", "Installed: liberation-mono-fonts-1:2.00.3-7.el8.noarch", "Installed: libmcpp-2.7.2-20.el8.x86_64", "Installed: urw-base35-nimbus-mono-ps-fonts-20170801-10.el8.noarch", "Installed: cups-filesystem-1:2.2.6-38.el8.noarch", "Installed: urw-base35-nimbus-roman-fonts-20170801-10.el8.noarch", "Installed: perl-XML-NamespaceSupport-1.12-4.el8.noarch", "Installed: perl-WWW-RobotRules-6.02-18.module+el8.3.0+6498+9eecfe51.noarch", "Installed: libXmu-1.1.3-1.el8.x86_64", "Installed: urw-base35-z003-fonts-20170801-10.el8.noarch", "Installed: xorg-x11-server-utils-7.7-27.el8.x86_64", "Installed: mcpp-2.7.2-20.el8.x86_64", "Installed: google-droid-sans-fonts-20120715-13.el8.noarch", "Installed: urw-base35-fonts-common-20170801-10.el8.noarch", "Installed: librados2-1:12.2.7-9.el8.x86_64", "Installed: lcms2-2.9-2.el8.x86_64", "Installed: poppler-data-0.4.9-1.el8.noarch", "Installed: perl-File-CheckTree-4.42-303.el8.noarch" 2021-07-31T01:25:30.061 INFO:teuthology.task.ansible.out:, "Installed: qemu-img-15:4.2.0-34.module+el8.3.0+10437+1ca0c2ba.5.x86_64", "Installed: libpaper-1.1.24-22.el8.x86_64", "Installed: adobe-mappings-cmap-deprecated-20171205-3.el8.noarch", "Installed: perl-HTTP-Negotiate-6.01-19.module+el8.3.0+6498+9eecfe51.noarch", "Installed: perl-XML-LibXML-1:2.0132-2.el8.x86_64", "Installed: libgs-9.25-7.el8.x86_64", "Installed: libSM-1.2.3-1.el8.x86_64", "Installed: urw-base35-nimbus-sans-fonts-20170801-10.el8.noarch", "Installed: libidn-1.34-5.el8.x86_64", "Installed: perl-Try-Tiny-0.30-7.module+el8.3.0+6498+9eecfe51.noarch", "Installed: redhat-lsb-cxx-4.1-47.el8.x86_64", "Installed: cups-filters-1.20.0-20.el8.x86_64", "Installed: qemu-kvm-block-rbd-15:4.2.0-34.module+el8.3.0+10437+1ca0c2ba.5.x86_64", "Installed: avahi-0.7-19.el8.x86_64", "Installed: qpdf-libs-7.1.1-10.el8.x86_64", "Installed: urw-base35-gothic-fonts-20170801-10.el8.noarch", "Installed: redhat-lsb-languages-4.1-47.el8.x86_64", "Installed: urw-base35-d050000l-fonts-20170801-10.el8.noarch", "Installed: libXt-1.1.5-12.el8.x86_64", "Installed: perl-Data-Dump-1.23-7.module+el8.3.0+6498+9eecfe51.noarch", "Installed: urw-base35-c059-fonts-20170801-10.el8.noarch", "Installed: redhat-lsb-4.1-47.el8.x86_64", "Installed: librbd1-1:12.2.7-9.el8.x86_64", "Installed: redhat-lsb-submod-multimedia-4.1-47.el8.x86_64", "Installed: mesa-libGLU-9.0.0-15.el8.x86_64", "Installed: urw-base35-fonts-20170801-10.el8.noarch", "Installed: perl-XML-SAX-1.00-1.el8.noarch", "Installed: perl-File-Listing-6.04-17.module+el8.3.0+6498+9eecfe51.noarch", "Installed: perl-Pod-Plainer-1.04-7.el8.noarch", "Installed: libpng12-1.2.57-5.el8.x86_64", "Installed: perl-Class-ISA-0.36-1022.el8.noarch", "Installed: jbig2dec-libs-0.14-4.el8_2.x86_64", "Installed: qemu-kvm-core-15:4.2.0-34.module+el8.3.0+10437+1ca0c2ba.5.x86_64", "Installed: adobe-mappings-cmap-20171205-3.el8.noarch", "Installed: perl-HTTP-Cookies-6.04-2.module+el8.3.0+6498+9eecfe51.noarch", "Installed: cups-ipptool-1:2.2.6-38.el8.x86_64", "In 2021-07-31T01:25:30.063 INFO:teuthology.task.ansible.out:stalled: perl-B-Lint-1.20-11.el8.noarch", "Installed: redhat-lsb-printing-4.1-47.el8.x86_64", "Installed: perl-Text-Unidecode-1.30-5.el8.noarch", "Installed: libijs-0.35-5.el8.x86_64", "Installed: adobe-mappings-pdf-20180407-1.el8.noarch", "Installed: urw-base35-standard-symbols-ps-fonts-20170801-10.el8.noarch", "Installed: boost-random-1.66.0-10.el8.x86_64", "Installed: libICE-1.0.9-15.el8.x86_64", "Installed: perl-Text-Soundex-3.05-8.el8.x86_64", "Installed: avahi-glib-0.7-19.el8.x86_64", "Installed: perl-libwww-perl-6.34-1.module+el8.3.0+6498+9eecfe51.noarch", "Installed: perl-Pod-LaTeX-0.61-302.el8.noarch", "Removed: qemu-img-15:4.2.0-34.module+el8.3.0+7976+077be4ec.x86_64", "Removed: qemu-kvm-common-15:4.2.0-34.module+el8.3.0+7976+077be4ec.x86_64", "Removed: qemu-kvm-core-15:4.2.0-34.module+el8.3.0+7976+077be4ec.x86_64"]} 2021-07-31T01:25:30.080 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:30 +0000 (0:00:00.082) 0:07:14.061 ********* 2021-07-31T01:25:30.087 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2021-07-31T01:25:30.088 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:30.110 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:30 +0000 (0:00:00.029) 0:07:14.090 ********* 2021-07-31T01:25:30.179 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2021-07-31T01:25:30.181 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:30.204 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:30 +0000 (0:00:00.094) 0:07:14.184 ********* 2021-07-31T01:25:30.273 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2021-07-31T01:25:30.275 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:30.295 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:30 +0000 (0:00:00.090) 0:07:14.275 ********* 2021-07-31T01:25:30.363 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2021-07-31T01:25:30.364 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:30.387 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:30 +0000 (0:00:00.092) 0:07:14.367 ********* 2021-07-31T01:25:30.446 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:30.471 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:30 +0000 (0:00:00.080) 0:07:14.448 ********* 2021-07-31T01:25:30.537 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-07-31T01:25:30.538 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:30.558 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:30 +0000 (0:00:00.090) 0:07:14.539 ********* 2021-07-31T01:25:30.629 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-07-31T01:25:30.631 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:30.651 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:30 +0000 (0:00:00.092) 0:07:14.631 ********* 2021-07-31T01:25:30.721 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-07-31T01:25:30.722 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:30.750 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:30 +0000 (0:00:00.099) 0:07:14.730 ********* 2021-07-31T01:25:30.822 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-07-31T01:25:30.824 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:30.846 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:30 +0000 (0:00:00.095) 0:07:14.826 ********* 2021-07-31T01:25:30.927 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-07-31T01:25:30.930 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:30.949 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:30 +0000 (0:00:00.103) 0:07:14.930 ********* 2021-07-31T01:25:31.019 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:31.041 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:31 +0000 (0:00:00.091) 0:07:15.021 ********* 2021-07-31T01:25:31.110 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-07-31T01:25:31.112 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:31.131 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:31 +0000 (0:00:00.090) 0:07:15.112 ********* 2021-07-31T01:25:31.200 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-07-31T01:25:31.202 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:31.222 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:31 +0000 (0:00:00.090) 0:07:15.202 ********* 2021-07-31T01:25:31.291 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:31.313 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:31 +0000 (0:00:00.091) 0:07:15.293 ********* 2021-07-31T01:25:31.382 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-07-31T01:25:31.384 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:31.404 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:31 +0000 (0:00:00.090) 0:07:15.384 ********* 2021-07-31T01:25:31.473 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-07-31T01:25:31.475 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:31.499 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:31 +0000 (0:00:00.094) 0:07:15.478 ********* 2021-07-31T01:25:31.621 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-07-31T01:25:31.623 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:31.646 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:31 +0000 (0:00:00.147) 0:07:15.626 ********* 2021-07-31T01:25:31.716 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-07-31T01:25:31.717 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:31.739 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:31 +0000 (0:00:00.093) 0:07:15.719 ********* 2021-07-31T01:25:31.809 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-07-31T01:25:31.811 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:31.831 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:31 +0000 (0:00:00.091) 0:07:15.811 ********* 2021-07-31T01:25:31.900 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:31.922 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:31 +0000 (0:00:00.091) 0:07:15.903 ********* 2021-07-31T01:25:31.991 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-07-31T01:25:31.993 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:32.013 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:32 +0000 (0:00:00.090) 0:07:15.994 ********* 2021-07-31T01:25:32.082 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-07-31T01:25:32.084 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:32.105 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:32 +0000 (0:00:00.091) 0:07:16.085 ********* 2021-07-31T01:25:32.176 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-07-31T01:25:32.177 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:32.198 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:32 +0000 (0:00:00.093) 0:07:16.179 ********* 2021-07-31T01:25:32.289 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:32.320 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:32 +0000 (0:00:00.120) 0:07:16.300 ********* 2021-07-31T01:25:32.390 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-07-31T01:25:32.392 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:32.414 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:32 +0000 (0:00:00.094) 0:07:16.394 ********* 2021-07-31T01:25:32.500 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:32.528 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:32 +0000 (0:00:00.113) 0:07:16.508 ********* 2021-07-31T01:25:32.621 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-07-31T01:25:32.622 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:32.643 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:32 +0000 (0:00:00.115) 0:07:16.623 ********* 2021-07-31T01:25:32.711 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-07-31T01:25:32.713 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:32.735 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:32 +0000 (0:00:00.092) 0:07:16.716 ********* 2021-07-31T01:25:32.795 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-07-31T01:25:32.797 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:32.817 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:32 +0000 (0:00:00.081) 0:07:16.797 ********* 2021-07-31T01:25:32.874 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2021-07-31T01:25:32.876 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:32.896 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:32 +0000 (0:00:00.078) 0:07:16.876 ********* 2021-07-31T01:25:32.965 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:32.987 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:32 +0000 (0:00:00.091) 0:07:16.968 ********* 2021-07-31T01:25:33.057 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-07-31T01:25:33.059 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:33.080 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:33 +0000 (0:00:00.092) 0:07:17.060 ********* 2021-07-31T01:25:33.148 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-07-31T01:25:33.150 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:33.169 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:33 +0000 (0:00:00.089) 0:07:17.150 ********* 2021-07-31T01:25:33.238 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2021-07-31T01:25:33.240 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:33.260 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:33 +0000 (0:00:00.090) 0:07:17.240 ********* 2021-07-31T01:25:33.478 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2021-07-31T01:25:33.480 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:33.500 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:33 +0000 (0:00:00.240) 0:07:17.480 ********* 2021-07-31T01:25:33.569 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:33.590 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:33 +0000 (0:00:00.090) 0:07:17.571 ********* 2021-07-31T01:25:33.659 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:33.682 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:33 +0000 (0:00:00.091) 0:07:17.662 ********* 2021-07-31T01:25:33.750 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2021-07-31T01:25:33.752 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:33.776 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:33 +0000 (0:00:00.093) 0:07:17.755 ********* 2021-07-31T01:25:33.845 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2021-07-31T01:25:33.847 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:33.872 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:33 +0000 (0:00:00.097) 0:07:17.853 ********* 2021-07-31T01:25:33.941 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2021-07-31T01:25:33.943 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:33.966 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:33 +0000 (0:00:00.093) 0:07:17.946 ********* 2021-07-31T01:25:34.035 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:34.057 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:34 +0000 (0:00:00.090) 0:07:18.037 ********* 2021-07-31T01:25:34.125 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2021-07-31T01:25:34.127 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:34.147 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:34 +0000 (0:00:00.090) 0:07:18.127 ********* 2021-07-31T01:25:34.216 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-07-31T01:25:34.218 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2021-07-31T01:25:34.239 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:34 +0000 (0:00:00.091) 0:07:18.219 ********* 2021-07-31T01:25:34.319 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2021-07-31T01:25:34.321 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2021-07-31T01:25:34.330 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2021-07-31T01:25:34.341 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2021-07-31T01:25:34.343 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2021-07-31T01:25:34.353 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2021-07-31T01:25:34.553 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2021-07-31T01:25:34.555 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2021-07-31T01:25:34.579 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:34 +0000 (0:00:00.340) 0:07:18.559 ********* 2021-07-31T01:25:34.817 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2021-07-31T01:25:34.818 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003960", "end": "2021-07-31 01:25:34.789890", "rc": 0, "start": "2021-07-31 01:25:34.785930", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_73aa7e3b960c7ffac669297b6aa86606265edd1b/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_73aa7e3b960c7ffac669297b6aa86606265edd1b/virtualenv/bin/teuthology-lock"]} 2021-07-31T01:25:34.841 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:34 +0000 (0:00:00.262) 0:07:18.821 ********* 2021-07-31T01:25:35.068 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:35.093 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:35 +0000 (0:00:00.251) 0:07:19.073 ********* 2021-07-31T01:25:35.318 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2021-07-31T01:25:35.320 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:35.340 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:35 +0000 (0:00:00.247) 0:07:19.320 ********* 2021-07-31T01:25:35.578 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2021-07-31T01:25:35.580 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:35.603 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:35 +0000 (0:00:00.263) 0:07:19.584 ********* 2021-07-31T01:25:35.676 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:35.705 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:35 +0000 (0:00:00.101) 0:07:19.685 ********* 2021-07-31T01:25:38.491 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* ok: [smithi154.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-07-31T01:25:38.513 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:38 +0000 (0:00:02.808) 0:07:22.493 ********* 2021-07-31T01:25:38.752 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-07-31T01:25:38.753 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930416566272, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227152482, 'block_used': 12942626, 'inode_total': 61054976, 'inode_available': 60986957, 'inode_used': 68019, 'uuid': '782d0384-b480-44b3-9c95-fe95daf886ae'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 227152482, "block_size": 4096, "block_total": 240095108, "block_used": 12942626, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60986957, "inode_total": 61054976, "inode_used": 68019, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930416566272, "size_total": 983429562368, "uuid": "782d0384-b480-44b3-9c95-fe95daf886ae"}} 2021-07-31T01:25:38.775 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:38 +0000 (0:00:00.261) 0:07:22.755 ********* 2021-07-31T01:25:39.193 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2021-07-31T01:25:39.194 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.004990", "end": "2021-07-31 01:25:39.170410", "rc": 0, "start": "2021-07-31 01:25:39.165420", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-07-31T01:25:39.216 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:39 +0000 (0:00:00.441) 0:07:23.197 ********* 2021-07-31T01:25:39.468 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** skipping: [smithi154.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930416566272, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227152482, 'block_used': 12942626, 'inode_total': 61054976, 'inode_available': 60986957, 'inode_used': 68019, 'uuid': '782d0384-b480-44b3-9c95-fe95daf886ae'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227152482, "block_size": 4096, "block_total": 240095108, "block_used": 12942626, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60986957, "inode_total": 61054976, "inode_used": 68019, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930416566272, "size_total": 983429562368, "uuid": "782d0384-b480-44b3-9c95-fe95daf886ae"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:39.491 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:39 +0000 (0:00:00.274) 0:07:23.471 ********* 2021-07-31T01:25:39.562 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2021-07-31T01:25:39.581 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:39 +0000 (0:00:00.090) 0:07:23.561 ********* 2021-07-31T01:25:40.963 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-07-31T01:25:40.965 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-nmRau0B2PUS0HM3DJdbaPCK3xitjSGEajBJVYkAc0f1YFFZGGr1nmRHLFnvNxTqi'], 'uuids': ['271f07ad-d337-4a04-9873-52d8d2fc3514'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.123828", "end": "2021-07-31 01:25:40.936111", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-nmRau0B2PUS0HM3DJdbaPCK3xitjSGEajBJVYkAc0f1YFFZGGr1nmRHLFnvNxTqi"], "labels": [], "masters": [], "uuids": ["271f07ad-d337-4a04-9873-52d8d2fc3514"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-07-31 01:25:39.812283", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-07-31T01:25:42.218 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-2ql69c-8I5v-gsyi-eLjV-ueiV-i0D8-YCbZ2X', 'nvme-INTEL_SSDPEDMD400G4_CVFT6233003B400BGN', 'nvme-nvme.8086-43564654363233333030334234303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPEDMD400G4', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.065742", "end": "2021-07-31 01:25:42.183883", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-2ql69c-8I5v-gsyi-eLjV-ueiV-i0D8-YCbZ2X", "nvme-INTEL_SSDPEDMD400G4_CVFT6233003B400BGN", "nvme-nvme.8086-43564654363233333030334234303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-07-31 01:25:41.118141", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! 2021-07-31T01:25:42.220 INFO:teuthology.task.ansible.out:You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-07-31T01:25:42.357 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2021-07-31T01:25:42.359 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-07-31T01:25:42.396 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:42 +0000 (0:00:02.813) 0:07:26.375 ********* 2021-07-31T01:25:43.446 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** changed: [smithi154.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-nmRau0B2PUS0HM3DJdbaPCK3xitjSGEapWOKThu9qKbfLUv3TsdrqYoQzTDwAp3G'], 'uuids': ['113a62c4-8c36-4003-a463-ebe53f6196e0'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.051820", "end": "2021-07-31 01:25:43.421195", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-nmRau0B2PUS0HM3DJdbaPCK3xitjSGEapWOKThu9qKbfLUv3TsdrqYoQzTDwAp3G"], "labels": [], "masters": [], "uuids": ["113a62c4-8c36-4003-a463-ebe53f6196e0"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-07-31 01:25:42.369375", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk usi 2021-07-31T01:25:43.448 INFO:teuthology.task.ansible.out:ng fdisk or", "other utilities."]} 2021-07-31T01:25:44.680 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-nmRau0B2PUS0HM3DJdbaPCK3xitjSGEaAQ3xfjtvxlfBYHJRIOW5mUBjkYyl8LK7'], 'uuids': ['6311b30f-2dde-402d-827f-081768bdcc30'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.047416", "end": "2021-07-31 01:25:44.653169", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-nmRau0B2PUS0HM3DJdbaPCK3xitjSGEaAQ3xfjtvxlfBYHJRIOW5mUBjkYyl8LK7"], "labels": [], "masters": [], "uuids": ["6311b30f-2dde-402d-827f-081768bdcc30"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-07-31 01:25:43.605753", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-07-31T01:25:45.490 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-07-31T01:25:45.492 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-07-31T01:25:45.511 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:45 +0000 (0:00:03.116) 0:07:29.492 ********* 2021-07-31T01:25:45.580 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-07-31T01:25:45.582 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:45.602 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:45 +0000 (0:00:00.090) 0:07:29.582 ********* 2021-07-31T01:25:45.909 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-07-31T01:25:45.911 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-nmRau0B2PUS0HM3DJdbaPCK3xitjSGEaxSuzznubKQDRA6p3SY62Zo0B3YHyw50e'], 'uuids': ['da6d6252-629a-4b6e-8d09-734a9e63b926'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.046706", "end": "2021-07-31 01:25:45.878481", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-nmRau0B2PUS0HM3DJdbaPCK3xitjSGEaxSuzznubKQDRA6p3SY62Zo0B3YHyw50e"], "labels": [], "masters": [], "uuids": ["da6d6252-629a-4b6e-8d09-734a9e63b926"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-07-31 01:25:44.831775", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-07-31T01:25:45.911 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => 2021-07-31T01:25:45.913 INFO:teuthology.task.ansible.out:(item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XFNL', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5XFNL', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XFNL', 'scsi-35000c50092924855', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5XFNL', 'wwn-0x5000c50092924855'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W5XFNL', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c50092924855', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XFNL-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5XFNL-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XFNL-part1', 'scsi-35000c50092924855-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5XFNL-part1', 'wwn-0x5000c50092924855-part1'], 'uuids': ['782d0384-b480-44b3-9c95-fe95daf886ae'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': '782d0384-b480-44b3-9c95-fe95daf886ae', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XFNL", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5XFNL", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XFNL", "scsi-35000c50092924855", "scsi-SATA_ST1000NM0033-9ZM_Z1W5XFNL", "wwn-0x5000c50092924855"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XFNL-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5XFNL-part1", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XFNL-part1", "scsi-35000c50092924855-part1 2021-07-31T01:25:45.914 INFO:teuthology.task.ansible.out:", "scsi-SATA_ST1000NM0033-9ZM_Z1W5XFNL-part1", "wwn-0x5000c50092924855-part1"], "labels": [], "masters": [], "uuids": ["782d0384-b480-44b3-9c95-fe95daf886ae"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "782d0384-b480-44b3-9c95-fe95daf886ae"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XFNL", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50092924855"}}, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:45.954 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2021-07-31T01:25:45.955 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "enabled": false, "name": "firewalld", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "polkit.service dbus.service sysinit.target system.slice basic.target dbus.socket", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "shutdown.target network-pre.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "BusName": "org.fedoraproject.FirewallD1", "CPUAccounting": "no", "CPUAffinity": "", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "ebtables.service ip6tables.service ipset.service shutdown.target iptables.service nftables.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "firewalld - dynamic firewall daemon", "D 2021-07-31T01:25:45.957 INFO:teuthology.task.ansible.out:evicePolicy": "auto", "Documentation": "man:firewalld(1)", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/firewalld (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/firewalld ; argv[]=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/firewalld.service", "FreezerState": "running", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "firewalld.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "mixed", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "Limi 2021-07-31T01:25:45.959 INFO:teuthology.task.ansible.out:tNOFILESoft": "1024", "LimitNPROC": "127262", "LimitNPROCSoft": "127262", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127262", "LimitSIGPENDINGSoft": "127262", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "0", "MemoryAccounting": "yes", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "firewalld.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice dbus.socket sysinit.target", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "null", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "null", "S 2021-07-31T01:25:45.960 INFO:teuthology.task.ansible.out:tartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestampMonotonic": "0", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "203619", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "dbus", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "UtmpMode": "init", "Wants": "network-pre.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-07-31T01:25:45.978 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:45 +0000 (0:00:00.376) 0:07:29.959 ********* 2021-07-31T01:25:46.317 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ fatal: [smithi045.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service iptables: host"} ...ignoring 2021-07-31T01:25:46.339 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:46 +0000 (0:00:00.360) 0:07:30.319 ********* 2021-07-31T01:25:46.668 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2021-07-31T01:25:46.669 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "configfile": "/etc/selinux/config", "msg": "", "policy": "targeted", "reboot_required": false, "state": "permissive"} 2021-07-31T01:25:46.689 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:46 +0000 (0:00:00.350) 0:07:30.669 ********* 2021-07-31T01:25:46.767 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:46.789 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:46 +0000 (0:00:00.100) 0:07:30.769 ********* 2021-07-31T01:25:46.857 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2021-07-31T01:25:46.969 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Saturday 31 July 2021 01:25:46 +0000 (0:00:00.090) 0:07:30.860 ********* 2021-07-31T01:25:46.979 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2021-07-31T01:25:46.980 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:47.000 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:46 +0000 (0:00:00.120) 0:07:30.980 ********* 2021-07-31T01:25:47.068 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** 2021-07-31T01:25:47.069 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:47.089 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:47 +0000 (0:00:00.088) 0:07:31.069 ********* 2021-07-31T01:25:47.156 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ 2021-07-31T01:25:47.158 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:47.178 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:47 +0000 (0:00:00.088) 0:07:31.158 ********* 2021-07-31T01:25:47.245 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:47.267 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:47 +0000 (0:00:00.089) 0:07:31.247 ********* 2021-07-31T01:25:47.295 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-07-31T01:25:47.297 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-nmRau0B2PUS0HM3DJdbaPCK3xitjSGEakjfrMD4M7WVRYpiyzlRrdR1KWQzn43MF'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.049188", "end": "2021-07-31 01:25:47.261328", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-nmRau0B2PUS0HM3DJdbaPCK3xitjSGEakjfrMD4M7WVRYpiyzlRrdR1KWQzn43MF"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-07-31 01:25:46.212140", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-07-31T01:25:47.325 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:47 +0000 (0:00:00.057) 0:07:31.305 ********* 2021-07-31T01:25:47.343 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2021-07-31T01:25:47.366 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:47 +0000 (0:00:00.041) 0:07:31.346 ********* 2021-07-31T01:25:47.445 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2021-07-31T01:25:47.446 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:47.467 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:47 +0000 (0:00:00.100) 0:07:31.447 ********* 2021-07-31T01:25:47.535 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2021-07-31T01:25:47.537 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:47.556 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:47 +0000 (0:00:00.089) 0:07:31.537 ********* 2021-07-31T01:25:47.635 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2021-07-31T01:25:47.637 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2021-07-31T01:25:47.661 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:47 +0000 (0:00:00.104) 0:07:31.642 ********* 2021-07-31T01:25:47.730 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:47.754 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:47 +0000 (0:00:00.090) 0:07:31.733 ********* 2021-07-31T01:25:47.760 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-07-31T01:25:47.762 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-nmRau0B2PUS0HM3DJdbaPCK3xitjSGEajBJVYkAc0f1YFFZGGr1nmRHLFnvNxTqi'], 'uuids': ['271f07ad-d337-4a04-9873-52d8d2fc3514'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.140389", "end": "2021-07-31 01:25:47.711689", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-nmRau0B2PUS0HM3DJdbaPCK3xitjSGEajBJVYkAc0f1YFFZGGr1nmRHLFnvNxTqi"], "labels": [], "masters": [], "uuids": ["271f07ad-d337-4a04-9873-52d8d2fc3514"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-07-31 01:25:47.571300", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.137669 s, 838 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.137669 s, 838 MB/s"], "stdout": "", "stdout_lines": []} 2021-07-31T01:25:47.814 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2021-07-31T01:25:47.837 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:47 +0000 (0:00:00.084) 0:07:31.817 ********* 2021-07-31T01:25:47.909 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2021-07-31T01:25:47.910 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:47.931 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:47 +0000 (0:00:00.093) 0:07:31.911 ********* 2021-07-31T01:25:47.948 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-07-31T01:25:47.950 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-2ql69c-8I5v-gsyi-eLjV-ueiV-i0D8-YCbZ2X', 'nvme-INTEL_SSDPEDMD400G4_CVFT6233003B400BGN', 'nvme-nvme.8086-43564654363233333030334234303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPEDMD400G4', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.037613", "end": "2021-07-31 01:25:47.922249", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-2ql69c-8I5v-gsyi-eLjV-ueiV-i0D8-YCbZ2X", "nvme-INTEL_SSDPEDMD400G4_CVFT6233003B400BGN", "nvme-nvme.8086-43564654363233333030334234303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-07-31 01:25:47.884636", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0350024 s, 3.3 GB/s", "std 2021-07-31T01:25:47.951 INFO:teuthology.task.ansible.out:err_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0350024 s, 3.3 GB/s"], "stdout": "", "stdout_lines": []} 2021-07-31T01:25:47.992 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:48.014 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:48 +0000 (0:00:00.082) 0:07:31.994 ********* 2021-07-31T01:25:48.082 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2021-07-31T01:25:48.084 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:48.104 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:48 +0000 (0:00:00.090) 0:07:32.085 ********* 2021-07-31T01:25:48.173 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-07-31T01:25:48.175 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:48.194 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:48 +0000 (0:00:00.089) 0:07:32.175 ********* 2021-07-31T01:25:48.263 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2021-07-31T01:25:48.264 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** changed: [smithi154.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-nmRau0B2PUS0HM3DJdbaPCK3xitjSGEapWOKThu9qKbfLUv3TsdrqYoQzTDwAp3G'], 'uuids': ['113a62c4-8c36-4003-a463-ebe53f6196e0'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.146124", "end": "2021-07-31 01:25:48.241013", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-nmRau0B2PUS0HM3DJdbaPCK3xitjSGEapWOKThu9qKbfLUv3TsdrqYoQzTDwAp3G"], "labels": [], "masters": [], "uuids": ["113a62c4-8c36-4003-a463-ebe53f6196e0"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-07-31 01:25:48.094889", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.143447 s, 804 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.143447 s, 804 MB/s"], "stdout": "", "stdout_lines": []} 2021-07-31T01:25:48.286 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:48 +0000 (0:00:00.091) 0:07:32.266 ********* 2021-07-31T01:25:48.354 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-07-31T01:25:48.356 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:48.376 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:48 +0000 (0:00:00.090) 0:07:32.356 ********* 2021-07-31T01:25:48.445 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2021-07-31T01:25:48.447 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:48.467 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:48 +0000 (0:00:00.091) 0:07:32.447 ********* 2021-07-31T01:25:48.536 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages with --no-install-recommends] **************** 2021-07-31T01:25:48.537 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:48.558 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:48 +0000 (0:00:00.090) 0:07:32.538 ********* 2021-07-31T01:25:48.575 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-07-31T01:25:48.576 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-nmRau0B2PUS0HM3DJdbaPCK3xitjSGEaAQ3xfjtvxlfBYHJRIOW5mUBjkYyl8LK7'], 'uuids': ['6311b30f-2dde-402d-827f-081768bdcc30'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.141107", "end": "2021-07-31 01:25:48.552467", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-nmRau0B2PUS0HM3DJdbaPCK3xitjSGEaAQ3xfjtvxlfBYHJRIOW5mUBjkYyl8LK7"], "labels": [], "masters": [], "uuids": ["6311b30f-2dde-402d-827f-081768bdcc30"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-07-31 01:25:48.411360", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.138465 s, 833 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.138465 s, 833 MB/s"], "stdout": "", "stdout_lines": []} 2021-07-31T01:25:48.639 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:48.661 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:48 +0000 (0:00:00.103) 0:07:32.641 ********* 2021-07-31T01:25:48.732 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:48.760 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:48 +0000 (0:00:00.098) 0:07:32.740 ********* 2021-07-31T01:25:48.831 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2021-07-31T01:25:48.833 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:48.853 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:48 +0000 (0:00:00.092) 0:07:32.833 ********* 2021-07-31T01:25:48.904 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-07-31T01:25:48.906 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-nmRau0B2PUS0HM3DJdbaPCK3xitjSGEaxSuzznubKQDRA6p3SY62Zo0B3YHyw50e'], 'uuids': ['da6d6252-629a-4b6e-8d09-734a9e63b926'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.150563", "end": "2021-07-31 01:25:48.878321", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-nmRau0B2PUS0HM3DJdbaPCK3xitjSGEaxSuzznubKQDRA6p3SY62Zo0B3YHyw50e"], "labels": [], "masters": [], "uuids": ["da6d6252-629a-4b6e-8d09-734a9e63b926"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-07-31 01:25:48.727758", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.147955 s, 780 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.147955 s, 780 MB/s"], "stdout": "", "stdout_lines": []} 2021-07-31T01:25:48.916 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XFNL', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5XFNL', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XFNL', 'scsi-35000c50092924855', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5XFNL', 'wwn-0x5000c50092924855'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W5XFNL', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c50092924855', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XFNL-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5XFNL-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XFNL-part1', 'scsi-35000c50092924855-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5XFNL-part1', 'wwn-0x5000c50092924855-part1'], 'uuids': ['782d0384-b480-44b3-9c95-fe95daf886ae'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': '782d0384-b480-44b3-9c95-fe95daf886ae', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XFNL", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5XFNL", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XFNL", "scsi-35000c50092924855", "scsi-SATA_ST1000NM0033-9ZM_Z1W5XFNL", "wwn-0x5000c50092924855"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XFNL-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5XFNL-part1", "scsi-1ATA_ST1000NM0033-9ZM173_ 2021-07-31T01:25:48.918 INFO:teuthology.task.ansible.out:Z1W5XFNL-part1", "scsi-35000c50092924855-part1", "scsi-SATA_ST1000NM0033-9ZM_Z1W5XFNL-part1", "wwn-0x5000c50092924855-part1"], "labels": [], "masters": [], "uuids": ["782d0384-b480-44b3-9c95-fe95daf886ae"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "782d0384-b480-44b3-9c95-fe95daf886ae"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XFNL", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50092924855"}}, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:48.938 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2021-07-31T01:25:48.941 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:48.964 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:48 +0000 (0:00:00.111) 0:07:32.944 ********* 2021-07-31T01:25:49.044 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:49.067 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:49 +0000 (0:00:00.103) 0:07:33.047 ********* 2021-07-31T01:25:49.137 INFO:teuthology.task.ansible.out: 2021-07-31T01:25:49.138 INFO:teuthology.task.ansible.out:TASK [testnode : Set up static IP in /etc/hosts.] ****************************** skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:49.161 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:49 +0000 (0:00:00.093) 0:07:33.141 ********* 2021-07-31T01:25:49.223 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-07-31T01:25:49.224 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-nmRau0B2PUS0HM3DJdbaPCK3xitjSGEakjfrMD4M7WVRYpiyzlRrdR1KWQzn43MF'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.139514", "end": "2021-07-31 01:25:49.198175", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-nmRau0B2PUS0HM3DJdbaPCK3xitjSGEakjfrMD4M7WVRYpiyzlRrdR1KWQzn43MF"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-07-31 01:25:49.058661", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.136864 s, 843 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.136864 s, 843 MB/s"], "stdout": "", "stdout_lines": []} 2021-07-31T01:25:49.230 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2021-07-31T01:25:49.232 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:49.253 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:49 +0000 (0:00:00.092) 0:07:33.233 ********* 2021-07-31T01:25:49.275 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:49 +0000 (0:00:00.022) 0:07:33.256 ********* 2021-07-31T01:25:49.325 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-07-31T01:25:49.327 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:49.349 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:49 +0000 (0:00:00.073) 0:07:33.329 ********* 2021-07-31T01:25:49.419 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-07-31T01:25:49.420 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:49.441 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:49 +0000 (0:00:00.091) 0:07:33.421 ********* 2021-07-31T01:25:49.509 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-07-31T01:25:49.511 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:49.531 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:49 +0000 (0:00:00.089) 0:07:33.511 ********* 2021-07-31T01:25:49.568 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2021-07-31T01:25:49.570 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.060165", "end": "2021-07-31 01:25:49.493663", "rc": 0, "start": "2021-07-31 01:25:49.433498", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-07-31T01:25:49.592 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:49 +0000 (0:00:00.060) 0:07:33.572 ********* 2021-07-31T01:25:49.610 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:49.632 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:49 +0000 (0:00:00.040) 0:07:33.612 ********* 2021-07-31T01:25:49.721 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-07-31T01:25:49.722 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:49.742 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:49 +0000 (0:00:00.110) 0:07:33.723 ********* 2021-07-31T01:25:49.811 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-07-31T01:25:49.813 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:49.834 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:49 +0000 (0:00:00.091) 0:07:33.814 ********* 2021-07-31T01:25:49.841 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2021-07-31T01:25:49.842 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.051611", "end": "2021-07-31 01:25:49.803379", "rc": 0, "start": "2021-07-31 01:25:49.751768", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-07-31T01:25:49.867 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:49 +0000 (0:00:00.032) 0:07:33.847 ********* 2021-07-31T01:25:49.905 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-07-31T01:25:49.907 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:49.939 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:49 +0000 (0:00:00.071) 0:07:33.919 ********* 2021-07-31T01:25:49.949 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2021-07-31T01:25:50.020 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:50 +0000 (0:00:00.080) 0:07:34.000 ********* 2021-07-31T01:25:50.027 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-07-31T01:25:50.028 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:50.050 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:50 +0000 (0:00:00.030) 0:07:34.030 ********* 2021-07-31T01:25:50.057 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2021-07-31T01:25:50.058 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:50.083 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:50 +0000 (0:00:00.032) 0:07:34.063 ********* 2021-07-31T01:25:50.090 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-07-31T01:25:50.092 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:50.116 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:50 +0000 (0:00:00.033) 0:07:34.096 ********* 2021-07-31T01:25:50.123 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2021-07-31T01:25:50.125 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:50.150 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:50 +0000 (0:00:00.033) 0:07:34.130 ********* 2021-07-31T01:25:50.156 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-07-31T01:25:50.158 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:50.180 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:50 +0000 (0:00:00.030) 0:07:34.160 ********* 2021-07-31T01:25:50.186 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-07-31T01:25:50.189 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:50.214 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:50 +0000 (0:00:00.033) 0:07:34.194 ********* 2021-07-31T01:25:50.221 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-07-31T01:25:50.223 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:50.247 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:50 +0000 (0:00:00.033) 0:07:34.227 ********* 2021-07-31T01:25:50.255 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2021-07-31T01:25:50.257 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:50.281 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:50 +0000 (0:00:00.034) 0:07:34.261 ********* 2021-07-31T01:25:50.288 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-07-31T01:25:50.290 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:50.317 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:50 +0000 (0:00:00.035) 0:07:34.297 ********* 2021-07-31T01:25:50.349 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-07-31T01:25:50.351 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:50.373 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:50 +0000 (0:00:00.055) 0:07:34.353 ********* 2021-07-31T01:25:50.415 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-07-31T01:25:50.417 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:50.446 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:50 +0000 (0:00:00.073) 0:07:34.426 ********* 2021-07-31T01:25:50.469 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2021-07-31T01:25:50.470 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-07-31T01:25:50.493 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:50 +0000 (0:00:00.047) 0:07:34.474 ********* 2021-07-31T01:25:50.500 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-07-31T01:25:50.502 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:50.527 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:50 +0000 (0:00:00.033) 0:07:34.507 ********* 2021-07-31T01:25:50.534 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-07-31T01:25:50.536 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930416566272, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227152482, 'block_used': 12942626, 'inode_total': 61054976, 'inode_available': 60986957, 'inode_used': 68019, 'uuid': '782d0384-b480-44b3-9c95-fe95daf886ae'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227152482, "block_size": 4096, "block_total": 240095108, "block_used": 12942626, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60986957, "inode_total": 61054976, "inode_used": 68019, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930416566272, "size_total": 983429562368, "uuid": "782d0384-b480-44b3-9c95-fe95daf886ae"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:50.569 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:50 +0000 (0:00:00.041) 0:07:34.549 ********* 2021-07-31T01:25:50.580 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-07-31T01:25:50.582 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:50.617 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:50 +0000 (0:00:00.048) 0:07:34.597 ********* 2021-07-31T01:25:50.629 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2021-07-31T01:25:50.631 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:50.657 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:50 +0000 (0:00:00.039) 0:07:34.637 ********* 2021-07-31T01:25:50.664 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-07-31T01:25:50.666 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:50.698 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:50 +0000 (0:00:00.041) 0:07:34.678 ********* 2021-07-31T01:25:50.708 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-07-31T01:25:50.711 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:50.733 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:50 +0000 (0:00:00.035) 0:07:34.714 ********* 2021-07-31T01:25:50.740 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:50.765 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:50 +0000 (0:00:00.032) 0:07:34.746 ********* 2021-07-31T01:25:50.772 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:50.796 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:50 +0000 (0:00:00.030) 0:07:34.776 ********* 2021-07-31T01:25:50.804 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-07-31T01:25:50.806 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:50.830 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:50 +0000 (0:00:00.034) 0:07:34.810 ********* 2021-07-31T01:25:50.838 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2021-07-31T01:25:50.839 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:50.864 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:50 +0000 (0:00:00.033) 0:07:34.844 ********* 2021-07-31T01:25:50.874 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-07-31T01:25:50.876 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:50.906 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:50 +0000 (0:00:00.042) 0:07:34.887 ********* 2021-07-31T01:25:50.917 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2021-07-31T01:25:50.919 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:50.943 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:50 +0000 (0:00:00.036) 0:07:34.923 ********* 2021-07-31T01:25:50.950 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-07-31T01:25:50.952 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:50.974 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:50 +0000 (0:00:00.031) 0:07:34.954 ********* 2021-07-31T01:25:50.981 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2021-07-31T01:25:50.983 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:51.005 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:51 +0000 (0:00:00.031) 0:07:34.986 ********* 2021-07-31T01:25:51.013 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-07-31T01:25:51.014 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:51.038 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:51 +0000 (0:00:00.031) 0:07:35.018 ********* 2021-07-31T01:25:51.067 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-07-31T01:25:51.069 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:51.090 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:51 +0000 (0:00:00.052) 0:07:35.070 ********* 2021-07-31T01:25:51.118 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-07-31T01:25:51.120 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:51.140 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:51 +0000 (0:00:00.049) 0:07:35.120 ********* 2021-07-31T01:25:51.180 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-07-31T01:25:51.182 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:51.202 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:51 +0000 (0:00:00.062) 0:07:35.183 ********* 2021-07-31T01:25:51.230 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2021-07-31T01:25:51.232 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:51.253 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:51 +0000 (0:00:00.050) 0:07:35.233 ********* 2021-07-31T01:25:51.282 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2021-07-31T01:25:51.284 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:51.305 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:51 +0000 (0:00:00.051) 0:07:35.285 ********* 2021-07-31T01:25:51.343 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-07-31T01:25:51.344 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:51.370 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:51 +0000 (0:00:00.064) 0:07:35.350 ********* 2021-07-31T01:25:51.397 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-07-31T01:25:51.399 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:51.420 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:51 +0000 (0:00:00.049) 0:07:35.400 ********* 2021-07-31T01:25:51.447 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2021-07-31T01:25:51.449 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:51.469 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:51 +0000 (0:00:00.049) 0:07:35.449 ********* 2021-07-31T01:25:51.496 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2021-07-31T01:25:51.498 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:51.518 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:51 +0000 (0:00:00.048) 0:07:35.498 ********* 2021-07-31T01:25:51.545 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2021-07-31T01:25:51.547 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:51.567 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:51 +0000 (0:00:00.049) 0:07:35.548 ********* 2021-07-31T01:25:51.584 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2021-07-31T01:25:51.586 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2021-07-31T01:25:51.609 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:51 +0000 (0:00:00.041) 0:07:35.589 ********* 2021-07-31T01:25:51.616 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2021-07-31T01:25:51.617 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:51.640 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:51 +0000 (0:00:00.031) 0:07:35.620 ********* 2021-07-31T01:25:51.667 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2021-07-31T01:25:51.669 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:51.689 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:51 +0000 (0:00:00.049) 0:07:35.670 ********* 2021-07-31T01:25:51.716 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2021-07-31T01:25:51.718 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:51.739 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:51 +0000 (0:00:00.049) 0:07:35.719 ********* 2021-07-31T01:25:51.766 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2021-07-31T01:25:51.768 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:51.789 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:51 +0000 (0:00:00.050) 0:07:35.769 ********* 2021-07-31T01:25:51.816 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2021-07-31T01:25:51.818 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:51.840 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:51 +0000 (0:00:00.050) 0:07:35.820 ********* 2021-07-31T01:25:51.868 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2021-07-31T01:25:51.870 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:51.890 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:51 +0000 (0:00:00.049) 0:07:35.870 ********* 2021-07-31T01:25:51.921 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-07-31T01:25:51.922 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2021-07-31T01:25:51.943 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:51 +0000 (0:00:00.053) 0:07:35.924 ********* 2021-07-31T01:25:51.982 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2021-07-31T01:25:51.983 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2021-07-31T01:25:51.993 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2021-07-31T01:25:52.004 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2021-07-31T01:25:52.006 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2021-07-31T01:25:52.015 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2021-07-31T01:25:52.018 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2021-07-31T01:25:52.026 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2021-07-31T01:25:52.053 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:52 +0000 (0:00:00.109) 0:07:36.033 ********* 2021-07-31T01:25:52.112 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-07-31T01:25:52.114 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-07-31T01:25:52.249 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2021-07-31T01:25:52.251 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003096", "end": "2021-07-31 01:25:52.228646", "rc": 0, "start": "2021-07-31 01:25:52.225550", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_73aa7e3b960c7ffac669297b6aa86606265edd1b/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_73aa7e3b960c7ffac669297b6aa86606265edd1b/virtualenv/bin/teuthology-lock"]} 2021-07-31T01:25:52.272 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:52 +0000 (0:00:00.219) 0:07:36.253 ********* 2021-07-31T01:25:52.299 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2021-07-31T01:25:52.301 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:52.323 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:52 +0000 (0:00:00.050) 0:07:36.304 ********* 2021-07-31T01:25:52.350 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2021-07-31T01:25:52.353 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:52.372 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:52 +0000 (0:00:00.049) 0:07:36.353 ********* 2021-07-31T01:25:52.410 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2021-07-31T01:25:52.412 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:52.436 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:52 +0000 (0:00:00.063) 0:07:36.416 ********* 2021-07-31T01:25:52.443 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-07-31T01:25:52.445 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-07-31T01:25:52.456 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2021-07-31T01:25:52.458 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:52.479 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:52 +0000 (0:00:00.043) 0:07:36.459 ********* 2021-07-31T01:25:52.746 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-07-31T01:25:52.748 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-07-31T01:25:53.072 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-07-31T01:25:53.394 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2021-07-31T01:25:53.419 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:53 +0000 (0:00:00.939) 0:07:37.399 ********* 2021-07-31T01:25:53.614 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-07-31T01:25:53.616 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2021-07-31T01:25:53.637 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:53 +0000 (0:00:00.218) 0:07:37.618 ********* 2021-07-31T01:25:53.850 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-07-31T01:25:53.853 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-07-31T01:25:54.024 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-07-31T01:25:54.203 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-07-31T01:25:54.371 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-07-31T01:25:54.382 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:54.408 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:54 +0000 (0:00:00.770) 0:07:38.389 ********* 2021-07-31T01:25:54.439 INFO:teuthology.task.ansible.out: TASK [testnode : Create filesystems] ******************************************* 2021-07-31T01:25:54.442 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:54.463 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:54 +0000 (0:00:00.054) 0:07:38.443 ********* 2021-07-31T01:25:54.489 INFO:teuthology.task.ansible.out: TASK [testnode : Mount filesystems] ******************************************** 2021-07-31T01:25:54.491 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:54.512 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:54 +0000 (0:00:00.049) 0:07:38.492 ********* 2021-07-31T01:25:54.708 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2021-07-31T01:25:54.711 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "secontext": "unconfined_u:object_r:var_lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-07-31T01:25:54.740 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:54 +0000 (0:00:00.228) 0:07:38.720 ********* 2021-07-31T01:25:54.767 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2021-07-31T01:25:54.769 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:54.794 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:54 +0000 (0:00:00.053) 0:07:38.774 ********* 2021-07-31T01:25:55.196 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2021-07-31T01:25:55.197 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => {"changed": true} 2021-07-31T01:25:55.219 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:55 +0000 (0:00:00.425) 0:07:39.200 ********* 2021-07-31T01:25:55.454 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2021-07-31T01:25:55.456 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-07-31T01:25:55.478 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:55 +0000 (0:00:00.258) 0:07:39.458 ********* 2021-07-31T01:25:55.517 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ ok: [smithi045.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930416504832, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227152467, 'block_used': 12942641, 'inode_total': 61054976, 'inode_available': 60986953, 'inode_used': 68023, 'uuid': '782d0384-b480-44b3-9c95-fe95daf886ae'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 227152467, "block_size": 4096, "block_total": 240095108, "block_used": 12942641, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60986953, "inode_total": 61054976, "inode_used": 68023, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930416504832, "size_total": 983429562368, "uuid": "782d0384-b480-44b3-9c95-fe95daf886ae"}} 2021-07-31T01:25:55.540 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:55 +0000 (0:00:00.061) 0:07:39.520 ********* 2021-07-31T01:25:55.672 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2021-07-31T01:25:55.673 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2021-07-31T01:25:55.695 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:55 +0000 (0:00:00.155) 0:07:39.675 ********* 2021-07-31T01:25:55.722 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd] ****************************************** 2021-07-31T01:25:55.724 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:55.744 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:55 +0000 (0:00:00.049) 0:07:39.725 ********* 2021-07-31T01:25:55.751 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2021-07-31T01:25:55.753 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.005306", "end": "2021-07-31 01:25:55.714873", "rc": 0, "start": "2021-07-31 01:25:55.709567", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-07-31T01:25:55.775 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:55 +0000 (0:00:00.030) 0:07:39.755 ********* 2021-07-31T01:25:55.781 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd conf file] ******************************** 2021-07-31T01:25:55.783 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:55.805 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:55 +0000 (0:00:00.030) 0:07:39.785 ********* 2021-07-31T01:25:55.812 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2021-07-31T01:25:55.814 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930416504832, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227152467, 'block_used': 12942641, 'inode_total': 61054976, 'inode_available': 60986953, 'inode_used': 68023, 'uuid': '782d0384-b480-44b3-9c95-fe95daf886ae'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227152467, "block_size": 4096, "block_total": 240095108, "block_used": 12942641, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60986953, "inode_total": 61054976, "inode_used": 68023, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930416504832, "size_total": 983429562368, "uuid": "782d0384-b480-44b3-9c95-fe95daf886ae"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:55.837 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:55 +0000 (0:00:00.032) 0:07:39.818 ********* 2021-07-31T01:25:55.844 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-07-31T01:25:55.846 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:55.867 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:55 +0000 (0:00:00.029) 0:07:39.847 ********* 2021-07-31T01:25:55.873 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2021-07-31T01:25:55.897 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:55 +0000 (0:00:00.030) 0:07:39.877 ********* 2021-07-31T01:25:55.903 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-07-31T01:25:55.905 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:55.927 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:55 +0000 (0:00:00.030) 0:07:39.907 ********* 2021-07-31T01:25:55.954 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2021-07-31T01:25:55.956 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:55.976 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:55 +0000 (0:00:00.049) 0:07:39.957 ********* 2021-07-31T01:25:56.003 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2021-07-31T01:25:56.005 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:56.025 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:56 +0000 (0:00:00.048) 0:07:40.005 ********* 2021-07-31T01:25:56.052 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2021-07-31T01:25:56.054 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:56.074 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:56 +0000 (0:00:00.048) 0:07:40.054 ********* 2021-07-31T01:25:56.101 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2021-07-31T01:25:56.102 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:56.129 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:56 +0000 (0:00:00.054) 0:07:40.109 ********* 2021-07-31T01:25:56.155 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2021-07-31T01:25:56.157 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:56.177 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:56 +0000 (0:00:00.048) 0:07:40.157 ********* 2021-07-31T01:25:56.204 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2021-07-31T01:25:56.206 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:56.226 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:56 +0000 (0:00:00.049) 0:07:40.207 ********* 2021-07-31T01:25:56.253 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:56.275 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:56 +0000 (0:00:00.048) 0:07:40.255 ********* 2021-07-31T01:25:56.302 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2021-07-31T01:25:56.304 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:56.324 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:56 +0000 (0:00:00.048) 0:07:40.304 ********* 2021-07-31T01:25:56.350 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2021-07-31T01:25:56.352 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:56.373 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:56 +0000 (0:00:00.048) 0:07:40.353 ********* 2021-07-31T01:25:56.399 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2021-07-31T01:25:56.401 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:56.421 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:56 +0000 (0:00:00.048) 0:07:40.402 ********* 2021-07-31T01:25:56.448 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2021-07-31T01:25:56.450 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:56.470 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:56 +0000 (0:00:00.048) 0:07:40.450 ********* 2021-07-31T01:25:56.934 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** ok: [smithi154.front.sepia.ceph.com] => {"changed": false, "checksum": "a7ecef00073feab34f868065d34e9dac55134241", "dest": "/etc/chrony.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/chrony.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 215, "state": "file", "uid": 0} 2021-07-31T01:25:56.957 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:56 +0000 (0:00:00.485) 0:07:40.936 ********* 2021-07-31T01:25:57.256 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2021-07-31T01:25:57.257 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "chronyd", "state": "started", "status": {"ActiveEnterTimestamp": "Sat 2021-07-31 01:17:13 UTC", "ActiveEnterTimestampMonotonic": "15540312", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "ntpdate.service system.slice ntpd.service -.mount sntp.service basic.target systemd-tmpfiles-setup.service tmp.mount systemd-journald.socket sysinit.target", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Sat 2021-07-31 01:17:11 UTC", "AssertTimestampMonotonic": "14420304", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Sat 2021-07-31 01:17:11 UTC", "ConditionTimestampMonotonic": "14420251", "ConfigurationDirectoryMode": "0755", "Conflicts": "ntpd.service shutdown.target systemd-timesyncd.service", "ControlGro 2021-07-31T01:25:57.258 INFO:teuthology.task.ansible.out:up": "/system.slice/chronyd.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "NTP client/server", "DevicePolicy": "auto", "Documentation": "man:chronyd(8) man:chrony.conf(5)", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/chronyd (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "906", "ExecMainStartTimestamp": "Sat 2021-07-31 01:17:13 UTC", "ExecMainStartTimestampMonotonic": "15478283", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/sbin/chronyd ; argv[]=/usr/sbin/chronyd $OPTIONS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartPost": "{ path=/usr/libexec/chrony-helper ; argv[]=/usr/libexec/chrony-helper update-daemon ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/chronyd.service", "FreezerState": "running", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "chronyd.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Sat 2021-07-31 01:17:12 UTC", "InactiveExitTimestampMonotonic": "14530599", "InvocationID": "5f33d35a2e7f4fbba0910d215e6c8fb9", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity" 2021-07-31T01:25:57.260 INFO:teuthology.task.ansible.out:, "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127237", "LimitNPROCSoft": "127237", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127237", "LimitSIGPENDINGSoft": "127237", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "906", "MemoryAccounting": "yes", "MemoryCurrent": "3190784", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "chronyd.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PIDFile": "/run/chrony/chronyd.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "yes", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "full", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount sysinit.target system.slice", "RequiresMountsFor": "/var/tmp", "Restart": "no 2021-07-31T01:25:57.262 INFO:teuthology.task.ansible.out:", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Sat 2021-07-31 01:17:13 UTC", "StateChangeTimestampMonotonic": "15540312", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203580", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Sat 2021-07-31 01:17:13 UTC", "WatchdogTimestampMonotonic": "15478293", "WatchdogUSec": "0"}} 2021-07-31T01:25:57.281 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:57 +0000 (0:00:00.325) 0:07:41.261 ********* 2021-07-31T01:25:57.288 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-07-31T01:25:57.289 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-UuRb0PtKkCf2pl3XWdG2Y7AzQldqcOsdrm3YR7ofCqEsKSEyO2rlgpOfoKh6TNBB'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.155173", "end": "2021-07-31 01:25:57.236967", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-UuRb0PtKkCf2pl3XWdG2Y7AzQldqcOsdrm3YR7ofCqEsKSEyO2rlgpOfoKh6TNBB"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-07-31 01:25:56.081794", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-07-31T01:25:57.301 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-07-31T01:25:57.303 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2021-07-31T01:25:57.324 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:57 +0000 (0:00:00.042) 0:07:41.304 ********* 2021-07-31T01:25:57.363 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-07-31T01:25:57.365 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"ansible_facts": {"ansible_python_interpreter": "/usr/bin/python3", "pip_executable": "pip3", "pip_version": "python3-pip"}, "changed": false} 2021-07-31T01:25:57.386 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:57 +0000 (0:00:00.062) 0:07:41.367 ********* 2021-07-31T01:25:57.416 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2021-07-31T01:25:57.418 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:57.444 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:57 +0000 (0:00:00.057) 0:07:41.425 ********* 2021-07-31T01:25:57.472 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2021-07-31T01:25:57.473 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:57.494 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:57 +0000 (0:00:00.050) 0:07:41.475 ********* 2021-07-31T01:25:57.523 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:57.555 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:57 +0000 (0:00:00.060) 0:07:41.535 ********* 2021-07-31T01:25:57.753 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2021-07-31T01:25:57.755 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"changed": false, "gid": 1236, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 4096, "state": "directory", "uid": 1000} 2021-07-31T01:25:57.776 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:57 +0000 (0:00:00.220) 0:07:41.756 ********* 2021-07-31T01:25:58.241 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2021-07-31T01:25:58.243 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1236, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 73, "state": "file", "uid": 1000} 2021-07-31T01:25:58.264 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:58 +0000 (0:00:00.488) 0:07:42.244 ********* 2021-07-31T01:25:58.560 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-07-31T01:25:58.561 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-vSHmA1-lJyE-6L0M-9esQ-GuQ1-6wBF-yf1yLx', 'nvme-INTEL_SSDPED1D480GA_PHMB75120098480DGN', 'nvme-nvme.8086-50484d42373531323030393834383044474e-494e54454c2053534450454431443438304741-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPED1D480GA', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '937703088', 'sectorsize': '512', 'size': '447.13 GB', 'host': 'Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.092308", "end": "2021-07-31 01:25:58.525819", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series", "links": {"ids": ["lvm-pv-uuid-vSHmA1-lJyE-6L0M-9esQ-GuQ1-6wBF-yf1yLx", "nvme-INTEL_SSDPED1D480GA_PHMB75120098480DGN", "nvme-nvme.8086-50484d42373531323030393834383044474e-494e54454c2053534450454431443438304741-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPED1D480GA", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "937703088", "sectorsize": "512", "size": "447.13 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-07-31 01:25:57.433511", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! You may now pa 2021-07-31T01:25:58.563 INFO:teuthology.task.ansible.out:rtition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-07-31T01:25:59.449 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2021-07-31T01:25:59.451 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip3", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages\nRequirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages", "Requirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2021-07-31T01:25:59.472 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:59 +0000 (0:00:01.207) 0:07:43.452 ********* 2021-07-31T01:25:59.499 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2021-07-31T01:25:59.500 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:59.521 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:59 +0000 (0:00:00.048) 0:07:43.501 ********* 2021-07-31T01:25:59.548 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2021-07-31T01:25:59.550 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2021-07-31T01:25:59.575 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:59 +0000 (0:00:00.054) 0:07:43.555 ********* 2021-07-31T01:25:59.760 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2021-07-31T01:25:59.762 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2021-07-31T01:25:59.783 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:59 +0000 (0:00:00.208) 0:07:43.764 ********* 2021-07-31T01:25:59.801 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-07-31T01:25:59.804 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-UuRb0PtKkCf2pl3XWdG2Y7AzQldqcOsd1Wyjyc4HkSORJweNAeTzd0sv4jrY9Ot9'], 'uuids': ['09234565-c83d-4cf7-a402-8c91d04cc5ee'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.049356", "end": "2021-07-31 01:25:59.770539", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-UuRb0PtKkCf2pl3XWdG2Y7AzQldqcOsd1Wyjyc4HkSORJweNAeTzd0sv4jrY9Ot9"], "labels": [], "masters": [], "uuids": ["09234565-c83d-4cf7-a402-8c91d04cc5ee"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-07-31 01:25:58.721183", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-07-31T01:25:59.814 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:59.847 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:59 +0000 (0:00:00.063) 0:07:43.827 ********* 2021-07-31T01:25:59.877 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** ok: [smithi154.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2021-07-31T01:25:59.899 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:59 +0000 (0:00:00.052) 0:07:43.879 ********* 2021-07-31T01:25:59.936 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2021-07-31T01:25:59.938 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:25:59.959 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:25:59 +0000 (0:00:00.059) 0:07:43.939 ********* 2021-07-31T01:26:00.155 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** changed: [smithi154.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2021-07-31T01:26:00.176 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:00 +0000 (0:00:00.217) 0:07:44.156 ********* 2021-07-31T01:26:00.382 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2021-07-31T01:26:00.384 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2021-07-31T01:26:00.407 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:00 +0000 (0:00:00.230) 0:07:44.387 ********* 2021-07-31T01:26:00.436 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:26:00.463 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:00 +0000 (0:00:00.056) 0:07:44.443 ********* 2021-07-31T01:26:00.502 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2021-07-31T01:26:00.504 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} 2021-07-31T01:26:00.515 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2021-07-31T01:26:00.516 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2021-07-31T01:26:00.538 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:00 +0000 (0:00:00.075) 0:07:44.519 ********* 2021-07-31T01:26:00.577 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2021-07-31T01:26:00.579 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-07-31T01:26:00.589 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-07-31T01:26:00.591 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-07-31T01:26:00.620 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:00 +0000 (0:00:00.081) 0:07:44.601 ********* 2021-07-31T01:26:00.651 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-07-31T01:26:00.653 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:26:00.675 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:00 +0000 (0:00:00.055) 0:07:44.656 ********* 2021-07-31T01:26:00.703 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2021-07-31T01:26:00.705 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:26:00.726 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:00 +0000 (0:00:00.051) 0:07:44.707 ********* 2021-07-31T01:26:00.754 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2021-07-31T01:26:00.756 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:26:00.779 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:00 +0000 (0:00:00.053) 0:07:44.760 ********* 2021-07-31T01:26:00.808 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:26:00.830 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:00 +0000 (0:00:00.050) 0:07:44.810 ********* 2021-07-31T01:26:00.857 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2021-07-31T01:26:00.859 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:26:00.888 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:00 +0000 (0:00:00.057) 0:07:44.868 ********* 2021-07-31T01:26:00.915 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-07-31T01:26:00.916 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:26:00.942 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:00 +0000 (0:00:00.053) 0:07:44.922 ********* 2021-07-31T01:26:00.968 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2021-07-31T01:26:00.970 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:26:00.992 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:00 +0000 (0:00:00.050) 0:07:44.972 ********* 2021-07-31T01:26:01.019 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:26:01.044 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:01 +0000 (0:00:00.051) 0:07:45.024 ********* 2021-07-31T01:26:01.052 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-07-31T01:26:01.054 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-UuRb0PtKkCf2pl3XWdG2Y7AzQldqcOsdr8JPpnlUbKqtAFcH8Py4tSzDPlhT030d'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.052120", "end": "2021-07-31 01:26:01.019911", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-UuRb0PtKkCf2pl3XWdG2Y7AzQldqcOsdr8JPpnlUbKqtAFcH8Py4tSzDPlhT030d"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-07-31 01:25:59.967791", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-07-31T01:26:01.270 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** ok: [smithi154.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.005223", "end": "2021-07-31 01:26:01.235375", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2021-07-31 01:26:01.230152", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-07-31T01:26:01.439 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.003132", "end": "2021-07-31 01:26:01.407038", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2021-07-31 01:26:01.403906", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-07-31T01:26:01.619 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.002779", "end": "2021-07-31 01:26:01.577026", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2021-07-31 01:26:01.574247", "stderr": "find: ‘/etc/ceph’: No such file or directory", "stderr_lines": ["find: ‘/etc/ceph’: No such file or directory"], "stdout": "", "stdout_lines": []} 2021-07-31T01:26:01.787 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.003156", "end": "2021-07-31 01:26:01.759941", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2021-07-31 01:26:01.756785", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-07-31T01:26:01.810 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:01 +0000 (0:00:00.766) 0:07:45.790 ********* 2021-07-31T01:26:01.839 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2021-07-31T01:26:01.861 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:01 +0000 (0:00:00.050) 0:07:45.841 ********* 2021-07-31T01:26:01.889 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2021-07-31T01:26:01.891 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:26:02.288 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-07-31T01:26:02.290 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-UuRb0PtKkCf2pl3XWdG2Y7AzQldqcOsdbs8WCFJXQumvg7BHtNSwZJs9QBDHQBvh'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.042294", "end": "2021-07-31 01:26:02.255401", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-UuRb0PtKkCf2pl3XWdG2Y7AzQldqcOsdbs8WCFJXQumvg7BHtNSwZJs9QBDHQBvh"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-07-31 01:26:01.213107", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-07-31T01:26:02.291 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM 2021-07-31T01:26:02.292 INFO:teuthology.task.ansible.out:0033-9ZM173_Z1W4HPNL', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W4HPNL', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W4HPNL', 'scsi-35000c500871a9bab', 'scsi-SATA_ST1000NM0033-9ZM_Z1W4HPNL', 'wwn-0x5000c500871a9bab'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W4HPNL', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c500871a9bab', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HPNL-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W4HPNL-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W4HPNL-part1', 'scsi-35000c500871a9bab-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W4HPNL-part1', 'wwn-0x5000c500871a9bab-part1'], 'uuids': ['782d0384-b480-44b3-9c95-fe95daf886ae'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': '782d0384-b480-44b3-9c95-fe95daf886ae', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HPNL", "scsi-0ATA_ST1000NM0033-9ZM_Z1W4HPNL", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W4HPNL", "scsi-35000c500871a9bab", "scsi-SATA_ST1000NM0033-9ZM_Z1W4HPNL", "wwn-0x5000c500871a9bab"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HPNL-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W4HPNL-part1", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W4HPNL-part1", "scsi-35000c500871a9bab-part1", "scsi-SATA_ST1000NM0033-9ZM_Z1W4HPNL-part1", "wwn-0x5000c500871a9bab-part 2021-07-31T01:26:02.294 INFO:teuthology.task.ansible.out:1"], "labels": [], "masters": [], "uuids": ["782d0384-b480-44b3-9c95-fe95daf886ae"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "782d0384-b480-44b3-9c95-fe95daf886ae"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4HPNL", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500871a9bab"}}, "skip_reason": "Conditional result was False"} 2021-07-31T01:26:03.603 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-UuRb0PtKkCf2pl3XWdG2Y7AzQldqcOsdUHEhW3TJbtvbpKxaHgIaEGVKfImoUOFB'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.119619", "end": "2021-07-31 01:26:03.572130", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-UuRb0PtKkCf2pl3XWdG2Y7AzQldqcOsdUHEhW3TJbtvbpKxaHgIaEGVKfImoUOFB"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-07-31 01:26:02.452511", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-07-31T01:26:03.631 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:03 +0000 (0:00:01.769) 0:07:47.611 ********* 2021-07-31T01:26:03.950 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** changed: [smithi045.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-UuRb0PtKkCf2pl3XWdG2Y7AzQldqcOsdrm3YR7ofCqEsKSEyO2rlgpOfoKh6TNBB'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.094752", "end": "2021-07-31 01:26:03.924521", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-UuRb0PtKkCf2pl3XWdG2Y7AzQldqcOsdrm3YR7ofCqEsKSEyO2rlgpOfoKh6TNBB"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-07-31 01:26:03.829769", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0920224 s, 1.3 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0920224 s, 1.3 GB/s"], "stdout": "", "stdout_lines": []} 2021-07-31T01:26:04.169 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-vSHmA1-lJyE-6L0M-9esQ-GuQ1-6wBF-yf1yLx', 'nvme-INTEL_SSDPED1D480GA_PHMB75120098480DGN', 'nvme-nvme.8086-50484d42373531323030393834383044474e-494e54454c2053534450454431443438304741-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPED1D480GA', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '937703088', 'sectorsize': '512', 'size': '447.13 GB', 'host': 'Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.036803", "end": "2021-07-31 01:26:04.141621", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series", "links": {"ids": ["lvm-pv-uuid-vSHmA1-lJyE-6L0M-9esQ-GuQ1-6wBF-yf1yLx", "nvme-INTEL_SSDPED1D480GA_PHMB75120098480DGN", "nvme-nvme.8086-50484d42373531323030393834383044474e-494e54454c2053534450454431443438304741-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPED1D480GA", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "937703088", "sectorsize": "512", "size": "447.13 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-07-31 01:26:04.104818", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0340929 s, 3.4 GB/s", "stderr_lines": [" 2021-07-31T01:26:04.171 INFO:teuthology.task.ansible.out:110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0340929 s, 3.4 GB/s"], "stdout": "", "stdout_lines": []} 2021-07-31T01:26:04.442 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-UuRb0PtKkCf2pl3XWdG2Y7AzQldqcOsd1Wyjyc4HkSORJweNAeTzd0sv4jrY9Ot9'], 'uuids': ['09234565-c83d-4cf7-a402-8c91d04cc5ee'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.094835", "end": "2021-07-31 01:26:04.414472", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-UuRb0PtKkCf2pl3XWdG2Y7AzQldqcOsd1Wyjyc4HkSORJweNAeTzd0sv4jrY9Ot9"], "labels": [], "masters": [], "uuids": ["09234565-c83d-4cf7-a402-8c91d04cc5ee"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-07-31 01:26:04.319637", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0920329 s, 1.3 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0920329 s, 1.3 GB/s"], "stdout": "", "stdout_lines": []} 2021-07-31T01:26:04.716 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-UuRb0PtKkCf2pl3XWdG2Y7AzQldqcOsdr8JPpnlUbKqtAFcH8Py4tSzDPlhT030d'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.096053", "end": "2021-07-31 01:26:04.692622", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-UuRb0PtKkCf2pl3XWdG2Y7AzQldqcOsdr8JPpnlUbKqtAFcH8Py4tSzDPlhT030d"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-07-31 01:26:04.596569", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0933483 s, 1.2 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0933483 s, 1.2 GB/s"], "stdout": "", "stdout_lines": []} 2021-07-31T01:26:04.997 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-UuRb0PtKkCf2pl3XWdG2Y7AzQldqcOsdbs8WCFJXQumvg7BHtNSwZJs9QBDHQBvh'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.095863", "end": "2021-07-31 01:26:04.969213", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-UuRb0PtKkCf2pl3XWdG2Y7AzQldqcOsdbs8WCFJXQumvg7BHtNSwZJs9QBDHQBvh"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-07-31 01:26:04.873350", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0931687 s, 1.2 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0931687 s, 1.2 GB/s"], "stdout": "", "stdout_lines": []} 2021-07-31T01:26:04.999 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HPNL', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W4HPNL', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W4HPNL', 'scsi-35000c500871a9bab', 'scsi-SATA_ST1000NM0033-9ZM_Z1W4HPNL', 'wwn-0x5000c500871a9bab'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W4HPNL', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c500871a9bab', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HPNL-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W4HPNL-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W4HPNL-part1', 'scsi-35000c500871a9bab-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W4HPNL-part1', 'wwn-0x5000c500871a9bab-part1'], 'uuids': ['782d0384-b480-44b3-9c95-fe95daf886ae'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': '782d0384-b480-44b3-9c95-fe95daf886ae', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HPNL", "scsi-0ATA_ST1000NM0033-9ZM_Z1W4HPNL", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W4HPNL", "scsi-35000c500871a9bab", "scsi-SATA_ST1000NM0033-9ZM_Z1W4HPNL", "wwn-0x5000c500871a9bab"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HPNL-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W4HPNL-part1", "scsi-1ATA_ST1000NM0033-9ZM173_ 2021-07-31T01:26:05.000 INFO:teuthology.task.ansible.out:Z1W4HPNL-part1", "scsi-35000c500871a9bab-part1", "scsi-SATA_ST1000NM0033-9ZM_Z1W4HPNL-part1", "wwn-0x5000c500871a9bab-part1"], "labels": [], "masters": [], "uuids": ["782d0384-b480-44b3-9c95-fe95daf886ae"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "782d0384-b480-44b3-9c95-fe95daf886ae"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4HPNL", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500871a9bab"}}, "skip_reason": "Conditional result was False"} 2021-07-31T01:26:05.271 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-UuRb0PtKkCf2pl3XWdG2Y7AzQldqcOsdUHEhW3TJbtvbpKxaHgIaEGVKfImoUOFB'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.095347", "end": "2021-07-31 01:26:05.250301", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-UuRb0PtKkCf2pl3XWdG2Y7AzQldqcOsdUHEhW3TJbtvbpKxaHgIaEGVKfImoUOFB"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-07-31 01:26:05.154954", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0927233 s, 1.2 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0927233 s, 1.2 GB/s"], "stdout": "", "stdout_lines": []} 2021-07-31T01:26:05.312 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:05 +0000 (0:00:01.681) 0:07:49.292 ********* 2021-07-31T01:26:05.591 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2021-07-31T01:26:05.593 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.074404", "end": "2021-07-31 01:26:05.558014", "rc": 0, "start": "2021-07-31 01:26:05.483610", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-07-31T01:26:05.614 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:05 +0000 (0:00:00.301) 0:07:49.594 ********* 2021-07-31T01:26:05.852 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2021-07-31T01:26:05.854 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.050480", "end": "2021-07-31 01:26:05.829661", "rc": 0, "start": "2021-07-31 01:26:05.779181", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-07-31T01:26:05.874 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:05 +0000 (0:00:00.259) 0:07:49.854 ********* 2021-07-31T01:26:05.902 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2021-07-31T01:26:05.925 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:05 +0000 (0:00:00.051) 0:07:49.906 ********* 2021-07-31T01:26:05.952 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2021-07-31T01:26:05.954 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:26:05.977 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:05 +0000 (0:00:00.051) 0:07:49.957 ********* 2021-07-31T01:26:06.004 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2021-07-31T01:26:06.006 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:26:06.026 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:06 +0000 (0:00:00.048) 0:07:50.006 ********* 2021-07-31T01:26:06.053 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-07-31T01:26:06.054 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:26:06.078 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:06 +0000 (0:00:00.052) 0:07:50.059 ********* 2021-07-31T01:26:06.106 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2021-07-31T01:26:06.108 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:26:06.130 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:06 +0000 (0:00:00.050) 0:07:50.109 ********* 2021-07-31T01:26:06.330 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ ok: [smithi045.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-07-31T01:26:06.354 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:06 +0000 (0:00:00.225) 0:07:50.335 ********* 2021-07-31T01:26:06.394 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-07-31T01:26:06.396 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930416504832, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227152467, 'block_used': 12942641, 'inode_total': 61054976, 'inode_available': 60986953, 'inode_used': 68023, 'uuid': '782d0384-b480-44b3-9c95-fe95daf886ae'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227152467, "block_size": 4096, "block_total": 240095108, "block_used": 12942641, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60986953, "inode_total": 61054976, "inode_used": 68023, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930416504832, "size_total": 983429562368, "uuid": "782d0384-b480-44b3-9c95-fe95daf886ae"}, "skip_reason": "Conditional result was False"} 2021-07-31T01:26:06.418 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:06 +0000 (0:00:00.063) 0:07:50.398 ********* 2021-07-31T01:26:06.448 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2021-07-31T01:26:06.450 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:26:06.471 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:06 +0000 (0:00:00.052) 0:07:50.451 ********* 2021-07-31T01:26:06.499 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:26:06.521 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:06 +0000 (0:00:00.050) 0:07:50.501 ********* 2021-07-31T01:26:06.548 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-07-31T01:26:06.550 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:26:06.572 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:06 +0000 (0:00:00.050) 0:07:50.551 ********* 2021-07-31T01:26:06.599 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2021-07-31T01:26:06.601 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:26:06.621 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:06 +0000 (0:00:00.049) 0:07:50.601 ********* 2021-07-31T01:26:06.648 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2021-07-31T01:26:06.649 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:26:06.670 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:06 +0000 (0:00:00.048) 0:07:50.650 ********* 2021-07-31T01:26:06.697 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2021-07-31T01:26:06.698 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:26:06.719 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:06 +0000 (0:00:00.048) 0:07:50.699 ********* 2021-07-31T01:26:07.168 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2021-07-31T01:26:07.170 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2021-07-31T01:26:07.191 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:07 +0000 (0:00:00.472) 0:07:51.171 ********* 2021-07-31T01:26:07.683 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-07-31T01:26:07.685 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-07-31T01:26:08.012 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-07-31T01:26:08.331 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-07-31T01:26:08.675 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-07-31T01:26:09.034 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2021-07-31T01:26:09.062 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:09 +0000 (0:00:01.871) 0:07:53.043 ********* 2021-07-31T01:26:09.273 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-07-31T01:26:09.275 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2021-07-31T01:26:09.297 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:09 +0000 (0:00:00.234) 0:07:53.277 ********* 2021-07-31T01:26:09.515 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-07-31T01:26:09.516 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-07-31T01:26:09.694 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-07-31T01:26:09.872 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-07-31T01:26:10.045 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-07-31T01:26:10.058 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2021-07-31T01:26:10.089 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:10 +0000 (0:00:00.791) 0:07:54.069 ********* 2021-07-31T01:26:10.119 INFO:teuthology.task.ansible.out: TASK [testnode : Create filesystems] ******************************************* 2021-07-31T01:26:10.121 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:26:10.146 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:10 +0000 (0:00:00.057) 0:07:54.126 ********* 2021-07-31T01:26:10.174 INFO:teuthology.task.ansible.out: TASK [testnode : Mount filesystems] ******************************************** 2021-07-31T01:26:10.176 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:26:10.196 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:10 +0000 (0:00:00.050) 0:07:54.177 ********* 2021-07-31T01:26:10.412 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2021-07-31T01:26:10.414 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "secontext": "unconfined_u:object_r:var_lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-07-31T01:26:10.434 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:10 +0000 (0:00:00.237) 0:07:54.414 ********* 2021-07-31T01:26:10.472 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2021-07-31T01:26:10.474 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:26:10.494 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:10 +0000 (0:00:00.060) 0:07:54.474 ********* 2021-07-31T01:26:10.740 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2021-07-31T01:26:10.741 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => {"changed": true} 2021-07-31T01:26:10.763 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:10 +0000 (0:00:00.268) 0:07:54.743 ********* 2021-07-31T01:26:11.073 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2021-07-31T01:26:11.075 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2021-07-31T01:26:11.096 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:11 +0000 (0:00:00.332) 0:07:55.076 ********* 2021-07-31T01:26:11.122 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd] ****************************************** 2021-07-31T01:26:11.124 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:26:11.144 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:11 +0000 (0:00:00.048) 0:07:55.124 ********* 2021-07-31T01:26:11.172 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd conf file] ******************************** 2021-07-31T01:26:11.174 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:26:11.194 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:11 +0000 (0:00:00.049) 0:07:55.174 ********* 2021-07-31T01:26:11.220 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-07-31T01:26:11.221 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:26:11.241 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:11 +0000 (0:00:00.047) 0:07:55.221 ********* 2021-07-31T01:26:11.277 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-07-31T01:26:11.279 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:26:11.302 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:11 +0000 (0:00:00.060) 0:07:55.282 ********* 2021-07-31T01:26:11.329 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2021-07-31T01:26:11.331 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:26:11.351 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:11 +0000 (0:00:00.049) 0:07:55.332 ********* 2021-07-31T01:26:11.379 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2021-07-31T01:26:11.381 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:26:11.401 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:11 +0000 (0:00:00.049) 0:07:55.381 ********* 2021-07-31T01:26:11.428 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2021-07-31T01:26:11.430 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:26:11.451 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:11 +0000 (0:00:00.050) 0:07:55.431 ********* 2021-07-31T01:26:11.479 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2021-07-31T01:26:11.481 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:26:11.502 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:11 +0000 (0:00:00.050) 0:07:55.482 ********* 2021-07-31T01:26:11.529 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2021-07-31T01:26:11.531 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:26:11.550 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:11 +0000 (0:00:00.048) 0:07:55.531 ********* 2021-07-31T01:26:11.577 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:26:11.601 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:11 +0000 (0:00:00.050) 0:07:55.581 ********* 2021-07-31T01:26:11.630 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:26:11.652 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:11 +0000 (0:00:00.051) 0:07:55.633 ********* 2021-07-31T01:26:11.679 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:26:11.702 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:11 +0000 (0:00:00.049) 0:07:55.682 ********* 2021-07-31T01:26:11.729 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2021-07-31T01:26:11.731 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:26:11.751 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:11 +0000 (0:00:00.048) 0:07:55.731 ********* 2021-07-31T01:26:11.778 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2021-07-31T01:26:11.780 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:26:11.799 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:11 +0000 (0:00:00.048) 0:07:55.780 ********* 2021-07-31T01:26:11.826 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2021-07-31T01:26:11.828 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:26:11.847 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:11 +0000 (0:00:00.047) 0:07:55.827 ********* 2021-07-31T01:26:12.330 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2021-07-31T01:26:12.332 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "checksum": "a7ecef00073feab34f868065d34e9dac55134241", "dest": "/etc/chrony.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/chrony.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 215, "state": "file", "uid": 0} 2021-07-31T01:26:12.354 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:12 +0000 (0:00:00.506) 0:07:56.334 ********* 2021-07-31T01:26:12.655 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2021-07-31T01:26:12.657 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "chronyd", "state": "started", "status": {"ActiveEnterTimestamp": "Sat 2021-07-31 01:16:29 UTC", "ActiveEnterTimestampMonotonic": "19093156", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "ntpdate.service sysinit.target system.slice basic.target -.mount systemd-journald.socket tmp.mount sntp.service ntpd.service systemd-tmpfiles-setup.service", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Sat 2021-07-31 01:16:28 UTC", "AssertTimestampMonotonic": "17820626", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Sat 2021-07-31 01:16:28 UTC", "ConditionTimestampMonotonic": "17820543", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target systemd-timesyncd.service ntpd.service", "ControlGro 2021-07-31T01:26:12.658 INFO:teuthology.task.ansible.out:up": "/system.slice/chronyd.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "NTP client/server", "DevicePolicy": "auto", "Documentation": "man:chronyd(8) man:chrony.conf(5)", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/chronyd (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "890", "ExecMainStartTimestamp": "Sat 2021-07-31 01:16:28 UTC", "ExecMainStartTimestampMonotonic": "17992316", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/sbin/chronyd ; argv[]=/usr/sbin/chronyd $OPTIONS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartPost": "{ path=/usr/libexec/chrony-helper ; argv[]=/usr/libexec/chrony-helper update-daemon ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/chronyd.service", "FreezerState": "running", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "chronyd.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Sat 2021-07-31 01:16:28 UTC", "InactiveExitTimestampMonotonic": "17846833", "InvocationID": "b35b3630096149b6abfb5449af87dcdf", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity" 2021-07-31T01:26:12.660 INFO:teuthology.task.ansible.out:, "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127262", "LimitNPROCSoft": "127262", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127262", "LimitSIGPENDINGSoft": "127262", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "890", "MemoryAccounting": "yes", "MemoryCurrent": "3342336", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "chronyd.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PIDFile": "/run/chrony/chronyd.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "yes", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "full", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target system.slice -.mount", "RequiresMountsFor": "/var/tmp", "Restart": "no 2021-07-31T01:26:12.662 INFO:teuthology.task.ansible.out:", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Sat 2021-07-31 01:16:29 UTC", "StateChangeTimestampMonotonic": "19093156", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203619", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Sat 2021-07-31 01:16:28 UTC", "WatchdogTimestampMonotonic": "17992338", "WatchdogUSec": "0"}} 2021-07-31T01:26:12.680 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:12 +0000 (0:00:00.326) 0:07:56.660 ********* 2021-07-31T01:26:12.708 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-07-31T01:26:12.710 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2021-07-31T01:26:12.732 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:12 +0000 (0:00:00.051) 0:07:56.712 ********* 2021-07-31T01:26:12.772 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-07-31T01:26:12.774 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"ansible_facts": {"ansible_python_interpreter": "/usr/bin/python3", "pip_executable": "pip3", "pip_version": "python3-pip"}, "changed": false} 2021-07-31T01:26:12.794 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:12 +0000 (0:00:00.062) 0:07:56.775 ********* 2021-07-31T01:26:12.833 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:26:12.855 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:12 +0000 (0:00:00.060) 0:07:56.835 ********* 2021-07-31T01:26:12.893 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2021-07-31T01:26:12.895 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:26:12.917 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:12 +0000 (0:00:00.062) 0:07:56.897 ********* 2021-07-31T01:26:12.946 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2021-07-31T01:26:12.949 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:26:12.968 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:12 +0000 (0:00:00.050) 0:07:56.948 ********* 2021-07-31T01:26:13.175 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2021-07-31T01:26:13.177 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "gid": 1236, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 4096, "state": "directory", "uid": 1000} 2021-07-31T01:26:13.197 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:13 +0000 (0:00:00.229) 0:07:57.177 ********* 2021-07-31T01:26:13.693 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2021-07-31T01:26:13.695 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1236, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 73, "state": "file", "uid": 1000} 2021-07-31T01:26:13.721 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:13 +0000 (0:00:00.524) 0:07:57.701 ********* 2021-07-31T01:26:14.808 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2021-07-31T01:26:14.810 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip3", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages\nRequirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages", "Requirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2021-07-31T01:26:14.831 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:14 +0000 (0:00:01.109) 0:07:58.811 ********* 2021-07-31T01:26:14.858 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:26:14.880 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:14 +0000 (0:00:00.049) 0:07:58.861 ********* 2021-07-31T01:26:14.908 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2021-07-31T01:26:14.909 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"ansible_facts": {"interface": "ens1f0"}, "changed": false} 2021-07-31T01:26:14.929 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:14 +0000 (0:00:00.048) 0:07:58.909 ********* 2021-07-31T01:26:15.112 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2021-07-31T01:26:15.116 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2021-07-31T01:26:15.135 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:15 +0000 (0:00:00.205) 0:07:59.115 ********* 2021-07-31T01:26:15.162 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2021-07-31T01:26:15.164 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:26:15.185 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:15 +0000 (0:00:00.050) 0:07:59.165 ********* 2021-07-31T01:26:15.224 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2021-07-31T01:26:15.226 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2021-07-31T01:26:15.246 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:15 +0000 (0:00:00.061) 0:07:59.227 ********* 2021-07-31T01:26:15.273 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2021-07-31T01:26:15.275 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:26:15.295 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:15 +0000 (0:00:00.048) 0:07:59.275 ********* 2021-07-31T01:26:15.503 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** changed: [smithi045.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2021-07-31T01:26:15.524 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:15 +0000 (0:00:00.229) 0:07:59.504 ********* 2021-07-31T01:26:15.720 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2021-07-31T01:26:15.722 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2021-07-31T01:26:15.742 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:15 +0000 (0:00:00.218) 0:07:59.723 ********* 2021-07-31T01:26:15.770 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:26:15.792 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:15 +0000 (0:00:00.049) 0:07:59.773 ********* 2021-07-31T01:26:15.830 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2021-07-31T01:26:15.831 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item=ens1f1) => {"ansible_loop_var": "item", "changed": false, "item": "ens1f1", "skip_reason": "Conditional result was False"} 2021-07-31T01:26:15.842 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=ens1f0) => {"ansible_loop_var": "item", "changed": false, "item": "ens1f0", "skip_reason": "Conditional result was False"} 2021-07-31T01:26:15.843 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2021-07-31T01:26:15.866 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:15 +0000 (0:00:00.073) 0:07:59.846 ********* 2021-07-31T01:26:15.905 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2021-07-31T01:26:15.907 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-07-31T01:26:15.919 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'ens1f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "ens1f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-07-31T01:26:15.928 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'ens1f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "ens1f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-07-31T01:26:15.930 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-07-31T01:26:15.952 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:15 +0000 (0:00:00.085) 0:07:59.932 ********* 2021-07-31T01:26:15.979 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-07-31T01:26:15.981 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:26:16.003 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:16 +0000 (0:00:00.051) 0:07:59.983 ********* 2021-07-31T01:26:16.030 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2021-07-31T01:26:16.032 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:26:16.053 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:16 +0000 (0:00:00.050) 0:08:00.034 ********* 2021-07-31T01:26:16.093 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2021-07-31T01:26:16.095 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:26:16.123 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:16 +0000 (0:00:00.068) 0:08:00.102 ********* 2021-07-31T01:26:16.151 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2021-07-31T01:26:16.153 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:26:16.173 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:16 +0000 (0:00:00.051) 0:08:00.154 ********* 2021-07-31T01:26:16.203 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2021-07-31T01:26:16.204 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:26:16.228 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:16 +0000 (0:00:00.054) 0:08:00.208 ********* 2021-07-31T01:26:16.257 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-07-31T01:26:16.259 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:26:16.287 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:16 +0000 (0:00:00.059) 0:08:00.268 ********* 2021-07-31T01:26:16.319 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2021-07-31T01:26:16.321 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:26:16.344 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:16 +0000 (0:00:00.056) 0:08:00.324 ********* 2021-07-31T01:26:16.373 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:26:16.394 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:16 +0000 (0:00:00.049) 0:08:00.374 ********* 2021-07-31T01:26:16.621 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2021-07-31T01:26:16.623 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.005560", "end": "2021-07-31 01:26:16.587076", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2021-07-31 01:26:16.581516", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-07-31T01:26:16.798 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.003231", "end": "2021-07-31 01:26:16.771804", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2021-07-31 01:26:16.768573", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-07-31T01:26:16.975 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.002922", "end": "2021-07-31 01:26:16.946884", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2021-07-31 01:26:16.943962", "stderr": "find: ‘/etc/ceph’: No such file or directory", "stderr_lines": ["find: ‘/etc/ceph’: No such file or directory"], "stdout": "", "stdout_lines": []} 2021-07-31T01:26:17.153 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.003373", "end": "2021-07-31 01:26:17.120960", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2021-07-31 01:26:17.117587", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-07-31T01:26:17.175 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:17 +0000 (0:00:00.781) 0:08:01.156 ********* 2021-07-31T01:26:17.203 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2021-07-31T01:26:17.227 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:17 +0000 (0:00:00.051) 0:08:01.208 ********* 2021-07-31T01:26:17.254 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2021-07-31T01:26:17.256 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:26:17.321 INFO:teuthology.task.ansible.out: PLAY [testnodes,senta,vossi,jenkins_slaves] ************************************ 2021-07-31T01:26:17.338 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:17 +0000 (0:00:00.110) 0:08:01.318 ********* 2021-07-31T01:26:17.358 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:17 +0000 (0:00:00.019) 0:08:01.338 ********* 2021-07-31T01:26:18.010 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi154.front.sepia.ceph.com] 2021-07-31T01:26:18.064 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:18 +0000 (0:00:00.705) 0:08:02.044 ********* 2021-07-31T01:26:18.094 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:26:18.116 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:18 +0000 (0:00:00.052) 0:08:02.096 ********* 2021-07-31T01:26:18.155 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-07-31T01:26:18.157 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"ansible_facts": {"package_manager": "yum"}, "changed": false} 2021-07-31T01:26:18.189 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:18 +0000 (0:00:00.072) 0:08:02.169 ********* 2021-07-31T01:26:18.231 INFO:teuthology.task.ansible.out: TASK [container-host : Including distro specific variables] ******************** 2021-07-31T01:26:18.233 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml) => {"ansible_facts": {"container_packages": ["podman", "podman-docker"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"} 2021-07-31T01:26:18.256 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:18 +0000 (0:00:00.065) 0:08:02.235 ********* 2021-07-31T01:26:20.003 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* 2021-07-31T01:26:20.005 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] 2021-07-31T01:26:20.075 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:20 +0000 (0:00:01.820) 0:08:04.055 ********* 2021-07-31T01:26:20.107 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-07-31T01:26:20.109 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:26:20.129 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:20 +0000 (0:00:00.053) 0:08:04.109 ********* 2021-07-31T01:26:20.171 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-07-31T01:26:20.174 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"ansible_facts": {"package_manager": "yum"}, "changed": false} 2021-07-31T01:26:20.194 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:20 +0000 (0:00:00.065) 0:08:04.175 ********* 2021-07-31T01:26:20.236 INFO:teuthology.task.ansible.out: TASK [container-host : Including distro specific variables] ******************** 2021-07-31T01:26:20.238 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml) => {"ansible_facts": {"container_packages": ["podman", "podman-docker"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"} 2021-07-31T01:26:20.259 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:20 +0000 (0:00:00.064) 0:08:04.240 ********* 2021-07-31T01:26:38.476 INFO:teuthology.task.ansible.out: TASK [container-host : Install container packages] ***************************** 2021-07-31T01:26:38.478 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: podman-catatonit-2.2.1-7.module+el8.3.1+9857+68fb1526.x86_64", "Installed: runc-1.0.0-70.rc92.module+el8.3.1+9857+68fb1526.x86_64", "Installed: podman-2.2.1-7.module+el8.3.1+9857+68fb1526.x86_64", "Installed: podman-docker-2.2.1-7.module+el8.3.1+9857+68fb1526.noarch", "Removed: podman-catatonit-2.0.5-5.module+el8.3.0+8221+97165c3f.x86_64", "Removed: podman-docker-2.0.5-5.module+el8.3.0+8221+97165c3f.noarch", "Removed: runc-1.0.0-68.rc92.module+el8.3.0+8221+97165c3f.x86_64", "Removed: podman-2.0.5-5.module+el8.3.0+8221+97165c3f.x86_64"]} 2021-07-31T01:26:38.500 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:38 +0000 (0:00:18.240) 0:08:22.480 ********* 2021-07-31T01:26:38.527 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-07-31T01:26:38.529 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"ansible_facts": {"container_service_conf": "/etc/containers/registries.conf"}, "changed": false} 2021-07-31T01:26:38.552 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:38 +0000 (0:00:00.051) 0:08:22.531 ********* 2021-07-31T01:26:38.578 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-07-31T01:26:38.581 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:26:38.602 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:38 +0000 (0:00:00.050) 0:08:22.582 ********* 2021-07-31T01:26:38.815 INFO:teuthology.task.ansible.out: TASK [container-host : Create container_mirror_cert_paths] ********************* 2021-07-31T01:26:38.817 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-07-31T01:26:38.996 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-07-31T01:26:39.023 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:39 +0000 (0:00:00.419) 0:08:23.002 ********* 2021-07-31T01:26:39.537 INFO:teuthology.task.ansible.out: TASK [container-host : Copy docker-mirror.front.sepia.ceph.com:5000 self-signed cert] *** 2021-07-31T01:26:39.539 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:cert_t:s0", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1627694799.0912309-18253-44360247887630/source", "state": "file", "uid": 0} 2021-07-31T01:26:40.007 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1627694799.5706544-18253-23303800851727/source", "state": "file", "uid": 0} 2021-07-31T01:26:40.030 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:40 +0000 (0:00:01.008) 0:08:24.011 ********* 2021-07-31T01:26:41.729 INFO:teuthology.task.ansible.out: TASK [container-host : Install container packages] ***************************** 2021-07-31T01:26:41.731 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: podman-catatonit-2.2.1-7.module+el8.3.1+9857+68fb1526.x86_64", "Installed: runc-1.0.0-70.rc92.module+el8.3.1+9857+68fb1526.x86_64", "Installed: podman-2.2.1-7.module+el8.3.1+9857+68fb1526.x86_64", "Installed: podman-docker-2.2.1-7.module+el8.3.1+9857+68fb1526.noarch", "Removed: podman-catatonit-2.0.5-5.module+el8.3.0+8221+97165c3f.x86_64", "Removed: podman-docker-2.0.5-5.module+el8.3.0+8221+97165c3f.noarch", "Removed: runc-1.0.0-68.rc92.module+el8.3.0+8221+97165c3f.x86_64", "Removed: podman-2.0.5-5.module+el8.3.0+8221+97165c3f.x86_64"]} 2021-07-31T01:26:41.753 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:41 +0000 (0:00:01.722) 0:08:25.733 ********* 2021-07-31T01:26:41.782 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-07-31T01:26:41.783 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"ansible_facts": {"container_service_conf": "/etc/containers/registries.conf"}, "changed": false} 2021-07-31T01:26:41.805 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:41 +0000 (0:00:00.051) 0:08:25.785 ********* 2021-07-31T01:26:41.832 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:26:41.855 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:41 +0000 (0:00:00.050) 0:08:25.835 ********* 2021-07-31T01:26:42.073 INFO:teuthology.task.ansible.out: TASK [container-host : Create container_mirror_cert_paths] ********************* 2021-07-31T01:26:42.074 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-07-31T01:26:42.276 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-07-31T01:26:42.305 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:42 +0000 (0:00:00.450) 0:08:26.286 ********* 2021-07-31T01:26:42.852 INFO:teuthology.task.ansible.out: TASK [container-host : Copy docker-mirror.front.sepia.ceph.com:5000 self-signed cert] *** 2021-07-31T01:26:42.853 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:cert_t:s0", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1627694802.373779-18592-206653192123653/source", "state": "file", "uid": 0} 2021-07-31T01:26:43.351 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1627694802.8951292-18592-83072698610760/source", "state": "file", "uid": 0} 2021-07-31T01:26:43.375 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:43 +0000 (0:00:01.069) 0:08:27.355 ********* 2021-07-31T01:26:51.393 INFO:teuthology.task.ansible.out: TASK [container-host : Ensure git is installed] ******************************** 2021-07-31T01:26:51.395 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-07-31T01:26:51.418 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:51 +0000 (0:00:08.043) 0:08:35.399 ********* 2021-07-31T01:26:54.688 INFO:teuthology.task.ansible.out: TASK [container-host : Install registries-conf-ctl] **************************** 2021-07-31T01:26:54.690 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => {"changed": true, "cmd": ["/bin/pip3", "install", "-U", "git+https://github.com/sebastian-philipp/registries-conf-ctl"], "name": ["git+https://github.com/sebastian-philipp/registries-conf-ctl"], "requirements": null, "state": "latest", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl\n Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-ecjxkzkf-build\nCollecting toml (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl\nCollecting docopt (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz\nInstalling collected packages: toml, docopt, registries-conf-ctl\n Running setup.py install for docopt: started\n Running setup.py install for docopt: finished with status 'done'\n Running setup.py install for registries-conf-ctl: started\n Running setup.py install for registries-conf-ctl: finished with status 'done'\nSuccessfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2\n", "stdout_lines": ["Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl", " Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-ecjxkzkf-build", "Collecting toml (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl", "Collecting docopt (from registries-conf-ctl==1.0)", " Downloading https://files. 2021-07-31T01:26:54.692 INFO:teuthology.task.ansible.out:pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz", "Installing collected packages: toml, docopt, registries-conf-ctl", " Running setup.py install for docopt: started", " Running setup.py install for docopt: finished with status 'done'", " Running setup.py install for registries-conf-ctl: started", " Running setup.py install for registries-conf-ctl: finished with status 'done'", "Successfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2"], "version": null, "virtualenv": null} 2021-07-31T01:26:54.711 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:54 +0000 (0:00:03.292) 0:08:38.691 ********* 2021-07-31T01:26:54.738 INFO:teuthology.task.ansible.out: TASK [container-host : Check for docker's daemon.json] ************************* 2021-07-31T01:26:54.740 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:26:54.764 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:54 +0000 (0:00:00.052) 0:08:38.743 ********* 2021-07-31T01:26:54.791 INFO:teuthology.task.ansible.out: TASK [container-host : Create /etc/containers/registries.conf if necessary] **** 2021-07-31T01:26:54.793 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:26:54.818 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:54 +0000 (0:00:00.054) 0:08:38.798 ********* 2021-07-31T01:26:55.172 INFO:teuthology.task.ansible.out: TASK [container-host : Add local docker.io registry mirror] ******************** 2021-07-31T01:26:55.175 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => {"changed": true, "cmd": ["registries-conf-ctl", "add-mirror", "docker.io", "docker-mirror.front.sepia.ceph.com:5000"], "delta": "0:00:00.152011", "end": "2021-07-31 01:26:55.144891", "rc": 0, "start": "2021-07-31 01:26:54.992880", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-07-31T01:26:55.194 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:55 +0000 (0:00:00.376) 0:08:39.174 ********* 2021-07-31T01:26:55.222 INFO:teuthology.task.ansible.out: TASK [container-host : Restart docker service] ********************************* 2021-07-31T01:26:55.224 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:26:55.362 INFO:teuthology.task.ansible.out: TASK [container-host : Ensure git is installed] ******************************** ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-07-31T01:26:55.384 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:55 +0000 (0:00:00.189) 0:08:39.364 ********* 2021-07-31T01:26:58.992 INFO:teuthology.task.ansible.out: TASK [container-host : Install registries-conf-ctl] **************************** 2021-07-31T01:26:58.994 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => {"changed": true, "cmd": ["/bin/pip3", "install", "-U", "git+https://github.com/sebastian-philipp/registries-conf-ctl"], "name": ["git+https://github.com/sebastian-philipp/registries-conf-ctl"], "requirements": null, "state": "latest", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl\n Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-u40s83k1-build\nCollecting toml (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl\nCollecting docopt (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz\nInstalling collected packages: toml, docopt, registries-conf-ctl\n Running setup.py install for docopt: started\n Running setup.py install for docopt: finished with status 'done'\n Running setup.py install for registries-conf-ctl: started\n Running setup.py install for registries-conf-ctl: finished with status 'done'\nSuccessfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2\n", "stdout_lines": ["Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl", " Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-u40s83k1-build", "Collecting toml (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl", "Collecting docopt (from registries-conf-ctl==1.0)", " Downloading https://files. 2021-07-31T01:26:58.996 INFO:teuthology.task.ansible.out:pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz", "Installing collected packages: toml, docopt, registries-conf-ctl", " Running setup.py install for docopt: started", " Running setup.py install for docopt: finished with status 'done'", " Running setup.py install for registries-conf-ctl: started", " Running setup.py install for registries-conf-ctl: finished with status 'done'", "Successfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2"], "version": null, "virtualenv": null} 2021-07-31T01:26:59.015 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:59 +0000 (0:00:03.630) 0:08:42.995 ********* 2021-07-31T01:26:59.043 INFO:teuthology.task.ansible.out: TASK [container-host : Check for docker's daemon.json] ************************* skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:26:59.066 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:59 +0000 (0:00:00.050) 0:08:43.046 ********* 2021-07-31T01:26:59.094 INFO:teuthology.task.ansible.out: TASK [container-host : Create /etc/containers/registries.conf if necessary] **** 2021-07-31T01:26:59.096 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:26:59.116 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:59 +0000 (0:00:00.050) 0:08:43.096 ********* 2021-07-31T01:26:59.490 INFO:teuthology.task.ansible.out: TASK [container-host : Add local docker.io registry mirror] ******************** 2021-07-31T01:26:59.494 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => {"changed": true, "cmd": ["registries-conf-ctl", "add-mirror", "docker.io", "docker-mirror.front.sepia.ceph.com:5000"], "delta": "0:00:00.180691", "end": "2021-07-31 01:26:59.471517", "rc": 0, "start": "2021-07-31 01:26:59.290826", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-07-31T01:26:59.516 INFO:teuthology.task.ansible.out:Saturday 31 July 2021 01:26:59 +0000 (0:00:00.400) 0:08:43.496 ********* 2021-07-31T01:26:59.554 INFO:teuthology.task.ansible.out: TASK [container-host : Restart docker service] ********************************* 2021-07-31T01:26:59.556 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-31T01:26:59.627 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** skipping: no hosts matched 2021-07-31T01:26:59.630 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** skipping: no hosts matched 2021-07-31T01:26:59.631 INFO:teuthology.task.ansible.out: PLAY [pulpito] ***************************************************************** skipping: no hosts matched 2021-07-31T01:26:59.632 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2021-07-31T01:26:59.652 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* Saturday 31 July 2021 01:26:59 +0000 (0:00:00.135) 0:08:43.632 ********* 2021-07-31T01:27:00.326 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] 2021-07-31T01:27:01.350 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] 2021-07-31T01:27:01.370 INFO:teuthology.task.ansible.out: TASK [Touch /ceph-qa-ready] **************************************************** Saturday 31 July 2021 01:27:01 +0000 (0:00:01.717) 0:08:45.350 ********* 2021-07-31T01:27:01.585 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_runtime_t:s0", "size": 0, "state": "file", "uid": 0} 2021-07-31T01:27:01.597 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_runtime_t:s0", "size": 0, "state": "file", "uid": 0} 2021-07-31T01:27:01.599 INFO:teuthology.task.ansible.out: PLAY RECAP ********************************************************************* 2021-07-31T01:27:01.601 INFO:teuthology.task.ansible.out:smithi045.front.sepia.ceph.com : ok=142 changed=48 unreachable=0 failed=0 skipped=178 rescued=0 ignored=4 smithi154.front.sepia.ceph.com : ok=142 changed=48 unreachable=0 failed=0 skipped=178 rescued=0 ignored=4 Saturday 31 July 2021 01:27:01 +0000 (0:00:00.230) 0:08:45.580 ********* =============================================================================== Create all admin users with sudo access. ------------------------------- 56.57s common : nrpe - Ensure SELinux policy is up to date -------------------- 43.80s testnode : Install packages -------------------------------------------- 40.29s 2021-07-31T01:27:01.602 INFO:teuthology.task.ansible.out:users : Update authorized_keys using the keys repo --------------------- 32.69s container-host : Install container packages ---------------------------- 18.24s testnode : Ensure ceph packages are not present. ----------------------- 16.15s common : Register with subscription-manager. --------------------------- 13.30s common : Install nrpe package and dependencies (RHEL/CentOS) ----------- 13.22s testnode : Ensure ceph dependency packages are not present. ------------ 12.30s Remove revoked users ---------------------------------------------------- 9.37s container-host : Ensure git is installed -------------------------------- 8.04s common : Install Kerberos Packages (RedHat) ----------------------------- 6.63s common : nrpe - Remove SELinux policy package --------------------------- 6.00s common : nrpe - Load SELinux policy package ----------------------------- 5.97s common : nrpe - Install SELinux dependencies ---------------------------- 4.88s container-host : Install registries-conf-ctl ---------------------------- 3.63s testnode : Remove packages ---------------------------------------------- 3.12s testnode : Rebuild rpmdb ------------------------------------------------ 2.58s users : Update authorized_keys for each user with literal keys ---------- 1.89s testnode : Create logical volume(s) ------------- 2021-07-31T01:27:01.603 INFO:teuthology.task.ansible.out:------------------------ 1.87s 2021-07-31T01:27:01.984 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi045.front.sepia.ceph.com'), Remote(name='ubuntu@smithi154.front.sepia.ceph.com')] 2021-07-31T01:27:01.985 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-07-31T01:27:01.987 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-07-31T01:27:02.104 DEBUG:teuthology.orchestra.run.smithi045:> true 2021-07-31T01:27:02.204 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-07-31T01:27:02.204 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-07-31T01:27:02.274 DEBUG:teuthology.orchestra.run.smithi154:> true 2021-07-31T01:27:02.366 INFO:teuthology.run_tasks:Running task clock... 2021-07-31T01:27:02.383 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2021-07-31T01:27:02.384 INFO:teuthology.orchestra.run:Running command with timeout 360 2021-07-31T01:27:02.386 DEBUG:teuthology.orchestra.run.smithi045:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-07-31T01:27:02.422 INFO:teuthology.orchestra.run.smithi045.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2021-07-31T01:27:02.439 INFO:teuthology.orchestra.run.smithi045.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2021-07-31T01:27:02.476 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: ntpd: command not found 2021-07-31T01:27:02.487 INFO:teuthology.orchestra.run.smithi045.stdout:506 Cannot talk to daemon 2021-07-31T01:27:02.503 INFO:teuthology.orchestra.run.smithi045.stderr:Failed to start ntp.service: Unit ntp.service not found. 2021-07-31T01:27:02.517 INFO:teuthology.orchestra.run.smithi045.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2021-07-31T01:27:02.580 INFO:teuthology.orchestra.run.smithi045.stderr:bash: ntpq: command not found 2021-07-31T01:27:02.583 INFO:teuthology.orchestra.run.smithi045.stdout:210 Number of sources = 4 2021-07-31T01:27:02.583 INFO:teuthology.orchestra.run.smithi045.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2021-07-31T01:27:02.583 INFO:teuthology.orchestra.run.smithi045.stdout:=============================================================================== 2021-07-31T01:27:02.584 INFO:teuthology.orchestra.run.smithi045.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-07-31T01:27:02.585 INFO:teuthology.orchestra.run.smithi045.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-07-31T01:27:02.585 INFO:teuthology.orchestra.run.smithi045.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-07-31T01:27:02.585 INFO:teuthology.orchestra.run.smithi045.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-07-31T01:27:02.586 INFO:teuthology.orchestra.run:Running command with timeout 360 2021-07-31T01:27:02.586 DEBUG:teuthology.orchestra.run.smithi154:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-07-31T01:27:02.619 INFO:teuthology.orchestra.run.smithi154.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2021-07-31T01:27:02.635 INFO:teuthology.orchestra.run.smithi154.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2021-07-31T01:27:02.673 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: ntpd: command not found 2021-07-31T01:27:02.683 INFO:teuthology.orchestra.run.smithi154.stdout:506 Cannot talk to daemon 2021-07-31T01:27:02.696 INFO:teuthology.orchestra.run.smithi154.stderr:Failed to start ntp.service: Unit ntp.service not found. 2021-07-31T01:27:02.710 INFO:teuthology.orchestra.run.smithi154.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2021-07-31T01:27:02.768 INFO:teuthology.orchestra.run.smithi154.stderr:bash: ntpq: command not found 2021-07-31T01:27:02.771 INFO:teuthology.orchestra.run.smithi154.stdout:210 Number of sources = 4 2021-07-31T01:27:02.772 INFO:teuthology.orchestra.run.smithi154.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2021-07-31T01:27:02.772 INFO:teuthology.orchestra.run.smithi154.stdout:=============================================================================== 2021-07-31T01:27:02.772 INFO:teuthology.orchestra.run.smithi154.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-07-31T01:27:02.772 INFO:teuthology.orchestra.run.smithi154.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-07-31T01:27:02.773 INFO:teuthology.orchestra.run.smithi154.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-07-31T01:27:02.773 INFO:teuthology.orchestra.run.smithi154.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-07-31T01:27:02.773 INFO:teuthology.run_tasks:Running task pexec... 2021-07-31T01:27:02.783 INFO:teuthology.task.pexec:Executing custom commands... 2021-07-31T01:27:02.784 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi045.front.sepia.ceph.com 2021-07-31T01:27:02.785 DEBUG:teuthology.orchestra.run.smithi045:> TESTDIR=/home/ubuntu/cephtest bash -s 2021-07-31T01:27:02.785 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi154.front.sepia.ceph.com 2021-07-31T01:27:02.786 DEBUG:teuthology.orchestra.run.smithi154:> TESTDIR=/home/ubuntu/cephtest bash -s 2021-07-31T01:27:03.080 INFO:teuthology.orchestra.run.smithi045.stdout:Updating Subscription Management repositories. 2021-07-31T01:27:03.088 INFO:teuthology.orchestra.run.smithi154.stdout:Updating Subscription Management repositories. 2021-07-31T01:27:03.581 INFO:teuthology.orchestra.run.smithi154.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 97 kB/s | 2.8 kB 00:00 2021-07-31T01:27:03.625 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 78 kB/s | 2.8 kB 00:00 2021-07-31T01:27:03.727 INFO:teuthology.orchestra.run.smithi154.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 84 kB/s | 2.4 kB 00:00 2021-07-31T01:27:03.788 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 63 kB/s | 2.4 kB 00:00 2021-07-31T01:27:05.496 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2021-07-31T01:27:05.508 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2021-07-31T01:27:05.508 INFO:teuthology.orchestra.run.smithi154.stdout: Package Architecture Version Repository Size 2021-07-31T01:27:05.509 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2021-07-31T01:27:05.509 INFO:teuthology.orchestra.run.smithi154.stdout:Disabling modules: 2021-07-31T01:27:05.509 INFO:teuthology.orchestra.run.smithi154.stdout: container-tools 2021-07-31T01:27:05.509 INFO:teuthology.orchestra.run.smithi154.stdout: 2021-07-31T01:27:05.510 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2021-07-31T01:27:05.510 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2021-07-31T01:27:05.510 INFO:teuthology.orchestra.run.smithi154.stdout: 2021-07-31T01:27:05.648 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2021-07-31T01:27:05.657 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2021-07-31T01:27:05.658 INFO:teuthology.orchestra.run.smithi045.stdout: Package Architecture Version Repository Size 2021-07-31T01:27:05.658 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2021-07-31T01:27:05.659 INFO:teuthology.orchestra.run.smithi045.stdout:Disabling modules: 2021-07-31T01:27:05.659 INFO:teuthology.orchestra.run.smithi045.stdout: container-tools 2021-07-31T01:27:05.660 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:27:05.660 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2021-07-31T01:27:05.660 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2021-07-31T01:27:05.661 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:27:16.035 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:00:04 ago on Sat 31 Jul 2021 01:26:31 AM UTC. 2021-07-31T01:27:16.373 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:05 ago on Sat 31 Jul 2021 01:26:34 AM UTC. 2021-07-31T01:27:18.180 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2021-07-31T01:27:18.525 INFO:teuthology.orchestra.run.smithi154.stdout:Updating Subscription Management repositories. 2021-07-31T01:27:18.620 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2021-07-31T01:27:19.030 INFO:teuthology.orchestra.run.smithi154.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 94 kB/s | 2.8 kB 00:00 2021-07-31T01:27:19.031 INFO:teuthology.orchestra.run.smithi045.stdout:Updating Subscription Management repositories. 2021-07-31T01:27:19.577 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 75 kB/s | 2.8 kB 00:00 2021-07-31T01:27:23.047 INFO:teuthology.orchestra.run.smithi154.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 80 kB/s | 2.4 kB 00:00 2021-07-31T01:27:23.992 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 65 kB/s | 2.4 kB 00:00 2021-07-31T01:27:27.257 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:00:03 ago on Sat 31 Jul 2021 01:23:28 AM UTC. 2021-07-31T01:27:28.612 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:04 ago on Sat 31 Jul 2021 01:23:39 AM UTC. 2021-07-31T01:27:29.055 INFO:teuthology.orchestra.run.smithi154.stdout:Package dnf-plugins-core-4.0.17-5.el8.noarch is already installed. 2021-07-31T01:27:29.407 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2021-07-31T01:27:29.408 INFO:teuthology.orchestra.run.smithi154.stdout:Nothing to do. 2021-07-31T01:27:29.408 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2021-07-31T01:27:29.742 INFO:teuthology.orchestra.run.smithi154.stdout:Updating Subscription Management repositories. 2021-07-31T01:27:30.072 INFO:teuthology.orchestra.run.smithi154.stdout:Repository successfully enabled. 2021-07-31T01:27:30.077 INFO:teuthology.orchestra.run.smithi154.stderr:Enabling a Copr repository. Please note that this repository is not part 2021-07-31T01:27:30.078 INFO:teuthology.orchestra.run.smithi154.stderr:of the main distribution, and quality may vary. 2021-07-31T01:27:30.078 INFO:teuthology.orchestra.run.smithi154.stderr: 2021-07-31T01:27:30.078 INFO:teuthology.orchestra.run.smithi154.stderr:The Fedora Project does not exercise any power over the contents of 2021-07-31T01:27:30.078 INFO:teuthology.orchestra.run.smithi154.stderr:this repository beyond the rules outlined in the Copr FAQ at 2021-07-31T01:27:30.079 INFO:teuthology.orchestra.run.smithi154.stderr:, 2021-07-31T01:27:30.079 INFO:teuthology.orchestra.run.smithi154.stderr:and packages are not held to any quality or security level. 2021-07-31T01:27:30.079 INFO:teuthology.orchestra.run.smithi154.stderr: 2021-07-31T01:27:30.079 INFO:teuthology.orchestra.run.smithi154.stderr:Please do not file bug reports about these packages in Fedora 2021-07-31T01:27:30.079 INFO:teuthology.orchestra.run.smithi154.stderr:Bugzilla. In case of problems, contact the owner of this repository. 2021-07-31T01:27:30.136 INFO:teuthology.orchestra.run.smithi154.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-07-31T01:27:30.136 INFO:teuthology.orchestra.run.smithi154.stderr: Dload Upload Total Spent Left Speed 2021-07-31T01:27:30.470 INFO:teuthology.orchestra.run.smithi045.stdout:Package dnf-plugins-core-4.0.17-5.el8.noarch is already installed. 2021-07-31T01:27:30.486 INFO:teuthology.orchestra.run.smithi154.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 359 100 359 0 0 1025 0 --:--:-- --:--:-- --:--:-- 1025 2021-07-31T01:27:30.739 INFO:teuthology.orchestra.run.smithi154.stdout:Updating Subscription Management repositories. 2021-07-31T01:27:30.835 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2021-07-31T01:27:30.836 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2021-07-31T01:27:30.836 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2021-07-31T01:27:31.180 INFO:teuthology.orchestra.run.smithi045.stdout:Updating Subscription Management repositories. 2021-07-31T01:27:31.201 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2021-07-31T01:27:31.205 INFO:teuthology.orchestra.run.smithi154.stdout:========================================================================================================================== 2021-07-31T01:27:31.205 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repository Size 2021-07-31T01:27:31.205 INFO:teuthology.orchestra.run.smithi154.stdout:========================================================================================================================== 2021-07-31T01:27:31.205 INFO:teuthology.orchestra.run.smithi154.stdout:Removing: 2021-07-31T01:27:31.206 INFO:teuthology.orchestra.run.smithi154.stdout: podman x86_64 2.2.1-7.module+el8.3.1+9857+68fb1526 @rhel-8-for-x86_64-appstream-rpms 52 M 2021-07-31T01:27:31.206 INFO:teuthology.orchestra.run.smithi154.stdout:Removing dependent packages: 2021-07-31T01:27:31.206 INFO:teuthology.orchestra.run.smithi154.stdout: podman-docker noarch 2.2.1-7.module+el8.3.1+9857+68fb1526 @rhel-8-for-x86_64-appstream-rpms 5.5 k 2021-07-31T01:27:31.206 INFO:teuthology.orchestra.run.smithi154.stdout:Removing unused dependencies: 2021-07-31T01:27:31.207 INFO:teuthology.orchestra.run.smithi154.stdout: conmon x86_64 2:2.0.20-2.module+el8.3.0+8221+97165c3f @rhel-8-for-x86_64-appstream-rpms 118 k 2021-07-31T01:27:31.208 INFO:teuthology.orchestra.run.smithi154.stdout: container-selinux noarch 2:2.144.0-1.module+el8.3.0+8221+97165c3f @rhel-8-for-x86_64-appstream-rpms 46 k 2021-07-31T01:27:31.209 INFO:teuthology.orchestra.run.smithi154.stdout: containernetworking-plugins x86_64 0.8.6-2.module+el8.3.0+8221+97165c3f @rhel-8-for-x86_64-appstream-rpms 66 M 2021-07-31T01:27:31.209 INFO:teuthology.orchestra.run.smithi154.stdout: containers-common x86_64 1:1.1.1-3.module+el8.3.0+8221+97165c3f @rhel-8-for-x86_64-appstream-rpms 69 k 2021-07-31T01:27:31.209 INFO:teuthology.orchestra.run.smithi154.stdout: criu x86_64 3.14-2.module+el8.3.0+8221+97165c3f @rhel-8-for-x86_64-appstream-rpms 1.3 M 2021-07-31T01:27:31.209 INFO:teuthology.orchestra.run.smithi154.stdout: fuse-overlayfs x86_64 1.1.2-3.module+el8.3.0+8221+97165c3f @rhel-8-for-x86_64-appstream-rpms 134 k 2021-07-31T01:27:31.210 INFO:teuthology.orchestra.run.smithi154.stdout: fuse3-libs x86_64 3.2.1-12.el8 @rhel-8-for-x86_64-baseos-rpms 279 k 2021-07-31T01:27:31.210 INFO:teuthology.orchestra.run.smithi154.stdout: libnet x86_64 1.1.6-15.el8 @rhel-8-for-x86_64-appstream-rpms 166 k 2021-07-31T01:27:31.210 INFO:teuthology.orchestra.run.smithi154.stdout: libslirp x86_64 4.3.1-1.module+el8.3.0+8221+97165c3f @rhel-8-for-x86_64-appstream-rpms 129 k 2021-07-31T01:27:31.210 INFO:teuthology.orchestra.run.smithi154.stdout: libvarlink x86_64 18-3.el8 @rhel-8-for-x86_64-baseos-rpms 117 k 2021-07-31T01:27:31.211 INFO:teuthology.orchestra.run.smithi154.stdout: podman-catatonit x86_64 2.2.1-7.module+el8.3.1+9857+68fb1526 @rhel-8-for-x86_64-appstream-rpms 753 k 2021-07-31T01:27:31.211 INFO:teuthology.orchestra.run.smithi154.stdout: protobuf-c x86_64 1.3.0-4.el8 @rhel-8-for-x86_64-appstream-rpms 56 k 2021-07-31T01:27:31.211 INFO:teuthology.orchestra.run.smithi154.stdout: runc x86_64 1.0.0-70.rc92.module+el8.3.1+9857+68fb1526 @rhel-8-for-x86_64-appstream-rpms 13 M 2021-07-31T01:27:31.211 INFO:teuthology.orchestra.run.smithi154.stdout: slirp4netns x86_64 1.1.4-2.module+el8.3.0+8221+97165c3f @rhel-8-for-x86_64-appstream-rpms 95 k 2021-07-31T01:27:31.212 INFO:teuthology.orchestra.run.smithi154.stdout: 2021-07-31T01:27:31.212 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2021-07-31T01:27:31.212 INFO:teuthology.orchestra.run.smithi154.stdout:========================================================================================================================== 2021-07-31T01:27:31.212 INFO:teuthology.orchestra.run.smithi154.stdout:Remove 16 Packages 2021-07-31T01:27:31.212 INFO:teuthology.orchestra.run.smithi154.stdout: 2021-07-31T01:27:31.213 INFO:teuthology.orchestra.run.smithi154.stdout:Freed space: 134 M 2021-07-31T01:27:31.214 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2021-07-31T01:27:31.214 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2021-07-31T01:27:31.214 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2021-07-31T01:27:31.316 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2021-07-31T01:27:31.317 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2021-07-31T01:27:31.511 INFO:teuthology.orchestra.run.smithi045.stdout:Repository successfully enabled. 2021-07-31T01:27:31.516 INFO:teuthology.orchestra.run.smithi045.stderr:Enabling a Copr repository. Please note that this repository is not part 2021-07-31T01:27:31.516 INFO:teuthology.orchestra.run.smithi045.stderr:of the main distribution, and quality may vary. 2021-07-31T01:27:31.517 INFO:teuthology.orchestra.run.smithi045.stderr: 2021-07-31T01:27:31.517 INFO:teuthology.orchestra.run.smithi045.stderr:The Fedora Project does not exercise any power over the contents of 2021-07-31T01:27:31.517 INFO:teuthology.orchestra.run.smithi045.stderr:this repository beyond the rules outlined in the Copr FAQ at 2021-07-31T01:27:31.517 INFO:teuthology.orchestra.run.smithi045.stderr:, 2021-07-31T01:27:31.518 INFO:teuthology.orchestra.run.smithi045.stderr:and packages are not held to any quality or security level. 2021-07-31T01:27:31.518 INFO:teuthology.orchestra.run.smithi045.stderr: 2021-07-31T01:27:31.518 INFO:teuthology.orchestra.run.smithi045.stderr:Please do not file bug reports about these packages in Fedora 2021-07-31T01:27:31.518 INFO:teuthology.orchestra.run.smithi045.stderr:Bugzilla. In case of problems, contact the owner of this repository. 2021-07-31T01:27:31.520 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2021-07-31T01:27:31.536 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: podman-docker-2.2.1-7.module+el8.3.1+9857+68fb1526.n 1/1 2021-07-31T01:27:31.579 INFO:teuthology.orchestra.run.smithi045.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-07-31T01:27:31.580 INFO:teuthology.orchestra.run.smithi045.stderr: Dload Upload Total Spent Left Speed 2021-07-31T01:27:31.668 INFO:teuthology.orchestra.run.smithi154.stdout: Erasing : podman-docker-2.2.1-7.module+el8.3.1+9857+68fb1526 1/16 2021-07-31T01:27:31.668 INFO:teuthology.orchestra.run.smithi154.stdout: Erasing : podman-2.2.1-7.module+el8.3.1+9857+68fb1526.x86_64 2/16 2021-07-31T01:27:31.901 INFO:teuthology.orchestra.run.smithi045.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 359 100 359 0 0 1104 0 --:--:-- --:--:-- --:--:-- 1104 2021-07-31T01:27:31.969 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: podman-2.2.1-7.module+el8.3.1+9857+68fb1526.x86_64 2/16 2021-07-31T01:27:31.969 INFO:teuthology.orchestra.run.smithi154.stdout: Erasing : containers-common-1:1.1.1-3.module+el8.3.0+8221+97 3/16 2021-07-31T01:27:31.969 INFO:teuthology.orchestra.run.smithi154.stdout:warning: /etc/containers/registries.conf saved as /etc/containers/registries.conf.rpmsave 2021-07-31T01:27:31.970 INFO:teuthology.orchestra.run.smithi154.stdout: 2021-07-31T01:27:32.166 INFO:teuthology.orchestra.run.smithi154.stdout: Erasing : container-selinux-2:2.144.0-1.module+el8.3.0+8221+ 4/16 2021-07-31T01:27:32.171 INFO:teuthology.orchestra.run.smithi045.stdout:Updating Subscription Management repositories. 2021-07-31T01:27:32.671 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2021-07-31T01:27:32.675 INFO:teuthology.orchestra.run.smithi045.stdout:========================================================================================================================== 2021-07-31T01:27:32.675 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2021-07-31T01:27:32.676 INFO:teuthology.orchestra.run.smithi045.stdout:========================================================================================================================== 2021-07-31T01:27:32.676 INFO:teuthology.orchestra.run.smithi045.stdout:Removing: 2021-07-31T01:27:32.676 INFO:teuthology.orchestra.run.smithi045.stdout: podman x86_64 2.2.1-7.module+el8.3.1+9857+68fb1526 @rhel-8-for-x86_64-appstream-rpms 52 M 2021-07-31T01:27:32.676 INFO:teuthology.orchestra.run.smithi045.stdout:Removing dependent packages: 2021-07-31T01:27:32.677 INFO:teuthology.orchestra.run.smithi045.stdout: podman-docker noarch 2.2.1-7.module+el8.3.1+9857+68fb1526 @rhel-8-for-x86_64-appstream-rpms 5.5 k 2021-07-31T01:27:32.677 INFO:teuthology.orchestra.run.smithi045.stdout:Removing unused dependencies: 2021-07-31T01:27:32.678 INFO:teuthology.orchestra.run.smithi045.stdout: conmon x86_64 2:2.0.20-2.module+el8.3.0+8221+97165c3f @rhel-8-for-x86_64-appstream-rpms 118 k 2021-07-31T01:27:32.678 INFO:teuthology.orchestra.run.smithi045.stdout: container-selinux noarch 2:2.144.0-1.module+el8.3.0+8221+97165c3f @rhel-8-for-x86_64-appstream-rpms 46 k 2021-07-31T01:27:32.678 INFO:teuthology.orchestra.run.smithi045.stdout: containernetworking-plugins x86_64 0.8.6-2.module+el8.3.0+8221+97165c3f @rhel-8-for-x86_64-appstream-rpms 66 M 2021-07-31T01:27:32.679 INFO:teuthology.orchestra.run.smithi045.stdout: containers-common x86_64 1:1.1.1-3.module+el8.3.0+8221+97165c3f @rhel-8-for-x86_64-appstream-rpms 69 k 2021-07-31T01:27:32.679 INFO:teuthology.orchestra.run.smithi045.stdout: criu x86_64 3.14-2.module+el8.3.0+8221+97165c3f @rhel-8-for-x86_64-appstream-rpms 1.3 M 2021-07-31T01:27:32.680 INFO:teuthology.orchestra.run.smithi045.stdout: fuse-overlayfs x86_64 1.1.2-3.module+el8.3.0+8221+97165c3f @rhel-8-for-x86_64-appstream-rpms 134 k 2021-07-31T01:27:32.680 INFO:teuthology.orchestra.run.smithi045.stdout: fuse3-libs x86_64 3.2.1-12.el8 @rhel-8-for-x86_64-baseos-rpms 279 k 2021-07-31T01:27:32.681 INFO:teuthology.orchestra.run.smithi045.stdout: libnet x86_64 1.1.6-15.el8 @rhel-8-for-x86_64-appstream-rpms 166 k 2021-07-31T01:27:32.681 INFO:teuthology.orchestra.run.smithi045.stdout: libslirp x86_64 4.3.1-1.module+el8.3.0+8221+97165c3f @rhel-8-for-x86_64-appstream-rpms 129 k 2021-07-31T01:27:32.682 INFO:teuthology.orchestra.run.smithi045.stdout: libvarlink x86_64 18-3.el8 @rhel-8-for-x86_64-baseos-rpms 117 k 2021-07-31T01:27:32.682 INFO:teuthology.orchestra.run.smithi045.stdout: podman-catatonit x86_64 2.2.1-7.module+el8.3.1+9857+68fb1526 @rhel-8-for-x86_64-appstream-rpms 753 k 2021-07-31T01:27:32.683 INFO:teuthology.orchestra.run.smithi045.stdout: protobuf-c x86_64 1.3.0-4.el8 @rhel-8-for-x86_64-appstream-rpms 56 k 2021-07-31T01:27:32.683 INFO:teuthology.orchestra.run.smithi045.stdout: runc x86_64 1.0.0-70.rc92.module+el8.3.1+9857+68fb1526 @rhel-8-for-x86_64-appstream-rpms 13 M 2021-07-31T01:27:32.683 INFO:teuthology.orchestra.run.smithi045.stdout: slirp4netns x86_64 1.1.4-2.module+el8.3.0+8221+97165c3f @rhel-8-for-x86_64-appstream-rpms 95 k 2021-07-31T01:27:32.684 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:27:32.684 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2021-07-31T01:27:32.685 INFO:teuthology.orchestra.run.smithi045.stdout:========================================================================================================================== 2021-07-31T01:27:32.685 INFO:teuthology.orchestra.run.smithi045.stdout:Remove 16 Packages 2021-07-31T01:27:32.685 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:27:32.686 INFO:teuthology.orchestra.run.smithi045.stdout:Freed space: 134 M 2021-07-31T01:27:32.687 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2021-07-31T01:27:32.687 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2021-07-31T01:27:32.688 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2021-07-31T01:27:32.786 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2021-07-31T01:27:32.787 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2021-07-31T01:27:33.028 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2021-07-31T01:27:33.045 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: podman-docker-2.2.1-7.module+el8.3.1+9857+68fb1526.n 1/1 2021-07-31T01:27:33.229 INFO:teuthology.orchestra.run.smithi045.stdout: Erasing : podman-docker-2.2.1-7.module+el8.3.1+9857+68fb1526 1/16 2021-07-31T01:27:33.229 INFO:teuthology.orchestra.run.smithi045.stdout: Erasing : podman-2.2.1-7.module+el8.3.1+9857+68fb1526.x86_64 2/16 2021-07-31T01:27:33.561 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: podman-2.2.1-7.module+el8.3.1+9857+68fb1526.x86_64 2/16 2021-07-31T01:27:33.561 INFO:teuthology.orchestra.run.smithi045.stdout: Erasing : containers-common-1:1.1.1-3.module+el8.3.0+8221+97 3/16 2021-07-31T01:27:33.562 INFO:teuthology.orchestra.run.smithi045.stdout:warning: /etc/containers/registries.conf saved as /etc/containers/registries.conf.rpmsave 2021-07-31T01:27:33.562 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:27:33.825 INFO:teuthology.orchestra.run.smithi045.stdout: Erasing : container-selinux-2:2.144.0-1.module+el8.3.0+8221+ 4/16 2021-07-31T01:27:48.400 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: container-selinux-2:2.144.0-1.module+el8.3.0+8221+ 4/16 2021-07-31T01:27:48.522 INFO:teuthology.orchestra.run.smithi154.stdout: Erasing : podman-catatonit-2.2.1-7.module+el8.3.1+9857+68fb1 5/16 2021-07-31T01:27:48.738 INFO:teuthology.orchestra.run.smithi154.stdout: Erasing : fuse-overlayfs-1.1.2-3.module+el8.3.0+8221+97165c3 6/16 2021-07-31T01:27:48.922 INFO:teuthology.orchestra.run.smithi154.stdout: Erasing : slirp4netns-1.1.4-2.module+el8.3.0+8221+97165c3f.x 7/16 2021-07-31T01:27:49.091 INFO:teuthology.orchestra.run.smithi154.stdout: Erasing : runc-1.0.0-70.rc92.module+el8.3.1+9857+68fb1526.x8 8/16 2021-07-31T01:27:49.265 INFO:teuthology.orchestra.run.smithi154.stdout: Erasing : criu-3.14-2.module+el8.3.0+8221+97165c3f.x86_64 9/16 2021-07-31T01:27:49.265 INFO:teuthology.orchestra.run.smithi154.stdout: Erasing : libnet-1.1.6-15.el8.x86_64 10/16 2021-07-31T01:27:49.514 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: libnet-1.1.6-15.el8.x86_64 10/16 2021-07-31T01:27:49.690 INFO:teuthology.orchestra.run.smithi154.stdout: Erasing : protobuf-c-1.3.0-4.el8.x86_64 11/16 2021-07-31T01:27:49.872 INFO:teuthology.orchestra.run.smithi154.stdout: Erasing : libslirp-4.3.1-1.module+el8.3.0+8221+97165c3f.x86_ 12/16 2021-07-31T01:27:49.872 INFO:teuthology.orchestra.run.smithi154.stdout: Erasing : fuse3-libs-3.2.1-12.el8.x86_64 13/16 2021-07-31T01:27:50.140 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: fuse3-libs-3.2.1-12.el8.x86_64 13/16 2021-07-31T01:27:50.333 INFO:teuthology.orchestra.run.smithi154.stdout: Erasing : conmon-2:2.0.20-2.module+el8.3.0+8221+97165c3f.x86 14/16 2021-07-31T01:27:50.519 INFO:teuthology.orchestra.run.smithi154.stdout: Erasing : containernetworking-plugins-0.8.6-2.module+el8.3.0 15/16 2021-07-31T01:27:50.519 INFO:teuthology.orchestra.run.smithi154.stdout: Erasing : libvarlink-18-3.el8.x86_64 16/16 2021-07-31T01:27:50.791 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: container-selinux-2:2.144.0-1.module+el8.3.0+8221+ 4/16 2021-07-31T01:27:50.957 INFO:teuthology.orchestra.run.smithi045.stdout: Erasing : podman-catatonit-2.2.1-7.module+el8.3.1+9857+68fb1 5/16 2021-07-31T01:27:50.978 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: libvarlink-18-3.el8.x86_64 16/16 2021-07-31T01:27:50.978 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : conmon-2:2.0.20-2.module+el8.3.0+8221+97165c3f.x86 1/16 2021-07-31T01:27:50.978 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : container-selinux-2:2.144.0-1.module+el8.3.0+8221+ 2/16 2021-07-31T01:27:50.979 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : containernetworking-plugins-0.8.6-2.module+el8.3.0 3/16 2021-07-31T01:27:50.979 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : containers-common-1:1.1.1-3.module+el8.3.0+8221+97 4/16 2021-07-31T01:27:50.979 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : criu-3.14-2.module+el8.3.0+8221+97165c3f.x86_64 5/16 2021-07-31T01:27:50.979 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : fuse-overlayfs-1.1.2-3.module+el8.3.0+8221+97165c3 6/16 2021-07-31T01:27:50.979 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : fuse3-libs-3.2.1-12.el8.x86_64 7/16 2021-07-31T01:27:50.980 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libnet-1.1.6-15.el8.x86_64 8/16 2021-07-31T01:27:50.980 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libslirp-4.3.1-1.module+el8.3.0+8221+97165c3f.x86_ 9/16 2021-07-31T01:27:50.980 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libvarlink-18-3.el8.x86_64 10/16 2021-07-31T01:27:50.980 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : podman-2.2.1-7.module+el8.3.1+9857+68fb1526.x86_64 11/16 2021-07-31T01:27:50.981 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : podman-catatonit-2.2.1-7.module+el8.3.1+9857+68fb1 12/16 2021-07-31T01:27:50.981 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : podman-docker-2.2.1-7.module+el8.3.1+9857+68fb1526 13/16 2021-07-31T01:27:50.981 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : protobuf-c-1.3.0-4.el8.x86_64 14/16 2021-07-31T01:27:50.981 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : runc-1.0.0-70.rc92.module+el8.3.1+9857+68fb1526.x8 15/16 2021-07-31T01:27:51.173 INFO:teuthology.orchestra.run.smithi045.stdout: Erasing : fuse-overlayfs-1.1.2-3.module+el8.3.0+8221+97165c3 6/16 2021-07-31T01:27:51.357 INFO:teuthology.orchestra.run.smithi045.stdout: Erasing : slirp4netns-1.1.4-2.module+el8.3.0+8221+97165c3f.x 7/16 2021-07-31T01:27:51.524 INFO:teuthology.orchestra.run.smithi045.stdout: Erasing : runc-1.0.0-70.rc92.module+el8.3.1+9857+68fb1526.x8 8/16 2021-07-31T01:27:51.717 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : slirp4netns-1.1.4-2.module+el8.3.0+8221+97165c3f.x 16/16 2021-07-31T01:27:51.718 INFO:teuthology.orchestra.run.smithi154.stdout:Installed products updated. 2021-07-31T01:27:51.727 INFO:teuthology.orchestra.run.smithi045.stdout: Erasing : criu-3.14-2.module+el8.3.0+8221+97165c3f.x86_64 9/16 2021-07-31T01:27:51.727 INFO:teuthology.orchestra.run.smithi045.stdout: Erasing : libnet-1.1.6-15.el8.x86_64 10/16 2021-07-31T01:27:51.965 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: libnet-1.1.6-15.el8.x86_64 10/16 2021-07-31T01:27:52.149 INFO:teuthology.orchestra.run.smithi045.stdout: Erasing : protobuf-c-1.3.0-4.el8.x86_64 11/16 2021-07-31T01:27:52.355 INFO:teuthology.orchestra.run.smithi045.stdout: Erasing : libslirp-4.3.1-1.module+el8.3.0+8221+97165c3f.x86_ 12/16 2021-07-31T01:27:52.356 INFO:teuthology.orchestra.run.smithi045.stdout: Erasing : fuse3-libs-3.2.1-12.el8.x86_64 13/16 2021-07-31T01:27:52.606 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: fuse3-libs-3.2.1-12.el8.x86_64 13/16 2021-07-31T01:27:52.798 INFO:teuthology.orchestra.run.smithi045.stdout: Erasing : conmon-2:2.0.20-2.module+el8.3.0+8221+97165c3f.x86 14/16 2021-07-31T01:27:53.002 INFO:teuthology.orchestra.run.smithi045.stdout: Erasing : containernetworking-plugins-0.8.6-2.module+el8.3.0 15/16 2021-07-31T01:27:53.003 INFO:teuthology.orchestra.run.smithi045.stdout: Erasing : libvarlink-18-3.el8.x86_64 16/16 2021-07-31T01:27:53.537 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: libvarlink-18-3.el8.x86_64 16/16 2021-07-31T01:27:53.537 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : conmon-2:2.0.20-2.module+el8.3.0+8221+97165c3f.x86 1/16 2021-07-31T01:27:53.538 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : container-selinux-2:2.144.0-1.module+el8.3.0+8221+ 2/16 2021-07-31T01:27:53.538 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : containernetworking-plugins-0.8.6-2.module+el8.3.0 3/16 2021-07-31T01:27:53.538 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : containers-common-1:1.1.1-3.module+el8.3.0+8221+97 4/16 2021-07-31T01:27:53.538 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : criu-3.14-2.module+el8.3.0+8221+97165c3f.x86_64 5/16 2021-07-31T01:27:53.539 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : fuse-overlayfs-1.1.2-3.module+el8.3.0+8221+97165c3 6/16 2021-07-31T01:27:53.539 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : fuse3-libs-3.2.1-12.el8.x86_64 7/16 2021-07-31T01:27:53.539 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libnet-1.1.6-15.el8.x86_64 8/16 2021-07-31T01:27:53.539 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libslirp-4.3.1-1.module+el8.3.0+8221+97165c3f.x86_ 9/16 2021-07-31T01:27:53.540 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libvarlink-18-3.el8.x86_64 10/16 2021-07-31T01:27:53.540 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : podman-2.2.1-7.module+el8.3.1+9857+68fb1526.x86_64 11/16 2021-07-31T01:27:53.540 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : podman-catatonit-2.2.1-7.module+el8.3.1+9857+68fb1 12/16 2021-07-31T01:27:53.541 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : podman-docker-2.2.1-7.module+el8.3.1+9857+68fb1526 13/16 2021-07-31T01:27:53.542 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : protobuf-c-1.3.0-4.el8.x86_64 14/16 2021-07-31T01:27:53.542 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : runc-1.0.0-70.rc92.module+el8.3.1+9857+68fb1526.x8 15/16 2021-07-31T01:27:54.298 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : slirp4netns-1.1.4-2.module+el8.3.0+8221+97165c3f.x 16/16 2021-07-31T01:27:54.298 INFO:teuthology.orchestra.run.smithi045.stdout:Installed products updated. 2021-07-31T01:28:08.209 INFO:teuthology.orchestra.run.smithi154.stdout: 2021-07-31T01:28:08.210 INFO:teuthology.orchestra.run.smithi154.stdout:Removed: 2021-07-31T01:28:08.210 INFO:teuthology.orchestra.run.smithi154.stdout: conmon-2:2.0.20-2.module+el8.3.0+8221+97165c3f.x86_64 2021-07-31T01:28:08.210 INFO:teuthology.orchestra.run.smithi154.stdout: container-selinux-2:2.144.0-1.module+el8.3.0+8221+97165c3f.noarch 2021-07-31T01:28:08.211 INFO:teuthology.orchestra.run.smithi154.stdout: containernetworking-plugins-0.8.6-2.module+el8.3.0+8221+97165c3f.x86_64 2021-07-31T01:28:08.211 INFO:teuthology.orchestra.run.smithi154.stdout: containers-common-1:1.1.1-3.module+el8.3.0+8221+97165c3f.x86_64 2021-07-31T01:28:08.211 INFO:teuthology.orchestra.run.smithi154.stdout: criu-3.14-2.module+el8.3.0+8221+97165c3f.x86_64 2021-07-31T01:28:08.211 INFO:teuthology.orchestra.run.smithi154.stdout: fuse-overlayfs-1.1.2-3.module+el8.3.0+8221+97165c3f.x86_64 2021-07-31T01:28:08.212 INFO:teuthology.orchestra.run.smithi154.stdout: fuse3-libs-3.2.1-12.el8.x86_64 2021-07-31T01:28:08.212 INFO:teuthology.orchestra.run.smithi154.stdout: libnet-1.1.6-15.el8.x86_64 2021-07-31T01:28:08.212 INFO:teuthology.orchestra.run.smithi154.stdout: libslirp-4.3.1-1.module+el8.3.0+8221+97165c3f.x86_64 2021-07-31T01:28:08.212 INFO:teuthology.orchestra.run.smithi154.stdout: libvarlink-18-3.el8.x86_64 2021-07-31T01:28:08.212 INFO:teuthology.orchestra.run.smithi154.stdout: podman-2.2.1-7.module+el8.3.1+9857+68fb1526.x86_64 2021-07-31T01:28:08.213 INFO:teuthology.orchestra.run.smithi154.stdout: podman-catatonit-2.2.1-7.module+el8.3.1+9857+68fb1526.x86_64 2021-07-31T01:28:08.213 INFO:teuthology.orchestra.run.smithi154.stdout: podman-docker-2.2.1-7.module+el8.3.1+9857+68fb1526.noarch 2021-07-31T01:28:08.213 INFO:teuthology.orchestra.run.smithi154.stdout: protobuf-c-1.3.0-4.el8.x86_64 2021-07-31T01:28:08.213 INFO:teuthology.orchestra.run.smithi154.stdout: runc-1.0.0-70.rc92.module+el8.3.1+9857+68fb1526.x86_64 2021-07-31T01:28:08.213 INFO:teuthology.orchestra.run.smithi154.stdout: slirp4netns-1.1.4-2.module+el8.3.0+8221+97165c3f.x86_64 2021-07-31T01:28:08.214 INFO:teuthology.orchestra.run.smithi154.stdout: 2021-07-31T01:28:08.214 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2021-07-31T01:28:08.591 INFO:teuthology.orchestra.run.smithi154.stdout:Updating Subscription Management repositories. 2021-07-31T01:28:09.091 INFO:teuthology.orchestra.run.smithi154.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 98 kB/s | 2.8 kB 00:00 2021-07-31T01:28:11.492 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:28:11.492 INFO:teuthology.orchestra.run.smithi045.stdout:Removed: 2021-07-31T01:28:11.493 INFO:teuthology.orchestra.run.smithi045.stdout: conmon-2:2.0.20-2.module+el8.3.0+8221+97165c3f.x86_64 2021-07-31T01:28:11.494 INFO:teuthology.orchestra.run.smithi045.stdout: container-selinux-2:2.144.0-1.module+el8.3.0+8221+97165c3f.noarch 2021-07-31T01:28:11.494 INFO:teuthology.orchestra.run.smithi045.stdout: containernetworking-plugins-0.8.6-2.module+el8.3.0+8221+97165c3f.x86_64 2021-07-31T01:28:11.495 INFO:teuthology.orchestra.run.smithi045.stdout: containers-common-1:1.1.1-3.module+el8.3.0+8221+97165c3f.x86_64 2021-07-31T01:28:11.495 INFO:teuthology.orchestra.run.smithi045.stdout: criu-3.14-2.module+el8.3.0+8221+97165c3f.x86_64 2021-07-31T01:28:11.496 INFO:teuthology.orchestra.run.smithi045.stdout: fuse-overlayfs-1.1.2-3.module+el8.3.0+8221+97165c3f.x86_64 2021-07-31T01:28:11.498 INFO:teuthology.orchestra.run.smithi045.stdout: fuse3-libs-3.2.1-12.el8.x86_64 2021-07-31T01:28:11.499 INFO:teuthology.orchestra.run.smithi045.stdout: libnet-1.1.6-15.el8.x86_64 2021-07-31T01:28:11.499 INFO:teuthology.orchestra.run.smithi045.stdout: libslirp-4.3.1-1.module+el8.3.0+8221+97165c3f.x86_64 2021-07-31T01:28:11.499 INFO:teuthology.orchestra.run.smithi045.stdout: libvarlink-18-3.el8.x86_64 2021-07-31T01:28:11.499 INFO:teuthology.orchestra.run.smithi045.stdout: podman-2.2.1-7.module+el8.3.1+9857+68fb1526.x86_64 2021-07-31T01:28:11.500 INFO:teuthology.orchestra.run.smithi045.stdout: podman-catatonit-2.2.1-7.module+el8.3.1+9857+68fb1526.x86_64 2021-07-31T01:28:11.500 INFO:teuthology.orchestra.run.smithi045.stdout: podman-docker-2.2.1-7.module+el8.3.1+9857+68fb1526.noarch 2021-07-31T01:28:11.500 INFO:teuthology.orchestra.run.smithi045.stdout: protobuf-c-1.3.0-4.el8.x86_64 2021-07-31T01:28:11.500 INFO:teuthology.orchestra.run.smithi045.stdout: runc-1.0.0-70.rc92.module+el8.3.1+9857+68fb1526.x86_64 2021-07-31T01:28:11.501 INFO:teuthology.orchestra.run.smithi045.stdout: slirp4netns-1.1.4-2.module+el8.3.0+8221+97165c3f.x86_64 2021-07-31T01:28:11.501 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:28:11.501 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2021-07-31T01:28:11.852 INFO:teuthology.orchestra.run.smithi045.stdout:Updating Subscription Management repositories. 2021-07-31T01:28:12.440 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 76 kB/s | 2.8 kB 00:00 2021-07-31T01:28:13.246 INFO:teuthology.orchestra.run.smithi154.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 83 kB/s | 2.4 kB 00:00 2021-07-31T01:28:16.791 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 63 kB/s | 2.4 kB 00:00 2021-07-31T01:28:17.623 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:00:04 ago on Sat 31 Jul 2021 01:27:55 AM UTC. 2021-07-31T01:28:19.741 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2021-07-31T01:28:19.743 INFO:teuthology.orchestra.run.smithi154.stdout:=================================================================================================================================== 2021-07-31T01:28:19.743 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repository Size 2021-07-31T01:28:19.743 INFO:teuthology.orchestra.run.smithi154.stdout:=================================================================================================================================== 2021-07-31T01:28:19.743 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2021-07-31T01:28:19.744 INFO:teuthology.orchestra.run.smithi154.stdout: podman x86_64 3.2.3-1.el8.1.3 devel_kubic_libcontainers_stable 13 M 2021-07-31T01:28:19.744 INFO:teuthology.orchestra.run.smithi154.stdout:Installing dependencies: 2021-07-31T01:28:19.744 INFO:teuthology.orchestra.run.smithi154.stdout: conmon x86_64 2:2.0.29-1.el8.3.7 devel_kubic_libcontainers_stable 50 k 2021-07-31T01:28:19.744 INFO:teuthology.orchestra.run.smithi154.stdout: container-selinux noarch 2:2.145.0-1.el8 copr:copr.fedorainfracloud.org:rhcontainerbot:container-selinux 59 k 2021-07-31T01:28:19.744 INFO:teuthology.orchestra.run.smithi154.stdout: containernetworking-plugins x86_64 1.0.0-0.2.rc1.el8.6.4 devel_kubic_libcontainers_stable 21 M 2021-07-31T01:28:19.745 INFO:teuthology.orchestra.run.smithi154.stdout: containers-common noarch 4:1-18.el8.20.3 devel_kubic_libcontainers_stable 61 k 2021-07-31T01:28:19.745 INFO:teuthology.orchestra.run.smithi154.stdout: crun x86_64 0.20.1-1.el8.3.4 devel_kubic_libcontainers_stable 194 k 2021-07-31T01:28:19.745 INFO:teuthology.orchestra.run.smithi154.stdout: dnsmasq x86_64 2.79-13.el8_3.1 rhel-8-for-x86_64-appstream-rpms 318 k 2021-07-31T01:28:19.746 INFO:teuthology.orchestra.run.smithi154.stdout: fuse-common x86_64 3.2.1-12.el8 rhel-8-for-x86_64-baseos-rpms 21 k 2021-07-31T01:28:19.746 INFO:teuthology.orchestra.run.smithi154.stdout: fuse3 x86_64 3.2.1-12.el8 rhel-8-for-x86_64-baseos-rpms 50 k 2021-07-31T01:28:19.746 INFO:teuthology.orchestra.run.smithi154.stdout: fuse3-libs x86_64 3.2.1-12.el8 rhel-8-for-x86_64-baseos-rpms 94 k 2021-07-31T01:28:19.747 INFO:teuthology.orchestra.run.smithi154.stdout: libslirp x86_64 4.3.1-4.el8.4.10 devel_kubic_libcontainers_stable 73 k 2021-07-31T01:28:19.747 INFO:teuthology.orchestra.run.smithi154.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2021-07-31T01:28:19.747 INFO:teuthology.orchestra.run.smithi154.stdout:Installing weak dependencies: 2021-07-31T01:28:19.747 INFO:teuthology.orchestra.run.smithi154.stdout: catatonit x86_64 0.1.5-6.el8.3.10 devel_kubic_libcontainers_stable 290 k 2021-07-31T01:28:19.748 INFO:teuthology.orchestra.run.smithi154.stdout: fuse-overlayfs x86_64 1.5.0-1.el8.1.7 devel_kubic_libcontainers_stable 73 k 2021-07-31T01:28:19.749 INFO:teuthology.orchestra.run.smithi154.stdout: podman-plugins x86_64 3.2.3-1.el8.1.3 devel_kubic_libcontainers_stable 3.4 M 2021-07-31T01:28:19.749 INFO:teuthology.orchestra.run.smithi154.stdout: slirp4netns x86_64 1.1.8-4.el8.7.11 devel_kubic_libcontainers_stable 55 k 2021-07-31T01:28:19.749 INFO:teuthology.orchestra.run.smithi154.stdout: 2021-07-31T01:28:19.750 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2021-07-31T01:28:19.750 INFO:teuthology.orchestra.run.smithi154.stdout:=================================================================================================================================== 2021-07-31T01:28:19.750 INFO:teuthology.orchestra.run.smithi154.stdout:Install 16 Packages 2021-07-31T01:28:19.750 INFO:teuthology.orchestra.run.smithi154.stdout: 2021-07-31T01:28:19.751 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 38 M 2021-07-31T01:28:19.751 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 139 M 2021-07-31T01:28:19.752 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2021-07-31T01:28:20.848 INFO:teuthology.orchestra.run.smithi154.stdout:(1/16): container-selinux-2.145.0-1.el8.noarch. 54 kB/s | 59 kB 00:01 2021-07-31T01:28:21.258 INFO:teuthology.orchestra.run.smithi154.stdout:(2/16): conmon-2.0.29-1.el8.3.7.x86_64.rpm 33 kB/s | 50 kB 00:01 2021-07-31T01:28:21.407 INFO:teuthology.orchestra.run.smithi154.stdout:(3/16): catatonit-0.1.5-6.el8.3.10.x86_64.rpm 175 kB/s | 290 kB 00:01 2021-07-31T01:28:21.543 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:04 ago on Sat 31 Jul 2021 01:27:58 AM UTC. 2021-07-31T01:28:21.675 INFO:teuthology.orchestra.run.smithi154.stdout:(4/16): containers-common-1-18.el8.20.3.noarch. 148 kB/s | 61 kB 00:00 2021-07-31T01:28:21.838 INFO:teuthology.orchestra.run.smithi154.stdout:(5/16): crun-0.20.1-1.el8.3.4.x86_64.rpm 451 kB/s | 194 kB 00:00 2021-07-31T01:28:22.088 INFO:teuthology.orchestra.run.smithi154.stdout:(6/16): fuse-overlayfs-1.5.0-1.el8.1.7.x86_64.r 176 kB/s | 73 kB 00:00 2021-07-31T01:28:22.252 INFO:teuthology.orchestra.run.smithi154.stdout:(7/16): libslirp-4.3.1-4.el8.4.10.x86_64.rpm 177 kB/s | 73 kB 00:00 2021-07-31T01:28:23.275 INFO:teuthology.orchestra.run.smithi154.stdout:(8/16): containernetworking-plugins-1.0.0-0.2.r 8.6 MB/s | 21 MB 00:02 2021-07-31T01:28:23.376 INFO:teuthology.orchestra.run.smithi154.stdout:(9/16): podman-plugins-3.2.3-1.el8.1.3.x86_64.r 3.0 MB/s | 3.4 MB 00:01 2021-07-31T01:28:23.408 INFO:teuthology.orchestra.run.smithi154.stdout:(10/16): yajl-2.1.0-10.el8.x86_64.rpm 1.3 MB/s | 41 kB 00:00 2021-07-31T01:28:23.432 INFO:teuthology.orchestra.run.smithi154.stdout:(11/16): dnsmasq-2.79-13.el8_3.1.x86_64.rpm 14 MB/s | 318 kB 00:00 2021-07-31T01:28:23.452 INFO:teuthology.orchestra.run.smithi154.stdout:(12/16): fuse3-libs-3.2.1-12.el8.x86_64.rpm 4.2 MB/s | 94 kB 00:00 2021-07-31T01:28:23.569 INFO:teuthology.orchestra.run.smithi154.stdout:(13/16): podman-3.2.3-1.el8.1.3.x86_64.rpm 8.6 MB/s | 13 MB 00:01 2021-07-31T01:28:23.569 INFO:teuthology.orchestra.run.smithi154.stdout:(14/16): fuse-common-3.2.1-12.el8.x86_64.rpm 182 kB/s | 21 kB 00:00 2021-07-31T01:28:23.598 INFO:teuthology.orchestra.run.smithi154.stdout:(15/16): fuse3-3.2.1-12.el8.x86_64.rpm 1.6 MB/s | 50 kB 00:00 2021-07-31T01:28:23.811 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2021-07-31T01:28:23.812 INFO:teuthology.orchestra.run.smithi154.stdout:(16/16): slirp4netns-1.1.8-4.el8.7.11.x86_64.rp 102 kB/s | 55 kB 00:00 2021-07-31T01:28:23.813 INFO:teuthology.orchestra.run.smithi045.stdout:=================================================================================================================================== 2021-07-31T01:28:23.813 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2021-07-31T01:28:23.813 INFO:teuthology.orchestra.run.smithi045.stdout:=================================================================================================================================== 2021-07-31T01:28:23.813 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2021-07-31T01:28:23.814 INFO:teuthology.orchestra.run.smithi045.stdout: podman x86_64 3.2.3-1.el8.1.3 devel_kubic_libcontainers_stable 13 M 2021-07-31T01:28:23.814 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2021-07-31T01:28:23.815 INFO:teuthology.orchestra.run.smithi045.stdout: conmon x86_64 2:2.0.29-1.el8.3.7 devel_kubic_libcontainers_stable 50 k 2021-07-31T01:28:23.815 INFO:teuthology.orchestra.run.smithi045.stdout: container-selinux noarch 2:2.145.0-1.el8 copr:copr.fedorainfracloud.org:rhcontainerbot:container-selinux 59 k 2021-07-31T01:28:23.815 INFO:teuthology.orchestra.run.smithi045.stdout: containernetworking-plugins x86_64 1.0.0-0.2.rc1.el8.6.4 devel_kubic_libcontainers_stable 21 M 2021-07-31T01:28:23.815 INFO:teuthology.orchestra.run.smithi045.stdout: containers-common noarch 4:1-18.el8.20.3 devel_kubic_libcontainers_stable 61 k 2021-07-31T01:28:23.816 INFO:teuthology.orchestra.run.smithi045.stdout: crun x86_64 0.20.1-1.el8.3.4 devel_kubic_libcontainers_stable 194 k 2021-07-31T01:28:23.816 INFO:teuthology.orchestra.run.smithi045.stdout: dnsmasq x86_64 2.79-13.el8_3.1 rhel-8-for-x86_64-appstream-rpms 318 k 2021-07-31T01:28:23.816 INFO:teuthology.orchestra.run.smithi045.stdout: fuse-common x86_64 3.2.1-12.el8 rhel-8-for-x86_64-baseos-rpms 21 k 2021-07-31T01:28:23.817 INFO:teuthology.orchestra.run.smithi045.stdout: fuse3 x86_64 3.2.1-12.el8 rhel-8-for-x86_64-baseos-rpms 50 k 2021-07-31T01:28:23.817 INFO:teuthology.orchestra.run.smithi045.stdout: fuse3-libs x86_64 3.2.1-12.el8 rhel-8-for-x86_64-baseos-rpms 94 k 2021-07-31T01:28:23.817 INFO:teuthology.orchestra.run.smithi045.stdout: libslirp x86_64 4.3.1-4.el8.4.10 devel_kubic_libcontainers_stable 73 k 2021-07-31T01:28:23.817 INFO:teuthology.orchestra.run.smithi045.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2021-07-31T01:28:23.818 INFO:teuthology.orchestra.run.smithi045.stdout:Installing weak dependencies: 2021-07-31T01:28:23.818 INFO:teuthology.orchestra.run.smithi045.stdout: catatonit x86_64 0.1.5-6.el8.3.10 devel_kubic_libcontainers_stable 290 k 2021-07-31T01:28:23.818 INFO:teuthology.orchestra.run.smithi045.stdout: fuse-overlayfs x86_64 1.5.0-1.el8.1.7 devel_kubic_libcontainers_stable 73 k 2021-07-31T01:28:23.818 INFO:teuthology.orchestra.run.smithi045.stdout: podman-plugins x86_64 3.2.3-1.el8.1.3 devel_kubic_libcontainers_stable 3.4 M 2021-07-31T01:28:23.818 INFO:teuthology.orchestra.run.smithi045.stdout: slirp4netns x86_64 1.1.8-4.el8.7.11 devel_kubic_libcontainers_stable 55 k 2021-07-31T01:28:23.819 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:28:23.819 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2021-07-31T01:28:23.820 INFO:teuthology.orchestra.run.smithi045.stdout:=================================================================================================================================== 2021-07-31T01:28:23.820 INFO:teuthology.orchestra.run.smithi045.stdout:Install 16 Packages 2021-07-31T01:28:23.820 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:28:23.822 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2021-07-31T01:28:23.822 INFO:teuthology.orchestra.run.smithi154.stdout:Total 9.4 MB/s | 38 MB 00:04 2021-07-31T01:28:23.822 INFO:teuthology.orchestra.run.smithi154.stderr:warning: /var/cache/dnf/copr:copr.fedorainfracloud.org:rhcontainerbot:container-selinux-57cf30cbbab27eef/packages/container-selinux-2.145.0-1.el8.noarch.rpm: Header V3 RSA/SHA1 Signature, key ID 62035f7b: NOKEY 2021-07-31T01:28:23.823 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 38 M 2021-07-31T01:28:23.823 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 139 M 2021-07-31T01:28:23.823 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2021-07-31T01:28:24.635 INFO:teuthology.orchestra.run.smithi154.stdout:Copr repo for container-selinux owned by rhcont 1.3 kB/s | 1.0 kB 00:00 2021-07-31T01:28:24.770 INFO:teuthology.orchestra.run.smithi154.stderr:Importing GPG key 0x62035F7B: 2021-07-31T01:28:24.771 INFO:teuthology.orchestra.run.smithi154.stderr: Userid : "rhcontainerbot_container-selinux (None) " 2021-07-31T01:28:24.771 INFO:teuthology.orchestra.run.smithi154.stderr: Fingerprint: F5D6 6271 4052 49DE A189 569F C907 EBD4 6203 5F7B 2021-07-31T01:28:24.771 INFO:teuthology.orchestra.run.smithi154.stderr: From : https://download.copr.fedorainfracloud.org/results/rhcontainerbot/container-selinux/pubkey.gpg 2021-07-31T01:28:24.880 INFO:teuthology.orchestra.run.smithi154.stdout:Key imported successfully 2021-07-31T01:28:24.881 INFO:teuthology.orchestra.run.smithi154.stderr:warning: /var/cache/dnf/devel_kubic_libcontainers_stable-37b272243bc11f7c/packages/catatonit-0.1.5-6.el8.3.10.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID 75060aa4: NOKEY 2021-07-31T01:28:24.927 INFO:teuthology.orchestra.run.smithi045.stdout:(1/16): container-selinux-2.145.0-1.el8.noarch. 53 kB/s | 59 kB 00:01 2021-07-31T01:28:25.211 INFO:teuthology.orchestra.run.smithi154.stdout:Stable Releases of Upstream github.com/containe 3.3 kB/s | 1.1 kB 00:00 2021-07-31T01:28:25.250 INFO:teuthology.orchestra.run.smithi154.stderr:Importing GPG key 0x75060AA4: 2021-07-31T01:28:25.250 INFO:teuthology.orchestra.run.smithi154.stderr: Userid : "devel:kubic OBS Project " 2021-07-31T01:28:25.250 INFO:teuthology.orchestra.run.smithi154.stderr: Fingerprint: 2472 D6D0 D2F6 6AF8 7ABA 8DA3 4D64 3903 7506 0AA4 2021-07-31T01:28:25.251 INFO:teuthology.orchestra.run.smithi154.stderr: From : https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/CentOS_8/repodata/repomd.xml.key 2021-07-31T01:28:25.299 INFO:teuthology.orchestra.run.smithi045.stdout:(2/16): conmon-2.0.29-1.el8.3.7.x86_64.rpm 34 kB/s | 50 kB 00:01 2021-07-31T01:28:25.322 INFO:teuthology.orchestra.run.smithi154.stdout:Key imported successfully 2021-07-31T01:28:25.337 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2021-07-31T01:28:25.378 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2021-07-31T01:28:25.379 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2021-07-31T01:28:25.538 INFO:teuthology.orchestra.run.smithi045.stdout:(3/16): catatonit-0.1.5-6.el8.3.10.x86_64.rpm 169 kB/s | 290 kB 00:01 2021-07-31T01:28:25.690 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2021-07-31T01:28:25.694 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2021-07-31T01:28:25.809 INFO:teuthology.orchestra.run.smithi045.stdout:(4/16): containers-common-1-18.el8.20.3.noarch. 120 kB/s | 61 kB 00:00 2021-07-31T01:28:26.044 INFO:teuthology.orchestra.run.smithi045.stdout:(5/16): crun-0.20.1-1.el8.3.4.x86_64.rpm 384 kB/s | 194 kB 00:00 2021-07-31T01:28:26.232 INFO:teuthology.orchestra.run.smithi045.stdout:(6/16): fuse-overlayfs-1.5.0-1.el8.1.7.x86_64.r 172 kB/s | 73 kB 00:00 2021-07-31T01:28:26.459 INFO:teuthology.orchestra.run.smithi045.stdout:(7/16): libslirp-4.3.1-4.el8.4.10.x86_64.rpm 176 kB/s | 73 kB 00:00 2021-07-31T01:28:26.460 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2021-07-31T01:28:26.461 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : fuse-common-3.2.1-12.el8.x86_64 1/16 2021-07-31T01:28:26.461 INFO:teuthology.orchestra.run.smithi154.stdout:warning: /etc/fuse.conf created as /etc/fuse.conf.rpmnew 2021-07-31T01:28:26.461 INFO:teuthology.orchestra.run.smithi154.stdout: 2021-07-31T01:28:26.796 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : fuse3-3.2.1-12.el8.x86_64 2/16 2021-07-31T01:28:26.938 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : fuse3-libs-3.2.1-12.el8.x86_64 3/16 2021-07-31T01:28:27.044 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: fuse3-libs-3.2.1-12.el8.x86_64 3/16 2021-07-31T01:28:27.197 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : fuse-overlayfs-1.5.0-1.el8.1.7.x86_64 4/16 2021-07-31T01:28:27.235 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: fuse-overlayfs-1.5.0-1.el8.1.7.x86_64 4/16 2021-07-31T01:28:27.245 INFO:teuthology.orchestra.run.smithi045.stdout:(8/16): containernetworking-plugins-1.0.0-0.2.r 9.0 MB/s | 21 MB 00:02 2021-07-31T01:28:27.321 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: dnsmasq-2.79-13.el8_3.1.x86_64 5/16 2021-07-31T01:28:27.321 INFO:teuthology.orchestra.run.smithi154.stdout:/var/tmp/rpm-tmp.IBuQIx: line 5: 26152 Segmentation fault (core dumped) systemd-sysusers - &> /dev/null < /dev/null 2021-07-31T01:28:27.596 INFO:teuthology.orchestra.run.smithi154.stdout: 2021-07-31T01:28:27.601 INFO:teuthology.orchestra.run.smithi045.stdout:(10/16): yajl-2.1.0-10.el8.x86_64.rpm 1.0 MB/s | 41 kB 00:00 2021-07-31T01:28:27.629 INFO:teuthology.orchestra.run.smithi045.stdout:(11/16): dnsmasq-2.79-13.el8_3.1.x86_64.rpm 11 MB/s | 318 kB 00:00 2021-07-31T01:28:27.655 INFO:teuthology.orchestra.run.smithi045.stdout:(12/16): fuse3-libs-3.2.1-12.el8.x86_64.rpm 3.6 MB/s | 94 kB 00:00 2021-07-31T01:28:27.681 INFO:teuthology.orchestra.run.smithi045.stdout:(13/16): fuse-common-3.2.1-12.el8.x86_64.rpm 823 kB/s | 21 kB 00:00 2021-07-31T01:28:27.707 INFO:teuthology.orchestra.run.smithi045.stdout:(14/16): fuse3-3.2.1-12.el8.x86_64.rpm 1.9 MB/s | 50 kB 00:00 2021-07-31T01:28:27.786 INFO:teuthology.orchestra.run.smithi045.stdout:(15/16): slirp4netns-1.1.8-4.el8.7.11.x86_64.rp 101 kB/s | 55 kB 00:00 2021-07-31T01:28:27.813 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : yajl-2.1.0-10.el8.x86_64 6/16 2021-07-31T01:28:27.941 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : crun-0.20.1-1.el8.3.4.x86_64 7/16 2021-07-31T01:28:28.119 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libslirp-4.3.1-4.el8.4.10.x86_64 8/16 2021-07-31T01:28:28.178 INFO:teuthology.orchestra.run.smithi045.stdout:(16/16): podman-plugins-3.2.3-1.el8.1.3.x86_64. 2.0 MB/s | 3.4 MB 00:01 2021-07-31T01:28:28.184 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2021-07-31T01:28:28.184 INFO:teuthology.orchestra.run.smithi045.stdout:Total 8.8 MB/s | 38 MB 00:04 2021-07-31T01:28:28.185 INFO:teuthology.orchestra.run.smithi045.stderr:warning: /var/cache/dnf/copr:copr.fedorainfracloud.org:rhcontainerbot:container-selinux-57cf30cbbab27eef/packages/container-selinux-2.145.0-1.el8.noarch.rpm: Header V3 RSA/SHA1 Signature, key ID 62035f7b: NOKEY 2021-07-31T01:28:28.993 INFO:teuthology.orchestra.run.smithi045.stdout:Copr repo for container-selinux owned by rhcont 1.3 kB/s | 1.0 kB 00:00 2021-07-31T01:28:29.113 INFO:teuthology.orchestra.run.smithi045.stderr:Importing GPG key 0x62035F7B: 2021-07-31T01:28:29.114 INFO:teuthology.orchestra.run.smithi045.stderr: Userid : "rhcontainerbot_container-selinux (None) " 2021-07-31T01:28:29.114 INFO:teuthology.orchestra.run.smithi045.stderr: Fingerprint: F5D6 6271 4052 49DE A189 569F C907 EBD4 6203 5F7B 2021-07-31T01:28:29.114 INFO:teuthology.orchestra.run.smithi045.stderr: From : https://download.copr.fedorainfracloud.org/results/rhcontainerbot/container-selinux/pubkey.gpg 2021-07-31T01:28:29.223 INFO:teuthology.orchestra.run.smithi045.stdout:Key imported successfully 2021-07-31T01:28:29.225 INFO:teuthology.orchestra.run.smithi045.stderr:warning: /var/cache/dnf/devel_kubic_libcontainers_stable-37b272243bc11f7c/packages/catatonit-0.1.5-6.el8.3.10.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID 75060aa4: NOKEY 2021-07-31T01:28:29.554 INFO:teuthology.orchestra.run.smithi045.stdout:Stable Releases of Upstream github.com/containe 3.3 kB/s | 1.1 kB 00:00 2021-07-31T01:28:29.600 INFO:teuthology.orchestra.run.smithi045.stderr:Importing GPG key 0x75060AA4: 2021-07-31T01:28:29.601 INFO:teuthology.orchestra.run.smithi045.stderr: Userid : "devel:kubic OBS Project " 2021-07-31T01:28:29.601 INFO:teuthology.orchestra.run.smithi045.stderr: Fingerprint: 2472 D6D0 D2F6 6AF8 7ABA 8DA3 4D64 3903 7506 0AA4 2021-07-31T01:28:29.601 INFO:teuthology.orchestra.run.smithi045.stderr: From : https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/CentOS_8/repodata/repomd.xml.key 2021-07-31T01:28:29.681 INFO:teuthology.orchestra.run.smithi045.stdout:Key imported successfully 2021-07-31T01:28:29.698 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2021-07-31T01:28:29.746 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2021-07-31T01:28:29.746 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2021-07-31T01:28:30.028 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : slirp4netns-1.1.8-4.el8.7.11.x86_64 9/16 2021-07-31T01:28:30.089 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2021-07-31T01:28:30.093 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2021-07-31T01:28:30.203 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : containernetworking-plugins-1.0.0-0.2.rc1.el8.6.4. 10/16 2021-07-31T01:28:30.403 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : conmon-2:2.0.29-1.el8.3.7.x86_64 11/16 2021-07-31T01:28:30.538 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : catatonit-0.1.5-6.el8.3.10.x86_64 12/16 2021-07-31T01:28:30.555 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 13/16 2021-07-31T01:28:30.712 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : container-selinux-2:2.145.0-1.el8.noarch 13/16 2021-07-31T01:28:30.919 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2021-07-31T01:28:30.919 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : fuse-common-3.2.1-12.el8.x86_64 1/16 2021-07-31T01:28:30.920 INFO:teuthology.orchestra.run.smithi045.stdout:warning: /etc/fuse.conf created as /etc/fuse.conf.rpmnew 2021-07-31T01:28:30.920 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:28:31.324 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : fuse3-3.2.1-12.el8.x86_64 2/16 2021-07-31T01:28:31.490 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : fuse3-libs-3.2.1-12.el8.x86_64 3/16 2021-07-31T01:28:31.554 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: fuse3-libs-3.2.1-12.el8.x86_64 3/16 2021-07-31T01:28:31.740 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : fuse-overlayfs-1.5.0-1.el8.1.7.x86_64 4/16 2021-07-31T01:28:31.823 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: fuse-overlayfs-1.5.0-1.el8.1.7.x86_64 4/16 2021-07-31T01:28:31.910 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: dnsmasq-2.79-13.el8_3.1.x86_64 5/16 2021-07-31T01:28:31.910 INFO:teuthology.orchestra.run.smithi045.stdout:/var/tmp/rpm-tmp.JbU15C: line 5: 26186 Segmentation fault (core dumped) systemd-sysusers - &> /dev/null < /dev/null 2021-07-31T01:28:32.206 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:28:32.440 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : yajl-2.1.0-10.el8.x86_64 6/16 2021-07-31T01:28:32.635 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : crun-0.20.1-1.el8.3.4.x86_64 7/16 2021-07-31T01:28:32.812 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libslirp-4.3.1-4.el8.4.10.x86_64 8/16 2021-07-31T01:28:34.893 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : slirp4netns-1.1.8-4.el8.7.11.x86_64 9/16 2021-07-31T01:28:35.097 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : containernetworking-plugins-1.0.0-0.2.rc1.el8.6.4. 10/16 2021-07-31T01:28:35.291 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : conmon-2:2.0.29-1.el8.3.7.x86_64 11/16 2021-07-31T01:28:35.472 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : catatonit-0.1.5-6.el8.3.10.x86_64 12/16 2021-07-31T01:28:35.491 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 13/16 2021-07-31T01:28:35.673 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : container-selinux-2:2.145.0-1.el8.noarch 13/16 2021-07-31T01:28:48.610 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 13/16 2021-07-31T01:28:49.069 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : containers-common-4:1-18.el8.20.3.noarch 14/16 2021-07-31T01:28:50.462 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : podman-plugins-3.2.3-1.el8.1.3.x86_64 15/16 2021-07-31T01:28:50.688 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : podman-3.2.3-1.el8.1.3.x86_64 16/16 2021-07-31T01:28:51.261 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 16/16 2021-07-31T01:28:51.584 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: podman-3.2.3-1.el8.1.3.x86_64 16/16 2021-07-31T01:28:51.584 INFO:teuthology.orchestra.run.smithi154.stdout:/var/tmp/rpm-tmp.QCWeQd: line 6: 27141 Segmentation fault (core dumped) /usr/bin/systemd-sysusers 2021-07-31T01:28:51.585 INFO:teuthology.orchestra.run.smithi154.stdout:warning: %triggerin(systemd-239-40.el8.x86_64) scriptlet failed, exit status 139 2021-07-31T01:28:51.585 INFO:teuthology.orchestra.run.smithi154.stdout: 2021-07-31T01:28:51.585 INFO:teuthology.orchestra.run.smithi154.stderr:Error in scriptlet in rpm package podman 2021-07-31T01:28:51.767 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : container-selinux-2:2.145.0-1.el8.noarch 1/16 2021-07-31T01:28:51.767 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : catatonit-0.1.5-6.el8.3.10.x86_64 2/16 2021-07-31T01:28:51.768 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : conmon-2:2.0.29-1.el8.3.7.x86_64 3/16 2021-07-31T01:28:51.768 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : containernetworking-plugins-1.0.0-0.2.rc1.el8.6.4. 4/16 2021-07-31T01:28:51.768 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : containers-common-4:1-18.el8.20.3.noarch 5/16 2021-07-31T01:28:51.768 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : crun-0.20.1-1.el8.3.4.x86_64 6/16 2021-07-31T01:28:51.769 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : fuse-overlayfs-1.5.0-1.el8.1.7.x86_64 7/16 2021-07-31T01:28:51.769 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libslirp-4.3.1-4.el8.4.10.x86_64 8/16 2021-07-31T01:28:51.769 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : podman-3.2.3-1.el8.1.3.x86_64 9/16 2021-07-31T01:28:51.769 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : podman-plugins-3.2.3-1.el8.1.3.x86_64 10/16 2021-07-31T01:28:51.770 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : slirp4netns-1.1.8-4.el8.7.11.x86_64 11/16 2021-07-31T01:28:51.770 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 12/16 2021-07-31T01:28:51.770 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : dnsmasq-2.79-13.el8_3.1.x86_64 13/16 2021-07-31T01:28:51.770 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : fuse3-libs-3.2.1-12.el8.x86_64 14/16 2021-07-31T01:28:51.770 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : fuse-common-3.2.1-12.el8.x86_64 15/16 2021-07-31T01:28:52.517 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : fuse3-3.2.1-12.el8.x86_64 16/16 2021-07-31T01:28:52.517 INFO:teuthology.orchestra.run.smithi154.stdout:Installed products updated. 2021-07-31T01:28:54.403 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 13/16 2021-07-31T01:28:55.047 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : containers-common-4:1-18.el8.20.3.noarch 14/16 2021-07-31T01:28:56.545 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : podman-plugins-3.2.3-1.el8.1.3.x86_64 15/16 2021-07-31T01:28:56.818 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : podman-3.2.3-1.el8.1.3.x86_64 16/16 2021-07-31T01:28:57.412 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 16/16 2021-07-31T01:28:57.709 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: podman-3.2.3-1.el8.1.3.x86_64 16/16 2021-07-31T01:28:57.710 INFO:teuthology.orchestra.run.smithi045.stdout:/var/tmp/rpm-tmp.M2Lv6Y: line 6: 27175 Segmentation fault (core dumped) /usr/bin/systemd-sysusers 2021-07-31T01:28:57.710 INFO:teuthology.orchestra.run.smithi045.stdout:warning: %triggerin(systemd-239-40.el8.x86_64) scriptlet failed, exit status 139 2021-07-31T01:28:57.710 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:28:57.711 INFO:teuthology.orchestra.run.smithi045.stderr:Error in scriptlet in rpm package podman 2021-07-31T01:28:57.907 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : container-selinux-2:2.145.0-1.el8.noarch 1/16 2021-07-31T01:28:57.908 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : catatonit-0.1.5-6.el8.3.10.x86_64 2/16 2021-07-31T01:28:57.908 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : conmon-2:2.0.29-1.el8.3.7.x86_64 3/16 2021-07-31T01:28:57.908 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : containernetworking-plugins-1.0.0-0.2.rc1.el8.6.4. 4/16 2021-07-31T01:28:57.908 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : containers-common-4:1-18.el8.20.3.noarch 5/16 2021-07-31T01:28:57.909 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : crun-0.20.1-1.el8.3.4.x86_64 6/16 2021-07-31T01:28:57.909 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : fuse-overlayfs-1.5.0-1.el8.1.7.x86_64 7/16 2021-07-31T01:28:57.909 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libslirp-4.3.1-4.el8.4.10.x86_64 8/16 2021-07-31T01:28:57.910 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : podman-3.2.3-1.el8.1.3.x86_64 9/16 2021-07-31T01:28:57.910 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : podman-plugins-3.2.3-1.el8.1.3.x86_64 10/16 2021-07-31T01:28:57.910 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : slirp4netns-1.1.8-4.el8.7.11.x86_64 11/16 2021-07-31T01:28:57.910 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 12/16 2021-07-31T01:28:57.910 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : dnsmasq-2.79-13.el8_3.1.x86_64 13/16 2021-07-31T01:28:57.911 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : fuse3-libs-3.2.1-12.el8.x86_64 14/16 2021-07-31T01:28:57.911 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : fuse-common-3.2.1-12.el8.x86_64 15/16 2021-07-31T01:28:58.705 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : fuse3-3.2.1-12.el8.x86_64 16/16 2021-07-31T01:28:58.705 INFO:teuthology.orchestra.run.smithi045.stdout:Installed products updated. 2021-07-31T01:29:02.696 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:00:05 ago on Sat 31 Jul 2021 01:28:00 AM UTC. 2021-07-31T01:29:04.872 INFO:teuthology.orchestra.run.smithi154.stdout: 2021-07-31T01:29:04.872 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2021-07-31T01:29:04.873 INFO:teuthology.orchestra.run.smithi154.stdout: catatonit-0.1.5-6.el8.3.10.x86_64 2021-07-31T01:29:04.873 INFO:teuthology.orchestra.run.smithi154.stdout: conmon-2:2.0.29-1.el8.3.7.x86_64 2021-07-31T01:29:04.873 INFO:teuthology.orchestra.run.smithi154.stdout: container-selinux-2:2.145.0-1.el8.noarch 2021-07-31T01:29:04.873 INFO:teuthology.orchestra.run.smithi154.stdout: containernetworking-plugins-1.0.0-0.2.rc1.el8.6.4.x86_64 2021-07-31T01:29:04.874 INFO:teuthology.orchestra.run.smithi154.stdout: containers-common-4:1-18.el8.20.3.noarch 2021-07-31T01:29:04.874 INFO:teuthology.orchestra.run.smithi154.stdout: crun-0.20.1-1.el8.3.4.x86_64 2021-07-31T01:29:04.874 INFO:teuthology.orchestra.run.smithi154.stdout: dnsmasq-2.79-13.el8_3.1.x86_64 2021-07-31T01:29:04.874 INFO:teuthology.orchestra.run.smithi154.stdout: fuse-common-3.2.1-12.el8.x86_64 2021-07-31T01:29:04.874 INFO:teuthology.orchestra.run.smithi154.stdout: fuse-overlayfs-1.5.0-1.el8.1.7.x86_64 2021-07-31T01:29:04.875 INFO:teuthology.orchestra.run.smithi154.stdout: fuse3-3.2.1-12.el8.x86_64 2021-07-31T01:29:04.875 INFO:teuthology.orchestra.run.smithi154.stdout: fuse3-libs-3.2.1-12.el8.x86_64 2021-07-31T01:29:04.875 INFO:teuthology.orchestra.run.smithi154.stdout: libslirp-4.3.1-4.el8.4.10.x86_64 2021-07-31T01:29:04.875 INFO:teuthology.orchestra.run.smithi154.stdout: podman-3.2.3-1.el8.1.3.x86_64 2021-07-31T01:29:04.876 INFO:teuthology.orchestra.run.smithi154.stdout: podman-plugins-3.2.3-1.el8.1.3.x86_64 2021-07-31T01:29:04.876 INFO:teuthology.orchestra.run.smithi154.stdout: slirp4netns-1.1.8-4.el8.7.11.x86_64 2021-07-31T01:29:04.876 INFO:teuthology.orchestra.run.smithi154.stdout: yajl-2.1.0-10.el8.x86_64 2021-07-31T01:29:04.876 INFO:teuthology.orchestra.run.smithi154.stdout: 2021-07-31T01:29:04.876 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2021-07-31T01:29:04.987 DEBUG:teuthology.parallel:result is None 2021-07-31T01:29:09.608 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:05 ago on Sat 31 Jul 2021 01:28:03 AM UTC. 2021-07-31T01:29:11.941 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:29:11.941 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2021-07-31T01:29:11.942 INFO:teuthology.orchestra.run.smithi045.stdout: catatonit-0.1.5-6.el8.3.10.x86_64 2021-07-31T01:29:11.942 INFO:teuthology.orchestra.run.smithi045.stdout: conmon-2:2.0.29-1.el8.3.7.x86_64 2021-07-31T01:29:11.942 INFO:teuthology.orchestra.run.smithi045.stdout: container-selinux-2:2.145.0-1.el8.noarch 2021-07-31T01:29:11.942 INFO:teuthology.orchestra.run.smithi045.stdout: containernetworking-plugins-1.0.0-0.2.rc1.el8.6.4.x86_64 2021-07-31T01:29:11.942 INFO:teuthology.orchestra.run.smithi045.stdout: containers-common-4:1-18.el8.20.3.noarch 2021-07-31T01:29:11.943 INFO:teuthology.orchestra.run.smithi045.stdout: crun-0.20.1-1.el8.3.4.x86_64 2021-07-31T01:29:11.943 INFO:teuthology.orchestra.run.smithi045.stdout: dnsmasq-2.79-13.el8_3.1.x86_64 2021-07-31T01:29:11.943 INFO:teuthology.orchestra.run.smithi045.stdout: fuse-common-3.2.1-12.el8.x86_64 2021-07-31T01:29:11.943 INFO:teuthology.orchestra.run.smithi045.stdout: fuse-overlayfs-1.5.0-1.el8.1.7.x86_64 2021-07-31T01:29:11.944 INFO:teuthology.orchestra.run.smithi045.stdout: fuse3-3.2.1-12.el8.x86_64 2021-07-31T01:29:11.944 INFO:teuthology.orchestra.run.smithi045.stdout: fuse3-libs-3.2.1-12.el8.x86_64 2021-07-31T01:29:11.944 INFO:teuthology.orchestra.run.smithi045.stdout: libslirp-4.3.1-4.el8.4.10.x86_64 2021-07-31T01:29:11.944 INFO:teuthology.orchestra.run.smithi045.stdout: podman-3.2.3-1.el8.1.3.x86_64 2021-07-31T01:29:11.944 INFO:teuthology.orchestra.run.smithi045.stdout: podman-plugins-3.2.3-1.el8.1.3.x86_64 2021-07-31T01:29:11.945 INFO:teuthology.orchestra.run.smithi045.stdout: slirp4netns-1.1.8-4.el8.7.11.x86_64 2021-07-31T01:29:11.945 INFO:teuthology.orchestra.run.smithi045.stdout: yajl-2.1.0-10.el8.x86_64 2021-07-31T01:29:11.945 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:29:11.945 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2021-07-31T01:29:12.094 DEBUG:teuthology.parallel:result is None 2021-07-31T01:29:12.094 INFO:teuthology.run_tasks:Running task install... 2021-07-31T01:29:12.105 DEBUG:teuthology.task.install:project ceph 2021-07-31T01:29:12.106 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'sha1': 'f25860a2cae30438a8a8d15f1f74f6043cc73d75'}} 2021-07-31T01:29:12.107 DEBUG:teuthology.task.install:config {'branch': 'pacific', 'sha1': 'f25860a2cae30438a8a8d15f1f74f6043cc73d75'} 2021-07-31T01:29:12.107 INFO:teuthology.task.install:Using flavor: basic 2021-07-31T01:29:12.114 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2021-07-31T01:29:12.115 INFO:teuthology.task.install:extra packages: [] 2021-07-31T01:29:12.115 DEBUG:teuthology.orchestra.run.smithi045:> sudo lsb_release -is 2021-07-31T01:29:12.155 INFO:teuthology.orchestra.run.smithi045.stdout:RedHatEnterprise 2021-07-31T01:29:12.157 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2021-07-31T01:29:12.158 DEBUG:teuthology.orchestra.run.smithi154:> sudo lsb_release -is 2021-07-31T01:29:12.159 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'pacific', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'flavor': 'basic', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'f25860a2cae30438a8a8d15f1f74f6043cc73d75', 'tag': None, 'wait_for_package': False} 2021-07-31T01:29:12.159 DEBUG:teuthology.orchestra.run.smithi045:> sudo lsb_release -is 2021-07-31T01:29:12.205 INFO:teuthology.orchestra.run.smithi154.stdout:RedHatEnterprise 2021-07-31T01:29:12.206 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2021-07-31T01:29:12.207 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'pacific', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'flavor': 'basic', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'f25860a2cae30438a8a8d15f1f74f6043cc73d75', 'tag': None, 'wait_for_package': False} 2021-07-31T01:29:12.207 DEBUG:teuthology.orchestra.run.smithi154:> sudo lsb_release -is 2021-07-31T01:29:12.237 INFO:teuthology.orchestra.run.smithi045.stdout:RedHatEnterprise 2021-07-31T01:29:12.238 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2021-07-31T01:29:12.239 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2021-07-31T01:29:12.239 INFO:teuthology.packaging:ref: None 2021-07-31T01:29:12.239 INFO:teuthology.packaging:tag: None 2021-07-31T01:29:12.240 INFO:teuthology.packaging:branch: pacific 2021-07-31T01:29:12.240 INFO:teuthology.packaging:sha1: f25860a2cae30438a8a8d15f1f74f6043cc73d75 2021-07-31T01:29:12.241 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=pacific 2021-07-31T01:29:12.248 INFO:teuthology.orchestra.run.smithi154.stdout:RedHatEnterprise 2021-07-31T01:29:12.249 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2021-07-31T01:29:12.249 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2021-07-31T01:29:12.250 INFO:teuthology.packaging:ref: None 2021-07-31T01:29:12.250 INFO:teuthology.packaging:tag: None 2021-07-31T01:29:12.250 INFO:teuthology.packaging:branch: pacific 2021-07-31T01:29:12.251 INFO:teuthology.packaging:sha1: f25860a2cae30438a8a8d15f1f74f6043cc73d75 2021-07-31T01:29:12.251 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=pacific 2021-07-31T01:29:12.433 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/pacific/5a784c87b94e0ea996e88d93f2de2a1a9497d277/centos/8/flavors/default/ 2021-07-31T01:29:12.434 INFO:teuthology.task.install.rpm:Package version is 16.2.5-150.g5a784c87 2021-07-31T01:29:12.455 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/pacific/5a784c87b94e0ea996e88d93f2de2a1a9497d277/centos/8/flavors/default/ 2021-07-31T01:29:12.456 INFO:teuthology.task.install.rpm:Package version is 16.2.5-150.g5a784c87 2021-07-31T01:29:12.579 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/pacific/5a784c87b94e0ea996e88d93f2de2a1a9497d277/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/pacific/5a784c87b94e0ea996e88d93f2de2a1a9497d277/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/pacific/5a784c87b94e0ea996e88d93f2de2a1a9497d277/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2021-07-31T01:29:12.579 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2021-07-31T01:29:12.580 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/etc/yum.repos.d/ceph.repo 2021-07-31T01:29:12.585 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/pacific/5a784c87b94e0ea996e88d93f2de2a1a9497d277/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/pacific/5a784c87b94e0ea996e88d93f2de2a1a9497d277/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/pacific/5a784c87b94e0ea996e88d93f2de2a1a9497d277/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2021-07-31T01:29:12.586 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2021-07-31T01:29:12.586 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/etc/yum.repos.d/ceph.repo 2021-07-31T01:29:12.606 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2021-07-31T01:29:12.606 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2021-07-31T01:29:12.607 INFO:teuthology.packaging:ref: None 2021-07-31T01:29:12.607 INFO:teuthology.packaging:tag: None 2021-07-31T01:29:12.607 INFO:teuthology.packaging:branch: pacific 2021-07-31T01:29:12.607 INFO:teuthology.packaging:sha1: f25860a2cae30438a8a8d15f1f74f6043cc73d75 2021-07-31T01:29:12.608 DEBUG:teuthology.orchestra.run.smithi045:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/pacific/;g' /etc/yum.repos.d/ceph.repo ; fi 2021-07-31T01:29:12.613 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2021-07-31T01:29:12.614 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2021-07-31T01:29:12.614 INFO:teuthology.packaging:ref: None 2021-07-31T01:29:12.614 INFO:teuthology.packaging:tag: None 2021-07-31T01:29:12.615 INFO:teuthology.packaging:branch: pacific 2021-07-31T01:29:12.615 INFO:teuthology.packaging:sha1: f25860a2cae30438a8a8d15f1f74f6043cc73d75 2021-07-31T01:29:12.615 DEBUG:teuthology.orchestra.run.smithi154:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/pacific/;g' /etc/yum.repos.d/ceph.repo ; fi 2021-07-31T01:29:12.673 DEBUG:teuthology.orchestra.run.smithi045:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2021-07-31T01:29:12.682 DEBUG:teuthology.orchestra.run.smithi154:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2021-07-31T01:29:12.750 DEBUG:teuthology.orchestra.run.smithi045:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2021-07-31T01:29:12.760 DEBUG:teuthology.orchestra.run.smithi154:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2021-07-31T01:29:12.815 INFO:teuthology.orchestra.run.smithi045.stdout:check_obsoletes = 1 2021-07-31T01:29:12.816 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum clean all 2021-07-31T01:29:12.825 INFO:teuthology.orchestra.run.smithi154.stdout:check_obsoletes = 1 2021-07-31T01:29:12.827 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum clean all 2021-07-31T01:29:13.132 INFO:teuthology.orchestra.run.smithi154.stdout:Updating Subscription Management repositories. 2021-07-31T01:29:13.136 INFO:teuthology.orchestra.run.smithi045.stdout:Updating Subscription Management repositories. 2021-07-31T01:29:13.445 INFO:teuthology.orchestra.run.smithi154.stdout:152 files removed 2021-07-31T01:29:13.483 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-radosgw 2021-07-31T01:29:13.484 INFO:teuthology.orchestra.run.smithi045.stdout:152 files removed 2021-07-31T01:29:13.521 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-radosgw 2021-07-31T01:29:13.749 INFO:teuthology.orchestra.run.smithi154.stdout:Updating Subscription Management repositories. 2021-07-31T01:29:13.802 INFO:teuthology.orchestra.run.smithi045.stdout:Updating Subscription Management repositories. 2021-07-31T01:29:14.478 INFO:teuthology.orchestra.run.smithi154.stdout:ceph packages for x86_64 258 kB/s | 75 kB 00:00 2021-07-31T01:29:14.524 INFO:teuthology.orchestra.run.smithi045.stdout:ceph packages for x86_64 265 kB/s | 75 kB 00:00 2021-07-31T01:29:14.724 INFO:teuthology.orchestra.run.smithi154.stdout:ceph noarch packages 71 kB/s | 15 kB 00:00 2021-07-31T01:29:14.774 INFO:teuthology.orchestra.run.smithi045.stdout:ceph noarch packages 71 kB/s | 15 kB 00:00 2021-07-31T01:29:14.954 INFO:teuthology.orchestra.run.smithi154.stdout:ceph source packages 7.0 kB/s | 1.5 kB 00:00 2021-07-31T01:29:15.004 INFO:teuthology.orchestra.run.smithi045.stdout:ceph source packages 7.1 kB/s | 1.5 kB 00:00 2021-07-31T01:29:16.796 INFO:teuthology.orchestra.run.smithi045.stdout:Copr repo for python3-asyncssh owned by ceph 2.0 kB/s | 3.5 kB 00:01 2021-07-31T01:29:16.799 INFO:teuthology.orchestra.run.smithi154.stdout:Copr repo for python3-asyncssh owned by ceph 1.9 kB/s | 3.5 kB 00:01 2021-07-31T01:29:18.619 INFO:teuthology.orchestra.run.smithi154.stdout:Copr repo for container-selinux owned by rhcont 768 B/s | 1.4 kB 00:01 2021-07-31T01:29:18.620 INFO:teuthology.orchestra.run.smithi045.stdout:Copr repo for container-selinux owned by rhcont 768 B/s | 1.4 kB 00:01 2021-07-31T01:29:20.481 INFO:teuthology.orchestra.run.smithi045.stdout:Stable Releases of Upstream github.com/containe 36 kB/s | 66 kB 00:01 2021-07-31T01:29:20.492 INFO:teuthology.orchestra.run.smithi154.stdout:Stable Releases of Upstream github.com/containe 36 kB/s | 66 kB 00:01 2021-07-31T01:29:21.231 INFO:teuthology.orchestra.run.smithi045.stdout:Extra Packages for Enterprise Linux 14 MB/s | 10 MB 00:00 2021-07-31T01:29:21.240 INFO:teuthology.orchestra.run.smithi154.stdout:Extra Packages for Enterprise Linux 14 MB/s | 10 MB 00:00 2021-07-31T01:29:23.246 INFO:teuthology.orchestra.run.smithi154.stdout:lab-extras 4.3 MB/s | 24 kB 00:00 2021-07-31T01:29:23.385 INFO:teuthology.orchestra.run.smithi045.stdout:lab-extras 4.3 MB/s | 24 kB 00:00 2021-07-31T01:29:23.529 INFO:teuthology.orchestra.run.smithi154.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 115 MB/s | 26 MB 00:00 2021-07-31T01:29:23.731 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 92 MB/s | 26 MB 00:00 2021-07-31T01:29:27.649 INFO:teuthology.orchestra.run.smithi154.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 127 MB/s | 29 MB 00:00 2021-07-31T01:29:28.251 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 103 MB/s | 29 MB 00:00 2021-07-31T01:29:34.452 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2021-07-31T01:29:34.454 INFO:teuthology.orchestra.run.smithi154.stdout:=================================================================================================== 2021-07-31T01:29:34.455 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repository Size 2021-07-31T01:29:34.455 INFO:teuthology.orchestra.run.smithi154.stdout:=================================================================================================== 2021-07-31T01:29:34.455 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2021-07-31T01:29:34.455 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-radosgw x86_64 2:16.2.5-150.g5a784c87.el8 ceph 12 M 2021-07-31T01:29:34.456 INFO:teuthology.orchestra.run.smithi154.stdout:Upgrading: 2021-07-31T01:29:34.456 INFO:teuthology.orchestra.run.smithi154.stdout: librados2 x86_64 2:16.2.5-150.g5a784c87.el8 ceph 3.7 M 2021-07-31T01:29:34.456 INFO:teuthology.orchestra.run.smithi154.stdout: librbd1 x86_64 2:16.2.5-150.g5a784c87.el8 ceph 4.0 M 2021-07-31T01:29:34.456 INFO:teuthology.orchestra.run.smithi154.stdout:Installing dependencies: 2021-07-31T01:29:34.457 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-base x86_64 2:16.2.5-150.g5a784c87.el8 ceph 5.9 M 2021-07-31T01:29:34.457 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-common x86_64 2:16.2.5-150.g5a784c87.el8 ceph 24 M 2021-07-31T01:29:34.457 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-selinux x86_64 2:16.2.5-150.g5a784c87.el8 ceph 25 k 2021-07-31T01:29:34.457 INFO:teuthology.orchestra.run.smithi154.stdout: daxctl-libs x86_64 67-2.el8 rhel-8-for-x86_64-baseos-rpms 40 k 2021-07-31T01:29:34.458 INFO:teuthology.orchestra.run.smithi154.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2021-07-31T01:29:34.458 INFO:teuthology.orchestra.run.smithi154.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2021-07-31T01:29:34.458 INFO:teuthology.orchestra.run.smithi154.stdout: libcephfs2 x86_64 2:16.2.5-150.g5a784c87.el8 ceph 810 k 2021-07-31T01:29:34.459 INFO:teuthology.orchestra.run.smithi154.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2021-07-31T01:29:34.459 INFO:teuthology.orchestra.run.smithi154.stdout: libpmemobj x86_64 1.6.1-1.el8 rhel-8-for-x86_64-appstream-rpms 145 k 2021-07-31T01:29:34.459 INFO:teuthology.orchestra.run.smithi154.stdout: librabbitmq x86_64 0.9.0-2.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2021-07-31T01:29:34.459 INFO:teuthology.orchestra.run.smithi154.stdout: libradosstriper1 x86_64 2:16.2.5-150.g5a784c87.el8 ceph 492 k 2021-07-31T01:29:34.460 INFO:teuthology.orchestra.run.smithi154.stdout: librdkafka x86_64 0.11.4-1.el8 rhel-8-for-x86_64-appstream-rpms 353 k 2021-07-31T01:29:34.460 INFO:teuthology.orchestra.run.smithi154.stdout: librgw2 x86_64 2:16.2.5-150.g5a784c87.el8 ceph 3.7 M 2021-07-31T01:29:34.460 INFO:teuthology.orchestra.run.smithi154.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2021-07-31T01:29:34.460 INFO:teuthology.orchestra.run.smithi154.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2021-07-31T01:29:34.461 INFO:teuthology.orchestra.run.smithi154.stdout: ndctl-libs x86_64 67-2.el8 rhel-8-for-x86_64-baseos-rpms 74 k 2021-07-31T01:29:34.461 INFO:teuthology.orchestra.run.smithi154.stdout: python3-ceph-argparse x86_64 2:16.2.5-150.g5a784c87.el8 ceph 45 k 2021-07-31T01:29:34.461 INFO:teuthology.orchestra.run.smithi154.stdout: python3-ceph-common x86_64 2:16.2.5-150.g5a784c87.el8 ceph 83 k 2021-07-31T01:29:34.461 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cephfs x86_64 2:16.2.5-150.g5a784c87.el8 ceph 214 k 2021-07-31T01:29:34.461 INFO:teuthology.orchestra.run.smithi154.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2021-07-31T01:29:34.462 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rados x86_64 2:16.2.5-150.g5a784c87.el8 ceph 387 k 2021-07-31T01:29:34.462 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rbd x86_64 2:16.2.5-150.g5a784c87.el8 ceph 368 k 2021-07-31T01:29:34.462 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rgw x86_64 2:16.2.5-150.g5a784c87.el8 ceph 114 k 2021-07-31T01:29:34.462 INFO:teuthology.orchestra.run.smithi154.stdout: 2021-07-31T01:29:34.463 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2021-07-31T01:29:34.463 INFO:teuthology.orchestra.run.smithi154.stdout:=================================================================================================== 2021-07-31T01:29:34.463 INFO:teuthology.orchestra.run.smithi154.stdout:Install 24 Packages 2021-07-31T01:29:34.463 INFO:teuthology.orchestra.run.smithi154.stdout:Upgrade 2 Packages 2021-07-31T01:29:34.464 INFO:teuthology.orchestra.run.smithi154.stdout: 2021-07-31T01:29:34.464 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 57 M 2021-07-31T01:29:34.465 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2021-07-31T01:29:35.667 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2021-07-31T01:29:35.670 INFO:teuthology.orchestra.run.smithi045.stdout:=================================================================================================== 2021-07-31T01:29:35.670 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2021-07-31T01:29:35.671 INFO:teuthology.orchestra.run.smithi045.stdout:=================================================================================================== 2021-07-31T01:29:35.671 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2021-07-31T01:29:35.671 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-radosgw x86_64 2:16.2.5-150.g5a784c87.el8 ceph 12 M 2021-07-31T01:29:35.671 INFO:teuthology.orchestra.run.smithi045.stdout:Upgrading: 2021-07-31T01:29:35.672 INFO:teuthology.orchestra.run.smithi045.stdout: librados2 x86_64 2:16.2.5-150.g5a784c87.el8 ceph 3.7 M 2021-07-31T01:29:35.672 INFO:teuthology.orchestra.run.smithi045.stdout: librbd1 x86_64 2:16.2.5-150.g5a784c87.el8 ceph 4.0 M 2021-07-31T01:29:35.672 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2021-07-31T01:29:35.672 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-base x86_64 2:16.2.5-150.g5a784c87.el8 ceph 5.9 M 2021-07-31T01:29:35.673 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-common x86_64 2:16.2.5-150.g5a784c87.el8 ceph 24 M 2021-07-31T01:29:35.673 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-selinux x86_64 2:16.2.5-150.g5a784c87.el8 ceph 25 k 2021-07-31T01:29:35.673 INFO:teuthology.orchestra.run.smithi045.stdout: daxctl-libs x86_64 67-2.el8 rhel-8-for-x86_64-baseos-rpms 40 k 2021-07-31T01:29:35.673 INFO:teuthology.orchestra.run.smithi045.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2021-07-31T01:29:35.674 INFO:teuthology.orchestra.run.smithi045.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2021-07-31T01:29:35.674 INFO:teuthology.orchestra.run.smithi045.stdout: libcephfs2 x86_64 2:16.2.5-150.g5a784c87.el8 ceph 810 k 2021-07-31T01:29:35.674 INFO:teuthology.orchestra.run.smithi045.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2021-07-31T01:29:35.675 INFO:teuthology.orchestra.run.smithi045.stdout: libpmemobj x86_64 1.6.1-1.el8 rhel-8-for-x86_64-appstream-rpms 145 k 2021-07-31T01:29:35.675 INFO:teuthology.orchestra.run.smithi045.stdout: librabbitmq x86_64 0.9.0-2.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2021-07-31T01:29:35.675 INFO:teuthology.orchestra.run.smithi045.stdout: libradosstriper1 x86_64 2:16.2.5-150.g5a784c87.el8 ceph 492 k 2021-07-31T01:29:35.675 INFO:teuthology.orchestra.run.smithi045.stdout: librdkafka x86_64 0.11.4-1.el8 rhel-8-for-x86_64-appstream-rpms 353 k 2021-07-31T01:29:35.676 INFO:teuthology.orchestra.run.smithi045.stdout: librgw2 x86_64 2:16.2.5-150.g5a784c87.el8 ceph 3.7 M 2021-07-31T01:29:35.676 INFO:teuthology.orchestra.run.smithi045.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2021-07-31T01:29:35.676 INFO:teuthology.orchestra.run.smithi045.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2021-07-31T01:29:35.676 INFO:teuthology.orchestra.run.smithi045.stdout: ndctl-libs x86_64 67-2.el8 rhel-8-for-x86_64-baseos-rpms 74 k 2021-07-31T01:29:35.676 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ceph-argparse x86_64 2:16.2.5-150.g5a784c87.el8 ceph 45 k 2021-07-31T01:29:35.677 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ceph-common x86_64 2:16.2.5-150.g5a784c87.el8 ceph 83 k 2021-07-31T01:29:35.677 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cephfs x86_64 2:16.2.5-150.g5a784c87.el8 ceph 214 k 2021-07-31T01:29:35.677 INFO:teuthology.orchestra.run.smithi045.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2021-07-31T01:29:35.677 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rados x86_64 2:16.2.5-150.g5a784c87.el8 ceph 387 k 2021-07-31T01:29:35.678 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rbd x86_64 2:16.2.5-150.g5a784c87.el8 ceph 368 k 2021-07-31T01:29:35.678 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rgw x86_64 2:16.2.5-150.g5a784c87.el8 ceph 114 k 2021-07-31T01:29:35.678 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:29:35.678 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2021-07-31T01:29:35.679 INFO:teuthology.orchestra.run.smithi045.stdout:=================================================================================================== 2021-07-31T01:29:35.679 INFO:teuthology.orchestra.run.smithi045.stdout:Install 24 Packages 2021-07-31T01:29:35.679 INFO:teuthology.orchestra.run.smithi045.stdout:Upgrade 2 Packages 2021-07-31T01:29:35.679 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:29:35.680 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 57 M 2021-07-31T01:29:35.680 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2021-07-31T01:29:37.131 INFO:teuthology.orchestra.run.smithi154.stdout:(1/26): ceph-base-16.2.5-150.g5a784c87.el8.x86_ 2.2 MB/s | 5.9 MB 00:02 2021-07-31T01:29:37.188 INFO:teuthology.orchestra.run.smithi154.stdout:(2/26): ceph-selinux-16.2.5-150.g5a784c87.el8.x 439 kB/s | 25 kB 00:00 2021-07-31T01:29:37.429 INFO:teuthology.orchestra.run.smithi154.stdout:(3/26): libcephfs2-16.2.5-150.g5a784c87.el8.x86 3.3 MB/s | 810 kB 00:00 2021-07-31T01:29:37.629 INFO:teuthology.orchestra.run.smithi154.stdout:(4/26): libradosstriper1-16.2.5-150.g5a784c87.e 2.4 MB/s | 492 kB 00:00 2021-07-31T01:29:38.752 INFO:teuthology.orchestra.run.smithi154.stdout:(5/26): librgw2-16.2.5-150.g5a784c87.el8.x86_64 3.3 MB/s | 3.7 MB 00:01 2021-07-31T01:29:38.815 INFO:teuthology.orchestra.run.smithi154.stdout:(6/26): python3-ceph-argparse-16.2.5-150.g5a784 712 kB/s | 45 kB 00:00 2021-07-31T01:29:38.875 INFO:teuthology.orchestra.run.smithi154.stdout:(7/26): python3-ceph-common-16.2.5-150.g5a784c8 1.4 MB/s | 83 kB 00:00 2021-07-31T01:29:38.942 INFO:teuthology.orchestra.run.smithi154.stdout:(8/26): python3-cephfs-16.2.5-150.g5a784c87.el8 3.1 MB/s | 214 kB 00:00 2021-07-31T01:29:39.068 INFO:teuthology.orchestra.run.smithi154.stdout:(9/26): python3-rados-16.2.5-150.g5a784c87.el8. 3.0 MB/s | 387 kB 00:00 2021-07-31T01:29:39.199 INFO:teuthology.orchestra.run.smithi154.stdout:(10/26): python3-rbd-16.2.5-150.g5a784c87.el8.x 2.7 MB/s | 368 kB 00:00 2021-07-31T01:29:39.264 INFO:teuthology.orchestra.run.smithi154.stdout:(11/26): python3-rgw-16.2.5-150.g5a784c87.el8.x 1.7 MB/s | 114 kB 00:00 2021-07-31T01:29:39.398 INFO:teuthology.orchestra.run.smithi154.stdout:(12/26): gperftools-libs-2.7-9.el8.x86_64.rpm 2.2 MB/s | 306 kB 00:00 2021-07-31T01:29:39.422 INFO:teuthology.orchestra.run.smithi154.stdout:(13/26): leveldb-1.22-1.el8.x86_64.rpm 7.6 MB/s | 181 kB 00:00 2021-07-31T01:29:39.444 INFO:teuthology.orchestra.run.smithi154.stdout:(14/26): liboath-2.6.2-3.el8.x86_64.rpm 2.5 MB/s | 59 kB 00:00 2021-07-31T01:29:39.467 INFO:teuthology.orchestra.run.smithi154.stdout:(15/26): libunwind-1.3.1-3.el8.x86_64.rpm 3.2 MB/s | 75 kB 00:00 2021-07-31T01:29:39.500 INFO:teuthology.orchestra.run.smithi154.stdout:(16/26): python3-prettytable-0.7.2-14.el8.noarc 1.3 MB/s | 44 kB 00:00 2021-07-31T01:29:39.525 INFO:teuthology.orchestra.run.smithi154.stdout:(17/26): libpmemobj-1.6.1-1.el8.x86_64.rpm 5.8 MB/s | 145 kB 00:00 2021-07-31T01:29:39.547 INFO:teuthology.orchestra.run.smithi154.stdout:(18/26): lttng-ust-2.8.1-11.el8.x86_64.rpm 12 MB/s | 259 kB 00:00 2021-07-31T01:29:39.570 INFO:teuthology.orchestra.run.smithi154.stdout:(19/26): librdkafka-0.11.4-1.el8.x86_64.rpm 15 MB/s | 353 kB 00:00 2021-07-31T01:29:39.592 INFO:teuthology.orchestra.run.smithi154.stdout:(20/26): daxctl-libs-67-2.el8.x86_64.rpm 1.7 MB/s | 40 kB 00:00 2021-07-31T01:29:39.613 INFO:teuthology.orchestra.run.smithi154.stdout:(21/26): librabbitmq-0.9.0-2.el8.x86_64.rpm 2.2 MB/s | 47 kB 00:00 2021-07-31T01:29:39.635 INFO:teuthology.orchestra.run.smithi154.stdout:(22/26): ndctl-libs-67-2.el8.x86_64.rpm 3.4 MB/s | 74 kB 00:00 2021-07-31T01:29:40.700 INFO:teuthology.orchestra.run.smithi045.stdout:(1/26): ceph-common-16.2.5-150.g5a784c87.el8.x8 4.7 MB/s | 24 MB 00:05 2021-07-31T01:29:40.711 INFO:teuthology.orchestra.run.smithi154.stdout:(23/26): librados2-16.2.5-150.g5a784c87.el8.x86 3.4 MB/s | 3.7 MB 00:01 2021-07-31T01:29:40.738 INFO:teuthology.orchestra.run.smithi045.stdout:(2/26): ceph-selinux-16.2.5-150.g5a784c87.el8.x 656 kB/s | 25 kB 00:00 2021-07-31T01:29:40.859 INFO:teuthology.orchestra.run.smithi045.stdout:(3/26): libcephfs2-16.2.5-150.g5a784c87.el8.x86 6.6 MB/s | 810 kB 00:00 2021-07-31T01:29:40.925 INFO:teuthology.orchestra.run.smithi045.stdout:(4/26): libradosstriper1-16.2.5-150.g5a784c87.e 7.3 MB/s | 492 kB 00:00 2021-07-31T01:29:41.342 INFO:teuthology.orchestra.run.smithi045.stdout:(5/26): librgw2-16.2.5-150.g5a784c87.el8.x86_64 9.0 MB/s | 3.7 MB 00:00 2021-07-31T01:29:41.392 INFO:teuthology.orchestra.run.smithi045.stdout:(6/26): python3-ceph-argparse-16.2.5-150.g5a784 904 kB/s | 45 kB 00:00 2021-07-31T01:29:41.442 INFO:teuthology.orchestra.run.smithi045.stdout:(7/26): python3-ceph-common-16.2.5-150.g5a784c8 1.6 MB/s | 83 kB 00:00 2021-07-31T01:29:41.499 INFO:teuthology.orchestra.run.smithi045.stdout:(8/26): python3-cephfs-16.2.5-150.g5a784c87.el8 3.7 MB/s | 214 kB 00:00 2021-07-31T01:29:41.563 INFO:teuthology.orchestra.run.smithi045.stdout:(9/26): python3-rados-16.2.5-150.g5a784c87.el8. 6.0 MB/s | 387 kB 00:00 2021-07-31T01:29:41.625 INFO:teuthology.orchestra.run.smithi045.stdout:(10/26): python3-rbd-16.2.5-150.g5a784c87.el8.x 5.8 MB/s | 368 kB 00:00 2021-07-31T01:29:41.673 INFO:teuthology.orchestra.run.smithi045.stdout:(11/26): python3-rgw-16.2.5-150.g5a784c87.el8.x 2.4 MB/s | 114 kB 00:00 2021-07-31T01:29:41.673 INFO:teuthology.orchestra.run.smithi154.stdout:(24/26): librbd1-16.2.5-150.g5a784c87.el8.x86_6 4.1 MB/s | 4.0 MB 00:00 2021-07-31T01:29:41.823 INFO:teuthology.orchestra.run.smithi045.stdout:(12/26): gperftools-libs-2.7-9.el8.x86_64.rpm 2.0 MB/s | 306 kB 00:00 2021-07-31T01:29:41.846 INFO:teuthology.orchestra.run.smithi045.stdout:(13/26): leveldb-1.22-1.el8.x86_64.rpm 7.7 MB/s | 181 kB 00:00 2021-07-31T01:29:41.869 INFO:teuthology.orchestra.run.smithi045.stdout:(14/26): liboath-2.6.2-3.el8.x86_64.rpm 2.6 MB/s | 59 kB 00:00 2021-07-31T01:29:41.893 INFO:teuthology.orchestra.run.smithi045.stdout:(15/26): libunwind-1.3.1-3.el8.x86_64.rpm 3.3 MB/s | 75 kB 00:00 2021-07-31T01:29:41.933 INFO:teuthology.orchestra.run.smithi045.stdout:(16/26): python3-prettytable-0.7.2-14.el8.noarc 1.0 MB/s | 44 kB 00:00 2021-07-31T01:29:41.939 INFO:teuthology.orchestra.run.smithi154.stdout:(25/26): ceph-common-16.2.5-150.g5a784c87.el8.x 3.2 MB/s | 24 MB 00:07 2021-07-31T01:29:41.959 INFO:teuthology.orchestra.run.smithi045.stdout:(17/26): libpmemobj-1.6.1-1.el8.x86_64.rpm 5.5 MB/s | 145 kB 00:00 2021-07-31T01:29:41.986 INFO:teuthology.orchestra.run.smithi045.stdout:(18/26): lttng-ust-2.8.1-11.el8.x86_64.rpm 9.3 MB/s | 259 kB 00:00 2021-07-31T01:29:42.013 INFO:teuthology.orchestra.run.smithi045.stdout:(19/26): librdkafka-0.11.4-1.el8.x86_64.rpm 13 MB/s | 353 kB 00:00 2021-07-31T01:29:42.040 INFO:teuthology.orchestra.run.smithi045.stdout:(20/26): daxctl-libs-67-2.el8.x86_64.rpm 1.5 MB/s | 40 kB 00:00 2021-07-31T01:29:42.067 INFO:teuthology.orchestra.run.smithi045.stdout:(21/26): librabbitmq-0.9.0-2.el8.x86_64.rpm 1.7 MB/s | 47 kB 00:00 2021-07-31T01:29:42.092 INFO:teuthology.orchestra.run.smithi045.stdout:(22/26): ceph-base-16.2.5-150.g5a784c87.el8.x86 948 kB/s | 5.9 MB 00:06 2021-07-31T01:29:42.109 INFO:teuthology.orchestra.run.smithi045.stdout:(23/26): ndctl-libs-67-2.el8.x86_64.rpm 1.7 MB/s | 74 kB 00:00 2021-07-31T01:29:42.451 INFO:teuthology.orchestra.run.smithi045.stdout:(24/26): librbd1-16.2.5-150.g5a784c87.el8.x86_6 12 MB/s | 4.0 MB 00:00 2021-07-31T01:29:43.004 INFO:teuthology.orchestra.run.smithi154.stdout:(26/26): ceph-radosgw-16.2.5-150.g5a784c87.el8. 1.4 MB/s | 12 MB 00:08 2021-07-31T01:29:43.007 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2021-07-31T01:29:43.008 INFO:teuthology.orchestra.run.smithi154.stdout:Total 6.6 MB/s | 57 MB 00:08 2021-07-31T01:29:43.015 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2021-07-31T01:29:43.064 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2021-07-31T01:29:43.064 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2021-07-31T01:29:43.399 INFO:teuthology.orchestra.run.smithi045.stdout:(25/26): librados2-16.2.5-150.g5a784c87.el8.x86 2.8 MB/s | 3.7 MB 00:01 2021-07-31T01:29:43.480 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2021-07-31T01:29:43.484 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2021-07-31T01:29:44.502 INFO:teuthology.orchestra.run.smithi045.stdout:(26/26): ceph-radosgw-16.2.5-150.g5a784c87.el8. 1.3 MB/s | 12 MB 00:08 2021-07-31T01:29:44.506 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2021-07-31T01:29:44.506 INFO:teuthology.orchestra.run.smithi045.stdout:Total 6.4 MB/s | 57 MB 00:08 2021-07-31T01:29:44.514 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2021-07-31T01:29:44.568 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2021-07-31T01:29:44.569 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2021-07-31T01:29:44.709 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2021-07-31T01:29:44.833 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 1/28 2021-07-31T01:29:45.098 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2021-07-31T01:29:45.101 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2021-07-31T01:29:45.220 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 1/28 2021-07-31T01:29:45.383 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : librados2-2:16.2.5-150.g5a784c87.el8.x86_64 2/28 2021-07-31T01:29:45.479 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: librados2-2:16.2.5-150.g5a784c87.el8.x86_64 2/28 2021-07-31T01:29:45.665 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-rados-2:16.2.5-150.g5a784c87.el8.x86_64 3/28 2021-07-31T01:29:45.863 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : liboath-2.6.2-3.el8.x86_64 4/28 2021-07-31T01:29:46.066 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libcephfs2-2:16.2.5-150.g5a784c87.el8.x86_64 5/28 2021-07-31T01:29:46.134 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: libcephfs2-2:16.2.5-150.g5a784c87.el8.x86_64 5/28 2021-07-31T01:29:46.357 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : librabbitmq-0.9.0-2.el8.x86_64 6/28 2021-07-31T01:29:46.446 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2021-07-31T01:29:46.491 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 7/28 2021-07-31T01:29:46.586 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 1/28 2021-07-31T01:29:46.867 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 7/28 2021-07-31T01:29:47.019 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 1/28 2021-07-31T01:29:47.091 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : librgw2-2:16.2.5-150.g5a784c87.el8.x86_64 8/28 2021-07-31T01:29:47.154 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: librgw2-2:16.2.5-150.g5a784c87.el8.x86_64 8/28 2021-07-31T01:29:47.219 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : librados2-2:16.2.5-150.g5a784c87.el8.x86_64 2/28 2021-07-31T01:29:47.310 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librados2-2:16.2.5-150.g5a784c87.el8.x86_64 2/28 2021-07-31T01:29:47.341 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : daxctl-libs-67-2.el8.x86_64 9/28 2021-07-31T01:29:47.408 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: daxctl-libs-67-2.el8.x86_64 9/28 2021-07-31T01:29:47.527 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-rados-2:16.2.5-150.g5a784c87.el8.x86_64 3/28 2021-07-31T01:29:47.589 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/28 2021-07-31T01:29:47.763 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-ceph-argparse-2:16.2.5-150.g5a784c87.el8.x 11/28 2021-07-31T01:29:47.771 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : liboath-2.6.2-3.el8.x86_64 4/28 2021-07-31T01:29:47.969 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libcephfs2-2:16.2.5-150.g5a784c87.el8.x86_64 5/28 2021-07-31T01:29:47.999 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cephfs-2:16.2.5-150.g5a784c87.el8.x86_64 12/28 2021-07-31T01:29:48.022 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: libcephfs2-2:16.2.5-150.g5a784c87.el8.x86_64 5/28 2021-07-31T01:29:48.199 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ndctl-libs-67-2.el8.x86_64 13/28 2021-07-31T01:29:48.248 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : librabbitmq-0.9.0-2.el8.x86_64 6/28 2021-07-31T01:29:48.290 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ndctl-libs-67-2.el8.x86_64 13/28 2021-07-31T01:29:48.436 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 7/28 2021-07-31T01:29:48.516 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 14/28 2021-07-31T01:29:48.833 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 7/28 2021-07-31T01:29:48.915 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 14/28 2021-07-31T01:29:49.011 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : librgw2-2:16.2.5-150.g5a784c87.el8.x86_64 8/28 2021-07-31T01:29:49.124 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : librbd1-2:16.2.5-150.g5a784c87.el8.x86_64 15/28 2021-07-31T01:29:49.223 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: librbd1-2:16.2.5-150.g5a784c87.el8.x86_64 15/28 2021-07-31T01:29:49.269 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librgw2-2:16.2.5-150.g5a784c87.el8.x86_64 8/28 2021-07-31T01:29:49.422 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-rbd-2:16.2.5-150.g5a784c87.el8.x86_64 16/28 2021-07-31T01:29:49.511 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : daxctl-libs-67-2.el8.x86_64 9/28 2021-07-31T01:29:49.615 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: daxctl-libs-67-2.el8.x86_64 9/28 2021-07-31T01:29:49.689 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-rgw-2:16.2.5-150.g5a784c87.el8.x86_64 17/28 2021-07-31T01:29:49.835 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/28 2021-07-31T01:29:49.849 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libradosstriper1-2:16.2.5-150.g5a784c87.el8.x86_64 18/28 2021-07-31T01:29:49.925 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: libradosstriper1-2:16.2.5-150.g5a784c87.el8.x86_64 18/28 2021-07-31T01:29:50.003 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-ceph-argparse-2:16.2.5-150.g5a784c87.el8.x 11/28 2021-07-31T01:29:50.115 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 19/28 2021-07-31T01:29:50.256 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cephfs-2:16.2.5-150.g5a784c87.el8.x86_64 12/28 2021-07-31T01:29:50.358 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/28 2021-07-31T01:29:50.452 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ndctl-libs-67-2.el8.x86_64 13/28 2021-07-31T01:29:50.520 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ndctl-libs-67-2.el8.x86_64 13/28 2021-07-31T01:29:50.561 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/28 2021-07-31T01:29:50.730 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 14/28 2021-07-31T01:29:50.757 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-ceph-common-2:16.2.5-150.g5a784c87.el8.x86 22/28 2021-07-31T01:29:51.131 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 14/28 2021-07-31T01:29:51.369 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : librbd1-2:16.2.5-150.g5a784c87.el8.x86_64 15/28 2021-07-31T01:29:51.473 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librbd1-2:16.2.5-150.g5a784c87.el8.x86_64 15/28 2021-07-31T01:29:51.675 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-rbd-2:16.2.5-150.g5a784c87.el8.x86_64 16/28 2021-07-31T01:29:51.921 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-rgw-2:16.2.5-150.g5a784c87.el8.x86_64 17/28 2021-07-31T01:29:52.119 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libradosstriper1-2:16.2.5-150.g5a784c87.el8.x86_64 18/28 2021-07-31T01:29:52.198 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: libradosstriper1-2:16.2.5-150.g5a784c87.el8.x86_64 18/28 2021-07-31T01:29:52.403 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 19/28 2021-07-31T01:29:52.631 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/28 2021-07-31T01:29:52.829 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/28 2021-07-31T01:29:53.011 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-ceph-common-2:16.2.5-150.g5a784c87.el8.x86 22/28 2021-07-31T01:29:54.940 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-common-2:16.2.5-150.g5a784c87.el8.x86_64 23/28 2021-07-31T01:29:55.140 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-common-2:16.2.5-150.g5a784c87.el8.x86_64 23/28 2021-07-31T01:29:55.823 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-common-2:16.2.5-150.g5a784c87.el8.x86_64 23/28 2021-07-31T01:29:56.100 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-base-2:16.2.5-150.g5a784c87.el8.x86_64 24/28 2021-07-31T01:29:56.192 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-base-2:16.2.5-150.g5a784c87.el8.x86_64 24/28 2021-07-31T01:29:56.322 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-selinux-2:16.2.5-150.g5a784c87.el8.x86_64 25/28 2021-07-31T01:29:58.078 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-common-2:16.2.5-150.g5a784c87.el8.x86_64 23/28 2021-07-31T01:29:58.226 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-common-2:16.2.5-150.g5a784c87.el8.x86_64 23/28 2021-07-31T01:29:58.971 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-common-2:16.2.5-150.g5a784c87.el8.x86_64 23/28 2021-07-31T01:29:59.282 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-base-2:16.2.5-150.g5a784c87.el8.x86_64 24/28 2021-07-31T01:29:59.372 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-base-2:16.2.5-150.g5a784c87.el8.x86_64 24/28 2021-07-31T01:29:59.509 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-selinux-2:16.2.5-150.g5a784c87.el8.x86_64 25/28 2021-07-31T01:30:13.392 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-selinux-2:16.2.5-150.g5a784c87.el8.x86_64 25/28 2021-07-31T01:30:13.393 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /sys 2021-07-31T01:30:13.393 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /proc 2021-07-31T01:30:13.393 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /dev 2021-07-31T01:30:13.393 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /run 2021-07-31T01:30:13.394 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /mnt 2021-07-31T01:30:13.394 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /var/tmp 2021-07-31T01:30:13.394 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /home 2021-07-31T01:30:13.394 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /tmp 2021-07-31T01:30:13.395 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /dev 2021-07-31T01:30:13.395 INFO:teuthology.orchestra.run.smithi154.stdout: 2021-07-31T01:30:14.606 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-radosgw-2:16.2.5-150.g5a784c87.el8.x86_64 26/28 2021-07-31T01:30:14.685 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-radosgw-2:16.2.5-150.g5a784c87.el8.x86_64 26/28 2021-07-31T01:30:14.686 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 27/28 2021-07-31T01:30:15.430 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 27/28 2021-07-31T01:30:15.430 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 28/28 2021-07-31T01:30:16.106 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 28/28 2021-07-31T01:30:16.106 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-base-2:16.2.5-150.g5a784c87.el8.x86_64 1/28 2021-07-31T01:30:16.107 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-common-2:16.2.5-150.g5a784c87.el8.x86_64 2/28 2021-07-31T01:30:16.107 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-radosgw-2:16.2.5-150.g5a784c87.el8.x86_64 3/28 2021-07-31T01:30:16.107 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-selinux-2:16.2.5-150.g5a784c87.el8.x86_64 4/28 2021-07-31T01:30:16.107 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libcephfs2-2:16.2.5-150.g5a784c87.el8.x86_64 5/28 2021-07-31T01:30:16.108 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libradosstriper1-2:16.2.5-150.g5a784c87.el8.x86_64 6/28 2021-07-31T01:30:16.108 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librgw2-2:16.2.5-150.g5a784c87.el8.x86_64 7/28 2021-07-31T01:30:16.108 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-ceph-argparse-2:16.2.5-150.g5a784c87.el8.x 8/28 2021-07-31T01:30:16.108 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-ceph-common-2:16.2.5-150.g5a784c87.el8.x86 9/28 2021-07-31T01:30:16.108 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cephfs-2:16.2.5-150.g5a784c87.el8.x86_64 10/28 2021-07-31T01:30:16.109 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-rados-2:16.2.5-150.g5a784c87.el8.x86_64 11/28 2021-07-31T01:30:16.109 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-rbd-2:16.2.5-150.g5a784c87.el8.x86_64 12/28 2021-07-31T01:30:16.109 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-rgw-2:16.2.5-150.g5a784c87.el8.x86_64 13/28 2021-07-31T01:30:16.109 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 14/28 2021-07-31T01:30:16.110 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : leveldb-1.22-1.el8.x86_64 15/28 2021-07-31T01:30:16.110 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 16/28 2021-07-31T01:30:16.110 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 17/28 2021-07-31T01:30:16.110 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 18/28 2021-07-31T01:30:16.111 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 19/28 2021-07-31T01:30:16.111 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 20/28 2021-07-31T01:30:16.111 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 21/28 2021-07-31T01:30:16.111 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : daxctl-libs-67-2.el8.x86_64 22/28 2021-07-31T01:30:16.111 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librabbitmq-0.9.0-2.el8.x86_64 23/28 2021-07-31T01:30:16.112 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ndctl-libs-67-2.el8.x86_64 24/28 2021-07-31T01:30:16.112 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librados2-2:16.2.5-150.g5a784c87.el8.x86_64 25/28 2021-07-31T01:30:16.112 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 26/28 2021-07-31T01:30:16.112 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librbd1-2:16.2.5-150.g5a784c87.el8.x86_64 27/28 2021-07-31T01:30:17.236 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 28/28 2021-07-31T01:30:17.237 INFO:teuthology.orchestra.run.smithi154.stdout:Installed products updated. 2021-07-31T01:30:17.396 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-selinux-2:16.2.5-150.g5a784c87.el8.x86_64 25/28 2021-07-31T01:30:17.397 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /sys 2021-07-31T01:30:17.397 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /proc 2021-07-31T01:30:17.398 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /dev 2021-07-31T01:30:17.399 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /run 2021-07-31T01:30:17.399 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /mnt 2021-07-31T01:30:17.400 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /var/tmp 2021-07-31T01:30:17.400 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /home 2021-07-31T01:30:17.400 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /tmp 2021-07-31T01:30:17.401 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /dev 2021-07-31T01:30:17.401 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:30:18.646 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-radosgw-2:16.2.5-150.g5a784c87.el8.x86_64 26/28 2021-07-31T01:30:18.720 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-radosgw-2:16.2.5-150.g5a784c87.el8.x86_64 26/28 2021-07-31T01:30:18.720 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 27/28 2021-07-31T01:30:18.950 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 27/28 2021-07-31T01:30:18.951 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 28/28 2021-07-31T01:30:19.682 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 28/28 2021-07-31T01:30:19.682 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-base-2:16.2.5-150.g5a784c87.el8.x86_64 1/28 2021-07-31T01:30:19.683 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-common-2:16.2.5-150.g5a784c87.el8.x86_64 2/28 2021-07-31T01:30:19.683 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-radosgw-2:16.2.5-150.g5a784c87.el8.x86_64 3/28 2021-07-31T01:30:19.683 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-selinux-2:16.2.5-150.g5a784c87.el8.x86_64 4/28 2021-07-31T01:30:19.683 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libcephfs2-2:16.2.5-150.g5a784c87.el8.x86_64 5/28 2021-07-31T01:30:19.684 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libradosstriper1-2:16.2.5-150.g5a784c87.el8.x86_64 6/28 2021-07-31T01:30:19.684 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librgw2-2:16.2.5-150.g5a784c87.el8.x86_64 7/28 2021-07-31T01:30:19.684 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-ceph-argparse-2:16.2.5-150.g5a784c87.el8.x 8/28 2021-07-31T01:30:19.685 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-ceph-common-2:16.2.5-150.g5a784c87.el8.x86 9/28 2021-07-31T01:30:19.685 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cephfs-2:16.2.5-150.g5a784c87.el8.x86_64 10/28 2021-07-31T01:30:19.685 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-rados-2:16.2.5-150.g5a784c87.el8.x86_64 11/28 2021-07-31T01:30:19.685 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-rbd-2:16.2.5-150.g5a784c87.el8.x86_64 12/28 2021-07-31T01:30:19.686 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-rgw-2:16.2.5-150.g5a784c87.el8.x86_64 13/28 2021-07-31T01:30:19.686 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 14/28 2021-07-31T01:30:19.686 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : leveldb-1.22-1.el8.x86_64 15/28 2021-07-31T01:30:19.686 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 16/28 2021-07-31T01:30:19.687 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 17/28 2021-07-31T01:30:19.687 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 18/28 2021-07-31T01:30:19.687 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 19/28 2021-07-31T01:30:19.687 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 20/28 2021-07-31T01:30:19.688 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 21/28 2021-07-31T01:30:19.688 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : daxctl-libs-67-2.el8.x86_64 22/28 2021-07-31T01:30:19.688 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librabbitmq-0.9.0-2.el8.x86_64 23/28 2021-07-31T01:30:19.688 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ndctl-libs-67-2.el8.x86_64 24/28 2021-07-31T01:30:19.689 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librados2-2:16.2.5-150.g5a784c87.el8.x86_64 25/28 2021-07-31T01:30:19.689 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 26/28 2021-07-31T01:30:19.689 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librbd1-2:16.2.5-150.g5a784c87.el8.x86_64 27/28 2021-07-31T01:30:20.643 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 28/28 2021-07-31T01:30:20.644 INFO:teuthology.orchestra.run.smithi045.stdout:Installed products updated. 2021-07-31T01:30:27.625 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:00:05 ago on Sat 31 Jul 2021 01:30:22 AM UTC. 2021-07-31T01:30:30.472 INFO:teuthology.orchestra.run.smithi154.stdout: 2021-07-31T01:30:30.472 INFO:teuthology.orchestra.run.smithi154.stdout:Upgraded: 2021-07-31T01:30:30.473 INFO:teuthology.orchestra.run.smithi154.stdout: librados2-2:16.2.5-150.g5a784c87.el8.x86_64 2021-07-31T01:30:30.473 INFO:teuthology.orchestra.run.smithi154.stdout: librbd1-2:16.2.5-150.g5a784c87.el8.x86_64 2021-07-31T01:30:30.473 INFO:teuthology.orchestra.run.smithi154.stdout: 2021-07-31T01:30:30.473 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2021-07-31T01:30:30.474 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-base-2:16.2.5-150.g5a784c87.el8.x86_64 2021-07-31T01:30:30.474 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-common-2:16.2.5-150.g5a784c87.el8.x86_64 2021-07-31T01:30:30.474 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-radosgw-2:16.2.5-150.g5a784c87.el8.x86_64 2021-07-31T01:30:30.475 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-selinux-2:16.2.5-150.g5a784c87.el8.x86_64 2021-07-31T01:30:30.475 INFO:teuthology.orchestra.run.smithi154.stdout: daxctl-libs-67-2.el8.x86_64 2021-07-31T01:30:30.475 INFO:teuthology.orchestra.run.smithi154.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2021-07-31T01:30:30.476 INFO:teuthology.orchestra.run.smithi154.stdout: leveldb-1.22-1.el8.x86_64 2021-07-31T01:30:30.476 INFO:teuthology.orchestra.run.smithi154.stdout: libcephfs2-2:16.2.5-150.g5a784c87.el8.x86_64 2021-07-31T01:30:30.476 INFO:teuthology.orchestra.run.smithi154.stdout: liboath-2.6.2-3.el8.x86_64 2021-07-31T01:30:30.476 INFO:teuthology.orchestra.run.smithi154.stdout: libpmemobj-1.6.1-1.el8.x86_64 2021-07-31T01:30:30.476 INFO:teuthology.orchestra.run.smithi154.stdout: librabbitmq-0.9.0-2.el8.x86_64 2021-07-31T01:30:30.477 INFO:teuthology.orchestra.run.smithi154.stdout: libradosstriper1-2:16.2.5-150.g5a784c87.el8.x86_64 2021-07-31T01:30:30.477 INFO:teuthology.orchestra.run.smithi154.stdout: librdkafka-0.11.4-1.el8.x86_64 2021-07-31T01:30:30.477 INFO:teuthology.orchestra.run.smithi154.stdout: librgw2-2:16.2.5-150.g5a784c87.el8.x86_64 2021-07-31T01:30:30.477 INFO:teuthology.orchestra.run.smithi154.stdout: libunwind-1.3.1-3.el8.x86_64 2021-07-31T01:30:30.477 INFO:teuthology.orchestra.run.smithi154.stdout: lttng-ust-2.8.1-11.el8.x86_64 2021-07-31T01:30:30.478 INFO:teuthology.orchestra.run.smithi154.stdout: ndctl-libs-67-2.el8.x86_64 2021-07-31T01:30:30.478 INFO:teuthology.orchestra.run.smithi154.stdout: python3-ceph-argparse-2:16.2.5-150.g5a784c87.el8.x86_64 2021-07-31T01:30:30.478 INFO:teuthology.orchestra.run.smithi154.stdout: python3-ceph-common-2:16.2.5-150.g5a784c87.el8.x86_64 2021-07-31T01:30:30.479 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cephfs-2:16.2.5-150.g5a784c87.el8.x86_64 2021-07-31T01:30:30.479 INFO:teuthology.orchestra.run.smithi154.stdout: python3-prettytable-0.7.2-14.el8.noarch 2021-07-31T01:30:30.479 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rados-2:16.2.5-150.g5a784c87.el8.x86_64 2021-07-31T01:30:30.479 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rbd-2:16.2.5-150.g5a784c87.el8.x86_64 2021-07-31T01:30:30.479 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rgw-2:16.2.5-150.g5a784c87.el8.x86_64 2021-07-31T01:30:30.480 INFO:teuthology.orchestra.run.smithi154.stdout: 2021-07-31T01:30:30.480 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2021-07-31T01:30:30.594 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-test 2021-07-31T01:30:30.873 INFO:teuthology.orchestra.run.smithi154.stdout:Updating Subscription Management repositories. 2021-07-31T01:30:31.357 INFO:teuthology.orchestra.run.smithi154.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 95 kB/s | 2.8 kB 00:00 2021-07-31T01:30:32.136 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:05 ago on Sat 31 Jul 2021 01:30:26 AM UTC. 2021-07-31T01:30:35.176 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:30:35.177 INFO:teuthology.orchestra.run.smithi045.stdout:Upgraded: 2021-07-31T01:30:35.177 INFO:teuthology.orchestra.run.smithi045.stdout: librados2-2:16.2.5-150.g5a784c87.el8.x86_64 2021-07-31T01:30:35.177 INFO:teuthology.orchestra.run.smithi045.stdout: librbd1-2:16.2.5-150.g5a784c87.el8.x86_64 2021-07-31T01:30:35.178 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:30:35.178 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2021-07-31T01:30:35.178 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-base-2:16.2.5-150.g5a784c87.el8.x86_64 2021-07-31T01:30:35.178 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-common-2:16.2.5-150.g5a784c87.el8.x86_64 2021-07-31T01:30:35.179 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-radosgw-2:16.2.5-150.g5a784c87.el8.x86_64 2021-07-31T01:30:35.179 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-selinux-2:16.2.5-150.g5a784c87.el8.x86_64 2021-07-31T01:30:35.179 INFO:teuthology.orchestra.run.smithi045.stdout: daxctl-libs-67-2.el8.x86_64 2021-07-31T01:30:35.180 INFO:teuthology.orchestra.run.smithi045.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2021-07-31T01:30:35.180 INFO:teuthology.orchestra.run.smithi045.stdout: leveldb-1.22-1.el8.x86_64 2021-07-31T01:30:35.180 INFO:teuthology.orchestra.run.smithi045.stdout: libcephfs2-2:16.2.5-150.g5a784c87.el8.x86_64 2021-07-31T01:30:35.180 INFO:teuthology.orchestra.run.smithi045.stdout: liboath-2.6.2-3.el8.x86_64 2021-07-31T01:30:35.180 INFO:teuthology.orchestra.run.smithi045.stdout: libpmemobj-1.6.1-1.el8.x86_64 2021-07-31T01:30:35.181 INFO:teuthology.orchestra.run.smithi045.stdout: librabbitmq-0.9.0-2.el8.x86_64 2021-07-31T01:30:35.181 INFO:teuthology.orchestra.run.smithi045.stdout: libradosstriper1-2:16.2.5-150.g5a784c87.el8.x86_64 2021-07-31T01:30:35.181 INFO:teuthology.orchestra.run.smithi045.stdout: librdkafka-0.11.4-1.el8.x86_64 2021-07-31T01:30:35.181 INFO:teuthology.orchestra.run.smithi045.stdout: librgw2-2:16.2.5-150.g5a784c87.el8.x86_64 2021-07-31T01:30:35.182 INFO:teuthology.orchestra.run.smithi045.stdout: libunwind-1.3.1-3.el8.x86_64 2021-07-31T01:30:35.182 INFO:teuthology.orchestra.run.smithi045.stdout: lttng-ust-2.8.1-11.el8.x86_64 2021-07-31T01:30:35.182 INFO:teuthology.orchestra.run.smithi045.stdout: ndctl-libs-67-2.el8.x86_64 2021-07-31T01:30:35.182 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ceph-argparse-2:16.2.5-150.g5a784c87.el8.x86_64 2021-07-31T01:30:35.182 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ceph-common-2:16.2.5-150.g5a784c87.el8.x86_64 2021-07-31T01:30:35.183 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cephfs-2:16.2.5-150.g5a784c87.el8.x86_64 2021-07-31T01:30:35.183 INFO:teuthology.orchestra.run.smithi045.stdout: python3-prettytable-0.7.2-14.el8.noarch 2021-07-31T01:30:35.183 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rados-2:16.2.5-150.g5a784c87.el8.x86_64 2021-07-31T01:30:35.183 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rbd-2:16.2.5-150.g5a784c87.el8.x86_64 2021-07-31T01:30:35.184 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rgw-2:16.2.5-150.g5a784c87.el8.x86_64 2021-07-31T01:30:35.184 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:30:35.184 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2021-07-31T01:30:35.318 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-test 2021-07-31T01:30:35.409 INFO:teuthology.orchestra.run.smithi154.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 72 kB/s | 2.4 kB 00:00 2021-07-31T01:30:35.614 INFO:teuthology.orchestra.run.smithi045.stdout:Updating Subscription Management repositories. 2021-07-31T01:30:36.162 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 76 kB/s | 2.8 kB 00:00 2021-07-31T01:30:39.758 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:00:04 ago on Sat 31 Jul 2021 01:29:23 AM UTC. 2021-07-31T01:30:40.375 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 63 kB/s | 2.4 kB 00:00 2021-07-31T01:30:41.840 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2021-07-31T01:30:41.841 INFO:teuthology.orchestra.run.smithi154.stdout:=========================================================================================== 2021-07-31T01:30:41.842 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repository Size 2021-07-31T01:30:41.842 INFO:teuthology.orchestra.run.smithi154.stdout:=========================================================================================== 2021-07-31T01:30:41.842 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2021-07-31T01:30:41.842 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-test x86_64 2:16.2.5-150.g5a784c87.el8 ceph 52 M 2021-07-31T01:30:41.843 INFO:teuthology.orchestra.run.smithi154.stdout:Installing dependencies: 2021-07-31T01:30:41.843 INFO:teuthology.orchestra.run.smithi154.stdout: jq x86_64 1.5-12.el8 rhel-8-for-x86_64-appstream-rpms 161 k 2021-07-31T01:30:41.843 INFO:teuthology.orchestra.run.smithi154.stdout: libcephsqlite x86_64 2:16.2.5-150.g5a784c87.el8 ceph 173 k 2021-07-31T01:30:41.843 INFO:teuthology.orchestra.run.smithi154.stdout: libxslt x86_64 1.1.32-5.el8 rhel-8-for-x86_64-baseos-rpms 250 k 2021-07-31T01:30:41.843 INFO:teuthology.orchestra.run.smithi154.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2021-07-31T01:30:41.844 INFO:teuthology.orchestra.run.smithi154.stdout: socat x86_64 1.7.3.3-2.el8 rhel-8-for-x86_64-appstream-rpms 302 k 2021-07-31T01:30:41.844 INFO:teuthology.orchestra.run.smithi154.stdout: xmlstarlet x86_64 1.6.1-11.el8 epel 69 k 2021-07-31T01:30:41.844 INFO:teuthology.orchestra.run.smithi154.stdout: 2021-07-31T01:30:41.844 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2021-07-31T01:30:41.844 INFO:teuthology.orchestra.run.smithi154.stdout:=========================================================================================== 2021-07-31T01:30:41.845 INFO:teuthology.orchestra.run.smithi154.stdout:Install 7 Packages 2021-07-31T01:30:41.845 INFO:teuthology.orchestra.run.smithi154.stdout: 2021-07-31T01:30:41.846 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 53 M 2021-07-31T01:30:41.846 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 194 M 2021-07-31T01:30:41.846 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2021-07-31T01:30:41.958 INFO:teuthology.orchestra.run.smithi154.stdout:(1/7): xmlstarlet-1.6.1-11.el8.x86_64.rpm 606 kB/s | 69 kB 00:00 2021-07-31T01:30:41.990 INFO:teuthology.orchestra.run.smithi154.stdout:(2/7): socat-1.7.3.3-2.el8.x86_64.rpm 9.2 MB/s | 302 kB 00:00 2021-07-31T01:30:42.011 INFO:teuthology.orchestra.run.smithi154.stdout:(3/7): oniguruma-6.8.2-2.el8.x86_64.rpm 8.7 MB/s | 187 kB 00:00 2021-07-31T01:30:42.032 INFO:teuthology.orchestra.run.smithi154.stdout:(4/7): jq-1.5-12.el8.x86_64.rpm 7.7 MB/s | 161 kB 00:00 2021-07-31T01:30:42.054 INFO:teuthology.orchestra.run.smithi154.stdout:(5/7): libxslt-1.1.32-5.el8.x86_64.rpm 11 MB/s | 250 kB 00:00 2021-07-31T01:30:43.524 INFO:teuthology.orchestra.run.smithi154.stdout:(6/7): libcephsqlite-16.2.5-150.g5a784c87.el8.x 103 kB/s | 173 kB 00:01 2021-07-31T01:30:45.117 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:04 ago on Sat 31 Jul 2021 01:29:23 AM UTC. 2021-07-31T01:30:46.676 INFO:teuthology.orchestra.run.smithi154.stdout:(7/7): ceph-test-16.2.5-150.g5a784c87.el8.x86_6 11 MB/s | 52 MB 00:04 2021-07-31T01:30:46.680 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2021-07-31T01:30:46.680 INFO:teuthology.orchestra.run.smithi154.stdout:Total 11 MB/s | 53 MB 00:04 2021-07-31T01:30:46.685 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2021-07-31T01:30:46.709 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2021-07-31T01:30:46.709 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2021-07-31T01:30:46.986 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2021-07-31T01:30:46.990 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2021-07-31T01:30:47.472 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2021-07-31T01:30:47.473 INFO:teuthology.orchestra.run.smithi045.stdout:=========================================================================================== 2021-07-31T01:30:47.475 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2021-07-31T01:30:47.475 INFO:teuthology.orchestra.run.smithi045.stdout:=========================================================================================== 2021-07-31T01:30:47.475 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2021-07-31T01:30:47.475 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-test x86_64 2:16.2.5-150.g5a784c87.el8 ceph 52 M 2021-07-31T01:30:47.476 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2021-07-31T01:30:47.476 INFO:teuthology.orchestra.run.smithi045.stdout: jq x86_64 1.5-12.el8 rhel-8-for-x86_64-appstream-rpms 161 k 2021-07-31T01:30:47.476 INFO:teuthology.orchestra.run.smithi045.stdout: libcephsqlite x86_64 2:16.2.5-150.g5a784c87.el8 ceph 173 k 2021-07-31T01:30:47.476 INFO:teuthology.orchestra.run.smithi045.stdout: libxslt x86_64 1.1.32-5.el8 rhel-8-for-x86_64-baseos-rpms 250 k 2021-07-31T01:30:47.476 INFO:teuthology.orchestra.run.smithi045.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2021-07-31T01:30:47.477 INFO:teuthology.orchestra.run.smithi045.stdout: socat x86_64 1.7.3.3-2.el8 rhel-8-for-x86_64-appstream-rpms 302 k 2021-07-31T01:30:47.477 INFO:teuthology.orchestra.run.smithi045.stdout: xmlstarlet x86_64 1.6.1-11.el8 epel 69 k 2021-07-31T01:30:47.477 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:30:47.477 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2021-07-31T01:30:47.478 INFO:teuthology.orchestra.run.smithi045.stdout:=========================================================================================== 2021-07-31T01:30:47.478 INFO:teuthology.orchestra.run.smithi045.stdout:Install 7 Packages 2021-07-31T01:30:47.478 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:30:47.479 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 53 M 2021-07-31T01:30:47.479 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 194 M 2021-07-31T01:30:47.479 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2021-07-31T01:30:47.545 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2021-07-31T01:30:47.565 INFO:teuthology.orchestra.run.smithi045.stdout:(1/7): xmlstarlet-1.6.1-11.el8.x86_64.rpm 778 kB/s | 69 kB 00:00 2021-07-31T01:30:47.605 INFO:teuthology.orchestra.run.smithi045.stdout:(2/7): socat-1.7.3.3-2.el8.x86_64.rpm 7.3 MB/s | 302 kB 00:00 2021-07-31T01:30:47.631 INFO:teuthology.orchestra.run.smithi045.stdout:(3/7): oniguruma-6.8.2-2.el8.x86_64.rpm 7.2 MB/s | 187 kB 00:00 2021-07-31T01:30:47.650 INFO:teuthology.orchestra.run.smithi045.stdout:(4/7): libcephsqlite-16.2.5-150.g5a784c87.el8.x 994 kB/s | 173 kB 00:00 2021-07-31T01:30:47.662 INFO:teuthology.orchestra.run.smithi045.stdout:(5/7): jq-1.5-12.el8.x86_64.rpm 5.1 MB/s | 161 kB 00:00 2021-07-31T01:30:47.688 INFO:teuthology.orchestra.run.smithi045.stdout:(6/7): libxslt-1.1.32-5.el8.x86_64.rpm 6.4 MB/s | 250 kB 00:00 2021-07-31T01:30:48.535 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libxslt-1.1.32-5.el8.x86_64 1/7 2021-07-31T01:30:48.710 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : xmlstarlet-1.6.1-11.el8.x86_64 2/7 2021-07-31T01:30:48.864 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/7 2021-07-31T01:30:48.939 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/7 2021-07-31T01:30:49.145 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : jq-1.5-12.el8.x86_64 4/7 2021-07-31T01:30:49.345 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : socat-1.7.3.3-2.el8.x86_64 5/7 2021-07-31T01:30:49.530 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libcephsqlite-2:16.2.5-150.g5a784c87.el8.x86_64 6/7 2021-07-31T01:30:50.465 INFO:teuthology.orchestra.run.smithi045.stdout:(7/7): ceph-test-16.2.5-150.g5a784c87.el8.x86_6 17 MB/s | 52 MB 00:02 2021-07-31T01:30:50.469 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2021-07-31T01:30:50.469 INFO:teuthology.orchestra.run.smithi045.stdout:Total 18 MB/s | 53 MB 00:02 2021-07-31T01:30:50.474 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2021-07-31T01:30:50.501 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2021-07-31T01:30:50.501 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2021-07-31T01:30:50.792 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2021-07-31T01:30:50.796 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2021-07-31T01:30:51.359 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2021-07-31T01:30:51.604 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libxslt-1.1.32-5.el8.x86_64 1/7 2021-07-31T01:30:51.804 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : xmlstarlet-1.6.1-11.el8.x86_64 2/7 2021-07-31T01:30:51.994 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/7 2021-07-31T01:30:52.069 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/7 2021-07-31T01:30:52.310 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : jq-1.5-12.el8.x86_64 4/7 2021-07-31T01:30:52.517 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : socat-1.7.3.3-2.el8.x86_64 5/7 2021-07-31T01:30:52.651 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libcephsqlite-2:16.2.5-150.g5a784c87.el8.x86_64 6/7 2021-07-31T01:30:54.058 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: libcephsqlite-2:16.2.5-150.g5a784c87.el8.x86_64 6/7 2021-07-31T01:30:54.318 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-test-2:16.2.5-150.g5a784c87.el8.x86_64 7/7 2021-07-31T01:30:54.536 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-test-2:16.2.5-150.g5a784c87.el8.x86_64 7/7 2021-07-31T01:30:54.536 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-test-2:16.2.5-150.g5a784c87.el8.x86_64 1/7 2021-07-31T01:30:54.536 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libcephsqlite-2:16.2.5-150.g5a784c87.el8.x86_64 2/7 2021-07-31T01:30:54.537 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : xmlstarlet-1.6.1-11.el8.x86_64 3/7 2021-07-31T01:30:54.537 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : socat-1.7.3.3-2.el8.x86_64 4/7 2021-07-31T01:30:54.537 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 5/7 2021-07-31T01:30:54.537 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : jq-1.5-12.el8.x86_64 6/7 2021-07-31T01:30:55.066 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libxslt-1.1.32-5.el8.x86_64 7/7 2021-07-31T01:30:55.066 INFO:teuthology.orchestra.run.smithi154.stdout:Installed products updated. 2021-07-31T01:30:57.538 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: libcephsqlite-2:16.2.5-150.g5a784c87.el8.x86_64 6/7 2021-07-31T01:30:57.840 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-test-2:16.2.5-150.g5a784c87.el8.x86_64 7/7 2021-07-31T01:30:58.083 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-test-2:16.2.5-150.g5a784c87.el8.x86_64 7/7 2021-07-31T01:30:58.084 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-test-2:16.2.5-150.g5a784c87.el8.x86_64 1/7 2021-07-31T01:30:58.084 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libcephsqlite-2:16.2.5-150.g5a784c87.el8.x86_64 2/7 2021-07-31T01:30:58.084 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : xmlstarlet-1.6.1-11.el8.x86_64 3/7 2021-07-31T01:30:58.085 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : socat-1.7.3.3-2.el8.x86_64 4/7 2021-07-31T01:30:58.085 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 5/7 2021-07-31T01:30:58.086 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : jq-1.5-12.el8.x86_64 6/7 2021-07-31T01:30:58.637 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libxslt-1.1.32-5.el8.x86_64 7/7 2021-07-31T01:30:58.637 INFO:teuthology.orchestra.run.smithi045.stdout:Installed products updated. 2021-07-31T01:31:05.319 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:00:05 ago on Sat 31 Jul 2021 01:30:22 AM UTC. 2021-07-31T01:31:07.478 INFO:teuthology.orchestra.run.smithi154.stdout: 2021-07-31T01:31:07.479 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2021-07-31T01:31:07.479 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-test-2:16.2.5-150.g5a784c87.el8.x86_64 jq-1.5-12.el8.x86_64 2021-07-31T01:31:07.479 INFO:teuthology.orchestra.run.smithi154.stdout: libcephsqlite-2:16.2.5-150.g5a784c87.el8.x86_64 libxslt-1.1.32-5.el8.x86_64 2021-07-31T01:31:07.479 INFO:teuthology.orchestra.run.smithi154.stdout: oniguruma-6.8.2-2.el8.x86_64 socat-1.7.3.3-2.el8.x86_64 2021-07-31T01:31:07.480 INFO:teuthology.orchestra.run.smithi154.stdout: xmlstarlet-1.6.1-11.el8.x86_64 2021-07-31T01:31:07.480 INFO:teuthology.orchestra.run.smithi154.stdout: 2021-07-31T01:31:07.480 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2021-07-31T01:31:07.564 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph 2021-07-31T01:31:07.835 INFO:teuthology.orchestra.run.smithi154.stdout:Updating Subscription Management repositories. 2021-07-31T01:31:08.318 INFO:teuthology.orchestra.run.smithi154.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 95 kB/s | 2.8 kB 00:00 2021-07-31T01:31:09.242 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:05 ago on Sat 31 Jul 2021 01:30:26 AM UTC. 2021-07-31T01:31:11.548 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:31:11.548 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2021-07-31T01:31:11.549 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-test-2:16.2.5-150.g5a784c87.el8.x86_64 jq-1.5-12.el8.x86_64 2021-07-31T01:31:11.549 INFO:teuthology.orchestra.run.smithi045.stdout: libcephsqlite-2:16.2.5-150.g5a784c87.el8.x86_64 libxslt-1.1.32-5.el8.x86_64 2021-07-31T01:31:11.549 INFO:teuthology.orchestra.run.smithi045.stdout: oniguruma-6.8.2-2.el8.x86_64 socat-1.7.3.3-2.el8.x86_64 2021-07-31T01:31:11.549 INFO:teuthology.orchestra.run.smithi045.stdout: xmlstarlet-1.6.1-11.el8.x86_64 2021-07-31T01:31:11.550 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:31:11.551 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2021-07-31T01:31:11.645 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph 2021-07-31T01:31:11.930 INFO:teuthology.orchestra.run.smithi045.stdout:Updating Subscription Management repositories. 2021-07-31T01:31:12.308 INFO:teuthology.orchestra.run.smithi154.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 83 kB/s | 2.4 kB 00:00 2021-07-31T01:31:12.510 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 73 kB/s | 2.8 kB 00:00 2021-07-31T01:31:16.495 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:00:04 ago on Sat 31 Jul 2021 01:29:23 AM UTC. 2021-07-31T01:31:16.733 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 64 kB/s | 2.4 kB 00:00 2021-07-31T01:31:18.615 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2021-07-31T01:31:18.619 INFO:teuthology.orchestra.run.smithi154.stdout:====================================================================================================== 2021-07-31T01:31:18.619 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repository Size 2021-07-31T01:31:18.619 INFO:teuthology.orchestra.run.smithi154.stdout:====================================================================================================== 2021-07-31T01:31:18.620 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2021-07-31T01:31:18.620 INFO:teuthology.orchestra.run.smithi154.stdout: ceph x86_64 2:16.2.5-150.g5a784c87.el8 ceph 6.4 k 2021-07-31T01:31:18.621 INFO:teuthology.orchestra.run.smithi154.stdout:Installing dependencies: 2021-07-31T01:31:18.621 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mds x86_64 2:16.2.5-150.g5a784c87.el8 ceph 2.4 M 2021-07-31T01:31:18.621 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr x86_64 2:16.2.5-150.g5a784c87.el8 ceph 1.6 M 2021-07-31T01:31:18.622 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-modules-core noarch 2:16.2.5-150.g5a784c87.el8 ceph-noarch 224 k 2021-07-31T01:31:18.623 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mon x86_64 2:16.2.5-150.g5a784c87.el8 ceph 5.0 M 2021-07-31T01:31:18.623 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-osd x86_64 2:16.2.5-150.g5a784c87.el8 ceph 18 M 2021-07-31T01:31:18.623 INFO:teuthology.orchestra.run.smithi154.stdout: python3-asn1crypto noarch 0.24.0-3.el8 rhel-8-for-x86_64-baseos-rpms 181 k 2021-07-31T01:31:18.624 INFO:teuthology.orchestra.run.smithi154.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2021-07-31T01:31:18.625 INFO:teuthology.orchestra.run.smithi154.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2021-07-31T01:31:18.626 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2021-07-31T01:31:18.626 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2021-07-31T01:31:18.626 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2021-07-31T01:31:18.626 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cryptography x86_64 2.3-3.el8 rhel-8-for-x86_64-baseos-rpms 511 k 2021-07-31T01:31:18.627 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2021-07-31T01:31:18.627 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2021-07-31T01:31:18.627 INFO:teuthology.orchestra.run.smithi154.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2021-07-31T01:31:18.627 INFO:teuthology.orchestra.run.smithi154.stdout: python3-lxml x86_64 4.2.3-1.el8 rhel-8-for-x86_64-appstream-rpms 1.5 M 2021-07-31T01:31:18.627 INFO:teuthology.orchestra.run.smithi154.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2021-07-31T01:31:18.628 INFO:teuthology.orchestra.run.smithi154.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2021-07-31T01:31:18.628 INFO:teuthology.orchestra.run.smithi154.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2021-07-31T01:31:18.628 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2021-07-31T01:31:18.628 INFO:teuthology.orchestra.run.smithi154.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2021-07-31T01:31:18.629 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyOpenSSL noarch 18.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2021-07-31T01:31:18.629 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2021-07-31T01:31:18.629 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2021-07-31T01:31:18.629 INFO:teuthology.orchestra.run.smithi154.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2021-07-31T01:31:18.629 INFO:teuthology.orchestra.run.smithi154.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2021-07-31T01:31:18.630 INFO:teuthology.orchestra.run.smithi154.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2021-07-31T01:31:18.630 INFO:teuthology.orchestra.run.smithi154.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2021-07-31T01:31:18.630 INFO:teuthology.orchestra.run.smithi154.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2021-07-31T01:31:18.630 INFO:teuthology.orchestra.run.smithi154.stdout: python3-webencodings noarch 0.5.1-6.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2021-07-31T01:31:18.631 INFO:teuthology.orchestra.run.smithi154.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2021-07-31T01:31:18.631 INFO:teuthology.orchestra.run.smithi154.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2021-07-31T01:31:18.631 INFO:teuthology.orchestra.run.smithi154.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2021-07-31T01:31:18.631 INFO:teuthology.orchestra.run.smithi154.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2021-07-31T01:31:18.631 INFO:teuthology.orchestra.run.smithi154.stdout:Installing weak dependencies: 2021-07-31T01:31:18.632 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cssselect noarch 0.9.2-10.el8 epel 40 k 2021-07-31T01:31:18.632 INFO:teuthology.orchestra.run.smithi154.stdout: python3-html5lib noarch 1:0.999999999-6.el8 rhel-8-for-x86_64-appstream-rpms 214 k 2021-07-31T01:31:18.632 INFO:teuthology.orchestra.run.smithi154.stdout: 2021-07-31T01:31:18.632 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2021-07-31T01:31:18.633 INFO:teuthology.orchestra.run.smithi154.stdout:====================================================================================================== 2021-07-31T01:31:18.633 INFO:teuthology.orchestra.run.smithi154.stdout:Install 37 Packages 2021-07-31T01:31:18.633 INFO:teuthology.orchestra.run.smithi154.stdout: 2021-07-31T01:31:18.634 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 33 M 2021-07-31T01:31:18.634 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 107 M 2021-07-31T01:31:18.634 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2021-07-31T01:31:18.745 INFO:teuthology.orchestra.run.smithi154.stdout:(1/37): ceph-16.2.5-150.g5a784c87.el8.x86_64.rp 53 kB/s | 6.4 kB 00:00 2021-07-31T01:31:19.064 INFO:teuthology.orchestra.run.smithi154.stdout:(2/37): ceph-mds-16.2.5-150.g5a784c87.el8.x86_6 5.4 MB/s | 2.4 MB 00:00 2021-07-31T01:31:19.072 INFO:teuthology.orchestra.run.smithi154.stdout:(3/37): ceph-mgr-16.2.5-150.g5a784c87.el8.x86_6 3.5 MB/s | 1.6 MB 00:00 2021-07-31T01:31:19.133 INFO:teuthology.orchestra.run.smithi154.stdout:(4/37): ceph-mgr-modules-core-16.2.5-150.g5a784 3.6 MB/s | 224 kB 00:00 2021-07-31T01:31:19.222 INFO:teuthology.orchestra.run.smithi154.stdout:(5/37): python3-bcrypt-3.1.6-2.el8.1.x86_64.rpm 511 kB/s | 44 kB 00:00 2021-07-31T01:31:19.282 INFO:teuthology.orchestra.run.smithi154.stdout:(6/37): python3-beautifulsoup4-4.6.3-2.el8.1.no 2.8 MB/s | 185 kB 00:00 2021-07-31T01:31:19.305 INFO:teuthology.orchestra.run.smithi154.stdout:(7/37): python3-cheroot-8.5.2-1.el8.noarch.rpm 7.4 MB/s | 173 kB 00:00 2021-07-31T01:31:19.349 INFO:teuthology.orchestra.run.smithi154.stdout:(8/37): python3-cherrypy-18.4.0-1.el8.noarch.rp 8.5 MB/s | 384 kB 00:00 2021-07-31T01:31:19.372 INFO:teuthology.orchestra.run.smithi154.stdout:(9/37): python3-cssselect-0.9.2-10.el8.noarch.r 1.8 MB/s | 40 kB 00:00 2021-07-31T01:31:19.393 INFO:teuthology.orchestra.run.smithi154.stdout:(10/37): python3-jaraco-6.2-6.el8.noarch.rpm 521 kB/s | 11 kB 00:00 2021-07-31T01:31:19.415 INFO:teuthology.orchestra.run.smithi154.stdout:(11/37): python3-jaraco-functools-2.0-4.el8.noa 818 kB/s | 18 kB 00:00 2021-07-31T01:31:19.437 INFO:teuthology.orchestra.run.smithi154.stdout:(12/37): python3-logutils-0.3.5-11.el8.noarch.r 2.2 MB/s | 49 kB 00:00 2021-07-31T01:31:19.459 INFO:teuthology.orchestra.run.smithi154.stdout:(13/37): python3-more-itertools-7.2.0-3.el8.noa 2.6 MB/s | 59 kB 00:00 2021-07-31T01:31:19.483 INFO:teuthology.orchestra.run.smithi154.stdout:(14/37): python3-pecan-1.3.2-9.el8.noarch.rpm 12 MB/s | 283 kB 00:00 2021-07-31T01:31:19.504 INFO:teuthology.orchestra.run.smithi154.stdout:(15/37): python3-portend-2.6-1.el8.noarch.rpm 718 kB/s | 16 kB 00:00 2021-07-31T01:31:19.526 INFO:teuthology.orchestra.run.smithi154.stdout:(16/37): python3-simplegeneric-0.8.1-17.el8.noa 860 kB/s | 19 kB 00:00 2021-07-31T01:31:19.548 INFO:teuthology.orchestra.run.smithi154.stdout:(17/37): python3-singledispatch-3.4.0.3-18.el8. 1.1 MB/s | 24 kB 00:00 2021-07-31T01:31:19.570 INFO:teuthology.orchestra.run.smithi154.stdout:(18/37): python3-tempora-1.14.1-5.el8.noarch.rp 1.3 MB/s | 29 kB 00:00 2021-07-31T01:31:19.592 INFO:teuthology.orchestra.run.smithi154.stdout:(19/37): python3-trustme-0.6.0-4.el8.noarch.rpm 1.2 MB/s | 27 kB 00:00 2021-07-31T01:31:19.615 INFO:teuthology.orchestra.run.smithi154.stdout:(20/37): python3-waitress-1.2.1-2.el8.1.noarch. 10 MB/s | 241 kB 00:00 2021-07-31T01:31:19.639 INFO:teuthology.orchestra.run.smithi154.stdout:(21/37): python3-webob-1.8.5-1.el8.1.noarch.rpm 11 MB/s | 251 kB 00:00 2021-07-31T01:31:19.661 INFO:teuthology.orchestra.run.smithi154.stdout:(22/37): python3-webtest-2.0.33-1.el8.noarch.rp 3.8 MB/s | 85 kB 00:00 2021-07-31T01:31:19.683 INFO:teuthology.orchestra.run.smithi154.stdout:(23/37): python3-zc-lockfile-2.0-2.el8.noarch.r 1.0 MB/s | 23 kB 00:00 2021-07-31T01:31:19.732 INFO:teuthology.orchestra.run.smithi154.stdout:(24/37): python3-lxml-4.2.3-1.el8.x86_64.rpm 31 MB/s | 1.5 MB 00:00 2021-07-31T01:31:19.773 INFO:teuthology.orchestra.run.smithi154.stdout:(25/37): python3-webencodings-0.5.1-6.el8.noarc 677 kB/s | 27 kB 00:00 2021-07-31T01:31:19.805 INFO:teuthology.orchestra.run.smithi154.stdout:(26/37): python3-pyOpenSSL-18.0.0-1.el8.noarch. 3.1 MB/s | 103 kB 00:00 2021-07-31T01:31:19.828 INFO:teuthology.orchestra.run.smithi154.stdout:(27/37): python3-werkzeug-0.12.2-4.el8.noarch.r 19 MB/s | 457 kB 00:00 2021-07-31T01:31:19.851 INFO:teuthology.orchestra.run.smithi154.stdout:(28/37): python3-markupsafe-0.23-19.el8.x86_64. 1.7 MB/s | 39 kB 00:00 2021-07-31T01:31:19.871 INFO:teuthology.orchestra.run.smithi154.stdout:(29/37): python3-pytz-2017.2-9.el8.noarch.rpm 2.7 MB/s | 54 kB 00:00 2021-07-31T01:31:19.891 INFO:teuthology.orchestra.run.smithi154.stdout:(30/37): python3-mako-1.0.6-13.el8.noarch.rpm 7.6 MB/s | 157 kB 00:00 2021-07-31T01:31:19.913 INFO:teuthology.orchestra.run.smithi154.stdout:(31/37): python3-html5lib-0.999999999-6.el8.noa 9.5 MB/s | 214 kB 00:00 2021-07-31T01:31:19.933 INFO:teuthology.orchestra.run.smithi154.stdout:(32/37): python3-pycparser-2.14-14.el8.noarch.r 5.3 MB/s | 109 kB 00:00 2021-07-31T01:31:19.954 INFO:teuthology.orchestra.run.smithi154.stdout:(33/37): python3-asn1crypto-0.24.0-3.el8.noarch 8.6 MB/s | 181 kB 00:00 2021-07-31T01:31:19.975 INFO:teuthology.orchestra.run.smithi154.stdout:(34/37): python3-cffi-1.11.5-5.el8.x86_64.rpm 11 MB/s | 238 kB 00:00 2021-07-31T01:31:19.999 INFO:teuthology.orchestra.run.smithi154.stdout:(35/37): python3-cryptography-2.3-3.el8.x86_64. 21 MB/s | 511 kB 00:00 2021-07-31T01:31:20.376 INFO:teuthology.orchestra.run.smithi154.stdout:(36/37): ceph-osd-16.2.5-150.g5a784c87.el8.x86_ 14 MB/s | 18 MB 00:01 2021-07-31T01:31:20.531 INFO:teuthology.orchestra.run.smithi154.stdout:(37/37): ceph-mon-16.2.5-150.g5a784c87.el8.x86_ 2.8 MB/s | 5.0 MB 00:01 2021-07-31T01:31:20.535 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2021-07-31T01:31:20.535 INFO:teuthology.orchestra.run.smithi154.stdout:Total 17 MB/s | 33 MB 00:01 2021-07-31T01:31:20.546 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2021-07-31T01:31:20.585 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2021-07-31T01:31:20.586 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2021-07-31T01:31:20.851 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2021-07-31T01:31:20.855 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2021-07-31T01:31:21.137 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:04 ago on Sat 31 Jul 2021 01:29:23 AM UTC. 2021-07-31T01:31:22.202 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2021-07-31T01:31:22.410 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/37 2021-07-31T01:31:22.628 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/37 2021-07-31T01:31:22.823 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-asn1crypto-0.24.0-3.el8.noarch 3/37 2021-07-31T01:31:23.055 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 4/37 2021-07-31T01:31:23.380 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2021-07-31T01:31:23.384 INFO:teuthology.orchestra.run.smithi045.stdout:====================================================================================================== 2021-07-31T01:31:23.384 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2021-07-31T01:31:23.384 INFO:teuthology.orchestra.run.smithi045.stdout:====================================================================================================== 2021-07-31T01:31:23.384 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2021-07-31T01:31:23.385 INFO:teuthology.orchestra.run.smithi045.stdout: ceph x86_64 2:16.2.5-150.g5a784c87.el8 ceph 6.4 k 2021-07-31T01:31:23.385 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2021-07-31T01:31:23.385 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mds x86_64 2:16.2.5-150.g5a784c87.el8 ceph 2.4 M 2021-07-31T01:31:23.385 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr x86_64 2:16.2.5-150.g5a784c87.el8 ceph 1.6 M 2021-07-31T01:31:23.385 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-modules-core noarch 2:16.2.5-150.g5a784c87.el8 ceph-noarch 224 k 2021-07-31T01:31:23.386 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mon x86_64 2:16.2.5-150.g5a784c87.el8 ceph 5.0 M 2021-07-31T01:31:23.386 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-osd x86_64 2:16.2.5-150.g5a784c87.el8 ceph 18 M 2021-07-31T01:31:23.386 INFO:teuthology.orchestra.run.smithi045.stdout: python3-asn1crypto noarch 0.24.0-3.el8 rhel-8-for-x86_64-baseos-rpms 181 k 2021-07-31T01:31:23.386 INFO:teuthology.orchestra.run.smithi045.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2021-07-31T01:31:23.387 INFO:teuthology.orchestra.run.smithi045.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2021-07-31T01:31:23.387 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2021-07-31T01:31:23.387 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2021-07-31T01:31:23.387 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2021-07-31T01:31:23.387 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cryptography x86_64 2.3-3.el8 rhel-8-for-x86_64-baseos-rpms 511 k 2021-07-31T01:31:23.388 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2021-07-31T01:31:23.388 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2021-07-31T01:31:23.388 INFO:teuthology.orchestra.run.smithi045.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2021-07-31T01:31:23.388 INFO:teuthology.orchestra.run.smithi045.stdout: python3-lxml x86_64 4.2.3-1.el8 rhel-8-for-x86_64-appstream-rpms 1.5 M 2021-07-31T01:31:23.388 INFO:teuthology.orchestra.run.smithi045.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2021-07-31T01:31:23.389 INFO:teuthology.orchestra.run.smithi045.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2021-07-31T01:31:23.389 INFO:teuthology.orchestra.run.smithi045.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2021-07-31T01:31:23.389 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2021-07-31T01:31:23.389 INFO:teuthology.orchestra.run.smithi045.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2021-07-31T01:31:23.390 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyOpenSSL noarch 18.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2021-07-31T01:31:23.390 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2021-07-31T01:31:23.390 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2021-07-31T01:31:23.390 INFO:teuthology.orchestra.run.smithi045.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2021-07-31T01:31:23.390 INFO:teuthology.orchestra.run.smithi045.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2021-07-31T01:31:23.391 INFO:teuthology.orchestra.run.smithi045.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2021-07-31T01:31:23.391 INFO:teuthology.orchestra.run.smithi045.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2021-07-31T01:31:23.391 INFO:teuthology.orchestra.run.smithi045.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2021-07-31T01:31:23.391 INFO:teuthology.orchestra.run.smithi045.stdout: python3-webencodings noarch 0.5.1-6.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2021-07-31T01:31:23.392 INFO:teuthology.orchestra.run.smithi045.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2021-07-31T01:31:23.392 INFO:teuthology.orchestra.run.smithi045.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2021-07-31T01:31:23.392 INFO:teuthology.orchestra.run.smithi045.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2021-07-31T01:31:23.392 INFO:teuthology.orchestra.run.smithi045.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2021-07-31T01:31:23.392 INFO:teuthology.orchestra.run.smithi045.stdout:Installing weak dependencies: 2021-07-31T01:31:23.393 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cssselect noarch 0.9.2-10.el8 epel 40 k 2021-07-31T01:31:23.393 INFO:teuthology.orchestra.run.smithi045.stdout: python3-html5lib noarch 1:0.999999999-6.el8 rhel-8-for-x86_64-appstream-rpms 214 k 2021-07-31T01:31:23.393 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:31:23.393 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2021-07-31T01:31:23.394 INFO:teuthology.orchestra.run.smithi045.stdout:====================================================================================================== 2021-07-31T01:31:23.394 INFO:teuthology.orchestra.run.smithi045.stdout:Install 37 Packages 2021-07-31T01:31:23.394 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:31:23.395 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 33 M 2021-07-31T01:31:23.395 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 107 M 2021-07-31T01:31:23.395 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2021-07-31T01:31:23.438 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 5/37 2021-07-31T01:31:23.490 INFO:teuthology.orchestra.run.smithi045.stdout:(1/37): ceph-16.2.5-150.g5a784c87.el8.x86_64.rp 64 kB/s | 6.4 kB 00:00 2021-07-31T01:31:23.696 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cryptography-2.3-3.el8.x86_64 6/37 2021-07-31T01:31:23.875 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pyOpenSSL-18.0.0-1.el8.noarch 7/37 2021-07-31T01:31:23.954 INFO:teuthology.orchestra.run.smithi045.stdout:(2/37): ceph-mgr-16.2.5-150.g5a784c87.el8.x86_6 2.8 MB/s | 1.6 MB 00:00 2021-07-31T01:31:24.027 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 8/37 2021-07-31T01:31:24.124 INFO:teuthology.orchestra.run.smithi045.stdout:(3/37): ceph-mds-16.2.5-150.g5a784c87.el8.x86_6 3.3 MB/s | 2.4 MB 00:00 2021-07-31T01:31:24.196 INFO:teuthology.orchestra.run.smithi045.stdout:(4/37): ceph-mgr-modules-core-16.2.5-150.g5a784 3.0 MB/s | 224 kB 00:00 2021-07-31T01:31:24.249 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 9/37 2021-07-31T01:31:24.300 INFO:teuthology.orchestra.run.smithi045.stdout:(5/37): python3-bcrypt-3.1.6-2.el8.1.x86_64.rpm 419 kB/s | 44 kB 00:00 2021-07-31T01:31:24.378 INFO:teuthology.orchestra.run.smithi045.stdout:(6/37): python3-beautifulsoup4-4.6.3-2.el8.1.no 2.3 MB/s | 185 kB 00:00 2021-07-31T01:31:24.407 INFO:teuthology.orchestra.run.smithi045.stdout:(7/37): python3-cheroot-8.5.2-1.el8.noarch.rpm 6.0 MB/s | 173 kB 00:00 2021-07-31T01:31:24.413 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 10/37 2021-07-31T01:31:24.460 INFO:teuthology.orchestra.run.smithi045.stdout:(8/37): python3-cherrypy-18.4.0-1.el8.noarch.rp 7.0 MB/s | 384 kB 00:00 2021-07-31T01:31:24.487 INFO:teuthology.orchestra.run.smithi045.stdout:(9/37): python3-cssselect-0.9.2-10.el8.noarch.r 1.5 MB/s | 40 kB 00:00 2021-07-31T01:31:24.514 INFO:teuthology.orchestra.run.smithi045.stdout:(10/37): python3-jaraco-6.2-6.el8.noarch.rpm 423 kB/s | 11 kB 00:00 2021-07-31T01:31:24.540 INFO:teuthology.orchestra.run.smithi045.stdout:(11/37): python3-jaraco-functools-2.0-4.el8.noa 662 kB/s | 18 kB 00:00 2021-07-31T01:31:24.564 INFO:teuthology.orchestra.run.smithi045.stdout:(12/37): ceph-mon-16.2.5-150.g5a784c87.el8.x86_ 4.6 MB/s | 5.0 MB 00:01 2021-07-31T01:31:24.567 INFO:teuthology.orchestra.run.smithi045.stdout:(13/37): python3-logutils-0.3.5-11.el8.noarch.r 1.8 MB/s | 49 kB 00:00 2021-07-31T01:31:24.620 INFO:teuthology.orchestra.run.smithi045.stdout:(14/37): python3-pecan-1.3.2-9.el8.noarch.rpm 5.3 MB/s | 283 kB 00:00 2021-07-31T01:31:24.647 INFO:teuthology.orchestra.run.smithi045.stdout:(15/37): python3-portend-2.6-1.el8.noarch.rpm 593 kB/s | 16 kB 00:00 2021-07-31T01:31:24.649 INFO:teuthology.orchestra.run.smithi045.stdout:(16/37): python3-more-itertools-7.2.0-3.el8.noa 689 kB/s | 59 kB 00:00 2021-07-31T01:31:24.657 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 11/37 2021-07-31T01:31:24.672 INFO:teuthology.orchestra.run.smithi045.stdout:(17/37): python3-singledispatch-3.4.0.3-18.el8. 1.1 MB/s | 24 kB 00:00 2021-07-31T01:31:24.673 INFO:teuthology.orchestra.run.smithi045.stdout:(18/37): python3-simplegeneric-0.8.1-17.el8.noa 717 kB/s | 19 kB 00:00 2021-07-31T01:31:24.693 INFO:teuthology.orchestra.run.smithi045.stdout:(19/37): python3-tempora-1.14.1-5.el8.noarch.rp 1.3 MB/s | 29 kB 00:00 2021-07-31T01:31:24.699 INFO:teuthology.orchestra.run.smithi045.stdout:(20/37): python3-trustme-0.6.0-4.el8.noarch.rpm 1.0 MB/s | 27 kB 00:00 2021-07-31T01:31:24.728 INFO:teuthology.orchestra.run.smithi045.stdout:(21/37): python3-webob-1.8.5-1.el8.1.noarch.rpm 8.5 MB/s | 251 kB 00:00 2021-07-31T01:31:24.755 INFO:teuthology.orchestra.run.smithi045.stdout:(22/37): python3-webtest-2.0.33-1.el8.noarch.rp 3.1 MB/s | 85 kB 00:00 2021-07-31T01:31:24.757 INFO:teuthology.orchestra.run.smithi045.stdout:(23/37): python3-waitress-1.2.1-2.el8.1.noarch. 3.7 MB/s | 241 kB 00:00 2021-07-31T01:31:24.788 INFO:teuthology.orchestra.run.smithi045.stdout:(24/37): python3-zc-lockfile-2.0-2.el8.noarch.r 695 kB/s | 23 kB 00:00 2021-07-31T01:31:24.823 INFO:teuthology.orchestra.run.smithi045.stdout:(25/37): python3-lxml-4.2.3-1.el8.x86_64.rpm 23 MB/s | 1.5 MB 00:00 2021-07-31T01:31:24.825 INFO:teuthology.orchestra.run.smithi045.stdout:(26/37): python3-webencodings-0.5.1-6.el8.noarc 747 kB/s | 27 kB 00:00 2021-07-31T01:31:24.858 INFO:teuthology.orchestra.run.smithi045.stdout:(27/37): python3-pyOpenSSL-18.0.0-1.el8.noarch. 2.9 MB/s | 103 kB 00:00 2021-07-31T01:31:24.872 INFO:teuthology.orchestra.run.smithi045.stdout:(28/37): python3-werkzeug-0.12.2-4.el8.noarch.r 9.5 MB/s | 457 kB 00:00 2021-07-31T01:31:24.888 INFO:teuthology.orchestra.run.smithi045.stdout:(29/37): python3-markupsafe-0.23-19.el8.x86_64. 1.3 MB/s | 39 kB 00:00 2021-07-31T01:31:24.906 INFO:teuthology.orchestra.run.smithi045.stdout:(30/37): python3-pytz-2017.2-9.el8.noarch.rpm 2.1 MB/s | 54 kB 00:00 2021-07-31T01:31:24.921 INFO:teuthology.orchestra.run.smithi045.stdout:(31/37): python3-mako-1.0.6-13.el8.noarch.rpm 4.7 MB/s | 157 kB 00:00 2021-07-31T01:31:24.938 INFO:teuthology.orchestra.run.smithi045.stdout:(32/37): python3-html5lib-0.999999999-6.el8.noa 6.2 MB/s | 214 kB 00:00 2021-07-31T01:31:24.954 INFO:teuthology.orchestra.run.smithi045.stdout:(33/37): python3-pycparser-2.14-14.el8.noarch.r 3.2 MB/s | 109 kB 00:00 2021-07-31T01:31:24.968 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 12/37 2021-07-31T01:31:24.972 INFO:teuthology.orchestra.run.smithi045.stdout:(34/37): python3-asn1crypto-0.24.0-3.el8.noarch 5.3 MB/s | 181 kB 00:00 2021-07-31T01:31:24.989 INFO:teuthology.orchestra.run.smithi045.stdout:(35/37): python3-cffi-1.11.5-5.el8.x86_64.rpm 6.9 MB/s | 238 kB 00:00 2021-07-31T01:31:25.007 INFO:teuthology.orchestra.run.smithi045.stdout:(36/37): python3-cryptography-2.3-3.el8.x86_64. 14 MB/s | 511 kB 00:00 2021-07-31T01:31:25.202 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 13/37 2021-07-31T01:31:25.851 INFO:teuthology.orchestra.run.smithi045.stdout:(37/37): ceph-osd-16.2.5-150.g5a784c87.el8.x86_ 9.5 MB/s | 18 MB 00:01 2021-07-31T01:31:25.854 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2021-07-31T01:31:25.854 INFO:teuthology.orchestra.run.smithi045.stdout:Total 13 MB/s | 33 MB 00:02 2021-07-31T01:31:25.868 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2021-07-31T01:31:25.911 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2021-07-31T01:31:25.912 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2021-07-31T01:31:26.049 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-webencodings-0.5.1-6.el8.noarch 14/37 2021-07-31T01:31:26.363 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2021-07-31T01:31:26.375 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2021-07-31T01:31:27.228 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-html5lib-1:0.999999999-6.el8.noarch 15/37 2021-07-31T01:31:27.463 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 16/37 2021-07-31T01:31:27.682 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 17/37 2021-07-31T01:31:27.863 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 18/37 2021-07-31T01:31:28.032 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 19/37 2021-07-31T01:31:28.232 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 20/37 2021-07-31T01:31:28.372 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 21/37 2021-07-31T01:31:28.613 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 22/37 2021-07-31T01:31:28.855 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 23/37 2021-07-31T01:31:29.023 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 24/37 2021-07-31T01:31:29.076 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2021-07-31T01:31:29.318 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/37 2021-07-31T01:31:29.327 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-portend-2.6-1.el8.noarch 25/37 2021-07-31T01:31:29.577 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/37 2021-07-31T01:31:29.587 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 26/37 2021-07-31T01:31:29.796 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-asn1crypto-0.24.0-3.el8.noarch 3/37 2021-07-31T01:31:29.915 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cssselect-0.9.2-10.el8.noarch 27/37 2021-07-31T01:31:30.021 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 4/37 2021-07-31T01:31:30.148 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-lxml-4.2.3-1.el8.x86_64 28/37 2021-07-31T01:31:30.400 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 29/37 2021-07-31T01:31:30.413 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 5/37 2021-07-31T01:31:30.702 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cryptography-2.3-3.el8.x86_64 6/37 2021-07-31T01:31:30.720 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 30/37 2021-07-31T01:31:30.863 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pyOpenSSL-18.0.0-1.el8.noarch 7/37 2021-07-31T01:31:31.057 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 8/37 2021-07-31T01:31:31.076 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 31/37 2021-07-31T01:31:31.257 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 9/37 2021-07-31T01:31:31.475 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mgr-modules-core-2:16.2.5-150.g5a784c87.el8.n 32/37 2021-07-31T01:31:31.488 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 10/37 2021-07-31T01:31:31.667 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mgr-2:16.2.5-150.g5a784c87.el8.x86_64 33/37 2021-07-31T01:31:31.751 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 11/37 2021-07-31T01:31:32.140 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 12/37 2021-07-31T01:31:32.326 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 13/37 2021-07-31T01:31:32.597 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-webencodings-0.5.1-6.el8.noarch 14/37 2021-07-31T01:31:32.797 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-html5lib-1:0.999999999-6.el8.noarch 15/37 2021-07-31T01:31:33.049 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 16/37 2021-07-31T01:31:33.159 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mgr-2:16.2.5-150.g5a784c87.el8.x86_64 33/37 2021-07-31T01:31:33.271 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 17/37 2021-07-31T01:31:33.342 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-osd-2:16.2.5-150.g5a784c87.el8.x86_64 34/37 2021-07-31T01:31:33.451 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 18/37 2021-07-31T01:31:33.647 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 19/37 2021-07-31T01:31:33.768 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-osd-2:16.2.5-150.g5a784c87.el8.x86_64 34/37 2021-07-31T01:31:33.832 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 20/37 2021-07-31T01:31:33.953 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 21/37 2021-07-31T01:31:33.958 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mon-2:16.2.5-150.g5a784c87.el8.x86_64 35/37 2021-07-31T01:31:34.173 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mon-2:16.2.5-150.g5a784c87.el8.x86_64 35/37 2021-07-31T01:31:34.180 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 22/37 2021-07-31T01:31:34.322 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mds-2:16.2.5-150.g5a784c87.el8.x86_64 36/37 2021-07-31T01:31:34.341 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mds-2:16.2.5-150.g5a784c87.el8.x86_64 36/37 2021-07-31T01:31:34.410 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 23/37 2021-07-31T01:31:34.462 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-2:16.2.5-150.g5a784c87.el8.x86_64 37/37 2021-07-31T01:31:34.562 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 24/37 2021-07-31T01:31:34.821 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-portend-2.6-1.el8.noarch 25/37 2021-07-31T01:31:34.851 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-2:16.2.5-150.g5a784c87.el8.x86_64 37/37 2021-07-31T01:31:34.852 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-2:16.2.5-150.g5a784c87.el8.x86_64 1/37 2021-07-31T01:31:34.852 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mds-2:16.2.5-150.g5a784c87.el8.x86_64 2/37 2021-07-31T01:31:34.852 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mgr-2:16.2.5-150.g5a784c87.el8.x86_64 3/37 2021-07-31T01:31:34.852 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mon-2:16.2.5-150.g5a784c87.el8.x86_64 4/37 2021-07-31T01:31:34.852 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-osd-2:16.2.5-150.g5a784c87.el8.x86_64 5/37 2021-07-31T01:31:34.853 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mgr-modules-core-2:16.2.5-150.g5a784c87.el8.n 6/37 2021-07-31T01:31:34.854 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 7/37 2021-07-31T01:31:34.854 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 8/37 2021-07-31T01:31:34.854 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 9/37 2021-07-31T01:31:34.855 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 10/37 2021-07-31T01:31:34.855 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cssselect-0.9.2-10.el8.noarch 11/37 2021-07-31T01:31:34.855 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 12/37 2021-07-31T01:31:34.855 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 13/37 2021-07-31T01:31:34.855 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 14/37 2021-07-31T01:31:34.856 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 15/37 2021-07-31T01:31:34.856 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 16/37 2021-07-31T01:31:34.856 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-portend-2.6-1.el8.noarch 17/37 2021-07-31T01:31:34.856 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 18/37 2021-07-31T01:31:34.857 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 19/37 2021-07-31T01:31:34.857 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 20/37 2021-07-31T01:31:34.857 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 21/37 2021-07-31T01:31:34.857 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 22/37 2021-07-31T01:31:34.857 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 23/37 2021-07-31T01:31:34.858 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 24/37 2021-07-31T01:31:34.858 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 25/37 2021-07-31T01:31:34.858 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-lxml-4.2.3-1.el8.x86_64 26/37 2021-07-31T01:31:34.858 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-webencodings-0.5.1-6.el8.noarch 27/37 2021-07-31T01:31:34.858 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pyOpenSSL-18.0.0-1.el8.noarch 28/37 2021-07-31T01:31:34.859 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 29/37 2021-07-31T01:31:34.859 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 30/37 2021-07-31T01:31:34.859 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 31/37 2021-07-31T01:31:34.859 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 32/37 2021-07-31T01:31:34.860 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-html5lib-1:0.999999999-6.el8.noarch 33/37 2021-07-31T01:31:34.860 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 34/37 2021-07-31T01:31:34.861 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-asn1crypto-0.24.0-3.el8.noarch 35/37 2021-07-31T01:31:34.861 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 36/37 2021-07-31T01:31:35.068 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 26/37 2021-07-31T01:31:35.388 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cssselect-0.9.2-10.el8.noarch 27/37 2021-07-31T01:31:35.642 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-lxml-4.2.3-1.el8.x86_64 28/37 2021-07-31T01:31:35.760 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cryptography-2.3-3.el8.x86_64 37/37 2021-07-31T01:31:35.761 INFO:teuthology.orchestra.run.smithi154.stdout:Installed products updated. 2021-07-31T01:31:35.916 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 29/37 2021-07-31T01:31:36.226 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 30/37 2021-07-31T01:31:36.561 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 31/37 2021-07-31T01:31:36.949 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-modules-core-2:16.2.5-150.g5a784c87.el8.n 32/37 2021-07-31T01:31:37.142 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-2:16.2.5-150.g5a784c87.el8.x86_64 33/37 2021-07-31T01:31:38.730 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mgr-2:16.2.5-150.g5a784c87.el8.x86_64 33/37 2021-07-31T01:31:38.912 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-osd-2:16.2.5-150.g5a784c87.el8.x86_64 34/37 2021-07-31T01:31:39.365 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-osd-2:16.2.5-150.g5a784c87.el8.x86_64 34/37 2021-07-31T01:31:39.575 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mon-2:16.2.5-150.g5a784c87.el8.x86_64 35/37 2021-07-31T01:31:39.805 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mon-2:16.2.5-150.g5a784c87.el8.x86_64 35/37 2021-07-31T01:31:39.995 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mds-2:16.2.5-150.g5a784c87.el8.x86_64 36/37 2021-07-31T01:31:40.018 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mds-2:16.2.5-150.g5a784c87.el8.x86_64 36/37 2021-07-31T01:31:40.126 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-2:16.2.5-150.g5a784c87.el8.x86_64 37/37 2021-07-31T01:31:40.550 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-2:16.2.5-150.g5a784c87.el8.x86_64 37/37 2021-07-31T01:31:40.550 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-2:16.2.5-150.g5a784c87.el8.x86_64 1/37 2021-07-31T01:31:40.550 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mds-2:16.2.5-150.g5a784c87.el8.x86_64 2/37 2021-07-31T01:31:40.551 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-2:16.2.5-150.g5a784c87.el8.x86_64 3/37 2021-07-31T01:31:40.551 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mon-2:16.2.5-150.g5a784c87.el8.x86_64 4/37 2021-07-31T01:31:40.551 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-osd-2:16.2.5-150.g5a784c87.el8.x86_64 5/37 2021-07-31T01:31:40.551 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-modules-core-2:16.2.5-150.g5a784c87.el8.n 6/37 2021-07-31T01:31:40.552 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 7/37 2021-07-31T01:31:40.552 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 8/37 2021-07-31T01:31:40.553 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 9/37 2021-07-31T01:31:40.553 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 10/37 2021-07-31T01:31:40.553 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cssselect-0.9.2-10.el8.noarch 11/37 2021-07-31T01:31:40.554 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 12/37 2021-07-31T01:31:40.554 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 13/37 2021-07-31T01:31:40.554 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 14/37 2021-07-31T01:31:40.554 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 15/37 2021-07-31T01:31:40.554 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 16/37 2021-07-31T01:31:40.555 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-portend-2.6-1.el8.noarch 17/37 2021-07-31T01:31:40.555 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 18/37 2021-07-31T01:31:40.555 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 19/37 2021-07-31T01:31:40.555 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 20/37 2021-07-31T01:31:40.555 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 21/37 2021-07-31T01:31:40.556 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 22/37 2021-07-31T01:31:40.556 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 23/37 2021-07-31T01:31:40.556 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 24/37 2021-07-31T01:31:40.556 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 25/37 2021-07-31T01:31:40.557 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-lxml-4.2.3-1.el8.x86_64 26/37 2021-07-31T01:31:40.557 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-webencodings-0.5.1-6.el8.noarch 27/37 2021-07-31T01:31:40.557 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pyOpenSSL-18.0.0-1.el8.noarch 28/37 2021-07-31T01:31:40.557 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 29/37 2021-07-31T01:31:40.557 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 30/37 2021-07-31T01:31:40.558 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 31/37 2021-07-31T01:31:40.558 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 32/37 2021-07-31T01:31:40.559 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-html5lib-1:0.999999999-6.el8.noarch 33/37 2021-07-31T01:31:40.559 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 34/37 2021-07-31T01:31:40.559 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-asn1crypto-0.24.0-3.el8.noarch 35/37 2021-07-31T01:31:40.559 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 36/37 2021-07-31T01:31:41.541 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cryptography-2.3-3.el8.x86_64 37/37 2021-07-31T01:31:41.542 INFO:teuthology.orchestra.run.smithi045.stdout:Installed products updated. 2021-07-31T01:31:45.629 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:00:04 ago on Sat 31 Jul 2021 01:31:00 AM UTC. 2021-07-31T01:31:47.831 INFO:teuthology.orchestra.run.smithi154.stdout: 2021-07-31T01:31:47.832 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2021-07-31T01:31:47.832 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-2:16.2.5-150.g5a784c87.el8.x86_64 2021-07-31T01:31:47.832 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mds-2:16.2.5-150.g5a784c87.el8.x86_64 2021-07-31T01:31:47.833 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-2:16.2.5-150.g5a784c87.el8.x86_64 2021-07-31T01:31:47.833 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-modules-core-2:16.2.5-150.g5a784c87.el8.noarch 2021-07-31T01:31:47.833 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mon-2:16.2.5-150.g5a784c87.el8.x86_64 2021-07-31T01:31:47.833 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-osd-2:16.2.5-150.g5a784c87.el8.x86_64 2021-07-31T01:31:47.833 INFO:teuthology.orchestra.run.smithi154.stdout: python3-asn1crypto-0.24.0-3.el8.noarch 2021-07-31T01:31:47.834 INFO:teuthology.orchestra.run.smithi154.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2021-07-31T01:31:47.834 INFO:teuthology.orchestra.run.smithi154.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2021-07-31T01:31:47.834 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cffi-1.11.5-5.el8.x86_64 2021-07-31T01:31:47.834 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cheroot-8.5.2-1.el8.noarch 2021-07-31T01:31:47.835 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2021-07-31T01:31:47.835 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cryptography-2.3-3.el8.x86_64 2021-07-31T01:31:47.835 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cssselect-0.9.2-10.el8.noarch 2021-07-31T01:31:47.835 INFO:teuthology.orchestra.run.smithi154.stdout: python3-html5lib-1:0.999999999-6.el8.noarch 2021-07-31T01:31:47.835 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-6.2-6.el8.noarch 2021-07-31T01:31:47.836 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2021-07-31T01:31:47.836 INFO:teuthology.orchestra.run.smithi154.stdout: python3-logutils-0.3.5-11.el8.noarch 2021-07-31T01:31:47.836 INFO:teuthology.orchestra.run.smithi154.stdout: python3-lxml-4.2.3-1.el8.x86_64 2021-07-31T01:31:47.836 INFO:teuthology.orchestra.run.smithi154.stdout: python3-mako-1.0.6-13.el8.noarch 2021-07-31T01:31:47.837 INFO:teuthology.orchestra.run.smithi154.stdout: python3-markupsafe-0.23-19.el8.x86_64 2021-07-31T01:31:47.837 INFO:teuthology.orchestra.run.smithi154.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2021-07-31T01:31:47.837 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pecan-1.3.2-9.el8.noarch 2021-07-31T01:31:47.837 INFO:teuthology.orchestra.run.smithi154.stdout: python3-portend-2.6-1.el8.noarch 2021-07-31T01:31:47.837 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyOpenSSL-18.0.0-1.el8.noarch 2021-07-31T01:31:47.838 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pycparser-2.14-14.el8.noarch 2021-07-31T01:31:47.838 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pytz-2017.2-9.el8.noarch 2021-07-31T01:31:47.838 INFO:teuthology.orchestra.run.smithi154.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2021-07-31T01:31:47.838 INFO:teuthology.orchestra.run.smithi154.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2021-07-31T01:31:47.839 INFO:teuthology.orchestra.run.smithi154.stdout: python3-tempora-1.14.1-5.el8.noarch 2021-07-31T01:31:47.839 INFO:teuthology.orchestra.run.smithi154.stdout: python3-trustme-0.6.0-4.el8.noarch 2021-07-31T01:31:47.839 INFO:teuthology.orchestra.run.smithi154.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2021-07-31T01:31:47.839 INFO:teuthology.orchestra.run.smithi154.stdout: python3-webencodings-0.5.1-6.el8.noarch 2021-07-31T01:31:47.840 INFO:teuthology.orchestra.run.smithi154.stdout: python3-webob-1.8.5-1.el8.1.noarch 2021-07-31T01:31:47.840 INFO:teuthology.orchestra.run.smithi154.stdout: python3-webtest-2.0.33-1.el8.noarch 2021-07-31T01:31:47.840 INFO:teuthology.orchestra.run.smithi154.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2021-07-31T01:31:47.840 INFO:teuthology.orchestra.run.smithi154.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2021-07-31T01:31:47.840 INFO:teuthology.orchestra.run.smithi154.stdout: 2021-07-31T01:31:47.841 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2021-07-31T01:31:47.937 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-base 2021-07-31T01:31:48.221 INFO:teuthology.orchestra.run.smithi154.stdout:Updating Subscription Management repositories. 2021-07-31T01:31:48.740 INFO:teuthology.orchestra.run.smithi154.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 93 kB/s | 2.8 kB 00:00 2021-07-31T01:31:52.100 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:05 ago on Sat 31 Jul 2021 01:31:04 AM UTC. 2021-07-31T01:31:52.775 INFO:teuthology.orchestra.run.smithi154.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 81 kB/s | 2.4 kB 00:00 2021-07-31T01:31:54.466 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:31:54.467 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2021-07-31T01:31:54.467 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-2:16.2.5-150.g5a784c87.el8.x86_64 2021-07-31T01:31:54.467 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mds-2:16.2.5-150.g5a784c87.el8.x86_64 2021-07-31T01:31:54.467 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-2:16.2.5-150.g5a784c87.el8.x86_64 2021-07-31T01:31:54.468 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-modules-core-2:16.2.5-150.g5a784c87.el8.noarch 2021-07-31T01:31:54.468 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mon-2:16.2.5-150.g5a784c87.el8.x86_64 2021-07-31T01:31:54.468 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-osd-2:16.2.5-150.g5a784c87.el8.x86_64 2021-07-31T01:31:54.468 INFO:teuthology.orchestra.run.smithi045.stdout: python3-asn1crypto-0.24.0-3.el8.noarch 2021-07-31T01:31:54.469 INFO:teuthology.orchestra.run.smithi045.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2021-07-31T01:31:54.469 INFO:teuthology.orchestra.run.smithi045.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2021-07-31T01:31:54.469 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cffi-1.11.5-5.el8.x86_64 2021-07-31T01:31:54.469 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cheroot-8.5.2-1.el8.noarch 2021-07-31T01:31:54.469 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2021-07-31T01:31:54.470 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cryptography-2.3-3.el8.x86_64 2021-07-31T01:31:54.470 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cssselect-0.9.2-10.el8.noarch 2021-07-31T01:31:54.470 INFO:teuthology.orchestra.run.smithi045.stdout: python3-html5lib-1:0.999999999-6.el8.noarch 2021-07-31T01:31:54.470 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-6.2-6.el8.noarch 2021-07-31T01:31:54.471 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2021-07-31T01:31:54.471 INFO:teuthology.orchestra.run.smithi045.stdout: python3-logutils-0.3.5-11.el8.noarch 2021-07-31T01:31:54.471 INFO:teuthology.orchestra.run.smithi045.stdout: python3-lxml-4.2.3-1.el8.x86_64 2021-07-31T01:31:54.471 INFO:teuthology.orchestra.run.smithi045.stdout: python3-mako-1.0.6-13.el8.noarch 2021-07-31T01:31:54.472 INFO:teuthology.orchestra.run.smithi045.stdout: python3-markupsafe-0.23-19.el8.x86_64 2021-07-31T01:31:54.472 INFO:teuthology.orchestra.run.smithi045.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2021-07-31T01:31:54.472 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pecan-1.3.2-9.el8.noarch 2021-07-31T01:31:54.472 INFO:teuthology.orchestra.run.smithi045.stdout: python3-portend-2.6-1.el8.noarch 2021-07-31T01:31:54.472 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyOpenSSL-18.0.0-1.el8.noarch 2021-07-31T01:31:54.473 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pycparser-2.14-14.el8.noarch 2021-07-31T01:31:54.473 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pytz-2017.2-9.el8.noarch 2021-07-31T01:31:54.473 INFO:teuthology.orchestra.run.smithi045.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2021-07-31T01:31:54.473 INFO:teuthology.orchestra.run.smithi045.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2021-07-31T01:31:54.474 INFO:teuthology.orchestra.run.smithi045.stdout: python3-tempora-1.14.1-5.el8.noarch 2021-07-31T01:31:54.474 INFO:teuthology.orchestra.run.smithi045.stdout: python3-trustme-0.6.0-4.el8.noarch 2021-07-31T01:31:54.474 INFO:teuthology.orchestra.run.smithi045.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2021-07-31T01:31:54.474 INFO:teuthology.orchestra.run.smithi045.stdout: python3-webencodings-0.5.1-6.el8.noarch 2021-07-31T01:31:54.474 INFO:teuthology.orchestra.run.smithi045.stdout: python3-webob-1.8.5-1.el8.1.noarch 2021-07-31T01:31:54.475 INFO:teuthology.orchestra.run.smithi045.stdout: python3-webtest-2.0.33-1.el8.noarch 2021-07-31T01:31:54.475 INFO:teuthology.orchestra.run.smithi045.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2021-07-31T01:31:54.475 INFO:teuthology.orchestra.run.smithi045.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2021-07-31T01:31:54.475 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:31:54.475 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2021-07-31T01:31:54.729 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-base 2021-07-31T01:31:55.028 INFO:teuthology.orchestra.run.smithi045.stdout:Updating Subscription Management repositories. 2021-07-31T01:31:55.611 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 73 kB/s | 2.8 kB 00:00 2021-07-31T01:31:56.968 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:00:04 ago on Sat 31 Jul 2021 01:29:23 AM UTC. 2021-07-31T01:31:58.860 INFO:teuthology.orchestra.run.smithi154.stdout:Package ceph-base-2:16.2.5-150.g5a784c87.el8.x86_64 is already installed. 2021-07-31T01:31:59.204 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2021-07-31T01:31:59.205 INFO:teuthology.orchestra.run.smithi154.stdout:Nothing to do. 2021-07-31T01:31:59.206 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2021-07-31T01:31:59.274 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install cephadm 2021-07-31T01:31:59.556 INFO:teuthology.orchestra.run.smithi154.stdout:Updating Subscription Management repositories. 2021-07-31T01:31:59.925 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 63 kB/s | 2.4 kB 00:00 2021-07-31T01:32:00.064 INFO:teuthology.orchestra.run.smithi154.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 100 kB/s | 2.8 kB 00:00 2021-07-31T01:32:00.211 INFO:teuthology.orchestra.run.smithi154.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 84 kB/s | 2.4 kB 00:00 2021-07-31T01:32:01.706 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2021-07-31T01:32:01.707 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2021-07-31T01:32:01.707 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repository Size 2021-07-31T01:32:01.707 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2021-07-31T01:32:01.708 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2021-07-31T01:32:01.708 INFO:teuthology.orchestra.run.smithi154.stdout: cephadm noarch 2:16.2.5-150.g5a784c87.el8 ceph-noarch 73 k 2021-07-31T01:32:01.708 INFO:teuthology.orchestra.run.smithi154.stdout: 2021-07-31T01:32:01.708 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2021-07-31T01:32:01.708 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2021-07-31T01:32:01.709 INFO:teuthology.orchestra.run.smithi154.stdout:Install 1 Package 2021-07-31T01:32:01.709 INFO:teuthology.orchestra.run.smithi154.stdout: 2021-07-31T01:32:01.709 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 73 k 2021-07-31T01:32:01.709 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 296 k 2021-07-31T01:32:01.710 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2021-07-31T01:32:01.866 INFO:teuthology.orchestra.run.smithi154.stdout:cephadm-16.2.5-150.g5a784c87.el8.noarch.rpm 459 kB/s | 73 kB 00:00 2021-07-31T01:32:01.866 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2021-07-31T01:32:01.867 INFO:teuthology.orchestra.run.smithi154.stdout:Total 455 kB/s | 73 kB 00:00 2021-07-31T01:32:01.867 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2021-07-31T01:32:01.876 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2021-07-31T01:32:01.877 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2021-07-31T01:32:01.877 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2021-07-31T01:32:01.879 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2021-07-31T01:32:01.928 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2021-07-31T01:32:04.222 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: cephadm-2:16.2.5-150.g5a784c87.el8.noarch 1/1 2021-07-31T01:32:04.353 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:04 ago on Sat 31 Jul 2021 01:29:23 AM UTC. 2021-07-31T01:32:05.457 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : cephadm-2:16.2.5-150.g5a784c87.el8.noarch 1/1 2021-07-31T01:32:05.631 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: cephadm-2:16.2.5-150.g5a784c87.el8.noarch 1/1 2021-07-31T01:32:06.104 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : cephadm-2:16.2.5-150.g5a784c87.el8.noarch 1/1 2021-07-31T01:32:06.105 INFO:teuthology.orchestra.run.smithi154.stdout:Installed products updated. 2021-07-31T01:32:06.243 INFO:teuthology.orchestra.run.smithi045.stdout:Package ceph-base-2:16.2.5-150.g5a784c87.el8.x86_64 is already installed. 2021-07-31T01:32:06.618 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2021-07-31T01:32:06.619 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2021-07-31T01:32:06.619 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2021-07-31T01:32:06.701 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install cephadm 2021-07-31T01:32:06.997 INFO:teuthology.orchestra.run.smithi045.stdout:Updating Subscription Management repositories. 2021-07-31T01:32:07.553 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 74 kB/s | 2.8 kB 00:00 2021-07-31T01:32:07.715 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 65 kB/s | 2.4 kB 00:00 2021-07-31T01:32:09.539 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2021-07-31T01:32:09.540 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2021-07-31T01:32:09.540 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2021-07-31T01:32:09.541 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2021-07-31T01:32:09.541 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2021-07-31T01:32:09.541 INFO:teuthology.orchestra.run.smithi045.stdout: cephadm noarch 2:16.2.5-150.g5a784c87.el8 ceph-noarch 73 k 2021-07-31T01:32:09.541 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:32:09.542 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2021-07-31T01:32:09.542 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2021-07-31T01:32:09.542 INFO:teuthology.orchestra.run.smithi045.stdout:Install 1 Package 2021-07-31T01:32:09.542 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:32:09.543 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 73 k 2021-07-31T01:32:09.543 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 296 k 2021-07-31T01:32:09.543 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2021-07-31T01:32:09.706 INFO:teuthology.orchestra.run.smithi045.stdout:cephadm-16.2.5-150.g5a784c87.el8.noarch.rpm 440 kB/s | 73 kB 00:00 2021-07-31T01:32:09.707 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2021-07-31T01:32:09.707 INFO:teuthology.orchestra.run.smithi045.stdout:Total 436 kB/s | 73 kB 00:00 2021-07-31T01:32:09.708 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2021-07-31T01:32:09.715 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2021-07-31T01:32:09.715 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2021-07-31T01:32:09.717 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2021-07-31T01:32:09.721 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2021-07-31T01:32:10.030 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2021-07-31T01:32:12.411 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: cephadm-2:16.2.5-150.g5a784c87.el8.noarch 1/1 2021-07-31T01:32:12.641 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : cephadm-2:16.2.5-150.g5a784c87.el8.noarch 1/1 2021-07-31T01:32:12.817 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: cephadm-2:16.2.5-150.g5a784c87.el8.noarch 1/1 2021-07-31T01:32:13.333 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : cephadm-2:16.2.5-150.g5a784c87.el8.noarch 1/1 2021-07-31T01:32:13.333 INFO:teuthology.orchestra.run.smithi045.stdout:Installed products updated. 2021-07-31T01:32:16.348 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:00:05 ago on Sat 31 Jul 2021 01:31:40 AM UTC. 2021-07-31T01:32:18.550 INFO:teuthology.orchestra.run.smithi154.stdout: 2021-07-31T01:32:18.551 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2021-07-31T01:32:18.551 INFO:teuthology.orchestra.run.smithi154.stdout: cephadm-2:16.2.5-150.g5a784c87.el8.noarch 2021-07-31T01:32:18.551 INFO:teuthology.orchestra.run.smithi154.stdout: 2021-07-31T01:32:18.551 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2021-07-31T01:32:18.639 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-immutable-object-cache 2021-07-31T01:32:18.930 INFO:teuthology.orchestra.run.smithi154.stdout:Updating Subscription Management repositories. 2021-07-31T01:32:19.449 INFO:teuthology.orchestra.run.smithi154.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 88 kB/s | 2.8 kB 00:00 2021-07-31T01:32:23.396 INFO:teuthology.orchestra.run.smithi154.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 78 kB/s | 2.4 kB 00:00 2021-07-31T01:32:23.948 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:05 ago on Sat 31 Jul 2021 01:31:46 AM UTC. 2021-07-31T01:32:26.276 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:32:26.276 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2021-07-31T01:32:26.276 INFO:teuthology.orchestra.run.smithi045.stdout: cephadm-2:16.2.5-150.g5a784c87.el8.noarch 2021-07-31T01:32:26.276 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:32:26.277 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2021-07-31T01:32:26.393 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-immutable-object-cache 2021-07-31T01:32:26.704 INFO:teuthology.orchestra.run.smithi045.stdout:Updating Subscription Management repositories. 2021-07-31T01:32:27.280 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 74 kB/s | 2.8 kB 00:00 2021-07-31T01:32:27.595 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:00:04 ago on Sat 31 Jul 2021 01:29:23 AM UTC. 2021-07-31T01:32:29.735 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2021-07-31T01:32:29.736 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2021-07-31T01:32:29.737 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repo Size 2021-07-31T01:32:29.737 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2021-07-31T01:32:29.738 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2021-07-31T01:32:29.738 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-immutable-object-cache x86_64 2:16.2.5-150.g5a784c87.el8 ceph 161 k 2021-07-31T01:32:29.738 INFO:teuthology.orchestra.run.smithi154.stdout: 2021-07-31T01:32:29.739 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2021-07-31T01:32:29.739 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2021-07-31T01:32:29.739 INFO:teuthology.orchestra.run.smithi154.stdout:Install 1 Package 2021-07-31T01:32:29.740 INFO:teuthology.orchestra.run.smithi154.stdout: 2021-07-31T01:32:29.740 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 161 k 2021-07-31T01:32:29.741 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 471 k 2021-07-31T01:32:29.741 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2021-07-31T01:32:29.907 INFO:teuthology.orchestra.run.smithi154.stdout:ceph-immutable-object-cache-16.2.5-150.g5a784c8 947 kB/s | 161 kB 00:00 2021-07-31T01:32:29.908 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2021-07-31T01:32:29.909 INFO:teuthology.orchestra.run.smithi154.stdout:Total 939 kB/s | 161 kB 00:00 2021-07-31T01:32:29.910 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2021-07-31T01:32:29.920 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2021-07-31T01:32:29.921 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2021-07-31T01:32:29.972 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2021-07-31T01:32:29.976 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2021-07-31T01:32:30.091 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2021-07-31T01:32:30.239 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-immutable-object-cache-2:16.2.5-150.g5a784c87.e 1/1 2021-07-31T01:32:30.627 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.5-150.g5a784c87.e 1/1 2021-07-31T01:32:31.146 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-immutable-object-cache-2:16.2.5-150.g5a784c87.e 1/1 2021-07-31T01:32:31.150 INFO:teuthology.orchestra.run.smithi154.stdout:Installed products updated. 2021-07-31T01:32:31.653 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 63 kB/s | 2.4 kB 00:00 2021-07-31T01:32:36.272 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:04 ago on Sat 31 Jul 2021 01:29:23 AM UTC. 2021-07-31T01:32:38.532 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2021-07-31T01:32:38.533 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2021-07-31T01:32:38.533 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repo Size 2021-07-31T01:32:38.534 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2021-07-31T01:32:38.534 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2021-07-31T01:32:38.534 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-immutable-object-cache x86_64 2:16.2.5-150.g5a784c87.el8 ceph 161 k 2021-07-31T01:32:38.535 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:32:38.535 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2021-07-31T01:32:38.535 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2021-07-31T01:32:38.535 INFO:teuthology.orchestra.run.smithi045.stdout:Install 1 Package 2021-07-31T01:32:38.535 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:32:38.536 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 161 k 2021-07-31T01:32:38.536 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 471 k 2021-07-31T01:32:38.536 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2021-07-31T01:32:38.723 INFO:teuthology.orchestra.run.smithi045.stdout:ceph-immutable-object-cache-16.2.5-150.g5a784c8 856 kB/s | 161 kB 00:00 2021-07-31T01:32:38.724 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2021-07-31T01:32:38.724 INFO:teuthology.orchestra.run.smithi045.stdout:Total 850 kB/s | 161 kB 00:00 2021-07-31T01:32:38.724 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2021-07-31T01:32:38.738 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2021-07-31T01:32:38.739 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2021-07-31T01:32:38.794 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2021-07-31T01:32:38.798 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2021-07-31T01:32:38.949 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2021-07-31T01:32:39.136 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-immutable-object-cache-2:16.2.5-150.g5a784c87.e 1/1 2021-07-31T01:32:39.514 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.5-150.g5a784c87.e 1/1 2021-07-31T01:32:40.065 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-immutable-object-cache-2:16.2.5-150.g5a784c87.e 1/1 2021-07-31T01:32:40.065 INFO:teuthology.orchestra.run.smithi045.stdout:Installed products updated. 2021-07-31T01:32:41.195 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:00:05 ago on Sat 31 Jul 2021 01:32:11 AM UTC. 2021-07-31T01:32:43.379 INFO:teuthology.orchestra.run.smithi154.stdout: 2021-07-31T01:32:43.380 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2021-07-31T01:32:43.380 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-immutable-object-cache-2:16.2.5-150.g5a784c87.el8.x86_64 2021-07-31T01:32:43.380 INFO:teuthology.orchestra.run.smithi154.stdout: 2021-07-31T01:32:43.380 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2021-07-31T01:32:43.471 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-mgr 2021-07-31T01:32:43.755 INFO:teuthology.orchestra.run.smithi154.stdout:Updating Subscription Management repositories. 2021-07-31T01:32:44.265 INFO:teuthology.orchestra.run.smithi154.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 101 kB/s | 2.8 kB 00:00 2021-07-31T01:32:48.337 INFO:teuthology.orchestra.run.smithi154.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 81 kB/s | 2.4 kB 00:00 2021-07-31T01:32:50.983 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:05 ago on Sat 31 Jul 2021 01:32:18 AM UTC. 2021-07-31T01:32:52.561 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:00:04 ago on Sat 31 Jul 2021 01:29:23 AM UTC. 2021-07-31T01:32:53.343 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:32:53.343 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2021-07-31T01:32:53.343 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-immutable-object-cache-2:16.2.5-150.g5a784c87.el8.x86_64 2021-07-31T01:32:53.344 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:32:53.344 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2021-07-31T01:32:53.467 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-mgr 2021-07-31T01:32:53.765 INFO:teuthology.orchestra.run.smithi045.stdout:Updating Subscription Management repositories. 2021-07-31T01:32:54.320 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 73 kB/s | 2.8 kB 00:00 2021-07-31T01:32:54.350 INFO:teuthology.orchestra.run.smithi154.stdout:Package ceph-mgr-2:16.2.5-150.g5a784c87.el8.x86_64 is already installed. 2021-07-31T01:32:54.694 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2021-07-31T01:32:54.695 INFO:teuthology.orchestra.run.smithi154.stdout:Nothing to do. 2021-07-31T01:32:54.695 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2021-07-31T01:32:54.768 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-mgr-dashboard 2021-07-31T01:32:55.048 INFO:teuthology.orchestra.run.smithi154.stdout:Updating Subscription Management repositories. 2021-07-31T01:32:55.556 INFO:teuthology.orchestra.run.smithi154.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 97 kB/s | 2.8 kB 00:00 2021-07-31T01:32:55.703 INFO:teuthology.orchestra.run.smithi154.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 82 kB/s | 2.4 kB 00:00 2021-07-31T01:32:57.218 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2021-07-31T01:32:57.219 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================================== 2021-07-31T01:32:57.219 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repository Size 2021-07-31T01:32:57.220 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================================== 2021-07-31T01:32:57.220 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2021-07-31T01:32:57.220 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-dashboard noarch 2:16.2.5-150.g5a784c87.el8 ceph-noarch 2.7 M 2021-07-31T01:32:57.220 INFO:teuthology.orchestra.run.smithi154.stdout:Installing dependencies: 2021-07-31T01:32:57.221 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-grafana-dashboards noarch 2:16.2.5-150.g5a784c87.el8 ceph-noarch 25 k 2021-07-31T01:32:57.221 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-prometheus-alerts noarch 2:16.2.5-150.g5a784c87.el8 ceph-noarch 9.0 k 2021-07-31T01:32:57.221 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2021-07-31T01:32:57.221 INFO:teuthology.orchestra.run.smithi154.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2021-07-31T01:32:57.222 INFO:teuthology.orchestra.run.smithi154.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2021-07-31T01:32:57.222 INFO:teuthology.orchestra.run.smithi154.stdout: 2021-07-31T01:32:57.222 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2021-07-31T01:32:57.223 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================================== 2021-07-31T01:32:57.224 INFO:teuthology.orchestra.run.smithi154.stdout:Install 6 Packages 2021-07-31T01:32:57.224 INFO:teuthology.orchestra.run.smithi154.stdout: 2021-07-31T01:32:57.225 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 3.0 M 2021-07-31T01:32:57.225 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 15 M 2021-07-31T01:32:57.225 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2021-07-31T01:32:57.394 INFO:teuthology.orchestra.run.smithi154.stdout:(1/6): ceph-prometheus-alerts-16.2.5-150.g5a784 52 kB/s | 9.0 kB 00:00 2021-07-31T01:32:57.431 INFO:teuthology.orchestra.run.smithi154.stdout:(2/6): ceph-grafana-dashboards-16.2.5-150.g5a78 118 kB/s | 25 kB 00:00 2021-07-31T01:32:57.542 INFO:teuthology.orchestra.run.smithi154.stdout:(3/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 228 kB/s | 34 kB 00:00 2021-07-31T01:32:57.576 INFO:teuthology.orchestra.run.smithi154.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 1.3 MB/s | 43 kB 00:00 2021-07-31T01:32:57.577 INFO:teuthology.orchestra.run.smithi154.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.3 MB/s | 196 kB 00:00 2021-07-31T01:32:58.430 INFO:teuthology.orchestra.run.smithi154.stdout:(6/6): ceph-mgr-dashboard-16.2.5-150.g5a784c87. 2.2 MB/s | 2.7 MB 00:01 2021-07-31T01:32:58.433 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2021-07-31T01:32:58.433 INFO:teuthology.orchestra.run.smithi154.stdout:Total 2.5 MB/s | 3.0 MB 00:01 2021-07-31T01:32:58.435 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2021-07-31T01:32:58.444 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2021-07-31T01:32:58.444 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2021-07-31T01:32:58.534 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2021-07-31T01:32:58.537 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2021-07-31T01:32:58.567 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 63 kB/s | 2.4 kB 00:00 2021-07-31T01:32:58.798 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2021-07-31T01:32:58.969 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 1/6 2021-07-31T01:32:59.172 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 2/6 2021-07-31T01:32:59.344 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 3/6 2021-07-31T01:32:59.498 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-prometheus-alerts-2:16.2.5-150.g5a784c87.el8.no 4/6 2021-07-31T01:33:00.553 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-grafana-dashboards-2:16.2.5-150.g5a784c87.el8.n 5/6 2021-07-31T01:33:00.851 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mgr-dashboard-2:16.2.5-150.g5a784c87.el8.noarch 6/6 2021-07-31T01:33:01.090 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.5-150.g5a784c87.el8.noarch 6/6 2021-07-31T01:33:01.090 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-grafana-dashboards-2:16.2.5-150.g5a784c87.el8.n 1/6 2021-07-31T01:33:01.090 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mgr-dashboard-2:16.2.5-150.g5a784c87.el8.noarch 2/6 2021-07-31T01:33:01.091 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-prometheus-alerts-2:16.2.5-150.g5a784c87.el8.no 3/6 2021-07-31T01:33:01.091 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 4/6 2021-07-31T01:33:01.091 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 5/6 2021-07-31T01:33:01.569 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 6/6 2021-07-31T01:33:01.570 INFO:teuthology.orchestra.run.smithi154.stdout:Installed products updated. 2021-07-31T01:33:03.147 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:04 ago on Sat 31 Jul 2021 01:29:23 AM UTC. 2021-07-31T01:33:05.136 INFO:teuthology.orchestra.run.smithi045.stdout:Package ceph-mgr-2:16.2.5-150.g5a784c87.el8.x86_64 is already installed. 2021-07-31T01:33:05.496 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2021-07-31T01:33:05.497 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2021-07-31T01:33:05.498 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2021-07-31T01:33:05.585 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-mgr-dashboard 2021-07-31T01:33:05.883 INFO:teuthology.orchestra.run.smithi045.stdout:Updating Subscription Management repositories. 2021-07-31T01:33:06.425 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 74 kB/s | 2.8 kB 00:00 2021-07-31T01:33:06.589 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 63 kB/s | 2.4 kB 00:00 2021-07-31T01:33:08.168 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2021-07-31T01:33:08.169 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================================== 2021-07-31T01:33:08.169 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2021-07-31T01:33:08.169 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================================== 2021-07-31T01:33:08.170 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2021-07-31T01:33:08.170 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-dashboard noarch 2:16.2.5-150.g5a784c87.el8 ceph-noarch 2.7 M 2021-07-31T01:33:08.170 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2021-07-31T01:33:08.170 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-grafana-dashboards noarch 2:16.2.5-150.g5a784c87.el8 ceph-noarch 25 k 2021-07-31T01:33:08.171 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-prometheus-alerts noarch 2:16.2.5-150.g5a784c87.el8 ceph-noarch 9.0 k 2021-07-31T01:33:08.171 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2021-07-31T01:33:08.171 INFO:teuthology.orchestra.run.smithi045.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2021-07-31T01:33:08.171 INFO:teuthology.orchestra.run.smithi045.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2021-07-31T01:33:08.172 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:33:08.172 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2021-07-31T01:33:08.172 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================================== 2021-07-31T01:33:08.172 INFO:teuthology.orchestra.run.smithi045.stdout:Install 6 Packages 2021-07-31T01:33:08.172 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:33:08.173 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 3.0 M 2021-07-31T01:33:08.173 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 15 M 2021-07-31T01:33:08.174 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2021-07-31T01:33:08.488 INFO:teuthology.orchestra.run.smithi045.stdout:(1/6): ceph-prometheus-alerts-16.2.5-150.g5a784 28 kB/s | 9.0 kB 00:00 2021-07-31T01:33:08.556 INFO:teuthology.orchestra.run.smithi045.stdout:(2/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 497 kB/s | 34 kB 00:00 2021-07-31T01:33:08.578 INFO:teuthology.orchestra.run.smithi045.stdout:(3/6): ceph-grafana-dashboards-16.2.5-150.g5a78 61 kB/s | 25 kB 00:00 2021-07-31T01:33:08.615 INFO:teuthology.orchestra.run.smithi045.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 1.1 MB/s | 43 kB 00:00 2021-07-31T01:33:08.622 INFO:teuthology.orchestra.run.smithi045.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 2.9 MB/s | 196 kB 00:00 2021-07-31T01:33:10.983 INFO:teuthology.orchestra.run.smithi045.stdout:(6/6): ceph-mgr-dashboard-16.2.5-150.g5a784c87. 977 kB/s | 2.7 MB 00:02 2021-07-31T01:33:10.986 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2021-07-31T01:33:10.987 INFO:teuthology.orchestra.run.smithi045.stdout:Total 1.1 MB/s | 3.0 MB 00:02 2021-07-31T01:33:10.989 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2021-07-31T01:33:10.999 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2021-07-31T01:33:11.000 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2021-07-31T01:33:11.097 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2021-07-31T01:33:11.101 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2021-07-31T01:33:11.411 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2021-07-31T01:33:11.453 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:00:05 ago on Sat 31 Jul 2021 01:32:36 AM UTC. 2021-07-31T01:33:11.601 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 1/6 2021-07-31T01:33:11.796 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 2/6 2021-07-31T01:33:11.958 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 3/6 2021-07-31T01:33:12.122 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-prometheus-alerts-2:16.2.5-150.g5a784c87.el8.no 4/6 2021-07-31T01:33:13.273 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-grafana-dashboards-2:16.2.5-150.g5a784c87.el8.n 5/6 2021-07-31T01:33:13.543 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-dashboard-2:16.2.5-150.g5a784c87.el8.noarch 6/6 2021-07-31T01:33:13.679 INFO:teuthology.orchestra.run.smithi154.stdout: 2021-07-31T01:33:13.680 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2021-07-31T01:33:13.680 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-grafana-dashboards-2:16.2.5-150.g5a784c87.el8.noarch 2021-07-31T01:33:13.680 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-dashboard-2:16.2.5-150.g5a784c87.el8.noarch 2021-07-31T01:33:13.681 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-prometheus-alerts-2:16.2.5-150.g5a784c87.el8.noarch 2021-07-31T01:33:13.681 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jwt-1.6.1-2.el8.noarch 2021-07-31T01:33:13.682 INFO:teuthology.orchestra.run.smithi154.stdout: python3-repoze-lru-0.7-6.el8.noarch 2021-07-31T01:33:13.682 INFO:teuthology.orchestra.run.smithi154.stdout: python3-routes-2.4.1-12.el8.noarch 2021-07-31T01:33:13.682 INFO:teuthology.orchestra.run.smithi154.stdout: 2021-07-31T01:33:13.683 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2021-07-31T01:33:13.766 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-mgr-diskprediction-local 2021-07-31T01:33:13.790 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.5-150.g5a784c87.el8.noarch 6/6 2021-07-31T01:33:13.791 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-grafana-dashboards-2:16.2.5-150.g5a784c87.el8.n 1/6 2021-07-31T01:33:13.791 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-dashboard-2:16.2.5-150.g5a784c87.el8.noarch 2/6 2021-07-31T01:33:13.791 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-prometheus-alerts-2:16.2.5-150.g5a784c87.el8.no 3/6 2021-07-31T01:33:13.791 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 4/6 2021-07-31T01:33:13.792 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 5/6 2021-07-31T01:33:14.051 INFO:teuthology.orchestra.run.smithi154.stdout:Updating Subscription Management repositories. 2021-07-31T01:33:14.293 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 6/6 2021-07-31T01:33:14.293 INFO:teuthology.orchestra.run.smithi045.stdout:Installed products updated. 2021-07-31T01:33:14.575 INFO:teuthology.orchestra.run.smithi154.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 97 kB/s | 2.8 kB 00:00 2021-07-31T01:33:18.628 INFO:teuthology.orchestra.run.smithi154.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 63 kB/s | 2.4 kB 00:00 2021-07-31T01:33:22.816 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:00:04 ago on Sat 31 Jul 2021 01:29:23 AM UTC. 2021-07-31T01:33:24.912 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:05 ago on Sat 31 Jul 2021 01:32:45 AM UTC. 2021-07-31T01:33:24.965 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2021-07-31T01:33:24.966 INFO:teuthology.orchestra.run.smithi154.stdout:====================================================================================================================== 2021-07-31T01:33:24.966 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repository Size 2021-07-31T01:33:24.966 INFO:teuthology.orchestra.run.smithi154.stdout:====================================================================================================================== 2021-07-31T01:33:24.966 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2021-07-31T01:33:24.967 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.5-150.g5a784c87.el8 ceph-noarch 8.6 M 2021-07-31T01:33:24.967 INFO:teuthology.orchestra.run.smithi154.stdout:Installing dependencies: 2021-07-31T01:33:24.968 INFO:teuthology.orchestra.run.smithi154.stdout: libgfortran x86_64 8.3.1-5.1.el8 rhel-8-for-x86_64-baseos-rpms 639 k 2021-07-31T01:33:24.968 INFO:teuthology.orchestra.run.smithi154.stdout: libquadmath x86_64 8.3.1-5.1.el8 rhel-8-for-x86_64-baseos-rpms 169 k 2021-07-31T01:33:24.968 INFO:teuthology.orchestra.run.smithi154.stdout: openblas x86_64 0.3.3-5.el8 rhel-8-for-x86_64-appstream-rpms 4.3 M 2021-07-31T01:33:24.968 INFO:teuthology.orchestra.run.smithi154.stdout: openblas-threads x86_64 0.3.3-5.el8 rhel-8-for-x86_64-appstream-rpms 4.4 M 2021-07-31T01:33:24.969 INFO:teuthology.orchestra.run.smithi154.stdout: python3-numpy x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2021-07-31T01:33:24.969 INFO:teuthology.orchestra.run.smithi154.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2021-07-31T01:33:24.969 INFO:teuthology.orchestra.run.smithi154.stdout: python3-scipy x86_64 1.0.0-20.module+el8.1.0+3334+5cb623d7 rhel-8-for-x86_64-appstream-rpms 15 M 2021-07-31T01:33:24.969 INFO:teuthology.orchestra.run.smithi154.stdout: 2021-07-31T01:33:24.970 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2021-07-31T01:33:24.970 INFO:teuthology.orchestra.run.smithi154.stdout:====================================================================================================================== 2021-07-31T01:33:24.970 INFO:teuthology.orchestra.run.smithi154.stdout:Install 8 Packages 2021-07-31T01:33:24.970 INFO:teuthology.orchestra.run.smithi154.stdout: 2021-07-31T01:33:24.971 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 37 M 2021-07-31T01:33:24.972 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 196 M 2021-07-31T01:33:24.972 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2021-07-31T01:33:25.048 INFO:teuthology.orchestra.run.smithi154.stdout:(1/8): openblas-threads-0.3.3-5.el8.x86_64.rpm 55 MB/s | 4.4 MB 00:00 2021-07-31T01:33:25.080 INFO:teuthology.orchestra.run.smithi154.stdout:(2/8): python3-numpy-f2py-1.14.3-9.el8.x86_64.r 8.2 MB/s | 225 kB 00:00 2021-07-31T01:33:25.206 INFO:teuthology.orchestra.run.smithi154.stdout:(3/8): python3-scipy-1.0.0-20.module+el8.1.0+33 61 MB/s | 15 MB 00:00 2021-07-31T01:33:25.244 INFO:teuthology.orchestra.run.smithi154.stdout:(4/8): openblas-0.3.3-5.el8.x86_64.rpm 26 MB/s | 4.3 MB 00:00 2021-07-31T01:33:25.272 INFO:teuthology.orchestra.run.smithi154.stdout:(5/8): python3-numpy-1.14.3-9.el8.x86_64.rpm 57 MB/s | 3.7 MB 00:00 2021-07-31T01:33:25.280 INFO:teuthology.orchestra.run.smithi154.stdout:(6/8): libgfortran-8.3.1-5.1.el8.x86_64.rpm 17 MB/s | 639 kB 00:00 2021-07-31T01:33:25.294 INFO:teuthology.orchestra.run.smithi154.stdout:(7/8): libquadmath-8.3.1-5.1.el8.x86_64.rpm 7.6 MB/s | 169 kB 00:00 2021-07-31T01:33:26.621 INFO:teuthology.orchestra.run.smithi154.stdout:(8/8): ceph-mgr-diskprediction-local-16.2.5-150 5.2 MB/s | 8.6 MB 00:01 2021-07-31T01:33:26.622 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2021-07-31T01:33:26.623 INFO:teuthology.orchestra.run.smithi154.stdout:Total 22 MB/s | 37 MB 00:01 2021-07-31T01:33:26.632 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2021-07-31T01:33:26.651 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2021-07-31T01:33:26.652 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2021-07-31T01:33:26.999 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2021-07-31T01:33:27.002 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2021-07-31T01:33:27.271 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:33:27.271 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2021-07-31T01:33:27.272 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-grafana-dashboards-2:16.2.5-150.g5a784c87.el8.noarch 2021-07-31T01:33:27.272 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-dashboard-2:16.2.5-150.g5a784c87.el8.noarch 2021-07-31T01:33:27.272 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-prometheus-alerts-2:16.2.5-150.g5a784c87.el8.noarch 2021-07-31T01:33:27.272 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jwt-1.6.1-2.el8.noarch 2021-07-31T01:33:27.272 INFO:teuthology.orchestra.run.smithi045.stdout: python3-repoze-lru-0.7-6.el8.noarch 2021-07-31T01:33:27.273 INFO:teuthology.orchestra.run.smithi045.stdout: python3-routes-2.4.1-12.el8.noarch 2021-07-31T01:33:27.273 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:33:27.273 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2021-07-31T01:33:27.369 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-mgr-diskprediction-local 2021-07-31T01:33:27.564 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2021-07-31T01:33:27.671 INFO:teuthology.orchestra.run.smithi045.stdout:Updating Subscription Management repositories. 2021-07-31T01:33:27.719 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libquadmath-8.3.1-5.1.el8.x86_64 1/8 2021-07-31T01:33:27.850 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: libquadmath-8.3.1-5.1.el8.x86_64 1/8 2021-07-31T01:33:28.019 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libgfortran-8.3.1-5.1.el8.x86_64 2/8 2021-07-31T01:33:28.260 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 74 kB/s | 2.8 kB 00:00 2021-07-31T01:33:28.529 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: libgfortran-8.3.1-5.1.el8.x86_64 2/8 2021-07-31T01:33:28.710 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : openblas-threads-0.3.3-5.el8.x86_64 3/8 2021-07-31T01:33:29.214 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: openblas-threads-0.3.3-5.el8.x86_64 3/8 2021-07-31T01:33:29.427 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : openblas-0.3.3-5.el8.x86_64 4/8 2021-07-31T01:33:30.143 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: openblas-0.3.3-5.el8.x86_64 4/8 2021-07-31T01:33:30.511 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 5/8 2021-07-31T01:33:30.760 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2021-07-31T01:33:32.552 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 62 kB/s | 2.4 kB 00:00 2021-07-31T01:33:32.785 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2021-07-31T01:33:34.355 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7. 7/8 2021-07-31T01:33:34.544 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.5-150.g5a784c87 8/8 2021-07-31T01:33:34.824 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.5-150.g5a784c87 8/8 2021-07-31T01:33:34.825 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.5-150.g5a784c87 1/8 2021-07-31T01:33:34.825 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7. 2/8 2021-07-31T01:33:34.825 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : openblas-threads-0.3.3-5.el8.x86_64 3/8 2021-07-31T01:33:34.828 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 4/8 2021-07-31T01:33:34.829 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : openblas-0.3.3-5.el8.x86_64 5/8 2021-07-31T01:33:34.829 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 6/8 2021-07-31T01:33:34.830 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libgfortran-8.3.1-5.1.el8.x86_64 7/8 2021-07-31T01:33:35.326 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libquadmath-8.3.1-5.1.el8.x86_64 8/8 2021-07-31T01:33:35.327 INFO:teuthology.orchestra.run.smithi154.stdout:Installed products updated. 2021-07-31T01:33:37.197 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:04 ago on Sat 31 Jul 2021 01:29:23 AM UTC. 2021-07-31T01:33:39.437 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2021-07-31T01:33:39.441 INFO:teuthology.orchestra.run.smithi045.stdout:====================================================================================================================== 2021-07-31T01:33:39.442 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2021-07-31T01:33:39.442 INFO:teuthology.orchestra.run.smithi045.stdout:====================================================================================================================== 2021-07-31T01:33:39.442 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2021-07-31T01:33:39.442 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.5-150.g5a784c87.el8 ceph-noarch 8.6 M 2021-07-31T01:33:39.443 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2021-07-31T01:33:39.443 INFO:teuthology.orchestra.run.smithi045.stdout: libgfortran x86_64 8.3.1-5.1.el8 rhel-8-for-x86_64-baseos-rpms 639 k 2021-07-31T01:33:39.443 INFO:teuthology.orchestra.run.smithi045.stdout: libquadmath x86_64 8.3.1-5.1.el8 rhel-8-for-x86_64-baseos-rpms 169 k 2021-07-31T01:33:39.443 INFO:teuthology.orchestra.run.smithi045.stdout: openblas x86_64 0.3.3-5.el8 rhel-8-for-x86_64-appstream-rpms 4.3 M 2021-07-31T01:33:39.443 INFO:teuthology.orchestra.run.smithi045.stdout: openblas-threads x86_64 0.3.3-5.el8 rhel-8-for-x86_64-appstream-rpms 4.4 M 2021-07-31T01:33:39.444 INFO:teuthology.orchestra.run.smithi045.stdout: python3-numpy x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2021-07-31T01:33:39.444 INFO:teuthology.orchestra.run.smithi045.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2021-07-31T01:33:39.444 INFO:teuthology.orchestra.run.smithi045.stdout: python3-scipy x86_64 1.0.0-20.module+el8.1.0+3334+5cb623d7 rhel-8-for-x86_64-appstream-rpms 15 M 2021-07-31T01:33:39.444 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:33:39.445 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2021-07-31T01:33:39.445 INFO:teuthology.orchestra.run.smithi045.stdout:====================================================================================================================== 2021-07-31T01:33:39.445 INFO:teuthology.orchestra.run.smithi045.stdout:Install 8 Packages 2021-07-31T01:33:39.445 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:33:39.446 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 37 M 2021-07-31T01:33:39.446 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 196 M 2021-07-31T01:33:39.446 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2021-07-31T01:33:39.543 INFO:teuthology.orchestra.run.smithi045.stdout:(1/8): openblas-threads-0.3.3-5.el8.x86_64.rpm 44 MB/s | 4.4 MB 00:00 2021-07-31T01:33:39.571 INFO:teuthology.orchestra.run.smithi045.stdout:(2/8): python3-numpy-f2py-1.14.3-9.el8.x86_64.r 8.0 MB/s | 225 kB 00:00 2021-07-31T01:33:39.735 INFO:teuthology.orchestra.run.smithi045.stdout:(3/8): python3-scipy-1.0.0-20.module+el8.1.0+33 50 MB/s | 15 MB 00:00 2021-07-31T01:33:39.767 INFO:teuthology.orchestra.run.smithi045.stdout:(4/8): openblas-0.3.3-5.el8.x86_64.rpm 22 MB/s | 4.3 MB 00:00 2021-07-31T01:33:39.799 INFO:teuthology.orchestra.run.smithi045.stdout:(5/8): libgfortran-8.3.1-5.1.el8.x86_64.rpm 20 MB/s | 639 kB 00:00 2021-07-31T01:33:39.810 INFO:teuthology.orchestra.run.smithi045.stdout:(6/8): python3-numpy-1.14.3-9.el8.x86_64.rpm 50 MB/s | 3.7 MB 00:00 2021-07-31T01:33:39.840 INFO:teuthology.orchestra.run.smithi045.stdout:(7/8): libquadmath-8.3.1-5.1.el8.x86_64.rpm 4.0 MB/s | 169 kB 00:00 2021-07-31T01:33:40.243 INFO:teuthology.orchestra.run.smithi045.stdout:(8/8): ceph-mgr-diskprediction-local-16.2.5-150 11 MB/s | 8.6 MB 00:00 2021-07-31T01:33:40.244 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2021-07-31T01:33:40.245 INFO:teuthology.orchestra.run.smithi045.stdout:Total 46 MB/s | 37 MB 00:00 2021-07-31T01:33:40.255 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2021-07-31T01:33:40.275 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2021-07-31T01:33:40.275 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2021-07-31T01:33:40.642 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2021-07-31T01:33:40.646 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2021-07-31T01:33:41.241 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2021-07-31T01:33:41.418 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libquadmath-8.3.1-5.1.el8.x86_64 1/8 2021-07-31T01:33:41.551 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: libquadmath-8.3.1-5.1.el8.x86_64 1/8 2021-07-31T01:33:41.684 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libgfortran-8.3.1-5.1.el8.x86_64 2/8 2021-07-31T01:33:42.263 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: libgfortran-8.3.1-5.1.el8.x86_64 2/8 2021-07-31T01:33:42.409 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : openblas-threads-0.3.3-5.el8.x86_64 3/8 2021-07-31T01:33:42.941 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: openblas-threads-0.3.3-5.el8.x86_64 3/8 2021-07-31T01:33:43.176 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : openblas-0.3.3-5.el8.x86_64 4/8 2021-07-31T01:33:43.975 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: openblas-0.3.3-5.el8.x86_64 4/8 2021-07-31T01:33:44.340 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 5/8 2021-07-31T01:33:44.584 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2021-07-31T01:33:45.283 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:00:05 ago on Sat 31 Jul 2021 01:33:06 AM UTC. 2021-07-31T01:33:46.735 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2021-07-31T01:33:47.995 INFO:teuthology.orchestra.run.smithi154.stdout: 2021-07-31T01:33:47.995 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2021-07-31T01:33:47.995 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-diskprediction-local-2:16.2.5-150.g5a784c87.el8.noarch 2021-07-31T01:33:47.996 INFO:teuthology.orchestra.run.smithi154.stdout: libgfortran-8.3.1-5.1.el8.x86_64 2021-07-31T01:33:47.996 INFO:teuthology.orchestra.run.smithi154.stdout: libquadmath-8.3.1-5.1.el8.x86_64 2021-07-31T01:33:47.996 INFO:teuthology.orchestra.run.smithi154.stdout: openblas-0.3.3-5.el8.x86_64 2021-07-31T01:33:47.996 INFO:teuthology.orchestra.run.smithi154.stdout: openblas-threads-0.3.3-5.el8.x86_64 2021-07-31T01:33:47.996 INFO:teuthology.orchestra.run.smithi154.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2021-07-31T01:33:47.997 INFO:teuthology.orchestra.run.smithi154.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2021-07-31T01:33:47.997 INFO:teuthology.orchestra.run.smithi154.stdout: python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7.x86_64 2021-07-31T01:33:47.997 INFO:teuthology.orchestra.run.smithi154.stdout: 2021-07-31T01:33:47.997 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2021-07-31T01:33:48.087 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-mgr-rook 2021-07-31T01:33:48.130 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7. 7/8 2021-07-31T01:33:48.346 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.5-150.g5a784c87 8/8 2021-07-31T01:33:48.372 INFO:teuthology.orchestra.run.smithi154.stdout:Updating Subscription Management repositories. 2021-07-31T01:33:48.604 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.5-150.g5a784c87 8/8 2021-07-31T01:33:48.604 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.5-150.g5a784c87 1/8 2021-07-31T01:33:48.604 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7. 2/8 2021-07-31T01:33:48.604 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : openblas-threads-0.3.3-5.el8.x86_64 3/8 2021-07-31T01:33:48.605 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 4/8 2021-07-31T01:33:48.605 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : openblas-0.3.3-5.el8.x86_64 5/8 2021-07-31T01:33:48.605 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 6/8 2021-07-31T01:33:48.605 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libgfortran-8.3.1-5.1.el8.x86_64 7/8 2021-07-31T01:33:48.874 INFO:teuthology.orchestra.run.smithi154.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 93 kB/s | 2.8 kB 00:00 2021-07-31T01:33:49.123 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libquadmath-8.3.1-5.1.el8.x86_64 8/8 2021-07-31T01:33:49.123 INFO:teuthology.orchestra.run.smithi045.stdout:Installed products updated. 2021-07-31T01:33:53.119 INFO:teuthology.orchestra.run.smithi154.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 77 kB/s | 2.4 kB 00:00 2021-07-31T01:33:57.474 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:00:04 ago on Sat 31 Jul 2021 01:29:23 AM UTC. 2021-07-31T01:33:59.618 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:05 ago on Sat 31 Jul 2021 01:33:19 AM UTC. 2021-07-31T01:34:00.017 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2021-07-31T01:34:00.018 INFO:teuthology.orchestra.run.smithi154.stdout:======================================================================================================= 2021-07-31T01:34:00.019 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repository Size 2021-07-31T01:34:00.019 INFO:teuthology.orchestra.run.smithi154.stdout:======================================================================================================= 2021-07-31T01:34:00.019 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2021-07-31T01:34:00.019 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-rook noarch 2:16.2.5-150.g5a784c87.el8 ceph-noarch 27 k 2021-07-31T01:34:00.020 INFO:teuthology.orchestra.run.smithi154.stdout:Installing dependencies: 2021-07-31T01:34:00.020 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2021-07-31T01:34:00.020 INFO:teuthology.orchestra.run.smithi154.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2021-07-31T01:34:00.020 INFO:teuthology.orchestra.run.smithi154.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2021-07-31T01:34:00.020 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2021-07-31T01:34:00.021 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2021-07-31T01:34:00.021 INFO:teuthology.orchestra.run.smithi154.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2021-07-31T01:34:00.021 INFO:teuthology.orchestra.run.smithi154.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2021-07-31T01:34:00.021 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2021-07-31T01:34:00.021 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2021-07-31T01:34:00.022 INFO:teuthology.orchestra.run.smithi154.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2021-07-31T01:34:00.022 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rsa noarch 3.4.2-16.el8 epel 73 k 2021-07-31T01:34:00.022 INFO:teuthology.orchestra.run.smithi154.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2021-07-31T01:34:00.022 INFO:teuthology.orchestra.run.smithi154.stdout: 2021-07-31T01:34:00.023 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2021-07-31T01:34:00.023 INFO:teuthology.orchestra.run.smithi154.stdout:======================================================================================================= 2021-07-31T01:34:00.023 INFO:teuthology.orchestra.run.smithi154.stdout:Install 13 Packages 2021-07-31T01:34:00.023 INFO:teuthology.orchestra.run.smithi154.stdout: 2021-07-31T01:34:00.024 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 1.9 M 2021-07-31T01:34:00.024 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 24 M 2021-07-31T01:34:00.025 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2021-07-31T01:34:00.089 INFO:teuthology.orchestra.run.smithi154.stdout:(1/13): python3-certifi-2018.10.15-7.el8.noarch 242 kB/s | 16 kB 00:00 2021-07-31T01:34:00.090 INFO:teuthology.orchestra.run.smithi154.stdout:(2/13): python3-cachetools-3.1.1-4.el8.noarch.r 486 kB/s | 33 kB 00:00 2021-07-31T01:34:00.133 INFO:teuthology.orchestra.run.smithi154.stdout:(3/13): python3-google-auth-1.1.1-10.el8.noarch 1.8 MB/s | 81 kB 00:00 2021-07-31T01:34:00.156 INFO:teuthology.orchestra.run.smithi154.stdout:(4/13): python3-rsa-3.4.2-16.el8.noarch.rpm 3.2 MB/s | 73 kB 00:00 2021-07-31T01:34:00.179 INFO:teuthology.orchestra.run.smithi154.stdout:(5/13): python3-websocket-client-0.56.0-5.el8.n 2.6 MB/s | 61 kB 00:00 2021-07-31T01:34:00.209 INFO:teuthology.orchestra.run.smithi154.stdout:(6/13): python3-jsonpointer-1.10-11.el8.noarch. 660 kB/s | 20 kB 00:00 2021-07-31T01:34:00.229 INFO:teuthology.orchestra.run.smithi154.stdout:(7/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 6.1 MB/s | 126 kB 00:00 2021-07-31T01:34:00.246 INFO:teuthology.orchestra.run.smithi154.stdout:(8/13): python3-kubernetes-11.0.0-6.el8.noarch. 7.4 MB/s | 1.2 MB 00:00 2021-07-31T01:34:00.254 INFO:teuthology.orchestra.run.smithi154.stdout:(9/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 1.1 MB/s | 27 kB 00:00 2021-07-31T01:34:00.278 INFO:teuthology.orchestra.run.smithi154.stdout:(10/13): python3-oauthlib-2.1.0-1.el8.noarch.rp 6.7 MB/s | 155 kB 00:00 2021-07-31T01:34:00.284 INFO:teuthology.orchestra.run.smithi154.stdout:(11/13): python3-pyasn1-modules-0.3.7-6.el8.noa 2.8 MB/s | 110 kB 00:00 2021-07-31T01:34:00.298 INFO:teuthology.orchestra.run.smithi154.stdout:(12/13): python3-requests-oauthlib-1.0.0-1.el8. 2.0 MB/s | 43 kB 00:00 2021-07-31T01:34:00.302 INFO:teuthology.orchestra.run.smithi154.stdout:(13/13): ceph-mgr-rook-16.2.5-150.g5a784c87.el8 98 kB/s | 27 kB 00:00 2021-07-31T01:34:00.303 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2021-07-31T01:34:00.303 INFO:teuthology.orchestra.run.smithi154.stdout:Total 6.8 MB/s | 1.9 MB 00:00 2021-07-31T01:34:00.308 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2021-07-31T01:34:00.321 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2021-07-31T01:34:00.322 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2021-07-31T01:34:00.392 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2021-07-31T01:34:00.395 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2021-07-31T01:34:00.985 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2021-07-31T01:34:01.247 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2021-07-31T01:34:01.453 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-rsa-3.4.2-16.el8.noarch 2/13 2021-07-31T01:34:01.742 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 3/13 2021-07-31T01:34:01.970 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:34:01.971 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2021-07-31T01:34:01.971 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-diskprediction-local-2:16.2.5-150.g5a784c87.el8.noarch 2021-07-31T01:34:01.971 INFO:teuthology.orchestra.run.smithi045.stdout: libgfortran-8.3.1-5.1.el8.x86_64 2021-07-31T01:34:01.972 INFO:teuthology.orchestra.run.smithi045.stdout: libquadmath-8.3.1-5.1.el8.x86_64 2021-07-31T01:34:01.972 INFO:teuthology.orchestra.run.smithi045.stdout: openblas-0.3.3-5.el8.x86_64 2021-07-31T01:34:01.972 INFO:teuthology.orchestra.run.smithi045.stdout: openblas-threads-0.3.3-5.el8.x86_64 2021-07-31T01:34:01.972 INFO:teuthology.orchestra.run.smithi045.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2021-07-31T01:34:01.972 INFO:teuthology.orchestra.run.smithi045.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2021-07-31T01:34:01.973 INFO:teuthology.orchestra.run.smithi045.stdout: python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7.x86_64 2021-07-31T01:34:01.973 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:34:01.973 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2021-07-31T01:34:01.987 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 4/13 2021-07-31T01:34:02.068 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-mgr-rook 2021-07-31T01:34:02.295 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 5/13 2021-07-31T01:34:02.371 INFO:teuthology.orchestra.run.smithi045.stdout:Updating Subscription Management repositories. 2021-07-31T01:34:02.455 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 6/13 2021-07-31T01:34:02.720 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 7/13 2021-07-31T01:34:02.919 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 73 kB/s | 2.8 kB 00:00 2021-07-31T01:34:02.982 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 8/13 2021-07-31T01:34:03.184 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 9/13 2021-07-31T01:34:03.447 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 10/13 2021-07-31T01:34:04.662 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 11/13 2021-07-31T01:34:06.182 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 12/13 2021-07-31T01:34:06.412 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mgr-rook-2:16.2.5-150.g5a784c87.el8.noarch 13/13 2021-07-31T01:34:06.658 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.5-150.g5a784c87.el8.noarch 13/13 2021-07-31T01:34:06.659 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mgr-rook-2:16.2.5-150.g5a784c87.el8.noarch 1/13 2021-07-31T01:34:06.659 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 2/13 2021-07-31T01:34:06.659 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 3/13 2021-07-31T01:34:06.659 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 4/13 2021-07-31T01:34:06.659 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 5/13 2021-07-31T01:34:06.660 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-rsa-3.4.2-16.el8.noarch 6/13 2021-07-31T01:34:06.660 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 7/13 2021-07-31T01:34:06.660 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 8/13 2021-07-31T01:34:06.660 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 9/13 2021-07-31T01:34:06.661 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 10/13 2021-07-31T01:34:06.661 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 11/13 2021-07-31T01:34:06.661 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 12/13 2021-07-31T01:34:07.241 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 13/13 2021-07-31T01:34:07.242 INFO:teuthology.orchestra.run.smithi154.stdout:Installed products updated. 2021-07-31T01:34:07.262 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 62 kB/s | 2.4 kB 00:00 2021-07-31T01:34:11.921 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:04 ago on Sat 31 Jul 2021 01:29:23 AM UTC. 2021-07-31T01:34:14.433 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2021-07-31T01:34:14.434 INFO:teuthology.orchestra.run.smithi045.stdout:======================================================================================================= 2021-07-31T01:34:14.435 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2021-07-31T01:34:14.435 INFO:teuthology.orchestra.run.smithi045.stdout:======================================================================================================= 2021-07-31T01:34:14.435 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2021-07-31T01:34:14.435 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-rook noarch 2:16.2.5-150.g5a784c87.el8 ceph-noarch 27 k 2021-07-31T01:34:14.436 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2021-07-31T01:34:14.436 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2021-07-31T01:34:14.436 INFO:teuthology.orchestra.run.smithi045.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2021-07-31T01:34:14.436 INFO:teuthology.orchestra.run.smithi045.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2021-07-31T01:34:14.437 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2021-07-31T01:34:14.437 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2021-07-31T01:34:14.437 INFO:teuthology.orchestra.run.smithi045.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2021-07-31T01:34:14.437 INFO:teuthology.orchestra.run.smithi045.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2021-07-31T01:34:14.437 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2021-07-31T01:34:14.438 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2021-07-31T01:34:14.438 INFO:teuthology.orchestra.run.smithi045.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2021-07-31T01:34:14.438 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rsa noarch 3.4.2-16.el8 epel 73 k 2021-07-31T01:34:14.438 INFO:teuthology.orchestra.run.smithi045.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2021-07-31T01:34:14.439 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:34:14.439 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2021-07-31T01:34:14.439 INFO:teuthology.orchestra.run.smithi045.stdout:======================================================================================================= 2021-07-31T01:34:14.439 INFO:teuthology.orchestra.run.smithi045.stdout:Install 13 Packages 2021-07-31T01:34:14.440 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:34:14.441 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 1.9 M 2021-07-31T01:34:14.441 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 24 M 2021-07-31T01:34:14.441 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2021-07-31T01:34:14.519 INFO:teuthology.orchestra.run.smithi045.stdout:(1/13): python3-certifi-2018.10.15-7.el8.noarch 202 kB/s | 16 kB 00:00 2021-07-31T01:34:14.519 INFO:teuthology.orchestra.run.smithi045.stdout:(2/13): python3-cachetools-3.1.1-4.el8.noarch.r 407 kB/s | 33 kB 00:00 2021-07-31T01:34:14.572 INFO:teuthology.orchestra.run.smithi045.stdout:(3/13): python3-google-auth-1.1.1-10.el8.noarch 1.5 MB/s | 81 kB 00:00 2021-07-31T01:34:14.600 INFO:teuthology.orchestra.run.smithi045.stdout:(4/13): python3-rsa-3.4.2-16.el8.noarch.rpm 2.6 MB/s | 73 kB 00:00 2021-07-31T01:34:14.628 INFO:teuthology.orchestra.run.smithi045.stdout:(5/13): python3-websocket-client-0.56.0-5.el8.n 2.2 MB/s | 61 kB 00:00 2021-07-31T01:34:14.667 INFO:teuthology.orchestra.run.smithi045.stdout:(6/13): python3-jsonpointer-1.10-11.el8.noarch. 517 kB/s | 20 kB 00:00 2021-07-31T01:34:14.676 INFO:teuthology.orchestra.run.smithi045.stdout:(7/13): ceph-mgr-rook-16.2.5-150.g5a784c87.el8. 115 kB/s | 27 kB 00:00 2021-07-31T01:34:14.694 INFO:teuthology.orchestra.run.smithi045.stdout:(8/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 4.5 MB/s | 126 kB 00:00 2021-07-31T01:34:14.727 INFO:teuthology.orchestra.run.smithi045.stdout:(9/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 528 kB/s | 27 kB 00:00 2021-07-31T01:34:14.737 INFO:teuthology.orchestra.run.smithi045.stdout:(10/13): python3-pyasn1-modules-0.3.7-6.el8.noa 2.5 MB/s | 110 kB 00:00 2021-07-31T01:34:14.753 INFO:teuthology.orchestra.run.smithi045.stdout:(11/13): python3-oauthlib-2.1.0-1.el8.noarch.rp 5.7 MB/s | 155 kB 00:00 2021-07-31T01:34:14.766 INFO:teuthology.orchestra.run.smithi045.stdout:(12/13): python3-kubernetes-11.0.0-6.el8.noarch 4.7 MB/s | 1.2 MB 00:00 2021-07-31T01:34:14.770 INFO:teuthology.orchestra.run.smithi045.stdout:(13/13): python3-requests-oauthlib-1.0.0-1.el8. 1.3 MB/s | 43 kB 00:00 2021-07-31T01:34:14.771 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2021-07-31T01:34:14.771 INFO:teuthology.orchestra.run.smithi045.stdout:Total 5.7 MB/s | 1.9 MB 00:00 2021-07-31T01:34:14.777 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2021-07-31T01:34:14.793 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2021-07-31T01:34:14.793 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2021-07-31T01:34:14.868 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2021-07-31T01:34:14.872 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2021-07-31T01:34:15.482 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2021-07-31T01:34:15.706 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2021-07-31T01:34:15.948 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-rsa-3.4.2-16.el8.noarch 2/13 2021-07-31T01:34:16.382 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 3/13 2021-07-31T01:34:16.631 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 4/13 2021-07-31T01:34:16.845 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 5/13 2021-07-31T01:34:17.013 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 6/13 2021-07-31T01:34:17.283 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 7/13 2021-07-31T01:34:17.541 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:00:05 ago on Sat 31 Jul 2021 01:33:40 AM UTC. 2021-07-31T01:34:17.565 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 8/13 2021-07-31T01:34:17.793 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 9/13 2021-07-31T01:34:18.041 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 10/13 2021-07-31T01:34:19.277 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 11/13 2021-07-31T01:34:19.808 INFO:teuthology.orchestra.run.smithi154.stdout: 2021-07-31T01:34:19.808 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2021-07-31T01:34:19.808 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-rook-2:16.2.5-150.g5a784c87.el8.noarch 2021-07-31T01:34:19.808 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cachetools-3.1.1-4.el8.noarch 2021-07-31T01:34:19.809 INFO:teuthology.orchestra.run.smithi154.stdout: python3-certifi-2018.10.15-7.el8.noarch 2021-07-31T01:34:19.809 INFO:teuthology.orchestra.run.smithi154.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2021-07-31T01:34:19.809 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jsonpatch-1.21-2.el8.noarch 2021-07-31T01:34:19.809 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jsonpointer-1.10-11.el8.noarch 2021-07-31T01:34:19.810 INFO:teuthology.orchestra.run.smithi154.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2021-07-31T01:34:19.810 INFO:teuthology.orchestra.run.smithi154.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2021-07-31T01:34:19.810 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2021-07-31T01:34:19.810 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2021-07-31T01:34:19.810 INFO:teuthology.orchestra.run.smithi154.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2021-07-31T01:34:19.811 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rsa-3.4.2-16.el8.noarch 2021-07-31T01:34:19.811 INFO:teuthology.orchestra.run.smithi154.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2021-07-31T01:34:19.811 INFO:teuthology.orchestra.run.smithi154.stdout: 2021-07-31T01:34:19.811 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2021-07-31T01:34:19.886 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-mgr-cephadm 2021-07-31T01:34:20.172 INFO:teuthology.orchestra.run.smithi154.stdout:Updating Subscription Management repositories. 2021-07-31T01:34:20.291 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 12/13 2021-07-31T01:34:20.487 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-rook-2:16.2.5-150.g5a784c87.el8.noarch 13/13 2021-07-31T01:34:20.682 INFO:teuthology.orchestra.run.smithi154.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 99 kB/s | 2.8 kB 00:00 2021-07-31T01:34:20.743 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.5-150.g5a784c87.el8.noarch 13/13 2021-07-31T01:34:20.743 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-rook-2:16.2.5-150.g5a784c87.el8.noarch 1/13 2021-07-31T01:34:20.744 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 2/13 2021-07-31T01:34:20.744 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 3/13 2021-07-31T01:34:20.744 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 4/13 2021-07-31T01:34:20.745 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 5/13 2021-07-31T01:34:20.745 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-rsa-3.4.2-16.el8.noarch 6/13 2021-07-31T01:34:20.745 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 7/13 2021-07-31T01:34:20.746 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 8/13 2021-07-31T01:34:20.746 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 9/13 2021-07-31T01:34:20.746 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 10/13 2021-07-31T01:34:20.746 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 11/13 2021-07-31T01:34:20.746 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 12/13 2021-07-31T01:34:21.428 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 13/13 2021-07-31T01:34:21.429 INFO:teuthology.orchestra.run.smithi045.stdout:Installed products updated. 2021-07-31T01:34:24.808 INFO:teuthology.orchestra.run.smithi154.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 82 kB/s | 2.4 kB 00:00 2021-07-31T01:34:29.142 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:00:04 ago on Sat 31 Jul 2021 01:29:23 AM UTC. 2021-07-31T01:34:31.247 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2021-07-31T01:34:31.247 INFO:teuthology.orchestra.run.smithi154.stdout:============================================================================================== 2021-07-31T01:34:31.248 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repository Size 2021-07-31T01:34:31.248 INFO:teuthology.orchestra.run.smithi154.stdout:============================================================================================== 2021-07-31T01:34:31.248 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2021-07-31T01:34:31.248 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-cephadm noarch 2:16.2.5-150.g5a784c87.el8 ceph-noarch 98 k 2021-07-31T01:34:31.249 INFO:teuthology.orchestra.run.smithi154.stdout:Installing dependencies: 2021-07-31T01:34:31.249 INFO:teuthology.orchestra.run.smithi154.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2021-07-31T01:34:31.249 INFO:teuthology.orchestra.run.smithi154.stdout: python3-babel noarch 2.5.1-5.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2021-07-31T01:34:31.250 INFO:teuthology.orchestra.run.smithi154.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2021-07-31T01:34:31.250 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jinja2 noarch 2.10.1-2.el8_0 rhel-8-for-x86_64-appstream-rpms 538 k 2021-07-31T01:34:31.250 INFO:teuthology.orchestra.run.smithi154.stdout: python3-remoto noarch 1.1.4-4.el8 epel 48 k 2021-07-31T01:34:31.250 INFO:teuthology.orchestra.run.smithi154.stdout: 2021-07-31T01:34:31.251 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2021-07-31T01:34:31.251 INFO:teuthology.orchestra.run.smithi154.stdout:============================================================================================== 2021-07-31T01:34:31.251 INFO:teuthology.orchestra.run.smithi154.stdout:Install 6 Packages 2021-07-31T01:34:31.252 INFO:teuthology.orchestra.run.smithi154.stdout: 2021-07-31T01:34:31.252 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 5.8 M 2021-07-31T01:34:31.253 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 24 M 2021-07-31T01:34:31.253 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2021-07-31T01:34:31.331 INFO:teuthology.orchestra.run.smithi154.stdout:(1/6): python3-apipkg-1.5-6.el8.noarch.rpm 260 kB/s | 21 kB 00:00 2021-07-31T01:34:31.379 INFO:teuthology.orchestra.run.smithi154.stdout:(2/6): python3-remoto-1.1.4-4.el8.noarch.rpm 1.0 MB/s | 48 kB 00:00 2021-07-31T01:34:31.404 INFO:teuthology.orchestra.run.smithi154.stdout:(3/6): python3-execnet-1.7.1-1.el8.noarch.rpm 2.2 MB/s | 355 kB 00:00 2021-07-31T01:34:31.416 INFO:teuthology.orchestra.run.smithi154.stdout:(4/6): python3-jinja2-2.10.1-2.el8_0.noarch.rpm 14 MB/s | 538 kB 00:00 2021-07-31T01:34:31.460 INFO:teuthology.orchestra.run.smithi154.stdout:(5/6): python3-babel-2.5.1-5.el8.noarch.rpm 87 MB/s | 4.8 MB 00:00 2021-07-31T01:34:31.738 INFO:teuthology.orchestra.run.smithi154.stdout:(6/6): ceph-mgr-cephadm-16.2.5-150.g5a784c87.el 201 kB/s | 98 kB 00:00 2021-07-31T01:34:31.739 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2021-07-31T01:34:31.739 INFO:teuthology.orchestra.run.smithi154.stdout:Total 12 MB/s | 5.8 MB 00:00 2021-07-31T01:34:31.740 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2021-07-31T01:34:31.750 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2021-07-31T01:34:31.750 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2021-07-31T01:34:31.827 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2021-07-31T01:34:31.830 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2021-07-31T01:34:32.253 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:05 ago on Sat 31 Jul 2021 01:33:54 AM UTC. 2021-07-31T01:34:32.881 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2021-07-31T01:34:33.275 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-babel-2.5.1-5.el8.noarch 1/6 2021-07-31T01:34:33.473 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jinja2-2.10.1-2.el8_0.noarch 2/6 2021-07-31T01:34:33.749 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 3/6 2021-07-31T01:34:34.013 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 4/6 2021-07-31T01:34:34.291 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-remoto-1.1.4-4.el8.noarch 5/6 2021-07-31T01:34:34.441 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mgr-cephadm-2:16.2.5-150.g5a784c87.el8.noarch 6/6 2021-07-31T01:34:34.631 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:34:34.632 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2021-07-31T01:34:34.632 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-rook-2:16.2.5-150.g5a784c87.el8.noarch 2021-07-31T01:34:34.632 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cachetools-3.1.1-4.el8.noarch 2021-07-31T01:34:34.633 INFO:teuthology.orchestra.run.smithi045.stdout: python3-certifi-2018.10.15-7.el8.noarch 2021-07-31T01:34:34.633 INFO:teuthology.orchestra.run.smithi045.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2021-07-31T01:34:34.633 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jsonpatch-1.21-2.el8.noarch 2021-07-31T01:34:34.633 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jsonpointer-1.10-11.el8.noarch 2021-07-31T01:34:34.633 INFO:teuthology.orchestra.run.smithi045.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2021-07-31T01:34:34.634 INFO:teuthology.orchestra.run.smithi045.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2021-07-31T01:34:34.634 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2021-07-31T01:34:34.634 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2021-07-31T01:34:34.635 INFO:teuthology.orchestra.run.smithi045.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2021-07-31T01:34:34.635 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rsa-3.4.2-16.el8.noarch 2021-07-31T01:34:34.635 INFO:teuthology.orchestra.run.smithi045.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2021-07-31T01:34:34.635 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:34:34.635 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2021-07-31T01:34:34.663 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.5-150.g5a784c87.el8.noarch 6/6 2021-07-31T01:34:34.664 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mgr-cephadm-2:16.2.5-150.g5a784c87.el8.noarch 1/6 2021-07-31T01:34:34.664 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 2/6 2021-07-31T01:34:34.664 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 3/6 2021-07-31T01:34:34.665 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-remoto-1.1.4-4.el8.noarch 4/6 2021-07-31T01:34:34.666 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jinja2-2.10.1-2.el8_0.noarch 5/6 2021-07-31T01:34:34.750 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-mgr-cephadm 2021-07-31T01:34:35.055 INFO:teuthology.orchestra.run.smithi045.stdout:Updating Subscription Management repositories. 2021-07-31T01:34:35.158 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-babel-2.5.1-5.el8.noarch 6/6 2021-07-31T01:34:35.158 INFO:teuthology.orchestra.run.smithi154.stdout:Installed products updated. 2021-07-31T01:34:35.648 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 73 kB/s | 2.8 kB 00:00 2021-07-31T01:34:39.897 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 61 kB/s | 2.4 kB 00:00 2021-07-31T01:34:44.395 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:04 ago on Sat 31 Jul 2021 01:29:23 AM UTC. 2021-07-31T01:34:45.480 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:00:05 ago on Sat 31 Jul 2021 01:34:12 AM UTC. 2021-07-31T01:34:46.860 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2021-07-31T01:34:46.861 INFO:teuthology.orchestra.run.smithi045.stdout:============================================================================================== 2021-07-31T01:34:46.861 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2021-07-31T01:34:46.862 INFO:teuthology.orchestra.run.smithi045.stdout:============================================================================================== 2021-07-31T01:34:46.862 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2021-07-31T01:34:46.862 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-cephadm noarch 2:16.2.5-150.g5a784c87.el8 ceph-noarch 98 k 2021-07-31T01:34:46.862 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2021-07-31T01:34:46.863 INFO:teuthology.orchestra.run.smithi045.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2021-07-31T01:34:46.863 INFO:teuthology.orchestra.run.smithi045.stdout: python3-babel noarch 2.5.1-5.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2021-07-31T01:34:46.863 INFO:teuthology.orchestra.run.smithi045.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2021-07-31T01:34:46.863 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jinja2 noarch 2.10.1-2.el8_0 rhel-8-for-x86_64-appstream-rpms 538 k 2021-07-31T01:34:46.863 INFO:teuthology.orchestra.run.smithi045.stdout: python3-remoto noarch 1.1.4-4.el8 epel 48 k 2021-07-31T01:34:46.864 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:34:46.864 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2021-07-31T01:34:46.864 INFO:teuthology.orchestra.run.smithi045.stdout:============================================================================================== 2021-07-31T01:34:46.864 INFO:teuthology.orchestra.run.smithi045.stdout:Install 6 Packages 2021-07-31T01:34:46.865 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:34:46.865 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 5.8 M 2021-07-31T01:34:46.866 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 24 M 2021-07-31T01:34:46.866 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2021-07-31T01:34:46.944 INFO:teuthology.orchestra.run.smithi045.stdout:(1/6): python3-apipkg-1.5-6.el8.noarch.rpm 263 kB/s | 21 kB 00:00 2021-07-31T01:34:46.997 INFO:teuthology.orchestra.run.smithi045.stdout:(2/6): python3-execnet-1.7.1-1.el8.noarch.rpm 2.6 MB/s | 355 kB 00:00 2021-07-31T01:34:46.998 INFO:teuthology.orchestra.run.smithi045.stdout:(3/6): python3-remoto-1.1.4-4.el8.noarch.rpm 898 kB/s | 48 kB 00:00 2021-07-31T01:34:47.065 INFO:teuthology.orchestra.run.smithi045.stdout:(4/6): python3-jinja2-2.10.1-2.el8_0.noarch.rpm 7.8 MB/s | 538 kB 00:00 2021-07-31T01:34:47.084 INFO:teuthology.orchestra.run.smithi045.stdout:(5/6): python3-babel-2.5.1-5.el8.noarch.rpm 56 MB/s | 4.8 MB 00:00 2021-07-31T01:34:47.103 INFO:teuthology.orchestra.run.smithi045.stdout:(6/6): ceph-mgr-cephadm-16.2.5-150.g5a784c87.el 409 kB/s | 98 kB 00:00 2021-07-31T01:34:47.104 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2021-07-31T01:34:47.105 INFO:teuthology.orchestra.run.smithi045.stdout:Total 24 MB/s | 5.8 MB 00:00 2021-07-31T01:34:47.108 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2021-07-31T01:34:47.117 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2021-07-31T01:34:47.118 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2021-07-31T01:34:47.200 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2021-07-31T01:34:47.204 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2021-07-31T01:34:47.737 INFO:teuthology.orchestra.run.smithi154.stdout: 2021-07-31T01:34:47.738 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2021-07-31T01:34:47.738 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-cephadm-2:16.2.5-150.g5a784c87.el8.noarch 2021-07-31T01:34:47.738 INFO:teuthology.orchestra.run.smithi154.stdout: python3-apipkg-1.5-6.el8.noarch 2021-07-31T01:34:47.738 INFO:teuthology.orchestra.run.smithi154.stdout: python3-babel-2.5.1-5.el8.noarch 2021-07-31T01:34:47.739 INFO:teuthology.orchestra.run.smithi154.stdout: python3-execnet-1.7.1-1.el8.noarch 2021-07-31T01:34:47.739 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jinja2-2.10.1-2.el8_0.noarch 2021-07-31T01:34:47.739 INFO:teuthology.orchestra.run.smithi154.stdout: python3-remoto-1.1.4-4.el8.noarch 2021-07-31T01:34:47.740 INFO:teuthology.orchestra.run.smithi154.stdout: 2021-07-31T01:34:47.740 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2021-07-31T01:34:47.819 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-fuse 2021-07-31T01:34:48.108 INFO:teuthology.orchestra.run.smithi154.stdout:Updating Subscription Management repositories. 2021-07-31T01:34:48.313 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2021-07-31T01:34:48.648 INFO:teuthology.orchestra.run.smithi154.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 96 kB/s | 2.8 kB 00:00 2021-07-31T01:34:48.682 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-babel-2.5.1-5.el8.noarch 1/6 2021-07-31T01:34:48.890 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jinja2-2.10.1-2.el8_0.noarch 2/6 2021-07-31T01:34:49.147 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 3/6 2021-07-31T01:34:49.399 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 4/6 2021-07-31T01:34:49.668 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-remoto-1.1.4-4.el8.noarch 5/6 2021-07-31T01:34:50.024 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-cephadm-2:16.2.5-150.g5a784c87.el8.noarch 6/6 2021-07-31T01:34:50.272 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.5-150.g5a784c87.el8.noarch 6/6 2021-07-31T01:34:50.273 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-cephadm-2:16.2.5-150.g5a784c87.el8.noarch 1/6 2021-07-31T01:34:50.273 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 2/6 2021-07-31T01:34:50.274 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 3/6 2021-07-31T01:34:50.274 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-remoto-1.1.4-4.el8.noarch 4/6 2021-07-31T01:34:50.274 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jinja2-2.10.1-2.el8_0.noarch 5/6 2021-07-31T01:34:50.782 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-babel-2.5.1-5.el8.noarch 6/6 2021-07-31T01:34:50.783 INFO:teuthology.orchestra.run.smithi045.stdout:Installed products updated. 2021-07-31T01:34:52.623 INFO:teuthology.orchestra.run.smithi154.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 77 kB/s | 2.4 kB 00:00 2021-07-31T01:34:56.874 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:00:04 ago on Sat 31 Jul 2021 01:29:23 AM UTC. 2021-07-31T01:34:58.975 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2021-07-31T01:34:58.976 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2021-07-31T01:34:58.976 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repository Size 2021-07-31T01:34:58.976 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2021-07-31T01:34:58.977 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2021-07-31T01:34:58.977 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-fuse 2021-07-31T01:34:58.977 INFO:teuthology.orchestra.run.smithi154.stdout: x86_64 2:16.2.5-150.g5a784c87.el8 ceph 893 k 2021-07-31T01:34:58.977 INFO:teuthology.orchestra.run.smithi154.stdout:Installing dependencies: 2021-07-31T01:34:58.977 INFO:teuthology.orchestra.run.smithi154.stdout: fuse x86_64 2.9.7-12.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2021-07-31T01:34:58.978 INFO:teuthology.orchestra.run.smithi154.stdout: 2021-07-31T01:34:58.978 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2021-07-31T01:34:58.978 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2021-07-31T01:34:58.978 INFO:teuthology.orchestra.run.smithi154.stdout:Install 2 Packages 2021-07-31T01:34:58.979 INFO:teuthology.orchestra.run.smithi154.stdout: 2021-07-31T01:34:58.979 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 976 k 2021-07-31T01:34:58.979 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 2.8 M 2021-07-31T01:34:58.979 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2021-07-31T01:34:59.007 INFO:teuthology.orchestra.run.smithi154.stdout:(1/2): fuse-2.9.7-12.el8.x86_64.rpm 2.7 MB/s | 83 kB 00:00 2021-07-31T01:34:59.693 INFO:teuthology.orchestra.run.smithi154.stdout:(2/2): ceph-fuse-16.2.5-150.g5a784c87.el8.x86_6 1.2 MB/s | 893 kB 00:00 2021-07-31T01:34:59.694 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2021-07-31T01:34:59.695 INFO:teuthology.orchestra.run.smithi154.stdout:Total 1.3 MB/s | 976 kB 00:00 2021-07-31T01:34:59.696 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2021-07-31T01:34:59.717 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2021-07-31T01:34:59.718 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2021-07-31T01:34:59.795 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2021-07-31T01:34:59.799 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2021-07-31T01:34:59.955 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2021-07-31T01:35:00.217 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : fuse-2.9.7-12.el8.x86_64 1/2 2021-07-31T01:35:00.419 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-fuse-2:16.2.5-150.g5a784c87.el8.x86_64 2/2 2021-07-31T01:35:00.747 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-fuse-2:16.2.5-150.g5a784c87.el8.x86_64 2/2 2021-07-31T01:35:00.747 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-fuse-2:16.2.5-150.g5a784c87.el8.x86_64 1/2 2021-07-31T01:35:01.213 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 2/2 2021-07-31T01:35:01.213 INFO:teuthology.orchestra.run.smithi154.stdout:Installed products updated. 2021-07-31T01:35:01.293 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:05 ago on Sat 31 Jul 2021 01:34:27 AM UTC. 2021-07-31T01:35:03.592 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:35:03.592 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2021-07-31T01:35:03.592 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-cephadm-2:16.2.5-150.g5a784c87.el8.noarch 2021-07-31T01:35:03.592 INFO:teuthology.orchestra.run.smithi045.stdout: python3-apipkg-1.5-6.el8.noarch 2021-07-31T01:35:03.593 INFO:teuthology.orchestra.run.smithi045.stdout: python3-babel-2.5.1-5.el8.noarch 2021-07-31T01:35:03.593 INFO:teuthology.orchestra.run.smithi045.stdout: python3-execnet-1.7.1-1.el8.noarch 2021-07-31T01:35:03.593 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jinja2-2.10.1-2.el8_0.noarch 2021-07-31T01:35:03.593 INFO:teuthology.orchestra.run.smithi045.stdout: python3-remoto-1.1.4-4.el8.noarch 2021-07-31T01:35:03.594 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:35:03.594 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2021-07-31T01:35:03.684 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-fuse 2021-07-31T01:35:03.992 INFO:teuthology.orchestra.run.smithi045.stdout:Updating Subscription Management repositories. 2021-07-31T01:35:04.556 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 76 kB/s | 2.8 kB 00:00 2021-07-31T01:35:08.935 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 66 kB/s | 2.4 kB 00:00 2021-07-31T01:35:11.541 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:00:05 ago on Sat 31 Jul 2021 01:34:40 AM UTC. 2021-07-31T01:35:13.583 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:04 ago on Sat 31 Jul 2021 01:29:23 AM UTC. 2021-07-31T01:35:13.729 INFO:teuthology.orchestra.run.smithi154.stdout: 2021-07-31T01:35:13.729 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2021-07-31T01:35:13.730 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-fuse-2:16.2.5-150.g5a784c87.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2021-07-31T01:35:13.730 INFO:teuthology.orchestra.run.smithi154.stdout: 2021-07-31T01:35:13.730 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2021-07-31T01:35:13.810 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install librados-devel 2021-07-31T01:35:14.100 INFO:teuthology.orchestra.run.smithi154.stdout:Updating Subscription Management repositories. 2021-07-31T01:35:14.633 INFO:teuthology.orchestra.run.smithi154.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 93 kB/s | 2.8 kB 00:00 2021-07-31T01:35:15.856 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2021-07-31T01:35:15.857 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2021-07-31T01:35:15.858 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2021-07-31T01:35:15.858 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2021-07-31T01:35:15.858 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2021-07-31T01:35:15.858 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-fuse 2021-07-31T01:35:15.859 INFO:teuthology.orchestra.run.smithi045.stdout: x86_64 2:16.2.5-150.g5a784c87.el8 ceph 893 k 2021-07-31T01:35:15.859 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2021-07-31T01:35:15.859 INFO:teuthology.orchestra.run.smithi045.stdout: fuse x86_64 2.9.7-12.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2021-07-31T01:35:15.859 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:35:15.859 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2021-07-31T01:35:15.860 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2021-07-31T01:35:15.860 INFO:teuthology.orchestra.run.smithi045.stdout:Install 2 Packages 2021-07-31T01:35:15.860 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:35:15.860 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 976 k 2021-07-31T01:35:15.861 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 2.8 M 2021-07-31T01:35:15.861 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2021-07-31T01:35:15.896 INFO:teuthology.orchestra.run.smithi045.stdout:(1/2): fuse-2.9.7-12.el8.x86_64.rpm 2.1 MB/s | 83 kB 00:00 2021-07-31T01:35:16.194 INFO:teuthology.orchestra.run.smithi045.stdout:(2/2): ceph-fuse-16.2.5-150.g5a784c87.el8.x86_6 2.6 MB/s | 893 kB 00:00 2021-07-31T01:35:16.195 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2021-07-31T01:35:16.195 INFO:teuthology.orchestra.run.smithi045.stdout:Total 2.8 MB/s | 976 kB 00:00 2021-07-31T01:35:16.197 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2021-07-31T01:35:16.220 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2021-07-31T01:35:16.220 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2021-07-31T01:35:16.304 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2021-07-31T01:35:16.308 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2021-07-31T01:35:16.483 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2021-07-31T01:35:16.739 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : fuse-2.9.7-12.el8.x86_64 1/2 2021-07-31T01:35:16.895 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-fuse-2:16.2.5-150.g5a784c87.el8.x86_64 2/2 2021-07-31T01:35:17.267 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-fuse-2:16.2.5-150.g5a784c87.el8.x86_64 2/2 2021-07-31T01:35:17.267 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-fuse-2:16.2.5-150.g5a784c87.el8.x86_64 1/2 2021-07-31T01:35:17.796 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 2/2 2021-07-31T01:35:17.796 INFO:teuthology.orchestra.run.smithi045.stdout:Installed products updated. 2021-07-31T01:35:18.665 INFO:teuthology.orchestra.run.smithi154.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 72 kB/s | 2.4 kB 00:00 2021-07-31T01:35:22.861 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:00:04 ago on Sat 31 Jul 2021 01:29:23 AM UTC. 2021-07-31T01:35:24.975 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2021-07-31T01:35:24.977 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2021-07-31T01:35:24.977 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repo Size 2021-07-31T01:35:24.977 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2021-07-31T01:35:24.978 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2021-07-31T01:35:24.978 INFO:teuthology.orchestra.run.smithi154.stdout: librados-devel x86_64 2:16.2.5-150.g5a784c87.el8 ceph 131 k 2021-07-31T01:35:24.978 INFO:teuthology.orchestra.run.smithi154.stdout: 2021-07-31T01:35:24.978 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2021-07-31T01:35:24.978 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2021-07-31T01:35:24.979 INFO:teuthology.orchestra.run.smithi154.stdout:Install 1 Package 2021-07-31T01:35:24.979 INFO:teuthology.orchestra.run.smithi154.stdout: 2021-07-31T01:35:24.979 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 131 k 2021-07-31T01:35:24.979 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 432 k 2021-07-31T01:35:24.980 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2021-07-31T01:35:25.148 INFO:teuthology.orchestra.run.smithi154.stdout:librados-devel-16.2.5-150.g5a784c87.el8.x86_64. 765 kB/s | 131 kB 00:00 2021-07-31T01:35:25.149 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2021-07-31T01:35:25.149 INFO:teuthology.orchestra.run.smithi154.stdout:Total 760 kB/s | 131 kB 00:00 2021-07-31T01:35:25.150 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2021-07-31T01:35:25.156 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2021-07-31T01:35:25.156 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2021-07-31T01:35:25.210 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2021-07-31T01:35:25.213 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2021-07-31T01:35:25.334 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2021-07-31T01:35:25.531 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : librados-devel-2:16.2.5-150.g5a784c87.el8.x86_64 1/1 2021-07-31T01:35:25.756 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: librados-devel-2:16.2.5-150.g5a784c87.el8.x86_64 1/1 2021-07-31T01:35:26.256 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librados-devel-2:16.2.5-150.g5a784c87.el8.x86_64 1/1 2021-07-31T01:35:26.257 INFO:teuthology.orchestra.run.smithi154.stdout:Installed products updated. 2021-07-31T01:35:28.860 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:05 ago on Sat 31 Jul 2021 01:34:56 AM UTC. 2021-07-31T01:35:31.182 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:35:31.183 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2021-07-31T01:35:31.183 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-fuse-2:16.2.5-150.g5a784c87.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2021-07-31T01:35:31.183 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:35:31.184 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2021-07-31T01:35:31.261 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install librados-devel 2021-07-31T01:35:31.573 INFO:teuthology.orchestra.run.smithi045.stdout:Updating Subscription Management repositories. 2021-07-31T01:35:32.143 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 76 kB/s | 2.8 kB 00:00 2021-07-31T01:35:36.222 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:00:05 ago on Sat 31 Jul 2021 01:35:06 AM UTC. 2021-07-31T01:35:36.355 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 64 kB/s | 2.4 kB 00:00 2021-07-31T01:35:38.416 INFO:teuthology.orchestra.run.smithi154.stdout: 2021-07-31T01:35:38.416 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2021-07-31T01:35:38.417 INFO:teuthology.orchestra.run.smithi154.stdout: librados-devel-2:16.2.5-150.g5a784c87.el8.x86_64 2021-07-31T01:35:38.417 INFO:teuthology.orchestra.run.smithi154.stdout: 2021-07-31T01:35:38.417 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2021-07-31T01:35:38.496 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install libcephfs2 2021-07-31T01:35:38.785 INFO:teuthology.orchestra.run.smithi154.stdout:Updating Subscription Management repositories. 2021-07-31T01:35:39.285 INFO:teuthology.orchestra.run.smithi154.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 97 kB/s | 2.8 kB 00:00 2021-07-31T01:35:40.843 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:04 ago on Sat 31 Jul 2021 01:29:23 AM UTC. 2021-07-31T01:35:43.087 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2021-07-31T01:35:43.089 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2021-07-31T01:35:43.089 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repo Size 2021-07-31T01:35:43.089 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2021-07-31T01:35:43.090 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2021-07-31T01:35:43.090 INFO:teuthology.orchestra.run.smithi045.stdout: librados-devel x86_64 2:16.2.5-150.g5a784c87.el8 ceph 131 k 2021-07-31T01:35:43.090 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:35:43.090 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2021-07-31T01:35:43.091 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2021-07-31T01:35:43.091 INFO:teuthology.orchestra.run.smithi045.stdout:Install 1 Package 2021-07-31T01:35:43.091 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:35:43.092 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 131 k 2021-07-31T01:35:43.092 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 432 k 2021-07-31T01:35:43.092 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2021-07-31T01:35:43.259 INFO:teuthology.orchestra.run.smithi045.stdout:librados-devel-16.2.5-150.g5a784c87.el8.x86_64. 773 kB/s | 131 kB 00:00 2021-07-31T01:35:43.260 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2021-07-31T01:35:43.260 INFO:teuthology.orchestra.run.smithi045.stdout:Total 767 kB/s | 131 kB 00:00 2021-07-31T01:35:43.261 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2021-07-31T01:35:43.267 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2021-07-31T01:35:43.267 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2021-07-31T01:35:43.328 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2021-07-31T01:35:43.332 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2021-07-31T01:35:43.468 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2021-07-31T01:35:43.491 INFO:teuthology.orchestra.run.smithi154.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 72 kB/s | 2.4 kB 00:00 2021-07-31T01:35:43.665 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : librados-devel-2:16.2.5-150.g5a784c87.el8.x86_64 1/1 2021-07-31T01:35:43.893 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librados-devel-2:16.2.5-150.g5a784c87.el8.x86_64 1/1 2021-07-31T01:35:44.396 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librados-devel-2:16.2.5-150.g5a784c87.el8.x86_64 1/1 2021-07-31T01:35:44.396 INFO:teuthology.orchestra.run.smithi045.stdout:Installed products updated. 2021-07-31T01:35:47.832 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:00:04 ago on Sat 31 Jul 2021 01:29:23 AM UTC. 2021-07-31T01:35:49.586 INFO:teuthology.orchestra.run.smithi154.stdout:Package libcephfs2-2:16.2.5-150.g5a784c87.el8.x86_64 is already installed. 2021-07-31T01:35:49.927 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2021-07-31T01:35:49.928 INFO:teuthology.orchestra.run.smithi154.stdout:Nothing to do. 2021-07-31T01:35:49.928 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2021-07-31T01:35:50.016 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install libcephfs-devel 2021-07-31T01:35:50.304 INFO:teuthology.orchestra.run.smithi154.stdout:Updating Subscription Management repositories. 2021-07-31T01:35:50.795 INFO:teuthology.orchestra.run.smithi154.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 94 kB/s | 2.8 kB 00:00 2021-07-31T01:35:50.944 INFO:teuthology.orchestra.run.smithi154.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 79 kB/s | 2.4 kB 00:00 2021-07-31T01:35:52.478 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2021-07-31T01:35:52.479 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2021-07-31T01:35:52.480 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repo Size 2021-07-31T01:35:52.480 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2021-07-31T01:35:52.480 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2021-07-31T01:35:52.480 INFO:teuthology.orchestra.run.smithi154.stdout: libcephfs-devel x86_64 2:16.2.5-150.g5a784c87.el8 ceph 25 k 2021-07-31T01:35:52.481 INFO:teuthology.orchestra.run.smithi154.stdout: 2021-07-31T01:35:52.481 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2021-07-31T01:35:52.481 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2021-07-31T01:35:52.481 INFO:teuthology.orchestra.run.smithi154.stdout:Install 1 Package 2021-07-31T01:35:52.481 INFO:teuthology.orchestra.run.smithi154.stdout: 2021-07-31T01:35:52.482 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 25 k 2021-07-31T01:35:52.482 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 101 k 2021-07-31T01:35:52.482 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2021-07-31T01:35:52.630 INFO:teuthology.orchestra.run.smithi154.stdout:libcephfs-devel-16.2.5-150.g5a784c87.el8.x86_64 166 kB/s | 25 kB 00:00 2021-07-31T01:35:52.631 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2021-07-31T01:35:52.631 INFO:teuthology.orchestra.run.smithi154.stdout:Total 165 kB/s | 25 kB 00:00 2021-07-31T01:35:52.631 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2021-07-31T01:35:52.635 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2021-07-31T01:35:52.635 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2021-07-31T01:35:52.637 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2021-07-31T01:35:52.640 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2021-07-31T01:35:52.699 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2021-07-31T01:35:52.867 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libcephfs-devel-2:16.2.5-150.g5a784c87.el8.x86_64 1/1 2021-07-31T01:35:53.093 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: libcephfs-devel-2:16.2.5-150.g5a784c87.el8.x86_64 1/1 2021-07-31T01:35:53.531 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libcephfs-devel-2:16.2.5-150.g5a784c87.el8.x86_64 1/1 2021-07-31T01:35:53.531 INFO:teuthology.orchestra.run.smithi154.stdout:Installed products updated. 2021-07-31T01:35:54.929 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:05 ago on Sat 31 Jul 2021 01:35:23 AM UTC. 2021-07-31T01:35:57.244 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:35:57.244 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2021-07-31T01:35:57.244 INFO:teuthology.orchestra.run.smithi045.stdout: librados-devel-2:16.2.5-150.g5a784c87.el8.x86_64 2021-07-31T01:35:57.245 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:35:57.245 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2021-07-31T01:35:57.323 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install libcephfs2 2021-07-31T01:35:57.631 INFO:teuthology.orchestra.run.smithi045.stdout:Updating Subscription Management repositories. 2021-07-31T01:35:58.204 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 74 kB/s | 2.8 kB 00:00 2021-07-31T01:36:02.565 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 63 kB/s | 2.4 kB 00:00 2021-07-31T01:36:03.801 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:00:05 ago on Sat 31 Jul 2021 01:35:31 AM UTC. 2021-07-31T01:36:05.979 INFO:teuthology.orchestra.run.smithi154.stdout: 2021-07-31T01:36:05.980 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2021-07-31T01:36:05.980 INFO:teuthology.orchestra.run.smithi154.stdout: libcephfs-devel-2:16.2.5-150.g5a784c87.el8.x86_64 2021-07-31T01:36:05.980 INFO:teuthology.orchestra.run.smithi154.stdout: 2021-07-31T01:36:05.980 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2021-07-31T01:36:06.057 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install librados2 2021-07-31T01:36:06.347 INFO:teuthology.orchestra.run.smithi154.stdout:Updating Subscription Management repositories. 2021-07-31T01:36:06.857 INFO:teuthology.orchestra.run.smithi154.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 95 kB/s | 2.8 kB 00:00 2021-07-31T01:36:07.221 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:04 ago on Sat 31 Jul 2021 01:29:23 AM UTC. 2021-07-31T01:36:09.070 INFO:teuthology.orchestra.run.smithi045.stdout:Package libcephfs2-2:16.2.5-150.g5a784c87.el8.x86_64 is already installed. 2021-07-31T01:36:09.430 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2021-07-31T01:36:09.431 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2021-07-31T01:36:09.431 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2021-07-31T01:36:09.514 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install libcephfs-devel 2021-07-31T01:36:09.821 INFO:teuthology.orchestra.run.smithi045.stdout:Updating Subscription Management repositories. 2021-07-31T01:36:10.419 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 75 kB/s | 2.8 kB 00:00 2021-07-31T01:36:10.582 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 64 kB/s | 2.4 kB 00:00 2021-07-31T01:36:10.885 INFO:teuthology.orchestra.run.smithi154.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 80 kB/s | 2.4 kB 00:00 2021-07-31T01:36:12.242 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2021-07-31T01:36:12.243 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2021-07-31T01:36:12.243 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repo Size 2021-07-31T01:36:12.244 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2021-07-31T01:36:12.244 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2021-07-31T01:36:12.244 INFO:teuthology.orchestra.run.smithi045.stdout: libcephfs-devel x86_64 2:16.2.5-150.g5a784c87.el8 ceph 25 k 2021-07-31T01:36:12.245 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:36:12.245 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2021-07-31T01:36:12.245 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2021-07-31T01:36:12.245 INFO:teuthology.orchestra.run.smithi045.stdout:Install 1 Package 2021-07-31T01:36:12.246 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:36:12.246 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 25 k 2021-07-31T01:36:12.246 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 101 k 2021-07-31T01:36:12.246 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2021-07-31T01:36:12.368 INFO:teuthology.orchestra.run.smithi045.stdout:libcephfs-devel-16.2.5-150.g5a784c87.el8.x86_64 200 kB/s | 25 kB 00:00 2021-07-31T01:36:12.369 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2021-07-31T01:36:12.369 INFO:teuthology.orchestra.run.smithi045.stdout:Total 198 kB/s | 25 kB 00:00 2021-07-31T01:36:12.369 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2021-07-31T01:36:12.373 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2021-07-31T01:36:12.374 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2021-07-31T01:36:12.376 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2021-07-31T01:36:12.379 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2021-07-31T01:36:12.429 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2021-07-31T01:36:12.635 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libcephfs-devel-2:16.2.5-150.g5a784c87.el8.x86_64 1/1 2021-07-31T01:36:12.843 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: libcephfs-devel-2:16.2.5-150.g5a784c87.el8.x86_64 1/1 2021-07-31T01:36:13.346 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libcephfs-devel-2:16.2.5-150.g5a784c87.el8.x86_64 1/1 2021-07-31T01:36:13.346 INFO:teuthology.orchestra.run.smithi045.stdout:Installed products updated. 2021-07-31T01:36:15.122 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:00:04 ago on Sat 31 Jul 2021 01:29:23 AM UTC. 2021-07-31T01:36:16.890 INFO:teuthology.orchestra.run.smithi154.stdout:Package librados2-2:16.2.5-150.g5a784c87.el8.x86_64 is already installed. 2021-07-31T01:36:17.240 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2021-07-31T01:36:17.241 INFO:teuthology.orchestra.run.smithi154.stdout:Nothing to do. 2021-07-31T01:36:17.241 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2021-07-31T01:36:17.312 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install librbd1 2021-07-31T01:36:17.599 INFO:teuthology.orchestra.run.smithi154.stdout:Updating Subscription Management repositories. 2021-07-31T01:36:18.153 INFO:teuthology.orchestra.run.smithi154.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 93 kB/s | 2.8 kB 00:00 2021-07-31T01:36:18.303 INFO:teuthology.orchestra.run.smithi154.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 77 kB/s | 2.4 kB 00:00 2021-07-31T01:36:19.514 INFO:teuthology.orchestra.run.smithi154.stdout:Package librbd1-2:16.2.5-150.g5a784c87.el8.x86_64 is already installed. 2021-07-31T01:36:19.866 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2021-07-31T01:36:19.867 INFO:teuthology.orchestra.run.smithi154.stdout:Nothing to do. 2021-07-31T01:36:19.867 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2021-07-31T01:36:19.940 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install python3-rados 2021-07-31T01:36:20.230 INFO:teuthology.orchestra.run.smithi154.stdout:Updating Subscription Management repositories. 2021-07-31T01:36:20.749 INFO:teuthology.orchestra.run.smithi154.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 100 kB/s | 2.8 kB 00:00 2021-07-31T01:36:20.900 INFO:teuthology.orchestra.run.smithi154.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 85 kB/s | 2.4 kB 00:00 2021-07-31T01:36:22.064 INFO:teuthology.orchestra.run.smithi154.stdout:Package python3-rados-2:16.2.5-150.g5a784c87.el8.x86_64 is already installed. 2021-07-31T01:36:22.408 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2021-07-31T01:36:22.409 INFO:teuthology.orchestra.run.smithi154.stdout:Nothing to do. 2021-07-31T01:36:22.410 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2021-07-31T01:36:22.482 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install python3-rgw 2021-07-31T01:36:22.771 INFO:teuthology.orchestra.run.smithi154.stdout:Updating Subscription Management repositories. 2021-07-31T01:36:23.295 INFO:teuthology.orchestra.run.smithi154.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 95 kB/s | 2.8 kB 00:00 2021-07-31T01:36:23.444 INFO:teuthology.orchestra.run.smithi154.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 79 kB/s | 2.4 kB 00:00 2021-07-31T01:36:24.373 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:05 ago on Sat 31 Jul 2021 01:35:49 AM UTC. 2021-07-31T01:36:24.617 INFO:teuthology.orchestra.run.smithi154.stdout:Package python3-rgw-2:16.2.5-150.g5a784c87.el8.x86_64 is already installed. 2021-07-31T01:36:24.961 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2021-07-31T01:36:24.961 INFO:teuthology.orchestra.run.smithi154.stdout:Nothing to do. 2021-07-31T01:36:24.962 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2021-07-31T01:36:25.028 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install python3-cephfs 2021-07-31T01:36:25.316 INFO:teuthology.orchestra.run.smithi154.stdout:Updating Subscription Management repositories. 2021-07-31T01:36:25.816 INFO:teuthology.orchestra.run.smithi154.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 93 kB/s | 2.8 kB 00:00 2021-07-31T01:36:25.964 INFO:teuthology.orchestra.run.smithi154.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 81 kB/s | 2.4 kB 00:00 2021-07-31T01:36:26.768 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:36:26.768 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2021-07-31T01:36:26.769 INFO:teuthology.orchestra.run.smithi045.stdout: libcephfs-devel-2:16.2.5-150.g5a784c87.el8.x86_64 2021-07-31T01:36:26.769 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:36:26.769 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2021-07-31T01:36:26.883 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install librados2 2021-07-31T01:36:27.121 INFO:teuthology.orchestra.run.smithi154.stdout:Package python3-cephfs-2:16.2.5-150.g5a784c87.el8.x86_64 is already installed. 2021-07-31T01:36:27.192 INFO:teuthology.orchestra.run.smithi045.stdout:Updating Subscription Management repositories. 2021-07-31T01:36:27.464 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2021-07-31T01:36:27.465 INFO:teuthology.orchestra.run.smithi154.stdout:Nothing to do. 2021-07-31T01:36:27.465 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2021-07-31T01:36:27.555 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install python3-rbd 2021-07-31T01:36:27.775 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 75 kB/s | 2.8 kB 00:00 2021-07-31T01:36:27.842 INFO:teuthology.orchestra.run.smithi154.stdout:Updating Subscription Management repositories. 2021-07-31T01:36:28.359 INFO:teuthology.orchestra.run.smithi154.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 97 kB/s | 2.8 kB 00:00 2021-07-31T01:36:28.507 INFO:teuthology.orchestra.run.smithi154.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 82 kB/s | 2.4 kB 00:00 2021-07-31T01:36:29.696 INFO:teuthology.orchestra.run.smithi154.stdout:Package python3-rbd-2:16.2.5-150.g5a784c87.el8.x86_64 is already installed. 2021-07-31T01:36:30.041 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2021-07-31T01:36:30.042 INFO:teuthology.orchestra.run.smithi154.stdout:Nothing to do. 2021-07-31T01:36:30.043 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2021-07-31T01:36:30.102 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install rbd-fuse 2021-07-31T01:36:30.391 INFO:teuthology.orchestra.run.smithi154.stdout:Updating Subscription Management repositories. 2021-07-31T01:36:30.898 INFO:teuthology.orchestra.run.smithi154.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 96 kB/s | 2.8 kB 00:00 2021-07-31T01:36:31.046 INFO:teuthology.orchestra.run.smithi154.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 82 kB/s | 2.4 kB 00:00 2021-07-31T01:36:31.970 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 63 kB/s | 2.4 kB 00:00 2021-07-31T01:36:32.607 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2021-07-31T01:36:32.608 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2021-07-31T01:36:32.608 INFO:teuthology.orchestra.run.smithi154.stdout: Package Architecture Version Repository Size 2021-07-31T01:36:32.608 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2021-07-31T01:36:32.609 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2021-07-31T01:36:32.609 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-fuse x86_64 2:16.2.5-150.g5a784c87.el8 ceph 88 k 2021-07-31T01:36:32.609 INFO:teuthology.orchestra.run.smithi154.stdout: 2021-07-31T01:36:32.609 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2021-07-31T01:36:32.610 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2021-07-31T01:36:32.610 INFO:teuthology.orchestra.run.smithi154.stdout:Install 1 Package 2021-07-31T01:36:32.610 INFO:teuthology.orchestra.run.smithi154.stdout: 2021-07-31T01:36:32.610 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 88 k 2021-07-31T01:36:32.611 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 236 k 2021-07-31T01:36:32.611 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2021-07-31T01:36:32.772 INFO:teuthology.orchestra.run.smithi154.stdout:rbd-fuse-16.2.5-150.g5a784c87.el8.x86_64.rpm 536 kB/s | 88 kB 00:00 2021-07-31T01:36:32.773 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2021-07-31T01:36:32.774 INFO:teuthology.orchestra.run.smithi154.stdout:Total 532 kB/s | 88 kB 00:00 2021-07-31T01:36:32.774 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2021-07-31T01:36:32.785 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2021-07-31T01:36:32.785 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2021-07-31T01:36:32.842 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2021-07-31T01:36:32.845 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2021-07-31T01:36:32.960 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2021-07-31T01:36:33.185 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : rbd-fuse-2:16.2.5-150.g5a784c87.el8.x86_64 1/1 2021-07-31T01:36:33.413 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: rbd-fuse-2:16.2.5-150.g5a784c87.el8.x86_64 1/1 2021-07-31T01:36:33.865 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : rbd-fuse-2:16.2.5-150.g5a784c87.el8.x86_64 1/1 2021-07-31T01:36:33.866 INFO:teuthology.orchestra.run.smithi154.stdout:Installed products updated. 2021-07-31T01:36:36.438 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:04 ago on Sat 31 Jul 2021 01:29:23 AM UTC. 2021-07-31T01:36:38.347 INFO:teuthology.orchestra.run.smithi045.stdout:Package librados2-2:16.2.5-150.g5a784c87.el8.x86_64 is already installed. 2021-07-31T01:36:38.711 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2021-07-31T01:36:38.712 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2021-07-31T01:36:38.712 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2021-07-31T01:36:38.782 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install librbd1 2021-07-31T01:36:39.091 INFO:teuthology.orchestra.run.smithi045.stdout:Updating Subscription Management repositories. 2021-07-31T01:36:39.640 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 75 kB/s | 2.8 kB 00:00 2021-07-31T01:36:39.801 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 66 kB/s | 2.4 kB 00:00 2021-07-31T01:36:41.085 INFO:teuthology.orchestra.run.smithi045.stdout:Package librbd1-2:16.2.5-150.g5a784c87.el8.x86_64 is already installed. 2021-07-31T01:36:41.453 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2021-07-31T01:36:41.454 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2021-07-31T01:36:41.455 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2021-07-31T01:36:41.523 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install python3-rados 2021-07-31T01:36:41.829 INFO:teuthology.orchestra.run.smithi045.stdout:Updating Subscription Management repositories. 2021-07-31T01:36:42.440 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 75 kB/s | 2.8 kB 00:00 2021-07-31T01:36:42.602 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 64 kB/s | 2.4 kB 00:00 2021-07-31T01:36:43.829 INFO:teuthology.orchestra.run.smithi045.stdout:Package python3-rados-2:16.2.5-150.g5a784c87.el8.x86_64 is already installed. 2021-07-31T01:36:44.195 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2021-07-31T01:36:44.196 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2021-07-31T01:36:44.196 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2021-07-31T01:36:44.268 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:00:05 ago on Sat 31 Jul 2021 01:35:58 AM UTC. 2021-07-31T01:36:44.270 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install python3-rgw 2021-07-31T01:36:44.577 INFO:teuthology.orchestra.run.smithi045.stdout:Updating Subscription Management repositories. 2021-07-31T01:36:45.172 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 75 kB/s | 2.8 kB 00:00 2021-07-31T01:36:45.327 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 64 kB/s | 2.4 kB 00:00 2021-07-31T01:36:46.532 INFO:teuthology.orchestra.run.smithi154.stdout: 2021-07-31T01:36:46.533 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2021-07-31T01:36:46.533 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-fuse-2:16.2.5-150.g5a784c87.el8.x86_64 2021-07-31T01:36:46.533 INFO:teuthology.orchestra.run.smithi154.stdout: 2021-07-31T01:36:46.534 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2021-07-31T01:36:46.570 INFO:teuthology.orchestra.run.smithi045.stdout:Package python3-rgw-2:16.2.5-150.g5a784c87.el8.x86_64 is already installed. 2021-07-31T01:36:46.612 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install rbd-mirror 2021-07-31T01:36:46.901 INFO:teuthology.orchestra.run.smithi154.stdout:Updating Subscription Management repositories. 2021-07-31T01:36:46.945 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2021-07-31T01:36:46.946 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2021-07-31T01:36:46.946 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2021-07-31T01:36:47.021 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install python3-cephfs 2021-07-31T01:36:47.327 INFO:teuthology.orchestra.run.smithi045.stdout:Updating Subscription Management repositories. 2021-07-31T01:36:47.424 INFO:teuthology.orchestra.run.smithi154.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 94 kB/s | 2.8 kB 00:00 2021-07-31T01:36:47.897 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 73 kB/s | 2.8 kB 00:00 2021-07-31T01:36:48.060 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 62 kB/s | 2.4 kB 00:00 2021-07-31T01:36:49.345 INFO:teuthology.orchestra.run.smithi045.stdout:Package python3-cephfs-2:16.2.5-150.g5a784c87.el8.x86_64 is already installed. 2021-07-31T01:36:49.705 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2021-07-31T01:36:49.707 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2021-07-31T01:36:49.707 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2021-07-31T01:36:49.813 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install python3-rbd 2021-07-31T01:36:50.121 INFO:teuthology.orchestra.run.smithi045.stdout:Updating Subscription Management repositories. 2021-07-31T01:36:50.713 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 74 kB/s | 2.8 kB 00:00 2021-07-31T01:36:50.876 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 63 kB/s | 2.4 kB 00:00 2021-07-31T01:36:51.576 INFO:teuthology.orchestra.run.smithi154.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 83 kB/s | 2.4 kB 00:00 2021-07-31T01:36:52.144 INFO:teuthology.orchestra.run.smithi045.stdout:Package python3-rbd-2:16.2.5-150.g5a784c87.el8.x86_64 is already installed. 2021-07-31T01:36:52.507 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2021-07-31T01:36:52.509 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2021-07-31T01:36:52.510 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2021-07-31T01:36:52.595 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install rbd-fuse 2021-07-31T01:36:52.903 INFO:teuthology.orchestra.run.smithi045.stdout:Updating Subscription Management repositories. 2021-07-31T01:36:53.495 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 59 kB/s | 2.8 kB 00:00 2021-07-31T01:36:53.658 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 62 kB/s | 2.4 kB 00:00 2021-07-31T01:36:55.315 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2021-07-31T01:36:55.316 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2021-07-31T01:36:55.316 INFO:teuthology.orchestra.run.smithi045.stdout: Package Architecture Version Repository Size 2021-07-31T01:36:55.317 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2021-07-31T01:36:55.317 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2021-07-31T01:36:55.317 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-fuse x86_64 2:16.2.5-150.g5a784c87.el8 ceph 88 k 2021-07-31T01:36:55.317 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:36:55.318 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2021-07-31T01:36:55.318 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2021-07-31T01:36:55.318 INFO:teuthology.orchestra.run.smithi045.stdout:Install 1 Package 2021-07-31T01:36:55.318 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:36:55.320 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 88 k 2021-07-31T01:36:55.321 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 236 k 2021-07-31T01:36:55.321 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2021-07-31T01:36:55.611 INFO:teuthology.orchestra.run.smithi045.stdout:rbd-fuse-16.2.5-150.g5a784c87.el8.x86_64.rpm 298 kB/s | 88 kB 00:00 2021-07-31T01:36:55.612 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2021-07-31T01:36:55.612 INFO:teuthology.orchestra.run.smithi045.stdout:Total 297 kB/s | 88 kB 00:00 2021-07-31T01:36:55.613 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2021-07-31T01:36:55.625 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2021-07-31T01:36:55.625 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2021-07-31T01:36:55.685 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2021-07-31T01:36:55.689 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2021-07-31T01:36:55.811 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2021-07-31T01:36:55.954 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:00:04 ago on Sat 31 Jul 2021 01:29:23 AM UTC. 2021-07-31T01:36:56.045 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : rbd-fuse-2:16.2.5-150.g5a784c87.el8.x86_64 1/1 2021-07-31T01:36:56.302 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: rbd-fuse-2:16.2.5-150.g5a784c87.el8.x86_64 1/1 2021-07-31T01:36:56.830 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : rbd-fuse-2:16.2.5-150.g5a784c87.el8.x86_64 1/1 2021-07-31T01:36:56.831 INFO:teuthology.orchestra.run.smithi045.stdout:Installed products updated. 2021-07-31T01:36:58.116 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2021-07-31T01:36:58.117 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2021-07-31T01:36:58.118 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repo Size 2021-07-31T01:36:58.118 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2021-07-31T01:36:58.118 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2021-07-31T01:36:58.118 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-mirror x86_64 2:16.2.5-150.g5a784c87.el8 ceph 3.7 M 2021-07-31T01:36:58.119 INFO:teuthology.orchestra.run.smithi154.stdout: 2021-07-31T01:36:58.119 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2021-07-31T01:36:58.119 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2021-07-31T01:36:58.119 INFO:teuthology.orchestra.run.smithi154.stdout:Install 1 Package 2021-07-31T01:36:58.119 INFO:teuthology.orchestra.run.smithi154.stdout: 2021-07-31T01:36:58.120 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 3.7 M 2021-07-31T01:36:58.120 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 15 M 2021-07-31T01:36:58.120 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2021-07-31T01:36:58.483 INFO:teuthology.orchestra.run.smithi154.stdout:rbd-mirror-16.2.5-150.g5a784c87.el8.x86_64.rpm 10 MB/s | 3.7 MB 00:00 2021-07-31T01:36:58.483 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2021-07-31T01:36:58.484 INFO:teuthology.orchestra.run.smithi154.stdout:Total 10 MB/s | 3.7 MB 00:00 2021-07-31T01:36:58.484 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2021-07-31T01:36:58.498 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2021-07-31T01:36:58.498 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2021-07-31T01:36:58.568 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2021-07-31T01:36:58.571 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2021-07-31T01:36:59.008 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2021-07-31T01:36:59.191 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : rbd-mirror-2:16.2.5-150.g5a784c87.el8.x86_64 1/1 2021-07-31T01:36:59.551 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: rbd-mirror-2:16.2.5-150.g5a784c87.el8.x86_64 1/1 2021-07-31T01:37:00.015 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : rbd-mirror-2:16.2.5-150.g5a784c87.el8.x86_64 1/1 2021-07-31T01:37:00.016 INFO:teuthology.orchestra.run.smithi154.stdout:Installed products updated. 2021-07-31T01:37:07.816 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:05 ago on Sat 31 Jul 2021 01:36:19 AM UTC. 2021-07-31T01:37:10.276 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:00:05 ago on Sat 31 Jul 2021 01:36:39 AM UTC. 2021-07-31T01:37:10.286 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:37:10.286 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2021-07-31T01:37:10.286 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-fuse-2:16.2.5-150.g5a784c87.el8.x86_64 2021-07-31T01:37:10.287 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:37:10.287 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2021-07-31T01:37:10.379 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install rbd-mirror 2021-07-31T01:37:10.686 INFO:teuthology.orchestra.run.smithi045.stdout:Updating Subscription Management repositories. 2021-07-31T01:37:11.252 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 77 kB/s | 2.8 kB 00:00 2021-07-31T01:37:12.467 INFO:teuthology.orchestra.run.smithi154.stdout: 2021-07-31T01:37:12.467 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2021-07-31T01:37:12.468 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-mirror-2:16.2.5-150.g5a784c87.el8.x86_64 2021-07-31T01:37:12.468 INFO:teuthology.orchestra.run.smithi154.stdout: 2021-07-31T01:37:12.468 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2021-07-31T01:37:12.576 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install rbd-nbd 2021-07-31T01:37:12.870 INFO:teuthology.orchestra.run.smithi154.stdout:Updating Subscription Management repositories. 2021-07-31T01:37:13.395 INFO:teuthology.orchestra.run.smithi154.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 100 kB/s | 2.8 kB 00:00 2021-07-31T01:37:15.539 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 63 kB/s | 2.4 kB 00:00 2021-07-31T01:37:17.497 INFO:teuthology.orchestra.run.smithi154.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 78 kB/s | 2.4 kB 00:00 2021-07-31T01:37:20.110 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:04 ago on Sat 31 Jul 2021 01:29:23 AM UTC. 2021-07-31T01:37:21.827 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:00:04 ago on Sat 31 Jul 2021 01:29:23 AM UTC. 2021-07-31T01:37:22.380 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2021-07-31T01:37:22.381 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2021-07-31T01:37:22.382 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repo Size 2021-07-31T01:37:22.382 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2021-07-31T01:37:22.382 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2021-07-31T01:37:22.382 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-mirror x86_64 2:16.2.5-150.g5a784c87.el8 ceph 3.7 M 2021-07-31T01:37:22.383 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:37:22.383 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2021-07-31T01:37:22.383 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2021-07-31T01:37:22.383 INFO:teuthology.orchestra.run.smithi045.stdout:Install 1 Package 2021-07-31T01:37:22.384 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:37:22.384 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 3.7 M 2021-07-31T01:37:22.384 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 15 M 2021-07-31T01:37:22.384 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2021-07-31T01:37:23.988 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2021-07-31T01:37:23.989 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2021-07-31T01:37:23.989 INFO:teuthology.orchestra.run.smithi154.stdout: Package Architecture Version Repository Size 2021-07-31T01:37:23.989 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2021-07-31T01:37:23.990 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2021-07-31T01:37:23.990 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-nbd x86_64 2:16.2.5-150.g5a784c87.el8 ceph 172 k 2021-07-31T01:37:23.990 INFO:teuthology.orchestra.run.smithi154.stdout: 2021-07-31T01:37:23.990 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2021-07-31T01:37:23.991 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2021-07-31T01:37:23.991 INFO:teuthology.orchestra.run.smithi154.stdout:Install 1 Package 2021-07-31T01:37:23.991 INFO:teuthology.orchestra.run.smithi154.stdout: 2021-07-31T01:37:23.991 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 172 k 2021-07-31T01:37:23.992 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 511 k 2021-07-31T01:37:23.992 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2021-07-31T01:37:24.206 INFO:teuthology.orchestra.run.smithi154.stdout:rbd-nbd-16.2.5-150.g5a784c87.el8.x86_64.rpm 793 kB/s | 172 kB 00:00 2021-07-31T01:37:24.207 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2021-07-31T01:37:24.207 INFO:teuthology.orchestra.run.smithi154.stdout:Total 788 kB/s | 172 kB 00:00 2021-07-31T01:37:24.207 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2021-07-31T01:37:24.219 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2021-07-31T01:37:24.220 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2021-07-31T01:37:24.275 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2021-07-31T01:37:24.278 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2021-07-31T01:37:24.401 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2021-07-31T01:37:24.609 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : rbd-nbd-2:16.2.5-150.g5a784c87.el8.x86_64 1/1 2021-07-31T01:37:24.832 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: rbd-nbd-2:16.2.5-150.g5a784c87.el8.x86_64 1/1 2021-07-31T01:37:25.336 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : rbd-nbd-2:16.2.5-150.g5a784c87.el8.x86_64 1/1 2021-07-31T01:37:25.337 INFO:teuthology.orchestra.run.smithi154.stdout:Installed products updated. 2021-07-31T01:37:25.350 INFO:teuthology.orchestra.run.smithi045.stdout:rbd-mirror-16.2.5-150.g5a784c87.el8.x86_64.rpm 1.2 MB/s | 3.7 MB 00:02 2021-07-31T01:37:25.351 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2021-07-31T01:37:25.352 INFO:teuthology.orchestra.run.smithi045.stdout:Total 1.2 MB/s | 3.7 MB 00:02 2021-07-31T01:37:25.352 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2021-07-31T01:37:25.370 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2021-07-31T01:37:25.370 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2021-07-31T01:37:25.448 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2021-07-31T01:37:25.452 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2021-07-31T01:37:25.923 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2021-07-31T01:37:26.085 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : rbd-mirror-2:16.2.5-150.g5a784c87.el8.x86_64 1/1 2021-07-31T01:37:26.479 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: rbd-mirror-2:16.2.5-150.g5a784c87.el8.x86_64 1/1 2021-07-31T01:37:26.974 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : rbd-mirror-2:16.2.5-150.g5a784c87.el8.x86_64 1/1 2021-07-31T01:37:26.975 INFO:teuthology.orchestra.run.smithi045.stdout:Installed products updated. 2021-07-31T01:37:35.592 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:00:05 ago on Sat 31 Jul 2021 01:37:05 AM UTC. 2021-07-31T01:37:37.830 INFO:teuthology.orchestra.run.smithi154.stdout: 2021-07-31T01:37:37.831 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2021-07-31T01:37:37.831 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-nbd-2:16.2.5-150.g5a784c87.el8.x86_64 2021-07-31T01:37:37.832 INFO:teuthology.orchestra.run.smithi154.stdout: 2021-07-31T01:37:37.832 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2021-07-31T01:37:37.901 DEBUG:teuthology.parallel:result is None 2021-07-31T01:37:37.943 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:05 ago on Sat 31 Jul 2021 01:37:02 AM UTC. 2021-07-31T01:37:40.345 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:37:40.345 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2021-07-31T01:37:40.345 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-mirror-2:16.2.5-150.g5a784c87.el8.x86_64 2021-07-31T01:37:40.345 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:37:40.346 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2021-07-31T01:37:40.467 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install rbd-nbd 2021-07-31T01:37:40.786 INFO:teuthology.orchestra.run.smithi045.stdout:Updating Subscription Management repositories. 2021-07-31T01:37:41.396 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 74 kB/s | 2.8 kB 00:00 2021-07-31T01:37:45.615 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 62 kB/s | 2.4 kB 00:00 2021-07-31T01:37:49.990 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:04 ago on Sat 31 Jul 2021 01:29:23 AM UTC. 2021-07-31T01:37:52.243 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2021-07-31T01:37:52.244 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2021-07-31T01:37:52.245 INFO:teuthology.orchestra.run.smithi045.stdout: Package Architecture Version Repository Size 2021-07-31T01:37:52.245 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2021-07-31T01:37:52.245 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2021-07-31T01:37:52.245 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-nbd x86_64 2:16.2.5-150.g5a784c87.el8 ceph 172 k 2021-07-31T01:37:52.246 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:37:52.246 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2021-07-31T01:37:52.246 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2021-07-31T01:37:52.246 INFO:teuthology.orchestra.run.smithi045.stdout:Install 1 Package 2021-07-31T01:37:52.247 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:37:52.247 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 172 k 2021-07-31T01:37:52.247 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 511 k 2021-07-31T01:37:52.247 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2021-07-31T01:37:52.415 INFO:teuthology.orchestra.run.smithi045.stdout:rbd-nbd-16.2.5-150.g5a784c87.el8.x86_64.rpm 1.0 MB/s | 172 kB 00:00 2021-07-31T01:37:52.416 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2021-07-31T01:37:52.417 INFO:teuthology.orchestra.run.smithi045.stdout:Total 997 kB/s | 172 kB 00:00 2021-07-31T01:37:52.417 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2021-07-31T01:37:52.430 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2021-07-31T01:37:52.431 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2021-07-31T01:37:52.488 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2021-07-31T01:37:52.492 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2021-07-31T01:37:52.685 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2021-07-31T01:37:53.003 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : rbd-nbd-2:16.2.5-150.g5a784c87.el8.x86_64 1/1 2021-07-31T01:37:53.248 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: rbd-nbd-2:16.2.5-150.g5a784c87.el8.x86_64 1/1 2021-07-31T01:37:53.773 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : rbd-nbd-2:16.2.5-150.g5a784c87.el8.x86_64 1/1 2021-07-31T01:37:53.774 INFO:teuthology.orchestra.run.smithi045.stdout:Installed products updated. 2021-07-31T01:38:04.418 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:05 ago on Sat 31 Jul 2021 01:37:32 AM UTC. 2021-07-31T01:38:06.712 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:38:06.712 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2021-07-31T01:38:06.713 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-nbd-2:16.2.5-150.g5a784c87.el8.x86_64 2021-07-31T01:38:06.713 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:38:06.713 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2021-07-31T01:38:06.835 DEBUG:teuthology.parallel:result is None 2021-07-31T01:38:06.836 DEBUG:teuthology.orchestra.run.smithi045:> sudo lsb_release -is 2021-07-31T01:38:06.883 INFO:teuthology.orchestra.run.smithi045.stdout:RedHatEnterprise 2021-07-31T01:38:06.884 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2021-07-31T01:38:06.885 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2021-07-31T01:38:06.885 INFO:teuthology.packaging:ref: None 2021-07-31T01:38:06.885 INFO:teuthology.packaging:tag: None 2021-07-31T01:38:06.886 INFO:teuthology.packaging:branch: pacific 2021-07-31T01:38:06.886 INFO:teuthology.packaging:sha1: f25860a2cae30438a8a8d15f1f74f6043cc73d75 2021-07-31T01:38:06.887 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=pacific 2021-07-31T01:38:07.097 DEBUG:teuthology.orchestra.run.smithi045:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2021-07-31T01:38:07.118 INFO:teuthology.orchestra.run.smithi045.stdout:16.2.5-150.g5a784c87.el8 2021-07-31T01:38:07.119 INFO:teuthology.packaging:The installed version of ceph is 16.2.5-150.g5a784c87.el8 2021-07-31T01:38:07.119 INFO:teuthology.task.install:The correct ceph version 16.2.5-150.g5a784c87 is installed. 2021-07-31T01:38:07.120 DEBUG:teuthology.orchestra.run.smithi154:> sudo lsb_release -is 2021-07-31T01:38:07.177 INFO:teuthology.orchestra.run.smithi154.stdout:RedHatEnterprise 2021-07-31T01:38:07.179 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2021-07-31T01:38:07.181 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2021-07-31T01:38:07.181 INFO:teuthology.packaging:ref: None 2021-07-31T01:38:07.182 INFO:teuthology.packaging:tag: None 2021-07-31T01:38:07.183 INFO:teuthology.packaging:branch: pacific 2021-07-31T01:38:07.183 INFO:teuthology.packaging:sha1: f25860a2cae30438a8a8d15f1f74f6043cc73d75 2021-07-31T01:38:07.184 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=pacific 2021-07-31T01:38:07.410 DEBUG:teuthology.orchestra.run.smithi154:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2021-07-31T01:38:07.433 INFO:teuthology.orchestra.run.smithi154.stdout:16.2.5-150.g5a784c87.el8 2021-07-31T01:38:07.433 INFO:teuthology.packaging:The installed version of ceph is 16.2.5-150.g5a784c87.el8 2021-07-31T01:38:07.434 INFO:teuthology.task.install:The correct ceph version 16.2.5-150.g5a784c87 is installed. 2021-07-31T01:38:07.434 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2021-07-31T01:38:07.435 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2021-07-31T01:38:07.435 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2021-07-31T01:38:07.464 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2021-07-31T01:38:07.465 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2021-07-31T01:38:07.502 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2021-07-31T01:38:07.503 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2021-07-31T01:38:07.503 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/usr/bin/daemon-helper 2021-07-31T01:38:07.536 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod a=rx -- /usr/bin/daemon-helper 2021-07-31T01:38:07.604 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2021-07-31T01:38:07.604 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/usr/bin/daemon-helper 2021-07-31T01:38:07.631 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod a=rx -- /usr/bin/daemon-helper 2021-07-31T01:38:07.699 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2021-07-31T01:38:07.700 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2021-07-31T01:38:07.700 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/usr/bin/adjust-ulimits 2021-07-31T01:38:07.727 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2021-07-31T01:38:07.795 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2021-07-31T01:38:07.795 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/usr/bin/adjust-ulimits 2021-07-31T01:38:07.822 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2021-07-31T01:38:07.890 INFO:teuthology.run_tasks:Running task print... 2021-07-31T01:38:07.906 INFO:teuthology.task.print:**** done install task... 2021-07-31T01:38:07.907 INFO:teuthology.run_tasks:Running task print... 2021-07-31T01:38:07.942 INFO:teuthology.task.print:**** done start installing pacific cephadm ... 2021-07-31T01:38:07.942 INFO:teuthology.run_tasks:Running task cephadm... 2021-07-31T01:38:08.039 INFO:tasks.cephadm:Config: {'cephadm_branch': 'pacific', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'conf': {'osd': {'osd_class_default_list': '*', 'osd_class_load_list': '*', 'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}, 'global': {'mon election default strategy': 1}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}}, 'image': 'docker.io/ceph/daemon-base:latest-pacific', 'create_rbd_pool': True, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': 'f25860a2cae30438a8a8d15f1f74f6043cc73d75'} 2021-07-31T01:38:08.039 INFO:tasks.cephadm:Cluster image is docker.io/ceph/daemon-base:latest-pacific 2021-07-31T01:38:08.040 INFO:tasks.cephadm:Cluster fsid is f52d2c14-f19f-11eb-8c24-001a4aab830c 2021-07-31T01:38:08.040 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2021-07-31T01:38:08.041 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.45', 'mon.c': '[v2:172.21.15.45:3301,v1:172.21.15.45:6790]', 'mon.b': '172.21.15.154'} 2021-07-31T01:38:08.041 INFO:tasks.cephadm:First mon is mon.a on smithi045 2021-07-31T01:38:08.042 INFO:tasks.cephadm:First mgr is y 2021-07-31T01:38:08.042 INFO:tasks.cephadm:Normalizing hostnames... 2021-07-31T01:38:08.042 DEBUG:teuthology.orchestra.run.smithi045:> sudo hostname $(hostname -s) 2021-07-31T01:38:08.071 DEBUG:teuthology.orchestra.run.smithi154:> sudo hostname $(hostname -s) 2021-07-31T01:38:08.099 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref pacific)... 2021-07-31T01:38:08.100 DEBUG:teuthology.orchestra.run.smithi045:> curl --silent https://raw.githubusercontent.com/ceph/ceph/pacific/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2021-07-31T01:38:08.326 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-rw-r--. 1 ubuntu ubuntu 298325 Jul 31 01:38 /home/ubuntu/cephtest/cephadm 2021-07-31T01:38:08.326 DEBUG:teuthology.orchestra.run.smithi154:> curl --silent https://raw.githubusercontent.com/ceph/ceph/pacific/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2021-07-31T01:38:08.431 INFO:teuthology.orchestra.run.smithi154.stdout:-rw-rw-r--. 1 ubuntu ubuntu 298325 Jul 31 01:38 /home/ubuntu/cephtest/cephadm 2021-07-31T01:38:08.432 DEBUG:teuthology.orchestra.run.smithi045:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2021-07-31T01:38:08.451 DEBUG:teuthology.orchestra.run.smithi154:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2021-07-31T01:38:08.484 DEBUG:teuthology.orchestra.run.smithi045:> sudo mkdir -p /etc/ceph 2021-07-31T01:38:08.521 DEBUG:teuthology.orchestra.run.smithi154:> sudo mkdir -p /etc/ceph 2021-07-31T01:38:08.551 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 777 /etc/ceph 2021-07-31T01:38:08.593 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod 777 /etc/ceph 2021-07-31T01:38:08.621 INFO:tasks.cephadm:Writing seed config... 2021-07-31T01:38:08.623 INFO:tasks.cephadm: override: [osd] osd_class_default_list = * 2021-07-31T01:38:08.623 INFO:tasks.cephadm: override: [osd] osd_class_load_list = * 2021-07-31T01:38:08.623 INFO:tasks.cephadm: override: [osd] debug ms = 1 2021-07-31T01:38:08.624 INFO:tasks.cephadm: override: [osd] debug osd = 20 2021-07-31T01:38:08.624 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2021-07-31T01:38:08.624 INFO:tasks.cephadm: override: [global] mon election default strategy = 1 2021-07-31T01:38:08.624 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2021-07-31T01:38:08.625 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2021-07-31T01:38:08.625 INFO:tasks.cephadm: override: [mon] debug mon = 20 2021-07-31T01:38:08.625 INFO:tasks.cephadm: override: [mon] debug ms = 1 2021-07-31T01:38:08.626 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2021-07-31T01:38:08.627 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2021-07-31T01:38:08.627 DEBUG:teuthology.orchestra.run.smithi045:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2021-07-31T01:38:08.650 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = f52d2c14-f19f-11eb-8c24-001a4aab830c mon election default strategy = 1 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true osd_class_default_list = * osd_class_load_list = * debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2021-07-31T01:38:08.650 DEBUG:teuthology.orchestra.run.smithi045:mon.a> sudo journalctl -f -n 0 -u ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c@mon.a.service 2021-07-31T01:38:08.692 DEBUG:teuthology.orchestra.run.smithi045:mgr.y> sudo journalctl -f -n 0 -u ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c@mgr.y.service 2021-07-31T01:38:08.720 INFO:journalctl@ceph.mon.a.smithi045.stdout:-- Logs begin at Sat 2021-07-31 01:16:15 UTC. -- 2021-07-31T01:38:08.721 INFO:tasks.cephadm:Bootstrapping... 2021-07-31T01:38:08.721 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific -v bootstrap --fsid f52d2c14-f19f-11eb-8c24-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.45 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2021-07-31T01:38:08.747 INFO:journalctl@ceph.mgr.y.smithi045.stdout:-- Logs begin at Sat 2021-07-31 01:16:15 UTC. -- 2021-07-31T01:38:09.051 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2021-07-31T01:38:09.787 INFO:teuthology.orchestra.run.smithi045.stderr:/bin/podman: 3.2.3 2021-07-31T01:38:09.843 INFO:teuthology.orchestra.run.smithi045.stderr:Verifying podman|docker is present... 2021-07-31T01:38:09.843 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2021-07-31T01:38:09.893 INFO:teuthology.orchestra.run.smithi045.stderr:/bin/podman: 3.2.3 2021-07-31T01:38:09.989 INFO:teuthology.orchestra.run.smithi045.stderr:Verifying lvm2 is present... 2021-07-31T01:38:09.989 INFO:teuthology.orchestra.run.smithi045.stderr:Verifying time synchronization is in place... 2021-07-31T01:38:09.990 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: systemctl is-enabled chrony.service 2021-07-31T01:38:09.998 INFO:teuthology.orchestra.run.smithi045.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2021-07-31T01:38:09.998 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: systemctl is-active chrony.service 2021-07-31T01:38:10.003 INFO:teuthology.orchestra.run.smithi045.stderr:systemctl: inactive 2021-07-31T01:38:10.003 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: systemctl is-enabled chronyd.service 2021-07-31T01:38:10.009 INFO:teuthology.orchestra.run.smithi045.stderr:systemctl: enabled 2021-07-31T01:38:10.010 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: systemctl is-active chronyd.service 2021-07-31T01:38:10.017 INFO:teuthology.orchestra.run.smithi045.stderr:systemctl: active 2021-07-31T01:38:10.017 INFO:teuthology.orchestra.run.smithi045.stderr:Unit chronyd.service is enabled and running 2021-07-31T01:38:10.017 INFO:teuthology.orchestra.run.smithi045.stderr:Repeating the final host check... 2021-07-31T01:38:10.017 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2021-07-31T01:38:10.060 INFO:teuthology.orchestra.run.smithi045.stderr:/bin/podman: 3.2.3 2021-07-31T01:38:10.157 INFO:teuthology.orchestra.run.smithi045.stderr:podman|docker (/bin/podman) is present 2021-07-31T01:38:10.158 INFO:teuthology.orchestra.run.smithi045.stderr:systemctl is present 2021-07-31T01:38:10.158 INFO:teuthology.orchestra.run.smithi045.stderr:lvcreate is present 2021-07-31T01:38:10.158 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: systemctl is-enabled chrony.service 2021-07-31T01:38:10.163 INFO:teuthology.orchestra.run.smithi045.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2021-07-31T01:38:10.164 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: systemctl is-active chrony.service 2021-07-31T01:38:10.169 INFO:teuthology.orchestra.run.smithi045.stderr:systemctl: inactive 2021-07-31T01:38:10.169 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: systemctl is-enabled chronyd.service 2021-07-31T01:38:10.175 INFO:teuthology.orchestra.run.smithi045.stderr:systemctl: enabled 2021-07-31T01:38:10.176 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: systemctl is-active chronyd.service 2021-07-31T01:38:10.181 INFO:teuthology.orchestra.run.smithi045.stderr:systemctl: active 2021-07-31T01:38:10.182 INFO:teuthology.orchestra.run.smithi045.stderr:Unit chronyd.service is enabled and running 2021-07-31T01:38:10.182 INFO:teuthology.orchestra.run.smithi045.stderr:Host looks OK 2021-07-31T01:38:10.182 INFO:teuthology.orchestra.run.smithi045.stderr:Cluster fsid: f52d2c14-f19f-11eb-8c24-001a4aab830c 2021-07-31T01:38:10.183 INFO:teuthology.orchestra.run.smithi045.stderr:Acquiring lock 139753647880624 on /run/cephadm/f52d2c14-f19f-11eb-8c24-001a4aab830c.lock 2021-07-31T01:38:10.183 INFO:teuthology.orchestra.run.smithi045.stderr:Lock 139753647880624 acquired on /run/cephadm/f52d2c14-f19f-11eb-8c24-001a4aab830c.lock 2021-07-31T01:38:10.185 INFO:teuthology.orchestra.run.smithi045.stderr:Verifying IP 172.21.15.45 port 3300 ... 2021-07-31T01:38:10.185 INFO:teuthology.orchestra.run.smithi045.stderr:Verifying IP 172.21.15.45 port 6789 ... 2021-07-31T01:38:10.185 INFO:teuthology.orchestra.run.smithi045.stderr:Base mon IP is 172.21.15.45, final addrv is [v2:172.21.15.45:3300,v1:172.21.15.45:6789] 2021-07-31T01:38:10.185 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /sbin/ip route ls 2021-07-31T01:38:10.186 INFO:teuthology.orchestra.run.smithi045.stderr:/sbin/ip: default via 172.21.15.254 dev ens1f0 proto dhcp metric 100 2021-07-31T01:38:10.186 INFO:teuthology.orchestra.run.smithi045.stderr:/sbin/ip: 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.45 metric 100 2021-07-31T01:38:10.187 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /sbin/ip -6 route ls 2021-07-31T01:38:10.187 INFO:teuthology.orchestra.run.smithi045.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2021-07-31T01:38:10.188 INFO:teuthology.orchestra.run.smithi045.stderr:/sbin/ip: fe80::/64 dev ens1f0 proto kernel metric 256 pref medium 2021-07-31T01:38:10.189 INFO:teuthology.orchestra.run.smithi045.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 1024 expires 1790sec hoplimit 64 pref medium 2021-07-31T01:38:10.189 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /sbin/ip -6 addr ls 2021-07-31T01:38:10.191 INFO:teuthology.orchestra.run.smithi045.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2021-07-31T01:38:10.191 INFO:teuthology.orchestra.run.smithi045.stderr:/sbin/ip: inet6 ::1/128 scope host 2021-07-31T01:38:10.192 INFO:teuthology.orchestra.run.smithi045.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2021-07-31T01:38:10.192 INFO:teuthology.orchestra.run.smithi045.stderr:/sbin/ip: 3: ens1f0: mtu 1500 state UP qlen 1000 2021-07-31T01:38:10.192 INFO:teuthology.orchestra.run.smithi045.stderr:/sbin/ip: inet6 fe80::ec4:7aff:febd:12be/64 scope link 2021-07-31T01:38:10.192 INFO:teuthology.orchestra.run.smithi045.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2021-07-31T01:38:10.193 INFO:teuthology.orchestra.run.smithi045.stderr:Mon IP `172.21.15.45` is in CIDR network `172.21.0.0/20` 2021-07-31T01:38:10.193 INFO:teuthology.orchestra.run.smithi045.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2021-07-31T01:38:10.194 INFO:teuthology.orchestra.run.smithi045.stderr:Pulling container image docker.io/ceph/daemon-base:latest-pacific... 2021-07-31T01:38:10.194 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /bin/podman pull docker.io/ceph/daemon-base:latest-pacific 2021-07-31T01:38:10.237 INFO:teuthology.orchestra.run.smithi045.stderr:/bin/podman: Trying to pull docker.io/ceph/daemon-base:latest-pacific... 2021-07-31T01:38:11.025 INFO:teuthology.orchestra.run.smithi045.stderr:/bin/podman: Getting image source signatures 2021-07-31T01:38:11.178 INFO:teuthology.orchestra.run.smithi045.stderr:/bin/podman: Copying blob sha256:7a0437f04f83f084b7ed68ad9c4a4947e12fc4e1b006b38129bac89114ec3621 2021-07-31T01:38:11.252 INFO:teuthology.orchestra.run.smithi045.stderr:/bin/podman: Copying blob sha256:79b8cfb73720e504eea96652816606d2e35443d17dcf4b03436c86eb0ed3724e 2021-07-31T01:38:19.571 INFO:teuthology.orchestra.run.smithi045.stderr:/bin/podman: Copying config sha256:a29040f3000ee18e001dfd32eb32838b2c80c805aeee25ad55400d7637cf29e0 2021-07-31T01:38:19.635 INFO:teuthology.orchestra.run.smithi045.stderr:/bin/podman: Writing manifest to image destination 2021-07-31T01:38:19.635 INFO:teuthology.orchestra.run.smithi045.stderr:/bin/podman: Storing signatures 2021-07-31T01:38:30.516 INFO:teuthology.orchestra.run.smithi045.stderr:/bin/podman: a29040f3000ee18e001dfd32eb32838b2c80c805aeee25ad55400d7637cf29e0 2021-07-31T01:38:30.570 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint ceph --init -e CONTAINER_IMAGE=docker.io/ceph/daemon-base:latest-pacific -e NODE_NAME=smithi045 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/daemon-base:latest-pacific --version 2021-07-31T01:38:31.265 INFO:teuthology.orchestra.run.smithi045.stderr:ceph: ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable) 2021-07-31T01:38:42.934 INFO:teuthology.orchestra.run.smithi045.stderr:Ceph version: ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable) 2021-07-31T01:38:42.935 INFO:teuthology.orchestra.run.smithi045.stderr:Extracting ceph user uid/gid from container image... 2021-07-31T01:38:42.935 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/ceph/daemon-base:latest-pacific -e NODE_NAME=smithi045 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/daemon-base:latest-pacific -c %u %g /var/lib/ceph 2021-07-31T01:38:43.494 INFO:teuthology.orchestra.run.smithi045.stderr:stat: 167 167 2021-07-31T01:38:43.985 INFO:teuthology.orchestra.run.smithi045.stderr:Creating initial keys... 2021-07-31T01:38:43.985 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/daemon-base:latest-pacific -e NODE_NAME=smithi045 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/daemon-base:latest-pacific --gen-print-key 2021-07-31T01:38:44.555 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-authtool: AQCkqQRhxaYuIBAA7W2uptGX8rNPuVsS7puKMg== 2021-07-31T01:38:45.130 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/daemon-base:latest-pacific -e NODE_NAME=smithi045 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/daemon-base:latest-pacific --gen-print-key 2021-07-31T01:38:45.655 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-authtool: AQClqQRhBMluJhAA9bOIkHGvpTnwZvMzLOhCfw== 2021-07-31T01:38:46.185 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/daemon-base:latest-pacific -e NODE_NAME=smithi045 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/daemon-base:latest-pacific --gen-print-key 2021-07-31T01:38:46.765 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-authtool: AQCmqQRhU2ypKRAABVpQn+4BnpUGfO+guimfnA== 2021-07-31T01:38:47.333 INFO:teuthology.orchestra.run.smithi045.stderr:Creating initial monmap... 2021-07-31T01:38:47.334 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/monmaptool --init -e CONTAINER_IMAGE=docker.io/ceph/daemon-base:latest-pacific -e NODE_NAME=smithi045 -e CEPH_USE_RANDOM_NONCE=1 -v /tmp/ceph-tmp0wmcz0d0:/tmp/monmap:z docker.io/ceph/daemon-base:latest-pacific --create --clobber --fsid f52d2c14-f19f-11eb-8c24-001a4aab830c --addv a [v2:172.21.15.45:3300,v1:172.21.15.45:6789] /tmp/monmap 2021-07-31T01:38:47.864 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2021-07-31T01:38:47.864 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to f52d2c14-f19f-11eb-8c24-001a4aab830c 2021-07-31T01:38:47.864 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2021-07-31T01:38:48.486 INFO:teuthology.orchestra.run.smithi045.stderr:monmaptool for a [v2:172.21.15.45:3300,v1:172.21.15.45:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2021-07-31T01:38:48.487 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/monmaptool: set fsid to f52d2c14-f19f-11eb-8c24-001a4aab830c 2021-07-31T01:38:48.487 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2021-07-31T01:38:48.487 INFO:teuthology.orchestra.run.smithi045.stderr: 2021-07-31T01:38:48.487 INFO:teuthology.orchestra.run.smithi045.stderr:Creating mon... 2021-07-31T01:38:48.488 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph-mon --init -e CONTAINER_IMAGE=docker.io/ceph/daemon-base:latest-pacific -e NODE_NAME=smithi045 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/f52d2c14-f19f-11eb-8c24-001a4aab830c:/var/log/ceph:z -v /var/lib/ceph/f52d2c14-f19f-11eb-8c24-001a4aab830c/mon.a:/var/lib/ceph/mon/ceph-a:z -v /tmp/ceph-tmprm29vdvd:/tmp/keyring:z -v /tmp/ceph-tmp0wmcz0d0:/tmp/monmap:z docker.io/ceph/daemon-base:latest-pacific --mkfs -i a --fsid f52d2c14-f19f-11eb-8c24-001a4aab830c -c /dev/null --monmap /tmp/monmap --keyring /tmp/keyring --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true 2021-07-31T01:38:49.098 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.060+0000 7f2467fc3700 0 set uid:gid to 167:167 (ceph:ceph) 2021-07-31T01:38:49.099 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.060+0000 7f2467fc3700 1 imported monmap: 2021-07-31T01:38:49.099 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: epoch 0 2021-07-31T01:38:49.099 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: fsid f52d2c14-f19f-11eb-8c24-001a4aab830c 2021-07-31T01:38:49.100 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: last_changed 2021-07-31T01:38:47.857698+0000 2021-07-31T01:38:49.100 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: created 2021-07-31T01:38:47.857698+0000 2021-07-31T01:38:49.100 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: min_mon_release 0 (unknown) 2021-07-31T01:38:49.100 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: election_strategy: 1 2021-07-31T01:38:49.100 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: 0: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.a 2021-07-31T01:38:49.101 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: 2021-07-31T01:38:49.101 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.060+0000 7f2467fc3700 0 /usr/bin/ceph-mon: set fsid to f52d2c14-f19f-11eb-8c24-001a4aab830c 2021-07-31T01:38:49.101 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: RocksDB version: 6.8.1 2021-07-31T01:38:49.101 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: 2021-07-31T01:38:49.103 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2021-07-31T01:38:49.103 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: Compile date Jul 8 2021 2021-07-31T01:38:49.103 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: DB SUMMARY 2021-07-31T01:38:49.103 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: 2021-07-31T01:38:49.104 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 0, files: 2021-07-31T01:38:49.104 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: 2021-07-31T01:38:49.105 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 2021-07-31T01:38:49.105 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: 2021-07-31T01:38:49.105 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: Options.error_if_exists: 0 2021-07-31T01:38:49.105 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: Options.create_if_missing: 1 2021-07-31T01:38:49.106 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: Options.paranoid_checks: 1 2021-07-31T01:38:49.106 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: Options.env: 0x55a672e4d1c0 2021-07-31T01:38:49.106 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: Options.fs: Posix File System 2021-07-31T01:38:49.106 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: Options.info_log: 0x55a674a50f00 2021-07-31T01:38:49.106 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: Options.max_file_opening_threads: 16 2021-07-31T01:38:49.107 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: Options.statistics: (nil) 2021-07-31T01:38:49.107 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: Options.use_fsync: 0 2021-07-31T01:38:49.107 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: Options.max_log_file_size: 0 2021-07-31T01:38:49.107 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2021-07-31T01:38:49.108 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: Options.log_file_time_to_roll: 0 2021-07-31T01:38:49.108 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: Options.keep_log_file_num: 1000 2021-07-31T01:38:49.108 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: Options.recycle_log_file_num: 0 2021-07-31T01:38:49.108 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: Options.allow_fallocate: 1 2021-07-31T01:38:49.108 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: Options.allow_mmap_reads: 0 2021-07-31T01:38:49.109 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: Options.allow_mmap_writes: 0 2021-07-31T01:38:49.109 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: Options.use_direct_reads: 0 2021-07-31T01:38:49.109 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2021-07-31T01:38:49.110 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: Options.create_missing_column_families: 0 2021-07-31T01:38:49.110 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: Options.db_log_dir: 2021-07-31T01:38:49.110 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2021-07-31T01:38:49.110 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: Options.table_cache_numshardbits: 6 2021-07-31T01:38:49.110 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: Options.max_subcompactions: 1 2021-07-31T01:38:49.111 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: Options.max_background_flushes: -1 2021-07-31T01:38:49.111 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: Options.WAL_ttl_seconds: 0 2021-07-31T01:38:49.111 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: Options.WAL_size_limit_MB: 0 2021-07-31T01:38:49.111 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2021-07-31T01:38:49.112 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2021-07-31T01:38:49.112 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: Options.is_fd_close_on_exec: 1 2021-07-31T01:38:49.112 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: Options.advise_random_on_open: 1 2021-07-31T01:38:49.112 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: Options.db_write_buffer_size: 0 2021-07-31T01:38:49.112 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: Options.write_buffer_manager: 0x55a674a59500 2021-07-31T01:38:49.113 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2021-07-31T01:38:49.113 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2021-07-31T01:38:49.113 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2021-07-31T01:38:49.114 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: Options.use_adaptive_mutex: 0 2021-07-31T01:38:49.114 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: Options.rate_limiter: (nil) 2021-07-31T01:38:49.115 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2021-07-31T01:38:49.115 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: Options.wal_recovery_mode: 2 2021-07-31T01:38:49.116 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: Options.enable_thread_tracking: 0 2021-07-31T01:38:49.117 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: Options.enable_pipelined_write: 0 2021-07-31T01:38:49.117 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: Options.unordered_write: 0 2021-07-31T01:38:49.117 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2021-07-31T01:38:49.118 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2021-07-31T01:38:49.118 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2021-07-31T01:38:49.118 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2021-07-31T01:38:49.118 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: Options.row_cache: None 2021-07-31T01:38:49.119 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: Options.wal_filter: None 2021-07-31T01:38:49.119 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2021-07-31T01:38:49.119 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: Options.allow_ingest_behind: 0 2021-07-31T01:38:49.119 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: Options.preserve_deletes: 0 2021-07-31T01:38:49.120 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: Options.two_write_queues: 0 2021-07-31T01:38:49.121 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: Options.manual_wal_flush: 0 2021-07-31T01:38:49.121 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: Options.atomic_flush: 0 2021-07-31T01:38:49.122 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2021-07-31T01:38:49.122 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: Options.persist_stats_to_disk: 0 2021-07-31T01:38:49.122 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: Options.write_dbid_to_manifest: 0 2021-07-31T01:38:49.122 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: Options.log_readahead_size: 0 2021-07-31T01:38:49.123 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2021-07-31T01:38:49.123 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: Options.max_background_jobs: 2 2021-07-31T01:38:49.123 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: Options.max_background_compactions: -1 2021-07-31T01:38:49.123 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2021-07-31T01:38:49.123 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2021-07-31T01:38:49.124 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: Options.delayed_write_rate : 16777216 2021-07-31T01:38:49.125 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: Options.max_total_wal_size: 0 2021-07-31T01:38:49.125 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2021-07-31T01:38:49.126 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: Options.stats_dump_period_sec: 600 2021-07-31T01:38:49.127 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: Options.stats_persist_period_sec: 600 2021-07-31T01:38:49.127 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2021-07-31T01:38:49.127 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: Options.max_open_files: -1 2021-07-31T01:38:49.128 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: Options.bytes_per_sync: 0 2021-07-31T01:38:49.128 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: Options.wal_bytes_per_sync: 0 2021-07-31T01:38:49.128 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: Options.strict_bytes_per_sync: 0 2021-07-31T01:38:49.128 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: Options.compaction_readahead_size: 0 2021-07-31T01:38:49.128 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: Compression algorithms supported: 2021-07-31T01:38:49.129 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2021-07-31T01:38:49.129 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: kZSTD supported: 0 2021-07-31T01:38:49.129 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: kXpressCompression supported: 0 2021-07-31T01:38:49.129 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: kLZ4HCCompression supported: 1 2021-07-31T01:38:49.129 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: kLZ4Compression supported: 1 2021-07-31T01:38:49.130 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: kBZip2Compression supported: 0 2021-07-31T01:38:49.131 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: kZlibCompression supported: 1 2021-07-31T01:38:49.131 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: kSnappyCompression supported: 1 2021-07-31T01:38:49.131 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: Fast CRC32 supported: Supported on x86 2021-07-31T01:38:49.131 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.061+0000 7f2467fc3700 4 rocksdb: [db_impl/db_impl_open.cc:273] Creating manifest 1 2021-07-31T01:38:49.132 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: 2021-07-31T01:38:49.132 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.063+0000 7f2467fc3700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000001 2021-07-31T01:38:49.132 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: 2021-07-31T01:38:49.132 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.063+0000 7f2467fc3700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2021-07-31T01:38:49.132 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: 2021-07-31T01:38:49.133 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.063+0000 7f2467fc3700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2021-07-31T01:38:49.133 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.063+0000 7f2467fc3700 4 rocksdb: Options.merge_operator: 2021-07-31T01:38:49.133 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.063+0000 7f2467fc3700 4 rocksdb: Options.compaction_filter: None 2021-07-31T01:38:49.133 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.063+0000 7f2467fc3700 4 rocksdb: Options.compaction_filter_factory: None 2021-07-31T01:38:49.133 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.063+0000 7f2467fc3700 4 rocksdb: Options.memtable_factory: SkipListFactory 2021-07-31T01:38:49.134 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.063+0000 7f2467fc3700 4 rocksdb: Options.table_factory: BlockBasedTable 2021-07-31T01:38:49.134 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.063+0000 7f2467fc3700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55a674964160) 2021-07-31T01:38:49.134 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks: 1 2021-07-31T01:38:49.134 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks_with_high_priority: 0 2021-07-31T01:38:49.135 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: pin_l0_filter_and_index_blocks_in_cache: 0 2021-07-31T01:38:49.135 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: pin_top_level_index_and_filter: 1 2021-07-31T01:38:49.135 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: index_type: 0 2021-07-31T01:38:49.135 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: data_block_index_type: 0 2021-07-31T01:38:49.135 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: index_shortening: 1 2021-07-31T01:38:49.136 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: data_block_hash_table_util_ratio: 0.750000 2021-07-31T01:38:49.137 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: hash_index_allow_collision: 1 2021-07-31T01:38:49.137 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: checksum: 1 2021-07-31T01:38:49.138 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: no_block_cache: 0 2021-07-31T01:38:49.138 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: block_cache: 0x55a67499ed10 2021-07-31T01:38:49.139 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: block_cache_name: BinnedLRUCache 2021-07-31T01:38:49.139 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: block_cache_options: 2021-07-31T01:38:49.139 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: capacity : 536870912 2021-07-31T01:38:49.139 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: num_shard_bits : 4 2021-07-31T01:38:49.139 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: strict_capacity_limit : 0 2021-07-31T01:38:49.140 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: high_pri_pool_ratio: 0.000 2021-07-31T01:38:49.140 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: block_cache_compressed: (nil) 2021-07-31T01:38:49.140 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: persistent_cache: (nil) 2021-07-31T01:38:49.141 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: block_size: 4096 2021-07-31T01:38:49.141 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: block_size_deviation: 10 2021-07-31T01:38:49.141 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: block_restart_interval: 16 2021-07-31T01:38:49.141 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: index_block_restart_interval: 1 2021-07-31T01:38:49.141 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: metadata_block_size: 4096 2021-07-31T01:38:49.142 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: partition_filters: 0 2021-07-31T01:38:49.142 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: use_delta_encoding: 1 2021-07-31T01:38:49.142 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: filter_policy: rocksdb.BuiltinBloomFilter 2021-07-31T01:38:49.142 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: whole_key_filtering: 1 2021-07-31T01:38:49.143 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: verify_compression: 0 2021-07-31T01:38:49.143 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: read_amp_bytes_per_bit: 0 2021-07-31T01:38:49.143 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: format_version: 2 2021-07-31T01:38:49.143 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: enable_index_compression: 1 2021-07-31T01:38:49.143 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: block_align: 0 2021-07-31T01:38:49.144 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: 2021-07-31T01:38:49.144 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.063+0000 7f2467fc3700 4 rocksdb: Options.write_buffer_size: 33554432 2021-07-31T01:38:49.144 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.063+0000 7f2467fc3700 4 rocksdb: Options.max_write_buffer_number: 2 2021-07-31T01:38:49.144 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.063+0000 7f2467fc3700 4 rocksdb: Options.compression: NoCompression 2021-07-31T01:38:49.144 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.063+0000 7f2467fc3700 4 rocksdb: Options.bottommost_compression: Disabled 2021-07-31T01:38:49.145 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.063+0000 7f2467fc3700 4 rocksdb: Options.prefix_extractor: nullptr 2021-07-31T01:38:49.145 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.063+0000 7f2467fc3700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2021-07-31T01:38:49.145 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.063+0000 7f2467fc3700 4 rocksdb: Options.num_levels: 7 2021-07-31T01:38:49.145 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.063+0000 7f2467fc3700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2021-07-31T01:38:49.145 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.063+0000 7f2467fc3700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2021-07-31T01:38:49.146 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.063+0000 7f2467fc3700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2021-07-31T01:38:49.146 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.063+0000 7f2467fc3700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2021-07-31T01:38:49.147 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.063+0000 7f2467fc3700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2021-07-31T01:38:49.147 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.063+0000 7f2467fc3700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2021-07-31T01:38:49.147 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.063+0000 7f2467fc3700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2021-07-31T01:38:49.148 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.063+0000 7f2467fc3700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2021-07-31T01:38:49.149 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.063+0000 7f2467fc3700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2021-07-31T01:38:49.149 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.063+0000 7f2467fc3700 4 rocksdb: Options.compression_opts.window_bits: -14 2021-07-31T01:38:49.149 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.063+0000 7f2467fc3700 4 rocksdb: Options.compression_opts.level: 32767 2021-07-31T01:38:49.149 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.063+0000 7f2467fc3700 4 rocksdb: Options.compression_opts.strategy: 0 2021-07-31T01:38:49.150 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.063+0000 7f2467fc3700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2021-07-31T01:38:49.150 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.063+0000 7f2467fc3700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2021-07-31T01:38:49.150 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.063+0000 7f2467fc3700 4 rocksdb: Options.compression_opts.enabled: false 2021-07-31T01:38:49.151 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.063+0000 7f2467fc3700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2021-07-31T01:38:49.151 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.063+0000 7f2467fc3700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2021-07-31T01:38:49.151 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.063+0000 7f2467fc3700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2021-07-31T01:38:49.151 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.063+0000 7f2467fc3700 4 rocksdb: Options.target_file_size_base: 67108864 2021-07-31T01:38:49.152 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.063+0000 7f2467fc3700 4 rocksdb: Options.target_file_size_multiplier: 1 2021-07-31T01:38:49.152 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.063+0000 7f2467fc3700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2021-07-31T01:38:49.152 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.063+0000 7f2467fc3700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2021-07-31T01:38:49.152 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.063+0000 7f2467fc3700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2021-07-31T01:38:49.153 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.063+0000 7f2467fc3700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2021-07-31T01:38:49.153 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.063+0000 7f2467fc3700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2021-07-31T01:38:49.153 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.063+0000 7f2467fc3700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2021-07-31T01:38:49.154 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.063+0000 7f2467fc3700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2021-07-31T01:38:49.154 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.063+0000 7f2467fc3700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2021-07-31T01:38:49.154 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.063+0000 7f2467fc3700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2021-07-31T01:38:49.154 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.063+0000 7f2467fc3700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2021-07-31T01:38:49.155 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.063+0000 7f2467fc3700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2021-07-31T01:38:49.155 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.063+0000 7f2467fc3700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2021-07-31T01:38:49.155 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.063+0000 7f2467fc3700 4 rocksdb: Options.arena_block_size: 4194304 2021-07-31T01:38:49.155 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.063+0000 7f2467fc3700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2021-07-31T01:38:49.156 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.063+0000 7f2467fc3700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2021-07-31T01:38:49.156 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.063+0000 7f2467fc3700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2021-07-31T01:38:49.156 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.063+0000 7f2467fc3700 4 rocksdb: Options.disable_auto_compactions: 0 2021-07-31T01:38:49.156 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.063+0000 7f2467fc3700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2021-07-31T01:38:49.156 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.063+0000 7f2467fc3700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2021-07-31T01:38:49.157 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.063+0000 7f2467fc3700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2021-07-31T01:38:49.157 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.063+0000 7f2467fc3700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2021-07-31T01:38:49.158 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.063+0000 7f2467fc3700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2021-07-31T01:38:49.158 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.063+0000 7f2467fc3700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2021-07-31T01:38:49.158 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.063+0000 7f2467fc3700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2021-07-31T01:38:49.158 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.063+0000 7f2467fc3700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2021-07-31T01:38:49.158 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.063+0000 7f2467fc3700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2021-07-31T01:38:49.159 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.063+0000 7f2467fc3700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2021-07-31T01:38:49.160 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.063+0000 7f2467fc3700 4 rocksdb: Options.table_properties_collectors: 2021-07-31T01:38:49.160 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.063+0000 7f2467fc3700 4 rocksdb: Options.inplace_update_support: 0 2021-07-31T01:38:49.160 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.063+0000 7f2467fc3700 4 rocksdb: Options.inplace_update_num_locks: 10000 2021-07-31T01:38:49.160 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.063+0000 7f2467fc3700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2021-07-31T01:38:49.160 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.063+0000 7f2467fc3700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2021-07-31T01:38:49.161 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.063+0000 7f2467fc3700 4 rocksdb: Options.memtable_huge_page_size: 0 2021-07-31T01:38:49.161 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.063+0000 7f2467fc3700 4 rocksdb: Options.bloom_locality: 0 2021-07-31T01:38:49.162 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.063+0000 7f2467fc3700 4 rocksdb: Options.max_successive_merges: 0 2021-07-31T01:38:49.163 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.063+0000 7f2467fc3700 4 rocksdb: Options.optimize_filters_for_hits: 0 2021-07-31T01:38:49.163 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.063+0000 7f2467fc3700 4 rocksdb: Options.paranoid_file_checks: 0 2021-07-31T01:38:49.163 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.063+0000 7f2467fc3700 4 rocksdb: Options.force_consistency_checks: 0 2021-07-31T01:38:49.163 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.063+0000 7f2467fc3700 4 rocksdb: Options.report_bg_io_stats: 0 2021-07-31T01:38:49.163 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.063+0000 7f2467fc3700 4 rocksdb: Options.ttl: 2592000 2021-07-31T01:38:49.164 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.063+0000 7f2467fc3700 4 rocksdb: Options.periodic_compaction_seconds: 0 2021-07-31T01:38:49.164 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.063+0000 7f2467fc3700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2021-07-31T01:38:49.165 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: 2021-07-31T01:38:49.165 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.063+0000 7f2467fc3700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2021-07-31T01:38:49.165 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: 2021-07-31T01:38:49.165 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.064+0000 7f2467fc3700 4 rocksdb: DB pointer 0x55a674a65800 2021-07-31T01:38:49.166 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.064+0000 7f2450e4c700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2021-07-31T01:38:49.166 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.064+0000 7f2450e4c700 4 rocksdb: [db_impl/db_impl.cc:851] 2021-07-31T01:38:49.166 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: ** DB Stats ** 2021-07-31T01:38:49.166 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2021-07-31T01:38:49.166 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2021-07-31T01:38:49.167 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2021-07-31T01:38:49.167 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2021-07-31T01:38:49.167 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2021-07-31T01:38:49.167 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2021-07-31T01:38:49.168 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2021-07-31T01:38:49.168 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: 2021-07-31T01:38:49.168 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2021-07-31T01:38:49.168 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-07-31T01:38:49.168 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-07-31T01:38:49.169 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2021-07-31T01:38:49.169 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2021-07-31T01:38:49.169 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: 2021-07-31T01:38:49.169 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2021-07-31T01:38:49.170 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-07-31T01:38:49.170 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-07-31T01:38:49.170 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2021-07-31T01:38:49.171 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2021-07-31T01:38:49.171 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2021-07-31T01:38:49.171 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2021-07-31T01:38:49.171 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2021-07-31T01:38:49.171 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2021-07-31T01:38:49.172 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-07-31T01:38:49.172 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-07-31T01:38:49.172 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2021-07-31T01:38:49.172 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: 2021-07-31T01:38:49.173 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2021-07-31T01:38:49.173 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: 2021-07-31T01:38:49.173 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2021-07-31T01:38:49.173 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-07-31T01:38:49.173 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-07-31T01:38:49.174 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2021-07-31T01:38:49.174 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2021-07-31T01:38:49.174 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: 2021-07-31T01:38:49.174 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2021-07-31T01:38:49.175 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-07-31T01:38:49.175 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-07-31T01:38:49.175 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2021-07-31T01:38:49.175 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2021-07-31T01:38:49.175 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2021-07-31T01:38:49.176 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2021-07-31T01:38:49.176 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2021-07-31T01:38:49.176 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2021-07-31T01:38:49.176 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-07-31T01:38:49.177 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-07-31T01:38:49.177 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2021-07-31T01:38:49.177 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: 2021-07-31T01:38:49.177 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2021-07-31T01:38:49.177 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: 2021-07-31T01:38:49.178 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.065+0000 7f2467fc3700 4 rocksdb: [db_impl/db_impl.cc:397] Shutdown: canceling all background work 2021-07-31T01:38:49.178 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.066+0000 7f2467fc3700 4 rocksdb: [db_impl/db_impl.cc:573] Shutdown complete 2021-07-31T01:38:49.178 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2021-07-31T01:38:49.066+0000 7f2467fc3700 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-a for mon.a 2021-07-31T01:38:49.630 INFO:teuthology.orchestra.run.smithi045.stderr:create mon.a on 2021-07-31T01:38:49.631 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: install -d -m0770 -o 167 -g 167 /var/run/ceph/f52d2c14-f19f-11eb-8c24-001a4aab830c 2021-07-31T01:38:49.637 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: systemctl disable ceph.target 2021-07-31T01:38:49.645 INFO:teuthology.orchestra.run.smithi045.stderr:systemctl: Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2021-07-31T01:38:49.730 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: systemctl enable ceph.target 2021-07-31T01:38:49.737 INFO:teuthology.orchestra.run.smithi045.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2021-07-31T01:38:49.820 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: systemctl start ceph.target 2021-07-31T01:38:49.828 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: systemctl enable ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c.target 2021-07-31T01:38:49.834 INFO:teuthology.orchestra.run.smithi045.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c.target → /etc/systemd/system/ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c.target. 2021-07-31T01:38:49.835 INFO:teuthology.orchestra.run.smithi045.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c.target → /etc/systemd/system/ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c.target. 2021-07-31T01:38:49.914 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: systemctl start ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c.target 2021-07-31T01:38:49.921 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: systemctl daemon-reload 2021-07-31T01:38:50.008 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: systemctl stop ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c@mon.a 2021-07-31T01:38:50.041 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: systemctl reset-failed ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c@mon.a 2021-07-31T01:38:50.048 INFO:teuthology.orchestra.run.smithi045.stderr:systemctl: Failed to reset failed state of unit ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c@mon.a.service: Unit ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c@mon.a.service not loaded. 2021-07-31T01:38:50.049 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: systemctl enable ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c@mon.a 2021-07-31T01:38:50.054 INFO:teuthology.orchestra.run.smithi045.stderr:systemctl: Created symlink /etc/systemd/system/ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c.target.wants/ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c@mon.a.service → /etc/systemd/system/ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c@.service. 2021-07-31T01:38:50.142 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: systemctl start ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c@mon.a 2021-07-31T01:38:50.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:50 smithi045 systemd[1]: Starting Ceph mon.a for f52d2c14-f19f-11eb-8c24-001a4aab830c... 2021-07-31T01:38:50.683 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:50 smithi045 podman[33024]: 2021-07-31 01:38:50.289570121 +0000 UTC m=+0.031255073 image pull docker.io/ceph/daemon-base:latest-pacific 2021-07-31T01:38:50.683 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:50 smithi045 podman[33024]: 2021-07-31 01:38:50.470960616 +0000 UTC m=+0.212645695 container create 4b01d5c27484a5548cf48ce5bcabc287c8a7fb69fba679a15fbf0e1b01cbb61c (image=docker.io/ceph/daemon-base:latest-pacific, name=ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c-mon.a, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Dimitri Savineau , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Base Image, RELEASE=master-0b3dfd0, org.label-schema.build-date=20201204, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=0b3dfd099f6a46b8c8fecd28a50330b63408461f, CEPH_POINT_RELEASE=) 2021-07-31T01:38:50.683 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:50 smithi045 podman[33024]: 2021-07-31 01:38:50.620934969 +0000 UTC m=+0.362620047 container init 4b01d5c27484a5548cf48ce5bcabc287c8a7fb69fba679a15fbf0e1b01cbb61c (image=docker.io/ceph/daemon-base:latest-pacific, name=ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c-mon.a, maintainer=Dimitri Savineau , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Base Image, ceph=True, GIT_COMMIT=0b3dfd099f6a46b8c8fecd28a50330b63408461f, RELEASE=master-0b3dfd0, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20201204, org.label-schema.schema-version=1.0) 2021-07-31T01:38:50.785 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: systemctl is-enabled firewalld.service 2021-07-31T01:38:50.794 INFO:teuthology.orchestra.run.smithi045.stderr:systemctl: disabled 2021-07-31T01:38:50.794 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: systemctl is-active firewalld.service 2021-07-31T01:38:50.802 INFO:teuthology.orchestra.run.smithi045.stderr:systemctl: inactive 2021-07-31T01:38:50.803 INFO:teuthology.orchestra.run.smithi045.stderr:firewalld.service is not enabled 2021-07-31T01:38:50.803 INFO:teuthology.orchestra.run.smithi045.stderr:Not possible to enable service . firewalld.service is not available 2021-07-31T01:38:50.803 INFO:teuthology.orchestra.run.smithi045.stderr:Waiting for mon to start... 2021-07-31T01:38:50.804 INFO:teuthology.orchestra.run.smithi045.stderr:Waiting for mon... 2021-07-31T01:38:50.804 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/daemon-base:latest-pacific -e NODE_NAME=smithi045 -e CEPH_USE_RANDOM_NONCE=1 -v /var/lib/ceph/f52d2c14-f19f-11eb-8c24-001a4aab830c/mon.a:/var/lib/ceph/mon/ceph-a:z -v /tmp/ceph-tmput3kn1b8:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp5r80c4w5:/etc/ceph/ceph.conf:z docker.io/ceph/daemon-base:latest-pacific status 2021-07-31T01:38:50.941 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:50 smithi045 conmon[33064]: cluster 2021-07-31T01:38:50.681094+0000 mon.a (mon.0) 0 : [INF] mkfs f52d2c14-f19f-11eb-8c24-001a4aab830c 2021-07-31T01:38:50.942 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:50 smithi045 conmon[33064]: cluster 2021-07-31T01 2021-07-31T01:38:50.942 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:50 smithi045 conmon[33064]: :38:50.678002+0000 mon.a (mon.0) 1 : cluster [INF] mon.a is new leader, mons a in quorum (ranks 0) 2021-07-31T01:38:50.942 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:50 smithi045 podman[33024]: 2021-07-31 01:38:50.704311354 +0000 UTC m=+0.445996416 container start 4b01d5c27484a5548cf48ce5bcabc287c8a7fb69fba679a15fbf0e1b01cbb61c (image=docker.io/ceph/daemon-base:latest-pacific, name=ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c-mon.a, maintainer=Dimitri Savineau , org.label-schema.build-date=20201204, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Base Image, ceph=True, GIT_COMMIT=0b3dfd099f6a46b8c8fecd28a50330b63408461f, RELEASE=master-0b3dfd0, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2021-07-31T01:38:50.943 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:50 smithi045 bash[32950]: 4b01d5c27484a5548cf48ce5bcabc287c8a7fb69fba679a15fbf0e1b01cbb61c 2021-07-31T01:38:50.944 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:50 smithi045 systemd[1]: Started Ceph mon.a for f52d2c14-f19f-11eb-8c24-001a4aab830c. 2021-07-31T01:38:51.490 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: cluster: 2021-07-31T01:38:51.490 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: id: f52d2c14-f19f-11eb-8c24-001a4aab830c 2021-07-31T01:38:51.491 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: health: HEALTH_OK 2021-07-31T01:38:51.491 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2021-07-31T01:38:51.491 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: services: 2021-07-31T01:38:51.491 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: mon: 1 daemons, quorum a (age 0.807444s) 2021-07-31T01:38:51.491 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: mgr: no daemons active 2021-07-31T01:38:51.492 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2021-07-31T01:38:51.492 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2021-07-31T01:38:51.492 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: data: 2021-07-31T01:38:51.492 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2021-07-31T01:38:51.492 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2021-07-31T01:38:51.493 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2021-07-31T01:38:51.493 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: pgs: 2021-07-31T01:38:51.493 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2021-07-31T01:38:52.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:51 smithi045 conmon[33064]: cluster 2021-07-31T01:38:50.680661+0000 mon.a (mon.0) 2 : cluster [INF] mon.a is new leader, mons a in quorum (ranks 0) 2021-07-31T01:38:52.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:51 smithi045 conmon[33064]: cluster 2021-07-31T01:38:50.680782+0000 mon.a (mon.0) 3 : cluster [DBG] monmap e1: 1 mons at {a=[v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0]} 2021-07-31T01:38:52.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:51 smithi045 conmon[33064]: cluster 2021-07-31T01:38:50.682191+0000 mon.a (mon.0) 4 : cluster [DBG] fsmap 2021-07-31T01:38:52.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:51 smithi045 conmon[33064]: cluster 2021-07-31T01:38:50.683877+0000 mon.a (mon.0) 5 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2021-07-31T01:38:52.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:51 smithi045 conmon[33064]: cluster 2021-07-31T01:38:50.684234+0000 mon.a (mon.0) 6 : cluster [DBG] 2021-07-31T01:38:52.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:51 smithi045 conmon[33064]: mgrmap e1: no daemons active 2021-07-31T01:38:52.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:51 smithi045 conmon[33064]: audit 2021-07-31T01:38:51.488073+0000 mon.a (mon.0) 7 : audit [DBG] from='client.? 172.21.15.45:0/3853017551' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2021-07-31T01:38:52.345 INFO:teuthology.orchestra.run.smithi045.stderr:mon is available 2021-07-31T01:38:52.345 INFO:teuthology.orchestra.run.smithi045.stderr:Assimilating anything we can from ceph.conf... 2021-07-31T01:38:52.346 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/daemon-base:latest-pacific -e NODE_NAME=smithi045 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/f52d2c14-f19f-11eb-8c24-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmput3kn1b8:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp5r80c4w5:/etc/ceph/ceph.conf:z -v /var/lib/ceph/f52d2c14-f19f-11eb-8c24-001a4aab830c/mon.a:/var/lib/ceph/mon/ceph-a:z docker.io/ceph/daemon-base:latest-pacific config assimilate-conf -i /var/lib/ceph/mon/ceph-a/config 2021-07-31T01:38:53.010 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2021-07-31T01:38:53.010 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: [global] 2021-07-31T01:38:53.010 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: fsid = f52d2c14-f19f-11eb-8c24-001a4aab830c 2021-07-31T01:38:53.011 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: log_to_journald = false 2021-07-31T01:38:53.011 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.45:3300,v1:172.21.15.45:6789] 2021-07-31T01:38:53.011 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2021-07-31T01:38:53.011 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2021-07-31T01:38:53.012 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2021-07-31T01:38:53.012 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2021-07-31T01:38:53.012 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2021-07-31T01:38:53.012 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: [mgr] 2021-07-31T01:38:53.012 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2021-07-31T01:38:53.013 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2021-07-31T01:38:53.013 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: [osd] 2021-07-31T01:38:53.013 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2021-07-31T01:38:53.013 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2021-07-31T01:38:53.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:53 smithi045 conmon[33064]: audit 2021-07-31T01:38:53.005041+0000 mon.a (mon.0) 8 : audit 2021-07-31T01:38:53.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:53 smithi045 conmon[33064]: [INF] from='client.? 172.21.15.45:0/4107488902' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2021-07-31T01:38:53.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:53 smithi045 conmon[33064]: audit 2021-07-31T01:38:53.007804+0000 mon.a (mon.0) 9 : audit [INF] from='client.? 172.21.15.45:0/4107488902' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2021-07-31T01:38:53.684 INFO:teuthology.orchestra.run.smithi045.stderr:Generating new minimal ceph.conf... 2021-07-31T01:38:53.684 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/daemon-base:latest-pacific -e NODE_NAME=smithi045 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/f52d2c14-f19f-11eb-8c24-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmput3kn1b8:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp5r80c4w5:/etc/ceph/ceph.conf:z -v /var/lib/ceph/f52d2c14-f19f-11eb-8c24-001a4aab830c/mon.a:/var/lib/ceph/mon/ceph-a:z docker.io/ceph/daemon-base:latest-pacific config generate-minimal-conf -o /var/lib/ceph/mon/ceph-a/config 2021-07-31T01:38:54.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:54 smithi045 conmon[33064]: audit 2021-07-31T01:38:54.396590+0000 mon.a 2021-07-31T01:38:54.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:54 smithi045 conmon[33064]: (mon.0) 10 : audit [DBG] from='client.? 172.21.15.45:0/445911739' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:38:55.023 INFO:teuthology.orchestra.run.smithi045.stderr:Restarting the monitor... 2021-07-31T01:38:55.024 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: systemctl restart ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c@mon.a 2021-07-31T01:38:55.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:55 smithi045 systemd[1]: Stopping Ceph mon.a for f52d2c14-f19f-11eb-8c24-001a4aab830c... 2021-07-31T01:38:55.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:55 smithi045 conmon[33064]: debug 2021-07-31T01:38:55.140+0000 7f2361fc1700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2021-07-31T01:38:55.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:55 smithi045 conmon[33064]: debug 2021-07-31T01:38:55.140+0000 7f2361fc1700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2021-07-31T01:38:55.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:55 smithi045 podman[33453]: 2021-07-31 01:38:55.341340975 +0000 UTC m=+0.297378435 container stop 4b01d5c27484a5548cf48ce5bcabc287c8a7fb69fba679a15fbf0e1b01cbb61c (image=docker.io/ceph/daemon-base:latest-pacific, name=ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c-mon.a, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=master-0b3dfd0, maintainer=Dimitri Savineau , org.label-schema.build-date=20201204, org.label-schema.name=CentOS Base Image, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=0b3dfd099f6a46b8c8fecd28a50330b63408461f, ceph=True, GIT_BRANCH=HEAD) 2021-07-31T01:38:55.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:55 smithi045 podman[33453]: 2021-07-31 01:38:55.414249358 +0000 UTC m=+0.370286820 container died 4b01d5c27484a5548cf48ce5bcabc287c8a7fb69fba679a15fbf0e1b01cbb61c (image=docker.io/ceph/daemon-base:latest-pacific, name=ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c-mon.a) 2021-07-31T01:38:55.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:55 smithi045 podman[33453]: ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c-mon.a 2021-07-31T01:38:56.158 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:55 smithi045 podman[33495]: 2021-07-31 01:38:55.895148732 +0000 UTC m=+0.556678736 container remove 4b01d5c27484a5548cf48ce5bcabc287c8a7fb69fba679a15fbf0e1b01cbb61c (image=docker.io/ceph/daemon-base:latest-pacific, name=ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c-mon.a, RELEASE=master-0b3dfd0, org.label-schema.name=CentOS Base Image, CEPH_POINT_RELEASE=, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20201204, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=0b3dfd099f6a46b8c8fecd28a50330b63408461f, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Dimitri Savineau ) 2021-07-31T01:38:56.159 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:55 smithi045 systemd[1]: ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c@mon.a.service: Succeeded. 2021-07-31T01:38:56.159 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:55 smithi045 systemd[1]: Stopped Ceph mon.a for f52d2c14-f19f-11eb-8c24-001a4aab830c. 2021-07-31T01:38:56.159 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:55 smithi045 systemd[1]: Starting Ceph mon.a for f52d2c14-f19f-11eb-8c24-001a4aab830c... 2021-07-31T01:38:56.448 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 podman[33615]: 2021-07-31 01:38:56.068257948 +0000 UTC m=+0.031411723 image pull docker.io/ceph/daemon-base:latest-pacific 2021-07-31T01:38:56.449 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 podman[33615]: 2021-07-31 01:38:56.203374073 +0000 UTC m=+0.166527839 container create 98c0a6dfe7097e97052ea5141c226baa3683b6d40ddf043556b02573203cc932 (image=docker.io/ceph/daemon-base:latest-pacific, name=ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c-mon.a, RELEASE=master-0b3dfd0, ceph=True, maintainer=Dimitri Savineau , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20201204, GIT_COMMIT=0b3dfd099f6a46b8c8fecd28a50330b63408461f, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git) 2021-07-31T01:38:56.449 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 podman[33615]: 2021-07-31 01:38:56.336740987 +0000 UTC m=+0.299894849 container init 98c0a6dfe7097e97052ea5141c226baa3683b6d40ddf043556b02573203cc932 (image=docker.io/ceph/daemon-base:latest-pacific, name=ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c-mon.a, org.label-schema.build-date=20201204, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=0b3dfd099f6a46b8c8fecd28a50330b63408461f, GIT_CLEAN=True, org.label-schema.name=CentOS Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Dimitri Savineau , CEPH_POINT_RELEASE=, RELEASE=master-0b3dfd0) 2021-07-31T01:38:56.449 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.367+0000 7f23b4aa0700 0 set uid:gid to 167:167 (ceph:ceph) 2021-07-31T01:38:56.450 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.367+0000 7f23b4aa0700 0 ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable), process ceph-mon, pid 2 2021-07-31T01:38:56.450 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.367+0000 7f23b4aa0700 0 pidfile_write: ignore empty --pid-file 2021-07-31T01:38:56.450 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 0 load: jerasure load: lrc load: isa 2021-07-31T01:38:56.451 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: RocksDB version: 6.8.1 2021-07-31T01:38:56.451 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: 2021-07-31T01:38:56.451 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2021-07-31T01:38:56.452 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: Compile date Jul 8 2021 2021-07-31T01:38:56.452 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: DB SUMMARY 2021-07-31T01:38:56.452 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: 2021-07-31T01:38:56.453 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: CURRENT file: CURRENT 2021-07-31T01:38:56.453 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: 2021-07-31T01:38:56.454 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: IDENTITY file: IDENTITY 2021-07-31T01:38:56.455 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: 2021-07-31T01:38:56.455 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 131 Bytes 2021-07-31T01:38:56.455 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: 2021-07-31T01:38:56.456 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000004.sst 2021-07-31T01:38:56.456 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: 2021-07-31T01:38:56.456 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000006.log size: 86029 ; 2021-07-31T01:38:56.456 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: 2021-07-31T01:38:56.457 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: Options.error_if_exists: 0 2021-07-31T01:38:56.457 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: Options.create_if_missing: 0 2021-07-31T01:38:56.457 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: Options.paranoid_checks: 1 2021-07-31T01:38:56.457 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: Options.env: 0x5565ad8891c0 2021-07-31T01:38:56.458 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: Options.fs: Posix File System 2021-07-31T01:38:56.458 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: Options.info_log: 0x5565aef5a6a0 2021-07-31T01:38:56.458 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: Options.max_file_opening_threads: 16 2021-07-31T01:38:56.458 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: Options.statistics: (nil) 2021-07-31T01:38:56.459 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: Options.use_fsync: 0 2021-07-31T01:38:56.459 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: Options.max_log_file_size: 0 2021-07-31T01:38:56.459 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2021-07-31T01:38:56.459 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.459 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: Options.log_file_time_to_roll: 0 2021-07-31T01:38:56.460 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: Options.keep_log_file_num: 1000 2021-07-31T01:38:56.460 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: Options.recycle_log_file_num: 0 2021-07-31T01:38:56.460 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: Options.allow_fallocate: 1 2021-07-31T01:38:56.460 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: Options.allow_mmap_reads: 0 2021-07-31T01:38:56.460 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.461 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: Options.allow_mmap_writes: 0 2021-07-31T01:38:56.461 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: Options.use_direct_reads: 0 2021-07-31T01:38:56.461 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2021-07-31T01:38:56.462 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: Options.create_missing_column_families: 0 2021-07-31T01:38:56.462 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: Options.db_log_dir: 2021-07-31T01:38:56.462 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.463 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2021-07-31T01:38:56.463 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: Options.table_cache_numshardbits: 6 2021-07-31T01:38:56.463 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: Options.max_subcompactions: 1 2021-07-31T01:38:56.463 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: Options.max_background_flushes: -1 2021-07-31T01:38:56.464 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: Options.WAL_ttl_seconds: 0 2021-07-31T01:38:56.464 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: Options.WAL_size_limit_MB: 0 2021-07-31T01:38:56.465 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2021-07-31T01:38:56.465 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: 2021-07-31T01:38:56.465 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2021-07-31T01:38:56.465 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: Options.is_fd_close_on_exec: 1 2021-07-31T01:38:56.466 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: Options.advise_random_on_open: 1 2021-07-31T01:38:56.466 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: Options.db_write_buffer_size: 0 2021-07-31T01:38:56.466 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: Options.write_buffer_manager: 0x5565aefedf80 2021-07-31T01:38:56.467 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2021-07-31T01:38:56.467 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2021-07-31T01:38:56.467 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2021-07-31T01:38:56.467 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: Options.use_adaptive_mutex: 0 2021-07-31T01:38:56.468 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: Options.rate_limiter: (nil) 2021-07-31T01:38:56.468 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.468 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2021-07-31T01:38:56.468 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: Options.wal_recovery_mode: 2 2021-07-31T01:38:56.468 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: Options.enable_thread_tracking: 0 2021-07-31T01:38:56.469 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: Options.enable_pipelined_write: 0 2021-07-31T01:38:56.469 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: Options.unordered_write: 0 2021-07-31T01:38:56.469 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2021-07-31T01:38:56.469 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2021-07-31T01:38:56.470 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2021-07-31T01:38:56.470 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2021-07-31T01:38:56.470 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: Options.row_cache: None 2021-07-31T01:38:56.470 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: Options.wal_filter: None 2021-07-31T01:38:56.471 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2021-07-31T01:38:56.471 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: 2021-07-31T01:38:56.471 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: Options.allow_ingest_behind: 0 2021-07-31T01:38:56.471 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: Options.preserve_deletes: 0 2021-07-31T01:38:56.472 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: Options.two_write_queues: 0 2021-07-31T01:38:56.472 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: Options.manual_wal_flush: 0 2021-07-31T01:38:56.472 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: Options.atomic_flush: 0 2021-07-31T01:38:56.472 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2021-07-31T01:38:56.473 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: Options.persist_stats_to_disk: 0 2021-07-31T01:38:56.473 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.473 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: Options.write_dbid_to_manifest: 0 2021-07-31T01:38:56.473 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: Options.log_readahead_size: 0 2021-07-31T01:38:56.473 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2021-07-31T01:38:56.474 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: Options.max_background_jobs: 2 2021-07-31T01:38:56.474 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 bash[33542]: 98c0a6dfe7097e97052ea5141c226baa3683b6d40ddf043556b02573203cc932 2021-07-31T01:38:56.474 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: Options.max_background_compactions: -1 2021-07-31T01:38:56.474 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2021-07-31T01:38:56.475 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2021-07-31T01:38:56.475 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: Options.delayed_write_rate : 16777216 2021-07-31T01:38:56.476 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: Options.max_total_wal_size: 0 2021-07-31T01:38:56.476 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2021-07-31T01:38:56.476 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: Options.stats_dump_period_sec: 600 2021-07-31T01:38:56.476 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: Options.stats_persist_period_sec: 600 2021-07-31T01:38:56.477 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2021-07-31T01:38:56.477 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: Options.max_open_files: -1 2021-07-31T01:38:56.477 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: Options.bytes_per_sync: 0 2021-07-31T01:38:56.478 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: Options.wal_bytes_per_sync: 0 2021-07-31T01:38:56.478 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: Options.strict_bytes_per_sync: 0 2021-07-31T01:38:56.478 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: Options.compaction_readahead_size: 0 2021-07-31T01:38:56.478 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: Compression algorithms supported: 2021-07-31T01:38:56.479 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2021-07-31T01:38:56.479 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: kZSTD supported: 0 2021-07-31T01:38:56.479 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.479 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: kXpressCompression supported: 0 2021-07-31T01:38:56.479 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: kLZ4HCCompression supported: 1 2021-07-31T01:38:56.481 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: kLZ4Compression supported: 1 2021-07-31T01:38:56.482 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: kBZip2Compression supported: 0 2021-07-31T01:38:56.482 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: kZlibCompression supported: 1 2021-07-31T01:38:56.482 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: kSnappyCompression supported: 1 2021-07-31T01:38:56.482 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: Fast CRC32 supported: Supported on x86 2021-07-31T01:38:56.482 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.372+0000 7f23b4aa0700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000005 2021-07-31T01:38:56.483 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: 2021-07-31T01:38:56.483 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.373+0000 7f23b4aa0700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2021-07-31T01:38:56.483 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: 2021-07-31T01:38:56.484 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.373+0000 7f23b4aa0700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2021-07-31T01:38:56.484 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.373+0000 7f23b4aa0700 4 rocksdb: Options.merge_operator: 2021-07-31T01:38:56.484 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.373+0000 7f23b4aa0700 4 rocksdb: Options.compaction_filter: None 2021-07-31T01:38:56.484 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.373+0000 7f23b4aa0700 4 rocksdb: Options.compaction_filter_factory: None 2021-07-31T01:38:56.485 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.373+0000 7f23b4aa0700 4 rocksdb: Options.memtable_factory: SkipListFactory 2021-07-31T01:38:56.485 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.373+0000 7f23b4aa0700 4 rocksdb: Options.table_factory: BlockBasedTable 2021-07-31T01:38:56.486 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.373+0000 7f23b4aa0700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5565aeef9cc0) 2021-07-31T01:38:56.486 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: cache_index_and_filter_blocks: 1 2021-07-31T01:38:56.486 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: cache_index_and_filter_blocks_with_high_priority: 0 2021-07-31T01:38:56.487 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: pin_l0_filter_and_index_blocks_in_cache: 0 2021-07-31T01:38:56.487 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: pin_top_level_index_and_filter: 1 2021-07-31T01:38:56.487 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: index_type: 0 2021-07-31T01:38:56.488 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: data_block_index_type: 0 2021-07-31T01:38:56.488 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: index_shortening: 1 2021-07-31T01:38:56.488 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: data_block_hash_table_util_ratio: 0.750000 2021-07-31T01:38:56.488 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: hash_index_allow_collision: 1 2021-07-31T01:38:56.489 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: checksum: 1 2021-07-31T01:38:56.489 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: no_block_cache: 0 2021-07-31T01:38:56.489 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: block_cache: 0x5565aef32f10 2021-07-31T01:38:56.490 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: block_cache_name: BinnedLRUCache 2021-07-31T01:38:56.490 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: block_cache_options: 2021-07-31T01:38:56.490 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: capacity : 536870912 2021-07-31T01:38:56.490 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: num_shard_bits : 4 2021-07-31T01:38:56.491 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: strict_capacity_limit : 0 2021-07-31T01:38:56.491 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: high_pri_pool_ratio: 0.000 2021-07-31T01:38:56.491 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: block_cache_compressed: (nil) 2021-07-31T01:38:56.491 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: persistent_cache: (nil) 2021-07-31T01:38:56.492 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: block_size: 4096 2021-07-31T01:38:56.492 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: block_size_deviation: 10 2021-07-31T01:38:56.492 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: block_restart_interval: 16 2021-07-31T01:38:56.493 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: index_block_restart_interval: 1 2021-07-31T01:38:56.493 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: metadata_block_size: 4096 2021-07-31T01:38:56.493 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: partition_filters: 0 2021-07-31T01:38:56.493 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: use_delta_encoding: 1 2021-07-31T01:38:56.494 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: filter_policy: rocksdb.BuiltinBloomFilter 2021-07-31T01:38:56.494 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: whole_key_filtering: 1 2021-07-31T01:38:56.495 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: verify_compression: 0 2021-07-31T01:38:56.495 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: read_amp_bytes_per_bit: 0 2021-07-31T01:38:56.495 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: format_version: 2 2021-07-31T01:38:56.496 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: enable_index_compression: 1 2021-07-31T01:38:56.499 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: block_align: 0 2021-07-31T01:38:56.502 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: 2021-07-31T01:38:56.503 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.373+0000 7f23b4aa0700 4 rocksdb: Options.write_buffer_size: 33554432 2021-07-31T01:38:56.503 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.373+0000 7f23b4aa0700 4 rocksdb: Options.max_write_buffer_number: 2 2021-07-31T01:38:56.503 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.373+0000 7f23b4aa0700 4 rocksdb: Options.compression: NoCompression 2021-07-31T01:38:56.504 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.373+0000 7f23b4aa0700 4 rocksdb: Options.bottommost_compression: Disabled 2021-07-31T01:38:56.504 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.373+0000 7f23b4aa0700 4 rocksdb: Options.prefix_extractor: nullptr 2021-07-31T01:38:56.504 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: 2021-07-31T01:38:56.505 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.373+0000 7f23b4aa0700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2021-07-31T01:38:56.505 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.373+0000 7f23b4aa0700 4 rocksdb: Options.num_levels: 7 2021-07-31T01:38:56.505 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.373+0000 7f23b4aa0700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2021-07-31T01:38:56.505 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.373+0000 7f23b4aa0700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2021-07-31T01:38:56.506 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.373+0000 7f23b4aa0700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2021-07-31T01:38:56.507 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.373+0000 7f23b4aa0700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2021-07-31T01:38:56.507 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.373+0000 7f23b4aa0700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2021-07-31T01:38:56.507 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.373+0000 7f23b4aa0700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2021-07-31T01:38:56.508 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.373+0000 7f23b4aa0700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2021-07-31T01:38:56.508 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.373+0000 7f23b4aa0700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2021-07-31T01:38:56.508 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.373+0000 7f23b4aa0700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2021-07-31T01:38:56.508 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.373+0000 7f23b4aa0700 4 rocksdb: Options.compression_opts.window_bits: -14 2021-07-31T01:38:56.508 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.373+0000 7f23b4aa0700 4 rocksdb: Options.compression_opts.level: 32767 2021-07-31T01:38:56.509 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.373+0000 7f23b4aa0700 4 rocksdb: Options.compression_opts.strategy: 0 2021-07-31T01:38:56.509 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.373+0000 7f23b4aa0700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2021-07-31T01:38:56.509 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.373+0000 7f23b4aa0700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2021-07-31T01:38:56.509 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.373+0000 7f23b4aa0700 4 rocksdb: Options.compression_opts.enabled: false 2021-07-31T01:38:56.510 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.373+0000 7f23b4aa0700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2021-07-31T01:38:56.510 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.373+0000 7f23b4aa0700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2021-07-31T01:38:56.510 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.373+0000 7f23b4aa0700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2021-07-31T01:38:56.510 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.373+0000 7f23b4aa0700 4 rocksdb: Options.target_file_size_base: 67108864 2021-07-31T01:38:56.510 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.373+0000 7f23b4aa0700 4 rocksdb: Options.target_file_size_multiplier: 1 2021-07-31T01:38:56.511 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.373+0000 7f23b4aa0700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2021-07-31T01:38:56.511 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.373+0000 7f23b4aa0700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2021-07-31T01:38:56.511 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.373+0000 7f23b4aa0700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2021-07-31T01:38:56.511 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.373+0000 7f23b4aa0700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2021-07-31T01:38:56.511 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.373+0000 7f23b4aa0700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2021-07-31T01:38:56.512 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.373+0000 7f23b4aa0700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2021-07-31T01:38:56.512 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.373+0000 7f23b4aa0700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2021-07-31T01:38:56.513 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.373+0000 7f23b4aa0700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2021-07-31T01:38:56.513 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.373+0000 7f23b4aa0700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2021-07-31T01:38:56.513 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.373+0000 7f23b4aa0700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2021-07-31T01:38:56.513 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.373+0000 7f23b4aa0700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2021-07-31T01:38:56.514 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.373+0000 7f23b4aa0700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2021-07-31T01:38:56.514 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.373+0000 7f23b4aa0700 4 rocksdb: Options.arena_block_size: 4194304 2021-07-31T01:38:56.514 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.373+0000 7f23b4aa0700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2021-07-31T01:38:56.514 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.373+0000 7f23b4aa0700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2021-07-31T01:38:56.515 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.373+0000 7f23b4aa0700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2021-07-31T01:38:56.515 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.373+0000 7f23b4aa0700 4 rocksdb: Options.disable_auto_compactions: 0 2021-07-31T01:38:56.515 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.373+0000 7f23b4aa0700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2021-07-31T01:38:56.516 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.373+0000 7f23b4aa0700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2021-07-31T01:38:56.516 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: 2021-07-31T01:38:56.516 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.373+0000 7f23b4aa0700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2021-07-31T01:38:56.516 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.373+0000 7f23b4aa0700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2021-07-31T01:38:56.517 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.373+0000 7f23b4aa0700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2021-07-31T01:38:56.517 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.373+0000 7f23b4aa0700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2021-07-31T01:38:56.517 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.373+0000 7f23b4aa0700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2021-07-31T01:38:56.517 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.373+0000 7f23b4aa0700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2021-07-31T01:38:56.517 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.373+0000 7f23b4aa0700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2021-07-31T01:38:56.518 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.373+0000 7f23b4aa0700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2021-07-31T01:38:56.518 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.373+0000 7f23b4aa0700 4 rocksdb: Options.table_properties_collectors: 2021-07-31T01:38:56.518 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.373+0000 7f23b4aa0700 4 rocksdb: Options.inplace_update_support: 0 2021-07-31T01:38:56.518 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: 2021-07-31T01:38:56.519 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.373+0000 7f23b4aa0700 4 rocksdb: Options.inplace_update_num_locks: 10000 2021-07-31T01:38:56.519 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.373+0000 7f23b4aa0700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2021-07-31T01:38:56.519 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.373+0000 7f23b4aa0700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2021-07-31T01:38:56.519 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.373+0000 7f23b4aa0700 4 rocksdb: Options.memtable_huge_page_size: 0 2021-07-31T01:38:56.520 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.373+0000 7f23b4aa0700 4 rocksdb: Options.bloom_locality: 0 2021-07-31T01:38:56.520 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.373+0000 7f23b4aa0700 4 rocksdb: Options.max_successive_merges: 0 2021-07-31T01:38:56.520 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.373+0000 7f23b4aa0700 4 rocksdb: Options.optimize_filters_for_hits: 0 2021-07-31T01:38:56.520 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.373+0000 7f23b4aa0700 4 rocksdb: Options.paranoid_file_checks: 0 2021-07-31T01:38:56.521 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.373+0000 7f23b4aa0700 4 rocksdb: Options.force_consistency_checks: 0 2021-07-31T01:38:56.521 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.373+0000 7f23b4aa0700 4 rocksdb: Options.report_bg_io_stats: 0 2021-07-31T01:38:56.521 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.373+0000 7f23b4aa0700 4 rocksdb: Options.ttl: 2592000 2021-07-31T01:38:56.522 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.373+0000 7f23b4aa0700 4 rocksdb: Options.periodic_compaction_seconds: 0 2021-07-31T01:38:56.522 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.373+0000 7f23b4aa0700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 5, log_number is 4,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2021-07-31T01:38:56.522 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: 2021-07-31T01:38:56.522 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.373+0000 7f23b4aa0700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 4 2021-07-31T01:38:56.523 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: 2021-07-31T01:38:56.523 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.373+0000 7f23b4aa0700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1627695536375072, "job": 1, "event": "recovery_started", "log_files": [6]} 2021-07-31T01:38:56.523 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.524 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: 2021-07-31T01:38:56.373+0000 7f23b4aa0700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #6 mode 2 2021-07-31T01:38:56.524 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.374+0000 7f23b4aa0700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2021-07-31T01:38:56.524 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.375+0000 7f23b4aa0700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1627695536376323, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 7, "file_size": 82633, "table_properties": {"data_size": 80902, "index_size": 336, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 581, "raw_key_size": 9714, "raw_average_key_size": 47, "raw_value_size": 75328, "raw_average_value_size": 369, "num_data_blocks": 11, "num_entries": 204, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1627695536, "oldest_key_time": 3, "file_creation_time": 0}} 2021-07-31T01:38:56.524 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.375+0000 7f23b4aa0700 4 rocksdb: [version_set.cc:3826] Creating manifest 8 2021-07-31T01:38:56.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: 2021-07-31T01:38:56.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.375+0000 7f23b4aa0700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1627695536376885, "job": 1, "event": "recovery_finished"} 2021-07-31T01:38:56.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.376+0000 7f23b4aa0700 4 rocksdb: DB pointer 0x5565aeff9800 2021-07-31T01:38:56.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.376+0000 7f239b8a8700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2021-07-31T01:38:56.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.376+0000 7f239b8a8700 4 rocksdb: [db_impl/db_impl.cc:851] 2021-07-31T01:38:56.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: ** DB Stats ** 2021-07-31T01:38:56.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: Uptime(secs): 0.0 total, 0.0 interval 2021-07-31T01:38:56.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2021-07-31T01:38:56.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2021-07-31T01:38:56.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2021-07-31T01:38:56.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2021-07-31T01:38:56.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2021-07-31T01:38:56.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2021-07-31T01:38:56.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: 2021-07-31T01:38:56.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: ** Compaction Stats [default] ** 2021-07-31T01:38:56.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-07-31T01:38:56.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-07-31T01:38:56.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: L0 2/0 82.35 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 98.5 0.00 0.00 1 0.001 0 0 2021-07-31T01:38:56.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: Sum 2/0 82.35 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 98.5 0.00 0.00 1 0.001 0 0 2021-07-31T01:38:56.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 98.5 0.00 0.00 1 0.001 0 0 2021-07-31T01:38:56.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: 2021-07-31T01:38:56.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: ** Compaction Stats [default] ** 2021-07-31T01:38:56.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-07-31T01:38:56.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-07-31T01:38:56.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 98.5 0.00 0.00 1 0.001 0 0 2021-07-31T01:38:56.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: Uptime(secs): 0.0 total, 0.0 interval 2021-07-31T01:38:56.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: Flush(GB): cumulative 0.000, interval 0.000 2021-07-31T01:38:56.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: AddFile(GB): cumulative 0.000, interval 0.000 2021-07-31T01:38:56.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: AddFile(Total Files): cumulative 0, interval 0 2021-07-31T01:38:56.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: AddFile(L0 Files): cumulative 0, interval 0 2021-07-31T01:38:56.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: AddFile(Keys): cumulative 0, interval 0 2021-07-31T01:38:56.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: Cumulative compaction: 0.00 GB write, 21.02 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-07-31T01:38:56.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: Interval compaction: 0.00 GB write, 21.02 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-07-31T01:38:56.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2021-07-31T01:38:56.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: 2021-07-31T01:38:56.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: ** File Read Latency Histogram By Level [default] ** 2021-07-31T01:38:56.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: 2021-07-31T01:38:56.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: ** Compaction Stats [default] ** 2021-07-31T01:38:56.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-07-31T01:38:56.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-07-31T01:38:56.535 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: L0 2/0 82.35 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 98.5 0.00 0.00 1 0.001 0 0 2021-07-31T01:38:56.535 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: Sum 2/0 82.35 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 98.5 0.00 0.00 1 0.001 0 0 2021-07-31T01:38:56.535 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2021-07-31T01:38:56.535 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: 2021-07-31T01:38:56.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: ** Compaction Stats [default] ** 2021-07-31T01:38:56.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-07-31T01:38:56.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-07-31T01:38:56.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 98.5 0.00 0.00 1 0.001 0 0 2021-07-31T01:38:56.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: Uptime(secs): 0.0 total, 0.0 interval 2021-07-31T01:38:56.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: Flush(GB): cumulative 0.000, interval 0.000 2021-07-31T01:38:56.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: AddFile(GB): cumulative 0.000, interval 0.000 2021-07-31T01:38:56.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: AddFile(Total Files): cumulative 0, interval 0 2021-07-31T01:38:56.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: AddFile(L0 Files): cumulative 0, interval 0 2021-07-31T01:38:56.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: AddFile(Keys): cumulative 0, interval 0 2021-07-31T01:38:56.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: Cumulative compaction: 0.00 GB write, 20.86 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-07-31T01:38:56.539 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-07-31T01:38:56.539 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2021-07-31T01:38:56.539 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: 2021-07-31T01:38:56.539 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: ** File Read Latency Histogram By Level [default] ** 2021-07-31T01:38:56.540 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: 2021-07-31T01:38:56.540 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 podman[33615]: 2021-07-31 01:38:56.378315907 +0000 UTC m=+0.341469671 container start 98c0a6dfe7097e97052ea5141c226baa3683b6d40ddf043556b02573203cc932 (image=docker.io/ceph/daemon-base:latest-pacific, name=ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c-mon.a, maintainer=Dimitri Savineau , org.label-schema.build-date=20201204, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=0b3dfd099f6a46b8c8fecd28a50330b63408461f, RELEASE=master-0b3dfd0) 2021-07-31T01:38:56.540 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.377+0000 7f23b4aa0700 0 starting mon.a rank 0 at public addrs [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] at bind addrs [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid f52d2c14-f19f-11eb-8c24-001a4aab830c 2021-07-31T01:38:56.540 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.379+0000 7f23b4aa0700 1 mon.a@-1(???) e1 preinit fsid f52d2c14-f19f-11eb-8c24-001a4aab830c 2021-07-31T01:38:56.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.379+0000 7f23b4aa0700 0 mon.a@-1(???).mds e1 new map 2021-07-31T01:38:56.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.379+0000 7f23b4aa0700 0 mon.a@-1(???).mds e1 print_map 2021-07-31T01:38:56.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: e1 2021-07-31T01:38:56.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: enable_multiple, ever_enabled_multiple: 1,1 2021-07-31T01:38:56.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2021-07-31T01:38:56.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: legacy client fscid: -1 2021-07-31T01:38:56.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: 2021-07-31T01:38:56.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: No filesystems configured 2021-07-31T01:38:56.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: 2021-07-31T01:38:56.543 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.379+0000 7f23b4aa0700 0 mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2021-07-31T01:38:56.543 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.543 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: 2021-07-31T01:38:56.379+0000 7f23b4aa0700 0 mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2021-07-31T01:38:56.544 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.379+0000 7f23b4aa0700 0 mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2021-07-31T01:38:56.544 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.379+0000 7f23b4aa0700 0 mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2021-07-31T01:38:56.544 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: audit 2021-07-31T01:38:56.544 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: 2021-07-31 2021-07-31T01:38:56.545 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: T01:38:54.396590 2021-07-31T01:38:56.545 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: +0000 mon.a (mon. 2021-07-31T01:38:56.545 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: 0) 10 : audit [DBG] 2021-07-31T01:38:56.545 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: from='client.? 172.21.15.45:0/445911739' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:38:56.546 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.379+0000 7f23b4aa0700 1 mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2021-07-31T01:38:56.546 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.381+0000 7f23b4aa0700 20 expand_channel_meta expand map: {default=false} 2021-07-31T01:38:56.546 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.381+0000 7f23b4aa0700 20 expand_channel_meta from 'false' to 'false' 2021-07-31T01:38:56.546 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.381+0000 7f23b4aa0700 20 expand_channel_meta expanded map: {default=false} 2021-07-31T01:38:56.546 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.381+0000 7f23b4aa0700 20 expand_channel_meta expand map: {default=info} 2021-07-31T01:38:56.547 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.381+0000 7f23b4aa0700 20 expand_channel_meta from 'info' to 'info' 2021-07-31T01:38:56.547 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.381+0000 7f23b4aa0700 20 expand_channel_meta expanded map: {default=info} 2021-07-31T01:38:56.547 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.381+0000 7f23b4aa0700 20 expand_channel_meta expand map: {default=daemon} 2021-07-31T01:38:56.547 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.381+0000 7f23b4aa0700 20 expand_channel_meta from 'daemon' to 'daemon' 2021-07-31T01:38:56.548 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.381+0000 7f23b4aa0700 20 expand_channel_meta expanded map: {default=daemon} 2021-07-31T01:38:56.548 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.381+0000 7f23b4aa0700 20 expand_channel_meta expand map: {default=debug} 2021-07-31T01:38:56.548 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.381+0000 7f23b4aa0700 20 expand_channel_meta from 'debug' to 'debug' 2021-07-31T01:38:56.548 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: debug 2021-07-31T01:38:56.381+0000 7f23b4aa0700 20 expand_channel_meta expanded map: {default=debug} 2021-07-31T01:38:56.549 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: cluster 2021-07-31T01:38: 2021-07-31T01:38:56.549 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: 56.396790+0000 mon.a (mon.0) 1 : cluster [INF] mon.a is new leader, mons a in quorum (ranks 0) 2021-07-31T01:38:56.549 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: cluster 2021-07-31T01:38: 2021-07-31T01:38:56.549 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: 56.396829+0000 mon.a (mon.0) 2 2021-07-31T01:38:56.549 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: : cluster [DBG] monmap e1: 1 mons at {a=[v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0]} 2021-07-31T01:38:56.550 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: cluster 2021-07-31T01:38:56.550 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: 2021-07-31T01:38:56. 2021-07-31T01:38:56.550 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: 396861+0000 mon.a (mon.0) 3 : cluster 2021-07-31T01:38:56.550 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: [DBG] fsmap 2021-07-31T01:38:56.551 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: cluster 2021-07-31T01:38:56.551 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: 2021-07- 2021-07-31T01:38:56.551 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: 31T01:38:56.396890+0000 mon.a 2021-07-31T01:38:56.551 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: (mon.0) 4 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2021-07-31T01:38:56.551 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: 2021-07-31T01:38:56.552 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: cluster 2021-07-31T01: 2021-07-31T01:38:56.552 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: 38:56.397207+0000 mon.a (mon.0) 2021-07-31T01:38:56.552 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 conmon[33655]: 5 : cluster [DBG] mgrmap e1: no daemons active 2021-07-31T01:38:56.553 INFO:teuthology.orchestra.run.smithi045.stderr:Setting mon public_network to 172.21.0.0/20 2021-07-31T01:38:56.553 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/daemon-base:latest-pacific -e NODE_NAME=smithi045 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/f52d2c14-f19f-11eb-8c24-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmput3kn1b8:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp5r80c4w5:/etc/ceph/ceph.conf:z docker.io/ceph/daemon-base:latest-pacific config set mon public_network 172.21.0.0/20 2021-07-31T01:38:56.714 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:56 smithi045 systemd[1]: Started Ceph mon.a for f52d2c14-f19f-11eb-8c24-001a4aab830c. 2021-07-31T01:38:57.903 INFO:teuthology.orchestra.run.smithi045.stderr:Wrote config to /etc/ceph/ceph.conf 2021-07-31T01:38:57.903 INFO:teuthology.orchestra.run.smithi045.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2021-07-31T01:38:57.904 INFO:teuthology.orchestra.run.smithi045.stderr:Creating mgr... 2021-07-31T01:38:57.905 INFO:teuthology.orchestra.run.smithi045.stderr:Verifying port 9283 ... 2021-07-31T01:38:57.906 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: systemctl daemon-reload 2021-07-31T01:38:58.002 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: systemctl stop ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c@mgr.y 2021-07-31T01:38:58.033 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: systemctl reset-failed ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c@mgr.y 2021-07-31T01:38:58.039 INFO:teuthology.orchestra.run.smithi045.stderr:systemctl: Failed to reset failed state of unit ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c@mgr.y.service: Unit ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c@mgr.y.service not loaded. 2021-07-31T01:38:58.039 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: systemctl enable ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c@mgr.y 2021-07-31T01:38:58.046 INFO:teuthology.orchestra.run.smithi045.stderr:systemctl: Created symlink /etc/systemd/system/ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c.target.wants/ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c@mgr.y.service → /etc/systemd/system/ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c@.service. 2021-07-31T01:38:58.131 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: systemctl start ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c@mgr.y 2021-07-31T01:38:58.437 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:58 smithi045 conmon[33655]: audit 2021-07-31T01:38:57.196982+0000 2021-07-31T01:38:58.437 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:58 smithi045 conmon[33655]: mon.a (mon.0) 6 : audit [INF] from='client.? 172.21.15.45:0/3035464346' entity='client.admin' 2021-07-31T01:38:58.437 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Jul 31 01:38:58 smithi045 systemd[1]: Starting Ceph mgr.y for f52d2c14-f19f-11eb-8c24-001a4aab830c... 2021-07-31T01:38:58.438 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Jul 31 01:38:58 smithi045 podman[33939]: 2021-07-31 01:38:58.277679233 +0000 UTC m=+0.030623001 image pull docker.io/ceph/daemon-base:latest-pacific 2021-07-31T01:38:58.438 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Jul 31 01:38:58 smithi045 podman[33939]: 2021-07-31 01:38:58.436373217 +0000 UTC m=+0.189317091 container create 4472872c68fa40a752e9f659d4c4d35e9bd0bccf8272e16f03235056118534ec (image=docker.io/ceph/daemon-base:latest-pacific, name=ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c-mgr.y, GIT_COMMIT=0b3dfd099f6a46b8c8fecd28a50330b63408461f, maintainer=Dimitri Savineau , org.label-schema.build-date=20201204, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=master-0b3dfd0, ceph=True, org.label-schema.vendor=CentOS) 2021-07-31T01:38:58.872 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: systemctl is-enabled firewalld.service 2021-07-31T01:38:58.881 INFO:teuthology.orchestra.run.smithi045.stderr:systemctl: disabled 2021-07-31T01:38:58.882 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: systemctl is-active firewalld.service 2021-07-31T01:38:58.889 INFO:teuthology.orchestra.run.smithi045.stderr:systemctl: inactive 2021-07-31T01:38:58.890 INFO:teuthology.orchestra.run.smithi045.stderr:firewalld.service is not enabled 2021-07-31T01:38:58.890 INFO:teuthology.orchestra.run.smithi045.stderr:Not possible to enable service . firewalld.service is not available 2021-07-31T01:38:58.891 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: systemctl is-enabled firewalld.service 2021-07-31T01:38:58.899 INFO:teuthology.orchestra.run.smithi045.stderr:systemctl: disabled 2021-07-31T01:38:58.900 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: systemctl is-active firewalld.service 2021-07-31T01:38:58.907 INFO:teuthology.orchestra.run.smithi045.stderr:systemctl: inactive 2021-07-31T01:38:58.910 INFO:teuthology.orchestra.run.smithi045.stderr:firewalld.service is not enabled 2021-07-31T01:38:58.910 INFO:teuthology.orchestra.run.smithi045.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2021-07-31T01:38:58.910 INFO:teuthology.orchestra.run.smithi045.stderr:Waiting for mgr to start... 2021-07-31T01:38:58.911 INFO:teuthology.orchestra.run.smithi045.stderr:Waiting for mgr... 2021-07-31T01:38:58.911 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/daemon-base:latest-pacific -e NODE_NAME=smithi045 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/f52d2c14-f19f-11eb-8c24-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmput3kn1b8:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp5r80c4w5:/etc/ceph/ceph.conf:z docker.io/ceph/daemon-base:latest-pacific status -f json-pretty 2021-07-31T01:38:59.048 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Jul 31 01:38:58 smithi045 podman[33939]: 2021-07-31 01:38:58.77804432 +0000 UTC m=+0.530988211 container init 4472872c68fa40a752e9f659d4c4d35e9bd0bccf8272e16f03235056118534ec (image=docker.io/ceph/daemon-base:latest-pacific, name=ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c-mgr.y, org.label-schema.name=CentOS Base Image, maintainer=Dimitri Savineau , ceph=True, GIT_COMMIT=0b3dfd099f6a46b8c8fecd28a50330b63408461f, GIT_BRANCH=HEAD, RELEASE=master-0b3dfd0, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20201204, org.label-schema.license=GPLv2) 2021-07-31T01:38:59.048 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Jul 31 01:38:58 smithi045 podman[33939]: 2021-07-31 01:38:58.827951354 +0000 UTC m=+0.580895275 container start 4472872c68fa40a752e9f659d4c4d35e9bd0bccf8272e16f03235056118534ec (image=docker.io/ceph/daemon-base:latest-pacific, name=ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c-mgr.y, RELEASE=master-0b3dfd0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20201204, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=0b3dfd099f6a46b8c8fecd28a50330b63408461f, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Dimitri Savineau ) 2021-07-31T01:38:59.049 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Jul 31 01:38:58 smithi045 bash[33864]: 4472872c68fa40a752e9f659d4c4d35e9bd0bccf8272e16f03235056118534ec 2021-07-31T01:38:59.049 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Jul 31 01:38:58 smithi045 systemd[1]: Started Ceph mgr.y for f52d2c14-f19f-11eb-8c24-001a4aab830c. 2021-07-31T01:38:59.640 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2021-07-31T01:38:59.641 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: { 2021-07-31T01:38:59.641 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "fsid": "f52d2c14-f19f-11eb-8c24-001a4aab830c", 2021-07-31T01:38:59.641 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "health": { 2021-07-31T01:38:59.642 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2021-07-31T01:38:59.642 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "checks": {}, 2021-07-31T01:38:59.642 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "mutes": [] 2021-07-31T01:38:59.642 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: }, 2021-07-31T01:38:59.643 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "election_epoch": 5, 2021-07-31T01:38:59.643 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "quorum": [ 2021-07-31T01:38:59.643 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 0 2021-07-31T01:38:59.643 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: ], 2021-07-31T01:38:59.643 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "quorum_names": [ 2021-07-31T01:38:59.644 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "a" 2021-07-31T01:38:59.644 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: ], 2021-07-31T01:38:59.644 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "quorum_age": 3, 2021-07-31T01:38:59.644 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "monmap": { 2021-07-31T01:38:59.645 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "epoch": 1, 2021-07-31T01:38:59.645 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2021-07-31T01:38:59.645 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "num_mons": 1 2021-07-31T01:38:59.645 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: }, 2021-07-31T01:38:59.646 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "osdmap": { 2021-07-31T01:38:59.646 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "epoch": 1, 2021-07-31T01:38:59.646 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "num_osds": 0, 2021-07-31T01:38:59.646 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "num_up_osds": 0, 2021-07-31T01:38:59.646 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "osd_up_since": 0, 2021-07-31T01:38:59.647 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "num_in_osds": 0, 2021-07-31T01:38:59.647 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "osd_in_since": 0, 2021-07-31T01:38:59.647 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2021-07-31T01:38:59.647 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: }, 2021-07-31T01:38:59.648 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "pgmap": { 2021-07-31T01:38:59.648 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "pgs_by_state": [], 2021-07-31T01:38:59.648 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "num_pgs": 0, 2021-07-31T01:38:59.648 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "num_pools": 0, 2021-07-31T01:38:59.649 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "num_objects": 0, 2021-07-31T01:38:59.649 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "data_bytes": 0, 2021-07-31T01:38:59.649 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "bytes_used": 0, 2021-07-31T01:38:59.649 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "bytes_avail": 0, 2021-07-31T01:38:59.649 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "bytes_total": 0 2021-07-31T01:38:59.650 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: }, 2021-07-31T01:38:59.650 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "fsmap": { 2021-07-31T01:38:59.650 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "epoch": 1, 2021-07-31T01:38:59.650 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "by_rank": [], 2021-07-31T01:38:59.651 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "up:standby": 0 2021-07-31T01:38:59.651 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: }, 2021-07-31T01:38:59.651 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "mgrmap": { 2021-07-31T01:38:59.651 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "available": false, 2021-07-31T01:38:59.652 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "num_standbys": 0, 2021-07-31T01:38:59.652 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "modules": [ 2021-07-31T01:38:59.653 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "iostat", 2021-07-31T01:38:59.653 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "nfs", 2021-07-31T01:38:59.653 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "restful" 2021-07-31T01:38:59.653 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: ], 2021-07-31T01:38:59.653 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "services": {} 2021-07-31T01:38:59.654 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: }, 2021-07-31T01:38:59.654 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "servicemap": { 2021-07-31T01:38:59.654 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "epoch": 1, 2021-07-31T01:38:59.654 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "modified": "2021-07-31T01:38:50.681462+0000", 2021-07-31T01:38:59.655 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "services": {} 2021-07-31T01:38:59.655 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: }, 2021-07-31T01:38:59.655 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "progress_events": {} 2021-07-31T01:38:59.655 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: } 2021-07-31T01:38:59.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:38:59 smithi045 conmon[33655]: audit 2021-07-31T01:38:59.633567+0000 mon.a (mon.0) 7 : audit [DBG] from='client.? 172.21.15.45:0/2612021618' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2021-07-31T01:39:00.355 INFO:teuthology.orchestra.run.smithi045.stderr:mgr not available, waiting (1/15)... 2021-07-31T01:39:02.353 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/daemon-base:latest-pacific -e NODE_NAME=smithi045 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/f52d2c14-f19f-11eb-8c24-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmput3kn1b8:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp5r80c4w5:/etc/ceph/ceph.conf:z docker.io/ceph/daemon-base:latest-pacific status -f json-pretty 2021-07-31T01:39:03.076 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2021-07-31T01:39:03.076 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: { 2021-07-31T01:39:03.077 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "fsid": "f52d2c14-f19f-11eb-8c24-001a4aab830c", 2021-07-31T01:39:03.077 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "health": { 2021-07-31T01:39:03.078 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2021-07-31T01:39:03.078 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "checks": {}, 2021-07-31T01:39:03.079 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "mutes": [] 2021-07-31T01:39:03.079 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: }, 2021-07-31T01:39:03.079 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "election_epoch": 5, 2021-07-31T01:39:03.080 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "quorum": [ 2021-07-31T01:39:03.080 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 0 2021-07-31T01:39:03.080 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: ], 2021-07-31T01:39:03.080 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "quorum_names": [ 2021-07-31T01:39:03.081 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "a" 2021-07-31T01:39:03.081 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: ], 2021-07-31T01:39:03.081 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "quorum_age": 6, 2021-07-31T01:39:03.082 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "monmap": { 2021-07-31T01:39:03.084 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "epoch": 1, 2021-07-31T01:39:03.084 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2021-07-31T01:39:03.084 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "num_mons": 1 2021-07-31T01:39:03.084 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: }, 2021-07-31T01:39:03.085 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "osdmap": { 2021-07-31T01:39:03.085 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "epoch": 1, 2021-07-31T01:39:03.085 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "num_osds": 0, 2021-07-31T01:39:03.085 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "num_up_osds": 0, 2021-07-31T01:39:03.086 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "osd_up_since": 0, 2021-07-31T01:39:03.086 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "num_in_osds": 0, 2021-07-31T01:39:03.086 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "osd_in_since": 0, 2021-07-31T01:39:03.087 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2021-07-31T01:39:03.087 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: }, 2021-07-31T01:39:03.088 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "pgmap": { 2021-07-31T01:39:03.088 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "pgs_by_state": [], 2021-07-31T01:39:03.088 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "num_pgs": 0, 2021-07-31T01:39:03.088 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "num_pools": 0, 2021-07-31T01:39:03.088 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "num_objects": 0, 2021-07-31T01:39:03.089 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "data_bytes": 0, 2021-07-31T01:39:03.091 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "bytes_used": 0, 2021-07-31T01:39:03.091 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "bytes_avail": 0, 2021-07-31T01:39:03.091 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "bytes_total": 0 2021-07-31T01:39:03.091 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: }, 2021-07-31T01:39:03.091 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "fsmap": { 2021-07-31T01:39:03.092 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "epoch": 1, 2021-07-31T01:39:03.092 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "by_rank": [], 2021-07-31T01:39:03.092 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "up:standby": 0 2021-07-31T01:39:03.092 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: }, 2021-07-31T01:39:03.092 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "mgrmap": { 2021-07-31T01:39:03.093 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "available": false, 2021-07-31T01:39:03.093 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "num_standbys": 0, 2021-07-31T01:39:03.093 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "modules": [ 2021-07-31T01:39:03.093 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "iostat", 2021-07-31T01:39:03.094 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "nfs", 2021-07-31T01:39:03.094 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "restful" 2021-07-31T01:39:03.094 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: ], 2021-07-31T01:39:03.094 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "services": {} 2021-07-31T01:39:03.094 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: }, 2021-07-31T01:39:03.095 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "servicemap": { 2021-07-31T01:39:03.095 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "epoch": 1, 2021-07-31T01:39:03.096 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "modified": "2021-07-31T01:38:50.681462+0000", 2021-07-31T01:39:03.096 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "services": {} 2021-07-31T01:39:03.096 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: }, 2021-07-31T01:39:03.097 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "progress_events": {} 2021-07-31T01:39:03.097 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: } 2021-07-31T01:39:03.138 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:03 smithi045 conmon[33655]: audit 2021-07-31T01:39:03.061823+0000 mon.a ( 2021-07-31T01:39:03.139 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:03 smithi045 conmon[33655]: mon.0) 8 : audit [DBG] from='client.? 172.21.15.45:0/579123371' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2021-07-31T01:39:03.691 INFO:teuthology.orchestra.run.smithi045.stderr:mgr not available, waiting (2/15)... 2021-07-31T01:39:04.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:04 smithi045 conmon[33655]: cluster 2021-07-31T01:39:04.157171+0000 mon.a ( 2021-07-31T01:39:04.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:04 smithi045 conmon[33655]: mon.0) 9 : cluster [INF] Activating manager daemon y 2021-07-31T01:39:04.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:04 smithi045 conmon[33655]: cluster 2021-07-31T01:39:04.159743+0000 mon.a (mon.0) 10 : 2021-07-31T01:39:04.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:04 smithi045 conmon[33655]: cluster [DBG] mgrmap e2: y(active, starting, since 0.00266901s) 2021-07-31T01:39:04.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:04 smithi045 conmon[33655]: audit 2021-07 2021-07-31T01:39:04.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:04 smithi045 conmon[33655]: -31T01:39:04. 2021-07-31T01:39:04.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:04 smithi045 conmon[33655]: 160702+0000 mon.a (mon.0) 11 2021-07-31T01:39:04.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:04 smithi045 conmon[33655]: : audit [DBG] from='mgr.14100 172.21.15.45:0/2137962814' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2021-07-31T01:39:04.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:04 smithi045 conmon[33655]: audit 2021 2021-07-31T01:39:04.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:04 smithi045 conmon[33655]: -07-31T01:39 2021-07-31T01:39:04.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:04 smithi045 conmon[33655]: :04.160940+0000 mon.a (mon. 2021-07-31T01:39:04.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:04 smithi045 conmon[33655]: 0) 12 : audit [DBG] from='mgr.14100 172.21.15.45:0/2137962814' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2021-07-31T01:39:04.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:04 smithi045 conmon[33655]: audit 2021 2021-07-31T01:39:04.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:04 smithi045 conmon[33655]: -07-31T01:39:04.161136+0000 2021-07-31T01:39:04.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:04 smithi045 conmon[33655]: mon.a (mon.0) 13 : audit 2021-07-31T01:39:04.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:04 smithi045 conmon[33655]: [DBG] from='mgr.14100 172.21.15.45:0/2137962814' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2021-07-31T01:39:04.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:04 smithi045 conmon[33655]: audit 2021- 2021-07-31T01:39:04.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:04 smithi045 conmon[33655]: 07-31T01:39: 2021-07-31T01:39:04.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:04 smithi045 conmon[33655]: 04.161339+0000 mon.a (mon 2021-07-31T01:39:04.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:04 smithi045 conmon[33655]: .0) 14 : audit [DBG] from='mgr.14100 172.21.15.45:0/2137962814' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2021-07-31T01:39:04.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:04 smithi045 conmon[33655]: audit 2021- 2021-07-31T01:39:04.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:04 smithi045 conmon[33655]: 07-31T01:39:04 2021-07-31T01:39:04.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:04 smithi045 conmon[33655]: .161533+0000 mon.a (mon. 2021-07-31T01:39:04.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:04 smithi045 conmon[33655]: 0) 15 : audit [DBG] from='mgr.14100 172.21.15.45:0/2137962814' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2021-07-31T01:39:04.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:04 smithi045 conmon[33655]: cluster 2021- 2021-07-31T01:39:04.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:04 smithi045 conmon[33655]: 07-31T01:39: 2021-07-31T01:39:04.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:04 smithi045 conmon[33655]: 04.163224+0000 mon.a (mon.0) 2021-07-31T01:39:04.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:04 smithi045 conmon[33655]: 16 : cluster [INF] Manager daemon y is now available 2021-07-31T01:39:04.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:04 smithi045 conmon[33655]: audit 2021-07-31T01:39:04.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:04 smithi045 conmon[33655]: 2021-07-31T01: 2021-07-31T01:39:04.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:04 smithi045 conmon[33655]: 39:04.172824+0000 mon.a (mon 2021-07-31T01:39:04.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:04 smithi045 conmon[33655]: .0) 17 : audit [INF] from='mgr.14100 172.21.15.45:0/2137962814' entity='mgr.y' 2021-07-31T01:39:04.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:04 smithi045 conmon[33655]: audit 2021 2021-07-31T01:39:04.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:04 smithi045 conmon[33655]: -07-31T01:39: 2021-07-31T01:39:04.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:04 smithi045 conmon[33655]: 04.176072+0000 mon.a (mon.0) 18 : audit 2021-07-31T01:39:04.535 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:04 smithi045 conmon[33655]: [INF] from='mgr.14100 172.21.15.45:0/2137962814' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:39:04.535 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:04 smithi045 conmon[33655]: audit 2021 2021-07-31T01:39:04.535 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:04 smithi045 conmon[33655]: -07-31T01:39 2021-07-31T01:39:04.535 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:04 smithi045 conmon[33655]: :04.177278+0000 mon.a (mon. 2021-07-31T01:39:04.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:04 smithi045 conmon[33655]: 0) 19 : audit [INF] from='mgr.14100 172.21.15.45:0/2137962814' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-07-31T01:39:04.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:04 smithi045 conmon[33655]: audit 2021-07-31T01:39:04.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:04 smithi045 conmon[33655]: 2021-07-31T01 2021-07-31T01:39:04.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:04 smithi045 conmon[33655]: :39:04.182111+0000 mon.a ( 2021-07-31T01:39:04.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:04 smithi045 conmon[33655]: mon.0) 20 : audit [INF] from='mgr.14100 172.21.15.45:0/2137962814' entity='mgr.y' 2021-07-31T01:39:04.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:04 smithi045 conmon[33655]: 2021-07-31T01:39:04.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:04 smithi045 conmon[33655]: audit 2021-07- 2021-07-31T01:39:04.539 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:04 smithi045 conmon[33655]: 31T01:39:04.183747 2021-07-31T01:39:04.539 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:04 smithi045 conmon[33655]: +0000 mon.a (mon.0) 21 : audit 2021-07-31T01:39:04.539 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:04 smithi045 conmon[33655]: [INF] from='mgr.14100 172.21.15.45:0/2137962814' entity='mgr.y' 2021-07-31T01:39:05.693 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/daemon-base:latest-pacific -e NODE_NAME=smithi045 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/f52d2c14-f19f-11eb-8c24-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmput3kn1b8:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp5r80c4w5:/etc/ceph/ceph.conf:z docker.io/ceph/daemon-base:latest-pacific status -f json-pretty 2021-07-31T01:39:06.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:06 smithi045 conmon[33655]: cluster 2021-07-31T01:39:05.162866+0000 2021-07-31T01:39:06.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:06 smithi045 conmon[33655]: mon.a (mon.0) 22 : cluster [DBG] mgrmap e3: y(active, since 1.00578s) 2021-07-31T01:39:06.617 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2021-07-31T01:39:06.617 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: { 2021-07-31T01:39:06.618 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "fsid": "f52d2c14-f19f-11eb-8c24-001a4aab830c", 2021-07-31T01:39:06.618 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "health": { 2021-07-31T01:39:06.618 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2021-07-31T01:39:06.618 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "checks": {}, 2021-07-31T01:39:06.619 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "mutes": [] 2021-07-31T01:39:06.619 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: }, 2021-07-31T01:39:06.619 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "election_epoch": 5, 2021-07-31T01:39:06.620 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "quorum": [ 2021-07-31T01:39:06.620 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 0 2021-07-31T01:39:06.620 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: ], 2021-07-31T01:39:06.620 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "quorum_names": [ 2021-07-31T01:39:06.620 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "a" 2021-07-31T01:39:06.621 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: ], 2021-07-31T01:39:06.621 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "quorum_age": 10, 2021-07-31T01:39:06.621 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "monmap": { 2021-07-31T01:39:06.621 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "epoch": 1, 2021-07-31T01:39:06.621 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2021-07-31T01:39:06.622 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "num_mons": 1 2021-07-31T01:39:06.622 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: }, 2021-07-31T01:39:06.622 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "osdmap": { 2021-07-31T01:39:06.622 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "epoch": 1, 2021-07-31T01:39:06.622 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "num_osds": 0, 2021-07-31T01:39:06.623 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "num_up_osds": 0, 2021-07-31T01:39:06.627 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "osd_up_since": 0, 2021-07-31T01:39:06.627 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "num_in_osds": 0, 2021-07-31T01:39:06.627 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "osd_in_since": 0, 2021-07-31T01:39:06.627 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2021-07-31T01:39:06.628 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: }, 2021-07-31T01:39:06.628 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "pgmap": { 2021-07-31T01:39:06.628 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "pgs_by_state": [], 2021-07-31T01:39:06.628 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "num_pgs": 0, 2021-07-31T01:39:06.628 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "num_pools": 0, 2021-07-31T01:39:06.629 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "num_objects": 0, 2021-07-31T01:39:06.629 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "data_bytes": 0, 2021-07-31T01:39:06.629 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "bytes_used": 0, 2021-07-31T01:39:06.629 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "bytes_avail": 0, 2021-07-31T01:39:06.630 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "bytes_total": 0 2021-07-31T01:39:06.630 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: }, 2021-07-31T01:39:06.630 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "fsmap": { 2021-07-31T01:39:06.630 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "epoch": 1, 2021-07-31T01:39:06.630 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "by_rank": [], 2021-07-31T01:39:06.631 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "up:standby": 0 2021-07-31T01:39:06.631 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: }, 2021-07-31T01:39:06.631 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "mgrmap": { 2021-07-31T01:39:06.631 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "available": true, 2021-07-31T01:39:06.631 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "num_standbys": 0, 2021-07-31T01:39:06.632 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "modules": [ 2021-07-31T01:39:06.632 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "iostat", 2021-07-31T01:39:06.632 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "nfs", 2021-07-31T01:39:06.632 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "restful" 2021-07-31T01:39:06.633 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: ], 2021-07-31T01:39:06.633 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "services": {} 2021-07-31T01:39:06.633 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: }, 2021-07-31T01:39:06.633 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "servicemap": { 2021-07-31T01:39:06.633 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "epoch": 1, 2021-07-31T01:39:06.634 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "modified": "2021-07-31T01:38:50.681462+0000", 2021-07-31T01:39:06.634 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "services": {} 2021-07-31T01:39:06.634 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: }, 2021-07-31T01:39:06.634 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "progress_events": {} 2021-07-31T01:39:06.635 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: } 2021-07-31T01:39:07.436 INFO:teuthology.orchestra.run.smithi045.stderr:mgr is available 2021-07-31T01:39:07.437 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/daemon-base:latest-pacific -e NODE_NAME=smithi045 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/f52d2c14-f19f-11eb-8c24-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmput3kn1b8:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp5r80c4w5:/etc/ceph/ceph.conf:z -v /tmp/tmpioaaipjn:/var/lib/ceph/user.conf:z docker.io/ceph/daemon-base:latest-pacific config assimilate-conf -i /var/lib/ceph/user.conf 2021-07-31T01:39:07.643 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:07 smithi045 conmon[33655]: audit 2021-07-31T01:39:06.614316+0000 mon.a (mon. 2021-07-31T01:39:07.643 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:07 smithi045 conmon[33655]: 0) 23 : audit [DBG] from='client.? 172.21.15.45:0/3303049060' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2021-07-31T01:39:07.643 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:07 smithi045 conmon[33655]: cluster 2021-07-31T01:39:07 2021-07-31T01:39:07.644 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:07 smithi045 conmon[33655]: .162592+0000 mon.a (mon.0) 24 : cluster 2021-07-31T01:39:07.644 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:07 smithi045 conmon[33655]: [DBG] mgrmap e4: y(active, since 3s) 2021-07-31T01:39:08.256 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2021-07-31T01:39:08.256 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: [global] 2021-07-31T01:39:08.256 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: fsid = f52d2c14-f19f-11eb-8c24-001a4aab830c 2021-07-31T01:39:08.256 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: log_to_journald = false 2021-07-31T01:39:08.257 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2021-07-31T01:39:08.257 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2021-07-31T01:39:08.257 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2021-07-31T01:39:08.258 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2021-07-31T01:39:08.258 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2021-07-31T01:39:08.258 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: [mgr] 2021-07-31T01:39:08.258 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2021-07-31T01:39:08.259 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2021-07-31T01:39:08.259 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: [osd] 2021-07-31T01:39:08.259 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2021-07-31T01:39:08.259 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2021-07-31T01:39:08.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:08 smithi045 conmon[33655]: audit 2021-07-31T01:39:08.253523+0000 mon.a (mon.0) 25 : audit [INF] from='client.? 172.21.15.45:0/301568982' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2021-07-31T01:39:08.875 INFO:teuthology.orchestra.run.smithi045.stderr:Enabling cephadm module... 2021-07-31T01:39:08.875 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/daemon-base:latest-pacific -e NODE_NAME=smithi045 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/f52d2c14-f19f-11eb-8c24-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmput3kn1b8:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp5r80c4w5:/etc/ceph/ceph.conf:z docker.io/ceph/daemon-base:latest-pacific mgr module enable cephadm 2021-07-31T01:39:10.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:09 smithi045 conmon[33655]: audit 2021-07-31T01:39:09.832246+0000 mon.a (mon.0) 26 : audit [INF] from='client.? 172.21.15.45:0/3593180727' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2021-07-31T01:39:10.276 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Jul 31 01:39:09 smithi045 conmon[33979]: ignoring --setuser ceph since I am not root 2021-07-31T01:39:10.276 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Jul 31 01:39:09 smithi045 conmon[33979]: ignoring --setgroup ceph since I am not root 2021-07-31T01:39:10.612 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/daemon-base:latest-pacific -e NODE_NAME=smithi045 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/f52d2c14-f19f-11eb-8c24-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmput3kn1b8:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp5r80c4w5:/etc/ceph/ceph.conf:z docker.io/ceph/daemon-base:latest-pacific mgr stat 2021-07-31T01:39:11.133 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:10 smithi045 conmon[33655]: audit 2021-07-31T01:39:09.885253+0000 2021-07-31T01:39:11.133 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:10 smithi045 conmon[33655]: mon.a (mon.0) 27 : audit [INF] from='client.? 172.21.15.45:0/3593180727' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2021-07-31T01:39:11.134 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:10 smithi045 conmon[33655]: cluster 2021-07-31T01:39:09.885311+0000 mon.a (mon.0) 28 : cluster 2021-07-31T01:39:11.134 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:10 smithi045 conmon[33655]: [DBG] mgrmap e5: y(active, since 5s) 2021-07-31T01:39:11.515 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: { 2021-07-31T01:39:11.515 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "epoch": 5, 2021-07-31T01:39:11.515 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "available": true, 2021-07-31T01:39:11.516 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "active_name": "y", 2021-07-31T01:39:11.516 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "num_standby": 0 2021-07-31T01:39:11.516 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: } 2021-07-31T01:39:12.250 INFO:teuthology.orchestra.run.smithi045.stderr:Waiting for the mgr to restart... 2021-07-31T01:39:12.250 INFO:teuthology.orchestra.run.smithi045.stderr:Waiting for mgr epoch 5... 2021-07-31T01:39:12.250 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/daemon-base:latest-pacific -e NODE_NAME=smithi045 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/f52d2c14-f19f-11eb-8c24-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmput3kn1b8:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp5r80c4w5:/etc/ceph/ceph.conf:z docker.io/ceph/daemon-base:latest-pacific tell mgr mgr_status 2021-07-31T01:39:12.461 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:12 smithi045 conmon[33655]: audit 2021-07-31T01:39:11.514198+0000 mon.a ( 2021-07-31T01:39:12.462 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:12 smithi045 conmon[33655]: mon.0) 29 : audit [DBG] from='client.? 172.21.15.45:0/2526777716' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2021-07-31T01:39:16.264 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: { 2021-07-31T01:39:16.264 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2021-07-31T01:39:16.264 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "initialized": true 2021-07-31T01:39:16.265 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: } 2021-07-31T01:39:16.485 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:16 smithi045 conmon[33655]: cluster 2021-07-31T01:39:15.200127+0000 mon.a (mon.0) 30 : cluster 2021-07-31T01:39:16.486 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:16 smithi045 conmon[33655]: [INF] Active manager daemon y restarted 2021-07-31T01:39:16.486 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:16 smithi045 conmon[33655]: cluster 2021-07-31T01:39:15.200523+0000 mon.a (mon.0) 31 : cluster [INF] Activating manager daemon y 2021-07-31T01:39:16.486 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:16 smithi045 conmon[33655]: cluster 2021-07-31T01:39:15.202131+0000 mon.a (mon.0) 32 : cluster [DBG] osdmap e2: 0 total, 0 up, 0 in 2021-07-31T01:39:16.486 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:16 smithi045 conmon[33655]: cluster 2021-07-31T01:39:15.252622+0000 mon.a (mon.0) 33 : cluster [DBG] mgrmap e6: y(active, starting, since 0.0521708s) 2021-07-31T01:39:16.487 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:16 smithi045 conmon[33655]: audit 2021-07-31T01:39:15.253442+0000 mon.a (mon.0) 34 : audit [DBG] from='mgr.14116 172.21.15.45:0/2301005870' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2021-07-31T01:39:16.487 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:16 smithi045 conmon[33655]: audit 2021-07-31T01:39:16.487 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:16 smithi045 conmon[33655]: 2021-07-31T01:39:15.254051+0000 mon.a (mon.0) 35 : audit [DBG] from='mgr.14116 172.21.15.45:0/2301005870' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2021-07-31T01:39:16.487 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:16 smithi045 conmon[33655]: audit 2021-07-31T01 2021-07-31T01:39:16.488 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:16 smithi045 conmon[33655]: :39:15.254766+0000 mon.a (mon.0) 36 2021-07-31T01:39:16.488 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:16 smithi045 conmon[33655]: : audit [DBG] from='mgr.14116 172.21.15.45:0/2301005870' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2021-07-31T01:39:16.488 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:16 smithi045 conmon[33655]: audit 2021-07-31T01:39:16.488 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:16 smithi045 conmon[33655]: 2021-07-31T01: 2021-07-31T01:39:16.488 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:16 smithi045 conmon[33655]: 39:15.254984+0000 mon.a (mon.0) 37 2021-07-31T01:39:16.489 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:16 smithi045 conmon[33655]: : audit [DBG] from='mgr.14116 172.21.15.45:0/2301005870' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2021-07-31T01:39:16.489 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:16 smithi045 conmon[33655]: audit 2021-07-31T01:39:16.489 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:16 smithi045 conmon[33655]: 2021-07-31T01 2021-07-31T01:39:16.489 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:16 smithi045 conmon[33655]: :39:15.255210+0000 mon.a (mon.0) 2021-07-31T01:39:16.490 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:16 smithi045 conmon[33655]: 38 : audit [DBG] from='mgr.14116 172.21.15.45:0/2301005870' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2021-07-31T01:39:16.490 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:16 smithi045 conmon[33655]: cluster 2021-07-31T01:39:16.490 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:16 smithi045 conmon[33655]: 2021-07-31T01:39:15.256970+0000 mon.a (mon.0) 39 : cluster 2021-07-31T01:39:16.490 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:16 smithi045 conmon[33655]: [INF] Manager daemon y is now available 2021-07-31T01:39:16.491 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:16 smithi045 conmon[33655]: audit 2021-07-31T 2021-07-31T01:39:16.491 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:16 smithi045 conmon[33655]: 01:39:15.270667+0000 mon.a (mon.0) 40 : audit 2021-07-31T01:39:16.491 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:16 smithi045 conmon[33655]: [INF] from='mgr.14116 172.21.15.45:0/2301005870' entity='mgr.y' 2021-07-31T01:39:16.491 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:16 smithi045 conmon[33655]: audit 2021-07-31T01:39:16.491 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:16 smithi045 conmon[33655]: 2021-07-31T01:39:15.273233+0000 2021-07-31T01:39:16.492 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:16 smithi045 conmon[33655]: mon.a (mon.0) 41 : audit [INF] 2021-07-31T01:39:16.492 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:16 smithi045 conmon[33655]: from='mgr.14116 172.21.15.45:0/2301005870' entity='mgr.y' 2021-07-31T01:39:16.492 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:16 smithi045 conmon[33655]: audit 2021-07-31T 2021-07-31T01:39:16.492 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:16 smithi045 conmon[33655]: 01:39:15.275589+0000 mon.a (mon.0) 2021-07-31T01:39:16.493 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:16 smithi045 conmon[33655]: 42 : audit [INF] from='mgr.14116 172.21.15.45:0/2301005870' entity='mgr.y' 2021-07-31T01:39:16.493 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:16 smithi045 conmon[33655]: audit 2021-07-31T01:39:16.494 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:16 smithi045 conmon[33655]: 2021-07-31T01:39:15.368833+0000 2021-07-31T01:39:16.494 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:16 smithi045 conmon[33655]: mon.a (mon.0) 43 : audit [INF] from='mgr.14116 172.21.15.45:0/2301005870' entity='mgr.y' 2021-07-31T01:39:16.494 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:16 smithi045 conmon[33655]: 2021-07-31T01:39:16.494 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:16 smithi045 conmon[33655]: audit 2021-07-31T01 2021-07-31T01:39:16.495 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:16 smithi045 conmon[33655]: :39:15.369522+0000 mon.a (mon.0) 44 : audit [DBG] from='mgr.14116 172.21.15.45:0/2301005870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:39:16.495 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:16 smithi045 conmon[33655]: audit 2021-07-31T01:39:16.495 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:16 smithi045 conmon[33655]: 2021-07-31T01:39:15.371100+0000 mon.a 2021-07-31T01:39:16.495 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:16 smithi045 conmon[33655]: (mon.0) 45 : audit [DBG] from='mgr.14116 172.21.15.45:0/2301005870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:39:16.495 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:16 smithi045 conmon[33655]: audit 2021-07-31T01:39:16.496 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:16 smithi045 conmon[33655]: 2021-07-31T01:39:15.372614+0000 2021-07-31T01:39:16.496 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:16 smithi045 conmon[33655]: mon.a (mon.0) 46 : audit [DBG] from='mgr.14116 172.21.15.45:0/2301005870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:39:16.496 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:16 smithi045 conmon[33655]: 2021-07-31T01:39:16.496 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:16 smithi045 conmon[33655]: audit 2021-07-31T01: 2021-07-31T01:39:16.497 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:16 smithi045 conmon[33655]: 39:15.383282+0000 mon.a (mon.0) 2021-07-31T01:39:16.497 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:16 smithi045 conmon[33655]: 47 : audit [INF] from='mgr.14116 172.21.15.45:0/2301005870' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:39:16.497 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:16 smithi045 conmon[33655]: audit 2021-07-31T01:39:16.497 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:16 smithi045 conmon[33655]: 2021-07-31T01:39:15.384422+0000 mon.a (mon.0 2021-07-31T01:39:16.497 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:16 smithi045 conmon[33655]: ) 48 : audit [INF] from='mgr.14116 172.21.15.45:0/2301005870' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-07-31T01:39:16.993 INFO:teuthology.orchestra.run.smithi045.stderr:mgr epoch 5 is available 2021-07-31T01:39:16.993 INFO:teuthology.orchestra.run.smithi045.stderr:Setting orchestrator backend to cephadm... 2021-07-31T01:39:16.993 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/daemon-base:latest-pacific -e NODE_NAME=smithi045 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/f52d2c14-f19f-11eb-8c24-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmput3kn1b8:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp5r80c4w5:/etc/ceph/ceph.conf:z docker.io/ceph/daemon-base:latest-pacific orch set backend cephadm 2021-07-31T01:39:17.500 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:17 smithi045 conmon[33655]: audit 2021-07-31T01:39:16.255208+0000 mgr.y (mgr. 2021-07-31T01:39:17.500 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:17 smithi045 conmon[33655]: 14116) 1 : audit [DBG] from='client.14120 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2021-07-31T01:39:17.500 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:17 smithi045 conmon[33655]: cluster 2021-07-31T01:39:16.256058+0000 mon.a (mon.0) 49 : cluster [DBG] mgrmap e7: y(active, since 1.0556s) 2021-07-31T01:39:17.501 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:17 smithi045 conmon[33655]: audit 2021-07-31T01:39:16.262462+0000 mgr.y (mgr.14116) 2 : audit [DBG] from='client.14120 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2021-07-31T01:39:18.536 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/daemon-base:latest-pacific -e NODE_NAME=smithi045 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/f52d2c14-f19f-11eb-8c24-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmput3kn1b8:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp5r80c4w5:/etc/ceph/ceph.conf:z docker.io/ceph/daemon-base:latest-pacific cephadm set-user root 2021-07-31T01:39:18.672 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:18 smithi045 conmon[33655]: cluster 2021-07-31T01:39:17.389344+0000 mon.a (mon.0) 50 : cluster [DBG] mgrmap e8: y(active, since 2s) 2021-07-31T01:39:18.673 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:18 smithi045 conmon[33655]: audit 2021-07-31T01:39:17.811458+0000 mgr.y (mgr.14116) 3 : audit [DBG] from='client.14126 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:39:18.673 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:18 smithi045 conmon[33655]: audit 2021-07-31T01:39:17.815284+0000 mon.a (mon.0) 51 : audit [INF] from='mgr.14116 172.21.15.45:0/2301005870' entity='mgr.y' 2021-07-31T01:39:18.673 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:18 smithi045 conmon[33655]: audit 2021-07-31T01:39:17.818900+0000 mon.a (mon.0) 52 : audit [DBG] from='mgr.14116 172.21.15.45:0/2301005870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:39:19.350 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: value unchanged 2021-07-31T01:39:20.078 INFO:teuthology.orchestra.run.smithi045.stderr:Generating ssh key... 2021-07-31T01:39:20.078 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/daemon-base:latest-pacific -e NODE_NAME=smithi045 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/f52d2c14-f19f-11eb-8c24-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmput3kn1b8:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp5r80c4w5:/etc/ceph/ceph.conf:z docker.io/ceph/daemon-base:latest-pacific cephadm generate-key 2021-07-31T01:39:20.274 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:19 smithi045 conmon[33655]: audit 2021-07-31T01:39:19.349398+0000 mgr.y (mgr.14116) 4 : audit [DBG] from='client.14128 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:39:21.240 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:20 smithi045 conmon[33655]: cephadm 2021-07-31T01:39:20.880037+0000 2021-07-31T01:39:21.241 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:20 smithi045 conmon[33655]: mgr.y (mgr.14116) 6 : cephadm [INF] Generating ssh key... 2021-07-31T01:39:21.241 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Jul 31 01:39:20 smithi045 conmon[33979]: Generating public/private rsa key pair. 2021-07-31T01:39:21.241 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Jul 31 01:39:20 smithi045 conmon[33979]: Your identification has been saved in /tmp/tmp6e4chb3s/key. 2021-07-31T01:39:21.242 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Jul 31 01:39:20 smithi045 conmon[33979]: Your public key has been saved in /tmp/tmp6e4chb3s/key.pub. 2021-07-31T01:39:21.242 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Jul 31 01:39:20 smithi045 conmon[33979]: The key fingerprint is: 2021-07-31T01:39:21.242 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Jul 31 01:39:20 smithi045 conmon[33979]: SHA256:P9XQQDLj3L8ILVRFvp5wDhgTqjIyx4VGUud7zLobS5w ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c 2021-07-31T01:39:21.242 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Jul 31 01:39:20 smithi045 conmon[33979]: The key's randomart image is: 2021-07-31T01:39:21.242 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Jul 31 01:39:20 smithi045 conmon[33979]: +---[RSA 3072]----+ 2021-07-31T01:39:21.243 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Jul 31 01:39:20 smithi045 conmon[33979]: | ..o . =.+oo | 2021-07-31T01:39:21.243 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Jul 31 01:39:20 smithi045 conmon[33979]: | o + + B + | 2021-07-31T01:39:21.243 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Jul 31 01:39:20 smithi045 conmon[33979]: | o o . * o o | 2021-07-31T01:39:21.243 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Jul 31 01:39:20 smithi045 conmon[33979]: | o . = . = + . | 2021-07-31T01:39:21.244 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Jul 31 01:39:20 smithi045 conmon[33979]: | o = o S + = = | 2021-07-31T01:39:21.244 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Jul 31 01:39:20 smithi045 conmon[33979]: | + + + . + B o | 2021-07-31T01:39:21.244 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Jul 31 01:39:20 smithi045 conmon[33979]: | E o . = | 2021-07-31T01:39:21.244 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Jul 31 01:39:20 smithi045 conmon[33979]: | . + . | 2021-07-31T01:39:21.244 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Jul 31 01:39:20 smithi045 conmon[33979]: | +. | 2021-07-31T01:39:21.245 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Jul 31 01:39:20 smithi045 conmon[33979]: +----[SHA256]-----+ 2021-07-31T01:39:21.718 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/daemon-base:latest-pacific -e NODE_NAME=smithi045 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/f52d2c14-f19f-11eb-8c24-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmput3kn1b8:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp5r80c4w5:/etc/ceph/ceph.conf:z docker.io/ceph/daemon-base:latest-pacific cephadm get-pub-key 2021-07-31T01:39:22.208 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:21 smithi045 conmon[33655]: audit 2021-07-31T01:39:20.993415+0000 mon.a (mon.0) 53 : audit [INF] from='mgr.14116 172.21.15.45:0/2301005870' entity='mgr.y' 2021-07-31T01:39:22.208 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:21 smithi045 conmon[33655]: audit 2021-07-31T01:39:20.995018+0000 mon.a (mon.0) 54 : audit [INF] from='mgr.14116 172.21.15.45:0/2301005870' entity='mgr.y' 2021-07-31T01:39:22.483 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkC8H2tsAfMLGzttfncpA6BfMj0ekTcEUiN/sZfXvmSCLmgSsXg1me2oQgQxuiYEg3gYN2dXESt3pqV2pC+2JsbGcba9SgXwP3ejESLMF6ZP9lo8M0Pj/XDu4NaP5sctlL2/GDWJCYpGrlL7LvizkRoAGZI+H1a3B+OfBk57gVzIuoYhIha2aJ3QzHhqSjIazXSkWETza49PGAMMe/00s+6iUaY3kLby211n0Z07aov1yYKSMDqtCG6UIPdF6yuVXmnUvM51/D8Z3ANojTiMPyiBRAzA5s7iPVW+bRpfkeF7fyMaJtd1S6HrLIba3PSlmbscYCrJn1Y8hW5403XpwA89LozZKG9bVyy8FRvilqYz17wCg3GKGB3oi8J491mmzGdgFZlpeFdtbAfC2oclodRO1vIQC+9hans4W5tuowTNsN8t8JBvHzsZcR+7ig2hWHJd1jq1ayJonAh8kAZLMwTfaNPUi0uNKp/ulQV10+MJEnaAaxcICys9gV3c3kBek= ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c 2021-07-31T01:39:23.256 INFO:teuthology.orchestra.run.smithi045.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2021-07-31T01:39:23.257 INFO:teuthology.orchestra.run.smithi045.stderr:Adding key to root@localhost authorized_keys... 2021-07-31T01:39:23.258 INFO:teuthology.orchestra.run.smithi045.stderr:Adding host smithi045... 2021-07-31T01:39:23.258 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/daemon-base:latest-pacific -e NODE_NAME=smithi045 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/f52d2c14-f19f-11eb-8c24-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmput3kn1b8:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp5r80c4w5:/etc/ceph/ceph.conf:z docker.io/ceph/daemon-base:latest-pacific orch host add smithi045 172.21.15.45 2021-07-31T01:39:23.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:22 smithi045 conmon[33655]: audit 2021-07-31T01:39:22.482634+0000 mgr.y 2021-07-31T01:39:23.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:22 smithi045 conmon[33655]: (mgr.14116) 7 : audit [DBG] from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:39:24.526 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Jul 31 01:39:24 smithi045 conmon[33979]: [79B blob data] 2021-07-31T01:39:24.861 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: Added host 'smithi045' with addr '172.21.15.45' 2021-07-31T01:39:25.796 INFO:teuthology.orchestra.run.smithi045.stderr:Deploying unmanaged mon service... 2021-07-31T01:39:25.797 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/daemon-base:latest-pacific -e NODE_NAME=smithi045 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/f52d2c14-f19f-11eb-8c24-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmput3kn1b8:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp5r80c4w5:/etc/ceph/ceph.conf:z docker.io/ceph/daemon-base:latest-pacific orch apply mon --unmanaged 2021-07-31T01:39:25.982 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:25 smithi045 conmon[33655]: audit 2021-07-31T01:39:24.062990+0000 mgr.y (mgr.14116) 8 : audit [DBG] from='client.14134 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi045", "addr": "172.21.15.45", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:39:25.982 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:25 smithi045 conmon[33655]: audit 2021-07-31T01:39:24.858567+0000 mon.a (mon.0) 55 : audit [INF] from='mgr.14116 172.21.15.45:0/2301005870' entity='mgr.y' 2021-07-31T01:39:25.983 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:25 smithi045 conmon[33655]: cephadm 2021-07-31T01:39:24.858889+0000 mgr.y (mgr.14116) 9 : cephadm [INF] Added host smithi045 2021-07-31T01:39:25.983 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:25 smithi045 conmon[33655]: audit 2021-07-31T01:39:24.859116+0000 mon.a (mon.0) 56 : audit [DBG] from='mgr.14116 172.21.15.45:0/2301005870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:39:26.612 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: Scheduled mon update... 2021-07-31T01:39:27.533 INFO:teuthology.orchestra.run.smithi045.stderr:Deploying unmanaged mgr service... 2021-07-31T01:39:27.533 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/daemon-base:latest-pacific -e NODE_NAME=smithi045 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/f52d2c14-f19f-11eb-8c24-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmput3kn1b8:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp5r80c4w5:/etc/ceph/ceph.conf:z docker.io/ceph/daemon-base:latest-pacific orch apply mgr --unmanaged 2021-07-31T01:39:27.867 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:27 smithi045 conmon[33655]: audit 2021-07-31T01:39:26.608487+0000 mgr.y (mgr.14116) 10 : audit [DBG] from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:39:27.868 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:27 smithi045 conmon[33655]: cephadm 2021-07-31T01:39:26.609219+0000 mgr.y (mgr.14116) 11 : cephadm [INF] Saving service mon spec with placement count:5 2021-07-31T01:39:27.868 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:27 smithi045 conmon[33655]: audit 2021-07-31T01:39:26.611477+0000 mon.a (mon.0) 57 : audit [INF] from='mgr.14116 172.21.15.45:0/2301005870' entity='mgr.y' 2021-07-31T01:39:28.741 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: Scheduled mgr update... 2021-07-31T01:39:29.269 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/daemon-base:latest-pacific -e NODE_NAME=smithi045 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/f52d2c14-f19f-11eb-8c24-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmput3kn1b8:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp5r80c4w5:/etc/ceph/ceph.conf:z docker.io/ceph/daemon-base:latest-pacific config set mgr mgr/cephadm/container_init True --force 2021-07-31T01:39:29.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:29 smithi045 conmon[33655]: audit 2021-07-31T01:39 2021-07-31T01:39:29.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:29 smithi045 conmon[33655]: :28.490661+0000 mon.a (mon.0) 58 : audit [INF] from='mgr.14116 172.21.15.45:0/2301005870' entity='mgr.y' 2021-07-31T01:39:29.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:29 smithi045 conmon[33655]: audit 2021-07-31T01:39:28.738326+0000 mgr.y (mgr.14116) 12 : audit [DBG] from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:39:29.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:29 smithi045 conmon[33655]: cephadm 2021-07-31T01:39:28.738874+0000 mgr.y (mgr.14116) 13 : cephadm [INF] Saving service mgr spec with placement count:2 2021-07-31T01:39:29.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:29 smithi045 conmon[33655]: audit 2021-07-31T01:39:28.740981+0000 mon.a (mon.0) 59 : audit [INF] from='mgr.14116 172.21.15.45:0/2301005870' entity='mgr.y' 2021-07-31T01:39:29.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:29 smithi045 conmon[33655]: audit 2021-07-31T01:39:28.840695+0000 mon.a (mon.0) 60 : audit [INF] from='mgr.14116 172.21.15.45:0/2301005870' entity='mgr.y' 2021-07-31T01:39:30.711 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/daemon-base:latest-pacific -e NODE_NAME=smithi045 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/f52d2c14-f19f-11eb-8c24-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmput3kn1b8:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp5r80c4w5:/etc/ceph/ceph.conf:z docker.io/ceph/daemon-base:latest-pacific config set mgr mgr/dashboard/ssl_server_port 8443 2021-07-31T01:39:31.183 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:31 smithi045 conmon[33655]: audit 2021-07-31T01:39:30. 2021-07-31T01:39:31.183 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:31 smithi045 conmon[33655]: 086026+0000 mon.a (mon.0) 61 : audit [INF] from='client.? 172.21.15.45:0/229749307' entity='client.admin' 2021-07-31T01:39:31.183 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:31 smithi045 conmon[33655]: audit 2021-07-31T01:39:30.237715+0000 mon.a (mon.0) 62 : audit [INF] from='mgr.14116 172.21.15.45:0/2301005870' entity='mgr.y' 2021-07-31T01:39:32.446 INFO:teuthology.orchestra.run.smithi045.stderr:Enabling the dashboard module... 2021-07-31T01:39:32.447 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/daemon-base:latest-pacific -e NODE_NAME=smithi045 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/f52d2c14-f19f-11eb-8c24-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmput3kn1b8:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp5r80c4w5:/etc/ceph/ceph.conf:z docker.io/ceph/daemon-base:latest-pacific mgr module enable dashboard 2021-07-31T01:39:32.923 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:32 smithi045 conmon[33655]: audit 2021-07-31T01:39:31.902451 2021-07-31T01:39:32.923 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:32 smithi045 conmon[33655]: +0000 mon.a (mon.0) 63 : audit [INF] from='client.? 172.21.15.45:0/2714955607' entity='client.admin' 2021-07-31T01:39:34.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:33 smithi045 conmon[33655]: audit 2021-07-31T01:39:33.234801+0000 mon.a (mon.0 2021-07-31T01:39:34.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:33 smithi045 conmon[33655]: ) 64 : audit [INF] from='client.? 172.21.15.45:0/1881955107' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2021-07-31T01:39:34.277 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Jul 31 01:39:33 smithi045 conmon[33979]: ignoring --setuser ceph since I am not root 2021-07-31T01:39:34.277 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Jul 31 01:39:33 smithi045 conmon[33979]: ignoring --setgroup ceph since I am not root 2021-07-31T01:39:34.585 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/daemon-base:latest-pacific -e NODE_NAME=smithi045 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/f52d2c14-f19f-11eb-8c24-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmput3kn1b8:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp5r80c4w5:/etc/ceph/ceph.conf:z docker.io/ceph/daemon-base:latest-pacific mgr stat 2021-07-31T01:39:35.097 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:34 smithi045 conmon[33655]: audit 2021-07-31T01:39:33.906532+0000 mon.a (mon 2021-07-31T01:39:35.098 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:34 smithi045 conmon[33655]: .0) 65 : audit [INF] from='client.? 172.21.15.45:0/1881955107' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2021-07-31T01:39:35.098 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:34 smithi045 conmon[33655]: cluster 2021-07-31T01:39:33.906597+0000 mon.a (mon.0) 66 : cluster [DBG] mgrmap e9: y(active, since 18s) 2021-07-31T01:39:35.460 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: { 2021-07-31T01:39:35.462 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "epoch": 9, 2021-07-31T01:39:35.463 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "available": true, 2021-07-31T01:39:35.463 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "active_name": "y", 2021-07-31T01:39:35.463 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "num_standby": 0 2021-07-31T01:39:35.463 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: } 2021-07-31T01:39:36.128 INFO:teuthology.orchestra.run.smithi045.stderr:Waiting for the mgr to restart... 2021-07-31T01:39:36.129 INFO:teuthology.orchestra.run.smithi045.stderr:Waiting for mgr epoch 9... 2021-07-31T01:39:36.129 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/daemon-base:latest-pacific -e NODE_NAME=smithi045 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/f52d2c14-f19f-11eb-8c24-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmput3kn1b8:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp5r80c4w5:/etc/ceph/ceph.conf:z docker.io/ceph/daemon-base:latest-pacific tell mgr mgr_status 2021-07-31T01:39:36.261 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:35 smithi045 conmon[33655]: audit 2021-07-31T01:39:35.459403+0000 mon.a (mon.0 2021-07-31T01:39:36.263 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:35 smithi045 conmon[33655]: ) 67 : audit [DBG] from='client.? 172.21.15.45:0/157804940' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2021-07-31T01:39:39.776 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Jul 31 01:39:39 smithi045 conmon[33979]: [79B blob data] 2021-07-31T01:39:40.301 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: { 2021-07-31T01:39:40.301 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "mgrmap_epoch": 11, 2021-07-31T01:39:40.302 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "initialized": true 2021-07-31T01:39:40.302 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: } 2021-07-31T01:39:40.506 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:40 smithi045 conmon[33655]: cluster 2021-07-31T01:39:39.238182+0000 mon.a (mon.0) 68 : cluster [INF] Active manager daemon y restarted 2021-07-31T01:39:40.507 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:40 smithi045 conmon[33655]: cluster 2021-07-31T01:39:39.238568+0000 mon.a (mon.0) 69 : cluster [INF] Activating manager daemon y 2021-07-31T01:39:40.507 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:40 smithi045 conmon[33655]: cluster 2021-07-31T01:39:39.240084+0000 mon.a (mon.0) 70 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2021-07-31T01:39:40.507 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:40 smithi045 conmon[33655]: cluster 2021-07-31T01:39:39.291112+0000 mon.a (mon.0) 71 : cluster [DBG] mgrmap e10: y(active, starting, since 0.0526284s) 2021-07-31T01:39:40.507 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:40 smithi045 conmon[33655]: audit 2021-07-31T01:39:39.292184+0000 mon.a (mon.0) 72 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2021-07-31T01:39:40.508 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:40 smithi045 conmon[33655]: audit 2021-07-31T01:39:39.293030+0000 mon.a (mon.0) 73 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2021-07-31T01:39:40.508 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:40 smithi045 conmon[33655]: audit 2021-07-31T01:39:39.293752+0000 mon.a (mon.0) 74 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2021-07-31T01:39:40.508 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:40 smithi045 conmon[33655]: audit 2021-07-31T01:39:39.293905+0000 mon.a (mon.0) 75 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2021-07-31T01:39:40.508 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:40 smithi045 conmon[33655]: audit 2021-07-31T01:39:39.294047+0000 mon.a (mon.0) 76 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2021-07-31T01:39:40.509 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:40 smithi045 conmon[33655]: cluster 2021-07-31T01:39:39.296050+0000 mon.a (mon.0) 77 : cluster [INF] Manager daemon y is now available 2021-07-31T01:39:40.509 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:40 smithi045 conmon[33655]: audit 2021-07-31T01:39:39.398668+0000 mon.a (mon.0) 78 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:39:40.509 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:40 smithi045 conmon[33655]: audit 2021-07-31T01:39:39.400784+0000 mon.a (mon.0) 79 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:39:40.510 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:40 smithi045 conmon[33655]: audit 2021-07-31T01:39:39.421524+0000 mon.a (mon.0) 80 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:39:40.510 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:40 smithi045 conmon[33655]: audit 2021-07-31T01:39:39.423272+0000 mon.a (mon.0) 81 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-07-31T01:39:41.070 INFO:teuthology.orchestra.run.smithi045.stderr:mgr epoch 9 is available 2021-07-31T01:39:41.070 INFO:teuthology.orchestra.run.smithi045.stderr:Generating a dashboard self-signed certificate... 2021-07-31T01:39:41.070 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/daemon-base:latest-pacific -e NODE_NAME=smithi045 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/f52d2c14-f19f-11eb-8c24-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmput3kn1b8:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp5r80c4w5:/etc/ceph/ceph.conf:z docker.io/ceph/daemon-base:latest-pacific dashboard create-self-signed-cert 2021-07-31T01:39:41.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:41 smithi045 conmon[33655]: audit 2021-07-31T01:39:40.292843+0000 mgr.y ( 2021-07-31T01:39:41.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:41 smithi045 conmon[33655]: mgr.14146) 1 : audit [DBG] from='client.14150 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2021-07-31T01:39:41.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:41 smithi045 conmon[33655]: cluster 2021-07-31T01:39:40.293082+0000 mon.a (mon.0) 82 : cluster [DBG] mgrmap e11: y(active, since 1.0546s) 2021-07-31T01:39:41.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:41 smithi045 conmon[33655]: audit 2021-07-31T01:39:40.299213+0000 mgr.y (mgr.14146) 2 : audit [DBG] from='client.14150 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2021-07-31T01:39:41.989 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: Self-signed certificate created 2021-07-31T01:39:42.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:42 smithi045 conmon[33655]: cluster 2021-07-31T01:39:41.295341+0000 mon.a (mon.0) 83 : cluster [DBG] mgrmap e12: y(active, since 2s) 2021-07-31T01:39:42.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:42 smithi045 conmon[33655]: audit 2021-07-31T01:39:41.475089+0000 mon.a (mon.0) 84 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:39:42.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:42 smithi045 conmon[33655]: audit 2021-07-31T01:39:41.910688+0000 mgr.y (mgr.14146) 3 : audit [DBG] from='client.14156 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:39:42.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:42 smithi045 conmon[33655]: audit 2021-07-31T01:39:41.986930+0000 mon.a (mon.0) 85 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:39:42.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:42 smithi045 conmon[33655]: audit 2021-07-31T01:39:41.988757+0000 mon.a (mon.0) 86 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:39:42.756 INFO:teuthology.orchestra.run.smithi045.stderr:Creating initial admin user... 2021-07-31T01:39:42.757 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/daemon-base:latest-pacific -e NODE_NAME=smithi045 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/f52d2c14-f19f-11eb-8c24-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmput3kn1b8:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp5r80c4w5:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpvqseklca:/tmp/dashboard.pw:z docker.io/ceph/daemon-base:latest-pacific dashboard ac-user-create admin -i /tmp/dashboard.pw administrator --force-password --pwd-update-required 2021-07-31T01:39:44.077 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$g65lksasIWQ/8Wk/.Wj43uQV9rLoFb5hWiikj8Uhk8/D5MdKCN/72", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1627695584, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2021-07-31T01:39:44.255 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:43 smithi045 conmon[33655]: audit 2021-07-31T01:39:43.832456+0000 mgr.y (mgr.14146) 4 : audit [DBG] from='client.14158 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:39:44.695 INFO:teuthology.orchestra.run.smithi045.stderr:Fetching dashboard port number... 2021-07-31T01:39:44.696 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/daemon-base:latest-pacific -e NODE_NAME=smithi045 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/f52d2c14-f19f-11eb-8c24-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmput3kn1b8:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp5r80c4w5:/etc/ceph/ceph.conf:z docker.io/ceph/daemon-base:latest-pacific config get mgr mgr/dashboard/ssl_server_port 2021-07-31T01:39:45.204 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:45 smithi045 conmon[33655]: audit 2021-07-31T01:39 2021-07-31T01:39:45.205 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:45 smithi045 conmon[33655]: :44.076790+0000 mon.a (mon.0) 87 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:39:45.477 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 8443 2021-07-31T01:39:46.133 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: systemctl is-enabled firewalld.service 2021-07-31T01:39:46.143 INFO:teuthology.orchestra.run.smithi045.stderr:systemctl: disabled 2021-07-31T01:39:46.143 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: systemctl is-active firewalld.service 2021-07-31T01:39:46.150 INFO:teuthology.orchestra.run.smithi045.stderr:systemctl: inactive 2021-07-31T01:39:46.151 INFO:teuthology.orchestra.run.smithi045.stderr:firewalld.service is not enabled 2021-07-31T01:39:46.152 INFO:teuthology.orchestra.run.smithi045.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2021-07-31T01:39:46.154 INFO:teuthology.orchestra.run.smithi045.stderr:Ceph Dashboard is now available at: 2021-07-31T01:39:46.154 INFO:teuthology.orchestra.run.smithi045.stderr: 2021-07-31T01:39:46.154 INFO:teuthology.orchestra.run.smithi045.stderr: URL: https://smithi045:8443/ 2021-07-31T01:39:46.154 INFO:teuthology.orchestra.run.smithi045.stderr: User: admin 2021-07-31T01:39:46.155 INFO:teuthology.orchestra.run.smithi045.stderr: Password: 0kfhonhyj4 2021-07-31T01:39:46.155 INFO:teuthology.orchestra.run.smithi045.stderr: 2021-07-31T01:39:46.156 INFO:teuthology.orchestra.run.smithi045.stderr:You can access the Ceph CLI with: 2021-07-31T01:39:46.156 INFO:teuthology.orchestra.run.smithi045.stderr: 2021-07-31T01:39:46.156 INFO:teuthology.orchestra.run.smithi045.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid f52d2c14-f19f-11eb-8c24-001a4aab830c -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2021-07-31T01:39:46.156 INFO:teuthology.orchestra.run.smithi045.stderr: 2021-07-31T01:39:46.157 INFO:teuthology.orchestra.run.smithi045.stderr:Please consider enabling telemetry to help improve Ceph: 2021-07-31T01:39:46.157 INFO:teuthology.orchestra.run.smithi045.stderr: 2021-07-31T01:39:46.157 INFO:teuthology.orchestra.run.smithi045.stderr: ceph telemetry on 2021-07-31T01:39:46.157 INFO:teuthology.orchestra.run.smithi045.stderr: 2021-07-31T01:39:46.157 INFO:teuthology.orchestra.run.smithi045.stderr:For more information see: 2021-07-31T01:39:46.158 INFO:teuthology.orchestra.run.smithi045.stderr: 2021-07-31T01:39:46.158 INFO:teuthology.orchestra.run.smithi045.stderr: https://docs.ceph.com/docs/pacific/mgr/telemetry/ 2021-07-31T01:39:46.158 INFO:teuthology.orchestra.run.smithi045.stderr: 2021-07-31T01:39:46.158 INFO:teuthology.orchestra.run.smithi045.stderr:Bootstrap complete. 2021-07-31T01:39:46.159 INFO:teuthology.orchestra.run.smithi045.stderr:Releasing lock 139753647880624 on /run/cephadm/f52d2c14-f19f-11eb-8c24-001a4aab830c.lock 2021-07-31T01:39:46.159 INFO:teuthology.orchestra.run.smithi045.stderr:Lock 139753647880624 released on /run/cephadm/f52d2c14-f19f-11eb-8c24-001a4aab830c.lock 2021-07-31T01:39:46.191 INFO:tasks.cephadm:Fetching config... 2021-07-31T01:39:46.191 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2021-07-31T01:39:46.192 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2021-07-31T01:39:46.209 INFO:tasks.cephadm:Fetching client.admin keyring... 2021-07-31T01:39:46.209 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2021-07-31T01:39:46.210 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2021-07-31T01:39:46.266 INFO:tasks.cephadm:Fetching mon keyring... 2021-07-31T01:39:46.267 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2021-07-31T01:39:46.267 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/var/lib/ceph/f52d2c14-f19f-11eb-8c24-001a4aab830c/mon.a/keyring of=/dev/stdout 2021-07-31T01:39:46.317 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:46 smithi045 conmon[33655]: audit 2021-07-31T01:39:45.477503+0000 2021-07-31T01:39:46.317 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:46 smithi045 conmon[33655]: mon.a (mon.0) 88 : audit [DBG] from='client.? 172.21.15.45:0/2105185210' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2021-07-31T01:39:46.333 INFO:tasks.cephadm:Fetching pub ssh key... 2021-07-31T01:39:46.334 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2021-07-31T01:39:46.334 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2021-07-31T01:39:46.389 INFO:tasks.cephadm:Installing pub ssh key for root users... 2021-07-31T01:39:46.390 DEBUG:teuthology.orchestra.run.smithi045:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkC8H2tsAfMLGzttfncpA6BfMj0ekTcEUiN/sZfXvmSCLmgSsXg1me2oQgQxuiYEg3gYN2dXESt3pqV2pC+2JsbGcba9SgXwP3ejESLMF6ZP9lo8M0Pj/XDu4NaP5sctlL2/GDWJCYpGrlL7LvizkRoAGZI+H1a3B+OfBk57gVzIuoYhIha2aJ3QzHhqSjIazXSkWETza49PGAMMe/00s+6iUaY3kLby211n0Z07aov1yYKSMDqtCG6UIPdF6yuVXmnUvM51/D8Z3ANojTiMPyiBRAzA5s7iPVW+bRpfkeF7fyMaJtd1S6HrLIba3PSlmbscYCrJn1Y8hW5403XpwA89LozZKG9bVyy8FRvilqYz17wCg3GKGB3oi8J491mmzGdgFZlpeFdtbAfC2oclodRO1vIQC+9hans4W5tuowTNsN8t8JBvHzsZcR+7ig2hWHJd1jq1ayJonAh8kAZLMwTfaNPUi0uNKp/ulQV10+MJEnaAaxcICys9gV3c3kBek= ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2021-07-31T01:39:46.468 INFO:teuthology.orchestra.run.smithi045.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkC8H2tsAfMLGzttfncpA6BfMj0ekTcEUiN/sZfXvmSCLmgSsXg1me2oQgQxuiYEg3gYN2dXESt3pqV2pC+2JsbGcba9SgXwP3ejESLMF6ZP9lo8M0Pj/XDu4NaP5sctlL2/GDWJCYpGrlL7LvizkRoAGZI+H1a3B+OfBk57gVzIuoYhIha2aJ3QzHhqSjIazXSkWETza49PGAMMe/00s+6iUaY3kLby211n0Z07aov1yYKSMDqtCG6UIPdF6yuVXmnUvM51/D8Z3ANojTiMPyiBRAzA5s7iPVW+bRpfkeF7fyMaJtd1S6HrLIba3PSlmbscYCrJn1Y8hW5403XpwA89LozZKG9bVyy8FRvilqYz17wCg3GKGB3oi8J491mmzGdgFZlpeFdtbAfC2oclodRO1vIQC+9hans4W5tuowTNsN8t8JBvHzsZcR+7ig2hWHJd1jq1ayJonAh8kAZLMwTfaNPUi0uNKp/ulQV10+MJEnaAaxcICys9gV3c3kBek= ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c 2021-07-31T01:39:46.482 DEBUG:teuthology.orchestra.run.smithi154:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkC8H2tsAfMLGzttfncpA6BfMj0ekTcEUiN/sZfXvmSCLmgSsXg1me2oQgQxuiYEg3gYN2dXESt3pqV2pC+2JsbGcba9SgXwP3ejESLMF6ZP9lo8M0Pj/XDu4NaP5sctlL2/GDWJCYpGrlL7LvizkRoAGZI+H1a3B+OfBk57gVzIuoYhIha2aJ3QzHhqSjIazXSkWETza49PGAMMe/00s+6iUaY3kLby211n0Z07aov1yYKSMDqtCG6UIPdF6yuVXmnUvM51/D8Z3ANojTiMPyiBRAzA5s7iPVW+bRpfkeF7fyMaJtd1S6HrLIba3PSlmbscYCrJn1Y8hW5403XpwA89LozZKG9bVyy8FRvilqYz17wCg3GKGB3oi8J491mmzGdgFZlpeFdtbAfC2oclodRO1vIQC+9hans4W5tuowTNsN8t8JBvHzsZcR+7ig2hWHJd1jq1ayJonAh8kAZLMwTfaNPUi0uNKp/ulQV10+MJEnaAaxcICys9gV3c3kBek= ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2021-07-31T01:39:46.523 INFO:teuthology.orchestra.run.smithi154.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkC8H2tsAfMLGzttfncpA6BfMj0ekTcEUiN/sZfXvmSCLmgSsXg1me2oQgQxuiYEg3gYN2dXESt3pqV2pC+2JsbGcba9SgXwP3ejESLMF6ZP9lo8M0Pj/XDu4NaP5sctlL2/GDWJCYpGrlL7LvizkRoAGZI+H1a3B+OfBk57gVzIuoYhIha2aJ3QzHhqSjIazXSkWETza49PGAMMe/00s+6iUaY3kLby211n0Z07aov1yYKSMDqtCG6UIPdF6yuVXmnUvM51/D8Z3ANojTiMPyiBRAzA5s7iPVW+bRpfkeF7fyMaJtd1S6HrLIba3PSlmbscYCrJn1Y8hW5403XpwA89LozZKG9bVyy8FRvilqYz17wCg3GKGB3oi8J491mmzGdgFZlpeFdtbAfC2oclodRO1vIQC+9hans4W5tuowTNsN8t8JBvHzsZcR+7ig2hWHJd1jq1ayJonAh8kAZLMwTfaNPUi0uNKp/ulQV10+MJEnaAaxcICys9gV3c3kBek= ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c 2021-07-31T01:39:46.536 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f52d2c14-f19f-11eb-8c24-001a4aab830c -- ceph config set mgr mgr/cephadm/allow_ptrace true 2021-07-31T01:39:47.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:47 smithi045 conmon[33655]: cluster 2021-07-31T01:39:46.079451+0000 mon.a (mon.0) 89 : cluster [DBG] mgrmap e13: y(active, since 6s) 2021-07-31T01:39:49.669 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2021-07-31T01:39:49.669 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f52d2c14-f19f-11eb-8c24-001a4aab830c -- ceph orch client-keyring set client.admin '*' --mode 0755 2021-07-31T01:39:50.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:49 smithi045 conmon[33655]: audit 2021-07-31T01:39:48.985700+0000 mon.a (mon.0) 90 : audit [INF] from='client.? 172.21.15.45:0/394709013' entity='client.admin' 2021-07-31T01:39:52.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:52 smithi045 conmon[33655]: audit 2021-07-31T01:39:51.330979+0000 mon.a (mon.0) 91 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:39:52.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:52 smithi045 conmon[33655]: audit 2021-07-31T01:39:51.749356+0000 mon.a (mon.0) 92 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi045", "name": "osd_memory_target"}]: dispatch 2021-07-31T01:39:52.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:52 smithi045 conmon[33655]: audit 2021-07-31T01:39:51.752545+0000 mon.a (mon.0) 93 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:39:52.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:52 smithi045 conmon[33655]: audit 2021-07-31T01:39:51.753591+0000 mon.a (mon.0) 94 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:39:52.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:52 smithi045 conmon[33655]: audit 2021-07-31T01:39:51.756553+0000 mon.a (mon.0) 95 : 2021-07-31T01:39:52.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:52 smithi045 conmon[33655]: audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:39:52.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:52 smithi045 conmon[33655]: audit 2021-07-31T01:39:52.114404+0000 mon.a (mon.0) 96 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:39:52.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:52 smithi045 conmon[33655]: audit 2021-07-31T01:39:52.114917+0000 mon.a (mon.0) 97 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:39:52.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:52 smithi045 conmon[33655]: audit 2021-07-31T01:39:52.115857+0000 mon.a (mon.0) 98 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:39:52.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:52 smithi045 conmon[33655]: audit 2021-07-31T01:39:52.116708+0000 mon.a (mon.0) 99 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:39:52.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:52 smithi045 conmon[33655]: audit 2021-07-31T01:39:52.198996+0000 mon.a (mon.0) 100 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:39:52.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:52 smithi045 conmon[33655]: audit 2021-07-31T01:39:52.201003+0000 mon.a (mon.0) 101 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:39:52.754 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi154 2021-07-31T01:39:52.756 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2021-07-31T01:39:52.756 DEBUG:teuthology.orchestra.run.smithi154:> dd of=/etc/ceph/ceph.conf 2021-07-31T01:39:52.774 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2021-07-31T01:39:52.774 DEBUG:teuthology.orchestra.run.smithi154:> dd of=/etc/ceph/ceph.client.admin.keyring 2021-07-31T01:39:52.832 INFO:tasks.cephadm:Adding host smithi154 to orchestrator... 2021-07-31T01:39:52.832 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f52d2c14-f19f-11eb-8c24-001a4aab830c -- ceph orch host add smithi154 2021-07-31T01:39:53.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:53 smithi045 conmon[33655]: cephadm 2021-07-31T01:39:52.117957+0000 mgr.y (mgr.14146 2021-07-31T01:39:53.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:53 smithi045 conmon[33655]: ) 6 : cephadm [INF] Updating smithi045:/etc/ceph/ceph.conf 2021-07-31T01:39:53.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:53 smithi045 conmon[33655]: cephadm 2021-07-31T01:39:52.146531+0000 mgr.y (mgr.14146) 7 : 2021-07-31T01:39:53.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:39:53 smithi045 conmon[33655]: cephadm [INF] Updating smithi045:/etc/ceph/ceph.client.admin.keyring 2021-07-31T01:40:00.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:00 smithi045 conmon[33655]: cluster 2021-07-31T01:39:59.294929+0000 mgr.y (mgr. 2021-07-31T01:40:00.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:00 smithi045 conmon[33655]: 14146) 8 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-07-31T01:40:02.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:02 smithi045 conmon[33655]: cluster 2021-07-31T01:40:01.295215+0000 2021-07-31T01:40:02.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:02 smithi045 conmon[33655]: mgr.y (mgr.14146) 9 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-07-31T01:40:04.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:04 smithi045 conmon[33655]: cluster 2021-07-31T01:40:03.295516 2021-07-31T01:40:04.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:04 smithi045 conmon[33655]: +0000 mgr.y (mgr.14146) 10 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-07-31T01:40:06.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:06 smithi045 conmon[33655]: cluster 2021-07-31T01:40:05.295897+0000 2021-07-31T01:40:06.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:06 smithi045 conmon[33655]: mgr.y (mgr.14146) 11 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-07-31T01:40:08.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:08 smithi045 conmon[33655]: cluster 2021-07-31T01:40:07.296257+0000 mgr.y (mgr.14146) 12 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-07-31T01:40:10.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:10 smithi045 conmon[33655]: cluster 2021-07-31T01:40:09.296603+0000 2021-07-31T01:40:10.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:10 smithi045 conmon[33655]: mgr.y (mgr.14146) 13 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-07-31T01:40:12.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:12 smithi045 conmon[33655]: cluster 2021-07-31T01:40:11.296978 2021-07-31T01:40:12.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:12 smithi045 conmon[33655]: +0000 mgr.y (mgr.14146) 14 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-07-31T01:40:14.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:14 smithi045 conmon[33655]: cluster 2021-07-31T01:40:13.297378+0000 mgr.y (mgr. 2021-07-31T01:40:14.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:14 smithi045 conmon[33655]: 14146) 15 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-07-31T01:40:16.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:16 smithi045 conmon[33655]: cluster 2021-07-31T01:40:15.297856+0000 2021-07-31T01:40:16.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:16 smithi045 conmon[33655]: mgr.y (mgr.14146) 16 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-07-31T01:40:18.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:18 smithi045 conmon[33655]: cluster 2021-07-31T01:40:17.298204 2021-07-31T01:40:18.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:18 smithi045 conmon[33655]: +0000 mgr.y (mgr.14146) 17 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-07-31T01:40:20.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:20 smithi045 conmon[33655]: cluster 2021-07-31T01:40:19.298406+0000 2021-07-31T01:40:20.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:20 smithi045 conmon[33655]: mgr.y (mgr.14146) 18 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-07-31T01:40:22.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:22 smithi045 conmon[33655]: cluster 2021-07-31T01:40:21.298773+0000 mgr.y 2021-07-31T01:40:22.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:22 smithi045 conmon[33655]: (mgr.14146) 19 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-07-31T01:40:24.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:24 smithi045 conmon[33655]: cluster 2021-07-31T01:40:23.299189+0000 2021-07-31T01:40:24.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:24 smithi045 conmon[33655]: mgr.y (mgr.14146) 20 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-07-31T01:40:25.776 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Jul 31 01:40:25 smithi045 conmon[33979]: [90B blob data] 2021-07-31T01:40:26.006 INFO:teuthology.orchestra.run.smithi154.stdout:Added host 'smithi154' with addr '172.21.15.154' 2021-07-31T01:40:26.410 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Jul 31 01:40:26 smithi045 conmon[33979]: [80B blob data] 2021-07-31T01:40:26.724 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f52d2c14-f19f-11eb-8c24-001a4aab830c -- ceph orch host ls --format=json 2021-07-31T01:40:26.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:26 smithi045 conmon[33655]: audit 2021-07-31T01:40:25.262154+0000 mgr.y (mgr.14146 2021-07-31T01:40:26.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:26 smithi045 conmon[33655]: ) 21 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi154", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:40:26.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:26 smithi045 conmon[33655]: cluster 2021-07-31T01:40:25.299524+0000 mgr.y (mgr.14146) 22 : 2021-07-31T01:40:26.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:26 smithi045 conmon[33655]: cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-07-31T01:40:26.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:26 smithi045 conmon[33655]: audit 2021-07-31T01:40:26.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:26 smithi045 conmon[33655]: 2021-07-31T01:40 2021-07-31T01:40:26.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:26 smithi045 conmon[33655]: :26.004051+0000 mon.a (mon 2021-07-31T01:40:26.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:26 smithi045 conmon[33655]: .0) 102 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:40:26.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:26 smithi045 conmon[33655]: 2021-07-31T01:40:26.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:26 smithi045 conmon[33655]: audit 2021-07- 2021-07-31T01:40:26.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:26 smithi045 conmon[33655]: 31T01:40:26.004844+0000 mon.a (mon 2021-07-31T01:40:26.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:26 smithi045 conmon[33655]: .0) 103 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:40:26.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:26 smithi045 conmon[33655]: audit 2021-07-31T01:40:26.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:26 smithi045 conmon[33655]: 2021-07-31T01:40 2021-07-31T01:40:26.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:26 smithi045 conmon[33655]: :26.006251+0000 mon.a (mon 2021-07-31T01:40:26.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:26 smithi045 conmon[33655]: .0) 104 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:40:26.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:26 smithi045 conmon[33655]: 2021-07-31T01:40:26.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:26 smithi045 conmon[33655]: audit 2021-07-31T01:40:26 2021-07-31T01:40:26.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:26 smithi045 conmon[33655]: .007300+0000 mon.a (mon.0) 105 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:40:28.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:27 smithi045 conmon[33655]: cephadm 2021-07-31T01:40:26.004475+0000 mgr.y (mgr.14146 2021-07-31T01:40:28.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:27 smithi045 conmon[33655]: ) 23 : cephadm [INF] Added host smithi154 2021-07-31T01:40:28.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:27 smithi045 conmon[33655]: audit 2021-07-31T01:40:26.576326+0000 mon.a (mon.0) 106 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:40:28.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:27 smithi045 conmon[33655]: 2021-07-31T01:40:28.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:27 smithi045 conmon[33655]: audit 2021-07-31T01 2021-07-31T01:40:28.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:27 smithi045 conmon[33655]: :40:27.046440+0000 mon.a (mon.0) 107 : 2021-07-31T01:40:28.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:27 smithi045 conmon[33655]: audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:40:29.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:28 smithi045 conmon[33655]: cluster 2021-07-31T01:40:27.299849 2021-07-31T01:40:29.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:28 smithi045 conmon[33655]: +0000 mgr.y (mgr.14146) 24 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-07-31T01:40:29.346 INFO:teuthology.orchestra.run.smithi154.stdout: 2021-07-31T01:40:29.347 INFO:teuthology.orchestra.run.smithi154.stdout:[{"addr": "172.21.15.45", "hostname": "smithi045", "labels": [], "status": ""}, {"addr": "172.21.15.154", "hostname": "smithi154", "labels": [], "status": ""}] 2021-07-31T01:40:30.311 INFO:tasks.cephadm:Setting crush tunables to default 2021-07-31T01:40:30.311 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f52d2c14-f19f-11eb-8c24-001a4aab830c -- ceph osd crush tunables default 2021-07-31T01:40:30.773 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:30 smithi045 conmon[33655]: cluster 2021-07-31T01:40:29.300269+0000 mgr.y (mgr.14146) 25 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-07-31T01:40:30.773 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:30 smithi045 conmon[33655]: audit 2021-07-31T01:40:29.345403+0000 mgr.y (mgr.14146) 26 : audit [DBG] from='client.14168 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-07-31T01:40:32.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:32 smithi045 conmon[33655]: audit 2021-07-31T01:40:31.221724+0000 2021-07-31T01:40:32.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:32 smithi045 conmon[33655]: mon.a (mon.0) 108 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:40:32.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:32 smithi045 conmon[33655]: cluster 2021-07-31T01:40:31.300533+0000 mgr.y (mgr.14146) 27 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-07-31T01:40:32.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:32 smithi045 conmon[33655]: audit 2021-07-31T01:40:31.596466+0000 mon.a (mon.0) 109 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:40:32.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:32 smithi045 conmon[33655]: audit 2021-07-31T01:40:31.597011+0000 mon.a (mon.0) 110 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi154", "name": "osd_memory_target"}]: dispatch 2021-07-31T01:40:32.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:32 smithi045 conmon[33655]: audit 2021-07-31T01:40:31.599650+0000 mon.a (mon.0) 111 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:40:33.231 INFO:teuthology.orchestra.run.smithi045.stderr:adjusted tunables profile to default 2021-07-31T01:40:33.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:33 smithi045 conmon[33655]: audit 2021-07-31T01:40:32.509298+0000 mon.a (mon.0) 112 : audit [INF] from='client.? 172.21.15.45:0/375905286' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2021-07-31T01:40:33.953 INFO:tasks.cephadm:Adding mon.a on smithi045 2021-07-31T01:40:33.954 INFO:tasks.cephadm:Adding mon.c on smithi045 2021-07-31T01:40:33.954 INFO:tasks.cephadm:Adding mon.b on smithi154 2021-07-31T01:40:33.955 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f52d2c14-f19f-11eb-8c24-001a4aab830c -- ceph orch apply mon '3;smithi045:172.21.15.45=a;smithi045:[v2:172.21.15.45:3301,v1:172.21.15.45:6790]=c;smithi154:172.21.15.154=b' 2021-07-31T01:40:34.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:34 smithi045 conmon[33655]: audit 2021-07-31T01:40:33.227271+0000 mon.a (mon.0) 113 : audit [INF] from='client.? 172.21.15.45:0/375905286' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2021-07-31T01:40:34.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:34 smithi045 conmon[33655]: cluster 2021-07-31T01:40:33.227338+0000 mon.a (mon.0) 114 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2021-07-31T01:40:34.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:34 smithi045 conmon[33655]: cluster 2021-07-31T01:40:33.300825+0000 mgr.y (mgr.14146) 28 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-07-31T01:40:35.853 INFO:teuthology.orchestra.run.smithi154.stdout:Scheduled mon update... 2021-07-31T01:40:36.507 DEBUG:teuthology.orchestra.run.smithi045:mon.c> sudo journalctl -f -n 0 -u ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c@mon.c.service 2021-07-31T01:40:36.509 DEBUG:teuthology.orchestra.run.smithi154:mon.b> sudo journalctl -f -n 0 -u ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c@mon.b.service 2021-07-31T01:40:36.512 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2021-07-31T01:40:36.512 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f52d2c14-f19f-11eb-8c24-001a4aab830c -- ceph mon dump -f json 2021-07-31T01:40:36.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:-- Logs begin at Sat 2021-07-31 01:17:02 UTC. -- 2021-07-31T01:40:36.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:-- Logs begin at Sat 2021-07-31 01:16:15 UTC. -- 2021-07-31T01:40:37.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:36 smithi045 conmon[33655]: cluster 2021-07-31T01:40:35.301125+0000 mgr.y (mgr.14146) 29 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-07-31T01:40:37.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:36 smithi045 conmon[33655]: audit 2021-07-31T01:40:35.848331+0000 mgr.y (mgr.14146) 30 : audit [DBG] from='client.14172 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi045:172.21.15.45=a;smithi045:[v2:172.21.15.45:3301,v1:172.21.15.45:6790]=c;smithi154:172.21.15.154=b", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:40:37.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:36 smithi045 conmon[33655]: cephadm 2021-07-31T01:40:35.849802+0000 mgr.y (mgr.14146) 31 : cephadm [INF] Saving service mon spec with placement smithi045:172.21.15.45=a;smithi045:[v2:172.21.15.45:3301,v1:172.21.15.45:6790]=c;smithi154:172.21.15.154=b;count:3 2021-07-31T01:40:37.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:36 smithi045 conmon[33655]: audit 2021-07-31T01:40:35.852436+0000 mon.a (mon.0) 115 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:40:37.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:36 smithi045 conmon[33655]: audit 2021-07-31T01:40:35.853210+0000 mon.a (mon.0) 116 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:40:37.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:36 smithi045 conmon[33655]: audit 2021-07-31T01:40:35.854480+0000 mon.a (mon.0) 117 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:40:37.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:36 smithi045 conmon[33655]: audit 2021-07-31T01:40:35.855546+0000 mon.a 2021-07-31T01:40:37.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:36 smithi045 conmon[33655]: (mon.0) 118 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:40:37.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:36 smithi045 conmon[33655]: cephadm 2021-07-31T01:40:35.857059+0000 mgr.y (mgr.14146) 32 : cephadm [INF] Updating smithi154:/etc/ceph/ceph.conf 2021-07-31T01:40:37.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:36 smithi045 conmon[33655]: cephadm 2021-07-31T01:40:35.880803+0000 mgr.y (mgr.14146) 33 : cephadm [INF] Updating smithi154:/etc/ceph/ceph.client.admin.keyring 2021-07-31T01:40:37.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:36 smithi045 conmon[33655]: audit 2021-07-31T01:40:35.933468+0000 mon.a (mon.0) 119 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:40:37.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:36 smithi045 conmon[33655]: audit 2021-07-31T01:40:35.936105+0000 mon.a (mon.0) 120 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:40:37.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:36 smithi045 conmon[33655]: audit 2021-07-31T01:40:35.937772+0000 mon.a (mon.0) 121 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2021-07-31T01:40:37.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:36 smithi045 conmon[33655]: audit 2021-07-31T01:40:35.938612+0000 mon.a (mon.0) 122 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:40:37.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:36 smithi045 conmon[33655]: cephadm 2021-07-31T01:40:35.939613+0000 mgr.y (mgr.14146) 34 : cephadm [INF] Deploying daemon mon.c on smithi045 2021-07-31T01:40:38.451 INFO:teuthology.orchestra.run.smithi154.stdout: 2021-07-31T01:40:38.452 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"f52d2c14-f19f-11eb-8c24-001a4aab830c","modified":"2021-07-31T01:38:47.857698Z","created":"2021-07-31T01:38:47.857698Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2021-07-31T01:40:38.453 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2021-07-31T01:40:39.240 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:38 smithi045 conmon[33655]: cluster 2021-07-31T01:40:37.301469+0000 mgr.y 2021-07-31T01:40:39.240 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:38 smithi045 conmon[33655]: (mgr.14146) 35 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-07-31T01:40:39.241 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:38 smithi045 conmon[33655]: audit 2021-07-31T01:40:38.450964+0000 mon.a (mon.0) 123 : audit [DBG] from='client.? 172.21.15.154:0/1184789943' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-07-31T01:40:39.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 systemd[1]: Starting Ceph mon.c for f52d2c14-f19f-11eb-8c24-001a4aab830c... 2021-07-31T01:40:39.871 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 podman[39678]: 2021-07-31 01:40:39.477750463 +0000 UTC m=+0.031648644 image pull docker.io/ceph/daemon-base@sha256:0191c38131eb5e8b2e3cce0839223fa943f24bf2153c2767cb5bdb4a2973d559 2021-07-31T01:40:39.872 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 podman[39678]: 2021-07-31 01:40:39.612853569 +0000 UTC m=+0.166751827 container create a763c867885ccbedc8b7ac6b592e7eac68844348286c51b7256c203560955c6b (image=docker.io/ceph/daemon-base@sha256:0191c38131eb5e8b2e3cce0839223fa943f24bf2153c2767cb5bdb4a2973d559, name=ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c-mon.c, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20201204, GIT_BRANCH=HEAD, GIT_COMMIT=0b3dfd099f6a46b8c8fecd28a50330b63408461f, maintainer=Dimitri Savineau , org.label-schema.license=GPLv2, RELEASE=master-0b3dfd0, org.label-schema.name=CentOS Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True) 2021-07-31T01:40:39.872 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 podman[39678]: 2021-07-31 01:40:39.737816207 +0000 UTC m=+0.291714469 container init a763c867885ccbedc8b7ac6b592e7eac68844348286c51b7256c203560955c6b (image=docker.io/ceph/daemon-base@sha256:0191c38131eb5e8b2e3cce0839223fa943f24bf2153c2767cb5bdb4a2973d559, name=ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c-mon.c, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=master-0b3dfd0, org.label-schema.name=CentOS Base Image, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=0b3dfd099f6a46b8c8fecd28a50330b63408461f, maintainer=Dimitri Savineau , org.label-schema.license=GPLv2, org.label-schema.build-date=20201204, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2021-07-31T01:40:39.872 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.768+0000 7f13a761e700 0 set uid:gid to 167:167 (ceph:ceph) 2021-07-31T01:40:39.873 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.768+0000 7f13a761e700 0 ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable), process ceph-mon, pid 2 2021-07-31T01:40:39.873 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.768+0000 7f13a761e700 0 pidfile_write: ignore empty --pid-file 2021-07-31T01:40:39.873 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 podman[39678]: 2021-07-31 01:40:39.771066539 +0000 UTC m=+0.324964719 container start a763c867885ccbedc8b7ac6b592e7eac68844348286c51b7256c203560955c6b (image=docker.io/ceph/daemon-base@sha256:0191c38131eb5e8b2e3cce0839223fa943f24bf2153c2767cb5bdb4a2973d559, name=ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c-mon.c, CEPH_POINT_RELEASE=, maintainer=Dimitri Savineau , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=0b3dfd099f6a46b8c8fecd28a50330b63408461f, org.label-schema.name=CentOS Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20201204, RELEASE=master-0b3dfd0) 2021-07-31T01:40:39.873 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 bash[39602]: a763c867885ccbedc8b7ac6b592e7eac68844348286c51b7256c203560955c6b 2021-07-31T01:40:39.873 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.772+0000 7f13a761e700 0 load: jerasure load: lrc load: isa 2021-07-31T01:40:39.874 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: RocksDB version: 6.8.1 2021-07-31T01:40:39.874 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: 2021-07-31T01:40:39.874 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2021-07-31T01:40:39.874 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Compile date Jul 8 2021 2021-07-31T01:40:39.875 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: DB SUMMARY 2021-07-31T01:40:39.875 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: 2021-07-31T01:40:39.875 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: CURRENT file: CURRENT 2021-07-31T01:40:39.875 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: 2021-07-31T01:40:39.875 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: IDENTITY file: IDENTITY 2021-07-31T01:40:39.876 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: 2021-07-31T01:40:39.876 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: MANIFEST file: MANIFEST-000001 size: 13 Bytes 2021-07-31T01:40:39.876 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: 2021-07-31T01:40:39.877 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2021-07-31T01:40:39.877 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: 2021-07-31T01:40:39.877 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000003.log size: 511 ; 2021-07-31T01:40:39.877 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: 2021-07-31T01:40:39.878 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.error_if_exists: 0 2021-07-31T01:40:39.878 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.create_if_missing: 0 2021-07-31T01:40:39.879 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.paranoid_checks: 1 2021-07-31T01:40:39.879 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.env: 0x559a032131c0 2021-07-31T01:40:39.879 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.fs: Posix File System 2021-07-31T01:40:39.879 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.info_log: 0x559a03c2c6a0 2021-07-31T01:40:39.880 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.max_file_opening_threads: 16 2021-07-31T01:40:39.880 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.statistics: (nil) 2021-07-31T01:40:39.880 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.use_fsync: 0 2021-07-31T01:40:39.880 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.max_log_file_size: 0 2021-07-31T01:40:39.881 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2021-07-31T01:40:39.881 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.log_file_time_to_roll: 0 2021-07-31T01:40:39.882 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.keep_log_file_num: 1000 2021-07-31T01:40:39.882 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.recycle_log_file_num: 0 2021-07-31T01:40:39.882 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.allow_fallocate: 1 2021-07-31T01:40:39.882 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.allow_mmap_reads: 0 2021-07-31T01:40:39.882 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.883 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.allow_mmap_writes: 0 2021-07-31T01:40:39.883 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.use_direct_reads: 0 2021-07-31T01:40:39.883 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2021-07-31T01:40:39.883 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.create_missing_column_families: 0 2021-07-31T01:40:39.884 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.db_log_dir: 2021-07-31T01:40:39.884 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-c/store.db 2021-07-31T01:40:39.884 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.table_cache_numshardbits: 6 2021-07-31T01:40:39.884 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.max_subcompactions: 1 2021-07-31T01:40:39.885 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.max_background_flushes: -1 2021-07-31T01:40:39.885 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.WAL_ttl_seconds: 0 2021-07-31T01:40:39.885 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.WAL_size_limit_MB: 0 2021-07-31T01:40:39.886 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2021-07-31T01:40:39.886 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2021-07-31T01:40:39.886 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.is_fd_close_on_exec: 1 2021-07-31T01:40:39.886 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.advise_random_on_open: 1 2021-07-31T01:40:39.887 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.db_write_buffer_size: 0 2021-07-31T01:40:39.887 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.write_buffer_manager: 0x559a03cbffb0 2021-07-31T01:40:39.887 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2021-07-31T01:40:39.888 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2021-07-31T01:40:39.888 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2021-07-31T01:40:39.888 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.use_adaptive_mutex: 0 2021-07-31T01:40:39.889 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.rate_limiter: (nil) 2021-07-31T01:40:39.889 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2021-07-31T01:40:39.889 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.wal_recovery_mode: 2 2021-07-31T01:40:39.890 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.enable_thread_tracking: 0 2021-07-31T01:40:39.890 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.enable_pipelined_write: 0 2021-07-31T01:40:39.890 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.unordered_write: 0 2021-07-31T01:40:39.890 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2021-07-31T01:40:39.891 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2021-07-31T01:40:39.891 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2021-07-31T01:40:39.891 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2021-07-31T01:40:39.892 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.row_cache: None 2021-07-31T01:40:39.892 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.wal_filter: None 2021-07-31T01:40:39.892 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2021-07-31T01:40:39.892 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.allow_ingest_behind: 0 2021-07-31T01:40:39.893 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.preserve_deletes: 0 2021-07-31T01:40:39.893 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.two_write_queues: 0 2021-07-31T01:40:39.893 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.manual_wal_flush: 0 2021-07-31T01:40:39.893 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.atomic_flush: 0 2021-07-31T01:40:39.894 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.895 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2021-07-31T01:40:39.895 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.persist_stats_to_disk: 0 2021-07-31T01:40:39.896 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.write_dbid_to_manifest: 0 2021-07-31T01:40:39.896 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.log_readahead_size: 0 2021-07-31T01:40:39.896 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2021-07-31T01:40:39.896 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.max_background_jobs: 2 2021-07-31T01:40:39.896 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.max_background_compactions: -1 2021-07-31T01:40:39.897 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2021-07-31T01:40:39.897 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2021-07-31T01:40:39.897 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.delayed_write_rate : 16777216 2021-07-31T01:40:39.897 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.max_total_wal_size: 0 2021-07-31T01:40:39.898 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.898 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2021-07-31T01:40:39.898 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.stats_dump_period_sec: 600 2021-07-31T01:40:39.898 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.stats_persist_period_sec: 600 2021-07-31T01:40:39.898 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2021-07-31T01:40:39.899 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.max_open_files: -1 2021-07-31T01:40:39.899 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.bytes_per_sync: 0 2021-07-31T01:40:39.899 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.wal_bytes_per_sync: 0 2021-07-31T01:40:39.899 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.strict_bytes_per_sync: 0 2021-07-31T01:40:39.900 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.compaction_readahead_size: 0 2021-07-31T01:40:39.900 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Compression algorithms supported: 2021-07-31T01:40:39.900 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2021-07-31T01:40:39.900 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.900 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: kZSTD supported: 0 2021-07-31T01:40:39.901 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: kXpressCompression supported: 0 2021-07-31T01:40:39.901 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: kLZ4HCCompression supported: 1 2021-07-31T01:40:39.901 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: kLZ4Compression supported: 1 2021-07-31T01:40:39.901 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: kBZip2Compression supported: 0 2021-07-31T01:40:39.902 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: kZlibCompression supported: 1 2021-07-31T01:40:39.902 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: kSnappyCompression supported: 1 2021-07-31T01:40:39.902 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Fast CRC32 supported: Supported on x86 2021-07-31T01:40:39.902 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000001 2021-07-31T01:40:39.902 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: 2021-07-31T01:40:39.903 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2021-07-31T01:40:39.903 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: 2021-07-31T01:40:39.903 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2021-07-31T01:40:39.903 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.merge_operator: 2021-07-31T01:40:39.903 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: 2021-07-31T01:40:39.904 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.compaction_filter: None 2021-07-31T01:40:39.904 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.compaction_filter_factory: None 2021-07-31T01:40:39.904 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.memtable_factory: SkipListFactory 2021-07-31T01:40:39.904 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.table_factory: BlockBasedTable 2021-07-31T01:40:39.905 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x559a03bcbcc0) 2021-07-31T01:40:39.905 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: cache_index_and_filter_blocks: 1 2021-07-31T01:40:39.905 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: cache_index_and_filter_blocks_with_high_priority: 0 2021-07-31T01:40:39.905 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: pin_l0_filter_and_index_blocks_in_cache: 0 2021-07-31T01:40:39.905 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: pin_top_level_index_and_filter: 1 2021-07-31T01:40:39.906 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: index_type: 0 2021-07-31T01:40:39.906 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: data_block_index_type: 0 2021-07-31T01:40:39.906 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: index_shortening: 1 2021-07-31T01:40:39.906 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: data_block_hash_table_util_ratio: 0.750000 2021-07-31T01:40:39.906 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: hash_index_allow_collision: 1 2021-07-31T01:40:39.907 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: checksum: 1 2021-07-31T01:40:39.907 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: no_block_cache: 0 2021-07-31T01:40:39.907 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: block_cache: 0x559a03c04f10 2021-07-31T01:40:39.907 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: block_cache_name: BinnedLRUCache 2021-07-31T01:40:39.908 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: block_cache_options: 2021-07-31T01:40:39.909 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: capacity : 536870912 2021-07-31T01:40:39.909 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: num_shard_bits : 4 2021-07-31T01:40:39.910 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: strict_capacity_limit : 0 2021-07-31T01:40:39.910 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: high_pri_pool_ratio: 0.000 2021-07-31T01:40:39.910 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: block_cache_compressed: (nil) 2021-07-31T01:40:39.910 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: persistent_cache: (nil) 2021-07-31T01:40:39.910 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: block_size: 4096 2021-07-31T01:40:39.911 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: block_size_deviation: 10 2021-07-31T01:40:39.911 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: block_restart_interval: 16 2021-07-31T01:40:39.911 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: index_block_restart_interval: 1 2021-07-31T01:40:39.911 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: metadata_block_size: 4096 2021-07-31T01:40:39.911 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: partition_filters: 0 2021-07-31T01:40:39.912 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: use_delta_encoding: 1 2021-07-31T01:40:39.912 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: filter_policy: rocksdb.BuiltinBloomFilter 2021-07-31T01:40:39.912 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: whole_key_filtering: 1 2021-07-31T01:40:39.912 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: verify_compression: 0 2021-07-31T01:40:39.913 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: read_amp_bytes_per_bit: 0 2021-07-31T01:40:39.913 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: format_version: 2 2021-07-31T01:40:39.913 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: enable_index_compression: 1 2021-07-31T01:40:39.913 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: block_align: 0 2021-07-31T01:40:39.913 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: 2021-07-31T01:40:39.914 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.write_buffer_size: 33554432 2021-07-31T01:40:39.914 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.max_write_buffer_number: 2 2021-07-31T01:40:39.914 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: 2021-07-31T01:40:39.914 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.compression: NoCompression 2021-07-31T01:40:39.914 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.bottommost_compression: Disabled 2021-07-31T01:40:39.915 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.prefix_extractor: nullptr 2021-07-31T01:40:39.915 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2021-07-31T01:40:39.915 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.num_levels: 7 2021-07-31T01:40:39.915 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2021-07-31T01:40:39.915 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2021-07-31T01:40:39.916 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2021-07-31T01:40:39.916 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2021-07-31T01:40:39.916 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2021-07-31T01:40:39.916 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2021-07-31T01:40:39.917 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2021-07-31T01:40:39.917 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2021-07-31T01:40:39.917 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2021-07-31T01:40:39.917 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.compression_opts.window_bits: -14 2021-07-31T01:40:39.917 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.compression_opts.level: 32767 2021-07-31T01:40:39.918 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.compression_opts.strategy: 0 2021-07-31T01:40:39.918 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2021-07-31T01:40:39.918 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2021-07-31T01:40:39.918 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.compression_opts.enabled: false 2021-07-31T01:40:39.918 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2021-07-31T01:40:39.919 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2021-07-31T01:40:39.919 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2021-07-31T01:40:39.919 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.target_file_size_base: 67108864 2021-07-31T01:40:39.919 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.target_file_size_multiplier: 1 2021-07-31T01:40:39.920 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2021-07-31T01:40:39.920 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2021-07-31T01:40:39.920 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2021-07-31T01:40:39.920 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2021-07-31T01:40:39.920 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2021-07-31T01:40:39.921 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2021-07-31T01:40:39.921 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2021-07-31T01:40:39.921 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2021-07-31T01:40:39.921 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2021-07-31T01:40:39.921 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2021-07-31T01:40:39.922 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2021-07-31T01:40:39.922 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2021-07-31T01:40:39.922 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.arena_block_size: 4194304 2021-07-31T01:40:39.922 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2021-07-31T01:40:39.922 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2021-07-31T01:40:39.923 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2021-07-31T01:40:39.923 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.disable_auto_compactions: 0 2021-07-31T01:40:39.923 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2021-07-31T01:40:39.923 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2021-07-31T01:40:39.924 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2021-07-31T01:40:39.924 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2021-07-31T01:40:39.924 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2021-07-31T01:40:39.924 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2021-07-31T01:40:39.924 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2021-07-31T01:40:39.925 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2021-07-31T01:40:39.925 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2021-07-31T01:40:39.925 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2021-07-31T01:40:39.925 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.table_properties_collectors: 2021-07-31T01:40:39.925 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.inplace_update_support: 0 2021-07-31T01:40:39.926 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.inplace_update_num_locks: 10000 2021-07-31T01:40:39.926 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2021-07-31T01:40:39.926 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2021-07-31T01:40:39.926 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.memtable_huge_page_size: 0 2021-07-31T01:40:39.927 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.bloom_locality: 0 2021-07-31T01:40:39.927 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.max_successive_merges: 0 2021-07-31T01:40:39.927 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.optimize_filters_for_hits: 0 2021-07-31T01:40:39.927 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.paranoid_file_checks: 0 2021-07-31T01:40:39.927 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.force_consistency_checks: 0 2021-07-31T01:40:39.928 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.report_bg_io_stats: 0 2021-07-31T01:40:39.928 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.ttl: 2592000 2021-07-31T01:40:39.928 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.773+0000 7f13a761e700 4 rocksdb: Options.periodic_compaction_seconds: 0 2021-07-31T01:40:39.928 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.774+0000 7f13a761e700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2021-07-31T01:40:39.928 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: 2021-07-31T01:40:39.929 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.774+0000 7f13a761e700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2021-07-31T01:40:39.929 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: 2021-07-31T01:40:39.929 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.774+0000 7f13a761e700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1627695639775519, "job": 1, "event": "recovery_started", "log_files": [3]} 2021-07-31T01:40:39.929 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.929 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: 2021-07-31T01:40:39.774+0000 7f13a761e700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #3 mode 2 2021-07-31T01:40:39.930 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.774+0000 7f13a761e700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2021-07-31T01:40:39.930 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.774+0000 7f13a761e700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1627695639775970, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 4, "file_size": 1437, "table_properties": {"data_size": 523, "index_size": 40, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1627695639, "oldest_key_time": 3, "file_creation_time": 0}} 2021-07-31T01:40:39.930 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.774+0000 7f13a761e700 4 rocksdb: [version_set.cc:3826] Creating manifest 5 2021-07-31T01:40:39.930 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: 2021-07-31T01:40:39.931 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.775+0000 7f13a761e700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1627695639776398, "job": 1, "event": "recovery_finished"} 2021-07-31T01:40:39.931 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.775+0000 7f13a761e700 4 rocksdb: DB pointer 0x559a03ccb800 2021-07-31T01:40:39.931 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.776+0000 7f138e426700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2021-07-31T01:40:39.931 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.776+0000 7f138e426700 4 rocksdb: [db_impl/db_impl.cc:851] 2021-07-31T01:40:39.931 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: ** DB Stats ** 2021-07-31T01:40:39.932 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: Uptime(secs): 0.0 total, 0.0 interval 2021-07-31T01:40:39.932 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2021-07-31T01:40:39.932 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2021-07-31T01:40:39.932 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2021-07-31T01:40:39.932 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2021-07-31T01:40:39.933 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2021-07-31T01:40:39.933 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2021-07-31T01:40:39.933 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: 2021-07-31T01:40:39.933 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: ** Compaction Stats [default] ** 2021-07-31T01:40:39.934 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-07-31T01:40:39.934 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-07-31T01:40:39.934 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: L0 1/0 1.40 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.3 0.00 0.00 1 0.000 0 0 2021-07-31T01:40:39.934 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: Sum 1/0 1.40 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.3 0.00 0.00 1 0.000 0 0 2021-07-31T01:40:39.934 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.3 0.00 0.00 1 0.000 0 0 2021-07-31T01:40:39.935 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: 2021-07-31T01:40:39.935 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: ** Compaction Stats [default] ** 2021-07-31T01:40:39.935 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-07-31T01:40:39.935 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-07-31T01:40:39.935 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.3 0.00 0.00 1 0.000 0 0 2021-07-31T01:40:39.936 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: Uptime(secs): 0.0 total, 0.0 interval 2021-07-31T01:40:39.936 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: Flush(GB): cumulative 0.000, interval 0.000 2021-07-31T01:40:39.936 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: AddFile(GB): cumulative 0.000, interval 0.000 2021-07-31T01:40:39.936 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: AddFile(Total Files): cumulative 0, interval 0 2021-07-31T01:40:39.936 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: AddFile(L0 Files): cumulative 0, interval 0 2021-07-31T01:40:39.937 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: AddFile(Keys): cumulative 0, interval 0 2021-07-31T01:40:39.937 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: Cumulative compaction: 0.00 GB write, 0.58 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-07-31T01:40:39.937 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: Interval compaction: 0.00 GB write, 0.58 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-07-31T01:40:39.937 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2021-07-31T01:40:39.938 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: 2021-07-31T01:40:39.938 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: ** File Read Latency Histogram By Level [default] ** 2021-07-31T01:40:39.938 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: 2021-07-31T01:40:39.938 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: ** Compaction Stats [default] ** 2021-07-31T01:40:39.938 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-07-31T01:40:39.939 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-07-31T01:40:39.939 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: L0 1/0 1.40 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.3 0.00 0.00 1 0.000 0 0 2021-07-31T01:40:39.939 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: Sum 1/0 1.40 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.3 0.00 0.00 1 0.000 0 0 2021-07-31T01:40:39.939 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2021-07-31T01:40:39.939 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: 2021-07-31T01:40:39.940 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: ** Compaction Stats [default] ** 2021-07-31T01:40:39.940 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-07-31T01:40:39.940 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-07-31T01:40:39.940 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.3 0.00 0.00 1 0.000 0 0 2021-07-31T01:40:39.940 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: Uptime(secs): 0.0 total, 0.0 interval 2021-07-31T01:40:39.941 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: Flush(GB): cumulative 0.000, interval 0.000 2021-07-31T01:40:39.941 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: AddFile(GB): cumulative 0.000, interval 0.000 2021-07-31T01:40:39.941 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: AddFile(Total Files): cumulative 0, interval 0 2021-07-31T01:40:39.941 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: AddFile(L0 Files): cumulative 0, interval 0 2021-07-31T01:40:39.942 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: AddFile(Keys): cumulative 0, interval 0 2021-07-31T01:40:39.942 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: Cumulative compaction: 0.00 GB write, 0.57 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-07-31T01:40:39.942 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-07-31T01:40:39.942 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2021-07-31T01:40:39.942 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: 2021-07-31T01:40:39.943 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: ** File Read Latency Histogram By Level [default] ** 2021-07-31T01:40:39.943 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: 2021-07-31T01:40:39.943 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.776+0000 7f13a761e700 0 mon.c does not exist in monmap, will attempt to join an existing cluster 2021-07-31T01:40:39.943 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.776+0000 7f13a761e700 0 using public_addrv [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] 2021-07-31T01:40:39.943 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.776+0000 7f13a761e700 0 starting mon.c rank -1 at public addrs [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] at bind addrs [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid f52d2c14-f19f-11eb-8c24-001a4aab830c 2021-07-31T01:40:39.944 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.777+0000 7f13a761e700 1 mon.c@-1(???) e0 preinit fsid f52d2c14-f19f-11eb-8c24-001a4aab830c 2021-07-31T01:40:39.944 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.792+0000 7f1390c2b700 0 mon.c@-1(synchronizing).mds e1 new map 2021-07-31T01:40:39.944 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.792+0000 7f1390c2b700 0 mon.c@-1(synchronizing).mds e1 print_map 2021-07-31T01:40:39.944 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: e1 2021-07-31T01:40:39.944 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: enable_multiple, ever_enabled_multiple: 1,1 2021-07-31T01:40:39.945 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2021-07-31T01:40:39.945 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: legacy client fscid: -1 2021-07-31T01:40:39.945 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: 2021-07-31T01:40:39.945 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: No filesystems configured 2021-07-31T01:40:39.945 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: 2021-07-31T01:40:39.946 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.792+0000 7f1390c2b700 1 mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2021-07-31T01:40:39.946 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.792+0000 7f1390c2b700 1 mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2021-07-31T01:40:39.946 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.792+0000 7f1390c2b700 1 mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2021-07-31T01:40:39.946 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.792+0000 7f1390c2b700 1 mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2021-07-31T01:40:39.947 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.793+0000 7f1390c2b700 1 mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2021-07-31T01:40:39.947 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.793+0000 7f1390c2b700 1 mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2021-07-31T01:40:39.947 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.793+0000 7f1390c2b700 0 mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2021-07-31T01:40:39.947 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.793+0000 7f1390c2b700 0 mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2021-07-31T01:40:39.947 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.793+0000 7f1390c2b700 0 mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2021-07-31T01:40:39.948 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.793+0000 7f1390c2b700 0 mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2021-07-31T01:40:39.948 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: 2021-07-31T01:40:39.948 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: cluster 2021-07-31T01:40:39.948 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: 2021-07-31T01:40 2021-07-31T01:40:39.948 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: :37.301469+0000 mgr.y (mgr 2021-07-31T01:40:39.949 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: .14146) 35 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-07-31T01:40:39.949 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: 2021-07-31T01:40:39.949 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: audit 2021-07-31T01:40:39.949 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: 2021-07-31T01:40:38.450964+0000 2021-07-31T01:40:39.949 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: mon.a (mon.0) 123 : audit 2021-07-31T01:40:39.950 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: [DBG] from='client.? 172.21.15.154:0/1184789943' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-07-31T01:40:39.950 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 conmon[39718]: debug 2021-07-31T01:40:39.793+0000 7f1390c2b700 1 mon.c@-1(synchronizing).paxosservice(auth 1..4) refresh upgraded, format 0 -> 3 2021-07-31T01:40:39.950 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:39 smithi045 systemd[1]: Started Ceph mon.c for f52d2c14-f19f-11eb-8c24-001a4aab830c. 2021-07-31T01:40:40.062 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2021-07-31T01:40:40.063 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f52d2c14-f19f-11eb-8c24-001a4aab830c -- ceph mon dump -f json 2021-07-31T01:40:45.168 INFO:teuthology.orchestra.run.smithi154.stdout: 2021-07-31T01:40:45.168 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":2,"fsid":"f52d2c14-f19f-11eb-8c24-001a4aab830c","modified":"2021-07-31T01:40:39.797850Z","created":"2021-07-31T01:38:47.857698Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3301","nonce":0},{"type":"v1","addr":"172.21.15.45:6790","nonce":0}]},"addr":"172.21.15.45:6790/0","public_addr":"172.21.15.45:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2021-07-31T01:40:45.170 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 2 2021-07-31T01:40:45.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[33655]: cluster 2021-07-31T01:40:39.301677+0000 mgr.y (mgr.14146) 36 2021-07-31T01:40:45.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[33655]: : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-07-31T01:40:45.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[33655]: audit 2021-07-31T01:40:39.800022+0000 mon.a (mon.0) 127 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2021-07-31T01:40:45.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[33655]: cluster 2021-07-31T01:40:39.800149+0000 mon.a (mon.0) 128 : cluster [INF] mon.a calling monitor election 2021-07-31T01:40:45.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[33655]: audit 2021-07-31T01:40:39.801107+0000 mon.a (mon.0) 129 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-07-31T01:40:45.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[33655]: audit 2021-07-31T01:40:40.797681+0000 mon.a (mon.0) 130 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-07-31T01:40:45.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[33655]: cluster 2021-07-31T01:40:41.302045+0000 mgr.y (mgr.14146 2021-07-31T01:40:45.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[33655]: ) 37 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-07-31T01:40:45.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[33655]: audit 2021-07-31T01:40:41.797868+0000 mon.a (mon.0) 131 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-07-31T01:40:45.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[33655]: cluster 2021-07-31T01:40:45.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[33655]: 2021-07-31T01:40:41.798997+0000 mon.c (mon.1) 1 : cluster [INF] mon.c calling monitor election 2021-07-31T01:40:45.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[33655]: audit 2021-07-31T01:40:42 2021-07-31T01:40:45.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[33655]: .797981+0000 mon.a (mon.0) 132 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-07-31T01:40:45.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[33655]: cluster 2021-07-31T 2021-07-31T01:40:45.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[33655]: 01:40:43.302375+0000 mgr.y (mgr.14146) 38 : cluster [DBG] 2021-07-31T01:40:45.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[33655]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-07-31T01:40:45.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[33655]: audit 2021-07-31T01:40:45.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[33655]: 2021-07-31T01:40:43.798179+0000 mon.a (mon.0) 133 : audit 2021-07-31T01:40:45.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[33655]: [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-07-31T01:40:45.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[33655]: audit 2021-07-31T01:40:45.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[33655]: 2021-07-31T01:40:44.798178+0000 mon.a (mon.0) 134 : audit 2021-07-31T01:40:45.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[33655]: [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-07-31T01:40:45.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[33655]: cluster 2021-07-31T01:40:44.813991+0000 mon.a ( 2021-07-31T01:40:45.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[33655]: mon.0) 135 : cluster [INF] mon.a is new leader, mons a,c in quorum (ranks 0,1) 2021-07-31T01:40:45.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[33655]: cluster 2021-07-31T01:40: 2021-07-31T01:40:45.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[33655]: 44.816897+0000 mon.a (mon.0) 136 : cluster [DBG] monmap e2: 2 mons at {a=[v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0],c=[v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0]} 2021-07-31T01:40:45.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[33655]: cluster 2021-07-31 2021-07-31T01:40:45.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[33655]: T01:40:44.816945+0000 mon.a (mon.0) 137 : cluster [DBG] 2021-07-31T01:40:45.283 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[33655]: fsmap 2021-07-31T01:40:45.283 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[33655]: cluster 2021-07-31T01:40:45.283 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[33655]: 2021-07-31T01:40:44.816986+0000 mon.a (mon.0) 138 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2021-07-31T01:40:45.283 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[33655]: cluster 2021-07-31T01:40:44.817199+0000 2021-07-31T01:40:45.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[33655]: mon.a (mon.0) 139 : cluster [DBG] mgrmap e13: y(active, since 65s) 2021-07-31T01:40:45.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[33655]: cluster 2021-07-31T01 2021-07-31T01:40:45.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[33655]: :40:44.820391+0000 mon.a (mon.0) 140 : cluster [INF] overall HEALTH_OK 2021-07-31T01:40:45.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[33655]: audit 2021-07-31T01: 2021-07-31T01:40:45.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[33655]: 40:44.822690+0000 mon.a (mon.0) 141 : audit 2021-07-31T01:40:45.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[33655]: [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:40:45.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[33655]: audit 2021-07- 2021-07-31T01:40:45.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[33655]: 31T01:40:44.823864+0000 mon.a (mon.0) 142 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2021-07-31T01:40:45.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[33655]: audit 2021-07-31T01 2021-07-31T01:40:45.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[33655]: :40:44.824801+0000 mon.a (mon.0) 143 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:40:45.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[33655]: 2021-07-31T01:40:45.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[39718]: cluster 2021-07-31T01:40:39.301677 2021-07-31T01:40:45.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[39718]: +0000 mgr.y (mgr.14146) 36 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-07-31T01:40:45.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[39718]: audit 2021-07-31T01:40 2021-07-31T01:40:45.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[39718]: :39.800022+0000 mon.a (mon.0) 127 : audit 2021-07-31T01:40:45.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[39718]: [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2021-07-31T01:40:45.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[39718]: cluster 2021-07-31T 2021-07-31T01:40:45.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[39718]: 01:40:39.800149+0000 mon.a (mon.0 2021-07-31T01:40:45.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[39718]: ) 128 : cluster [INF] mon.a calling monitor election 2021-07-31T01:40:45.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T01:40:45.289 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[39718]: 31T01:40:39.801107+0000 mon.a (mon.0) 129 : 2021-07-31T01:40:45.289 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[39718]: audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-07-31T01:40:45.289 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T01:40:45.289 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[39718]: T01:40:40.797681+0000 mon.a (mon.0 2021-07-31T01:40:45.289 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[39718]: ) 130 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-07-31T01:40:45.290 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[39718]: cluster 2021- 2021-07-31T01:40:45.290 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[39718]: 07-31T01:40:41. 2021-07-31T01:40:45.290 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[39718]: 302045+0000 mgr.y (mgr.14146 2021-07-31T01:40:45.290 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[39718]: ) 37 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-07-31T01:40:45.291 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[39718]: 2021-07-31T01:40:45.291 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[39718]: audit 2021 2021-07-31T01:40:45.291 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[39718]: -07-31T01:40:41.797868+0000 mon.a (mon.0) 131 : audit 2021-07-31T01:40:45.291 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[39718]: [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-07-31T01:40:45.291 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[39718]: cluster 2021- 2021-07-31T01:40:45.292 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[39718]: 07-31T01:40:41.798997+0000 mon.c ( 2021-07-31T01:40:45.292 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[39718]: mon.1) 1 : cluster [INF] mon.c calling monitor election 2021-07-31T01:40:45.292 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[39718]: audit 2021-07-31T01:40:45.292 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[39718]: 2021-07- 2021-07-31T01:40:45.293 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[39718]: 31T01:40:42.797981+0000 mon.a (mon. 2021-07-31T01:40:45.293 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[39718]: 0) 132 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-07-31T01:40:45.293 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[39718]: cluster 2021 2021-07-31T01:40:45.293 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[39718]: -07-31T01:40:43.302375+0000 mgr.y (mgr 2021-07-31T01:40:45.293 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[39718]: .14146) 38 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-07-31T01:40:45.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[39718]: audit 2021 2021-07-31T01:40:45.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[39718]: -07-31T01:40:43.798179+0000 mon.a (mon.0) 133 : audit 2021-07-31T01:40:45.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[39718]: [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-07-31T01:40:45.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T01:40:45.295 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[39718]: 31T01:40:44.798178+0000 mon.a (mon.0) 134 : audit [DBG] 2021-07-31T01:40:45.295 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[39718]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-07-31T01:40:45.295 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[39718]: cluster 2021-07-31T01:40 2021-07-31T01:40:45.295 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[39718]: :44.813991+0000 mon.a (mon.0) 135 : cluster 2021-07-31T01:40:45.295 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[39718]: [INF] mon.a is new leader, mons a,c in quorum (ranks 0,1) 2021-07-31T01:40:45.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[39718]: cluster 2021-07-31 2021-07-31T01:40:45.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[39718]: T01:40:44.816897+0000 mon.a (mon.0 2021-07-31T01:40:45.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[39718]: ) 136 : cluster [DBG] monmap e2: 2 mons at {a=[v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0],c=[v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0]} 2021-07-31T01:40:45.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[39718]: cluster 2021-07 2021-07-31T01:40:45.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[39718]: -31T01:40:44.816945+0000 mon.a (mon.0 2021-07-31T01:40:45.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[39718]: ) 137 : cluster [DBG] fsmap 2021-07-31T01:40:45.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[39718]: cluster 2021-07-31T 2021-07-31T01:40:45.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[39718]: 01:40:44.816986+0000 mon.a (mon.0) 138 : cluster [DBG] 2021-07-31T01:40:45.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[39718]: osdmap e4: 0 total, 0 up, 0 in 2021-07-31T01:40:45.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[39718]: cluster 2021-07-31T01:40:45.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[39718]: 2021-07-31T01 2021-07-31T01:40:45.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[39718]: :40:44.817199+0000 mon.a (mon.0) 139 : cluster [DBG] mgrmap e13: y(active, since 65s) 2021-07-31T01:40:45.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[39718]: cluster 2021-07- 2021-07-31T01:40:45.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[39718]: 31T01:40:44.820391+0000 mon.a (mon.0) 140 2021-07-31T01:40:45.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[39718]: : cluster [INF] overall HEALTH_OK 2021-07-31T01:40:45.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T01:40:45.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[39718]: 31T01:40:44.822690+0000 mon.a (mon.0 2021-07-31T01:40:45.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[39718]: ) 141 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:40:45.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[39718]: audit 2021- 2021-07-31T01:40:45.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[39718]: 07-31T01:40:44.823864+0000 mon.a ( 2021-07-31T01:40:45.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[39718]: mon.0) 142 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2021-07-31T01:40:45.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[39718]: audit 2021 2021-07-31T01:40:45.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[39718]: -07-31T01:40:44.824801+0000 mon.a 2021-07-31T01:40:45.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:44 smithi045 conmon[39718]: (mon.0) 143 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:40:46.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:45 smithi045 conmon[33655]: cephadm 2021-07-31T01:40:44.825839+0000 mgr.y (mgr.14146) 2021-07-31T01:40:46.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:45 smithi045 conmon[33655]: 39 : cephadm [INF] Deploying daemon mon.b on smithi154 2021-07-31T01:40:46.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:45 smithi045 conmon[33655]: audit 2021-07-31T01:40:45.167709+0000 mon.a (mon.0) 144 2021-07-31T01:40:46.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:45 smithi045 conmon[33655]: : audit [DBG] from='client.? 172.21.15.154:0/786553110' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-07-31T01:40:46.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:45 smithi045 conmon[33655]: audit 2021-07-31T01:40:46.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:45 smithi045 conmon[33655]: 2021-07-31T01:40 2021-07-31T01:40:46.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:45 smithi045 conmon[33655]: :45.798394+0000 mon.a ( 2021-07-31T01:40:46.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:45 smithi045 conmon[33655]: mon.0) 145 : audit [DBG] 2021-07-31T01:40:46.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:45 smithi045 conmon[33655]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-07-31T01:40:46.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:45 smithi045 conmon[39718]: cephadm 2021-07-31T01:40:44.825839+0000 2021-07-31T01:40:46.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:45 smithi045 conmon[39718]: mgr.y (mgr.14146) 39 : cephadm [INF] Deploying daemon mon.b on smithi154 2021-07-31T01:40:46.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:45 smithi045 conmon[39718]: audit 2021-07-31T01:40 2021-07-31T01:40:46.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:45 smithi045 conmon[39718]: :45.167709+0000 mon.a (mon 2021-07-31T01:40:46.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:45 smithi045 conmon[39718]: .0) 144 : audit [DBG] from='client.? 172.21.15.154:0/786553110' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-07-31T01:40:46.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:45 smithi045 conmon[39718]: audit 2021-07 2021-07-31T01:40:46.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:45 smithi045 conmon[39718]: -31T01:40:45.798394+0000 2021-07-31T01:40:46.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:45 smithi045 conmon[39718]: mon.a (mon.0) 145 : audit 2021-07-31T01:40:46.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:45 smithi045 conmon[39718]: [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-07-31T01:40:46.905 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2021-07-31T01:40:46.906 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f52d2c14-f19f-11eb-8c24-001a4aab830c -- ceph mon dump -f json 2021-07-31T01:40:47.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:46 smithi045 conmon[33655]: cluster 2021-07-31T01:40:45.302736+0000 mgr.y (mgr 2021-07-31T01:40:47.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:46 smithi045 conmon[33655]: .14146) 40 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-07-31T01:40:47.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:46 smithi045 conmon[39718]: cluster 2021-07-31T01:40 2021-07-31T01:40:47.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:46 smithi045 conmon[39718]: :45.302736+0000 mgr.y (mgr.14146) 40 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-07-31T01:40:49.107 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 podman[34474]: 2021-07-31 01:40:48.948083011 +0000 UTC m=+0.030500683 image pull docker.io/ceph/daemon-base@sha256:0191c38131eb5e8b2e3cce0839223fa943f24bf2153c2767cb5bdb4a2973d559 2021-07-31T01:40:49.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:48 smithi045 conmon[33655]: cluster 2021-07-31T01:40:47.303033+0000 mgr.y 2021-07-31T01:40:49.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:48 smithi045 conmon[33655]: (mgr.14146) 41 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-07-31T01:40:49.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:48 smithi045 conmon[39718]: cluster 2021-07-31T01:40:47.303033+0000 2021-07-31T01:40:49.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:48 smithi045 conmon[39718]: mgr.y (mgr.14146) 41 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-07-31T01:40:49.368 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 podman[34474]: 2021-07-31 01:40:49.107333865 +0000 UTC m=+0.189751544 container create ea0311fb642566ebb2300a7906f85b72f0a004bf189bb356465d2b3f07be6e88 (image=docker.io/ceph/daemon-base@sha256:0191c38131eb5e8b2e3cce0839223fa943f24bf2153c2767cb5bdb4a2973d559, name=ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c-mon.b, org.label-schema.vendor=CentOS, GIT_COMMIT=0b3dfd099f6a46b8c8fecd28a50330b63408461f, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Dimitri Savineau , org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20201204, RELEASE=master-0b3dfd0, org.label-schema.name=CentOS Base Image, ceph=True, CEPH_POINT_RELEASE=) 2021-07-31T01:40:49.368 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 podman[34474]: 2021-07-31 01:40:49.264866895 +0000 UTC m=+0.347284690 container init ea0311fb642566ebb2300a7906f85b72f0a004bf189bb356465d2b3f07be6e88 (image=docker.io/ceph/daemon-base@sha256:0191c38131eb5e8b2e3cce0839223fa943f24bf2153c2767cb5bdb4a2973d559, name=ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c-mon.b, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Dimitri Savineau , org.label-schema.name=CentOS Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=0b3dfd099f6a46b8c8fecd28a50330b63408461f, org.label-schema.build-date=20201204, RELEASE=master-0b3dfd0) 2021-07-31T01:40:49.369 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.294+0000 7fba8b023700 0 set uid:gid to 167:167 (ceph:ceph) 2021-07-31T01:40:49.369 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.294+0000 7fba8b023700 0 ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable), process ceph-mon, pid 2 2021-07-31T01:40:49.369 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.295+0000 7fba8b023700 0 pidfile_write: ignore empty --pid-file 2021-07-31T01:40:49.369 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.299+0000 7fba8b023700 0 load: jerasure load: lrc load: isa 2021-07-31T01:40:49.370 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: RocksDB version: 6.8.1 2021-07-31T01:40:49.370 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: 2021-07-31T01:40:49.370 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2021-07-31T01:40:49.370 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.371 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Compile date Jul 8 2021 2021-07-31T01:40:49.371 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: DB SUMMARY 2021-07-31T01:40:49.371 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: 2021-07-31T01:40:49.371 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: CURRENT file: CURRENT 2021-07-31T01:40:49.372 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: 2021-07-31T01:40:49.372 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: IDENTITY file: IDENTITY 2021-07-31T01:40:49.372 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: 2021-07-31T01:40:49.372 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: MANIFEST file: MANIFEST-000001 size: 13 Bytes 2021-07-31T01:40:49.372 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: 2021-07-31T01:40:49.373 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2021-07-31T01:40:49.373 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: 2021-07-31T01:40:49.373 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000003.log size: 636 ; 2021-07-31T01:40:49.373 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: 2021-07-31T01:40:49.374 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.error_if_exists: 0 2021-07-31T01:40:49.374 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.create_if_missing: 0 2021-07-31T01:40:49.374 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.paranoid_checks: 1 2021-07-31T01:40:49.374 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.env: 0x556648fc21c0 2021-07-31T01:40:49.375 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.fs: Posix File System 2021-07-31T01:40:49.375 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.info_log: 0x55664ae706a0 2021-07-31T01:40:49.375 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.max_file_opening_threads: 16 2021-07-31T01:40:49.375 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.statistics: (nil) 2021-07-31T01:40:49.375 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.use_fsync: 0 2021-07-31T01:40:49.376 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.max_log_file_size: 0 2021-07-31T01:40:49.376 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2021-07-31T01:40:49.376 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.log_file_time_to_roll: 0 2021-07-31T01:40:49.376 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.keep_log_file_num: 1000 2021-07-31T01:40:49.377 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.recycle_log_file_num: 0 2021-07-31T01:40:49.377 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.allow_fallocate: 1 2021-07-31T01:40:49.377 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.allow_mmap_reads: 0 2021-07-31T01:40:49.377 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.allow_mmap_writes: 0 2021-07-31T01:40:49.377 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.use_direct_reads: 0 2021-07-31T01:40:49.378 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2021-07-31T01:40:49.378 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.create_missing_column_families: 0 2021-07-31T01:40:49.378 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.db_log_dir: 2021-07-31T01:40:49.378 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-b/store.db 2021-07-31T01:40:49.379 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.table_cache_numshardbits: 6 2021-07-31T01:40:49.379 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.max_subcompactions: 1 2021-07-31T01:40:49.379 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.max_background_flushes: -1 2021-07-31T01:40:49.379 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.WAL_ttl_seconds: 0 2021-07-31T01:40:49.379 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.WAL_size_limit_MB: 0 2021-07-31T01:40:49.380 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2021-07-31T01:40:49.380 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2021-07-31T01:40:49.380 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.is_fd_close_on_exec: 1 2021-07-31T01:40:49.380 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.advise_random_on_open: 1 2021-07-31T01:40:49.381 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.db_write_buffer_size: 0 2021-07-31T01:40:49.381 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.write_buffer_manager: 0x55664af03f20 2021-07-31T01:40:49.381 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: 2021-07-31T01:40:49.381 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2021-07-31T01:40:49.381 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2021-07-31T01:40:49.382 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2021-07-31T01:40:49.382 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.use_adaptive_mutex: 0 2021-07-31T01:40:49.382 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.rate_limiter: (nil) 2021-07-31T01:40:49.382 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2021-07-31T01:40:49.382 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.wal_recovery_mode: 2 2021-07-31T01:40:49.383 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.enable_thread_tracking: 0 2021-07-31T01:40:49.383 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.enable_pipelined_write: 0 2021-07-31T01:40:49.383 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.unordered_write: 0 2021-07-31T01:40:49.383 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2021-07-31T01:40:49.384 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: 2021-07-31T01:40:49.384 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2021-07-31T01:40:49.384 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2021-07-31T01:40:49.384 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2021-07-31T01:40:49.385 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.row_cache: None 2021-07-31T01:40:49.385 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.wal_filter: None 2021-07-31T01:40:49.385 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2021-07-31T01:40:49.385 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.allow_ingest_behind: 0 2021-07-31T01:40:49.385 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.preserve_deletes: 0 2021-07-31T01:40:49.386 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.two_write_queues: 0 2021-07-31T01:40:49.386 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.manual_wal_flush: 0 2021-07-31T01:40:49.386 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.atomic_flush: 0 2021-07-31T01:40:49.386 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2021-07-31T01:40:49.387 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.persist_stats_to_disk: 0 2021-07-31T01:40:49.387 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.write_dbid_to_manifest: 0 2021-07-31T01:40:49.387 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.log_readahead_size: 0 2021-07-31T01:40:49.387 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2021-07-31T01:40:49.387 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.max_background_jobs: 2 2021-07-31T01:40:49.388 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.max_background_compactions: -1 2021-07-31T01:40:49.388 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2021-07-31T01:40:49.388 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2021-07-31T01:40:49.388 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.delayed_write_rate : 16777216 2021-07-31T01:40:49.388 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.max_total_wal_size: 0 2021-07-31T01:40:49.389 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2021-07-31T01:40:49.389 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.stats_dump_period_sec: 600 2021-07-31T01:40:49.389 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.stats_persist_period_sec: 600 2021-07-31T01:40:49.389 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2021-07-31T01:40:49.390 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.max_open_files: -1 2021-07-31T01:40:49.390 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.bytes_per_sync: 0 2021-07-31T01:40:49.390 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.wal_bytes_per_sync: 0 2021-07-31T01:40:49.390 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.strict_bytes_per_sync: 0 2021-07-31T01:40:49.391 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.compaction_readahead_size: 0 2021-07-31T01:40:49.391 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Compression algorithms supported: 2021-07-31T01:40:49.391 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2021-07-31T01:40:49.391 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: kZSTD supported: 0 2021-07-31T01:40:49.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: kXpressCompression supported: 0 2021-07-31T01:40:49.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: kLZ4HCCompression supported: 1 2021-07-31T01:40:49.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: kLZ4Compression supported: 1 2021-07-31T01:40:49.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: 2021-07-31T01:40:49.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: kBZip2Compression supported: 0 2021-07-31T01:40:49.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: kZlibCompression supported: 1 2021-07-31T01:40:49.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: kSnappyCompression supported: 1 2021-07-31T01:40:49.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Fast CRC32 supported: Supported on x86 2021-07-31T01:40:49.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000001 2021-07-31T01:40:49.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: 2021-07-31T01:40:49.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2021-07-31T01:40:49.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: 2021-07-31T01:40:49.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2021-07-31T01:40:49.395 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.merge_operator: 2021-07-31T01:40:49.395 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.compaction_filter: None 2021-07-31T01:40:49.395 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.compaction_filter_factory: None 2021-07-31T01:40:49.395 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.memtable_factory: SkipListFactory 2021-07-31T01:40:49.396 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.table_factory: BlockBasedTable 2021-07-31T01:40:49.396 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55664ae0fcc0) 2021-07-31T01:40:49.396 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: cache_index_and_filter_blocks: 1 2021-07-31T01:40:49.396 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: cache_index_and_filter_blocks_with_high_priority: 0 2021-07-31T01:40:49.397 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: pin_l0_filter_and_index_blocks_in_cache: 0 2021-07-31T01:40:49.397 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: pin_top_level_index_and_filter: 1 2021-07-31T01:40:49.397 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: index_type: 0 2021-07-31T01:40:49.397 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: data_block_index_type: 0 2021-07-31T01:40:49.398 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: index_shortening: 1 2021-07-31T01:40:49.398 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: data_block_hash_table_util_ratio: 0.750000 2021-07-31T01:40:49.398 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: hash_index_allow_collision: 1 2021-07-31T01:40:49.398 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: checksum: 1 2021-07-31T01:40:49.399 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: no_block_cache: 0 2021-07-31T01:40:49.399 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: block_cache: 0x55664ae48f10 2021-07-31T01:40:49.399 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: block_cache_name: BinnedLRUCache 2021-07-31T01:40:49.399 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: block_cache_options: 2021-07-31T01:40:49.400 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: capacity : 536870912 2021-07-31T01:40:49.400 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: num_shard_bits : 4 2021-07-31T01:40:49.400 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: strict_capacity_limit : 0 2021-07-31T01:40:49.401 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: high_pri_pool_ratio: 0.000 2021-07-31T01:40:49.401 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: block_cache_compressed: (nil) 2021-07-31T01:40:49.401 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: persistent_cache: (nil) 2021-07-31T01:40:49.401 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: block_size: 4096 2021-07-31T01:40:49.402 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: block_size_deviation: 10 2021-07-31T01:40:49.402 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: block_restart_interval: 16 2021-07-31T01:40:49.402 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: index_block_restart_interval: 1 2021-07-31T01:40:49.402 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: metadata_block_size: 4096 2021-07-31T01:40:49.403 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: partition_filters: 0 2021-07-31T01:40:49.403 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: use_delta_encoding: 1 2021-07-31T01:40:49.403 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: filter_policy: rocksdb.BuiltinBloomFilter 2021-07-31T01:40:49.403 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: whole_key_filtering: 1 2021-07-31T01:40:49.404 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: verify_compression: 0 2021-07-31T01:40:49.404 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: read_amp_bytes_per_bit: 0 2021-07-31T01:40:49.405 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: format_version: 2 2021-07-31T01:40:49.405 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: enable_index_compression: 1 2021-07-31T01:40:49.406 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: block_align: 0 2021-07-31T01:40:49.406 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: 2021-07-31T01:40:49.406 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.write_buffer_size: 33554432 2021-07-31T01:40:49.406 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.max_write_buffer_number: 2 2021-07-31T01:40:49.407 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.compression: NoCompression 2021-07-31T01:40:49.407 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.bottommost_compression: Disabled 2021-07-31T01:40:49.407 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.prefix_extractor: nullptr 2021-07-31T01:40:49.407 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2021-07-31T01:40:49.408 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.num_levels: 7 2021-07-31T01:40:49.408 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2021-07-31T01:40:49.408 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2021-07-31T01:40:49.408 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2021-07-31T01:40:49.409 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2021-07-31T01:40:49.409 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2021-07-31T01:40:49.409 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2021-07-31T01:40:49.409 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2021-07-31T01:40:49.409 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2021-07-31T01:40:49.410 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2021-07-31T01:40:49.410 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.compression_opts.window_bits: -14 2021-07-31T01:40:49.410 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.compression_opts.level: 32767 2021-07-31T01:40:49.411 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.compression_opts.strategy: 0 2021-07-31T01:40:49.411 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2021-07-31T01:40:49.411 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2021-07-31T01:40:49.411 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.compression_opts.enabled: false 2021-07-31T01:40:49.412 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2021-07-31T01:40:49.412 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2021-07-31T01:40:49.412 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2021-07-31T01:40:49.412 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.target_file_size_base: 67108864 2021-07-31T01:40:49.412 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.target_file_size_multiplier: 1 2021-07-31T01:40:49.413 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2021-07-31T01:40:49.413 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2021-07-31T01:40:49.413 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2021-07-31T01:40:49.413 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2021-07-31T01:40:49.414 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2021-07-31T01:40:49.414 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2021-07-31T01:40:49.414 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2021-07-31T01:40:49.414 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2021-07-31T01:40:49.414 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2021-07-31T01:40:49.415 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2021-07-31T01:40:49.415 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2021-07-31T01:40:49.415 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2021-07-31T01:40:49.415 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.arena_block_size: 4194304 2021-07-31T01:40:49.416 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2021-07-31T01:40:49.416 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2021-07-31T01:40:49.416 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2021-07-31T01:40:49.416 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.disable_auto_compactions: 0 2021-07-31T01:40:49.416 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2021-07-31T01:40:49.417 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2021-07-31T01:40:49.417 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2021-07-31T01:40:49.417 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2021-07-31T01:40:49.417 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2021-07-31T01:40:49.418 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.418 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2021-07-31T01:40:49.418 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2021-07-31T01:40:49.418 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2021-07-31T01:40:49.418 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 bash[34359]: ea0311fb642566ebb2300a7906f85b72f0a004bf189bb356465d2b3f07be6e88 2021-07-31T01:40:49.419 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2021-07-31T01:40:49.419 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2021-07-31T01:40:49.419 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.table_properties_collectors: 2021-07-31T01:40:49.419 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.inplace_update_support: 0 2021-07-31T01:40:49.420 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.inplace_update_num_locks: 10000 2021-07-31T01:40:49.420 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2021-07-31T01:40:49.420 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2021-07-31T01:40:49.420 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.memtable_huge_page_size: 0 2021-07-31T01:40:49.420 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.bloom_locality: 0 2021-07-31T01:40:49.421 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.max_successive_merges: 0 2021-07-31T01:40:49.421 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.optimize_filters_for_hits: 0 2021-07-31T01:40:49.421 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.paranoid_file_checks: 0 2021-07-31T01:40:49.421 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.force_consistency_checks: 0 2021-07-31T01:40:49.421 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.report_bg_io_stats: 0 2021-07-31T01:40:49.422 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.ttl: 2592000 2021-07-31T01:40:49.422 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: Options.periodic_compaction_seconds: 0 2021-07-31T01:40:49.422 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2021-07-31T01:40:49.422 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: 2021-07-31T01:40:49.423 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.423 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2021-07-31T01:40:49.423 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: 2021-07-31T01:40:49.423 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1627695649301794, "job": 1, "event": "recovery_started", "log_files": [3]} 2021-07-31T01:40:49.423 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.300+0000 7fba8b023700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #3 mode 2 2021-07-31T01:40:49.424 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.301+0000 7fba8b023700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2021-07-31T01:40:49.424 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.301+0000 7fba8b023700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1627695649302255, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 4, "file_size": 1562, "table_properties": {"data_size": 648, "index_size": 40, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1627695649, "oldest_key_time": 3, "file_creation_time": 0}} 2021-07-31T01:40:49.424 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.301+0000 7fba8b023700 4 rocksdb: [version_set.cc:3826] Creating manifest 5 2021-07-31T01:40:49.424 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: 2021-07-31T01:40:49.425 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.301+0000 7fba8b023700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1627695649302629, "job": 1, "event": "recovery_finished"} 2021-07-31T01:40:49.425 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.302+0000 7fba8b023700 4 rocksdb: DB pointer 0x55664af0f800 2021-07-31T01:40:49.425 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.302+0000 7fba71e2b700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2021-07-31T01:40:49.425 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.302+0000 7fba71e2b700 4 rocksdb: [db_impl/db_impl.cc:851] 2021-07-31T01:40:49.426 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: ** DB Stats ** 2021-07-31T01:40:49.426 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: Uptime(secs): 0.0 total, 0.0 interval 2021-07-31T01:40:49.426 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2021-07-31T01:40:49.426 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2021-07-31T01:40:49.426 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2021-07-31T01:40:49.427 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2021-07-31T01:40:49.427 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2021-07-31T01:40:49.427 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2021-07-31T01:40:49.427 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: 2021-07-31T01:40:49.428 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: ** Compaction Stats [default] ** 2021-07-31T01:40:49.428 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-07-31T01:40:49.428 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-07-31T01:40:49.428 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: L0 1/0 1.53 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.5 0.00 0.00 1 0.000 0 0 2021-07-31T01:40:49.428 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: Sum 1/0 1.53 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.5 0.00 0.00 1 0.000 0 0 2021-07-31T01:40:49.429 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.5 0.00 0.00 1 0.000 0 0 2021-07-31T01:40:49.429 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: 2021-07-31T01:40:49.429 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: ** Compaction Stats [default] ** 2021-07-31T01:40:49.429 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-07-31T01:40:49.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-07-31T01:40:49.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.5 0.00 0.00 1 0.000 0 0 2021-07-31T01:40:49.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: Uptime(secs): 0.0 total, 0.0 interval 2021-07-31T01:40:49.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: Flush(GB): cumulative 0.000, interval 0.000 2021-07-31T01:40:49.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: AddFile(GB): cumulative 0.000, interval 0.000 2021-07-31T01:40:49.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: AddFile(Total Files): cumulative 0, interval 0 2021-07-31T01:40:49.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: AddFile(L0 Files): cumulative 0, interval 0 2021-07-31T01:40:49.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: AddFile(Keys): cumulative 0, interval 0 2021-07-31T01:40:49.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: Cumulative compaction: 0.00 GB write, 0.70 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-07-31T01:40:49.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: Interval compaction: 0.00 GB write, 0.70 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-07-31T01:40:49.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2021-07-31T01:40:49.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: 2021-07-31T01:40:49.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: ** File Read Latency Histogram By Level [default] ** 2021-07-31T01:40:49.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: 2021-07-31T01:40:49.433 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: ** Compaction Stats [default] ** 2021-07-31T01:40:49.433 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-07-31T01:40:49.433 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-07-31T01:40:49.433 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: L0 1/0 1.53 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.5 0.00 0.00 1 0.000 0 0 2021-07-31T01:40:49.434 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: Sum 1/0 1.53 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.5 0.00 0.00 1 0.000 0 0 2021-07-31T01:40:49.434 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2021-07-31T01:40:49.434 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: 2021-07-31T01:40:49.434 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: ** Compaction Stats [default] ** 2021-07-31T01:40:49.435 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-07-31T01:40:49.435 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-07-31T01:40:49.435 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.5 0.00 0.00 1 0.000 0 0 2021-07-31T01:40:49.435 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: Uptime(secs): 0.0 total, 0.0 interval 2021-07-31T01:40:49.436 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: Flush(GB): cumulative 0.000, interval 0.000 2021-07-31T01:40:49.436 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: AddFile(GB): cumulative 0.000, interval 0.000 2021-07-31T01:40:49.436 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: AddFile(Total Files): cumulative 0, interval 0 2021-07-31T01:40:49.436 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: AddFile(L0 Files): cumulative 0, interval 0 2021-07-31T01:40:49.436 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: AddFile(Keys): cumulative 0, interval 0 2021-07-31T01:40:49.437 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: Cumulative compaction: 0.00 GB write, 0.69 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-07-31T01:40:49.437 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-07-31T01:40:49.437 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2021-07-31T01:40:49.437 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: 2021-07-31T01:40:49.437 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: ** File Read Latency Histogram By Level [default] ** 2021-07-31T01:40:49.438 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: 2021-07-31T01:40:49.438 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.302+0000 7fba8b023700 0 mon.b does not exist in monmap, will attempt to join an existing cluster 2021-07-31T01:40:49.438 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.302+0000 7fba8b023700 0 using public_addr v2:172.21.15.154:0/0 -> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2021-07-31T01:40:49.438 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.303+0000 7fba8b023700 0 starting mon.b rank -1 at public addrs [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] at bind addrs [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid f52d2c14-f19f-11eb-8c24-001a4aab830c 2021-07-31T01:40:49.439 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.304+0000 7fba8b023700 1 mon.b@-1(???) e0 preinit fsid f52d2c14-f19f-11eb-8c24-001a4aab830c 2021-07-31T01:40:49.439 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 podman[34474]: 2021-07-31 01:40:49.306417469 +0000 UTC m=+0.388835140 container start ea0311fb642566ebb2300a7906f85b72f0a004bf189bb356465d2b3f07be6e88 (image=docker.io/ceph/daemon-base@sha256:0191c38131eb5e8b2e3cce0839223fa943f24bf2153c2767cb5bdb4a2973d559, name=ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c-mon.b, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Dimitri Savineau , org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20201204, org.label-schema.vendor=CentOS, GIT_COMMIT=0b3dfd099f6a46b8c8fecd28a50330b63408461f, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=master-0b3dfd0) 2021-07-31T01:40:49.439 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.324+0000 7fba74630700 0 mon.b@-1(synchronizing).mds e1 new map 2021-07-31T01:40:49.439 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.324+0000 7fba74630700 0 mon.b@-1(synchronizing).mds e1 print_map 2021-07-31T01:40:49.440 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: e1 2021-07-31T01:40:49.440 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: enable_multiple, ever_enabled_multiple: 1,1 2021-07-31T01:40:49.440 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2021-07-31T01:40:49.440 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: legacy client fscid: -1 2021-07-31T01:40:49.440 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: 2021-07-31T01:40:49.441 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: No filesystems configured 2021-07-31T01:40:49.441 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: 2021-07-31T01:40:49.441 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.324+0000 7fba74630700 1 mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2021-07-31T01:40:49.441 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.324+0000 7fba74630700 1 mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2021-07-31T01:40:49.441 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.324+0000 7fba74630700 1 mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2021-07-31T01:40:49.442 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.324+0000 7fba74630700 1 mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2021-07-31T01:40:49.442 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.324+0000 7fba74630700 1 mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2021-07-31T01:40:49.442 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.324+0000 7fba74630700 1 mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2021-07-31T01:40:49.442 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.325+0000 7fba74630700 0 mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2021-07-31T01:40:49.443 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.325+0000 7fba74630700 0 mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2021-07-31T01:40:49.443 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.325+0000 7fba74630700 0 mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2021-07-31T01:40:49.443 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.325+0000 7fba74630700 0 mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2021-07-31T01:40:49.443 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: cluster 2021-07-31T01:40:49.443 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: 2021-07-31T01:40:47.303033+0000 mgr.y (mgr.14146 2021-07-31T01:40:49.444 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: ) 41 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-07-31T01:40:49.444 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.325+0000 7fba74630700 1 mon.b@-1(synchronizing).paxosservice(auth 1..4) refresh upgraded, format 0 -> 3 2021-07-31T01:40:49.444 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.326+0000 7fba74630700 20 expand_channel_meta expand map: {default=false} 2021-07-31T01:40:49.444 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.326+0000 7fba74630700 20 expand_channel_meta from 'false' to 'false' 2021-07-31T01:40:49.445 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.326+0000 7fba74630700 20 expand_channel_meta expanded map: {default=false} 2021-07-31T01:40:49.445 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.326+0000 7fba74630700 20 expand_channel_meta expand map: {default=info} 2021-07-31T01:40:49.445 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.326+0000 7fba74630700 20 expand_channel_meta from 'info' to 'info' 2021-07-31T01:40:49.445 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.326+0000 7fba74630700 20 expand_channel_meta expanded map: {default=info} 2021-07-31T01:40:49.445 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.326+0000 7fba74630700 20 expand_channel_meta expand map: {default=daemon} 2021-07-31T01:40:49.446 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.326+0000 7fba74630700 20 expand_channel_meta from 'daemon' to 'daemon' 2021-07-31T01:40:49.446 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.326+0000 7fba74630700 20 expand_channel_meta expanded map: {default=daemon} 2021-07-31T01:40:49.446 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.326+0000 7fba74630700 20 expand_channel_meta expand map: {default=debug} 2021-07-31T01:40:49.446 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.326+0000 7fba74630700 20 expand_channel_meta from 'debug' to 'debug' 2021-07-31T01:40:49.446 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 conmon[34553]: debug 2021-07-31T01:40:49.326+0000 7fba74630700 20 expand_channel_meta expanded map: {default=debug} 2021-07-31T01:40:49.447 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:49 smithi154 systemd[1]: Started Ceph mon.b for f52d2c14-f19f-11eb-8c24-001a4aab830c. 2021-07-31T01:40:54.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[33655]: cluster 2021-07-31T01:40:49.303371+0000 mgr.y ( 2021-07-31T01:40:54.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[33655]: mgr.14146) 42 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-07-31T01:40:54.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[33655]: audit 2021-07-31T01:40:49.332924+0000 mon.a (mon.0) 147 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2021-07-31T01:40:54.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[33655]: cluster 2021-07-31T01:40:54.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[33655]: 2021-07-31T01:40:49.333114+0000 mon.a (mon.0) 148 : cluster [INF] mon.a calling monitor election 2021-07-31T01:40:54.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[33655]: cluster 2021-07-31T01:40:54.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[33655]: 2021-07-31T01:40:49.333352+0000 mon.c (mon.1) 2 : 2021-07-31T01:40:54.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[33655]: cluster [INF] mon.c calling monitor election 2021-07-31T01:40:54.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[33655]: audit 2021-07-31T01:40:54.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[33655]: 2021-07-31T01:40:49.345958+0000 mon.a (mon.0) 149 2021-07-31T01:40:54.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[33655]: : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-07-31T01:40:54.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[33655]: audit 2021-07-31T01:40:54.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[33655]: 2021-07-31T01:40:49.346102+0000 mon.a (mon.0) 150 : audit 2021-07-31T01:40:54.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[33655]: [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-07-31T01:40:54.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[33655]: audit 2021- 2021-07-31T01:40:54.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[33655]: 07-31T01:40:50.329771+0000 mon.a (mon.0) 151 : audit [DBG] 2021-07-31T01:40:54.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[33655]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-07-31T01:40:54.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[33655]: cluster 2021-07 2021-07-31T01:40:54.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[33655]: -31T01:40:51.303683+0000 mgr.y (mgr.14146) 43 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-07-31T01:40:54.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[33655]: 2021-07-31T01:40:54.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[33655]: audit 2021-07- 2021-07-31T01:40:54.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[33655]: 31T01:40:51.329593+0000 mon.a (mon.0) 152 : audit [DBG] 2021-07-31T01:40:54.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[33655]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-07-31T01:40:54.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[33655]: audit 2021-07-31 2021-07-31T01:40:54.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[33655]: T01:40:52.329796+0000 mon.a (mon.0) 153 : audit 2021-07-31T01:40:54.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[33655]: [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-07-31T01:40:54.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[33655]: cluster 2021-07- 2021-07-31T01:40:54.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[33655]: 31T01:40:53.303945+0000 mgr.y 2021-07-31T01:40:54.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[33655]: (mgr.14146) 44 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-07-31T01:40:54.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[33655]: 2021-07-31T01:40:54.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[33655]: audit 2021 2021-07-31T01:40:54.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[33655]: -07-31T01:40:53.330078 2021-07-31T01:40:54.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[33655]: +0000 mon.a (mon.0) 154 : audit 2021-07-31T01:40:54.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[33655]: [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-07-31T01:40:54.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[33655]: audit 2021-07-31T01:40:54.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[33655]: 2021-07-31T01:40:54 2021-07-31T01:40:54.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[33655]: .330088+0000 mon.a (mon.0) 155 2021-07-31T01:40:54.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[33655]: : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-07-31T01:40:54.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[33655]: cluster 2021-07-31 2021-07-31T01:40:54.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[33655]: T01:40:54.346142+0000 mon.a (mon 2021-07-31T01:40:54.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[33655]: .0) 156 : cluster [INF] mon.a is new leader, mons a,c in quorum (ranks 0,1) 2021-07-31T01:40:54.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[33655]: cluster 2021-07-31T 2021-07-31T01:40:54.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[33655]: 01:40:54.348901+0000 mon.a (mon.0) 2021-07-31T01:40:54.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[39718]: cluster 2021-07-31T01:40:49.303371+0000 mgr.y (mgr.14146) 42 2021-07-31T01:40:54.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[39718]: : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-07-31T01:40:54.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[39718]: audit 2021-07-31T01:40:49.332924+0000 mon.a (mon.0) 147 : audit [DBG] 2021-07-31T01:40:54.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[39718]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2021-07-31T01:40:54.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[39718]: cluster 2021-07-31T01:40: 2021-07-31T01:40:54.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[39718]: 49.333114+0000 mon.a (mon.0) 148 : cluster [INF] mon.a calling monitor election 2021-07-31T01:40:54.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[39718]: cluster 2021-07-31T01:40:49.333352 2021-07-31T01:40:54.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[39718]: +0000 mon.c (mon.1) 2 : cluster [INF] mon.c calling monitor election 2021-07-31T01:40:54.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[39718]: audit 2021-07 2021-07-31T01:40:54.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[39718]: -31T01:40:49.345958+0000 mon.a (mon.0) 149 : audit 2021-07-31T01:40:54.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[39718]: [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-07-31T01:40:54.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[39718]: audit 2021-07 2021-07-31T01:40:54.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[39718]: -31T01:40:49.346102+0000 mon.a (mon.0) 150 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-07-31T01:40:54.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[39718]: 2021-07-31T01:40:54.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[39718]: audit 2021-07-31T01:40:50.329771+0000 mon.a 2021-07-31T01:40:54.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[39718]: (mon.0) 151 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-07-31T01:40:54.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[39718]: cluster 2021-07-31T01:40 2021-07-31T01:40:54.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[39718]: :51.303683+0000 mgr.y (mgr.14146) 43 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-07-31T01:40:54.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T01:40:54.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[39718]: T01:40:51.329593+0000 mon.a (mon.0) 152 2021-07-31T01:40:54.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[39718]: : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-07-31T01:40:54.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[39718]: audit 2021-07-31T01:40:52.329796+0000 mon.a ( 2021-07-31T01:40:54.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[39718]: mon.0) 153 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-07-31T01:40:54.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[39718]: cluster 2021- 2021-07-31T01:40:54.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[39718]: 07-31T01:40:53.303945+0000 mgr.y (mgr.14146) 44 : 2021-07-31T01:40:54.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[39718]: cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-07-31T01:40:54.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[39718]: audit 2021-07-31T01:40:53.330078+0000 mon.a 2021-07-31T01:40:54.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[39718]: (mon.0) 154 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-07-31T01:40:54.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[39718]: audit 2021-07-31T01: 2021-07-31T01:40:54.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[39718]: 40:54.330088+0000 mon.a (mon.0) 155 : audit [DBG] 2021-07-31T01:40:54.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[39718]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-07-31T01:40:54.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[39718]: cluster 2021-07-31T01:40:54.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[39718]: 2021-07-31T01:40:54.346142+0000 mon.a (mon.0) 156 : cluster [INF] mon.a is new leader, mons a,c in quorum (ranks 0,1) 2021-07-31T01:40:54.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[39718]: cluster 2021-07-31T01:40:54.348901+0000 mon.a ( 2021-07-31T01:40:54.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[39718]: mon.0) 157 : cluster [DBG] monmap e3: 3 mons at {a=[v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0],b=[v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0],c=[v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0]} 2021-07-31T01:40:54.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[39718]: cluster 2021-07-31T01:40:54.348968+0000 2021-07-31T01:40:54.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[39718]: mon.a (mon.0) 158 : cluster [DBG] fsmap 2021-07-31T01:40:54.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[39718]: cluster 2021-07-31T01:40:54.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[39718]: 2021-07-31T01:40:54.349015+0000 mon.a (mon 2021-07-31T01:40:54.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[39718]: .0) 159 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2021-07-31T01:40:54.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[39718]: cluster 2021-07-31T01:40:54 2021-07-31T01:40:54.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[39718]: .349252+0000 mon.a (mon.0) 160 : cluster [DBG] mgrmap e13: y(active, since 75s) 2021-07-31T01:40:54.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[39718]: cluster 2021 2021-07-31T01:40:54.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[39718]: -07-31T01:40:54.349471+0000 mon.a (mon.0) 2021-07-31T01:40:54.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[39718]: 161 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2021-07-31T01:40:54.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[39718]: cluster 2021-07-31T 2021-07-31T01:40:54.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[39718]: 01:40:54.353260+0000 mon.a (mon.0) 162 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2021-07-31T01:40:54.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[39718]: 2021-07-31T01:40:54.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[39718]: cluster 2021-07- 2021-07-31T01:40:54.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[39718]: 31T01:40:54.353298+0000 mon.a (mon.0) 163 : cluster [WRN] 2021-07-31T01:40:54.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[39718]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2021-07-31T01:40:54.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[39718]: cluster 2021-07-31T01:40:54.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[39718]: 2021-07-31T01:40:54.353309+0000 mon.a (mon.0) 164 : cluster [WRN] mon.b (rank 2) addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] is down (out of quorum) 2021-07-31T01:40:54.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[39718]: 2021-07-31T01:40:54.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T01:40:54.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[39718]: 01:40:54.355768+0000 mon.a (mon.0) 165 : audit [INF] 2021-07-31T01:40:54.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[39718]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:40:54.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[39718]: audit 2021-07 2021-07-31T01:40:54.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[39718]: -31T01:40:54.356985+0000 mon.a (mon.0) 166 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:40:54.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T01:40:54.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[39718]: 31T01:40:54.358282+0000 mon.a (mon.0) 167 : audit 2021-07-31T01:40:54.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[39718]: [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:40:54.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[39718]: audit 2021-07-31T01:40:54.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[39718]: 2021-07-31T01:40:54.359185+0000 mon.a (mon.0) 168 : 2021-07-31T01:40:54.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[39718]: audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:40:54.802 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[33655]: 157 : cluster [DBG] monmap e3: 3 mons at {a=[v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0],b=[v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0],c=[v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0]} 2021-07-31T01:40:54.802 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[33655]: cluster 2021-07-31T01:40:54.802 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[33655]: 2021-07-31T01:40:54.348968 2021-07-31T01:40:54.803 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[33655]: +0000 mon.a (mon.0) 158 : cluster [DBG] 2021-07-31T01:40:54.803 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[33655]: fsmap 2021-07-31T01:40:54.803 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[33655]: cluster 2021-07-31T01:40:54.803 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[33655]: 2021-07-31T01:40 2021-07-31T01:40:54.803 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[33655]: :54.349015+0000 mon.a (mon.0 2021-07-31T01:40:54.804 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[33655]: ) 159 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2021-07-31T01:40:54.804 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[33655]: cluster 2021-07-31T01:40:54.804 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[33655]: 2021-07-31T01:40:54. 2021-07-31T01:40:54.805 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[33655]: 349252+0000 mon.a (mon.0) 160 : 2021-07-31T01:40:54.805 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[33655]: cluster [DBG] mgrmap e13: y(active, since 75s) 2021-07-31T01:40:54.805 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[33655]: cluster 2021-07-31T01:40:54.805 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[33655]: 2021-07-31T 2021-07-31T01:40:54.806 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[33655]: 01:40:54.349471+0000 mon.a (mon. 2021-07-31T01:40:54.806 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[33655]: 0) 161 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2021-07-31T01:40:54.806 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[33655]: cluster 2021-07-31 2021-07-31T01:40:54.806 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[33655]: T01:40:54.353260+0000 mon.a (mon.0) 162 : cluster 2021-07-31T01:40:54.807 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[33655]: [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2021-07-31T01:40:54.807 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[33655]: cluster 2021- 2021-07-31T01:40:54.807 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[33655]: 07-31T01:40:54.353298+0000 mon.a ( 2021-07-31T01:40:54.807 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[33655]: mon.0) 163 : cluster [WRN] [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2021-07-31T01:40:54.808 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[33655]: cluster 2021-07-31T01:40:54.808 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[33655]: 2021-07-31T01:40:54. 2021-07-31T01:40:54.808 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[33655]: 353309+0000 mon.a (mon.0) 164 : cluster 2021-07-31T01:40:54.808 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[33655]: [WRN] mon.b (rank 2) addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] is down (out of quorum) 2021-07-31T01:40:54.809 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[33655]: audit 2021-07-31T01:40:54.809 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[33655]: 2021-07-31T01:40: 2021-07-31T01:40:54.809 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[33655]: 54.355768+0000 mon.a (mon.0) 2021-07-31T01:40:54.809 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[33655]: 165 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:40:54.810 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[33655]: audit 2021-07-31T01:40:54.810 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[33655]: 2021-07-31T 2021-07-31T01:40:54.810 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[33655]: 01:40:54.356985+0000 mon.a (mon 2021-07-31T01:40:54.810 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[33655]: .0) 166 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:40:54.811 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[33655]: audit 2021-07 2021-07-31T01:40:54.811 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[33655]: -31T01:40:54.358282+0000 mon.a (mon 2021-07-31T01:40:54.811 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[33655]: .0) 167 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:40:54.811 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[33655]: audit 2021-07- 2021-07-31T01:40:54.812 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[33655]: 31T01:40:54.359185+0000 mon.a (mon. 2021-07-31T01:40:54.812 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:54 smithi045 conmon[33655]: 0) 168 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:40:55.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:55 smithi045 conmon[33655]: audit 2021-07-31T01:40:54.418583+0000 mon.a (mon.0) 169 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:40:55.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:55 smithi045 conmon[33655]: audit 2021-07-31T01:40:55.330118+0000 mon.a (mon.0) 170 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-07-31T01:40:55.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:55 smithi045 conmon[39718]: audit 2021-07-31T01:40:54.418583+0000 mon.a (mon.0) 169 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:40:55.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:55 smithi045 conmon[39718]: audit 2021-07-31T01:40:55.330118+0000 mon.a (mon.0) 170 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-07-31T01:40:55.823 INFO:teuthology.orchestra.run.smithi154.stdout: 2021-07-31T01:40:55.823 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":3,"fsid":"f52d2c14-f19f-11eb-8c24-001a4aab830c","modified":"2021-07-31T01:40:49.329783Z","created":"2021-07-31T01:38:47.857698Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3301","nonce":0},{"type":"v1","addr":"172.21.15.45:6790","nonce":0}]},"addr":"172.21.15.45:6790/0","public_addr":"172.21.15.45:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2021-07-31T01:40:55.825 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 3 2021-07-31T01:40:56.553 INFO:tasks.cephadm:Generating final ceph.conf file... 2021-07-31T01:40:56.554 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f52d2c14-f19f-11eb-8c24-001a4aab830c -- ceph config generate-minimal-conf 2021-07-31T01:40:56.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:56 smithi154 conmon[34553]: audit 2021-07-31T01:40:54.418583+0000 mon.a (mon.0) 169 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:40:56.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:56 smithi154 conmon[34553]: audit 2021-07-31T01:40:55.330118+0000 mon.a (mon.0) 170 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-07-31T01:40:57.420 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[33655]: cluster 2021-07-31T01: 2021-07-31T01:40:57.421 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[33655]: 40:51.330878+0000 mon.b (mon.2) 1 : cluster [INF] mon.b calling monitor election 2021-07-31T01:40:57.421 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[33655]: cephadm 2021-07-31T01:40:56.294048+0000 mgr.y (mgr.14146) 47 : cephadm [INF] Updating smithi045:/etc/ceph/ceph.conf 2021-07-31T01:40:57.421 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[33655]: cephadm 2021-07-31T01:40:56.325440+0000 mgr.y (mgr.14146) 48 : cephadm [INF] Reconfiguring mon.a (unknown last config time)... 2021-07-31T01:40:57.422 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[33655]: cephadm 2021-07-31T01:40:56.327428+0000 mgr.y (mgr.14146) 49 : cephadm [INF] Reconfiguring daemon mon.a on smithi045 2021-07-31T01:40:57.422 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[33655]: cluster 2021-07-31T01:40:56.348250+0000 mon.b (mon.2) 2 : cluster [INF] mon.b calling monitor election 2021-07-31T01:40:57.422 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[33655]: cluster 2021-07-31T01:40:56.348836+0000 mon.c (mon.1) 3 : cluster [INF] mon.c calling monitor election 2021-07-31T01:40:57.422 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[33655]: cluster 2021-07-31T01:40:56.348949+0000 mon.a 2021-07-31T01:40:57.422 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[33655]: (mon.0) 181 : cluster [INF] mon.a calling monitor election 2021-07-31T01:40:57.423 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[33655]: cluster 2021-07-31T01:40:56 2021-07-31T01:40:57.423 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[33655]: .361817+0000 mon.a (mon.0) 182 : 2021-07-31T01:40:57.423 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[33655]: cluster [INF] mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2021-07-31T01:40:57.423 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[33655]: cluster 2021- 2021-07-31T01:40:57.424 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[33655]: 07-31T01:40:56. 2021-07-31T01:40:57.424 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[33655]: 371423+0000 mon.a (mon. 2021-07-31T01:40:57.424 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[33655]: 0) 183 : cluster [DBG] 2021-07-31T01:40:57.424 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[33655]: monmap e3: 3 mons at {a=[v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0],b=[v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0],c=[v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0]} 2021-07-31T01:40:57.424 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[33655]: cluster 2021-07-31T01:40:57.425 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[33655]: 2021-07-31T01 2021-07-31T01:40:57.425 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[33655]: :40:56.371471+0000 mon.a (mon 2021-07-31T01:40:57.425 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[33655]: .0) 184 : cluster [DBG] fsmap 2021-07-31T01:40:57.425 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[33655]: 2021-07-31T01:40:57.425 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[33655]: cluster 2021-07-31 2021-07-31T01:40:57.426 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[33655]: T01:40:56.371497 2021-07-31T01:40:57.426 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[33655]: +0000 mon.a (mon.0) 185 : 2021-07-31T01:40:57.426 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[33655]: cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2021-07-31T01:40:57.426 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[33655]: cluster 2021-07 2021-07-31T01:40:57.426 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[33655]: -31T01:40:56. 2021-07-31T01:40:57.427 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[33655]: 371649+0000 mon.a (mon. 2021-07-31T01:40:57.427 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[33655]: 0) 186 : cluster [DBG] mgrmap e13: y(active, since 77s) 2021-07-31T01:40:57.427 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[33655]: 2021-07-31T01:40:57.427 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[33655]: cluster 2021-07-31T01: 2021-07-31T01:40:57.428 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[33655]: 40:56.371872+0000 mon.a 2021-07-31T01:40:57.428 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[33655]: (mon.0) 187 : cluster 2021-07-31T01:40:57.428 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[33655]: [INF] Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2021-07-31T01:40:57.428 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[33655]: cluster 2021-07 2021-07-31T01:40:57.428 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[33655]: -31T01:40:56.371890+0000 2021-07-31T01:40:57.429 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[33655]: mon.a (mon.0) 188 2021-07-31T01:40:57.429 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[33655]: : cluster [INF] Cluster is now healthy 2021-07-31T01:40:57.429 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[33655]: cluster 2021-07-31T01:40:57.429 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[33655]: 2021-07-31T01:40:56. 2021-07-31T01:40:57.429 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[33655]: 377614+0000 mon.a (mon. 2021-07-31T01:40:57.430 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[33655]: 0) 189 : cluster [INF] 2021-07-31T01:40:57.430 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[33655]: overall HEALTH_OK 2021-07-31T01:40:57.430 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[39718]: cluster 2021-07-31T01:40:51. 2021-07-31T01:40:57.430 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[39718]: 330878+0000 mon.b (mon.2) 1 : cluster [INF] mon.b calling monitor election 2021-07-31T01:40:57.431 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[39718]: cephadm 2021-07-31T01:40:56. 2021-07-31T01:40:57.431 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[39718]: 294048+0000 mgr.y (mgr.14146) 47 : 2021-07-31T01:40:57.431 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[39718]: cephadm [INF] Updating smithi045:/etc/ceph/ceph.conf 2021-07-31T01:40:57.431 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[39718]: cephadm 2021 2021-07-31T01:40:57.432 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[39718]: -07-31T01:40:56.325440 2021-07-31T01:40:57.432 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[39718]: +0000 mgr.y (mgr.14146) 48 2021-07-31T01:40:57.432 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[39718]: : cephadm [INF] Reconfiguring mon.a (unknown last config time)... 2021-07-31T01:40:57.432 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[39718]: cephadm 2021-07- 2021-07-31T01:40:57.432 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[39718]: 31T01:40:56.327428+0000 2021-07-31T01:40:57.433 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[39718]: mgr.y (mgr.14146) 49 : 2021-07-31T01:40:57.433 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[39718]: cephadm [INF] Reconfiguring daemon mon.a on smithi045 2021-07-31T01:40:57.433 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[39718]: cluster 2021-07-31T01: 2021-07-31T01:40:57.433 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[39718]: 40:56.348250+0000 mon.b ( 2021-07-31T01:40:57.433 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[39718]: mon.2) 2 : cluster [INF] mon.b calling monitor election 2021-07-31T01:40:57.434 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[39718]: cluster 2021- 2021-07-31T01:40:57.434 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[39718]: 07-31T01:40:56.348836+0000 2021-07-31T01:40:57.434 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[39718]: mon.c (mon.1) 3 : cluster 2021-07-31T01:40:57.434 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[39718]: [INF] mon.c calling monitor election 2021-07-31T01:40:57.434 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[39718]: cluster 2021 2021-07-31T01:40:57.435 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[39718]: -07-31T01:40:56.348949+0000 2021-07-31T01:40:57.435 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[39718]: mon.a (mon.0) 181 : cluster [INF] 2021-07-31T01:40:57.435 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[39718]: mon.a calling monitor election 2021-07-31T01:40:57.435 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[39718]: cluster 2021-07-31T01:40:57.435 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[39718]: 2021-07-31T 2021-07-31T01:40:57.436 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[39718]: 01:40:56.361817+0000 mon.a (mon. 2021-07-31T01:40:57.436 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[39718]: 0) 182 : cluster [INF] mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2021-07-31T01:40:57.436 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[39718]: cluster 2021-07-31T01:40: 2021-07-31T01:40:57.436 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[39718]: 56.371423+0000 mon.a (mon.0 2021-07-31T01:40:57.437 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[39718]: ) 183 : cluster [DBG] monmap e3: 3 mons at {a=[v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0],b=[v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0],c=[v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0]} 2021-07-31T01:40:57.437 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[39718]: 2021-07-31T01:40:57.437 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[39718]: cluster 2021-07-31T01:40:57.437 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[39718]: 2021-07-31T01:40:56.371471+0000 2021-07-31T01:40:57.437 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[39718]: mon.a (mon.0) 184 : cluster 2021-07-31T01:40:57.438 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[39718]: [DBG] fsmap 2021-07-31T01:40:57.438 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[39718]: cluster 2021-07-31T01: 2021-07-31T01:40:57.438 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[39718]: 40:56.371497+0000 mon.a (mon.0) 185 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2021-07-31T01:40:57.438 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[39718]: cluster 2021-07- 2021-07-31T01:40:57.438 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[39718]: 31T01:40:56.371649+0000 mon.a ( 2021-07-31T01:40:57.439 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[39718]: mon.0) 186 : cluster [DBG] mgrmap e13: y(active, since 77s) 2021-07-31T01:40:57.439 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[39718]: cluster 2021 2021-07-31T01:40:57.439 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[39718]: -07-31T01:40:56.371872+0000 2021-07-31T01:40:57.439 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[39718]: mon.a (mon.0) 187 : cluster [INF] 2021-07-31T01:40:57.439 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[39718]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2021-07-31T01:40:57.440 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[39718]: cluster 2021-07-31T01: 2021-07-31T01:40:57.440 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[39718]: 40:56.371890+0000 mon.a (mon.0 2021-07-31T01:40:57.440 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[39718]: ) 188 : cluster [INF] Cluster is now healthy 2021-07-31T01:40:57.440 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[39718]: cluster 2021-07-31T01:40:57.440 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[39718]: 2021-07-31 2021-07-31T01:40:57.441 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[39718]: T01:40:56.377614+0000 mon.a ( 2021-07-31T01:40:57.441 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[39718]: mon.0) 189 : cluster [INF] overall HEALTH_OK 2021-07-31T01:40:57.441 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:57 smithi045 conmon[39718]: 2021-07-31T01:40:57.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:57 smithi154 conmon[34553]: cluster 2021-07-31T01:40:51.330878+0000 mon.b (mon.2) 1 : cluster [INF] mon.b calling monitor election 2021-07-31T01:40:57.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:57 smithi154 conmon[34553]: cephadm 2021-07-31T01:40:56.294048+0000 mgr.y (mgr.14146) 47 : cephadm [INF] Updating smithi045:/etc/ceph/ceph.conf 2021-07-31T01:40:57.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:57 smithi154 conmon[34553]: cephadm 2021-07-31T01 2021-07-31T01:40:57.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:57 smithi154 conmon[34553]: :40:56.325440+0000 mgr.y (mgr. 2021-07-31T01:40:57.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:57 smithi154 conmon[34553]: 14146) 48 : cephadm [INF] Reconfiguring mon.a (unknown last config time)... 2021-07-31T01:40:57.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:57 smithi154 conmon[34553]: cephadm 2021-07-31T01:40:57.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:57 smithi154 conmon[34553]: 2021-07-31T01:40 2021-07-31T01:40:57.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:57 smithi154 conmon[34553]: :56.327428+0000 mgr.y (mgr.14146 2021-07-31T01:40:57.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:57 smithi154 conmon[34553]: ) 49 : cephadm [INF] Reconfiguring daemon mon.a on smithi045 2021-07-31T01:40:57.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:57 smithi154 conmon[34553]: cluster 2021-07-31T01:40:57.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:57 smithi154 conmon[34553]: 2021-07-31T01:40 2021-07-31T01:40:57.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:57 smithi154 conmon[34553]: :56.348250+0000 mon.b (mon 2021-07-31T01:40:57.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:57 smithi154 conmon[34553]: .2) 2 : cluster [INF] mon.b calling monitor election 2021-07-31T01:40:57.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:57 smithi154 conmon[34553]: 2021-07-31T01:40:57.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:57 smithi154 conmon[34553]: cluster 2021-07-31T 2021-07-31T01:40:57.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:57 smithi154 conmon[34553]: 01:40:56.348836+0000 2021-07-31T01:40:57.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:57 smithi154 conmon[34553]: mon.c (mon.1) 3 : cluster 2021-07-31T01:40:57.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:57 smithi154 conmon[34553]: [INF] mon.c calling monitor election 2021-07-31T01:40:57.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:57 smithi154 conmon[34553]: cluster 2021-07-31T01:40:57.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:57 smithi154 conmon[34553]: 2021-07-31T01: 2021-07-31T01:40:57.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:57 smithi154 conmon[34553]: 40:56.348949+0000 mon.a (mon.0) 2021-07-31T01:40:57.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:57 smithi154 conmon[34553]: 181 : cluster [INF] mon.a calling monitor election 2021-07-31T01:40:57.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:57 smithi154 conmon[34553]: cluster 2021 2021-07-31T01:40:57.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:57 smithi154 conmon[34553]: -07-31T01:40: 2021-07-31T01:40:57.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:57 smithi154 conmon[34553]: 56.361817+0000 mon.a (mon.0) 182 2021-07-31T01:40:57.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:57 smithi154 conmon[34553]: : cluster [INF] mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2021-07-31T01:40:57.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:57 smithi154 conmon[34553]: cluster 2021 2021-07-31T01:40:57.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:57 smithi154 conmon[34553]: -07-31T01:40: 2021-07-31T01:40:57.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:57 smithi154 conmon[34553]: 56.371423+0000 mon.a (mon.0 2021-07-31T01:40:57.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:57 smithi154 conmon[34553]: ) 183 : cluster [DBG] monmap e3: 3 mons at {a=[v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0],b=[v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0],c=[v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0]} 2021-07-31T01:40:57.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:57 smithi154 conmon[34553]: cluster 2021-07 2021-07-31T01:40:57.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:57 smithi154 conmon[34553]: -31T01:40:56.371471+0000 mon.a (mon.0 2021-07-31T01:40:57.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:57 smithi154 conmon[34553]: ) 184 : cluster [DBG] fsmap 2021-07-31T01:40:57.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:57 smithi154 conmon[34553]: cluster 2021-07-31T01:40 2021-07-31T01:40:57.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:57 smithi154 conmon[34553]: :56.371497+0000 mon.a (mon.0) 185 2021-07-31T01:40:57.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:57 smithi154 conmon[34553]: : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2021-07-31T01:40:57.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:57 smithi154 conmon[34553]: cluster 2021-07-31T01:40:57.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:57 smithi154 conmon[34553]: 2021-07-31T01:40: 2021-07-31T01:40:57.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:57 smithi154 conmon[34553]: 56.371649+0000 mon.a (mon.0) 186 : 2021-07-31T01:40:57.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:57 smithi154 conmon[34553]: cluster [DBG] mgrmap e13: y(active, since 77s) 2021-07-31T01:40:57.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:57 smithi154 conmon[34553]: cluster 2021-07-31T01:40:57.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:57 smithi154 conmon[34553]: 2021-07-31T01:40:56. 2021-07-31T01:40:57.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:57 smithi154 conmon[34553]: 371872+0000 mon.a (mon.0) 187 : cluster 2021-07-31T01:40:57.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:57 smithi154 conmon[34553]: [INF] Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2021-07-31T01:40:57.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:57 smithi154 conmon[34553]: cluster 2021-07-31T01:40:57.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:57 smithi154 conmon[34553]: 2021-07-31T01:40 2021-07-31T01:40:57.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:57 smithi154 conmon[34553]: :56.371890+0000 mon.a (mon.0) 2021-07-31T01:40:57.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:57 smithi154 conmon[34553]: 188 : cluster [INF] Cluster is now healthy 2021-07-31T01:40:57.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:57 smithi154 conmon[34553]: cluster 2021 2021-07-31T01:40:57.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:57 smithi154 conmon[34553]: -07-31T01:40: 2021-07-31T01:40:57.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:57 smithi154 conmon[34553]: 56.377614+0000 mon.a (mon.0) 189 : 2021-07-31T01:40:57.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:57 smithi154 conmon[34553]: cluster [INF] overall HEALTH_OK 2021-07-31T01:40:58.333 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:58 smithi045 conmon[33655]: cluster 2021-07-31T01:40:57.304380+0000 mgr.y (mgr. 2021-07-31T01:40:58.334 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:58 smithi045 conmon[33655]: 14146) 50 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-07-31T01:40:58.334 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:58 smithi045 conmon[33655]: audit 2021-07-31T01:40:57.354128+0000 mon.a (mon.0) 190 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-07-31T01:40:58.335 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:58 smithi045 conmon[33655]: audit 2021-07-31T01:40:58.127129+0000 mon.a (mon.0) 191 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:40:58.335 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:58 smithi045 conmon[33655]: audit 2021-07-31T01:40:58.127849+0000 mon.a (mon.0) 192 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2021-07-31T01:40:58.336 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:58 smithi045 conmon[33655]: audit 2021-07-31T01:40:58.128466+0000 mon.a (mon.0) 193 : audit [DBG] 2021-07-31T01:40:58.336 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:58 smithi045 conmon[33655]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2021-07-31T01:40:58.337 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:58 smithi045 conmon[33655]: audit 2021-07-31T01:40:58.129078+0000 mon.a (mon.0) 2021-07-31T01:40:58.337 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:58 smithi045 conmon[33655]: 194 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:40:58.338 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:58 smithi045 conmon[39718]: cluster 2021-07-31T01:40: 2021-07-31T01:40:58.339 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:58 smithi045 conmon[39718]: 57.304380+0000 mgr.y (mgr.14146) 50 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-07-31T01:40:58.339 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:58 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T01:40:58.339 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:58 smithi045 conmon[39718]: 01:40:57.354128+0000 mon.a (mon.0 2021-07-31T01:40:58.340 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:58 smithi045 conmon[39718]: ) 190 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-07-31T01:40:58.340 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:58 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T01:40:58.341 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:58 smithi045 conmon[39718]: T01:40:58.127129+0000 mon.a (mon.0) 191 : 2021-07-31T01:40:58.341 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:58 smithi045 conmon[39718]: audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:40:58.342 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:58 smithi045 conmon[39718]: audit 2021-07-31T01:40:58.342 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:58 smithi045 conmon[39718]: 2021-07-31T01:40:58.127849+0000 mon.a (mon 2021-07-31T01:40:58.342 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:58 smithi045 conmon[39718]: .0) 192 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2021-07-31T01:40:58.343 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:58 smithi045 conmon[39718]: audit 2021-07-31T01 2021-07-31T01:40:58.343 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:58 smithi045 conmon[39718]: :40:58.128466+0000 mon.a (mon.0) 193 : audit 2021-07-31T01:40:58.343 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:58 smithi045 conmon[39718]: [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2021-07-31T01:40:58.344 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:58 smithi045 conmon[39718]: audit 2021- 2021-07-31T01:40:58.344 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:58 smithi045 conmon[39718]: 07-31T01:40:58.129078+0000 mon.a (mon.0) 194 2021-07-31T01:40:58.344 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:58 smithi045 conmon[39718]: : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:40:58.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:58 smithi154 conmon[34553]: cluster 2021-07-31T01:40:57.304380+0000 mgr.y (mgr.14146) 50 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-07-31T01:40:58.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:58 smithi154 conmon[34553]: audit 2021-07-31T01:40:58.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:58 smithi154 conmon[34553]: 2021-07-31T01:40:57.354128+0000 mon.a (mon.0) 190 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-07-31T01:40:58.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:58 smithi154 conmon[34553]: audit 2021-07-31T01 2021-07-31T01:40:58.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:58 smithi154 conmon[34553]: :40:58.127129+0000 mon.a (mon.0) 191 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:40:58.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:58 smithi154 conmon[34553]: 2021-07-31T01:40:58.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:58 smithi154 conmon[34553]: audit 2021-07- 2021-07-31T01:40:58.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:58 smithi154 conmon[34553]: 31T01:40:58.127849+0000 2021-07-31T01:40:58.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:58 smithi154 conmon[34553]: mon.a (mon.0) 192 : audit [INF] 2021-07-31T01:40:58.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:58 smithi154 conmon[34553]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2021-07-31T01:40:58.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:58 smithi154 conmon[34553]: audit 2021 2021-07-31T01:40:58.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:58 smithi154 conmon[34553]: -07-31T01:40:58 2021-07-31T01:40:58.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:58 smithi154 conmon[34553]: .128466+0000 mon.a (mon. 2021-07-31T01:40:58.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:58 smithi154 conmon[34553]: 0) 193 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2021-07-31T01:40:58.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:58 smithi154 conmon[34553]: audit 2021 2021-07-31T01:40:58.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:58 smithi154 conmon[34553]: -07-31T01:40:58 2021-07-31T01:40:58.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:58 smithi154 conmon[34553]: .129078+0000 mon.a (mon.0) 194 : 2021-07-31T01:40:58.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:58 smithi154 conmon[34553]: audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:40:59.246 INFO:teuthology.orchestra.run.smithi045.stdout:# minimal ceph.conf for f52d2c14-f19f-11eb-8c24-001a4aab830c 2021-07-31T01:40:59.247 INFO:teuthology.orchestra.run.smithi045.stdout:[global] 2021-07-31T01:40:59.247 INFO:teuthology.orchestra.run.smithi045.stdout: fsid = f52d2c14-f19f-11eb-8c24-001a4aab830c 2021-07-31T01:40:59.247 INFO:teuthology.orchestra.run.smithi045.stdout: mon_host = [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] 2021-07-31T01:40:59.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:59 smithi045 conmon[33655]: cephadm 2021-07-31T01:40:58.127630+0000 mgr.y (mgr.14146) 51 : cephadm [INF] Reconfiguring mon.c (monmap changed)... 2021-07-31T01:40:59.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:59 smithi045 conmon[33655]: cephadm 2021-07-31T01:40:58.129605+0000 mgr.y (mgr.14146) 52 : cephadm [INF] Reconfiguring daemon mon.c on smithi045 2021-07-31T01:40:59.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:40:59 smithi045 conmon[33655]: audit 2021-07-31T01:40:59.246420+0000 mon.c (mon.1) 4 : audit [DBG] from='client.? 172.21.15.45:0/1173843885' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:40:59.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:59 smithi045 conmon[39718]: cephadm 2021-07-31T01:40:58.127630+0000 mgr.y (mgr.14146) 51 : cephadm [INF] Reconfiguring mon.c (monmap changed)... 2021-07-31T01:40:59.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:59 smithi045 conmon[39718]: cephadm 2021-07-31T01:40:58.129605+0000 mgr.y (mgr.14146) 52 : cephadm [INF] Reconfiguring daemon mon.c on smithi045 2021-07-31T01:40:59.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:59 smithi045 conmon[39718]: audit 2021-07-31T01:40:59.246420+0000 mon.c ( 2021-07-31T01:40:59.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:40:59 smithi045 conmon[39718]: mon.1) 4 : audit [DBG] from='client.? 172.21.15.45:0/1173843885' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:40:59.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:59 smithi154 conmon[34553]: cephadm 2021-07-31T01:40:58.127630+0000 mgr.y (mgr.14146) 51 : cephadm [INF] 2021-07-31T01:40:59.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:59 smithi154 conmon[34553]: Reconfiguring mon.c (monmap changed)... 2021-07-31T01:40:59.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:59 smithi154 conmon[34553]: cephadm 2021-07-31T01:40:58.129605 2021-07-31T01:40:59.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:59 smithi154 conmon[34553]: +0000 mgr.y (mgr.14146) 52 : cephadm [INF] Reconfiguring daemon mon.c on smithi045 2021-07-31T01:40:59.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:59 smithi154 conmon[34553]: audit 2021-07-31 2021-07-31T01:40:59.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:59 smithi154 conmon[34553]: T01:40:59.246420+0000 mon.c (mon 2021-07-31T01:40:59.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:40:59 smithi154 conmon[34553]: .1) 4 : audit [DBG] from='client.? 172.21.15.45:0/1173843885' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:41:00.318 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2021-07-31T01:41:00.318 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2021-07-31T01:41:00.319 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/etc/ceph/ceph.conf 2021-07-31T01:41:00.349 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2021-07-31T01:41:00.349 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2021-07-31T01:41:00.415 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2021-07-31T01:41:00.416 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/etc/ceph/ceph.conf 2021-07-31T01:41:00.444 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2021-07-31T01:41:00.444 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2021-07-31T01:41:00.508 INFO:tasks.cephadm:Adding mgr.y on smithi045 2021-07-31T01:41:00.508 INFO:tasks.cephadm:Adding mgr.x on smithi154 2021-07-31T01:41:00.509 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f52d2c14-f19f-11eb-8c24-001a4aab830c -- ceph orch apply mgr '2;smithi045=y;smithi154=x' 2021-07-31T01:41:00.588 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:00 smithi154 conmon[34553]: cluster 2021-07-31T01:40:59.304614+0000 mgr.y (mgr.14146) 53 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-07-31T01:41:00.588 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:00 smithi154 conmon[34553]: audit 2021-07 2021-07-31T01:41:00.589 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:00 smithi154 conmon[34553]: -31T01:41:00.061267+0000 mon.a (mon.0) 195 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:41:00.589 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:00 smithi154 conmon[34553]: audit 2021-07-31T01:41:00.061970+0000 mon.a (mon.0) 196 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2021-07-31T01:41:00.589 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:00 smithi154 conmon[34553]: audit 2021-07-31T01:41:00.062602+0000 mon.a (mon.0) 197 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2021-07-31T01:41:00.589 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:00 smithi154 conmon[34553]: audit 2021-07-31T01:41:00.590 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:00 smithi154 conmon[34553]: 2021-07-31T01:41:00.063252+0000 mon.a (mon.0) 198 : audit 2021-07-31T01:41:00.590 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:00 smithi154 conmon[34553]: [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:41:00.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:00 smithi045 conmon[33655]: cluster 2021-07-31T01:40:59.304614+0000 mgr.y ( 2021-07-31T01:41:00.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:00 smithi045 conmon[33655]: mgr.14146) 53 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-07-31T01:41:00.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:00 smithi045 conmon[33655]: audit 2021-07-31T01:41:00.061267+0000 mon.a (mon.0) 195 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:41:00.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:00 smithi045 conmon[33655]: audit 2021-07-31T01:41:00.061970+0000 mon.a (mon.0) 196 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2021-07-31T01:41:00.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:00 smithi045 conmon[33655]: audit 2021-07-31T01:41:00.062602+0000 mon.a (mon.0) 197 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2021-07-31T01:41:00.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:00 smithi045 conmon[33655]: audit 2021-07-31T01:41:00.063252+0000 mon.a (mon.0) 198 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:41:00.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:00 smithi045 conmon[39718]: cluster 2021-07-31T01:40:59.304614+0000 mgr.y ( 2021-07-31T01:41:00.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:00 smithi045 conmon[39718]: mgr.14146) 53 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-07-31T01:41:00.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:00 smithi045 conmon[39718]: audit 2021-07-31T01:41:00.061267+0000 mon.a (mon.0) 195 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:41:00.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:00 smithi045 conmon[39718]: audit 2021-07-31T01:41:00.061970+0000 mon.a (mon.0) 196 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2021-07-31T01:41:00.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:00 smithi045 conmon[39718]: audit 2021-07-31T01:41:00.062602+0000 mon.a (mon.0) 197 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2021-07-31T01:41:00.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:00 smithi045 conmon[39718]: audit 2021-07-31T01:41:00.063252+0000 mon.a (mon.0) 198 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:41:01.580 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:01 smithi154 conmon[34553]: cephadm 2021-07-31T01:41:00.061731+0000 mgr.y (mgr.14146) 54 : cephadm [INF] Reconfiguring mon.b (monmap changed)... 2021-07-31T01:41:01.580 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:01 smithi154 conmon[34553]: cephadm 2021-07-31T01:41:00.063812+0000 mgr.y (mgr.14146) 55 : cephadm [INF] Reconfiguring daemon mon.b on smithi154 2021-07-31T01:41:01.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:01 smithi045 conmon[33655]: cephadm 2021-07-31T01:41:00.061731+0000 mgr.y (mgr.14146) 54 : cephadm [INF] Reconfiguring mon.b (monmap changed)... 2021-07-31T01:41:01.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:01 smithi045 conmon[33655]: 2021-07-31T01:41:01.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:01 smithi045 conmon[33655]: cephadm 2021-07-31T01:41:00.063812+0000 mgr.y (mgr.14146) 55 : cephadm [INF] Reconfiguring daemon mon.b on smithi154 2021-07-31T01:41:01.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:01 smithi045 conmon[39718]: cephadm 2021-07-31T01:41:00.061731+0000 mgr.y (mgr. 2021-07-31T01:41:01.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:01 smithi045 conmon[39718]: 14146) 54 : cephadm [INF] Reconfiguring mon.b (monmap changed)... 2021-07-31T01:41:01.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:01 smithi045 conmon[39718]: cephadm 2021-07-31T01:41:01.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:01 smithi045 conmon[39718]: 2021-07-31T01:41:00.063812+0000 mgr.y (mgr.14146) 55 : cephadm [INF] Reconfiguring daemon mon.b on smithi154 2021-07-31T01:41:01.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:01 smithi045 conmon[39718]: 2021-07-31T01:41:03.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:02 smithi154 conmon[34553]: cluster 2021-07-31T01:41:01.304994+0000 mgr.y (mgr. 2021-07-31T01:41:03.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:02 smithi154 conmon[34553]: 14146) 56 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-07-31T01:41:03.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:02 smithi154 conmon[34553]: audit 2021-07-31T01:41:01.748170+0000 mon.a (mon.0) 199 : audit [INF] 2021-07-31T01:41:03.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:02 smithi154 conmon[34553]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:41:03.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:02 smithi154 conmon[34553]: audit 2021-07-31 2021-07-31T01:41:03.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:02 smithi154 conmon[34553]: T01:41:01.748995+0000 mon.a (mon.0) 200 : audit 2021-07-31T01:41:03.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:02 smithi154 conmon[34553]: [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:41:03.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:02 smithi154 conmon[34553]: audit 2021-07-31T01:41:01.750221 2021-07-31T01:41:03.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:02 smithi154 conmon[34553]: +0000 mon.a (mon.0) 201 : audit [DBG] 2021-07-31T01:41:03.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:02 smithi154 conmon[34553]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:41:03.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:02 smithi154 conmon[34553]: audit 2021-07-31T01:41 2021-07-31T01:41:03.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:02 smithi154 conmon[34553]: :01.751122+0000 mon.a (mon.0) 2021-07-31T01:41:03.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:02 smithi154 conmon[34553]: 202 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:41:03.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:02 smithi154 conmon[34553]: audit 2021-07 2021-07-31T01:41:03.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:02 smithi154 conmon[34553]: -31T01:41:01.755858+0000 2021-07-31T01:41:03.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:02 smithi154 conmon[34553]: mon.a (mon.0) 203 : audit [INF] 2021-07-31T01:41:03.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:02 smithi154 conmon[34553]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:41:03.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:02 smithi045 conmon[33655]: cluster 2021-07-31T01:41:01.304994+0000 mgr.y (mgr.14146) 56 : cluster 2021-07-31T01:41:03.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:02 smithi045 conmon[33655]: [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-07-31T01:41:03.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:02 smithi045 conmon[33655]: audit 2021-07-31T01:41:01.748170+0000 mon.a (mon.0) 199 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:41:03.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:02 smithi045 conmon[33655]: audit 2021-07-31T01:41:01.748995 2021-07-31T01:41:03.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:02 smithi045 conmon[33655]: +0000 mon.a (mon.0) 200 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:41:03.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:02 smithi045 conmon[33655]: audit 2021-07-31T 2021-07-31T01:41:03.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:02 smithi045 conmon[33655]: 01:41:01.750221+0000 mon.a (mon.0) 201 : 2021-07-31T01:41:03.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:02 smithi045 conmon[33655]: audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:41:03.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:02 smithi045 conmon[33655]: audit 2021-07-31T01:41: 2021-07-31T01:41:03.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:02 smithi045 conmon[33655]: 01.751122+0000 mon.a (mon.0) 202 : audit [INF] 2021-07-31T01:41:03.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:02 smithi045 conmon[33655]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:41:03.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:02 smithi045 conmon[33655]: audit 2021-07-31T01:41:01.755858 2021-07-31T01:41:03.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:02 smithi045 conmon[33655]: +0000 mon.a (mon.0) 203 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:41:03.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:02 smithi045 conmon[39718]: cluster 2021-07-31T01:41:01.304994+0000 mgr.y (mgr.14146) 2021-07-31T01:41:03.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:02 smithi045 conmon[39718]: 56 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-07-31T01:41:03.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:02 smithi045 conmon[39718]: audit 2021-07-31T01:41:01. 2021-07-31T01:41:03.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:02 smithi045 conmon[39718]: 748170+0000 mon.a (mon.0) 199 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:41:03.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:02 smithi045 conmon[39718]: audit 2021-07-31T01:41:03.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:02 smithi045 conmon[39718]: 2021-07-31T01:41:01.748995+0000 mon.a 2021-07-31T01:41:03.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:02 smithi045 conmon[39718]: (mon.0) 200 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:41:03.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:02 smithi045 conmon[39718]: audit 2021-07-31T01:41:03.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:02 smithi045 conmon[39718]: 2021-07-31T01:41:01.750221+0000 mon.a (mon 2021-07-31T01:41:03.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:02 smithi045 conmon[39718]: .0) 201 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:41:03.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:02 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T01:41:03.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:02 smithi045 conmon[39718]: T01:41:01.751122+0000 mon.a (mon.0) 202 : audit 2021-07-31T01:41:03.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:02 smithi045 conmon[39718]: [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:41:03.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:02 smithi045 conmon[39718]: audit 2021- 2021-07-31T01:41:03.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:02 smithi045 conmon[39718]: 07-31T01:41:01.755858+0000 mon.a ( 2021-07-31T01:41:03.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:02 smithi045 conmon[39718]: mon.0) 203 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:41:03.071 INFO:teuthology.orchestra.run.smithi154.stdout:Scheduled mgr update... 2021-07-31T01:41:03.726 DEBUG:teuthology.orchestra.run.smithi154:mgr.x> sudo journalctl -f -n 0 -u ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c@mgr.x.service 2021-07-31T01:41:03.770 INFO:tasks.cephadm:Deploying OSDs... 2021-07-31T01:41:03.770 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2021-07-31T01:41:03.771 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/scratch_devs of=/dev/stdout 2021-07-31T01:41:03.791 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2021-07-31T01:41:03.791 DEBUG:teuthology.orchestra.run.smithi045:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2021-07-31T01:41:03.799 INFO:journalctl@ceph.mgr.x.smithi154.stdout:-- Logs begin at Sat 2021-07-31 01:17:02 UTC. -- 2021-07-31T01:41:03.850 INFO:teuthology.orchestra.run.smithi045.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2021-07-31T01:41:03.851 INFO:teuthology.orchestra.run.smithi045.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-07-31T01:41:03.851 INFO:teuthology.orchestra.run.smithi045.stdout:Device: 6h/6d Inode: 108486 Links: 1 2021-07-31T01:41:03.852 INFO:teuthology.orchestra.run.smithi045.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-07-31T01:41:03.852 INFO:teuthology.orchestra.run.smithi045.stdout:Context: system_u:object_r:device_t:s0 2021-07-31T01:41:03.852 INFO:teuthology.orchestra.run.smithi045.stdout:Access: 2021-07-31 01:40:39.291973686 +0000 2021-07-31T01:41:03.853 INFO:teuthology.orchestra.run.smithi045.stdout:Modify: 2021-07-31 01:26:07.644893555 +0000 2021-07-31T01:41:03.853 INFO:teuthology.orchestra.run.smithi045.stdout:Change: 2021-07-31 01:26:07.644893555 +0000 2021-07-31T01:41:03.853 INFO:teuthology.orchestra.run.smithi045.stdout: Birth: - 2021-07-31T01:41:03.863 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2021-07-31T01:41:03.864 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2021-07-31T01:41:03.864 INFO:teuthology.orchestra.run.smithi045.stderr:512 bytes copied, 0.000127322 s, 4.0 MB/s 2021-07-31T01:41:03.867 DEBUG:teuthology.orchestra.run.smithi045:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2021-07-31T01:41:03.924 INFO:teuthology.orchestra.run.smithi045.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2021-07-31T01:41:03.925 INFO:teuthology.orchestra.run.smithi045.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-07-31T01:41:03.925 INFO:teuthology.orchestra.run.smithi045.stdout:Device: 6h/6d Inode: 117396 Links: 1 2021-07-31T01:41:03.925 INFO:teuthology.orchestra.run.smithi045.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-07-31T01:41:03.925 INFO:teuthology.orchestra.run.smithi045.stdout:Context: system_u:object_r:device_t:s0 2021-07-31T01:41:03.926 INFO:teuthology.orchestra.run.smithi045.stdout:Access: 2021-07-31 01:40:39.291973686 +0000 2021-07-31T01:41:03.926 INFO:teuthology.orchestra.run.smithi045.stdout:Modify: 2021-07-31 01:26:07.974888522 +0000 2021-07-31T01:41:03.926 INFO:teuthology.orchestra.run.smithi045.stdout:Change: 2021-07-31 01:26:07.974888522 +0000 2021-07-31T01:41:03.926 INFO:teuthology.orchestra.run.smithi045.stdout: Birth: - 2021-07-31T01:41:03.936 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2021-07-31T01:41:03.937 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2021-07-31T01:41:03.937 INFO:teuthology.orchestra.run.smithi045.stderr:512 bytes copied, 0.000115372 s, 4.4 MB/s 2021-07-31T01:41:03.939 DEBUG:teuthology.orchestra.run.smithi045:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2021-07-31T01:41:03.996 INFO:teuthology.orchestra.run.smithi045.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2021-07-31T01:41:03.997 INFO:teuthology.orchestra.run.smithi045.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-07-31T01:41:03.997 INFO:teuthology.orchestra.run.smithi045.stdout:Device: 6h/6d Inode: 115416 Links: 1 2021-07-31T01:41:03.997 INFO:teuthology.orchestra.run.smithi045.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-07-31T01:41:03.997 INFO:teuthology.orchestra.run.smithi045.stdout:Context: system_u:object_r:device_t:s0 2021-07-31T01:41:03.998 INFO:teuthology.orchestra.run.smithi045.stdout:Access: 2021-07-31 01:40:39.291973686 +0000 2021-07-31T01:41:03.998 INFO:teuthology.orchestra.run.smithi045.stdout:Modify: 2021-07-31 01:26:08.305883477 +0000 2021-07-31T01:41:03.998 INFO:teuthology.orchestra.run.smithi045.stdout:Change: 2021-07-31 01:26:08.305883477 +0000 2021-07-31T01:41:03.998 INFO:teuthology.orchestra.run.smithi045.stdout: Birth: - 2021-07-31T01:41:04.007 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2021-07-31T01:41:04.007 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2021-07-31T01:41:04.008 INFO:teuthology.orchestra.run.smithi045.stderr:512 bytes copied, 9.9357e-05 s, 5.2 MB/s 2021-07-31T01:41:04.011 DEBUG:teuthology.orchestra.run.smithi045:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2021-07-31T01:41:04.068 INFO:teuthology.orchestra.run.smithi045.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2021-07-31T01:41:04.069 INFO:teuthology.orchestra.run.smithi045.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-07-31T01:41:04.069 INFO:teuthology.orchestra.run.smithi045.stdout:Device: 6h/6d Inode: 115468 Links: 1 2021-07-31T01:41:04.069 INFO:teuthology.orchestra.run.smithi045.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-07-31T01:41:04.069 INFO:teuthology.orchestra.run.smithi045.stdout:Context: system_u:object_r:device_t:s0 2021-07-31T01:41:04.070 INFO:teuthology.orchestra.run.smithi045.stdout:Access: 2021-07-31 01:40:39.291973686 +0000 2021-07-31T01:41:04.070 INFO:teuthology.orchestra.run.smithi045.stdout:Modify: 2021-07-31 01:26:08.648878249 +0000 2021-07-31T01:41:04.070 INFO:teuthology.orchestra.run.smithi045.stdout:Change: 2021-07-31 01:26:08.648878249 +0000 2021-07-31T01:41:04.070 INFO:teuthology.orchestra.run.smithi045.stdout: Birth: - 2021-07-31T01:41:04.080 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2021-07-31T01:41:04.081 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2021-07-31T01:41:04.082 INFO:teuthology.orchestra.run.smithi045.stderr:512 bytes copied, 0.000109891 s, 4.7 MB/s 2021-07-31T01:41:04.084 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2021-07-31T01:41:04.085 DEBUG:teuthology.orchestra.run.smithi154:> dd if=/scratch_devs of=/dev/stdout 2021-07-31T01:41:04.100 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2021-07-31T01:41:04.101 DEBUG:teuthology.orchestra.run.smithi154:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2021-07-31T01:41:04.155 INFO:teuthology.orchestra.run.smithi154.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2021-07-31T01:41:04.155 INFO:teuthology.orchestra.run.smithi154.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-07-31T01:41:04.155 INFO:teuthology.orchestra.run.smithi154.stdout:Device: 6h/6d Inode: 115513 Links: 1 2021-07-31T01:41:04.155 INFO:teuthology.orchestra.run.smithi154.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-07-31T01:41:04.156 INFO:teuthology.orchestra.run.smithi154.stdout:Context: system_u:object_r:device_t:s0 2021-07-31T01:41:04.156 INFO:teuthology.orchestra.run.smithi154.stdout:Access: 2021-07-31 01:40:48.220831890 +0000 2021-07-31T01:41:04.156 INFO:teuthology.orchestra.run.smithi154.stdout:Modify: 2021-07-31 01:25:52.077734808 +0000 2021-07-31T01:41:04.157 INFO:teuthology.orchestra.run.smithi154.stdout:Change: 2021-07-31 01:25:52.077734808 +0000 2021-07-31T01:41:04.157 INFO:teuthology.orchestra.run.smithi154.stdout: Birth: - 2021-07-31T01:41:04.165 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records in 2021-07-31T01:41:04.166 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records out 2021-07-31T01:41:04.166 INFO:teuthology.orchestra.run.smithi154.stderr:512 bytes copied, 0.000240424 s, 2.1 MB/s 2021-07-31T01:41:04.168 DEBUG:teuthology.orchestra.run.smithi154:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2021-07-31T01:41:04.204 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:04 smithi154 conmon[34553]: audit 2021-07-31T01:41:03.066709+0000 mgr.y (mgr.14146) 57 : audit [DBG] from='client.24110 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi045=y;smithi154=x", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:41:04.204 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:04 smithi154 conmon[34553]: cephadm 2021-07-31T01:41:03.067478+0000 mgr.y (mgr.14146) 58 : cephadm [INF] Saving service mgr spec with placement smithi045=y;smithi154=x;count:2 2021-07-31T01:41:04.205 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:04 smithi154 conmon[34553]: audit 2021-07-31T01:41:03.070870+0000 mon.a (mon.0) 204 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:41:04.205 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:04 smithi154 conmon[34553]: audit 2021-07-31T01:41:03.071687+0000 mon.a (mon.0) 205 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:41:04.205 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:04 smithi154 conmon[34553]: audit 2021-07-31T01:41:03.072959+0000 mon.a (mon.0) 206 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:41:04.205 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:04 smithi154 conmon[34553]: audit 2021-07-31T01:41:03.074174+0000 mon.a (mon.0) 207 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:41:04.205 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:04 smithi154 conmon[34553]: audit 2021-07-31T01:41:03.078957+0000 mon.a (mon.0) 208 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:41:04.206 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:04 smithi154 conmon[34553]: audit 2021 2021-07-31T01:41:04.206 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:04 smithi154 conmon[34553]: -07-31T01:41:03.080275+0000 mon.a (mon.0) 209 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2021-07-31T01:41:04.206 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:04 smithi154 conmon[34553]: audit 2021-07-31T01:41:03.082825+0000 mon.a (mon.0) 210 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2021-07-31T01:41:04.207 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:04 smithi154 conmon[34553]: audit 2021-07-31T01:41:03.083707+0000 mon.a (mon.0) 211 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2021-07-31T01:41:04.207 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:04 smithi154 conmon[34553]: audit 2021-07-31T01:41:03.084680+0000 mon.a (mon.0) 212 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:41:04.207 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:04 smithi154 conmon[34553]: cephadm 2021-07-31T01:41:03.085523+0000 mgr.y (mgr.14146) 59 : cephadm [INF] Deploying daemon mgr.x on smithi154 2021-07-31T01:41:04.208 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:04 smithi154 conmon[34553]: cluster 2021-07-31T01:41:03.305328+0000 mgr.y (mgr.14146) 60 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-07-31T01:41:04.222 INFO:teuthology.orchestra.run.smithi154.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2021-07-31T01:41:04.222 INFO:teuthology.orchestra.run.smithi154.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-07-31T01:41:04.222 INFO:teuthology.orchestra.run.smithi154.stdout:Device: 6h/6d Inode: 112287 Links: 1 2021-07-31T01:41:04.223 INFO:teuthology.orchestra.run.smithi154.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-07-31T01:41:04.223 INFO:teuthology.orchestra.run.smithi154.stdout:Context: system_u:object_r:device_t:s0 2021-07-31T01:41:04.223 INFO:teuthology.orchestra.run.smithi154.stdout:Access: 2021-07-31 01:40:48.220831890 +0000 2021-07-31T01:41:04.223 INFO:teuthology.orchestra.run.smithi154.stdout:Modify: 2021-07-31 01:25:52.403726370 +0000 2021-07-31T01:41:04.223 INFO:teuthology.orchestra.run.smithi154.stdout:Change: 2021-07-31 01:25:52.403726370 +0000 2021-07-31T01:41:04.224 INFO:teuthology.orchestra.run.smithi154.stdout: Birth: - 2021-07-31T01:41:04.232 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records in 2021-07-31T01:41:04.233 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records out 2021-07-31T01:41:04.233 INFO:teuthology.orchestra.run.smithi154.stderr:512 bytes copied, 0.000272394 s, 1.9 MB/s 2021-07-31T01:41:04.237 DEBUG:teuthology.orchestra.run.smithi154:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2021-07-31T01:41:04.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:04 smithi045 conmon[33655]: audit 2021-07-31T01:41:03.066709+0000 2021-07-31T01:41:04.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:04 smithi045 conmon[33655]: mgr.y (mgr.14146) 57 : audit [DBG] from='client.24110 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi045=y;smithi154=x", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:41:04.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:04 smithi045 conmon[33655]: cephadm 2021-07-31T01:41:03.067478+0000 mgr.y (mgr.14146) 58 : cephadm [INF] Saving service mgr spec with placement smithi045=y;smithi154=x;count:2 2021-07-31T01:41:04.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:04 smithi045 conmon[33655]: audit 2021-07-31T01:41:03.070870+0000 mon.a (mon.0) 204 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:41:04.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:04 smithi045 conmon[33655]: 2021-07-31T01:41:04.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:04 smithi045 conmon[33655]: audit 2021-07-31T01:41 2021-07-31T01:41:04.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:04 smithi045 conmon[33655]: :03.071687+0000 mon.a (mon.0) 205 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:41:04.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:04 smithi045 conmon[33655]: audit 2021-07-31T01:41:03.072959+0000 mon.a ( 2021-07-31T01:41:04.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:04 smithi045 conmon[33655]: mon.0) 206 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:41:04.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:04 smithi045 conmon[33655]: audit 2021-07-31T01:41:04.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:04 smithi045 conmon[33655]: 2021-07-31T01:41:03.074174+0000 mon.a ( 2021-07-31T01:41:04.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:04 smithi045 conmon[33655]: mon.0) 207 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:41:04.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:04 smithi045 conmon[33655]: audit 2021-07- 2021-07-31T01:41:04.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:04 smithi045 conmon[33655]: 31T01:41:03.078957+0000 mon.a (mon.0) 208 2021-07-31T01:41:04.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:04 smithi045 conmon[33655]: : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:41:04.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:04 smithi045 conmon[33655]: audit 2021-07-31T01:41:04.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:04 smithi045 conmon[33655]: 2021-07-31T01:41:03.080275+0000 2021-07-31T01:41:04.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:04 smithi045 conmon[33655]: mon.a (mon.0) 209 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2021-07-31T01:41:04.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:04 smithi045 conmon[33655]: audit 2021-07-31T01:41:04.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:04 smithi045 conmon[33655]: 2021-07-31T01:41:03.082825+0000 2021-07-31T01:41:04.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:04 smithi045 conmon[33655]: mon.a (mon.0) 210 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2021-07-31T01:41:04.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:04 smithi045 conmon[33655]: audit 2021 2021-07-31T01:41:04.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:04 smithi045 conmon[33655]: -07-31T01:41:03.083707+0000 mon.a 2021-07-31T01:41:04.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:04 smithi045 conmon[33655]: (mon.0) 211 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2021-07-31T01:41:04.283 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:04 smithi045 conmon[33655]: audit 2021-07-31T 2021-07-31T01:41:04.283 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:04 smithi045 conmon[33655]: 01:41:03.084680+0000 mon.a (mon.0) 2021-07-31T01:41:04.283 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:04 smithi045 conmon[33655]: 212 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:41:04.283 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:04 smithi045 conmon[33655]: cephadm 2021-07-31T01:41:04.283 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:04 smithi045 conmon[33655]: 2021-07-31T01:41:03.085523+0000 mgr.y (mgr. 2021-07-31T01:41:04.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:04 smithi045 conmon[33655]: 14146) 59 : cephadm [INF] Deploying daemon mgr.x on smithi154 2021-07-31T01:41:04.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:04 smithi045 conmon[33655]: cluster 2021-07-31T01:41 2021-07-31T01:41:04.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:04 smithi045 conmon[33655]: :03.305328+0000 mgr.y (mgr.14146) 60 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-07-31T01:41:04.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:04 smithi045 conmon[39718]: audit 2021-07-31T01:41:03.066709 2021-07-31T01:41:04.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:04 smithi045 conmon[39718]: +0000 mgr.y (mgr.14146) 57 : audit [DBG] from='client.24110 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi045=y;smithi154=x", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:41:04.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:04 smithi045 conmon[39718]: cephadm 2021 2021-07-31T01:41:04.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:04 smithi045 conmon[39718]: -07-31T01:41:03.067478+0000 mgr.y (mgr.14146) 58 : 2021-07-31T01:41:04.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:04 smithi045 conmon[39718]: cephadm [INF] Saving service mgr spec with placement smithi045=y;smithi154=x;count:2 2021-07-31T01:41:04.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:04 smithi045 conmon[39718]: audit 2021-07-31T01:41:03 2021-07-31T01:41:04.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:04 smithi045 conmon[39718]: .070870+0000 mon.a (mon.0) 204 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:41:04.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:04 smithi045 conmon[39718]: 2021-07-31T01:41:04.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:04 smithi045 conmon[39718]: audit 2021-07-31T01 2021-07-31T01:41:04.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:04 smithi045 conmon[39718]: :41:03.071687+0000 mon.a (mon.0) 205 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:41:04.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:04 smithi045 conmon[39718]: audit 2021-07-31T01:41:04.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:04 smithi045 conmon[39718]: 2021-07-31 2021-07-31T01:41:04.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:04 smithi045 conmon[39718]: T01:41:03.072959+0000 mon.a 2021-07-31T01:41:04.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:04 smithi045 conmon[39718]: (mon.0) 206 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:41:04.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:04 smithi045 conmon[39718]: audit 2021 2021-07-31T01:41:04.289 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:04 smithi045 conmon[39718]: -07-31T01:41:03.074174+0000 mon.a (mon.0) 2021-07-31T01:41:04.289 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:04 smithi045 conmon[39718]: 207 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:41:04.289 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:04 smithi045 conmon[39718]: audit 2021-07-31T01:41:03.078957+0000 2021-07-31T01:41:04.289 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:04 smithi045 conmon[39718]: mon.a (mon.0) 208 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:41:04.290 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:04 smithi045 conmon[39718]: 2021-07-31T01:41:04.290 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:04 smithi045 conmon[39718]: audit 2021-07-31T01:41:04.290 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:04 smithi045 conmon[39718]: 2021-07-31T01: 2021-07-31T01:41:04.290 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:04 smithi045 conmon[39718]: 41:03.080275+0000 mon.a (mon.0) 2021-07-31T01:41:04.290 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:04 smithi045 conmon[39718]: 209 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2021-07-31T01:41:04.291 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:04 smithi045 conmon[39718]: audit 2021-07-31T01:41:04.291 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:04 smithi045 conmon[39718]: 2021-07-31T01:41:03.082825+0000 2021-07-31T01:41:04.291 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:04 smithi045 conmon[39718]: mon.a (mon.0) 210 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2021-07-31T01:41:04.291 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:04 smithi045 conmon[39718]: audit 2021-07 2021-07-31T01:41:04.292 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:04 smithi045 conmon[39718]: -31T01:41:03.083707+0000 2021-07-31T01:41:04.292 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:04 smithi045 conmon[39718]: mon.a (mon.0) 211 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2021-07-31T01:41:04.292 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:04 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T01:41:04.292 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:04 smithi045 conmon[39718]: 01:41:03.084680+0000 mon.a (mon. 2021-07-31T01:41:04.293 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:04 smithi045 conmon[39718]: 0) 212 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:41:04.293 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:04 smithi045 conmon[39718]: cephadm 2021-07-31T01:41 2021-07-31T01:41:04.293 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:04 smithi045 conmon[39718]: :03.085523+0000 mgr.y (mgr.14146) 59 : cephadm 2021-07-31T01:41:04.293 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:04 smithi045 conmon[39718]: [INF] Deploying daemon mgr.x on smithi154 2021-07-31T01:41:04.293 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:04 smithi045 conmon[39718]: cluster 2021- 2021-07-31T01:41:04.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:04 smithi045 conmon[39718]: 07-31T01:41:03.305328+0000 mgr.y (mgr.14146) 60 : 2021-07-31T01:41:04.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:04 smithi045 conmon[39718]: cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-07-31T01:41:04.308 INFO:teuthology.orchestra.run.smithi154.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2021-07-31T01:41:04.309 INFO:teuthology.orchestra.run.smithi154.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-07-31T01:41:04.309 INFO:teuthology.orchestra.run.smithi154.stdout:Device: 6h/6d Inode: 117240 Links: 1 2021-07-31T01:41:04.309 INFO:teuthology.orchestra.run.smithi154.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-07-31T01:41:04.309 INFO:teuthology.orchestra.run.smithi154.stdout:Context: system_u:object_r:device_t:s0 2021-07-31T01:41:04.310 INFO:teuthology.orchestra.run.smithi154.stdout:Access: 2021-07-31 01:40:48.220831890 +0000 2021-07-31T01:41:04.310 INFO:teuthology.orchestra.run.smithi154.stdout:Modify: 2021-07-31 01:25:52.714718321 +0000 2021-07-31T01:41:04.310 INFO:teuthology.orchestra.run.smithi154.stdout:Change: 2021-07-31 01:25:52.714718321 +0000 2021-07-31T01:41:04.310 INFO:teuthology.orchestra.run.smithi154.stdout: Birth: - 2021-07-31T01:41:04.319 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records in 2021-07-31T01:41:04.319 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records out 2021-07-31T01:41:04.319 INFO:teuthology.orchestra.run.smithi154.stderr:512 bytes copied, 0.000211525 s, 2.4 MB/s 2021-07-31T01:41:04.321 DEBUG:teuthology.orchestra.run.smithi154:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2021-07-31T01:41:04.376 INFO:teuthology.orchestra.run.smithi154.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2021-07-31T01:41:04.377 INFO:teuthology.orchestra.run.smithi154.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-07-31T01:41:04.377 INFO:teuthology.orchestra.run.smithi154.stdout:Device: 6h/6d Inode: 116253 Links: 1 2021-07-31T01:41:04.378 INFO:teuthology.orchestra.run.smithi154.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-07-31T01:41:04.378 INFO:teuthology.orchestra.run.smithi154.stdout:Context: system_u:object_r:device_t:s0 2021-07-31T01:41:04.378 INFO:teuthology.orchestra.run.smithi154.stdout:Access: 2021-07-31 01:40:48.221831862 +0000 2021-07-31T01:41:04.378 INFO:teuthology.orchestra.run.smithi154.stdout:Modify: 2021-07-31 01:25:53.036709987 +0000 2021-07-31T01:41:04.379 INFO:teuthology.orchestra.run.smithi154.stdout:Change: 2021-07-31 01:25:53.036709987 +0000 2021-07-31T01:41:04.379 INFO:teuthology.orchestra.run.smithi154.stdout: Birth: - 2021-07-31T01:41:04.400 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records in 2021-07-31T01:41:04.400 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records out 2021-07-31T01:41:04.401 INFO:teuthology.orchestra.run.smithi154.stderr:512 bytes copied, 0.000192076 s, 2.7 MB/s 2021-07-31T01:41:04.403 INFO:tasks.cephadm:Deploying osd.0 on smithi045 with /dev/vg_nvme/lv_4... 2021-07-31T01:41:04.403 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f52d2c14-f19f-11eb-8c24-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2021-07-31T01:41:05.398 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:41:05 smithi154 systemd[1]: Starting Ceph mgr.x for f52d2c14-f19f-11eb-8c24-001a4aab830c... 2021-07-31T01:41:05.399 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:41:05 smithi154 podman[35740]: 2021-07-31 01:41:05.235065264 +0000 UTC m=+0.030179111 image pull docker.io/ceph/daemon-base@sha256:0191c38131eb5e8b2e3cce0839223fa943f24bf2153c2767cb5bdb4a2973d559 2021-07-31T01:41:05.399 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:41:05 smithi154 podman[35740]: 2021-07-31 01:41:05.397576726 +0000 UTC m=+0.192690649 container create f9af406258e19b744cd21dcc3c4c155e8f0149130edfbf220700512ce2c5672d (image=docker.io/ceph/daemon-base@sha256:0191c38131eb5e8b2e3cce0839223fa943f24bf2153c2767cb5bdb4a2973d559, name=ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c-mgr.x, ceph=True, maintainer=Dimitri Savineau , CEPH_POINT_RELEASE=, RELEASE=master-0b3dfd0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Base Image, GIT_COMMIT=0b3dfd099f6a46b8c8fecd28a50330b63408461f, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20201204) 2021-07-31T01:41:06.003 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:41:05 smithi154 podman[35740]: 2021-07-31 01:41:05.745306126 +0000 UTC m=+0.540420062 container init f9af406258e19b744cd21dcc3c4c155e8f0149130edfbf220700512ce2c5672d (image=docker.io/ceph/daemon-base@sha256:0191c38131eb5e8b2e3cce0839223fa943f24bf2153c2767cb5bdb4a2973d559, name=ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c-mgr.x, org.label-schema.name=CentOS Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=0b3dfd099f6a46b8c8fecd28a50330b63408461f, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=master-0b3dfd0, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20201204, GIT_CLEAN=True, maintainer=Dimitri Savineau , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2021-07-31T01:41:06.003 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:41:05 smithi154 conmon[35780]: debug 2021-07-31T01:41:05.777+0000 7fb155d74700 1 -- 172.21.15.154:0/939747309 <== mon.2 v2:172.21.15.154:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55c85d2964e0 con 0x55c85d26d000 2021-07-31T01:41:06.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:41:05 smithi154 podman[35740]: 2021-07-31 01:41:05.786884938 +0000 UTC m=+0.581998814 container start f9af406258e19b744cd21dcc3c4c155e8f0149130edfbf220700512ce2c5672d (image=docker.io/ceph/daemon-base@sha256:0191c38131eb5e8b2e3cce0839223fa943f24bf2153c2767cb5bdb4a2973d559, name=ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c-mgr.x, org.label-schema.name=CentOS Base Image, org.label-schema.build-date=20201204, maintainer=Dimitri Savineau , GIT_COMMIT=0b3dfd099f6a46b8c8fecd28a50330b63408461f, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=master-0b3dfd0, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS) 2021-07-31T01:41:06.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:41:05 smithi154 bash[35667]: f9af406258e19b744cd21dcc3c4c155e8f0149130edfbf220700512ce2c5672d 2021-07-31T01:41:06.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:41:05 smithi154 systemd[1]: Started Ceph mgr.x for f52d2c14-f19f-11eb-8c24-001a4aab830c. 2021-07-31T01:41:06.679 INFO:teuthology.orchestra.run.smithi045.stderr:--> Zapping: /dev/vg_nvme/lv_4 2021-07-31T01:41:06.679 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2021-07-31T01:41:06.679 INFO:teuthology.orchestra.run.smithi045.stderr: stderr: 10+0 records in 2021-07-31T01:41:06.679 INFO:teuthology.orchestra.run.smithi045.stderr:10+0 records out 2021-07-31T01:41:06.680 INFO:teuthology.orchestra.run.smithi045.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0079304 s, 1.3 GB/s 2021-07-31T01:41:06.680 INFO:teuthology.orchestra.run.smithi045.stderr:--> Zapping successful for: 2021-07-31T01:41:07.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:06 smithi045 conmon[33655]: cluster 2021-07-31T01:41:05.305632+0000 mgr.y (mgr.14146) 61 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-07-31T01:41:07.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:06 smithi045 conmon[33655]: audit 2021-07-31T01:41:05.896230+0000 mon.a (mon.0) 213 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:41:07.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:06 smithi045 conmon[33655]: audit 2021-07-31T01:41:05.897618+0000 mon.a (mon.0) 214 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:41:07.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:06 smithi045 conmon[33655]: audit 2021-07-31T01:41:05.898606+0000 mon.a (mon.0) 215 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:41:07.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:06 smithi045 conmon[33655]: audit 2021-07-31T01:41:05.899271+0000 mon.a (mon.0) 216 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:41:07.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:06 smithi045 conmon[39718]: cluster 2021-07-31T01:41:05.305632+0000 mgr.y (mgr.14146) 61 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-07-31T01:41:07.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:06 smithi045 conmon[39718]: audit 2021-07-31T01:41:05.896230+0000 mon.a (mon.0) 213 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:41:07.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:06 smithi045 conmon[39718]: audit 2021-07-31T01:41:05.897618+0000 mon.a (mon.0) 214 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:41:07.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:06 smithi045 conmon[39718]: audit 2021-07-31T01:41:05.898606+0000 mon.a (mon.0) 215 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:41:07.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:06 smithi045 conmon[39718]: audit 2021-07-31T01:41:05.899271+0000 mon.a (mon.0) 216 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:41:07.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:06 smithi154 conmon[34553]: cluster 2021-07-31T01:41:05.305632+0000 mgr.y (mgr.14146 2021-07-31T01:41:07.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:06 smithi154 conmon[34553]: ) 61 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-07-31T01:41:07.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:06 smithi154 conmon[34553]: audit 2021-07-31T01:41:05.896230+0000 mon.a (mon.0) 213 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:41:07.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:06 smithi154 conmon[34553]: audit 2021-07-31T01:41:05.897618+0000 mon.a (mon.0) 214 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:41:07.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:06 smithi154 conmon[34553]: audit 2021-07-31T01:41:05.898606+0000 mon.a (mon.0) 215 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:41:07.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:06 smithi154 conmon[34553]: audit 2021-07-31T01:41:05.899271+0000 mon.a (mon.0) 216 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:41:07.432 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f52d2c14-f19f-11eb-8c24-001a4aab830c -- ceph orch daemon add osd smithi045:vg_nvme/lv_4 2021-07-31T01:41:08.254 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:08 smithi045 conmon[39718]: audit 2021-07-31T01:41:07.252403+0000 mon.a (mon.0) 217 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:41:08.255 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:08 smithi045 conmon[33655]: audit 2021-07-31T01:41:07.252403+0000 mon.a (mon.0) 217 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:41:08.255 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:08 smithi045 conmon[33655]: audit 2021-07-31T01:41:07.255220+0000 mon.a (mon.0) 218 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:41:08.255 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:08 smithi045 conmon[33655]: cephadm 2021-07-31T01:41:07.256619+0000 mgr.y (mgr.14146) 62 : cephadm [INF] Reconfiguring mgr.y (unknown last config time)... 2021-07-31T01:41:08.255 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:08 smithi045 conmon[33655]: audit 2021-07-31T01:41:07.256873+0000 mon.a (mon.0) 219 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2021-07-31T01:41:08.256 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:08 smithi045 conmon[33655]: audit 2021-07-31T01:41:07.257665+0000 mon.a (mon.0) 220 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2021-07-31T01:41:08.256 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:08 smithi045 conmon[33655]: audit 2021 2021-07-31T01:41:08.256 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:08 smithi045 conmon[33655]: -07-31T01:41:07.258376+0000 mon.a (mon.0) 221 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:41:08.256 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:08 smithi045 conmon[33655]: cephadm 2021-07-31T01:41:07.259226+0000 mgr.y (mgr.14146) 63 : cephadm [INF] Reconfiguring daemon mgr.y on smithi045 2021-07-31T01:41:08.256 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:08 smithi045 conmon[33655]: cluster 2021-07-31T01:41:07.305957+0000 mgr.y (mgr.14146) 64 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-07-31T01:41:08.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:08 smithi154 conmon[34553]: audit 2021-07-31T01:41:07.252403+0000 mon.a (mon.0) 217 2021-07-31T01:41:08.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:08 smithi154 conmon[34553]: : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:41:08.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:08 smithi154 conmon[34553]: audit 2021-07-31T01:41:07.255220+0000 mon.a (mon.0) 218 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:41:08.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:08 smithi154 conmon[34553]: cephadm 2021-07-31T01:41:07.256619 2021-07-31T01:41:08.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:08 smithi154 conmon[34553]: +0000 mgr.y (mgr.14146) 62 : cephadm [INF] 2021-07-31T01:41:08.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:08 smithi154 conmon[34553]: Reconfiguring mgr.y (unknown last config time)... 2021-07-31T01:41:08.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:08 smithi154 conmon[34553]: audit 2021-07-31 2021-07-31T01:41:08.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:08 smithi154 conmon[34553]: T01:41:07.256873+0000 mon.a (mon 2021-07-31T01:41:08.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:08 smithi154 conmon[34553]: .0) 219 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2021-07-31T01:41:08.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:08 smithi154 conmon[34553]: audit 2021-07-31T01:41:08.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:08 smithi154 conmon[34553]: 2021-07-31T01:41:07.257665 2021-07-31T01:41:08.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:08 smithi154 conmon[34553]: +0000 mon.a (mon.0) 220 : audit 2021-07-31T01:41:08.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:08 smithi154 conmon[34553]: [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2021-07-31T01:41:08.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:08 smithi154 conmon[34553]: audit 2021-07-31T 2021-07-31T01:41:08.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:08 smithi154 conmon[34553]: 01:41:07.258376+0000 mon.a (mon 2021-07-31T01:41:08.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:08 smithi154 conmon[34553]: .0) 221 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:41:08.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:08 smithi154 conmon[34553]: cephadm 2021-07-31T01:41:08.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:08 smithi154 conmon[34553]: 2021-07-31T01:41:07. 2021-07-31T01:41:08.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:08 smithi154 conmon[34553]: 259226+0000 mgr.y (mgr.14146) 63 : cephadm 2021-07-31T01:41:08.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:08 smithi154 conmon[34553]: [INF] Reconfiguring daemon mgr.y on smithi045 2021-07-31T01:41:08.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:08 smithi154 conmon[34553]: cluster 2021-07- 2021-07-31T01:41:08.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:08 smithi154 conmon[34553]: 31T01:41:07.305957+0000 mgr.y ( 2021-07-31T01:41:08.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:08 smithi154 conmon[34553]: mgr.14146) 64 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-07-31T01:41:08.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:08 smithi154 conmon[34553]: 2021-07-31T01:41:08.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:08 smithi045 conmon[39718]: audit 2021-07-31T01:41:07.255220+0000 mon.a (mon.0) 218 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:41:08.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:08 smithi045 conmon[39718]: cephadm 2021-07-31T01:41:07.256619+0000 mgr.y (mgr.14146) 62 : cephadm [INF] Reconfiguring mgr.y (unknown last config time)... 2021-07-31T01:41:08.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:08 smithi045 conmon[39718]: audit 2021-07-31T01:41:07.256873+0000 mon.a (mon.0) 219 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2021-07-31T01:41:08.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:08 smithi045 conmon[39718]: audit 2021-07-31T01:41:07.257665+0000 mon.a (mon.0) 2021-07-31T01:41:08.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:08 smithi045 conmon[39718]: 220 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2021-07-31T01:41:08.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:08 smithi045 conmon[39718]: audit 2021-07-31T01:41:07.258376+0000 mon.a (mon.0) 221 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:41:08.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:08 smithi045 conmon[39718]: cephadm 2021-07-31T01:41:07.259226+0000 mgr.y (mgr.14146) 63 : cephadm [INF] Reconfiguring daemon mgr.y on smithi045 2021-07-31T01:41:08.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:08 smithi045 conmon[39718]: cluster 2021-07-31T01:41:07.305957+0000 mgr.y (mgr.14146) 64 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-07-31T01:41:10.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:10 smithi154 conmon[34553]: audit 2021-07-31T01:41:09.053360+0000 mon.a (mon.0) 222 : 2021-07-31T01:41:10.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:10 smithi154 conmon[34553]: audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:41:10.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:10 smithi154 conmon[34553]: audit 2021-07-31T01:41:09.054074+0000 mon.a (mon.0) 223 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:41:10.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:10 smithi154 conmon[34553]: audit 2021-07-31T01:41:09.054994 2021-07-31T01:41:10.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:10 smithi154 conmon[34553]: +0000 mon.a (mon.0) 224 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:41:10.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:10 smithi154 conmon[34553]: 2021-07-31T01:41:10.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:10 smithi154 conmon[34553]: audit 2021-07-31T01:41:10.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:10 smithi154 conmon[34553]: 2021-07-31T01:41:09.055721+0000 mon.a (mon. 2021-07-31T01:41:10.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:10 smithi154 conmon[34553]: 0) 225 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:41:10.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:10 smithi154 conmon[34553]: audit 2021- 2021-07-31T01:41:10.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:10 smithi154 conmon[34553]: 07-31T01:41:09.059129+0000 mon.a 2021-07-31T01:41:10.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:10 smithi154 conmon[34553]: (mon.0) 226 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:41:10.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:10 smithi154 conmon[34553]: 2021-07-31T01:41:10.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:10 smithi154 conmon[34553]: cluster 2021-07-31T01:41 2021-07-31T01:41:10.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:10 smithi154 conmon[34553]: :09.306344+0000 mgr.y (mgr.14146) 2021-07-31T01:41:10.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:10 smithi154 conmon[34553]: 65 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-07-31T01:41:10.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:10 smithi154 conmon[34553]: audit 2021 2021-07-31T01:41:10.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:10 smithi154 conmon[34553]: -07-31T01:41:09.422871+0000 2021-07-31T01:41:10.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:10 smithi154 conmon[34553]: mon.a (mon.0) 227 : audit [INF] 2021-07-31T01:41:10.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:10 smithi154 conmon[34553]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:41:10.508 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:10 smithi045 conmon[33655]: audit 2021-07-31T01:41:09.053360+0000 2021-07-31T01:41:10.509 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:10 smithi045 conmon[33655]: mon.a (mon.0) 222 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:41:10.509 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:10 smithi045 conmon[33655]: audit 2021-07-31T01:41:09.054074+0000 mon.a (mon.0) 223 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:41:10.509 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:10 smithi045 conmon[33655]: audit 2021-07-31T01:41:09.054994+0000 mon.a (mon.0) 224 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:41:10.509 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:10 smithi045 conmon[33655]: audit 2021-07-31T01:41:09.055721+0000 mon.a (mon.0) 225 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:41:10.510 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:10 smithi045 conmon[33655]: audit 2021-07-31T01:41:09.059129+0000 mon.a (mon.0) 226 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:41:10.510 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:10 smithi045 conmon[33655]: cluster 2021-07-31T01:41:09.306344+0000 mgr.y (mgr.14146) 65 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-07-31T01:41:10.510 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:10 smithi045 conmon[33655]: audit 2021-07-31T01:41:09.422871+0000 mon.a (mon.0) 227 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:41:10.510 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:10 smithi045 conmon[39718]: audit 2021-07-31T01 2021-07-31T01:41:10.511 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:10 smithi045 conmon[39718]: :41:09.053360+0000 mon.a (mon.0) 222 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:41:10.511 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:10 smithi045 conmon[39718]: audit 2021-07-31T01:41:09.054074+0000 mon.a (mon.0) 223 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:41:10.511 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:10 smithi045 conmon[39718]: audit 2021-07-31T01:41:09.054994+0000 mon.a (mon.0) 224 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:41:10.512 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:10 smithi045 conmon[39718]: audit 2021-07-31T01:41:09.055721+0000 mon.a (mon.0) 225 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:41:10.512 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:10 smithi045 conmon[39718]: audit 2021-07-31T01:41:09.059129+0000 mon.a (mon.0) 226 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:41:10.512 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:10 smithi045 conmon[39718]: cluster 2021-07-31T01:41:09.306344+0000 mgr.y (mgr.14146) 2021-07-31T01:41:10.512 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:10 smithi045 conmon[39718]: 65 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-07-31T01:41:10.513 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:10 smithi045 conmon[39718]: audit 2021-07-31T01:41:09. 2021-07-31T01:41:10.513 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:10 smithi045 conmon[39718]: 422871+0000 mon.a (mon.0) 227 : audit 2021-07-31T01:41:10.513 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:10 smithi045 conmon[39718]: [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:41:11.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:11 smithi045 conmon[33655]: audit 2021-07-31T01:41:10.078553 2021-07-31T01:41:11.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:11 smithi045 conmon[33655]: +0000 mgr.y (mgr.14146) 66 : audit [DBG] from='client.24121 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:41:11.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:11 smithi045 conmon[33655]: audit 2021-07-31T01:41:10.079276+0000 mon.a (mon.0) 228 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-07-31T01:41:11.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:11 smithi045 conmon[33655]: audit 2021-07-31T01:41:10.080685+0000 mon.a (mon.0) 229 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-07-31T01:41:11.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:11 smithi045 conmon[33655]: audit 2021-07-31T01:41:10.081238+0000 mon.a (mon.0) 230 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:41:11.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:11 smithi045 conmon[33655]: cluster 2021-07-31T01:41:10.845557+0000 mon.a (mon.0) 231 : cluster [DBG] Standby manager daemon x started 2021-07-31T01:41:11.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:11 smithi045 conmon[33655]: 2021-07-31T01:41:11.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:11 smithi045 conmon[33655]: audit 2021-07-31T01:41:10.846567+0000 mon.b (mon.2) 3 : 2021-07-31T01:41:11.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:11 smithi045 conmon[33655]: audit [DBG] from='mgr.? 172.21.15.154:0/3003068096' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2021-07-31T01:41:11.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:11 smithi045 conmon[33655]: audit 2021-07-31T01:41: 2021-07-31T01:41:11.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:11 smithi045 conmon[33655]: 10.846870+0000 mon.b (mon.2) 4 : 2021-07-31T01:41:11.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:11 smithi045 conmon[33655]: audit [DBG] from='mgr.? 172.21.15.154:0/3003068096' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2021-07-31T01:41:11.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:11 smithi045 conmon[33655]: audit 2021-07 2021-07-31T01:41:11.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:11 smithi045 conmon[33655]: -31T01:41:10.848032+0000 mon.b 2021-07-31T01:41:11.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:11 smithi045 conmon[33655]: (mon.2) 5 : audit [DBG] from='mgr.? 172.21.15.154:0/3003068096' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2021-07-31T01:41:11.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:11 smithi045 conmon[33655]: audit 2021- 2021-07-31T01:41:11.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:11 smithi045 conmon[33655]: 07-31T01:41:10.848279+0000 mon.b 2021-07-31T01:41:11.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:11 smithi045 conmon[33655]: (mon.2) 6 : audit [DBG] from='mgr.? 172.21.15.154:0/3003068096' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2021-07-31T01:41:11.282 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:11 smithi045 conmon[39718]: audit 2021-07-31T01:41:10 2021-07-31T01:41:11.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:11 smithi045 conmon[39718]: .078553+0000 mgr.y (mgr.14146) 66 : audit [DBG] from='client.24121 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:41:11.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:11 smithi045 conmon[39718]: audit 2021-07-31T01:41:10.079276+0000 mon.a (mon.0) 228 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-07-31T01:41:11.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:11 smithi045 conmon[39718]: audit 2021-07-31T01:41:10.080685+0000 mon.a (mon.0) 229 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-07-31T01:41:11.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:11 smithi045 conmon[39718]: audit 2021-07-31T01:41:10.081238+0000 2021-07-31T01:41:11.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:11 smithi045 conmon[39718]: mon.a (mon.0) 230 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:41:11.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:11 smithi045 conmon[39718]: cluster 2021-07-31T01:41:10.845557+0000 mon.a (mon.0 2021-07-31T01:41:11.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:11 smithi045 conmon[39718]: ) 231 : cluster [DBG] Standby manager daemon x started 2021-07-31T01:41:11.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:11 smithi045 conmon[39718]: audit 2021-07-31T01 2021-07-31T01:41:11.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:11 smithi045 conmon[39718]: :41:10.846567+0000 mon.b (mon.2) 3 : audit [DBG] from='mgr.? 172.21.15.154:0/3003068096' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2021-07-31T01:41:11.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:11 smithi045 conmon[39718]: 2021-07-31T01:41:11.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:11 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T01:41:11.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:11 smithi045 conmon[39718]: 31T01:41:10.846870+0000 mon.b (mon.2) 4 : audit 2021-07-31T01:41:11.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:11 smithi045 conmon[39718]: [DBG] from='mgr.? 172.21.15.154:0/3003068096' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2021-07-31T01:41:11.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:11 smithi045 conmon[39718]: audit 2021-07-31T01:41:11.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:11 smithi045 conmon[39718]: 2021-07-31T01:41:10.848032+0000 mon.b (mon.2 2021-07-31T01:41:11.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:11 smithi045 conmon[39718]: ) 5 : audit [DBG] from='mgr.? 172.21.15.154:0/3003068096' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2021-07-31T01:41:11.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:11 smithi045 conmon[39718]: audit 2021-07-31T01:41:10 2021-07-31T01:41:11.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:11 smithi045 conmon[39718]: .848279+0000 mon.b (mon.2) 6 : audit [DBG] from='mgr.? 172.21.15.154:0/3003068096' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2021-07-31T01:41:11.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:11 smithi045 conmon[39718]: 2021-07-31T01:41:11.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:11 smithi154 conmon[34553]: audit 2021-07-31T01:41:10.078553+0000 mgr.y (mgr.14146) 66 2021-07-31T01:41:11.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:11 smithi154 conmon[34553]: : audit [DBG] from='client.24121 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:41:11.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:11 smithi154 conmon[34553]: audit 2021-07-31T01:41:10.079276+0000 mon.a (mon.0) 228 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-07-31T01:41:11.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:11 smithi154 conmon[34553]: audit 2021-07-31T01:41:10.080685+0000 2021-07-31T01:41:11.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:11 smithi154 conmon[34553]: mon.a (mon.0) 229 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-07-31T01:41:11.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:11 smithi154 conmon[34553]: 2021-07-31T01:41:11.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:11 smithi154 conmon[34553]: audit 2021-07-31T01:41: 2021-07-31T01:41:11.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:11 smithi154 conmon[34553]: 10.081238+0000 mon.a (mon.0) 230 : 2021-07-31T01:41:11.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:11 smithi154 conmon[34553]: audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:41:11.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:11 smithi154 conmon[34553]: cluster 2021-07 2021-07-31T01:41:11.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:11 smithi154 conmon[34553]: -31T01:41:10.845557+0000 mon.a ( 2021-07-31T01:41:11.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:11 smithi154 conmon[34553]: mon.0) 231 : cluster [DBG] Standby manager daemon x started 2021-07-31T01:41:11.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:11 smithi154 conmon[34553]: audit 2021-07-31T 2021-07-31T01:41:11.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:11 smithi154 conmon[34553]: 01:41:10.846567+0000 mon.b (mon.2) 3 : 2021-07-31T01:41:11.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:11 smithi154 conmon[34553]: audit [DBG] from='mgr.? 172.21.15.154:0/3003068096' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2021-07-31T01:41:11.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:11 smithi154 conmon[34553]: audit 2021-07 2021-07-31T01:41:11.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:11 smithi154 conmon[34553]: -31T01:41:10.846870+0000 mon.b 2021-07-31T01:41:11.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:11 smithi154 conmon[34553]: (mon.2) 4 : audit [DBG] from='mgr.? 172.21.15.154:0/3003068096' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2021-07-31T01:41:11.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:11 smithi154 conmon[34553]: audit 2021 2021-07-31T01:41:11.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:11 smithi154 conmon[34553]: -07-31T01:41:10.848032+0000 mon.b 2021-07-31T01:41:11.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:11 smithi154 conmon[34553]: (mon.2) 5 : audit [DBG] from='mgr.? 172.21.15.154:0/3003068096' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2021-07-31T01:41:11.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:11 smithi154 conmon[34553]: 2021-07-31T01:41:11.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:11 smithi154 conmon[34553]: audit 2021-07-31T01:41 2021-07-31T01:41:11.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:11 smithi154 conmon[34553]: :10.848279+0000 mon.b (mon.2) 2021-07-31T01:41:11.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:11 smithi154 conmon[34553]: 6 : audit [DBG] from='mgr.? 172.21.15.154:0/3003068096' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2021-07-31T01:41:12.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:12 smithi045 conmon[33655]: cluster 2021-07-31T01:41:11.066729+0000 mon.a (mon.0) 232 : cluster [DBG] mgrmap e14: y(active, since 91s), standbys: x 2021-07-31T01:41:12.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:12 smithi045 conmon[33655]: audit 2021-07-31T01:41:11.066858+0000 mon.a (mon.0) 233 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2021-07-31T01:41:12.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:12 smithi045 conmon[33655]: cluster 2021-07-31T01:41:11.306762+0000 mgr.y (mgr.14146) 67 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-07-31T01:41:12.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:12 smithi045 conmon[39718]: cluster 2021-07-31T01:41:11.066729+0000 mon.a (mon.0) 232 : cluster [DBG] mgrmap e14: y(active, since 91s), standbys: x 2021-07-31T01:41:12.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:12 smithi045 conmon[39718]: audit 2021-07-31T01:41:11.066858+0000 mon.a (mon.0) 233 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2021-07-31T01:41:12.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:12 smithi045 conmon[39718]: cluster 2021-07-31T01:41:11.306762+0000 mgr.y (mgr.14146) 67 : cluster [DBG] 2021-07-31T01:41:12.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:12 smithi045 conmon[39718]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-07-31T01:41:12.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:12 smithi154 conmon[34553]: cluster 2021-07-31T01:41:11.066729+0000 mon.a (mon.0) 232 : cluster [DBG] 2021-07-31T01:41:12.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:12 smithi154 conmon[34553]: mgrmap e14: y(active, since 91s), standbys: x 2021-07-31T01:41:12.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:12 smithi154 conmon[34553]: audit 2021-07-31T01:41:11.066858+0000 mon.a (mon.0) 233 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2021-07-31T01:41:12.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:12 smithi154 conmon[34553]: cluster 2021-07-31T01:41:11.306762+0000 mgr.y (mgr.14146) 2021-07-31T01:41:12.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:12 smithi154 conmon[34553]: 67 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-07-31T01:41:13.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:13 smithi154 conmon[34553]: audit 2021-07-31T01:41:12.643107+0000 mon.a ( 2021-07-31T01:41:13.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:13 smithi154 conmon[34553]: mon.0) 234 : audit [INF] from='client.? 172.21.15.45:0/2083264684' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "20724b9a-515c-4878-9e4f-301e21a5a985"}]: dispatch 2021-07-31T01:41:13.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:13 smithi154 conmon[34553]: audit 2021-07-31T01:41:12.646559+0000 2021-07-31T01:41:13.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:13 smithi154 conmon[34553]: mon.a (mon.0) 235 : audit [INF] from='client.? 172.21.15.45:0/2083264684' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "20724b9a-515c-4878-9e4f-301e21a5a985"}]': finished 2021-07-31T01:41:13.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:13 smithi154 conmon[34553]: cluster 2021-07-31T01:41:12.646613 2021-07-31T01:41:13.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:13 smithi154 conmon[34553]: +0000 mon.a (mon.0) 236 : cluster [DBG] 2021-07-31T01:41:13.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:13 smithi154 conmon[34553]: osdmap e5: 1 total, 0 up, 1 in 2021-07-31T01:41:13.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:13 smithi154 conmon[34553]: audit 2021-07-31 2021-07-31T01:41:13.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:13 smithi154 conmon[34553]: T01:41:12.646717+0000 mon.a (mon. 2021-07-31T01:41:13.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:13 smithi154 conmon[34553]: 0) 237 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-07-31T01:41:13.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:13 smithi045 conmon[33655]: audit 2021-07-31T01:41:12.643107+0000 mon.a (mon. 2021-07-31T01:41:13.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:13 smithi045 conmon[33655]: 0) 234 : audit [INF] from='client.? 172.21.15.45:0/2083264684' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "20724b9a-515c-4878-9e4f-301e21a5a985"}]: dispatch 2021-07-31T01:41:13.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:13 smithi045 conmon[33655]: audit 2021-07-31T01:41:12.646559+0000 mon.a (mon.0) 235 : audit [INF] from='client.? 172.21.15.45:0/2083264684' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "20724b9a-515c-4878-9e4f-301e21a5a985"}]': finished 2021-07-31T01:41:13.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:13 smithi045 conmon[33655]: cluster 2021-07-31T01:41:12.646613+0000 mon.a (mon.0) 236 : cluster [DBG] osdmap e5: 1 total, 0 up, 1 in 2021-07-31T01:41:13.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:13 smithi045 conmon[33655]: audit 2021-07-31T01:41:12.646717+0000 mon.a (mon.0) 237 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-07-31T01:41:13.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:13 smithi045 conmon[39718]: audit 2021-07-31T01:41:12.643107+0000 mon.a (mon.0 2021-07-31T01:41:13.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:13 smithi045 conmon[39718]: ) 234 : audit [INF] from='client.? 172.21.15.45:0/2083264684' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "20724b9a-515c-4878-9e4f-301e21a5a985"}]: dispatch 2021-07-31T01:41:13.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:13 smithi045 conmon[39718]: audit 2021-07-31T01 2021-07-31T01:41:13.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:13 smithi045 conmon[39718]: :41:12.646559+0000 mon.a (mon.0) 235 : audit [INF] from='client.? 172.21.15.45:0/2083264684' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "20724b9a-515c-4878-9e4f-301e21a5a985"}]': finished 2021-07-31T01:41:13.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:13 smithi045 conmon[39718]: 2021-07-31T01:41:13.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:13 smithi045 conmon[39718]: cluster 2021-07-31T01:41:13.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:13 smithi045 conmon[39718]: 2021-07-31T 2021-07-31T01:41:13.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:13 smithi045 conmon[39718]: 01:41:12.646613+0000 mon.a (mon 2021-07-31T01:41:13.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:13 smithi045 conmon[39718]: .0) 236 : cluster [DBG] osdmap e5: 1 total, 0 up, 1 in 2021-07-31T01:41:13.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:13 smithi045 conmon[39718]: audit 2021-07-31T01:41:13.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:13 smithi045 conmon[39718]: 2021-07- 2021-07-31T01:41:13.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:13 smithi045 conmon[39718]: 31T01:41:12.646717+0000 mon.a (mon 2021-07-31T01:41:13.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:13 smithi045 conmon[39718]: .0) 237 : audit [DBG] 2021-07-31T01:41:13.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:13 smithi045 conmon[39718]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-07-31T01:41:14.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:14 smithi154 conmon[34553]: audit 2021-07-31T01:41:13.167445+0000 mon.c 2021-07-31T01:41:14.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:14 smithi154 conmon[34553]: (mon.1) 5 : audit [DBG] from='client.? 172.21.15.45:0/2306288179' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-07-31T01:41:14.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:14 smithi154 conmon[34553]: cluster 2021-07-31T 2021-07-31T01:41:14.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:14 smithi154 conmon[34553]: 01:41:13.307149+0000 mgr.y (mgr.14146) 68 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-07-31T01:41:14.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:14 smithi045 conmon[33655]: audit 2021-07-31T01:41:13. 2021-07-31T01:41:14.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:14 smithi045 conmon[33655]: 167445+0000 mon.c (mon.1) 5 : audit [DBG] from='client.? 172.21.15.45:0/2306288179' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-07-31T01:41:14.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:14 smithi045 conmon[33655]: cluster 2021-07-31T01:41:13.307149+0000 mgr.y (mgr.14146) 68 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-07-31T01:41:14.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:14 smithi045 conmon[39718]: audit 2021-07-31T01:41:13 2021-07-31T01:41:14.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:14 smithi045 conmon[39718]: .167445+0000 mon.c (mon.1) 5 : audit [DBG] from='client.? 172.21.15.45:0/2306288179' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-07-31T01:41:14.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:14 smithi045 conmon[39718]: cluster 2021-07-31T01:41:13.307149+0000 mgr.y (mgr.14146) 68 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-07-31T01:41:16.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:16 smithi045 conmon[33655]: cluster 2021-07-31T01:41 2021-07-31T01:41:16.616 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:16 smithi045 conmon[33655]: :15.307544+0000 mgr.y (mgr.14146) 69 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-07-31T01:41:16.616 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:16 smithi045 conmon[39718]: cluster 2021-07-31T01:41:15.307544 2021-07-31T01:41:16.617 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:16 smithi045 conmon[39718]: +0000 mgr.y (mgr.14146) 69 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-07-31T01:41:16.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:16 smithi154 conmon[34553]: cluster 2021-07-31T01:41:15.307544+0000 mgr.y (mgr 2021-07-31T01:41:16.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:16 smithi154 conmon[34553]: .14146) 69 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-07-31T01:41:18.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:18 smithi154 conmon[34553]: cluster 2021-07-31T01:41:17.307992+0000 mgr.y (mgr 2021-07-31T01:41:18.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:18 smithi154 conmon[34553]: .14146) 70 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-07-31T01:41:18.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:18 smithi045 conmon[33655]: cluster 2021-07-31T01:41:17.307992+0000 mgr.y (mgr.14146) 70 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-07-31T01:41:18.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:18 smithi045 conmon[39718]: cluster 2021-07-31T01:41:17.307992+0000 mgr.y (mgr.14146) 70 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-07-31T01:41:19.731 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:19 smithi045 conmon[33655]: audit 2021-07-31T01 2021-07-31T01:41:19.732 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:19 smithi045 conmon[33655]: :41:18.912539+0000 mon.a (mon.0) 238 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2021-07-31T01:41:19.732 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:19 smithi045 conmon[33655]: audit 2021-07-31T01:41:18.913318+0000 mon.a (mon.0) 239 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:41:19.732 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:19 smithi045 conmon[33655]: cephadm 2021-07-31T01:41:18.913986+0000 mgr.y (mgr.14146) 71 : cephadm [INF] Deploying daemon osd.0 on smithi045 2021-07-31T01:41:19.733 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:19 smithi045 conmon[39718]: audit 2021-07-31T01:41:19.733 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:19 smithi045 conmon[39718]: 2021-07-31T01:41:19.733 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:19 smithi045 conmon[39718]: 2021 2021-07-31T01:41:19.733 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:19 smithi045 conmon[39718]: - 2021-07-31T01:41:19.734 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:19 smithi045 conmon[39718]: 07 2021-07-31T01:41:19.734 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:19 smithi045 conmon[39718]: - 2021-07-31T01:41:19.734 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:19 smithi045 conmon[39718]: 31 2021-07-31T01:41:19.734 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:19 smithi045 conmon[39718]: T 2021-07-31T01:41:19.734 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:19 smithi045 conmon[39718]: 01 2021-07-31T01:41:19.735 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:19 smithi045 conmon[39718]: : 2021-07-31T01:41:19.735 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:19 smithi045 conmon[39718]: 41 2021-07-31T01:41:19.735 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:19 smithi045 conmon[39718]: : 2021-07-31T01:41:19.735 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:19 smithi045 conmon[39718]: 18 2021-07-31T01:41:19.736 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:19 smithi045 conmon[39718]: . 2021-07-31T01:41:19.736 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:19 smithi045 conmon[39718]: 912539 2021-07-31T01:41:19.736 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:19 smithi045 conmon[39718]: +0000 2021-07-31T01:41:19.736 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:19 smithi045 conmon[39718]: 2021-07-31T01:41:19.737 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:19 smithi045 conmon[39718]: mon.a 2021-07-31T01:41:19.737 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:19 smithi045 conmon[39718]: ( 2021-07-31T01:41:19.737 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:19 smithi045 conmon[39718]: mon 2021-07-31T01:41:19.737 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:19 smithi045 conmon[39718]: . 2021-07-31T01:41:19.737 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:19 smithi045 conmon[39718]: 0 2021-07-31T01:41:19.738 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:19 smithi045 conmon[39718]: ) 2021-07-31T01:41:19.738 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:19 smithi045 conmon[39718]: 238 2021-07-31T01:41:19.738 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:19 smithi045 conmon[39718]: : 2021-07-31T01:41:19.738 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:19 smithi045 conmon[39718]: audit 2021-07-31T01:41:19.739 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:19 smithi045 conmon[39718]: 2021-07-31T01:41:19.739 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:19 smithi045 conmon[39718]: [INF] 2021-07-31T01:41:19.739 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:19 smithi045 conmon[39718]: 2021-07-31T01:41:19.739 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:19 smithi045 conmon[39718]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2021-07-31T01:41:19.740 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:19 smithi045 conmon[39718]: 2021-07-31T01:41:19.740 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:19 smithi045 conmon[39718]: audit 2021-07-31T01:41:19.740 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:19 smithi045 conmon[39718]: 2021-07-31T01 2021-07-31T01:41:19.740 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:19 smithi045 conmon[39718]: :41:18.913318+0000 mon.a (mon.0) 239 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:41:19.741 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:19 smithi045 conmon[39718]: 2021-07-31T01:41:19.741 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:19 smithi045 conmon[39718]: cephadm 2021-07- 2021-07-31T01:41:19.741 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:19 smithi045 conmon[39718]: 31T01:41:18. 2021-07-31T01:41:19.741 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:19 smithi045 conmon[39718]: 913986+0000 mgr.y (mgr.14146 2021-07-31T01:41:19.741 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:19 smithi045 conmon[39718]: ) 71 : cephadm [INF] Deploying daemon osd.0 on smithi045 2021-07-31T01:41:19.742 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:19 smithi045 conmon[39718]: 2021-07-31T01:41:19.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:19 smithi154 conmon[34553]: audit 2021-07-31T01:41 2021-07-31T01:41:19.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:19 smithi154 conmon[34553]: :18.912539+0000 mon.a (mon.0) 238 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2021-07-31T01:41:19.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:19 smithi154 conmon[34553]: audit 2021-07-31T01:41:19.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:19 smithi154 conmon[34553]: 2021-07-31T01:41:18.913318+0000 mon.a (mon.0) 239 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:41:19.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:19 smithi154 conmon[34553]: 2021-07-31T01:41:19.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:19 smithi154 conmon[34553]: cephadm 2021-07-31T01:41: 2021-07-31T01:41:19.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:19 smithi154 conmon[34553]: 18.913986+0000 mgr.y (mgr.14146) 71 : 2021-07-31T01:41:19.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:19 smithi154 conmon[34553]: cephadm [INF] Deploying daemon osd.0 on smithi045 2021-07-31T01:41:20.670 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:20 smithi045 conmon[33655]: cluster 2021-07-31T01:41:19.308285+0000 2021-07-31T01:41:20.670 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:20 smithi045 conmon[33655]: mgr.y (mgr.14146) 72 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-07-31T01:41:20.670 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:20 smithi045 conmon[39718]: cluster 2021-07-31T01:41:19. 2021-07-31T01:41:20.671 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:20 smithi045 conmon[39718]: 308285+0000 mgr.y (mgr.14146) 72 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-07-31T01:41:20.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:20 smithi154 conmon[34553]: cluster 2021-07-31T01:41:19.308285+0000 mgr.y 2021-07-31T01:41:20.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:20 smithi154 conmon[34553]: (mgr.14146) 72 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-07-31T01:41:22.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:22 smithi045 conmon[33655]: cluster 2021-07-31T01:41:21 2021-07-31T01:41:22.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:22 smithi045 conmon[33655]: .308529+0000 mgr.y (mgr.14146) 73 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-07-31T01:41:22.620 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:22 smithi045 conmon[39718]: cluster 2021-07-31T01:41:21 2021-07-31T01:41:22.621 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:22 smithi045 conmon[39718]: .308529+0000 mgr.y (mgr.14146) 73 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-07-31T01:41:22.667 INFO:teuthology.orchestra.run.smithi045.stdout:Created osd(s) 0 on host 'smithi045' 2021-07-31T01:41:22.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:22 smithi154 conmon[34553]: cluster 2021-07-31T01:41:21.308529+0000 mgr.y ( 2021-07-31T01:41:22.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:22 smithi154 conmon[34553]: mgr.14146) 73 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-07-31T01:41:23.436 DEBUG:teuthology.orchestra.run.smithi045:osd.0> sudo journalctl -f -n 0 -u ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c@osd.0.service 2021-07-31T01:41:23.438 INFO:tasks.cephadm:Deploying osd.1 on smithi045 with /dev/vg_nvme/lv_3... 2021-07-31T01:41:23.439 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f52d2c14-f19f-11eb-8c24-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2021-07-31T01:41:23.467 INFO:journalctl@ceph.osd.0.smithi045.stdout:-- Logs begin at Sat 2021-07-31 01:16:15 UTC. -- 2021-07-31T01:41:23.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:23 smithi154 conmon[34553]: audit 2021-07-31T01:41:22.657856+0000 mon.a (mon. 2021-07-31T01:41:23.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:23 smithi154 conmon[34553]: 0) 240 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:41:23.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:23 smithi154 conmon[34553]: audit 2021-07-31T01:41:22.659274+0000 mon.a 2021-07-31T01:41:23.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:23 smithi154 conmon[34553]: (mon.0) 241 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:41:23.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:23 smithi154 conmon[34553]: audit 2021-07-31T01:41:22.661178+0000 mon.a (mon.0) 2021-07-31T01:41:23.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:23 smithi154 conmon[34553]: 242 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:41:23.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:23 smithi154 conmon[34553]: audit 2021-07- 2021-07-31T01:41:23.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:23 smithi154 conmon[34553]: 31T01:41:22.662570+0000 mon.a (mon.0) 2021-07-31T01:41:23.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:23 smithi154 conmon[34553]: 243 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:41:23.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:23 smithi154 conmon[34553]: audit 2021-07-31T01:41:23.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:23 smithi154 conmon[34553]: 2021-07-31T01:41:22.664879+0000 mon.a 2021-07-31T01:41:23.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:23 smithi154 conmon[34553]: (mon.0) 244 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:41:23.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:23 smithi045 conmon[39718]: audit 2021-07-31T01:41:22.657856+0000 mon.a (mon.0) 240 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:41:23.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:23 smithi045 conmon[39718]: audit 2021-07-31T01:41:22.659274+0000 mon.a (mon.0) 241 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:41:23.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:23 smithi045 conmon[39718]: audit 2021-07-31T01:41:22.661178+0000 mon.a (mon.0) 242 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:41:23.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:23 smithi045 conmon[39718]: audit 2021-07-31T01:41:22.662570+0000 mon.a (mon.0) 243 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:41:23.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:23 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T01:41:23.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:23 smithi045 conmon[39718]: T01:41:22.664879+0000 mon.a (mon.0) 244 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:41:23.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:23 smithi045 conmon[33655]: audit 2021-07-31T01:41:22.657856+0000 mon.a (mon.0) 240 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:41:23.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:23 smithi045 conmon[33655]: audit 2021-07-31T01:41:22.659274+0000 mon.a (mon.0) 241 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:41:23.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:23 smithi045 conmon[33655]: audit 2021-07-31T01:41:22.661178+0000 mon.a (mon.0) 242 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:41:23.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:23 smithi045 conmon[33655]: audit 2021-07-31T01:41:22.662570+0000 mon.a (mon.0) 243 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:41:23.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:23 smithi045 conmon[33655]: audit 2021-07-31T01:41:22.664879+0000 mon.a (mon.0) 244 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:41:25.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:24 smithi154 conmon[34553]: cluster 2021-07-31T01:41:23.308803+0000 mgr.y ( 2021-07-31T01:41:25.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:24 smithi154 conmon[34553]: mgr.14146) 74 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-07-31T01:41:25.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:24 smithi045 conmon[33655]: cluster 2021-07-31T01:41:23. 2021-07-31T01:41:25.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:24 smithi045 conmon[33655]: 308803+0000 mgr.y (mgr.14146) 74 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-07-31T01:41:25.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:24 smithi045 conmon[39718]: cluster 2021-07-31T01:41:23.308803 2021-07-31T01:41:25.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:24 smithi045 conmon[39718]: +0000 mgr.y (mgr.14146) 74 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-07-31T01:41:25.849 INFO:teuthology.orchestra.run.smithi045.stderr:--> Zapping: /dev/vg_nvme/lv_3 2021-07-31T01:41:25.849 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2021-07-31T01:41:25.849 INFO:teuthology.orchestra.run.smithi045.stderr: stderr: 10+0 records in 2021-07-31T01:41:25.850 INFO:teuthology.orchestra.run.smithi045.stderr:10+0 records out 2021-07-31T01:41:25.850 INFO:teuthology.orchestra.run.smithi045.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00943674 s, 1.1 GB/s 2021-07-31T01:41:25.850 INFO:teuthology.orchestra.run.smithi045.stderr:--> Zapping successful for: 2021-07-31T01:41:25.934 INFO:journalctl@ceph.osd.0.smithi045.stdout:Jul 31 01:41:25 smithi045 conmon[43243]: debug 2021-07-31T01:41:25.766+0000 7fdf95c12080 -1 osd.0 0 log_to_monitors {default=true} 2021-07-31T01:41:26.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:26 smithi154 conmon[34553]: audit 2021-07-31T01:41:25.240653+0000 mon.a (mon.0 2021-07-31T01:41:26.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:26 smithi154 conmon[34553]: ) 245 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:41:26.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:26 smithi154 conmon[34553]: cluster 2021-07-31T01:41:25.309233+0000 mgr.y (mgr.14146) 2021-07-31T01:41:26.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:26 smithi154 conmon[34553]: 75 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-07-31T01:41:26.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:26 smithi154 conmon[34553]: audit 2021-07-31T01:41:25.769338+0000 2021-07-31T01:41:26.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:26 smithi154 conmon[34553]: mon.a (mon.0) 246 : audit [INF] from='osd.0 [v2:172.21.15.45:6802/2841572524,v1:172.21.15.45:6803/2841572524]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2021-07-31T01:41:26.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:26 smithi045 conmon[39718]: audit 2021-07-31T01:41:25.240653+0000 mon.a (mon.0) 245 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:41:26.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:26 smithi045 conmon[39718]: cluster 2021-07-31T01:41:25.309233+0000 mgr.y (mgr.14146) 75 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-07-31T01:41:26.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:26 smithi045 conmon[39718]: audit 2021-07-31T01:41:25.769338+0000 mon.a (mon.0) 246 : audit [INF] from='osd.0 [v2:172.21.15.45:6802/2841572524,v1:172.21.15.45:6803/2841572524]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2021-07-31T01:41:26.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:26 smithi045 conmon[33655]: audit 2021-07-31T01:41:25.240653+0000 mon.a (mon.0) 245 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:41:26.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:26 smithi045 conmon[33655]: cluster 2021-07-31T01:41:25.309233+0000 mgr.y (mgr.14146) 75 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-07-31T01:41:26.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:26 smithi045 conmon[33655]: audit 2021-07-31T01:41:25.769338+0000 mon.a (mon.0) 246 : audit [INF] from='osd.0 [v2:172.21.15.45:6802/2841572524,v1:172.21.15.45:6803/2841572524]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2021-07-31T01:41:26.742 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f52d2c14-f19f-11eb-8c24-001a4aab830c -- ceph orch daemon add osd smithi045:vg_nvme/lv_3 2021-07-31T01:41:27.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:27 smithi154 conmon[34553]: audit 2021-07-31T01:41:26.246219+0000 mon.a 2021-07-31T01:41:27.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:27 smithi154 conmon[34553]: (mon.0) 247 : audit [INF] from='osd.0 [v2:172.21.15.45:6802/2841572524,v1:172.21.15.45:6803/2841572524]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2021-07-31T01:41:27.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:27 smithi154 conmon[34553]: cluster 2021-07-31T01:41:26.246272 2021-07-31T01:41:27.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:27 smithi154 conmon[34553]: +0000 mon.a (mon.0) 248 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2021-07-31T01:41:27.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:27 smithi154 conmon[34553]: audit 2021-07-31T01:41:26 2021-07-31T01:41:27.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:27 smithi154 conmon[34553]: .246526+0000 mon.a (mon.0) 249 : audit [DBG] 2021-07-31T01:41:27.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:27 smithi154 conmon[34553]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-07-31T01:41:27.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:27 smithi154 conmon[34553]: audit 2021-07- 2021-07-31T01:41:27.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:27 smithi154 conmon[34553]: 31T01:41:26.246900+0000 mon.a (mon. 2021-07-31T01:41:27.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:27 smithi154 conmon[34553]: 0) 250 : audit [INF] from='osd.0 [v2:172.21.15.45:6802/2841572524,v1:172.21.15.45:6803/2841572524]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2021-07-31T01:41:27.526 INFO:journalctl@ceph.osd.0.smithi045.stdout:Jul 31 01:41:27 smithi045 conmon[43243]: debug 2021-07-31T01:41:27.250+0000 7fdf8c175700 -1 osd.0 0 waiting for initial osdmap 2021-07-31T01:41:27.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:27 smithi045 conmon[33655]: audit 2021-07-31T01:41:26.246219 2021-07-31T01:41:27.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:27 smithi045 conmon[33655]: +0000 mon.a (mon.0) 247 : audit [INF] from='osd.0 [v2:172.21.15.45:6802/2841572524,v1:172.21.15.45:6803/2841572524]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2021-07-31T01:41:27.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:27 smithi045 conmon[33655]: cluster 2021-07-31T01:41:26.246272+0000 mon.a (mon.0) 248 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2021-07-31T01:41:27.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:27 smithi045 conmon[33655]: audit 2021-07-31T01:41:26.246526+0000 mon.a (mon.0) 249 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-07-31T01:41:27.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:27 smithi045 conmon[33655]: audit 2021-07-31T01:41:26.246900+0000 mon.a (mon.0) 250 : audit [INF] from='osd.0 [v2:172.21.15.45:6802/2841572524,v1:172.21.15.45:6803/2841572524]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2021-07-31T01:41:27.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:27 smithi045 conmon[39718]: audit 2021-07-31T01:41:26.246219+0000 mon.a (mon.0 2021-07-31T01:41:27.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:27 smithi045 conmon[39718]: ) 247 : audit [INF] from='osd.0 [v2:172.21.15.45:6802/2841572524,v1:172.21.15.45:6803/2841572524]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2021-07-31T01:41:27.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:27 smithi045 conmon[39718]: cluster 2021-07-31T01:41:26.246272+0000 mon.a (mon.0) 2021-07-31T01:41:27.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:27 smithi045 conmon[39718]: 248 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2021-07-31T01:41:27.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:27 smithi045 conmon[39718]: audit 2021-07-31T01 2021-07-31T01:41:27.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:27 smithi045 conmon[39718]: :41:26.246526+0000 mon.a (mon.0) 249 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-07-31T01:41:27.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:27 smithi045 conmon[39718]: audit 2021-07-31T01:41 2021-07-31T01:41:27.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:27 smithi045 conmon[39718]: :26.246900+0000 mon.a (mon.0) 250 : audit [INF] 2021-07-31T01:41:27.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:27 smithi045 conmon[39718]: from='osd.0 [v2:172.21.15.45:6802/2841572524,v1:172.21.15.45:6803/2841572524]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2021-07-31T01:41:28.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:28 smithi045 conmon[39718]: audit 2021-07-31T01:41:27.246573+0000 2021-07-31T01:41:28.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:28 smithi045 conmon[39718]: mon.a (mon.0) 251 : audit [INF] from='osd.0 [v2:172.21.15.45:6802/2841572524,v1:172.21.15.45:6803/2841572524]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2021-07-31T01:41:28.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:28 smithi045 conmon[39718]: cluster 2021-07-31T01:41:27.246638+0000 mon.a (mon.0) 252 : cluster [DBG] osdmap e7: 1 total, 0 up, 1 in 2021-07-31T01:41:28.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:28 smithi045 conmon[39718]: audit 2021-07-31T01:41:27.246994+0000 mon.a (mon.0) 253 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-07-31T01:41:28.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:28 smithi045 conmon[39718]: audit 2021-07-31T01:41:27.250798+0000 mon.a (mon.0) 254 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-07-31T01:41:28.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:28 smithi045 conmon[39718]: cluster 2021-07-31T01:41:27.309523+0000 mgr.y (mgr.14146) 76 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-07-31T01:41:28.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:28 smithi045 conmon[33655]: audit 2021-07-31T01:41 2021-07-31T01:41:28.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:28 smithi045 conmon[33655]: :27.246573+0000 mon.a (mon.0) 251 : audit [INF] from='osd.0 [v2:172.21.15.45:6802/2841572524,v1:172.21.15.45:6803/2841572524]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2021-07-31T01:41:28.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:28 smithi045 conmon[33655]: cluster 2021-07-31T01:41:27.246638+0000 mon.a (mon.0) 252 : cluster [DBG] osdmap e7: 1 total, 0 up, 1 in 2021-07-31T01:41:28.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:28 smithi045 conmon[33655]: audit 2021-07-31T01:41:27.246994+0000 mon.a (mon.0) 253 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-07-31T01:41:28.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:28 smithi045 conmon[33655]: audit 2021-07-31T01:41:27.250798+0000 mon.a (mon.0) 254 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-07-31T01:41:28.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:28 smithi045 conmon[33655]: cluster 2021-07-31T01:41:27.309523+0000 mgr.y (mgr.14146) 76 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-07-31T01:41:28.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:28 smithi154 conmon[34553]: audit 2021-07-31T01:41:27.246573+0000 2021-07-31T01:41:28.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:28 smithi154 conmon[34553]: mon.a (mon.0) 251 : audit [INF] from='osd.0 [v2:172.21.15.45:6802/2841572524,v1:172.21.15.45:6803/2841572524]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2021-07-31T01:41:28.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:28 smithi154 conmon[34553]: cluster 2021-07- 2021-07-31T01:41:28.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:28 smithi154 conmon[34553]: 31T01:41:27.246638+0000 mon.a (mon.0) 252 : cluster [DBG] osdmap e7: 1 total, 0 up, 1 in 2021-07-31T01:41:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:28 smithi154 conmon[34553]: audit 2021-07-31T01:41:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:28 smithi154 conmon[34553]: 2021-07-31T01:41:27.246994+0000 mon.a ( 2021-07-31T01:41:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:28 smithi154 conmon[34553]: mon.0) 253 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-07-31T01:41:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:28 smithi154 conmon[34553]: audit 2021-07-31 2021-07-31T01:41:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:28 smithi154 conmon[34553]: T01:41:27.250798+0000 mon.a (mon.0) 254 : audit [DBG] 2021-07-31T01:41:28.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:28 smithi154 conmon[34553]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-07-31T01:41:28.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:28 smithi154 conmon[34553]: cluster 2021-07 2021-07-31T01:41:28.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:28 smithi154 conmon[34553]: -31T01:41:27.309523+0000 mgr.y (mgr 2021-07-31T01:41:28.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:28 smithi154 conmon[34553]: .14146) 76 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-07-31T01:41:29.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:29 smithi045 conmon[33655]: cluster 2021-07-31T01:41:26.799427+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2021-07-31T01:41:29.539 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:29 smithi045 conmon[33655]: cluster 2021-07-31T01:41:26.799527+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2021-07-31T01:41:29.540 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:29 smithi045 conmon[33655]: audit 2021-07-31T01:41:28.251042+0000 mon.a (mon.0) 255 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-07-31T01:41:29.540 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:29 smithi045 conmon[33655]: cluster 2021-07-31T01:41:28.254893+0000 mon.a (mon.0) 256 : cluster [INF] osd.0 [v2:172.21.15.45:6802/2841572524,v1:172.21.15.45:6803/2841572524] boot 2021-07-31T01:41:29.540 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:29 smithi045 conmon[33655]: cluster 2021-07-31T01:41:28.254926+0000 mon.a (mon.0) 257 : cluster [DBG] osdmap e8: 1 total, 1 up, 1 in 2021-07-31T01:41:29.540 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:29 smithi045 conmon[33655]: audit 2021-07-31T01:41:28.255337+0000 mon.a (mon.0) 258 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-07-31T01:41:29.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:29 smithi045 conmon[33655]: audit 2021-07-31T01:41:29.179948+0000 mon.a (mon.0) 259 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-07-31T01:41:29.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:29 smithi045 conmon[33655]: audit 2021-07-31T01:41:29.181233+0000 mon.a (mon.0) 260 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-07-31T01:41:29.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:29 smithi045 conmon[33655]: audit 2021-07-31T01:41:29.181802+0000 mon.a (mon.0) 261 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:41:29.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:29 smithi045 conmon[39718]: cluster 2021-07-31T01:41:26.799427+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2021-07-31T01:41:29.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:29 smithi045 conmon[39718]: cluster 2021-07-31T01:41:26.799527+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2021-07-31T01:41:29.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:29 smithi045 conmon[39718]: audit 2021-07-31T01:41:28.251042+0000 mon.a (mon.0) 255 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-07-31T01:41:29.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:29 smithi045 conmon[39718]: cluster 2021-07-31T01:41:28.254893+0000 mon.a (mon.0) 256 : cluster [INF] osd.0 [v2:172.21.15.45:6802/2841572524,v1:172.21.15.45:6803/2841572524] boot 2021-07-31T01:41:29.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:29 smithi045 conmon[39718]: cluster 2021-07-31T01:41 2021-07-31T01:41:29.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:29 smithi045 conmon[39718]: :28.254926+0000 mon.a (mon.0) 257 : cluster [DBG] osdmap e8: 1 total, 1 up, 1 in 2021-07-31T01:41:29.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:29 smithi045 conmon[39718]: audit 2021-07-31T01:41:28.255337+0000 mon.a (mon.0) 258 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-07-31T01:41:29.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:29 smithi045 conmon[39718]: audit 2021-07-31T01:41:29.179948+0000 mon.a (mon.0) 259 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-07-31T01:41:29.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:29 smithi045 conmon[39718]: audit 2021-07-31T01:41:29.181233+0000 mon.a (mon.0) 260 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-07-31T01:41:29.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:29 smithi045 conmon[39718]: audit 2021-07-31T01:41:29.181802+0000 mon.a (mon.0) 261 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:41:29.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:29 smithi154 conmon[34553]: cluster 2021-07-31T01:41:26.799427+0000 2021-07-31T01:41:29.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:29 smithi154 conmon[34553]: osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2021-07-31T01:41:29.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:29 smithi154 conmon[34553]: cluster 2021-07-31T01 2021-07-31T01:41:29.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:29 smithi154 conmon[34553]: :41:26.799527+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2021-07-31T01:41:29.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:29 smithi154 conmon[34553]: audit 2021 2021-07-31T01:41:29.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:29 smithi154 conmon[34553]: -07-31T01:41:28.251042+0000 mon.a 2021-07-31T01:41:29.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:29 smithi154 conmon[34553]: (mon.0) 255 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-07-31T01:41:29.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:29 smithi154 conmon[34553]: cluster 2021-07 2021-07-31T01:41:29.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:29 smithi154 conmon[34553]: -31T01:41:28.254893+0000 mon.a (mon.0) 256 : cluster 2021-07-31T01:41:29.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:29 smithi154 conmon[34553]: [INF] osd.0 [v2:172.21.15.45:6802/2841572524,v1:172.21.15.45:6803/2841572524] boot 2021-07-31T01:41:29.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:29 smithi154 conmon[34553]: cluster 2021 2021-07-31T01:41:29.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:29 smithi154 conmon[34553]: -07-31T01:41:28.254926+0000 mon.a 2021-07-31T01:41:29.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:29 smithi154 conmon[34553]: (mon.0) 257 : cluster [DBG] osdmap e8: 1 total, 1 up, 1 in 2021-07-31T01:41:29.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:29 smithi154 conmon[34553]: audit 2021-07-31T01:41:29.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:29 smithi154 conmon[34553]: 2021-07-31T01:41:28.255337 2021-07-31T01:41:29.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:29 smithi154 conmon[34553]: +0000 mon.a (mon.0) 258 : audit [DBG] 2021-07-31T01:41:29.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:29 smithi154 conmon[34553]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-07-31T01:41:29.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:29 smithi154 conmon[34553]: audit 2021- 2021-07-31T01:41:29.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:29 smithi154 conmon[34553]: 07-31T01:41:29.179948+0000 mon.a ( 2021-07-31T01:41:29.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:29 smithi154 conmon[34553]: mon.0) 259 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-07-31T01:41:29.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:29 smithi154 conmon[34553]: audit 2021-07-31T01:41:29.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:29 smithi154 conmon[34553]: 2021-07-31T01:41:29.181233+0000 mon.a ( 2021-07-31T01:41:29.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:29 smithi154 conmon[34553]: mon.0) 260 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-07-31T01:41:29.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:29 smithi154 conmon[34553]: audit 2021-07-31T01:41:29.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:29 smithi154 conmon[34553]: 2021-07-31T01:41:29.181802+0000 mon.a ( 2021-07-31T01:41:29.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:29 smithi154 conmon[34553]: mon.0) 261 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:41:30.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:30 smithi045 conmon[33655]: audit 2021-07-31T01:41:29.179238+0000 2021-07-31T01:41:30.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:30 smithi045 conmon[33655]: mgr.y (mgr.14146) 77 : audit [DBG] from='client.24148 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:41:30.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:30 smithi045 conmon[33655]: cluster 2021-07-31T01:41:29.258188+0000 mon.a (mon.0) 262 : cluster [DBG] osdmap e9: 1 total, 1 up, 1 in 2021-07-31T01:41:30.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:30 smithi045 conmon[33655]: cluster 2021-07-31T01:41:29.309809+0000 mgr.y (mgr.14146) 78 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-07-31T01:41:30.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:30 smithi045 conmon[33655]: audit 2021-07-31T01:41:29.983718+0000 mon.a (mon.0) 263 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:41:30.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:30 smithi045 conmon[33655]: audit 2021-07-31T01:41:29.984499+0000 mon.a (mon.0) 264 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-07-31T01:41:30.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:30 smithi045 conmon[33655]: audit 2021-07-31T01:41:29.988267+0000 mon.a (mon.0) 265 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:41:30.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:30 smithi045 conmon[33655]: audit 2021-07-31T01:41:29.991348+0000 mon.a (mon.0) 266 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:41:30.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:30 smithi045 conmon[33655]: audit 2021-07-31T01:41:29.992710+0000 mon.a (mon.0) 267 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:41:30.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:30 smithi045 conmon[33655]: audit 2021-07-31T01:41:29.993538+0000 mon.a (mon.0) 268 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:41:30.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:30 smithi045 conmon[33655]: audit 2021-07-31T01:41:29.994050+0000 mon.a (mon.0) 269 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:41:30.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:30 smithi045 conmon[33655]: audit 2021-07-31T01:41:29. 2021-07-31T01:41:30.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:30 smithi045 conmon[33655]: 997583+0000 mon.a (mon.0) 270 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:41:30.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:30 smithi045 conmon[39718]: audit 2021-07-31T01:41:29.179238+0000 2021-07-31T01:41:30.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:30 smithi045 conmon[39718]: mgr.y (mgr.14146) 77 : audit [DBG] from='client.24148 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:41:30.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:30 smithi045 conmon[39718]: cluster 2021-07-31T01:41:29.258188+0000 mon.a (mon.0) 262 : cluster [DBG] osdmap e9: 1 total, 1 up, 1 in 2021-07-31T01:41:30.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:30 smithi045 conmon[39718]: cluster 2021-07-31T01:41:29.309809+0000 mgr.y (mgr.14146) 78 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-07-31T01:41:30.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:30 smithi045 conmon[39718]: audit 2021-07-31T01:41:29.983718+0000 mon.a (mon.0) 263 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:41:30.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:30 smithi045 conmon[39718]: audit 2021-07-31T01:41:29.984499+0000 mon.a (mon.0) 264 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-07-31T01:41:30.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:30 smithi045 conmon[39718]: audit 2021-07-31T01:41:29.988267+0000 mon.a (mon.0) 265 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:41:30.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:30 smithi045 conmon[39718]: audit 2021-07-31T01:41:29.991348+0000 mon.a (mon.0) 266 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:41:30.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:30 smithi045 conmon[39718]: audit 2021-07-31T01:41:29.992710+0000 mon.a (mon.0) 267 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:41:30.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:30 smithi045 conmon[39718]: audit 2021-07-31T01:41:29.993538+0000 mon.a (mon.0) 268 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:41:30.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:30 smithi045 conmon[39718]: audit 2021-07 2021-07-31T01:41:30.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:30 smithi045 conmon[39718]: -31T01:41:29.994050+0000 mon.a (mon 2021-07-31T01:41:30.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:30 smithi045 conmon[39718]: .0) 269 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:41:30.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:30 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T01:41:30.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:30 smithi045 conmon[39718]: T01:41:29.997583+0000 mon.a (mon 2021-07-31T01:41:30.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:30 smithi045 conmon[39718]: .0) 270 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:41:30.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:30 smithi154 conmon[34553]: audit 2021-07-31T01:41:29.179238+0000 mgr.y (mgr 2021-07-31T01:41:30.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:30 smithi154 conmon[34553]: .14146) 77 : audit [DBG] from='client.24148 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:41:30.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:30 smithi154 conmon[34553]: cluster 2021-07-31T01:41:29.258188+0000 mon.a (mon 2021-07-31T01:41:30.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:30 smithi154 conmon[34553]: .0) 262 : cluster [DBG] osdmap e9: 1 total, 1 up, 1 in 2021-07-31T01:41:30.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:30 smithi154 conmon[34553]: cluster 2021-07-31T01:41:29.309809+0000 2021-07-31T01:41:30.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:30 smithi154 conmon[34553]: mgr.y (mgr.14146) 78 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-07-31T01:41:30.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:30 smithi154 conmon[34553]: audit 2021 2021-07-31T01:41:30.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:30 smithi154 conmon[34553]: -07-31T01:41:29.983718+0000 mon.a (mon.0) 263 : 2021-07-31T01:41:30.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:30 smithi154 conmon[34553]: audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:41:30.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:30 smithi154 conmon[34553]: audit 2021-07-31T01:41:30.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:30 smithi154 conmon[34553]: 2021-07-31T01:41:29.984499+0000 mon.a ( 2021-07-31T01:41:30.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:30 smithi154 conmon[34553]: mon.0) 264 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-07-31T01:41:30.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:30 smithi154 conmon[34553]: audit 2021-07-31T01:41:30.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:30 smithi154 conmon[34553]: 2021-07-31T01:41:29.988267+0000 mon.a (mon. 2021-07-31T01:41:30.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:30 smithi154 conmon[34553]: 0) 265 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:41:30.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:30 smithi154 conmon[34553]: audit 2021-07-31T01:41:29.991348+0000 mon.a (mon.0 2021-07-31T01:41:30.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:30 smithi154 conmon[34553]: ) 266 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:41:30.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:30 smithi154 conmon[34553]: audit 2021-07-31T01:41:30.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:30 smithi154 conmon[34553]: 2021-07-31T01:41:29.992710 2021-07-31T01:41:30.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:30 smithi154 conmon[34553]: +0000 mon.a (mon.0) 267 : audit [DBG] 2021-07-31T01:41:30.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:30 smithi154 conmon[34553]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:41:30.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:30 smithi154 conmon[34553]: audit 2021-07-31T 2021-07-31T01:41:30.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:30 smithi154 conmon[34553]: 01:41:29.993538+0000 mon.a (mon.0) 2021-07-31T01:41:30.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:30 smithi154 conmon[34553]: 268 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:41:30.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:30 smithi154 conmon[34553]: audit 2021-07-31T01:41:30.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:30 smithi154 conmon[34553]: 2021-07-31T01:41:29.994050+0000 2021-07-31T01:41:30.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:30 smithi154 conmon[34553]: mon.a (mon.0) 269 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:41:30.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:30 smithi154 conmon[34553]: 2021-07-31T01:41:30.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:30 smithi154 conmon[34553]: audit 2021-07-31T01:41:29.997583 2021-07-31T01:41:30.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:30 smithi154 conmon[34553]: +0000 mon.a (mon.0) 270 : audit [INF] 2021-07-31T01:41:30.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:30 smithi154 conmon[34553]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:41:31.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:31 smithi045 conmon[33655]: cephadm 2021-07-31T01:41:29.979732+0000 mgr.y (mgr.14146) 79 : cephadm [INF] Detected new or changed devices on smithi045 2021-07-31T01:41:31.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:31 smithi045 conmon[33655]: cephadm 2021-07-31T01:41:29.985034+0000 mgr.y (mgr.14146) 80 : cephadm [INF] Adjusting osd_memory_target on smithi045 to 16152M 2021-07-31T01:41:31.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:31 smithi045 conmon[39718]: cephadm 2021-07-31T01:41:29.979732+0000 mgr.y (mgr.14146) 79 : cephadm [INF] Detected new or changed devices on smithi045 2021-07-31T01:41:31.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:31 smithi045 conmon[39718]: cephadm 2021-07-31T01:41:29.985034+0000 mgr.y (mgr.14146) 80 : cephadm [INF] Adjusting osd_memory_target on smithi045 to 16152M 2021-07-31T01:41:31.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:31 smithi154 conmon[34553]: cephadm 2021-07-31T01:41:29.979732+0000 mgr.y 2021-07-31T01:41:31.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:31 smithi154 conmon[34553]: (mgr.14146) 79 : cephadm [INF] Detected new or changed devices on smithi045 2021-07-31T01:41:31.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:31 smithi154 conmon[34553]: cephadm 2021-07-31T01: 2021-07-31T01:41:31.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:31 smithi154 conmon[34553]: 41:29.985034+0000 mgr.y (mgr.14146) 80 : cephadm [INF] Adjusting osd_memory_target on smithi045 to 16152M 2021-07-31T01:41:32.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:32 smithi154 conmon[34553]: cluster 2021-07-31T01:41:31.310089+0000 mgr.y (mgr 2021-07-31T01:41:32.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:32 smithi154 conmon[34553]: .14146) 81 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-07-31T01:41:32.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:32 smithi154 conmon[34553]: audit 2021-07-31T01:41:31.771292+0000 mon.a 2021-07-31T01:41:32.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:32 smithi154 conmon[34553]: (mon.0) 271 : audit [INF] from='client.? 172.21.15.45:0/2811762386' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f47f025a-6f29-417a-9bd1-9af9f91f38a6"}]: dispatch 2021-07-31T01:41:32.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:32 smithi154 conmon[34553]: audit 2021 2021-07-31T01:41:32.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:32 smithi154 conmon[34553]: -07-31T01:41:31.775114+0000 mon.a (mon. 2021-07-31T01:41:32.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:32 smithi154 conmon[34553]: 0) 272 : audit [INF] from='client.? 172.21.15.45:0/2811762386' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f47f025a-6f29-417a-9bd1-9af9f91f38a6"}]': finished 2021-07-31T01:41:32.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:32 smithi154 conmon[34553]: cluster 2021-07-31T01: 2021-07-31T01:41:32.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:32 smithi154 conmon[34553]: 41:31.775188+0000 mon.a (mon.0) 273 : cluster [DBG] osdmap e10: 2 total, 1 up, 2 in 2021-07-31T01:41:32.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:32 smithi154 conmon[34553]: 2021-07-31T01:41:32.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:32 smithi154 conmon[34553]: audit 2021-07-31T01 2021-07-31T01:41:32.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:32 smithi154 conmon[34553]: :41:31.775294+0000 mon.a (mon.0) 274 2021-07-31T01:41:32.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:32 smithi154 conmon[34553]: : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-07-31T01:41:32.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:32 smithi154 conmon[34553]: audit 2021-07-31T01:41:32.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:32 smithi154 conmon[34553]: 2021-07- 2021-07-31T01:41:32.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:32 smithi154 conmon[34553]: 31T01:41:32.315076+0000 mon.a (mon. 2021-07-31T01:41:32.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:32 smithi154 conmon[34553]: 0) 275 : audit [DBG] from='client.? 172.21.15.45:0/629360527' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-07-31T01:41:32.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:32 smithi045 conmon[33655]: cluster 2021-07-31T01:41: 2021-07-31T01:41:32.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:32 smithi045 conmon[33655]: 31.310089+0000 mgr.y (mgr.14146) 81 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-07-31T01:41:32.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:32 smithi045 conmon[33655]: audit 2021-07-31T01:41:31.771292+0000 mon.a (mon.0) 271 : audit [INF] from='client.? 172.21.15.45:0/2811762386' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f47f025a-6f29-417a-9bd1-9af9f91f38a6"}]: dispatch 2021-07-31T01:41:32.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:32 smithi045 conmon[33655]: audit 2021-07-31T01:41:31.775114+0000 mon.a (mon.0) 272 : audit [INF] from='client.? 172.21.15.45:0/2811762386' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f47f025a-6f29-417a-9bd1-9af9f91f38a6"}]': finished 2021-07-31T01:41:32.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:32 smithi045 conmon[33655]: cluster 2021-07-31T01:41:31.775188+0000 mon.a (mon.0) 273 : cluster [DBG] osdmap e10: 2 total, 1 up, 2 in 2021-07-31T01:41:32.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:32 smithi045 conmon[33655]: audit 2021-07-31T01:41:31.775294+0000 mon.a (mon.0) 274 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-07-31T01:41:32.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:32 smithi045 conmon[33655]: audit 2021-07-31T01:41:32.315076+0000 mon.a (mon.0) 275 : audit [DBG] from='client.? 172.21.15.45:0/629360527' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-07-31T01:41:32.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:32 smithi045 conmon[39718]: cluster 2021-07-31T01:41:31.310089+0000 mgr.y (mgr 2021-07-31T01:41:32.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:32 smithi045 conmon[39718]: .14146) 81 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-07-31T01:41:32.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:32 smithi045 conmon[39718]: audit 2021-07-31T01:41:31.771292+0000 mon.a (mon.0) 271 : audit [INF] from='client.? 172.21.15.45:0/2811762386' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f47f025a-6f29-417a-9bd1-9af9f91f38a6"}]: dispatch 2021-07-31T01:41:32.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:32 smithi045 conmon[39718]: audit 2021-07-31T01:41:31.775114+0000 mon.a (mon.0) 272 : audit [INF] from='client.? 172.21.15.45:0/2811762386' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f47f025a-6f29-417a-9bd1-9af9f91f38a6"}]': finished 2021-07-31T01:41:32.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:32 smithi045 conmon[39718]: cluster 2021-07-31T01:41:31.775188+0000 mon.a (mon.0) 273 : cluster [DBG] osdmap e10: 2 total, 1 up, 2 in 2021-07-31T01:41:32.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:32 smithi045 conmon[39718]: audit 2021-07-31T01:41:31.775294+0000 mon.a (mon.0) 274 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-07-31T01:41:32.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:32 smithi045 conmon[39718]: audit 2021-07-31T01:41:32.315076+0000 2021-07-31T01:41:32.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:32 smithi045 conmon[39718]: mon.a (mon.0) 275 : audit [DBG] from='client.? 172.21.15.45:0/629360527' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-07-31T01:41:34.739 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:34 smithi045 conmon[39718]: cluster 2021-07-31T01:41:33.310411+0000 mgr.y (mgr 2021-07-31T01:41:34.739 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:34 smithi045 conmon[39718]: .14146) 82 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-07-31T01:41:34.740 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:34 smithi045 conmon[33655]: cluster 2021-07-31T01:41:33.310411+0000 mgr.y (mgr.14146) 2021-07-31T01:41:34.740 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:34 smithi045 conmon[33655]: 82 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-07-31T01:41:34.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:34 smithi154 conmon[34553]: cluster 2021-07-31T01:41:33.310411+0000 mgr.y (mgr. 2021-07-31T01:41:34.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:34 smithi154 conmon[34553]: 14146) 82 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-07-31T01:41:36.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:36 smithi045 conmon[33655]: cluster 2021-07-31T01:41:35.310750+0000 mgr.y (mgr.14146) 83 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-07-31T01:41:36.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:36 smithi045 conmon[39718]: cluster 2021-07-31T01:41:35.310750+0000 mgr.y (mgr.14146) 83 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-07-31T01:41:36.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:36 smithi154 conmon[34553]: cluster 2021-07-31T01:41:35.310750+0000 mgr.y 2021-07-31T01:41:36.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:36 smithi154 conmon[34553]: (mgr.14146) 83 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-07-31T01:41:38.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:38 smithi045 conmon[33655]: cluster 2021-07-31T01:41:37.311023+0000 2021-07-31T01:41:38.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:38 smithi045 conmon[33655]: mgr.y (mgr.14146) 84 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-07-31T01:41:38.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:38 smithi045 conmon[33655]: audit 2021-07-31T01:41:38.274872+0000 mon.a (mon.0) 276 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2021-07-31T01:41:38.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:38 smithi045 conmon[33655]: audit 2021-07-31T01:41:38.275557+0000 mon.a (mon.0) 277 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:41:38.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:38 smithi045 conmon[39718]: cluster 2021-07-31T01:41 2021-07-31T01:41:38.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:38 smithi045 conmon[39718]: :37.311023+0000 mgr.y (mgr.14146) 84 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-07-31T01:41:38.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:38 smithi045 conmon[39718]: audit 2021-07-31T01:41:38.274872+0000 mon.a (mon.0) 276 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2021-07-31T01:41:38.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:38 smithi045 conmon[39718]: audit 2021-07-31T01:41:38.275557+0000 mon.a (mon.0) 277 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:41:38.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:38 smithi154 conmon[34553]: cluster 2021-07-31T01:41:37.311023+0000 mgr.y ( 2021-07-31T01:41:38.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:38 smithi154 conmon[34553]: mgr.14146) 84 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-07-31T01:41:38.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:38 smithi154 conmon[34553]: audit 2021-07-31T01:41:38.274872+0000 2021-07-31T01:41:38.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:38 smithi154 conmon[34553]: mon.a (mon.0) 276 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2021-07-31T01:41:38.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:38 smithi154 conmon[34553]: audit 2021-07-31T01:41:38. 2021-07-31T01:41:38.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:38 smithi154 conmon[34553]: 275557+0000 mon.a (mon.0) 277 : audit [DBG] 2021-07-31T01:41:38.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:38 smithi154 conmon[34553]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:41:39.423 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:39 smithi045 conmon[33655]: cephadm 2021-07-31T 2021-07-31T01:41:39.423 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:39 smithi045 conmon[33655]: 01:41:38.276074+0000 mgr.y (mgr.14146) 85 : cephadm [INF] Deploying daemon osd.1 on smithi045 2021-07-31T01:41:39.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:39 smithi154 conmon[34553]: cephadm 2021-07-31T01:41:38.276074+0000 2021-07-31T01:41:39.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:39 smithi154 conmon[34553]: mgr.y (mgr.14146) 85 : cephadm [INF] Deploying daemon osd.1 on smithi045 2021-07-31T01:41:39.758 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:39 smithi045 conmon[39718]: cephadm 2021-07-31T01:41 2021-07-31T01:41:39.759 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:39 smithi045 conmon[39718]: :38.276074+0000 mgr.y (mgr.14146) 85 : cephadm [INF] Deploying daemon osd.1 on smithi045 2021-07-31T01:41:40.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:40 smithi154 conmon[34553]: cluster 2021-07-31T01:41:39.311340 2021-07-31T01:41:40.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:40 smithi154 conmon[34553]: +0000 mgr.y (mgr.14146) 86 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-07-31T01:41:40.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:40 smithi154 conmon[34553]: audit 2021-07 2021-07-31T01:41:40.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:40 smithi154 conmon[34553]: -31T01:41:39.424501+0000 mon.a (mon.0) 278 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:41:40.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:40 smithi154 conmon[34553]: audit 2021-07-31T 2021-07-31T01:41:40.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:40 smithi154 conmon[34553]: 01:41:39.425208+0000 mon.a (mon.0) 279 : audit [INF] 2021-07-31T01:41:40.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:40 smithi154 conmon[34553]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-07-31T01:41:40.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:40 smithi045 conmon[33655]: cluster 2021-07-31T01:41:39 2021-07-31T01:41:40.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:40 smithi045 conmon[33655]: .311340+0000 mgr.y (mgr.14146) 86 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-07-31T01:41:40.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:40 smithi045 conmon[33655]: audit 2021-07-31T 2021-07-31T01:41:40.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:40 smithi045 conmon[33655]: 01:41:39.424501+0000 mon.a (mon.0) 278 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:41:40.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:40 smithi045 conmon[33655]: audit 2021-07-31T01:41:39 2021-07-31T01:41:40.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:40 smithi045 conmon[33655]: .425208+0000 mon.a (mon.0) 279 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-07-31T01:41:40.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:40 smithi045 conmon[33655]: 2021-07-31T01:41:40.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:40 smithi045 conmon[39718]: cluster 2021-07-31T01:41:39.311340+0000 2021-07-31T01:41:40.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:40 smithi045 conmon[39718]: mgr.y (mgr.14146) 86 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-07-31T01:41:40.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:40 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T01:41:40.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:40 smithi045 conmon[39718]: 01:41:39.424501+0000 mon.a (mon.0) 278 : audit [INF] 2021-07-31T01:41:40.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:40 smithi045 conmon[39718]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:41:40.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:40 smithi045 conmon[39718]: audit 2021-07-31T01:41:40.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:40 smithi045 conmon[39718]: 2021-07-31T01:41:39. 2021-07-31T01:41:40.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:40 smithi045 conmon[39718]: 425208+0000 mon.a (mon.0) 279 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-07-31T01:41:40.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:40 smithi045 conmon[39718]: 2021-07-31T01:41:42.161 INFO:teuthology.orchestra.run.smithi045.stdout:Created osd(s) 1 on host 'smithi045' 2021-07-31T01:41:42.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:42 smithi154 conmon[34553]: cluster 2021-07-31T01:41:41.311666+0000 mgr.y (mgr.14146 2021-07-31T01:41:42.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:42 smithi154 conmon[34553]: ) 87 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-07-31T01:41:42.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:42 smithi154 conmon[34553]: audit 2021-07-31T01:41:42.150401+0000 mon.a (mon.0) 280 : audit [DBG] 2021-07-31T01:41:42.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:42 smithi154 conmon[34553]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:41:42.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:42 smithi154 conmon[34553]: audit 2021 2021-07-31T01:41:42.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:42 smithi154 conmon[34553]: -07-31T01:41:42.151887+0000 mon.a (mon.0) 2021-07-31T01:41:42.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:42 smithi154 conmon[34553]: 281 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:41:42.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:42 smithi154 conmon[34553]: audit 2021- 2021-07-31T01:41:42.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:42 smithi154 conmon[34553]: 07-31T01:41:42.153952+0000 2021-07-31T01:41:42.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:42 smithi154 conmon[34553]: mon.a (mon.0) 282 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:41:42.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:42 smithi154 conmon[34553]: 2021-07-31T01:41:42.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:42 smithi154 conmon[34553]: audit 2021-07-31T01:41: 2021-07-31T01:41:42.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:42 smithi154 conmon[34553]: 42.155402+0000 mon.a (mon.0) 283 2021-07-31T01:41:42.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:42 smithi154 conmon[34553]: : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:41:42.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:42 smithi154 conmon[34553]: audit 2021-07 2021-07-31T01:41:42.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:42 smithi154 conmon[34553]: -31T01:41:42.158109+0000 mon.a 2021-07-31T01:41:42.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:42 smithi154 conmon[34553]: (mon.0) 284 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:41:42.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:42 smithi045 conmon[33655]: cluster 2021-07-31T01:41:41. 2021-07-31T01:41:42.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:42 smithi045 conmon[33655]: 311666+0000 mgr.y (mgr.14146) 87 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-07-31T01:41:42.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:42 smithi045 conmon[33655]: audit 2021-07-31T01:41:42.150401+0000 mon.a (mon.0) 280 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:41:42.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:42 smithi045 conmon[33655]: audit 2021-07-31T01:41:42.151887+0000 mon.a (mon.0) 281 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:41:42.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:42 smithi045 conmon[33655]: audit 2021-07-31T01:41:42.153952+0000 mon.a (mon.0) 282 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:41:42.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:42 smithi045 conmon[33655]: audit 2021-07-31T01:41:42.155402+0000 mon.a (mon.0) 283 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:41:42.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:42 smithi045 conmon[33655]: audit 2021-07-31T01:41:42.158109+0000 mon.a (mon.0) 284 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:41:42.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:42 smithi045 conmon[39718]: cluster 2021-07-31T01:41:41.311666 2021-07-31T01:41:42.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:42 smithi045 conmon[39718]: +0000 mgr.y (mgr.14146) 87 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-07-31T01:41:42.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:42 smithi045 conmon[39718]: audit 2021-07-31T01:41:42.150401+0000 mon.a (mon.0) 280 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:41:42.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:42 smithi045 conmon[39718]: audit 2021-07-31T01:41:42.151887+0000 mon.a (mon.0) 281 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:41:42.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:42 smithi045 conmon[39718]: audit 2021-07-31T01:41:42.153952+0000 mon.a (mon.0) 282 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:41:42.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:42 smithi045 conmon[39718]: audit 2021-07-31T01:41:42.155402+0000 mon.a (mon.0) 283 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:41:42.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:42 smithi045 conmon[39718]: audit 2021-07-31T01:41:42.158109+0000 mon.a (mon.0) 284 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:41:42.962 DEBUG:teuthology.orchestra.run.smithi045:osd.1> sudo journalctl -f -n 0 -u ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c@osd.1.service 2021-07-31T01:41:42.966 INFO:tasks.cephadm:Deploying osd.2 on smithi045 with /dev/vg_nvme/lv_2... 2021-07-31T01:41:42.967 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f52d2c14-f19f-11eb-8c24-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2021-07-31T01:41:42.994 INFO:journalctl@ceph.osd.1.smithi045.stdout:-- Logs begin at Sat 2021-07-31 01:16:15 UTC. -- 2021-07-31T01:41:44.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:44 smithi154 conmon[34553]: cluster 2021-07-31T01:41:43.311940+0000 mgr.y (mgr.14146) 88 2021-07-31T01:41:44.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:44 smithi154 conmon[34553]: : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-07-31T01:41:44.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:44 smithi045 conmon[33655]: cluster 2021-07-31T01:41:43.311940+0000 mgr.y (mgr.14146) 88 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-07-31T01:41:44.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:44 smithi045 conmon[39718]: cluster 2021-07-31T01:41:43.311940+0000 mgr.y (mgr.14146) 88 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-07-31T01:41:45.526 INFO:journalctl@ceph.osd.1.smithi045.stdout:Jul 31 01:41:45 smithi045 conmon[46771]: debug 2021-07-31T01:41:45.229+0000 7f41bce23080 -1 osd.1 0 log_to_monitors {default=true} 2021-07-31T01:41:45.739 INFO:teuthology.orchestra.run.smithi045.stderr:--> Zapping: /dev/vg_nvme/lv_2 2021-07-31T01:41:45.740 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2021-07-31T01:41:45.740 INFO:teuthology.orchestra.run.smithi045.stderr: stderr: 10+0 records in 2021-07-31T01:41:45.740 INFO:teuthology.orchestra.run.smithi045.stderr:10+0 records out 2021-07-31T01:41:45.741 INFO:teuthology.orchestra.run.smithi045.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00807779 s, 1.3 GB/s 2021-07-31T01:41:45.741 INFO:teuthology.orchestra.run.smithi045.stderr:--> Zapping successful for: 2021-07-31T01:41:46.197 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:46 smithi045 conmon[33655]: audit 2021-07-31T01:41:46.198 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:46 smithi045 conmon[33655]: 2021-07-31T01 2021-07-31T01:41:46.198 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:46 smithi045 conmon[33655]: :41:45.139827+0000 mon.a (mon.0) 285 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:41:46.198 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:46 smithi045 conmon[33655]: audit 2021-07 2021-07-31T01:41:46.199 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:46 smithi045 conmon[33655]: -31T01:41:45.234318+0000 mon.a (mon 2021-07-31T01:41:46.199 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:46 smithi045 conmon[33655]: .0) 286 : audit [INF] from='osd.1 [v2:172.21.15.45:6810/1747309179,v1:172.21.15.45:6811/1747309179]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2021-07-31T01:41:46.199 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:46 smithi045 conmon[33655]: cluster 2021-07-31T01:41:45 2021-07-31T01:41:46.199 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:46 smithi045 conmon[33655]: .312193+0000 mgr.y (mgr.14146) 89 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-07-31T01:41:46.200 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:46 smithi045 conmon[33655]: 2021-07-31T01:41:46.200 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:46 smithi045 conmon[39718]: audit 2021-07-31T01:41:46.200 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:46 smithi045 conmon[39718]: 2021-07-31T01:41:45 2021-07-31T01:41:46.200 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:46 smithi045 conmon[39718]: .139827+0000 mon.a (mon.0) 285 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:41:46.201 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:46 smithi045 conmon[39718]: audit 2021-07-31T01:41:45 2021-07-31T01:41:46.201 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:46 smithi045 conmon[39718]: .234318+0000 mon.a (mon.0) 286 : audit [INF] from='osd.1 [v2:172.21.15.45:6810/1747309179,v1:172.21.15.45:6811/1747309179]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2021-07-31T01:41:46.201 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:46 smithi045 conmon[39718]: cluster 2021- 2021-07-31T01:41:46.201 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:46 smithi045 conmon[39718]: 07-31T01:41:45.312193+0000 mgr.y (mgr 2021-07-31T01:41:46.202 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:46 smithi045 conmon[39718]: .14146) 89 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-07-31T01:41:46.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:46 smithi154 conmon[34553]: audit 2021-07-31T01:41:45.139827+0000 mon.a ( 2021-07-31T01:41:46.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:46 smithi154 conmon[34553]: mon.0) 285 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:41:46.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:46 smithi154 conmon[34553]: audit 2021-07-31T01:41:45.234318+0000 2021-07-31T01:41:46.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:46 smithi154 conmon[34553]: mon.a (mon.0) 286 : audit [INF] from='osd.1 [v2:172.21.15.45:6810/1747309179,v1:172.21.15.45:6811/1747309179]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2021-07-31T01:41:46.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:46 smithi154 conmon[34553]: cluster 2021-07-31T01:41:45 2021-07-31T01:41:46.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:46 smithi154 conmon[34553]: .312193+0000 mgr.y (mgr.14146) 89 : cluster [DBG] 2021-07-31T01:41:46.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:46 smithi154 conmon[34553]: pgmap v63: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-07-31T01:41:46.532 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f52d2c14-f19f-11eb-8c24-001a4aab830c -- ceph orch daemon add osd smithi045:vg_nvme/lv_2 2021-07-31T01:41:47.217 INFO:journalctl@ceph.osd.1.smithi045.stdout:Jul 31 01:41:47 smithi045 conmon[46771]: debug 2021-07-31T01:41:47.149+0000 7f41b3386700 -1 osd.1 0 waiting for initial osdmap 2021-07-31T01:41:47.218 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:47 smithi045 conmon[33655]: audit 2021-07-31T01:41:46.145323+0000 mon.a (mon.0) 287 : audit [INF] from='osd.1 [v2:172.21.15.45:6810/1747309179,v1:172.21.15.45:6811/1747309179]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2021-07-31T01:41:47.218 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:47 smithi045 conmon[33655]: cluster 2021-07-31T01:41:46.145393+0000 mon.a (mon.0) 288 : cluster [DBG] osdmap e11: 2 total, 1 up, 2 in 2021-07-31T01:41:47.219 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:47 smithi045 conmon[33655]: audit 2021-07-31T01:41:46.145624+0000 mon.a (mon.0) 289 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-07-31T01:41:47.219 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:47 smithi045 conmon[33655]: audit 2021-07-31T01:41:46.145994+0000 mon.a (mon.0) 290 : audit [INF] from='osd.1 [v2:172.21.15.45:6810/1747309179,v1:172.21.15.45:6811/1747309179]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2021-07-31T01:41:47.219 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:47 smithi045 conmon[39718]: audit 2021-07-31T01:41:46.145323+0000 mon.a (mon.0) 287 : audit [INF] from='osd.1 [v2:172.21.15.45:6810/1747309179,v1:172.21.15.45:6811/1747309179]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2021-07-31T01:41:47.219 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:47 smithi045 conmon[39718]: cluster 2021-07-31T01:41:46.145393+0000 mon.a (mon.0) 288 : cluster [DBG] osdmap e11: 2 total, 1 up, 2 in 2021-07-31T01:41:47.220 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:47 smithi045 conmon[39718]: audit 2021-07-31T01:41:46.145624+0000 mon.a (mon.0) 289 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-07-31T01:41:47.220 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:47 smithi045 conmon[39718]: audit 2021-07-31T01:41:46.145994+0000 mon.a (mon.0) 290 : audit [INF] from='osd.1 [v2:172.21.15.45:6810/1747309179,v1:172.21.15.45:6811/1747309179]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2021-07-31T01:41:47.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:47 smithi154 conmon[34553]: audit 2021-07-31T01:41:46.145323+0000 mon.a 2021-07-31T01:41:47.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:47 smithi154 conmon[34553]: (mon.0) 287 : audit [INF] from='osd.1 [v2:172.21.15.45:6810/1747309179,v1:172.21.15.45:6811/1747309179]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2021-07-31T01:41:47.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:47 smithi154 conmon[34553]: cluster 2021-07-31T01:41:46. 2021-07-31T01:41:47.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:47 smithi154 conmon[34553]: 145393+0000 mon.a (mon.0) 288 : cluster [DBG] osdmap e11: 2 total, 1 up, 2 in 2021-07-31T01:41:47.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:47 smithi154 conmon[34553]: audit 2021-07-31 2021-07-31T01:41:47.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:47 smithi154 conmon[34553]: T01:41:46.145624+0000 mon.a (mon.0) 289 : audit 2021-07-31T01:41:47.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:47 smithi154 conmon[34553]: [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-07-31T01:41:47.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:47 smithi154 conmon[34553]: audit 2021-07-31T01:41: 2021-07-31T01:41:47.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:47 smithi154 conmon[34553]: 46.145994+0000 mon.a (mon.0) 290 : audit 2021-07-31T01:41:47.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:47 smithi154 conmon[34553]: [INF] from='osd.1 [v2:172.21.15.45:6810/1747309179,v1:172.21.15.45:6811/1747309179]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2021-07-31T01:41:48.155 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:48 smithi045 conmon[39718]: audit 2021-07-31T01:41:47.146242+0000 mon.a (mon.0) 291 : audit [INF] from='osd.1 [v2:172.21.15.45:6810/1747309179,v1:172.21.15.45:6811/1747309179]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2021-07-31T01:41:48.156 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:48 smithi045 conmon[39718]: cluster 2021-07-31T01:41:47.146390+0000 mon.a (mon.0) 292 : cluster [DBG] osdmap e12: 2 total, 1 up, 2 in 2021-07-31T01:41:48.156 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:48 smithi045 conmon[39718]: audit 2021-07-31T01:41:47.146785+0000 mon.a (mon.0) 293 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-07-31T01:41:48.156 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:48 smithi045 conmon[39718]: audit 2021-07-31T01:41:47.151209+0000 mon.a (mon.0) 294 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-07-31T01:41:48.157 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:48 smithi045 conmon[39718]: 2021-07-31T01:41:48.157 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:48 smithi045 conmon[39718]: cluster 2021-07-31T01:41:47.312490+0000 mgr.y (mgr.14146) 90 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-07-31T01:41:48.419 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:48 smithi045 conmon[33655]: audit 2021-07-31T01:41:48.419 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:48 smithi045 conmon[33655]: 2021-07-31T01:41:48.420 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:48 smithi045 conmon[33655]: 2021 2021-07-31T01:41:48.420 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:48 smithi045 conmon[33655]: - 2021-07-31T01:41:48.420 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:48 smithi045 conmon[33655]: 07 2021-07-31T01:41:48.420 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:48 smithi045 conmon[33655]: - 2021-07-31T01:41:48.421 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:48 smithi045 conmon[33655]: 31T01:41:47 2021-07-31T01:41:48.421 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:48 smithi045 conmon[33655]: .146242+0000 mon.a (mon.0) 291 : audit [INF] from='osd.1 [v2:172.21.15.45:6810/1747309179,v1:172.21.15.45:6811/1747309179]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2021-07-31T01:41:48.421 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:48 smithi045 conmon[33655]: cluster 2021-07 2021-07-31T01:41:48.421 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:48 smithi045 conmon[33655]: -31T01:41:47.146390+0000 mon.a (mon.0) 292 : 2021-07-31T01:41:48.422 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:48 smithi045 conmon[33655]: cluster [DBG] osdmap e12: 2 total, 1 up, 2 in 2021-07-31T01:41:48.422 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:48 smithi045 conmon[33655]: audit 2021- 2021-07-31T01:41:48.422 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:48 smithi045 conmon[33655]: 07-31T01:41:47.146785+0000 2021-07-31T01:41:48.423 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:48 smithi045 conmon[33655]: mon.a (mon.0) 293 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-07-31T01:41:48.423 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:48 smithi045 conmon[33655]: 2021-07-31T01:41:48.423 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:48 smithi045 conmon[33655]: audit 2021- 2021-07-31T01:41:48.423 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:48 smithi045 conmon[33655]: 07-31T01:41:47.151209+0000 mon.a (mon.0 2021-07-31T01:41:48.423 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:48 smithi045 conmon[33655]: ) 294 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-07-31T01:41:48.424 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:48 smithi045 conmon[33655]: cluster 2021-07- 2021-07-31T01:41:48.424 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:48 smithi045 conmon[33655]: 31T01:41:47.312490+0000 mgr.y ( 2021-07-31T01:41:48.424 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:48 smithi045 conmon[33655]: mgr.14146) 90 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-07-31T01:41:48.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:48 smithi154 conmon[34553]: audit 2021-07-31T01:41:47.146242+0000 mon.a 2021-07-31T01:41:48.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:48 smithi154 conmon[34553]: (mon.0) 291 : audit [INF] from='osd.1 [v2:172.21.15.45:6810/1747309179,v1:172.21.15.45:6811/1747309179]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2021-07-31T01:41:48.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:48 smithi154 conmon[34553]: cluster 2021-07-31 2021-07-31T01:41:48.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:48 smithi154 conmon[34553]: T01:41:47.146390+0000 mon.a (mon.0) 292 : cluster [DBG] osdmap e12: 2 total, 1 up, 2 in 2021-07-31T01:41:48.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:48 smithi154 conmon[34553]: audit 2021-07-31T01:41:48.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:48 smithi154 conmon[34553]: 2021-07-31T01:41:47.146785+0000 mon.a ( 2021-07-31T01:41:48.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:48 smithi154 conmon[34553]: mon.0) 293 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-07-31T01:41:48.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:48 smithi154 conmon[34553]: audit 2021-07-31T01:41:47.151209+0000 mon.a (mon 2021-07-31T01:41:48.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:48 smithi154 conmon[34553]: .0) 294 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-07-31T01:41:48.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:48 smithi154 conmon[34553]: cluster 2021-07-31T01: 2021-07-31T01:41:48.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:48 smithi154 conmon[34553]: 41:47.312490+0000 mgr.y (mgr.14146) 90 2021-07-31T01:41:48.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:48 smithi154 conmon[34553]: : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-07-31T01:41:49.469 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:49 smithi045 conmon[33655]: cluster 2021-07-31T01:41:46.224317+0000 osd.1 (osd.1) 2021-07-31T01:41:49.469 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:49 smithi045 conmon[33655]: 1 : cluster [DBG] purged_snaps scrub starts 2021-07-31T01:41:49.470 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:49 smithi045 conmon[33655]: cluster 2021-07-31T01:41:46.224501+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2021-07-31T01:41:49.470 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:49 smithi045 conmon[33655]: audit 2021-07-31T01:41:48.151245+0000 mon.a (mon.0) 295 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-07-31T01:41:49.470 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:49 smithi045 conmon[33655]: cluster 2021-07-31T01:41:48.156096+0000 mon.a (mon.0) 296 : cluster [INF] osd.1 [v2:172.21.15.45:6810/1747309179,v1:172.21.15.45:6811/1747309179] boot 2021-07-31T01:41:49.470 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:49 smithi045 conmon[33655]: cluster 2021-07-31T01:41:48.156139+0000 mon.a (mon.0) 297 : cluster [DBG] osdmap e13: 2 total, 2 up, 2 in 2021-07-31T01:41:49.471 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:49 smithi045 conmon[33655]: audit 2021-07-31T01:41:48.156252+0000 mon.a (mon.0) 298 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-07-31T01:41:49.471 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:49 smithi045 conmon[33655]: audit 2021-07-31T01:41:48.712677+0000 mgr.y (mgr.14146) 91 : audit [DBG] from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:41:49.471 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:49 smithi045 conmon[33655]: audit 2021-07-31T01:41:48.713658+0000 mon.a (mon.0) 299 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-07-31T01:41:49.471 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:49 smithi045 conmon[33655]: audit 2021-07-31T01:41:48.715054+0000 mon.a (mon.0) 300 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-07-31T01:41:49.471 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:49 smithi045 conmon[33655]: audit 2021-07-31T01:41:48.715685+0000 mon.a (mon.0) 301 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:41:49.472 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:49 smithi045 conmon[39718]: cluster 2021-07-31T01:41:46.224317 2021-07-31T01:41:49.472 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:49 smithi045 conmon[39718]: +0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2021-07-31T01:41:49.472 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:49 smithi045 conmon[39718]: cluster 2021-07-31T01:41:46.224501+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2021-07-31T01:41:49.473 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:49 smithi045 conmon[39718]: audit 2021-07-31T01:41:48.151245+0000 mon.a (mon.0) 295 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-07-31T01:41:49.473 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:49 smithi045 conmon[39718]: cluster 2021-07-31T01:41:48.156096+0000 mon.a (mon.0) 296 : cluster [INF] osd.1 [v2:172.21.15.45:6810/1747309179,v1:172.21.15.45:6811/1747309179] boot 2021-07-31T01:41:49.473 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:49 smithi045 conmon[39718]: cluster 2021-07-31T01:41:48.156139+0000 mon.a (mon.0) 297 : cluster [DBG] osdmap e13: 2 total, 2 up, 2 in 2021-07-31T01:41:49.473 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:49 smithi045 conmon[39718]: audit 2021-07-31T01:41:48.156252+0000 mon.a (mon.0) 298 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-07-31T01:41:49.474 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:49 smithi045 conmon[39718]: audit 2021-07-31T01:41:48.712677+0000 mgr.y (mgr.14146) 91 : audit [DBG] 2021-07-31T01:41:49.474 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:49 smithi045 conmon[39718]: from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:41:49.474 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:49 smithi045 conmon[39718]: audit 2021-07-31T01:41:48.713658+0000 mon.a (mon.0) 299 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-07-31T01:41:49.474 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:49 smithi045 conmon[39718]: audit 2021-07 2021-07-31T01:41:49.474 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:49 smithi045 conmon[39718]: -31T01:41:48.715054+0000 mon.a (mon.0 2021-07-31T01:41:49.475 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:49 smithi045 conmon[39718]: ) 300 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-07-31T01:41:49.475 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:49 smithi045 conmon[39718]: audit 2021-07-31T01:41:49.475 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:49 smithi045 conmon[39718]: 2021-07-31T01:41:48.715685+0000 2021-07-31T01:41:49.475 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:49 smithi045 conmon[39718]: mon.a (mon.0) 301 : audit [DBG] 2021-07-31T01:41:49.476 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:49 smithi045 conmon[39718]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:41:49.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:49 smithi154 conmon[34553]: cluster 2021-07-31T01:41:46.224317 2021-07-31T01:41:49.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:49 smithi154 conmon[34553]: +0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2021-07-31T01:41:49.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:49 smithi154 conmon[34553]: cluster 2021-07-31T 2021-07-31T01:41:49.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:49 smithi154 conmon[34553]: 01:41:46.224501+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2021-07-31T01:41:49.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:49 smithi154 conmon[34553]: audit 2021-07-31T01: 2021-07-31T01:41:49.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:49 smithi154 conmon[34553]: 41:48.151245+0000 mon.a (mon.0) 295 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-07-31T01:41:49.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:49 smithi154 conmon[34553]: 2021-07-31T01:41:49.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:49 smithi154 conmon[34553]: cluster 2021-07-31T01:41:48.156096 2021-07-31T01:41:49.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:49 smithi154 conmon[34553]: +0000 mon.a (mon.0) 296 : cluster [INF] 2021-07-31T01:41:49.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:49 smithi154 conmon[34553]: osd.1 [v2:172.21.15.45:6810/1747309179,v1:172.21.15.45:6811/1747309179] boot 2021-07-31T01:41:49.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:49 smithi154 conmon[34553]: cluster 2021-07-31T01 2021-07-31T01:41:49.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:49 smithi154 conmon[34553]: :41:48.156139+0000 mon.a (mon.0) 2021-07-31T01:41:49.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:49 smithi154 conmon[34553]: 297 : cluster [DBG] osdmap e13: 2 total, 2 up, 2 in 2021-07-31T01:41:49.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:49 smithi154 conmon[34553]: audit 2021 2021-07-31T01:41:49.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:49 smithi154 conmon[34553]: -07-31T01:41:48.156252+0000 mon.a (mon 2021-07-31T01:41:49.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:49 smithi154 conmon[34553]: .0) 298 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-07-31T01:41:49.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:49 smithi154 conmon[34553]: audit 2021- 2021-07-31T01:41:49.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:49 smithi154 conmon[34553]: 07-31T01:41:48.712677+0000 mgr.y (mgr.14146) 2021-07-31T01:41:49.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:49 smithi154 conmon[34553]: 91 : audit [DBG] from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:41:49.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:49 smithi154 conmon[34553]: audit 2021 2021-07-31T01:41:49.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:49 smithi154 conmon[34553]: -07-31T01:41:48.713658+0000 mon.a ( 2021-07-31T01:41:49.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:49 smithi154 conmon[34553]: mon.0) 299 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-07-31T01:41:49.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:49 smithi154 conmon[34553]: audit 2021-07-31 2021-07-31T01:41:49.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:49 smithi154 conmon[34553]: T01:41:48.715054+0000 mon.a (mon.0) 300 : audit 2021-07-31T01:41:49.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:49 smithi154 conmon[34553]: [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-07-31T01:41:49.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:49 smithi154 conmon[34553]: audit 2021-07- 2021-07-31T01:41:49.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:49 smithi154 conmon[34553]: 31T01:41:48.715685+0000 mon.a (mon. 2021-07-31T01:41:49.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:49 smithi154 conmon[34553]: 0) 301 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:41:50.469 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[39718]: cluster 2021-07-31T01:41: 2021-07-31T01:41:50.469 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[39718]: 49.158588+0000 mon.a (mon.0) 302 : cluster [DBG] osdmap e14: 2 total, 2 up, 2 in 2021-07-31T01:41:50.470 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[39718]: cluster 2021- 2021-07-31T01:41:50.470 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[39718]: 07-31T01:41:49.312778+0000 2021-07-31T01:41:50.471 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[39718]: mgr.y (mgr.14146) 92 : cluster [DBG] 2021-07-31T01:41:50.471 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[39718]: pgmap v69: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2021-07-31T01:41:50.471 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[39718]: cephadm 2021-07-31T01:41:50.471 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[39718]: 2021-07-31T01 2021-07-31T01:41:50.472 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[39718]: :41:49.722580+0000 mgr.y (mgr 2021-07-31T01:41:50.472 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[39718]: .14146) 93 : cephadm [INF] Detected new or changed devices on smithi045 2021-07-31T01:41:50.472 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[39718]: 2021-07-31T01:41:50.472 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[39718]: audit 2021-07-31T01:41:50.473 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[39718]: 2021-07-31T01 2021-07-31T01:41:50.473 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[39718]: :41:49.727323+0000 mon.a ( 2021-07-31T01:41:50.473 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[39718]: mon.0) 303 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:41:50.473 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[39718]: 2021-07-31T01:41:50.473 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[39718]: audit 2021-07 2021-07-31T01:41:50.474 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[39718]: -31T01:41:49.728227+0000 2021-07-31T01:41:50.474 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[39718]: mon.a (mon.0) 304 : audit [INF] 2021-07-31T01:41:50.474 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[39718]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-07-31T01:41:50.474 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T01:41:50.475 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[39718]: 01:41:49.728892+0000 mon.a (mon. 2021-07-31T01:41:50.475 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[39718]: 0) 305 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-07-31T01:41:50.475 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[39718]: cephadm 2021-07-31T01:41:50.475 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[39718]: 2021-07-31T01:41:49. 2021-07-31T01:41:50.475 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[39718]: 729360+0000 mgr.y (mgr.14146) 94 : cephadm 2021-07-31T01:41:50.476 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[39718]: [INF] Adjusting osd_memory_target on smithi045 to 8076M 2021-07-31T01:41:50.476 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[39718]: audit 2021-07 2021-07-31T01:41:50.476 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[39718]: -31T01:41:49.733732+0000 2021-07-31T01:41:50.476 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[39718]: mon.a (mon.0) 306 : audit [INF] 2021-07-31T01:41:50.477 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[39718]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:41:50.477 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T01:41:50.477 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[39718]: 01:41:49.738151+0000 mon.a (mon 2021-07-31T01:41:50.477 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[39718]: .0) 307 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:41:50.477 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[39718]: audit 2021-07-31T01:41: 2021-07-31T01:41:50.478 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[39718]: 49.739765+0000 mon.a (mon.0) 308 2021-07-31T01:41:50.478 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[39718]: : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:41:50.478 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[39718]: audit 2021-07-31T01:41 2021-07-31T01:41:50.478 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[39718]: :49.740677+0000 mon.a (mon.0 2021-07-31T01:41:50.479 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[39718]: ) 309 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:41:50.479 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T01:41:50.479 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[39718]: 01:41:49.741262+0000 mon.a (mon 2021-07-31T01:41:50.480 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[39718]: .0) 310 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:41:50.480 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[39718]: audit 2021-07 2021-07-31T01:41:50.480 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[39718]: -31T01:41:49.745524+0000 2021-07-31T01:41:50.480 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[39718]: mon.a (mon.0) 311 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:41:50.480 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[39718]: 2021-07-31T01:41:50.481 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[33655]: cluster 2021-07-31T01:41:49.158588 2021-07-31T01:41:50.481 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[33655]: +0000 mon.a (mon.0) 302 : cluster [DBG] osdmap e14: 2 total, 2 up, 2 in 2021-07-31T01:41:50.481 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[33655]: cluster 2021-07-31T01:41:49.312778+0000 mgr.y (mgr.14146) 92 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2021-07-31T01:41:50.481 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[33655]: cephadm 2021-07-31T01 2021-07-31T01:41:50.482 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[33655]: :41:49.722580+0000 mgr.y (mgr.14146) 93 : cephadm [INF] 2021-07-31T01:41:50.482 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[33655]: Detected new or changed devices on smithi045 2021-07-31T01:41:50.482 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[33655]: audit 2021 2021-07-31T01:41:50.482 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[33655]: -07-31T01:41:49.727323+0000 mon.a (mon 2021-07-31T01:41:50.483 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[33655]: .0) 303 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:41:50.483 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[33655]: audit 2021-07-31T01 2021-07-31T01:41:50.483 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[33655]: :41:49.728227+0000 mon.a (mon 2021-07-31T01:41:50.483 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[33655]: .0) 304 : audit [INF] 2021-07-31T01:41:50.484 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[33655]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-07-31T01:41:50.484 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[33655]: audit 2021-07-31T01:41:50.484 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[33655]: 2021-07-31T01: 2021-07-31T01:41:50.485 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[33655]: 41:49.728892+0000 2021-07-31T01:41:50.485 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[33655]: mon.a (mon.0) 305 2021-07-31T01:41:50.486 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[33655]: : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-07-31T01:41:50.486 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[33655]: cephadm 2021-07-31T01:41:49.729360+0000 mgr.y (mgr.14146) 94 : cephadm [INF] Adjusting osd_memory_target on smithi045 to 8076M 2021-07-31T01:41:50.486 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[33655]: audit 2021- 2021-07-31T01:41:50.487 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[33655]: 07-31T01:41: 2021-07-31T01:41:50.487 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[33655]: 49.733732+0000 mon.a ( 2021-07-31T01:41:50.487 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[33655]: mon.0) 306 : audit 2021-07-31T01:41:50.488 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[33655]: [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:41:50.488 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[33655]: audit 2021-07-31T01:41:50.488 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[33655]: 2021-07-31T01: 2021-07-31T01:41:50.489 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[33655]: 41:49.738151+0000 mon.a 2021-07-31T01:41:50.489 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[33655]: (mon.0) 307 2021-07-31T01:41:50.490 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[33655]: : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:41:50.490 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[33655]: audit 2021 2021-07-31T01:41:50.491 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[33655]: -07-31T01:41:49 2021-07-31T01:41:50.491 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[33655]: .739765+0000 mon.a (mon.0) 2021-07-31T01:41:50.491 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[33655]: 308 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:41:50.492 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[33655]: audit 2021- 2021-07-31T01:41:50.492 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[33655]: 07-31T01:41:49.740677+0000 mon.a ( 2021-07-31T01:41:50.493 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[33655]: mon.0) 309 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:41:50.493 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[33655]: audit 2021 2021-07-31T01:41:50.493 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[33655]: -07-31T01: 2021-07-31T01:41:50.494 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[33655]: 41:49.741262+0000 2021-07-31T01:41:50.494 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[33655]: mon.a (mon.0) 2021-07-31T01:41:50.494 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[33655]: 310 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:41:50.494 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[33655]: audit 2021 2021-07-31T01:41:50.495 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[33655]: -07-31T01:41:49 2021-07-31T01:41:50.495 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[33655]: .745524+0000 mon.a (mon 2021-07-31T01:41:50.495 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[33655]: .0) 311 : audit 2021-07-31T01:41:50.495 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:50 smithi045 conmon[33655]: [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:41:50.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:50 smithi154 conmon[34553]: cluster 2021-07-31T01:41:49.158588+0000 mon.a ( 2021-07-31T01:41:50.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:50 smithi154 conmon[34553]: mon.0) 302 : cluster [DBG] osdmap e14: 2 total, 2 up, 2 in 2021-07-31T01:41:50.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:50 smithi154 conmon[34553]: cluster 2021-07-31T01:41:49.312778+0000 2021-07-31T01:41:50.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:50 smithi154 conmon[34553]: mgr.y (mgr.14146) 92 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2021-07-31T01:41:50.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:50 smithi154 conmon[34553]: cephadm 2021-07-31T01:41 2021-07-31T01:41:50.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:50 smithi154 conmon[34553]: :49.722580+0000 mgr.y (mgr.14146) 93 : 2021-07-31T01:41:50.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:50 smithi154 conmon[34553]: cephadm [INF] Detected new or changed devices on smithi045 2021-07-31T01:41:50.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:50 smithi154 conmon[34553]: audit 2021- 2021-07-31T01:41:50.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:50 smithi154 conmon[34553]: 07-31T01:41:49.727323+0000 mon.a (mon. 2021-07-31T01:41:50.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:50 smithi154 conmon[34553]: 0) 303 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:41:50.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:50 smithi154 conmon[34553]: audit 2021-07-31T01:41:50.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:50 smithi154 conmon[34553]: 2021-07-31T01:41:49.728227+0000 mon.a ( 2021-07-31T01:41:50.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:50 smithi154 conmon[34553]: mon.0) 304 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-07-31T01:41:50.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:50 smithi154 conmon[34553]: audit 2021-07-31T01:41:49.728892 2021-07-31T01:41:50.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:50 smithi154 conmon[34553]: +0000 mon.a (mon.0) 305 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-07-31T01:41:50.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:50 smithi154 conmon[34553]: cephadm 2021-07-31T01:41:50.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:50 smithi154 conmon[34553]: 2021-07-31T01:41:49.729360+0000 mgr.y 2021-07-31T01:41:50.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:50 smithi154 conmon[34553]: (mgr.14146) 94 : cephadm [INF] Adjusting osd_memory_target on smithi045 to 8076M 2021-07-31T01:41:50.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:50 smithi154 conmon[34553]: audit 2021-07 2021-07-31T01:41:50.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:50 smithi154 conmon[34553]: -31T01:41:49.733732+0000 mon.a (mon.0) 306 : audit 2021-07-31T01:41:50.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:50 smithi154 conmon[34553]: [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:41:50.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:50 smithi154 conmon[34553]: audit 2021-07 2021-07-31T01:41:50.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:50 smithi154 conmon[34553]: -31T01:41:49.738151+0000 mon.a (mon 2021-07-31T01:41:50.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:50 smithi154 conmon[34553]: .0) 307 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:41:50.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:50 smithi154 conmon[34553]: audit 2021-07-31T01 2021-07-31T01:41:50.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:50 smithi154 conmon[34553]: :41:49.739765+0000 mon.a (mon.0) 308 : audit [DBG] 2021-07-31T01:41:50.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:50 smithi154 conmon[34553]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:41:50.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:50 smithi154 conmon[34553]: audit 2021-07-31T01: 2021-07-31T01:41:50.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:50 smithi154 conmon[34553]: 41:49.740677+0000 mon.a (mon.0) 2021-07-31T01:41:50.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:50 smithi154 conmon[34553]: 309 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:41:50.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:50 smithi154 conmon[34553]: audit 2021-07- 2021-07-31T01:41:50.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:50 smithi154 conmon[34553]: 31T01:41:49.741262+0000 mon.a (mon. 2021-07-31T01:41:50.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:50 smithi154 conmon[34553]: 0) 310 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:41:50.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:50 smithi154 conmon[34553]: audit 2021-07-31T01:41:50.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:50 smithi154 conmon[34553]: 2021-07-31T01:41:49.745524+0000 mon.a (mon 2021-07-31T01:41:50.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:50 smithi154 conmon[34553]: .0) 311 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:41:52.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:52 smithi154 conmon[34553]: cluster 2021-07-31T01:41:51.313043+0000 mgr.y (mgr.14146) 95 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2021-07-31T01:41:52.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:52 smithi154 conmon[34553]: audit 2021-07-31 2021-07-31T01:41:52.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:52 smithi154 conmon[34553]: T01:41:51.492661+0000 mon.a (mon.0) 312 : audit [INF] from='client.? 172.21.15.45:0/2913310728' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bfbe641e-b9d7-4005-8fb5-bc7d5af76388"}]: dispatch 2021-07-31T01:41:52.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:52 smithi154 conmon[34553]: audit 2021-07-31T01:41:51.496270+0000 mon.a (mon.0) 313 : audit [INF] from='client.? 172.21.15.45:0/2913310728' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bfbe641e-b9d7-4005-8fb5-bc7d5af76388"}]': finished 2021-07-31T01:41:52.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:52 smithi154 conmon[34553]: cluster 2021-07-31T01:41:52.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:52 smithi154 conmon[34553]: 2021-07-31T01:41:51 2021-07-31T01:41:52.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:52 smithi154 conmon[34553]: .496318+0000 mon.a (mon.0) 2021-07-31T01:41:52.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:52 smithi154 conmon[34553]: 314 : cluster [DBG] osdmap e15: 3 total, 2 up, 3 in 2021-07-31T01:41:52.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:52 smithi154 conmon[34553]: audit 2021- 2021-07-31T01:41:52.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:52 smithi154 conmon[34553]: 07-31T01:41:51.496438+0000 mon.a ( 2021-07-31T01:41:52.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:52 smithi154 conmon[34553]: mon.0) 315 : audit [DBG] 2021-07-31T01:41:52.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:52 smithi154 conmon[34553]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-07-31T01:41:52.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:52 smithi154 conmon[34553]: audit 2021- 2021-07-31T01:41:52.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:52 smithi154 conmon[34553]: 07-31T01:41:51.998702 2021-07-31T01:41:52.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:52 smithi154 conmon[34553]: +0000 mon.a (mon.0) 316 2021-07-31T01:41:52.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:52 smithi154 conmon[34553]: : audit [DBG] from='client.? 172.21.15.45:0/768979908' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-07-31T01:41:52.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:52 smithi045 conmon[33655]: cluster 2021-07-31T01:41:51.313043+0000 mgr.y (mgr.14146) 95 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2021-07-31T01:41:52.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:52 smithi045 conmon[33655]: audit 2021-07-31T01:41:51.492661+0000 mon.a (mon.0) 312 : audit [INF] from='client.? 172.21.15.45:0/2913310728' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bfbe641e-b9d7-4005-8fb5-bc7d5af76388"}]: dispatch 2021-07-31T01:41:52.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:52 smithi045 conmon[33655]: audit 2021-07-31T01:41:51.496270+0000 mon.a (mon.0) 313 : audit [INF] from='client.? 172.21.15.45:0/2913310728' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bfbe641e-b9d7-4005-8fb5-bc7d5af76388"}]': finished 2021-07-31T01:41:52.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:52 smithi045 conmon[33655]: cluster 2021-07-31T01:41:51.496318+0000 mon.a (mon.0) 314 : cluster [DBG] osdmap e15: 3 total, 2 up, 3 in 2021-07-31T01:41:52.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:52 smithi045 conmon[33655]: audit 2021-07-31T01:41:51.496438+0000 mon.a (mon.0) 315 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-07-31T01:41:52.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:52 smithi045 conmon[33655]: audit 2021-07-31T01:41:51.998702+0000 mon.a (mon.0) 316 : audit [DBG] from='client.? 172.21.15.45:0/768979908' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-07-31T01:41:52.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:52 smithi045 conmon[39718]: cluster 2021-07-31T01:41:51.313043+0000 mgr.y (mgr.14146) 95 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2021-07-31T01:41:52.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:52 smithi045 conmon[39718]: audit 2021-07-31T01:41:51.492661+0000 mon.a (mon.0) 312 : audit [INF] from='client.? 172.21.15.45:0/2913310728' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bfbe641e-b9d7-4005-8fb5-bc7d5af76388"}]: dispatch 2021-07-31T01:41:52.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:52 smithi045 conmon[39718]: audit 2021-07-31T01:41:51.496270+0000 mon.a (mon.0) 313 : audit [INF] from='client.? 172.21.15.45:0/2913310728' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bfbe641e-b9d7-4005-8fb5-bc7d5af76388"}]': finished 2021-07-31T01:41:52.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:52 smithi045 conmon[39718]: cluster 2021-07-31T01:41:51.496318+0000 mon.a (mon.0) 314 : cluster [DBG] osdmap e15: 3 total, 2 up, 3 in 2021-07-31T01:41:52.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:52 smithi045 conmon[39718]: audit 2021-07-31T01:41:51.496438 2021-07-31T01:41:52.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:52 smithi045 conmon[39718]: +0000 mon.a (mon.0) 315 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-07-31T01:41:52.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:52 smithi045 conmon[39718]: audit 2021-07-31T01:41:51.998702+0000 mon.a (mon.0) 316 : audit [DBG] from='client.? 172.21.15.45:0/768979908' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-07-31T01:41:54.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:54 smithi045 conmon[33655]: cluster 2021-07-31T01:41:53.313523+0000 2021-07-31T01:41:54.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:54 smithi045 conmon[33655]: mgr.y (mgr.14146) 96 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2021-07-31T01:41:54.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:54 smithi045 conmon[39718]: cluster 2021-07-31T01:41:53.313523 2021-07-31T01:41:54.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:54 smithi045 conmon[39718]: +0000 mgr.y (mgr.14146) 96 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2021-07-31T01:41:54.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:54 smithi154 conmon[34553]: cluster 2021-07-31T01:41:53.313523+0000 mgr.y (mgr.14146) 96 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2021-07-31T01:41:56.752 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:56 smithi045 conmon[39718]: cluster 2021-07-31T01:41:55.313830+0000 mgr.y ( 2021-07-31T01:41:56.752 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:56 smithi045 conmon[39718]: mgr.14146) 97 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-07-31T01:41:56.753 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:56 smithi045 conmon[33655]: cluster 2021-07-31T01:41:55. 2021-07-31T01:41:56.753 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:56 smithi045 conmon[33655]: 313830+0000 mgr.y (mgr.14146) 97 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-07-31T01:41:56.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:56 smithi154 conmon[34553]: cluster 2021-07-31T01:41:55.313830+0000 mgr.y (mgr.14146) 97 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-07-31T01:41:58.701 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:58 smithi045 conmon[39718]: cluster 2021-07-31T01:41:57.314091+0000 mgr.y (mgr.14146) 98 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-07-31T01:41:58.702 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:58 smithi045 conmon[39718]: audit 2021-07-31T01:41:58.078285+0000 mon.a (mon.0) 317 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2021-07-31T01:41:58.702 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:58 smithi045 conmon[39718]: audit 2021-07-31T01:41:58.078988+0000 mon.a (mon.0) 318 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:41:58.703 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:58 smithi045 conmon[33655]: cluster 2021-07-31T01:41:57.314091+0000 mgr.y (mgr.14146) 98 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-07-31T01:41:58.703 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:58 smithi045 conmon[33655]: audit 2021-07-31T01:41:58.078285+0000 mon.a (mon.0) 317 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2021-07-31T01:41:58.704 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:58 smithi045 conmon[33655]: audit 2021-07-31T01:41:58.078988+0000 mon.a (mon.0) 318 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:41:58.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:58 smithi154 conmon[34553]: cluster 2021-07-31T01:41:57.314091+0000 mgr.y (mgr.14146) 98 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-07-31T01:41:58.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:58 smithi154 conmon[34553]: audit 2021-07-31T01:41:58.078285+0000 mon.a (mon. 2021-07-31T01:41:58.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:58 smithi154 conmon[34553]: 0) 317 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2021-07-31T01:41:58.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:58 smithi154 conmon[34553]: audit 2021-07-31T01:41:58.078988+0000 mon.a (mon.0) 318 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:41:59.701 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:59 smithi045 conmon[39718]: cephadm 2021-07-31T01:41:59.701 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:59 smithi045 conmon[39718]: 2021-07-31T01:41: 2021-07-31T01:41:59.701 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:41:59 smithi045 conmon[39718]: 58.079582+0000 mgr.y (mgr.14146) 99 : cephadm [INF] Deploying daemon osd.2 on smithi045 2021-07-31T01:41:59.702 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:59 smithi045 conmon[33655]: cephadm 2021-07-31T01:41: 2021-07-31T01:41:59.702 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:41:59 smithi045 conmon[33655]: 58.079582+0000 mgr.y (mgr.14146) 99 : cephadm [INF] Deploying daemon osd.2 on smithi045 2021-07-31T01:41:59.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:41:59 smithi154 conmon[34553]: cephadm 2021-07-31T01:41:58.079582+0000 mgr.y (mgr.14146) 99 : cephadm [INF] Deploying daemon osd.2 on smithi045 2021-07-31T01:42:00.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:00 smithi045 conmon[33655]: cluster 2021-07-31T01:41:59 2021-07-31T01:42:00.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:00 smithi045 conmon[33655]: .314417+0000 mgr.y (mgr.14146) 100 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-07-31T01:42:00.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:00 smithi045 conmon[39718]: cluster 2021-07-31T01:41:59.314417 2021-07-31T01:42:00.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:00 smithi045 conmon[39718]: +0000 mgr.y (mgr.14146) 100 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-07-31T01:42:00.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:00 smithi154 conmon[34553]: cluster 2021-07-31T01:41:59.314417+0000 mgr.y (mgr.14146) 100 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-07-31T01:42:02.165 INFO:teuthology.orchestra.run.smithi045.stdout:Created osd(s) 2 on host 'smithi045' 2021-07-31T01:42:02.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:02 smithi154 conmon[34553]: cluster 2021-07-31T01:42:01.314724+0000 mgr.y (mgr.14146) 101 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-07-31T01:42:02.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:02 smithi154 conmon[34553]: audit 2021-07-31T01:42:02. 2021-07-31T01:42:02.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:02 smithi154 conmon[34553]: 156006+0000 mon.a (mon.0) 319 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:42:02.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:02 smithi154 conmon[34553]: audit 2021-07-31T01:42:02.157566+0000 mon.a (mon.0) 320 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:42:02.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:02 smithi154 conmon[34553]: audit 2021-07-31T01: 2021-07-31T01:42:02.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:02 smithi154 conmon[34553]: 42:02.159840+0000 mon.a (mon 2021-07-31T01:42:02.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:02 smithi154 conmon[34553]: .0) 321 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:42:02.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:02 smithi154 conmon[34553]: 2021-07-31T01:42:02.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:02 smithi154 conmon[34553]: audit 2021-07- 2021-07-31T01:42:02.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:02 smithi154 conmon[34553]: 31T01:42:02.161437+0000 mon.a ( 2021-07-31T01:42:02.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:02 smithi154 conmon[34553]: mon.0) 322 : audit [INF] 2021-07-31T01:42:02.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:02 smithi154 conmon[34553]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:42:02.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:02 smithi154 conmon[34553]: audit 2021-07-31T01:42:02.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:02 smithi154 conmon[34553]: 2021-07-31T01:42 2021-07-31T01:42:02.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:02 smithi154 conmon[34553]: :02.163559+0000 mon.a (mon 2021-07-31T01:42:02.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:02 smithi154 conmon[34553]: .0) 323 : audit [INF] 2021-07-31T01:42:02.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:02 smithi154 conmon[34553]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:42:02.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:02 smithi045 conmon[33655]: cluster 2021-07-31T01:42:01. 2021-07-31T01:42:02.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:02 smithi045 conmon[33655]: 314724+0000 mgr.y (mgr.14146) 101 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-07-31T01:42:02.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:02 smithi045 conmon[33655]: audit 2021-07-31T01:42:02.156006+0000 mon.a (mon.0) 319 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:42:02.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:02 smithi045 conmon[33655]: audit 2021-07-31T01:42:02.157566+0000 mon.a (mon.0) 320 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:42:02.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:02 smithi045 conmon[33655]: audit 2021-07-31T01:42:02.159840+0000 mon.a (mon.0) 321 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:42:02.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:02 smithi045 conmon[33655]: audit 2021-07-31T01:42:02.161437+0000 mon.a (mon.0) 322 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:42:02.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:02 smithi045 conmon[33655]: audit 2021-07-31T01:42:02.163559+0000 mon.a (mon.0) 323 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:42:02.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:02 smithi045 conmon[39718]: cluster 2021-07-31T01:42:01. 2021-07-31T01:42:02.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:02 smithi045 conmon[39718]: 314724+0000 mgr.y (mgr.14146) 101 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-07-31T01:42:02.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:02 smithi045 conmon[39718]: audit 2021-07-31T01:42:02.156006+0000 mon.a (mon.0) 319 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:42:02.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:02 smithi045 conmon[39718]: audit 2021-07-31T01:42:02.157566+0000 mon.a (mon.0) 320 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:42:02.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:02 smithi045 conmon[39718]: audit 2021-07-31T01:42:02.159840+0000 mon.a (mon.0) 321 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:42:02.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:02 smithi045 conmon[39718]: audit 2021-07-31T01:42:02.161437+0000 mon.a (mon.0) 322 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:42:02.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:02 smithi045 conmon[39718]: audit 2021-07-31T01:42:02.163559+0000 mon.a (mon.0) 323 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:42:03.034 DEBUG:teuthology.orchestra.run.smithi045:osd.2> sudo journalctl -f -n 0 -u ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c@osd.2.service 2021-07-31T01:42:03.037 INFO:tasks.cephadm:Deploying osd.3 on smithi045 with /dev/vg_nvme/lv_1... 2021-07-31T01:42:03.037 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f52d2c14-f19f-11eb-8c24-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2021-07-31T01:42:03.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:-- Logs begin at Sat 2021-07-31 01:16:15 UTC. -- 2021-07-31T01:42:04.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:04 smithi045 conmon[33655]: cluster 2021-07-31T01:42:03.315005+0000 mgr.y (mgr.14146) 102 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-07-31T01:42:04.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:04 smithi045 conmon[39718]: cluster 2021-07-31T01:42:03.315005+0000 mgr.y (mgr.14146) 102 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-07-31T01:42:04.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:04 smithi154 conmon[34553]: cluster 2021-07-31T01:42:03.315005+0000 mgr.y (mgr.14146) 102 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-07-31T01:42:05.526 INFO:journalctl@ceph.osd.2.smithi045.stdout:Jul 31 01:42:05 smithi045 conmon[50337]: debug 2021-07-31T01:42:05.249+0000 7f9609907080 -1 osd.2 0 log_to_monitors {default=true} 2021-07-31T01:42:05.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:05 smithi045 conmon[33655]: audit 2021-07-31T01:42:05.253836+0000 2021-07-31T01:42:05.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:05 smithi045 conmon[33655]: mon.a (mon.0) 324 : audit [INF] from='osd.2 [v2:172.21.15.45:6818/2518799988,v1:172.21.15.45:6819/2518799988]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2021-07-31T01:42:05.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:05 smithi045 conmon[33655]: audit 2021-07-31T01:42:05.311280+0000 mon.a (mon.0) 325 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:42:05.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:05 smithi045 conmon[39718]: audit 2021-07-31T01:42:05 2021-07-31T01:42:05.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:05 smithi045 conmon[39718]: .253836+0000 mon.a (mon.0) 324 : audit [INF] from='osd.2 [v2:172.21.15.45:6818/2518799988,v1:172.21.15.45:6819/2518799988]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2021-07-31T01:42:05.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:05 smithi045 conmon[39718]: audit 2021-07-31T01:42:05.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:05 smithi045 conmon[39718]: 2021-07-31T01 2021-07-31T01:42:05.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:05 smithi045 conmon[39718]: :42:05.311280+0000 mon.a (mon.0) 325 : 2021-07-31T01:42:05.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:05 smithi045 conmon[39718]: audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:42:05.744 INFO:teuthology.orchestra.run.smithi045.stderr:--> Zapping: /dev/vg_nvme/lv_1 2021-07-31T01:42:05.744 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2021-07-31T01:42:05.744 INFO:teuthology.orchestra.run.smithi045.stderr: stderr: 10+0 records in 2021-07-31T01:42:05.745 INFO:teuthology.orchestra.run.smithi045.stderr:10+0 records out 2021-07-31T01:42:05.745 INFO:teuthology.orchestra.run.smithi045.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00830916 s, 1.3 GB/s 2021-07-31T01:42:05.745 INFO:teuthology.orchestra.run.smithi045.stderr:--> Zapping successful for: 2021-07-31T01:42:05.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:05 smithi154 conmon[34553]: audit 2021-07-31T01:42:05.253836+0000 mon.a (mon.0) 324 : audit [INF] from='osd.2 [v2:172.21.15.45:6818/2518799988,v1:172.21.15.45:6819/2518799988]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2021-07-31T01:42:05.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:05 smithi154 conmon[34553]: audit 2021-07-31T01: 2021-07-31T01:42:05.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:05 smithi154 conmon[34553]: 42:05.311280+0000 mon.a (mon.0) 325 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:42:06.593 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f52d2c14-f19f-11eb-8c24-001a4aab830c -- ceph orch daemon add osd smithi045:vg_nvme/lv_1 2021-07-31T01:42:06.734 INFO:journalctl@ceph.osd.2.smithi045.stdout:Jul 31 01:42:06 smithi045 conmon[50337]: debug 2021-07-31T01:42:06.442+0000 7f95ffe6a700 -1 osd.2 0 waiting for initial osdmap 2021-07-31T01:42:06.735 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:06 smithi045 conmon[33655]: cluster 2021-07-31T01:42:05.315263 2021-07-31T01:42:06.735 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:06 smithi045 conmon[33655]: +0000 mgr.y (mgr.14146) 103 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-07-31T01:42:06.736 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:06 smithi045 conmon[33655]: audit 2021-07-31T01:42:05.439095 2021-07-31T01:42:06.736 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:06 smithi045 conmon[33655]: +0000 mon.a (mon.0) 326 : audit [INF] from='osd.2 [v2:172.21.15.45:6818/2518799988,v1:172.21.15.45:6819/2518799988]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2021-07-31T01:42:06.737 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:06 smithi045 conmon[33655]: 2021-07-31T01:42:06.737 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:06 smithi045 conmon[33655]: cluster 2021-07-31T01:42:06.737 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:06 smithi045 conmon[33655]: 2021-07 2021-07-31T01:42:06.737 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:06 smithi045 conmon[33655]: -31T01:42:05 2021-07-31T01:42:06.738 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:06 smithi045 conmon[33655]: .439182+0000 mon.a (mon 2021-07-31T01:42:06.738 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:06 smithi045 conmon[33655]: .0) 327 : cluster [DBG] 2021-07-31T01:42:06.738 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:06 smithi045 conmon[33655]: osdmap e16: 3 total, 2 up, 3 in 2021-07-31T01:42:06.738 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:06 smithi045 conmon[33655]: audit 2021 2021-07-31T01:42:06.738 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:06 smithi045 conmon[33655]: -07-31T01:42:05. 2021-07-31T01:42:06.739 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:06 smithi045 conmon[33655]: 439331+0000 mon.a (mon.0 2021-07-31T01:42:06.739 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:06 smithi045 conmon[33655]: ) 328 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-07-31T01:42:06.739 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:06 smithi045 conmon[33655]: 2021-07-31T01:42:06.739 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:06 smithi045 conmon[33655]: audit 2021-07 2021-07-31T01:42:06.740 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:06 smithi045 conmon[33655]: -31T01:42:05.439822 2021-07-31T01:42:06.740 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:06 smithi045 conmon[33655]: +0000 mon.a (mon.0) 329 2021-07-31T01:42:06.740 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:06 smithi045 conmon[33655]: : audit [INF] from='osd.2 [v2:172.21.15.45:6818/2518799988,v1:172.21.15.45:6819/2518799988]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2021-07-31T01:42:06.740 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:06 smithi045 conmon[39718]: cluster 2021-07-31T01:42:05. 2021-07-31T01:42:06.741 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:06 smithi045 conmon[39718]: 315263+0000 mgr.y (mgr.14146) 103 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-07-31T01:42:06.741 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:06 smithi045 conmon[39718]: audit 2021-07-31T01: 2021-07-31T01:42:06.741 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:06 smithi045 conmon[39718]: 42:05.439095+0000 mon.a (mon.0 2021-07-31T01:42:06.741 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:06 smithi045 conmon[39718]: ) 326 : audit [INF] from='osd.2 [v2:172.21.15.45:6818/2518799988,v1:172.21.15.45:6819/2518799988]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2021-07-31T01:42:06.742 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:06 smithi045 conmon[39718]: cluster 2021-07-31T01:42:06.742 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:06 smithi045 conmon[39718]: 2021-07-31T01:42:05. 2021-07-31T01:42:06.742 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:06 smithi045 conmon[39718]: 439182+0000 mon.a (mon.0) 327 : cluster [DBG] 2021-07-31T01:42:06.742 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:06 smithi045 conmon[39718]: osdmap e16: 3 total, 2 up, 3 in 2021-07-31T01:42:06.743 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:06 smithi045 conmon[39718]: audit 2021-07-31T01:42:05 2021-07-31T01:42:06.743 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:06 smithi045 conmon[39718]: .439331+0000 mon.a (mon.0) 328 2021-07-31T01:42:06.743 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:06 smithi045 conmon[39718]: : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-07-31T01:42:06.744 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:06 smithi045 conmon[39718]: audit 2021-07 2021-07-31T01:42:06.744 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:06 smithi045 conmon[39718]: -31T01:42:05. 2021-07-31T01:42:06.744 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:06 smithi045 conmon[39718]: 439822+0000 mon.a (mon. 2021-07-31T01:42:06.744 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:06 smithi045 conmon[39718]: 0) 329 : audit [INF] 2021-07-31T01:42:06.744 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:06 smithi045 conmon[39718]: from='osd.2 [v2:172.21.15.45:6818/2518799988,v1:172.21.15.45:6819/2518799988]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2021-07-31T01:42:06.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:06 smithi154 conmon[34553]: cluster 2021-07-31T01:42:05.315263+0000 mgr.y (mgr.14146) 103 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-07-31T01:42:06.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:06 smithi154 conmon[34553]: audit 2021-07-31T01:42:05.439095+0000 mon.a (mon.0) 326 : audit [INF] from='osd.2 [v2:172.21.15.45:6818/2518799988,v1:172.21.15.45:6819/2518799988]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2021-07-31T01:42:06.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:06 smithi154 conmon[34553]: cluster 2021-07-31T01:42:06.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:06 smithi154 conmon[34553]: 2021-07-31T01:42:05.439182+0000 mon.a ( 2021-07-31T01:42:06.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:06 smithi154 conmon[34553]: mon.0) 327 : cluster [DBG] osdmap e16: 3 total, 2 up, 3 in 2021-07-31T01:42:06.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:06 smithi154 conmon[34553]: 2021-07-31T01:42:06.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:06 smithi154 conmon[34553]: audit 2021- 2021-07-31T01:42:06.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:06 smithi154 conmon[34553]: 07-31T01:42:05.439331+0000 2021-07-31T01:42:06.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:06 smithi154 conmon[34553]: mon.a (mon.0) 328 : audit 2021-07-31T01:42:06.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:06 smithi154 conmon[34553]: [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-07-31T01:42:06.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:06 smithi154 conmon[34553]: audit 2021- 2021-07-31T01:42:06.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:06 smithi154 conmon[34553]: 07-31T01:42:05.439822 2021-07-31T01:42:06.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:06 smithi154 conmon[34553]: +0000 mon.a (mon.0) 329 : 2021-07-31T01:42:06.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:06 smithi154 conmon[34553]: audit [INF] from='osd.2 [v2:172.21.15.45:6818/2518799988,v1:172.21.15.45:6819/2518799988]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2021-07-31T01:42:08.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:07 smithi154 conmon[34553]: audit 2021-07-31T01:42:06.440381+0000 mon.a (mon.0) 330 : audit [INF] from='osd.2 [v2:172.21.15.45:6818/2518799988,v1:172.21.15.45:6819/2518799988]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2021-07-31T01:42:08.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:07 smithi154 conmon[34553]: cluster 2021-07-31T01:42:06.440458+0000 mon.a (mon.0) 331 : cluster [DBG] osdmap e17: 3 total, 2 up, 3 in 2021-07-31T01:42:08.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:07 smithi154 conmon[34553]: audit 2021-07-31T01:42:06.440926+0000 mon.a (mon.0) 332 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-07-31T01:42:08.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:07 smithi154 conmon[34553]: audit 2021-07-31T01:42:06.444147+0000 mon.a (mon.0) 333 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-07-31T01:42:08.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:07 smithi154 conmon[34553]: audit 2021-07-31T01:42:07.443883+0000 mon.a (mon.0) 334 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-07-31T01:42:08.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:07 smithi045 conmon[39718]: audit 2021-07-31T01:42:06.440381+0000 mon.a (mon.0) 330 : audit [INF] from='osd.2 [v2:172.21.15.45:6818/2518799988,v1:172.21.15.45:6819/2518799988]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2021-07-31T01:42:08.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:07 smithi045 conmon[39718]: cluster 2021-07-31T01:42:06.440458+0000 mon.a (mon.0) 331 : cluster [DBG] osdmap e17: 3 total, 2 up, 3 in 2021-07-31T01:42:08.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:07 smithi045 conmon[39718]: audit 2021-07-31T01:42:06.440926+0000 mon.a (mon.0) 332 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-07-31T01:42:08.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:07 smithi045 conmon[39718]: audit 2021-07-31T01:42:06.444147+0000 mon.a (mon.0) 333 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-07-31T01:42:08.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:07 smithi045 conmon[39718]: audit 2021-07-31T01:42:07.443883+0000 mon.a (mon.0) 334 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-07-31T01:42:08.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:07 smithi045 conmon[33655]: audit 2021-07-31T01:42:06.440381+0000 mon.a (mon.0) 330 : audit [INF] from='osd.2 [v2:172.21.15.45:6818/2518799988,v1:172.21.15.45:6819/2518799988]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2021-07-31T01:42:08.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:07 smithi045 conmon[33655]: cluster 2021-07-31T01:42:06.440458+0000 mon.a (mon.0) 331 : cluster [DBG] osdmap e17: 3 total, 2 up, 3 in 2021-07-31T01:42:08.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:07 smithi045 conmon[33655]: audit 2021-07-31T01:42:06.440926+0000 mon.a (mon.0) 332 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-07-31T01:42:08.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:07 smithi045 conmon[33655]: audit 2021-07-31T01:42:06.444147+0000 mon.a (mon.0) 333 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-07-31T01:42:08.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:07 smithi045 conmon[33655]: audit 2021-07-31T01:42:07.443883+0000 mon.a (mon.0) 334 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-07-31T01:42:09.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:08 smithi045 conmon[39718]: cluster 2021-07-31T01:42:06.245955+0000 osd.2 (osd.2) 1 : cluster [DBG] purged_snaps scrub starts 2021-07-31T01:42:09.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:08 smithi045 conmon[39718]: cluster 2021-07-31T01:42:06.246089+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2021-07-31T01:42:09.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:08 smithi045 conmon[39718]: cluster 2021-07-31T01:42:07.315491+0000 mgr.y (mgr.14146) 104 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-07-31T01:42:09.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:08 smithi045 conmon[39718]: cluster 2021-07-31T01:42:07.449260+0000 mon.a (mon.0) 335 : cluster [INF] osd.2 [v2:172.21.15.45:6818/2518799988,v1:172.21.15.45:6819/2518799988] boot 2021-07-31T01:42:09.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:08 smithi045 conmon[39718]: cluster 2021-07-31T01:42:07.449296+0000 mon.a (mon.0) 336 : cluster [DBG] 2021-07-31T01:42:09.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:08 smithi045 conmon[39718]: osdmap e18: 3 total, 3 up, 3 in 2021-07-31T01:42:09.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:08 smithi045 conmon[39718]: audit 2021-07-31T01:42:07.449742+0000 mon.a (mon.0) 337 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-07-31T01:42:09.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:08 smithi045 conmon[39718]: audit 2021-07-31T01:42:07.450071+0000 mon.a (mon.0) 338 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]: dispatch 2021-07-31T01:42:09.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:08 smithi045 conmon[39718]: audit 2021-07-31T01:42:08.450847+0000 mon.a (mon.0) 339 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]': finished 2021-07-31T01:42:09.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:08 smithi045 conmon[39718]: cluster 2021-07-31T01:42:08.450935+0000 mon.a (mon.0) 340 : cluster [DBG] osdmap e19: 3 total, 3 up, 3 in 2021-07-31T01:42:09.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:08 smithi045 conmon[39718]: audit 2021-07-31T01:42:08.452650+0000 mon.a (mon.0) 341 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2021-07-31T01:42:09.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:08 smithi045 conmon[33655]: cluster 2021-07-31T01:42:06.245955+0000 osd.2 (osd.2) 1 : cluster [DBG] purged_snaps scrub starts 2021-07-31T01:42:09.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:08 smithi045 conmon[33655]: cluster 2021-07-31T01:42:06.246089+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2021-07-31T01:42:09.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:08 smithi045 conmon[33655]: cluster 2021-07-31T01:42:07.315491+0000 mgr.y (mgr.14146) 104 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-07-31T01:42:09.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:08 smithi045 conmon[33655]: cluster 2021-07-31T01:42:07.449260+0000 mon.a (mon.0) 335 : cluster [INF] osd.2 [v2:172.21.15.45:6818/2518799988,v1:172.21.15.45:6819/2518799988] boot 2021-07-31T01:42:09.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:08 smithi045 conmon[33655]: cluster 2021-07-31T01:42:07.449296+0000 mon.a (mon.0) 336 : cluster [DBG] osdmap e18: 3 total, 3 up, 3 in 2021-07-31T01:42:09.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:08 smithi045 conmon[33655]: audit 2021-07-31T01:42:07.449742+0000 mon.a (mon.0) 337 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-07-31T01:42:09.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:08 smithi045 conmon[33655]: audit 2021-07-31T01:42:07.450071+0000 mon.a (mon.0) 338 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]: dispatch 2021-07-31T01:42:09.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:08 smithi045 conmon[33655]: audit 2021-07-31T01:42:08.450847+0000 mon.a (mon.0) 339 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]': finished 2021-07-31T01:42:09.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:08 smithi045 conmon[33655]: cluster 2021-07-31T01:42:08.450935+0000 mon.a (mon. 2021-07-31T01:42:09.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:08 smithi045 conmon[33655]: 0) 340 : cluster [DBG] osdmap e19: 3 total, 3 up, 3 in 2021-07-31T01:42:09.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:08 smithi045 conmon[33655]: audit 2021-07-31T01:42:08.452650+0000 mon.a (mon.0) 341 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2021-07-31T01:42:09.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:08 smithi154 conmon[34553]: cluster 2021-07-31T01:42:06.245955+0000 osd.2 (osd.2) 1 : cluster [DBG] purged_snaps scrub starts 2021-07-31T01:42:09.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:08 smithi154 conmon[34553]: cluster 2021-07-31T01:42:06.246089+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2021-07-31T01:42:09.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:08 smithi154 conmon[34553]: cluster 2021-07-31T 2021-07-31T01:42:09.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:08 smithi154 conmon[34553]: 01:42:07.315491+0000 mgr.y 2021-07-31T01:42:09.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:08 smithi154 conmon[34553]: (mgr.14146) 104 : cluster [DBG] 2021-07-31T01:42:09.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:08 smithi154 conmon[34553]: pgmap v81: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-07-31T01:42:09.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:08 smithi154 conmon[34553]: cluster 2021- 2021-07-31T01:42:09.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:08 smithi154 conmon[34553]: 07-31T01:42:07.449260+0000 2021-07-31T01:42:09.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:08 smithi154 conmon[34553]: mon.a (mon.0) 335 : cluster 2021-07-31T01:42:09.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:08 smithi154 conmon[34553]: [INF] osd.2 [v2:172.21.15.45:6818/2518799988,v1:172.21.15.45:6819/2518799988] boot 2021-07-31T01:42:09.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:08 smithi154 conmon[34553]: cluster 2021- 2021-07-31T01:42:09.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:08 smithi154 conmon[34553]: 07-31T01:42:07.449296 2021-07-31T01:42:09.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:08 smithi154 conmon[34553]: +0000 mon.a (mon.0) 336 2021-07-31T01:42:09.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:08 smithi154 conmon[34553]: : cluster [DBG] osdmap e18: 3 total, 3 up, 3 in 2021-07-31T01:42:09.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:08 smithi154 conmon[34553]: audit 2021 2021-07-31T01:42:09.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:08 smithi154 conmon[34553]: -07-31T01:42:07.449742 2021-07-31T01:42:09.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:08 smithi154 conmon[34553]: +0000 mon.a (mon.0) 2021-07-31T01:42:09.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:08 smithi154 conmon[34553]: 337 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-07-31T01:42:09.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:08 smithi154 conmon[34553]: audit 2021- 2021-07-31T01:42:09.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:08 smithi154 conmon[34553]: 07-31T01:42:07.450071 2021-07-31T01:42:09.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:08 smithi154 conmon[34553]: +0000 mon.a (mon.0) 338 2021-07-31T01:42:09.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:08 smithi154 conmon[34553]: : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]: dispatch 2021-07-31T01:42:09.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:08 smithi154 conmon[34553]: audit 2021 2021-07-31T01:42:09.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:08 smithi154 conmon[34553]: -07-31T01:42:08. 2021-07-31T01:42:09.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:08 smithi154 conmon[34553]: 450847+0000 mon.a (mon.0 2021-07-31T01:42:09.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:08 smithi154 conmon[34553]: ) 339 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]': finished 2021-07-31T01:42:09.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:08 smithi154 conmon[34553]: cluster 2021- 2021-07-31T01:42:09.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:08 smithi154 conmon[34553]: 07-31T01:42:08.450935+0000 2021-07-31T01:42:09.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:08 smithi154 conmon[34553]: mon.a (mon.0) 340 : cluster 2021-07-31T01:42:09.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:08 smithi154 conmon[34553]: [DBG] osdmap e19: 3 total, 3 up, 3 in 2021-07-31T01:42:09.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:08 smithi154 conmon[34553]: audit 2021 2021-07-31T01:42:09.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:08 smithi154 conmon[34553]: -07-31T01:42:08. 2021-07-31T01:42:09.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:08 smithi154 conmon[34553]: 452650+0000 mon.a (mon.0 2021-07-31T01:42:09.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:08 smithi154 conmon[34553]: ) 341 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2021-07-31T01:42:10.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:10 smithi154 conmon[34553]: cluster 2021-07-31T01:42:09.315768+0000 mgr.y (mgr.14146) 105 : cluster [DBG] pgmap v84: 1 pgs: 1 creating+peering; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2021-07-31T01:42:10.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:10 smithi154 conmon[34553]: audit 2021-07-31T01:42:09.452400+0000 mon.a (mon.0) 342 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2021-07-31T01:42:10.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:10 smithi154 conmon[34553]: cluster 2021-07-31T01:42:10.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:10 smithi154 conmon[34553]: 2021-07-31T01:42:09.452478+0000 mon.a ( 2021-07-31T01:42:10.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:10 smithi154 conmon[34553]: mon.0) 343 : cluster [DBG] osdmap e20: 3 total, 3 up, 3 in 2021-07-31T01:42:10.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:10 smithi154 conmon[34553]: 2021-07-31T01:42:10.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:10 smithi154 conmon[34553]: audit 2021-07 2021-07-31T01:42:10.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:10 smithi154 conmon[34553]: -31T01:42:09.505259+0000 mgr.y 2021-07-31T01:42:10.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:10 smithi154 conmon[34553]: (mgr.14146) 106 : audit [DBG] 2021-07-31T01:42:10.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:10 smithi154 conmon[34553]: from='client.24202 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:42:10.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:10 smithi154 conmon[34553]: audit 2021- 2021-07-31T01:42:10.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:10 smithi154 conmon[34553]: 07-31T01:42:09.506025 2021-07-31T01:42:10.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:10 smithi154 conmon[34553]: +0000 mon.a (mon.0) 2021-07-31T01:42:10.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:10 smithi154 conmon[34553]: 344 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-07-31T01:42:10.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:10 smithi154 conmon[34553]: audit 2021 2021-07-31T01:42:10.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:10 smithi154 conmon[34553]: -07-31T01:42:09.507294+0000 mon.a 2021-07-31T01:42:10.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:10 smithi154 conmon[34553]: (mon.0) 345 : audit [INF] 2021-07-31T01:42:10.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:10 smithi154 conmon[34553]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-07-31T01:42:10.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:10 smithi154 conmon[34553]: audit 2021- 2021-07-31T01:42:10.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:10 smithi154 conmon[34553]: 07-31T01:42:09.507908 2021-07-31T01:42:10.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:10 smithi154 conmon[34553]: +0000 mon.a (mon.0) 346 2021-07-31T01:42:10.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:10 smithi154 conmon[34553]: : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:42:10.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:10 smithi154 conmon[34553]: cephadm 2021 2021-07-31T01:42:10.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:10 smithi154 conmon[34553]: -07-31T01:42:09. 2021-07-31T01:42:10.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:10 smithi154 conmon[34553]: 692369+0000 mgr.y (mgr.14146) 2021-07-31T01:42:10.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:10 smithi154 conmon[34553]: 107 : cephadm [INF] Detected new or changed devices on smithi045 2021-07-31T01:42:10.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:10 smithi154 conmon[34553]: audit 2021 2021-07-31T01:42:10.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:10 smithi154 conmon[34553]: -07-31T01:42:09.698003 2021-07-31T01:42:10.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:10 smithi154 conmon[34553]: +0000 mon.a (mon.0) 2021-07-31T01:42:10.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:10 smithi154 conmon[34553]: 347 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:42:10.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:10 smithi154 conmon[34553]: audit 2021- 2021-07-31T01:42:10.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:10 smithi154 conmon[34553]: 07-31T01:42:10.090912 2021-07-31T01:42:10.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:10 smithi154 conmon[34553]: +0000 mon.a (mon.0) 348 2021-07-31T01:42:10.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:10 smithi154 conmon[34553]: : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-07-31T01:42:10.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:10 smithi154 conmon[34553]: audit 2021 2021-07-31T01:42:10.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:10 smithi154 conmon[34553]: -07-31T01:42:10.091551+0000 mon.a (mon. 2021-07-31T01:42:10.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:10 smithi154 conmon[34553]: 0) 349 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-07-31T01:42:10.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:10 smithi154 conmon[34553]: 2021-07-31T01:42:10.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:10 smithi154 conmon[34553]: audit 2021- 2021-07-31T01:42:10.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:10 smithi154 conmon[34553]: 07-31T01:42:10.092059 2021-07-31T01:42:10.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:10 smithi154 conmon[34553]: +0000 mon.a (mon.0) 350 2021-07-31T01:42:10.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:10 smithi154 conmon[34553]: : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-07-31T01:42:10.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:10 smithi154 conmon[34553]: audit 2021- 2021-07-31T01:42:10.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:10 smithi154 conmon[34553]: 07-31T01:42:10.097264 2021-07-31T01:42:10.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:10 smithi154 conmon[34553]: +0000 mon.a (mon.0) 351 2021-07-31T01:42:10.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:10 smithi154 conmon[34553]: : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:42:10.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:10 smithi154 conmon[34553]: audit 2021- 2021-07-31T01:42:10.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:10 smithi154 conmon[34553]: 07-31T01:42:10. 2021-07-31T01:42:10.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:10 smithi154 conmon[34553]: 101376+0000 mon.a (mon.0) 2021-07-31T01:42:10.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:10 smithi154 conmon[34553]: 352 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:42:10.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:10 smithi154 conmon[34553]: audit 2021 2021-07-31T01:42:10.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:10 smithi154 conmon[34553]: -07-31T01:42:10. 2021-07-31T01:42:10.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:10 smithi154 conmon[34553]: 102778+0000 mon.a (mon.0) 2021-07-31T01:42:10.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:10 smithi154 conmon[34553]: 353 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:42:10.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:10 smithi154 conmon[34553]: 2021-07-31T01:42:10.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:10 smithi154 conmon[34553]: audit 2021- 2021-07-31T01:42:10.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:10 smithi154 conmon[34553]: 07-31T01:42:10.103697 2021-07-31T01:42:10.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:10 smithi154 conmon[34553]: +0000 mon.a (mon.0) 354 2021-07-31T01:42:10.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:10 smithi154 conmon[34553]: : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:42:10.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:10 smithi154 conmon[34553]: audit 2021 2021-07-31T01:42:10.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:10 smithi154 conmon[34553]: -07-31T01:42:10. 2021-07-31T01:42:10.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:10 smithi154 conmon[34553]: 104306+0000 mon.a (mon.0) 2021-07-31T01:42:10.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:10 smithi154 conmon[34553]: 355 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:42:10.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:10 smithi154 conmon[34553]: audit 2021 2021-07-31T01:42:10.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:10 smithi154 conmon[34553]: -07-31T01:42:10. 2021-07-31T01:42:10.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:10 smithi154 conmon[34553]: 107865+0000 mon.a (mon.0 2021-07-31T01:42:10.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:10 smithi154 conmon[34553]: ) 356 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:42:10.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:10 smithi045 conmon[39718]: cluster 2021-07-31T01:42:09.315768+0000 mgr.y (mgr. 2021-07-31T01:42:10.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:10 smithi045 conmon[39718]: 14146) 105 : cluster [DBG] pgmap v84: 1 pgs: 1 creating+peering; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2021-07-31T01:42:10.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:10 smithi045 conmon[39718]: audit 2021-07-31T01:42:09.452400+0000 mon.a (mon.0) 342 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2021-07-31T01:42:10.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:10 smithi045 conmon[39718]: cluster 2021-07-31T01:42:09.452478+0000 mon.a (mon.0) 343 : cluster [DBG] osdmap e20: 3 total, 3 up, 3 in 2021-07-31T01:42:10.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:10 smithi045 conmon[39718]: audit 2021-07-31T01:42:09.505259+0000 mgr.y (mgr.14146) 106 : audit [DBG] from='client.24202 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:42:10.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:10 smithi045 conmon[39718]: audit 2021-07-31T01:42:09.506025+0000 mon.a (mon.0) 344 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-07-31T01:42:10.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:10 smithi045 conmon[39718]: audit 2021-07-31T01:42:09.507294+0000 mon.a (mon.0) 345 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-07-31T01:42:10.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:10 smithi045 conmon[39718]: audit 2021-07-31T01:42:09.507908+0000 mon.a (mon.0) 346 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:42:10.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:10 smithi045 conmon[39718]: cephadm 2021-07-31T01:42:09.692369+0000 mgr.y (mgr.14146) 107 : cephadm [INF] Detected new or changed devices on smithi045 2021-07-31T01:42:10.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:10 smithi045 conmon[39718]: audit 2021-07-31T01:42:09.698003+0000 mon.a (mon.0) 347 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:42:10.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:10 smithi045 conmon[39718]: audit 2021-07-31T01:42:10.090912+0000 mon.a (mon 2021-07-31T01:42:10.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:10 smithi045 conmon[39718]: .0) 348 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-07-31T01:42:10.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:10 smithi045 conmon[39718]: audit 2021-07-31T01:42:10.091551+0000 mon.a (mon.0) 349 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-07-31T01:42:10.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:10 smithi045 conmon[39718]: audit 2021-07-31T01:42:10.092059+0000 mon.a (mon.0) 350 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-07-31T01:42:10.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:10 smithi045 conmon[39718]: audit 2021-07-31T01:42:10.097264+0000 2021-07-31T01:42:10.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:10 smithi045 conmon[39718]: mon.a (mon.0) 351 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:42:10.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:10 smithi045 conmon[39718]: audit 2021-07-31T01:42 2021-07-31T01:42:10.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:10 smithi045 conmon[39718]: :10.101376+0000 mon.a (mon.0) 352 : audit 2021-07-31T01:42:10.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:10 smithi045 conmon[39718]: [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:42:10.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:10 smithi045 conmon[39718]: audit 2021-07-31T01: 2021-07-31T01:42:10.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:10 smithi045 conmon[39718]: 42:10.102778+0000 mon.a (mon.0) 353 : 2021-07-31T01:42:10.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:10 smithi045 conmon[39718]: audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:42:10.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:10 smithi045 conmon[39718]: audit 2021-07-31T01:42 2021-07-31T01:42:10.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:10 smithi045 conmon[39718]: :10.103697+0000 mon.a (mon.0) 354 : audit [DBG] 2021-07-31T01:42:10.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:10 smithi045 conmon[39718]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:42:10.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:10 smithi045 conmon[39718]: audit 2021-07-31T01:42:10.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:10 smithi045 conmon[39718]: 2021-07-31T01:42:10.104306 2021-07-31T01:42:10.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:10 smithi045 conmon[39718]: +0000 mon.a (mon.0) 355 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:42:10.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:10 smithi045 conmon[39718]: audit 2021 2021-07-31T01:42:10.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:10 smithi045 conmon[39718]: -07-31T01:42:10.107865+0000 mon.a 2021-07-31T01:42:10.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:10 smithi045 conmon[39718]: (mon.0) 356 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:42:10.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:10 smithi045 conmon[33655]: cluster 2021-07-31T01:42:09.315768+0000 mgr.y 2021-07-31T01:42:10.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:10 smithi045 conmon[33655]: (mgr.14146) 105 : cluster [DBG] pgmap v84: 1 pgs: 1 creating+peering; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2021-07-31T01:42:10.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:10 smithi045 conmon[33655]: audit 2021-07-31T01:42:09.452400+0000 mon.a (mon.0) 342 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2021-07-31T01:42:10.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:10 smithi045 conmon[33655]: cluster 2021-07-31T01:42:09.452478+0000 mon.a (mon.0) 343 : cluster [DBG] osdmap e20: 3 total, 3 up, 3 in 2021-07-31T01:42:10.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:10 smithi045 conmon[33655]: audit 2021-07-31T01:42:09.505259+0000 mgr.y (mgr.14146) 106 : audit [DBG] from='client.24202 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:42:10.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:10 smithi045 conmon[33655]: audit 2021-07-31T01:42:09.506025+0000 mon.a (mon.0) 344 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-07-31T01:42:10.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:10 smithi045 conmon[33655]: audit 2021-07-31T01:42:09.507294+0000 mon.a (mon.0) 345 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-07-31T01:42:10.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:10 smithi045 conmon[33655]: audit 2021-07-31T01:42:09.507908+0000 mon.a (mon.0) 346 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:42:10.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:10 smithi045 conmon[33655]: cephadm 2021-07-31T01:42:09.692369+0000 mgr.y (mgr.14146) 107 : cephadm [INF] Detected new or changed devices on smithi045 2021-07-31T01:42:10.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:10 smithi045 conmon[33655]: audit 2021-07-31T01:42:09.698003+0000 mon.a (mon.0) 347 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:42:10.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:10 smithi045 conmon[33655]: audit 2021-07-31T01:42:10.090912+0000 mon.a (mon.0) 348 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-07-31T01:42:10.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:10 smithi045 conmon[33655]: audit 2021-07-31T01:42:10.091551+0000 mon.a (mon.0) 349 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-07-31T01:42:10.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:10 smithi045 conmon[33655]: audit 2021-07-31T01:42:10.092059+0000 mon.a (mon.0) 350 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-07-31T01:42:10.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:10 smithi045 conmon[33655]: audit 2021-07-31T01:42: 2021-07-31T01:42:10.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:10 smithi045 conmon[33655]: 10.097264+0000 mon.a (mon.0) 351 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:42:10.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:10 smithi045 conmon[33655]: audit 2021-07-31T01:42:10.101376+0000 mon.a (mon.0) 352 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:42:10.788 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:10 smithi045 conmon[33655]: audit 2021-07-31T01: 2021-07-31T01:42:10.788 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:10 smithi045 conmon[33655]: 42:10.102778+0000 mon.a (mon. 2021-07-31T01:42:10.788 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:10 smithi045 conmon[33655]: 0) 353 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:42:10.788 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:10 smithi045 conmon[33655]: 2021-07-31T01:42:10.788 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:10 smithi045 conmon[33655]: audit 2021- 2021-07-31T01:42:10.789 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:10 smithi045 conmon[33655]: 07-31T01:42:10. 2021-07-31T01:42:10.789 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:10 smithi045 conmon[33655]: 103697+0000 mon.a (mon.0) 354 2021-07-31T01:42:10.789 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:10 smithi045 conmon[33655]: : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:42:10.789 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:10 smithi045 conmon[33655]: audit 2021-07-31T01:42:10.789 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:10 smithi045 conmon[33655]: 2021-07-31T01:42: 2021-07-31T01:42:10.790 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:10 smithi045 conmon[33655]: 10.104306+0000 mon.a (mon. 2021-07-31T01:42:10.790 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:10 smithi045 conmon[33655]: 0) 355 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:42:10.790 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:10 smithi045 conmon[33655]: 2021-07-31T01:42:10.790 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:10 smithi045 conmon[33655]: audit 2021-07- 2021-07-31T01:42:10.791 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:10 smithi045 conmon[33655]: 31T01:42:10.107865 2021-07-31T01:42:10.791 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:10 smithi045 conmon[33655]: +0000 mon.a (mon.0) 356 2021-07-31T01:42:10.791 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:10 smithi045 conmon[33655]: : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:42:11.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:11 smithi154 conmon[34553]: cephadm 2021-07-31T01:42:10.092539+0000 mgr.y (mgr.14146) 108 : cephadm [INF] Adjusting osd_memory_target on smithi045 to 5384M 2021-07-31T01:42:11.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:11 smithi154 conmon[34553]: cluster 2021-07-31T01:42:10.456916+0000 mon.a (mon.0) 357 : cluster [DBG] osdmap e21: 3 total, 3 up, 3 in 2021-07-31T01:42:11.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:11 smithi045 conmon[33655]: cephadm 2021-07-31T01:42:10.092539+0000 mgr.y (mgr.14146) 108 : cephadm [INF] Adjusting osd_memory_target on smithi045 to 5384M 2021-07-31T01:42:11.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:11 smithi045 conmon[33655]: cluster 2021-07-31T01:42:10.456916+0000 mon.a (mon.0) 357 : cluster [DBG] osdmap e21: 3 total, 3 up, 3 in 2021-07-31T01:42:11.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:11 smithi045 conmon[39718]: cephadm 2021-07-31T01:42:10.092539+0000 mgr.y (mgr.14146) 108 : cephadm [INF] Adjusting osd_memory_target on smithi045 to 5384M 2021-07-31T01:42:11.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:11 smithi045 conmon[39718]: cluster 2021-07-31T01:42:10.456916+0000 mon.a (mon.0) 357 : cluster [DBG] osdmap e21: 3 total, 3 up, 3 in 2021-07-31T01:42:12.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:12 smithi154 conmon[34553]: cluster 2021-07-31T01:42:11.316040+0000 mgr.y (mgr.14146) 109 : cluster [DBG] pgmap v87: 1 pgs: 1 creating+peering; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2021-07-31T01:42:12.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:12 smithi154 conmon[34553]: audit 2021-07-31T01:42:12.041766+0000 mon.c (mon.1) 6 : audit [INF] from='client.? 172.21.15.45:0/2091064407' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "37b826a3-b611-4001-b9a7-1966032e56d6"}]: dispatch 2021-07-31T01:42:12.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:12 smithi154 conmon[34553]: audit 2021-07-31T01:42: 2021-07-31T01:42:12.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:12 smithi154 conmon[34553]: 12.042097+0000 mon.a (mon.0) 2021-07-31T01:42:12.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:12 smithi154 conmon[34553]: 358 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "37b826a3-b611-4001-b9a7-1966032e56d6"}]: dispatch 2021-07-31T01:42:12.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:12 smithi154 conmon[34553]: audit 2021- 2021-07-31T01:42:12.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:12 smithi154 conmon[34553]: 07-31T01:42:12.045770+0000 2021-07-31T01:42:12.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:12 smithi154 conmon[34553]: mon.a (mon.0) 359 : audit 2021-07-31T01:42:12.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:12 smithi154 conmon[34553]: [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "37b826a3-b611-4001-b9a7-1966032e56d6"}]': finished 2021-07-31T01:42:12.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:12 smithi154 conmon[34553]: cluster 2021- 2021-07-31T01:42:12.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:12 smithi154 conmon[34553]: 07-31T01:42:12.045823+0000 2021-07-31T01:42:12.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:12 smithi154 conmon[34553]: mon.a (mon.0) 360 : 2021-07-31T01:42:12.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:12 smithi154 conmon[34553]: cluster [DBG] osdmap e22: 4 total, 3 up, 4 in 2021-07-31T01:42:12.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:12 smithi154 conmon[34553]: audit 2021 2021-07-31T01:42:12.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:12 smithi154 conmon[34553]: -07-31T01:42:12. 2021-07-31T01:42:12.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:12 smithi154 conmon[34553]: 045922+0000 mon.a (mon.0) 2021-07-31T01:42:12.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:12 smithi154 conmon[34553]: 361 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-07-31T01:42:12.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:12 smithi045 conmon[33655]: cluster 2021-07-31T01:42:11.316040+0000 mgr.y (mgr.14146) 109 2021-07-31T01:42:12.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:12 smithi045 conmon[33655]: : cluster [DBG] pgmap v87: 1 pgs: 1 creating+peering; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2021-07-31T01:42:12.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:12 smithi045 conmon[33655]: audit 2021-07-31T01:42:12.041766+0000 mon.c (mon.1) 6 : audit [INF] from='client.? 172.21.15.45:0/2091064407' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "37b826a3-b611-4001-b9a7-1966032e56d6"}]: dispatch 2021-07-31T01:42:12.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:12 smithi045 conmon[33655]: audit 2021-07-31T01:42:12.042097+0000 mon.a (mon.0) 358 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "37b826a3-b611-4001-b9a7-1966032e56d6"}]: dispatch 2021-07-31T01:42:12.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:12 smithi045 conmon[33655]: audit 2021-07-31T01:42:12.045770+0000 mon.a (mon.0) 359 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "37b826a3-b611-4001-b9a7-1966032e56d6"}]': finished 2021-07-31T01:42:12.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:12 smithi045 conmon[33655]: cluster 2021-07-31T01:42:12.045823+0000 mon.a (mon.0) 360 : cluster [DBG] osdmap e22: 4 total, 3 up, 4 in 2021-07-31T01:42:12.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:12 smithi045 conmon[33655]: audit 2021-07-31T01:42:12.045922+0000 mon.a (mon.0) 361 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-07-31T01:42:12.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:12 smithi045 conmon[39718]: cluster 2021-07-31T01:42:11.316040+0000 mgr.y (mgr 2021-07-31T01:42:12.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:12 smithi045 conmon[39718]: .14146) 109 : cluster [DBG] pgmap v87: 1 pgs: 1 creating+peering; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2021-07-31T01:42:12.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:12 smithi045 conmon[39718]: audit 2021-07-31T01 2021-07-31T01:42:12.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:12 smithi045 conmon[39718]: :42:12.041766+0000 mon.c (mon.1) 6 : audit 2021-07-31T01:42:12.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:12 smithi045 conmon[39718]: [INF] from='client.? 172.21.15.45:0/2091064407' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "37b826a3-b611-4001-b9a7-1966032e56d6"}]: dispatch 2021-07-31T01:42:12.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:12 smithi045 conmon[39718]: audit 2021- 2021-07-31T01:42:12.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:12 smithi045 conmon[39718]: 07-31T01:42:12.042097 2021-07-31T01:42:12.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:12 smithi045 conmon[39718]: +0000 mon.a (mon.0) 358 : audit 2021-07-31T01:42:12.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:12 smithi045 conmon[39718]: [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "37b826a3-b611-4001-b9a7-1966032e56d6"}]: dispatch 2021-07-31T01:42:12.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:12 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T01:42:12.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:12 smithi045 conmon[39718]: 31T01:42:12.045770+0000 mon.a ( 2021-07-31T01:42:12.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:12 smithi045 conmon[39718]: mon.0) 359 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "37b826a3-b611-4001-b9a7-1966032e56d6"}]': finished 2021-07-31T01:42:12.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:12 smithi045 conmon[39718]: cluster 2021-07-31T01 2021-07-31T01:42:12.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:12 smithi045 conmon[39718]: :42:12.045823+0000 mon.a (mon.0) 360 : cluster 2021-07-31T01:42:12.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:12 smithi045 conmon[39718]: [DBG] osdmap e22: 4 total, 3 up, 4 in 2021-07-31T01:42:12.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:12 smithi045 conmon[39718]: audit 2021 2021-07-31T01:42:12.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:12 smithi045 conmon[39718]: -07-31T01:42:12.045922+0000 mon.a (mon.0) 361 : audit 2021-07-31T01:42:12.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:12 smithi045 conmon[39718]: [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-07-31T01:42:13.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:13 smithi154 conmon[34553]: audit 2021-07-31T01:42:12.599325+0000 mon.c (mon.1) 7 : audit [DBG] from='client.? 172.21.15.45:0/2234973343' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-07-31T01:42:13.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:13 smithi045 conmon[33655]: audit 2021-07-31T01:42:12. 2021-07-31T01:42:13.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:13 smithi045 conmon[33655]: 599325+0000 mon.c (mon.1) 7 : audit [DBG] from='client.? 172.21.15.45:0/2234973343' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-07-31T01:42:13.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:13 smithi045 conmon[39718]: audit 2021-07-31T01: 2021-07-31T01:42:13.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:13 smithi045 conmon[39718]: 42:12.599325+0000 mon.c (mon.1) 7 : audit [DBG] from='client.? 172.21.15.45:0/2234973343' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-07-31T01:42:14.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:14 smithi154 conmon[34553]: cluster 2021-07-31T01:42:13.316451+0000 mgr.y (mgr.14146) 110 : cluster 2021-07-31T01:42:14.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:14 smithi154 conmon[34553]: [DBG] pgmap v89: 1 pgs: 1 creating+peering; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2021-07-31T01:42:14.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:14 smithi045 conmon[33655]: cluster 2021-07-31T01:42:13.316451+0000 2021-07-31T01:42:14.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:14 smithi045 conmon[33655]: mgr.y (mgr.14146) 110 : cluster [DBG] pgmap v89: 1 pgs: 1 creating+peering; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2021-07-31T01:42:14.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:14 smithi045 conmon[39718]: cluster 2021-07-31T01:42:13.316451+0000 2021-07-31T01:42:14.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:14 smithi045 conmon[39718]: mgr.y (mgr.14146) 110 : cluster [DBG] pgmap v89: 1 pgs: 1 creating+peering; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2021-07-31T01:42:16.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:16 smithi154 conmon[34553]: cluster 2021-07-31T01:42:15.316789+0000 mgr.y (mgr.14146) 111 : cluster [DBG] pgmap v90: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-07-31T01:42:16.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:16 smithi045 conmon[33655]: cluster 2021-07-31T01:42:15.316789+0000 mgr.y (mgr.14146) 111 : cluster [DBG] pgmap v90: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-07-31T01:42:16.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:16 smithi045 conmon[39718]: cluster 2021-07-31T01:42:15.316789+0000 mgr.y (mgr.14146) 111 : cluster [DBG] pgmap v90: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-07-31T01:42:18.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:18 smithi154 conmon[34553]: cluster 2021-07-31T01:42:17.317195+0000 mgr.y (mgr.14146) 112 : cluster [DBG] pgmap v91: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-07-31T01:42:18.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:18 smithi045 conmon[39718]: cluster 2021-07-31T01:42:17.317195+0000 mgr.y (mgr 2021-07-31T01:42:18.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:18 smithi045 conmon[39718]: .14146) 112 : cluster [DBG] pgmap v91: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-07-31T01:42:18.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:18 smithi045 conmon[33655]: cluster 2021-07-31T01:42:17.317195 2021-07-31T01:42:18.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:18 smithi045 conmon[33655]: +0000 mgr.y (mgr.14146) 112 : cluster [DBG] pgmap v91: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-07-31T01:42:19.740 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:19 smithi045 conmon[39718]: audit 2021-07-31T01:42:18.655385+0000 2021-07-31T01:42:19.741 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:19 smithi045 conmon[39718]: mon.a (mon.0) 362 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2021-07-31T01:42:19.741 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:19 smithi045 conmon[39718]: audit 2021-07-31T01:42:18.656055+0000 mon.a (mon.0) 363 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:42:19.741 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:19 smithi045 conmon[39718]: cephadm 2021-07-31T01:42:18.656630+0000 mgr.y (mgr.14146) 113 : cephadm [INF] Deploying daemon osd.3 on smithi045 2021-07-31T01:42:19.742 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:19 smithi045 conmon[33655]: audit 2021-07-31T01:42:18. 2021-07-31T01:42:19.742 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:19 smithi045 conmon[33655]: 655385+0000 mon.a (mon.0) 362 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2021-07-31T01:42:19.742 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:19 smithi045 conmon[33655]: audit 2021-07-31T01:42:18.656055+0000 mon.a (mon.0) 363 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:42:19.742 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:19 smithi045 conmon[33655]: cephadm 2021-07-31T01:42:18.656630+0000 mgr.y (mgr.14146) 113 : cephadm [INF] Deploying daemon osd.3 on smithi045 2021-07-31T01:42:19.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:19 smithi154 conmon[34553]: audit 2021-07-31T01:42:18.655385+0000 mon.a (mon.0) 362 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2021-07-31T01:42:19.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:19 smithi154 conmon[34553]: audit 2021-07-31T01:42:18.656055+0000 mon.a (mon.0) 363 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:42:19.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:19 smithi154 conmon[34553]: cephadm 2021-07-31T01:42:19.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:19 smithi154 conmon[34553]: 2021-07-31T01:42:18.656630+0000 mgr.y (mgr 2021-07-31T01:42:19.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:19 smithi154 conmon[34553]: .14146) 113 : cephadm [INF] Deploying daemon osd.3 on smithi045 2021-07-31T01:42:19.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:19 smithi154 conmon[34553]: 2021-07-31T01:42:20.751 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:20 smithi045 conmon[33655]: cluster 2021-07-31T01:42:19.317483+0000 mgr.y (mgr.14146) 114 : cluster [DBG] pgmap v92: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-07-31T01:42:20.752 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:20 smithi045 conmon[39718]: cluster 2021-07-31T01:42 2021-07-31T01:42:20.752 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:20 smithi045 conmon[39718]: :19.317483+0000 mgr.y (mgr.14146) 114 : cluster [DBG] pgmap v92: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-07-31T01:42:20.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:20 smithi154 conmon[34553]: cluster 2021-07-31T01:42:19.317483+0000 mgr.y (mgr.14146) 114 : cluster [DBG] pgmap v92: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-07-31T01:42:22.535 INFO:teuthology.orchestra.run.smithi045.stdout:Created osd(s) 3 on host 'smithi045' 2021-07-31T01:42:22.733 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:22 smithi045 conmon[39718]: cluster 2021-07-31T01:42:21.317755+0000 2021-07-31T01:42:22.733 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:22 smithi045 conmon[39718]: mgr.y (mgr.14146) 115 : cluster [DBG] pgmap v93: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-07-31T01:42:22.733 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:22 smithi045 conmon[33655]: cluster 2021-07-31T01:42:21.317755+0000 mgr.y 2021-07-31T01:42:22.734 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:22 smithi045 conmon[33655]: (mgr.14146) 115 : cluster [DBG] pgmap v93: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-07-31T01:42:22.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:22 smithi154 conmon[34553]: cluster 2021-07-31T01:42:21.317755+0000 mgr.y (mgr.14146) 115 : cluster [DBG] pgmap v93: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-07-31T01:42:23.275 DEBUG:teuthology.orchestra.run.smithi045:osd.3> sudo journalctl -f -n 0 -u ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c@osd.3.service 2021-07-31T01:42:23.278 INFO:tasks.cephadm:Deploying osd.4 on smithi154 with /dev/vg_nvme/lv_4... 2021-07-31T01:42:23.278 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f52d2c14-f19f-11eb-8c24-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2021-07-31T01:42:23.306 INFO:journalctl@ceph.osd.3.smithi045.stdout:-- Logs begin at Sat 2021-07-31 01:16:15 UTC. -- 2021-07-31T01:42:23.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:23 smithi045 conmon[33655]: audit 2021-07-31T01:42:22.524101+0000 mon.a (mon.0) 364 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:42:23.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:23 smithi045 conmon[33655]: audit 2021-07-31T01:42:22.525602+0000 mon.a (mon.0) 365 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:42:23.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:23 smithi045 conmon[33655]: audit 2021-07-31T01:42:22.527607+0000 mon.a (mon.0) 366 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:42:23.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:23 smithi045 conmon[33655]: audit 2021-07-31T01:42:22.529247+0000 mon.a (mon.0) 367 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:42:23.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:23 smithi045 conmon[33655]: audit 2021-07-31T01:42:22.531916+0000 mon.a (mon.0) 368 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:42:23.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:23 smithi045 conmon[39718]: audit 2021-07-31T01:42:22.524101+0000 mon.a (mon.0) 364 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:42:23.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:23 smithi045 conmon[39718]: audit 2021-07-31T01:42:22.525602+0000 mon.a (mon.0) 365 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:42:23.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:23 smithi045 conmon[39718]: audit 2021-07-31T01:42:22.527607+0000 mon.a (mon.0) 366 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:42:23.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:23 smithi045 conmon[39718]: audit 2021-07-31T01:42:22.529247+0000 mon.a (mon.0) 367 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:42:23.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:23 smithi045 conmon[39718]: audit 2021 2021-07-31T01:42:23.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:23 smithi045 conmon[39718]: -07-31T01:42:22.531916+0000 mon.a (mon.0) 368 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:42:23.726 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:23 smithi154 conmon[34553]: audit 2021-07-31T01:42: 2021-07-31T01:42:23.726 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:23 smithi154 conmon[34553]: 22.524101+0000 mon.a (mon.0) 364 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:42:23.726 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:23 smithi154 conmon[34553]: audit 2021-07-31T01:42:22.525602+0000 mon.a (mon.0) 365 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:42:23.726 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:23 smithi154 conmon[34553]: audit 2021-07-31 2021-07-31T01:42:23.728 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:23 smithi154 conmon[34553]: T01:42:22.527607+0000 2021-07-31T01:42:23.728 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:23 smithi154 conmon[34553]: mon.a (mon.0) 366 2021-07-31T01:42:23.729 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:23 smithi154 conmon[34553]: : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:42:23.729 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:23 smithi154 conmon[34553]: audit 2021-07-31T01:42:23.730 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:23 smithi154 conmon[34553]: 2021-07-31T01 2021-07-31T01:42:23.730 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:23 smithi154 conmon[34553]: :42:22.529247+0000 2021-07-31T01:42:23.730 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:23 smithi154 conmon[34553]: mon.a (mon.0) 367 : 2021-07-31T01:42:23.731 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:23 smithi154 conmon[34553]: audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:42:23.731 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:23 smithi154 conmon[34553]: audit 2021-07 2021-07-31T01:42:23.731 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:23 smithi154 conmon[34553]: -31T01:42:22. 2021-07-31T01:42:23.732 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:23 smithi154 conmon[34553]: 531916+0000 mon.a (mon.0 2021-07-31T01:42:23.732 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:23 smithi154 conmon[34553]: ) 368 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:42:23.732 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:23 smithi154 conmon[34553]: 2021-07-31T01:42:24.802 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:24 smithi154 conmon[34553]: cluster 2021-07-31T01:42:23.318063+0000 mgr.y (mgr.14146) 116 : cluster [DBG] pgmap v94: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-07-31T01:42:25.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:24 smithi045 conmon[39718]: cluster 2021-07-31T01:42:23.318063+0000 2021-07-31T01:42:25.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:24 smithi045 conmon[39718]: mgr.y (mgr.14146) 116 : cluster [DBG] pgmap v94: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-07-31T01:42:25.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:24 smithi045 conmon[33655]: cluster 2021-07-31T01:42:23.318063+0000 mgr.y (mgr. 2021-07-31T01:42:25.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:24 smithi045 conmon[33655]: 14146) 116 : cluster [DBG] pgmap v94: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-07-31T01:42:25.573 INFO:teuthology.orchestra.run.smithi154.stderr:--> Zapping: /dev/vg_nvme/lv_4 2021-07-31T01:42:25.573 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2021-07-31T01:42:25.574 INFO:teuthology.orchestra.run.smithi154.stderr: stderr: 10+0 records in 2021-07-31T01:42:25.574 INFO:teuthology.orchestra.run.smithi154.stderr:10+0 records out 2021-07-31T01:42:25.574 INFO:teuthology.orchestra.run.smithi154.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0109427 s, 958 MB/s 2021-07-31T01:42:25.574 INFO:teuthology.orchestra.run.smithi154.stderr:--> Zapping successful for: 2021-07-31T01:42:25.764 INFO:journalctl@ceph.osd.3.smithi045.stdout:Jul 31 01:42:25 smithi045 conmon[54013]: debug 2021-07-31T01:42:25.589+0000 7f652554d080 -1 osd.3 0 log_to_monitors {default=true} 2021-07-31T01:42:26.214 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:25 smithi154 conmon[34553]: audit 2021-07-31T01:42:24.960008+0000 mon.a (mon.0) 369 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:42:26.215 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:25 smithi154 conmon[34553]: audit 2021-07-31T01:42:25.594042+0000 mon.a (mon.0) 370 : audit [INF] from='osd.3 [v2:172.21.15.45:6826/3024698157,v1:172.21.15.45:6827/3024698157]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2021-07-31T01:42:26.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:25 smithi045 conmon[33655]: audit 2021-07-31T01:42:24.960008+0000 mon.a (mon 2021-07-31T01:42:26.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:25 smithi045 conmon[33655]: .0) 369 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:42:26.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:25 smithi045 conmon[33655]: audit 2021-07-31T01:42:25.594042+0000 mon.a (mon.0) 370 : audit [INF] from='osd.3 [v2:172.21.15.45:6826/3024698157,v1:172.21.15.45:6827/3024698157]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2021-07-31T01:42:26.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:25 smithi045 conmon[39718]: audit 2021-07-31T01:42:26.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:25 smithi045 conmon[39718]: 2021-07-31T01:42:24 2021-07-31T01:42:26.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:25 smithi045 conmon[39718]: .960008+0000 mon.a (mon.0) 369 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:42:26.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:25 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T01:42:26.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:25 smithi045 conmon[39718]: 01:42:25.594042+0000 2021-07-31T01:42:26.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:25 smithi045 conmon[39718]: mon.a (mon.0) 370 : audit 2021-07-31T01:42:26.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:25 smithi045 conmon[39718]: [INF] from='osd.3 [v2:172.21.15.45:6826/3024698157,v1:172.21.15.45:6827/3024698157]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2021-07-31T01:42:26.305 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f52d2c14-f19f-11eb-8c24-001a4aab830c -- ceph orch daemon add osd smithi154:vg_nvme/lv_4 2021-07-31T01:42:27.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:26 smithi154 conmon[34553]: cluster 2021-07-31T01:42:25.318372+0000 mgr.y (mgr.14146) 2021-07-31T01:42:27.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:26 smithi154 conmon[34553]: 117 : cluster [DBG] pgmap v95: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-07-31T01:42:27.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:26 smithi154 conmon[34553]: audit 2021-07-31T01:42:25.964048+0000 mon.a (mon.0) 371 : audit [INF] from='osd.3 [v2:172.21.15.45:6826/3024698157,v1:172.21.15.45:6827/3024698157]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2021-07-31T01:42:27.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:26 smithi154 conmon[34553]: cluster 2021-07-31T01:42:25.964127+0000 mon.a (mon.0) 372 : cluster [DBG] osdmap e23: 4 total, 3 up, 4 in 2021-07-31T01:42:27.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:26 smithi154 conmon[34553]: audit 2021-07-31T01:42:25.964450+0000 mon.a (mon.0) 373 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-07-31T01:42:27.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:26 smithi154 conmon[34553]: audit 2021-07-31T01:42:25.964773+0000 mon.a (mon.0) 374 : audit [INF] from='osd.3 [v2:172.21.15.45:6826/3024698157,v1:172.21.15.45:6827/3024698157]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2021-07-31T01:42:27.276 INFO:journalctl@ceph.osd.3.smithi045.stdout:Jul 31 01:42:26 smithi045 conmon[54013]: debug 2021-07-31T01:42:26.969+0000 7f651bab0700 -1 osd.3 0 waiting for initial osdmap 2021-07-31T01:42:27.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:26 smithi045 conmon[39718]: cluster 2021-07-31T01:42:25.318372+0000 mgr.y (mgr.14146) 117 : cluster [DBG] pgmap v95: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-07-31T01:42:27.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:26 smithi045 conmon[39718]: audit 2021-07-31T01:42:25.964048+0000 mon.a (mon.0) 371 : audit [INF] from='osd.3 [v2:172.21.15.45:6826/3024698157,v1:172.21.15.45:6827/3024698157]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2021-07-31T01:42:27.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:26 smithi045 conmon[39718]: cluster 2021-07-31T01:42:25.964127+0000 mon.a (mon.0) 372 : cluster [DBG] osdmap e23: 4 total, 3 up, 4 in 2021-07-31T01:42:27.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:26 smithi045 conmon[39718]: audit 2021-07-31T01:42:25.964450+0000 mon.a (mon.0) 373 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-07-31T01:42:27.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:26 smithi045 conmon[39718]: audit 2021-07-31T01:42:25.964773+0000 mon.a (mon.0) 374 : audit [INF] from='osd.3 [v2:172.21.15.45:6826/3024698157,v1:172.21.15.45:6827/3024698157]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2021-07-31T01:42:27.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:26 smithi045 conmon[33655]: cluster 2021-07-31T01:42:25.318372+0000 mgr.y (mgr.14146) 117 : cluster [DBG] pgmap v95: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-07-31T01:42:27.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:26 smithi045 conmon[33655]: audit 2021-07-31T01:42:25.964048+0000 mon.a (mon.0) 371 : audit [INF] from='osd.3 [v2:172.21.15.45:6826/3024698157,v1:172.21.15.45:6827/3024698157]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2021-07-31T01:42:27.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:26 smithi045 conmon[33655]: cluster 2021-07-31T01:42:25.964127+0000 mon.a (mon.0) 372 : cluster [DBG] osdmap e23: 4 total, 3 up, 4 in 2021-07-31T01:42:27.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:26 smithi045 conmon[33655]: audit 2021-07-31T01:42:25.964450+0000 mon.a (mon.0) 373 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-07-31T01:42:27.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:26 smithi045 conmon[33655]: audit 2021-07-31T01:42:25.964773+0000 mon.a (mon.0) 374 : audit [INF] from='osd.3 [v2:172.21.15.45:6826/3024698157,v1:172.21.15.45:6827/3024698157]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2021-07-31T01:42:27.973 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:27 smithi045 conmon[33655]: audit 2021-07-31T01:42:26.965205+0000 mon.a (mon 2021-07-31T01:42:27.973 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:27 smithi045 conmon[33655]: .0) 375 : audit [INF] from='osd.3 [v2:172.21.15.45:6826/3024698157,v1:172.21.15.45:6827/3024698157]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2021-07-31T01:42:27.973 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:27 smithi045 conmon[33655]: cluster 2021-07-31T01:42:26.965279+0000 mon.a (mon.0) 376 : cluster [DBG] osdmap e24: 4 total, 3 up, 4 in 2021-07-31T01:42:27.974 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:27 smithi045 conmon[33655]: audit 2021-07-31T01:42:26.965690+0000 mon.a (mon.0) 377 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-07-31T01:42:27.974 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:27 smithi045 conmon[33655]: audit 2021-07-31T01:42:26.970020+0000 mon.a (mon.0) 378 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-07-31T01:42:28.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:27 smithi154 conmon[34553]: audit 2021-07-31T01:42:26.965205+0000 mon.a (mon.0) 375 : 2021-07-31T01:42:28.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:27 smithi154 conmon[34553]: audit [INF] from='osd.3 [v2:172.21.15.45:6826/3024698157,v1:172.21.15.45:6827/3024698157]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2021-07-31T01:42:28.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:27 smithi154 conmon[34553]: cluster 2021-07-31T01:42:26.965279+0000 mon.a (mon.0) 376 : cluster [DBG] osdmap e24: 4 total, 3 up, 4 in 2021-07-31T01:42:28.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:27 smithi154 conmon[34553]: audit 2021-07-31T01:42:26.965690+0000 mon.a (mon.0) 377 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-07-31T01:42:28.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:27 smithi154 conmon[34553]: audit 2021-07-31T01:42:26.970020+0000 mon.a (mon.0) 378 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-07-31T01:42:28.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:27 smithi045 conmon[39718]: audit 2021-07-31T01:42: 2021-07-31T01:42:28.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:27 smithi045 conmon[39718]: 26.965205+0000 mon.a (mon.0) 375 : audit [INF] from='osd.3 [v2:172.21.15.45:6826/3024698157,v1:172.21.15.45:6827/3024698157]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2021-07-31T01:42:28.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:27 smithi045 conmon[39718]: cluster 2021-07-31T01:42 2021-07-31T01:42:28.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:27 smithi045 conmon[39718]: :26.965279+0000 mon.a (mon.0) 376 : cluster [DBG] 2021-07-31T01:42:28.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:27 smithi045 conmon[39718]: osdmap e24: 4 total, 3 up, 4 in 2021-07-31T01:42:28.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:27 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T01:42:28.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:27 smithi045 conmon[39718]: 31T01:42:26.965690+0000 mon.a (mon.0) 377 : audit 2021-07-31T01:42:28.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:27 smithi045 conmon[39718]: [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-07-31T01:42:28.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:27 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T01:42:28.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:27 smithi045 conmon[39718]: T01:42:26.970020+0000 mon.a (mon.0) 378 : audit [DBG] 2021-07-31T01:42:28.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:27 smithi045 conmon[39718]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-07-31T01:42:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:28 smithi154 conmon[34553]: cluster 2021-07-31T01:42:26.612879+0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2021-07-31T01:42:29.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:28 smithi154 conmon[34553]: cluster 2021-07-31T01:42:26.612984+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2021-07-31T01:42:29.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:28 smithi154 conmon[34553]: cluster 2021-07-31T01:42:27.318808+0000 mgr.y (mgr.14146) 118 : cluster [DBG] pgmap v98: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-07-31T01:42:29.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:28 smithi154 conmon[34553]: audit 2021-07-31T01:42:27.970136+0000 mon.a (mon.0) 379 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-07-31T01:42:29.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:28 smithi154 conmon[34553]: cluster 2021-07-31T01:42:27.973675+0000 mon.a (mon.0) 380 : cluster [INF] osd.3 [v2:172.21.15.45:6826/3024698157,v1:172.21.15.45:6827/3024698157] boot 2021-07-31T01:42:29.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:28 smithi154 conmon[34553]: cluster 2021-07-31T01:42:27.973701+0000 mon.a (mon.0) 381 : cluster [DBG] osdmap e25: 4 total, 4 up, 4 in 2021-07-31T01:42:29.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:28 smithi154 conmon[34553]: audit 2021-07-31T01:42:27.973793+0000 mon.a (mon.0) 382 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-07-31T01:42:29.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:28 smithi154 conmon[34553]: audit 2021-07-31T01:42:28.494456+0000 mon.a (mon.0) 383 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-07-31T01:42:29.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:28 smithi154 conmon[34553]: audit 2021-07-31T01:42:28.495885+0000 mon.a (mon.0) 384 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-07-31T01:42:29.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:28 smithi154 conmon[34553]: audit 2021-07-31T01:42:28.496516+0000 mon.a (mon.0) 385 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:42:29.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:28 smithi154 conmon[34553]: audit 2021-07-31T01:42:28.785406+0000 mon.a (mon.0) 386 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:42:29.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:28 smithi154 conmon[34553]: audit 2021-07-31T01:42:28.786254+0000 mon.a (mon.0) 387 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-07-31T01:42:29.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:28 smithi154 conmon[34553]: audit 2021-07-31T01:42:28.786846+0000 mon.a (mon.0) 388 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-07-31T01:42:29.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:28 smithi154 conmon[34553]: audit 2021-07-31T01:42:28.787440+0000 mon.a (mon.0) 389 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-07-31T01:42:29.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:28 smithi154 conmon[34553]: audit 2021-07-31T01:42:28.787991+0000 mon.a (mon.0) 390 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-07-31T01:42:29.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:28 smithi154 conmon[34553]: audit 2021-07-31T01:42 2021-07-31T01:42:29.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:28 smithi154 conmon[34553]: :28.791625+0000 mon.a (mon.0) 391 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:42:29.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:28 smithi154 conmon[34553]: audit 2021-07-31T01:42:28.794591+0000 mon.a (mon.0) 392 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:42:29.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:28 smithi154 conmon[34553]: audit 2021-07-31T01:42:29.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:28 smithi154 conmon[34553]: 2021-07-31T01:42:28.796015+0000 mon.a (mon.0) 393 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:42:29.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:28 smithi154 conmon[34553]: audit 2021- 2021-07-31T01:42:29.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:28 smithi154 conmon[34553]: 07-31T01:42:28.796908+0000 mon.a 2021-07-31T01:42:29.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:28 smithi154 conmon[34553]: (mon.0) 394 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:42:29.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:28 smithi154 conmon[34553]: audit 2021-07-31T01 2021-07-31T01:42:29.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:28 smithi154 conmon[34553]: :42:28.797460+0000 mon.a (mon.0) 395 : audit 2021-07-31T01:42:29.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:28 smithi154 conmon[34553]: [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:42:29.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:28 smithi154 conmon[34553]: audit 2021-07-31T01:42:29.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:28 smithi154 conmon[34553]: 2021-07-31T01:42:28.800798+0000 mon.a (mon.0 2021-07-31T01:42:29.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:28 smithi154 conmon[34553]: ) 396 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:42:29.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[39718]: cluster 2021-07-31T01:42:26.612879+0000 2021-07-31T01:42:29.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[39718]: osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2021-07-31T01:42:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[39718]: cluster 2021-07-31T01:42:26 2021-07-31T01:42:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[39718]: .612984+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2021-07-31T01:42:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[39718]: cluster 2021-07-31T01:42:27.318808+0000 mgr.y 2021-07-31T01:42:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[39718]: (mgr.14146) 118 : cluster [DBG] pgmap v98: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-07-31T01:42:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[39718]: audit 2021-07-31T01:42 2021-07-31T01:42:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[39718]: :27.970136+0000 mon.a (mon.0) 379 : audit [DBG] 2021-07-31T01:42:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[39718]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-07-31T01:42:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[39718]: cluster 2021-07-31T01:42:29.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[39718]: 2021-07-31T01:42:27.973675+0000 mon.a (mon.0) 380 : cluster 2021-07-31T01:42:29.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[39718]: [INF] osd.3 [v2:172.21.15.45:6826/3024698157,v1:172.21.15.45:6827/3024698157] boot 2021-07-31T01:42:29.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[39718]: cluster 2021-07-31T01:42:27.973701+0000 2021-07-31T01:42:29.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[39718]: mon.a (mon.0) 381 : cluster [DBG] osdmap e25: 4 total, 4 up, 4 in 2021-07-31T01:42:29.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T01:42:29.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[39718]: 31T01:42:27.973793+0000 mon.a (mon.0) 382 2021-07-31T01:42:29.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[39718]: : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-07-31T01:42:29.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[39718]: audit 2021-07-31T01:42:28. 2021-07-31T01:42:29.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[39718]: 494456+0000 mon.a (mon.0) 383 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-07-31T01:42:29.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[39718]: audit 2021-07-31T01:42:28 2021-07-31T01:42:29.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[39718]: .495885+0000 mon.a (mon.0) 384 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-07-31T01:42:29.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[39718]: 2021-07-31T01:42:29.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[39718]: audit 2021-07 2021-07-31T01:42:29.282 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[39718]: -31T01:42:28.496516+0000 mon.a (mon.0) 385 : audit [DBG] 2021-07-31T01:42:29.282 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[39718]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:42:29.282 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[39718]: audit 2021-07-31T01:42:29.282 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[39718]: 2021-07-31T01:42:28.785406+0000 mon.a (mon.0) 386 : 2021-07-31T01:42:29.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[39718]: audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:42:29.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[39718]: audit 2021-07-31T01:42:28.786254+0000 2021-07-31T01:42:29.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[39718]: mon.a (mon.0) 387 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-07-31T01:42:29.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[39718]: audit 2021-07-31T01 2021-07-31T01:42:29.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[39718]: :42:28.786846+0000 mon.a (mon.0) 388 : audit 2021-07-31T01:42:29.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[39718]: [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-07-31T01:42:29.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[39718]: audit 2021-07-31T01:42:29.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[39718]: 2021-07-31T01:42:28.787440+0000 mon.a (mon.0) 389 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-07-31T01:42:29.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[39718]: audit 2021-07-31T01:42:28.787991+0000 mon.a (mon. 2021-07-31T01:42:29.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[39718]: 0) 390 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-07-31T01:42:29.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[33655]: cluster 2021-07-31T01:42:26.612879+0000 osd.3 (osd 2021-07-31T01:42:29.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[33655]: .3) 1 : cluster [DBG] purged_snaps scrub starts 2021-07-31T01:42:29.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[33655]: cluster 2021-07-31T01:42:26.612984+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2021-07-31T01:42:29.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[33655]: cluster 2021-07-31T01:42:27.318808+0000 mgr.y (mgr.14146) 118 : cluster [DBG] pgmap v98: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-07-31T01:42:29.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[33655]: audit 2021-07-31T01:42:27.970136+0000 mon.a (mon.0) 379 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-07-31T01:42:29.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[33655]: cluster 2021-07-31T01:42:27.973675+0000 mon.a (mon.0) 380 2021-07-31T01:42:29.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[33655]: : cluster [INF] osd.3 [v2:172.21.15.45:6826/3024698157,v1:172.21.15.45:6827/3024698157] boot 2021-07-31T01:42:29.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[33655]: cluster 2021-07-31T01:42:27 2021-07-31T01:42:29.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[33655]: .973701+0000 mon.a (mon.0) 381 : cluster [DBG] osdmap e25: 4 total, 4 up, 4 in 2021-07-31T01:42:29.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[33655]: audit 2021- 2021-07-31T01:42:29.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[33655]: 07-31T01:42:27.973793+0000 mon.a (mon. 2021-07-31T01:42:29.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[33655]: 0) 382 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-07-31T01:42:29.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[33655]: audit 2021-07-31T01:42:29.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[33655]: 2021-07-31T01:42:28.494456+0000 2021-07-31T01:42:29.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[33655]: mon.a (mon.0) 383 : audit [DBG] 2021-07-31T01:42:29.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[33655]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-07-31T01:42:29.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[33655]: audit 2021-07-31T01 2021-07-31T01:42:29.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[33655]: :42:28.495885+0000 mon.a (mon.0 2021-07-31T01:42:29.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[33655]: ) 384 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-07-31T01:42:29.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[33655]: audit 2021-07-31T01:42:29.290 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[33655]: 2021-07-31T01: 2021-07-31T01:42:29.290 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[33655]: 42:28.496516+0000 mon.a (mon.0) 2021-07-31T01:42:29.290 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[33655]: 385 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:42:29.290 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[33655]: audit 2021-07-31T01:42:29.291 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[33655]: 2021-07-31T01:42:28.785406 2021-07-31T01:42:29.291 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[33655]: +0000 mon.a (mon.0) 386 : audit [INF] 2021-07-31T01:42:29.291 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[33655]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:42:29.291 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[33655]: audit 2021-07-31T01:42:29.291 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[33655]: 2021-07-31T01:42:28.786254 2021-07-31T01:42:29.292 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[33655]: +0000 mon.a (mon.0) 387 : audit [INF] 2021-07-31T01:42:29.292 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[33655]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-07-31T01:42:29.292 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[33655]: audit 2021-07-31T01:42:29.292 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[33655]: 2021-07-31T01:42:28.786846 2021-07-31T01:42:29.293 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[33655]: +0000 mon.a (mon.0) 388 : audit [INF] 2021-07-31T01:42:29.293 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[33655]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-07-31T01:42:29.293 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[33655]: audit 2021-07-31 2021-07-31T01:42:29.293 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[33655]: T01:42:28.787440+0000 mon.a (mon. 2021-07-31T01:42:29.293 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[33655]: 0) 389 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-07-31T01:42:29.294 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[33655]: audit 2021-07-31T01:42:29.294 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[33655]: 2021-07-31T01:42:28.787991+0000 2021-07-31T01:42:29.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[39718]: audit 2021-07-31T01:42:28 2021-07-31T01:42:29.295 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[39718]: .791625+0000 mon.a (mon.0) 391 : audit [INF] 2021-07-31T01:42:29.295 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[39718]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:42:29.295 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[39718]: audit 2021 2021-07-31T01:42:29.295 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[39718]: -07-31T01:42:28.794591+0000 mon.a (mon. 2021-07-31T01:42:29.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[39718]: 0) 392 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:42:29.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[39718]: 2021-07-31T01:42:29.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[39718]: audit 2021-07-31T01:42:29.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[39718]: 2021-07-31T01:42:28 2021-07-31T01:42:29.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[39718]: .796015+0000 mon.a (mon.0 2021-07-31T01:42:29.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[39718]: ) 393 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:42:29.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T01:42:29.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[39718]: T01:42:28.796908+0000 mon.a (mon.0) 394 : 2021-07-31T01:42:29.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[39718]: audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:42:29.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[39718]: audit 2021-07-31T01:42:28.797460+0000 2021-07-31T01:42:29.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[39718]: mon.a (mon.0) 395 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:42:29.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[39718]: audit 2021-07 2021-07-31T01:42:29.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[39718]: -31T01:42:28.800798+0000 mon.a (mon.0) 2021-07-31T01:42:29.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[39718]: 396 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:42:29.299 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[33655]: mon.a (mon.0) 390 : audit [INF] 2021-07-31T01:42:29.299 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[33655]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-07-31T01:42:29.300 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[33655]: audit 2021-07-31 2021-07-31T01:42:29.300 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[33655]: T01:42:28.791625+0000 mon.a (mon 2021-07-31T01:42:29.300 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[33655]: .0) 391 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:42:29.300 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[33655]: audit 2021-07-31T01:42:29.300 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[33655]: 2021-07-31T01 2021-07-31T01:42:29.301 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[33655]: :42:28.794591+0000 mon.a (mon. 2021-07-31T01:42:29.301 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[33655]: 0) 392 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:42:29.301 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[33655]: audit 2021-07-31T01:42:29.302 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[33655]: 2021-07-31T01 2021-07-31T01:42:29.302 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[33655]: :42:28.796015+0000 mon.a (mon.0 2021-07-31T01:42:29.302 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[33655]: ) 393 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:42:29.302 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[33655]: audit 2021-07-31T01:42:29.303 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[33655]: 2021-07-31T01:42 2021-07-31T01:42:29.303 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[33655]: :28.796908+0000 mon.a (mon.0) 394 2021-07-31T01:42:29.303 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[33655]: : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:42:29.303 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[33655]: audit 2021-07-31T01:42:29.303 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[33655]: 2021-07-31T01: 2021-07-31T01:42:29.304 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[33655]: 42:28.797460+0000 mon.a (mon.0 2021-07-31T01:42:29.304 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[33655]: ) 395 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:42:29.304 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[33655]: audit 2021-07-31T01:42:29.304 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[33655]: 2021-07-31T01 2021-07-31T01:42:29.305 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[33655]: :42:28.800798+0000 mon.a (mon. 2021-07-31T01:42:29.305 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:28 smithi045 conmon[33655]: 0) 396 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:42:30.241 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:29 smithi154 conmon[34553]: audit 2021-07-31T01:42:28.493583+0000 mgr.y (mgr.14146) 119 : audit [DBG] from='client.24203 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:42:30.242 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:29 smithi154 conmon[34553]: cephadm 2021-07-31T01:42:28.781096+0000 mgr.y (mgr.14146) 120 : cephadm [INF] Detected new or changed devices on smithi045 2021-07-31T01:42:30.242 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:29 smithi154 conmon[34553]: cephadm 2021-07-31T01:42:28.788423+0000 mgr.y (mgr.14146) 121 : cephadm [INF] Adjusting osd_memory_target on smithi045 to 4038M 2021-07-31T01:42:30.242 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:29 smithi154 conmon[34553]: cluster 2021-07-31T01:42:28.974226+0000 mon.a (mon.0) 397 : cluster 2021-07-31T01:42:30.243 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:29 smithi154 conmon[34553]: [DBG] osdmap e26: 4 total, 4 up, 4 in 2021-07-31T01:42:30.243 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:29 smithi154 conmon[34553]: cluster 2021-07-31T01:42:29.319139+0000 mgr.y (mgr 2021-07-31T01:42:30.243 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:29 smithi154 conmon[34553]: .14146) 122 : cluster [DBG] pgmap v101: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-07-31T01:42:30.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:29 smithi045 conmon[33655]: audit 2021-07-31T01:42:28.493583+0000 mgr.y (mgr.14146) 119 : audit [DBG] from='client.24203 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:42:30.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:29 smithi045 conmon[33655]: cephadm 2021-07-31T01:42:28.781096+0000 mgr.y (mgr.14146) 120 : cephadm [INF] Detected new or changed devices on smithi045 2021-07-31T01:42:30.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:29 smithi045 conmon[33655]: cephadm 2021-07-31T01:42:28.788423+0000 mgr.y (mgr.14146) 121 : 2021-07-31T01:42:30.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:29 smithi045 conmon[33655]: cephadm [INF] Adjusting osd_memory_target on smithi045 to 4038M 2021-07-31T01:42:30.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:29 smithi045 conmon[33655]: cluster 2021-07-31T01:42:28.974226+0000 mon.a (mon.0) 397 : 2021-07-31T01:42:30.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:29 smithi045 conmon[33655]: cluster [DBG] osdmap e26: 4 total, 4 up, 4 in 2021-07-31T01:42:30.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:29 smithi045 conmon[33655]: cluster 2021-07-31T01:42: 2021-07-31T01:42:30.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:29 smithi045 conmon[33655]: 29.319139+0000 mgr.y (mgr.14146) 122 : cluster [DBG] pgmap v101: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-07-31T01:42:30.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:29 smithi045 conmon[39718]: audit 2021-07-31T01:42:28.493583+0000 mgr.y ( 2021-07-31T01:42:30.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:29 smithi045 conmon[39718]: mgr.14146) 119 : audit [DBG] from='client.24203 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:42:30.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:29 smithi045 conmon[39718]: cephadm 2021-07-31T01:42 2021-07-31T01:42:30.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:29 smithi045 conmon[39718]: :28.781096+0000 mgr.y (mgr.14146) 120 : cephadm [INF] 2021-07-31T01:42:30.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:29 smithi045 conmon[39718]: Detected new or changed devices on smithi045 2021-07-31T01:42:30.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:29 smithi045 conmon[39718]: cephadm 2021-07- 2021-07-31T01:42:30.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:29 smithi045 conmon[39718]: 31T01:42:28.788423+0000 2021-07-31T01:42:30.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:29 smithi045 conmon[39718]: mgr.y (mgr.14146) 121 2021-07-31T01:42:30.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:29 smithi045 conmon[39718]: : cephadm [INF] Adjusting osd_memory_target on smithi045 to 4038M 2021-07-31T01:42:30.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:29 smithi045 conmon[39718]: cluster 2021-07- 2021-07-31T01:42:30.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:29 smithi045 conmon[39718]: 31T01:42:28.974226+0000 mon.a (mon 2021-07-31T01:42:30.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:29 smithi045 conmon[39718]: .0) 397 : cluster [DBG] osdmap e26: 4 total, 4 up, 4 in 2021-07-31T01:42:30.282 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:29 smithi045 conmon[39718]: cluster 2021-07-31T01:42:29. 2021-07-31T01:42:30.282 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:29 smithi045 conmon[39718]: 319139+0000 mgr.y (mgr.14146) 122 : cluster [DBG] 2021-07-31T01:42:30.282 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:29 smithi045 conmon[39718]: pgmap v101: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-07-31T01:42:31.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:30 smithi154 conmon[34553]: audit 2021-07-31T01:42:30.967413+0000 mon.b (mon.2) 2021-07-31T01:42:31.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:30 smithi154 conmon[34553]: 7 : audit [INF] from='client.? 172.21.15.154:0/1365789729' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "69881d9d-3e56-4483-b76b-7dba14cfaac3"}]: dispatch 2021-07-31T01:42:31.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:30 smithi154 conmon[34553]: audit 2021-07-31T01:42:30.967994+0000 mon.a (mon.0) 398 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "69881d9d-3e56-4483-b76b-7dba14cfaac3"}]: dispatch 2021-07-31T01:42:31.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:30 smithi154 conmon[34553]: audit 2021-07-31T01:42:30.973448+0000 mon.a (mon.0) 399 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "69881d9d-3e56-4483-b76b-7dba14cfaac3"}]': finished 2021-07-31T01:42:31.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:30 smithi154 conmon[34553]: cluster 2021-07-31T01:42:30.973553+0000 mon.a (mon.0) 400 : cluster [DBG] osdmap e27: 5 total, 4 up, 5 in 2021-07-31T01:42:31.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:30 smithi154 conmon[34553]: audit 2021-07-31T01:42:30.973713+0000 mon.a (mon.0) 401 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-07-31T01:42:31.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:30 smithi045 conmon[33655]: audit 2021-07-31T01:42:30.967413 2021-07-31T01:42:31.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:30 smithi045 conmon[33655]: +0000 mon.b (mon.2) 7 : audit [INF] from='client.? 172.21.15.154:0/1365789729' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "69881d9d-3e56-4483-b76b-7dba14cfaac3"}]: dispatch 2021-07-31T01:42:31.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:30 smithi045 conmon[33655]: audit 2021-07-31T01:42:30.967994+0000 mon.a (mon.0) 398 2021-07-31T01:42:31.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:30 smithi045 conmon[33655]: : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "69881d9d-3e56-4483-b76b-7dba14cfaac3"}]: dispatch 2021-07-31T01:42:31.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:30 smithi045 conmon[33655]: audit 2021-07- 2021-07-31T01:42:31.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:30 smithi045 conmon[33655]: 31T01:42:30.973448 2021-07-31T01:42:31.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:30 smithi045 conmon[33655]: +0000 mon.a (mon.0) 399 2021-07-31T01:42:31.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:30 smithi045 conmon[33655]: : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "69881d9d-3e56-4483-b76b-7dba14cfaac3"}]': finished 2021-07-31T01:42:31.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:30 smithi045 conmon[33655]: cluster 2021- 2021-07-31T01:42:31.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:30 smithi045 conmon[33655]: 07-31T01:42:30.973553 2021-07-31T01:42:31.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:30 smithi045 conmon[33655]: +0000 mon.a (mon.0) 400 2021-07-31T01:42:31.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:30 smithi045 conmon[33655]: : cluster [DBG] osdmap e27: 5 total, 4 up, 5 in 2021-07-31T01:42:31.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:30 smithi045 conmon[33655]: audit 2021 2021-07-31T01:42:31.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:30 smithi045 conmon[33655]: -07-31T01:42:30. 2021-07-31T01:42:31.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:30 smithi045 conmon[33655]: 973713+0000 mon.a (mon.0 2021-07-31T01:42:31.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:30 smithi045 conmon[33655]: ) 401 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-07-31T01:42:31.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:30 smithi045 conmon[39718]: audit 2021-07-31T01:42:30. 2021-07-31T01:42:31.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:30 smithi045 conmon[39718]: 967413+0000 mon.b (mon.2) 7 : audit [INF] from='client.? 172.21.15.154:0/1365789729' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "69881d9d-3e56-4483-b76b-7dba14cfaac3"}]: dispatch 2021-07-31T01:42:31.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:30 smithi045 conmon[39718]: audit 2021-07-31T01:42:31.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:30 smithi045 conmon[39718]: 2021-07-31T01 2021-07-31T01:42:31.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:30 smithi045 conmon[39718]: :42:30.967994+0000 mon.a 2021-07-31T01:42:31.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:30 smithi045 conmon[39718]: (mon.0) 398 : audit 2021-07-31T01:42:31.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:30 smithi045 conmon[39718]: [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "69881d9d-3e56-4483-b76b-7dba14cfaac3"}]: dispatch 2021-07-31T01:42:31.282 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:30 smithi045 conmon[39718]: audit 2021- 2021-07-31T01:42:31.282 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:30 smithi045 conmon[39718]: 07-31T01:42:30.973448+0000 2021-07-31T01:42:31.282 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:30 smithi045 conmon[39718]: mon.a (mon.0) 399 : audit [INF] 2021-07-31T01:42:31.282 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:30 smithi045 conmon[39718]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "69881d9d-3e56-4483-b76b-7dba14cfaac3"}]': finished 2021-07-31T01:42:31.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:30 smithi045 conmon[39718]: cluster 2021-07-31T 2021-07-31T01:42:31.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:30 smithi045 conmon[39718]: 01:42:30.973553+0000 mon.a ( 2021-07-31T01:42:31.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:30 smithi045 conmon[39718]: mon.0) 400 : cluster [DBG] osdmap e27: 5 total, 4 up, 5 in 2021-07-31T01:42:31.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:30 smithi045 conmon[39718]: 2021-07-31T01:42:31.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:30 smithi045 conmon[39718]: audit 2021-07-31T01:42 2021-07-31T01:42:31.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:30 smithi045 conmon[39718]: :30.973713+0000 mon.a (mon.0 2021-07-31T01:42:31.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:30 smithi045 conmon[39718]: ) 401 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-07-31T01:42:32.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:32 smithi154 conmon[34553]: cluster 2021-07-31T01:42:31.319521+0000 mgr.y (mgr.14146) 123 : cluster [DBG] pgmap v103: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-07-31T01:42:32.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:32 smithi154 conmon[34553]: audit 2021-07-31T01:42:31.480660+0000 mon.b (mon.2) 8 : audit [DBG] from='client.? 172.21.15.154:0/2068853873' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-07-31T01:42:32.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:32 smithi045 conmon[33655]: cluster 2021-07-31T01:42:31.319521+0000 2021-07-31T01:42:32.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:32 smithi045 conmon[33655]: mgr.y (mgr.14146) 123 : cluster [DBG] pgmap v103: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-07-31T01:42:32.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:32 smithi045 conmon[33655]: audit 2021-07-31T01:42:31.480660+0000 mon.b (mon.2) 8 : audit [DBG] 2021-07-31T01:42:32.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:32 smithi045 conmon[33655]: from='client.? 172.21.15.154:0/2068853873' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-07-31T01:42:32.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:32 smithi045 conmon[39718]: cluster 2021-07-31T01:42:31.319521+0000 mgr.y (mgr.14146) 123 2021-07-31T01:42:32.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:32 smithi045 conmon[39718]: : cluster [DBG] pgmap v103: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-07-31T01:42:32.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:32 smithi045 conmon[39718]: audit 2021-07-31T01:42:31.480660+0000 mon.b (mon.2) 8 : audit [DBG] from='client.? 172.21.15.154:0/2068853873' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-07-31T01:42:32.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:32 smithi045 conmon[39718]: 2021-07-31T01:42:34.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:34 smithi154 conmon[34553]: cluster 2021-07-31T01:42:33.319963+0000 mgr.y (mgr.14146) 124 : cluster [DBG] pgmap v104: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-07-31T01:42:34.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:34 smithi045 conmon[33655]: cluster 2021-07-31T01:42:33.319963+0000 mgr.y (mgr. 2021-07-31T01:42:34.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:34 smithi045 conmon[33655]: 14146) 124 : cluster [DBG] pgmap v104: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-07-31T01:42:34.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:34 smithi045 conmon[39718]: cluster 2021-07-31T01:42: 2021-07-31T01:42:34.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:34 smithi045 conmon[39718]: 33.319963+0000 mgr.y (mgr.14146) 124 : cluster [DBG] pgmap v104: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-07-31T01:42:36.495 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:36 smithi154 conmon[34553]: cluster 2021-07-31T01:42:35.320494+0000 mgr.y (mgr.14146) 125 : cluster [DBG] pgmap v105: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-07-31T01:42:36.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:36 smithi045 conmon[33655]: cluster 2021-07-31T01:42:35.320494+0000 mgr.y (mgr.14146) 125 : cluster [DBG] 2021-07-31T01:42:36.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:36 smithi045 conmon[33655]: pgmap v105: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-07-31T01:42:36.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:36 smithi045 conmon[39718]: cluster 2021-07-31T01:42:35.320494+0000 mgr.y (mgr. 2021-07-31T01:42:36.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:36 smithi045 conmon[39718]: 14146) 125 : cluster [DBG] pgmap v105: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-07-31T01:42:37.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:37 smithi154 conmon[34553]: audit 2021-07-31T01:42:37.372497+0000 mon.a (mon.0) 402 : audit [INF] 2021-07-31T01:42:37.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:37 smithi154 conmon[34553]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2021-07-31T01:42:37.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:37 smithi154 conmon[34553]: audit 2021-07-31T01:42:37.373358+0000 mon.a (mon.0) 403 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:42:37.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:37 smithi045 conmon[33655]: audit 2021-07-31T01:42:37.372497+0000 2021-07-31T01:42:37.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:37 smithi045 conmon[33655]: mon.a (mon.0) 402 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2021-07-31T01:42:37.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:37 smithi045 conmon[33655]: audit 2021-07-31T01:42:37.373358+0000 mon.a (mon.0) 403 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:42:37.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:37 smithi045 conmon[39718]: audit 2021-07-31T01:42:37.372497+0000 mon.a ( 2021-07-31T01:42:37.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:37 smithi045 conmon[39718]: mon.0) 402 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2021-07-31T01:42:37.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:37 smithi045 conmon[39718]: audit 2021-07-31T01:42:37.373358+0000 2021-07-31T01:42:37.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:37 smithi045 conmon[39718]: mon.a (mon.0) 403 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:42:38.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:38 smithi154 conmon[34553]: cluster 2021-07-31T01:42:37.320874+0000 mgr.y (mgr 2021-07-31T01:42:38.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:38 smithi154 conmon[34553]: .14146) 126 : cluster [DBG] pgmap v106: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-07-31T01:42:38.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:38 smithi154 conmon[34553]: cephadm 2021-07-31T01:42:37.374085+0000 mgr.y (mgr.14146) 127 : cephadm [INF] Deploying daemon osd.4 on smithi154 2021-07-31T01:42:38.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:38 smithi045 conmon[33655]: cluster 2021-07-31T01:42:37. 2021-07-31T01:42:38.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:38 smithi045 conmon[33655]: 320874+0000 mgr.y (mgr.14146) 126 : cluster [DBG] pgmap v106: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-07-31T01:42:38.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:38 smithi045 conmon[33655]: cephadm 2021-07-31T01:42:37. 2021-07-31T01:42:38.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:38 smithi045 conmon[33655]: 374085+0000 mgr.y (mgr.14146) 127 : cephadm [INF] Deploying daemon osd.4 on smithi154 2021-07-31T01:42:38.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:38 smithi045 conmon[39718]: cluster 2021-07-31T01:42:37.320874+0000 mgr.y 2021-07-31T01:42:38.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:38 smithi045 conmon[39718]: (mgr.14146) 126 : cluster [DBG] pgmap v106: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-07-31T01:42:38.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:38 smithi045 conmon[39718]: cephadm 2021-07 2021-07-31T01:42:38.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:38 smithi045 conmon[39718]: -31T01:42:37.374085+0000 mgr.y ( 2021-07-31T01:42:38.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:38 smithi045 conmon[39718]: mgr.14146) 127 : cephadm [INF] Deploying daemon osd.4 on smithi154 2021-07-31T01:42:38.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:38 smithi045 conmon[39718]: 2021-07-31T01:42:39.479 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:39 smithi154 conmon[34553]: audit 2021-07-31T01:42:39.425635+0000 mon.a (mon.0) 404 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:42:39.479 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:39 smithi154 conmon[34553]: audit 2021-07-31T01:42:39.426086+0000 mon.a (mon.0) 405 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-07-31T01:42:39.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:39 smithi045 conmon[39718]: audit 2021-07-31T01:42:39.425635+0000 mon.a (mon.0) 2021-07-31T01:42:39.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:39 smithi045 conmon[39718]: 404 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:42:39.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:39 smithi045 conmon[39718]: audit 2021-07-31T01:42:39.426086+0000 2021-07-31T01:42:39.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:39 smithi045 conmon[39718]: mon.a (mon.0) 405 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-07-31T01:42:39.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:39 smithi045 conmon[33655]: audit 2021-07-31T01:42:39.425635 2021-07-31T01:42:39.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:39 smithi045 conmon[33655]: +0000 mon.a (mon.0) 404 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:42:39.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:39 smithi045 conmon[33655]: audit 2021-07-31T01:42:39.426086+0000 mon.a (mon.0) 405 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-07-31T01:42:40.695 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:40 smithi154 conmon[34553]: cluster 2021-07-31T01:42:39.321244+0000 mgr.y (mgr.14146) 128 : 2021-07-31T01:42:40.696 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:40 smithi154 conmon[34553]: cluster [DBG] pgmap v107: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-07-31T01:42:40.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:40 smithi045 conmon[33655]: cluster 2021-07-31T01:42:39.321244+0000 mgr.y (mgr.14146) 128 : cluster 2021-07-31T01:42:40.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:40 smithi045 conmon[33655]: [DBG] pgmap v107: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-07-31T01:42:40.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:40 smithi045 conmon[39718]: cluster 2021-07-31T01:42:39.321244+0000 2021-07-31T01:42:40.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:40 smithi045 conmon[39718]: mgr.y (mgr.14146) 128 : cluster [DBG] pgmap v107: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-07-31T01:42:41.402 INFO:teuthology.orchestra.run.smithi154.stdout:Created osd(s) 4 on host 'smithi154' 2021-07-31T01:42:41.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:41 smithi154 conmon[34553]: audit 2021-07-31T01:42: 2021-07-31T01:42:41.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:41 smithi154 conmon[34553]: 41.392103+0000 mon.a (mon.0) 406 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:42:41.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:41 smithi154 conmon[34553]: audit 2021-07-31T01:42:41.395443+0000 mon.a (mon.0) 407 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:42:41.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:41 smithi154 conmon[34553]: audit 2021-07-31T01:42:41.395686+0000 mon.a (mon.0) 408 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:42:41.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:41 smithi154 conmon[34553]: audit 2021-07-31T01:42:41.397052+0000 mon.a (mon.0) 409 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:42:41.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:41 smithi154 conmon[34553]: audit 2021-07-31T01:42:41.399815+0000 mon.a (mon.0) 410 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:42:41.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:41 smithi045 conmon[39718]: audit 2021-07-31T01:42:41.392103+0000 mon.a (mon.0) 406 2021-07-31T01:42:41.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:41 smithi045 conmon[39718]: : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:42:41.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:41 smithi045 conmon[39718]: audit 2021-07-31T01:42:41.395443+0000 mon.a (mon.0) 407 : audit [INF] 2021-07-31T01:42:41.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:41 smithi045 conmon[39718]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:42:41.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:41 smithi045 conmon[39718]: audit 2021-07-31T01:42:41.395686+0000 mon.a (mon 2021-07-31T01:42:41.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:41 smithi045 conmon[39718]: .0) 408 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:42:41.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:41 smithi045 conmon[39718]: audit 2021-07-31T01:42:41.397052+0000 mon.a (mon. 2021-07-31T01:42:41.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:41 smithi045 conmon[39718]: 0) 409 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:42:41.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:41 smithi045 conmon[39718]: audit 2021-07-31T01:42:41.399815+0000 mon.a (mon.0) 410 : 2021-07-31T01:42:41.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:41 smithi045 conmon[39718]: audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:42:41.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:41 smithi045 conmon[33655]: audit 2021-07-31T01:42:41.392103+0000 mon.a (mon.0) 406 : 2021-07-31T01:42:41.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:41 smithi045 conmon[33655]: audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:42:41.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:41 smithi045 conmon[33655]: audit 2021-07-31T01:42:41.395443+0000 mon.a (mon.0) 407 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:42:41.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:41 smithi045 conmon[33655]: audit 2021-07-31T01:42:41.395686+0000 mon.a (mon.0) 408 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:42:41.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:41 smithi045 conmon[33655]: 2021-07-31T01:42:41.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:41 smithi045 conmon[33655]: audit 2021-07-31T01:42:41.397052+0000 mon.a (mon.0) 409 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:42:41.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:41 smithi045 conmon[33655]: audit 2021-07-31T01:42:41.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:41 smithi045 conmon[33655]: 2021-07-31T01:42:41.399815+0000 mon.a (mon.0) 410 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:42:42.125 DEBUG:teuthology.orchestra.run.smithi154:osd.4> sudo journalctl -f -n 0 -u ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c@osd.4.service 2021-07-31T01:42:42.128 INFO:tasks.cephadm:Deploying osd.5 on smithi154 with /dev/vg_nvme/lv_3... 2021-07-31T01:42:42.128 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f52d2c14-f19f-11eb-8c24-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2021-07-31T01:42:42.155 INFO:journalctl@ceph.osd.4.smithi154.stdout:-- Logs begin at Sat 2021-07-31 01:17:02 UTC. -- 2021-07-31T01:42:42.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:42 smithi154 conmon[34553]: cluster 2021-07-31T01:42:41.321644+0000 mgr.y ( 2021-07-31T01:42:42.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:42 smithi154 conmon[34553]: mgr.14146) 129 : cluster [DBG] pgmap v108: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-07-31T01:42:42.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:42 smithi045 conmon[33655]: cluster 2021-07-31T01:42:41.321644+0000 mgr.y (mgr. 2021-07-31T01:42:42.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:42 smithi045 conmon[33655]: 14146) 129 : cluster [DBG] pgmap v108: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-07-31T01:42:42.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:42 smithi045 conmon[39718]: cluster 2021-07-31T01:42:41.321644+0000 mgr.y (mgr.14146 2021-07-31T01:42:42.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:42 smithi045 conmon[39718]: ) 129 : cluster [DBG] pgmap v108: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-07-31T01:42:44.400 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:44 smithi154 conmon[34553]: audit 2021-07-31T01:42:43.268523+0000 mon.a (mon.0) 411 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:42:44.401 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:44 smithi154 conmon[34553]: cluster 2021-07-31T01:42:43.322093+0000 mgr.y (mgr.14146) 130 : cluster [DBG] pgmap v109: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-07-31T01:42:44.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:44 smithi045 conmon[33655]: audit 2021-07-31T01:42:43.268523 2021-07-31T01:42:44.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:44 smithi045 conmon[33655]: +0000 mon.a (mon.0) 411 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:42:44.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:44 smithi045 conmon[33655]: cluster 2021-07-31T01:42:43.322093 2021-07-31T01:42:44.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:44 smithi045 conmon[33655]: +0000 mgr.y (mgr.14146) 130 : cluster [DBG] pgmap v109: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-07-31T01:42:44.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:44 smithi045 conmon[39718]: audit 2021-07-31T01:42:43. 2021-07-31T01:42:44.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:44 smithi045 conmon[39718]: 268523+0000 mon.a (mon.0) 411 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:42:44.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:44 smithi045 conmon[39718]: cluster 2021-07-31T01:42:44.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:44 smithi045 conmon[39718]: 2021-07-31T01:42:43. 2021-07-31T01:42:44.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:44 smithi045 conmon[39718]: 322093+0000 mgr.y (mgr.14146) 130 : cluster 2021-07-31T01:42:44.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:44 smithi045 conmon[39718]: [DBG] pgmap v109: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-07-31T01:42:44.753 INFO:journalctl@ceph.osd.4.smithi154.stdout:Jul 31 01:42:44 smithi154 conmon[38506]: debug 2021-07-31T01:42:44.455+0000 7feb0ee54080 -1 osd.4 0 log_to_monitors {default=true} 2021-07-31T01:42:45.275 INFO:teuthology.orchestra.run.smithi154.stderr:--> Zapping: /dev/vg_nvme/lv_3 2021-07-31T01:42:45.275 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2021-07-31T01:42:45.275 INFO:teuthology.orchestra.run.smithi154.stderr: stderr: 10+0 records in 2021-07-31T01:42:45.276 INFO:teuthology.orchestra.run.smithi154.stderr:10+0 records out 2021-07-31T01:42:45.276 INFO:teuthology.orchestra.run.smithi154.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0121666 s, 862 MB/s 2021-07-31T01:42:45.276 INFO:teuthology.orchestra.run.smithi154.stderr:--> Zapping successful for: 2021-07-31T01:42:45.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:45 smithi154 conmon[34553]: audit 2021-07-31T01:42:44.458754+0000 mon.b (mon.2) 9 : 2021-07-31T01:42:45.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:45 smithi154 conmon[34553]: audit [INF] from='osd.4 [v2:172.21.15.154:6800/4277144246,v1:172.21.15.154:6801/4277144246]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2021-07-31T01:42:45.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:45 smithi154 conmon[34553]: audit 2021-07-31T01:42:44.459154+0000 mon.a (mon.0) 412 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2021-07-31T01:42:45.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:45 smithi045 conmon[33655]: audit 2021-07-31T01:42:44.458754+0000 2021-07-31T01:42:45.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:45 smithi045 conmon[33655]: mon.b (mon.2) 9 : audit [INF] from='osd.4 [v2:172.21.15.154:6800/4277144246,v1:172.21.15.154:6801/4277144246]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2021-07-31T01:42:45.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:45 smithi045 conmon[33655]: audit 2021-07-31T01:42:45.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:45 smithi045 conmon[33655]: 2021-07-31T01:42:44.459154+0000 mon.a (mon.0) 412 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2021-07-31T01:42:45.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:45 smithi045 conmon[39718]: audit 2021-07-31T01:42:44.458754+0000 mon.b ( 2021-07-31T01:42:45.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:45 smithi045 conmon[39718]: mon.2) 9 : audit [INF] from='osd.4 [v2:172.21.15.154:6800/4277144246,v1:172.21.15.154:6801/4277144246]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2021-07-31T01:42:45.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:45 smithi045 conmon[39718]: audit 2021-07-31T01:42:45.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:45 smithi045 conmon[39718]: 2021-07-31T01:42:44.459154+0000 mon.a (mon.0) 412 : audit [INF] 2021-07-31T01:42:45.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:45 smithi045 conmon[39718]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2021-07-31T01:42:46.166 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f52d2c14-f19f-11eb-8c24-001a4aab830c -- ceph orch daemon add osd smithi154:vg_nvme/lv_3 2021-07-31T01:42:46.503 INFO:journalctl@ceph.osd.4.smithi154.stdout:Jul 31 01:42:46 smithi154 conmon[38506]: debug 2021-07-31T01:42:46.291+0000 7feb053b7700 -1 osd.4 0 waiting for initial osdmap 2021-07-31T01:42:46.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:46 smithi154 conmon[34553]: audit 2021-07-31T01:42:45.279650+0000 mon.a (mon.0) 413 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2021-07-31T01:42:46.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:46 smithi154 conmon[34553]: cluster 2021-07-31T01:42:45.279742+0000 mon.a (mon.0) 414 : cluster [DBG] osdmap e28: 5 total, 4 up, 5 in 2021-07-31T01:42:46.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:46 smithi154 conmon[34553]: audit 2021-07-31T01:42:45.279935+0000 mon.a (mon.0) 415 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-07-31T01:42:46.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:46 smithi154 conmon[34553]: audit 2021-07-31T01:42:45.283022+0000 mon.b (mon.2) 10 : audit [INF] from='osd.4 [v2:172.21.15.154:6800/4277144246,v1:172.21.15.154:6801/4277144246]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2021-07-31T01:42:46.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:46 smithi154 conmon[34553]: audit 2021-07-31T01:42:45.283524+0000 mon.a (mon.0) 416 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2021-07-31T01:42:46.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:46 smithi154 conmon[34553]: cluster 2021-07-31T01:42:45.322508+0000 mgr.y (mgr.14146) 131 : cluster [DBG] pgmap v111: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-07-31T01:42:46.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:46 smithi045 conmon[33655]: audit 2021-07-31T01:42:45.279650+0000 mon.a (mon. 2021-07-31T01:42:46.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:46 smithi045 conmon[33655]: 0) 413 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2021-07-31T01:42:46.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:46 smithi045 conmon[33655]: cluster 2021-07-31T01:42:45.279742+0000 mon.a (mon.0) 414 : cluster [DBG] osdmap e28: 5 total, 4 up, 5 in 2021-07-31T01:42:46.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:46 smithi045 conmon[33655]: audit 2021-07-31T01:42:45.279935+0000 mon.a (mon.0) 415 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-07-31T01:42:46.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:46 smithi045 conmon[33655]: audit 2021-07- 2021-07-31T01:42:46.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:46 smithi045 conmon[33655]: 31T01:42:45.283022+0000 mon.b (mon.2) 10 : audit [INF] 2021-07-31T01:42:46.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:46 smithi045 conmon[33655]: from='osd.4 [v2:172.21.15.154:6800/4277144246,v1:172.21.15.154:6801/4277144246]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2021-07-31T01:42:46.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:46 smithi045 conmon[33655]: audit 2021-07-31T01:42:45 2021-07-31T01:42:46.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:46 smithi045 conmon[33655]: .283524+0000 mon.a (mon.0) 416 : audit 2021-07-31T01:42:46.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:46 smithi045 conmon[33655]: [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2021-07-31T01:42:46.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:46 smithi045 conmon[33655]: cluster 2021-07-31T 2021-07-31T01:42:46.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:46 smithi045 conmon[33655]: 01:42:45.322508+0000 mgr.y (mgr. 2021-07-31T01:42:46.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:46 smithi045 conmon[33655]: 14146) 131 : cluster [DBG] pgmap v111: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-07-31T01:42:46.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:46 smithi045 conmon[39718]: audit 2021-07-31T01:42:45.279650+0000 mon.a ( 2021-07-31T01:42:46.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:46 smithi045 conmon[39718]: mon.0) 413 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2021-07-31T01:42:46.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:46 smithi045 conmon[39718]: cluster 2021-07-31T01: 2021-07-31T01:42:46.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:46 smithi045 conmon[39718]: 42:45.279742+0000 mon.a (mon.0) 414 : cluster [DBG] osdmap e28: 5 total, 4 up, 5 in 2021-07-31T01:42:46.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:46 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T01:42:46.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:46 smithi045 conmon[39718]: 31T01:42:45.279935+0000 mon.a (mon 2021-07-31T01:42:46.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:46 smithi045 conmon[39718]: .0) 415 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-07-31T01:42:46.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:46 smithi045 conmon[39718]: audit 2021-07-31T01:42:45 2021-07-31T01:42:46.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:46 smithi045 conmon[39718]: .283022+0000 mon.b (mon.2) 10 : audit [INF] from='osd.4 [v2:172.21.15.154:6800/4277144246,v1:172.21.15.154:6801/4277144246]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2021-07-31T01:42:46.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:46 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T01:42:46.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:46 smithi045 conmon[39718]: 01:42:45.283524+0000 mon.a (mon.0) 416 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2021-07-31T01:42:46.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:46 smithi045 conmon[39718]: cluster 2021-07-31 2021-07-31T01:42:46.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:46 smithi045 conmon[39718]: T01:42:45.322508+0000 mgr.y (mgr.14146) 131 : cluster [DBG] pgmap v111: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-07-31T01:42:47.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:47 smithi154 conmon[34553]: audit 2021-07-31T01:42:46.286631+0000 mon.a ( 2021-07-31T01:42:47.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:47 smithi154 conmon[34553]: mon.0) 417 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2021-07-31T01:42:47.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:47 smithi154 conmon[34553]: cluster 2021-07-31T01:42:46.286749+0000 mon.a (mon.0) 418 : cluster [DBG] osdmap e29: 5 total, 4 up, 5 in 2021-07-31T01:42:47.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:47 smithi154 conmon[34553]: audit 2021-07-31T01:42:46.287074+0000 mon.a (mon.0) 419 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-07-31T01:42:47.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:47 smithi154 conmon[34553]: audit 2021-07-31T01:42:46.293483+0000 mon.a (mon.0) 420 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-07-31T01:42:47.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:47 smithi154 conmon[34553]: cluster 2021-07-31T01:42:46.425231+0000 mon.a (mon.0) 421 : cluster [INF] osd.4 [v2:172.21.15.154:6800/4277144246,v1:172.21.15.154:6801/4277144246] boot 2021-07-31T01:42:47.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:47 smithi154 conmon[34553]: cluster 2021-07-31T01:42:46.425293+0000 mon.a (mon.0) 422 : cluster [DBG] osdmap e30: 5 total, 5 up, 5 in 2021-07-31T01:42:47.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:47 smithi154 conmon[34553]: audit 2021-07-31T01:42:46.425703+0000 mon.a (mon.0) 423 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-07-31T01:42:47.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:47 smithi045 conmon[33655]: audit 2021-07-31T01:42:46.286631+0000 mon.a (mon. 2021-07-31T01:42:47.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:47 smithi045 conmon[33655]: 0) 417 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2021-07-31T01:42:47.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:47 smithi045 conmon[33655]: cluster 2021-07-31T01:42:46.286749+0000 mon.a (mon.0) 418 : cluster [DBG] osdmap e29: 5 total, 4 up, 5 in 2021-07-31T01:42:47.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:47 smithi045 conmon[33655]: audit 2021-07-31T01:42:47.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:47 smithi045 conmon[33655]: 2021-07-31T01:42:46.287074+0000 mon.a (mon.0) 419 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-07-31T01:42:47.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:47 smithi045 conmon[33655]: audit 2021-07-31T 2021-07-31T01:42:47.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:47 smithi045 conmon[33655]: 01:42:46.293483+0000 mon.a (mon.0) 420 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-07-31T01:42:47.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:47 smithi045 conmon[33655]: cluster 2021-07-31T01:42:46 2021-07-31T01:42:47.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:47 smithi045 conmon[33655]: .425231+0000 mon.a (mon.0) 421 : cluster [INF] osd.4 [v2:172.21.15.154:6800/4277144246,v1:172.21.15.154:6801/4277144246] boot 2021-07-31T01:42:47.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:47 smithi045 conmon[33655]: 2021-07-31T01:42:47.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:47 smithi045 conmon[33655]: cluster 2021-07-31T01:42:46.425293+0000 2021-07-31T01:42:47.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:47 smithi045 conmon[33655]: mon.a (mon.0) 422 : cluster [DBG] osdmap e30: 5 total, 5 up, 5 in 2021-07-31T01:42:47.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:47 smithi045 conmon[33655]: audit 2021-07-31T01:42:46.425703+0000 mon.a (mon.0 2021-07-31T01:42:47.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:47 smithi045 conmon[33655]: ) 423 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-07-31T01:42:47.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:47 smithi045 conmon[39718]: audit 2021-07-31T01:42:46.286631+0000 mon.a ( 2021-07-31T01:42:47.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:47 smithi045 conmon[39718]: mon.0) 417 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2021-07-31T01:42:47.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:47 smithi045 conmon[39718]: cluster 2021 2021-07-31T01:42:47.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:47 smithi045 conmon[39718]: -07-31T01:42:46.286749+0000 mon.a (mon.0) 418 : cluster 2021-07-31T01:42:47.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:47 smithi045 conmon[39718]: [DBG] osdmap e29: 5 total, 4 up, 5 in 2021-07-31T01:42:47.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:47 smithi045 conmon[39718]: audit 2021-07-31T01:42:46.287074+0000 mon.a (mon.0) 419 2021-07-31T01:42:47.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:47 smithi045 conmon[39718]: : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-07-31T01:42:47.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:47 smithi045 conmon[39718]: audit 2021- 2021-07-31T01:42:47.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:47 smithi045 conmon[39718]: 07-31T01:42:46.293483 2021-07-31T01:42:47.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:47 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2021-07-31T01:42:47.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:47 smithi045 conmon[39718]: 420 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-07-31T01:42:47.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:47 smithi045 conmon[39718]: cluster 2021-07-31T01:42:47.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:47 smithi045 conmon[39718]: 2021-07- 2021-07-31T01:42:47.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:47 smithi045 conmon[39718]: 31T01:42:46.425231+0000 mon.a 2021-07-31T01:42:47.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:47 smithi045 conmon[39718]: (mon.0) 421 : cluster [INF] osd.4 [v2:172.21.15.154:6800/4277144246,v1:172.21.15.154:6801/4277144246] boot 2021-07-31T01:42:47.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:47 smithi045 conmon[39718]: cluster 2021-07-31T01: 2021-07-31T01:42:47.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:47 smithi045 conmon[39718]: 42:46.425293+0000 mon.a (mon.0) 422 : cluster 2021-07-31T01:42:47.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:47 smithi045 conmon[39718]: [DBG] osdmap e30: 5 total, 5 up, 5 in 2021-07-31T01:42:47.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:47 smithi045 conmon[39718]: audit 2021 2021-07-31T01:42:47.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:47 smithi045 conmon[39718]: -07-31T01:42:46.425703+0000 mon.a (mon.0) 2021-07-31T01:42:47.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:47 smithi045 conmon[39718]: 423 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-07-31T01:42:48.727 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:48 smithi154 conmon[34553]: cluster 2021-07-31T01:42:45.424103+0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2021-07-31T01:42:48.728 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:48 smithi154 conmon[34553]: cluster 2021-07-31T01:42:45.424191+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2021-07-31T01:42:48.728 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:48 smithi154 conmon[34553]: cluster 2021-07-31T01:42:47.322941+0000 mgr.y (mgr.14146) 132 : cluster [DBG] pgmap v114: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-07-31T01:42:48.728 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:48 smithi154 conmon[34553]: cluster 2021-07-31T01:42:47.431169+0000 mon.a (mon.0) 424 : cluster [DBG] osdmap e31: 5 total, 5 up, 5 in 2021-07-31T01:42:48.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:48 smithi045 conmon[33655]: cluster 2021-07-31T01:42:45.424103 2021-07-31T01:42:48.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:48 smithi045 conmon[33655]: +0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2021-07-31T01:42:48.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:48 smithi045 conmon[33655]: cluster 2021-07-31T01:42:45.424191+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2021-07-31T01:42:48.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:48 smithi045 conmon[33655]: cluster 2021-07-31T01:42:47 2021-07-31T01:42:48.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:48 smithi045 conmon[33655]: .322941+0000 mgr.y (mgr.14146) 132 : cluster [DBG] pgmap v114: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-07-31T01:42:48.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:48 smithi045 conmon[33655]: cluster 2021- 2021-07-31T01:42:48.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:48 smithi045 conmon[33655]: 07-31T01:42:47.431169+0000 2021-07-31T01:42:48.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:48 smithi045 conmon[33655]: mon.a (mon.0) 424 : cluster 2021-07-31T01:42:48.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:48 smithi045 conmon[33655]: [DBG] osdmap e31: 5 total, 5 up, 5 in 2021-07-31T01:42:48.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:48 smithi045 conmon[39718]: cluster 2021-07-31T01:42 2021-07-31T01:42:48.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:48 smithi045 conmon[39718]: :45.424103+0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2021-07-31T01:42:48.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:48 smithi045 conmon[39718]: cluster 2021 2021-07-31T01:42:48.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:48 smithi045 conmon[39718]: -07-31T01:42:45.424191 2021-07-31T01:42:48.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:48 smithi045 conmon[39718]: +0000 osd.4 (osd.4) 2021-07-31T01:42:48.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:48 smithi045 conmon[39718]: 2 : cluster [DBG] purged_snaps scrub ok 2021-07-31T01:42:48.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:48 smithi045 conmon[39718]: cluster 2021 2021-07-31T01:42:48.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:48 smithi045 conmon[39718]: -07-31T01:42:47 2021-07-31T01:42:48.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:48 smithi045 conmon[39718]: .322941+0000 mgr.y (mgr. 2021-07-31T01:42:48.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:48 smithi045 conmon[39718]: 14146) 132 : cluster [DBG] pgmap v114: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-07-31T01:42:48.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:48 smithi045 conmon[39718]: 2021-07-31T01:42:48.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:48 smithi045 conmon[39718]: cluster 2021 2021-07-31T01:42:48.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:48 smithi045 conmon[39718]: -07-31T01:42 2021-07-31T01:42:48.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:48 smithi045 conmon[39718]: :47.431169+0000 mon.a ( 2021-07-31T01:42:48.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:48 smithi045 conmon[39718]: mon.0) 424 : cluster [DBG] 2021-07-31T01:42:48.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:48 smithi045 conmon[39718]: osdmap e31: 5 total, 5 up, 5 in 2021-07-31T01:42:49.717 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:49 smithi154 conmon[34553]: cluster 2021-07-31T01:42:48.435606+0000 mon.a (mon.0) 425 : cluster 2021-07-31T01:42:49.717 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:49 smithi154 conmon[34553]: [DBG] osdmap e32: 5 total, 5 up, 5 in 2021-07-31T01:42:49.718 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:49 smithi154 conmon[34553]: cephadm 2021-07-31T01:42:48.553895+0000 mgr.y (mgr.14146) 133 : cephadm [INF] Detected new or changed devices on smithi154 2021-07-31T01:42:49.718 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:49 smithi154 conmon[34553]: audit 2021-07-31T01:42:48.558633+0000 mon.a (mon.0) 426 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:42:49.718 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:49 smithi154 conmon[34553]: audit 2021-07-31T01:42:48.912047+0000 mon.a (mon.0) 427 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-07-31T01:42:49.718 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:49 smithi154 conmon[34553]: cephadm 2021-07-31T01:42:48.912838+0000 mgr.y (mgr.14146) 134 : cephadm [INF] Adjusting osd_memory_target on smithi154 to 17171M 2021-07-31T01:42:49.719 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:49 smithi154 conmon[34553]: audit 2021-07-31T01:42:48.917822+0000 mon.a (mon.0) 428 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:42:49.719 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:49 smithi154 conmon[34553]: audit 2021-07-31T01:42:48.922434+0000 mon.a (mon.0) 429 : audit [INF] 2021-07-31T01:42:49.719 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:49 smithi154 conmon[34553]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:42:49.720 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:49 smithi154 conmon[34553]: audit 2021-07-31 2021-07-31T01:42:49.720 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:49 smithi154 conmon[34553]: T01:42:48.924515+0000 mon.a ( 2021-07-31T01:42:49.720 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:49 smithi154 conmon[34553]: mon.0) 430 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:42:49.720 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:49 smithi154 conmon[34553]: audit 2021-07-31T01:42:49.720 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:49 smithi154 conmon[34553]: 2021-07-31T01:42 2021-07-31T01:42:49.721 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:49 smithi154 conmon[34553]: :48.925749+0000 mon.a (mon.0) 2021-07-31T01:42:49.721 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:49 smithi154 conmon[34553]: 431 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:42:49.721 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:49 smithi154 conmon[34553]: audit 2021 2021-07-31T01:42:49.721 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:49 smithi154 conmon[34553]: -07-31T01:42:48 2021-07-31T01:42:49.722 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:49 smithi154 conmon[34553]: .926745+0000 mon.a (mon.0) 432 : 2021-07-31T01:42:49.722 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:49 smithi154 conmon[34553]: audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:42:49.722 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:49 smithi154 conmon[34553]: audit 2021- 2021-07-31T01:42:49.722 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:49 smithi154 conmon[34553]: 07-31T01:42:48 2021-07-31T01:42:49.722 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:49 smithi154 conmon[34553]: .931574+0000 mon.a (mon.0) 433 : 2021-07-31T01:42:49.723 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:49 smithi154 conmon[34553]: audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:42:49.723 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:49 smithi154 conmon[34553]: audit 2021- 2021-07-31T01:42:49.723 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:49 smithi154 conmon[34553]: 07-31T01:42:48 2021-07-31T01:42:49.723 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:49 smithi154 conmon[34553]: .937629+0000 mgr.y (mgr.14146) 135 : 2021-07-31T01:42:49.723 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:49 smithi154 conmon[34553]: audit [DBG] from='client.24230 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:42:49.724 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:49 smithi154 conmon[34553]: audit 2021-07 2021-07-31T01:42:49.724 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:49 smithi154 conmon[34553]: -31T01:42:48. 2021-07-31T01:42:49.724 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:49 smithi154 conmon[34553]: 938397+0000 mon.a (mon.0) 434 : audit 2021-07-31T01:42:49.724 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:49 smithi154 conmon[34553]: [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-07-31T01:42:49.724 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:49 smithi154 conmon[34553]: audit 2021-07 2021-07-31T01:42:49.725 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:49 smithi154 conmon[34553]: -31T01:42:48. 2021-07-31T01:42:49.725 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:49 smithi154 conmon[34553]: 939927+0000 mon.a (mon.0) 435 : 2021-07-31T01:42:49.725 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:49 smithi154 conmon[34553]: audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-07-31T01:42:49.725 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:49 smithi154 conmon[34553]: audit 2021- 2021-07-31T01:42:49.726 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:49 smithi154 conmon[34553]: 07-31T01:42:48.940702+0000 mon.a (mon.0) 2021-07-31T01:42:49.726 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:49 smithi154 conmon[34553]: 436 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:42:49.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:49 smithi045 conmon[33655]: cluster 2021-07-31T01:42:48.435606+0000 mon.a (mon.0) 425 : cluster 2021-07-31T01:42:49.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:49 smithi045 conmon[33655]: [DBG] osdmap e32: 5 total, 5 up, 5 in 2021-07-31T01:42:49.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:49 smithi045 conmon[33655]: cephadm 2021-07-31T01:42:48.553895+0000 mgr.y (mgr.14146) 133 : cephadm [INF] Detected new or changed devices on smithi154 2021-07-31T01:42:49.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:49 smithi045 conmon[33655]: 2021-07-31T01:42:49.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:49 smithi045 conmon[33655]: audit 2021-07-31T01:42:48 2021-07-31T01:42:49.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:49 smithi045 conmon[33655]: .558633+0000 mon.a (mon.0) 426 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:42:49.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:49 smithi045 conmon[33655]: audit 2021-07-31T01: 2021-07-31T01:42:49.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:49 smithi045 conmon[33655]: 42:48.912047+0000 mon.a (mon.0) 427 2021-07-31T01:42:49.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:49 smithi045 conmon[33655]: : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-07-31T01:42:49.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:49 smithi045 conmon[33655]: cephadm 2021-07-31T01:42:48.912838+0000 mgr.y (mgr. 2021-07-31T01:42:49.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:49 smithi045 conmon[33655]: 14146) 134 : cephadm [INF] Adjusting osd_memory_target on smithi154 to 17171M 2021-07-31T01:42:49.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:49 smithi045 conmon[33655]: audit 2021-07-31T01:42:49.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:49 smithi045 conmon[33655]: 2021-07-31T01:42:48.917822+0000 mon.a (mon.0) 428 : audit 2021-07-31T01:42:49.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:49 smithi045 conmon[33655]: [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:42:49.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:49 smithi045 conmon[33655]: audit 2021- 2021-07-31T01:42:49.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:49 smithi045 conmon[33655]: 07-31T01:42:48.922434+0000 mon.a (mon.0) 429 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:42:49.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:49 smithi045 conmon[33655]: audit 2021-07-31T01:42: 2021-07-31T01:42:49.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:49 smithi045 conmon[33655]: 48.924515+0000 mon.a (mon.0) 430 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:42:49.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:49 smithi045 conmon[33655]: audit 2021-07-31T01: 2021-07-31T01:42:49.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:49 smithi045 conmon[33655]: 42:48.925749+0000 mon.a (mon.0) 431 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:42:49.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:49 smithi045 conmon[33655]: 2021-07-31T01:42:49.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:49 smithi045 conmon[33655]: audit 2021-07-31T01 2021-07-31T01:42:49.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:49 smithi045 conmon[33655]: :42:48.926745+0000 mon.a (mon.0) 432 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:42:49.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:49 smithi045 conmon[33655]: audit 2021-07-31T01: 2021-07-31T01:42:49.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:49 smithi045 conmon[33655]: 42:48.931574+0000 mon.a (mon.0) 433 : audit 2021-07-31T01:42:49.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:49 smithi045 conmon[33655]: [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:42:49.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:49 smithi045 conmon[33655]: audit 2021 2021-07-31T01:42:49.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:49 smithi045 conmon[33655]: -07-31T01:42:48.937629+0000 mgr.y (mgr.14146) 135 : audit [DBG] 2021-07-31T01:42:49.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:49 smithi045 conmon[33655]: from='client.24230 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:42:49.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:49 smithi045 conmon[33655]: audit 2021-07- 2021-07-31T01:42:49.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:49 smithi045 conmon[33655]: 31T01:42:48.938397+0000 mon.a (mon.0) 434 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-07-31T01:42:49.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:49 smithi045 conmon[33655]: audit 2021-07-31T01:42: 2021-07-31T01:42:49.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:49 smithi045 conmon[33655]: 48.939927+0000 mon.a (mon.0) 435 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-07-31T01:42:49.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:49 smithi045 conmon[33655]: audit 2021-07-31T01: 2021-07-31T01:42:49.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:49 smithi045 conmon[33655]: 42:48.940702+0000 mon.a (mon.0) 436 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:42:49.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:49 smithi045 conmon[33655]: 2021-07-31T01:42:49.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:49 smithi045 conmon[39718]: cluster 2021-07-31T01:42:48.435606+0000 mon.a (mon. 2021-07-31T01:42:49.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:49 smithi045 conmon[39718]: 0) 425 : cluster [DBG] osdmap e32: 5 total, 5 up, 5 in 2021-07-31T01:42:49.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:49 smithi045 conmon[39718]: cephadm 2021-07-31T01:42:48.553895 2021-07-31T01:42:49.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:49 smithi045 conmon[39718]: +0000 mgr.y (mgr.14146) 133 : cephadm [INF] Detected new or changed devices on smithi154 2021-07-31T01:42:49.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:49 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T01:42:49.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:49 smithi045 conmon[39718]: 01:42:48.558633+0000 mon.a (mon.0) 426 : 2021-07-31T01:42:49.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:49 smithi045 conmon[39718]: audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:42:49.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:49 smithi045 conmon[39718]: audit 2021-07 2021-07-31T01:42:49.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:49 smithi045 conmon[39718]: -31T01:42:48.912047+0000 mon.a (mon. 2021-07-31T01:42:49.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:49 smithi045 conmon[39718]: 0) 427 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-07-31T01:42:49.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:49 smithi045 conmon[39718]: cephadm 2021-07-31T01:42:49.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:49 smithi045 conmon[39718]: 2021-07-31T01:42:48.912838+0000 mgr.y ( 2021-07-31T01:42:49.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:49 smithi045 conmon[39718]: mgr.14146) 134 : cephadm [INF] Adjusting osd_memory_target on smithi154 to 17171M 2021-07-31T01:42:49.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:49 smithi045 conmon[39718]: audit 2021- 2021-07-31T01:42:49.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:49 smithi045 conmon[39718]: 07-31T01:42:48.917822+0000 mon.a (mon.0) 428 : audit 2021-07-31T01:42:49.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:49 smithi045 conmon[39718]: [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:42:49.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:49 smithi045 conmon[39718]: audit 2021 2021-07-31T01:42:49.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:49 smithi045 conmon[39718]: -07-31T01:42:48.922434+0000 mon.a ( 2021-07-31T01:42:49.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:49 smithi045 conmon[39718]: mon.0) 429 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:42:49.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:49 smithi045 conmon[39718]: audit 2021-07 2021-07-31T01:42:49.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:49 smithi045 conmon[39718]: -31T01:42:48.924515+0000 mon.a (mon.0) 430 : audit [DBG] 2021-07-31T01:42:49.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:49 smithi045 conmon[39718]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:42:49.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:49 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T01:42:49.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:49 smithi045 conmon[39718]: 31T01:42:48.925749+0000 mon.a (mon 2021-07-31T01:42:49.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:49 smithi045 conmon[39718]: .0) 431 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:42:49.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:49 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T01:42:49.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:49 smithi045 conmon[39718]: 31T01:42:48.926745+0000 mon.a (mon.0) 432 : audit 2021-07-31T01:42:49.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:49 smithi045 conmon[39718]: [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:42:49.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:49 smithi045 conmon[39718]: audit 2021-07-31T01:42:49.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:49 smithi045 conmon[39718]: 2021-07-31T01: 2021-07-31T01:42:49.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:49 smithi045 conmon[39718]: 42:48.931574+0000 mon.a (mon.0) 433 2021-07-31T01:42:49.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:49 smithi045 conmon[39718]: : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:42:49.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:49 smithi045 conmon[39718]: audit 2021-07-31T01:42:49.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:49 smithi045 conmon[39718]: 2021-07-31T01:42:48.937629+0000 mgr.y (mgr. 2021-07-31T01:42:49.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:49 smithi045 conmon[39718]: 14146) 135 : audit [DBG] from='client.24230 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:42:49.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:49 smithi045 conmon[39718]: audit 2021-07-31T01:42:48.938397+0000 2021-07-31T01:42:49.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:49 smithi045 conmon[39718]: mon.a (mon.0) 434 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-07-31T01:42:49.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:49 smithi045 conmon[39718]: 2021-07-31T01:42:49.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:49 smithi045 conmon[39718]: audit 2021-07-31T01: 2021-07-31T01:42:49.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:49 smithi045 conmon[39718]: 42:48.939927+0000 mon.a (mon.0) 435 : 2021-07-31T01:42:49.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:49 smithi045 conmon[39718]: audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-07-31T01:42:49.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:49 smithi045 conmon[39718]: audit 2021-07-31T01:42: 2021-07-31T01:42:49.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:49 smithi045 conmon[39718]: 48.940702+0000 mon.a (mon.0) 436 : audit [DBG] 2021-07-31T01:42:49.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:49 smithi045 conmon[39718]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:42:51.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:50 smithi154 conmon[34553]: cluster 2021-07-31T01:42:49.323338+0000 mgr.y (mgr.14146) 136 : cluster [DBG] pgmap v117: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-07-31T01:42:51.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:50 smithi154 conmon[34553]: 2021-07-31T01:42:51.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:50 smithi045 conmon[39718]: cluster 2021-07-31T01:42:49.323338+0000 mgr.y (mgr.14146) 136 : cluster [DBG] pgmap v117: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-07-31T01:42:51.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:50 smithi045 conmon[33655]: cluster 2021-07-31T01:42:49.323338+0000 mgr.y (mgr.14146) 2021-07-31T01:42:51.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:50 smithi045 conmon[33655]: 136 : cluster [DBG] pgmap v117: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-07-31T01:42:52.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:51 smithi154 conmon[34553]: audit 2021-07-31T01:42:51.360252+0000 mon.b (mon.2) 11 : audit [INF] from='client.? 172.21.15.154:0/4076294704' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7dcad412-f42a-478b-886a-8f4bd82a04ce"}]: dispatch 2021-07-31T01:42:52.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:51 smithi154 conmon[34553]: audit 2021-07-31T01:42:51.360727+0000 mon.a (mon.0) 437 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7dcad412-f42a-478b-886a-8f4bd82a04ce"}]: dispatch 2021-07-31T01:42:52.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:51 smithi154 conmon[34553]: audit 2021-07-31T01:42:51.365300+0000 mon.a (mon.0) 438 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7dcad412-f42a-478b-886a-8f4bd82a04ce"}]': finished 2021-07-31T01:42:52.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:51 smithi154 conmon[34553]: cluster 2021-07-31T01:42:51.365427+0000 mon.a (mon.0) 439 : cluster [DBG] osdmap e33: 6 total, 5 up, 6 in 2021-07-31T01:42:52.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:51 smithi154 conmon[34553]: audit 2021-07-31T01:42:51.365583+0000 mon.a (mon.0) 440 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-07-31T01:42:52.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:51 smithi154 conmon[34553]: audit 2021-07-31T01:42:51.894539+0000 mon.b (mon.2) 12 : audit [DBG] from='client.? 172.21.15.154:0/1391554944' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-07-31T01:42:52.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:51 smithi045 conmon[33655]: audit 2021-07-31T01:42:51.360252+0000 mon.b (mon.2) 11 : audit [INF] from='client.? 172.21.15.154:0/4076294704' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7dcad412-f42a-478b-886a-8f4bd82a04ce"}]: dispatch 2021-07-31T01:42:52.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:51 smithi045 conmon[33655]: audit 2021-07-31T01:42:51.360727+0000 mon.a (mon.0) 437 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7dcad412-f42a-478b-886a-8f4bd82a04ce"}]: dispatch 2021-07-31T01:42:52.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:51 smithi045 conmon[33655]: audit 2021- 2021-07-31T01:42:52.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:51 smithi045 conmon[33655]: 07-31T01:42:51.365300+0000 mon.a (mon.0) 2021-07-31T01:42:52.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:51 smithi045 conmon[33655]: 438 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7dcad412-f42a-478b-886a-8f4bd82a04ce"}]': finished 2021-07-31T01:42:52.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:51 smithi045 conmon[33655]: cluster 2021-07-31T01:42:51.365427+0000 2021-07-31T01:42:52.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:51 smithi045 conmon[33655]: mon.a (mon.0) 439 : cluster [DBG] osdmap e33: 6 total, 5 up, 6 in 2021-07-31T01:42:52.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:51 smithi045 conmon[33655]: audit 2021-07-31T01:42:51 2021-07-31T01:42:52.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:51 smithi045 conmon[33655]: .365583+0000 mon.a (mon.0) 440 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-07-31T01:42:52.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:51 smithi045 conmon[33655]: audit 2021-07-31T01: 2021-07-31T01:42:52.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:51 smithi045 conmon[33655]: 42:51.894539+0000 mon.b (mon.2) 12 : audit [DBG] from='client.? 172.21.15.154:0/1391554944' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-07-31T01:42:52.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:51 smithi045 conmon[39718]: audit 2021-07-31T01:42:51.360252+0000 mon.b (mon.2 2021-07-31T01:42:52.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:51 smithi045 conmon[39718]: ) 11 : audit [INF] from='client.? 172.21.15.154:0/4076294704' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7dcad412-f42a-478b-886a-8f4bd82a04ce"}]: dispatch 2021-07-31T01:42:52.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:51 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T01:42:52.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:51 smithi045 conmon[39718]: T01:42:51.360727+0000 mon.a (mon.0) 437 : audit [INF] 2021-07-31T01:42:52.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:51 smithi045 conmon[39718]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7dcad412-f42a-478b-886a-8f4bd82a04ce"}]: dispatch 2021-07-31T01:42:52.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:51 smithi045 conmon[39718]: audit 2021-07-31T01:42:52.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:51 smithi045 conmon[39718]: 2021-07-31T01:42:51.365300+0000 2021-07-31T01:42:52.282 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:51 smithi045 conmon[39718]: mon.a (mon.0) 438 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7dcad412-f42a-478b-886a-8f4bd82a04ce"}]': finished 2021-07-31T01:42:52.282 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:51 smithi045 conmon[39718]: cluster 2021-07-31T01: 2021-07-31T01:42:52.282 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:51 smithi045 conmon[39718]: 42:51.365427+0000 mon.a (mon.0) 439 : cluster [DBG] osdmap e33: 6 total, 5 up, 6 in 2021-07-31T01:42:52.282 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:51 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T01:42:52.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:51 smithi045 conmon[39718]: 31T01:42:51.365583+0000 mon.a (mon.0) 440 2021-07-31T01:42:52.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:51 smithi045 conmon[39718]: : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-07-31T01:42:52.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:51 smithi045 conmon[39718]: audit 2021-07-31T01:42:51.894539+0000 mon.b (mon 2021-07-31T01:42:52.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:51 smithi045 conmon[39718]: .2) 12 : audit [DBG] from='client.? 172.21.15.154:0/1391554944' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-07-31T01:42:53.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:52 smithi154 conmon[34553]: cluster 2021-07-31T01:42:51.323875+0000 mgr.y (mgr.14146) 137 : cluster [DBG] pgmap v118: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-07-31T01:42:53.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:52 smithi045 conmon[33655]: cluster 2021-07-31T01:42:51.323875+0000 mgr.y (mgr.14146 2021-07-31T01:42:53.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:52 smithi045 conmon[33655]: ) 137 : cluster [DBG] pgmap v118: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-07-31T01:42:53.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:52 smithi045 conmon[39718]: cluster 2021-07-31T01:42:51.323875+0000 mgr.y ( 2021-07-31T01:42:53.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:52 smithi045 conmon[39718]: mgr.14146) 137 : cluster [DBG] pgmap v118: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-07-31T01:42:55.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:54 smithi154 conmon[34553]: cluster 2021-07-31T01:42:53.324276+0000 mgr.y (mgr.14146) 138 : cluster [DBG] pgmap v120: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-07-31T01:42:55.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:54 smithi045 conmon[33655]: cluster 2021-07-31T01:42 2021-07-31T01:42:55.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:54 smithi045 conmon[33655]: :53.324276+0000 mgr.y (mgr.14146) 138 : cluster [DBG] pgmap v120: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-07-31T01:42:55.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:54 smithi045 conmon[39718]: cluster 2021-07-31T01:42:53.324276+0000 mgr.y ( 2021-07-31T01:42:55.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:54 smithi045 conmon[39718]: mgr.14146) 138 : cluster [DBG] pgmap v120: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-07-31T01:42:57.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:56 smithi154 conmon[34553]: cluster 2021-07-31T01:42:55.324681+0000 mgr.y (mgr.14146) 139 : cluster [DBG] pgmap v121: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-07-31T01:42:57.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:56 smithi045 conmon[33655]: cluster 2021-07-31T01:42:55.324681+0000 mgr.y (mgr.14146) 139 : 2021-07-31T01:42:57.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:56 smithi045 conmon[33655]: cluster [DBG] pgmap v121: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-07-31T01:42:57.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:56 smithi045 conmon[39718]: cluster 2021-07-31T01:42:55.324681+0000 mgr.y (mgr.14146) 139 : cluster [DBG] 2021-07-31T01:42:57.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:56 smithi045 conmon[39718]: pgmap v121: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-07-31T01:42:59.202 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:58 smithi154 conmon[34553]: cluster 2021-07-31T01:42:57.325056+0000 mgr.y (mgr.14146) 140 : cluster [DBG] pgmap v122: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-07-31T01:42:59.203 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:58 smithi154 conmon[34553]: audit 2021-07-31T01:42:58.047620+0000 mon.a (mon.0) 441 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2021-07-31T01:42:59.203 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:58 smithi154 conmon[34553]: audit 2021-07-31T01:42:58.048536+0000 mon.a (mon.0) 442 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:42:59.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:58 smithi045 conmon[33655]: cluster 2021-07-31T01:42:57.325056+0000 mgr.y (mgr.14146 2021-07-31T01:42:59.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:58 smithi045 conmon[33655]: ) 140 : cluster [DBG] pgmap v122: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-07-31T01:42:59.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:58 smithi045 conmon[33655]: audit 2021-07-31T01:42:58.047620+0000 mon.a (mon.0) 441 : audit [INF] 2021-07-31T01:42:59.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:58 smithi045 conmon[33655]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2021-07-31T01:42:59.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:58 smithi045 conmon[33655]: audit 2021- 2021-07-31T01:42:59.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:58 smithi045 conmon[33655]: 07-31T01:42:58.048536+0000 mon.a (mon.0) 442 : 2021-07-31T01:42:59.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:58 smithi045 conmon[33655]: audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:42:59.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:58 smithi045 conmon[39718]: cluster 2021-07-31T01:42:57.325056+0000 mgr.y (mgr.14146) 140 2021-07-31T01:42:59.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:58 smithi045 conmon[39718]: : cluster [DBG] pgmap v122: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-07-31T01:42:59.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:58 smithi045 conmon[39718]: audit 2021-07-31T01:42:58.047620 2021-07-31T01:42:59.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:58 smithi045 conmon[39718]: +0000 mon.a (mon.0) 441 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2021-07-31T01:42:59.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:58 smithi045 conmon[39718]: audit 2021-07-31T01:42:59.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:58 smithi045 conmon[39718]: 2021-07-31T01:42:58 2021-07-31T01:42:59.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:58 smithi045 conmon[39718]: .048536+0000 mon.a (mon.0) 442 : audit 2021-07-31T01:42:59.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:58 smithi045 conmon[39718]: [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:43:00.159 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:59 smithi154 conmon[34553]: cephadm 2021-07-31T01:42:58.049259+0000 2021-07-31T01:43:00.159 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:42:59 smithi154 conmon[34553]: mgr.y (mgr.14146) 141 : cephadm [INF] Deploying daemon osd.5 on smithi154 2021-07-31T01:43:00.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:59 smithi045 conmon[39718]: cephadm 2021-07-31T01:42:58.049259+0000 mgr.y (mgr.14146) 2021-07-31T01:43:00.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:42:59 smithi045 conmon[39718]: 141 : cephadm [INF] Deploying daemon osd.5 on smithi154 2021-07-31T01:43:00.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:59 smithi045 conmon[33655]: cephadm 2021-07-31T01:42:58.049259+0000 mgr.y (mgr.14146) 2021-07-31T01:43:00.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:42:59 smithi045 conmon[33655]: 141 : cephadm [INF] Deploying daemon osd.5 on smithi154 2021-07-31T01:43:01.250 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:00 smithi154 conmon[34553]: cluster 2021-07-31T01:42:59.325416+0000 mgr.y (mgr.14146) 142 : cluster [DBG] pgmap v123: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-07-31T01:43:01.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:00 smithi045 conmon[33655]: cluster 2021-07-31T01:42:59.325416+0000 mgr.y (mgr 2021-07-31T01:43:01.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:00 smithi045 conmon[33655]: .14146) 142 : cluster [DBG] pgmap v123: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-07-31T01:43:01.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:00 smithi045 conmon[39718]: cluster 2021-07-31T01:42:59.325416+0000 mgr.y 2021-07-31T01:43:01.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:00 smithi045 conmon[39718]: (mgr.14146) 142 : cluster [DBG] pgmap v123: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-07-31T01:43:01.880 INFO:teuthology.orchestra.run.smithi154.stdout:Created osd(s) 5 on host 'smithi154' 2021-07-31T01:43:02.594 DEBUG:teuthology.orchestra.run.smithi154:osd.5> sudo journalctl -f -n 0 -u ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c@osd.5.service 2021-07-31T01:43:02.597 INFO:tasks.cephadm:Deploying osd.6 on smithi154 with /dev/vg_nvme/lv_2... 2021-07-31T01:43:02.597 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f52d2c14-f19f-11eb-8c24-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2021-07-31T01:43:02.626 INFO:journalctl@ceph.osd.5.smithi154.stdout:-- Logs begin at Sat 2021-07-31 01:17:02 UTC. -- 2021-07-31T01:43:02.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:02 smithi154 conmon[34553]: cluster 2021-07-31T01:43:01.325914+0000 mgr.y (mgr.14146 2021-07-31T01:43:02.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:02 smithi154 conmon[34553]: ) 143 : cluster [DBG] pgmap v124: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-07-31T01:43:02.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:02 smithi154 conmon[34553]: audit 2021-07-31T01:43:01.868609+0000 mon.a (mon.0) 443 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:43:02.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:02 smithi154 conmon[34553]: audit 2021-07-31T01:43:01.870355+0000 mon.a (mon.0) 444 : audit 2021-07-31T01:43:02.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:02 smithi154 conmon[34553]: [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:43:02.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:02 smithi154 conmon[34553]: audit 2021-07-31T01:43:01.872901+0000 mon.a (mon. 2021-07-31T01:43:02.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:02 smithi154 conmon[34553]: 0) 445 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:43:02.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:02 smithi154 conmon[34553]: audit 2021 2021-07-31T01:43:02.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:02 smithi154 conmon[34553]: -07-31T01:43:01.874689+0000 2021-07-31T01:43:02.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:02 smithi154 conmon[34553]: mon.a (mon.0) 446 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:43:02.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:02 smithi154 conmon[34553]: 2021-07-31T01:43:02.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:02 smithi154 conmon[34553]: audit 2021-07-31T01:43:02.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:02 smithi154 conmon[34553]: 2021-07-31T01:43 2021-07-31T01:43:02.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:02 smithi154 conmon[34553]: :01.877567+0000 mon.a (mon 2021-07-31T01:43:02.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:02 smithi154 conmon[34553]: .0) 447 : audit [INF] 2021-07-31T01:43:02.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:02 smithi154 conmon[34553]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:43:02.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:02 smithi045 conmon[39718]: cluster 2021-07-31T01:43:01.325914+0000 mgr.y ( 2021-07-31T01:43:02.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:02 smithi045 conmon[39718]: mgr.14146) 143 : cluster [DBG] pgmap v124: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-07-31T01:43:02.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:02 smithi045 conmon[39718]: audit 2021- 2021-07-31T01:43:02.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:02 smithi045 conmon[39718]: 07-31T01:43:01.868609+0000 mon.a (mon.0) 443 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:43:02.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:02 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T01:43:02.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:02 smithi045 conmon[39718]: T01:43:01.870355+0000 mon.a (mon.0) 444 : audit 2021-07-31T01:43:02.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:02 smithi045 conmon[39718]: [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:43:02.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:02 smithi045 conmon[39718]: audit 2021-07-31T01:43:01 2021-07-31T01:43:02.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:02 smithi045 conmon[39718]: .872901+0000 mon.a (mon.0) 445 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:43:02.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:02 smithi045 conmon[39718]: 2021-07-31T01:43:02.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:02 smithi045 conmon[39718]: audit 2021-07-31T01:43:02.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:02 smithi045 conmon[39718]: 2021-07-31T01:43:01.874689 2021-07-31T01:43:02.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:02 smithi045 conmon[39718]: +0000 mon.a (mon.0) 446 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:43:02.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:02 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T01:43:02.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:02 smithi045 conmon[39718]: 31T01:43:01.877567+0000 mon.a (mon.0) 447 : audit [INF] 2021-07-31T01:43:02.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:02 smithi045 conmon[39718]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:43:02.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:02 smithi045 conmon[33655]: cluster 2021-07-31T01:43:01.325914+0000 mgr.y (mgr.14146) 143 2021-07-31T01:43:02.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:02 smithi045 conmon[33655]: : cluster [DBG] pgmap v124: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-07-31T01:43:02.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:02 smithi045 conmon[33655]: audit 2021-07-31T01:43:01.868609+0000 mon.a (mon.0) 443 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:43:02.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:02 smithi045 conmon[33655]: audit 2021-07-31T01:43: 2021-07-31T01:43:02.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:02 smithi045 conmon[33655]: 01.870355+0000 mon.a (mon.0) 444 : 2021-07-31T01:43:02.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:02 smithi045 conmon[33655]: audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:43:02.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:02 smithi045 conmon[33655]: audit 2021-07-31T 2021-07-31T01:43:02.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:02 smithi045 conmon[33655]: 01:43:01.872901+0000 mon.a ( 2021-07-31T01:43:02.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:02 smithi045 conmon[33655]: mon.0) 445 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:43:02.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:02 smithi045 conmon[33655]: 2021-07-31T01:43:02.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:02 smithi045 conmon[33655]: audit 2021-07-31T01:43:02.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:02 smithi045 conmon[33655]: 2021-07-31T01: 2021-07-31T01:43:02.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:02 smithi045 conmon[33655]: 43:01.874689+0000 mon.a 2021-07-31T01:43:02.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:02 smithi045 conmon[33655]: (mon.0) 446 : audit 2021-07-31T01:43:02.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:02 smithi045 conmon[33655]: [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:43:02.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:02 smithi045 conmon[33655]: audit 2021- 2021-07-31T01:43:02.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:02 smithi045 conmon[33655]: 07-31T01:43:01.877567+0000 2021-07-31T01:43:02.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:02 smithi045 conmon[33655]: mon.a (mon.0) 447 : audit 2021-07-31T01:43:02.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:02 smithi045 conmon[33655]: [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:43:04.853 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:04 smithi154 conmon[34553]: cluster 2021-07-31T01:43:03.326432+0000 mgr.y (mgr.14146) 144 : cluster [DBG] pgmap v125: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-07-31T01:43:04.853 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:04 smithi154 conmon[34553]: audit 2021-07-31T01:43:03.713107+0000 mon.a (mon.0) 448 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:43:05.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:04 smithi045 conmon[33655]: cluster 2021-07-31T01:43:03.326432+0000 mgr.y (mgr.14146) 144 : 2021-07-31T01:43:05.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:04 smithi045 conmon[33655]: cluster [DBG] pgmap v125: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-07-31T01:43:05.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:04 smithi045 conmon[33655]: audit 2021-07-31T01:43:03.713107+0000 mon.a (mon.0) 448 : 2021-07-31T01:43:05.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:04 smithi045 conmon[33655]: audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:43:05.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:04 smithi045 conmon[39718]: cluster 2021-07-31T01:43:03.326432+0000 mgr.y (mgr. 2021-07-31T01:43:05.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:04 smithi045 conmon[39718]: 14146) 144 : cluster [DBG] pgmap v125: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-07-31T01:43:05.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:04 smithi045 conmon[39718]: audit 2021-07-31T01:43:03.713107+0000 mon.a 2021-07-31T01:43:05.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:04 smithi045 conmon[39718]: (mon.0) 448 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:43:05.252 INFO:journalctl@ceph.osd.5.smithi154.stdout:Jul 31 01:43:05 smithi154 conmon[42050]: debug 2021-07-31T01:43:05.030+0000 7fc9dacde080 -1 osd.5 0 log_to_monitors {default=true} 2021-07-31T01:43:05.708 INFO:teuthology.orchestra.run.smithi154.stderr:--> Zapping: /dev/vg_nvme/lv_2 2021-07-31T01:43:05.708 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2021-07-31T01:43:05.709 INFO:teuthology.orchestra.run.smithi154.stderr: stderr: 10+0 records in 2021-07-31T01:43:05.709 INFO:teuthology.orchestra.run.smithi154.stderr:10+0 records out 2021-07-31T01:43:05.709 INFO:teuthology.orchestra.run.smithi154.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0140816 s, 745 MB/s 2021-07-31T01:43:05.710 INFO:teuthology.orchestra.run.smithi154.stderr:--> Zapping successful for: 2021-07-31T01:43:05.982 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:05 smithi154 conmon[34553]: audit 2021-07-31T01:43:05.033953+0000 mon.b (mon.2) 13 : 2021-07-31T01:43:05.983 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:05 smithi154 conmon[34553]: audit [INF] from='osd.5 [v2:172.21.15.154:6808/579203063,v1:172.21.15.154:6809/579203063]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2021-07-31T01:43:05.983 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:05 smithi154 conmon[34553]: audit 2021-07-31T01:43:05.034290+0000 mon.a (mon.0) 449 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2021-07-31T01:43:06.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:05 smithi045 conmon[33655]: audit 2021-07-31T01:43:05.033953+0000 mon.b (mon.2) 13 : audit [INF] from='osd.5 [v2:172.21.15.154:6808/579203063,v1:172.21.15.154:6809/579203063]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2021-07-31T01:43:06.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:05 smithi045 conmon[33655]: 2021-07-31T01:43:06.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:05 smithi045 conmon[33655]: audit 2021-07-31T01:43:05.034290+0000 mon.a (mon.0) 449 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2021-07-31T01:43:06.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:05 smithi045 conmon[39718]: audit 2021-07-31T01:43:05.033953+0000 mon.b (mon.2) 2021-07-31T01:43:06.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:05 smithi045 conmon[39718]: 13 : audit [INF] from='osd.5 [v2:172.21.15.154:6808/579203063,v1:172.21.15.154:6809/579203063]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2021-07-31T01:43:06.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:05 smithi045 conmon[39718]: audit 2021-07-31T01:43:05.034290+0000 mon.a (mon.0 2021-07-31T01:43:06.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:05 smithi045 conmon[39718]: ) 449 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2021-07-31T01:43:06.624 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f52d2c14-f19f-11eb-8c24-001a4aab830c -- ceph orch daemon add osd smithi154:vg_nvme/lv_2 2021-07-31T01:43:07.003 INFO:journalctl@ceph.osd.5.smithi154.stdout:Jul 31 01:43:06 smithi154 conmon[42050]: debug 2021-07-31T01:43:06.726+0000 7fc9d1241700 -1 osd.5 0 waiting for initial osdmap 2021-07-31T01:43:07.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:06 smithi154 conmon[34553]: cluster 2021-07-31T01:43:05.326931+0000 mgr.y (mgr.14146) 145 : cluster [DBG] pgmap v126: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-07-31T01:43:07.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:06 smithi154 conmon[34553]: audit 2021-07-31T01:43:05.720243+0000 mon.a (mon.0) 450 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2021-07-31T01:43:07.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:06 smithi154 conmon[34553]: cluster 2021-07-31T01:43:05.720342+0000 mon.a (mon.0) 451 : cluster [DBG] osdmap e34: 6 total, 5 up, 6 in 2021-07-31T01:43:07.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:06 smithi154 conmon[34553]: audit 2021-07-31T01:43:05.720515+0000 mon.a (mon.0) 452 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-07-31T01:43:07.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:06 smithi154 conmon[34553]: audit 2021-07-31T01:43:05.720891+0000 mon.b (mon.2) 14 : audit [INF] from='osd.5 [v2:172.21.15.154:6808/579203063,v1:172.21.15.154:6809/579203063]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2021-07-31T01:43:07.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:06 smithi154 conmon[34553]: audit 2021-07-31T01:43:05.721215+0000 mon.a (mon.0) 453 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2021-07-31T01:43:07.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:06 smithi045 conmon[33655]: cluster 2021-07-31T01:43:05.326931+0000 mgr.y (mgr.14146) 145 : 2021-07-31T01:43:07.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:06 smithi045 conmon[33655]: cluster [DBG] pgmap v126: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-07-31T01:43:07.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:06 smithi045 conmon[33655]: audit 2021-07-31T01:43:05.720243+0000 mon.a (mon.0) 450 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2021-07-31T01:43:07.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:06 smithi045 conmon[33655]: cluster 2021-07-31T01:43:05.720342+0000 mon.a (mon.0) 451 : cluster [DBG] osdmap e34: 6 total, 5 up, 6 in 2021-07-31T01:43:07.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:06 smithi045 conmon[33655]: audit 2021-07-31T01:43:07.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:06 smithi045 conmon[33655]: 2021-07-31 2021-07-31T01:43:07.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:06 smithi045 conmon[33655]: T01:43:05.720515+0000 mon.a (mon.0) 2021-07-31T01:43:07.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:06 smithi045 conmon[33655]: 452 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-07-31T01:43:07.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:06 smithi045 conmon[33655]: audit 2021-07-31T01:43: 2021-07-31T01:43:07.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:06 smithi045 conmon[33655]: 05.720891+0000 mon.b (mon.2) 14 : audit [INF] from='osd.5 [v2:172.21.15.154:6808/579203063,v1:172.21.15.154:6809/579203063]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2021-07-31T01:43:07.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:06 smithi045 conmon[33655]: audit 2021-07-31T01:43:05.721215+0000 mon.a (mon.0) 453 : audit [INF] 2021-07-31T01:43:07.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:06 smithi045 conmon[33655]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2021-07-31T01:43:07.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:06 smithi045 conmon[39718]: cluster 2021-07-31T01:43:05.326931+0000 mgr.y (mgr 2021-07-31T01:43:07.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:06 smithi045 conmon[39718]: .14146) 145 : cluster [DBG] pgmap v126: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-07-31T01:43:07.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:06 smithi045 conmon[39718]: audit 2021-07-31T01:43: 2021-07-31T01:43:07.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:06 smithi045 conmon[39718]: 05.720243+0000 mon.a (mon.0) 450 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2021-07-31T01:43:07.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:06 smithi045 conmon[39718]: cluster 2021-07 2021-07-31T01:43:07.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:06 smithi045 conmon[39718]: -31T01:43:05.720342+0000 mon.a ( 2021-07-31T01:43:07.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:06 smithi045 conmon[39718]: mon.0) 451 : cluster [DBG] osdmap e34: 6 total, 5 up, 6 in 2021-07-31T01:43:07.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:06 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T01:43:07.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:06 smithi045 conmon[39718]: 01:43:05.720515+0000 mon.a (mon.0) 452 : 2021-07-31T01:43:07.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:06 smithi045 conmon[39718]: audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-07-31T01:43:07.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:06 smithi045 conmon[39718]: audit 2021-07-31T01:43:07.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:06 smithi045 conmon[39718]: 2021-07- 2021-07-31T01:43:07.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:06 smithi045 conmon[39718]: 31T01:43:05.720891+0000 mon.b (mon.2) 14 : audit [INF] from='osd.5 [v2:172.21.15.154:6808/579203063,v1:172.21.15.154:6809/579203063]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2021-07-31T01:43:07.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:06 smithi045 conmon[39718]: audit 2021-07-31T01:43:07.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:06 smithi045 conmon[39718]: 2021-07-31T01:43: 2021-07-31T01:43:07.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:06 smithi045 conmon[39718]: 05.721215+0000 mon.a (mon.0) 453 : 2021-07-31T01:43:07.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:06 smithi045 conmon[39718]: audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2021-07-31T01:43:07.995 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:07 smithi154 conmon[34553]: audit 2021-07-31T01:43:06.721170+0000 mon.a (mon.0) 454 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2021-07-31T01:43:07.995 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:07 smithi154 conmon[34553]: cluster 2021-07-31T01:43:06.721277+0000 mon.a (mon.0) 455 : cluster [DBG] osdmap e35: 6 total, 5 up, 6 in 2021-07-31T01:43:07.996 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:07 smithi154 conmon[34553]: audit 2021-07-31T01:43:06.721854+0000 mon.a (mon.0) 456 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-07-31T01:43:07.996 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:07 smithi154 conmon[34553]: audit 2021-07-31T01:43:06.727536+0000 mon.a (mon.0) 457 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-07-31T01:43:07.996 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:07 smithi154 conmon[34553]: cluster 2021-07-31T01:43:07.722739+0000 mon.a (mon.0) 458 : cluster [INF] osd.5 [v2:172.21.15.154:6808/579203063,v1:172.21.15.154:6809/579203063] boot 2021-07-31T01:43:08.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:07 smithi045 conmon[33655]: audit 2021-07-31T 2021-07-31T01:43:08.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:07 smithi045 conmon[33655]: 01:43:06.721170+0000 mon.a (mon.0) 454 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2021-07-31T01:43:08.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:07 smithi045 conmon[33655]: cluster 2021-07-31T01:43:06.721277+0000 mon.a (mon.0) 455 : cluster [DBG] 2021-07-31T01:43:08.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:07 smithi045 conmon[33655]: osdmap e35: 6 total, 5 up, 6 in 2021-07-31T01:43:08.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:07 smithi045 conmon[33655]: audit 2021-07-31T01:43: 2021-07-31T01:43:08.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:07 smithi045 conmon[33655]: 06.721854+0000 mon.a (mon.0) 456 : audit [DBG] 2021-07-31T01:43:08.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:07 smithi045 conmon[33655]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-07-31T01:43:08.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:07 smithi045 conmon[33655]: audit 2021-07-31T01:43 2021-07-31T01:43:08.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:07 smithi045 conmon[33655]: :06.727536+0000 mon.a (mon.0) 2021-07-31T01:43:08.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:07 smithi045 conmon[33655]: 457 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-07-31T01:43:08.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:07 smithi045 conmon[33655]: cluster 2021-07-31T01: 2021-07-31T01:43:08.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:07 smithi045 conmon[33655]: 43:07.722739+0000 mon.a (mon.0) 458 : cluster [INF] 2021-07-31T01:43:08.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:07 smithi045 conmon[33655]: osd.5 [v2:172.21.15.154:6808/579203063,v1:172.21.15.154:6809/579203063] boot 2021-07-31T01:43:08.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:07 smithi045 conmon[39718]: audit 2021-07-31T01:43:06.721170+0000 mon.a (mon.0 2021-07-31T01:43:08.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:07 smithi045 conmon[39718]: ) 454 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2021-07-31T01:43:08.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:07 smithi045 conmon[39718]: cluster 2021-07-31T01:43:06.721277+0000 2021-07-31T01:43:08.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:07 smithi045 conmon[39718]: mon.a (mon.0) 455 : cluster [DBG] osdmap e35: 6 total, 5 up, 6 in 2021-07-31T01:43:08.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:07 smithi045 conmon[39718]: audit 2021-07-31T01:43 2021-07-31T01:43:08.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:07 smithi045 conmon[39718]: :06.721854+0000 mon.a (mon.0) 456 : audit 2021-07-31T01:43:08.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:07 smithi045 conmon[39718]: [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-07-31T01:43:08.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:07 smithi045 conmon[39718]: audit 2021- 2021-07-31T01:43:08.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:07 smithi045 conmon[39718]: 07-31T01:43:06.727536+0000 mon.a (mon 2021-07-31T01:43:08.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:07 smithi045 conmon[39718]: .0) 457 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-07-31T01:43:08.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:07 smithi045 conmon[39718]: cluster 2021-07-31T01: 2021-07-31T01:43:08.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:07 smithi045 conmon[39718]: 43:07.722739+0000 mon.a (mon.0) 458 : cluster [INF] osd.5 [v2:172.21.15.154:6808/579203063,v1:172.21.15.154:6809/579203063] boot 2021-07-31T01:43:08.800 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:08 smithi154 conmon[34553]: cluster 2021-07-31T01:43:06.036373+0000 osd.5 (osd.5 2021-07-31T01:43:08.801 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:08 smithi154 conmon[34553]: ) 1 : cluster [DBG] purged_snaps scrub starts 2021-07-31T01:43:08.801 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:08 smithi154 conmon[34553]: cluster 2021-07-31T01:43:06.036559+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2021-07-31T01:43:08.801 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:08 smithi154 conmon[34553]: cluster 2021-07-31T01:43:07.327381+0000 mgr.y (mgr.14146) 146 : cluster [DBG] pgmap v129: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-07-31T01:43:08.802 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:08 smithi154 conmon[34553]: cluster 2021-07-31T01:43:07.722809+0000 mon.a (mon 2021-07-31T01:43:08.802 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:08 smithi154 conmon[34553]: .0) 459 : cluster [DBG] osdmap e36: 6 total, 6 up, 6 in 2021-07-31T01:43:08.802 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:08 smithi154 conmon[34553]: audit 2021-07-31T01:43:08.802 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:08 smithi154 conmon[34553]: 2021-07-31T01:43:07.722996+0000 2021-07-31T01:43:08.802 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:08 smithi154 conmon[34553]: mon.a (mon.0) 460 : audit [DBG] 2021-07-31T01:43:08.803 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:08 smithi154 conmon[34553]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-07-31T01:43:08.803 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:08 smithi154 conmon[34553]: audit 2021-07-31T01:43 2021-07-31T01:43:08.803 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:08 smithi154 conmon[34553]: :08.317135+0000 mon.a (mon.0) 461 2021-07-31T01:43:08.803 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:08 smithi154 conmon[34553]: : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:43:08.804 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:08 smithi154 conmon[34553]: audit 2021- 2021-07-31T01:43:08.804 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:08 smithi154 conmon[34553]: 07-31T01:43:08.318190+0000 2021-07-31T01:43:08.804 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:08 smithi154 conmon[34553]: mon.a (mon.0) 462 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-07-31T01:43:08.804 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:08 smithi154 conmon[34553]: 2021-07-31T01:43:08.804 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:08 smithi154 conmon[34553]: audit 2021-07-31T01: 2021-07-31T01:43:08.805 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:08 smithi154 conmon[34553]: 43:08.319384+0000 mon.a (mon.0 2021-07-31T01:43:08.805 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:08 smithi154 conmon[34553]: ) 463 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-07-31T01:43:08.805 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:08 smithi154 conmon[34553]: audit 2021 2021-07-31T01:43:08.805 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:08 smithi154 conmon[34553]: -07-31T01:43:08.324321+0000 2021-07-31T01:43:08.806 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:08 smithi154 conmon[34553]: mon.a (mon.0) 464 : audit [INF] 2021-07-31T01:43:08.806 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:08 smithi154 conmon[34553]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:43:08.806 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:08 smithi154 conmon[34553]: audit 2021-07-31T01 2021-07-31T01:43:08.806 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:08 smithi154 conmon[34553]: :43:08.328824+0000 mon.a (mon. 2021-07-31T01:43:08.807 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:08 smithi154 conmon[34553]: 0) 465 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:43:08.807 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:08 smithi154 conmon[34553]: audit 2021-07-31T01:43:08.807 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:08 smithi154 conmon[34553]: 2021-07-31T01:43:08.330895 2021-07-31T01:43:08.807 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:08 smithi154 conmon[34553]: +0000 mon.a (mon.0) 466 : audit [DBG] 2021-07-31T01:43:08.807 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:08 smithi154 conmon[34553]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:43:08.808 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:08 smithi154 conmon[34553]: audit 2021-07-31T01: 2021-07-31T01:43:08.808 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:08 smithi154 conmon[34553]: 43:08.332060+0000 mon.a (mon.0 2021-07-31T01:43:08.808 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:08 smithi154 conmon[34553]: ) 467 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:43:08.808 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:08 smithi154 conmon[34553]: audit 2021-07-31T01:43:08.809 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:08 smithi154 conmon[34553]: 2021-07-31T01:43:08.333244+0000 mon.a (mon.0) 2021-07-31T01:43:08.809 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:08 smithi154 conmon[34553]: 468 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:43:08.809 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:08 smithi154 conmon[34553]: audit 2021 2021-07-31T01:43:08.810 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:08 smithi154 conmon[34553]: -07-31T01:43:08.338147+0000 mon.a (mon. 2021-07-31T01:43:08.810 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:08 smithi154 conmon[34553]: 0) 469 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:43:09.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[39718]: cluster 2021-07-31T01:43:06.036373 2021-07-31T01:43:09.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[39718]: +0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2021-07-31T01:43:09.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[39718]: cluster 2021-07-31T 2021-07-31T01:43:09.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[39718]: 01:43:06.036559+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2021-07-31T01:43:09.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[39718]: cluster 2021-07-31 2021-07-31T01:43:09.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[39718]: T01:43:07.327381+0000 mgr.y (mgr.14146 2021-07-31T01:43:09.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[39718]: ) 146 : cluster [DBG] pgmap v129: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-07-31T01:43:09.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[39718]: cluster 2021 2021-07-31T01:43:09.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[39718]: -07-31T01:43:07.722809+0000 2021-07-31T01:43:09.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[39718]: mon.a (mon.0) 459 : cluster [DBG] osdmap e36: 6 total, 6 up, 6 in 2021-07-31T01:43:09.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[39718]: 2021-07-31T01:43:09.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[39718]: audit 2021-07 2021-07-31T01:43:09.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[39718]: -31T01:43:07.722996+0000 mon.a (mon.0) 460 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-07-31T01:43:09.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[39718]: audit 2021-07-31T01:43:09.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[39718]: 2021-07-31T01 2021-07-31T01:43:09.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[39718]: :43:08.317135+0000 mon.a (mon.0) 461 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:43:09.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[39718]: audit 2021-07-31T01:43:08 2021-07-31T01:43:09.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[39718]: .318190+0000 mon.a (mon.0) 462 : audit [INF] 2021-07-31T01:43:09.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[39718]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-07-31T01:43:09.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[39718]: audit 2021-07-31T01:43:09.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[39718]: 2021-07-31T 2021-07-31T01:43:09.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[39718]: 01:43:08.319384+0000 mon.a (mon.0) 463 2021-07-31T01:43:09.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[39718]: : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-07-31T01:43:09.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[39718]: audit 2021 2021-07-31T01:43:09.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[39718]: -07-31T01:43:08.324321 2021-07-31T01:43:09.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[39718]: +0000 mon.a (mon.0) 464 : audit 2021-07-31T01:43:09.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[39718]: [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:43:09.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[39718]: audit 2021-07 2021-07-31T01:43:09.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[39718]: -31T01:43:08.328824+0000 mon.a (mon.0) 465 : 2021-07-31T01:43:09.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[39718]: audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:43:09.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[39718]: audit 2021-07 2021-07-31T01:43:09.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[39718]: -31T01:43:08.330895+0000 mon.a (mon 2021-07-31T01:43:09.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[39718]: .0) 466 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:43:09.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[39718]: audit 2021- 2021-07-31T01:43:09.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[39718]: 07-31T01:43:08.332060+0000 mon.a (mon 2021-07-31T01:43:09.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[39718]: .0) 467 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:43:09.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[39718]: audit 2021 2021-07-31T01:43:09.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[39718]: -07-31T01:43:08.333244+0000 2021-07-31T01:43:09.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[39718]: mon.a (mon.0) 468 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:43:09.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[39718]: 2021-07-31T01:43:09.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[39718]: audit 2021-07-31T01:43 2021-07-31T01:43:09.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[39718]: :08.338147+0000 mon.a (mon.0) 469 2021-07-31T01:43:09.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[39718]: : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:43:09.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[33655]: cluster 2021-07-31T01:43:06.036373+0000 osd.5 (osd.5 2021-07-31T01:43:09.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[33655]: ) 1 : cluster [DBG] purged_snaps scrub starts 2021-07-31T01:43:09.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[33655]: cluster 2021-07-31T01:43:06.036559+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2021-07-31T01:43:09.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[33655]: cluster 2021-07-31T01:43:07.327381 2021-07-31T01:43:09.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[33655]: +0000 mgr.y (mgr.14146) 146 : cluster [DBG] 2021-07-31T01:43:09.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[33655]: pgmap v129: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-07-31T01:43:09.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[33655]: cluster 2021-07-31T 2021-07-31T01:43:09.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[33655]: 01:43:07.722809+0000 mon.a (mon. 2021-07-31T01:43:09.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[33655]: 0) 459 : cluster [DBG] osdmap e36: 6 total, 6 up, 6 in 2021-07-31T01:43:09.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[33655]: audit 2021 2021-07-31T01:43:09.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[33655]: -07-31T01:43:07.722996 2021-07-31T01:43:09.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[33655]: +0000 mon.a (mon.0) 460 : audit 2021-07-31T01:43:09.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[33655]: [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-07-31T01:43:09.040 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[33655]: audit 2021-07-31T01 2021-07-31T01:43:09.040 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[33655]: :43:08.317135+0000 2021-07-31T01:43:09.040 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[33655]: mon.a (mon.0) 461 : audit 2021-07-31T01:43:09.040 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[33655]: [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:43:09.040 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[33655]: audit 2021-07-31T01:43:09.041 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[33655]: 2021-07-31T01:43:08.318190+0000 2021-07-31T01:43:09.041 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[33655]: mon.a (mon.0) 462 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-07-31T01:43:09.041 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[33655]: audit 2021 2021-07-31T01:43:09.041 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[33655]: -07-31T01:43:08 2021-07-31T01:43:09.042 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[33655]: .319384+0000 mon.a (mon 2021-07-31T01:43:09.042 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[33655]: .0) 463 : audit [INF] 2021-07-31T01:43:09.042 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[33655]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-07-31T01:43:09.042 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[33655]: audit 2021 2021-07-31T01:43:09.042 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[33655]: -07-31T01:43:08 2021-07-31T01:43:09.043 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[33655]: .324321+0000 mon.a (mon. 2021-07-31T01:43:09.043 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[33655]: 0) 464 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:43:09.043 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[33655]: 2021-07-31T01:43:09.043 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[33655]: audit 2021 2021-07-31T01:43:09.044 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[33655]: -07-31T01:43:08. 2021-07-31T01:43:09.044 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[33655]: 328824+0000 mon.a (mon. 2021-07-31T01:43:09.044 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[33655]: 0) 465 : audit [INF] 2021-07-31T01:43:09.044 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[33655]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:43:09.044 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[33655]: audit 2021-07 2021-07-31T01:43:09.045 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[33655]: -31T01:43:08 2021-07-31T01:43:09.045 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[33655]: .330895+0000 mon.a (mon.0 2021-07-31T01:43:09.045 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[33655]: ) 466 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:43:09.045 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[33655]: audit 2021-07-31 2021-07-31T01:43:09.045 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[33655]: T01:43:08.332060+0000 mon.a (mon.0) 467 2021-07-31T01:43:09.046 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[33655]: : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:43:09.046 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[33655]: audit 2021 2021-07-31T01:43:09.046 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[33655]: -07-31T01:43:08.333244+0000 mon.a ( 2021-07-31T01:43:09.046 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[33655]: mon.0) 468 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:43:09.046 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[33655]: audit 2021-07-31T01:43:09.047 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[33655]: 2021-07-31T01:43:08.338147 2021-07-31T01:43:09.047 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:08 smithi045 conmon[33655]: +0000 mon.a (mon.0) 469 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:43:09.978 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:09 smithi154 conmon[34553]: cephadm 2021-07-31T01:43:08. 2021-07-31T01:43:09.978 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:09 smithi154 conmon[34553]: 312152+0000 mgr.y (mgr.14146) 147 : cephadm [INF] Detected new or changed devices on smithi154 2021-07-31T01:43:09.979 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:09 smithi154 conmon[34553]: cephadm 2021-07-31T01:43:08.320089+0000 mgr.y (mgr.14146) 148 : cephadm [INF] Adjusting osd_memory_target on smithi154 to 8585M 2021-07-31T01:43:09.979 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:09 smithi154 conmon[34553]: cluster 2021-07-31T01:43:08.728848+0000 mon.a (mon.0) 470 : cluster [DBG] osdmap e37: 6 total, 6 up, 6 in 2021-07-31T01:43:09.979 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:09 smithi154 conmon[34553]: audit 2021-07-31T01:43:09.210257+0000 mon.a (mon.0) 471 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-07-31T01:43:09.979 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:09 smithi154 conmon[34553]: audit 2021-07-31T01:43:09.212060+0000 mon.a (mon.0) 472 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-07-31T01:43:09.980 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:09 smithi154 conmon[34553]: audit 2021-07-31T01:43:09.212896+0000 mon.a (mon.0) 473 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:43:10.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:09 smithi045 conmon[33655]: cephadm 2021-07-31T01:43:08.312152+0000 mgr.y (mgr. 2021-07-31T01:43:10.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:09 smithi045 conmon[33655]: 14146) 147 : cephadm [INF] Detected new or changed devices on smithi154 2021-07-31T01:43:10.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:09 smithi045 conmon[33655]: cephadm 2021-07-31T01:43:08.320089+0000 mgr.y (mgr.14146) 148 : cephadm [INF] Adjusting osd_memory_target on smithi154 to 8585M 2021-07-31T01:43:10.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:09 smithi045 conmon[33655]: cluster 2021-07-31T01:43:08.728848+0000 mon.a (mon.0) 470 : cluster [DBG] osdmap e37: 6 total, 6 up, 6 in 2021-07-31T01:43:10.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:09 smithi045 conmon[33655]: audit 2021-07-31T01:43:09.210257+0000 mon.a (mon.0) 471 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-07-31T01:43:10.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:09 smithi045 conmon[33655]: audit 2021-07-31T01:43:09.212060+0000 mon.a (mon.0) 472 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-07-31T01:43:10.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:09 smithi045 conmon[33655]: audit 2021-07-31T01:43:09.212896+0000 mon.a (mon.0) 473 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:43:10.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:09 smithi045 conmon[39718]: cephadm 2021-07-31T01:43:08.312152+0000 mgr.y ( 2021-07-31T01:43:10.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:09 smithi045 conmon[39718]: mgr.14146) 147 : cephadm [INF] Detected new or changed devices on smithi154 2021-07-31T01:43:10.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:09 smithi045 conmon[39718]: cephadm 2021-07-31T01:43:08.320089+0000 mgr.y (mgr.14146) 2021-07-31T01:43:10.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:09 smithi045 conmon[39718]: 148 : cephadm [INF] Adjusting osd_memory_target on smithi154 to 8585M 2021-07-31T01:43:10.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:09 smithi045 conmon[39718]: cluster 2021-07- 2021-07-31T01:43:10.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:09 smithi045 conmon[39718]: 31T01:43:08.728848+0000 mon.a (mon.0) 470 : cluster 2021-07-31T01:43:10.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:09 smithi045 conmon[39718]: [DBG] osdmap e37: 6 total, 6 up, 6 in 2021-07-31T01:43:10.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:09 smithi045 conmon[39718]: audit 2021-07-31T01 2021-07-31T01:43:10.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:09 smithi045 conmon[39718]: :43:09.210257+0000 mon.a (mon.0) 471 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-07-31T01:43:10.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:09 smithi045 conmon[39718]: 2021-07-31T01:43:10.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:09 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T01:43:10.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:09 smithi045 conmon[39718]: 31T01:43:09.212060+0000 mon.a (mon. 2021-07-31T01:43:10.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:09 smithi045 conmon[39718]: 0) 472 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-07-31T01:43:10.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:09 smithi045 conmon[39718]: audit 2021-07-31T01:43:10.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:09 smithi045 conmon[39718]: 2021-07-31T01:43:09.212896 2021-07-31T01:43:10.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:09 smithi045 conmon[39718]: +0000 mon.a (mon.0) 473 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:43:10.965 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:10 smithi154 conmon[34553]: audit 2021-07-31T01:43:09.209321+0000 mgr.y (mgr.14146) 149 : audit 2021-07-31T01:43:10.966 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:10 smithi154 conmon[34553]: [DBG] from='client.24257 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:43:10.966 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:10 smithi154 conmon[34553]: cluster 2021-07-31T01:43:09.327815+0000 mgr.y (mgr.14146) 150 : cluster [DBG] pgmap v132: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-07-31T01:43:10.966 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:10 smithi154 conmon[34553]: cluster 2021-07-31T01:43:09.738572+0000 mon.a (mon.0) 474 : cluster 2021-07-31T01:43:10.967 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:10 smithi154 conmon[34553]: [DBG] osdmap e38: 6 total, 6 up, 6 in 2021-07-31T01:43:11.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:10 smithi045 conmon[33655]: audit 2021-07-31T01:43:09.209321+0000 mgr.y (mgr.14146) 149 : audit [DBG] from='client.24257 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:43:11.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:10 smithi045 conmon[33655]: cluster 2021- 2021-07-31T01:43:11.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:10 smithi045 conmon[33655]: 07-31T01:43:09.327815+0000 mgr.y (mgr.14146) 150 : cluster [DBG] pgmap v132: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-07-31T01:43:11.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:10 smithi045 conmon[33655]: cluster 2021-07-31T01:43:09.738572+0000 mon.a (mon.0) 474 : 2021-07-31T01:43:11.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:10 smithi045 conmon[33655]: cluster [DBG] osdmap e38: 6 total, 6 up, 6 in 2021-07-31T01:43:11.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:10 smithi045 conmon[39718]: audit 2021-07-31T01:43:09.209321+0000 mgr.y 2021-07-31T01:43:11.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:10 smithi045 conmon[39718]: (mgr.14146) 149 : audit [DBG] from='client.24257 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:43:11.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:10 smithi045 conmon[39718]: cluster 2021-07 2021-07-31T01:43:11.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:10 smithi045 conmon[39718]: -31T01:43:09.327815+0000 mgr.y ( 2021-07-31T01:43:11.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:10 smithi045 conmon[39718]: mgr.14146) 150 : cluster [DBG] pgmap v132: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-07-31T01:43:11.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:10 smithi045 conmon[39718]: cluster 2021- 2021-07-31T01:43:11.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:10 smithi045 conmon[39718]: 07-31T01:43:09.738572 2021-07-31T01:43:11.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:10 smithi045 conmon[39718]: +0000 mon.a (mon.0) 474 : 2021-07-31T01:43:11.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:10 smithi045 conmon[39718]: cluster [DBG] osdmap e38: 6 total, 6 up, 6 in 2021-07-31T01:43:12.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:12 smithi154 conmon[34553]: cluster 2021-07-31T01:43:11.328305+0000 mgr.y (mgr.14146) 151 : cluster [DBG] pgmap v134: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-07-31T01:43:12.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:12 smithi154 conmon[34553]: audit 2021-07-31T01:43:11.595132+0000 mon.b (mon.2) 15 : audit [INF] from='client.? 172.21.15.154:0/3323537278' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "94258018-67f7-49d7-83c3-85070ff8c2f3"}]: dispatch 2021-07-31T01:43:12.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:12 smithi154 conmon[34553]: audit 2021-07-31T01:43:11.595497+0000 mon.a (mon.0) 475 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "94258018-67f7-49d7-83c3-85070ff8c2f3"}]: dispatch 2021-07-31T01:43:12.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:12 smithi154 conmon[34553]: audit 2021-07-31T01:43:11.600042+0000 mon.a (mon.0) 476 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "94258018-67f7-49d7-83c3-85070ff8c2f3"}]': finished 2021-07-31T01:43:12.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:12 smithi154 conmon[34553]: cluster 2021-07-31T01:43:11.600196+0000 mon.a (mon.0) 477 : cluster [DBG] osdmap e39: 7 total, 6 up, 7 in 2021-07-31T01:43:12.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:12 smithi154 conmon[34553]: audit 2021-07-31T01:43:11.600540+0000 mon.a (mon.0) 478 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-07-31T01:43:12.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:12 smithi154 conmon[34553]: audit 2021-07-31T01:43:12.077191+0000 mon.b (mon.2) 16 : audit [DBG] from='client.? 172.21.15.154:0/593517675' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-07-31T01:43:12.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:12 smithi045 conmon[39718]: cluster 2021-07-31T01:43:11.328305+0000 mgr.y (mgr.14146 2021-07-31T01:43:12.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:12 smithi045 conmon[39718]: ) 151 : cluster [DBG] pgmap v134: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-07-31T01:43:12.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:12 smithi045 conmon[39718]: audit 2021-07-31T01:43:12.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:12 smithi045 conmon[39718]: 2021-07-31T01:43:11.595132+0000 mon.b (mon. 2021-07-31T01:43:12.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:12 smithi045 conmon[39718]: 2) 15 : audit [INF] from='client.? 172.21.15.154:0/3323537278' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "94258018-67f7-49d7-83c3-85070ff8c2f3"}]: dispatch 2021-07-31T01:43:12.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:12 smithi045 conmon[39718]: 2021-07-31T01:43:12.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:12 smithi045 conmon[39718]: audit 2021 2021-07-31T01:43:12.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:12 smithi045 conmon[39718]: -07-31T01:43:11.595497+0000 2021-07-31T01:43:12.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:12 smithi045 conmon[39718]: mon.a (mon.0) 475 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "94258018-67f7-49d7-83c3-85070ff8c2f3"}]: dispatch 2021-07-31T01:43:12.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:12 smithi045 conmon[39718]: audit 2021-07-31T01:43:11.600042+0000 2021-07-31T01:43:12.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:12 smithi045 conmon[39718]: mon.a (mon.0) 476 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "94258018-67f7-49d7-83c3-85070ff8c2f3"}]': finished 2021-07-31T01:43:12.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:12 smithi045 conmon[39718]: cluster 2021-07-31T 2021-07-31T01:43:12.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:12 smithi045 conmon[39718]: 01:43:11.600196+0000 mon.a (mon 2021-07-31T01:43:12.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:12 smithi045 conmon[39718]: .0) 477 : cluster [DBG] osdmap e39: 7 total, 6 up, 7 in 2021-07-31T01:43:12.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:12 smithi045 conmon[39718]: audit 2021- 2021-07-31T01:43:12.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:12 smithi045 conmon[39718]: 07-31T01:43:11 2021-07-31T01:43:12.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:12 smithi045 conmon[39718]: .600540+0000 mon.a (mon. 2021-07-31T01:43:12.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:12 smithi045 conmon[39718]: 0) 478 : audit [DBG] 2021-07-31T01:43:12.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:12 smithi045 conmon[39718]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-07-31T01:43:12.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:12 smithi045 conmon[39718]: audit 2021- 2021-07-31T01:43:12.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:12 smithi045 conmon[39718]: 07-31T01:43:12. 2021-07-31T01:43:12.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:12 smithi045 conmon[39718]: 077191+0000 mon.b (mon.2 2021-07-31T01:43:12.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:12 smithi045 conmon[39718]: ) 16 : audit [DBG] from='client.? 172.21.15.154:0/593517675' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-07-31T01:43:12.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:12 smithi045 conmon[39718]: 2021-07-31T01:43:12.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:12 smithi045 conmon[33655]: cluster 2021-07-31T01:43:11.328305+0000 mgr.y (mgr. 2021-07-31T01:43:12.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:12 smithi045 conmon[33655]: 14146) 151 : cluster [DBG] pgmap v134: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-07-31T01:43:12.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:12 smithi045 conmon[33655]: audit 2021-07-31T01:43:11 2021-07-31T01:43:12.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:12 smithi045 conmon[33655]: .595132+0000 mon.b (mon.2) 15 : audit [INF] 2021-07-31T01:43:12.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:12 smithi045 conmon[33655]: from='client.? 172.21.15.154:0/3323537278' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "94258018-67f7-49d7-83c3-85070ff8c2f3"}]: dispatch 2021-07-31T01:43:12.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:12 smithi045 conmon[33655]: audit 2021- 2021-07-31T01:43:12.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:12 smithi045 conmon[33655]: 07-31T01:43:11.595497+0000 2021-07-31T01:43:12.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:12 smithi045 conmon[33655]: mon.a (mon.0) 475 : audit 2021-07-31T01:43:12.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:12 smithi045 conmon[33655]: [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "94258018-67f7-49d7-83c3-85070ff8c2f3"}]: dispatch 2021-07-31T01:43:12.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:12 smithi045 conmon[33655]: audit 2021- 2021-07-31T01:43:12.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:12 smithi045 conmon[33655]: 07-31T01:43:11.600042+0000 mon.a 2021-07-31T01:43:12.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:12 smithi045 conmon[33655]: (mon.0) 476 : audit [INF] 2021-07-31T01:43:12.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:12 smithi045 conmon[33655]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "94258018-67f7-49d7-83c3-85070ff8c2f3"}]': finished 2021-07-31T01:43:12.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:12 smithi045 conmon[33655]: cluster 2021-07-31T01:43:12.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:12 smithi045 conmon[33655]: 2021-07-31T 2021-07-31T01:43:12.788 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:12 smithi045 conmon[33655]: 01:43:11.600196+0000 2021-07-31T01:43:12.788 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:12 smithi045 conmon[33655]: mon.a (mon.0) 477 2021-07-31T01:43:12.788 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:12 smithi045 conmon[33655]: : cluster [DBG] osdmap e39: 7 total, 6 up, 7 in 2021-07-31T01:43:12.788 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:12 smithi045 conmon[33655]: audit 2021 2021-07-31T01:43:12.789 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:12 smithi045 conmon[33655]: -07-31T01:43:11 2021-07-31T01:43:12.789 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:12 smithi045 conmon[33655]: .600540+0000 mon.a (mon 2021-07-31T01:43:12.790 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:12 smithi045 conmon[33655]: .0) 478 : audit [DBG] 2021-07-31T01:43:12.790 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:12 smithi045 conmon[33655]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-07-31T01:43:12.790 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:12 smithi045 conmon[33655]: audit 2021 2021-07-31T01:43:12.791 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:12 smithi045 conmon[33655]: -07-31T01:43:12 2021-07-31T01:43:12.791 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:12 smithi045 conmon[33655]: .077191+0000 mon.b (mon. 2021-07-31T01:43:12.792 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:12 smithi045 conmon[33655]: 2) 16 : audit [DBG] from='client.? 172.21.15.154:0/593517675' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-07-31T01:43:12.792 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:12 smithi045 conmon[33655]: 2021-07-31T01:43:14.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:14 smithi154 conmon[34553]: cluster 2021-07-31T01:43:13.328803+0000 mgr.y (mgr 2021-07-31T01:43:14.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:14 smithi154 conmon[34553]: .14146) 152 : cluster [DBG] pgmap v136: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-07-31T01:43:14.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:14 smithi045 conmon[33655]: cluster 2021-07-31T01:43:13.328803+0000 mgr.y (mgr.14146) 152 : cluster 2021-07-31T01:43:14.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:14 smithi045 conmon[33655]: [DBG] pgmap v136: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-07-31T01:43:14.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:14 smithi045 conmon[39718]: cluster 2021-07-31T 2021-07-31T01:43:14.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:14 smithi045 conmon[39718]: 01:43:13.328803+0000 mgr.y (mgr.14146) 152 : cluster [DBG] pgmap v136: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-07-31T01:43:16.714 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:16 smithi154 conmon[34553]: cluster 2021-07-31T01:43:15.329247+0000 mgr.y 2021-07-31T01:43:16.714 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:16 smithi154 conmon[34553]: (mgr.14146) 153 : cluster [DBG] pgmap v137: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-07-31T01:43:16.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:16 smithi045 conmon[33655]: cluster 2021-07-31T01:43:15.329247+0000 mgr.y (mgr.14146) 153 2021-07-31T01:43:16.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:16 smithi045 conmon[33655]: : cluster [DBG] pgmap v137: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-07-31T01:43:16.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:16 smithi045 conmon[39718]: cluster 2021-07-31T01:43:15.329247+0000 mgr.y ( 2021-07-31T01:43:16.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:16 smithi045 conmon[39718]: mgr.14146) 153 : cluster [DBG] pgmap v137: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-07-31T01:43:18.713 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:18 smithi154 conmon[34553]: cluster 2021-07-31T01:43:17.329605+0000 mgr.y (mgr.14146) 154 : cluster [DBG] pgmap v138: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-07-31T01:43:18.713 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:18 smithi154 conmon[34553]: audit 2021-07-31T01:43:17.910828+0000 mon.a (mon.0) 479 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2021-07-31T01:43:18.713 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:18 smithi154 conmon[34553]: audit 2021-07-31T01:43:17.911707+0000 mon.a (mon.0) 480 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:43:18.714 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:18 smithi154 conmon[34553]: cephadm 2021-07-31T01:43:17.912491+0000 mgr.y (mgr.14146) 155 : cephadm [INF] Deploying daemon osd.6 on smithi154 2021-07-31T01:43:18.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:18 smithi045 conmon[33655]: cluster 2021-07-31T01:43:17.329605+0000 mgr.y (mgr.14146) 154 : 2021-07-31T01:43:18.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:18 smithi045 conmon[33655]: cluster [DBG] pgmap v138: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-07-31T01:43:18.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:18 smithi045 conmon[33655]: audit 2021-07-31T01:43:17.910828+0000 mon.a (mon.0) 479 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2021-07-31T01:43:18.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:18 smithi045 conmon[33655]: audit 2021-07-31T01:43:17.911707+0000 2021-07-31T01:43:18.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:18 smithi045 conmon[33655]: mon.a (mon.0) 480 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:43:18.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:18 smithi045 conmon[33655]: cephadm 2021-07-31T01:43:17.912491+0000 mgr.y (mgr.14146) 155 2021-07-31T01:43:18.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:18 smithi045 conmon[33655]: : cephadm [INF] Deploying daemon osd.6 on smithi154 2021-07-31T01:43:18.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:18 smithi045 conmon[39718]: cluster 2021-07-31T01:43:17.329605+0000 mgr.y (mgr 2021-07-31T01:43:18.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:18 smithi045 conmon[39718]: .14146) 154 : cluster [DBG] pgmap v138: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-07-31T01:43:18.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:18 smithi045 conmon[39718]: audit 2021-07-31T01:43:17.910828+0000 mon.a ( 2021-07-31T01:43:18.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:18 smithi045 conmon[39718]: mon.0) 479 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2021-07-31T01:43:18.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:18 smithi045 conmon[39718]: audit 2021-07-31T01 2021-07-31T01:43:18.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:18 smithi045 conmon[39718]: :43:17.911707+0000 mon.a (mon.0 2021-07-31T01:43:18.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:18 smithi045 conmon[39718]: ) 480 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:43:18.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:18 smithi045 conmon[39718]: cephadm 2021- 2021-07-31T01:43:18.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:18 smithi045 conmon[39718]: 07-31T01:43:17.912491+0000 2021-07-31T01:43:18.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:18 smithi045 conmon[39718]: mgr.y (mgr.14146) 155 : cephadm 2021-07-31T01:43:18.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:18 smithi045 conmon[39718]: [INF] Deploying daemon osd.6 on smithi154 2021-07-31T01:43:20.715 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:20 smithi154 conmon[34553]: cluster 2021-07-31T01:43:19.330100+0000 mgr.y (mgr 2021-07-31T01:43:20.716 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:20 smithi154 conmon[34553]: .14146) 156 : cluster [DBG] pgmap v139: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-07-31T01:43:20.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:20 smithi045 conmon[33655]: cluster 2021-07-31T01:43:19.330100 2021-07-31T01:43:20.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:20 smithi045 conmon[33655]: +0000 mgr.y (mgr.14146) 156 : cluster [DBG] pgmap v139: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-07-31T01:43:20.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:20 smithi045 conmon[39718]: cluster 2021-07-31T01:43:19.330100+0000 mgr.y (mgr 2021-07-31T01:43:20.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:20 smithi045 conmon[39718]: .14146) 156 : cluster [DBG] pgmap v139: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-07-31T01:43:21.729 INFO:teuthology.orchestra.run.smithi154.stdout:Created osd(s) 6 on host 'smithi154' 2021-07-31T01:43:22.470 DEBUG:teuthology.orchestra.run.smithi154:osd.6> sudo journalctl -f -n 0 -u ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c@osd.6.service 2021-07-31T01:43:22.473 INFO:tasks.cephadm:Deploying osd.7 on smithi154 with /dev/vg_nvme/lv_1... 2021-07-31T01:43:22.473 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f52d2c14-f19f-11eb-8c24-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2021-07-31T01:43:22.511 INFO:journalctl@ceph.osd.6.smithi154.stdout:-- Logs begin at Sat 2021-07-31 01:17:02 UTC. -- 2021-07-31T01:43:22.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:22 smithi154 conmon[34553]: cluster 2021-07-31T01:43:21.330554+0000 mgr.y (mgr. 2021-07-31T01:43:22.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:22 smithi154 conmon[34553]: 14146) 157 : cluster [DBG] pgmap v140: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-07-31T01:43:22.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:22 smithi154 conmon[34553]: audit 2021-07-31T01:43:21.717247+0000 mon.a (mon.0) 481 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:43:22.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:22 smithi154 conmon[34553]: audit 2021-07-31T01:43:21.719125+0000 mon.a (mon.0) 482 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:43:22.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:22 smithi154 conmon[34553]: audit 2021-07-31T01:43:21.721519+0000 mon.a (mon.0) 483 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:43:22.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:22 smithi154 conmon[34553]: audit 2021-07-31T01:43:21.723599+0000 mon.a (mon.0) 484 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:43:22.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:22 smithi154 conmon[34553]: audit 2021-07-31T01:43:21.726361+0000 mon.a (mon.0) 485 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:43:22.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:22 smithi045 conmon[33655]: cluster 2021-07-31T01:43:21.330554+0000 mgr.y (mgr.14146) 157 : cluster [DBG] pgmap v140: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-07-31T01:43:22.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:22 smithi045 conmon[33655]: audit 2021-07-31T01:43:21.717247+0000 mon.a (mon.0) 481 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:43:22.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:22 smithi045 conmon[33655]: audit 2021-07-31T01:43:21.719125+0000 mon.a (mon.0) 482 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:43:22.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:22 smithi045 conmon[33655]: audit 2021-07-31T01:43:21.721519+0000 mon.a (mon.0) 483 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:43:22.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:22 smithi045 conmon[33655]: audit 2021-07-31T01:43:21.723599+0000 mon.a (mon.0) 484 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:43:22.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:22 smithi045 conmon[33655]: audit 2021-07-31T01:43:21.726361+0000 mon.a (mon.0) 485 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:43:22.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:22 smithi045 conmon[39718]: cluster 2021-07-31T01:43:21.330554+0000 mgr.y (mgr.14146) 157 : cluster [DBG] pgmap v140: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-07-31T01:43:22.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:22 smithi045 conmon[39718]: audit 2021-07-31T01:43:21.717247+0000 mon.a (mon.0) 481 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:43:22.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:22 smithi045 conmon[39718]: audit 2021-07-31T01:43:21.719125+0000 mon.a (mon.0) 482 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:43:22.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:22 smithi045 conmon[39718]: audit 2021-07-31T01:43:21.721519+0000 mon.a (mon.0) 483 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:43:22.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:22 smithi045 conmon[39718]: audit 2021-07-31T01:43:21.723599+0000 mon.a (mon.0) 484 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:43:22.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:22 smithi045 conmon[39718]: audit 2021-07-31T01:43:21.726361+0000 mon.a (mon.0) 485 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:43:24.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:24 smithi154 conmon[34553]: cluster 2021-07-31T01:43:23.331029+0000 mgr.y (mgr.14146) 158 : cluster [DBG] pgmap v141: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-07-31T01:43:24.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:24 smithi154 conmon[34553]: audit 2021-07-31T01:43:24.305980+0000 mon.a (mon.0) 486 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:43:24.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:24 smithi045 conmon[33655]: cluster 2021-07-31T01:43:23.331029+0000 mgr.y (mgr.14146) 158 : 2021-07-31T01:43:24.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:24 smithi045 conmon[33655]: cluster [DBG] pgmap v141: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-07-31T01:43:24.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:24 smithi045 conmon[33655]: audit 2021-07-31T01:43:24.305980+0000 mon.a (mon.0) 486 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:43:24.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:24 smithi045 conmon[39718]: cluster 2021-07-31T01:43:23.331029+0000 mgr.y (mgr.14146 2021-07-31T01:43:24.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:24 smithi045 conmon[39718]: ) 158 : cluster [DBG] pgmap v141: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-07-31T01:43:24.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:24 smithi045 conmon[39718]: audit 2021-07-31T01:43:24.305980+0000 2021-07-31T01:43:24.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:24 smithi045 conmon[39718]: mon.a (mon.0) 486 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:43:24.990 INFO:teuthology.orchestra.run.smithi154.stderr:--> Zapping: /dev/vg_nvme/lv_1 2021-07-31T01:43:24.990 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2021-07-31T01:43:24.991 INFO:teuthology.orchestra.run.smithi154.stderr: stderr: 10+0 records in 2021-07-31T01:43:24.991 INFO:teuthology.orchestra.run.smithi154.stderr:10+0 records out 2021-07-31T01:43:24.991 INFO:teuthology.orchestra.run.smithi154.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0118164 s, 887 MB/s 2021-07-31T01:43:24.991 INFO:teuthology.orchestra.run.smithi154.stderr:--> Zapping successful for: 2021-07-31T01:43:25.086 INFO:journalctl@ceph.osd.6.smithi154.stdout:Jul 31 01:43:24 smithi154 conmon[45576]: debug 2021-07-31T01:43:24.877+0000 7f47ec52a080 -1 osd.6 0 log_to_monitors {default=true} 2021-07-31T01:43:25.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:25 smithi154 conmon[34553]: audit 2021-07-31T01:43:24.881035 2021-07-31T01:43:25.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:25 smithi154 conmon[34553]: +0000 mon.b (mon.2) 17 : audit [INF] from='osd.6 [v2:172.21.15.154:6816/3963397470,v1:172.21.15.154:6817/3963397470]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2021-07-31T01:43:25.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:25 smithi154 conmon[34553]: audit 2021-07-31T01:43:24.881336+0000 mon.a (mon.0) 487 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2021-07-31T01:43:25.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:25 smithi045 conmon[33655]: audit 2021-07-31T01:43:24.881035+0000 mon.b (mon.2) 2021-07-31T01:43:25.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:25 smithi045 conmon[33655]: 17 : audit [INF] from='osd.6 [v2:172.21.15.154:6816/3963397470,v1:172.21.15.154:6817/3963397470]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2021-07-31T01:43:25.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:25 smithi045 conmon[33655]: audit 2021-07-31T01:43:24.881336+0000 mon.a (mon.0) 487 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2021-07-31T01:43:25.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:25 smithi045 conmon[39718]: audit 2021-07-31T01:43:24.881035+0000 mon.b (mon 2021-07-31T01:43:25.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:25 smithi045 conmon[39718]: .2) 17 : audit [INF] from='osd.6 [v2:172.21.15.154:6816/3963397470,v1:172.21.15.154:6817/3963397470]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2021-07-31T01:43:25.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:25 smithi045 conmon[39718]: audit 2021-07-31T01:43:24.881336+0000 mon.a 2021-07-31T01:43:25.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:25 smithi045 conmon[39718]: (mon.0) 487 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2021-07-31T01:43:25.826 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f52d2c14-f19f-11eb-8c24-001a4aab830c -- ceph orch daemon add osd smithi154:vg_nvme/lv_1 2021-07-31T01:43:26.629 INFO:journalctl@ceph.osd.6.smithi154.stdout:Jul 31 01:43:26 smithi154 conmon[45576]: debug 2021-07-31T01:43:26.466+0000 7f47e2a8d700 -1 osd.6 0 waiting for initial osdmap 2021-07-31T01:43:26.630 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:26 smithi154 conmon[34553]: cluster 2021-07-31T01:43:25.331511+0000 mgr.y (mgr 2021-07-31T01:43:26.630 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:26 smithi154 conmon[34553]: .14146) 159 : cluster [DBG] pgmap v142: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-07-31T01:43:26.631 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:26 smithi154 conmon[34553]: audit 2021-07-31T01:43:25.454297+0000 mon.a (mon.0) 488 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2021-07-31T01:43:26.631 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:26 smithi154 conmon[34553]: cluster 2021-07-31T01:43:25.454396+0000 mon.a (mon.0) 489 : cluster [DBG] osdmap e40: 7 total, 6 up, 7 in 2021-07-31T01:43:26.631 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:26 smithi154 conmon[34553]: audit 2021-07-31T01:43:25.454604+0000 mon.a (mon.0) 490 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-07-31T01:43:26.631 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:26 smithi154 conmon[34553]: audit 2021-07-31T01:43:25.456306+0000 mon.b (mon.2) 18 : audit [INF] from='osd.6 [v2:172.21.15.154:6816/3963397470,v1:172.21.15.154:6817/3963397470]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2021-07-31T01:43:26.632 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:26 smithi154 conmon[34553]: audit 2021-07-31T01:43:25.456584+0000 mon.a (mon.0) 491 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2021-07-31T01:43:26.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:26 smithi045 conmon[33655]: cluster 2021-07-31T01:43:25.331511+0000 mgr.y (mgr. 2021-07-31T01:43:26.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:26 smithi045 conmon[33655]: 14146) 159 : cluster [DBG] pgmap v142: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-07-31T01:43:26.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:26 smithi045 conmon[33655]: audit 2021-07-31T01:43:25.454297+0000 mon.a (mon.0) 488 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2021-07-31T01:43:26.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:26 smithi045 conmon[33655]: cluster 2021-07-31T01:43:25.454396+0000 mon.a (mon 2021-07-31T01:43:26.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:26 smithi045 conmon[33655]: .0) 489 : cluster [DBG] osdmap e40: 7 total, 6 up, 7 in 2021-07-31T01:43:26.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:26 smithi045 conmon[33655]: audit 2021-07-31T 2021-07-31T01:43:26.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:26 smithi045 conmon[33655]: 01:43:25.454604+0000 mon.a (mon.0) 490 : audit 2021-07-31T01:43:26.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:26 smithi045 conmon[33655]: [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-07-31T01:43:26.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:26 smithi045 conmon[33655]: audit 2021- 2021-07-31T01:43:26.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:26 smithi045 conmon[33655]: 07-31T01:43:25.456306+0000 mon.b (mon.2 2021-07-31T01:43:26.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:26 smithi045 conmon[33655]: ) 18 : audit [INF] from='osd.6 [v2:172.21.15.154:6816/3963397470,v1:172.21.15.154:6817/3963397470]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2021-07-31T01:43:26.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:26 smithi045 conmon[33655]: audit 2021-07-31T01:43:26.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:26 smithi045 conmon[33655]: 2021-07-31T01:43 2021-07-31T01:43:26.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:26 smithi045 conmon[33655]: :25.456584+0000 mon.a (mon.0) 491 : audit 2021-07-31T01:43:26.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:26 smithi045 conmon[33655]: [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2021-07-31T01:43:26.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:26 smithi045 conmon[39718]: cluster 2021-07-31T01:43:25.331511+0000 mgr.y (mgr.14146 2021-07-31T01:43:26.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:26 smithi045 conmon[39718]: ) 159 : cluster [DBG] pgmap v142: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-07-31T01:43:26.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:26 smithi045 conmon[39718]: audit 2021-07-31T01:43: 2021-07-31T01:43:26.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:26 smithi045 conmon[39718]: 25.454297+0000 mon.a (mon.0) 488 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2021-07-31T01:43:26.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:26 smithi045 conmon[39718]: cluster 2021- 2021-07-31T01:43:26.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:26 smithi045 conmon[39718]: 07-31T01:43:25. 2021-07-31T01:43:26.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:26 smithi045 conmon[39718]: 454396+0000 mon.a (mon.0 2021-07-31T01:43:26.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:26 smithi045 conmon[39718]: ) 489 : cluster [DBG] osdmap e40: 7 total, 6 up, 7 in 2021-07-31T01:43:26.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:26 smithi045 conmon[39718]: 2021-07-31T01:43:26.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:26 smithi045 conmon[39718]: audit 2021-07-31T01:43:26.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:26 smithi045 conmon[39718]: 2021-07-31T 2021-07-31T01:43:26.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:26 smithi045 conmon[39718]: 01:43:25.454604 2021-07-31T01:43:26.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:26 smithi045 conmon[39718]: +0000 mon.a (mon.0) 490 : audit 2021-07-31T01:43:26.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:26 smithi045 conmon[39718]: [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-07-31T01:43:26.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:26 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T01:43:26.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:26 smithi045 conmon[39718]: T01:43:25. 2021-07-31T01:43:26.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:26 smithi045 conmon[39718]: 456306+0000 mon.b (mon.2) 2021-07-31T01:43:26.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:26 smithi045 conmon[39718]: 18 : audit [INF] from='osd.6 [v2:172.21.15.154:6816/3963397470,v1:172.21.15.154:6817/3963397470]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2021-07-31T01:43:26.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:26 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T01:43:26.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:26 smithi045 conmon[39718]: 31T01:43:25.456584+0000 mon.a 2021-07-31T01:43:26.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:26 smithi045 conmon[39718]: (mon.0) 491 : audit 2021-07-31T01:43:26.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:26 smithi045 conmon[39718]: [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2021-07-31T01:43:27.596 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:27 smithi154 conmon[34553]: audit 2021-07-31T01:43:26.455639+0000 mon.a (mon.0) 492 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2021-07-31T01:43:27.596 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:27 smithi154 conmon[34553]: cluster 2021-07-31T01:43:26.455773+0000 mon.a (mon.0) 493 : cluster [DBG] osdmap e41: 7 total, 6 up, 7 in 2021-07-31T01:43:27.597 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:27 smithi154 conmon[34553]: audit 2021-07-31T01:43:26.456222+0000 mon.a (mon.0) 494 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-07-31T01:43:27.597 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:27 smithi154 conmon[34553]: audit 2021-07-31T01:43:26.468036+0000 mon.a (mon.0) 495 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-07-31T01:43:27.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:27 smithi045 conmon[33655]: audit 2021-07-31T01:43:27.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:27 smithi045 conmon[33655]: 2021-07-31T01:43:26. 2021-07-31T01:43:27.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:27 smithi045 conmon[33655]: 455639+0000 mon.a (mon.0) 492 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2021-07-31T01:43:27.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:27 smithi045 conmon[33655]: cluster 2021-07-31T01:43:26.455773+0000 mon.a (mon.0) 493 : cluster [DBG] 2021-07-31T01:43:27.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:27 smithi045 conmon[33655]: osdmap e41: 7 total, 6 up, 7 in 2021-07-31T01:43:27.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:27 smithi045 conmon[33655]: audit 2021-07 2021-07-31T01:43:27.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:27 smithi045 conmon[33655]: -31T01:43:26.456222+0000 mon.a (mon.0) 494 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-07-31T01:43:27.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:27 smithi045 conmon[33655]: audit 2021-07-31T01 2021-07-31T01:43:27.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:27 smithi045 conmon[33655]: :43:26.468036+0000 mon.a (mon.0) 495 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-07-31T01:43:27.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:27 smithi045 conmon[33655]: 2021-07-31T01:43:27.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:27 smithi045 conmon[39718]: audit 2021-07-31T01:43:26.455639+0000 mon.a ( 2021-07-31T01:43:27.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:27 smithi045 conmon[39718]: mon.0) 492 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2021-07-31T01:43:27.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:27 smithi045 conmon[39718]: cluster 2021-07-31T01:43:27.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:27 smithi045 conmon[39718]: 2021-07-31T01:43:26.455773+0000 mon.a (mon.0) 493 : cluster [DBG] 2021-07-31T01:43:27.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:27 smithi045 conmon[39718]: osdmap e41: 7 total, 6 up, 7 in 2021-07-31T01:43:27.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:27 smithi045 conmon[39718]: audit 2021-07-31T01:43:27.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:27 smithi045 conmon[39718]: 2021-07-31T01:43: 2021-07-31T01:43:27.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:27 smithi045 conmon[39718]: 26.456222+0000 mon.a (mon.0) 494 2021-07-31T01:43:27.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:27 smithi045 conmon[39718]: : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-07-31T01:43:27.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:27 smithi045 conmon[39718]: audit 2021-07-31T01:43:27.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:27 smithi045 conmon[39718]: 2021-07-31T01:43:26.468036+0000 2021-07-31T01:43:27.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:27 smithi045 conmon[39718]: mon.a (mon.0) 495 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-07-31T01:43:27.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:27 smithi045 conmon[39718]: 2021-07-31T01:43:28.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:28 smithi154 conmon[34553]: cluster 2021-07-31T01:43:25.839557 2021-07-31T01:43:28.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:28 smithi154 conmon[34553]: +0000 osd.6 (osd.6) 1 : cluster [DBG] purged_snaps scrub starts 2021-07-31T01:43:28.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:28 smithi154 conmon[34553]: cluster 2021-07-31T01:43:25.839641+0000 osd.6 (osd.6) 2 : cluster [DBG] purged_snaps scrub ok 2021-07-31T01:43:28.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:28 smithi154 conmon[34553]: cluster 2021-07-31T01:43:27.331883+0000 mgr.y (mgr.14146) 160 : cluster [DBG] pgmap v145: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-07-31T01:43:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:28 smithi154 conmon[34553]: cluster 2021-07-31T01:43:27.466215+0000 mon.a (mon.0) 496 : cluster [INF] osd.6 [v2:172.21.15.154:6816/3963397470,v1:172.21.15.154:6817/3963397470] boot 2021-07-31T01:43:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:28 smithi154 conmon[34553]: cluster 2021-07-31T01:43:27.466327+0000 mon.a (mon.0) 497 : cluster [DBG] osdmap e42: 7 total, 7 up, 7 in 2021-07-31T01:43:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:28 smithi154 conmon[34553]: audit 2021-07-31T01:43:27.467038+0000 mon.a (mon.0) 498 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-07-31T01:43:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:28 smithi154 conmon[34553]: audit 2021-07-31T01:43:28.386320+0000 mon.a (mon.0) 499 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-07-31T01:43:28.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:28 smithi154 conmon[34553]: audit 2021-07-31T01:43:28.388041+0000 mon.a (mon.0) 500 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-07-31T01:43:28.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:28 smithi154 conmon[34553]: audit 2021-07-31T01:43:28.388842+0000 mon.a (mon.0) 501 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:43:28.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:28 smithi154 conmon[34553]: cluster 2021-07-31T01:43:28.467091+0000 mon.a (mon.0) 502 : cluster [DBG] osdmap e43: 7 total, 7 up, 7 in 2021-07-31T01:43:28.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:28 smithi045 conmon[39718]: cluster 2021-07-31T01:43: 2021-07-31T01:43:28.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:28 smithi045 conmon[39718]: 25.839557+0000 osd.6 (osd.6) 1 : cluster [DBG] purged_snaps scrub starts 2021-07-31T01:43:28.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:28 smithi045 conmon[39718]: cluster 2021-07-31T01:43:25.839641+0000 osd.6 (osd.6) 2 : cluster [DBG] purged_snaps scrub ok 2021-07-31T01:43:28.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:28 smithi045 conmon[39718]: cluster 2021-07-31T01:43:27.331883+0000 mgr.y (mgr.14146) 160 : cluster [DBG] 2021-07-31T01:43:28.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:28 smithi045 conmon[39718]: pgmap v145: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-07-31T01:43:28.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:28 smithi045 conmon[39718]: cluster 2021-07-31T01:43:27.466215+0000 2021-07-31T01:43:28.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:28 smithi045 conmon[39718]: mon.a (mon.0) 496 : cluster [INF] osd.6 [v2:172.21.15.154:6816/3963397470,v1:172.21.15.154:6817/3963397470] boot 2021-07-31T01:43:28.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:28 smithi045 conmon[39718]: cluster 2021-07- 2021-07-31T01:43:28.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:28 smithi045 conmon[39718]: 31T01:43:27.466327+0000 mon.a ( 2021-07-31T01:43:28.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:28 smithi045 conmon[39718]: mon.0) 497 : cluster [DBG] osdmap e42: 7 total, 7 up, 7 in 2021-07-31T01:43:28.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:28 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T01:43:28.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:28 smithi045 conmon[39718]: T01:43:27.467038+0000 mon.a (mon. 2021-07-31T01:43:28.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:28 smithi045 conmon[39718]: 0) 498 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-07-31T01:43:28.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:28 smithi045 conmon[39718]: audit 2021-07 2021-07-31T01:43:28.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:28 smithi045 conmon[39718]: -31T01:43:28.386320+0000 mon.a (mon. 2021-07-31T01:43:28.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:28 smithi045 conmon[39718]: 0) 499 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-07-31T01:43:28.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:28 smithi045 conmon[39718]: audit 2021-07 2021-07-31T01:43:28.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:28 smithi045 conmon[39718]: -31T01:43:28.388041+0000 mon.a (mon.0 2021-07-31T01:43:28.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:28 smithi045 conmon[39718]: ) 500 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-07-31T01:43:28.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:28 smithi045 conmon[39718]: 2021-07-31T01:43:28.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:28 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T01:43:28.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:28 smithi045 conmon[39718]: 31T01:43:28.388842+0000 mon.a (mon.0) 2021-07-31T01:43:28.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:28 smithi045 conmon[39718]: 501 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:43:28.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:28 smithi045 conmon[39718]: cluster 2021-07-31T01:43:28.467091+0000 mon.a ( 2021-07-31T01:43:28.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:28 smithi045 conmon[39718]: mon.0) 502 : cluster [DBG] osdmap e43: 7 total, 7 up, 7 in 2021-07-31T01:43:28.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:28 smithi045 conmon[33655]: cluster 2021-07-31T01:43:25. 2021-07-31T01:43:28.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:28 smithi045 conmon[33655]: 839557+0000 osd.6 (osd.6) 1 : cluster [DBG] purged_snaps scrub starts 2021-07-31T01:43:28.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:28 smithi045 conmon[33655]: cluster 2021-07-31T01:43:25.839641+0000 osd.6 (osd.6 2021-07-31T01:43:28.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:28 smithi045 conmon[33655]: ) 2 : cluster [DBG] purged_snaps scrub ok 2021-07-31T01:43:28.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:28 smithi045 conmon[33655]: cluster 2021-07-31T01:43:28.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:28 smithi045 conmon[33655]: 2021-07-31T01:43:27.331883+0000 mgr.y (mgr. 2021-07-31T01:43:28.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:28 smithi045 conmon[33655]: 14146) 160 : cluster [DBG] pgmap v145: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-07-31T01:43:28.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:28 smithi045 conmon[33655]: cluster 2021 2021-07-31T01:43:28.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:28 smithi045 conmon[33655]: -07-31T01:43:27.466215 2021-07-31T01:43:28.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:28 smithi045 conmon[33655]: +0000 mon.a (mon.0) 496 : cluster 2021-07-31T01:43:28.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:28 smithi045 conmon[33655]: [INF] osd.6 [v2:172.21.15.154:6816/3963397470,v1:172.21.15.154:6817/3963397470] boot 2021-07-31T01:43:28.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:28 smithi045 conmon[33655]: cluster 2021-07-31 2021-07-31T01:43:28.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:28 smithi045 conmon[33655]: T01:43:27.466327+0000 mon.a ( 2021-07-31T01:43:28.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:28 smithi045 conmon[33655]: mon.0) 497 : cluster [DBG] osdmap e42: 7 total, 7 up, 7 in 2021-07-31T01:43:28.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:28 smithi045 conmon[33655]: 2021-07-31T01:43:28.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:28 smithi045 conmon[33655]: audit 2021-07-31T01:43: 2021-07-31T01:43:28.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:28 smithi045 conmon[33655]: 27.467038+0000 mon.a (mon.0) 2021-07-31T01:43:28.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:28 smithi045 conmon[33655]: 498 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-07-31T01:43:28.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:28 smithi045 conmon[33655]: audit 2021 2021-07-31T01:43:28.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:28 smithi045 conmon[33655]: -07-31T01:43 2021-07-31T01:43:28.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:28 smithi045 conmon[33655]: :28.386320+0000 mon.a ( 2021-07-31T01:43:28.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:28 smithi045 conmon[33655]: mon.0) 499 : audit [DBG] 2021-07-31T01:43:28.788 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:28 smithi045 conmon[33655]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-07-31T01:43:28.788 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:28 smithi045 conmon[33655]: audit 2021- 2021-07-31T01:43:28.788 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:28 smithi045 conmon[33655]: 07-31T01:43: 2021-07-31T01:43:28.789 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:28 smithi045 conmon[33655]: 28.388041+0000 mon.a (mon. 2021-07-31T01:43:28.789 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:28 smithi045 conmon[33655]: 0) 500 : audit [INF] 2021-07-31T01:43:28.789 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:28 smithi045 conmon[33655]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-07-31T01:43:28.789 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:28 smithi045 conmon[33655]: audit 2021-07-31T01: 2021-07-31T01:43:28.790 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:28 smithi045 conmon[33655]: 43:28.388842+0000 mon.a (mon.0) 501 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:43:28.790 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:28 smithi045 conmon[33655]: cluster 2021-07 2021-07-31T01:43:28.790 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:28 smithi045 conmon[33655]: -31T01:43:28.467091+0000 mon.a 2021-07-31T01:43:28.790 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:28 smithi045 conmon[33655]: (mon.0) 502 : cluster [DBG] osdmap e43: 7 total, 7 up, 7 in 2021-07-31T01:43:29.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:29 smithi154 conmon[34553]: audit 2021-07-31T01:43:28.385352+0000 mgr.y (mgr.14146) 161 : audit [DBG] from='client.24284 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:43:29.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:29 smithi154 conmon[34553]: audit 2021-07-31T01:43:29.232264+0000 mon.a (mon.0) 503 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:43:29.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:29 smithi154 conmon[34553]: audit 2021-07-31T01:43:29.233355+0000 mon.a (mon.0) 504 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-07-31T01:43:29.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:29 smithi154 conmon[34553]: audit 2021-07-31T01:43:29.234219+0000 mon.a (mon.0) 505 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-07-31T01:43:29.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:29 smithi154 conmon[34553]: audit 2021-07-31T01:43:29.234991+0000 mon.a (mon.0) 506 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-07-31T01:43:29.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:29 smithi154 conmon[34553]: audit 2021-07-31T01:43:29.239769+0000 mon.a (mon.0) 507 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:43:29.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:29 smithi154 conmon[34553]: audit 2021-07-31T01:43 2021-07-31T01:43:29.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:29 smithi154 conmon[34553]: :29.243663+0000 mon.a (mon.0) 508 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:43:29.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:29 smithi154 conmon[34553]: audit 2021-07-31T01:43:29.245693+0000 mon.a (mon.0) 509 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:43:29.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:29 smithi154 conmon[34553]: audit 2021-07-31T01:43:29.246856+0000 mon.a (mon.0) 510 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:43:29.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:29 smithi154 conmon[34553]: audit 2021-07-31T01:43:29.247605+0000 mon.a (mon.0) 511 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:43:29.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:29 smithi154 conmon[34553]: audit 2021-07-31T01:43:29.251973+0000 mon.a (mon.0) 512 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:43:29.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:29 smithi045 conmon[33655]: audit 2021-07-31T01:43:28.385352+0000 mgr.y ( 2021-07-31T01:43:29.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:29 smithi045 conmon[33655]: mgr.14146) 161 : audit [DBG] from='client.24284 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:43:29.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:29 smithi045 conmon[33655]: audit 2021-07-31T01:43:29.232264+0000 2021-07-31T01:43:29.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:29 smithi045 conmon[33655]: mon.a (mon.0) 503 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:43:29.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:29 smithi045 conmon[33655]: audit 2021-07- 2021-07-31T01:43:29.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:29 smithi045 conmon[33655]: 31T01:43:29.233355+0000 mon.a (mon 2021-07-31T01:43:29.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:29 smithi045 conmon[33655]: .0) 504 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-07-31T01:43:29.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:29 smithi045 conmon[33655]: 2021-07-31T01:43:29.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:29 smithi045 conmon[33655]: audit 2021-07 2021-07-31T01:43:29.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:29 smithi045 conmon[33655]: -31T01:43:29. 2021-07-31T01:43:29.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:29 smithi045 conmon[33655]: 234219+0000 mon.a (mon.0 2021-07-31T01:43:29.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:29 smithi045 conmon[33655]: ) 505 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-07-31T01:43:29.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:29 smithi045 conmon[33655]: audit 2021- 2021-07-31T01:43:29.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:29 smithi045 conmon[33655]: 07-31T01:43:29.234991+0000 2021-07-31T01:43:29.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:29 smithi045 conmon[33655]: mon.a (mon.0) 506 2021-07-31T01:43:29.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:29 smithi045 conmon[33655]: : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-07-31T01:43:29.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:29 smithi045 conmon[33655]: audit 2021 2021-07-31T01:43:29.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:29 smithi045 conmon[33655]: -07-31T01:43:29. 2021-07-31T01:43:29.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:29 smithi045 conmon[33655]: 239769+0000 mon.a (mon.0 2021-07-31T01:43:29.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:29 smithi045 conmon[33655]: ) 507 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:43:29.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:29 smithi045 conmon[33655]: 2021-07-31T01:43:29.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:29 smithi045 conmon[33655]: audit 2021-07-31T01:43:29.243663+0000 mon.a (mon.0) 508 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:43:29.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:29 smithi045 conmon[33655]: audit 2021-07-31T01:43:29.245693+0000 mon.a (mon.0) 509 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:43:29.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:29 smithi045 conmon[33655]: audit 2021-07-31T01:43:29.246856+0000 mon.a (mon.0) 510 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:43:29.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:29 smithi045 conmon[33655]: audit 2021-07-31T01:43:29.247605+0000 mon.a (mon.0) 511 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:43:29.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:29 smithi045 conmon[33655]: audit 2021-07-31T01:43:29.251973+0000 2021-07-31T01:43:29.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:29 smithi045 conmon[33655]: mon.a (mon.0) 512 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:43:29.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:29 smithi045 conmon[39718]: audit 2021-07-31T01:43:28.385352+0000 2021-07-31T01:43:29.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:29 smithi045 conmon[39718]: mgr.y (mgr.14146) 161 : audit [DBG] from='client.24284 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:43:29.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:29 smithi045 conmon[39718]: audit 2021-07-31T01:43:29 2021-07-31T01:43:29.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:29 smithi045 conmon[39718]: .232264+0000 mon.a (mon.0) 503 : 2021-07-31T01:43:29.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:29 smithi045 conmon[39718]: audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:43:29.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:29 smithi045 conmon[39718]: audit 2021- 2021-07-31T01:43:29.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:29 smithi045 conmon[39718]: 07-31T01:43:29.233355+0000 2021-07-31T01:43:29.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:29 smithi045 conmon[39718]: mon.a (mon.0) 504 : audit 2021-07-31T01:43:29.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:29 smithi045 conmon[39718]: [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-07-31T01:43:29.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:29 smithi045 conmon[39718]: audit 2021-07-31T01 2021-07-31T01:43:29.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:29 smithi045 conmon[39718]: :43:29.234219+0000 mon.a (mon 2021-07-31T01:43:29.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:29 smithi045 conmon[39718]: .0) 505 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-07-31T01:43:29.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:29 smithi045 conmon[39718]: audit 2021 2021-07-31T01:43:29.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:29 smithi045 conmon[39718]: -07-31T01:43:29. 2021-07-31T01:43:29.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:29 smithi045 conmon[39718]: 234991+0000 mon.a (mon.0) 506 : 2021-07-31T01:43:29.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:29 smithi045 conmon[39718]: audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-07-31T01:43:29.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:29 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T01:43:29.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:29 smithi045 conmon[39718]: 31T01:43:29.239769+0000 mon.a 2021-07-31T01:43:29.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:29 smithi045 conmon[39718]: (mon.0) 507 : audit [INF] 2021-07-31T01:43:29.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:29 smithi045 conmon[39718]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:43:29.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:29 smithi045 conmon[39718]: audit 2021-07-31T01: 2021-07-31T01:43:29.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:29 smithi045 conmon[39718]: 43:29.243663+0000 mon.a (mon.0) 508 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:43:29.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:29 smithi045 conmon[39718]: audit 2021-07-31T01:43:29.245693+0000 mon.a ( 2021-07-31T01:43:29.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:29 smithi045 conmon[39718]: mon.0) 509 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:43:29.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:29 smithi045 conmon[39718]: audit 2021-07-31T01: 2021-07-31T01:43:29.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:29 smithi045 conmon[39718]: 43:29.246856+0000 mon.a (mon. 2021-07-31T01:43:29.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:29 smithi045 conmon[39718]: 0) 510 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:43:29.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:29 smithi045 conmon[39718]: audit 2021-07-31T01:43:29.247605+0000 2021-07-31T01:43:29.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:29 smithi045 conmon[39718]: mon.a (mon.0) 511 : audit [INF] 2021-07-31T01:43:29.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:29 smithi045 conmon[39718]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:43:29.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:29 smithi045 conmon[39718]: audit 2021-07-31T01 2021-07-31T01:43:29.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:29 smithi045 conmon[39718]: :43:29.251973+0000 mon.a (mon. 2021-07-31T01:43:29.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:29 smithi045 conmon[39718]: 0) 512 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:43:30.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:30 smithi154 conmon[34553]: cephadm 2021-07-31T01:43:29.226987+0000 mgr.y (mgr.14146) 162 : cephadm [INF] Detected new or changed devices on smithi154 2021-07-31T01:43:30.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:30 smithi154 conmon[34553]: cephadm 2021-07-31T01:43:29 2021-07-31T01:43:30.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:30 smithi154 conmon[34553]: .235621+0000 mgr.y (mgr.14146) 163 : cephadm [INF] Adjusting osd_memory_target on smithi154 to 5723M 2021-07-31T01:43:30.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:30 smithi154 conmon[34553]: cluster 2021-07-31T01:43:29.332273+0000 mgr.y (mgr.14146) 164 : cluster [DBG] pgmap v148: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2021-07-31T01:43:30.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:30 smithi154 conmon[34553]: cluster 2021-07-31T01:43:29.478080+0000 mon.a (mon.0) 513 : cluster [DBG] osdmap e44: 7 total, 7 up, 7 in 2021-07-31T01:43:30.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:30 smithi045 conmon[33655]: cephadm 2021-07-31T01:43:29.226987+0000 mgr.y (mgr. 2021-07-31T01:43:30.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:30 smithi045 conmon[33655]: 14146) 162 : cephadm [INF] Detected new or changed devices on smithi154 2021-07-31T01:43:30.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:30 smithi045 conmon[33655]: cephadm 2021-07-31T01:43:29.235621+0000 mgr.y (mgr.14146) 163 : cephadm [INF] Adjusting osd_memory_target on smithi154 to 5723M 2021-07-31T01:43:30.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:30 smithi045 conmon[33655]: cluster 2021-07-31T 2021-07-31T01:43:30.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:30 smithi045 conmon[33655]: 01:43:29.332273+0000 mgr.y (mgr.14146 2021-07-31T01:43:30.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:30 smithi045 conmon[33655]: ) 164 : cluster [DBG] pgmap v148: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2021-07-31T01:43:30.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:30 smithi045 conmon[33655]: cluster 2021-07-31T01:43:30.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:30 smithi045 conmon[33655]: 2021-07 2021-07-31T01:43:30.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:30 smithi045 conmon[33655]: -31T01:43:29. 2021-07-31T01:43:30.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:30 smithi045 conmon[33655]: 478080+0000 mon.a (mon.0 2021-07-31T01:43:30.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:30 smithi045 conmon[33655]: ) 513 : cluster [DBG] osdmap e44: 7 total, 7 up, 7 in 2021-07-31T01:43:30.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:30 smithi045 conmon[39718]: cephadm 2021-07-31T01:43:29.226987+0000 mgr.y (mgr 2021-07-31T01:43:30.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:30 smithi045 conmon[39718]: .14146) 162 : cephadm [INF] Detected new or changed devices on smithi154 2021-07-31T01:43:30.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:30 smithi045 conmon[39718]: cephadm 2021-07-31T01:43:29.235621+0000 mgr.y (mgr.14146) 163 : 2021-07-31T01:43:30.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:30 smithi045 conmon[39718]: cephadm [INF] Adjusting osd_memory_target on smithi154 to 5723M 2021-07-31T01:43:30.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:30 smithi045 conmon[39718]: cluster 2021-07-31T 2021-07-31T01:43:30.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:30 smithi045 conmon[39718]: 01:43:29.332273+0000 mgr.y (mgr.14146) 164 2021-07-31T01:43:30.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:30 smithi045 conmon[39718]: : cluster [DBG] pgmap v148: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2021-07-31T01:43:30.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:30 smithi045 conmon[39718]: cluster 2021-07-31T01:43 2021-07-31T01:43:30.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:30 smithi045 conmon[39718]: :29.478080+0000 mon.a (mon.0) 2021-07-31T01:43:30.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:30 smithi045 conmon[39718]: 513 : cluster [DBG] osdmap e44: 7 total, 7 up, 7 in 2021-07-31T01:43:31.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:31 smithi154 conmon[34553]: audit 2021-07-31T01:43:30.969062+0000 mon.b (mon.2) 19 : audit [INF] from='client.? 172.21.15.154:0/3210412899' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "da82715a-b8b7-4786-bbc2-ee0cc3ac9b18"}]: dispatch 2021-07-31T01:43:31.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:31 smithi154 conmon[34553]: audit 2021-07-31T01:43 2021-07-31T01:43:31.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:31 smithi154 conmon[34553]: :30.969686+0000 mon.a (mon.0) 514 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "da82715a-b8b7-4786-bbc2-ee0cc3ac9b18"}]: dispatch 2021-07-31T01:43:31.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:31 smithi154 conmon[34553]: audit 2021-07-31T01:43:30.974321+0000 mon.a (mon.0) 515 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "da82715a-b8b7-4786-bbc2-ee0cc3ac9b18"}]': finished 2021-07-31T01:43:31.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:31 smithi154 conmon[34553]: cluster 2021-07-31T01:43:30.974437+0000 mon.a (mon.0) 516 : cluster [DBG] osdmap e45: 8 total, 7 up, 8 in 2021-07-31T01:43:31.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:31 smithi154 conmon[34553]: audit 2021-07-31T01:43:30.974586+0000 mon.a (mon.0) 517 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-07-31T01:43:31.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:31 smithi045 conmon[33655]: audit 2021-07-31T01:43:30.969062+0000 mon.b (mon.2) 19 : audit [INF] from='client.? 172.21.15.154:0/3210412899' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "da82715a-b8b7-4786-bbc2-ee0cc3ac9b18"}]: dispatch 2021-07-31T01:43:31.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:31 smithi045 conmon[33655]: audit 2021-07-31T01:43:30.969686+0000 mon.a (mon.0) 514 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "da82715a-b8b7-4786-bbc2-ee0cc3ac9b18"}]: dispatch 2021-07-31T01:43:31.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:31 smithi045 conmon[33655]: audit 2021-07-31T01:43:30.974321 2021-07-31T01:43:31.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:31 smithi045 conmon[33655]: +0000 mon.a (mon.0) 515 : 2021-07-31T01:43:31.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:31 smithi045 conmon[33655]: audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "da82715a-b8b7-4786-bbc2-ee0cc3ac9b18"}]': finished 2021-07-31T01:43:31.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:31 smithi045 conmon[33655]: cluster 2021-07-31 2021-07-31T01:43:31.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:31 smithi045 conmon[33655]: T01:43:30.974437+0000 mon.a (mon 2021-07-31T01:43:31.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:31 smithi045 conmon[33655]: .0) 516 : cluster [DBG] 2021-07-31T01:43:31.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:31 smithi045 conmon[33655]: osdmap e45: 8 total, 7 up, 8 in 2021-07-31T01:43:31.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:31 smithi045 conmon[33655]: audit 2021 2021-07-31T01:43:31.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:31 smithi045 conmon[33655]: -07-31T01:43:30.974586 2021-07-31T01:43:31.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:31 smithi045 conmon[33655]: +0000 mon.a (mon.0) 2021-07-31T01:43:31.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:31 smithi045 conmon[33655]: 517 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-07-31T01:43:31.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:31 smithi045 conmon[39718]: audit 2021-07-31T01:43:30.969062 2021-07-31T01:43:31.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:31 smithi045 conmon[39718]: +0000 mon.b (mon.2) 19 : audit [INF] from='client.? 172.21.15.154:0/3210412899' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "da82715a-b8b7-4786-bbc2-ee0cc3ac9b18"}]: dispatch 2021-07-31T01:43:31.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:31 smithi045 conmon[39718]: audit 2021-07-31T01:43:30.969686 2021-07-31T01:43:31.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:31 smithi045 conmon[39718]: +0000 mon.a (mon.0) 514 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "da82715a-b8b7-4786-bbc2-ee0cc3ac9b18"}]: dispatch 2021-07-31T01:43:31.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:31 smithi045 conmon[39718]: audit 2021-07-31T01:43:31.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:31 smithi045 conmon[39718]: 2021-07-31T01:43:30.974321+0000 2021-07-31T01:43:31.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:31 smithi045 conmon[39718]: mon.a (mon.0) 515 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "da82715a-b8b7-4786-bbc2-ee0cc3ac9b18"}]': finished 2021-07-31T01:43:31.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:31 smithi045 conmon[39718]: cluster 2021 2021-07-31T01:43:31.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:31 smithi045 conmon[39718]: -07-31T01:43:30.974437+0000 2021-07-31T01:43:31.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:31 smithi045 conmon[39718]: mon.a (mon.0) 516 : cluster [DBG] osdmap e45: 8 total, 7 up, 8 in 2021-07-31T01:43:31.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:31 smithi045 conmon[39718]: audit 2021-07-31T01 2021-07-31T01:43:31.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:31 smithi045 conmon[39718]: :43:30.974586+0000 mon.a (mon.0) 517 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-07-31T01:43:31.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:31 smithi045 conmon[39718]: 2021-07-31T01:43:32.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:32 smithi154 conmon[34553]: cluster 2021-07-31T01:43:31.332660+0000 mgr.y (mgr.14146) 165 : cluster 2021-07-31T01:43:32.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:32 smithi154 conmon[34553]: [DBG] pgmap v151: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2021-07-31T01:43:32.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:32 smithi154 conmon[34553]: audit 2021-07-31T01:43:31.478383+0000 mon.b (mon.2) 20 : audit [DBG] from='client.? 172.21.15.154:0/3188334950' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-07-31T01:43:32.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:32 smithi045 conmon[33655]: cluster 2021-07-31T01:43:31.332660+0000 mgr.y (mgr 2021-07-31T01:43:32.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:32 smithi045 conmon[33655]: .14146) 165 : cluster [DBG] pgmap v151: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2021-07-31T01:43:32.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:32 smithi045 conmon[33655]: audit 2021-07-31T01:43:31.478383+0000 mon.b (mon.2) 20 : audit [DBG] from='client.? 172.21.15.154:0/3188334950' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-07-31T01:43:32.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:32 smithi045 conmon[39718]: cluster 2021-07-31T01:43:31.332660+0000 mgr.y ( 2021-07-31T01:43:32.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:32 smithi045 conmon[39718]: mgr.14146) 165 : cluster [DBG] pgmap v151: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2021-07-31T01:43:32.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:32 smithi045 conmon[39718]: audit 2021-07-31T01:43:31.478383 2021-07-31T01:43:32.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:32 smithi045 conmon[39718]: +0000 mon.b (mon.2) 20 : audit [DBG] from='client.? 172.21.15.154:0/3188334950' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-07-31T01:43:34.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:34 smithi154 conmon[34553]: cluster 2021-07-31T01:43:33.333155+0000 mgr.y (mgr.14146) 166 2021-07-31T01:43:34.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:34 smithi154 conmon[34553]: : cluster [DBG] pgmap v152: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2021-07-31T01:43:34.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:34 smithi045 conmon[33655]: cluster 2021-07-31T01:43:33.333155+0000 mgr.y (mgr.14146) 166 : cluster [DBG] pgmap v152: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2021-07-31T01:43:34.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:34 smithi045 conmon[39718]: cluster 2021-07-31T01:43:33.333155+0000 mgr.y (mgr.14146) 166 : cluster [DBG] 2021-07-31T01:43:34.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:34 smithi045 conmon[39718]: pgmap v152: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2021-07-31T01:43:36.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:36 smithi154 conmon[34553]: cluster 2021-07-31T01:43:35.333612+0000 mgr.y (mgr.14146) 167 : cluster [DBG] pgmap v153: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2021-07-31T01:43:36.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:36 smithi045 conmon[33655]: cluster 2021 2021-07-31T01:43:36.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:36 smithi045 conmon[33655]: -07-31T01:43:35.333612+0000 mgr.y (mgr.14146) 167 : cluster [DBG] pgmap v153: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2021-07-31T01:43:36.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:36 smithi045 conmon[39718]: cluster 2021-07-31T01:43:35 2021-07-31T01:43:36.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:36 smithi045 conmon[39718]: .333612+0000 mgr.y (mgr.14146) 167 : cluster [DBG] pgmap v153: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2021-07-31T01:43:37.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:37 smithi154 conmon[34553]: audit 2021-07-31T01:43:37.442013+0000 mon.a (mon.0) 518 : audit 2021-07-31T01:43:37.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:37 smithi154 conmon[34553]: [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2021-07-31T01:43:37.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:37 smithi154 conmon[34553]: audit 2021-07-31T01:43:37.442925+0000 mon.a (mon.0) 519 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:43:37.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:37 smithi045 conmon[33655]: audit 2021-07-31T01:43:37.442013+0000 mon.a (mon.0 2021-07-31T01:43:37.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:37 smithi045 conmon[33655]: ) 518 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2021-07-31T01:43:37.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:37 smithi045 conmon[33655]: audit 2021-07-31T01:43:37.442925+0000 mon.a ( 2021-07-31T01:43:37.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:37 smithi045 conmon[33655]: mon.0) 519 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:43:37.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:37 smithi045 conmon[39718]: audit 2021-07-31T01:43:37.442013+0000 mon.a ( 2021-07-31T01:43:37.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:37 smithi045 conmon[39718]: mon.0) 518 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2021-07-31T01:43:37.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:37 smithi045 conmon[39718]: audit 2021-07-31T01:43:37.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:37 smithi045 conmon[39718]: 2021-07-31T01:43:37.442925+0000 mon.a (mon.0) 519 : 2021-07-31T01:43:37.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:37 smithi045 conmon[39718]: audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:43:38.562 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:38 smithi154 conmon[34553]: cluster 2021-07-31T01:43:37.334097+0000 mgr.y ( 2021-07-31T01:43:38.563 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:38 smithi154 conmon[34553]: mgr.14146) 168 : cluster [DBG] pgmap v154: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2021-07-31T01:43:38.563 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:38 smithi154 conmon[34553]: cephadm 2021-07-31T01:43:37.443712+0000 mgr.y (mgr.14146) 169 : cephadm [INF] Deploying daemon osd.7 on smithi154 2021-07-31T01:43:38.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:38 smithi045 conmon[33655]: cluster 2021-07-31T01:43:37.334097+0000 mgr.y (mgr 2021-07-31T01:43:38.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:38 smithi045 conmon[33655]: .14146) 168 : cluster [DBG] pgmap v154: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2021-07-31T01:43:38.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:38 smithi045 conmon[33655]: cephadm 2021-07-31 2021-07-31T01:43:38.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:38 smithi045 conmon[33655]: T01:43:37.443712+0000 mgr.y (mgr.14146) 2021-07-31T01:43:38.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:38 smithi045 conmon[33655]: 169 : cephadm [INF] Deploying daemon osd.7 on smithi154 2021-07-31T01:43:38.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:38 smithi045 conmon[39718]: cluster 2021-07-31T01:43:37 2021-07-31T01:43:38.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:38 smithi045 conmon[39718]: .334097+0000 mgr.y (mgr.14146) 168 : cluster [DBG] pgmap v154: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2021-07-31T01:43:38.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:38 smithi045 conmon[39718]: cephadm 2021 2021-07-31T01:43:38.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:38 smithi045 conmon[39718]: -07-31T01:43: 2021-07-31T01:43:38.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:38 smithi045 conmon[39718]: 37.443712+0000 mgr.y (mgr 2021-07-31T01:43:38.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:38 smithi045 conmon[39718]: .14146) 169 : cephadm [INF] 2021-07-31T01:43:38.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:38 smithi045 conmon[39718]: Deploying daemon osd.7 on smithi154 2021-07-31T01:43:39.544 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:39 smithi154 conmon[34553]: audit 2021-07-31T01:43:39.427234+0000 mon.a ( 2021-07-31T01:43:39.545 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:39 smithi154 conmon[34553]: mon.0) 520 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:43:39.545 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:39 smithi154 conmon[34553]: audit 2021-07-31T01:43:39.427478+0000 mon.a (mon.0) 521 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-07-31T01:43:39.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:39 smithi045 conmon[33655]: audit 2021-07-31T01:43:39.427234+0000 mon.a (mon.0) 2021-07-31T01:43:39.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:39 smithi045 conmon[33655]: 520 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:43:39.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:39 smithi045 conmon[33655]: audit 2021-07-31T01:43:39.427478+0000 mon.a (mon.0) 521 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-07-31T01:43:39.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:39 smithi045 conmon[39718]: audit 2021-07-31T01:43:39.427234+0000 mon.a (mon.0) 520 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:43:39.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:39 smithi045 conmon[39718]: audit 2021-07-31T01:43:39.427478 2021-07-31T01:43:39.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:39 smithi045 conmon[39718]: +0000 mon.a (mon.0) 521 : 2021-07-31T01:43:39.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:39 smithi045 conmon[39718]: audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-07-31T01:43:40.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:40 smithi154 conmon[34553]: cluster 2021-07-31T01:43:39.334510+0000 mgr.y (mgr 2021-07-31T01:43:40.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:40 smithi154 conmon[34553]: .14146) 170 : cluster [DBG] pgmap v155: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2021-07-31T01:43:40.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:40 smithi045 conmon[33655]: cluster 2021-07-31T01:43:39.334510+0000 mgr.y (mgr.14146) 170 2021-07-31T01:43:40.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:40 smithi045 conmon[33655]: : cluster [DBG] pgmap v155: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2021-07-31T01:43:40.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:40 smithi045 conmon[39718]: cluster 2021-07-31T01:43:39. 2021-07-31T01:43:40.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:40 smithi045 conmon[39718]: 334510+0000 mgr.y (mgr.14146) 170 : cluster [DBG] pgmap v155: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2021-07-31T01:43:41.295 INFO:teuthology.orchestra.run.smithi154.stdout:Created osd(s) 7 on host 'smithi154' 2021-07-31T01:43:41.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:41 smithi154 conmon[34553]: audit 2021-07-31T01:43:41.281995+0000 mon.a (mon.0) 522 2021-07-31T01:43:41.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:41 smithi154 conmon[34553]: : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:43:41.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:41 smithi154 conmon[34553]: audit 2021-07-31T01:43:41.286489+0000 mon.a (mon.0) 523 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:43:41.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:41 smithi154 conmon[34553]: audit 2021-07-31T01:43:41.286845+0000 mon.a (mon.0) 524 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:43:41.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:41 smithi154 conmon[34553]: audit 2021-07-31T01:43:41.288583+0000 mon.a (mon.0) 525 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:43:41.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:41 smithi154 conmon[34553]: audit 2021-07-31T01:43:41.291881+0000 mon.a (mon.0) 526 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:43:41.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:41 smithi045 conmon[33655]: audit 2021-07-31T01:43:41.281995+0000 mon.a (mon.0) 2021-07-31T01:43:41.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:41 smithi045 conmon[33655]: 522 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:43:41.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:41 smithi045 conmon[33655]: audit 2021-07-31T01:43:41.286489+0000 mon.a (mon.0) 523 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:43:41.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:41 smithi045 conmon[33655]: audit 2021-07-31T01:43:41.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:41 smithi045 conmon[33655]: 2021-07-31T01:43:41 2021-07-31T01:43:41.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:41 smithi045 conmon[33655]: .286845+0000 mon.a (mon. 2021-07-31T01:43:41.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:41 smithi045 conmon[33655]: 0) 524 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:43:41.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:41 smithi045 conmon[33655]: 2021-07-31T01:43:41.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:41 smithi045 conmon[33655]: audit 2021-07- 2021-07-31T01:43:41.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:41 smithi045 conmon[33655]: 31T01:43:41.288583+0000 2021-07-31T01:43:41.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:41 smithi045 conmon[33655]: mon.a (mon.0) 525 : 2021-07-31T01:43:41.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:41 smithi045 conmon[33655]: audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:43:41.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:41 smithi045 conmon[33655]: audit 2021-07-31T 2021-07-31T01:43:41.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:41 smithi045 conmon[33655]: 01:43:41.291881+0000 mon.a (mon.0 2021-07-31T01:43:41.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:41 smithi045 conmon[33655]: ) 526 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:43:41.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:41 smithi045 conmon[33655]: 2021-07-31T01:43:41.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:41 smithi045 conmon[39718]: audit 2021-07-31T01:43:41.281995+0000 mon.a ( 2021-07-31T01:43:41.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:41 smithi045 conmon[39718]: mon.0) 522 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:43:41.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:41 smithi045 conmon[39718]: audit 2021-07-31T01:43 2021-07-31T01:43:41.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:41 smithi045 conmon[39718]: :41.286489+0000 mon.a (mon.0) 523 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:43:41.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:41 smithi045 conmon[39718]: audit 2021-07-31T01:43:41.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:41 smithi045 conmon[39718]: 2021-07-31T 2021-07-31T01:43:41.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:41 smithi045 conmon[39718]: 01:43:41.286845+0000 2021-07-31T01:43:41.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:41 smithi045 conmon[39718]: mon.a (mon.0) 524 : audit [DBG] 2021-07-31T01:43:41.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:41 smithi045 conmon[39718]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:43:41.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:41 smithi045 conmon[39718]: audit 2021-07-31T01:43:41.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:41 smithi045 conmon[39718]: 2021- 2021-07-31T01:43:41.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:41 smithi045 conmon[39718]: 07-31T01:43:41 2021-07-31T01:43:41.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:41 smithi045 conmon[39718]: .288583+0000 mon.a (mon.0) 2021-07-31T01:43:41.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:41 smithi045 conmon[39718]: 525 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:43:41.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:41 smithi045 conmon[39718]: audit 2021- 2021-07-31T01:43:41.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:41 smithi045 conmon[39718]: 07-31T01:43:41.291881+0000 mon.a (mon.0) 526 : 2021-07-31T01:43:41.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:41 smithi045 conmon[39718]: audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:43:42.070 DEBUG:teuthology.orchestra.run.smithi154:osd.7> sudo journalctl -f -n 0 -u ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c@osd.7.service 2021-07-31T01:43:42.073 INFO:tasks.cephadm:Adding prometheus.a on smithi154 2021-07-31T01:43:42.073 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f52d2c14-f19f-11eb-8c24-001a4aab830c -- ceph orch apply prometheus '1;smithi154=a' 2021-07-31T01:43:42.100 INFO:journalctl@ceph.osd.7.smithi154.stdout:-- Logs begin at Sat 2021-07-31 01:17:02 UTC. -- 2021-07-31T01:43:42.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:42 smithi154 conmon[34553]: cluster 2021-07-31T01:43:41.334970+0000 mgr.y (mgr.14146) 171 : cluster [DBG] pgmap v156: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2021-07-31T01:43:42.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:42 smithi045 conmon[33655]: cluster 2021-07-31T01:43:41.334970+0000 mgr.y (mgr.14146) 2021-07-31T01:43:42.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:42 smithi045 conmon[33655]: 171 : cluster [DBG] pgmap v156: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2021-07-31T01:43:42.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:42 smithi045 conmon[39718]: cluster 2021-07-31T01:43:41.334970+0000 mgr.y (mgr 2021-07-31T01:43:42.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:42 smithi045 conmon[39718]: .14146) 171 : cluster [DBG] pgmap v156: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2021-07-31T01:43:44.436 INFO:teuthology.orchestra.run.smithi154.stdout:Scheduled prometheus update... 2021-07-31T01:43:44.753 INFO:journalctl@ceph.osd.7.smithi154.stdout:Jul 31 01:43:44 smithi154 conmon[49149]: debug 2021-07-31T01:43:44.387+0000 7f8a9d682080 -1 osd.7 0 log_to_monitors {default=true} 2021-07-31T01:43:45.176 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:44 smithi154 conmon[34553]: cluster 2021-07-31T01:43:43.335554+0000 mgr.y (mgr.14146) 172 : 2021-07-31T01:43:45.177 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:44 smithi154 conmon[34553]: cluster [DBG] pgmap v157: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2021-07-31T01:43:45.177 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:44 smithi154 conmon[34553]: audit 2021-07-31T01:43:43.974911+0000 mon.a (mon.0) 527 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:43:45.177 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:44 smithi154 conmon[34553]: audit 2021-07-31T01:43:44.391573+0000 mon.b (mon.2) 21 : audit [INF] from='osd.7 [v2:172.21.15.154:6824/1263188610,v1:172.21.15.154:6825/1263188610]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2021-07-31T01:43:45.178 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:44 smithi154 conmon[34553]: audit 2021-07-31T01:43:44.392139+0000 mon.a (mon.0) 528 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2021-07-31T01:43:45.178 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:44 smithi154 conmon[34553]: audit 2021-07-31T01:43:44.434498+0000 mon.a (mon.0) 529 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:43:45.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:44 smithi045 conmon[39718]: cluster 2021-07-31T01:43:43.335554+0000 mgr.y (mgr 2021-07-31T01:43:45.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:44 smithi045 conmon[39718]: .14146) 172 : cluster [DBG] pgmap v157: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2021-07-31T01:43:45.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:44 smithi045 conmon[39718]: audit 2021-07-31T01:43:45.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:44 smithi045 conmon[39718]: 2021-07-31T01:43 2021-07-31T01:43:45.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:44 smithi045 conmon[39718]: :43.974911+0000 mon.a (mon.0 2021-07-31T01:43:45.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:44 smithi045 conmon[39718]: ) 527 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:43:45.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:44 smithi045 conmon[39718]: audit 2021-07-31T01:43:44.391573+0000 mon.b (mon.2) 21 : audit [INF] from='osd.7 [v2:172.21.15.154:6824/1263188610,v1:172.21.15.154:6825/1263188610]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2021-07-31T01:43:45.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:44 smithi045 conmon[39718]: audit 2021-07-31T01:43:44 2021-07-31T01:43:45.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:44 smithi045 conmon[39718]: .392139+0000 mon.a (mon.0) 528 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2021-07-31T01:43:45.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:44 smithi045 conmon[39718]: audit 2021- 2021-07-31T01:43:45.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:44 smithi045 conmon[39718]: 07-31T01:43:44 2021-07-31T01:43:45.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:44 smithi045 conmon[39718]: .434498+0000 mon.a (mon. 2021-07-31T01:43:45.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:44 smithi045 conmon[39718]: 0) 529 : audit [INF] 2021-07-31T01:43:45.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:44 smithi045 conmon[39718]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:43:45.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:44 smithi045 conmon[33655]: cluster 2021-07-31T01:43:43.335554+0000 mgr.y (mgr.14146) 172 : cluster 2021-07-31T01:43:45.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:44 smithi045 conmon[33655]: [DBG] pgmap v157: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2021-07-31T01:43:45.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:44 smithi045 conmon[33655]: audit 2021-07-31T01:43:43.974911+0000 mon.a (mon.0) 527 2021-07-31T01:43:45.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:44 smithi045 conmon[33655]: : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:43:45.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:44 smithi045 conmon[33655]: audit 2021-07-31T 2021-07-31T01:43:45.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:44 smithi045 conmon[33655]: 01:43 2021-07-31T01:43:45.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:44 smithi045 conmon[33655]: :44.391573+0000 mon.b (mon 2021-07-31T01:43:45.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:44 smithi045 conmon[33655]: .2) 21 : audit [INF] 2021-07-31T01:43:45.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:44 smithi045 conmon[33655]: from='osd.7 [v2:172.21.15.154:6824/1263188610,v1:172.21.15.154:6825/1263188610]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2021-07-31T01:43:45.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:44 smithi045 conmon[33655]: audit 2021-07-31T01:43:44.392139+0000 mon.a 2021-07-31T01:43:45.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:44 smithi045 conmon[33655]: (mon.0) 528 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2021-07-31T01:43:45.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:44 smithi045 conmon[33655]: audit 2021-07-31T01:43:45.283 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:44 smithi045 conmon[33655]: 2021-07-31T01:43:44.434498+0000 mon.a (mon.0) 529 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:43:45.461 DEBUG:teuthology.orchestra.run.smithi154:prometheus.a> sudo journalctl -f -n 0 -u ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c@prometheus.a.service 2021-07-31T01:43:45.464 INFO:tasks.cephadm:Adding node-exporter.a on smithi045 2021-07-31T01:43:45.464 INFO:tasks.cephadm:Adding node-exporter.b on smithi154 2021-07-31T01:43:45.465 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f52d2c14-f19f-11eb-8c24-001a4aab830c -- ceph orch apply node-exporter '2;smithi045=a;smithi154=b' 2021-07-31T01:43:45.492 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:-- Logs begin at Sat 2021-07-31 01:17:02 UTC. -- 2021-07-31T01:43:46.187 INFO:journalctl@ceph.osd.7.smithi154.stdout:Jul 31 01:43:45 smithi154 conmon[49149]: debug 2021-07-31T01:43:45.982+0000 7f8a93be5700 -1 osd.7 0 waiting for initial osdmap 2021-07-31T01:43:46.188 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:45 smithi154 conmon[34553]: audit 2021-07-31T01:43:44.429886+0000 mgr.y (mgr.14146) 173 : audit [DBG] 2021-07-31T01:43:46.188 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:45 smithi154 conmon[34553]: from='client.24308 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi154=a", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:43:46.188 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:45 smithi154 conmon[34553]: cephadm 2021-07-31T01:43:44.430636+0000 mgr.y (mgr.14146) 174 : cephadm [INF] Saving service prometheus spec with placement smithi154=a;count:1 2021-07-31T01:43:46.188 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:45 smithi154 conmon[34553]: audit 2021 2021-07-31T01:43:46.189 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:45 smithi154 conmon[34553]: -07-31T01:43:44.977589+0000 mon.a (mon.0) 530 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2021-07-31T01:43:46.189 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:45 smithi154 conmon[34553]: cluster 2021-07-31T 2021-07-31T01:43:46.189 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:45 smithi154 conmon[34553]: 01:43:44.977794+0000 mon.a (mon.0) 531 2021-07-31T01:43:46.189 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:45 smithi154 conmon[34553]: : cluster [DBG] osdmap e46: 8 total, 7 up, 8 in 2021-07-31T01:43:46.189 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:45 smithi154 conmon[34553]: audit 2021- 2021-07-31T01:43:46.190 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:45 smithi154 conmon[34553]: 07-31T01:43:44.978321+0000 mon.a ( 2021-07-31T01:43:46.190 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:45 smithi154 conmon[34553]: mon.0) 532 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-07-31T01:43:46.190 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:45 smithi154 conmon[34553]: audit 2021-07 2021-07-31T01:43:46.190 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:45 smithi154 conmon[34553]: -31T01:43:44.978358+0000 mon.b (mon.2 2021-07-31T01:43:46.191 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:45 smithi154 conmon[34553]: ) 22 : audit [INF] from='osd.7 [v2:172.21.15.154:6824/1263188610,v1:172.21.15.154:6825/1263188610]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2021-07-31T01:43:46.191 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:45 smithi154 conmon[34553]: audit 2021-07-31T01 2021-07-31T01:43:46.191 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:45 smithi154 conmon[34553]: :43:44.978831+0000 mon.a (mon.0) 533 2021-07-31T01:43:46.191 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:45 smithi154 conmon[34553]: : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2021-07-31T01:43:46.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:45 smithi045 conmon[39718]: audit 2021-07-31T01:43:44.429886+0000 mgr.y (mgr.14146) 2021-07-31T01:43:46.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:45 smithi045 conmon[39718]: 173 : audit [DBG] from='client.24308 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi154=a", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:43:46.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:45 smithi045 conmon[39718]: cephadm 2021-07-31T01:43:44.430636+0000 mgr.y (mgr.14146) 174 : cephadm [INF] Saving service prometheus spec with placement smithi154=a;count:1 2021-07-31T01:43:46.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:45 smithi045 conmon[39718]: audit 2021-07-31T01:43:46.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:45 smithi045 conmon[39718]: 2021-07-31T01:43:44.977589+0000 2021-07-31T01:43:46.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:45 smithi045 conmon[39718]: mon.a (mon.0) 530 : audit 2021-07-31T01:43:46.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:45 smithi045 conmon[39718]: [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2021-07-31T01:43:46.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:45 smithi045 conmon[39718]: cluster 2021-07-31T01:43:44.977794+0000 mon.a (mon.0) 531 : cluster [DBG] osdmap e46: 8 total, 7 up, 8 in 2021-07-31T01:43:46.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:45 smithi045 conmon[39718]: audit 2021-07-31T01:43:44.978321+0000 mon.a (mon.0) 532 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-07-31T01:43:46.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:45 smithi045 conmon[39718]: audit 2021-07-31T01:43:44.978358+0000 mon.b (mon.2) 22 : audit [INF] from='osd.7 [v2:172.21.15.154:6824/1263188610,v1:172.21.15.154:6825/1263188610]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2021-07-31T01:43:46.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:45 smithi045 conmon[39718]: audit 2021 2021-07-31T01:43:46.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:45 smithi045 conmon[39718]: -07-31T01:43:44.978831 2021-07-31T01:43:46.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:45 smithi045 conmon[39718]: +0000 mon.a (mon.0) 533 : audit 2021-07-31T01:43:46.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:45 smithi045 conmon[39718]: [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2021-07-31T01:43:46.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:45 smithi045 conmon[33655]: audit 2021-07-31T01:43:44.429886+0000 mgr.y ( 2021-07-31T01:43:46.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:45 smithi045 conmon[33655]: mgr.14146) 173 : audit [DBG] from='client.24308 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi154=a", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:43:46.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:45 smithi045 conmon[33655]: cephadm 2021-07-31T01:43:44.430636+0000 mgr.y (mgr.14146) 174 : cephadm [INF] Saving service prometheus spec with placement smithi154=a;count:1 2021-07-31T01:43:46.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:45 smithi045 conmon[33655]: audit 2021-07-31T01:43:44.977589+0000 mon.a ( 2021-07-31T01:43:46.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:45 smithi045 conmon[33655]: mon.0) 530 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2021-07-31T01:43:46.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:45 smithi045 conmon[33655]: cluster 2021-07-31T01:43 2021-07-31T01:43:46.283 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:45 smithi045 conmon[33655]: :44.977794+0000 mon.a (mon.0) 531 : cluster [DBG] osdmap e46: 8 total, 7 up, 8 in 2021-07-31T01:43:46.283 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:45 smithi045 conmon[33655]: audit 2021 2021-07-31T01:43:46.283 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:45 smithi045 conmon[33655]: -07-31T01:43:44.978321+0000 mon.a (mon.0) 532 2021-07-31T01:43:46.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:45 smithi045 conmon[33655]: : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-07-31T01:43:46.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:45 smithi045 conmon[33655]: audit 2021-07-31T01:43:46.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:45 smithi045 conmon[33655]: 2021-07-31T01:43:44.978358+0000 2021-07-31T01:43:46.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:45 smithi045 conmon[33655]: mon.b (mon.2) 22 : audit [INF] from='osd.7 [v2:172.21.15.154:6824/1263188610,v1:172.21.15.154:6825/1263188610]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2021-07-31T01:43:46.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:45 smithi045 conmon[33655]: 2021-07-31T01:43:46.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:45 smithi045 conmon[33655]: audit 2021-07 2021-07-31T01:43:46.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:45 smithi045 conmon[33655]: -31T01:43:44.978831+0000 mon.a ( 2021-07-31T01:43:46.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:45 smithi045 conmon[33655]: mon.0) 533 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2021-07-31T01:43:47.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:46 smithi154 conmon[34553]: cluster 2021-07-31T01:43:45.335981+0000 mgr.y (mgr.14146 2021-07-31T01:43:47.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:46 smithi154 conmon[34553]: ) 175 : cluster [DBG] pgmap v159: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2021-07-31T01:43:47.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:46 smithi154 conmon[34553]: audit 2021-07-31T01:43:45.979696+0000 mon.a (mon.0) 534 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2021-07-31T01:43:47.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:46 smithi154 conmon[34553]: cluster 2021-07-31T01:43:45.979856+0000 mon.a (mon.0) 535 : cluster [DBG] osdmap e47: 8 total, 7 up, 8 in 2021-07-31T01:43:47.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:46 smithi154 conmon[34553]: audit 2021-07-31T01:43:45.979982+0000 mon.a (mon.0) 536 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-07-31T01:43:47.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:46 smithi154 conmon[34553]: audit 2021-07-31T01:43:45.984284+0000 mon.a (mon.0) 537 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-07-31T01:43:47.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:46 smithi154 conmon[34553]: audit 2021-07-31T01:43:46.983524+0000 mon.a (mon.0) 538 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-07-31T01:43:47.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:46 smithi045 conmon[39718]: cluster 2021-07-31T01:43:45.335981+0000 mgr.y ( 2021-07-31T01:43:47.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:46 smithi045 conmon[39718]: mgr.14146) 175 : cluster [DBG] pgmap v159: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2021-07-31T01:43:47.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:46 smithi045 conmon[39718]: audit 2021-07-31T01:43:45.979696+0000 mon.a (mon.0) 534 : audit [INF] 2021-07-31T01:43:47.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:46 smithi045 conmon[39718]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2021-07-31T01:43:47.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:46 smithi045 conmon[39718]: cluster 2021-07-31T01:43:45.979856+0000 mon.a (mon.0) 535 : cluster [DBG] 2021-07-31T01:43:47.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:46 smithi045 conmon[39718]: osdmap e47: 8 total, 7 up, 8 in 2021-07-31T01:43:47.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:46 smithi045 conmon[39718]: audit 2021-07-31T01:43:45.979982+0000 mon.a (mon.0) 2021-07-31T01:43:47.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:46 smithi045 conmon[39718]: 536 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-07-31T01:43:47.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:46 smithi045 conmon[39718]: audit 2021 2021-07-31T01:43:47.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:46 smithi045 conmon[39718]: -07-31T01:43:45.984284+0000 mon.a (mon.0) 537 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-07-31T01:43:47.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:46 smithi045 conmon[39718]: audit 2021 2021-07-31T01:43:47.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:46 smithi045 conmon[39718]: -07-31T01:43:46.983524+0000 mon.a (mon 2021-07-31T01:43:47.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:46 smithi045 conmon[39718]: .0) 538 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-07-31T01:43:47.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:46 smithi045 conmon[33655]: cluster 2021-07-31T01:43:45.335981+0000 mgr.y (mgr.14146) 2021-07-31T01:43:47.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:46 smithi045 conmon[33655]: 175 : cluster [DBG] pgmap v159: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2021-07-31T01:43:47.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:46 smithi045 conmon[33655]: audit 2021-07-31T01:43:45.979696+0000 mon.a (mon.0) 534 : audit 2021-07-31T01:43:47.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:46 smithi045 conmon[33655]: [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2021-07-31T01:43:47.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:46 smithi045 conmon[33655]: cluster 2021-07-31T01:43:47.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:46 smithi045 conmon[33655]: 2021-07-31T01:43:45.979856+0000 mon.a (mon.0) 535 2021-07-31T01:43:47.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:46 smithi045 conmon[33655]: : cluster [DBG] osdmap e47: 8 total, 7 up, 8 in 2021-07-31T01:43:47.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:46 smithi045 conmon[33655]: audit 2021-07-31T01:43:45. 2021-07-31T01:43:47.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:46 smithi045 conmon[33655]: 979982+0000 mon.a (mon.0) 536 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-07-31T01:43:47.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:46 smithi045 conmon[33655]: audit 2021-07-31T01:43:45 2021-07-31T01:43:47.283 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:46 smithi045 conmon[33655]: .984284+0000 mon.a (mon.0) 537 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-07-31T01:43:47.283 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:46 smithi045 conmon[33655]: audit 2021-07-31T01:43:46.983524 2021-07-31T01:43:47.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:46 smithi045 conmon[33655]: +0000 mon.a (mon.0) 538 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-07-31T01:43:48.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:47 smithi154 conmon[34553]: cluster 2021-07-31T01:43:45.387695+0000 osd.7 (osd.7) 1 : cluster [DBG] purged_snaps scrub starts 2021-07-31T01:43:48.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:47 smithi154 conmon[34553]: cluster 2021-07-31T01:43:45.387894+0000 osd.7 (osd.7) 2 : cluster [DBG] purged_snaps scrub ok 2021-07-31T01:43:48.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:47 smithi154 conmon[34553]: cluster 2021-07-31T01:43:46.989504+0000 mon.a (mon.0) 539 : cluster [INF] osd.7 [v2:172.21.15.154:6824/1263188610,v1:172.21.15.154:6825/1263188610] boot 2021-07-31T01:43:48.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:47 smithi154 conmon[34553]: cluster 2021-07-31T01:43:46.989579+0000 mon.a (mon.0) 540 : cluster [DBG] osdmap e48: 8 total, 8 up, 8 in 2021-07-31T01:43:48.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:47 smithi154 conmon[34553]: audit 2021-07-31T01:43:46.990078+0000 mon.a (mon.0) 541 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-07-31T01:43:48.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:47 smithi045 conmon[33655]: cluster 2021-07-31T01:43:45.387695+0000 2021-07-31T01:43:48.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:47 smithi045 conmon[33655]: osd.7 (osd.7) 1 : cluster [DBG] purged_snaps scrub starts 2021-07-31T01:43:48.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:47 smithi045 conmon[33655]: cluster 2021-07-31T01:43:45.387894+0000 osd.7 (osd.7 2021-07-31T01:43:48.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:47 smithi045 conmon[33655]: ) 2 : cluster [DBG] purged_snaps scrub ok 2021-07-31T01:43:48.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:47 smithi045 conmon[33655]: cluster 2021-07-31T01:43:46.989504+0000 mon.a (mon. 2021-07-31T01:43:48.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:47 smithi045 conmon[33655]: 0) 539 : cluster [INF] osd.7 [v2:172.21.15.154:6824/1263188610,v1:172.21.15.154:6825/1263188610] boot 2021-07-31T01:43:48.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:47 smithi045 conmon[33655]: cluster 2021 2021-07-31T01:43:48.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:47 smithi045 conmon[33655]: -07-31T01:43:46.989579+0000 2021-07-31T01:43:48.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:47 smithi045 conmon[33655]: mon.a (mon.0) 540 : cluster [DBG] osdmap e48: 8 total, 8 up, 8 in 2021-07-31T01:43:48.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:47 smithi045 conmon[33655]: audit 2021-07-31T 2021-07-31T01:43:48.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:47 smithi045 conmon[33655]: 01:43:46.990078+0000 mon.a (mon.0) 541 : 2021-07-31T01:43:48.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:47 smithi045 conmon[33655]: audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-07-31T01:43:48.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:47 smithi045 conmon[39718]: cluster 2021-07-31T01:43:45.387695+0000 osd.7 (osd.7) 1 : cluster 2021-07-31T01:43:48.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:47 smithi045 conmon[39718]: [DBG] purged_snaps scrub starts 2021-07-31T01:43:48.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:47 smithi045 conmon[39718]: cluster 2021-07-31T01:43:45.387894+0000 osd.7 (osd.7) 2 2021-07-31T01:43:48.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:47 smithi045 conmon[39718]: : cluster [DBG] purged_snaps scrub ok 2021-07-31T01:43:48.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:47 smithi045 conmon[39718]: cluster 2021-07-31T01:43:46.989504+0000 2021-07-31T01:43:48.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:47 smithi045 conmon[39718]: mon.a (mon.0) 539 : cluster [INF] osd.7 [v2:172.21.15.154:6824/1263188610,v1:172.21.15.154:6825/1263188610] boot 2021-07-31T01:43:48.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:47 smithi045 conmon[39718]: cluster 2021-07-31T01 2021-07-31T01:43:48.282 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:47 smithi045 conmon[39718]: :43:46.989579+0000 mon.a (mon.0) 540 : cluster 2021-07-31T01:43:48.282 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:47 smithi045 conmon[39718]: [DBG] osdmap e48: 8 total, 8 up, 8 in 2021-07-31T01:43:48.282 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:47 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T01:43:48.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:47 smithi045 conmon[39718]: 31T01:43:46.990078+0000 mon.a (mon.0) 541 : audit [DBG] 2021-07-31T01:43:48.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:47 smithi045 conmon[39718]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-07-31T01:43:48.598 INFO:teuthology.orchestra.run.smithi154.stdout:Scheduled node-exporter update... 2021-07-31T01:43:48.998 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:48 smithi154 conmon[34553]: cluster 2021-07-31T01:43:47.336502+0000 mgr.y (mgr.14146) 176 : cluster [DBG] pgmap v162: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2021-07-31T01:43:48.998 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:48 smithi154 conmon[34553]: cluster 2021-07-31T01:43:47.998961+0000 mon.a (mon.0) 542 : cluster [DBG] osdmap e49: 8 total, 8 up, 8 in 2021-07-31T01:43:48.998 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:48 smithi154 conmon[34553]: cephadm 2021-07-31T01:43:48.532138+0000 mgr.y (mgr.14146) 177 : cephadm [INF] Detected new or changed devices on smithi154 2021-07-31T01:43:49.203 DEBUG:teuthology.orchestra.run.smithi045:node-exporter.a> sudo journalctl -f -n 0 -u ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c@node-exporter.a.service 2021-07-31T01:43:49.207 DEBUG:teuthology.orchestra.run.smithi154:node-exporter.b> sudo journalctl -f -n 0 -u ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c@node-exporter.b.service 2021-07-31T01:43:49.212 INFO:tasks.cephadm:Adding alertmanager.a on smithi045 2021-07-31T01:43:49.212 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f52d2c14-f19f-11eb-8c24-001a4aab830c -- ceph orch apply alertmanager '1;smithi045=a' 2021-07-31T01:43:49.240 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:-- Logs begin at Sat 2021-07-31 01:17:02 UTC. -- 2021-07-31T01:43:49.251 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:-- Logs begin at Sat 2021-07-31 01:16:15 UTC. -- 2021-07-31T01:43:49.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:48 smithi154 conmon[34553]: audit 2021-07-31T01:43:48.537365+0000 mon.a (mon.0) 543 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:43:49.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:48 smithi154 conmon[34553]: audit 2021-07-31T01:43:48.538480+0000 2021-07-31T01:43:49.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:48 smithi154 conmon[34553]: mon.a (mon.0) 544 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-07-31T01:43:49.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:48 smithi154 conmon[34553]: audit 2021-07 2021-07-31T01:43:49.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:48 smithi154 conmon[34553]: -31T01:43:48.539604+0000 mon.a (mon.0) 545 2021-07-31T01:43:49.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:48 smithi154 conmon[34553]: : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-07-31T01:43:49.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:48 smithi154 conmon[34553]: audit 2021 2021-07-31T01:43:49.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:48 smithi154 conmon[34553]: -07-31T01:43:48.540588 2021-07-31T01:43:49.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:48 smithi154 conmon[34553]: +0000 mon.a (mon.0) 546 : audit [INF] 2021-07-31T01:43:49.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:48 smithi154 conmon[34553]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-07-31T01:43:49.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:48 smithi154 conmon[34553]: audit 2021-07-31T01:43:49.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:48 smithi154 conmon[34553]: 2021-07-31T01:43:48.541761 2021-07-31T01:43:49.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:48 smithi154 conmon[34553]: +0000 mon.a (mon.0) 547 : audit [INF] 2021-07-31T01:43:49.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:48 smithi154 conmon[34553]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-07-31T01:43:49.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:48 smithi154 conmon[34553]: cephadm 2021-07- 2021-07-31T01:43:49.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:48 smithi154 conmon[34553]: 31T01:43:48.542440+0000 mgr.y 2021-07-31T01:43:49.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:48 smithi154 conmon[34553]: (mgr.14146) 178 : cephadm [INF] Adjusting osd_memory_target on smithi154 to 4292M 2021-07-31T01:43:49.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:48 smithi154 conmon[34553]: 2021-07-31T01:43:49.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:48 smithi154 conmon[34553]: audit 2021-07-31T 2021-07-31T01:43:49.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:48 smithi154 conmon[34553]: 01:43:48.546775+0000 mon.a (mon.0 2021-07-31T01:43:49.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:48 smithi154 conmon[34553]: ) 548 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:43:49.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:48 smithi154 conmon[34553]: audit 2021- 2021-07-31T01:43:49.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:48 smithi154 conmon[34553]: 07-31T01:43:48.550954+0000 2021-07-31T01:43:49.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:48 smithi154 conmon[34553]: mon.a (mon.0) 549 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:43:49.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:48 smithi154 conmon[34553]: cephadm 2021-07-31T01 2021-07-31T01:43:49.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:48 smithi154 conmon[34553]: :43:48.563346+0000 mgr.y (mgr.14146) 179 2021-07-31T01:43:49.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:48 smithi154 conmon[34553]: : cephadm [INF] Deploying daemon prometheus.a on smithi154 2021-07-31T01:43:49.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:48 smithi154 conmon[34553]: audit 2021-07- 2021-07-31T01:43:49.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:48 smithi154 conmon[34553]: 31T01:43:48.592816+0000 mgr.y ( 2021-07-31T01:43:49.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:48 smithi154 conmon[34553]: mgr.14146) 180 : audit [DBG] from='client.24317 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi045=a;smithi154=b", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:43:49.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:48 smithi154 conmon[34553]: cephadm 2021 2021-07-31T01:43:49.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:48 smithi154 conmon[34553]: -07-31T01:43:48.593500+0000 2021-07-31T01:43:49.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:48 smithi154 conmon[34553]: mgr.y (mgr.14146) 181 : cephadm [INF] 2021-07-31T01:43:49.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:48 smithi154 conmon[34553]: Saving service node-exporter spec with placement smithi045=a;smithi154=b;count:2 2021-07-31T01:43:49.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:48 smithi154 conmon[34553]: audit 2021-07-31T 2021-07-31T01:43:49.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:48 smithi154 conmon[34553]: 01:43:48.597330+0000 mon.a (mon.0) 550 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:43:49.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:48 smithi045 conmon[33655]: cluster 2021-07-31T01:43:47.336502+0000 mgr.y (mgr.14146) 176 : cluster [DBG] 2021-07-31T01:43:49.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:48 smithi045 conmon[33655]: pgmap v162: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2021-07-31T01:43:49.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:48 smithi045 conmon[33655]: cluster 2021-07-31T01:43:47.998961+0000 mon.a (mon.0) 542 : cluster [DBG] osdmap e49: 8 total, 8 up, 8 in 2021-07-31T01:43:49.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:48 smithi045 conmon[33655]: cephadm 2021-07-31T01:43:48.532138+0000 mgr.y (mgr.14146) 177 : cephadm [INF] 2021-07-31T01:43:49.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:48 smithi045 conmon[33655]: Detected new or changed devices on smithi154 2021-07-31T01:43:49.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:48 smithi045 conmon[33655]: audit 2021-07-31T01:43:48.537365+0000 mon.a (mon.0) 543 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:43:49.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:48 smithi045 conmon[33655]: audit 2021-07-31T01:43:48.538480+0000 mon.a (mon.0) 544 : audit [INF] 2021-07-31T01:43:49.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:48 smithi045 conmon[33655]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-07-31T01:43:49.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:48 smithi045 conmon[33655]: audit 2021-07-31 2021-07-31T01:43:49.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:48 smithi045 conmon[33655]: T01:43:48.539604+0000 mon.a (mon.0) 545 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-07-31T01:43:49.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:48 smithi045 conmon[33655]: audit 2021-07-31T01:43:48.540588+0000 mon.a ( 2021-07-31T01:43:49.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:48 smithi045 conmon[33655]: mon.0) 546 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-07-31T01:43:49.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:48 smithi045 conmon[33655]: audit 2021-07-31T01:43:48.541761+0000 mon.a (mon.0 2021-07-31T01:43:49.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:48 smithi045 conmon[33655]: ) 547 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-07-31T01:43:49.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:48 smithi045 conmon[33655]: cephadm 2021-07-31T01:43:49.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:48 smithi045 conmon[33655]: 2021-07-31T01:43:48.542440+0000 mgr.y (mgr.14146) 178 : cephadm [INF] Adjusting osd_memory_target on smithi154 to 4292M 2021-07-31T01:43:49.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:48 smithi045 conmon[33655]: audit 2021-07-31T 2021-07-31T01:43:49.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:48 smithi045 conmon[33655]: 01:43:48.546775+0000 mon.a (mon.0) 548 : audit [INF] 2021-07-31T01:43:49.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:48 smithi045 conmon[33655]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:43:49.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:48 smithi045 conmon[33655]: audit 2021-07-31T01:43:49.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:48 smithi045 conmon[33655]: 2021-07-31T01:43:48.550954+0000 mon.a (mon.0) 549 : audit [INF] 2021-07-31T01:43:49.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:48 smithi045 conmon[33655]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:43:49.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:48 smithi045 conmon[33655]: cephadm 2021-07-31T01: 2021-07-31T01:43:49.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:48 smithi045 conmon[33655]: 43:48.563346+0000 mgr.y (mgr.14146) 179 : 2021-07-31T01:43:49.283 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:48 smithi045 conmon[33655]: cephadm [INF] Deploying daemon prometheus.a on smithi154 2021-07-31T01:43:49.283 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:48 smithi045 conmon[33655]: audit 2021-07- 2021-07-31T01:43:49.283 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:48 smithi045 conmon[33655]: 31T01:43:48.592816+0000 mgr.y (mgr.14146) 2021-07-31T01:43:49.283 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:48 smithi045 conmon[33655]: 180 : audit [DBG] from='client.24317 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi045=a;smithi154=b", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:43:49.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:48 smithi045 conmon[33655]: cephadm 2021-07 2021-07-31T01:43:49.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:48 smithi045 conmon[33655]: -31T01:43:48.593500+0000 mgr.y (mgr 2021-07-31T01:43:49.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:48 smithi045 conmon[33655]: .14146) 181 : cephadm [INF] 2021-07-31T01:43:49.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:48 smithi045 conmon[33655]: Saving service node-exporter spec with placement smithi045=a;smithi154=b;count:2 2021-07-31T01:43:49.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:48 smithi045 conmon[33655]: audit 2021-07 2021-07-31T01:43:49.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:48 smithi045 conmon[33655]: -31T01:43:48.597330+0000 mon.a ( 2021-07-31T01:43:49.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:48 smithi045 conmon[33655]: mon.0) 550 : audit [INF] 2021-07-31T01:43:49.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:48 smithi045 conmon[33655]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:43:49.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:48 smithi045 conmon[39718]: cluster 2021-07-31T01:43:47.336502+0000 mgr.y (mgr.14146) 176 : cluster 2021-07-31T01:43:49.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:48 smithi045 conmon[39718]: [DBG] pgmap v162: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2021-07-31T01:43:49.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:48 smithi045 conmon[39718]: cluster 2021-07-31T01:43:47.998961+0000 mon.a (mon.0) 542 : cluster [DBG] osdmap e49: 8 total, 8 up, 8 in 2021-07-31T01:43:49.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:48 smithi045 conmon[39718]: cephadm 2021-07-31T01:43:48.532138+0000 mgr.y (mgr.14146) 2021-07-31T01:43:49.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:48 smithi045 conmon[39718]: 177 : cephadm [INF] Detected new or changed devices on smithi154 2021-07-31T01:43:49.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:48 smithi045 conmon[39718]: audit 2021-07-31T01:43:49.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:48 smithi045 conmon[39718]: 2021-07-31T01:43:48.537365+0000 mon.a (mon.0) 543 : 2021-07-31T01:43:49.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:48 smithi045 conmon[39718]: audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:43:49.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:48 smithi045 conmon[39718]: audit 2021-07-31T01:43 2021-07-31T01:43:49.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:48 smithi045 conmon[39718]: :48.538480+0000 mon.a (mon.0) 544 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-07-31T01:43:49.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:48 smithi045 conmon[39718]: 2021-07-31T01:43:49.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:48 smithi045 conmon[39718]: audit 2021-07-31T01:43:49.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:48 smithi045 conmon[39718]: 2021-07-31T01:43:48.539604+0000 2021-07-31T01:43:49.289 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:48 smithi045 conmon[39718]: mon.a (mon.0) 545 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-07-31T01:43:49.289 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:48 smithi045 conmon[39718]: 2021-07-31T01:43:49.289 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:48 smithi045 conmon[39718]: audit 2021-07-31T01:43:49.289 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:48 smithi045 conmon[39718]: 2021-07-31T01:43:48.540588+0000 mon.a (mon.0) 2021-07-31T01:43:49.290 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:48 smithi045 conmon[39718]: 546 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-07-31T01:43:49.290 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:48 smithi045 conmon[39718]: audit 2021-07 2021-07-31T01:43:49.290 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:48 smithi045 conmon[39718]: -31T01:43:48.541761+0000 mon.a (mon.0) 547 2021-07-31T01:43:49.290 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:48 smithi045 conmon[39718]: : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-07-31T01:43:49.291 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:48 smithi045 conmon[39718]: cephadm 2021-07-31T01:43:48.542440+0000 mgr.y (mgr.14146) 2021-07-31T01:43:49.291 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:48 smithi045 conmon[39718]: 178 : cephadm [INF] Adjusting osd_memory_target on smithi154 to 4292M 2021-07-31T01:43:49.292 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:48 smithi045 conmon[39718]: audit 2021- 2021-07-31T01:43:49.292 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:48 smithi045 conmon[39718]: 07-31T01:43:48.546775+0000 mon.a (mon.0) 548 : audit 2021-07-31T01:43:49.292 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:48 smithi045 conmon[39718]: [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:43:49.293 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:48 smithi045 conmon[39718]: audit 2021-07-31T01:43:49.293 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:48 smithi045 conmon[39718]: 2021-07-31T01:43:48.550954+0000 2021-07-31T01:43:49.293 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:48 smithi045 conmon[39718]: mon.a (mon.0) 549 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:43:49.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:48 smithi045 conmon[39718]: cephadm 2021-07 2021-07-31T01:43:49.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:48 smithi045 conmon[39718]: -31T01:43:48.563346+0000 mgr.y (mgr.14146) 179 2021-07-31T01:43:49.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:48 smithi045 conmon[39718]: : cephadm [INF] Deploying daemon prometheus.a on smithi154 2021-07-31T01:43:49.295 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:48 smithi045 conmon[39718]: audit 2021-07 2021-07-31T01:43:49.295 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:48 smithi045 conmon[39718]: -31T01:43:48.592816+0000 mgr.y (mgr.14146) 180 : 2021-07-31T01:43:49.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:48 smithi045 conmon[39718]: audit [DBG] from='client.24317 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi045=a;smithi154=b", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:43:49.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:48 smithi045 conmon[39718]: cephadm 2021-07-31 2021-07-31T01:43:49.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:48 smithi045 conmon[39718]: T01:43:48.593500+0000 mgr.y (mgr 2021-07-31T01:43:49.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:48 smithi045 conmon[39718]: .14146) 181 : cephadm [INF] Saving service node-exporter spec with placement smithi045=a;smithi154=b;count:2 2021-07-31T01:43:49.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:48 smithi045 conmon[39718]: audit 2021-07-31T01:43:49.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:48 smithi045 conmon[39718]: 2021-07-31T01:43:48 2021-07-31T01:43:49.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:48 smithi045 conmon[39718]: .597330+0000 mon.a (mon.0) 550 : 2021-07-31T01:43:49.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:48 smithi045 conmon[39718]: audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:43:50.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:50 smithi045 conmon[33655]: cluster 2021-07-31T01:43:49.336865+0000 mgr.y (mgr.14146) 182 : 2021-07-31T01:43:50.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:50 smithi045 conmon[33655]: cluster [DBG] pgmap v164: 1 pgs: 1 peering; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:43:50.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:50 smithi045 conmon[39718]: cluster 2021-07-31T01:43:49 2021-07-31T01:43:50.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:50 smithi045 conmon[39718]: .336865+0000 mgr.y (mgr.14146) 182 : cluster [DBG] pgmap v164: 1 pgs: 1 peering; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:43:50.410 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:50 smithi154 conmon[34553]: cluster 2021-07-31T01:43:49.336865+0000 mgr.y (mgr.14146) 182 : cluster [DBG] pgmap v164: 1 pgs: 1 peering; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:43:52.583 INFO:teuthology.orchestra.run.smithi154.stdout:Scheduled alertmanager update... 2021-07-31T01:43:52.607 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:52 smithi154 conmon[34553]: cluster 2021-07-31T01:43:51.337393+0000 mgr.y (mgr.14146) 183 : cluster [DBG] pgmap v165: 1 pgs: 1 peering; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:43:52.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:52 smithi045 conmon[39718]: cluster 2021-07-31T01:43:51.337393+0000 mgr.y (mgr.14146) 183 : cluster [DBG] 2021-07-31T01:43:52.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:52 smithi045 conmon[39718]: pgmap v165: 1 pgs: 1 peering; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:43:52.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:52 smithi045 conmon[33655]: cluster 2021-07-31T01:43:51. 2021-07-31T01:43:52.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:52 smithi045 conmon[33655]: 337393+0000 mgr.y (mgr.14146) 183 : cluster [DBG] pgmap v165: 1 pgs: 1 peering; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:43:53.959 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:53 smithi154 conmon[34553]: audit 2021-07-31T01:43:52.577739+0000 mgr.y (mgr.14146) 184 2021-07-31T01:43:53.960 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:53 smithi154 conmon[34553]: : audit [DBG] from='client.24323 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi045=a", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:43:53.960 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:53 smithi154 conmon[34553]: cephadm 2021-07-31T01:43:52.578492+0000 mgr.y (mgr.14146) 185 : cephadm [INF] Saving service alertmanager spec with placement smithi045=a;count:1 2021-07-31T01:43:53.960 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:53 smithi154 conmon[34553]: audit 2021-07-31T01:43:52.582230+0000 mon.a (mon.0) 551 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:43:53.979 DEBUG:teuthology.orchestra.run.smithi045:alertmanager.a> sudo journalctl -f -n 0 -u ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c@alertmanager.a.service 2021-07-31T01:43:53.982 INFO:tasks.cephadm:Adding grafana.a on smithi154 2021-07-31T01:43:53.983 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f52d2c14-f19f-11eb-8c24-001a4aab830c -- ceph orch apply grafana '1;smithi154=a' 2021-07-31T01:43:54.007 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:53 smithi045 conmon[33655]: audit 2021-07-31T01:43:52.577739+0000 mgr.y 2021-07-31T01:43:54.007 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:53 smithi045 conmon[33655]: (mgr.14146) 184 : audit [DBG] from='client.24323 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi045=a", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:43:54.007 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:53 smithi045 conmon[33655]: cephadm 2021-07-31T01:43:52.578492+0000 mgr.y (mgr.14146) 185 : cephadm [INF] Saving service alertmanager spec with placement smithi045=a;count:1 2021-07-31T01:43:54.008 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:53 smithi045 conmon[33655]: audit 2021-07-31T01:43:52.582230+0000 2021-07-31T01:43:54.008 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:53 smithi045 conmon[33655]: mon.a (mon.0) 551 : audit [INF] 2021-07-31T01:43:54.008 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:53 smithi045 conmon[33655]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:43:54.009 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:53 smithi045 conmon[39718]: audit 2021-07-31T01:43:52 2021-07-31T01:43:54.009 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:53 smithi045 conmon[39718]: .577739+0000 mgr.y (mgr.14146) 184 : audit [DBG] from='client.24323 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi045=a", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:43:54.009 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:53 smithi045 conmon[39718]: cephadm 2021-07-31T01:43:54.010 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:53 smithi045 conmon[39718]: 2021-07-31T01: 2021-07-31T01:43:54.010 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:53 smithi045 conmon[39718]: 43:52.578492+0000 mgr.y 2021-07-31T01:43:54.010 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:53 smithi045 conmon[39718]: (mgr.14146) 185 : cephadm 2021-07-31T01:43:54.010 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:53 smithi045 conmon[39718]: [INF] Saving service alertmanager spec with placement smithi045=a;count:1 2021-07-31T01:43:54.011 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:53 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T01:43:54.011 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:53 smithi045 conmon[39718]: 31T01:43:52.582230+0000 mon.a (mon.0) 551 : audit [INF] 2021-07-31T01:43:54.011 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:53 smithi045 conmon[39718]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:43:54.013 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:-- Logs begin at Sat 2021-07-31 01:16:15 UTC. -- 2021-07-31T01:43:55.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:54 smithi154 conmon[34553]: cluster 2021-07-31T01:43:53.337915+0000 mgr.y ( 2021-07-31T01:43:55.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:54 smithi154 conmon[34553]: mgr.14146) 186 : cluster [DBG] pgmap v166: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:43:55.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:54 smithi045 conmon[33655]: cluster 2021-07-31T01:43:53.337915+0000 mgr.y (mgr.14146) 186 : cluster [DBG] 2021-07-31T01:43:55.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:54 smithi045 conmon[33655]: pgmap v166: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:43:55.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:54 smithi045 conmon[39718]: cluster 2021-07-31T01:43:53.337915+0000 mgr.y ( 2021-07-31T01:43:55.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:54 smithi045 conmon[39718]: mgr.14146) 186 : cluster [DBG] pgmap v166: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:43:56.907 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:56 smithi154 conmon[34553]: cluster 2021-07-31T01:43:55.338522+0000 mgr.y (mgr.14146) 187 : cluster 2021-07-31T01:43:56.907 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:56 smithi154 conmon[34553]: [DBG] pgmap v167: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:43:57.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:56 smithi045 conmon[33655]: cluster 2021-07-31T01:43:55.338522+0000 mgr.y (mgr.14146) 2021-07-31T01:43:57.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:56 smithi045 conmon[33655]: 187 : cluster [DBG] pgmap v167: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:43:57.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:56 smithi045 conmon[39718]: cluster 2021-07-31T01:43:55 2021-07-31T01:43:57.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:56 smithi045 conmon[39718]: .338522+0000 mgr.y (mgr.14146) 187 : cluster [DBG] pgmap v167: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:43:58.798 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:58 smithi154 conmon[34553]: cluster 2021-07-31T01:43:57.338982+0000 mgr.y (mgr.14146) 188 : 2021-07-31T01:43:58.798 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:43:58 smithi154 conmon[34553]: cluster [DBG] pgmap v168: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:43:59.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:58 smithi045 conmon[33655]: cluster 2021-07-31T01 2021-07-31T01:43:59.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:43:58 smithi045 conmon[33655]: :43:57.338982+0000 mgr.y (mgr.14146) 188 : cluster [DBG] pgmap v168: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:43:59.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:58 smithi045 conmon[39718]: cluster 2021-07-31T01:43:57.338982+0000 mgr.y 2021-07-31T01:43:59.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:43:58 smithi045 conmon[39718]: (mgr.14146) 188 : cluster [DBG] pgmap v168: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:43:59.911 INFO:teuthology.orchestra.run.smithi154.stdout:Scheduled grafana update... 2021-07-31T01:44:00.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:00 smithi154 conmon[34553]: cluster 2021-07-31T01:43:59.339408+0000 mgr.y (mgr.14146) 189 : cluster [DBG] pgmap v169: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:44:00.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:00 smithi154 conmon[34553]: audit 2021-07-31T01:43:59.431856+0000 mon.a (mon.0) 552 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:44:00.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:00 smithi154 conmon[34553]: audit 2021-07-31T01:43:59.905579+0000 mgr.y (mgr.14146) 190 : audit [DBG] from='client.24329 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi154=a", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:44:00.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:00 smithi154 conmon[34553]: cephadm 2021-07-31T01:43:59.906384+0000 mgr.y (mgr.14146) 191 : cephadm [INF] Saving service grafana spec with placement smithi154=a;count:1 2021-07-31T01:44:00.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:00 smithi154 conmon[34553]: audit 2021-07-31T01:43:59.910206+0000 mon.a (mon.0) 553 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:44:00.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:00 smithi045 conmon[33655]: cluster 2021-07-31T01:43:59.339408+0000 2021-07-31T01:44:00.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:00 smithi045 conmon[33655]: mgr.y (mgr.14146) 189 : cluster [DBG] pgmap v169: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:44:00.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:00 smithi045 conmon[33655]: audit 2021-07-31T01:43:59.431856+0000 mon.a ( 2021-07-31T01:44:00.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:00 smithi045 conmon[33655]: mon.0) 552 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:44:00.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:00 smithi045 conmon[33655]: audit 2021-07-31T01 2021-07-31T01:44:00.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:00 smithi045 conmon[33655]: :43:59.905579+0000 mgr.y (mgr. 2021-07-31T01:44:00.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:00 smithi045 conmon[33655]: 14146) 190 : audit [DBG] from='client.24329 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi154=a", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:44:00.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:00 smithi045 conmon[33655]: cephadm 2021- 2021-07-31T01:44:00.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:00 smithi045 conmon[33655]: 07-31T01:43:59.906384+0000 2021-07-31T01:44:00.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:00 smithi045 conmon[33655]: mgr.y (mgr.14146) 191 : cephadm 2021-07-31T01:44:00.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:00 smithi045 conmon[33655]: [INF] Saving service grafana spec with placement smithi154=a;count:1 2021-07-31T01:44:00.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:00 smithi045 conmon[33655]: audit 2021-07-31 2021-07-31T01:44:00.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:00 smithi045 conmon[33655]: T01:43:59.910206+0000 mon.a ( 2021-07-31T01:44:00.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:00 smithi045 conmon[33655]: mon.0) 553 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:44:00.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:00 smithi045 conmon[33655]: 2021-07-31T01:44:00.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:00 smithi045 conmon[39718]: cluster 2021-07-31T01:43:59.339408+0000 mgr.y ( 2021-07-31T01:44:00.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:00 smithi045 conmon[39718]: mgr.14146) 189 : cluster [DBG] pgmap v169: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:44:00.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:00 smithi045 conmon[39718]: audit 2021-07-31T01:43 2021-07-31T01:44:00.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:00 smithi045 conmon[39718]: :59.431856+0000 mon.a (mon.0) 552 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:44:00.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:00 smithi045 conmon[39718]: audit 2021-07-31T01:44:00.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:00 smithi045 conmon[39718]: 2021-07-31T01:43: 2021-07-31T01:44:00.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:00 smithi045 conmon[39718]: 59.905579+0000 mgr.y (mgr.14146) 190 : audit 2021-07-31T01:44:00.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:00 smithi045 conmon[39718]: [DBG] from='client.24329 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi154=a", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:44:00.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:00 smithi045 conmon[39718]: cephadm 2021-07-31T 2021-07-31T01:44:00.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:00 smithi045 conmon[39718]: 01:43:59.906384+0000 mgr.y (mgr.14146) 191 2021-07-31T01:44:00.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:00 smithi045 conmon[39718]: : cephadm [INF] Saving service grafana spec with placement smithi154=a;count:1 2021-07-31T01:44:00.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:00 smithi045 conmon[39718]: audit 2021- 2021-07-31T01:44:00.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:00 smithi045 conmon[39718]: 07-31T01:43:59.910206+0000 mon.a (mon 2021-07-31T01:44:00.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:00 smithi045 conmon[39718]: .0) 553 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:44:01.129 DEBUG:teuthology.orchestra.run.smithi154:grafana.a> sudo journalctl -f -n 0 -u ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c@grafana.a.service 2021-07-31T01:44:01.133 INFO:tasks.cephadm:Setting up client nodes... 2021-07-31T01:44:01.133 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f52d2c14-f19f-11eb-8c24-001a4aab830c -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2021-07-31T01:44:01.245 INFO:journalctl@ceph.grafana.a.smithi154.stdout:-- Logs begin at Sat 2021-07-31 01:17:02 UTC. -- 2021-07-31T01:44:01.491 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Jul 31 01:44:01 smithi154 systemd[1]: Starting Ceph prometheus.a for f52d2c14-f19f-11eb-8c24-001a4aab830c... 2021-07-31T01:44:01.491 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Jul 31 01:44:01 smithi154 podman[52044]: 2021-07-31 01:44:01.377123013 +0000 UTC m=+0.029805308 image pull docker.io/prom/prometheus:v2.18.1 2021-07-31T01:44:01.491 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Jul 31 01:44:01 smithi154 podman[52044]: 2021-07-31 01:44:01.489794519 +0000 UTC m=+0.142476887 container create 5db25a29728b6bd32fed7793d92a2ae6f1f8bdc8a0be6efb5a96b8f730e4787d (image=docker.io/prom/prometheus:v2.18.1, name=ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c-prometheus.a, maintainer=The Prometheus Authors ) 2021-07-31T01:44:02.252 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Jul 31 01:44:01 smithi154 podman[52044]: 2021-07-31 01:44:01.806331889 +0000 UTC m=+0.459014159 container init 5db25a29728b6bd32fed7793d92a2ae6f1f8bdc8a0be6efb5a96b8f730e4787d (image=docker.io/prom/prometheus:v2.18.1, name=ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c-prometheus.a, maintainer=The Prometheus Authors ) 2021-07-31T01:44:02.253 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Jul 31 01:44:01 smithi154 podman[52044]: 2021-07-31 01:44:01.848151182 +0000 UTC m=+0.500833458 container start 5db25a29728b6bd32fed7793d92a2ae6f1f8bdc8a0be6efb5a96b8f730e4787d (image=docker.io/prom/prometheus:v2.18.1, name=ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c-prometheus.a, maintainer=The Prometheus Authors ) 2021-07-31T01:44:02.253 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Jul 31 01:44:01 smithi154 bash[51970]: 5db25a29728b6bd32fed7793d92a2ae6f1f8bdc8a0be6efb5a96b8f730e4787d 2021-07-31T01:44:02.253 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Jul 31 01:44:01 smithi154 conmon[52084]: level=info ts=2021-07-31T01:44:01.848Z caller=main.go:302 msg="No time or size retention was set so using the default time retention" duration=15d 2021-07-31T01:44:02.253 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Jul 31 01:44:01 smithi154 conmon[52084]: level=info ts=2021-07-31T01:44:01.849Z caller=main.go:337 msg="Starting Prometheus" version="(version=2.18.1, branch=HEAD, revision=ecee9c8abfd118f139014cb1b174b08db3f342cf)" 2021-07-31T01:44:02.254 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Jul 31 01:44:01 smithi154 conmon[52084]: level=info ts=2021-07-31T01:44:01.849Z caller=main.go:338 build_context="(go=go1.14.2, user=root@2117a9e64a7e, date=20200507-16:51:47)" 2021-07-31T01:44:02.254 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Jul 31 01:44:01 smithi154 conmon[52084]: level=info ts=2021-07-31T01:44:01.849Z caller=main.go:339 host_details="(Linux 4.18.0-240.1.1.el8_3.x86_64 #1 SMP Fri Oct 16 13:36:46 EDT 2020 x86_64 smithi154 (none))" 2021-07-31T01:44:02.254 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Jul 31 01:44:01 smithi154 conmon[52084]: level=info ts=2021-07-31T01:44:01.849Z caller=main.go:340 fd_limits="(soft=1048576, hard=1048576)" 2021-07-31T01:44:02.254 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Jul 31 01:44:01 smithi154 conmon[52084]: level=info ts=2021-07-31T01:44:01.849Z caller=main.go:341 vm_limits="(soft=unlimited, hard=unlimited)" 2021-07-31T01:44:02.255 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Jul 31 01:44:01 smithi154 conmon[52084]: level=info ts=2021-07-31T01:44:01.850Z caller=main.go:678 msg="Starting TSDB ..." 2021-07-31T01:44:02.255 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Jul 31 01:44:01 smithi154 conmon[52084]: level=info ts=2021-07-31T01:44:01.850Z caller=web.go:523 component=web msg="Start listening for connections" address=:9095 2021-07-31T01:44:02.255 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Jul 31 01:44:01 smithi154 conmon[52084]: level=info ts=2021-07-31T01:44:01.852Z caller=head.go:575 component=tsdb msg="Replaying WAL, this may take awhile" 2021-07-31T01:44:02.255 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Jul 31 01:44:01 smithi154 conmon[52084]: level=info ts=2021-07-31T01:44:01.852Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2021-07-31T01:44:02.255 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Jul 31 01:44:01 smithi154 conmon[52084]: level=info ts=2021-07-31T01:44:01.852Z caller=head.go:627 component=tsdb msg="WAL replay completed" duration=150.42µs 2021-07-31T01:44:02.256 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Jul 31 01:44:01 smithi154 conmon[52084]: level=info ts=2021-07-31T01:44:01.853Z caller=main.go:694 fs_type=XFS_SUPER_MAGIC 2021-07-31T01:44:02.256 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Jul 31 01:44:01 smithi154 conmon[52084]: level=info ts=2021-07-31T01:44:01.853Z caller=main.go:695 msg="TSDB started" 2021-07-31T01:44:02.256 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Jul 31 01:44:01 smithi154 conmon[52084]: level=info ts=2021-07-31T01:44:01.853Z caller=main.go:799 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2021-07-31T01:44:02.256 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Jul 31 01:44:01 smithi154 conmon[52084]: level=info ts=2021-07-31T01:44:01.858Z caller=main.go:827 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml 2021-07-31T01:44:02.257 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Jul 31 01:44:01 smithi154 conmon[52084]: level=info ts=2021-07-31T01:44:01.858Z caller=main.go:646 msg="Server is ready to receive web requests." 2021-07-31T01:44:02.257 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Jul 31 01:44:01 smithi154 systemd[1]: Started Ceph prometheus.a for f52d2c14-f19f-11eb-8c24-001a4aab830c. 2021-07-31T01:44:03.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:02 smithi154 conmon[34553]: cluster 2021-07-31T01:44:01.339740+0000 mgr.y (mgr.14146) 192 : cluster [DBG] pgmap v170: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:44:03.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:02 smithi154 conmon[34553]: audit 2021-07-31T01:44:01.925602+0000 mon.a (mon.0) 554 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:44:03.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:02 smithi154 conmon[34553]: audit 2021-07-31T01:44:01.926598+0000 mon.a (mon.0) 555 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:44:03.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:02 smithi154 conmon[34553]: audit 2021-07-31T01:44:01.927595+0000 mon.a (mon.0) 556 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:44:03.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:02 smithi154 conmon[34553]: audit 2021-07-31T01:44:01.928235+0000 mon.a (mon.0) 557 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:44:03.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:02 smithi045 conmon[33655]: cluster 2021-07-31T01:44:01.339740+0000 mgr.y ( 2021-07-31T01:44:03.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:02 smithi045 conmon[33655]: mgr.14146) 192 : cluster [DBG] pgmap v170: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:44:03.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:02 smithi045 conmon[33655]: audit 2021-07-31T01:44:01.925602+0000 mon.a (mon.0) 554 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:44:03.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:02 smithi045 conmon[33655]: audit 2021-07-31T01:44:01.926598+0000 mon.a (mon.0) 555 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:44:03.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:02 smithi045 conmon[33655]: audit 2021-07-31T01:44:01.927595+0000 mon.a (mon.0) 556 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:44:03.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:02 smithi045 conmon[33655]: audit 2021-07-31T01:44:01.928235+0000 mon.a (mon.0) 557 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:44:03.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:02 smithi045 conmon[39718]: cluster 2021-07-31T01:44: 2021-07-31T01:44:03.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:02 smithi045 conmon[39718]: 01.339740+0000 mgr.y (mgr.14146) 192 : cluster [DBG] pgmap v170: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:44:03.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:02 smithi045 conmon[39718]: audit 2021-07-31T01:44:01.925602+0000 mon.a (mon.0) 554 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:44:03.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:02 smithi045 conmon[39718]: audit 2021-07-31T01:44:01.926598+0000 mon.a (mon.0) 555 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:44:03.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:02 smithi045 conmon[39718]: audit 2021-07-31T01:44:01.927595+0000 mon.a (mon.0) 556 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:44:03.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:02 smithi045 conmon[39718]: audit 2021-07-31T01:44:01.928235+0000 mon.a (mon.0) 557 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:44:03.381 INFO:teuthology.orchestra.run.smithi045.stdout:[client.0] 2021-07-31T01:44:03.382 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQDjqgRhbR6LFhAAieRFEv6kg2TGIGkM1Jo/6A== 2021-07-31T01:44:03.935 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:03 smithi045 conmon[33655]: audit 2021-07-31T01:44:03.377749+0000 mon.c (mon.1) 8 : audit [INF] from='client.? 172.21.15.45:0/974378415' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2021-07-31T01:44:03.936 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:03 smithi045 conmon[39718]: audit 2021-07-31T01:44:03.377749+0000 mon.c (mon.1) 8 : audit [INF] from='client.? 172.21.15.45:0/974378415' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2021-07-31T01:44:03.936 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:03 smithi045 conmon[39718]: audit 2021-07-31T01:44:03.378058+0000 mon.a (mon.0) 558 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2021-07-31T01:44:03.937 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:03 smithi045 conmon[39718]: audit 2021-07-31T01:44:03.381268+0000 mon.a (mon.0) 559 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2021-07-31T01:44:04.168 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2021-07-31T01:44:04.169 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2021-07-31T01:44:04.170 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2021-07-31T01:44:04.197 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:03 smithi045 conmon[33655]: audit 2021-07-31T01:44:03.378058+0000 mon.a (mon.0) 558 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2021-07-31T01:44:04.197 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:03 smithi045 conmon[33655]: audit 2021-07-31T01:44:03.381268+0000 mon.a (mon.0) 559 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2021-07-31T01:44:04.213 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f52d2c14-f19f-11eb-8c24-001a4aab830c -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2021-07-31T01:44:04.232 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:03 smithi154 conmon[34553]: audit 2021-07-31T01:44:03.377749+0000 mon.c (mon.1 2021-07-31T01:44:04.232 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:03 smithi154 conmon[34553]: ) 8 : audit [INF] from='client.? 172.21.15.45:0/974378415' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2021-07-31T01:44:04.233 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:03 smithi154 conmon[34553]: audit 2021-07-31T01:44:03.378058+0000 mon.a (mon.0) 558 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2021-07-31T01:44:04.233 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:03 smithi154 conmon[34553]: audit 2021-07-31T01:44:03.381268+0000 mon.a (mon.0) 559 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2021-07-31T01:44:04.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:04 smithi154 conmon[34553]: cluster 2021-07-31T01:44:03.340066+0000 mgr.y (mgr. 2021-07-31T01:44:04.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:04 smithi154 conmon[34553]: 14146) 193 : cluster [DBG] pgmap v171: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:44:04.933 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:04 smithi154 conmon[34553]: audit 2021-07-31T01:44:04.436325+0000 mon.a (mon.0) 560 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:44:05.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:04 smithi154 conmon[34553]: audit 2021-07-31T01:44:04.533997+0000 mon.a (mon.0) 561 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:44:05.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:04 smithi045 conmon[33655]: cluster 2021-07-31T01:44:03.340066+0000 mgr.y (mgr.14146) 2021-07-31T01:44:05.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:04 smithi045 conmon[33655]: 193 : cluster [DBG] pgmap v171: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:44:05.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:04 smithi045 conmon[33655]: audit 2021-07-31T01:44:04.436325+0000 mon.a (mon.0) 560 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:44:05.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:04 smithi045 conmon[33655]: audit 2021-07-31T01:44:04.533997+0000 mon.a (mon.0) 561 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:44:05.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:04 smithi045 conmon[39718]: cluster 2021-07-31T01:44:03.340066+0000 mgr.y ( 2021-07-31T01:44:05.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:04 smithi045 conmon[39718]: mgr.14146) 193 : cluster [DBG] pgmap v171: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:44:05.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:04 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T01:44:05.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:04 smithi045 conmon[39718]: T01:44:04.436325+0000 mon.a (mon.0) 560 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:44:05.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:04 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T01:44:05.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:04 smithi045 conmon[39718]: 31T01:44:04.533997+0000 mon.a (mon.0) 561 : audit [INF] 2021-07-31T01:44:05.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:04 smithi045 conmon[39718]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:44:05.633 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Jul 31 01:44:05 smithi045 systemd[1]: Starting Ceph node-exporter.a for f52d2c14-f19f-11eb-8c24-001a4aab830c... 2021-07-31T01:44:06.026 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Jul 31 01:44:05 smithi045 bash[55861]: Trying to pull docker.io/prom/node-exporter:v0.18.1... 2021-07-31T01:44:06.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:06 smithi045 conmon[39718]: audit 2021-07-31T01:44:05.005660+0000 mon.a (mon.0) 562 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:44:06.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:06 smithi045 conmon[39718]: cephadm 2021-07-31T01:44:05.008297+0000 mgr.y (mgr.14146) 194 : cephadm [INF] Deploying daemon node-exporter.a on smithi045 2021-07-31T01:44:06.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:06 smithi045 conmon[39718]: cluster 2021-07-31T01:44:05.340438+0000 mgr.y (mgr.14146) 195 : cluster [DBG] pgmap v172: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:44:06.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:06 smithi045 conmon[33655]: audit 2021-07-31T01:44:05.005660+0000 2021-07-31T01:44:06.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:06 smithi045 conmon[33655]: mon.a (mon.0) 562 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:44:06.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:06 smithi045 conmon[33655]: cephadm 2021-07-31T01:44:05.008297+0000 mgr.y (mgr.14146) 194 : cephadm [INF] Deploying daemon node-exporter.a on smithi045 2021-07-31T01:44:06.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:06 smithi045 conmon[33655]: cluster 2021-07-31T01:44:05.340438+0000 mgr.y (mgr.14146) 195 : cluster [DBG] pgmap v172: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:44:06.100 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:06 smithi154 conmon[34553]: audit 2021-07-31T01:44:05.005660+0000 mon.a (mon. 2021-07-31T01:44:06.100 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:06 smithi154 conmon[34553]: 0) 562 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:44:06.101 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:06 smithi154 conmon[34553]: cephadm 2021-07-31T01:44:05.008297+0000 mgr.y (mgr.14146) 194 : cephadm [INF] Deploying daemon node-exporter.a on smithi045 2021-07-31T01:44:06.101 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:06 smithi154 conmon[34553]: cluster 2021-07-31T01:44:05.340438+0000 mgr.y (mgr.14146) 195 : cluster [DBG] pgmap v172: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:44:06.424 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Jul 31 01:44:06 smithi045 bash[55861]: Getting image source signatures 2021-07-31T01:44:06.424 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Jul 31 01:44:06 smithi045 bash[55861]: Copying blob sha256:3589a6efd9ce96b2f2218a80bacb84f60d5e584d024023617b0f56b87fe798c3 2021-07-31T01:44:06.424 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Jul 31 01:44:06 smithi045 bash[55861]: Copying blob sha256:49a2d53aa1af115e87f7e23a7fe4b7a3e12304d37982fb6528b035d0f68a3b56 2021-07-31T01:44:06.425 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Jul 31 01:44:06 smithi045 bash[55861]: Copying blob sha256:190160031744b0810999dd6f293478be0b41524654c70782cf09437095430f2f 2021-07-31T01:44:06.500 INFO:teuthology.orchestra.run.smithi154.stdout:[client.1] 2021-07-31T01:44:06.500 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQDmqgRhCR6MHRAAMijHuZtJkhx/Y/EBQ/EIPQ== 2021-07-31T01:44:06.776 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Jul 31 01:44:06 smithi045 bash[55861]: Copying config sha256:e5a616e4b9cf68dfcad7782b78e118be4310022e874d52da85c55923fb615f87 2021-07-31T01:44:06.776 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Jul 31 01:44:06 smithi045 bash[55861]: Writing manifest to image destination 2021-07-31T01:44:06.776 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Jul 31 01:44:06 smithi045 bash[55861]: Storing signatures 2021-07-31T01:44:07.234 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2021-07-31T01:44:07.235 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2021-07-31T01:44:07.235 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2021-07-31T01:44:07.312 INFO:tasks.cephadm:Waiting for OSDs to come up 2021-07-31T01:44:07.313 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2021-07-31T01:44:07.616 INFO:teuthology.misc.health.smithi045.stdout: 2021-07-31T01:44:07.617 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":49,"fsid":"f52d2c14-f19f-11eb-8c24-001a4aab830c","created":"2021-07-31T01:38:50.680976+0000","modified":"2021-07-31T01:43:47.991258+0000","last_up_change":"2021-07-31T01:43:46.985098+0000","last_in_change":"2021-07-31T01:43:30.970323+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-07-31T01:42:07.901706+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"20724b9a-515c-4878-9e4f-301e21a5a985","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":2841572524},{"type":"v1","addr":"172.21.15.45:6803","nonce":2841572524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":2841572524},{"type":"v1","addr":"172.21.15.45:6805","nonce":2841572524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":2841572524},{"type":"v1","addr":"172.21.15.45:6809","nonce":2841572524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":2841572524},{"type":"v1","addr":"172.21.15.45:6807","nonce":2841572524}]},"public_addr":"172.21.15.45:6803/2841572524","cluster_addr":"172.21.15.45:6805/2841572524","heartbeat_back_addr":"172.21.15.45:6809/2841572524","heartbeat_front_addr":"172.21.15.45:6807/2841572524","state":["exists","up"]},{"osd":1,"uuid":"f47f025a-6f29-417a-9bd1-9af9f91f38a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1747309179},{"type":"v1","addr":"172.21.15.45:6811","nonce":1747309179}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1747309179},{"type":"v1","addr":"172.21.15.45:6813","nonce":1747309179}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":1747309179},{"type":"v1","addr":"172.21.15.45:6817","nonce":1747309179}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1747309179},{"type":"v1","addr":"172.21.15.45:6815","nonce":1747309179}]},"public_addr":"172.21.15.45:6811/1747309179","cluster_addr":"172.21.15.45:6813/1747309179","heartbeat_back_addr":"172.21.15.45:6817/1747309179","heartbeat_front_addr":"172.21.15.45:6815/1747309179","state":["exists","up"]},{"osd":2,"uuid":"bfbe641e-b9d7-4005-8fb5-bc7d5af76388","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2518799988},{"type":"v1","addr":"172.21.15.45:6819","nonce":2518799988}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2518799988},{"type":"v1","addr":"172.21.15.45:6821","nonce":2518799988}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2518799988},{"type":"v1","addr":"172.21.15.45:6825","nonce":2518799988}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2518799988},{"type":"v1","addr":"172.21.15.45:6823","nonce":2518799988}]},"public_addr":"172.21.15.45:6819/2518799988","cluster_addr":"172.21.15.45:6821/2518799988","heartbeat_back_addr":"172.21.15.45:6825/2518799988","heartbeat_front_addr":"172.21.15.45:6823/2518799988","state":["exists","up"]},{"osd":3,"uuid":"37b826a3-b611-4001-b9a7-1966032e56d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3024698157},{"type":"v1","addr":"172.21.15.45:6827","nonce":3024698157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3024698157},{"type":"v1","addr":"172.21.15.45:6829","nonce":3024698157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3024698157},{"type":"v1","addr":"172.21.15.45:6833","nonce":3024698157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3024698157},{"type":"v1","addr":"172.21.15.45:6831","nonce":3024698157}]},"public_addr":"172.21.15.45:6827/3024698157","cluster_addr":"172.21.15.45:6829/3024698157","heartbeat_back_addr":"172.21.15.45:6833/3024698157","heartbeat_front_addr":"172.21.15.45:6831/3024698157","state":["exists","up"]},{"osd":4,"uuid":"69881d9d-3e56-4483-b76b-7dba14cfaac3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":4277144246},{"type":"v1","addr":"172.21.15.154:6801","nonce":4277144246}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":4277144246},{"type":"v1","addr":"172.21.15.154:6803","nonce":4277144246}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":4277144246},{"type":"v1","addr":"172.21.15.154:6807","nonce":4277144246}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":4277144246},{"type":"v1","addr":"172.21.15.154:6805","nonce":4277144246}]},"public_addr":"172.21.15.154:6801/4277144246","cluster_addr":"172.21.15.154:6803/4277144246","heartbeat_back_addr":"172.21.15.154:6807/4277144246","heartbeat_front_addr":"172.21.15.154:6805/4277144246","state":["exists","up"]},{"osd":5,"uuid":"7dcad412-f42a-478b-886a-8f4bd82a04ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":579203063},{"type":"v1","addr":"172.21.15.154:6809","nonce":579203063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":579203063},{"type":"v1","addr":"172.21.15.154:6811","nonce":579203063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":579203063},{"type":"v1","addr":"172.21.15.154:6815","nonce":579203063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":579203063},{"type":"v1","addr":"172.21.15.154:6813","nonce":579203063}]},"public_addr":"172.21.15.154:6809/579203063","cluster_addr":"172.21.15.154:6811/579203063","heartbeat_back_addr":"172.21.15.154:6815/579203063","heartbeat_front_addr":"172.21.15.154:6813/579203063","state":["exists","up"]},{"osd":6,"uuid":"94258018-67f7-49d7-83c3-85070ff8c2f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":3963397470},{"type":"v1","addr":"172.21.15.154:6817","nonce":3963397470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":3963397470},{"type":"v1","addr":"172.21.15.154:6819","nonce":3963397470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":3963397470},{"type":"v1","addr":"172.21.15.154:6823","nonce":3963397470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":3963397470},{"type":"v1","addr":"172.21.15.154:6821","nonce":3963397470}]},"public_addr":"172.21.15.154:6817/3963397470","cluster_addr":"172.21.15.154:6819/3963397470","heartbeat_back_addr":"172.21.15.154:6823/3963397470","heartbeat_front_addr":"172.21.15.154:6821/3963397470","state":["exists","up"]},{"osd":7,"uuid":"da82715a-b8b7-4786-bbc2-ee0cc3ac9b18","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1263188610},{"type":"v1","addr":"172.21.15.154:6825","nonce":1263188610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1263188610},{"type":"v1","addr":"172.21.15.154:6827","nonce":1263188610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1263188610},{"type":"v1","addr":"172.21.15.154:6831","nonce":1263188610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1263188610},{"type":"v1","addr":"172.21.15.154:6829","nonce":1263188610}]},"public_addr":"172.21.15.154:6825/1263188610","cluster_addr":"172.21.15.154:6827/1263188610","heartbeat_back_addr":"172.21.15.154:6831/1263188610","heartbeat_front_addr":"172.21.15.154:6829/1263188610","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-07-31T01:41:26.799531+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-07-31T01:41:46.224506+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-07-31T01:42:06.246092+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-07-31T01:42:26.612987+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-07-31T01:42:45.424194+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-07-31T01:43:06.036564+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-07-31T01:43:25.839643+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-07-31T01:43:45.387900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:0/3355281269":"2021-08-01T01:39:39.238202+0000","172.21.15.45:6801/2610418727":"2021-08-01T01:39:39.238202+0000","172.21.15.45:6800/2610418727":"2021-08-01T01:39:39.238202+0000","172.21.15.45:6800/487754334":"2021-08-01T01:39:15.200151+0000","172.21.15.45:0/536736800":"2021-08-01T01:39:39.238202+0000","172.21.15.45:0/4040877953":"2021-08-01T01:39:15.200151+0000","172.21.15.45:0/2377876452":"2021-08-01T01:39:15.200151+0000","172.21.15.45:6801/487754334":"2021-08-01T01:39:15.200151+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-07-31T01:44:07.644 DEBUG:teuthology.misc:8 of 8 OSDs are up 2021-07-31T01:44:07.644 INFO:tasks.cephadm:Creating RBD pool 2021-07-31T01:44:07.645 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f52d2c14-f19f-11eb-8c24-001a4aab830c -- sudo ceph --cluster ceph osd pool create rbd 8 2021-07-31T01:44:07.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:07 smithi154 conmon[34553]: audit 2021-07-31T01:44:06.495005+0000 mon.b 2021-07-31T01:44:07.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:07 smithi154 conmon[34553]: (mon.2) 23 : audit [INF] from='client.? 172.21.15.154:0/1814554942' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2021-07-31T01:44:07.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:07 smithi154 conmon[34553]: audit 2021-07-31T01:44:06.495551+0000 mon.a (mon.0) 563 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2021-07-31T01:44:07.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:07 smithi154 conmon[34553]: audit 2021-07-31T01:44:06.499609+0000 mon.a (mon.0) 564 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2021-07-31T01:44:07.776 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Jul 31 01:44:07 smithi045 podman[55935]: 2021-07-31 01:44:05.765578184 +0000 UTC m=+0.031198449 image pull docker.io/prom/node-exporter:v0.18.1 2021-07-31T01:44:07.776 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Jul 31 01:44:07 smithi045 podman[55935]: 2021-07-31 01:44:07.614793183 +0000 UTC m=+1.880413442 container create 71bdbcbf130babc3419e28b0b49fb68fc4006b556f5c3c238acd0625d4060892 (image=docker.io/prom/node-exporter:v0.18.1, name=ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c-node-exporter.a, maintainer=The Prometheus Authors ) 2021-07-31T01:44:07.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:07 smithi045 conmon[33655]: audit 2021-07-31T01:44:06 2021-07-31T01:44:07.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:07 smithi045 conmon[33655]: .495005+0000 mon.b (mon.2) 23 : audit [INF] from='client.? 172.21.15.154:0/1814554942' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2021-07-31T01:44:07.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:07 smithi045 conmon[33655]: audit 2021-07-31T01:44:06.495551+0000 mon.a (mon.0) 563 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2021-07-31T01:44:07.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:07 smithi045 conmon[33655]: audit 2021-07-31T01:44:06 2021-07-31T01:44:07.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:07 smithi045 conmon[33655]: .499609+0000 mon.a (mon.0) 564 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2021-07-31T01:44:07.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:07 smithi045 conmon[39718]: audit 2021-07-31T01:44:07.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:07 smithi045 conmon[39718]: 2021-07-31T01 2021-07-31T01:44:07.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:07 smithi045 conmon[39718]: :44:06.495005+0000 mon.b (mon.2) 23 : audit [INF] from='client.? 172.21.15.154:0/1814554942' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2021-07-31T01:44:07.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:07 smithi045 conmon[39718]: audit 2021-07-31T01:44:07.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:07 smithi045 conmon[39718]: 2021-07-31T01:44:06. 2021-07-31T01:44:07.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:07 smithi045 conmon[39718]: 495551+0000 mon.a (mon.0) 563 : audit 2021-07-31T01:44:07.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:07 smithi045 conmon[39718]: [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2021-07-31T01:44:07.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:07 smithi045 conmon[39718]: audit 2021- 2021-07-31T01:44:07.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:07 smithi045 conmon[39718]: 07-31T01:44:06.499609+0000 2021-07-31T01:44:07.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:07 smithi045 conmon[39718]: mon.a (mon.0) 564 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2021-07-31T01:44:07.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:07 smithi045 conmon[39718]: 2021-07-31T01:44:08.193 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Jul 31 01:44:07 smithi045 podman[55935]: 2021-07-31 01:44:07.914817186 +0000 UTC m=+2.180437454 container init 71bdbcbf130babc3419e28b0b49fb68fc4006b556f5c3c238acd0625d4060892 (image=docker.io/prom/node-exporter:v0.18.1, name=ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c-node-exporter.a, maintainer=The Prometheus Authors ) 2021-07-31T01:44:08.193 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Jul 31 01:44:07 smithi045 conmon[56059]: time="2021-07-31T01:44:07Z" level=info msg="Starting node_exporter (version=0.18.1, branch=HEAD, revision=3db77732e925c08f675d7404a8c46466b2ece83e)" source="node_exporter.go:156" 2021-07-31T01:44:08.194 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Jul 31 01:44:07 smithi045 conmon[56059]: time="2021-07-31T01:44:07Z" level=info msg="Build context (go=go1.12.5, user=root@b50852a1acba, date=20190604-16:41:18)" source="node_exporter.go:157" 2021-07-31T01:44:08.194 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Jul 31 01:44:07 smithi045 conmon[56059]: time="2021-07-31T01:44:07Z" level=info msg="Enabled collectors:" source="node_exporter.go:97" 2021-07-31T01:44:08.194 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Jul 31 01:44:07 smithi045 conmon[56059]: time="2021-07-31T01:44:07Z" level=info msg=" - arp" source="node_exporter.go:104" 2021-07-31T01:44:08.195 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Jul 31 01:44:07 smithi045 conmon[56059]: time="2021-07-31T01:44:07Z" level=info msg=" - bcache" source="node_exporter.go:104" 2021-07-31T01:44:08.195 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Jul 31 01:44:07 smithi045 conmon[56059]: time="2021-07-31T01:44:07Z" level=info msg=" - bonding" source="node_exporter.go:104" 2021-07-31T01:44:08.196 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Jul 31 01:44:07 smithi045 conmon[56059]: time="2021-07-31T01:44:07Z" level=info msg=" - conntrack" source="node_exporter.go:104" 2021-07-31T01:44:08.196 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Jul 31 01:44:07 smithi045 conmon[56059]: time="2021-07-31T01:44:07Z" level=info msg=" - cpu" source="node_exporter.go:104" 2021-07-31T01:44:08.196 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Jul 31 01:44:07 smithi045 conmon[56059]: time="2021-07-31T01:44:07Z" level=info msg=" - cpufreq" source="node_exporter.go:104" 2021-07-31T01:44:08.196 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Jul 31 01:44:07 smithi045 conmon[56059]: time="2021-07-31T01:44:07Z" level=info msg=" - diskstats" source="node_exporter.go:104" 2021-07-31T01:44:08.197 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Jul 31 01:44:07 smithi045 conmon[56059]: time="2021-07-31T01:44:07Z" level=info msg=" - edac" source="node_exporter.go:104" 2021-07-31T01:44:08.197 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Jul 31 01:44:07 smithi045 conmon[56059]: time="2021-07-31T01:44:07Z" level=info msg=" - entropy" source="node_exporter.go:104" 2021-07-31T01:44:08.197 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Jul 31 01:44:07 smithi045 conmon[56059]: time="2021-07-31T01:44:07Z" level=info msg=" - filefd" source="node_exporter.go:104" 2021-07-31T01:44:08.197 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Jul 31 01:44:07 smithi045 conmon[56059]: time="2021-07-31T01:44:07Z" level=info msg=" - filesystem" source="node_exporter.go:104" 2021-07-31T01:44:08.198 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Jul 31 01:44:07 smithi045 conmon[56059]: time="2021-07-31T01:44:07Z" level=info msg=" - hwmon" source="node_exporter.go:104" 2021-07-31T01:44:08.198 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Jul 31 01:44:07 smithi045 conmon[56059]: time="2021-07-31T01:44:07Z" level=info msg=" - infiniband" source="node_exporter.go:104" 2021-07-31T01:44:08.198 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Jul 31 01:44:07 smithi045 conmon[56059]: time="2021-07-31T01:44:07Z" level=info msg=" - ipvs" source="node_exporter.go:104" 2021-07-31T01:44:08.198 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Jul 31 01:44:07 smithi045 conmon[56059]: time="2021-07-31T01:44:07Z" level=info msg=" - loadavg" source="node_exporter.go:104" 2021-07-31T01:44:08.199 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Jul 31 01:44:07 smithi045 conmon[56059]: time="2021-07-31T01:44:07Z" level=info msg=" - mdadm" source="node_exporter.go:104" 2021-07-31T01:44:08.199 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Jul 31 01:44:07 smithi045 conmon[56059]: time="2021-07-31T01:44:07Z" level=info msg=" - meminfo" source="node_exporter.go:104" 2021-07-31T01:44:08.199 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Jul 31 01:44:07 smithi045 conmon[56059]: time="2021-07-31T01:44:07Z" level=info msg=" - netclass" source="node_exporter.go:104" 2021-07-31T01:44:08.199 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Jul 31 01:44:07 smithi045 conmon[56059]: time="2021-07-31T01:44:07Z" level=info msg=" - netdev" source="node_exporter.go:104" 2021-07-31T01:44:08.199 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Jul 31 01:44:07 smithi045 conmon[56059]: time="2021-07-31T01:44:07Z" level=info msg=" - netstat" source="node_exporter.go:104" 2021-07-31T01:44:08.200 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Jul 31 01:44:07 smithi045 conmon[56059]: time="2021-07-31T01:44:07Z" level=info msg=" - nfs" source="node_exporter.go:104" 2021-07-31T01:44:08.200 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Jul 31 01:44:07 smithi045 conmon[56059]: time="2021-07-31T01:44:07Z" level=info msg=" - nfsd" source="node_exporter.go:104" 2021-07-31T01:44:08.200 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Jul 31 01:44:07 smithi045 conmon[56059]: time="2021-07-31T01:44:07Z" level=info msg=" - pressure" source="node_exporter.go:104" 2021-07-31T01:44:08.201 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Jul 31 01:44:07 smithi045 conmon[56059]: time="2021-07-31T01:44:07Z" level=info msg=" - sockstat" source="node_exporter.go:104" 2021-07-31T01:44:08.201 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Jul 31 01:44:07 smithi045 conmon[56059]: time="2021-07-31T01:44:07Z" level=info msg=" - stat" source="node_exporter.go:104" 2021-07-31T01:44:08.201 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Jul 31 01:44:07 smithi045 conmon[56059]: time="2021-07-31T01:44:07Z" level=info msg=" - textfile" source="node_exporter.go:104" 2021-07-31T01:44:08.201 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Jul 31 01:44:07 smithi045 conmon[56059]: time="2021-07-31T01:44:07Z" level=info msg=" - time" source="node_exporter.go:104" 2021-07-31T01:44:08.202 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Jul 31 01:44:07 smithi045 conmon[56059]: time="2021-07-31T01:44:07Z" level=info msg=" - uname" source="node_exporter.go:104" 2021-07-31T01:44:08.202 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Jul 31 01:44:07 smithi045 conmon[56059]: time="2021-07-31T01:44:07Z" level=info msg=" - vmstat" source="node_exporter.go:104" 2021-07-31T01:44:08.203 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Jul 31 01:44:07 smithi045 conmon[56059]: time="2021-07-31T01:44:07Z" level=info msg=" - xfs" source="node_exporter.go:104" 2021-07-31T01:44:08.203 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Jul 31 01:44:07 smithi045 conmon[56059]: time="2021-07-31T01:44:07Z" level=info msg=" - zfs" source="node_exporter.go:104" 2021-07-31T01:44:08.203 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Jul 31 01:44:07 smithi045 conmon[56059]: time="2021-07-31T01:44:07Z" level=info msg="Listening on :9100" source="node_exporter.go:170" 2021-07-31T01:44:08.203 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Jul 31 01:44:07 smithi045 podman[55935]: 2021-07-31 01:44:07.948143517 +0000 UTC m=+2.213763897 container start 71bdbcbf130babc3419e28b0b49fb68fc4006b556f5c3c238acd0625d4060892 (image=docker.io/prom/node-exporter:v0.18.1, name=ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c-node-exporter.a, maintainer=The Prometheus Authors ) 2021-07-31T01:44:08.204 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Jul 31 01:44:07 smithi045 bash[55861]: 71bdbcbf130babc3419e28b0b49fb68fc4006b556f5c3c238acd0625d4060892 2021-07-31T01:44:08.204 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Jul 31 01:44:07 smithi045 systemd[1]: Started Ceph node-exporter.a for f52d2c14-f19f-11eb-8c24-001a4aab830c. 2021-07-31T01:44:08.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:08 smithi154 conmon[34553]: cluster 2021-07-31T01:44:07.340743+0000 mgr.y ( 2021-07-31T01:44:08.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:08 smithi154 conmon[34553]: mgr.14146) 196 : cluster [DBG] pgmap v173: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:44:08.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:08 smithi154 conmon[34553]: audit 2021-07-31T01:44:07.615729+0000 mon.c (mon.1) 9 : audit [DBG] from='client.? 172.21.15.45:0/3960070804' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-07-31T01:44:08.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:08 smithi154 conmon[34553]: audit 2021-07-31T01:44:08.029212+0000 mon.a (mon.0) 565 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:44:08.752 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Jul 31 01:44:08 smithi154 systemd[1]: Starting Ceph node-exporter.b for f52d2c14-f19f-11eb-8c24-001a4aab830c... 2021-07-31T01:44:08.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:08 smithi045 conmon[33655]: cluster 2021-07-31T 2021-07-31T01:44:08.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:08 smithi045 conmon[33655]: 01:44:07.340743+0000 mgr.y (mgr.14146) 196 : cluster [DBG] pgmap v173: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:44:08.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:08 smithi045 conmon[33655]: audit 2021-07-31T01:44:07.615729+0000 mon.c (mon.1) 9 : audit [DBG] from='client.? 172.21.15.45:0/3960070804' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-07-31T01:44:08.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:08 smithi045 conmon[33655]: audit 2021-07-31T01:44:08.029212+0000 mon.a (mon.0) 565 : audit [INF] 2021-07-31T01:44:08.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:08 smithi045 conmon[33655]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:44:08.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:08 smithi045 conmon[39718]: cluster 2021-07-31T01:44:07.340743+0000 2021-07-31T01:44:08.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:08 smithi045 conmon[39718]: mgr.y (mgr.14146) 196 : cluster [DBG] pgmap v173: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:44:08.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:08 smithi045 conmon[39718]: audit 2021-07-31T01:44:07.615729+0000 mon.c (mon.1) 9 : audit [DBG] from='client.? 172.21.15.45:0/3960070804' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-07-31T01:44:08.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:08 smithi045 conmon[39718]: audit 2021-07-31T01:44:08.029212+0000 mon.a (mon.0) 2021-07-31T01:44:08.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:08 smithi045 conmon[39718]: 565 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:44:09.002 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Jul 31 01:44:08 smithi154 bash[53127]: Trying to pull docker.io/prom/node-exporter:v0.18.1... 2021-07-31T01:44:09.003 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Jul 31 01:44:08 smithi154 bash[53127]: Getting image source signatures 2021-07-31T01:44:09.446 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Jul 31 01:44:09 smithi154 bash[53127]: Copying blob sha256:190160031744b0810999dd6f293478be0b41524654c70782cf09437095430f2f 2021-07-31T01:44:09.446 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Jul 31 01:44:09 smithi154 bash[53127]: Copying blob sha256:3589a6efd9ce96b2f2218a80bacb84f60d5e584d024023617b0f56b87fe798c3 2021-07-31T01:44:09.447 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Jul 31 01:44:09 smithi154 bash[53127]: Copying blob sha256:49a2d53aa1af115e87f7e23a7fe4b7a3e12304d37982fb6528b035d0f68a3b56 2021-07-31T01:44:09.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:09 smithi154 conmon[34553]: cephadm 2021-07-31T01:44:08.030040+0000 mgr.y (mgr.14146) 197 : cephadm [INF] Deploying daemon node-exporter.b on smithi154 2021-07-31T01:44:09.753 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Jul 31 01:44:09 smithi154 bash[53127]: Copying config sha256:e5a616e4b9cf68dfcad7782b78e118be4310022e874d52da85c55923fb615f87 2021-07-31T01:44:09.753 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Jul 31 01:44:09 smithi154 bash[53127]: Writing manifest to image destination 2021-07-31T01:44:09.753 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Jul 31 01:44:09 smithi154 bash[53127]: Storing signatures 2021-07-31T01:44:09.761 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:09 smithi045 conmon[33655]: cephadm 2021-07-31T01:44:08.030040+0000 mgr.y (mgr.14146) 197 : cephadm [INF] Deploying daemon node-exporter.b on smithi154 2021-07-31T01:44:09.761 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:09 smithi045 conmon[39718]: cephadm 2021-07-31T01:44:08.030040+0000 mgr.y (mgr.14146 2021-07-31T01:44:09.761 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:09 smithi045 conmon[39718]: ) 197 : cephadm [INF] Deploying daemon node-exporter.b on smithi154 2021-07-31T01:44:10.743 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'rbd' created 2021-07-31T01:44:10.949 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:10 smithi045 conmon[39718]: cluster 2021-07-31T01:44:09. 2021-07-31T01:44:10.950 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:10 smithi045 conmon[39718]: 341324+0000 mgr.y (mgr.14146) 198 : cluster [DBG] pgmap v174: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:44:10.950 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:10 smithi045 conmon[33655]: cluster 2021-07-31T01:44:09.341324+0000 2021-07-31T01:44:10.951 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:10 smithi045 conmon[33655]: mgr.y (mgr.14146) 198 : cluster [DBG] pgmap v174: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:44:11.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:10 smithi154 conmon[34553]: cluster 2021-07-31T01:44:09.341324+0000 mgr.y (mgr.14146) 2021-07-31T01:44:11.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:10 smithi154 conmon[34553]: 198 : cluster [DBG] pgmap v174: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:44:11.003 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Jul 31 01:44:10 smithi154 podman[53203]: 2021-07-31 01:44:08.749221612 +0000 UTC m=+0.031004242 image pull docker.io/prom/node-exporter:v0.18.1 2021-07-31T01:44:11.004 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Jul 31 01:44:10 smithi154 podman[53203]: 2021-07-31 01:44:10.73806062 +0000 UTC m=+2.019843252 container create f373a51f817a6bc539d04520e0e1e4e7941897ebcc87f2824dbc0d43ba6bb43b (image=docker.io/prom/node-exporter:v0.18.1, name=ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c-node-exporter.b, maintainer=The Prometheus Authors ) 2021-07-31T01:44:11.487 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f52d2c14-f19f-11eb-8c24-001a4aab830c -- sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2021-07-31T01:44:11.503 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Jul 31 01:44:11 smithi154 podman[53203]: 2021-07-31 01:44:11.054701539 +0000 UTC m=+2.336484140 container init f373a51f817a6bc539d04520e0e1e4e7941897ebcc87f2824dbc0d43ba6bb43b (image=docker.io/prom/node-exporter:v0.18.1, name=ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c-node-exporter.b, maintainer=The Prometheus Authors ) 2021-07-31T01:44:11.503 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Jul 31 01:44:11 smithi154 conmon[53276]: time="2021-07-31T01:44:11Z" level=info msg="Starting node_exporter (version=0.18.1, branch=HEAD, revision=3db77732e925c08f675d7404a8c46466b2ece83e)" source="node_exporter.go:156" 2021-07-31T01:44:11.504 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Jul 31 01:44:11 smithi154 conmon[53276]: time="2021-07-31T01:44:11Z" level=info msg="Build context (go=go1.12.5, user=root@b50852a1acba, date=20190604-16:41:18)" source="node_exporter.go:157" 2021-07-31T01:44:11.505 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Jul 31 01:44:11 smithi154 conmon[53276]: time="2021-07-31T01:44:11Z" level=info msg="Enabled collectors:" source="node_exporter.go:97" 2021-07-31T01:44:11.506 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Jul 31 01:44:11 smithi154 conmon[53276]: time="2021-07-31T01:44:11Z" level=info msg=" - arp" source="node_exporter.go:104" 2021-07-31T01:44:11.506 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Jul 31 01:44:11 smithi154 conmon[53276]: time="2021-07-31T01:44:11Z" level=info msg=" - bcache" source="node_exporter.go:104" 2021-07-31T01:44:11.506 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Jul 31 01:44:11 smithi154 conmon[53276]: time="2021-07-31T01:44:11Z" level=info msg=" - bonding" source="node_exporter.go:104" 2021-07-31T01:44:11.506 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Jul 31 01:44:11 smithi154 conmon[53276]: time="2021-07-31T01:44:11Z" level=info msg=" - conntrack" source="node_exporter.go:104" 2021-07-31T01:44:11.507 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Jul 31 01:44:11 smithi154 conmon[53276]: time="2021-07-31T01:44:11Z" level=info msg=" - cpu" source="node_exporter.go:104" 2021-07-31T01:44:11.507 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Jul 31 01:44:11 smithi154 conmon[53276]: time="2021-07-31T01:44:11Z" level=info msg=" - cpufreq" source="node_exporter.go:104" 2021-07-31T01:44:11.507 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Jul 31 01:44:11 smithi154 conmon[53276]: time="2021-07-31T01:44:11Z" level=info msg=" - diskstats" source="node_exporter.go:104" 2021-07-31T01:44:11.508 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Jul 31 01:44:11 smithi154 conmon[53276]: time="2021-07-31T01:44:11Z" level=info msg=" - edac" source="node_exporter.go:104" 2021-07-31T01:44:11.508 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Jul 31 01:44:11 smithi154 conmon[53276]: time="2021-07-31T01:44:11Z" level=info msg=" - entropy" source="node_exporter.go:104" 2021-07-31T01:44:11.508 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Jul 31 01:44:11 smithi154 conmon[53276]: time="2021-07-31T01:44:11Z" level=info msg=" - filefd" source="node_exporter.go:104" 2021-07-31T01:44:11.508 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Jul 31 01:44:11 smithi154 conmon[53276]: time="2021-07-31T01:44:11Z" level=info msg=" - filesystem" source="node_exporter.go:104" 2021-07-31T01:44:11.509 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Jul 31 01:44:11 smithi154 conmon[53276]: time="2021-07-31T01:44:11Z" level=info msg=" - hwmon" source="node_exporter.go:104" 2021-07-31T01:44:11.509 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Jul 31 01:44:11 smithi154 conmon[53276]: time="2021-07-31T01:44:11Z" level=info msg=" - infiniband" source="node_exporter.go:104" 2021-07-31T01:44:11.509 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Jul 31 01:44:11 smithi154 conmon[53276]: time="2021-07-31T01:44:11Z" level=info msg=" - ipvs" source="node_exporter.go:104" 2021-07-31T01:44:11.509 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Jul 31 01:44:11 smithi154 conmon[53276]: time="2021-07-31T01:44:11Z" level=info msg=" - loadavg" source="node_exporter.go:104" 2021-07-31T01:44:11.510 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Jul 31 01:44:11 smithi154 conmon[53276]: time="2021-07-31T01:44:11Z" level=info msg=" - mdadm" source="node_exporter.go:104" 2021-07-31T01:44:11.510 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Jul 31 01:44:11 smithi154 conmon[53276]: time="2021-07-31T01:44:11Z" level=info msg=" - meminfo" source="node_exporter.go:104" 2021-07-31T01:44:11.510 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Jul 31 01:44:11 smithi154 conmon[53276]: time="2021-07-31T01:44:11Z" level=info msg=" - netclass" source="node_exporter.go:104" 2021-07-31T01:44:11.512 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Jul 31 01:44:11 smithi154 conmon[53276]: time="2021-07-31T01:44:11Z" level=info msg=" - netdev" source="node_exporter.go:104" 2021-07-31T01:44:11.512 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Jul 31 01:44:11 smithi154 conmon[53276]: time="2021-07-31T01:44:11Z" level=info msg=" - netstat" source="node_exporter.go:104" 2021-07-31T01:44:11.512 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Jul 31 01:44:11 smithi154 conmon[53276]: time="2021-07-31T01:44:11Z" level=info msg=" - nfs" source="node_exporter.go:104" 2021-07-31T01:44:11.512 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Jul 31 01:44:11 smithi154 conmon[53276]: time="2021-07-31T01:44:11Z" level=info msg=" - nfsd" source="node_exporter.go:104" 2021-07-31T01:44:11.512 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Jul 31 01:44:11 smithi154 conmon[53276]: time="2021-07-31T01:44:11Z" level=info msg=" - pressure" source="node_exporter.go:104" 2021-07-31T01:44:11.513 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Jul 31 01:44:11 smithi154 conmon[53276]: time="2021-07-31T01:44:11Z" level=info msg=" - sockstat" source="node_exporter.go:104" 2021-07-31T01:44:11.513 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Jul 31 01:44:11 smithi154 conmon[53276]: time="2021-07-31T01:44:11Z" level=info msg=" - stat" source="node_exporter.go:104" 2021-07-31T01:44:11.513 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Jul 31 01:44:11 smithi154 conmon[53276]: time="2021-07-31T01:44:11Z" level=info msg=" - textfile" source="node_exporter.go:104" 2021-07-31T01:44:11.513 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Jul 31 01:44:11 smithi154 conmon[53276]: time="2021-07-31T01:44:11Z" level=info msg=" - time" source="node_exporter.go:104" 2021-07-31T01:44:11.514 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Jul 31 01:44:11 smithi154 conmon[53276]: time="2021-07-31T01:44:11Z" level=info msg=" - uname" source="node_exporter.go:104" 2021-07-31T01:44:11.514 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Jul 31 01:44:11 smithi154 conmon[53276]: time="2021-07-31T01:44:11Z" level=info msg=" - vmstat" source="node_exporter.go:104" 2021-07-31T01:44:11.514 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Jul 31 01:44:11 smithi154 conmon[53276]: time="2021-07-31T01:44:11Z" level=info msg=" - xfs" source="node_exporter.go:104" 2021-07-31T01:44:11.514 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Jul 31 01:44:11 smithi154 conmon[53276]: time="2021-07-31T01:44:11Z" level=info msg=" - zfs" source="node_exporter.go:104" 2021-07-31T01:44:11.514 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Jul 31 01:44:11 smithi154 conmon[53276]: time="2021-07-31T01:44:11Z" level=info msg="Listening on :9100" source="node_exporter.go:170" 2021-07-31T01:44:11.515 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Jul 31 01:44:11 smithi154 podman[53203]: 2021-07-31 01:44:11.088079837 +0000 UTC m=+2.369862451 container start f373a51f817a6bc539d04520e0e1e4e7941897ebcc87f2824dbc0d43ba6bb43b (image=docker.io/prom/node-exporter:v0.18.1, name=ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c-node-exporter.b, maintainer=The Prometheus Authors ) 2021-07-31T01:44:11.515 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Jul 31 01:44:11 smithi154 bash[53127]: f373a51f817a6bc539d04520e0e1e4e7941897ebcc87f2824dbc0d43ba6bb43b 2021-07-31T01:44:11.516 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Jul 31 01:44:11 smithi154 systemd[1]: Started Ceph node-exporter.b for f52d2c14-f19f-11eb-8c24-001a4aab830c. 2021-07-31T01:44:11.690 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:11 smithi045 conmon[39718]: audit 2021-07-31T01:44:10 2021-07-31T01:44:11.690 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:11 smithi045 conmon[39718]: .248037+0000 mon.a (mon.0) 566 : audit [INF] from='client.? 172.21.15.45:0/3520791073' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]: dispatch 2021-07-31T01:44:11.691 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:11 smithi045 conmon[39718]: audit 2021-07-31T01:44:10.738872+0000 mon.a (mon.0) 567 : audit [INF] from='client.? 172.21.15.45:0/3520791073' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]': finished 2021-07-31T01:44:11.691 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:11 smithi045 conmon[39718]: cluster 2021-07-31T01:44:10.738958+0000 mon.a (mon.0) 568 : cluster [DBG] osdmap e50: 8 total, 8 up, 8 in 2021-07-31T01:44:11.691 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:11 smithi045 conmon[39718]: audit 2021-07-31T01:44:11.205457+0000 mon.a (mon.0) 569 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:44:11.692 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:11 smithi045 conmon[33655]: audit 2021-07-31T01:44:10.248037 2021-07-31T01:44:11.692 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:11 smithi045 conmon[33655]: +0000 mon.a (mon.0) 566 : audit [INF] from='client.? 172.21.15.45:0/3520791073' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]: dispatch 2021-07-31T01:44:11.692 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:11 smithi045 conmon[33655]: audit 2021-07-31T01:44:10.738872+0000 mon.a (mon.0) 567 : audit [INF] from='client.? 172.21.15.45:0/3520791073' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]': finished 2021-07-31T01:44:11.692 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:11 smithi045 conmon[33655]: cluster 2021-07-31T01:44:10.738958+0000 mon.a (mon.0) 568 : cluster [DBG] osdmap e50: 8 total, 8 up, 8 in 2021-07-31T01:44:11.693 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:11 smithi045 conmon[33655]: audit 2021-07-31T01:44:11.205457+0000 mon.a (mon.0) 569 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:44:12.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:11 smithi154 conmon[34553]: audit 2021-07-31T01:44:10.248037+0000 mon.a (mon.0) 566 : audit [INF] from='client.? 172.21.15.45:0/3520791073' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]: dispatch 2021-07-31T01:44:12.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:11 smithi154 conmon[34553]: audit 2021-07-31T01:44:10.738872+0000 mon.a (mon.0) 567 : audit [INF] from='client.? 172.21.15.45:0/3520791073' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]': finished 2021-07-31T01:44:12.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:11 smithi154 conmon[34553]: cluster 2021-07-31T01:44:10.738958+0000 mon.a (mon.0) 568 : cluster [DBG] osdmap e50: 8 total, 8 up, 8 in 2021-07-31T01:44:12.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:11 smithi154 conmon[34553]: audit 2021-07-31T01:44: 2021-07-31T01:44:12.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:11 smithi154 conmon[34553]: 11.205457+0000 mon.a (mon.0) 569 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:44:13.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:12 smithi154 conmon[34553]: cephadm 2021-07-31T01:44:11.211293+0000 mgr.y (mgr.14146) 199 : 2021-07-31T01:44:13.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:12 smithi154 conmon[34553]: cephadm [INF] Deploying daemon alertmanager.a on smithi045 2021-07-31T01:44:13.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:12 smithi154 conmon[34553]: cluster 2021-07-31T01:44:11.341673+0000 mgr.y (mgr.14146) 200 : cluster [DBG] pgmap v176: 9 pgs: 8 unknown, 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:44:13.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:12 smithi154 conmon[34553]: cluster 2021-07-31T01:44:11.740840+0000 2021-07-31T01:44:13.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:12 smithi154 conmon[34553]: mon.a (mon.0) 570 : cluster [DBG] osdmap e51: 8 total, 8 up, 8 in 2021-07-31T01:44:13.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:12 smithi045 conmon[33655]: cephadm 2021-07-31T01:44:11.211293+0000 mgr.y (mgr.14146) 199 : cephadm [INF] Deploying daemon alertmanager.a on smithi045 2021-07-31T01:44:13.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:12 smithi045 conmon[33655]: cluster 2021-07-31T01:44:11.341673+0000 mgr.y (mgr.14146) 200 : cluster [DBG] pgmap v176: 9 pgs: 8 unknown, 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:44:13.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:12 smithi045 conmon[33655]: cluster 2021-07-31T01:44:11.740840+0000 mon.a (mon.0) 570 : cluster [DBG] osdmap e51: 8 total, 8 up, 8 in 2021-07-31T01:44:13.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:12 smithi045 conmon[39718]: cephadm 2021-07-31T01:44:11.211293+0000 mgr.y (mgr.14146) 199 : cephadm [INF] Deploying daemon alertmanager.a on smithi045 2021-07-31T01:44:13.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:12 smithi045 conmon[39718]: cluster 2021-07-31T01:44:11.341673+0000 mgr.y (mgr.14146) 200 : cluster [DBG] pgmap v176: 9 pgs: 8 unknown, 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:44:13.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:12 smithi045 conmon[39718]: cluster 2021-07-31T01:44:11.740840+0000 mon.a (mon.0) 570 : cluster [DBG] osdmap e51: 8 total, 8 up, 8 in 2021-07-31T01:44:14.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:13 smithi154 conmon[34553]: cluster 2021-07-31T01:44:12.741350+0000 mon.a (mon.0) 571 : cluster [DBG] osdmap e52: 8 total, 8 up, 8 in 2021-07-31T01:44:14.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:13 smithi045 conmon[33655]: cluster 2021-07-31T01:44:12.741350+0000 2021-07-31T01:44:14.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:13 smithi045 conmon[33655]: mon.a (mon.0) 571 : cluster [DBG] osdmap e52: 8 total, 8 up, 8 in 2021-07-31T01:44:14.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:13 smithi045 conmon[39718]: cluster 2021-07-31T01:44:12.741350+0000 mon.a 2021-07-31T01:44:14.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:13 smithi045 conmon[39718]: (mon.0) 571 : cluster [DBG] osdmap e52: 8 total, 8 up, 8 in 2021-07-31T01:44:14.750 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:14 smithi045 conmon[33655]: cluster 2021-07-31T01:44:13.341995+0000 mgr.y (mgr.14146 2021-07-31T01:44:14.750 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:14 smithi045 conmon[33655]: ) 201 : cluster [DBG] pgmap v179: 9 pgs: 3 unknown, 6 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:44:15.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:14 smithi154 conmon[34553]: cluster 2021-07-31T01:44:13.341995+0000 mgr.y (mgr.14146) 201 : cluster [DBG] pgmap v179: 9 pgs: 3 unknown, 6 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:44:15.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:14 smithi154 conmon[34553]: audit 2021-07-31T01:44:14.441070+0000 mon.a (mon.0) 572 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:44:15.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:14 smithi045 conmon[33655]: audit 2021-07-31T01:44:14.441070+0000 mon.a (mon.0) 572 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:44:15.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:14 smithi045 conmon[39718]: cluster 2021-07-31T01:44:15.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:14 smithi045 conmon[39718]: 2021-07 2021-07-31T01:44:15.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:14 smithi045 conmon[39718]: -31T01:44:13.341995+0000 mgr.y (mgr.14146) 201 : cluster [DBG] pgmap v179: 9 pgs: 3 unknown, 6 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:44:15.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:14 smithi045 conmon[39718]: 2021-07-31T01:44:15.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:14 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T01:44:15.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:14 smithi045 conmon[39718]: 01:44:14.441070+0000 mon.a (mon. 2021-07-31T01:44:15.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:14 smithi045 conmon[39718]: 0) 572 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:44:16.758 INFO:teuthology.orchestra.run.smithi045.stderr:enabled application 'rbd' on pool 'rbd' 2021-07-31T01:44:17.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:16 smithi045 conmon[33655]: cluster 2021-07-31T01:44:15.342480+0000 mgr.y (mgr.14146) 2021-07-31T01:44:17.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:16 smithi045 conmon[33655]: 202 : cluster [DBG] pgmap v180: 9 pgs: 9 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:44:17.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:16 smithi045 conmon[33655]: audit 2021-07-31T01:44:16.115792+0000 mon.a (mon.0) 573 : audit [INF] from='client.? 172.21.15.45:0/1359319820' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]: dispatch 2021-07-31T01:44:17.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:16 smithi045 conmon[39718]: cluster 2021-07-31T01:44:15.342480+0000 mgr.y (mgr.14146) 202 : cluster [DBG] pgmap v180: 9 pgs: 9 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:44:17.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:16 smithi045 conmon[39718]: audit 2021-07-31T01:44:16.115792+0000 mon.a (mon.0) 573 : audit [INF] from='client.? 172.21.15.45:0/1359319820' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]: dispatch 2021-07-31T01:44:17.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:16 smithi154 conmon[34553]: cluster 2021-07-31T01:44:15.342480+0000 mgr.y (mgr.14146) 202 : cluster [DBG] pgmap v180: 9 pgs: 9 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:44:17.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:16 smithi154 conmon[34553]: 2021-07-31T01:44:17.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:16 smithi154 conmon[34553]: audit 2021-07-31T01:44:16.115792+0000 mon.a (mon.0) 573 : audit [INF] from='client.? 172.21.15.45:0/1359319820' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]: dispatch 2021-07-31T01:44:17.761 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:17 smithi045 conmon[39718]: audit 2021-07-31T01:44:16.754534+0000 mon.a 2021-07-31T01:44:17.761 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:17 smithi045 conmon[39718]: (mon.0) 574 : audit [INF] from='client.? 172.21.15.45:0/1359319820' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]': finished 2021-07-31T01:44:17.762 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:17 smithi045 conmon[39718]: cluster 2021-07-31T01:44:16.754609+0000 mon.a (mon 2021-07-31T01:44:17.762 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:17 smithi045 conmon[39718]: .0) 575 : cluster [DBG] osdmap e53: 8 total, 8 up, 8 in 2021-07-31T01:44:17.762 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:17 smithi045 conmon[33655]: audit 2021-07-31T01:44:16.754534+0000 mon.a 2021-07-31T01:44:17.763 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:17 smithi045 conmon[33655]: (mon.0) 574 : audit [INF] from='client.? 172.21.15.45:0/1359319820' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]': finished 2021-07-31T01:44:17.763 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:17 smithi045 conmon[33655]: cluster 2021-07-31T01:44:16.754609+0000 mon.a (mon.0) 575 : cluster [DBG] osdmap e53: 8 total, 8 up, 8 in 2021-07-31T01:44:17.840 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell --fsid f52d2c14-f19f-11eb-8c24-001a4aab830c -- ceph osd dump --format=json 2021-07-31T01:44:18.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:17 smithi154 conmon[34553]: audit 2021-07-31T01:44:16.754534+0000 mon.a (mon.0) 574 : audit [INF] from='client.? 172.21.15.45:0/1359319820' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]': finished 2021-07-31T01:44:18.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:17 smithi154 conmon[34553]: cluster 2021-07-31T01:44:16.754609+0000 mon.a (mon.0) 575 : cluster [DBG] osdmap e53: 8 total, 8 up, 8 in 2021-07-31T01:44:18.324 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f52d2c14-f19f-11eb-8c24-001a4aab830c/mon.a/config 2021-07-31T01:44:18.889 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:18 smithi045 conmon[33655]: cluster 2021-07-31T01:44:17.343033+0000 mgr.y (mgr.14146) 203 : cluster [DBG] pgmap v182: 9 pgs: 9 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:44:18.889 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:18 smithi045 conmon[39718]: cluster 2021-07-31T01:44:17.343033+0000 mgr.y (mgr.14146) 203 : cluster [DBG] pgmap v182: 9 pgs: 9 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:44:19.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:18 smithi154 conmon[34553]: cluster 2021-07-31T01:44:17.343033+0000 mgr.y (mgr.14146) 203 : cluster [DBG] pgmap v182: 9 pgs: 9 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:44:19.432 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:44:19 smithi045 systemd[1]: Starting Ceph alertmanager.a for f52d2c14-f19f-11eb-8c24-001a4aab830c... 2021-07-31T01:44:19.750 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:44:19 smithi045 podman[57174]: 2021-07-31 01:44:19.601008514 +0000 UTC m=+0.183210401 image pull docker.io/prom/alertmanager:v0.20.0 2021-07-31T01:44:20.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:44:19 smithi045 podman[57174]: 2021-07-31 01:44:19.889782391 +0000 UTC m=+0.471984324 volume create 62c6ace8113ab34184606b54029191d1cadfcda6112bbf139564135daabea533 2021-07-31T01:44:20.329 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:44:20 smithi045 podman[57174]: 2021-07-31 01:44:20.049658421 +0000 UTC m=+0.631860321 container create 06ec3286016a82b740b2a51234acbf4149d4048e72d0ca2cccf42bd05711b31c (image=docker.io/prom/alertmanager:v0.20.0, name=ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c-alertmanager.a, maintainer=The Prometheus Authors ) 2021-07-31T01:44:20.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:20 smithi045 conmon[33655]: cluster 2021-07-31T01:44:19.343454+0000 mgr.y (mgr.14146) 204 : cluster [DBG] pgmap v183: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:44:20.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:20 smithi045 conmon[33655]: audit 2021-07-31T01:44:19.445306+0000 mon.a (mon.0) 576 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:44:20.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:20 smithi045 conmon[39718]: cluster 2021-07-31T01:44:19.343454+0000 mgr.y (mgr.14146) 204 : cluster [DBG] pgmap v183: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:44:20.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:20 smithi045 conmon[39718]: audit 2021-07-31T01:44:19.445306+0000 mon.a (mon.0) 576 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:44:20.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:20 smithi154 conmon[34553]: cluster 2021-07-31T01:44:19.343454+0000 mgr.y (mgr.14146) 204 : cluster [DBG] 2021-07-31T01:44:20.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:20 smithi154 conmon[34553]: pgmap v183: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:44:20.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:20 smithi154 conmon[34553]: audit 2021-07-31T01:44:19.445306+0000 mon.a (mon.0) 576 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:44:20.846 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:44:20 smithi045 podman[57174]: 2021-07-31 01:44:20.696251171 +0000 UTC m=+1.278453125 container init 06ec3286016a82b740b2a51234acbf4149d4048e72d0ca2cccf42bd05711b31c (image=docker.io/prom/alertmanager:v0.20.0, name=ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c-alertmanager.a, maintainer=The Prometheus Authors ) 2021-07-31T01:44:20.847 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:44:20 smithi045 conmon[57269]: level=info ts=2021-07-31T01:44:20.734Z caller=main.go:231 msg="Starting Alertmanager" version="(version=0.20.0, branch=HEAD, revision=f74be0400a6243d10bb53812d6fa408ad71ff32d)" 2021-07-31T01:44:20.847 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:44:20 smithi045 conmon[57269]: level=info ts=2021-07-31T01:44:20.734Z caller=main.go:232 build_context="(go=go1.13.5, user=root@00c3106655f8, date=20191211-14:13:14)" 2021-07-31T01:44:20.847 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:44:20 smithi045 conmon[57269]: level=info ts=2021-07-31T01:44:20.735Z caller=cluster.go:161 component=cluster msg="setting advertise address explicitly" addr=172.21.15.45 port=9094 2021-07-31T01:44:20.848 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:44:20 smithi045 conmon[57269]: level=info ts=2021-07-31T01:44:20.736Z caller=cluster.go:623 component=cluster msg="Waiting for gossip to settle..." interval=2s 2021-07-31T01:44:20.848 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:44:20 smithi045 conmon[57269]: level=info ts=2021-07-31T01:44:20.755Z caller=coordinator.go:119 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2021-07-31T01:44:20.848 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:44:20 smithi045 conmon[57269]: level=info ts=2021-07-31T01:44:20.755Z caller=coordinator.go:131 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2021-07-31T01:44:20.849 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:44:20 smithi045 conmon[57269]: level=info ts=2021-07-31T01:44:20.758Z caller=main.go:497 msg=Listening address=:9093 2021-07-31T01:44:20.849 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:44:20 smithi045 podman[57174]: 2021-07-31 01:44:20.771185052 +0000 UTC m=+1.353386899 container start 06ec3286016a82b740b2a51234acbf4149d4048e72d0ca2cccf42bd05711b31c (image=docker.io/prom/alertmanager:v0.20.0, name=ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c-alertmanager.a, maintainer=The Prometheus Authors ) 2021-07-31T01:44:20.849 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:44:20 smithi045 bash[57102]: 06ec3286016a82b740b2a51234acbf4149d4048e72d0ca2cccf42bd05711b31c 2021-07-31T01:44:20.849 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:44:20 smithi045 systemd[1]: Started Ceph alertmanager.a for f52d2c14-f19f-11eb-8c24-001a4aab830c. 2021-07-31T01:44:21.089 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:44:21.090 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":53,"fsid":"f52d2c14-f19f-11eb-8c24-001a4aab830c","created":"2021-07-31T01:38:50.680976+0000","modified":"2021-07-31T01:44:16.750759+0000","last_up_change":"2021-07-31T01:43:46.985098+0000","last_in_change":"2021-07-31T01:43:30.970323+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-07-31T01:42:07.901706+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-07-31T01:44:10.685434+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"20724b9a-515c-4878-9e4f-301e21a5a985","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":2841572524},{"type":"v1","addr":"172.21.15.45:6803","nonce":2841572524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":2841572524},{"type":"v1","addr":"172.21.15.45:6805","nonce":2841572524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":2841572524},{"type":"v1","addr":"172.21.15.45:6809","nonce":2841572524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":2841572524},{"type":"v1","addr":"172.21.15.45:6807","nonce":2841572524}]},"public_addr":"172.21.15.45:6803/2841572524","cluster_addr":"172.21.15.45:6805/2841572524","heartbeat_back_addr":"172.21.15.45:6809/2841572524","heartbeat_front_addr":"172.21.15.45:6807/2841572524","state":["exists","up"]},{"osd":1,"uuid":"f47f025a-6f29-417a-9bd1-9af9f91f38a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1747309179},{"type":"v1","addr":"172.21.15.45:6811","nonce":1747309179}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1747309179},{"type":"v1","addr":"172.21.15.45:6813","nonce":1747309179}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":1747309179},{"type":"v1","addr":"172.21.15.45:6817","nonce":1747309179}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1747309179},{"type":"v1","addr":"172.21.15.45:6815","nonce":1747309179}]},"public_addr":"172.21.15.45:6811/1747309179","cluster_addr":"172.21.15.45:6813/1747309179","heartbeat_back_addr":"172.21.15.45:6817/1747309179","heartbeat_front_addr":"172.21.15.45:6815/1747309179","state":["exists","up"]},{"osd":2,"uuid":"bfbe641e-b9d7-4005-8fb5-bc7d5af76388","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2518799988},{"type":"v1","addr":"172.21.15.45:6819","nonce":2518799988}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2518799988},{"type":"v1","addr":"172.21.15.45:6821","nonce":2518799988}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2518799988},{"type":"v1","addr":"172.21.15.45:6825","nonce":2518799988}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2518799988},{"type":"v1","addr":"172.21.15.45:6823","nonce":2518799988}]},"public_addr":"172.21.15.45:6819/2518799988","cluster_addr":"172.21.15.45:6821/2518799988","heartbeat_back_addr":"172.21.15.45:6825/2518799988","heartbeat_front_addr":"172.21.15.45:6823/2518799988","state":["exists","up"]},{"osd":3,"uuid":"37b826a3-b611-4001-b9a7-1966032e56d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3024698157},{"type":"v1","addr":"172.21.15.45:6827","nonce":3024698157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3024698157},{"type":"v1","addr":"172.21.15.45:6829","nonce":3024698157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3024698157},{"type":"v1","addr":"172.21.15.45:6833","nonce":3024698157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3024698157},{"type":"v1","addr":"172.21.15.45:6831","nonce":3024698157}]},"public_addr":"172.21.15.45:6827/3024698157","cluster_addr":"172.21.15.45:6829/3024698157","heartbeat_back_addr":"172.21.15.45:6833/3024698157","heartbeat_front_addr":"172.21.15.45:6831/3024698157","state":["exists","up"]},{"osd":4,"uuid":"69881d9d-3e56-4483-b76b-7dba14cfaac3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":4277144246},{"type":"v1","addr":"172.21.15.154:6801","nonce":4277144246}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":4277144246},{"type":"v1","addr":"172.21.15.154:6803","nonce":4277144246}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":4277144246},{"type":"v1","addr":"172.21.15.154:6807","nonce":4277144246}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":4277144246},{"type":"v1","addr":"172.21.15.154:6805","nonce":4277144246}]},"public_addr":"172.21.15.154:6801/4277144246","cluster_addr":"172.21.15.154:6803/4277144246","heartbeat_back_addr":"172.21.15.154:6807/4277144246","heartbeat_front_addr":"172.21.15.154:6805/4277144246","state":["exists","up"]},{"osd":5,"uuid":"7dcad412-f42a-478b-886a-8f4bd82a04ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":579203063},{"type":"v1","addr":"172.21.15.154:6809","nonce":579203063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":579203063},{"type":"v1","addr":"172.21.15.154:6811","nonce":579203063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":579203063},{"type":"v1","addr":"172.21.15.154:6815","nonce":579203063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":579203063},{"type":"v1","addr":"172.21.15.154:6813","nonce":579203063}]},"public_addr":"172.21.15.154:6809/579203063","cluster_addr":"172.21.15.154:6811/579203063","heartbeat_back_addr":"172.21.15.154:6815/579203063","heartbeat_front_addr":"172.21.15.154:6813/579203063","state":["exists","up"]},{"osd":6,"uuid":"94258018-67f7-49d7-83c3-85070ff8c2f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":3963397470},{"type":"v1","addr":"172.21.15.154:6817","nonce":3963397470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":3963397470},{"type":"v1","addr":"172.21.15.154:6819","nonce":3963397470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":3963397470},{"type":"v1","addr":"172.21.15.154:6823","nonce":3963397470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":3963397470},{"type":"v1","addr":"172.21.15.154:6821","nonce":3963397470}]},"public_addr":"172.21.15.154:6817/3963397470","cluster_addr":"172.21.15.154:6819/3963397470","heartbeat_back_addr":"172.21.15.154:6823/3963397470","heartbeat_front_addr":"172.21.15.154:6821/3963397470","state":["exists","up"]},{"osd":7,"uuid":"da82715a-b8b7-4786-bbc2-ee0cc3ac9b18","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1263188610},{"type":"v1","addr":"172.21.15.154:6825","nonce":1263188610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1263188610},{"type":"v1","addr":"172.21.15.154:6827","nonce":1263188610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1263188610},{"type":"v1","addr":"172.21.15.154:6831","nonce":1263188610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1263188610},{"type":"v1","addr":"172.21.15.154:6829","nonce":1263188610}]},"public_addr":"172.21.15.154:6825/1263188610","cluster_addr":"172.21.15.154:6827/1263188610","heartbeat_back_addr":"172.21.15.154:6831/1263188610","heartbeat_front_addr":"172.21.15.154:6829/1263188610","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-07-31T01:41:26.799531+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-07-31T01:41:46.224506+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-07-31T01:42:06.246092+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-07-31T01:42:26.612987+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-07-31T01:42:45.424194+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-07-31T01:43:06.036564+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-07-31T01:43:25.839643+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-07-31T01:43:45.387900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:0/3355281269":"2021-08-01T01:39:39.238202+0000","172.21.15.45:6801/2610418727":"2021-08-01T01:39:39.238202+0000","172.21.15.45:6800/2610418727":"2021-08-01T01:39:39.238202+0000","172.21.15.45:6800/487754334":"2021-08-01T01:39:15.200151+0000","172.21.15.45:0/536736800":"2021-08-01T01:39:39.238202+0000","172.21.15.45:0/4040877953":"2021-08-01T01:39:15.200151+0000","172.21.15.45:0/2377876452":"2021-08-01T01:39:15.200151+0000","172.21.15.45:6801/487754334":"2021-08-01T01:39:15.200151+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-07-31T01:44:21.851 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2021-07-31T01:42:07.901706+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2021-07-31T01:44:10.685434+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '53', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-07-31T01:44:21.852 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell --fsid f52d2c14-f19f-11eb-8c24-001a4aab830c -- ceph osd pool get device_health_metrics pg_num 2021-07-31T01:44:22.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:21 smithi045 conmon[33655]: audit 2021-07-31T01:44:20 2021-07-31T01:44:22.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:21 smithi045 conmon[33655]: .854397+0000 mon.a (mon.0) 577 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:44:22.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:21 smithi045 conmon[33655]: audit 2021-07-31T01:44:20.918048+0000 mon.a (mon.0) 578 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:44:22.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:21 smithi045 conmon[33655]: audit 2021-07-31T01:44:20.921606+0000 mon.a (mon.0) 579 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:44:22.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:21 smithi045 conmon[33655]: audit 2021-07-31T01:44:20.922032+0000 mon.a (mon.0) 580 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2021-07-31T01:44:22.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:21 smithi045 conmon[33655]: audit 2021-07-31T01:44:20.922367+0000 mgr.y (mgr.14146) 205 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2021-07-31T01:44:22.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:21 smithi045 conmon[33655]: audit 2021-07-31T01:44:20.928680+0000 mon.a (mon.0) 581 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:44:22.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:21 smithi045 conmon[33655]: cephadm 2021-07-31T01:44:20.932469+0000 mgr.y (mgr.14146) 206 : cephadm [INF] Deploying daemon grafana.a on smithi154 2021-07-31T01:44:22.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:21 smithi045 conmon[33655]: audit 2021-07-31T01:44:21.088494+0000 mon.b (mon.2) 24 : audit [DBG] from='client.? 172.21.15.45:0/3988634109' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-07-31T01:44:22.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:21 smithi045 conmon[39718]: audit 2021-07-31T01 2021-07-31T01:44:22.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:21 smithi045 conmon[39718]: :44:20.854397+0000 mon.a (mon.0) 577 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:44:22.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:21 smithi045 conmon[39718]: audit 2021-07-31T01:44:22.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:21 smithi045 conmon[39718]: 2021-07-31T01:44: 2021-07-31T01:44:22.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:21 smithi045 conmon[39718]: 20.918048+0000 mon.a ( 2021-07-31T01:44:22.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:21 smithi045 conmon[39718]: mon.0) 578 : audit [INF] 2021-07-31T01:44:22.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:21 smithi045 conmon[39718]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:44:22.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:21 smithi045 conmon[39718]: audit 2021-07-31T01:44:22.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:21 smithi045 conmon[39718]: 2021-07-31T01:44:20.921606 2021-07-31T01:44:22.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:21 smithi045 conmon[39718]: +0000 mon.a (mon.0) 579 : audit 2021-07-31T01:44:22.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:21 smithi045 conmon[39718]: [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:44:22.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:21 smithi045 conmon[39718]: audit 2021-07 2021-07-31T01:44:22.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:21 smithi045 conmon[39718]: -31T01:44:20.922032+0000 mon.a (mon.0) 580 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2021-07-31T01:44:22.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:21 smithi045 conmon[39718]: audit 2021-07 2021-07-31T01:44:22.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:21 smithi045 conmon[39718]: -31T01:44:20.922367+0000 2021-07-31T01:44:22.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:21 smithi045 conmon[39718]: mgr.y (mgr.14146) 205 2021-07-31T01:44:22.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:21 smithi045 conmon[39718]: : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2021-07-31T01:44:22.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:21 smithi045 conmon[39718]: audit 2021-07-31T01:44:22.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:21 smithi045 conmon[39718]: 2021-07-31T01:44: 2021-07-31T01:44:22.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:21 smithi045 conmon[39718]: 20.928680+0000 mon.a (mon.0 2021-07-31T01:44:22.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:21 smithi045 conmon[39718]: ) 581 : audit [INF] 2021-07-31T01:44:22.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:21 smithi045 conmon[39718]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:44:22.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:21 smithi045 conmon[39718]: cephadm 2021 2021-07-31T01:44:22.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:21 smithi045 conmon[39718]: -07-31T01:44:20.932469+0000 2021-07-31T01:44:22.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:21 smithi045 conmon[39718]: mgr.y (mgr.14146) 206 : cephadm 2021-07-31T01:44:22.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:21 smithi045 conmon[39718]: [INF] Deploying daemon grafana.a on smithi154 2021-07-31T01:44:22.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:21 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T01:44:22.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:21 smithi045 conmon[39718]: T01:44:21.088494+0000 mon.b (mon. 2021-07-31T01:44:22.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:21 smithi045 conmon[39718]: 2) 24 : audit [DBG] from='client.? 172.21.15.45:0/3988634109' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-07-31T01:44:22.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:21 smithi045 conmon[39718]: 2021-07-31T01:44:22.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:21 smithi154 conmon[34553]: audit 2021-07-31T01:44:20. 2021-07-31T01:44:22.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:21 smithi154 conmon[34553]: 854397+0000 mon.a (mon.0) 577 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:44:22.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:21 smithi154 conmon[34553]: audit 2021-07-31T01:44:20.918048+0000 mon.a (mon.0) 578 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:44:22.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:21 smithi154 conmon[34553]: audit 2021-07-31T01:44:20.921606+0000 mon.a (mon.0) 579 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:44:22.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:21 smithi154 conmon[34553]: audit 2021-07-31T01:44:20.922032+0000 mon.a (mon.0) 580 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2021-07-31T01:44:22.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:21 smithi154 conmon[34553]: audit 2021-07-31T01:44:20.922367+0000 mgr.y (mgr.14146) 205 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2021-07-31T01:44:22.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:21 smithi154 conmon[34553]: audit 2021-07-31T01:44:20.928680+0000 mon.a (mon.0) 581 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:44:22.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:21 smithi154 conmon[34553]: cephadm 2021-07-31T01:44:20.932469+0000 mgr.y (mgr.14146) 206 : cephadm [INF] Deploying daemon grafana.a on smithi154 2021-07-31T01:44:22.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:21 smithi154 conmon[34553]: audit 2021-07-31T01:44:21.088494+0000 mon.b (mon.2) 24 : audit [DBG] from='client.? 172.21.15.45:0/3988634109' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-07-31T01:44:22.304 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f52d2c14-f19f-11eb-8c24-001a4aab830c/mon.a/config 2021-07-31T01:44:23.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:22 smithi045 conmon[39718]: cluster 2021-07-31T01:44:21.343788+0000 mgr.y (mgr.14146) 207 : cluster [DBG] pgmap v184: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:44:23.027 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:44:22 smithi045 conmon[57269]: level=info ts=2021-07-31T01:44:22.736Z caller=cluster.go:648 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000084122s 2021-07-31T01:44:23.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:22 smithi045 conmon[33655]: cluster 2021-07-31T01:44:21.343788+0000 mgr.y (mgr.14146) 207 : cluster [DBG] pgmap v184: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:44:23.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:22 smithi154 conmon[34553]: cluster 2021-07-31T01:44:21.343788+0000 mgr.y (mgr. 2021-07-31T01:44:23.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:22 smithi154 conmon[34553]: 14146) 207 : cluster [DBG] pgmap v184: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:44:24.142 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2021-07-31T01:44:24.911 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell --fsid f52d2c14-f19f-11eb-8c24-001a4aab830c -- ceph osd pool get rbd pg_num 2021-07-31T01:44:25.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:24 smithi154 conmon[34553]: cluster 2021-07-31T01:44:23.344207+0000 mgr.y (mgr.14146) 2021-07-31T01:44:25.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:24 smithi154 conmon[34553]: 208 : cluster [DBG] pgmap v185: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:44:25.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:24 smithi154 conmon[34553]: audit 2021-07-31T01:44:24.141927+0000 mon.a (mon.0) 582 : audit [DBG] from='client.? 172.21.15.45:0/175461702' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2021-07-31T01:44:25.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:24 smithi154 conmon[34553]: audit 2021-07-31T01:44:25.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:24 smithi154 conmon[34553]: 2021-07-31T01:44:24.449082+0000 mon.a (mon.0) 2021-07-31T01:44:25.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:24 smithi154 conmon[34553]: 583 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:44:25.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:24 smithi045 conmon[33655]: cluster 2021-07-31T01:44:23.344207+0000 mgr.y ( 2021-07-31T01:44:25.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:24 smithi045 conmon[33655]: mgr.14146) 208 : cluster [DBG] pgmap v185: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:44:25.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:24 smithi045 conmon[33655]: audit 2021-07-31T01:44:24.141927+0000 mon.a (mon.0) 582 : audit [DBG] from='client.? 172.21.15.45:0/175461702' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2021-07-31T01:44:25.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:24 smithi045 conmon[33655]: audit 2021-07-31T01:44:24.449082+0000 mon.a (mon.0) 583 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:44:25.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:24 smithi045 conmon[39718]: cluster 2021-07-31T01:44: 2021-07-31T01:44:25.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:24 smithi045 conmon[39718]: 23.344207+0000 mgr.y (mgr.14146) 208 : cluster [DBG] pgmap v185: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:44:25.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:24 smithi045 conmon[39718]: audit 2021-07-31T01:44:24.141927+0000 mon.a (mon.0) 582 : audit [DBG] from='client.? 172.21.15.45:0/175461702' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2021-07-31T01:44:25.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:24 smithi045 conmon[39718]: audit 2021-07-31T01:44:24.449082+0000 mon.a (mon.0) 583 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:44:25.368 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f52d2c14-f19f-11eb-8c24-001a4aab830c/mon.a/config 2021-07-31T01:44:27.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:26 smithi045 conmon[33655]: cluster 2021-07-31T01:44:25.344612+0000 mgr.y (mgr.14146) 209 : cluster [DBG] pgmap v186: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:44:27.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:26 smithi045 conmon[39718]: cluster 2021-07-31T01:44:25.344612+0000 mgr.y (mgr.14146) 209 : cluster [DBG] pgmap v186: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:44:27.121 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 8 2021-07-31T01:44:27.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:26 smithi154 conmon[34553]: cluster 2021-07-31T01:44:25.344612+0000 mgr.y (mgr 2021-07-31T01:44:27.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:26 smithi154 conmon[34553]: .14146) 209 : cluster [DBG] pgmap v186: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:44:27.784 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2021-07-31T01:44:27.785 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2021-07-31T01:44:27.786 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell --fsid f52d2c14-f19f-11eb-8c24-001a4aab830c -- ceph mgr dump --format=json 2021-07-31T01:44:28.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:27 smithi045 conmon[39718]: audit 2021-07-31T01:44:27.121124+0000 2021-07-31T01:44:28.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:27 smithi045 conmon[39718]: mon.c (mon.1) 10 : audit [DBG] from='client.? 172.21.15.45:0/1166594718' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]: dispatch 2021-07-31T01:44:28.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:27 smithi045 conmon[33655]: audit 2021-07-31T01:44:27.121124+0000 mon.c 2021-07-31T01:44:28.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:27 smithi045 conmon[33655]: (mon.1) 10 : audit [DBG] from='client.? 172.21.15.45:0/1166594718' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]: dispatch 2021-07-31T01:44:28.237 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f52d2c14-f19f-11eb-8c24-001a4aab830c/mon.a/config 2021-07-31T01:44:28.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:27 smithi154 conmon[34553]: audit 2021-07-31T01:44:27.121124+0000 2021-07-31T01:44:28.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:27 smithi154 conmon[34553]: mon.c (mon.1) 10 : audit [DBG] from='client.? 172.21.15.45:0/1166594718' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]: dispatch 2021-07-31T01:44:29.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:28 smithi045 conmon[39718]: cluster 2021-07-31T01 2021-07-31T01:44:29.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:28 smithi045 conmon[39718]: :44:27.345068+0000 mgr.y (mgr.14146) 210 : cluster [DBG] pgmap v187: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:44:29.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:28 smithi045 conmon[33655]: cluster 2021-07-31T01:44:27 2021-07-31T01:44:29.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:28 smithi045 conmon[33655]: .345068+0000 mgr.y (mgr.14146) 210 : cluster [DBG] pgmap v187: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:44:29.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:28 smithi154 conmon[34553]: cluster 2021-07-31T01:44:27.345068+0000 2021-07-31T01:44:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:28 smithi154 conmon[34553]: mgr.y (mgr.14146) 210 : cluster [DBG] pgmap v187: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:44:30.067 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:44:30.841 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":14,"active_gid":14146,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1006624185},{"type":"v1","addr":"172.21.15.45:6801","nonce":1006624185}]},"active_addr":"172.21.15.45:6801/1006624185","active_change":"2021-07-31T01:39:39.238472+0000","active_mgr_features":4540138297136906239,"available":true,"standbys":[{"gid":24116,"name":"x","mgr_features":4540138297136906239,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"docker.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Registry to which we should normalize unqualified image names","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"str","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"docker.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Registry to which we should normalize unqualified image names","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"str","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.45:8443/"},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":3,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.45:0","nonce":4027209229}]},{"addrvec":[{"type":"v2","addr":"172.21.15.45:0","nonce":2869241044}]}]}} 2021-07-31T01:44:30.846 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2021-07-31T01:44:30.847 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2021-07-31T01:44:30.847 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell --fsid f52d2c14-f19f-11eb-8c24-001a4aab830c -- ceph osd dump --format=json 2021-07-31T01:44:31.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:44:30 smithi045 conmon[57269]: level=info ts=2021-07-31T01:44:30.738Z caller=cluster.go:640 component=cluster msg="gossip settled; proceeding" elapsed=10.001209743s 2021-07-31T01:44:31.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:30 smithi045 conmon[39718]: cluster 2021-07-31T01:44:29.345472+0000 mgr.y (mgr.14146 2021-07-31T01:44:31.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:30 smithi045 conmon[39718]: ) 211 : cluster [DBG] pgmap v188: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:44:31.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:30 smithi045 conmon[39718]: audit 2021-07-31T01:44:30.061745 2021-07-31T01:44:31.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:30 smithi045 conmon[39718]: +0000 mon.a (mon.0) 584 : audit [DBG] from='client.? 172.21.15.45:0/2034129281' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2021-07-31T01:44:31.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:30 smithi045 conmon[33655]: cluster 2021-07-31T01:44:29.345472+0000 mgr.y (mgr.14146) 2021-07-31T01:44:31.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:30 smithi045 conmon[33655]: 211 : cluster [DBG] pgmap v188: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:44:31.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:30 smithi045 conmon[33655]: audit 2021-07-31T01:44:30.061745+0000 mon.a (mon.0) 584 : audit [DBG] from='client.? 172.21.15.45:0/2034129281' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2021-07-31T01:44:31.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:30 smithi154 conmon[34553]: cluster 2021-07-31T01:44:29.345472+0000 mgr.y (mgr 2021-07-31T01:44:31.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:30 smithi154 conmon[34553]: .14146) 211 : cluster [DBG] pgmap v188: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:44:31.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:30 smithi154 conmon[34553]: audit 2021-07-31T01:44:30.061745+0000 mon.a (mon.0) 584 : audit [DBG] from='client.? 172.21.15.45:0/2034129281' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2021-07-31T01:44:31.295 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f52d2c14-f19f-11eb-8c24-001a4aab830c/mon.a/config 2021-07-31T01:44:33.056 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:44:33.057 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":53,"fsid":"f52d2c14-f19f-11eb-8c24-001a4aab830c","created":"2021-07-31T01:38:50.680976+0000","modified":"2021-07-31T01:44:16.750759+0000","last_up_change":"2021-07-31T01:43:46.985098+0000","last_in_change":"2021-07-31T01:43:30.970323+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-07-31T01:42:07.901706+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-07-31T01:44:10.685434+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"20724b9a-515c-4878-9e4f-301e21a5a985","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":2841572524},{"type":"v1","addr":"172.21.15.45:6803","nonce":2841572524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":2841572524},{"type":"v1","addr":"172.21.15.45:6805","nonce":2841572524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":2841572524},{"type":"v1","addr":"172.21.15.45:6809","nonce":2841572524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":2841572524},{"type":"v1","addr":"172.21.15.45:6807","nonce":2841572524}]},"public_addr":"172.21.15.45:6803/2841572524","cluster_addr":"172.21.15.45:6805/2841572524","heartbeat_back_addr":"172.21.15.45:6809/2841572524","heartbeat_front_addr":"172.21.15.45:6807/2841572524","state":["exists","up"]},{"osd":1,"uuid":"f47f025a-6f29-417a-9bd1-9af9f91f38a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1747309179},{"type":"v1","addr":"172.21.15.45:6811","nonce":1747309179}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1747309179},{"type":"v1","addr":"172.21.15.45:6813","nonce":1747309179}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":1747309179},{"type":"v1","addr":"172.21.15.45:6817","nonce":1747309179}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1747309179},{"type":"v1","addr":"172.21.15.45:6815","nonce":1747309179}]},"public_addr":"172.21.15.45:6811/1747309179","cluster_addr":"172.21.15.45:6813/1747309179","heartbeat_back_addr":"172.21.15.45:6817/1747309179","heartbeat_front_addr":"172.21.15.45:6815/1747309179","state":["exists","up"]},{"osd":2,"uuid":"bfbe641e-b9d7-4005-8fb5-bc7d5af76388","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2518799988},{"type":"v1","addr":"172.21.15.45:6819","nonce":2518799988}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2518799988},{"type":"v1","addr":"172.21.15.45:6821","nonce":2518799988}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2518799988},{"type":"v1","addr":"172.21.15.45:6825","nonce":2518799988}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2518799988},{"type":"v1","addr":"172.21.15.45:6823","nonce":2518799988}]},"public_addr":"172.21.15.45:6819/2518799988","cluster_addr":"172.21.15.45:6821/2518799988","heartbeat_back_addr":"172.21.15.45:6825/2518799988","heartbeat_front_addr":"172.21.15.45:6823/2518799988","state":["exists","up"]},{"osd":3,"uuid":"37b826a3-b611-4001-b9a7-1966032e56d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3024698157},{"type":"v1","addr":"172.21.15.45:6827","nonce":3024698157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3024698157},{"type":"v1","addr":"172.21.15.45:6829","nonce":3024698157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3024698157},{"type":"v1","addr":"172.21.15.45:6833","nonce":3024698157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3024698157},{"type":"v1","addr":"172.21.15.45:6831","nonce":3024698157}]},"public_addr":"172.21.15.45:6827/3024698157","cluster_addr":"172.21.15.45:6829/3024698157","heartbeat_back_addr":"172.21.15.45:6833/3024698157","heartbeat_front_addr":"172.21.15.45:6831/3024698157","state":["exists","up"]},{"osd":4,"uuid":"69881d9d-3e56-4483-b76b-7dba14cfaac3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":4277144246},{"type":"v1","addr":"172.21.15.154:6801","nonce":4277144246}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":4277144246},{"type":"v1","addr":"172.21.15.154:6803","nonce":4277144246}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":4277144246},{"type":"v1","addr":"172.21.15.154:6807","nonce":4277144246}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":4277144246},{"type":"v1","addr":"172.21.15.154:6805","nonce":4277144246}]},"public_addr":"172.21.15.154:6801/4277144246","cluster_addr":"172.21.15.154:6803/4277144246","heartbeat_back_addr":"172.21.15.154:6807/4277144246","heartbeat_front_addr":"172.21.15.154:6805/4277144246","state":["exists","up"]},{"osd":5,"uuid":"7dcad412-f42a-478b-886a-8f4bd82a04ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":579203063},{"type":"v1","addr":"172.21.15.154:6809","nonce":579203063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":579203063},{"type":"v1","addr":"172.21.15.154:6811","nonce":579203063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":579203063},{"type":"v1","addr":"172.21.15.154:6815","nonce":579203063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":579203063},{"type":"v1","addr":"172.21.15.154:6813","nonce":579203063}]},"public_addr":"172.21.15.154:6809/579203063","cluster_addr":"172.21.15.154:6811/579203063","heartbeat_back_addr":"172.21.15.154:6815/579203063","heartbeat_front_addr":"172.21.15.154:6813/579203063","state":["exists","up"]},{"osd":6,"uuid":"94258018-67f7-49d7-83c3-85070ff8c2f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":3963397470},{"type":"v1","addr":"172.21.15.154:6817","nonce":3963397470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":3963397470},{"type":"v1","addr":"172.21.15.154:6819","nonce":3963397470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":3963397470},{"type":"v1","addr":"172.21.15.154:6823","nonce":3963397470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":3963397470},{"type":"v1","addr":"172.21.15.154:6821","nonce":3963397470}]},"public_addr":"172.21.15.154:6817/3963397470","cluster_addr":"172.21.15.154:6819/3963397470","heartbeat_back_addr":"172.21.15.154:6823/3963397470","heartbeat_front_addr":"172.21.15.154:6821/3963397470","state":["exists","up"]},{"osd":7,"uuid":"da82715a-b8b7-4786-bbc2-ee0cc3ac9b18","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1263188610},{"type":"v1","addr":"172.21.15.154:6825","nonce":1263188610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1263188610},{"type":"v1","addr":"172.21.15.154:6827","nonce":1263188610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1263188610},{"type":"v1","addr":"172.21.15.154:6831","nonce":1263188610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1263188610},{"type":"v1","addr":"172.21.15.154:6829","nonce":1263188610}]},"public_addr":"172.21.15.154:6825/1263188610","cluster_addr":"172.21.15.154:6827/1263188610","heartbeat_back_addr":"172.21.15.154:6831/1263188610","heartbeat_front_addr":"172.21.15.154:6829/1263188610","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-07-31T01:41:26.799531+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-07-31T01:41:46.224506+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-07-31T01:42:06.246092+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-07-31T01:42:26.612987+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-07-31T01:42:45.424194+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-07-31T01:43:06.036564+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-07-31T01:43:25.839643+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-07-31T01:43:45.387900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:0/3355281269":"2021-08-01T01:39:39.238202+0000","172.21.15.45:6801/2610418727":"2021-08-01T01:39:39.238202+0000","172.21.15.45:6800/2610418727":"2021-08-01T01:39:39.238202+0000","172.21.15.45:6800/487754334":"2021-08-01T01:39:15.200151+0000","172.21.15.45:0/536736800":"2021-08-01T01:39:39.238202+0000","172.21.15.45:0/4040877953":"2021-08-01T01:39:15.200151+0000","172.21.15.45:0/2377876452":"2021-08-01T01:39:15.200151+0000","172.21.15.45:6801/487754334":"2021-08-01T01:39:15.200151+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-07-31T01:44:33.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:32 smithi154 conmon[34553]: cluster 2021-07-31T01:44:31.345771+0000 mgr.y (mgr.14146) 212 : cluster [DBG] pgmap v189: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:44:33.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:32 smithi045 conmon[33655]: cluster 2021-07-31T01:44:31.345771 2021-07-31T01:44:33.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:32 smithi045 conmon[33655]: +0000 mgr.y (mgr.14146) 212 : cluster [DBG] pgmap v189: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:44:33.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:32 smithi045 conmon[39718]: cluster 2021-07-31T01:44:31.345771+0000 mgr.y (mgr 2021-07-31T01:44:33.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:32 smithi045 conmon[39718]: .14146) 212 : cluster [DBG] pgmap v189: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:44:33.820 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2021-07-31T01:44:33.836 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell --fsid f52d2c14-f19f-11eb-8c24-001a4aab830c -- ceph osd dump --format=json 2021-07-31T01:44:34.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:33 smithi045 conmon[33655]: audit 2021-07-31T01:44:33.054672+0000 mon.c (mon.1) 11 : audit [DBG] from='client.? 172.21.15.45:0/3762221390' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-07-31T01:44:34.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:33 smithi045 conmon[39718]: audit 2021-07-31T01:44:33.054672+0000 mon.c (mon.1) 11 : audit [DBG] from='client.? 172.21.15.45:0/3762221390' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-07-31T01:44:34.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:33 smithi154 conmon[34553]: audit 2021-07-31T01:44:33.054672+0000 mon.c (mon.1) 11 : audit [DBG] from='client.? 172.21.15.45:0/3762221390' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-07-31T01:44:34.291 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f52d2c14-f19f-11eb-8c24-001a4aab830c/mon.a/config 2021-07-31T01:44:35.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:34 smithi154 conmon[34553]: cluster 2021-07-31T01:44:33.346094+0000 mgr.y (mgr.14146) 213 : cluster [DBG] pgmap v190: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:44:35.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:34 smithi154 conmon[34553]: 2021-07-31T01:44:35.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:34 smithi045 conmon[33655]: cluster 2021-07-31T01:44 2021-07-31T01:44:35.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:34 smithi045 conmon[33655]: :33.346094+0000 mgr.y (mgr.14146) 213 : cluster [DBG] pgmap v190: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:44:35.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:34 smithi045 conmon[39718]: cluster 2021-07-31T01:44:33. 2021-07-31T01:44:35.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:34 smithi045 conmon[39718]: 346094+0000 mgr.y (mgr.14146) 213 : cluster [DBG] pgmap v190: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:44:36.143 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:44:36.143 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":53,"fsid":"f52d2c14-f19f-11eb-8c24-001a4aab830c","created":"2021-07-31T01:38:50.680976+0000","modified":"2021-07-31T01:44:16.750759+0000","last_up_change":"2021-07-31T01:43:46.985098+0000","last_in_change":"2021-07-31T01:43:30.970323+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-07-31T01:42:07.901706+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-07-31T01:44:10.685434+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"20724b9a-515c-4878-9e4f-301e21a5a985","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":2841572524},{"type":"v1","addr":"172.21.15.45:6803","nonce":2841572524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":2841572524},{"type":"v1","addr":"172.21.15.45:6805","nonce":2841572524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":2841572524},{"type":"v1","addr":"172.21.15.45:6809","nonce":2841572524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":2841572524},{"type":"v1","addr":"172.21.15.45:6807","nonce":2841572524}]},"public_addr":"172.21.15.45:6803/2841572524","cluster_addr":"172.21.15.45:6805/2841572524","heartbeat_back_addr":"172.21.15.45:6809/2841572524","heartbeat_front_addr":"172.21.15.45:6807/2841572524","state":["exists","up"]},{"osd":1,"uuid":"f47f025a-6f29-417a-9bd1-9af9f91f38a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1747309179},{"type":"v1","addr":"172.21.15.45:6811","nonce":1747309179}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1747309179},{"type":"v1","addr":"172.21.15.45:6813","nonce":1747309179}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":1747309179},{"type":"v1","addr":"172.21.15.45:6817","nonce":1747309179}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1747309179},{"type":"v1","addr":"172.21.15.45:6815","nonce":1747309179}]},"public_addr":"172.21.15.45:6811/1747309179","cluster_addr":"172.21.15.45:6813/1747309179","heartbeat_back_addr":"172.21.15.45:6817/1747309179","heartbeat_front_addr":"172.21.15.45:6815/1747309179","state":["exists","up"]},{"osd":2,"uuid":"bfbe641e-b9d7-4005-8fb5-bc7d5af76388","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2518799988},{"type":"v1","addr":"172.21.15.45:6819","nonce":2518799988}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2518799988},{"type":"v1","addr":"172.21.15.45:6821","nonce":2518799988}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2518799988},{"type":"v1","addr":"172.21.15.45:6825","nonce":2518799988}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2518799988},{"type":"v1","addr":"172.21.15.45:6823","nonce":2518799988}]},"public_addr":"172.21.15.45:6819/2518799988","cluster_addr":"172.21.15.45:6821/2518799988","heartbeat_back_addr":"172.21.15.45:6825/2518799988","heartbeat_front_addr":"172.21.15.45:6823/2518799988","state":["exists","up"]},{"osd":3,"uuid":"37b826a3-b611-4001-b9a7-1966032e56d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3024698157},{"type":"v1","addr":"172.21.15.45:6827","nonce":3024698157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3024698157},{"type":"v1","addr":"172.21.15.45:6829","nonce":3024698157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3024698157},{"type":"v1","addr":"172.21.15.45:6833","nonce":3024698157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3024698157},{"type":"v1","addr":"172.21.15.45:6831","nonce":3024698157}]},"public_addr":"172.21.15.45:6827/3024698157","cluster_addr":"172.21.15.45:6829/3024698157","heartbeat_back_addr":"172.21.15.45:6833/3024698157","heartbeat_front_addr":"172.21.15.45:6831/3024698157","state":["exists","up"]},{"osd":4,"uuid":"69881d9d-3e56-4483-b76b-7dba14cfaac3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":4277144246},{"type":"v1","addr":"172.21.15.154:6801","nonce":4277144246}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":4277144246},{"type":"v1","addr":"172.21.15.154:6803","nonce":4277144246}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":4277144246},{"type":"v1","addr":"172.21.15.154:6807","nonce":4277144246}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":4277144246},{"type":"v1","addr":"172.21.15.154:6805","nonce":4277144246}]},"public_addr":"172.21.15.154:6801/4277144246","cluster_addr":"172.21.15.154:6803/4277144246","heartbeat_back_addr":"172.21.15.154:6807/4277144246","heartbeat_front_addr":"172.21.15.154:6805/4277144246","state":["exists","up"]},{"osd":5,"uuid":"7dcad412-f42a-478b-886a-8f4bd82a04ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":579203063},{"type":"v1","addr":"172.21.15.154:6809","nonce":579203063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":579203063},{"type":"v1","addr":"172.21.15.154:6811","nonce":579203063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":579203063},{"type":"v1","addr":"172.21.15.154:6815","nonce":579203063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":579203063},{"type":"v1","addr":"172.21.15.154:6813","nonce":579203063}]},"public_addr":"172.21.15.154:6809/579203063","cluster_addr":"172.21.15.154:6811/579203063","heartbeat_back_addr":"172.21.15.154:6815/579203063","heartbeat_front_addr":"172.21.15.154:6813/579203063","state":["exists","up"]},{"osd":6,"uuid":"94258018-67f7-49d7-83c3-85070ff8c2f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":3963397470},{"type":"v1","addr":"172.21.15.154:6817","nonce":3963397470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":3963397470},{"type":"v1","addr":"172.21.15.154:6819","nonce":3963397470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":3963397470},{"type":"v1","addr":"172.21.15.154:6823","nonce":3963397470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":3963397470},{"type":"v1","addr":"172.21.15.154:6821","nonce":3963397470}]},"public_addr":"172.21.15.154:6817/3963397470","cluster_addr":"172.21.15.154:6819/3963397470","heartbeat_back_addr":"172.21.15.154:6823/3963397470","heartbeat_front_addr":"172.21.15.154:6821/3963397470","state":["exists","up"]},{"osd":7,"uuid":"da82715a-b8b7-4786-bbc2-ee0cc3ac9b18","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1263188610},{"type":"v1","addr":"172.21.15.154:6825","nonce":1263188610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1263188610},{"type":"v1","addr":"172.21.15.154:6827","nonce":1263188610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1263188610},{"type":"v1","addr":"172.21.15.154:6831","nonce":1263188610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1263188610},{"type":"v1","addr":"172.21.15.154:6829","nonce":1263188610}]},"public_addr":"172.21.15.154:6825/1263188610","cluster_addr":"172.21.15.154:6827/1263188610","heartbeat_back_addr":"172.21.15.154:6831/1263188610","heartbeat_front_addr":"172.21.15.154:6829/1263188610","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-07-31T01:41:26.799531+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-07-31T01:41:46.224506+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-07-31T01:42:06.246092+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-07-31T01:42:26.612987+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-07-31T01:42:45.424194+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-07-31T01:43:06.036564+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-07-31T01:43:25.839643+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-07-31T01:43:45.387900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:0/3355281269":"2021-08-01T01:39:39.238202+0000","172.21.15.45:6801/2610418727":"2021-08-01T01:39:39.238202+0000","172.21.15.45:6800/2610418727":"2021-08-01T01:39:39.238202+0000","172.21.15.45:6800/487754334":"2021-08-01T01:39:15.200151+0000","172.21.15.45:0/536736800":"2021-08-01T01:39:39.238202+0000","172.21.15.45:0/4040877953":"2021-08-01T01:39:15.200151+0000","172.21.15.45:0/2377876452":"2021-08-01T01:39:15.200151+0000","172.21.15.45:6801/487754334":"2021-08-01T01:39:15.200151+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-07-31T01:44:36.898 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell --fsid f52d2c14-f19f-11eb-8c24-001a4aab830c -- ceph tell osd.0 flush_pg_stats 2021-07-31T01:44:37.205 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:36 smithi154 conmon[34553]: cluster 2021-07-31T01:44:35.346528+0000 mgr.y (mgr.14146) 214 : cluster [DBG] pgmap v191: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:44:37.205 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:36 smithi154 conmon[34553]: audit 2021-07-31T01:44:36.141658+0000 mon.c (mon.1) 12 : audit [DBG] from='client.? 172.21.15.45:0/3560637167' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-07-31T01:44:37.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:36 smithi045 conmon[39718]: cluster 2021-07-31T01:44: 2021-07-31T01:44:37.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:36 smithi045 conmon[39718]: 35.346528+0000 mgr.y (mgr.14146) 214 : cluster [DBG] pgmap v191: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:44:37.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:36 smithi045 conmon[39718]: audit 2021-07-31T01:44:36.141658+0000 mon.c (mon.1) 12 : audit [DBG] from='client.? 172.21.15.45:0/3560637167' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-07-31T01:44:37.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:36 smithi045 conmon[33655]: cluster 2021-07-31T01:44:35.346528+0000 mgr.y (mgr.14146 2021-07-31T01:44:37.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:36 smithi045 conmon[33655]: ) 214 : cluster [DBG] pgmap v191: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:44:37.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:36 smithi045 conmon[33655]: audit 2021-07-31T01:44:36.141658+0000 mon.c (mon.1) 12 : audit [DBG] from='client.? 172.21.15.45:0/3560637167' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-07-31T01:44:37.355 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f52d2c14-f19f-11eb-8c24-001a4aab830c/mon.a/config 2021-07-31T01:44:37.503 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:37 smithi154 systemd[1]: Starting Ceph grafana.a for f52d2c14-f19f-11eb-8c24-001a4aab830c... 2021-07-31T01:44:37.985 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:37 smithi154 podman[53692]: 2021-07-31 01:44:37.563015758 +0000 UTC m=+0.030327835 image pull docker.io/ceph/ceph-grafana:6.7.4 2021-07-31T01:44:37.986 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:37 smithi154 podman[53692]: 2021-07-31 01:44:37.700176776 +0000 UTC m=+0.167488866 container create 60b9d954f74702aaad1f0a4f40b8ab27f72c5da65f8efdc38ed143cf43a6aac1 (image=docker.io/ceph/ceph-grafana:6.7.4, name=ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c-grafana.a, org.label-schema.schema-version=1.0, io.buildah.version=1.19.8, org.label-schema.name=CentOS Base Image, description=Ceph Grafana Container, org.label-schema.build-date=20201204, org.label-schema.vendor=CentOS, maintainer=Paul Cuzner , summary=Grafana Container configured for Ceph mgr/dashboard integration, org.label-schema.license=GPLv2) 2021-07-31T01:44:38.280 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:38 smithi154 podman[53692]: 2021-07-31 01:44:38.02512152 +0000 UTC m=+0.492433745 container init 60b9d954f74702aaad1f0a4f40b8ab27f72c5da65f8efdc38ed143cf43a6aac1 (image=docker.io/ceph/ceph-grafana:6.7.4, name=ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c-grafana.a, summary=Grafana Container configured for Ceph mgr/dashboard integration, org.label-schema.build-date=20201204, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, description=Ceph Grafana Container, maintainer=Paul Cuzner , io.buildah.version=1.19.8, org.label-schema.name=CentOS Base Image, org.label-schema.vendor=CentOS) 2021-07-31T01:44:38.281 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:38 smithi154 conmon[53732]: t=2021-07-31T01:44:38+0000 lvl=info msg="Starting Grafana" logger=server version=6.7.4 commit=8e44bbc5f5 branch=HEAD compiled=2020-05-26T17:35:38+0000 2021-07-31T01:44:38.281 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:38 smithi154 conmon[53732]: t=2021-07-31T01:44:38+0000 lvl=info msg="Config loaded from" logger=settings file=/usr/share/grafana/conf/defaults.ini 2021-07-31T01:44:38.281 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:38 smithi154 conmon[53732]: t=2021-07-31T01:44:38+0000 lvl=info msg="Config loaded from" logger=settings file=/etc/grafana/grafana.ini 2021-07-31T01:44:38.281 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:38 smithi154 conmon[53732]: t=2021-07-31T01:44:38+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_DATA=/var/lib/grafana" 2021-07-31T01:44:38.282 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:38 smithi154 conmon[53732]: t=2021-07-31T01:44:38+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_LOGS=/var/log/grafana" 2021-07-31T01:44:38.282 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:38 smithi154 conmon[53732]: t=2021-07-31T01:44:38+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2021-07-31T01:44:38.282 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:38 smithi154 conmon[53732]: t=2021-07-31T01:44:38+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2021-07-31T01:44:38.282 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:38 smithi154 conmon[53732]: t=2021-07-31T01:44:38+0000 lvl=info msg="Path Home" logger=settings path=/usr/share/grafana 2021-07-31T01:44:38.282 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:38 smithi154 conmon[53732]: t=2021-07-31T01:44:38+0000 lvl=info msg="Path Data" logger=settings path=/var/lib/grafana 2021-07-31T01:44:38.283 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:38 smithi154 conmon[53732]: t=2021-07-31T01:44:38+0000 lvl=info msg="Path Logs" logger=settings path=/var/log/grafana 2021-07-31T01:44:38.283 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:38 smithi154 conmon[53732]: t=2021-07-31T01:44:38+0000 lvl=info msg="Path Plugins" logger=settings path=/var/lib/grafana/plugins 2021-07-31T01:44:38.283 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:38 smithi154 conmon[53732]: t=2021-07-31T01:44:38+0000 lvl=info msg="Path Provisioning" logger=settings path=/etc/grafana/provisioning 2021-07-31T01:44:38.283 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:38 smithi154 conmon[53732]: t=2021-07-31T01:44:38+0000 lvl=info msg="App mode production" logger=settings 2021-07-31T01:44:38.284 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:38 smithi154 conmon[53732]: t=2021-07-31T01:44:38+0000 lvl=info msg="Initializing SqlStore" logger=server 2021-07-31T01:44:38.284 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:38 smithi154 conmon[53732]: t=2021-07-31T01:44:38+0000 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3 2021-07-31T01:44:38.284 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:38 smithi154 conmon[53732]: t=2021-07-31T01:44:38+0000 lvl=info msg="Starting DB migration" logger=migrator 2021-07-31T01:44:38.285 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:38 smithi154 conmon[53732]: t=2021-07-31T01:44:38+0000 lvl=info msg="Executing migration" logger=migrator id="create migration_log table" 2021-07-31T01:44:38.285 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:38 smithi154 podman[53692]: 2021-07-31 01:44:38.085402924 +0000 UTC m=+0.552715139 container start 60b9d954f74702aaad1f0a4f40b8ab27f72c5da65f8efdc38ed143cf43a6aac1 (image=docker.io/ceph/ceph-grafana:6.7.4, name=ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c-grafana.a, io.buildah.version=1.19.8, org.label-schema.name=CentOS Base Image, maintainer=Paul Cuzner , org.label-schema.build-date=20201204, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, summary=Grafana Container configured for Ceph mgr/dashboard integration, org.label-schema.vendor=CentOS, description=Ceph Grafana Container) 2021-07-31T01:44:38.285 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:38 smithi154 bash[53621]: 60b9d954f74702aaad1f0a4f40b8ab27f72c5da65f8efdc38ed143cf43a6aac1 2021-07-31T01:44:38.285 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:38 smithi154 conmon[53732]: t=2021-07-31T01:44:38+0000 lvl=info msg="Executing migration" logger=migrator id="create user table" 2021-07-31T01:44:38.285 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:38 smithi154 conmon[53732]: t=2021-07-31T01:44:38+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.login" 2021-07-31T01:44:38.286 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:38 smithi154 systemd[1]: Started Ceph grafana.a for f52d2c14-f19f-11eb-8c24-001a4aab830c. 2021-07-31T01:44:38.286 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:38 smithi154 conmon[53732]: t=2021-07-31T01:44:38+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.email" 2021-07-31T01:44:38.286 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:38 smithi154 conmon[53732]: t=2021-07-31T01:44:38+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_login - v1" 2021-07-31T01:44:38.286 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:38 smithi154 conmon[53732]: t=2021-07-31T01:44:38+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_email - v1" 2021-07-31T01:44:38.549 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:38 smithi154 conmon[53732]: t=2021-07-31T01:44:38+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table user to user_v1 - v1" 2021-07-31T01:44:38.549 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:38 smithi154 conmon[53732]: t=2021-07-31T01:44:38+0000 lvl=info msg="Executing migration" logger=migrator id="create user table v2" 2021-07-31T01:44:38.549 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:38 smithi154 conmon[53732]: t=2021-07-31T01:44:38+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_login - v2" 2021-07-31T01:44:38.550 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:38 smithi154 conmon[53732]: t=2021-07-31T01:44:38+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_email - v2" 2021-07-31T01:44:38.550 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:38 smithi154 conmon[53732]: t=2021-07-31T01:44:38+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2021-07-31T01:44:38.550 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:38 smithi154 conmon[53732]: t=2021-07-31T01:44:38+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table user_v1" 2021-07-31T01:44:38.811 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:38 smithi154 conmon[53732]: t=2021-07-31T01:44:38+0000 lvl=info msg="Executing migration" logger=migrator id="Add column help_flags1 to user table" 2021-07-31T01:44:38.811 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:38 smithi154 conmon[53732]: t=2021-07-31T01:44:38+0000 lvl=info msg="Executing migration" logger=migrator id="Update user table charset" 2021-07-31T01:44:38.811 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:38 smithi154 conmon[53732]: t=2021-07-31T01:44:38+0000 lvl=info msg="Executing migration" logger=migrator id="Add last_seen_at column to user" 2021-07-31T01:44:38.812 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:38 smithi154 conmon[53732]: t=2021-07-31T01:44:38+0000 lvl=info msg="Executing migration" logger=migrator id="Add missing user data" 2021-07-31T01:44:38.812 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:38 smithi154 conmon[53732]: t=2021-07-31T01:44:38+0000 lvl=info msg="Executing migration" logger=migrator id="Add is_disabled column to user" 2021-07-31T01:44:38.812 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:38 smithi154 conmon[53732]: t=2021-07-31T01:44:38+0000 lvl=info msg="Executing migration" logger=migrator id="create temp user table v1-7" 2021-07-31T01:44:38.812 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:38 smithi154 conmon[53732]: t=2021-07-31T01:44:38+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_email - v1-7" 2021-07-31T01:44:39.085 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:38 smithi154 conmon[53732]: t=2021-07-31T01:44:38+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_org_id - v1-7" 2021-07-31T01:44:39.086 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:38 smithi154 conmon[53732]: t=2021-07-31T01:44:38+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_code - v1-7" 2021-07-31T01:44:39.086 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:38 smithi154 conmon[53732]: t=2021-07-31T01:44:38+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_status - v1-7" 2021-07-31T01:44:39.086 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:38 smithi154 conmon[53732]: t=2021-07-31T01:44:38+0000 lvl=info msg="Executing migration" logger=migrator id="Update temp_user table charset" 2021-07-31T01:44:39.087 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:39 smithi154 conmon[53732]: t=2021-07-31T01:44:39+0000 lvl=info msg="Executing migration" logger=migrator id="create star table" 2021-07-31T01:44:39.087 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:38 smithi154 conmon[34553]: cluster 2021-07-31T01:44:37.346894+0000 mgr.y (mgr.14146 2021-07-31T01:44:39.087 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:38 smithi154 conmon[34553]: ) 215 : cluster [DBG] pgmap v192: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:44:39.087 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:38 smithi154 conmon[34553]: audit 2021-07-31T01:44:38.210834+0000 mon.a (mon.0) 585 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:44:39.088 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:38 smithi154 conmon[34553]: audit 2021-07-31T 2021-07-31T01:44:39.088 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:38 smithi154 conmon[34553]: 01:44:38.211871+0000 mon.a (mon 2021-07-31T01:44:39.088 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:38 smithi154 conmon[34553]: .0) 586 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:44:39.088 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:38 smithi154 conmon[34553]: audit 2021-07-31T01:44:39.089 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:38 smithi154 conmon[34553]: 2021-07-31T01: 2021-07-31T01:44:39.089 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:38 smithi154 conmon[34553]: 44:38.212856+0000 mon.a (mon.0 2021-07-31T01:44:39.089 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:38 smithi154 conmon[34553]: ) 587 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:44:39.089 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:38 smithi154 conmon[34553]: audit 2021-07- 2021-07-31T01:44:39.090 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:38 smithi154 conmon[34553]: 31T01:44:38.213607+0000 2021-07-31T01:44:39.090 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:38 smithi154 conmon[34553]: mon.a (mon.0) 588 : audit [INF] 2021-07-31T01:44:39.090 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:38 smithi154 conmon[34553]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:44:39.153 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:38 smithi045 conmon[33655]: cluster 2021-07-31T01:44:37.346894+0000 mgr.y (mgr.14146) 215 : cluster [DBG] pgmap v192: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:44:39.154 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:38 smithi045 conmon[33655]: audit 2021-07-31T01:44:38.210834+0000 mon.a (mon.0) 585 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:44:39.154 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:38 smithi045 conmon[33655]: audit 2021-07-31T01:44:38.211871+0000 mon.a (mon.0) 586 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:44:39.154 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:38 smithi045 conmon[33655]: audit 2021-07-31T01:44:38.212856+0000 mon.a (mon.0) 587 : 2021-07-31T01:44:39.154 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:38 smithi045 conmon[33655]: audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:44:39.155 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:38 smithi045 conmon[33655]: audit 2021-07-31T01:44:38.213607+0000 mon.a (mon.0) 588 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:44:39.155 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:38 smithi045 conmon[39718]: cluster 2021-07-31T01:44:37.346894+0000 mgr.y (mgr.14146) 215 : cluster [DBG] pgmap v192: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:44:39.155 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:38 smithi045 conmon[39718]: audit 2021-07-31T01:44:38.210834+0000 mon.a (mon.0) 585 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:44:39.156 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:38 smithi045 conmon[39718]: audit 2021-07-31T01:44:38.211871+0000 mon.a (mon.0) 586 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:44:39.156 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:38 smithi045 conmon[39718]: audit 2021-07-31T01:44:38.212856+0000 mon.a (mon.0) 587 : 2021-07-31T01:44:39.156 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:38 smithi045 conmon[39718]: audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:44:39.156 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:38 smithi045 conmon[39718]: audit 2021-07-31T01:44:38.213607+0000 mon.a (mon.0) 588 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:44:39.381 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:39 smithi154 conmon[53732]: t=2021-07-31T01:44:39+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index star.user_id_dashboard_id" 2021-07-31T01:44:39.382 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:39 smithi154 conmon[53732]: t=2021-07-31T01:44:39+0000 lvl=info msg="Executing migration" logger=migrator id="create org table v1" 2021-07-31T01:44:39.382 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:39 smithi154 conmon[53732]: t=2021-07-31T01:44:39+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_name - v1" 2021-07-31T01:44:39.382 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:39 smithi154 conmon[53732]: t=2021-07-31T01:44:39+0000 lvl=info msg="Executing migration" logger=migrator id="create org_user table v1" 2021-07-31T01:44:39.382 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:39 smithi154 conmon[53732]: t=2021-07-31T01:44:39+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_org_user_org_id - v1" 2021-07-31T01:44:39.633 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:39 smithi154 conmon[53732]: t=2021-07-31T01:44:39+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_user_org_id_user_id - v1" 2021-07-31T01:44:39.633 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:39 smithi154 conmon[53732]: t=2021-07-31T01:44:39+0000 lvl=info msg="Executing migration" logger=migrator id="Update org table charset" 2021-07-31T01:44:39.633 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:39 smithi154 conmon[53732]: t=2021-07-31T01:44:39+0000 lvl=info msg="Executing migration" logger=migrator id="Update org_user table charset" 2021-07-31T01:44:39.634 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:39 smithi154 conmon[53732]: t=2021-07-31T01:44:39+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate all Read Only Viewers to Viewers" 2021-07-31T01:44:39.634 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:39 smithi154 conmon[53732]: t=2021-07-31T01:44:39+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard table" 2021-07-31T01:44:39.634 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:39 smithi154 conmon[53732]: t=2021-07-31T01:44:39+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard.account_id" 2021-07-31T01:44:39.634 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:39 smithi154 conmon[53732]: t=2021-07-31T01:44:39+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_account_id_slug" 2021-07-31T01:44:39.901 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:39 smithi154 conmon[53732]: t=2021-07-31T01:44:39+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_tag table" 2021-07-31T01:44:39.901 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:39 smithi154 conmon[53732]: t=2021-07-31T01:44:39+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_tag.dasboard_id_term" 2021-07-31T01:44:39.902 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:39 smithi154 conmon[53732]: t=2021-07-31T01:44:39+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2021-07-31T01:44:39.902 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:39 smithi154 conmon[53732]: t=2021-07-31T01:44:39+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard to dashboard_v1 - v1" 2021-07-31T01:44:39.902 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:39 smithi154 conmon[53732]: t=2021-07-31T01:44:39+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard v2" 2021-07-31T01:44:39.902 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:39 smithi154 conmon[53732]: t=2021-07-31T01:44:39+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_org_id - v2" 2021-07-31T01:44:39.919 INFO:teuthology.orchestra.run.smithi045.stdout:34359738408 2021-07-31T01:44:39.919 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell --fsid f52d2c14-f19f-11eb-8c24-001a4aab830c -- ceph tell osd.1 flush_pg_stats 2021-07-31T01:44:40.169 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:39 smithi154 conmon[34553]: audit 2021-07-31T01:44:39.428646+0000 mon.a (mon.0) 589 : audit [INF] 2021-07-31T01:44:40.170 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:39 smithi154 conmon[34553]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:44:40.170 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:39 smithi154 conmon[34553]: audit 2021-07-31T01:44:39.429549+0000 mon.a (mon.0) 590 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-07-31T01:44:40.170 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:39 smithi154 conmon[34553]: audit 2021-07-31T01:44:39.453042+0000 mon.a (mon.0) 591 : audit 2021-07-31T01:44:40.170 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:39 smithi154 conmon[34553]: [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:44:40.171 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:39 smithi154 conmon[53732]: t=2021-07-31T01:44:39+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_org_id_slug - v2" 2021-07-31T01:44:40.171 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:39 smithi154 conmon[53732]: t=2021-07-31T01:44:39+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard v1 to v2" 2021-07-31T01:44:40.171 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:39 smithi154 conmon[53732]: t=2021-07-31T01:44:39+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_v1" 2021-07-31T01:44:40.171 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:40 smithi154 conmon[53732]: t=2021-07-31T01:44:40+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard.data to mediumtext v1" 2021-07-31T01:44:40.172 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:40 smithi154 conmon[53732]: t=2021-07-31T01:44:40+0000 lvl=info msg="Executing migration" logger=migrator id="Add column updated_by in dashboard - v2" 2021-07-31T01:44:40.172 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:40 smithi154 conmon[53732]: t=2021-07-31T01:44:40+0000 lvl=info msg="Executing migration" logger=migrator id="Add column created_by in dashboard - v2" 2021-07-31T01:44:40.172 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:40 smithi154 conmon[53732]: t=2021-07-31T01:44:40+0000 lvl=info msg="Executing migration" logger=migrator id="Add column gnetId in dashboard" 2021-07-31T01:44:40.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:39 smithi045 conmon[39718]: audit 2021-07-31T01:44:39 2021-07-31T01:44:40.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:39 smithi045 conmon[39718]: .428646+0000 mon.a (mon.0) 589 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:44:40.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:39 smithi045 conmon[39718]: audit 2021-07-31T01:44:39.429549+0000 mon.a (mon.0) 590 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-07-31T01:44:40.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:39 smithi045 conmon[39718]: audit 2021-07-31T01:44:39.453042+0000 mon.a (mon.0) 591 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:44:40.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:39 smithi045 conmon[33655]: audit 2021-07-31T01: 2021-07-31T01:44:40.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:39 smithi045 conmon[33655]: 44:39.428646+0000 mon.a (mon.0) 589 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:44:40.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:39 smithi045 conmon[33655]: audit 2021-07-31T01:44:39.429549+0000 mon.a (mon.0) 590 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-07-31T01:44:40.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:39 smithi045 conmon[33655]: audit 2021-07-31T01:44:40.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:39 smithi045 conmon[33655]: 2021-07-31T01:44:39.453042+0000 mon.a (mon.0) 2021-07-31T01:44:40.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:39 smithi045 conmon[33655]: 591 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:44:40.373 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f52d2c14-f19f-11eb-8c24-001a4aab830c/mon.a/config 2021-07-31T01:44:40.448 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:40 smithi154 conmon[53732]: t=2021-07-31T01:44:40+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for gnetId in dashboard" 2021-07-31T01:44:40.448 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:40 smithi154 conmon[53732]: t=2021-07-31T01:44:40+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_id in dashboard" 2021-07-31T01:44:40.448 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:40 smithi154 conmon[53732]: t=2021-07-31T01:44:40+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for plugin_id in dashboard" 2021-07-31T01:44:40.448 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:40 smithi154 conmon[53732]: t=2021-07-31T01:44:40+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_id in dashboard_tag" 2021-07-31T01:44:40.449 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:40 smithi154 conmon[53732]: t=2021-07-31T01:44:40+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard table charset" 2021-07-31T01:44:40.449 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:40 smithi154 conmon[53732]: t=2021-07-31T01:44:40+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_tag table charset" 2021-07-31T01:44:40.449 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:40 smithi154 conmon[53732]: t=2021-07-31T01:44:40+0000 lvl=info msg="Executing migration" logger=migrator id="Add column folder_id in dashboard" 2021-07-31T01:44:40.449 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:40 smithi154 conmon[53732]: t=2021-07-31T01:44:40+0000 lvl=info msg="Executing migration" logger=migrator id="Add column isFolder in dashboard" 2021-07-31T01:44:40.700 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:40 smithi154 conmon[53732]: t=2021-07-31T01:44:40+0000 lvl=info msg="Executing migration" logger=migrator id="Add column has_acl in dashboard" 2021-07-31T01:44:40.700 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:40 smithi154 conmon[53732]: t=2021-07-31T01:44:40+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in dashboard" 2021-07-31T01:44:40.700 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:40 smithi154 conmon[53732]: t=2021-07-31T01:44:40+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in dashboard" 2021-07-31T01:44:40.960 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:40 smithi154 conmon[53732]: t=2021-07-31T01:44:40+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index dashboard_org_id_uid" 2021-07-31T01:44:40.960 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:40 smithi154 conmon[53732]: t=2021-07-31T01:44:40+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_slug" 2021-07-31T01:44:40.960 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:40 smithi154 conmon[53732]: t=2021-07-31T01:44:40+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard title length" 2021-07-31T01:44:40.960 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:40 smithi154 conmon[53732]: t=2021-07-31T01:44:40+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index for dashboard_org_id_title_folder_id" 2021-07-31T01:44:41.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:40 smithi154 conmon[34553]: cluster 2021-07-31T01:44:39.347300+0000 mgr.y (mgr.14146 2021-07-31T01:44:41.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:40 smithi154 conmon[34553]: ) 216 : cluster [DBG] pgmap v193: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:44:41.253 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:40 smithi154 conmon[53732]: t=2021-07-31T01:44:40+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning" 2021-07-31T01:44:41.253 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:41 smithi154 conmon[53732]: t=2021-07-31T01:44:41+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2021-07-31T01:44:41.254 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:41 smithi154 conmon[53732]: t=2021-07-31T01:44:41+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning v2" 2021-07-31T01:44:41.254 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:41 smithi154 conmon[53732]: t=2021-07-31T01:44:41+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2021-07-31T01:44:41.254 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:41 smithi154 conmon[53732]: t=2021-07-31T01:44:41+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2021-07-31T01:44:41.254 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:41 smithi154 conmon[53732]: t=2021-07-31T01:44:41+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard_provisioning v1 to v2" 2021-07-31T01:44:41.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:40 smithi045 conmon[39718]: cluster 2021-07-31T01:44:39.347300+0000 mgr.y (mgr.14146) 216 : cluster [DBG] pgmap v193: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:44:41.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:40 smithi045 conmon[33655]: cluster 2021-07-31T01:44:39.347300+0000 mgr.y (mgr.14146) 216 : cluster [DBG] pgmap v193: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:44:41.577 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:41 smithi154 conmon[53732]: t=2021-07-31T01:44:41+0000 lvl=info msg="Executing migration" logger=migrator id="drop dashboard_provisioning_tmp_qwerty" 2021-07-31T01:44:41.577 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:41 smithi154 conmon[53732]: t=2021-07-31T01:44:41+0000 lvl=info msg="Executing migration" logger=migrator id="Add check_sum column" 2021-07-31T01:44:41.577 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:41 smithi154 conmon[53732]: t=2021-07-31T01:44:41+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table" 2021-07-31T01:44:41.577 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:41 smithi154 conmon[53732]: t=2021-07-31T01:44:41+0000 lvl=info msg="Executing migration" logger=migrator id="add index data_source.account_id" 2021-07-31T01:44:41.578 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:41 smithi154 conmon[53732]: t=2021-07-31T01:44:41+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index data_source.account_id_name" 2021-07-31T01:44:41.578 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:41 smithi154 conmon[53732]: t=2021-07-31T01:44:41+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_data_source_account_id - v1" 2021-07-31T01:44:41.830 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:41 smithi154 conmon[53732]: t=2021-07-31T01:44:41+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_data_source_account_id_name - v1" 2021-07-31T01:44:41.831 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:41 smithi154 conmon[53732]: t=2021-07-31T01:44:41+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table data_source to data_source_v1 - v1" 2021-07-31T01:44:41.831 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:41 smithi154 conmon[53732]: t=2021-07-31T01:44:41+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table v2" 2021-07-31T01:44:41.831 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:41 smithi154 conmon[53732]: t=2021-07-31T01:44:41+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_data_source_org_id - v2" 2021-07-31T01:44:41.831 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:41 smithi154 conmon[53732]: t=2021-07-31T01:44:41+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_data_source_org_id_name - v2" 2021-07-31T01:44:41.832 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:41 smithi154 conmon[53732]: t=2021-07-31T01:44:41+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2021-07-31T01:44:41.832 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:41 smithi154 conmon[53732]: t=2021-07-31T01:44:41+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table data_source_v1 #2" 2021-07-31T01:44:41.832 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:41 smithi154 conmon[53732]: t=2021-07-31T01:44:41+0000 lvl=info msg="Executing migration" logger=migrator id="Add column with_credentials" 2021-07-31T01:44:42.084 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:41 smithi154 conmon[53732]: t=2021-07-31T01:44:41+0000 lvl=info msg="Executing migration" logger=migrator id="Add secure json data column" 2021-07-31T01:44:42.085 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:41 smithi154 conmon[53732]: t=2021-07-31T01:44:41+0000 lvl=info msg="Executing migration" logger=migrator id="Update data_source table charset" 2021-07-31T01:44:42.085 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:41 smithi154 conmon[53732]: t=2021-07-31T01:44:41+0000 lvl=info msg="Executing migration" logger=migrator id="Update initial version to 1" 2021-07-31T01:44:42.085 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:41 smithi154 conmon[53732]: t=2021-07-31T01:44:41+0000 lvl=info msg="Executing migration" logger=migrator id="Add read_only data column" 2021-07-31T01:44:42.349 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:42 smithi154 conmon[53732]: t=2021-07-31T01:44:42+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate logging ds to loki ds" 2021-07-31T01:44:42.350 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:42 smithi154 conmon[53732]: t=2021-07-31T01:44:42+0000 lvl=info msg="Executing migration" logger=migrator id="Update json_data with nulls" 2021-07-31T01:44:42.350 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:42 smithi154 conmon[53732]: t=2021-07-31T01:44:42+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table" 2021-07-31T01:44:42.350 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:42 smithi154 conmon[53732]: t=2021-07-31T01:44:42+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id" 2021-07-31T01:44:42.351 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:42 smithi154 conmon[53732]: t=2021-07-31T01:44:42+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.key" 2021-07-31T01:44:42.643 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:42 smithi154 conmon[53732]: t=2021-07-31T01:44:42+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id_name" 2021-07-31T01:44:42.644 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:42 smithi154 conmon[53732]: t=2021-07-31T01:44:42+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_api_key_account_id - v1" 2021-07-31T01:44:42.644 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:42 smithi154 conmon[53732]: t=2021-07-31T01:44:42+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_key - v1" 2021-07-31T01:44:42.644 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:42 smithi154 conmon[53732]: t=2021-07-31T01:44:42+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_account_id_name - v1" 2021-07-31T01:44:42.645 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:42 smithi154 conmon[53732]: t=2021-07-31T01:44:42+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table api_key to api_key_v1 - v1" 2021-07-31T01:44:42.895 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:42 smithi154 conmon[53732]: t=2021-07-31T01:44:42+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table v2" 2021-07-31T01:44:42.896 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:42 smithi154 conmon[53732]: t=2021-07-31T01:44:42+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_api_key_org_id - v2" 2021-07-31T01:44:42.896 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:42 smithi154 conmon[53732]: t=2021-07-31T01:44:42+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_key - v2" 2021-07-31T01:44:42.896 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:42 smithi154 conmon[53732]: t=2021-07-31T01:44:42+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_org_id_name - v2" 2021-07-31T01:44:42.896 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:42 smithi154 conmon[53732]: t=2021-07-31T01:44:42+0000 lvl=info msg="Executing migration" logger=migrator id="copy api_key v1 to v2" 2021-07-31T01:44:42.897 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:42 smithi154 conmon[53732]: t=2021-07-31T01:44:42+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table api_key_v1" 2021-07-31T01:44:42.897 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:42 smithi154 conmon[53732]: t=2021-07-31T01:44:42+0000 lvl=info msg="Executing migration" logger=migrator id="Update api_key table charset" 2021-07-31T01:44:42.897 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:42 smithi154 conmon[53732]: t=2021-07-31T01:44:42+0000 lvl=info msg="Executing migration" logger=migrator id="Add expires to api_key table" 2021-07-31T01:44:43.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:42 smithi045 conmon[33655]: cluster 2021-07-31T01:44:41.347806+0000 2021-07-31T01:44:43.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:42 smithi045 conmon[33655]: mgr.y (mgr.14146) 217 : cluster [DBG] pgmap v194: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:44:43.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:42 smithi045 conmon[33655]: audit 2021-07-31T01:44:42.451415+0000 mon.a (mon.0) 592 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:44:43.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:42 smithi045 conmon[39718]: cluster 2021-07-31T01:44:41.347806+0000 2021-07-31T01:44:43.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:42 smithi045 conmon[39718]: mgr.y (mgr.14146) 217 : cluster [DBG] pgmap v194: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:44:43.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:42 smithi045 conmon[39718]: audit 2021-07-31T01:44:42.451415+0000 mon.a (mon.0) 592 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:44:43.155 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:42 smithi154 conmon[34553]: cluster 2021-07-31T01:44:41.347806+0000 mgr.y (mgr.14146) 217 2021-07-31T01:44:43.155 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:42 smithi154 conmon[34553]: : cluster [DBG] pgmap v194: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:44:43.156 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:42 smithi154 conmon[34553]: audit 2021-07-31T01:44:42.451415+0000 mon.a (mon.0) 592 : audit [INF] 2021-07-31T01:44:43.156 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:42 smithi154 conmon[34553]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:44:43.156 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:42 smithi154 conmon[53732]: t=2021-07-31T01:44:42+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v4" 2021-07-31T01:44:43.156 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:43 smithi154 conmon[53732]: t=2021-07-31T01:44:43+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_snapshot_v4 #1" 2021-07-31T01:44:43.157 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:43 smithi154 conmon[53732]: t=2021-07-31T01:44:43+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v5 #2" 2021-07-31T01:44:43.157 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:43 smithi154 conmon[53732]: t=2021-07-31T01:44:43+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_key - v5" 2021-07-31T01:44:43.157 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:43 smithi154 conmon[53732]: t=2021-07-31T01:44:43+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_delete_key - v5" 2021-07-31T01:44:43.157 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:43 smithi154 conmon[53732]: t=2021-07-31T01:44:43+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_snapshot_user_id - v5" 2021-07-31T01:44:43.158 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:43 smithi154 conmon[53732]: t=2021-07-31T01:44:43+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_snapshot to mediumtext v2" 2021-07-31T01:44:43.389 INFO:teuthology.orchestra.run.smithi045.stdout:55834574886 2021-07-31T01:44:43.389 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell --fsid f52d2c14-f19f-11eb-8c24-001a4aab830c -- ceph tell osd.2 flush_pg_stats 2021-07-31T01:44:43.414 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:43 smithi154 conmon[53732]: t=2021-07-31T01:44:43+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_snapshot table charset" 2021-07-31T01:44:43.415 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:43 smithi154 conmon[53732]: t=2021-07-31T01:44:43+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external_delete_url to dashboard_snapshots table" 2021-07-31T01:44:43.415 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:43 smithi154 conmon[53732]: t=2021-07-31T01:44:43+0000 lvl=info msg="Executing migration" logger=migrator id="create quota table v1" 2021-07-31T01:44:43.415 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:43 smithi154 conmon[53732]: t=2021-07-31T01:44:43+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_quota_org_id_user_id_target - v1" 2021-07-31T01:44:43.416 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:43 smithi154 conmon[53732]: t=2021-07-31T01:44:43+0000 lvl=info msg="Executing migration" logger=migrator id="Update quota table charset" 2021-07-31T01:44:43.666 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:43 smithi154 conmon[53732]: t=2021-07-31T01:44:43+0000 lvl=info msg="Executing migration" logger=migrator id="create plugin_setting table" 2021-07-31T01:44:43.666 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:43 smithi154 conmon[53732]: t=2021-07-31T01:44:43+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2021-07-31T01:44:43.667 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:43 smithi154 conmon[53732]: t=2021-07-31T01:44:43+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_version to plugin_settings" 2021-07-31T01:44:43.667 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:43 smithi154 conmon[53732]: t=2021-07-31T01:44:43+0000 lvl=info msg="Executing migration" logger=migrator id="Update plugin_setting table charset" 2021-07-31T01:44:43.667 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:43 smithi154 conmon[53732]: t=2021-07-31T01:44:43+0000 lvl=info msg="Executing migration" logger=migrator id="create session table" 2021-07-31T01:44:43.667 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:43 smithi154 conmon[53732]: t=2021-07-31T01:44:43+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist table" 2021-07-31T01:44:43.668 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:43 smithi154 conmon[53732]: t=2021-07-31T01:44:43+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist_item table" 2021-07-31T01:44:43.668 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:43 smithi154 conmon[53732]: t=2021-07-31T01:44:43+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist table v2" 2021-07-31T01:44:43.836 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f52d2c14-f19f-11eb-8c24-001a4aab830c/mon.a/config 2021-07-31T01:44:43.994 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:43 smithi154 conmon[53732]: t=2021-07-31T01:44:43+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist item table v2" 2021-07-31T01:44:43.994 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:43 smithi154 conmon[53732]: t=2021-07-31T01:44:43+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist table charset" 2021-07-31T01:44:43.995 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:43 smithi154 conmon[53732]: t=2021-07-31T01:44:43+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist_item table charset" 2021-07-31T01:44:43.995 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:43 smithi154 conmon[53732]: t=2021-07-31T01:44:43+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v2" 2021-07-31T01:44:43.995 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:43 smithi154 conmon[53732]: t=2021-07-31T01:44:43+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v3" 2021-07-31T01:44:43.995 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:43 smithi154 conmon[53732]: t=2021-07-31T01:44:43+0000 lvl=info msg="Executing migration" logger=migrator id="create preferences table v3" 2021-07-31T01:44:43.996 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:43 smithi154 conmon[53732]: t=2021-07-31T01:44:43+0000 lvl=info msg="Executing migration" logger=migrator id="Update preferences table charset" 2021-07-31T01:44:43.996 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:43 smithi154 conmon[53732]: t=2021-07-31T01:44:43+0000 lvl=info msg="Executing migration" logger=migrator id="Add column team_id in preferences" 2021-07-31T01:44:43.996 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:43 smithi154 conmon[53732]: t=2021-07-31T01:44:43+0000 lvl=info msg="Executing migration" logger=migrator id="Update team_id column values in preferences" 2021-07-31T01:44:43.996 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:43 smithi154 conmon[53732]: t=2021-07-31T01:44:43+0000 lvl=info msg="Executing migration" logger=migrator id="create alert table v1" 2021-07-31T01:44:43.997 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:43 smithi154 conmon[53732]: t=2021-07-31T01:44:43+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert org_id & id " 2021-07-31T01:44:44.252 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:44 smithi154 conmon[53732]: t=2021-07-31T01:44:44+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert state" 2021-07-31T01:44:44.253 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:44 smithi154 conmon[53732]: t=2021-07-31T01:44:44+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert dashboard_id" 2021-07-31T01:44:44.253 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:44 smithi154 conmon[53732]: t=2021-07-31T01:44:44+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v1" 2021-07-31T01:44:44.253 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:44 smithi154 conmon[53732]: t=2021-07-31T01:44:44+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_rule_tag.alert_id_tag_id" 2021-07-31T01:44:44.253 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:44 smithi154 conmon[53732]: t=2021-07-31T01:44:44+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification table v1" 2021-07-31T01:44:44.254 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:44 smithi154 conmon[53732]: t=2021-07-31T01:44:44+0000 lvl=info msg="Executing migration" logger=migrator id="Add column is_default" 2021-07-31T01:44:44.564 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:44 smithi154 conmon[53732]: t=2021-07-31T01:44:44+0000 lvl=info msg="Executing migration" logger=migrator id="Add column frequency" 2021-07-31T01:44:44.564 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:44 smithi154 conmon[53732]: t=2021-07-31T01:44:44+0000 lvl=info msg="Executing migration" logger=migrator id="Add column send_reminder" 2021-07-31T01:44:44.565 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:44 smithi154 conmon[53732]: t=2021-07-31T01:44:44+0000 lvl=info msg="Executing migration" logger=migrator id="Add column disable_resolve_message" 2021-07-31T01:44:44.565 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:44 smithi154 conmon[53732]: t=2021-07-31T01:44:44+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification org_id & name" 2021-07-31T01:44:44.565 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:44 smithi154 conmon[53732]: t=2021-07-31T01:44:44+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert table charset" 2021-07-31T01:44:44.565 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:44 smithi154 conmon[53732]: t=2021-07-31T01:44:44+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert_notification table charset" 2021-07-31T01:44:44.566 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:44 smithi154 conmon[53732]: t=2021-07-31T01:44:44+0000 lvl=info msg="Executing migration" logger=migrator id="create notification_journal table v1" 2021-07-31T01:44:44.566 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:44 smithi154 conmon[53732]: t=2021-07-31T01:44:44+0000 lvl=info msg="Executing migration" logger=migrator id="add index notification_journal org_id & alert_id & notifier_id" 2021-07-31T01:44:44.566 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:44 smithi154 conmon[53732]: t=2021-07-31T01:44:44+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_notification_journal" 2021-07-31T01:44:44.698 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:44 smithi045 conmon[39718]: cluster 2021-07-31T01:44:43.348244+0000 mgr.y (mgr.14146) 218 : cluster [DBG] pgmap v195: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:44:44.698 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:44 smithi045 conmon[39718]: audit 2021-07-31T01:44:43.658822+0000 mon.a (mon.0) 593 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:44:44.698 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:44 smithi045 conmon[39718]: audit 2021-07-31T01:44:44.075468+0000 mon.a (mon.0) 594 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:44:44.699 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:44 smithi045 conmon[33655]: cluster 2021-07-31T01:44:43.348244+0000 mgr.y (mgr.14146) 218 : cluster [DBG] pgmap v195: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:44:44.699 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:44 smithi045 conmon[33655]: audit 2021-07-31T01:44:43.658822+0000 mon.a (mon.0) 593 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:44:44.699 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:44 smithi045 conmon[33655]: audit 2021-07-31T01:44:44.075468+0000 mon.a (mon.0) 594 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:44:44.832 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:44 smithi154 conmon[34553]: cluster 2021-07-31T01:44:43.348244+0000 mgr.y (mgr.14146) 218 : cluster [DBG] 2021-07-31T01:44:44.833 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:44 smithi154 conmon[34553]: pgmap v195: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:44:44.833 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:44 smithi154 conmon[34553]: audit 2021-07-31T01:44:43.658822+0000 mon.a (mon.0) 593 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:44:44.833 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:44 smithi154 conmon[34553]: audit 2021-07-31T01:44:44.075468+0000 mon.a (mon.0) 594 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:44:44.834 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:44 smithi154 conmon[53732]: t=2021-07-31T01:44:44+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification_state table v1" 2021-07-31T01:44:44.834 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:44 smithi154 conmon[53732]: t=2021-07-31T01:44:44+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification_state org_id & alert_id & notifier_id" 2021-07-31T01:44:44.834 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:44 smithi154 conmon[53732]: t=2021-07-31T01:44:44+0000 lvl=info msg="Executing migration" logger=migrator id="Add for to alert table" 2021-07-31T01:44:44.834 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:44 smithi154 conmon[53732]: t=2021-07-31T01:44:44+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in alert_notification" 2021-07-31T01:44:44.835 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:44 smithi154 conmon[53732]: t=2021-07-31T01:44:44+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in alert_notification" 2021-07-31T01:44:44.835 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:44 smithi154 conmon[53732]: t=2021-07-31T01:44:44+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_notification_org_id_uid" 2021-07-31T01:44:44.835 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:44 smithi154 conmon[53732]: t=2021-07-31T01:44:44+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_name" 2021-07-31T01:44:45.084 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:44 smithi154 conmon[53732]: t=2021-07-31T01:44:44+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old annotation table v4" 2021-07-31T01:44:45.084 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:44 smithi154 conmon[53732]: t=2021-07-31T01:44:44+0000 lvl=info msg="Executing migration" logger=migrator id="create annotation table v5" 2021-07-31T01:44:45.084 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:44 smithi154 conmon[53732]: t=2021-07-31T01:44:44+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 0 v3" 2021-07-31T01:44:45.085 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:45 smithi154 conmon[53732]: t=2021-07-31T01:44:45+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 1 v3" 2021-07-31T01:44:45.085 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:45 smithi154 conmon[53732]: t=2021-07-31T01:44:45+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 2 v3" 2021-07-31T01:44:45.085 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:45 smithi154 conmon[53732]: t=2021-07-31T01:44:45+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 3 v3" 2021-07-31T01:44:45.085 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:45 smithi154 conmon[53732]: t=2021-07-31T01:44:45+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 4 v3" 2021-07-31T01:44:45.361 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:45 smithi154 conmon[53732]: t=2021-07-31T01:44:45+0000 lvl=info msg="Executing migration" logger=migrator id="Update annotation table charset" 2021-07-31T01:44:45.361 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:45 smithi154 conmon[53732]: t=2021-07-31T01:44:45+0000 lvl=info msg="Executing migration" logger=migrator id="Add column region_id to annotation table" 2021-07-31T01:44:45.361 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:45 smithi154 conmon[53732]: t=2021-07-31T01:44:45+0000 lvl=info msg="Executing migration" logger=migrator id="Drop category_id index" 2021-07-31T01:44:45.361 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:45 smithi154 conmon[53732]: t=2021-07-31T01:44:45+0000 lvl=info msg="Executing migration" logger=migrator id="Add column tags to annotation table" 2021-07-31T01:44:45.362 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:45 smithi154 conmon[53732]: t=2021-07-31T01:44:45+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v2" 2021-07-31T01:44:45.362 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:45 smithi154 conmon[53732]: t=2021-07-31T01:44:45+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index annotation_tag.annotation_id_tag_id" 2021-07-31T01:44:45.362 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:45 smithi154 conmon[53732]: t=2021-07-31T01:44:45+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert annotations and set TEXT to empty" 2021-07-31T01:44:45.663 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:45 smithi154 conmon[53732]: t=2021-07-31T01:44:45+0000 lvl=info msg="Executing migration" logger=migrator id="Add created time to annotation table" 2021-07-31T01:44:45.663 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:45 smithi154 conmon[53732]: t=2021-07-31T01:44:45+0000 lvl=info msg="Executing migration" logger=migrator id="Add updated time to annotation table" 2021-07-31T01:44:45.664 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:45 smithi154 conmon[53732]: t=2021-07-31T01:44:45+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for created in annotation table" 2021-07-31T01:44:45.664 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:45 smithi154 conmon[53732]: t=2021-07-31T01:44:45+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for updated in annotation table" 2021-07-31T01:44:45.664 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:45 smithi154 conmon[53732]: t=2021-07-31T01:44:45+0000 lvl=info msg="Executing migration" logger=migrator id="Convert existing annotations from seconds to milliseconds" 2021-07-31T01:44:45.664 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:45 smithi154 conmon[53732]: t=2021-07-31T01:44:45+0000 lvl=info msg="Executing migration" logger=migrator id="Add epoch_end column" 2021-07-31T01:44:45.665 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:45 smithi154 conmon[53732]: t=2021-07-31T01:44:45+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for epoch_end" 2021-07-31T01:44:45.665 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:45 smithi154 conmon[53732]: t=2021-07-31T01:44:45+0000 lvl=info msg="Executing migration" logger=migrator id="Make epoch_end the same as epoch" 2021-07-31T01:44:45.665 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:45 smithi154 conmon[53732]: t=2021-07-31T01:44:45+0000 lvl=info msg="Executing migration" logger=migrator id="Move region to single row" 2021-07-31T01:44:45.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:45 smithi045 conmon[33655]: cephadm 2021-07-31T01: 2021-07-31T01:44:45.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:45 smithi045 conmon[33655]: 44:44.077971+0000 mgr.y (mgr.14146) 219 : cephadm [INF] Reconfiguring alertmanager.a (dependencies changed)... 2021-07-31T01:44:45.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:45 smithi045 conmon[33655]: cephadm 2021-07-31T01:44:44.080445+0000 mgr.y (mgr.14146) 220 : cephadm [INF] Reconfiguring daemon alertmanager.a on smithi045 2021-07-31T01:44:45.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:45 smithi045 conmon[39718]: cephadm 2021-07-31T01:44: 2021-07-31T01:44:45.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:45 smithi045 conmon[39718]: 44.077971+0000 mgr.y (mgr.14146) 219 : cephadm [INF] Reconfiguring alertmanager.a (dependencies changed)... 2021-07-31T01:44:45.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:45 smithi045 conmon[39718]: cephadm 2021-07-31T01:44:44.080445+0000 mgr.y (mgr.14146) 220 : cephadm [INF] Reconfiguring daemon alertmanager.a on smithi045 2021-07-31T01:44:45.931 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:45 smithi154 conmon[34553]: cephadm 2021-07-31T01:44:44.077971+0000 mgr.y (mgr.14146) 219 : 2021-07-31T01:44:45.931 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:45 smithi154 conmon[34553]: cephadm [INF] Reconfiguring alertmanager.a (dependencies changed)... 2021-07-31T01:44:45.931 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:45 smithi154 conmon[34553]: cephadm 2021-07-31T01:44:44.080445+0000 mgr.y (mgr.14146) 220 : cephadm [INF] Reconfiguring daemon alertmanager.a on smithi045 2021-07-31T01:44:45.932 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:45 smithi154 conmon[53732]: t=2021-07-31T01:44:45+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch from annotation table" 2021-07-31T01:44:45.932 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:45 smithi154 conmon[53732]: t=2021-07-31T01:44:45+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2021-07-31T01:44:45.932 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:45 smithi154 conmon[53732]: t=2021-07-31T01:44:45+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2021-07-31T01:44:45.932 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:45 smithi154 conmon[53732]: t=2021-07-31T01:44:45+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_epoch_end_epoch on annotation table" 2021-07-31T01:44:45.933 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:45 smithi154 conmon[53732]: t=2021-07-31T01:44:45+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch_epoch_end from annotation table" 2021-07-31T01:44:45.933 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:45 smithi154 conmon[53732]: t=2021-07-31T01:44:45+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for alert_id on annotation table" 2021-07-31T01:44:45.933 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:45 smithi154 conmon[53732]: t=2021-07-31T01:44:45+0000 lvl=info msg="Executing migration" logger=migrator id="create test_data table" 2021-07-31T01:44:45.933 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:45 smithi154 conmon[53732]: t=2021-07-31T01:44:45+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_version table v1" 2021-07-31T01:44:46.191 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:45 smithi154 conmon[53732]: t=2021-07-31T01:44:45+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_version.dashboard_id" 2021-07-31T01:44:46.191 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:46 smithi154 conmon[53732]: t=2021-07-31T01:44:46+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2021-07-31T01:44:46.192 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:46 smithi154 conmon[53732]: t=2021-07-31T01:44:46+0000 lvl=info msg="Executing migration" logger=migrator id="Set dashboard version to 1 where 0" 2021-07-31T01:44:46.192 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:46 smithi154 conmon[53732]: t=2021-07-31T01:44:46+0000 lvl=info msg="Executing migration" logger=migrator id="save existing dashboard data in dashboard_version table v1" 2021-07-31T01:44:46.192 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:46 smithi154 conmon[53732]: t=2021-07-31T01:44:46+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_version.data to mediumtext v1" 2021-07-31T01:44:46.192 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:46 smithi154 conmon[53732]: t=2021-07-31T01:44:46+0000 lvl=info msg="Executing migration" logger=migrator id="create team table" 2021-07-31T01:44:46.193 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:46 smithi154 conmon[53732]: t=2021-07-31T01:44:46+0000 lvl=info msg="Executing migration" logger=migrator id="add index team.org_id" 2021-07-31T01:44:46.193 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:46 smithi154 conmon[53732]: t=2021-07-31T01:44:46+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_org_id_name" 2021-07-31T01:44:46.442 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:46 smithi154 conmon[53732]: t=2021-07-31T01:44:46+0000 lvl=info msg="Executing migration" logger=migrator id="create team member table" 2021-07-31T01:44:46.443 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:46 smithi154 conmon[53732]: t=2021-07-31T01:44:46+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.org_id" 2021-07-31T01:44:46.443 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:46 smithi154 conmon[53732]: t=2021-07-31T01:44:46+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_member_org_id_team_id_user_id" 2021-07-31T01:44:46.443 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:46 smithi154 conmon[53732]: t=2021-07-31T01:44:46+0000 lvl=info msg="Executing migration" logger=migrator id="Add column email to team table" 2021-07-31T01:44:46.443 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:46 smithi154 conmon[53732]: t=2021-07-31T01:44:46+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external to team_member table" 2021-07-31T01:44:46.444 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:46 smithi154 conmon[53732]: t=2021-07-31T01:44:46+0000 lvl=info msg="Executing migration" logger=migrator id="Add column permission to team_member table" 2021-07-31T01:44:46.719 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:46 smithi154 conmon[34553]: cluster 2021-07-31T01:44:45.348601+0000 mgr.y (mgr.14146) 221 : 2021-07-31T01:44:46.720 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:46 smithi154 conmon[34553]: cluster [DBG] pgmap v196: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:44:46.720 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:46 smithi154 conmon[53732]: t=2021-07-31T01:44:46+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard acl table" 2021-07-31T01:44:46.720 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:46 smithi154 conmon[53732]: t=2021-07-31T01:44:46+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_dashboard_id" 2021-07-31T01:44:46.720 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:46 smithi154 conmon[53732]: t=2021-07-31T01:44:46+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_user_id" 2021-07-31T01:44:46.721 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:46 smithi154 conmon[53732]: t=2021-07-31T01:44:46+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_team_id" 2021-07-31T01:44:46.721 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:46 smithi154 conmon[53732]: t=2021-07-31T01:44:46+0000 lvl=info msg="Executing migration" logger=migrator id="save default acl rules in dashboard_acl table" 2021-07-31T01:44:46.721 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:46 smithi154 conmon[53732]: t=2021-07-31T01:44:46+0000 lvl=info msg="Executing migration" logger=migrator id="create tag table" 2021-07-31T01:44:46.721 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:46 smithi154 conmon[53732]: t=2021-07-31T01:44:46+0000 lvl=info msg="Executing migration" logger=migrator id="add index tag.key_value" 2021-07-31T01:44:46.721 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:46 smithi154 conmon[53732]: t=2021-07-31T01:44:46+0000 lvl=info msg="Executing migration" logger=migrator id="create login attempt table" 2021-07-31T01:44:46.909 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:46 smithi045 conmon[33655]: cluster 2021-07-31T01:44:45.348601+0000 mgr.y ( 2021-07-31T01:44:46.909 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:46 smithi045 conmon[33655]: mgr.14146) 221 : cluster [DBG] pgmap v196: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:44:46.910 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:46 smithi045 conmon[39718]: cluster 2021-07-31T01:44 2021-07-31T01:44:46.911 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:46 smithi045 conmon[39718]: :45.348601+0000 mgr.y (mgr.14146) 221 : cluster [DBG] pgmap v196: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:44:47.003 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:46 smithi154 conmon[53732]: t=2021-07-31T01:44:46+0000 lvl=info msg="Executing migration" logger=migrator id="add index login_attempt.username" 2021-07-31T01:44:47.003 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:46 smithi154 conmon[53732]: t=2021-07-31T01:44:46+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_login_attempt_username - v1" 2021-07-31T01:44:47.003 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:46 smithi154 conmon[53732]: t=2021-07-31T01:44:46+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2021-07-31T01:44:47.003 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:46 smithi154 conmon[53732]: t=2021-07-31T01:44:46+0000 lvl=info msg="Executing migration" logger=migrator id="create login_attempt v2" 2021-07-31T01:44:47.004 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:46 smithi154 conmon[53732]: t=2021-07-31T01:44:46+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_login_attempt_username - v2" 2021-07-31T01:44:47.004 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:46 smithi154 conmon[53732]: t=2021-07-31T01:44:46+0000 lvl=info msg="Executing migration" logger=migrator id="copy login_attempt v1 to v2" 2021-07-31T01:44:47.012 INFO:teuthology.orchestra.run.smithi045.stdout:77309411363 2021-07-31T01:44:47.013 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell --fsid f52d2c14-f19f-11eb-8c24-001a4aab830c -- ceph tell osd.3 flush_pg_stats 2021-07-31T01:44:47.273 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:47 smithi154 conmon[53732]: t=2021-07-31T01:44:47+0000 lvl=info msg="Executing migration" logger=migrator id="drop login_attempt_tmp_qwerty" 2021-07-31T01:44:47.273 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:47 smithi154 conmon[53732]: t=2021-07-31T01:44:47+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth table" 2021-07-31T01:44:47.273 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:47 smithi154 conmon[53732]: t=2021-07-31T01:44:47+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_user_auth_auth_module_auth_id - v1" 2021-07-31T01:44:47.274 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:47 smithi154 conmon[53732]: t=2021-07-31T01:44:47+0000 lvl=info msg="Executing migration" logger=migrator id="alter user_auth.auth_id to length 190" 2021-07-31T01:44:47.274 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:47 smithi154 conmon[53732]: t=2021-07-31T01:44:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth access token to user_auth" 2021-07-31T01:44:47.274 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:47 smithi154 conmon[53732]: t=2021-07-31T01:44:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth refresh token to user_auth" 2021-07-31T01:44:47.274 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:47 smithi154 conmon[53732]: t=2021-07-31T01:44:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth token type to user_auth" 2021-07-31T01:44:47.275 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:47 smithi154 conmon[53732]: t=2021-07-31T01:44:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth expiry to user_auth" 2021-07-31T01:44:47.524 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:47 smithi154 conmon[53732]: t=2021-07-31T01:44:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add index to user_id column in user_auth" 2021-07-31T01:44:47.524 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:47 smithi154 conmon[53732]: t=2021-07-31T01:44:47+0000 lvl=info msg="Executing migration" logger=migrator id="create server_lock table" 2021-07-31T01:44:47.524 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:47 smithi154 conmon[53732]: t=2021-07-31T01:44:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index server_lock.operation_uid" 2021-07-31T01:44:47.525 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:47 smithi154 conmon[53732]: t=2021-07-31T01:44:47+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth token table" 2021-07-31T01:44:47.525 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:47 smithi154 conmon[53732]: t=2021-07-31T01:44:47+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.auth_token" 2021-07-31T01:44:47.526 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:47 smithi154 conmon[53732]: t=2021-07-31T01:44:47+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.prev_auth_token" 2021-07-31T01:44:47.690 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f52d2c14-f19f-11eb-8c24-001a4aab830c/mon.a/config 2021-07-31T01:44:47.801 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:47 smithi154 conmon[53732]: t=2021-07-31T01:44:47+0000 lvl=info msg="Executing migration" logger=migrator id="create cache_data table" 2021-07-31T01:44:47.801 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:47 smithi154 conmon[53732]: t=2021-07-31T01:44:47+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index cache_data.cache_key" 2021-07-31T01:44:47.802 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:47 smithi154 conmon[53732]: t=2021-07-31T01:44:47+0000 lvl=info msg="Created default admin" logger=sqlstore user=admin 2021-07-31T01:44:47.802 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:47 smithi154 conmon[53732]: t=2021-07-31T01:44:47+0000 lvl=info msg="Initializing HTTPServer" logger=server 2021-07-31T01:44:47.802 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:47 smithi154 conmon[53732]: t=2021-07-31T01:44:47+0000 lvl=info msg="Initializing BackendPluginManager" logger=server 2021-07-31T01:44:47.802 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:47 smithi154 conmon[53732]: t=2021-07-31T01:44:47+0000 lvl=info msg="Initializing PluginManager" logger=server 2021-07-31T01:44:47.802 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:47 smithi154 conmon[53732]: t=2021-07-31T01:44:47+0000 lvl=info msg="Starting plugin search" logger=plugins 2021-07-31T01:44:47.803 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:47 smithi154 conmon[53732]: t=2021-07-31T01:44:47+0000 lvl=info msg="Registering plugin" logger=plugins name="Pie Chart" 2021-07-31T01:44:47.803 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:47 smithi154 conmon[53732]: t=2021-07-31T01:44:47+0000 lvl=info msg="Registering plugin" logger=plugins name="Status Panel" 2021-07-31T01:44:47.803 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:47 smithi154 conmon[53732]: t=2021-07-31T01:44:47+0000 lvl=info msg="Initializing HooksService" logger=server 2021-07-31T01:44:47.803 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:47 smithi154 conmon[53732]: t=2021-07-31T01:44:47+0000 lvl=info msg="Initializing OSSLicensingService" logger=server 2021-07-31T01:44:47.804 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:47 smithi154 conmon[53732]: t=2021-07-31T01:44:47+0000 lvl=info msg="Initializing InternalMetricsService" logger=server 2021-07-31T01:44:47.804 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:47 smithi154 conmon[53732]: t=2021-07-31T01:44:47+0000 lvl=info msg="Initializing RemoteCache" logger=server 2021-07-31T01:44:47.804 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:47 smithi154 conmon[53732]: t=2021-07-31T01:44:47+0000 lvl=info msg="Initializing RenderingService" logger=server 2021-07-31T01:44:47.804 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:47 smithi154 conmon[53732]: t=2021-07-31T01:44:47+0000 lvl=info msg="Initializing AlertEngine" logger=server 2021-07-31T01:44:47.804 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:47 smithi154 conmon[53732]: t=2021-07-31T01:44:47+0000 lvl=info msg="Initializing QuotaService" logger=server 2021-07-31T01:44:47.805 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:47 smithi154 conmon[53732]: t=2021-07-31T01:44:47+0000 lvl=info msg="Initializing ServerLockService" logger=server 2021-07-31T01:44:47.805 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:47 smithi154 conmon[53732]: t=2021-07-31T01:44:47+0000 lvl=info msg="Initializing UserAuthTokenService" logger=server 2021-07-31T01:44:47.805 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:47 smithi154 conmon[53732]: t=2021-07-31T01:44:47+0000 lvl=info msg="Initializing DatasourceCacheService" logger=server 2021-07-31T01:44:47.805 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:47 smithi154 conmon[53732]: t=2021-07-31T01:44:47+0000 lvl=info msg="Initializing LoginService" logger=server 2021-07-31T01:44:47.806 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:47 smithi154 conmon[53732]: t=2021-07-31T01:44:47+0000 lvl=info msg="Initializing SearchService" logger=server 2021-07-31T01:44:47.806 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:47 smithi154 conmon[53732]: t=2021-07-31T01:44:47+0000 lvl=info msg="Initializing TracingService" logger=server 2021-07-31T01:44:47.806 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:47 smithi154 conmon[53732]: t=2021-07-31T01:44:47+0000 lvl=info msg="Initializing UsageStatsService" logger=server 2021-07-31T01:44:47.806 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:47 smithi154 conmon[53732]: t=2021-07-31T01:44:47+0000 lvl=info msg="Initializing CleanUpService" logger=server 2021-07-31T01:44:47.807 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:47 smithi154 conmon[53732]: t=2021-07-31T01:44:47+0000 lvl=info msg="Initializing NotificationService" logger=server 2021-07-31T01:44:47.807 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:47 smithi154 conmon[53732]: t=2021-07-31T01:44:47+0000 lvl=info msg="Initializing provisioningServiceImpl" logger=server 2021-07-31T01:44:47.807 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:47 smithi154 conmon[53732]: t=2021-07-31T01:44:47+0000 lvl=warn msg="[Deprecated] the datasource provisioning config is outdated. please upgrade" logger=provisioning.datasources filename=/etc/grafana/provisioning/datasources/ceph-dashboard.yml 2021-07-31T01:44:47.807 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:47 smithi154 conmon[53732]: t=2021-07-31T01:44:47+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Dashboard1 2021-07-31T01:44:47.807 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:47 smithi154 conmon[53732]: t=2021-07-31T01:44:47+0000 lvl=info msg="Backend rendering via phantomJS" logger=rendering renderer=phantomJS 2021-07-31T01:44:47.808 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:47 smithi154 conmon[53732]: t=2021-07-31T01:44:47+0000 lvl=warn msg="phantomJS is deprecated and will be removed in a future release. You should consider migrating from phantomJS to grafana-image-renderer plugin. Read more at https://grafana.com/docs/grafana/latest/administration/image_rendering/" logger=rendering renderer=phantomJS 2021-07-31T01:44:48.253 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:47 smithi154 conmon[53732]: t=2021-07-31T01:44:47+0000 lvl=info msg="Initializing Stream Manager" 2021-07-31T01:44:48.253 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Jul 31 01:44:47 smithi154 conmon[53732]: t=2021-07-31T01:44:47+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=https subUrl= socket= 2021-07-31T01:44:48.667 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:44:48 smithi045 systemd[1]: Stopping Ceph alertmanager.a for f52d2c14-f19f-11eb-8c24-001a4aab830c... 2021-07-31T01:44:48.976 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:44:48 smithi045 conmon[57269]: level=info ts=2021-07-31T01:44:48.670Z caller=main.go:536 msg="Received SIGTERM, exiting gracefully..." 2021-07-31T01:44:48.977 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:48 smithi045 conmon[33655]: cluster 2021-07-31T01:44:47.349045+0000 2021-07-31T01:44:48.977 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:48 smithi045 conmon[33655]: mgr.y (mgr.14146) 222 : cluster [DBG] pgmap v197: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:44:48.978 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:48 smithi045 conmon[39718]: cluster 2021-07-31T01:44: 2021-07-31T01:44:48.978 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:48 smithi045 conmon[39718]: 47.349045+0000 mgr.y (mgr.14146) 222 : cluster [DBG] pgmap v197: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:44:49.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:48 smithi154 conmon[34553]: cluster 2021-07-31T01:44:47.349045+0000 mgr.y (mgr.14146) 222 : cluster [DBG] pgmap v197: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:44:49.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:44:49 smithi045 podman[60965]: 2021-07-31 01:44:49.400251163 +0000 UTC m=+0.673944432 container remove 06ec3286016a82b740b2a51234acbf4149d4048e72d0ca2cccf42bd05711b31c (image=docker.io/prom/alertmanager:v0.20.0, name=ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c-alertmanager.a, maintainer=The Prometheus Authors ) 2021-07-31T01:44:49.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:44:49 smithi045 podman[60965]: 2021-07-31 01:44:49.417135648 +0000 UTC m=+0.690828926 volume remove 62c6ace8113ab34184606b54029191d1cadfcda6112bbf139564135daabea533 2021-07-31T01:44:49.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:44:49 smithi045 podman[60924]: ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c-alertmanager.a 2021-07-31T01:44:49.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:44:49 smithi045 systemd[1]: ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c@alertmanager.a.service: Succeeded. 2021-07-31T01:44:49.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:44:49 smithi045 systemd[1]: Stopped Ceph alertmanager.a for f52d2c14-f19f-11eb-8c24-001a4aab830c. 2021-07-31T01:44:49.778 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:44:49 smithi045 systemd[1]: Starting Ceph alertmanager.a for f52d2c14-f19f-11eb-8c24-001a4aab830c... 2021-07-31T01:44:50.276 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:44:50 smithi045 podman[61127]: 2021-07-31 01:44:50.018148392 +0000 UTC m=+0.032667065 image pull docker.io/prom/alertmanager:v0.20.0 2021-07-31T01:44:50.276 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:44:50 smithi045 podman[61127]: 2021-07-31 01:44:50.133397764 +0000 UTC m=+0.147916438 volume create e7af50a5c98169e9e4a9f45bd3b272420cf392c9c2142d79902138569aa9847c 2021-07-31T01:44:50.276 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:44:50 smithi045 podman[61127]: 2021-07-31 01:44:50.166765695 +0000 UTC m=+0.181284370 container create 3c43b4d8c4d32c7d5249accf22326a062bf47ec776df3c8652928d70fb9ac9e3 (image=docker.io/prom/alertmanager:v0.20.0, name=ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c-alertmanager.a, maintainer=The Prometheus Authors ) 2021-07-31T01:44:50.614 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:44:50 smithi045 podman[61127]: 2021-07-31 01:44:50.583344555 +0000 UTC m=+0.597863226 container init 3c43b4d8c4d32c7d5249accf22326a062bf47ec776df3c8652928d70fb9ac9e3 (image=docker.io/prom/alertmanager:v0.20.0, name=ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c-alertmanager.a, maintainer=The Prometheus Authors ) 2021-07-31T01:44:51.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:50 smithi154 conmon[34553]: cluster 2021-07-31T01:44:49.349575+0000 mgr.y (mgr.14146) 223 : cluster [DBG] pgmap v198: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:44:51.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:50 smithi154 conmon[34553]: 2021-07-31T01:44:51.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:50 smithi045 conmon[33655]: cluster 2021-07-31T01: 2021-07-31T01:44:51.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:50 smithi045 conmon[33655]: 44:49.349575+0000 mgr.y (mgr.14146) 223 : cluster [DBG] pgmap v198: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:44:51.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:50 smithi045 conmon[39718]: cluster 2021-07-31T01:44:49 2021-07-31T01:44:51.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:50 smithi045 conmon[39718]: .349575+0000 mgr.y (mgr.14146) 223 : cluster [DBG] pgmap v198: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:44:51.028 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:44:50 smithi045 conmon[61210]: level=info ts=2021-07-31T01:44:50.617Z caller=main.go:231 msg="Starting Alertmanager" version="(version=0.20.0, branch=HEAD, revision=f74be0400a6243d10bb53812d6fa408ad71ff32d)" 2021-07-31T01:44:51.028 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:44:50 smithi045 conmon[61210]: level=info ts=2021-07-31T01:44:50.617Z caller=main.go:232 build_context="(go=go1.13.5, user=root@00c3106655f8, date=20191211-14:13:14)" 2021-07-31T01:44:51.028 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:44:50 smithi045 conmon[61210]: level=info ts=2021-07-31T01:44:50.617Z caller=cluster.go:161 component=cluster msg="setting advertise address explicitly" addr=172.21.15.45 port=9094 2021-07-31T01:44:51.029 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:44:50 smithi045 conmon[61210]: level=info ts=2021-07-31T01:44:50.618Z caller=cluster.go:623 component=cluster msg="Waiting for gossip to settle..." interval=2s 2021-07-31T01:44:51.029 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:44:50 smithi045 conmon[61210]: level=info ts=2021-07-31T01:44:50.637Z caller=coordinator.go:119 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2021-07-31T01:44:51.029 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:44:50 smithi045 conmon[61210]: level=info ts=2021-07-31T01:44:50.638Z caller=coordinator.go:131 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2021-07-31T01:44:51.029 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:44:50 smithi045 conmon[61210]: level=info ts=2021-07-31T01:44:50.641Z caller=main.go:497 msg=Listening address=:9093 2021-07-31T01:44:51.029 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:44:50 smithi045 podman[61127]: 2021-07-31 01:44:50.641668763 +0000 UTC m=+0.656187435 container start 3c43b4d8c4d32c7d5249accf22326a062bf47ec776df3c8652928d70fb9ac9e3 (image=docker.io/prom/alertmanager:v0.20.0, name=ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c-alertmanager.a, maintainer=The Prometheus Authors ) 2021-07-31T01:44:51.030 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:44:50 smithi045 bash[61026]: 3c43b4d8c4d32c7d5249accf22326a062bf47ec776df3c8652928d70fb9ac9e3 2021-07-31T01:44:51.030 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:44:50 smithi045 systemd[1]: Started Ceph alertmanager.a for f52d2c14-f19f-11eb-8c24-001a4aab830c. 2021-07-31T01:44:51.316 INFO:teuthology.orchestra.run.smithi045.stdout:107374182432 2021-07-31T01:44:51.317 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell --fsid f52d2c14-f19f-11eb-8c24-001a4aab830c -- ceph tell osd.4 flush_pg_stats 2021-07-31T01:44:51.766 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f52d2c14-f19f-11eb-8c24-001a4aab830c/mon.a/config 2021-07-31T01:44:52.000 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:51 smithi045 conmon[33655]: audit 2021-07-31T01:44:50.735767+0000 mon.a (mon.0) 595 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:44:52.000 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:51 smithi045 conmon[33655]: cephadm 2021-07-31T01:44:50.736616+0000 mgr.y (mgr.14146) 224 : cephadm [INF] Reconfiguring prometheus.a (dependencies changed)... 2021-07-31T01:44:52.000 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:51 smithi045 conmon[33655]: cephadm 2021-07-31T01:44:50.738788+0000 mgr.y (mgr.14146) 225 : cephadm [INF] Reconfiguring daemon prometheus.a on smithi154 2021-07-31T01:44:52.001 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:51 smithi045 conmon[39718]: audit 2021-07-31T01:44:50.735767+0000 mon.a (mon.0) 595 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:44:52.001 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:51 smithi045 conmon[39718]: cephadm 2021-07-31T01:44:50.736616+0000 mgr.y (mgr.14146) 224 : cephadm [INF] Reconfiguring prometheus.a (dependencies changed)... 2021-07-31T01:44:52.001 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:51 smithi045 conmon[39718]: cephadm 2021-07-31T01:44:50.738788+0000 mgr.y (mgr.14146) 225 : cephadm [INF] Reconfiguring daemon prometheus.a on smithi154 2021-07-31T01:44:52.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:51 smithi154 conmon[34553]: audit 2021-07-31T01:44:50.735767+0000 mon.a (mon.0) 595 : audit [INF] 2021-07-31T01:44:52.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:51 smithi154 conmon[34553]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:44:52.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:51 smithi154 conmon[34553]: cephadm 2021-07-31T01:44:50.736616+0000 mgr.y (mgr.14146) 224 : cephadm [INF] Reconfiguring prometheus.a (dependencies changed)... 2021-07-31T01:44:52.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:51 smithi154 conmon[34553]: cephadm 2021-07-31 2021-07-31T01:44:52.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:51 smithi154 conmon[34553]: T01:44:50.738788+0000 mgr.y (mgr.14146) 225 : cephadm [INF] Reconfiguring daemon prometheus.a on smithi154 2021-07-31T01:44:52.902 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:52 smithi154 conmon[34553]: cluster 2021-07-31T01:44:51.349881+0000 mgr.y (mgr 2021-07-31T01:44:52.902 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:52 smithi154 conmon[34553]: .14146) 226 : cluster [DBG] pgmap v199: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:44:53.016 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:52 smithi045 conmon[33655]: cluster 2021-07-31T01:44:51.349881+0000 mgr.y (mgr.14146 2021-07-31T01:44:53.017 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:52 smithi045 conmon[33655]: ) 226 : cluster [DBG] pgmap v199: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:44:53.017 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:52 smithi045 conmon[39718]: cluster 2021-07-31T01:44:51 2021-07-31T01:44:53.018 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:52 smithi045 conmon[39718]: .349881+0000 mgr.y (mgr.14146) 226 : cluster [DBG] pgmap v199: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:44:53.018 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:44:52 smithi045 conmon[61210]: level=info ts=2021-07-31T01:44:52.619Z caller=cluster.go:648 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000087592s 2021-07-31T01:44:54.049 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Jul 31 01:44:53 smithi154 systemd[1]: Stopping Ceph prometheus.a for f52d2c14-f19f-11eb-8c24-001a4aab830c... 2021-07-31T01:44:54.050 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Jul 31 01:44:53 smithi154 conmon[52084]: level=warn ts=2021-07-31T01:44:53.946Z caller=main.go:524 msg="Received SIGTERM, exiting gracefully..." 2021-07-31T01:44:54.050 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Jul 31 01:44:53 smithi154 conmon[52084]: level=info ts=2021-07-31T01:44:53.947Z caller=main.go:547 msg="Stopping scrape discovery manager..." 2021-07-31T01:44:54.050 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Jul 31 01:44:53 smithi154 conmon[52084]: level=info ts=2021-07-31T01:44:53.947Z caller=main.go:561 msg="Stopping notify discovery manager..." 2021-07-31T01:44:54.050 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Jul 31 01:44:53 smithi154 conmon[52084]: level=info ts=2021-07-31T01:44:53.947Z caller=main.go:583 msg="Stopping scrape manager..." 2021-07-31T01:44:54.051 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Jul 31 01:44:53 smithi154 conmon[52084]: level=info ts=2021-07-31T01:44:53.947Z caller=main.go:543 msg="Scrape discovery manager stopped" 2021-07-31T01:44:54.051 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Jul 31 01:44:53 smithi154 conmon[52084]: level=info ts=2021-07-31T01:44:53.947Z caller=manager.go:882 component="rule manager" msg="Stopping rule manager..." 2021-07-31T01:44:54.052 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Jul 31 01:44:53 smithi154 conmon[52084]: level=info ts=2021-07-31T01:44:53.947Z caller=manager.go:892 component="rule manager" msg="Rule manager stopped" 2021-07-31T01:44:54.052 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Jul 31 01:44:53 smithi154 conmon[52084]: level=info ts=2021-07-31T01:44:53.947Z caller=main.go:557 msg="Notify discovery manager stopped" 2021-07-31T01:44:54.053 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Jul 31 01:44:53 smithi154 conmon[52084]: level=info ts=2021-07-31T01:44:53.947Z caller=main.go:577 msg="Scrape manager stopped" 2021-07-31T01:44:54.053 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Jul 31 01:44:53 smithi154 conmon[52084]: level=info ts=2021-07-31T01:44:53.947Z caller=notifier.go:601 component=notifier msg="Stopping notification manager..." 2021-07-31T01:44:54.053 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Jul 31 01:44:53 smithi154 conmon[52084]: level=info ts=2021-07-31T01:44:53.947Z caller=main.go:749 msg="Notifier manager stopped" 2021-07-31T01:44:54.053 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Jul 31 01:44:53 smithi154 conmon[52084]: level=info ts=2021-07-31T01:44:53.947Z caller=main.go:761 msg="See you next time!" 2021-07-31T01:44:54.374 INFO:teuthology.orchestra.run.smithi045.stdout:128849018907 2021-07-31T01:44:54.375 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell --fsid f52d2c14-f19f-11eb-8c24-001a4aab830c -- ceph tell osd.5 flush_pg_stats 2021-07-31T01:44:54.742 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Jul 31 01:44:54 smithi154 podman[54864]: 2021-07-31 01:44:54.480515235 +0000 UTC m=+0.509963686 container remove 5db25a29728b6bd32fed7793d92a2ae6f1f8bdc8a0be6efb5a96b8f730e4787d (image=docker.io/prom/prometheus:v2.18.1, name=ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c-prometheus.a, maintainer=The Prometheus Authors ) 2021-07-31T01:44:54.743 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Jul 31 01:44:54 smithi154 podman[54822]: ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c-prometheus.a 2021-07-31T01:44:54.743 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Jul 31 01:44:54 smithi154 systemd[1]: ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c@prometheus.a.service: Succeeded. 2021-07-31T01:44:54.743 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Jul 31 01:44:54 smithi154 systemd[1]: Stopped Ceph prometheus.a for f52d2c14-f19f-11eb-8c24-001a4aab830c. 2021-07-31T01:44:54.744 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Jul 31 01:44:54 smithi154 systemd[1]: Starting Ceph prometheus.a for f52d2c14-f19f-11eb-8c24-001a4aab830c... 2021-07-31T01:44:54.744 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:54 smithi045 conmon[39718]: cluster 2021-07-31T01:44:53.350615 2021-07-31T01:44:54.744 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:54 smithi045 conmon[39718]: +0000 mgr.y (mgr.14146) 227 : cluster [DBG] pgmap v200: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:44:54.826 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f52d2c14-f19f-11eb-8c24-001a4aab830c/mon.a/config 2021-07-31T01:44:55.002 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Jul 31 01:44:54 smithi154 podman[54985]: 2021-07-31 01:44:54.72136804 +0000 UTC m=+0.030393367 image pull docker.io/prom/prometheus:v2.18.1 2021-07-31T01:44:55.003 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Jul 31 01:44:54 smithi154 podman[54985]: 2021-07-31 01:44:54.847129165 +0000 UTC m=+0.156154591 container create 535e16fd8ecab1357b374f14bf7cefe4fd53a65d5a7cf08b46834201472a58bf (image=docker.io/prom/prometheus:v2.18.1, name=ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c-prometheus.a, maintainer=The Prometheus Authors ) 2021-07-31T01:44:55.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:54 smithi154 conmon[34553]: cluster 2021-07-31T01:44:53.350615+0000 mgr.y (mgr.14146) 227 : cluster 2021-07-31T01:44:55.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:54 smithi154 conmon[34553]: [DBG] pgmap v200: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:44:55.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:54 smithi045 conmon[33655]: cluster 2021-07-31T01:44:55.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:54 smithi045 conmon[33655]: 2021- 2021-07-31T01:44:55.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:54 smithi045 conmon[33655]: 07-31T01:44:53.350615+0000 mgr.y (mgr.14146) 227 : cluster [DBG] pgmap v200: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:44:55.503 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Jul 31 01:44:55 smithi154 podman[54985]: 2021-07-31 01:44:55.172096335 +0000 UTC m=+0.481121634 container init 535e16fd8ecab1357b374f14bf7cefe4fd53a65d5a7cf08b46834201472a58bf (image=docker.io/prom/prometheus:v2.18.1, name=ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c-prometheus.a, maintainer=The Prometheus Authors ) 2021-07-31T01:44:55.503 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Jul 31 01:44:55 smithi154 conmon[55025]: level=info ts=2021-07-31T01:44:55.218Z caller=main.go:302 msg="No time or size retention was set so using the default time retention" duration=15d 2021-07-31T01:44:55.504 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Jul 31 01:44:55 smithi154 conmon[55025]: level=info ts=2021-07-31T01:44:55.218Z caller=main.go:337 msg="Starting Prometheus" version="(version=2.18.1, branch=HEAD, revision=ecee9c8abfd118f139014cb1b174b08db3f342cf)" 2021-07-31T01:44:55.504 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Jul 31 01:44:55 smithi154 conmon[55025]: level=info ts=2021-07-31T01:44:55.218Z caller=main.go:338 build_context="(go=go1.14.2, user=root@2117a9e64a7e, date=20200507-16:51:47)" 2021-07-31T01:44:55.505 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Jul 31 01:44:55 smithi154 conmon[55025]: level=info ts=2021-07-31T01:44:55.218Z caller=main.go:339 host_details="(Linux 4.18.0-240.1.1.el8_3.x86_64 #1 SMP Fri Oct 16 13:36:46 EDT 2020 x86_64 smithi154 (none))" 2021-07-31T01:44:55.505 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Jul 31 01:44:55 smithi154 conmon[55025]: level=info ts=2021-07-31T01:44:55.218Z caller=main.go:340 fd_limits="(soft=1048576, hard=1048576)" 2021-07-31T01:44:55.505 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Jul 31 01:44:55 smithi154 conmon[55025]: level=info ts=2021-07-31T01:44:55.218Z caller=main.go:341 vm_limits="(soft=unlimited, hard=unlimited)" 2021-07-31T01:44:55.505 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Jul 31 01:44:55 smithi154 conmon[55025]: level=info ts=2021-07-31T01:44:55.220Z caller=main.go:678 msg="Starting TSDB ..." 2021-07-31T01:44:55.506 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Jul 31 01:44:55 smithi154 conmon[55025]: level=info ts=2021-07-31T01:44:55.220Z caller=web.go:523 component=web msg="Start listening for connections" address=:9095 2021-07-31T01:44:55.506 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Jul 31 01:44:55 smithi154 conmon[55025]: level=info ts=2021-07-31T01:44:55.222Z caller=head.go:575 component=tsdb msg="Replaying WAL, this may take awhile" 2021-07-31T01:44:55.506 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Jul 31 01:44:55 smithi154 conmon[55025]: level=info ts=2021-07-31T01:44:55.223Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2021-07-31T01:44:55.506 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Jul 31 01:44:55 smithi154 conmon[55025]: level=info ts=2021-07-31T01:44:55.223Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2021-07-31T01:44:55.507 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Jul 31 01:44:55 smithi154 conmon[55025]: level=info ts=2021-07-31T01:44:55.223Z caller=head.go:627 component=tsdb msg="WAL replay completed" duration=487.054µs 2021-07-31T01:44:55.507 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Jul 31 01:44:55 smithi154 conmon[55025]: level=info ts=2021-07-31T01:44:55.224Z caller=main.go:694 fs_type=XFS_SUPER_MAGIC 2021-07-31T01:44:55.507 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Jul 31 01:44:55 smithi154 conmon[55025]: level=info ts=2021-07-31T01:44:55.224Z caller=main.go:695 msg="TSDB started" 2021-07-31T01:44:55.508 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Jul 31 01:44:55 smithi154 conmon[55025]: level=info ts=2021-07-31T01:44:55.224Z caller=main.go:799 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2021-07-31T01:44:55.508 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Jul 31 01:44:55 smithi154 conmon[55025]: level=info ts=2021-07-31T01:44:55.228Z caller=main.go:827 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml 2021-07-31T01:44:55.508 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Jul 31 01:44:55 smithi154 conmon[55025]: level=info ts=2021-07-31T01:44:55.228Z caller=main.go:646 msg="Server is ready to receive web requests." 2021-07-31T01:44:55.508 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Jul 31 01:44:55 smithi154 podman[54985]: 2021-07-31 01:44:55.247115202 +0000 UTC m=+0.556140638 container start 535e16fd8ecab1357b374f14bf7cefe4fd53a65d5a7cf08b46834201472a58bf (image=docker.io/prom/prometheus:v2.18.1, name=ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c-prometheus.a, maintainer=The Prometheus Authors ) 2021-07-31T01:44:55.508 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Jul 31 01:44:55 smithi154 bash[54911]: 535e16fd8ecab1357b374f14bf7cefe4fd53a65d5a7cf08b46834201472a58bf 2021-07-31T01:44:55.509 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Jul 31 01:44:55 smithi154 systemd[1]: Started Ceph prometheus.a for f52d2c14-f19f-11eb-8c24-001a4aab830c. 2021-07-31T01:44:56.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:56 smithi154 conmon[34553]: audit 2021-07-31T01:44:55.338962+0000 mon.a (mon.0) 596 2021-07-31T01:44:56.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:56 smithi154 conmon[34553]: : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:44:56.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:56 smithi154 conmon[34553]: audit 2021-07-31T01:44:55.341797+0000 mon.a (mon.0) 597 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2021-07-31T01:44:56.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:56 smithi154 conmon[34553]: audit 2021-07-31T01:44:55.342113+0000 mgr.y ( 2021-07-31T01:44:56.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:56 smithi154 conmon[34553]: mgr.14146) 228 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2021-07-31T01:44:56.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:56 smithi154 conmon[34553]: audit 2021 2021-07-31T01:44:56.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:56 smithi154 conmon[34553]: -07-31T01:44:55.342871+0000 mon.a ( 2021-07-31T01:44:56.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:56 smithi154 conmon[34553]: mon.0) 598 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.45:9093"}]: dispatch 2021-07-31T01:44:56.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:56 smithi154 conmon[34553]: audit 2021-07 2021-07-31T01:44:56.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:56 smithi154 conmon[34553]: -31T01:44:55.343117+0000 2021-07-31T01:44:56.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:56 smithi154 conmon[34553]: mgr.y (mgr.14146) 229 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.45:9093"}]: dispatch 2021-07-31T01:44:56.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:56 smithi154 conmon[34553]: 2021-07-31T01:44:56.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:56 smithi154 conmon[34553]: audit 2021-07-31T01:44: 2021-07-31T01:44:56.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:56 smithi154 conmon[34553]: 55.346778+0000 mon.a (mon.0) 599 2021-07-31T01:44:56.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:56 smithi154 conmon[34553]: : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:44:56.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:56 smithi154 conmon[34553]: audit 2021-07- 2021-07-31T01:44:56.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:56 smithi154 conmon[34553]: 31T01:44:55.349624+0000 mon.a (mon.0) 2021-07-31T01:44:56.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:56 smithi154 conmon[34553]: 600 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2021-07-31T01:44:56.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:56 smithi154 conmon[34553]: audit 2021-07-31T 2021-07-31T01:44:56.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:56 smithi154 conmon[34553]: 01:44:55.349943+0000 mgr.y (mgr. 2021-07-31T01:44:56.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:56 smithi154 conmon[34553]: 14146) 230 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2021-07-31T01:44:56.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:56 smithi154 conmon[34553]: cluster 2021- 2021-07-31T01:44:56.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:56 smithi154 conmon[34553]: 07-31T01:44:55.350965+0000 2021-07-31T01:44:56.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:56 smithi154 conmon[34553]: mgr.y (mgr.14146) 231 : cluster [DBG] 2021-07-31T01:44:56.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:56 smithi154 conmon[34553]: pgmap v201: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:44:56.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:56 smithi154 conmon[34553]: audit 2021-07-31T01: 2021-07-31T01:44:56.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:56 smithi154 conmon[34553]: 44:55.353984+0000 mon.a (mon.0) 601 : audit 2021-07-31T01:44:56.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:56 smithi154 conmon[34553]: [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.154:9095"}]: dispatch 2021-07-31T01:44:56.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:56 smithi154 conmon[34553]: audit 2021-07-31T01: 2021-07-31T01:44:56.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:56 smithi154 conmon[34553]: 44:55.354279+0000 mgr.y (mgr.14146) 2021-07-31T01:44:56.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:56 smithi154 conmon[34553]: 232 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.154:9095"}]: dispatch 2021-07-31T01:44:56.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:56 smithi154 conmon[34553]: audit 2021-07- 2021-07-31T01:44:56.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:56 smithi154 conmon[34553]: 31T01:44:55.358097+0000 mon.a 2021-07-31T01:44:56.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:56 smithi154 conmon[34553]: (mon.0) 602 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:44:56.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:56 smithi154 conmon[34553]: audit 2021-07-31T01:44:55. 2021-07-31T01:44:56.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:56 smithi154 conmon[34553]: 361491+0000 mon.a (mon.0) 603 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2021-07-31T01:44:56.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:56 smithi154 conmon[34553]: audit 2021- 2021-07-31T01:44:56.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:56 smithi154 conmon[34553]: 07-31T01:44:55.361794+0000 2021-07-31T01:44:56.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:56 smithi154 conmon[34553]: mgr.y (mgr.14146) 233 : audit [DBG] 2021-07-31T01:44:56.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:56 smithi154 conmon[34553]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2021-07-31T01:44:56.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:56 smithi154 conmon[34553]: audit 2021-07-31T01:44 2021-07-31T01:44:56.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:56 smithi154 conmon[34553]: :55.365835+0000 mon.a (mon.0) 2021-07-31T01:44:56.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:56 smithi154 conmon[34553]: 604 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.154:3000"}]: dispatch 2021-07-31T01:44:56.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:56 smithi154 conmon[34553]: audit 2021-07- 2021-07-31T01:44:56.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:56 smithi154 conmon[34553]: 31T01:44:55.366104+0000 mgr.y 2021-07-31T01:44:56.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:56 smithi154 conmon[34553]: (mgr.14146) 234 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.154:3000"}]: dispatch 2021-07-31T01:44:56.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:56 smithi154 conmon[34553]: audit 2021- 2021-07-31T01:44:56.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:56 smithi154 conmon[34553]: 07-31T01:44:55.370158+0000 mon.a ( 2021-07-31T01:44:56.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:56 smithi154 conmon[34553]: mon.0) 605 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:44:56.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:56 smithi154 conmon[34553]: audit 2021 2021-07-31T01:44:56.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:56 smithi154 conmon[34553]: -07-31T01:44:55.371967 2021-07-31T01:44:56.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:56 smithi154 conmon[34553]: +0000 mon.a (mon.0) 606 : audit 2021-07-31T01:44:56.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:56 smithi154 conmon[34553]: [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:44:56.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:56 smithi154 conmon[34553]: audit 2021-07-31T01:44:55.372920 2021-07-31T01:44:56.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:56 smithi154 conmon[34553]: +0000 mon.a (mon.0) 607 : audit [DBG] 2021-07-31T01:44:56.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:56 smithi154 conmon[34553]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:44:56.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:56 smithi154 conmon[34553]: audit 2021-07-31T01 2021-07-31T01:44:56.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:56 smithi154 conmon[34553]: :44:55.373622+0000 mon.a (mon. 2021-07-31T01:44:56.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:56 smithi154 conmon[34553]: 0) 608 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:44:56.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[33655]: audit 2021-07-31T01:44:55.338962+0000 mon.a (mon.0) 596 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:44:56.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[33655]: audit 2021-07-31T01:44:55.341797+0000 mon.a (mon.0) 597 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2021-07-31T01:44:56.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[33655]: audit 2021-07-31T01:44:55.342113+0000 mgr.y (mgr.14146) 228 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2021-07-31T01:44:56.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[33655]: audit 2021-07-31T01:44:55.342871+0000 mon.a (mon.0) 598 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.45:9093"}]: dispatch 2021-07-31T01:44:56.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[33655]: audit 2021-07-31T01:44:55.343117+0000 mgr.y (mgr.14146) 229 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.45:9093"}]: dispatch 2021-07-31T01:44:56.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[33655]: audit 2021-07-31T01:44:55.346778+0000 mon.a (mon.0) 599 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:44:56.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[33655]: audit 2021-07-31T01:44:55.349624+0000 mon.a (mon.0 2021-07-31T01:44:56.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[33655]: ) 600 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2021-07-31T01:44:56.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[33655]: audit 2021-07-31T01:44:55.349943+0000 mgr.y (mgr.14146) 230 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2021-07-31T01:44:56.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[33655]: cluster 2021-07-31T01:44:55.350965+0000 mgr.y (mgr.14146) 231 : cluster [DBG] pgmap v201: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:44:56.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[33655]: audit 2021-07-31T01:44:55.353984+0000 mon.a (mon.0) 601 : audit [INF] 2021-07-31T01:44:56.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[33655]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.154:9095"}]: dispatch 2021-07-31T01:44:56.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[33655]: audit 2021-07-31T01:44:56.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[33655]: 2021-07-31T01:44 2021-07-31T01:44:56.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[33655]: :55.354279+0000 mgr.y ( 2021-07-31T01:44:56.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[33655]: mgr.14146) 232 : audit 2021-07-31T01:44:56.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[33655]: [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.154:9095"}]: dispatch 2021-07-31T01:44:56.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[33655]: audit 2021-07 2021-07-31T01:44:56.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[33655]: -31T01:44:55.358097+0000 mon.a (mon 2021-07-31T01:44:56.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[33655]: .0) 602 : audit 2021-07-31T01:44:56.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[33655]: [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:44:56.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[33655]: audit 2021- 2021-07-31T01:44:56.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[33655]: 07-31T01:44:55.361491+0000 2021-07-31T01:44:56.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[33655]: mon.a (mon.0) 603 2021-07-31T01:44:56.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[33655]: : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2021-07-31T01:44:56.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[33655]: audit 2021- 2021-07-31T01:44:56.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[33655]: 07-31T01:44:55.361794+0000 2021-07-31T01:44:56.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[33655]: mgr.y (mgr.14146) 233 2021-07-31T01:44:56.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[33655]: : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2021-07-31T01:44:56.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[33655]: 2021-07-31T01:44:56.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[33655]: audit 2021-07-31T01:44:56.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[33655]: 2021-07-31T01 2021-07-31T01:44:56.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[33655]: :44:55.365835+0000 2021-07-31T01:44:56.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[33655]: mon.a (mon.0) 2021-07-31T01:44:56.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[39718]: audit 2021-07-31T01 2021-07-31T01:44:56.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[39718]: :44:55.338962+0000 mon.a (mon.0) 596 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:44:56.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[39718]: audit 2021-07-31T01:44:55.341797+0000 mon.a (mon.0) 597 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2021-07-31T01:44:56.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T01:44:56.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[39718]: 31T01:44:55.342113+0000 2021-07-31T01:44:56.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[39718]: mgr.y (mgr.14146) 228 : audit [DBG] 2021-07-31T01:44:56.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[39718]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2021-07-31T01:44:56.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[39718]: audit 2021-07 2021-07-31T01:44:56.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[39718]: -31T01:44:55.342871 2021-07-31T01:44:56.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[39718]: +0000 mon.a (mon.0) 598 2021-07-31T01:44:56.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[39718]: : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.45:9093"}]: dispatch 2021-07-31T01:44:56.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[39718]: audit 2021 2021-07-31T01:44:56.585 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[39718]: -07-31T01:44:55 2021-07-31T01:44:56.585 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[39718]: .343117+0000 mgr.y (mgr.14146) 229 : 2021-07-31T01:44:56.585 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[39718]: audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.45:9093"}]: dispatch 2021-07-31T01:44:56.585 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T01:44:56.586 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[39718]: T01:44:55.346778+0000 mon.a ( 2021-07-31T01:44:56.586 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[39718]: mon.0) 599 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:44:56.586 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[39718]: audit 2021- 2021-07-31T01:44:56.586 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[39718]: 07-31T01:44:55.349624+0000 2021-07-31T01:44:56.586 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[39718]: mon.a (mon.0) 600 : audit [DBG] 2021-07-31T01:44:56.587 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[39718]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2021-07-31T01:44:56.587 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T01:44:56.587 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[39718]: T01:44:55.349943+0000 2021-07-31T01:44:56.587 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[39718]: mgr.y (mgr.14146) 230 : audit [DBG] 2021-07-31T01:44:56.587 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[39718]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2021-07-31T01:44:56.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[39718]: cluster 2021-07 2021-07-31T01:44:56.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[39718]: -31T01:44:55.350965 2021-07-31T01:44:56.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[39718]: +0000 mgr.y (mgr.14146) 231 : cluster 2021-07-31T01:44:56.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[39718]: [DBG] pgmap v201: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:44:56.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[39718]: audit 2021-07 2021-07-31T01:44:56.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[39718]: -31T01:44:55.353984 2021-07-31T01:44:56.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2021-07-31T01:44:56.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[39718]: 601 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.154:9095"}]: dispatch 2021-07-31T01:44:56.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[39718]: audit 2021-07-31T01: 2021-07-31T01:44:56.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[39718]: 44:55.354279+0000 mgr.y (mgr.14146 2021-07-31T01:44:56.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[39718]: ) 232 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.154:9095"}]: dispatch 2021-07-31T01:44:56.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[39718]: audit 2021 2021-07-31T01:44:56.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[39718]: -07-31T01:44:55.358097 2021-07-31T01:44:56.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[39718]: +0000 mon.a (mon.0) 602 : audit 2021-07-31T01:44:56.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[39718]: [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:44:56.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[39718]: audit 2021-07 2021-07-31T01:44:56.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[39718]: -31T01:44:55.361491+0000 2021-07-31T01:44:56.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[39718]: mon.a (mon.0) 603 : audit 2021-07-31T01:44:56.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[33655]: 604 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.154:3000"}]: dispatch 2021-07-31T01:44:56.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[33655]: audit 2021- 2021-07-31T01:44:56.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[33655]: 07-31T01:44:55.366104+0000 2021-07-31T01:44:56.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[33655]: mgr.y (mgr.14146) 2021-07-31T01:44:56.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[33655]: 234 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.154:3000"}]: dispatch 2021-07-31T01:44:56.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[33655]: 2021-07-31T01:44:56.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[33655]: audit 2021- 2021-07-31T01:44:56.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[33655]: 07-31T01:44:55.370158 2021-07-31T01:44:56.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[33655]: +0000 mon.a (mon. 2021-07-31T01:44:56.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[33655]: 0) 605 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:44:56.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[33655]: 2021-07-31T01:44:56.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[33655]: audit 2021-07-31 2021-07-31T01:44:56.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[33655]: T01:44:55. 2021-07-31T01:44:56.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[33655]: 371967+0000 mon.a (mon 2021-07-31T01:44:56.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[33655]: .0) 606 : audit [DBG] 2021-07-31T01:44:56.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[33655]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:44:56.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[33655]: audit 2021- 2021-07-31T01:44:56.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[33655]: 07-31T01:44 2021-07-31T01:44:56.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[33655]: :55.372920+0000 mon.a 2021-07-31T01:44:56.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[33655]: (mon.0) 607 : 2021-07-31T01:44:56.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[33655]: audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:44:56.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[33655]: audit 2021 2021-07-31T01:44:56.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[33655]: -07-31T01: 2021-07-31T01:44:56.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[33655]: 44:55.373622+0000 2021-07-31T01:44:56.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[33655]: mon.a (mon.0) 608 2021-07-31T01:44:56.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[33655]: : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:44:56.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[39718]: [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2021-07-31T01:44:56.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T01:44:56.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[39718]: 31T01:44:55.361794+0000 mgr.y ( 2021-07-31T01:44:56.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[39718]: mgr.14146) 233 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2021-07-31T01:44:56.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T01:44:56.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[39718]: 01:44:55.365835+0000 mon.a (mon.0 2021-07-31T01:44:56.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[39718]: ) 604 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.154:3000"}]: dispatch 2021-07-31T01:44:56.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[39718]: audit 2021-07-31T01:44:56.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[39718]: 2021-07-31T01:44:55. 2021-07-31T01:44:56.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[39718]: 366104+0000 mgr.y (mgr.14146) 234 : audit 2021-07-31T01:44:56.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[39718]: [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.154:3000"}]: dispatch 2021-07-31T01:44:56.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[39718]: audit 2021- 2021-07-31T01:44:56.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[39718]: 07-31T01:44:55.370158+0000 mon.a (mon 2021-07-31T01:44:56.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[39718]: .0) 605 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:44:56.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[39718]: audit 2021-07-31T01:44:56.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[39718]: 2021-07-31T01:44: 2021-07-31T01:44:56.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[39718]: 55.371967+0000 mon.a (mon.0) 606 : 2021-07-31T01:44:56.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[39718]: audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:44:56.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T01:44:56.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[39718]: 31T01:44:55.372920+0000 mon.a ( 2021-07-31T01:44:56.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[39718]: mon.0) 607 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:44:56.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[39718]: audit 2021-07-31T01:44:56.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[39718]: 2021-07-31T01 2021-07-31T01:44:56.603 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[39718]: :44:55.373622+0000 mon.a (mon.0) 608 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:44:56.603 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:56 smithi045 conmon[39718]: 2021-07-31T01:44:57.544 INFO:teuthology.orchestra.run.smithi045.stdout:154618822679 2021-07-31T01:44:57.544 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell --fsid f52d2c14-f19f-11eb-8c24-001a4aab830c -- ceph tell osd.6 flush_pg_stats 2021-07-31T01:44:58.004 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f52d2c14-f19f-11eb-8c24-001a4aab830c/mon.a/config 2021-07-31T01:44:58.452 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:44:58 smithi045 conmon[33655]: cluster 2021-07-31T01:44:57.351489+0000 mgr.y (mgr.14146) 235 : cluster [DBG] pgmap v202: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:44:58.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:44:58 smithi154 conmon[34553]: cluster 2021-07-31T01:44:57.351489+0000 mgr.y (mgr.14146) 235 : cluster [DBG] pgmap v202: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:44:58.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:44:58 smithi045 conmon[39718]: cluster 2021-07-31T01:44:57.351489+0000 mgr.y (mgr.14146) 235 : cluster [DBG] pgmap v202: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:00.208 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:00 smithi045 conmon[33655]: audit 2021-07-31T01:44:59.111200+0000 mon.a ( 2021-07-31T01:45:00.209 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:00 smithi045 conmon[33655]: mon.0) 609 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:45:00.210 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:00 smithi045 conmon[33655]: cluster 2021-07-31T01:44:59.351886+0000 mgr.y (mgr.14146) 236 : cluster [DBG] pgmap v203: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:00.210 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:00 smithi045 conmon[39718]: audit 2021-07-31T01:44:59.111200+0000 mon.a 2021-07-31T01:45:00.211 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:00 smithi045 conmon[39718]: (mon.0) 609 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:45:00.211 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:00 smithi045 conmon[39718]: cluster 2021-07-31T01:44:59.351886+0000 mgr.y (mgr.14146) 236 : cluster [DBG] pgmap v203: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:00.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:00 smithi154 conmon[34553]: audit 2021-07-31T01:44:59.111200+0000 mon.a (mon.0) 609 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:45:00.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:00 smithi154 conmon[34553]: cluster 2021-07-31T01:44:59.351886+0000 mgr.y (mgr.14146) 236 : cluster [DBG] pgmap v203: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:00.713 INFO:teuthology.orchestra.run.smithi045.stdout:180388626453 2021-07-31T01:45:00.714 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell --fsid f52d2c14-f19f-11eb-8c24-001a4aab830c -- ceph tell osd.7 flush_pg_stats 2021-07-31T01:45:01.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:45:00 smithi045 conmon[61210]: level=info ts=2021-07-31T01:45:00.619Z caller=cluster.go:640 component=cluster msg="gossip settled; proceeding" elapsed=10.0005478s 2021-07-31T01:45:01.214 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f52d2c14-f19f-11eb-8c24-001a4aab830c/mon.a/config 2021-07-31T01:45:02.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:02 smithi045 conmon[33655]: audit 2021-07-31T01:45:01.012121+0000 mon.a (mon.0) 610 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:45:02.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:02 smithi045 conmon[33655]: audit 2021-07-31T01:45:01.015515+0000 mon.a (mon.0) 611 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:45:02.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:02 smithi045 conmon[33655]: audit 2021-07-31T01:45:01.018354+0000 mon.a (mon.0) 612 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:45:02.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:02 smithi045 conmon[33655]: audit 2021-07-31T01:45:01.019270+0000 mon.a (mon.0) 613 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:45:02.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:02 smithi045 conmon[33655]: audit 2021-07-31T01:45:01.019866+0000 mon.a (mon.0) 614 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:45:02.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:02 smithi045 conmon[33655]: cluster 2021-07-31T01:45:01.352272+0000 mgr.y (mgr.14146) 237 : cluster [DBG] pgmap v204: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:02.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:02 smithi045 conmon[39718]: audit 2021-07-31T01:45:01.012121+0000 mon.a (mon.0) 610 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:45:02.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:02 smithi045 conmon[39718]: audit 2021-07-31T01:45:01.015515+0000 mon.a (mon.0) 611 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:45:02.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:02 smithi045 conmon[39718]: audit 2021-07-31T01:45:01.018354+0000 mon.a (mon.0) 612 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:45:02.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:02 smithi045 conmon[39718]: audit 2021-07-31T01:45:01.019270+0000 mon.a (mon.0) 613 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:45:02.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:02 smithi045 conmon[39718]: audit 2021-07-31T01:45:01.019866+0000 mon.a (mon.0) 614 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:45:02.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:02 smithi045 conmon[39718]: cluster 2021-07-31T01:45:01.352272+0000 mgr.y (mgr.14146) 237 : cluster [DBG] pgmap v204: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:02.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:02 smithi154 conmon[34553]: audit 2021-07-31T01:45:01.012121+0000 mon.a (mon.0) 610 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:45:02.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:02 smithi154 conmon[34553]: 2021-07-31T01:45:02.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:02 smithi154 conmon[34553]: audit 2021-07-31T01:45:01.015515+0000 mon.a (mon.0) 611 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:45:02.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:02 smithi154 conmon[34553]: audit 2021-07-31 2021-07-31T01:45:02.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:02 smithi154 conmon[34553]: T01:45:01.018354+0000 2021-07-31T01:45:02.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:02 smithi154 conmon[34553]: mon.a (mon.0) 612 : audit 2021-07-31T01:45:02.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:02 smithi154 conmon[34553]: [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:45:02.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:02 smithi154 conmon[34553]: audit 2021-07- 2021-07-31T01:45:02.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:02 smithi154 conmon[34553]: 31T01:45:01.019270+0000 2021-07-31T01:45:02.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:02 smithi154 conmon[34553]: mon.a (mon.0) 613 : 2021-07-31T01:45:02.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:02 smithi154 conmon[34553]: audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:45:02.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:02 smithi154 conmon[34553]: audit 2021 2021-07-31T01:45:02.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:02 smithi154 conmon[34553]: -07-31T01:45:01. 2021-07-31T01:45:02.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:02 smithi154 conmon[34553]: 019866+0000 mon.a (mon.0 2021-07-31T01:45:02.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:02 smithi154 conmon[34553]: ) 614 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:45:02.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:02 smithi154 conmon[34553]: cluster 2021 2021-07-31T01:45:02.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:02 smithi154 conmon[34553]: -07-31T01:45:01. 2021-07-31T01:45:02.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:02 smithi154 conmon[34553]: 352272+0000 mgr.y (mgr.14146 2021-07-31T01:45:02.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:02 smithi154 conmon[34553]: ) 237 : cluster [DBG] pgmap v204: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:04.031 INFO:teuthology.orchestra.run.smithi045.stdout:206158430227 2021-07-31T01:45:04.031 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell --fsid f52d2c14-f19f-11eb-8c24-001a4aab830c -- ceph osd last-stat-seq osd.0 2021-07-31T01:45:04.485 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f52d2c14-f19f-11eb-8c24-001a4aab830c/mon.a/config 2021-07-31T01:45:04.657 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:04 smithi045 conmon[33655]: cluster 2021-07-31T01:45:03.352617+0000 mgr.y (mgr.14146) 238 : cluster [DBG] pgmap v205: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:04.658 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:04 smithi045 conmon[39718]: cluster 2021-07-31T01:45:03.352617+0000 mgr.y (mgr.14146) 238 : cluster [DBG] pgmap v205: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:04.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:04 smithi154 conmon[34553]: cluster 2021-07-31T01:45:03.352617+0000 mgr.y (mgr.14146) 238 : cluster [DBG] 2021-07-31T01:45:04.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:04 smithi154 conmon[34553]: pgmap v205: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:06.005 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:05 smithi045 conmon[39718]: audit 2021-07-31T01:45:04. 2021-07-31T01:45:06.005 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:05 smithi045 conmon[39718]: 861461+0000 mon.a (mon.0) 615 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:45:06.006 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:05 smithi045 conmon[33655]: audit 2021-07-31T01:45:04.861461+0000 mon.a (mon.0) 615 : audit 2021-07-31T01:45:06.006 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:05 smithi045 conmon[33655]: [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:45:06.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:05 smithi154 conmon[34553]: audit 2021-07-31T01:45:04.861461+0000 mon.a (mon.0) 615 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:45:06.602 INFO:teuthology.orchestra.run.smithi045.stdout:34359738414 2021-07-31T01:45:07.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:06 smithi045 conmon[39718]: cluster 2021-07-31T01:45:05. 2021-07-31T01:45:07.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:06 smithi045 conmon[39718]: 353153+0000 mgr.y (mgr.14146) 239 : cluster [DBG] pgmap v206: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:07.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:06 smithi045 conmon[39718]: audit 2021-07-31T01:45:06.602274+0000 mon.a (mon.0) 616 : audit [DBG] from='client.? 172.21.15.45:0/337216410' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2021-07-31T01:45:07.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:06 smithi045 conmon[39718]: audit 2021-07-31T01:45:06.696843+0000 mon.a (mon.0) 617 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:45:07.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:06 smithi045 conmon[39718]: audit 2021-07-31T01:45:06.700803+0000 mon.a (mon.0) 618 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:45:07.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:06 smithi045 conmon[33655]: cluster 2021-07-31T01:45:05.353153+0000 2021-07-31T01:45:07.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:06 smithi045 conmon[33655]: mgr.y (mgr.14146) 239 : cluster [DBG] pgmap v206: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:07.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:06 smithi045 conmon[33655]: audit 2021-07-31T01:45:06.602274+0000 mon.a (mon.0) 616 : audit [DBG] from='client.? 172.21.15.45:0/337216410' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2021-07-31T01:45:07.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:06 smithi045 conmon[33655]: audit 2021-07-31T01:45:06.696843+0000 mon.a (mon.0) 617 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:45:07.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:06 smithi045 conmon[33655]: audit 2021-07-31T01:45:06.700803+0000 mon.a (mon.0) 618 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:45:07.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:06 smithi154 conmon[34553]: cluster 2021-07-31T01:45:05.353153+0000 mgr.y (mgr.14146) 239 : cluster [DBG] pgmap v206: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:07.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:06 smithi154 conmon[34553]: audit 2021-07-31T01:45:06.602274+0000 mon.a (mon.0) 616 : audit [DBG] from='client.? 172.21.15.45:0/337216410' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2021-07-31T01:45:07.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:06 smithi154 conmon[34553]: audit 2021-07-31T01:45:06.696843+0000 mon.a ( 2021-07-31T01:45:07.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:06 smithi154 conmon[34553]: mon.0) 617 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:45:07.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:06 smithi154 conmon[34553]: audit 2021-07-31T01:45:06. 2021-07-31T01:45:07.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:06 smithi154 conmon[34553]: 700803+0000 mon.a (mon.0) 618 : audit [INF] 2021-07-31T01:45:07.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:06 smithi154 conmon[34553]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:45:07.442 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738408 got 34359738414 for osd.0 2021-07-31T01:45:07.443 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell --fsid f52d2c14-f19f-11eb-8c24-001a4aab830c -- ceph osd last-stat-seq osd.1 2021-07-31T01:45:07.896 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f52d2c14-f19f-11eb-8c24-001a4aab830c/mon.a/config 2021-07-31T01:45:09.147 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:08 smithi045 conmon[33655]: cluster 2021-07-31T01:45:07.353629+0000 mgr.y (mgr.14146) 240 : cluster [DBG] pgmap v207: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:09.148 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:08 smithi045 conmon[39718]: cluster 2021-07-31T01:45:07.353629+0000 mgr.y (mgr.14146) 240 : cluster [DBG] pgmap v207: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:09.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:08 smithi154 conmon[34553]: cluster 2021-07-31T01:45:07.353629+0000 mgr.y (mgr.14146) 240 : cluster [DBG] pgmap v207: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:09.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:08 smithi154 conmon[34553]: 2021-07-31T01:45:09.894 INFO:teuthology.orchestra.run.smithi045.stdout:55834574892 2021-07-31T01:45:10.711 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574886 got 55834574892 for osd.1 2021-07-31T01:45:10.712 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell --fsid f52d2c14-f19f-11eb-8c24-001a4aab830c -- ceph osd last-stat-seq osd.2 2021-07-31T01:45:11.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:10 smithi045 conmon[33655]: cluster 2021-07-31T01:45:09.354104+0000 mgr.y (mgr 2021-07-31T01:45:11.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:10 smithi045 conmon[33655]: .14146) 241 : cluster [DBG] pgmap v208: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:11.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:10 smithi045 conmon[33655]: audit 2021-07-31T01:45:09.893936+0000 mon.a (mon.0) 619 : audit [DBG] from='client.? 172.21.15.45:0/3088895457' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2021-07-31T01:45:11.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:10 smithi045 conmon[39718]: cluster 2021-07-31T01:45:09.354104+0000 mgr.y (mgr.14146) 241 2021-07-31T01:45:11.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:10 smithi045 conmon[39718]: : cluster [DBG] pgmap v208: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:11.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:10 smithi045 conmon[39718]: audit 2021-07-31T01:45:09.893936+0000 mon.a (mon.0) 619 : audit [DBG] from='client.? 172.21.15.45:0/3088895457' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2021-07-31T01:45:11.163 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f52d2c14-f19f-11eb-8c24-001a4aab830c/mon.a/config 2021-07-31T01:45:11.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:10 smithi154 conmon[34553]: cluster 2021-07-31T01:45:09.354104+0000 mgr.y (mgr.14146) 241 : cluster 2021-07-31T01:45:11.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:10 smithi154 conmon[34553]: [DBG] pgmap v208: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:11.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:10 smithi154 conmon[34553]: audit 2021-07-31T01:45:09.893936+0000 mon.a (mon.0) 619 : audit [DBG] from='client.? 172.21.15.45:0/3088895457' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2021-07-31T01:45:13.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:12 smithi045 conmon[33655]: cluster 2021-07-31T01:45:11.354406+0000 mgr.y (mgr.14146) 242 : cluster [DBG] pgmap v209: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:13.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:12 smithi045 conmon[39718]: cluster 2021-07-31T01:45:11.354406+0000 mgr.y (mgr.14146) 242 : cluster [DBG] pgmap v209: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:13.116 INFO:teuthology.orchestra.run.smithi045.stdout:77309411368 2021-07-31T01:45:13.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:12 smithi154 conmon[34553]: cluster 2021-07-31T01:45:11.354406+0000 mgr.y (mgr.14146 2021-07-31T01:45:13.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:12 smithi154 conmon[34553]: ) 242 : cluster [DBG] pgmap v209: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:13.884 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411363 got 77309411368 for osd.2 2021-07-31T01:45:13.885 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell --fsid f52d2c14-f19f-11eb-8c24-001a4aab830c -- ceph osd last-stat-seq osd.3 2021-07-31T01:45:14.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:13 smithi154 conmon[34553]: audit 2021-07-31T01:45:13.116124+0000 mon.a (mon.0) 620 2021-07-31T01:45:14.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:13 smithi154 conmon[34553]: : audit [DBG] from='client.? 172.21.15.45:0/1077012270' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2021-07-31T01:45:14.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:13 smithi045 conmon[33655]: audit 2021-07-31 2021-07-31T01:45:14.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:13 smithi045 conmon[33655]: T01:45:13.116124+0000 mon.a (mon.0) 620 : audit [DBG] from='client.? 172.21.15.45:0/1077012270' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2021-07-31T01:45:14.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:13 smithi045 conmon[39718]: audit 2021-07-31T01:45 2021-07-31T01:45:14.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:13 smithi045 conmon[39718]: :13.116124+0000 mon.a (mon.0) 620 : audit [DBG] from='client.? 172.21.15.45:0/1077012270' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2021-07-31T01:45:14.338 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f52d2c14-f19f-11eb-8c24-001a4aab830c/mon.a/config 2021-07-31T01:45:15.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:14 smithi154 conmon[34553]: cluster 2021-07-31T01:45:13.354796+0000 mgr.y (mgr 2021-07-31T01:45:15.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:14 smithi154 conmon[34553]: .14146) 243 : cluster [DBG] pgmap v210: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:15.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:14 smithi045 conmon[33655]: cluster 2021-07-31T01:45:13.354796+0000 mgr.y (mgr.14146) 243 : cluster [DBG] pgmap v210: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:15.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:14 smithi045 conmon[39718]: cluster 2021-07-31T01:45:13.354796+0000 mgr.y (mgr.14146) 243 : cluster [DBG] pgmap v210: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:16.317 INFO:teuthology.orchestra.run.smithi045.stdout:107374182437 2021-07-31T01:45:17.144 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:16 smithi045 conmon[39718]: cluster 2021-07-31T01:45:15.355169+0000 2021-07-31T01:45:17.145 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:16 smithi045 conmon[39718]: mgr.y (mgr.14146) 244 : cluster [DBG] pgmap v211: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:17.146 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:16 smithi045 conmon[39718]: audit 2021-07-31T01:45:16.316976+0000 mon.c (mon.1) 13 : audit [DBG] from='client.? 172.21.15.45:0/2202882396' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2021-07-31T01:45:17.146 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:16 smithi045 conmon[33655]: cluster 2021-07-31T01:45:15.355169+0000 mgr.y (mgr.14146) 244 : cluster [DBG] 2021-07-31T01:45:17.147 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:16 smithi045 conmon[33655]: pgmap v211: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:17.147 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:16 smithi045 conmon[33655]: audit 2021-07-31T01:45:16.316976+0000 mon.c (mon.1) 13 : audit [DBG] from='client.? 172.21.15.45:0/2202882396' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2021-07-31T01:45:17.148 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182432 got 107374182437 for osd.3 2021-07-31T01:45:17.148 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell --fsid f52d2c14-f19f-11eb-8c24-001a4aab830c -- ceph osd last-stat-seq osd.4 2021-07-31T01:45:17.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:16 smithi154 conmon[34553]: cluster 2021-07-31T01:45:15.355169+0000 mgr.y (mgr.14146) 244 : cluster [DBG] 2021-07-31T01:45:17.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:16 smithi154 conmon[34553]: pgmap v211: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:17.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:16 smithi154 conmon[34553]: audit 2021-07-31T01:45:16.316976+0000 mon.c (mon.1) 13 : audit [DBG] from='client.? 172.21.15.45:0/2202882396' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2021-07-31T01:45:17.598 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f52d2c14-f19f-11eb-8c24-001a4aab830c/mon.a/config 2021-07-31T01:45:19.187 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:18 smithi045 conmon[33655]: cluster 2021-07-31T01:45:17.355447+0000 mgr.y (mgr.14146) 245 : cluster [DBG] pgmap v212: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:19.188 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:18 smithi045 conmon[39718]: cluster 2021-07-31T01:45:17.355447+0000 mgr.y (mgr.14146) 245 : cluster [DBG] pgmap v212: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:19.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:18 smithi154 conmon[34553]: cluster 2021-07-31T01:45:17.355447+0000 mgr.y (mgr.14146) 245 : cluster [DBG] 2021-07-31T01:45:19.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:18 smithi154 conmon[34553]: pgmap v212: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:19.619 INFO:teuthology.orchestra.run.smithi045.stdout:128849018912 2021-07-31T01:45:20.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:19 smithi045 conmon[33655]: audit 2021-07-31T01:45:19.618370+0000 mon.a (mon 2021-07-31T01:45:20.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:19 smithi045 conmon[33655]: .0) 621 : audit [DBG] from='client.? 172.21.15.45:0/1075074545' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2021-07-31T01:45:20.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:19 smithi045 conmon[39718]: audit 2021-07-31T01:45:19.618370+0000 2021-07-31T01:45:20.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:19 smithi045 conmon[39718]: mon.a (mon.0) 621 : audit [DBG] from='client.? 172.21.15.45:0/1075074545' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2021-07-31T01:45:20.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:19 smithi154 conmon[34553]: audit 2021-07-31T01:45:19.618370+0000 mon.a (mon.0) 621 : audit [DBG] from='client.? 172.21.15.45:0/1075074545' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2021-07-31T01:45:20.316 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849018907 got 128849018912 for osd.4 2021-07-31T01:45:20.317 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell --fsid f52d2c14-f19f-11eb-8c24-001a4aab830c -- ceph osd last-stat-seq osd.5 2021-07-31T01:45:20.772 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f52d2c14-f19f-11eb-8c24-001a4aab830c/mon.a/config 2021-07-31T01:45:21.241 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:20 smithi045 conmon[39718]: cluster 2021-07-31T01:45:19 2021-07-31T01:45:21.241 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:20 smithi045 conmon[39718]: .355796+0000 mgr.y (mgr.14146) 246 : cluster [DBG] pgmap v213: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:21.242 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:20 smithi045 conmon[33655]: cluster 2021-07-31T01:45:19.355796 2021-07-31T01:45:21.242 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:20 smithi045 conmon[33655]: +0000 mgr.y (mgr.14146) 246 : cluster [DBG] pgmap v213: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:21.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:20 smithi154 conmon[34553]: cluster 2021-07-31T01:45:19.355796+0000 mgr.y (mgr.14146) 246 : cluster [DBG] pgmap v213: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:22.621 INFO:teuthology.orchestra.run.smithi045.stdout:154618822684 2021-07-31T01:45:22.896 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:22 smithi045 conmon[39718]: cluster 2021-07-31T01:45:21.356116+0000 mgr.y (mgr.14146) 247 : cluster [DBG] pgmap v214: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:22.897 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:22 smithi045 conmon[33655]: cluster 2021-07-31T01:45:21.356116+0000 mgr.y (mgr.14146) 247 : cluster [DBG] pgmap v214: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:22.897 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:22 smithi045 conmon[33655]: audit 2021-07-31T01:45:22.620921+0000 mon.a (mon.0) 622 : audit [DBG] from='client.? 172.21.15.45:0/1220035114' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2021-07-31T01:45:23.212 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:22 smithi045 conmon[39718]: audit 2021-07-31T01:45:22.620921+0000 mon.a (mon.0) 622 : audit [DBG] from='client.? 172.21.15.45:0/1220035114' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2021-07-31T01:45:23.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:22 smithi154 conmon[34553]: cluster 2021-07-31T01:45:21.356116+0000 mgr.y (mgr.14146) 247 : cluster [DBG] pgmap v214: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:23.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:22 smithi154 conmon[34553]: audit 2021-07-31T01: 2021-07-31T01:45:23.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:22 smithi154 conmon[34553]: 45:22.620921+0000 mon.a (mon.0) 622 : audit [DBG] from='client.? 172.21.15.45:0/1220035114' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2021-07-31T01:45:23.280 INFO:tasks.cephadm.ceph_manager.ceph:need seq 154618822679 got 154618822684 for osd.5 2021-07-31T01:45:23.281 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell --fsid f52d2c14-f19f-11eb-8c24-001a4aab830c -- ceph osd last-stat-seq osd.6 2021-07-31T01:45:23.733 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f52d2c14-f19f-11eb-8c24-001a4aab830c/mon.a/config 2021-07-31T01:45:25.220 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:24 smithi045 conmon[33655]: cluster 2021-07-31T01:45:23 2021-07-31T01:45:25.220 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:24 smithi045 conmon[33655]: .356493+0000 mgr.y (mgr.14146) 248 : cluster [DBG] pgmap v215: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:25.221 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:24 smithi045 conmon[39718]: cluster 2021-07-31T01:45:23.356493+0000 2021-07-31T01:45:25.221 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:24 smithi045 conmon[39718]: mgr.y (mgr.14146) 248 : cluster [DBG] pgmap v215: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:25.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:24 smithi154 conmon[34553]: cluster 2021-07-31T01:45:23.356493+0000 mgr.y (mgr.14146) 248 : cluster [DBG] pgmap v215: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:25.664 INFO:teuthology.orchestra.run.smithi045.stdout:180388626458 2021-07-31T01:45:26.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:25 smithi045 conmon[33655]: audit 2021-07-31T01:45:25.664119+0000 mon.a (mon.0) 623 : audit [DBG] from='client.? 172.21.15.45:0/211727666' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2021-07-31T01:45:26.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:25 smithi045 conmon[39718]: audit 2021-07-31T01:45:25.664119+0000 mon.a (mon.0) 623 : audit [DBG] from='client.? 172.21.15.45:0/211727666' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2021-07-31T01:45:26.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:25 smithi154 conmon[34553]: audit 2021-07-31T01:45:25.664119+0000 mon.a (mon.0) 623 : audit [DBG] from='client.? 172.21.15.45:0/211727666' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2021-07-31T01:45:26.451 INFO:tasks.cephadm.ceph_manager.ceph:need seq 180388626453 got 180388626458 for osd.6 2021-07-31T01:45:26.451 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell --fsid f52d2c14-f19f-11eb-8c24-001a4aab830c -- ceph osd last-stat-seq osd.7 2021-07-31T01:45:26.902 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f52d2c14-f19f-11eb-8c24-001a4aab830c/mon.a/config 2021-07-31T01:45:27.168 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:26 smithi045 conmon[39718]: cluster 2021-07-31T01:45:25 2021-07-31T01:45:27.168 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:26 smithi045 conmon[39718]: .356992+0000 mgr.y (mgr.14146) 249 : cluster [DBG] pgmap v216: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:27.168 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:26 smithi045 conmon[33655]: cluster 2021-07-31T01:45:25.356992+0000 mgr.y (mgr. 2021-07-31T01:45:27.169 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:26 smithi045 conmon[33655]: 14146) 249 : cluster [DBG] pgmap v216: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:27.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:26 smithi154 conmon[34553]: cluster 2021-07-31T01:45:25.356992+0000 mgr.y (mgr.14146) 249 : cluster [DBG] pgmap v216: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:28.807 INFO:teuthology.orchestra.run.smithi045.stdout:206158430231 2021-07-31T01:45:29.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:28 smithi154 conmon[34553]: cluster 2021-07-31T01:45:27.357279+0000 mgr.y (mgr.14146) 250 : cluster [DBG] pgmap v217: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:28 smithi154 conmon[34553]: audit 2021-07-31T01:45:28.806370+0000 mon.a (mon.0) 624 : 2021-07-31T01:45:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:28 smithi154 conmon[34553]: audit [DBG] from='client.? 172.21.15.45:0/288704010' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2021-07-31T01:45:29.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:28 smithi045 conmon[33655]: cluster 2021-07-31T01:45:27.357279+0000 mgr.y (mgr.14146) 250 : cluster [DBG] pgmap v217: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:29.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:28 smithi045 conmon[33655]: audit 2021-07-31T01:45:28.806370+0000 mon.a (mon.0) 624 : audit [DBG] from='client.? 172.21.15.45:0/288704010' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2021-07-31T01:45:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:28 smithi045 conmon[39718]: cluster 2021-07-31T01:45:27.357279+0000 mgr.y (mgr.14146) 250 : cluster [DBG] pgmap v217: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:28 smithi045 conmon[39718]: audit 2021-07-31T01:45:28.806370+0000 mon.a (mon.0) 624 : audit [DBG] from='client.? 172.21.15.45:0/288704010' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2021-07-31T01:45:29.528 INFO:tasks.cephadm.ceph_manager.ceph:need seq 206158430227 got 206158430231 for osd.7 2021-07-31T01:45:29.528 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2021-07-31T01:45:29.529 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell --fsid f52d2c14-f19f-11eb-8c24-001a4aab830c -- ceph pg dump --format=json 2021-07-31T01:45:29.980 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f52d2c14-f19f-11eb-8c24-001a4aab830c/mon.a/config 2021-07-31T01:45:31.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:30 smithi154 conmon[34553]: cluster 2021-07-31T01:45:29.357610+0000 mgr.y (mgr.14146) 251 : cluster [DBG] pgmap v218: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:31.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:30 smithi045 conmon[39718]: cluster 2021-07-31T01:45:29.357610+0000 2021-07-31T01:45:31.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:30 smithi045 conmon[39718]: mgr.y (mgr.14146) 251 : cluster [DBG] pgmap v218: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:31.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:30 smithi045 conmon[33655]: cluster 2021-07-31T01:45:29.357610+0000 2021-07-31T01:45:31.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:30 smithi045 conmon[33655]: mgr.y (mgr.14146) 251 : cluster [DBG] pgmap v218: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:31.987 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:45:31.988 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2021-07-31T01:45:32.688 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":219,"stamp":"2021-07-31T01:45:31.357791+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":27,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":43816,"kb_used_data":4008,"kb_used_omap":0,"kb_used_meta":39744,"kb_avail":749917400,"statfs":{"total":767960285184,"available":767915417600,"internally_reserved":0,"allocated":4104192,"data_stored":1658371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40697856},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":52,"state":"active+clean","last_fresh":"2021-07-31T01:44:12.740721+0000","last_change":"2021-07-31T01:44:11.747859+0000","last_active":"2021-07-31T01:44:12.740721+0000","last_peered":"2021-07-31T01:44:12.740721+0000","last_clean":"2021-07-31T01:44:12.740721+0000","last_became_active":"2021-07-31T01:44:11.747575+0000","last_became_peered":"2021-07-31T01:44:11.747575+0000","last_unstale":"2021-07-31T01:44:12.740721+0000","last_undegraded":"2021-07-31T01:44:12.740721+0000","last_fullsized":"2021-07-31T01:44:12.740721+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-07-31T01:44:10.735599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-07-31T01:44:10.735599+0000","last_clean_scrub_stamp":"2021-07-31T01:44:10.735599+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":52,"state":"active+clean","last_fresh":"2021-07-31T01:44:13.286749+0000","last_change":"2021-07-31T01:44:11.748087+0000","last_active":"2021-07-31T01:44:13.286749+0000","last_peered":"2021-07-31T01:44:13.286749+0000","last_clean":"2021-07-31T01:44:13.286749+0000","last_became_active":"2021-07-31T01:44:11.747789+0000","last_became_peered":"2021-07-31T01:44:11.747789+0000","last_unstale":"2021-07-31T01:44:13.286749+0000","last_undegraded":"2021-07-31T01:44:13.286749+0000","last_fullsized":"2021-07-31T01:44:13.286749+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-07-31T01:44:10.735599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-07-31T01:44:10.735599+0000","last_clean_scrub_stamp":"2021-07-31T01:44:10.735599+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":52,"state":"active+clean","last_fresh":"2021-07-31T01:44:13.286619+0000","last_change":"2021-07-31T01:44:12.036816+0000","last_active":"2021-07-31T01:44:13.286619+0000","last_peered":"2021-07-31T01:44:13.286619+0000","last_clean":"2021-07-31T01:44:13.286619+0000","last_became_active":"2021-07-31T01:44:12.036440+0000","last_became_peered":"2021-07-31T01:44:12.036440+0000","last_unstale":"2021-07-31T01:44:13.286619+0000","last_undegraded":"2021-07-31T01:44:13.286619+0000","last_fullsized":"2021-07-31T01:44:13.286619+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-07-31T01:44:10.735599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-07-31T01:44:10.735599+0000","last_clean_scrub_stamp":"2021-07-31T01:44:10.735599+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":52,"state":"active+clean","last_fresh":"2021-07-31T01:44:13.286499+0000","last_change":"2021-07-31T01:44:12.036812+0000","last_active":"2021-07-31T01:44:13.286499+0000","last_peered":"2021-07-31T01:44:13.286499+0000","last_clean":"2021-07-31T01:44:13.286499+0000","last_became_active":"2021-07-31T01:44:12.036370+0000","last_became_peered":"2021-07-31T01:44:12.036370+0000","last_unstale":"2021-07-31T01:44:13.286499+0000","last_undegraded":"2021-07-31T01:44:13.286499+0000","last_fullsized":"2021-07-31T01:44:13.286499+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-07-31T01:44:10.735599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-07-31T01:44:10.735599+0000","last_clean_scrub_stamp":"2021-07-31T01:44:10.735599+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2021-07-31T01:44:13.286484+0000","last_change":"2021-07-31T01:43:48.139877+0000","last_active":"2021-07-31T01:44:13.286484+0000","last_peered":"2021-07-31T01:44:13.286484+0000","last_clean":"2021-07-31T01:44:13.286484+0000","last_became_active":"2021-07-31T01:43:48.139559+0000","last_became_peered":"2021-07-31T01:43:48.139559+0000","last_unstale":"2021-07-31T01:44:13.286484+0000","last_undegraded":"2021-07-31T01:44:13.286484+0000","last_fullsized":"2021-07-31T01:44:13.286484+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-07-31T01:42:08.446854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-07-31T01:42:08.446854+0000","last_clean_scrub_stamp":"2021-07-31T01:42:08.446854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":52,"state":"active+clean","last_fresh":"2021-07-31T01:44:13.286384+0000","last_change":"2021-07-31T01:44:11.747832+0000","last_active":"2021-07-31T01:44:13.286384+0000","last_peered":"2021-07-31T01:44:13.286384+0000","last_clean":"2021-07-31T01:44:13.286384+0000","last_became_active":"2021-07-31T01:44:11.747418+0000","last_became_peered":"2021-07-31T01:44:11.747418+0000","last_unstale":"2021-07-31T01:44:13.286384+0000","last_undegraded":"2021-07-31T01:44:13.286384+0000","last_fullsized":"2021-07-31T01:44:13.286384+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-07-31T01:44:10.735599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-07-31T01:44:10.735599+0000","last_clean_scrub_stamp":"2021-07-31T01:44:10.735599+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":52,"state":"active+clean","last_fresh":"2021-07-31T01:44:13.286359+0000","last_change":"2021-07-31T01:44:12.036886+0000","last_active":"2021-07-31T01:44:13.286359+0000","last_peered":"2021-07-31T01:44:13.286359+0000","last_clean":"2021-07-31T01:44:13.286359+0000","last_became_active":"2021-07-31T01:44:12.036017+0000","last_became_peered":"2021-07-31T01:44:12.036017+0000","last_unstale":"2021-07-31T01:44:13.286359+0000","last_undegraded":"2021-07-31T01:44:13.286359+0000","last_fullsized":"2021-07-31T01:44:13.286359+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-07-31T01:44:10.735599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-07-31T01:44:10.735599+0000","last_clean_scrub_stamp":"2021-07-31T01:44:10.735599+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":52,"state":"active+clean","last_fresh":"2021-07-31T01:44:13.286384+0000","last_change":"2021-07-31T01:44:11.748203+0000","last_active":"2021-07-31T01:44:13.286384+0000","last_peered":"2021-07-31T01:44:13.286384+0000","last_clean":"2021-07-31T01:44:13.286384+0000","last_became_active":"2021-07-31T01:44:11.747897+0000","last_became_peered":"2021-07-31T01:44:11.747897+0000","last_unstale":"2021-07-31T01:44:13.286384+0000","last_undegraded":"2021-07-31T01:44:13.286384+0000","last_fullsized":"2021-07-31T01:44:13.286384+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-07-31T01:44:10.735599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-07-31T01:44:10.735599+0000","last_clean_scrub_stamp":"2021-07-31T01:44:10.735599+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":52,"state":"active+clean","last_fresh":"2021-07-31T01:44:13.286852+0000","last_change":"2021-07-31T01:44:12.036948+0000","last_active":"2021-07-31T01:44:13.286852+0000","last_peered":"2021-07-31T01:44:13.286852+0000","last_clean":"2021-07-31T01:44:13.286852+0000","last_became_active":"2021-07-31T01:44:12.036181+0000","last_became_peered":"2021-07-31T01:44:12.036181+0000","last_unstale":"2021-07-31T01:44:13.286852+0000","last_undegraded":"2021-07-31T01:44:13.286852+0000","last_fullsized":"2021-07-31T01:44:13.286852+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-07-31T01:44:10.735599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-07-31T01:44:10.735599+0000","last_clean_scrub_stamp":"2021-07-31T01:44:10.735599+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":48,"seq":206158430232,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5312,"kb_used_data":504,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":93739840,"statfs":{"total":95995035648,"available":95989596160,"internally_reserved":0,"allocated":516096,"data_stored":207576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 31 01:44:48 2021","interfaces":[{"interface":"back","average":{"1min":1.379,"5min":1.379,"15min":1.379},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":22.087,"5min":22.087,"15min":22.087},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.42999999999999999}]},{"osd":1,"last update":"Sat Jul 31 01:44:48 2021","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.46000000000000002}]},{"osd":2,"last update":"Sat Jul 31 01:44:48 2021","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.47899999999999998}]},{"osd":3,"last update":"Sat Jul 31 01:44:48 2021","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.504},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.63200000000000001}]},{"osd":4,"last update":"Sat Jul 31 01:44:48 2021","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.53100000000000003}]},{"osd":5,"last update":"Sat Jul 31 01:44:48 2021","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.65200000000000002}]},{"osd":6,"last update":"Sat Jul 31 01:44:48 2021","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.64100000000000001}]}]},{"osd":6,"up_from":42,"seq":180388626459,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5376,"kb_used_data":504,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739776,"statfs":{"total":95995035648,"available":95989530624,"internally_reserved":0,"allocated":516096,"data_stored":207576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 31 01:44:28 2021","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.79500000000000004}]},{"osd":1,"last update":"Sat Jul 31 01:44:28 2021","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.95899999999999996}]},{"osd":2,"last update":"Sat Jul 31 01:44:28 2021","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":1.3140000000000001},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":1.73}]},{"osd":3,"last update":"Sat Jul 31 01:44:28 2021","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":1.3939999999999999},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.63600000000000001}]},{"osd":4,"last update":"Sat Jul 31 01:44:28 2021","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.625,"5min":0.625,"15min":0.625},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.52900000000000003}]},{"osd":5,"last update":"Sat Jul 31 01:44:28 2021","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.47899999999999998}]},{"osd":7,"last update":"Sat Jul 31 01:44:50 2021","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.435},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.5}]}]},{"osd":1,"up_from":13,"seq":55834574896,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5560,"kb_used_data":496,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739592,"statfs":{"total":95995035648,"available":95989342208,"internally_reserved":0,"allocated":507904,"data_stored":206946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 31 01:44:49 2021","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.40999999999999998,"15min":0.40999999999999998},"min":{"1min":0.24199999999999999,"5min":0.221,"15min":0.221},"max":{"1min":0.53200000000000003,"5min":2.1859999999999999,"15min":2.1859999999999999},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.41799999999999998,"15min":0.41399999999999998},"min":{"1min":0.26400000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.68300000000000005,"5min":2.2290000000000001,"15min":2.2290000000000001},"last":0.36099999999999999}]},{"osd":2,"last update":"Sat Jul 31 01:45:13 2021","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.504,"15min":0.54400000000000004},"min":{"1min":0.27300000000000002,"5min":0.249,"15min":0.249},"max":{"1min":0.58099999999999996,"5min":3.7719999999999998,"15min":3.7719999999999998},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.505,"15min":0.52500000000000002},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.59699999999999998,"5min":2.5939999999999999,"15min":2.5939999999999999},"last":0.40799999999999997}]},{"osd":3,"last update":"Sat Jul 31 01:44:31 2021","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.48599999999999999,"15min":0.49099999999999999},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.47499999999999998,"15min":0.48299999999999998},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.622,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.40100000000000002}]},{"osd":4,"last update":"Sat Jul 31 01:44:49 2021","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.63400000000000001,"15min":0.64400000000000002},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.77500000000000002,"5min":0.995,"15min":0.995},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.61699999999999999,"15min":0.624},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.83599999999999997,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.70799999999999996}]},{"osd":5,"last update":"Sat Jul 31 01:45:13 2021","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.66200000000000003,"15min":0.66600000000000004},"min":{"1min":0.435,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.94999999999999996,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.66000000000000003,"15min":0.66100000000000003},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.93100000000000005,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.67300000000000004}]},{"osd":6,"last update":"Sat Jul 31 01:44:28 2021","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.0549999999999999,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.71399999999999997}]},{"osd":7,"last update":"Sat Jul 31 01:44:47 2021","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.62},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.66100000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738419,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5632,"kb_used_data":504,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739520,"statfs":{"total":95995035648,"available":95989268480,"internally_reserved":0,"allocated":516096,"data_stored":207576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jul 31 01:44:51 2021","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.38,"15min":0.35799999999999998},"min":{"1min":0.25,"5min":0.222,"15min":0.222},"max":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.31},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.39000000000000001,"15min":0.372},"min":{"1min":0.27400000000000002,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.53400000000000003,"5min":0.57999999999999996,"15min":0.57999999999999996},"last":0.38700000000000001}]},{"osd":2,"last update":"Sat Jul 31 01:45:16 2021","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.41299999999999998,"15min":0.40600000000000003},"min":{"1min":0.26500000000000001,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.44500000000000001,"15min":0.437},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.55400000000000005,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.39400000000000002}]},{"osd":3,"last update":"Sat Jul 31 01:44:34 2021","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.52000000000000002,"15min":0.52200000000000002},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":1.198,"5min":1.198,"15min":1.198},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.503,"15min":0.503},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.34999999999999998}]},{"osd":4,"last update":"Sat Jul 31 01:44:51 2021","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.66700000000000004,"15min":0.67300000000000004},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.0960000000000001,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":1.0860000000000001,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.47299999999999998}]},{"osd":5,"last update":"Sat Jul 31 01:45:12 2021","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.69299999999999995,"15min":0.69899999999999995},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.175,"5min":1.175,"15min":1.175},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.71599999999999997,"15min":0.72999999999999998},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.88900000000000001,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.55200000000000005}]},{"osd":6,"last update":"Sat Jul 31 01:45:28 2021","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.76100000000000001,"15min":0.76000000000000001},"min":{"1min":0.41499999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":3.1800000000000002,"5min":3.1800000000000002,"15min":3.1800000000000002},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.75900000000000001,"15min":0.75800000000000001},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":3.028,"5min":3.028,"15min":3.028},"last":0.57199999999999995}]},{"osd":7,"last update":"Sat Jul 31 01:44:51 2021","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.159,"5min":1.159,"15min":1.159},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.46200000000000002}]}]},{"osd":2,"up_from":18,"seq":77309411372,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5560,"kb_used_data":496,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739592,"statfs":{"total":95995035648,"available":95989342208,"internally_reserved":0,"allocated":507904,"data_stored":206599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 31 01:45:16 2021","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.40999999999999998,"15min":0.40300000000000002},"min":{"1min":0.22800000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.58499999999999996,"5min":1.9810000000000001,"15min":1.9810000000000001},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.379,"5min":0.40699999999999997,"15min":0.40300000000000002},"min":{"1min":0.248,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.63400000000000001,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.30599999999999999}]},{"osd":1,"last update":"Sat Jul 31 01:45:16 2021","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.42199999999999999,"15min":0.41599999999999998},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.752,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.432,"15min":0.42199999999999999},"min":{"1min":0.27100000000000002,"5min":0.249,"15min":0.249},"max":{"1min":0.55200000000000005,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.35099999999999998}]},{"osd":3,"last update":"Sat Jul 31 01:44:31 2021","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.48699999999999999,"15min":0.47899999999999998},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":1.552,"5min":1.552,"15min":1.552},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.49299999999999999,"15min":0.48699999999999999},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.318}]},{"osd":4,"last update":"Sat Jul 31 01:44:47 2021","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.622,"15min":0.622},"min":{"1min":0.432,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.004,"5min":1.004,"15min":1.004},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.64200000000000002,"15min":0.63600000000000001},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.395,"5min":1.395,"15min":1.395},"last":0.66300000000000003}]},{"osd":5,"last update":"Sat Jul 31 01:45:16 2021","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.71999999999999997,"15min":0.73599999999999999},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.093,"5min":1.486,"15min":1.486},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.628,"5min":0.66300000000000003,"15min":0.66900000000000004},"min":{"1min":0.41799999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.224,"5min":1.224,"15min":1.224},"last":0.48299999999999998}]},{"osd":6,"last update":"Sat Jul 31 01:44:31 2021","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.327,"5min":1.327,"15min":1.327},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.633,"5min":1.633,"15min":1.633},"last":0.69999999999999996}]},{"osd":7,"last update":"Sat Jul 31 01:44:47 2021","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.157,"5min":1.157,"15min":1.157},"last":0.53500000000000003}]}]},{"osd":3,"up_from":25,"seq":107374182440,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5504,"kb_used_data":504,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739648,"statfs":{"total":95995035648,"available":95989399552,"internally_reserved":0,"allocated":516096,"data_stored":207576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 31 01:44:28 2021","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.45900000000000002,"15min":0.439},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":1.8879999999999999,"5min":1.8879999999999999,"15min":1.8879999999999999},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.45900000000000002,"15min":0.45300000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":1.083,"5min":1.083,"15min":1.083},"last":0.26300000000000001}]},{"osd":1,"last update":"Sat Jul 31 01:44:28 2021","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.46999999999999997,"15min":0.46200000000000002},"min":{"1min":0.311,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.48099999999999998,"15min":0.47599999999999998},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.22700000000000001}]},{"osd":2,"last update":"Sat Jul 31 01:44:28 2021","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.51800000000000002,"15min":0.50700000000000001},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":1.734,"5min":1.734,"15min":1.734},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.499,"15min":0.48099999999999998},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.786,"5min":1.786,"15min":1.786},"last":0.34000000000000002}]},{"osd":4,"last update":"Sat Jul 31 01:44:50 2021","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.60999999999999999,"15min":0.60599999999999998},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.258,"5min":1.258,"15min":1.258},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.65500000000000003,"15min":0.65400000000000003},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.6739999999999999,"5min":1.6739999999999999,"15min":1.6739999999999999},"last":0.57399999999999995}]},{"osd":5,"last update":"Sat Jul 31 01:45:12 2021","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.69899999999999995,"15min":0.70599999999999996},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.645,"5min":1.645,"15min":1.645},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.68700000000000006,"15min":0.69499999999999995},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.194,"5min":1.194,"15min":1.194},"last":0.58099999999999996}]},{"osd":6,"last update":"Sat Jul 31 01:44:28 2021","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.4530000000000001,"5min":1.4530000000000001,"15min":1.4530000000000001},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.498,"5min":1.498,"15min":1.498},"last":0.59199999999999997}]},{"osd":7,"last update":"Sat Jul 31 01:44:50 2021","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.7130000000000001,"5min":1.7130000000000001,"15min":1.7130000000000001},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.147,"5min":1.147,"15min":1.147},"last":0.63800000000000001}]}]},{"osd":4,"up_from":30,"seq":128849018914,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5440,"kb_used_data":504,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739712,"statfs":{"total":95995035648,"available":95989465088,"internally_reserved":0,"allocated":516096,"data_stored":207576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 31 01:44:50 2021","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.59799999999999998,"15min":0.61199999999999999},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.84999999999999998,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.58999999999999997,"15min":0.59199999999999997},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":1.357,"5min":1.357,"15min":1.357},"last":0.61599999999999999}]},{"osd":1,"last update":"Sat Jul 31 01:44:50 2021","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.59199999999999997,"15min":0.59299999999999997},"min":{"1min":0.38500000000000001,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.431,"5min":1.431,"15min":1.431},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.59699999999999998,"15min":0.60399999999999998},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.331,"5min":1.331,"15min":1.331},"last":0.626}]},{"osd":2,"last update":"Sat Jul 31 01:44:50 2021","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.59999999999999998,"15min":0.60199999999999998},"min":{"1min":0.40600000000000003,"5min":0.38,"15min":0.38},"max":{"1min":1.375,"5min":1.375,"15min":1.375},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.504,"5min":0.58699999999999997,"15min":0.60099999999999998},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.73099999999999998,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.69399999999999995}]},{"osd":3,"last update":"Sat Jul 31 01:44:50 2021","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.60199999999999998,"15min":0.60999999999999999},"min":{"1min":0.40999999999999998,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.79100000000000004,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.60499999999999998,"15min":0.60799999999999998},"min":{"1min":0.41499999999999998,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.389,"5min":1.389,"15min":1.389},"last":0.67600000000000005}]},{"osd":5,"last update":"Sat Jul 31 01:45:09 2021","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.42499999999999999,"15min":0.40799999999999997},"min":{"1min":0.33500000000000002,"5min":0.312,"15min":0.312},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.438,"15min":0.42499999999999999},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.46200000000000002}]},{"osd":6,"last update":"Sat Jul 31 01:44:30 2021","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.56999999999999995}]},{"osd":7,"last update":"Sat Jul 31 01:44:48 2021","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.502}]}]},{"osd":5,"up_from":36,"seq":154618822686,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5432,"kb_used_data":496,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739720,"statfs":{"total":95995035648,"available":95989473280,"internally_reserved":0,"allocated":507904,"data_stored":206946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 31 01:45:08 2021","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.58199999999999996,"15min":0.59199999999999997},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.80700000000000005,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.57199999999999995,"15min":0.57999999999999996},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.73299999999999998,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.53800000000000003}]},{"osd":1,"last update":"Sat Jul 31 01:45:08 2021","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.1890000000000001,"5min":1.1890000000000001,"15min":1.1890000000000001},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.58399999999999996,"15min":0.59299999999999997},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.78000000000000003,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.433}]},{"osd":2,"last update":"Sat Jul 31 01:45:08 2021","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.59899999999999998,"15min":0.60999999999999999},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.59299999999999997,"15min":0.59399999999999997},"min":{"1min":0.46000000000000002,"5min":0.436,"15min":0.436},"max":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.42599999999999999}]},{"osd":3,"last update":"Sat Jul 31 01:45:08 2021","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.60099999999999998,"15min":0.60899999999999999},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.74399999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.57999999999999996,"15min":0.58199999999999996},"min":{"1min":0.39700000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.69899999999999995,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.47199999999999998}]},{"osd":4,"last update":"Sat Jul 31 01:45:08 2021","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.44,"15min":0.436},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.501},{"interface":"front","average":{"1min":0.504,"5min":0.46500000000000002,"15min":0.45900000000000002},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.48799999999999999}]},{"osd":6,"last update":"Sat Jul 31 01:44:29 2021","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.56599999999999995}]},{"osd":7,"last update":"Sat Jul 31 01:44:48 2021","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.61399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-07-31T01:45:32.691 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell --fsid f52d2c14-f19f-11eb-8c24-001a4aab830c -- ceph pg dump --format=json 2021-07-31T01:45:32.914 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:32 smithi045 conmon[39718]: cluster 2021-07-31T01:45:31.357930+0000 mgr.y (mgr.14146) 252 : cluster [DBG] pgmap v219: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:32.915 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:32 smithi045 conmon[39718]: audit 2021-07-31T01:45:31.977460+0000 mgr.y (mgr.14146) 253 : audit [DBG] from='client.24469 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-07-31T01:45:33.140 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f52d2c14-f19f-11eb-8c24-001a4aab830c/mon.a/config 2021-07-31T01:45:33.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:32 smithi154 conmon[34553]: cluster 2021-07-31T01:45:31.357930+0000 mgr.y (mgr.14146) 252 : cluster [DBG] pgmap v219: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:33.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:32 smithi154 conmon[34553]: audit 2021-07-31T01 2021-07-31T01:45:33.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:32 smithi154 conmon[34553]: :45:31.977460+0000 mgr.y (mgr.14146) 253 : audit [DBG] from='client.24469 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-07-31T01:45:33.273 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:32 smithi045 conmon[33655]: cluster 2021-07-31T01:45:31.357930+0000 mgr.y (mgr.14146) 252 : cluster [DBG] pgmap v219: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:33.273 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:32 smithi045 conmon[33655]: audit 2021-07-31T01:45:31.977460+0000 mgr.y (mgr.14146) 253 : audit [DBG] from='client.24469 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-07-31T01:45:35.091 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:45:35.092 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2021-07-31T01:45:35.151 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:34 smithi045 conmon[33655]: cluster 2021-07-31T01:45:33.358535+0000 mgr.y ( 2021-07-31T01:45:35.152 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:34 smithi045 conmon[33655]: mgr.14146) 254 : cluster [DBG] pgmap v220: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:35.152 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:34 smithi045 conmon[39718]: cluster 2021-07-31T01:45:33.358535+0000 2021-07-31T01:45:35.152 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:34 smithi045 conmon[39718]: mgr.y (mgr.14146) 254 : cluster [DBG] pgmap v220: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:35.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:34 smithi154 conmon[34553]: cluster 2021-07-31T01:45:33.358535+0000 mgr.y (mgr.14146) 254 : cluster [DBG] pgmap v220: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:35.860 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":220,"stamp":"2021-07-31T01:45:33.358209+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":27,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":43816,"kb_used_data":4008,"kb_used_omap":0,"kb_used_meta":39744,"kb_avail":749917400,"statfs":{"total":767960285184,"available":767915417600,"internally_reserved":0,"allocated":4104192,"data_stored":1658371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40697856},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":52,"state":"active+clean","last_fresh":"2021-07-31T01:44:12.740721+0000","last_change":"2021-07-31T01:44:11.747859+0000","last_active":"2021-07-31T01:44:12.740721+0000","last_peered":"2021-07-31T01:44:12.740721+0000","last_clean":"2021-07-31T01:44:12.740721+0000","last_became_active":"2021-07-31T01:44:11.747575+0000","last_became_peered":"2021-07-31T01:44:11.747575+0000","last_unstale":"2021-07-31T01:44:12.740721+0000","last_undegraded":"2021-07-31T01:44:12.740721+0000","last_fullsized":"2021-07-31T01:44:12.740721+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-07-31T01:44:10.735599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-07-31T01:44:10.735599+0000","last_clean_scrub_stamp":"2021-07-31T01:44:10.735599+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":52,"state":"active+clean","last_fresh":"2021-07-31T01:44:13.286749+0000","last_change":"2021-07-31T01:44:11.748087+0000","last_active":"2021-07-31T01:44:13.286749+0000","last_peered":"2021-07-31T01:44:13.286749+0000","last_clean":"2021-07-31T01:44:13.286749+0000","last_became_active":"2021-07-31T01:44:11.747789+0000","last_became_peered":"2021-07-31T01:44:11.747789+0000","last_unstale":"2021-07-31T01:44:13.286749+0000","last_undegraded":"2021-07-31T01:44:13.286749+0000","last_fullsized":"2021-07-31T01:44:13.286749+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-07-31T01:44:10.735599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-07-31T01:44:10.735599+0000","last_clean_scrub_stamp":"2021-07-31T01:44:10.735599+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":52,"state":"active+clean","last_fresh":"2021-07-31T01:44:13.286619+0000","last_change":"2021-07-31T01:44:12.036816+0000","last_active":"2021-07-31T01:44:13.286619+0000","last_peered":"2021-07-31T01:44:13.286619+0000","last_clean":"2021-07-31T01:44:13.286619+0000","last_became_active":"2021-07-31T01:44:12.036440+0000","last_became_peered":"2021-07-31T01:44:12.036440+0000","last_unstale":"2021-07-31T01:44:13.286619+0000","last_undegraded":"2021-07-31T01:44:13.286619+0000","last_fullsized":"2021-07-31T01:44:13.286619+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-07-31T01:44:10.735599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-07-31T01:44:10.735599+0000","last_clean_scrub_stamp":"2021-07-31T01:44:10.735599+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":52,"state":"active+clean","last_fresh":"2021-07-31T01:44:13.286499+0000","last_change":"2021-07-31T01:44:12.036812+0000","last_active":"2021-07-31T01:44:13.286499+0000","last_peered":"2021-07-31T01:44:13.286499+0000","last_clean":"2021-07-31T01:44:13.286499+0000","last_became_active":"2021-07-31T01:44:12.036370+0000","last_became_peered":"2021-07-31T01:44:12.036370+0000","last_unstale":"2021-07-31T01:44:13.286499+0000","last_undegraded":"2021-07-31T01:44:13.286499+0000","last_fullsized":"2021-07-31T01:44:13.286499+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-07-31T01:44:10.735599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-07-31T01:44:10.735599+0000","last_clean_scrub_stamp":"2021-07-31T01:44:10.735599+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2021-07-31T01:44:13.286484+0000","last_change":"2021-07-31T01:43:48.139877+0000","last_active":"2021-07-31T01:44:13.286484+0000","last_peered":"2021-07-31T01:44:13.286484+0000","last_clean":"2021-07-31T01:44:13.286484+0000","last_became_active":"2021-07-31T01:43:48.139559+0000","last_became_peered":"2021-07-31T01:43:48.139559+0000","last_unstale":"2021-07-31T01:44:13.286484+0000","last_undegraded":"2021-07-31T01:44:13.286484+0000","last_fullsized":"2021-07-31T01:44:13.286484+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-07-31T01:42:08.446854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-07-31T01:42:08.446854+0000","last_clean_scrub_stamp":"2021-07-31T01:42:08.446854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":52,"state":"active+clean","last_fresh":"2021-07-31T01:44:13.286384+0000","last_change":"2021-07-31T01:44:11.747832+0000","last_active":"2021-07-31T01:44:13.286384+0000","last_peered":"2021-07-31T01:44:13.286384+0000","last_clean":"2021-07-31T01:44:13.286384+0000","last_became_active":"2021-07-31T01:44:11.747418+0000","last_became_peered":"2021-07-31T01:44:11.747418+0000","last_unstale":"2021-07-31T01:44:13.286384+0000","last_undegraded":"2021-07-31T01:44:13.286384+0000","last_fullsized":"2021-07-31T01:44:13.286384+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-07-31T01:44:10.735599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-07-31T01:44:10.735599+0000","last_clean_scrub_stamp":"2021-07-31T01:44:10.735599+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":52,"state":"active+clean","last_fresh":"2021-07-31T01:44:13.286359+0000","last_change":"2021-07-31T01:44:12.036886+0000","last_active":"2021-07-31T01:44:13.286359+0000","last_peered":"2021-07-31T01:44:13.286359+0000","last_clean":"2021-07-31T01:44:13.286359+0000","last_became_active":"2021-07-31T01:44:12.036017+0000","last_became_peered":"2021-07-31T01:44:12.036017+0000","last_unstale":"2021-07-31T01:44:13.286359+0000","last_undegraded":"2021-07-31T01:44:13.286359+0000","last_fullsized":"2021-07-31T01:44:13.286359+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-07-31T01:44:10.735599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-07-31T01:44:10.735599+0000","last_clean_scrub_stamp":"2021-07-31T01:44:10.735599+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":52,"state":"active+clean","last_fresh":"2021-07-31T01:44:13.286384+0000","last_change":"2021-07-31T01:44:11.748203+0000","last_active":"2021-07-31T01:44:13.286384+0000","last_peered":"2021-07-31T01:44:13.286384+0000","last_clean":"2021-07-31T01:44:13.286384+0000","last_became_active":"2021-07-31T01:44:11.747897+0000","last_became_peered":"2021-07-31T01:44:11.747897+0000","last_unstale":"2021-07-31T01:44:13.286384+0000","last_undegraded":"2021-07-31T01:44:13.286384+0000","last_fullsized":"2021-07-31T01:44:13.286384+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-07-31T01:44:10.735599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-07-31T01:44:10.735599+0000","last_clean_scrub_stamp":"2021-07-31T01:44:10.735599+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":52,"state":"active+clean","last_fresh":"2021-07-31T01:44:13.286852+0000","last_change":"2021-07-31T01:44:12.036948+0000","last_active":"2021-07-31T01:44:13.286852+0000","last_peered":"2021-07-31T01:44:13.286852+0000","last_clean":"2021-07-31T01:44:13.286852+0000","last_became_active":"2021-07-31T01:44:12.036181+0000","last_became_peered":"2021-07-31T01:44:12.036181+0000","last_unstale":"2021-07-31T01:44:13.286852+0000","last_undegraded":"2021-07-31T01:44:13.286852+0000","last_fullsized":"2021-07-31T01:44:13.286852+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-07-31T01:44:10.735599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-07-31T01:44:10.735599+0000","last_clean_scrub_stamp":"2021-07-31T01:44:10.735599+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":48,"seq":206158430233,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5312,"kb_used_data":504,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":93739840,"statfs":{"total":95995035648,"available":95989596160,"internally_reserved":0,"allocated":516096,"data_stored":207576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 31 01:44:48 2021","interfaces":[{"interface":"back","average":{"1min":1.379,"5min":1.379,"15min":1.379},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":22.087,"5min":22.087,"15min":22.087},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.44}]},{"osd":1,"last update":"Sat Jul 31 01:44:48 2021","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.498},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.47799999999999998}]},{"osd":2,"last update":"Sat Jul 31 01:44:48 2021","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.55900000000000005}]},{"osd":3,"last update":"Sat Jul 31 01:44:48 2021","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.61599999999999999}]},{"osd":4,"last update":"Sat Jul 31 01:44:48 2021","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.627}]},{"osd":5,"last update":"Sat Jul 31 01:44:48 2021","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.67300000000000004}]},{"osd":6,"last update":"Sat Jul 31 01:44:48 2021","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.68700000000000006}]}]},{"osd":6,"up_from":42,"seq":180388626460,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5376,"kb_used_data":504,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739776,"statfs":{"total":95995035648,"available":95989530624,"internally_reserved":0,"allocated":516096,"data_stored":207576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 31 01:45:32 2021","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.55200000000000005,"15min":0.55600000000000005},"min":{"1min":0.32900000000000001,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.374},{"interface":"front","average":{"1min":0.505,"5min":0.55700000000000005,"15min":0.56599999999999995},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.79500000000000004,"5min":0.998,"15min":0.998},"last":0.505}]},{"osd":1,"last update":"Sat Jul 31 01:45:32 2021","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.54600000000000004,"15min":0.54800000000000004},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.434}]},{"osd":2,"last update":"Sat Jul 31 01:45:32 2021","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.53900000000000003,"15min":0.53700000000000003},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":1.3140000000000001,"5min":1.3140000000000001,"15min":1.3140000000000001},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.56599999999999995,"15min":0.56200000000000006},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":1.73,"5min":1.73,"15min":1.73},"last":0.45300000000000001}]},{"osd":3,"last update":"Sat Jul 31 01:45:32 2021","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.54900000000000004,"15min":0.54200000000000004},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.3939999999999999,"5min":1.3939999999999999,"15min":1.3939999999999999},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.56299999999999994,"15min":0.56599999999999995},"min":{"1min":0.39000000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.47499999999999998}]},{"osd":4,"last update":"Sat Jul 31 01:45:32 2021","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"min":{"1min":0.26500000000000001,"5min":0.253,"15min":0.253},"max":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.49199999999999999,"15min":0.48599999999999999},"min":{"1min":0.35299999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.40899999999999997}]},{"osd":5,"last update":"Sat Jul 31 01:45:32 2021","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.48699999999999999,"15min":0.48499999999999999},"min":{"1min":0.316,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.66500000000000004,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.51700000000000002,"15min":0.51400000000000001},"min":{"1min":0.35299999999999998,"5min":0.309,"15min":0.309},"max":{"1min":0.73999999999999999,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.61299999999999999}]},{"osd":7,"last update":"Sat Jul 31 01:44:50 2021","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.52600000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574896,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5560,"kb_used_data":496,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739592,"statfs":{"total":95995035648,"available":95989342208,"internally_reserved":0,"allocated":507904,"data_stored":206946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 31 01:44:49 2021","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.40999999999999998,"15min":0.40999999999999998},"min":{"1min":0.24199999999999999,"5min":0.221,"15min":0.221},"max":{"1min":0.53200000000000003,"5min":2.1859999999999999,"15min":2.1859999999999999},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.41799999999999998,"15min":0.41399999999999998},"min":{"1min":0.26400000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.68300000000000005,"5min":2.2290000000000001,"15min":2.2290000000000001},"last":0.36099999999999999}]},{"osd":2,"last update":"Sat Jul 31 01:45:13 2021","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.504,"15min":0.54400000000000004},"min":{"1min":0.27300000000000002,"5min":0.249,"15min":0.249},"max":{"1min":0.58099999999999996,"5min":3.7719999999999998,"15min":3.7719999999999998},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.505,"15min":0.52500000000000002},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.59699999999999998,"5min":2.5939999999999999,"15min":2.5939999999999999},"last":0.40799999999999997}]},{"osd":3,"last update":"Sat Jul 31 01:44:31 2021","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.48599999999999999,"15min":0.49099999999999999},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.47499999999999998,"15min":0.48299999999999998},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.622,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.40100000000000002}]},{"osd":4,"last update":"Sat Jul 31 01:44:49 2021","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.63400000000000001,"15min":0.64400000000000002},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.77500000000000002,"5min":0.995,"15min":0.995},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.61699999999999999,"15min":0.624},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.83599999999999997,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.70799999999999996}]},{"osd":5,"last update":"Sat Jul 31 01:45:13 2021","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.66200000000000003,"15min":0.66600000000000004},"min":{"1min":0.435,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.94999999999999996,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.66000000000000003,"15min":0.66100000000000003},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.93100000000000005,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.67300000000000004}]},{"osd":6,"last update":"Sat Jul 31 01:44:28 2021","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.0549999999999999,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.71399999999999997}]},{"osd":7,"last update":"Sat Jul 31 01:44:47 2021","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.62},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.66100000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738419,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5632,"kb_used_data":504,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739520,"statfs":{"total":95995035648,"available":95989268480,"internally_reserved":0,"allocated":516096,"data_stored":207576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jul 31 01:44:51 2021","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.38,"15min":0.35799999999999998},"min":{"1min":0.25,"5min":0.222,"15min":0.222},"max":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.31},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.39000000000000001,"15min":0.372},"min":{"1min":0.27400000000000002,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.53400000000000003,"5min":0.57999999999999996,"15min":0.57999999999999996},"last":0.38700000000000001}]},{"osd":2,"last update":"Sat Jul 31 01:45:16 2021","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.41299999999999998,"15min":0.40600000000000003},"min":{"1min":0.26500000000000001,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.44500000000000001,"15min":0.437},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.55400000000000005,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.39400000000000002}]},{"osd":3,"last update":"Sat Jul 31 01:44:34 2021","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.52000000000000002,"15min":0.52200000000000002},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":1.198,"5min":1.198,"15min":1.198},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.503,"15min":0.503},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.34999999999999998}]},{"osd":4,"last update":"Sat Jul 31 01:44:51 2021","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.66700000000000004,"15min":0.67300000000000004},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.0960000000000001,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":1.0860000000000001,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.47299999999999998}]},{"osd":5,"last update":"Sat Jul 31 01:45:12 2021","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.69299999999999995,"15min":0.69899999999999995},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.175,"5min":1.175,"15min":1.175},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.71599999999999997,"15min":0.72999999999999998},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.88900000000000001,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.55200000000000005}]},{"osd":6,"last update":"Sat Jul 31 01:45:28 2021","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.76100000000000001,"15min":0.76000000000000001},"min":{"1min":0.41499999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":3.1800000000000002,"5min":3.1800000000000002,"15min":3.1800000000000002},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.75900000000000001,"15min":0.75800000000000001},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":3.028,"5min":3.028,"15min":3.028},"last":0.57199999999999995}]},{"osd":7,"last update":"Sat Jul 31 01:44:51 2021","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.159,"5min":1.159,"15min":1.159},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.46200000000000002}]}]},{"osd":2,"up_from":18,"seq":77309411372,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5560,"kb_used_data":496,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739592,"statfs":{"total":95995035648,"available":95989342208,"internally_reserved":0,"allocated":507904,"data_stored":206599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 31 01:45:16 2021","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.40999999999999998,"15min":0.40300000000000002},"min":{"1min":0.22800000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.58499999999999996,"5min":1.9810000000000001,"15min":1.9810000000000001},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.379,"5min":0.40699999999999997,"15min":0.40300000000000002},"min":{"1min":0.248,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.63400000000000001,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.30599999999999999}]},{"osd":1,"last update":"Sat Jul 31 01:45:16 2021","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.42199999999999999,"15min":0.41599999999999998},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.752,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.432,"15min":0.42199999999999999},"min":{"1min":0.27100000000000002,"5min":0.249,"15min":0.249},"max":{"1min":0.55200000000000005,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.35099999999999998}]},{"osd":3,"last update":"Sat Jul 31 01:44:31 2021","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.48699999999999999,"15min":0.47899999999999998},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":1.552,"5min":1.552,"15min":1.552},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.49299999999999999,"15min":0.48699999999999999},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.318}]},{"osd":4,"last update":"Sat Jul 31 01:44:47 2021","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.622,"15min":0.622},"min":{"1min":0.432,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.004,"5min":1.004,"15min":1.004},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.64200000000000002,"15min":0.63600000000000001},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.395,"5min":1.395,"15min":1.395},"last":0.66300000000000003}]},{"osd":5,"last update":"Sat Jul 31 01:45:16 2021","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.71999999999999997,"15min":0.73599999999999999},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.093,"5min":1.486,"15min":1.486},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.628,"5min":0.66300000000000003,"15min":0.66900000000000004},"min":{"1min":0.41799999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.224,"5min":1.224,"15min":1.224},"last":0.48299999999999998}]},{"osd":6,"last update":"Sat Jul 31 01:44:31 2021","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.327,"5min":1.327,"15min":1.327},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.633,"5min":1.633,"15min":1.633},"last":0.69999999999999996}]},{"osd":7,"last update":"Sat Jul 31 01:44:47 2021","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.157,"5min":1.157,"15min":1.157},"last":0.53500000000000003}]}]},{"osd":3,"up_from":25,"seq":107374182440,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5504,"kb_used_data":504,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739648,"statfs":{"total":95995035648,"available":95989399552,"internally_reserved":0,"allocated":516096,"data_stored":207576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 31 01:44:28 2021","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.45900000000000002,"15min":0.439},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":1.8879999999999999,"5min":1.8879999999999999,"15min":1.8879999999999999},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.45900000000000002,"15min":0.45300000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":1.083,"5min":1.083,"15min":1.083},"last":0.26300000000000001}]},{"osd":1,"last update":"Sat Jul 31 01:44:28 2021","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.46999999999999997,"15min":0.46200000000000002},"min":{"1min":0.311,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.48099999999999998,"15min":0.47599999999999998},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.22700000000000001}]},{"osd":2,"last update":"Sat Jul 31 01:44:28 2021","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.51800000000000002,"15min":0.50700000000000001},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":1.734,"5min":1.734,"15min":1.734},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.499,"15min":0.48099999999999998},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.786,"5min":1.786,"15min":1.786},"last":0.34000000000000002}]},{"osd":4,"last update":"Sat Jul 31 01:44:50 2021","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.60999999999999999,"15min":0.60599999999999998},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.258,"5min":1.258,"15min":1.258},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.65500000000000003,"15min":0.65400000000000003},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.6739999999999999,"5min":1.6739999999999999,"15min":1.6739999999999999},"last":0.57399999999999995}]},{"osd":5,"last update":"Sat Jul 31 01:45:12 2021","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.69899999999999995,"15min":0.70599999999999996},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.645,"5min":1.645,"15min":1.645},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.68700000000000006,"15min":0.69499999999999995},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.194,"5min":1.194,"15min":1.194},"last":0.58099999999999996}]},{"osd":6,"last update":"Sat Jul 31 01:44:28 2021","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.4530000000000001,"5min":1.4530000000000001,"15min":1.4530000000000001},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.498,"5min":1.498,"15min":1.498},"last":0.59199999999999997}]},{"osd":7,"last update":"Sat Jul 31 01:44:50 2021","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.7130000000000001,"5min":1.7130000000000001,"15min":1.7130000000000001},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.147,"5min":1.147,"15min":1.147},"last":0.63800000000000001}]}]},{"osd":4,"up_from":30,"seq":128849018915,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5440,"kb_used_data":504,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739712,"statfs":{"total":95995035648,"available":95989465088,"internally_reserved":0,"allocated":516096,"data_stored":207576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 31 01:44:50 2021","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.59799999999999998,"15min":0.61199999999999999},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.84999999999999998,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.58999999999999997,"15min":0.59199999999999997},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":1.357,"5min":1.357,"15min":1.357},"last":0.67000000000000004}]},{"osd":1,"last update":"Sat Jul 31 01:44:50 2021","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.59199999999999997,"15min":0.59299999999999997},"min":{"1min":0.38500000000000001,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.431,"5min":1.431,"15min":1.431},"last":0.621},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.59699999999999998,"15min":0.60399999999999998},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.331,"5min":1.331,"15min":1.331},"last":0.51700000000000002}]},{"osd":2,"last update":"Sat Jul 31 01:44:50 2021","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.59999999999999998,"15min":0.60199999999999998},"min":{"1min":0.40600000000000003,"5min":0.38,"15min":0.38},"max":{"1min":1.375,"5min":1.375,"15min":1.375},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.504,"5min":0.58699999999999997,"15min":0.60099999999999998},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.73099999999999998,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.60099999999999998}]},{"osd":3,"last update":"Sat Jul 31 01:44:50 2021","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.60199999999999998,"15min":0.60999999999999999},"min":{"1min":0.40999999999999998,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.79100000000000004,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.60499999999999998,"15min":0.60799999999999998},"min":{"1min":0.41499999999999998,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.389,"5min":1.389,"15min":1.389},"last":0.58199999999999996}]},{"osd":5,"last update":"Sat Jul 31 01:45:09 2021","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.42499999999999999,"15min":0.40799999999999997},"min":{"1min":0.33500000000000002,"5min":0.312,"15min":0.312},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.438,"15min":0.42499999999999999},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.48399999999999999}]},{"osd":6,"last update":"Sat Jul 31 01:45:31 2021","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.45800000000000002,"15min":0.44900000000000001},"min":{"1min":0.38800000000000001,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.54100000000000004,"15min":0.52900000000000003},"min":{"1min":0.432,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.78900000000000003,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.69199999999999995}]},{"osd":7,"last update":"Sat Jul 31 01:44:48 2021","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.53200000000000003}]}]},{"osd":5,"up_from":36,"seq":154618822687,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5432,"kb_used_data":496,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739720,"statfs":{"total":95995035648,"available":95989473280,"internally_reserved":0,"allocated":507904,"data_stored":206946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 31 01:45:08 2021","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.58199999999999996,"15min":0.59199999999999997},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.80700000000000005,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.57199999999999995,"15min":0.57999999999999996},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.73299999999999998,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.71599999999999997}]},{"osd":1,"last update":"Sat Jul 31 01:45:08 2021","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.1890000000000001,"5min":1.1890000000000001,"15min":1.1890000000000001},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.58399999999999996,"15min":0.59299999999999997},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.78000000000000003,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.59999999999999998}]},{"osd":2,"last update":"Sat Jul 31 01:45:08 2021","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.59899999999999998,"15min":0.60999999999999999},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.59299999999999997,"15min":0.59399999999999997},"min":{"1min":0.46000000000000002,"5min":0.436,"15min":0.436},"max":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.61599999999999999}]},{"osd":3,"last update":"Sat Jul 31 01:45:08 2021","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.60099999999999998,"15min":0.60899999999999999},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.74399999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.57999999999999996,"15min":0.58199999999999996},"min":{"1min":0.39700000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.69899999999999995,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.58699999999999997}]},{"osd":4,"last update":"Sat Jul 31 01:45:08 2021","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.44,"15min":0.436},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.504,"5min":0.46500000000000002,"15min":0.45900000000000002},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.53700000000000003}]},{"osd":6,"last update":"Sat Jul 31 01:45:30 2021","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.48199999999999998,"15min":0.46600000000000003},"min":{"1min":0.32500000000000001,"5min":0.314,"15min":0.314},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.53800000000000003,"15min":0.52800000000000002},"min":{"1min":0.40799999999999997,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.76100000000000001}]},{"osd":7,"last update":"Sat Jul 31 01:44:48 2021","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.55700000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-07-31T01:45:35.862 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2021-07-31T01:45:35.863 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2021-07-31T01:45:35.863 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2021-07-31T01:45:35.863 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell --fsid f52d2c14-f19f-11eb-8c24-001a4aab830c -- ceph health --format=json 2021-07-31T01:45:36.313 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f52d2c14-f19f-11eb-8c24-001a4aab830c/mon.a/config 2021-07-31T01:45:37.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:36 smithi154 conmon[34553]: audit 2021-07-31T01:45:35.087690+0000 mgr.y (mgr.14146) 255 : audit [DBG] from='client.14568 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-07-31T01:45:37.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:36 smithi154 conmon[34553]: cluster 2021-07-31T01:45:35.358942+0000 2021-07-31T01:45:37.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:36 smithi154 conmon[34553]: mgr.y (mgr.14146) 256 : cluster [DBG] pgmap v221: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:37.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:36 smithi045 conmon[33655]: audit 2021-07-31T01:45:35.087690+0000 mgr.y (mgr.14146) 255 : audit [DBG] from='client.14568 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-07-31T01:45:37.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:36 smithi045 conmon[33655]: cluster 2021-07-31T01:45:35.358942+0000 mgr.y (mgr.14146) 256 : cluster [DBG] pgmap v221: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:37.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:36 smithi045 conmon[39718]: audit 2021-07-31T01:45:35.087690+0000 mgr.y (mgr.14146) 255 : audit [DBG] from='client.14568 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-07-31T01:45:37.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:36 smithi045 conmon[39718]: cluster 2021-07-31T01:45:35.358942+0000 mgr.y (mgr.14146) 256 : cluster [DBG] pgmap v221: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:38.380 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:45:39.118 INFO:teuthology.orchestra.run.smithi045.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2021-07-31T01:45:39.119 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2021-07-31T01:45:39.119 INFO:tasks.cephadm:Setup complete, yielding 2021-07-31T01:45:39.120 INFO:teuthology.run_tasks:Running task print... 2021-07-31T01:45:39.141 INFO:teuthology.task.print:**** done end installing pacific cephadm ... 2021-07-31T01:45:39.141 INFO:teuthology.run_tasks:Running task print... 2021-07-31T01:45:39.201 INFO:teuthology.task.print:**** done start cephadm.shell ceph config set mgr... 2021-07-31T01:45:39.202 INFO:teuthology.run_tasks:Running task cephadm.shell... 2021-07-31T01:45:39.223 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi045.front.sepia.ceph.com 2021-07-31T01:45:39.224 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f52d2c14-f19f-11eb-8c24-001a4aab830c -- bash -c 'ceph config set mgr mgr/cephadm/use_repo_digest true --force' 2021-07-31T01:45:39.250 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:38 smithi045 conmon[39718]: cluster 2021-07-31T01:45:37 2021-07-31T01:45:39.250 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:38 smithi045 conmon[39718]: .359297+0000 mgr.y (mgr.14146) 257 : cluster [DBG] pgmap v222: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:39.251 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:38 smithi045 conmon[39718]: audit 2021-07-31T01:45:38.379156+0000 mon.a (mon.0) 625 : audit [DBG] from='client.? 172.21.15.45:0/4094795289' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2021-07-31T01:45:39.251 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:38 smithi045 conmon[33655]: cluster 2021-07-31T01:45:37.359297+0000 mgr.y (mgr.14146) 2021-07-31T01:45:39.251 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:38 smithi045 conmon[33655]: 257 : cluster [DBG] pgmap v222: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:39.251 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:38 smithi045 conmon[33655]: audit 2021-07-31T01:45:38.379156+0000 mon.a (mon.0) 625 : audit [DBG] from='client.? 172.21.15.45:0/4094795289' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2021-07-31T01:45:39.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:38 smithi154 conmon[34553]: cluster 2021-07-31T01:45:37.359297+0000 mgr.y (mgr.14146) 257 : cluster [DBG] pgmap v222: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:39.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:38 smithi154 conmon[34553]: audit 2021-07-31T 2021-07-31T01:45:39.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:38 smithi154 conmon[34553]: 01:45:38.379156+0000 mon.a (mon.0) 625 : audit [DBG] from='client.? 172.21.15.45:0/4094795289' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2021-07-31T01:45:40.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:39 smithi154 conmon[34553]: audit 2021-07-31T01:45:39.432940+0000 mon.a (mon.0) 626 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:45:40.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:39 smithi154 conmon[34553]: audit 2021-07-31T01:45:39.433155+0000 mon.a 2021-07-31T01:45:40.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:39 smithi154 conmon[34553]: (mon.0) 627 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-07-31T01:45:40.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:39 smithi045 conmon[39718]: audit 2021-07-31T01:45:39.432940+0000 2021-07-31T01:45:40.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:39 smithi045 conmon[39718]: mon.a (mon.0) 626 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:45:40.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:39 smithi045 conmon[39718]: audit 2021-07-31T01:45:39.433155+0000 mon.a (mon.0) 627 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-07-31T01:45:40.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:39 smithi045 conmon[33655]: audit 2021-07-31T01:45:39. 2021-07-31T01:45:40.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:39 smithi045 conmon[33655]: 432940+0000 mon.a (mon.0) 626 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:45:40.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:39 smithi045 conmon[33655]: audit 2021-07-31T01:45:39.433155+0000 mon.a (mon.0) 627 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-07-31T01:45:41.017 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:40 smithi045 conmon[33655]: cluster 2021-07-31T01:45:39.359632+0000 mgr.y (mgr.14146) 258 : cluster [DBG] pgmap v223: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:41.017 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:40 smithi045 conmon[39718]: cluster 2021-07-31T01:45:39.359632+0000 mgr.y (mgr.14146) 258 : cluster [DBG] pgmap v223: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:41.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:40 smithi154 conmon[34553]: cluster 2021-07-31T01:45:39.359632+0000 mgr.y (mgr.14146) 258 : cluster [DBG] pgmap v223: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:42.145 INFO:teuthology.run_tasks:Running task print... 2021-07-31T01:45:42.155 INFO:teuthology.task.print:**** done cephadm.shell ceph config set mgr... 2021-07-31T01:45:42.155 INFO:teuthology.run_tasks:Running task print... 2021-07-31T01:45:42.185 INFO:teuthology.task.print:**** done start parallel 2021-07-31T01:45:42.185 INFO:teuthology.run_tasks:Running task parallel... 2021-07-31T01:45:42.219 INFO:teuthology.task.parallel:starting parallel... 2021-07-31T01:45:42.220 INFO:teuthology.task.parallel:In parallel, running task full_sequential... 2021-07-31T01:45:42.221 INFO:teuthology.task.full_sequential:In full_sequential, running task print... 2021-07-31T01:45:42.221 INFO:teuthology.task.print:**** done start ec-rados-default.yaml 2021-07-31T01:45:42.221 INFO:teuthology.task.full_sequential:In full_sequential, running task rados... 2021-07-31T01:45:42.224 INFO:tasks.rados:Beginning rados... 2021-07-31T01:45:42.224 INFO:tasks.rados:joining rados 2021-07-31T01:45:42.224 INFO:teuthology.task.parallel:In parallel, running task sequential... 2021-07-31T01:45:42.226 INFO:teuthology.task.sequential:In sequential, running task print... 2021-07-31T01:45:42.227 INFO:teuthology.task.print:**** done start upgrade, wait 2021-07-31T01:45:42.227 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2021-07-31T01:45:42.227 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi045.front.sepia.ceph.com 2021-07-31T01:45:42.228 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f52d2c14-f19f-11eb-8c24-001a4aab830c -e sha1=f25860a2cae30438a8a8d15f1f74f6043cc73d75 -- bash -c 'ceph config set global log_to_journald false --force' 2021-07-31T01:45:42.229 INFO:tasks.rados:clients are ['client.0', 'client.1'] 2021-07-31T01:45:42.229 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell --fsid f52d2c14-f19f-11eb-8c24-001a4aab830c -- ceph osd erasure-code-profile set teuthologyprofile k=2 m=1 crush-failure-domain=osd 2021-07-31T01:45:42.457 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:42 smithi045 conmon[33655]: cluster 2021-07- 2021-07-31T01:45:42.457 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:42 smithi045 conmon[33655]: 31T01:45:41.359940+0000 mgr.y (mgr.14146) 259 : cluster [DBG] pgmap v224: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:42.457 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:42 smithi045 conmon[33655]: audit 2021-07-31T01:45:41.455877+0000 mon.a (mon.0) 628 : audit [INF] from='client.? ' entity='client.admin' 2021-07-31T01:45:42.458 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:42 smithi045 conmon[33655]: audit 2021-07-31T01:45:41.460652+0000 mon.a (mon.0) 629 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:45:42.458 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:42 smithi045 conmon[33655]: audit 2021-07-31T01:45:41.461896+0000 mon.a (mon.0) 630 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:45:42.458 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:42 smithi045 conmon[33655]: audit 2021-07-31T01:45:41.462680+0000 mon.a (mon.0) 631 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:45:42.458 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:42 smithi045 conmon[33655]: audit 2021-07-31T01:45:41.828410+0000 mon.a (mon.0) 632 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:45:42.459 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:42 smithi045 conmon[39718]: cluster 2021-07-31T01:45:41.359940 2021-07-31T01:45:42.459 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:42 smithi045 conmon[39718]: +0000 mgr.y (mgr.14146) 259 : cluster [DBG] pgmap v224: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:42.459 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:42 smithi045 conmon[39718]: audit 2021-07-31T01:45:41.455877+0000 mon.a (mon.0) 628 : audit [INF] from='client.? ' entity='client.admin' 2021-07-31T01:45:42.459 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:42 smithi045 conmon[39718]: audit 2021-07-31T01:45:41.460652+0000 mon.a (mon.0) 629 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:45:42.460 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:42 smithi045 conmon[39718]: audit 2021-07-31T01:45:41.461896+0000 mon.a (mon.0) 630 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:45:42.460 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:42 smithi045 conmon[39718]: audit 2021-07-31T01:45:41.462680+0000 mon.a (mon.0) 631 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:45:42.460 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:42 smithi045 conmon[39718]: audit 2021-07-31T01:45:41.828410+0000 mon.a (mon.0) 632 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:45:42.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:42 smithi154 conmon[34553]: cluster 2021-07-31T01:45:41.359940+0000 mgr.y (mgr.14146) 259 : cluster [DBG] pgmap v224: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:42.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:42 smithi154 conmon[34553]: 2021-07-31T01:45:42.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:42 smithi154 conmon[34553]: audit 2021-07-31T01:45:41.455877+0000 mon.a (mon.0) 628 : audit [INF] from='client.? ' entity='client.admin' 2021-07-31T01:45:42.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:42 smithi154 conmon[34553]: audit 2021-07-31T01:45:41.460652+0000 mon.a (mon.0) 629 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:45:42.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:42 smithi154 conmon[34553]: audit 2021-07-31T01:45:41.461896+0000 mon.a (mon.0) 630 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:45:42.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:42 smithi154 conmon[34553]: audit 2021-07-31T01:45:41.462680+0000 mon.a (mon.0) 631 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:45:42.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:42 smithi154 conmon[34553]: audit 2021-07-31T01:45:41.828410+0000 mon.a (mon.0) 632 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:45:42.855 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f52d2c14-f19f-11eb-8c24-001a4aab830c/mon.a/config 2021-07-31T01:45:44.484 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:44 smithi045 conmon[39718]: cluster 2021-07-31T01:45:43.360268+0000 mgr.y (mgr.14146) 260 : cluster [DBG] pgmap v225: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:44.484 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:44 smithi045 conmon[33655]: cluster 2021-07-31T01:45:43.360268+0000 mgr.y (mgr.14146) 260 : cluster [DBG] pgmap v225: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:44.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:44 smithi154 conmon[34553]: cluster 2021-07-31T01:45:43.360268+0000 mgr.y (mgr.14146) 260 : cluster [DBG] pgmap v225: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:45.333 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f52d2c14-f19f-11eb-8c24-001a4aab830c -e sha1=f25860a2cae30438a8a8d15f1f74f6043cc73d75 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2021-07-31T01:45:46.160 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:45 smithi045 conmon[33655]: audit 2021-07-31T01:45:44.788524+0000 mon.a 2021-07-31T01:45:46.160 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:45 smithi045 conmon[33655]: (mon.0) 633 : audit [INF] from='client.? 172.21.15.45:0/953489784' entity='client.admin' 2021-07-31T01:45:46.160 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:45 smithi045 conmon[33655]: audit 2021-07-31T01:45:44.793127+0000 mon.a (mon.0) 634 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:45:46.160 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:45 smithi045 conmon[33655]: audit 2021-07-31T01:45:44.794623+0000 mon.a (mon.0) 635 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:45:46.161 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:45 smithi045 conmon[33655]: audit 2021-07-31T01:45:44.795544+0000 mon.a (mon.0) 636 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:45:46.161 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:45 smithi045 conmon[33655]: audit 2021-07-31T01:45:45.097541+0000 mon.c (mon.1) 14 : audit [INF] from='client.? 172.21.15.45:0/973896976' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "teuthologyprofile", "profile": ["k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2021-07-31T01:45:46.161 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:45 smithi045 conmon[33655]: audit 2021-07-31T01:45:45.097863+0000 mon.a (mon.0) 637 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "teuthologyprofile", "profile": ["k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2021-07-31T01:45:46.161 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:45 smithi045 conmon[33655]: audit 2021-07-31T01:45:45.222123 2021-07-31T01:45:46.162 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:45 smithi045 conmon[33655]: +0000 mon.a (mon.0) 638 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:45:46.162 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:45 smithi045 conmon[39718]: audit 2021-07-31T01:45:44.788524+0000 mon.a ( 2021-07-31T01:45:46.162 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:45 smithi045 conmon[39718]: mon.0) 633 : audit [INF] from='client.? 172.21.15.45:0/953489784' entity='client.admin' 2021-07-31T01:45:46.163 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:45 smithi045 conmon[39718]: audit 2021-07-31T01:45:44.793127+0000 mon.a (mon.0) 634 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:45:46.163 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:45 smithi045 conmon[39718]: audit 2021-07-31T01:45:44.794623+0000 mon.a (mon.0) 635 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:45:46.163 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:45 smithi045 conmon[39718]: audit 2021-07-31T01:45:44.795544+0000 mon.a (mon.0) 636 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:45:46.163 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:45 smithi045 conmon[39718]: audit 2021-07-31T01:45:45.097541+0000 mon.c (mon.1) 14 : audit [INF] from='client.? 172.21.15.45:0/973896976' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "teuthologyprofile", "profile": ["k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2021-07-31T01:45:46.164 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:45 smithi045 conmon[39718]: audit 2021-07-31T01:45:45.097863+0000 mon.a (mon.0) 637 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "teuthologyprofile", "profile": ["k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2021-07-31T01:45:46.164 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:45 smithi045 conmon[39718]: audit 2021-07-31T01:45:45.222123+0000 mon.a (mon.0) 638 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:45:46.164 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:45 smithi045 conmon[39718]: 2021-07-31T01:45:46.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:45 smithi154 conmon[34553]: audit 2021-07-31T01:45:44.788524+0000 mon.a (mon.0) 633 : audit [INF] from='client.? 172.21.15.45:0/953489784' entity='client.admin' 2021-07-31T01:45:46.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:45 smithi154 conmon[34553]: audit 2021-07-31T01:45:44.793127+0000 mon.a (mon.0) 634 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:45:46.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:45 smithi154 conmon[34553]: audit 2021-07-31T01:45:44.794623+0000 mon.a (mon.0) 635 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:45:46.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:45 smithi154 conmon[34553]: audit 2021-07-31T01:45:46.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:45 smithi154 conmon[34553]: 2021-07-31T01:45:44.795544+0000 mon.a (mon.0) 636 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:45:46.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:45 smithi154 conmon[34553]: audit 2021-07-31T01 2021-07-31T01:45:46.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:45 smithi154 conmon[34553]: :45:45.097541+0000 mon.c (mon.1) 14 : audit [INF] from='client.? 172.21.15.45:0/973896976' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "teuthologyprofile", "profile": ["k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2021-07-31T01:45:46.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:45 smithi154 conmon[34553]: audit 2021-07-31T01:45:45.097863+0000 mon.a (mon.0) 637 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "teuthologyprofile", "profile": ["k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2021-07-31T01:45:46.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:45 smithi154 conmon[34553]: audit 2021-07-31T01:45:45.222123+0000 mon.a (mon.0) 638 : audit [INF] 2021-07-31T01:45:46.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:45 smithi154 conmon[34553]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:45:46.727 INFO:tasks.rados:starting run 0 out of 1 2021-07-31T01:45:46.728 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_0 2021-07-31T01:45:46.728 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell --fsid f52d2c14-f19f-11eb-8c24-001a4aab830c -- ceph osd pool create unique_pool_0 16 16 erasure teuthologyprofile 2021-07-31T01:45:47.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:46 smithi045 conmon[33655]: cluster 2021-07-31T01:45:45.360757+0000 mgr.y (mgr.14146) 261 : cluster [DBG] pgmap v226: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:47.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:46 smithi045 conmon[33655]: audit 2021-07-31T01:45:45.792286+0000 mon.a (mon.0) 639 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "teuthologyprofile", "profile": ["k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2021-07-31T01:45:47.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:46 smithi045 conmon[33655]: cluster 2021-07-31T01:45:45.792382+0000 mon.a (mon.0) 640 : cluster [DBG] osdmap e54: 8 total, 8 up, 8 in 2021-07-31T01:45:47.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:46 smithi045 conmon[39718]: cluster 2021-07-31T01:45:45.360757+0000 mgr.y (mgr.14146) 261 : cluster [DBG] pgmap v226: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:47.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:46 smithi045 conmon[39718]: audit 2021-07-31T01:45:45.792286+0000 mon.a (mon.0) 639 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "teuthologyprofile", "profile": ["k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2021-07-31T01:45:47.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:46 smithi045 conmon[39718]: cluster 2021-07-31T01:45:45.792382+0000 mon.a (mon.0) 640 : cluster [DBG] osdmap e54: 8 total, 8 up, 8 in 2021-07-31T01:45:47.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:46 smithi154 conmon[34553]: cluster 2021-07-31T01:45:45.360757+0000 mgr.y (mgr.14146) 261 : cluster [DBG] pgmap v226: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:47.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:46 smithi154 conmon[34553]: audit 2021-07-31T01:45:45.792286+0000 mon.a (mon.0) 639 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "teuthologyprofile", "profile": ["k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2021-07-31T01:45:47.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:46 smithi154 conmon[34553]: cluster 2021-07-31T01:45:45.792382+0000 mon.a (mon.0) 640 : cluster 2021-07-31T01:45:47.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:46 smithi154 conmon[34553]: [DBG] osdmap e54: 8 total, 8 up, 8 in 2021-07-31T01:45:47.452 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f52d2c14-f19f-11eb-8c24-001a4aab830c/mon.a/config 2021-07-31T01:45:47.719 INFO:teuthology.orchestra.run.smithi045.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:f25860a2cae30438a8a8d15f1f74f6043cc73d75 2021-07-31T01:45:48.650 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f52d2c14-f19f-11eb-8c24-001a4aab830c -e sha1=f25860a2cae30438a8a8d15f1f74f6043cc73d75 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true ; do ceph orch ps ; ceph versions ; sleep 30 ; done' 2021-07-31T01:45:48.835 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:48 smithi045 conmon[39718]: cluster 2021-07-31T01:45:47.361084+0000 mgr.y (mgr.14146) 2021-07-31T01:45:48.835 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:48 smithi045 conmon[39718]: 262 : cluster [DBG] pgmap v228: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:48.835 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:48 smithi045 conmon[39718]: audit 2021-07-31T01:45:47.714313+0000 mgr.y (mgr.14146) 263 : audit [DBG] from='client.14598 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:f25860a2cae30438a8a8d15f1f74f6043cc73d75", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:45:48.836 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:48 smithi045 conmon[39718]: cephadm 2021-07-31T01:45:47.714991+0000 mgr.y (mgr.14146) 264 : cephadm [INF] Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:f25860a2cae30438a8a8d15f1f74f6043cc73d75 2021-07-31T01:45:48.836 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:48 smithi045 conmon[39718]: audit 2021-07-31T01:45:47.719208+0000 mon.a (mon.0) 641 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:45:48.836 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:48 smithi045 conmon[39718]: audit 2021-07-31T01:45:47.719794+0000 mon.a (mon.0) 642 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:45:48.836 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:48 smithi045 conmon[39718]: audit 2021-07-31T01:45:47.720861+0000 mon.a (mon.0) 643 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:45:48.837 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:48 smithi045 conmon[39718]: audit 2021-07-31T01:45:47.721758+0000 mon.a (mon.0) 644 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:45:48.837 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:48 smithi045 conmon[39718]: audit 2021-07-31T01:45:47.726508+0000 mon.a (mon.0) 645 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:45:48.837 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:48 smithi045 conmon[39718]: cephadm 2021-07-31T01:45:47.729425+0000 mgr.y (mgr.14146) 265 : cephadm [INF] Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:f25860a2cae30438a8a8d15f1f74f6043cc73d75 2021-07-31T01:45:48.837 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:48 smithi045 conmon[33655]: cluster 2021-07-31T01:45 2021-07-31T01:45:48.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:48 smithi045 conmon[33655]: :47.361084+0000 mgr.y (mgr.14146) 262 : cluster [DBG] pgmap v228: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:48.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:48 smithi045 conmon[33655]: audit 2021-07-31T01:45:47.714313+0000 mgr.y (mgr.14146) 263 : audit [DBG] from='client.14598 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:f25860a2cae30438a8a8d15f1f74f6043cc73d75", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:45:48.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:48 smithi045 conmon[33655]: cephadm 2021-07-31T01:45:47.714991+0000 mgr.y (mgr.14146) 264 : cephadm [INF] Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:f25860a2cae30438a8a8d15f1f74f6043cc73d75 2021-07-31T01:45:48.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:48 smithi045 conmon[33655]: audit 2021-07-31T01:45:47.719208+0000 mon.a (mon.0) 641 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:45:48.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:48 smithi045 conmon[33655]: audit 2021-07-31T01:45:47.719794+0000 mon.a (mon.0) 642 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:45:48.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:48 smithi045 conmon[33655]: audit 2021-07-31T01:45:47.720861+0000 mon.a (mon.0) 643 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:45:48.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:48 smithi045 conmon[33655]: audit 2021-07-31T01:45:47.721758+0000 mon.a (mon.0) 644 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:45:48.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:48 smithi045 conmon[33655]: audit 2021-07-31T01:45:47.726508+0000 mon.a (mon.0) 645 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:45:48.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:48 smithi045 conmon[33655]: cephadm 2021-07-31T01:45:47.729425+0000 mgr.y (mgr.14146) 265 : cephadm [INF] Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:f25860a2cae30438a8a8d15f1f74f6043cc73d75 2021-07-31T01:45:49.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:48 smithi154 conmon[34553]: cluster 2021-07-31T01:45:47.361084+0000 mgr.y (mgr.14146) 262 : cluster [DBG] pgmap v228: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:49.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:48 smithi154 conmon[34553]: audit 2021-07-31T01:45:47.714313+0000 mgr.y (mgr.14146) 263 : audit [DBG] from='client.14598 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:f25860a2cae30438a8a8d15f1f74f6043cc73d75", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:45:49.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:48 smithi154 conmon[34553]: cephadm 2021-07-31T01:45:47.714991+0000 mgr.y (mgr.14146) 264 : cephadm [INF] Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:f25860a2cae30438a8a8d15f1f74f6043cc73d75 2021-07-31T01:45:49.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:48 smithi154 conmon[34553]: audit 2021-07 2021-07-31T01:45:49.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:48 smithi154 conmon[34553]: -31T01:45:47.719208+0000 mon.a (mon.0) 641 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:45:49.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:48 smithi154 conmon[34553]: audit 2021-07-31T01 2021-07-31T01:45:49.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:48 smithi154 conmon[34553]: :45:47.719794+0000 mon.a (mon.0) 642 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:45:49.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:48 smithi154 conmon[34553]: audit 2021-07-31T01:45:47.720861+0000 mon.a (mon.0) 643 2021-07-31T01:45:49.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:48 smithi154 conmon[34553]: : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:45:49.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:48 smithi154 conmon[34553]: audit 2021-07-31T01:45:49.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:48 smithi154 conmon[34553]: 2021-07-31T01:45:47.721758+0000 mon.a (mon.0) 644 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:45:49.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:48 smithi154 conmon[34553]: audit 2021-07-31T01 2021-07-31T01:45:49.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:48 smithi154 conmon[34553]: :45:47.726508+0000 mon.a (mon.0) 645 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:45:49.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:48 smithi154 conmon[34553]: cephadm 2021-07-31 2021-07-31T01:45:49.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:48 smithi154 conmon[34553]: T01:45:47.729425+0000 mgr.y (mgr.14146) 265 : cephadm [INF] Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:f25860a2cae30438a8a8d15f1f74f6043cc73d75 2021-07-31T01:45:51.491 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:51 smithi045 conmon[33655]: cluster 2021-07-31T01:45:49.361374+0000 mgr.y (mgr.14146) 266 : cluster [DBG] pgmap v229: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:51.492 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:51 smithi045 conmon[33655]: audit 2021-07-31T01:45:50.320927+0000 mon.c (mon.1) 15 : audit [INF] from='client.? 172.21.15.45:0/775973152' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16, "pgp_num": 16, "pool_type": "erasure", "erasure_code_profile": "teuthologyprofile"}]: dispatch 2021-07-31T01:45:51.492 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:51 smithi045 conmon[33655]: audit 2021-07-31T01:45:50.321238+0000 mon.a (mon.0) 646 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16, "pgp_num": 16, "pool_type": "erasure", "erasure_code_profile": "teuthologyprofile"}]: dispatch 2021-07-31T01:45:51.492 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:51 smithi045 conmon[39718]: cluster 2021-07-31T01:45:51.493 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:51 smithi045 conmon[39718]: 2021-07-31T01:45:51.493 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:51 smithi045 conmon[39718]: 2021-07-31T01:45 2021-07-31T01:45:51.493 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:51 smithi045 conmon[39718]: :49.361374+0000 mgr.y (mgr.14146) 266 : cluster [DBG] pgmap v229: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:51.493 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:51 smithi045 conmon[39718]: 2021-07-31T01:45:51.493 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:51 smithi045 conmon[39718]: audit 2021-07-31T01:45:51.494 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:51 smithi045 conmon[39718]: 2021-07-31T 2021-07-31T01:45:51.494 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:51 smithi045 conmon[39718]: 01:45:50.320927+0000 mon.c (mon.1) 15 : audit [INF] 2021-07-31T01:45:51.494 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:51 smithi045 conmon[39718]: from='client.? 172.21.15.45:0/775973152' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16, "pgp_num": 16, "pool_type": "erasure", "erasure_code_profile": "teuthologyprofile"}]: dispatch 2021-07-31T01:45:51.494 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:51 smithi045 conmon[39718]: audit 2021-07-31T01:45:50 2021-07-31T01:45:51.495 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:51 smithi045 conmon[39718]: .321238+0000 mon.a (mon.0) 646 : audit 2021-07-31T01:45:51.495 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:51 smithi045 conmon[39718]: [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16, "pgp_num": 16, "pool_type": "erasure", "erasure_code_profile": "teuthologyprofile"}]: dispatch 2021-07-31T01:45:51.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:51 smithi154 conmon[34553]: cluster 2021-07-31T01:45:49.361374+0000 mgr.y (mgr.14146) 266 : cluster [DBG] pgmap v229: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:51.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:51 smithi154 conmon[34553]: audit 2021-07-31T01:45:50.320927+0000 mon.c (mon.1) 15 : audit [INF] from='client.? 172.21.15.45:0/775973152' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16, "pgp_num": 16, "pool_type": "erasure", "erasure_code_profile": "teuthologyprofile"}]: dispatch 2021-07-31T01:45:51.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:51 smithi154 conmon[34553]: audit 2021-07-31T01:45:50.321238+0000 mon.a (mon.0) 646 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16, "pgp_num": 16, "pool_type": "erasure", "erasure_code_profile": "teuthologyprofile"}]: dispatch 2021-07-31T01:45:51.737 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'unique_pool_0' created 2021-07-31T01:45:52.123 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T01:45:52.479 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T01:45:52.479 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 running (62s) 45s ago 91s - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T01:45:52.480 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 running (74s) 47s ago 74s 37.1M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T01:45:52.480 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 running (4m) 47s ago 4m 377M - 16.2.5 a29040f3000e f9af406258e1 2021-07-31T01:45:52.480 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 running (6m) 45s ago 6m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T01:45:52.480 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 running (6m) 45s ago 7m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T01:45:52.481 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 running (5m) 47s ago 5m 92.9M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T01:45:52.481 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 running (5m) 45s ago 5m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T01:45:52.481 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 running (104s) 45s ago 104s - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T01:45:52.481 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 running (101s) 47s ago 101s 8144k - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T01:45:52.481 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 running (4m) 45s ago 4m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T01:45:52.482 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 running (4m) 45s ago 4m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T01:45:52.482 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 running (3m) 45s ago 3m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T01:45:52.482 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 running (3m) 45s ago 3m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T01:45:52.482 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 running (3m) 47s ago 3m 43.3M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T01:45:52.483 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 running (2m) 47s ago 2m 41.6M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T01:45:52.483 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 running (2m) 47s ago 2m 41.2M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T01:45:52.483 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 running (2m) 47s ago 2m 42.6M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T01:45:52.483 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 running (57s) 47s ago 110s 28.9M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T01:45:52.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:52 smithi154 conmon[34553]: cluster 2021-07-31T01:45:51.225442+0000 mon.a (mon.0) 647 : cluster [DBG] osdmap e55: 8 total, 8 up, 8 in 2021-07-31T01:45:52.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:52 smithi154 conmon[34553]: cluster 2021-07-31T01:45:51.361696+0000 mgr.y (mgr.14146) 267 : cluster [DBG] pgmap v231: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:52.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:52 smithi154 conmon[34553]: audit 2021-07-31T01:45:51.730873+0000 mon.a (mon.0) 648 : 2021-07-31T01:45:52.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:52 smithi154 conmon[34553]: audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16, "pgp_num": 16, "pool_type": "erasure", "erasure_code_profile": "teuthologyprofile"}]': finished 2021-07-31T01:45:52.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:52 smithi154 conmon[34553]: cluster 2021-07- 2021-07-31T01:45:52.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:52 smithi154 conmon[34553]: 31T01:45:51.731100+0000 mon.a (mon.0) 649 : cluster [DBG] 2021-07-31T01:45:52.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:52 smithi154 conmon[34553]: osdmap e56: 8 total, 8 up, 8 in 2021-07-31T01:45:52.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:52 smithi045 conmon[33655]: cluster 2021-07 2021-07-31T01:45:52.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:52 smithi045 conmon[33655]: -31T01:45:51.225442+0000 mon.a (mon.0) 647 : cluster [DBG] osdmap e55: 8 total, 8 up, 8 in 2021-07-31T01:45:52.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:52 smithi045 conmon[33655]: cluster 2021-07-31T01:45:51.361696+0000 mgr.y (mgr.14146) 267 : cluster [DBG] pgmap v231: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:52.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:52 smithi045 conmon[33655]: audit 2021-07- 2021-07-31T01:45:52.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:52 smithi045 conmon[33655]: 31T01:45:51.730873+0000 mon.a (mon.0) 648 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16, "pgp_num": 16, "pool_type": "erasure", "erasure_code_profile": "teuthologyprofile"}]': finished 2021-07-31T01:45:52.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:52 smithi045 conmon[33655]: cluster 2021-07- 2021-07-31T01:45:52.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:52 smithi045 conmon[33655]: 31T01:45:51.731100+0000 mon.a (mon.0) 649 : cluster [DBG] osdmap e56: 8 total, 8 up, 8 in 2021-07-31T01:45:52.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:52 smithi045 conmon[39718]: cluster 2021-07-31T01:45:52.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:52 smithi045 conmon[39718]: 2021-07-31T01:45:52.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:52 smithi045 conmon[39718]: 2021 2021-07-31T01:45:52.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:52 smithi045 conmon[39718]: - 2021-07-31T01:45:52.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:52 smithi045 conmon[39718]: 07 2021-07-31T01:45:52.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:52 smithi045 conmon[39718]: - 2021-07-31T01:45:52.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:52 smithi045 conmon[39718]: 31 2021-07-31T01:45:52.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:52 smithi045 conmon[39718]: T 2021-07-31T01:45:52.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:52 smithi045 conmon[39718]: 01 2021-07-31T01:45:52.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:52 smithi045 conmon[39718]: : 2021-07-31T01:45:52.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:52 smithi045 conmon[39718]: 45 2021-07-31T01:45:52.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:52 smithi045 conmon[39718]: : 2021-07-31T01:45:52.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:52 smithi045 conmon[39718]: 51 2021-07-31T01:45:52.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:52 smithi045 conmon[39718]: . 2021-07-31T01:45:52.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:52 smithi045 conmon[39718]: 225442 2021-07-31T01:45:52.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:52 smithi045 conmon[39718]: +0000 2021-07-31T01:45:52.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:52 smithi045 conmon[39718]: 2021-07-31T01:45:52.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:52 smithi045 conmon[39718]: mon.a 2021-07-31T01:45:52.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:52 smithi045 conmon[39718]: ( 2021-07-31T01:45:52.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:52 smithi045 conmon[39718]: mon 2021-07-31T01:45:52.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:52 smithi045 conmon[39718]: . 2021-07-31T01:45:52.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:52 smithi045 conmon[39718]: 0 2021-07-31T01:45:52.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:52 smithi045 conmon[39718]: ) 2021-07-31T01:45:52.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:52 smithi045 conmon[39718]: 647 2021-07-31T01:45:52.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:52 smithi045 conmon[39718]: : 2021-07-31T01:45:52.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:52 smithi045 conmon[39718]: cluster 2021-07-31T01:45:52.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:52 smithi045 conmon[39718]: 2021-07-31T01:45:52.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:52 smithi045 conmon[39718]: [DBG] 2021-07-31T01:45:52.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:52 smithi045 conmon[39718]: 2021-07-31T01:45:52.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:52 smithi045 conmon[39718]: osdmap e55: 8 total, 8 up, 8 in 2021-07-31T01:45:52.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:52 smithi045 conmon[39718]: 2021-07-31T01:45:52.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:52 smithi045 conmon[39718]: cluster 2021-07-31T01:45:52.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:52 smithi045 conmon[39718]: 2021-07-31T01:45:52.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:52 smithi045 conmon[39718]: 2021 2021-07-31T01:45:52.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:52 smithi045 conmon[39718]: - 2021-07-31T01:45:52.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:52 smithi045 conmon[39718]: 07 2021-07-31T01:45:52.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:52 smithi045 conmon[39718]: - 2021-07-31T01:45:52.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:52 smithi045 conmon[39718]: 31 2021-07-31T01:45:52.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:52 smithi045 conmon[39718]: T 2021-07-31T01:45:52.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:52 smithi045 conmon[39718]: 01 2021-07-31T01:45:52.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:52 smithi045 conmon[39718]: : 2021-07-31T01:45:52.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:52 smithi045 conmon[39718]: 45 2021-07-31T01:45:52.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:52 smithi045 conmon[39718]: : 2021-07-31T01:45:52.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:52 smithi045 conmon[39718]: 51 2021-07-31T01:45:52.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:52 smithi045 conmon[39718]: . 2021-07-31T01:45:52.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:52 smithi045 conmon[39718]: 361696 2021-07-31T01:45:52.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:52 smithi045 conmon[39718]: +0000 2021-07-31T01:45:52.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:52 smithi045 conmon[39718]: 2021-07-31T01:45:52.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:52 smithi045 conmon[39718]: mgr.y 2021-07-31T01:45:52.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:52 smithi045 conmon[39718]: ( 2021-07-31T01:45:52.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:52 smithi045 conmon[39718]: mgr 2021-07-31T01:45:52.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:52 smithi045 conmon[39718]: . 2021-07-31T01:45:52.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:52 smithi045 conmon[39718]: 14146) 267 : cluster [DBG] 2021-07-31T01:45:52.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:52 smithi045 conmon[39718]: pgmap v231: 9 pgs: 9 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:52.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:52 smithi045 conmon[39718]: audit 2021-07-31T01:45:52.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:52 smithi045 conmon[39718]: 2021-07-31T01:45:51 2021-07-31T01:45:52.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:52 smithi045 conmon[39718]: .730873+0000 mon.a (mon.0) 648 : 2021-07-31T01:45:52.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:52 smithi045 conmon[39718]: audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16, "pgp_num": 16, "pool_type": "erasure", "erasure_code_profile": "teuthologyprofile"}]': finished 2021-07-31T01:45:52.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:52 smithi045 conmon[39718]: cluster 2021-07-31T01:45:52.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:52 smithi045 conmon[39718]: 2021-07-31T01:45: 2021-07-31T01:45:52.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:52 smithi045 conmon[39718]: 51.731100+0000 mon.a ( 2021-07-31T01:45:52.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:52 smithi045 conmon[39718]: mon.0) 649 : cluster 2021-07-31T01:45:52.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:52 smithi045 conmon[39718]: [DBG] osdmap e56: 8 total, 8 up, 8 in 2021-07-31T01:45:52.852 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell --fsid f52d2c14-f19f-11eb-8c24-001a4aab830c -- ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2021-07-31T01:45:52.942 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T01:45:52.942 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T01:45:52.943 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T01:45:52.943 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T01:45:52.943 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T01:45:52.943 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 2 2021-07-31T01:45:52.943 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T01:45:52.944 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T01:45:52.944 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T01:45:52.944 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T01:45:52.944 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T01:45:52.945 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T01:45:52.945 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 13 2021-07-31T01:45:52.945 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T01:45:52.945 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T01:45:53.355 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f52d2c14-f19f-11eb-8c24-001a4aab830c/mon.a/config 2021-07-31T01:45:53.739 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[33655]: audit 2021-07-31T01:45:53.739 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[33655]: 2021-07-31T01:45:53.740 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[33655]: 2021 2021-07-31T01:45:53.740 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[33655]: - 2021-07-31T01:45:53.740 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[33655]: 07 2021-07-31T01:45:53.740 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[33655]: - 2021-07-31T01:45:53.740 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[33655]: 31 2021-07-31T01:45:53.741 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[33655]: T 2021-07-31T01:45:53.741 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[33655]: 01 2021-07-31T01:45:53.741 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[33655]: : 2021-07-31T01:45:53.741 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[33655]: 45 2021-07-31T01:45:53.741 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[33655]: : 2021-07-31T01:45:53.742 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[33655]: 52 2021-07-31T01:45:53.742 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[33655]: . 2021-07-31T01:45:53.742 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[33655]: 107395 2021-07-31T01:45:53.742 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[33655]: +0000 2021-07-31T01:45:53.743 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[33655]: 2021-07-31T01:45:53.743 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[33655]: mgr.y 2021-07-31T01:45:53.743 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[33655]: ( 2021-07-31T01:45:53.743 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[33655]: mgr 2021-07-31T01:45:53.743 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[33655]: . 2021-07-31T01:45:53.744 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[33655]: 14146 2021-07-31T01:45:53.744 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[33655]: ) 2021-07-31T01:45:53.744 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[33655]: 268 2021-07-31T01:45:53.744 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[33655]: : 2021-07-31T01:45:53.744 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[33655]: audit 2021-07-31T01:45:53.745 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[33655]: 2021-07-31T01:45:53.745 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[33655]: [DBG] 2021-07-31T01:45:53.745 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[33655]: 2021-07-31T01:45:53.745 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[33655]: from='client.24517 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:45:53.745 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[33655]: 2021-07-31T01:45:53.746 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[33655]: audit 2021-07-31T01:45:53.746 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[33655]: 2021-07-31T01:45:53.746 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[33655]: 2021 2021-07-31T01:45:53.746 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[33655]: - 2021-07-31T01:45:53.747 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[33655]: 07 2021-07-31T01:45:53.747 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[33655]: - 2021-07-31T01:45:53.747 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[33655]: 31 2021-07-31T01:45:53.747 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[33655]: T 2021-07-31T01:45:53.747 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[33655]: 01 2021-07-31T01:45:53.748 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[33655]: : 2021-07-31T01:45:53.748 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[33655]: 45 2021-07-31T01:45:53.748 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[33655]: : 2021-07-31T01:45:53.748 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[33655]: 52 2021-07-31T01:45:53.748 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[33655]: . 2021-07-31T01:45:53.749 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[33655]: 466704 2021-07-31T01:45:53.749 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[33655]: +0000 2021-07-31T01:45:53.749 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[33655]: 2021-07-31T01:45:53.749 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[33655]: mgr.y 2021-07-31T01:45:53.750 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[33655]: ( 2021-07-31T01:45:53.750 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[33655]: mgr 2021-07-31T01:45:53.750 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[33655]: . 2021-07-31T01:45:53.750 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[33655]: 14146 2021-07-31T01:45:53.751 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[33655]: ) 2021-07-31T01:45:53.751 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[33655]: 269 2021-07-31T01:45:53.751 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[33655]: : audit [DBG] from='client.14616 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:45:53.751 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[33655]: cluster 2021-07-31T01:45:53.751 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[33655]: 2021-07-31T01:45:52.730263+0000 2021-07-31T01:45:53.752 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[33655]: mon.a (mon.0) 650 : cluster [DBG] 2021-07-31T01:45:53.752 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[33655]: osdmap e57: 8 total, 8 up, 8 in 2021-07-31T01:45:53.752 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[33655]: audit 2021-07-31T01:45:53.752 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[33655]: 2021-07-31T01:45:52.941558 2021-07-31T01:45:53.753 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[33655]: +0000 mon.c (mon.1) 16 : audit [DBG] 2021-07-31T01:45:53.753 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[33655]: from='client.? 172.21.15.45:0/3332270394' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T01:45:53.753 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[39718]: audit 2021-07-31T01:45:53.753 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[39718]: 2021-07-31T01:45:53.754 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[39718]: 2021 2021-07-31T01:45:53.754 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[39718]: - 2021-07-31T01:45:53.754 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[39718]: 07 2021-07-31T01:45:53.754 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[39718]: - 2021-07-31T01:45:53.754 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[39718]: 31 2021-07-31T01:45:53.755 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[39718]: T 2021-07-31T01:45:53.755 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[39718]: 01 2021-07-31T01:45:53.755 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[39718]: : 2021-07-31T01:45:53.755 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[39718]: 45 2021-07-31T01:45:53.755 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[39718]: : 2021-07-31T01:45:53.756 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[39718]: 52 2021-07-31T01:45:53.756 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[39718]: . 2021-07-31T01:45:53.756 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[39718]: 107395 2021-07-31T01:45:53.756 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[39718]: +0000 2021-07-31T01:45:53.757 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[39718]: 2021-07-31T01:45:53.757 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[39718]: mgr.y 2021-07-31T01:45:53.757 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[39718]: ( 2021-07-31T01:45:53.757 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[39718]: mgr 2021-07-31T01:45:53.757 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[39718]: . 2021-07-31T01:45:53.758 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[39718]: 14146 2021-07-31T01:45:53.758 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[39718]: ) 2021-07-31T01:45:53.758 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[39718]: 268 2021-07-31T01:45:53.758 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[39718]: : 2021-07-31T01:45:53.758 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[39718]: audit 2021-07-31T01:45:53.759 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[39718]: 2021-07-31T01:45:53.759 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[39718]: [DBG] 2021-07-31T01:45:53.759 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[39718]: 2021-07-31T01:45:53.760 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[39718]: from='client.24517 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:45:53.760 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[39718]: 2021-07-31T01:45:53.760 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T01:45:53.761 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[39718]: 01:45:52.466704+0000 mgr.y (mgr.14146) 269 : audit [DBG] from='client.14616 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:45:53.762 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[39718]: cluster 2021-07 2021-07-31T01:45:53.762 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[39718]: -31T01:45:52.730263+0000 mon.a 2021-07-31T01:45:53.762 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[39718]: (mon.0) 650 : cluster [DBG] osdmap e57: 8 total, 8 up, 8 in 2021-07-31T01:45:53.762 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[39718]: 2021-07-31T01:45:53.763 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[39718]: audit 2021-07-31T01:45: 2021-07-31T01:45:53.763 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[39718]: 52.941558+0000 mon.c (mon.1) 2021-07-31T01:45:53.763 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:53 smithi045 conmon[39718]: 16 : audit [DBG] from='client.? 172.21.15.45:0/3332270394' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T01:45:54.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:53 smithi154 conmon[34553]: audit 2021-07-31T01:45:52.107395+0000 mgr.y (mgr.14146) 268 : audit [DBG] from='client.24517 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:45:54.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:53 smithi154 conmon[34553]: audit 2021-07-31T01:45:52.466704+0000 mgr.y (mgr.14146) 269 : audit [DBG] from='client.14616 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:45:54.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:53 smithi154 conmon[34553]: cluster 2021-07-31T01:45:52.730263+0000 mon.a (mon.0) 650 : cluster [DBG] osdmap e57: 8 total, 8 up, 8 in 2021-07-31T01:45:54.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:53 smithi154 conmon[34553]: audit 2021-07-31T01:45:54.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:53 smithi154 conmon[34553]: 2021-07-31T01:45:52 2021-07-31T01:45:54.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:53 smithi154 conmon[34553]: .941558+0000 mon.c (mon.1) 16 2021-07-31T01:45:54.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:53 smithi154 conmon[34553]: : audit [DBG] from='client.? 172.21.15.45:0/3332270394' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T01:45:55.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:54 smithi154 conmon[34553]: cluster 2021-07-31T01:45:53.362073+0000 mgr.y (mgr.14146) 270 : cluster [DBG] pgmap v234: 25 pgs: 3 creating+peering, 14 active+clean, 8 unknown; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:55.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:54 smithi154 conmon[34553]: cluster 2021-07-31T01:45:53.737398+0000 mon.a (mon.0) 651 : cluster [DBG] osdmap e58: 8 total, 8 up, 8 in 2021-07-31T01:45:55.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:54 smithi045 conmon[33655]: cluster 2021-07-31T01:45:53.362073 2021-07-31T01:45:55.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:54 smithi045 conmon[33655]: +0000 mgr.y (mgr.14146) 270 : cluster [DBG] pgmap v234: 25 pgs: 3 creating+peering, 14 active+clean, 8 unknown; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:55.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:54 smithi045 conmon[33655]: cluster 2021-07-31T01:45:53.737398+0000 mon.a (mon.0) 651 : cluster [DBG] osdmap e58: 8 total, 8 up, 8 in 2021-07-31T01:45:55.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:54 smithi045 conmon[39718]: cluster 2021-07-31T01:45:53.362073 2021-07-31T01:45:55.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:54 smithi045 conmon[39718]: +0000 mgr.y (mgr.14146) 270 : cluster [DBG] pgmap v234: 25 pgs: 3 creating+peering, 14 active+clean, 8 unknown; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:55.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:54 smithi045 conmon[39718]: cluster 2021-07-31T01:45:55.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:54 smithi045 conmon[39718]: 2021-07-31T01:45:53.737398+0000 mon.a (mon.0) 651 : cluster [DBG] 2021-07-31T01:45:55.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:54 smithi045 conmon[39718]: osdmap e58: 8 total, 8 up, 8 in 2021-07-31T01:45:56.747 INFO:teuthology.orchestra.run.smithi045.stderr:enabled application 'rados' on pool 'unique_pool_0' 2021-07-31T01:45:57.001 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:56 smithi045 conmon[39718]: cluster 2021-07-31T01:45:55. 2021-07-31T01:45:57.002 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:56 smithi045 conmon[39718]: 362379+0000 mgr.y (mgr.14146) 271 : cluster [DBG] pgmap v236: 25 pgs: 3 creating+peering, 22 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:57.002 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:56 smithi045 conmon[39718]: audit 2021-07-31T01:45:55.852230+0000 mon.c (mon.1) 17 : audit [INF] from='client.? 172.21.15.45:0/1315042109' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2021-07-31T01:45:57.002 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:56 smithi045 conmon[39718]: audit 2021-07-31T01:45:55.852609+0000 mon.a (mon.0) 652 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2021-07-31T01:45:57.006 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:56 smithi045 conmon[33655]: cluster 2021-07-31T01:45:55.362379+0000 mgr.y 2021-07-31T01:45:57.007 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:56 smithi045 conmon[33655]: (mgr.14146) 271 : cluster [DBG] pgmap v236: 25 pgs: 3 creating+peering, 22 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:57.007 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:56 smithi045 conmon[33655]: audit 2021-07-31T01:45:55.852230+0000 mon.c (mon.1) 17 : audit [INF] from='client.? 172.21.15.45:0/1315042109' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2021-07-31T01:45:57.007 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:56 smithi045 conmon[33655]: audit 2021-07-31T01:45:55.852609+0000 mon.a (mon.0) 652 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2021-07-31T01:45:57.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:56 smithi154 conmon[34553]: cluster 2021-07-31T01:45:55.362379+0000 mgr.y (mgr.14146) 271 : cluster [DBG] pgmap v236: 25 pgs: 3 creating+peering, 22 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:57.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:56 smithi154 conmon[34553]: audit 2021 2021-07-31T01:45:57.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:56 smithi154 conmon[34553]: -07-31T01:45:55.852230+0000 mon.c (mon.1) 17 : audit [INF] from='client.? 172.21.15.45:0/1315042109' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2021-07-31T01:45:57.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:56 smithi154 conmon[34553]: audit 2021-07-31T01:45:55.852609+0000 2021-07-31T01:45:57.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:56 smithi154 conmon[34553]: mon.a (mon.0) 652 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2021-07-31T01:45:58.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:57 smithi154 conmon[34553]: audit 2021-07-31T01:45:56.743436+0000 mon.a (mon.0) 653 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2021-07-31T01:45:58.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:57 smithi154 conmon[34553]: cluster 2021-07-31T01:45:56.743511+0000 mon.a (mon.0) 654 : cluster [DBG] osdmap e59: 8 total, 8 up, 8 in 2021-07-31T01:45:58.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:57 smithi045 conmon[33655]: audit 2021-07-31T01:45:56.743436+0000 mon.a (mon. 2021-07-31T01:45:58.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:57 smithi045 conmon[33655]: 0) 653 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2021-07-31T01:45:58.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:57 smithi045 conmon[33655]: cluster 2021-07-31T01:45:56.743511+0000 mon.a (mon.0) 654 : cluster [DBG] osdmap e59: 8 total, 8 up, 8 in 2021-07-31T01:45:58.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:57 smithi045 conmon[39718]: audit 2021-07-31T01:45:56.743436 2021-07-31T01:45:58.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:57 smithi045 conmon[39718]: +0000 mon.a (mon.0) 653 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2021-07-31T01:45:58.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:57 smithi045 conmon[39718]: cluster 2021-07-31T01:45:58.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:57 smithi045 conmon[39718]: 2021-07-31T01:45: 2021-07-31T01:45:58.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:57 smithi045 conmon[39718]: 56.743511+0000 mon.a (mon. 2021-07-31T01:45:58.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:57 smithi045 conmon[39718]: 0) 654 : cluster [DBG] osdmap e59: 8 total, 8 up, 8 in 2021-07-31T01:45:58.961 DEBUG:teuthology.orchestra.run.smithi045:> CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --ec-pool --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 0 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op append 100 --pool unique_pool_0 2021-07-31T01:45:58.987 INFO:tasks.rados.rados.0.smithi045.stdout:adding op weight read -> 100 2021-07-31T01:45:58.988 INFO:tasks.rados.rados.0.smithi045.stdout:adding op weight delete -> 50 2021-07-31T01:45:58.988 INFO:tasks.rados.rados.0.smithi045.stdout:adding op weight snap_create -> 50 2021-07-31T01:45:58.988 INFO:tasks.rados.rados.0.smithi045.stdout:adding op weight snap_remove -> 50 2021-07-31T01:45:58.988 INFO:tasks.rados.rados.0.smithi045.stdout:adding op weight rollback -> 50 2021-07-31T01:45:58.989 INFO:tasks.rados.rados.0.smithi045.stdout:adding op weight setattr -> 25 2021-07-31T01:45:58.989 INFO:tasks.rados.rados.0.smithi045.stdout:adding op weight rmattr -> 25 2021-07-31T01:45:58.989 INFO:tasks.rados.rados.0.smithi045.stdout:adding op weight copy_from -> 50 2021-07-31T01:45:58.989 INFO:tasks.rados.rados.0.smithi045.stdout:adding op weight append -> 100 2021-07-31T01:45:58.990 INFO:tasks.rados.rados.0.smithi045.stdout:ceph version 16.2.5-150-g5a784c87 (5a784c87b94e0ea996e88d93f2de2a1a9497d277) pacific (stable) 2021-07-31T01:45:58.990 INFO:tasks.rados.rados.0.smithi045.stdout:Configuration: 2021-07-31T01:45:58.990 INFO:tasks.rados.rados.0.smithi045.stdout: Number of operations: 4000 2021-07-31T01:45:58.990 INFO:tasks.rados.rados.0.smithi045.stdout: Number of objects: 50 2021-07-31T01:45:58.990 INFO:tasks.rados.rados.0.smithi045.stdout: Max in flight operations: 16 2021-07-31T01:45:58.991 INFO:tasks.rados.rados.0.smithi045.stdout: Object size (in bytes): 4000000 2021-07-31T01:45:58.991 INFO:tasks.rados.rados.0.smithi045.stdout: Write stride min: 400000 2021-07-31T01:45:58.991 INFO:tasks.rados.rados.0.smithi045.stdout: Write stride max: 800000 2021-07-31T01:45:59.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:58 smithi154 conmon[34553]: cluster 2021-07-31T01:45:57.362680+0000 mgr.y (mgr.14146) 272 : cluster [DBG] pgmap v238: 25 pgs: 1 creating+peering, 24 active+clean; 0 B data, 44 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:59.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:58 smithi154 conmon[34553]: 2021-07-31T01:45:59.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:58 smithi045 conmon[33655]: cluster 2021-07-31T01:45:57.362680+0000 2021-07-31T01:45:59.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:58 smithi045 conmon[33655]: mgr.y (mgr.14146) 272 : cluster [DBG] pgmap v238: 25 pgs: 1 creating+peering, 24 active+clean; 0 B data, 44 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:59.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:58 smithi045 conmon[39718]: cluster 2021-07-31T01:45:57.362680+0000 mgr.y (mgr 2021-07-31T01:45:59.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:58 smithi045 conmon[39718]: .14146) 272 : cluster [DBG] pgmap v238: 25 pgs: 1 creating+peering, 24 active+clean; 0 B data, 44 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:45:59.754 INFO:tasks.rados.rados.0.smithi045.stdout:1: write initial oid 1 2021-07-31T01:45:59.755 INFO:tasks.rados.rados.0.smithi045.stdout:1: seq_num 0 ranges {0=24576} 2021-07-31T01:45:59.755 INFO:tasks.rados.rados.0.smithi045.stdout:1: writing smithi04569269-1 from 0 to 24576 tid 1 2021-07-31T01:45:59.755 INFO:tasks.rados.rados.0.smithi045.stdout:2: write initial oid 2 2021-07-31T01:45:59.755 INFO:tasks.rados.rados.0.smithi045.stdout:2: seq_num 1 ranges {0=24576} 2021-07-31T01:45:59.755 INFO:tasks.rados.rados.0.smithi045.stdout:2: writing smithi04569269-2 from 0 to 24576 tid 1 2021-07-31T01:45:59.756 INFO:tasks.rados.rados.0.smithi045.stdout:3: write initial oid 3 2021-07-31T01:45:59.756 INFO:tasks.rados.rados.0.smithi045.stdout:3: seq_num 2 ranges {0=40960} 2021-07-31T01:45:59.757 INFO:tasks.rados.rados.0.smithi045.stdout:3: writing smithi04569269-3 from 0 to 40960 tid 1 2021-07-31T01:45:59.758 INFO:tasks.rados.rados.0.smithi045.stdout:4: write initial oid 4 2021-07-31T01:45:59.758 INFO:tasks.rados.rados.0.smithi045.stdout:4: seq_num 3 ranges {0=57344} 2021-07-31T01:45:59.758 INFO:tasks.rados.rados.0.smithi045.stdout:4: writing smithi04569269-4 from 0 to 57344 tid 1 2021-07-31T01:45:59.758 INFO:tasks.rados.rados.0.smithi045.stdout:5: write initial oid 5 2021-07-31T01:45:59.759 INFO:tasks.rados.rados.0.smithi045.stdout:5: seq_num 4 ranges {0=73728} 2021-07-31T01:45:59.759 INFO:tasks.rados.rados.0.smithi045.stdout:5: writing smithi04569269-5 from 0 to 73728 tid 1 2021-07-31T01:45:59.759 INFO:tasks.rados.rados.0.smithi045.stdout:6: write initial oid 6 2021-07-31T01:45:59.759 INFO:tasks.rados.rados.0.smithi045.stdout:6: seq_num 5 ranges {0=90112} 2021-07-31T01:45:59.760 INFO:tasks.rados.rados.0.smithi045.stdout:6: writing smithi04569269-6 from 0 to 90112 tid 1 2021-07-31T01:45:59.760 INFO:tasks.rados.rados.0.smithi045.stdout:7: write initial oid 7 2021-07-31T01:45:59.760 INFO:tasks.rados.rados.0.smithi045.stdout:7: seq_num 6 ranges {0=106496} 2021-07-31T01:45:59.760 INFO:tasks.rados.rados.0.smithi045.stdout:7: writing smithi04569269-7 from 0 to 106496 tid 1 2021-07-31T01:45:59.760 INFO:tasks.rados.rados.0.smithi045.stdout:8: write initial oid 8 2021-07-31T01:45:59.761 INFO:tasks.rados.rados.0.smithi045.stdout:8: seq_num 7 ranges {0=122880} 2021-07-31T01:45:59.761 INFO:tasks.rados.rados.0.smithi045.stdout:8: writing smithi04569269-8 from 0 to 122880 tid 1 2021-07-31T01:45:59.762 INFO:tasks.rados.rados.0.smithi045.stdout:9: write initial oid 9 2021-07-31T01:45:59.762 INFO:tasks.rados.rados.0.smithi045.stdout:9: seq_num 8 ranges {0=139264} 2021-07-31T01:45:59.762 INFO:tasks.rados.rados.0.smithi045.stdout:9: writing smithi04569269-9 from 0 to 139264 tid 1 2021-07-31T01:45:59.762 INFO:tasks.rados.rados.0.smithi045.stdout:10: write initial oid 10 2021-07-31T01:45:59.762 INFO:tasks.rados.rados.0.smithi045.stdout:10: seq_num 9 ranges {0=155648} 2021-07-31T01:45:59.763 INFO:tasks.rados.rados.0.smithi045.stdout:10: writing smithi04569269-10 from 0 to 155648 tid 1 2021-07-31T01:45:59.763 INFO:tasks.rados.rados.0.smithi045.stdout:11: write initial oid 11 2021-07-31T01:45:59.763 INFO:tasks.rados.rados.0.smithi045.stdout:11: seq_num 10 ranges {0=172032} 2021-07-31T01:45:59.763 INFO:tasks.rados.rados.0.smithi045.stdout:11: writing smithi04569269-11 from 0 to 172032 tid 1 2021-07-31T01:45:59.764 INFO:tasks.rados.rados.0.smithi045.stdout:1: finishing write tid 1 to smithi04569269-1 2021-07-31T01:45:59.765 INFO:tasks.rados.rados.0.smithi045.stdout:2: finishing write tid 1 to smithi04569269-2 2021-07-31T01:45:59.765 INFO:tasks.rados.rados.0.smithi045.stdout:2: finishing write tid 2 to smithi04569269-2 2021-07-31T01:45:59.765 INFO:tasks.rados.rados.0.smithi045.stdout:12: write initial oid 12 2021-07-31T01:45:59.765 INFO:tasks.rados.rados.0.smithi045.stdout:12: seq_num 11 ranges {0=188416} 2021-07-31T01:45:59.765 INFO:tasks.rados.rados.0.smithi045.stdout:12: writing smithi04569269-12 from 0 to 188416 tid 1 2021-07-31T01:45:59.766 INFO:tasks.rados.rados.0.smithi045.stdout:7: finishing write tid 1 to smithi04569269-7 2021-07-31T01:45:59.766 INFO:tasks.rados.rados.0.smithi045.stdout:5: finishing write tid 1 to smithi04569269-5 2021-07-31T01:45:59.766 INFO:tasks.rados.rados.0.smithi045.stdout:1: finishing write tid 2 to smithi04569269-1 2021-07-31T01:45:59.766 INFO:tasks.rados.rados.0.smithi045.stdout:4: finishing write tid 1 to smithi04569269-4 2021-07-31T01:45:59.767 INFO:tasks.rados.rados.0.smithi045.stdout:13: write initial oid 13 2021-07-31T01:45:59.767 INFO:tasks.rados.rados.0.smithi045.stdout:13: seq_num 12 ranges {0=204800} 2021-07-31T01:45:59.768 INFO:tasks.rados.rados.0.smithi045.stdout:13: writing smithi04569269-13 from 0 to 204800 tid 1 2021-07-31T01:45:59.768 INFO:tasks.rados.rados.0.smithi045.stdout:14: write initial oid 14 2021-07-31T01:45:59.768 INFO:tasks.rados.rados.0.smithi045.stdout:14: seq_num 13 ranges {0=221184} 2021-07-31T01:45:59.769 INFO:tasks.rados.rados.0.smithi045.stdout:14: writing smithi04569269-14 from 0 to 221184 tid 1 2021-07-31T01:45:59.769 INFO:tasks.rados.rados.0.smithi045.stdout:3: finishing write tid 1 to smithi04569269-3 2021-07-31T01:45:59.769 INFO:tasks.rados.rados.0.smithi045.stdout:1: finishing write tid 3 to smithi04569269-1 2021-07-31T01:45:59.769 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 1 v 2 (ObjNum 0 snap 0 seq_num 0) dirty exists 2021-07-31T01:45:59.770 INFO:tasks.rados.rados.0.smithi045.stdout:1: left oid 1 (ObjNum 0 snap 0 seq_num 0) 2021-07-31T01:45:59.770 INFO:tasks.rados.rados.0.smithi045.stdout:3: finishing write tid 2 to smithi04569269-3 2021-07-31T01:45:59.770 INFO:tasks.rados.rados.0.smithi045.stdout:2: finishing write tid 3 to smithi04569269-2 2021-07-31T01:45:59.770 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 2 v 2 (ObjNum 1 snap 0 seq_num 1) dirty exists 2021-07-31T01:45:59.770 INFO:tasks.rados.rados.0.smithi045.stdout:2: left oid 2 (ObjNum 1 snap 0 seq_num 1) 2021-07-31T01:45:59.771 INFO:tasks.rados.rados.0.smithi045.stdout:1: done (13 left) 2021-07-31T01:45:59.771 INFO:tasks.rados.rados.0.smithi045.stdout:2: done (12 left) 2021-07-31T01:45:59.771 INFO:tasks.rados.rados.0.smithi045.stdout:15: write initial oid 15 2021-07-31T01:45:59.771 INFO:tasks.rados.rados.0.smithi045.stdout:15: seq_num 14 ranges {0=237568} 2021-07-31T01:45:59.773 INFO:tasks.rados.rados.0.smithi045.stdout:15: writing smithi04569269-15 from 0 to 237568 tid 1 2021-07-31T01:45:59.773 INFO:tasks.rados.rados.0.smithi045.stdout:16: write initial oid 16 2021-07-31T01:45:59.773 INFO:tasks.rados.rados.0.smithi045.stdout:16: seq_num 15 ranges {0=253952} 2021-07-31T01:45:59.774 INFO:tasks.rados.rados.0.smithi045.stdout:16: writing smithi04569269-16 from 0 to 253952 tid 1 2021-07-31T01:45:59.774 INFO:tasks.rados.rados.0.smithi045.stdout:9: finishing write tid 1 to smithi04569269-9 2021-07-31T01:45:59.774 INFO:tasks.rados.rados.0.smithi045.stdout:5: finishing write tid 2 to smithi04569269-5 2021-07-31T01:45:59.774 INFO:tasks.rados.rados.0.smithi045.stdout:5: finishing write tid 3 to smithi04569269-5 2021-07-31T01:45:59.775 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 5 v 2 (ObjNum 4 snap 0 seq_num 4) dirty exists 2021-07-31T01:45:59.775 INFO:tasks.rados.rados.0.smithi045.stdout:5: left oid 5 (ObjNum 4 snap 0 seq_num 4) 2021-07-31T01:45:59.775 INFO:tasks.rados.rados.0.smithi045.stdout:4: finishing write tid 2 to smithi04569269-4 2021-07-31T01:45:59.775 INFO:tasks.rados.rados.0.smithi045.stdout:3: finishing write tid 3 to smithi04569269-3 2021-07-31T01:45:59.775 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 3 v 2 (ObjNum 2 snap 0 seq_num 2) dirty exists 2021-07-31T01:45:59.776 INFO:tasks.rados.rados.0.smithi045.stdout:3: left oid 3 (ObjNum 2 snap 0 seq_num 2) 2021-07-31T01:45:59.776 INFO:tasks.rados.rados.0.smithi045.stdout:6: finishing write tid 1 to smithi04569269-6 2021-07-31T01:45:59.776 INFO:tasks.rados.rados.0.smithi045.stdout:8: finishing write tid 1 to smithi04569269-8 2021-07-31T01:45:59.776 INFO:tasks.rados.rados.0.smithi045.stdout:7: finishing write tid 2 to smithi04569269-7 2021-07-31T01:45:59.776 INFO:tasks.rados.rados.0.smithi045.stdout:4: finishing write tid 3 to smithi04569269-4 2021-07-31T01:45:59.777 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 4 v 2 (ObjNum 3 snap 0 seq_num 3) dirty exists 2021-07-31T01:45:59.777 INFO:tasks.rados.rados.0.smithi045.stdout:4: left oid 4 (ObjNum 3 snap 0 seq_num 3) 2021-07-31T01:45:59.777 INFO:tasks.rados.rados.0.smithi045.stdout:10: finishing write tid 1 to smithi04569269-10 2021-07-31T01:45:59.778 INFO:tasks.rados.rados.0.smithi045.stdout:7: finishing write tid 3 to smithi04569269-7 2021-07-31T01:45:59.778 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 7 v 2 (ObjNum 6 snap 0 seq_num 6) dirty exists 2021-07-31T01:45:59.778 INFO:tasks.rados.rados.0.smithi045.stdout:7: left oid 7 (ObjNum 6 snap 0 seq_num 6) 2021-07-31T01:45:59.778 INFO:tasks.rados.rados.0.smithi045.stdout:11: finishing write tid 1 to smithi04569269-11 2021-07-31T01:45:59.779 INFO:tasks.rados.rados.0.smithi045.stdout:8: finishing write tid 2 to smithi04569269-8 2021-07-31T01:45:59.779 INFO:tasks.rados.rados.0.smithi045.stdout:3: done (13 left) 2021-07-31T01:45:59.779 INFO:tasks.rados.rados.0.smithi045.stdout:4: done (12 left) 2021-07-31T01:45:59.779 INFO:tasks.rados.rados.0.smithi045.stdout:5: done (11 left) 2021-07-31T01:45:59.779 INFO:tasks.rados.rados.0.smithi045.stdout:7: done (10 left) 2021-07-31T01:45:59.780 INFO:tasks.rados.rados.0.smithi045.stdout:17: write initial oid 17 2021-07-31T01:45:59.780 INFO:tasks.rados.rados.0.smithi045.stdout:12: finishing write tid 1 to smithi04569269-12 2021-07-31T01:45:59.780 INFO:tasks.rados.rados.0.smithi045.stdout:10: finishing write tid 2 to smithi04569269-10 2021-07-31T01:45:59.780 INFO:tasks.rados.rados.0.smithi045.stdout:9: finishing write tid 2 to smithi04569269-9 2021-07-31T01:45:59.780 INFO:tasks.rados.rados.0.smithi045.stdout:17: seq_num 16 ranges {0=270336} 2021-07-31T01:45:59.782 INFO:tasks.rados.rados.0.smithi045.stdout:17: writing smithi04569269-17 from 0 to 270336 tid 1 2021-07-31T01:45:59.782 INFO:tasks.rados.rados.0.smithi045.stdout:18: write initial oid 18 2021-07-31T01:45:59.783 INFO:tasks.rados.rados.0.smithi045.stdout:18: seq_num 17 ranges {0=286720} 2021-07-31T01:45:59.783 INFO:tasks.rados.rados.0.smithi045.stdout:18: writing smithi04569269-18 from 0 to 286720 tid 1 2021-07-31T01:45:59.783 INFO:tasks.rados.rados.0.smithi045.stdout:19: write initial oid 19 2021-07-31T01:45:59.783 INFO:tasks.rados.rados.0.smithi045.stdout:19: seq_num 18 ranges {0=303104} 2021-07-31T01:45:59.783 INFO:tasks.rados.rados.0.smithi045.stdout:19: writing smithi04569269-19 from 0 to 303104 tid 1 2021-07-31T01:45:59.784 INFO:tasks.rados.rados.0.smithi045.stdout:20: write initial oid 20 2021-07-31T01:45:59.784 INFO:tasks.rados.rados.0.smithi045.stdout:20: seq_num 19 ranges {0=319488} 2021-07-31T01:45:59.784 INFO:tasks.rados.rados.0.smithi045.stdout:20: writing smithi04569269-20 from 0 to 319488 tid 1 2021-07-31T01:45:59.784 INFO:tasks.rados.rados.0.smithi045.stdout:21: write initial oid 21 2021-07-31T01:45:59.785 INFO:tasks.rados.rados.0.smithi045.stdout:21: seq_num 20 ranges {0=344064} 2021-07-31T01:45:59.787 INFO:tasks.rados.rados.0.smithi045.stdout:21: writing smithi04569269-21 from 0 to 344064 tid 1 2021-07-31T01:45:59.788 INFO:tasks.rados.rados.0.smithi045.stdout:11: finishing write tid 2 to smithi04569269-11 2021-07-31T01:45:59.788 INFO:tasks.rados.rados.0.smithi045.stdout:11: finishing write tid 3 to smithi04569269-11 2021-07-31T01:45:59.788 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 11 v 4 (ObjNum 10 snap 0 seq_num 10) dirty exists 2021-07-31T01:45:59.788 INFO:tasks.rados.rados.0.smithi045.stdout:11: left oid 11 (ObjNum 10 snap 0 seq_num 10) 2021-07-31T01:45:59.789 INFO:tasks.rados.rados.0.smithi045.stdout:8: finishing write tid 3 to smithi04569269-8 2021-07-31T01:45:59.789 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 8 v 4 (ObjNum 7 snap 0 seq_num 7) dirty exists 2021-07-31T01:45:59.789 INFO:tasks.rados.rados.0.smithi045.stdout:8: left oid 8 (ObjNum 7 snap 0 seq_num 7) 2021-07-31T01:45:59.789 INFO:tasks.rados.rados.0.smithi045.stdout:10: finishing write tid 3 to smithi04569269-10 2021-07-31T01:45:59.789 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 10 v 2 (ObjNum 9 snap 0 seq_num 9) dirty exists 2021-07-31T01:45:59.790 INFO:tasks.rados.rados.0.smithi045.stdout:10: left oid 10 (ObjNum 9 snap 0 seq_num 9) 2021-07-31T01:45:59.790 INFO:tasks.rados.rados.0.smithi045.stdout:9: finishing write tid 3 to smithi04569269-9 2021-07-31T01:45:59.790 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 9 v 4 (ObjNum 8 snap 0 seq_num 8) dirty exists 2021-07-31T01:45:59.790 INFO:tasks.rados.rados.0.smithi045.stdout:9: left oid 9 (ObjNum 8 snap 0 seq_num 8) 2021-07-31T01:45:59.791 INFO:tasks.rados.rados.0.smithi045.stdout:15: finishing write tid 1 to smithi04569269-15 2021-07-31T01:45:59.791 INFO:tasks.rados.rados.0.smithi045.stdout:6: finishing write tid 2 to smithi04569269-6 2021-07-31T01:45:59.791 INFO:tasks.rados.rados.0.smithi045.stdout:14: finishing write tid 1 to smithi04569269-14 2021-07-31T01:45:59.791 INFO:tasks.rados.rados.0.smithi045.stdout:13: finishing write tid 1 to smithi04569269-13 2021-07-31T01:45:59.791 INFO:tasks.rados.rados.0.smithi045.stdout:14: finishing write tid 2 to smithi04569269-14 2021-07-31T01:45:59.792 INFO:tasks.rados.rados.0.smithi045.stdout:15: finishing write tid 2 to smithi04569269-15 2021-07-31T01:45:59.792 INFO:tasks.rados.rados.0.smithi045.stdout:16: finishing write tid 1 to smithi04569269-16 2021-07-31T01:45:59.792 INFO:tasks.rados.rados.0.smithi045.stdout:12: finishing write tid 2 to smithi04569269-12 2021-07-31T01:45:59.792 INFO:tasks.rados.rados.0.smithi045.stdout:15: finishing write tid 3 to smithi04569269-15 2021-07-31T01:45:59.792 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 15 v 4 (ObjNum 14 snap 0 seq_num 14) dirty exists 2021-07-31T01:45:59.793 INFO:tasks.rados.rados.0.smithi045.stdout:15: left oid 15 (ObjNum 14 snap 0 seq_num 14) 2021-07-31T01:45:59.793 INFO:tasks.rados.rados.0.smithi045.stdout:16: finishing write tid 2 to smithi04569269-16 2021-07-31T01:45:59.793 INFO:tasks.rados.rados.0.smithi045.stdout:14: finishing write tid 3 to smithi04569269-14 2021-07-31T01:45:59.793 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 14 v 2 (ObjNum 13 snap 0 seq_num 13) dirty exists 2021-07-31T01:45:59.794 INFO:tasks.rados.rados.0.smithi045.stdout:14: left oid 14 (ObjNum 13 snap 0 seq_num 13) 2021-07-31T01:45:59.794 INFO:tasks.rados.rados.0.smithi045.stdout:13: finishing write tid 2 to smithi04569269-13 2021-07-31T01:45:59.794 INFO:tasks.rados.rados.0.smithi045.stdout:6: finishing write tid 3 to smithi04569269-6 2021-07-31T01:45:59.794 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 6 v 3 (ObjNum 5 snap 0 seq_num 5) dirty exists 2021-07-31T01:45:59.794 INFO:tasks.rados.rados.0.smithi045.stdout:6: left oid 6 (ObjNum 5 snap 0 seq_num 5) 2021-07-31T01:45:59.795 INFO:tasks.rados.rados.0.smithi045.stdout:16: finishing write tid 3 to smithi04569269-16 2021-07-31T01:45:59.795 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 16 v 2 (ObjNum 15 snap 0 seq_num 15) dirty exists 2021-07-31T01:45:59.795 INFO:tasks.rados.rados.0.smithi045.stdout:16: left oid 16 (ObjNum 15 snap 0 seq_num 15) 2021-07-31T01:45:59.795 INFO:tasks.rados.rados.0.smithi045.stdout:17: finishing write tid 1 to smithi04569269-17 2021-07-31T01:45:59.795 INFO:tasks.rados.rados.0.smithi045.stdout:12: finishing write tid 3 to smithi04569269-12 2021-07-31T01:45:59.796 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 12 v 5 (ObjNum 11 snap 0 seq_num 11) dirty exists 2021-07-31T01:45:59.796 INFO:tasks.rados.rados.0.smithi045.stdout:12: left oid 12 (ObjNum 11 snap 0 seq_num 11) 2021-07-31T01:45:59.796 INFO:tasks.rados.rados.0.smithi045.stdout:18: finishing write tid 1 to smithi04569269-18 2021-07-31T01:45:59.796 INFO:tasks.rados.rados.0.smithi045.stdout:13: finishing write tid 3 to smithi04569269-13 2021-07-31T01:45:59.796 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 13 v 6 (ObjNum 12 snap 0 seq_num 12) dirty exists 2021-07-31T01:45:59.797 INFO:tasks.rados.rados.0.smithi045.stdout:13: left oid 13 (ObjNum 12 snap 0 seq_num 12) 2021-07-31T01:45:59.797 INFO:tasks.rados.rados.0.smithi045.stdout:17: finishing write tid 2 to smithi04569269-17 2021-07-31T01:45:59.797 INFO:tasks.rados.rados.0.smithi045.stdout:18: finishing write tid 2 to smithi04569269-18 2021-07-31T01:45:59.797 INFO:tasks.rados.rados.0.smithi045.stdout:19: finishing write tid 1 to smithi04569269-19 2021-07-31T01:45:59.798 INFO:tasks.rados.rados.0.smithi045.stdout:17: finishing write tid 3 to smithi04569269-17 2021-07-31T01:45:59.798 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 17 v 4 (ObjNum 16 snap 0 seq_num 16) dirty exists 2021-07-31T01:45:59.798 INFO:tasks.rados.rados.0.smithi045.stdout:17: left oid 17 (ObjNum 16 snap 0 seq_num 16) 2021-07-31T01:45:59.798 INFO:tasks.rados.rados.0.smithi045.stdout:18: finishing write tid 3 to smithi04569269-18 2021-07-31T01:45:59.798 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 18 v 6 (ObjNum 17 snap 0 seq_num 17) dirty exists 2021-07-31T01:45:59.799 INFO:tasks.rados.rados.0.smithi045.stdout:18: left oid 18 (ObjNum 17 snap 0 seq_num 17) 2021-07-31T01:45:59.799 INFO:tasks.rados.rados.0.smithi045.stdout:19: finishing write tid 2 to smithi04569269-19 2021-07-31T01:45:59.799 INFO:tasks.rados.rados.0.smithi045.stdout:6: done (14 left) 2021-07-31T01:45:59.800 INFO:tasks.rados.rados.0.smithi045.stdout:8: done (13 left) 2021-07-31T01:45:59.800 INFO:tasks.rados.rados.0.smithi045.stdout:9: done (12 left) 2021-07-31T01:45:59.800 INFO:tasks.rados.rados.0.smithi045.stdout:10: done (11 left) 2021-07-31T01:45:59.800 INFO:tasks.rados.rados.0.smithi045.stdout:11: done (10 left) 2021-07-31T01:45:59.800 INFO:tasks.rados.rados.0.smithi045.stdout:12: done (9 left) 2021-07-31T01:45:59.801 INFO:tasks.rados.rados.0.smithi045.stdout:13: done (8 left) 2021-07-31T01:45:59.801 INFO:tasks.rados.rados.0.smithi045.stdout:14: done (7 left) 2021-07-31T01:45:59.801 INFO:tasks.rados.rados.0.smithi045.stdout:15: done (6 left) 2021-07-31T01:45:59.801 INFO:tasks.rados.rados.0.smithi045.stdout:16: done (5 left) 2021-07-31T01:45:59.801 INFO:tasks.rados.rados.0.smithi045.stdout:17: done (4 left) 2021-07-31T01:45:59.802 INFO:tasks.rados.rados.0.smithi045.stdout:18: done (3 left) 2021-07-31T01:45:59.802 INFO:tasks.rados.rados.0.smithi045.stdout:22: write initial oid 22 2021-07-31T01:45:59.802 INFO:tasks.rados.rados.0.smithi045.stdout:22: seq_num 21 ranges {0=360448} 2021-07-31T01:45:59.802 INFO:tasks.rados.rados.0.smithi045.stdout:22: writing smithi04569269-22 from 0 to 360448 tid 1 2021-07-31T01:45:59.803 INFO:tasks.rados.rados.0.smithi045.stdout:20: finishing write tid 1 to smithi04569269-20 2021-07-31T01:45:59.803 INFO:tasks.rados.rados.0.smithi045.stdout:19: finishing write tid 3 to smithi04569269-19 2021-07-31T01:45:59.803 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 19 v 4 (ObjNum 18 snap 0 seq_num 18) dirty exists 2021-07-31T01:45:59.803 INFO:tasks.rados.rados.0.smithi045.stdout:19: left oid 19 (ObjNum 18 snap 0 seq_num 18) 2021-07-31T01:45:59.803 INFO:tasks.rados.rados.0.smithi045.stdout:19: done (3 left) 2021-07-31T01:45:59.804 INFO:tasks.rados.rados.0.smithi045.stdout:23: write initial oid 23 2021-07-31T01:45:59.804 INFO:tasks.rados.rados.0.smithi045.stdout:23: seq_num 22 ranges {0=376832} 2021-07-31T01:45:59.807 INFO:tasks.rados.rados.0.smithi045.stdout:23: writing smithi04569269-23 from 0 to 376832 tid 1 2021-07-31T01:45:59.807 INFO:tasks.rados.rados.0.smithi045.stdout:20: finishing write tid 2 to smithi04569269-20 2021-07-31T01:45:59.807 INFO:tasks.rados.rados.0.smithi045.stdout:21: finishing write tid 1 to smithi04569269-21 2021-07-31T01:45:59.807 INFO:tasks.rados.rados.0.smithi045.stdout:24: write initial oid 24 2021-07-31T01:45:59.808 INFO:tasks.rados.rados.0.smithi045.stdout:24: seq_num 23 ranges {0=393216} 2021-07-31T01:45:59.808 INFO:tasks.rados.rados.0.smithi045.stdout:24: writing smithi04569269-24 from 0 to 393216 tid 1 2021-07-31T01:45:59.808 INFO:tasks.rados.rados.0.smithi045.stdout:20: finishing write tid 3 to smithi04569269-20 2021-07-31T01:45:59.808 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 20 v 6 (ObjNum 19 snap 0 seq_num 19) dirty exists 2021-07-31T01:45:59.808 INFO:tasks.rados.rados.0.smithi045.stdout:20: left oid 20 (ObjNum 19 snap 0 seq_num 19) 2021-07-31T01:45:59.809 INFO:tasks.rados.rados.0.smithi045.stdout:20: done (4 left) 2021-07-31T01:45:59.809 INFO:tasks.rados.rados.0.smithi045.stdout:25: write initial oid 25 2021-07-31T01:45:59.809 INFO:tasks.rados.rados.0.smithi045.stdout:22: finishing write tid 1 to smithi04569269-22 2021-07-31T01:45:59.809 INFO:tasks.rados.rados.0.smithi045.stdout:25: seq_num 24 ranges {0=409600} 2021-07-31T01:45:59.810 INFO:tasks.rados.rados.0.smithi045.stdout:25: writing smithi04569269-25 from 0 to 409600 tid 1 2021-07-31T01:45:59.810 INFO:tasks.rados.rados.0.smithi045.stdout:26: write initial oid 26 2021-07-31T01:45:59.810 INFO:tasks.rados.rados.0.smithi045.stdout:26: seq_num 25 ranges {0=425984} 2021-07-31T01:45:59.810 INFO:tasks.rados.rados.0.smithi045.stdout:26: writing smithi04569269-26 from 0 to 425984 tid 1 2021-07-31T01:45:59.810 INFO:tasks.rados.rados.0.smithi045.stdout:21: finishing write tid 2 to smithi04569269-21 2021-07-31T01:45:59.811 INFO:tasks.rados.rados.0.smithi045.stdout:23: finishing write tid 1 to smithi04569269-23 2021-07-31T01:45:59.811 INFO:tasks.rados.rados.0.smithi045.stdout:21: finishing write tid 3 to smithi04569269-21 2021-07-31T01:45:59.811 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 21 v 8 (ObjNum 20 snap 0 seq_num 20) dirty exists 2021-07-31T01:45:59.811 INFO:tasks.rados.rados.0.smithi045.stdout:21: left oid 21 (ObjNum 20 snap 0 seq_num 20) 2021-07-31T01:45:59.811 INFO:tasks.rados.rados.0.smithi045.stdout:23: finishing write tid 2 to smithi04569269-23 2021-07-31T01:45:59.812 INFO:tasks.rados.rados.0.smithi045.stdout:22: finishing write tid 2 to smithi04569269-22 2021-07-31T01:45:59.812 INFO:tasks.rados.rados.0.smithi045.stdout:24: finishing write tid 1 to smithi04569269-24 2021-07-31T01:45:59.812 INFO:tasks.rados.rados.0.smithi045.stdout:23: finishing write tid 3 to smithi04569269-23 2021-07-31T01:45:59.812 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 23 v 6 (ObjNum 22 snap 0 seq_num 22) dirty exists 2021-07-31T01:45:59.813 INFO:tasks.rados.rados.0.smithi045.stdout:23: left oid 23 (ObjNum 22 snap 0 seq_num 22) 2021-07-31T01:45:59.813 INFO:tasks.rados.rados.0.smithi045.stdout:22: finishing write tid 3 to smithi04569269-22 2021-07-31T01:45:59.813 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 22 v 6 (ObjNum 21 snap 0 seq_num 21) dirty exists 2021-07-31T01:45:59.813 INFO:tasks.rados.rados.0.smithi045.stdout:22: left oid 22 (ObjNum 21 snap 0 seq_num 21) 2021-07-31T01:45:59.813 INFO:tasks.rados.rados.0.smithi045.stdout:24: finishing write tid 2 to smithi04569269-24 2021-07-31T01:45:59.814 INFO:tasks.rados.rados.0.smithi045.stdout:21: done (5 left) 2021-07-31T01:45:59.814 INFO:tasks.rados.rados.0.smithi045.stdout:22: done (4 left) 2021-07-31T01:45:59.814 INFO:tasks.rados.rados.0.smithi045.stdout:23: done (3 left) 2021-07-31T01:45:59.814 INFO:tasks.rados.rados.0.smithi045.stdout:27: write initial oid 27 2021-07-31T01:45:59.814 INFO:tasks.rados.rados.0.smithi045.stdout:27: seq_num 26 ranges {0=442368} 2021-07-31T01:45:59.815 INFO:tasks.rados.rados.0.smithi045.stdout:27: writing smithi04569269-27 from 0 to 442368 tid 1 2021-07-31T01:45:59.815 INFO:tasks.rados.rados.0.smithi045.stdout:28: write initial oid 28 2021-07-31T01:45:59.815 INFO:tasks.rados.rados.0.smithi045.stdout:28: seq_num 27 ranges {0=458752} 2021-07-31T01:45:59.815 INFO:tasks.rados.rados.0.smithi045.stdout:28: writing smithi04569269-28 from 0 to 458752 tid 1 2021-07-31T01:45:59.816 INFO:tasks.rados.rados.0.smithi045.stdout:29: write initial oid 29 2021-07-31T01:45:59.816 INFO:tasks.rados.rados.0.smithi045.stdout:29: seq_num 28 ranges {0=475136} 2021-07-31T01:45:59.816 INFO:tasks.rados.rados.0.smithi045.stdout:29: writing smithi04569269-29 from 0 to 475136 tid 1 2021-07-31T01:45:59.816 INFO:tasks.rados.rados.0.smithi045.stdout:25: finishing write tid 1 to smithi04569269-25 2021-07-31T01:45:59.816 INFO:tasks.rados.rados.0.smithi045.stdout:24: finishing write tid 3 to smithi04569269-24 2021-07-31T01:45:59.817 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 24 v 2 (ObjNum 23 snap 0 seq_num 23) dirty exists 2021-07-31T01:45:59.817 INFO:tasks.rados.rados.0.smithi045.stdout:24: left oid 24 (ObjNum 23 snap 0 seq_num 23) 2021-07-31T01:45:59.817 INFO:tasks.rados.rados.0.smithi045.stdout:25: finishing write tid 2 to smithi04569269-25 2021-07-31T01:45:59.817 INFO:tasks.rados.rados.0.smithi045.stdout:25: finishing write tid 3 to smithi04569269-25 2021-07-31T01:45:59.817 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 25 v 2 (ObjNum 24 snap 0 seq_num 24) dirty exists 2021-07-31T01:45:59.818 INFO:tasks.rados.rados.0.smithi045.stdout:25: left oid 25 (ObjNum 24 snap 0 seq_num 24) 2021-07-31T01:45:59.818 INFO:tasks.rados.rados.0.smithi045.stdout:26: finishing write tid 1 to smithi04569269-26 2021-07-31T01:45:59.818 INFO:tasks.rados.rados.0.smithi045.stdout:26: finishing write tid 2 to smithi04569269-26 2021-07-31T01:45:59.818 INFO:tasks.rados.rados.0.smithi045.stdout:26: finishing write tid 3 to smithi04569269-26 2021-07-31T01:45:59.818 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 26 v 2 (ObjNum 25 snap 0 seq_num 25) dirty exists 2021-07-31T01:45:59.819 INFO:tasks.rados.rados.0.smithi045.stdout:26: left oid 26 (ObjNum 25 snap 0 seq_num 25) 2021-07-31T01:45:59.819 INFO:tasks.rados.rados.0.smithi045.stdout:27: finishing write tid 1 to smithi04569269-27 2021-07-31T01:45:59.819 INFO:tasks.rados.rados.0.smithi045.stdout:28: finishing write tid 1 to smithi04569269-28 2021-07-31T01:45:59.819 INFO:tasks.rados.rados.0.smithi045.stdout:27: finishing write tid 2 to smithi04569269-27 2021-07-31T01:45:59.819 INFO:tasks.rados.rados.0.smithi045.stdout:24: done (5 left) 2021-07-31T01:45:59.820 INFO:tasks.rados.rados.0.smithi045.stdout:25: done (4 left) 2021-07-31T01:45:59.820 INFO:tasks.rados.rados.0.smithi045.stdout:26: done (3 left) 2021-07-31T01:45:59.820 INFO:tasks.rados.rados.0.smithi045.stdout:30: write initial oid 30 2021-07-31T01:45:59.820 INFO:tasks.rados.rados.0.smithi045.stdout:30: seq_num 29 ranges {0=491520} 2021-07-31T01:45:59.822 INFO:tasks.rados.rados.0.smithi045.stdout:30: writing smithi04569269-30 from 0 to 491520 tid 1 2021-07-31T01:45:59.823 INFO:tasks.rados.rados.0.smithi045.stdout:31: write initial oid 31 2021-07-31T01:45:59.823 INFO:tasks.rados.rados.0.smithi045.stdout:31: seq_num 30 ranges {0=507904} 2021-07-31T01:45:59.823 INFO:tasks.rados.rados.0.smithi045.stdout:31: writing smithi04569269-31 from 0 to 507904 tid 1 2021-07-31T01:45:59.823 INFO:tasks.rados.rados.0.smithi045.stdout:28: finishing write tid 2 to smithi04569269-28 2021-07-31T01:45:59.824 INFO:tasks.rados.rados.0.smithi045.stdout:27: finishing write tid 3 to smithi04569269-27 2021-07-31T01:45:59.824 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 27 v 4 (ObjNum 26 snap 0 seq_num 26) dirty exists 2021-07-31T01:45:59.824 INFO:tasks.rados.rados.0.smithi045.stdout:27: left oid 27 (ObjNum 26 snap 0 seq_num 26) 2021-07-31T01:45:59.824 INFO:tasks.rados.rados.0.smithi045.stdout:28: finishing write tid 3 to smithi04569269-28 2021-07-31T01:45:59.824 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 28 v 6 (ObjNum 27 snap 0 seq_num 27) dirty exists 2021-07-31T01:45:59.825 INFO:tasks.rados.rados.0.smithi045.stdout:28: left oid 28 (ObjNum 27 snap 0 seq_num 27) 2021-07-31T01:45:59.825 INFO:tasks.rados.rados.0.smithi045.stdout:29: finishing write tid 1 to smithi04569269-29 2021-07-31T01:45:59.825 INFO:tasks.rados.rados.0.smithi045.stdout:30: finishing write tid 1 to smithi04569269-30 2021-07-31T01:45:59.825 INFO:tasks.rados.rados.0.smithi045.stdout:27: done (4 left) 2021-07-31T01:45:59.826 INFO:tasks.rados.rados.0.smithi045.stdout:28: done (3 left) 2021-07-31T01:45:59.826 INFO:tasks.rados.rados.0.smithi045.stdout:32: write initial oid 32 2021-07-31T01:45:59.826 INFO:tasks.rados.rados.0.smithi045.stdout:32: seq_num 31 ranges {0=524288} 2021-07-31T01:45:59.826 INFO:tasks.rados.rados.0.smithi045.stdout:32: writing smithi04569269-32 from 0 to 524288 tid 1 2021-07-31T01:45:59.826 INFO:tasks.rados.rados.0.smithi045.stdout:33: write initial oid 33 2021-07-31T01:45:59.827 INFO:tasks.rados.rados.0.smithi045.stdout:33: seq_num 32 ranges {0=540672} 2021-07-31T01:45:59.827 INFO:tasks.rados.rados.0.smithi045.stdout:33: writing smithi04569269-33 from 0 to 540672 tid 1 2021-07-31T01:45:59.827 INFO:tasks.rados.rados.0.smithi045.stdout:34: write initial oid 34 2021-07-31T01:45:59.827 INFO:tasks.rados.rados.0.smithi045.stdout:34: seq_num 33 ranges {0=557056} 2021-07-31T01:45:59.829 INFO:tasks.rados.rados.0.smithi045.stdout:34: writing smithi04569269-34 from 0 to 557056 tid 1 2021-07-31T01:45:59.829 INFO:tasks.rados.rados.0.smithi045.stdout:35: write initial oid 35 2021-07-31T01:45:59.829 INFO:tasks.rados.rados.0.smithi045.stdout:35: seq_num 34 ranges {0=573440} 2021-07-31T01:45:59.830 INFO:tasks.rados.rados.0.smithi045.stdout:35: writing smithi04569269-35 from 0 to 573440 tid 1 2021-07-31T01:45:59.830 INFO:tasks.rados.rados.0.smithi045.stdout:30: finishing write tid 2 to smithi04569269-30 2021-07-31T01:45:59.830 INFO:tasks.rados.rados.0.smithi045.stdout:29: finishing write tid 2 to smithi04569269-29 2021-07-31T01:45:59.830 INFO:tasks.rados.rados.0.smithi045.stdout:30: finishing write tid 3 to smithi04569269-30 2021-07-31T01:45:59.831 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 30 v 4 (ObjNum 29 snap 0 seq_num 29) dirty exists 2021-07-31T01:45:59.831 INFO:tasks.rados.rados.0.smithi045.stdout:30: left oid 30 (ObjNum 29 snap 0 seq_num 29) 2021-07-31T01:45:59.831 INFO:tasks.rados.rados.0.smithi045.stdout:29: finishing write tid 3 to smithi04569269-29 2021-07-31T01:45:59.831 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 29 v 2 (ObjNum 28 snap 0 seq_num 28) dirty exists 2021-07-31T01:45:59.831 INFO:tasks.rados.rados.0.smithi045.stdout:29: left oid 29 (ObjNum 28 snap 0 seq_num 28) 2021-07-31T01:45:59.832 INFO:tasks.rados.rados.0.smithi045.stdout:31: finishing write tid 1 to smithi04569269-31 2021-07-31T01:45:59.832 INFO:tasks.rados.rados.0.smithi045.stdout:31: finishing write tid 2 to smithi04569269-31 2021-07-31T01:45:59.832 INFO:tasks.rados.rados.0.smithi045.stdout:31: finishing write tid 3 to smithi04569269-31 2021-07-31T01:45:59.832 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 31 v 10 (ObjNum 30 snap 0 seq_num 30) dirty exists 2021-07-31T01:45:59.833 INFO:tasks.rados.rados.0.smithi045.stdout:31: left oid 31 (ObjNum 30 snap 0 seq_num 30) 2021-07-31T01:45:59.833 INFO:tasks.rados.rados.0.smithi045.stdout:32: finishing write tid 1 to smithi04569269-32 2021-07-31T01:45:59.833 INFO:tasks.rados.rados.0.smithi045.stdout:32: finishing write tid 2 to smithi04569269-32 2021-07-31T01:45:59.833 INFO:tasks.rados.rados.0.smithi045.stdout:32: finishing write tid 3 to smithi04569269-32 2021-07-31T01:45:59.833 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 32 v 4 (ObjNum 31 snap 0 seq_num 31) dirty exists 2021-07-31T01:45:59.834 INFO:tasks.rados.rados.0.smithi045.stdout:32: left oid 32 (ObjNum 31 snap 0 seq_num 31) 2021-07-31T01:45:59.834 INFO:tasks.rados.rados.0.smithi045.stdout:33: finishing write tid 1 to smithi04569269-33 2021-07-31T01:45:59.834 INFO:tasks.rados.rados.0.smithi045.stdout:33: finishing write tid 2 to smithi04569269-33 2021-07-31T01:45:59.834 INFO:tasks.rados.rados.0.smithi045.stdout:33: finishing write tid 3 to smithi04569269-33 2021-07-31T01:45:59.834 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 33 v 8 (ObjNum 32 snap 0 seq_num 32) dirty exists 2021-07-31T01:45:59.835 INFO:tasks.rados.rados.0.smithi045.stdout:33: left oid 33 (ObjNum 32 snap 0 seq_num 32) 2021-07-31T01:45:59.835 INFO:tasks.rados.rados.0.smithi045.stdout:34: finishing write tid 1 to smithi04569269-34 2021-07-31T01:45:59.835 INFO:tasks.rados.rados.0.smithi045.stdout:34: finishing write tid 2 to smithi04569269-34 2021-07-31T01:45:59.835 INFO:tasks.rados.rados.0.smithi045.stdout:29: done (6 left) 2021-07-31T01:45:59.836 INFO:tasks.rados.rados.0.smithi045.stdout:30: done (5 left) 2021-07-31T01:45:59.836 INFO:tasks.rados.rados.0.smithi045.stdout:31: done (4 left) 2021-07-31T01:45:59.836 INFO:tasks.rados.rados.0.smithi045.stdout:32: done (3 left) 2021-07-31T01:45:59.836 INFO:tasks.rados.rados.0.smithi045.stdout:33: done (2 left) 2021-07-31T01:45:59.836 INFO:tasks.rados.rados.0.smithi045.stdout:36: write initial oid 36 2021-07-31T01:45:59.837 INFO:tasks.rados.rados.0.smithi045.stdout:36: seq_num 35 ranges {0=589824} 2021-07-31T01:45:59.838 INFO:tasks.rados.rados.0.smithi045.stdout:36: writing smithi04569269-36 from 0 to 589824 tid 1 2021-07-31T01:45:59.838 INFO:tasks.rados.rados.0.smithi045.stdout:34: finishing write tid 3 to smithi04569269-34 2021-07-31T01:45:59.838 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 34 v 4 (ObjNum 33 snap 0 seq_num 33) dirty exists 2021-07-31T01:45:59.838 INFO:tasks.rados.rados.0.smithi045.stdout:34: left oid 34 (ObjNum 33 snap 0 seq_num 33) 2021-07-31T01:45:59.839 INFO:tasks.rados.rados.0.smithi045.stdout:35: finishing write tid 1 to smithi04569269-35 2021-07-31T01:45:59.839 INFO:tasks.rados.rados.0.smithi045.stdout:35: finishing write tid 2 to smithi04569269-35 2021-07-31T01:45:59.839 INFO:tasks.rados.rados.0.smithi045.stdout:34: done (2 left) 2021-07-31T01:45:59.839 INFO:tasks.rados.rados.0.smithi045.stdout:37: write initial oid 37 2021-07-31T01:45:59.839 INFO:tasks.rados.rados.0.smithi045.stdout:37: seq_num 36 ranges {0=606208} 2021-07-31T01:45:59.840 INFO:tasks.rados.rados.0.smithi045.stdout:37: writing smithi04569269-37 from 0 to 606208 tid 1 2021-07-31T01:45:59.840 INFO:tasks.rados.rados.0.smithi045.stdout:38: write initial oid 38 2021-07-31T01:45:59.840 INFO:tasks.rados.rados.0.smithi045.stdout:38: seq_num 37 ranges {0=622592} 2021-07-31T01:45:59.841 INFO:tasks.rados.rados.0.smithi045.stdout:38: writing smithi04569269-38 from 0 to 622592 tid 1 2021-07-31T01:45:59.841 INFO:tasks.rados.rados.0.smithi045.stdout:39: write initial oid 39 2021-07-31T01:45:59.842 INFO:tasks.rados.rados.0.smithi045.stdout:35: finishing write tid 3 to smithi04569269-35 2021-07-31T01:45:59.842 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 35 v 12 (ObjNum 34 snap 0 seq_num 34) dirty exists 2021-07-31T01:45:59.842 INFO:tasks.rados.rados.0.smithi045.stdout:35: left oid 35 (ObjNum 34 snap 0 seq_num 34) 2021-07-31T01:45:59.842 INFO:tasks.rados.rados.0.smithi045.stdout:36: finishing write tid 1 to smithi04569269-36 2021-07-31T01:45:59.842 INFO:tasks.rados.rados.0.smithi045.stdout:36: finishing write tid 2 to smithi04569269-36 2021-07-31T01:45:59.843 INFO:tasks.rados.rados.0.smithi045.stdout:36: finishing write tid 3 to smithi04569269-36 2021-07-31T01:45:59.843 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 36 v 14 (ObjNum 35 snap 0 seq_num 35) dirty exists 2021-07-31T01:45:59.843 INFO:tasks.rados.rados.0.smithi045.stdout:36: left oid 36 (ObjNum 35 snap 0 seq_num 35) 2021-07-31T01:45:59.843 INFO:tasks.rados.rados.0.smithi045.stdout:37: finishing write tid 1 to smithi04569269-37 2021-07-31T01:45:59.844 INFO:tasks.rados.rados.0.smithi045.stdout:39: seq_num 38 ranges {0=638976} 2021-07-31T01:45:59.844 INFO:tasks.rados.rados.0.smithi045.stdout:39: writing smithi04569269-39 from 0 to 638976 tid 1 2021-07-31T01:45:59.844 INFO:tasks.rados.rados.0.smithi045.stdout:35: done (4 left) 2021-07-31T01:45:59.845 INFO:tasks.rados.rados.0.smithi045.stdout:36: done (3 left) 2021-07-31T01:45:59.845 INFO:tasks.rados.rados.0.smithi045.stdout:40: write initial oid 40 2021-07-31T01:45:59.845 INFO:tasks.rados.rados.0.smithi045.stdout:40: seq_num 39 ranges {0=663552} 2021-07-31T01:45:59.849 INFO:tasks.rados.rados.0.smithi045.stdout:40: writing smithi04569269-40 from 0 to 663552 tid 1 2021-07-31T01:45:59.850 INFO:tasks.rados.rados.0.smithi045.stdout:41: write initial oid 41 2021-07-31T01:45:59.850 INFO:tasks.rados.rados.0.smithi045.stdout:41: seq_num 40 ranges {0=679936} 2021-07-31T01:45:59.854 INFO:tasks.rados.rados.0.smithi045.stdout:41: writing smithi04569269-41 from 0 to 679936 tid 1 2021-07-31T01:45:59.854 INFO:tasks.rados.rados.0.smithi045.stdout:42: write initial oid 42 2021-07-31T01:45:59.855 INFO:tasks.rados.rados.0.smithi045.stdout:42: seq_num 41 ranges {0=696320} 2021-07-31T01:45:59.859 INFO:tasks.rados.rados.0.smithi045.stdout:42: writing smithi04569269-42 from 0 to 696320 tid 1 2021-07-31T01:45:59.860 INFO:tasks.rados.rados.0.smithi045.stdout:37: finishing write tid 2 to smithi04569269-37 2021-07-31T01:45:59.860 INFO:tasks.rados.rados.0.smithi045.stdout:37: finishing write tid 3 to smithi04569269-37 2021-07-31T01:45:59.860 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 37 v 8 (ObjNum 36 snap 0 seq_num 36) dirty exists 2021-07-31T01:45:59.861 INFO:tasks.rados.rados.0.smithi045.stdout:37: left oid 37 (ObjNum 36 snap 0 seq_num 36) 2021-07-31T01:45:59.861 INFO:tasks.rados.rados.0.smithi045.stdout:38: finishing write tid 1 to smithi04569269-38 2021-07-31T01:45:59.862 INFO:tasks.rados.rados.0.smithi045.stdout:38: finishing write tid 2 to smithi04569269-38 2021-07-31T01:45:59.862 INFO:tasks.rados.rados.0.smithi045.stdout:38: finishing write tid 3 to smithi04569269-38 2021-07-31T01:45:59.863 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 38 v 10 (ObjNum 37 snap 0 seq_num 37) dirty exists 2021-07-31T01:45:59.864 INFO:tasks.rados.rados.0.smithi045.stdout:38: left oid 38 (ObjNum 37 snap 0 seq_num 37) 2021-07-31T01:45:59.866 INFO:tasks.rados.rados.0.smithi045.stdout:39: finishing write tid 1 to smithi04569269-39 2021-07-31T01:45:59.866 INFO:tasks.rados.rados.0.smithi045.stdout:39: finishing write tid 2 to smithi04569269-39 2021-07-31T01:45:59.867 INFO:tasks.rados.rados.0.smithi045.stdout:39: finishing write tid 3 to smithi04569269-39 2021-07-31T01:45:59.867 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 39 v 12 (ObjNum 38 snap 0 seq_num 38) dirty exists 2021-07-31T01:45:59.868 INFO:tasks.rados.rados.0.smithi045.stdout:39: left oid 39 (ObjNum 38 snap 0 seq_num 38) 2021-07-31T01:45:59.868 INFO:tasks.rados.rados.0.smithi045.stdout:40: finishing write tid 1 to smithi04569269-40 2021-07-31T01:45:59.869 INFO:tasks.rados.rados.0.smithi045.stdout:40: finishing write tid 2 to smithi04569269-40 2021-07-31T01:45:59.869 INFO:tasks.rados.rados.0.smithi045.stdout:40: finishing write tid 3 to smithi04569269-40 2021-07-31T01:45:59.870 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 40 v 8 (ObjNum 39 snap 0 seq_num 39) dirty exists 2021-07-31T01:45:59.870 INFO:tasks.rados.rados.0.smithi045.stdout:40: left oid 40 (ObjNum 39 snap 0 seq_num 39) 2021-07-31T01:45:59.870 INFO:tasks.rados.rados.0.smithi045.stdout:41: finishing write tid 1 to smithi04569269-41 2021-07-31T01:45:59.871 INFO:tasks.rados.rados.0.smithi045.stdout:41: finishing write tid 2 to smithi04569269-41 2021-07-31T01:45:59.871 INFO:tasks.rados.rados.0.smithi045.stdout:37: done (5 left) 2021-07-31T01:45:59.872 INFO:tasks.rados.rados.0.smithi045.stdout:38: done (4 left) 2021-07-31T01:45:59.872 INFO:tasks.rados.rados.0.smithi045.stdout:39: done (3 left) 2021-07-31T01:45:59.872 INFO:tasks.rados.rados.0.smithi045.stdout:40: done (2 left) 2021-07-31T01:45:59.873 INFO:tasks.rados.rados.0.smithi045.stdout:43: write initial oid 43 2021-07-31T01:45:59.873 INFO:tasks.rados.rados.0.smithi045.stdout:43: seq_num 42 ranges {0=712704} 2021-07-31T01:45:59.878 INFO:tasks.rados.rados.0.smithi045.stdout:43: writing smithi04569269-43 from 0 to 712704 tid 1 2021-07-31T01:45:59.878 INFO:tasks.rados.rados.0.smithi045.stdout:44: write initial oid 44 2021-07-31T01:45:59.879 INFO:tasks.rados.rados.0.smithi045.stdout:44: seq_num 43 ranges {0=729088} 2021-07-31T01:45:59.879 INFO:tasks.rados.rados.0.smithi045.stdout:44: writing smithi04569269-44 from 0 to 729088 tid 1 2021-07-31T01:45:59.880 INFO:tasks.rados.rados.0.smithi045.stdout:45: write initial oid 45 2021-07-31T01:45:59.880 INFO:tasks.rados.rados.0.smithi045.stdout:45: seq_num 44 ranges {0=745472} 2021-07-31T01:45:59.881 INFO:tasks.rados.rados.0.smithi045.stdout:45: writing smithi04569269-45 from 0 to 745472 tid 1 2021-07-31T01:45:59.881 INFO:tasks.rados.rados.0.smithi045.stdout:46: write initial oid 46 2021-07-31T01:45:59.881 INFO:tasks.rados.rados.0.smithi045.stdout:46: seq_num 45 ranges {0=761856} 2021-07-31T01:45:59.881 INFO:tasks.rados.rados.0.smithi045.stdout:46: writing smithi04569269-46 from 0 to 761856 tid 1 2021-07-31T01:45:59.882 INFO:tasks.rados.rados.0.smithi045.stdout:47: write initial oid 47 2021-07-31T01:45:59.882 INFO:tasks.rados.rados.0.smithi045.stdout:47: seq_num 46 ranges {0=778240} 2021-07-31T01:45:59.883 INFO:tasks.rados.rados.0.smithi045.stdout:47: writing smithi04569269-47 from 0 to 778240 tid 1 2021-07-31T01:45:59.883 INFO:tasks.rados.rados.0.smithi045.stdout:48: write initial oid 48 2021-07-31T01:45:59.883 INFO:tasks.rados.rados.0.smithi045.stdout:48: seq_num 47 ranges {0=794624} 2021-07-31T01:45:59.886 INFO:tasks.rados.rados.0.smithi045.stdout:48: writing smithi04569269-48 from 0 to 794624 tid 1 2021-07-31T01:45:59.886 INFO:tasks.rados.rados.0.smithi045.stdout:49: write initial oid 49 2021-07-31T01:45:59.887 INFO:tasks.rados.rados.0.smithi045.stdout:49: seq_num 48 ranges {0=409600,409600=401408} 2021-07-31T01:45:59.889 INFO:tasks.rados.rados.0.smithi045.stdout:49: writing smithi04569269-49 from 0 to 409600 tid 1 2021-07-31T01:45:59.891 INFO:tasks.rados.rados.0.smithi045.stdout:49: writing smithi04569269-49 from 409600 to 811008 tid 2 2021-07-31T01:45:59.892 INFO:tasks.rados.rados.0.smithi045.stdout:50: write initial oid 50 2021-07-31T01:45:59.892 INFO:tasks.rados.rados.0.smithi045.stdout:50: seq_num 49 ranges {0=425984,425984=401408} 2021-07-31T01:45:59.894 INFO:tasks.rados.rados.0.smithi045.stdout:50: writing smithi04569269-50 from 0 to 425984 tid 1 2021-07-31T01:45:59.897 INFO:tasks.rados.rados.0.smithi045.stdout:50: writing smithi04569269-50 from 425984 to 827392 tid 2 2021-07-31T01:45:59.897 INFO:tasks.rados.rados.0.smithi045.stdout:51: copy_from oid 15 from oid 25 current snap is 0 2021-07-31T01:45:59.897 INFO:tasks.rados.rados.0.smithi045.stdout:41: finishing write tid 3 to smithi04569269-41 2021-07-31T01:45:59.898 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 41 v 4 (ObjNum 40 snap 0 seq_num 40) dirty exists 2021-07-31T01:45:59.899 INFO:tasks.rados.rados.0.smithi045.stdout:41: left oid 41 (ObjNum 40 snap 0 seq_num 40) 2021-07-31T01:45:59.899 INFO:tasks.rados.rados.0.smithi045.stdout:41: done (10 left) 2021-07-31T01:45:59.899 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 23 current snap is 0 2021-07-31T01:45:59.900 INFO:tasks.rados.rados.0.smithi045.stdout:42: finishing write tid 1 to smithi04569269-42 2021-07-31T01:45:59.900 INFO:tasks.rados.rados.0.smithi045.stdout:52: seq_num 50 ranges {376832=442368,819200=401408} 2021-07-31T01:45:59.902 INFO:tasks.rados.rados.0.smithi045.stdout:52: writing smithi04569269-23 from 376832 to 819200 tid 1 2021-07-31T01:45:59.907 INFO:tasks.rados.rados.0.smithi045.stdout:52: writing smithi04569269-23 from 819200 to 1220608 tid 2 2021-07-31T01:45:59.908 INFO:tasks.rados.rados.0.smithi045.stdout:53: rmattr oid 2 current snap is 0 2021-07-31T01:45:59.908 INFO:tasks.rados.rados.0.smithi045.stdout:53: done (11 left) 2021-07-31T01:45:59.908 INFO:tasks.rados.rados.0.smithi045.stdout:54: delete oid 17 current snap is 0 2021-07-31T01:45:59.908 INFO:tasks.rados.rados.0.smithi045.stdout:42: finishing write tid 2 to smithi04569269-42 2021-07-31T01:45:59.908 INFO:tasks.rados.rados.0.smithi045.stdout:42: finishing write tid 3 to smithi04569269-42 2021-07-31T01:45:59.909 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 42 v 8 (ObjNum 41 snap 0 seq_num 41) dirty exists 2021-07-31T01:45:59.909 INFO:tasks.rados.rados.0.smithi045.stdout:42: left oid 42 (ObjNum 41 snap 0 seq_num 41) 2021-07-31T01:45:59.909 INFO:tasks.rados.rados.0.smithi045.stdout:43: finishing write tid 1 to smithi04569269-43 2021-07-31T01:45:59.909 INFO:tasks.rados.rados.0.smithi045.stdout:43: finishing write tid 2 to smithi04569269-43 2021-07-31T01:45:59.910 INFO:tasks.rados.rados.0.smithi045.stdout:43: finishing write tid 3 to smithi04569269-43 2021-07-31T01:45:59.910 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 43 v 6 (ObjNum 42 snap 0 seq_num 42) dirty exists 2021-07-31T01:45:59.910 INFO:tasks.rados.rados.0.smithi045.stdout:43: left oid 43 (ObjNum 42 snap 0 seq_num 42) 2021-07-31T01:45:59.910 INFO:tasks.rados.rados.0.smithi045.stdout:44: finishing write tid 1 to smithi04569269-44 2021-07-31T01:45:59.911 INFO:tasks.rados.rados.0.smithi045.stdout:44: finishing write tid 2 to smithi04569269-44 2021-07-31T01:45:59.911 INFO:tasks.rados.rados.0.smithi045.stdout:44: finishing write tid 3 to smithi04569269-44 2021-07-31T01:45:59.911 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 44 v 8 (ObjNum 43 snap 0 seq_num 43) dirty exists 2021-07-31T01:45:59.912 INFO:tasks.rados.rados.0.smithi045.stdout:44: left oid 44 (ObjNum 43 snap 0 seq_num 43) 2021-07-31T01:45:59.912 INFO:tasks.rados.rados.0.smithi045.stdout:45: finishing write tid 1 to smithi04569269-45 2021-07-31T01:45:59.912 INFO:tasks.rados.rados.0.smithi045.stdout:45: finishing write tid 2 to smithi04569269-45 2021-07-31T01:45:59.913 INFO:tasks.rados.rados.0.smithi045.stdout:45: finishing write tid 3 to smithi04569269-45 2021-07-31T01:45:59.913 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 45 v 10 (ObjNum 44 snap 0 seq_num 44) dirty exists 2021-07-31T01:45:59.913 INFO:tasks.rados.rados.0.smithi045.stdout:45: left oid 45 (ObjNum 44 snap 0 seq_num 44) 2021-07-31T01:45:59.913 INFO:tasks.rados.rados.0.smithi045.stdout:46: finishing write tid 1 to smithi04569269-46 2021-07-31T01:45:59.913 INFO:tasks.rados.rados.0.smithi045.stdout:46: finishing write tid 2 to smithi04569269-46 2021-07-31T01:45:59.914 INFO:tasks.rados.rados.0.smithi045.stdout:46: finishing write tid 3 to smithi04569269-46 2021-07-31T01:45:59.914 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 46 v 2 (ObjNum 45 snap 0 seq_num 45) dirty exists 2021-07-31T01:45:59.914 INFO:tasks.rados.rados.0.smithi045.stdout:46: left oid 46 (ObjNum 45 snap 0 seq_num 45) 2021-07-31T01:45:59.914 INFO:tasks.rados.rados.0.smithi045.stdout:47: finishing write tid 1 to smithi04569269-47 2021-07-31T01:45:59.915 INFO:tasks.rados.rados.0.smithi045.stdout:47: finishing write tid 2 to smithi04569269-47 2021-07-31T01:45:59.915 INFO:tasks.rados.rados.0.smithi045.stdout:47: finishing write tid 3 to smithi04569269-47 2021-07-31T01:45:59.915 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 47 v 4 (ObjNum 46 snap 0 seq_num 46) dirty exists 2021-07-31T01:45:59.915 INFO:tasks.rados.rados.0.smithi045.stdout:47: left oid 47 (ObjNum 46 snap 0 seq_num 46) 2021-07-31T01:45:59.916 INFO:tasks.rados.rados.0.smithi045.stdout:48: finishing write tid 1 to smithi04569269-48 2021-07-31T01:45:59.916 INFO:tasks.rados.rados.0.smithi045.stdout:49: finishing write tid 1 to smithi04569269-49 2021-07-31T01:45:59.916 INFO:tasks.rados.rados.0.smithi045.stdout:48: finishing write tid 2 to smithi04569269-48 2021-07-31T01:45:59.916 INFO:tasks.rados.rados.0.smithi045.stdout:48: finishing write tid 3 to smithi04569269-48 2021-07-31T01:45:59.917 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 48 v 6 (ObjNum 47 snap 0 seq_num 47) dirty exists 2021-07-31T01:45:59.917 INFO:tasks.rados.rados.0.smithi045.stdout:48: left oid 48 (ObjNum 47 snap 0 seq_num 47) 2021-07-31T01:45:59.917 INFO:tasks.rados.rados.0.smithi045.stdout:49: finishing write tid 2 to smithi04569269-49 2021-07-31T01:45:59.917 INFO:tasks.rados.rados.0.smithi045.stdout:49: finishing write tid 3 to smithi04569269-49 2021-07-31T01:45:59.918 INFO:tasks.rados.rados.0.smithi045.stdout:49: finishing write tid 4 to smithi04569269-49 2021-07-31T01:45:59.918 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 49 v 17 (ObjNum 48 snap 0 seq_num 48) dirty exists 2021-07-31T01:45:59.918 INFO:tasks.rados.rados.0.smithi045.stdout:49: left oid 49 (ObjNum 48 snap 0 seq_num 48) 2021-07-31T01:45:59.918 INFO:tasks.rados.rados.0.smithi045.stdout:50: finishing write tid 1 to smithi04569269-50 2021-07-31T01:45:59.918 INFO:tasks.rados.rados.0.smithi045.stdout:50: finishing write tid 2 to smithi04569269-50 2021-07-31T01:45:59.919 INFO:tasks.rados.rados.0.smithi045.stdout:50: finishing write tid 3 to smithi04569269-50 2021-07-31T01:45:59.919 INFO:tasks.rados.rados.0.smithi045.stdout:50: finishing write tid 4 to smithi04569269-50 2021-07-31T01:45:59.919 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 50 v 13 (ObjNum 49 snap 0 seq_num 49) dirty exists 2021-07-31T01:45:59.919 INFO:tasks.rados.rados.0.smithi045.stdout:50: left oid 50 (ObjNum 49 snap 0 seq_num 49) 2021-07-31T01:45:59.919 INFO:tasks.rados.rados.0.smithi045.stdout:51: finishing copy_from to smithi04569269-15 2021-07-31T01:45:59.920 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 15 v 2 (ObjNum 24 snap 0 seq_num 24) dirty exists 2021-07-31T01:45:59.920 INFO:tasks.rados.rados.0.smithi045.stdout:51: finishing copy_from racing read to smithi04569269-15 2021-07-31T01:45:59.920 INFO:tasks.rados.rados.0.smithi045.stdout:52: finishing write tid 1 to smithi04569269-23 2021-07-31T01:45:59.921 INFO:tasks.rados.rados.0.smithi045.stdout:42: done (11 left) 2021-07-31T01:45:59.921 INFO:tasks.rados.rados.0.smithi045.stdout:43: done (10 left) 2021-07-31T01:45:59.921 INFO:tasks.rados.rados.0.smithi045.stdout:44: done (9 left) 2021-07-31T01:45:59.921 INFO:tasks.rados.rados.0.smithi045.stdout:45: done (8 left) 2021-07-31T01:45:59.921 INFO:tasks.rados.rados.0.smithi045.stdout:46: done (7 left) 2021-07-31T01:45:59.922 INFO:tasks.rados.rados.0.smithi045.stdout:47: done (6 left) 2021-07-31T01:45:59.922 INFO:tasks.rados.rados.0.smithi045.stdout:48: done (5 left) 2021-07-31T01:45:59.922 INFO:tasks.rados.rados.0.smithi045.stdout:49: done (4 left) 2021-07-31T01:45:59.923 INFO:tasks.rados.rados.0.smithi045.stdout:50: done (3 left) 2021-07-31T01:45:59.923 INFO:tasks.rados.rados.0.smithi045.stdout:51: done (2 left) 2021-07-31T01:45:59.923 INFO:tasks.rados.rados.0.smithi045.stdout:54: done (1 left) 2021-07-31T01:45:59.923 INFO:tasks.rados.rados.0.smithi045.stdout:55: read oid 15 snap -1 2021-07-31T01:45:59.924 INFO:tasks.rados.rados.0.smithi045.stdout:55: expect (ObjNum 24 snap 0 seq_num 24) 2021-07-31T01:45:59.924 INFO:tasks.rados.rados.0.smithi045.stdout:56: rollback oid 31 current snap is 0 2021-07-31T01:45:59.924 INFO:tasks.rados.rados.0.smithi045.stdout:56: done (2 left) 2021-07-31T01:45:59.924 INFO:tasks.rados.rados.0.smithi045.stdout:57: read oid 28 snap -1 2021-07-31T01:45:59.924 INFO:tasks.rados.rados.0.smithi045.stdout:57: expect (ObjNum 27 snap 0 seq_num 27) 2021-07-31T01:45:59.925 INFO:tasks.rados.rados.0.smithi045.stdout:58: copy_from oid 29 from oid 47 current snap is 0 2021-07-31T01:45:59.925 INFO:tasks.rados.rados.0.smithi045.stdout:59: rollback oid 45 current snap is 0 2021-07-31T01:45:59.925 INFO:tasks.rados.rados.0.smithi045.stdout:59: done (4 left) 2021-07-31T01:45:59.926 INFO:tasks.rados.rados.0.smithi045.stdout:60: copy_from oid 22 from oid 48 current snap is 0 2021-07-31T01:45:59.926 INFO:tasks.rados.rados.0.smithi045.stdout:61: delete oid 1 current snap is 0 2021-07-31T01:45:59.929 INFO:tasks.rados.rados.0.smithi045.stdout:52: finishing write tid 2 to smithi04569269-23 2021-07-31T01:45:59.930 INFO:tasks.rados.rados.0.smithi045.stdout:52: finishing write tid 3 to smithi04569269-23 2021-07-31T01:45:59.930 INFO:tasks.rados.rados.0.smithi045.stdout:52: finishing write tid 4 to smithi04569269-23 2021-07-31T01:45:59.931 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 23 v 15 (ObjNum 50 snap 0 seq_num 50) dirty exists 2021-07-31T01:45:59.932 INFO:tasks.rados.rados.0.smithi045.stdout:52: left oid 23 (ObjNum 50 snap 0 seq_num 50) 2021-07-31T01:45:59.932 INFO:tasks.rados.rados.0.smithi045.stdout:52: done (5 left) 2021-07-31T01:45:59.932 INFO:tasks.rados.rados.0.smithi045.stdout:61: done (4 left) 2021-07-31T01:45:59.932 INFO:tasks.rados.rados.0.smithi045.stdout:62: read oid 24 snap -1 2021-07-31T01:45:59.933 INFO:tasks.rados.rados.0.smithi045.stdout:62: expect (ObjNum 23 snap 0 seq_num 23) 2021-07-31T01:45:59.933 INFO:tasks.rados.rados.0.smithi045.stdout:63: snap_create 2021-07-31T01:45:59.933 INFO:tasks.rados.rados.0.smithi045.stdout:55: expect (ObjNum 24 snap 0 seq_num 24) 2021-07-31T01:45:59.934 INFO:tasks.rados.rados.0.smithi045.stdout:57: expect (ObjNum 27 snap 0 seq_num 27) 2021-07-31T01:45:59.942 INFO:tasks.rados.rados.0.smithi045.stdout:60: finishing copy_from to smithi04569269-22 2021-07-31T01:45:59.943 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 22 v 6 (ObjNum 47 snap 0 seq_num 47) dirty exists 2021-07-31T01:45:59.943 INFO:tasks.rados.rados.0.smithi045.stdout:60: finishing copy_from racing read to smithi04569269-22 2021-07-31T01:45:59.943 INFO:tasks.rados.rados.0.smithi045.stdout:62: expect (ObjNum 23 snap 0 seq_num 23) 2021-07-31T01:45:59.953 INFO:tasks.rados.rados.0.smithi045.stdout:58: finishing copy_from to smithi04569269-29 2021-07-31T01:45:59.953 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 29 v 4 (ObjNum 46 snap 0 seq_num 46) dirty exists 2021-07-31T01:45:59.953 INFO:tasks.rados.rados.0.smithi045.stdout:58: finishing copy_from racing read to smithi04569269-29 2021-07-31T01:46:00.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:59 smithi154 conmon[34553]: audit 2021-07-31T01:45:58.997254+0000 mon.c (mon.1) 18 : audit [INF] from='client.? 172.21.15.45:0/2361293534' entity='client.0' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "write_fadvise_dontneed", "val": "false"}]: dispatch 2021-07-31T01:46:00.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:59 smithi154 conmon[34553]: audit 2021-07-31T 2021-07-31T01:46:00.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:45:59 smithi154 conmon[34553]: 01:45:58.997570+0000 mon.a (mon.0) 655 : audit [INF] from='client.? ' entity='client.0' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "write_fadvise_dontneed", "val": "false"}]: dispatch 2021-07-31T01:46:00.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:59 smithi045 conmon[33655]: audit 2021-07-31T01:45:58 2021-07-31T01:46:00.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:59 smithi045 conmon[33655]: .997254+0000 mon.c (mon.1) 18 : audit [INF] from='client.? 172.21.15.45:0/2361293534' entity='client.0' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "write_fadvise_dontneed", "val": "false"}]: dispatch 2021-07-31T01:46:00.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:45:59 smithi045 conmon[33655]: audit 2021-07-31T01:45:58.997570+0000 mon.a (mon.0) 655 : audit [INF] from='client.? ' entity='client.0' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "write_fadvise_dontneed", "val": "false"}]: dispatch 2021-07-31T01:46:00.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:59 smithi045 conmon[39718]: audit 2021-07-31T01:45:58. 2021-07-31T01:46:00.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:59 smithi045 conmon[39718]: 997254+0000 mon.c (mon.1) 18 : audit [INF] from='client.? 172.21.15.45:0/2361293534' entity='client.0' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "write_fadvise_dontneed", "val": "false"}]: dispatch 2021-07-31T01:46:00.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:59 smithi045 conmon[39718]: audit 2021-07-31T01:46:00.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:59 smithi045 conmon[39718]: 2021-07-31T01:45:58.997570+0000 mon.a (mon.0 2021-07-31T01:46:00.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:45:59 smithi045 conmon[39718]: ) 655 : audit [INF] from='client.? ' entity='client.0' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "write_fadvise_dontneed", "val": "false"}]: dispatch 2021-07-31T01:46:00.761 INFO:tasks.rados.rados.0.smithi045.stdout:55: done (5 left) 2021-07-31T01:46:00.761 INFO:tasks.rados.rados.0.smithi045.stdout:57: done (4 left) 2021-07-31T01:46:00.761 INFO:tasks.rados.rados.0.smithi045.stdout:58: done (3 left) 2021-07-31T01:46:00.762 INFO:tasks.rados.rados.0.smithi045.stdout:60: done (2 left) 2021-07-31T01:46:00.763 INFO:tasks.rados.rados.0.smithi045.stdout:62: done (1 left) 2021-07-31T01:46:00.763 INFO:tasks.rados.rados.0.smithi045.stdout:63: done (0 left) 2021-07-31T01:46:00.763 INFO:tasks.rados.rados.0.smithi045.stdout:64: copy_from oid 15 from oid 2 current snap is 1 2021-07-31T01:46:00.763 INFO:tasks.rados.rados.0.smithi045.stdout:65: read oid 30 snap -1 2021-07-31T01:46:00.764 INFO:tasks.rados.rados.0.smithi045.stdout:65: expect (ObjNum 29 snap 0 seq_num 29) 2021-07-31T01:46:00.764 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 23 current snap is 1 2021-07-31T01:46:00.764 INFO:tasks.rados.rados.0.smithi045.stdout:66: seq_num 53 ranges {1220608=491520,1712128=401408} 2021-07-31T01:46:00.770 INFO:tasks.rados.rados.0.smithi045.stdout:66: writing smithi04569269-23 from 1220608 to 1712128 tid 1 2021-07-31T01:46:00.772 INFO:tasks.rados.rados.0.smithi045.stdout:66: writing smithi04569269-23 from 1712128 to 2113536 tid 2 2021-07-31T01:46:00.773 INFO:tasks.rados.rados.0.smithi045.stdout:67: rollback oid 10 current snap is 1 2021-07-31T01:46:00.773 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 10 to 0 2021-07-31T01:46:00.773 INFO:tasks.rados.rados.0.smithi045.stdout:64: finishing copy_from to smithi04569269-15 2021-07-31T01:46:00.773 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 15 v 2 (ObjNum 1 snap 0 seq_num 1) dirty exists 2021-07-31T01:46:00.773 INFO:tasks.rados.rados.0.smithi045.stdout:64: finishing copy_from racing read to smithi04569269-15 2021-07-31T01:46:00.774 INFO:tasks.rados.rados.0.smithi045.stdout:65: expect (ObjNum 29 snap 0 seq_num 29) 2021-07-31T01:46:00.786 INFO:tasks.rados.rados.0.smithi045.stdout:67: finishing rollback tid 0 to smithi04569269-10 2021-07-31T01:46:00.786 INFO:tasks.rados.rados.0.smithi045.stdout:67: finishing rollback tid 1 to smithi04569269-10 2021-07-31T01:46:00.787 INFO:tasks.rados.rados.0.smithi045.stdout:67: finishing rollback tid 2 to smithi04569269-10 2021-07-31T01:46:00.787 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 10 v 18 (ObjNum 9 snap 0 seq_num 9) dirty exists 2021-07-31T01:46:00.787 INFO:tasks.rados.rados.0.smithi045.stdout:66: finishing write tid 1 to smithi04569269-23 2021-07-31T01:46:00.787 INFO:tasks.rados.rados.0.smithi045.stdout:64: done (3 left) 2021-07-31T01:46:00.788 INFO:tasks.rados.rados.0.smithi045.stdout:65: done (2 left) 2021-07-31T01:46:00.788 INFO:tasks.rados.rados.0.smithi045.stdout:67: done (1 left) 2021-07-31T01:46:00.788 INFO:tasks.rados.rados.0.smithi045.stdout:68: setattr oid 31 current snap is 1 2021-07-31T01:46:00.788 INFO:tasks.rados.rados.0.smithi045.stdout:66: finishing write tid 2 to smithi04569269-23 2021-07-31T01:46:00.788 INFO:tasks.rados.rados.0.smithi045.stdout:66: finishing write tid 3 to smithi04569269-23 2021-07-31T01:46:00.789 INFO:tasks.rados.rados.0.smithi045.stdout:66: finishing write tid 4 to smithi04569269-23 2021-07-31T01:46:00.789 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 23 v 19 (ObjNum 53 snap 1 seq_num 53) dirty exists 2021-07-31T01:46:00.789 INFO:tasks.rados.rados.0.smithi045.stdout:66: left oid 23 (ObjNum 53 snap 1 seq_num 53) 2021-07-31T01:46:00.791 INFO:tasks.rados.rados.0.smithi045.stdout:66: done (1 left) 2021-07-31T01:46:00.791 INFO:tasks.rados.rados.0.smithi045.stdout:69: copy_from oid 45 from oid 34 current snap is 1 2021-07-31T01:46:00.791 INFO:tasks.rados.rados.0.smithi045.stdout:70: read oid 23 snap 0 2021-07-31T01:46:00.792 INFO:tasks.rados.rados.0.smithi045.stdout:70: expect (ObjNum 50 snap 0 seq_num 50) 2021-07-31T01:46:00.792 INFO:tasks.rados.rados.0.smithi045.stdout:71: copy_from oid 36 from oid 27 current snap is 1 2021-07-31T01:46:00.792 INFO:tasks.rados.rados.0.smithi045.stdout:72: rollback oid 49 current snap is 1 2021-07-31T01:46:00.792 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 49 to 0 2021-07-31T01:46:00.793 INFO:tasks.rados.rados.0.smithi045.stdout:73: read oid 4 snap 0 2021-07-31T01:46:00.793 INFO:tasks.rados.rados.0.smithi045.stdout:73: expect (ObjNum 3 snap 0 seq_num 3) 2021-07-31T01:46:00.793 INFO:tasks.rados.rados.0.smithi045.stdout:74: rmattr oid 7 current snap is 1 2021-07-31T01:46:00.793 INFO:tasks.rados.rados.0.smithi045.stdout:74: done (6 left) 2021-07-31T01:46:00.793 INFO:tasks.rados.rados.0.smithi045.stdout:75: copy_from oid 20 from oid 12 current snap is 1 2021-07-31T01:46:00.794 INFO:tasks.rados.rados.0.smithi045.stdout:76: read oid 39 snap -1 2021-07-31T01:46:00.794 INFO:tasks.rados.rados.0.smithi045.stdout:76: expect (ObjNum 38 snap 0 seq_num 38) 2021-07-31T01:46:00.794 INFO:tasks.rados.rados.0.smithi045.stdout:77: copy_from oid 6 from oid 37 current snap is 1 2021-07-31T01:46:00.794 INFO:tasks.rados.rados.0.smithi045.stdout:78: read oid 2 snap -1 2021-07-31T01:46:00.795 INFO:tasks.rados.rados.0.smithi045.stdout:78: expect (ObjNum 1 snap 0 seq_num 1) 2021-07-31T01:46:00.795 INFO:tasks.rados.rados.0.smithi045.stdout:79: setattr oid 41 current snap is 1 2021-07-31T01:46:00.795 INFO:tasks.rados.rados.0.smithi045.stdout:80: snap_create 2021-07-31T01:46:00.795 INFO:tasks.rados.rados.0.smithi045.stdout:78: expect (ObjNum 1 snap 0 seq_num 1) 2021-07-31T01:46:00.796 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 31 v 19 (ObjNum 30 snap 0 seq_num 30) dirty exists 2021-07-31T01:46:00.796 INFO:tasks.rados.rados.0.smithi045.stdout:73: expect (ObjNum 3 snap 0 seq_num 3) 2021-07-31T01:46:00.797 INFO:tasks.rados.rados.0.smithi045.stdout:72: finishing rollback tid 0 to smithi04569269-49 2021-07-31T01:46:00.797 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 41 v 6 (ObjNum 40 snap 0 seq_num 40) dirty exists 2021-07-31T01:46:00.797 INFO:tasks.rados.rados.0.smithi045.stdout:72: finishing rollback tid 1 to smithi04569269-49 2021-07-31T01:46:00.797 INFO:tasks.rados.rados.0.smithi045.stdout:72: finishing rollback tid 2 to smithi04569269-49 2021-07-31T01:46:00.797 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 49 v 23 (ObjNum 48 snap 0 seq_num 48) dirty exists 2021-07-31T01:46:00.803 INFO:tasks.rados.rados.0.smithi045.stdout:69: finishing copy_from to smithi04569269-45 2021-07-31T01:46:00.803 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 45 v 4 (ObjNum 33 snap 0 seq_num 33) dirty exists 2021-07-31T01:46:00.803 INFO:tasks.rados.rados.0.smithi045.stdout:75: finishing copy_from to smithi04569269-20 2021-07-31T01:46:00.804 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 20 v 5 (ObjNum 11 snap 0 seq_num 11) dirty exists 2021-07-31T01:46:00.804 INFO:tasks.rados.rados.0.smithi045.stdout:69: finishing copy_from racing read to smithi04569269-45 2021-07-31T01:46:00.804 INFO:tasks.rados.rados.0.smithi045.stdout:75: finishing copy_from racing read to smithi04569269-20 2021-07-31T01:46:00.804 INFO:tasks.rados.rados.0.smithi045.stdout:77: finishing copy_from to smithi04569269-6 2021-07-31T01:46:00.804 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 6 v 8 (ObjNum 36 snap 0 seq_num 36) dirty exists 2021-07-31T01:46:00.805 INFO:tasks.rados.rados.0.smithi045.stdout:77: finishing copy_from racing read to smithi04569269-6 2021-07-31T01:46:00.805 INFO:tasks.rados.rados.0.smithi045.stdout:71: finishing copy_from to smithi04569269-36 2021-07-31T01:46:00.805 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 36 v 4 (ObjNum 26 snap 0 seq_num 26) dirty exists 2021-07-31T01:46:00.806 INFO:tasks.rados.rados.0.smithi045.stdout:71: finishing copy_from racing read to smithi04569269-36 2021-07-31T01:46:00.809 INFO:tasks.rados.rados.0.smithi045.stdout:70: expect (ObjNum 50 snap 0 seq_num 50) 2021-07-31T01:46:00.846 INFO:tasks.rados.rados.0.smithi045.stdout:76: expect (ObjNum 38 snap 0 seq_num 38) 2021-07-31T01:46:01.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:00 smithi045 conmon[33655]: cluster 2021-07-31T01:45: 2021-07-31T01:46:01.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:00 smithi045 conmon[33655]: 59.363054+0000 mgr.y (mgr.14146) 273 : cluster [DBG] pgmap v239: 25 pgs: 25 active+clean; 0 B data, 44 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:46:01.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:00 smithi045 conmon[33655]: audit 2021-07-31T 2021-07-31T01:46:01.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:00 smithi045 conmon[33655]: 01:45:59.753546+0000 mon.a (mon.0) 656 : audit [INF] 2021-07-31T01:46:01.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:00 smithi045 conmon[33655]: from='client.? ' entity='client.0' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "write_fadvise_dontneed", "val": "false"}]': finished 2021-07-31T01:46:01.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:00 smithi045 conmon[33655]: cluster 2021-07-31 2021-07-31T01:46:01.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:00 smithi045 conmon[33655]: T01:45:59.753644+0000 mon.a (mon.0) 657 2021-07-31T01:46:01.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:00 smithi045 conmon[33655]: : cluster [DBG] osdmap e60: 8 total, 8 up, 8 in 2021-07-31T01:46:01.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:00 smithi045 conmon[39718]: cluster 2021-07- 2021-07-31T01:46:01.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:00 smithi045 conmon[39718]: 31T01:45:59.363054+0000 mgr.y (mgr.14146) 273 : cluster [DBG] pgmap v239: 25 pgs: 25 active+clean; 0 B data, 44 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:46:01.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:00 smithi045 conmon[39718]: audit 2021-07-31T01:46:01.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:00 smithi045 conmon[39718]: 2021-07-31T01 2021-07-31T01:46:01.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:00 smithi045 conmon[39718]: :45:59.753546+0000 mon.a (mon.0 2021-07-31T01:46:01.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:00 smithi045 conmon[39718]: ) 656 : audit [INF] from='client.? ' entity='client.0' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "write_fadvise_dontneed", "val": "false"}]': finished 2021-07-31T01:46:01.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:00 smithi045 conmon[39718]: cluster 2021-07-31T01:46:01.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:00 smithi045 conmon[39718]: 2021-07-31 2021-07-31T01:46:01.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:00 smithi045 conmon[39718]: T01:45:59.753644+0000 mon.a 2021-07-31T01:46:01.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:00 smithi045 conmon[39718]: (mon.0) 657 : cluster [DBG] osdmap e60: 8 total, 8 up, 8 in 2021-07-31T01:46:01.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:00 smithi045 conmon[39718]: 2021-07-31T01:46:01.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:00 smithi154 conmon[34553]: cluster 2021-07-31T01:45:59.363054+0000 mgr.y (mgr.14146) 273 : cluster [DBG] pgmap v239: 25 pgs: 25 active+clean; 0 B data, 44 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:46:01.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:00 smithi154 conmon[34553]: audit 2021-07-31T01:45:59.753546+0000 mon.a (mon.0) 656 : audit [INF] from='client.? ' entity='client.0' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "write_fadvise_dontneed", "val": "false"}]': finished 2021-07-31T01:46:01.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:00 smithi154 conmon[34553]: cluster 2021-07-31T01:45:59.753644+0000 mon.a (mon.0) 657 : cluster [DBG] osdmap e60: 8 total, 8 up, 8 in 2021-07-31T01:46:01.767 INFO:tasks.rados.rados.0.smithi045.stdout:68: done (11 left) 2021-07-31T01:46:01.767 INFO:tasks.rados.rados.0.smithi045.stdout:69: done (10 left) 2021-07-31T01:46:01.768 INFO:tasks.rados.rados.0.smithi045.stdout:70: done (9 left) 2021-07-31T01:46:01.768 INFO:tasks.rados.rados.0.smithi045.stdout:71: done (8 left) 2021-07-31T01:46:01.769 INFO:tasks.rados.rados.0.smithi045.stdout:72: done (7 left) 2021-07-31T01:46:01.769 INFO:tasks.rados.rados.0.smithi045.stdout:73: done (6 left) 2021-07-31T01:46:01.769 INFO:tasks.rados.rados.0.smithi045.stdout:75: done (5 left) 2021-07-31T01:46:01.770 INFO:tasks.rados.rados.0.smithi045.stdout:76: done (4 left) 2021-07-31T01:46:01.770 INFO:tasks.rados.rados.0.smithi045.stdout:77: done (3 left) 2021-07-31T01:46:01.771 INFO:tasks.rados.rados.0.smithi045.stdout:78: done (2 left) 2021-07-31T01:46:01.772 INFO:tasks.rados.rados.0.smithi045.stdout:79: done (1 left) 2021-07-31T01:46:01.772 INFO:tasks.rados.rados.0.smithi045.stdout:80: done (0 left) 2021-07-31T01:46:01.773 INFO:tasks.rados.rados.0.smithi045.stdout:81: rollback oid 40 current snap is 2 2021-07-31T01:46:01.773 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 40 to 1 2021-07-31T01:46:01.773 INFO:tasks.rados.rados.0.smithi045.stdout:82: snap_create 2021-07-31T01:46:01.775 INFO:tasks.rados.rados.0.smithi045.stdout:81: finishing rollback tid 0 to smithi04569269-40 2021-07-31T01:46:01.775 INFO:tasks.rados.rados.0.smithi045.stdout:81: finishing rollback tid 1 to smithi04569269-40 2021-07-31T01:46:01.776 INFO:tasks.rados.rados.0.smithi045.stdout:81: finishing rollback tid 2 to smithi04569269-40 2021-07-31T01:46:01.776 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 40 v 14 (ObjNum 39 snap 0 seq_num 39) dirty exists 2021-07-31T01:46:02.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:01 smithi045 conmon[33655]: cluster 2021-07-31T01:46:00.759446 2021-07-31T01:46:02.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:01 smithi045 conmon[33655]: +0000 mon.a (mon.0) 658 : cluster [DBG] osdmap e61: 8 total, 8 up, 8 in 2021-07-31T01:46:02.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:01 smithi045 conmon[39718]: cluster 2021-07-31T01:46:00. 2021-07-31T01:46:02.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:01 smithi045 conmon[39718]: 759446+0000 mon.a (mon.0) 658 : cluster [DBG] osdmap e61: 8 total, 8 up, 8 in 2021-07-31T01:46:02.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:01 smithi154 conmon[34553]: cluster 2021-07-31T01:46:00.759446+0000 mon.a (mon.0) 658 : cluster [DBG] osdmap e61: 8 total, 8 up, 8 in 2021-07-31T01:46:02.771 INFO:tasks.rados.rados.0.smithi045.stdout:81: done (1 left) 2021-07-31T01:46:02.771 INFO:tasks.rados.rados.0.smithi045.stdout:82: done (0 left) 2021-07-31T01:46:02.772 INFO:tasks.rados.rados.0.smithi045.stdout:83: rollback oid 9 current snap is 3 2021-07-31T01:46:02.772 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 9 to 0 2021-07-31T01:46:02.772 INFO:tasks.rados.rados.0.smithi045.stdout:84: read oid 20 snap 1 2021-07-31T01:46:02.772 INFO:tasks.rados.rados.0.smithi045.stdout:84: expect (ObjNum 11 snap 0 seq_num 11) 2021-07-31T01:46:02.772 INFO:tasks.rados.rados.0.smithi045.stdout:85: read oid 41 snap -1 2021-07-31T01:46:02.773 INFO:tasks.rados.rados.0.smithi045.stdout:85: expect (ObjNum 40 snap 0 seq_num 40) 2021-07-31T01:46:02.773 INFO:tasks.rados.rados.0.smithi045.stdout:86: snap_create 2021-07-31T01:46:02.776 INFO:tasks.rados.rados.0.smithi045.stdout:83: finishing rollback tid 0 to smithi04569269-9 2021-07-31T01:46:02.777 INFO:tasks.rados.rados.0.smithi045.stdout:84: expect (ObjNum 11 snap 0 seq_num 11) 2021-07-31T01:46:02.782 INFO:tasks.rados.rados.0.smithi045.stdout:83: finishing rollback tid 1 to smithi04569269-9 2021-07-31T01:46:02.782 INFO:tasks.rados.rados.0.smithi045.stdout:83: finishing rollback tid 2 to smithi04569269-9 2021-07-31T01:46:02.782 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 9 v 8 (ObjNum 8 snap 0 seq_num 8) dirty exists 2021-07-31T01:46:02.783 INFO:tasks.rados.rados.0.smithi045.stdout:85: expect (ObjNum 40 snap 0 seq_num 40) 2021-07-31T01:46:03.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:02 smithi045 conmon[33655]: cluster 2021-07-31T01:46:01.363393+0000 mgr.y (mgr. 2021-07-31T01:46:03.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:02 smithi045 conmon[33655]: 14146) 274 : cluster [DBG] pgmap v242: 25 pgs: 25 active+clean; 0 B data, 44 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:46:03.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:02 smithi045 conmon[33655]: cluster 2021-07-31T01:46:01 2021-07-31T01:46:03.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:02 smithi045 conmon[33655]: .765350+0000 mon.a (mon.0) 659 : cluster [DBG] osdmap e62: 8 total, 8 up, 8 in 2021-07-31T01:46:03.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:02 smithi045 conmon[39718]: cluster 2021-07-31T01:46:01.363393+0000 mgr.y 2021-07-31T01:46:03.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:02 smithi045 conmon[39718]: (mgr.14146) 274 : cluster [DBG] pgmap v242: 25 pgs: 25 active+clean; 0 B data, 44 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:46:03.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:02 smithi045 conmon[39718]: cluster 2021-07-31T01:46:03.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:02 smithi045 conmon[39718]: 2021-07-31T01:46:01.765350+0000 mon.a (mon.0) 659 : cluster [DBG] 2021-07-31T01:46:03.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:02 smithi045 conmon[39718]: osdmap e62: 8 total, 8 up, 8 in 2021-07-31T01:46:03.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:02 smithi154 conmon[34553]: cluster 2021-07-31T01:46:01.363393+0000 mgr.y (mgr.14146) 274 : cluster [DBG] pgmap v242: 25 pgs: 25 active+clean; 0 B data, 44 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:46:03.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:02 smithi154 conmon[34553]: cluster 2021-07-31T01:46:01.765350+0000 mon.a (mon.0) 659 : cluster [DBG] osdmap e62: 8 total, 8 up, 8 in 2021-07-31T01:46:03.775 INFO:tasks.rados.rados.0.smithi045.stdout:83: done (3 left) 2021-07-31T01:46:03.776 INFO:tasks.rados.rados.0.smithi045.stdout:84: done (2 left) 2021-07-31T01:46:03.776 INFO:tasks.rados.rados.0.smithi045.stdout:85: done (1 left) 2021-07-31T01:46:03.776 INFO:tasks.rados.rados.0.smithi045.stdout:86: done (0 left) 2021-07-31T01:46:03.777 INFO:tasks.rados.rados.0.smithi045.stdout:87: snap_remove snap 0 2021-07-31T01:46:04.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:03 smithi045 conmon[39718]: cluster 2021-07-31T01:46:02.770279+0000 mon.a 2021-07-31T01:46:04.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:03 smithi045 conmon[39718]: (mon.0) 660 : cluster [DBG] osdmap e63: 8 total, 8 up, 8 in 2021-07-31T01:46:04.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:03 smithi045 conmon[33655]: cluster 2021-07-31T01:46:02.770279 2021-07-31T01:46:04.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:03 smithi045 conmon[33655]: +0000 mon.a (mon.0) 660 : cluster [DBG] osdmap e63: 8 total, 8 up, 8 in 2021-07-31T01:46:04.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:03 smithi154 conmon[34553]: cluster 2021-07-31T01:46:02.770279+0000 mon.a (mon.0) 660 : cluster [DBG] osdmap e63: 8 total, 8 up, 8 in 2021-07-31T01:46:04.776 INFO:tasks.rados.rados.0.smithi045.stdout:87: done (0 left) 2021-07-31T01:46:04.777 INFO:tasks.rados.rados.0.smithi045.stdout:88: read oid 29 snap 3 2021-07-31T01:46:04.777 INFO:tasks.rados.rados.0.smithi045.stdout:88: expect (ObjNum 46 snap 0 seq_num 46) 2021-07-31T01:46:04.777 INFO:tasks.rados.rados.0.smithi045.stdout:89: read oid 46 snap -1 2021-07-31T01:46:04.777 INFO:tasks.rados.rados.0.smithi045.stdout:89: expect (ObjNum 45 snap 0 seq_num 45) 2021-07-31T01:46:04.778 INFO:tasks.rados.rados.0.smithi045.stdout:90: copy_from oid 36 from oid 2 current snap is 4 2021-07-31T01:46:04.778 INFO:tasks.rados.rados.0.smithi045.stdout:91: rmattr oid 49 current snap is 4 2021-07-31T01:46:04.778 INFO:tasks.rados.rados.0.smithi045.stdout:91: done (3 left) 2021-07-31T01:46:04.779 INFO:tasks.rados.rados.0.smithi045.stdout:92: snap_remove snap 2 2021-07-31T01:46:05.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:04 smithi154 conmon[34553]: cluster 2021-07-31T01:46:03.363695+0000 mgr.y (mgr. 2021-07-31T01:46:05.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:04 smithi154 conmon[34553]: 14146) 275 : cluster [DBG] pgmap v245: 25 pgs: 25 active+clean; 14 MiB data, 64 MiB used, 715 GiB / 715 GiB avail; 2.2 MiB/s rd, 10 MiB/s wr, 104 op/s 2021-07-31T01:46:05.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:04 smithi154 conmon[34553]: cluster 2021-07-31T01:46:03.775187+0000 mon.a (mon.0) 661 : cluster [DBG] osdmap e64: 8 total, 8 up, 8 in 2021-07-31T01:46:05.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:04 smithi154 conmon[34553]: audit 2021- 2021-07-31T01:46:05.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:04 smithi154 conmon[34553]: 07-31T01:46:04.458510+0000 mon.a (mon 2021-07-31T01:46:05.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:04 smithi154 conmon[34553]: .0) 662 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:46:05.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:04 smithi045 conmon[39718]: cluster 2021-07-31T01:46:05.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:04 smithi045 conmon[39718]: 2021-07 2021-07-31T01:46:05.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:04 smithi045 conmon[39718]: -31T01:46:03.363695+0000 mgr.y (mgr.14146) 275 : cluster [DBG] pgmap v245: 25 pgs: 25 active+clean; 14 MiB data, 64 MiB used, 715 GiB / 715 GiB avail; 2.2 MiB/s rd, 10 MiB/s wr, 104 op/s 2021-07-31T01:46:05.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:04 smithi045 conmon[39718]: 2021-07-31T01:46:05.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:04 smithi045 conmon[39718]: cluster 2021-07-31T01:46:03.775187+0000 mon.a 2021-07-31T01:46:05.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:04 smithi045 conmon[39718]: (mon.0) 661 : cluster [DBG] osdmap e64: 8 total, 8 up, 8 in 2021-07-31T01:46:05.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:04 smithi045 conmon[39718]: audit 2021 2021-07-31T01:46:05.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:04 smithi045 conmon[39718]: -07-31T01:46:04.458510+0000 2021-07-31T01:46:05.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:04 smithi045 conmon[39718]: mon.a (mon.0) 662 : audit [INF] 2021-07-31T01:46:05.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:04 smithi045 conmon[39718]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:46:05.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:04 smithi045 conmon[33655]: cluster 2021-07-31T01:46:05.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:04 smithi045 conmon[33655]: 2021-07-31T01:46:03.363695+0000 mgr.y (mgr 2021-07-31T01:46:05.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:04 smithi045 conmon[33655]: .14146) 275 : cluster [DBG] pgmap v245: 25 pgs: 25 active+clean; 14 MiB data, 64 MiB used, 715 GiB / 715 GiB avail; 2.2 MiB/s rd, 10 MiB/s wr, 104 op/s 2021-07-31T01:46:05.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:04 smithi045 conmon[33655]: cluster 2021-07-31T01:46:03. 2021-07-31T01:46:05.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:04 smithi045 conmon[33655]: 775187+0000 mon.a (mon.0) 661 : cluster [DBG] osdmap e64: 8 total, 8 up, 8 in 2021-07-31T01:46:05.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:04 smithi045 conmon[33655]: audit 2021-07 2021-07-31T01:46:05.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:04 smithi045 conmon[33655]: -31T01:46:04.458510+0000 mon.a (mon.0) 662 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:46:05.778 INFO:tasks.rados.rados.0.smithi045.stdout:92: done (3 left) 2021-07-31T01:46:05.779 INFO:tasks.rados.rados.0.smithi045.stdout:93: rmattr oid 30 current snap is 4 2021-07-31T01:46:05.779 INFO:tasks.rados.rados.0.smithi045.stdout:93: done (3 left) 2021-07-31T01:46:05.779 INFO:tasks.rados.rados.0.smithi045.stdout:94: read oid 42 snap 1 2021-07-31T01:46:05.780 INFO:tasks.rados.rados.0.smithi045.stdout:94: expect (ObjNum 41 snap 0 seq_num 41) 2021-07-31T01:46:05.780 INFO:tasks.rados.rados.0.smithi045.stdout:95: rmattr oid 14 current snap is 4 2021-07-31T01:46:05.780 INFO:tasks.rados.rados.0.smithi045.stdout:95: done (4 left) 2021-07-31T01:46:05.780 INFO:tasks.rados.rados.0.smithi045.stdout:96: read oid 9 snap 1 2021-07-31T01:46:05.780 INFO:tasks.rados.rados.0.smithi045.stdout:96: expect (ObjNum 8 snap 0 seq_num 8) 2021-07-31T01:46:05.781 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 24 current snap is 4 2021-07-31T01:46:05.781 INFO:tasks.rados.rados.0.smithi045.stdout:97: seq_num 54 ranges {393216=507904,901120=401408} 2021-07-31T01:46:05.784 INFO:tasks.rados.rados.0.smithi045.stdout:97: writing smithi04569269-24 from 393216 to 901120 tid 1 2021-07-31T01:46:05.786 INFO:tasks.rados.rados.0.smithi045.stdout:97: writing smithi04569269-24 from 901120 to 1302528 tid 2 2021-07-31T01:46:05.792 INFO:tasks.rados.rados.0.smithi045.stdout:88: expect (ObjNum 46 snap 0 seq_num 46) 2021-07-31T01:46:05.810 INFO:tasks.rados.rados.0.smithi045.stdout:90: finishing copy_from to smithi04569269-36 2021-07-31T01:46:05.810 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 36 v 2 (ObjNum 1 snap 0 seq_num 1) dirty exists 2021-07-31T01:46:05.810 INFO:tasks.rados.rados.0.smithi045.stdout:90: finishing copy_from racing read to smithi04569269-36 2021-07-31T01:46:05.811 INFO:tasks.rados.rados.0.smithi045.stdout:88: done (5 left) 2021-07-31T01:46:05.811 INFO:tasks.rados.rados.0.smithi045.stdout:90: done (4 left) 2021-07-31T01:46:05.811 INFO:tasks.rados.rados.0.smithi045.stdout:98: copy_from oid 16 from oid 5 current snap is 4 2021-07-31T01:46:05.811 INFO:tasks.rados.rados.0.smithi045.stdout:89: expect (ObjNum 45 snap 0 seq_num 45) 2021-07-31T01:46:05.830 INFO:tasks.rados.rados.0.smithi045.stdout:89: done (4 left) 2021-07-31T01:46:05.830 INFO:tasks.rados.rados.0.smithi045.stdout:99: snap_remove snap 3 2021-07-31T01:46:06.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:05 smithi154 conmon[34553]: cluster 2021-07-31T01:46:04.775660+0000 mon.a (mon.0) 663 : cluster [DBG] osdmap e65: 8 total, 8 up, 8 in 2021-07-31T01:46:06.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:05 smithi154 conmon[34553]: 2021-07-31T01:46:06.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:05 smithi154 conmon[34553]: cluster 2021-07-31T01:46:05.777668+0000 mon.a (mon.0) 664 : cluster [DBG] osdmap e66: 8 total, 8 up, 8 in 2021-07-31T01:46:06.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:05 smithi045 conmon[33655]: cluster 2021-07-31T01:46:06.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:05 smithi045 conmon[33655]: 2021-07-31 2021-07-31T01:46:06.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:05 smithi045 conmon[33655]: T01:46:04.775660+0000 mon.a (mon.0) 663 : cluster [DBG] osdmap e65: 8 total, 8 up, 8 in 2021-07-31T01:46:06.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:05 smithi045 conmon[33655]: 2021-07-31T01:46:06.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:05 smithi045 conmon[33655]: cluster 2021 2021-07-31T01:46:06.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:05 smithi045 conmon[33655]: -07-31T01:46:05.777668 2021-07-31T01:46:06.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:05 smithi045 conmon[33655]: +0000 mon.a (mon.0) 664 : cluster 2021-07-31T01:46:06.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:05 smithi045 conmon[33655]: [DBG] osdmap e66: 8 total, 8 up, 8 in 2021-07-31T01:46:06.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:05 smithi045 conmon[39718]: cluster 2021-07-31T01:46:04.775660 2021-07-31T01:46:06.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:05 smithi045 conmon[39718]: +0000 mon.a (mon.0) 663 : cluster [DBG] osdmap e65: 8 total, 8 up, 8 in 2021-07-31T01:46:06.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:05 smithi045 conmon[39718]: cluster 2021-07-31T01:46:06.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:05 smithi045 conmon[39718]: 2021- 2021-07-31T01:46:06.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:05 smithi045 conmon[39718]: 07-31T01:46:05.777668 2021-07-31T01:46:06.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:05 smithi045 conmon[39718]: +0000 mon.a (mon.0) 664 : cluster 2021-07-31T01:46:06.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:05 smithi045 conmon[39718]: [DBG] osdmap e66: 8 total, 8 up, 8 in 2021-07-31T01:46:06.794 INFO:tasks.rados.rados.0.smithi045.stdout:99: done (4 left) 2021-07-31T01:46:06.795 INFO:tasks.rados.rados.0.smithi045.stdout:100: copy_from oid 26 from oid 31 current snap is 4 2021-07-31T01:46:06.795 INFO:tasks.rados.rados.0.smithi045.stdout:96: expect (ObjNum 8 snap 0 seq_num 8) 2021-07-31T01:46:06.798 INFO:tasks.rados.rados.0.smithi045.stdout:94: expect (ObjNum 41 snap 0 seq_num 41) 2021-07-31T01:46:06.820 INFO:tasks.rados.rados.0.smithi045.stdout:97: finishing write tid 1 to smithi04569269-24 2021-07-31T01:46:06.820 INFO:tasks.rados.rados.0.smithi045.stdout:97: finishing write tid 2 to smithi04569269-24 2021-07-31T01:46:06.821 INFO:tasks.rados.rados.0.smithi045.stdout:97: finishing write tid 3 to smithi04569269-24 2021-07-31T01:46:06.821 INFO:tasks.rados.rados.0.smithi045.stdout:97: finishing write tid 4 to smithi04569269-24 2021-07-31T01:46:06.821 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 24 v 8 (ObjNum 54 snap 4 seq_num 54) dirty exists 2021-07-31T01:46:06.821 INFO:tasks.rados.rados.0.smithi045.stdout:97: left oid 24 (ObjNum 54 snap 4 seq_num 54) 2021-07-31T01:46:06.821 INFO:tasks.rados.rados.0.smithi045.stdout:98: finishing copy_from to smithi04569269-16 2021-07-31T01:46:06.822 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 16 v 2 (ObjNum 4 snap 0 seq_num 4) dirty exists 2021-07-31T01:46:06.822 INFO:tasks.rados.rados.0.smithi045.stdout:98: finishing copy_from racing read to smithi04569269-16 2021-07-31T01:46:06.822 INFO:tasks.rados.rados.0.smithi045.stdout:94: done (4 left) 2021-07-31T01:46:06.822 INFO:tasks.rados.rados.0.smithi045.stdout:96: done (3 left) 2021-07-31T01:46:06.823 INFO:tasks.rados.rados.0.smithi045.stdout:97: done (2 left) 2021-07-31T01:46:06.823 INFO:tasks.rados.rados.0.smithi045.stdout:98: done (1 left) 2021-07-31T01:46:06.823 INFO:tasks.rados.rados.0.smithi045.stdout:101: read oid 14 snap 1 2021-07-31T01:46:06.823 INFO:tasks.rados.rados.0.smithi045.stdout:101: expect (ObjNum 13 snap 0 seq_num 13) 2021-07-31T01:46:06.823 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 49 current snap is 4 2021-07-31T01:46:06.824 INFO:tasks.rados.rados.0.smithi045.stdout:100: finishing copy_from to smithi04569269-26 2021-07-31T01:46:06.824 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 26 v 19 (ObjNum 30 snap 0 seq_num 30) dirty exists 2021-07-31T01:46:06.824 INFO:tasks.rados.rados.0.smithi045.stdout:100: finishing copy_from racing read to smithi04569269-26 2021-07-31T01:46:06.824 INFO:tasks.rados.rados.0.smithi045.stdout:102: seq_num 55 ranges {811008=524288,1335296=401408} 2021-07-31T01:46:06.827 INFO:tasks.rados.rados.0.smithi045.stdout:102: writing smithi04569269-49 from 811008 to 1335296 tid 1 2021-07-31T01:46:06.830 INFO:tasks.rados.rados.0.smithi045.stdout:102: writing smithi04569269-49 from 1335296 to 1736704 tid 2 2021-07-31T01:46:06.830 INFO:tasks.rados.rados.0.smithi045.stdout:100: done (2 left) 2021-07-31T01:46:06.831 INFO:tasks.rados.rados.0.smithi045.stdout:103: delete oid 34 current snap is 4 2021-07-31T01:46:06.832 INFO:tasks.rados.rados.0.smithi045.stdout:101: expect (ObjNum 13 snap 0 seq_num 13) 2021-07-31T01:46:06.841 INFO:tasks.rados.rados.0.smithi045.stdout:102: finishing write tid 1 to smithi04569269-49 2021-07-31T01:46:06.842 INFO:tasks.rados.rados.0.smithi045.stdout:101: done (2 left) 2021-07-31T01:46:06.842 INFO:tasks.rados.rados.0.smithi045.stdout:103: done (1 left) 2021-07-31T01:46:06.843 INFO:tasks.rados.rados.0.smithi045.stdout:104: read oid 2 snap -1 2021-07-31T01:46:06.843 INFO:tasks.rados.rados.0.smithi045.stdout:104: expect (ObjNum 1 snap 0 seq_num 1) 2021-07-31T01:46:06.844 INFO:tasks.rados.rados.0.smithi045.stdout:105: rollback oid 34 current snap is 4 2021-07-31T01:46:06.844 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 34 to 1 2021-07-31T01:46:06.844 INFO:tasks.rados.rados.0.smithi045.stdout:102: finishing write tid 2 to smithi04569269-49 2021-07-31T01:46:06.845 INFO:tasks.rados.rados.0.smithi045.stdout:102: finishing write tid 3 to smithi04569269-49 2021-07-31T01:46:06.845 INFO:tasks.rados.rados.0.smithi045.stdout:102: finishing write tid 4 to smithi04569269-49 2021-07-31T01:46:06.846 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 49 v 45 (ObjNum 55 snap 4 seq_num 55) dirty exists 2021-07-31T01:46:06.846 INFO:tasks.rados.rados.0.smithi045.stdout:102: left oid 49 (ObjNum 55 snap 4 seq_num 55) 2021-07-31T01:46:06.846 INFO:tasks.rados.rados.0.smithi045.stdout:102: done (2 left) 2021-07-31T01:46:06.847 INFO:tasks.rados.rados.0.smithi045.stdout:106: delete oid 12 current snap is 4 2021-07-31T01:46:06.850 INFO:tasks.rados.rados.0.smithi045.stdout:105: finishing rollback tid 1 to smithi04569269-34 2021-07-31T01:46:06.851 INFO:tasks.rados.rados.0.smithi045.stdout:106: done (2 left) 2021-07-31T01:46:06.851 INFO:tasks.rados.rados.0.smithi045.stdout:107: read oid 17 snap 1 2021-07-31T01:46:06.851 INFO:tasks.rados.rados.0.smithi045.stdout:107: expect deleted 2021-07-31T01:46:06.852 INFO:tasks.rados.rados.0.smithi045.stdout:108: rollback oid 41 current snap is 4 2021-07-31T01:46:06.852 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 41 to 1 2021-07-31T01:46:06.853 INFO:tasks.rados.rados.0.smithi045.stdout:109: delete oid 8 current snap is 4 2021-07-31T01:46:06.853 INFO:tasks.rados.rados.0.smithi045.stdout:105: finishing rollback tid 2 to smithi04569269-34 2021-07-31T01:46:06.854 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 34 v 11 (ObjNum 33 snap 0 seq_num 33) dirty exists 2021-07-31T01:46:06.854 INFO:tasks.rados.rados.0.smithi045.stdout:104: expect (ObjNum 1 snap 0 seq_num 1) 2021-07-31T01:46:06.854 INFO:tasks.rados.rados.0.smithi045.stdout:104: done (4 left) 2021-07-31T01:46:06.855 INFO:tasks.rados.rados.0.smithi045.stdout:105: done (3 left) 2021-07-31T01:46:06.855 INFO:tasks.rados.rados.0.smithi045.stdout:107: done (2 left) 2021-07-31T01:46:06.856 INFO:tasks.rados.rados.0.smithi045.stdout:109: done (1 left) 2021-07-31T01:46:06.856 INFO:tasks.rados.rados.0.smithi045.stdout:110: setattr oid 22 current snap is 4 2021-07-31T01:46:06.856 INFO:tasks.rados.rados.0.smithi045.stdout:108: finishing rollback tid 0 to smithi04569269-41 2021-07-31T01:46:06.857 INFO:tasks.rados.rados.0.smithi045.stdout:108: finishing rollback tid 1 to smithi04569269-41 2021-07-31T01:46:06.857 INFO:tasks.rados.rados.0.smithi045.stdout:108: finishing rollback tid 2 to smithi04569269-41 2021-07-31T01:46:06.858 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 41 v 12 (ObjNum 40 snap 0 seq_num 40) dirty exists 2021-07-31T01:46:06.858 INFO:tasks.rados.rados.0.smithi045.stdout:108: done (1 left) 2021-07-31T01:46:06.859 INFO:tasks.rados.rados.0.smithi045.stdout:111: snap_create 2021-07-31T01:46:06.859 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 22 v 13 (ObjNum 47 snap 0 seq_num 47) dirty exists 2021-07-31T01:46:07.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:06 smithi154 conmon[34553]: cluster 2021-07-31T01:46:05.364001+0000 mgr.y (mgr.14146) 276 : 2021-07-31T01:46:07.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:06 smithi154 conmon[34553]: cluster [DBG] pgmap v248: 25 pgs: 25 active+clean; 24 MiB data, 82 MiB used, 715 GiB / 715 GiB avail; 2.2 MiB/s rd, 6.4 MiB/s wr, 83 op/s 2021-07-31T01:46:07.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:06 smithi045 conmon[33655]: cluster 2021-07-31T01:46:05.364001+0000 2021-07-31T01:46:07.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:06 smithi045 conmon[33655]: mgr.y (mgr.14146) 276 : cluster [DBG] pgmap v248: 25 pgs: 25 active+clean; 24 MiB data, 82 MiB used, 715 GiB / 715 GiB avail; 2.2 MiB/s rd, 6.4 MiB/s wr, 83 op/s 2021-07-31T01:46:07.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:06 smithi045 conmon[39718]: cluster 2021-07-31T01:46 2021-07-31T01:46:07.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:06 smithi045 conmon[39718]: :05.364001+0000 mgr.y (mgr.14146) 276 : cluster [DBG] pgmap v248: 25 pgs: 25 active+clean; 24 MiB data, 82 MiB used, 715 GiB / 715 GiB avail; 2.2 MiB/s rd, 6.4 MiB/s wr, 83 op/s 2021-07-31T01:46:07.799 INFO:tasks.rados.rados.0.smithi045.stdout:110: done (1 left) 2021-07-31T01:46:07.800 INFO:tasks.rados.rados.0.smithi045.stdout:111: done (0 left) 2021-07-31T01:46:07.800 INFO:tasks.rados.rados.0.smithi045.stdout:112: copy_from oid 25 from oid 6 current snap is 5 2021-07-31T01:46:07.800 INFO:tasks.rados.rados.0.smithi045.stdout:113: setattr oid 23 current snap is 5 2021-07-31T01:46:07.801 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 15 current snap is 5 2021-07-31T01:46:07.801 INFO:tasks.rados.rados.0.smithi045.stdout:114: seq_num 59 ranges {24576=598016,622592=401408} 2021-07-31T01:46:07.803 INFO:tasks.rados.rados.0.smithi045.stdout:114: writing smithi04569269-15 from 24576 to 622592 tid 1 2021-07-31T01:46:07.806 INFO:tasks.rados.rados.0.smithi045.stdout:114: writing smithi04569269-15 from 622592 to 1024000 tid 2 2021-07-31T01:46:07.806 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 20 current snap is 5 2021-07-31T01:46:07.806 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 23 v 23 (ObjNum 53 snap 1 seq_num 53) dirty exists 2021-07-31T01:46:07.807 INFO:tasks.rados.rados.0.smithi045.stdout:115: seq_num 60 ranges {188416=614400,802816=401408} 2021-07-31T01:46:07.811 INFO:tasks.rados.rados.0.smithi045.stdout:115: writing smithi04569269-20 from 188416 to 802816 tid 1 2021-07-31T01:46:07.817 INFO:tasks.rados.rados.0.smithi045.stdout:115: writing smithi04569269-20 from 802816 to 1204224 tid 2 2021-07-31T01:46:07.817 INFO:tasks.rados.rados.0.smithi045.stdout:113: done (3 left) 2021-07-31T01:46:07.818 INFO:tasks.rados.rados.0.smithi045.stdout:116: rollback oid 14 current snap is 5 2021-07-31T01:46:07.818 INFO:tasks.rados.rados.0.smithi045.stdout:114: finishing write tid 1 to smithi04569269-15 2021-07-31T01:46:07.819 INFO:tasks.rados.rados.0.smithi045.stdout:114: finishing write tid 2 to smithi04569269-15 2021-07-31T01:46:07.819 INFO:tasks.rados.rados.0.smithi045.stdout:114: finishing write tid 3 to smithi04569269-15 2021-07-31T01:46:07.820 INFO:tasks.rados.rados.0.smithi045.stdout:112: finishing copy_from to smithi04569269-25 2021-07-31T01:46:07.820 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 25 v 8 (ObjNum 36 snap 0 seq_num 36) dirty exists 2021-07-31T01:46:07.821 INFO:tasks.rados.rados.0.smithi045.stdout:112: finishing copy_from racing read to smithi04569269-25 2021-07-31T01:46:07.822 INFO:tasks.rados.rados.0.smithi045.stdout:114: finishing write tid 4 to smithi04569269-15 2021-07-31T01:46:07.822 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 15 v 17 (ObjNum 59 snap 5 seq_num 59) dirty exists 2021-07-31T01:46:07.822 INFO:tasks.rados.rados.0.smithi045.stdout:114: left oid 15 (ObjNum 59 snap 5 seq_num 59) 2021-07-31T01:46:07.823 INFO:tasks.rados.rados.0.smithi045.stdout:115: finishing write tid 1 to smithi04569269-20 2021-07-31T01:46:07.823 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 14 to 1 2021-07-31T01:46:07.824 INFO:tasks.rados.rados.0.smithi045.stdout:112: done (3 left) 2021-07-31T01:46:07.824 INFO:tasks.rados.rados.0.smithi045.stdout:114: done (2 left) 2021-07-31T01:46:07.824 INFO:tasks.rados.rados.0.smithi045.stdout:117: read oid 27 snap -1 2021-07-31T01:46:07.824 INFO:tasks.rados.rados.0.smithi045.stdout:117: expect (ObjNum 26 snap 0 seq_num 26) 2021-07-31T01:46:07.825 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 21 current snap is 5 2021-07-31T01:46:07.825 INFO:tasks.rados.rados.0.smithi045.stdout:118: seq_num 61 ranges {344064=630784,974848=401408} 2021-07-31T01:46:07.826 INFO:tasks.rados.rados.0.smithi045.stdout:118: writing smithi04569269-21 from 344064 to 974848 tid 1 2021-07-31T01:46:07.826 INFO:tasks.rados.rados.0.smithi045.stdout:118: writing smithi04569269-21 from 974848 to 1376256 tid 2 2021-07-31T01:46:07.826 INFO:tasks.rados.rados.0.smithi045.stdout:119: snap_create 2021-07-31T01:46:07.826 INFO:tasks.rados.rados.0.smithi045.stdout:115: finishing write tid 2 to smithi04569269-20 2021-07-31T01:46:07.827 INFO:tasks.rados.rados.0.smithi045.stdout:115: finishing write tid 3 to smithi04569269-20 2021-07-31T01:46:07.827 INFO:tasks.rados.rados.0.smithi045.stdout:116: finishing rollback tid 0 to smithi04569269-14 2021-07-31T01:46:07.827 INFO:tasks.rados.rados.0.smithi045.stdout:116: finishing rollback tid 1 to smithi04569269-14 2021-07-31T01:46:07.827 INFO:tasks.rados.rados.0.smithi045.stdout:115: finishing write tid 4 to smithi04569269-20 2021-07-31T01:46:07.828 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 20 v 51 (ObjNum 60 snap 5 seq_num 60) dirty exists 2021-07-31T01:46:07.828 INFO:tasks.rados.rados.0.smithi045.stdout:115: left oid 20 (ObjNum 60 snap 5 seq_num 60) 2021-07-31T01:46:07.828 INFO:tasks.rados.rados.0.smithi045.stdout:116: finishing rollback tid 2 to smithi04569269-14 2021-07-31T01:46:07.828 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 14 v 12 (ObjNum 13 snap 0 seq_num 13) dirty exists 2021-07-31T01:46:07.828 INFO:tasks.rados.rados.0.smithi045.stdout:118: finishing write tid 1 to smithi04569269-21 2021-07-31T01:46:07.829 INFO:tasks.rados.rados.0.smithi045.stdout:117: expect (ObjNum 26 snap 0 seq_num 26) 2021-07-31T01:46:07.844 INFO:tasks.rados.rados.0.smithi045.stdout:118: finishing write tid 2 to smithi04569269-21 2021-07-31T01:46:07.844 INFO:tasks.rados.rados.0.smithi045.stdout:118: finishing write tid 3 to smithi04569269-21 2021-07-31T01:46:07.844 INFO:tasks.rados.rados.0.smithi045.stdout:118: finishing write tid 4 to smithi04569269-21 2021-07-31T01:46:07.845 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 21 v 55 (ObjNum 61 snap 5 seq_num 61) dirty exists 2021-07-31T01:46:07.845 INFO:tasks.rados.rados.0.smithi045.stdout:118: left oid 21 (ObjNum 61 snap 5 seq_num 61) 2021-07-31T01:46:08.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:07 smithi154 conmon[34553]: cluster 2021-07-31T01:46:06.793681+0000 mon.a (mon.0) 665 : cluster [DBG] osdmap e67: 8 total, 8 up, 8 in 2021-07-31T01:46:08.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:07 smithi045 conmon[33655]: cluster 2021-07-31T01:46:06.793681+0000 mon.a (mon.0) 665 : cluster [DBG] osdmap e67: 8 total, 8 up, 8 in 2021-07-31T01:46:08.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:07 smithi045 conmon[39718]: cluster 2021-07-31T01:46:08.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:07 smithi045 conmon[39718]: 2021-07- 2021-07-31T01:46:08.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:07 smithi045 conmon[39718]: 31T01:46:06.793681+0000 mon.a (mon.0) 665 : cluster [DBG] osdmap e67: 8 total, 8 up, 8 in 2021-07-31T01:46:08.800 INFO:tasks.rados.rados.0.smithi045.stdout:115: done (4 left) 2021-07-31T01:46:08.800 INFO:tasks.rados.rados.0.smithi045.stdout:116: done (3 left) 2021-07-31T01:46:08.800 INFO:tasks.rados.rados.0.smithi045.stdout:117: done (2 left) 2021-07-31T01:46:08.801 INFO:tasks.rados.rados.0.smithi045.stdout:118: done (1 left) 2021-07-31T01:46:08.801 INFO:tasks.rados.rados.0.smithi045.stdout:119: done (0 left) 2021-07-31T01:46:08.801 INFO:tasks.rados.rados.0.smithi045.stdout:120: read oid 48 snap 4 2021-07-31T01:46:08.801 INFO:tasks.rados.rados.0.smithi045.stdout:120: expect (ObjNum 47 snap 0 seq_num 47) 2021-07-31T01:46:08.801 INFO:tasks.rados.rados.0.smithi045.stdout:121: read oid 49 snap 5 2021-07-31T01:46:08.802 INFO:tasks.rados.rados.0.smithi045.stdout:121: expect (ObjNum 55 snap 4 seq_num 55) 2021-07-31T01:46:08.802 INFO:tasks.rados.rados.0.smithi045.stdout:122: delete oid 44 current snap is 6 2021-07-31T01:46:08.806 INFO:tasks.rados.rados.0.smithi045.stdout:120: expect (ObjNum 47 snap 0 seq_num 47) 2021-07-31T01:46:08.828 INFO:tasks.rados.rados.0.smithi045.stdout:121: expect (ObjNum 55 snap 4 seq_num 55) 2021-07-31T01:46:08.889 INFO:tasks.rados.rados.0.smithi045.stdout:120: done (2 left) 2021-07-31T01:46:08.889 INFO:tasks.rados.rados.0.smithi045.stdout:121: done (1 left) 2021-07-31T01:46:08.889 INFO:tasks.rados.rados.0.smithi045.stdout:122: done (0 left) 2021-07-31T01:46:08.890 INFO:tasks.rados.rados.0.smithi045.stdout:123: rollback oid 49 current snap is 6 2021-07-31T01:46:08.890 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 49 to 4 2021-07-31T01:46:08.890 INFO:tasks.rados.rados.0.smithi045.stdout:124: read oid 27 snap -1 2021-07-31T01:46:08.890 INFO:tasks.rados.rados.0.smithi045.stdout:124: expect (ObjNum 26 snap 0 seq_num 26) 2021-07-31T01:46:08.890 INFO:tasks.rados.rados.0.smithi045.stdout:125: setattr oid 41 current snap is 6 2021-07-31T01:46:08.891 INFO:tasks.rados.rados.0.smithi045.stdout:126: delete oid 28 current snap is 6 2021-07-31T01:46:08.891 INFO:tasks.rados.rados.0.smithi045.stdout:123: finishing rollback tid 0 to smithi04569269-49 2021-07-31T01:46:08.892 INFO:tasks.rados.rados.0.smithi045.stdout:123: finishing rollback tid 1 to smithi04569269-49 2021-07-31T01:46:08.892 INFO:tasks.rados.rados.0.smithi045.stdout:123: finishing rollback tid 2 to smithi04569269-49 2021-07-31T01:46:08.892 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 49 v 59 (ObjNum 55 snap 4 seq_num 55) dirty exists 2021-07-31T01:46:08.893 INFO:tasks.rados.rados.0.smithi045.stdout:123: done (3 left) 2021-07-31T01:46:08.893 INFO:tasks.rados.rados.0.smithi045.stdout:126: done (2 left) 2021-07-31T01:46:08.894 INFO:tasks.rados.rados.0.smithi045.stdout:127: snap_remove snap 4 2021-07-31T01:46:09.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:08 smithi154 conmon[34553]: cluster 2021-07-31T01:46:07.364338+0000 mgr.y (mgr.14146) 277 : cluster [DBG] pgmap v251: 25 pgs: 25 active+clean; 24 MiB data, 81 MiB used, 715 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.6 MiB/s wr, 41 op/s 2021-07-31T01:46:09.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:08 smithi154 conmon[34553]: 2021-07-31T01:46:09.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:08 smithi154 conmon[34553]: cluster 2021-07-31T01:46:07.798003+0000 mon.a (mon.0) 666 : cluster [DBG] osdmap e68: 8 total, 8 up, 8 in 2021-07-31T01:46:09.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:08 smithi045 conmon[33655]: cluster 2021-07-31T01:46:09.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:08 smithi045 conmon[33655]: 2021-07-31T01: 2021-07-31T01:46:09.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:08 smithi045 conmon[33655]: 46:07.364338+0000 mgr.y (mgr.14146) 277 : cluster [DBG] pgmap v251: 25 pgs: 25 active+clean; 24 MiB data, 81 MiB used, 715 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.6 MiB/s wr, 41 op/s 2021-07-31T01:46:09.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:08 smithi045 conmon[33655]: cluster 2021-07- 2021-07-31T01:46:09.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:08 smithi045 conmon[33655]: 31T01:46:07.798003+0000 mon.a (mon. 2021-07-31T01:46:09.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:08 smithi045 conmon[33655]: 0) 666 : cluster [DBG] osdmap e68: 8 total, 8 up, 8 in 2021-07-31T01:46:09.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:08 smithi045 conmon[39718]: cluster 2021-07-31T01:46:07.364338+0000 2021-07-31T01:46:09.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:08 smithi045 conmon[39718]: mgr.y (mgr.14146) 277 : cluster [DBG] pgmap v251: 25 pgs: 25 active+clean; 24 MiB data, 81 MiB used, 715 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.6 MiB/s wr, 41 op/s 2021-07-31T01:46:09.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:08 smithi045 conmon[39718]: cluster 2021-07-31T01 2021-07-31T01:46:09.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:08 smithi045 conmon[39718]: :46:07.798003+0000 mon.a (mon.0) 666 : cluster [DBG] osdmap e68: 8 total, 8 up, 8 in 2021-07-31T01:46:09.809 INFO:tasks.rados.rados.0.smithi045.stdout:127: done (2 left) 2021-07-31T01:46:09.809 INFO:tasks.rados.rados.0.smithi045.stdout:128: setattr oid 32 current snap is 6 2021-07-31T01:46:09.810 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 41 v 14 (ObjNum 40 snap 0 seq_num 40) dirty exists 2021-07-31T01:46:09.810 INFO:tasks.rados.rados.0.smithi045.stdout:124: expect (ObjNum 26 snap 0 seq_num 26) 2021-07-31T01:46:09.831 INFO:tasks.rados.rados.0.smithi045.stdout:124: done (2 left) 2021-07-31T01:46:09.831 INFO:tasks.rados.rados.0.smithi045.stdout:125: done (1 left) 2021-07-31T01:46:09.831 INFO:tasks.rados.rados.0.smithi045.stdout:129: snap_remove snap 1 2021-07-31T01:46:10.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:09 smithi154 conmon[34553]: cluster 2021-07-31T01:46:08.799527+0000 mon.a (mon.0) 667 : cluster [DBG] osdmap e69: 8 total, 8 up, 8 in 2021-07-31T01:46:10.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:09 smithi045 conmon[33655]: cluster 2021-07-31T01:46:08.799527+0000 2021-07-31T01:46:10.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:09 smithi045 conmon[33655]: mon.a (mon.0) 667 : cluster [DBG] osdmap e69: 8 total, 8 up, 8 in 2021-07-31T01:46:10.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:09 smithi045 conmon[39718]: cluster 2021-07-31T01:46:08 2021-07-31T01:46:10.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:09 smithi045 conmon[39718]: .799527+0000 mon.a (mon.0) 667 : cluster [DBG] osdmap e69: 8 total, 8 up, 8 in 2021-07-31T01:46:10.819 INFO:tasks.rados.rados.0.smithi045.stdout:129: done (1 left) 2021-07-31T01:46:10.819 INFO:tasks.rados.rados.0.smithi045.stdout:130: delete oid 6 current snap is 6 2021-07-31T01:46:10.820 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 32 v 20 (ObjNum 31 snap 0 seq_num 31) dirty exists 2021-07-31T01:46:10.826 INFO:tasks.rados.rados.0.smithi045.stdout:128: done (1 left) 2021-07-31T01:46:10.827 INFO:tasks.rados.rados.0.smithi045.stdout:130: done (0 left) 2021-07-31T01:46:10.827 INFO:tasks.rados.rados.0.smithi045.stdout:131: snap_remove snap 5 2021-07-31T01:46:11.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:10 smithi154 conmon[34553]: cluster 2021-07-31T01:46:09.364695+0000 mgr.y (mgr.14146) 278 : cluster [DBG] 2021-07-31T01:46:11.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:10 smithi154 conmon[34553]: pgmap v254: 25 pgs: 25 active+clean; 28 MiB data, 86 MiB used, 715 GiB / 715 GiB avail; 4.2 MiB/s rd, 1.5 MiB/s wr, 23 op/s 2021-07-31T01:46:11.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:10 smithi154 conmon[34553]: cluster 2021-07-31T01:46:09.808619+0000 mon.a (mon.0) 668 : cluster [DBG] osdmap e70: 8 total, 8 up, 8 in 2021-07-31T01:46:11.254 INFO:journalctl@ceph.osd.7.smithi154.stdout:Jul 31 01:46:10 smithi154 conmon[49149]: debug 2021-07-31T01:46:11.254 INFO:journalctl@ceph.osd.7.smithi154.stdout:Jul 31 01:46:10 smithi154 conmon[49149]: 2021-07-31T01:46:10.818+0000 7f8a711fb700 -1 osd.7 pg_epoch: 71 pg[3.es0( v 71'22 (0'0,71'22] local-lis/les=56/57 n=5 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [7,4,1]p7(0) r=0 lpr=56 luod=67'20 crt=67'20 lcod 65'18 mlcod 65'18 active+clean+snaptrim trimq=[3~1] ps=[2~1,4~2,9~1]] removing snap head 2021-07-31T01:46:11.276 INFO:journalctl@ceph.osd.3.smithi045.stdout:Jul 31 01:46:10 smithi045 conmon[54013]: debug 2021-07-31T01:46:11.276 INFO:journalctl@ceph.osd.3.smithi045.stdout:Jul 31 01:46:10 smithi045 conmon[54013]: 2021-07-31T01:46:10.825+0000 7f64f88c5700 -1 osd.3 pg_epoch: 71 pg[3.bs0( v 71'69 (0'0,71'69] local-lis/les=56/57 n=12 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [3,0,4]p3(0) r=0 lpr=56 crt=71'69 lcod 71'67 mlcod 71'67 active+clean+snaptrim trimq=[3~1] ps=[2~1,4~2,9~1]] removing snap head 2021-07-31T01:46:11.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:10 smithi045 conmon[39718]: cluster 2021-07-31T01:46:09.364695 2021-07-31T01:46:11.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:10 smithi045 conmon[39718]: +0000 mgr.y (mgr.14146) 278 : cluster [DBG] pgmap v254: 25 pgs: 25 active+clean; 28 MiB data, 86 MiB used, 715 GiB / 715 GiB avail; 4.2 MiB/s rd, 1.5 MiB/s wr, 23 op/s 2021-07-31T01:46:11.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:10 smithi045 conmon[39718]: cluster 2021-07-31T01:46:09.808619+0000 mon.a (mon. 2021-07-31T01:46:11.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:10 smithi045 conmon[39718]: 0) 668 : cluster [DBG] osdmap e70: 8 total, 8 up, 8 in 2021-07-31T01:46:11.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:10 smithi045 conmon[33655]: cluster 2021-07-31T01:46:09 2021-07-31T01:46:11.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:10 smithi045 conmon[33655]: .364695+0000 mgr.y (mgr.14146) 278 : cluster [DBG] pgmap v254: 25 pgs: 25 active+clean; 28 MiB data, 86 MiB used, 715 GiB / 715 GiB avail; 4.2 MiB/s rd, 1.5 MiB/s wr, 23 op/s 2021-07-31T01:46:11.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:10 smithi045 conmon[33655]: cluster 2021-07-31T01:46:11.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:10 smithi045 conmon[33655]: 2021-07-31T01:46:09.808619+0000 2021-07-31T01:46:11.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:10 smithi045 conmon[33655]: mon.a (mon.0) 668 : 2021-07-31T01:46:11.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:10 smithi045 conmon[33655]: cluster [DBG] osdmap e70: 8 total, 8 up, 8 in 2021-07-31T01:46:11.470 INFO:tasks.rados.rados.0.smithi045.stdout:131: done (0 left) 2021-07-31T01:46:11.471 INFO:tasks.rados.rados.0.smithi045.stdout:132: rollback oid 17 current snap is 6 2021-07-31T01:46:11.471 INFO:tasks.rados.rados.0.smithi045.stdout:132: done (0 left) 2021-07-31T01:46:11.471 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 43 current snap is 6 2021-07-31T01:46:11.471 INFO:tasks.rados.rados.0.smithi045.stdout:133: seq_num 65 ranges {712704=696320,1409024=401408} 2021-07-31T01:46:11.479 INFO:tasks.rados.rados.0.smithi045.stdout:133: writing smithi04569269-43 from 712704 to 1409024 tid 1 2021-07-31T01:46:11.482 INFO:tasks.rados.rados.0.smithi045.stdout:133: writing smithi04569269-43 from 1409024 to 1810432 tid 2 2021-07-31T01:46:11.482 INFO:tasks.rados.rados.0.smithi045.stdout:134: read oid 11 snap -1 2021-07-31T01:46:11.483 INFO:tasks.rados.rados.0.smithi045.stdout:134: expect (ObjNum 10 snap 0 seq_num 10) 2021-07-31T01:46:11.483 INFO:tasks.rados.rados.0.smithi045.stdout:135: read oid 6 snap -1 2021-07-31T01:46:11.483 INFO:tasks.rados.rados.0.smithi045.stdout:135: expect deleted 2021-07-31T01:46:11.483 INFO:tasks.rados.rados.0.smithi045.stdout:136: copy_from oid 18 from oid 46 current snap is 6 2021-07-31T01:46:11.484 INFO:tasks.rados.rados.0.smithi045.stdout:137: read oid 41 snap -1 2021-07-31T01:46:11.484 INFO:tasks.rados.rados.0.smithi045.stdout:137: expect (ObjNum 40 snap 0 seq_num 40) 2021-07-31T01:46:11.484 INFO:tasks.rados.rados.0.smithi045.stdout:138: rollback oid 1 current snap is 6 2021-07-31T01:46:11.484 INFO:tasks.rados.rados.0.smithi045.stdout:138: done (5 left) 2021-07-31T01:46:11.484 INFO:tasks.rados.rados.0.smithi045.stdout:139: snap_create 2021-07-31T01:46:11.485 INFO:tasks.rados.rados.0.smithi045.stdout:133: finishing write tid 1 to smithi04569269-43 2021-07-31T01:46:11.486 INFO:tasks.rados.rados.0.smithi045.stdout:133: finishing write tid 2 to smithi04569269-43 2021-07-31T01:46:11.486 INFO:tasks.rados.rados.0.smithi045.stdout:133: finishing write tid 3 to smithi04569269-43 2021-07-31T01:46:11.487 INFO:tasks.rados.rados.0.smithi045.stdout:134: expect (ObjNum 10 snap 0 seq_num 10) 2021-07-31T01:46:11.492 INFO:tasks.rados.rados.0.smithi045.stdout:133: finishing write tid 4 to smithi04569269-43 2021-07-31T01:46:11.492 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 43 v 27 (ObjNum 65 snap 6 seq_num 65) dirty exists 2021-07-31T01:46:11.492 INFO:tasks.rados.rados.0.smithi045.stdout:133: left oid 43 (ObjNum 65 snap 6 seq_num 65) 2021-07-31T01:46:11.492 INFO:tasks.rados.rados.0.smithi045.stdout:137: expect (ObjNum 40 snap 0 seq_num 40) 2021-07-31T01:46:11.525 INFO:tasks.rados.rados.0.smithi045.stdout:136: finishing copy_from to smithi04569269-18 2021-07-31T01:46:11.525 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 18 v 2 (ObjNum 45 snap 0 seq_num 45) dirty exists 2021-07-31T01:46:11.526 INFO:tasks.rados.rados.0.smithi045.stdout:136: finishing copy_from racing read to smithi04569269-18 2021-07-31T01:46:11.752 INFO:journalctl@ceph.osd.6.smithi154.stdout:Jul 31 01:46:11 smithi154 conmon[45576]: debug 2021-07-31T01:46:11.466+0000 7f47c50ad700 -1 osd.6 pg_epoch: 72 pg[3.as0( v 71'22 (0'0,71'22] local-lis/les=56/57 n=5 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [6,4,1]p6(0) r=0 lpr=56 crt=71'22 lcod 71'20 mlcod 71'20 active+clean+snaptrim trimq=[a~1] ps=[3~1,9~1]] removing snap head 2021-07-31T01:46:11.753 INFO:journalctl@ceph.osd.7.smithi154.stdout:Jul 31 01:46:11 smithi154 conmon[49149]: debug 2021-07-31T01:46:11.753 INFO:journalctl@ceph.osd.7.smithi154.stdout:Jul 31 01:46:11 smithi154 conmon[49149]: 2021-07-31T01:46:11.471+0000 7f8a75203700 -1 osd.7 pg_epoch: 72 pg[3.es0( v 71'26 (0'0,71'26] local-lis/les=56/57 n=4 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [7,4,1]p7(0) r=0 lpr=56 crt=71'26 lcod 71'24 mlcod 71'24 active+clean+snaptrim trimq=[a~1] ps=[3~1,9~1]] removing snap head 2021-07-31T01:46:11.753 INFO:journalctl@ceph.osd.7.smithi154.stdout:Jul 31 01:46:11 smithi154 conmon[49149]: 2021-07-31T01:46:11.754 INFO:journalctl@ceph.osd.7.smithi154.stdout:Jul 31 01:46:11 smithi154 conmon[49149]: debug 2021-07-31T01:46:11.754 INFO:journalctl@ceph.osd.7.smithi154.stdout:Jul 31 01:46:11 smithi154 conmon[49149]: 2021-07-31T01:46:11.474+0000 7f8a719fc700 -1 osd.7 pg_epoch: 72 pg[3.ds0( v 71'30 (0'0,71'30] local-lis/les=56/57 n=6 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [7,5,3]p7(0) r=0 lpr=56 crt=71'30 lcod 70'28 mlcod 70'28 active+clean+snaptrim trimq=[a~1] ps=[3~1,9~1]] removing snap head 2021-07-31T01:46:11.755 INFO:journalctl@ceph.osd.7.smithi154.stdout:Jul 31 01:46:11 smithi154 conmon[49149]: 2021-07-31T01:46:12.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:11 smithi154 conmon[34553]: cluster 2021-07-31T01:46:10.815799+0000 mon.a (mon.0) 669 : cluster [DBG] osdmap e71: 8 total, 8 up, 8 in 2021-07-31T01:46:12.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:11 smithi154 conmon[34553]: cluster 2021-07-31T01:46:11.467236+0000 mon.a (mon.0) 670 : cluster [DBG] osdmap e72: 8 total, 8 up, 8 in 2021-07-31T01:46:12.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:11 smithi045 conmon[39718]: cluster 2021-07-31T01:46:10.815799 2021-07-31T01:46:12.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:11 smithi045 conmon[39718]: +0000 mon.a (mon.0) 669 : cluster [DBG] osdmap e71: 8 total, 8 up, 8 in 2021-07-31T01:46:12.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:11 smithi045 conmon[39718]: cluster 2021 2021-07-31T01:46:12.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:11 smithi045 conmon[39718]: -07-31T01:46:11.467236+0000 mon.a (mon.0 2021-07-31T01:46:12.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:11 smithi045 conmon[39718]: ) 670 : cluster [DBG] osdmap e72: 8 total, 8 up, 8 in 2021-07-31T01:46:12.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:11 smithi045 conmon[33655]: cluster 2021-07-31T01:46:10 2021-07-31T01:46:12.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:11 smithi045 conmon[33655]: .815799+0000 mon.a (mon.0) 669 : cluster [DBG] osdmap e71: 8 total, 8 up, 8 in 2021-07-31T01:46:12.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:11 smithi045 conmon[33655]: cluster 2021-07-31T01:46:11 2021-07-31T01:46:12.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:11 smithi045 conmon[33655]: .467236+0000 mon.a (mon.0) 670 : cluster [DBG] osdmap e72: 8 total, 8 up, 8 in 2021-07-31T01:46:12.474 INFO:tasks.rados.rados.0.smithi045.stdout:133: done (5 left) 2021-07-31T01:46:12.475 INFO:tasks.rados.rados.0.smithi045.stdout:134: done (4 left) 2021-07-31T01:46:12.475 INFO:tasks.rados.rados.0.smithi045.stdout:135: done (3 left) 2021-07-31T01:46:12.475 INFO:tasks.rados.rados.0.smithi045.stdout:136: done (2 left) 2021-07-31T01:46:12.475 INFO:tasks.rados.rados.0.smithi045.stdout:137: done (1 left) 2021-07-31T01:46:12.476 INFO:tasks.rados.rados.0.smithi045.stdout:139: done (0 left) 2021-07-31T01:46:12.476 INFO:tasks.rados.rados.0.smithi045.stdout:140: copy_from oid 32 from oid 41 current snap is 7 2021-07-31T01:46:12.476 INFO:tasks.rados.rados.0.smithi045.stdout:141: snap_remove snap 6 2021-07-31T01:46:13.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:12 smithi154 conmon[34553]: cluster 2021-07-31T01:46:11.365008+0000 mgr.y (mgr.14146) 279 : cluster [DBG] pgmap v257: 25 pgs: 25 active+clean; 28 MiB data, 86 MiB used, 715 GiB / 715 GiB avail; 3.1 MiB/s rd, 1.5 MiB/s wr, 20 op/s 2021-07-31T01:46:13.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:12 smithi154 conmon[34553]: cluster 2021-07-31T01:46:12 2021-07-31T01:46:13.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:12 smithi154 conmon[34553]: .473724+0000 mon.a (mon.0) 671 : cluster [DBG] osdmap e73: 8 total, 8 up, 8 in 2021-07-31T01:46:13.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:12 smithi045 conmon[33655]: cluster 2021-07-31T01:46:11.365008+0000 mgr.y (mgr.14146) 279 : cluster [DBG] pgmap v257: 25 pgs: 25 active+clean; 28 MiB data, 86 MiB used, 715 GiB / 715 GiB avail; 3.1 MiB/s rd, 1.5 MiB/s wr, 20 op/s 2021-07-31T01:46:13.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:12 smithi045 conmon[33655]: 2021-07-31T01:46:13.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:12 smithi045 conmon[33655]: cluster 2021-07-31T01:46:12.473724+0000 mon.a (mon.0) 671 : cluster [DBG] osdmap e73: 8 total, 8 up, 8 in 2021-07-31T01:46:13.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:12 smithi045 conmon[39718]: cluster 2021-07-31T01:46:11.365008+0000 mgr.y (mgr.14146) 2021-07-31T01:46:13.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:12 smithi045 conmon[39718]: 279 : cluster [DBG] pgmap v257: 25 pgs: 25 active+clean; 28 MiB data, 86 MiB used, 715 GiB / 715 GiB avail; 3.1 MiB/s rd, 1.5 MiB/s wr, 20 op/s 2021-07-31T01:46:13.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:12 smithi045 conmon[39718]: cluster 2021-07-31T01:46:13.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:12 smithi045 conmon[39718]: 2021-07-31T01:46:12.473724+0000 mon.a ( 2021-07-31T01:46:13.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:12 smithi045 conmon[39718]: mon.0) 671 : cluster [DBG] osdmap e73: 8 total, 8 up, 8 in 2021-07-31T01:46:13.476 INFO:tasks.rados.rados.0.smithi045.stdout:141: done (1 left) 2021-07-31T01:46:13.476 INFO:tasks.rados.rados.0.smithi045.stdout:142: delete oid 34 current snap is 7 2021-07-31T01:46:13.476 INFO:tasks.rados.rados.0.smithi045.stdout:140: finishing copy_from to smithi04569269-32 2021-07-31T01:46:13.476 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 32 v 14 (ObjNum 40 snap 0 seq_num 40) dirty exists 2021-07-31T01:46:13.477 INFO:tasks.rados.rados.0.smithi045.stdout:140: finishing copy_from racing read to smithi04569269-32 2021-07-31T01:46:13.478 INFO:tasks.rados.rados.0.smithi045.stdout:140: done (1 left) 2021-07-31T01:46:13.479 INFO:tasks.rados.rados.0.smithi045.stdout:142: done (0 left) 2021-07-31T01:46:13.479 INFO:tasks.rados.rados.0.smithi045.stdout:143: rollback oid 46 current snap is 7 2021-07-31T01:46:13.479 INFO:tasks.rados.rados.0.smithi045.stdout:143: done (0 left) 2021-07-31T01:46:13.479 INFO:tasks.rados.rados.0.smithi045.stdout:144: read oid 46 snap -1 2021-07-31T01:46:13.480 INFO:tasks.rados.rados.0.smithi045.stdout:144: expect (ObjNum 45 snap 0 seq_num 45) 2021-07-31T01:46:13.480 INFO:tasks.rados.rados.0.smithi045.stdout:145: read oid 7 snap -1 2021-07-31T01:46:13.480 INFO:tasks.rados.rados.0.smithi045.stdout:145: expect (ObjNum 6 snap 0 seq_num 6) 2021-07-31T01:46:13.480 INFO:tasks.rados.rados.0.smithi045.stdout:146: snap_create 2021-07-31T01:46:13.482 INFO:tasks.rados.rados.0.smithi045.stdout:145: expect (ObjNum 6 snap 0 seq_num 6) 2021-07-31T01:46:13.485 INFO:tasks.rados.rados.0.smithi045.stdout:144: expect (ObjNum 45 snap 0 seq_num 45) 2021-07-31T01:46:14.477 INFO:tasks.rados.rados.0.smithi045.stdout:144: done (2 left) 2021-07-31T01:46:14.478 INFO:tasks.rados.rados.0.smithi045.stdout:145: done (1 left) 2021-07-31T01:46:14.478 INFO:tasks.rados.rados.0.smithi045.stdout:146: done (0 left) 2021-07-31T01:46:14.478 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 13 current snap is 8 2021-07-31T01:46:14.479 INFO:tasks.rados.rados.0.smithi045.stdout:147: seq_num 67 ranges {204800=729088,933888=401408} 2021-07-31T01:46:14.482 INFO:tasks.rados.rados.0.smithi045.stdout:147: writing smithi04569269-13 from 204800 to 933888 tid 1 2021-07-31T01:46:14.485 INFO:tasks.rados.rados.0.smithi045.stdout:147: writing smithi04569269-13 from 933888 to 1335296 tid 2 2021-07-31T01:46:14.486 INFO:tasks.rados.rados.0.smithi045.stdout:148: copy_from oid 15 from oid 5 current snap is 8 2021-07-31T01:46:14.486 INFO:tasks.rados.rados.0.smithi045.stdout:149: rollback oid 19 current snap is 8 2021-07-31T01:46:14.486 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 19 to 7 2021-07-31T01:46:14.486 INFO:tasks.rados.rados.0.smithi045.stdout:150: copy_from oid 21 from oid 24 current snap is 8 2021-07-31T01:46:14.487 INFO:tasks.rados.rados.0.smithi045.stdout:151: copy_from oid 3 from oid 7 current snap is 8 2021-07-31T01:46:14.487 INFO:tasks.rados.rados.0.smithi045.stdout:152: read oid 33 snap -1 2021-07-31T01:46:14.487 INFO:tasks.rados.rados.0.smithi045.stdout:152: expect (ObjNum 32 snap 0 seq_num 32) 2021-07-31T01:46:14.487 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 16 current snap is 8 2021-07-31T01:46:14.488 INFO:tasks.rados.rados.0.smithi045.stdout:153: seq_num 68 ranges {73728=745472,819200=401408} 2021-07-31T01:46:14.492 INFO:tasks.rados.rados.0.smithi045.stdout:153: writing smithi04569269-16 from 73728 to 819200 tid 1 2021-07-31T01:46:14.495 INFO:tasks.rados.rados.0.smithi045.stdout:153: writing smithi04569269-16 from 819200 to 1220608 tid 2 2021-07-31T01:46:14.496 INFO:tasks.rados.rados.0.smithi045.stdout:149: finishing rollback tid 0 to smithi04569269-19 2021-07-31T01:46:14.496 INFO:tasks.rados.rados.0.smithi045.stdout:147: finishing write tid 1 to smithi04569269-13 2021-07-31T01:46:14.496 INFO:tasks.rados.rados.0.smithi045.stdout:147: finishing write tid 2 to smithi04569269-13 2021-07-31T01:46:14.497 INFO:tasks.rados.rados.0.smithi045.stdout:149: finishing rollback tid 1 to smithi04569269-19 2021-07-31T01:46:14.497 INFO:tasks.rados.rados.0.smithi045.stdout:147: finishing write tid 3 to smithi04569269-13 2021-07-31T01:46:14.497 INFO:tasks.rados.rados.0.smithi045.stdout:149: finishing rollback tid 2 to smithi04569269-19 2021-07-31T01:46:14.497 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 19 v 21 (ObjNum 18 snap 0 seq_num 18) dirty exists 2021-07-31T01:46:14.498 INFO:tasks.rados.rados.0.smithi045.stdout:147: finishing write tid 4 to smithi04569269-13 2021-07-31T01:46:14.498 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 13 v 32 (ObjNum 67 snap 8 seq_num 67) dirty exists 2021-07-31T01:46:14.498 INFO:tasks.rados.rados.0.smithi045.stdout:147: left oid 13 (ObjNum 67 snap 8 seq_num 67) 2021-07-31T01:46:14.499 INFO:tasks.rados.rados.0.smithi045.stdout:148: finishing copy_from to smithi04569269-15 2021-07-31T01:46:14.499 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 15 v 2 (ObjNum 4 snap 0 seq_num 4) dirty exists 2021-07-31T01:46:14.499 INFO:tasks.rados.rados.0.smithi045.stdout:148: finishing copy_from racing read to smithi04569269-15 2021-07-31T01:46:14.499 INFO:tasks.rados.rados.0.smithi045.stdout:147: done (6 left) 2021-07-31T01:46:14.500 INFO:tasks.rados.rados.0.smithi045.stdout:148: done (5 left) 2021-07-31T01:46:14.500 INFO:tasks.rados.rados.0.smithi045.stdout:149: done (4 left) 2021-07-31T01:46:14.500 INFO:tasks.rados.rados.0.smithi045.stdout:154: rmattr oid 42 current snap is 8 2021-07-31T01:46:14.500 INFO:tasks.rados.rados.0.smithi045.stdout:154: done (4 left) 2021-07-31T01:46:14.500 INFO:tasks.rados.rados.0.smithi045.stdout:155: snap_remove snap 7 2021-07-31T01:46:14.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:14 smithi154 conmon[34553]: cluster 2021-07-31T01:46:13.365435+0000 mgr.y (mgr.14146) 280 : cluster [DBG] 2021-07-31T01:46:14.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:14 smithi154 conmon[34553]: pgmap v260: 25 pgs: 25 active+clean; 28 MiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.3 MiB/s rd, 434 KiB/s wr, 22 op/s 2021-07-31T01:46:14.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:14 smithi154 conmon[34553]: cluster 2021-07-31T01:46:13.475431+0000 mon.a (mon.0) 672 : cluster [DBG] osdmap e74: 8 total, 8 up, 8 in 2021-07-31T01:46:14.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:14 smithi045 conmon[39718]: cluster 2021-07-31T01:46:13.365435+0000 mgr.y (mgr.14146) 280 : cluster [DBG] pgmap v260: 25 pgs: 25 active+clean; 28 MiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.3 MiB/s rd, 434 KiB/s wr, 22 op/s 2021-07-31T01:46:14.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:14 smithi045 conmon[39718]: cluster 2021-07-31T01:46:13.475431+0000 mon.a (mon.0) 672 : cluster [DBG] osdmap e74: 8 total, 8 up, 8 in 2021-07-31T01:46:14.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:14 smithi045 conmon[33655]: cluster 2021-07-31T01:46:13.365435+0000 mgr.y (mgr.14146) 280 : cluster [DBG] pgmap v260: 25 pgs: 25 active+clean; 28 MiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.3 MiB/s rd, 434 KiB/s wr, 22 op/s 2021-07-31T01:46:14.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:14 smithi045 conmon[33655]: cluster 2021-07-31T01:46:13.475431+0000 mon.a (mon.0) 672 : cluster [DBG] osdmap e74: 8 total, 8 up, 8 in 2021-07-31T01:46:15.487 INFO:tasks.rados.rados.0.smithi045.stdout:155: done (4 left) 2021-07-31T01:46:15.487 INFO:tasks.rados.rados.0.smithi045.stdout:156: read oid 27 snap -1 2021-07-31T01:46:15.487 INFO:tasks.rados.rados.0.smithi045.stdout:156: expect (ObjNum 26 snap 0 seq_num 26) 2021-07-31T01:46:15.488 INFO:tasks.rados.rados.0.smithi045.stdout:157: rmattr oid 19 current snap is 8 2021-07-31T01:46:15.488 INFO:tasks.rados.rados.0.smithi045.stdout:157: done (5 left) 2021-07-31T01:46:15.488 INFO:tasks.rados.rados.0.smithi045.stdout:158: snap_create 2021-07-31T01:46:15.488 INFO:tasks.rados.rados.0.smithi045.stdout:152: expect (ObjNum 32 snap 0 seq_num 32) 2021-07-31T01:46:15.503 INFO:tasks.rados.rados.0.smithi045.stdout:151: finishing copy_from to smithi04569269-3 2021-07-31T01:46:15.503 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 3 v 2 (ObjNum 6 snap 0 seq_num 6) dirty exists 2021-07-31T01:46:15.503 INFO:tasks.rados.rados.0.smithi045.stdout:151: finishing copy_from racing read to smithi04569269-3 2021-07-31T01:46:15.504 INFO:tasks.rados.rados.0.smithi045.stdout:150: finishing copy_from to smithi04569269-21 2021-07-31T01:46:15.504 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 21 v 8 (ObjNum 54 snap 4 seq_num 54) dirty exists 2021-07-31T01:46:15.504 INFO:tasks.rados.rados.0.smithi045.stdout:153: finishing write tid 1 to smithi04569269-16 2021-07-31T01:46:15.504 INFO:tasks.rados.rados.0.smithi045.stdout:150: finishing copy_from racing read to smithi04569269-21 2021-07-31T01:46:15.505 INFO:tasks.rados.rados.0.smithi045.stdout:153: finishing write tid 2 to smithi04569269-16 2021-07-31T01:46:15.505 INFO:tasks.rados.rados.0.smithi045.stdout:153: finishing write tid 3 to smithi04569269-16 2021-07-31T01:46:15.505 INFO:tasks.rados.rados.0.smithi045.stdout:153: finishing write tid 4 to smithi04569269-16 2021-07-31T01:46:15.505 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 16 v 25 (ObjNum 68 snap 8 seq_num 68) dirty exists 2021-07-31T01:46:15.505 INFO:tasks.rados.rados.0.smithi045.stdout:153: left oid 16 (ObjNum 68 snap 8 seq_num 68) 2021-07-31T01:46:15.506 INFO:tasks.rados.rados.0.smithi045.stdout:156: expect (ObjNum 26 snap 0 seq_num 26) 2021-07-31T01:46:15.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:15 smithi154 conmon[34553]: cluster 2021-07-31T01:46:14.477067+0000 mon.a (mon.0) 673 : cluster [DBG] osdmap e75: 8 total, 8 up, 8 in 2021-07-31T01:46:15.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:15 smithi045 conmon[33655]: cluster 2021-07-31T01:46:14.477067+0000 mon.a ( 2021-07-31T01:46:15.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:15 smithi045 conmon[33655]: mon.0) 673 : cluster [DBG] osdmap e75: 8 total, 8 up, 8 in 2021-07-31T01:46:15.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:15 smithi045 conmon[39718]: cluster 2021-07-31T01:46:15.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:15 smithi045 conmon[39718]: 2021-07-31T01:46:15.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:15 smithi045 conmon[39718]: 2021 2021-07-31T01:46:15.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:15 smithi045 conmon[39718]: - 2021-07-31T01:46:15.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:15 smithi045 conmon[39718]: 07 2021-07-31T01:46:15.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:15 smithi045 conmon[39718]: - 2021-07-31T01:46:15.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:15 smithi045 conmon[39718]: 31 2021-07-31T01:46:15.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:15 smithi045 conmon[39718]: T 2021-07-31T01:46:15.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:15 smithi045 conmon[39718]: 01:46:14.477067+0000 2021-07-31T01:46:15.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:15 smithi045 conmon[39718]: mon.a (mon.0) 673 : cluster [DBG] osdmap e75: 8 total, 8 up, 8 in 2021-07-31T01:46:16.469 INFO:tasks.rados.rados.0.smithi045.stdout:150: done (5 left) 2021-07-31T01:46:16.469 INFO:tasks.rados.rados.0.smithi045.stdout:151: done (4 left) 2021-07-31T01:46:16.470 INFO:tasks.rados.rados.0.smithi045.stdout:152: done (3 left) 2021-07-31T01:46:16.470 INFO:tasks.rados.rados.0.smithi045.stdout:153: done (2 left) 2021-07-31T01:46:16.470 INFO:tasks.rados.rados.0.smithi045.stdout:156: done (1 left) 2021-07-31T01:46:16.470 INFO:tasks.rados.rados.0.smithi045.stdout:158: done (0 left) 2021-07-31T01:46:16.471 INFO:tasks.rados.rados.0.smithi045.stdout:159: copy_from oid 17 from oid 8 current snap is 9 2021-07-31T01:46:16.471 INFO:tasks.rados.rados.0.smithi045.stdout:160: read oid 32 snap -1 2021-07-31T01:46:16.471 INFO:tasks.rados.rados.0.smithi045.stdout:160: expect (ObjNum 40 snap 0 seq_num 40) 2021-07-31T01:46:16.471 INFO:tasks.rados.rados.0.smithi045.stdout:161: snap_remove snap 8 2021-07-31T01:46:16.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:16 smithi154 conmon[34553]: cluster 2021-07-31T01:46:15.365790+0000 mgr.y (mgr.14146) 281 : cluster 2021-07-31T01:46:16.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:16 smithi154 conmon[34553]: [DBG] pgmap v263: 25 pgs: 25 active+clean; 26 MiB data, 92 MiB used, 715 GiB / 715 GiB avail; 1.3 MiB/s rd, 620 KiB/s wr, 22 op/s 2021-07-31T01:46:16.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:16 smithi154 conmon[34553]: cluster 2021-07-31T01:46:15.485963+0000 mon.a (mon.0) 674 : cluster [DBG] osdmap e76: 8 total, 8 up, 8 in 2021-07-31T01:46:16.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:16 smithi154 conmon[34553]: cluster 2021-07-31T01:46:16.468345+0000 mon.a (mon.0) 675 : cluster [DBG] osdmap e77: 8 total, 8 up, 8 in 2021-07-31T01:46:16.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:16 smithi045 conmon[39718]: cluster 2021-07-31T01:46:15. 2021-07-31T01:46:16.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:16 smithi045 conmon[39718]: 365790+0000 mgr.y (mgr.14146) 281 : cluster [DBG] pgmap v263: 25 pgs: 25 active+clean; 26 MiB data, 92 MiB used, 715 GiB / 715 GiB avail; 1.3 MiB/s rd, 620 KiB/s wr, 22 op/s 2021-07-31T01:46:16.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:16 smithi045 conmon[39718]: cluster 2021-07-31T01:46:15.485963+0000 mon.a ( 2021-07-31T01:46:16.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:16 smithi045 conmon[39718]: mon.0) 674 : cluster [DBG] osdmap e76: 8 total, 8 up, 8 in 2021-07-31T01:46:16.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:16 smithi045 conmon[39718]: cluster 2021-07-31T01:46:16. 2021-07-31T01:46:16.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:16 smithi045 conmon[39718]: 468345+0000 mon.a (mon.0) 675 : cluster [DBG] osdmap e77: 8 total, 8 up, 8 in 2021-07-31T01:46:16.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:16 smithi045 conmon[33655]: cluster 2021-07-31T01:46:15.365790+0000 mgr.y 2021-07-31T01:46:16.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:16 smithi045 conmon[33655]: (mgr.14146) 281 : cluster [DBG] pgmap v263: 25 pgs: 25 active+clean; 26 MiB data, 92 MiB used, 715 GiB / 715 GiB avail; 1.3 MiB/s rd, 620 KiB/s wr, 22 op/s 2021-07-31T01:46:16.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:16 smithi045 conmon[33655]: cluster 2021-07-31T01:46:15.485963+0000 mon.a (mon.0) 674 : cluster [DBG] osdmap e76: 8 total, 8 up, 8 in 2021-07-31T01:46:16.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:16 smithi045 conmon[33655]: cluster 2021-07-31T01:46:16.468345+0000 mon.a (mon.0) 675 : 2021-07-31T01:46:16.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:16 smithi045 conmon[33655]: cluster [DBG] osdmap e77: 8 total, 8 up, 8 in 2021-07-31T01:46:17.474 INFO:tasks.rados.rados.0.smithi045.stdout:161: done (2 left) 2021-07-31T01:46:17.474 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 2 current snap is 9 2021-07-31T01:46:17.475 INFO:tasks.rados.rados.0.smithi045.stdout:159: finishing copy_from to smithi04569269-17 2021-07-31T01:46:17.475 INFO:tasks.rados.rados.0.smithi045.stdout:159: got expected ENOENT (src dne) 2021-07-31T01:46:17.475 INFO:tasks.rados.rados.0.smithi045.stdout:162: seq_num 69 ranges {24576=761856,786432=401408} 2021-07-31T01:46:17.478 INFO:tasks.rados.rados.0.smithi045.stdout:162: writing smithi04569269-2 from 24576 to 786432 tid 1 2021-07-31T01:46:17.481 INFO:tasks.rados.rados.0.smithi045.stdout:162: writing smithi04569269-2 from 786432 to 1187840 tid 2 2021-07-31T01:46:17.483 INFO:tasks.rados.rados.0.smithi045.stdout:159: finishing copy_from racing read to smithi04569269-17 2021-07-31T01:46:17.483 INFO:tasks.rados.rados.0.smithi045.stdout:160: expect (ObjNum 40 snap 0 seq_num 40) 2021-07-31T01:46:17.508 INFO:tasks.rados.rados.0.smithi045.stdout:159: done (2 left) 2021-07-31T01:46:17.508 INFO:tasks.rados.rados.0.smithi045.stdout:160: done (1 left) 2021-07-31T01:46:17.509 INFO:tasks.rados.rados.0.smithi045.stdout:163: copy_from oid 29 from oid 40 current snap is 9 2021-07-31T01:46:17.509 INFO:tasks.rados.rados.0.smithi045.stdout:162: finishing write tid 1 to smithi04569269-2 2021-07-31T01:46:17.509 INFO:tasks.rados.rados.0.smithi045.stdout:162: finishing write tid 2 to smithi04569269-2 2021-07-31T01:46:17.510 INFO:tasks.rados.rados.0.smithi045.stdout:162: finishing write tid 3 to smithi04569269-2 2021-07-31T01:46:17.510 INFO:tasks.rados.rados.0.smithi045.stdout:162: finishing write tid 4 to smithi04569269-2 2021-07-31T01:46:17.510 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 2 v 17 (ObjNum 69 snap 9 seq_num 69) dirty exists 2021-07-31T01:46:17.511 INFO:tasks.rados.rados.0.smithi045.stdout:162: left oid 2 (ObjNum 69 snap 9 seq_num 69) 2021-07-31T01:46:17.511 INFO:tasks.rados.rados.0.smithi045.stdout:162: done (1 left) 2021-07-31T01:46:17.511 INFO:tasks.rados.rados.0.smithi045.stdout:164: rollback oid 31 current snap is 9 2021-07-31T01:46:17.511 INFO:tasks.rados.rados.0.smithi045.stdout:164: done (1 left) 2021-07-31T01:46:17.511 INFO:tasks.rados.rados.0.smithi045.stdout:165: delete oid 7 current snap is 9 2021-07-31T01:46:17.512 INFO:tasks.rados.rados.0.smithi045.stdout:165: done (1 left) 2021-07-31T01:46:17.512 INFO:tasks.rados.rados.0.smithi045.stdout:166: snap_create 2021-07-31T01:46:17.514 INFO:tasks.rados.rados.0.smithi045.stdout:163: finishing copy_from to smithi04569269-29 2021-07-31T01:46:17.515 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 29 v 14 (ObjNum 39 snap 0 seq_num 39) dirty exists 2021-07-31T01:46:17.515 INFO:tasks.rados.rados.0.smithi045.stdout:163: finishing copy_from racing read to smithi04569269-29 2021-07-31T01:46:18.490 INFO:tasks.rados.rados.0.smithi045.stdout:163: done (1 left) 2021-07-31T01:46:18.491 INFO:tasks.rados.rados.0.smithi045.stdout:166: done (0 left) 2021-07-31T01:46:18.491 INFO:tasks.rados.rados.0.smithi045.stdout:167: rmattr oid 47 current snap is 10 2021-07-31T01:46:18.491 INFO:tasks.rados.rados.0.smithi045.stdout:167: done (0 left) 2021-07-31T01:46:18.492 INFO:tasks.rados.rados.0.smithi045.stdout:168: rmattr oid 39 current snap is 10 2021-07-31T01:46:18.492 INFO:tasks.rados.rados.0.smithi045.stdout:168: done (0 left) 2021-07-31T01:46:18.492 INFO:tasks.rados.rados.0.smithi045.stdout:169: read oid 19 snap -1 2021-07-31T01:46:18.492 INFO:tasks.rados.rados.0.smithi045.stdout:169: expect (ObjNum 18 snap 0 seq_num 18) 2021-07-31T01:46:18.492 INFO:tasks.rados.rados.0.smithi045.stdout:170: read oid 38 snap -1 2021-07-31T01:46:18.493 INFO:tasks.rados.rados.0.smithi045.stdout:170: expect (ObjNum 37 snap 0 seq_num 37) 2021-07-31T01:46:18.493 INFO:tasks.rados.rados.0.smithi045.stdout:171: setattr oid 15 current snap is 10 2021-07-31T01:46:18.494 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 1 current snap is 10 2021-07-31T01:46:18.494 INFO:tasks.rados.rados.0.smithi045.stdout:172: seq_num 71 ranges {0=794624,794624=401408} 2021-07-31T01:46:18.497 INFO:tasks.rados.rados.0.smithi045.stdout:172: writing smithi04569269-1 from 0 to 794624 tid 1 2021-07-31T01:46:18.499 INFO:tasks.rados.rados.0.smithi045.stdout:172: writing smithi04569269-1 from 794624 to 1196032 tid 2 2021-07-31T01:46:18.504 INFO:tasks.rados.rados.0.smithi045.stdout:169: expect (ObjNum 18 snap 0 seq_num 18) 2021-07-31T01:46:18.508 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 15 v 28 (ObjNum 4 snap 0 seq_num 4) dirty exists 2021-07-31T01:46:18.509 INFO:tasks.rados.rados.0.smithi045.stdout:169: done (3 left) 2021-07-31T01:46:18.509 INFO:tasks.rados.rados.0.smithi045.stdout:171: done (2 left) 2021-07-31T01:46:18.509 INFO:tasks.rados.rados.0.smithi045.stdout:173: copy_from oid 15 from oid 5 current snap is 10 2021-07-31T01:46:18.509 INFO:tasks.rados.rados.0.smithi045.stdout:170: expect (ObjNum 37 snap 0 seq_num 37) 2021-07-31T01:46:18.529 INFO:tasks.rados.rados.0.smithi045.stdout:172: finishing write tid 1 to smithi04569269-1 2021-07-31T01:46:18.529 INFO:tasks.rados.rados.0.smithi045.stdout:172: finishing write tid 2 to smithi04569269-1 2021-07-31T01:46:18.529 INFO:tasks.rados.rados.0.smithi045.stdout:172: finishing write tid 3 to smithi04569269-1 2021-07-31T01:46:18.529 INFO:tasks.rados.rados.0.smithi045.stdout:172: finishing write tid 4 to smithi04569269-1 2021-07-31T01:46:18.530 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 1 v 17 (ObjNum 71 snap 10 seq_num 71) dirty exists 2021-07-31T01:46:18.530 INFO:tasks.rados.rados.0.smithi045.stdout:172: left oid 1 (ObjNum 71 snap 10 seq_num 71) 2021-07-31T01:46:18.530 INFO:tasks.rados.rados.0.smithi045.stdout:170: done (2 left) 2021-07-31T01:46:18.530 INFO:tasks.rados.rados.0.smithi045.stdout:172: done (1 left) 2021-07-31T01:46:18.531 INFO:tasks.rados.rados.0.smithi045.stdout:174: snap_create 2021-07-31T01:46:18.531 INFO:tasks.rados.rados.0.smithi045.stdout:173: finishing copy_from to smithi04569269-15 2021-07-31T01:46:18.531 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 15 v 2 (ObjNum 4 snap 0 seq_num 4) dirty exists 2021-07-31T01:46:18.531 INFO:tasks.rados.rados.0.smithi045.stdout:173: finishing copy_from racing read to smithi04569269-15 2021-07-31T01:46:18.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:18 smithi154 conmon[34553]: cluster 2021-07-31T01:46:17.366136+0000 mgr.y (mgr.14146) 282 : cluster [DBG] 2021-07-31T01:46:18.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:18 smithi154 conmon[34553]: pgmap v266: 25 pgs: 25 active+clean; 26 MiB data, 94 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 186 KiB/s wr, 1 op/s 2021-07-31T01:46:18.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:18 smithi154 conmon[34553]: cluster 2021-07-31T01:46:17.473537+0000 mon.a (mon.0) 676 : cluster [DBG] osdmap e78: 8 total, 8 up, 8 in 2021-07-31T01:46:18.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:18 smithi045 conmon[33655]: cluster 2021-07- 2021-07-31T01:46:18.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:18 smithi045 conmon[33655]: 31T01:46:17.366136+0000 mgr.y (mgr.14146) 282 : cluster [DBG] pgmap v266: 25 pgs: 25 active+clean; 26 MiB data, 94 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 186 KiB/s wr, 1 op/s 2021-07-31T01:46:18.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:18 smithi045 conmon[33655]: cluster 2021-07-31T01 2021-07-31T01:46:18.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:18 smithi045 conmon[33655]: :46:17.473537+0000 mon.a (mon.0) 676 2021-07-31T01:46:18.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:18 smithi045 conmon[33655]: : cluster [DBG] osdmap e78: 8 total, 8 up, 8 in 2021-07-31T01:46:18.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:18 smithi045 conmon[39718]: cluster 2021-07-31T01: 2021-07-31T01:46:18.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:18 smithi045 conmon[39718]: 46:17.366136+0000 mgr.y (mgr.14146) 282 : cluster [DBG] pgmap v266: 25 pgs: 25 active+clean; 26 MiB data, 94 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 186 KiB/s wr, 1 op/s 2021-07-31T01:46:18.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:18 smithi045 conmon[39718]: 2021-07-31T01:46:18.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:18 smithi045 conmon[39718]: cluster 2021- 2021-07-31T01:46:18.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:18 smithi045 conmon[39718]: 07-31T01:46:17 2021-07-31T01:46:18.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:18 smithi045 conmon[39718]: .473537+0000 mon.a (mon. 2021-07-31T01:46:18.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:18 smithi045 conmon[39718]: 0) 676 : cluster [DBG] 2021-07-31T01:46:18.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:18 smithi045 conmon[39718]: osdmap e78: 8 total, 8 up, 8 in 2021-07-31T01:46:19.496 INFO:tasks.rados.rados.0.smithi045.stdout:173: done (1 left) 2021-07-31T01:46:19.496 INFO:tasks.rados.rados.0.smithi045.stdout:174: done (0 left) 2021-07-31T01:46:19.497 INFO:tasks.rados.rados.0.smithi045.stdout:175: delete oid 31 current snap is 11 2021-07-31T01:46:19.499 INFO:tasks.rados.rados.0.smithi045.stdout:175: done (0 left) 2021-07-31T01:46:19.499 INFO:tasks.rados.rados.0.smithi045.stdout:176: delete oid 26 current snap is 11 2021-07-31T01:46:19.503 INFO:tasks.rados.rados.0.smithi045.stdout:176: done (0 left) 2021-07-31T01:46:19.503 INFO:tasks.rados.rados.0.smithi045.stdout:177: copy_from oid 20 from oid 42 current snap is 11 2021-07-31T01:46:19.503 INFO:tasks.rados.rados.0.smithi045.stdout:178: delete oid 5 current snap is 11 2021-07-31T01:46:19.506 INFO:tasks.rados.rados.0.smithi045.stdout:178: done (1 left) 2021-07-31T01:46:19.506 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 13 current snap is 11 2021-07-31T01:46:19.507 INFO:tasks.rados.rados.0.smithi045.stdout:179: seq_num 75 ranges {1335296=458752,1794048=581632,2375680=221184} 2021-07-31T01:46:19.517 INFO:tasks.rados.rados.0.smithi045.stdout:179: writing smithi04569269-13 from 1335296 to 1794048 tid 1 2021-07-31T01:46:19.521 INFO:tasks.rados.rados.0.smithi045.stdout:179: writing smithi04569269-13 from 1794048 to 2375680 tid 2 2021-07-31T01:46:19.523 INFO:tasks.rados.rados.0.smithi045.stdout:179: writing smithi04569269-13 from 2375680 to 2596864 tid 3 2021-07-31T01:46:19.523 INFO:tasks.rados.rados.0.smithi045.stdout:177: finishing copy_from to smithi04569269-20 2021-07-31T01:46:19.523 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 20 v 8 (ObjNum 41 snap 0 seq_num 41) dirty exists 2021-07-31T01:46:19.524 INFO:tasks.rados.rados.0.smithi045.stdout:177: finishing copy_from racing read to smithi04569269-20 2021-07-31T01:46:19.524 INFO:tasks.rados.rados.0.smithi045.stdout:179: finishing write tid 1 to smithi04569269-13 2021-07-31T01:46:19.524 INFO:tasks.rados.rados.0.smithi045.stdout:177: done (1 left) 2021-07-31T01:46:19.524 INFO:tasks.rados.rados.0.smithi045.stdout:180: read oid 34 snap -1 2021-07-31T01:46:19.524 INFO:tasks.rados.rados.0.smithi045.stdout:180: expect deleted 2021-07-31T01:46:19.525 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 42 current snap is 11 2021-07-31T01:46:19.525 INFO:tasks.rados.rados.0.smithi045.stdout:181: seq_num 76 ranges {696320=475136,1171456=458752,1630208=344064} 2021-07-31T01:46:19.530 INFO:tasks.rados.rados.0.smithi045.stdout:181: writing smithi04569269-42 from 696320 to 1171456 tid 1 2021-07-31T01:46:19.533 INFO:tasks.rados.rados.0.smithi045.stdout:181: writing smithi04569269-42 from 1171456 to 1630208 tid 2 2021-07-31T01:46:19.536 INFO:tasks.rados.rados.0.smithi045.stdout:181: writing smithi04569269-42 from 1630208 to 1974272 tid 3 2021-07-31T01:46:19.536 INFO:tasks.rados.rados.0.smithi045.stdout:179: finishing write tid 2 to smithi04569269-13 2021-07-31T01:46:19.536 INFO:tasks.rados.rados.0.smithi045.stdout:179: finishing write tid 3 to smithi04569269-13 2021-07-31T01:46:19.536 INFO:tasks.rados.rados.0.smithi045.stdout:179: finishing write tid 4 to smithi04569269-13 2021-07-31T01:46:19.537 INFO:tasks.rados.rados.0.smithi045.stdout:179: finishing write tid 5 to smithi04569269-13 2021-07-31T01:46:19.537 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 13 v 39 (ObjNum 75 snap 11 seq_num 75) dirty exists 2021-07-31T01:46:19.537 INFO:tasks.rados.rados.0.smithi045.stdout:179: left oid 13 (ObjNum 75 snap 11 seq_num 75) 2021-07-31T01:46:19.537 INFO:tasks.rados.rados.0.smithi045.stdout:181: finishing write tid 1 to smithi04569269-42 2021-07-31T01:46:19.538 INFO:tasks.rados.rados.0.smithi045.stdout:179: done (2 left) 2021-07-31T01:46:19.538 INFO:tasks.rados.rados.0.smithi045.stdout:180: done (1 left) 2021-07-31T01:46:19.538 INFO:tasks.rados.rados.0.smithi045.stdout:182: delete oid 34 current snap is 11 2021-07-31T01:46:19.538 INFO:tasks.rados.rados.0.smithi045.stdout:181: finishing write tid 2 to smithi04569269-42 2021-07-31T01:46:19.540 INFO:tasks.rados.rados.0.smithi045.stdout:182: done (1 left) 2021-07-31T01:46:19.540 INFO:tasks.rados.rados.0.smithi045.stdout:183: read oid 31 snap -1 2021-07-31T01:46:19.540 INFO:tasks.rados.rados.0.smithi045.stdout:183: expect deleted 2021-07-31T01:46:19.541 INFO:tasks.rados.rados.0.smithi045.stdout:184: copy_from oid 27 from oid 4 current snap is 11 2021-07-31T01:46:19.541 INFO:tasks.rados.rados.0.smithi045.stdout:185: copy_from oid 20 from oid 21 current snap is 11 2021-07-31T01:46:19.541 INFO:tasks.rados.rados.0.smithi045.stdout:186: read oid 8 snap 10 2021-07-31T01:46:19.541 INFO:tasks.rados.rados.0.smithi045.stdout:186: expect deleted 2021-07-31T01:46:19.541 INFO:tasks.rados.rados.0.smithi045.stdout:187: delete oid 32 current snap is 11 2021-07-31T01:46:19.542 INFO:tasks.rados.rados.0.smithi045.stdout:181: finishing write tid 3 to smithi04569269-42 2021-07-31T01:46:19.542 INFO:tasks.rados.rados.0.smithi045.stdout:181: finishing write tid 4 to smithi04569269-42 2021-07-31T01:46:19.542 INFO:tasks.rados.rados.0.smithi045.stdout:181: finishing write tid 5 to smithi04569269-42 2021-07-31T01:46:19.542 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 42 v 37 (ObjNum 76 snap 11 seq_num 76) dirty exists 2021-07-31T01:46:19.542 INFO:tasks.rados.rados.0.smithi045.stdout:181: left oid 42 (ObjNum 76 snap 11 seq_num 76) 2021-07-31T01:46:19.543 INFO:tasks.rados.rados.0.smithi045.stdout:181: done (5 left) 2021-07-31T01:46:19.543 INFO:tasks.rados.rados.0.smithi045.stdout:183: done (4 left) 2021-07-31T01:46:19.543 INFO:tasks.rados.rados.0.smithi045.stdout:186: done (3 left) 2021-07-31T01:46:19.543 INFO:tasks.rados.rados.0.smithi045.stdout:187: done (2 left) 2021-07-31T01:46:19.543 INFO:tasks.rados.rados.0.smithi045.stdout:188: rollback oid 31 current snap is 11 2021-07-31T01:46:19.544 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 31 to 10 2021-07-31T01:46:19.544 INFO:tasks.rados.rados.0.smithi045.stdout:189: delete oid 44 current snap is 11 2021-07-31T01:46:19.545 INFO:tasks.rados.rados.0.smithi045.stdout:184: finishing copy_from to smithi04569269-27 2021-07-31T01:46:19.545 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 27 v 2 (ObjNum 3 snap 0 seq_num 3) dirty exists 2021-07-31T01:46:19.545 INFO:tasks.rados.rados.0.smithi045.stdout:184: finishing copy_from racing read to smithi04569269-27 2021-07-31T01:46:19.545 INFO:tasks.rados.rados.0.smithi045.stdout:184: done (3 left) 2021-07-31T01:46:19.546 INFO:tasks.rados.rados.0.smithi045.stdout:189: done (2 left) 2021-07-31T01:46:19.546 INFO:tasks.rados.rados.0.smithi045.stdout:190: copy_from oid 41 from oid 12 current snap is 11 2021-07-31T01:46:19.546 INFO:tasks.rados.rados.0.smithi045.stdout:191: rollback oid 45 current snap is 11 2021-07-31T01:46:19.546 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 45 to 9 2021-07-31T01:46:19.547 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 16 current snap is 11 2021-07-31T01:46:19.547 INFO:tasks.rados.rados.0.smithi045.stdout:192: seq_num 80 ranges {1220608=548864,1769472=450560,2220032=352256} 2021-07-31T01:46:19.554 INFO:tasks.rados.rados.0.smithi045.stdout:192: writing smithi04569269-16 from 1220608 to 1769472 tid 1 2021-07-31T01:46:19.557 INFO:tasks.rados.rados.0.smithi045.stdout:192: writing smithi04569269-16 from 1769472 to 2220032 tid 2 2021-07-31T01:46:19.560 INFO:tasks.rados.rados.0.smithi045.stdout:192: writing smithi04569269-16 from 2220032 to 2572288 tid 3 2021-07-31T01:46:19.560 INFO:tasks.rados.rados.0.smithi045.stdout:190: finishing copy_from to smithi04569269-41 2021-07-31T01:46:19.561 INFO:tasks.rados.rados.0.smithi045.stdout:190: got expected ENOENT (src dne) 2021-07-31T01:46:19.561 INFO:tasks.rados.rados.0.smithi045.stdout:190: finishing copy_from racing read to smithi04569269-41 2021-07-31T01:46:19.561 INFO:tasks.rados.rados.0.smithi045.stdout:188: finishing rollback tid 1 to smithi04569269-31 2021-07-31T01:46:19.561 INFO:tasks.rados.rados.0.smithi045.stdout:188: finishing rollback tid 2 to smithi04569269-31 2021-07-31T01:46:19.561 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 31 v 91 (ObjNum 30 snap 0 seq_num 30) dirty exists 2021-07-31T01:46:19.562 INFO:tasks.rados.rados.0.smithi045.stdout:185: finishing copy_from to smithi04569269-20 2021-07-31T01:46:19.562 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 20 v 8 (ObjNum 54 snap 4 seq_num 54) dirty exists 2021-07-31T01:46:19.562 INFO:tasks.rados.rados.0.smithi045.stdout:191: finishing rollback tid 0 to smithi04569269-45 2021-07-31T01:46:19.562 INFO:tasks.rados.rados.0.smithi045.stdout:185: finishing copy_from racing read to smithi04569269-20 2021-07-31T01:46:19.563 INFO:tasks.rados.rados.0.smithi045.stdout:191: finishing rollback tid 1 to smithi04569269-45 2021-07-31T01:46:19.563 INFO:tasks.rados.rados.0.smithi045.stdout:191: finishing rollback tid 2 to smithi04569269-45 2021-07-31T01:46:19.563 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 45 v 41 (ObjNum 33 snap 0 seq_num 33) dirty exists 2021-07-31T01:46:19.563 INFO:tasks.rados.rados.0.smithi045.stdout:192: finishing write tid 1 to smithi04569269-16 2021-07-31T01:46:19.563 INFO:tasks.rados.rados.0.smithi045.stdout:192: finishing write tid 2 to smithi04569269-16 2021-07-31T01:46:19.564 INFO:tasks.rados.rados.0.smithi045.stdout:185: done (4 left) 2021-07-31T01:46:19.564 INFO:tasks.rados.rados.0.smithi045.stdout:188: done (3 left) 2021-07-31T01:46:19.564 INFO:tasks.rados.rados.0.smithi045.stdout:190: done (2 left) 2021-07-31T01:46:19.564 INFO:tasks.rados.rados.0.smithi045.stdout:191: done (1 left) 2021-07-31T01:46:19.564 INFO:tasks.rados.rados.0.smithi045.stdout:193: rollback oid 41 current snap is 11 2021-07-31T01:46:19.565 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 41 to 10 2021-07-31T01:46:19.565 INFO:tasks.rados.rados.0.smithi045.stdout:194: snap_remove snap 9 2021-07-31T01:46:19.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:19 smithi154 conmon[34553]: cluster 2021-07-31T01:46:18.489065+0000 mon.a (mon.0) 677 : cluster [DBG] osdmap e79: 8 total, 8 up, 8 in 2021-07-31T01:46:19.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:19 smithi045 conmon[39718]: cluster 2021-07-31T01:46:18 2021-07-31T01:46:19.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:19 smithi045 conmon[39718]: .489065+0000 mon.a (mon.0) 677 : cluster [DBG] osdmap e79: 8 total, 8 up, 8 in 2021-07-31T01:46:19.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:19 smithi045 conmon[33655]: cluster 2021-07-31T01:46:19.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:19 smithi045 conmon[33655]: 2021-07-31T01:46:18.489065 2021-07-31T01:46:19.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:19 smithi045 conmon[33655]: +0000 mon.a (mon.0) 677 : cluster [DBG] osdmap e79: 8 total, 8 up, 8 in 2021-07-31T01:46:20.502 INFO:tasks.rados.rados.0.smithi045.stdout:193: finishing rollback tid 0 to smithi04569269-41 2021-07-31T01:46:20.502 INFO:tasks.rados.rados.0.smithi045.stdout:194: done (2 left) 2021-07-31T01:46:20.503 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 24 current snap is 11 2021-07-31T01:46:20.503 INFO:tasks.rados.rados.0.smithi045.stdout:193: finishing rollback tid 1 to smithi04569269-41 2021-07-31T01:46:20.503 INFO:tasks.rados.rados.0.smithi045.stdout:192: finishing write tid 3 to smithi04569269-16 2021-07-31T01:46:20.503 INFO:tasks.rados.rados.0.smithi045.stdout:193: finishing rollback tid 2 to smithi04569269-41 2021-07-31T01:46:20.504 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 41 v 25 (ObjNum 40 snap 0 seq_num 40) dirty exists 2021-07-31T01:46:20.504 INFO:tasks.rados.rados.0.smithi045.stdout:192: finishing write tid 4 to smithi04569269-16 2021-07-31T01:46:20.504 INFO:tasks.rados.rados.0.smithi045.stdout:195: seq_num 81 ranges {1302528=565248,1867776=737280,2605056=65536} 2021-07-31T01:46:20.513 INFO:tasks.rados.rados.0.smithi045.stdout:195: writing smithi04569269-24 from 1302528 to 1867776 tid 1 2021-07-31T01:46:20.518 INFO:tasks.rados.rados.0.smithi045.stdout:195: writing smithi04569269-24 from 1867776 to 2605056 tid 2 2021-07-31T01:46:20.519 INFO:tasks.rados.rados.0.smithi045.stdout:195: writing smithi04569269-24 from 2605056 to 2670592 tid 3 2021-07-31T01:46:20.519 INFO:tasks.rados.rados.0.smithi045.stdout:193: done (2 left) 2021-07-31T01:46:20.519 INFO:tasks.rados.rados.0.smithi045.stdout:196: copy_from oid 35 from oid 49 current snap is 11 2021-07-31T01:46:20.519 INFO:tasks.rados.rados.0.smithi045.stdout:192: finishing write tid 5 to smithi04569269-16 2021-07-31T01:46:20.520 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 16 v 34 (ObjNum 80 snap 11 seq_num 80) dirty exists 2021-07-31T01:46:20.520 INFO:tasks.rados.rados.0.smithi045.stdout:192: left oid 16 (ObjNum 80 snap 11 seq_num 80) 2021-07-31T01:46:20.520 INFO:tasks.rados.rados.0.smithi045.stdout:195: finishing write tid 1 to smithi04569269-24 2021-07-31T01:46:20.520 INFO:tasks.rados.rados.0.smithi045.stdout:192: done (2 left) 2021-07-31T01:46:20.520 INFO:tasks.rados.rados.0.smithi045.stdout:197: delete oid 15 current snap is 11 2021-07-31T01:46:20.521 INFO:tasks.rados.rados.0.smithi045.stdout:197: done (2 left) 2021-07-31T01:46:20.521 INFO:tasks.rados.rados.0.smithi045.stdout:198: delete oid 40 current snap is 11 2021-07-31T01:46:20.522 INFO:tasks.rados.rados.0.smithi045.stdout:195: finishing write tid 2 to smithi04569269-24 2021-07-31T01:46:20.523 INFO:tasks.rados.rados.0.smithi045.stdout:195: finishing write tid 3 to smithi04569269-24 2021-07-31T01:46:20.523 INFO:tasks.rados.rados.0.smithi045.stdout:195: finishing write tid 4 to smithi04569269-24 2021-07-31T01:46:20.523 INFO:tasks.rados.rados.0.smithi045.stdout:198: done (2 left) 2021-07-31T01:46:20.524 INFO:tasks.rados.rados.0.smithi045.stdout:199: setattr oid 8 current snap is 11 2021-07-31T01:46:20.526 INFO:tasks.rados.rados.0.smithi045.stdout:195: finishing write tid 5 to smithi04569269-24 2021-07-31T01:46:20.526 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 24 v 17 (ObjNum 81 snap 11 seq_num 81) dirty exists 2021-07-31T01:46:20.526 INFO:tasks.rados.rados.0.smithi045.stdout:195: left oid 24 (ObjNum 81 snap 11 seq_num 81) 2021-07-31T01:46:20.527 INFO:tasks.rados.rados.0.smithi045.stdout:196: finishing copy_from to smithi04569269-35 2021-07-31T01:46:20.527 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 35 v 59 (ObjNum 55 snap 4 seq_num 55) dirty exists 2021-07-31T01:46:20.527 INFO:tasks.rados.rados.0.smithi045.stdout:196: finishing copy_from racing read to smithi04569269-35 2021-07-31T01:46:20.528 INFO:tasks.rados.rados.0.smithi045.stdout:195: done (2 left) 2021-07-31T01:46:20.528 INFO:tasks.rados.rados.0.smithi045.stdout:196: done (1 left) 2021-07-31T01:46:20.528 INFO:tasks.rados.rados.0.smithi045.stdout:200: snap_remove snap 10 2021-07-31T01:46:20.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:20 smithi154 conmon[34553]: cluster 2021-07-31T01:46:19.366493+0000 mgr.y (mgr.14146) 283 : cluster [DBG] pgmap v269: 25 pgs: 25 active+clean; 29 MiB data, 96 MiB used, 715 GiB / 715 GiB avail; 2.0 MiB/s rd, 1.5 MiB/s wr, 17 op/s 2021-07-31T01:46:20.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:20 smithi154 conmon[34553]: cluster 2021-07-31T01:46:19.495156+0000 mon.a (mon.0) 678 : cluster [DBG] osdmap e80: 8 total, 8 up, 8 in 2021-07-31T01:46:20.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:20 smithi045 conmon[33655]: cluster 2021-07-31T01:46:19.366493 2021-07-31T01:46:20.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:20 smithi045 conmon[33655]: +0000 mgr.y (mgr.14146) 283 : cluster [DBG] pgmap v269: 25 pgs: 25 active+clean; 29 MiB data, 96 MiB used, 715 GiB / 715 GiB avail; 2.0 MiB/s rd, 1.5 MiB/s wr, 17 op/s 2021-07-31T01:46:20.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:20 smithi045 conmon[33655]: cluster 2021-07-31T01:46:19.495156+0000 mon.a (mon.0) 678 : cluster [DBG] osdmap e80: 8 total, 8 up, 8 in 2021-07-31T01:46:20.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:20 smithi045 conmon[39718]: cluster 2021-07-31T01:46:19.366493 2021-07-31T01:46:20.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:20 smithi045 conmon[39718]: +0000 mgr.y (mgr.14146) 283 : cluster [DBG] pgmap v269: 25 pgs: 25 active+clean; 29 MiB data, 96 MiB used, 715 GiB / 715 GiB avail; 2.0 MiB/s rd, 1.5 MiB/s wr, 17 op/s 2021-07-31T01:46:20.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:20 smithi045 conmon[39718]: cluster 2021-07-31T01:46:19.495156 2021-07-31T01:46:20.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:20 smithi045 conmon[39718]: +0000 mon.a (mon.0) 678 : cluster [DBG] 2021-07-31T01:46:20.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:20 smithi045 conmon[39718]: osdmap e80: 8 total, 8 up, 8 in 2021-07-31T01:46:21.471 INFO:tasks.rados.rados.0.smithi045.stdout:200: done (1 left) 2021-07-31T01:46:21.472 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 37 current snap is 11 2021-07-31T01:46:21.472 INFO:tasks.rados.rados.0.smithi045.stdout:201: seq_num 84 ranges {606208=614400,1220608=434176,1654784=368640} 2021-07-31T01:46:21.479 INFO:tasks.rados.rados.0.smithi045.stdout:201: writing smithi04569269-37 from 606208 to 1220608 tid 1 2021-07-31T01:46:21.482 INFO:tasks.rados.rados.0.smithi045.stdout:201: writing smithi04569269-37 from 1220608 to 1654784 tid 2 2021-07-31T01:46:21.487 INFO:tasks.rados.rados.0.smithi045.stdout:201: writing smithi04569269-37 from 1654784 to 2023424 tid 3 2021-07-31T01:46:21.488 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 8 v 99 (ObjNum -1126011800 snap 21873 seq_num 1) dirty exists 2021-07-31T01:46:21.489 INFO:tasks.rados.rados.0.smithi045.stdout:201: finishing write tid 1 to smithi04569269-37 2021-07-31T01:46:21.489 INFO:tasks.rados.rados.0.smithi045.stdout:201: finishing write tid 2 to smithi04569269-37 2021-07-31T01:46:21.489 INFO:tasks.rados.rados.0.smithi045.stdout:199: done (1 left) 2021-07-31T01:46:21.490 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 2 current snap is 11 2021-07-31T01:46:21.490 INFO:tasks.rados.rados.0.smithi045.stdout:202: seq_num 85 ranges {1187840=630784,1818624=720896,2539520=81920} 2021-07-31T01:46:21.501 INFO:tasks.rados.rados.0.smithi045.stdout:202: writing smithi04569269-2 from 1187840 to 1818624 tid 1 2021-07-31T01:46:21.506 INFO:tasks.rados.rados.0.smithi045.stdout:202: writing smithi04569269-2 from 1818624 to 2539520 tid 2 2021-07-31T01:46:21.507 INFO:tasks.rados.rados.0.smithi045.stdout:202: writing smithi04569269-2 from 2539520 to 2621440 tid 3 2021-07-31T01:46:21.507 INFO:tasks.rados.rados.0.smithi045.stdout:201: finishing write tid 3 to smithi04569269-37 2021-07-31T01:46:21.507 INFO:tasks.rados.rados.0.smithi045.stdout:201: finishing write tid 4 to smithi04569269-37 2021-07-31T01:46:21.508 INFO:tasks.rados.rados.0.smithi045.stdout:201: finishing write tid 5 to smithi04569269-37 2021-07-31T01:46:21.508 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 37 v 39 (ObjNum 84 snap 11 seq_num 84) dirty exists 2021-07-31T01:46:21.508 INFO:tasks.rados.rados.0.smithi045.stdout:201: left oid 37 (ObjNum 84 snap 11 seq_num 84) 2021-07-31T01:46:21.508 INFO:tasks.rados.rados.0.smithi045.stdout:202: finishing write tid 1 to smithi04569269-2 2021-07-31T01:46:21.509 INFO:tasks.rados.rados.0.smithi045.stdout:201: done (1 left) 2021-07-31T01:46:21.509 INFO:tasks.rados.rados.0.smithi045.stdout:203: copy_from oid 19 from oid 46 current snap is 11 2021-07-31T01:46:21.509 INFO:tasks.rados.rados.0.smithi045.stdout:204: snap_create 2021-07-31T01:46:21.510 INFO:tasks.rados.rados.0.smithi045.stdout:202: finishing write tid 2 to smithi04569269-2 2021-07-31T01:46:21.510 INFO:tasks.rados.rados.0.smithi045.stdout:202: finishing write tid 3 to smithi04569269-2 2021-07-31T01:46:21.511 INFO:tasks.rados.rados.0.smithi045.stdout:202: finishing write tid 4 to smithi04569269-2 2021-07-31T01:46:21.511 INFO:tasks.rados.rados.0.smithi045.stdout:202: finishing write tid 5 to smithi04569269-2 2021-07-31T01:46:21.512 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 2 v 21 (ObjNum 85 snap 11 seq_num 85) dirty exists 2021-07-31T01:46:21.512 INFO:tasks.rados.rados.0.smithi045.stdout:202: left oid 2 (ObjNum 85 snap 11 seq_num 85) 2021-07-31T01:46:21.516 INFO:tasks.rados.rados.0.smithi045.stdout:203: finishing copy_from to smithi04569269-19 2021-07-31T01:46:21.516 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 19 v 2 (ObjNum 45 snap 0 seq_num 45) dirty exists 2021-07-31T01:46:21.516 INFO:tasks.rados.rados.0.smithi045.stdout:203: finishing copy_from racing read to smithi04569269-19 2021-07-31T01:46:21.753 INFO:journalctl@ceph.osd.5.smithi154.stdout:Jul 31 01:46:21 smithi154 conmon[42050]: debug 2021-07-31T01:46:21.478+0000 7fc9b5064700 -1 osd.5 pg_epoch: 82 pg[3.cs0( v 81'31 (0'0,81'31] local-lis/les=56/57 n=7 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [5,3,6]p5(0) r=0 lpr=56 crt=81'31 lcod 80'29 mlcod 80'29 active+clean+snaptrim trimq=[15~1] ps=[12~1,14~1]] removing snap head 2021-07-31T01:46:21.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:21 smithi154 conmon[34553]: cluster 2021-07-31T01:46: 2021-07-31T01:46:21.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:21 smithi154 conmon[34553]: 20.500360+0000 mon.a (mon.0) 679 : cluster [DBG] osdmap e81: 8 total, 8 up, 8 in 2021-07-31T01:46:21.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:21 smithi154 conmon[34553]: cluster 2021-07-31T01:46 2021-07-31T01:46:21.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:21 smithi154 conmon[34553]: :21.469457+0000 mon.a (mon.0) 680 : cluster [DBG] osdmap e82: 8 total, 8 up, 8 in 2021-07-31T01:46:21.755 INFO:journalctl@ceph.osd.7.smithi154.stdout:Jul 31 01:46:21 smithi154 conmon[49149]: debug 2021-07-31T01:46:21.473+0000 7f8a711fb700 -1 osd.7 pg_epoch: 82 pg[3.es0( v 81'43 (0'0,81'43] local-lis/les=56/57 n=4 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [7,4,1]p7(0) r=0 lpr=56 crt=81'43 lcod 81'41 mlcod 81'41 active+clean+snaptrim trimq=[15~1] ps=[12~1,14~1]] removing snap head 2021-07-31T01:46:21.755 INFO:journalctl@ceph.osd.7.smithi154.stdout:Jul 31 01:46:21 smithi154 conmon[49149]: debug 2021-07-31T01:46:21.473+0000 7f8a709fa700 -1 osd.7 pg_epoch: 82 pg[3.fs0( v 81'26 (0'0,81'26] local-lis/les=56/57 n=4 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [7,4,0]p7(0) r=0 lpr=56 crt=81'26 lcod 81'24 mlcod 81'24 active+clean+snaptrim trimq=[15~1] ps=[12~1,14~1]] removing snap head 2021-07-31T01:46:21.755 INFO:journalctl@ceph.osd.7.smithi154.stdout:Jul 31 01:46:21 smithi154 conmon[49149]: debug 2021-07-31T01:46:21.474+0000 7f8a709fa700 -1 osd.7 pg_epoch: 82 pg[3.fs0( v 82'28 (0'0,82'28] local-lis/les=56/57 n=2 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [7,4,0]p7(0) r=0 lpr=56 luod=81'26 crt=81'26 lcod 81'24 mlcod 81'24 active+clean+snaptrim trimq=[15~1] ps=[12~1,14~1]] removing snap head 2021-07-31T01:46:21.783 INFO:journalctl@ceph.osd.1.smithi045.stdout:Jul 31 01:46:21 smithi045 conmon[46771]: debug 2021-07-31T01:46:21.472+0000 7f41979aa700 -1 osd.1 pg_epoch: 82 pg[3.0s0( v 81'33 (0'0,81'33] local-lis/les=56/57 n=5 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [1,2,6]p1(0) r=0 lpr=56 crt=81'33 lcod 81'31 mlcod 81'31 active+clean+snaptrim trimq=[15~1] ps=[12~1,14~1]] removing snap head 2021-07-31T01:46:21.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:21 smithi045 conmon[39718]: cluster 2021-07-31T01:46:20.500360+0000 mon.a 2021-07-31T01:46:21.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:21 smithi045 conmon[39718]: (mon.0) 679 : cluster [DBG] osdmap e81: 8 total, 8 up, 8 in 2021-07-31T01:46:21.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:21 smithi045 conmon[39718]: cluster 2021-07-31T01 2021-07-31T01:46:21.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:21 smithi045 conmon[39718]: :46:21.469457+0000 mon.a (mon.0) 680 : cluster [DBG] osdmap e82: 8 total, 8 up, 8 in 2021-07-31T01:46:21.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:21 smithi045 conmon[33655]: cluster 2021-07-31T01:46:21.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:21 smithi045 conmon[33655]: 2021-07-31T01:46:21.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:21 smithi045 conmon[33655]: 2021 2021-07-31T01:46:21.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:21 smithi045 conmon[33655]: - 2021-07-31T01:46:21.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:21 smithi045 conmon[33655]: 07 2021-07-31T01:46:21.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:21 smithi045 conmon[33655]: - 2021-07-31T01:46:21.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:21 smithi045 conmon[33655]: 31 2021-07-31T01:46:21.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:21 smithi045 conmon[33655]: T 2021-07-31T01:46:21.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:21 smithi045 conmon[33655]: 01 2021-07-31T01:46:21.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:21 smithi045 conmon[33655]: : 2021-07-31T01:46:21.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:21 smithi045 conmon[33655]: 46 2021-07-31T01:46:21.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:21 smithi045 conmon[33655]: : 2021-07-31T01:46:21.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:21 smithi045 conmon[33655]: 20 2021-07-31T01:46:21.789 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:21 smithi045 conmon[33655]: . 2021-07-31T01:46:21.789 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:21 smithi045 conmon[33655]: 500360 2021-07-31T01:46:21.790 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:21 smithi045 conmon[33655]: +0000 2021-07-31T01:46:21.790 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:21 smithi045 conmon[33655]: 2021-07-31T01:46:21.790 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:21 smithi045 conmon[33655]: mon.a 2021-07-31T01:46:21.790 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:21 smithi045 conmon[33655]: ( 2021-07-31T01:46:21.791 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:21 smithi045 conmon[33655]: mon 2021-07-31T01:46:21.791 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:21 smithi045 conmon[33655]: . 2021-07-31T01:46:21.791 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:21 smithi045 conmon[33655]: 0 2021-07-31T01:46:21.792 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:21 smithi045 conmon[33655]: ) 2021-07-31T01:46:21.792 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:21 smithi045 conmon[33655]: 679 2021-07-31T01:46:21.792 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:21 smithi045 conmon[33655]: : 2021-07-31T01:46:21.792 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:21 smithi045 conmon[33655]: cluster 2021-07-31T01:46:21.793 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:21 smithi045 conmon[33655]: [DBG] osdmap e81: 8 total, 8 up, 8 in 2021-07-31T01:46:21.793 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:21 smithi045 conmon[33655]: cluster 2021-07-31T01:46:21.793 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:21 smithi045 conmon[33655]: 2021-07-31 2021-07-31T01:46:21.793 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:21 smithi045 conmon[33655]: T01:46:21.469457+0000 mon.a (mon 2021-07-31T01:46:21.793 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:21 smithi045 conmon[33655]: .0) 680 : cluster [DBG] osdmap e82: 8 total, 8 up, 8 in 2021-07-31T01:46:22.512 INFO:tasks.rados.rados.0.smithi045.stdout:202: done (2 left) 2021-07-31T01:46:22.512 INFO:tasks.rados.rados.0.smithi045.stdout:203: done (1 left) 2021-07-31T01:46:22.512 INFO:tasks.rados.rados.0.smithi045.stdout:204: done (0 left) 2021-07-31T01:46:22.512 INFO:tasks.rados.rados.0.smithi045.stdout:205: snap_create 2021-07-31T01:46:22.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:22 smithi045 conmon[33655]: cluster 2021-07-31T01:46:21.366858+0000 mgr.y (mgr 2021-07-31T01:46:22.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:22 smithi045 conmon[33655]: .14146) 284 : cluster [DBG] pgmap v272: 25 pgs: 25 active+clean; 29 MiB data, 96 MiB used, 715 GiB / 715 GiB avail; 2.0 MiB/s rd, 1.5 MiB/s wr, 16 op/s 2021-07-31T01:46:22.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:22 smithi045 conmon[39718]: cluster 2021-07-31T01:46:21 2021-07-31T01:46:22.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:22 smithi045 conmon[39718]: .366858+0000 mgr.y (mgr.14146) 284 : cluster [DBG] pgmap v272: 25 pgs: 25 active+clean; 29 MiB data, 96 MiB used, 715 GiB / 715 GiB avail; 2.0 MiB/s rd, 1.5 MiB/s wr, 16 op/s 2021-07-31T01:46:23.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:22 smithi154 conmon[34553]: cluster 2021-07-31T01:46:21.366858+0000 mgr.y (mgr.14146) 284 : cluster [DBG] pgmap v272: 25 pgs: 25 active+clean; 29 MiB data, 96 MiB used, 715 GiB / 715 GiB avail; 2.0 MiB/s rd, 1.5 MiB/s wr, 16 op/s 2021-07-31T01:46:23.249 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T01:46:23.518 INFO:tasks.rados.rados.0.smithi045.stdout:205: done (0 left) 2021-07-31T01:46:23.518 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 17 current snap is 13 2021-07-31T01:46:23.518 INFO:tasks.rados.rados.0.smithi045.stdout:206: seq_num 86 ranges {0=647168,647168=606208,1253376=196608} 2021-07-31T01:46:23.521 INFO:tasks.rados.rados.0.smithi045.stdout:206: writing smithi04569269-17 from 0 to 647168 tid 1 2021-07-31T01:46:23.524 INFO:tasks.rados.rados.0.smithi045.stdout:206: writing smithi04569269-17 from 647168 to 1253376 tid 2 2021-07-31T01:46:23.526 INFO:tasks.rados.rados.0.smithi045.stdout:206: writing smithi04569269-17 from 1253376 to 1449984 tid 3 2021-07-31T01:46:23.526 INFO:tasks.rados.rados.0.smithi045.stdout:207: rollback oid 8 current snap is 13 2021-07-31T01:46:23.526 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 8 to 12 2021-07-31T01:46:23.526 INFO:tasks.rados.rados.0.smithi045.stdout:208: rmattr oid 26 current snap is 13 2021-07-31T01:46:23.527 INFO:tasks.rados.rados.0.smithi045.stdout:208: done (2 left) 2021-07-31T01:46:23.527 INFO:tasks.rados.rados.0.smithi045.stdout:209: delete oid 2 current snap is 13 2021-07-31T01:46:23.529 INFO:tasks.rados.rados.0.smithi045.stdout:206: finishing write tid 1 to smithi04569269-17 2021-07-31T01:46:23.530 INFO:tasks.rados.rados.0.smithi045.stdout:206: finishing write tid 2 to smithi04569269-17 2021-07-31T01:46:23.531 INFO:tasks.rados.rados.0.smithi045.stdout:206: finishing write tid 3 to smithi04569269-17 2021-07-31T01:46:23.532 INFO:tasks.rados.rados.0.smithi045.stdout:207: finishing rollback tid 0 to smithi04569269-8 2021-07-31T01:46:23.532 INFO:tasks.rados.rados.0.smithi045.stdout:206: finishing write tid 4 to smithi04569269-17 2021-07-31T01:46:23.532 INFO:tasks.rados.rados.0.smithi045.stdout:209: done (2 left) 2021-07-31T01:46:23.532 INFO:tasks.rados.rados.0.smithi045.stdout:210: rmattr oid 24 current snap is 13 2021-07-31T01:46:23.532 INFO:tasks.rados.rados.0.smithi045.stdout:210: done (2 left) 2021-07-31T01:46:23.533 INFO:tasks.rados.rados.0.smithi045.stdout:211: snap_create 2021-07-31T01:46:23.533 INFO:tasks.rados.rados.0.smithi045.stdout:207: finishing rollback tid 1 to smithi04569269-8 2021-07-31T01:46:23.534 INFO:tasks.rados.rados.0.smithi045.stdout:207: finishing rollback tid 2 to smithi04569269-8 2021-07-31T01:46:23.534 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 8 v 109 (ObjNum -1105723456 snap 21873 seq_num 3189313248) dirty exists 2021-07-31T01:46:23.534 INFO:tasks.rados.rados.0.smithi045.stdout:206: finishing write tid 5 to smithi04569269-17 2021-07-31T01:46:23.534 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 17 v 53 (ObjNum 86 snap 13 seq_num 86) dirty exists 2021-07-31T01:46:23.535 INFO:tasks.rados.rados.0.smithi045.stdout:206: left oid 17 (ObjNum 86 snap 13 seq_num 86) 2021-07-31T01:46:23.549 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T01:46:23.549 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 running (93s) 76s ago 2m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T01:46:23.550 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 running (105s) 78s ago 105s 37.1M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T01:46:23.550 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 running (5m) 78s ago 5m 377M - 16.2.5 a29040f3000e f9af406258e1 2021-07-31T01:46:23.550 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 running (7m) 76s ago 7m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T01:46:23.551 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 running (7m) 76s ago 7m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T01:46:23.551 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 running (5m) 78s ago 5m 92.9M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T01:46:23.551 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 running (5m) 76s ago 5m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T01:46:23.551 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 running (2m) 76s ago 2m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T01:46:23.551 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 running (2m) 78s ago 2m 8144k - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T01:46:23.552 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 running (5m) 76s ago 5m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T01:46:23.552 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 running (4m) 76s ago 4m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T01:46:23.552 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 running (4m) 76s ago 4m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T01:46:23.553 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 running (4m) 76s ago 4m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T01:46:23.554 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 running (3m) 78s ago 3m 43.3M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T01:46:23.554 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 running (3m) 78s ago 3m 41.6M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T01:46:23.554 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 running (3m) 78s ago 3m 41.2M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T01:46:23.555 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 running (2m) 78s ago 2m 42.6M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T01:46:23.555 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 running (88s) 78s ago 2m 28.9M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T01:46:23.610 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:23 smithi045 conmon[39718]: cluster 2021-07-31T01:46:23.611 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:23 smithi045 conmon[39718]: 2021-07-31T01: 2021-07-31T01:46:23.611 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:23 smithi045 conmon[39718]: 46:22.511612+0000 mon.a (mon.0) 681 : cluster [DBG] osdmap e83: 8 total, 8 up, 8 in 2021-07-31T01:46:23.611 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:46:23 smithi045 conmon[61210]: level=error ts=2021-07-31T01:46:23.505Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:46:23.612 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:46:23 smithi045 conmon[61210]: level=error ts=2021-07-31T01:46:23.505Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:46:23.612 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:46:23 smithi045 conmon[61210]: level=error ts=2021-07-31T01:46:23.505Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T01:46:23.613 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:23 smithi045 conmon[33655]: cluster 2021-07-31T01:46:23.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:23 smithi045 conmon[33655]: 2021-07-31T01:46:23.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:23 smithi045 conmon[33655]: 2021 2021-07-31T01:46:23.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:23 smithi045 conmon[33655]: - 2021-07-31T01:46:23.622 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:23 smithi045 conmon[33655]: 07 2021-07-31T01:46:23.622 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:23 smithi045 conmon[33655]: - 2021-07-31T01:46:23.622 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:23 smithi045 conmon[33655]: 31 2021-07-31T01:46:23.622 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:23 smithi045 conmon[33655]: T 2021-07-31T01:46:23.623 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:23 smithi045 conmon[33655]: 01 2021-07-31T01:46:23.623 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:23 smithi045 conmon[33655]: : 2021-07-31T01:46:23.623 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:23 smithi045 conmon[33655]: 46:22.511612+0000 mon.a (mon.0) 681 : cluster [DBG] osdmap e83: 8 total, 8 up, 8 in 2021-07-31T01:46:23.885 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T01:46:23.886 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T01:46:23.886 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T01:46:23.886 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T01:46:23.887 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T01:46:23.887 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 2 2021-07-31T01:46:23.887 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T01:46:23.887 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T01:46:23.888 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T01:46:23.888 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T01:46:23.888 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T01:46:23.888 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T01:46:23.889 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 13 2021-07-31T01:46:23.889 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T01:46:23.889 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T01:46:24.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:23 smithi154 conmon[34553]: cluster 2021-07-31T01:46:22.511612+0000 mon.a (mon.0) 681 : cluster [DBG] osdmap e83: 8 total, 8 up, 8 in 2021-07-31T01:46:24.522 INFO:tasks.rados.rados.0.smithi045.stdout:206: done (2 left) 2021-07-31T01:46:24.523 INFO:tasks.rados.rados.0.smithi045.stdout:207: done (1 left) 2021-07-31T01:46:24.523 INFO:tasks.rados.rados.0.smithi045.stdout:211: done (0 left) 2021-07-31T01:46:24.523 INFO:tasks.rados.rados.0.smithi045.stdout:212: copy_from oid 48 from oid 47 current snap is 14 2021-07-31T01:46:24.523 INFO:tasks.rados.rados.0.smithi045.stdout:213: read oid 34 snap -1 2021-07-31T01:46:24.524 INFO:tasks.rados.rados.0.smithi045.stdout:213: expect deleted 2021-07-31T01:46:24.524 INFO:tasks.rados.rados.0.smithi045.stdout:214: snap_create 2021-07-31T01:46:24.532 INFO:tasks.rados.rados.0.smithi045.stdout:212: finishing copy_from to smithi04569269-48 2021-07-31T01:46:24.532 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 48 v 4 (ObjNum 46 snap 0 seq_num 46) dirty exists 2021-07-31T01:46:24.532 INFO:tasks.rados.rados.0.smithi045.stdout:212: finishing copy_from racing read to smithi04569269-48 2021-07-31T01:46:24.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:24 smithi045 conmon[39718]: audit 2021-07-31T01:46:23.236293+0000 mgr.y (mgr.14146) 285 : audit [DBG] from='client.14640 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:46:24.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:24 smithi045 conmon[39718]: cluster 2021-07-31T01:46:23.367222+0000 mgr.y (mgr.14146) 286 : cluster [DBG] pgmap v275: 25 pgs: 25 active+clean; 29 MiB data, 104 MiB used, 715 GiB / 715 GiB avail; 457 KiB/s rd, 796 KiB/s wr, 7 op/s 2021-07-31T01:46:24.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:24 smithi045 conmon[39718]: cluster 2021-07-31T01:46:23.516739+0000 mon.a (mon.0) 682 : cluster [DBG] osdmap e84: 8 total, 8 up, 8 in 2021-07-31T01:46:24.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:24 smithi045 conmon[39718]: audit 2021-07-31T01:46:23.541027+0000 mgr.y (mgr.14146) 287 : audit [DBG] from='client.24553 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:46:24.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:24 smithi045 conmon[39718]: audit 2021-07-31T01:46:23.885311+0000 mon.c (mon.1) 19 : audit [DBG] from='client.? 172.21.15.45:0/757984581' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T01:46:24.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:24 smithi045 conmon[33655]: audit 2021-07-31T01:46:23.236293+0000 mgr.y (mgr.14146) 285 2021-07-31T01:46:24.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:24 smithi045 conmon[33655]: : audit [DBG] from='client.14640 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:46:24.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:24 smithi045 conmon[33655]: cluster 2021-07-31T01:46:23.367222+0000 mgr.y (mgr.14146) 286 : cluster [DBG] pgmap v275: 25 pgs: 25 active+clean; 29 MiB data, 104 MiB used, 715 GiB / 715 GiB avail; 457 KiB/s rd, 796 KiB/s wr, 7 op/s 2021-07-31T01:46:24.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:24 smithi045 conmon[33655]: cluster 2021-07-31T01:46:23.516739+0000 mon.a (mon.0) 682 : cluster [DBG] osdmap e84: 8 total, 8 up, 8 in 2021-07-31T01:46:24.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:24 smithi045 conmon[33655]: audit 2021-07-31T01:46:23.541027+0000 mgr.y (mgr.14146) 287 : audit [DBG] from='client.24553 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:46:24.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:24 smithi045 conmon[33655]: audit 2021-07-31T01:46:24.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:24 smithi045 conmon[33655]: 2021-07-31T01:46:23.885311+0000 mon.c (mon.1) 19 : audit [DBG] from='client.? 172.21.15.45:0/757984581' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T01:46:25.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:24 smithi154 conmon[34553]: audit 2021-07-31T01:46:23.236293+0000 mgr.y (mgr.14146) 285 : audit [DBG] from='client.14640 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:46:25.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:24 smithi154 conmon[34553]: cluster 2021-07-31T01:46:23.367222+0000 mgr.y (mgr.14146) 286 : cluster [DBG] pgmap v275: 25 pgs: 25 active+clean; 29 MiB data, 104 MiB used, 715 GiB / 715 GiB avail; 457 KiB/s rd, 796 KiB/s wr, 7 op/s 2021-07-31T01:46:25.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:24 smithi154 conmon[34553]: cluster 2021-07-31T01:46:23.516739+0000 mon.a (mon.0) 682 : cluster [DBG] osdmap e84: 8 total, 8 up, 8 in 2021-07-31T01:46:25.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:24 smithi154 conmon[34553]: 2021-07-31T01:46:25.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:24 smithi154 conmon[34553]: audit 2021-07-31T01:46:23 2021-07-31T01:46:25.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:24 smithi154 conmon[34553]: .541027+0000 mgr.y (mgr.14146) 287 : audit [DBG] from='client.24553 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:46:25.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:24 smithi154 conmon[34553]: audit 2021-07-31T01:46:25.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:24 smithi154 conmon[34553]: 2021-07-31T01:46:23.885311 2021-07-31T01:46:25.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:24 smithi154 conmon[34553]: +0000 mon.c (mon.1) 19 : audit [DBG] 2021-07-31T01:46:25.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:24 smithi154 conmon[34553]: from='client.? 172.21.15.45:0/757984581' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T01:46:25.528 INFO:tasks.rados.rados.0.smithi045.stdout:212: done (2 left) 2021-07-31T01:46:25.528 INFO:tasks.rados.rados.0.smithi045.stdout:213: done (1 left) 2021-07-31T01:46:25.528 INFO:tasks.rados.rados.0.smithi045.stdout:214: done (0 left) 2021-07-31T01:46:25.529 INFO:tasks.rados.rados.0.smithi045.stdout:215: read oid 47 snap -1 2021-07-31T01:46:25.529 INFO:tasks.rados.rados.0.smithi045.stdout:215: expect (ObjNum 46 snap 0 seq_num 46) 2021-07-31T01:46:25.529 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 45 current snap is 15 2021-07-31T01:46:25.529 INFO:tasks.rados.rados.0.smithi045.stdout:216: seq_num 88 ranges {557056=679936,1236992=778240,2015232=24576} 2021-07-31T01:46:25.534 INFO:tasks.rados.rados.0.smithi045.stdout:216: writing smithi04569269-45 from 557056 to 1236992 tid 1 2021-07-31T01:46:25.539 INFO:tasks.rados.rados.0.smithi045.stdout:216: writing smithi04569269-45 from 1236992 to 2015232 tid 2 2021-07-31T01:46:25.539 INFO:tasks.rados.rados.0.smithi045.stdout:216: writing smithi04569269-45 from 2015232 to 2039808 tid 3 2021-07-31T01:46:25.540 INFO:tasks.rados.rados.0.smithi045.stdout:217: read oid 49 snap 13 2021-07-31T01:46:25.540 INFO:tasks.rados.rados.0.smithi045.stdout:217: expect (ObjNum 55 snap 4 seq_num 55) 2021-07-31T01:46:25.540 INFO:tasks.rados.rados.0.smithi045.stdout:218: read oid 28 snap -1 2021-07-31T01:46:25.540 INFO:tasks.rados.rados.0.smithi045.stdout:218: expect deleted 2021-07-31T01:46:25.540 INFO:tasks.rados.rados.0.smithi045.stdout:219: read oid 46 snap 13 2021-07-31T01:46:25.541 INFO:tasks.rados.rados.0.smithi045.stdout:219: expect (ObjNum 45 snap 0 seq_num 45) 2021-07-31T01:46:25.541 INFO:tasks.rados.rados.0.smithi045.stdout:220: snap_remove snap 14 2021-07-31T01:46:25.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:25 smithi045 conmon[33655]: cluster 2021-07-31T01:46:24.521839+0000 mon.a (mon.0) 683 : cluster 2021-07-31T01:46:25.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:25 smithi045 conmon[33655]: [DBG] osdmap e85: 8 total, 8 up, 8 in 2021-07-31T01:46:25.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:25 smithi045 conmon[39718]: cluster 2021-07-31T01:46:25.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:25 smithi045 conmon[39718]: 2021-07-31T01:46:24. 2021-07-31T01:46:25.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:25 smithi045 conmon[39718]: 521839+0000 mon.a (mon.0) 683 : cluster [DBG] osdmap e85: 8 total, 8 up, 8 in 2021-07-31T01:46:26.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:25 smithi154 conmon[34553]: cluster 2021-07-31T01:46:24.521839+0000 mon.a (mon.0) 683 : cluster [DBG] osdmap e85: 8 total, 8 up, 8 in 2021-07-31T01:46:26.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:25 smithi154 conmon[34553]: 2021-07-31T01:46:26.472 INFO:tasks.rados.rados.0.smithi045.stdout:220: done (5 left) 2021-07-31T01:46:26.472 INFO:tasks.rados.rados.0.smithi045.stdout:221: rollback oid 13 current snap is 15 2021-07-31T01:46:26.472 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 13 to 13 2021-07-31T01:46:26.473 INFO:tasks.rados.rados.0.smithi045.stdout:215: expect (ObjNum 46 snap 0 seq_num 46) 2021-07-31T01:46:26.494 INFO:tasks.rados.rados.0.smithi045.stdout:216: finishing write tid 1 to smithi04569269-45 2021-07-31T01:46:26.495 INFO:tasks.rados.rados.0.smithi045.stdout:216: finishing write tid 2 to smithi04569269-45 2021-07-31T01:46:26.495 INFO:tasks.rados.rados.0.smithi045.stdout:216: finishing write tid 3 to smithi04569269-45 2021-07-31T01:46:26.496 INFO:tasks.rados.rados.0.smithi045.stdout:216: finishing write tid 4 to smithi04569269-45 2021-07-31T01:46:26.496 INFO:tasks.rados.rados.0.smithi045.stdout:215: done (5 left) 2021-07-31T01:46:26.496 INFO:tasks.rados.rados.0.smithi045.stdout:222: read oid 12 snap -1 2021-07-31T01:46:26.496 INFO:tasks.rados.rados.0.smithi045.stdout:222: expect deleted 2021-07-31T01:46:26.496 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 24 current snap is 15 2021-07-31T01:46:26.497 INFO:tasks.rados.rados.0.smithi045.stdout:223: seq_num 89 ranges {2670592=696320,3366912=655360,4022272=147456} 2021-07-31T01:46:26.513 INFO:tasks.rados.rados.0.smithi045.stdout:223: writing smithi04569269-24 from 2670592 to 3366912 tid 1 2021-07-31T01:46:26.516 INFO:tasks.rados.rados.0.smithi045.stdout:223: writing smithi04569269-24 from 3366912 to 4022272 tid 2 2021-07-31T01:46:26.517 INFO:tasks.rados.rados.0.smithi045.stdout:223: writing smithi04569269-24 from 4022272 to 4169728 tid 3 2021-07-31T01:46:26.517 INFO:tasks.rados.rados.0.smithi045.stdout:224: setattr oid 6 current snap is 15 2021-07-31T01:46:26.518 INFO:tasks.rados.rados.0.smithi045.stdout:219: expect (ObjNum 45 snap 0 seq_num 45) 2021-07-31T01:46:26.539 INFO:tasks.rados.rados.0.smithi045.stdout:216: finishing write tid 5 to smithi04569269-45 2021-07-31T01:46:26.539 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 45 v 58 (ObjNum 88 snap 15 seq_num 88) dirty exists 2021-07-31T01:46:26.540 INFO:tasks.rados.rados.0.smithi045.stdout:216: left oid 45 (ObjNum 88 snap 15 seq_num 88) 2021-07-31T01:46:26.540 INFO:tasks.rados.rados.0.smithi045.stdout:217: expect (ObjNum 55 snap 4 seq_num 55) 2021-07-31T01:46:26.598 INFO:tasks.rados.rados.0.smithi045.stdout:221: finishing rollback tid 0 to smithi04569269-13 2021-07-31T01:46:26.598 INFO:tasks.rados.rados.0.smithi045.stdout:221: finishing rollback tid 1 to smithi04569269-13 2021-07-31T01:46:26.599 INFO:tasks.rados.rados.0.smithi045.stdout:221: finishing rollback tid 2 to smithi04569269-13 2021-07-31T01:46:26.599 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 13 v 51 (ObjNum 75 snap 11 seq_num 75) dirty exists 2021-07-31T01:46:26.599 INFO:tasks.rados.rados.0.smithi045.stdout:223: finishing write tid 1 to smithi04569269-24 2021-07-31T01:46:26.599 INFO:tasks.rados.rados.0.smithi045.stdout:223: finishing write tid 2 to smithi04569269-24 2021-07-31T01:46:26.599 INFO:tasks.rados.rados.0.smithi045.stdout:223: finishing write tid 3 to smithi04569269-24 2021-07-31T01:46:26.600 INFO:tasks.rados.rados.0.smithi045.stdout:223: finishing write tid 4 to smithi04569269-24 2021-07-31T01:46:26.600 INFO:tasks.rados.rados.0.smithi045.stdout:223: finishing write tid 5 to smithi04569269-24 2021-07-31T01:46:26.600 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 24 v 24 (ObjNum 89 snap 15 seq_num 89) dirty exists 2021-07-31T01:46:26.600 INFO:tasks.rados.rados.0.smithi045.stdout:223: left oid 24 (ObjNum 89 snap 15 seq_num 89) 2021-07-31T01:46:26.601 INFO:tasks.rados.rados.0.smithi045.stdout:216: done (7 left) 2021-07-31T01:46:26.601 INFO:tasks.rados.rados.0.smithi045.stdout:217: done (6 left) 2021-07-31T01:46:26.601 INFO:tasks.rados.rados.0.smithi045.stdout:218: done (5 left) 2021-07-31T01:46:26.601 INFO:tasks.rados.rados.0.smithi045.stdout:219: done (4 left) 2021-07-31T01:46:26.602 INFO:tasks.rados.rados.0.smithi045.stdout:221: done (3 left) 2021-07-31T01:46:26.602 INFO:tasks.rados.rados.0.smithi045.stdout:222: done (2 left) 2021-07-31T01:46:26.602 INFO:tasks.rados.rados.0.smithi045.stdout:223: done (1 left) 2021-07-31T01:46:26.602 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 19 current snap is 15 2021-07-31T01:46:26.602 INFO:tasks.rados.rados.0.smithi045.stdout:225: seq_num 90 ranges {761856=712704,1474560=540672,2015232=262144} 2021-07-31T01:46:26.605 INFO:tasks.rados.rados.0.smithi045.stdout:225: writing smithi04569269-19 from 761856 to 1474560 tid 1 2021-07-31T01:46:26.608 INFO:tasks.rados.rados.0.smithi045.stdout:225: writing smithi04569269-19 from 1474560 to 2015232 tid 2 2021-07-31T01:46:26.610 INFO:tasks.rados.rados.0.smithi045.stdout:225: writing smithi04569269-19 from 2015232 to 2277376 tid 3 2021-07-31T01:46:26.610 INFO:tasks.rados.rados.0.smithi045.stdout:226: read oid 1 snap 11 2021-07-31T01:46:26.611 INFO:tasks.rados.rados.0.smithi045.stdout:226: expect (ObjNum 71 snap 10 seq_num 71) 2021-07-31T01:46:26.611 INFO:tasks.rados.rados.0.smithi045.stdout:227: snap_remove snap 12 2021-07-31T01:46:27.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:26 smithi154 conmon[34553]: cluster 2021-07-31T01:46:25.367747+0000 mgr.y (mgr.14146) 288 : cluster [DBG] pgmap v278: 25 pgs: 25 active+clean; 36 MiB data, 116 MiB used, 715 GiB / 715 GiB avail; 458 KiB/s rd, 3.2 MiB/s wr, 21 op/s 2021-07-31T01:46:27.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:26 smithi154 conmon[34553]: cluster 2021-07-31T01:46:25 2021-07-31T01:46:27.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:26 smithi154 conmon[34553]: .527204+0000 mon.a (mon.0) 684 : cluster [DBG] osdmap e86: 8 total, 8 up, 8 in 2021-07-31T01:46:27.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:26 smithi154 conmon[34553]: cluster 2021-07-31T01:46:26.471330+0000 mon.a (mon.0) 685 : cluster [DBG] osdmap e87: 8 total, 8 up, 8 in 2021-07-31T01:46:27.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:26 smithi045 conmon[33655]: cluster 2021-07-31T01:46:25.367747+0000 mgr.y (mgr.14146) 2021-07-31T01:46:27.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:26 smithi045 conmon[33655]: 288 : cluster [DBG] pgmap v278: 25 pgs: 25 active+clean; 36 MiB data, 116 MiB used, 715 GiB / 715 GiB avail; 458 KiB/s rd, 3.2 MiB/s wr, 21 op/s 2021-07-31T01:46:27.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:26 smithi045 conmon[33655]: cluster 2021-07-31T01:46:25.527204+0000 mon.a (mon.0) 684 : cluster [DBG] osdmap e86: 8 total, 8 up, 8 in 2021-07-31T01:46:27.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:26 smithi045 conmon[33655]: cluster 2021-07-31T01:46:26.471330 2021-07-31T01:46:27.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:26 smithi045 conmon[33655]: +0000 mon.a (mon.0) 685 : cluster [DBG] osdmap e87: 8 total, 8 up, 8 in 2021-07-31T01:46:27.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:26 smithi045 conmon[33655]: 2021-07-31T01:46:27.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:26 smithi045 conmon[39718]: cluster 2021-07-31T01:46:25.367747+0000 mgr.y (mgr 2021-07-31T01:46:27.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:26 smithi045 conmon[39718]: .14146) 288 : cluster [DBG] pgmap v278: 25 pgs: 25 active+clean; 36 MiB data, 116 MiB used, 715 GiB / 715 GiB avail; 458 KiB/s rd, 3.2 MiB/s wr, 21 op/s 2021-07-31T01:46:27.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:26 smithi045 conmon[39718]: cluster 2021-07-31T01:46:27.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:26 smithi045 conmon[39718]: 2021-07-31T01:46:25.527204+0000 mon.a (mon.0) 684 : cluster [DBG] osdmap e86: 8 total, 8 up, 8 in 2021-07-31T01:46:27.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:26 smithi045 conmon[39718]: cluster 2021 2021-07-31T01:46:27.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:26 smithi045 conmon[39718]: -07-31T01:46:26.471330 2021-07-31T01:46:27.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:26 smithi045 conmon[39718]: +0000 mon.a (mon.0) 685 : cluster [DBG] 2021-07-31T01:46:27.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:26 smithi045 conmon[39718]: osdmap e87: 8 total, 8 up, 8 in 2021-07-31T01:46:27.535 INFO:tasks.rados.rados.0.smithi045.stdout:227: done (3 left) 2021-07-31T01:46:27.535 INFO:tasks.rados.rados.0.smithi045.stdout:228: copy_from oid 49 from oid 25 current snap is 15 2021-07-31T01:46:27.536 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 6 v 52 (ObjNum -1105991824 snap 21873 seq_num 3188950640) dirty exists 2021-07-31T01:46:27.536 INFO:tasks.rados.rados.0.smithi045.stdout:225: finishing write tid 1 to smithi04569269-19 2021-07-31T01:46:27.536 INFO:tasks.rados.rados.0.smithi045.stdout:225: finishing write tid 2 to smithi04569269-19 2021-07-31T01:46:27.536 INFO:tasks.rados.rados.0.smithi045.stdout:225: finishing write tid 3 to smithi04569269-19 2021-07-31T01:46:27.537 INFO:tasks.rados.rados.0.smithi045.stdout:225: finishing write tid 4 to smithi04569269-19 2021-07-31T01:46:27.537 INFO:tasks.rados.rados.0.smithi045.stdout:226: expect (ObjNum 71 snap 10 seq_num 71) 2021-07-31T01:46:27.570 INFO:tasks.rados.rados.0.smithi045.stdout:225: finishing write tid 5 to smithi04569269-19 2021-07-31T01:46:27.570 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 19 v 44 (ObjNum 90 snap 15 seq_num 90) dirty exists 2021-07-31T01:46:27.571 INFO:tasks.rados.rados.0.smithi045.stdout:225: left oid 19 (ObjNum 90 snap 15 seq_num 90) 2021-07-31T01:46:27.571 INFO:tasks.rados.rados.0.smithi045.stdout:224: done (3 left) 2021-07-31T01:46:27.571 INFO:tasks.rados.rados.0.smithi045.stdout:225: done (2 left) 2021-07-31T01:46:27.571 INFO:tasks.rados.rados.0.smithi045.stdout:226: done (1 left) 2021-07-31T01:46:27.572 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 9 current snap is 15 2021-07-31T01:46:27.572 INFO:tasks.rados.rados.0.smithi045.stdout:228: finishing copy_from to smithi04569269-49 2021-07-31T01:46:27.572 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 49 v 8 (ObjNum 36 snap 0 seq_num 36) dirty exists 2021-07-31T01:46:27.572 INFO:tasks.rados.rados.0.smithi045.stdout:228: finishing copy_from racing read to smithi04569269-49 2021-07-31T01:46:27.572 INFO:tasks.rados.rados.0.smithi045.stdout:229: seq_num 91 ranges {139264=729088,868352=425984,1294336=376832} 2021-07-31T01:46:27.574 INFO:tasks.rados.rados.0.smithi045.stdout:229: writing smithi04569269-9 from 139264 to 868352 tid 1 2021-07-31T01:46:27.577 INFO:tasks.rados.rados.0.smithi045.stdout:229: writing smithi04569269-9 from 868352 to 1294336 tid 2 2021-07-31T01:46:27.580 INFO:tasks.rados.rados.0.smithi045.stdout:229: writing smithi04569269-9 from 1294336 to 1671168 tid 3 2021-07-31T01:46:27.580 INFO:tasks.rados.rados.0.smithi045.stdout:228: done (1 left) 2021-07-31T01:46:27.580 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 48 current snap is 15 2021-07-31T01:46:27.580 INFO:tasks.rados.rados.0.smithi045.stdout:229: finishing write tid 1 to smithi04569269-9 2021-07-31T01:46:27.581 INFO:tasks.rados.rados.0.smithi045.stdout:229: finishing write tid 2 to smithi04569269-9 2021-07-31T01:46:27.581 INFO:tasks.rados.rados.0.smithi045.stdout:230: seq_num 92 ranges {778240=745472,1523712=761856,2285568=40960} 2021-07-31T01:46:27.588 INFO:tasks.rados.rados.0.smithi045.stdout:230: writing smithi04569269-48 from 778240 to 1523712 tid 1 2021-07-31T01:46:27.593 INFO:tasks.rados.rados.0.smithi045.stdout:230: writing smithi04569269-48 from 1523712 to 2285568 tid 2 2021-07-31T01:46:27.594 INFO:tasks.rados.rados.0.smithi045.stdout:230: writing smithi04569269-48 from 2285568 to 2326528 tid 3 2021-07-31T01:46:27.594 INFO:tasks.rados.rados.0.smithi045.stdout:231: read oid 49 snap -1 2021-07-31T01:46:27.594 INFO:tasks.rados.rados.0.smithi045.stdout:231: expect (ObjNum 36 snap 0 seq_num 36) 2021-07-31T01:46:27.595 INFO:tasks.rados.rados.0.smithi045.stdout:232: read oid 1 snap 13 2021-07-31T01:46:27.595 INFO:tasks.rados.rados.0.smithi045.stdout:232: expect (ObjNum 71 snap 10 seq_num 71) 2021-07-31T01:46:27.595 INFO:tasks.rados.rados.0.smithi045.stdout:233: setattr oid 36 current snap is 15 2021-07-31T01:46:27.595 INFO:tasks.rados.rados.0.smithi045.stdout:229: finishing write tid 3 to smithi04569269-9 2021-07-31T01:46:27.596 INFO:tasks.rados.rados.0.smithi045.stdout:229: finishing write tid 4 to smithi04569269-9 2021-07-31T01:46:27.596 INFO:tasks.rados.rados.0.smithi045.stdout:229: finishing write tid 5 to smithi04569269-9 2021-07-31T01:46:27.596 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 9 v 30 (ObjNum 91 snap 15 seq_num 91) dirty exists 2021-07-31T01:46:27.596 INFO:tasks.rados.rados.0.smithi045.stdout:229: left oid 9 (ObjNum 91 snap 15 seq_num 91) 2021-07-31T01:46:27.596 INFO:tasks.rados.rados.0.smithi045.stdout:230: finishing write tid 1 to smithi04569269-48 2021-07-31T01:46:27.597 INFO:tasks.rados.rados.0.smithi045.stdout:229: done (4 left) 2021-07-31T01:46:27.598 INFO:tasks.rados.rados.0.smithi045.stdout:234: rollback oid 15 current snap is 15 2021-07-31T01:46:27.598 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 15 to 13 2021-07-31T01:46:27.598 INFO:tasks.rados.rados.0.smithi045.stdout:235: delete oid 14 current snap is 15 2021-07-31T01:46:27.598 INFO:tasks.rados.rados.0.smithi045.stdout:234: finishing rollback tid 1 to smithi04569269-15 2021-07-31T01:46:27.599 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 15 v 0 (ObjNum 6 snap 4 seq_num 1) dirty dne 2021-07-31T01:46:27.599 INFO:tasks.rados.rados.0.smithi045.stdout:234: done (5 left) 2021-07-31T01:46:27.599 INFO:tasks.rados.rados.0.smithi045.stdout:235: done (4 left) 2021-07-31T01:46:27.599 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 27 current snap is 15 2021-07-31T01:46:27.600 INFO:tasks.rados.rados.0.smithi045.stdout:236: seq_num 94 ranges {57344=778240,835584=532480,1368064=270336} 2021-07-31T01:46:27.602 INFO:tasks.rados.rados.0.smithi045.stdout:236: writing smithi04569269-27 from 57344 to 835584 tid 1 2021-07-31T01:46:27.605 INFO:tasks.rados.rados.0.smithi045.stdout:236: writing smithi04569269-27 from 835584 to 1368064 tid 2 2021-07-31T01:46:27.606 INFO:tasks.rados.rados.0.smithi045.stdout:236: writing smithi04569269-27 from 1368064 to 1638400 tid 3 2021-07-31T01:46:27.607 INFO:tasks.rados.rados.0.smithi045.stdout:237: setattr oid 17 current snap is 15 2021-07-31T01:46:27.607 INFO:tasks.rados.rados.0.smithi045.stdout:231: expect (ObjNum 36 snap 0 seq_num 36) 2021-07-31T01:46:27.622 INFO:tasks.rados.rados.0.smithi045.stdout:230: finishing write tid 2 to smithi04569269-48 2021-07-31T01:46:27.623 INFO:tasks.rados.rados.0.smithi045.stdout:230: finishing write tid 3 to smithi04569269-48 2021-07-31T01:46:27.623 INFO:tasks.rados.rados.0.smithi045.stdout:230: finishing write tid 4 to smithi04569269-48 2021-07-31T01:46:27.623 INFO:tasks.rados.rados.0.smithi045.stdout:230: finishing write tid 5 to smithi04569269-48 2021-07-31T01:46:27.624 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 48 v 26 (ObjNum 92 snap 15 seq_num 92) dirty exists 2021-07-31T01:46:27.624 INFO:tasks.rados.rados.0.smithi045.stdout:230: left oid 48 (ObjNum 92 snap 15 seq_num 92) 2021-07-31T01:46:27.626 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 36 v 115 (ObjNum 1 snap 0 seq_num 1) dirty exists 2021-07-31T01:46:27.626 INFO:tasks.rados.rados.0.smithi045.stdout:232: expect (ObjNum 71 snap 10 seq_num 71) 2021-07-31T01:46:27.654 INFO:tasks.rados.rados.0.smithi045.stdout:236: finishing write tid 1 to smithi04569269-27 2021-07-31T01:46:27.655 INFO:tasks.rados.rados.0.smithi045.stdout:236: finishing write tid 2 to smithi04569269-27 2021-07-31T01:46:27.655 INFO:tasks.rados.rados.0.smithi045.stdout:236: finishing write tid 3 to smithi04569269-27 2021-07-31T01:46:27.655 INFO:tasks.rados.rados.0.smithi045.stdout:236: finishing write tid 4 to smithi04569269-27 2021-07-31T01:46:27.656 INFO:tasks.rados.rados.0.smithi045.stdout:236: finishing write tid 5 to smithi04569269-27 2021-07-31T01:46:27.656 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 27 v 41 (ObjNum 94 snap 15 seq_num 94) dirty exists 2021-07-31T01:46:27.656 INFO:tasks.rados.rados.0.smithi045.stdout:236: left oid 27 (ObjNum 94 snap 15 seq_num 94) 2021-07-31T01:46:27.656 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 17 v 64 (ObjNum 86 snap 13 seq_num 86) dirty exists 2021-07-31T01:46:27.656 INFO:tasks.rados.rados.0.smithi045.stdout:230: done (5 left) 2021-07-31T01:46:27.657 INFO:tasks.rados.rados.0.smithi045.stdout:231: done (4 left) 2021-07-31T01:46:27.657 INFO:tasks.rados.rados.0.smithi045.stdout:232: done (3 left) 2021-07-31T01:46:27.657 INFO:tasks.rados.rados.0.smithi045.stdout:233: done (2 left) 2021-07-31T01:46:27.657 INFO:tasks.rados.rados.0.smithi045.stdout:236: done (1 left) 2021-07-31T01:46:27.658 INFO:tasks.rados.rados.0.smithi045.stdout:237: done (0 left) 2021-07-31T01:46:27.658 INFO:tasks.rados.rados.0.smithi045.stdout:238: setattr oid 39 current snap is 15 2021-07-31T01:46:27.659 INFO:tasks.rados.rados.0.smithi045.stdout:239: snap_remove snap 11 2021-07-31T01:46:28.542 INFO:tasks.rados.rados.0.smithi045.stdout:239: done (1 left) 2021-07-31T01:46:28.542 INFO:tasks.rados.rados.0.smithi045.stdout:240: read oid 17 snap -1 2021-07-31T01:46:28.543 INFO:tasks.rados.rados.0.smithi045.stdout:240: expect (ObjNum 86 snap 13 seq_num 86) 2021-07-31T01:46:28.543 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 39 v 35 (ObjNum 38 snap 0 seq_num 38) dirty exists 2021-07-31T01:46:28.543 INFO:tasks.rados.rados.0.smithi045.stdout:238: done (1 left) 2021-07-31T01:46:28.543 INFO:tasks.rados.rados.0.smithi045.stdout:241: read oid 29 snap -1 2021-07-31T01:46:28.543 INFO:tasks.rados.rados.0.smithi045.stdout:241: expect (ObjNum 39 snap 0 seq_num 39) 2021-07-31T01:46:28.544 INFO:tasks.rados.rados.0.smithi045.stdout:242: snap_create 2021-07-31T01:46:28.556 INFO:tasks.rados.rados.0.smithi045.stdout:240: expect (ObjNum 86 snap 13 seq_num 86) 2021-07-31T01:46:28.596 INFO:tasks.rados.rados.0.smithi045.stdout:241: expect (ObjNum 39 snap 0 seq_num 39) 2021-07-31T01:46:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:28 smithi154 conmon[34553]: cluster 2021-07-31T01:46:27.368306+0000 mgr.y (mgr.14146) 289 : cluster [DBG] pgmap v281: 25 pgs: 25 active+clean; 36 MiB data, 117 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 2.4 MiB/s wr, 14 op/s 2021-07-31T01:46:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:28 smithi154 conmon[34553]: cluster 2021-07-31T01:46:27.534261+0000 mon.a (mon.0) 686 : cluster [DBG] osdmap e88: 8 total, 8 up, 8 in 2021-07-31T01:46:29.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:28 smithi045 conmon[39718]: cluster 2021-07-31T01:46:27.368306+0000 mgr.y 2021-07-31T01:46:29.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:28 smithi045 conmon[39718]: (mgr.14146) 289 : cluster [DBG] pgmap v281: 25 pgs: 25 active+clean; 36 MiB data, 117 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 2.4 MiB/s wr, 14 op/s 2021-07-31T01:46:29.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:28 smithi045 conmon[39718]: cluster 2021-07-31T 2021-07-31T01:46:29.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:28 smithi045 conmon[39718]: 01:46:27.534261+0000 mon.a (mon.0) 686 : cluster [DBG] 2021-07-31T01:46:29.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:28 smithi045 conmon[39718]: osdmap e88: 8 total, 8 up, 8 in 2021-07-31T01:46:29.029 INFO:journalctl@ceph.osd.0.smithi045.stdout:Jul 31 01:46:28 smithi045 conmon[43243]: debug 2021-07-31T01:46:29.029 INFO:journalctl@ceph.osd.0.smithi045.stdout:Jul 31 01:46:28 smithi045 conmon[43243]: 2021-07-31T01:46:28.538+0000 7fdf6978b700 -1 osd.0 pg_epoch: 89 pg[3.6s0( v 89'32 (0'0,89'32] local-lis/les=56/57 n=4 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [0,1,4]p0(0) r=0 lpr=56 luod=88'30 crt=88'30 lcod 88'29 mlcod 88'29 active+clean+snaptrim trimq=[18~1] ps=[19~1,1b~1]] removing snap head 2021-07-31T01:46:29.030 INFO:journalctl@ceph.osd.0.smithi045.stdout:Jul 31 01:46:28 smithi045 conmon[43243]: 2021-07-31T01:46:29.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:28 smithi045 conmon[33655]: cluster 2021-07-31T01:46:27.368306 2021-07-31T01:46:29.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:28 smithi045 conmon[33655]: +0000 mgr.y (mgr.14146) 289 : cluster [DBG] pgmap v281: 25 pgs: 25 active+clean; 36 MiB data, 117 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 2.4 MiB/s wr, 14 op/s 2021-07-31T01:46:29.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:28 smithi045 conmon[33655]: cluster 2021-07-31T01:46:27.534261+0000 2021-07-31T01:46:29.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:28 smithi045 conmon[33655]: mon.a (mon.0) 686 : cluster [DBG] osdmap e88: 8 total, 8 up, 8 in 2021-07-31T01:46:29.546 INFO:tasks.rados.rados.0.smithi045.stdout:240: done (2 left) 2021-07-31T01:46:29.546 INFO:tasks.rados.rados.0.smithi045.stdout:241: done (1 left) 2021-07-31T01:46:29.546 INFO:tasks.rados.rados.0.smithi045.stdout:242: done (0 left) 2021-07-31T01:46:29.546 INFO:tasks.rados.rados.0.smithi045.stdout:243: rollback oid 30 current snap is 16 2021-07-31T01:46:29.547 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 30 to 13 2021-07-31T01:46:29.547 INFO:tasks.rados.rados.0.smithi045.stdout:244: copy_from oid 9 from oid 25 current snap is 16 2021-07-31T01:46:29.547 INFO:tasks.rados.rados.0.smithi045.stdout:245: copy_from oid 12 from oid 28 current snap is 16 2021-07-31T01:46:29.547 INFO:tasks.rados.rados.0.smithi045.stdout:246: read oid 1 snap -1 2021-07-31T01:46:29.547 INFO:tasks.rados.rados.0.smithi045.stdout:246: expect (ObjNum 71 snap 10 seq_num 71) 2021-07-31T01:46:29.548 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 32 current snap is 16 2021-07-31T01:46:29.548 INFO:tasks.rados.rados.0.smithi045.stdout:247: seq_num 95 ranges {0=794624,794624=409600,1204224=393216} 2021-07-31T01:46:29.550 INFO:tasks.rados.rados.0.smithi045.stdout:247: writing smithi04569269-32 from 0 to 794624 tid 1 2021-07-31T01:46:29.553 INFO:tasks.rados.rados.0.smithi045.stdout:247: writing smithi04569269-32 from 794624 to 1204224 tid 2 2021-07-31T01:46:29.555 INFO:tasks.rados.rados.0.smithi045.stdout:247: writing smithi04569269-32 from 1204224 to 1597440 tid 3 2021-07-31T01:46:29.556 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 36 current snap is 16 2021-07-31T01:46:29.556 INFO:tasks.rados.rados.0.smithi045.stdout:245: finishing copy_from to smithi04569269-12 2021-07-31T01:46:29.556 INFO:tasks.rados.rados.0.smithi045.stdout:245: got expected ENOENT (src dne) 2021-07-31T01:46:29.556 INFO:tasks.rados.rados.0.smithi045.stdout:248: seq_num 96 ranges {24576=409600,434176=696320,1130496=507904} 2021-07-31T01:46:29.558 INFO:tasks.rados.rados.0.smithi045.stdout:248: writing smithi04569269-36 from 24576 to 434176 tid 1 2021-07-31T01:46:29.562 INFO:tasks.rados.rados.0.smithi045.stdout:248: writing smithi04569269-36 from 434176 to 1130496 tid 2 2021-07-31T01:46:29.565 INFO:tasks.rados.rados.0.smithi045.stdout:248: writing smithi04569269-36 from 1130496 to 1638400 tid 3 2021-07-31T01:46:29.567 INFO:tasks.rados.rados.0.smithi045.stdout:249: rollback oid 2 current snap is 16 2021-07-31T01:46:29.567 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 2 to 13 2021-07-31T01:46:29.567 INFO:tasks.rados.rados.0.smithi045.stdout:245: finishing copy_from racing read to smithi04569269-12 2021-07-31T01:46:29.567 INFO:tasks.rados.rados.0.smithi045.stdout:243: finishing rollback tid 0 to smithi04569269-30 2021-07-31T01:46:29.568 INFO:tasks.rados.rados.0.smithi045.stdout:243: finishing rollback tid 1 to smithi04569269-30 2021-07-31T01:46:29.568 INFO:tasks.rados.rados.0.smithi045.stdout:243: finishing rollback tid 2 to smithi04569269-30 2021-07-31T01:46:29.568 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 30 v 32 (ObjNum 29 snap 0 seq_num 29) dirty exists 2021-07-31T01:46:29.568 INFO:tasks.rados.rados.0.smithi045.stdout:244: finishing copy_from to smithi04569269-9 2021-07-31T01:46:29.569 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 9 v 8 (ObjNum 36 snap 0 seq_num 36) dirty exists 2021-07-31T01:46:29.569 INFO:tasks.rados.rados.0.smithi045.stdout:244: finishing copy_from racing read to smithi04569269-9 2021-07-31T01:46:29.569 INFO:tasks.rados.rados.0.smithi045.stdout:247: finishing write tid 1 to smithi04569269-32 2021-07-31T01:46:29.569 INFO:tasks.rados.rados.0.smithi045.stdout:247: finishing write tid 2 to smithi04569269-32 2021-07-31T01:46:29.569 INFO:tasks.rados.rados.0.smithi045.stdout:246: expect (ObjNum 71 snap 10 seq_num 71) 2021-07-31T01:46:29.598 INFO:tasks.rados.rados.0.smithi045.stdout:247: finishing write tid 3 to smithi04569269-32 2021-07-31T01:46:29.598 INFO:tasks.rados.rados.0.smithi045.stdout:247: finishing write tid 4 to smithi04569269-32 2021-07-31T01:46:29.598 INFO:tasks.rados.rados.0.smithi045.stdout:247: finishing write tid 5 to smithi04569269-32 2021-07-31T01:46:29.599 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 32 v 34 (ObjNum 95 snap 16 seq_num 95) dirty exists 2021-07-31T01:46:29.599 INFO:tasks.rados.rados.0.smithi045.stdout:247: left oid 32 (ObjNum 95 snap 16 seq_num 95) 2021-07-31T01:46:29.599 INFO:tasks.rados.rados.0.smithi045.stdout:248: finishing write tid 1 to smithi04569269-36 2021-07-31T01:46:29.599 INFO:tasks.rados.rados.0.smithi045.stdout:248: finishing write tid 2 to smithi04569269-36 2021-07-31T01:46:29.600 INFO:tasks.rados.rados.0.smithi045.stdout:243: done (6 left) 2021-07-31T01:46:29.600 INFO:tasks.rados.rados.0.smithi045.stdout:244: done (5 left) 2021-07-31T01:46:29.600 INFO:tasks.rados.rados.0.smithi045.stdout:245: done (4 left) 2021-07-31T01:46:29.600 INFO:tasks.rados.rados.0.smithi045.stdout:246: done (3 left) 2021-07-31T01:46:29.601 INFO:tasks.rados.rados.0.smithi045.stdout:247: done (2 left) 2021-07-31T01:46:29.601 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 10 current snap is 16 2021-07-31T01:46:29.601 INFO:tasks.rados.rados.0.smithi045.stdout:249: finishing rollback tid 1 to smithi04569269-2 2021-07-31T01:46:29.601 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 2 v 0 (ObjNum -1126017128 snap 21873 seq_num 1) dirty dne 2021-07-31T01:46:29.601 INFO:tasks.rados.rados.0.smithi045.stdout:248: finishing write tid 3 to smithi04569269-36 2021-07-31T01:46:29.602 INFO:tasks.rados.rados.0.smithi045.stdout:248: finishing write tid 4 to smithi04569269-36 2021-07-31T01:46:29.602 INFO:tasks.rados.rados.0.smithi045.stdout:248: finishing write tid 5 to smithi04569269-36 2021-07-31T01:46:29.602 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 36 v 126 (ObjNum 96 snap 16 seq_num 96) dirty exists 2021-07-31T01:46:29.602 INFO:tasks.rados.rados.0.smithi045.stdout:248: left oid 36 (ObjNum 96 snap 16 seq_num 96) 2021-07-31T01:46:29.603 INFO:tasks.rados.rados.0.smithi045.stdout:250: seq_num 97 ranges {155648=434176,589824=581632,1171456=540672,1712128=81920} 2021-07-31T01:46:29.603 INFO:tasks.rados.rados.0.smithi045.stdout:250: writing smithi04569269-10 from 155648 to 589824 tid 1 2021-07-31T01:46:29.604 INFO:tasks.rados.rados.0.smithi045.stdout:250: writing smithi04569269-10 from 589824 to 1171456 tid 2 2021-07-31T01:46:29.609 INFO:tasks.rados.rados.0.smithi045.stdout:250: writing smithi04569269-10 from 1171456 to 1712128 tid 3 2021-07-31T01:46:29.610 INFO:tasks.rados.rados.0.smithi045.stdout:250: writing smithi04569269-10 from 1712128 to 1794048 tid 4 2021-07-31T01:46:29.611 INFO:tasks.rados.rados.0.smithi045.stdout:248: done (2 left) 2021-07-31T01:46:29.612 INFO:tasks.rados.rados.0.smithi045.stdout:249: done (1 left) 2021-07-31T01:46:29.612 INFO:tasks.rados.rados.0.smithi045.stdout:251: read oid 33 snap -1 2021-07-31T01:46:29.612 INFO:tasks.rados.rados.0.smithi045.stdout:251: expect (ObjNum 32 snap 0 seq_num 32) 2021-07-31T01:46:29.613 INFO:tasks.rados.rados.0.smithi045.stdout:252: rollback oid 39 current snap is 16 2021-07-31T01:46:29.613 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 39 to 15 2021-07-31T01:46:29.613 INFO:tasks.rados.rados.0.smithi045.stdout:250: finishing write tid 1 to smithi04569269-10 2021-07-31T01:46:29.613 INFO:tasks.rados.rados.0.smithi045.stdout:250: finishing write tid 2 to smithi04569269-10 2021-07-31T01:46:29.614 INFO:tasks.rados.rados.0.smithi045.stdout:253: rmattr oid 23 current snap is 16 2021-07-31T01:46:29.614 INFO:tasks.rados.rados.0.smithi045.stdout:254: copy_from oid 29 from oid 18 current snap is 16 2021-07-31T01:46:29.614 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 14 current snap is 16 2021-07-31T01:46:29.614 INFO:tasks.rados.rados.0.smithi045.stdout:255: seq_num 98 ranges {0=450560,450560=466944,917504=737280} 2021-07-31T01:46:29.615 INFO:tasks.rados.rados.0.smithi045.stdout:255: writing smithi04569269-14 from 0 to 450560 tid 1 2021-07-31T01:46:29.615 INFO:tasks.rados.rados.0.smithi045.stdout:255: writing smithi04569269-14 from 450560 to 917504 tid 2 2021-07-31T01:46:29.618 INFO:tasks.rados.rados.0.smithi045.stdout:255: writing smithi04569269-14 from 917504 to 1654784 tid 3 2021-07-31T01:46:29.618 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 28 current snap is 16 2021-07-31T01:46:29.618 INFO:tasks.rados.rados.0.smithi045.stdout:256: seq_num 99 ranges {0=466944,466944=802816,1269760=401408} 2021-07-31T01:46:29.620 INFO:tasks.rados.rados.0.smithi045.stdout:256: writing smithi04569269-28 from 0 to 466944 tid 1 2021-07-31T01:46:29.625 INFO:tasks.rados.rados.0.smithi045.stdout:256: writing smithi04569269-28 from 466944 to 1269760 tid 2 2021-07-31T01:46:29.627 INFO:tasks.rados.rados.0.smithi045.stdout:256: writing smithi04569269-28 from 1269760 to 1671168 tid 3 2021-07-31T01:46:29.628 INFO:tasks.rados.rados.0.smithi045.stdout:257: read oid 13 snap 13 2021-07-31T01:46:29.628 INFO:tasks.rados.rados.0.smithi045.stdout:257: expect (ObjNum 75 snap 11 seq_num 75) 2021-07-31T01:46:29.628 INFO:tasks.rados.rados.0.smithi045.stdout:258: rmattr oid 4 current snap is 16 2021-07-31T01:46:29.628 INFO:tasks.rados.rados.0.smithi045.stdout:258: done (8 left) 2021-07-31T01:46:29.629 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 43 current snap is 16 2021-07-31T01:46:29.629 INFO:tasks.rados.rados.0.smithi045.stdout:250: finishing write tid 3 to smithi04569269-10 2021-07-31T01:46:29.629 INFO:tasks.rados.rados.0.smithi045.stdout:250: finishing write tid 4 to smithi04569269-10 2021-07-31T01:46:29.629 INFO:tasks.rados.rados.0.smithi045.stdout:250: finishing write tid 5 to smithi04569269-10 2021-07-31T01:46:29.629 INFO:tasks.rados.rados.0.smithi045.stdout:251: expect (ObjNum 32 snap 0 seq_num 32) 2021-07-31T01:46:29.642 INFO:tasks.rados.rados.0.smithi045.stdout:252: finishing rollback tid 0 to smithi04569269-39 2021-07-31T01:46:29.642 INFO:tasks.rados.rados.0.smithi045.stdout:252: finishing rollback tid 1 to smithi04569269-39 2021-07-31T01:46:29.642 INFO:tasks.rados.rados.0.smithi045.stdout:259: seq_num 100 ranges {1810432=483328,2293760=688128,2981888=516096} 2021-07-31T01:46:29.654 INFO:tasks.rados.rados.0.smithi045.stdout:259: writing smithi04569269-43 from 1810432 to 2293760 tid 1 2021-07-31T01:46:29.658 INFO:tasks.rados.rados.0.smithi045.stdout:259: writing smithi04569269-43 from 2293760 to 2981888 tid 2 2021-07-31T01:46:29.662 INFO:tasks.rados.rados.0.smithi045.stdout:259: writing smithi04569269-43 from 2981888 to 3497984 tid 3 2021-07-31T01:46:29.662 INFO:tasks.rados.rados.0.smithi045.stdout:251: done (8 left) 2021-07-31T01:46:29.662 INFO:tasks.rados.rados.0.smithi045.stdout:260: read oid 5 snap -1 2021-07-31T01:46:29.663 INFO:tasks.rados.rados.0.smithi045.stdout:260: expect deleted 2021-07-31T01:46:29.663 INFO:tasks.rados.rados.0.smithi045.stdout:261: read oid 2 snap 13 2021-07-31T01:46:29.663 INFO:tasks.rados.rados.0.smithi045.stdout:261: expect deleted 2021-07-31T01:46:29.663 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 26 current snap is 16 2021-07-31T01:46:29.663 INFO:tasks.rados.rados.0.smithi045.stdout:252: finishing rollback tid 2 to smithi04569269-39 2021-07-31T01:46:29.664 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 39 v 41 (ObjNum 38 snap 0 seq_num 38) dirty exists 2021-07-31T01:46:29.664 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 23 v 43 (ObjNum 53 snap 1 seq_num 53) dirty exists 2021-07-31T01:46:29.664 INFO:tasks.rados.rados.0.smithi045.stdout:250: finishing write tid 6 to smithi04569269-10 2021-07-31T01:46:29.664 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 10 v 72 (ObjNum 97 snap 16 seq_num 97) dirty exists 2021-07-31T01:46:29.665 INFO:tasks.rados.rados.0.smithi045.stdout:250: left oid 10 (ObjNum 97 snap 16 seq_num 97) 2021-07-31T01:46:29.665 INFO:tasks.rados.rados.0.smithi045.stdout:255: finishing write tid 1 to smithi04569269-14 2021-07-31T01:46:29.665 INFO:tasks.rados.rados.0.smithi045.stdout:255: finishing write tid 2 to smithi04569269-14 2021-07-31T01:46:29.665 INFO:tasks.rados.rados.0.smithi045.stdout:254: finishing copy_from to smithi04569269-29 2021-07-31T01:46:29.665 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 29 v 2 (ObjNum 45 snap 0 seq_num 45) dirty exists 2021-07-31T01:46:29.666 INFO:tasks.rados.rados.0.smithi045.stdout:254: finishing copy_from racing read to smithi04569269-29 2021-07-31T01:46:29.666 INFO:tasks.rados.rados.0.smithi045.stdout:255: finishing write tid 3 to smithi04569269-14 2021-07-31T01:46:29.666 INFO:tasks.rados.rados.0.smithi045.stdout:255: finishing write tid 4 to smithi04569269-14 2021-07-31T01:46:29.666 INFO:tasks.rados.rados.0.smithi045.stdout:255: finishing write tid 5 to smithi04569269-14 2021-07-31T01:46:29.667 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 14 v 49 (ObjNum 98 snap 16 seq_num 98) dirty exists 2021-07-31T01:46:29.667 INFO:tasks.rados.rados.0.smithi045.stdout:255: left oid 14 (ObjNum 98 snap 16 seq_num 98) 2021-07-31T01:46:29.667 INFO:tasks.rados.rados.0.smithi045.stdout:256: finishing write tid 1 to smithi04569269-28 2021-07-31T01:46:29.667 INFO:tasks.rados.rados.0.smithi045.stdout:256: finishing write tid 2 to smithi04569269-28 2021-07-31T01:46:29.667 INFO:tasks.rados.rados.0.smithi045.stdout:256: finishing write tid 3 to smithi04569269-28 2021-07-31T01:46:29.668 INFO:tasks.rados.rados.0.smithi045.stdout:256: finishing write tid 4 to smithi04569269-28 2021-07-31T01:46:29.668 INFO:tasks.rados.rados.0.smithi045.stdout:262: seq_num 101 ranges {0=499712,499712=573440,1073152=524288,1597440=106496} 2021-07-31T01:46:29.668 INFO:tasks.rados.rados.0.smithi045.stdout:262: writing smithi04569269-26 from 0 to 499712 tid 1 2021-07-31T01:46:29.669 INFO:tasks.rados.rados.0.smithi045.stdout:262: writing smithi04569269-26 from 499712 to 1073152 tid 2 2021-07-31T01:46:29.670 INFO:tasks.rados.rados.0.smithi045.stdout:262: writing smithi04569269-26 from 1073152 to 1597440 tid 3 2021-07-31T01:46:29.671 INFO:tasks.rados.rados.0.smithi045.stdout:262: writing smithi04569269-26 from 1597440 to 1703936 tid 4 2021-07-31T01:46:29.671 INFO:tasks.rados.rados.0.smithi045.stdout:250: done (10 left) 2021-07-31T01:46:29.671 INFO:tasks.rados.rados.0.smithi045.stdout:252: done (9 left) 2021-07-31T01:46:29.672 INFO:tasks.rados.rados.0.smithi045.stdout:253: done (8 left) 2021-07-31T01:46:29.672 INFO:tasks.rados.rados.0.smithi045.stdout:254: done (7 left) 2021-07-31T01:46:29.672 INFO:tasks.rados.rados.0.smithi045.stdout:255: done (6 left) 2021-07-31T01:46:29.673 INFO:tasks.rados.rados.0.smithi045.stdout:263: snap_remove snap 15 2021-07-31T01:46:29.673 INFO:tasks.rados.rados.0.smithi045.stdout:256: finishing write tid 5 to smithi04569269-28 2021-07-31T01:46:29.673 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 28 v 76 (ObjNum 99 snap 16 seq_num 99) dirty exists 2021-07-31T01:46:29.674 INFO:tasks.rados.rados.0.smithi045.stdout:256: left oid 28 (ObjNum 99 snap 16 seq_num 99) 2021-07-31T01:46:29.674 INFO:tasks.rados.rados.0.smithi045.stdout:257: expect (ObjNum 75 snap 11 seq_num 75) 2021-07-31T01:46:29.763 INFO:tasks.rados.rados.0.smithi045.stdout:259: finishing write tid 1 to smithi04569269-43 2021-07-31T01:46:29.763 INFO:tasks.rados.rados.0.smithi045.stdout:259: finishing write tid 2 to smithi04569269-43 2021-07-31T01:46:29.764 INFO:tasks.rados.rados.0.smithi045.stdout:259: finishing write tid 3 to smithi04569269-43 2021-07-31T01:46:29.764 INFO:tasks.rados.rados.0.smithi045.stdout:259: finishing write tid 4 to smithi04569269-43 2021-07-31T01:46:29.764 INFO:tasks.rados.rados.0.smithi045.stdout:259: finishing write tid 5 to smithi04569269-43 2021-07-31T01:46:29.764 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 43 v 54 (ObjNum 100 snap 16 seq_num 100) dirty exists 2021-07-31T01:46:29.764 INFO:tasks.rados.rados.0.smithi045.stdout:259: left oid 43 (ObjNum 100 snap 16 seq_num 100) 2021-07-31T01:46:29.765 INFO:tasks.rados.rados.0.smithi045.stdout:262: finishing write tid 1 to smithi04569269-26 2021-07-31T01:46:30.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:29 smithi154 conmon[34553]: cluster 2021-07-31T01:46:28.539835+0000 mon.a (mon.0) 687 : cluster [DBG] 2021-07-31T01:46:30.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:29 smithi154 conmon[34553]: osdmap e89: 8 total, 8 up, 8 in 2021-07-31T01:46:30.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:29 smithi045 conmon[33655]: cluster 2021-07-31T01:46:28.539835+0000 mon.a (mon.0) 687 : cluster 2021-07-31T01:46:30.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:29 smithi045 conmon[33655]: [DBG] osdmap e89: 8 total, 8 up, 8 in 2021-07-31T01:46:30.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:29 smithi045 conmon[39718]: cluster 2021-07-31T01:46:28.539835+0000 2021-07-31T01:46:30.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:29 smithi045 conmon[39718]: mon.a (mon.0) 687 : cluster [DBG] osdmap e89: 8 total, 8 up, 8 in 2021-07-31T01:46:30.548 INFO:tasks.rados.rados.0.smithi045.stdout:256: done (6 left) 2021-07-31T01:46:30.548 INFO:tasks.rados.rados.0.smithi045.stdout:257: done (5 left) 2021-07-31T01:46:30.548 INFO:tasks.rados.rados.0.smithi045.stdout:259: done (4 left) 2021-07-31T01:46:30.548 INFO:tasks.rados.rados.0.smithi045.stdout:260: done (3 left) 2021-07-31T01:46:30.549 INFO:tasks.rados.rados.0.smithi045.stdout:261: done (2 left) 2021-07-31T01:46:30.549 INFO:tasks.rados.rados.0.smithi045.stdout:263: done (1 left) 2021-07-31T01:46:30.549 INFO:tasks.rados.rados.0.smithi045.stdout:264: copy_from oid 38 from oid 19 current snap is 16 2021-07-31T01:46:30.549 INFO:tasks.rados.rados.0.smithi045.stdout:262: finishing write tid 2 to smithi04569269-26 2021-07-31T01:46:30.549 INFO:tasks.rados.rados.0.smithi045.stdout:262: finishing write tid 3 to smithi04569269-26 2021-07-31T01:46:30.550 INFO:tasks.rados.rados.0.smithi045.stdout:262: finishing write tid 4 to smithi04569269-26 2021-07-31T01:46:30.550 INFO:tasks.rados.rados.0.smithi045.stdout:262: finishing write tid 5 to smithi04569269-26 2021-07-31T01:46:30.550 INFO:tasks.rados.rados.0.smithi045.stdout:262: finishing write tid 6 to smithi04569269-26 2021-07-31T01:46:30.550 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 26 v 39 (ObjNum 101 snap 16 seq_num 101) dirty exists 2021-07-31T01:46:30.551 INFO:tasks.rados.rados.0.smithi045.stdout:262: left oid 26 (ObjNum 101 snap 16 seq_num 101) 2021-07-31T01:46:30.551 INFO:tasks.rados.rados.0.smithi045.stdout:262: done (1 left) 2021-07-31T01:46:30.551 INFO:tasks.rados.rados.0.smithi045.stdout:265: rollback oid 16 current snap is 16 2021-07-31T01:46:30.551 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 16 to 13 2021-07-31T01:46:30.551 INFO:tasks.rados.rados.0.smithi045.stdout:266: rollback oid 3 current snap is 16 2021-07-31T01:46:30.552 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 3 to 13 2021-07-31T01:46:30.552 INFO:tasks.rados.rados.0.smithi045.stdout:267: rmattr oid 21 current snap is 16 2021-07-31T01:46:30.552 INFO:tasks.rados.rados.0.smithi045.stdout:267: done (3 left) 2021-07-31T01:46:30.552 INFO:tasks.rados.rados.0.smithi045.stdout:268: rollback oid 13 current snap is 16 2021-07-31T01:46:30.553 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 13 to 13 2021-07-31T01:46:30.553 INFO:tasks.rados.rados.0.smithi045.stdout:269: rmattr oid 47 current snap is 16 2021-07-31T01:46:30.553 INFO:tasks.rados.rados.0.smithi045.stdout:269: done (4 left) 2021-07-31T01:46:30.554 INFO:tasks.rados.rados.0.smithi045.stdout:270: copy_from oid 39 from oid 27 current snap is 16 2021-07-31T01:46:30.554 INFO:tasks.rados.rados.0.smithi045.stdout:271: rollback oid 9 current snap is 16 2021-07-31T01:46:30.554 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 9 to 13 2021-07-31T01:46:30.554 INFO:tasks.rados.rados.0.smithi045.stdout:272: snap_create 2021-07-31T01:46:30.557 INFO:tasks.rados.rados.0.smithi045.stdout:266: finishing rollback tid 0 to smithi04569269-3 2021-07-31T01:46:30.558 INFO:tasks.rados.rados.0.smithi045.stdout:266: finishing rollback tid 1 to smithi04569269-3 2021-07-31T01:46:30.558 INFO:tasks.rados.rados.0.smithi045.stdout:266: finishing rollback tid 2 to smithi04569269-3 2021-07-31T01:46:30.558 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 3 v 130 (ObjNum 6 snap 0 seq_num 6) dirty exists 2021-07-31T01:46:30.559 INFO:tasks.rados.rados.0.smithi045.stdout:265: finishing rollback tid 0 to smithi04569269-16 2021-07-31T01:46:30.559 INFO:tasks.rados.rados.0.smithi045.stdout:265: finishing rollback tid 1 to smithi04569269-16 2021-07-31T01:46:30.560 INFO:tasks.rados.rados.0.smithi045.stdout:268: finishing rollback tid 0 to smithi04569269-13 2021-07-31T01:46:30.560 INFO:tasks.rados.rados.0.smithi045.stdout:271: finishing rollback tid 0 to smithi04569269-9 2021-07-31T01:46:30.560 INFO:tasks.rados.rados.0.smithi045.stdout:271: finishing rollback tid 1 to smithi04569269-9 2021-07-31T01:46:30.561 INFO:tasks.rados.rados.0.smithi045.stdout:268: finishing rollback tid 1 to smithi04569269-13 2021-07-31T01:46:30.561 INFO:tasks.rados.rados.0.smithi045.stdout:271: finishing rollback tid 2 to smithi04569269-9 2021-07-31T01:46:30.561 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 9 v 39 (ObjNum 8 snap 0 seq_num 8) dirty exists 2021-07-31T01:46:30.561 INFO:tasks.rados.rados.0.smithi045.stdout:268: finishing rollback tid 2 to smithi04569269-13 2021-07-31T01:46:30.561 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 13 v 59 (ObjNum 75 snap 11 seq_num 75) dirty exists 2021-07-31T01:46:30.564 INFO:tasks.rados.rados.0.smithi045.stdout:265: finishing rollback tid 2 to smithi04569269-16 2021-07-31T01:46:30.564 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 16 v 52 (ObjNum 80 snap 11 seq_num 80) dirty exists 2021-07-31T01:46:30.573 INFO:tasks.rados.rados.0.smithi045.stdout:270: finishing copy_from to smithi04569269-39 2021-07-31T01:46:30.573 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 39 v 41 (ObjNum 94 snap 15 seq_num 94) dirty exists 2021-07-31T01:46:30.574 INFO:tasks.rados.rados.0.smithi045.stdout:270: finishing copy_from racing read to smithi04569269-39 2021-07-31T01:46:30.574 INFO:tasks.rados.rados.0.smithi045.stdout:264: finishing copy_from to smithi04569269-38 2021-07-31T01:46:30.575 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 38 v 44 (ObjNum 90 snap 15 seq_num 90) dirty exists 2021-07-31T01:46:30.576 INFO:tasks.rados.rados.0.smithi045.stdout:264: finishing copy_from racing read to smithi04569269-38 2021-07-31T01:46:31.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:30 smithi154 conmon[34553]: cluster 2021-07-31T01:46:31.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:30 smithi154 conmon[34553]: 2021-07-31T01:46:31.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:30 smithi154 conmon[34553]: 2021 2021-07-31T01:46:31.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:30 smithi154 conmon[34553]: - 2021-07-31T01:46:31.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:30 smithi154 conmon[34553]: 07-31T01:46:29.368863+0000 mgr.y (mgr 2021-07-31T01:46:31.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:30 smithi154 conmon[34553]: .14146) 290 : cluster [DBG] pgmap v284: 25 pgs: 25 active+clean; 43 MiB data, 131 MiB used, 715 GiB / 715 GiB avail; 4.5 MiB/s rd, 2.5 MiB/s wr, 34 op/s 2021-07-31T01:46:31.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:30 smithi154 conmon[34553]: cluster 2021-07-31T01:46:31.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:30 smithi154 conmon[34553]: 2021-07-31T01:46:29.544824+0000 mon.a (mon.0) 688 : cluster [DBG] osdmap e90: 8 total, 8 up, 8 in 2021-07-31T01:46:31.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:30 smithi045 conmon[33655]: cluster 2021-07-31T01:46:29.368863+0000 mgr.y (mgr.14146) 290 : cluster [DBG] pgmap v284: 25 pgs: 25 active+clean; 43 MiB data, 131 MiB used, 715 GiB / 715 GiB avail; 4.5 MiB/s rd, 2.5 MiB/s wr, 34 op/s 2021-07-31T01:46:31.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:30 smithi045 conmon[33655]: cluster 2021- 2021-07-31T01:46:31.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:30 smithi045 conmon[33655]: 07-31T01:46:29.544824+0000 mon.a (mon.0) 688 : cluster [DBG] osdmap e90: 8 total, 8 up, 8 in 2021-07-31T01:46:31.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:30 smithi045 conmon[39718]: cluster 2021-07-31T01:46:31.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:30 smithi045 conmon[39718]: 2021-07-31T01: 2021-07-31T01:46:31.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:30 smithi045 conmon[39718]: 46:29.368863+0000 mgr.y (mgr.14146) 290 : cluster [DBG] pgmap v284: 25 pgs: 25 active+clean; 43 MiB data, 131 MiB used, 715 GiB / 715 GiB avail; 4.5 MiB/s rd, 2.5 MiB/s wr, 34 op/s 2021-07-31T01:46:31.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:30 smithi045 conmon[39718]: cluster 2021-07-31T01:46:29.544824+0000 mon.a (mon.0) 688 2021-07-31T01:46:31.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:30 smithi045 conmon[39718]: : cluster [DBG] osdmap e90: 8 total, 8 up, 8 in 2021-07-31T01:46:31.474 INFO:tasks.rados.rados.0.smithi045.stdout:264: done (6 left) 2021-07-31T01:46:31.474 INFO:tasks.rados.rados.0.smithi045.stdout:265: done (5 left) 2021-07-31T01:46:31.474 INFO:tasks.rados.rados.0.smithi045.stdout:266: done (4 left) 2021-07-31T01:46:31.475 INFO:tasks.rados.rados.0.smithi045.stdout:268: done (3 left) 2021-07-31T01:46:31.475 INFO:tasks.rados.rados.0.smithi045.stdout:270: done (2 left) 2021-07-31T01:46:31.475 INFO:tasks.rados.rados.0.smithi045.stdout:271: done (1 left) 2021-07-31T01:46:31.476 INFO:tasks.rados.rados.0.smithi045.stdout:272: done (0 left) 2021-07-31T01:46:31.476 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 16 current snap is 17 2021-07-31T01:46:31.476 INFO:tasks.rados.rados.0.smithi045.stdout:273: seq_num 102 ranges {2572288=516096,3088384=450560,3538944=729088,4268032=24576} 2021-07-31T01:46:31.490 INFO:tasks.rados.rados.0.smithi045.stdout:273: writing smithi04569269-16 from 2572288 to 3088384 tid 1 2021-07-31T01:46:31.493 INFO:tasks.rados.rados.0.smithi045.stdout:273: writing smithi04569269-16 from 3088384 to 3538944 tid 2 2021-07-31T01:46:31.498 INFO:tasks.rados.rados.0.smithi045.stdout:273: writing smithi04569269-16 from 3538944 to 4268032 tid 3 2021-07-31T01:46:31.498 INFO:tasks.rados.rados.0.smithi045.stdout:273: writing smithi04569269-16 from 4268032 to 4292608 tid 4 2021-07-31T01:46:31.499 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 18 current snap is 17 2021-07-31T01:46:31.499 INFO:tasks.rados.rados.0.smithi045.stdout:273: finishing write tid 1 to smithi04569269-16 2021-07-31T01:46:31.499 INFO:tasks.rados.rados.0.smithi045.stdout:273: finishing write tid 2 to smithi04569269-16 2021-07-31T01:46:31.500 INFO:tasks.rados.rados.0.smithi045.stdout:274: seq_num 103 ranges {761856=532480,1294336=737280,2031616=466944} 2021-07-31T01:46:31.505 INFO:tasks.rados.rados.0.smithi045.stdout:274: writing smithi04569269-18 from 761856 to 1294336 tid 1 2021-07-31T01:46:31.509 INFO:tasks.rados.rados.0.smithi045.stdout:274: writing smithi04569269-18 from 1294336 to 2031616 tid 2 2021-07-31T01:46:31.513 INFO:tasks.rados.rados.0.smithi045.stdout:274: writing smithi04569269-18 from 2031616 to 2498560 tid 3 2021-07-31T01:46:31.513 INFO:tasks.rados.rados.0.smithi045.stdout:275: setattr oid 45 current snap is 17 2021-07-31T01:46:31.513 INFO:tasks.rados.rados.0.smithi045.stdout:273: finishing write tid 3 to smithi04569269-16 2021-07-31T01:46:31.513 INFO:tasks.rados.rados.0.smithi045.stdout:273: finishing write tid 4 to smithi04569269-16 2021-07-31T01:46:31.514 INFO:tasks.rados.rados.0.smithi045.stdout:273: finishing write tid 5 to smithi04569269-16 2021-07-31T01:46:31.514 INFO:tasks.rados.rados.0.smithi045.stdout:273: finishing write tid 6 to smithi04569269-16 2021-07-31T01:46:31.514 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 16 v 58 (ObjNum 102 snap 17 seq_num 102) dirty exists 2021-07-31T01:46:31.514 INFO:tasks.rados.rados.0.smithi045.stdout:273: left oid 16 (ObjNum 102 snap 17 seq_num 102) 2021-07-31T01:46:31.515 INFO:tasks.rados.rados.0.smithi045.stdout:274: finishing write tid 1 to smithi04569269-18 2021-07-31T01:46:31.515 INFO:tasks.rados.rados.0.smithi045.stdout:273: done (2 left) 2021-07-31T01:46:31.515 INFO:tasks.rados.rados.0.smithi045.stdout:276: snap_remove snap 13 2021-07-31T01:46:32.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:31 smithi154 conmon[34553]: cluster 2021-07-31T01:46:30.546206+0000 mon.a (mon.0) 689 : cluster [DBG] 2021-07-31T01:46:32.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:31 smithi154 conmon[34553]: osdmap e91: 8 total, 8 up, 8 in 2021-07-31T01:46:32.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:31 smithi154 conmon[34553]: cluster 2021-07-31T01:46:31.473206+0000 mon.a (mon.0) 690 : cluster [DBG] osdmap e92: 8 total, 8 up, 8 in 2021-07-31T01:46:32.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:31 smithi045 conmon[39718]: cluster 2021-07-31T01:46:30.546206+0000 mon.a (mon.0) 2021-07-31T01:46:32.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:31 smithi045 conmon[39718]: 689 : cluster [DBG] osdmap e91: 8 total, 8 up, 8 in 2021-07-31T01:46:32.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:31 smithi045 conmon[39718]: cluster 2021-07-31T01:46:31.473206 2021-07-31T01:46:32.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:31 smithi045 conmon[39718]: +0000 mon.a (mon.0) 690 : cluster [DBG] osdmap e92: 8 total, 8 up, 8 in 2021-07-31T01:46:32.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:31 smithi045 conmon[33655]: cluster 2021-07-31T01:46:32.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:31 smithi045 conmon[33655]: 2021-07-31T01:46:30 2021-07-31T01:46:32.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:31 smithi045 conmon[33655]: .546206+0000 mon.a (mon.0) 689 : cluster [DBG] osdmap e91: 8 total, 8 up, 8 in 2021-07-31T01:46:32.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:31 smithi045 conmon[33655]: cluster 2021-07-31T01:46:31.473206+0000 mon.a (mon.0) 690 2021-07-31T01:46:32.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:31 smithi045 conmon[33655]: : cluster [DBG] osdmap e92: 8 total, 8 up, 8 in 2021-07-31T01:46:32.482 INFO:tasks.rados.rados.0.smithi045.stdout:274: finishing write tid 2 to smithi04569269-18 2021-07-31T01:46:32.482 INFO:tasks.rados.rados.0.smithi045.stdout:274: finishing write tid 3 to smithi04569269-18 2021-07-31T01:46:32.483 INFO:tasks.rados.rados.0.smithi045.stdout:274: finishing write tid 4 to smithi04569269-18 2021-07-31T01:46:32.483 INFO:tasks.rados.rados.0.smithi045.stdout:274: finishing write tid 5 to smithi04569269-18 2021-07-31T01:46:32.483 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 18 v 44 (ObjNum 103 snap 17 seq_num 103) dirty exists 2021-07-31T01:46:32.483 INFO:tasks.rados.rados.0.smithi045.stdout:274: left oid 18 (ObjNum 103 snap 17 seq_num 103) 2021-07-31T01:46:32.484 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 45 v 80 (ObjNum 88 snap 15 seq_num 88) dirty exists 2021-07-31T01:46:32.484 INFO:tasks.rados.rados.0.smithi045.stdout:274: done (2 left) 2021-07-31T01:46:32.484 INFO:tasks.rados.rados.0.smithi045.stdout:275: done (1 left) 2021-07-31T01:46:32.484 INFO:tasks.rados.rados.0.smithi045.stdout:276: done (0 left) 2021-07-31T01:46:32.485 INFO:tasks.rados.rados.0.smithi045.stdout:277: setattr oid 40 current snap is 17 2021-07-31T01:46:32.485 INFO:tasks.rados.rados.0.smithi045.stdout:278: snap_create 2021-07-31T01:46:32.491 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 40 v 60 (ObjNum 0 snap 0 seq_num 0) dirty exists 2021-07-31T01:46:33.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:32 smithi154 conmon[34553]: cluster 2021-07-31T01:46:31.369300+0000 mgr.y (mgr.14146 2021-07-31T01:46:33.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:32 smithi154 conmon[34553]: ) 291 : cluster [DBG] pgmap v287: 25 pgs: 25 active+clean; 43 MiB data, 131 MiB used, 715 GiB / 715 GiB avail; 4.5 MiB/s rd, 2.5 MiB/s wr, 34 op/s 2021-07-31T01:46:33.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:32 smithi154 conmon[34553]: cluster 2021-07-31T01:46:32.481389+0000 mon.a (mon.0) 691 : cluster [DBG] osdmap e93: 8 total, 8 up, 8 in 2021-07-31T01:46:33.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:32 smithi045 conmon[39718]: cluster 2021-07-31T01:46:31.369300+0000 mgr.y (mgr.14146) 2021-07-31T01:46:33.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:32 smithi045 conmon[39718]: 291 : cluster [DBG] pgmap v287: 25 pgs: 25 active+clean; 43 MiB data, 131 MiB used, 715 GiB / 715 GiB avail; 4.5 MiB/s rd, 2.5 MiB/s wr, 34 op/s 2021-07-31T01:46:33.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:32 smithi045 conmon[39718]: cluster 2021-07-31T01:46:32. 2021-07-31T01:46:33.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:32 smithi045 conmon[39718]: 481389+0000 mon.a (mon.0) 691 : cluster [DBG] osdmap e93: 8 total, 8 up, 8 in 2021-07-31T01:46:33.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:32 smithi045 conmon[33655]: cluster 2021-07-31T01:46:31.369300+0000 mgr.y (mgr.14146) 2021-07-31T01:46:33.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:32 smithi045 conmon[33655]: 291 : cluster [DBG] pgmap v287: 25 pgs: 25 active+clean; 43 MiB data, 131 MiB used, 715 GiB / 715 GiB avail; 4.5 MiB/s rd, 2.5 MiB/s wr, 34 op/s 2021-07-31T01:46:33.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:32 smithi045 conmon[33655]: cluster 2021-07-31T01:46:32.481389+0000 mon.a (mon.0) 691 : cluster [DBG] osdmap e93: 8 total, 8 up, 8 in 2021-07-31T01:46:33.484 INFO:tasks.rados.rados.0.smithi045.stdout:277: done (1 left) 2021-07-31T01:46:33.484 INFO:tasks.rados.rados.0.smithi045.stdout:278: done (0 left) 2021-07-31T01:46:33.484 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 33 current snap is 18 2021-07-31T01:46:33.485 INFO:tasks.rados.rados.0.smithi045.stdout:279: seq_num 104 ranges {540672=548864,1089536=622592,1712128=581632} 2021-07-31T01:46:33.489 INFO:tasks.rados.rados.0.smithi045.stdout:279: writing smithi04569269-33 from 540672 to 1089536 tid 1 2021-07-31T01:46:33.493 INFO:tasks.rados.rados.0.smithi045.stdout:279: writing smithi04569269-33 from 1089536 to 1712128 tid 2 2021-07-31T01:46:33.497 INFO:tasks.rados.rados.0.smithi045.stdout:279: writing smithi04569269-33 from 1712128 to 2293760 tid 3 2021-07-31T01:46:33.497 INFO:tasks.rados.rados.0.smithi045.stdout:280: copy_from oid 18 from oid 13 current snap is 18 2021-07-31T01:46:33.497 INFO:tasks.rados.rados.0.smithi045.stdout:279: finishing write tid 1 to smithi04569269-33 2021-07-31T01:46:33.497 INFO:tasks.rados.rados.0.smithi045.stdout:279: finishing write tid 2 to smithi04569269-33 2021-07-31T01:46:33.498 INFO:tasks.rados.rados.0.smithi045.stdout:281: snap_create 2021-07-31T01:46:33.499 INFO:tasks.rados.rados.0.smithi045.stdout:279: finishing write tid 3 to smithi04569269-33 2021-07-31T01:46:33.500 INFO:tasks.rados.rados.0.smithi045.stdout:279: finishing write tid 4 to smithi04569269-33 2021-07-31T01:46:33.501 INFO:tasks.rados.rados.0.smithi045.stdout:279: finishing write tid 5 to smithi04569269-33 2021-07-31T01:46:33.501 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 33 v 61 (ObjNum 104 snap 18 seq_num 104) dirty exists 2021-07-31T01:46:33.501 INFO:tasks.rados.rados.0.smithi045.stdout:279: left oid 33 (ObjNum 104 snap 18 seq_num 104) 2021-07-31T01:46:33.517 INFO:tasks.rados.rados.0.smithi045.stdout:280: finishing copy_from to smithi04569269-18 2021-07-31T01:46:33.518 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 18 v 59 (ObjNum 75 snap 11 seq_num 75) dirty exists 2021-07-31T01:46:33.518 INFO:tasks.rados.rados.0.smithi045.stdout:280: finishing copy_from racing read to smithi04569269-18 2021-07-31T01:46:33.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:46:33 smithi045 conmon[61210]: level=error ts=2021-07-31T01:46:33.505Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:46:33.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:46:33 smithi045 conmon[61210]: level=error ts=2021-07-31T01:46:33.505Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:46:33.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:46:33 smithi045 conmon[61210]: level=error ts=2021-07-31T01:46:33.505Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T01:46:34.489 INFO:tasks.rados.rados.0.smithi045.stdout:279: done (2 left) 2021-07-31T01:46:34.489 INFO:tasks.rados.rados.0.smithi045.stdout:280: done (1 left) 2021-07-31T01:46:34.490 INFO:tasks.rados.rados.0.smithi045.stdout:281: done (0 left) 2021-07-31T01:46:34.490 INFO:tasks.rados.rados.0.smithi045.stdout:282: snap_create 2021-07-31T01:46:34.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:34 smithi045 conmon[33655]: cluster 2021-07-31T01:46:33.369947+0000 mgr.y (mgr.14146 2021-07-31T01:46:34.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:34 smithi045 conmon[33655]: ) 292 : cluster [DBG] pgmap v290: 25 pgs: 1 active+clean+snaptrim, 24 active+clean; 57 MiB data, 151 MiB used, 715 GiB / 715 GiB avail; 3.8 MiB/s rd, 3.9 MiB/s wr, 27 op/s 2021-07-31T01:46:34.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:34 smithi045 conmon[33655]: cluster 2021-07-31T01:46:33.483123+0000 mon.a (mon.0) 692 : cluster [DBG] osdmap e94: 8 total, 8 up, 8 in 2021-07-31T01:46:34.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:34 smithi045 conmon[33655]: audit 2021-07-31T01:46:34.293916+0000 mon.a (mon.0) 693 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:46:34.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:34 smithi045 conmon[33655]: audit 2021-07-31T01:46:34.295511+0000 mon.a (mon.0) 694 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:46:34.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:34 smithi045 conmon[39718]: cluster 2021-07-31T01:46:33.369947+0000 mgr.y 2021-07-31T01:46:34.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:34 smithi045 conmon[39718]: (mgr.14146) 292 : cluster [DBG] pgmap v290: 25 pgs: 1 active+clean+snaptrim, 24 active+clean; 57 MiB data, 151 MiB used, 715 GiB / 715 GiB avail; 3.8 MiB/s rd, 3.9 MiB/s wr, 27 op/s 2021-07-31T01:46:34.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:34 smithi045 conmon[39718]: cluster 2021-07-31T01:46: 2021-07-31T01:46:34.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:34 smithi045 conmon[39718]: 33.483123+0000 mon.a (mon.0) 692 : cluster [DBG] osdmap e94: 8 total, 8 up, 8 in 2021-07-31T01:46:34.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:34 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T01:46:34.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:34 smithi045 conmon[39718]: 31T01:46:34.293916+0000 mon.a (mon 2021-07-31T01:46:34.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:34 smithi045 conmon[39718]: .0) 693 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:46:34.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:34 smithi045 conmon[39718]: audit 2021-07-31T01:46:34.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:34 smithi045 conmon[39718]: 2021-07- 2021-07-31T01:46:34.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:34 smithi045 conmon[39718]: 31T01:46:34.295511+0000 mon.a ( 2021-07-31T01:46:34.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:34 smithi045 conmon[39718]: mon.0) 694 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:46:34.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:34 smithi154 conmon[34553]: cluster 2021-07-31T01:46:33.369947+0000 mgr.y (mgr.14146 2021-07-31T01:46:34.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:34 smithi154 conmon[34553]: ) 292 : cluster [DBG] pgmap v290: 25 pgs: 1 active+clean+snaptrim, 24 active+clean; 57 MiB data, 151 MiB used, 715 GiB / 715 GiB avail; 3.8 MiB/s rd, 3.9 MiB/s wr, 27 op/s 2021-07-31T01:46:34.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:34 smithi154 conmon[34553]: cluster 2021-07-31T01:46:33.483123+0000 mon.a (mon.0 2021-07-31T01:46:34.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:34 smithi154 conmon[34553]: ) 692 : cluster [DBG] osdmap e94: 8 total, 8 up, 8 in 2021-07-31T01:46:34.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:34 smithi154 conmon[34553]: audit 2021-07-31T01:46:34.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:34 smithi154 conmon[34553]: 2021-07-31T01:46:34.293916+0000 mon.a 2021-07-31T01:46:34.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:34 smithi154 conmon[34553]: (mon.0) 693 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:46:34.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:34 smithi154 conmon[34553]: audit 2021 2021-07-31T01:46:34.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:34 smithi154 conmon[34553]: -07-31T01:46:34.295511 2021-07-31T01:46:34.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:34 smithi154 conmon[34553]: +0000 mon.a (mon.0) 694 : audit [DBG] 2021-07-31T01:46:34.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:34 smithi154 conmon[34553]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:46:35.495 INFO:tasks.rados.rados.0.smithi045.stdout:282: done (0 left) 2021-07-31T01:46:35.496 INFO:tasks.rados.rados.0.smithi045.stdout:283: read oid 15 snap -1 2021-07-31T01:46:35.496 INFO:tasks.rados.rados.0.smithi045.stdout:283: expect deleted 2021-07-31T01:46:35.496 INFO:tasks.rados.rados.0.smithi045.stdout:284: read oid 1 snap -1 2021-07-31T01:46:35.497 INFO:tasks.rados.rados.0.smithi045.stdout:284: expect (ObjNum 71 snap 10 seq_num 71) 2021-07-31T01:46:35.497 INFO:tasks.rados.rados.0.smithi045.stdout:285: read oid 10 snap -1 2021-07-31T01:46:35.497 INFO:tasks.rados.rados.0.smithi045.stdout:285: expect (ObjNum 97 snap 16 seq_num 97) 2021-07-31T01:46:35.497 INFO:tasks.rados.rados.0.smithi045.stdout:286: delete oid 49 current snap is 20 2021-07-31T01:46:35.498 INFO:tasks.rados.rados.0.smithi045.stdout:283: done (3 left) 2021-07-31T01:46:35.498 INFO:tasks.rados.rados.0.smithi045.stdout:286: done (2 left) 2021-07-31T01:46:35.499 INFO:tasks.rados.rados.0.smithi045.stdout:287: delete oid 5 current snap is 20 2021-07-31T01:46:35.500 INFO:tasks.rados.rados.0.smithi045.stdout:287: done (2 left) 2021-07-31T01:46:35.500 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 29 current snap is 20 2021-07-31T01:46:35.500 INFO:tasks.rados.rados.0.smithi045.stdout:288: seq_num 107 ranges {761856=598016,1359872=729088,2088960=466944,2555904=8192} 2021-07-31T01:46:35.508 INFO:tasks.rados.rados.0.smithi045.stdout:288: writing smithi04569269-29 from 761856 to 1359872 tid 1 2021-07-31T01:46:35.513 INFO:tasks.rados.rados.0.smithi045.stdout:288: writing smithi04569269-29 from 1359872 to 2088960 tid 2 2021-07-31T01:46:35.516 INFO:tasks.rados.rados.0.smithi045.stdout:288: writing smithi04569269-29 from 2088960 to 2555904 tid 3 2021-07-31T01:46:35.516 INFO:tasks.rados.rados.0.smithi045.stdout:288: writing smithi04569269-29 from 2555904 to 2564096 tid 4 2021-07-31T01:46:35.516 INFO:tasks.rados.rados.0.smithi045.stdout:289: delete oid 40 current snap is 20 2021-07-31T01:46:35.516 INFO:tasks.rados.rados.0.smithi045.stdout:284: expect (ObjNum 71 snap 10 seq_num 71) 2021-07-31T01:46:35.548 INFO:tasks.rados.rados.0.smithi045.stdout:285: expect (ObjNum 97 snap 16 seq_num 97) 2021-07-31T01:46:35.597 INFO:tasks.rados.rados.0.smithi045.stdout:288: finishing write tid 1 to smithi04569269-29 2021-07-31T01:46:35.597 INFO:tasks.rados.rados.0.smithi045.stdout:288: finishing write tid 2 to smithi04569269-29 2021-07-31T01:46:35.597 INFO:tasks.rados.rados.0.smithi045.stdout:288: finishing write tid 3 to smithi04569269-29 2021-07-31T01:46:35.597 INFO:tasks.rados.rados.0.smithi045.stdout:288: finishing write tid 4 to smithi04569269-29 2021-07-31T01:46:35.598 INFO:tasks.rados.rados.0.smithi045.stdout:288: finishing write tid 5 to smithi04569269-29 2021-07-31T01:46:35.598 INFO:tasks.rados.rados.0.smithi045.stdout:288: finishing write tid 6 to smithi04569269-29 2021-07-31T01:46:35.598 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 29 v 19 (ObjNum 107 snap 20 seq_num 107) dirty exists 2021-07-31T01:46:35.598 INFO:tasks.rados.rados.0.smithi045.stdout:288: left oid 29 (ObjNum 107 snap 20 seq_num 107) 2021-07-31T01:46:35.599 INFO:tasks.rados.rados.0.smithi045.stdout:284: done (3 left) 2021-07-31T01:46:35.599 INFO:tasks.rados.rados.0.smithi045.stdout:285: done (2 left) 2021-07-31T01:46:35.599 INFO:tasks.rados.rados.0.smithi045.stdout:288: done (1 left) 2021-07-31T01:46:35.599 INFO:tasks.rados.rados.0.smithi045.stdout:289: done (0 left) 2021-07-31T01:46:35.599 INFO:tasks.rados.rados.0.smithi045.stdout:290: snap_remove snap 16 2021-07-31T01:46:35.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:35 smithi154 conmon[34553]: cephadm 2021-07-31T01:46:34.294375+0000 mgr.y (mgr.14146 2021-07-31T01:46:35.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:35 smithi154 conmon[34553]: ) 293 : cephadm [INF] Upgrade: Target is version 17.0.0-6562-gf25860a2 (unknown) 2021-07-31T01:46:35.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:35 smithi154 conmon[34553]: cephadm 2021-07-31T01:46:34.294627+0000 mgr.y (mgr.14146) 294 : cephadm [INF] Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:58353ac33be8de5bd041245814b4a069acff6ffcb54969da96873a191aef6c81, digests ['quay.ceph.io/ceph-ci/ceph@sha256:58353ac33be8de5bd041245814b4a069acff6ffcb54969da96873a191aef6c81'] 2021-07-31T01:46:35.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:35 smithi154 conmon[34553]: cephadm 2021-07-31T01:46:34.296487+0000 mgr.y (mgr.14146) 295 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.y) 2021-07-31T01:46:35.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:35 smithi154 conmon[34553]: cluster 2021-07-31T01:46:34.488013+0000 mon.a (mon.0) 695 : cluster [DBG] osdmap e95: 8 total, 8 up, 8 in 2021-07-31T01:46:35.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:35 smithi154 conmon[34553]: cephadm 2021-07-31T01:46:34.655084+0000 mgr.y (mgr.14146) 296 : cephadm [INF] Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:58353ac33be8de5bd041245814b4a069acff6ffcb54969da96873a191aef6c81 on smithi154 2021-07-31T01:46:35.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:35 smithi045 conmon[39718]: cephadm 2021-07-31T01:46:34.294375+0000 mgr.y ( 2021-07-31T01:46:35.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:35 smithi045 conmon[39718]: mgr.14146) 293 : cephadm [INF] Upgrade: Target is version 17.0.0-6562-gf25860a2 (unknown) 2021-07-31T01:46:35.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:35 smithi045 conmon[39718]: cephadm 2021-07-31T01:46:34.294627+0000 mgr.y (mgr.14146) 294 : cephadm [INF] 2021-07-31T01:46:35.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:35 smithi045 conmon[39718]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:58353ac33be8de5bd041245814b4a069acff6ffcb54969da96873a191aef6c81, digests ['quay.ceph.io/ceph-ci/ceph@sha256:58353ac33be8de5bd041245814b4a069acff6ffcb54969da96873a191aef6c81'] 2021-07-31T01:46:35.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:35 smithi045 conmon[39718]: cephadm 2021-07-31T01:46:34.296487+0000 mgr.y (mgr.14146) 295 : 2021-07-31T01:46:35.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:35 smithi045 conmon[39718]: cephadm [INF] Upgrade: Need to upgrade myself (mgr.y) 2021-07-31T01:46:35.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:35 smithi045 conmon[39718]: cluster 2021-07-31T01:46:34.488013+0000 mon.a 2021-07-31T01:46:35.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:35 smithi045 conmon[39718]: (mon.0) 695 : cluster [DBG] osdmap e95: 8 total, 8 up, 8 in 2021-07-31T01:46:35.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:35 smithi045 conmon[39718]: cephadm 2021-07-31T01: 2021-07-31T01:46:35.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:35 smithi045 conmon[39718]: 46:34.655084+0000 mgr.y (mgr.14146) 296 : cephadm [INF] Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:58353ac33be8de5bd041245814b4a069acff6ffcb54969da96873a191aef6c81 on smithi154 2021-07-31T01:46:35.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:35 smithi045 conmon[33655]: cephadm 2021-07-31T01:46:34.294375+0000 mgr.y (mgr.14146) 293 : cephadm [INF] Upgrade: Target is version 17.0.0-6562-gf25860a2 (unknown) 2021-07-31T01:46:35.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:35 smithi045 conmon[33655]: cephadm 2021-07-31T01:46:34.294627+0000 mgr.y (mgr.14146) 294 : cephadm [INF] Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:58353ac33be8de5bd041245814b4a069acff6ffcb54969da96873a191aef6c81, digests ['quay.ceph.io/ceph-ci/ceph@sha256:58353ac33be8de5bd041245814b4a069acff6ffcb54969da96873a191aef6c81'] 2021-07-31T01:46:35.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:35 smithi045 conmon[33655]: cephadm 2021-07-31T01:46:34.296487+0000 mgr.y (mgr.14146) 295 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.y) 2021-07-31T01:46:35.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:35 smithi045 conmon[33655]: cluster 2021-07-31T01:46:34.488013+0000 mon.a (mon.0) 695 : cluster [DBG] osdmap e95: 8 total, 8 up, 8 in 2021-07-31T01:46:35.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:35 smithi045 conmon[33655]: cephadm 2021-07-31T01:46:34.655084+0000 mgr.y (mgr.14146) 296 : cephadm [INF] Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:58353ac33be8de5bd041245814b4a069acff6ffcb54969da96873a191aef6c81 on smithi154 2021-07-31T01:46:36.475 INFO:tasks.rados.rados.0.smithi045.stdout:290: done (0 left) 2021-07-31T01:46:36.475 INFO:tasks.rados.rados.0.smithi045.stdout:291: delete oid 38 current snap is 20 2021-07-31T01:46:36.482 INFO:tasks.rados.rados.0.smithi045.stdout:291: done (0 left) 2021-07-31T01:46:36.482 INFO:tasks.rados.rados.0.smithi045.stdout:292: copy_from oid 36 from oid 46 current snap is 20 2021-07-31T01:46:36.482 INFO:tasks.rados.rados.0.smithi045.stdout:293: rollback oid 15 current snap is 20 2021-07-31T01:46:36.482 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 15 to 17 2021-07-31T01:46:36.483 INFO:tasks.rados.rados.0.smithi045.stdout:294: setattr oid 14 current snap is 20 2021-07-31T01:46:36.483 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 50 current snap is 20 2021-07-31T01:46:36.483 INFO:tasks.rados.rados.0.smithi045.stdout:295: seq_num 110 ranges {827392=647168,1474560=778240,2252800=425984} 2021-07-31T01:46:36.490 INFO:tasks.rados.rados.0.smithi045.stdout:295: writing smithi04569269-50 from 827392 to 1474560 tid 1 2021-07-31T01:46:36.497 INFO:tasks.rados.rados.0.smithi045.stdout:295: writing smithi04569269-50 from 1474560 to 2252800 tid 2 2021-07-31T01:46:36.499 INFO:tasks.rados.rados.0.smithi045.stdout:295: writing smithi04569269-50 from 2252800 to 2678784 tid 3 2021-07-31T01:46:36.499 INFO:tasks.rados.rados.0.smithi045.stdout:296: read oid 12 snap 19 2021-07-31T01:46:36.500 INFO:tasks.rados.rados.0.smithi045.stdout:296: expect deleted 2021-07-31T01:46:36.500 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 20 current snap is 20 2021-07-31T01:46:36.500 INFO:tasks.rados.rados.0.smithi045.stdout:293: finishing rollback tid 1 to smithi04569269-15 2021-07-31T01:46:36.501 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 15 v 0 (ObjNum 1 snap 32731 seq_num 3189247136) dirty dne 2021-07-31T01:46:36.501 INFO:tasks.rados.rados.0.smithi045.stdout:297: seq_num 111 ranges {1302528=663552,1966080=663552,2629632=450560,3080192=90112} 2021-07-31T01:46:36.508 INFO:tasks.rados.rados.0.smithi045.stdout:297: writing smithi04569269-20 from 1302528 to 1966080 tid 1 2021-07-31T01:46:36.512 INFO:tasks.rados.rados.0.smithi045.stdout:297: writing smithi04569269-20 from 1966080 to 2629632 tid 2 2021-07-31T01:46:36.514 INFO:tasks.rados.rados.0.smithi045.stdout:297: writing smithi04569269-20 from 2629632 to 3080192 tid 3 2021-07-31T01:46:36.515 INFO:tasks.rados.rados.0.smithi045.stdout:297: writing smithi04569269-20 from 3080192 to 3170304 tid 4 2021-07-31T01:46:36.516 INFO:tasks.rados.rados.0.smithi045.stdout:293: done (5 left) 2021-07-31T01:46:36.516 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 33 current snap is 20 2021-07-31T01:46:36.516 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 14 v 64 (ObjNum 98 snap 16 seq_num 98) dirty exists 2021-07-31T01:46:36.516 INFO:tasks.rados.rados.0.smithi045.stdout:292: finishing copy_from to smithi04569269-36 2021-07-31T01:46:36.517 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 36 v 2 (ObjNum 45 snap 0 seq_num 45) dirty exists 2021-07-31T01:46:36.517 INFO:tasks.rados.rados.0.smithi045.stdout:292: finishing copy_from racing read to smithi04569269-36 2021-07-31T01:46:36.517 INFO:tasks.rados.rados.0.smithi045.stdout:295: finishing write tid 1 to smithi04569269-50 2021-07-31T01:46:36.517 INFO:tasks.rados.rados.0.smithi045.stdout:298: seq_num 112 ranges {2293760=679936,2973696=548864,3522560=655360} 2021-07-31T01:46:36.532 INFO:tasks.rados.rados.0.smithi045.stdout:298: writing smithi04569269-33 from 2293760 to 2973696 tid 1 2021-07-31T01:46:36.535 INFO:tasks.rados.rados.0.smithi045.stdout:298: writing smithi04569269-33 from 2973696 to 3522560 tid 2 2021-07-31T01:46:36.539 INFO:tasks.rados.rados.0.smithi045.stdout:298: writing smithi04569269-33 from 3522560 to 4177920 tid 3 2021-07-31T01:46:36.539 INFO:tasks.rados.rados.0.smithi045.stdout:292: done (5 left) 2021-07-31T01:46:36.540 INFO:tasks.rados.rados.0.smithi045.stdout:294: done (4 left) 2021-07-31T01:46:36.540 INFO:tasks.rados.rados.0.smithi045.stdout:299: rollback oid 9 current snap is 20 2021-07-31T01:46:36.540 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 9 to 18 2021-07-31T01:46:36.540 INFO:tasks.rados.rados.0.smithi045.stdout:295: finishing write tid 2 to smithi04569269-50 2021-07-31T01:46:36.541 INFO:tasks.rados.rados.0.smithi045.stdout:295: finishing write tid 3 to smithi04569269-50 2021-07-31T01:46:36.541 INFO:tasks.rados.rados.0.smithi045.stdout:295: finishing write tid 4 to smithi04569269-50 2021-07-31T01:46:36.541 INFO:tasks.rados.rados.0.smithi045.stdout:295: finishing write tid 5 to smithi04569269-50 2021-07-31T01:46:36.541 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 50 v 93 (ObjNum 110 snap 20 seq_num 110) dirty exists 2021-07-31T01:46:36.541 INFO:tasks.rados.rados.0.smithi045.stdout:295: left oid 50 (ObjNum 110 snap 20 seq_num 110) 2021-07-31T01:46:36.542 INFO:tasks.rados.rados.0.smithi045.stdout:297: finishing write tid 1 to smithi04569269-20 2021-07-31T01:46:36.542 INFO:tasks.rados.rados.0.smithi045.stdout:297: finishing write tid 2 to smithi04569269-20 2021-07-31T01:46:36.542 INFO:tasks.rados.rados.0.smithi045.stdout:297: finishing write tid 3 to smithi04569269-20 2021-07-31T01:46:36.542 INFO:tasks.rados.rados.0.smithi045.stdout:297: finishing write tid 4 to smithi04569269-20 2021-07-31T01:46:36.542 INFO:tasks.rados.rados.0.smithi045.stdout:297: finishing write tid 5 to smithi04569269-20 2021-07-31T01:46:36.543 INFO:tasks.rados.rados.0.smithi045.stdout:297: finishing write tid 6 to smithi04569269-20 2021-07-31T01:46:36.543 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 20 v 150 (ObjNum 111 snap 20 seq_num 111) dirty exists 2021-07-31T01:46:36.543 INFO:tasks.rados.rados.0.smithi045.stdout:297: left oid 20 (ObjNum 111 snap 20 seq_num 111) 2021-07-31T01:46:36.543 INFO:tasks.rados.rados.0.smithi045.stdout:298: finishing write tid 1 to smithi04569269-33 2021-07-31T01:46:36.544 INFO:tasks.rados.rados.0.smithi045.stdout:295: done (4 left) 2021-07-31T01:46:36.544 INFO:tasks.rados.rados.0.smithi045.stdout:296: done (3 left) 2021-07-31T01:46:36.545 INFO:tasks.rados.rados.0.smithi045.stdout:297: done (2 left) 2021-07-31T01:46:36.545 INFO:tasks.rados.rados.0.smithi045.stdout:300: snap_remove snap 17 2021-07-31T01:46:36.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:36 smithi154 conmon[34553]: cluster 2021-07-31T01:46:35.370365+0000 mgr.y (mgr.14146) 297 : cluster [DBG] pgmap v293: 25 pgs: 1 active+clean+snaptrim, 24 active+clean; 61 MiB data, 162 MiB used, 715 GiB / 715 GiB avail; 4.6 MiB/s rd, 5.8 MiB/s wr, 35 op/s 2021-07-31T01:46:36.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:36 smithi154 conmon[34553]: cluster 2021-07-31T01:46:35.493213+0000 mon.a (mon.0) 696 : cluster [DBG] osdmap e96: 8 total, 8 up, 8 in 2021-07-31T01:46:36.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:36 smithi154 conmon[34553]: cluster 2021-07-31T01:46:36.474301+0000 mon.a (mon.0) 697 : cluster [DBG] osdmap e97: 8 total, 8 up, 8 in 2021-07-31T01:46:36.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:36 smithi045 conmon[39718]: cluster 2021-07-31T01:46:35.370365+0000 2021-07-31T01:46:36.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:36 smithi045 conmon[39718]: mgr.y (mgr.14146) 297 : cluster [DBG] pgmap v293: 25 pgs: 1 active+clean+snaptrim, 24 active+clean; 61 MiB data, 162 MiB used, 715 GiB / 715 GiB avail; 4.6 MiB/s rd, 5.8 MiB/s wr, 35 op/s 2021-07-31T01:46:36.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:36 smithi045 conmon[39718]: cluster 2021-07-31T01: 2021-07-31T01:46:36.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:36 smithi045 conmon[39718]: 46:35.493213+0000 mon.a (mon.0) 696 : cluster [DBG] osdmap e96: 8 total, 8 up, 8 in 2021-07-31T01:46:36.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:36 smithi045 conmon[39718]: cluster 2021-07-31T01:46: 2021-07-31T01:46:36.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:36 smithi045 conmon[39718]: 36.474301+0000 mon.a (mon.0) 697 : cluster 2021-07-31T01:46:36.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:36 smithi045 conmon[39718]: [DBG] osdmap e97: 8 total, 8 up, 8 in 2021-07-31T01:46:36.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:36 smithi045 conmon[33655]: cluster 2021-07-31T01:46:36.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:36 smithi045 conmon[33655]: 2021-07-31T01:46:36.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:36 smithi045 conmon[33655]: 2021 2021-07-31T01:46:36.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:36 smithi045 conmon[33655]: - 2021-07-31T01:46:36.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:36 smithi045 conmon[33655]: 07-31T01:46:35.370365+0000 2021-07-31T01:46:36.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:36 smithi045 conmon[33655]: mgr.y (mgr.14146) 297 : cluster [DBG] pgmap v293: 25 pgs: 1 active+clean+snaptrim, 24 active+clean; 61 MiB data, 162 MiB used, 715 GiB / 715 GiB avail; 4.6 MiB/s rd, 5.8 MiB/s wr, 35 op/s 2021-07-31T01:46:36.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:36 smithi045 conmon[33655]: cluster 2021-07-31 2021-07-31T01:46:36.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:36 smithi045 conmon[33655]: T01:46:35.493213+0000 mon.a ( 2021-07-31T01:46:36.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:36 smithi045 conmon[33655]: mon.0) 696 : cluster [DBG] osdmap e96: 8 total, 8 up, 8 in 2021-07-31T01:46:36.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:36 smithi045 conmon[33655]: cluster 2021-07 2021-07-31T01:46:36.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:36 smithi045 conmon[33655]: -31T01:46:36.474301+0000 mon.a (mon 2021-07-31T01:46:36.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:36 smithi045 conmon[33655]: .0) 697 : cluster [DBG] osdmap e97: 8 total, 8 up, 8 in 2021-07-31T01:46:37.500 INFO:tasks.rados.rados.0.smithi045.stdout:300: done (2 left) 2021-07-31T01:46:37.500 INFO:tasks.rados.rados.0.smithi045.stdout:301: rollback oid 24 current snap is 20 2021-07-31T01:46:37.500 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 24 to 18 2021-07-31T01:46:37.501 INFO:tasks.rados.rados.0.smithi045.stdout:298: finishing write tid 2 to smithi04569269-33 2021-07-31T01:46:37.501 INFO:tasks.rados.rados.0.smithi045.stdout:298: finishing write tid 3 to smithi04569269-33 2021-07-31T01:46:37.501 INFO:tasks.rados.rados.0.smithi045.stdout:298: finishing write tid 4 to smithi04569269-33 2021-07-31T01:46:37.501 INFO:tasks.rados.rados.0.smithi045.stdout:299: finishing rollback tid 0 to smithi04569269-9 2021-07-31T01:46:37.502 INFO:tasks.rados.rados.0.smithi045.stdout:299: finishing rollback tid 1 to smithi04569269-9 2021-07-31T01:46:37.502 INFO:tasks.rados.rados.0.smithi045.stdout:299: finishing rollback tid 2 to smithi04569269-9 2021-07-31T01:46:37.502 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 9 v 47 (ObjNum 8 snap 0 seq_num 8) dirty exists 2021-07-31T01:46:37.502 INFO:tasks.rados.rados.0.smithi045.stdout:298: finishing write tid 5 to smithi04569269-33 2021-07-31T01:46:37.503 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 33 v 71 (ObjNum 112 snap 20 seq_num 112) dirty exists 2021-07-31T01:46:37.503 INFO:tasks.rados.rados.0.smithi045.stdout:298: left oid 33 (ObjNum 112 snap 20 seq_num 112) 2021-07-31T01:46:37.503 INFO:tasks.rados.rados.0.smithi045.stdout:298: done (2 left) 2021-07-31T01:46:37.503 INFO:tasks.rados.rados.0.smithi045.stdout:299: done (1 left) 2021-07-31T01:46:37.503 INFO:tasks.rados.rados.0.smithi045.stdout:302: snap_create 2021-07-31T01:46:37.505 INFO:tasks.rados.rados.0.smithi045.stdout:301: finishing rollback tid 0 to smithi04569269-24 2021-07-31T01:46:37.506 INFO:tasks.rados.rados.0.smithi045.stdout:301: finishing rollback tid 1 to smithi04569269-24 2021-07-31T01:46:37.507 INFO:tasks.rados.rados.0.smithi045.stdout:301: finishing rollback tid 2 to smithi04569269-24 2021-07-31T01:46:37.507 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 24 v 42 (ObjNum 89 snap 15 seq_num 89) dirty exists 2021-07-31T01:46:38.501 INFO:tasks.rados.rados.0.smithi045.stdout:301: done (1 left) 2021-07-31T01:46:38.501 INFO:tasks.rados.rados.0.smithi045.stdout:302: done (0 left) 2021-07-31T01:46:38.501 INFO:tasks.rados.rados.0.smithi045.stdout:303: read oid 23 snap -1 2021-07-31T01:46:38.501 INFO:tasks.rados.rados.0.smithi045.stdout:303: expect (ObjNum 53 snap 1 seq_num 53) 2021-07-31T01:46:38.502 INFO:tasks.rados.rados.0.smithi045.stdout:304: read oid 38 snap 19 2021-07-31T01:46:38.502 INFO:tasks.rados.rados.0.smithi045.stdout:304: expect (ObjNum 90 snap 15 seq_num 90) 2021-07-31T01:46:38.502 INFO:tasks.rados.rados.0.smithi045.stdout:305: setattr oid 44 current snap is 21 2021-07-31T01:46:38.503 INFO:tasks.rados.rados.0.smithi045.stdout:306: read oid 33 snap -1 2021-07-31T01:46:38.503 INFO:tasks.rados.rados.0.smithi045.stdout:306: expect (ObjNum 112 snap 20 seq_num 112) 2021-07-31T01:46:38.503 INFO:tasks.rados.rados.0.smithi045.stdout:307: read oid 20 snap 18 2021-07-31T01:46:38.504 INFO:tasks.rados.rados.0.smithi045.stdout:307: expect (ObjNum 54 snap 4 seq_num 54) 2021-07-31T01:46:38.504 INFO:tasks.rados.rados.0.smithi045.stdout:308: delete oid 49 current snap is 21 2021-07-31T01:46:38.510 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 44 v 67 (ObjNum 0 snap 21873 seq_num 3189332928) dirty exists 2021-07-31T01:46:38.512 INFO:tasks.rados.rados.0.smithi045.stdout:307: expect (ObjNum 54 snap 4 seq_num 54) 2021-07-31T01:46:38.554 INFO:tasks.rados.rados.0.smithi045.stdout:305: done (5 left) 2021-07-31T01:46:38.554 INFO:tasks.rados.rados.0.smithi045.stdout:307: done (4 left) 2021-07-31T01:46:38.554 INFO:tasks.rados.rados.0.smithi045.stdout:308: done (3 left) 2021-07-31T01:46:38.554 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 44 current snap is 21 2021-07-31T01:46:38.555 INFO:tasks.rados.rados.0.smithi045.stdout:303: expect (ObjNum 53 snap 1 seq_num 53) 2021-07-31T01:46:38.635 INFO:tasks.rados.rados.0.smithi045.stdout:304: expect (ObjNum 90 snap 15 seq_num 90) 2021-07-31T01:46:38.706 INFO:tasks.rados.rados.0.smithi045.stdout:306: expect (ObjNum 112 snap 20 seq_num 112) 2021-07-31T01:46:38.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:38 smithi045 conmon[33655]: cluster 2021-07-31T01:46:38.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:38 smithi045 conmon[33655]: 2021-07-31T01:46:37.370903+0000 mgr.y (mgr.14146) 298 : cluster [DBG] pgmap v296: 25 pgs: 1 active+clean+snaptrim, 24 active+clean; 62 MiB data, 168 MiB used, 715 GiB / 715 GiB avail; 877 KiB/s rd, 2.4 MiB/s wr, 9 op/s 2021-07-31T01:46:38.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:38 smithi045 conmon[33655]: cluster 2021-07-31T01 2021-07-31T01:46:38.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:38 smithi045 conmon[33655]: :46:37.498926+0000 mon.a (mon.0) 698 : cluster [DBG] osdmap e98: 8 total, 8 up, 8 in 2021-07-31T01:46:38.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:38 smithi045 conmon[39718]: cluster 2021-07-31T01:46:37.370903+0000 mgr.y ( 2021-07-31T01:46:38.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:38 smithi045 conmon[39718]: mgr.14146) 298 : cluster [DBG] pgmap v296: 25 pgs: 1 active+clean+snaptrim, 24 active+clean; 62 MiB data, 168 MiB used, 715 GiB / 715 GiB avail; 877 KiB/s rd, 2.4 MiB/s wr, 9 op/s 2021-07-31T01:46:38.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:38 smithi045 conmon[39718]: cluster 2021-07-31T01:46:37 2021-07-31T01:46:38.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:38 smithi045 conmon[39718]: .498926+0000 mon.a (mon.0) 698 : cluster [DBG] 2021-07-31T01:46:38.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:38 smithi045 conmon[39718]: osdmap e98: 8 total, 8 up, 8 in 2021-07-31T01:46:38.859 INFO:tasks.rados.rados.0.smithi045.stdout:309: seq_num 114 ranges {0=712704,712704=712704,1425408=491520} 2021-07-31T01:46:38.862 INFO:tasks.rados.rados.0.smithi045.stdout:309: writing smithi04569269-44 from 0 to 712704 tid 1 2021-07-31T01:46:38.866 INFO:tasks.rados.rados.0.smithi045.stdout:309: writing smithi04569269-44 from 712704 to 1425408 tid 2 2021-07-31T01:46:38.869 INFO:tasks.rados.rados.0.smithi045.stdout:309: writing smithi04569269-44 from 1425408 to 1916928 tid 3 2021-07-31T01:46:38.870 INFO:tasks.rados.rados.0.smithi045.stdout:303: done (3 left) 2021-07-31T01:46:38.870 INFO:tasks.rados.rados.0.smithi045.stdout:304: done (2 left) 2021-07-31T01:46:38.870 INFO:tasks.rados.rados.0.smithi045.stdout:306: done (1 left) 2021-07-31T01:46:38.871 INFO:tasks.rados.rados.0.smithi045.stdout:310: delete oid 50 current snap is 21 2021-07-31T01:46:38.871 INFO:tasks.rados.rados.0.smithi045.stdout:309: finishing write tid 1 to smithi04569269-44 2021-07-31T01:46:38.871 INFO:tasks.rados.rados.0.smithi045.stdout:309: finishing write tid 2 to smithi04569269-44 2021-07-31T01:46:38.872 INFO:tasks.rados.rados.0.smithi045.stdout:310: done (1 left) 2021-07-31T01:46:38.872 INFO:tasks.rados.rados.0.smithi045.stdout:311: copy_from oid 28 from oid 45 current snap is 21 2021-07-31T01:46:38.873 INFO:tasks.rados.rados.0.smithi045.stdout:312: read oid 35 snap -1 2021-07-31T01:46:38.873 INFO:tasks.rados.rados.0.smithi045.stdout:312: expect (ObjNum 55 snap 4 seq_num 55) 2021-07-31T01:46:38.873 INFO:tasks.rados.rados.0.smithi045.stdout:313: read oid 22 snap -1 2021-07-31T01:46:38.873 INFO:tasks.rados.rados.0.smithi045.stdout:313: expect (ObjNum 47 snap 0 seq_num 47) 2021-07-31T01:46:38.874 INFO:tasks.rados.rados.0.smithi045.stdout:314: setattr oid 25 current snap is 21 2021-07-31T01:46:38.874 INFO:tasks.rados.rados.0.smithi045.stdout:309: finishing write tid 3 to smithi04569269-44 2021-07-31T01:46:38.875 INFO:tasks.rados.rados.0.smithi045.stdout:309: finishing write tid 4 to smithi04569269-44 2021-07-31T01:46:38.875 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 23 current snap is 21 2021-07-31T01:46:38.875 INFO:tasks.rados.rados.0.smithi045.stdout:315: seq_num 116 ranges {2113536=745472,2859008=532480,3391488=638976,4030464=32768} 2021-07-31T01:46:38.890 INFO:tasks.rados.rados.0.smithi045.stdout:315: writing smithi04569269-23 from 2113536 to 2859008 tid 1 2021-07-31T01:46:38.893 INFO:tasks.rados.rados.0.smithi045.stdout:315: writing smithi04569269-23 from 2859008 to 3391488 tid 2 2021-07-31T01:46:38.897 INFO:tasks.rados.rados.0.smithi045.stdout:315: writing smithi04569269-23 from 3391488 to 4030464 tid 3 2021-07-31T01:46:38.898 INFO:tasks.rados.rados.0.smithi045.stdout:315: writing smithi04569269-23 from 4030464 to 4063232 tid 4 2021-07-31T01:46:38.898 INFO:tasks.rados.rados.0.smithi045.stdout:316: read oid 49 snap -1 2021-07-31T01:46:38.899 INFO:tasks.rados.rados.0.smithi045.stdout:316: expect deleted 2021-07-31T01:46:38.899 INFO:tasks.rados.rados.0.smithi045.stdout:317: rollback oid 32 current snap is 21 2021-07-31T01:46:38.899 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 32 to 19 2021-07-31T01:46:38.900 INFO:tasks.rados.rados.0.smithi045.stdout:313: expect (ObjNum 47 snap 0 seq_num 47) 2021-07-31T01:46:38.920 INFO:tasks.rados.rados.0.smithi045.stdout:309: finishing write tid 5 to smithi04569269-44 2021-07-31T01:46:38.920 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 44 v 71 (ObjNum 114 snap 21 seq_num 114) dirty exists 2021-07-31T01:46:38.920 INFO:tasks.rados.rados.0.smithi045.stdout:309: left oid 44 (ObjNum 114 snap 21 seq_num 114) 2021-07-31T01:46:38.921 INFO:tasks.rados.rados.0.smithi045.stdout:312: expect (ObjNum 55 snap 4 seq_num 55) 2021-07-31T01:46:38.978 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 25 v 12 (ObjNum 36 snap 0 seq_num 36) dirty exists 2021-07-31T01:46:38.979 INFO:tasks.rados.rados.0.smithi045.stdout:311: finishing copy_from to smithi04569269-28 2021-07-31T01:46:38.979 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 28 v 80 (ObjNum 88 snap 15 seq_num 88) dirty exists 2021-07-31T01:46:38.979 INFO:tasks.rados.rados.0.smithi045.stdout:311: finishing copy_from racing read to smithi04569269-28 2021-07-31T01:46:38.979 INFO:tasks.rados.rados.0.smithi045.stdout:315: finishing write tid 1 to smithi04569269-23 2021-07-31T01:46:38.980 INFO:tasks.rados.rados.0.smithi045.stdout:309: done (7 left) 2021-07-31T01:46:38.980 INFO:tasks.rados.rados.0.smithi045.stdout:311: done (6 left) 2021-07-31T01:46:38.980 INFO:tasks.rados.rados.0.smithi045.stdout:312: done (5 left) 2021-07-31T01:46:38.980 INFO:tasks.rados.rados.0.smithi045.stdout:313: done (4 left) 2021-07-31T01:46:38.981 INFO:tasks.rados.rados.0.smithi045.stdout:314: done (3 left) 2021-07-31T01:46:38.981 INFO:tasks.rados.rados.0.smithi045.stdout:318: snap_create 2021-07-31T01:46:38.981 INFO:tasks.rados.rados.0.smithi045.stdout:315: finishing write tid 2 to smithi04569269-23 2021-07-31T01:46:38.981 INFO:tasks.rados.rados.0.smithi045.stdout:317: finishing rollback tid 0 to smithi04569269-32 2021-07-31T01:46:38.981 INFO:tasks.rados.rados.0.smithi045.stdout:317: finishing rollback tid 1 to smithi04569269-32 2021-07-31T01:46:38.982 INFO:tasks.rados.rados.0.smithi045.stdout:317: finishing rollback tid 2 to smithi04569269-32 2021-07-31T01:46:38.982 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 32 v 43 (ObjNum 95 snap 16 seq_num 95) dirty exists 2021-07-31T01:46:38.982 INFO:tasks.rados.rados.0.smithi045.stdout:315: finishing write tid 3 to smithi04569269-23 2021-07-31T01:46:38.982 INFO:tasks.rados.rados.0.smithi045.stdout:315: finishing write tid 4 to smithi04569269-23 2021-07-31T01:46:38.983 INFO:tasks.rados.rados.0.smithi045.stdout:315: finishing write tid 5 to smithi04569269-23 2021-07-31T01:46:38.983 INFO:tasks.rados.rados.0.smithi045.stdout:315: finishing write tid 6 to smithi04569269-23 2021-07-31T01:46:38.983 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 23 v 81 (ObjNum 116 snap 21 seq_num 116) dirty exists 2021-07-31T01:46:38.983 INFO:tasks.rados.rados.0.smithi045.stdout:315: left oid 23 (ObjNum 116 snap 21 seq_num 116) 2021-07-31T01:46:39.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:38 smithi154 conmon[34553]: cluster 2021-07-31T01:46:37.370903+0000 mgr.y (mgr.14146) 298 : cluster [DBG] pgmap v296: 25 pgs: 1 active+clean+snaptrim, 24 active+clean; 62 MiB data, 168 MiB used, 715 GiB / 715 GiB avail; 877 KiB/s rd, 2.4 MiB/s wr, 9 op/s 2021-07-31T01:46:39.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:38 smithi154 conmon[34553]: 2021-07-31T01:46:39.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:38 smithi154 conmon[34553]: cluster 2021-07-31T01:46:37.498926+0000 mon.a (mon.0) 698 : cluster [DBG] osdmap e98: 8 total, 8 up, 8 in 2021-07-31T01:46:39.512 INFO:tasks.rados.rados.0.smithi045.stdout:315: done (3 left) 2021-07-31T01:46:39.513 INFO:tasks.rados.rados.0.smithi045.stdout:316: done (2 left) 2021-07-31T01:46:39.513 INFO:tasks.rados.rados.0.smithi045.stdout:317: done (1 left) 2021-07-31T01:46:39.513 INFO:tasks.rados.rados.0.smithi045.stdout:318: done (0 left) 2021-07-31T01:46:39.513 INFO:tasks.rados.rados.0.smithi045.stdout:319: rollback oid 50 current snap is 22 2021-07-31T01:46:39.513 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 50 to 18 2021-07-31T01:46:39.514 INFO:tasks.rados.rados.0.smithi045.stdout:320: rmattr oid 38 current snap is 22 2021-07-31T01:46:39.514 INFO:tasks.rados.rados.0.smithi045.stdout:320: done (1 left) 2021-07-31T01:46:39.514 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 21 current snap is 22 2021-07-31T01:46:39.515 INFO:tasks.rados.rados.0.smithi045.stdout:321: seq_num 117 ranges {1302528=770048,2072576=417792,2490368=442368,2932736=344064} 2021-07-31T01:46:39.523 INFO:tasks.rados.rados.0.smithi045.stdout:321: writing smithi04569269-21 from 1302528 to 2072576 tid 1 2021-07-31T01:46:39.526 INFO:tasks.rados.rados.0.smithi045.stdout:321: writing smithi04569269-21 from 2072576 to 2490368 tid 2 2021-07-31T01:46:39.529 INFO:tasks.rados.rados.0.smithi045.stdout:321: writing smithi04569269-21 from 2490368 to 2932736 tid 3 2021-07-31T01:46:39.531 INFO:tasks.rados.rados.0.smithi045.stdout:321: writing smithi04569269-21 from 2932736 to 3276800 tid 4 2021-07-31T01:46:39.532 INFO:tasks.rados.rados.0.smithi045.stdout:322: copy_from oid 37 from oid 14 current snap is 22 2021-07-31T01:46:39.532 INFO:tasks.rados.rados.0.smithi045.stdout:319: finishing rollback tid 1 to smithi04569269-50 2021-07-31T01:46:39.532 INFO:tasks.rados.rados.0.smithi045.stdout:319: finishing rollback tid 2 to smithi04569269-50 2021-07-31T01:46:39.532 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 50 v 101 (ObjNum 49 snap 0 seq_num 49) dirty exists 2021-07-31T01:46:39.533 INFO:tasks.rados.rados.0.smithi045.stdout:321: finishing write tid 1 to smithi04569269-21 2021-07-31T01:46:39.533 INFO:tasks.rados.rados.0.smithi045.stdout:321: finishing write tid 2 to smithi04569269-21 2021-07-31T01:46:39.533 INFO:tasks.rados.rados.0.smithi045.stdout:321: finishing write tid 3 to smithi04569269-21 2021-07-31T01:46:39.533 INFO:tasks.rados.rados.0.smithi045.stdout:319: done (2 left) 2021-07-31T01:46:39.534 INFO:tasks.rados.rados.0.smithi045.stdout:323: rmattr oid 44 current snap is 22 2021-07-31T01:46:39.534 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 12 current snap is 22 2021-07-31T01:46:39.535 INFO:tasks.rados.rados.0.smithi045.stdout:324: seq_num 118 ranges {0=786432,786432=704512,1490944=499712} 2021-07-31T01:46:39.536 INFO:tasks.rados.rados.0.smithi045.stdout:324: writing smithi04569269-12 from 0 to 786432 tid 1 2021-07-31T01:46:39.540 INFO:tasks.rados.rados.0.smithi045.stdout:324: writing smithi04569269-12 from 786432 to 1490944 tid 2 2021-07-31T01:46:39.544 INFO:tasks.rados.rados.0.smithi045.stdout:324: writing smithi04569269-12 from 1490944 to 1990656 tid 3 2021-07-31T01:46:39.544 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 38 current snap is 22 2021-07-31T01:46:39.545 INFO:tasks.rados.rados.0.smithi045.stdout:321: finishing write tid 4 to smithi04569269-21 2021-07-31T01:46:39.545 INFO:tasks.rados.rados.0.smithi045.stdout:325: seq_num 119 ranges {0=802816,802816=589824,1392640=409600,1802240=204800} 2021-07-31T01:46:39.548 INFO:tasks.rados.rados.0.smithi045.stdout:325: writing smithi04569269-38 from 0 to 802816 tid 1 2021-07-31T01:46:39.552 INFO:tasks.rados.rados.0.smithi045.stdout:325: writing smithi04569269-38 from 802816 to 1392640 tid 2 2021-07-31T01:46:39.554 INFO:tasks.rados.rados.0.smithi045.stdout:325: writing smithi04569269-38 from 1392640 to 1802240 tid 3 2021-07-31T01:46:39.556 INFO:tasks.rados.rados.0.smithi045.stdout:325: writing smithi04569269-38 from 1802240 to 2007040 tid 4 2021-07-31T01:46:39.556 INFO:tasks.rados.rados.0.smithi045.stdout:326: copy_from oid 45 from oid 8 current snap is 22 2021-07-31T01:46:39.557 INFO:tasks.rados.rados.0.smithi045.stdout:321: finishing write tid 5 to smithi04569269-21 2021-07-31T01:46:39.557 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 44 v 73 (ObjNum 114 snap 21 seq_num 114) dirty exists 2021-07-31T01:46:39.557 INFO:tasks.rados.rados.0.smithi045.stdout:323: done (5 left) 2021-07-31T01:46:39.557 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 34 current snap is 22 2021-07-31T01:46:39.557 INFO:tasks.rados.rados.0.smithi045.stdout:321: finishing write tid 6 to smithi04569269-21 2021-07-31T01:46:39.558 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 21 v 163 (ObjNum 117 snap 22 seq_num 117) dirty exists 2021-07-31T01:46:39.558 INFO:tasks.rados.rados.0.smithi045.stdout:321: left oid 21 (ObjNum 117 snap 22 seq_num 117) 2021-07-31T01:46:39.558 INFO:tasks.rados.rados.0.smithi045.stdout:324: finishing write tid 1 to smithi04569269-12 2021-07-31T01:46:39.558 INFO:tasks.rados.rados.0.smithi045.stdout:324: finishing write tid 2 to smithi04569269-12 2021-07-31T01:46:39.559 INFO:tasks.rados.rados.0.smithi045.stdout:324: finishing write tid 3 to smithi04569269-12 2021-07-31T01:46:39.559 INFO:tasks.rados.rados.0.smithi045.stdout:324: finishing write tid 4 to smithi04569269-12 2021-07-31T01:46:39.559 INFO:tasks.rados.rados.0.smithi045.stdout:322: finishing copy_from to smithi04569269-37 2021-07-31T01:46:39.559 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 37 v 64 (ObjNum 98 snap 16 seq_num 98) dirty exists 2021-07-31T01:46:39.559 INFO:tasks.rados.rados.0.smithi045.stdout:322: finishing copy_from racing read to smithi04569269-37 2021-07-31T01:46:39.560 INFO:tasks.rados.rados.0.smithi045.stdout:324: finishing write tid 5 to smithi04569269-12 2021-07-31T01:46:39.560 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 12 v 70 (ObjNum 118 snap 22 seq_num 118) dirty exists 2021-07-31T01:46:39.560 INFO:tasks.rados.rados.0.smithi045.stdout:324: left oid 12 (ObjNum 118 snap 22 seq_num 118) 2021-07-31T01:46:39.560 INFO:tasks.rados.rados.0.smithi045.stdout:325: finishing write tid 1 to smithi04569269-38 2021-07-31T01:46:39.561 INFO:tasks.rados.rados.0.smithi045.stdout:325: finishing write tid 2 to smithi04569269-38 2021-07-31T01:46:39.561 INFO:tasks.rados.rados.0.smithi045.stdout:327: seq_num 120 ranges {0=417792,417792=475136,892928=622592,1515520=507904} 2021-07-31T01:46:39.561 INFO:tasks.rados.rados.0.smithi045.stdout:327: writing smithi04569269-34 from 0 to 417792 tid 1 2021-07-31T01:46:39.562 INFO:tasks.rados.rados.0.smithi045.stdout:327: writing smithi04569269-34 from 417792 to 892928 tid 2 2021-07-31T01:46:39.565 INFO:tasks.rados.rados.0.smithi045.stdout:327: writing smithi04569269-34 from 892928 to 1515520 tid 3 2021-07-31T01:46:39.568 INFO:tasks.rados.rados.0.smithi045.stdout:327: writing smithi04569269-34 from 1515520 to 2023424 tid 4 2021-07-31T01:46:39.568 INFO:tasks.rados.rados.0.smithi045.stdout:321: done (5 left) 2021-07-31T01:46:39.568 INFO:tasks.rados.rados.0.smithi045.stdout:322: done (4 left) 2021-07-31T01:46:39.569 INFO:tasks.rados.rados.0.smithi045.stdout:324: done (3 left) 2021-07-31T01:46:39.569 INFO:tasks.rados.rados.0.smithi045.stdout:328: snap_remove snap 18 2021-07-31T01:46:39.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:39 smithi045 conmon[33655]: cluster 2021-07-31T01:46:38.500115+0000 mon.a (mon. 2021-07-31T01:46:39.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:39 smithi045 conmon[33655]: 0) 699 : cluster [DBG] osdmap e99: 8 total, 8 up, 8 in 2021-07-31T01:46:39.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:39 smithi045 conmon[33655]: audit 2021-07-31T01:46:39.435673+0000 mon.a (mon.0) 700 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-07-31T01:46:39.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:39 smithi045 conmon[33655]: audit 2021-07-31T01:46:39.435881+0000 mon.a (mon.0) 701 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:46:39.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:39 smithi045 conmon[39718]: cluster 2021-07-31T01:46:38.500115+0000 mon.a (mon.0) 699 : cluster 2021-07-31T01:46:39.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:39 smithi045 conmon[39718]: [DBG] osdmap e99: 8 total, 8 up, 8 in 2021-07-31T01:46:39.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:39 smithi045 conmon[39718]: audit 2021-07-31T01:46:39.435673+0000 mon.a (mon.0) 700 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-07-31T01:46:39.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:39 smithi045 conmon[39718]: audit 2021-07-31T01:46 2021-07-31T01:46:39.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:39 smithi045 conmon[39718]: :39.435881+0000 mon.a (mon.0) 701 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:46:40.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:39 smithi154 conmon[34553]: cluster 2021-07-31T01:46:38.500115+0000 mon.a (mon.0) 699 : cluster 2021-07-31T01:46:40.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:39 smithi154 conmon[34553]: [DBG] osdmap e99: 8 total, 8 up, 8 in 2021-07-31T01:46:40.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:39 smithi154 conmon[34553]: audit 2021-07-31T01:46:39.435673+0000 mon.a (mon.0) 700 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-07-31T01:46:40.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:39 smithi154 conmon[34553]: audit 2021-07-31T01:46:39.435881+0000 mon.a (mon.0) 701 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:46:40.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:39 smithi154 conmon[34553]: 2021-07-31T01:46:40.518 INFO:tasks.rados.rados.0.smithi045.stdout:328: done (3 left) 2021-07-31T01:46:40.519 INFO:tasks.rados.rados.0.smithi045.stdout:329: copy_from oid 15 from oid 42 current snap is 22 2021-07-31T01:46:40.519 INFO:tasks.rados.rados.0.smithi045.stdout:325: finishing write tid 3 to smithi04569269-38 2021-07-31T01:46:40.519 INFO:tasks.rados.rados.0.smithi045.stdout:325: finishing write tid 4 to smithi04569269-38 2021-07-31T01:46:40.519 INFO:tasks.rados.rados.0.smithi045.stdout:325: finishing write tid 5 to smithi04569269-38 2021-07-31T01:46:40.520 INFO:tasks.rados.rados.0.smithi045.stdout:326: finishing copy_from to smithi04569269-45 2021-07-31T01:46:40.520 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 45 v 109 (ObjNum 1 snap 32731 seq_num 3189609064) dirty exists 2021-07-31T01:46:40.520 INFO:tasks.rados.rados.0.smithi045.stdout:325: finishing write tid 6 to smithi04569269-38 2021-07-31T01:46:40.520 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 38 v 86 (ObjNum 119 snap 22 seq_num 119) dirty exists 2021-07-31T01:46:40.520 INFO:tasks.rados.rados.0.smithi045.stdout:325: left oid 38 (ObjNum 119 snap 22 seq_num 119) 2021-07-31T01:46:40.521 INFO:tasks.rados.rados.0.smithi045.stdout:326: finishing copy_from racing read to smithi04569269-45 2021-07-31T01:46:40.521 INFO:tasks.rados.rados.0.smithi045.stdout:327: finishing write tid 1 to smithi04569269-34 2021-07-31T01:46:40.521 INFO:tasks.rados.rados.0.smithi045.stdout:327: finishing write tid 2 to smithi04569269-34 2021-07-31T01:46:40.521 INFO:tasks.rados.rados.0.smithi045.stdout:327: finishing write tid 3 to smithi04569269-34 2021-07-31T01:46:40.521 INFO:tasks.rados.rados.0.smithi045.stdout:327: finishing write tid 4 to smithi04569269-34 2021-07-31T01:46:40.522 INFO:tasks.rados.rados.0.smithi045.stdout:327: finishing write tid 5 to smithi04569269-34 2021-07-31T01:46:40.522 INFO:tasks.rados.rados.0.smithi045.stdout:327: finishing write tid 6 to smithi04569269-34 2021-07-31T01:46:40.522 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 34 v 35 (ObjNum 120 snap 22 seq_num 120) dirty exists 2021-07-31T01:46:40.522 INFO:tasks.rados.rados.0.smithi045.stdout:327: left oid 34 (ObjNum 120 snap 22 seq_num 120) 2021-07-31T01:46:40.523 INFO:tasks.rados.rados.0.smithi045.stdout:325: done (3 left) 2021-07-31T01:46:40.523 INFO:tasks.rados.rados.0.smithi045.stdout:326: done (2 left) 2021-07-31T01:46:40.523 INFO:tasks.rados.rados.0.smithi045.stdout:327: done (1 left) 2021-07-31T01:46:40.523 INFO:tasks.rados.rados.0.smithi045.stdout:330: copy_from oid 19 from oid 27 current snap is 22 2021-07-31T01:46:40.523 INFO:tasks.rados.rados.0.smithi045.stdout:331: snap_create 2021-07-31T01:46:40.537 INFO:tasks.rados.rados.0.smithi045.stdout:330: finishing copy_from to smithi04569269-19 2021-07-31T01:46:40.537 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 19 v 41 (ObjNum 94 snap 15 seq_num 94) dirty exists 2021-07-31T01:46:40.538 INFO:tasks.rados.rados.0.smithi045.stdout:330: finishing copy_from racing read to smithi04569269-19 2021-07-31T01:46:40.538 INFO:tasks.rados.rados.0.smithi045.stdout:329: finishing copy_from to smithi04569269-15 2021-07-31T01:46:40.539 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 15 v 37 (ObjNum 76 snap 11 seq_num 76) dirty exists 2021-07-31T01:46:40.539 INFO:tasks.rados.rados.0.smithi045.stdout:329: finishing copy_from racing read to smithi04569269-15 2021-07-31T01:46:40.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:40 smithi045 conmon[33655]: cluster 2021-07-31T01:46:39.371503+0000 2021-07-31T01:46:40.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:40 smithi045 conmon[33655]: mgr.y (mgr.14146) 299 : cluster [DBG] pgmap v299: 25 pgs: 25 active+clean; 67 MiB data, 183 MiB used, 715 GiB / 715 GiB avail; 4.9 MiB/s rd, 2.4 MiB/s wr, 24 op/s 2021-07-31T01:46:40.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:40 smithi045 conmon[33655]: cluster 2021-07-31T01:46:39.511697+0000 mon.a (mon.0) 702 : cluster [DBG] osdmap e100: 8 total, 8 up, 8 in 2021-07-31T01:46:40.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:40 smithi045 conmon[33655]: 2021-07-31T01:46:40.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:40 smithi045 conmon[39718]: cluster 2021-07-31T01:46:39.371503+0000 mgr.y (mgr.14146) 299 : cluster [DBG] pgmap v299: 25 pgs: 25 active+clean; 67 MiB data, 183 MiB used, 715 GiB / 715 GiB avail; 4.9 MiB/s rd, 2.4 MiB/s wr, 24 op/s 2021-07-31T01:46:40.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:40 smithi045 conmon[39718]: cluster 2021-07-31T01:46:39.511697+0000 mon.a (mon.0) 702 : cluster [DBG] osdmap e100: 8 total, 8 up, 8 in 2021-07-31T01:46:41.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:40 smithi154 conmon[34553]: cluster 2021-07-31T01:46:39.371503+0000 mgr.y (mgr.14146 2021-07-31T01:46:41.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:40 smithi154 conmon[34553]: ) 299 : cluster [DBG] pgmap v299: 25 pgs: 25 active+clean; 67 MiB data, 183 MiB used, 715 GiB / 715 GiB avail; 4.9 MiB/s rd, 2.4 MiB/s wr, 24 op/s 2021-07-31T01:46:41.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:40 smithi154 conmon[34553]: cluster 2021-07-31T01:46:39.511697+0000 mon.a (mon.0) 702 : cluster [DBG] osdmap e100: 8 total, 8 up, 8 in 2021-07-31T01:46:41.478 INFO:tasks.rados.rados.0.smithi045.stdout:329: done (2 left) 2021-07-31T01:46:41.478 INFO:tasks.rados.rados.0.smithi045.stdout:330: done (1 left) 2021-07-31T01:46:41.479 INFO:tasks.rados.rados.0.smithi045.stdout:331: done (0 left) 2021-07-31T01:46:41.479 INFO:tasks.rados.rados.0.smithi045.stdout:332: read oid 15 snap -1 2021-07-31T01:46:41.479 INFO:tasks.rados.rados.0.smithi045.stdout:332: expect (ObjNum 76 snap 11 seq_num 76) 2021-07-31T01:46:41.479 INFO:tasks.rados.rados.0.smithi045.stdout:333: copy_from oid 3 from oid 47 current snap is 23 2021-07-31T01:46:41.480 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 38 current snap is 23 2021-07-31T01:46:41.480 INFO:tasks.rados.rados.0.smithi045.stdout:334: seq_num 121 ranges {2007040=434176,2441216=753664,3194880=425984,3620864=425984} 2021-07-31T01:46:41.490 INFO:tasks.rados.rados.0.smithi045.stdout:334: writing smithi04569269-38 from 2007040 to 2441216 tid 1 2021-07-31T01:46:41.494 INFO:tasks.rados.rados.0.smithi045.stdout:334: writing smithi04569269-38 from 2441216 to 3194880 tid 2 2021-07-31T01:46:41.497 INFO:tasks.rados.rados.0.smithi045.stdout:334: writing smithi04569269-38 from 3194880 to 3620864 tid 3 2021-07-31T01:46:41.500 INFO:tasks.rados.rados.0.smithi045.stdout:334: writing smithi04569269-38 from 3620864 to 4046848 tid 4 2021-07-31T01:46:41.501 INFO:tasks.rados.rados.0.smithi045.stdout:335: read oid 39 snap -1 2021-07-31T01:46:41.501 INFO:tasks.rados.rados.0.smithi045.stdout:335: expect (ObjNum 94 snap 15 seq_num 94) 2021-07-31T01:46:41.501 INFO:tasks.rados.rados.0.smithi045.stdout:336: rmattr oid 46 current snap is 23 2021-07-31T01:46:41.501 INFO:tasks.rados.rados.0.smithi045.stdout:336: done (4 left) 2021-07-31T01:46:41.501 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 35 current snap is 23 2021-07-31T01:46:41.502 INFO:tasks.rados.rados.0.smithi045.stdout:333: finishing copy_from to smithi04569269-3 2021-07-31T01:46:41.502 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 3 v 4 (ObjNum 46 snap 0 seq_num 46) dirty exists 2021-07-31T01:46:41.502 INFO:tasks.rados.rados.0.smithi045.stdout:333: finishing copy_from racing read to smithi04569269-3 2021-07-31T01:46:41.502 INFO:tasks.rados.rados.0.smithi045.stdout:337: seq_num 122 ranges {1736704=450560,2187264=688128,2875392=630784,3506176=286720} 2021-07-31T01:46:41.512 INFO:tasks.rados.rados.0.smithi045.stdout:337: writing smithi04569269-35 from 1736704 to 2187264 tid 1 2021-07-31T01:46:41.516 INFO:tasks.rados.rados.0.smithi045.stdout:337: writing smithi04569269-35 from 2187264 to 2875392 tid 2 2021-07-31T01:46:41.520 INFO:tasks.rados.rados.0.smithi045.stdout:337: writing smithi04569269-35 from 2875392 to 3506176 tid 3 2021-07-31T01:46:41.522 INFO:tasks.rados.rados.0.smithi045.stdout:337: writing smithi04569269-35 from 3506176 to 3792896 tid 4 2021-07-31T01:46:41.522 INFO:tasks.rados.rados.0.smithi045.stdout:333: done (4 left) 2021-07-31T01:46:41.523 INFO:tasks.rados.rados.0.smithi045.stdout:338: read oid 26 snap -1 2021-07-31T01:46:41.523 INFO:tasks.rados.rados.0.smithi045.stdout:338: expect (ObjNum 101 snap 16 seq_num 101) 2021-07-31T01:46:41.523 INFO:tasks.rados.rados.0.smithi045.stdout:339: copy_from oid 34 from oid 42 current snap is 23 2021-07-31T01:46:41.523 INFO:tasks.rados.rados.0.smithi045.stdout:332: expect (ObjNum 76 snap 11 seq_num 76) 2021-07-31T01:46:41.587 INFO:tasks.rados.rados.0.smithi045.stdout:334: finishing write tid 1 to smithi04569269-38 2021-07-31T01:46:41.587 INFO:tasks.rados.rados.0.smithi045.stdout:334: finishing write tid 2 to smithi04569269-38 2021-07-31T01:46:41.587 INFO:tasks.rados.rados.0.smithi045.stdout:334: finishing write tid 3 to smithi04569269-38 2021-07-31T01:46:41.588 INFO:tasks.rados.rados.0.smithi045.stdout:334: finishing write tid 4 to smithi04569269-38 2021-07-31T01:46:41.588 INFO:tasks.rados.rados.0.smithi045.stdout:332: done (5 left) 2021-07-31T01:46:41.588 INFO:tasks.rados.rados.0.smithi045.stdout:340: read oid 18 snap -1 2021-07-31T01:46:41.588 INFO:tasks.rados.rados.0.smithi045.stdout:340: expect (ObjNum 75 snap 11 seq_num 75) 2021-07-31T01:46:41.589 INFO:tasks.rados.rados.0.smithi045.stdout:341: read oid 3 snap 19 2021-07-31T01:46:41.589 INFO:tasks.rados.rados.0.smithi045.stdout:341: expect (ObjNum 6 snap 0 seq_num 6) 2021-07-31T01:46:41.589 INFO:tasks.rados.rados.0.smithi045.stdout:342: read oid 21 snap -1 2021-07-31T01:46:41.589 INFO:tasks.rados.rados.0.smithi045.stdout:342: expect (ObjNum 117 snap 22 seq_num 117) 2021-07-31T01:46:41.590 INFO:tasks.rados.rados.0.smithi045.stdout:343: copy_from oid 16 from oid 12 current snap is 23 2021-07-31T01:46:41.590 INFO:tasks.rados.rados.0.smithi045.stdout:334: finishing write tid 5 to smithi04569269-38 2021-07-31T01:46:41.590 INFO:tasks.rados.rados.0.smithi045.stdout:334: finishing write tid 6 to smithi04569269-38 2021-07-31T01:46:41.590 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 38 v 98 (ObjNum 121 snap 23 seq_num 121) dirty exists 2021-07-31T01:46:41.590 INFO:tasks.rados.rados.0.smithi045.stdout:334: left oid 38 (ObjNum 121 snap 23 seq_num 121) 2021-07-31T01:46:41.591 INFO:tasks.rados.rados.0.smithi045.stdout:335: expect (ObjNum 94 snap 15 seq_num 94) 2021-07-31T01:46:41.634 INFO:tasks.rados.rados.0.smithi045.stdout:337: finishing write tid 1 to smithi04569269-35 2021-07-31T01:46:41.635 INFO:tasks.rados.rados.0.smithi045.stdout:337: finishing write tid 2 to smithi04569269-35 2021-07-31T01:46:41.635 INFO:tasks.rados.rados.0.smithi045.stdout:337: finishing write tid 3 to smithi04569269-35 2021-07-31T01:46:41.635 INFO:tasks.rados.rados.0.smithi045.stdout:337: finishing write tid 4 to smithi04569269-35 2021-07-31T01:46:41.635 INFO:tasks.rados.rados.0.smithi045.stdout:334: done (8 left) 2021-07-31T01:46:41.636 INFO:tasks.rados.rados.0.smithi045.stdout:335: done (7 left) 2021-07-31T01:46:41.636 INFO:tasks.rados.rados.0.smithi045.stdout:344: snap_create 2021-07-31T01:46:41.636 INFO:tasks.rados.rados.0.smithi045.stdout:337: finishing write tid 5 to smithi04569269-35 2021-07-31T01:46:41.636 INFO:tasks.rados.rados.0.smithi045.stdout:337: finishing write tid 6 to smithi04569269-35 2021-07-31T01:46:41.637 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 35 v 179 (ObjNum 122 snap 23 seq_num 122) dirty exists 2021-07-31T01:46:41.637 INFO:tasks.rados.rados.0.smithi045.stdout:337: left oid 35 (ObjNum 122 snap 23 seq_num 122) 2021-07-31T01:46:41.637 INFO:tasks.rados.rados.0.smithi045.stdout:338: expect (ObjNum 101 snap 16 seq_num 101) 2021-07-31T01:46:41.678 INFO:tasks.rados.rados.0.smithi045.stdout:339: finishing copy_from to smithi04569269-34 2021-07-31T01:46:41.679 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 34 v 37 (ObjNum 76 snap 11 seq_num 76) dirty exists 2021-07-31T01:46:41.679 INFO:tasks.rados.rados.0.smithi045.stdout:339: finishing copy_from racing read to smithi04569269-34 2021-07-31T01:46:41.679 INFO:tasks.rados.rados.0.smithi045.stdout:341: expect (ObjNum 6 snap 0 seq_num 6) 2021-07-31T01:46:41.681 INFO:tasks.rados.rados.0.smithi045.stdout:340: expect (ObjNum 75 snap 11 seq_num 75) 2021-07-31T01:46:41.773 INFO:tasks.rados.rados.0.smithi045.stdout:343: finishing copy_from to smithi04569269-16 2021-07-31T01:46:41.773 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 16 v 70 (ObjNum 118 snap 22 seq_num 118) dirty exists 2021-07-31T01:46:41.773 INFO:tasks.rados.rados.0.smithi045.stdout:342: expect (ObjNum 117 snap 22 seq_num 117) 2021-07-31T01:46:41.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:41 smithi045 conmon[33655]: cluster 2021-07-31T01:46:41.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:41 smithi045 conmon[33655]: 2021-07-31T01:46:40.517224 2021-07-31T01:46:41.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:41 smithi045 conmon[33655]: +0000 mon.a (mon.0) 703 : cluster [DBG] osdmap e101: 8 total, 8 up, 8 in 2021-07-31T01:46:41.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:41 smithi045 conmon[33655]: cluster 2021-07-31T01: 2021-07-31T01:46:41.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:41 smithi045 conmon[33655]: 46:41.475692+0000 mon.a (mon.0) 704 : cluster [DBG] osdmap e102: 8 total, 8 up, 8 in 2021-07-31T01:46:41.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:41 smithi045 conmon[39718]: cluster 2021-07-31T01:46:40. 2021-07-31T01:46:41.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:41 smithi045 conmon[39718]: 517224+0000 mon.a (mon.0) 703 : cluster [DBG] osdmap e101: 8 total, 8 up, 8 in 2021-07-31T01:46:41.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:41 smithi045 conmon[39718]: cluster 2021-07-31T01:46:41 2021-07-31T01:46:41.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:41 smithi045 conmon[39718]: .475692+0000 mon.a (mon.0) 704 : cluster 2021-07-31T01:46:41.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:41 smithi045 conmon[39718]: [DBG] osdmap e102: 8 total, 8 up, 8 in 2021-07-31T01:46:41.884 INFO:tasks.rados.rados.0.smithi045.stdout:343: finishing copy_from racing read to smithi04569269-16 2021-07-31T01:46:42.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:41 smithi154 conmon[34553]: cluster 2021-07-31T01:46:40.517224+0000 mon.a (mon 2021-07-31T01:46:42.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:41 smithi154 conmon[34553]: .0) 703 : cluster [DBG] osdmap e101: 8 total, 8 up, 8 in 2021-07-31T01:46:42.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:41 smithi154 conmon[34553]: cluster 2021-07-31T01:46:41.475692+0000 mon.a (mon.0) 704 : cluster [DBG] osdmap e102: 8 total, 8 up, 8 in 2021-07-31T01:46:42.526 INFO:tasks.rados.rados.0.smithi045.stdout:337: done (7 left) 2021-07-31T01:46:42.526 INFO:tasks.rados.rados.0.smithi045.stdout:338: done (6 left) 2021-07-31T01:46:42.526 INFO:tasks.rados.rados.0.smithi045.stdout:339: done (5 left) 2021-07-31T01:46:42.527 INFO:tasks.rados.rados.0.smithi045.stdout:340: done (4 left) 2021-07-31T01:46:42.527 INFO:tasks.rados.rados.0.smithi045.stdout:341: done (3 left) 2021-07-31T01:46:42.527 INFO:tasks.rados.rados.0.smithi045.stdout:342: done (2 left) 2021-07-31T01:46:42.527 INFO:tasks.rados.rados.0.smithi045.stdout:343: done (1 left) 2021-07-31T01:46:42.527 INFO:tasks.rados.rados.0.smithi045.stdout:344: done (0 left) 2021-07-31T01:46:42.528 INFO:tasks.rados.rados.0.smithi045.stdout:345: delete oid 32 current snap is 24 2021-07-31T01:46:42.528 INFO:tasks.rados.rados.0.smithi045.stdout:345: done (0 left) 2021-07-31T01:46:42.528 INFO:tasks.rados.rados.0.smithi045.stdout:346: delete oid 20 current snap is 24 2021-07-31T01:46:42.532 INFO:tasks.rados.rados.0.smithi045.stdout:346: done (0 left) 2021-07-31T01:46:42.532 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 42 current snap is 24 2021-07-31T01:46:42.532 INFO:tasks.rados.rados.0.smithi045.stdout:347: seq_num 125 ranges {1974272=499712,2473984=745472,3219456=409600,3629056=450560} 2021-07-31T01:46:42.546 INFO:tasks.rados.rados.0.smithi045.stdout:347: writing smithi04569269-42 from 1974272 to 2473984 tid 1 2021-07-31T01:46:42.550 INFO:tasks.rados.rados.0.smithi045.stdout:347: writing smithi04569269-42 from 2473984 to 3219456 tid 2 2021-07-31T01:46:42.552 INFO:tasks.rados.rados.0.smithi045.stdout:347: writing smithi04569269-42 from 3219456 to 3629056 tid 3 2021-07-31T01:46:42.555 INFO:tasks.rados.rados.0.smithi045.stdout:347: writing smithi04569269-42 from 3629056 to 4079616 tid 4 2021-07-31T01:46:42.556 INFO:tasks.rados.rados.0.smithi045.stdout:348: snap_create 2021-07-31T01:46:42.556 INFO:tasks.rados.rados.0.smithi045.stdout:347: finishing write tid 1 to smithi04569269-42 2021-07-31T01:46:42.556 INFO:tasks.rados.rados.0.smithi045.stdout:347: finishing write tid 2 to smithi04569269-42 2021-07-31T01:46:42.557 INFO:tasks.rados.rados.0.smithi045.stdout:347: finishing write tid 3 to smithi04569269-42 2021-07-31T01:46:42.557 INFO:tasks.rados.rados.0.smithi045.stdout:347: finishing write tid 4 to smithi04569269-42 2021-07-31T01:46:42.558 INFO:tasks.rados.rados.0.smithi045.stdout:347: finishing write tid 5 to smithi04569269-42 2021-07-31T01:46:42.559 INFO:tasks.rados.rados.0.smithi045.stdout:347: finishing write tid 6 to smithi04569269-42 2021-07-31T01:46:42.559 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 42 v 115 (ObjNum 125 snap 24 seq_num 125) dirty exists 2021-07-31T01:46:42.560 INFO:tasks.rados.rados.0.smithi045.stdout:347: left oid 42 (ObjNum 125 snap 24 seq_num 125) 2021-07-31T01:46:42.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:42 smithi045 conmon[33655]: cluster 2021-07-31T01:46:41.371919+0000 mgr.y (mgr.14146) 300 : cluster 2021-07-31T01:46:42.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:42 smithi045 conmon[33655]: [DBG] pgmap v302: 25 pgs: 25 active+clean; 67 MiB data, 183 MiB used, 715 GiB / 715 GiB avail; 4.9 MiB/s rd, 1.9 MiB/s wr, 23 op/s 2021-07-31T01:46:42.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:42 smithi045 conmon[39718]: cluster 2021-07-31T01:46:41.371919+0000 mgr.y (mgr.14146) 300 : cluster 2021-07-31T01:46:42.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:42 smithi045 conmon[39718]: [DBG] pgmap v302: 25 pgs: 25 active+clean; 67 MiB data, 183 MiB used, 715 GiB / 715 GiB avail; 4.9 MiB/s rd, 1.9 MiB/s wr, 23 op/s 2021-07-31T01:46:43.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:42 smithi154 conmon[34553]: cluster 2021-07-31T01:46:41.371919 2021-07-31T01:46:43.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:42 smithi154 conmon[34553]: +0000 mgr.y (mgr.14146) 300 : cluster [DBG] pgmap v302: 25 pgs: 25 active+clean; 67 MiB data, 183 MiB used, 715 GiB / 715 GiB avail; 4.9 MiB/s rd, 1.9 MiB/s wr, 23 op/s 2021-07-31T01:46:43.534 INFO:tasks.rados.rados.0.smithi045.stdout:347: done (1 left) 2021-07-31T01:46:43.534 INFO:tasks.rados.rados.0.smithi045.stdout:348: done (0 left) 2021-07-31T01:46:43.534 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 5 current snap is 25 2021-07-31T01:46:43.535 INFO:tasks.rados.rados.0.smithi045.stdout:349: seq_num 126 ranges {0=516096,516096=630784,1146880=614400,1761280=360448} 2021-07-31T01:46:43.536 INFO:tasks.rados.rados.0.smithi045.stdout:349: writing smithi04569269-5 from 0 to 516096 tid 1 2021-07-31T01:46:43.540 INFO:tasks.rados.rados.0.smithi045.stdout:349: writing smithi04569269-5 from 516096 to 1146880 tid 2 2021-07-31T01:46:43.543 INFO:tasks.rados.rados.0.smithi045.stdout:349: writing smithi04569269-5 from 1146880 to 1761280 tid 3 2021-07-31T01:46:43.546 INFO:tasks.rados.rados.0.smithi045.stdout:349: writing smithi04569269-5 from 1761280 to 2121728 tid 4 2021-07-31T01:46:43.546 INFO:tasks.rados.rados.0.smithi045.stdout:350: rollback oid 31 current snap is 25 2021-07-31T01:46:43.547 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 31 to 23 2021-07-31T01:46:43.547 INFO:tasks.rados.rados.0.smithi045.stdout:349: finishing write tid 1 to smithi04569269-5 2021-07-31T01:46:43.547 INFO:tasks.rados.rados.0.smithi045.stdout:349: finishing write tid 2 to smithi04569269-5 2021-07-31T01:46:43.547 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 49 current snap is 25 2021-07-31T01:46:43.547 INFO:tasks.rados.rados.0.smithi045.stdout:351: seq_num 127 ranges {0=532480,532480=507904,1040384=778240,1818624=319488} 2021-07-31T01:46:43.549 INFO:tasks.rados.rados.0.smithi045.stdout:351: writing smithi04569269-49 from 0 to 532480 tid 1 2021-07-31T01:46:43.553 INFO:tasks.rados.rados.0.smithi045.stdout:351: writing smithi04569269-49 from 532480 to 1040384 tid 2 2021-07-31T01:46:43.557 INFO:tasks.rados.rados.0.smithi045.stdout:351: writing smithi04569269-49 from 1040384 to 1818624 tid 3 2021-07-31T01:46:43.560 INFO:tasks.rados.rados.0.smithi045.stdout:351: writing smithi04569269-49 from 1818624 to 2138112 tid 4 2021-07-31T01:46:43.560 INFO:tasks.rados.rados.0.smithi045.stdout:352: rmattr oid 41 current snap is 25 2021-07-31T01:46:43.561 INFO:tasks.rados.rados.0.smithi045.stdout:349: finishing write tid 3 to smithi04569269-5 2021-07-31T01:46:43.561 INFO:tasks.rados.rados.0.smithi045.stdout:353: read oid 18 snap -1 2021-07-31T01:46:43.561 INFO:tasks.rados.rados.0.smithi045.stdout:353: expect (ObjNum 75 snap 11 seq_num 75) 2021-07-31T01:46:43.561 INFO:tasks.rados.rados.0.smithi045.stdout:354: snap_create 2021-07-31T01:46:43.562 INFO:tasks.rados.rados.0.smithi045.stdout:349: finishing write tid 4 to smithi04569269-5 2021-07-31T01:46:43.562 INFO:tasks.rados.rados.0.smithi045.stdout:349: finishing write tid 5 to smithi04569269-5 2021-07-31T01:46:43.562 INFO:tasks.rados.rados.0.smithi045.stdout:350: finishing rollback tid 0 to smithi04569269-31 2021-07-31T01:46:43.563 INFO:tasks.rados.rados.0.smithi045.stdout:350: finishing rollback tid 1 to smithi04569269-31 2021-07-31T01:46:43.563 INFO:tasks.rados.rados.0.smithi045.stdout:350: finishing rollback tid 2 to smithi04569269-31 2021-07-31T01:46:43.563 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 31 v 185 (ObjNum 30 snap 0 seq_num 30) dirty exists 2021-07-31T01:46:43.563 INFO:tasks.rados.rados.0.smithi045.stdout:349: finishing write tid 6 to smithi04569269-5 2021-07-31T01:46:43.563 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 5 v 103 (ObjNum 126 snap 25 seq_num 126) dirty exists 2021-07-31T01:46:43.564 INFO:tasks.rados.rados.0.smithi045.stdout:349: left oid 5 (ObjNum 126 snap 25 seq_num 126) 2021-07-31T01:46:43.564 INFO:tasks.rados.rados.0.smithi045.stdout:351: finishing write tid 1 to smithi04569269-49 2021-07-31T01:46:43.564 INFO:tasks.rados.rados.0.smithi045.stdout:351: finishing write tid 2 to smithi04569269-49 2021-07-31T01:46:43.565 INFO:tasks.rados.rados.0.smithi045.stdout:351: finishing write tid 3 to smithi04569269-49 2021-07-31T01:46:43.566 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 41 v 31 (ObjNum 40 snap 0 seq_num 40) dirty exists 2021-07-31T01:46:43.566 INFO:tasks.rados.rados.0.smithi045.stdout:351: finishing write tid 4 to smithi04569269-49 2021-07-31T01:46:43.566 INFO:tasks.rados.rados.0.smithi045.stdout:351: finishing write tid 5 to smithi04569269-49 2021-07-31T01:46:43.567 INFO:tasks.rados.rados.0.smithi045.stdout:351: finishing write tid 6 to smithi04569269-49 2021-07-31T01:46:43.567 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 49 v 190 (ObjNum 127 snap 25 seq_num 127) dirty exists 2021-07-31T01:46:43.567 INFO:tasks.rados.rados.0.smithi045.stdout:351: left oid 49 (ObjNum 127 snap 25 seq_num 127) 2021-07-31T01:46:43.571 INFO:tasks.rados.rados.0.smithi045.stdout:353: expect (ObjNum 75 snap 11 seq_num 75) 2021-07-31T01:46:43.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:46:43 smithi045 conmon[61210]: level=error ts=2021-07-31T01:46:43.505Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:46:43.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:46:43 smithi045 conmon[61210]: level=error ts=2021-07-31T01:46:43.505Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:46:43.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:46:43 smithi045 conmon[61210]: level=error ts=2021-07-31T01:46:43.505Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T01:46:43.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:43 smithi045 conmon[33655]: cluster 2021-07-31T01:46:42.523327+0000 2021-07-31T01:46:43.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:43 smithi045 conmon[33655]: mon.a (mon.0) 705 : cluster [DBG] osdmap e103: 8 total, 8 up, 8 in 2021-07-31T01:46:43.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:43 smithi045 conmon[39718]: cluster 2021-07-31T01:46:43.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:43 smithi045 conmon[39718]: 2021-07-31T01:46:43.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:43 smithi045 conmon[39718]: 2021 2021-07-31T01:46:43.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:43 smithi045 conmon[39718]: -07-31T01:46:42.523327+0000 mon.a 2021-07-31T01:46:43.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:43 smithi045 conmon[39718]: (mon.0) 705 : cluster [DBG] osdmap e103: 8 total, 8 up, 8 in 2021-07-31T01:46:44.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:43 smithi154 conmon[34553]: cluster 2021-07-31T01:46:44.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:43 smithi154 conmon[34553]: 2021-07-31T 2021-07-31T01:46:44.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:43 smithi154 conmon[34553]: 01:46:42.523327+0000 mon.a (mon.0) 705 : cluster [DBG] osdmap e103: 8 total, 8 up, 8 in 2021-07-31T01:46:44.535 INFO:tasks.rados.rados.0.smithi045.stdout:349: done (5 left) 2021-07-31T01:46:44.535 INFO:tasks.rados.rados.0.smithi045.stdout:350: done (4 left) 2021-07-31T01:46:44.535 INFO:tasks.rados.rados.0.smithi045.stdout:351: done (3 left) 2021-07-31T01:46:44.536 INFO:tasks.rados.rados.0.smithi045.stdout:352: done (2 left) 2021-07-31T01:46:44.536 INFO:tasks.rados.rados.0.smithi045.stdout:353: done (1 left) 2021-07-31T01:46:44.536 INFO:tasks.rados.rados.0.smithi045.stdout:354: done (0 left) 2021-07-31T01:46:44.536 INFO:tasks.rados.rados.0.smithi045.stdout:355: delete oid 49 current snap is 26 2021-07-31T01:46:44.541 INFO:tasks.rados.rados.0.smithi045.stdout:355: done (0 left) 2021-07-31T01:46:44.542 INFO:tasks.rados.rados.0.smithi045.stdout:356: rollback oid 18 current snap is 26 2021-07-31T01:46:44.542 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 18 to 25 2021-07-31T01:46:44.542 INFO:tasks.rados.rados.0.smithi045.stdout:357: snap_create 2021-07-31T01:46:44.546 INFO:tasks.rados.rados.0.smithi045.stdout:356: finishing rollback tid 0 to smithi04569269-18 2021-07-31T01:46:44.546 INFO:tasks.rados.rados.0.smithi045.stdout:356: finishing rollback tid 1 to smithi04569269-18 2021-07-31T01:46:44.546 INFO:tasks.rados.rados.0.smithi045.stdout:356: finishing rollback tid 2 to smithi04569269-18 2021-07-31T01:46:44.547 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 18 v 67 (ObjNum 75 snap 11 seq_num 75) dirty exists 2021-07-31T01:46:45.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:44 smithi154 conmon[34553]: cluster 2021-07-31T01:46:43.372565+0000 mgr.y (mgr. 2021-07-31T01:46:45.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:44 smithi154 conmon[34553]: 14146) 301 : cluster [DBG] pgmap v305: 25 pgs: 25 active+clean; 81 MiB data, 206 MiB used, 715 GiB / 715 GiB avail; 8.5 MiB/s rd, 3.3 MiB/s wr, 22 op/s 2021-07-31T01:46:45.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:44 smithi154 conmon[34553]: cluster 2021-07-31T01:46:43.533172+0000 mon.a (mon.0) 706 : cluster [DBG] osdmap e104: 8 total, 8 up, 8 in 2021-07-31T01:46:45.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:44 smithi045 conmon[39718]: cluster 2021-07-31T01:46:43.372565+0000 mgr.y 2021-07-31T01:46:45.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:44 smithi045 conmon[39718]: (mgr.14146) 301 : cluster [DBG] pgmap v305: 25 pgs: 25 active+clean; 81 MiB data, 206 MiB used, 715 GiB / 715 GiB avail; 8.5 MiB/s rd, 3.3 MiB/s wr, 22 op/s 2021-07-31T01:46:45.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:44 smithi045 conmon[39718]: cluster 2021-07- 2021-07-31T01:46:45.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:44 smithi045 conmon[39718]: 31T01:46:43.533172+0000 mon.a (mon. 2021-07-31T01:46:45.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:44 smithi045 conmon[39718]: 0) 706 : cluster [DBG] osdmap e104: 8 total, 8 up, 8 in 2021-07-31T01:46:45.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:44 smithi045 conmon[39718]: 2021-07-31T01:46:45.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:44 smithi045 conmon[33655]: cluster 2021-07-31T01:46:43.372565+0000 mgr.y (mgr.14146) 301 : cluster [DBG] 2021-07-31T01:46:45.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:44 smithi045 conmon[33655]: pgmap v305: 25 pgs: 25 active+clean; 81 MiB data, 206 MiB used, 715 GiB / 715 GiB avail; 8.5 MiB/s rd, 3.3 MiB/s wr, 22 op/s 2021-07-31T01:46:45.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:44 smithi045 conmon[33655]: cluster 2021-07-31T01:46:43.533172+0000 mon.a (mon.0) 706 : cluster [DBG] osdmap e104: 8 total, 8 up, 8 in 2021-07-31T01:46:45.547 INFO:tasks.rados.rados.0.smithi045.stdout:356: done (1 left) 2021-07-31T01:46:45.547 INFO:tasks.rados.rados.0.smithi045.stdout:357: done (0 left) 2021-07-31T01:46:45.548 INFO:tasks.rados.rados.0.smithi045.stdout:358: rollback oid 41 current snap is 27 2021-07-31T01:46:45.548 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 41 to 23 2021-07-31T01:46:45.548 INFO:tasks.rados.rados.0.smithi045.stdout:359: rollback oid 18 current snap is 27 2021-07-31T01:46:45.548 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 18 to 20 2021-07-31T01:46:45.549 INFO:tasks.rados.rados.0.smithi045.stdout:360: rmattr oid 23 current snap is 27 2021-07-31T01:46:45.549 INFO:tasks.rados.rados.0.smithi045.stdout:361: copy_from oid 13 from oid 37 current snap is 27 2021-07-31T01:46:45.549 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 36 current snap is 27 2021-07-31T01:46:45.549 INFO:tasks.rados.rados.0.smithi045.stdout:362: seq_num 129 ranges {761856=565248,1327104=679936,2007040=786432,2793472=139264} 2021-07-31T01:46:45.553 INFO:tasks.rados.rados.0.smithi045.stdout:362: writing smithi04569269-36 from 761856 to 1327104 tid 1 2021-07-31T01:46:45.557 INFO:tasks.rados.rados.0.smithi045.stdout:362: writing smithi04569269-36 from 1327104 to 2007040 tid 2 2021-07-31T01:46:45.562 INFO:tasks.rados.rados.0.smithi045.stdout:362: writing smithi04569269-36 from 2007040 to 2793472 tid 3 2021-07-31T01:46:45.563 INFO:tasks.rados.rados.0.smithi045.stdout:362: writing smithi04569269-36 from 2793472 to 2932736 tid 4 2021-07-31T01:46:45.563 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 31 current snap is 27 2021-07-31T01:46:45.564 INFO:tasks.rados.rados.0.smithi045.stdout:363: seq_num 130 ranges {507904=581632,1089536=614400,1703936=598016,2301952=393216} 2021-07-31T01:46:45.569 INFO:tasks.rados.rados.0.smithi045.stdout:363: writing smithi04569269-31 from 507904 to 1089536 tid 1 2021-07-31T01:46:45.573 INFO:tasks.rados.rados.0.smithi045.stdout:363: writing smithi04569269-31 from 1089536 to 1703936 tid 2 2021-07-31T01:46:45.576 INFO:tasks.rados.rados.0.smithi045.stdout:363: writing smithi04569269-31 from 1703936 to 2301952 tid 3 2021-07-31T01:46:45.579 INFO:tasks.rados.rados.0.smithi045.stdout:363: writing smithi04569269-31 from 2301952 to 2695168 tid 4 2021-07-31T01:46:45.579 INFO:tasks.rados.rados.0.smithi045.stdout:364: setattr oid 39 current snap is 27 2021-07-31T01:46:45.580 INFO:tasks.rados.rados.0.smithi045.stdout:358: finishing rollback tid 0 to smithi04569269-41 2021-07-31T01:46:45.580 INFO:tasks.rados.rados.0.smithi045.stdout:358: finishing rollback tid 1 to smithi04569269-41 2021-07-31T01:46:45.580 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 23 v 105 (ObjNum 116 snap 21 seq_num 116) dirty exists 2021-07-31T01:46:45.580 INFO:tasks.rados.rados.0.smithi045.stdout:358: finishing rollback tid 2 to smithi04569269-41 2021-07-31T01:46:45.581 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 41 v 35 (ObjNum 40 snap 0 seq_num 40) dirty exists 2021-07-31T01:46:45.581 INFO:tasks.rados.rados.0.smithi045.stdout:359: finishing rollback tid 0 to smithi04569269-18 2021-07-31T01:46:45.581 INFO:tasks.rados.rados.0.smithi045.stdout:359: finishing rollback tid 1 to smithi04569269-18 2021-07-31T01:46:45.582 INFO:tasks.rados.rados.0.smithi045.stdout:359: finishing rollback tid 2 to smithi04569269-18 2021-07-31T01:46:45.582 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 18 v 70 (ObjNum 75 snap 11 seq_num 75) dirty exists 2021-07-31T01:46:45.582 INFO:tasks.rados.rados.0.smithi045.stdout:362: finishing write tid 1 to smithi04569269-36 2021-07-31T01:46:45.582 INFO:tasks.rados.rados.0.smithi045.stdout:362: finishing write tid 2 to smithi04569269-36 2021-07-31T01:46:45.582 INFO:tasks.rados.rados.0.smithi045.stdout:361: finishing copy_from to smithi04569269-13 2021-07-31T01:46:45.583 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 13 v 64 (ObjNum 98 snap 16 seq_num 98) dirty exists 2021-07-31T01:46:45.583 INFO:tasks.rados.rados.0.smithi045.stdout:361: finishing copy_from racing read to smithi04569269-13 2021-07-31T01:46:45.583 INFO:tasks.rados.rados.0.smithi045.stdout:362: finishing write tid 3 to smithi04569269-36 2021-07-31T01:46:45.583 INFO:tasks.rados.rados.0.smithi045.stdout:362: finishing write tid 4 to smithi04569269-36 2021-07-31T01:46:45.584 INFO:tasks.rados.rados.0.smithi045.stdout:362: finishing write tid 5 to smithi04569269-36 2021-07-31T01:46:45.584 INFO:tasks.rados.rados.0.smithi045.stdout:362: finishing write tid 6 to smithi04569269-36 2021-07-31T01:46:45.584 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 36 v 198 (ObjNum 129 snap 27 seq_num 129) dirty exists 2021-07-31T01:46:45.584 INFO:tasks.rados.rados.0.smithi045.stdout:362: left oid 36 (ObjNum 129 snap 27 seq_num 129) 2021-07-31T01:46:45.585 INFO:tasks.rados.rados.0.smithi045.stdout:363: finishing write tid 1 to smithi04569269-31 2021-07-31T01:46:45.585 INFO:tasks.rados.rados.0.smithi045.stdout:363: finishing write tid 2 to smithi04569269-31 2021-07-31T01:46:45.585 INFO:tasks.rados.rados.0.smithi045.stdout:363: finishing write tid 3 to smithi04569269-31 2021-07-31T01:46:45.586 INFO:tasks.rados.rados.0.smithi045.stdout:358: done (6 left) 2021-07-31T01:46:45.586 INFO:tasks.rados.rados.0.smithi045.stdout:359: done (5 left) 2021-07-31T01:46:45.587 INFO:tasks.rados.rados.0.smithi045.stdout:360: done (4 left) 2021-07-31T01:46:45.587 INFO:tasks.rados.rados.0.smithi045.stdout:361: done (3 left) 2021-07-31T01:46:45.587 INFO:tasks.rados.rados.0.smithi045.stdout:362: done (2 left) 2021-07-31T01:46:45.587 INFO:tasks.rados.rados.0.smithi045.stdout:365: snap_create 2021-07-31T01:46:45.588 INFO:tasks.rados.rados.0.smithi045.stdout:363: finishing write tid 4 to smithi04569269-31 2021-07-31T01:46:45.588 INFO:tasks.rados.rados.0.smithi045.stdout:363: finishing write tid 5 to smithi04569269-31 2021-07-31T01:46:45.588 INFO:tasks.rados.rados.0.smithi045.stdout:363: finishing write tid 6 to smithi04569269-31 2021-07-31T01:46:45.588 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 31 v 204 (ObjNum 130 snap 27 seq_num 130) dirty exists 2021-07-31T01:46:45.589 INFO:tasks.rados.rados.0.smithi045.stdout:363: left oid 31 (ObjNum 130 snap 27 seq_num 130) 2021-07-31T01:46:45.589 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 39 v 107 (ObjNum 94 snap 15 seq_num 94) dirty exists 2021-07-31T01:46:46.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:45 smithi154 conmon[34553]: cluster 2021-07-31T01:46:44.534500+0000 mon.a (mon 2021-07-31T01:46:46.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:45 smithi154 conmon[34553]: .0) 707 : cluster [DBG] osdmap e105: 8 total, 8 up, 8 in 2021-07-31T01:46:46.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:45 smithi045 conmon[33655]: cluster 2021-07-31T01:46:44.534500+0000 2021-07-31T01:46:46.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:45 smithi045 conmon[33655]: mon.a (mon.0) 707 : cluster [DBG] osdmap e105: 8 total, 8 up, 8 in 2021-07-31T01:46:46.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:45 smithi045 conmon[39718]: cluster 2021-07-31T01:46:46.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:45 smithi045 conmon[39718]: 2021-07-31T01:46 2021-07-31T01:46:46.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:45 smithi045 conmon[39718]: :44.534500+0000 mon.a (mon.0) 707 : cluster [DBG] osdmap e105: 8 total, 8 up, 8 in 2021-07-31T01:46:46.477 INFO:tasks.rados.rados.0.smithi045.stdout:363: done (2 left) 2021-07-31T01:46:46.478 INFO:tasks.rados.rados.0.smithi045.stdout:364: done (1 left) 2021-07-31T01:46:46.478 INFO:tasks.rados.rados.0.smithi045.stdout:365: done (0 left) 2021-07-31T01:46:46.478 INFO:tasks.rados.rados.0.smithi045.stdout:366: copy_from oid 34 from oid 46 current snap is 28 2021-07-31T01:46:46.479 INFO:tasks.rados.rados.0.smithi045.stdout:367: snap_remove snap 25 2021-07-31T01:46:47.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:46 smithi154 conmon[34553]: cluster 2021-07-31T01:46:45.373178+0000 mgr.y (mgr. 2021-07-31T01:46:47.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:46 smithi154 conmon[34553]: 14146) 302 : cluster [DBG] pgmap v308: 25 pgs: 25 active+clean; 99 MiB data, 232 MiB used, 715 GiB / 715 GiB avail; 16 MiB/s rd, 7.7 MiB/s wr, 37 op/s 2021-07-31T01:46:47.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:46 smithi154 conmon[34553]: cluster 2021-07-31T01:46:45.545131+0000 mon.a (mon.0) 708 : cluster [DBG] osdmap e106: 8 total, 8 up, 8 in 2021-07-31T01:46:47.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:46 smithi154 conmon[34553]: cluster 2021-07-31T01:46:46.476575+0000 2021-07-31T01:46:47.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:46 smithi154 conmon[34553]: mon.a (mon.0) 709 : cluster [DBG] osdmap e107: 8 total, 8 up, 8 in 2021-07-31T01:46:47.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:46 smithi154 conmon[34553]: 2021-07-31T01:46:47.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:46 smithi045 conmon[33655]: cluster 2021-07-31T01:46:45.373178+0000 2021-07-31T01:46:47.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:46 smithi045 conmon[33655]: mgr.y (mgr.14146) 302 : cluster [DBG] pgmap v308: 25 pgs: 25 active+clean; 99 MiB data, 232 MiB used, 715 GiB / 715 GiB avail; 16 MiB/s rd, 7.7 MiB/s wr, 37 op/s 2021-07-31T01:46:47.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:46 smithi045 conmon[33655]: cluster 2021-07-31T01:46:45.545131+0000 2021-07-31T01:46:47.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:46 smithi045 conmon[33655]: mon.a (mon.0) 708 : cluster [DBG] osdmap e106: 8 total, 8 up, 8 in 2021-07-31T01:46:47.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:46 smithi045 conmon[33655]: cluster 2021- 2021-07-31T01:46:47.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:46 smithi045 conmon[33655]: 07-31T01:46:46.476575+0000 2021-07-31T01:46:47.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:46 smithi045 conmon[33655]: mon.a (mon.0) 709 : 2021-07-31T01:46:47.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:46 smithi045 conmon[33655]: cluster [DBG] osdmap e107: 8 total, 8 up, 8 in 2021-07-31T01:46:47.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:46 smithi045 conmon[39718]: cluster 2021-07-31T01:46:45.373178+0000 mgr.y (mgr 2021-07-31T01:46:47.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:46 smithi045 conmon[39718]: .14146) 302 : cluster [DBG] pgmap v308: 25 pgs: 25 active+clean; 99 MiB data, 232 MiB used, 715 GiB / 715 GiB avail; 16 MiB/s rd, 7.7 MiB/s wr, 37 op/s 2021-07-31T01:46:47.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:46 smithi045 conmon[39718]: cluster 2021-07-31T01:46:45.545131 2021-07-31T01:46:47.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:46 smithi045 conmon[39718]: +0000 mon.a (mon.0) 708 : cluster [DBG] osdmap e106: 8 total, 8 up, 8 in 2021-07-31T01:46:47.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:46 smithi045 conmon[39718]: cluster 2021-07 2021-07-31T01:46:47.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:46 smithi045 conmon[39718]: -31T01:46:46.476575+0000 mon.a (mon.0) 709 : cluster [DBG] osdmap e107: 8 total, 8 up, 8 in 2021-07-31T01:46:47.485 INFO:tasks.rados.rados.0.smithi045.stdout:367: done (1 left) 2021-07-31T01:46:47.486 INFO:tasks.rados.rados.0.smithi045.stdout:368: rollback oid 45 current snap is 28 2021-07-31T01:46:47.486 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 45 to 22 2021-07-31T01:46:47.486 INFO:tasks.rados.rados.0.smithi045.stdout:369: read oid 20 snap -1 2021-07-31T01:46:47.487 INFO:tasks.rados.rados.0.smithi045.stdout:369: expect deleted 2021-07-31T01:46:47.487 INFO:tasks.rados.rados.0.smithi045.stdout:370: read oid 37 snap -1 2021-07-31T01:46:47.487 INFO:tasks.rados.rados.0.smithi045.stdout:370: expect (ObjNum 98 snap 16 seq_num 98) 2021-07-31T01:46:47.487 INFO:tasks.rados.rados.0.smithi045.stdout:371: delete oid 17 current snap is 28 2021-07-31T01:46:47.487 INFO:tasks.rados.rados.0.smithi045.stdout:366: finishing copy_from to smithi04569269-34 2021-07-31T01:46:47.488 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 34 v 2 (ObjNum 45 snap 0 seq_num 45) dirty exists 2021-07-31T01:46:47.488 INFO:tasks.rados.rados.0.smithi045.stdout:366: finishing copy_from racing read to smithi04569269-34 2021-07-31T01:46:47.491 INFO:tasks.rados.rados.0.smithi045.stdout:368: finishing rollback tid 0 to smithi04569269-45 2021-07-31T01:46:47.491 INFO:tasks.rados.rados.0.smithi045.stdout:368: finishing rollback tid 1 to smithi04569269-45 2021-07-31T01:46:47.493 INFO:tasks.rados.rados.0.smithi045.stdout:368: finishing rollback tid 2 to smithi04569269-45 2021-07-31T01:46:47.493 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 45 v 119 (ObjNum 1567451484 snap 1651003754 seq_num 1851092577) dirty exists 2021-07-31T01:46:47.493 INFO:tasks.rados.rados.0.smithi045.stdout:366: done (4 left) 2021-07-31T01:46:47.493 INFO:tasks.rados.rados.0.smithi045.stdout:368: done (3 left) 2021-07-31T01:46:47.493 INFO:tasks.rados.rados.0.smithi045.stdout:369: done (2 left) 2021-07-31T01:46:47.494 INFO:tasks.rados.rados.0.smithi045.stdout:371: done (1 left) 2021-07-31T01:46:47.494 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 18 current snap is 28 2021-07-31T01:46:47.494 INFO:tasks.rados.rados.0.smithi045.stdout:372: seq_num 132 ranges {2596864=614400,3211264=786432,3997696=565248,4562944=253952} 2021-07-31T01:46:47.511 INFO:tasks.rados.rados.0.smithi045.stdout:372: writing smithi04569269-18 from 2596864 to 3211264 tid 1 2021-07-31T01:46:47.515 INFO:tasks.rados.rados.0.smithi045.stdout:372: writing smithi04569269-18 from 3211264 to 3997696 tid 2 2021-07-31T01:46:47.518 INFO:tasks.rados.rados.0.smithi045.stdout:372: writing smithi04569269-18 from 3997696 to 4562944 tid 3 2021-07-31T01:46:47.520 INFO:tasks.rados.rados.0.smithi045.stdout:372: writing smithi04569269-18 from 4562944 to 4816896 tid 4 2021-07-31T01:46:47.520 INFO:tasks.rados.rados.0.smithi045.stdout:373: delete oid 49 current snap is 28 2021-07-31T01:46:47.520 INFO:tasks.rados.rados.0.smithi045.stdout:370: expect (ObjNum 98 snap 16 seq_num 98) 2021-07-31T01:46:47.566 INFO:tasks.rados.rados.0.smithi045.stdout:372: finishing write tid 1 to smithi04569269-18 2021-07-31T01:46:47.567 INFO:tasks.rados.rados.0.smithi045.stdout:372: finishing write tid 2 to smithi04569269-18 2021-07-31T01:46:47.567 INFO:tasks.rados.rados.0.smithi045.stdout:372: finishing write tid 3 to smithi04569269-18 2021-07-31T01:46:47.567 INFO:tasks.rados.rados.0.smithi045.stdout:372: finishing write tid 4 to smithi04569269-18 2021-07-31T01:46:47.567 INFO:tasks.rados.rados.0.smithi045.stdout:370: done (2 left) 2021-07-31T01:46:47.568 INFO:tasks.rados.rados.0.smithi045.stdout:373: done (1 left) 2021-07-31T01:46:47.568 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 37 current snap is 28 2021-07-31T01:46:47.568 INFO:tasks.rados.rados.0.smithi045.stdout:372: finishing write tid 5 to smithi04569269-18 2021-07-31T01:46:47.568 INFO:tasks.rados.rados.0.smithi045.stdout:372: finishing write tid 6 to smithi04569269-18 2021-07-31T01:46:47.568 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 18 v 75 (ObjNum 132 snap 28 seq_num 132) dirty exists 2021-07-31T01:46:47.569 INFO:tasks.rados.rados.0.smithi045.stdout:372: left oid 18 (ObjNum 132 snap 28 seq_num 132) 2021-07-31T01:46:47.569 INFO:tasks.rados.rados.0.smithi045.stdout:374: seq_num 134 ranges {1654784=647168,2301952=548864,2850816=573440,3424256=483328} 2021-07-31T01:46:47.578 INFO:tasks.rados.rados.0.smithi045.stdout:374: writing smithi04569269-37 from 1654784 to 2301952 tid 1 2021-07-31T01:46:47.581 INFO:tasks.rados.rados.0.smithi045.stdout:374: writing smithi04569269-37 from 2301952 to 2850816 tid 2 2021-07-31T01:46:47.585 INFO:tasks.rados.rados.0.smithi045.stdout:374: writing smithi04569269-37 from 2850816 to 3424256 tid 3 2021-07-31T01:46:47.588 INFO:tasks.rados.rados.0.smithi045.stdout:374: writing smithi04569269-37 from 3424256 to 3907584 tid 4 2021-07-31T01:46:47.588 INFO:tasks.rados.rados.0.smithi045.stdout:372: done (1 left) 2021-07-31T01:46:47.588 INFO:tasks.rados.rados.0.smithi045.stdout:375: copy_from oid 21 from oid 32 current snap is 28 2021-07-31T01:46:47.589 INFO:tasks.rados.rados.0.smithi045.stdout:374: finishing write tid 1 to smithi04569269-37 2021-07-31T01:46:47.589 INFO:tasks.rados.rados.0.smithi045.stdout:374: finishing write tid 2 to smithi04569269-37 2021-07-31T01:46:47.589 INFO:tasks.rados.rados.0.smithi045.stdout:374: finishing write tid 3 to smithi04569269-37 2021-07-31T01:46:47.589 INFO:tasks.rados.rados.0.smithi045.stdout:376: setattr oid 43 current snap is 28 2021-07-31T01:46:47.591 INFO:tasks.rados.rados.0.smithi045.stdout:375: finishing copy_from to smithi04569269-21 2021-07-31T01:46:47.591 INFO:tasks.rados.rados.0.smithi045.stdout:375: got expected ENOENT (src dne) 2021-07-31T01:46:47.592 INFO:tasks.rados.rados.0.smithi045.stdout:375: finishing copy_from racing read to smithi04569269-21 2021-07-31T01:46:47.592 INFO:tasks.rados.rados.0.smithi045.stdout:375: done (2 left) 2021-07-31T01:46:47.592 INFO:tasks.rados.rados.0.smithi045.stdout:377: read oid 30 snap -1 2021-07-31T01:46:47.592 INFO:tasks.rados.rados.0.smithi045.stdout:377: expect (ObjNum 29 snap 0 seq_num 29) 2021-07-31T01:46:47.593 INFO:tasks.rados.rados.0.smithi045.stdout:378: snap_create 2021-07-31T01:46:47.593 INFO:tasks.rados.rados.0.smithi045.stdout:374: finishing write tid 4 to smithi04569269-37 2021-07-31T01:46:47.593 INFO:tasks.rados.rados.0.smithi045.stdout:374: finishing write tid 5 to smithi04569269-37 2021-07-31T01:46:47.594 INFO:tasks.rados.rados.0.smithi045.stdout:377: expect (ObjNum 29 snap 0 seq_num 29) 2021-07-31T01:46:47.604 INFO:tasks.rados.rados.0.smithi045.stdout:374: finishing write tid 6 to smithi04569269-37 2021-07-31T01:46:47.605 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 37 v 80 (ObjNum 134 snap 28 seq_num 134) dirty exists 2021-07-31T01:46:47.605 INFO:tasks.rados.rados.0.smithi045.stdout:374: left oid 37 (ObjNum 134 snap 28 seq_num 134) 2021-07-31T01:46:47.605 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 43 v 77 (ObjNum 100 snap 16 seq_num 100) dirty exists 2021-07-31T01:46:48.549 INFO:tasks.rados.rados.0.smithi045.stdout:374: done (3 left) 2021-07-31T01:46:48.549 INFO:tasks.rados.rados.0.smithi045.stdout:376: done (2 left) 2021-07-31T01:46:48.549 INFO:tasks.rados.rados.0.smithi045.stdout:377: done (1 left) 2021-07-31T01:46:48.549 INFO:tasks.rados.rados.0.smithi045.stdout:378: done (0 left) 2021-07-31T01:46:48.550 INFO:tasks.rados.rados.0.smithi045.stdout:379: read oid 31 snap -1 2021-07-31T01:46:48.550 INFO:tasks.rados.rados.0.smithi045.stdout:379: expect (ObjNum 130 snap 27 seq_num 130) 2021-07-31T01:46:48.550 INFO:tasks.rados.rados.0.smithi045.stdout:380: copy_from oid 13 from oid 33 current snap is 29 2021-07-31T01:46:48.550 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 27 current snap is 29 2021-07-31T01:46:48.551 INFO:tasks.rados.rados.0.smithi045.stdout:381: seq_num 135 ranges {1638400=663552,2301952=434176,2736128=786432,3522560=385024} 2021-07-31T01:46:48.561 INFO:tasks.rados.rados.0.smithi045.stdout:381: writing smithi04569269-27 from 1638400 to 2301952 tid 1 2021-07-31T01:46:48.563 INFO:tasks.rados.rados.0.smithi045.stdout:381: writing smithi04569269-27 from 2301952 to 2736128 tid 2 2021-07-31T01:46:48.568 INFO:tasks.rados.rados.0.smithi045.stdout:381: writing smithi04569269-27 from 2736128 to 3522560 tid 3 2021-07-31T01:46:48.570 INFO:tasks.rados.rados.0.smithi045.stdout:381: writing smithi04569269-27 from 3522560 to 3907584 tid 4 2021-07-31T01:46:48.571 INFO:tasks.rados.rados.0.smithi045.stdout:382: copy_from oid 5 from oid 22 current snap is 29 2021-07-31T01:46:48.571 INFO:tasks.rados.rados.0.smithi045.stdout:379: expect (ObjNum 130 snap 27 seq_num 130) 2021-07-31T01:46:48.651 INFO:tasks.rados.rados.0.smithi045.stdout:381: finishing write tid 1 to smithi04569269-27 2021-07-31T01:46:48.651 INFO:tasks.rados.rados.0.smithi045.stdout:381: finishing write tid 2 to smithi04569269-27 2021-07-31T01:46:48.651 INFO:tasks.rados.rados.0.smithi045.stdout:380: finishing copy_from to smithi04569269-13 2021-07-31T01:46:48.652 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 13 v 71 (ObjNum 112 snap 20 seq_num 112) dirty exists 2021-07-31T01:46:48.652 INFO:tasks.rados.rados.0.smithi045.stdout:380: finishing copy_from racing read to smithi04569269-13 2021-07-31T01:46:48.652 INFO:tasks.rados.rados.0.smithi045.stdout:381: finishing write tid 3 to smithi04569269-27 2021-07-31T01:46:48.653 INFO:tasks.rados.rados.0.smithi045.stdout:379: done (3 left) 2021-07-31T01:46:48.653 INFO:tasks.rados.rados.0.smithi045.stdout:380: done (2 left) 2021-07-31T01:46:48.653 INFO:tasks.rados.rados.0.smithi045.stdout:383: snap_create 2021-07-31T01:46:48.654 INFO:tasks.rados.rados.0.smithi045.stdout:381: finishing write tid 4 to smithi04569269-27 2021-07-31T01:46:48.654 INFO:tasks.rados.rados.0.smithi045.stdout:381: finishing write tid 5 to smithi04569269-27 2021-07-31T01:46:48.654 INFO:tasks.rados.rados.0.smithi045.stdout:381: finishing write tid 6 to smithi04569269-27 2021-07-31T01:46:48.655 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 27 v 83 (ObjNum 135 snap 29 seq_num 135) dirty exists 2021-07-31T01:46:48.655 INFO:tasks.rados.rados.0.smithi045.stdout:381: left oid 27 (ObjNum 135 snap 29 seq_num 135) 2021-07-31T01:46:48.655 INFO:tasks.rados.rados.0.smithi045.stdout:382: finishing copy_from to smithi04569269-5 2021-07-31T01:46:48.656 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 5 v 13 (ObjNum 47 snap 0 seq_num 47) dirty exists 2021-07-31T01:46:48.656 INFO:tasks.rados.rados.0.smithi045.stdout:382: finishing copy_from racing read to smithi04569269-5 2021-07-31T01:46:48.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:48 smithi154 conmon[34553]: cluster 2021-07-31T01:46:47.373540+0000 mgr.y (mgr.14146) 303 2021-07-31T01:46:48.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:48 smithi154 conmon[34553]: : cluster [DBG] pgmap v311: 25 pgs: 25 active+clean; 99 MiB data, 237 MiB used, 715 GiB / 715 GiB avail; 7.5 MiB/s rd, 4.3 MiB/s wr, 17 op/s 2021-07-31T01:46:48.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:48 smithi154 conmon[34553]: cluster 2021-07-31T01:46:47.484508+0000 mon.a (mon.0) 710 : cluster [DBG] 2021-07-31T01:46:48.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:48 smithi154 conmon[34553]: osdmap e108: 8 total, 8 up, 8 in 2021-07-31T01:46:48.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:48 smithi045 conmon[39718]: cluster 2021-07-31T01:46:47.373540+0000 mgr.y (mgr.14146) 303 : cluster [DBG] pgmap v311: 25 pgs: 25 active+clean; 99 MiB data, 237 MiB used, 715 GiB / 715 GiB avail; 7.5 MiB/s rd, 4.3 MiB/s wr, 17 op/s 2021-07-31T01:46:48.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:48 smithi045 conmon[39718]: 2021-07-31T01:46:48.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:48 smithi045 conmon[39718]: cluster 2021-07-31T01:46:47.484508+0000 mon.a (mon.0) 710 : cluster [DBG] osdmap e108: 8 total, 8 up, 8 in 2021-07-31T01:46:48.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:48 smithi045 conmon[33655]: cluster 2021-07-31T01:46:47.373540+0000 mgr.y 2021-07-31T01:46:48.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:48 smithi045 conmon[33655]: (mgr.14146) 303 : cluster [DBG] pgmap v311: 25 pgs: 25 active+clean; 99 MiB data, 237 MiB used, 715 GiB / 715 GiB avail; 7.5 MiB/s rd, 4.3 MiB/s wr, 17 op/s 2021-07-31T01:46:48.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:48 smithi045 conmon[33655]: cluster 2021-07-31T01:46:47.484508+0000 mon.a (mon.0) 710 : cluster [DBG] osdmap e108: 8 total, 8 up, 8 in 2021-07-31T01:46:49.554 INFO:tasks.rados.rados.0.smithi045.stdout:381: done (2 left) 2021-07-31T01:46:49.554 INFO:tasks.rados.rados.0.smithi045.stdout:382: done (1 left) 2021-07-31T01:46:49.554 INFO:tasks.rados.rados.0.smithi045.stdout:383: done (0 left) 2021-07-31T01:46:49.555 INFO:tasks.rados.rados.0.smithi045.stdout:384: read oid 36 snap -1 2021-07-31T01:46:49.555 INFO:tasks.rados.rados.0.smithi045.stdout:384: expect (ObjNum 129 snap 27 seq_num 129) 2021-07-31T01:46:49.555 INFO:tasks.rados.rados.0.smithi045.stdout:385: delete oid 21 current snap is 30 2021-07-31T01:46:49.566 INFO:tasks.rados.rados.0.smithi045.stdout:384: expect (ObjNum 129 snap 27 seq_num 129) 2021-07-31T01:46:49.655 INFO:tasks.rados.rados.0.smithi045.stdout:384: done (1 left) 2021-07-31T01:46:49.655 INFO:tasks.rados.rados.0.smithi045.stdout:385: done (0 left) 2021-07-31T01:46:49.655 INFO:tasks.rados.rados.0.smithi045.stdout:386: rmattr oid 22 current snap is 30 2021-07-31T01:46:49.655 INFO:tasks.rados.rados.0.smithi045.stdout:387: rmattr oid 33 current snap is 30 2021-07-31T01:46:49.656 INFO:tasks.rados.rados.0.smithi045.stdout:387: done (1 left) 2021-07-31T01:46:49.656 INFO:tasks.rados.rados.0.smithi045.stdout:388: rmattr oid 6 current snap is 30 2021-07-31T01:46:49.656 INFO:tasks.rados.rados.0.smithi045.stdout:389: copy_from oid 12 from oid 11 current snap is 30 2021-07-31T01:46:49.656 INFO:tasks.rados.rados.0.smithi045.stdout:390: delete oid 7 current snap is 30 2021-07-31T01:46:49.657 INFO:tasks.rados.rados.0.smithi045.stdout:390: done (3 left) 2021-07-31T01:46:49.658 INFO:tasks.rados.rados.0.smithi045.stdout:391: rollback oid 44 current snap is 30 2021-07-31T01:46:49.658 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 44 to 22 2021-07-31T01:46:49.658 INFO:tasks.rados.rados.0.smithi045.stdout:392: read oid 18 snap -1 2021-07-31T01:46:49.658 INFO:tasks.rados.rados.0.smithi045.stdout:392: expect (ObjNum 132 snap 28 seq_num 132) 2021-07-31T01:46:49.659 INFO:tasks.rados.rados.0.smithi045.stdout:393: snap_remove snap 23 2021-07-31T01:46:50.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:49 smithi154 conmon[34553]: cluster 2021-07-31T01:46:48.547986+0000 mon.a (mon.0 2021-07-31T01:46:50.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:49 smithi154 conmon[34553]: ) 711 : cluster [DBG] osdmap e109: 8 total, 8 up, 8 in 2021-07-31T01:46:50.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:49 smithi045 conmon[39718]: cluster 2021-07-31T01:46:48.547986+0000 mon.a (mon.0) 2021-07-31T01:46:50.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:49 smithi045 conmon[39718]: 711 : cluster [DBG] osdmap e109: 8 total, 8 up, 8 in 2021-07-31T01:46:50.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:49 smithi045 conmon[33655]: cluster 2021-07-31T01:46:50.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:49 smithi045 conmon[33655]: 2021-07 2021-07-31T01:46:50.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:49 smithi045 conmon[33655]: -31T01:46:48.547986+0000 mon.a (mon.0) 711 : cluster [DBG] osdmap e109: 8 total, 8 up, 8 in 2021-07-31T01:46:50.559 INFO:tasks.rados.rados.0.smithi045.stdout:393: done (5 left) 2021-07-31T01:46:50.559 INFO:tasks.rados.rados.0.smithi045.stdout:394: delete oid 46 current snap is 30 2021-07-31T01:46:50.559 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 22 v 71 (ObjNum 47 snap 0 seq_num 47) dirty exists 2021-07-31T01:46:50.559 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 6 v 80 (ObjNum 1482185829 snap 1800825175 seq_num 1315465582) dirty exists 2021-07-31T01:46:50.560 INFO:tasks.rados.rados.0.smithi045.stdout:391: finishing rollback tid 0 to smithi04569269-44 2021-07-31T01:46:50.560 INFO:tasks.rados.rados.0.smithi045.stdout:391: finishing rollback tid 1 to smithi04569269-44 2021-07-31T01:46:50.560 INFO:tasks.rados.rados.0.smithi045.stdout:391: finishing rollback tid 2 to smithi04569269-44 2021-07-31T01:46:50.560 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 44 v 87 (ObjNum 114 snap 21 seq_num 114) dirty exists 2021-07-31T01:46:50.561 INFO:tasks.rados.rados.0.smithi045.stdout:389: finishing copy_from to smithi04569269-12 2021-07-31T01:46:50.561 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 12 v 4 (ObjNum 10 snap 0 seq_num 10) dirty exists 2021-07-31T01:46:50.561 INFO:tasks.rados.rados.0.smithi045.stdout:389: finishing copy_from racing read to smithi04569269-12 2021-07-31T01:46:50.562 INFO:tasks.rados.rados.0.smithi045.stdout:392: expect (ObjNum 132 snap 28 seq_num 132) 2021-07-31T01:46:50.757 INFO:tasks.rados.rados.0.smithi045.stdout:386: done (5 left) 2021-07-31T01:46:50.758 INFO:tasks.rados.rados.0.smithi045.stdout:388: done (4 left) 2021-07-31T01:46:50.758 INFO:tasks.rados.rados.0.smithi045.stdout:389: done (3 left) 2021-07-31T01:46:50.758 INFO:tasks.rados.rados.0.smithi045.stdout:391: done (2 left) 2021-07-31T01:46:50.758 INFO:tasks.rados.rados.0.smithi045.stdout:392: done (1 left) 2021-07-31T01:46:50.759 INFO:tasks.rados.rados.0.smithi045.stdout:394: done (0 left) 2021-07-31T01:46:50.759 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 12 current snap is 30 2021-07-31T01:46:50.760 INFO:tasks.rados.rados.0.smithi045.stdout:395: seq_num 139 ranges {172032=737280,909312=425984,1335296=761856,2097152=417792} 2021-07-31T01:46:50.762 INFO:tasks.rados.rados.0.smithi045.stdout:395: writing smithi04569269-12 from 172032 to 909312 tid 1 2021-07-31T01:46:50.764 INFO:tasks.rados.rados.0.smithi045.stdout:395: writing smithi04569269-12 from 909312 to 1335296 tid 2 2021-07-31T01:46:50.768 INFO:tasks.rados.rados.0.smithi045.stdout:395: writing smithi04569269-12 from 1335296 to 2097152 tid 3 2021-07-31T01:46:50.771 INFO:tasks.rados.rados.0.smithi045.stdout:395: writing smithi04569269-12 from 2097152 to 2514944 tid 4 2021-07-31T01:46:50.771 INFO:tasks.rados.rados.0.smithi045.stdout:396: snap_remove snap 28 2021-07-31T01:46:51.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:50 smithi154 conmon[34553]: cluster 2021-07-31T01:46:49.374139+0000 mgr.y ( 2021-07-31T01:46:51.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:50 smithi154 conmon[34553]: mgr.14146) 304 : cluster [DBG] pgmap v314: 25 pgs: 25 active+clean; 109 MiB data, 261 MiB used, 715 GiB / 715 GiB avail; 3.5 MiB/s rd, 3.2 MiB/s wr, 23 op/s 2021-07-31T01:46:51.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:50 smithi154 conmon[34553]: cluster 2021-07-31T01:46:49.553109+0000 mon.a (mon.0) 712 : cluster [DBG] osdmap e110: 8 total, 8 up, 8 in 2021-07-31T01:46:51.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:50 smithi045 conmon[39718]: cluster 2021-07-31T01:46:49 2021-07-31T01:46:51.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:50 smithi045 conmon[39718]: .374139+0000 mgr.y (mgr.14146) 304 : cluster [DBG] pgmap v314: 25 pgs: 25 active+clean; 109 MiB data, 261 MiB used, 715 GiB / 715 GiB avail; 3.5 MiB/s rd, 3.2 MiB/s wr, 23 op/s 2021-07-31T01:46:51.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:50 smithi045 conmon[39718]: cluster 2021-07-31T01:46:51.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:50 smithi045 conmon[39718]: 2021-07-31T01: 2021-07-31T01:46:51.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:50 smithi045 conmon[39718]: 46:49.553109+0000 mon.a 2021-07-31T01:46:51.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:50 smithi045 conmon[39718]: (mon.0) 712 : cluster 2021-07-31T01:46:51.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:50 smithi045 conmon[39718]: [DBG] osdmap e110: 8 total, 8 up, 8 in 2021-07-31T01:46:51.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:50 smithi045 conmon[33655]: cluster 2021-07-31T01:46:49.374139+0000 mgr.y (mgr.14146) 304 : cluster [DBG] pgmap v314: 25 pgs: 25 active+clean; 109 MiB data, 261 MiB used, 715 GiB / 715 GiB avail; 3.5 MiB/s rd, 3.2 MiB/s wr, 23 op/s 2021-07-31T01:46:51.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:50 smithi045 conmon[33655]: cluster 2021-07-31T01:46:49.553109+0000 mon.a (mon.0) 712 : cluster [DBG] osdmap e110: 8 total, 8 up, 8 in 2021-07-31T01:46:51.564 INFO:tasks.rados.rados.0.smithi045.stdout:396: done (1 left) 2021-07-31T01:46:51.564 INFO:tasks.rados.rados.0.smithi045.stdout:397: snap_remove snap 24 2021-07-31T01:46:52.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:51 smithi154 conmon[34553]: cluster 2021-07-31T01:46:50.557996+0000 mon.a (mon.0) 713 : cluster 2021-07-31T01:46:52.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:51 smithi154 conmon[34553]: [DBG] osdmap e111: 8 total, 8 up, 8 in 2021-07-31T01:46:52.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:51 smithi045 conmon[39718]: cluster 2021-07-31T01:46:50.557996 2021-07-31T01:46:52.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:51 smithi045 conmon[39718]: +0000 mon.a (mon.0) 713 : cluster [DBG] osdmap e111: 8 total, 8 up, 8 in 2021-07-31T01:46:52.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:51 smithi045 conmon[33655]: cluster 2021-07-31T01:46:50.557996+0000 mon.a (mon.0) 2021-07-31T01:46:52.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:51 smithi045 conmon[33655]: 713 : cluster [DBG] osdmap e111: 8 total, 8 up, 8 in 2021-07-31T01:46:52.565 INFO:tasks.rados.rados.0.smithi045.stdout:397: done (1 left) 2021-07-31T01:46:52.566 INFO:tasks.rados.rados.0.smithi045.stdout:398: snap_remove snap 21 2021-07-31T01:46:53.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:52 smithi154 conmon[34553]: cluster 2021-07-31T01:46:51.374568+0000 mgr.y (mgr.14146) 305 : cluster [DBG] pgmap v317: 25 pgs: 25 active+clean; 109 MiB data, 261 MiB used, 715 GiB / 715 GiB avail; 3.5 MiB/s rd, 3.2 MiB/s wr, 19 op/s 2021-07-31T01:46:53.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:52 smithi154 conmon[34553]: cluster 2021-07-31T01:46:51.563422+0000 mon.a (mon.0) 714 : cluster [DBG] osdmap e112: 8 total, 8 up, 8 in 2021-07-31T01:46:53.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:52 smithi154 conmon[34553]: 2021-07-31T01:46:53.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:52 smithi045 conmon[33655]: cluster 2021-07-31T01:46:53.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:52 smithi045 conmon[33655]: 2021-07-31T01 2021-07-31T01:46:53.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:52 smithi045 conmon[33655]: :46:51.374568+0000 mgr.y (mgr.14146) 305 : cluster [DBG] pgmap v317: 25 pgs: 25 active+clean; 109 MiB data, 261 MiB used, 715 GiB / 715 GiB avail; 3.5 MiB/s rd, 3.2 MiB/s wr, 19 op/s 2021-07-31T01:46:53.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:52 smithi045 conmon[33655]: cluster 2021-07-31T01:46:53.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:52 smithi045 conmon[33655]: 2021-07-31T01:46:51.563422+0000 2021-07-31T01:46:53.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:52 smithi045 conmon[33655]: mon.a (mon.0) 714 : cluster [DBG] osdmap e112: 8 total, 8 up, 8 in 2021-07-31T01:46:53.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:52 smithi045 conmon[33655]: 2021-07-31T01:46:53.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:52 smithi045 conmon[39718]: cluster 2021-07-31T01:46:51.374568+0000 mgr.y 2021-07-31T01:46:53.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:52 smithi045 conmon[39718]: (mgr.14146) 305 : cluster [DBG] pgmap v317: 25 pgs: 25 active+clean; 109 MiB data, 261 MiB used, 715 GiB / 715 GiB avail; 3.5 MiB/s rd, 3.2 MiB/s wr, 19 op/s 2021-07-31T01:46:53.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:52 smithi045 conmon[39718]: cluster 2021-07- 2021-07-31T01:46:53.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:52 smithi045 conmon[39718]: 31T01:46:51.563422+0000 mon.a (mon.0) 714 : cluster [DBG] osdmap e112: 8 total, 8 up, 8 in 2021-07-31T01:46:53.576 INFO:tasks.rados.rados.0.smithi045.stdout:398: done (1 left) 2021-07-31T01:46:53.577 INFO:tasks.rados.rados.0.smithi045.stdout:399: snap_remove snap 20 2021-07-31T01:46:53.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:53 smithi045 conmon[39718]: cluster 2021-07-31T01:46:52.564867+0000 mon.a ( 2021-07-31T01:46:53.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:53 smithi045 conmon[39718]: mon.0) 715 : cluster [DBG] osdmap e113: 8 total, 8 up, 8 in 2021-07-31T01:46:53.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:46:53 smithi045 conmon[61210]: level=error ts=2021-07-31T01:46:53.505Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:46:53.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:46:53 smithi045 conmon[61210]: level=error ts=2021-07-31T01:46:53.505Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:46:53.778 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:46:53 smithi045 conmon[61210]: level=error ts=2021-07-31T01:46:53.506Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T01:46:53.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:53 smithi045 conmon[33655]: cluster 2021-07-31T01:46:52.564867+0000 mon.a (mon.0) 715 : cluster 2021-07-31T01:46:53.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:53 smithi045 conmon[33655]: [DBG] osdmap e113: 8 total, 8 up, 8 in 2021-07-31T01:46:54.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:53 smithi154 conmon[34553]: cluster 2021-07-31T01:46:52.564867+0000 mon.a (mon.0) 715 2021-07-31T01:46:54.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:53 smithi154 conmon[34553]: : cluster [DBG] osdmap e113: 8 total, 8 up, 8 in 2021-07-31T01:46:54.194 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T01:46:54.471 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T01:46:54.471 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 running (2m) 107s ago 2m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T01:46:54.471 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 running (2m) 109s ago 2m 37.1M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T01:46:54.471 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 running (5m) 109s ago 5m 377M - 16.2.5 a29040f3000e f9af406258e1 2021-07-31T01:46:54.472 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 running (7m) 107s ago 7m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T01:46:54.472 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 running (7m) 107s ago 8m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T01:46:54.473 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 running (6m) 109s ago 6m 92.9M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T01:46:54.473 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 running (6m) 107s ago 6m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T01:46:54.473 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 running (2m) 107s ago 2m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T01:46:54.473 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 running (2m) 109s ago 2m 8144k - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T01:46:54.473 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 running (5m) 107s ago 5m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T01:46:54.474 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 running (5m) 107s ago 5m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T01:46:54.474 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 running (4m) 107s ago 4m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T01:46:54.474 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 running (4m) 107s ago 4m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T01:46:54.474 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 running (4m) 109s ago 4m 43.3M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T01:46:54.475 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 running (3m) 109s ago 3m 41.6M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T01:46:54.475 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 running (3m) 109s ago 3m 41.2M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T01:46:54.475 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 running (3m) 109s ago 3m 42.6M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T01:46:54.475 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 running (119s) 109s ago 2m 28.9M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T01:46:54.584 INFO:tasks.rados.rados.0.smithi045.stdout:399: done (1 left) 2021-07-31T01:46:54.584 INFO:tasks.rados.rados.0.smithi045.stdout:400: setattr oid 21 current snap is 30 2021-07-31T01:46:54.585 INFO:tasks.rados.rados.0.smithi045.stdout:395: finishing write tid 1 to smithi04569269-12 2021-07-31T01:46:54.585 INFO:tasks.rados.rados.0.smithi045.stdout:395: finishing write tid 2 to smithi04569269-12 2021-07-31T01:46:54.585 INFO:tasks.rados.rados.0.smithi045.stdout:395: finishing write tid 3 to smithi04569269-12 2021-07-31T01:46:54.585 INFO:tasks.rados.rados.0.smithi045.stdout:395: finishing write tid 4 to smithi04569269-12 2021-07-31T01:46:54.586 INFO:tasks.rados.rados.0.smithi045.stdout:395: finishing write tid 5 to smithi04569269-12 2021-07-31T01:46:54.586 INFO:tasks.rados.rados.0.smithi045.stdout:395: finishing write tid 6 to smithi04569269-12 2021-07-31T01:46:54.586 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 12 v 93 (ObjNum 139 snap 30 seq_num 139) dirty exists 2021-07-31T01:46:54.586 INFO:tasks.rados.rados.0.smithi045.stdout:395: left oid 12 (ObjNum 139 snap 30 seq_num 139) 2021-07-31T01:46:54.591 INFO:tasks.rados.rados.0.smithi045.stdout:395: done (1 left) 2021-07-31T01:46:54.591 INFO:tasks.rados.rados.0.smithi045.stdout:401: read oid 27 snap -1 2021-07-31T01:46:54.591 INFO:tasks.rados.rados.0.smithi045.stdout:401: expect (ObjNum 135 snap 29 seq_num 135) 2021-07-31T01:46:54.591 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 32 current snap is 30 2021-07-31T01:46:54.592 INFO:tasks.rados.rados.0.smithi045.stdout:402: seq_num 140 ranges {0=753664,753664=704512,1458176=524288,1982464=376832} 2021-07-31T01:46:54.594 INFO:tasks.rados.rados.0.smithi045.stdout:402: writing smithi04569269-32 from 0 to 753664 tid 1 2021-07-31T01:46:54.596 INFO:tasks.rados.rados.0.smithi045.stdout:402: writing smithi04569269-32 from 753664 to 1458176 tid 2 2021-07-31T01:46:54.599 INFO:tasks.rados.rados.0.smithi045.stdout:402: writing smithi04569269-32 from 1458176 to 1982464 tid 3 2021-07-31T01:46:54.601 INFO:tasks.rados.rados.0.smithi045.stdout:402: writing smithi04569269-32 from 1982464 to 2359296 tid 4 2021-07-31T01:46:54.602 INFO:tasks.rados.rados.0.smithi045.stdout:403: setattr oid 8 current snap is 30 2021-07-31T01:46:54.602 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 21 v 246 (ObjNum 0 snap 0 seq_num 0) dirty exists 2021-07-31T01:46:54.602 INFO:tasks.rados.rados.0.smithi045.stdout:402: finishing write tid 1 to smithi04569269-32 2021-07-31T01:46:54.602 INFO:tasks.rados.rados.0.smithi045.stdout:402: finishing write tid 2 to smithi04569269-32 2021-07-31T01:46:54.603 INFO:tasks.rados.rados.0.smithi045.stdout:400: done (3 left) 2021-07-31T01:46:54.604 INFO:tasks.rados.rados.0.smithi045.stdout:404: read oid 26 snap -1 2021-07-31T01:46:54.604 INFO:tasks.rados.rados.0.smithi045.stdout:404: expect (ObjNum 101 snap 16 seq_num 101) 2021-07-31T01:46:54.604 INFO:tasks.rados.rados.0.smithi045.stdout:405: snap_remove snap 19 2021-07-31T01:46:54.816 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T01:46:54.816 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T01:46:54.816 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T01:46:54.817 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T01:46:54.817 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T01:46:54.817 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 2 2021-07-31T01:46:54.817 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T01:46:54.817 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T01:46:54.818 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T01:46:54.818 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T01:46:54.819 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T01:46:54.819 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T01:46:54.819 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 13 2021-07-31T01:46:54.819 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T01:46:54.820 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T01:46:55.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:54 smithi154 conmon[34553]: cluster 2021-07-31T01:46:53.375024+0000 mgr.y (mgr.14146) 306 : cluster [DBG] pgmap v320: 25 pgs: 25 active+clean; 119 MiB data, 276 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 2.3 MiB/s wr, 12 op/s 2021-07-31T01:46:55.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:54 smithi154 conmon[34553]: 2021-07-31T01:46:55.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:54 smithi154 conmon[34553]: cluster 2021-07-31T01:46:53.575402+0000 mon.a (mon.0) 716 : cluster [DBG] osdmap e114: 8 total, 8 up, 8 in 2021-07-31T01:46:55.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:54 smithi045 conmon[39718]: cluster 2021-07-31T01:46:53.375024+0000 mgr.y (mgr.14146) 2021-07-31T01:46:55.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:54 smithi045 conmon[39718]: 306 : cluster [DBG] pgmap v320: 25 pgs: 25 active+clean; 119 MiB data, 276 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 2.3 MiB/s wr, 12 op/s 2021-07-31T01:46:55.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:54 smithi045 conmon[39718]: cluster 2021-07-31T01:46:53.575402+0000 mon.a (mon.0) 716 : cluster [DBG] osdmap e114: 8 total, 8 up, 8 in 2021-07-31T01:46:55.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:54 smithi045 conmon[33655]: cluster 2021-07-31T01:46:53.375024+0000 2021-07-31T01:46:55.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:54 smithi045 conmon[33655]: mgr.y (mgr.14146) 306 : cluster [DBG] pgmap v320: 25 pgs: 25 active+clean; 119 MiB data, 276 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 2.3 MiB/s wr, 12 op/s 2021-07-31T01:46:55.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:54 smithi045 conmon[33655]: cluster 2021-07-31T01:46:53.575402+0000 mon.a (mon.0) 716 : cluster [DBG] osdmap e114: 8 total, 8 up, 8 in 2021-07-31T01:46:55.598 INFO:tasks.rados.rados.0.smithi045.stdout:402: finishing write tid 3 to smithi04569269-32 2021-07-31T01:46:55.599 INFO:tasks.rados.rados.0.smithi045.stdout:402: finishing write tid 4 to smithi04569269-32 2021-07-31T01:46:55.599 INFO:tasks.rados.rados.0.smithi045.stdout:402: finishing write tid 5 to smithi04569269-32 2021-07-31T01:46:55.599 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 8 v 256 (ObjNum 0 snap 32731 seq_num 3189069520) dirty exists 2021-07-31T01:46:55.599 INFO:tasks.rados.rados.0.smithi045.stdout:401: expect (ObjNum 135 snap 29 seq_num 135) 2021-07-31T01:46:55.735 INFO:tasks.rados.rados.0.smithi045.stdout:402: finishing write tid 6 to smithi04569269-32 2021-07-31T01:46:55.735 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 32 v 58 (ObjNum 140 snap 30 seq_num 140) dirty exists 2021-07-31T01:46:55.735 INFO:tasks.rados.rados.0.smithi045.stdout:402: left oid 32 (ObjNum 140 snap 30 seq_num 140) 2021-07-31T01:46:55.735 INFO:tasks.rados.rados.0.smithi045.stdout:404: expect (ObjNum 101 snap 16 seq_num 101) 2021-07-31T01:46:55.781 INFO:tasks.rados.rados.0.smithi045.stdout:401: done (4 left) 2021-07-31T01:46:55.781 INFO:tasks.rados.rados.0.smithi045.stdout:402: done (3 left) 2021-07-31T01:46:55.782 INFO:tasks.rados.rados.0.smithi045.stdout:403: done (2 left) 2021-07-31T01:46:55.782 INFO:tasks.rados.rados.0.smithi045.stdout:404: done (1 left) 2021-07-31T01:46:55.782 INFO:tasks.rados.rados.0.smithi045.stdout:405: done (0 left) 2021-07-31T01:46:55.782 INFO:tasks.rados.rados.0.smithi045.stdout:406: snap_remove snap 29 2021-07-31T01:46:56.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:55 smithi154 conmon[34553]: audit 2021-07-31T01:46:54.182397+0000 mgr.y (mgr. 2021-07-31T01:46:56.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:55 smithi154 conmon[34553]: 14146) 307 : audit [DBG] from='client.24565 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:46:56.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:55 smithi154 conmon[34553]: audit 2021-07-31T01:46:54.462801+0000 mgr.y (mgr.14146) 308 : audit [DBG] from='client.14664 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:46:56.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:55 smithi154 conmon[34553]: cluster 2021-07-31T01 2021-07-31T01:46:56.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:55 smithi154 conmon[34553]: :46:54.576325+0000 mon.a (mon.0) 717 : cluster [DBG] osdmap e115: 8 total, 8 up, 8 in 2021-07-31T01:46:56.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:55 smithi154 conmon[34553]: audit 2021 2021-07-31T01:46:56.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:55 smithi154 conmon[34553]: -07-31T01:46:54.815916+0000 mon.c (mon. 2021-07-31T01:46:56.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:55 smithi154 conmon[34553]: 1) 20 : audit [DBG] from='client.? 172.21.15.45:0/390797325' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T01:46:56.005 INFO:journalctl@ceph.osd.7.smithi154.stdout:Jul 31 01:46:55 smithi154 conmon[49149]: debug 2021-07-31T01:46:55.598+0000 7f8a75203700 -1 osd.7 pg_epoch: 116 pg[3.es0( v 116'115 (0'0,116'115] local-lis/les=56/57 n=9 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [7,4,1]p7(0) r=0 lpr=56 luod=115'113 crt=115'113 lcod 115'111 mlcod 115'111 active+clean+snaptrim trimq=[25~1] ps=[28~2,2c~3,32~1]] removing snap head 2021-07-31T01:46:56.026 INFO:journalctl@ceph.osd.3.smithi045.stdout:Jul 31 01:46:55 smithi045 conmon[54013]: debug 2021-07-31T01:46:56.027 INFO:journalctl@ceph.osd.3.smithi045.stdout:Jul 31 01:46:55 smithi045 conmon[54013]: 2021-07-31T01:46:55.609+0000 7f64f88c5700 -1 osd.3 pg_epoch: 116 pg[3.bs0( v 116'258 (0'0,116'258] local-lis/les=56/57 n=20 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [3,0,4]p3(0) r=0 lpr=56 luod=115'256 crt=115'256 lcod 115'254 mlcod 115'254 active+clean+snaptrim trimq=[25~1] ps=[28~2,2c~3,32~1]] removing snap head 2021-07-31T01:46:56.028 INFO:journalctl@ceph.osd.3.smithi045.stdout:Jul 31 01:46:55 smithi045 conmon[54013]: 2021-07-31T01:46:56.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:55 smithi045 conmon[39718]: audit 2021-07-31T01:46:54.182397+0000 mgr.y (mgr 2021-07-31T01:46:56.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:55 smithi045 conmon[39718]: .14146) 307 : audit [DBG] from='client.24565 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:46:56.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:55 smithi045 conmon[39718]: audit 2021-07-31T01:46:54.462801+0000 mgr.y (mgr.14146) 308 : audit 2021-07-31T01:46:56.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:55 smithi045 conmon[39718]: [DBG] from='client.14664 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:46:56.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:55 smithi045 conmon[39718]: cluster 2021-07-31T01:46:54.576325+0000 mon.a 2021-07-31T01:46:56.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:55 smithi045 conmon[39718]: (mon.0) 717 : cluster [DBG] osdmap e115: 8 total, 8 up, 8 in 2021-07-31T01:46:56.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:55 smithi045 conmon[39718]: audit 2021-07-31T01:46 2021-07-31T01:46:56.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:55 smithi045 conmon[39718]: :54.815916+0000 mon.c (mon.1) 20 : audit [DBG] from='client.? 172.21.15.45:0/390797325' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T01:46:56.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:55 smithi045 conmon[33655]: audit 2021-07-31T01:46:54.182397+0000 mgr.y (mgr.14146) 307 : 2021-07-31T01:46:56.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:55 smithi045 conmon[33655]: audit [DBG] from='client.24565 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:46:56.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:55 smithi045 conmon[33655]: audit 2021-07-31T01:46:54.462801+0000 mgr.y (mgr.14146) 308 : audit [DBG] from='client.14664 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:46:56.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:55 smithi045 conmon[33655]: cluster 2021-07-31T01:46:54.576325+0000 2021-07-31T01:46:56.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:55 smithi045 conmon[33655]: mon.a (mon.0) 717 : cluster [DBG] osdmap e115: 8 total, 8 up, 8 in 2021-07-31T01:46:56.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:55 smithi045 conmon[33655]: audit 2021-07-31 2021-07-31T01:46:56.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:55 smithi045 conmon[33655]: T01:46:54.815916+0000 mon.c (mon.1) 20 : audit [DBG] from='client.? 172.21.15.45:0/390797325' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T01:46:56.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:55 smithi045 conmon[33655]: 2021-07-31T01:46:56.480 INFO:tasks.rados.rados.0.smithi045.stdout:406: done (0 left) 2021-07-31T01:46:56.481 INFO:tasks.rados.rados.0.smithi045.stdout:407: read oid 36 snap -1 2021-07-31T01:46:56.481 INFO:tasks.rados.rados.0.smithi045.stdout:407: expect (ObjNum 129 snap 27 seq_num 129) 2021-07-31T01:46:56.481 INFO:tasks.rados.rados.0.smithi045.stdout:408: setattr oid 42 current snap is 30 2021-07-31T01:46:56.482 INFO:tasks.rados.rados.0.smithi045.stdout:409: setattr oid 11 current snap is 30 2021-07-31T01:46:56.484 INFO:tasks.rados.rados.0.smithi045.stdout:410: delete oid 15 current snap is 30 2021-07-31T01:46:56.491 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 42 v 159 (ObjNum 125 snap 24 seq_num 125) dirty exists 2021-07-31T01:46:56.493 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 11 v 141 (ObjNum 10 snap 0 seq_num 10) dirty exists 2021-07-31T01:46:56.497 INFO:tasks.rados.rados.0.smithi045.stdout:407: expect (ObjNum 129 snap 27 seq_num 129) 2021-07-31T01:46:56.587 INFO:tasks.rados.rados.0.smithi045.stdout:407: done (3 left) 2021-07-31T01:46:56.587 INFO:tasks.rados.rados.0.smithi045.stdout:408: done (2 left) 2021-07-31T01:46:56.587 INFO:tasks.rados.rados.0.smithi045.stdout:409: done (1 left) 2021-07-31T01:46:56.588 INFO:tasks.rados.rados.0.smithi045.stdout:410: done (0 left) 2021-07-31T01:46:56.588 INFO:tasks.rados.rados.0.smithi045.stdout:411: copy_from oid 35 from oid 7 current snap is 30 2021-07-31T01:46:56.588 INFO:tasks.rados.rados.0.smithi045.stdout:412: snap_remove snap 27 2021-07-31T01:46:57.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:56 smithi154 conmon[34553]: cluster 2021-07-31T01:46:55.375437+0000 mgr.y (mgr.14146) 309 2021-07-31T01:46:57.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:56 smithi154 conmon[34553]: : cluster [DBG] pgmap v323: 25 pgs: 25 active+clean; 119 MiB data, 290 MiB used, 715 GiB / 715 GiB avail; 5.5 MiB/s rd, 2.3 MiB/s wr, 16 op/s 2021-07-31T01:46:57.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:56 smithi154 conmon[34553]: cluster 2021-07-31T01:46:55.597448+0000 mon.a (mon.0) 718 : cluster [DBG] osdmap e116: 8 total, 8 up, 8 in 2021-07-31T01:46:57.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:56 smithi154 conmon[34553]: cluster 2021-07-31T01:46:56.479607+0000 mon.a (mon.0) 2021-07-31T01:46:57.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:56 smithi154 conmon[34553]: 719 : cluster [DBG] osdmap e117: 8 total, 8 up, 8 in 2021-07-31T01:46:57.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:56 smithi045 conmon[39718]: cluster 2021-07-31T01: 2021-07-31T01:46:57.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:56 smithi045 conmon[39718]: 46:55.375437+0000 mgr.y (mgr.14146) 309 : cluster [DBG] pgmap v323: 25 pgs: 25 active+clean; 119 MiB data, 290 MiB used, 715 GiB / 715 GiB avail; 5.5 MiB/s rd, 2.3 MiB/s wr, 16 op/s 2021-07-31T01:46:57.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:56 smithi045 conmon[39718]: cluster 2021- 2021-07-31T01:46:57.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:56 smithi045 conmon[39718]: 07-31T01:46:55 2021-07-31T01:46:57.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:56 smithi045 conmon[39718]: .597448+0000 mon.a (mon. 2021-07-31T01:46:57.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:56 smithi045 conmon[39718]: 0) 718 : cluster [DBG] 2021-07-31T01:46:57.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:56 smithi045 conmon[39718]: osdmap e116: 8 total, 8 up, 8 in 2021-07-31T01:46:57.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:56 smithi045 conmon[39718]: cluster 2021-07-31T01:46:57.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:56 smithi045 conmon[39718]: 2021-07-31T01: 2021-07-31T01:46:57.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:56 smithi045 conmon[39718]: 46:56.479607+0000 mon.a 2021-07-31T01:46:57.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:56 smithi045 conmon[39718]: (mon.0) 719 : cluster 2021-07-31T01:46:57.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:56 smithi045 conmon[39718]: [DBG] osdmap e117: 8 total, 8 up, 8 in 2021-07-31T01:46:57.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:56 smithi045 conmon[33655]: cluster 2021-07-31T01:46:55 2021-07-31T01:46:57.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:56 smithi045 conmon[33655]: .375437+0000 mgr.y (mgr.14146) 309 : cluster [DBG] pgmap v323: 25 pgs: 25 active+clean; 119 MiB data, 290 MiB used, 715 GiB / 715 GiB avail; 5.5 MiB/s rd, 2.3 MiB/s wr, 16 op/s 2021-07-31T01:46:57.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:56 smithi045 conmon[33655]: cluster 2021-07-31T01:46:57.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:56 smithi045 conmon[33655]: 2021-07-31T01:46:55.597448+0000 2021-07-31T01:46:57.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:56 smithi045 conmon[33655]: mon.a (mon.0) 718 2021-07-31T01:46:57.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:56 smithi045 conmon[33655]: : cluster [DBG] osdmap e116: 8 total, 8 up, 8 in 2021-07-31T01:46:57.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:56 smithi045 conmon[33655]: cluster 2021 2021-07-31T01:46:57.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:56 smithi045 conmon[33655]: -07-31T01:46:56 2021-07-31T01:46:57.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:56 smithi045 conmon[33655]: .479607+0000 mon.a (mon. 2021-07-31T01:46:57.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:56 smithi045 conmon[33655]: 0) 719 : cluster [DBG] osdmap e117: 8 total, 8 up, 8 in 2021-07-31T01:46:57.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:56 smithi045 conmon[33655]: 2021-07-31T01:46:57.487 INFO:tasks.rados.rados.0.smithi045.stdout:412: done (1 left) 2021-07-31T01:46:57.487 INFO:tasks.rados.rados.0.smithi045.stdout:413: copy_from oid 5 from oid 12 current snap is 30 2021-07-31T01:46:57.487 INFO:tasks.rados.rados.0.smithi045.stdout:411: finishing copy_from to smithi04569269-35 2021-07-31T01:46:57.488 INFO:tasks.rados.rados.0.smithi045.stdout:411: got expected ENOENT (src dne) 2021-07-31T01:46:57.488 INFO:tasks.rados.rados.0.smithi045.stdout:411: finishing copy_from racing read to smithi04569269-35 2021-07-31T01:46:57.488 INFO:tasks.rados.rados.0.smithi045.stdout:411: done (1 left) 2021-07-31T01:46:57.488 INFO:tasks.rados.rados.0.smithi045.stdout:414: read oid 22 snap -1 2021-07-31T01:46:57.489 INFO:tasks.rados.rados.0.smithi045.stdout:414: expect (ObjNum 47 snap 0 seq_num 47) 2021-07-31T01:46:57.489 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 44 current snap is 30 2021-07-31T01:46:57.489 INFO:tasks.rados.rados.0.smithi045.stdout:415: seq_num 142 ranges {1916928=786432,2703360=475136,3178496=540672,3719168=589824} 2021-07-31T01:46:57.502 INFO:tasks.rados.rados.0.smithi045.stdout:415: writing smithi04569269-44 from 1916928 to 2703360 tid 1 2021-07-31T01:46:57.505 INFO:tasks.rados.rados.0.smithi045.stdout:415: writing smithi04569269-44 from 2703360 to 3178496 tid 2 2021-07-31T01:46:57.508 INFO:tasks.rados.rados.0.smithi045.stdout:415: writing smithi04569269-44 from 3178496 to 3719168 tid 3 2021-07-31T01:46:57.512 INFO:tasks.rados.rados.0.smithi045.stdout:415: writing smithi04569269-44 from 3719168 to 4308992 tid 4 2021-07-31T01:46:57.512 INFO:tasks.rados.rados.0.smithi045.stdout:416: setattr oid 48 current snap is 30 2021-07-31T01:46:57.512 INFO:tasks.rados.rados.0.smithi045.stdout:414: expect (ObjNum 47 snap 0 seq_num 47) 2021-07-31T01:46:57.535 INFO:tasks.rados.rados.0.smithi045.stdout:415: finishing write tid 1 to smithi04569269-44 2021-07-31T01:46:57.535 INFO:tasks.rados.rados.0.smithi045.stdout:413: finishing copy_from to smithi04569269-5 2021-07-31T01:46:57.535 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 5 v 93 (ObjNum 139 snap 30 seq_num 139) dirty exists 2021-07-31T01:46:57.535 INFO:tasks.rados.rados.0.smithi045.stdout:415: finishing write tid 2 to smithi04569269-44 2021-07-31T01:46:57.536 INFO:tasks.rados.rados.0.smithi045.stdout:413: finishing copy_from racing read to smithi04569269-5 2021-07-31T01:46:57.536 INFO:tasks.rados.rados.0.smithi045.stdout:415: finishing write tid 3 to smithi04569269-44 2021-07-31T01:46:57.536 INFO:tasks.rados.rados.0.smithi045.stdout:415: finishing write tid 4 to smithi04569269-44 2021-07-31T01:46:57.536 INFO:tasks.rados.rados.0.smithi045.stdout:415: finishing write tid 5 to smithi04569269-44 2021-07-31T01:46:57.537 INFO:tasks.rados.rados.0.smithi045.stdout:415: finishing write tid 6 to smithi04569269-44 2021-07-31T01:46:57.537 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 44 v 132 (ObjNum 142 snap 30 seq_num 142) dirty exists 2021-07-31T01:46:57.537 INFO:tasks.rados.rados.0.smithi045.stdout:415: left oid 44 (ObjNum 142 snap 30 seq_num 142) 2021-07-31T01:46:57.537 INFO:tasks.rados.rados.0.smithi045.stdout:413: done (3 left) 2021-07-31T01:46:57.537 INFO:tasks.rados.rados.0.smithi045.stdout:414: done (2 left) 2021-07-31T01:46:57.538 INFO:tasks.rados.rados.0.smithi045.stdout:415: done (1 left) 2021-07-31T01:46:57.538 INFO:tasks.rados.rados.0.smithi045.stdout:417: read oid 12 snap -1 2021-07-31T01:46:57.539 INFO:tasks.rados.rados.0.smithi045.stdout:417: expect (ObjNum 139 snap 30 seq_num 139) 2021-07-31T01:46:57.539 INFO:tasks.rados.rados.0.smithi045.stdout:418: snap_create 2021-07-31T01:46:57.539 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 48 v 49 (ObjNum 92 snap 15 seq_num 92) dirty exists 2021-07-31T01:46:57.545 INFO:tasks.rados.rados.0.smithi045.stdout:417: expect (ObjNum 139 snap 30 seq_num 139) 2021-07-31T01:46:58.491 INFO:tasks.rados.rados.0.smithi045.stdout:416: done (2 left) 2021-07-31T01:46:58.491 INFO:tasks.rados.rados.0.smithi045.stdout:417: done (1 left) 2021-07-31T01:46:58.491 INFO:tasks.rados.rados.0.smithi045.stdout:418: done (0 left) 2021-07-31T01:46:58.492 INFO:tasks.rados.rados.0.smithi045.stdout:419: delete oid 27 current snap is 31 2021-07-31T01:46:58.494 INFO:tasks.rados.rados.0.smithi045.stdout:419: done (0 left) 2021-07-31T01:46:58.494 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 3 current snap is 31 2021-07-31T01:46:58.494 INFO:tasks.rados.rados.0.smithi045.stdout:420: seq_num 144 ranges {778240=16384} 2021-07-31T01:46:58.498 INFO:tasks.rados.rados.0.smithi045.stdout:420: writing smithi04569269-3 from 778240 to 794624 tid 1 2021-07-31T01:46:58.499 INFO:tasks.rados.rados.0.smithi045.stdout:421: snap_create 2021-07-31T01:46:58.502 INFO:tasks.rados.rados.0.smithi045.stdout:420: finishing write tid 1 to smithi04569269-3 2021-07-31T01:46:58.503 INFO:tasks.rados.rados.0.smithi045.stdout:420: finishing write tid 2 to smithi04569269-3 2021-07-31T01:46:58.504 INFO:tasks.rados.rados.0.smithi045.stdout:420: finishing write tid 3 to smithi04569269-3 2021-07-31T01:46:58.504 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 3 v 283 (ObjNum 144 snap 31 seq_num 144) dirty exists 2021-07-31T01:46:58.504 INFO:tasks.rados.rados.0.smithi045.stdout:420: left oid 3 (ObjNum 144 snap 31 seq_num 144) 2021-07-31T01:46:58.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:58 smithi154 conmon[34553]: cluster 2021-07-31T01:46:57.375989+0000 mgr.y (mgr 2021-07-31T01:46:58.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:58 smithi154 conmon[34553]: .14146) 310 : cluster [DBG] pgmap v326: 25 pgs: 25 active+clean; 119 MiB data, 293 MiB used, 715 GiB / 715 GiB avail; 5.5 MiB/s rd, 0 B/s wr, 4 op/s 2021-07-31T01:46:58.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:58 smithi154 conmon[34553]: cluster 2021-07-31T01:46:57.485507+0000 mon.a (mon.0) 720 : cluster [DBG] osdmap e118: 8 total, 8 up, 8 in 2021-07-31T01:46:58.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:58 smithi045 conmon[33655]: cluster 2021-07-31T01:46:57. 2021-07-31T01:46:58.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:58 smithi045 conmon[33655]: 375989+0000 mgr.y (mgr.14146) 310 : cluster [DBG] pgmap v326: 25 pgs: 25 active+clean; 119 MiB data, 293 MiB used, 715 GiB / 715 GiB avail; 5.5 MiB/s rd, 0 B/s wr, 4 op/s 2021-07-31T01:46:58.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:58 smithi045 conmon[33655]: cluster 2021-07-31T01 2021-07-31T01:46:58.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:58 smithi045 conmon[33655]: :46:57.485507+0000 mon.a (mon.0) 720 : cluster 2021-07-31T01:46:58.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:58 smithi045 conmon[33655]: [DBG] osdmap e118: 8 total, 8 up, 8 in 2021-07-31T01:46:58.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:58 smithi045 conmon[39718]: cluster 2021-07-31T01:46:57.375989+0000 mgr.y (mgr 2021-07-31T01:46:58.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:58 smithi045 conmon[39718]: .14146) 310 : cluster [DBG] pgmap v326: 25 pgs: 25 active+clean; 119 MiB data, 293 MiB used, 715 GiB / 715 GiB avail; 5.5 MiB/s rd, 0 B/s wr, 4 op/s 2021-07-31T01:46:58.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:58 smithi045 conmon[39718]: cluster 2021-07-31T01:46: 2021-07-31T01:46:58.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:58 smithi045 conmon[39718]: 57.485507+0000 mon.a (mon.0) 720 : cluster [DBG] osdmap e118: 8 total, 8 up, 8 in 2021-07-31T01:46:59.497 INFO:tasks.rados.rados.0.smithi045.stdout:420: done (1 left) 2021-07-31T01:46:59.497 INFO:tasks.rados.rados.0.smithi045.stdout:421: done (0 left) 2021-07-31T01:46:59.497 INFO:tasks.rados.rados.0.smithi045.stdout:422: rollback oid 27 current snap is 32 2021-07-31T01:46:59.497 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 27 to 31 2021-07-31T01:46:59.498 INFO:tasks.rados.rados.0.smithi045.stdout:423: setattr oid 20 current snap is 32 2021-07-31T01:46:59.498 INFO:tasks.rados.rados.0.smithi045.stdout:424: snap_create 2021-07-31T01:46:59.499 INFO:tasks.rados.rados.0.smithi045.stdout:422: finishing rollback tid 1 to smithi04569269-27 2021-07-31T01:46:59.499 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 27 v 134 (ObjNum -1105652464 snap 21873 seq_num 3189671936) dirty dne 2021-07-31T01:46:59.501 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 20 v 284 (ObjNum 3 snap 0 seq_num 4) dirty exists 2021-07-31T01:46:59.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:59 smithi154 conmon[34553]: cluster 2021-07-31T01:46:58.490492+0000 mon.a (mon.0) 721 : 2021-07-31T01:46:59.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:46:59 smithi154 conmon[34553]: cluster [DBG] osdmap e119: 8 total, 8 up, 8 in 2021-07-31T01:46:59.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:59 smithi045 conmon[33655]: cluster 2021-07-31T01:46:59.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:59 smithi045 conmon[33655]: 2021-07-31T01:46:58. 2021-07-31T01:46:59.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:46:59 smithi045 conmon[33655]: 490492+0000 mon.a (mon.0) 721 : cluster [DBG] osdmap e119: 8 total, 8 up, 8 in 2021-07-31T01:46:59.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:59 smithi045 conmon[39718]: cluster 2021-07-31T01:46:59.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:59 smithi045 conmon[39718]: 2021- 2021-07-31T01:46:59.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:46:59 smithi045 conmon[39718]: 07-31T01:46:58.490492+0000 mon.a (mon.0) 721 : cluster [DBG] osdmap e119: 8 total, 8 up, 8 in 2021-07-31T01:47:00.498 INFO:tasks.rados.rados.0.smithi045.stdout:422: done (2 left) 2021-07-31T01:47:00.498 INFO:tasks.rados.rados.0.smithi045.stdout:423: done (1 left) 2021-07-31T01:47:00.498 INFO:tasks.rados.rados.0.smithi045.stdout:424: done (0 left) 2021-07-31T01:47:00.499 INFO:tasks.rados.rados.0.smithi045.stdout:425: read oid 46 snap -1 2021-07-31T01:47:00.499 INFO:tasks.rados.rados.0.smithi045.stdout:425: expect deleted 2021-07-31T01:47:00.499 INFO:tasks.rados.rados.0.smithi045.stdout:426: snap_remove snap 30 2021-07-31T01:47:00.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:00 smithi154 conmon[34553]: cluster 2021-07-31T01:46:59.376434+0000 mgr.y (mgr.14146) 311 : cluster 2021-07-31T01:47:00.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:00 smithi154 conmon[34553]: [DBG] pgmap v329: 25 pgs: 1 active+clean+snaptrim, 24 active+clean; 106 MiB data, 303 MiB used, 715 GiB / 715 GiB avail; 8.5 MiB/s rd, 1.2 MiB/s wr, 33 op/s 2021-07-31T01:47:00.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:00 smithi154 conmon[34553]: cluster 2021-07-31T01:46:59.496014+0000 mon.a (mon.0) 722 : cluster [DBG] osdmap e120: 8 total, 8 up, 8 in 2021-07-31T01:47:00.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:00 smithi045 conmon[33655]: cluster 2021-07-31T01:46:59.376434+0000 mgr.y (mgr.14146) 2021-07-31T01:47:00.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:00 smithi045 conmon[33655]: 311 : cluster [DBG] pgmap v329: 25 pgs: 1 active+clean+snaptrim, 24 active+clean; 106 MiB data, 303 MiB used, 715 GiB / 715 GiB avail; 8.5 MiB/s rd, 1.2 MiB/s wr, 33 op/s 2021-07-31T01:47:00.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:00 smithi045 conmon[33655]: cluster 2021-07-31T01:46:59.496014+0000 mon.a (mon.0) 722 : cluster [DBG] osdmap e120: 8 total, 8 up, 8 in 2021-07-31T01:47:00.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:00 smithi045 conmon[39718]: cluster 2021-07-31T01:46:59.376434+0000 mgr.y (mgr 2021-07-31T01:47:00.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:00 smithi045 conmon[39718]: .14146) 311 : cluster [DBG] pgmap v329: 25 pgs: 1 active+clean+snaptrim, 24 active+clean; 106 MiB data, 303 MiB used, 715 GiB / 715 GiB avail; 8.5 MiB/s rd, 1.2 MiB/s wr, 33 op/s 2021-07-31T01:47:00.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:00 smithi045 conmon[39718]: cluster 2021-07-31T01:46:59.496014+0000 mon.a (mon.0) 722 : cluster 2021-07-31T01:47:00.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:00 smithi045 conmon[39718]: [DBG] osdmap e120: 8 total, 8 up, 8 in 2021-07-31T01:47:01.481 INFO:tasks.rados.rados.0.smithi045.stdout:426: done (1 left) 2021-07-31T01:47:01.482 INFO:tasks.rados.rados.0.smithi045.stdout:427: read oid 12 snap -1 2021-07-31T01:47:01.482 INFO:tasks.rados.rados.0.smithi045.stdout:427: expect (ObjNum 139 snap 30 seq_num 139) 2021-07-31T01:47:01.482 INFO:tasks.rados.rados.0.smithi045.stdout:428: read oid 38 snap 31 2021-07-31T01:47:01.483 INFO:tasks.rados.rados.0.smithi045.stdout:428: expect (ObjNum 121 snap 23 seq_num 121) 2021-07-31T01:47:01.483 INFO:tasks.rados.rados.0.smithi045.stdout:429: snap_create 2021-07-31T01:47:01.499 INFO:tasks.rados.rados.0.smithi045.stdout:427: expect (ObjNum 139 snap 30 seq_num 139) 2021-07-31T01:47:01.572 INFO:tasks.rados.rados.0.smithi045.stdout:428: expect (ObjNum 121 snap 23 seq_num 121) 2021-07-31T01:47:01.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:01 smithi045 conmon[33655]: cluster 2021-07-31T01:47:00.497443+0000 2021-07-31T01:47:01.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:01 smithi045 conmon[33655]: mon.a (mon.0) 723 : cluster [DBG] osdmap e121: 8 total, 8 up, 8 in 2021-07-31T01:47:01.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:01 smithi045 conmon[33655]: cluster 2021-07-31T 2021-07-31T01:47:01.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:01 smithi045 conmon[33655]: 01:47:01. 2021-07-31T01:47:01.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:01 smithi045 conmon[33655]: 480977+0000 2021-07-31T01:47:01.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:01 smithi045 conmon[33655]: mon.a (mon.0) 724 : cluster 2021-07-31T01:47:01.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:01 smithi045 conmon[33655]: [DBG] osdmap e122: 8 total, 8 up, 8 in 2021-07-31T01:47:01.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:01 smithi045 conmon[39718]: cluster 2021-07-31T01:47:01.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:01 smithi045 conmon[39718]: 2021-07-31T01:47:01.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:01 smithi045 conmon[39718]: 2021 2021-07-31T01:47:01.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:01 smithi045 conmon[39718]: -07-31T01:47:00.497443 2021-07-31T01:47:01.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:01 smithi045 conmon[39718]: +0000 mon.a (mon.0) 723 : cluster [DBG] osdmap e121: 8 total, 8 up, 8 in 2021-07-31T01:47:01.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:01 smithi045 conmon[39718]: cluster 2021- 2021-07-31T01:47:01.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:01 smithi045 conmon[39718]: 07-31T01:47:01.480977+0000 2021-07-31T01:47:01.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:01 smithi045 conmon[39718]: mon.a (mon.0) 724 : cluster [DBG] 2021-07-31T01:47:01.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:01 smithi045 conmon[39718]: osdmap e122: 8 total, 8 up, 8 in 2021-07-31T01:47:02.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:01 smithi154 conmon[34553]: cluster 2021-07-31T01:47:00.497443+0000 2021-07-31T01:47:02.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:01 smithi154 conmon[34553]: mon.a (mon.0) 723 : cluster [DBG] osdmap e121: 8 total, 8 up, 8 in 2021-07-31T01:47:02.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:01 smithi154 conmon[34553]: cluster 2021-07-31T01:47:01.480977+0000 mon.a (mon.0) 724 : cluster [DBG] osdmap e122: 8 total, 8 up, 8 in 2021-07-31T01:47:02.488 INFO:tasks.rados.rados.0.smithi045.stdout:425: done (3 left) 2021-07-31T01:47:02.488 INFO:tasks.rados.rados.0.smithi045.stdout:427: done (2 left) 2021-07-31T01:47:02.488 INFO:tasks.rados.rados.0.smithi045.stdout:428: done (1 left) 2021-07-31T01:47:02.488 INFO:tasks.rados.rados.0.smithi045.stdout:429: done (0 left) 2021-07-31T01:47:02.489 INFO:tasks.rados.rados.0.smithi045.stdout:430: delete oid 4 current snap is 34 2021-07-31T01:47:02.489 INFO:tasks.rados.rados.0.smithi045.stdout:430: done (0 left) 2021-07-31T01:47:02.490 INFO:tasks.rados.rados.0.smithi045.stdout:431: snap_create 2021-07-31T01:47:02.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:02 smithi045 conmon[33655]: cluster 2021-07-31T01:47:01.376841+0000 mgr.y (mgr.14146) 312 2021-07-31T01:47:02.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:02 smithi045 conmon[33655]: : cluster [DBG] pgmap v332: 25 pgs: 1 active+clean+snaptrim, 24 active+clean; 106 MiB data, 303 MiB used, 715 GiB / 715 GiB avail; 8.5 MiB/s rd, 1.2 MiB/s wr, 33 op/s 2021-07-31T01:47:02.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:02 smithi045 conmon[33655]: cluster 2021-07-31T01:47:02.487110+0000 mon.a (mon.0) 725 : 2021-07-31T01:47:02.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:02 smithi045 conmon[33655]: cluster [DBG] osdmap e123: 8 total, 8 up, 8 in 2021-07-31T01:47:02.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:02 smithi045 conmon[39718]: cluster 2021-07-31T01:47:01. 2021-07-31T01:47:02.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:02 smithi045 conmon[39718]: 376841+0000 mgr.y (mgr.14146) 312 : cluster [DBG] pgmap v332: 25 pgs: 1 active+clean+snaptrim, 24 active+clean; 106 MiB data, 303 MiB used, 715 GiB / 715 GiB avail; 8.5 MiB/s rd, 1.2 MiB/s wr, 33 op/s 2021-07-31T01:47:02.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:02 smithi045 conmon[39718]: cluster 2021-07-31T01:47:02.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:02 smithi045 conmon[39718]: 2021-07-31T01 2021-07-31T01:47:02.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:02 smithi045 conmon[39718]: :47:02.487110+0000 2021-07-31T01:47:02.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:02 smithi045 conmon[39718]: mon.a (mon.0) 725 : 2021-07-31T01:47:02.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:02 smithi045 conmon[39718]: cluster [DBG] osdmap e123: 8 total, 8 up, 8 in 2021-07-31T01:47:03.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:02 smithi154 conmon[34553]: cluster 2021-07-31T01:47:01.376841+0000 mgr.y 2021-07-31T01:47:03.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:02 smithi154 conmon[34553]: (mgr.14146) 312 : cluster [DBG] pgmap v332: 25 pgs: 1 active+clean+snaptrim, 24 active+clean; 106 MiB data, 303 MiB used, 715 GiB / 715 GiB avail; 8.5 MiB/s rd, 1.2 MiB/s wr, 33 op/s 2021-07-31T01:47:03.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:02 smithi154 conmon[34553]: cluster 2021-07-31T01:47:02.487110+0000 mon.a (mon.0) 725 : cluster [DBG] 2021-07-31T01:47:03.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:02 smithi154 conmon[34553]: osdmap e123: 8 total, 8 up, 8 in 2021-07-31T01:47:03.488 INFO:tasks.rados.rados.0.smithi045.stdout:431: done (0 left) 2021-07-31T01:47:03.489 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 28 current snap is 35 2021-07-31T01:47:03.489 INFO:tasks.rados.rados.0.smithi045.stdout:432: seq_num 146 ranges {2039808=49152} 2021-07-31T01:47:03.500 INFO:tasks.rados.rados.0.smithi045.stdout:432: writing smithi04569269-28 from 2039808 to 2088960 tid 1 2021-07-31T01:47:03.500 INFO:tasks.rados.rados.0.smithi045.stdout:433: rollback oid 34 current snap is 35 2021-07-31T01:47:03.500 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 34 to 26 2021-07-31T01:47:03.501 INFO:tasks.rados.rados.0.smithi045.stdout:434: delete oid 8 current snap is 35 2021-07-31T01:47:03.503 INFO:tasks.rados.rados.0.smithi045.stdout:433: finishing rollback tid 0 to smithi04569269-34 2021-07-31T01:47:03.504 INFO:tasks.rados.rados.0.smithi045.stdout:433: finishing rollback tid 1 to smithi04569269-34 2021-07-31T01:47:03.504 INFO:tasks.rados.rados.0.smithi045.stdout:433: finishing rollback tid 2 to smithi04569269-34 2021-07-31T01:47:03.504 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 34 v 53 (ObjNum 76 snap 11 seq_num 76) dirty exists 2021-07-31T01:47:03.505 INFO:tasks.rados.rados.0.smithi045.stdout:432: finishing write tid 1 to smithi04569269-28 2021-07-31T01:47:03.506 INFO:tasks.rados.rados.0.smithi045.stdout:432: finishing write tid 2 to smithi04569269-28 2021-07-31T01:47:03.507 INFO:tasks.rados.rados.0.smithi045.stdout:433: done (2 left) 2021-07-31T01:47:03.507 INFO:tasks.rados.rados.0.smithi045.stdout:434: done (1 left) 2021-07-31T01:47:03.507 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 41 current snap is 35 2021-07-31T01:47:03.507 INFO:tasks.rados.rados.0.smithi045.stdout:435: seq_num 148 ranges {679936=81920} 2021-07-31T01:47:03.512 INFO:tasks.rados.rados.0.smithi045.stdout:435: writing smithi04569269-41 from 679936 to 761856 tid 1 2021-07-31T01:47:03.512 INFO:tasks.rados.rados.0.smithi045.stdout:436: snap_create 2021-07-31T01:47:03.513 INFO:tasks.rados.rados.0.smithi045.stdout:432: finishing write tid 3 to smithi04569269-28 2021-07-31T01:47:03.513 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 28 v 168 (ObjNum 146 snap 35 seq_num 146) dirty exists 2021-07-31T01:47:03.513 INFO:tasks.rados.rados.0.smithi045.stdout:432: left oid 28 (ObjNum 146 snap 35 seq_num 146) 2021-07-31T01:47:03.514 INFO:tasks.rados.rados.0.smithi045.stdout:435: finishing write tid 1 to smithi04569269-41 2021-07-31T01:47:03.515 INFO:tasks.rados.rados.0.smithi045.stdout:435: finishing write tid 2 to smithi04569269-41 2021-07-31T01:47:03.517 INFO:tasks.rados.rados.0.smithi045.stdout:435: finishing write tid 3 to smithi04569269-41 2021-07-31T01:47:03.518 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 41 v 51 (ObjNum 148 snap 35 seq_num 148) dirty exists 2021-07-31T01:47:03.518 INFO:tasks.rados.rados.0.smithi045.stdout:435: left oid 41 (ObjNum 148 snap 35 seq_num 148) 2021-07-31T01:47:03.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:47:03 smithi045 conmon[61210]: level=error ts=2021-07-31T01:47:03.506Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:47:03.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:47:03 smithi045 conmon[61210]: level=error ts=2021-07-31T01:47:03.506Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:47:03.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:47:03 smithi045 conmon[61210]: level=error ts=2021-07-31T01:47:03.506Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T01:47:04.514 INFO:tasks.rados.rados.0.smithi045.stdout:432: done (2 left) 2021-07-31T01:47:04.514 INFO:tasks.rados.rados.0.smithi045.stdout:435: done (1 left) 2021-07-31T01:47:04.515 INFO:tasks.rados.rados.0.smithi045.stdout:436: done (0 left) 2021-07-31T01:47:04.515 INFO:tasks.rados.rados.0.smithi045.stdout:437: read oid 45 snap -1 2021-07-31T01:47:04.515 INFO:tasks.rados.rados.0.smithi045.stdout:437: expect (ObjNum 0 snap 0 seq_num 0) 2021-07-31T01:47:04.515 INFO:tasks.rados.rados.0.smithi045.stdout:438: rmattr oid 18 current snap is 36 2021-07-31T01:47:04.515 INFO:tasks.rados.rados.0.smithi045.stdout:438: done (1 left) 2021-07-31T01:47:04.516 INFO:tasks.rados.rados.0.smithi045.stdout:439: read oid 28 snap -1 2021-07-31T01:47:04.516 INFO:tasks.rados.rados.0.smithi045.stdout:439: expect (ObjNum 146 snap 35 seq_num 146) 2021-07-31T01:47:04.516 INFO:tasks.rados.rados.0.smithi045.stdout:440: snap_create 2021-07-31T01:47:04.519 INFO:tasks.rados.rados.0.smithi045.stdout:437: expect (ObjNum 0 snap 0 seq_num 0) 2021-07-31T01:47:04.532 INFO:tasks.rados.rados.0.smithi045.stdout:439: expect (ObjNum 146 snap 35 seq_num 146) 2021-07-31T01:47:04.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:04 smithi154 conmon[34553]: cluster 2021-07-31T01:47:03.377289+0000 mgr.y (mgr. 2021-07-31T01:47:04.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:04 smithi154 conmon[34553]: 14146) 313 : cluster [DBG] pgmap v335: 25 pgs: 25 active+clean; 107 MiB data, 308 MiB used, 715 GiB / 715 GiB avail; 4.7 MiB/s rd, 584 KiB/s wr, 3 op/s 2021-07-31T01:47:04.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:04 smithi154 conmon[34553]: cluster 2021-07-31T01:47:03.488044+0000 mon.a (mon.0) 726 : cluster [DBG] osdmap e124: 8 total, 8 up, 8 in 2021-07-31T01:47:04.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:04 smithi045 conmon[33655]: cluster 2021-07-31T01:47:03.377289+0000 mgr.y ( 2021-07-31T01:47:04.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:04 smithi045 conmon[33655]: mgr.14146) 313 : cluster [DBG] pgmap v335: 25 pgs: 25 active+clean; 107 MiB data, 308 MiB used, 715 GiB / 715 GiB avail; 4.7 MiB/s rd, 584 KiB/s wr, 3 op/s 2021-07-31T01:47:04.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:04 smithi045 conmon[33655]: cluster 2021-07-31T01:47:03.488044+0000 mon.a (mon.0) 726 : cluster [DBG] 2021-07-31T01:47:04.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:04 smithi045 conmon[33655]: osdmap e124: 8 total, 8 up, 8 in 2021-07-31T01:47:04.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:04 smithi045 conmon[39718]: cluster 2021-07-31T01:47:03.377289 2021-07-31T01:47:04.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:04 smithi045 conmon[39718]: +0000 mgr.y (mgr.14146) 313 : cluster [DBG] pgmap v335: 25 pgs: 25 active+clean; 107 MiB data, 308 MiB used, 715 GiB / 715 GiB avail; 4.7 MiB/s rd, 584 KiB/s wr, 3 op/s 2021-07-31T01:47:04.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:04 smithi045 conmon[39718]: cluster 2021-07-31T01:47:04.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:04 smithi045 conmon[39718]: 2021-07-31T01:47: 2021-07-31T01:47:04.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:04 smithi045 conmon[39718]: 03.488044+0000 mon.a (mon 2021-07-31T01:47:04.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:04 smithi045 conmon[39718]: .0) 726 : cluster [DBG] 2021-07-31T01:47:04.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:04 smithi045 conmon[39718]: osdmap e124: 8 total, 8 up, 8 in 2021-07-31T01:47:05.521 INFO:tasks.rados.rados.0.smithi045.stdout:437: done (2 left) 2021-07-31T01:47:05.522 INFO:tasks.rados.rados.0.smithi045.stdout:439: done (1 left) 2021-07-31T01:47:05.522 INFO:tasks.rados.rados.0.smithi045.stdout:440: done (0 left) 2021-07-31T01:47:05.523 INFO:tasks.rados.rados.0.smithi045.stdout:441: read oid 10 snap -1 2021-07-31T01:47:05.523 INFO:tasks.rados.rados.0.smithi045.stdout:441: expect (ObjNum 97 snap 16 seq_num 97) 2021-07-31T01:47:05.523 INFO:tasks.rados.rados.0.smithi045.stdout:442: rollback oid 42 current snap is 37 2021-07-31T01:47:05.523 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 42 to 32 2021-07-31T01:47:05.524 INFO:tasks.rados.rados.0.smithi045.stdout:443: snap_remove snap 34 2021-07-31T01:47:05.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:05 smithi045 conmon[33655]: cluster 2021-07-31T01:47:04.512915+0000 mon.a (mon.0) 727 : cluster 2021-07-31T01:47:05.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:05 smithi045 conmon[33655]: [DBG] osdmap e125: 8 total, 8 up, 8 in 2021-07-31T01:47:05.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:05 smithi045 conmon[39718]: cluster 2021-07-31T01:47:04.512915+0000 2021-07-31T01:47:05.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:05 smithi045 conmon[39718]: mon.a (mon.0) 727 : cluster [DBG] osdmap e125: 8 total, 8 up, 8 in 2021-07-31T01:47:06.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:05 smithi154 conmon[34553]: cluster 2021-07-31T01:47:04.512915+0000 2021-07-31T01:47:06.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:05 smithi154 conmon[34553]: mon.a (mon.0) 727 : cluster [DBG] osdmap e125: 8 total, 8 up, 8 in 2021-07-31T01:47:06.484 INFO:tasks.rados.rados.0.smithi045.stdout:443: done (2 left) 2021-07-31T01:47:06.484 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 23 current snap is 37 2021-07-31T01:47:06.484 INFO:tasks.rados.rados.0.smithi045.stdout:441: expect (ObjNum 97 snap 16 seq_num 97) 2021-07-31T01:47:06.537 INFO:tasks.rados.rados.0.smithi045.stdout:442: finishing rollback tid 0 to smithi04569269-42 2021-07-31T01:47:06.538 INFO:tasks.rados.rados.0.smithi045.stdout:442: finishing rollback tid 1 to smithi04569269-42 2021-07-31T01:47:06.538 INFO:tasks.rados.rados.0.smithi045.stdout:442: finishing rollback tid 2 to smithi04569269-42 2021-07-31T01:47:06.539 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 42 v 172 (ObjNum 125 snap 24 seq_num 125) dirty exists 2021-07-31T01:47:06.539 INFO:tasks.rados.rados.0.smithi045.stdout:444: seq_num 149 ranges {4063232=98304} 2021-07-31T01:47:06.560 INFO:tasks.rados.rados.0.smithi045.stdout:444: writing smithi04569269-23 from 4063232 to 4161536 tid 1 2021-07-31T01:47:06.561 INFO:tasks.rados.rados.0.smithi045.stdout:441: done (2 left) 2021-07-31T01:47:06.561 INFO:tasks.rados.rados.0.smithi045.stdout:442: done (1 left) 2021-07-31T01:47:06.561 INFO:tasks.rados.rados.0.smithi045.stdout:445: delete oid 5 current snap is 37 2021-07-31T01:47:06.564 INFO:tasks.rados.rados.0.smithi045.stdout:444: finishing write tid 1 to smithi04569269-23 2021-07-31T01:47:06.565 INFO:tasks.rados.rados.0.smithi045.stdout:444: finishing write tid 2 to smithi04569269-23 2021-07-31T01:47:06.566 INFO:tasks.rados.rados.0.smithi045.stdout:445: done (1 left) 2021-07-31T01:47:06.566 INFO:tasks.rados.rados.0.smithi045.stdout:446: read oid 26 snap 35 2021-07-31T01:47:06.567 INFO:tasks.rados.rados.0.smithi045.stdout:446: expect (ObjNum 101 snap 16 seq_num 101) 2021-07-31T01:47:06.567 INFO:tasks.rados.rados.0.smithi045.stdout:447: read oid 47 snap -1 2021-07-31T01:47:06.567 INFO:tasks.rados.rados.0.smithi045.stdout:447: expect (ObjNum 46 snap 0 seq_num 46) 2021-07-31T01:47:06.567 INFO:tasks.rados.rados.0.smithi045.stdout:448: read oid 41 snap -1 2021-07-31T01:47:06.568 INFO:tasks.rados.rados.0.smithi045.stdout:448: expect (ObjNum 148 snap 35 seq_num 148) 2021-07-31T01:47:06.568 INFO:tasks.rados.rados.0.smithi045.stdout:449: rmattr oid 4 current snap is 37 2021-07-31T01:47:06.568 INFO:tasks.rados.rados.0.smithi045.stdout:449: done (4 left) 2021-07-31T01:47:06.568 INFO:tasks.rados.rados.0.smithi045.stdout:450: copy_from oid 13 from oid 48 current snap is 37 2021-07-31T01:47:06.568 INFO:tasks.rados.rados.0.smithi045.stdout:451: read oid 3 snap -1 2021-07-31T01:47:06.569 INFO:tasks.rados.rados.0.smithi045.stdout:451: expect (ObjNum 144 snap 31 seq_num 144) 2021-07-31T01:47:06.569 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 4 current snap is 37 2021-07-31T01:47:06.569 INFO:tasks.rados.rados.0.smithi045.stdout:452: seq_num 151 ranges {0=131072} 2021-07-31T01:47:06.570 INFO:tasks.rados.rados.0.smithi045.stdout:452: writing smithi04569269-4 from 0 to 131072 tid 1 2021-07-31T01:47:06.570 INFO:tasks.rados.rados.0.smithi045.stdout:453: delete oid 42 current snap is 37 2021-07-31T01:47:06.570 INFO:tasks.rados.rados.0.smithi045.stdout:444: finishing write tid 3 to smithi04569269-23 2021-07-31T01:47:06.571 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 23 v 149 (ObjNum 149 snap 37 seq_num 149) dirty exists 2021-07-31T01:47:06.571 INFO:tasks.rados.rados.0.smithi045.stdout:444: left oid 23 (ObjNum 149 snap 37 seq_num 149) 2021-07-31T01:47:06.573 INFO:tasks.rados.rados.0.smithi045.stdout:451: expect (ObjNum 144 snap 31 seq_num 144) 2021-07-31T01:47:06.607 INFO:tasks.rados.rados.0.smithi045.stdout:447: expect (ObjNum 46 snap 0 seq_num 46) 2021-07-31T01:47:06.628 INFO:tasks.rados.rados.0.smithi045.stdout:444: done (7 left) 2021-07-31T01:47:06.628 INFO:tasks.rados.rados.0.smithi045.stdout:447: done (6 left) 2021-07-31T01:47:06.628 INFO:tasks.rados.rados.0.smithi045.stdout:451: done (5 left) 2021-07-31T01:47:06.628 INFO:tasks.rados.rados.0.smithi045.stdout:453: done (4 left) 2021-07-31T01:47:06.629 INFO:tasks.rados.rados.0.smithi045.stdout:454: delete oid 29 current snap is 37 2021-07-31T01:47:06.629 INFO:tasks.rados.rados.0.smithi045.stdout:452: finishing write tid 1 to smithi04569269-4 2021-07-31T01:47:06.629 INFO:tasks.rados.rados.0.smithi045.stdout:452: finishing write tid 2 to smithi04569269-4 2021-07-31T01:47:06.629 INFO:tasks.rados.rados.0.smithi045.stdout:452: finishing write tid 3 to smithi04569269-4 2021-07-31T01:47:06.629 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 4 v 112 (ObjNum 151 snap 37 seq_num 151) dirty exists 2021-07-31T01:47:06.630 INFO:tasks.rados.rados.0.smithi045.stdout:452: left oid 4 (ObjNum 151 snap 37 seq_num 151) 2021-07-31T01:47:06.630 INFO:tasks.rados.rados.0.smithi045.stdout:446: expect (ObjNum 101 snap 16 seq_num 101) 2021-07-31T01:47:06.671 INFO:tasks.rados.rados.0.smithi045.stdout:448: expect (ObjNum 148 snap 35 seq_num 148) 2021-07-31T01:47:06.704 INFO:tasks.rados.rados.0.smithi045.stdout:450: finishing copy_from to smithi04569269-13 2021-07-31T01:47:06.705 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 13 v 49 (ObjNum 92 snap 15 seq_num 92) dirty exists 2021-07-31T01:47:06.705 INFO:tasks.rados.rados.0.smithi045.stdout:450: finishing copy_from racing read to smithi04569269-13 2021-07-31T01:47:06.706 INFO:tasks.rados.rados.0.smithi045.stdout:446: done (4 left) 2021-07-31T01:47:06.706 INFO:tasks.rados.rados.0.smithi045.stdout:448: done (3 left) 2021-07-31T01:47:06.706 INFO:tasks.rados.rados.0.smithi045.stdout:450: done (2 left) 2021-07-31T01:47:06.706 INFO:tasks.rados.rados.0.smithi045.stdout:452: done (1 left) 2021-07-31T01:47:06.706 INFO:tasks.rados.rados.0.smithi045.stdout:454: done (0 left) 2021-07-31T01:47:06.707 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 24 current snap is 37 2021-07-31T01:47:06.707 INFO:tasks.rados.rados.0.smithi045.stdout:455: seq_num 154 ranges {4169728=180224} 2021-07-31T01:47:06.728 INFO:tasks.rados.rados.0.smithi045.stdout:455: writing smithi04569269-24 from 4169728 to 4349952 tid 1 2021-07-31T01:47:06.729 INFO:tasks.rados.rados.0.smithi045.stdout:456: read oid 4 snap -1 2021-07-31T01:47:06.729 INFO:tasks.rados.rados.0.smithi045.stdout:456: expect (ObjNum 151 snap 37 seq_num 151) 2021-07-31T01:47:06.729 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 1 current snap is 37 2021-07-31T01:47:06.729 INFO:tasks.rados.rados.0.smithi045.stdout:457: seq_num 155 ranges {1196032=204800} 2021-07-31T01:47:06.736 INFO:tasks.rados.rados.0.smithi045.stdout:457: writing smithi04569269-1 from 1196032 to 1400832 tid 1 2021-07-31T01:47:06.736 INFO:tasks.rados.rados.0.smithi045.stdout:458: rmattr oid 50 current snap is 37 2021-07-31T01:47:06.737 INFO:tasks.rados.rados.0.smithi045.stdout:458: done (3 left) 2021-07-31T01:47:06.737 INFO:tasks.rados.rados.0.smithi045.stdout:459: setattr oid 26 current snap is 37 2021-07-31T01:47:06.737 INFO:tasks.rados.rados.0.smithi045.stdout:456: expect (ObjNum 151 snap 37 seq_num 151) 2021-07-31T01:47:06.741 INFO:tasks.rados.rados.0.smithi045.stdout:455: finishing write tid 1 to smithi04569269-24 2021-07-31T01:47:06.741 INFO:tasks.rados.rados.0.smithi045.stdout:455: finishing write tid 2 to smithi04569269-24 2021-07-31T01:47:06.741 INFO:tasks.rados.rados.0.smithi045.stdout:455: finishing write tid 3 to smithi04569269-24 2021-07-31T01:47:06.742 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 24 v 49 (ObjNum 154 snap 37 seq_num 154) dirty exists 2021-07-31T01:47:06.742 INFO:tasks.rados.rados.0.smithi045.stdout:455: left oid 24 (ObjNum 154 snap 37 seq_num 154) 2021-07-31T01:47:06.742 INFO:tasks.rados.rados.0.smithi045.stdout:457: finishing write tid 1 to smithi04569269-1 2021-07-31T01:47:06.742 INFO:tasks.rados.rados.0.smithi045.stdout:457: finishing write tid 2 to smithi04569269-1 2021-07-31T01:47:06.742 INFO:tasks.rados.rados.0.smithi045.stdout:457: finishing write tid 3 to smithi04569269-1 2021-07-31T01:47:06.743 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 1 v 58 (ObjNum 155 snap 37 seq_num 155) dirty exists 2021-07-31T01:47:06.743 INFO:tasks.rados.rados.0.smithi045.stdout:457: left oid 1 (ObjNum 155 snap 37 seq_num 155) 2021-07-31T01:47:06.743 INFO:tasks.rados.rados.0.smithi045.stdout:455: done (3 left) 2021-07-31T01:47:06.743 INFO:tasks.rados.rados.0.smithi045.stdout:456: done (2 left) 2021-07-31T01:47:06.744 INFO:tasks.rados.rados.0.smithi045.stdout:457: done (1 left) 2021-07-31T01:47:06.744 INFO:tasks.rados.rados.0.smithi045.stdout:460: rollback oid 35 current snap is 37 2021-07-31T01:47:06.744 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 35 to 31 2021-07-31T01:47:06.744 INFO:tasks.rados.rados.0.smithi045.stdout:461: delete oid 37 current snap is 37 2021-07-31T01:47:06.745 INFO:tasks.rados.rados.0.smithi045.stdout:461: done (2 left) 2021-07-31T01:47:06.745 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 31 current snap is 37 2021-07-31T01:47:06.745 INFO:tasks.rados.rados.0.smithi045.stdout:462: seq_num 157 ranges {2695168=237568} 2021-07-31T01:47:06.759 INFO:tasks.rados.rados.0.smithi045.stdout:462: writing smithi04569269-31 from 2695168 to 2932736 tid 1 2021-07-31T01:47:06.759 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 40 current snap is 37 2021-07-31T01:47:06.760 INFO:tasks.rados.rados.0.smithi045.stdout:460: finishing rollback tid 0 to smithi04569269-35 2021-07-31T01:47:06.760 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 26 v 62 (ObjNum 101 snap 16 seq_num 101) dirty exists 2021-07-31T01:47:06.760 INFO:tasks.rados.rados.0.smithi045.stdout:460: finishing rollback tid 1 to smithi04569269-35 2021-07-31T01:47:06.760 INFO:tasks.rados.rados.0.smithi045.stdout:460: finishing rollback tid 2 to smithi04569269-35 2021-07-31T01:47:06.760 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 35 v 294 (ObjNum 122 snap 23 seq_num 122) dirty exists 2021-07-31T01:47:06.761 INFO:tasks.rados.rados.0.smithi045.stdout:463: seq_num 158 ranges {0=253952} 2021-07-31T01:47:06.762 INFO:tasks.rados.rados.0.smithi045.stdout:463: writing smithi04569269-40 from 0 to 253952 tid 1 2021-07-31T01:47:06.762 INFO:tasks.rados.rados.0.smithi045.stdout:459: done (3 left) 2021-07-31T01:47:06.762 INFO:tasks.rados.rados.0.smithi045.stdout:460: done (2 left) 2021-07-31T01:47:06.763 INFO:tasks.rados.rados.0.smithi045.stdout:464: read oid 17 snap -1 2021-07-31T01:47:06.763 INFO:tasks.rados.rados.0.smithi045.stdout:464: expect deleted 2021-07-31T01:47:06.763 INFO:tasks.rados.rados.0.smithi045.stdout:465: read oid 7 snap -1 2021-07-31T01:47:06.763 INFO:tasks.rados.rados.0.smithi045.stdout:465: expect deleted 2021-07-31T01:47:06.764 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 33 current snap is 37 2021-07-31T01:47:06.764 INFO:tasks.rados.rados.0.smithi045.stdout:466: seq_num 159 ranges {4177920=270336} 2021-07-31T01:47:06.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:06 smithi045 conmon[33655]: cluster 2021-07-31T01:47:05.377793+0000 mgr.y (mgr 2021-07-31T01:47:06.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:06 smithi045 conmon[33655]: .14146) 314 : cluster [DBG] pgmap v338: 25 pgs: 25 active+clean; 108 MiB data, 309 MiB used, 715 GiB / 715 GiB avail; 4.7 MiB/s rd, 584 KiB/s wr, 10 op/s 2021-07-31T01:47:06.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:06 smithi045 conmon[33655]: cluster 2021-07-31T01:47:05.519905+0000 mon.a (mon.0) 728 : cluster [DBG] osdmap e126: 8 total, 8 up, 8 in 2021-07-31T01:47:06.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:06 smithi045 conmon[33655]: cluster 2021-07-31T01:47:06.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:06 smithi045 conmon[33655]: 2021-07-31T01:47:06.483078+0000 mon.a (mon.0) 729 : cluster [DBG] osdmap e127: 8 total, 8 up, 8 in 2021-07-31T01:47:06.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:06 smithi045 conmon[39718]: cluster 2021-07-31T01:47:05.377793+0000 2021-07-31T01:47:06.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:06 smithi045 conmon[39718]: mgr.y (mgr.14146) 314 : cluster [DBG] pgmap v338: 25 pgs: 25 active+clean; 108 MiB data, 309 MiB used, 715 GiB / 715 GiB avail; 4.7 MiB/s rd, 584 KiB/s wr, 10 op/s 2021-07-31T01:47:06.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:06 smithi045 conmon[39718]: cluster 2021-07- 2021-07-31T01:47:06.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:06 smithi045 conmon[39718]: 31T01:47:05.519905+0000 mon.a (mon.0) 728 : cluster 2021-07-31T01:47:06.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:06 smithi045 conmon[39718]: [DBG] osdmap e126: 8 total, 8 up, 8 in 2021-07-31T01:47:06.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:06 smithi045 conmon[39718]: cluster 2021-07-31 2021-07-31T01:47:06.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:06 smithi045 conmon[39718]: T01:47:06.483078+0000 mon.a 2021-07-31T01:47:06.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:06 smithi045 conmon[39718]: (mon.0) 729 : cluster [DBG] 2021-07-31T01:47:06.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:06 smithi045 conmon[39718]: osdmap e127: 8 total, 8 up, 8 in 2021-07-31T01:47:06.785 INFO:tasks.rados.rados.0.smithi045.stdout:466: writing smithi04569269-33 from 4177920 to 4448256 tid 1 2021-07-31T01:47:06.785 INFO:tasks.rados.rados.0.smithi045.stdout:467: read oid 8 snap -1 2021-07-31T01:47:06.785 INFO:tasks.rados.rados.0.smithi045.stdout:467: expect deleted 2021-07-31T01:47:06.786 INFO:tasks.rados.rados.0.smithi045.stdout:468: read oid 16 snap -1 2021-07-31T01:47:06.786 INFO:tasks.rados.rados.0.smithi045.stdout:468: expect (ObjNum 118 snap 22 seq_num 118) 2021-07-31T01:47:06.786 INFO:tasks.rados.rados.0.smithi045.stdout:469: rmattr oid 48 current snap is 37 2021-07-31T01:47:06.786 INFO:tasks.rados.rados.0.smithi045.stdout:470: setattr oid 22 current snap is 37 2021-07-31T01:47:06.787 INFO:tasks.rados.rados.0.smithi045.stdout:462: finishing write tid 1 to smithi04569269-31 2021-07-31T01:47:06.787 INFO:tasks.rados.rados.0.smithi045.stdout:462: finishing write tid 2 to smithi04569269-31 2021-07-31T01:47:06.787 INFO:tasks.rados.rados.0.smithi045.stdout:462: finishing write tid 3 to smithi04569269-31 2021-07-31T01:47:06.787 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 31 v 297 (ObjNum 157 snap 37 seq_num 157) dirty exists 2021-07-31T01:47:06.788 INFO:tasks.rados.rados.0.smithi045.stdout:462: left oid 31 (ObjNum 157 snap 37 seq_num 157) 2021-07-31T01:47:06.788 INFO:tasks.rados.rados.0.smithi045.stdout:463: finishing write tid 1 to smithi04569269-40 2021-07-31T01:47:06.788 INFO:tasks.rados.rados.0.smithi045.stdout:463: finishing write tid 2 to smithi04569269-40 2021-07-31T01:47:06.788 INFO:tasks.rados.rados.0.smithi045.stdout:463: finishing write tid 3 to smithi04569269-40 2021-07-31T01:47:06.788 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 40 v 133 (ObjNum 158 snap 37 seq_num 158) dirty exists 2021-07-31T01:47:06.789 INFO:tasks.rados.rados.0.smithi045.stdout:463: left oid 40 (ObjNum 158 snap 37 seq_num 158) 2021-07-31T01:47:06.789 INFO:tasks.rados.rados.0.smithi045.stdout:462: done (8 left) 2021-07-31T01:47:06.790 INFO:tasks.rados.rados.0.smithi045.stdout:463: done (7 left) 2021-07-31T01:47:06.790 INFO:tasks.rados.rados.0.smithi045.stdout:464: done (6 left) 2021-07-31T01:47:06.790 INFO:tasks.rados.rados.0.smithi045.stdout:465: done (5 left) 2021-07-31T01:47:06.790 INFO:tasks.rados.rados.0.smithi045.stdout:467: done (4 left) 2021-07-31T01:47:06.791 INFO:tasks.rados.rados.0.smithi045.stdout:471: rmattr oid 42 current snap is 37 2021-07-31T01:47:06.791 INFO:tasks.rados.rados.0.smithi045.stdout:471: done (4 left) 2021-07-31T01:47:06.791 INFO:tasks.rados.rados.0.smithi045.stdout:472: setattr oid 31 current snap is 37 2021-07-31T01:47:06.791 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 48 v 60 (ObjNum 92 snap 15 seq_num 92) dirty exists 2021-07-31T01:47:06.792 INFO:tasks.rados.rados.0.smithi045.stdout:469: done (4 left) 2021-07-31T01:47:06.792 INFO:tasks.rados.rados.0.smithi045.stdout:473: snap_create 2021-07-31T01:47:06.793 INFO:tasks.rados.rados.0.smithi045.stdout:466: finishing write tid 1 to smithi04569269-33 2021-07-31T01:47:06.793 INFO:tasks.rados.rados.0.smithi045.stdout:466: finishing write tid 2 to smithi04569269-33 2021-07-31T01:47:06.793 INFO:tasks.rados.rados.0.smithi045.stdout:466: finishing write tid 3 to smithi04569269-33 2021-07-31T01:47:06.793 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 33 v 154 (ObjNum 159 snap 37 seq_num 159) dirty exists 2021-07-31T01:47:06.794 INFO:tasks.rados.rados.0.smithi045.stdout:466: left oid 33 (ObjNum 159 snap 37 seq_num 159) 2021-07-31T01:47:06.794 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 31 v 298 (ObjNum 157 snap 37 seq_num 157) dirty exists 2021-07-31T01:47:06.794 INFO:tasks.rados.rados.0.smithi045.stdout:468: expect (ObjNum 118 snap 22 seq_num 118) 2021-07-31T01:47:06.844 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 22 v 100 (ObjNum 47 snap 0 seq_num 47) dirty exists 2021-07-31T01:47:07.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:06 smithi154 conmon[34553]: cluster 2021-07-31T01:47:05.377793+0000 mgr.y 2021-07-31T01:47:07.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:06 smithi154 conmon[34553]: (mgr.14146) 314 : cluster [DBG] pgmap v338: 25 pgs: 25 active+clean; 108 MiB data, 309 MiB used, 715 GiB / 715 GiB avail; 4.7 MiB/s rd, 584 KiB/s wr, 10 op/s 2021-07-31T01:47:07.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:06 smithi154 conmon[34553]: cluster 2021-07-31T01:47:05.519905+0000 mon.a (mon.0) 728 : cluster [DBG] osdmap e126: 8 total, 8 up, 8 in 2021-07-31T01:47:07.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:06 smithi154 conmon[34553]: cluster 2021-07-31T01:47:07.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:06 smithi154 conmon[34553]: 2021-07-31T01:47:06.483078+0000 2021-07-31T01:47:07.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:06 smithi154 conmon[34553]: mon.a (mon.0) 729 : cluster [DBG] 2021-07-31T01:47:07.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:06 smithi154 conmon[34553]: osdmap e127: 8 total, 8 up, 8 in 2021-07-31T01:47:07.527 INFO:tasks.rados.rados.0.smithi045.stdout:466: done (4 left) 2021-07-31T01:47:07.527 INFO:tasks.rados.rados.0.smithi045.stdout:468: done (3 left) 2021-07-31T01:47:07.527 INFO:tasks.rados.rados.0.smithi045.stdout:470: done (2 left) 2021-07-31T01:47:07.528 INFO:tasks.rados.rados.0.smithi045.stdout:472: done (1 left) 2021-07-31T01:47:07.528 INFO:tasks.rados.rados.0.smithi045.stdout:473: done (0 left) 2021-07-31T01:47:07.528 INFO:tasks.rados.rados.0.smithi045.stdout:474: setattr oid 39 current snap is 38 2021-07-31T01:47:07.529 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 22 current snap is 38 2021-07-31T01:47:07.529 INFO:tasks.rados.rados.0.smithi045.stdout:475: seq_num 160 ranges {794624=286720} 2021-07-31T01:47:07.534 INFO:tasks.rados.rados.0.smithi045.stdout:475: writing smithi04569269-22 from 794624 to 1081344 tid 1 2021-07-31T01:47:07.534 INFO:tasks.rados.rados.0.smithi045.stdout:476: snap_remove snap 26 2021-07-31T01:47:08.535 INFO:tasks.rados.rados.0.smithi045.stdout:476: done (2 left) 2021-07-31T01:47:08.535 INFO:tasks.rados.rados.0.smithi045.stdout:477: rmattr oid 7 current snap is 38 2021-07-31T01:47:08.535 INFO:tasks.rados.rados.0.smithi045.stdout:477: done (2 left) 2021-07-31T01:47:08.535 INFO:tasks.rados.rados.0.smithi045.stdout:478: copy_from oid 19 from oid 37 current snap is 38 2021-07-31T01:47:08.536 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 39 v 156 (ObjNum 94 snap 15 seq_num 94) dirty exists 2021-07-31T01:47:08.536 INFO:tasks.rados.rados.0.smithi045.stdout:475: finishing write tid 1 to smithi04569269-22 2021-07-31T01:47:08.536 INFO:tasks.rados.rados.0.smithi045.stdout:475: finishing write tid 2 to smithi04569269-22 2021-07-31T01:47:08.536 INFO:tasks.rados.rados.0.smithi045.stdout:475: finishing write tid 3 to smithi04569269-22 2021-07-31T01:47:08.536 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 22 v 103 (ObjNum 160 snap 38 seq_num 160) dirty exists 2021-07-31T01:47:08.537 INFO:tasks.rados.rados.0.smithi045.stdout:475: left oid 22 (ObjNum 160 snap 38 seq_num 160) 2021-07-31T01:47:08.537 INFO:tasks.rados.rados.0.smithi045.stdout:474: done (2 left) 2021-07-31T01:47:08.537 INFO:tasks.rados.rados.0.smithi045.stdout:475: done (1 left) 2021-07-31T01:47:08.537 INFO:tasks.rados.rados.0.smithi045.stdout:479: copy_from oid 48 from oid 31 current snap is 38 2021-07-31T01:47:08.538 INFO:tasks.rados.rados.0.smithi045.stdout:480: delete oid 16 current snap is 38 2021-07-31T01:47:08.538 INFO:tasks.rados.rados.0.smithi045.stdout:478: finishing copy_from to smithi04569269-19 2021-07-31T01:47:08.538 INFO:tasks.rados.rados.0.smithi045.stdout:478: got expected ENOENT (src dne) 2021-07-31T01:47:08.539 INFO:tasks.rados.rados.0.smithi045.stdout:478: finishing copy_from racing read to smithi04569269-19 2021-07-31T01:47:08.549 INFO:tasks.rados.rados.0.smithi045.stdout:478: done (2 left) 2021-07-31T01:47:08.550 INFO:tasks.rados.rados.0.smithi045.stdout:480: done (1 left) 2021-07-31T01:47:08.550 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 43 current snap is 38 2021-07-31T01:47:08.551 INFO:tasks.rados.rados.0.smithi045.stdout:481: seq_num 162 ranges {3497984=319488} 2021-07-31T01:47:08.563 INFO:tasks.rados.rados.0.smithi045.stdout:481: writing smithi04569269-43 from 3497984 to 3817472 tid 1 2021-07-31T01:47:08.563 INFO:tasks.rados.rados.0.smithi045.stdout:482: delete oid 18 current snap is 38 2021-07-31T01:47:08.563 INFO:tasks.rados.rados.0.smithi045.stdout:479: finishing copy_from to smithi04569269-48 2021-07-31T01:47:08.564 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 48 v 298 (ObjNum 157 snap 37 seq_num 157) dirty exists 2021-07-31T01:47:08.564 INFO:tasks.rados.rados.0.smithi045.stdout:479: finishing copy_from racing read to smithi04569269-48 2021-07-31T01:47:08.568 INFO:tasks.rados.rados.0.smithi045.stdout:479: done (2 left) 2021-07-31T01:47:08.568 INFO:tasks.rados.rados.0.smithi045.stdout:482: done (1 left) 2021-07-31T01:47:08.569 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 15 current snap is 38 2021-07-31T01:47:08.569 INFO:tasks.rados.rados.0.smithi045.stdout:483: seq_num 164 ranges {0=352256} 2021-07-31T01:47:08.569 INFO:tasks.rados.rados.0.smithi045.stdout:483: writing smithi04569269-15 from 0 to 352256 tid 1 2021-07-31T01:47:08.569 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 28 current snap is 38 2021-07-31T01:47:08.569 INFO:tasks.rados.rados.0.smithi045.stdout:481: finishing write tid 1 to smithi04569269-43 2021-07-31T01:47:08.570 INFO:tasks.rados.rados.0.smithi045.stdout:481: finishing write tid 2 to smithi04569269-43 2021-07-31T01:47:08.570 INFO:tasks.rados.rados.0.smithi045.stdout:484: seq_num 165 ranges {2088960=368640} 2021-07-31T01:47:08.583 INFO:tasks.rados.rados.0.smithi045.stdout:484: writing smithi04569269-28 from 2088960 to 2457600 tid 1 2021-07-31T01:47:08.584 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 3 current snap is 38 2021-07-31T01:47:08.584 INFO:tasks.rados.rados.0.smithi045.stdout:485: seq_num 166 ranges {794624=385024} 2021-07-31T01:47:08.588 INFO:tasks.rados.rados.0.smithi045.stdout:485: writing smithi04569269-3 from 794624 to 1179648 tid 1 2021-07-31T01:47:08.588 INFO:tasks.rados.rados.0.smithi045.stdout:486: read oid 40 snap -1 2021-07-31T01:47:08.588 INFO:tasks.rados.rados.0.smithi045.stdout:486: expect (ObjNum 158 snap 37 seq_num 158) 2021-07-31T01:47:08.589 INFO:tasks.rados.rados.0.smithi045.stdout:487: snap_remove snap 36 2021-07-31T01:47:08.626 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:08 smithi154 conmon[34553]: cluster 2021-07-31T01:47:07.378192+0000 mgr.y (mgr.14146) 315 : cluster [DBG] pgmap v341: 25 pgs: 25 active+clean; 108 MiB data, 308 MiB used, 715 GiB / 715 GiB avail; 576 KiB/s rd, 20 KiB/s wr, 8 op/s 2021-07-31T01:47:08.627 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:08 smithi154 conmon[34553]: cluster 2021-07-31T01:47:07.526329+0000 mon.a (mon.0) 730 : cluster [DBG] osdmap e128: 8 total, 8 up, 8 in 2021-07-31T01:47:08.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:08 smithi045 conmon[39718]: cluster 2021-07-31T01:47:07 2021-07-31T01:47:08.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:08 smithi045 conmon[39718]: .378192+0000 mgr.y (mgr.14146) 315 : cluster [DBG] pgmap v341: 25 pgs: 25 active+clean; 108 MiB data, 308 MiB used, 715 GiB / 715 GiB avail; 576 KiB/s rd, 20 KiB/s wr, 8 op/s 2021-07-31T01:47:08.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:08 smithi045 conmon[39718]: cluster 2021 2021-07-31T01:47:08.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:08 smithi045 conmon[39718]: -07-31T01:47:07. 2021-07-31T01:47:08.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:08 smithi045 conmon[39718]: 526329+0000 mon.a (mon.0 2021-07-31T01:47:08.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:08 smithi045 conmon[39718]: ) 730 : cluster [DBG] osdmap e128: 8 total, 8 up, 8 in 2021-07-31T01:47:08.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:08 smithi045 conmon[39718]: 2021-07-31T01:47:08.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:08 smithi045 conmon[33655]: cluster 2021-07-31T01:47:07.378192+0000 2021-07-31T01:47:08.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:08 smithi045 conmon[33655]: mgr.y (mgr.14146) 315 : cluster [DBG] pgmap v341: 25 pgs: 25 active+clean; 108 MiB data, 308 MiB used, 715 GiB / 715 GiB avail; 576 KiB/s rd, 20 KiB/s wr, 8 op/s 2021-07-31T01:47:08.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:08 smithi045 conmon[33655]: cluster 2021-07-31T01:47:07.526329+0000 mon.a (mon. 2021-07-31T01:47:08.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:08 smithi045 conmon[33655]: 0) 730 : cluster [DBG] osdmap e128: 8 total, 8 up, 8 in 2021-07-31T01:47:09.539 INFO:tasks.rados.rados.0.smithi045.stdout:487: done (5 left) 2021-07-31T01:47:09.540 INFO:tasks.rados.rados.0.smithi045.stdout:488: rollback oid 24 current snap is 38 2021-07-31T01:47:09.540 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 24 to 31 2021-07-31T01:47:09.540 INFO:tasks.rados.rados.0.smithi045.stdout:489: delete oid 50 current snap is 38 2021-07-31T01:47:09.543 INFO:tasks.rados.rados.0.smithi045.stdout:481: finishing write tid 3 to smithi04569269-43 2021-07-31T01:47:09.544 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 43 v 146 (ObjNum 162 snap 38 seq_num 162) dirty exists 2021-07-31T01:47:09.544 INFO:tasks.rados.rados.0.smithi045.stdout:481: left oid 43 (ObjNum 162 snap 38 seq_num 162) 2021-07-31T01:47:09.544 INFO:tasks.rados.rados.0.smithi045.stdout:483: finishing write tid 1 to smithi04569269-15 2021-07-31T01:47:09.544 INFO:tasks.rados.rados.0.smithi045.stdout:483: finishing write tid 2 to smithi04569269-15 2021-07-31T01:47:09.545 INFO:tasks.rados.rados.0.smithi045.stdout:483: finishing write tid 3 to smithi04569269-15 2021-07-31T01:47:09.545 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 15 v 126 (ObjNum 164 snap 38 seq_num 164) dirty exists 2021-07-31T01:47:09.545 INFO:tasks.rados.rados.0.smithi045.stdout:483: left oid 15 (ObjNum 164 snap 38 seq_num 164) 2021-07-31T01:47:09.545 INFO:tasks.rados.rados.0.smithi045.stdout:484: finishing write tid 1 to smithi04569269-28 2021-07-31T01:47:09.545 INFO:tasks.rados.rados.0.smithi045.stdout:484: finishing write tid 2 to smithi04569269-28 2021-07-31T01:47:09.546 INFO:tasks.rados.rados.0.smithi045.stdout:484: finishing write tid 3 to smithi04569269-28 2021-07-31T01:47:09.546 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 28 v 188 (ObjNum 165 snap 38 seq_num 165) dirty exists 2021-07-31T01:47:09.546 INFO:tasks.rados.rados.0.smithi045.stdout:484: left oid 28 (ObjNum 165 snap 38 seq_num 165) 2021-07-31T01:47:09.546 INFO:tasks.rados.rados.0.smithi045.stdout:485: finishing write tid 1 to smithi04569269-3 2021-07-31T01:47:09.547 INFO:tasks.rados.rados.0.smithi045.stdout:486: expect (ObjNum 158 snap 37 seq_num 158) 2021-07-31T01:47:09.551 INFO:tasks.rados.rados.0.smithi045.stdout:485: finishing write tid 2 to smithi04569269-3 2021-07-31T01:47:09.551 INFO:tasks.rados.rados.0.smithi045.stdout:485: finishing write tid 3 to smithi04569269-3 2021-07-31T01:47:09.551 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 3 v 313 (ObjNum 166 snap 38 seq_num 166) dirty exists 2021-07-31T01:47:09.551 INFO:tasks.rados.rados.0.smithi045.stdout:485: left oid 3 (ObjNum 166 snap 38 seq_num 166) 2021-07-31T01:47:09.551 INFO:tasks.rados.rados.0.smithi045.stdout:481: done (6 left) 2021-07-31T01:47:09.552 INFO:tasks.rados.rados.0.smithi045.stdout:483: done (5 left) 2021-07-31T01:47:09.552 INFO:tasks.rados.rados.0.smithi045.stdout:484: done (4 left) 2021-07-31T01:47:09.552 INFO:tasks.rados.rados.0.smithi045.stdout:485: done (3 left) 2021-07-31T01:47:09.552 INFO:tasks.rados.rados.0.smithi045.stdout:486: done (2 left) 2021-07-31T01:47:09.553 INFO:tasks.rados.rados.0.smithi045.stdout:489: done (1 left) 2021-07-31T01:47:09.553 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 4 current snap is 38 2021-07-31T01:47:09.553 INFO:tasks.rados.rados.0.smithi045.stdout:490: seq_num 168 ranges {131072=417792} 2021-07-31T01:47:09.554 INFO:tasks.rados.rados.0.smithi045.stdout:490: writing smithi04569269-4 from 131072 to 548864 tid 1 2021-07-31T01:47:09.554 INFO:tasks.rados.rados.0.smithi045.stdout:491: snap_remove snap 22 2021-07-31T01:47:10.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:09 smithi154 conmon[34553]: cluster 2021-07-31T01:47:08.531568+0000 mon.a (mon.0) 731 : cluster [DBG] osdmap e129: 8 total, 8 up, 8 in 2021-07-31T01:47:10.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:09 smithi154 conmon[34553]: 2021-07-31T01:47:10.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:09 smithi045 conmon[33655]: cluster 2021-07-31T01:47:08.531568+0000 2021-07-31T01:47:10.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:09 smithi045 conmon[33655]: mon.a (mon.0) 731 : cluster [DBG] osdmap e129: 8 total, 8 up, 8 in 2021-07-31T01:47:10.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:09 smithi045 conmon[39718]: cluster 2021-07-31T01:47:10.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:09 smithi045 conmon[39718]: 2021-07-31T01:47:10.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:09 smithi045 conmon[39718]: 2021 2021-07-31T01:47:10.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:09 smithi045 conmon[39718]: - 2021-07-31T01:47:10.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:09 smithi045 conmon[39718]: 07 2021-07-31T01:47:10.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:09 smithi045 conmon[39718]: - 2021-07-31T01:47:10.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:09 smithi045 conmon[39718]: 31 2021-07-31T01:47:10.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:09 smithi045 conmon[39718]: T 2021-07-31T01:47:10.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:09 smithi045 conmon[39718]: 01 2021-07-31T01:47:10.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:09 smithi045 conmon[39718]: : 2021-07-31T01:47:10.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:09 smithi045 conmon[39718]: 47 2021-07-31T01:47:10.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:09 smithi045 conmon[39718]: : 2021-07-31T01:47:10.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:09 smithi045 conmon[39718]: 08 2021-07-31T01:47:10.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:09 smithi045 conmon[39718]: . 2021-07-31T01:47:10.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:09 smithi045 conmon[39718]: 531568 2021-07-31T01:47:10.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:09 smithi045 conmon[39718]: +0000 2021-07-31T01:47:10.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:09 smithi045 conmon[39718]: 2021-07-31T01:47:10.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:09 smithi045 conmon[39718]: mon.a 2021-07-31T01:47:10.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:09 smithi045 conmon[39718]: ( 2021-07-31T01:47:10.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:09 smithi045 conmon[39718]: mon 2021-07-31T01:47:10.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:09 smithi045 conmon[39718]: . 2021-07-31T01:47:10.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:09 smithi045 conmon[39718]: 0 2021-07-31T01:47:10.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:09 smithi045 conmon[39718]: ) 2021-07-31T01:47:10.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:09 smithi045 conmon[39718]: 731 2021-07-31T01:47:10.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:09 smithi045 conmon[39718]: : 2021-07-31T01:47:10.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:09 smithi045 conmon[39718]: cluster 2021-07-31T01:47:10.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:09 smithi045 conmon[39718]: 2021-07-31T01:47:10.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:09 smithi045 conmon[39718]: [DBG] 2021-07-31T01:47:10.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:09 smithi045 conmon[39718]: 2021-07-31T01:47:10.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:09 smithi045 conmon[39718]: osdmap e129: 8 total, 8 up, 8 in 2021-07-31T01:47:10.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:09 smithi045 conmon[39718]: 2021-07-31T01:47:10.544 INFO:tasks.rados.rados.0.smithi045.stdout:491: done (2 left) 2021-07-31T01:47:10.544 INFO:tasks.rados.rados.0.smithi045.stdout:492: copy_from oid 13 from oid 43 current snap is 38 2021-07-31T01:47:10.544 INFO:tasks.rados.rados.0.smithi045.stdout:488: finishing rollback tid 0 to smithi04569269-24 2021-07-31T01:47:10.545 INFO:tasks.rados.rados.0.smithi045.stdout:488: finishing rollback tid 1 to smithi04569269-24 2021-07-31T01:47:10.545 INFO:tasks.rados.rados.0.smithi045.stdout:488: finishing rollback tid 2 to smithi04569269-24 2021-07-31T01:47:10.545 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 24 v 57 (ObjNum 89 snap 15 seq_num 89) dirty exists 2021-07-31T01:47:10.545 INFO:tasks.rados.rados.0.smithi045.stdout:490: finishing write tid 1 to smithi04569269-4 2021-07-31T01:47:10.546 INFO:tasks.rados.rados.0.smithi045.stdout:490: finishing write tid 2 to smithi04569269-4 2021-07-31T01:47:10.546 INFO:tasks.rados.rados.0.smithi045.stdout:490: finishing write tid 3 to smithi04569269-4 2021-07-31T01:47:10.546 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 4 v 133 (ObjNum 168 snap 38 seq_num 168) dirty exists 2021-07-31T01:47:10.546 INFO:tasks.rados.rados.0.smithi045.stdout:490: left oid 4 (ObjNum 168 snap 38 seq_num 168) 2021-07-31T01:47:10.547 INFO:tasks.rados.rados.0.smithi045.stdout:488: done (2 left) 2021-07-31T01:47:10.547 INFO:tasks.rados.rados.0.smithi045.stdout:490: done (1 left) 2021-07-31T01:47:10.547 INFO:tasks.rados.rados.0.smithi045.stdout:493: setattr oid 27 current snap is 38 2021-07-31T01:47:10.548 INFO:tasks.rados.rados.0.smithi045.stdout:494: rollback oid 3 current snap is 38 2021-07-31T01:47:10.548 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 3 to 35 2021-07-31T01:47:10.548 INFO:tasks.rados.rados.0.smithi045.stdout:495: setattr oid 40 current snap is 38 2021-07-31T01:47:10.549 INFO:tasks.rados.rados.0.smithi045.stdout:496: snap_create 2021-07-31T01:47:10.552 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 27 v 149 (ObjNum 0 snap 0 seq_num 0) dirty exists 2021-07-31T01:47:10.553 INFO:tasks.rados.rados.0.smithi045.stdout:494: finishing rollback tid 0 to smithi04569269-3 2021-07-31T01:47:10.554 INFO:tasks.rados.rados.0.smithi045.stdout:494: finishing rollback tid 1 to smithi04569269-3 2021-07-31T01:47:10.554 INFO:tasks.rados.rados.0.smithi045.stdout:494: finishing rollback tid 2 to smithi04569269-3 2021-07-31T01:47:10.555 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 3 v 322 (ObjNum 144 snap 31 seq_num 144) dirty exists 2021-07-31T01:47:10.557 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 40 v 143 (ObjNum 158 snap 37 seq_num 158) dirty exists 2021-07-31T01:47:10.578 INFO:tasks.rados.rados.0.smithi045.stdout:492: finishing copy_from to smithi04569269-13 2021-07-31T01:47:10.578 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 13 v 146 (ObjNum 162 snap 38 seq_num 162) dirty exists 2021-07-31T01:47:10.579 INFO:tasks.rados.rados.0.smithi045.stdout:492: finishing copy_from racing read to smithi04569269-13 2021-07-31T01:47:11.003 INFO:journalctl@ceph.osd.5.smithi154.stdout:Jul 31 01:47:10 smithi154 conmon[42050]: debug 2021-07-31T01:47:10.560+0000 7fc9b4863700 -1 osd.5 pg_epoch: 131 pg[3.5s0( v 129'20 (0'0,129'20] local-lis/les=56/57 n=2 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [5,3,2]p5(0) r=0 lpr=56 crt=129'20 lcod 118'18 mlcod 118'18 active+clean+snaptrim trimq=[2b~1] ps=[2f~1,41~1,43~1]] removing snap head 2021-07-31T01:47:11.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:10 smithi154 conmon[34553]: cluster 2021-07-31T01:47:09.378693+0000 mgr.y (mgr.14146) 316 2021-07-31T01:47:11.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:10 smithi154 conmon[34553]: : cluster [DBG] pgmap v344: 25 pgs: 25 active+clean; 114 MiB data, 321 MiB used, 715 GiB / 715 GiB avail; 7.2 MiB/s rd, 1.9 MiB/s wr, 43 op/s 2021-07-31T01:47:11.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:10 smithi154 conmon[34553]: cluster 2021-07-31T01:47:09.537297+0000 mon.a (mon.0) 732 : cluster [DBG] osdmap e130: 8 total, 8 up, 8 in 2021-07-31T01:47:11.004 INFO:journalctl@ceph.osd.7.smithi154.stdout:Jul 31 01:47:10 smithi154 conmon[49149]: debug 2021-07-31T01:47:10.564+0000 7f8a719fc700 -1 osd.7 pg_epoch: 131 pg[3.ds0( v 131'200 (0'0,131'200] local-lis/les=56/57 n=11 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [7,5,3]p7(0) r=0 lpr=56 luod=131'198 crt=131'198 lcod 131'196 mlcod 131'196 active+clean+snaptrim trimq=[2b~1] ps=[2f~1,41~1,43~1]] removing snap head 2021-07-31T01:47:11.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:10 smithi045 conmon[39718]: cluster 2021-07-31T01:47:09.378693+0000 mgr.y (mgr.14146) 316 : cluster 2021-07-31T01:47:11.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:10 smithi045 conmon[39718]: [DBG] pgmap v344: 25 pgs: 25 active+clean; 114 MiB data, 321 MiB used, 715 GiB / 715 GiB avail; 7.2 MiB/s rd, 1.9 MiB/s wr, 43 op/s 2021-07-31T01:47:11.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:10 smithi045 conmon[39718]: cluster 2021-07-31T01:47:09.537297+0000 mon.a (mon.0) 732 : cluster [DBG] osdmap e130: 8 total, 8 up, 8 in 2021-07-31T01:47:11.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:10 smithi045 conmon[33655]: cluster 2021-07-31T01:47:09.378693+0000 mgr.y (mgr.14146) 316 : 2021-07-31T01:47:11.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:10 smithi045 conmon[33655]: cluster [DBG] pgmap v344: 25 pgs: 25 active+clean; 114 MiB data, 321 MiB used, 715 GiB / 715 GiB avail; 7.2 MiB/s rd, 1.9 MiB/s wr, 43 op/s 2021-07-31T01:47:11.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:10 smithi045 conmon[33655]: cluster 2021-07-31T01:47:09.537297+0000 mon.a (mon.0) 732 : cluster [DBG] osdmap e130: 8 total, 8 up, 8 in 2021-07-31T01:47:11.484 INFO:tasks.rados.rados.0.smithi045.stdout:492: done (4 left) 2021-07-31T01:47:11.485 INFO:tasks.rados.rados.0.smithi045.stdout:493: done (3 left) 2021-07-31T01:47:11.485 INFO:tasks.rados.rados.0.smithi045.stdout:494: done (2 left) 2021-07-31T01:47:11.486 INFO:tasks.rados.rados.0.smithi045.stdout:495: done (1 left) 2021-07-31T01:47:11.486 INFO:tasks.rados.rados.0.smithi045.stdout:496: done (0 left) 2021-07-31T01:47:11.486 INFO:tasks.rados.rados.0.smithi045.stdout:497: snap_create 2021-07-31T01:47:12.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:11 smithi154 conmon[34553]: cluster 2021-07-31T01:47:10.542294+0000 mon.a (mon.0) 733 : cluster [DBG] osdmap e131: 8 total, 8 up, 8 in 2021-07-31T01:47:12.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:11 smithi154 conmon[34553]: cluster 2021-07-31T01:47:11.484048+0000 mon.a (mon.0) 734 : cluster [DBG] osdmap e132: 8 total, 8 up, 8 in 2021-07-31T01:47:12.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:11 smithi045 conmon[33655]: cluster 2021-07-31T01:47:10.542294+0000 2021-07-31T01:47:12.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:11 smithi045 conmon[33655]: mon.a (mon.0) 733 : cluster [DBG] osdmap e131: 8 total, 8 up, 8 in 2021-07-31T01:47:12.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:11 smithi045 conmon[33655]: cluster 2021-07-31T01:47:11.484048+0000 mon.a (mon.0) 734 : 2021-07-31T01:47:12.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:11 smithi045 conmon[33655]: cluster [DBG] osdmap e132: 8 total, 8 up, 8 in 2021-07-31T01:47:12.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:11 smithi045 conmon[39718]: cluster 2021-07-31T01:47:10.542294+0000 mon.a ( 2021-07-31T01:47:12.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:11 smithi045 conmon[39718]: mon.0) 733 : cluster [DBG] osdmap e131: 8 total, 8 up, 8 in 2021-07-31T01:47:12.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:11 smithi045 conmon[39718]: cluster 2021-07-31T01:47:11.484048+0000 2021-07-31T01:47:12.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:11 smithi045 conmon[39718]: mon.a (mon.0) 734 : cluster [DBG] osdmap e132: 8 total, 8 up, 8 in 2021-07-31T01:47:12.492 INFO:tasks.rados.rados.0.smithi045.stdout:497: done (0 left) 2021-07-31T01:47:12.493 INFO:tasks.rados.rados.0.smithi045.stdout:498: setattr oid 4 current snap is 40 2021-07-31T01:47:12.493 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 14 current snap is 40 2021-07-31T01:47:12.493 INFO:tasks.rados.rados.0.smithi045.stdout:499: seq_num 169 ranges {1654784=434176} 2021-07-31T01:47:12.505 INFO:tasks.rados.rados.0.smithi045.stdout:499: writing smithi04569269-14 from 1654784 to 2088960 tid 1 2021-07-31T01:47:12.506 INFO:tasks.rados.rados.0.smithi045.stdout:500: read oid 16 snap 31 2021-07-31T01:47:12.506 INFO:tasks.rados.rados.0.smithi045.stdout:500: expect (ObjNum 118 snap 22 seq_num 118) 2021-07-31T01:47:12.506 INFO:tasks.rados.rados.0.smithi045.stdout:501: rmattr oid 10 current snap is 40 2021-07-31T01:47:12.506 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 4 v 143 (ObjNum 168 snap 38 seq_num 168) dirty exists 2021-07-31T01:47:12.507 INFO:tasks.rados.rados.0.smithi045.stdout:498: done (3 left) 2021-07-31T01:47:12.507 INFO:tasks.rados.rados.0.smithi045.stdout:501: done (2 left) 2021-07-31T01:47:12.507 INFO:tasks.rados.rados.0.smithi045.stdout:502: copy_from oid 37 from oid 50 current snap is 40 2021-07-31T01:47:12.507 INFO:tasks.rados.rados.0.smithi045.stdout:503: delete oid 21 current snap is 40 2021-07-31T01:47:12.509 INFO:tasks.rados.rados.0.smithi045.stdout:503: done (3 left) 2021-07-31T01:47:12.510 INFO:tasks.rados.rados.0.smithi045.stdout:504: rmattr oid 27 current snap is 40 2021-07-31T01:47:12.510 INFO:tasks.rados.rados.0.smithi045.stdout:505: rollback oid 30 current snap is 40 2021-07-31T01:47:12.510 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 30 to 39 2021-07-31T01:47:12.511 INFO:tasks.rados.rados.0.smithi045.stdout:506: copy_from oid 18 from oid 45 current snap is 40 2021-07-31T01:47:12.511 INFO:tasks.rados.rados.0.smithi045.stdout:507: read oid 44 snap -1 2021-07-31T01:47:12.511 INFO:tasks.rados.rados.0.smithi045.stdout:507: expect (ObjNum 142 snap 30 seq_num 142) 2021-07-31T01:47:12.511 INFO:tasks.rados.rados.0.smithi045.stdout:508: snap_create 2021-07-31T01:47:12.511 INFO:tasks.rados.rados.0.smithi045.stdout:502: finishing copy_from to smithi04569269-37 2021-07-31T01:47:12.512 INFO:tasks.rados.rados.0.smithi045.stdout:502: got expected ENOENT (src dne) 2021-07-31T01:47:12.512 INFO:tasks.rados.rados.0.smithi045.stdout:502: finishing copy_from racing read to smithi04569269-37 2021-07-31T01:47:12.512 INFO:tasks.rados.rados.0.smithi045.stdout:499: finishing write tid 1 to smithi04569269-14 2021-07-31T01:47:12.513 INFO:tasks.rados.rados.0.smithi045.stdout:499: finishing write tid 2 to smithi04569269-14 2021-07-31T01:47:12.513 INFO:tasks.rados.rados.0.smithi045.stdout:505: finishing rollback tid 0 to smithi04569269-30 2021-07-31T01:47:12.514 INFO:tasks.rados.rados.0.smithi045.stdout:505: finishing rollback tid 1 to smithi04569269-30 2021-07-31T01:47:12.514 INFO:tasks.rados.rados.0.smithi045.stdout:505: finishing rollback tid 2 to smithi04569269-30 2021-07-31T01:47:12.514 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 30 v 63 (ObjNum 29 snap 0 seq_num 29) dirty exists 2021-07-31T01:47:12.515 INFO:tasks.rados.rados.0.smithi045.stdout:506: finishing copy_from to smithi04569269-18 2021-07-31T01:47:12.515 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 18 v 119 (ObjNum 0 snap 21873 seq_num 3189649408) dirty exists 2021-07-31T01:47:12.515 INFO:tasks.rados.rados.0.smithi045.stdout:506: finishing copy_from racing read to smithi04569269-18 2021-07-31T01:47:12.516 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 27 v 160 (ObjNum 1364417124 snap 1800492651 seq_num 1666410069) dirty exists 2021-07-31T01:47:12.517 INFO:tasks.rados.rados.0.smithi045.stdout:500: expect (ObjNum 118 snap 22 seq_num 118) 2021-07-31T01:47:12.570 INFO:tasks.rados.rados.0.smithi045.stdout:499: finishing write tid 3 to smithi04569269-14 2021-07-31T01:47:12.571 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 14 v 158 (ObjNum 169 snap 40 seq_num 169) dirty exists 2021-07-31T01:47:12.571 INFO:tasks.rados.rados.0.smithi045.stdout:499: left oid 14 (ObjNum 169 snap 40 seq_num 169) 2021-07-31T01:47:12.571 INFO:tasks.rados.rados.0.smithi045.stdout:507: expect (ObjNum 142 snap 30 seq_num 142) 2021-07-31T01:47:13.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:12 smithi154 conmon[34553]: cluster 2021-07-31T01:47:11.379245+0000 mgr.y (mgr.14146) 317 : 2021-07-31T01:47:13.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:12 smithi154 conmon[34553]: cluster [DBG] pgmap v347: 25 pgs: 25 active+clean; 114 MiB data, 321 MiB used, 715 GiB / 715 GiB avail; 6.7 MiB/s rd, 1.8 MiB/s wr, 41 op/s 2021-07-31T01:47:13.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:12 smithi154 conmon[34553]: cluster 2021-07-31T01:47:12.491667+0000 mon.a (mon.0) 735 : cluster [DBG] osdmap e133: 8 total, 8 up, 8 in 2021-07-31T01:47:13.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:12 smithi045 conmon[39718]: cluster 2021-07-31T01:47:11.379245+0000 mgr.y 2021-07-31T01:47:13.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:12 smithi045 conmon[39718]: (mgr.14146) 317 : cluster [DBG] pgmap v347: 25 pgs: 25 active+clean; 114 MiB data, 321 MiB used, 715 GiB / 715 GiB avail; 6.7 MiB/s rd, 1.8 MiB/s wr, 41 op/s 2021-07-31T01:47:13.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:12 smithi045 conmon[39718]: cluster 2021-07-31T01 2021-07-31T01:47:13.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:12 smithi045 conmon[39718]: :47:12.491667+0000 mon.a (mon.0) 735 : cluster [DBG] osdmap e133: 8 total, 8 up, 8 in 2021-07-31T01:47:13.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:12 smithi045 conmon[33655]: cluster 2021-07-31T01:47:11.379245+0000 2021-07-31T01:47:13.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:12 smithi045 conmon[33655]: mgr.y (mgr.14146) 317 : cluster [DBG] pgmap v347: 25 pgs: 25 active+clean; 114 MiB data, 321 MiB used, 715 GiB / 715 GiB avail; 6.7 MiB/s rd, 1.8 MiB/s wr, 41 op/s 2021-07-31T01:47:13.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:12 smithi045 conmon[33655]: cluster 2021-07-31T01:47:12.491667+0000 mon.a (mon.0) 735 2021-07-31T01:47:13.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:12 smithi045 conmon[33655]: : cluster [DBG] osdmap e133: 8 total, 8 up, 8 in 2021-07-31T01:47:13.495 INFO:tasks.rados.rados.0.smithi045.stdout:499: done (7 left) 2021-07-31T01:47:13.495 INFO:tasks.rados.rados.0.smithi045.stdout:500: done (6 left) 2021-07-31T01:47:13.495 INFO:tasks.rados.rados.0.smithi045.stdout:502: done (5 left) 2021-07-31T01:47:13.496 INFO:tasks.rados.rados.0.smithi045.stdout:504: done (4 left) 2021-07-31T01:47:13.496 INFO:tasks.rados.rados.0.smithi045.stdout:505: done (3 left) 2021-07-31T01:47:13.496 INFO:tasks.rados.rados.0.smithi045.stdout:506: done (2 left) 2021-07-31T01:47:13.496 INFO:tasks.rados.rados.0.smithi045.stdout:507: done (1 left) 2021-07-31T01:47:13.497 INFO:tasks.rados.rados.0.smithi045.stdout:508: done (0 left) 2021-07-31T01:47:13.497 INFO:tasks.rados.rados.0.smithi045.stdout:509: rollback oid 11 current snap is 41 2021-07-31T01:47:13.497 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 11 to 37 2021-07-31T01:47:13.497 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 17 current snap is 41 2021-07-31T01:47:13.498 INFO:tasks.rados.rados.0.smithi045.stdout:510: seq_num 171 ranges {0=466944} 2021-07-31T01:47:13.499 INFO:tasks.rados.rados.0.smithi045.stdout:510: writing smithi04569269-17 from 0 to 466944 tid 1 2021-07-31T01:47:13.499 INFO:tasks.rados.rados.0.smithi045.stdout:511: rollback oid 47 current snap is 41 2021-07-31T01:47:13.499 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 47 to 37 2021-07-31T01:47:13.499 INFO:tasks.rados.rados.0.smithi045.stdout:512: snap_remove snap 33 2021-07-31T01:47:13.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:47:13 smithi045 conmon[61210]: level=error ts=2021-07-31T01:47:13.506Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:47:13.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:47:13 smithi045 conmon[61210]: level=error ts=2021-07-31T01:47:13.506Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:47:13.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:47:13 smithi045 conmon[61210]: level=error ts=2021-07-31T01:47:13.506Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T01:47:14.501 INFO:tasks.rados.rados.0.smithi045.stdout:512: done (3 left) 2021-07-31T01:47:14.501 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 18 current snap is 41 2021-07-31T01:47:14.502 INFO:tasks.rados.rados.0.smithi045.stdout:509: finishing rollback tid 0 to smithi04569269-11 2021-07-31T01:47:14.502 INFO:tasks.rados.rados.0.smithi045.stdout:513: seq_num 172 ranges {0=483328} 2021-07-31T01:47:14.504 INFO:tasks.rados.rados.0.smithi045.stdout:513: writing smithi04569269-18 from 0 to 483328 tid 1 2021-07-31T01:47:14.504 INFO:tasks.rados.rados.0.smithi045.stdout:514: delete oid 2 current snap is 41 2021-07-31T01:47:14.504 INFO:tasks.rados.rados.0.smithi045.stdout:509: finishing rollback tid 1 to smithi04569269-11 2021-07-31T01:47:14.505 INFO:tasks.rados.rados.0.smithi045.stdout:509: finishing rollback tid 2 to smithi04569269-11 2021-07-31T01:47:14.505 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 11 v 188 (ObjNum 10 snap 0 seq_num 10) dirty exists 2021-07-31T01:47:14.505 INFO:tasks.rados.rados.0.smithi045.stdout:511: finishing rollback tid 0 to smithi04569269-47 2021-07-31T01:47:14.505 INFO:tasks.rados.rados.0.smithi045.stdout:511: finishing rollback tid 1 to smithi04569269-47 2021-07-31T01:47:14.505 INFO:tasks.rados.rados.0.smithi045.stdout:511: finishing rollback tid 2 to smithi04569269-47 2021-07-31T01:47:14.506 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 47 v 22 (ObjNum 46 snap 0 seq_num 46) dirty exists 2021-07-31T01:47:14.506 INFO:tasks.rados.rados.0.smithi045.stdout:510: finishing write tid 1 to smithi04569269-17 2021-07-31T01:47:14.506 INFO:tasks.rados.rados.0.smithi045.stdout:510: finishing write tid 2 to smithi04569269-17 2021-07-31T01:47:14.506 INFO:tasks.rados.rados.0.smithi045.stdout:510: finishing write tid 3 to smithi04569269-17 2021-07-31T01:47:14.507 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 17 v 206 (ObjNum 171 snap 41 seq_num 171) dirty exists 2021-07-31T01:47:14.507 INFO:tasks.rados.rados.0.smithi045.stdout:510: left oid 17 (ObjNum 171 snap 41 seq_num 171) 2021-07-31T01:47:14.507 INFO:tasks.rados.rados.0.smithi045.stdout:509: done (4 left) 2021-07-31T01:47:14.507 INFO:tasks.rados.rados.0.smithi045.stdout:510: done (3 left) 2021-07-31T01:47:14.508 INFO:tasks.rados.rados.0.smithi045.stdout:511: done (2 left) 2021-07-31T01:47:14.508 INFO:tasks.rados.rados.0.smithi045.stdout:514: done (1 left) 2021-07-31T01:47:14.508 INFO:tasks.rados.rados.0.smithi045.stdout:515: snap_create 2021-07-31T01:47:14.513 INFO:tasks.rados.rados.0.smithi045.stdout:513: finishing write tid 1 to smithi04569269-18 2021-07-31T01:47:14.513 INFO:tasks.rados.rados.0.smithi045.stdout:513: finishing write tid 2 to smithi04569269-18 2021-07-31T01:47:14.514 INFO:tasks.rados.rados.0.smithi045.stdout:513: finishing write tid 3 to smithi04569269-18 2021-07-31T01:47:14.515 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 18 v 147 (ObjNum 172 snap 41 seq_num 172) dirty exists 2021-07-31T01:47:14.515 INFO:tasks.rados.rados.0.smithi045.stdout:513: left oid 18 (ObjNum 172 snap 41 seq_num 172) 2021-07-31T01:47:14.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:14 smithi154 conmon[34553]: cluster 2021-07-31T01:47:13.379688+0000 mgr.y (mgr.14146) 318 : cluster [DBG] pgmap v350: 25 pgs: 25 active+clean; 107 MiB data, 322 MiB used, 715 GiB / 715 GiB avail; 187 KiB/s rd, 1.1 MiB/s wr, 13 op/s 2021-07-31T01:47:14.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:14 smithi154 conmon[34553]: cluster 2021-07-31T01:47:13.493784+0000 mon.a (mon.0) 736 : cluster [DBG] osdmap e134: 8 total, 8 up, 8 in 2021-07-31T01:47:14.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:14 smithi045 conmon[33655]: cluster 2021-07-31T01:47:13.379688+0000 mgr.y (mgr.14146 2021-07-31T01:47:14.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:14 smithi045 conmon[33655]: ) 318 : cluster [DBG] pgmap v350: 25 pgs: 25 active+clean; 107 MiB data, 322 MiB used, 715 GiB / 715 GiB avail; 187 KiB/s rd, 1.1 MiB/s wr, 13 op/s 2021-07-31T01:47:14.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:14 smithi045 conmon[33655]: cluster 2021-07-31T01:47:13.493784+0000 mon.a (mon.0) 736 : cluster [DBG] osdmap e134: 8 total, 8 up, 8 in 2021-07-31T01:47:14.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:14 smithi045 conmon[39718]: cluster 2021-07-31T01:47:13.379688+0000 mgr.y (mgr.14146) 318 : cluster 2021-07-31T01:47:14.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:14 smithi045 conmon[39718]: [DBG] pgmap v350: 25 pgs: 25 active+clean; 107 MiB data, 322 MiB used, 715 GiB / 715 GiB avail; 187 KiB/s rd, 1.1 MiB/s wr, 13 op/s 2021-07-31T01:47:14.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:14 smithi045 conmon[39718]: cluster 2021-07-31T01:47:13.493784+0000 mon.a (mon.0) 736 : cluster [DBG] osdmap e134: 8 total, 8 up, 8 in 2021-07-31T01:47:15.504 INFO:tasks.rados.rados.0.smithi045.stdout:513: done (1 left) 2021-07-31T01:47:15.505 INFO:tasks.rados.rados.0.smithi045.stdout:515: done (0 left) 2021-07-31T01:47:15.505 INFO:tasks.rados.rados.0.smithi045.stdout:516: snap_create 2021-07-31T01:47:15.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:15 smithi154 conmon[34553]: cluster 2021-07-31T01:47:14.499124+0000 mon.a (mon.0) 737 : 2021-07-31T01:47:15.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:15 smithi154 conmon[34553]: cluster [DBG] osdmap e135: 8 total, 8 up, 8 in 2021-07-31T01:47:15.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:15 smithi045 conmon[39718]: cluster 2021-07-31T01:47:14 2021-07-31T01:47:15.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:15 smithi045 conmon[39718]: .499124+0000 mon.a (mon.0) 737 : cluster [DBG] osdmap e135: 8 total, 8 up, 8 in 2021-07-31T01:47:15.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:15 smithi045 conmon[33655]: cluster 2021-07-31T01:47:14.499124+0000 2021-07-31T01:47:15.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:15 smithi045 conmon[33655]: mon.a (mon.0) 737 : cluster [DBG] osdmap e135: 8 total, 8 up, 8 in 2021-07-31T01:47:16.486 INFO:tasks.rados.rados.0.smithi045.stdout:516: done (0 left) 2021-07-31T01:47:16.486 INFO:tasks.rados.rados.0.smithi045.stdout:517: snap_remove snap 37 2021-07-31T01:47:16.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:16 smithi045 conmon[39718]: cluster 2021-07-31T01:47:15.380108+0000 mgr.y (mgr. 2021-07-31T01:47:16.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:16 smithi045 conmon[39718]: 14146) 319 : cluster [DBG] pgmap v353: 25 pgs: 25 active+clean; 93 MiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.6 MiB/s rd, 1.2 MiB/s wr, 22 op/s 2021-07-31T01:47:16.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:16 smithi045 conmon[39718]: cluster 2021 2021-07-31T01:47:16.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:16 smithi045 conmon[39718]: -07-31T01:47:15.504313+0000 mon.a (mon.0) 738 : 2021-07-31T01:47:16.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:16 smithi045 conmon[39718]: cluster [DBG] osdmap e136: 8 total, 8 up, 8 in 2021-07-31T01:47:16.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:16 smithi045 conmon[39718]: cluster 2021-07-31 2021-07-31T01:47:16.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:16 smithi045 conmon[39718]: T01:47:16.485598+0000 mon.a (mon.0 2021-07-31T01:47:16.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:16 smithi045 conmon[39718]: ) 739 : cluster [DBG] osdmap e137: 8 total, 8 up, 8 in 2021-07-31T01:47:16.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:16 smithi045 conmon[33655]: cluster 2021-07-31T01:47:15.380108 2021-07-31T01:47:16.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:16 smithi045 conmon[33655]: +0000 mgr.y (mgr.14146) 319 : cluster [DBG] pgmap v353: 25 pgs: 25 active+clean; 93 MiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.6 MiB/s rd, 1.2 MiB/s wr, 22 op/s 2021-07-31T01:47:16.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:16 smithi045 conmon[33655]: cluster 2021-07-31T01:47:15.504313+0000 mon.a (mon 2021-07-31T01:47:16.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:16 smithi045 conmon[33655]: .0) 738 : cluster [DBG] osdmap e136: 8 total, 8 up, 8 in 2021-07-31T01:47:16.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:16 smithi045 conmon[33655]: cluster 2021 2021-07-31T01:47:16.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:16 smithi045 conmon[33655]: -07-31T01:47:16.485598+0000 mon.a 2021-07-31T01:47:16.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:16 smithi045 conmon[33655]: (mon.0) 739 : cluster [DBG] osdmap e137: 8 total, 8 up, 8 in 2021-07-31T01:47:16.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:16 smithi045 conmon[33655]: 2021-07-31T01:47:17.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:16 smithi154 conmon[34553]: cluster 2021-07-31T01:47:15.380108+0000 mgr.y (mgr.14146) 319 : cluster [DBG] pgmap v353: 25 pgs: 25 active+clean; 93 MiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.6 MiB/s rd, 1.2 MiB/s wr, 22 op/s 2021-07-31T01:47:17.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:16 smithi154 conmon[34553]: 2021-07-31T01:47:17.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:16 smithi154 conmon[34553]: cluster 2021-07-31T01:47:15.504313+0000 mon.a (mon.0) 738 : cluster [DBG] osdmap e136: 8 total, 8 up, 8 in 2021-07-31T01:47:17.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:16 smithi154 conmon[34553]: cluster 2021-07-31T01:47:16.485598+0000 mon.a (mon.0) 739 : cluster [DBG] osdmap e137: 8 total, 8 up, 8 in 2021-07-31T01:47:17.494 INFO:tasks.rados.rados.0.smithi045.stdout:517: done (0 left) 2021-07-31T01:47:17.495 INFO:tasks.rados.rados.0.smithi045.stdout:518: read oid 21 snap -1 2021-07-31T01:47:17.495 INFO:tasks.rados.rados.0.smithi045.stdout:518: expect deleted 2021-07-31T01:47:17.495 INFO:tasks.rados.rados.0.smithi045.stdout:519: snap_create 2021-07-31T01:47:18.497 INFO:tasks.rados.rados.0.smithi045.stdout:518: done (1 left) 2021-07-31T01:47:18.498 INFO:tasks.rados.rados.0.smithi045.stdout:519: done (0 left) 2021-07-31T01:47:18.498 INFO:tasks.rados.rados.0.smithi045.stdout:520: copy_from oid 24 from oid 25 current snap is 44 2021-07-31T01:47:18.498 INFO:tasks.rados.rados.0.smithi045.stdout:521: snap_remove snap 43 2021-07-31T01:47:18.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:18 smithi154 conmon[34553]: cluster 2021-07-31T01:47:17.380476+0000 mgr.y (mgr.14146) 320 : cluster [DBG] 2021-07-31T01:47:18.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:18 smithi154 conmon[34553]: pgmap v356: 25 pgs: 25 active+clean; 93 MiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.4 MiB/s rd, 102 KiB/s wr, 9 op/s 2021-07-31T01:47:18.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:18 smithi154 conmon[34553]: cluster 2021-07-31T01:47:17.491990+0000 mon.a (mon.0) 740 : cluster [DBG] osdmap e138: 8 total, 8 up, 8 in 2021-07-31T01:47:18.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:18 smithi045 conmon[33655]: cluster 2021-07-31T01:47:17.380476+0000 mgr.y (mgr 2021-07-31T01:47:18.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:18 smithi045 conmon[33655]: .14146) 320 : cluster [DBG] pgmap v356: 25 pgs: 25 active+clean; 93 MiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.4 MiB/s rd, 102 KiB/s wr, 9 op/s 2021-07-31T01:47:18.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:18 smithi045 conmon[33655]: cluster 2021-07-31T01:47:17.491990+0000 mon.a (mon.0) 740 : cluster [DBG] osdmap e138: 8 total, 8 up, 8 in 2021-07-31T01:47:18.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:18 smithi045 conmon[39718]: cluster 2021-07-31T01:47:17.380476+0000 mgr.y (mgr 2021-07-31T01:47:18.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:18 smithi045 conmon[39718]: .14146) 320 : cluster [DBG] pgmap v356: 25 pgs: 25 active+clean; 93 MiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.4 MiB/s rd, 102 KiB/s wr, 9 op/s 2021-07-31T01:47:18.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:18 smithi045 conmon[39718]: cluster 2021-07-31T01:47:17.491990+0000 mon.a (mon.0) 740 : cluster 2021-07-31T01:47:18.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:18 smithi045 conmon[39718]: [DBG] osdmap e138: 8 total, 8 up, 8 in 2021-07-31T01:47:19.503 INFO:tasks.rados.rados.0.smithi045.stdout:521: done (1 left) 2021-07-31T01:47:19.503 INFO:tasks.rados.rados.0.smithi045.stdout:522: rollback oid 16 current snap is 44 2021-07-31T01:47:19.503 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 16 to 38 2021-07-31T01:47:19.504 INFO:tasks.rados.rados.0.smithi045.stdout:520: finishing copy_from to smithi04569269-24 2021-07-31T01:47:19.504 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 24 v 12 (ObjNum 36 snap 0 seq_num 36) dirty exists 2021-07-31T01:47:19.504 INFO:tasks.rados.rados.0.smithi045.stdout:520: finishing copy_from racing read to smithi04569269-24 2021-07-31T01:47:19.504 INFO:tasks.rados.rados.0.smithi045.stdout:520: done (1 left) 2021-07-31T01:47:19.505 INFO:tasks.rados.rados.0.smithi045.stdout:523: read oid 47 snap 41 2021-07-31T01:47:19.505 INFO:tasks.rados.rados.0.smithi045.stdout:523: expect (ObjNum 46 snap 0 seq_num 46) 2021-07-31T01:47:19.505 INFO:tasks.rados.rados.0.smithi045.stdout:524: rollback oid 17 current snap is 44 2021-07-31T01:47:19.505 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 17 to 41 2021-07-31T01:47:19.505 INFO:tasks.rados.rados.0.smithi045.stdout:525: snap_create 2021-07-31T01:47:19.508 INFO:tasks.rados.rados.0.smithi045.stdout:522: finishing rollback tid 1 to smithi04569269-16 2021-07-31T01:47:19.509 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 16 v 105 (ObjNum 0 snap 0 seq_num 0) dirty dne 2021-07-31T01:47:19.509 INFO:tasks.rados.rados.0.smithi045.stdout:524: finishing rollback tid 0 to smithi04569269-17 2021-07-31T01:47:19.509 INFO:tasks.rados.rados.0.smithi045.stdout:524: finishing rollback tid 1 to smithi04569269-17 2021-07-31T01:47:19.509 INFO:tasks.rados.rados.0.smithi045.stdout:524: finishing rollback tid 2 to smithi04569269-17 2021-07-31T01:47:19.509 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 17 v 220 (ObjNum 171 snap 41 seq_num 171) dirty exists 2021-07-31T01:47:19.512 INFO:tasks.rados.rados.0.smithi045.stdout:523: expect (ObjNum 46 snap 0 seq_num 46) 2021-07-31T01:47:19.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:19 smithi154 conmon[34553]: cluster 2021-07-31T01:47:18.496602+0000 mon.a 2021-07-31T01:47:19.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:19 smithi154 conmon[34553]: (mon.0) 741 : cluster [DBG] osdmap e139: 8 total, 8 up, 8 in 2021-07-31T01:47:19.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:19 smithi154 conmon[34553]: cephadm 2021-07-31T01:47:18.800360+0000 mgr.y (mgr.14146) 321 : cephadm [INF] Upgrade: Updating mgr.x 2021-07-31T01:47:19.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:19 smithi154 conmon[34553]: audit 2021-07-31T01:47:18.806701+0000 mon.a (mon.0) 742 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:47:19.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:19 smithi154 conmon[34553]: audit 2021-07-31T01:47:18.807672+0000 mon.a (mon.0) 743 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2021-07-31T01:47:19.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:19 smithi154 conmon[34553]: audit 2021-07-31T01:47:18.808613+0000 mon.a (mon.0) 744 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2021-07-31T01:47:19.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:19 smithi154 conmon[34553]: audit 2021-07-31T01:47:18.809498+0000 mon.a (mon.0) 745 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:47:19.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:19 smithi154 conmon[34553]: cephadm 2021-07-31T01:47:18.810451+0000 mgr.y (mgr.14146) 322 : cephadm [INF] Deploying daemon mgr.x on smithi154 2021-07-31T01:47:19.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:19 smithi045 conmon[33655]: cluster 2021-07-31T01:47:18.496602+0000 mon.a (mon.0) 741 2021-07-31T01:47:19.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:19 smithi045 conmon[33655]: : cluster [DBG] osdmap e139: 8 total, 8 up, 8 in 2021-07-31T01:47:19.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:19 smithi045 conmon[33655]: cephadm 2021-07-31T01:47:18.800360+0000 mgr.y (mgr.14146) 321 : cephadm [INF] Upgrade: Updating mgr.x 2021-07-31T01:47:19.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:19 smithi045 conmon[33655]: audit 2021-07-31T01:47:18.806701+0000 mon.a (mon.0) 2021-07-31T01:47:19.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:19 smithi045 conmon[33655]: 742 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:47:19.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:19 smithi045 conmon[33655]: audit 2021- 2021-07-31T01:47:19.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:19 smithi045 conmon[33655]: 07-31T01:47:18.807672+0000 mon.a (mon. 2021-07-31T01:47:19.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:19 smithi045 conmon[33655]: 0) 743 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2021-07-31T01:47:19.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:19 smithi045 conmon[33655]: audit 2021- 2021-07-31T01:47:19.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:19 smithi045 conmon[33655]: 07-31T01:47:18.808613+0000 mon.a (mon. 2021-07-31T01:47:19.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:19 smithi045 conmon[33655]: 0) 744 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2021-07-31T01:47:19.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:19 smithi045 conmon[33655]: audit 2021 2021-07-31T01:47:19.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:19 smithi045 conmon[33655]: -07-31T01:47:18.809498+0000 mon.a ( 2021-07-31T01:47:19.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:19 smithi045 conmon[33655]: mon.0) 745 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:47:19.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:19 smithi045 conmon[33655]: cephadm 2021-07-31T01:47:18 2021-07-31T01:47:19.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:19 smithi045 conmon[33655]: .810451+0000 mgr.y (mgr.14146) 322 : cephadm [INF] Deploying daemon mgr.x on smithi154 2021-07-31T01:47:19.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:19 smithi045 conmon[39718]: cluster 2021-07-31T01:47:18.496602+0000 mon.a (mon.0) 741 : 2021-07-31T01:47:19.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:19 smithi045 conmon[39718]: cluster [DBG] osdmap e139: 8 total, 8 up, 8 in 2021-07-31T01:47:19.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:19 smithi045 conmon[39718]: cephadm 2021-07-31T01:47:18.800360+0000 mgr.y (mgr 2021-07-31T01:47:19.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:19 smithi045 conmon[39718]: .14146) 321 : cephadm [INF] Upgrade: Updating mgr.x 2021-07-31T01:47:19.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:19 smithi045 conmon[39718]: audit 2021-07-31T01:47 2021-07-31T01:47:19.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:19 smithi045 conmon[39718]: :18.806701+0000 mon.a (mon.0) 742 : audit [INF] 2021-07-31T01:47:19.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:19 smithi045 conmon[39718]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:47:19.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:19 smithi045 conmon[39718]: audit 2021-07-31T01:47:19.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:19 smithi045 conmon[39718]: 2021-07-31T01:47:18.807672+0000 mon.a 2021-07-31T01:47:19.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:19 smithi045 conmon[39718]: (mon.0) 743 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2021-07-31T01:47:19.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:19 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T01:47:19.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:19 smithi045 conmon[39718]: T01:47:18.808613+0000 mon.a (mon. 2021-07-31T01:47:19.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:19 smithi045 conmon[39718]: 0) 744 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2021-07-31T01:47:19.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:19 smithi045 conmon[39718]: audit 2021-07-31T01 2021-07-31T01:47:19.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:19 smithi045 conmon[39718]: :47:18.809498+0000 mon.a (mon.0 2021-07-31T01:47:19.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:19 smithi045 conmon[39718]: ) 745 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:47:19.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:19 smithi045 conmon[39718]: cephadm 2021-07 2021-07-31T01:47:19.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:19 smithi045 conmon[39718]: -31T01:47:18.810451+0000 2021-07-31T01:47:19.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:19 smithi045 conmon[39718]: mgr.y (mgr.14146) 322 : cephadm [INF] Deploying daemon mgr.x on smithi154 2021-07-31T01:47:20.508 INFO:tasks.rados.rados.0.smithi045.stdout:522: done (3 left) 2021-07-31T01:47:20.509 INFO:tasks.rados.rados.0.smithi045.stdout:523: done (2 left) 2021-07-31T01:47:20.509 INFO:tasks.rados.rados.0.smithi045.stdout:524: done (1 left) 2021-07-31T01:47:20.509 INFO:tasks.rados.rados.0.smithi045.stdout:525: done (0 left) 2021-07-31T01:47:20.510 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 33 current snap is 45 2021-07-31T01:47:20.510 INFO:tasks.rados.rados.0.smithi045.stdout:526: seq_num 174 ranges {4448256=516096} 2021-07-31T01:47:20.535 INFO:tasks.rados.rados.0.smithi045.stdout:526: writing smithi04569269-33 from 4448256 to 4964352 tid 1 2021-07-31T01:47:20.536 INFO:tasks.rados.rados.0.smithi045.stdout:527: read oid 27 snap -1 2021-07-31T01:47:20.536 INFO:tasks.rados.rados.0.smithi045.stdout:527: expect (ObjNum 0 snap 0 seq_num 0) 2021-07-31T01:47:20.536 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 2 current snap is 45 2021-07-31T01:47:20.536 INFO:tasks.rados.rados.0.smithi045.stdout:528: seq_num 175 ranges {0=540672} 2021-07-31T01:47:20.538 INFO:tasks.rados.rados.0.smithi045.stdout:528: writing smithi04569269-2 from 0 to 540672 tid 1 2021-07-31T01:47:20.538 INFO:tasks.rados.rados.0.smithi045.stdout:529: snap_create 2021-07-31T01:47:20.539 INFO:tasks.rados.rados.0.smithi045.stdout:527: expect (ObjNum 0 snap 0 seq_num 0) 2021-07-31T01:47:20.540 INFO:tasks.rados.rados.0.smithi045.stdout:526: finishing write tid 1 to smithi04569269-33 2021-07-31T01:47:20.540 INFO:tasks.rados.rados.0.smithi045.stdout:526: finishing write tid 2 to smithi04569269-33 2021-07-31T01:47:20.541 INFO:tasks.rados.rados.0.smithi045.stdout:526: finishing write tid 3 to smithi04569269-33 2021-07-31T01:47:20.541 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 33 v 205 (ObjNum 174 snap 45 seq_num 174) dirty exists 2021-07-31T01:47:20.541 INFO:tasks.rados.rados.0.smithi045.stdout:526: left oid 33 (ObjNum 174 snap 45 seq_num 174) 2021-07-31T01:47:20.542 INFO:tasks.rados.rados.0.smithi045.stdout:528: finishing write tid 1 to smithi04569269-2 2021-07-31T01:47:20.543 INFO:tasks.rados.rados.0.smithi045.stdout:528: finishing write tid 2 to smithi04569269-2 2021-07-31T01:47:20.543 INFO:tasks.rados.rados.0.smithi045.stdout:528: finishing write tid 3 to smithi04569269-2 2021-07-31T01:47:20.544 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 2 v 56 (ObjNum 175 snap 45 seq_num 175) dirty exists 2021-07-31T01:47:20.544 INFO:tasks.rados.rados.0.smithi045.stdout:528: left oid 2 (ObjNum 175 snap 45 seq_num 175) 2021-07-31T01:47:20.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:20 smithi154 conmon[34553]: cluster 2021-07-31T01:47:19.381005+0000 mgr.y ( 2021-07-31T01:47:20.683 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:20 smithi154 conmon[34553]: mgr.14146) 323 : cluster [DBG] pgmap v359: 25 pgs: 25 active+clean; 92 MiB data, 332 MiB used, 715 GiB / 715 GiB avail; 3.1 MiB/s rd, 486 KiB/s wr, 5 op/s 2021-07-31T01:47:20.683 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:20 smithi154 conmon[34553]: cluster 2021-07-31T01:47:19.502133+0000 mon.a (mon.0) 746 : cluster [DBG] osdmap e140: 8 total, 8 up, 8 in 2021-07-31T01:47:20.683 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:20 smithi154 systemd[1]: Stopping Ceph mgr.x for f52d2c14-f19f-11eb-8c24-001a4aab830c... 2021-07-31T01:47:20.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:20 smithi045 conmon[33655]: cluster 2021-07-31T01:47:19.381005+0000 mgr.y (mgr.14146) 323 2021-07-31T01:47:20.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:20 smithi045 conmon[33655]: : cluster [DBG] pgmap v359: 25 pgs: 25 active+clean; 92 MiB data, 332 MiB used, 715 GiB / 715 GiB avail; 3.1 MiB/s rd, 486 KiB/s wr, 5 op/s 2021-07-31T01:47:20.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:20 smithi045 conmon[33655]: cluster 2021-07-31T01:47:19.502133+0000 mon.a (mon.0) 746 : cluster [DBG] osdmap e140: 8 total, 8 up, 8 in 2021-07-31T01:47:20.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:20 smithi045 conmon[39718]: cluster 2021-07-31T01:47:19.381005+0000 mgr.y (mgr 2021-07-31T01:47:20.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:20 smithi045 conmon[39718]: .14146) 323 : cluster [DBG] pgmap v359: 25 pgs: 25 active+clean; 92 MiB data, 332 MiB used, 715 GiB / 715 GiB avail; 3.1 MiB/s rd, 486 KiB/s wr, 5 op/s 2021-07-31T01:47:20.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:20 smithi045 conmon[39718]: cluster 2021-07-31T01:47: 2021-07-31T01:47:20.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:20 smithi045 conmon[39718]: 19.502133+0000 mon.a (mon.0) 746 : cluster [DBG] osdmap e140: 8 total, 8 up, 8 in 2021-07-31T01:47:21.458 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:21 smithi154 podman[57305]: 2021-07-31 01:47:21.079969866 +0000 UTC m=+0.486717657 container remove f9af406258e19b744cd21dcc3c4c155e8f0149130edfbf220700512ce2c5672d (image=docker.io/ceph/daemon-base@sha256:0191c38131eb5e8b2e3cce0839223fa943f24bf2153c2767cb5bdb4a2973d559, name=ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c-mgr.x, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Base Image, maintainer=Dimitri Savineau , org.label-schema.build-date=20201204, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=0b3dfd099f6a46b8c8fecd28a50330b63408461f, ceph=True, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=master-0b3dfd0) 2021-07-31T01:47:21.458 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:21 smithi154 podman[57265]: ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c-mgr.x 2021-07-31T01:47:21.458 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:21 smithi154 systemd[1]: ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c@mgr.x.service: Main process exited, code=exited, status=143/n/a 2021-07-31T01:47:21.459 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:21 smithi154 systemd[1]: ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c@mgr.x.service: Failed with result 'exit-code'. 2021-07-31T01:47:21.459 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:21 smithi154 systemd[1]: Stopped Ceph mgr.x for f52d2c14-f19f-11eb-8c24-001a4aab830c. 2021-07-31T01:47:21.459 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:21 smithi154 systemd[1]: Starting Ceph mgr.x for f52d2c14-f19f-11eb-8c24-001a4aab830c... 2021-07-31T01:47:21.487 INFO:tasks.rados.rados.0.smithi045.stdout:526: done (3 left) 2021-07-31T01:47:21.488 INFO:tasks.rados.rados.0.smithi045.stdout:527: done (2 left) 2021-07-31T01:47:21.488 INFO:tasks.rados.rados.0.smithi045.stdout:528: done (1 left) 2021-07-31T01:47:21.488 INFO:tasks.rados.rados.0.smithi045.stdout:529: done (0 left) 2021-07-31T01:47:21.488 INFO:tasks.rados.rados.0.smithi045.stdout:530: read oid 37 snap -1 2021-07-31T01:47:21.489 INFO:tasks.rados.rados.0.smithi045.stdout:530: expect deleted 2021-07-31T01:47:21.489 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 42 current snap is 46 2021-07-31T01:47:21.489 INFO:tasks.rados.rados.0.smithi045.stdout:531: seq_num 176 ranges {0=557056} 2021-07-31T01:47:21.491 INFO:tasks.rados.rados.0.smithi045.stdout:531: writing smithi04569269-42 from 0 to 557056 tid 1 2021-07-31T01:47:21.491 INFO:tasks.rados.rados.0.smithi045.stdout:532: read oid 4 snap 35 2021-07-31T01:47:21.491 INFO:tasks.rados.rados.0.smithi045.stdout:532: expect deleted 2021-07-31T01:47:21.491 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 13 current snap is 46 2021-07-31T01:47:21.492 INFO:tasks.rados.rados.0.smithi045.stdout:533: seq_num 177 ranges {3817472=573440} 2021-07-31T01:47:21.516 INFO:tasks.rados.rados.0.smithi045.stdout:533: writing smithi04569269-13 from 3817472 to 4390912 tid 1 2021-07-31T01:47:21.516 INFO:tasks.rados.rados.0.smithi045.stdout:534: read oid 38 snap -1 2021-07-31T01:47:21.516 INFO:tasks.rados.rados.0.smithi045.stdout:534: expect (ObjNum 121 snap 23 seq_num 121) 2021-07-31T01:47:21.516 INFO:tasks.rados.rados.0.smithi045.stdout:535: copy_from oid 46 from oid 19 current snap is 46 2021-07-31T01:47:21.517 INFO:tasks.rados.rados.0.smithi045.stdout:530: done (5 left) 2021-07-31T01:47:21.517 INFO:tasks.rados.rados.0.smithi045.stdout:536: delete oid 35 current snap is 46 2021-07-31T01:47:21.517 INFO:tasks.rados.rados.0.smithi045.stdout:531: finishing write tid 1 to smithi04569269-42 2021-07-31T01:47:21.517 INFO:tasks.rados.rados.0.smithi045.stdout:531: finishing write tid 2 to smithi04569269-42 2021-07-31T01:47:21.518 INFO:tasks.rados.rados.0.smithi045.stdout:531: finishing write tid 3 to smithi04569269-42 2021-07-31T01:47:21.518 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 42 v 222 (ObjNum 176 snap 46 seq_num 176) dirty exists 2021-07-31T01:47:21.518 INFO:tasks.rados.rados.0.smithi045.stdout:531: left oid 42 (ObjNum 176 snap 46 seq_num 176) 2021-07-31T01:47:21.519 INFO:tasks.rados.rados.0.smithi045.stdout:531: done (5 left) 2021-07-31T01:47:21.520 INFO:tasks.rados.rados.0.smithi045.stdout:532: done (4 left) 2021-07-31T01:47:21.520 INFO:tasks.rados.rados.0.smithi045.stdout:536: done (3 left) 2021-07-31T01:47:21.520 INFO:tasks.rados.rados.0.smithi045.stdout:537: read oid 45 snap -1 2021-07-31T01:47:21.520 INFO:tasks.rados.rados.0.smithi045.stdout:537: expect (ObjNum 0 snap 0 seq_num 0) 2021-07-31T01:47:21.521 INFO:tasks.rados.rados.0.smithi045.stdout:538: snap_remove snap 44 2021-07-31T01:47:21.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:21 smithi154 conmon[34553]: cluster 2021-07-31T01:47:20.507421+0000 mon.a (mon. 2021-07-31T01:47:21.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:21 smithi154 conmon[34553]: 0) 747 : cluster [DBG] osdmap e141: 8 total, 8 up, 8 in 2021-07-31T01:47:21.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:21 smithi154 conmon[34553]: cluster 2021-07-31T01:47:21.486557+0000 mon.a (mon.0) 748 : cluster [DBG] osdmap e142: 8 total, 8 up, 8 in 2021-07-31T01:47:21.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:21 smithi154 podman[57452]: 2021-07-31 01:47:21.367579162 +0000 UTC m=+0.031084659 image pull quay.ceph.io/ceph-ci/ceph@sha256:58353ac33be8de5bd041245814b4a069acff6ffcb54969da96873a191aef6c81 2021-07-31T01:47:21.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:21 smithi154 podman[57452]: 2021-07-31 01:47:21.528333208 +0000 UTC m=+0.191838706 container create 1260dfd1d8793604105a6154d3936de035848afa9fc79a93fd7615cf50939bee (image=quay.ceph.io/ceph-ci/ceph@sha256:58353ac33be8de5bd041245814b4a069acff6ffcb54969da96873a191aef6c81, name=ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c-mgr.x, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, GIT_CLEAN=True, org.label-schema.name=CentOS Base Image, ceph=True, GIT_COMMIT=4d96298d43ed77d80efdb44c23a181c2021ebba7, maintainer=Dimitri Savineau , org.label-schema.vendor=CentOS, org.label-schema.build-date=20201204, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-kefu-testing-2021-07-31-0023-f25860a, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2021-07-31T01:47:21.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:21 smithi045 conmon[39718]: cluster 2021-07-31T01:47:20.507421 2021-07-31T01:47:21.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:21 smithi045 conmon[39718]: +0000 mon.a (mon.0) 747 : cluster [DBG] osdmap e141: 8 total, 8 up, 8 in 2021-07-31T01:47:21.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:21 smithi045 conmon[39718]: cluster 2021-07-31T01:47:21.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:21 smithi045 conmon[39718]: 2021-07-31T01: 2021-07-31T01:47:21.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:21 smithi045 conmon[39718]: 47:21.486557+0000 mon.a (mon. 2021-07-31T01:47:21.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:21 smithi045 conmon[39718]: 0) 748 : cluster [DBG] osdmap e142: 8 total, 8 up, 8 in 2021-07-31T01:47:21.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:21 smithi045 conmon[33655]: cluster 2021-07-31T01:47:20.507421+0000 mon.a (mon.0) 747 : cluster [DBG] osdmap e141: 8 total, 8 up, 8 in 2021-07-31T01:47:21.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:21 smithi045 conmon[33655]: cluster 2021-07-31T01:47:21.486557+0000 mon.a (mon.0) 748 : cluster [DBG] osdmap e142: 8 total, 8 up, 8 in 2021-07-31T01:47:22.253 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:21 smithi154 podman[57452]: 2021-07-31 01:47:21.888154778 +0000 UTC m=+0.551660265 container init 1260dfd1d8793604105a6154d3936de035848afa9fc79a93fd7615cf50939bee (image=quay.ceph.io/ceph-ci/ceph@sha256:58353ac33be8de5bd041245814b4a069acff6ffcb54969da96873a191aef6c81, name=ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c-mgr.x, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=4d96298d43ed77d80efdb44c23a181c2021ebba7, org.label-schema.name=CentOS Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Dimitri Savineau , CEPH_POINT_RELEASE=, org.label-schema.build-date=20201204, ceph=True, GIT_CLEAN=True, io.buildah.version=1.19.8, RELEASE=wip-kefu-testing-2021-07-31-0023-f25860a, org.label-schema.vendor=CentOS) 2021-07-31T01:47:22.253 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:21 smithi154 conmon[57492]: debug 2021-07-31T01:47:21.921+0000 7fcbf9dda700 1 -- 172.21.15.154:0/2025517513 <== mon.2 v2:172.21.15.154:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55c9d85f6340 con 0x55c9d85cf000 2021-07-31T01:47:22.253 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:21 smithi154 podman[57452]: 2021-07-31 01:47:21.938102706 +0000 UTC m=+0.601608192 container start 1260dfd1d8793604105a6154d3936de035848afa9fc79a93fd7615cf50939bee (image=quay.ceph.io/ceph-ci/ceph@sha256:58353ac33be8de5bd041245814b4a069acff6ffcb54969da96873a191aef6c81, name=ceph-f52d2c14-f19f-11eb-8c24-001a4aab830c-mgr.x, org.label-schema.build-date=20201204, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=4d96298d43ed77d80efdb44c23a181c2021ebba7, maintainer=Dimitri Savineau , org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-kefu-testing-2021-07-31-0023-f25860a, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.19.8, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Base Image) 2021-07-31T01:47:22.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:21 smithi154 bash[57379]: 1260dfd1d8793604105a6154d3936de035848afa9fc79a93fd7615cf50939bee 2021-07-31T01:47:22.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:21 smithi154 systemd[1]: Started Ceph mgr.x for f52d2c14-f19f-11eb-8c24-001a4aab830c. 2021-07-31T01:47:22.513 INFO:tasks.rados.rados.0.smithi045.stdout:538: done (4 left) 2021-07-31T01:47:22.514 INFO:tasks.rados.rados.0.smithi045.stdout:539: snap_create 2021-07-31T01:47:22.514 INFO:tasks.rados.rados.0.smithi045.stdout:533: finishing write tid 1 to smithi04569269-13 2021-07-31T01:47:22.514 INFO:tasks.rados.rados.0.smithi045.stdout:537: expect (ObjNum 0 snap 0 seq_num 0) 2021-07-31T01:47:22.516 INFO:tasks.rados.rados.0.smithi045.stdout:533: finishing write tid 2 to smithi04569269-13 2021-07-31T01:47:22.517 INFO:tasks.rados.rados.0.smithi045.stdout:533: finishing write tid 3 to smithi04569269-13 2021-07-31T01:47:22.517 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 13 v 160 (ObjNum 177 snap 46 seq_num 177) dirty exists 2021-07-31T01:47:22.517 INFO:tasks.rados.rados.0.smithi045.stdout:533: left oid 13 (ObjNum 177 snap 46 seq_num 177) 2021-07-31T01:47:22.517 INFO:tasks.rados.rados.0.smithi045.stdout:534: expect (ObjNum 121 snap 23 seq_num 121) 2021-07-31T01:47:22.654 INFO:tasks.rados.rados.0.smithi045.stdout:535: finishing copy_from to smithi04569269-46 2021-07-31T01:47:22.654 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 46 v 41 (ObjNum 94 snap 15 seq_num 94) dirty exists 2021-07-31T01:47:22.655 INFO:tasks.rados.rados.0.smithi045.stdout:535: finishing copy_from racing read to smithi04569269-46 2021-07-31T01:47:23.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:23 smithi154 conmon[34553]: cluster 2021-07-31T01:47:21.381560+0000 mgr.y (mgr.14146) 324 : cluster [DBG] pgmap v362: 25 pgs: 25 active+clean; 92 MiB data, 332 MiB used, 715 GiB / 715 GiB avail; 3.1 MiB/s rd, 486 KiB/s wr, 5 op/s 2021-07-31T01:47:23.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:23 smithi154 conmon[34553]: audit 2021-07-31T01:47:22.006329+0000 mon.a (mon.0) 749 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:47:23.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:23 smithi154 conmon[34553]: audit 2021-07-31T01:47:22.007283+0000 mon.a (mon.0) 750 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:47:23.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:23 smithi154 conmon[34553]: audit 2021-07-31T01:47:22.008615+0000 mon.a (mon.0) 751 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:47:23.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:23 smithi154 conmon[34553]: audit 2021-07-31T01:47:22.009498+0000 mon.a (mon.0) 752 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:47:23.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:23 smithi154 conmon[34553]: cluster 2021-07-31T01:47:22.513124+0000 mon.a (mon.0) 753 : cluster [DBG] osdmap e143: 8 total, 8 up, 8 in 2021-07-31T01:47:23.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:23 smithi045 conmon[39718]: cluster 2021-07-31T01:47:21. 2021-07-31T01:47:23.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:23 smithi045 conmon[39718]: 381560+0000 mgr.y (mgr.14146) 324 : cluster [DBG] pgmap v362: 25 pgs: 25 active+clean; 92 MiB data, 332 MiB used, 715 GiB / 715 GiB avail; 3.1 MiB/s rd, 486 KiB/s wr, 5 op/s 2021-07-31T01:47:23.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:23 smithi045 conmon[39718]: audit 2021-07-31T01: 2021-07-31T01:47:23.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:23 smithi045 conmon[39718]: 47:22.006329+0000 mon.a (mon.0) 2021-07-31T01:47:23.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:23 smithi045 conmon[39718]: 749 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:47:23.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:23 smithi045 conmon[39718]: audit 2021-07-31T01: 2021-07-31T01:47:23.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:23 smithi045 conmon[39718]: 47:22.007283+0000 mon.a ( 2021-07-31T01:47:23.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:23 smithi045 conmon[39718]: mon.0) 750 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:47:23.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:23 smithi045 conmon[39718]: 2021-07-31T01:47:23.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:23 smithi045 conmon[39718]: audit 2021-07-31T01:47:23.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:23 smithi045 conmon[39718]: 2021-07-31T01:47:22.008615+0000 mon.a ( 2021-07-31T01:47:23.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:23 smithi045 conmon[39718]: mon.0) 751 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:47:23.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:23 smithi045 conmon[39718]: 2021-07-31T01:47:23.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:23 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T01:47:23.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:23 smithi045 conmon[39718]: 31T01:47:22.009498 2021-07-31T01:47:23.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:23 smithi045 conmon[39718]: +0000 mon.a (mon.0) 752 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:47:23.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:23 smithi045 conmon[39718]: cluster 2021-07-31T01:47:23.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:23 smithi045 conmon[39718]: 2021-07-31T 2021-07-31T01:47:23.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:23 smithi045 conmon[39718]: 01:47:22.513124+0000 mon.a ( 2021-07-31T01:47:23.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:23 smithi045 conmon[39718]: mon.0) 753 : cluster [DBG] osdmap e143: 8 total, 8 up, 8 in 2021-07-31T01:47:23.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:23 smithi045 conmon[33655]: cluster 2021-07-31T01:47:21.381560 2021-07-31T01:47:23.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:23 smithi045 conmon[33655]: +0000 mgr.y (mgr.14146) 324 : cluster [DBG] pgmap v362: 25 pgs: 25 active+clean; 92 MiB data, 332 MiB used, 715 GiB / 715 GiB avail; 3.1 MiB/s rd, 486 KiB/s wr, 5 op/s 2021-07-31T01:47:23.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:23 smithi045 conmon[33655]: audit 2021-07-31T01:47:22.006329+0000 mon.a (mon.0) 749 : audit 2021-07-31T01:47:23.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:23 smithi045 conmon[33655]: [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:47:23.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:23 smithi045 conmon[33655]: audit 2021-07-31T01:47:23.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:23 smithi045 conmon[33655]: 2021-07-31T01:47:22.007283+0000 mon.a (mon.0 2021-07-31T01:47:23.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:23 smithi045 conmon[33655]: ) 750 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:47:23.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:23 smithi045 conmon[33655]: audit 2021-07-31T01:47:23.283 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:23 smithi045 conmon[33655]: 2021-07-31T01:47:22.008615+0000 2021-07-31T01:47:23.283 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:23 smithi045 conmon[33655]: mon.a (mon.0) 751 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:47:23.283 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:23 smithi045 conmon[33655]: audit 2021-07-31T01:47:23.283 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:23 smithi045 conmon[33655]: 2021-07-31T01:47:22 2021-07-31T01:47:23.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:23 smithi045 conmon[33655]: .009498+0000 mon.a (mon.0) 752 : audit 2021-07-31T01:47:23.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:23 smithi045 conmon[33655]: [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:47:23.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:23 smithi045 conmon[33655]: cluster 2021 2021-07-31T01:47:23.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:23 smithi045 conmon[33655]: -07-31T01:47:22.513124+0000 2021-07-31T01:47:23.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:23 smithi045 conmon[33655]: mon.a (mon.0) 753 : cluster [DBG] osdmap e143: 8 total, 8 up, 8 in 2021-07-31T01:47:23.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:23 smithi045 conmon[33655]: 2021-07-31T01:47:23.515 INFO:tasks.rados.rados.0.smithi045.stdout:533: done (4 left) 2021-07-31T01:47:23.516 INFO:tasks.rados.rados.0.smithi045.stdout:534: done (3 left) 2021-07-31T01:47:23.516 INFO:tasks.rados.rados.0.smithi045.stdout:535: done (2 left) 2021-07-31T01:47:23.516 INFO:tasks.rados.rados.0.smithi045.stdout:537: done (1 left) 2021-07-31T01:47:23.517 INFO:tasks.rados.rados.0.smithi045.stdout:539: done (0 left) 2021-07-31T01:47:23.517 INFO:tasks.rados.rados.0.smithi045.stdout:540: snap_remove snap 31 2021-07-31T01:47:23.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:47:23 smithi045 conmon[61210]: level=error ts=2021-07-31T01:47:23.506Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:47:23.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:47:23 smithi045 conmon[61210]: level=error ts=2021-07-31T01:47:23.506Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: dial tcp 172.21.15.154:8443: connect: connection refused" context_err="context deadline exceeded" 2021-07-31T01:47:23.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:47:23 smithi045 conmon[61210]: level=error ts=2021-07-31T01:47:23.506Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: dial tcp 172.21.15.154:8443: connect: connection refused" 2021-07-31T01:47:24.519 INFO:tasks.rados.rados.0.smithi045.stdout:540: done (0 left) 2021-07-31T01:47:24.519 INFO:tasks.rados.rados.0.smithi045.stdout:541: read oid 16 snap 41 2021-07-31T01:47:24.520 INFO:tasks.rados.rados.0.smithi045.stdout:541: expect deleted 2021-07-31T01:47:24.520 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 2 current snap is 47 2021-07-31T01:47:24.520 INFO:tasks.rados.rados.0.smithi045.stdout:542: seq_num 179 ranges {540672=606208} 2021-07-31T01:47:24.526 INFO:tasks.rados.rados.0.smithi045.stdout:542: writing smithi04569269-2 from 540672 to 1146880 tid 1 2021-07-31T01:47:24.527 INFO:tasks.rados.rados.0.smithi045.stdout:543: setattr oid 49 current snap is 47 2021-07-31T01:47:24.529 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 45 current snap is 47 2021-07-31T01:47:24.529 INFO:tasks.rados.rados.0.smithi045.stdout:544: seq_num 180 ranges {0=622592} 2021-07-31T01:47:24.534 INFO:tasks.rados.rados.0.smithi045.stdout:544: writing smithi04569269-45 from 0 to 622592 tid 1 2021-07-31T01:47:24.534 INFO:tasks.rados.rados.0.smithi045.stdout:545: read oid 35 snap -1 2021-07-31T01:47:24.534 INFO:tasks.rados.rados.0.smithi045.stdout:545: expect deleted 2021-07-31T01:47:24.534 INFO:tasks.rados.rados.0.smithi045.stdout:542: finishing write tid 1 to smithi04569269-2 2021-07-31T01:47:24.535 INFO:tasks.rados.rados.0.smithi045.stdout:542: finishing write tid 2 to smithi04569269-2 2021-07-31T01:47:24.535 INFO:tasks.rados.rados.0.smithi045.stdout:541: done (4 left) 2021-07-31T01:47:24.535 INFO:tasks.rados.rados.0.smithi045.stdout:546: read oid 5 snap -1 2021-07-31T01:47:24.536 INFO:tasks.rados.rados.0.smithi045.stdout:546: expect deleted 2021-07-31T01:47:24.536 INFO:tasks.rados.rados.0.smithi045.stdout:547: copy_from oid 1 from oid 7 current snap is 47 2021-07-31T01:47:24.536 INFO:tasks.rados.rados.0.smithi045.stdout:548: rollback oid 29 current snap is 47 2021-07-31T01:47:24.536 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 29 to 39 2021-07-31T01:47:24.537 INFO:tasks.rados.rados.0.smithi045.stdout:549: snap_remove snap 38 2021-07-31T01:47:24.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:24 smithi045 conmon[39718]: cluster 2021-07-31T01:47: 2021-07-31T01:47:24.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:24 smithi045 conmon[39718]: 23.382058+0000 mgr.y (mgr.14146) 325 : cluster [DBG] pgmap v365: 25 pgs: 25 active+clean; 95 MiB data, 337 MiB used, 715 GiB / 715 GiB avail; 2.9 MiB/s rd, 802 KiB/s wr, 6 op/s 2021-07-31T01:47:24.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:24 smithi045 conmon[39718]: cluster 2021-07-31T01:47:23.514823+0000 mon.a (mon.0) 754 : cluster [DBG] osdmap e144: 8 total, 8 up, 8 in 2021-07-31T01:47:24.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:24 smithi045 conmon[33655]: cluster 2021-07-31T01:47:24.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:24 smithi045 conmon[33655]: 2021-07-31T01:47:24.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:24 smithi045 conmon[33655]: 2021 2021-07-31T01:47:24.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:24 smithi045 conmon[33655]: - 2021-07-31T01:47:24.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:24 smithi045 conmon[33655]: 07 2021-07-31T01:47:24.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:24 smithi045 conmon[33655]: - 2021-07-31T01:47:24.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:24 smithi045 conmon[33655]: 31 2021-07-31T01:47:24.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:24 smithi045 conmon[33655]: T 2021-07-31T01:47:24.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:24 smithi045 conmon[33655]: 01 2021-07-31T01:47:24.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:24 smithi045 conmon[33655]: : 2021-07-31T01:47:24.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:24 smithi045 conmon[33655]: 47 2021-07-31T01:47:24.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:24 smithi045 conmon[33655]: : 2021-07-31T01:47:24.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:24 smithi045 conmon[33655]: 23 2021-07-31T01:47:24.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:24 smithi045 conmon[33655]: . 2021-07-31T01:47:24.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:24 smithi045 conmon[33655]: 382058 2021-07-31T01:47:24.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:24 smithi045 conmon[33655]: +0000 2021-07-31T01:47:24.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:24 smithi045 conmon[33655]: 2021-07-31T01:47:24.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:24 smithi045 conmon[33655]: mgr.y 2021-07-31T01:47:24.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:24 smithi045 conmon[33655]: ( 2021-07-31T01:47:24.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:24 smithi045 conmon[33655]: mgr 2021-07-31T01:47:24.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:24 smithi045 conmon[33655]: . 2021-07-31T01:47:24.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:24 smithi045 conmon[33655]: 14146 2021-07-31T01:47:24.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:24 smithi045 conmon[33655]: ) 2021-07-31T01:47:24.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:24 smithi045 conmon[33655]: 325 2021-07-31T01:47:24.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:24 smithi045 conmon[33655]: : 2021-07-31T01:47:24.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:24 smithi045 conmon[33655]: cluster 2021-07-31T01:47:24.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:24 smithi045 conmon[33655]: [DBG] pgmap v365: 25 pgs: 25 active+clean; 95 MiB data, 337 MiB used, 715 GiB / 715 GiB avail; 2.9 MiB/s rd, 802 KiB/s wr, 6 op/s 2021-07-31T01:47:24.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:24 smithi045 conmon[33655]: cluster 2021-07-31 2021-07-31T01:47:24.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:24 smithi045 conmon[33655]: T01:47:23.514823+0000 mon.a (mon.0) 754 : cluster [DBG] 2021-07-31T01:47:24.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:24 smithi045 conmon[33655]: osdmap e144: 8 total, 8 up, 8 in 2021-07-31T01:47:24.955 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:24 smithi154 conmon[34553]: cluster 2021-07-31T01:47:23.382058+0000 mgr.y (mgr.14146 2021-07-31T01:47:24.955 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:24 smithi154 conmon[34553]: ) 325 : cluster [DBG] pgmap v365: 25 pgs: 25 active+clean; 95 MiB data, 337 MiB used, 715 GiB / 715 GiB avail; 2.9 MiB/s rd, 802 KiB/s wr, 6 op/s 2021-07-31T01:47:24.956 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:24 smithi154 conmon[34553]: cluster 2021-07-31T01:47:23.514823+0000 mon.a (mon.0) 754 : cluster [DBG] osdmap e144: 8 total, 8 up, 8 in 2021-07-31T01:47:25.128 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T01:47:25.398 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T01:47:25.399 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 running (2m) 2m ago 3m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T01:47:25.399 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 running (2m) 2m ago 2m 37.1M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T01:47:25.399 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 starting - - - - 2021-07-31T01:47:25.399 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 running (8m) 2m ago 8m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T01:47:25.400 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 running (8m) 2m ago 8m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T01:47:25.400 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 running (6m) 2m ago 6m 92.9M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T01:47:25.400 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 running (6m) 2m ago 6m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T01:47:25.400 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 running (3m) 2m ago 3m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T01:47:25.400 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 running (3m) 2m ago 3m 8144k - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T01:47:25.401 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 running (6m) 2m ago 6m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T01:47:25.401 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 running (5m) 2m ago 5m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T01:47:25.401 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 running (5m) 2m ago 5m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T01:47:25.402 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 running (5m) 2m ago 5m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T01:47:25.402 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 running (4m) 2m ago 4m 43.3M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T01:47:25.402 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 running (4m) 2m ago 4m 41.6M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T01:47:25.403 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 running (4m) 2m ago 4m 41.2M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T01:47:25.403 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 running (3m) 2m ago 3m 42.6M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T01:47:25.403 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 running (2m) 2m ago 3m 28.9M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T01:47:25.533 INFO:tasks.rados.rados.0.smithi045.stdout:549: done (7 left) 2021-07-31T01:47:25.533 INFO:tasks.rados.rados.0.smithi045.stdout:550: read oid 17 snap -1 2021-07-31T01:47:25.533 INFO:tasks.rados.rados.0.smithi045.stdout:550: expect (ObjNum 171 snap 41 seq_num 171) 2021-07-31T01:47:25.534 INFO:tasks.rados.rados.0.smithi045.stdout:551: rollback oid 6 current snap is 47 2021-07-31T01:47:25.534 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 6 to 45 2021-07-31T01:47:25.534 INFO:tasks.rados.rados.0.smithi045.stdout:552: delete oid 46 current snap is 47 2021-07-31T01:47:25.534 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 49 v 361 (ObjNum 1684951897 snap 1901158758 seq_num 1364285036) dirty exists 2021-07-31T01:47:25.534 INFO:tasks.rados.rados.0.smithi045.stdout:542: finishing write tid 3 to smithi04569269-2 2021-07-31T01:47:25.535 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 2 v 59 (ObjNum 179 snap 47 seq_num 179) dirty exists 2021-07-31T01:47:25.535 INFO:tasks.rados.rados.0.smithi045.stdout:542: left oid 2 (ObjNum 179 snap 47 seq_num 179) 2021-07-31T01:47:25.535 INFO:tasks.rados.rados.0.smithi045.stdout:548: finishing rollback tid 1 to smithi04569269-29 2021-07-31T01:47:25.535 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 29 v 26 (ObjNum 1 snap 32731 seq_num 3189088976) dirty dne 2021-07-31T01:47:25.536 INFO:tasks.rados.rados.0.smithi045.stdout:547: finishing copy_from to smithi04569269-1 2021-07-31T01:47:25.536 INFO:tasks.rados.rados.0.smithi045.stdout:547: got expected ENOENT (src dne) 2021-07-31T01:47:25.536 INFO:tasks.rados.rados.0.smithi045.stdout:547: finishing copy_from racing read to smithi04569269-1 2021-07-31T01:47:25.536 INFO:tasks.rados.rados.0.smithi045.stdout:544: finishing write tid 1 to smithi04569269-45 2021-07-31T01:47:25.536 INFO:tasks.rados.rados.0.smithi045.stdout:544: finishing write tid 2 to smithi04569269-45 2021-07-31T01:47:25.537 INFO:tasks.rados.rados.0.smithi045.stdout:544: finishing write tid 3 to smithi04569269-45 2021-07-31T01:47:25.537 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 45 v 231 (ObjNum 180 snap 47 seq_num 180) dirty exists 2021-07-31T01:47:25.537 INFO:tasks.rados.rados.0.smithi045.stdout:544: left oid 45 (ObjNum 180 snap 47 seq_num 180) 2021-07-31T01:47:25.538 INFO:tasks.rados.rados.0.smithi045.stdout:542: done (9 left) 2021-07-31T01:47:25.538 INFO:tasks.rados.rados.0.smithi045.stdout:543: done (8 left) 2021-07-31T01:47:25.538 INFO:tasks.rados.rados.0.smithi045.stdout:544: done (7 left) 2021-07-31T01:47:25.538 INFO:tasks.rados.rados.0.smithi045.stdout:545: done (6 left) 2021-07-31T01:47:25.539 INFO:tasks.rados.rados.0.smithi045.stdout:546: done (5 left) 2021-07-31T01:47:25.539 INFO:tasks.rados.rados.0.smithi045.stdout:547: done (4 left) 2021-07-31T01:47:25.539 INFO:tasks.rados.rados.0.smithi045.stdout:548: done (3 left) 2021-07-31T01:47:25.539 INFO:tasks.rados.rados.0.smithi045.stdout:552: done (2 left) 2021-07-31T01:47:25.540 INFO:tasks.rados.rados.0.smithi045.stdout:553: rmattr oid 9 current snap is 47 2021-07-31T01:47:25.540 INFO:tasks.rados.rados.0.smithi045.stdout:553: done (2 left) 2021-07-31T01:47:25.540 INFO:tasks.rados.rados.0.smithi045.stdout:554: snap_create 2021-07-31T01:47:25.543 INFO:tasks.rados.rados.0.smithi045.stdout:550: expect (ObjNum 171 snap 41 seq_num 171) 2021-07-31T01:47:25.555 INFO:tasks.rados.rados.0.smithi045.stdout:551: finishing rollback tid 0 to smithi04569269-6 2021-07-31T01:47:25.555 INFO:tasks.rados.rados.0.smithi045.stdout:551: finishing rollback tid 1 to smithi04569269-6 2021-07-31T01:47:25.555 INFO:tasks.rados.rados.0.smithi045.stdout:551: finishing rollback tid 2 to smithi04569269-6 2021-07-31T01:47:25.555 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 6 v 168 (ObjNum -1105778384 snap 21873 seq_num 3190202672) dirty exists 2021-07-31T01:47:25.737 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T01:47:25.738 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T01:47:25.738 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T01:47:25.738 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T01:47:25.738 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T01:47:25.739 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 2 2021-07-31T01:47:25.739 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T01:47:25.739 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T01:47:25.739 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T01:47:25.739 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T01:47:25.740 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T01:47:25.740 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T01:47:25.740 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 13 2021-07-31T01:47:25.740 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T01:47:25.740 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T01:47:26.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:25 smithi154 conmon[34553]: cluster 2021-07-31T01:47:24.516622+0000 mon.a (mon.0) 755 : cluster [DBG] osdmap e145: 8 total, 8 up, 8 in 2021-07-31T01:47:26.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:25 smithi154 conmon[34553]: 2021-07-31T01:47:26.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:25 smithi045 conmon[33655]: cluster 2021-07-31T01:47:24.516622+0000 mon.a (mon.0) 755 : cluster [DBG] osdmap e145: 8 total, 8 up, 8 in 2021-07-31T01:47:26.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:25 smithi045 conmon[39718]: cluster 2021-07-31T01:47:24.516622+0000 mon.a (mon.0) 755 : cluster [DBG] osdmap e145: 8 total, 8 up, 8 in 2021-07-31T01:47:26.489 INFO:tasks.rados.rados.0.smithi045.stdout:550: done (2 left) 2021-07-31T01:47:26.489 INFO:tasks.rados.rados.0.smithi045.stdout:551: done (1 left) 2021-07-31T01:47:26.489 INFO:tasks.rados.rados.0.smithi045.stdout:554: done (0 left) 2021-07-31T01:47:26.489 INFO:tasks.rados.rados.0.smithi045.stdout:555: delete oid 37 current snap is 48 2021-07-31T01:47:26.495 INFO:tasks.rados.rados.0.smithi045.stdout:555: done (0 left) 2021-07-31T01:47:26.495 INFO:tasks.rados.rados.0.smithi045.stdout:556: read oid 28 snap 47 2021-07-31T01:47:26.495 INFO:tasks.rados.rados.0.smithi045.stdout:556: expect (ObjNum 165 snap 38 seq_num 165) 2021-07-31T01:47:26.496 INFO:tasks.rados.rados.0.smithi045.stdout:557: read oid 6 snap -1 2021-07-31T01:47:26.496 INFO:tasks.rados.rados.0.smithi045.stdout:557: expect (ObjNum 0 snap 0 seq_num 0) 2021-07-31T01:47:26.496 INFO:tasks.rados.rados.0.smithi045.stdout:558: rollback oid 2 current snap is 48 2021-07-31T01:47:26.496 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 2 to 39 2021-07-31T01:47:26.497 INFO:tasks.rados.rados.0.smithi045.stdout:559: snap_remove snap 32 2021-07-31T01:47:26.799 INFO:tasks.rados.rados.0.smithi045.stdout:559: done (3 left) 2021-07-31T01:47:26.800 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 31 current snap is 48 2021-07-31T01:47:26.800 INFO:tasks.rados.rados.0.smithi045.stdout:560: seq_num 183 ranges {2932736=671744} 2021-07-31T01:47:26.816 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:26 smithi045 conmon[33655]: audit 2021-07-31T01:47:25.116421+0000 mgr.y ( 2021-07-31T01:47:26.816 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:26 smithi045 conmon[33655]: mgr.14146) 326 : audit [DBG] from='client.14679 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:47:26.817 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:26 smithi045 conmon[33655]: cluster 2021-07-31T01:47:25.382356+0000 mgr.y (mgr.14146) 327 : cluster [DBG] 2021-07-31T01:47:26.817 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:26 smithi045 conmon[33655]: pgmap v368: 25 pgs: 25 active+clean; 96 MiB data, 341 MiB used, 715 GiB / 715 GiB avail; 3.5 MiB/s rd, 934 KiB/s wr, 8 op/s 2021-07-31T01:47:26.817 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:26 smithi045 conmon[33655]: audit 2021-07-31T 2021-07-31T01:47:26.817 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:26 smithi045 conmon[33655]: 01:47:25.391066+0000 mgr.y (mgr 2021-07-31T01:47:26.817 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:26 smithi045 conmon[33655]: .14146) 328 : audit [DBG] from='client.24592 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:47:26.818 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:26 smithi045 conmon[33655]: cluster 2021-07-31T01:47:26.818 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:26 smithi045 conmon[33655]: 2021-07-31T01:47:25. 2021-07-31T01:47:26.818 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:26 smithi045 conmon[33655]: 531942+0000 mon.a (mon.0) 756 : 2021-07-31T01:47:26.818 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:26 smithi045 conmon[33655]: cluster [DBG] osdmap e146: 8 total, 8 up, 8 in 2021-07-31T01:47:26.819 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:26 smithi045 conmon[33655]: audit 2021-07-31 2021-07-31T01:47:26.819 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:26 smithi045 conmon[33655]: T01:47:25.737203+0000 mon.c ( 2021-07-31T01:47:26.819 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:26 smithi045 conmon[33655]: mon.1) 21 : audit [DBG] from='client.? 172.21.15.45:0/3477264497' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T01:47:26.819 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:26 smithi045 conmon[33655]: audit 2021- 2021-07-31T01:47:26.820 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:26 smithi045 conmon[33655]: 07-31T01:47:26.404482+0000 mon.a (mon.0 2021-07-31T01:47:26.820 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:26 smithi045 conmon[33655]: ) 757 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:47:26.820 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:26 smithi045 conmon[33655]: cluster 2021-07 2021-07-31T01:47:26.820 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:26 smithi045 conmon[33655]: -31T01:47:26.488293+0000 2021-07-31T01:47:26.821 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:26 smithi045 conmon[33655]: mon.a (mon.0) 758 : cluster [DBG] 2021-07-31T01:47:26.821 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:26 smithi045 conmon[33655]: osdmap e147: 8 total, 8 up, 8 in 2021-07-31T01:47:26.821 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:26 smithi045 conmon[39718]: audit 2021-07-31T01:47:25.116421+0000 mgr.y (mgr.14146) 326 : audit [DBG] from='client.14679 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:47:26.821 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:26 smithi045 conmon[39718]: cluster 2021-07-31T01:47:25.382356+0000 mgr.y (mgr.14146) 327 : cluster [DBG] pgmap v368: 25 pgs: 25 active+clean; 96 MiB data, 341 MiB used, 715 GiB / 715 GiB avail; 3.5 MiB/s rd, 934 KiB/s wr, 8 op/s 2021-07-31T01:47:26.822 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:26 smithi045 conmon[39718]: audit 2021-07-31T01:47:26.822 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:26 smithi045 conmon[39718]: 2021-07-31T01:47:25.391066+0000 mgr.y (mgr.14146) 328 : audit [DBG] from='client.24592 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:47:26.822 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:26 smithi045 conmon[39718]: cluster 2021 2021-07-31T01:47:26.822 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:26 smithi045 conmon[39718]: -07-31T01:47:25.531942+0000 mon.a (mon.0) 756 : cluster [DBG] osdmap e146: 8 total, 8 up, 8 in 2021-07-31T01:47:26.823 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:26 smithi045 conmon[39718]: 2021-07-31T01:47:26.823 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:26 smithi045 conmon[39718]: audit 2021-07-31T01:47:26.823 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:26 smithi045 conmon[39718]: 2021-07-31T01:47:25.737203+0000 2021-07-31T01:47:26.823 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:26 smithi045 conmon[39718]: mon.c (mon.1) 21 : audit [DBG] from='client.? 172.21.15.45:0/3477264497' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T01:47:26.824 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:26 smithi045 conmon[39718]: audit 2021- 2021-07-31T01:47:26.824 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:26 smithi045 conmon[39718]: 07-31T01:47:26.404482+0000 mon.a 2021-07-31T01:47:26.824 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:26 smithi045 conmon[39718]: (mon.0) 757 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:47:26.824 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:26 smithi045 conmon[39718]: cluster 2021- 2021-07-31T01:47:26.824 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:26 smithi045 conmon[39718]: 07-31T01:47:26.488293+0000 mon.a ( 2021-07-31T01:47:26.825 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:26 smithi045 conmon[39718]: mon.0) 758 : cluster [DBG] osdmap e147: 8 total, 8 up, 8 in 2021-07-31T01:47:26.826 INFO:tasks.rados.rados.0.smithi045.stdout:560: writing smithi04569269-31 from 2932736 to 3604480 tid 1 2021-07-31T01:47:26.826 INFO:tasks.rados.rados.0.smithi045.stdout:561: copy_from oid 29 from oid 47 current snap is 48 2021-07-31T01:47:26.826 INFO:tasks.rados.rados.0.smithi045.stdout:557: expect (ObjNum 0 snap 0 seq_num 0) 2021-07-31T01:47:26.826 INFO:tasks.rados.rados.0.smithi045.stdout:558: finishing rollback tid 0 to smithi04569269-2 2021-07-31T01:47:26.827 INFO:tasks.rados.rados.0.smithi045.stdout:558: finishing rollback tid 1 to smithi04569269-2 2021-07-31T01:47:26.827 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 2 v 62 (ObjNum 0 snap 0 seq_num 0) dirty dne 2021-07-31T01:47:26.827 INFO:tasks.rados.rados.0.smithi045.stdout:557: done (4 left) 2021-07-31T01:47:26.827 INFO:tasks.rados.rados.0.smithi045.stdout:558: done (3 left) 2021-07-31T01:47:26.828 INFO:tasks.rados.rados.0.smithi045.stdout:562: rollback oid 1 current snap is 48 2021-07-31T01:47:26.828 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 1 to 42 2021-07-31T01:47:26.828 INFO:tasks.rados.rados.0.smithi045.stdout:556: expect (ObjNum 165 snap 38 seq_num 165) 2021-07-31T01:47:26.967 INFO:tasks.rados.rados.0.smithi045.stdout:560: finishing write tid 1 to smithi04569269-31 2021-07-31T01:47:26.967 INFO:tasks.rados.rados.0.smithi045.stdout:560: finishing write tid 2 to smithi04569269-31 2021-07-31T01:47:26.968 INFO:tasks.rados.rados.0.smithi045.stdout:562: finishing rollback tid 0 to smithi04569269-1 2021-07-31T01:47:26.968 INFO:tasks.rados.rados.0.smithi045.stdout:562: finishing rollback tid 1 to smithi04569269-1 2021-07-31T01:47:26.968 INFO:tasks.rados.rados.0.smithi045.stdout:560: finishing write tid 3 to smithi04569269-31 2021-07-31T01:47:26.968 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 31 v 384 (ObjNum 183 snap 48 seq_num 183) dirty exists 2021-07-31T01:47:26.968 INFO:tasks.rados.rados.0.smithi045.stdout:560: left oid 31 (ObjNum 183 snap 48 seq_num 183) 2021-07-31T01:47:26.969 INFO:tasks.rados.rados.0.smithi045.stdout:561: finishing copy_from to smithi04569269-29 2021-07-31T01:47:26.969 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 29 v 22 (ObjNum 46 snap 0 seq_num 46) dirty exists 2021-07-31T01:47:26.969 INFO:tasks.rados.rados.0.smithi045.stdout:561: finishing copy_from racing read to smithi04569269-29 2021-07-31T01:47:26.969 INFO:tasks.rados.rados.0.smithi045.stdout:562: finishing rollback tid 2 to smithi04569269-1 2021-07-31T01:47:26.970 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 1 v 301 (ObjNum 155 snap 37 seq_num 155) dirty exists 2021-07-31T01:47:26.970 INFO:tasks.rados.rados.0.smithi045.stdout:556: done (3 left) 2021-07-31T01:47:26.970 INFO:tasks.rados.rados.0.smithi045.stdout:560: done (2 left) 2021-07-31T01:47:26.970 INFO:tasks.rados.rados.0.smithi045.stdout:561: done (1 left) 2021-07-31T01:47:26.971 INFO:tasks.rados.rados.0.smithi045.stdout:562: done (0 left) 2021-07-31T01:47:26.971 INFO:tasks.rados.rados.0.smithi045.stdout:563: snap_remove snap 42 2021-07-31T01:47:27.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:26 smithi154 conmon[34553]: audit 2021-07-31T01:47:25.116421+0000 mgr.y 2021-07-31T01:47:27.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:26 smithi154 conmon[34553]: (mgr.14146) 326 : audit [DBG] from='client.14679 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:47:27.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:26 smithi154 conmon[34553]: cluster 2021-07-31T01:47:25.382356+0000 mgr.y (mgr.14146) 327 : cluster [DBG] pgmap v368: 25 pgs: 25 active+clean; 96 MiB data, 341 MiB used, 715 GiB / 715 GiB avail; 3.5 MiB/s rd, 934 KiB/s wr, 8 op/s 2021-07-31T01:47:27.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:26 smithi154 conmon[34553]: 2021-07-31T01:47:27.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:26 smithi154 conmon[34553]: audit 2021-07-31T01 2021-07-31T01:47:27.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:26 smithi154 conmon[34553]: :47:25.391066+0000 mgr.y (mgr.14146) 328 : audit [DBG] 2021-07-31T01:47:27.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:26 smithi154 conmon[34553]: from='client.24592 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:47:27.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:26 smithi154 conmon[34553]: cluster 2021-07-31T01:47 2021-07-31T01:47:27.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:26 smithi154 conmon[34553]: :25.531942+0000 mon.a (mon.0) 756 2021-07-31T01:47:27.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:26 smithi154 conmon[34553]: : cluster [DBG] osdmap e146: 8 total, 8 up, 8 in 2021-07-31T01:47:27.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:26 smithi154 conmon[34553]: audit 2021-07 2021-07-31T01:47:27.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:26 smithi154 conmon[34553]: -31T01:47:25.737203+0000 mon.c ( 2021-07-31T01:47:27.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:26 smithi154 conmon[34553]: mon.1) 21 : audit [DBG] from='client.? 172.21.15.45:0/3477264497' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T01:47:27.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:26 smithi154 conmon[34553]: audit 2021-07-31T01 2021-07-31T01:47:27.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:26 smithi154 conmon[34553]: :47:26.404482+0000 mon.a (mon.0) 757 : 2021-07-31T01:47:27.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:26 smithi154 conmon[34553]: audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:47:27.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:26 smithi154 conmon[34553]: cluster 2021-07 2021-07-31T01:47:27.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:26 smithi154 conmon[34553]: -31T01:47:26.488293+0000 mon.a ( 2021-07-31T01:47:27.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:26 smithi154 conmon[34553]: mon.0) 758 : cluster [DBG] osdmap e147: 8 total, 8 up, 8 in 2021-07-31T01:47:27.276 INFO:journalctl@ceph.osd.3.smithi045.stdout:Jul 31 01:47:26 smithi045 conmon[54013]: debug 2021-07-31T01:47:26.814+0000 7f64fc8cd700 -1 osd.3 pg_epoch: 148 pg[3.bs0( v 148'375 (0'0,148'375] local-lis/les=56/57 n=14 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [3,0,4]p3(0) r=0 lpr=56 crt=148'375 lcod 148'373 mlcod 148'373 active+clean+snaptrim trimq=[3e~1] ps=[3d~1,49~1,52~1]] removing snap head 2021-07-31T01:47:27.510 INFO:tasks.rados.rados.0.smithi045.stdout:563: done (0 left) 2021-07-31T01:47:27.510 INFO:tasks.rados.rados.0.smithi045.stdout:564: snap_remove snap 46 2021-07-31T01:47:28.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:27 smithi154 conmon[34553]: audit 2021-07-31T01:47:26. 2021-07-31T01:47:28.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:27 smithi154 conmon[34553]: 784518+0000 mon.a (mon.0) 759 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:47:28.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:27 smithi154 conmon[34553]: audit 2021-07-31T01 2021-07-31T01:47:28.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:27 smithi154 conmon[34553]: :47:26.789277+0000 mon.a (mon.0) 760 : audit [DBG] 2021-07-31T01:47:28.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:27 smithi154 conmon[34553]: from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:47:28.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:27 smithi154 conmon[34553]: cephadm 2021-07-31 2021-07-31T01:47:28.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:27 smithi154 conmon[34553]: T01:47:26.790340+0000 mgr.y (mgr 2021-07-31T01:47:28.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:27 smithi154 conmon[34553]: .14146) 329 : cephadm [INF] 2021-07-31T01:47:28.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:27 smithi154 conmon[34553]: Upgrade: Need to upgrade myself (mgr.y) 2021-07-31T01:47:28.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:27 smithi154 conmon[34553]: cephadm 2021-07 2021-07-31T01:47:28.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:27 smithi154 conmon[34553]: -31T01:47:26.793057+0000 mgr.y 2021-07-31T01:47:28.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:27 smithi154 conmon[34553]: (mgr.14146) 330 : cephadm 2021-07-31T01:47:28.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:27 smithi154 conmon[34553]: [INF] Failing over to other MGR 2021-07-31T01:47:28.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:27 smithi154 conmon[34553]: audit 2021- 2021-07-31T01:47:28.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:27 smithi154 conmon[34553]: 07-31T01:47:26.793343+0000 2021-07-31T01:47:28.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:27 smithi154 conmon[34553]: mon.a (mon.0) 761 : audit 2021-07-31T01:47:28.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:27 smithi154 conmon[34553]: [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2021-07-31T01:47:28.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:27 smithi154 conmon[34553]: cluster 2021- 2021-07-31T01:47:28.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:27 smithi154 conmon[34553]: 07-31T01:47:26.797649+0000 2021-07-31T01:47:28.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:27 smithi154 conmon[34553]: mon.a (mon.0) 762 : 2021-07-31T01:47:28.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:27 smithi154 conmon[34553]: cluster [DBG] osdmap e148: 8 total, 8 up, 8 in 2021-07-31T01:47:28.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:27 smithi154 conmon[34553]: cluster 2021- 2021-07-31T01:47:28.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:27 smithi154 conmon[34553]: 07-31T01:47:27.503849+0000 2021-07-31T01:47:28.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:27 smithi154 conmon[34553]: mon.a (mon.0) 763 : 2021-07-31T01:47:28.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:27 smithi154 conmon[34553]: cluster [INF] Active manager daemon x restarted 2021-07-31T01:47:28.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:27 smithi154 conmon[34553]: cluster 2021-07 2021-07-31T01:47:28.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:27 smithi154 conmon[34553]: -31T01:47:27.504694+0000 mon.a (mon 2021-07-31T01:47:28.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:27 smithi154 conmon[34553]: .0) 764 : cluster [INF] 2021-07-31T01:47:28.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:27 smithi154 conmon[34553]: Activating manager daemon x 2021-07-31T01:47:28.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:27 smithi154 conmon[34553]: audit 2021-07- 2021-07-31T01:47:28.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:27 smithi154 conmon[34553]: 31T01:47:27.504949+0000 mon.b ( 2021-07-31T01:47:28.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:27 smithi154 conmon[34553]: mon.2) 25 : audit [DBG] 2021-07-31T01:47:28.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:27 smithi154 conmon[34553]: from='mgr.? 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2021-07-31T01:47:28.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:27 smithi154 conmon[34553]: cluster 2021-07 2021-07-31T01:47:28.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:27 smithi154 conmon[34553]: -31T01:47:27.508848+0000 mon.a (mon 2021-07-31T01:47:28.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:27 smithi154 conmon[34553]: .0) 765 : cluster [DBG] 2021-07-31T01:47:28.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:27 smithi154 conmon[34553]: osdmap e149: 8 total, 8 up, 8 in 2021-07-31T01:47:28.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:27 smithi154 conmon[34553]: audit 2021-07- 2021-07-31T01:47:28.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:27 smithi154 conmon[34553]: 31T01:47:27.509722+0000 mon.b ( 2021-07-31T01:47:28.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:27 smithi154 conmon[34553]: mon.2) 26 : audit [DBG] 2021-07-31T01:47:28.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:27 smithi154 conmon[34553]: from='mgr.? 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2021-07-31T01:47:28.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:27 smithi154 conmon[34553]: audit 2021- 2021-07-31T01:47:28.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:27 smithi154 conmon[34553]: 07-31T01:47:27.511582+0000 mon.b 2021-07-31T01:47:28.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:27 smithi154 conmon[34553]: (mon.2) 27 : audit 2021-07-31T01:47:28.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:27 smithi154 conmon[34553]: [DBG] from='mgr.? 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2021-07-31T01:47:28.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:27 smithi154 conmon[34553]: audit 2021-07 2021-07-31T01:47:28.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:27 smithi154 conmon[34553]: -31T01:47:27.512322+0000 mon.b 2021-07-31T01:47:28.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:27 smithi154 conmon[34553]: (mon.2) 28 : audit [DBG] from='mgr.? 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2021-07-31T01:47:28.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:27 smithi154 conmon[34553]: 2021-07-31T01:47:28.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:27 smithi154 conmon[34553]: cluster 2021-07-31T01:47:28.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:27 smithi154 conmon[34553]: 2021-07-31T01:47: 2021-07-31T01:47:28.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:27 smithi154 conmon[34553]: 27.645288+0000 mon.a (mon. 2021-07-31T01:47:28.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:27 smithi154 conmon[34553]: 0) 766 : cluster [DBG] Standby manager daemon y started 2021-07-31T01:47:28.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:27 smithi154 conmon[34553]: 2021-07-31T01:47:28.276 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Jul 31 01:47:27 smithi045 conmon[33979]: debug 2021-07-31T01:47:28.277 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Jul 31 01:47:27 smithi045 conmon[33979]: 2021-07-31T01:47:27.786+0000 7f3578ae4700 -1 mgr handle_mgr_map I was active but no longer am 2021-07-31T01:47:28.277 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Jul 31 01:47:27 smithi045 conmon[33979]: 2021-07-31T01:47:28.277 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Jul 31 01:47:27 smithi045 conmon[33979]: ignoring --setuser ceph since I am not root 2021-07-31T01:47:28.277 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Jul 31 01:47:27 smithi045 conmon[33979]: ignoring --setgroup ceph since I am not root 2021-07-31T01:47:28.278 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Jul 31 01:47:27 smithi045 conmon[33979]: debug 2021-07-31T01:47:27.828+0000 7f179db70700 1 -- 172.21.15.45:0/2693968453 <== mon.1 v2:172.21.15.45:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55b92db524e0 con 0x55b92db2b000 2021-07-31T01:47:28.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:27 smithi045 conmon[39718]: audit 2021-07-31T01:47:26.784518+0000 mon.a (mon 2021-07-31T01:47:28.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:27 smithi045 conmon[39718]: .0) 759 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:47:28.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:27 smithi045 conmon[39718]: audit 2021-07-31T01:47: 2021-07-31T01:47:28.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:27 smithi045 conmon[39718]: 26.789277+0000 mon.a (mon.0) 760 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:47:28.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:27 smithi045 conmon[39718]: cephadm 2021- 2021-07-31T01:47:28.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:27 smithi045 conmon[39718]: 07-31T01:47:26.790340+0000 mgr.y (mgr 2021-07-31T01:47:28.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:27 smithi045 conmon[39718]: .14146) 329 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.y) 2021-07-31T01:47:28.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:27 smithi045 conmon[39718]: cephadm 2021-07-31T01: 2021-07-31T01:47:28.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:27 smithi045 conmon[39718]: 47:26.793057+0000 mgr.y (mgr.14146) 330 : cephadm [INF] Failing over to other MGR 2021-07-31T01:47:28.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:27 smithi045 conmon[39718]: 2021-07-31T01:47:28.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:27 smithi045 conmon[39718]: audit 2021-07-31T01 2021-07-31T01:47:28.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:27 smithi045 conmon[39718]: :47:26.793343+0000 mon.a (mon.0) 2021-07-31T01:47:28.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:27 smithi045 conmon[39718]: 761 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2021-07-31T01:47:28.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:27 smithi045 conmon[39718]: cluster 2021 2021-07-31T01:47:28.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:27 smithi045 conmon[39718]: -07-31T01:47:26.797649+0000 mon.a 2021-07-31T01:47:28.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:27 smithi045 conmon[39718]: (mon.0) 762 : cluster [DBG] osdmap e148: 8 total, 8 up, 8 in 2021-07-31T01:47:28.282 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:27 smithi045 conmon[39718]: cluster 2021-07 2021-07-31T01:47:28.282 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:27 smithi045 conmon[39718]: -31T01:47:27.503849+0000 mon.a (mon.0) 763 : 2021-07-31T01:47:28.282 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:27 smithi045 conmon[39718]: cluster [INF] Active manager daemon x restarted 2021-07-31T01:47:28.282 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:27 smithi045 conmon[39718]: cluster 2021- 2021-07-31T01:47:28.282 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:27 smithi045 conmon[39718]: 07-31T01:47:27.504694+0000 mon.a ( 2021-07-31T01:47:28.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:27 smithi045 conmon[39718]: mon.0) 764 : cluster [INF] Activating manager daemon x 2021-07-31T01:47:28.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:27 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T01:47:28.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:27 smithi045 conmon[39718]: 31T01:47:27.504949+0000 mon.b (mon.2) 25 : audit 2021-07-31T01:47:28.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:27 smithi045 conmon[39718]: [DBG] from='mgr.? 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2021-07-31T01:47:28.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:27 smithi045 conmon[39718]: cluster 2021-07 2021-07-31T01:47:28.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:27 smithi045 conmon[39718]: -31T01:47:27.508848+0000 mon.a (mon 2021-07-31T01:47:28.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:27 smithi045 conmon[39718]: .0) 765 : cluster [DBG] osdmap e149: 8 total, 8 up, 8 in 2021-07-31T01:47:28.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:27 smithi045 conmon[39718]: audit 2021- 2021-07-31T01:47:28.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:27 smithi045 conmon[39718]: 07-31T01:47:27.509722+0000 mon.b (mon.2) 2021-07-31T01:47:28.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:27 smithi045 conmon[39718]: 26 : audit [DBG] from='mgr.? 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2021-07-31T01:47:28.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:27 smithi045 conmon[39718]: audit 2021-07-31T01:47:28.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:27 smithi045 conmon[39718]: 2021-07-31T01:47:27.511582 2021-07-31T01:47:28.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:27 smithi045 conmon[39718]: +0000 mon.b (mon.2) 27 : audit [DBG] from='mgr.? 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2021-07-31T01:47:28.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:27 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T01:47:28.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:27 smithi045 conmon[39718]: T01:47:27.512322+0000 mon.b (mon. 2021-07-31T01:47:28.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:27 smithi045 conmon[39718]: 2) 28 : audit [DBG] from='mgr.? 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2021-07-31T01:47:28.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:27 smithi045 conmon[39718]: cluster 2021-07-31T 2021-07-31T01:47:28.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:27 smithi045 conmon[39718]: 01:47:27.645288+0000 mon.a (mon.0) 766 : cluster [DBG] Standby manager daemon y started 2021-07-31T01:47:28.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:27 smithi045 conmon[33655]: audit 2021-07-31T01:47:26.784518+0000 mon.a (mon.0) 759 : audit 2021-07-31T01:47:28.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:27 smithi045 conmon[33655]: [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' 2021-07-31T01:47:28.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:27 smithi045 conmon[33655]: audit 2021-07-31T01:47:26.789277+0000 mon.a (mon.0) 760 : audit [DBG] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:47:28.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:27 smithi045 conmon[33655]: cephadm 2021-07-31T01:47:26.790340+0000 mgr.y (mgr.14146) 329 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.y) 2021-07-31T01:47:28.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:27 smithi045 conmon[33655]: cephadm 2021-07-31T01:47:28.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:27 smithi045 conmon[33655]: 2021-07-31T01 2021-07-31T01:47:28.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:27 smithi045 conmon[33655]: :47:26.793057+0000 mgr.y (mgr.14146) 330 2021-07-31T01:47:28.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:27 smithi045 conmon[33655]: : cephadm [INF] Failing over to other MGR 2021-07-31T01:47:28.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:27 smithi045 conmon[33655]: audit 2021-07-31T01:47:28.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:27 smithi045 conmon[33655]: 2021-07-31T01:47: 2021-07-31T01:47:28.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:27 smithi045 conmon[33655]: 26.793343+0000 mon.a (mon.0) 761 : audit 2021-07-31T01:47:28.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:27 smithi045 conmon[33655]: [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2021-07-31T01:47:28.290 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:27 smithi045 conmon[33655]: cluster 2021-07-31T01:47:28.290 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:27 smithi045 conmon[33655]: 2021-07-31T01:47: 2021-07-31T01:47:28.290 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:27 smithi045 conmon[33655]: 26.797649+0000 mon.a (mon.0) 762 : 2021-07-31T01:47:28.290 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:27 smithi045 conmon[33655]: cluster [DBG] osdmap e148: 8 total, 8 up, 8 in 2021-07-31T01:47:28.291 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:27 smithi045 conmon[33655]: cluster 2021-07-31T01:47:28.291 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:27 smithi045 conmon[33655]: 2021-07-31T01 2021-07-31T01:47:28.291 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:27 smithi045 conmon[33655]: :47:27.503849+0000 mon.a (mon.0) 2021-07-31T01:47:28.291 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:27 smithi045 conmon[33655]: 763 : cluster [INF] Active manager daemon x restarted 2021-07-31T01:47:28.291 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:27 smithi045 conmon[33655]: cluster 2021-07-31T01:47:28.292 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:27 smithi045 conmon[33655]: 2021-07-31T01: 2021-07-31T01:47:28.292 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:27 smithi045 conmon[33655]: 47:27.504694+0000 mon.a (mon.0) 2021-07-31T01:47:28.292 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:27 smithi045 conmon[33655]: 764 : cluster [INF] Activating manager daemon x 2021-07-31T01:47:28.292 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:27 smithi045 conmon[33655]: audit 2021-07-31T01:47:28.293 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:27 smithi045 conmon[33655]: 2021-07-31T01: 2021-07-31T01:47:28.293 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:27 smithi045 conmon[33655]: 47:27.504949+0000 mon.b (mon.2) 2021-07-31T01:47:28.293 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:27 smithi045 conmon[33655]: 25 : audit [DBG] from='mgr.? 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2021-07-31T01:47:28.293 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:27 smithi045 conmon[33655]: cluster 2021 2021-07-31T01:47:28.293 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:27 smithi045 conmon[33655]: -07-31T01:47:27.508848+0000 mon.a 2021-07-31T01:47:28.294 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:27 smithi045 conmon[33655]: (mon.0) 765 : cluster [DBG] osdmap e149: 8 total, 8 up, 8 in 2021-07-31T01:47:28.294 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:27 smithi045 conmon[33655]: audit 2021-07-31T01:47:28.294 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:27 smithi045 conmon[33655]: 2021-07 2021-07-31T01:47:28.295 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:27 smithi045 conmon[33655]: -31T01:47:27.509722+0000 mon.b (mon.2) 26 : audit [DBG] from='mgr.? 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2021-07-31T01:47:28.295 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:27 smithi045 conmon[33655]: audit 2021 2021-07-31T01:47:28.295 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:27 smithi045 conmon[33655]: -07-31T01:47:27.511582+0000 mon.b (mon.2) 27 : audit 2021-07-31T01:47:28.295 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:27 smithi045 conmon[33655]: [DBG] from='mgr.? 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2021-07-31T01:47:28.295 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:27 smithi045 conmon[33655]: audit 2021-07-31T01:47:28.296 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:27 smithi045 conmon[33655]: 2021-07-31T01:47: 2021-07-31T01:47:28.296 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:27 smithi045 conmon[33655]: 27.512322+0000 mon.b (mon.2) 28 : 2021-07-31T01:47:28.296 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:27 smithi045 conmon[33655]: audit [DBG] from='mgr.? 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2021-07-31T01:47:28.296 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:27 smithi045 conmon[33655]: cluster 2021 2021-07-31T01:47:28.296 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:27 smithi045 conmon[33655]: -07-31T01:47:27.645288+0000 mon.a 2021-07-31T01:47:28.297 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:27 smithi045 conmon[33655]: (mon.0) 766 : cluster [DBG] Standby manager daemon y started 2021-07-31T01:47:28.510 INFO:tasks.rados.rados.0.smithi045.stdout:564: done (0 left) 2021-07-31T01:47:28.511 INFO:tasks.rados.rados.0.smithi045.stdout:565: snap_create 2021-07-31T01:47:28.752 INFO:journalctl@ceph.osd.5.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[42050]: debug 2021-07-31T01:47:28.753 INFO:journalctl@ceph.osd.5.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[42050]: 2021-07-31T01:47:28.519+0000 7fc9b4863700 -1 osd.5 pg_epoch: 150 pg[3.5s0( v 146'25 (0'0,146'25] local-lis/les=56/57 n=2 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [5,3,2]p5(0) r=0 lpr=56 crt=146'25 lcod 142'23 mlcod 142'23 active+clean+snaptrim trimq=[55~1] ps=[3d~2,49~1,4e~1,52~1]] removing snap head 2021-07-31T01:47:28.753 INFO:journalctl@ceph.osd.5.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[42050]: 2021-07-31T01:47:28.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: debug 2021-07-31T01:47:28.302+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T01:47:28.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:47:28.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:47:28.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: yield 2021-07-31T01:47:28.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:47:28.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:47:28.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:47:28.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T01:47:28.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:47:28.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T01:47:28.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:47:28.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: await self._waiter 2021-07-31T01:47:28.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:47:28.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: task.result() 2021-07-31T01:47:28.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T01:47:28.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: debug 2021-07-31T01:47:28.304+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T01:47:28.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:47:28.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:47:28.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: yield 2021-07-31T01:47:28.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:47:28.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:47:28.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:47:28.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T01:47:28.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:47:28.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T01:47:28.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:47:28.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: await self._waiter 2021-07-31T01:47:28.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:47:28.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: task.result() 2021-07-31T01:47:28.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T01:47:28.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: debug 2021-07-31T01:47:28.338+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T01:47:28.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:47:28.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:47:28.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: yield 2021-07-31T01:47:28.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:47:28.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:47:28.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:47:28.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T01:47:28.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:47:28.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T01:47:28.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:47:28.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: await self._waiter 2021-07-31T01:47:28.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:47:28.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: task.result() 2021-07-31T01:47:28.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T01:47:28.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: debug 2021-07-31T01:47:28.338+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : Unable to write smithi045:/etc/ceph/ceph.conf: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T01:47:28.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:47:28.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:47:28.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: yield 2021-07-31T01:47:28.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:47:28.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:47:28.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:47:28.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T01:47:28.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:47:28.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T01:47:28.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:47:28.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: await self._waiter 2021-07-31T01:47:28.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:47:28.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: task.result() 2021-07-31T01:47:28.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T01:47:28.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: 2021-07-31T01:47:28.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T01:47:28.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: 2021-07-31T01:47:28.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:47:28.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 175, in _write_remote_file 2021-07-31T01:47:28.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: await self._check_execute_command(host, ['mkdir', '-p', dirname], addr=addr) 2021-07-31T01:47:28.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:47:28.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:47:28.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:47:28.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T01:47:28.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:47:28.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: raise 2021-07-31T01:47:28.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:47:28.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T01:47:28.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:47:28.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:47:28.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T01:47:28.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: debug 2021-07-31T01:47:28.340+0000 7fcb9816a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:47:28.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:47:28.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:47:28.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: yield 2021-07-31T01:47:28.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:47:28.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:47:28.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:47:28.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T01:47:28.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:47:28.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T01:47:28.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:47:28.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: await self._waiter 2021-07-31T01:47:28.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:47:28.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: task.result() 2021-07-31T01:47:28.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T01:47:28.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: 2021-07-31T01:47:28.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T01:47:28.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: 2021-07-31T01:47:28.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:47:28.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 175, in _write_remote_file 2021-07-31T01:47:28.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: await self._check_execute_command(host, ['mkdir', '-p', dirname], addr=addr) 2021-07-31T01:47:28.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:47:28.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:47:28.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:47:28.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T01:47:28.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:47:28.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: raise 2021-07-31T01:47:28.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:47:28.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T01:47:28.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:47:28.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:47:28.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T01:47:28.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: 2021-07-31T01:47:28.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T01:47:28.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: 2021-07-31T01:47:28.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:47:28.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:47:28.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: return f(*arg) 2021-07-31T01:47:28.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 295, in refresh 2021-07-31T01:47:28.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: self.mgr.ssh.write_remote_file(host, path, content, mode, uid, gid) 2021-07-31T01:47:28.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in write_remote_file 2021-07-31T01:47:28.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: self.mgr.event_loop.get_result(self._write_remote_file(*args, **kwargs)) 2021-07-31T01:47:28.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:47:28.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:47:28.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:47:28.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T01:47:28.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:47:28.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: raise self._exception 2021-07-31T01:47:28.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 187, in _write_remote_file 2021-07-31T01:47:28.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: raise OrchestratorError(msg) 2021-07-31T01:47:28.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Unable to write smithi045:/etc/ceph/ceph.conf: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T01:47:28.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: debug 2021-07-31T01:47:28.346+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T01:47:28.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:47:28.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:47:28.786 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: yield 2021-07-31T01:47:28.786 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:47:28.786 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:47:28.786 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:47:28.787 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T01:47:28.787 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:47:28.787 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T01:47:28.787 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:47:28.787 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: await self._waiter 2021-07-31T01:47:28.788 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:47:28.788 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: task.result() 2021-07-31T01:47:28.788 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T01:47:28.788 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: debug 2021-07-31T01:47:28.346+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : Unable to write smithi154:/etc/ceph/ceph.conf: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T01:47:28.788 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:47:28.789 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:47:28.789 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: yield 2021-07-31T01:47:28.789 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:47:28.789 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:47:28.790 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:47:28.790 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T01:47:28.790 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:47:28.790 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T01:47:28.790 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:47:28.791 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: await self._waiter 2021-07-31T01:47:28.791 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:47:28.791 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: task.result() 2021-07-31T01:47:28.791 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T01:47:28.792 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: 2021-07-31T01:47:28.792 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T01:47:28.792 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: 2021-07-31T01:47:28.792 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:47:28.793 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 175, in _write_remote_file 2021-07-31T01:47:28.793 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: await self._check_execute_command(host, ['mkdir', '-p', dirname], addr=addr) 2021-07-31T01:47:28.793 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:47:28.793 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:47:28.793 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:47:28.794 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T01:47:28.794 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:47:28.794 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: raise 2021-07-31T01:47:28.794 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:47:28.795 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T01:47:28.795 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:47:28.795 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:47:28.795 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T01:47:28.795 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: debug 2021-07-31T01:47:28.346+0000 7fcb97969700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:47:28.796 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:47:28.796 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:47:28.796 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: yield 2021-07-31T01:47:28.796 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:47:28.797 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:47:28.797 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:47:28.797 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T01:47:28.797 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:47:28.797 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T01:47:28.798 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:47:28.798 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: await self._waiter 2021-07-31T01:47:28.798 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:47:28.798 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: task.result() 2021-07-31T01:47:28.799 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T01:47:28.799 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: 2021-07-31T01:47:28.799 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T01:47:28.799 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: 2021-07-31T01:47:28.799 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:47:28.800 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 175, in _write_remote_file 2021-07-31T01:47:28.800 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: await self._check_execute_command(host, ['mkdir', '-p', dirname], addr=addr) 2021-07-31T01:47:28.800 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:47:28.800 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:47:28.800 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:47:28.801 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T01:47:28.801 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:47:28.801 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: raise 2021-07-31T01:47:28.801 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:47:28.802 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T01:47:28.802 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:47:28.802 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:47:28.802 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T01:47:28.803 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: 2021-07-31T01:47:28.803 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T01:47:28.803 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: 2021-07-31T01:47:28.803 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:47:28.803 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:47:28.804 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: return f(*arg) 2021-07-31T01:47:28.804 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 295, in refresh 2021-07-31T01:47:28.804 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: self.mgr.ssh.write_remote_file(host, path, content, mode, uid, gid) 2021-07-31T01:47:28.804 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in write_remote_file 2021-07-31T01:47:28.805 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: self.mgr.event_loop.get_result(self._write_remote_file(*args, **kwargs)) 2021-07-31T01:47:28.805 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:47:28.805 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:47:28.805 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:47:28.805 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T01:47:28.806 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:47:28.806 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: raise self._exception 2021-07-31T01:47:28.806 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 187, in _write_remote_file 2021-07-31T01:47:28.806 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: raise OrchestratorError(msg) 2021-07-31T01:47:28.807 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Unable to write smithi154:/etc/ceph/ceph.conf: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T01:47:28.807 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: debug 2021-07-31T01:47:28.390+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T01:47:28.807 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:47:28.807 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:47:28.808 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: yield 2021-07-31T01:47:28.808 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:47:28.808 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:47:28.808 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:47:28.809 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T01:47:28.809 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:47:28.809 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T01:47:28.809 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:47:28.809 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: await self._waiter 2021-07-31T01:47:28.810 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:47:28.810 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: task.result() 2021-07-31T01:47:28.810 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T01:47:28.810 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: debug 2021-07-31T01:47:28.390+0000 7fcb94162700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:47:28.811 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:47:28.811 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:47:28.811 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: yield 2021-07-31T01:47:28.811 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:47:28.811 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:47:28.812 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:47:28.812 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T01:47:28.812 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:47:28.812 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T01:47:28.812 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:47:28.813 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: await self._waiter 2021-07-31T01:47:28.813 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:47:28.813 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: task.result() 2021-07-31T01:47:28.813 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T01:47:28.814 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: 2021-07-31T01:47:28.814 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T01:47:28.814 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: 2021-07-31T01:47:28.814 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:47:28.814 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:47:28.815 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: return f(*arg) 2021-07-31T01:47:28.815 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:47:28.815 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:47:28.815 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:47:28.815 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:47:28.816 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:47:28.816 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:47:28.816 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:47:28.816 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T01:47:28.816 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:47:28.817 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: raise self._exception 2021-07-31T01:47:28.817 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:47:28.817 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:47:28.817 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:47:28.817 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T01:47:28.818 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:47:28.818 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: raise 2021-07-31T01:47:28.818 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:47:28.818 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T01:47:28.819 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:47:28.819 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:47:28.819 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T01:47:28.819 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: debug 2021-07-31T01:47:28.392+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T01:47:28.819 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:47:28.820 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:47:28.820 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: yield 2021-07-31T01:47:28.820 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:47:28.820 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:47:28.820 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:47:28.821 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T01:47:28.821 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:47:28.821 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T01:47:28.821 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:47:28.822 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: await self._waiter 2021-07-31T01:47:28.822 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:47:28.822 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: task.result() 2021-07-31T01:47:28.822 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T01:47:28.822 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: debug 2021-07-31T01:47:28.392+0000 7fcb93961700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:47:28.823 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:47:28.823 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:47:28.823 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: yield 2021-07-31T01:47:28.823 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:47:28.823 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:47:28.824 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:47:28.824 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T01:47:28.824 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:47:28.824 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T01:47:28.825 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:47:28.825 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: await self._waiter 2021-07-31T01:47:28.825 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:47:28.825 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: task.result() 2021-07-31T01:47:28.825 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T01:47:28.826 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: 2021-07-31T01:47:28.826 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T01:47:28.826 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: 2021-07-31T01:47:28.826 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:47:28.826 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:47:28.827 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: return f(*arg) 2021-07-31T01:47:28.827 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:47:28.827 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:47:28.827 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:47:28.828 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:47:28.828 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:47:28.828 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:47:28.828 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:47:28.828 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T01:47:28.829 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:47:28.829 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: raise self._exception 2021-07-31T01:47:28.829 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:47:28.829 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:47:28.829 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:47:28.830 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T01:47:28.830 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:47:28.830 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: raise 2021-07-31T01:47:28.830 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:47:28.830 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T01:47:28.831 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:47:28.831 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:47:28.831 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T01:47:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: audit 2021-07-31T01:47:27.790183+0000 mon.b (mon.2) 29 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2021-07-31T01:47:29.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: audit 2021-07-31T01:47:27.790307+0000 mon.b (mon.2) 30 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-07-31T01:47:29.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: audit 2021-07-31T01:47:27.790414+0000 2021-07-31T01:47:29.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: mon.b (mon.2) 31 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-07-31T01:47:29.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: 2021-07-31T01:47:29.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: audit 2021-07-31T01:47 2021-07-31T01:47:29.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: :27.791415+0000 mon.b (mon.2) 32 2021-07-31T01:47:29.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2021-07-31T01:47:29.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: audit 2021-07- 2021-07-31T01:47:29.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: 31T01:47:27.791620+0000 mon.a ( 2021-07-31T01:47:29.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: mon.0) 767 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2021-07-31T01:47:29.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: audit 2021- 2021-07-31T01:47:29.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: 07-31T01:47:27.791630+0000 2021-07-31T01:47:29.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: mon.b (mon.2) 33 : audit [DBG] 2021-07-31T01:47:29.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2021-07-31T01:47:29.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: cluster 2021-07-31T01: 2021-07-31T01:47:29.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: 47:27.791700+0000 mon.a (mon.0) 2021-07-31T01:47:29.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: 768 : cluster [DBG] mgrmap e15: x(active, starting, since 0.287282s), standbys: y 2021-07-31T01:47:29.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: audit 2021-07 2021-07-31T01:47:29.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: -31T01:47:27.791855+0000 mon.b 2021-07-31T01:47:29.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: (mon.2) 34 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-07-31T01:47:29.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: 2021-07-31T01:47:29.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: audit 2021-07-31T01:47:29.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: 2021-07-31T01:47:27.792091+0000 2021-07-31T01:47:29.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: mon.b (mon.2) 35 : audit [DBG] 2021-07-31T01:47:29.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-07-31T01:47:29.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: audit 2021-07-31T 2021-07-31T01:47:29.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: 01:47:27.792502+0000 mon.b (mon 2021-07-31T01:47:29.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: .2) 36 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-07-31T01:47:29.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: audit 2021-07-31T01:47:29.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: 2021-07-31T01:47:27. 2021-07-31T01:47:29.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: 792591+0000 mon.b (mon.2) 37 : audit 2021-07-31T01:47:29.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-07-31T01:47:29.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: audit 2021-07-31 2021-07-31T01:47:29.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: T01:47:27.792676+0000 mon.b (mon.2) 38 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-07-31T01:47:29.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: audit 2021 2021-07-31T01:47:29.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: -07-31T01:47:27.792760 2021-07-31T01:47:29.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: +0000 mon.b (mon.2) 39 : audit 2021-07-31T01:47:29.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-07-31T01:47:29.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: audit 2021-07-31T01 2021-07-31T01:47:29.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: :47:27.792840+0000 mon.b (mon. 2021-07-31T01:47:29.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: 2) 40 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-07-31T01:47:29.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: audit 2021- 2021-07-31T01:47:29.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: 07-31T01:47:27.792930+0000 2021-07-31T01:47:29.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: mon.b (mon.2) 41 : audit [DBG] 2021-07-31T01:47:29.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-07-31T01:47:29.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: audit 2021-07-31T01 2021-07-31T01:47:29.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: :47:27.796027+0000 mon.b (mon. 2021-07-31T01:47:29.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: 2) 42 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2021-07-31T01:47:29.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: audit 2021-07-31T01:47:29.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: 2021-07-31T01:47:27.796116 2021-07-31T01:47:29.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: +0000 mon.b (mon.2) 43 : audit 2021-07-31T01:47:29.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2021-07-31T01:47:29.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: audit 2021-07-31T 2021-07-31T01:47:29.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: 01:47:27.797839+0000 mon.b (mon 2021-07-31T01:47:29.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: .2) 44 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2021-07-31T01:47:29.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: cluster 2021-07-31T01:47:29.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: 2021-07-31T01:47:27. 2021-07-31T01:47:29.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: 817985+0000 mon.a (mon.0) 769 : cluster 2021-07-31T01:47:29.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: [INF] Manager daemon x is now available 2021-07-31T01:47:29.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: audit 2021-07-31 2021-07-31T01:47:29.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: T01:47:27.922023+0000 mon.b ( 2021-07-31T01:47:29.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: mon.2) 45 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:47:29.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: audit 2021-07-31T01:47:29.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: 2021-07-31T01:47:27. 2021-07-31T01:47:29.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: 923467+0000 mon.b (mon.2) 46 : audit 2021-07-31T01:47:29.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:47:29.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: audit 2021-07 2021-07-31T01:47:29.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: -31T01:47:27.924667+0000 mon.b (mon.2) 2021-07-31T01:47:29.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: 47 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:47:29.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: audit 2021-07-31T 2021-07-31T01:47:29.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: 01:47:27.926065+0000 mon.b (mon.2 2021-07-31T01:47:29.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: ) 48 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:47:29.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: audit 2021 2021-07-31T01:47:29.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: -07-31T01:47:27.928756 2021-07-31T01:47:29.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: +0000 mon.b (mon.2) 49 : audit [INF] 2021-07-31T01:47:29.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "osd pool rename", "format": "json", "srcpool": "device_health_metrics", "destpool": ".mgr"}]: dispatch 2021-07-31T01:47:29.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: audit 2021-07-31T01 2021-07-31T01:47:29.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: :47:27.929238+0000 mon.a (mon. 2021-07-31T01:47:29.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: 0) 770 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix": "osd pool rename", "format": "json", "srcpool": "device_health_metrics", "destpool": ".mgr"}]: dispatch 2021-07-31T01:47:29.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: audit 2021-07-31T01:47:29.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: 2021-07-31T01:47:27. 2021-07-31T01:47:29.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: 943291+0000 mon.b (mon.2) 50 : audit 2021-07-31T01:47:29.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:47:29.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: audit 2021-07- 2021-07-31T01:47:29.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: 31T01:47:27.943737+0000 mon.a 2021-07-31T01:47:29.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: (mon.0) 771 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:47:29.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: 2021-07-31T01:47:29.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: audit 2021-07-31T01:47:29.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: 2021-07-31T01:47:27.951366+0000 2021-07-31T01:47:29.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: mon.b (mon.2) 51 : audit [INF] 2021-07-31T01:47:29.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T01:47:29.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: audit 2021-07-31T01 2021-07-31T01:47:29.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: :47:27.951832+0000 mon.a (mon. 2021-07-31T01:47:29.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: 0) 772 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T01:47:29.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: audit 2021 2021-07-31T01:47:29.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: -07-31T01:47:28.348613 2021-07-31T01:47:29.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: +0000 mon.b (mon.2) 52 : audit 2021-07-31T01:47:29.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:47:29.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: audit 2021-07-31T01:47:29.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: 2021-07-31T01:47:28.349588+0000 mon.b ( 2021-07-31T01:47:29.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: mon.2) 53 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:47:29.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: audit 2021- 2021-07-31T01:47:29.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: 07-31T01:47:28.350206+0000 2021-07-31T01:47:29.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: mon.b (mon.2) 54 : audit [INF] 2021-07-31T01:47:29.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:47:29.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: audit 2021-07-31T01: 2021-07-31T01:47:29.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: 47:28.509150+0000 mon.a (mon.0 2021-07-31T01:47:29.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: ) 773 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd='[{"prefix": "osd pool rename", "format": "json", "srcpool": "device_health_metrics", "destpool": ".mgr"}]': finished 2021-07-31T01:47:29.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: cluster 2021- 2021-07-31T01:47:29.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: 07-31T01:47:28.509263+0000 2021-07-31T01:47:29.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: mon.a (mon.0) 774 : cluster [DBG] 2021-07-31T01:47:29.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: osdmap e150: 8 total, 8 up, 8 in 2021-07-31T01:47:29.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: audit 2021-07-31T01 2021-07-31T01:47:29.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: :47:28.512473+0000 mon.b (mon. 2021-07-31T01:47:29.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: 2) 55 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2021-07-31T01:47:29.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: audit 2021-07-31T01:47:29.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: 2021-07-31T01:47:28. 2021-07-31T01:47:29.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: 512916+0000 mon.a (mon.0) 775 : audit 2021-07-31T01:47:29.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:28 smithi154 conmon[34553]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2021-07-31T01:47:29.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: audit 2021-07-31T01:47:27.790183+0000 2021-07-31T01:47:29.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: mon.b (mon.2) 29 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2021-07-31T01:47:29.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: audit 2021-07-31T01:47:27.790307+0000 mon.b (mon.2) 30 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-07-31T01:47:29.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: audit 2021-07-31T01:47:27.790414+0000 mon.b (mon.2) 31 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-07-31T01:47:29.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: audit 2021-07-31T01:47:27.791415+0000 mon.b (mon.2) 32 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2021-07-31T01:47:29.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: audit 2021-07-31T01:47:27.791620+0000 mon.a (mon.0) 767 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2021-07-31T01:47:29.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: audit 2021-07-31T01:47:27.791630+0000 mon.b (mon.2) 33 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2021-07-31T01:47:29.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: cluster 2021-07-31T01:47:27.791700+0000 mon.a (mon.0) 768 : cluster [DBG] mgrmap e15: x(active, starting, since 0.287282s), standbys: y 2021-07-31T01:47:29.282 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: audit 2021-07-31T01:47:27.791855+0000 mon.b (mon.2) 34 : audit 2021-07-31T01:47:29.282 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-07-31T01:47:29.282 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: audit 2021-07-31T01:47:29.282 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: 2021-07-31T01:47:27.792091 2021-07-31T01:47:29.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: +0000 mon.b (mon.2) 35 : audit [DBG] 2021-07-31T01:47:29.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-07-31T01:47:29.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: audit 2021-07-31T01:47:29.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: 2021-07-31T01:47:27.792502 2021-07-31T01:47:29.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: +0000 mon.b (mon.2) 36 : audit [DBG] 2021-07-31T01:47:29.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-07-31T01:47:29.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: audit 2021-07-31T01:47:29.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: 2021-07-31T01:47:27.792591+0000 2021-07-31T01:47:29.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: mon.b (mon.2) 37 : audit [DBG] 2021-07-31T01:47:29.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-07-31T01:47:29.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T01:47:29.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: T01:47:27.792676+0000 mon.b (mon 2021-07-31T01:47:29.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: .2) 38 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-07-31T01:47:29.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: audit 2021-07-31T01:47:29.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: 2021-07-31T01: 2021-07-31T01:47:29.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: 47:27.792760+0000 mon.b (mon.2) 2021-07-31T01:47:29.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: 39 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-07-31T01:47:29.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: audit 2021-07-31T01:47:29.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: 2021-07-31T01:47:27.792840 2021-07-31T01:47:29.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: +0000 mon.b (mon.2) 40 : audit 2021-07-31T01:47:29.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-07-31T01:47:29.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: audit 2021-07-31T01:47: 2021-07-31T01:47:29.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: 27.792930+0000 mon.b (mon.2) 41 : audit 2021-07-31T01:47:29.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: audit 2021-07-31T01:47:27. 2021-07-31T01:47:29.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: 790183+0000 mon.b (mon.2) 29 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2021-07-31T01:47:29.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: audit 2021-07-31T01:47:27.790307+0000 mon.b (mon.2) 30 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-07-31T01:47:29.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: audit 2021-07-31T01:47:27.790414+0000 mon.b (mon.2) 31 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-07-31T01:47:29.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: audit 2021-07-31T01:47:27.791415+0000 mon.b (mon.2) 32 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2021-07-31T01:47:29.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: audit 2021-07-31T01:47:27.791620+0000 mon.a (mon.0) 767 : audit [INF] from='mgr.14146 172.21.15.45:0/3382313699' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2021-07-31T01:47:29.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: audit 2021-07-31T01:47:27.791630+0000 mon.b (mon.2) 33 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2021-07-31T01:47:29.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: cluster 2021-07-31T01:47:27.791700+0000 mon.a (mon.0) 768 : cluster [DBG] mgrmap e15: x(active, starting, since 0.287282s), standbys: y 2021-07-31T01:47:29.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: audit 2021-07-31T01:47:27.791855+0000 mon.b (mon.2) 34 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-07-31T01:47:29.290 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: audit 2021-07-31T01:47:27.792091+0000 mon.b (mon.2) 35 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-07-31T01:47:29.290 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: audit 2021-07-31T01:47:27.792502+0000 mon.b (mon.2) 36 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-07-31T01:47:29.290 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: audit 2021-07-31T01:47:27.792591+0000 mon.b (mon.2) 37 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-07-31T01:47:29.290 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: audit 2021-07-31T01:47:27.792676+0000 mon.b (mon.2) 38 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-07-31T01:47:29.290 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: audit 2021-07-31T01:47:27.792760+0000 mon.b (mon.2) 39 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-07-31T01:47:29.291 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: audit 2021-07-31T01:47:27.792840+0000 mon.b (mon.2) 40 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-07-31T01:47:29.291 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: audit 2021-07-31T01:47:27.792930+0000 mon.b (mon.2) 41 : audit 2021-07-31T01:47:29.291 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-07-31T01:47:29.291 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: audit 2021-07-31T01:47:27.796027+0000 mon.b (mon.2) 42 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2021-07-31T01:47:29.292 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: audit 2021-07-31T01:47:29.292 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: 2021-07-31T01:47:27.796116+0000 mon.b (mon.2) 43 2021-07-31T01:47:29.292 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2021-07-31T01:47:29.292 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: audit 2021-07-31T01:47:27 2021-07-31T01:47:29.292 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: .797839+0000 mon.b (mon.2) 44 : audit [DBG] 2021-07-31T01:47:29.293 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-07-31T01:47:29.293 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: audit 2021-07-31T01:47:29.293 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: 2021-07-31T01:47:27 2021-07-31T01:47:29.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: .796027+0000 mon.b (mon.2) 42 : audit 2021-07-31T01:47:29.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2021-07-31T01:47:29.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: audit 2021-07-31T01:47:29.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: 2021-07-31T01:47:27. 2021-07-31T01:47:29.295 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: 796116+0000 mon.b (mon.2) 43 : audit 2021-07-31T01:47:29.295 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2021-07-31T01:47:29.295 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: audit 2021-07-31T01:47:29.295 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: 2021-07-31T01:47:27 2021-07-31T01:47:29.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: .797839+0000 mon.b (mon.2) 44 : 2021-07-31T01:47:29.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2021-07-31T01:47:29.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: cluster 2021-07-31T01:47:29.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: 2021-07-31T01:47: 2021-07-31T01:47:29.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: 27.817985+0000 mon.a (mon.0) 769 2021-07-31T01:47:29.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: : cluster [INF] Manager daemon x is now available 2021-07-31T01:47:29.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: audit 2021-07-31T01:47:29.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: 2021-07-31T01:47 2021-07-31T01:47:29.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: :27.922023+0000 mon.b (mon.2) 45 2021-07-31T01:47:29.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:47:29.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T01:47:29.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: 01:47:27.923467+0000 mon.b (mon.2) 46 : 2021-07-31T01:47:29.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:47:29.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: audit 2021- 2021-07-31T01:47:29.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: 07-31T01:47:27.924667+0000 2021-07-31T01:47:29.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: mon.b (mon.2) 47 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:47:29.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: 2021-07-31T01:47:29.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: audit 2021- 2021-07-31T01:47:29.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: 07-31T01:47:27.926065+0000 2021-07-31T01:47:29.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: mon.b (mon.2) 48 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:47:29.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: audit 2021 2021-07-31T01:47:29.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: -07-31T01:47:27.928756+0000 2021-07-31T01:47:29.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: mon.b (mon.2) 49 : audit [INF] 2021-07-31T01:47:29.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "osd pool rename", "format": "json", "srcpool": "device_health_metrics", "destpool": ".mgr"}]: dispatch 2021-07-31T01:47:29.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: audit 2021-07-31T01 2021-07-31T01:47:29.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: :47:27.929238+0000 mon.a (mon.0) 770 2021-07-31T01:47:29.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix": "osd pool rename", "format": "json", "srcpool": "device_health_metrics", "destpool": ".mgr"}]: dispatch 2021-07-31T01:47:29.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: audit 2021-07 2021-07-31T01:47:29.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: -31T01:47:27.943291+0000 mon.b 2021-07-31T01:47:29.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: (mon.2) 50 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:47:29.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: 2021-07-31T01:47:29.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: audit 2021-07-31T01:47 2021-07-31T01:47:29.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: :27.943737+0000 mon.a (mon.0) 2021-07-31T01:47:29.303 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2021-07-31T01:47:29.303 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: cluster 2021-07-31T01:47:29.303 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: 2021-07- 2021-07-31T01:47:29.304 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: 31T01:47:27.817985+0000 mon.a ( 2021-07-31T01:47:29.304 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: mon.0) 769 : cluster [INF] 2021-07-31T01:47:29.304 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: Manager daemon x is now available 2021-07-31T01:47:29.304 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: audit 2021-07-31T01:47:29.304 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: 2021-07-31 2021-07-31T01:47:29.305 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: T01:47:27.922023+0000 2021-07-31T01:47:29.305 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: mon.b (mon.2) 45 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:47:29.305 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: audit 2021-07-31T01:47:29.305 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: 2021-07-31T01 2021-07-31T01:47:29.306 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: :47:27.923467+0000 2021-07-31T01:47:29.306 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: mon.b (mon.2) 46 : 2021-07-31T01:47:29.306 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:47:29.306 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: audit 2021-07- 2021-07-31T01:47:29.306 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: 31T01:47:27.924667 2021-07-31T01:47:29.307 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: +0000 mon.b (mon.2) 47 2021-07-31T01:47:29.307 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:47:29.307 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: audit 2021-07-31T01:47:29.307 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: 2021-07-31T01:47:27.926065 2021-07-31T01:47:29.307 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: +0000 mon.b (mon.2) 48 : audit [INF] 2021-07-31T01:47:29.308 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:47:29.308 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: audit 2021-07 2021-07-31T01:47:29.308 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: -31T01:47:27 2021-07-31T01:47:29.308 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: .928756+0000 mon.b (mon.2 2021-07-31T01:47:29.309 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: ) 49 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "osd pool rename", "format": "json", "srcpool": "device_health_metrics", "destpool": ".mgr"}]: dispatch 2021-07-31T01:47:29.309 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: audit 2021-07-31T01:47:29.309 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: 2021-07- 2021-07-31T01:47:29.309 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: 31T01:47:27. 2021-07-31T01:47:29.309 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: 929238+0000 mon.a (mon.0) 2021-07-31T01:47:29.310 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: 770 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix": "osd pool rename", "format": "json", "srcpool": "device_health_metrics", "destpool": ".mgr"}]: dispatch 2021-07-31T01:47:29.310 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: audit 2021-07-31T01:47:29.310 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: 2021-07 2021-07-31T01:47:29.310 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: -31T01:47:27 2021-07-31T01:47:29.311 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: .943291+0000 mon.b (mon 2021-07-31T01:47:29.311 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: .2) 50 : audit [INF] 2021-07-31T01:47:29.311 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:47:29.311 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: audit 2021-07-31T01:47:29.311 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: 2021-07-31T 2021-07-31T01:47:29.312 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: 01:47:27.943737+0000 2021-07-31T01:47:29.312 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: mon.a (mon.0) 2021-07-31T01:47:29.312 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: 771 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:47:29.312 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: audit 2021-07-31T01:47:29.313 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: 2021-07-31T01 2021-07-31T01:47:29.313 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: :47:27.951366+0000 2021-07-31T01:47:29.313 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: mon.b (mon.2) 51 : audit 2021-07-31T01:47:29.313 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T01:47:29.313 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: audit 2021-07-31T01:47:29.314 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: 2021-07-31T01:47 2021-07-31T01:47:29.314 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: :27.951832+0000 mon.a (mon 2021-07-31T01:47:29.314 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: .0) 772 : audit [INF] 2021-07-31T01:47:29.314 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T01:47:29.314 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: audit 2021-07-31T01:47:29.315 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: 2021-07-31T01:47:28.348613 2021-07-31T01:47:29.315 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: +0000 mon.b (mon.2) 52 2021-07-31T01:47:29.315 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:47:29.315 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: audit 2021 2021-07-31T01:47:29.316 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: -07-31T01:47:28.349588+0000 mon.b (mon 2021-07-31T01:47:29.316 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: .2) 53 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:47:29.316 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: audit 2021 2021-07-31T01:47:29.316 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: -07-31T01:47:28 2021-07-31T01:47:29.316 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: .350206+0000 mon.b (mon.2 2021-07-31T01:47:29.317 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: ) 54 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:47:29.317 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: audit 2021- 2021-07-31T01:47:29.317 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: 07-31T01:47:28 2021-07-31T01:47:29.317 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: .509150+0000 mon.a (mon 2021-07-31T01:47:29.317 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: .0) 773 : audit [INF] 2021-07-31T01:47:29.318 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: from='mgr.24542 ' entity='mgr.x' cmd='[{"prefix": "osd pool rename", "format": "json", "srcpool": "device_health_metrics", "destpool": ".mgr"}]': finished 2021-07-31T01:47:29.318 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: cluster 2021-07 2021-07-31T01:47:29.318 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: -31T01:47:28 2021-07-31T01:47:29.318 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: .509263+0000 mon.a (mon. 2021-07-31T01:47:29.318 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: 0) 774 : cluster [DBG] 2021-07-31T01:47:29.319 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: osdmap e150: 8 total, 8 up, 8 in 2021-07-31T01:47:29.319 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: audit 2021 2021-07-31T01:47:29.319 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: -07-31T01:47:28. 2021-07-31T01:47:29.319 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: 512473+0000 mon.b (mon.2 2021-07-31T01:47:29.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: ) 55 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2021-07-31T01:47:29.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: 2021-07-31T01:47:29.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: audit 2021-07-31T01:47:29.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: 2021-07-31 2021-07-31T01:47:29.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: T01:47:28.512916 2021-07-31T01:47:29.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: +0000 mon.a (mon.0 2021-07-31T01:47:29.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: ) 775 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2021-07-31T01:47:29.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[33655]: 2021-07-31T01:47:29.321 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: 771 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:47:29.322 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: audit 2021- 2021-07-31T01:47:29.322 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: 07-31T01:47:27.951366+0000 2021-07-31T01:47:29.322 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: mon.b (mon.2) 51 : audit [INF] 2021-07-31T01:47:29.322 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T01:47:29.322 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: audit 2021-07-31T01: 2021-07-31T01:47:29.323 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: 47:27.951832+0000 mon.a (mon.0) 772 2021-07-31T01:47:29.323 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T01:47:29.323 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: audit 2021 2021-07-31T01:47:29.323 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: -07-31T01:47:28.348613+0000 2021-07-31T01:47:29.324 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: mon.b (mon.2) 52 : audit [DBG] 2021-07-31T01:47:29.324 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:47:29.324 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: audit 2021-07-31T01 2021-07-31T01:47:29.324 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: :47:28.349588+0000 mon.b (mon 2021-07-31T01:47:29.324 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: .2) 53 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:47:29.325 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: audit 2021-07-31T01:47:29.325 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: 2021-07-31T01:47:28.350206 2021-07-31T01:47:29.325 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: +0000 mon.b (mon.2) 54 : audit [INF] 2021-07-31T01:47:29.325 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:47:29.325 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: audit 2021-07-31T01:47:28. 2021-07-31T01:47:29.326 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: 509150+0000 mon.a (mon.0) 773 : audit 2021-07-31T01:47:29.326 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: [INF] from='mgr.24542 ' entity='mgr.x' cmd='[{"prefix": "osd pool rename", "format": "json", "srcpool": "device_health_metrics", "destpool": ".mgr"}]': finished 2021-07-31T01:47:29.326 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: cluster 2021-07-31 2021-07-31T01:47:29.326 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: T01:47:28.509263+0000 mon.a ( 2021-07-31T01:47:29.326 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: mon.0) 774 : cluster [DBG] osdmap e150: 8 total, 8 up, 8 in 2021-07-31T01:47:29.327 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: audit 2021 2021-07-31T01:47:29.327 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: -07-31T01:47:28.512473+0000 2021-07-31T01:47:29.327 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: mon.b (mon.2) 55 : audit [INF] 2021-07-31T01:47:29.327 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2021-07-31T01:47:29.327 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: audit 2021-07-31T01 2021-07-31T01:47:29.328 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: :47:28.512916+0000 mon.a (mon. 2021-07-31T01:47:29.328 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:28 smithi045 conmon[39718]: 0) 775 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2021-07-31T01:47:29.511 INFO:tasks.rados.rados.0.smithi045.stdout:565: done (0 left) 2021-07-31T01:47:29.512 INFO:tasks.rados.rados.0.smithi045.stdout:566: read oid 47 snap -1 2021-07-31T01:47:29.512 INFO:tasks.rados.rados.0.smithi045.stdout:566: expect (ObjNum 46 snap 0 seq_num 46) 2021-07-31T01:47:29.512 INFO:tasks.rados.rados.0.smithi045.stdout:567: delete oid 33 current snap is 49 2021-07-31T01:47:29.514 INFO:tasks.rados.rados.0.smithi045.stdout:566: expect (ObjNum 46 snap 0 seq_num 46) 2021-07-31T01:47:29.536 INFO:tasks.rados.rados.0.smithi045.stdout:566: done (1 left) 2021-07-31T01:47:29.536 INFO:tasks.rados.rados.0.smithi045.stdout:567: done (0 left) 2021-07-31T01:47:29.536 INFO:tasks.rados.rados.0.smithi045.stdout:568: rollback oid 46 current snap is 49 2021-07-31T01:47:29.537 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 46 to 40 2021-07-31T01:47:29.537 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 49 current snap is 49 2021-07-31T01:47:29.537 INFO:tasks.rados.rados.0.smithi045.stdout:569: seq_num 185 ranges {0=704512} 2021-07-31T01:47:29.540 INFO:tasks.rados.rados.0.smithi045.stdout:569: writing smithi04569269-49 from 0 to 704512 tid 1 2021-07-31T01:47:29.540 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 10 current snap is 49 2021-07-31T01:47:29.540 INFO:tasks.rados.rados.0.smithi045.stdout:568: finishing rollback tid 1 to smithi04569269-46 2021-07-31T01:47:29.540 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 46 v 0 (ObjNum 0 snap 0 seq_num 0) dirty dne 2021-07-31T01:47:29.540 INFO:tasks.rados.rados.0.smithi045.stdout:570: seq_num 186 ranges {1794048=720896} 2021-07-31T01:47:29.555 INFO:tasks.rados.rados.0.smithi045.stdout:570: writing smithi04569269-10 from 1794048 to 2514944 tid 1 2021-07-31T01:47:29.555 INFO:tasks.rados.rados.0.smithi045.stdout:568: done (2 left) 2021-07-31T01:47:29.555 INFO:tasks.rados.rados.0.smithi045.stdout:571: rollback oid 31 current snap is 49 2021-07-31T01:47:29.556 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 31 to 45 2021-07-31T01:47:29.556 INFO:tasks.rados.rados.0.smithi045.stdout:569: finishing write tid 1 to smithi04569269-49 2021-07-31T01:47:29.556 INFO:tasks.rados.rados.0.smithi045.stdout:569: finishing write tid 2 to smithi04569269-49 2021-07-31T01:47:29.556 INFO:tasks.rados.rados.0.smithi045.stdout:569: finishing write tid 3 to smithi04569269-49 2021-07-31T01:47:29.556 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 49 v 393 (ObjNum 185 snap 49 seq_num 185) dirty exists 2021-07-31T01:47:29.557 INFO:tasks.rados.rados.0.smithi045.stdout:569: left oid 49 (ObjNum 185 snap 49 seq_num 185) 2021-07-31T01:47:29.557 INFO:tasks.rados.rados.0.smithi045.stdout:569: done (2 left) 2021-07-31T01:47:29.557 INFO:tasks.rados.rados.0.smithi045.stdout:572: read oid 39 snap -1 2021-07-31T01:47:29.557 INFO:tasks.rados.rados.0.smithi045.stdout:572: expect (ObjNum 94 snap 15 seq_num 94) 2021-07-31T01:47:29.557 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 3 current snap is 49 2021-07-31T01:47:29.558 INFO:tasks.rados.rados.0.smithi045.stdout:573: seq_num 187 ranges {794624=737280} 2021-07-31T01:47:29.565 INFO:tasks.rados.rados.0.smithi045.stdout:573: writing smithi04569269-3 from 794624 to 1531904 tid 1 2021-07-31T01:47:29.565 INFO:tasks.rados.rados.0.smithi045.stdout:574: read oid 49 snap -1 2021-07-31T01:47:29.565 INFO:tasks.rados.rados.0.smithi045.stdout:574: expect (ObjNum 185 snap 49 seq_num 185) 2021-07-31T01:47:29.565 INFO:tasks.rados.rados.0.smithi045.stdout:575: read oid 2 snap -1 2021-07-31T01:47:29.566 INFO:tasks.rados.rados.0.smithi045.stdout:575: expect deleted 2021-07-31T01:47:29.566 INFO:tasks.rados.rados.0.smithi045.stdout:576: rollback oid 28 current snap is 49 2021-07-31T01:47:29.566 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 28 to 39 2021-07-31T01:47:29.566 INFO:tasks.rados.rados.0.smithi045.stdout:571: finishing rollback tid 0 to smithi04569269-31 2021-07-31T01:47:29.567 INFO:tasks.rados.rados.0.smithi045.stdout:571: finishing rollback tid 1 to smithi04569269-31 2021-07-31T01:47:29.567 INFO:tasks.rados.rados.0.smithi045.stdout:571: finishing rollback tid 2 to smithi04569269-31 2021-07-31T01:47:29.567 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 31 v 397 (ObjNum 157 snap 37 seq_num 157) dirty exists 2021-07-31T01:47:29.567 INFO:tasks.rados.rados.0.smithi045.stdout:571: done (6 left) 2021-07-31T01:47:29.568 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 41 current snap is 49 2021-07-31T01:47:29.568 INFO:tasks.rados.rados.0.smithi045.stdout:570: finishing write tid 1 to smithi04569269-10 2021-07-31T01:47:29.568 INFO:tasks.rados.rados.0.smithi045.stdout:577: seq_num 188 ranges {761856=753664} 2021-07-31T01:47:29.573 INFO:tasks.rados.rados.0.smithi045.stdout:577: writing smithi04569269-41 from 761856 to 1515520 tid 1 2021-07-31T01:47:29.573 INFO:tasks.rados.rados.0.smithi045.stdout:578: copy_from oid 43 from oid 8 current snap is 49 2021-07-31T01:47:29.573 INFO:tasks.rados.rados.0.smithi045.stdout:570: finishing write tid 2 to smithi04569269-10 2021-07-31T01:47:29.573 INFO:tasks.rados.rados.0.smithi045.stdout:570: finishing write tid 3 to smithi04569269-10 2021-07-31T01:47:29.574 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 10 v 250 (ObjNum 186 snap 49 seq_num 186) dirty exists 2021-07-31T01:47:29.574 INFO:tasks.rados.rados.0.smithi045.stdout:570: left oid 10 (ObjNum 186 snap 49 seq_num 186) 2021-07-31T01:47:29.574 INFO:tasks.rados.rados.0.smithi045.stdout:572: expect (ObjNum 94 snap 15 seq_num 94) 2021-07-31T01:47:29.624 INFO:tasks.rados.rados.0.smithi045.stdout:570: done (7 left) 2021-07-31T01:47:29.624 INFO:tasks.rados.rados.0.smithi045.stdout:572: done (6 left) 2021-07-31T01:47:29.624 INFO:tasks.rados.rados.0.smithi045.stdout:575: done (5 left) 2021-07-31T01:47:29.625 INFO:tasks.rados.rados.0.smithi045.stdout:579: rollback oid 36 current snap is 49 2021-07-31T01:47:29.625 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 36 to 45 2021-07-31T01:47:29.625 INFO:tasks.rados.rados.0.smithi045.stdout:573: finishing write tid 1 to smithi04569269-3 2021-07-31T01:47:29.625 INFO:tasks.rados.rados.0.smithi045.stdout:576: finishing rollback tid 0 to smithi04569269-28 2021-07-31T01:47:29.626 INFO:tasks.rados.rados.0.smithi045.stdout:576: finishing rollback tid 1 to smithi04569269-28 2021-07-31T01:47:29.626 INFO:tasks.rados.rados.0.smithi045.stdout:573: finishing write tid 2 to smithi04569269-3 2021-07-31T01:47:29.626 INFO:tasks.rados.rados.0.smithi045.stdout:576: finishing rollback tid 2 to smithi04569269-28 2021-07-31T01:47:29.626 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 28 v 254 (ObjNum 165 snap 38 seq_num 165) dirty exists 2021-07-31T01:47:29.626 INFO:tasks.rados.rados.0.smithi045.stdout:573: finishing write tid 3 to smithi04569269-3 2021-07-31T01:47:29.627 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 3 v 400 (ObjNum 187 snap 49 seq_num 187) dirty exists 2021-07-31T01:47:29.627 INFO:tasks.rados.rados.0.smithi045.stdout:573: left oid 3 (ObjNum 187 snap 49 seq_num 187) 2021-07-31T01:47:29.627 INFO:tasks.rados.rados.0.smithi045.stdout:574: expect (ObjNum 185 snap 49 seq_num 185) 2021-07-31T01:47:29.648 INFO:tasks.rados.rados.0.smithi045.stdout:578: finishing copy_from to smithi04569269-43 2021-07-31T01:47:29.649 INFO:tasks.rados.rados.0.smithi045.stdout:578: got expected ENOENT (src dne) 2021-07-31T01:47:29.649 INFO:tasks.rados.rados.0.smithi045.stdout:578: finishing copy_from racing read to smithi04569269-43 2021-07-31T01:47:29.649 INFO:tasks.rados.rados.0.smithi045.stdout:577: finishing write tid 1 to smithi04569269-41 2021-07-31T01:47:29.649 INFO:tasks.rados.rados.0.smithi045.stdout:577: finishing write tid 2 to smithi04569269-41 2021-07-31T01:47:29.649 INFO:tasks.rados.rados.0.smithi045.stdout:577: finishing write tid 3 to smithi04569269-41 2021-07-31T01:47:29.650 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 41 v 66 (ObjNum 188 snap 49 seq_num 188) dirty exists 2021-07-31T01:47:29.650 INFO:tasks.rados.rados.0.smithi045.stdout:577: left oid 41 (ObjNum 188 snap 49 seq_num 188) 2021-07-31T01:47:29.650 INFO:tasks.rados.rados.0.smithi045.stdout:579: finishing rollback tid 0 to smithi04569269-36 2021-07-31T01:47:29.650 INFO:tasks.rados.rados.0.smithi045.stdout:579: finishing rollback tid 1 to smithi04569269-36 2021-07-31T01:47:29.651 INFO:tasks.rados.rados.0.smithi045.stdout:579: finishing rollback tid 2 to smithi04569269-36 2021-07-31T01:47:29.651 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 36 v 404 (ObjNum 129 snap 27 seq_num 129) dirty exists 2021-07-31T01:47:29.651 INFO:tasks.rados.rados.0.smithi045.stdout:573: done (5 left) 2021-07-31T01:47:29.651 INFO:tasks.rados.rados.0.smithi045.stdout:574: done (4 left) 2021-07-31T01:47:29.652 INFO:tasks.rados.rados.0.smithi045.stdout:576: done (3 left) 2021-07-31T01:47:29.652 INFO:tasks.rados.rados.0.smithi045.stdout:577: done (2 left) 2021-07-31T01:47:29.652 INFO:tasks.rados.rados.0.smithi045.stdout:578: done (1 left) 2021-07-31T01:47:29.653 INFO:tasks.rados.rados.0.smithi045.stdout:579: done (0 left) 2021-07-31T01:47:29.653 INFO:tasks.rados.rados.0.smithi045.stdout:580: copy_from oid 5 from oid 40 current snap is 49 2021-07-31T01:47:29.653 INFO:tasks.rados.rados.0.smithi045.stdout:581: rollback oid 46 current snap is 49 2021-07-31T01:47:29.653 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 46 to 47 2021-07-31T01:47:29.653 INFO:tasks.rados.rados.0.smithi045.stdout:582: delete oid 50 current snap is 49 2021-07-31T01:47:29.654 INFO:tasks.rados.rados.0.smithi045.stdout:581: finishing rollback tid 1 to smithi04569269-46 2021-07-31T01:47:29.654 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 46 v 0 (ObjNum 0 snap 0 seq_num 0) dirty dne 2021-07-31T01:47:29.654 INFO:tasks.rados.rados.0.smithi045.stdout:581: done (2 left) 2021-07-31T01:47:29.654 INFO:tasks.rados.rados.0.smithi045.stdout:582: done (1 left) 2021-07-31T01:47:29.655 INFO:tasks.rados.rados.0.smithi045.stdout:583: delete oid 41 current snap is 49 2021-07-31T01:47:29.656 INFO:tasks.rados.rados.0.smithi045.stdout:583: done (1 left) 2021-07-31T01:47:29.656 INFO:tasks.rados.rados.0.smithi045.stdout:584: read oid 2 snap -1 2021-07-31T01:47:29.656 INFO:tasks.rados.rados.0.smithi045.stdout:584: expect deleted 2021-07-31T01:47:29.656 INFO:tasks.rados.rados.0.smithi045.stdout:585: rollback oid 3 current snap is 49 2021-07-31T01:47:29.657 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 3 to 41 2021-07-31T01:47:29.658 INFO:tasks.rados.rados.0.smithi045.stdout:586: delete oid 28 current snap is 49 2021-07-31T01:47:29.658 INFO:tasks.rados.rados.0.smithi045.stdout:580: finishing copy_from to smithi04569269-5 2021-07-31T01:47:29.658 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 5 v 143 (ObjNum 158 snap 37 seq_num 158) dirty exists 2021-07-31T01:47:29.660 INFO:tasks.rados.rados.0.smithi045.stdout:580: finishing copy_from racing read to smithi04569269-5 2021-07-31T01:47:29.660 INFO:tasks.rados.rados.0.smithi045.stdout:580: done (3 left) 2021-07-31T01:47:29.660 INFO:tasks.rados.rados.0.smithi045.stdout:584: done (2 left) 2021-07-31T01:47:29.660 INFO:tasks.rados.rados.0.smithi045.stdout:586: done (1 left) 2021-07-31T01:47:29.661 INFO:tasks.rados.rados.0.smithi045.stdout:587: read oid 6 snap -1 2021-07-31T01:47:29.661 INFO:tasks.rados.rados.0.smithi045.stdout:587: expect (ObjNum 0 snap 0 seq_num 0) 2021-07-31T01:47:29.661 INFO:tasks.rados.rados.0.smithi045.stdout:588: delete oid 12 current snap is 49 2021-07-31T01:47:29.661 INFO:tasks.rados.rados.0.smithi045.stdout:587: expect (ObjNum 0 snap 0 seq_num 0) 2021-07-31T01:47:29.662 INFO:tasks.rados.rados.0.smithi045.stdout:587: done (2 left) 2021-07-31T01:47:29.662 INFO:tasks.rados.rados.0.smithi045.stdout:588: done (1 left) 2021-07-31T01:47:29.662 INFO:tasks.rados.rados.0.smithi045.stdout:589: read oid 31 snap -1 2021-07-31T01:47:29.663 INFO:tasks.rados.rados.0.smithi045.stdout:589: expect (ObjNum 157 snap 37 seq_num 157) 2021-07-31T01:47:29.663 INFO:tasks.rados.rados.0.smithi045.stdout:590: snap_create 2021-07-31T01:47:29.663 INFO:tasks.rados.rados.0.smithi045.stdout:585: finishing rollback tid 0 to smithi04569269-3 2021-07-31T01:47:29.663 INFO:tasks.rados.rados.0.smithi045.stdout:585: finishing rollback tid 1 to smithi04569269-3 2021-07-31T01:47:29.664 INFO:tasks.rados.rados.0.smithi045.stdout:585: finishing rollback tid 2 to smithi04569269-3 2021-07-31T01:47:29.664 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 3 v 407 (ObjNum 144 snap 31 seq_num 144) dirty exists 2021-07-31T01:47:29.664 INFO:tasks.rados.rados.0.smithi045.stdout:589: expect (ObjNum 157 snap 37 seq_num 157) 2021-07-31T01:47:30.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: cephadm 2021-07-31T01:47:28.303399+0000 mgr.x (mgr.24542) 1 : cephadm [ERR] 2021-07-31T01:47:30.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:47:30.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:47:30.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: yield 2021-07-31T01:47:30.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:47:30.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:47:30.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:47:30.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T01:47:30.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:47:30.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T01:47:30.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:47:30.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: await self._waiter 2021-07-31T01:47:30.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:47:30.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: task.result() 2021-07-31T01:47:30.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T01:47:30.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: cephadm 2021-07-31T01:47:28.303956+0000 mgr.x (mgr.24542) 2 : cephadm [INF] Updating smithi045:/etc/ceph/ceph.conf 2021-07-31T01:47:30.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: cephadm 2021-07-31T01:47:28.305248+0000 mgr.x (mgr.24542) 3 : cephadm [ERR] 2021-07-31T01:47:30.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:47:30.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:47:30.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: yield 2021-07-31T01:47:30.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:47:30.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:47:30.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:47:30.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T01:47:30.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:47:30.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T01:47:30.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:47:30.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: await self._waiter 2021-07-31T01:47:30.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:47:30.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: task.result() 2021-07-31T01:47:30.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T01:47:30.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: cephadm 2021-07-31T01:47:28.314982+0000 mgr.x (mgr.24542) 4 : cephadm [INF] Updating smithi154:/etc/ceph/ceph.conf 2021-07-31T01:47:30.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: cephadm 2021-07-31T01:47:28.339314+0000 mgr.x (mgr.24542) 5 : cephadm [ERR] 2021-07-31T01:47:30.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:47:30.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:47:30.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: yield 2021-07-31T01:47:30.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:47:30.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:47:30.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:47:30.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T01:47:30.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:47:30.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T01:47:30.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:47:30.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: await self._waiter 2021-07-31T01:47:30.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:47:30.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: task.result() 2021-07-31T01:47:30.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T01:47:30.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: cephadm 2021-07-31T01:47:28.339665+0000 mgr.x (mgr.24542) 6 : cephadm [ERR] Unable to write smithi045:/etc/ceph/ceph.conf: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T01:47:30.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:47:30.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:47:30.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: yield 2021-07-31T01:47:30.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:47:30.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:47:30.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:47:30.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T01:47:30.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:47:30.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T01:47:30.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:47:30.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: await self._waiter 2021-07-31T01:47:30.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:47:30.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: task.result() 2021-07-31T01:47:30.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T01:47:30.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: 2021-07-31T01:47:30.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T01:47:30.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: 2021-07-31T01:47:30.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:47:30.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 175, in _write_remote_file 2021-07-31T01:47:30.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: await self._check_execute_command(host, ['mkdir', '-p', dirname], addr=addr) 2021-07-31T01:47:30.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:47:30.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:47:30.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:47:30.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T01:47:30.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:47:30.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: raise 2021-07-31T01:47:30.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:47:30.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T01:47:30.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:47:30.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:47:30.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T01:47:30.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: cephadm 2021-07-31T01:47:28.340968+0000 mgr.x (mgr.24542) 7 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:47:30.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:47:30.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:47:30.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: yield 2021-07-31T01:47:30.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:47:30.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:47:30.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:47:30.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T01:47:30.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:47:30.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T01:47:30.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:47:30.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: await self._waiter 2021-07-31T01:47:30.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:47:30.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: task.result() 2021-07-31T01:47:30.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T01:47:30.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: 2021-07-31T01:47:30.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T01:47:30.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: 2021-07-31T01:47:30.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:47:30.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 175, in _write_remote_file 2021-07-31T01:47:30.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: await self._check_execute_command(host, ['mkdir', '-p', dirname], addr=addr) 2021-07-31T01:47:30.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:47:30.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:47:30.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:47:30.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T01:47:30.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:47:30.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: raise 2021-07-31T01:47:30.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:47:30.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T01:47:30.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:47:30.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:47:30.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T01:47:30.285 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: 2021-07-31T01:47:30.285 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T01:47:30.285 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: 2021-07-31T01:47:30.285 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:47:30.285 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:47:30.286 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: return f(*arg) 2021-07-31T01:47:30.286 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 295, in refresh 2021-07-31T01:47:30.286 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: self.mgr.ssh.write_remote_file(host, path, content, mode, uid, gid) 2021-07-31T01:47:30.286 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in write_remote_file 2021-07-31T01:47:30.287 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: self.mgr.event_loop.get_result(self._write_remote_file(*args, **kwargs)) 2021-07-31T01:47:30.287 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:47:30.287 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:47:30.288 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:47:30.288 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T01:47:30.288 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:47:30.288 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: raise self._exception 2021-07-31T01:47:30.288 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 187, in _write_remote_file 2021-07-31T01:47:30.289 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: raise OrchestratorError(msg) 2021-07-31T01:47:30.289 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Unable to write smithi045:/etc/ceph/ceph.conf: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T01:47:30.289 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: cephadm 2021-07-31T01:47:28.347166+0000 mgr.x (mgr.24542) 8 : cephadm [ERR] 2021-07-31T01:47:30.289 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:47:30.290 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_l 2021-07-31T01:47:30.290 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: og 2021-07-31T01:47:30.290 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: yield 2021-07-31T01:47:30.290 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:47:30.290 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:47:30.291 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:47:30.291 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T01:47:30.291 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:47:30.291 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T01:47:30.292 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:47:30.292 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: await self._waiter 2021-07-31T01:47:30.292 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:47:30.292 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: task.result() 2021-07-31T01:47:30.292 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T01:47:30.293 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: cephadm 2021-07-31T01:47:28.347367+0000 mgr.x (mgr.24542) 9 : cephadm [ERR] Unable to write smithi154:/etc/ceph/ceph.conf: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T01:47:30.293 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:47:30.293 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:47:30.293 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: yield 2021-07-31T01:47:30.293 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:47:30.294 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:47:30.294 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:47:30.294 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T01:47:30.294 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:47:30.295 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T01:47:30.295 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:47:30.295 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: await self._waiter 2021-07-31T01:47:30.296 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: cephadm 2021-07-31T01:47:28.303399+0000 mgr.x (mgr.24542) 1 : cephadm [ERR] 2021-07-31T01:47:30.296 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:47:30.296 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:47:30.297 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: yield 2021-07-31T01:47:30.297 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:47:30.297 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:47:30.297 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:47:30.298 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T01:47:30.298 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:47:30.298 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T01:47:30.298 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:47:30.299 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: await self._waiter 2021-07-31T01:47:30.299 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:47:30.299 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: task.result() 2021-07-31T01:47:30.299 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T01:47:30.299 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: cephadm 2021-07-31T01:47:30.300 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: 2021-07-31T01:47:28.303956+0000 mgr.x (mgr.24542) 2 : cephadm [INF] Updating smithi045:/etc/ceph/ceph.conf 2021-07-31T01:47:30.300 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: cephadm 2021-07-31T01:47:28.305248+0000 mgr.x (mgr.24542) 3 : cephadm [ERR] 2021-07-31T01:47:30.300 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:47:30.300 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:47:30.300 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: yield 2021-07-31T01:47:30.301 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:47:30.301 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:47:30.301 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:47:30.301 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T01:47:30.301 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:47:30.302 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T01:47:30.302 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:47:30.302 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: await self._waiter 2021-07-31T01:47:30.302 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:47:30.303 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: task.result() 2021-07-31T01:47:30.303 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T01:47:30.303 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: cephadm 2021-07-31T01:47:28.314982+0000 mgr.x (mgr.24542) 4 : cephadm [INF] Updating smithi154:/etc/ceph/ceph.conf 2021-07-31T01:47:30.303 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: cephadm 2021-07-31T01:47:28.339314+0000 mgr.x (mgr.24542) 5 : cephadm [ERR] 2021-07-31T01:47:30.303 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:47:30.304 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:47:30.304 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: yield 2021-07-31T01:47:30.304 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:47:30.304 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:47:30.305 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: cephadm 2021-07-31T01:47:28. 2021-07-31T01:47:30.305 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: 303399+0000 mgr.x (mgr.24542) 1 : cephadm [ERR] 2021-07-31T01:47:30.305 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:47:30.306 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:47:30.306 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: yield 2021-07-31T01:47:30.306 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:47:30.306 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:47:30.306 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:47:30.307 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T01:47:30.307 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:47:30.307 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T01:47:30.307 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:47:30.308 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: await self._waiter 2021-07-31T01:47:30.308 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:47:30.308 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: task.result() 2021-07-31T01:47:30.308 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T01:47:30.308 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: cephadm 2021-07-31T01:47:28.303956+0000 mgr.x (mgr.24542) 2 : cephadm [INF] Updating smithi045:/etc/ceph/ceph.conf 2021-07-31T01:47:30.309 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: cephadm 2021-07-31T01:47:28.305248+0000 mgr.x ( 2021-07-31T01:47:30.309 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: mgr.24542) 3 : cephadm [ERR] 2021-07-31T01:47:30.309 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:47:30.309 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:47:30.309 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: yield 2021-07-31T01:47:30.310 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:47:30.310 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:47:30.310 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:47:30.310 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T01:47:30.311 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:47:30.311 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T01:47:30.311 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:47:30.311 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: await self._waiter 2021-07-31T01:47:30.311 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:47:30.312 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: task.result() 2021-07-31T01:47:30.312 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T01:47:30.312 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: cephadm 2021-07-31T01:47:28.314982+0000 mgr.x (mgr.24542) 4 : cephadm [INF] Updating smithi154:/etc/ceph/ceph.conf 2021-07-31T01:47:30.312 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: cephadm 2021-07-31T01:47:30.312 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: 2021-07-31 2021-07-31T01:47:30.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: T01:47:28.339314+0000 2021-07-31T01:47:30.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: mgr.x (mgr.24542) 5 : cephadm 2021-07-31T01:47:30.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: [ERR] 2021-07-31T01:47:30.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:47:30.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:47:30.314 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: yield 2021-07-31T01:47:30.315 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:47:30.319 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T01:47:30.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:47:30.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T01:47:30.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:47:30.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: await self._waiter 2021-07-31T01:47:30.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:47:30.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: task.result() 2021-07-31T01:47:30.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T01:47:30.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: cephadm 2021-07-31T01:47:30.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: 2021-07-31T01:47:28.339665+0000 mgr.x (mgr.24542) 6 : cephadm [ERR] Unable to write smithi045:/etc/ceph/ceph.conf: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T01:47:30.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:47:30.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:47:30.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: yield 2021-07-31T01:47:30.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:47:30.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:47:30.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:47:30.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T01:47:30.325 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:47:30.325 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T01:47:30.325 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:47:30.325 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: await self._waiter 2021-07-31T01:47:30.325 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:47:30.326 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: task.result() 2021-07-31T01:47:30.326 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T01:47:30.326 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: 2021-07-31T01:47:30.326 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T01:47:30.326 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: 2021-07-31T01:47:30.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:47:30.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 175, in _write_remote_file 2021-07-31T01:47:30.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: await self._check_execute_command(host, ['mkdir', '-p', dirname], addr=addr) 2021-07-31T01:47:30.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:47:30.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:47:30.328 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:47:30.328 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T01:47:30.328 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:47:30.328 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: raise 2021-07-31T01:47:30.329 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:47:30.329 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T01:47:30.329 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:47:30.329 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:47:30.330 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:47:30.330 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: task.result() 2021-07-31T01:47:30.330 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T01:47:30.330 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: 2021-07-31T01:47:30.331 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T01:47:30.331 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: 2021-07-31T01:47:30.331 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:47:30.331 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 175, in _write_remote_file 2021-07-31T01:47:30.331 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: await self._check_execute_command(host, ['mkdir', '-p', dirname], addr=addr) 2021-07-31T01:47:30.332 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:47:30.332 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:47:30.332 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:47:30.332 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T01:47:30.332 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:47:30.333 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: raise 2021-07-31T01:47:30.333 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:47:30.333 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T01:47:30.333 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:47:30.333 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:47:30.334 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T01:47:30.334 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: cephadm 2021-07-31T01:47:28.347707+0000 mgr.x (mgr.24542) 10 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:47:30.334 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:47:30.334 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:47:30.335 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: yield 2021-07-31T01:47:30.335 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:47:30.335 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:47:30.335 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:47:30.335 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T01:47:30.336 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:47:30.336 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T01:47:30.336 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:47:30.336 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: await self._waiter 2021-07-31T01:47:30.336 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:47:30.337 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: task.result() 2021-07-31T01:47:30.337 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T01:47:30.337 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: 2021-07-31T01:47:30.338 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T01:47:30.338 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: 2021-07-31T01:47:30.338 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:47:30.339 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 175, in _write_remote_file 2021-07-31T01:47:30.339 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: await self._check_execute_command(host, ['mkdir', '-p', dirname], addr=addr) 2021-07-31T01:47:30.339 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:47:30.339 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:47:30.339 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:47:30.340 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T01:47:30.340 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:47:30.340 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: raise 2021-07-31T01:47:30.340 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:47:30.341 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T01:47:30.341 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:47:30.341 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:47:30.341 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T01:47:30.341 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: 2021-07-31T01:47:30.342 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T01:47:30.342 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: 2021-07-31T01:47:30.342 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:47:30.342 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:47:30.342 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: return f(*arg) 2021-07-31T01:47:30.343 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 295, in refresh 2021-07-31T01:47:30.343 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: self.mgr.ssh.write_remote_file(host, path, content, mode, uid, gid) 2021-07-31T01:47:30.343 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in write_remote_file 2021-07-31T01:47:30.343 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: self.mgr.event_loop.get_result(self._write_remote_file(*args, **kwargs)) 2021-07-31T01:47:30.344 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:47:30.344 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:47:30.344 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:47:30.344 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T01:47:30.344 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:47:30.345 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: raise self._exception 2021-07-31T01:47:30.345 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 187, in _write_remote_file 2021-07-31T01:47:30.345 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: raise OrchestratorError(msg) 2021-07-31T01:47:30.345 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Unable to write smithi154:/etc/ceph/ceph.conf: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T01:47:30.345 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: cephadm 2021-07-31T01:47:28.350733+0000 mgr.x (mgr.24542) 11 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T01:47:30.346 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: cephadm 2021-07-31T01:47:28.351185+0000 mgr.x (mgr.24542) 12 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T01:47:30.346 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: cephadm 2021-07-31T01:47:28.351424+0000 mgr.x (mgr.24542) 13 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T01:47:30.346 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: cephadm 2021-07-31T01:47:28.391370+0000 mgr.x (mgr.24542) 14 : cephadm [ERR] 2021-07-31T01:47:30.346 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:47:30.346 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:47:30.347 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: yield 2021-07-31T01:47:30.347 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:47:30.347 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:47:30.347 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:47:30.348 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T01:47:30.348 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:47:30.348 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T01:47:30.348 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:47:30.348 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: await self._waiter 2021-07-31T01:47:30.349 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:47:30.349 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: task.result() 2021-07-31T01:47:30.349 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T01:47:30.349 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: cephadm 2021-07-31T01:47:28.391779+0000 mgr.x (mgr.24542) 15 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:47:30.349 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:47:30.350 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:47:30.350 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: yield 2021-07-31T01:47:30.350 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:47:30.350 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:47:30.350 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:47:30.351 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T01:47:30.351 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:47:30.351 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T01:47:30.351 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:47:30.352 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: await self._waiter 2021-07-31T01:47:30.352 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:47:30.352 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: task.result() 2021-07-31T01:47:30.352 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T01:47:30.352 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: 2021-07-31T01:47:30.353 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: During handling of the above exception, another exception occu 2021-07-31T01:47:30.353 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: rred: 2021-07-31T01:47:30.353 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: 2021-07-31T01:47:30.353 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:47:30.353 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:47:30.354 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: return f(*arg) 2021-07-31T01:47:30.354 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:47:30.354 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:47:30.354 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:47:30.354 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:47:30.355 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:47:30.355 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:47:30.355 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:47:30.355 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T01:47:30.356 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:47:30.356 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: raise self._exception 2021-07-31T01:47:30.356 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:47:30.356 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:47:30.356 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:47:30.357 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T01:47:30.357 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:47:30.357 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: raise 2021-07-31T01:47:30.357 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:47:30.357 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T01:47:30.358 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:47:30.358 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:47:30.358 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T01:47:30.358 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: cephadm 2021-07-31T01:47:28.393295+0000 mgr.x (mgr.24542) 16 : cephadm [ERR] 2021-07-31T01:47:30.358 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:47:30.359 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:47:30.359 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: yield 2021-07-31T01:47:30.359 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:47:30.359 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:47:30.359 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:47:30.360 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T01:47:30.360 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:47:30.360 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T01:47:30.360 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:47:30.361 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: await self._waiter 2021-07-31T01:47:30.361 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:47:30.361 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: task.result() 2021-07-31T01:47:30.361 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T01:47:30.361 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: cephadm 2021-07-31T01:47:28.393701+0000 mgr.x (mgr.24542) 17 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:47:30.362 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:47:30.362 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:47:30.362 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: yield 2021-07-31T01:47:30.362 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:47:30.362 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:47:30.363 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:47:30.363 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T01:47:30.363 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:47:30.363 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T01:47:30.363 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:47:30.364 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: await self._waiter 2021-07-31T01:47:30.364 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:47:30.364 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: task.result() 2021-07-31T01:47:30.364 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T01:47:30.365 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: 2021-07-31T01:47:30.365 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T01:47:30.365 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: 2021-07-31T01:47:30.365 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:47:30.365 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:47:30.366 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: return f(*arg) 2021-07-31T01:47:30.366 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:47:30.366 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:47:30.366 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:47:30.366 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:47:30.367 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:47:30.367 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:47:30.367 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:47:30.367 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T01:47:30.367 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:47:30.368 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: raise self._exception 2021-07-31T01:47:30.368 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:47:30.368 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:47:30.368 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:47:30.368 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T01:47:30.369 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:47:30.369 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: raise 2021-07-31T01:47:30.369 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:47:30.369 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T01:47:30.370 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:47:30.370 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:47:30.370 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T01:47:30.370 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: cluster 2021-07-31T01:47:28.807231+0000 mon.a (mon.0) 776 : cluster [DBG] mgrmap e16: x(active, since 1.30281s), standbys: y 2021-07-31T01:47:30.370 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: cluster 2021-07-31T01:47:28.817824+0000 mgr.x (mgr.24542) 18 : cluster [DBG] pgmap v4: 25 pgs: 25 active+clean; 96 MiB data, 367 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:47:30.371 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: audit 2021-07-31T01:47:29.510312+0000 mon.a (mon.0) 777 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2021-07-31T01:47:30.371 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:29 smithi154 conmon[34553]: cluster 2021-07-31T01:47:29.510397+0000 mon.a (mon.0) 778 : cluster [DBG] osdmap e151: 8 total, 8 up, 8 in 2021-07-31T01:47:30.371 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:47:30.371 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:47:30.372 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:47:30.372 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T01:47:30.372 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:47:30.372 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T01:47:30.372 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:47:30.373 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: await self._waiter 2021-07-31T01:47:30.373 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:47:30.373 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: task.result() 2021-07-31T01:47:30.373 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T01:47:30.373 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: cephadm 2021-07-31T01:47:28.339665+0000 mgr.x (mgr.24542) 6 : cephadm [ERR] Unable to write smithi045:/etc/ceph/ceph.conf: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T01:47:30.374 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:47:30.374 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:47:30.374 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: yield 2021-07-31T01:47:30.374 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:47:30.375 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:47:30.375 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:47:30.375 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T01:47:30.375 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:47:30.375 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T01:47:30.376 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:47:30.376 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: await self._waiter 2021-07-31T01:47:30.376 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:47:30.376 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: task.result() 2021-07-31T01:47:30.376 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T01:47:30.377 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: 2021-07-31T01:47:30.377 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T01:47:30.377 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: 2021-07-31T01:47:30.377 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:47:30.377 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 175, in _write_remote_file 2021-07-31T01:47:30.378 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: await self._check_execute_command(host, ['mkdir', '-p', dirname], addr=addr) 2021-07-31T01:47:30.378 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:47:30.378 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:47:30.378 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:47:30.378 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T01:47:30.379 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:47:30.379 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: raise 2021-07-31T01:47:30.379 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:47:30.379 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T01:47:30.380 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T01:47:30.380 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: cephadm 2021-07-31T01:47:28.340968+0000 mgr.x (mgr 2021-07-31T01:47:30.380 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: .24542) 7 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:47:30.381 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:47:30.381 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:47:30.381 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: yield 2021-07-31T01:47:30.381 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:47:30.381 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:47:30.382 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:47:30.382 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T01:47:30.382 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:47:30.382 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T01:47:30.383 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:47:30.383 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: await self._waiter 2021-07-31T01:47:30.383 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:47:30.383 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: task.result() 2021-07-31T01:47:30.383 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T01:47:30.384 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: 2021-07-31T01:47:30.384 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T01:47:30.384 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: 2021-07-31T01:47:30.384 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:47:30.384 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 175, in _write_remote_file 2021-07-31T01:47:30.385 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: await self._check_execute_command(host, ['mkdir', '-p', dirname], addr=addr) 2021-07-31T01:47:30.385 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:47:30.385 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:47:30.385 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:47:30.385 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T01:47:30.386 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:47:30.386 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: raise 2021-07-31T01:47:30.386 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:47:30.386 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T01:47:30.386 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:47:30.387 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:47:30.387 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T01:47:30.387 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: 2021-07-31T01:47:30.387 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T01:47:30.388 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: 2021-07-31T01:47:30.388 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:47:30.388 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:47:30.388 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: return f(*arg) 2021-07-31T01:47:30.388 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 295, in refresh 2021-07-31T01:47:30.389 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: self.mgr.ssh.write_remote_file(host, path, content, mode, uid, gid) 2021-07-31T01:47:30.389 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:47:30.389 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:47:30.389 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T01:47:30.390 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: cephadm 2021-07-31T01:47:28.340968+0000 mgr.x (mgr.24542) 7 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:47:30.390 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:47:30.390 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:47:30.390 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: yield 2021-07-31T01:47:30.391 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:47:30.391 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:47:30.391 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:47:30.391 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T01:47:30.391 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:47:30.392 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T01:47:30.392 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:47:30.392 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: await self._waiter 2021-07-31T01:47:30.392 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:47:30.392 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: task.result() 2021-07-31T01:47:30.393 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T01:47:30.393 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: 2021-07-31T01:47:30.393 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T01:47:30.393 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: 2021-07-31T01:47:30.393 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:47:30.394 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 175, in _write_remote_file 2021-07-31T01:47:30.394 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: await self._check_execute_command(host, ['mkdir', '-p', dirname], addr=addr) 2021-07-31T01:47:30.394 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:47:30.394 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:47:30.395 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:47:30.395 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T01:47:30.395 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:47:30.395 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: raise 2021-07-31T01:47:30.395 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:47:30.396 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T01:47:30.396 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:47:30.396 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:47:30.396 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T01:47:30.396 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: 2021-07-31T01:47:30.397 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T01:47:30.397 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: 2021-07-31T01:47:30.397 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:47:30.397 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:47:30.397 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: return f(*arg) 2021-07-31T01:47:30.398 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 295, in refresh 2021-07-31T01:47:30.398 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in write_remote_file 2021-07-31T01:47:30.398 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: self.mgr.event_loop.get_result(self._write_remote_file(*args, **kwargs)) 2021-07-31T01:47:30.399 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:47:30.399 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:47:30.399 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:47:30.399 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T01:47:30.400 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:47:30.400 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: raise self._exception 2021-07-31T01:47:30.400 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 187, in _write_remote_file 2021-07-31T01:47:30.400 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: raise OrchestratorError(msg) 2021-07-31T01:47:30.400 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Unable to write smithi045:/etc/ceph/ceph.conf: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T01:47:30.401 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: cephadm 2021-07-31T01:47:28.347166+0000 mgr.x (mgr.24542) 8 : cephadm [ERR] 2021-07-31T01:47:30.401 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:47:30.401 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:47:30.401 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: yield 2021-07-31T01:47:30.401 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:47:30.402 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:47:30.402 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:47:30.402 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T01:47:30.402 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:47:30.403 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T01:47:30.403 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:47:30.403 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: await self._waiter 2021-07-31T01:47:30.403 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:47:30.403 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: task.result() 2021-07-31T01:47:30.404 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T01:47:30.404 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: cephadm 2021-07-31T01:47:28.347367+0000 mgr.x (mgr.24542) 9 : cephadm [ERR] Unable to write smithi154:/etc/ceph/ceph.conf: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T01:47:30.404 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:47:30.404 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:47:30.404 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: yield 2021-07-31T01:47:30.405 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:47:30.405 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:47:30.405 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:47:30.405 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T01:47:30.405 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:47:30.406 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T01:47:30.406 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:47:30.406 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: await self._waiter 2021-07-31T01:47:30.407 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: self.mgr.ssh.write_remote_file(host, path, content, mode, uid, gid) 2021-07-31T01:47:30.407 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in write_remote_file 2021-07-31T01:47:30.407 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: self.mgr.event_loop.get_result(self._write_remote_file(*args, **kwargs)) 2021-07-31T01:47:30.407 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:47:30.407 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:47:30.408 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:47:30.408 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T01:47:30.408 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:47:30.408 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: raise self._exception 2021-07-31T01:47:30.409 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 187, in _write_remote_file 2021-07-31T01:47:30.409 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: raise OrchestratorError(msg) 2021-07-31T01:47:30.409 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Unable to write smithi045:/etc/ceph/ceph.conf: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T01:47:30.409 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: cephadm 2021-07-31T01:47:28.347166+0000 mgr.x (mgr.24542) 8 : cephadm [ERR] 2021-07-31T01:47:30.409 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:47:30.410 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:47:30.410 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: yield 2021-07-31T01:47:30.410 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:47:30.410 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:47:30.410 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:47:30.411 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T01:47:30.411 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:47:30.411 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T01:47:30.411 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:47:30.411 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: await self._waiter 2021-07-31T01:47:30.412 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:47:30.412 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: task.result() 2021-07-31T01:47:30.412 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T01:47:30.412 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: cephadm 2021-07- 2021-07-31T01:47:30.413 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: 31T01:47:28.347367+0000 mgr.x (mgr.24542) 9 : cephadm [ERR] Unable to write smithi154:/etc/ceph/ceph.conf: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T01:47:30.413 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:47:30.413 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:47:30.413 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: yield 2021-07-31T01:47:30.413 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:47:30.414 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:47:30.414 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:47:30.414 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T01:47:30.414 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:47:30.414 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T01:47:30.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:47:30.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: task.result() 2021-07-31T01:47:30.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T01:47:30.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: 2021-07-31T01:47:30.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T01:47:30.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: 2021-07-31T01:47:30.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:47:30.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 175, in _write_remote_file 2021-07-31T01:47:30.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: await self._check_execute_command(host, ['mkdir', '-p', dirname], addr=addr) 2021-07-31T01:47:30.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:47:30.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:47:30.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:47:30.418 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T01:47:30.418 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:47:30.418 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: raise 2021-07-31T01:47:30.418 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:47:30.418 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T01:47:30.419 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:47:30.419 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:47:30.419 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T01:47:30.419 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: cephadm 2021-07-31T01:47:28.347707+0000 mgr.x 2021-07-31T01:47:30.419 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: (mgr.24542) 10 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:47:30.420 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:47:30.420 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:47:30.420 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: yield 2021-07-31T01:47:30.420 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:47:30.420 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:47:30.421 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:47:30.421 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T01:47:30.421 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:47:30.421 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T01:47:30.422 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:47:30.422 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: await self._waiter 2021-07-31T01:47:30.422 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:47:30.422 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: task.result() 2021-07-31T01:47:30.422 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T01:47:30.423 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: 2021-07-31T01:47:30.423 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T01:47:30.423 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: 2021-07-31T01:47:30.423 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:47:30.423 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 175, in _write_remote_file 2021-07-31T01:47:30.424 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: await self._check_execute_command(host, ['mkdir', '-p', dirname], addr=addr) 2021-07-31T01:47:30.424 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:47:30.424 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: await self._waiter 2021-07-31T01:47:30.424 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:47:30.425 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: task.result() 2021-07-31T01:47:30.425 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T01:47:30.425 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: 2021-07-31T01:47:30.425 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T01:47:30.426 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: 2021-07-31T01:47:30.426 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:47:30.426 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 175, in _write_remote_file 2021-07-31T01:47:30.426 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: await self._check_execute_command(host, ['mkdir', '-p', dirname], addr=addr) 2021-07-31T01:47:30.426 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:47:30.427 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:47:30.427 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:47:30.427 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T01:47:30.427 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:47:30.427 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: raise 2021-07-31T01:47:30.428 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:47:30.428 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T01:47:30.428 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:47:30.428 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:47:30.428 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T01:47:30.429 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: cephadm 2021-07-31T01:47:28.347707+0000 mgr.x (mgr.24542) 10 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:47:30.429 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:47:30.429 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:47:30.429 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: yield 2021-07-31T01:47:30.430 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:47:30.430 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:47:30.430 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:47:30.430 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T01:47:30.430 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:47:30.431 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T01:47:30.431 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:47:30.431 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: await self._waiter 2021-07-31T01:47:30.431 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:47:30.431 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: task.result() 2021-07-31T01:47:30.432 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T01:47:30.432 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: 2021-07-31T01:47:30.432 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T01:47:30.432 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: 2021-07-31T01:47:30.432 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:47:30.433 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 175, in _write_remote_file 2021-07-31T01:47:30.433 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:47:30.433 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:47:30.434 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:47:30.434 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T01:47:30.434 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:47:30.434 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: raise 2021-07-31T01:47:30.434 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:47:30.435 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T01:47:30.435 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:47:30.435 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:47:30.435 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T01:47:30.436 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: 2021-07-31T01:47:30.436 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T01:47:30.436 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: 2021-07-31T01:47:30.436 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:47:30.436 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:47:30.437 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: return f(*arg) 2021-07-31T01:47:30.437 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 295, in refresh 2021-07-31T01:47:30.437 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: self.mgr.ssh.write_remote_file(host, path, content, mode, uid, gid) 2021-07-31T01:47:30.437 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in write_remote_file 2021-07-31T01:47:30.437 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: self.mgr.event_loop.get_result(self._write_remote_file(*args, **kwargs)) 2021-07-31T01:47:30.438 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:47:30.438 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:47:30.438 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:47:30.439 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T01:47:30.439 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:47:30.439 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: raise self._exception 2021-07-31T01:47:30.439 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 187, in _write_remote_file 2021-07-31T01:47:30.439 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: raise OrchestratorError(msg) 2021-07-31T01:47:30.440 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Unable to write smithi154:/etc/ceph/ceph.conf: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T01:47:30.440 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: cephadm 2021-07-31T01:47:28.350733+0000 mgr.x (mgr.24542) 11 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T01:47:30.440 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: cephadm 2021-07-31T01:47:28.351185+0000 mgr.x (mgr.24542) 12 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T01:47:30.440 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: cephadm 2021-07-31T01:47:28.351424+0000 mgr.x (mgr.24542) 13 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T01:47:30.441 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: cephadm 2021-07-31T01:47:28.391370+0000 mgr.x (mgr.24542) 14 : cephadm [ERR] 2021-07-31T01:47:30.441 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:47:30.441 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:47:30.441 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: yield 2021-07-31T01:47:30.441 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:47:30.442 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: await self._check_execute_command(host, ['mkdir', '-p', dirname], addr=addr) 2021-07-31T01:47:30.442 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:47:30.442 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:47:30.443 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:47:30.443 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T01:47:30.443 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:47:30.443 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: raise 2021-07-31T01:47:30.444 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:47:30.444 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T01:47:30.444 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:47:30.444 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:47:30.444 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T01:47:30.445 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: 2021-07-31T01:47:30.445 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T01:47:30.445 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: 2021-07-31T01:47:30.445 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:47:30.446 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:47:30.446 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: return f(*arg) 2021-07-31T01:47:30.446 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 295, in refresh 2021-07-31T01:47:30.446 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: self.mgr.ssh.write_remote_file(host, path, content, mode, uid, gid) 2021-07-31T01:47:30.446 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in write_remote_file 2021-07-31T01:47:30.447 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: self.mgr.event_loop.get_result(self._write_remote_file(*args, **kwargs)) 2021-07-31T01:47:30.447 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:47:30.447 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:47:30.447 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:47:30.448 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T01:47:30.448 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:47:30.448 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: raise self._exception 2021-07-31T01:47:30.448 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 187, in _write_remote_file 2021-07-31T01:47:30.448 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: raise OrchestratorError(msg) 2021-07-31T01:47:30.449 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Unable to write smithi154:/etc/ceph/ceph.conf: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T01:47:30.449 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: cephadm 2021-07-31T01:47:28.350733+0000 mgr.x (mgr.24542) 11 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T01:47:30.449 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: cephadm 2021-07-31T01:47:28.351185+0000 mgr.x (mgr.24542) 12 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T01:47:30.449 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: cephadm 2021 2021-07-31T01:47:30.450 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: -07-31T01:47:28.351424+0000 mgr.x (mgr.24542) 13 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T01:47:30.450 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: cephadm 2021-07-31T01:47:28.391370+0000 mgr.x (mgr.24542) 14 : cephadm [ERR] 2021-07-31T01:47:30.450 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:47:30.450 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:47:30.451 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:47:30.451 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:47:30.451 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T01:47:30.451 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:47:30.452 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T01:47:30.452 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:47:30.452 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: await self._waiter 2021-07-31T01:47:30.452 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:47:30.452 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: task.result() 2021-07-31T01:47:30.453 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T01:47:30.453 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: cephadm 2021-07-31T01:47:28.391779+0000 mgr.x (mgr.24542) 15 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:47:30.453 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:47:30.453 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:47:30.454 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: yield 2021-07-31T01:47:30.454 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:47:30.454 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:47:30.454 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:47:30.454 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T01:47:30.455 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:47:30.455 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T01:47:30.455 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:47:30.455 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: await self._waiter 2021-07-31T01:47:30.456 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:47:30.456 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: task.result() 2021-07-31T01:47:30.456 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T01:47:30.456 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: 2021-07-31T01:47:30.456 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T01:47:30.457 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: 2021-07-31T01:47:30.457 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:47:30.457 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:47:30.457 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: return f(*arg) 2021-07-31T01:47:30.457 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:47:30.458 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:47:30.458 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:47:30.458 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:47:30.458 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:47:30.459 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:47:30.459 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:47:30.459 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T01:47:30.459 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:47:30.460 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: yield 2021-07-31T01:47:30.460 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:47:30.460 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:47:30.460 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:47:30.461 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T01:47:30.461 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:47:30.461 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T01:47:30.461 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:47:30.461 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: await self._waiter 2021-07-31T01:47:30.462 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:47:30.462 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: task.result() 2021-07-31T01:47:30.462 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T01:47:30.462 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: cephadm 2021-07-31T01:47:28.391779+0000 mgr.x (mgr.24542) 15 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:47:30.462 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:47:30.463 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:47:30.463 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: yield 2021-07-31T01:47:30.463 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:47:30.463 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:47:30.464 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:47:30.464 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T01:47:30.464 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:47:30.464 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T01:47:30.464 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:47:30.465 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: await self._waiter 2021-07-31T01:47:30.465 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:47:30.465 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: task.result() 2021-07-31T01:47:30.465 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T01:47:30.465 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: 2021-07-31T01:47:30.466 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T01:47:30.466 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: 2021-07-31T01:47:30.466 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:47:30.466 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:47:30.466 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: return f(*arg) 2021-07-31T01:47:30.467 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:47:30.467 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:47:30.467 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:47:30.467 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:47:30.468 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:47:30.468 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:47:30.468 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:47:30.468 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T01:47:30.469 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: raise self._exception 2021-07-31T01:47:30.469 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:47:30.469 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:47:30.469 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:47:30.469 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T01:47:30.470 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:47:30.470 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: raise 2021-07-31T01:47:30.470 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:47:30.470 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T01:47:30.471 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:47:30.471 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:47:30.471 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T01:47:30.471 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: cephadm 2021-07-31T01:47:28.393295+0000 mgr.x (mgr.24542) 16 : cephadm [ERR] 2021-07-31T01:47:30.471 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:47:30.472 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:47:30.472 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: yield 2021-07-31T01:47:30.472 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:47:30.472 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:47:30.473 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:47:30.473 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T01:47:30.473 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:47:30.473 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T01:47:30.473 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:47:30.474 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: await self._waiter 2021-07-31T01:47:30.474 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:47:30.474 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: task.result() 2021-07-31T01:47:30.474 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T01:47:30.475 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: cephadm 2021-07-31T01:47:28.393701+0000 mgr.x (mgr.24542) 17 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:47:30.475 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:47:30.475 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:47:30.475 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: yield 2021-07-31T01:47:30.475 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:47:30.476 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=sel 2021-07-31T01:47:30.476 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: f.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:47:30.476 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:47:30.476 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T01:47:30.477 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:47:30.477 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T01:47:30.477 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:47:30.477 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: await self._waiter 2021-07-31T01:47:30.478 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:47:30.478 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: raise self._exception 2021-07-31T01:47:30.478 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:47:30.478 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:47:30.478 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:47:30.479 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T01:47:30.479 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:47:30.479 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: raise 2021-07-31T01:47:30.479 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:47:30.480 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T01:47:30.480 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:47:30.480 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:47:30.480 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T01:47:30.480 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: cephadm 2021-07-31T01:47:28.393295+0000 mgr.x (mgr.24542) 16 : cephadm [ERR] 2021-07-31T01:47:30.481 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:47:30.481 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:47:30.481 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: yield 2021-07-31T01:47:30.481 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:47:30.481 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:47:30.482 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:47:30.482 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T01:47:30.482 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:47:30.482 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T01:47:30.483 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:47:30.483 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: await self._waiter 2021-07-31T01:47:30.483 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:47:30.483 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: task.result() 2021-07-31T01:47:30.483 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T01:47:30.484 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: cephadm 2021-07-31T01:47:28.393701+0000 mgr.x (mgr.24542) 17 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:47:30.484 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:47:30.484 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:47:30.484 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: yield 2021-07-31T01:47:30.484 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:47:30.485 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:47:30.485 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:47:30.485 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T01:47:30.485 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:47:30.485 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T01:47:30.486 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:47:30.486 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: task.result() 2021-07-31T01:47:30.486 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T01:47:30.487 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: 2021-07-31T01:47:30.487 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T01:47:30.487 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: 2021-07-31T01:47:30.487 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:47:30.488 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:47:30.488 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: return f(*arg) 2021-07-31T01:47:30.488 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:47:30.488 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:47:30.488 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:47:30.489 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:47:30.489 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:47:30.489 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:47:30.489 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:47:30.489 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T01:47:30.490 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:47:30.490 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: raise self._exception 2021-07-31T01:47:30.490 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:47:30.490 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:47:30.490 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:47:30.491 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T01:47:30.491 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:47:30.491 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: raise 2021-07-31T01:47:30.491 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:47:30.492 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T01:47:30.492 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:47:30.492 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:47:30.492 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T01:47:30.492 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: cluster 2021-07-31T01:47:28.807231+0000 mon.a (mon.0) 776 : cluster [DBG] mgrmap e16: x(active, since 1.30281s), standbys: y 2021-07-31T01:47:30.493 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: cluster 2021-07-31T01:47:28.817824+0000 mgr.x (mgr.24542) 18 : cluster [DBG] pgmap v4: 25 pgs: 25 active+clean; 96 MiB data, 367 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:47:30.493 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: audit 2021-07-31T01:47:29.510312+0000 mon.a (mon.0) 777 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2021-07-31T01:47:30.493 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[33655]: cluster 2021-07-31T01:47:29.510397+0000 mon.a (mon.0) 778 : cluster [DBG] osdmap e151: 8 total, 8 up, 8 in 2021-07-31T01:47:30.493 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:47:30.494 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: await self._waiter 2021-07-31T01:47:30.494 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:47:30.494 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: task.result() 2021-07-31T01:47:30.494 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T01:47:30.494 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: 2021-07-31T01:47:30.495 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T01:47:30.495 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: 2021-07-31T01:47:30.495 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:47:30.495 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:47:30.495 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: return f(*arg) 2021-07-31T01:47:30.496 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:47:30.496 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:47:30.496 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:47:30.496 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:47:30.497 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:47:30.497 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:47:30.497 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:47:30.497 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T01:47:30.497 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:47:30.498 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: raise self._exception 2021-07-31T01:47:30.498 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:47:30.498 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:47:30.498 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:47:30.498 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T01:47:30.499 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:47:30.499 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: raise 2021-07-31T01:47:30.499 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:47:30.499 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T01:47:30.500 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:47:30.500 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:47:30.500 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T01:47:30.500 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: cluster 2021-07-31T01:47:28.807231+0000 mon.a (mon.0) 776 : cluster [DBG] mgrmap e16: x(active, since 1.30281s), standbys: y 2021-07-31T01:47:30.500 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: cluster 2021-07-31T01:47:28.817824+0000 mgr.x (mgr.24542) 18 : cluster [DBG] pgmap v4: 25 pgs: 25 active+clean; 96 MiB data, 367 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:47:30.501 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: audit 2021-07-31T01:47:29.510312+0000 mon.a (mon.0) 777 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2021-07-31T01:47:30.501 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: cluster 2021-07- 2021-07-31T01:47:30.501 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:29 smithi045 conmon[39718]: 31T01:47:29.510397+0000 mon.a (mon.0) 778 : cluster [DBG] osdmap e151: 8 total, 8 up, 8 in 2021-07-31T01:47:30.521 INFO:tasks.rados.rados.0.smithi045.stdout:585: done (2 left) 2021-07-31T01:47:30.522 INFO:tasks.rados.rados.0.smithi045.stdout:589: done (1 left) 2021-07-31T01:47:30.522 INFO:tasks.rados.rados.0.smithi045.stdout:590: done (0 left) 2021-07-31T01:47:30.522 INFO:tasks.rados.rados.0.smithi045.stdout:591: snap_remove snap 47 2021-07-31T01:47:31.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:30 smithi154 conmon[34553]: cluster 2021-07-31T01:47:29.794578+0000 mgr.x (mgr.24542) 19 : cluster [DBG] pgmap v6: 25 pgs: 25 active+clean; 96 MiB data, 367 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:47:31.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:30 smithi154 conmon[34553]: cluster 2021-07-31T01:47:30. 2021-07-31T01:47:31.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:30 smithi154 conmon[34553]: 514247+0000 mon.a (mon.0) 779 : cluster [DBG] mgrmap e17: x(active, since 3s), standbys: y 2021-07-31T01:47:31.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:30 smithi154 conmon[34553]: cluster 2021-07-31T01:47:30.520722+0000 mon.a (mon.0) 780 : cluster [DBG] osdmap e152: 8 total, 8 up, 8 in 2021-07-31T01:47:31.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:30 smithi154 conmon[34553]: audit 2021-07-31T01:47:30.531888+0000 mon.a (mon.0) 781 : audit [INF] 2021-07-31T01:47:31.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:30 smithi154 conmon[34553]: from='mgr.24542 ' entity='mgr.x' 2021-07-31T01:47:31.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:30 smithi045 conmon[39718]: cluster 2021-07-31T01: 2021-07-31T01:47:31.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:30 smithi045 conmon[39718]: 47:29.794578+0000 mgr.x (mgr.24542) 19 : cluster [DBG] pgmap v6: 25 pgs: 25 active+clean; 96 MiB data, 367 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:47:31.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:30 smithi045 conmon[39718]: cluster 2021-07-31T01:47:31.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:30 smithi045 conmon[39718]: 2021-07-31 2021-07-31T01:47:31.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:30 smithi045 conmon[39718]: T01:47:30.514247+0000 2021-07-31T01:47:31.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:30 smithi045 conmon[39718]: mon.a (mon.0) 779 : cluster [DBG] 2021-07-31T01:47:31.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:30 smithi045 conmon[39718]: mgrmap e17: x(active, since 3s), standbys: y 2021-07-31T01:47:31.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:30 smithi045 conmon[39718]: cluster 2021-07 2021-07-31T01:47:31.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:30 smithi045 conmon[39718]: -31T01:47:30.520722+0000 2021-07-31T01:47:31.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:30 smithi045 conmon[39718]: mon.a (mon.0) 780 : cluster 2021-07-31T01:47:31.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:30 smithi045 conmon[39718]: [DBG] osdmap e152: 8 total, 8 up, 8 in 2021-07-31T01:47:31.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:30 smithi045 conmon[39718]: audit 2021-07 2021-07-31T01:47:31.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:30 smithi045 conmon[39718]: -31T01:47:30.531888 2021-07-31T01:47:31.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:30 smithi045 conmon[39718]: +0000 mon.a (mon.0) 781 : 2021-07-31T01:47:31.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:30 smithi045 conmon[39718]: audit [INF] from='mgr.24542 ' entity='mgr.x' 2021-07-31T01:47:31.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:30 smithi045 conmon[33655]: cluster 2021-07-31T01:47:31.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:30 smithi045 conmon[33655]: 2021-07-31T01:47 2021-07-31T01:47:31.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:30 smithi045 conmon[33655]: :29.794578+0000 mgr.x (mgr.24542) 19 : cluster [DBG] pgmap v6: 25 pgs: 25 active+clean; 96 MiB data, 367 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:47:31.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:30 smithi045 conmon[33655]: cluster 2021- 2021-07-31T01:47:31.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:30 smithi045 conmon[33655]: 07-31T01:47:30.514247+0000 mon.a 2021-07-31T01:47:31.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:30 smithi045 conmon[33655]: (mon.0) 779 : cluster [DBG] mgrmap e17: x(active, since 3s), standbys: y 2021-07-31T01:47:31.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:30 smithi045 conmon[33655]: cluster 2021-07-31T01:47:31.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:30 smithi045 conmon[33655]: 2021-07-31T01:47:30.520722+0000 mon.a ( 2021-07-31T01:47:31.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:30 smithi045 conmon[33655]: mon.0) 780 : cluster [DBG] osdmap e152: 8 total, 8 up, 8 in 2021-07-31T01:47:31.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:30 smithi045 conmon[33655]: audit 2021-07 2021-07-31T01:47:31.283 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:30 smithi045 conmon[33655]: -31T01:47:30.531888+0000 mon.a (mon 2021-07-31T01:47:31.283 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:30 smithi045 conmon[33655]: .0) 781 : audit [INF] from='mgr.24542 ' entity='mgr.x' 2021-07-31T01:47:31.490 INFO:tasks.rados.rados.0.smithi045.stdout:591: done (0 left) 2021-07-31T01:47:31.490 INFO:tasks.rados.rados.0.smithi045.stdout:592: read oid 10 snap -1 2021-07-31T01:47:31.496 INFO:tasks.rados.rados.0.smithi045.stdout:592: expect (ObjNum 186 snap 49 seq_num 186) 2021-07-31T01:47:31.496 INFO:tasks.rados.rados.0.smithi045.stdout:593: delete oid 17 current snap is 50 2021-07-31T01:47:31.507 INFO:tasks.rados.rados.0.smithi045.stdout:592: expect (ObjNum 186 snap 49 seq_num 186) 2021-07-31T01:47:31.608 INFO:tasks.rados.rados.0.smithi045.stdout:592: done (1 left) 2021-07-31T01:47:31.609 INFO:tasks.rados.rados.0.smithi045.stdout:593: done (0 left) 2021-07-31T01:47:31.609 INFO:tasks.rados.rados.0.smithi045.stdout:594: rollback oid 44 current snap is 50 2021-07-31T01:47:31.609 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 44 to 40 2021-07-31T01:47:31.610 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 30 current snap is 50 2021-07-31T01:47:31.610 INFO:tasks.rados.rados.0.smithi045.stdout:595: seq_num 194 ranges {491520=458752,950272=401408} 2021-07-31T01:47:31.610 INFO:tasks.rados.rados.0.smithi045.stdout:595: writing smithi04569269-30 from 491520 to 950272 tid 1 2021-07-31T01:47:31.613 INFO:tasks.rados.rados.0.smithi045.stdout:595: writing smithi04569269-30 from 950272 to 1351680 tid 2 2021-07-31T01:47:31.613 INFO:tasks.rados.rados.0.smithi045.stdout:596: rollback oid 40 current snap is 50 2021-07-31T01:47:31.613 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 40 to 49 2021-07-31T01:47:31.614 INFO:tasks.rados.rados.0.smithi045.stdout:594: finishing rollback tid 0 to smithi04569269-44 2021-07-31T01:47:31.614 INFO:tasks.rados.rados.0.smithi045.stdout:594: finishing rollback tid 1 to smithi04569269-44 2021-07-31T01:47:31.614 INFO:tasks.rados.rados.0.smithi045.stdout:594: finishing rollback tid 2 to smithi04569269-44 2021-07-31T01:47:31.614 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 44 v 184 (ObjNum 142 snap 30 seq_num 142) dirty exists 2021-07-31T01:47:31.615 INFO:tasks.rados.rados.0.smithi045.stdout:594: done (2 left) 2021-07-31T01:47:31.615 INFO:tasks.rados.rados.0.smithi045.stdout:597: snap_remove snap 41 2021-07-31T01:47:32.510 INFO:tasks.rados.rados.0.smithi045.stdout:597: done (2 left) 2021-07-31T01:47:32.511 INFO:tasks.rados.rados.0.smithi045.stdout:598: snap_create 2021-07-31T01:47:32.511 INFO:tasks.rados.rados.0.smithi045.stdout:595: finishing write tid 1 to smithi04569269-30 2021-07-31T01:47:32.511 INFO:tasks.rados.rados.0.smithi045.stdout:595: finishing write tid 2 to smithi04569269-30 2021-07-31T01:47:32.512 INFO:tasks.rados.rados.0.smithi045.stdout:595: finishing write tid 3 to smithi04569269-30 2021-07-31T01:47:32.512 INFO:tasks.rados.rados.0.smithi045.stdout:596: finishing rollback tid 0 to smithi04569269-40 2021-07-31T01:47:32.512 INFO:tasks.rados.rados.0.smithi045.stdout:596: finishing rollback tid 1 to smithi04569269-40 2021-07-31T01:47:32.512 INFO:tasks.rados.rados.0.smithi045.stdout:596: finishing rollback tid 2 to smithi04569269-40 2021-07-31T01:47:32.512 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 40 v 188 (ObjNum 158 snap 37 seq_num 158) dirty exists 2021-07-31T01:47:32.513 INFO:tasks.rados.rados.0.smithi045.stdout:595: finishing write tid 4 to smithi04569269-30 2021-07-31T01:47:32.513 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 30 v 93 (ObjNum 194 snap 50 seq_num 194) dirty exists 2021-07-31T01:47:32.513 INFO:tasks.rados.rados.0.smithi045.stdout:595: left oid 30 (ObjNum 194 snap 50 seq_num 194) 2021-07-31T01:47:32.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:32 smithi154 conmon[34553]: cluster 2021-07-31T01:47:31.489457+0000 mon.a (mon.0) 782 : cluster [DBG] osdmap e153: 8 total, 8 up, 8 in 2021-07-31T01:47:32.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:32 smithi154 conmon[34553]: cluster 2021-07-31T01:47:31.795194+0000 mgr.x (mgr.24542) 20 : cluster [DBG] pgmap v9: 25 pgs: 25 active+clean; 96 MiB data, 367 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:47:32.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:32 smithi045 conmon[33655]: cluster 2021-07-31T01:47:31.489457+0000 mon.a (mon.0) 782 : cluster [DBG] osdmap e153: 8 total, 8 up, 8 in 2021-07-31T01:47:32.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:32 smithi045 conmon[33655]: cluster 2021-07-31T01:47:31.795194+0000 mgr.x (mgr.24542) 20 : cluster [DBG] pgmap v9: 25 pgs: 25 active+clean; 96 MiB data, 367 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:47:32.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:32 smithi045 conmon[39718]: cluster 2021-07-31T01:47:31.489457+0000 mon.a 2021-07-31T01:47:32.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:32 smithi045 conmon[39718]: (mon.0) 782 : cluster [DBG] osdmap e153: 8 total, 8 up, 8 in 2021-07-31T01:47:32.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:32 smithi045 conmon[39718]: cluster 2021-07-31T01:47:31. 2021-07-31T01:47:32.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:32 smithi045 conmon[39718]: 795194+0000 mgr.x (mgr.24542) 20 : cluster [DBG] pgmap v9: 25 pgs: 25 active+clean; 96 MiB data, 367 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:47:33.509 INFO:tasks.rados.rados.0.smithi045.stdout:595: done (2 left) 2021-07-31T01:47:33.510 INFO:tasks.rados.rados.0.smithi045.stdout:596: done (1 left) 2021-07-31T01:47:33.510 INFO:tasks.rados.rados.0.smithi045.stdout:598: done (0 left) 2021-07-31T01:47:33.510 INFO:tasks.rados.rados.0.smithi045.stdout:599: rmattr oid 40 current snap is 51 2021-07-31T01:47:33.510 INFO:tasks.rados.rados.0.smithi045.stdout:600: read oid 48 snap 39 2021-07-31T01:47:33.511 INFO:tasks.rados.rados.0.smithi045.stdout:600: expect (ObjNum 157 snap 37 seq_num 157) 2021-07-31T01:47:33.511 INFO:tasks.rados.rados.0.smithi045.stdout:601: snap_create 2021-07-31T01:47:33.516 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 40 v 198 (ObjNum 158 snap 37 seq_num 158) dirty exists 2021-07-31T01:47:33.520 INFO:tasks.rados.rados.0.smithi045.stdout:600: expect (ObjNum 157 snap 37 seq_num 157) 2021-07-31T01:47:33.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:33 smithi154 conmon[34553]: cluster 2021-07-31T01:47:32.503493+0000 mon.a (mon.0) 783 : 2021-07-31T01:47:33.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:33 smithi154 conmon[34553]: cluster [DBG] osdmap e154: 8 total, 8 up, 8 in 2021-07-31T01:47:33.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:33 smithi154 conmon[34553]: cluster 2021-07-31T01:47:32.504058+0000 mon.a (mon.0) 784 : cluster [DBG] mgrmap e18: x(active, since 4s), standbys: y 2021-07-31T01:47:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:33 smithi154 conmon[34553]: cluster 2021-07-31T01:47:33.151528+0000 mon.a (mon.0 2021-07-31T01:47:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:33 smithi154 conmon[34553]: ) 785 : cluster [DBG] Standby manager daemon y restarted 2021-07-31T01:47:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:33 smithi154 conmon[34553]: cluster 2021-07 2021-07-31T01:47:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:33 smithi154 conmon[34553]: -31T01:47:33.151624+0000 mon.a 2021-07-31T01:47:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:33 smithi154 conmon[34553]: (mon.0) 786 : cluster [DBG] Standby manager daemon y started 2021-07-31T01:47:33.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:33 smithi154 conmon[34553]: audit 2021 2021-07-31T01:47:33.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:33 smithi154 conmon[34553]: -07-31T01:47:33.152403+0000 2021-07-31T01:47:33.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:33 smithi154 conmon[34553]: mon.c (mon.1) 22 : audit [DBG] 2021-07-31T01:47:33.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:33 smithi154 conmon[34553]: from='mgr.? 172.21.15.45:0/1241761794' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2021-07-31T01:47:33.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:33 smithi154 conmon[34553]: audit 2021-07-31T01:47: 2021-07-31T01:47:33.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:33 smithi154 conmon[34553]: 33.154511+0000 mon.c (mon.1) 23 : 2021-07-31T01:47:33.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:33 smithi154 conmon[34553]: audit [DBG] from='mgr.? 172.21.15.45:0/1241761794' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2021-07-31T01:47:33.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:33 smithi154 conmon[34553]: audit 2021-07-31T01:47:33 2021-07-31T01:47:33.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:33 smithi154 conmon[34553]: .155627+0000 mon.c (mon.1) 24 : 2021-07-31T01:47:33.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:33 smithi154 conmon[34553]: audit [DBG] from='mgr.? 172.21.15.45:0/1241761794' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2021-07-31T01:47:33.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:33 smithi154 conmon[34553]: audit 2021-07-31T 2021-07-31T01:47:33.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:33 smithi154 conmon[34553]: 01:47:33.156049+0000 mon.c (mon. 2021-07-31T01:47:33.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:33 smithi154 conmon[34553]: 1) 25 : audit [DBG] from='mgr.? 172.21.15.45:0/1241761794' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2021-07-31T01:47:33.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:47:33 smithi045 conmon[61210]: level=error ts=2021-07-31T01:47:33.508Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:47:33.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:47:33 smithi045 conmon[61210]: level=error ts=2021-07-31T01:47:33.508Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: dial tcp 172.21.15.45:8443: connect: connection refused" context_err="context deadline exceeded" 2021-07-31T01:47:33.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:47:33 smithi045 conmon[61210]: level=error ts=2021-07-31T01:47:33.508Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: dial tcp 172.21.15.45:8443: connect: connection refused; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T01:47:33.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:33 smithi045 conmon[39718]: cluster 2021-07-31T01:47:32.503493+0000 mon.a (mon 2021-07-31T01:47:33.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:33 smithi045 conmon[39718]: .0) 783 : cluster [DBG] osdmap e154: 8 total, 8 up, 8 in 2021-07-31T01:47:33.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:33 smithi045 conmon[39718]: cluster 2021-07-31T01 2021-07-31T01:47:33.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:33 smithi045 conmon[39718]: :47:32.504058+0000 mon.a (mon.0) 784 : cluster [DBG] mgrmap e18: x(active, since 4s), standbys: y 2021-07-31T01:47:33.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:33 smithi045 conmon[39718]: cluster 2021- 2021-07-31T01:47:33.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:33 smithi045 conmon[39718]: 07-31T01:47:33.151528+0000 mon.a (mon.0) 785 2021-07-31T01:47:33.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:33 smithi045 conmon[39718]: : cluster [DBG] Standby manager daemon y restarted 2021-07-31T01:47:33.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:33 smithi045 conmon[39718]: cluster 2021- 2021-07-31T01:47:33.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:33 smithi045 conmon[39718]: 07-31T01:47:33.151624+0000 mon.a ( 2021-07-31T01:47:33.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:33 smithi045 conmon[39718]: mon.0) 786 : cluster [DBG] Standby manager daemon y started 2021-07-31T01:47:33.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:33 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T01:47:33.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:33 smithi045 conmon[39718]: T01:47:33.152403+0000 mon.c (mon.1) 22 : audit 2021-07-31T01:47:33.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:33 smithi045 conmon[39718]: [DBG] from='mgr.? 172.21.15.45:0/1241761794' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2021-07-31T01:47:33.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:33 smithi045 conmon[39718]: audit 2021- 2021-07-31T01:47:33.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:33 smithi045 conmon[39718]: 07-31T01:47:33.154511+0000 mon.c 2021-07-31T01:47:33.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:33 smithi045 conmon[39718]: (mon.1) 23 : audit [DBG] from='mgr.? 172.21.15.45:0/1241761794' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2021-07-31T01:47:33.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:33 smithi045 conmon[39718]: 2021-07-31T01:47:33.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:33 smithi045 conmon[39718]: audit 2021-07-31T01:47:33 2021-07-31T01:47:33.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:33 smithi045 conmon[39718]: .155627+0000 mon.c (mon.1) 24 : 2021-07-31T01:47:33.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:33 smithi045 conmon[39718]: audit [DBG] from='mgr.? 172.21.15.45:0/1241761794' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2021-07-31T01:47:33.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:33 smithi045 conmon[39718]: audit 2021 2021-07-31T01:47:33.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:33 smithi045 conmon[39718]: -07-31T01:47:33.156049+0000 mon.c 2021-07-31T01:47:33.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:33 smithi045 conmon[39718]: (mon.1) 25 : audit [DBG] from='mgr.? 172.21.15.45:0/1241761794' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2021-07-31T01:47:33.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:33 smithi045 conmon[33655]: cluster 2021-07-31T01:47:32.503493+0000 mon.a 2021-07-31T01:47:33.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:33 smithi045 conmon[33655]: (mon.0) 783 : cluster [DBG] osdmap e154: 8 total, 8 up, 8 in 2021-07-31T01:47:33.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:33 smithi045 conmon[33655]: cluster 2021-07-31T01:47:32.504058+0000 mon.a (mon.0 2021-07-31T01:47:33.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:33 smithi045 conmon[33655]: ) 784 : cluster [DBG] mgrmap e18: x(active, since 4s), standbys: y 2021-07-31T01:47:33.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:33 smithi045 conmon[33655]: cluster 2021-07-31T01:47:33.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:33 smithi045 conmon[33655]: 2021-07-31T 2021-07-31T01:47:33.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:33 smithi045 conmon[33655]: 01:47:33.151528+0000 mon.a (mon. 2021-07-31T01:47:33.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:33 smithi045 conmon[33655]: 0) 785 : cluster [DBG] Standby manager daemon y restarted 2021-07-31T01:47:33.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:33 smithi045 conmon[33655]: cluster 2021-07-31T01:47: 2021-07-31T01:47:33.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:33 smithi045 conmon[33655]: 33.151624+0000 mon.a (mon.0) 786 : cluster [DBG] Standby manager daemon y started 2021-07-31T01:47:33.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:33 smithi045 conmon[33655]: audit 2021-07-31T01:47: 2021-07-31T01:47:33.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:33 smithi045 conmon[33655]: 33.152403+0000 mon.c (mon.1) 22 : audit [DBG] 2021-07-31T01:47:33.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:33 smithi045 conmon[33655]: from='mgr.? 172.21.15.45:0/1241761794' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2021-07-31T01:47:33.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:33 smithi045 conmon[33655]: audit 2021-07- 2021-07-31T01:47:33.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:33 smithi045 conmon[33655]: 31T01:47:33.154511+0000 mon.c (mon.1) 23 : audit [DBG] from='mgr.? 172.21.15.45:0/1241761794' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2021-07-31T01:47:33.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:33 smithi045 conmon[33655]: audit 2021-07-31T01 2021-07-31T01:47:33.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:33 smithi045 conmon[33655]: :47:33.155627+0000 mon.c (mon.1) 24 : audit [DBG] 2021-07-31T01:47:33.788 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:33 smithi045 conmon[33655]: from='mgr.? 172.21.15.45:0/1241761794' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2021-07-31T01:47:33.788 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:33 smithi045 conmon[33655]: audit 2021-07-31T 2021-07-31T01:47:33.788 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:33 smithi045 conmon[33655]: 01:47:33.156049+0000 mon.c (mon.1) 25 : audit [DBG] from='mgr.? 172.21.15.45:0/1241761794' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2021-07-31T01:47:34.510 INFO:tasks.rados.rados.0.smithi045.stdout:599: done (2 left) 2021-07-31T01:47:34.510 INFO:tasks.rados.rados.0.smithi045.stdout:600: done (1 left) 2021-07-31T01:47:34.510 INFO:tasks.rados.rados.0.smithi045.stdout:601: done (0 left) 2021-07-31T01:47:34.510 INFO:tasks.rados.rados.0.smithi045.stdout:602: snap_remove snap 50 2021-07-31T01:47:34.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:34 smithi045 conmon[39718]: cluster 2021-07-31T01:47:33.508837+0000 2021-07-31T01:47:34.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:34 smithi045 conmon[39718]: mon.a (mon.0) 787 : cluster [DBG] osdmap e155: 8 total, 8 up, 8 in 2021-07-31T01:47:34.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:34 smithi045 conmon[39718]: cluster 2021- 2021-07-31T01:47:34.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:34 smithi045 conmon[39718]: 07-31T01:47:33.509381+0000 mon.a (mon. 2021-07-31T01:47:34.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:34 smithi045 conmon[39718]: 0) 788 : cluster [DBG] mgrmap e19: x(active, since 6s), standbys: y 2021-07-31T01:47:34.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:34 smithi045 conmon[39718]: audit 2021 2021-07-31T01:47:34.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:34 smithi045 conmon[39718]: -07-31T01:47:33. 2021-07-31T01:47:34.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:34 smithi045 conmon[39718]: 753228+0000 mon.a (mon.0) 789 : 2021-07-31T01:47:34.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:34 smithi045 conmon[39718]: audit [INF] from='mgr.24542 ' entity='mgr.x' 2021-07-31T01:47:34.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:34 smithi045 conmon[39718]: cluster 2021-07- 2021-07-31T01:47:34.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:34 smithi045 conmon[39718]: 31T01:47:33.795565+0000 mgr.x (mgr 2021-07-31T01:47:34.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:34 smithi045 conmon[39718]: .24542) 21 : cluster [DBG] pgmap v12: 25 pgs: 25 active+clean; 96 MiB data, 367 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:47:34.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:34 smithi045 conmon[33655]: cluster 2021-07-31T01:47:33.508837+0000 mon.a (mon.0) 787 : cluster [DBG] osdmap e155: 8 total, 8 up, 8 in 2021-07-31T01:47:34.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:34 smithi045 conmon[33655]: cluster 2021-07-31T01:47:33.509381+0000 mon.a (mon.0) 788 : cluster [DBG] mgrmap e19: x(active, since 6s), standbys: y 2021-07-31T01:47:34.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:34 smithi045 conmon[33655]: audit 2021-07-31T01: 2021-07-31T01:47:34.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:34 smithi045 conmon[33655]: 47:33.753228+0000 mon.a (mon 2021-07-31T01:47:34.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:34 smithi045 conmon[33655]: .0) 789 : audit [INF] 2021-07-31T01:47:34.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:34 smithi045 conmon[33655]: from='mgr.24542 ' entity='mgr.x' 2021-07-31T01:47:34.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:34 smithi045 conmon[33655]: cluster 2021-07-31 2021-07-31T01:47:34.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:34 smithi045 conmon[33655]: T01:47:33.795565+0000 2021-07-31T01:47:34.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:34 smithi045 conmon[33655]: mgr.x (mgr.24542) 21 : cluster 2021-07-31T01:47:34.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:34 smithi045 conmon[33655]: [DBG] pgmap v12: 25 pgs: 25 active+clean; 96 MiB data, 367 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:47:35.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:34 smithi154 conmon[34553]: cluster 2021-07-31T01:47:33.508837+0000 mon.a (mon.0 2021-07-31T01:47:35.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:34 smithi154 conmon[34553]: ) 787 : cluster [DBG] osdmap e155: 8 total, 8 up, 8 in 2021-07-31T01:47:35.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:34 smithi154 conmon[34553]: cluster 2021-07-31T01:47:33.509381+0000 mon.a (mon.0) 788 : cluster [DBG] mgrmap e19: x(active, since 6s), standbys: y 2021-07-31T01:47:35.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:34 smithi154 conmon[34553]: audit 2021-07-31T01:47:33 2021-07-31T01:47:35.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:34 smithi154 conmon[34553]: .753228+0000 mon.a (mon.0) 789 : audit 2021-07-31T01:47:35.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:34 smithi154 conmon[34553]: [INF] from='mgr.24542 ' entity='mgr.x' 2021-07-31T01:47:35.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:34 smithi154 conmon[34553]: cluster 2021-07-31 2021-07-31T01:47:35.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:34 smithi154 conmon[34553]: T01:47:33.795565+0000 mgr.x (mgr 2021-07-31T01:47:35.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:34 smithi154 conmon[34553]: .24542) 21 : cluster [DBG] pgmap v12: 25 pgs: 25 active+clean; 96 MiB data, 367 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:47:37.036 INFO:tasks.rados.rados.0.smithi045.stdout:602: done (0 left) 2021-07-31T01:47:37.037 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 17 current snap is 52 2021-07-31T01:47:37.037 INFO:tasks.rados.rados.0.smithi045.stdout:603: seq_num 195 ranges {0=475136,475136=401408} 2021-07-31T01:47:37.039 INFO:tasks.rados.rados.0.smithi045.stdout:603: writing smithi04569269-17 from 0 to 475136 tid 1 2021-07-31T01:47:37.041 INFO:tasks.rados.rados.0.smithi045.stdout:603: writing smithi04569269-17 from 475136 to 876544 tid 2 2021-07-31T01:47:37.042 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 38 current snap is 52 2021-07-31T01:47:37.042 INFO:tasks.rados.rados.0.smithi045.stdout:604: seq_num 196 ranges {4046848=491520,4538368=401408} 2021-07-31T01:47:37.066 INFO:tasks.rados.rados.0.smithi045.stdout:604: writing smithi04569269-38 from 4046848 to 4538368 tid 1 2021-07-31T01:47:37.068 INFO:tasks.rados.rados.0.smithi045.stdout:604: writing smithi04569269-38 from 4538368 to 4939776 tid 2 2021-07-31T01:47:37.069 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 39 current snap is 52 2021-07-31T01:47:37.069 INFO:tasks.rados.rados.0.smithi045.stdout:605: seq_num 197 ranges {1638400=507904,2146304=401408} 2021-07-31T01:47:37.081 INFO:tasks.rados.rados.0.smithi045.stdout:605: writing smithi04569269-39 from 1638400 to 2146304 tid 1 2021-07-31T01:47:37.088 INFO:tasks.rados.rados.0.smithi045.stdout:605: writing smithi04569269-39 from 2146304 to 2547712 tid 2 2021-07-31T01:47:37.088 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 22 current snap is 52 2021-07-31T01:47:37.089 INFO:tasks.rados.rados.0.smithi045.stdout:603: finishing write tid 1 to smithi04569269-17 2021-07-31T01:47:37.089 INFO:tasks.rados.rados.0.smithi045.stdout:606: seq_num 198 ranges {1081344=524288,1605632=401408} 2021-07-31T01:47:37.092 INFO:tasks.rados.rados.0.smithi045.stdout:606: writing smithi04569269-22 from 1081344 to 1605632 tid 1 2021-07-31T01:47:37.095 INFO:tasks.rados.rados.0.smithi045.stdout:606: writing smithi04569269-22 from 1605632 to 2007040 tid 2 2021-07-31T01:47:37.095 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 10 current snap is 52 2021-07-31T01:47:37.095 INFO:tasks.rados.rados.0.smithi045.stdout:603: finishing write tid 2 to smithi04569269-17 2021-07-31T01:47:37.095 INFO:tasks.rados.rados.0.smithi045.stdout:603: finishing write tid 3 to smithi04569269-17 2021-07-31T01:47:37.096 INFO:tasks.rados.rados.0.smithi045.stdout:603: finishing write tid 4 to smithi04569269-17 2021-07-31T01:47:37.096 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 17 v 273 (ObjNum 195 snap 52 seq_num 195) dirty exists 2021-07-31T01:47:37.096 INFO:tasks.rados.rados.0.smithi045.stdout:603: left oid 17 (ObjNum 195 snap 52 seq_num 195) 2021-07-31T01:47:37.096 INFO:tasks.rados.rados.0.smithi045.stdout:604: finishing write tid 1 to smithi04569269-38 2021-07-31T01:47:37.096 INFO:tasks.rados.rados.0.smithi045.stdout:604: finishing write tid 2 to smithi04569269-38 2021-07-31T01:47:37.097 INFO:tasks.rados.rados.0.smithi045.stdout:607: seq_num 199 ranges {2514944=540672,3055616=401408} 2021-07-31T01:47:37.111 INFO:tasks.rados.rados.0.smithi045.stdout:607: writing smithi04569269-10 from 2514944 to 3055616 tid 1 2021-07-31T01:47:37.114 INFO:tasks.rados.rados.0.smithi045.stdout:607: writing smithi04569269-10 from 3055616 to 3457024 tid 2 2021-07-31T01:47:37.114 INFO:tasks.rados.rados.0.smithi045.stdout:603: done (4 left) 2021-07-31T01:47:37.114 INFO:tasks.rados.rados.0.smithi045.stdout:608: setattr oid 29 current snap is 52 2021-07-31T01:47:37.115 INFO:tasks.rados.rados.0.smithi045.stdout:604: finishing write tid 3 to smithi04569269-38 2021-07-31T01:47:37.115 INFO:tasks.rados.rados.0.smithi045.stdout:604: finishing write tid 4 to smithi04569269-38 2021-07-31T01:47:37.115 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 38 v 244 (ObjNum 196 snap 52 seq_num 196) dirty exists 2021-07-31T01:47:37.115 INFO:tasks.rados.rados.0.smithi045.stdout:604: left oid 38 (ObjNum 196 snap 52 seq_num 196) 2021-07-31T01:47:37.116 INFO:tasks.rados.rados.0.smithi045.stdout:605: finishing write tid 1 to smithi04569269-39 2021-07-31T01:47:37.116 INFO:tasks.rados.rados.0.smithi045.stdout:605: finishing write tid 2 to smithi04569269-39 2021-07-31T01:47:37.116 INFO:tasks.rados.rados.0.smithi045.stdout:605: finishing write tid 3 to smithi04569269-39 2021-07-31T01:47:37.116 INFO:tasks.rados.rados.0.smithi045.stdout:605: finishing write tid 4 to smithi04569269-39 2021-07-31T01:47:37.117 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 39 v 248 (ObjNum 197 snap 52 seq_num 197) dirty exists 2021-07-31T01:47:37.117 INFO:tasks.rados.rados.0.smithi045.stdout:605: left oid 39 (ObjNum 197 snap 52 seq_num 197) 2021-07-31T01:47:37.117 INFO:tasks.rados.rados.0.smithi045.stdout:606: finishing write tid 1 to smithi04569269-22 2021-07-31T01:47:37.117 INFO:tasks.rados.rados.0.smithi045.stdout:606: finishing write tid 2 to smithi04569269-22 2021-07-31T01:47:37.118 INFO:tasks.rados.rados.0.smithi045.stdout:606: finishing write tid 3 to smithi04569269-22 2021-07-31T01:47:37.118 INFO:tasks.rados.rados.0.smithi045.stdout:606: finishing write tid 4 to smithi04569269-22 2021-07-31T01:47:37.118 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 22 v 136 (ObjNum 198 snap 52 seq_num 198) dirty exists 2021-07-31T01:47:37.118 INFO:tasks.rados.rados.0.smithi045.stdout:606: left oid 22 (ObjNum 198 snap 52 seq_num 198) 2021-07-31T01:47:37.120 INFO:tasks.rados.rados.0.smithi045.stdout:604: done (4 left) 2021-07-31T01:47:37.120 INFO:tasks.rados.rados.0.smithi045.stdout:605: done (3 left) 2021-07-31T01:47:37.121 INFO:tasks.rados.rados.0.smithi045.stdout:606: done (2 left) 2021-07-31T01:47:37.121 INFO:tasks.rados.rados.0.smithi045.stdout:609: delete oid 7 current snap is 52 2021-07-31T01:47:37.121 INFO:tasks.rados.rados.0.smithi045.stdout:607: finishing write tid 1 to smithi04569269-10 2021-07-31T01:47:37.122 INFO:tasks.rados.rados.0.smithi045.stdout:607: finishing write tid 2 to smithi04569269-10 2021-07-31T01:47:37.122 INFO:tasks.rados.rados.0.smithi045.stdout:607: finishing write tid 3 to smithi04569269-10 2021-07-31T01:47:37.122 INFO:tasks.rados.rados.0.smithi045.stdout:609: done (2 left) 2021-07-31T01:47:37.122 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 25 current snap is 52 2021-07-31T01:47:37.123 INFO:tasks.rados.rados.0.smithi045.stdout:610: seq_num 201 ranges {606208=573440,1179648=401408} 2021-07-31T01:47:37.124 INFO:tasks.rados.rados.0.smithi045.stdout:610: writing smithi04569269-25 from 606208 to 1179648 tid 1 2021-07-31T01:47:37.127 INFO:tasks.rados.rados.0.smithi045.stdout:610: writing smithi04569269-25 from 1179648 to 1581056 tid 2 2021-07-31T01:47:37.128 INFO:tasks.rados.rados.0.smithi045.stdout:611: copy_from oid 45 from oid 6 current snap is 52 2021-07-31T01:47:37.128 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 29 v 42 (ObjNum 46 snap 0 seq_num 46) dirty exists 2021-07-31T01:47:37.128 INFO:tasks.rados.rados.0.smithi045.stdout:608: done (3 left) 2021-07-31T01:47:37.128 INFO:tasks.rados.rados.0.smithi045.stdout:612: rollback oid 16 current snap is 52 2021-07-31T01:47:37.129 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 16 to 49 2021-07-31T01:47:37.129 INFO:tasks.rados.rados.0.smithi045.stdout:607: finishing write tid 4 to smithi04569269-10 2021-07-31T01:47:37.129 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 10 v 277 (ObjNum 199 snap 52 seq_num 199) dirty exists 2021-07-31T01:47:37.129 INFO:tasks.rados.rados.0.smithi045.stdout:607: left oid 10 (ObjNum 199 snap 52 seq_num 199) 2021-07-31T01:47:37.129 INFO:tasks.rados.rados.0.smithi045.stdout:607: done (3 left) 2021-07-31T01:47:37.130 INFO:tasks.rados.rados.0.smithi045.stdout:613: copy_from oid 33 from oid 19 current snap is 52 2021-07-31T01:47:37.130 INFO:tasks.rados.rados.0.smithi045.stdout:614: copy_from oid 18 from oid 8 current snap is 52 2021-07-31T01:47:37.130 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 17 current snap is 52 2021-07-31T01:47:37.130 INFO:tasks.rados.rados.0.smithi045.stdout:615: seq_num 202 ranges {876544=589824,1466368=401408} 2021-07-31T01:47:37.135 INFO:tasks.rados.rados.0.smithi045.stdout:615: writing smithi04569269-17 from 876544 to 1466368 tid 1 2021-07-31T01:47:37.137 INFO:tasks.rados.rados.0.smithi045.stdout:615: writing smithi04569269-17 from 1466368 to 1867776 tid 2 2021-07-31T01:47:37.137 INFO:tasks.rados.rados.0.smithi045.stdout:616: snap_remove snap 45 2021-07-31T01:47:37.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:37 smithi154 conmon[34553]: cluster 2021-07-31T01:47:34.508928+0000 mon.a (mon.0) 790 : cluster [DBG] osdmap e156: 8 total, 8 up, 8 in 2021-07-31T01:47:37.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:37 smithi154 conmon[34553]: audit 2021-07-31T01:47:35.255181+0000 mon.a (mon.0) 791 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2021-07-31T01:47:37.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:37 smithi045 conmon[39718]: cluster 2021-07-31T01 2021-07-31T01:47:37.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:37 smithi045 conmon[39718]: :47:34.508928+0000 mon.a (mon.0) 790 : cluster [DBG] osdmap e156: 8 total, 8 up, 8 in 2021-07-31T01:47:37.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:37 smithi045 conmon[39718]: audit 2021 2021-07-31T01:47:37.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:37 smithi045 conmon[39718]: -07-31T01:47:35 2021-07-31T01:47:37.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:37 smithi045 conmon[39718]: .255181+0000 mon.a (mon 2021-07-31T01:47:37.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:37 smithi045 conmon[39718]: .0) 791 : audit [INF] 2021-07-31T01:47:37.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:37 smithi045 conmon[39718]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2021-07-31T01:47:37.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:37 smithi045 conmon[33655]: cluster 2021-07-31T01 2021-07-31T01:47:37.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:37 smithi045 conmon[33655]: :47:34.508928+0000 mon.a (mon.0) 790 : cluster [DBG] osdmap e156: 8 total, 8 up, 8 in 2021-07-31T01:47:37.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:37 smithi045 conmon[33655]: audit 2021-07-31T01:47:37.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:37 smithi045 conmon[33655]: 2021-07-31T01:47:35.255181 2021-07-31T01:47:37.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:37 smithi045 conmon[33655]: +0000 mon.a (mon.0) 791 : audit [INF] 2021-07-31T01:47:37.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:37 smithi045 conmon[33655]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2021-07-31T01:47:38.050 INFO:tasks.rados.rados.0.smithi045.stdout:616: done (6 left) 2021-07-31T01:47:38.051 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 28 current snap is 52 2021-07-31T01:47:38.051 INFO:tasks.rados.rados.0.smithi045.stdout:614: finishing copy_from to smithi04569269-18 2021-07-31T01:47:38.051 INFO:tasks.rados.rados.0.smithi045.stdout:614: got expected ENOENT (src dne) 2021-07-31T01:47:38.051 INFO:tasks.rados.rados.0.smithi045.stdout:614: finishing copy_from racing read to smithi04569269-18 2021-07-31T01:47:38.052 INFO:tasks.rados.rados.0.smithi045.stdout:612: finishing rollback tid 1 to smithi04569269-16 2021-07-31T01:47:38.052 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 16 v 105 (ObjNum -1107059520 snap 21873 seq_num 3189092672) dirty dne 2021-07-31T01:47:38.052 INFO:tasks.rados.rados.0.smithi045.stdout:611: finishing copy_from to smithi04569269-45 2021-07-31T01:47:38.053 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 45 v 168 (ObjNum 1599304042 snap 1701011050 seq_num 1633182054) dirty exists 2021-07-31T01:47:38.053 INFO:tasks.rados.rados.0.smithi045.stdout:610: finishing write tid 1 to smithi04569269-25 2021-07-31T01:47:38.053 INFO:tasks.rados.rados.0.smithi045.stdout:610: finishing write tid 2 to smithi04569269-25 2021-07-31T01:47:38.053 INFO:tasks.rados.rados.0.smithi045.stdout:611: finishing copy_from racing read to smithi04569269-45 2021-07-31T01:47:38.054 INFO:tasks.rados.rados.0.smithi045.stdout:610: finishing write tid 3 to smithi04569269-25 2021-07-31T01:47:38.054 INFO:tasks.rados.rados.0.smithi045.stdout:610: finishing write tid 4 to smithi04569269-25 2021-07-31T01:47:38.054 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 25 v 36 (ObjNum 201 snap 52 seq_num 201) dirty exists 2021-07-31T01:47:38.054 INFO:tasks.rados.rados.0.smithi045.stdout:610: left oid 25 (ObjNum 201 snap 52 seq_num 201) 2021-07-31T01:47:38.055 INFO:tasks.rados.rados.0.smithi045.stdout:613: finishing copy_from to smithi04569269-33 2021-07-31T01:47:38.055 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 33 v 41 (ObjNum 94 snap 15 seq_num 94) dirty exists 2021-07-31T01:47:38.055 INFO:tasks.rados.rados.0.smithi045.stdout:613: finishing copy_from racing read to smithi04569269-33 2021-07-31T01:47:38.055 INFO:tasks.rados.rados.0.smithi045.stdout:615: finishing write tid 1 to smithi04569269-17 2021-07-31T01:47:38.056 INFO:tasks.rados.rados.0.smithi045.stdout:615: finishing write tid 2 to smithi04569269-17 2021-07-31T01:47:38.056 INFO:tasks.rados.rados.0.smithi045.stdout:615: finishing write tid 3 to smithi04569269-17 2021-07-31T01:47:38.056 INFO:tasks.rados.rados.0.smithi045.stdout:615: finishing write tid 4 to smithi04569269-17 2021-07-31T01:47:38.056 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 17 v 282 (ObjNum 202 snap 52 seq_num 202) dirty exists 2021-07-31T01:47:38.056 INFO:tasks.rados.rados.0.smithi045.stdout:615: left oid 17 (ObjNum 202 snap 52 seq_num 202) 2021-07-31T01:47:38.057 INFO:tasks.rados.rados.0.smithi045.stdout:617: seq_num 203 ranges {0=606208,606208=401408} 2021-07-31T01:47:38.058 INFO:tasks.rados.rados.0.smithi045.stdout:617: writing smithi04569269-28 from 0 to 606208 tid 1 2021-07-31T01:47:38.058 INFO:tasks.rados.rados.0.smithi045.stdout:617: writing smithi04569269-28 from 606208 to 1007616 tid 2 2021-07-31T01:47:38.058 INFO:tasks.rados.rados.0.smithi045.stdout:610: done (6 left) 2021-07-31T01:47:38.059 INFO:tasks.rados.rados.0.smithi045.stdout:611: done (5 left) 2021-07-31T01:47:38.059 INFO:tasks.rados.rados.0.smithi045.stdout:612: done (4 left) 2021-07-31T01:47:38.059 INFO:tasks.rados.rados.0.smithi045.stdout:613: done (3 left) 2021-07-31T01:47:38.059 INFO:tasks.rados.rados.0.smithi045.stdout:614: done (2 left) 2021-07-31T01:47:38.060 INFO:tasks.rados.rados.0.smithi045.stdout:615: done (1 left) 2021-07-31T01:47:38.060 INFO:tasks.rados.rados.0.smithi045.stdout:618: read oid 22 snap -1 2021-07-31T01:47:38.060 INFO:tasks.rados.rados.0.smithi045.stdout:618: expect (ObjNum 198 snap 52 seq_num 198) 2021-07-31T01:47:38.060 INFO:tasks.rados.rados.0.smithi045.stdout:619: read oid 7 snap -1 2021-07-31T01:47:38.060 INFO:tasks.rados.rados.0.smithi045.stdout:619: expect deleted 2021-07-31T01:47:38.061 INFO:tasks.rados.rados.0.smithi045.stdout:620: rollback oid 9 current snap is 52 2021-07-31T01:47:38.061 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 9 to 48 2021-07-31T01:47:38.061 INFO:tasks.rados.rados.0.smithi045.stdout:621: setattr oid 19 current snap is 52 2021-07-31T01:47:38.062 INFO:tasks.rados.rados.0.smithi045.stdout:619: done (4 left) 2021-07-31T01:47:38.062 INFO:tasks.rados.rados.0.smithi045.stdout:622: delete oid 36 current snap is 52 2021-07-31T01:47:38.063 INFO:tasks.rados.rados.0.smithi045.stdout:617: finishing write tid 1 to smithi04569269-28 2021-07-31T01:47:38.066 INFO:tasks.rados.rados.0.smithi045.stdout:622: done (4 left) 2021-07-31T01:47:38.067 INFO:tasks.rados.rados.0.smithi045.stdout:623: rollback oid 17 current snap is 52 2021-07-31T01:47:38.067 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 17 to 48 2021-07-31T01:47:38.067 INFO:tasks.rados.rados.0.smithi045.stdout:624: setattr oid 44 current snap is 52 2021-07-31T01:47:38.068 INFO:tasks.rados.rados.0.smithi045.stdout:617: finishing write tid 2 to smithi04569269-28 2021-07-31T01:47:38.068 INFO:tasks.rados.rados.0.smithi045.stdout:617: finishing write tid 3 to smithi04569269-28 2021-07-31T01:47:38.069 INFO:tasks.rados.rados.0.smithi045.stdout:625: setattr oid 37 current snap is 52 2021-07-31T01:47:38.069 INFO:tasks.rados.rados.0.smithi045.stdout:620: finishing rollback tid 0 to smithi04569269-9 2021-07-31T01:47:38.070 INFO:tasks.rados.rados.0.smithi045.stdout:620: finishing rollback tid 1 to smithi04569269-9 2021-07-31T01:47:38.070 INFO:tasks.rados.rados.0.smithi045.stdout:620: finishing rollback tid 2 to smithi04569269-9 2021-07-31T01:47:38.070 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 9 v 72 (ObjNum 8 snap 0 seq_num 8) dirty exists 2021-07-31T01:47:38.071 INFO:tasks.rados.rados.0.smithi045.stdout:620: done (6 left) 2021-07-31T01:47:38.071 INFO:tasks.rados.rados.0.smithi045.stdout:626: snap_remove snap 39 2021-07-31T01:47:38.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:38 smithi154 conmon[34553]: audit 2021-07-31T01:47:35.255340+0000 mon.b (mon. 2021-07-31T01:47:38.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:38 smithi154 conmon[34553]: 2) 56 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2021-07-31T01:47:38.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:38 smithi154 conmon[34553]: audit 2021-07-31T01:47:35.255535+0000 mon.b (mon.2) 57 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-07-31T01:47:38.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:38 smithi154 conmon[34553]: audit 2021-07-31T01:47:35.255695+0000 mon.b (mon.2) 58 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-07-31T01:47:38.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:38 smithi154 conmon[34553]: cluster 2021-07-31T01:47:35.795931 2021-07-31T01:47:38.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:38 smithi154 conmon[34553]: +0000 mgr.x (mgr.24542) 22 : cluster [DBG] pgmap v14: 25 pgs: 25 active+clean; 98 MiB data, 388 MiB used, 715 GiB / 715 GiB avail; 7.8 MiB/s rd, 1.4 MiB/s wr, 75 op/s 2021-07-31T01:47:38.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:38 smithi154 conmon[34553]: audit 2021-07- 2021-07-31T01:47:38.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:38 smithi154 conmon[34553]: 31T01:47:36.145315+0000 mon.a (mon.0) 792 : audit [INF] 2021-07-31T01:47:38.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:38 smithi154 conmon[34553]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2021-07-31T01:47:38.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:38 smithi154 conmon[34553]: audit 2021-07-31T01:47:36.149944+0000 mon.c ( 2021-07-31T01:47:38.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:38 smithi154 conmon[34553]: mon.1) 26 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2021-07-31T01:47:38.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:38 smithi154 conmon[34553]: cluster 2021-07-31T01:47: 2021-07-31T01:47:38.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:38 smithi154 conmon[34553]: 36.150007+0000 mon.a (mon.0) 793 : cluster [DBG] osdmap e157: 8 total, 8 up, 8 in 2021-07-31T01:47:38.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:38 smithi154 conmon[34553]: audit 2021-07- 2021-07-31T01:47:38.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:38 smithi154 conmon[34553]: 31T01:47:36.151226+0000 mon.b (mon.2) 59 : audit 2021-07-31T01:47:38.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:38 smithi154 conmon[34553]: [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2021-07-31T01:47:38.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:38 smithi154 conmon[34553]: audit 2021-07-31T01:47:36.151428+0000 2021-07-31T01:47:38.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:38 smithi154 conmon[34553]: mon.b (mon.2) 60 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-07-31T01:47:38.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:38 smithi154 conmon[34553]: audit 2021-07-31T01 2021-07-31T01:47:38.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:38 smithi154 conmon[34553]: :47:36.151610+0000 mon.b (mon.2) 61 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-07-31T01:47:38.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:38 smithi154 conmon[34553]: audit 2021-07- 2021-07-31T01:47:38.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:38 smithi154 conmon[34553]: 31T01:47:37.035632+0000 mon.c (mon.1) 27 : audit 2021-07-31T01:47:38.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:38 smithi154 conmon[34553]: [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2021-07-31T01:47:38.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:38 smithi154 conmon[34553]: audit 2021-07-31T01:47:37.044816+0000 2021-07-31T01:47:38.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:38 smithi154 conmon[34553]: mon.b (mon.2) 62 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2021-07-31T01:47:38.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:38 smithi154 conmon[34553]: audit 2021-07-31T01:47 2021-07-31T01:47:38.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:38 smithi154 conmon[34553]: :37.549914+0000 mon.b (mon.2) 63 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2021-07-31T01:47:38.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:38 smithi154 conmon[34553]: audit 2021 2021-07-31T01:47:38.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:38 smithi154 conmon[34553]: -07-31T01:47:37.550402+0000 mon.b (mon.2) 2021-07-31T01:47:38.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:38 smithi154 conmon[34553]: 64 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2021-07-31T01:47:38.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:38 smithi154 conmon[34553]: audit 2021-07-31T01:47: 2021-07-31T01:47:38.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:38 smithi154 conmon[34553]: 37.550605+0000 mon.b (mon.2) 65 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-07-31T01:47:38.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:38 smithi154 conmon[34553]: audit 2021-07- 2021-07-31T01:47:38.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:38 smithi154 conmon[34553]: 31T01:47:37.550812+0000 mon.b (mon.2) 66 : audit [DBG] 2021-07-31T01:47:38.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:38 smithi154 conmon[34553]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-07-31T01:47:38.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:38 smithi154 conmon[34553]: cluster 2021-07-31T01:47:37.796458+0000 2021-07-31T01:47:38.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:38 smithi154 conmon[34553]: mgr.x (mgr.24542) 23 : cluster [DBG] pgmap v16: 25 pgs: 25 active+clean; 98 MiB data, 388 MiB used, 715 GiB / 715 GiB avail; 6.3 MiB/s rd, 1.1 MiB/s wr, 61 op/s 2021-07-31T01:47:38.526 INFO:journalctl@ceph.osd.0.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[43243]: debug 2021-07-31T01:47:38.526 INFO:journalctl@ceph.osd.0.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[43243]: 2021-07-31T01:47:38.048+0000 7fdf6978b700 -1 osd.0 pg_epoch: 158 pg[3.6s0( v 153'66 (0'0,153'66] local-lis/les=56/57 n=3 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [0,1,4]p0(0) r=0 lpr=56 crt=153'66 lcod 150'64 mlcod 150'64 active+clean+snaptrim trimq=[53~1] ps=[4d~1,58~1,60~1]] removing snap head 2021-07-31T01:47:38.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[33655]: audit 2021-07-31T01:47:35.255340+0000 mon.b (mon.2) 56 : audit 2021-07-31T01:47:38.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[33655]: [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2021-07-31T01:47:38.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[33655]: audit 2021-07-31T01:47:35.255535+0000 mon.b (mon.2) 57 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-07-31T01:47:38.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[33655]: audit 2021-07-31T01: 2021-07-31T01:47:38.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[33655]: 47:35.255695+0000 mon.b (mon.2) 58 : audit [DBG] 2021-07-31T01:47:38.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[33655]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-07-31T01:47:38.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[33655]: cluster 2021-07-31 2021-07-31T01:47:38.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[33655]: T01:47:35.795931+0000 mgr.x (mgr.24542 2021-07-31T01:47:38.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[33655]: ) 22 : cluster [DBG] pgmap v14: 25 pgs: 25 active+clean; 98 MiB data, 388 MiB used, 715 GiB / 715 GiB avail; 7.8 MiB/s rd, 1.4 MiB/s wr, 75 op/s 2021-07-31T01:47:38.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[33655]: audit 2021-07-31T01:47: 2021-07-31T01:47:38.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[33655]: 36.145315+0000 mon.a (mon.0) 792 : audit [INF] 2021-07-31T01:47:38.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[33655]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2021-07-31T01:47:38.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[33655]: audit 2021-07-31 2021-07-31T01:47:38.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[33655]: T01:47:36.149944+0000 mon.c (mon.1 2021-07-31T01:47:38.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[33655]: ) 26 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2021-07-31T01:47:38.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[33655]: cluster 2021 2021-07-31T01:47:38.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[33655]: -07-31T01:47:36.150007+0000 mon.a ( 2021-07-31T01:47:38.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[33655]: mon.0) 793 : cluster [DBG] osdmap e157: 8 total, 8 up, 8 in 2021-07-31T01:47:38.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[33655]: audit 2021-07-31T01:47:36.151226 2021-07-31T01:47:38.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[33655]: +0000 mon.b (mon.2) 59 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2021-07-31T01:47:38.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[33655]: audit 2021 2021-07-31T01:47:38.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[33655]: -07-31T01:47:36.151428+0000 mon.b (mon 2021-07-31T01:47:38.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[33655]: .2) 60 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-07-31T01:47:38.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[33655]: audit 2021- 2021-07-31T01:47:38.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[33655]: 07-31T01:47:36.151610+0000 mon.b (mon.2) 61 : audit 2021-07-31T01:47:38.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[33655]: [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-07-31T01:47:38.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[33655]: audit 2021-07-31T01:47:38.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[33655]: 2021-07-31 2021-07-31T01:47:38.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[33655]: T01:47:37.035632+0000 mon.c (mon.1 2021-07-31T01:47:38.535 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[33655]: ) 27 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2021-07-31T01:47:38.535 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[33655]: audit 2021 2021-07-31T01:47:38.535 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[33655]: -07-31T01:47:37.044816+0000 mon.b (mon.2) 2021-07-31T01:47:38.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[33655]: 62 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2021-07-31T01:47:38.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[33655]: audit 2021-07-31T01:47:38.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[33655]: 2021-07-31T01: 2021-07-31T01:47:38.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[33655]: 47:37.549914+0000 mon.b (mon.2) 63 2021-07-31T01:47:38.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[33655]: : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2021-07-31T01:47:38.539 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[33655]: audit 2021-07-31T01:47:38.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[33655]: 2021-07-31T01:47:37.550402+0000 mon.b 2021-07-31T01:47:38.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[33655]: (mon.2) 64 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2021-07-31T01:47:38.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[33655]: audit 2021-07- 2021-07-31T01:47:38.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[33655]: 31T01:47:37.550605+0000 mon.b (mon.2) 65 : audit 2021-07-31T01:47:38.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[33655]: [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-07-31T01:47:38.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[33655]: audit 2021- 2021-07-31T01:47:38.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[33655]: 07-31T01:47:37.550812+0000 mon.b (mon 2021-07-31T01:47:38.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[33655]: .2) 66 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-07-31T01:47:38.543 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[33655]: cluster 2021-07-31T01:47:38.543 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[33655]: 2021-07-31T01:47:37.796458+0000 mgr.x ( 2021-07-31T01:47:38.543 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[33655]: mgr.24542) 23 : cluster [DBG] pgmap v16: 25 pgs: 25 active+clean; 98 MiB data, 388 MiB used, 715 GiB / 715 GiB avail; 6.3 MiB/s rd, 1.1 MiB/s wr, 61 op/s 2021-07-31T01:47:38.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[39718]: audit 2021-07-31T01:47:35. 2021-07-31T01:47:38.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[39718]: 255340+0000 mon.b (mon.2) 56 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2021-07-31T01:47:38.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[39718]: audit 2021-07-31T01:47:38.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[39718]: 2021-07-31T01:47:35.255535+0000 2021-07-31T01:47:38.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[39718]: mon.b (mon.2) 57 : audit 2021-07-31T01:47:38.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[39718]: [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-07-31T01:47:38.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[39718]: audit 2021-07-31T01:47:38.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[39718]: 2021-07-31T01:47 2021-07-31T01:47:38.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[39718]: :35.255695+0000 mon.b (mon 2021-07-31T01:47:38.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[39718]: .2) 58 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-07-31T01:47:38.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[39718]: cluster 2021-07-31 2021-07-31T01:47:38.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[39718]: T01:47:35.795931+0000 mgr.x 2021-07-31T01:47:38.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[39718]: (mgr.24542) 22 : cluster [DBG] pgmap v14: 25 pgs: 25 active+clean; 98 MiB data, 388 MiB used, 715 GiB / 715 GiB avail; 7.8 MiB/s rd, 1.4 MiB/s wr, 75 op/s 2021-07-31T01:47:38.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[39718]: 2021-07-31T01:47:38.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[39718]: audit 2021-07-31T01:47:38.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[39718]: 2021-07- 2021-07-31T01:47:38.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[39718]: 31T01:47:36.145315+0000 mon.a ( 2021-07-31T01:47:38.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[39718]: mon.0) 792 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2021-07-31T01:47:38.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[39718]: 2021-07-31T01:47:38.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[39718]: audit 2021- 2021-07-31T01:47:38.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[39718]: 07-31T01:47:36 2021-07-31T01:47:38.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[39718]: .149944+0000 mon.c (mon.1 2021-07-31T01:47:38.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[39718]: ) 26 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2021-07-31T01:47:38.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[39718]: cluster 2021- 2021-07-31T01:47:38.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[39718]: 07-31T01:47:36.150007+0000 2021-07-31T01:47:38.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[39718]: mon.a (mon.0) 793 : cluster [DBG] 2021-07-31T01:47:38.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[39718]: osdmap e157: 8 total, 8 up, 8 in 2021-07-31T01:47:38.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[39718]: audit 2021-07-31T01:47:38.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[39718]: 2021-07-31 2021-07-31T01:47:38.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[39718]: T01:47:36.151226+0000 2021-07-31T01:47:38.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[39718]: mon.b (mon.2) 59 : audit [DBG] 2021-07-31T01:47:38.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2021-07-31T01:47:38.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[39718]: audit 2021 2021-07-31T01:47:38.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[39718]: -07-31T01:47:36 2021-07-31T01:47:38.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[39718]: .151428+0000 mon.b (mon.2) 60 : 2021-07-31T01:47:38.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[39718]: audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-07-31T01:47:38.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[39718]: audit 2021-07-31T01:47:38.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[39718]: 2021-07 2021-07-31T01:47:38.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[39718]: -31T01:47:36. 2021-07-31T01:47:38.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[39718]: 151610+0000 mon.b (mon.2) 61 : 2021-07-31T01:47:38.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[39718]: audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-07-31T01:47:38.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[39718]: audit 2021-07-31T01:47:38.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[39718]: 2021-07-31T01:47 2021-07-31T01:47:38.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[39718]: :37.035632+0000 mon.c ( 2021-07-31T01:47:38.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[39718]: mon.1) 27 : audit [INF] 2021-07-31T01:47:38.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[39718]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2021-07-31T01:47:38.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[39718]: audit 2021 2021-07-31T01:47:38.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[39718]: -07-31T01:47: 2021-07-31T01:47:38.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[39718]: 37.044816+0000 mon.b (mon. 2021-07-31T01:47:38.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[39718]: 2) 62 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2021-07-31T01:47:38.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[39718]: audit 2021- 2021-07-31T01:47:38.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[39718]: 07-31T01:47:37. 2021-07-31T01:47:38.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[39718]: 549914+0000 mon.b (mon.2) 63 2021-07-31T01:47:38.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[39718]: : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2021-07-31T01:47:38.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[39718]: audit 2021-07-31T01:47:38.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[39718]: 2021-07-31T 2021-07-31T01:47:38.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[39718]: 01:47:37.550402+0000 2021-07-31T01:47:38.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[39718]: mon.b (mon.2) 64 : audit [DBG] 2021-07-31T01:47:38.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2021-07-31T01:47:38.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[39718]: audit 2021-07-31T01:47:38.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[39718]: 2021-07- 2021-07-31T01:47:38.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[39718]: 31T01:47:37.550605+0000 mon.b 2021-07-31T01:47:38.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[39718]: (mon.2) 65 : audit [DBG] 2021-07-31T01:47:38.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-07-31T01:47:38.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[39718]: audit 2021 2021-07-31T01:47:38.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[39718]: -07-31T01:47:37.550812+0000 mon.b (mon.2 2021-07-31T01:47:38.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[39718]: ) 66 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-07-31T01:47:38.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[39718]: cluster 2021-07-31T01:47:38.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[39718]: 2021-07-31T01: 2021-07-31T01:47:38.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[39718]: 47:37.796458+0000 mgr.x ( 2021-07-31T01:47:38.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[39718]: mgr.24542) 23 : cluster [DBG] pgmap v16: 25 pgs: 25 active+clean; 98 MiB data, 388 MiB used, 715 GiB / 715 GiB avail; 6.3 MiB/s rd, 1.1 MiB/s wr, 61 op/s 2021-07-31T01:47:38.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:38 smithi045 conmon[39718]: 2021-07-31T01:47:39.056 INFO:tasks.rados.rados.0.smithi045.stdout:626: done (6 left) 2021-07-31T01:47:39.056 INFO:tasks.rados.rados.0.smithi045.stdout:627: delete oid 29 current snap is 52 2021-07-31T01:47:39.057 INFO:tasks.rados.rados.0.smithi045.stdout:617: finishing write tid 4 to smithi04569269-28 2021-07-31T01:47:39.057 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 28 v 289 (ObjNum 203 snap 52 seq_num 203) dirty exists 2021-07-31T01:47:39.057 INFO:tasks.rados.rados.0.smithi045.stdout:617: left oid 28 (ObjNum 203 snap 52 seq_num 203) 2021-07-31T01:47:39.057 INFO:tasks.rados.rados.0.smithi045.stdout:618: expect (ObjNum 198 snap 52 seq_num 198) 2021-07-31T01:47:39.147 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 19 v 141 (ObjNum 94 snap 15 seq_num 94) dirty exists 2021-07-31T01:47:39.148 INFO:tasks.rados.rados.0.smithi045.stdout:623: finishing rollback tid 0 to smithi04569269-17 2021-07-31T01:47:39.148 INFO:tasks.rados.rados.0.smithi045.stdout:623: finishing rollback tid 1 to smithi04569269-17 2021-07-31T01:47:39.148 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 44 v 190 (ObjNum 142 snap 30 seq_num 142) dirty exists 2021-07-31T01:47:39.148 INFO:tasks.rados.rados.0.smithi045.stdout:623: finishing rollback tid 2 to smithi04569269-17 2021-07-31T01:47:39.149 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 17 v 296 (ObjNum 171 snap 41 seq_num 171) dirty exists 2021-07-31T01:47:39.149 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 37 v 173 (ObjNum -1104942000 snap 21873 seq_num 3190218160) dirty exists 2021-07-31T01:47:39.149 INFO:tasks.rados.rados.0.smithi045.stdout:617: done (6 left) 2021-07-31T01:47:39.149 INFO:tasks.rados.rados.0.smithi045.stdout:618: done (5 left) 2021-07-31T01:47:39.150 INFO:tasks.rados.rados.0.smithi045.stdout:621: done (4 left) 2021-07-31T01:47:39.150 INFO:tasks.rados.rados.0.smithi045.stdout:623: done (3 left) 2021-07-31T01:47:39.150 INFO:tasks.rados.rados.0.smithi045.stdout:624: done (2 left) 2021-07-31T01:47:39.150 INFO:tasks.rados.rados.0.smithi045.stdout:625: done (1 left) 2021-07-31T01:47:39.151 INFO:tasks.rados.rados.0.smithi045.stdout:627: done (0 left) 2021-07-31T01:47:39.151 INFO:tasks.rados.rados.0.smithi045.stdout:628: read oid 42 snap -1 2021-07-31T01:47:39.151 INFO:tasks.rados.rados.0.smithi045.stdout:628: expect (ObjNum 176 snap 46 seq_num 176) 2021-07-31T01:47:39.151 INFO:tasks.rados.rados.0.smithi045.stdout:629: read oid 24 snap -1 2021-07-31T01:47:39.152 INFO:tasks.rados.rados.0.smithi045.stdout:629: expect (ObjNum 36 snap 0 seq_num 36) 2021-07-31T01:47:39.152 INFO:tasks.rados.rados.0.smithi045.stdout:630: rmattr oid 45 current snap is 52 2021-07-31T01:47:39.152 INFO:tasks.rados.rados.0.smithi045.stdout:631: snap_remove snap 48 2021-07-31T01:47:39.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:39 smithi154 conmon[34553]: cluster 2021-07-31T01:47:38.049268+0000 mon.a (mon.0) 794 : cluster [DBG] osdmap e158: 8 total, 8 up, 8 in 2021-07-31T01:47:39.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:39 smithi045 conmon[33655]: cluster 2021-07-31T01:47:38.049268+0000 mon.a (mon.0) 2021-07-31T01:47:39.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:39 smithi045 conmon[33655]: 794 : cluster [DBG] osdmap e158: 8 total, 8 up, 8 in 2021-07-31T01:47:39.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:39 smithi045 conmon[39718]: cluster 2021-07-31T01:47:38.049268+0000 mon.a (mon.0) 794 : 2021-07-31T01:47:39.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:39 smithi045 conmon[39718]: cluster [DBG] osdmap e158: 8 total, 8 up, 8 in 2021-07-31T01:47:40.063 INFO:tasks.rados.rados.0.smithi045.stdout:629: expect (ObjNum 36 snap 0 seq_num 36) 2021-07-31T01:47:40.087 INFO:tasks.rados.rados.0.smithi045.stdout:628: expect (ObjNum 176 snap 46 seq_num 176) 2021-07-31T01:47:40.102 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 45 v 303 (ObjNum 1431399533 snap 1668048986 seq_num 1886151281) dirty exists 2021-07-31T01:47:40.102 INFO:tasks.rados.rados.0.smithi045.stdout:628: done (3 left) 2021-07-31T01:47:40.102 INFO:tasks.rados.rados.0.smithi045.stdout:629: done (2 left) 2021-07-31T01:47:40.102 INFO:tasks.rados.rados.0.smithi045.stdout:630: done (1 left) 2021-07-31T01:47:40.103 INFO:tasks.rados.rados.0.smithi045.stdout:631: done (0 left) 2021-07-31T01:47:40.103 INFO:tasks.rados.rados.0.smithi045.stdout:632: setattr oid 8 current snap is 52 2021-07-31T01:47:40.103 INFO:tasks.rados.rados.0.smithi045.stdout:633: snap_create 2021-07-31T01:47:40.104 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 8 v 452 (ObjNum 0 snap 0 seq_num 0) dirty exists 2021-07-31T01:47:40.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:40 smithi154 conmon[34553]: cluster 2021-07-31T01:47:39.055073+0000 mon.a (mon.0) 795 : cluster [DBG] osdmap e159: 8 total, 8 up, 8 in 2021-07-31T01:47:40.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:40 smithi154 conmon[34553]: cluster 2021-07-31T01:47:39.797026+0000 mgr.x (mgr.24542) 24 : cluster [DBG] pgmap v19: 25 pgs: 25 active+clean; 105 MiB data, 409 MiB used, 715 GiB / 715 GiB avail; 1.1 MiB/s rd, 1.7 MiB/s wr, 28 op/s 2021-07-31T01:47:40.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:40 smithi045 conmon[33655]: cluster 2021-07-31T01:47:39.055073+0000 mon.a (mon.0 2021-07-31T01:47:40.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:40 smithi045 conmon[33655]: ) 795 : cluster [DBG] osdmap e159: 8 total, 8 up, 8 in 2021-07-31T01:47:40.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:40 smithi045 conmon[33655]: cluster 2021-07-31T01:47:39.797026+0000 mgr.x (mgr.24542) 24 : cluster [DBG] pgmap v19: 25 pgs: 25 active+clean; 105 MiB data, 409 MiB used, 715 GiB / 715 GiB avail; 1.1 MiB/s rd, 1.7 MiB/s wr, 28 op/s 2021-07-31T01:47:40.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:40 smithi045 conmon[39718]: cluster 2021-07-31T01:47:39.055073+0000 2021-07-31T01:47:40.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:40 smithi045 conmon[39718]: mon.a (mon.0) 795 : cluster [DBG] osdmap e159: 8 total, 8 up, 8 in 2021-07-31T01:47:40.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:40 smithi045 conmon[39718]: cluster 2021-07-31T01:47: 2021-07-31T01:47:40.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:40 smithi045 conmon[39718]: 39.797026+0000 mgr.x (mgr.24542) 24 : cluster [DBG] pgmap v19: 25 pgs: 25 active+clean; 105 MiB data, 409 MiB used, 715 GiB / 715 GiB avail; 1.1 MiB/s rd, 1.7 MiB/s wr, 28 op/s 2021-07-31T01:47:41.063 INFO:tasks.rados.rados.0.smithi045.stdout:632: done (1 left) 2021-07-31T01:47:41.063 INFO:tasks.rados.rados.0.smithi045.stdout:633: done (0 left) 2021-07-31T01:47:41.064 INFO:tasks.rados.rados.0.smithi045.stdout:634: rollback oid 47 current snap is 53 2021-07-31T01:47:41.064 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 47 to 40 2021-07-31T01:47:41.064 INFO:tasks.rados.rados.0.smithi045.stdout:635: read oid 10 snap -1 2021-07-31T01:47:41.064 INFO:tasks.rados.rados.0.smithi045.stdout:635: expect (ObjNum 199 snap 52 seq_num 199) 2021-07-31T01:47:41.065 INFO:tasks.rados.rados.0.smithi045.stdout:636: read oid 42 snap 49 2021-07-31T01:47:41.065 INFO:tasks.rados.rados.0.smithi045.stdout:636: expect (ObjNum 176 snap 46 seq_num 176) 2021-07-31T01:47:41.065 INFO:tasks.rados.rados.0.smithi045.stdout:637: snap_remove snap 35 2021-07-31T01:47:41.492 INFO:tasks.rados.rados.0.smithi045.stdout:637: done (3 left) 2021-07-31T01:47:41.493 INFO:tasks.rados.rados.0.smithi045.stdout:638: rmattr oid 46 current snap is 53 2021-07-31T01:47:41.493 INFO:tasks.rados.rados.0.smithi045.stdout:638: done (3 left) 2021-07-31T01:47:41.493 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 40 current snap is 53 2021-07-31T01:47:41.493 INFO:tasks.rados.rados.0.smithi045.stdout:634: finishing rollback tid 0 to smithi04569269-47 2021-07-31T01:47:41.493 INFO:tasks.rados.rados.0.smithi045.stdout:639: seq_num 206 ranges {253952=655360,909312=401408} 2021-07-31T01:47:41.498 INFO:tasks.rados.rados.0.smithi045.stdout:639: writing smithi04569269-40 from 253952 to 909312 tid 1 2021-07-31T01:47:41.501 INFO:tasks.rados.rados.0.smithi045.stdout:639: writing smithi04569269-40 from 909312 to 1310720 tid 2 2021-07-31T01:47:41.502 INFO:tasks.rados.rados.0.smithi045.stdout:640: read oid 36 snap -1 2021-07-31T01:47:41.502 INFO:tasks.rados.rados.0.smithi045.stdout:640: expect deleted 2021-07-31T01:47:41.503 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 16 current snap is 53 2021-07-31T01:47:41.503 INFO:tasks.rados.rados.0.smithi045.stdout:634: finishing rollback tid 1 to smithi04569269-47 2021-07-31T01:47:41.503 INFO:tasks.rados.rados.0.smithi045.stdout:634: finishing rollback tid 2 to smithi04569269-47 2021-07-31T01:47:41.503 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 47 v 48 (ObjNum 46 snap 0 seq_num 46) dirty exists 2021-07-31T01:47:41.504 INFO:tasks.rados.rados.0.smithi045.stdout:635: expect (ObjNum 199 snap 52 seq_num 199) 2021-07-31T01:47:41.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:41 smithi154 conmon[34553]: cluster 2021-07-31T01:47:40.061331+0000 mon.a (mon.0) 796 : cluster 2021-07-31T01:47:41.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:41 smithi154 conmon[34553]: [DBG] osdmap e160: 8 total, 8 up, 8 in 2021-07-31T01:47:41.508 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:41 smithi045 conmon[39718]: cluster 2021-07-31T01:47:40.061331+0000 mon.a (mon.0) 796 : cluster [DBG] osdmap e160: 8 total, 8 up, 8 in 2021-07-31T01:47:41.509 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:41 smithi045 conmon[33655]: cluster 2021-07-31T01:47:40.061331+0000 mon.a 2021-07-31T01:47:41.509 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:41 smithi045 conmon[33655]: (mon.0) 796 : cluster [DBG] osdmap e160: 8 total, 8 up, 8 in 2021-07-31T01:47:41.670 INFO:tasks.rados.rados.0.smithi045.stdout:636: expect (ObjNum 176 snap 46 seq_num 176) 2021-07-31T01:47:41.684 INFO:tasks.rados.rados.0.smithi045.stdout:641: seq_num 207 ranges {0=671744,671744=401408} 2021-07-31T01:47:41.688 INFO:tasks.rados.rados.0.smithi045.stdout:641: writing smithi04569269-16 from 0 to 671744 tid 1 2021-07-31T01:47:41.691 INFO:tasks.rados.rados.0.smithi045.stdout:641: writing smithi04569269-16 from 671744 to 1073152 tid 2 2021-07-31T01:47:41.692 INFO:tasks.rados.rados.0.smithi045.stdout:634: done (5 left) 2021-07-31T01:47:41.693 INFO:tasks.rados.rados.0.smithi045.stdout:635: done (4 left) 2021-07-31T01:47:41.693 INFO:tasks.rados.rados.0.smithi045.stdout:636: done (3 left) 2021-07-31T01:47:41.693 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 15 current snap is 53 2021-07-31T01:47:41.693 INFO:tasks.rados.rados.0.smithi045.stdout:639: finishing write tid 1 to smithi04569269-40 2021-07-31T01:47:41.694 INFO:tasks.rados.rados.0.smithi045.stdout:639: finishing write tid 2 to smithi04569269-40 2021-07-31T01:47:41.694 INFO:tasks.rados.rados.0.smithi045.stdout:639: finishing write tid 3 to smithi04569269-40 2021-07-31T01:47:41.694 INFO:tasks.rados.rados.0.smithi045.stdout:639: finishing write tid 4 to smithi04569269-40 2021-07-31T01:47:41.694 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 40 v 230 (ObjNum 206 snap 53 seq_num 206) dirty exists 2021-07-31T01:47:41.694 INFO:tasks.rados.rados.0.smithi045.stdout:639: left oid 40 (ObjNum 206 snap 53 seq_num 206) 2021-07-31T01:47:41.695 INFO:tasks.rados.rados.0.smithi045.stdout:642: seq_num 208 ranges {352256=688128,1040384=401408} 2021-07-31T01:47:41.696 INFO:tasks.rados.rados.0.smithi045.stdout:642: writing smithi04569269-15 from 352256 to 1040384 tid 1 2021-07-31T01:47:41.699 INFO:tasks.rados.rados.0.smithi045.stdout:642: writing smithi04569269-15 from 1040384 to 1441792 tid 2 2021-07-31T01:47:41.700 INFO:tasks.rados.rados.0.smithi045.stdout:639: done (3 left) 2021-07-31T01:47:41.700 INFO:tasks.rados.rados.0.smithi045.stdout:640: done (2 left) 2021-07-31T01:47:41.701 INFO:tasks.rados.rados.0.smithi045.stdout:643: read oid 26 snap 49 2021-07-31T01:47:41.701 INFO:tasks.rados.rados.0.smithi045.stdout:643: expect (ObjNum 101 snap 16 seq_num 101) 2021-07-31T01:47:41.701 INFO:tasks.rados.rados.0.smithi045.stdout:644: snap_remove snap 51 2021-07-31T01:47:41.776 INFO:journalctl@ceph.osd.0.smithi045.stdout:Jul 31 01:47:41 smithi045 conmon[43243]: debug 2021-07-31T01:47:41.505+0000 7fdf6bf90700 -1 osd.0 pg_epoch: 162 pg[3.1s0( v 162'153 (0'0,162'153] local-lis/les=56/57 n=6 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [0,4,3]p0(0) r=0 lpr=56 crt=162'153 lcod 162'151 mlcod 162'151 active+clean+snaptrim trimq=[42~1] ps=[4a~1,5c~1]] removing snap head 2021-07-31T01:47:41.777 INFO:journalctl@ceph.osd.3.smithi045.stdout:Jul 31 01:47:41 smithi045 conmon[54013]: debug 2021-07-31T01:47:41.512+0000 7f64fc8cd700 -1 osd.3 pg_epoch: 162 pg[3.bs0( v 162'454 (0'0,162'454] local-lis/les=56/57 n=16 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [3,0,4]p3(0) r=0 lpr=56 luod=160'452 crt=160'452 lcod 160'451 mlcod 160'451 active+clean+snaptrim trimq=[42~1] ps=[4a~1,5c~1]] removing snap head 2021-07-31T01:47:42.003 INFO:journalctl@ceph.osd.7.smithi154.stdout:Jul 31 01:47:41 smithi154 conmon[49149]: debug 2021-07-31T01:47:41.505+0000 7f8a75a04700 -1 osd.7 pg_epoch: 162 pg[3.ds0( v 162'317 (0'0,162'317] local-lis/les=56/57 n=13 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [7,5,3]p7(0) r=0 lpr=56 luod=162'315 crt=162'315 lcod 162'313 mlcod 162'313 active+clean+snaptrim trimq=[42~1] ps=[4a~1,5c~1]] removing snap head 2021-07-31T01:47:42.501 INFO:tasks.rados.rados.0.smithi045.stdout:644: done (3 left) 2021-07-31T01:47:42.502 INFO:tasks.rados.rados.0.smithi045.stdout:645: snap_remove snap 52 2021-07-31T01:47:42.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:42 smithi154 conmon[34553]: cluster 2021-07-31T01:47:41.062370+0000 mon.a (mon.0) 797 : cluster [DBG] osdmap e161: 8 total, 8 up, 8 in 2021-07-31T01:47:42.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:42 smithi154 conmon[34553]: cluster 2021-07-31T01:47:41.491553+0000 mon.a (mon.0) 798 : cluster [DBG] osdmap e162: 8 total, 8 up, 8 in 2021-07-31T01:47:42.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:42 smithi154 conmon[34553]: cluster 2021-07-31T01:47:41.797486+0000 mgr.x (mgr.24542 2021-07-31T01:47:42.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:42 smithi154 conmon[34553]: ) 25 : cluster [DBG] pgmap v23: 25 pgs: 25 active+clean; 105 MiB data, 409 MiB used, 715 GiB / 715 GiB avail; 1.6 MiB/s rd, 2.4 MiB/s wr, 40 op/s 2021-07-31T01:47:42.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:42 smithi045 conmon[33655]: cluster 2021-07-31T01:47:41.062370+0000 mon.a ( 2021-07-31T01:47:42.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:42 smithi045 conmon[33655]: mon.0) 797 : cluster [DBG] osdmap e161: 8 total, 8 up, 8 in 2021-07-31T01:47:42.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:42 smithi045 conmon[33655]: cluster 2021-07-31T01:47:41.491553+0000 mon.a (mon.0) 798 : cluster 2021-07-31T01:47:42.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:42 smithi045 conmon[33655]: [DBG] osdmap e162: 8 total, 8 up, 8 in 2021-07-31T01:47:42.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:42 smithi045 conmon[33655]: cluster 2021-07-31T01:47:42.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:42 smithi045 conmon[33655]: 2021-07-31T01:47 2021-07-31T01:47:42.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:42 smithi045 conmon[33655]: :41.797486+0000 mgr.x (mgr.24542) 25 : cluster 2021-07-31T01:47:42.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:42 smithi045 conmon[33655]: [DBG] pgmap v23: 25 pgs: 25 active+clean; 105 MiB data, 409 MiB used, 715 GiB / 715 GiB avail; 1.6 MiB/s rd, 2.4 MiB/s wr, 40 op/s 2021-07-31T01:47:42.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:42 smithi045 conmon[39718]: cluster 2021-07-31T01:47:41.062370+0000 mon.a 2021-07-31T01:47:42.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:42 smithi045 conmon[39718]: (mon.0) 797 : cluster [DBG] osdmap e161: 8 total, 8 up, 8 in 2021-07-31T01:47:42.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:42 smithi045 conmon[39718]: cluster 2021-07-31T01:47:41.491553 2021-07-31T01:47:42.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:42 smithi045 conmon[39718]: +0000 mon.a (mon.0) 798 : cluster [DBG] osdmap e162: 8 total, 8 up, 8 in 2021-07-31T01:47:42.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:42 smithi045 conmon[39718]: cluster 2021- 2021-07-31T01:47:42.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:42 smithi045 conmon[39718]: 07-31T01:47:41.797486 2021-07-31T01:47:42.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:42 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 25 : cluster 2021-07-31T01:47:42.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:42 smithi045 conmon[39718]: [DBG] pgmap v23: 25 pgs: 25 active+clean; 105 MiB data, 409 MiB used, 715 GiB / 715 GiB avail; 1.6 MiB/s rd, 2.4 MiB/s wr, 40 op/s 2021-07-31T01:47:43.504 INFO:tasks.rados.rados.0.smithi045.stdout:645: done (3 left) 2021-07-31T01:47:43.504 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 30 current snap is 53 2021-07-31T01:47:43.504 INFO:tasks.rados.rados.0.smithi045.stdout:641: finishing write tid 1 to smithi04569269-16 2021-07-31T01:47:43.505 INFO:tasks.rados.rados.0.smithi045.stdout:646: seq_num 209 ranges {1351680=704512,2056192=401408} 2021-07-31T01:47:43.515 INFO:tasks.rados.rados.0.smithi045.stdout:646: writing smithi04569269-30 from 1351680 to 2056192 tid 1 2021-07-31T01:47:43.518 INFO:tasks.rados.rados.0.smithi045.stdout:646: writing smithi04569269-30 from 2056192 to 2457600 tid 2 2021-07-31T01:47:43.518 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 35 current snap is 53 2021-07-31T01:47:43.518 INFO:tasks.rados.rados.0.smithi045.stdout:647: seq_num 210 ranges {0=720896,720896=401408} 2021-07-31T01:47:43.522 INFO:tasks.rados.rados.0.smithi045.stdout:647: writing smithi04569269-35 from 0 to 720896 tid 1 2021-07-31T01:47:43.525 INFO:tasks.rados.rados.0.smithi045.stdout:647: writing smithi04569269-35 from 720896 to 1122304 tid 2 2021-07-31T01:47:43.525 INFO:tasks.rados.rados.0.smithi045.stdout:648: snap_create 2021-07-31T01:47:43.526 INFO:tasks.rados.rados.0.smithi045.stdout:641: finishing write tid 2 to smithi04569269-16 2021-07-31T01:47:43.526 INFO:tasks.rados.rados.0.smithi045.stdout:641: finishing write tid 3 to smithi04569269-16 2021-07-31T01:47:43.526 INFO:tasks.rados.rados.0.smithi045.stdout:641: finishing write tid 4 to smithi04569269-16 2021-07-31T01:47:43.526 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 16 v 158 (ObjNum 207 snap 53 seq_num 207) dirty exists 2021-07-31T01:47:43.527 INFO:tasks.rados.rados.0.smithi045.stdout:641: left oid 16 (ObjNum 207 snap 53 seq_num 207) 2021-07-31T01:47:43.527 INFO:tasks.rados.rados.0.smithi045.stdout:642: finishing write tid 1 to smithi04569269-15 2021-07-31T01:47:43.527 INFO:tasks.rados.rados.0.smithi045.stdout:642: finishing write tid 2 to smithi04569269-15 2021-07-31T01:47:43.527 INFO:tasks.rados.rados.0.smithi045.stdout:642: finishing write tid 3 to smithi04569269-15 2021-07-31T01:47:43.527 INFO:tasks.rados.rados.0.smithi045.stdout:642: finishing write tid 4 to smithi04569269-15 2021-07-31T01:47:43.528 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 15 v 183 (ObjNum 208 snap 53 seq_num 208) dirty exists 2021-07-31T01:47:43.528 INFO:tasks.rados.rados.0.smithi045.stdout:642: left oid 15 (ObjNum 208 snap 53 seq_num 208) 2021-07-31T01:47:43.528 INFO:tasks.rados.rados.0.smithi045.stdout:643: expect (ObjNum 101 snap 16 seq_num 101) 2021-07-31T01:47:43.573 INFO:tasks.rados.rados.0.smithi045.stdout:646: finishing write tid 1 to smithi04569269-30 2021-07-31T01:47:43.573 INFO:tasks.rados.rados.0.smithi045.stdout:646: finishing write tid 2 to smithi04569269-30 2021-07-31T01:47:43.574 INFO:tasks.rados.rados.0.smithi045.stdout:646: finishing write tid 3 to smithi04569269-30 2021-07-31T01:47:43.574 INFO:tasks.rados.rados.0.smithi045.stdout:646: finishing write tid 4 to smithi04569269-30 2021-07-31T01:47:43.574 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 30 v 112 (ObjNum 209 snap 53 seq_num 209) dirty exists 2021-07-31T01:47:43.575 INFO:tasks.rados.rados.0.smithi045.stdout:646: left oid 30 (ObjNum 209 snap 53 seq_num 209) 2021-07-31T01:47:43.575 INFO:tasks.rados.rados.0.smithi045.stdout:647: finishing write tid 1 to smithi04569269-35 2021-07-31T01:47:43.575 INFO:tasks.rados.rados.0.smithi045.stdout:647: finishing write tid 2 to smithi04569269-35 2021-07-31T01:47:43.575 INFO:tasks.rados.rados.0.smithi045.stdout:647: finishing write tid 3 to smithi04569269-35 2021-07-31T01:47:43.575 INFO:tasks.rados.rados.0.smithi045.stdout:647: finishing write tid 4 to smithi04569269-35 2021-07-31T01:47:43.576 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 35 v 467 (ObjNum 210 snap 53 seq_num 210) dirty exists 2021-07-31T01:47:43.576 INFO:tasks.rados.rados.0.smithi045.stdout:647: left oid 35 (ObjNum 210 snap 53 seq_num 210) 2021-07-31T01:47:43.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:43 smithi045 conmon[33655]: cluster 2021-07-31T01:47:42.500951+0000 mon.a ( 2021-07-31T01:47:43.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:43 smithi045 conmon[33655]: mon.0) 799 : cluster [DBG] osdmap e163: 8 total, 8 up, 8 in 2021-07-31T01:47:43.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:47:43 smithi045 conmon[61210]: level=error ts=2021-07-31T01:47:43.509Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:47:43.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:47:43 smithi045 conmon[61210]: level=error ts=2021-07-31T01:47:43.509Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:47:43.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:47:43 smithi045 conmon[61210]: level=error ts=2021-07-31T01:47:43.509Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T01:47:43.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:43 smithi045 conmon[39718]: cluster 2021-07-31T01:47:42.500951+0000 2021-07-31T01:47:43.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:43 smithi045 conmon[39718]: mon.a (mon.0) 799 : cluster [DBG] osdmap e163: 8 total, 8 up, 8 in 2021-07-31T01:47:44.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:43 smithi154 conmon[34553]: cluster 2021-07-31T01:47:42.500951+0000 mon.a (mon.0) 799 : cluster [DBG] osdmap e163: 8 total, 8 up, 8 in 2021-07-31T01:47:44.515 INFO:tasks.rados.rados.0.smithi045.stdout:641: done (5 left) 2021-07-31T01:47:44.516 INFO:tasks.rados.rados.0.smithi045.stdout:642: done (4 left) 2021-07-31T01:47:44.516 INFO:tasks.rados.rados.0.smithi045.stdout:643: done (3 left) 2021-07-31T01:47:44.517 INFO:tasks.rados.rados.0.smithi045.stdout:646: done (2 left) 2021-07-31T01:47:44.517 INFO:tasks.rados.rados.0.smithi045.stdout:647: done (1 left) 2021-07-31T01:47:44.517 INFO:tasks.rados.rados.0.smithi045.stdout:648: done (0 left) 2021-07-31T01:47:44.518 INFO:tasks.rados.rados.0.smithi045.stdout:649: rollback oid 13 current snap is 54 2021-07-31T01:47:44.518 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 13 to 53 2021-07-31T01:47:44.518 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 35 current snap is 54 2021-07-31T01:47:44.518 INFO:tasks.rados.rados.0.smithi045.stdout:650: seq_num 211 ranges {1122304=737280,1859584=401408} 2021-07-31T01:47:44.526 INFO:tasks.rados.rados.0.smithi045.stdout:650: writing smithi04569269-35 from 1122304 to 1859584 tid 1 2021-07-31T01:47:44.529 INFO:tasks.rados.rados.0.smithi045.stdout:650: writing smithi04569269-35 from 1859584 to 2260992 tid 2 2021-07-31T01:47:44.529 INFO:tasks.rados.rados.0.smithi045.stdout:651: read oid 1 snap -1 2021-07-31T01:47:44.529 INFO:tasks.rados.rados.0.smithi045.stdout:651: expect (ObjNum 155 snap 37 seq_num 155) 2021-07-31T01:47:44.530 INFO:tasks.rados.rados.0.smithi045.stdout:652: copy_from oid 17 from oid 38 current snap is 54 2021-07-31T01:47:44.530 INFO:tasks.rados.rados.0.smithi045.stdout:649: finishing rollback tid 0 to smithi04569269-13 2021-07-31T01:47:44.530 INFO:tasks.rados.rados.0.smithi045.stdout:649: finishing rollback tid 1 to smithi04569269-13 2021-07-31T01:47:44.530 INFO:tasks.rados.rados.0.smithi045.stdout:649: finishing rollback tid 2 to smithi04569269-13 2021-07-31T01:47:44.530 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 13 v 238 (ObjNum 177 snap 46 seq_num 177) dirty exists 2021-07-31T01:47:44.531 INFO:tasks.rados.rados.0.smithi045.stdout:649: done (3 left) 2021-07-31T01:47:44.531 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 6 current snap is 54 2021-07-31T01:47:44.531 INFO:tasks.rados.rados.0.smithi045.stdout:653: seq_num 212 ranges {0=753664,753664=401408} 2021-07-31T01:47:44.533 INFO:tasks.rados.rados.0.smithi045.stdout:653: writing smithi04569269-6 from 0 to 753664 tid 1 2021-07-31T01:47:44.536 INFO:tasks.rados.rados.0.smithi045.stdout:653: writing smithi04569269-6 from 753664 to 1155072 tid 2 2021-07-31T01:47:44.536 INFO:tasks.rados.rados.0.smithi045.stdout:654: setattr oid 27 current snap is 54 2021-07-31T01:47:44.537 INFO:tasks.rados.rados.0.smithi045.stdout:650: finishing write tid 1 to smithi04569269-35 2021-07-31T01:47:44.537 INFO:tasks.rados.rados.0.smithi045.stdout:651: expect (ObjNum 155 snap 37 seq_num 155) 2021-07-31T01:47:44.593 INFO:tasks.rados.rados.0.smithi045.stdout:650: finishing write tid 2 to smithi04569269-35 2021-07-31T01:47:44.594 INFO:tasks.rados.rados.0.smithi045.stdout:650: finishing write tid 3 to smithi04569269-35 2021-07-31T01:47:44.594 INFO:tasks.rados.rados.0.smithi045.stdout:650: finishing write tid 4 to smithi04569269-35 2021-07-31T01:47:44.594 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 35 v 471 (ObjNum 211 snap 54 seq_num 211) dirty exists 2021-07-31T01:47:44.595 INFO:tasks.rados.rados.0.smithi045.stdout:650: left oid 35 (ObjNum 211 snap 54 seq_num 211) 2021-07-31T01:47:44.595 INFO:tasks.rados.rados.0.smithi045.stdout:653: finishing write tid 1 to smithi04569269-6 2021-07-31T01:47:44.595 INFO:tasks.rados.rados.0.smithi045.stdout:653: finishing write tid 2 to smithi04569269-6 2021-07-31T01:47:44.596 INFO:tasks.rados.rados.0.smithi045.stdout:653: finishing write tid 3 to smithi04569269-6 2021-07-31T01:47:44.596 INFO:tasks.rados.rados.0.smithi045.stdout:653: finishing write tid 4 to smithi04569269-6 2021-07-31T01:47:44.596 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 6 v 242 (ObjNum 212 snap 54 seq_num 212) dirty exists 2021-07-31T01:47:44.597 INFO:tasks.rados.rados.0.smithi045.stdout:653: left oid 6 (ObjNum 212 snap 54 seq_num 212) 2021-07-31T01:47:44.597 INFO:tasks.rados.rados.0.smithi045.stdout:652: finishing copy_from to smithi04569269-17 2021-07-31T01:47:44.597 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 17 v 244 (ObjNum 196 snap 52 seq_num 196) dirty exists 2021-07-31T01:47:44.598 INFO:tasks.rados.rados.0.smithi045.stdout:652: finishing copy_from racing read to smithi04569269-17 2021-07-31T01:47:44.598 INFO:tasks.rados.rados.0.smithi045.stdout:650: done (4 left) 2021-07-31T01:47:44.598 INFO:tasks.rados.rados.0.smithi045.stdout:651: done (3 left) 2021-07-31T01:47:44.598 INFO:tasks.rados.rados.0.smithi045.stdout:652: done (2 left) 2021-07-31T01:47:44.598 INFO:tasks.rados.rados.0.smithi045.stdout:653: done (1 left) 2021-07-31T01:47:44.599 INFO:tasks.rados.rados.0.smithi045.stdout:655: setattr oid 34 current snap is 54 2021-07-31T01:47:44.599 INFO:tasks.rados.rados.0.smithi045.stdout:656: snap_create 2021-07-31T01:47:44.600 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 34 v 302 (ObjNum 76 snap 11 seq_num 76) dirty exists 2021-07-31T01:47:44.600 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 27 v 208 (ObjNum 0 snap 21873 seq_num 3189610128) dirty exists 2021-07-31T01:47:44.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:44 smithi045 conmon[33655]: cluster 2021-07-31T01:47:43.503005+0000 mon.a 2021-07-31T01:47:44.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:44 smithi045 conmon[33655]: (mon.0) 800 : cluster [DBG] osdmap e164: 8 total, 8 up, 8 in 2021-07-31T01:47:44.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:44 smithi045 conmon[33655]: cluster 2021-07-31T01:47:43.797931+0000 mgr.x (mgr.24542) 26 : cluster [DBG] 2021-07-31T01:47:44.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:44 smithi045 conmon[33655]: pgmap v26: 25 pgs: 25 active+clean; 105 MiB data, 409 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:47:44.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:44 smithi045 conmon[39718]: cluster 2021-07-31T01:47:43.503005+0000 2021-07-31T01:47:44.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:44 smithi045 conmon[39718]: mon.a (mon.0) 800 : cluster [DBG] osdmap e164: 8 total, 8 up, 8 in 2021-07-31T01:47:44.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:44 smithi045 conmon[39718]: cluster 2021 2021-07-31T01:47:44.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:44 smithi045 conmon[39718]: -07-31T01:47:43.797931+0000 mgr.x (mgr.24542) 26 : cluster [DBG] 2021-07-31T01:47:44.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:44 smithi045 conmon[39718]: pgmap v26: 25 pgs: 25 active+clean; 105 MiB data, 409 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:47:45.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:44 smithi154 conmon[34553]: cluster 2021-07-31T01:47:43.503005+0000 mon.a (mon.0) 800 : cluster [DBG] 2021-07-31T01:47:45.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:44 smithi154 conmon[34553]: osdmap e164: 8 total, 8 up, 8 in 2021-07-31T01:47:45.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:44 smithi154 conmon[34553]: cluster 2021-07-31T01:47:43.797931+0000 mgr.x (mgr.24542) 26 : cluster [DBG] pgmap v26: 25 pgs: 25 active+clean; 105 MiB data, 409 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:47:45.516 INFO:tasks.rados.rados.0.smithi045.stdout:654: done (2 left) 2021-07-31T01:47:45.517 INFO:tasks.rados.rados.0.smithi045.stdout:655: done (1 left) 2021-07-31T01:47:45.517 INFO:tasks.rados.rados.0.smithi045.stdout:656: done (0 left) 2021-07-31T01:47:45.517 INFO:tasks.rados.rados.0.smithi045.stdout:657: rollback oid 32 current snap is 55 2021-07-31T01:47:45.517 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 32 to 54 2021-07-31T01:47:45.518 INFO:tasks.rados.rados.0.smithi045.stdout:658: snap_remove snap 49 2021-07-31T01:47:45.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:45 smithi045 conmon[33655]: cluster 2021-07-31T01:47:44.515118+0000 2021-07-31T01:47:45.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:45 smithi045 conmon[33655]: mon.a (mon.0) 801 : cluster [DBG] osdmap e165: 8 total, 8 up, 8 in 2021-07-31T01:47:45.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:45 smithi045 conmon[39718]: cluster 2021-07-31T01:47:44.515118+0000 mon.a (mon.0) 801 2021-07-31T01:47:45.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:45 smithi045 conmon[39718]: : cluster [DBG] osdmap e165: 8 total, 8 up, 8 in 2021-07-31T01:47:46.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:45 smithi154 conmon[34553]: cluster 2021-07-31T01:47:44.515118+0000 mon.a (mon.0) 801 : cluster [DBG] osdmap e165: 8 total, 8 up, 8 in 2021-07-31T01:47:46.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:45 smithi154 conmon[34553]: 2021-07-31T01:47:46.529 INFO:tasks.rados.rados.0.smithi045.stdout:658: done (1 left) 2021-07-31T01:47:46.529 INFO:tasks.rados.rados.0.smithi045.stdout:659: rollback oid 46 current snap is 55 2021-07-31T01:47:46.529 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 46 to 40 2021-07-31T01:47:46.529 INFO:tasks.rados.rados.0.smithi045.stdout:657: finishing rollback tid 0 to smithi04569269-32 2021-07-31T01:47:46.530 INFO:tasks.rados.rados.0.smithi045.stdout:657: finishing rollback tid 1 to smithi04569269-32 2021-07-31T01:47:46.530 INFO:tasks.rados.rados.0.smithi045.stdout:657: finishing rollback tid 2 to smithi04569269-32 2021-07-31T01:47:46.530 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 32 v 82 (ObjNum 140 snap 30 seq_num 140) dirty exists 2021-07-31T01:47:46.530 INFO:tasks.rados.rados.0.smithi045.stdout:657: done (1 left) 2021-07-31T01:47:46.531 INFO:tasks.rados.rados.0.smithi045.stdout:660: snap_remove snap 53 2021-07-31T01:47:46.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:46 smithi045 conmon[33655]: cluster 2021-07-31T01:47:45.515593+0000 mon.a (mon.0) 802 : 2021-07-31T01:47:46.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:46 smithi045 conmon[33655]: cluster [DBG] osdmap e166: 8 total, 8 up, 8 in 2021-07-31T01:47:46.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:46 smithi045 conmon[33655]: cluster 2021-07-31T01:47:45.798400+0000 mgr.x (mgr.24542) 27 : cluster [DBG] pgmap v29: 25 pgs: 25 active+clean; 83 MiB data, 404 MiB used, 715 GiB / 715 GiB avail; 5.0 MiB/s rd, 1.3 MiB/s wr, 17 op/s 2021-07-31T01:47:46.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:46 smithi045 conmon[39718]: cluster 2021-07-31T01:47:45.515593 2021-07-31T01:47:46.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:46 smithi045 conmon[39718]: +0000 mon.a (mon.0) 802 : cluster [DBG] osdmap e166: 8 total, 8 up, 8 in 2021-07-31T01:47:46.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:46 smithi045 conmon[39718]: cluster 2021-07-31T 2021-07-31T01:47:46.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:46 smithi045 conmon[39718]: 01:47:45.798400+0000 mgr.x (mgr.24542) 27 : cluster [DBG] pgmap v29: 25 pgs: 25 active+clean; 83 MiB data, 404 MiB used, 715 GiB / 715 GiB avail; 5.0 MiB/s rd, 1.3 MiB/s wr, 17 op/s 2021-07-31T01:47:47.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:46 smithi154 conmon[34553]: cluster 2021-07-31T01:47:45.515593+0000 mon.a (mon.0) 802 2021-07-31T01:47:47.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:46 smithi154 conmon[34553]: : cluster [DBG] osdmap e166: 8 total, 8 up, 8 in 2021-07-31T01:47:47.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:46 smithi154 conmon[34553]: cluster 2021-07-31T01:47:45.798400+0000 mgr.x (mgr.24542) 27 : cluster [DBG] pgmap v29: 25 pgs: 25 active+clean; 83 MiB data, 404 MiB used, 715 GiB / 715 GiB avail; 5.0 MiB/s rd, 1.3 MiB/s wr, 17 op/s 2021-07-31T01:47:47.004 INFO:journalctl@ceph.osd.4.smithi154.stdout:Jul 31 01:47:46 smithi154 conmon[38506]: debug 2021-07-31T01:47:47.004 INFO:journalctl@ceph.osd.4.smithi154.stdout:Jul 31 01:47:46 smithi154 conmon[38506]: 2021-07-31T01:47:46.531+0000 7feae29cd700 -1 osd.4 pg_epoch: 167 pg[3.3s0( v 163'49 (0'0,163'49] local-lis/les=56/57 n=2 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [4,1,0]p4(0) r=0 lpr=56 crt=163'49 lcod 162'47 mlcod 162'47 active+clean+snaptrim trimq=[5d~1] ps=[42~1,4a~1,5c~1,61~1,66~1]] removing snap head 2021-07-31T01:47:47.004 INFO:journalctl@ceph.osd.4.smithi154.stdout:Jul 31 01:47:46 smithi154 conmon[38506]: 2021-07-31T01:47:47.531 INFO:tasks.rados.rados.0.smithi045.stdout:660: done (1 left) 2021-07-31T01:47:47.532 INFO:tasks.rados.rados.0.smithi045.stdout:661: read oid 21 snap -1 2021-07-31T01:47:47.532 INFO:tasks.rados.rados.0.smithi045.stdout:661: expect deleted 2021-07-31T01:47:47.532 INFO:tasks.rados.rados.0.smithi045.stdout:662: read oid 27 snap 40 2021-07-31T01:47:47.532 INFO:tasks.rados.rados.0.smithi045.stdout:662: expect (ObjNum -1105360576 snap 21873 seq_num 3190207248) 2021-07-31T01:47:47.532 INFO:tasks.rados.rados.0.smithi045.stdout:663: rollback oid 2 current snap is 55 2021-07-31T01:47:47.533 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 2 to 40 2021-07-31T01:47:47.533 INFO:tasks.rados.rados.0.smithi045.stdout:659: finishing rollback tid 1 to smithi04569269-46 2021-07-31T01:47:47.533 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 46 v 0 (ObjNum 1 snap 21873 seq_num 3189250944) dirty dne 2021-07-31T01:47:47.533 INFO:tasks.rados.rados.0.smithi045.stdout:659: done (3 left) 2021-07-31T01:47:47.534 INFO:tasks.rados.rados.0.smithi045.stdout:664: copy_from oid 8 from oid 29 current snap is 55 2021-07-31T01:47:47.534 INFO:tasks.rados.rados.0.smithi045.stdout:665: copy_from oid 30 from oid 46 current snap is 55 2021-07-31T01:47:47.534 INFO:tasks.rados.rados.0.smithi045.stdout:666: delete oid 31 current snap is 55 2021-07-31T01:47:47.535 INFO:tasks.rados.rados.0.smithi045.stdout:662: expect (ObjNum -1105360576 snap 21873 seq_num 3190207248) 2021-07-31T01:47:47.535 INFO:tasks.rados.rados.0.smithi045.stdout:663: finishing rollback tid 1 to smithi04569269-2 2021-07-31T01:47:47.535 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 2 v 0 (ObjNum 1 snap 32731 seq_num 3189211808) dirty dne 2021-07-31T01:47:47.537 INFO:tasks.rados.rados.0.smithi045.stdout:664: finishing copy_from to smithi04569269-8 2021-07-31T01:47:47.538 INFO:tasks.rados.rados.0.smithi045.stdout:664: got expected ENOENT (src dne) 2021-07-31T01:47:47.538 INFO:tasks.rados.rados.0.smithi045.stdout:665: finishing copy_from to smithi04569269-30 2021-07-31T01:47:47.538 INFO:tasks.rados.rados.0.smithi045.stdout:665: got expected ENOENT (src dne) 2021-07-31T01:47:47.539 INFO:tasks.rados.rados.0.smithi045.stdout:664: finishing copy_from racing read to smithi04569269-8 2021-07-31T01:47:47.539 INFO:tasks.rados.rados.0.smithi045.stdout:665: finishing copy_from racing read to smithi04569269-30 2021-07-31T01:47:47.539 INFO:tasks.rados.rados.0.smithi045.stdout:661: done (5 left) 2021-07-31T01:47:47.539 INFO:tasks.rados.rados.0.smithi045.stdout:662: done (4 left) 2021-07-31T01:47:47.539 INFO:tasks.rados.rados.0.smithi045.stdout:663: done (3 left) 2021-07-31T01:47:47.540 INFO:tasks.rados.rados.0.smithi045.stdout:664: done (2 left) 2021-07-31T01:47:47.540 INFO:tasks.rados.rados.0.smithi045.stdout:665: done (1 left) 2021-07-31T01:47:47.540 INFO:tasks.rados.rados.0.smithi045.stdout:666: done (0 left) 2021-07-31T01:47:47.540 INFO:tasks.rados.rados.0.smithi045.stdout:667: snap_create 2021-07-31T01:47:48.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:47 smithi154 conmon[34553]: cluster 2021-07-31T01:47:46.527799+0000 mon.a (mon.0) 803 : cluster 2021-07-31T01:47:48.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:47 smithi154 conmon[34553]: [DBG] osdmap e167: 8 total, 8 up, 8 in 2021-07-31T01:47:48.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:47 smithi045 conmon[33655]: cluster 2021-07-31T01:47:48.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:47 smithi045 conmon[33655]: 2021-07-31T01:47:48.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:47 smithi045 conmon[33655]: 2021-07-31T01:47:46.527799+0000 mon.a (mon.0) 803 : cluster [DBG] 2021-07-31T01:47:48.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:47 smithi045 conmon[33655]: osdmap e167: 8 total, 8 up, 8 in 2021-07-31T01:47:48.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:47 smithi045 conmon[39718]: cluster 2021-07-31T01:47:48.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:47 smithi045 conmon[39718]: 2021-07-31T01:47:48.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:47 smithi045 conmon[39718]: 2021 2021-07-31T01:47:48.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:47 smithi045 conmon[39718]: - 2021-07-31T01:47:48.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:47 smithi045 conmon[39718]: 07 2021-07-31T01:47:48.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:47 smithi045 conmon[39718]: - 2021-07-31T01:47:48.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:47 smithi045 conmon[39718]: 31 2021-07-31T01:47:48.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:47 smithi045 conmon[39718]: T 2021-07-31T01:47:48.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:47 smithi045 conmon[39718]: 01 2021-07-31T01:47:48.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:47 smithi045 conmon[39718]: :47:46.527799+0000 2021-07-31T01:47:48.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:47 smithi045 conmon[39718]: mon.a (mon.0) 803 : cluster [DBG] osdmap e167: 8 total, 8 up, 8 in 2021-07-31T01:47:48.540 INFO:tasks.rados.rados.0.smithi045.stdout:667: done (0 left) 2021-07-31T01:47:48.540 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 26 current snap is 56 2021-07-31T01:47:48.540 INFO:tasks.rados.rados.0.smithi045.stdout:668: seq_num 214 ranges {1703936=794624,2498560=401408} 2021-07-31T01:47:48.553 INFO:tasks.rados.rados.0.smithi045.stdout:668: writing smithi04569269-26 from 1703936 to 2498560 tid 1 2021-07-31T01:47:48.555 INFO:tasks.rados.rados.0.smithi045.stdout:668: writing smithi04569269-26 from 2498560 to 2899968 tid 2 2021-07-31T01:47:48.556 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 6 current snap is 56 2021-07-31T01:47:48.556 INFO:tasks.rados.rados.0.smithi045.stdout:669: seq_num 215 ranges {1155072=409600,1564672=532480,2097152=270336} 2021-07-31T01:47:48.564 INFO:tasks.rados.rados.0.smithi045.stdout:669: writing smithi04569269-6 from 1155072 to 1564672 tid 1 2021-07-31T01:47:48.567 INFO:tasks.rados.rados.0.smithi045.stdout:669: writing smithi04569269-6 from 1564672 to 2097152 tid 2 2021-07-31T01:47:48.569 INFO:tasks.rados.rados.0.smithi045.stdout:669: writing smithi04569269-6 from 2097152 to 2367488 tid 3 2021-07-31T01:47:48.570 INFO:tasks.rados.rados.0.smithi045.stdout:670: copy_from oid 41 from oid 39 current snap is 56 2021-07-31T01:47:48.570 INFO:tasks.rados.rados.0.smithi045.stdout:668: finishing write tid 1 to smithi04569269-26 2021-07-31T01:47:48.570 INFO:tasks.rados.rados.0.smithi045.stdout:668: finishing write tid 2 to smithi04569269-26 2021-07-31T01:47:48.571 INFO:tasks.rados.rados.0.smithi045.stdout:668: finishing write tid 3 to smithi04569269-26 2021-07-31T01:47:48.571 INFO:tasks.rados.rados.0.smithi045.stdout:668: finishing write tid 4 to smithi04569269-26 2021-07-31T01:47:48.571 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 26 v 90 (ObjNum 214 snap 56 seq_num 214) dirty exists 2021-07-31T01:47:48.571 INFO:tasks.rados.rados.0.smithi045.stdout:668: left oid 26 (ObjNum 214 snap 56 seq_num 214) 2021-07-31T01:47:48.571 INFO:tasks.rados.rados.0.smithi045.stdout:669: finishing write tid 1 to smithi04569269-6 2021-07-31T01:47:48.572 INFO:tasks.rados.rados.0.smithi045.stdout:668: done (2 left) 2021-07-31T01:47:48.572 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 19 current snap is 56 2021-07-31T01:47:48.572 INFO:tasks.rados.rados.0.smithi045.stdout:671: seq_num 216 ranges {1638400=425984,2064384=409600,2473984=393216} 2021-07-31T01:47:48.580 INFO:tasks.rados.rados.0.smithi045.stdout:671: writing smithi04569269-19 from 1638400 to 2064384 tid 1 2021-07-31T01:47:48.583 INFO:tasks.rados.rados.0.smithi045.stdout:671: writing smithi04569269-19 from 2064384 to 2473984 tid 2 2021-07-31T01:47:48.586 INFO:tasks.rados.rados.0.smithi045.stdout:671: writing smithi04569269-19 from 2473984 to 2867200 tid 3 2021-07-31T01:47:48.586 INFO:tasks.rados.rados.0.smithi045.stdout:672: delete oid 46 current snap is 56 2021-07-31T01:47:48.586 INFO:tasks.rados.rados.0.smithi045.stdout:669: finishing write tid 2 to smithi04569269-6 2021-07-31T01:47:48.586 INFO:tasks.rados.rados.0.smithi045.stdout:669: finishing write tid 3 to smithi04569269-6 2021-07-31T01:47:48.587 INFO:tasks.rados.rados.0.smithi045.stdout:669: finishing write tid 4 to smithi04569269-6 2021-07-31T01:47:48.587 INFO:tasks.rados.rados.0.smithi045.stdout:669: finishing write tid 5 to smithi04569269-6 2021-07-31T01:47:48.587 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 6 v 257 (ObjNum 215 snap 56 seq_num 215) dirty exists 2021-07-31T01:47:48.587 INFO:tasks.rados.rados.0.smithi045.stdout:669: left oid 6 (ObjNum 215 snap 56 seq_num 215) 2021-07-31T01:47:48.588 INFO:tasks.rados.rados.0.smithi045.stdout:671: finishing write tid 1 to smithi04569269-19 2021-07-31T01:47:48.588 INFO:tasks.rados.rados.0.smithi045.stdout:670: finishing copy_from to smithi04569269-41 2021-07-31T01:47:48.588 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 41 v 248 (ObjNum 197 snap 52 seq_num 197) dirty exists 2021-07-31T01:47:48.589 INFO:tasks.rados.rados.0.smithi045.stdout:670: finishing copy_from racing read to smithi04569269-41 2021-07-31T01:47:48.589 INFO:tasks.rados.rados.0.smithi045.stdout:669: done (3 left) 2021-07-31T01:47:48.589 INFO:tasks.rados.rados.0.smithi045.stdout:670: done (2 left) 2021-07-31T01:47:48.590 INFO:tasks.rados.rados.0.smithi045.stdout:672: done (1 left) 2021-07-31T01:47:48.590 INFO:tasks.rados.rados.0.smithi045.stdout:673: snap_remove snap 55 2021-07-31T01:47:49.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:48 smithi154 conmon[34553]: cluster 2021-07-31T01:47:47.528727+0000 mon.a (mon.0) 804 : cluster [DBG] osdmap e168: 8 total, 8 up, 8 in 2021-07-31T01:47:49.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:48 smithi154 conmon[34553]: 2021-07-31T01:47:49.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:48 smithi154 conmon[34553]: cluster 2021-07-31T01:47:47.798792+0000 mgr.x (mgr.24542) 28 : cluster [DBG] pgmap v32: 25 pgs: 25 active+clean; 83 MiB data, 404 MiB used, 715 GiB / 715 GiB avail; 5.0 MiB/s rd, 1.3 MiB/s wr, 17 op/s 2021-07-31T01:47:49.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:48 smithi045 conmon[39718]: cluster 2021-07-31T01:47:47.528727+0000 mon.a ( 2021-07-31T01:47:49.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:48 smithi045 conmon[39718]: mon.0) 804 : cluster [DBG] osdmap e168: 8 total, 8 up, 8 in 2021-07-31T01:47:49.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:48 smithi045 conmon[39718]: cluster 2021-07-31T01:47:47. 2021-07-31T01:47:49.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:48 smithi045 conmon[39718]: 798792+0000 mgr.x (mgr.24542) 28 : cluster [DBG] pgmap v32: 25 pgs: 25 active+clean; 83 MiB data, 404 MiB used, 715 GiB / 715 GiB avail; 5.0 MiB/s rd, 1.3 MiB/s wr, 17 op/s 2021-07-31T01:47:49.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:48 smithi045 conmon[33655]: cluster 2021-07-31T01:47:47.528727+0000 mon.a (mon. 2021-07-31T01:47:49.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:48 smithi045 conmon[33655]: 0) 804 : cluster [DBG] osdmap e168: 8 total, 8 up, 8 in 2021-07-31T01:47:49.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:48 smithi045 conmon[33655]: cluster 2021-07-31T01:47:47.798792+0000 mgr.x (mgr.24542) 28 : cluster 2021-07-31T01:47:49.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:48 smithi045 conmon[33655]: [DBG] pgmap v32: 25 pgs: 25 active+clean; 83 MiB data, 404 MiB used, 715 GiB / 715 GiB avail; 5.0 MiB/s rd, 1.3 MiB/s wr, 17 op/s 2021-07-31T01:47:49.545 INFO:tasks.rados.rados.0.smithi045.stdout:673: done (1 left) 2021-07-31T01:47:49.545 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 36 current snap is 56 2021-07-31T01:47:49.545 INFO:tasks.rados.rados.0.smithi045.stdout:671: finishing write tid 2 to smithi04569269-19 2021-07-31T01:47:49.546 INFO:tasks.rados.rados.0.smithi045.stdout:674: seq_num 218 ranges {0=458752,458752=581632,1040384=221184} 2021-07-31T01:47:49.547 INFO:tasks.rados.rados.0.smithi045.stdout:674: writing smithi04569269-36 from 0 to 458752 tid 1 2021-07-31T01:47:49.555 INFO:tasks.rados.rados.0.smithi045.stdout:674: writing smithi04569269-36 from 458752 to 1040384 tid 2 2021-07-31T01:47:49.555 INFO:tasks.rados.rados.0.smithi045.stdout:674: writing smithi04569269-36 from 1040384 to 1261568 tid 3 2021-07-31T01:47:49.556 INFO:tasks.rados.rados.0.smithi045.stdout:675: read oid 17 snap -1 2021-07-31T01:47:49.556 INFO:tasks.rados.rados.0.smithi045.stdout:675: expect (ObjNum 196 snap 52 seq_num 196) 2021-07-31T01:47:49.556 INFO:tasks.rados.rados.0.smithi045.stdout:676: snap_create 2021-07-31T01:47:49.556 INFO:tasks.rados.rados.0.smithi045.stdout:671: finishing write tid 3 to smithi04569269-19 2021-07-31T01:47:49.557 INFO:tasks.rados.rados.0.smithi045.stdout:671: finishing write tid 4 to smithi04569269-19 2021-07-31T01:47:49.557 INFO:tasks.rados.rados.0.smithi045.stdout:671: finishing write tid 5 to smithi04569269-19 2021-07-31T01:47:49.557 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 19 v 171 (ObjNum 216 snap 56 seq_num 216) dirty exists 2021-07-31T01:47:49.557 INFO:tasks.rados.rados.0.smithi045.stdout:671: left oid 19 (ObjNum 216 snap 56 seq_num 216) 2021-07-31T01:47:49.558 INFO:tasks.rados.rados.0.smithi045.stdout:674: finishing write tid 1 to smithi04569269-36 2021-07-31T01:47:49.559 INFO:tasks.rados.rados.0.smithi045.stdout:674: finishing write tid 2 to smithi04569269-36 2021-07-31T01:47:49.559 INFO:tasks.rados.rados.0.smithi045.stdout:674: finishing write tid 3 to smithi04569269-36 2021-07-31T01:47:49.559 INFO:tasks.rados.rados.0.smithi045.stdout:674: finishing write tid 4 to smithi04569269-36 2021-07-31T01:47:49.560 INFO:tasks.rados.rados.0.smithi045.stdout:674: finishing write tid 5 to smithi04569269-36 2021-07-31T01:47:49.560 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 36 v 481 (ObjNum 218 snap 56 seq_num 218) dirty exists 2021-07-31T01:47:49.560 INFO:tasks.rados.rados.0.smithi045.stdout:674: left oid 36 (ObjNum 218 snap 56 seq_num 218) 2021-07-31T01:47:49.579 INFO:tasks.rados.rados.0.smithi045.stdout:675: expect (ObjNum 196 snap 52 seq_num 196) 2021-07-31T01:47:50.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:49 smithi154 conmon[34553]: cluster 2021-07-31T01:47:48.538915+0000 mon.a (mon.0) 805 : cluster [DBG] osdmap e169: 8 total, 8 up, 8 in 2021-07-31T01:47:50.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:49 smithi045 conmon[39718]: cluster 2021-07-31T01:47:48.538915+0000 mon.a (mon. 2021-07-31T01:47:50.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:49 smithi045 conmon[39718]: 0) 805 : cluster [DBG] osdmap e169: 8 total, 8 up, 8 in 2021-07-31T01:47:50.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:49 smithi045 conmon[33655]: cluster 2021-07-31T01:47:48.538915+0000 mon.a (mon. 2021-07-31T01:47:50.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:49 smithi045 conmon[33655]: 0) 805 : cluster [DBG] osdmap e169: 8 total, 8 up, 8 in 2021-07-31T01:47:50.551 INFO:tasks.rados.rados.0.smithi045.stdout:671: done (3 left) 2021-07-31T01:47:50.551 INFO:tasks.rados.rados.0.smithi045.stdout:674: done (2 left) 2021-07-31T01:47:50.552 INFO:tasks.rados.rados.0.smithi045.stdout:675: done (1 left) 2021-07-31T01:47:50.552 INFO:tasks.rados.rados.0.smithi045.stdout:676: done (0 left) 2021-07-31T01:47:50.552 INFO:tasks.rados.rados.0.smithi045.stdout:677: setattr oid 33 current snap is 57 2021-07-31T01:47:50.553 INFO:tasks.rados.rados.0.smithi045.stdout:678: read oid 6 snap -1 2021-07-31T01:47:50.553 INFO:tasks.rados.rados.0.smithi045.stdout:678: expect (ObjNum 215 snap 56 seq_num 215) 2021-07-31T01:47:50.553 INFO:tasks.rados.rados.0.smithi045.stdout:679: delete oid 49 current snap is 57 2021-07-31T01:47:50.558 INFO:tasks.rados.rados.0.smithi045.stdout:679: done (2 left) 2021-07-31T01:47:50.558 INFO:tasks.rados.rados.0.smithi045.stdout:680: read oid 37 snap -1 2021-07-31T01:47:50.558 INFO:tasks.rados.rados.0.smithi045.stdout:680: expect (ObjNum 0 snap 0 seq_num 0) 2021-07-31T01:47:50.559 INFO:tasks.rados.rados.0.smithi045.stdout:681: snap_remove snap 56 2021-07-31T01:47:51.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:50 smithi154 conmon[34553]: cluster 2021-07-31T01:47:49.544273+0000 mon.a (mon.0) 806 : cluster [DBG] osdmap e170: 8 total, 8 up, 8 in 2021-07-31T01:47:51.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:50 smithi154 conmon[34553]: cluster 2021-07-31T01:47:49.799332+0000 mgr.x (mgr.24542) 29 : cluster [DBG] pgmap v35: 25 pgs: 25 active+clean; 94 MiB data, 430 MiB used, 715 GiB / 715 GiB avail; 1.0 MiB/s rd, 3.2 MiB/s wr, 20 op/s 2021-07-31T01:47:51.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:50 smithi045 conmon[33655]: cluster 2021-07-31T01:47:49.544273+0000 mon.a (mon.0) 806 : cluster [DBG] 2021-07-31T01:47:51.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:50 smithi045 conmon[33655]: osdmap e170: 8 total, 8 up, 8 in 2021-07-31T01:47:51.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:50 smithi045 conmon[33655]: cluster 2021-07-31T01:47:49.799332+0000 mgr.x (mgr.24542) 29 : cluster [DBG] pgmap v35: 25 pgs: 25 active+clean; 94 MiB data, 430 MiB used, 715 GiB / 715 GiB avail; 1.0 MiB/s rd, 3.2 MiB/s wr, 20 op/s 2021-07-31T01:47:51.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:50 smithi045 conmon[39718]: cluster 2021-07-31T01:47:49.544273+0000 mon.a 2021-07-31T01:47:51.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:50 smithi045 conmon[39718]: (mon.0) 806 : cluster [DBG] osdmap e170: 8 total, 8 up, 8 in 2021-07-31T01:47:51.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:50 smithi045 conmon[39718]: cluster 2021-07-31T01 2021-07-31T01:47:51.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:50 smithi045 conmon[39718]: :47:49.799332+0000 mgr.x (mgr.24542) 29 : cluster [DBG] pgmap v35: 25 pgs: 25 active+clean; 94 MiB data, 430 MiB used, 715 GiB / 715 GiB avail; 1.0 MiB/s rd, 3.2 MiB/s wr, 20 op/s 2021-07-31T01:47:51.495 INFO:tasks.rados.rados.0.smithi045.stdout:681: done (3 left) 2021-07-31T01:47:51.496 INFO:tasks.rados.rados.0.smithi045.stdout:682: rmattr oid 2 current snap is 57 2021-07-31T01:47:51.496 INFO:tasks.rados.rados.0.smithi045.stdout:682: done (3 left) 2021-07-31T01:47:51.496 INFO:tasks.rados.rados.0.smithi045.stdout:683: delete oid 45 current snap is 57 2021-07-31T01:47:51.496 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 33 v 291 (ObjNum 94 snap 15 seq_num 94) dirty exists 2021-07-31T01:47:51.497 INFO:tasks.rados.rados.0.smithi045.stdout:680: expect (ObjNum 0 snap 0 seq_num 0) 2021-07-31T01:47:51.498 INFO:tasks.rados.rados.0.smithi045.stdout:678: expect (ObjNum 215 snap 56 seq_num 215) 2021-07-31T01:47:51.584 INFO:tasks.rados.rados.0.smithi045.stdout:677: done (3 left) 2021-07-31T01:47:51.584 INFO:tasks.rados.rados.0.smithi045.stdout:678: done (2 left) 2021-07-31T01:47:51.585 INFO:tasks.rados.rados.0.smithi045.stdout:680: done (1 left) 2021-07-31T01:47:51.585 INFO:tasks.rados.rados.0.smithi045.stdout:683: done (0 left) 2021-07-31T01:47:51.585 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 4 current snap is 57 2021-07-31T01:47:51.585 INFO:tasks.rados.rados.0.smithi045.stdout:684: seq_num 221 ranges {548864=507904,1056768=688128,1744896=114688} 2021-07-31T01:47:51.589 INFO:tasks.rados.rados.0.smithi045.stdout:684: writing smithi04569269-4 from 548864 to 1056768 tid 1 2021-07-31T01:47:51.593 INFO:tasks.rados.rados.0.smithi045.stdout:684: writing smithi04569269-4 from 1056768 to 1744896 tid 2 2021-07-31T01:47:51.594 INFO:tasks.rados.rados.0.smithi045.stdout:684: writing smithi04569269-4 from 1744896 to 1859584 tid 3 2021-07-31T01:47:51.595 INFO:tasks.rados.rados.0.smithi045.stdout:685: read oid 11 snap -1 2021-07-31T01:47:51.595 INFO:tasks.rados.rados.0.smithi045.stdout:685: expect (ObjNum 10 snap 0 seq_num 10) 2021-07-31T01:47:51.595 INFO:tasks.rados.rados.0.smithi045.stdout:686: copy_from oid 43 from oid 2 current snap is 57 2021-07-31T01:47:51.595 INFO:tasks.rados.rados.0.smithi045.stdout:684: finishing write tid 1 to smithi04569269-4 2021-07-31T01:47:51.596 INFO:tasks.rados.rados.0.smithi045.stdout:687: delete oid 27 current snap is 57 2021-07-31T01:47:51.597 INFO:tasks.rados.rados.0.smithi045.stdout:686: finishing copy_from to smithi04569269-43 2021-07-31T01:47:51.597 INFO:tasks.rados.rados.0.smithi045.stdout:686: got expected ENOENT (src dne) 2021-07-31T01:47:51.597 INFO:tasks.rados.rados.0.smithi045.stdout:686: finishing copy_from racing read to smithi04569269-43 2021-07-31T01:47:51.598 INFO:tasks.rados.rados.0.smithi045.stdout:685: expect (ObjNum 10 snap 0 seq_num 10) 2021-07-31T01:47:51.606 INFO:tasks.rados.rados.0.smithi045.stdout:684: finishing write tid 2 to smithi04569269-4 2021-07-31T01:47:51.606 INFO:tasks.rados.rados.0.smithi045.stdout:684: finishing write tid 3 to smithi04569269-4 2021-07-31T01:47:51.607 INFO:tasks.rados.rados.0.smithi045.stdout:684: finishing write tid 4 to smithi04569269-4 2021-07-31T01:47:51.607 INFO:tasks.rados.rados.0.smithi045.stdout:684: finishing write tid 5 to smithi04569269-4 2021-07-31T01:47:51.607 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 4 v 194 (ObjNum 221 snap 57 seq_num 221) dirty exists 2021-07-31T01:47:51.607 INFO:tasks.rados.rados.0.smithi045.stdout:684: left oid 4 (ObjNum 221 snap 57 seq_num 221) 2021-07-31T01:47:51.608 INFO:tasks.rados.rados.0.smithi045.stdout:684: done (3 left) 2021-07-31T01:47:51.608 INFO:tasks.rados.rados.0.smithi045.stdout:685: done (2 left) 2021-07-31T01:47:51.608 INFO:tasks.rados.rados.0.smithi045.stdout:686: done (1 left) 2021-07-31T01:47:51.608 INFO:tasks.rados.rados.0.smithi045.stdout:687: done (0 left) 2021-07-31T01:47:51.608 INFO:tasks.rados.rados.0.smithi045.stdout:688: read oid 37 snap -1 2021-07-31T01:47:51.609 INFO:tasks.rados.rados.0.smithi045.stdout:688: expect (ObjNum 0 snap 0 seq_num 0) 2021-07-31T01:47:51.609 INFO:tasks.rados.rados.0.smithi045.stdout:689: copy_from oid 24 from oid 43 current snap is 57 2021-07-31T01:47:51.609 INFO:tasks.rados.rados.0.smithi045.stdout:690: read oid 45 snap -1 2021-07-31T01:47:51.609 INFO:tasks.rados.rados.0.smithi045.stdout:690: expect deleted 2021-07-31T01:47:51.610 INFO:tasks.rados.rados.0.smithi045.stdout:691: read oid 5 snap -1 2021-07-31T01:47:51.610 INFO:tasks.rados.rados.0.smithi045.stdout:691: expect (ObjNum 158 snap 37 seq_num 158) 2021-07-31T01:47:51.610 INFO:tasks.rados.rados.0.smithi045.stdout:692: setattr oid 30 current snap is 57 2021-07-31T01:47:51.610 INFO:tasks.rados.rados.0.smithi045.stdout:688: expect (ObjNum 0 snap 0 seq_num 0) 2021-07-31T01:47:51.611 INFO:tasks.rados.rados.0.smithi045.stdout:688: done (4 left) 2021-07-31T01:47:51.611 INFO:tasks.rados.rados.0.smithi045.stdout:690: done (3 left) 2021-07-31T01:47:51.612 INFO:tasks.rados.rados.0.smithi045.stdout:693: rollback oid 13 current snap is 57 2021-07-31T01:47:51.612 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 13 to 40 2021-07-31T01:47:51.612 INFO:tasks.rados.rados.0.smithi045.stdout:694: delete oid 33 current snap is 57 2021-07-31T01:47:51.612 INFO:tasks.rados.rados.0.smithi045.stdout:691: expect (ObjNum 158 snap 37 seq_num 158) 2021-07-31T01:47:51.618 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 30 v 116 (ObjNum 209 snap 53 seq_num 209) dirty exists 2021-07-31T01:47:51.619 INFO:tasks.rados.rados.0.smithi045.stdout:693: finishing rollback tid 0 to smithi04569269-13 2021-07-31T01:47:51.619 INFO:tasks.rados.rados.0.smithi045.stdout:693: finishing rollback tid 1 to smithi04569269-13 2021-07-31T01:47:51.619 INFO:tasks.rados.rados.0.smithi045.stdout:693: finishing rollback tid 2 to smithi04569269-13 2021-07-31T01:47:51.619 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 13 v 263 (ObjNum 162 snap 38 seq_num 162) dirty exists 2021-07-31T01:47:51.620 INFO:tasks.rados.rados.0.smithi045.stdout:691: done (4 left) 2021-07-31T01:47:51.620 INFO:tasks.rados.rados.0.smithi045.stdout:692: done (3 left) 2021-07-31T01:47:51.620 INFO:tasks.rados.rados.0.smithi045.stdout:693: done (2 left) 2021-07-31T01:47:51.620 INFO:tasks.rados.rados.0.smithi045.stdout:694: done (1 left) 2021-07-31T01:47:51.620 INFO:tasks.rados.rados.0.smithi045.stdout:695: rmattr oid 40 current snap is 57 2021-07-31T01:47:51.621 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 36 current snap is 57 2021-07-31T01:47:51.621 INFO:tasks.rados.rados.0.smithi045.stdout:696: seq_num 224 ranges {1261568=557056,1818624=737280,2555904=65536} 2021-07-31T01:47:51.629 INFO:tasks.rados.rados.0.smithi045.stdout:696: writing smithi04569269-36 from 1261568 to 1818624 tid 1 2021-07-31T01:47:51.633 INFO:tasks.rados.rados.0.smithi045.stdout:696: writing smithi04569269-36 from 1818624 to 2555904 tid 2 2021-07-31T01:47:51.634 INFO:tasks.rados.rados.0.smithi045.stdout:696: writing smithi04569269-36 from 2555904 to 2621440 tid 3 2021-07-31T01:47:51.634 INFO:tasks.rados.rados.0.smithi045.stdout:697: snap_remove snap 40 2021-07-31T01:47:52.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:51 smithi154 conmon[34553]: cluster 2021-07-31T01:47:50.550317+0000 mon.a (mon.0) 807 : cluster [DBG] osdmap e171: 8 total, 8 up, 8 in 2021-07-31T01:47:52.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:51 smithi154 conmon[34553]: cluster 2021-07-31T01:47:51.494622+0000 mon.a (mon.0) 808 : cluster [DBG] osdmap e172: 8 total, 8 up, 8 in 2021-07-31T01:47:52.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:51 smithi045 conmon[39718]: cluster 2021-07-31T01:47:50.550317+0000 mon.a ( 2021-07-31T01:47:52.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:51 smithi045 conmon[39718]: mon.0) 807 : cluster [DBG] osdmap e171: 8 total, 8 up, 8 in 2021-07-31T01:47:52.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:51 smithi045 conmon[39718]: cluster 2021-07- 2021-07-31T01:47:52.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:51 smithi045 conmon[39718]: 31T01:47:51.494622+0000 mon.a (mon.0) 808 : cluster [DBG] osdmap e172: 8 total, 8 up, 8 in 2021-07-31T01:47:52.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:51 smithi045 conmon[39718]: 2021-07-31T01:47:52.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:51 smithi045 conmon[33655]: cluster 2021-07-31T01:47:50.550317+0000 mon.a ( 2021-07-31T01:47:52.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:51 smithi045 conmon[33655]: mon.0) 807 : cluster [DBG] osdmap e171: 8 total, 8 up, 8 in 2021-07-31T01:47:52.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:51 smithi045 conmon[33655]: cluster 2021-07-31T01:47:51.494622+0000 mon.a (mon.0) 808 : cluster [DBG] osdmap e172: 8 total, 8 up, 8 in 2021-07-31T01:47:52.558 INFO:tasks.rados.rados.0.smithi045.stdout:697: done (3 left) 2021-07-31T01:47:52.559 INFO:tasks.rados.rados.0.smithi045.stdout:698: copy_from oid 26 from oid 44 current snap is 57 2021-07-31T01:47:52.559 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 40 v 265 (ObjNum 206 snap 53 seq_num 206) dirty exists 2021-07-31T01:47:52.559 INFO:tasks.rados.rados.0.smithi045.stdout:689: finishing copy_from to smithi04569269-24 2021-07-31T01:47:52.560 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 24 v 146 (ObjNum 162 snap 38 seq_num 162) dirty exists 2021-07-31T01:47:52.560 INFO:tasks.rados.rados.0.smithi045.stdout:689: finishing copy_from racing read to smithi04569269-24 2021-07-31T01:47:52.560 INFO:tasks.rados.rados.0.smithi045.stdout:696: finishing write tid 1 to smithi04569269-36 2021-07-31T01:47:52.560 INFO:tasks.rados.rados.0.smithi045.stdout:696: finishing write tid 2 to smithi04569269-36 2021-07-31T01:47:52.561 INFO:tasks.rados.rados.0.smithi045.stdout:696: finishing write tid 3 to smithi04569269-36 2021-07-31T01:47:52.561 INFO:tasks.rados.rados.0.smithi045.stdout:696: finishing write tid 4 to smithi04569269-36 2021-07-31T01:47:52.561 INFO:tasks.rados.rados.0.smithi045.stdout:696: finishing write tid 5 to smithi04569269-36 2021-07-31T01:47:52.561 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 36 v 489 (ObjNum 224 snap 57 seq_num 224) dirty exists 2021-07-31T01:47:52.561 INFO:tasks.rados.rados.0.smithi045.stdout:696: left oid 36 (ObjNum 224 snap 57 seq_num 224) 2021-07-31T01:47:52.562 INFO:tasks.rados.rados.0.smithi045.stdout:689: done (3 left) 2021-07-31T01:47:52.562 INFO:tasks.rados.rados.0.smithi045.stdout:695: done (2 left) 2021-07-31T01:47:52.562 INFO:tasks.rados.rados.0.smithi045.stdout:696: done (1 left) 2021-07-31T01:47:52.562 INFO:tasks.rados.rados.0.smithi045.stdout:699: rollback oid 1 current snap is 57 2021-07-31T01:47:52.563 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 1 to 54 2021-07-31T01:47:52.563 INFO:tasks.rados.rados.0.smithi045.stdout:700: setattr oid 6 current snap is 57 2021-07-31T01:47:52.563 INFO:tasks.rados.rados.0.smithi045.stdout:701: delete oid 37 current snap is 57 2021-07-31T01:47:52.568 INFO:tasks.rados.rados.0.smithi045.stdout:699: finishing rollback tid 0 to smithi04569269-1 2021-07-31T01:47:52.569 INFO:tasks.rados.rados.0.smithi045.stdout:699: finishing rollback tid 1 to smithi04569269-1 2021-07-31T01:47:52.570 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 6 v 270 (ObjNum 215 snap 56 seq_num 215) dirty exists 2021-07-31T01:47:52.572 INFO:tasks.rados.rados.0.smithi045.stdout:700: done (3 left) 2021-07-31T01:47:52.573 INFO:tasks.rados.rados.0.smithi045.stdout:701: done (2 left) 2021-07-31T01:47:52.573 INFO:tasks.rados.rados.0.smithi045.stdout:702: snap_create 2021-07-31T01:47:52.575 INFO:tasks.rados.rados.0.smithi045.stdout:699: finishing rollback tid 2 to smithi04569269-1 2021-07-31T01:47:52.575 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 1 v 305 (ObjNum 155 snap 37 seq_num 155) dirty exists 2021-07-31T01:47:52.586 INFO:tasks.rados.rados.0.smithi045.stdout:698: finishing copy_from to smithi04569269-26 2021-07-31T01:47:52.586 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 26 v 190 (ObjNum 142 snap 30 seq_num 142) dirty exists 2021-07-31T01:47:52.587 INFO:tasks.rados.rados.0.smithi045.stdout:698: finishing copy_from racing read to smithi04569269-26 2021-07-31T01:47:53.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:52 smithi154 conmon[34553]: cluster 2021-07-31T01:47:51.799822+0000 mgr.x (mgr.24542 2021-07-31T01:47:53.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:52 smithi154 conmon[34553]: ) 30 : cluster [DBG] pgmap v38: 25 pgs: 25 active+clean; 94 MiB data, 430 MiB used, 715 GiB / 715 GiB avail; 1.0 MiB/s rd, 3.2 MiB/s wr, 20 op/s 2021-07-31T01:47:53.003 INFO:journalctl@ceph.osd.7.smithi154.stdout:Jul 31 01:47:52 smithi154 conmon[49149]: debug 2021-07-31T01:47:52.564+0000 7f8a75203700 -1 osd.7 pg_epoch: 173 pg[3.es0( v 173'270 (0'0,173'270] local-lis/les=56/57 n=9 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [7,4,1]p7(0) r=0 lpr=56 luod=173'269 crt=172'265 lcod 173'267 mlcod 173'267 active+clean+snaptrim trimq=[4b~1] ps=[6e~1,70~1]] removing snap head 2021-07-31T01:47:53.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:52 smithi045 conmon[39718]: cluster 2021-07-31T01:47:51.799822+0000 mgr.x ( 2021-07-31T01:47:53.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:52 smithi045 conmon[39718]: mgr.24542) 30 : cluster [DBG] pgmap v38: 25 pgs: 25 active+clean; 94 MiB data, 430 MiB used, 715 GiB / 715 GiB avail; 1.0 MiB/s rd, 3.2 MiB/s wr, 20 op/s 2021-07-31T01:47:53.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:52 smithi045 conmon[33655]: cluster 2021-07-31T01:47:51.799822+0000 mgr.x (mgr.24542) 30 : cluster 2021-07-31T01:47:53.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:52 smithi045 conmon[33655]: [DBG] pgmap v38: 25 pgs: 25 active+clean; 94 MiB data, 430 MiB used, 715 GiB / 715 GiB avail; 1.0 MiB/s rd, 3.2 MiB/s wr, 20 op/s 2021-07-31T01:47:53.559 INFO:tasks.rados.rados.0.smithi045.stdout:698: done (2 left) 2021-07-31T01:47:53.559 INFO:tasks.rados.rados.0.smithi045.stdout:699: done (1 left) 2021-07-31T01:47:53.560 INFO:tasks.rados.rados.0.smithi045.stdout:702: done (0 left) 2021-07-31T01:47:53.560 INFO:tasks.rados.rados.0.smithi045.stdout:703: delete oid 9 current snap is 58 2021-07-31T01:47:53.562 INFO:tasks.rados.rados.0.smithi045.stdout:703: done (0 left) 2021-07-31T01:47:53.562 INFO:tasks.rados.rados.0.smithi045.stdout:704: snap_create 2021-07-31T01:47:53.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:53 smithi045 conmon[39718]: cluster 2021-07-31T01:47:52.556768+0000 mon.a (mon.0) 809 : cluster [DBG] osdmap e173: 8 total, 8 up, 8 in 2021-07-31T01:47:53.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:47:53 smithi045 conmon[61210]: level=error ts=2021-07-31T01:47:53.509Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:47:53.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:47:53 smithi045 conmon[61210]: level=error ts=2021-07-31T01:47:53.509Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:47:53.778 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:47:53 smithi045 conmon[61210]: level=error ts=2021-07-31T01:47:53.509Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T01:47:53.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:53 smithi045 conmon[33655]: cluster 2021-07-31T01:47:52.556768+0000 mon.a 2021-07-31T01:47:53.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:53 smithi045 conmon[33655]: (mon.0) 809 : cluster [DBG] osdmap e173: 8 total, 8 up, 8 in 2021-07-31T01:47:54.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:53 smithi154 conmon[34553]: cluster 2021-07-31T01:47:52.556768+0000 mon.a (mon.0) 809 : cluster [DBG] 2021-07-31T01:47:54.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:53 smithi154 conmon[34553]: osdmap e173: 8 total, 8 up, 8 in 2021-07-31T01:47:54.572 INFO:tasks.rados.rados.0.smithi045.stdout:704: done (0 left) 2021-07-31T01:47:54.572 INFO:tasks.rados.rados.0.smithi045.stdout:705: rmattr oid 13 current snap is 59 2021-07-31T01:47:54.572 INFO:tasks.rados.rados.0.smithi045.stdout:706: snap_create 2021-07-31T01:47:54.576 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 13 v 276 (ObjNum 162 snap 38 seq_num 162) dirty exists 2021-07-31T01:47:55.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:54 smithi154 conmon[34553]: cluster 2021-07-31T01:47:53.558034+0000 mon.a (mon.0) 810 2021-07-31T01:47:55.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:54 smithi154 conmon[34553]: : cluster [DBG] osdmap e174: 8 total, 8 up, 8 in 2021-07-31T01:47:55.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:54 smithi154 conmon[34553]: cluster 2021-07-31T01:47:53.800197+0000 mgr.x (mgr.24542) 31 : cluster [DBG] pgmap v41: 25 pgs: 25 active+clean; 94 MiB data, 430 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:47:55.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:54 smithi045 conmon[33655]: cluster 2021-07-31T01:47:53.558034+0000 mon.a 2021-07-31T01:47:55.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:54 smithi045 conmon[33655]: (mon.0) 810 : cluster [DBG] osdmap e174: 8 total, 8 up, 8 in 2021-07-31T01:47:55.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:54 smithi045 conmon[33655]: cluster 2021-07-31T01:47:53.800197+0000 mgr.x (mgr.24542) 31 : cluster 2021-07-31T01:47:55.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:54 smithi045 conmon[33655]: [DBG] pgmap v41: 25 pgs: 25 active+clean; 94 MiB data, 430 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:47:55.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:54 smithi045 conmon[39718]: cluster 2021-07-31T01:47:53 2021-07-31T01:47:55.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:54 smithi045 conmon[39718]: .558034+0000 mon.a (mon.0) 810 : cluster [DBG] osdmap e174: 8 total, 8 up, 8 in 2021-07-31T01:47:55.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:54 smithi045 conmon[39718]: cluster 2021-07-31T01:47:55.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:54 smithi045 conmon[39718]: 2021-07-31T01:47:53.800197+0000 2021-07-31T01:47:55.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:54 smithi045 conmon[39718]: mgr.x (mgr.24542) 31 : cluster 2021-07-31T01:47:55.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:54 smithi045 conmon[39718]: [DBG] pgmap v41: 25 pgs: 25 active+clean; 94 MiB data, 430 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:47:55.574 INFO:tasks.rados.rados.0.smithi045.stdout:705: done (1 left) 2021-07-31T01:47:55.575 INFO:tasks.rados.rados.0.smithi045.stdout:706: done (0 left) 2021-07-31T01:47:55.576 INFO:tasks.rados.rados.0.smithi045.stdout:707: copy_from oid 43 from oid 39 current snap is 60 2021-07-31T01:47:55.576 INFO:tasks.rados.rados.0.smithi045.stdout:708: delete oid 49 current snap is 60 2021-07-31T01:47:55.578 INFO:tasks.rados.rados.0.smithi045.stdout:708: done (1 left) 2021-07-31T01:47:55.579 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 23 current snap is 60 2021-07-31T01:47:55.579 INFO:tasks.rados.rados.0.smithi045.stdout:709: seq_num 228 ranges {4161536=622592,4784128=671744,5455872=131072} 2021-07-31T01:47:55.604 INFO:tasks.rados.rados.0.smithi045.stdout:709: writing smithi04569269-23 from 4161536 to 4784128 tid 1 2021-07-31T01:47:55.607 INFO:tasks.rados.rados.0.smithi045.stdout:709: writing smithi04569269-23 from 4784128 to 5455872 tid 2 2021-07-31T01:47:55.609 INFO:tasks.rados.rados.0.smithi045.stdout:709: writing smithi04569269-23 from 5455872 to 5586944 tid 3 2021-07-31T01:47:55.609 INFO:tasks.rados.rados.0.smithi045.stdout:710: rollback oid 3 current snap is 60 2021-07-31T01:47:55.609 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 3 to 57 2021-07-31T01:47:55.610 INFO:tasks.rados.rados.0.smithi045.stdout:707: finishing copy_from to smithi04569269-43 2021-07-31T01:47:55.610 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 43 v 248 (ObjNum 197 snap 52 seq_num 197) dirty exists 2021-07-31T01:47:55.610 INFO:tasks.rados.rados.0.smithi045.stdout:707: finishing copy_from racing read to smithi04569269-43 2021-07-31T01:47:55.610 INFO:tasks.rados.rados.0.smithi045.stdout:707: done (2 left) 2021-07-31T01:47:55.611 INFO:tasks.rados.rados.0.smithi045.stdout:711: snap_create 2021-07-31T01:47:55.611 INFO:tasks.rados.rados.0.smithi045.stdout:709: finishing write tid 1 to smithi04569269-23 2021-07-31T01:47:55.612 INFO:tasks.rados.rados.0.smithi045.stdout:710: finishing rollback tid 0 to smithi04569269-3 2021-07-31T01:47:55.612 INFO:tasks.rados.rados.0.smithi045.stdout:709: finishing write tid 2 to smithi04569269-23 2021-07-31T01:47:55.612 INFO:tasks.rados.rados.0.smithi045.stdout:710: finishing rollback tid 1 to smithi04569269-3 2021-07-31T01:47:55.612 INFO:tasks.rados.rados.0.smithi045.stdout:710: finishing rollback tid 2 to smithi04569269-3 2021-07-31T01:47:55.613 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 3 v 502 (ObjNum 144 snap 31 seq_num 144) dirty exists 2021-07-31T01:47:55.613 INFO:tasks.rados.rados.0.smithi045.stdout:709: finishing write tid 3 to smithi04569269-23 2021-07-31T01:47:55.613 INFO:tasks.rados.rados.0.smithi045.stdout:709: finishing write tid 4 to smithi04569269-23 2021-07-31T01:47:55.614 INFO:tasks.rados.rados.0.smithi045.stdout:709: finishing write tid 5 to smithi04569269-23 2021-07-31T01:47:55.614 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 23 v 307 (ObjNum 228 snap 60 seq_num 228) dirty exists 2021-07-31T01:47:55.615 INFO:tasks.rados.rados.0.smithi045.stdout:709: left oid 23 (ObjNum 228 snap 60 seq_num 228) 2021-07-31T01:47:56.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:55 smithi154 conmon[34553]: cluster 2021-07-31T01:47:54.571614+0000 mon.a (mon 2021-07-31T01:47:56.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:55 smithi154 conmon[34553]: .0) 811 : cluster [DBG] osdmap e175: 8 total, 8 up, 8 in 2021-07-31T01:47:56.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:55 smithi045 conmon[39718]: cluster 2021-07-31T01:47:54.571614+0000 2021-07-31T01:47:56.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:55 smithi045 conmon[39718]: mon.a (mon.0) 811 : cluster [DBG] osdmap e175: 8 total, 8 up, 8 in 2021-07-31T01:47:56.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:55 smithi045 conmon[33655]: cluster 2021-07-31T01:47:56.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:55 smithi045 conmon[33655]: 2021-07-31T01:47:54.571614 2021-07-31T01:47:56.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:55 smithi045 conmon[33655]: +0000 mon.a (mon.0) 811 : cluster [DBG] osdmap e175: 8 total, 8 up, 8 in 2021-07-31T01:47:56.049 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T01:47:56.324 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T01:47:56.325 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 2m ago 3m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T01:47:56.325 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 29s ago 3m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T01:47:56.325 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 29s ago 6m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T01:47:56.326 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 2m ago 8m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T01:47:56.326 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 2m ago 9m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T01:47:56.326 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 29s ago 7m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T01:47:56.327 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 2m ago 7m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T01:47:56.327 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 2m ago 3m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T01:47:56.327 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 29s ago 3m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T01:47:56.328 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 2m ago 6m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T01:47:56.328 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 2m ago 6m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T01:47:56.328 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 2m ago 5m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T01:47:56.329 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 2m ago 5m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T01:47:56.329 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 29s ago 5m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T01:47:56.329 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 29s ago 4m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T01:47:56.329 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 29s ago 4m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T01:47:56.329 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 29s ago 4m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T01:47:56.330 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 29s ago 3m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T01:47:56.583 INFO:tasks.rados.rados.0.smithi045.stdout:709: done (2 left) 2021-07-31T01:47:56.583 INFO:tasks.rados.rados.0.smithi045.stdout:710: done (1 left) 2021-07-31T01:47:56.583 INFO:tasks.rados.rados.0.smithi045.stdout:711: done (0 left) 2021-07-31T01:47:56.584 INFO:tasks.rados.rados.0.smithi045.stdout:712: read oid 42 snap -1 2021-07-31T01:47:56.584 INFO:tasks.rados.rados.0.smithi045.stdout:712: expect (ObjNum 176 snap 46 seq_num 176) 2021-07-31T01:47:56.584 INFO:tasks.rados.rados.0.smithi045.stdout:713: read oid 24 snap -1 2021-07-31T01:47:56.584 INFO:tasks.rados.rados.0.smithi045.stdout:713: expect (ObjNum 162 snap 38 seq_num 162) 2021-07-31T01:47:56.584 INFO:tasks.rados.rados.0.smithi045.stdout:714: copy_from oid 25 from oid 5 current snap is 61 2021-07-31T01:47:56.585 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 13 current snap is 61 2021-07-31T01:47:56.585 INFO:tasks.rados.rados.0.smithi045.stdout:715: seq_num 229 ranges {3817472=638976,4456448=606208,5062656=196608} 2021-07-31T01:47:56.607 INFO:tasks.rados.rados.0.smithi045.stdout:715: writing smithi04569269-13 from 3817472 to 4456448 tid 1 2021-07-31T01:47:56.611 INFO:tasks.rados.rados.0.smithi045.stdout:715: writing smithi04569269-13 from 4456448 to 5062656 tid 2 2021-07-31T01:47:56.611 INFO:tasks.rados.rados.0.smithi045.stdout:715: writing smithi04569269-13 from 5062656 to 5259264 tid 3 2021-07-31T01:47:56.612 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 4 current snap is 61 2021-07-31T01:47:56.612 INFO:tasks.rados.rados.0.smithi045.stdout:716: seq_num 230 ranges {1859584=655360,2514944=491520,3006464=311296} 2021-07-31T01:47:56.625 INFO:tasks.rados.rados.0.smithi045.stdout:716: writing smithi04569269-4 from 1859584 to 2514944 tid 1 2021-07-31T01:47:56.628 INFO:tasks.rados.rados.0.smithi045.stdout:716: writing smithi04569269-4 from 2514944 to 3006464 tid 2 2021-07-31T01:47:56.630 INFO:tasks.rados.rados.0.smithi045.stdout:716: writing smithi04569269-4 from 3006464 to 3317760 tid 3 2021-07-31T01:47:56.630 INFO:tasks.rados.rados.0.smithi045.stdout:717: rmattr oid 12 current snap is 61 2021-07-31T01:47:56.630 INFO:tasks.rados.rados.0.smithi045.stdout:717: done (5 left) 2021-07-31T01:47:56.631 INFO:tasks.rados.rados.0.smithi045.stdout:718: read oid 39 snap -1 2021-07-31T01:47:56.631 INFO:tasks.rados.rados.0.smithi045.stdout:718: expect (ObjNum 197 snap 52 seq_num 197) 2021-07-31T01:47:56.631 INFO:tasks.rados.rados.0.smithi045.stdout:719: copy_from oid 30 from oid 34 current snap is 61 2021-07-31T01:47:56.631 INFO:tasks.rados.rados.0.smithi045.stdout:712: expect (ObjNum 176 snap 46 seq_num 176) 2021-07-31T01:47:56.647 INFO:tasks.rados.rados.0.smithi045.stdout:714: finishing copy_from to smithi04569269-25 2021-07-31T01:47:56.648 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 25 v 143 (ObjNum 158 snap 37 seq_num 158) dirty exists 2021-07-31T01:47:56.648 INFO:tasks.rados.rados.0.smithi045.stdout:714: finishing copy_from racing read to smithi04569269-25 2021-07-31T01:47:56.648 INFO:tasks.rados.rados.0.smithi045.stdout:713: expect (ObjNum 162 snap 38 seq_num 162) 2021-07-31T01:47:56.675 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T01:47:56.676 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T01:47:56.676 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T01:47:56.676 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T01:47:56.676 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T01:47:56.677 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T01:47:56.677 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T01:47:56.677 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T01:47:56.677 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T01:47:56.678 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T01:47:56.678 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T01:47:56.679 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T01:47:56.679 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T01:47:56.679 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T01:47:56.680 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T01:47:56.680 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T01:47:56.681 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T01:47:56.846 INFO:tasks.rados.rados.0.smithi045.stdout:715: finishing write tid 1 to smithi04569269-13 2021-07-31T01:47:56.846 INFO:tasks.rados.rados.0.smithi045.stdout:712: done (6 left) 2021-07-31T01:47:56.846 INFO:tasks.rados.rados.0.smithi045.stdout:713: done (5 left) 2021-07-31T01:47:56.846 INFO:tasks.rados.rados.0.smithi045.stdout:714: done (4 left) 2021-07-31T01:47:56.847 INFO:tasks.rados.rados.0.smithi045.stdout:720: read oid 31 snap -1 2021-07-31T01:47:56.847 INFO:tasks.rados.rados.0.smithi045.stdout:720: expect deleted 2021-07-31T01:47:56.847 INFO:tasks.rados.rados.0.smithi045.stdout:721: snap_create 2021-07-31T01:47:56.848 INFO:tasks.rados.rados.0.smithi045.stdout:715: finishing write tid 2 to smithi04569269-13 2021-07-31T01:47:56.848 INFO:tasks.rados.rados.0.smithi045.stdout:715: finishing write tid 3 to smithi04569269-13 2021-07-31T01:47:56.848 INFO:tasks.rados.rados.0.smithi045.stdout:715: finishing write tid 4 to smithi04569269-13 2021-07-31T01:47:56.848 INFO:tasks.rados.rados.0.smithi045.stdout:715: finishing write tid 5 to smithi04569269-13 2021-07-31T01:47:56.848 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 13 v 281 (ObjNum 229 snap 61 seq_num 229) dirty exists 2021-07-31T01:47:56.849 INFO:tasks.rados.rados.0.smithi045.stdout:715: left oid 13 (ObjNum 229 snap 61 seq_num 229) 2021-07-31T01:47:56.850 INFO:tasks.rados.rados.0.smithi045.stdout:716: finishing write tid 1 to smithi04569269-4 2021-07-31T01:47:56.850 INFO:tasks.rados.rados.0.smithi045.stdout:716: finishing write tid 2 to smithi04569269-4 2021-07-31T01:47:56.850 INFO:tasks.rados.rados.0.smithi045.stdout:716: finishing write tid 3 to smithi04569269-4 2021-07-31T01:47:56.850 INFO:tasks.rados.rados.0.smithi045.stdout:716: finishing write tid 4 to smithi04569269-4 2021-07-31T01:47:56.851 INFO:tasks.rados.rados.0.smithi045.stdout:716: finishing write tid 5 to smithi04569269-4 2021-07-31T01:47:56.851 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 4 v 207 (ObjNum 230 snap 61 seq_num 230) dirty exists 2021-07-31T01:47:56.851 INFO:tasks.rados.rados.0.smithi045.stdout:716: left oid 4 (ObjNum 230 snap 61 seq_num 230) 2021-07-31T01:47:56.851 INFO:tasks.rados.rados.0.smithi045.stdout:719: finishing copy_from to smithi04569269-30 2021-07-31T01:47:56.851 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 30 v 302 (ObjNum 76 snap 11 seq_num 76) dirty exists 2021-07-31T01:47:56.852 INFO:tasks.rados.rados.0.smithi045.stdout:719: finishing copy_from racing read to smithi04569269-30 2021-07-31T01:47:56.852 INFO:tasks.rados.rados.0.smithi045.stdout:718: expect (ObjNum 197 snap 52 seq_num 197) 2021-07-31T01:47:57.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:56 smithi154 conmon[34553]: cluster 2021-07-31T01:47:55.572999+0000 mon.a (mon.0) 812 : cluster 2021-07-31T01:47:57.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:56 smithi154 conmon[34553]: [DBG] osdmap e176: 8 total, 8 up, 8 in 2021-07-31T01:47:57.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:56 smithi154 conmon[34553]: cluster 2021-07-31T01:47:55.800717+0000 mgr.x (mgr.24542) 32 : cluster [DBG] pgmap v44: 25 pgs: 25 active+clean; 84 MiB data, 440 MiB used, 715 GiB / 715 GiB avail; 5.6 MiB/s rd, 2.9 MiB/s wr, 28 op/s 2021-07-31T01:47:57.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:56 smithi045 conmon[33655]: cluster 2021-07-31T01:47:55.572999+0000 mon.a ( 2021-07-31T01:47:57.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:56 smithi045 conmon[33655]: mon.0) 812 : cluster [DBG] osdmap e176: 8 total, 8 up, 8 in 2021-07-31T01:47:57.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:56 smithi045 conmon[33655]: cluster 2021-07-31T01:47:55.800717+0000 mgr.x (mgr.24542) 32 : cluster [DBG] pgmap v44: 25 pgs: 25 active+clean; 84 MiB data, 440 MiB used, 715 GiB / 715 GiB avail; 5.6 MiB/s rd, 2.9 MiB/s wr, 28 op/s 2021-07-31T01:47:57.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:56 smithi045 conmon[39718]: cluster 2021-07-31T01:47:57.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:56 smithi045 conmon[39718]: 2021-07-31T01:47:57.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:56 smithi045 conmon[39718]: 2021 2021-07-31T01:47:57.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:56 smithi045 conmon[39718]: - 2021-07-31T01:47:57.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:56 smithi045 conmon[39718]: 07 2021-07-31T01:47:57.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:56 smithi045 conmon[39718]: - 2021-07-31T01:47:57.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:56 smithi045 conmon[39718]: 31 2021-07-31T01:47:57.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:56 smithi045 conmon[39718]: T 2021-07-31T01:47:57.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:56 smithi045 conmon[39718]: 01 2021-07-31T01:47:57.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:56 smithi045 conmon[39718]: : 2021-07-31T01:47:57.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:56 smithi045 conmon[39718]: 47 2021-07-31T01:47:57.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:56 smithi045 conmon[39718]: : 2021-07-31T01:47:57.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:56 smithi045 conmon[39718]: 55 2021-07-31T01:47:57.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:56 smithi045 conmon[39718]: . 2021-07-31T01:47:57.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:56 smithi045 conmon[39718]: 572999+0000 mon.a (mon.0 2021-07-31T01:47:57.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:56 smithi045 conmon[39718]: ) 812 : cluster [DBG] osdmap e176: 8 total, 8 up, 8 in 2021-07-31T01:47:57.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:56 smithi045 conmon[39718]: cluster 2021 2021-07-31T01:47:57.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:56 smithi045 conmon[39718]: -07-31T01:47:55 2021-07-31T01:47:57.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:56 smithi045 conmon[39718]: .800717+0000 mgr.x (mgr 2021-07-31T01:47:57.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:56 smithi045 conmon[39718]: .24542) 32 : cluster [DBG] 2021-07-31T01:47:57.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:56 smithi045 conmon[39718]: pgmap v44: 25 pgs: 25 active+clean; 84 MiB data, 440 MiB used, 715 GiB / 715 GiB avail; 5.6 MiB/s rd, 2.9 MiB/s wr, 28 op/s 2021-07-31T01:47:57.589 INFO:tasks.rados.rados.0.smithi045.stdout:715: done (5 left) 2021-07-31T01:47:57.589 INFO:tasks.rados.rados.0.smithi045.stdout:716: done (4 left) 2021-07-31T01:47:57.590 INFO:tasks.rados.rados.0.smithi045.stdout:718: done (3 left) 2021-07-31T01:47:57.590 INFO:tasks.rados.rados.0.smithi045.stdout:719: done (2 left) 2021-07-31T01:47:57.590 INFO:tasks.rados.rados.0.smithi045.stdout:720: done (1 left) 2021-07-31T01:47:57.590 INFO:tasks.rados.rados.0.smithi045.stdout:721: done (0 left) 2021-07-31T01:47:57.591 INFO:tasks.rados.rados.0.smithi045.stdout:722: snap_create 2021-07-31T01:47:58.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:57 smithi154 conmon[34553]: audit 2021-07-31T01:47:56.036711+0000 mgr.x (mgr.24542) 33 : audit [DBG] from='client.24637 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:47:58.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:57 smithi154 conmon[34553]: audit 2021-07-31T01:47:56.316503+0000 mgr.x (mgr.24542) 34 : audit [DBG] from='client.14736 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:47:58.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:57 smithi154 conmon[34553]: cluster 2021-07-31T01:47:56.580877+0000 mon.a (mon.0) 813 : cluster [DBG] osdmap e177: 8 total, 8 up, 8 in 2021-07-31T01:47:58.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:57 smithi154 conmon[34553]: audit 2021-07-31T01:47:56.675170+0000 mon.a (mon.0) 814 : audit [DBG] from='client.? 172.21.15.45:0/1139072556' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T01:47:58.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:57 smithi045 conmon[33655]: audit 2021-07-31T01:47:56.036711+0000 mgr.x ( 2021-07-31T01:47:58.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:57 smithi045 conmon[33655]: mgr.24542) 33 : audit [DBG] from='client.24637 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:47:58.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:57 smithi045 conmon[33655]: audit 2021-07-31T01:47:56.316503 2021-07-31T01:47:58.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:57 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 34 : audit [DBG] from='client.14736 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:47:58.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:57 smithi045 conmon[33655]: cluster 2021-07 2021-07-31T01:47:58.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:57 smithi045 conmon[33655]: -31T01:47:56.580877+0000 mon.a 2021-07-31T01:47:58.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:57 smithi045 conmon[33655]: (mon.0) 813 : cluster 2021-07-31T01:47:58.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:57 smithi045 conmon[33655]: [DBG] osdmap e177: 8 total, 8 up, 8 in 2021-07-31T01:47:58.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:57 smithi045 conmon[33655]: audit 2021-07-31T01:47:56 2021-07-31T01:47:58.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:57 smithi045 conmon[33655]: .675170+0000 mon.a (mon.0) 814 : audit [DBG] from='client.? 172.21.15.45:0/1139072556' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T01:47:58.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:57 smithi045 conmon[39718]: audit 2021-07-31T01:47: 2021-07-31T01:47:58.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:57 smithi045 conmon[39718]: 56.036711+0000 mgr.x (mgr.24542) 33 : audit [DBG] from='client.24637 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:47:58.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:57 smithi045 conmon[39718]: audit 2021- 2021-07-31T01:47:58.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:57 smithi045 conmon[39718]: 07-31T01:47:56.316503 2021-07-31T01:47:58.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:57 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542 2021-07-31T01:47:58.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:57 smithi045 conmon[39718]: ) 34 : audit [DBG] from='client.14736 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:47:58.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:57 smithi045 conmon[39718]: 2021-07-31T01:47:58.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:57 smithi045 conmon[39718]: cluster 2021-07-31T01:47:58.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:57 smithi045 conmon[39718]: 2021-07-31T01: 2021-07-31T01:47:58.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:57 smithi045 conmon[39718]: 47:56.580877+0000 mon.a 2021-07-31T01:47:58.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:57 smithi045 conmon[39718]: (mon.0) 813 : cluster 2021-07-31T01:47:58.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:57 smithi045 conmon[39718]: [DBG] osdmap e177: 8 total, 8 up, 8 in 2021-07-31T01:47:58.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:57 smithi045 conmon[39718]: audit 2021-07-31T01:47:58.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:57 smithi045 conmon[39718]: 2021- 2021-07-31T01:47:58.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:57 smithi045 conmon[39718]: 07-31T01:47: 2021-07-31T01:47:58.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:57 smithi045 conmon[39718]: 56.675170+0000 mon.a (mon 2021-07-31T01:47:58.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:57 smithi045 conmon[39718]: .0) 814 : audit [DBG] 2021-07-31T01:47:58.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:57 smithi045 conmon[39718]: from='client.? 172.21.15.45:0/1139072556' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T01:47:58.594 INFO:tasks.rados.rados.0.smithi045.stdout:722: done (0 left) 2021-07-31T01:47:58.595 INFO:tasks.rados.rados.0.smithi045.stdout:723: copy_from oid 22 from oid 47 current snap is 63 2021-07-31T01:47:58.595 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 27 current snap is 63 2021-07-31T01:47:58.595 INFO:tasks.rados.rados.0.smithi045.stdout:724: seq_num 231 ranges {0=671744,671744=778240,1449984=24576} 2021-07-31T01:47:58.598 INFO:tasks.rados.rados.0.smithi045.stdout:724: writing smithi04569269-27 from 0 to 671744 tid 1 2021-07-31T01:47:58.603 INFO:tasks.rados.rados.0.smithi045.stdout:724: writing smithi04569269-27 from 671744 to 1449984 tid 2 2021-07-31T01:47:58.603 INFO:tasks.rados.rados.0.smithi045.stdout:724: writing smithi04569269-27 from 1449984 to 1474560 tid 3 2021-07-31T01:47:58.603 INFO:tasks.rados.rados.0.smithi045.stdout:725: snap_create 2021-07-31T01:47:58.605 INFO:tasks.rados.rados.0.smithi045.stdout:723: finishing copy_from to smithi04569269-22 2021-07-31T01:47:58.605 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 22 v 48 (ObjNum 46 snap 0 seq_num 46) dirty exists 2021-07-31T01:47:58.606 INFO:tasks.rados.rados.0.smithi045.stdout:723: finishing copy_from racing read to smithi04569269-22 2021-07-31T01:47:58.607 INFO:tasks.rados.rados.0.smithi045.stdout:724: finishing write tid 1 to smithi04569269-27 2021-07-31T01:47:58.608 INFO:tasks.rados.rados.0.smithi045.stdout:724: finishing write tid 2 to smithi04569269-27 2021-07-31T01:47:58.608 INFO:tasks.rados.rados.0.smithi045.stdout:724: finishing write tid 3 to smithi04569269-27 2021-07-31T01:47:58.608 INFO:tasks.rados.rados.0.smithi045.stdout:724: finishing write tid 4 to smithi04569269-27 2021-07-31T01:47:58.609 INFO:tasks.rados.rados.0.smithi045.stdout:724: finishing write tid 5 to smithi04569269-27 2021-07-31T01:47:58.610 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 27 v 252 (ObjNum 231 snap 63 seq_num 231) dirty exists 2021-07-31T01:47:58.610 INFO:tasks.rados.rados.0.smithi045.stdout:724: left oid 27 (ObjNum 231 snap 63 seq_num 231) 2021-07-31T01:47:59.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:58 smithi154 conmon[34553]: cluster 2021-07-31T01:47:57.588723+0000 mon.a (mon.0) 815 : cluster [DBG] osdmap e178: 8 total, 8 up, 8 in 2021-07-31T01:47:59.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:58 smithi154 conmon[34553]: cluster 2021-07-31T01:47:57.801137+0000 mgr.x (mgr.24542) 35 : cluster [DBG] pgmap v47: 25 pgs: 25 active+clean; 84 MiB data, 440 MiB used, 715 GiB / 715 GiB avail; 5.6 MiB/s rd, 2.9 MiB/s wr, 28 op/s 2021-07-31T01:47:59.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:58 smithi045 conmon[39718]: cluster 2021-07-31T01:47:57. 2021-07-31T01:47:59.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:58 smithi045 conmon[39718]: 588723+0000 mon.a (mon.0) 815 : cluster [DBG] osdmap e178: 8 total, 8 up, 8 in 2021-07-31T01:47:59.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:58 smithi045 conmon[39718]: cluster 2021-07-31T01:47:59.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:58 smithi045 conmon[39718]: 2021-07-31T01:47 2021-07-31T01:47:59.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:58 smithi045 conmon[39718]: :57.801137+0000 mgr.x ( 2021-07-31T01:47:59.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:58 smithi045 conmon[39718]: mgr.24542) 35 : cluster 2021-07-31T01:47:59.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:58 smithi045 conmon[39718]: [DBG] pgmap v47: 25 pgs: 25 active+clean; 84 MiB data, 440 MiB used, 715 GiB / 715 GiB avail; 5.6 MiB/s rd, 2.9 MiB/s wr, 28 op/s 2021-07-31T01:47:59.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:58 smithi045 conmon[33655]: cluster 2021-07-31T01:47:57.588723+0000 2021-07-31T01:47:59.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:58 smithi045 conmon[33655]: mon.a (mon.0) 815 : cluster [DBG] osdmap e178: 8 total, 8 up, 8 in 2021-07-31T01:47:59.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:58 smithi045 conmon[33655]: cluster 2021-07-31T01:47:57.801137+0000 mgr.x ( 2021-07-31T01:47:59.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:58 smithi045 conmon[33655]: mgr.24542) 35 : cluster [DBG] pgmap v47: 25 pgs: 25 active+clean; 84 MiB data, 440 MiB used, 715 GiB / 715 GiB avail; 5.6 MiB/s rd, 2.9 MiB/s wr, 28 op/s 2021-07-31T01:47:59.600 INFO:tasks.rados.rados.0.smithi045.stdout:723: done (2 left) 2021-07-31T01:47:59.600 INFO:tasks.rados.rados.0.smithi045.stdout:724: done (1 left) 2021-07-31T01:47:59.601 INFO:tasks.rados.rados.0.smithi045.stdout:725: done (0 left) 2021-07-31T01:47:59.601 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 35 current snap is 64 2021-07-31T01:47:59.601 INFO:tasks.rados.rados.0.smithi045.stdout:726: seq_num 232 ranges {2260992=688128,2949120=663552,3612672=139264} 2021-07-31T01:47:59.615 INFO:tasks.rados.rados.0.smithi045.stdout:726: writing smithi04569269-35 from 2260992 to 2949120 tid 1 2021-07-31T01:47:59.619 INFO:tasks.rados.rados.0.smithi045.stdout:726: writing smithi04569269-35 from 2949120 to 3612672 tid 2 2021-07-31T01:47:59.620 INFO:tasks.rados.rados.0.smithi045.stdout:726: writing smithi04569269-35 from 3612672 to 3751936 tid 3 2021-07-31T01:47:59.621 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 40 current snap is 64 2021-07-31T01:47:59.621 INFO:tasks.rados.rados.0.smithi045.stdout:726: finishing write tid 1 to smithi04569269-35 2021-07-31T01:47:59.621 INFO:tasks.rados.rados.0.smithi045.stdout:727: seq_num 233 ranges {1310720=712704,2023424=548864,2572288=253952} 2021-07-31T01:47:59.631 INFO:tasks.rados.rados.0.smithi045.stdout:727: writing smithi04569269-40 from 1310720 to 2023424 tid 1 2021-07-31T01:47:59.635 INFO:tasks.rados.rados.0.smithi045.stdout:727: writing smithi04569269-40 from 2023424 to 2572288 tid 2 2021-07-31T01:47:59.636 INFO:tasks.rados.rados.0.smithi045.stdout:727: writing smithi04569269-40 from 2572288 to 2826240 tid 3 2021-07-31T01:47:59.637 INFO:tasks.rados.rados.0.smithi045.stdout:728: read oid 22 snap -1 2021-07-31T01:47:59.637 INFO:tasks.rados.rados.0.smithi045.stdout:728: expect (ObjNum 46 snap 0 seq_num 46) 2021-07-31T01:47:59.638 INFO:tasks.rados.rados.0.smithi045.stdout:729: read oid 8 snap -1 2021-07-31T01:47:59.638 INFO:tasks.rados.rados.0.smithi045.stdout:729: expect (ObjNum 0 snap 0 seq_num 0) 2021-07-31T01:47:59.639 INFO:tasks.rados.rados.0.smithi045.stdout:730: copy_from oid 13 from oid 3 current snap is 64 2021-07-31T01:47:59.640 INFO:tasks.rados.rados.0.smithi045.stdout:726: finishing write tid 2 to smithi04569269-35 2021-07-31T01:47:59.640 INFO:tasks.rados.rados.0.smithi045.stdout:726: finishing write tid 3 to smithi04569269-35 2021-07-31T01:47:59.641 INFO:tasks.rados.rados.0.smithi045.stdout:726: finishing write tid 4 to smithi04569269-35 2021-07-31T01:47:59.641 INFO:tasks.rados.rados.0.smithi045.stdout:726: finishing write tid 5 to smithi04569269-35 2021-07-31T01:47:59.642 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 35 v 507 (ObjNum 232 snap 64 seq_num 232) dirty exists 2021-07-31T01:47:59.642 INFO:tasks.rados.rados.0.smithi045.stdout:726: left oid 35 (ObjNum 232 snap 64 seq_num 232) 2021-07-31T01:47:59.643 INFO:tasks.rados.rados.0.smithi045.stdout:726: done (4 left) 2021-07-31T01:47:59.644 INFO:tasks.rados.rados.0.smithi045.stdout:731: copy_from oid 36 from oid 45 current snap is 64 2021-07-31T01:47:59.644 INFO:tasks.rados.rados.0.smithi045.stdout:732: snap_remove snap 54 2021-07-31T01:48:00.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:47:59 smithi154 conmon[34553]: cluster 2021-07-31T01:47:58.593543+0000 mon.a (mon.0) 816 : cluster [DBG] osdmap e179: 8 total, 8 up, 8 in 2021-07-31T01:48:00.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:59 smithi045 conmon[33655]: cluster 2021-07-31T01:47:58.593543+0000 mon.a (mon.0) 816 2021-07-31T01:48:00.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:47:59 smithi045 conmon[33655]: : cluster [DBG] osdmap e179: 8 total, 8 up, 8 in 2021-07-31T01:48:00.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:59 smithi045 conmon[39718]: cluster 2021-07-31T01:47:58 2021-07-31T01:48:00.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:47:59 smithi045 conmon[39718]: .593543+0000 mon.a (mon.0) 816 : cluster [DBG] osdmap e179: 8 total, 8 up, 8 in 2021-07-31T01:48:00.609 INFO:tasks.rados.rados.0.smithi045.stdout:732: done (5 left) 2021-07-31T01:48:00.609 INFO:tasks.rados.rados.0.smithi045.stdout:733: read oid 32 snap -1 2021-07-31T01:48:00.609 INFO:tasks.rados.rados.0.smithi045.stdout:733: expect (ObjNum 140 snap 30 seq_num 140) 2021-07-31T01:48:00.610 INFO:tasks.rados.rados.0.smithi045.stdout:734: copy_from oid 29 from oid 19 current snap is 64 2021-07-31T01:48:00.610 INFO:tasks.rados.rados.0.smithi045.stdout:735: snap_remove snap 62 2021-07-31T01:48:01.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:00 smithi154 conmon[34553]: cluster 2021-07-31T01:47:59.599280+0000 mon.a (mon.0) 2021-07-31T01:48:01.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:00 smithi154 conmon[34553]: 817 : cluster [DBG] osdmap e180: 8 total, 8 up, 8 in 2021-07-31T01:48:01.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:00 smithi154 conmon[34553]: cluster 2021-07-31T01:47:59.801708+0000 mgr.x (mgr.24542) 36 : cluster [DBG] pgmap v50: 25 pgs: 25 active+clean; 95 MiB data, 458 MiB used, 715 GiB / 715 GiB avail; 5.0 MiB/s rd, 2.7 MiB/s wr, 12 op/s 2021-07-31T01:48:01.004 INFO:journalctl@ceph.osd.5.smithi154.stdout:Jul 31 01:48:00 smithi154 conmon[42050]: debug 2021-07-31T01:48:00.605+0000 7fc9b5064700 -1 osd.5 pg_epoch: 181 pg[3.cs0( v 176'307 (0'0,176'307] local-lis/les=56/57 n=8 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [5,3,6]p5(0) r=0 lpr=56 crt=176'307 lcod 176'306 mlcod 176'306 active+clean+snaptrim trimq=[6b~1] ps=[4b~1,6e~1,70~1]] removing snap head 2021-07-31T01:48:01.004 INFO:journalctl@ceph.osd.7.smithi154.stdout:Jul 31 01:48:00 smithi154 conmon[49149]: debug 2021-07-31T01:48:01.004 INFO:journalctl@ceph.osd.7.smithi154.stdout:Jul 31 01:48:00 smithi154 conmon[49149]: 2021-07-31T01:48:00.611+0000 7f8a75a04700 -1 osd.7 pg_epoch: 181 pg[3.ds0( v 173'343 (0'0,173'343] local-lis/les=56/57 n=6 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [7,5,3]p7(0) r=0 lpr=56 crt=173'343 lcod 173'341 mlcod 173'341 active+clean+snaptrim trimq=[6b~1] ps=[4b~1,6e~1,70~1]] removing snap head 2021-07-31T01:48:01.005 INFO:journalctl@ceph.osd.7.smithi154.stdout:Jul 31 01:48:00 smithi154 conmon[49149]: 2021-07-31T01:48:01.026 INFO:journalctl@ceph.osd.1.smithi045.stdout:Jul 31 01:48:00 smithi045 conmon[46771]: debug 2021-07-31T01:48:01.026 INFO:journalctl@ceph.osd.1.smithi045.stdout:Jul 31 01:48:00 smithi045 conmon[46771]: 2021-07-31T01:48:00.608+0000 7f41979aa700 -1 osd.1 pg_epoch: 181 pg[3.0s0( v 177'207 (0'0,177'207] local-lis/les=56/57 n=7 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [1,2,6]p1(0) r=0 lpr=56 crt=177'207 lcod 177'206 mlcod 177'206 active+clean+snaptrim trimq=[6b~1] ps=[4b~1,6e~1,70~1]] removing snap head 2021-07-31T01:48:01.027 INFO:journalctl@ceph.osd.1.smithi045.stdout:Jul 31 01:48:00 smithi045 conmon[46771]: 2021-07-31T01:48:01.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:00 smithi045 conmon[33655]: cluster 2021-07-31T01:47:59.599280+0000 mon.a (mon.0) 817 : cluster [DBG] 2021-07-31T01:48:01.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:00 smithi045 conmon[33655]: osdmap e180: 8 total, 8 up, 8 in 2021-07-31T01:48:01.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:00 smithi045 conmon[33655]: cluster 2021-07-31T01:47:59.801708+0000 mgr.x (mgr.24542) 36 : cluster [DBG] pgmap v50: 25 pgs: 25 active+clean; 95 MiB data, 458 MiB used, 715 GiB / 715 GiB avail; 5.0 MiB/s rd, 2.7 MiB/s wr, 12 op/s 2021-07-31T01:48:01.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:00 smithi045 conmon[39718]: cluster 2021-07-31T01:47:59.599280+0000 mon.a ( 2021-07-31T01:48:01.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:00 smithi045 conmon[39718]: mon.0) 817 : cluster [DBG] osdmap e180: 8 total, 8 up, 8 in 2021-07-31T01:48:01.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:00 smithi045 conmon[39718]: cluster 2021-07-31T01:47:59 2021-07-31T01:48:01.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:00 smithi045 conmon[39718]: .801708+0000 mgr.x (mgr.24542) 36 : cluster [DBG] pgmap v50: 25 pgs: 25 active+clean; 95 MiB data, 458 MiB used, 715 GiB / 715 GiB avail; 5.0 MiB/s rd, 2.7 MiB/s wr, 12 op/s 2021-07-31T01:48:01.029 INFO:journalctl@ceph.osd.3.smithi045.stdout:Jul 31 01:48:00 smithi045 conmon[54013]: debug 2021-07-31T01:48:00.617+0000 7f64fc8cd700 -1 osd.3 pg_epoch: 181 pg[3.bs0( v 181'509 (0'0,181'509] local-lis/les=56/57 n=11 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [3,0,4]p3(0) r=0 lpr=56 luod=180'507 crt=180'507 lcod 180'506 mlcod 180'506 active+clean+snaptrim trimq=[6b~1] ps=[4b~1,6e~1,70~1]] removing snap head 2021-07-31T01:48:01.030 INFO:journalctl@ceph.osd.3.smithi045.stdout:Jul 31 01:48:00 smithi045 conmon[54013]: debug 2021-07-31T01:48:00.621+0000 7f64fc8cd700 -1 osd.3 pg_epoch: 181 pg[3.bs0( v 181'511 (0'0,181'511] local-lis/les=56/57 n=9 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [3,0,4]p3(0) r=0 lpr=56 crt=181'511 lcod 181'509 mlcod 181'509 active+clean+snaptrim trimq=[6b~1] ps=[4b~1,6e~1,70~1]] removing snap head 2021-07-31T01:48:01.498 INFO:tasks.rados.rados.0.smithi045.stdout:735: done (7 left) 2021-07-31T01:48:01.499 INFO:tasks.rados.rados.0.smithi045.stdout:736: rmattr oid 7 current snap is 64 2021-07-31T01:48:01.499 INFO:tasks.rados.rados.0.smithi045.stdout:736: done (7 left) 2021-07-31T01:48:01.499 INFO:tasks.rados.rados.0.smithi045.stdout:737: read oid 50 snap -1 2021-07-31T01:48:01.500 INFO:tasks.rados.rados.0.smithi045.stdout:737: expect deleted 2021-07-31T01:48:01.500 INFO:tasks.rados.rados.0.smithi045.stdout:738: rollback oid 47 current snap is 64 2021-07-31T01:48:01.500 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 47 to 58 2021-07-31T01:48:01.501 INFO:tasks.rados.rados.0.smithi045.stdout:729: expect (ObjNum 0 snap 0 seq_num 0) 2021-07-31T01:48:01.502 INFO:tasks.rados.rados.0.smithi045.stdout:731: finishing copy_from to smithi04569269-36 2021-07-31T01:48:01.503 INFO:tasks.rados.rados.0.smithi045.stdout:731: got expected ENOENT (src dne) 2021-07-31T01:48:01.504 INFO:tasks.rados.rados.0.smithi045.stdout:731: finishing copy_from racing read to smithi04569269-36 2021-07-31T01:48:01.504 INFO:tasks.rados.rados.0.smithi045.stdout:727: finishing write tid 1 to smithi04569269-40 2021-07-31T01:48:01.505 INFO:tasks.rados.rados.0.smithi045.stdout:727: finishing write tid 2 to smithi04569269-40 2021-07-31T01:48:01.505 INFO:tasks.rados.rados.0.smithi045.stdout:727: finishing write tid 3 to smithi04569269-40 2021-07-31T01:48:01.505 INFO:tasks.rados.rados.0.smithi045.stdout:727: finishing write tid 4 to smithi04569269-40 2021-07-31T01:48:01.506 INFO:tasks.rados.rados.0.smithi045.stdout:727: finishing write tid 5 to smithi04569269-40 2021-07-31T01:48:01.506 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 40 v 286 (ObjNum 233 snap 64 seq_num 233) dirty exists 2021-07-31T01:48:01.506 INFO:tasks.rados.rados.0.smithi045.stdout:727: left oid 40 (ObjNum 233 snap 64 seq_num 233) 2021-07-31T01:48:01.507 INFO:tasks.rados.rados.0.smithi045.stdout:728: expect (ObjNum 46 snap 0 seq_num 46) 2021-07-31T01:48:01.523 INFO:tasks.rados.rados.0.smithi045.stdout:730: finishing copy_from to smithi04569269-13 2021-07-31T01:48:01.523 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 13 v 502 (ObjNum 144 snap 31 seq_num 144) dirty exists 2021-07-31T01:48:01.523 INFO:tasks.rados.rados.0.smithi045.stdout:730: finishing copy_from racing read to smithi04569269-13 2021-07-31T01:48:01.524 INFO:tasks.rados.rados.0.smithi045.stdout:727: done (8 left) 2021-07-31T01:48:01.524 INFO:tasks.rados.rados.0.smithi045.stdout:728: done (7 left) 2021-07-31T01:48:01.524 INFO:tasks.rados.rados.0.smithi045.stdout:729: done (6 left) 2021-07-31T01:48:01.525 INFO:tasks.rados.rados.0.smithi045.stdout:730: done (5 left) 2021-07-31T01:48:01.525 INFO:tasks.rados.rados.0.smithi045.stdout:731: done (4 left) 2021-07-31T01:48:01.525 INFO:tasks.rados.rados.0.smithi045.stdout:739: copy_from oid 41 from oid 35 current snap is 64 2021-07-31T01:48:01.526 INFO:tasks.rados.rados.0.smithi045.stdout:733: expect (ObjNum 140 snap 30 seq_num 140) 2021-07-31T01:48:01.586 INFO:tasks.rados.rados.0.smithi045.stdout:734: finishing copy_from to smithi04569269-29 2021-07-31T01:48:01.586 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 29 v 171 (ObjNum 216 snap 56 seq_num 216) dirty exists 2021-07-31T01:48:01.587 INFO:tasks.rados.rados.0.smithi045.stdout:734: finishing copy_from racing read to smithi04569269-29 2021-07-31T01:48:01.587 INFO:tasks.rados.rados.0.smithi045.stdout:733: done (4 left) 2021-07-31T01:48:01.587 INFO:tasks.rados.rados.0.smithi045.stdout:734: done (3 left) 2021-07-31T01:48:01.587 INFO:tasks.rados.rados.0.smithi045.stdout:740: read oid 17 snap -1 2021-07-31T01:48:01.588 INFO:tasks.rados.rados.0.smithi045.stdout:740: expect (ObjNum 196 snap 52 seq_num 196) 2021-07-31T01:48:01.588 INFO:tasks.rados.rados.0.smithi045.stdout:741: rollback oid 37 current snap is 64 2021-07-31T01:48:01.588 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 37 to 63 2021-07-31T01:48:01.588 INFO:tasks.rados.rados.0.smithi045.stdout:738: finishing rollback tid 0 to smithi04569269-47 2021-07-31T01:48:01.589 INFO:tasks.rados.rados.0.smithi045.stdout:738: finishing rollback tid 1 to smithi04569269-47 2021-07-31T01:48:01.589 INFO:tasks.rados.rados.0.smithi045.stdout:738: finishing rollback tid 2 to smithi04569269-47 2021-07-31T01:48:01.589 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 47 v 146 (ObjNum 46 snap 0 seq_num 46) dirty exists 2021-07-31T01:48:01.589 INFO:tasks.rados.rados.0.smithi045.stdout:739: finishing copy_from to smithi04569269-41 2021-07-31T01:48:01.589 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 41 v 507 (ObjNum 232 snap 64 seq_num 232) dirty exists 2021-07-31T01:48:01.590 INFO:tasks.rados.rados.0.smithi045.stdout:739: finishing copy_from racing read to smithi04569269-41 2021-07-31T01:48:01.590 INFO:tasks.rados.rados.0.smithi045.stdout:737: done (4 left) 2021-07-31T01:48:01.591 INFO:tasks.rados.rados.0.smithi045.stdout:738: done (3 left) 2021-07-31T01:48:01.591 INFO:tasks.rados.rados.0.smithi045.stdout:739: done (2 left) 2021-07-31T01:48:01.591 INFO:tasks.rados.rados.0.smithi045.stdout:742: snap_create 2021-07-31T01:48:01.591 INFO:tasks.rados.rados.0.smithi045.stdout:741: finishing rollback tid 1 to smithi04569269-37 2021-07-31T01:48:01.592 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 37 v 0 (ObjNum 1 snap 32731 seq_num 3190195312) dirty dne 2021-07-31T01:48:01.605 INFO:tasks.rados.rados.0.smithi045.stdout:740: expect (ObjNum 196 snap 52 seq_num 196) 2021-07-31T01:48:02.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:01 smithi154 conmon[34553]: cluster 2021-07-31T01:48:00.607322+0000 mon.a (mon.0) 818 : cluster [DBG] osdmap e181: 8 total, 8 up, 8 in 2021-07-31T01:48:02.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:01 smithi154 conmon[34553]: cluster 2021-07-31T01:48:01.496665+0000 mon.a (mon.0) 819 : cluster [DBG] osdmap e182: 8 total, 8 up, 8 in 2021-07-31T01:48:02.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:01 smithi045 conmon[33655]: cluster 2021-07-31T01:48:00. 2021-07-31T01:48:02.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:01 smithi045 conmon[33655]: 607322+0000 mon.a (mon.0) 818 : cluster [DBG] osdmap e181: 8 total, 8 up, 8 in 2021-07-31T01:48:02.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:01 smithi045 conmon[33655]: cluster 2021-07-31T01:48:01.496665+0000 mon.a (mon.0 2021-07-31T01:48:02.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:01 smithi045 conmon[33655]: ) 819 : cluster [DBG] osdmap e182: 8 total, 8 up, 8 in 2021-07-31T01:48:02.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:01 smithi045 conmon[39718]: cluster 2021-07-31T01:48:00 2021-07-31T01:48:02.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:01 smithi045 conmon[39718]: .607322+0000 mon.a (mon.0) 818 : cluster [DBG] osdmap e181: 8 total, 8 up, 8 in 2021-07-31T01:48:02.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:01 smithi045 conmon[39718]: cluster 2021 2021-07-31T01:48:02.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:01 smithi045 conmon[39718]: -07-31T01:48: 2021-07-31T01:48:02.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:01 smithi045 conmon[39718]: 01.496665+0000 mon.a (mon 2021-07-31T01:48:02.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:01 smithi045 conmon[39718]: .0) 819 : cluster [DBG] 2021-07-31T01:48:02.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:01 smithi045 conmon[39718]: osdmap e182: 8 total, 8 up, 8 in 2021-07-31T01:48:02.504 INFO:tasks.rados.rados.0.smithi045.stdout:740: done (2 left) 2021-07-31T01:48:02.504 INFO:tasks.rados.rados.0.smithi045.stdout:741: done (1 left) 2021-07-31T01:48:02.504 INFO:tasks.rados.rados.0.smithi045.stdout:742: done (0 left) 2021-07-31T01:48:02.504 INFO:tasks.rados.rados.0.smithi045.stdout:743: copy_from oid 36 from oid 21 current snap is 65 2021-07-31T01:48:02.505 INFO:tasks.rados.rados.0.smithi045.stdout:744: read oid 12 snap -1 2021-07-31T01:48:02.505 INFO:tasks.rados.rados.0.smithi045.stdout:744: expect deleted 2021-07-31T01:48:02.505 INFO:tasks.rados.rados.0.smithi045.stdout:745: read oid 19 snap -1 2021-07-31T01:48:02.505 INFO:tasks.rados.rados.0.smithi045.stdout:745: expect (ObjNum 216 snap 56 seq_num 216) 2021-07-31T01:48:02.505 INFO:tasks.rados.rados.0.smithi045.stdout:746: snap_create 2021-07-31T01:48:02.506 INFO:tasks.rados.rados.0.smithi045.stdout:743: finishing copy_from to smithi04569269-36 2021-07-31T01:48:02.506 INFO:tasks.rados.rados.0.smithi045.stdout:743: got expected ENOENT (src dne) 2021-07-31T01:48:02.507 INFO:tasks.rados.rados.0.smithi045.stdout:743: finishing copy_from racing read to smithi04569269-36 2021-07-31T01:48:02.514 INFO:tasks.rados.rados.0.smithi045.stdout:745: expect (ObjNum 216 snap 56 seq_num 216) 2021-07-31T01:48:03.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:02 smithi154 conmon[34553]: cluster 2021-07-31T01:48:01.802121+0000 mgr.x (mgr.24542) 37 : cluster [DBG] pgmap v53: 25 pgs: 25 active+clean; 95 MiB data, 458 MiB used, 715 GiB / 715 GiB avail; 5.0 MiB/s rd, 2.7 MiB/s wr, 12 op/s 2021-07-31T01:48:03.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:02 smithi154 conmon[34553]: cluster 2021-07-31T01:48 2021-07-31T01:48:03.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:02 smithi154 conmon[34553]: :02.502895+0000 mon.a (mon.0) 820 : cluster [DBG] osdmap e183: 8 total, 8 up, 8 in 2021-07-31T01:48:03.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:02 smithi045 conmon[39718]: cluster 2021-07-31T01:48: 2021-07-31T01:48:03.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:02 smithi045 conmon[39718]: 01.802121+0000 mgr.x (mgr.24542) 37 : cluster [DBG] pgmap v53: 25 pgs: 25 active+clean; 95 MiB data, 458 MiB used, 715 GiB / 715 GiB avail; 5.0 MiB/s rd, 2.7 MiB/s wr, 12 op/s 2021-07-31T01:48:03.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:02 smithi045 conmon[39718]: cluster 2021 2021-07-31T01:48:03.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:02 smithi045 conmon[39718]: -07-31T01:48:02 2021-07-31T01:48:03.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:02 smithi045 conmon[39718]: .502895+0000 mon.a (mon 2021-07-31T01:48:03.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:02 smithi045 conmon[39718]: .0) 820 : cluster [DBG] 2021-07-31T01:48:03.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:02 smithi045 conmon[39718]: osdmap e183: 8 total, 8 up, 8 in 2021-07-31T01:48:03.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:02 smithi045 conmon[33655]: cluster 2021-07-31T01:48:03.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:02 smithi045 conmon[33655]: 2021-07-31T01: 2021-07-31T01:48:03.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:02 smithi045 conmon[33655]: 48:01.802121+0000 mgr.x (mgr.24542) 37 : cluster [DBG] pgmap v53: 25 pgs: 25 active+clean; 95 MiB data, 458 MiB used, 715 GiB / 715 GiB avail; 5.0 MiB/s rd, 2.7 MiB/s wr, 12 op/s 2021-07-31T01:48:03.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:02 smithi045 conmon[33655]: cluster 2021-07-31T01:48:03.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:02 smithi045 conmon[33655]: 2021-07-31T01 2021-07-31T01:48:03.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:02 smithi045 conmon[33655]: :48:02.502895+0000 2021-07-31T01:48:03.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:02 smithi045 conmon[33655]: mon.a (mon.0) 820 : cluster 2021-07-31T01:48:03.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:02 smithi045 conmon[33655]: [DBG] osdmap e183: 8 total, 8 up, 8 in 2021-07-31T01:48:03.505 INFO:tasks.rados.rados.0.smithi045.stdout:743: done (3 left) 2021-07-31T01:48:03.505 INFO:tasks.rados.rados.0.smithi045.stdout:744: done (2 left) 2021-07-31T01:48:03.505 INFO:tasks.rados.rados.0.smithi045.stdout:745: done (1 left) 2021-07-31T01:48:03.506 INFO:tasks.rados.rados.0.smithi045.stdout:746: done (0 left) 2021-07-31T01:48:03.506 INFO:tasks.rados.rados.0.smithi045.stdout:747: rollback oid 27 current snap is 66 2021-07-31T01:48:03.506 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 27 to 58 2021-07-31T01:48:03.506 INFO:tasks.rados.rados.0.smithi045.stdout:748: snap_remove snap 65 2021-07-31T01:48:03.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:48:03 smithi045 conmon[61210]: level=error ts=2021-07-31T01:48:03.509Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:48:03.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:48:03 smithi045 conmon[61210]: level=error ts=2021-07-31T01:48:03.509Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:48:03.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:48:03 smithi045 conmon[61210]: level=error ts=2021-07-31T01:48:03.509Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T01:48:04.511 INFO:tasks.rados.rados.0.smithi045.stdout:748: done (1 left) 2021-07-31T01:48:04.511 INFO:tasks.rados.rados.0.smithi045.stdout:749: copy_from oid 11 from oid 41 current snap is 66 2021-07-31T01:48:04.511 INFO:tasks.rados.rados.0.smithi045.stdout:747: finishing rollback tid 0 to smithi04569269-27 2021-07-31T01:48:04.512 INFO:tasks.rados.rados.0.smithi045.stdout:747: finishing rollback tid 1 to smithi04569269-27 2021-07-31T01:48:04.512 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 27 v 254 (ObjNum 0 snap 0 seq_num 0) dirty dne 2021-07-31T01:48:04.512 INFO:tasks.rados.rados.0.smithi045.stdout:747: done (1 left) 2021-07-31T01:48:04.512 INFO:tasks.rados.rados.0.smithi045.stdout:750: copy_from oid 46 from oid 17 current snap is 66 2021-07-31T01:48:04.512 INFO:tasks.rados.rados.0.smithi045.stdout:751: delete oid 29 current snap is 66 2021-07-31T01:48:04.513 INFO:tasks.rados.rados.0.smithi045.stdout:751: done (2 left) 2021-07-31T01:48:04.513 INFO:tasks.rados.rados.0.smithi045.stdout:752: snap_remove snap 61 2021-07-31T01:48:04.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:04 smithi045 conmon[33655]: cluster 2021-07-31T01:48:03.504432+0000 mon.a (mon.0) 821 : cluster [DBG] osdmap e184: 8 total, 8 up, 8 in 2021-07-31T01:48:04.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:04 smithi045 conmon[33655]: cluster 2021-07-31T01:48:03.802507+0000 mgr.x (mgr.24542) 38 : cluster [DBG] pgmap v56: 25 pgs: 25 active+clean; 95 MiB data, 458 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:48:04.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:04 smithi045 conmon[39718]: cluster 2021-07-31T01:48:03. 2021-07-31T01:48:04.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:04 smithi045 conmon[39718]: 504432+0000 mon.a (mon.0) 821 : cluster [DBG] osdmap e184: 8 total, 8 up, 8 in 2021-07-31T01:48:04.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:04 smithi045 conmon[39718]: cluster 2021-07-31 2021-07-31T01:48:04.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:04 smithi045 conmon[39718]: T01:48:03.802507+0000 2021-07-31T01:48:04.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:04 smithi045 conmon[39718]: mgr.x (mgr.24542) 38 : 2021-07-31T01:48:04.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:04 smithi045 conmon[39718]: cluster [DBG] pgmap v56: 25 pgs: 25 active+clean; 95 MiB data, 458 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:48:05.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:04 smithi154 conmon[34553]: cluster 2021-07-31T01:48:03.504432+0000 mon.a (mon.0) 821 : cluster [DBG] 2021-07-31T01:48:05.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:04 smithi154 conmon[34553]: osdmap e184: 8 total, 8 up, 8 in 2021-07-31T01:48:05.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:04 smithi154 conmon[34553]: cluster 2021-07-31T01:48:03.802507+0000 mgr.x (mgr.24542) 38 : cluster [DBG] pgmap v56: 25 pgs: 25 active+clean; 95 MiB data, 458 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:48:05.516 INFO:tasks.rados.rados.0.smithi045.stdout:752: done (2 left) 2021-07-31T01:48:05.516 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 12 current snap is 66 2021-07-31T01:48:05.517 INFO:tasks.rados.rados.0.smithi045.stdout:749: finishing copy_from to smithi04569269-11 2021-07-31T01:48:05.517 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 11 v 507 (ObjNum 232 snap 64 seq_num 232) dirty exists 2021-07-31T01:48:05.517 INFO:tasks.rados.rados.0.smithi045.stdout:749: finishing copy_from racing read to smithi04569269-11 2021-07-31T01:48:05.517 INFO:tasks.rados.rados.0.smithi045.stdout:750: finishing copy_from to smithi04569269-46 2021-07-31T01:48:05.518 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 46 v 244 (ObjNum 196 snap 52 seq_num 196) dirty exists 2021-07-31T01:48:05.518 INFO:tasks.rados.rados.0.smithi045.stdout:750: finishing copy_from racing read to smithi04569269-46 2021-07-31T01:48:05.518 INFO:tasks.rados.rados.0.smithi045.stdout:753: seq_num 235 ranges {0=745472,745472=712704,1458176=90112} 2021-07-31T01:48:05.524 INFO:tasks.rados.rados.0.smithi045.stdout:753: writing smithi04569269-12 from 0 to 745472 tid 1 2021-07-31T01:48:05.525 INFO:tasks.rados.rados.0.smithi045.stdout:753: writing smithi04569269-12 from 745472 to 1458176 tid 2 2021-07-31T01:48:05.526 INFO:tasks.rados.rados.0.smithi045.stdout:753: writing smithi04569269-12 from 1458176 to 1548288 tid 3 2021-07-31T01:48:05.526 INFO:tasks.rados.rados.0.smithi045.stdout:749: done (2 left) 2021-07-31T01:48:05.526 INFO:tasks.rados.rados.0.smithi045.stdout:750: done (1 left) 2021-07-31T01:48:05.526 INFO:tasks.rados.rados.0.smithi045.stdout:754: delete oid 3 current snap is 66 2021-07-31T01:48:05.527 INFO:tasks.rados.rados.0.smithi045.stdout:753: finishing write tid 1 to smithi04569269-12 2021-07-31T01:48:05.528 INFO:tasks.rados.rados.0.smithi045.stdout:754: done (1 left) 2021-07-31T01:48:05.528 INFO:tasks.rados.rados.0.smithi045.stdout:755: snap_remove snap 59 2021-07-31T01:48:05.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:05 smithi045 conmon[39718]: cluster 2021-07-31T01:48:04.509369+0000 mon.a (mon 2021-07-31T01:48:05.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:05 smithi045 conmon[39718]: .0) 822 : cluster [DBG] osdmap e185: 8 total, 8 up, 8 in 2021-07-31T01:48:05.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:05 smithi045 conmon[33655]: cluster 2021-07-31T01:48:04.509369+0000 mon.a (mon.0 2021-07-31T01:48:05.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:05 smithi045 conmon[33655]: ) 822 : cluster [DBG] osdmap e185: 8 total, 8 up, 8 in 2021-07-31T01:48:06.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:05 smithi154 conmon[34553]: cluster 2021-07-31T01:48:04.509369+0000 mon.a (mon.0) 822 : cluster [DBG] 2021-07-31T01:48:06.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:05 smithi154 conmon[34553]: osdmap e185: 8 total, 8 up, 8 in 2021-07-31T01:48:06.523 INFO:tasks.rados.rados.0.smithi045.stdout:755: done (1 left) 2021-07-31T01:48:06.523 INFO:tasks.rados.rados.0.smithi045.stdout:756: read oid 9 snap -1 2021-07-31T01:48:06.523 INFO:tasks.rados.rados.0.smithi045.stdout:756: expect deleted 2021-07-31T01:48:06.523 INFO:tasks.rados.rados.0.smithi045.stdout:757: snap_remove snap 60 2021-07-31T01:48:06.524 INFO:tasks.rados.rados.0.smithi045.stdout:753: finishing write tid 2 to smithi04569269-12 2021-07-31T01:48:06.524 INFO:tasks.rados.rados.0.smithi045.stdout:753: finishing write tid 3 to smithi04569269-12 2021-07-31T01:48:06.524 INFO:tasks.rados.rados.0.smithi045.stdout:753: finishing write tid 4 to smithi04569269-12 2021-07-31T01:48:06.525 INFO:tasks.rados.rados.0.smithi045.stdout:753: finishing write tid 5 to smithi04569269-12 2021-07-31T01:48:06.525 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 12 v 506 (ObjNum 235 snap 66 seq_num 235) dirty exists 2021-07-31T01:48:06.525 INFO:tasks.rados.rados.0.smithi045.stdout:753: left oid 12 (ObjNum 235 snap 66 seq_num 235) 2021-07-31T01:48:06.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:06 smithi045 conmon[33655]: cluster 2021-07-31T01:48:05.515435+0000 mon.a (mon.0) 2021-07-31T01:48:06.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:06 smithi045 conmon[33655]: 823 : cluster [DBG] osdmap e186: 8 total, 8 up, 8 in 2021-07-31T01:48:06.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:06 smithi045 conmon[33655]: cluster 2021-07-31T01:48:05.803049+0000 mgr.x (mgr.24542) 39 : cluster [DBG] pgmap v59: 25 pgs: 25 active+clean; 97 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 7.9 MiB/s rd, 2.5 MiB/s wr, 10 op/s 2021-07-31T01:48:06.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:06 smithi045 conmon[39718]: cluster 2021-07-31T01:48:05.515435+0000 mon.a ( 2021-07-31T01:48:06.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:06 smithi045 conmon[39718]: mon.0) 823 : cluster [DBG] osdmap e186: 8 total, 8 up, 8 in 2021-07-31T01:48:06.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:06 smithi045 conmon[39718]: cluster 2021-07-31T01 2021-07-31T01:48:06.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:06 smithi045 conmon[39718]: :48:05.803049+0000 mgr.x (mgr.24542) 39 : cluster [DBG] pgmap v59: 25 pgs: 25 active+clean; 97 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 7.9 MiB/s rd, 2.5 MiB/s wr, 10 op/s 2021-07-31T01:48:07.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:06 smithi154 conmon[34553]: cluster 2021-07-31T01:48:05.515435+0000 mon.a (mon.0) 2021-07-31T01:48:07.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:06 smithi154 conmon[34553]: 823 : cluster [DBG] osdmap e186: 8 total, 8 up, 8 in 2021-07-31T01:48:07.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:06 smithi154 conmon[34553]: cluster 2021-07-31T01:48:05.803049+0000 mgr.x (mgr.24542) 39 : cluster [DBG] pgmap v59: 25 pgs: 25 active+clean; 97 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 7.9 MiB/s rd, 2.5 MiB/s wr, 10 op/s 2021-07-31T01:48:07.524 INFO:tasks.rados.rados.0.smithi045.stdout:753: done (2 left) 2021-07-31T01:48:07.525 INFO:tasks.rados.rados.0.smithi045.stdout:757: done (1 left) 2021-07-31T01:48:07.525 INFO:tasks.rados.rados.0.smithi045.stdout:758: read oid 31 snap -1 2021-07-31T01:48:07.525 INFO:tasks.rados.rados.0.smithi045.stdout:758: expect deleted 2021-07-31T01:48:07.526 INFO:tasks.rados.rados.0.smithi045.stdout:759: rollback oid 44 current snap is 66 2021-07-31T01:48:07.526 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 44 to 63 2021-07-31T01:48:07.526 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 29 current snap is 66 2021-07-31T01:48:07.527 INFO:tasks.rados.rados.0.smithi045.stdout:760: seq_num 237 ranges {0=778240,778240=532480,1310720=270336} 2021-07-31T01:48:07.532 INFO:tasks.rados.rados.0.smithi045.stdout:760: writing smithi04569269-29 from 0 to 778240 tid 1 2021-07-31T01:48:07.535 INFO:tasks.rados.rados.0.smithi045.stdout:760: writing smithi04569269-29 from 778240 to 1310720 tid 2 2021-07-31T01:48:07.536 INFO:tasks.rados.rados.0.smithi045.stdout:760: writing smithi04569269-29 from 1310720 to 1581056 tid 3 2021-07-31T01:48:07.536 INFO:tasks.rados.rados.0.smithi045.stdout:761: read oid 8 snap -1 2021-07-31T01:48:07.536 INFO:tasks.rados.rados.0.smithi045.stdout:761: expect (ObjNum 0 snap 0 seq_num 0) 2021-07-31T01:48:07.536 INFO:tasks.rados.rados.0.smithi045.stdout:762: delete oid 14 current snap is 66 2021-07-31T01:48:07.537 INFO:tasks.rados.rados.0.smithi045.stdout:759: finishing rollback tid 0 to smithi04569269-44 2021-07-31T01:48:07.537 INFO:tasks.rados.rados.0.smithi045.stdout:759: finishing rollback tid 1 to smithi04569269-44 2021-07-31T01:48:07.537 INFO:tasks.rados.rados.0.smithi045.stdout:759: finishing rollback tid 2 to smithi04569269-44 2021-07-31T01:48:07.537 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 44 v 257 (ObjNum 142 snap 30 seq_num 142) dirty exists 2021-07-31T01:48:07.538 INFO:tasks.rados.rados.0.smithi045.stdout:761: expect (ObjNum 0 snap 0 seq_num 0) 2021-07-31T01:48:07.541 INFO:tasks.rados.rados.0.smithi045.stdout:760: finishing write tid 1 to smithi04569269-29 2021-07-31T01:48:07.541 INFO:tasks.rados.rados.0.smithi045.stdout:760: finishing write tid 2 to smithi04569269-29 2021-07-31T01:48:07.542 INFO:tasks.rados.rados.0.smithi045.stdout:760: finishing write tid 3 to smithi04569269-29 2021-07-31T01:48:07.542 INFO:tasks.rados.rados.0.smithi045.stdout:756: done (5 left) 2021-07-31T01:48:07.542 INFO:tasks.rados.rados.0.smithi045.stdout:758: done (4 left) 2021-07-31T01:48:07.542 INFO:tasks.rados.rados.0.smithi045.stdout:759: done (3 left) 2021-07-31T01:48:07.543 INFO:tasks.rados.rados.0.smithi045.stdout:761: done (2 left) 2021-07-31T01:48:07.543 INFO:tasks.rados.rados.0.smithi045.stdout:762: done (1 left) 2021-07-31T01:48:07.543 INFO:tasks.rados.rados.0.smithi045.stdout:763: setattr oid 48 current snap is 66 2021-07-31T01:48:07.543 INFO:tasks.rados.rados.0.smithi045.stdout:760: finishing write tid 4 to smithi04569269-29 2021-07-31T01:48:07.543 INFO:tasks.rados.rados.0.smithi045.stdout:760: finishing write tid 5 to smithi04569269-29 2021-07-31T01:48:07.544 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 29 v 176 (ObjNum 237 snap 66 seq_num 237) dirty exists 2021-07-31T01:48:07.544 INFO:tasks.rados.rados.0.smithi045.stdout:760: left oid 29 (ObjNum 237 snap 66 seq_num 237) 2021-07-31T01:48:07.545 INFO:tasks.rados.rados.0.smithi045.stdout:760: done (1 left) 2021-07-31T01:48:07.545 INFO:tasks.rados.rados.0.smithi045.stdout:764: rmattr oid 3 current snap is 66 2021-07-31T01:48:07.545 INFO:tasks.rados.rados.0.smithi045.stdout:764: done (1 left) 2021-07-31T01:48:07.546 INFO:tasks.rados.rados.0.smithi045.stdout:765: setattr oid 31 current snap is 66 2021-07-31T01:48:07.546 INFO:tasks.rados.rados.0.smithi045.stdout:766: delete oid 43 current snap is 66 2021-07-31T01:48:07.547 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 31 v 530 (ObjNum -1105381136 snap 21873 seq_num 0) dirty exists 2021-07-31T01:48:07.547 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 48 v 306 (ObjNum 157 snap 37 seq_num 157) dirty exists 2021-07-31T01:48:07.547 INFO:tasks.rados.rados.0.smithi045.stdout:763: done (2 left) 2021-07-31T01:48:07.548 INFO:tasks.rados.rados.0.smithi045.stdout:765: done (1 left) 2021-07-31T01:48:07.548 INFO:tasks.rados.rados.0.smithi045.stdout:766: done (0 left) 2021-07-31T01:48:07.548 INFO:tasks.rados.rados.0.smithi045.stdout:767: copy_from oid 8 from oid 47 current snap is 66 2021-07-31T01:48:07.548 INFO:tasks.rados.rados.0.smithi045.stdout:768: read oid 25 snap -1 2021-07-31T01:48:07.549 INFO:tasks.rados.rados.0.smithi045.stdout:768: expect (ObjNum 158 snap 37 seq_num 158) 2021-07-31T01:48:07.549 INFO:tasks.rados.rados.0.smithi045.stdout:769: snap_create 2021-07-31T01:48:07.550 INFO:tasks.rados.rados.0.smithi045.stdout:768: expect (ObjNum 158 snap 37 seq_num 158) 2021-07-31T01:48:07.558 INFO:tasks.rados.rados.0.smithi045.stdout:767: finishing copy_from to smithi04569269-8 2021-07-31T01:48:07.559 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 8 v 146 (ObjNum 46 snap 0 seq_num 46) dirty exists 2021-07-31T01:48:07.559 INFO:tasks.rados.rados.0.smithi045.stdout:767: finishing copy_from racing read to smithi04569269-8 2021-07-31T01:48:08.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:07 smithi154 conmon[34553]: cluster 2021-07-31T01:48:08.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:07 smithi154 conmon[34553]: 2021-07-31T01:48:08.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:07 smithi154 conmon[34553]: 2021 2021-07-31T01:48:08.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:07 smithi154 conmon[34553]: - 2021-07-31T01:48:08.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:07 smithi154 conmon[34553]: 07 2021-07-31T01:48:08.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:07 smithi154 conmon[34553]: - 2021-07-31T01:48:08.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:07 smithi154 conmon[34553]: 31 2021-07-31T01:48:08.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:07 smithi154 conmon[34553]: T 2021-07-31T01:48:08.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:07 smithi154 conmon[34553]: 01 2021-07-31T01:48:08.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:07 smithi154 conmon[34553]: : 2021-07-31T01:48:08.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:07 smithi154 conmon[34553]: 48 2021-07-31T01:48:08.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:07 smithi154 conmon[34553]: : 2021-07-31T01:48:08.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:07 smithi154 conmon[34553]: 06.521946+0000 mon.a (mon.0) 824 : cluster [DBG] osdmap e187: 8 total, 8 up, 8 in 2021-07-31T01:48:08.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:07 smithi045 conmon[39718]: cluster 2021-07-31T01:48:06.521946+0000 2021-07-31T01:48:08.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:07 smithi045 conmon[39718]: mon.a (mon.0) 824 : cluster [DBG] osdmap e187: 8 total, 8 up, 8 in 2021-07-31T01:48:08.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:07 smithi045 conmon[33655]: cluster 2021-07-31T01:48:08.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:07 smithi045 conmon[33655]: 2021-07-31T01:48:08.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:07 smithi045 conmon[33655]: 2021 2021-07-31T01:48:08.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:07 smithi045 conmon[33655]: - 2021-07-31T01:48:08.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:07 smithi045 conmon[33655]: 07 2021-07-31T01:48:08.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:07 smithi045 conmon[33655]: - 2021-07-31T01:48:08.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:07 smithi045 conmon[33655]: 31 2021-07-31T01:48:08.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:07 smithi045 conmon[33655]: T 2021-07-31T01:48:08.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:07 smithi045 conmon[33655]: 01:48:06.521946+0000 mon.a (mon.0) 824 2021-07-31T01:48:08.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:07 smithi045 conmon[33655]: : cluster [DBG] osdmap e187: 8 total, 8 up, 8 in 2021-07-31T01:48:08.539 INFO:tasks.rados.rados.0.smithi045.stdout:767: done (2 left) 2021-07-31T01:48:08.540 INFO:tasks.rados.rados.0.smithi045.stdout:768: done (1 left) 2021-07-31T01:48:08.540 INFO:tasks.rados.rados.0.smithi045.stdout:769: done (0 left) 2021-07-31T01:48:08.540 INFO:tasks.rados.rados.0.smithi045.stdout:770: rmattr oid 35 current snap is 67 2021-07-31T01:48:08.540 INFO:tasks.rados.rados.0.smithi045.stdout:770: done (0 left) 2021-07-31T01:48:08.541 INFO:tasks.rados.rados.0.smithi045.stdout:771: read oid 27 snap -1 2021-07-31T01:48:08.541 INFO:tasks.rados.rados.0.smithi045.stdout:771: expect deleted 2021-07-31T01:48:08.541 INFO:tasks.rados.rados.0.smithi045.stdout:772: setattr oid 18 current snap is 67 2021-07-31T01:48:08.542 INFO:tasks.rados.rados.0.smithi045.stdout:773: rollback oid 25 current snap is 67 2021-07-31T01:48:08.542 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 25 to 58 2021-07-31T01:48:08.542 INFO:tasks.rados.rados.0.smithi045.stdout:774: snap_remove snap 57 2021-07-31T01:48:09.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:08 smithi154 conmon[34553]: cluster 2021-07-31T01:48:07.523618+0000 mon.a (mon.0) 825 : cluster [DBG] osdmap e188: 8 total, 8 up, 8 in 2021-07-31T01:48:09.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:08 smithi154 conmon[34553]: 2021-07-31T01:48:09.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:08 smithi154 conmon[34553]: cluster 2021-07-31T01:48:07.803449+0000 mgr.x (mgr.24542) 40 : cluster [DBG] pgmap v62: 25 pgs: 25 active+clean; 97 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 7.9 MiB/s rd, 2.5 MiB/s wr, 10 op/s 2021-07-31T01:48:09.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:08 smithi045 conmon[33655]: cluster 2021-07-31T01:48:07.523618+0000 mon.a (mon.0 2021-07-31T01:48:09.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:08 smithi045 conmon[33655]: ) 825 : cluster [DBG] osdmap e188: 8 total, 8 up, 8 in 2021-07-31T01:48:09.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:08 smithi045 conmon[33655]: cluster 2021-07-31T01:48:07.803449+0000 mgr.x (mgr.24542) 40 : cluster [DBG] pgmap v62: 25 pgs: 25 active+clean; 97 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 7.9 MiB/s rd, 2.5 MiB/s wr, 10 op/s 2021-07-31T01:48:09.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:08 smithi045 conmon[39718]: cluster 2021-07-31T01:48:07.523618+0000 mon.a (mon 2021-07-31T01:48:09.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:08 smithi045 conmon[39718]: .0) 825 : cluster [DBG] osdmap e188: 8 total, 8 up, 8 in 2021-07-31T01:48:09.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:08 smithi045 conmon[39718]: cluster 2021-07-31T01:48:07.803449 2021-07-31T01:48:09.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:08 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 40 : cluster [DBG] pgmap v62: 25 pgs: 25 active+clean; 97 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 7.9 MiB/s rd, 2.5 MiB/s wr, 10 op/s 2021-07-31T01:48:09.542 INFO:tasks.rados.rados.0.smithi045.stdout:774: done (3 left) 2021-07-31T01:48:09.542 INFO:tasks.rados.rados.0.smithi045.stdout:775: snap_remove snap 63 2021-07-31T01:48:10.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:09 smithi154 conmon[34553]: cluster 2021-07-31T01:48:08.538495+0000 mon.a (mon.0) 826 2021-07-31T01:48:10.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:09 smithi154 conmon[34553]: : cluster [DBG] osdmap e189: 8 total, 8 up, 8 in 2021-07-31T01:48:10.026 INFO:journalctl@ceph.osd.0.smithi045.stdout:Jul 31 01:48:09 smithi045 conmon[43243]: debug 2021-07-31T01:48:09.787+0000 7fdf6978b700 -1 osd.0 pg_epoch: 190 pg[3.6s0( v 181'89 (0'0,181'89] local-lis/les=56/57 n=2 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [0,1,4]p0(0) r=0 lpr=56 crt=181'89 lcod 174'87 mlcod 174'87 active+clean+snaptrim trimq=[73~1] ps=[6b~1,75~4,7d~1]] removing snap head 2021-07-31T01:48:10.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:09 smithi045 conmon[33655]: cluster 2021-07-31T01:48:10.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:09 smithi045 conmon[33655]: 2021-07-31T01:48:10.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:09 smithi045 conmon[33655]: 2021-07-31T01:48 2021-07-31T01:48:10.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:09 smithi045 conmon[33655]: :08.538495+0000 mon.a (mon.0) 826 : cluster [DBG] osdmap e189: 8 total, 8 up, 8 in 2021-07-31T01:48:10.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:09 smithi045 conmon[39718]: cluster 2021-07-31T01:48:10.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:09 smithi045 conmon[39718]: 2021-07-31T01:48:10.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:09 smithi045 conmon[39718]: 2021 2021-07-31T01:48:10.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:09 smithi045 conmon[39718]: - 2021-07-31T01:48:10.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:09 smithi045 conmon[39718]: 07-31T01:48:08.538495+0000 mon.a (mon.0) 826 : cluster [DBG] osdmap e189: 8 total, 8 up, 8 in 2021-07-31T01:48:10.551 INFO:tasks.rados.rados.0.smithi045.stdout:775: done (3 left) 2021-07-31T01:48:10.552 INFO:tasks.rados.rados.0.smithi045.stdout:776: setattr oid 32 current snap is 67 2021-07-31T01:48:10.552 INFO:tasks.rados.rados.0.smithi045.stdout:773: finishing rollback tid 0 to smithi04569269-25 2021-07-31T01:48:10.552 INFO:tasks.rados.rados.0.smithi045.stdout:773: finishing rollback tid 1 to smithi04569269-25 2021-07-31T01:48:10.552 INFO:tasks.rados.rados.0.smithi045.stdout:773: finishing rollback tid 2 to smithi04569269-25 2021-07-31T01:48:10.553 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 25 v 149 (ObjNum 201 snap 52 seq_num 201) dirty exists 2021-07-31T01:48:10.553 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 18 v 217 (ObjNum 172 snap 41 seq_num 172) dirty exists 2021-07-31T01:48:10.554 INFO:tasks.rados.rados.0.smithi045.stdout:771: done (3 left) 2021-07-31T01:48:10.554 INFO:tasks.rados.rados.0.smithi045.stdout:772: done (2 left) 2021-07-31T01:48:10.554 INFO:tasks.rados.rados.0.smithi045.stdout:773: done (1 left) 2021-07-31T01:48:10.555 INFO:tasks.rados.rados.0.smithi045.stdout:777: read oid 38 snap -1 2021-07-31T01:48:10.555 INFO:tasks.rados.rados.0.smithi045.stdout:777: expect (ObjNum 196 snap 52 seq_num 196) 2021-07-31T01:48:10.555 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 49 current snap is 67 2021-07-31T01:48:10.555 INFO:tasks.rados.rados.0.smithi045.stdout:778: seq_num 240 ranges {0=425984,425984=589824,1015808=483328,1499136=131072} 2021-07-31T01:48:10.556 INFO:tasks.rados.rados.0.smithi045.stdout:778: writing smithi04569269-49 from 0 to 425984 tid 1 2021-07-31T01:48:10.560 INFO:tasks.rados.rados.0.smithi045.stdout:778: writing smithi04569269-49 from 425984 to 1015808 tid 2 2021-07-31T01:48:10.563 INFO:tasks.rados.rados.0.smithi045.stdout:778: writing smithi04569269-49 from 1015808 to 1499136 tid 3 2021-07-31T01:48:10.565 INFO:tasks.rados.rados.0.smithi045.stdout:778: writing smithi04569269-49 from 1499136 to 1630208 tid 4 2021-07-31T01:48:10.565 INFO:tasks.rados.rados.0.smithi045.stdout:779: copy_from oid 3 from oid 19 current snap is 67 2021-07-31T01:48:10.565 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 32 v 191 (ObjNum 140 snap 30 seq_num 140) dirty exists 2021-07-31T01:48:10.565 INFO:tasks.rados.rados.0.smithi045.stdout:776: done (3 left) 2021-07-31T01:48:10.566 INFO:tasks.rados.rados.0.smithi045.stdout:780: read oid 15 snap -1 2021-07-31T01:48:10.566 INFO:tasks.rados.rados.0.smithi045.stdout:780: expect (ObjNum 208 snap 53 seq_num 208) 2021-07-31T01:48:10.566 INFO:tasks.rados.rados.0.smithi045.stdout:781: delete oid 23 current snap is 67 2021-07-31T01:48:10.571 INFO:tasks.rados.rados.0.smithi045.stdout:778: finishing write tid 1 to smithi04569269-49 2021-07-31T01:48:10.572 INFO:tasks.rados.rados.0.smithi045.stdout:780: expect (ObjNum 208 snap 53 seq_num 208) 2021-07-31T01:48:10.619 INFO:tasks.rados.rados.0.smithi045.stdout:778: finishing write tid 2 to smithi04569269-49 2021-07-31T01:48:10.619 INFO:tasks.rados.rados.0.smithi045.stdout:778: finishing write tid 3 to smithi04569269-49 2021-07-31T01:48:10.619 INFO:tasks.rados.rados.0.smithi045.stdout:778: finishing write tid 4 to smithi04569269-49 2021-07-31T01:48:10.620 INFO:tasks.rados.rados.0.smithi045.stdout:778: finishing write tid 5 to smithi04569269-49 2021-07-31T01:48:10.620 INFO:tasks.rados.rados.0.smithi045.stdout:778: finishing write tid 6 to smithi04569269-49 2021-07-31T01:48:10.620 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 49 v 547 (ObjNum 240 snap 67 seq_num 240) dirty exists 2021-07-31T01:48:10.620 INFO:tasks.rados.rados.0.smithi045.stdout:778: left oid 49 (ObjNum 240 snap 67 seq_num 240) 2021-07-31T01:48:10.621 INFO:tasks.rados.rados.0.smithi045.stdout:777: expect (ObjNum 196 snap 52 seq_num 196) 2021-07-31T01:48:10.843 INFO:tasks.rados.rados.0.smithi045.stdout:779: finishing copy_from to smithi04569269-3 2021-07-31T01:48:10.844 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 3 v 171 (ObjNum 216 snap 56 seq_num 216) dirty exists 2021-07-31T01:48:10.844 INFO:tasks.rados.rados.0.smithi045.stdout:779: finishing copy_from racing read to smithi04569269-3 2021-07-31T01:48:10.844 INFO:tasks.rados.rados.0.smithi045.stdout:777: done (4 left) 2021-07-31T01:48:10.844 INFO:tasks.rados.rados.0.smithi045.stdout:778: done (3 left) 2021-07-31T01:48:10.845 INFO:tasks.rados.rados.0.smithi045.stdout:779: done (2 left) 2021-07-31T01:48:10.845 INFO:tasks.rados.rados.0.smithi045.stdout:780: done (1 left) 2021-07-31T01:48:10.845 INFO:tasks.rados.rados.0.smithi045.stdout:781: done (0 left) 2021-07-31T01:48:10.845 INFO:tasks.rados.rados.0.smithi045.stdout:782: snap_remove snap 66 2021-07-31T01:48:11.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:10 smithi154 conmon[34553]: cluster 2021-07-31T01:48:09.540437+0000 mon.a (mon.0) 827 : cluster 2021-07-31T01:48:11.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:10 smithi154 conmon[34553]: [DBG] osdmap e190: 8 total, 8 up, 8 in 2021-07-31T01:48:11.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:10 smithi154 conmon[34553]: cluster 2021-07-31T01:48:09.803889+0000 mgr.x (mgr.24542) 41 : cluster [DBG] pgmap v65: 25 pgs: 25 active+clean; 111 MiB data, 504 MiB used, 715 GiB / 715 GiB avail; 206 KiB/s rd, 3.0 MiB/s wr, 23 op/s 2021-07-31T01:48:11.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:10 smithi045 conmon[33655]: cluster 2021-07-31T01:48:09.540437+0000 mon.a ( 2021-07-31T01:48:11.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:10 smithi045 conmon[33655]: mon.0) 827 : cluster [DBG] osdmap e190: 8 total, 8 up, 8 in 2021-07-31T01:48:11.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:10 smithi045 conmon[33655]: cluster 2021-07-31T01:48:09.803889+0000 mgr.x (mgr.24542) 41 : cluster [DBG] pgmap v65: 25 pgs: 25 active+clean; 111 MiB data, 504 MiB used, 715 GiB / 715 GiB avail; 206 KiB/s rd, 3.0 MiB/s wr, 23 op/s 2021-07-31T01:48:11.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:10 smithi045 conmon[39718]: cluster 2021-07-31T01:48:09.540437+0000 mon.a (mon 2021-07-31T01:48:11.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:10 smithi045 conmon[39718]: .0) 827 : cluster [DBG] osdmap e190: 8 total, 8 up, 8 in 2021-07-31T01:48:11.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:10 smithi045 conmon[39718]: cluster 2021-07-31T01:48:09.803889+0000 mgr.x (mgr.24542) 41 : cluster 2021-07-31T01:48:11.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:10 smithi045 conmon[39718]: [DBG] pgmap v65: 25 pgs: 25 active+clean; 111 MiB data, 504 MiB used, 715 GiB / 715 GiB avail; 206 KiB/s rd, 3.0 MiB/s wr, 23 op/s 2021-07-31T01:48:11.500 INFO:tasks.rados.rados.0.smithi045.stdout:782: done (0 left) 2021-07-31T01:48:11.500 INFO:tasks.rados.rados.0.smithi045.stdout:783: snap_remove snap 64 2021-07-31T01:48:12.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:11 smithi154 conmon[34553]: cluster 2021-07-31T01:48:10.550443+0000 mon.a (mon.0) 828 : cluster [DBG] osdmap e191: 8 total, 8 up, 8 in 2021-07-31T01:48:12.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:11 smithi154 conmon[34553]: cluster 2021-07-31T01:48:11. 2021-07-31T01:48:12.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:11 smithi154 conmon[34553]: 499053+0000 mon.a (mon.0) 829 : cluster [DBG] osdmap e192: 8 total, 8 up, 8 in 2021-07-31T01:48:12.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:11 smithi045 conmon[33655]: cluster 2021-07-31T01:48:10.550443+0000 mon.a (mon 2021-07-31T01:48:12.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:11 smithi045 conmon[33655]: .0) 828 : cluster [DBG] osdmap e191: 8 total, 8 up, 8 in 2021-07-31T01:48:12.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:11 smithi045 conmon[33655]: cluster 2021-07-31T01:48 2021-07-31T01:48:12.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:11 smithi045 conmon[33655]: :11.499053+0000 mon.a (mon.0) 829 : cluster [DBG] osdmap e192: 8 total, 8 up, 8 in 2021-07-31T01:48:12.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:11 smithi045 conmon[39718]: cluster 2021-07- 2021-07-31T01:48:12.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:11 smithi045 conmon[39718]: 31T01:48:10.550443+0000 mon.a (mon.0) 828 : cluster [DBG] osdmap e191: 8 total, 8 up, 8 in 2021-07-31T01:48:12.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:11 smithi045 conmon[39718]: cluster 2021 2021-07-31T01:48:12.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:11 smithi045 conmon[39718]: -07-31T01:48:11.499053 2021-07-31T01:48:12.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:11 smithi045 conmon[39718]: +0000 mon.a (mon.0 2021-07-31T01:48:12.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:11 smithi045 conmon[39718]: ) 829 : cluster [DBG] osdmap e192: 8 total, 8 up, 8 in 2021-07-31T01:48:12.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:11 smithi045 conmon[39718]: 2021-07-31T01:48:12.507 INFO:tasks.rados.rados.0.smithi045.stdout:783: done (0 left) 2021-07-31T01:48:12.508 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 14 current snap is 67 2021-07-31T01:48:12.508 INFO:tasks.rados.rados.0.smithi045.stdout:784: seq_num 242 ranges {0=458752,458752=753664,1212416=450560} 2021-07-31T01:48:12.510 INFO:tasks.rados.rados.0.smithi045.stdout:784: writing smithi04569269-14 from 0 to 458752 tid 1 2021-07-31T01:48:12.515 INFO:tasks.rados.rados.0.smithi045.stdout:784: writing smithi04569269-14 from 458752 to 1212416 tid 2 2021-07-31T01:48:12.518 INFO:tasks.rados.rados.0.smithi045.stdout:784: writing smithi04569269-14 from 1212416 to 1662976 tid 3 2021-07-31T01:48:12.519 INFO:tasks.rados.rados.0.smithi045.stdout:785: read oid 31 snap -1 2021-07-31T01:48:12.519 INFO:tasks.rados.rados.0.smithi045.stdout:785: expect (ObjNum 0 snap 0 seq_num 0) 2021-07-31T01:48:12.519 INFO:tasks.rados.rados.0.smithi045.stdout:786: read oid 2 snap -1 2021-07-31T01:48:12.519 INFO:tasks.rados.rados.0.smithi045.stdout:786: expect deleted 2021-07-31T01:48:12.520 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 15 current snap is 67 2021-07-31T01:48:12.520 INFO:tasks.rados.rados.0.smithi045.stdout:787: seq_num 243 ranges {1441792=475136,1916928=638976,2555904=565248} 2021-07-31T01:48:12.528 INFO:tasks.rados.rados.0.smithi045.stdout:787: writing smithi04569269-15 from 1441792 to 1916928 tid 1 2021-07-31T01:48:12.532 INFO:tasks.rados.rados.0.smithi045.stdout:787: writing smithi04569269-15 from 1916928 to 2555904 tid 2 2021-07-31T01:48:12.536 INFO:tasks.rados.rados.0.smithi045.stdout:787: writing smithi04569269-15 from 2555904 to 3121152 tid 3 2021-07-31T01:48:12.536 INFO:tasks.rados.rados.0.smithi045.stdout:788: read oid 46 snap -1 2021-07-31T01:48:12.536 INFO:tasks.rados.rados.0.smithi045.stdout:788: expect (ObjNum 196 snap 52 seq_num 196) 2021-07-31T01:48:12.537 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 50 current snap is 67 2021-07-31T01:48:12.537 INFO:tasks.rados.rados.0.smithi045.stdout:789: seq_num 244 ranges {0=491520,491520=573440,1064960=458752,1523712=172032} 2021-07-31T01:48:12.538 INFO:tasks.rados.rados.0.smithi045.stdout:789: writing smithi04569269-50 from 0 to 491520 tid 1 2021-07-31T01:48:12.542 INFO:tasks.rados.rados.0.smithi045.stdout:789: writing smithi04569269-50 from 491520 to 1064960 tid 2 2021-07-31T01:48:12.544 INFO:tasks.rados.rados.0.smithi045.stdout:789: writing smithi04569269-50 from 1064960 to 1523712 tid 3 2021-07-31T01:48:12.546 INFO:tasks.rados.rados.0.smithi045.stdout:789: writing smithi04569269-50 from 1523712 to 1695744 tid 4 2021-07-31T01:48:12.546 INFO:tasks.rados.rados.0.smithi045.stdout:790: read oid 26 snap -1 2021-07-31T01:48:12.547 INFO:tasks.rados.rados.0.smithi045.stdout:790: expect (ObjNum 142 snap 30 seq_num 142) 2021-07-31T01:48:12.547 INFO:tasks.rados.rados.0.smithi045.stdout:791: snap_create 2021-07-31T01:48:12.547 INFO:tasks.rados.rados.0.smithi045.stdout:784: finishing write tid 1 to smithi04569269-14 2021-07-31T01:48:12.547 INFO:tasks.rados.rados.0.smithi045.stdout:785: expect (ObjNum 0 snap 0 seq_num 0) 2021-07-31T01:48:12.549 INFO:tasks.rados.rados.0.smithi045.stdout:784: finishing write tid 2 to smithi04569269-14 2021-07-31T01:48:12.550 INFO:tasks.rados.rados.0.smithi045.stdout:784: finishing write tid 3 to smithi04569269-14 2021-07-31T01:48:12.551 INFO:tasks.rados.rados.0.smithi045.stdout:784: finishing write tid 4 to smithi04569269-14 2021-07-31T01:48:12.551 INFO:tasks.rados.rados.0.smithi045.stdout:784: finishing write tid 5 to smithi04569269-14 2021-07-31T01:48:12.552 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 14 v 271 (ObjNum 242 snap 67 seq_num 242) dirty exists 2021-07-31T01:48:12.552 INFO:tasks.rados.rados.0.smithi045.stdout:784: left oid 14 (ObjNum 242 snap 67 seq_num 242) 2021-07-31T01:48:12.552 INFO:tasks.rados.rados.0.smithi045.stdout:787: finishing write tid 1 to smithi04569269-15 2021-07-31T01:48:12.552 INFO:tasks.rados.rados.0.smithi045.stdout:787: finishing write tid 2 to smithi04569269-15 2021-07-31T01:48:12.553 INFO:tasks.rados.rados.0.smithi045.stdout:787: finishing write tid 3 to smithi04569269-15 2021-07-31T01:48:12.553 INFO:tasks.rados.rados.0.smithi045.stdout:787: finishing write tid 4 to smithi04569269-15 2021-07-31T01:48:12.553 INFO:tasks.rados.rados.0.smithi045.stdout:787: finishing write tid 5 to smithi04569269-15 2021-07-31T01:48:12.553 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 15 v 232 (ObjNum 243 snap 67 seq_num 243) dirty exists 2021-07-31T01:48:12.553 INFO:tasks.rados.rados.0.smithi045.stdout:787: left oid 15 (ObjNum 243 snap 67 seq_num 243) 2021-07-31T01:48:12.554 INFO:tasks.rados.rados.0.smithi045.stdout:789: finishing write tid 1 to smithi04569269-50 2021-07-31T01:48:12.554 INFO:tasks.rados.rados.0.smithi045.stdout:789: finishing write tid 2 to smithi04569269-50 2021-07-31T01:48:12.556 INFO:tasks.rados.rados.0.smithi045.stdout:789: finishing write tid 3 to smithi04569269-50 2021-07-31T01:48:12.556 INFO:tasks.rados.rados.0.smithi045.stdout:789: finishing write tid 4 to smithi04569269-50 2021-07-31T01:48:12.557 INFO:tasks.rados.rados.0.smithi045.stdout:789: finishing write tid 5 to smithi04569269-50 2021-07-31T01:48:12.565 INFO:tasks.rados.rados.0.smithi045.stdout:789: finishing write tid 6 to smithi04569269-50 2021-07-31T01:48:12.565 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 50 v 350 (ObjNum 244 snap 67 seq_num 244) dirty exists 2021-07-31T01:48:12.566 INFO:tasks.rados.rados.0.smithi045.stdout:789: left oid 50 (ObjNum 244 snap 67 seq_num 244) 2021-07-31T01:48:12.567 INFO:tasks.rados.rados.0.smithi045.stdout:788: expect (ObjNum 196 snap 52 seq_num 196) 2021-07-31T01:48:12.793 INFO:tasks.rados.rados.0.smithi045.stdout:790: expect (ObjNum 142 snap 30 seq_num 142) 2021-07-31T01:48:13.003 INFO:journalctl@ceph.osd.6.smithi154.stdout:Jul 31 01:48:12 smithi154 conmon[45576]: debug 2021-07-31T01:48:13.003 INFO:journalctl@ceph.osd.6.smithi154.stdout:Jul 31 01:48:12 smithi154 conmon[45576]: 2021-07-31T01:48:12.508+0000 7f47c50ad700 -1 osd.6 pg_epoch: 193 pg[3.as0( v 193'261 (0'0,193'261] local-lis/les=56/57 n=8 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [6,4,1]p6(0) r=0 lpr=56 luod=191'259 crt=191'259 lcod 191'257 mlcod 191'257 active+clean+snaptrim trimq=[7c~1] ps=[73~1,79~1,82~1]] removing snap head 2021-07-31T01:48:13.003 INFO:journalctl@ceph.osd.6.smithi154.stdout:Jul 31 01:48:12 smithi154 conmon[45576]: 2021-07-31T01:48:13.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:12 smithi154 conmon[34553]: cluster 2021-07-31T01:48:11.804283+0000 mgr.x (mgr.24542 2021-07-31T01:48:13.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:12 smithi154 conmon[34553]: ) 42 : cluster [DBG] pgmap v68: 25 pgs: 25 active+clean; 111 MiB data, 504 MiB used, 715 GiB / 715 GiB avail; 206 KiB/s rd, 3.0 MiB/s wr, 23 op/s 2021-07-31T01:48:13.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:12 smithi154 conmon[34553]: cluster 2021-07-31T01:48:13.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:12 smithi154 conmon[34553]: 2021-07-31T01:48:12.506066+0000 mon.a (mon.0) 830 : cluster [DBG] osdmap e193: 8 total, 8 up, 8 in 2021-07-31T01:48:13.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:12 smithi045 conmon[39718]: cluster 2021-07-31T01: 2021-07-31T01:48:13.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:12 smithi045 conmon[39718]: 48:11.804283+0000 mgr.x (mgr.24542) 42 : cluster [DBG] pgmap v68: 25 pgs: 25 active+clean; 111 MiB data, 504 MiB used, 715 GiB / 715 GiB avail; 206 KiB/s rd, 3.0 MiB/s wr, 23 op/s 2021-07-31T01:48:13.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:12 smithi045 conmon[39718]: cluster 2021-07-31 2021-07-31T01:48:13.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:12 smithi045 conmon[39718]: T01:48:12.506066+0000 mon.a (mon.0) 2021-07-31T01:48:13.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:12 smithi045 conmon[39718]: 830 : cluster [DBG] osdmap e193: 8 total, 8 up, 8 in 2021-07-31T01:48:13.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:12 smithi045 conmon[33655]: cluster 2021-07-31T01:48:11 2021-07-31T01:48:13.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:12 smithi045 conmon[33655]: .804283+0000 mgr.x (mgr.24542) 42 : cluster [DBG] pgmap v68: 25 pgs: 25 active+clean; 111 MiB data, 504 MiB used, 715 GiB / 715 GiB avail; 206 KiB/s rd, 3.0 MiB/s wr, 23 op/s 2021-07-31T01:48:13.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:12 smithi045 conmon[33655]: cluster 2021-07-31T01:48: 2021-07-31T01:48:13.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:12 smithi045 conmon[33655]: 12.506066 2021-07-31T01:48:13.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:12 smithi045 conmon[33655]: +0000 mon.a 2021-07-31T01:48:13.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:12 smithi045 conmon[33655]: (mon.0) 830 : cluster [DBG] osdmap e193: 8 total, 8 up, 8 in 2021-07-31T01:48:13.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:12 smithi045 conmon[33655]: 2021-07-31T01:48:13.509 INFO:tasks.rados.rados.0.smithi045.stdout:784: done (7 left) 2021-07-31T01:48:13.509 INFO:tasks.rados.rados.0.smithi045.stdout:785: done (6 left) 2021-07-31T01:48:13.510 INFO:tasks.rados.rados.0.smithi045.stdout:786: done (5 left) 2021-07-31T01:48:13.510 INFO:tasks.rados.rados.0.smithi045.stdout:787: done (4 left) 2021-07-31T01:48:13.510 INFO:tasks.rados.rados.0.smithi045.stdout:788: done (3 left) 2021-07-31T01:48:13.510 INFO:tasks.rados.rados.0.smithi045.stdout:789: done (2 left) 2021-07-31T01:48:13.511 INFO:tasks.rados.rados.0.smithi045.stdout:790: done (1 left) 2021-07-31T01:48:13.511 INFO:tasks.rados.rados.0.smithi045.stdout:791: done (0 left) 2021-07-31T01:48:13.511 INFO:tasks.rados.rados.0.smithi045.stdout:792: copy_from oid 31 from oid 39 current snap is 68 2021-07-31T01:48:13.511 INFO:tasks.rados.rados.0.smithi045.stdout:793: copy_from oid 28 from oid 43 current snap is 68 2021-07-31T01:48:13.512 INFO:tasks.rados.rados.0.smithi045.stdout:794: delete oid 3 current snap is 68 2021-07-31T01:48:13.513 INFO:tasks.rados.rados.0.smithi045.stdout:793: finishing copy_from to smithi04569269-28 2021-07-31T01:48:13.514 INFO:tasks.rados.rados.0.smithi045.stdout:793: got expected ENOENT (src dne) 2021-07-31T01:48:13.514 INFO:tasks.rados.rados.0.smithi045.stdout:793: finishing copy_from racing read to smithi04569269-28 2021-07-31T01:48:13.515 INFO:tasks.rados.rados.0.smithi045.stdout:793: done (2 left) 2021-07-31T01:48:13.515 INFO:tasks.rados.rados.0.smithi045.stdout:794: done (1 left) 2021-07-31T01:48:13.515 INFO:tasks.rados.rados.0.smithi045.stdout:795: read oid 13 snap -1 2021-07-31T01:48:13.515 INFO:tasks.rados.rados.0.smithi045.stdout:795: expect (ObjNum 144 snap 31 seq_num 144) 2021-07-31T01:48:13.517 INFO:tasks.rados.rados.0.smithi045.stdout:796: read oid 35 snap 67 2021-07-31T01:48:13.517 INFO:tasks.rados.rados.0.smithi045.stdout:796: expect (ObjNum 232 snap 64 seq_num 232) 2021-07-31T01:48:13.517 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 26 current snap is 68 2021-07-31T01:48:13.517 INFO:tasks.rados.rados.0.smithi045.stdout:797: seq_num 246 ranges {4308992=524288,4833280=745472,5578752=425984,6004736=32768} 2021-07-31T01:48:13.540 INFO:tasks.rados.rados.0.smithi045.stdout:797: writing smithi04569269-26 from 4308992 to 4833280 tid 1 2021-07-31T01:48:13.544 INFO:tasks.rados.rados.0.smithi045.stdout:797: writing smithi04569269-26 from 4833280 to 5578752 tid 2 2021-07-31T01:48:13.547 INFO:tasks.rados.rados.0.smithi045.stdout:797: writing smithi04569269-26 from 5578752 to 6004736 tid 3 2021-07-31T01:48:13.547 INFO:tasks.rados.rados.0.smithi045.stdout:797: writing smithi04569269-26 from 6004736 to 6037504 tid 4 2021-07-31T01:48:13.547 INFO:tasks.rados.rados.0.smithi045.stdout:798: read oid 44 snap 67 2021-07-31T01:48:13.548 INFO:tasks.rados.rados.0.smithi045.stdout:798: expect (ObjNum 142 snap 30 seq_num 142) 2021-07-31T01:48:13.548 INFO:tasks.rados.rados.0.smithi045.stdout:799: copy_from oid 38 from oid 50 current snap is 68 2021-07-31T01:48:13.548 INFO:tasks.rados.rados.0.smithi045.stdout:795: expect (ObjNum 144 snap 31 seq_num 144) 2021-07-31T01:48:13.583 INFO:tasks.rados.rados.0.smithi045.stdout:796: expect (ObjNum 232 snap 64 seq_num 232) 2021-07-31T01:48:13.735 INFO:tasks.rados.rados.0.smithi045.stdout:792: finishing copy_from to smithi04569269-31 2021-07-31T01:48:13.735 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 31 v 248 (ObjNum 197 snap 52 seq_num 197) dirty exists 2021-07-31T01:48:13.736 INFO:tasks.rados.rados.0.smithi045.stdout:792: finishing copy_from racing read to smithi04569269-31 2021-07-31T01:48:13.736 INFO:tasks.rados.rados.0.smithi045.stdout:797: finishing write tid 1 to smithi04569269-26 2021-07-31T01:48:13.736 INFO:tasks.rados.rados.0.smithi045.stdout:797: finishing write tid 2 to smithi04569269-26 2021-07-31T01:48:13.736 INFO:tasks.rados.rados.0.smithi045.stdout:797: finishing write tid 3 to smithi04569269-26 2021-07-31T01:48:13.736 INFO:tasks.rados.rados.0.smithi045.stdout:797: finishing write tid 4 to smithi04569269-26 2021-07-31T01:48:13.737 INFO:tasks.rados.rados.0.smithi045.stdout:797: finishing write tid 5 to smithi04569269-26 2021-07-31T01:48:13.737 INFO:tasks.rados.rados.0.smithi045.stdout:797: finishing write tid 6 to smithi04569269-26 2021-07-31T01:48:13.737 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 26 v 196 (ObjNum 246 snap 68 seq_num 246) dirty exists 2021-07-31T01:48:13.737 INFO:tasks.rados.rados.0.smithi045.stdout:797: left oid 26 (ObjNum 246 snap 68 seq_num 246) 2021-07-31T01:48:13.738 INFO:tasks.rados.rados.0.smithi045.stdout:792: done (5 left) 2021-07-31T01:48:13.738 INFO:tasks.rados.rados.0.smithi045.stdout:795: done (4 left) 2021-07-31T01:48:13.738 INFO:tasks.rados.rados.0.smithi045.stdout:796: done (3 left) 2021-07-31T01:48:13.738 INFO:tasks.rados.rados.0.smithi045.stdout:797: done (2 left) 2021-07-31T01:48:13.738 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 47 current snap is 68 2021-07-31T01:48:13.739 INFO:tasks.rados.rados.0.smithi045.stdout:799: finishing copy_from to smithi04569269-38 2021-07-31T01:48:13.739 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 38 v 350 (ObjNum 244 snap 67 seq_num 244) dirty exists 2021-07-31T01:48:13.739 INFO:tasks.rados.rados.0.smithi045.stdout:799: finishing copy_from racing read to smithi04569269-38 2021-07-31T01:48:13.739 INFO:tasks.rados.rados.0.smithi045.stdout:798: expect (ObjNum 142 snap 30 seq_num 142) 2021-07-31T01:48:13.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:48:13 smithi045 conmon[61210]: level=error ts=2021-07-31T01:48:13.509Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:48:13.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:48:13 smithi045 conmon[61210]: level=error ts=2021-07-31T01:48:13.509Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:48:13.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:48:13 smithi045 conmon[61210]: level=error ts=2021-07-31T01:48:13.509Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T01:48:13.882 INFO:tasks.rados.rados.0.smithi045.stdout:800: seq_num 247 ranges {778240=540672,1318912=622592,1941504=581632} 2021-07-31T01:48:13.889 INFO:tasks.rados.rados.0.smithi045.stdout:800: writing smithi04569269-47 from 778240 to 1318912 tid 1 2021-07-31T01:48:13.892 INFO:tasks.rados.rados.0.smithi045.stdout:800: writing smithi04569269-47 from 1318912 to 1941504 tid 2 2021-07-31T01:48:13.896 INFO:tasks.rados.rados.0.smithi045.stdout:800: writing smithi04569269-47 from 1941504 to 2523136 tid 3 2021-07-31T01:48:13.896 INFO:tasks.rados.rados.0.smithi045.stdout:798: done (2 left) 2021-07-31T01:48:13.897 INFO:tasks.rados.rados.0.smithi045.stdout:799: done (1 left) 2021-07-31T01:48:13.897 INFO:tasks.rados.rados.0.smithi045.stdout:801: delete oid 50 current snap is 68 2021-07-31T01:48:13.897 INFO:tasks.rados.rados.0.smithi045.stdout:800: finishing write tid 1 to smithi04569269-47 2021-07-31T01:48:13.897 INFO:tasks.rados.rados.0.smithi045.stdout:800: finishing write tid 2 to smithi04569269-47 2021-07-31T01:48:13.898 INFO:tasks.rados.rados.0.smithi045.stdout:801: done (1 left) 2021-07-31T01:48:13.898 INFO:tasks.rados.rados.0.smithi045.stdout:802: snap_create 2021-07-31T01:48:13.899 INFO:tasks.rados.rados.0.smithi045.stdout:800: finishing write tid 3 to smithi04569269-47 2021-07-31T01:48:13.899 INFO:tasks.rados.rados.0.smithi045.stdout:800: finishing write tid 4 to smithi04569269-47 2021-07-31T01:48:13.899 INFO:tasks.rados.rados.0.smithi045.stdout:800: finishing write tid 5 to smithi04569269-47 2021-07-31T01:48:13.900 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 47 v 153 (ObjNum 247 snap 68 seq_num 247) dirty exists 2021-07-31T01:48:13.900 INFO:tasks.rados.rados.0.smithi045.stdout:800: left oid 47 (ObjNum 247 snap 68 seq_num 247) 2021-07-31T01:48:14.514 INFO:tasks.rados.rados.0.smithi045.stdout:800: done (1 left) 2021-07-31T01:48:14.515 INFO:tasks.rados.rados.0.smithi045.stdout:802: done (0 left) 2021-07-31T01:48:14.515 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 13 current snap is 69 2021-07-31T01:48:14.515 INFO:tasks.rados.rados.0.smithi045.stdout:803: seq_num 249 ranges {794624=573440,1368064=794624,2162688=409600} 2021-07-31T01:48:14.521 INFO:tasks.rados.rados.0.smithi045.stdout:803: writing smithi04569269-13 from 794624 to 1368064 tid 1 2021-07-31T01:48:14.526 INFO:tasks.rados.rados.0.smithi045.stdout:803: writing smithi04569269-13 from 1368064 to 2162688 tid 2 2021-07-31T01:48:14.528 INFO:tasks.rados.rados.0.smithi045.stdout:803: writing smithi04569269-13 from 2162688 to 2572288 tid 3 2021-07-31T01:48:14.528 INFO:tasks.rados.rados.0.smithi045.stdout:804: delete oid 50 current snap is 69 2021-07-31T01:48:14.529 INFO:tasks.rados.rados.0.smithi045.stdout:803: finishing write tid 1 to smithi04569269-13 2021-07-31T01:48:14.531 INFO:tasks.rados.rados.0.smithi045.stdout:804: done (1 left) 2021-07-31T01:48:14.531 INFO:tasks.rados.rados.0.smithi045.stdout:805: rollback oid 21 current snap is 69 2021-07-31T01:48:14.532 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 21 to 58 2021-07-31T01:48:14.532 INFO:tasks.rados.rados.0.smithi045.stdout:806: snap_remove snap 67 2021-07-31T01:48:14.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:14 smithi045 conmon[33655]: cluster 2021-07-31T01:48:13.506478+0000 mon.a 2021-07-31T01:48:14.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:14 smithi045 conmon[33655]: (mon.0) 831 : cluster [DBG] osdmap e194: 8 total, 8 up, 8 in 2021-07-31T01:48:14.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:14 smithi045 conmon[33655]: cluster 2021-07-31T01:48:13.804696+0000 mgr.x (mgr.24542) 43 : cluster [DBG] pgmap v71: 25 pgs: 25 active+clean; 111 MiB data, 504 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:48:14.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:14 smithi045 conmon[39718]: cluster 2021-07-31T01:48:13 2021-07-31T01:48:14.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:14 smithi045 conmon[39718]: .506478+0000 mon.a (mon.0) 831 : cluster [DBG] osdmap e194: 8 total, 8 up, 8 in 2021-07-31T01:48:14.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:14 smithi045 conmon[39718]: cluster 2021-07 2021-07-31T01:48:14.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:14 smithi045 conmon[39718]: -31T01:48:13.804696+0000 2021-07-31T01:48:14.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:14 smithi045 conmon[39718]: mgr.x (mgr.24542) 43 : 2021-07-31T01:48:14.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:14 smithi045 conmon[39718]: cluster [DBG] pgmap v71: 25 pgs: 25 active+clean; 111 MiB data, 504 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:48:15.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:14 smithi154 conmon[34553]: cluster 2021-07-31T01:48:13.506478+0000 mon.a (mon.0) 831 : cluster [DBG] osdmap e194: 8 total, 8 up, 8 in 2021-07-31T01:48:15.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:14 smithi154 conmon[34553]: cluster 2021-07-31T01:48:13.804696+0000 mgr.x (mgr.24542) 43 : cluster [DBG] pgmap v71: 25 pgs: 25 active+clean; 111 MiB data, 504 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:48:15.520 INFO:tasks.rados.rados.0.smithi045.stdout:806: done (2 left) 2021-07-31T01:48:15.520 INFO:tasks.rados.rados.0.smithi045.stdout:807: snap_create 2021-07-31T01:48:15.521 INFO:tasks.rados.rados.0.smithi045.stdout:803: finishing write tid 2 to smithi04569269-13 2021-07-31T01:48:15.521 INFO:tasks.rados.rados.0.smithi045.stdout:805: finishing rollback tid 1 to smithi04569269-21 2021-07-31T01:48:15.521 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 21 v 0 (ObjNum 1752328794 snap 1818653024 seq_num 1516589159) dirty dne 2021-07-31T01:48:15.521 INFO:tasks.rados.rados.0.smithi045.stdout:803: finishing write tid 3 to smithi04569269-13 2021-07-31T01:48:15.522 INFO:tasks.rados.rados.0.smithi045.stdout:803: finishing write tid 4 to smithi04569269-13 2021-07-31T01:48:15.522 INFO:tasks.rados.rados.0.smithi045.stdout:803: finishing write tid 5 to smithi04569269-13 2021-07-31T01:48:15.522 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 13 v 510 (ObjNum 249 snap 69 seq_num 249) dirty exists 2021-07-31T01:48:15.522 INFO:tasks.rados.rados.0.smithi045.stdout:803: left oid 13 (ObjNum 249 snap 69 seq_num 249) 2021-07-31T01:48:15.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:15 smithi045 conmon[39718]: cluster 2021-07-31T01:48:14. 2021-07-31T01:48:15.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:15 smithi045 conmon[39718]: 513858+0000 mon.a (mon.0) 832 : cluster [DBG] osdmap e195: 8 total, 8 up, 8 in 2021-07-31T01:48:15.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:15 smithi045 conmon[33655]: cluster 2021-07-31T01:48:14.513858+0000 mon.a (mon.0 2021-07-31T01:48:15.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:15 smithi045 conmon[33655]: ) 832 : cluster [DBG] osdmap e195: 8 total, 8 up, 8 in 2021-07-31T01:48:15.812 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:15 smithi154 conmon[34553]: cluster 2021-07-31T01:48:14.513858+0000 mon.a (mon.0) 832 : cluster [DBG] osdmap e195: 8 total, 8 up, 8 in 2021-07-31T01:48:16.253 INFO:journalctl@ceph.osd.7.smithi154.stdout:Jul 31 01:48:15 smithi154 conmon[49149]: debug 2021-07-31T01:48:15.809+0000 7f8a719fc700 -1 osd.7 pg_epoch: 196 pg[3.ds0( v 195'353 (0'0,195'353] local-lis/les=56/57 n=6 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [7,5,3]p7(0) r=0 lpr=56 crt=194'352 lcod 194'352 mlcod 194'352 active+clean+snaptrim trimq=[87~1] ps=[73~1,79~1,7c~1,82~1]] removing snap head 2021-07-31T01:48:16.522 INFO:tasks.rados.rados.0.smithi045.stdout:803: done (2 left) 2021-07-31T01:48:16.522 INFO:tasks.rados.rados.0.smithi045.stdout:805: done (1 left) 2021-07-31T01:48:16.522 INFO:tasks.rados.rados.0.smithi045.stdout:807: done (0 left) 2021-07-31T01:48:16.522 INFO:tasks.rados.rados.0.smithi045.stdout:808: read oid 46 snap -1 2021-07-31T01:48:16.523 INFO:tasks.rados.rados.0.smithi045.stdout:808: expect (ObjNum 196 snap 52 seq_num 196) 2021-07-31T01:48:16.523 INFO:tasks.rados.rados.0.smithi045.stdout:809: read oid 12 snap -1 2021-07-31T01:48:16.523 INFO:tasks.rados.rados.0.smithi045.stdout:809: expect (ObjNum 235 snap 66 seq_num 235) 2021-07-31T01:48:16.524 INFO:tasks.rados.rados.0.smithi045.stdout:810: read oid 29 snap -1 2021-07-31T01:48:16.524 INFO:tasks.rados.rados.0.smithi045.stdout:810: expect (ObjNum 237 snap 66 seq_num 237) 2021-07-31T01:48:16.524 INFO:tasks.rados.rados.0.smithi045.stdout:811: snap_remove snap 68 2021-07-31T01:48:17.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:16 smithi154 conmon[34553]: cluster 2021-07-31T01:48:15.519619+0000 mon.a (mon.0 2021-07-31T01:48:17.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:16 smithi154 conmon[34553]: ) 833 : cluster [DBG] osdmap e196: 8 total, 8 up, 8 in 2021-07-31T01:48:17.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:16 smithi154 conmon[34553]: cluster 2021-07-31T01:48:15.805217+0000 mgr.x (mgr.24542) 44 : cluster [DBG] pgmap v74: 25 pgs: 25 active+clean; 116 MiB data, 531 MiB used, 715 GiB / 715 GiB avail; 18 MiB/s rd, 3.7 MiB/s wr, 22 op/s 2021-07-31T01:48:17.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:16 smithi045 conmon[33655]: cluster 2021-07-31T01:48:17.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:16 smithi045 conmon[33655]: 2021-07-31T01:48:15 2021-07-31T01:48:17.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:16 smithi045 conmon[33655]: .519619+0000 mon.a (mon.0) 833 : cluster [DBG] osdmap e196: 8 total, 8 up, 8 in 2021-07-31T01:48:17.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:16 smithi045 conmon[33655]: cluster 2021-07-31T01:48:17.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:16 smithi045 conmon[33655]: 2021-07-31T01:48:15.805217+0000 mgr.x (mgr.24542) 44 2021-07-31T01:48:17.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:16 smithi045 conmon[33655]: : cluster [DBG] pgmap v74: 25 pgs: 25 active+clean; 116 MiB data, 531 MiB used, 715 GiB / 715 GiB avail; 18 MiB/s rd, 3.7 MiB/s wr, 22 op/s 2021-07-31T01:48:17.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:16 smithi045 conmon[39718]: cluster 2021-07-31T01:48:15.519619+0000 mon.a (mon. 2021-07-31T01:48:17.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:16 smithi045 conmon[39718]: 0) 833 : cluster [DBG] osdmap e196: 8 total, 8 up, 8 in 2021-07-31T01:48:17.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:16 smithi045 conmon[39718]: cluster 2021-07-31T01:48:15.805217+0000 mgr.x ( 2021-07-31T01:48:17.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:16 smithi045 conmon[39718]: mgr.24542) 44 : cluster [DBG] pgmap v74: 25 pgs: 25 active+clean; 116 MiB data, 531 MiB used, 715 GiB / 715 GiB avail; 18 MiB/s rd, 3.7 MiB/s wr, 22 op/s 2021-07-31T01:48:17.532 INFO:tasks.rados.rados.0.smithi045.stdout:811: done (3 left) 2021-07-31T01:48:17.532 INFO:tasks.rados.rados.0.smithi045.stdout:812: setattr oid 42 current snap is 70 2021-07-31T01:48:17.532 INFO:tasks.rados.rados.0.smithi045.stdout:810: expect (ObjNum 237 snap 66 seq_num 237) 2021-07-31T01:48:17.573 INFO:tasks.rados.rados.0.smithi045.stdout:809: expect (ObjNum 235 snap 66 seq_num 235) 2021-07-31T01:48:17.613 INFO:tasks.rados.rados.0.smithi045.stdout:808: expect (ObjNum 196 snap 52 seq_num 196) 2021-07-31T01:48:17.837 INFO:tasks.rados.rados.0.smithi045.stdout:808: done (3 left) 2021-07-31T01:48:17.838 INFO:tasks.rados.rados.0.smithi045.stdout:809: done (2 left) 2021-07-31T01:48:17.838 INFO:tasks.rados.rados.0.smithi045.stdout:810: done (1 left) 2021-07-31T01:48:17.838 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 12 current snap is 70 2021-07-31T01:48:17.839 INFO:tasks.rados.rados.0.smithi045.stdout:813: seq_num 251 ranges {1548288=606208,2154496=614400,2768896=589824} 2021-07-31T01:48:17.849 INFO:tasks.rados.rados.0.smithi045.stdout:813: writing smithi04569269-12 from 1548288 to 2154496 tid 1 2021-07-31T01:48:17.853 INFO:tasks.rados.rados.0.smithi045.stdout:813: writing smithi04569269-12 from 2154496 to 2768896 tid 2 2021-07-31T01:48:17.857 INFO:tasks.rados.rados.0.smithi045.stdout:813: writing smithi04569269-12 from 2768896 to 3358720 tid 3 2021-07-31T01:48:17.857 INFO:tasks.rados.rados.0.smithi045.stdout:814: read oid 21 snap 58 2021-07-31T01:48:17.858 INFO:tasks.rados.rados.0.smithi045.stdout:814: expect deleted 2021-07-31T01:48:17.858 INFO:tasks.rados.rados.0.smithi045.stdout:815: read oid 34 snap 69 2021-07-31T01:48:17.858 INFO:tasks.rados.rados.0.smithi045.stdout:815: expect (ObjNum 76 snap 11 seq_num 76) 2021-07-31T01:48:17.858 INFO:tasks.rados.rados.0.smithi045.stdout:816: rollback oid 32 current snap is 70 2021-07-31T01:48:17.859 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 32 to 58 2021-07-31T01:48:17.859 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 42 v 357 (ObjNum 176 snap 46 seq_num 176) dirty exists 2021-07-31T01:48:17.859 INFO:tasks.rados.rados.0.smithi045.stdout:813: finishing write tid 1 to smithi04569269-12 2021-07-31T01:48:17.859 INFO:tasks.rados.rados.0.smithi045.stdout:813: finishing write tid 2 to smithi04569269-12 2021-07-31T01:48:17.860 INFO:tasks.rados.rados.0.smithi045.stdout:812: done (4 left) 2021-07-31T01:48:17.860 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 11 current snap is 70 2021-07-31T01:48:17.860 INFO:tasks.rados.rados.0.smithi045.stdout:817: seq_num 252 ranges {3751936=630784,4382720=499712,4882432=425984,5308416=278528} 2021-07-31T01:48:17.881 INFO:tasks.rados.rados.0.smithi045.stdout:817: writing smithi04569269-11 from 3751936 to 4382720 tid 1 2021-07-31T01:48:17.884 INFO:tasks.rados.rados.0.smithi045.stdout:817: writing smithi04569269-11 from 4382720 to 4882432 tid 2 2021-07-31T01:48:17.886 INFO:tasks.rados.rados.0.smithi045.stdout:817: writing smithi04569269-11 from 4882432 to 5308416 tid 3 2021-07-31T01:48:17.888 INFO:tasks.rados.rados.0.smithi045.stdout:817: writing smithi04569269-11 from 5308416 to 5586944 tid 4 2021-07-31T01:48:17.888 INFO:tasks.rados.rados.0.smithi045.stdout:818: rollback oid 37 current snap is 70 2021-07-31T01:48:17.888 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 37 to 58 2021-07-31T01:48:17.888 INFO:tasks.rados.rados.0.smithi045.stdout:813: finishing write tid 3 to smithi04569269-12 2021-07-31T01:48:17.889 INFO:tasks.rados.rados.0.smithi045.stdout:813: finishing write tid 4 to smithi04569269-12 2021-07-31T01:48:17.889 INFO:tasks.rados.rados.0.smithi045.stdout:816: finishing rollback tid 0 to smithi04569269-32 2021-07-31T01:48:17.889 INFO:tasks.rados.rados.0.smithi045.stdout:816: finishing rollback tid 1 to smithi04569269-32 2021-07-31T01:48:17.889 INFO:tasks.rados.rados.0.smithi045.stdout:816: finishing rollback tid 2 to smithi04569269-32 2021-07-31T01:48:17.889 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 32 v 199 (ObjNum 140 snap 30 seq_num 140) dirty exists 2021-07-31T01:48:17.890 INFO:tasks.rados.rados.0.smithi045.stdout:813: finishing write tid 5 to smithi04569269-12 2021-07-31T01:48:17.890 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 12 v 514 (ObjNum 251 snap 70 seq_num 251) dirty exists 2021-07-31T01:48:17.890 INFO:tasks.rados.rados.0.smithi045.stdout:813: left oid 12 (ObjNum 251 snap 70 seq_num 251) 2021-07-31T01:48:17.890 INFO:tasks.rados.rados.0.smithi045.stdout:815: expect (ObjNum 76 snap 11 seq_num 76) 2021-07-31T01:48:17.953 INFO:tasks.rados.rados.0.smithi045.stdout:817: finishing write tid 1 to smithi04569269-11 2021-07-31T01:48:17.953 INFO:tasks.rados.rados.0.smithi045.stdout:817: finishing write tid 2 to smithi04569269-11 2021-07-31T01:48:17.953 INFO:tasks.rados.rados.0.smithi045.stdout:818: finishing rollback tid 1 to smithi04569269-37 2021-07-31T01:48:17.954 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 37 v 0 (ObjNum -1126017128 snap 21873 seq_num 1) dirty dne 2021-07-31T01:48:17.954 INFO:tasks.rados.rados.0.smithi045.stdout:813: done (5 left) 2021-07-31T01:48:17.954 INFO:tasks.rados.rados.0.smithi045.stdout:814: done (4 left) 2021-07-31T01:48:17.954 INFO:tasks.rados.rados.0.smithi045.stdout:815: done (3 left) 2021-07-31T01:48:17.955 INFO:tasks.rados.rados.0.smithi045.stdout:816: done (2 left) 2021-07-31T01:48:17.955 INFO:tasks.rados.rados.0.smithi045.stdout:818: done (1 left) 2021-07-31T01:48:17.955 INFO:tasks.rados.rados.0.smithi045.stdout:819: read oid 21 snap -1 2021-07-31T01:48:17.955 INFO:tasks.rados.rados.0.smithi045.stdout:819: expect deleted 2021-07-31T01:48:17.955 INFO:tasks.rados.rados.0.smithi045.stdout:820: snap_create 2021-07-31T01:48:17.956 INFO:tasks.rados.rados.0.smithi045.stdout:817: finishing write tid 3 to smithi04569269-11 2021-07-31T01:48:17.956 INFO:tasks.rados.rados.0.smithi045.stdout:817: finishing write tid 4 to smithi04569269-11 2021-07-31T01:48:17.956 INFO:tasks.rados.rados.0.smithi045.stdout:817: finishing write tid 5 to smithi04569269-11 2021-07-31T01:48:17.957 INFO:tasks.rados.rados.0.smithi045.stdout:817: finishing write tid 6 to smithi04569269-11 2021-07-31T01:48:17.957 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 11 v 513 (ObjNum 252 snap 70 seq_num 252) dirty exists 2021-07-31T01:48:17.957 INFO:tasks.rados.rados.0.smithi045.stdout:817: left oid 11 (ObjNum 252 snap 70 seq_num 252) 2021-07-31T01:48:18.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:17 smithi154 conmon[34553]: cluster 2021-07-31T01:48:16.520567+0000 mon.a (mon.0) 834 : cluster [DBG] 2021-07-31T01:48:18.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:17 smithi154 conmon[34553]: osdmap e197: 8 total, 8 up, 8 in 2021-07-31T01:48:18.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:17 smithi045 conmon[33655]: cluster 2021-07-31T01:48:16.520567+0000 2021-07-31T01:48:18.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:17 smithi045 conmon[33655]: mon.a (mon.0) 834 : cluster [DBG] osdmap e197: 8 total, 8 up, 8 in 2021-07-31T01:48:18.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:17 smithi045 conmon[39718]: cluster 2021-07-31T01:48:16.520567 2021-07-31T01:48:18.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:17 smithi045 conmon[39718]: +0000 mon.a (mon.0) 834 : cluster [DBG] osdmap e197: 8 total, 8 up, 8 in 2021-07-31T01:48:18.542 INFO:tasks.rados.rados.0.smithi045.stdout:817: done (2 left) 2021-07-31T01:48:18.542 INFO:tasks.rados.rados.0.smithi045.stdout:819: done (1 left) 2021-07-31T01:48:18.543 INFO:tasks.rados.rados.0.smithi045.stdout:820: done (0 left) 2021-07-31T01:48:18.543 INFO:tasks.rados.rados.0.smithi045.stdout:821: snap_remove snap 69 2021-07-31T01:48:19.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:18 smithi154 conmon[34553]: cluster 2021-07-31T01:48:17.530626+0000 mon.a (mon.0) 835 : cluster [DBG] osdmap e198: 8 total, 8 up, 8 in 2021-07-31T01:48:19.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:18 smithi154 conmon[34553]: cluster 2021-07-31T01:48:17.805662+0000 mgr.x (mgr.24542) 45 : cluster [DBG] pgmap v77: 25 pgs: 25 active+clean; 116 MiB data, 531 MiB used, 715 GiB / 715 GiB avail; 18 MiB/s rd, 3.7 MiB/s wr, 22 op/s 2021-07-31T01:48:19.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:18 smithi045 conmon[39718]: cluster 2021-07-31T01:48: 2021-07-31T01:48:19.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:18 smithi045 conmon[39718]: 17.530626+0000 mon.a (mon.0) 835 : cluster [DBG] osdmap e198: 8 total, 8 up, 8 in 2021-07-31T01:48:19.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:18 smithi045 conmon[39718]: cluster 2021-07 2021-07-31T01:48:19.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:18 smithi045 conmon[39718]: -31T01:48:17. 2021-07-31T01:48:19.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:18 smithi045 conmon[39718]: 805662+0000 mgr.x (mgr. 2021-07-31T01:48:19.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:18 smithi045 conmon[39718]: 24542) 45 : cluster [DBG] pgmap v77: 25 pgs: 25 active+clean; 116 MiB data, 531 MiB used, 715 GiB / 715 GiB avail; 18 MiB/s rd, 3.7 MiB/s wr, 22 op/s 2021-07-31T01:48:19.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:18 smithi045 conmon[39718]: 2021-07-31T01:48:19.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:18 smithi045 conmon[33655]: cluster 2021-07-31T01:48:17.530626+0000 mon.a ( 2021-07-31T01:48:19.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:18 smithi045 conmon[33655]: mon.0) 835 : cluster [DBG] osdmap e198: 8 total, 8 up, 8 in 2021-07-31T01:48:19.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:18 smithi045 conmon[33655]: cluster 2021-07-31T01:48:17.805662+0000 2021-07-31T01:48:19.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:18 smithi045 conmon[33655]: mgr.x (mgr.24542) 45 : cluster [DBG] pgmap v77: 25 pgs: 25 active+clean; 116 MiB data, 531 MiB used, 715 GiB / 715 GiB avail; 18 MiB/s rd, 3.7 MiB/s wr, 22 op/s 2021-07-31T01:48:19.543 INFO:tasks.rados.rados.0.smithi045.stdout:821: done (0 left) 2021-07-31T01:48:19.544 INFO:tasks.rados.rados.0.smithi045.stdout:822: rmattr oid 41 current snap is 71 2021-07-31T01:48:19.544 INFO:tasks.rados.rados.0.smithi045.stdout:822: done (0 left) 2021-07-31T01:48:19.545 INFO:tasks.rados.rados.0.smithi045.stdout:823: snap_remove snap 70 2021-07-31T01:48:20.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:19 smithi154 conmon[34553]: cluster 2021-07-31T01:48:18.541522+0000 mon.a (mon.0) 836 : 2021-07-31T01:48:20.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:19 smithi154 conmon[34553]: cluster [DBG] osdmap e199: 8 total, 8 up, 8 in 2021-07-31T01:48:20.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:19 smithi045 conmon[39718]: cluster 2021-07-31T01:48:20.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:19 smithi045 conmon[39718]: 2021-07-31T01:48:18 2021-07-31T01:48:20.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:19 smithi045 conmon[39718]: .541522+0000 mon.a (mon.0) 836 : cluster [DBG] osdmap e199: 8 total, 8 up, 8 in 2021-07-31T01:48:20.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:19 smithi045 conmon[33655]: cluster 2021-07-31T01:48:18.541522+0000 mon.a (mon.0) 836 : cluster [DBG] osdmap e199: 8 total, 8 up, 8 in 2021-07-31T01:48:20.556 INFO:tasks.rados.rados.0.smithi045.stdout:823: done (0 left) 2021-07-31T01:48:20.556 INFO:tasks.rados.rados.0.smithi045.stdout:824: setattr oid 11 current snap is 71 2021-07-31T01:48:20.557 INFO:tasks.rados.rados.0.smithi045.stdout:825: rmattr oid 31 current snap is 71 2021-07-31T01:48:20.557 INFO:tasks.rados.rados.0.smithi045.stdout:826: rollback oid 1 current snap is 71 2021-07-31T01:48:20.558 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 1 to 58 2021-07-31T01:48:20.558 INFO:tasks.rados.rados.0.smithi045.stdout:827: rollback oid 27 current snap is 71 2021-07-31T01:48:20.558 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 27 to 58 2021-07-31T01:48:20.558 INFO:tasks.rados.rados.0.smithi045.stdout:828: snap_create 2021-07-31T01:48:20.561 INFO:tasks.rados.rados.0.smithi045.stdout:826: finishing rollback tid 0 to smithi04569269-1 2021-07-31T01:48:20.561 INFO:tasks.rados.rados.0.smithi045.stdout:827: finishing rollback tid 1 to smithi04569269-27 2021-07-31T01:48:20.561 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 27 v 0 (ObjNum 1953066355 snap 875587944 seq_num 842610229) dirty dne 2021-07-31T01:48:20.562 INFO:tasks.rados.rados.0.smithi045.stdout:826: finishing rollback tid 1 to smithi04569269-1 2021-07-31T01:48:20.562 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 11 v 514 (ObjNum 252 snap 70 seq_num 252) dirty exists 2021-07-31T01:48:20.563 INFO:tasks.rados.rados.0.smithi045.stdout:826: finishing rollback tid 2 to smithi04569269-1 2021-07-31T01:48:20.563 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 1 v 309 (ObjNum 155 snap 37 seq_num 155) dirty exists 2021-07-31T01:48:20.564 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 31 v 563 (ObjNum 197 snap 52 seq_num 197) dirty exists 2021-07-31T01:48:21.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:20 smithi154 conmon[34553]: cluster 2021-07-31T01:48:19.543063+0000 mon.a (mon.0) 837 : 2021-07-31T01:48:21.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:20 smithi154 conmon[34553]: cluster [DBG] osdmap e200: 8 total, 8 up, 8 in 2021-07-31T01:48:21.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:20 smithi154 conmon[34553]: cluster 2021-07-31T01:48:19.806125+0000 mgr.x (mgr.24542) 46 : cluster [DBG] 2021-07-31T01:48:21.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:20 smithi154 conmon[34553]: pgmap v80: 25 pgs: 25 active+clean; 118 MiB data, 544 MiB used, 715 GiB / 715 GiB avail; 7.2 MiB/s rd, 1.7 MiB/s wr, 15 op/s 2021-07-31T01:48:21.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:20 smithi045 conmon[39718]: cluster 2021-07-31T01:48:19.543063+0000 mon.a ( 2021-07-31T01:48:21.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:20 smithi045 conmon[39718]: mon.0) 837 : cluster [DBG] osdmap e200: 8 total, 8 up, 8 in 2021-07-31T01:48:21.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:20 smithi045 conmon[39718]: cluster 2021-07-31T01 2021-07-31T01:48:21.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:20 smithi045 conmon[39718]: :48:19.806125+0000 mgr.x (mgr.24542) 46 : cluster [DBG] pgmap v80: 25 pgs: 25 active+clean; 118 MiB data, 544 MiB used, 715 GiB / 715 GiB avail; 7.2 MiB/s rd, 1.7 MiB/s wr, 15 op/s 2021-07-31T01:48:21.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:20 smithi045 conmon[33655]: cluster 2021-07-31T01:48:19.543063+0000 2021-07-31T01:48:21.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:20 smithi045 conmon[33655]: mon.a (mon.0) 837 : cluster [DBG] osdmap e200: 8 total, 8 up, 8 in 2021-07-31T01:48:21.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:20 smithi045 conmon[33655]: cluster 2021-07-31T01:48:19.806125+0000 mgr.x (mgr.24542) 46 2021-07-31T01:48:21.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:20 smithi045 conmon[33655]: : cluster [DBG] pgmap v80: 25 pgs: 25 active+clean; 118 MiB data, 544 MiB used, 715 GiB / 715 GiB avail; 7.2 MiB/s rd, 1.7 MiB/s wr, 15 op/s 2021-07-31T01:48:21.502 INFO:tasks.rados.rados.0.smithi045.stdout:824: done (4 left) 2021-07-31T01:48:21.502 INFO:tasks.rados.rados.0.smithi045.stdout:825: done (3 left) 2021-07-31T01:48:21.503 INFO:tasks.rados.rados.0.smithi045.stdout:826: done (2 left) 2021-07-31T01:48:21.504 INFO:tasks.rados.rados.0.smithi045.stdout:827: done (1 left) 2021-07-31T01:48:21.504 INFO:tasks.rados.rados.0.smithi045.stdout:828: done (0 left) 2021-07-31T01:48:21.504 INFO:tasks.rados.rados.0.smithi045.stdout:829: rollback oid 43 current snap is 72 2021-07-31T01:48:21.504 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 43 to 58 2021-07-31T01:48:21.504 INFO:tasks.rados.rados.0.smithi045.stdout:830: snap_create 2021-07-31T01:48:21.507 INFO:tasks.rados.rados.0.smithi045.stdout:829: finishing rollback tid 1 to smithi04569269-43 2021-07-31T01:48:21.508 INFO:tasks.rados.rados.0.smithi045.stdout:829: finishing rollback tid 2 to smithi04569269-43 2021-07-31T01:48:21.509 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 43 v 274 (ObjNum 162 snap 38 seq_num 162) dirty exists 2021-07-31T01:48:22.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:21 smithi154 conmon[34553]: cluster 2021-07-31T01:48:20.555388+0000 mon.a (mon.0) 838 : cluster [DBG] 2021-07-31T01:48:22.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:21 smithi154 conmon[34553]: osdmap e201: 8 total, 8 up, 8 in 2021-07-31T01:48:22.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:21 smithi154 conmon[34553]: cluster 2021-07-31T01:48:21.501408+0000 mon.a (mon.0) 839 : cluster [DBG] osdmap e202: 8 total, 8 up, 8 in 2021-07-31T01:48:22.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:21 smithi045 conmon[39718]: cluster 2021-07-31T01:48:20.555388+0000 mon.a (mon 2021-07-31T01:48:22.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:21 smithi045 conmon[39718]: .0) 838 : cluster [DBG] osdmap e201: 8 total, 8 up, 8 in 2021-07-31T01:48:22.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:21 smithi045 conmon[39718]: cluster 2021-07-31T01:48:21.501408+0000 2021-07-31T01:48:22.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:21 smithi045 conmon[39718]: mon.a (mon.0) 839 : cluster [DBG] osdmap e202: 8 total, 8 up, 8 in 2021-07-31T01:48:22.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:21 smithi045 conmon[33655]: cluster 2021-07-31T01:48:20.555388+0000 mon.a (mon 2021-07-31T01:48:22.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:21 smithi045 conmon[33655]: .0) 838 : cluster [DBG] osdmap e201: 8 total, 8 up, 8 in 2021-07-31T01:48:22.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:21 smithi045 conmon[33655]: cluster 2021-07-31T01:48:21.501408+0000 mon.a (mon.0) 839 : cluster [DBG] osdmap e202: 8 total, 8 up, 8 in 2021-07-31T01:48:22.507 INFO:tasks.rados.rados.0.smithi045.stdout:829: done (1 left) 2021-07-31T01:48:22.508 INFO:tasks.rados.rados.0.smithi045.stdout:830: done (0 left) 2021-07-31T01:48:22.508 INFO:tasks.rados.rados.0.smithi045.stdout:831: rmattr oid 29 current snap is 73 2021-07-31T01:48:22.508 INFO:tasks.rados.rados.0.smithi045.stdout:831: done (0 left) 2021-07-31T01:48:22.509 INFO:tasks.rados.rados.0.smithi045.stdout:832: snap_create 2021-07-31T01:48:23.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:22 smithi154 conmon[34553]: cluster 2021-07-31T01:48:21.806554+0000 mgr.x (mgr.24542) 47 : cluster [DBG] 2021-07-31T01:48:23.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:22 smithi154 conmon[34553]: pgmap v83: 25 pgs: 25 active+clean; 118 MiB data, 544 MiB used, 715 GiB / 715 GiB avail; 7.2 MiB/s rd, 1.7 MiB/s wr, 15 op/s 2021-07-31T01:48:23.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:22 smithi154 conmon[34553]: cluster 2021-07-31T01:48:22.507428+0000 mon.a (mon.0) 840 : cluster [DBG] osdmap e203: 8 total, 8 up, 8 in 2021-07-31T01:48:23.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:22 smithi045 conmon[33655]: cluster 2021-07-31T01:48:21.806554+0000 mgr.x ( 2021-07-31T01:48:23.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:22 smithi045 conmon[33655]: mgr.24542) 47 : cluster [DBG] pgmap v83: 25 pgs: 25 active+clean; 118 MiB data, 544 MiB used, 715 GiB / 715 GiB avail; 7.2 MiB/s rd, 1.7 MiB/s wr, 15 op/s 2021-07-31T01:48:23.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:22 smithi045 conmon[33655]: cluster 2021-07-31T01:48:22.507428+0000 mon.a (mon.0) 840 : cluster [DBG] 2021-07-31T01:48:23.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:22 smithi045 conmon[33655]: osdmap e203: 8 total, 8 up, 8 in 2021-07-31T01:48:23.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:22 smithi045 conmon[39718]: cluster 2021-07-31T01:48:21 2021-07-31T01:48:23.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:22 smithi045 conmon[39718]: .806554+0000 mgr.x (mgr.24542) 47 : cluster [DBG] pgmap v83: 25 pgs: 25 active+clean; 118 MiB data, 544 MiB used, 715 GiB / 715 GiB avail; 7.2 MiB/s rd, 1.7 MiB/s wr, 15 op/s 2021-07-31T01:48:23.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:22 smithi045 conmon[39718]: cluster 2021-07-31T01:48:22.507428 2021-07-31T01:48:23.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:22 smithi045 conmon[39718]: +0000 mon.a (mon.0) 840 : 2021-07-31T01:48:23.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:22 smithi045 conmon[39718]: cluster [DBG] osdmap e203: 8 total, 8 up, 8 in 2021-07-31T01:48:23.509 INFO:tasks.rados.rados.0.smithi045.stdout:832: done (0 left) 2021-07-31T01:48:23.509 INFO:tasks.rados.rados.0.smithi045.stdout:833: delete oid 33 current snap is 74 2021-07-31T01:48:23.512 INFO:tasks.rados.rados.0.smithi045.stdout:833: done (0 left) 2021-07-31T01:48:23.512 INFO:tasks.rados.rados.0.smithi045.stdout:834: setattr oid 48 current snap is 74 2021-07-31T01:48:23.514 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 2 current snap is 74 2021-07-31T01:48:23.515 INFO:tasks.rados.rados.0.smithi045.stdout:835: seq_num 254 ranges {0=663552,663552=663552,1327104=540672} 2021-07-31T01:48:23.518 INFO:tasks.rados.rados.0.smithi045.stdout:835: writing smithi04569269-2 from 0 to 663552 tid 1 2021-07-31T01:48:23.523 INFO:tasks.rados.rados.0.smithi045.stdout:835: writing smithi04569269-2 from 663552 to 1327104 tid 2 2021-07-31T01:48:23.527 INFO:tasks.rados.rados.0.smithi045.stdout:835: writing smithi04569269-2 from 1327104 to 1867776 tid 3 2021-07-31T01:48:23.527 INFO:tasks.rados.rados.0.smithi045.stdout:836: read oid 37 snap -1 2021-07-31T01:48:23.527 INFO:tasks.rados.rados.0.smithi045.stdout:836: expect deleted 2021-07-31T01:48:23.527 INFO:tasks.rados.rados.0.smithi045.stdout:837: copy_from oid 40 from oid 33 current snap is 74 2021-07-31T01:48:23.528 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 48 v 310 (ObjNum 157 snap 37 seq_num 157) dirty exists 2021-07-31T01:48:23.528 INFO:tasks.rados.rados.0.smithi045.stdout:835: finishing write tid 1 to smithi04569269-2 2021-07-31T01:48:23.528 INFO:tasks.rados.rados.0.smithi045.stdout:835: finishing write tid 2 to smithi04569269-2 2021-07-31T01:48:23.529 INFO:tasks.rados.rados.0.smithi045.stdout:834: done (3 left) 2021-07-31T01:48:23.529 INFO:tasks.rados.rados.0.smithi045.stdout:838: snap_remove snap 58 2021-07-31T01:48:23.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:48:23 smithi045 conmon[61210]: level=error ts=2021-07-31T01:48:23.510Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:48:23.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:48:23 smithi045 conmon[61210]: level=error ts=2021-07-31T01:48:23.510Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:48:23.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:48:23 smithi045 conmon[61210]: level=error ts=2021-07-31T01:48:23.510Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T01:48:24.516 INFO:tasks.rados.rados.0.smithi045.stdout:838: done (3 left) 2021-07-31T01:48:24.516 INFO:tasks.rados.rados.0.smithi045.stdout:839: setattr oid 27 current snap is 74 2021-07-31T01:48:24.517 INFO:tasks.rados.rados.0.smithi045.stdout:835: finishing write tid 3 to smithi04569269-2 2021-07-31T01:48:24.517 INFO:tasks.rados.rados.0.smithi045.stdout:835: finishing write tid 4 to smithi04569269-2 2021-07-31T01:48:24.517 INFO:tasks.rados.rados.0.smithi045.stdout:837: finishing copy_from to smithi04569269-40 2021-07-31T01:48:24.517 INFO:tasks.rados.rados.0.smithi045.stdout:837: got expected ENOENT (src dne) 2021-07-31T01:48:24.517 INFO:tasks.rados.rados.0.smithi045.stdout:837: finishing copy_from racing read to smithi04569269-40 2021-07-31T01:48:24.518 INFO:tasks.rados.rados.0.smithi045.stdout:835: finishing write tid 5 to smithi04569269-2 2021-07-31T01:48:24.518 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 2 v 95 (ObjNum 254 snap 74 seq_num 254) dirty exists 2021-07-31T01:48:24.518 INFO:tasks.rados.rados.0.smithi045.stdout:835: left oid 2 (ObjNum 254 snap 74 seq_num 254) 2021-07-31T01:48:24.519 INFO:tasks.rados.rados.0.smithi045.stdout:835: done (3 left) 2021-07-31T01:48:24.519 INFO:tasks.rados.rados.0.smithi045.stdout:836: done (2 left) 2021-07-31T01:48:24.520 INFO:tasks.rados.rados.0.smithi045.stdout:837: done (1 left) 2021-07-31T01:48:24.520 INFO:tasks.rados.rados.0.smithi045.stdout:840: snap_create 2021-07-31T01:48:24.528 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 27 v 279 (ObjNum -1104881600 snap 21873 seq_num 3187445776) dirty exists 2021-07-31T01:48:24.776 INFO:journalctl@ceph.osd.3.smithi045.stdout:Jul 31 01:48:24 smithi045 conmon[54013]: debug 2021-07-31T01:48:24.526+0000 7f64f88c5700 -1 osd.3 pg_epoch: 205 pg[3.bs0( v 205'567 (0'0,205'567] local-lis/les=56/57 n=8 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [3,0,4]p3(0) r=0 lpr=56 crt=205'567 lcod 205'565 mlcod 205'565 active+clean+snaptrim trimq=[74~1] ps=[8a~1,8c~1]] removing snap head 2021-07-31T01:48:24.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:24 smithi045 conmon[39718]: cluster 2021-07-31T01:48:23.508486+0000 mon.a (mon 2021-07-31T01:48:24.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:24 smithi045 conmon[39718]: .0) 841 : cluster [DBG] osdmap e204: 8 total, 8 up, 8 in 2021-07-31T01:48:24.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:24 smithi045 conmon[39718]: cluster 2021-07-31T01:48:23 2021-07-31T01:48:24.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:24 smithi045 conmon[39718]: .806984+0000 mgr.x (mgr.24542) 48 : cluster [DBG] pgmap v86: 25 pgs: 25 active+clean; 118 MiB data, 544 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:48:24.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:24 smithi045 conmon[33655]: cluster 2021-07-31T01:48:23.508486+0000 mon.a (mon.0) 841 : cluster [DBG] osdmap e204: 8 total, 8 up, 8 in 2021-07-31T01:48:24.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:24 smithi045 conmon[33655]: cluster 2021-07-31T01:48:23.806984+0000 mgr.x (mgr.24542) 48 : cluster [DBG] pgmap v86: 25 pgs: 25 active+clean; 118 MiB data, 544 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:48:25.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:24 smithi154 conmon[34553]: cluster 2021-07-31T01:48:23.508486+0000 mon.a (mon.0) 841 : cluster [DBG] osdmap e204: 8 total, 8 up, 8 in 2021-07-31T01:48:25.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:24 smithi154 conmon[34553]: cluster 2021-07-31T01:48:23.806984+0000 mgr.x (mgr.24542) 48 : cluster [DBG] pgmap v86: 25 pgs: 25 active+clean; 118 MiB data, 544 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:48:25.004 INFO:journalctl@ceph.osd.5.smithi154.stdout:Jul 31 01:48:24 smithi154 conmon[42050]: debug 2021-07-31T01:48:24.517+0000 7fc9b5064700 -1 osd.5 pg_epoch: 205 pg[3.cs0( v 205'351 (0'0,205'351] local-lis/les=56/57 n=7 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [5,3,6]p5(0) r=0 lpr=56 luod=204'349 crt=201'348 lcod 201'348 mlcod 201'348 active+clean+snaptrim trimq=[74~1] ps=[8a~1,8c~1]] removing snap head 2021-07-31T01:48:25.520 INFO:tasks.rados.rados.0.smithi045.stdout:839: done (1 left) 2021-07-31T01:48:25.520 INFO:tasks.rados.rados.0.smithi045.stdout:840: done (0 left) 2021-07-31T01:48:25.520 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 37 current snap is 75 2021-07-31T01:48:25.520 INFO:tasks.rados.rados.0.smithi045.stdout:841: seq_num 255 ranges {0=679936,679936=548864,1228800=598016,1826816=57344} 2021-07-31T01:48:25.523 INFO:tasks.rados.rados.0.smithi045.stdout:841: writing smithi04569269-37 from 0 to 679936 tid 1 2021-07-31T01:48:25.526 INFO:tasks.rados.rados.0.smithi045.stdout:841: writing smithi04569269-37 from 679936 to 1228800 tid 2 2021-07-31T01:48:25.530 INFO:tasks.rados.rados.0.smithi045.stdout:841: writing smithi04569269-37 from 1228800 to 1826816 tid 3 2021-07-31T01:48:25.530 INFO:tasks.rados.rados.0.smithi045.stdout:841: writing smithi04569269-37 from 1826816 to 1884160 tid 4 2021-07-31T01:48:25.531 INFO:tasks.rados.rados.0.smithi045.stdout:842: snap_create 2021-07-31T01:48:25.531 INFO:tasks.rados.rados.0.smithi045.stdout:841: finishing write tid 1 to smithi04569269-37 2021-07-31T01:48:25.531 INFO:tasks.rados.rados.0.smithi045.stdout:841: finishing write tid 2 to smithi04569269-37 2021-07-31T01:48:25.534 INFO:tasks.rados.rados.0.smithi045.stdout:841: finishing write tid 3 to smithi04569269-37 2021-07-31T01:48:25.534 INFO:tasks.rados.rados.0.smithi045.stdout:841: finishing write tid 4 to smithi04569269-37 2021-07-31T01:48:25.534 INFO:tasks.rados.rados.0.smithi045.stdout:841: finishing write tid 5 to smithi04569269-37 2021-07-31T01:48:25.535 INFO:tasks.rados.rados.0.smithi045.stdout:841: finishing write tid 6 to smithi04569269-37 2021-07-31T01:48:25.535 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 37 v 244 (ObjNum 255 snap 75 seq_num 255) dirty exists 2021-07-31T01:48:25.535 INFO:tasks.rados.rados.0.smithi045.stdout:841: left oid 37 (ObjNum 255 snap 75 seq_num 255) 2021-07-31T01:48:25.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:25 smithi045 conmon[39718]: cluster 2021-07-31T01:48:24.514222+0000 mon.a ( 2021-07-31T01:48:25.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:25 smithi045 conmon[39718]: mon.0) 842 : cluster [DBG] osdmap e205: 8 total, 8 up, 8 in 2021-07-31T01:48:25.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:25 smithi045 conmon[33655]: cluster 2021-07-31T01:48:24.514222+0000 mon.a (mon.0) 842 : cluster 2021-07-31T01:48:25.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:25 smithi045 conmon[33655]: [DBG] osdmap e205: 8 total, 8 up, 8 in 2021-07-31T01:48:26.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:25 smithi154 conmon[34553]: cluster 2021-07-31T01:48:24.514222+0000 mon.a (mon.0) 842 : cluster [DBG] osdmap e205: 8 total, 8 up, 8 in 2021-07-31T01:48:26.525 INFO:tasks.rados.rados.0.smithi045.stdout:841: done (1 left) 2021-07-31T01:48:26.526 INFO:tasks.rados.rados.0.smithi045.stdout:842: done (0 left) 2021-07-31T01:48:26.526 INFO:tasks.rados.rados.0.smithi045.stdout:843: read oid 11 snap 71 2021-07-31T01:48:26.527 INFO:tasks.rados.rados.0.smithi045.stdout:843: expect (ObjNum 252 snap 70 seq_num 252) 2021-07-31T01:48:26.527 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 21 current snap is 76 2021-07-31T01:48:26.527 INFO:tasks.rados.rados.0.smithi045.stdout:844: seq_num 256 ranges {0=696320,696320=434176,1130496=770048} 2021-07-31T01:48:26.529 INFO:tasks.rados.rados.0.smithi045.stdout:844: writing smithi04569269-21 from 0 to 696320 tid 1 2021-07-31T01:48:26.532 INFO:tasks.rados.rados.0.smithi045.stdout:844: writing smithi04569269-21 from 696320 to 1130496 tid 2 2021-07-31T01:48:26.536 INFO:tasks.rados.rados.0.smithi045.stdout:844: writing smithi04569269-21 from 1130496 to 1900544 tid 3 2021-07-31T01:48:26.537 INFO:tasks.rados.rados.0.smithi045.stdout:845: delete oid 47 current snap is 76 2021-07-31T01:48:26.537 INFO:tasks.rados.rados.0.smithi045.stdout:844: finishing write tid 1 to smithi04569269-21 2021-07-31T01:48:26.538 INFO:tasks.rados.rados.0.smithi045.stdout:844: finishing write tid 2 to smithi04569269-21 2021-07-31T01:48:26.541 INFO:tasks.rados.rados.0.smithi045.stdout:844: finishing write tid 3 to smithi04569269-21 2021-07-31T01:48:26.542 INFO:tasks.rados.rados.0.smithi045.stdout:844: finishing write tid 4 to smithi04569269-21 2021-07-31T01:48:26.542 INFO:tasks.rados.rados.0.smithi045.stdout:845: done (2 left) 2021-07-31T01:48:26.542 INFO:tasks.rados.rados.0.smithi045.stdout:846: snap_create 2021-07-31T01:48:26.543 INFO:tasks.rados.rados.0.smithi045.stdout:844: finishing write tid 5 to smithi04569269-21 2021-07-31T01:48:26.543 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 21 v 573 (ObjNum 256 snap 76 seq_num 256) dirty exists 2021-07-31T01:48:26.544 INFO:tasks.rados.rados.0.smithi045.stdout:844: left oid 21 (ObjNum 256 snap 76 seq_num 256) 2021-07-31T01:48:26.554 INFO:tasks.rados.rados.0.smithi045.stdout:843: expect (ObjNum 252 snap 70 seq_num 252) 2021-07-31T01:48:26.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:26 smithi045 conmon[33655]: cluster 2021-07-31T01:48:25.519081+0000 mon.a (mon.0) 843 : cluster [DBG] osdmap e206: 8 total, 8 up, 8 in 2021-07-31T01:48:26.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:26 smithi045 conmon[33655]: 2021-07-31T01:48:26.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:26 smithi045 conmon[33655]: cluster 2021-07-31T01:48:25.807518+0000 mgr.x (mgr.24542) 49 : cluster [DBG] pgmap v89: 25 pgs: 25 active+clean; 124 MiB data, 547 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 456 KiB/s wr, 15 op/s 2021-07-31T01:48:26.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:26 smithi045 conmon[39718]: cluster 2021-07-31T01:48:25.519081+0000 mon.a (mon 2021-07-31T01:48:26.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:26 smithi045 conmon[39718]: .0) 843 : cluster [DBG] osdmap e206: 8 total, 8 up, 8 in 2021-07-31T01:48:26.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:26 smithi045 conmon[39718]: cluster 2021-07-31T01 2021-07-31T01:48:26.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:26 smithi045 conmon[39718]: :48:25.807518+0000 mgr.x (mgr.24542) 49 : cluster [DBG] pgmap v89: 25 pgs: 25 active+clean; 124 MiB data, 547 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 456 KiB/s wr, 15 op/s 2021-07-31T01:48:26.980 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T01:48:27.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:26 smithi154 conmon[34553]: cluster 2021-07-31T01:48:25.519081+0000 mon.a (mon.0) 843 : cluster [DBG] osdmap e206: 8 total, 8 up, 8 in 2021-07-31T01:48:27.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:26 smithi154 conmon[34553]: cluster 2021-07-31T01:48:25.807518+0000 mgr.x (mgr.24542) 49 : cluster [DBG] pgmap v89: 25 pgs: 25 active+clean; 124 MiB data, 547 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 456 KiB/s wr, 15 op/s 2021-07-31T01:48:27.256 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T01:48:27.257 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 3m ago 4m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T01:48:27.257 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 60s ago 3m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T01:48:27.257 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 60s ago 7m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T01:48:27.258 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 3m ago 9m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T01:48:27.258 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 3m ago 9m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T01:48:27.258 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 60s ago 7m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T01:48:27.258 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 3m ago 7m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T01:48:27.259 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 3m ago 4m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T01:48:27.259 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 60s ago 4m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T01:48:27.259 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 3m ago 7m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T01:48:27.259 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 3m ago 6m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T01:48:27.259 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 3m ago 6m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T01:48:27.260 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 3m ago 6m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T01:48:27.260 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 60s ago 5m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T01:48:27.260 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 60s ago 5m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T01:48:27.260 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 60s ago 5m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T01:48:27.261 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 60s ago 4m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T01:48:27.261 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 60s ago 4m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T01:48:27.526 INFO:tasks.rados.rados.0.smithi045.stdout:843: done (2 left) 2021-07-31T01:48:27.526 INFO:tasks.rados.rados.0.smithi045.stdout:844: done (1 left) 2021-07-31T01:48:27.526 INFO:tasks.rados.rados.0.smithi045.stdout:846: done (0 left) 2021-07-31T01:48:27.526 INFO:tasks.rados.rados.0.smithi045.stdout:847: setattr oid 39 current snap is 77 2021-07-31T01:48:27.528 INFO:tasks.rados.rados.0.smithi045.stdout:848: read oid 37 snap -1 2021-07-31T01:48:27.529 INFO:tasks.rados.rados.0.smithi045.stdout:848: expect (ObjNum 255 snap 75 seq_num 255) 2021-07-31T01:48:27.529 INFO:tasks.rados.rados.0.smithi045.stdout:849: read oid 25 snap -1 2021-07-31T01:48:27.529 INFO:tasks.rados.rados.0.smithi045.stdout:849: expect (ObjNum 201 snap 52 seq_num 201) 2021-07-31T01:48:27.529 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 47 current snap is 77 2021-07-31T01:48:27.530 INFO:tasks.rados.rados.0.smithi045.stdout:850: seq_num 258 ranges {0=729088,729088=606208,1335296=598016} 2021-07-31T01:48:27.532 INFO:tasks.rados.rados.0.smithi045.stdout:850: writing smithi04569269-47 from 0 to 729088 tid 1 2021-07-31T01:48:27.537 INFO:tasks.rados.rados.0.smithi045.stdout:850: writing smithi04569269-47 from 729088 to 1335296 tid 2 2021-07-31T01:48:27.539 INFO:tasks.rados.rados.0.smithi045.stdout:850: writing smithi04569269-47 from 1335296 to 1933312 tid 3 2021-07-31T01:48:27.540 INFO:tasks.rados.rados.0.smithi045.stdout:851: copy_from oid 42 from oid 26 current snap is 77 2021-07-31T01:48:27.540 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 39 v 515 (ObjNum 197 snap 52 seq_num 197) dirty exists 2021-07-31T01:48:27.540 INFO:tasks.rados.rados.0.smithi045.stdout:848: expect (ObjNum 255 snap 75 seq_num 255) 2021-07-31T01:48:27.589 INFO:tasks.rados.rados.0.smithi045.stdout:849: expect (ObjNum 201 snap 52 seq_num 201) 2021-07-31T01:48:27.602 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T01:48:27.602 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T01:48:27.602 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T01:48:27.603 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T01:48:27.603 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T01:48:27.603 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T01:48:27.603 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T01:48:27.604 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T01:48:27.604 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T01:48:27.604 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T01:48:27.604 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T01:48:27.604 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T01:48:27.605 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T01:48:27.605 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T01:48:27.605 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T01:48:27.605 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T01:48:27.605 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T01:48:27.644 INFO:tasks.rados.rados.0.smithi045.stdout:850: finishing write tid 1 to smithi04569269-47 2021-07-31T01:48:27.644 INFO:tasks.rados.rados.0.smithi045.stdout:847: done (4 left) 2021-07-31T01:48:27.645 INFO:tasks.rados.rados.0.smithi045.stdout:848: done (3 left) 2021-07-31T01:48:27.645 INFO:tasks.rados.rados.0.smithi045.stdout:849: done (2 left) 2021-07-31T01:48:27.645 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 50 current snap is 77 2021-07-31T01:48:27.645 INFO:tasks.rados.rados.0.smithi045.stdout:850: finishing write tid 2 to smithi04569269-47 2021-07-31T01:48:27.645 INFO:tasks.rados.rados.0.smithi045.stdout:850: finishing write tid 3 to smithi04569269-47 2021-07-31T01:48:27.646 INFO:tasks.rados.rados.0.smithi045.stdout:850: finishing write tid 4 to smithi04569269-47 2021-07-31T01:48:27.646 INFO:tasks.rados.rados.0.smithi045.stdout:850: finishing write tid 5 to smithi04569269-47 2021-07-31T01:48:27.646 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 47 v 158 (ObjNum 258 snap 77 seq_num 258) dirty exists 2021-07-31T01:48:27.646 INFO:tasks.rados.rados.0.smithi045.stdout:850: left oid 47 (ObjNum 258 snap 77 seq_num 258) 2021-07-31T01:48:27.647 INFO:tasks.rados.rados.0.smithi045.stdout:851: finishing copy_from to smithi04569269-42 2021-07-31T01:48:27.647 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 42 v 196 (ObjNum 246 snap 68 seq_num 246) dirty exists 2021-07-31T01:48:27.647 INFO:tasks.rados.rados.0.smithi045.stdout:851: finishing copy_from racing read to smithi04569269-42 2021-07-31T01:48:27.647 INFO:tasks.rados.rados.0.smithi045.stdout:852: seq_num 259 ranges {0=745472,745472=540672,1286144=581632,1867776=81920} 2021-07-31T01:48:27.648 INFO:tasks.rados.rados.0.smithi045.stdout:852: writing smithi04569269-50 from 0 to 745472 tid 1 2021-07-31T01:48:27.650 INFO:tasks.rados.rados.0.smithi045.stdout:852: writing smithi04569269-50 from 745472 to 1286144 tid 2 2021-07-31T01:48:27.653 INFO:tasks.rados.rados.0.smithi045.stdout:852: writing smithi04569269-50 from 1286144 to 1867776 tid 3 2021-07-31T01:48:27.654 INFO:tasks.rados.rados.0.smithi045.stdout:852: writing smithi04569269-50 from 1867776 to 1949696 tid 4 2021-07-31T01:48:27.655 INFO:tasks.rados.rados.0.smithi045.stdout:850: done (2 left) 2021-07-31T01:48:27.655 INFO:tasks.rados.rados.0.smithi045.stdout:851: done (1 left) 2021-07-31T01:48:27.655 INFO:tasks.rados.rados.0.smithi045.stdout:853: delete oid 33 current snap is 77 2021-07-31T01:48:27.655 INFO:tasks.rados.rados.0.smithi045.stdout:852: finishing write tid 1 to smithi04569269-50 2021-07-31T01:48:27.655 INFO:tasks.rados.rados.0.smithi045.stdout:852: finishing write tid 2 to smithi04569269-50 2021-07-31T01:48:27.656 INFO:tasks.rados.rados.0.smithi045.stdout:853: done (1 left) 2021-07-31T01:48:27.656 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 47 current snap is 77 2021-07-31T01:48:27.657 INFO:tasks.rados.rados.0.smithi045.stdout:854: seq_num 261 ranges {1933312=778240,2711552=704512,3416064=499712} 2021-07-31T01:48:27.670 INFO:tasks.rados.rados.0.smithi045.stdout:854: writing smithi04569269-47 from 1933312 to 2711552 tid 1 2021-07-31T01:48:27.674 INFO:tasks.rados.rados.0.smithi045.stdout:854: writing smithi04569269-47 from 2711552 to 3416064 tid 2 2021-07-31T01:48:27.677 INFO:tasks.rados.rados.0.smithi045.stdout:854: writing smithi04569269-47 from 3416064 to 3915776 tid 3 2021-07-31T01:48:27.677 INFO:tasks.rados.rados.0.smithi045.stdout:855: snap_create 2021-07-31T01:48:27.678 INFO:tasks.rados.rados.0.smithi045.stdout:852: finishing write tid 3 to smithi04569269-50 2021-07-31T01:48:27.678 INFO:tasks.rados.rados.0.smithi045.stdout:852: finishing write tid 4 to smithi04569269-50 2021-07-31T01:48:27.678 INFO:tasks.rados.rados.0.smithi045.stdout:852: finishing write tid 5 to smithi04569269-50 2021-07-31T01:48:27.678 INFO:tasks.rados.rados.0.smithi045.stdout:852: finishing write tid 6 to smithi04569269-50 2021-07-31T01:48:27.679 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 50 v 368 (ObjNum 259 snap 77 seq_num 259) dirty exists 2021-07-31T01:48:27.679 INFO:tasks.rados.rados.0.smithi045.stdout:852: left oid 50 (ObjNum 259 snap 77 seq_num 259) 2021-07-31T01:48:27.679 INFO:tasks.rados.rados.0.smithi045.stdout:854: finishing write tid 1 to smithi04569269-47 2021-07-31T01:48:27.679 INFO:tasks.rados.rados.0.smithi045.stdout:854: finishing write tid 2 to smithi04569269-47 2021-07-31T01:48:27.680 INFO:tasks.rados.rados.0.smithi045.stdout:854: finishing write tid 3 to smithi04569269-47 2021-07-31T01:48:27.680 INFO:tasks.rados.rados.0.smithi045.stdout:854: finishing write tid 4 to smithi04569269-47 2021-07-31T01:48:27.681 INFO:tasks.rados.rados.0.smithi045.stdout:854: finishing write tid 5 to smithi04569269-47 2021-07-31T01:48:27.681 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 47 v 162 (ObjNum 261 snap 77 seq_num 261) dirty exists 2021-07-31T01:48:27.681 INFO:tasks.rados.rados.0.smithi045.stdout:854: left oid 47 (ObjNum 261 snap 77 seq_num 261) 2021-07-31T01:48:28.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:27 smithi154 conmon[34553]: cluster 2021-07-31T01:48:26.524439+0000 mon.a (mon.0) 844 : cluster 2021-07-31T01:48:28.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:27 smithi154 conmon[34553]: [DBG] osdmap e207: 8 total, 8 up, 8 in 2021-07-31T01:48:28.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:27 smithi045 conmon[33655]: cluster 2021-07-31T01:48:26 2021-07-31T01:48:28.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:27 smithi045 conmon[33655]: .524439+0000 mon.a (mon.0) 844 : cluster [DBG] osdmap e207: 8 total, 8 up, 8 in 2021-07-31T01:48:28.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:27 smithi045 conmon[39718]: cluster 2021-07-31T01 2021-07-31T01:48:28.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:27 smithi045 conmon[39718]: :48:26.524439+0000 mon.a (mon.0) 844 : cluster [DBG] osdmap e207: 8 total, 8 up, 8 in 2021-07-31T01:48:28.536 INFO:tasks.rados.rados.0.smithi045.stdout:852: done (2 left) 2021-07-31T01:48:28.536 INFO:tasks.rados.rados.0.smithi045.stdout:854: done (1 left) 2021-07-31T01:48:28.537 INFO:tasks.rados.rados.0.smithi045.stdout:855: done (0 left) 2021-07-31T01:48:28.537 INFO:tasks.rados.rados.0.smithi045.stdout:856: snap_remove snap 74 2021-07-31T01:48:28.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[34553]: audit 2021-07-31T 2021-07-31T01:48:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[34553]: 01:48:26.967772+0000 mgr.x (mgr.24542) 50 : audit [DBG] from='client.14748 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:48:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[34553]: audit 2021-07-31T01:48:27.248340+0000 mgr.x (mgr.24542) 51 : audit [DBG] 2021-07-31T01:48:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[34553]: from='client.24661 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:48:28.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[34553]: cluster 2021-07-31T01:48:27.525072+0000 2021-07-31T01:48:28.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[34553]: mon.a (mon.0) 845 : cluster [DBG] osdmap e208: 8 total, 8 up, 8 in 2021-07-31T01:48:28.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[34553]: audit 2021-07-31T01: 2021-07-31T01:48:28.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[34553]: 48:27.602068+0000 mon.a (mon.0) 846 : audit [DBG] 2021-07-31T01:48:28.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[34553]: from='client.? 172.21.15.45:0/1155433845' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T01:48:28.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[34553]: cluster 2021-07-31T01:48:27 2021-07-31T01:48:28.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[34553]: .807861+0000 mgr.x (mgr.24542) 52 : cluster 2021-07-31T01:48:28.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[34553]: [DBG] pgmap v92: 25 pgs: 25 active+clean; 124 MiB data, 547 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 456 KiB/s wr, 15 op/s 2021-07-31T01:48:28.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[34553]: audit 2021-07-31T01 2021-07-31T01:48:28.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[34553]: :48:27.947037+0000 mon.b (mon.2) 2021-07-31T01:48:28.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[34553]: 67 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:48:28.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[34553]: audit 2021-07-31 2021-07-31T01:48:28.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[34553]: T01:48:27.947536+0000 mon.a (mon.0 2021-07-31T01:48:28.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[34553]: ) 847 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:48:28.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[34553]: audit 2021-07 2021-07-31T01:48:28.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[34553]: -31T01:48:27.955863+0000 mon.b ( 2021-07-31T01:48:28.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[34553]: mon.2) 68 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T01:48:28.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[34553]: audit 2021- 2021-07-31T01:48:28.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[34553]: 07-31T01:48:27.956527+0000 mon.a ( 2021-07-31T01:48:28.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[34553]: mon.0) 848 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T01:48:28.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[34553]: audit 2021-07-31T01:48:28.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[34553]: 2021-07-31T01: 2021-07-31T01:48:28.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[34553]: 48:28.394283+0000 mon.b (mon.2) 69 : audit [DBG] 2021-07-31T01:48:28.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[34553]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:48:28.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[34553]: audit 2021-07-31T01:48:28.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[34553]: 2021-07-31T01:48:28.395675+0000 mon.b (mon. 2021-07-31T01:48:28.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[34553]: 2) 70 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:48:28.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[34553]: audit 2021-07 2021-07-31T01:48:28.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[34553]: -31T01:48:28.396655+0000 mon.b (mon.2) 71 : audit 2021-07-31T01:48:28.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[34553]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:48:28.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: debug 2021-07-31T01:48:28.440+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T01:48:28.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:48:28.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:48:28.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: yield 2021-07-31T01:48:28.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:48:28.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:48:28.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:48:28.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T01:48:28.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:48:28.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T01:48:28.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:48:28.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: await self._waiter 2021-07-31T01:48:28.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:48:28.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: task.result() 2021-07-31T01:48:28.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T01:48:28.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: debug 2021-07-31T01:48:28.442+0000 7fcb93160700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:48:28.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:48:28.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:48:28.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: yield 2021-07-31T01:48:28.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:48:28.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:48:28.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:48:28.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T01:48:28.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:48:28.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T01:48:28.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:48:28.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: await self._waiter 2021-07-31T01:48:28.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:48:28.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: task.result() 2021-07-31T01:48:28.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T01:48:28.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: 2021-07-31T01:48:28.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T01:48:28.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: 2021-07-31T01:48:28.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:48:28.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:48:28.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: return f(*arg) 2021-07-31T01:48:28.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:48:28.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:48:28.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:48:28.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:48:28.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:48:28.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:48:28.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:48:28.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T01:48:28.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:48:28.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: raise self._exception 2021-07-31T01:48:28.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:48:28.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:48:28.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:48:28.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T01:48:28.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:48:28.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: raise 2021-07-31T01:48:28.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:48:28.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T01:48:28.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:48:28.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:48:28.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T01:48:28.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: debug 2021-07-31T01:48:28.442+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T01:48:28.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:48:28.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:48:28.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: yield 2021-07-31T01:48:28.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:48:28.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:48:28.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:48:28.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T01:48:28.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:48:28.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T01:48:28.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:48:28.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: await self._waiter 2021-07-31T01:48:28.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:48:28.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: task.result() 2021-07-31T01:48:28.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T01:48:28.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: debug 2021-07-31T01:48:28.443+0000 7fcb9295f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:48:28.786 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:48:28.786 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:48:28.786 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: yield 2021-07-31T01:48:28.786 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:48:28.787 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:48:28.787 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:48:28.787 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T01:48:28.787 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:48:28.788 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T01:48:28.788 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:48:28.788 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: await self._waiter 2021-07-31T01:48:28.788 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:48:28.788 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: task.result() 2021-07-31T01:48:28.789 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T01:48:28.789 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: 2021-07-31T01:48:28.789 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T01:48:28.789 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: 2021-07-31T01:48:28.790 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:48:28.790 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:48:28.790 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: return f(*arg) 2021-07-31T01:48:28.790 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:48:28.791 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:48:28.791 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:48:28.791 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:48:28.791 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:48:28.792 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:48:28.792 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:48:28.792 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T01:48:28.792 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:48:28.792 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: raise self._exception 2021-07-31T01:48:28.793 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:48:28.793 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:48:28.793 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:48:28.793 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T01:48:28.794 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:48:28.794 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: raise 2021-07-31T01:48:28.794 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:48:28.794 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T01:48:28.795 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:48:28.795 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:48:28.795 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:48:28 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T01:48:28.796 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:28 smithi045 conmon[33655]: audit 2021-07-31T01:48 2021-07-31T01:48:28.796 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:28 smithi045 conmon[33655]: :26.967772+0000 mgr.x (mgr.24542) 50 : audit [DBG] from='client.14748 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:48:28.796 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:28 smithi045 conmon[33655]: audit 2021-07-31T01: 2021-07-31T01:48:28.797 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:28 smithi045 conmon[33655]: 48:27.248340+0000 mgr.x (mgr.24542) 51 : audit [DBG] from='client.24661 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:48:28.798 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:28 smithi045 conmon[33655]: cluster 2021-07 2021-07-31T01:48:28.798 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:28 smithi045 conmon[33655]: -31T01:48:27.525072+0000 mon.a (mon.0) 845 : cluster [DBG] osdmap e208: 8 total, 8 up, 8 in 2021-07-31T01:48:28.798 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:28 smithi045 conmon[33655]: audit 2021-07-31T 2021-07-31T01:48:28.798 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:28 smithi045 conmon[33655]: 01:48:27.602068+0000 mon.a (mon.0) 846 2021-07-31T01:48:28.799 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:28 smithi045 conmon[33655]: : audit [DBG] from='client.? 172.21.15.45:0/1155433845' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T01:48:28.799 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:28 smithi045 conmon[33655]: cluster 2021-07-31T01:48:27. 2021-07-31T01:48:28.799 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:28 smithi045 conmon[33655]: 807861+0000 mgr.x (mgr.24542) 52 : cluster [DBG] 2021-07-31T01:48:28.799 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:28 smithi045 conmon[33655]: pgmap v92: 25 pgs: 25 active+clean; 124 MiB data, 547 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 456 KiB/s wr, 15 op/s 2021-07-31T01:48:28.800 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:28 smithi045 conmon[33655]: audit 2021-07-31T01:48:28.800 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:28 smithi045 conmon[33655]: 2021-07-31T01:48:27.947037+0000 mon.b (mon. 2021-07-31T01:48:28.800 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:28 smithi045 conmon[33655]: 2) 67 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:48:28.800 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:28 smithi045 conmon[33655]: audit 2021-07 2021-07-31T01:48:28.801 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:28 smithi045 conmon[33655]: -31T01:48:27.947536+0000 2021-07-31T01:48:28.801 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:28 smithi045 conmon[33655]: mon.a (mon.0) 847 : audit 2021-07-31T01:48:28.801 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:28 smithi045 conmon[33655]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:48:28.801 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:28 smithi045 conmon[33655]: audit 2021-07-31T01:48:28.802 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:28 smithi045 conmon[33655]: 2021-07-31T01:48 2021-07-31T01:48:28.802 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:28 smithi045 conmon[33655]: :27.955863+0000 mon.b (mon. 2021-07-31T01:48:28.802 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:28 smithi045 conmon[33655]: 2) 68 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T01:48:28.802 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:28 smithi045 conmon[33655]: audit 2021- 2021-07-31T01:48:28.803 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:28 smithi045 conmon[33655]: 07-31T01:48:27.956527+0000 mon.a (mon.0 2021-07-31T01:48:28.803 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:28 smithi045 conmon[33655]: ) 848 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T01:48:28.803 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:28 smithi045 conmon[33655]: audit 2021-07-31T01:48:28.394283+0000 mon.b ( 2021-07-31T01:48:28.803 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:28 smithi045 conmon[33655]: mon.2) 69 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:48:28.803 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:28 smithi045 conmon[33655]: audit 2021-07-31T 2021-07-31T01:48:28.804 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:28 smithi045 conmon[33655]: 01:48:28.395675+0000 mon.b (mon.2) 2021-07-31T01:48:28.804 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:28 smithi045 conmon[33655]: 70 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:48:28.804 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:28 smithi045 conmon[33655]: audit 2021-07-31T01:48:28.396655+0000 2021-07-31T01:48:28.804 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:28 smithi045 conmon[33655]: mon.b (mon.2) 71 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:48:28.805 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:28 smithi045 conmon[39718]: audit 2021-07-31T01:48:26.967772+0000 mgr.x (mgr. 2021-07-31T01:48:28.805 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:28 smithi045 conmon[39718]: 24542) 50 : audit [DBG] from='client.14748 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:48:28.805 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:28 smithi045 conmon[39718]: audit 2021-07-31T01:48:27.248340+0000 mgr.x (mgr.24542 2021-07-31T01:48:28.805 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:28 smithi045 conmon[39718]: ) 51 : audit [DBG] from='client.24661 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:48:28.806 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:28 smithi045 conmon[39718]: cluster 2021-07-31 2021-07-31T01:48:28.806 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:28 smithi045 conmon[39718]: T01:48:27.525072+0000 mon.a (mon.0) 2021-07-31T01:48:28.806 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:28 smithi045 conmon[39718]: 845 : cluster [DBG] osdmap e208: 8 total, 8 up, 8 in 2021-07-31T01:48:28.806 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:28 smithi045 conmon[39718]: audit 2021- 2021-07-31T01:48:28.806 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:28 smithi045 conmon[39718]: 07-31T01:48:27.602068+0000 2021-07-31T01:48:28.807 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:28 smithi045 conmon[39718]: mon.a (mon.0) 846 : audit [DBG] 2021-07-31T01:48:28.807 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:28 smithi045 conmon[39718]: from='client.? 172.21.15.45:0/1155433845' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T01:48:28.807 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:28 smithi045 conmon[39718]: cluster 2021-07-31T01: 2021-07-31T01:48:28.807 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:28 smithi045 conmon[39718]: 48:27.807861+0000 mgr.x (mgr. 2021-07-31T01:48:28.807 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:28 smithi045 conmon[39718]: 24542) 52 : cluster [DBG] pgmap v92: 25 pgs: 25 active+clean; 124 MiB data, 547 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 456 KiB/s wr, 15 op/s 2021-07-31T01:48:28.808 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:28 smithi045 conmon[39718]: audit 2021- 2021-07-31T01:48:28.808 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:28 smithi045 conmon[39718]: 07-31T01:48:27.947037 2021-07-31T01:48:28.808 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:28 smithi045 conmon[39718]: +0000 mon.b (mon.2) 67 : audit 2021-07-31T01:48:28.808 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:28 smithi045 conmon[39718]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:48:28.809 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:28 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T01:48:28.809 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:28 smithi045 conmon[39718]: T01:48:27.947536+0000 mon.a 2021-07-31T01:48:28.809 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:28 smithi045 conmon[39718]: (mon.0) 847 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:48:28.809 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:28 smithi045 conmon[39718]: 2021-07-31T01:48:28.810 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:28 smithi045 conmon[39718]: audit 2021-07-31T01:48:28.810 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:28 smithi045 conmon[39718]: 2021-07-31T01:48:27.955863+0000 2021-07-31T01:48:28.810 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:28 smithi045 conmon[39718]: mon.b (mon.2) 68 : audit 2021-07-31T01:48:28.810 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:28 smithi045 conmon[39718]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T01:48:28.811 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:28 smithi045 conmon[39718]: audit 2021-07 2021-07-31T01:48:28.811 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:28 smithi045 conmon[39718]: -31T01:48:27 2021-07-31T01:48:28.811 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:28 smithi045 conmon[39718]: .956527+0000 mon.a (mon.0 2021-07-31T01:48:28.811 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:28 smithi045 conmon[39718]: ) 848 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T01:48:28.812 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:28 smithi045 conmon[39718]: 2021-07-31T01:48:28.812 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:28 smithi045 conmon[39718]: audit 2021-07-31T01:48:28 2021-07-31T01:48:28.812 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:28 smithi045 conmon[39718]: .394283+0000 mon.b (mon.2) 69 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:48:28.812 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:28 smithi045 conmon[39718]: audit 2021-07-31T01:48:28.812 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:28 smithi045 conmon[39718]: 2021-07-31T01:48:28. 2021-07-31T01:48:28.813 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:28 smithi045 conmon[39718]: 395675+0000 mon.b (mon.2) 70 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:48:28.813 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:28 smithi045 conmon[39718]: audit 2021-07-31T01 2021-07-31T01:48:28.813 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:28 smithi045 conmon[39718]: :48:28.396655+0000 mon.b (mon.2) 71 : audit [INF] 2021-07-31T01:48:28.813 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:28 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:48:29.538 INFO:tasks.rados.rados.0.smithi045.stdout:856: done (0 left) 2021-07-31T01:48:29.538 INFO:tasks.rados.rados.0.smithi045.stdout:857: rmattr oid 46 current snap is 78 2021-07-31T01:48:29.539 INFO:tasks.rados.rados.0.smithi045.stdout:857: done (0 left) 2021-07-31T01:48:29.539 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 21 current snap is 78 2021-07-31T01:48:29.539 INFO:tasks.rados.rados.0.smithi045.stdout:858: seq_num 262 ranges {1900544=794624,2695168=589824,3284992=614400} 2021-07-31T01:48:29.553 INFO:tasks.rados.rados.0.smithi045.stdout:858: writing smithi04569269-21 from 1900544 to 2695168 tid 1 2021-07-31T01:48:29.556 INFO:tasks.rados.rados.0.smithi045.stdout:858: writing smithi04569269-21 from 2695168 to 3284992 tid 2 2021-07-31T01:48:29.560 INFO:tasks.rados.rados.0.smithi045.stdout:858: writing smithi04569269-21 from 3284992 to 3899392 tid 3 2021-07-31T01:48:29.560 INFO:tasks.rados.rados.0.smithi045.stdout:859: delete oid 39 current snap is 78 2021-07-31T01:48:29.561 INFO:tasks.rados.rados.0.smithi045.stdout:858: finishing write tid 1 to smithi04569269-21 2021-07-31T01:48:29.561 INFO:tasks.rados.rados.0.smithi045.stdout:858: finishing write tid 2 to smithi04569269-21 2021-07-31T01:48:29.565 INFO:tasks.rados.rados.0.smithi045.stdout:858: finishing write tid 3 to smithi04569269-21 2021-07-31T01:48:29.566 INFO:tasks.rados.rados.0.smithi045.stdout:859: done (1 left) 2021-07-31T01:48:29.566 INFO:tasks.rados.rados.0.smithi045.stdout:860: read oid 49 snap -1 2021-07-31T01:48:29.567 INFO:tasks.rados.rados.0.smithi045.stdout:860: expect (ObjNum 240 snap 67 seq_num 240) 2021-07-31T01:48:29.567 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 28 current snap is 78 2021-07-31T01:48:29.567 INFO:tasks.rados.rados.0.smithi045.stdout:858: finishing write tid 4 to smithi04569269-21 2021-07-31T01:48:29.567 INFO:tasks.rados.rados.0.smithi045.stdout:861: seq_num 264 ranges {1007616=425984,1433600=761856,2195456=770048,2965504=73728} 2021-07-31T01:48:29.572 INFO:tasks.rados.rados.0.smithi045.stdout:861: writing smithi04569269-28 from 1007616 to 1433600 tid 1 2021-07-31T01:48:29.576 INFO:tasks.rados.rados.0.smithi045.stdout:861: writing smithi04569269-28 from 1433600 to 2195456 tid 2 2021-07-31T01:48:29.580 INFO:tasks.rados.rados.0.smithi045.stdout:861: writing smithi04569269-28 from 2195456 to 2965504 tid 3 2021-07-31T01:48:29.581 INFO:tasks.rados.rados.0.smithi045.stdout:861: writing smithi04569269-28 from 2965504 to 3039232 tid 4 2021-07-31T01:48:29.582 INFO:tasks.rados.rados.0.smithi045.stdout:862: read oid 39 snap -1 2021-07-31T01:48:29.582 INFO:tasks.rados.rados.0.smithi045.stdout:862: expect deleted 2021-07-31T01:48:29.582 INFO:tasks.rados.rados.0.smithi045.stdout:863: rmattr oid 12 current snap is 78 2021-07-31T01:48:29.583 INFO:tasks.rados.rados.0.smithi045.stdout:863: done (4 left) 2021-07-31T01:48:29.583 INFO:tasks.rados.rados.0.smithi045.stdout:864: delete oid 3 current snap is 78 2021-07-31T01:48:29.583 INFO:tasks.rados.rados.0.smithi045.stdout:858: finishing write tid 5 to smithi04569269-21 2021-07-31T01:48:29.584 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 21 v 578 (ObjNum 262 snap 78 seq_num 262) dirty exists 2021-07-31T01:48:29.584 INFO:tasks.rados.rados.0.smithi045.stdout:858: left oid 21 (ObjNum 262 snap 78 seq_num 262) 2021-07-31T01:48:29.584 INFO:tasks.rados.rados.0.smithi045.stdout:860: expect (ObjNum 240 snap 67 seq_num 240) 2021-07-31T01:48:29.625 INFO:tasks.rados.rados.0.smithi045.stdout:861: finishing write tid 1 to smithi04569269-28 2021-07-31T01:48:29.625 INFO:tasks.rados.rados.0.smithi045.stdout:861: finishing write tid 2 to smithi04569269-28 2021-07-31T01:48:29.626 INFO:tasks.rados.rados.0.smithi045.stdout:861: finishing write tid 3 to smithi04569269-28 2021-07-31T01:48:29.626 INFO:tasks.rados.rados.0.smithi045.stdout:861: finishing write tid 4 to smithi04569269-28 2021-07-31T01:48:29.626 INFO:tasks.rados.rados.0.smithi045.stdout:858: done (4 left) 2021-07-31T01:48:29.626 INFO:tasks.rados.rados.0.smithi045.stdout:860: done (3 left) 2021-07-31T01:48:29.627 INFO:tasks.rados.rados.0.smithi045.stdout:862: done (2 left) 2021-07-31T01:48:29.627 INFO:tasks.rados.rados.0.smithi045.stdout:864: done (1 left) 2021-07-31T01:48:29.627 INFO:tasks.rados.rados.0.smithi045.stdout:865: copy_from oid 32 from oid 19 current snap is 78 2021-07-31T01:48:29.627 INFO:tasks.rados.rados.0.smithi045.stdout:861: finishing write tid 5 to smithi04569269-28 2021-07-31T01:48:29.628 INFO:tasks.rados.rados.0.smithi045.stdout:861: finishing write tid 6 to smithi04569269-28 2021-07-31T01:48:29.628 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 28 v 376 (ObjNum 264 snap 78 seq_num 264) dirty exists 2021-07-31T01:48:29.628 INFO:tasks.rados.rados.0.smithi045.stdout:861: left oid 28 (ObjNum 264 snap 78 seq_num 264) 2021-07-31T01:48:29.628 INFO:tasks.rados.rados.0.smithi045.stdout:861: done (1 left) 2021-07-31T01:48:29.628 INFO:tasks.rados.rados.0.smithi045.stdout:866: snap_remove snap 73 2021-07-31T01:48:30.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: cephadm 2021-07-31T01:48:28.397440+0000 mgr.x (mgr. 2021-07-31T01:48:30.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: 24542) 53 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T01:48:30.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: cephadm 2021-07-31T01:48:28.398179+0000 mgr.x (mgr.24542) 54 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T01:48:30.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: cephadm 2021-07-31T01:48:28.398515+0000 mgr.x (mgr.24542) 55 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T01:48:30.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: cephadm 2021-07-31T01:48:28.441572+0000 mgr.x (mgr.24542) 56 : cephadm [ERR] 2021-07-31T01:48:30.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:48:30.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:48:30.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: yield 2021-07-31T01:48:30.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:48:30.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:48:30.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:48:30.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T01:48:30.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:48:30.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T01:48:30.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:48:30.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: await self._waiter 2021-07-31T01:48:30.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:48:30.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: task.result() 2021-07-31T01:48:30.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T01:48:30.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: cephadm 2021-07-31T01:48:28.442992+0000 mgr.x (mgr.24542) 57 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:48:30.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:48:30.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:48:30.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: yield 2021-07-31T01:48:30.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:48:30.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:48:30.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:48:30.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T01:48:30.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:48:30.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T01:48:30.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:48:30.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: await self._waiter 2021-07-31T01:48:30.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:48:30.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: task.result() 2021-07-31T01:48:30.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T01:48:30.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: 2021-07-31T01:48:30.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T01:48:30.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: 2021-07-31T01:48:30.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:48:30.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:48:30.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: return f(*arg) 2021-07-31T01:48:30.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:48:30.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:48:30.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:48:30.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:48:30.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:48:30.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:48:30.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:48:30.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T01:48:30.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:48:30.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: raise self._exception 2021-07-31T01:48:30.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:48:30.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:48:30.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:48:30.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T01:48:30.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:48:30.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: raise 2021-07-31T01:48:30.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:48:30.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T01:48:30.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:48:30.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:48:30.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T01:48:30.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: cephadm 2021-07-31T01:48:28.443850+0000 mgr.x (mgr.24542) 58 : cephadm [ERR] 2021-07-31T01:48:30.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:48:30.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:48:30.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: yield 2021-07-31T01:48:30.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:48:30.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:48:30.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:48:30.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T01:48:30.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:48:30.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T01:48:30.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:48:30.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: await self._waiter 2021-07-31T01:48:30.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:48:30.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: task.result() 2021-07-31T01:48:30.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T01:48:30.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: cephadm 2021-07-31T01:48:28.444261+0000 mgr.x (mgr.24542) 59 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:48:30.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:48:30.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:48:30.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: yield 2021-07-31T01:48:30.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:48:30.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:48:30.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:48:30.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T01:48:30.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:48:30.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T01:48:30.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:48:30.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: await self._waiter 2021-07-31T01:48:30.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:48:30.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: task.result() 2021-07-31T01:48:30.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T01:48:30.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: 2021-07-31T01:48:30.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T01:48:30.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: 2021-07-31T01:48:30.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:48:30.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:48:30.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: return f(*arg) 2021-07-31T01:48:30.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:48:30.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:48:30.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:48:30.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:48:30.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:48:30.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:48:30.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:48:30.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T01:48:30.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:48:30.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: raise self._exception 2021-07-31T01:48:30.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:48:30.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:48:30.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:48:30.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T01:48:30.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:48:30.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: raise 2021-07-31T01:48:30.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:48:30.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T01:48:30.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:48:30.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:48:30.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T01:48:30.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:29 smithi154 conmon[34553]: cluster 2021-07-31T01:48:28.535991+0000 mon.a (mon.0) 849 : cluster [DBG] osdmap e209: 8 total, 8 up, 8 in 2021-07-31T01:48:30.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: cephadm 2021-07-31T01:48:30.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: 2021-07-31T01:48:30.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: 2021 2021-07-31T01:48:30.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: -07-31T01:48:28.397440+0000 mgr.x (mgr 2021-07-31T01:48:30.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: .24542) 53 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T01:48:30.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: cephadm 2021-07-31T01:48:30.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: 2021-07-31T01: 2021-07-31T01:48:30.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: 48:28.398179+0000 mgr.x (mgr.24542) 2021-07-31T01:48:30.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: 54 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T01:48:30.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: cephadm 2021-07-31T01:48:30.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: 2021-07-31T01:48:28.398515 2021-07-31T01:48:30.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 55 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T01:48:30.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: cephadm 2021-07 2021-07-31T01:48:30.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: -31T01:48:28.441572+0000 mgr.x ( 2021-07-31T01:48:30.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: mgr.24542) 56 : cephadm [ERR] 2021-07-31T01:48:30.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:48:30.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:48:30.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: yield 2021-07-31T01:48:30.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:48:30.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:48:30.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:48:30.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T01:48:30.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:48:30.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T01:48:30.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:48:30.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: await self._waiter 2021-07-31T01:48:30.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:48:30.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: task.result() 2021-07-31T01:48:30.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T01:48:30.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: cephadm 2021-07-31T01:48:28.442992+0000 mgr.x (mgr.24542) 57 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:48:30.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:48:30.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:48:30.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: yield 2021-07-31T01:48:30.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:48:30.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:48:30.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:48:30.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T01:48:30.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:48:30.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T01:48:30.040 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:48:30.040 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: await self._waiter 2021-07-31T01:48:30.040 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:48:30.040 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: task.result() 2021-07-31T01:48:30.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: cephadm 2021-07-31T01:48:28.397440+0000 mgr.x ( 2021-07-31T01:48:30.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: mgr.24542) 53 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T01:48:30.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: cephadm 2021-07-31T01:48:28.398179+0000 2021-07-31T01:48:30.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: mgr.x (mgr.24542) 54 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T01:48:30.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: cephadm 2021 2021-07-31T01:48:30.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: -07-31T01:48:28.398515+0000 mgr.x (mgr.24542) 55 : cephadm 2021-07-31T01:48:30.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T01:48:30.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: cephadm 2021-07-31T01:48:28.441572 2021-07-31T01:48:30.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 56 : cephadm [ERR] 2021-07-31T01:48:30.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:48:30.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:48:30.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: yield 2021-07-31T01:48:30.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:48:30.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:48:30.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:48:30.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T01:48:30.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:48:30.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T01:48:30.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:48:30.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: await self._waiter 2021-07-31T01:48:30.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:48:30.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: task.result() 2021-07-31T01:48:30.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T01:48:30.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: cephadm 2021-07-31T01:48:28.442992+0000 mgr.x (mgr.24542) 57 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:48:30.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:48:30.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:48:30.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: yield 2021-07-31T01:48:30.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:48:30.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:48:30.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:48:30.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T01:48:30.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:48:30.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T01:48:30.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:48:30.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: await self._waiter 2021-07-31T01:48:30.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:48:30.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: task.result() 2021-07-31T01:48:30.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T01:48:30.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: 2021-07-31T01:48:30.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T01:48:30.050 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T01:48:30.050 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: 2021-07-31T01:48:30.050 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T01:48:30.050 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: 2021-07-31T01:48:30.051 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:48:30.051 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:48:30.051 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: return f(*arg) 2021-07-31T01:48:30.051 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:48:30.052 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:48:30.052 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:48:30.052 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:48:30.052 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:48:30.052 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:48:30.053 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:48:30.053 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T01:48:30.053 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:48:30.053 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: raise self._exception 2021-07-31T01:48:30.054 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:48:30.054 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:48:30.054 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:48:30.054 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T01:48:30.054 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:48:30.055 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: raise 2021-07-31T01:48:30.055 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:48:30.055 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T01:48:30.055 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:48:30.055 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:48:30.056 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T01:48:30.056 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: cephadm 2021-07-31T01:48:28. 2021-07-31T01:48:30.056 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: 443850+0000 mgr.x (mgr.24542) 58 : cephadm [ERR] 2021-07-31T01:48:30.056 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:48:30.056 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:48:30.057 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: yield 2021-07-31T01:48:30.057 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:48:30.057 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:48:30.057 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:48:30.058 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T01:48:30.058 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:48:30.058 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T01:48:30.058 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:48:30.058 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: await self._waiter 2021-07-31T01:48:30.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: 2021-07-31T01:48:30.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:48:30.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:48:30.060 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: return f(*arg) 2021-07-31T01:48:30.060 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:48:30.060 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:48:30.060 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:48:30.061 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:48:30.061 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:48:30.061 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:48:30.061 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:48:30.061 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T01:48:30.062 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:48:30.062 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: raise self._exception 2021-07-31T01:48:30.062 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:48:30.062 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:48:30.063 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:48:30.063 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T01:48:30.063 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:48:30.063 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: raise 2021-07-31T01:48:30.063 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:48:30.064 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T01:48:30.064 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:48:30.064 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:48:30.064 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T01:48:30.064 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: cephadm 2021-07-31T01:48:28.443850+0000 mgr.x (mgr.24542 2021-07-31T01:48:30.065 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: ) 58 : cephadm [ERR] 2021-07-31T01:48:30.065 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:48:30.065 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:48:30.065 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: yield 2021-07-31T01:48:30.065 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:48:30.066 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:48:30.066 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:48:30.066 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T01:48:30.066 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:48:30.067 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T01:48:30.067 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:48:30.067 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: await self._waiter 2021-07-31T01:48:30.067 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:48:30.067 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: task.result() 2021-07-31T01:48:30.068 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T01:48:30.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:48:30.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: task.result() 2021-07-31T01:48:30.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T01:48:30.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: cephadm 2021-07-31T01:48:28.444261+0000 mgr.x (mgr.24542) 59 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:48:30.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:48:30.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:48:30.070 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: yield 2021-07-31T01:48:30.070 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:48:30.070 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:48:30.070 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:48:30.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T01:48:30.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:48:30.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T01:48:30.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:48:30.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: await self._waiter 2021-07-31T01:48:30.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:48:30.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: task.result() 2021-07-31T01:48:30.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T01:48:30.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: 2021-07-31T01:48:30.073 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T01:48:30.073 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: 2021-07-31T01:48:30.073 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:48:30.073 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:48:30.073 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: return f(*arg) 2021-07-31T01:48:30.074 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:48:30.074 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:48:30.074 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:48:30.074 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:48:30.074 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:48:30.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:48:30.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:48:30.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T01:48:30.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:48:30.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: raise self._exception 2021-07-31T01:48:30.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:48:30.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:48:30.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:48:30.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T01:48:30.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:48:30.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: raise 2021-07-31T01:48:30.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:48:30.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T01:48:30.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:48:30.078 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:48:30.078 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T01:48:30.078 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[33655]: cluster 2021-07-31T01:48:28.535991+0000 mon.a (mon.0) 849 : cluster [DBG] osdmap e209: 8 total, 8 up, 8 in 2021-07-31T01:48:30.079 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: cephadm 2021-07-31T01:48:28.444261+0000 mgr.x (mgr.24542) 59 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:48:30.079 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:48:30.079 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:48:30.079 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: yield 2021-07-31T01:48:30.080 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:48:30.080 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:48:30.080 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:48:30.080 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T01:48:30.080 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:48:30.081 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T01:48:30.081 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:48:30.081 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: await self._waiter 2021-07-31T01:48:30.081 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:48:30.082 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: task.result() 2021-07-31T01:48:30.082 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T01:48:30.082 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: 2021-07-31T01:48:30.082 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T01:48:30.082 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: 2021-07-31T01:48:30.083 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:48:30.083 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:48:30.083 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: return f(*arg) 2021-07-31T01:48:30.083 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:48:30.083 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:48:30.084 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:48:30.084 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:48:30.084 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:48:30.084 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:48:30.085 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:48:30.085 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T01:48:30.085 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:48:30.085 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: raise self._exception 2021-07-31T01:48:30.085 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:48:30.086 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:48:30.086 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:48:30.086 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T01:48:30.086 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:48:30.087 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: raise 2021-07-31T01:48:30.087 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:48:30.087 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T01:48:30.087 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:48:30.087 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:48:30.088 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T01:48:30.088 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:29 smithi045 conmon[39718]: cluster 2021-07-31T01:48:28.535991+0000 mon.a (mon.0) 849 : cluster [DBG] osdmap e209: 8 total, 8 up, 8 in 2021-07-31T01:48:30.549 INFO:tasks.rados.rados.0.smithi045.stdout:866: done (1 left) 2021-07-31T01:48:30.549 INFO:tasks.rados.rados.0.smithi045.stdout:867: snap_create 2021-07-31T01:48:30.549 INFO:tasks.rados.rados.0.smithi045.stdout:865: finishing copy_from to smithi04569269-32 2021-07-31T01:48:30.550 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 32 v 171 (ObjNum 216 snap 56 seq_num 216) dirty exists 2021-07-31T01:48:30.550 INFO:tasks.rados.rados.0.smithi045.stdout:865: finishing copy_from racing read to smithi04569269-32 2021-07-31T01:48:31.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:30 smithi154 conmon[34553]: cluster 2021-07-31T01:48:31.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:30 smithi154 conmon[34553]: 2021-07-31T01:48:29.537636 2021-07-31T01:48:31.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:30 smithi154 conmon[34553]: +0000 mon.a (mon.0) 850 : cluster [DBG] osdmap e210: 8 total, 8 up, 8 in 2021-07-31T01:48:31.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:30 smithi154 conmon[34553]: cluster 2021- 2021-07-31T01:48:31.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:30 smithi154 conmon[34553]: 07-31T01:48:29.808443+0000 mgr.x (mgr. 2021-07-31T01:48:31.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:30 smithi154 conmon[34553]: 24542) 60 : cluster [DBG] pgmap v95: 25 pgs: 25 active+clean; 112 MiB data, 554 MiB used, 715 GiB / 715 GiB avail; 6.5 MiB/s rd, 3.7 MiB/s wr, 20 op/s 2021-07-31T01:48:31.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:30 smithi045 conmon[33655]: cluster 2021-07-31T01:48:29.537636+0000 2021-07-31T01:48:31.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:30 smithi045 conmon[33655]: mon.a (mon.0) 850 : cluster [DBG] osdmap e210: 8 total, 8 up, 8 in 2021-07-31T01:48:31.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:30 smithi045 conmon[33655]: cluster 2021-07-31T01:48:29.808443+0000 mgr.x (mgr. 2021-07-31T01:48:31.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:30 smithi045 conmon[33655]: 24542) 60 : cluster [DBG] pgmap v95: 25 pgs: 25 active+clean; 112 MiB data, 554 MiB used, 715 GiB / 715 GiB avail; 6.5 MiB/s rd, 3.7 MiB/s wr, 20 op/s 2021-07-31T01:48:31.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:30 smithi045 conmon[39718]: cluster 2021-07-31T01:48:29.537636+0000 mon.a ( 2021-07-31T01:48:31.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:30 smithi045 conmon[39718]: mon.0) 850 : cluster [DBG] osdmap e210: 8 total, 8 up, 8 in 2021-07-31T01:48:31.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:30 smithi045 conmon[39718]: cluster 2021-07-31T01:48 2021-07-31T01:48:31.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:30 smithi045 conmon[39718]: :29.808443+0000 mgr.x (mgr.24542) 60 : cluster [DBG] pgmap v95: 25 pgs: 25 active+clean; 112 MiB data, 554 MiB used, 715 GiB / 715 GiB avail; 6.5 MiB/s rd, 3.7 MiB/s wr, 20 op/s 2021-07-31T01:48:31.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:30 smithi045 conmon[39718]: 2021-07-31T01:48:31.505 INFO:tasks.rados.rados.0.smithi045.stdout:865: done (1 left) 2021-07-31T01:48:31.505 INFO:tasks.rados.rados.0.smithi045.stdout:867: done (0 left) 2021-07-31T01:48:31.505 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 21 current snap is 79 2021-07-31T01:48:31.505 INFO:tasks.rados.rados.0.smithi045.stdout:868: seq_num 266 ranges {3899392=458752,4358144=524288,4882432=729088,5611520=352256} 2021-07-31T01:48:31.528 INFO:tasks.rados.rados.0.smithi045.stdout:868: writing smithi04569269-21 from 3899392 to 4358144 tid 1 2021-07-31T01:48:31.531 INFO:tasks.rados.rados.0.smithi045.stdout:868: writing smithi04569269-21 from 4358144 to 4882432 tid 2 2021-07-31T01:48:31.535 INFO:tasks.rados.rados.0.smithi045.stdout:868: writing smithi04569269-21 from 4882432 to 5611520 tid 3 2021-07-31T01:48:31.538 INFO:tasks.rados.rados.0.smithi045.stdout:868: writing smithi04569269-21 from 5611520 to 5963776 tid 4 2021-07-31T01:48:31.538 INFO:tasks.rados.rados.0.smithi045.stdout:869: rollback oid 42 current snap is 79 2021-07-31T01:48:31.539 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 42 to 76 2021-07-31T01:48:31.539 INFO:tasks.rados.rados.0.smithi045.stdout:868: finishing write tid 1 to smithi04569269-21 2021-07-31T01:48:31.539 INFO:tasks.rados.rados.0.smithi045.stdout:868: finishing write tid 2 to smithi04569269-21 2021-07-31T01:48:31.539 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 47 current snap is 79 2021-07-31T01:48:31.540 INFO:tasks.rados.rados.0.smithi045.stdout:870: seq_num 267 ranges {3915776=475136,4390912=458752,4849664=540672,5390336=606208} 2021-07-31T01:48:31.562 INFO:tasks.rados.rados.0.smithi045.stdout:870: writing smithi04569269-47 from 3915776 to 4390912 tid 1 2021-07-31T01:48:31.565 INFO:tasks.rados.rados.0.smithi045.stdout:870: writing smithi04569269-47 from 4390912 to 4849664 tid 2 2021-07-31T01:48:31.568 INFO:tasks.rados.rados.0.smithi045.stdout:870: writing smithi04569269-47 from 4849664 to 5390336 tid 3 2021-07-31T01:48:31.572 INFO:tasks.rados.rados.0.smithi045.stdout:870: writing smithi04569269-47 from 5390336 to 5996544 tid 4 2021-07-31T01:48:31.573 INFO:tasks.rados.rados.0.smithi045.stdout:871: copy_from oid 13 from oid 7 current snap is 79 2021-07-31T01:48:31.573 INFO:tasks.rados.rados.0.smithi045.stdout:868: finishing write tid 3 to smithi04569269-21 2021-07-31T01:48:31.573 INFO:tasks.rados.rados.0.smithi045.stdout:868: finishing write tid 4 to smithi04569269-21 2021-07-31T01:48:31.573 INFO:tasks.rados.rados.0.smithi045.stdout:868: finishing write tid 5 to smithi04569269-21 2021-07-31T01:48:31.574 INFO:tasks.rados.rados.0.smithi045.stdout:868: finishing write tid 6 to smithi04569269-21 2021-07-31T01:48:31.574 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 21 v 585 (ObjNum 266 snap 79 seq_num 266) dirty exists 2021-07-31T01:48:31.574 INFO:tasks.rados.rados.0.smithi045.stdout:868: left oid 21 (ObjNum 266 snap 79 seq_num 266) 2021-07-31T01:48:31.574 INFO:tasks.rados.rados.0.smithi045.stdout:869: finishing rollback tid 0 to smithi04569269-42 2021-07-31T01:48:31.575 INFO:tasks.rados.rados.0.smithi045.stdout:869: finishing rollback tid 1 to smithi04569269-42 2021-07-31T01:48:31.575 INFO:tasks.rados.rados.0.smithi045.stdout:869: finishing rollback tid 2 to smithi04569269-42 2021-07-31T01:48:31.575 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 42 v 384 (ObjNum 176 snap 46 seq_num 176) dirty exists 2021-07-31T01:48:31.575 INFO:tasks.rados.rados.0.smithi045.stdout:870: finishing write tid 1 to smithi04569269-47 2021-07-31T01:48:31.575 INFO:tasks.rados.rados.0.smithi045.stdout:870: finishing write tid 2 to smithi04569269-47 2021-07-31T01:48:31.576 INFO:tasks.rados.rados.0.smithi045.stdout:870: finishing write tid 3 to smithi04569269-47 2021-07-31T01:48:31.576 INFO:tasks.rados.rados.0.smithi045.stdout:868: done (3 left) 2021-07-31T01:48:31.576 INFO:tasks.rados.rados.0.smithi045.stdout:869: done (2 left) 2021-07-31T01:48:31.576 INFO:tasks.rados.rados.0.smithi045.stdout:872: snap_create 2021-07-31T01:48:31.577 INFO:tasks.rados.rados.0.smithi045.stdout:871: finishing copy_from to smithi04569269-13 2021-07-31T01:48:31.577 INFO:tasks.rados.rados.0.smithi045.stdout:871: got expected ENOENT (src dne) 2021-07-31T01:48:31.577 INFO:tasks.rados.rados.0.smithi045.stdout:871: finishing copy_from racing read to smithi04569269-13 2021-07-31T01:48:31.578 INFO:tasks.rados.rados.0.smithi045.stdout:870: finishing write tid 4 to smithi04569269-47 2021-07-31T01:48:31.578 INFO:tasks.rados.rados.0.smithi045.stdout:870: finishing write tid 5 to smithi04569269-47 2021-07-31T01:48:31.578 INFO:tasks.rados.rados.0.smithi045.stdout:870: finishing write tid 6 to smithi04569269-47 2021-07-31T01:48:31.579 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 47 v 167 (ObjNum 267 snap 79 seq_num 267) dirty exists 2021-07-31T01:48:31.579 INFO:tasks.rados.rados.0.smithi045.stdout:870: left oid 47 (ObjNum 267 snap 79 seq_num 267) 2021-07-31T01:48:32.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:31 smithi154 conmon[34553]: cluster 2021-07-31T01:48:30.548582+0000 mon.a (mon.0) 851 : cluster [DBG] osdmap e211: 8 total, 8 up, 8 in 2021-07-31T01:48:32.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:31 smithi154 conmon[34553]: cluster 2021-07-31T01:48:31.504031+0000 mon.a (mon.0) 852 : cluster [DBG] osdmap e212: 8 total, 8 up, 8 in 2021-07-31T01:48:32.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:31 smithi045 conmon[39718]: cluster 2021-07-31T01:48:30.548582+0000 mon.a (mon.0) 851 2021-07-31T01:48:32.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:31 smithi045 conmon[39718]: : cluster [DBG] osdmap e211: 8 total, 8 up, 8 in 2021-07-31T01:48:32.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:31 smithi045 conmon[39718]: cluster 2021-07-31T01:48:31.504031+0000 mon.a (mon.0) 852 : cluster [DBG] osdmap e212: 8 total, 8 up, 8 in 2021-07-31T01:48:32.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:31 smithi045 conmon[33655]: cluster 2021-07-31T01:48:30.548582 2021-07-31T01:48:32.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:31 smithi045 conmon[33655]: +0000 mon.a (mon.0) 851 : cluster [DBG] osdmap e211: 8 total, 8 up, 8 in 2021-07-31T01:48:32.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:31 smithi045 conmon[33655]: cluster 2021-07-31T01:48:31.504031+0000 mon.a (mon.0) 852 : cluster [DBG] osdmap e212: 8 total, 8 up, 8 in 2021-07-31T01:48:32.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:31 smithi045 conmon[33655]: 2021-07-31T01:48:32.555 INFO:tasks.rados.rados.0.smithi045.stdout:870: done (2 left) 2021-07-31T01:48:32.556 INFO:tasks.rados.rados.0.smithi045.stdout:871: done (1 left) 2021-07-31T01:48:32.556 INFO:tasks.rados.rados.0.smithi045.stdout:872: done (0 left) 2021-07-31T01:48:32.556 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 15 current snap is 80 2021-07-31T01:48:32.556 INFO:tasks.rados.rados.0.smithi045.stdout:873: seq_num 268 ranges {3121152=491520,3612672=745472,4358144=745472,5103616=114688} 2021-07-31T01:48:32.575 INFO:tasks.rados.rados.0.smithi045.stdout:873: writing smithi04569269-15 from 3121152 to 3612672 tid 1 2021-07-31T01:48:32.579 INFO:tasks.rados.rados.0.smithi045.stdout:873: writing smithi04569269-15 from 3612672 to 4358144 tid 2 2021-07-31T01:48:32.583 INFO:tasks.rados.rados.0.smithi045.stdout:873: writing smithi04569269-15 from 4358144 to 5103616 tid 3 2021-07-31T01:48:32.584 INFO:tasks.rados.rados.0.smithi045.stdout:873: writing smithi04569269-15 from 5103616 to 5218304 tid 4 2021-07-31T01:48:32.585 INFO:tasks.rados.rados.0.smithi045.stdout:874: delete oid 31 current snap is 80 2021-07-31T01:48:32.585 INFO:tasks.rados.rados.0.smithi045.stdout:873: finishing write tid 1 to smithi04569269-15 2021-07-31T01:48:32.585 INFO:tasks.rados.rados.0.smithi045.stdout:873: finishing write tid 2 to smithi04569269-15 2021-07-31T01:48:32.588 INFO:tasks.rados.rados.0.smithi045.stdout:873: finishing write tid 3 to smithi04569269-15 2021-07-31T01:48:32.588 INFO:tasks.rados.rados.0.smithi045.stdout:873: finishing write tid 4 to smithi04569269-15 2021-07-31T01:48:32.590 INFO:tasks.rados.rados.0.smithi045.stdout:874: done (1 left) 2021-07-31T01:48:32.590 INFO:tasks.rados.rados.0.smithi045.stdout:875: snap_create 2021-07-31T01:48:32.590 INFO:tasks.rados.rados.0.smithi045.stdout:873: finishing write tid 5 to smithi04569269-15 2021-07-31T01:48:32.591 INFO:tasks.rados.rados.0.smithi045.stdout:873: finishing write tid 6 to smithi04569269-15 2021-07-31T01:48:32.591 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 15 v 250 (ObjNum 268 snap 80 seq_num 268) dirty exists 2021-07-31T01:48:32.591 INFO:tasks.rados.rados.0.smithi045.stdout:873: left oid 15 (ObjNum 268 snap 80 seq_num 268) 2021-07-31T01:48:33.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:32 smithi154 conmon[34553]: cluster 2021-07-31T01:48:31.808932+0000 mgr.x (mgr.24542) 2021-07-31T01:48:33.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:32 smithi154 conmon[34553]: 61 : cluster [DBG] pgmap v98: 25 pgs: 25 active+clean; 112 MiB data, 554 MiB used, 715 GiB / 715 GiB avail; 6.5 MiB/s rd, 3.7 MiB/s wr, 20 op/s 2021-07-31T01:48:33.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:32 smithi045 conmon[33655]: cluster 2021-07-31T01:48:31.808932+0000 mgr.x (mgr.24542) 61 : cluster [DBG] 2021-07-31T01:48:33.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:32 smithi045 conmon[33655]: pgmap v98: 25 pgs: 25 active+clean; 112 MiB data, 554 MiB used, 715 GiB / 715 GiB avail; 6.5 MiB/s rd, 3.7 MiB/s wr, 20 op/s 2021-07-31T01:48:33.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:32 smithi045 conmon[39718]: cluster 2021-07-31T01:48:31.808932+0000 mgr.x (mgr.24542) 61 : 2021-07-31T01:48:33.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:32 smithi045 conmon[39718]: cluster [DBG] pgmap v98: 25 pgs: 25 active+clean; 112 MiB data, 554 MiB used, 715 GiB / 715 GiB avail; 6.5 MiB/s rd, 3.7 MiB/s wr, 20 op/s 2021-07-31T01:48:33.557 INFO:tasks.rados.rados.0.smithi045.stdout:873: done (1 left) 2021-07-31T01:48:33.557 INFO:tasks.rados.rados.0.smithi045.stdout:875: done (0 left) 2021-07-31T01:48:33.557 INFO:tasks.rados.rados.0.smithi045.stdout:876: copy_from oid 33 from oid 44 current snap is 81 2021-07-31T01:48:33.558 INFO:tasks.rados.rados.0.smithi045.stdout:877: delete oid 3 current snap is 81 2021-07-31T01:48:33.560 INFO:tasks.rados.rados.0.smithi045.stdout:877: done (1 left) 2021-07-31T01:48:33.561 INFO:tasks.rados.rados.0.smithi045.stdout:878: snap_remove snap 80 2021-07-31T01:48:33.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:33 smithi045 conmon[33655]: cluster 2021-07-31T01:48:32.554741+0000 mon.a 2021-07-31T01:48:33.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:33 smithi045 conmon[33655]: (mon.0) 853 : cluster [DBG] osdmap e213: 8 total, 8 up, 8 in 2021-07-31T01:48:33.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:33 smithi045 conmon[39718]: cluster 2021-07-31T01:48:32.554741+0000 2021-07-31T01:48:33.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:33 smithi045 conmon[39718]: mon.a (mon.0) 853 : cluster [DBG] osdmap e213: 8 total, 8 up, 8 in 2021-07-31T01:48:33.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:48:33 smithi045 conmon[61210]: level=error ts=2021-07-31T01:48:33.510Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:48:33.778 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:48:33 smithi045 conmon[61210]: level=error ts=2021-07-31T01:48:33.510Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:48:33.778 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:48:33 smithi045 conmon[61210]: level=error ts=2021-07-31T01:48:33.510Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T01:48:34.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:33 smithi154 conmon[34553]: cluster 2021-07-31T01:48:32.554741+0000 mon.a (mon.0) 853 : 2021-07-31T01:48:34.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:33 smithi154 conmon[34553]: cluster [DBG] osdmap e213: 8 total, 8 up, 8 in 2021-07-31T01:48:34.568 INFO:tasks.rados.rados.0.smithi045.stdout:878: done (1 left) 2021-07-31T01:48:34.568 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 21 current snap is 81 2021-07-31T01:48:34.569 INFO:tasks.rados.rados.0.smithi045.stdout:876: finishing copy_from to smithi04569269-33 2021-07-31T01:48:34.569 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 33 v 257 (ObjNum 142 snap 30 seq_num 142) dirty exists 2021-07-31T01:48:34.569 INFO:tasks.rados.rados.0.smithi045.stdout:876: finishing copy_from racing read to smithi04569269-33 2021-07-31T01:48:34.569 INFO:tasks.rados.rados.0.smithi045.stdout:879: seq_num 271 ranges {5963776=540672,6504448=802816,7307264=516096,7823360=286720} 2021-07-31T01:48:34.603 INFO:tasks.rados.rados.0.smithi045.stdout:879: writing smithi04569269-21 from 5963776 to 6504448 tid 1 2021-07-31T01:48:34.608 INFO:tasks.rados.rados.0.smithi045.stdout:879: writing smithi04569269-21 from 6504448 to 7307264 tid 2 2021-07-31T01:48:34.611 INFO:tasks.rados.rados.0.smithi045.stdout:879: writing smithi04569269-21 from 7307264 to 7823360 tid 3 2021-07-31T01:48:34.613 INFO:tasks.rados.rados.0.smithi045.stdout:879: writing smithi04569269-21 from 7823360 to 8110080 tid 4 2021-07-31T01:48:34.614 INFO:tasks.rados.rados.0.smithi045.stdout:876: done (1 left) 2021-07-31T01:48:34.614 INFO:tasks.rados.rados.0.smithi045.stdout:880: read oid 26 snap -1 2021-07-31T01:48:34.614 INFO:tasks.rados.rados.0.smithi045.stdout:880: expect (ObjNum 246 snap 68 seq_num 246) 2021-07-31T01:48:34.614 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 49 current snap is 81 2021-07-31T01:48:34.615 INFO:tasks.rados.rados.0.smithi045.stdout:879: finishing write tid 1 to smithi04569269-21 2021-07-31T01:48:34.615 INFO:tasks.rados.rados.0.smithi045.stdout:881: seq_num 272 ranges {1630208=565248,2195456=688128,2883584=729088,3612672=188416} 2021-07-31T01:48:34.625 INFO:tasks.rados.rados.0.smithi045.stdout:881: writing smithi04569269-49 from 1630208 to 2195456 tid 1 2021-07-31T01:48:34.630 INFO:tasks.rados.rados.0.smithi045.stdout:881: writing smithi04569269-49 from 2195456 to 2883584 tid 2 2021-07-31T01:48:34.634 INFO:tasks.rados.rados.0.smithi045.stdout:881: writing smithi04569269-49 from 2883584 to 3612672 tid 3 2021-07-31T01:48:34.636 INFO:tasks.rados.rados.0.smithi045.stdout:881: writing smithi04569269-49 from 3612672 to 3801088 tid 4 2021-07-31T01:48:34.636 INFO:tasks.rados.rados.0.smithi045.stdout:882: setattr oid 50 current snap is 81 2021-07-31T01:48:34.636 INFO:tasks.rados.rados.0.smithi045.stdout:879: finishing write tid 2 to smithi04569269-21 2021-07-31T01:48:34.636 INFO:tasks.rados.rados.0.smithi045.stdout:879: finishing write tid 3 to smithi04569269-21 2021-07-31T01:48:34.637 INFO:tasks.rados.rados.0.smithi045.stdout:879: finishing write tid 4 to smithi04569269-21 2021-07-31T01:48:34.637 INFO:tasks.rados.rados.0.smithi045.stdout:879: finishing write tid 5 to smithi04569269-21 2021-07-31T01:48:34.637 INFO:tasks.rados.rados.0.smithi045.stdout:879: finishing write tid 6 to smithi04569269-21 2021-07-31T01:48:34.637 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 21 v 594 (ObjNum 271 snap 81 seq_num 271) dirty exists 2021-07-31T01:48:34.637 INFO:tasks.rados.rados.0.smithi045.stdout:879: left oid 21 (ObjNum 271 snap 81 seq_num 271) 2021-07-31T01:48:34.638 INFO:tasks.rados.rados.0.smithi045.stdout:881: finishing write tid 1 to smithi04569269-49 2021-07-31T01:48:34.638 INFO:tasks.rados.rados.0.smithi045.stdout:881: finishing write tid 2 to smithi04569269-49 2021-07-31T01:48:34.639 INFO:tasks.rados.rados.0.smithi045.stdout:879: done (3 left) 2021-07-31T01:48:34.644 INFO:tasks.rados.rados.0.smithi045.stdout:883: read oid 48 snap -1 2021-07-31T01:48:34.644 INFO:tasks.rados.rados.0.smithi045.stdout:883: expect (ObjNum 157 snap 37 seq_num 157) 2021-07-31T01:48:34.644 INFO:tasks.rados.rados.0.smithi045.stdout:884: setattr oid 38 current snap is 81 2021-07-31T01:48:34.646 INFO:tasks.rados.rados.0.smithi045.stdout:885: read oid 24 snap -1 2021-07-31T01:48:34.646 INFO:tasks.rados.rados.0.smithi045.stdout:885: expect (ObjNum 162 snap 38 seq_num 162) 2021-07-31T01:48:34.646 INFO:tasks.rados.rados.0.smithi045.stdout:886: delete oid 1 current snap is 81 2021-07-31T01:48:34.646 INFO:tasks.rados.rados.0.smithi045.stdout:881: finishing write tid 3 to smithi04569269-49 2021-07-31T01:48:34.647 INFO:tasks.rados.rados.0.smithi045.stdout:881: finishing write tid 4 to smithi04569269-49 2021-07-31T01:48:34.647 INFO:tasks.rados.rados.0.smithi045.stdout:881: finishing write tid 5 to smithi04569269-49 2021-07-31T01:48:34.647 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 38 v 517 (ObjNum 244 snap 67 seq_num 244) dirty exists 2021-07-31T01:48:34.648 INFO:tasks.rados.rados.0.smithi045.stdout:881: finishing write tid 6 to smithi04569269-49 2021-07-31T01:48:34.648 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 49 v 600 (ObjNum 272 snap 81 seq_num 272) dirty exists 2021-07-31T01:48:34.648 INFO:tasks.rados.rados.0.smithi045.stdout:881: left oid 49 (ObjNum 272 snap 81 seq_num 272) 2021-07-31T01:48:34.650 INFO:tasks.rados.rados.0.smithi045.stdout:881: done (6 left) 2021-07-31T01:48:34.650 INFO:tasks.rados.rados.0.smithi045.stdout:884: done (5 left) 2021-07-31T01:48:34.650 INFO:tasks.rados.rados.0.smithi045.stdout:886: done (4 left) 2021-07-31T01:48:34.651 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 40 current snap is 81 2021-07-31T01:48:34.651 INFO:tasks.rados.rados.0.smithi045.stdout:887: seq_num 274 ranges {2826240=598016,3424256=499712,3923968=745472,4669440=360448} 2021-07-31T01:48:34.669 INFO:tasks.rados.rados.0.smithi045.stdout:887: writing smithi04569269-40 from 2826240 to 3424256 tid 1 2021-07-31T01:48:34.671 INFO:tasks.rados.rados.0.smithi045.stdout:887: writing smithi04569269-40 from 3424256 to 3923968 tid 2 2021-07-31T01:48:34.676 INFO:tasks.rados.rados.0.smithi045.stdout:887: writing smithi04569269-40 from 3923968 to 4669440 tid 3 2021-07-31T01:48:34.679 INFO:tasks.rados.rados.0.smithi045.stdout:887: writing smithi04569269-40 from 4669440 to 5029888 tid 4 2021-07-31T01:48:34.679 INFO:tasks.rados.rados.0.smithi045.stdout:888: read oid 31 snap -1 2021-07-31T01:48:34.680 INFO:tasks.rados.rados.0.smithi045.stdout:888: expect deleted 2021-07-31T01:48:34.680 INFO:tasks.rados.rados.0.smithi045.stdout:889: read oid 13 snap -1 2021-07-31T01:48:34.680 INFO:tasks.rados.rados.0.smithi045.stdout:889: expect (ObjNum 249 snap 69 seq_num 249) 2021-07-31T01:48:34.680 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 14 current snap is 81 2021-07-31T01:48:34.681 INFO:tasks.rados.rados.0.smithi045.stdout:880: expect (ObjNum 246 snap 68 seq_num 246) 2021-07-31T01:48:34.939 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 50 v 386 (ObjNum 259 snap 77 seq_num 259) dirty exists 2021-07-31T01:48:34.940 INFO:tasks.rados.rados.0.smithi045.stdout:883: expect (ObjNum 157 snap 37 seq_num 157) 2021-07-31T01:48:35.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:34 smithi154 conmon[34553]: cluster 2021-07-31T01:48:33.556523+0000 mon.a (mon.0) 854 : cluster [DBG] osdmap e214: 8 total, 8 up, 8 in 2021-07-31T01:48:35.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:34 smithi154 conmon[34553]: cluster 2021-07-31T01:48:33.809317+0000 mgr.x (mgr.24542) 62 : cluster [DBG] pgmap v101: 25 pgs: 25 active+clean; 112 MiB data, 554 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:48:35.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:34 smithi045 conmon[39718]: cluster 2021-07-31T01:48:33.556523+0000 mon.a (mon.0 2021-07-31T01:48:35.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:34 smithi045 conmon[39718]: ) 854 : cluster [DBG] osdmap e214: 8 total, 8 up, 8 in 2021-07-31T01:48:35.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:34 smithi045 conmon[39718]: cluster 2021-07-31T01:48:33.809317+0000 mgr.x (mgr.24542) 62 : cluster [DBG] pgmap v101: 25 pgs: 25 active+clean; 112 MiB data, 554 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:48:35.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:34 smithi045 conmon[33655]: cluster 2021-07-31T01:48:33.556523+0000 2021-07-31T01:48:35.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:34 smithi045 conmon[33655]: mon.a (mon.0) 854 : cluster [DBG] osdmap e214: 8 total, 8 up, 8 in 2021-07-31T01:48:35.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:34 smithi045 conmon[33655]: cluster 2021-07-31T01:48:33.809317+0000 mgr.x ( 2021-07-31T01:48:35.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:34 smithi045 conmon[33655]: mgr.24542) 62 : cluster [DBG] pgmap v101: 25 pgs: 25 active+clean; 112 MiB data, 554 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:48:35.075 INFO:tasks.rados.rados.0.smithi045.stdout:885: expect (ObjNum 162 snap 38 seq_num 162) 2021-07-31T01:48:35.272 INFO:tasks.rados.rados.0.smithi045.stdout:887: finishing write tid 1 to smithi04569269-40 2021-07-31T01:48:35.273 INFO:tasks.rados.rados.0.smithi045.stdout:887: finishing write tid 2 to smithi04569269-40 2021-07-31T01:48:35.273 INFO:tasks.rados.rados.0.smithi045.stdout:887: finishing write tid 3 to smithi04569269-40 2021-07-31T01:48:35.273 INFO:tasks.rados.rados.0.smithi045.stdout:887: finishing write tid 4 to smithi04569269-40 2021-07-31T01:48:35.273 INFO:tasks.rados.rados.0.smithi045.stdout:890: seq_num 275 ranges {1662976=614400,2277376=786432,3063808=499712,3563520=319488} 2021-07-31T01:48:35.284 INFO:tasks.rados.rados.0.smithi045.stdout:890: writing smithi04569269-14 from 1662976 to 2277376 tid 1 2021-07-31T01:48:35.289 INFO:tasks.rados.rados.0.smithi045.stdout:890: writing smithi04569269-14 from 2277376 to 3063808 tid 2 2021-07-31T01:48:35.291 INFO:tasks.rados.rados.0.smithi045.stdout:890: writing smithi04569269-14 from 3063808 to 3563520 tid 3 2021-07-31T01:48:35.294 INFO:tasks.rados.rados.0.smithi045.stdout:890: writing smithi04569269-14 from 3563520 to 3883008 tid 4 2021-07-31T01:48:35.294 INFO:tasks.rados.rados.0.smithi045.stdout:880: done (7 left) 2021-07-31T01:48:35.295 INFO:tasks.rados.rados.0.smithi045.stdout:882: done (6 left) 2021-07-31T01:48:35.295 INFO:tasks.rados.rados.0.smithi045.stdout:883: done (5 left) 2021-07-31T01:48:35.295 INFO:tasks.rados.rados.0.smithi045.stdout:885: done (4 left) 2021-07-31T01:48:35.295 INFO:tasks.rados.rados.0.smithi045.stdout:888: done (3 left) 2021-07-31T01:48:35.295 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 32 current snap is 81 2021-07-31T01:48:35.296 INFO:tasks.rados.rados.0.smithi045.stdout:887: finishing write tid 5 to smithi04569269-40 2021-07-31T01:48:35.296 INFO:tasks.rados.rados.0.smithi045.stdout:887: finishing write tid 6 to smithi04569269-40 2021-07-31T01:48:35.296 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 40 v 519 (ObjNum 274 snap 81 seq_num 274) dirty exists 2021-07-31T01:48:35.296 INFO:tasks.rados.rados.0.smithi045.stdout:887: left oid 40 (ObjNum 274 snap 81 seq_num 274) 2021-07-31T01:48:35.297 INFO:tasks.rados.rados.0.smithi045.stdout:889: expect (ObjNum 249 snap 69 seq_num 249) 2021-07-31T01:48:35.385 INFO:tasks.rados.rados.0.smithi045.stdout:890: finishing write tid 1 to smithi04569269-14 2021-07-31T01:48:35.386 INFO:tasks.rados.rados.0.smithi045.stdout:890: finishing write tid 2 to smithi04569269-14 2021-07-31T01:48:35.386 INFO:tasks.rados.rados.0.smithi045.stdout:890: finishing write tid 3 to smithi04569269-14 2021-07-31T01:48:35.386 INFO:tasks.rados.rados.0.smithi045.stdout:890: finishing write tid 4 to smithi04569269-14 2021-07-31T01:48:35.387 INFO:tasks.rados.rados.0.smithi045.stdout:890: finishing write tid 5 to smithi04569269-14 2021-07-31T01:48:35.387 INFO:tasks.rados.rados.0.smithi045.stdout:891: seq_num 276 ranges {2867200=630784,3497984=671744,4169728=704512,4874240=229376} 2021-07-31T01:48:35.404 INFO:tasks.rados.rados.0.smithi045.stdout:891: writing smithi04569269-32 from 2867200 to 3497984 tid 1 2021-07-31T01:48:35.408 INFO:tasks.rados.rados.0.smithi045.stdout:891: writing smithi04569269-32 from 3497984 to 4169728 tid 2 2021-07-31T01:48:35.411 INFO:tasks.rados.rados.0.smithi045.stdout:891: writing smithi04569269-32 from 4169728 to 4874240 tid 3 2021-07-31T01:48:35.413 INFO:tasks.rados.rados.0.smithi045.stdout:891: writing smithi04569269-32 from 4874240 to 5103616 tid 4 2021-07-31T01:48:35.414 INFO:tasks.rados.rados.0.smithi045.stdout:887: done (3 left) 2021-07-31T01:48:35.414 INFO:tasks.rados.rados.0.smithi045.stdout:889: done (2 left) 2021-07-31T01:48:35.414 INFO:tasks.rados.rados.0.smithi045.stdout:892: rmattr oid 31 current snap is 81 2021-07-31T01:48:35.414 INFO:tasks.rados.rados.0.smithi045.stdout:892: done (2 left) 2021-07-31T01:48:35.415 INFO:tasks.rados.rados.0.smithi045.stdout:893: setattr oid 34 current snap is 81 2021-07-31T01:48:35.415 INFO:tasks.rados.rados.0.smithi045.stdout:890: finishing write tid 6 to smithi04569269-14 2021-07-31T01:48:35.415 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 14 v 287 (ObjNum 275 snap 81 seq_num 275) dirty exists 2021-07-31T01:48:35.415 INFO:tasks.rados.rados.0.smithi045.stdout:890: left oid 14 (ObjNum 275 snap 81 seq_num 275) 2021-07-31T01:48:35.416 INFO:tasks.rados.rados.0.smithi045.stdout:891: finishing write tid 1 to smithi04569269-32 2021-07-31T01:48:35.416 INFO:tasks.rados.rados.0.smithi045.stdout:891: finishing write tid 2 to smithi04569269-32 2021-07-31T01:48:35.416 INFO:tasks.rados.rados.0.smithi045.stdout:890: done (2 left) 2021-07-31T01:48:35.417 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 25 current snap is 81 2021-07-31T01:48:35.417 INFO:tasks.rados.rados.0.smithi045.stdout:894: seq_num 277 ranges {1581056=647168,2228224=557056,2785280=516096,3301376=532480} 2021-07-31T01:48:35.426 INFO:tasks.rados.rados.0.smithi045.stdout:894: writing smithi04569269-25 from 1581056 to 2228224 tid 1 2021-07-31T01:48:35.429 INFO:tasks.rados.rados.0.smithi045.stdout:894: writing smithi04569269-25 from 2228224 to 2785280 tid 2 2021-07-31T01:48:35.433 INFO:tasks.rados.rados.0.smithi045.stdout:894: writing smithi04569269-25 from 2785280 to 3301376 tid 3 2021-07-31T01:48:35.436 INFO:tasks.rados.rados.0.smithi045.stdout:894: writing smithi04569269-25 from 3301376 to 3833856 tid 4 2021-07-31T01:48:35.436 INFO:tasks.rados.rados.0.smithi045.stdout:895: delete oid 3 current snap is 81 2021-07-31T01:48:35.437 INFO:tasks.rados.rados.0.smithi045.stdout:891: finishing write tid 3 to smithi04569269-32 2021-07-31T01:48:35.437 INFO:tasks.rados.rados.0.smithi045.stdout:891: finishing write tid 4 to smithi04569269-32 2021-07-31T01:48:35.437 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 34 v 312 (ObjNum 76 snap 11 seq_num 76) dirty exists 2021-07-31T01:48:35.437 INFO:tasks.rados.rados.0.smithi045.stdout:891: finishing write tid 5 to smithi04569269-32 2021-07-31T01:48:35.438 INFO:tasks.rados.rados.0.smithi045.stdout:891: finishing write tid 6 to smithi04569269-32 2021-07-31T01:48:35.438 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 32 v 204 (ObjNum 276 snap 81 seq_num 276) dirty exists 2021-07-31T01:48:35.438 INFO:tasks.rados.rados.0.smithi045.stdout:891: left oid 32 (ObjNum 276 snap 81 seq_num 276) 2021-07-31T01:48:35.438 INFO:tasks.rados.rados.0.smithi045.stdout:894: finishing write tid 1 to smithi04569269-25 2021-07-31T01:48:35.439 INFO:tasks.rados.rados.0.smithi045.stdout:891: done (3 left) 2021-07-31T01:48:35.440 INFO:tasks.rados.rados.0.smithi045.stdout:893: done (2 left) 2021-07-31T01:48:35.440 INFO:tasks.rados.rados.0.smithi045.stdout:895: done (1 left) 2021-07-31T01:48:35.440 INFO:tasks.rados.rados.0.smithi045.stdout:896: read oid 5 snap -1 2021-07-31T01:48:35.440 INFO:tasks.rados.rados.0.smithi045.stdout:896: expect (ObjNum 158 snap 37 seq_num 158) 2021-07-31T01:48:35.440 INFO:tasks.rados.rados.0.smithi045.stdout:897: read oid 38 snap -1 2021-07-31T01:48:35.441 INFO:tasks.rados.rados.0.smithi045.stdout:897: expect (ObjNum 244 snap 67 seq_num 244) 2021-07-31T01:48:35.441 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 48 current snap is 81 2021-07-31T01:48:35.441 INFO:tasks.rados.rados.0.smithi045.stdout:894: finishing write tid 2 to smithi04569269-25 2021-07-31T01:48:35.441 INFO:tasks.rados.rados.0.smithi045.stdout:898: seq_num 279 ranges {2932736=679936,3612672=720896,4333568=483328,4816896=401408} 2021-07-31T01:48:35.458 INFO:tasks.rados.rados.0.smithi045.stdout:898: writing smithi04569269-48 from 2932736 to 3612672 tid 1 2021-07-31T01:48:35.462 INFO:tasks.rados.rados.0.smithi045.stdout:898: writing smithi04569269-48 from 3612672 to 4333568 tid 2 2021-07-31T01:48:35.465 INFO:tasks.rados.rados.0.smithi045.stdout:898: writing smithi04569269-48 from 4333568 to 4816896 tid 3 2021-07-31T01:48:35.468 INFO:tasks.rados.rados.0.smithi045.stdout:898: writing smithi04569269-48 from 4816896 to 5218304 tid 4 2021-07-31T01:48:35.468 INFO:tasks.rados.rados.0.smithi045.stdout:894: finishing write tid 3 to smithi04569269-25 2021-07-31T01:48:35.468 INFO:tasks.rados.rados.0.smithi045.stdout:899: snap_remove snap 76 2021-07-31T01:48:35.570 INFO:tasks.rados.rados.0.smithi045.stdout:899: done (4 left) 2021-07-31T01:48:35.571 INFO:tasks.rados.rados.0.smithi045.stdout:900: setattr oid 35 current snap is 81 2021-07-31T01:48:35.571 INFO:tasks.rados.rados.0.smithi045.stdout:894: finishing write tid 4 to smithi04569269-25 2021-07-31T01:48:35.571 INFO:tasks.rados.rados.0.smithi045.stdout:894: finishing write tid 5 to smithi04569269-25 2021-07-31T01:48:35.571 INFO:tasks.rados.rados.0.smithi045.stdout:894: finishing write tid 6 to smithi04569269-25 2021-07-31T01:48:35.572 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 25 v 172 (ObjNum 277 snap 81 seq_num 277) dirty exists 2021-07-31T01:48:35.572 INFO:tasks.rados.rados.0.smithi045.stdout:894: left oid 25 (ObjNum 277 snap 81 seq_num 277) 2021-07-31T01:48:35.572 INFO:tasks.rados.rados.0.smithi045.stdout:896: expect (ObjNum 158 snap 37 seq_num 158) 2021-07-31T01:48:35.580 INFO:tasks.rados.rados.0.smithi045.stdout:897: expect (ObjNum 244 snap 67 seq_num 244) 2021-07-31T01:48:35.627 INFO:tasks.rados.rados.0.smithi045.stdout:898: finishing write tid 1 to smithi04569269-48 2021-07-31T01:48:35.628 INFO:tasks.rados.rados.0.smithi045.stdout:898: finishing write tid 2 to smithi04569269-48 2021-07-31T01:48:35.628 INFO:tasks.rados.rados.0.smithi045.stdout:898: finishing write tid 3 to smithi04569269-48 2021-07-31T01:48:35.628 INFO:tasks.rados.rados.0.smithi045.stdout:898: finishing write tid 4 to smithi04569269-48 2021-07-31T01:48:35.628 INFO:tasks.rados.rados.0.smithi045.stdout:898: finishing write tid 5 to smithi04569269-48 2021-07-31T01:48:35.629 INFO:tasks.rados.rados.0.smithi045.stdout:898: finishing write tid 6 to smithi04569269-48 2021-07-31T01:48:35.629 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 48 v 317 (ObjNum 279 snap 81 seq_num 279) dirty exists 2021-07-31T01:48:35.629 INFO:tasks.rados.rados.0.smithi045.stdout:898: left oid 48 (ObjNum 279 snap 81 seq_num 279) 2021-07-31T01:48:35.629 INFO:tasks.rados.rados.0.smithi045.stdout:894: done (4 left) 2021-07-31T01:48:35.630 INFO:tasks.rados.rados.0.smithi045.stdout:896: done (3 left) 2021-07-31T01:48:35.630 INFO:tasks.rados.rados.0.smithi045.stdout:897: done (2 left) 2021-07-31T01:48:35.630 INFO:tasks.rados.rados.0.smithi045.stdout:898: done (1 left) 2021-07-31T01:48:35.631 INFO:tasks.rados.rados.0.smithi045.stdout:901: rmattr oid 43 current snap is 81 2021-07-31T01:48:35.631 INFO:tasks.rados.rados.0.smithi045.stdout:902: snap_remove snap 78 2021-07-31T01:48:36.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:35 smithi154 conmon[34553]: cluster 2021-07-31T01:48:34.567309+0000 mon.a ( 2021-07-31T01:48:36.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:35 smithi154 conmon[34553]: mon.0) 855 : cluster [DBG] osdmap e215: 8 total, 8 up, 8 in 2021-07-31T01:48:36.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:35 smithi045 conmon[39718]: cluster 2021-07-31T01:48:36.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:35 smithi045 conmon[39718]: 2021-07-31T01:48:36.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:35 smithi045 conmon[39718]: 2021 2021-07-31T01:48:36.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:35 smithi045 conmon[39718]: - 2021-07-31T01:48:36.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:35 smithi045 conmon[39718]: 07 2021-07-31T01:48:36.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:35 smithi045 conmon[39718]: - 2021-07-31T01:48:36.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:35 smithi045 conmon[39718]: 31 2021-07-31T01:48:36.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:35 smithi045 conmon[39718]: T 2021-07-31T01:48:36.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:35 smithi045 conmon[39718]: 01 2021-07-31T01:48:36.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:35 smithi045 conmon[39718]: : 2021-07-31T01:48:36.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:35 smithi045 conmon[39718]: 48 2021-07-31T01:48:36.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:35 smithi045 conmon[39718]: : 2021-07-31T01:48:36.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:35 smithi045 conmon[39718]: 34 2021-07-31T01:48:36.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:35 smithi045 conmon[39718]: . 2021-07-31T01:48:36.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:35 smithi045 conmon[39718]: 567309 2021-07-31T01:48:36.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:35 smithi045 conmon[39718]: +0000 2021-07-31T01:48:36.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:35 smithi045 conmon[39718]: 2021-07-31T01:48:36.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:35 smithi045 conmon[39718]: mon.a 2021-07-31T01:48:36.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:35 smithi045 conmon[39718]: ( 2021-07-31T01:48:36.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:35 smithi045 conmon[39718]: mon 2021-07-31T01:48:36.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:35 smithi045 conmon[39718]: . 2021-07-31T01:48:36.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:35 smithi045 conmon[39718]: 0 2021-07-31T01:48:36.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:35 smithi045 conmon[39718]: ) 2021-07-31T01:48:36.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:35 smithi045 conmon[39718]: 855 2021-07-31T01:48:36.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:35 smithi045 conmon[39718]: : 2021-07-31T01:48:36.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:35 smithi045 conmon[39718]: cluster 2021-07-31T01:48:36.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:35 smithi045 conmon[39718]: 2021-07-31T01:48:36.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:35 smithi045 conmon[39718]: [DBG] 2021-07-31T01:48:36.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:35 smithi045 conmon[39718]: 2021-07-31T01:48:36.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:35 smithi045 conmon[39718]: osdmap e215: 8 total, 8 up, 8 in 2021-07-31T01:48:36.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:35 smithi045 conmon[39718]: 2021-07-31T01:48:36.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:35 smithi045 conmon[33655]: cluster 2021-07-31T01:48:34.567309+0000 mon.a (mon.0) 855 : cluster [DBG] 2021-07-31T01:48:36.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:35 smithi045 conmon[33655]: osdmap e215: 8 total, 8 up, 8 in 2021-07-31T01:48:36.579 INFO:tasks.rados.rados.0.smithi045.stdout:902: done (2 left) 2021-07-31T01:48:36.579 INFO:tasks.rados.rados.0.smithi045.stdout:903: rollback oid 50 current snap is 81 2021-07-31T01:48:36.579 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 50 to 72 2021-07-31T01:48:36.580 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 43 v 291 (ObjNum 162 snap 38 seq_num 162) dirty exists 2021-07-31T01:48:36.580 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 35 v 609 (ObjNum 232 snap 64 seq_num 232) dirty exists 2021-07-31T01:48:36.580 INFO:tasks.rados.rados.0.smithi045.stdout:900: done (2 left) 2021-07-31T01:48:36.580 INFO:tasks.rados.rados.0.smithi045.stdout:901: done (1 left) 2021-07-31T01:48:36.580 INFO:tasks.rados.rados.0.smithi045.stdout:904: delete oid 49 current snap is 81 2021-07-31T01:48:36.587 INFO:tasks.rados.rados.0.smithi045.stdout:904: done (1 left) 2021-07-31T01:48:36.587 INFO:tasks.rados.rados.0.smithi045.stdout:905: delete oid 20 current snap is 81 2021-07-31T01:48:36.589 INFO:tasks.rados.rados.0.smithi045.stdout:903: finishing rollback tid 0 to smithi04569269-50 2021-07-31T01:48:36.590 INFO:tasks.rados.rados.0.smithi045.stdout:903: finishing rollback tid 1 to smithi04569269-50 2021-07-31T01:48:36.590 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 50 v 392 (ObjNum -1105496816 snap 21873 seq_num 0) dirty dne 2021-07-31T01:48:36.594 INFO:tasks.rados.rados.0.smithi045.stdout:903: done (1 left) 2021-07-31T01:48:36.594 INFO:tasks.rados.rados.0.smithi045.stdout:905: done (0 left) 2021-07-31T01:48:36.595 INFO:tasks.rados.rados.0.smithi045.stdout:906: snap_remove snap 72 2021-07-31T01:48:37.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:36 smithi154 conmon[34553]: cluster 2021-07-31T01:48:35.567566+0000 mon.a (mon.0) 856 : cluster [DBG] osdmap e216: 8 total, 8 up, 8 in 2021-07-31T01:48:37.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:36 smithi154 conmon[34553]: cluster 2021-07-31T01:48:35.809818+0000 mgr.x (mgr.24542) 63 : cluster [DBG] pgmap v104: 25 pgs: 25 active+clean; 129 MiB data, 589 MiB used, 715 GiB / 715 GiB avail; 1.2 MiB/s rd, 4.2 MiB/s wr, 10 op/s 2021-07-31T01:48:37.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:36 smithi045 conmon[33655]: cluster 2021-07-31T01:48:35.567566+0000 mon.a (mon.0 2021-07-31T01:48:37.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:36 smithi045 conmon[33655]: ) 856 : cluster [DBG] osdmap e216: 8 total, 8 up, 8 in 2021-07-31T01:48:37.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:36 smithi045 conmon[33655]: cluster 2021-07-31T01:48:35.809818+0000 mgr.x (mgr.24542 2021-07-31T01:48:37.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:36 smithi045 conmon[33655]: ) 63 : cluster [DBG] pgmap v104: 25 pgs: 25 active+clean; 129 MiB data, 589 MiB used, 715 GiB / 715 GiB avail; 1.2 MiB/s rd, 4.2 MiB/s wr, 10 op/s 2021-07-31T01:48:37.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:36 smithi045 conmon[39718]: cluster 2021-07-31T01:48:37.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:36 smithi045 conmon[39718]: 2021-07 2021-07-31T01:48:37.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:36 smithi045 conmon[39718]: -31T01:48:35.567566+0000 mon.a (mon.0) 856 : cluster [DBG] osdmap e216: 8 total, 8 up, 8 in 2021-07-31T01:48:37.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:36 smithi045 conmon[39718]: cluster 2021-07-31T01:48:37.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:36 smithi045 conmon[39718]: 2021-07-31T01:48:35.809818+0000 2021-07-31T01:48:37.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:36 smithi045 conmon[39718]: mgr.x (mgr.24542) 63 : cluster [DBG] pgmap v104: 25 pgs: 25 active+clean; 129 MiB data, 589 MiB used, 715 GiB / 715 GiB avail; 1.2 MiB/s rd, 4.2 MiB/s wr, 10 op/s 2021-07-31T01:48:37.580 INFO:tasks.rados.rados.0.smithi045.stdout:906: done (0 left) 2021-07-31T01:48:37.580 INFO:tasks.rados.rados.0.smithi045.stdout:907: snap_remove snap 77 2021-07-31T01:48:38.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:37 smithi154 conmon[34553]: cluster 2021-07-31T01:48:36.577768+0000 mon.a (mon.0) 857 : cluster [DBG] osdmap e217: 8 total, 8 up, 8 in 2021-07-31T01:48:38.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:37 smithi045 conmon[39718]: cluster 2021-07-31T01:48:36.577768+0000 mon.a (mon 2021-07-31T01:48:38.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:37 smithi045 conmon[39718]: .0) 857 : cluster [DBG] osdmap e217: 8 total, 8 up, 8 in 2021-07-31T01:48:38.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:37 smithi045 conmon[33655]: cluster 2021-07-31T01:48:36.577768+0000 mon.a (mon.0 2021-07-31T01:48:38.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:37 smithi045 conmon[33655]: ) 857 : cluster [DBG] osdmap e217: 8 total, 8 up, 8 in 2021-07-31T01:48:38.590 INFO:tasks.rados.rados.0.smithi045.stdout:907: done (0 left) 2021-07-31T01:48:38.590 INFO:tasks.rados.rados.0.smithi045.stdout:908: snap_create 2021-07-31T01:48:39.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:38 smithi154 conmon[34553]: cluster 2021-07-31T01:48:37.578888+0000 mon.a (mon.0) 858 : cluster [DBG] 2021-07-31T01:48:39.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:38 smithi154 conmon[34553]: osdmap e218: 8 total, 8 up, 8 in 2021-07-31T01:48:39.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:38 smithi154 conmon[34553]: cluster 2021-07-31T01:48:37.810359+0000 mgr.x (mgr.24542) 64 : cluster [DBG] pgmap v107: 25 pgs: 25 active+clean; 129 MiB data, 589 MiB used, 715 GiB / 715 GiB avail; 1.2 MiB/s rd, 4.2 MiB/s wr, 10 op/s 2021-07-31T01:48:39.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:38 smithi045 conmon[33655]: cluster 2021-07-31T01:48:37.578888+0000 mon.a (mon.0 2021-07-31T01:48:39.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:38 smithi045 conmon[33655]: ) 858 : cluster [DBG] osdmap e218: 8 total, 8 up, 8 in 2021-07-31T01:48:39.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:38 smithi045 conmon[33655]: cluster 2021-07-31T01:48:37.810359+0000 mgr.x (mgr.24542) 64 : cluster [DBG] pgmap v107: 25 pgs: 25 active+clean; 129 MiB data, 589 MiB used, 715 GiB / 715 GiB avail; 1.2 MiB/s rd, 4.2 MiB/s wr, 10 op/s 2021-07-31T01:48:39.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:38 smithi045 conmon[39718]: cluster 2021-07-31T01:48:37.578888+0000 mon.a (mon. 2021-07-31T01:48:39.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:38 smithi045 conmon[39718]: 0) 858 : cluster [DBG] osdmap e218: 8 total, 8 up, 8 in 2021-07-31T01:48:39.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:38 smithi045 conmon[39718]: cluster 2021-07-31T01:48:37.810359+0000 mgr.x (mgr.24542) 64 : cluster [DBG] pgmap v107: 25 pgs: 25 active+clean; 129 MiB data, 589 MiB used, 715 GiB / 715 GiB avail; 1.2 MiB/s rd, 4.2 MiB/s wr, 10 op/s 2021-07-31T01:48:39.591 INFO:tasks.rados.rados.0.smithi045.stdout:908: done (0 left) 2021-07-31T01:48:39.592 INFO:tasks.rados.rados.0.smithi045.stdout:909: rollback oid 48 current snap is 82 2021-07-31T01:48:39.592 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 48 to 81 2021-07-31T01:48:39.592 INFO:tasks.rados.rados.0.smithi045.stdout:910: copy_from oid 4 from oid 24 current snap is 82 2021-07-31T01:48:39.592 INFO:tasks.rados.rados.0.smithi045.stdout:911: snap_create 2021-07-31T01:48:39.599 INFO:tasks.rados.rados.0.smithi045.stdout:909: finishing rollback tid 0 to smithi04569269-48 2021-07-31T01:48:39.599 INFO:tasks.rados.rados.0.smithi045.stdout:909: finishing rollback tid 1 to smithi04569269-48 2021-07-31T01:48:39.600 INFO:tasks.rados.rados.0.smithi045.stdout:909: finishing rollback tid 2 to smithi04569269-48 2021-07-31T01:48:39.600 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 48 v 320 (ObjNum 279 snap 81 seq_num 279) dirty exists 2021-07-31T01:48:39.614 INFO:tasks.rados.rados.0.smithi045.stdout:910: finishing copy_from to smithi04569269-4 2021-07-31T01:48:39.614 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 4 v 146 (ObjNum 162 snap 38 seq_num 162) dirty exists 2021-07-31T01:48:39.615 INFO:tasks.rados.rados.0.smithi045.stdout:910: finishing copy_from racing read to smithi04569269-4 2021-07-31T01:48:40.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:39 smithi154 conmon[34553]: cluster 2021-07-31T01:48:38.589199+0000 mon.a (mon.0) 859 : cluster 2021-07-31T01:48:40.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:39 smithi154 conmon[34553]: [DBG] osdmap e219: 8 total, 8 up, 8 in 2021-07-31T01:48:40.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:39 smithi045 conmon[33655]: cluster 2021-07-31T01:48:40.113 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:39 smithi045 conmon[33655]: 2021-07-31T01:48: 2021-07-31T01:48:40.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:39 smithi045 conmon[33655]: 38.589199+0000 mon.a (mon.0) 859 : cluster [DBG] osdmap e219: 8 total, 8 up, 8 in 2021-07-31T01:48:40.114 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:39 smithi045 conmon[39718]: cluster 2021-07-31T01:48:40.114 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:39 smithi045 conmon[39718]: 2021-07-31T01:48:40.114 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:39 smithi045 conmon[39718]: 2021 2021-07-31T01:48:40.115 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:39 smithi045 conmon[39718]: - 2021-07-31T01:48:40.115 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:39 smithi045 conmon[39718]: 07 2021-07-31T01:48:40.115 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:39 smithi045 conmon[39718]: -31T01:48:38.589199+0000 mon.a 2021-07-31T01:48:40.115 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:39 smithi045 conmon[39718]: (mon.0) 859 : cluster [DBG] osdmap e219: 8 total, 8 up, 8 in 2021-07-31T01:48:40.602 INFO:tasks.rados.rados.0.smithi045.stdout:909: done (2 left) 2021-07-31T01:48:40.603 INFO:tasks.rados.rados.0.smithi045.stdout:910: done (1 left) 2021-07-31T01:48:40.603 INFO:tasks.rados.rados.0.smithi045.stdout:911: done (0 left) 2021-07-31T01:48:40.603 INFO:tasks.rados.rados.0.smithi045.stdout:912: read oid 34 snap 71 2021-07-31T01:48:40.603 INFO:tasks.rados.rados.0.smithi045.stdout:912: expect (ObjNum 76 snap 11 seq_num 76) 2021-07-31T01:48:40.603 INFO:tasks.rados.rados.0.smithi045.stdout:913: read oid 46 snap -1 2021-07-31T01:48:40.604 INFO:tasks.rados.rados.0.smithi045.stdout:913: expect (ObjNum 196 snap 52 seq_num 196) 2021-07-31T01:48:40.604 INFO:tasks.rados.rados.0.smithi045.stdout:914: read oid 41 snap -1 2021-07-31T01:48:40.604 INFO:tasks.rados.rados.0.smithi045.stdout:914: expect (ObjNum 232 snap 64 seq_num 232) 2021-07-31T01:48:40.604 INFO:tasks.rados.rados.0.smithi045.stdout:915: copy_from oid 4 from oid 18 current snap is 83 2021-07-31T01:48:40.605 INFO:tasks.rados.rados.0.smithi045.stdout:916: copy_from oid 9 from oid 44 current snap is 83 2021-07-31T01:48:40.605 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 2 current snap is 83 2021-07-31T01:48:40.605 INFO:tasks.rados.rados.0.smithi045.stdout:917: seq_num 282 ranges {1867776=729088,2596864=425984,3022848=704512,3727360=475136} 2021-07-31T01:48:40.618 INFO:tasks.rados.rados.0.smithi045.stdout:917: writing smithi04569269-2 from 1867776 to 2596864 tid 1 2021-07-31T01:48:40.621 INFO:tasks.rados.rados.0.smithi045.stdout:917: writing smithi04569269-2 from 2596864 to 3022848 tid 2 2021-07-31T01:48:40.627 INFO:tasks.rados.rados.0.smithi045.stdout:917: writing smithi04569269-2 from 3022848 to 3727360 tid 3 2021-07-31T01:48:40.630 INFO:tasks.rados.rados.0.smithi045.stdout:917: writing smithi04569269-2 from 3727360 to 4202496 tid 4 2021-07-31T01:48:40.630 INFO:tasks.rados.rados.0.smithi045.stdout:918: snap_remove snap 79 2021-07-31T01:48:41.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:40 smithi154 conmon[34553]: cluster 2021-07-31T01:48:39.590706+0000 mon.a (mon.0) 860 2021-07-31T01:48:41.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:40 smithi154 conmon[34553]: : cluster [DBG] osdmap e220: 8 total, 8 up, 8 in 2021-07-31T01:48:41.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:40 smithi154 conmon[34553]: cluster 2021-07-31T01:48:39.810903+0000 mgr.x (mgr.24542) 65 : cluster [DBG] pgmap v110: 25 pgs: 25 active+clean; 141 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 13 MiB/s rd, 3.7 MiB/s wr, 39 op/s 2021-07-31T01:48:41.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:40 smithi045 conmon[33655]: cluster 2021-07-31T01:48:39.590706+0000 mon.a (mon.0) 860 : cluster [DBG] osdmap e220: 8 total, 8 up, 8 in 2021-07-31T01:48:41.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:40 smithi045 conmon[33655]: cluster 2021-07-31T01:48:39.810903+0000 mgr.x (mgr.24542) 65 : cluster [DBG] pgmap v110: 25 pgs: 25 active+clean; 141 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 13 MiB/s rd, 3.7 MiB/s wr, 39 op/s 2021-07-31T01:48:41.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:40 smithi045 conmon[39718]: cluster 2021-07-31T01:48:39.590706+0000 mon.a ( 2021-07-31T01:48:41.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:40 smithi045 conmon[39718]: mon.0) 860 : cluster [DBG] osdmap e220: 8 total, 8 up, 8 in 2021-07-31T01:48:41.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:40 smithi045 conmon[39718]: cluster 2021-07-31T01:48:39.810903+0000 mgr.x ( 2021-07-31T01:48:41.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:40 smithi045 conmon[39718]: mgr.24542) 65 : cluster [DBG] pgmap v110: 25 pgs: 25 active+clean; 141 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 13 MiB/s rd, 3.7 MiB/s wr, 39 op/s 2021-07-31T01:48:41.512 INFO:tasks.rados.rados.0.smithi045.stdout:918: done (6 left) 2021-07-31T01:48:41.512 INFO:tasks.rados.rados.0.smithi045.stdout:919: rmattr oid 47 current snap is 83 2021-07-31T01:48:41.512 INFO:tasks.rados.rados.0.smithi045.stdout:919: done (6 left) 2021-07-31T01:48:41.512 INFO:tasks.rados.rados.0.smithi045.stdout:920: rollback oid 29 current snap is 83 2021-07-31T01:48:41.513 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 29 to 81 2021-07-31T01:48:41.513 INFO:tasks.rados.rados.0.smithi045.stdout:921: delete oid 39 current snap is 83 2021-07-31T01:48:41.513 INFO:tasks.rados.rados.0.smithi045.stdout:912: expect (ObjNum 76 snap 11 seq_num 76) 2021-07-31T01:48:41.600 INFO:tasks.rados.rados.0.smithi045.stdout:915: finishing copy_from to smithi04569269-4 2021-07-31T01:48:41.600 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 4 v 217 (ObjNum 172 snap 41 seq_num 172) dirty exists 2021-07-31T01:48:41.601 INFO:tasks.rados.rados.0.smithi045.stdout:915: finishing copy_from racing read to smithi04569269-4 2021-07-31T01:48:41.601 INFO:tasks.rados.rados.0.smithi045.stdout:917: finishing write tid 1 to smithi04569269-2 2021-07-31T01:48:41.601 INFO:tasks.rados.rados.0.smithi045.stdout:912: done (7 left) 2021-07-31T01:48:41.601 INFO:tasks.rados.rados.0.smithi045.stdout:915: done (6 left) 2021-07-31T01:48:41.601 INFO:tasks.rados.rados.0.smithi045.stdout:921: done (5 left) 2021-07-31T01:48:41.602 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 20 current snap is 83 2021-07-31T01:48:41.602 INFO:tasks.rados.rados.0.smithi045.stdout:917: finishing write tid 2 to smithi04569269-2 2021-07-31T01:48:41.602 INFO:tasks.rados.rados.0.smithi045.stdout:922: seq_num 284 ranges {0=761856,761856=598016,1359872=671744,2031616=335872} 2021-07-31T01:48:41.603 INFO:tasks.rados.rados.0.smithi045.stdout:922: writing smithi04569269-20 from 0 to 761856 tid 1 2021-07-31T01:48:41.607 INFO:tasks.rados.rados.0.smithi045.stdout:922: writing smithi04569269-20 from 761856 to 1359872 tid 2 2021-07-31T01:48:41.610 INFO:tasks.rados.rados.0.smithi045.stdout:922: writing smithi04569269-20 from 1359872 to 2031616 tid 3 2021-07-31T01:48:41.613 INFO:tasks.rados.rados.0.smithi045.stdout:922: writing smithi04569269-20 from 2031616 to 2367488 tid 4 2021-07-31T01:48:41.613 INFO:tasks.rados.rados.0.smithi045.stdout:923: read oid 7 snap -1 2021-07-31T01:48:41.613 INFO:tasks.rados.rados.0.smithi045.stdout:923: expect deleted 2021-07-31T01:48:41.613 INFO:tasks.rados.rados.0.smithi045.stdout:924: snap_remove snap 82 2021-07-31T01:48:42.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:41 smithi154 conmon[34553]: cluster 2021-07-31T01:48:40.600829+0000 mon.a (mon.0) 861 : cluster [DBG] osdmap e221: 8 total, 8 up, 8 in 2021-07-31T01:48:42.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:41 smithi154 conmon[34553]: cluster 2021-07-31T01:48:41.510526+0000 mon.a (mon.0) 862 : cluster [DBG] osdmap e222: 8 total, 8 up, 8 in 2021-07-31T01:48:42.003 INFO:journalctl@ceph.osd.7.smithi154.stdout:Jul 31 01:48:41 smithi154 conmon[49149]: debug 2021-07-31T01:48:41.509+0000 7f8a75a04700 -1 osd.7 pg_epoch: 222 pg[3.ds0( v 219'406 (0'0,219'406] local-lis/les=56/57 n=8 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [7,5,3]p7(0) r=0 lpr=56 crt=219'406 lcod 219'404 mlcod 219'404 active+clean+snaptrim trimq=[9a~1] ps=[96~1]] removing snap head 2021-07-31T01:48:42.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:41 smithi045 conmon[33655]: cluster 2021-07-31T01:48:40.600829+0000 mon.a (mon. 2021-07-31T01:48:42.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:41 smithi045 conmon[33655]: 0) 861 : cluster [DBG] osdmap e221: 8 total, 8 up, 8 in 2021-07-31T01:48:42.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:41 smithi045 conmon[33655]: cluster 2021-07-31T01:48:41.510526+0000 mon.a (mon.0) 862 : cluster [DBG] osdmap e222: 8 total, 8 up, 8 in 2021-07-31T01:48:42.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:41 smithi045 conmon[39718]: cluster 2021-07-31T01:48: 2021-07-31T01:48:42.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:41 smithi045 conmon[39718]: 40.600829+0000 mon.a (mon.0) 861 : cluster [DBG] osdmap e221: 8 total, 8 up, 8 in 2021-07-31T01:48:42.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:41 smithi045 conmon[39718]: cluster 2021-07 2021-07-31T01:48:42.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:41 smithi045 conmon[39718]: -31T01:48:41.510526+0000 mon.a (mon 2021-07-31T01:48:42.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:41 smithi045 conmon[39718]: .0) 862 : cluster [DBG] osdmap e222: 8 total, 8 up, 8 in 2021-07-31T01:48:42.608 INFO:tasks.rados.rados.0.smithi045.stdout:924: done (7 left) 2021-07-31T01:48:42.608 INFO:tasks.rados.rados.0.smithi045.stdout:925: read oid 43 snap -1 2021-07-31T01:48:42.609 INFO:tasks.rados.rados.0.smithi045.stdout:925: expect (ObjNum 162 snap 38 seq_num 162) 2021-07-31T01:48:42.609 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 24 current snap is 83 2021-07-31T01:48:42.609 INFO:tasks.rados.rados.0.smithi045.stdout:926: seq_num 285 ranges {3817472=778240,4595712=475136,5070848=475136,5545984=655360} 2021-07-31T01:48:42.634 INFO:tasks.rados.rados.0.smithi045.stdout:926: writing smithi04569269-24 from 3817472 to 4595712 tid 1 2021-07-31T01:48:42.636 INFO:tasks.rados.rados.0.smithi045.stdout:926: writing smithi04569269-24 from 4595712 to 5070848 tid 2 2021-07-31T01:48:42.639 INFO:tasks.rados.rados.0.smithi045.stdout:926: writing smithi04569269-24 from 5070848 to 5545984 tid 3 2021-07-31T01:48:42.644 INFO:tasks.rados.rados.0.smithi045.stdout:926: writing smithi04569269-24 from 5545984 to 6201344 tid 4 2021-07-31T01:48:42.644 INFO:tasks.rados.rados.0.smithi045.stdout:927: read oid 47 snap -1 2021-07-31T01:48:42.644 INFO:tasks.rados.rados.0.smithi045.stdout:927: expect (ObjNum 267 snap 79 seq_num 267) 2021-07-31T01:48:42.644 INFO:tasks.rados.rados.0.smithi045.stdout:928: setattr oid 25 current snap is 83 2021-07-31T01:48:42.645 INFO:tasks.rados.rados.0.smithi045.stdout:917: finishing write tid 3 to smithi04569269-2 2021-07-31T01:48:42.645 INFO:tasks.rados.rados.0.smithi045.stdout:914: expect (ObjNum 232 snap 64 seq_num 232) 2021-07-31T01:48:42.798 INFO:tasks.rados.rados.0.smithi045.stdout:913: expect (ObjNum 196 snap 52 seq_num 196) 2021-07-31T01:48:43.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:42 smithi154 conmon[34553]: cluster 2021-07-31T01:48:41.811337+0000 mgr.x (mgr.24542 2021-07-31T01:48:43.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:42 smithi154 conmon[34553]: ) 66 : cluster [DBG] pgmap v113: 25 pgs: 25 active+clean; 141 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 13 MiB/s rd, 3.7 MiB/s wr, 39 op/s 2021-07-31T01:48:43.026 INFO:tasks.rados.rados.0.smithi045.stdout:917: finishing write tid 4 to smithi04569269-2 2021-07-31T01:48:43.026 INFO:tasks.rados.rados.0.smithi045.stdout:916: finishing copy_from to smithi04569269-9 2021-07-31T01:48:43.026 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 9 v 257 (ObjNum 142 snap 30 seq_num 142) dirty exists 2021-07-31T01:48:43.026 INFO:tasks.rados.rados.0.smithi045.stdout:916: finishing copy_from racing read to smithi04569269-9 2021-07-31T01:48:43.027 INFO:tasks.rados.rados.0.smithi045.stdout:917: finishing write tid 5 to smithi04569269-2 2021-07-31T01:48:43.027 INFO:tasks.rados.rados.0.smithi045.stdout:917: finishing write tid 6 to smithi04569269-2 2021-07-31T01:48:43.027 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 2 v 258 (ObjNum 282 snap 83 seq_num 282) dirty exists 2021-07-31T01:48:43.027 INFO:tasks.rados.rados.0.smithi045.stdout:917: left oid 2 (ObjNum 282 snap 83 seq_num 282) 2021-07-31T01:48:43.028 INFO:tasks.rados.rados.0.smithi045.stdout:920: finishing rollback tid 0 to smithi04569269-29 2021-07-31T01:48:43.028 INFO:tasks.rados.rados.0.smithi045.stdout:920: finishing rollback tid 1 to smithi04569269-29 2021-07-31T01:48:43.028 INFO:tasks.rados.rados.0.smithi045.stdout:920: finishing rollback tid 2 to smithi04569269-29 2021-07-31T01:48:43.028 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 29 v 179 (ObjNum 237 snap 66 seq_num 237) dirty exists 2021-07-31T01:48:43.028 INFO:tasks.rados.rados.0.smithi045.stdout:922: finishing write tid 1 to smithi04569269-20 2021-07-31T01:48:43.029 INFO:tasks.rados.rados.0.smithi045.stdout:922: finishing write tid 2 to smithi04569269-20 2021-07-31T01:48:43.029 INFO:tasks.rados.rados.0.smithi045.stdout:922: finishing write tid 3 to smithi04569269-20 2021-07-31T01:48:43.029 INFO:tasks.rados.rados.0.smithi045.stdout:922: finishing write tid 4 to smithi04569269-20 2021-07-31T01:48:43.029 INFO:tasks.rados.rados.0.smithi045.stdout:922: finishing write tid 5 to smithi04569269-20 2021-07-31T01:48:43.030 INFO:tasks.rados.rados.0.smithi045.stdout:922: finishing write tid 6 to smithi04569269-20 2021-07-31T01:48:43.030 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 20 v 653 (ObjNum 284 snap 83 seq_num 284) dirty exists 2021-07-31T01:48:43.030 INFO:tasks.rados.rados.0.smithi045.stdout:922: left oid 20 (ObjNum 284 snap 83 seq_num 284) 2021-07-31T01:48:43.030 INFO:tasks.rados.rados.0.smithi045.stdout:925: expect (ObjNum 162 snap 38 seq_num 162) 2021-07-31T01:48:43.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:42 smithi045 conmon[33655]: cluster 2021-07-31T01:48:41.811337+0000 mgr.x (mgr.24542) 66 : cluster [DBG] 2021-07-31T01:48:43.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:42 smithi045 conmon[33655]: pgmap v113: 25 pgs: 25 active+clean; 141 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 13 MiB/s rd, 3.7 MiB/s wr, 39 op/s 2021-07-31T01:48:43.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:42 smithi045 conmon[39718]: cluster 2021-07-31T01:48:41.811337+0000 mgr.x (mgr.24542) 2021-07-31T01:48:43.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:42 smithi045 conmon[39718]: 66 : cluster [DBG] pgmap v113: 25 pgs: 25 active+clean; 141 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 13 MiB/s rd, 3.7 MiB/s wr, 39 op/s 2021-07-31T01:48:43.222 INFO:tasks.rados.rados.0.smithi045.stdout:926: finishing write tid 1 to smithi04569269-24 2021-07-31T01:48:43.223 INFO:tasks.rados.rados.0.smithi045.stdout:926: finishing write tid 2 to smithi04569269-24 2021-07-31T01:48:43.223 INFO:tasks.rados.rados.0.smithi045.stdout:926: finishing write tid 3 to smithi04569269-24 2021-07-31T01:48:43.223 INFO:tasks.rados.rados.0.smithi045.stdout:926: finishing write tid 4 to smithi04569269-24 2021-07-31T01:48:43.224 INFO:tasks.rados.rados.0.smithi045.stdout:926: finishing write tid 5 to smithi04569269-24 2021-07-31T01:48:43.224 INFO:tasks.rados.rados.0.smithi045.stdout:926: finishing write tid 6 to smithi04569269-24 2021-07-31T01:48:43.224 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 24 v 307 (ObjNum 285 snap 83 seq_num 285) dirty exists 2021-07-31T01:48:43.224 INFO:tasks.rados.rados.0.smithi045.stdout:926: left oid 24 (ObjNum 285 snap 83 seq_num 285) 2021-07-31T01:48:43.224 INFO:tasks.rados.rados.0.smithi045.stdout:913: done (10 left) 2021-07-31T01:48:43.225 INFO:tasks.rados.rados.0.smithi045.stdout:914: done (9 left) 2021-07-31T01:48:43.225 INFO:tasks.rados.rados.0.smithi045.stdout:916: done (8 left) 2021-07-31T01:48:43.225 INFO:tasks.rados.rados.0.smithi045.stdout:917: done (7 left) 2021-07-31T01:48:43.225 INFO:tasks.rados.rados.0.smithi045.stdout:920: done (6 left) 2021-07-31T01:48:43.225 INFO:tasks.rados.rados.0.smithi045.stdout:922: done (5 left) 2021-07-31T01:48:43.226 INFO:tasks.rados.rados.0.smithi045.stdout:923: done (4 left) 2021-07-31T01:48:43.226 INFO:tasks.rados.rados.0.smithi045.stdout:925: done (3 left) 2021-07-31T01:48:43.226 INFO:tasks.rados.rados.0.smithi045.stdout:926: done (2 left) 2021-07-31T01:48:43.226 INFO:tasks.rados.rados.0.smithi045.stdout:929: read oid 16 snap -1 2021-07-31T01:48:43.226 INFO:tasks.rados.rados.0.smithi045.stdout:929: expect (ObjNum 207 snap 53 seq_num 207) 2021-07-31T01:48:43.227 INFO:tasks.rados.rados.0.smithi045.stdout:930: copy_from oid 4 from oid 15 current snap is 83 2021-07-31T01:48:43.227 INFO:tasks.rados.rados.0.smithi045.stdout:927: expect (ObjNum 267 snap 79 seq_num 267) 2021-07-31T01:48:43.474 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 25 v 173 (ObjNum 277 snap 81 seq_num 277) dirty exists 2021-07-31T01:48:43.474 INFO:tasks.rados.rados.0.smithi045.stdout:927: done (3 left) 2021-07-31T01:48:43.475 INFO:tasks.rados.rados.0.smithi045.stdout:928: done (2 left) 2021-07-31T01:48:43.475 INFO:tasks.rados.rados.0.smithi045.stdout:931: rollback oid 43 current snap is 83 2021-07-31T01:48:43.475 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 43 to 71 2021-07-31T01:48:43.475 INFO:tasks.rados.rados.0.smithi045.stdout:929: expect (ObjNum 207 snap 53 seq_num 207) 2021-07-31T01:48:43.502 INFO:tasks.rados.rados.0.smithi045.stdout:930: finishing copy_from to smithi04569269-4 2021-07-31T01:48:43.503 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 4 v 250 (ObjNum 268 snap 80 seq_num 268) dirty exists 2021-07-31T01:48:43.503 INFO:tasks.rados.rados.0.smithi045.stdout:930: finishing copy_from racing read to smithi04569269-4 2021-07-31T01:48:43.504 INFO:tasks.rados.rados.0.smithi045.stdout:931: finishing rollback tid 0 to smithi04569269-43 2021-07-31T01:48:43.504 INFO:tasks.rados.rados.0.smithi045.stdout:931: finishing rollback tid 1 to smithi04569269-43 2021-07-31T01:48:43.504 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 43 v 310 (ObjNum 0 snap 32731 seq_num 3189063376) dirty dne 2021-07-31T01:48:43.504 INFO:tasks.rados.rados.0.smithi045.stdout:929: done (2 left) 2021-07-31T01:48:43.505 INFO:tasks.rados.rados.0.smithi045.stdout:930: done (1 left) 2021-07-31T01:48:43.505 INFO:tasks.rados.rados.0.smithi045.stdout:931: done (0 left) 2021-07-31T01:48:43.505 INFO:tasks.rados.rados.0.smithi045.stdout:932: copy_from oid 8 from oid 9 current snap is 83 2021-07-31T01:48:43.505 INFO:tasks.rados.rados.0.smithi045.stdout:933: read oid 15 snap 81 2021-07-31T01:48:43.506 INFO:tasks.rados.rados.0.smithi045.stdout:933: expect (ObjNum 268 snap 80 seq_num 268) 2021-07-31T01:48:43.506 INFO:tasks.rados.rados.0.smithi045.stdout:934: copy_from oid 50 from oid 18 current snap is 83 2021-07-31T01:48:43.506 INFO:tasks.rados.rados.0.smithi045.stdout:935: snap_create 2021-07-31T01:48:43.517 INFO:tasks.rados.rados.0.smithi045.stdout:934: finishing copy_from to smithi04569269-50 2021-07-31T01:48:43.517 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 50 v 217 (ObjNum 172 snap 41 seq_num 172) dirty exists 2021-07-31T01:48:43.518 INFO:tasks.rados.rados.0.smithi045.stdout:934: finishing copy_from racing read to smithi04569269-50 2021-07-31T01:48:43.519 INFO:tasks.rados.rados.0.smithi045.stdout:933: expect (ObjNum 268 snap 80 seq_num 268) 2021-07-31T01:48:43.735 INFO:tasks.rados.rados.0.smithi045.stdout:932: finishing copy_from to smithi04569269-8 2021-07-31T01:48:43.736 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 8 v 257 (ObjNum 142 snap 30 seq_num 142) dirty exists 2021-07-31T01:48:43.736 INFO:tasks.rados.rados.0.smithi045.stdout:932: finishing copy_from racing read to smithi04569269-8 2021-07-31T01:48:43.736 INFO:tasks.rados.rados.0.smithi045.stdout:932: done (3 left) 2021-07-31T01:48:43.737 INFO:tasks.rados.rados.0.smithi045.stdout:933: done (2 left) 2021-07-31T01:48:43.737 INFO:tasks.rados.rados.0.smithi045.stdout:934: done (1 left) 2021-07-31T01:48:43.737 INFO:tasks.rados.rados.0.smithi045.stdout:935: done (0 left) 2021-07-31T01:48:43.737 INFO:tasks.rados.rados.0.smithi045.stdout:936: read oid 48 snap -1 2021-07-31T01:48:43.737 INFO:tasks.rados.rados.0.smithi045.stdout:936: expect (ObjNum 279 snap 81 seq_num 279) 2021-07-31T01:48:43.738 INFO:tasks.rados.rados.0.smithi045.stdout:937: read oid 5 snap -1 2021-07-31T01:48:43.738 INFO:tasks.rados.rados.0.smithi045.stdout:937: expect (ObjNum 158 snap 37 seq_num 158) 2021-07-31T01:48:43.739 INFO:tasks.rados.rados.0.smithi045.stdout:938: read oid 49 snap -1 2021-07-31T01:48:43.739 INFO:tasks.rados.rados.0.smithi045.stdout:938: expect deleted 2021-07-31T01:48:43.739 INFO:tasks.rados.rados.0.smithi045.stdout:939: copy_from oid 23 from oid 7 current snap is 84 2021-07-31T01:48:43.740 INFO:tasks.rados.rados.0.smithi045.stdout:940: read oid 19 snap -1 2021-07-31T01:48:43.740 INFO:tasks.rados.rados.0.smithi045.stdout:940: expect (ObjNum 216 snap 56 seq_num 216) 2021-07-31T01:48:43.740 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 15 current snap is 84 2021-07-31T01:48:43.740 INFO:tasks.rados.rados.0.smithi045.stdout:941: seq_num 286 ranges {5218304=794624,6012928=761856,6774784=679936,7454720=163840} 2021-07-31T01:48:43.769 INFO:tasks.rados.rados.0.smithi045.stdout:941: writing smithi04569269-15 from 5218304 to 6012928 tid 1 2021-07-31T01:48:43.774 INFO:tasks.rados.rados.0.smithi045.stdout:941: writing smithi04569269-15 from 6012928 to 6774784 tid 2 2021-07-31T01:48:43.775 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:48:43 smithi045 conmon[61210]: level=error ts=2021-07-31T01:48:43.510Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:48:43.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:48:43 smithi045 conmon[61210]: level=error ts=2021-07-31T01:48:43.510Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:48:43.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:48:43 smithi045 conmon[61210]: level=error ts=2021-07-31T01:48:43.510Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T01:48:43.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:43 smithi045 conmon[39718]: cluster 2021-07-31T01:48:42.607315 2021-07-31T01:48:43.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:43 smithi045 conmon[39718]: +0000 mon.a (mon.0) 863 : cluster [DBG] osdmap e223: 8 total, 8 up, 8 in 2021-07-31T01:48:43.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:43 smithi045 conmon[33655]: cluster 2021-07-31T01:48:42.607315+0000 mon.a 2021-07-31T01:48:43.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:43 smithi045 conmon[33655]: (mon.0) 863 : cluster [DBG] osdmap e223: 8 total, 8 up, 8 in 2021-07-31T01:48:43.779 INFO:tasks.rados.rados.0.smithi045.stdout:941: writing smithi04569269-15 from 6774784 to 7454720 tid 3 2021-07-31T01:48:43.779 INFO:tasks.rados.rados.0.smithi045.stdout:941: writing smithi04569269-15 from 7454720 to 7618560 tid 4 2021-07-31T01:48:43.779 INFO:tasks.rados.rados.0.smithi045.stdout:942: setattr oid 50 current snap is 84 2021-07-31T01:48:43.779 INFO:tasks.rados.rados.0.smithi045.stdout:940: expect (ObjNum 216 snap 56 seq_num 216) 2021-07-31T01:48:43.883 INFO:tasks.rados.rados.0.smithi045.stdout:937: expect (ObjNum 158 snap 37 seq_num 158) 2021-07-31T01:48:43.891 INFO:tasks.rados.rados.0.smithi045.stdout:939: finishing copy_from to smithi04569269-23 2021-07-31T01:48:43.891 INFO:tasks.rados.rados.0.smithi045.stdout:939: got expected ENOENT (src dne) 2021-07-31T01:48:43.891 INFO:tasks.rados.rados.0.smithi045.stdout:939: finishing copy_from racing read to smithi04569269-23 2021-07-31T01:48:43.892 INFO:tasks.rados.rados.0.smithi045.stdout:936: expect (ObjNum 279 snap 81 seq_num 279) 2021-07-31T01:48:44.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:43 smithi154 conmon[34553]: cluster 2021-07-31T01:48:42.607315+0000 mon.a (mon.0) 863 : cluster [DBG] osdmap e223: 8 total, 8 up, 8 in 2021-07-31T01:48:44.135 INFO:tasks.rados.rados.0.smithi045.stdout:941: finishing write tid 1 to smithi04569269-15 2021-07-31T01:48:44.136 INFO:tasks.rados.rados.0.smithi045.stdout:941: finishing write tid 2 to smithi04569269-15 2021-07-31T01:48:44.136 INFO:tasks.rados.rados.0.smithi045.stdout:941: finishing write tid 3 to smithi04569269-15 2021-07-31T01:48:44.136 INFO:tasks.rados.rados.0.smithi045.stdout:941: finishing write tid 4 to smithi04569269-15 2021-07-31T01:48:44.137 INFO:tasks.rados.rados.0.smithi045.stdout:941: finishing write tid 5 to smithi04569269-15 2021-07-31T01:48:44.138 INFO:tasks.rados.rados.0.smithi045.stdout:941: finishing write tid 6 to smithi04569269-15 2021-07-31T01:48:44.138 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 15 v 275 (ObjNum 286 snap 84 seq_num 286) dirty exists 2021-07-31T01:48:44.138 INFO:tasks.rados.rados.0.smithi045.stdout:941: left oid 15 (ObjNum 286 snap 84 seq_num 286) 2021-07-31T01:48:44.139 INFO:tasks.rados.rados.0.smithi045.stdout:936: done (6 left) 2021-07-31T01:48:44.139 INFO:tasks.rados.rados.0.smithi045.stdout:937: done (5 left) 2021-07-31T01:48:44.139 INFO:tasks.rados.rados.0.smithi045.stdout:938: done (4 left) 2021-07-31T01:48:44.139 INFO:tasks.rados.rados.0.smithi045.stdout:939: done (3 left) 2021-07-31T01:48:44.140 INFO:tasks.rados.rados.0.smithi045.stdout:940: done (2 left) 2021-07-31T01:48:44.140 INFO:tasks.rados.rados.0.smithi045.stdout:941: done (1 left) 2021-07-31T01:48:44.140 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 2 current snap is 84 2021-07-31T01:48:44.140 INFO:tasks.rados.rados.0.smithi045.stdout:943: seq_num 287 ranges {4202496=8192} 2021-07-31T01:48:44.159 INFO:tasks.rados.rados.0.smithi045.stdout:943: writing smithi04569269-2 from 4202496 to 4210688 tid 1 2021-07-31T01:48:44.160 INFO:tasks.rados.rados.0.smithi045.stdout:944: rmattr oid 6 current snap is 84 2021-07-31T01:48:44.160 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 50 v 411 (ObjNum 172 snap 41 seq_num 172) dirty exists 2021-07-31T01:48:44.160 INFO:tasks.rados.rados.0.smithi045.stdout:942: done (2 left) 2021-07-31T01:48:44.161 INFO:tasks.rados.rados.0.smithi045.stdout:945: rmattr oid 49 current snap is 84 2021-07-31T01:48:44.161 INFO:tasks.rados.rados.0.smithi045.stdout:946: snap_remove snap 81 2021-07-31T01:48:44.618 INFO:tasks.rados.rados.0.smithi045.stdout:946: done (3 left) 2021-07-31T01:48:44.619 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 36 current snap is 84 2021-07-31T01:48:44.619 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 49 v 0 (ObjNum 1651209053 snap 1450470495 seq_num 1348755813) dirty dne 2021-07-31T01:48:44.619 INFO:tasks.rados.rados.0.smithi045.stdout:943: finishing write tid 1 to smithi04569269-2 2021-07-31T01:48:44.619 INFO:tasks.rados.rados.0.smithi045.stdout:943: finishing write tid 2 to smithi04569269-2 2021-07-31T01:48:44.620 INFO:tasks.rados.rados.0.smithi045.stdout:947: seq_num 288 ranges {2621440=24576} 2021-07-31T01:48:44.633 INFO:tasks.rados.rados.0.smithi045.stdout:947: writing smithi04569269-36 from 2621440 to 2646016 tid 1 2021-07-31T01:48:44.633 INFO:tasks.rados.rados.0.smithi045.stdout:945: done (3 left) 2021-07-31T01:48:44.634 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 35 current snap is 84 2021-07-31T01:48:44.634 INFO:tasks.rados.rados.0.smithi045.stdout:948: seq_num 289 ranges {3751936=40960} 2021-07-31T01:48:44.656 INFO:tasks.rados.rados.0.smithi045.stdout:948: writing smithi04569269-35 from 3751936 to 3792896 tid 1 2021-07-31T01:48:44.656 INFO:tasks.rados.rados.0.smithi045.stdout:949: snap_create 2021-07-31T01:48:44.656 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 6 v 520 (ObjNum 215 snap 56 seq_num 215) dirty exists 2021-07-31T01:48:44.656 INFO:tasks.rados.rados.0.smithi045.stdout:943: finishing write tid 3 to smithi04569269-2 2021-07-31T01:48:44.657 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 2 v 260 (ObjNum 287 snap 84 seq_num 287) dirty exists 2021-07-31T01:48:44.657 INFO:tasks.rados.rados.0.smithi045.stdout:943: left oid 2 (ObjNum 287 snap 84 seq_num 287) 2021-07-31T01:48:44.657 INFO:tasks.rados.rados.0.smithi045.stdout:947: finishing write tid 1 to smithi04569269-36 2021-07-31T01:48:44.657 INFO:tasks.rados.rados.0.smithi045.stdout:947: finishing write tid 2 to smithi04569269-36 2021-07-31T01:48:44.658 INFO:tasks.rados.rados.0.smithi045.stdout:947: finishing write tid 3 to smithi04569269-36 2021-07-31T01:48:44.658 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 36 v 660 (ObjNum 288 snap 84 seq_num 288) dirty exists 2021-07-31T01:48:44.658 INFO:tasks.rados.rados.0.smithi045.stdout:947: left oid 36 (ObjNum 288 snap 84 seq_num 288) 2021-07-31T01:48:44.659 INFO:tasks.rados.rados.0.smithi045.stdout:948: finishing write tid 1 to smithi04569269-35 2021-07-31T01:48:44.659 INFO:tasks.rados.rados.0.smithi045.stdout:948: finishing write tid 2 to smithi04569269-35 2021-07-31T01:48:44.659 INFO:tasks.rados.rados.0.smithi045.stdout:948: finishing write tid 3 to smithi04569269-35 2021-07-31T01:48:44.660 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 35 v 663 (ObjNum 289 snap 84 seq_num 289) dirty exists 2021-07-31T01:48:44.660 INFO:tasks.rados.rados.0.smithi045.stdout:948: left oid 35 (ObjNum 289 snap 84 seq_num 289) 2021-07-31T01:48:45.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:44 smithi154 conmon[34553]: cluster 2021-07-31T01:48:43.607402+0000 mon.a (mon.0) 864 : cluster [DBG] 2021-07-31T01:48:45.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:44 smithi154 conmon[34553]: osdmap e224: 8 total, 8 up, 8 in 2021-07-31T01:48:45.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:44 smithi154 conmon[34553]: cluster 2021-07-31T01:48:43.811890+0000 mgr.x (mgr.24542) 67 : cluster [DBG] pgmap v116: 25 pgs: 25 active+clean; 141 MiB data, 640 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:48:45.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:44 smithi045 conmon[33655]: cluster 2021-07-31T01:48:43.607402+0000 mon.a (mon.0) 864 : cluster 2021-07-31T01:48:45.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:44 smithi045 conmon[33655]: [DBG] osdmap e224: 8 total, 8 up, 8 in 2021-07-31T01:48:45.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:44 smithi045 conmon[33655]: cluster 2021-07-31T01:48:43.811890+0000 mgr.x (mgr.24542) 67 : cluster [DBG] pgmap v116: 25 pgs: 25 active+clean; 141 MiB data, 640 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:48:45.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:44 smithi045 conmon[39718]: cluster 2021-07-31T01:48:43 2021-07-31T01:48:45.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:44 smithi045 conmon[39718]: .607402+0000 mon.a (mon.0) 864 : cluster [DBG] osdmap e224: 8 total, 8 up, 8 in 2021-07-31T01:48:45.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:44 smithi045 conmon[39718]: cluster 2021-07-31T01:48:45.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:44 smithi045 conmon[39718]: 2021-07-31T 2021-07-31T01:48:45.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:44 smithi045 conmon[39718]: 01:48:43.811890+0000 mgr.x (mgr.24542) 67 : cluster 2021-07-31T01:48:45.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:44 smithi045 conmon[39718]: [DBG] pgmap v116: 25 pgs: 25 active+clean; 141 MiB data, 640 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:48:45.623 INFO:tasks.rados.rados.0.smithi045.stdout:943: done (4 left) 2021-07-31T01:48:45.623 INFO:tasks.rados.rados.0.smithi045.stdout:944: done (3 left) 2021-07-31T01:48:45.623 INFO:tasks.rados.rados.0.smithi045.stdout:947: done (2 left) 2021-07-31T01:48:45.623 INFO:tasks.rados.rados.0.smithi045.stdout:948: done (1 left) 2021-07-31T01:48:45.624 INFO:tasks.rados.rados.0.smithi045.stdout:949: done (0 left) 2021-07-31T01:48:45.624 INFO:tasks.rados.rados.0.smithi045.stdout:950: read oid 5 snap 71 2021-07-31T01:48:45.624 INFO:tasks.rados.rados.0.smithi045.stdout:950: expect (ObjNum 158 snap 37 seq_num 158) 2021-07-31T01:48:45.624 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 15 current snap is 85 2021-07-31T01:48:45.625 INFO:tasks.rados.rados.0.smithi045.stdout:951: seq_num 290 ranges {7618560=57344} 2021-07-31T01:48:45.665 INFO:tasks.rados.rados.0.smithi045.stdout:951: writing smithi04569269-15 from 7618560 to 7675904 tid 1 2021-07-31T01:48:45.665 INFO:tasks.rados.rados.0.smithi045.stdout:952: read oid 31 snap -1 2021-07-31T01:48:45.666 INFO:tasks.rados.rados.0.smithi045.stdout:952: expect deleted 2021-07-31T01:48:45.666 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 23 current snap is 85 2021-07-31T01:48:45.666 INFO:tasks.rados.rados.0.smithi045.stdout:953: seq_num 291 ranges {0=81920} 2021-07-31T01:48:45.666 INFO:tasks.rados.rados.0.smithi045.stdout:953: writing smithi04569269-23 from 0 to 81920 tid 1 2021-07-31T01:48:45.666 INFO:tasks.rados.rados.0.smithi045.stdout:954: snap_remove snap 84 2021-07-31T01:48:46.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:45 smithi154 conmon[34553]: cluster 2021-07-31T01:48:44.616801+0000 mon.a (mon.0) 865 : cluster [DBG] osdmap e225: 8 total, 8 up, 8 in 2021-07-31T01:48:46.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:45 smithi045 conmon[39718]: cluster 2021-07-31T01:48:44.616801+0000 mon.a ( 2021-07-31T01:48:46.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:45 smithi045 conmon[39718]: mon.0) 865 : cluster [DBG] osdmap e225: 8 total, 8 up, 8 in 2021-07-31T01:48:46.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:45 smithi045 conmon[33655]: cluster 2021-07-31T01:48:44.616801+0000 mon.a (mon.0) 865 : cluster [DBG] 2021-07-31T01:48:46.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:45 smithi045 conmon[33655]: osdmap e225: 8 total, 8 up, 8 in 2021-07-31T01:48:46.631 INFO:tasks.rados.rados.0.smithi045.stdout:954: done (4 left) 2021-07-31T01:48:46.631 INFO:tasks.rados.rados.0.smithi045.stdout:955: snap_create 2021-07-31T01:48:46.631 INFO:tasks.rados.rados.0.smithi045.stdout:950: expect (ObjNum 158 snap 37 seq_num 158) 2021-07-31T01:48:46.639 INFO:tasks.rados.rados.0.smithi045.stdout:953: finishing write tid 1 to smithi04569269-23 2021-07-31T01:48:46.639 INFO:tasks.rados.rados.0.smithi045.stdout:953: finishing write tid 2 to smithi04569269-23 2021-07-31T01:48:46.639 INFO:tasks.rados.rados.0.smithi045.stdout:951: finishing write tid 1 to smithi04569269-15 2021-07-31T01:48:46.640 INFO:tasks.rados.rados.0.smithi045.stdout:951: finishing write tid 2 to smithi04569269-15 2021-07-31T01:48:46.640 INFO:tasks.rados.rados.0.smithi045.stdout:953: finishing write tid 3 to smithi04569269-23 2021-07-31T01:48:46.640 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 23 v 519 (ObjNum 291 snap 85 seq_num 291) dirty exists 2021-07-31T01:48:46.640 INFO:tasks.rados.rados.0.smithi045.stdout:953: left oid 23 (ObjNum 291 snap 85 seq_num 291) 2021-07-31T01:48:46.640 INFO:tasks.rados.rados.0.smithi045.stdout:951: finishing write tid 3 to smithi04569269-15 2021-07-31T01:48:46.641 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 15 v 282 (ObjNum 290 snap 85 seq_num 290) dirty exists 2021-07-31T01:48:46.641 INFO:tasks.rados.rados.0.smithi045.stdout:951: left oid 15 (ObjNum 290 snap 85 seq_num 290) 2021-07-31T01:48:47.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:46 smithi154 conmon[34553]: cluster 2021-07-31T01:48:45.621915+0000 mon.a (mon.0) 866 : cluster [DBG] osdmap e226: 8 total, 8 up, 8 in 2021-07-31T01:48:47.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:46 smithi154 conmon[34553]: cluster 2021-07-31T01:48:45.812391+0000 mgr.x (mgr.24542) 68 : cluster [DBG] pgmap v119: 25 pgs: 25 active+clean; 157 MiB data, 677 MiB used, 715 GiB / 715 GiB avail; 25 MiB/s rd, 6.7 MiB/s wr, 25 op/s 2021-07-31T01:48:47.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:46 smithi045 conmon[39718]: cluster 2021-07-31T01:48 2021-07-31T01:48:47.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:46 smithi045 conmon[39718]: :45.621915+0000 mon.a (mon.0) 866 : cluster [DBG] osdmap e226: 8 total, 8 up, 8 in 2021-07-31T01:48:47.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:46 smithi045 conmon[39718]: cluster 2021-07-31T01:48:47.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:46 smithi045 conmon[39718]: 2021-07-31T01:48: 2021-07-31T01:48:47.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:46 smithi045 conmon[39718]: 45.812391+0000 mgr.x (mgr.24542) 68 : cluster [DBG] 2021-07-31T01:48:47.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:46 smithi045 conmon[39718]: pgmap v119: 25 pgs: 25 active+clean; 157 MiB data, 677 MiB used, 715 GiB / 715 GiB avail; 25 MiB/s rd, 6.7 MiB/s wr, 25 op/s 2021-07-31T01:48:47.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:46 smithi045 conmon[33655]: cluster 2021-07-31T01:48:45.621915+0000 mon.a (mon. 2021-07-31T01:48:47.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:46 smithi045 conmon[33655]: 0) 866 : cluster [DBG] osdmap e226: 8 total, 8 up, 8 in 2021-07-31T01:48:47.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:46 smithi045 conmon[33655]: cluster 2021-07-31T01:48:45.812391+0000 mgr.x (mgr.24542) 68 : cluster [DBG] pgmap v119: 25 pgs: 25 active+clean; 157 MiB data, 677 MiB used, 715 GiB / 715 GiB avail; 25 MiB/s rd, 6.7 MiB/s wr, 25 op/s 2021-07-31T01:48:47.631 INFO:tasks.rados.rados.0.smithi045.stdout:950: done (4 left) 2021-07-31T01:48:47.632 INFO:tasks.rados.rados.0.smithi045.stdout:951: done (3 left) 2021-07-31T01:48:47.633 INFO:tasks.rados.rados.0.smithi045.stdout:952: done (2 left) 2021-07-31T01:48:47.633 INFO:tasks.rados.rados.0.smithi045.stdout:953: done (1 left) 2021-07-31T01:48:47.633 INFO:tasks.rados.rados.0.smithi045.stdout:955: done (0 left) 2021-07-31T01:48:47.633 INFO:tasks.rados.rados.0.smithi045.stdout:956: rollback oid 2 current snap is 86 2021-07-31T01:48:47.634 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 2 to 85 2021-07-31T01:48:47.634 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 38 current snap is 86 2021-07-31T01:48:47.634 INFO:tasks.rados.rados.0.smithi045.stdout:957: seq_num 292 ranges {1695744=98304} 2021-07-31T01:48:47.643 INFO:tasks.rados.rados.0.smithi045.stdout:957: writing smithi04569269-38 from 1695744 to 1794048 tid 1 2021-07-31T01:48:47.643 INFO:tasks.rados.rados.0.smithi045.stdout:958: read oid 40 snap -1 2021-07-31T01:48:47.643 INFO:tasks.rados.rados.0.smithi045.stdout:958: expect (ObjNum 274 snap 81 seq_num 274) 2021-07-31T01:48:47.643 INFO:tasks.rados.rados.0.smithi045.stdout:959: rmattr oid 10 current snap is 86 2021-07-31T01:48:47.644 INFO:tasks.rados.rados.0.smithi045.stdout:959: done (3 left) 2021-07-31T01:48:47.644 INFO:tasks.rados.rados.0.smithi045.stdout:960: read oid 45 snap -1 2021-07-31T01:48:47.644 INFO:tasks.rados.rados.0.smithi045.stdout:960: expect deleted 2021-07-31T01:48:47.644 INFO:tasks.rados.rados.0.smithi045.stdout:961: copy_from oid 9 from oid 18 current snap is 86 2021-07-31T01:48:47.645 INFO:tasks.rados.rados.0.smithi045.stdout:956: finishing rollback tid 0 to smithi04569269-2 2021-07-31T01:48:47.645 INFO:tasks.rados.rados.0.smithi045.stdout:956: finishing rollback tid 1 to smithi04569269-2 2021-07-31T01:48:47.645 INFO:tasks.rados.rados.0.smithi045.stdout:962: snap_remove snap 75 2021-07-31T01:48:48.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:47 smithi154 conmon[34553]: cluster 2021-07-31T01:48:46.630334+0000 mon.a (mon 2021-07-31T01:48:48.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:47 smithi154 conmon[34553]: .0) 867 : cluster [DBG] osdmap e227: 8 total, 8 up, 8 in 2021-07-31T01:48:48.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:47 smithi045 conmon[39718]: cluster 2021-07-31T01:48:48.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:47 smithi045 conmon[39718]: 2021-07-31T01:48:46. 2021-07-31T01:48:48.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:47 smithi045 conmon[39718]: 630334+0000 mon.a (mon.0) 867 : cluster [DBG] osdmap e227: 8 total, 8 up, 8 in 2021-07-31T01:48:48.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:47 smithi045 conmon[33655]: cluster 2021-07-31T01:48:46.630334+0000 mon.a (mon.0) 867 : cluster [DBG] osdmap e227: 8 total, 8 up, 8 in 2021-07-31T01:48:48.647 INFO:tasks.rados.rados.0.smithi045.stdout:962: done (5 left) 2021-07-31T01:48:48.647 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 47 current snap is 86 2021-07-31T01:48:48.648 INFO:tasks.rados.rados.0.smithi045.stdout:963: seq_num 293 ranges {5996544=114688} 2021-07-31T01:48:48.681 INFO:tasks.rados.rados.0.smithi045.stdout:963: writing smithi04569269-47 from 5996544 to 6111232 tid 1 2021-07-31T01:48:48.681 INFO:tasks.rados.rados.0.smithi045.stdout:964: snap_create 2021-07-31T01:48:48.682 INFO:tasks.rados.rados.0.smithi045.stdout:956: finishing rollback tid 2 to smithi04569269-2 2021-07-31T01:48:48.682 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 2 v 263 (ObjNum 287 snap 84 seq_num 287) dirty exists 2021-07-31T01:48:48.682 INFO:tasks.rados.rados.0.smithi045.stdout:957: finishing write tid 1 to smithi04569269-38 2021-07-31T01:48:48.682 INFO:tasks.rados.rados.0.smithi045.stdout:957: finishing write tid 2 to smithi04569269-38 2021-07-31T01:48:48.683 INFO:tasks.rados.rados.0.smithi045.stdout:957: finishing write tid 3 to smithi04569269-38 2021-07-31T01:48:48.683 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 38 v 521 (ObjNum 292 snap 86 seq_num 292) dirty exists 2021-07-31T01:48:48.683 INFO:tasks.rados.rados.0.smithi045.stdout:957: left oid 38 (ObjNum 292 snap 86 seq_num 292) 2021-07-31T01:48:48.683 INFO:tasks.rados.rados.0.smithi045.stdout:961: finishing copy_from to smithi04569269-9 2021-07-31T01:48:48.683 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 9 v 217 (ObjNum 172 snap 41 seq_num 172) dirty exists 2021-07-31T01:48:48.684 INFO:tasks.rados.rados.0.smithi045.stdout:961: finishing copy_from racing read to smithi04569269-9 2021-07-31T01:48:48.684 INFO:tasks.rados.rados.0.smithi045.stdout:958: expect (ObjNum 274 snap 81 seq_num 274) 2021-07-31T01:48:48.884 INFO:tasks.rados.rados.0.smithi045.stdout:963: finishing write tid 1 to smithi04569269-47 2021-07-31T01:48:48.884 INFO:tasks.rados.rados.0.smithi045.stdout:963: finishing write tid 2 to smithi04569269-47 2021-07-31T01:48:48.884 INFO:tasks.rados.rados.0.smithi045.stdout:963: finishing write tid 3 to smithi04569269-47 2021-07-31T01:48:48.885 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 47 v 175 (ObjNum 293 snap 86 seq_num 293) dirty exists 2021-07-31T01:48:48.885 INFO:tasks.rados.rados.0.smithi045.stdout:963: left oid 47 (ObjNum 293 snap 86 seq_num 293) 2021-07-31T01:48:49.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:48 smithi154 conmon[34553]: cluster 2021-07-31T01:48:47.631246+0000 mon.a (mon.0) 868 : cluster 2021-07-31T01:48:49.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:48 smithi154 conmon[34553]: [DBG] osdmap e228: 8 total, 8 up, 8 in 2021-07-31T01:48:49.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:48 smithi154 conmon[34553]: cluster 2021-07-31T01:48:47.812850+0000 mgr.x (mgr.24542) 69 : cluster [DBG] pgmap v122: 25 pgs: 25 active+clean; 157 MiB data, 677 MiB used, 715 GiB / 715 GiB avail; 25 MiB/s rd, 6.7 MiB/s wr, 25 op/s 2021-07-31T01:48:49.005 INFO:journalctl@ceph.osd.6.smithi154.stdout:Jul 31 01:48:48 smithi154 conmon[45576]: debug 2021-07-31T01:48:49.005 INFO:journalctl@ceph.osd.6.smithi154.stdout:Jul 31 01:48:48 smithi154 conmon[45576]: 2021-07-31T01:48:48.650+0000 7f47c10a5700 -1 osd.6 pg_epoch: 229 pg[3.as0( v 225'312 (0'0,225'312] local-lis/les=56/57 n=6 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [6,4,1]p6(0) r=0 lpr=56 crt=225'312 lcod 223'310 mlcod 223'310 active+clean+snaptrim trimq=[94~1] ps=[96~1,9a~1,a1~2,a7~1]] removing snap head 2021-07-31T01:48:49.005 INFO:journalctl@ceph.osd.6.smithi154.stdout:Jul 31 01:48:48 smithi154 conmon[45576]: 2021-07-31T01:48:49.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:48 smithi045 conmon[39718]: cluster 2021-07-31T01:48:47.631246+0000 mon.a (mon.0) 868 : cluster [DBG] osdmap e228: 8 total, 8 up, 8 in 2021-07-31T01:48:49.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:48 smithi045 conmon[39718]: cluster 2021-07-31T01:48:47.812850+0000 mgr.x (mgr.24542) 69 : cluster [DBG] pgmap v122: 25 pgs: 25 active+clean; 157 MiB data, 677 MiB used, 715 GiB / 715 GiB avail; 25 MiB/s rd, 6.7 MiB/s wr, 25 op/s 2021-07-31T01:48:49.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:48 smithi045 conmon[33655]: cluster 2021-07-31T01:48:47.631246+0000 mon.a (mon.0 2021-07-31T01:48:49.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:48 smithi045 conmon[33655]: ) 868 : cluster [DBG] osdmap e228: 8 total, 8 up, 8 in 2021-07-31T01:48:49.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:48 smithi045 conmon[33655]: cluster 2021-07-31T01:48:47.812850+0000 mgr.x (mgr.24542) 69 : cluster [DBG] 2021-07-31T01:48:49.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:48 smithi045 conmon[33655]: pgmap v122: 25 pgs: 25 active+clean; 157 MiB data, 677 MiB used, 715 GiB / 715 GiB avail; 25 MiB/s rd, 6.7 MiB/s wr, 25 op/s 2021-07-31T01:48:49.647 INFO:tasks.rados.rados.0.smithi045.stdout:956: done (6 left) 2021-07-31T01:48:49.647 INFO:tasks.rados.rados.0.smithi045.stdout:957: done (5 left) 2021-07-31T01:48:49.647 INFO:tasks.rados.rados.0.smithi045.stdout:958: done (4 left) 2021-07-31T01:48:49.647 INFO:tasks.rados.rados.0.smithi045.stdout:960: done (3 left) 2021-07-31T01:48:49.648 INFO:tasks.rados.rados.0.smithi045.stdout:961: done (2 left) 2021-07-31T01:48:49.648 INFO:tasks.rados.rados.0.smithi045.stdout:963: done (1 left) 2021-07-31T01:48:49.648 INFO:tasks.rados.rados.0.smithi045.stdout:964: done (0 left) 2021-07-31T01:48:49.648 INFO:tasks.rados.rados.0.smithi045.stdout:965: snap_create 2021-07-31T01:48:50.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:49 smithi154 conmon[34553]: cluster 2021-07-31T01:48:48.645325+0000 mon.a (mon.0) 869 : cluster [DBG] osdmap e229: 8 total, 8 up, 8 in 2021-07-31T01:48:50.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:49 smithi045 conmon[33655]: cluster 2021-07-31T01:48:48.645325+0000 mon.a (mon.0 2021-07-31T01:48:50.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:49 smithi045 conmon[33655]: ) 869 : cluster [DBG] osdmap e229: 8 total, 8 up, 8 in 2021-07-31T01:48:50.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:49 smithi045 conmon[39718]: cluster 2021-07-31T01:48:48.645325+0000 mon.a (mon.0) 869 2021-07-31T01:48:50.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:49 smithi045 conmon[39718]: : cluster [DBG] osdmap e229: 8 total, 8 up, 8 in 2021-07-31T01:48:50.656 INFO:tasks.rados.rados.0.smithi045.stdout:965: done (0 left) 2021-07-31T01:48:50.657 INFO:tasks.rados.rados.0.smithi045.stdout:966: snap_remove snap 85 2021-07-31T01:48:51.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:50 smithi154 conmon[34553]: cluster 2021-07-31T01:48:49.646107+0000 mon.a (mon.0) 870 2021-07-31T01:48:51.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:50 smithi154 conmon[34553]: : cluster [DBG] osdmap e230: 8 total, 8 up, 8 in 2021-07-31T01:48:51.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:50 smithi154 conmon[34553]: cluster 2021-07-31T01:48:49.813392+0000 mgr.x (mgr.24542) 70 : cluster [DBG] pgmap v125: 25 pgs: 25 active+clean; 154 MiB data, 695 MiB used, 715 GiB / 715 GiB avail; 3.8 MiB/s rd, 222 KiB/s wr, 15 op/s 2021-07-31T01:48:51.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:50 smithi045 conmon[39718]: cluster 2021-07-31T01:48:49.646107+0000 mon.a (mon 2021-07-31T01:48:51.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:50 smithi045 conmon[39718]: .0) 870 : cluster [DBG] osdmap e230: 8 total, 8 up, 8 in 2021-07-31T01:48:51.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:50 smithi045 conmon[39718]: cluster 2021-07-31T01:48:49.813392 2021-07-31T01:48:51.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:50 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 70 : cluster [DBG] pgmap v125: 25 pgs: 25 active+clean; 154 MiB data, 695 MiB used, 715 GiB / 715 GiB avail; 3.8 MiB/s rd, 222 KiB/s wr, 15 op/s 2021-07-31T01:48:51.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:50 smithi045 conmon[33655]: cluster 2021-07-31T01:48:49.646107+0000 mon.a (mon.0) 870 2021-07-31T01:48:51.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:50 smithi045 conmon[33655]: : cluster [DBG] osdmap e230: 8 total, 8 up, 8 in 2021-07-31T01:48:51.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:50 smithi045 conmon[33655]: cluster 2021-07-31T01:48:49.813392+0000 mgr.x (mgr.24542) 70 : cluster [DBG] pgmap v125: 25 pgs: 25 active+clean; 154 MiB data, 695 MiB used, 715 GiB / 715 GiB avail; 3.8 MiB/s rd, 222 KiB/s wr, 15 op/s 2021-07-31T01:48:51.514 INFO:tasks.rados.rados.0.smithi045.stdout:966: done (0 left) 2021-07-31T01:48:51.515 INFO:tasks.rados.rados.0.smithi045.stdout:967: snap_remove snap 87 2021-07-31T01:48:52.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:51 smithi154 conmon[34553]: cluster 2021-07-31T01:48:50.655725+0000 mon.a (mon.0 2021-07-31T01:48:52.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:51 smithi154 conmon[34553]: ) 871 : cluster [DBG] osdmap e231: 8 total, 8 up, 8 in 2021-07-31T01:48:52.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:51 smithi154 conmon[34553]: cluster 2021-07-31T01:48:51.513535+0000 mon.a (mon.0) 872 : cluster [DBG] osdmap e232: 8 total, 8 up, 8 in 2021-07-31T01:48:52.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:51 smithi045 conmon[39718]: cluster 2021-07-31T01:48: 2021-07-31T01:48:52.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:51 smithi045 conmon[39718]: 50.655725+0000 mon.a (mon.0) 871 : cluster [DBG] osdmap e231: 8 total, 8 up, 8 in 2021-07-31T01:48:52.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:51 smithi045 conmon[39718]: cluster 2021-07-31T01:48:52.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:51 smithi045 conmon[39718]: 2021-07-31T01:48:51.513535+0000 2021-07-31T01:48:52.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:51 smithi045 conmon[39718]: mon.a (mon.0) 872 : 2021-07-31T01:48:52.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:51 smithi045 conmon[39718]: cluster [DBG] osdmap e232: 8 total, 8 up, 8 in 2021-07-31T01:48:52.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:51 smithi045 conmon[33655]: cluster 2021-07-31T01:48:50.655725+0000 mon.a (mon.0) 2021-07-31T01:48:52.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:51 smithi045 conmon[33655]: 871 : cluster [DBG] osdmap e231: 8 total, 8 up, 8 in 2021-07-31T01:48:52.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:51 smithi045 conmon[33655]: cluster 2021-07-31T01:48: 2021-07-31T01:48:52.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:51 smithi045 conmon[33655]: 51.513535+0000 mon.a (mon.0) 872 : cluster [DBG] osdmap e232: 8 total, 8 up, 8 in 2021-07-31T01:48:52.523 INFO:tasks.rados.rados.0.smithi045.stdout:967: done (0 left) 2021-07-31T01:48:52.524 INFO:tasks.rados.rados.0.smithi045.stdout:968: delete oid 6 current snap is 88 2021-07-31T01:48:52.527 INFO:tasks.rados.rados.0.smithi045.stdout:968: done (0 left) 2021-07-31T01:48:52.528 INFO:tasks.rados.rados.0.smithi045.stdout:969: read oid 23 snap 83 2021-07-31T01:48:52.528 INFO:tasks.rados.rados.0.smithi045.stdout:969: expect deleted 2021-07-31T01:48:52.529 INFO:tasks.rados.rados.0.smithi045.stdout:970: delete oid 13 current snap is 88 2021-07-31T01:48:52.530 INFO:tasks.rados.rados.0.smithi045.stdout:970: done (1 left) 2021-07-31T01:48:52.530 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 25 current snap is 88 2021-07-31T01:48:52.531 INFO:tasks.rados.rados.0.smithi045.stdout:971: seq_num 296 ranges {3833856=163840} 2021-07-31T01:48:52.553 INFO:tasks.rados.rados.0.smithi045.stdout:971: writing smithi04569269-25 from 3833856 to 3997696 tid 1 2021-07-31T01:48:52.554 INFO:tasks.rados.rados.0.smithi045.stdout:969: done (1 left) 2021-07-31T01:48:52.554 INFO:tasks.rados.rados.0.smithi045.stdout:972: read oid 15 snap -1 2021-07-31T01:48:52.555 INFO:tasks.rados.rados.0.smithi045.stdout:972: expect (ObjNum 290 snap 85 seq_num 290) 2021-07-31T01:48:52.555 INFO:tasks.rados.rados.0.smithi045.stdout:973: read oid 27 snap 83 2021-07-31T01:48:52.556 INFO:tasks.rados.rados.0.smithi045.stdout:973: expect (ObjNum -1105582352 snap 21873 seq_num 3189582144) 2021-07-31T01:48:52.556 INFO:tasks.rados.rados.0.smithi045.stdout:974: rollback oid 38 current snap is 88 2021-07-31T01:48:52.557 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 38 to 83 2021-07-31T01:48:52.557 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 26 current snap is 88 2021-07-31T01:48:52.557 INFO:tasks.rados.rados.0.smithi045.stdout:975: seq_num 297 ranges {6037504=180224} 2021-07-31T01:48:52.588 INFO:tasks.rados.rados.0.smithi045.stdout:975: writing smithi04569269-26 from 6037504 to 6217728 tid 1 2021-07-31T01:48:52.588 INFO:tasks.rados.rados.0.smithi045.stdout:976: snap_create 2021-07-31T01:48:52.588 INFO:tasks.rados.rados.0.smithi045.stdout:973: expect (ObjNum -1105582352 snap 21873 seq_num 3189582144) 2021-07-31T01:48:52.591 INFO:tasks.rados.rados.0.smithi045.stdout:974: finishing rollback tid 0 to smithi04569269-38 2021-07-31T01:48:52.592 INFO:tasks.rados.rados.0.smithi045.stdout:971: finishing write tid 1 to smithi04569269-25 2021-07-31T01:48:52.592 INFO:tasks.rados.rados.0.smithi045.stdout:974: finishing rollback tid 1 to smithi04569269-38 2021-07-31T01:48:52.592 INFO:tasks.rados.rados.0.smithi045.stdout:974: finishing rollback tid 2 to smithi04569269-38 2021-07-31T01:48:52.592 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 38 v 524 (ObjNum 244 snap 67 seq_num 244) dirty exists 2021-07-31T01:48:52.593 INFO:tasks.rados.rados.0.smithi045.stdout:971: finishing write tid 2 to smithi04569269-25 2021-07-31T01:48:52.593 INFO:tasks.rados.rados.0.smithi045.stdout:971: finishing write tid 3 to smithi04569269-25 2021-07-31T01:48:52.593 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 25 v 177 (ObjNum 296 snap 88 seq_num 296) dirty exists 2021-07-31T01:48:52.593 INFO:tasks.rados.rados.0.smithi045.stdout:971: left oid 25 (ObjNum 296 snap 88 seq_num 296) 2021-07-31T01:48:52.594 INFO:tasks.rados.rados.0.smithi045.stdout:972: expect (ObjNum 290 snap 85 seq_num 290) 2021-07-31T01:48:53.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:52 smithi154 conmon[34553]: cluster 2021-07-31T01:48:51.813784+0000 mgr.x (mgr.24542) 71 : cluster [DBG] pgmap v128: 25 pgs: 25 active+clean; 154 MiB data, 695 MiB used, 715 GiB / 715 GiB avail; 3.8 MiB/s rd, 222 KiB/s wr, 15 op/s 2021-07-31T01:48:53.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:52 smithi154 conmon[34553]: cluster 2021-07-31T01 2021-07-31T01:48:53.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:52 smithi154 conmon[34553]: :48:52.522547+0000 mon.a (mon.0) 873 : cluster [DBG] osdmap e233: 8 total, 8 up, 8 in 2021-07-31T01:48:53.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:52 smithi045 conmon[39718]: cluster 2021-07-31T01:48:51.813784+0000 2021-07-31T01:48:53.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:52 smithi045 conmon[39718]: mgr.x (mgr.24542) 71 : cluster [DBG] pgmap v128: 25 pgs: 25 active+clean; 154 MiB data, 695 MiB used, 715 GiB / 715 GiB avail; 3.8 MiB/s rd, 222 KiB/s wr, 15 op/s 2021-07-31T01:48:53.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:52 smithi045 conmon[39718]: cluster 2021-07-31T01: 2021-07-31T01:48:53.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:52 smithi045 conmon[39718]: 48:52.522547+0000 mon.a (mon.0) 873 : cluster [DBG] osdmap e233: 8 total, 8 up, 8 in 2021-07-31T01:48:53.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:52 smithi045 conmon[33655]: cluster 2021-07-31T01:48: 2021-07-31T01:48:53.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:52 smithi045 conmon[33655]: 51.813784+0000 mgr.x (mgr.24542) 71 : cluster [DBG] pgmap v128: 25 pgs: 25 active+clean; 154 MiB data, 695 MiB used, 715 GiB / 715 GiB avail; 3.8 MiB/s rd, 222 KiB/s wr, 15 op/s 2021-07-31T01:48:53.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:52 smithi045 conmon[33655]: cluster 2021-07-31T01:48:52.522547+0000 mon.a ( 2021-07-31T01:48:53.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:52 smithi045 conmon[33655]: mon.0) 873 : cluster [DBG] osdmap e233: 8 total, 8 up, 8 in 2021-07-31T01:48:53.078 INFO:tasks.rados.rados.0.smithi045.stdout:975: finishing write tid 1 to smithi04569269-26 2021-07-31T01:48:53.078 INFO:tasks.rados.rados.0.smithi045.stdout:975: finishing write tid 2 to smithi04569269-26 2021-07-31T01:48:53.078 INFO:tasks.rados.rados.0.smithi045.stdout:975: finishing write tid 3 to smithi04569269-26 2021-07-31T01:48:53.079 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 26 v 206 (ObjNum 297 snap 88 seq_num 297) dirty exists 2021-07-31T01:48:53.079 INFO:tasks.rados.rados.0.smithi045.stdout:975: left oid 26 (ObjNum 297 snap 88 seq_num 297) 2021-07-31T01:48:53.525 INFO:tasks.rados.rados.0.smithi045.stdout:971: done (5 left) 2021-07-31T01:48:53.525 INFO:tasks.rados.rados.0.smithi045.stdout:972: done (4 left) 2021-07-31T01:48:53.525 INFO:tasks.rados.rados.0.smithi045.stdout:973: done (3 left) 2021-07-31T01:48:53.525 INFO:tasks.rados.rados.0.smithi045.stdout:974: done (2 left) 2021-07-31T01:48:53.526 INFO:tasks.rados.rados.0.smithi045.stdout:975: done (1 left) 2021-07-31T01:48:53.526 INFO:tasks.rados.rados.0.smithi045.stdout:976: done (0 left) 2021-07-31T01:48:53.526 INFO:tasks.rados.rados.0.smithi045.stdout:977: rollback oid 29 current snap is 89 2021-07-31T01:48:53.526 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 29 to 88 2021-07-31T01:48:53.527 INFO:tasks.rados.rados.0.smithi045.stdout:978: snap_remove snap 83 2021-07-31T01:48:53.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:48:53 smithi045 conmon[61210]: level=error ts=2021-07-31T01:48:53.510Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:48:53.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:48:53 smithi045 conmon[61210]: level=error ts=2021-07-31T01:48:53.511Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:48:53.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:48:53 smithi045 conmon[61210]: level=error ts=2021-07-31T01:48:53.511Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T01:48:54.531 INFO:tasks.rados.rados.0.smithi045.stdout:978: done (1 left) 2021-07-31T01:48:54.532 INFO:tasks.rados.rados.0.smithi045.stdout:979: setattr oid 26 current snap is 89 2021-07-31T01:48:54.532 INFO:tasks.rados.rados.0.smithi045.stdout:977: finishing rollback tid 0 to smithi04569269-29 2021-07-31T01:48:54.532 INFO:tasks.rados.rados.0.smithi045.stdout:977: finishing rollback tid 1 to smithi04569269-29 2021-07-31T01:48:54.532 INFO:tasks.rados.rados.0.smithi045.stdout:977: finishing rollback tid 2 to smithi04569269-29 2021-07-31T01:48:54.533 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 29 v 182 (ObjNum 237 snap 66 seq_num 237) dirty exists 2021-07-31T01:48:54.533 INFO:tasks.rados.rados.0.smithi045.stdout:977: done (1 left) 2021-07-31T01:48:54.533 INFO:tasks.rados.rados.0.smithi045.stdout:980: delete oid 11 current snap is 89 2021-07-31T01:48:54.539 INFO:tasks.rados.rados.0.smithi045.stdout:980: done (1 left) 2021-07-31T01:48:54.539 INFO:tasks.rados.rados.0.smithi045.stdout:981: snap_remove snap 88 2021-07-31T01:48:55.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:54 smithi154 conmon[34553]: cluster 2021-07-31T01:48:53.524028+0000 mon.a (mon.0) 874 : cluster [DBG] 2021-07-31T01:48:55.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:54 smithi154 conmon[34553]: osdmap e234: 8 total, 8 up, 8 in 2021-07-31T01:48:55.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:54 smithi154 conmon[34553]: cluster 2021-07-31T01:48:53.814185+0000 mgr.x (mgr.24542) 72 : cluster [DBG] pgmap v131: 25 pgs: 25 active+clean; 154 MiB data, 695 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:48:55.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:54 smithi045 conmon[33655]: cluster 2021-07-31T01:48:53.524028+0000 2021-07-31T01:48:55.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:54 smithi045 conmon[33655]: mon.a (mon.0) 874 : cluster [DBG] osdmap e234: 8 total, 8 up, 8 in 2021-07-31T01:48:55.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:54 smithi045 conmon[33655]: cluster 2021-07-31T01:48:53.814185+0000 mgr.x (mgr.24542 2021-07-31T01:48:55.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:54 smithi045 conmon[33655]: ) 72 : cluster [DBG] pgmap v131: 25 pgs: 25 active+clean; 154 MiB data, 695 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:48:55.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:54 smithi045 conmon[39718]: cluster 2021-07-31T01:48:53.524028+0000 mon.a (mon.0) 874 : cluster 2021-07-31T01:48:55.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:54 smithi045 conmon[39718]: [DBG] osdmap e234: 8 total, 8 up, 8 in 2021-07-31T01:48:55.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:54 smithi045 conmon[39718]: cluster 2021-07-31T01:48:53.814185+0000 mgr.x (mgr.24542) 72 : cluster [DBG] 2021-07-31T01:48:55.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:54 smithi045 conmon[39718]: pgmap v131: 25 pgs: 25 active+clean; 154 MiB data, 695 MiB used, 715 GiB / 715 GiB avail 2021-07-31T01:48:55.535 INFO:tasks.rados.rados.0.smithi045.stdout:981: done (1 left) 2021-07-31T01:48:55.536 INFO:tasks.rados.rados.0.smithi045.stdout:982: delete oid 11 current snap is 89 2021-07-31T01:48:55.536 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 26 v 207 (ObjNum 297 snap 88 seq_num 297) dirty exists 2021-07-31T01:48:55.542 INFO:tasks.rados.rados.0.smithi045.stdout:979: done (1 left) 2021-07-31T01:48:55.543 INFO:tasks.rados.rados.0.smithi045.stdout:982: done (0 left) 2021-07-31T01:48:55.543 INFO:tasks.rados.rados.0.smithi045.stdout:983: snap_remove snap 86 2021-07-31T01:48:56.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:55 smithi154 conmon[34553]: cluster 2021-07-31T01:48:54.529625+0000 mon.a (mon.0) 875 : cluster [DBG] osdmap e235: 8 total, 8 up, 8 in 2021-07-31T01:48:56.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:55 smithi045 conmon[33655]: cluster 2021-07-31T01:48:54.529625+0000 mon.a (mon.0 2021-07-31T01:48:56.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:55 smithi045 conmon[33655]: ) 875 : cluster [DBG] osdmap e235: 8 total, 8 up, 8 in 2021-07-31T01:48:56.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:55 smithi045 conmon[39718]: cluster 2021-07-31T01:48:54.529625+0000 2021-07-31T01:48:56.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:55 smithi045 conmon[39718]: mon.a (mon.0) 875 : cluster [DBG] osdmap e235: 8 total, 8 up, 8 in 2021-07-31T01:48:56.541 INFO:tasks.rados.rados.0.smithi045.stdout:983: done (0 left) 2021-07-31T01:48:56.541 INFO:tasks.rados.rados.0.smithi045.stdout:984: read oid 6 snap -1 2021-07-31T01:48:56.542 INFO:tasks.rados.rados.0.smithi045.stdout:984: expect deleted 2021-07-31T01:48:56.542 INFO:tasks.rados.rados.0.smithi045.stdout:985: copy_from oid 50 from oid 30 current snap is 89 2021-07-31T01:48:56.542 INFO:tasks.rados.rados.0.smithi045.stdout:986: rmattr oid 49 current snap is 89 2021-07-31T01:48:56.542 INFO:tasks.rados.rados.0.smithi045.stdout:986: done (2 left) 2021-07-31T01:48:56.544 INFO:tasks.rados.rados.0.smithi045.stdout:987: delete oid 20 current snap is 89 2021-07-31T01:48:56.551 INFO:tasks.rados.rados.0.smithi045.stdout:984: done (2 left) 2021-07-31T01:48:56.551 INFO:tasks.rados.rados.0.smithi045.stdout:987: done (1 left) 2021-07-31T01:48:56.551 INFO:tasks.rados.rados.0.smithi045.stdout:988: delete oid 45 current snap is 89 2021-07-31T01:48:56.557 INFO:tasks.rados.rados.0.smithi045.stdout:985: finishing copy_from to smithi04569269-50 2021-07-31T01:48:56.558 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 50 v 302 (ObjNum 76 snap 11 seq_num 76) dirty exists 2021-07-31T01:48:56.559 INFO:tasks.rados.rados.0.smithi045.stdout:985: finishing copy_from racing read to smithi04569269-50 2021-07-31T01:48:56.559 INFO:tasks.rados.rados.0.smithi045.stdout:985: done (1 left) 2021-07-31T01:48:56.560 INFO:tasks.rados.rados.0.smithi045.stdout:988: done (0 left) 2021-07-31T01:48:56.560 INFO:tasks.rados.rados.0.smithi045.stdout:989: read oid 23 snap 71 2021-07-31T01:48:56.560 INFO:tasks.rados.rados.0.smithi045.stdout:989: expect deleted 2021-07-31T01:48:56.560 INFO:tasks.rados.rados.0.smithi045.stdout:990: copy_from oid 20 from oid 30 current snap is 89 2021-07-31T01:48:56.561 INFO:tasks.rados.rados.0.smithi045.stdout:991: copy_from oid 7 from oid 1 current snap is 89 2021-07-31T01:48:56.561 INFO:tasks.rados.rados.0.smithi045.stdout:992: rollback oid 6 current snap is 89 2021-07-31T01:48:56.561 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 6 to 71 2021-07-31T01:48:56.561 INFO:tasks.rados.rados.0.smithi045.stdout:993: read oid 50 snap -1 2021-07-31T01:48:56.561 INFO:tasks.rados.rados.0.smithi045.stdout:993: expect (ObjNum 76 snap 11 seq_num 76) 2021-07-31T01:48:56.562 INFO:tasks.rados.rados.0.smithi045.stdout:994: snap_create 2021-07-31T01:48:56.563 INFO:tasks.rados.rados.0.smithi045.stdout:991: finishing copy_from to smithi04569269-7 2021-07-31T01:48:56.563 INFO:tasks.rados.rados.0.smithi045.stdout:991: got expected ENOENT (src dne) 2021-07-31T01:48:56.564 INFO:tasks.rados.rados.0.smithi045.stdout:991: finishing copy_from racing read to smithi04569269-7 2021-07-31T01:48:56.564 INFO:tasks.rados.rados.0.smithi045.stdout:992: finishing rollback tid 1 to smithi04569269-6 2021-07-31T01:48:56.564 INFO:tasks.rados.rados.0.smithi045.stdout:992: finishing rollback tid 2 to smithi04569269-6 2021-07-31T01:48:56.565 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 6 v 524 (ObjNum 215 snap 56 seq_num 215) dirty exists 2021-07-31T01:48:56.568 INFO:tasks.rados.rados.0.smithi045.stdout:990: finishing copy_from to smithi04569269-20 2021-07-31T01:48:56.568 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 20 v 302 (ObjNum 76 snap 11 seq_num 76) dirty exists 2021-07-31T01:48:56.569 INFO:tasks.rados.rados.0.smithi045.stdout:990: finishing copy_from racing read to smithi04569269-20 2021-07-31T01:48:56.569 INFO:tasks.rados.rados.0.smithi045.stdout:993: expect (ObjNum 76 snap 11 seq_num 76) 2021-07-31T01:48:57.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:56 smithi154 conmon[34553]: cluster 2021-07-31T01:48:55.534734+0000 mon.a (mon.0) 876 : cluster 2021-07-31T01:48:57.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:56 smithi154 conmon[34553]: [DBG] osdmap e236: 8 total, 8 up, 8 in 2021-07-31T01:48:57.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:56 smithi154 conmon[34553]: cluster 2021-07-31T01:48:55.814631+0000 mgr.x (mgr.24542) 73 : cluster [DBG] pgmap v134: 25 pgs: 25 active+clean; 154 MiB data, 702 MiB used, 715 GiB / 715 GiB avail; 5.5 MiB/s rd, 84 KiB/s wr, 4 op/s 2021-07-31T01:48:57.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:56 smithi045 conmon[33655]: cluster 2021-07-31T01:48:55.534734+0000 mon.a (mon.0) 876 2021-07-31T01:48:57.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:56 smithi045 conmon[33655]: : cluster [DBG] osdmap e236: 8 total, 8 up, 8 in 2021-07-31T01:48:57.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:56 smithi045 conmon[33655]: cluster 2021-07-31T01:48:55.814631+0000 mgr.x (mgr.24542) 73 : cluster [DBG] pgmap v134: 25 pgs: 25 active+clean; 154 MiB data, 702 MiB used, 715 GiB / 715 GiB avail; 5.5 MiB/s rd, 84 KiB/s wr, 4 op/s 2021-07-31T01:48:57.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:56 smithi045 conmon[39718]: cluster 2021-07-31T01:48:55.534734+0000 2021-07-31T01:48:57.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:56 smithi045 conmon[39718]: mon.a (mon.0) 876 : cluster [DBG] osdmap e236: 8 total, 8 up, 8 in 2021-07-31T01:48:57.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:56 smithi045 conmon[39718]: cluster 2021-07-31T01:48:57.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:56 smithi045 conmon[39718]: 2021-07-31T01:48: 2021-07-31T01:48:57.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:56 smithi045 conmon[39718]: 55.814631+0000 mgr.x (mgr.24542) 73 : cluster [DBG] 2021-07-31T01:48:57.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:56 smithi045 conmon[39718]: pgmap v134: 25 pgs: 25 active+clean; 154 MiB data, 702 MiB used, 715 GiB / 715 GiB avail; 5.5 MiB/s rd, 84 KiB/s wr, 4 op/s 2021-07-31T01:48:57.546 INFO:tasks.rados.rados.0.smithi045.stdout:989: done (5 left) 2021-07-31T01:48:57.547 INFO:tasks.rados.rados.0.smithi045.stdout:990: done (4 left) 2021-07-31T01:48:57.547 INFO:tasks.rados.rados.0.smithi045.stdout:991: done (3 left) 2021-07-31T01:48:57.547 INFO:tasks.rados.rados.0.smithi045.stdout:992: done (2 left) 2021-07-31T01:48:57.548 INFO:tasks.rados.rados.0.smithi045.stdout:993: done (1 left) 2021-07-31T01:48:57.548 INFO:tasks.rados.rados.0.smithi045.stdout:994: done (0 left) 2021-07-31T01:48:57.548 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 13 current snap is 90 2021-07-31T01:48:57.548 INFO:tasks.rados.rados.0.smithi045.stdout:995: seq_num 302 ranges {0=262144} 2021-07-31T01:48:57.549 INFO:tasks.rados.rados.0.smithi045.stdout:995: writing smithi04569269-13 from 0 to 262144 tid 1 2021-07-31T01:48:57.549 INFO:tasks.rados.rados.0.smithi045.stdout:996: snap_create 2021-07-31T01:48:57.552 INFO:tasks.rados.rados.0.smithi045.stdout:995: finishing write tid 1 to smithi04569269-13 2021-07-31T01:48:57.552 INFO:tasks.rados.rados.0.smithi045.stdout:995: finishing write tid 2 to smithi04569269-13 2021-07-31T01:48:57.554 INFO:tasks.rados.rados.0.smithi045.stdout:995: finishing write tid 3 to smithi04569269-13 2021-07-31T01:48:57.554 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 13 v 526 (ObjNum 302 snap 90 seq_num 302) dirty exists 2021-07-31T01:48:57.555 INFO:tasks.rados.rados.0.smithi045.stdout:995: left oid 13 (ObjNum 302 snap 90 seq_num 302) 2021-07-31T01:48:57.910 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T01:48:58.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:57 smithi154 conmon[34553]: cluster 2021-07-31T01:48:56.540409+0000 mon.a (mon.0) 877 2021-07-31T01:48:58.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:57 smithi154 conmon[34553]: : cluster [DBG] osdmap e237: 8 total, 8 up, 8 in 2021-07-31T01:48:58.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:57 smithi045 conmon[33655]: cluster 2021-07-31T01:48:56.540409+0000 mon.a (mon. 2021-07-31T01:48:58.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:57 smithi045 conmon[33655]: 0) 877 : cluster [DBG] osdmap e237: 8 total, 8 up, 8 in 2021-07-31T01:48:58.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:57 smithi045 conmon[39718]: cluster 2021-07-31T01:48:56.540409+0000 mon.a ( 2021-07-31T01:48:58.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:57 smithi045 conmon[39718]: mon.0) 877 : cluster [DBG] osdmap e237: 8 total, 8 up, 8 in 2021-07-31T01:48:58.189 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T01:48:58.189 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 3m ago 4m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T01:48:58.189 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 91s ago 4m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T01:48:58.189 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 91s ago 7m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T01:48:58.190 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 3m ago 9m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T01:48:58.190 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 3m ago 10m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T01:48:58.190 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 91s ago 8m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T01:48:58.190 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 3m ago 8m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T01:48:58.190 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 3m ago 4m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T01:48:58.191 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 91s ago 4m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T01:48:58.191 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 3m ago 7m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T01:48:58.191 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 3m ago 7m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T01:48:58.191 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 3m ago 6m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T01:48:58.192 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 3m ago 6m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T01:48:58.192 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 91s ago 6m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T01:48:58.192 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 91s ago 5m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T01:48:58.192 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 91s ago 5m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T01:48:58.192 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 91s ago 5m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T01:48:58.193 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 91s ago 4m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T01:48:58.523 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T01:48:58.524 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T01:48:58.525 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T01:48:58.526 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T01:48:58.526 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T01:48:58.526 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T01:48:58.526 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T01:48:58.527 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T01:48:58.527 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T01:48:58.527 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T01:48:58.527 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T01:48:58.528 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T01:48:58.528 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T01:48:58.528 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T01:48:58.528 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T01:48:58.528 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T01:48:58.529 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T01:48:58.551 INFO:tasks.rados.rados.0.smithi045.stdout:995: done (1 left) 2021-07-31T01:48:58.552 INFO:tasks.rados.rados.0.smithi045.stdout:996: done (0 left) 2021-07-31T01:48:58.552 INFO:tasks.rados.rados.0.smithi045.stdout:997: rollback oid 26 current snap is 91 2021-07-31T01:48:58.552 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 26 to 71 2021-07-31T01:48:58.552 INFO:tasks.rados.rados.0.smithi045.stdout:998: copy_from oid 48 from oid 14 current snap is 91 2021-07-31T01:48:58.553 INFO:tasks.rados.rados.0.smithi045.stdout:999: snap_create 2021-07-31T01:48:58.562 INFO:tasks.rados.rados.0.smithi045.stdout:997: finishing rollback tid 0 to smithi04569269-26 2021-07-31T01:48:58.563 INFO:tasks.rados.rados.0.smithi045.stdout:997: finishing rollback tid 1 to smithi04569269-26 2021-07-31T01:48:58.563 INFO:tasks.rados.rados.0.smithi045.stdout:997: finishing rollback tid 2 to smithi04569269-26 2021-07-31T01:48:58.564 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 26 v 210 (ObjNum 246 snap 68 seq_num 246) dirty exists 2021-07-31T01:48:58.593 INFO:tasks.rados.rados.0.smithi045.stdout:998: finishing copy_from to smithi04569269-48 2021-07-31T01:48:58.593 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 48 v 287 (ObjNum 275 snap 81 seq_num 275) dirty exists 2021-07-31T01:48:58.593 INFO:tasks.rados.rados.0.smithi045.stdout:998: finishing copy_from racing read to smithi04569269-48 2021-07-31T01:48:59.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:58 smithi154 conmon[34553]: cluster 2021-07-31T01:48:57.545961+0000 mon.a (mon.0) 878 : 2021-07-31T01:48:59.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:58 smithi154 conmon[34553]: cluster [DBG] osdmap e238: 8 total, 8 up, 8 in 2021-07-31T01:48:59.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:58 smithi154 conmon[34553]: cluster 2021-07-31T01:48:57.815047+0000 mgr.x (mgr.24542) 74 : cluster [DBG] pgmap v137: 25 pgs: 25 active+clean; 154 MiB data, 702 MiB used, 715 GiB / 715 GiB avail; 5.5 MiB/s rd, 84 KiB/s wr, 4 op/s 2021-07-31T01:48:59.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:58 smithi154 conmon[34553]: audit 2021-07-31T01:48:59.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:58 smithi154 conmon[34553]: 2021-07-31T01:48:57.898638+0000 mgr.x (mgr.24542) 75 : audit [DBG] from='client.14766 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:48:59.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:58 smithi154 conmon[34553]: audit 2021-07 2021-07-31T01:48:59.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:58 smithi154 conmon[34553]: -31T01:48 2021-07-31T01:48:59.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:58 smithi154 conmon[34553]: :58.522723+0000 mon.c (mon.1) 28 : audit [DBG] from='client.? 172.21.15.45:0/1177494023' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T01:48:59.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:58 smithi154 conmon[34553]: 2021-07-31T01:48:59.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:58 smithi045 conmon[33655]: cluster 2021-07-31T01:48:57.545961+0000 mon.a (mon. 2021-07-31T01:48:59.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:58 smithi045 conmon[33655]: 0) 878 : cluster [DBG] osdmap e238: 8 total, 8 up, 8 in 2021-07-31T01:48:59.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:58 smithi045 conmon[33655]: cluster 2021-07-31T01:48:57.815047+0000 mgr.x (mgr.24542) 74 : cluster [DBG] pgmap v137: 25 pgs: 25 active+clean; 154 MiB data, 702 MiB used, 715 GiB / 715 GiB avail; 5.5 MiB/s rd, 84 KiB/s wr, 4 op/s 2021-07-31T01:48:59.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:58 smithi045 conmon[33655]: audit 2021-07-31T01:48:59.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:58 smithi045 conmon[33655]: 2021-07-31T01:48:57.898638+0000 2021-07-31T01:48:59.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:58 smithi045 conmon[33655]: mgr.x (mgr.24542) 75 : audit [DBG] 2021-07-31T01:48:59.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:58 smithi045 conmon[33655]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:48:59.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:58 smithi045 conmon[33655]: audit 2021-07-31T01:48:59.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:58 smithi045 conmon[33655]: 2021-07-31T01:48:58. 2021-07-31T01:48:59.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:58 smithi045 conmon[33655]: 522723+0000 mon.c (mon.1) 28 : audit 2021-07-31T01:48:59.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:58 smithi045 conmon[33655]: [DBG] from='client.? 172.21.15.45:0/1177494023' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T01:48:59.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:58 smithi045 conmon[39718]: cluster 2021-07-31T01:48:59.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:58 smithi045 conmon[39718]: 2021-07-31T01:48:57 2021-07-31T01:48:59.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:58 smithi045 conmon[39718]: .545961+0000 mon.a (mon.0) 878 : cluster [DBG] osdmap e238: 8 total, 8 up, 8 in 2021-07-31T01:48:59.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:58 smithi045 conmon[39718]: cluster 2021-07- 2021-07-31T01:48:59.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:58 smithi045 conmon[39718]: 31T01:48:57.815047+0000 mgr.x (mgr.24542) 74 2021-07-31T01:48:59.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:58 smithi045 conmon[39718]: : cluster [DBG] pgmap v137: 25 pgs: 25 active+clean; 154 MiB data, 702 MiB used, 715 GiB / 715 GiB avail; 5.5 MiB/s rd, 84 KiB/s wr, 4 op/s 2021-07-31T01:48:59.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:58 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T01:48:59.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:58 smithi045 conmon[39718]: T01:48:57.898638+0000 mgr.x (mgr.24542) 75 2021-07-31T01:48:59.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:58 smithi045 conmon[39718]: : audit [DBG] from='client.14766 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:48:59.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:58 smithi045 conmon[39718]: audit 2021-07-31T01:48: 2021-07-31T01:48:59.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:58 smithi045 conmon[39718]: 58.522723+0000 mon.c (mon.1) 28 : audit [DBG] 2021-07-31T01:48:59.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:58 smithi045 conmon[39718]: from='client.? 172.21.15.45:0/1177494023' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T01:48:59.554 INFO:tasks.rados.rados.0.smithi045.stdout:997: done (2 left) 2021-07-31T01:48:59.554 INFO:tasks.rados.rados.0.smithi045.stdout:998: done (1 left) 2021-07-31T01:48:59.554 INFO:tasks.rados.rados.0.smithi045.stdout:999: done (0 left) 2021-07-31T01:48:59.555 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 23 current snap is 92 2021-07-31T01:48:59.555 INFO:tasks.rados.rados.0.smithi045.stdout:1000: seq_num 303 ranges {81920=278528} 2021-07-31T01:48:59.556 INFO:tasks.rados.rados.0.smithi045.stdout:1000: writing smithi04569269-23 from 81920 to 360448 tid 1 2021-07-31T01:48:59.556 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 36 current snap is 92 2021-07-31T01:48:59.556 INFO:tasks.rados.rados.0.smithi045.stdout:1001: seq_num 304 ranges {2646016=294912} 2021-07-31T01:48:59.572 INFO:tasks.rados.rados.0.smithi045.stdout:1001: writing smithi04569269-36 from 2646016 to 2940928 tid 1 2021-07-31T01:48:59.573 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 12 current snap is 92 2021-07-31T01:48:59.573 INFO:tasks.rados.rados.0.smithi045.stdout:1002: seq_num 305 ranges {3358720=311296} 2021-07-31T01:48:59.593 INFO:tasks.rados.rados.0.smithi045.stdout:1002: writing smithi04569269-12 from 3358720 to 3670016 tid 1 2021-07-31T01:48:59.593 INFO:tasks.rados.rados.0.smithi045.stdout:1003: snap_create 2021-07-31T01:48:59.593 INFO:tasks.rados.rados.0.smithi045.stdout:1000: finishing write tid 1 to smithi04569269-23 2021-07-31T01:48:59.594 INFO:tasks.rados.rados.0.smithi045.stdout:1000: finishing write tid 2 to smithi04569269-23 2021-07-31T01:48:59.594 INFO:tasks.rados.rados.0.smithi045.stdout:1000: finishing write tid 3 to smithi04569269-23 2021-07-31T01:48:59.594 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 23 v 527 (ObjNum 303 snap 92 seq_num 303) dirty exists 2021-07-31T01:48:59.594 INFO:tasks.rados.rados.0.smithi045.stdout:1000: left oid 23 (ObjNum 303 snap 92 seq_num 303) 2021-07-31T01:48:59.595 INFO:tasks.rados.rados.0.smithi045.stdout:1001: finishing write tid 1 to smithi04569269-36 2021-07-31T01:48:59.595 INFO:tasks.rados.rados.0.smithi045.stdout:1001: finishing write tid 2 to smithi04569269-36 2021-07-31T01:48:59.595 INFO:tasks.rados.rados.0.smithi045.stdout:1001: finishing write tid 3 to smithi04569269-36 2021-07-31T01:48:59.595 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 36 v 684 (ObjNum 304 snap 92 seq_num 304) dirty exists 2021-07-31T01:48:59.595 INFO:tasks.rados.rados.0.smithi045.stdout:1001: left oid 36 (ObjNum 304 snap 92 seq_num 304) 2021-07-31T01:48:59.597 INFO:tasks.rados.rados.0.smithi045.stdout:1002: finishing write tid 1 to smithi04569269-12 2021-07-31T01:48:59.597 INFO:tasks.rados.rados.0.smithi045.stdout:1002: finishing write tid 2 to smithi04569269-12 2021-07-31T01:48:59.598 INFO:tasks.rados.rados.0.smithi045.stdout:1002: finishing write tid 3 to smithi04569269-12 2021-07-31T01:48:59.598 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 12 v 528 (ObjNum 305 snap 92 seq_num 305) dirty exists 2021-07-31T01:48:59.599 INFO:tasks.rados.rados.0.smithi045.stdout:1002: left oid 12 (ObjNum 305 snap 92 seq_num 305) 2021-07-31T01:49:00.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:59 smithi154 conmon[34553]: audit 2021-07-31T01:48:58.180489+0000 mgr.x (mgr.24542) 76 : audit [DBG] from='client.24679 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:49:00.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:59 smithi154 conmon[34553]: cluster 2021-07-31T01:49:00.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:48:59 smithi154 conmon[34553]: 2021-07-31T01:48:58.550576+0000 mon.a (mon.0) 879 : cluster [DBG] osdmap e239: 8 total, 8 up, 8 in 2021-07-31T01:49:00.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:59 smithi045 conmon[33655]: audit 2021-07-31T01:48:58.180489+0000 mgr.x (mgr.24542) 76 : audit [DBG] from='client.24679 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:49:00.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:48:59 smithi045 conmon[33655]: cluster 2021-07-31T01:48:58.550576+0000 mon.a (mon.0) 879 : cluster [DBG] osdmap e239: 8 total, 8 up, 8 in 2021-07-31T01:49:00.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:59 smithi045 conmon[39718]: audit 2021-07-31T01:48:58. 2021-07-31T01:49:00.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:59 smithi045 conmon[39718]: 180489+0000 mgr.x (mgr.24542) 76 : audit [DBG] from='client.24679 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:49:00.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:59 smithi045 conmon[39718]: cluster 2021-07-31T01:48:58. 2021-07-31T01:49:00.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:59 smithi045 conmon[39718]: 550576+0000 mon.a (mon.0) 879 2021-07-31T01:49:00.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:48:59 smithi045 conmon[39718]: : cluster [DBG] osdmap e239: 8 total, 8 up, 8 in 2021-07-31T01:49:00.570 INFO:tasks.rados.rados.0.smithi045.stdout:1000: done (3 left) 2021-07-31T01:49:00.570 INFO:tasks.rados.rados.0.smithi045.stdout:1001: done (2 left) 2021-07-31T01:49:00.571 INFO:tasks.rados.rados.0.smithi045.stdout:1002: done (1 left) 2021-07-31T01:49:00.571 INFO:tasks.rados.rados.0.smithi045.stdout:1003: done (0 left) 2021-07-31T01:49:00.571 INFO:tasks.rados.rados.0.smithi045.stdout:1004: read oid 16 snap -1 2021-07-31T01:49:00.571 INFO:tasks.rados.rados.0.smithi045.stdout:1004: expect (ObjNum 207 snap 53 seq_num 207) 2021-07-31T01:49:00.572 INFO:tasks.rados.rados.0.smithi045.stdout:1005: read oid 47 snap -1 2021-07-31T01:49:00.572 INFO:tasks.rados.rados.0.smithi045.stdout:1005: expect (ObjNum 293 snap 86 seq_num 293) 2021-07-31T01:49:00.572 INFO:tasks.rados.rados.0.smithi045.stdout:1006: read oid 29 snap -1 2021-07-31T01:49:00.572 INFO:tasks.rados.rados.0.smithi045.stdout:1006: expect (ObjNum 237 snap 66 seq_num 237) 2021-07-31T01:49:00.572 INFO:tasks.rados.rados.0.smithi045.stdout:1007: rollback oid 14 current snap is 93 2021-07-31T01:49:00.573 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 14 to 91 2021-07-31T01:49:00.573 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 13 current snap is 93 2021-07-31T01:49:00.573 INFO:tasks.rados.rados.0.smithi045.stdout:1008: seq_num 306 ranges {262144=327680} 2021-07-31T01:49:00.574 INFO:tasks.rados.rados.0.smithi045.stdout:1008: writing smithi04569269-13 from 262144 to 589824 tid 1 2021-07-31T01:49:00.574 INFO:tasks.rados.rados.0.smithi045.stdout:1009: read oid 9 snap -1 2021-07-31T01:49:00.575 INFO:tasks.rados.rados.0.smithi045.stdout:1009: expect (ObjNum 172 snap 41 seq_num 172) 2021-07-31T01:49:00.575 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 7 current snap is 93 2021-07-31T01:49:00.575 INFO:tasks.rados.rados.0.smithi045.stdout:1010: seq_num 307 ranges {0=344064} 2021-07-31T01:49:00.576 INFO:tasks.rados.rados.0.smithi045.stdout:1010: writing smithi04569269-7 from 0 to 344064 tid 1 2021-07-31T01:49:00.576 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 20 current snap is 93 2021-07-31T01:49:00.576 INFO:tasks.rados.rados.0.smithi045.stdout:1011: seq_num 308 ranges {1974272=360448} 2021-07-31T01:49:00.590 INFO:tasks.rados.rados.0.smithi045.stdout:1011: writing smithi04569269-20 from 1974272 to 2334720 tid 1 2021-07-31T01:49:00.590 INFO:tasks.rados.rados.0.smithi045.stdout:1012: delete oid 27 current snap is 93 2021-07-31T01:49:00.590 INFO:tasks.rados.rados.0.smithi045.stdout:1007: finishing rollback tid 0 to smithi04569269-14 2021-07-31T01:49:00.591 INFO:tasks.rados.rados.0.smithi045.stdout:1004: expect (ObjNum 207 snap 53 seq_num 207) 2021-07-31T01:49:00.623 INFO:tasks.rados.rados.0.smithi045.stdout:1007: finishing rollback tid 1 to smithi04569269-14 2021-07-31T01:49:00.623 INFO:tasks.rados.rados.0.smithi045.stdout:1007: finishing rollback tid 2 to smithi04569269-14 2021-07-31T01:49:00.623 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 14 v 320 (ObjNum 275 snap 81 seq_num 275) dirty exists 2021-07-31T01:49:00.623 INFO:tasks.rados.rados.0.smithi045.stdout:1009: expect (ObjNum 172 snap 41 seq_num 172) 2021-07-31T01:49:00.639 INFO:tasks.rados.rados.0.smithi045.stdout:1008: finishing write tid 1 to smithi04569269-13 2021-07-31T01:49:00.639 INFO:tasks.rados.rados.0.smithi045.stdout:1008: finishing write tid 2 to smithi04569269-13 2021-07-31T01:49:00.639 INFO:tasks.rados.rados.0.smithi045.stdout:1008: finishing write tid 3 to smithi04569269-13 2021-07-31T01:49:00.640 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 13 v 530 (ObjNum 306 snap 93 seq_num 306) dirty exists 2021-07-31T01:49:00.640 INFO:tasks.rados.rados.0.smithi045.stdout:1008: left oid 13 (ObjNum 306 snap 93 seq_num 306) 2021-07-31T01:49:00.640 INFO:tasks.rados.rados.0.smithi045.stdout:1006: expect (ObjNum 237 snap 66 seq_num 237) 2021-07-31T01:49:00.680 INFO:tasks.rados.rados.0.smithi045.stdout:1010: finishing write tid 1 to smithi04569269-7 2021-07-31T01:49:00.680 INFO:tasks.rados.rados.0.smithi045.stdout:1010: finishing write tid 2 to smithi04569269-7 2021-07-31T01:49:00.681 INFO:tasks.rados.rados.0.smithi045.stdout:1010: finishing write tid 3 to smithi04569269-7 2021-07-31T01:49:00.681 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 7 v 509 (ObjNum 307 snap 93 seq_num 307) dirty exists 2021-07-31T01:49:00.681 INFO:tasks.rados.rados.0.smithi045.stdout:1010: left oid 7 (ObjNum 307 snap 93 seq_num 307) 2021-07-31T01:49:00.681 INFO:tasks.rados.rados.0.smithi045.stdout:1011: finishing write tid 1 to smithi04569269-20 2021-07-31T01:49:00.682 INFO:tasks.rados.rados.0.smithi045.stdout:1011: finishing write tid 2 to smithi04569269-20 2021-07-31T01:49:00.682 INFO:tasks.rados.rados.0.smithi045.stdout:1005: expect (ObjNum 293 snap 86 seq_num 293) 2021-07-31T01:49:01.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:00 smithi154 conmon[34553]: cluster 2021-07-31T01:48:59.553285+0000 mon.a (mon.0) 880 : cluster [DBG] osdmap e240: 8 total, 8 up, 8 in 2021-07-31T01:49:01.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:00 smithi154 conmon[34553]: cluster 2021-07-31T01:48:59.815452+0000 mgr.x (mgr.24542) 77 : cluster [DBG] pgmap v140: 25 pgs: 25 active+clean; 157 MiB data, 717 MiB used, 715 GiB / 715 GiB avail; 1.4 MiB/s rd, 1.9 MiB/s wr, 9 op/s 2021-07-31T01:49:01.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:00 smithi045 conmon[33655]: cluster 2021-07-31T01:48:59.553285+0000 mon.a (mon.0) 880 : cluster 2021-07-31T01:49:01.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:00 smithi045 conmon[33655]: [DBG] osdmap e240: 8 total, 8 up, 8 in 2021-07-31T01:49:01.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:00 smithi045 conmon[33655]: cluster 2021-07-31T01:48:59.815452+0000 mgr.x (mgr.24542) 77 : cluster [DBG] pgmap v140: 25 pgs: 25 active+clean; 157 MiB data, 717 MiB used, 715 GiB / 715 GiB avail; 1.4 MiB/s rd, 1.9 MiB/s wr, 9 op/s 2021-07-31T01:49:01.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:00 smithi045 conmon[39718]: cluster 2021-07-31T01:48:59.553285+0000 mon.a (mon. 2021-07-31T01:49:01.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:00 smithi045 conmon[39718]: 0) 880 : cluster [DBG] osdmap e240: 8 total, 8 up, 8 in 2021-07-31T01:49:01.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:00 smithi045 conmon[39718]: cluster 2021-07-31T01:48:59 2021-07-31T01:49:01.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:00 smithi045 conmon[39718]: .815452+0000 mgr.x (mgr.24542) 77 : cluster [DBG] pgmap v140: 25 pgs: 25 active+clean; 157 MiB data, 717 MiB used, 715 GiB / 715 GiB avail; 1.4 MiB/s rd, 1.9 MiB/s wr, 9 op/s 2021-07-31T01:49:01.031 INFO:tasks.rados.rados.0.smithi045.stdout:1011: finishing write tid 3 to smithi04569269-20 2021-07-31T01:49:01.031 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 20 v 687 (ObjNum 308 snap 93 seq_num 308) dirty exists 2021-07-31T01:49:01.031 INFO:tasks.rados.rados.0.smithi045.stdout:1011: left oid 20 (ObjNum 308 snap 93 seq_num 308) 2021-07-31T01:49:01.031 INFO:tasks.rados.rados.0.smithi045.stdout:1004: done (8 left) 2021-07-31T01:49:01.031 INFO:tasks.rados.rados.0.smithi045.stdout:1005: done (7 left) 2021-07-31T01:49:01.032 INFO:tasks.rados.rados.0.smithi045.stdout:1006: done (6 left) 2021-07-31T01:49:01.032 INFO:tasks.rados.rados.0.smithi045.stdout:1007: done (5 left) 2021-07-31T01:49:01.032 INFO:tasks.rados.rados.0.smithi045.stdout:1008: done (4 left) 2021-07-31T01:49:01.032 INFO:tasks.rados.rados.0.smithi045.stdout:1009: done (3 left) 2021-07-31T01:49:01.033 INFO:tasks.rados.rados.0.smithi045.stdout:1010: done (2 left) 2021-07-31T01:49:01.033 INFO:tasks.rados.rados.0.smithi045.stdout:1011: done (1 left) 2021-07-31T01:49:01.033 INFO:tasks.rados.rados.0.smithi045.stdout:1012: done (0 left) 2021-07-31T01:49:01.033 INFO:tasks.rados.rados.0.smithi045.stdout:1013: rollback oid 50 current snap is 93 2021-07-31T01:49:01.033 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 50 to 90 2021-07-31T01:49:01.034 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 29 current snap is 93 2021-07-31T01:49:01.034 INFO:tasks.rados.rados.0.smithi045.stdout:1014: seq_num 310 ranges {1581056=401408} 2021-07-31T01:49:01.041 INFO:tasks.rados.rados.0.smithi045.stdout:1014: writing smithi04569269-29 from 1581056 to 1982464 tid 1 2021-07-31T01:49:01.041 INFO:tasks.rados.rados.0.smithi045.stdout:1015: snap_remove snap 92 2021-07-31T01:49:01.516 INFO:tasks.rados.rados.0.smithi045.stdout:1015: done (2 left) 2021-07-31T01:49:01.517 INFO:tasks.rados.rados.0.smithi045.stdout:1016: snap_remove snap 89 2021-07-31T01:49:02.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:01 smithi154 conmon[34553]: cluster 2021-07-31T01:49:00.569052+0000 mon.a (mon.0) 881 : cluster [DBG] osdmap e241: 8 total, 8 up, 8 in 2021-07-31T01:49:02.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:01 smithi154 conmon[34553]: cluster 2021-07-31T01:49:01.515675+0000 mon.a (mon.0) 882 : cluster [DBG] osdmap e242: 8 total, 8 up, 8 in 2021-07-31T01:49:02.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:01 smithi045 conmon[39718]: cluster 2021-07-31T01:49:00.569052+0000 2021-07-31T01:49:02.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:01 smithi045 conmon[39718]: mon.a (mon.0) 881 : cluster [DBG] osdmap e241: 8 total, 8 up, 8 in 2021-07-31T01:49:02.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:01 smithi045 conmon[39718]: cluster 2021-07-31T01 2021-07-31T01:49:02.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:01 smithi045 conmon[39718]: :49:01.515675+0000 mon.a (mon.0) 882 : cluster [DBG] osdmap e242: 8 total, 8 up, 8 in 2021-07-31T01:49:02.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:01 smithi045 conmon[33655]: cluster 2021-07-31T01:49:00.569052+0000 2021-07-31T01:49:02.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:01 smithi045 conmon[33655]: mon.a (mon.0) 881 : cluster [DBG] osdmap e241: 8 total, 8 up, 8 in 2021-07-31T01:49:02.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:01 smithi045 conmon[33655]: cluster 2021-07-31T01:49:01.515675+0000 mon.a (mon.0) 882 : cluster [DBG] osdmap e242: 8 total, 8 up, 8 in 2021-07-31T01:49:02.523 INFO:tasks.rados.rados.0.smithi045.stdout:1016: done (2 left) 2021-07-31T01:49:02.523 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 22 current snap is 93 2021-07-31T01:49:02.524 INFO:tasks.rados.rados.0.smithi045.stdout:1013: finishing rollback tid 0 to smithi04569269-50 2021-07-31T01:49:02.524 INFO:tasks.rados.rados.0.smithi045.stdout:1013: finishing rollback tid 1 to smithi04569269-50 2021-07-31T01:49:02.524 INFO:tasks.rados.rados.0.smithi045.stdout:1013: finishing rollback tid 2 to smithi04569269-50 2021-07-31T01:49:02.525 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 50 v 423 (ObjNum 76 snap 11 seq_num 76) dirty exists 2021-07-31T01:49:02.525 INFO:tasks.rados.rados.0.smithi045.stdout:1014: finishing write tid 1 to smithi04569269-29 2021-07-31T01:49:02.525 INFO:tasks.rados.rados.0.smithi045.stdout:1017: seq_num 311 ranges {778240=417792} 2021-07-31T01:49:02.530 INFO:tasks.rados.rados.0.smithi045.stdout:1017: writing smithi04569269-22 from 778240 to 1196032 tid 1 2021-07-31T01:49:02.530 INFO:tasks.rados.rados.0.smithi045.stdout:1013: done (2 left) 2021-07-31T01:49:02.531 INFO:tasks.rados.rados.0.smithi045.stdout:1018: read oid 48 snap -1 2021-07-31T01:49:02.531 INFO:tasks.rados.rados.0.smithi045.stdout:1018: expect (ObjNum 275 snap 81 seq_num 275) 2021-07-31T01:49:02.531 INFO:tasks.rados.rados.0.smithi045.stdout:1019: delete oid 7 current snap is 93 2021-07-31T01:49:02.531 INFO:tasks.rados.rados.0.smithi045.stdout:1014: finishing write tid 2 to smithi04569269-29 2021-07-31T01:49:02.532 INFO:tasks.rados.rados.0.smithi045.stdout:1014: finishing write tid 3 to smithi04569269-29 2021-07-31T01:49:02.532 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 29 v 184 (ObjNum 310 snap 93 seq_num 310) dirty exists 2021-07-31T01:49:02.533 INFO:tasks.rados.rados.0.smithi045.stdout:1014: left oid 29 (ObjNum 310 snap 93 seq_num 310) 2021-07-31T01:49:02.540 INFO:tasks.rados.rados.0.smithi045.stdout:1014: done (3 left) 2021-07-31T01:49:02.540 INFO:tasks.rados.rados.0.smithi045.stdout:1019: done (2 left) 2021-07-31T01:49:02.540 INFO:tasks.rados.rados.0.smithi045.stdout:1020: rollback oid 10 current snap is 93 2021-07-31T01:49:02.540 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 10 to 71 2021-07-31T01:49:02.540 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 8 current snap is 93 2021-07-31T01:49:02.541 INFO:tasks.rados.rados.0.smithi045.stdout:1021: seq_num 313 ranges {4308992=450560} 2021-07-31T01:49:02.561 INFO:tasks.rados.rados.0.smithi045.stdout:1021: writing smithi04569269-8 from 4308992 to 4759552 tid 1 2021-07-31T01:49:02.561 INFO:tasks.rados.rados.0.smithi045.stdout:1022: rollback oid 25 current snap is 93 2021-07-31T01:49:02.561 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 25 to 90 2021-07-31T01:49:02.562 INFO:tasks.rados.rados.0.smithi045.stdout:1017: finishing write tid 1 to smithi04569269-22 2021-07-31T01:49:02.562 INFO:tasks.rados.rados.0.smithi045.stdout:1017: finishing write tid 2 to smithi04569269-22 2021-07-31T01:49:02.562 INFO:tasks.rados.rados.0.smithi045.stdout:1020: finishing rollback tid 0 to smithi04569269-10 2021-07-31T01:49:02.562 INFO:tasks.rados.rados.0.smithi045.stdout:1017: finishing write tid 3 to smithi04569269-22 2021-07-31T01:49:02.563 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 22 v 198 (ObjNum 311 snap 93 seq_num 311) dirty exists 2021-07-31T01:49:02.563 INFO:tasks.rados.rados.0.smithi045.stdout:1017: left oid 22 (ObjNum 311 snap 93 seq_num 311) 2021-07-31T01:49:02.563 INFO:tasks.rados.rados.0.smithi045.stdout:1020: finishing rollback tid 1 to smithi04569269-10 2021-07-31T01:49:02.563 INFO:tasks.rados.rados.0.smithi045.stdout:1020: finishing rollback tid 2 to smithi04569269-10 2021-07-31T01:49:02.563 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 10 v 431 (ObjNum 199 snap 52 seq_num 199) dirty exists 2021-07-31T01:49:02.564 INFO:tasks.rados.rados.0.smithi045.stdout:1017: done (4 left) 2021-07-31T01:49:02.564 INFO:tasks.rados.rados.0.smithi045.stdout:1020: done (3 left) 2021-07-31T01:49:02.564 INFO:tasks.rados.rados.0.smithi045.stdout:1023: delete oid 39 current snap is 93 2021-07-31T01:49:02.564 INFO:tasks.rados.rados.0.smithi045.stdout:1018: expect (ObjNum 275 snap 81 seq_num 275) 2021-07-31T01:49:02.691 INFO:tasks.rados.rados.0.smithi045.stdout:1018: done (3 left) 2021-07-31T01:49:02.691 INFO:tasks.rados.rados.0.smithi045.stdout:1023: done (2 left) 2021-07-31T01:49:02.692 INFO:tasks.rados.rados.0.smithi045.stdout:1024: delete oid 41 current snap is 93 2021-07-31T01:49:02.692 INFO:tasks.rados.rados.0.smithi045.stdout:1022: finishing rollback tid 0 to smithi04569269-25 2021-07-31T01:49:02.692 INFO:tasks.rados.rados.0.smithi045.stdout:1021: finishing write tid 1 to smithi04569269-8 2021-07-31T01:49:02.693 INFO:tasks.rados.rados.0.smithi045.stdout:1021: finishing write tid 2 to smithi04569269-8 2021-07-31T01:49:02.693 INFO:tasks.rados.rados.0.smithi045.stdout:1022: finishing rollback tid 1 to smithi04569269-25 2021-07-31T01:49:02.694 INFO:tasks.rados.rados.0.smithi045.stdout:1022: finishing rollback tid 2 to smithi04569269-25 2021-07-31T01:49:02.694 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 25 v 180 (ObjNum 296 snap 88 seq_num 296) dirty exists 2021-07-31T01:49:02.694 INFO:tasks.rados.rados.0.smithi045.stdout:1021: finishing write tid 3 to smithi04569269-8 2021-07-31T01:49:02.694 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 8 v 696 (ObjNum 313 snap 93 seq_num 313) dirty exists 2021-07-31T01:49:02.695 INFO:tasks.rados.rados.0.smithi045.stdout:1021: left oid 8 (ObjNum 313 snap 93 seq_num 313) 2021-07-31T01:49:02.696 INFO:tasks.rados.rados.0.smithi045.stdout:1021: done (2 left) 2021-07-31T01:49:02.696 INFO:tasks.rados.rados.0.smithi045.stdout:1022: done (1 left) 2021-07-31T01:49:02.696 INFO:tasks.rados.rados.0.smithi045.stdout:1024: done (0 left) 2021-07-31T01:49:02.696 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 10 current snap is 93 2021-07-31T01:49:02.697 INFO:tasks.rados.rados.0.smithi045.stdout:1025: seq_num 316 ranges {3457024=499712} 2021-07-31T01:49:02.715 INFO:tasks.rados.rados.0.smithi045.stdout:1025: writing smithi04569269-10 from 3457024 to 3956736 tid 1 2021-07-31T01:49:02.715 INFO:tasks.rados.rados.0.smithi045.stdout:1026: rmattr oid 1 current snap is 93 2021-07-31T01:49:02.715 INFO:tasks.rados.rados.0.smithi045.stdout:1026: done (1 left) 2021-07-31T01:49:02.715 INFO:tasks.rados.rados.0.smithi045.stdout:1027: read oid 7 snap 71 2021-07-31T01:49:02.716 INFO:tasks.rados.rados.0.smithi045.stdout:1027: expect deleted 2021-07-31T01:49:02.716 INFO:tasks.rados.rados.0.smithi045.stdout:1028: snap_remove snap 91 2021-07-31T01:49:03.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:02 smithi154 conmon[34553]: cluster 2021-07-31T01:49:01.815931+0000 mgr.x (mgr.24542) 78 : cluster [DBG] 2021-07-31T01:49:03.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:02 smithi154 conmon[34553]: pgmap v143: 25 pgs: 25 active+clean; 157 MiB data, 717 MiB used, 715 GiB / 715 GiB avail; 1.4 MiB/s rd, 1.9 MiB/s wr, 9 op/s 2021-07-31T01:49:03.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:02 smithi154 conmon[34553]: cluster 2021-07-31T01:49:02.522053+0000 mon.a (mon.0) 883 : cluster [DBG] osdmap e243: 8 total, 8 up, 8 in 2021-07-31T01:49:03.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:02 smithi045 conmon[33655]: cluster 2021-07-31T01:49:01.815931+0000 mgr.x (mgr.24542) 78 : 2021-07-31T01:49:03.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:02 smithi045 conmon[33655]: cluster [DBG] pgmap v143: 25 pgs: 25 active+clean; 157 MiB data, 717 MiB used, 715 GiB / 715 GiB avail; 1.4 MiB/s rd, 1.9 MiB/s wr, 9 op/s 2021-07-31T01:49:03.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:02 smithi045 conmon[33655]: cluster 2021-07-31T01:49:02.522053+0000 mon.a (mon.0) 883 : cluster [DBG] osdmap e243: 8 total, 8 up, 8 in 2021-07-31T01:49:03.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:02 smithi045 conmon[39718]: cluster 2021-07-31T01:49:01.815931+0000 2021-07-31T01:49:03.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:02 smithi045 conmon[39718]: mgr.x (mgr.24542) 78 : cluster [DBG] pgmap v143: 25 pgs: 25 active+clean; 157 MiB data, 717 MiB used, 715 GiB / 715 GiB avail; 1.4 MiB/s rd, 1.9 MiB/s wr, 9 op/s 2021-07-31T01:49:03.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:02 smithi045 conmon[39718]: cluster 2021-07-31T01: 2021-07-31T01:49:03.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:02 smithi045 conmon[39718]: 49:02.522053+0000 mon.a (mon.0) 883 : cluster [DBG] osdmap e243: 8 total, 8 up, 8 in 2021-07-31T01:49:03.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:02 smithi045 conmon[39718]: 2021-07-31T01:49:03.579 INFO:tasks.rados.rados.0.smithi045.stdout:1025: finishing write tid 1 to smithi04569269-10 2021-07-31T01:49:03.580 INFO:tasks.rados.rados.0.smithi045.stdout:1025: finishing write tid 2 to smithi04569269-10 2021-07-31T01:49:03.580 INFO:tasks.rados.rados.0.smithi045.stdout:1025: finishing write tid 3 to smithi04569269-10 2021-07-31T01:49:03.580 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 10 v 433 (ObjNum 316 snap 93 seq_num 316) dirty exists 2021-07-31T01:49:03.580 INFO:tasks.rados.rados.0.smithi045.stdout:1025: left oid 10 (ObjNum 316 snap 93 seq_num 316) 2021-07-31T01:49:03.580 INFO:tasks.rados.rados.0.smithi045.stdout:1025: done (2 left) 2021-07-31T01:49:03.581 INFO:tasks.rados.rados.0.smithi045.stdout:1027: done (1 left) 2021-07-31T01:49:03.581 INFO:tasks.rados.rados.0.smithi045.stdout:1028: done (0 left) 2021-07-31T01:49:03.581 INFO:tasks.rados.rados.0.smithi045.stdout:1029: read oid 16 snap -1 2021-07-31T01:49:03.581 INFO:tasks.rados.rados.0.smithi045.stdout:1029: expect (ObjNum 207 snap 53 seq_num 207) 2021-07-31T01:49:03.582 INFO:tasks.rados.rados.0.smithi045.stdout:1030: snap_remove snap 71 2021-07-31T01:49:03.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:49:03 smithi045 conmon[61210]: level=error ts=2021-07-31T01:49:03.511Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:49:03.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:49:03 smithi045 conmon[61210]: level=error ts=2021-07-31T01:49:03.511Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:49:03.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:49:03 smithi045 conmon[61210]: level=error ts=2021-07-31T01:49:03.511Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T01:49:04.583 INFO:tasks.rados.rados.0.smithi045.stdout:1030: done (1 left) 2021-07-31T01:49:04.584 INFO:tasks.rados.rados.0.smithi045.stdout:1031: rollback oid 49 current snap is 93 2021-07-31T01:49:04.584 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 49 to 90 2021-07-31T01:49:04.584 INFO:tasks.rados.rados.0.smithi045.stdout:1032: snap_create 2021-07-31T01:49:04.584 INFO:tasks.rados.rados.0.smithi045.stdout:1029: expect (ObjNum 207 snap 53 seq_num 207) 2021-07-31T01:49:04.619 INFO:tasks.rados.rados.0.smithi045.stdout:1031: finishing rollback tid 1 to smithi04569269-49 2021-07-31T01:49:04.620 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 49 v 610 (ObjNum -1105720832 snap 21873 seq_num 3189536720) dirty dne 2021-07-31T01:49:05.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:04 smithi154 conmon[34553]: cluster 2021-07-31T01:49:03.577204+0000 mon.a (mon.0) 884 : cluster [DBG] 2021-07-31T01:49:05.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:04 smithi154 conmon[34553]: osdmap e244: 8 total, 8 up, 8 in 2021-07-31T01:49:05.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:04 smithi154 conmon[34553]: cluster 2021-07-31T01:49:03.816317+0000 mgr.x (mgr.24542) 79 : cluster [DBG] pgmap v146: 25 pgs: 25 active+clean; 157 MiB data, 724 MiB used, 715 GiB / 715 GiB avail; 2.8 MiB/s rd, 0 op/s 2021-07-31T01:49:05.004 INFO:journalctl@ceph.osd.5.smithi154.stdout:Jul 31 01:49:04 smithi154 conmon[42050]: debug 2021-07-31T01:49:04.587+0000 7fc9b5064700 -1 osd.5 pg_epoch: 245 pg[3.cs0( v 245'420 (0'0,245'420] local-lis/les=56/57 n=9 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [5,3,6]p5(0) r=0 lpr=56 luod=244'418 crt=244'418 lcod 243'416 mlcod 243'416 active+clean+snaptrim trimq=[8f~1] ps=[b3~1,b5~2]] removing snap head 2021-07-31T01:49:05.004 INFO:journalctl@ceph.osd.5.smithi154.stdout:Jul 31 01:49:04 smithi154 conmon[42050]: debug 2021-07-31T01:49:04.591+0000 7fc9b105c700 -1 osd.5 pg_epoch: 245 pg[3.cs0( v 245'422 (0'0,245'422] local-lis/les=56/57 n=7 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [5,3,6]p5(0) r=0 lpr=56 crt=245'422 lcod 245'420 mlcod 245'420 active+clean+snaptrim trimq=[8f~1] ps=[b3~1,b5~2]] removing snap head 2021-07-31T01:49:05.026 INFO:journalctl@ceph.osd.1.smithi045.stdout:Jul 31 01:49:04 smithi045 conmon[46771]: debug 2021-07-31T01:49:05.026 INFO:journalctl@ceph.osd.1.smithi045.stdout:Jul 31 01:49:04 smithi045 conmon[46771]: 2021-07-31T01:49:04.595+0000 7f41959a6700 -1 osd.1 pg_epoch: 245 pg[3.4s0( v 245'214 (0'0,245'214] local-lis/les=56/57 n=5 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [1,2,6]p1(0) r=0 lpr=56 crt=245'214 lcod 245'212 mlcod 245'212 active+clean+snaptrim trimq=[8f~1] ps=[b3~1,b5~2]] removing snap head 2021-07-31T01:49:05.027 INFO:journalctl@ceph.osd.1.smithi045.stdout:Jul 31 01:49:04 smithi045 conmon[46771]: 2021-07-31T01:49:05.027 INFO:journalctl@ceph.osd.3.smithi045.stdout:Jul 31 01:49:04 smithi045 conmon[54013]: debug 2021-07-31T01:49:05.027 INFO:journalctl@ceph.osd.3.smithi045.stdout:Jul 31 01:49:04 smithi045 conmon[54013]: 2021-07-31T01:49:04.586+0000 7f64f88c5700 -1 osd.3 pg_epoch: 245 pg[3.bs0( v 245'705 (0'0,245'705] local-lis/les=56/57 n=15 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [3,0,4]p3(0) r=0 lpr=56 luod=244'702 crt=244'702 lcod 244'700 mlcod 244'700 active+clean+snaptrim trimq=[8f~1] ps=[b3~1,b5~2]] removing snap head 2021-07-31T01:49:05.028 INFO:journalctl@ceph.osd.3.smithi045.stdout:Jul 31 01:49:04 smithi045 conmon[54013]: 2021-07-31T01:49:05.028 INFO:journalctl@ceph.osd.3.smithi045.stdout:Jul 31 01:49:04 smithi045 conmon[54013]: debug 2021-07-31T01:49:05.028 INFO:journalctl@ceph.osd.3.smithi045.stdout:Jul 31 01:49:04 smithi045 conmon[54013]: 2021-07-31T01:49:04.598+0000 7f64fc8cd700 -1 osd.3 pg_epoch: 245 pg[3.bs0( v 245'709 (0'0,245'709] local-lis/les=56/57 n=12 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [3,0,4]p3(0) r=0 lpr=56 luod=245'707 crt=245'707 lcod 245'705 mlcod 245'705 active+clean+snaptrim trimq=[8f~1] ps=[b3~1,b5~2]] removing snap head 2021-07-31T01:49:05.029 INFO:journalctl@ceph.osd.3.smithi045.stdout:Jul 31 01:49:04 smithi045 conmon[54013]: 2021-07-31T01:49:05.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:04 smithi045 conmon[33655]: cluster 2021-07-31T01:49:03.577204+0000 mon.a (mon.0) 884 : cluster [DBG] osdmap e244: 8 total, 8 up, 8 in 2021-07-31T01:49:05.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:04 smithi045 conmon[33655]: cluster 2021-07-31T01:49:05.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:04 smithi045 conmon[33655]: 2021-07-31T01:49:03.816317+0000 mgr.x (mgr.24542) 79 : cluster [DBG] pgmap v146: 25 pgs: 25 active+clean; 157 MiB data, 724 MiB used, 715 GiB / 715 GiB avail; 2.8 MiB/s rd, 0 op/s 2021-07-31T01:49:05.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:04 smithi045 conmon[39718]: cluster 2021-07-31T01:49:03.577204+0000 mon.a (mon 2021-07-31T01:49:05.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:04 smithi045 conmon[39718]: .0) 884 : cluster [DBG] osdmap e244: 8 total, 8 up, 8 in 2021-07-31T01:49:05.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:04 smithi045 conmon[39718]: cluster 2021-07-31T01:49:03.816317+0000 mgr.x ( 2021-07-31T01:49:05.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:04 smithi045 conmon[39718]: mgr.24542) 79 : cluster [DBG] pgmap v146: 25 pgs: 25 active+clean; 157 MiB data, 724 MiB used, 715 GiB / 715 GiB avail; 2.8 MiB/s rd, 0 op/s 2021-07-31T01:49:05.587 INFO:tasks.rados.rados.0.smithi045.stdout:1029: done (2 left) 2021-07-31T01:49:05.588 INFO:tasks.rados.rados.0.smithi045.stdout:1031: done (1 left) 2021-07-31T01:49:05.588 INFO:tasks.rados.rados.0.smithi045.stdout:1032: done (0 left) 2021-07-31T01:49:05.588 INFO:tasks.rados.rados.0.smithi045.stdout:1033: snap_create 2021-07-31T01:49:06.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:05 smithi154 conmon[34553]: cluster 2021-07-31T01:49:04.581993+0000 mon.a (mon.0) 885 : cluster [DBG] osdmap e245: 8 total, 8 up, 8 in 2021-07-31T01:49:06.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:05 smithi045 conmon[39718]: cluster 2021-07-31T01:49:04.581993+0000 mon.a 2021-07-31T01:49:06.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:05 smithi045 conmon[39718]: (mon.0) 885 : cluster [DBG] osdmap e245: 8 total, 8 up, 8 in 2021-07-31T01:49:06.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:05 smithi045 conmon[33655]: cluster 2021-07-31T01:49:04.581993 2021-07-31T01:49:06.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:05 smithi045 conmon[33655]: +0000 mon.a (mon.0) 885 : cluster [DBG] osdmap e245: 8 total, 8 up, 8 in 2021-07-31T01:49:06.592 INFO:tasks.rados.rados.0.smithi045.stdout:1033: done (0 left) 2021-07-31T01:49:06.593 INFO:tasks.rados.rados.0.smithi045.stdout:1034: delete oid 39 current snap is 95 2021-07-31T01:49:06.596 INFO:tasks.rados.rados.0.smithi045.stdout:1034: done (0 left) 2021-07-31T01:49:06.596 INFO:tasks.rados.rados.0.smithi045.stdout:1035: rollback oid 33 current snap is 95 2021-07-31T01:49:06.596 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 33 to 93 2021-07-31T01:49:06.596 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 12 current snap is 95 2021-07-31T01:49:06.597 INFO:tasks.rados.rados.0.smithi045.stdout:1036: seq_num 318 ranges {3670016=532480} 2021-07-31T01:49:06.620 INFO:tasks.rados.rados.0.smithi045.stdout:1036: writing smithi04569269-12 from 3670016 to 4202496 tid 1 2021-07-31T01:49:06.621 INFO:tasks.rados.rados.0.smithi045.stdout:1037: rollback oid 42 current snap is 95 2021-07-31T01:49:06.621 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 42 to 90 2021-07-31T01:49:06.621 INFO:tasks.rados.rados.0.smithi045.stdout:1035: finishing rollback tid 0 to smithi04569269-33 2021-07-31T01:49:06.621 INFO:tasks.rados.rados.0.smithi045.stdout:1035: finishing rollback tid 1 to smithi04569269-33 2021-07-31T01:49:06.622 INFO:tasks.rados.rados.0.smithi045.stdout:1035: finishing rollback tid 2 to smithi04569269-33 2021-07-31T01:49:06.622 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 33 v 530 (ObjNum 142 snap 30 seq_num 142) dirty exists 2021-07-31T01:49:06.623 INFO:tasks.rados.rados.0.smithi045.stdout:1035: done (2 left) 2021-07-31T01:49:06.623 INFO:tasks.rados.rados.0.smithi045.stdout:1038: snap_remove snap 93 2021-07-31T01:49:07.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:06 smithi154 conmon[34553]: cluster 2021-07-31T01:49:05.586833+0000 mon.a (mon.0) 886 : cluster [DBG] osdmap e246: 8 total, 8 up, 8 in 2021-07-31T01:49:07.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:06 smithi154 conmon[34553]: 2021-07-31T01:49:07.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:06 smithi154 conmon[34553]: cluster 2021-07-31T01:49:05.816714+0000 mgr.x (mgr.24542) 80 : cluster [DBG] pgmap v149: 25 pgs: 25 active+clean; 160 MiB data, 741 MiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 900 KiB/s wr, 16 op/s 2021-07-31T01:49:07.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:06 smithi045 conmon[39718]: cluster 2021-07-31T01:49:05. 2021-07-31T01:49:07.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:06 smithi045 conmon[39718]: 586833+0000 mon.a (mon.0) 886 : cluster [DBG] osdmap e246: 8 total, 8 up, 8 in 2021-07-31T01:49:07.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:06 smithi045 conmon[39718]: cluster 2021-07-31T01:49:05.816714 2021-07-31T01:49:07.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:06 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 80 : cluster [DBG] pgmap v149: 25 pgs: 25 active+clean; 160 MiB data, 741 MiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 900 KiB/s wr, 16 op/s 2021-07-31T01:49:07.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:06 smithi045 conmon[33655]: cluster 2021-07-31T01:49:05.586833+0000 mon.a (mon.0) 886 : cluster [DBG] osdmap e246: 8 total, 8 up, 8 in 2021-07-31T01:49:07.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:06 smithi045 conmon[33655]: cluster 2021-07-31T01:49:05.816714+0000 mgr.x (mgr.24542) 80 : cluster [DBG] pgmap v149: 25 pgs: 25 active+clean; 160 MiB data, 741 MiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 900 KiB/s wr, 16 op/s 2021-07-31T01:49:07.598 INFO:tasks.rados.rados.0.smithi045.stdout:1038: done (2 left) 2021-07-31T01:49:07.599 INFO:tasks.rados.rados.0.smithi045.stdout:1039: read oid 31 snap -1 2021-07-31T01:49:07.599 INFO:tasks.rados.rados.0.smithi045.stdout:1039: expect deleted 2021-07-31T01:49:07.599 INFO:tasks.rados.rados.0.smithi045.stdout:1040: delete oid 28 current snap is 95 2021-07-31T01:49:07.600 INFO:tasks.rados.rados.0.smithi045.stdout:1036: finishing write tid 1 to smithi04569269-12 2021-07-31T01:49:07.600 INFO:tasks.rados.rados.0.smithi045.stdout:1036: finishing write tid 2 to smithi04569269-12 2021-07-31T01:49:07.600 INFO:tasks.rados.rados.0.smithi045.stdout:1037: finishing rollback tid 0 to smithi04569269-42 2021-07-31T01:49:07.600 INFO:tasks.rados.rados.0.smithi045.stdout:1036: finishing write tid 3 to smithi04569269-12 2021-07-31T01:49:07.601 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 12 v 532 (ObjNum 318 snap 95 seq_num 318) dirty exists 2021-07-31T01:49:07.601 INFO:tasks.rados.rados.0.smithi045.stdout:1036: left oid 12 (ObjNum 318 snap 95 seq_num 318) 2021-07-31T01:49:07.601 INFO:tasks.rados.rados.0.smithi045.stdout:1037: finishing rollback tid 1 to smithi04569269-42 2021-07-31T01:49:07.601 INFO:tasks.rados.rados.0.smithi045.stdout:1037: finishing rollback tid 2 to smithi04569269-42 2021-07-31T01:49:07.602 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 42 v 447 (ObjNum 176 snap 46 seq_num 176) dirty exists 2021-07-31T01:49:07.606 INFO:tasks.rados.rados.0.smithi045.stdout:1036: done (3 left) 2021-07-31T01:49:07.607 INFO:tasks.rados.rados.0.smithi045.stdout:1037: done (2 left) 2021-07-31T01:49:07.607 INFO:tasks.rados.rados.0.smithi045.stdout:1039: done (1 left) 2021-07-31T01:49:07.607 INFO:tasks.rados.rados.0.smithi045.stdout:1040: done (0 left) 2021-07-31T01:49:07.607 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 43 current snap is 95 2021-07-31T01:49:07.608 INFO:tasks.rados.rados.0.smithi045.stdout:1041: seq_num 320 ranges {0=565248} 2021-07-31T01:49:07.609 INFO:tasks.rados.rados.0.smithi045.stdout:1041: writing smithi04569269-43 from 0 to 565248 tid 1 2021-07-31T01:49:07.609 INFO:tasks.rados.rados.0.smithi045.stdout:1042: rollback oid 23 current snap is 95 2021-07-31T01:49:07.610 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 23 to 90 2021-07-31T01:49:07.610 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 35 current snap is 95 2021-07-31T01:49:07.610 INFO:tasks.rados.rados.0.smithi045.stdout:1043: seq_num 321 ranges {3792896=581632} 2021-07-31T01:49:07.633 INFO:tasks.rados.rados.0.smithi045.stdout:1043: writing smithi04569269-35 from 3792896 to 4374528 tid 1 2021-07-31T01:49:07.633 INFO:tasks.rados.rados.0.smithi045.stdout:1044: delete oid 36 current snap is 95 2021-07-31T01:49:07.634 INFO:tasks.rados.rados.0.smithi045.stdout:1041: finishing write tid 1 to smithi04569269-43 2021-07-31T01:49:07.634 INFO:tasks.rados.rados.0.smithi045.stdout:1042: finishing rollback tid 0 to smithi04569269-23 2021-07-31T01:49:07.634 INFO:tasks.rados.rados.0.smithi045.stdout:1042: finishing rollback tid 1 to smithi04569269-23 2021-07-31T01:49:07.634 INFO:tasks.rados.rados.0.smithi045.stdout:1041: finishing write tid 2 to smithi04569269-43 2021-07-31T01:49:07.635 INFO:tasks.rados.rados.0.smithi045.stdout:1042: finishing rollback tid 2 to smithi04569269-23 2021-07-31T01:49:07.635 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 23 v 533 (ObjNum 291 snap 85 seq_num 291) dirty exists 2021-07-31T01:49:07.635 INFO:tasks.rados.rados.0.smithi045.stdout:1041: finishing write tid 3 to smithi04569269-43 2021-07-31T01:49:07.635 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 43 v 338 (ObjNum 320 snap 95 seq_num 320) dirty exists 2021-07-31T01:49:07.635 INFO:tasks.rados.rados.0.smithi045.stdout:1041: left oid 43 (ObjNum 320 snap 95 seq_num 320) 2021-07-31T01:49:07.639 INFO:tasks.rados.rados.0.smithi045.stdout:1043: finishing write tid 1 to smithi04569269-35 2021-07-31T01:49:07.639 INFO:tasks.rados.rados.0.smithi045.stdout:1043: finishing write tid 2 to smithi04569269-35 2021-07-31T01:49:07.639 INFO:tasks.rados.rados.0.smithi045.stdout:1041: done (3 left) 2021-07-31T01:49:07.639 INFO:tasks.rados.rados.0.smithi045.stdout:1042: done (2 left) 2021-07-31T01:49:07.640 INFO:tasks.rados.rados.0.smithi045.stdout:1044: done (1 left) 2021-07-31T01:49:07.640 INFO:tasks.rados.rados.0.smithi045.stdout:1045: rollback oid 6 current snap is 95 2021-07-31T01:49:07.640 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 6 to 90 2021-07-31T01:49:07.640 INFO:tasks.rados.rados.0.smithi045.stdout:1046: snap_remove snap 94 2021-07-31T01:49:08.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:07 smithi154 conmon[34553]: cluster 2021-07-31T01:49:06.591512+0000 mon.a (mon.0) 887 : cluster [DBG] osdmap e247: 8 total, 8 up, 8 in 2021-07-31T01:49:08.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:07 smithi045 conmon[39718]: cluster 2021-07-31T01:49:06.591512+0000 mon.a (mon.0) 887 : cluster 2021-07-31T01:49:08.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:07 smithi045 conmon[39718]: [DBG] osdmap e247: 8 total, 8 up, 8 in 2021-07-31T01:49:08.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:07 smithi045 conmon[33655]: cluster 2021-07-31T01:49:06.591512+0000 mon.a (mon.0) 887 : cluster [DBG] 2021-07-31T01:49:08.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:07 smithi045 conmon[33655]: osdmap e247: 8 total, 8 up, 8 in 2021-07-31T01:49:08.605 INFO:tasks.rados.rados.0.smithi045.stdout:1046: done (2 left) 2021-07-31T01:49:08.606 INFO:tasks.rados.rados.0.smithi045.stdout:1047: rmattr oid 26 current snap is 95 2021-07-31T01:49:08.606 INFO:tasks.rados.rados.0.smithi045.stdout:1043: finishing write tid 3 to smithi04569269-35 2021-07-31T01:49:08.606 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 35 v 718 (ObjNum 321 snap 95 seq_num 321) dirty exists 2021-07-31T01:49:08.606 INFO:tasks.rados.rados.0.smithi045.stdout:1043: left oid 35 (ObjNum 321 snap 95 seq_num 321) 2021-07-31T01:49:08.607 INFO:tasks.rados.rados.0.smithi045.stdout:1045: finishing rollback tid 0 to smithi04569269-6 2021-07-31T01:49:08.607 INFO:tasks.rados.rados.0.smithi045.stdout:1045: finishing rollback tid 1 to smithi04569269-6 2021-07-31T01:49:08.607 INFO:tasks.rados.rados.0.smithi045.stdout:1045: finishing rollback tid 2 to smithi04569269-6 2021-07-31T01:49:08.607 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 6 v 535 (ObjNum 215 snap 56 seq_num 215) dirty exists 2021-07-31T01:49:08.608 INFO:tasks.rados.rados.0.smithi045.stdout:1043: done (2 left) 2021-07-31T01:49:08.608 INFO:tasks.rados.rados.0.smithi045.stdout:1045: done (1 left) 2021-07-31T01:49:08.608 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 15 current snap is 95 2021-07-31T01:49:08.608 INFO:tasks.rados.rados.0.smithi045.stdout:1048: seq_num 323 ranges {7675904=614400} 2021-07-31T01:49:08.650 INFO:tasks.rados.rados.0.smithi045.stdout:1048: writing smithi04569269-15 from 7675904 to 8290304 tid 1 2021-07-31T01:49:08.651 INFO:tasks.rados.rados.0.smithi045.stdout:1049: rollback oid 46 current snap is 95 2021-07-31T01:49:08.651 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 46 to 90 2021-07-31T01:49:08.651 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 26 v 211 (ObjNum 246 snap 68 seq_num 246) dirty exists 2021-07-31T01:49:08.651 INFO:tasks.rados.rados.0.smithi045.stdout:1047: done (2 left) 2021-07-31T01:49:08.652 INFO:tasks.rados.rados.0.smithi045.stdout:1050: rmattr oid 43 current snap is 95 2021-07-31T01:49:08.652 INFO:tasks.rados.rados.0.smithi045.stdout:1050: done (2 left) 2021-07-31T01:49:08.652 INFO:tasks.rados.rados.0.smithi045.stdout:1051: delete oid 8 current snap is 95 2021-07-31T01:49:08.654 INFO:tasks.rados.rados.0.smithi045.stdout:1051: done (2 left) 2021-07-31T01:49:08.654 INFO:tasks.rados.rados.0.smithi045.stdout:1052: read oid 41 snap -1 2021-07-31T01:49:08.654 INFO:tasks.rados.rados.0.smithi045.stdout:1052: expect deleted 2021-07-31T01:49:08.655 INFO:tasks.rados.rados.0.smithi045.stdout:1053: snap_create 2021-07-31T01:49:08.655 INFO:tasks.rados.rados.0.smithi045.stdout:1049: finishing rollback tid 0 to smithi04569269-46 2021-07-31T01:49:08.656 INFO:tasks.rados.rados.0.smithi045.stdout:1049: finishing rollback tid 1 to smithi04569269-46 2021-07-31T01:49:08.656 INFO:tasks.rados.rados.0.smithi045.stdout:1048: finishing write tid 1 to smithi04569269-15 2021-07-31T01:49:08.656 INFO:tasks.rados.rados.0.smithi045.stdout:1048: finishing write tid 2 to smithi04569269-15 2021-07-31T01:49:08.656 INFO:tasks.rados.rados.0.smithi045.stdout:1049: finishing rollback tid 2 to smithi04569269-46 2021-07-31T01:49:08.657 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 46 v 247 (ObjNum 196 snap 52 seq_num 196) dirty exists 2021-07-31T01:49:08.657 INFO:tasks.rados.rados.0.smithi045.stdout:1048: finishing write tid 3 to smithi04569269-15 2021-07-31T01:49:08.657 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 15 v 297 (ObjNum 323 snap 95 seq_num 323) dirty exists 2021-07-31T01:49:08.658 INFO:tasks.rados.rados.0.smithi045.stdout:1048: left oid 15 (ObjNum 323 snap 95 seq_num 323) 2021-07-31T01:49:09.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:08 smithi154 conmon[34553]: cluster 2021-07-31T01:49:07.597442+0000 mon.a (mon. 2021-07-31T01:49:09.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:08 smithi154 conmon[34553]: 0) 888 : cluster [DBG] osdmap e248: 8 total, 8 up, 8 in 2021-07-31T01:49:09.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:08 smithi154 conmon[34553]: cluster 2021-07-31T01:49:07.817135+0000 mgr.x (mgr.24542) 81 : cluster [DBG] pgmap v152: 25 pgs: 25 active+clean; 160 MiB data, 741 MiB used, 714 GiB / 715 GiB avail; 7.4 MiB/s rd, 900 KiB/s wr, 15 op/s 2021-07-31T01:49:09.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:08 smithi045 conmon[33655]: cluster 2021-07-31T01:49:07.597442+0000 mon.a (mon.0) 888 2021-07-31T01:49:09.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:08 smithi045 conmon[33655]: : cluster [DBG] osdmap e248: 8 total, 8 up, 8 in 2021-07-31T01:49:09.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:08 smithi045 conmon[33655]: cluster 2021-07-31T01:49:07.817135+0000 mgr.x (mgr.24542) 81 : cluster [DBG] pgmap v152: 25 pgs: 25 active+clean; 160 MiB data, 741 MiB used, 714 GiB / 715 GiB avail; 7.4 MiB/s rd, 900 KiB/s wr, 15 op/s 2021-07-31T01:49:09.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:08 smithi045 conmon[39718]: cluster 2021-07-31T01:49:07.597442+0000 mon.a (mon.0 2021-07-31T01:49:09.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:08 smithi045 conmon[39718]: ) 888 : cluster [DBG] osdmap e248: 8 total, 8 up, 8 in 2021-07-31T01:49:09.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:08 smithi045 conmon[39718]: cluster 2021-07-31T01:49:07.817135+0000 mgr.x ( 2021-07-31T01:49:09.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:08 smithi045 conmon[39718]: mgr.24542) 81 : cluster [DBG] pgmap v152: 25 pgs: 25 active+clean; 160 MiB data, 741 MiB used, 714 GiB / 715 GiB avail; 7.4 MiB/s rd, 900 KiB/s wr, 15 op/s 2021-07-31T01:49:09.605 INFO:tasks.rados.rados.0.smithi045.stdout:1048: done (3 left) 2021-07-31T01:49:09.606 INFO:tasks.rados.rados.0.smithi045.stdout:1049: done (2 left) 2021-07-31T01:49:09.606 INFO:tasks.rados.rados.0.smithi045.stdout:1052: done (1 left) 2021-07-31T01:49:09.606 INFO:tasks.rados.rados.0.smithi045.stdout:1053: done (0 left) 2021-07-31T01:49:09.607 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 40 current snap is 96 2021-07-31T01:49:09.607 INFO:tasks.rados.rados.0.smithi045.stdout:1054: seq_num 325 ranges {5029888=647168} 2021-07-31T01:49:09.637 INFO:tasks.rados.rados.0.smithi045.stdout:1054: writing smithi04569269-40 from 5029888 to 5677056 tid 1 2021-07-31T01:49:09.638 INFO:tasks.rados.rados.0.smithi045.stdout:1055: delete oid 2 current snap is 96 2021-07-31T01:49:09.639 INFO:tasks.rados.rados.0.smithi045.stdout:1055: done (1 left) 2021-07-31T01:49:09.639 INFO:tasks.rados.rados.0.smithi045.stdout:1056: snap_remove snap 95 2021-07-31T01:49:10.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:09 smithi154 conmon[34553]: cluster 2021-07-31T01:49:08.603641+0000 mon.a (mon.0) 889 : cluster [DBG] osdmap e249: 8 total, 8 up, 8 in 2021-07-31T01:49:10.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:09 smithi045 conmon[39718]: cluster 2021-07-31T01:49:08.603641 2021-07-31T01:49:10.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:09 smithi045 conmon[39718]: +0000 mon.a (mon.0) 889 : cluster [DBG] osdmap e249: 8 total, 8 up, 8 in 2021-07-31T01:49:10.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:09 smithi045 conmon[33655]: cluster 2021-07-31T01:49:08.603641 2021-07-31T01:49:10.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:09 smithi045 conmon[33655]: +0000 mon.a (mon.0) 889 : cluster [DBG] osdmap e249: 8 total, 8 up, 8 in 2021-07-31T01:49:10.617 INFO:tasks.rados.rados.0.smithi045.stdout:1056: done (1 left) 2021-07-31T01:49:10.618 INFO:tasks.rados.rados.0.smithi045.stdout:1057: copy_from oid 43 from oid 39 current snap is 96 2021-07-31T01:49:10.618 INFO:tasks.rados.rados.0.smithi045.stdout:1054: finishing write tid 1 to smithi04569269-40 2021-07-31T01:49:10.619 INFO:tasks.rados.rados.0.smithi045.stdout:1054: finishing write tid 2 to smithi04569269-40 2021-07-31T01:49:10.619 INFO:tasks.rados.rados.0.smithi045.stdout:1054: finishing write tid 3 to smithi04569269-40 2021-07-31T01:49:10.619 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 40 v 537 (ObjNum 325 snap 96 seq_num 325) dirty exists 2021-07-31T01:49:10.620 INFO:tasks.rados.rados.0.smithi045.stdout:1054: left oid 40 (ObjNum 325 snap 96 seq_num 325) 2021-07-31T01:49:10.620 INFO:tasks.rados.rados.0.smithi045.stdout:1054: done (1 left) 2021-07-31T01:49:10.621 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 49 current snap is 96 2021-07-31T01:49:10.621 INFO:tasks.rados.rados.0.smithi045.stdout:1058: seq_num 327 ranges {0=679936} 2021-07-31T01:49:10.623 INFO:tasks.rados.rados.0.smithi045.stdout:1058: writing smithi04569269-49 from 0 to 679936 tid 1 2021-07-31T01:49:10.623 INFO:tasks.rados.rados.0.smithi045.stdout:1059: delete oid 26 current snap is 96 2021-07-31T01:49:10.624 INFO:tasks.rados.rados.0.smithi045.stdout:1057: finishing copy_from to smithi04569269-43 2021-07-31T01:49:10.624 INFO:tasks.rados.rados.0.smithi045.stdout:1057: got expected ENOENT (src dne) 2021-07-31T01:49:10.624 INFO:tasks.rados.rados.0.smithi045.stdout:1057: finishing copy_from racing read to smithi04569269-43 2021-07-31T01:49:10.625 INFO:tasks.rados.rados.0.smithi045.stdout:1058: finishing write tid 1 to smithi04569269-49 2021-07-31T01:49:10.625 INFO:tasks.rados.rados.0.smithi045.stdout:1057: done (2 left) 2021-07-31T01:49:10.626 INFO:tasks.rados.rados.0.smithi045.stdout:1059: done (1 left) 2021-07-31T01:49:10.626 INFO:tasks.rados.rados.0.smithi045.stdout:1060: read oid 7 snap 90 2021-07-31T01:49:10.627 INFO:tasks.rados.rados.0.smithi045.stdout:1060: expect deleted 2021-07-31T01:49:10.631 INFO:tasks.rados.rados.0.smithi045.stdout:1061: rollback oid 18 current snap is 96 2021-07-31T01:49:10.631 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 18 to 90 2021-07-31T01:49:10.632 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 40 current snap is 96 2021-07-31T01:49:10.632 INFO:tasks.rados.rados.0.smithi045.stdout:1062: seq_num 329 ranges {5677056=712704} 2021-07-31T01:49:10.662 INFO:tasks.rados.rados.0.smithi045.stdout:1062: writing smithi04569269-40 from 5677056 to 6389760 tid 1 2021-07-31T01:49:10.662 INFO:tasks.rados.rados.0.smithi045.stdout:1063: read oid 31 snap -1 2021-07-31T01:49:10.662 INFO:tasks.rados.rados.0.smithi045.stdout:1063: expect deleted 2021-07-31T01:49:10.663 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 32 current snap is 96 2021-07-31T01:49:10.663 INFO:tasks.rados.rados.0.smithi045.stdout:1064: seq_num 330 ranges {5103616=737280} 2021-07-31T01:49:10.696 INFO:tasks.rados.rados.0.smithi045.stdout:1064: writing smithi04569269-32 from 5103616 to 5840896 tid 1 2021-07-31T01:49:10.696 INFO:tasks.rados.rados.0.smithi045.stdout:1065: read oid 24 snap -1 2021-07-31T01:49:10.697 INFO:tasks.rados.rados.0.smithi045.stdout:1065: expect (ObjNum 285 snap 83 seq_num 285) 2021-07-31T01:49:10.697 INFO:tasks.rados.rados.0.smithi045.stdout:1066: rollback oid 6 current snap is 96 2021-07-31T01:49:10.697 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 6 to 90 2021-07-31T01:49:10.698 INFO:tasks.rados.rados.0.smithi045.stdout:1058: finishing write tid 2 to smithi04569269-49 2021-07-31T01:49:10.698 INFO:tasks.rados.rados.0.smithi045.stdout:1060: done (7 left) 2021-07-31T01:49:10.699 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 20 current snap is 96 2021-07-31T01:49:10.699 INFO:tasks.rados.rados.0.smithi045.stdout:1058: finishing write tid 3 to smithi04569269-49 2021-07-31T01:49:10.702 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 49 v 727 (ObjNum 327 snap 96 seq_num 327) dirty exists 2021-07-31T01:49:10.702 INFO:tasks.rados.rados.0.smithi045.stdout:1058: left oid 49 (ObjNum 327 snap 96 seq_num 327) 2021-07-31T01:49:10.702 INFO:tasks.rados.rados.0.smithi045.stdout:1061: finishing rollback tid 0 to smithi04569269-18 2021-07-31T01:49:10.703 INFO:tasks.rados.rados.0.smithi045.stdout:1067: seq_num 331 ranges {2334720=753664} 2021-07-31T01:49:10.712 INFO:tasks.rados.rados.0.smithi045.stdout:1067: writing smithi04569269-20 from 2334720 to 3088384 tid 1 2021-07-31T01:49:10.712 INFO:tasks.rados.rados.0.smithi045.stdout:1058: done (7 left) 2021-07-31T01:49:10.712 INFO:tasks.rados.rados.0.smithi045.stdout:1068: read oid 43 snap -1 2021-07-31T01:49:10.713 INFO:tasks.rados.rados.0.smithi045.stdout:1068: expect (ObjNum 320 snap 95 seq_num 320) 2021-07-31T01:49:10.713 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 12 current snap is 96 2021-07-31T01:49:10.713 INFO:tasks.rados.rados.0.smithi045.stdout:1061: finishing rollback tid 1 to smithi04569269-18 2021-07-31T01:49:10.713 INFO:tasks.rados.rados.0.smithi045.stdout:1069: seq_num 332 ranges {4202496=770048} 2021-07-31T01:49:10.741 INFO:tasks.rados.rados.0.smithi045.stdout:1069: writing smithi04569269-12 from 4202496 to 4972544 tid 1 2021-07-31T01:49:10.856 INFO:tasks.rados.rados.0.smithi045.stdout:1070: rmattr oid 9 current snap is 96 2021-07-31T01:49:10.856 INFO:tasks.rados.rados.0.smithi045.stdout:1061: finishing rollback tid 2 to smithi04569269-18 2021-07-31T01:49:10.857 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 18 v 301 (ObjNum 172 snap 41 seq_num 172) dirty exists 2021-07-31T01:49:10.857 INFO:tasks.rados.rados.0.smithi045.stdout:1062: finishing write tid 1 to smithi04569269-40 2021-07-31T01:49:10.857 INFO:tasks.rados.rados.0.smithi045.stdout:1062: finishing write tid 2 to smithi04569269-40 2021-07-31T01:49:10.858 INFO:tasks.rados.rados.0.smithi045.stdout:1062: finishing write tid 3 to smithi04569269-40 2021-07-31T01:49:10.858 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 40 v 539 (ObjNum 329 snap 96 seq_num 329) dirty exists 2021-07-31T01:49:10.858 INFO:tasks.rados.rados.0.smithi045.stdout:1062: left oid 40 (ObjNum 329 snap 96 seq_num 329) 2021-07-31T01:49:10.858 INFO:tasks.rados.rados.0.smithi045.stdout:1061: done (9 left) 2021-07-31T01:49:10.859 INFO:tasks.rados.rados.0.smithi045.stdout:1062: done (8 left) 2021-07-31T01:49:10.859 INFO:tasks.rados.rados.0.smithi045.stdout:1063: done (7 left) 2021-07-31T01:49:10.859 INFO:tasks.rados.rados.0.smithi045.stdout:1071: read oid 10 snap -1 2021-07-31T01:49:10.860 INFO:tasks.rados.rados.0.smithi045.stdout:1071: expect (ObjNum 316 snap 93 seq_num 316) 2021-07-31T01:49:10.860 INFO:tasks.rados.rados.0.smithi045.stdout:1072: rmattr oid 34 current snap is 96 2021-07-31T01:49:10.860 INFO:tasks.rados.rados.0.smithi045.stdout:1066: finishing rollback tid 0 to smithi04569269-6 2021-07-31T01:49:10.861 INFO:tasks.rados.rados.0.smithi045.stdout:1066: finishing rollback tid 1 to smithi04569269-6 2021-07-31T01:49:10.861 INFO:tasks.rados.rados.0.smithi045.stdout:1064: finishing write tid 1 to smithi04569269-32 2021-07-31T01:49:10.861 INFO:tasks.rados.rados.0.smithi045.stdout:1064: finishing write tid 2 to smithi04569269-32 2021-07-31T01:49:10.862 INFO:tasks.rados.rados.0.smithi045.stdout:1066: finishing rollback tid 2 to smithi04569269-6 2021-07-31T01:49:10.862 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 6 v 542 (ObjNum 215 snap 56 seq_num 215) dirty exists 2021-07-31T01:49:10.862 INFO:tasks.rados.rados.0.smithi045.stdout:1064: finishing write tid 3 to smithi04569269-32 2021-07-31T01:49:10.862 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 32 v 214 (ObjNum 330 snap 96 seq_num 330) dirty exists 2021-07-31T01:49:10.863 INFO:tasks.rados.rados.0.smithi045.stdout:1064: left oid 32 (ObjNum 330 snap 96 seq_num 330) 2021-07-31T01:49:10.863 INFO:tasks.rados.rados.0.smithi045.stdout:1064: done (8 left) 2021-07-31T01:49:10.863 INFO:tasks.rados.rados.0.smithi045.stdout:1066: done (7 left) 2021-07-31T01:49:10.864 INFO:tasks.rados.rados.0.smithi045.stdout:1073: rmattr oid 8 current snap is 96 2021-07-31T01:49:10.864 INFO:tasks.rados.rados.0.smithi045.stdout:1073: done (7 left) 2021-07-31T01:49:10.864 INFO:tasks.rados.rados.0.smithi045.stdout:1074: copy_from oid 2 from oid 26 current snap is 96 2021-07-31T01:49:10.865 INFO:tasks.rados.rados.0.smithi045.stdout:1065: expect (ObjNum 285 snap 83 seq_num 285) 2021-07-31T01:49:11.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:10 smithi154 conmon[34553]: cluster 2021-07-31T01:49:09.604803+0000 mon.a (mon.0) 890 : cluster [DBG] osdmap e250: 8 total, 8 up, 8 in 2021-07-31T01:49:11.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:10 smithi154 conmon[34553]: cluster 2021-07-31T01:49:09.817538+0000 mgr.x (mgr.24542) 82 : cluster [DBG] pgmap v155: 25 pgs: 25 active+clean; 135 MiB data, 723 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 560 KiB/s wr, 8 op/s 2021-07-31T01:49:11.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:10 smithi045 conmon[33655]: cluster 2021-07-31T01:49:09.604803+0000 mon.a (mon.0) 890 : cluster 2021-07-31T01:49:11.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:10 smithi045 conmon[33655]: [DBG] osdmap e250: 8 total, 8 up, 8 in 2021-07-31T01:49:11.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:10 smithi045 conmon[33655]: cluster 2021-07-31T01:49:09.817538+0000 mgr.x (mgr.24542) 82 : cluster [DBG] pgmap v155: 25 pgs: 25 active+clean; 135 MiB data, 723 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 560 KiB/s wr, 8 op/s 2021-07-31T01:49:11.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:10 smithi045 conmon[39718]: cluster 2021-07-31T01:49:09.604803+0000 mon.a ( 2021-07-31T01:49:11.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:10 smithi045 conmon[39718]: mon.0) 890 : cluster [DBG] osdmap e250: 8 total, 8 up, 8 in 2021-07-31T01:49:11.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:10 smithi045 conmon[39718]: cluster 2021-07-31T01: 2021-07-31T01:49:11.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:10 smithi045 conmon[39718]: 49:09.817538+0000 mgr.x (mgr.24542) 82 : cluster [DBG] 2021-07-31T01:49:11.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:10 smithi045 conmon[39718]: pgmap v155: 25 pgs: 25 active+clean; 135 MiB data, 723 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 560 KiB/s wr, 8 op/s 2021-07-31T01:49:11.062 INFO:tasks.rados.rados.0.smithi045.stdout:1068: expect (ObjNum 320 snap 95 seq_num 320) 2021-07-31T01:49:11.077 INFO:tasks.rados.rados.0.smithi045.stdout:1067: finishing write tid 1 to smithi04569269-20 2021-07-31T01:49:11.077 INFO:tasks.rados.rados.0.smithi045.stdout:1067: finishing write tid 2 to smithi04569269-20 2021-07-31T01:49:11.077 INFO:tasks.rados.rados.0.smithi045.stdout:1067: finishing write tid 3 to smithi04569269-20 2021-07-31T01:49:11.077 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 20 v 729 (ObjNum 331 snap 96 seq_num 331) dirty exists 2021-07-31T01:49:11.078 INFO:tasks.rados.rados.0.smithi045.stdout:1067: left oid 20 (ObjNum 331 snap 96 seq_num 331) 2021-07-31T01:49:11.078 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 9 v 265 (ObjNum 172 snap 41 seq_num 172) dirty exists 2021-07-31T01:49:11.081 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 34 v 321 (ObjNum 76 snap 11 seq_num 76) dirty exists 2021-07-31T01:49:11.081 INFO:tasks.rados.rados.0.smithi045.stdout:1074: finishing copy_from to smithi04569269-2 2021-07-31T01:49:11.081 INFO:tasks.rados.rados.0.smithi045.stdout:1074: got expected ENOENT (src dne) 2021-07-31T01:49:11.081 INFO:tasks.rados.rados.0.smithi045.stdout:1074: finishing copy_from racing read to smithi04569269-2 2021-07-31T01:49:11.082 INFO:tasks.rados.rados.0.smithi045.stdout:1069: finishing write tid 1 to smithi04569269-12 2021-07-31T01:49:11.082 INFO:tasks.rados.rados.0.smithi045.stdout:1065: done (7 left) 2021-07-31T01:49:11.083 INFO:tasks.rados.rados.0.smithi045.stdout:1067: done (6 left) 2021-07-31T01:49:11.083 INFO:tasks.rados.rados.0.smithi045.stdout:1068: done (5 left) 2021-07-31T01:49:11.083 INFO:tasks.rados.rados.0.smithi045.stdout:1070: done (4 left) 2021-07-31T01:49:11.084 INFO:tasks.rados.rados.0.smithi045.stdout:1072: done (3 left) 2021-07-31T01:49:11.084 INFO:tasks.rados.rados.0.smithi045.stdout:1074: done (2 left) 2021-07-31T01:49:11.084 INFO:tasks.rados.rados.0.smithi045.stdout:1075: snap_create 2021-07-31T01:49:11.084 INFO:tasks.rados.rados.0.smithi045.stdout:1069: finishing write tid 2 to smithi04569269-12 2021-07-31T01:49:11.085 INFO:tasks.rados.rados.0.smithi045.stdout:1069: finishing write tid 3 to smithi04569269-12 2021-07-31T01:49:11.085 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 12 v 544 (ObjNum 332 snap 96 seq_num 332) dirty exists 2021-07-31T01:49:11.085 INFO:tasks.rados.rados.0.smithi045.stdout:1069: left oid 12 (ObjNum 332 snap 96 seq_num 332) 2021-07-31T01:49:11.085 INFO:tasks.rados.rados.0.smithi045.stdout:1071: expect (ObjNum 316 snap 93 seq_num 316) 2021-07-31T01:49:11.519 INFO:tasks.rados.rados.0.smithi045.stdout:1069: done (2 left) 2021-07-31T01:49:11.519 INFO:tasks.rados.rados.0.smithi045.stdout:1071: done (1 left) 2021-07-31T01:49:11.520 INFO:tasks.rados.rados.0.smithi045.stdout:1075: done (0 left) 2021-07-31T01:49:11.520 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 28 current snap is 97 2021-07-31T01:49:11.520 INFO:tasks.rados.rados.0.smithi045.stdout:1076: seq_num 333 ranges {0=786432} 2021-07-31T01:49:11.523 INFO:tasks.rados.rados.0.smithi045.stdout:1076: writing smithi04569269-28 from 0 to 786432 tid 1 2021-07-31T01:49:11.523 INFO:tasks.rados.rados.0.smithi045.stdout:1077: rmattr oid 39 current snap is 97 2021-07-31T01:49:11.523 INFO:tasks.rados.rados.0.smithi045.stdout:1077: done (1 left) 2021-07-31T01:49:11.524 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 34 current snap is 97 2021-07-31T01:49:11.524 INFO:tasks.rados.rados.0.smithi045.stdout:1078: seq_num 334 ranges {1974272=802816} 2021-07-31T01:49:11.539 INFO:tasks.rados.rados.0.smithi045.stdout:1078: writing smithi04569269-34 from 1974272 to 2777088 tid 1 2021-07-31T01:49:11.539 INFO:tasks.rados.rados.0.smithi045.stdout:1079: copy_from oid 3 from oid 13 current snap is 97 2021-07-31T01:49:11.539 INFO:tasks.rados.rados.0.smithi045.stdout:1076: finishing write tid 1 to smithi04569269-28 2021-07-31T01:49:11.539 INFO:tasks.rados.rados.0.smithi045.stdout:1076: finishing write tid 2 to smithi04569269-28 2021-07-31T01:49:11.540 INFO:tasks.rados.rados.0.smithi045.stdout:1076: finishing write tid 3 to smithi04569269-28 2021-07-31T01:49:11.540 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 28 v 457 (ObjNum 333 snap 97 seq_num 333) dirty exists 2021-07-31T01:49:11.540 INFO:tasks.rados.rados.0.smithi045.stdout:1076: left oid 28 (ObjNum 333 snap 97 seq_num 333) 2021-07-31T01:49:11.540 INFO:tasks.rados.rados.0.smithi045.stdout:1076: done (2 left) 2021-07-31T01:49:11.540 INFO:tasks.rados.rados.0.smithi045.stdout:1080: copy_from oid 5 from oid 45 current snap is 97 2021-07-31T01:49:11.541 INFO:tasks.rados.rados.0.smithi045.stdout:1081: snap_create 2021-07-31T01:49:11.541 INFO:tasks.rados.rados.0.smithi045.stdout:1080: finishing copy_from to smithi04569269-5 2021-07-31T01:49:11.542 INFO:tasks.rados.rados.0.smithi045.stdout:1080: got expected ENOENT (src dne) 2021-07-31T01:49:11.542 INFO:tasks.rados.rados.0.smithi045.stdout:1080: finishing copy_from racing read to smithi04569269-5 2021-07-31T01:49:11.543 INFO:tasks.rados.rados.0.smithi045.stdout:1078: finishing write tid 1 to smithi04569269-34 2021-07-31T01:49:11.544 INFO:tasks.rados.rados.0.smithi045.stdout:1078: finishing write tid 2 to smithi04569269-34 2021-07-31T01:49:11.545 INFO:tasks.rados.rados.0.smithi045.stdout:1078: finishing write tid 3 to smithi04569269-34 2021-07-31T01:49:11.546 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 34 v 323 (ObjNum 334 snap 97 seq_num 334) dirty exists 2021-07-31T01:49:11.546 INFO:tasks.rados.rados.0.smithi045.stdout:1078: left oid 34 (ObjNum 334 snap 97 seq_num 334) 2021-07-31T01:49:11.546 INFO:tasks.rados.rados.0.smithi045.stdout:1079: finishing copy_from to smithi04569269-3 2021-07-31T01:49:11.546 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 3 v 530 (ObjNum 306 snap 93 seq_num 306) dirty exists 2021-07-31T01:49:11.547 INFO:tasks.rados.rados.0.smithi045.stdout:1079: finishing copy_from racing read to smithi04569269-3 2021-07-31T01:49:12.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:11 smithi154 conmon[34553]: cluster 2021-07-31T01:49:10.615152+0000 mon.a (mon.0) 891 : cluster [DBG] osdmap e251: 8 total, 8 up, 8 in 2021-07-31T01:49:12.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:11 smithi154 conmon[34553]: 2021-07-31T01:49:12.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:11 smithi154 conmon[34553]: cluster 2021-07-31T01:49:11.518387+0000 mon.a (mon.0) 892 : cluster [DBG] osdmap e252: 8 total, 8 up, 8 in 2021-07-31T01:49:12.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:11 smithi045 conmon[33655]: cluster 2021-07-31T01:49:10 2021-07-31T01:49:12.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:11 smithi045 conmon[33655]: .615152+0000 mon.a (mon.0) 891 : cluster [DBG] osdmap e251: 8 total, 8 up, 8 in 2021-07-31T01:49:12.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:11 smithi045 conmon[33655]: cluster 2021- 2021-07-31T01:49:12.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:11 smithi045 conmon[33655]: 07-31T01:49:11.518387+0000 mon.a (mon.0 2021-07-31T01:49:12.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:11 smithi045 conmon[33655]: ) 892 : cluster [DBG] osdmap e252: 8 total, 8 up, 8 in 2021-07-31T01:49:12.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:11 smithi045 conmon[39718]: cluster 2021-07-31T01:49:10.615152+0000 2021-07-31T01:49:12.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:11 smithi045 conmon[39718]: mon.a (mon.0) 891 : cluster [DBG] osdmap e251: 8 total, 8 up, 8 in 2021-07-31T01:49:12.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:11 smithi045 conmon[39718]: cluster 2021- 2021-07-31T01:49:12.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:11 smithi045 conmon[39718]: 07-31T01:49:11.518387+0000 mon.a (mon.0) 892 : cluster 2021-07-31T01:49:12.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:11 smithi045 conmon[39718]: [DBG] osdmap e252: 8 total, 8 up, 8 in 2021-07-31T01:49:12.524 INFO:tasks.rados.rados.0.smithi045.stdout:1078: done (3 left) 2021-07-31T01:49:12.525 INFO:tasks.rados.rados.0.smithi045.stdout:1079: done (2 left) 2021-07-31T01:49:12.525 INFO:tasks.rados.rados.0.smithi045.stdout:1080: done (1 left) 2021-07-31T01:49:12.525 INFO:tasks.rados.rados.0.smithi045.stdout:1081: done (0 left) 2021-07-31T01:49:12.525 INFO:tasks.rados.rados.0.smithi045.stdout:1082: snap_create 2021-07-31T01:49:13.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:12 smithi154 conmon[34553]: cluster 2021-07-31T01:49:11.817948+0000 mgr.x (mgr.24542) 83 : cluster [DBG] pgmap v158: 25 pgs: 25 active+clean; 135 MiB data, 723 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 560 KiB/s wr, 8 op/s 2021-07-31T01:49:13.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:12 smithi154 conmon[34553]: cluster 2021-07-31T01:49:12.523746+0000 mon.a (mon.0) 893 : cluster [DBG] osdmap e253: 8 total, 8 up, 8 in 2021-07-31T01:49:13.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:12 smithi045 conmon[39718]: cluster 2021-07-31T01:49:11 2021-07-31T01:49:13.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:12 smithi045 conmon[39718]: .817948+0000 mgr.x (mgr.24542) 83 : cluster [DBG] pgmap v158: 25 pgs: 25 active+clean; 135 MiB data, 723 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 560 KiB/s wr, 8 op/s 2021-07-31T01:49:13.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:12 smithi045 conmon[39718]: cluster 2021-07-31 2021-07-31T01:49:13.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:12 smithi045 conmon[39718]: T01:49:12.523746 2021-07-31T01:49:13.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:12 smithi045 conmon[39718]: +0000 mon.a (mon.0) 893 : cluster 2021-07-31T01:49:13.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:12 smithi045 conmon[39718]: [DBG] osdmap e253: 8 total, 8 up, 8 in 2021-07-31T01:49:13.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:12 smithi045 conmon[33655]: cluster 2021-07-31T01:49:11.817948+0000 2021-07-31T01:49:13.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:12 smithi045 conmon[33655]: mgr.x (mgr.24542) 83 : cluster [DBG] pgmap v158: 25 pgs: 25 active+clean; 135 MiB data, 723 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 560 KiB/s wr, 8 op/s 2021-07-31T01:49:13.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:12 smithi045 conmon[33655]: cluster 2021-07-31T01:49:12.523746+0000 2021-07-31T01:49:13.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:12 smithi045 conmon[33655]: mon.a (mon.0) 893 : cluster [DBG] osdmap e253: 8 total, 8 up, 8 in 2021-07-31T01:49:13.526 INFO:tasks.rados.rados.0.smithi045.stdout:1082: done (0 left) 2021-07-31T01:49:13.526 INFO:tasks.rados.rados.0.smithi045.stdout:1083: snap_remove snap 97 2021-07-31T01:49:13.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:49:13 smithi045 conmon[61210]: level=error ts=2021-07-31T01:49:13.511Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:49:13.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:49:13 smithi045 conmon[61210]: level=error ts=2021-07-31T01:49:13.511Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:49:13.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:49:13 smithi045 conmon[61210]: level=error ts=2021-07-31T01:49:13.511Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T01:49:14.532 INFO:tasks.rados.rados.0.smithi045.stdout:1083: done (0 left) 2021-07-31T01:49:14.533 INFO:tasks.rados.rados.0.smithi045.stdout:1084: rollback oid 5 current snap is 99 2021-07-31T01:49:14.533 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 5 to 96 2021-07-31T01:49:14.533 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 29 current snap is 99 2021-07-31T01:49:14.534 INFO:tasks.rados.rados.0.smithi045.stdout:1085: seq_num 335 ranges {1982464=417792,2400256=401408} 2021-07-31T01:49:14.545 INFO:tasks.rados.rados.0.smithi045.stdout:1085: writing smithi04569269-29 from 1982464 to 2400256 tid 1 2021-07-31T01:49:14.547 INFO:tasks.rados.rados.0.smithi045.stdout:1085: writing smithi04569269-29 from 2400256 to 2801664 tid 2 2021-07-31T01:49:14.548 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 10 current snap is 99 2021-07-31T01:49:14.548 INFO:tasks.rados.rados.0.smithi045.stdout:1086: seq_num 336 ranges {3956736=434176,4390912=401408} 2021-07-31T01:49:14.571 INFO:tasks.rados.rados.0.smithi045.stdout:1086: writing smithi04569269-10 from 3956736 to 4390912 tid 1 2021-07-31T01:49:14.574 INFO:tasks.rados.rados.0.smithi045.stdout:1086: writing smithi04569269-10 from 4390912 to 4792320 tid 2 2021-07-31T01:49:14.574 INFO:tasks.rados.rados.0.smithi045.stdout:1087: delete oid 26 current snap is 99 2021-07-31T01:49:14.574 INFO:tasks.rados.rados.0.smithi045.stdout:1084: finishing rollback tid 0 to smithi04569269-5 2021-07-31T01:49:14.575 INFO:tasks.rados.rados.0.smithi045.stdout:1084: finishing rollback tid 1 to smithi04569269-5 2021-07-31T01:49:14.575 INFO:tasks.rados.rados.0.smithi045.stdout:1084: finishing rollback tid 2 to smithi04569269-5 2021-07-31T01:49:14.575 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 5 v 536 (ObjNum 158 snap 37 seq_num 158) dirty exists 2021-07-31T01:49:14.575 INFO:tasks.rados.rados.0.smithi045.stdout:1085: finishing write tid 1 to smithi04569269-29 2021-07-31T01:49:14.576 INFO:tasks.rados.rados.0.smithi045.stdout:1085: finishing write tid 2 to smithi04569269-29 2021-07-31T01:49:14.576 INFO:tasks.rados.rados.0.smithi045.stdout:1085: finishing write tid 3 to smithi04569269-29 2021-07-31T01:49:14.576 INFO:tasks.rados.rados.0.smithi045.stdout:1085: finishing write tid 4 to smithi04569269-29 2021-07-31T01:49:14.577 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 29 v 187 (ObjNum 335 snap 99 seq_num 335) dirty exists 2021-07-31T01:49:14.577 INFO:tasks.rados.rados.0.smithi045.stdout:1085: left oid 29 (ObjNum 335 snap 99 seq_num 335) 2021-07-31T01:49:14.578 INFO:tasks.rados.rados.0.smithi045.stdout:1084: done (3 left) 2021-07-31T01:49:14.578 INFO:tasks.rados.rados.0.smithi045.stdout:1085: done (2 left) 2021-07-31T01:49:14.578 INFO:tasks.rados.rados.0.smithi045.stdout:1087: done (1 left) 2021-07-31T01:49:14.578 INFO:tasks.rados.rados.0.smithi045.stdout:1088: read oid 39 snap -1 2021-07-31T01:49:14.579 INFO:tasks.rados.rados.0.smithi045.stdout:1088: expect deleted 2021-07-31T01:49:14.579 INFO:tasks.rados.rados.0.smithi045.stdout:1089: snap_remove snap 96 2021-07-31T01:49:15.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:14 smithi154 conmon[34553]: cluster 2021-07-31T01:49:13.525526+0000 mon.a (mon.0) 894 2021-07-31T01:49:15.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:14 smithi154 conmon[34553]: : cluster [DBG] osdmap e254: 8 total, 8 up, 8 in 2021-07-31T01:49:15.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:14 smithi154 conmon[34553]: cluster 2021-07-31T01:49:13.818371+0000 mgr.x (mgr.24542) 84 : cluster [DBG] pgmap v161: 25 pgs: 25 active+clean; 139 MiB data, 740 MiB used, 714 GiB / 715 GiB avail; 3.2 MiB/s rd, 1.1 MiB/s wr, 11 op/s 2021-07-31T01:49:15.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:14 smithi045 conmon[33655]: cluster 2021-07-31T01:49:13.525526+0000 mon.a (mon.0) 894 : 2021-07-31T01:49:15.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:14 smithi045 conmon[33655]: cluster [DBG] osdmap e254: 8 total, 8 up, 8 in 2021-07-31T01:49:15.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:14 smithi045 conmon[33655]: cluster 2021-07-31T01:49:13.818371+0000 mgr.x (mgr.24542) 84 : cluster [DBG] pgmap v161: 25 pgs: 25 active+clean; 139 MiB data, 740 MiB used, 714 GiB / 715 GiB avail; 3.2 MiB/s rd, 1.1 MiB/s wr, 11 op/s 2021-07-31T01:49:15.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:14 smithi045 conmon[39718]: cluster 2021-07-31T01:49:13.525526+0000 mon.a (mon.0 2021-07-31T01:49:15.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:14 smithi045 conmon[39718]: ) 894 : cluster [DBG] osdmap e254: 8 total, 8 up, 8 in 2021-07-31T01:49:15.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:14 smithi045 conmon[39718]: cluster 2021-07-31T01:49:13.818371+0000 mgr.x (mgr.24542) 84 : cluster [DBG] pgmap v161: 25 pgs: 25 active+clean; 139 MiB data, 740 MiB used, 714 GiB / 715 GiB avail; 3.2 MiB/s rd, 1.1 MiB/s wr, 11 op/s 2021-07-31T01:49:15.537 INFO:tasks.rados.rados.0.smithi045.stdout:1089: done (2 left) 2021-07-31T01:49:15.537 INFO:tasks.rados.rados.0.smithi045.stdout:1090: snap_remove snap 90 2021-07-31T01:49:16.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:15 smithi154 conmon[34553]: cluster 2021-07-31T01:49:14.530299+0000 mon.a (mon.0) 895 : cluster [DBG] 2021-07-31T01:49:16.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:15 smithi154 conmon[34553]: osdmap e255: 8 total, 8 up, 8 in 2021-07-31T01:49:16.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:15 smithi045 conmon[33655]: cluster 2021-07-31T01:49:14.530299+0000 mon.a 2021-07-31T01:49:16.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:15 smithi045 conmon[33655]: (mon.0) 895 : cluster [DBG] osdmap e255: 8 total, 8 up, 8 in 2021-07-31T01:49:16.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:15 smithi045 conmon[39718]: cluster 2021-07-31T01:49:14.530299+0000 mon.a (mon.0 2021-07-31T01:49:16.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:15 smithi045 conmon[39718]: ) 895 : cluster [DBG] osdmap e255: 8 total, 8 up, 8 in 2021-07-31T01:49:16.543 INFO:tasks.rados.rados.0.smithi045.stdout:1090: done (2 left) 2021-07-31T01:49:16.544 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 47 current snap is 99 2021-07-31T01:49:16.544 INFO:tasks.rados.rados.0.smithi045.stdout:1091: seq_num 338 ranges {6111232=466944,6578176=401408} 2021-07-31T01:49:16.581 INFO:tasks.rados.rados.0.smithi045.stdout:1091: writing smithi04569269-47 from 6111232 to 6578176 tid 1 2021-07-31T01:49:16.583 INFO:tasks.rados.rados.0.smithi045.stdout:1091: writing smithi04569269-47 from 6578176 to 6979584 tid 2 2021-07-31T01:49:16.584 INFO:tasks.rados.rados.0.smithi045.stdout:1092: copy_from oid 46 from oid 44 current snap is 99 2021-07-31T01:49:16.584 INFO:tasks.rados.rados.0.smithi045.stdout:1086: finishing write tid 1 to smithi04569269-10 2021-07-31T01:49:16.585 INFO:tasks.rados.rados.0.smithi045.stdout:1086: finishing write tid 2 to smithi04569269-10 2021-07-31T01:49:16.585 INFO:tasks.rados.rados.0.smithi045.stdout:1088: done (3 left) 2021-07-31T01:49:16.585 INFO:tasks.rados.rados.0.smithi045.stdout:1093: snap_create 2021-07-31T01:49:16.585 INFO:tasks.rados.rados.0.smithi045.stdout:1086: finishing write tid 3 to smithi04569269-10 2021-07-31T01:49:16.585 INFO:tasks.rados.rados.0.smithi045.stdout:1086: finishing write tid 4 to smithi04569269-10 2021-07-31T01:49:16.586 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 10 v 461 (ObjNum 336 snap 99 seq_num 336) dirty exists 2021-07-31T01:49:16.586 INFO:tasks.rados.rados.0.smithi045.stdout:1086: left oid 10 (ObjNum 336 snap 99 seq_num 336) 2021-07-31T01:49:16.587 INFO:tasks.rados.rados.0.smithi045.stdout:1091: finishing write tid 1 to smithi04569269-47 2021-07-31T01:49:16.587 INFO:tasks.rados.rados.0.smithi045.stdout:1091: finishing write tid 2 to smithi04569269-47 2021-07-31T01:49:16.587 INFO:tasks.rados.rados.0.smithi045.stdout:1091: finishing write tid 3 to smithi04569269-47 2021-07-31T01:49:16.587 INFO:tasks.rados.rados.0.smithi045.stdout:1091: finishing write tid 4 to smithi04569269-47 2021-07-31T01:49:16.588 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 47 v 191 (ObjNum 338 snap 99 seq_num 338) dirty exists 2021-07-31T01:49:16.588 INFO:tasks.rados.rados.0.smithi045.stdout:1091: left oid 47 (ObjNum 338 snap 99 seq_num 338) 2021-07-31T01:49:16.602 INFO:tasks.rados.rados.0.smithi045.stdout:1092: finishing copy_from to smithi04569269-46 2021-07-31T01:49:16.602 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 46 v 257 (ObjNum 142 snap 30 seq_num 142) dirty exists 2021-07-31T01:49:16.603 INFO:tasks.rados.rados.0.smithi045.stdout:1092: finishing copy_from racing read to smithi04569269-46 2021-07-31T01:49:17.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:16 smithi154 conmon[34553]: cluster 2021-07-31T01:49:15.536665+0000 mon.a (mon.0) 896 : cluster [DBG] osdmap e256: 8 total, 8 up, 8 in 2021-07-31T01:49:17.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:16 smithi154 conmon[34553]: cluster 2021-07-31T01:49:15.818791+0000 mgr.x (mgr.24542) 85 : cluster [DBG] pgmap v164: 25 pgs: 25 active+clean; 141 MiB data, 748 MiB used, 714 GiB / 715 GiB avail; 7.7 MiB/s rd, 1.5 MiB/s wr, 18 op/s 2021-07-31T01:49:17.004 INFO:journalctl@ceph.osd.6.smithi154.stdout:Jul 31 01:49:16 smithi154 conmon[45576]: debug 2021-07-31T01:49:17.004 INFO:journalctl@ceph.osd.6.smithi154.stdout:Jul 31 01:49:16 smithi154 conmon[45576]: 2021-07-31T01:49:16.552+0000 7f47c50ad700 -1 osd.6 pg_epoch: 257 pg[3.as0( v 248'338 (0'0,248'338] local-lis/les=56/57 n=6 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [6,4,1]p6(0) r=0 lpr=56 crt=248'338 lcod 248'337 mlcod 248'337 active+clean+snaptrim trimq=[b4~1] ps=[bf~1,c1~2]] removing snap head 2021-07-31T01:49:17.004 INFO:journalctl@ceph.osd.4.smithi154.stdout:Jul 31 01:49:16 smithi154 conmon[38506]: debug 2021-07-31T01:49:16.545+0000 7feae79d7700 -1 osd.4 pg_epoch: 257 pg[3.9s0( v 245'106 (0'0,245'106] local-lis/les=56/57 n=2 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [4,2,7]p4(0) r=0 lpr=56 crt=245'106 lcod 244'104 mlcod 244'104 active+clean+snaptrim trimq=[b4~1] ps=[bf~1,c1~2]] removing snap head 2021-07-31T01:49:17.005 INFO:journalctl@ceph.osd.7.smithi154.stdout:Jul 31 01:49:16 smithi154 conmon[49149]: debug 2021-07-31T01:49:16.549+0000 7f8a74a02700 -1 osd.7 pg_epoch: 257 pg[3.fs0( v 257'176 (0'0,257'176] local-lis/les=56/57 n=3 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [7,4,0]p7(0) r=0 lpr=56 luod=255'174 crt=251'173 lcod 251'173 mlcod 251'173 active+clean+snaptrim trimq=[b4~1] ps=[bf~1,c1~2]] removing snap head 2021-07-31T01:49:17.026 INFO:journalctl@ceph.osd.0.smithi045.stdout:Jul 31 01:49:16 smithi045 conmon[43243]: debug 2021-07-31T01:49:17.026 INFO:journalctl@ceph.osd.0.smithi045.stdout:Jul 31 01:49:16 smithi045 conmon[43243]: 2021-07-31T01:49:16.549+0000 7fdf6978b700 -1 osd.0 pg_epoch: 257 pg[3.6s0( v 257'135 (0'0,257'135] local-lis/les=56/57 n=3 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [0,1,4]p0(0) r=0 lpr=56 luod=251'133 crt=251'133 lcod 251'131 mlcod 251'131 active+clean+snaptrim trimq=[b4~1] ps=[bf~1,c1~2]] removing snap head 2021-07-31T01:49:17.026 INFO:journalctl@ceph.osd.0.smithi045.stdout:Jul 31 01:49:16 smithi045 conmon[43243]: 2021-07-31T01:49:17.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:16 smithi045 conmon[33655]: cluster 2021-07-31T01:49:15.536665+0000 mon.a (mon.0) 896 : cluster 2021-07-31T01:49:17.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:16 smithi045 conmon[33655]: [DBG] osdmap e256: 8 total, 8 up, 8 in 2021-07-31T01:49:17.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:16 smithi045 conmon[33655]: cluster 2021-07-31T01:49:15.818791+0000 mgr.x (mgr.24542) 85 : cluster [DBG] pgmap v164: 25 pgs: 25 active+clean; 141 MiB data, 748 MiB used, 714 GiB / 715 GiB avail; 7.7 MiB/s rd, 1.5 MiB/s wr, 18 op/s 2021-07-31T01:49:17.028 INFO:journalctl@ceph.osd.3.smithi045.stdout:Jul 31 01:49:16 smithi045 conmon[54013]: debug 2021-07-31T01:49:16.557+0000 7f64fc8cd700 -1 osd.3 pg_epoch: 257 pg[3.bs0( v 257'732 (0'0,257'732] local-lis/les=56/57 n=10 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [3,0,4]p3(0) r=0 lpr=56 luod=252'730 crt=252'730 lcod 251'729 mlcod 251'729 active+clean+snaptrim trimq=[b4~1] ps=[bf~1,c1~2]] removing snap head 2021-07-31T01:49:17.028 INFO:journalctl@ceph.osd.3.smithi045.stdout:Jul 31 01:49:16 smithi045 conmon[54013]: debug 2021-07-31T01:49:17.028 INFO:journalctl@ceph.osd.3.smithi045.stdout:Jul 31 01:49:16 smithi045 conmon[54013]: 2021-07-31T01:49:16.564+0000 7f64f88c5700 -1 osd.3 pg_epoch: 257 pg[3.bs0( v 257'734 (0'0,257'734] local-lis/les=56/57 n=8 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [3,0,4]p3(0) r=0 lpr=56 crt=257'734 lcod 257'732 mlcod 257'732 active+clean+snaptrim trimq=[b4~1] ps=[bf~1,c1~2]] removing snap head 2021-07-31T01:49:17.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:16 smithi045 conmon[39718]: cluster 2021-07-31T01:49:15 2021-07-31T01:49:17.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:16 smithi045 conmon[39718]: .536665+0000 mon.a (mon.0) 896 : cluster [DBG] osdmap e256: 8 total, 8 up, 8 in 2021-07-31T01:49:17.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:16 smithi045 conmon[39718]: cluster 2021 2021-07-31T01:49:17.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:16 smithi045 conmon[39718]: -07-31T01:49:15. 2021-07-31T01:49:17.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:16 smithi045 conmon[39718]: 818791+0000 mgr.x (mgr. 2021-07-31T01:49:17.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:16 smithi045 conmon[39718]: 24542) 85 : cluster [DBG] pgmap v164: 25 pgs: 25 active+clean; 141 MiB data, 748 MiB used, 714 GiB / 715 GiB avail; 7.7 MiB/s rd, 1.5 MiB/s wr, 18 op/s 2021-07-31T01:49:17.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:16 smithi045 conmon[39718]: 2021-07-31T01:49:17.545 INFO:tasks.rados.rados.0.smithi045.stdout:1086: done (3 left) 2021-07-31T01:49:17.545 INFO:tasks.rados.rados.0.smithi045.stdout:1091: done (2 left) 2021-07-31T01:49:17.546 INFO:tasks.rados.rados.0.smithi045.stdout:1092: done (1 left) 2021-07-31T01:49:17.546 INFO:tasks.rados.rados.0.smithi045.stdout:1093: done (0 left) 2021-07-31T01:49:17.546 INFO:tasks.rados.rados.0.smithi045.stdout:1094: delete oid 3 current snap is 100 2021-07-31T01:49:17.552 INFO:tasks.rados.rados.0.smithi045.stdout:1094: done (0 left) 2021-07-31T01:49:17.552 INFO:tasks.rados.rados.0.smithi045.stdout:1095: read oid 6 snap -1 2021-07-31T01:49:17.553 INFO:tasks.rados.rados.0.smithi045.stdout:1095: expect (ObjNum 215 snap 56 seq_num 215) 2021-07-31T01:49:17.553 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 31 current snap is 100 2021-07-31T01:49:17.553 INFO:tasks.rados.rados.0.smithi045.stdout:1096: seq_num 340 ranges {0=499712,499712=401408} 2021-07-31T01:49:17.555 INFO:tasks.rados.rados.0.smithi045.stdout:1096: writing smithi04569269-31 from 0 to 499712 tid 1 2021-07-31T01:49:17.557 INFO:tasks.rados.rados.0.smithi045.stdout:1096: writing smithi04569269-31 from 499712 to 901120 tid 2 2021-07-31T01:49:17.557 INFO:tasks.rados.rados.0.smithi045.stdout:1097: snap_create 2021-07-31T01:49:17.562 INFO:tasks.rados.rados.0.smithi045.stdout:1096: finishing write tid 1 to smithi04569269-31 2021-07-31T01:49:17.562 INFO:tasks.rados.rados.0.smithi045.stdout:1096: finishing write tid 2 to smithi04569269-31 2021-07-31T01:49:17.563 INFO:tasks.rados.rados.0.smithi045.stdout:1096: finishing write tid 3 to smithi04569269-31 2021-07-31T01:49:17.563 INFO:tasks.rados.rados.0.smithi045.stdout:1096: finishing write tid 4 to smithi04569269-31 2021-07-31T01:49:17.564 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 31 v 743 (ObjNum 340 snap 100 seq_num 340) dirty exists 2021-07-31T01:49:17.564 INFO:tasks.rados.rados.0.smithi045.stdout:1096: left oid 31 (ObjNum 340 snap 100 seq_num 340) 2021-07-31T01:49:17.565 INFO:tasks.rados.rados.0.smithi045.stdout:1095: expect (ObjNum 215 snap 56 seq_num 215) 2021-07-31T01:49:18.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:17 smithi154 conmon[34553]: cluster 2021-07-31T01:49:16.543045+0000 mon.a (mon.0) 897 : cluster [DBG] 2021-07-31T01:49:18.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:17 smithi154 conmon[34553]: osdmap e257: 8 total, 8 up, 8 in 2021-07-31T01:49:18.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:17 smithi045 conmon[33655]: cluster 2021-07-31T01:49:16.543045+0000 mon.a (mon.0) 897 : cluster [DBG] 2021-07-31T01:49:18.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:17 smithi045 conmon[33655]: osdmap e257: 8 total, 8 up, 8 in 2021-07-31T01:49:18.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:17 smithi045 conmon[39718]: cluster 2021-07-31T01:49:16.543045+0000 mon.a ( 2021-07-31T01:49:18.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:17 smithi045 conmon[39718]: mon.0) 897 : cluster [DBG] osdmap e257: 8 total, 8 up, 8 in 2021-07-31T01:49:18.555 INFO:tasks.rados.rados.0.smithi045.stdout:1095: done (2 left) 2021-07-31T01:49:18.555 INFO:tasks.rados.rados.0.smithi045.stdout:1096: done (1 left) 2021-07-31T01:49:18.556 INFO:tasks.rados.rados.0.smithi045.stdout:1097: done (0 left) 2021-07-31T01:49:18.556 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 34 current snap is 101 2021-07-31T01:49:18.556 INFO:tasks.rados.rados.0.smithi045.stdout:1098: seq_num 341 ranges {2777088=516096,3293184=401408} 2021-07-31T01:49:18.573 INFO:tasks.rados.rados.0.smithi045.stdout:1098: writing smithi04569269-34 from 2777088 to 3293184 tid 1 2021-07-31T01:49:18.575 INFO:tasks.rados.rados.0.smithi045.stdout:1098: writing smithi04569269-34 from 3293184 to 3694592 tid 2 2021-07-31T01:49:18.575 INFO:tasks.rados.rados.0.smithi045.stdout:1099: read oid 37 snap -1 2021-07-31T01:49:18.575 INFO:tasks.rados.rados.0.smithi045.stdout:1099: expect (ObjNum 255 snap 75 seq_num 255) 2021-07-31T01:49:18.576 INFO:tasks.rados.rados.0.smithi045.stdout:1100: snap_create 2021-07-31T01:49:18.580 INFO:tasks.rados.rados.0.smithi045.stdout:1098: finishing write tid 1 to smithi04569269-34 2021-07-31T01:49:18.581 INFO:tasks.rados.rados.0.smithi045.stdout:1098: finishing write tid 2 to smithi04569269-34 2021-07-31T01:49:18.582 INFO:tasks.rados.rados.0.smithi045.stdout:1098: finishing write tid 3 to smithi04569269-34 2021-07-31T01:49:18.582 INFO:tasks.rados.rados.0.smithi045.stdout:1099: expect (ObjNum 255 snap 75 seq_num 255) 2021-07-31T01:49:18.631 INFO:tasks.rados.rados.0.smithi045.stdout:1098: finishing write tid 4 to smithi04569269-34 2021-07-31T01:49:18.632 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 34 v 326 (ObjNum 341 snap 101 seq_num 341) dirty exists 2021-07-31T01:49:18.632 INFO:tasks.rados.rados.0.smithi045.stdout:1098: left oid 34 (ObjNum 341 snap 101 seq_num 341) 2021-07-31T01:49:19.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:18 smithi154 conmon[34553]: cluster 2021-07-31T01:49:17.544123+0000 mon.a (mon.0) 898 : 2021-07-31T01:49:19.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:18 smithi154 conmon[34553]: cluster [DBG] osdmap e258: 8 total, 8 up, 8 in 2021-07-31T01:49:19.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:18 smithi154 conmon[34553]: cluster 2021-07-31T01:49:17.819204+0000 mgr.x (mgr.24542) 86 : 2021-07-31T01:49:19.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:18 smithi154 conmon[34553]: cluster [DBG] pgmap v167: 25 pgs: 25 active+clean; 141 MiB data, 748 MiB used, 714 GiB / 715 GiB avail; 4.4 MiB/s rd, 494 KiB/s wr, 6 op/s 2021-07-31T01:49:19.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:18 smithi045 conmon[33655]: cluster 2021-07-31T01:49:17.544123+0000 mon.a (mon.0) 2021-07-31T01:49:19.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:18 smithi045 conmon[33655]: 898 : cluster [DBG] osdmap e258: 8 total, 8 up, 8 in 2021-07-31T01:49:19.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:18 smithi045 conmon[33655]: cluster 2021-07-31T01:49:17.819204+0000 mgr.x (mgr.24542) 86 : cluster [DBG] pgmap v167: 25 pgs: 25 active+clean; 141 MiB data, 748 MiB used, 714 GiB / 715 GiB avail; 4.4 MiB/s rd, 494 KiB/s wr, 6 op/s 2021-07-31T01:49:19.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:18 smithi045 conmon[39718]: cluster 2021-07-31T01:49:17 2021-07-31T01:49:19.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:18 smithi045 conmon[39718]: .544123+0000 mon.a (mon.0) 898 : cluster [DBG] osdmap e258: 8 total, 8 up, 8 in 2021-07-31T01:49:19.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:18 smithi045 conmon[39718]: cluster 2021-07-31T01:49:19.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:18 smithi045 conmon[39718]: 2021-07-31T01 2021-07-31T01:49:19.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:18 smithi045 conmon[39718]: :49:17.819204+0000 mgr.x ( 2021-07-31T01:49:19.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:18 smithi045 conmon[39718]: mgr.24542) 86 : cluster [DBG] 2021-07-31T01:49:19.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:18 smithi045 conmon[39718]: pgmap v167: 25 pgs: 25 active+clean; 141 MiB data, 748 MiB used, 714 GiB / 715 GiB avail; 4.4 MiB/s rd, 494 KiB/s wr, 6 op/s 2021-07-31T01:49:19.560 INFO:tasks.rados.rados.0.smithi045.stdout:1098: done (2 left) 2021-07-31T01:49:19.560 INFO:tasks.rados.rados.0.smithi045.stdout:1099: done (1 left) 2021-07-31T01:49:19.561 INFO:tasks.rados.rados.0.smithi045.stdout:1100: done (0 left) 2021-07-31T01:49:19.561 INFO:tasks.rados.rados.0.smithi045.stdout:1101: snap_remove snap 98 2021-07-31T01:49:20.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:19 smithi154 conmon[34553]: cluster 2021-07-31T01:49:18.553995+0000 mon.a (mon.0) 899 : cluster [DBG] 2021-07-31T01:49:20.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:19 smithi154 conmon[34553]: osdmap e259: 8 total, 8 up, 8 in 2021-07-31T01:49:20.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:19 smithi045 conmon[39718]: cluster 2021-07-31T01:49:18.553995+0000 mon.a (mon.0) 899 : cluster [DBG] osdmap e259: 8 total, 8 up, 8 in 2021-07-31T01:49:20.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:19 smithi045 conmon[33655]: cluster 2021-07-31T01:49:18.553995+0000 2021-07-31T01:49:20.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:19 smithi045 conmon[33655]: mon.a (mon.0) 899 : cluster [DBG] osdmap e259: 8 total, 8 up, 8 in 2021-07-31T01:49:20.565 INFO:tasks.rados.rados.0.smithi045.stdout:1101: done (0 left) 2021-07-31T01:49:20.566 INFO:tasks.rados.rados.0.smithi045.stdout:1102: read oid 9 snap -1 2021-07-31T01:49:20.566 INFO:tasks.rados.rados.0.smithi045.stdout:1102: expect (ObjNum 172 snap 41 seq_num 172) 2021-07-31T01:49:20.566 INFO:tasks.rados.rados.0.smithi045.stdout:1103: snap_remove snap 100 2021-07-31T01:49:21.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:20 smithi154 conmon[34553]: cluster 2021-07-31T01:49:19.558951+0000 mon.a (mon.0) 900 : cluster [DBG] osdmap e260: 8 total, 8 up, 8 in 2021-07-31T01:49:21.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:20 smithi154 conmon[34553]: cluster 2021-07-31T01:49:19.819766+0000 mgr.x (mgr.24542) 87 : cluster [DBG] pgmap v170: 25 pgs: 25 active+clean; 121 MiB data, 734 MiB used, 715 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.1 MiB/s wr, 8 op/s 2021-07-31T01:49:21.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:20 smithi045 conmon[33655]: cluster 2021-07-31T01:49:19.558951+0000 mon.a 2021-07-31T01:49:21.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:20 smithi045 conmon[33655]: (mon.0) 900 : cluster [DBG] osdmap e260: 8 total, 8 up, 8 in 2021-07-31T01:49:21.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:20 smithi045 conmon[33655]: cluster 2021-07-31T01:49:19.819766+0000 mgr.x (mgr.24542 2021-07-31T01:49:21.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:20 smithi045 conmon[33655]: ) 87 : cluster [DBG] pgmap v170: 25 pgs: 25 active+clean; 121 MiB data, 734 MiB used, 715 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.1 MiB/s wr, 8 op/s 2021-07-31T01:49:21.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:20 smithi045 conmon[39718]: cluster 2021-07-31T01:49:19.558951+0000 mon.a (mon 2021-07-31T01:49:21.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:20 smithi045 conmon[39718]: .0) 900 : cluster [DBG] osdmap e260: 8 total, 8 up, 8 in 2021-07-31T01:49:21.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:20 smithi045 conmon[39718]: cluster 2021-07-31T01:49:19.819766 2021-07-31T01:49:21.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:20 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 87 : cluster [DBG] pgmap v170: 25 pgs: 25 active+clean; 121 MiB data, 734 MiB used, 715 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.1 MiB/s wr, 8 op/s 2021-07-31T01:49:21.522 INFO:tasks.rados.rados.0.smithi045.stdout:1103: done (1 left) 2021-07-31T01:49:21.522 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 39 current snap is 102 2021-07-31T01:49:21.522 INFO:tasks.rados.rados.0.smithi045.stdout:1104: seq_num 342 ranges {0=532480,532480=401408} 2021-07-31T01:49:21.525 INFO:tasks.rados.rados.0.smithi045.stdout:1104: writing smithi04569269-39 from 0 to 532480 tid 1 2021-07-31T01:49:21.527 INFO:tasks.rados.rados.0.smithi045.stdout:1104: writing smithi04569269-39 from 532480 to 933888 tid 2 2021-07-31T01:49:21.527 INFO:tasks.rados.rados.0.smithi045.stdout:1105: rollback oid 43 current snap is 102 2021-07-31T01:49:21.528 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 43 to 101 2021-07-31T01:49:21.528 INFO:tasks.rados.rados.0.smithi045.stdout:1102: expect (ObjNum 172 snap 41 seq_num 172) 2021-07-31T01:49:21.548 INFO:tasks.rados.rados.0.smithi045.stdout:1104: finishing write tid 1 to smithi04569269-39 2021-07-31T01:49:21.549 INFO:tasks.rados.rados.0.smithi045.stdout:1104: finishing write tid 2 to smithi04569269-39 2021-07-31T01:49:21.549 INFO:tasks.rados.rados.0.smithi045.stdout:1104: finishing write tid 3 to smithi04569269-39 2021-07-31T01:49:21.549 INFO:tasks.rados.rados.0.smithi045.stdout:1105: finishing rollback tid 0 to smithi04569269-43 2021-07-31T01:49:21.550 INFO:tasks.rados.rados.0.smithi045.stdout:1105: finishing rollback tid 1 to smithi04569269-43 2021-07-31T01:49:21.550 INFO:tasks.rados.rados.0.smithi045.stdout:1104: finishing write tid 4 to smithi04569269-39 2021-07-31T01:49:21.550 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 39 v 539 (ObjNum 342 snap 102 seq_num 342) dirty exists 2021-07-31T01:49:21.550 INFO:tasks.rados.rados.0.smithi045.stdout:1104: left oid 39 (ObjNum 342 snap 102 seq_num 342) 2021-07-31T01:49:21.550 INFO:tasks.rados.rados.0.smithi045.stdout:1105: finishing rollback tid 2 to smithi04569269-43 2021-07-31T01:49:21.551 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 43 v 346 (ObjNum 320 snap 95 seq_num 320) dirty exists 2021-07-31T01:49:21.551 INFO:tasks.rados.rados.0.smithi045.stdout:1102: done (2 left) 2021-07-31T01:49:21.551 INFO:tasks.rados.rados.0.smithi045.stdout:1104: done (1 left) 2021-07-31T01:49:21.551 INFO:tasks.rados.rados.0.smithi045.stdout:1105: done (0 left) 2021-07-31T01:49:21.551 INFO:tasks.rados.rados.0.smithi045.stdout:1106: snap_remove snap 99 2021-07-31T01:49:22.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:21 smithi154 conmon[34553]: cluster 2021-07-31T01:49:20.564772+0000 mon.a (mon.0) 901 : cluster [DBG] osdmap e261: 8 total, 8 up, 8 in 2021-07-31T01:49:22.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:21 smithi154 conmon[34553]: 2021-07-31T01:49:22.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:21 smithi154 conmon[34553]: cluster 2021-07-31T01:49:21.520819+0000 mon.a (mon.0) 902 : cluster [DBG] osdmap e262: 8 total, 8 up, 8 in 2021-07-31T01:49:22.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:21 smithi045 conmon[39718]: cluster 2021-07-31T01:49:20.564772+0000 mon.a (mon.0) 901 : cluster [DBG] osdmap e261: 8 total, 8 up, 8 in 2021-07-31T01:49:22.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:21 smithi045 conmon[39718]: cluster 2021-07-31T01:49:21.520819+0000 mon.a (mon.0) 902 : cluster [DBG] osdmap e262: 8 total, 8 up, 8 in 2021-07-31T01:49:22.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:21 smithi045 conmon[33655]: cluster 2021-07-31T01:49:20.564772+0000 mon.a (mon.0) 2021-07-31T01:49:22.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:21 smithi045 conmon[33655]: 901 : cluster [DBG] osdmap e261: 8 total, 8 up, 8 in 2021-07-31T01:49:22.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:21 smithi045 conmon[33655]: cluster 2021-07-31T01:49:21.520819+0000 mon.a (mon.0) 902 : cluster [DBG] osdmap e262: 8 total, 8 up, 8 in 2021-07-31T01:49:22.528 INFO:tasks.rados.rados.0.smithi045.stdout:1106: done (0 left) 2021-07-31T01:49:22.529 INFO:tasks.rados.rados.0.smithi045.stdout:1107: copy_from oid 33 from oid 48 current snap is 102 2021-07-31T01:49:22.529 INFO:tasks.rados.rados.0.smithi045.stdout:1108: snap_create 2021-07-31T01:49:22.555 INFO:tasks.rados.rados.0.smithi045.stdout:1107: finishing copy_from to smithi04569269-33 2021-07-31T01:49:22.556 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 33 v 287 (ObjNum 275 snap 81 seq_num 275) dirty exists 2021-07-31T01:49:22.556 INFO:tasks.rados.rados.0.smithi045.stdout:1107: finishing copy_from racing read to smithi04569269-33 2021-07-31T01:49:23.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:22 smithi154 conmon[34553]: cluster 2021-07-31T01:49:21.820291+0000 mgr.x (mgr.24542) 88 : cluster 2021-07-31T01:49:23.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:22 smithi154 conmon[34553]: [DBG] pgmap v173: 25 pgs: 25 active+clean; 121 MiB data, 734 MiB used, 715 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.1 MiB/s wr, 8 op/s 2021-07-31T01:49:23.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:22 smithi154 conmon[34553]: cluster 2021-07-31T01:49:22.527541+0000 mon.a (mon.0) 903 : cluster [DBG] osdmap e263: 8 total, 8 up, 8 in 2021-07-31T01:49:23.026 INFO:journalctl@ceph.osd.3.smithi045.stdout:Jul 31 01:49:22 smithi045 conmon[54013]: debug 2021-07-31T01:49:22.535+0000 7f64f88c5700 -1 osd.3 pg_epoch: 263 pg[3.bs0( v 261'745 (0'0,261'745] local-lis/les=56/57 n=7 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [3,0,4]p3(0) r=0 lpr=56 crt=261'745 lcod 258'743 mlcod 258'743 active+clean+snaptrim trimq=[c7~1] ps=[c3~1,c8~1]] removing snap head 2021-07-31T01:49:23.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:22 smithi045 conmon[39718]: cluster 2021-07-31T01: 2021-07-31T01:49:23.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:22 smithi045 conmon[39718]: 49:21.820291+0000 mgr.x (mgr.24542) 88 : cluster [DBG] pgmap v173: 25 pgs: 25 active+clean; 121 MiB data, 734 MiB used, 715 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.1 MiB/s wr, 8 op/s 2021-07-31T01:49:23.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:22 smithi045 conmon[39718]: cluster 2021 2021-07-31T01:49:23.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:22 smithi045 conmon[39718]: -07-31T01:49:22. 2021-07-31T01:49:23.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:22 smithi045 conmon[39718]: 527541+0000 mon.a (mon.0 2021-07-31T01:49:23.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:22 smithi045 conmon[39718]: ) 903 : cluster [DBG] osdmap e263: 8 total, 8 up, 8 in 2021-07-31T01:49:23.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:22 smithi045 conmon[39718]: 2021-07-31T01:49:23.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:22 smithi045 conmon[33655]: cluster 2021-07-31T01:49:21.820291+0000 mgr.x (mgr.24542) 88 : 2021-07-31T01:49:23.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:22 smithi045 conmon[33655]: cluster [DBG] pgmap v173: 25 pgs: 25 active+clean; 121 MiB data, 734 MiB used, 715 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.1 MiB/s wr, 8 op/s 2021-07-31T01:49:23.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:22 smithi045 conmon[33655]: cluster 2021-07-31T01:49:22.527541+0000 mon.a (mon.0) 903 : cluster [DBG] osdmap e263: 8 total, 8 up, 8 in 2021-07-31T01:49:23.529 INFO:tasks.rados.rados.0.smithi045.stdout:1107: done (1 left) 2021-07-31T01:49:23.529 INFO:tasks.rados.rados.0.smithi045.stdout:1108: done (0 left) 2021-07-31T01:49:23.530 INFO:tasks.rados.rados.0.smithi045.stdout:1109: copy_from oid 35 from oid 22 current snap is 103 2021-07-31T01:49:23.531 INFO:tasks.rados.rados.0.smithi045.stdout:1110: rollback oid 20 current snap is 103 2021-07-31T01:49:23.531 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 20 to 102 2021-07-31T01:49:23.531 INFO:tasks.rados.rados.0.smithi045.stdout:1111: delete oid 27 current snap is 103 2021-07-31T01:49:23.532 INFO:tasks.rados.rados.0.smithi045.stdout:1111: done (2 left) 2021-07-31T01:49:23.532 INFO:tasks.rados.rados.0.smithi045.stdout:1112: rollback oid 36 current snap is 103 2021-07-31T01:49:23.532 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 36 to 102 2021-07-31T01:49:23.533 INFO:tasks.rados.rados.0.smithi045.stdout:1113: delete oid 7 current snap is 103 2021-07-31T01:49:23.535 INFO:tasks.rados.rados.0.smithi045.stdout:1113: done (3 left) 2021-07-31T01:49:23.536 INFO:tasks.rados.rados.0.smithi045.stdout:1114: copy_from oid 1 from oid 4 current snap is 103 2021-07-31T01:49:23.536 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 21 current snap is 103 2021-07-31T01:49:23.536 INFO:tasks.rados.rados.0.smithi045.stdout:1115: seq_num 345 ranges {8110080=581632,8691712=401408} 2021-07-31T01:49:23.583 INFO:tasks.rados.rados.0.smithi045.stdout:1115: writing smithi04569269-21 from 8110080 to 8691712 tid 1 2021-07-31T01:49:23.586 INFO:tasks.rados.rados.0.smithi045.stdout:1115: writing smithi04569269-21 from 8691712 to 9093120 tid 2 2021-07-31T01:49:23.586 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 30 current snap is 103 2021-07-31T01:49:23.586 INFO:tasks.rados.rados.0.smithi045.stdout:1110: finishing rollback tid 0 to smithi04569269-20 2021-07-31T01:49:23.587 INFO:tasks.rados.rados.0.smithi045.stdout:1116: seq_num 346 ranges {1974272=598016,2572288=401408} 2021-07-31T01:49:23.600 INFO:tasks.rados.rados.0.smithi045.stdout:1116: writing smithi04569269-30 from 1974272 to 2572288 tid 1 2021-07-31T01:49:23.603 INFO:tasks.rados.rados.0.smithi045.stdout:1116: writing smithi04569269-30 from 2572288 to 2973696 tid 2 2021-07-31T01:49:23.603 INFO:tasks.rados.rados.0.smithi045.stdout:1117: rollback oid 10 current snap is 103 2021-07-31T01:49:23.603 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 10 to 102 2021-07-31T01:49:23.604 INFO:tasks.rados.rados.0.smithi045.stdout:1110: finishing rollback tid 1 to smithi04569269-20 2021-07-31T01:49:23.604 INFO:tasks.rados.rados.0.smithi045.stdout:1110: finishing rollback tid 2 to smithi04569269-20 2021-07-31T01:49:23.604 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 20 v 751 (ObjNum 331 snap 96 seq_num 331) dirty exists 2021-07-31T01:49:23.604 INFO:tasks.rados.rados.0.smithi045.stdout:1112: finishing rollback tid 1 to smithi04569269-36 2021-07-31T01:49:23.605 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 36 v 0 (ObjNum 1666339411 snap 1617261151 seq_num 1566729057) dirty dne 2021-07-31T01:49:23.605 INFO:tasks.rados.rados.0.smithi045.stdout:1109: finishing copy_from to smithi04569269-35 2021-07-31T01:49:23.605 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 35 v 198 (ObjNum 311 snap 93 seq_num 311) dirty exists 2021-07-31T01:49:23.605 INFO:tasks.rados.rados.0.smithi045.stdout:1109: finishing copy_from racing read to smithi04569269-35 2021-07-31T01:49:23.605 INFO:tasks.rados.rados.0.smithi045.stdout:1114: finishing copy_from to smithi04569269-1 2021-07-31T01:49:23.606 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 1 v 250 (ObjNum 268 snap 80 seq_num 268) dirty exists 2021-07-31T01:49:23.606 INFO:tasks.rados.rados.0.smithi045.stdout:1114: finishing copy_from racing read to smithi04569269-1 2021-07-31T01:49:23.606 INFO:tasks.rados.rados.0.smithi045.stdout:1109: done (6 left) 2021-07-31T01:49:23.606 INFO:tasks.rados.rados.0.smithi045.stdout:1110: done (5 left) 2021-07-31T01:49:23.607 INFO:tasks.rados.rados.0.smithi045.stdout:1112: done (4 left) 2021-07-31T01:49:23.607 INFO:tasks.rados.rados.0.smithi045.stdout:1114: done (3 left) 2021-07-31T01:49:23.607 INFO:tasks.rados.rados.0.smithi045.stdout:1118: read oid 11 snap -1 2021-07-31T01:49:23.607 INFO:tasks.rados.rados.0.smithi045.stdout:1118: expect deleted 2021-07-31T01:49:23.607 INFO:tasks.rados.rados.0.smithi045.stdout:1119: snap_remove snap 101 2021-07-31T01:49:23.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:49:23 smithi045 conmon[61210]: level=error ts=2021-07-31T01:49:23.512Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:49:23.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:49:23 smithi045 conmon[61210]: level=error ts=2021-07-31T01:49:23.512Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:49:23.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:49:23 smithi045 conmon[61210]: level=error ts=2021-07-31T01:49:23.512Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T01:49:24.535 INFO:tasks.rados.rados.0.smithi045.stdout:1119: done (4 left) 2021-07-31T01:49:24.535 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 49 current snap is 103 2021-07-31T01:49:24.536 INFO:tasks.rados.rados.0.smithi045.stdout:1115: finishing write tid 1 to smithi04569269-21 2021-07-31T01:49:24.536 INFO:tasks.rados.rados.0.smithi045.stdout:1120: seq_num 347 ranges {679936=614400,1294336=401408} 2021-07-31T01:49:24.541 INFO:tasks.rados.rados.0.smithi045.stdout:1120: writing smithi04569269-49 from 679936 to 1294336 tid 1 2021-07-31T01:49:24.544 INFO:tasks.rados.rados.0.smithi045.stdout:1120: writing smithi04569269-49 from 1294336 to 1695744 tid 2 2021-07-31T01:49:24.545 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 50 current snap is 103 2021-07-31T01:49:24.545 INFO:tasks.rados.rados.0.smithi045.stdout:1115: finishing write tid 2 to smithi04569269-21 2021-07-31T01:49:24.545 INFO:tasks.rados.rados.0.smithi045.stdout:1115: finishing write tid 3 to smithi04569269-21 2021-07-31T01:49:24.545 INFO:tasks.rados.rados.0.smithi045.stdout:1115: finishing write tid 4 to smithi04569269-21 2021-07-31T01:49:24.546 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 21 v 757 (ObjNum 345 snap 103 seq_num 345) dirty exists 2021-07-31T01:49:24.546 INFO:tasks.rados.rados.0.smithi045.stdout:1115: left oid 21 (ObjNum 345 snap 103 seq_num 345) 2021-07-31T01:49:24.546 INFO:tasks.rados.rados.0.smithi045.stdout:1116: finishing write tid 1 to smithi04569269-30 2021-07-31T01:49:24.546 INFO:tasks.rados.rados.0.smithi045.stdout:1116: finishing write tid 2 to smithi04569269-30 2021-07-31T01:49:24.546 INFO:tasks.rados.rados.0.smithi045.stdout:1116: finishing write tid 3 to smithi04569269-30 2021-07-31T01:49:24.547 INFO:tasks.rados.rados.0.smithi045.stdout:1117: finishing rollback tid 0 to smithi04569269-10 2021-07-31T01:49:24.547 INFO:tasks.rados.rados.0.smithi045.stdout:1116: finishing write tid 4 to smithi04569269-30 2021-07-31T01:49:24.547 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 30 v 310 (ObjNum 346 snap 103 seq_num 346) dirty exists 2021-07-31T01:49:24.547 INFO:tasks.rados.rados.0.smithi045.stdout:1116: left oid 30 (ObjNum 346 snap 103 seq_num 346) 2021-07-31T01:49:24.548 INFO:tasks.rados.rados.0.smithi045.stdout:1117: finishing rollback tid 1 to smithi04569269-10 2021-07-31T01:49:24.548 INFO:tasks.rados.rados.0.smithi045.stdout:1117: finishing rollback tid 2 to smithi04569269-10 2021-07-31T01:49:24.548 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 10 v 477 (ObjNum 336 snap 99 seq_num 336) dirty exists 2021-07-31T01:49:24.548 INFO:tasks.rados.rados.0.smithi045.stdout:1121: seq_num 348 ranges {1974272=630784,2605056=401408} 2021-07-31T01:49:24.559 INFO:tasks.rados.rados.0.smithi045.stdout:1121: writing smithi04569269-50 from 1974272 to 2605056 tid 1 2021-07-31T01:49:24.562 INFO:tasks.rados.rados.0.smithi045.stdout:1121: writing smithi04569269-50 from 2605056 to 3006464 tid 2 2021-07-31T01:49:24.563 INFO:tasks.rados.rados.0.smithi045.stdout:1115: done (5 left) 2021-07-31T01:49:24.563 INFO:tasks.rados.rados.0.smithi045.stdout:1116: done (4 left) 2021-07-31T01:49:24.563 INFO:tasks.rados.rados.0.smithi045.stdout:1117: done (3 left) 2021-07-31T01:49:24.563 INFO:tasks.rados.rados.0.smithi045.stdout:1118: done (2 left) 2021-07-31T01:49:24.564 INFO:tasks.rados.rados.0.smithi045.stdout:1122: snap_create 2021-07-31T01:49:24.564 INFO:tasks.rados.rados.0.smithi045.stdout:1120: finishing write tid 1 to smithi04569269-49 2021-07-31T01:49:24.564 INFO:tasks.rados.rados.0.smithi045.stdout:1120: finishing write tid 2 to smithi04569269-49 2021-07-31T01:49:24.564 INFO:tasks.rados.rados.0.smithi045.stdout:1120: finishing write tid 3 to smithi04569269-49 2021-07-31T01:49:24.564 INFO:tasks.rados.rados.0.smithi045.stdout:1120: finishing write tid 4 to smithi04569269-49 2021-07-31T01:49:24.565 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 49 v 761 (ObjNum 347 snap 103 seq_num 347) dirty exists 2021-07-31T01:49:24.565 INFO:tasks.rados.rados.0.smithi045.stdout:1120: left oid 49 (ObjNum 347 snap 103 seq_num 347) 2021-07-31T01:49:24.566 INFO:tasks.rados.rados.0.smithi045.stdout:1121: finishing write tid 1 to smithi04569269-50 2021-07-31T01:49:24.566 INFO:tasks.rados.rados.0.smithi045.stdout:1121: finishing write tid 2 to smithi04569269-50 2021-07-31T01:49:24.566 INFO:tasks.rados.rados.0.smithi045.stdout:1121: finishing write tid 3 to smithi04569269-50 2021-07-31T01:49:24.568 INFO:tasks.rados.rados.0.smithi045.stdout:1121: finishing write tid 4 to smithi04569269-50 2021-07-31T01:49:24.568 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 50 v 483 (ObjNum 348 snap 103 seq_num 348) dirty exists 2021-07-31T01:49:24.569 INFO:tasks.rados.rados.0.smithi045.stdout:1121: left oid 50 (ObjNum 348 snap 103 seq_num 348) 2021-07-31T01:49:25.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:24 smithi154 conmon[34553]: cluster 2021-07-31T01:49:23.528306+0000 mon.a (mon.0) 904 : cluster [DBG] 2021-07-31T01:49:25.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:24 smithi154 conmon[34553]: osdmap e264: 8 total, 8 up, 8 in 2021-07-31T01:49:25.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:24 smithi154 conmon[34553]: cluster 2021-07-31T01:49:23.820849+0000 mgr.x (mgr.24542) 89 : cluster [DBG] pgmap v176: 25 pgs: 25 active+clean; 126 MiB data, 742 MiB used, 714 GiB / 715 GiB avail; 255 B/s rd, 2.4 MiB/s wr, 2 op/s 2021-07-31T01:49:25.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:24 smithi045 conmon[33655]: cluster 2021-07-31T01:49:23.528306+0000 mon.a (mon.0) 904 2021-07-31T01:49:25.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:24 smithi045 conmon[33655]: : cluster [DBG] osdmap e264: 8 total, 8 up, 8 in 2021-07-31T01:49:25.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:24 smithi045 conmon[33655]: cluster 2021-07-31T01:49:23.820849+0000 mgr.x (mgr.24542) 89 : cluster [DBG] pgmap v176: 25 pgs: 25 active+clean; 126 MiB data, 742 MiB used, 714 GiB / 715 GiB avail; 255 B/s rd, 2.4 MiB/s wr, 2 op/s 2021-07-31T01:49:25.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:24 smithi045 conmon[39718]: cluster 2021-07-31T01:49:23 2021-07-31T01:49:25.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:24 smithi045 conmon[39718]: .528306+0000 mon.a (mon.0) 904 : cluster [DBG] osdmap e264: 8 total, 8 up, 8 in 2021-07-31T01:49:25.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:24 smithi045 conmon[39718]: cluster 2021-07-31T01:49:25.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:24 smithi045 conmon[39718]: 2021-07-31 2021-07-31T01:49:25.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:24 smithi045 conmon[39718]: T01:49:23.820849+0000 mgr.x (mgr.24542) 89 : cluster [DBG] pgmap v176: 25 pgs: 25 active+clean; 126 MiB data, 742 MiB used, 714 GiB / 715 GiB avail; 255 B/s rd, 2.4 MiB/s wr, 2 op/s 2021-07-31T01:49:25.536 INFO:tasks.rados.rados.0.smithi045.stdout:1120: done (2 left) 2021-07-31T01:49:25.537 INFO:tasks.rados.rados.0.smithi045.stdout:1121: done (1 left) 2021-07-31T01:49:25.537 INFO:tasks.rados.rados.0.smithi045.stdout:1122: done (0 left) 2021-07-31T01:49:25.537 INFO:tasks.rados.rados.0.smithi045.stdout:1123: snap_create 2021-07-31T01:49:26.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:25 smithi154 conmon[34553]: cluster 2021-07-31T01:49:24.534083+0000 mon.a (mon.0) 905 : cluster [DBG] osdmap e265: 8 total, 8 up, 8 in 2021-07-31T01:49:26.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:25 smithi045 conmon[39718]: cluster 2021-07-31T01:49:24.534083+0000 mon.a (mon 2021-07-31T01:49:26.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:25 smithi045 conmon[39718]: .0) 905 : cluster [DBG] osdmap e265: 8 total, 8 up, 8 in 2021-07-31T01:49:26.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:25 smithi045 conmon[33655]: cluster 2021-07-31T01:49:24. 2021-07-31T01:49:26.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:25 smithi045 conmon[33655]: 534083+0000 mon.a (mon.0) 905 : cluster [DBG] osdmap e265: 8 total, 8 up, 8 in 2021-07-31T01:49:26.548 INFO:tasks.rados.rados.0.smithi045.stdout:1123: done (0 left) 2021-07-31T01:49:26.548 INFO:tasks.rados.rados.0.smithi045.stdout:1124: rollback oid 44 current snap is 105 2021-07-31T01:49:26.549 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 44 to 102 2021-07-31T01:49:26.549 INFO:tasks.rados.rados.0.smithi045.stdout:1125: read oid 36 snap -1 2021-07-31T01:49:26.550 INFO:tasks.rados.rados.0.smithi045.stdout:1125: expect deleted 2021-07-31T01:49:26.550 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 1 current snap is 105 2021-07-31T01:49:26.550 INFO:tasks.rados.rados.0.smithi045.stdout:1126: seq_num 349 ranges {5218304=655360,5873664=401408} 2021-07-31T01:49:26.583 INFO:tasks.rados.rados.0.smithi045.stdout:1126: writing smithi04569269-1 from 5218304 to 5873664 tid 1 2021-07-31T01:49:26.583 INFO:tasks.rados.rados.0.smithi045.stdout:1126: writing smithi04569269-1 from 5873664 to 6275072 tid 2 2021-07-31T01:49:26.584 INFO:tasks.rados.rados.0.smithi045.stdout:1127: delete oid 50 current snap is 105 2021-07-31T01:49:26.584 INFO:tasks.rados.rados.0.smithi045.stdout:1124: finishing rollback tid 0 to smithi04569269-44 2021-07-31T01:49:26.584 INFO:tasks.rados.rados.0.smithi045.stdout:1124: finishing rollback tid 1 to smithi04569269-44 2021-07-31T01:49:26.584 INFO:tasks.rados.rados.0.smithi045.stdout:1124: finishing rollback tid 2 to smithi04569269-44 2021-07-31T01:49:26.585 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 44 v 355 (ObjNum 142 snap 30 seq_num 142) dirty exists 2021-07-31T01:49:26.585 INFO:tasks.rados.rados.0.smithi045.stdout:1126: finishing write tid 1 to smithi04569269-1 2021-07-31T01:49:26.586 INFO:tasks.rados.rados.0.smithi045.stdout:1126: finishing write tid 2 to smithi04569269-1 2021-07-31T01:49:26.586 INFO:tasks.rados.rados.0.smithi045.stdout:1126: finishing write tid 3 to smithi04569269-1 2021-07-31T01:49:26.587 INFO:tasks.rados.rados.0.smithi045.stdout:1126: finishing write tid 4 to smithi04569269-1 2021-07-31T01:49:26.587 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 1 v 329 (ObjNum 349 snap 105 seq_num 349) dirty exists 2021-07-31T01:49:26.587 INFO:tasks.rados.rados.0.smithi045.stdout:1126: left oid 1 (ObjNum 349 snap 105 seq_num 349) 2021-07-31T01:49:26.588 INFO:tasks.rados.rados.0.smithi045.stdout:1124: done (3 left) 2021-07-31T01:49:26.588 INFO:tasks.rados.rados.0.smithi045.stdout:1125: done (2 left) 2021-07-31T01:49:26.589 INFO:tasks.rados.rados.0.smithi045.stdout:1126: done (1 left) 2021-07-31T01:49:26.589 INFO:tasks.rados.rados.0.smithi045.stdout:1127: done (0 left) 2021-07-31T01:49:26.589 INFO:tasks.rados.rados.0.smithi045.stdout:1128: read oid 15 snap -1 2021-07-31T01:49:26.589 INFO:tasks.rados.rados.0.smithi045.stdout:1128: expect (ObjNum 323 snap 95 seq_num 323) 2021-07-31T01:49:26.590 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 44 current snap is 105 2021-07-31T01:49:26.590 INFO:tasks.rados.rados.0.smithi045.stdout:1129: seq_num 351 ranges {4308992=688128,4997120=401408} 2021-07-31T01:49:26.615 INFO:tasks.rados.rados.0.smithi045.stdout:1129: writing smithi04569269-44 from 4308992 to 4997120 tid 1 2021-07-31T01:49:26.618 INFO:tasks.rados.rados.0.smithi045.stdout:1129: writing smithi04569269-44 from 4997120 to 5398528 tid 2 2021-07-31T01:49:26.619 INFO:tasks.rados.rados.0.smithi045.stdout:1130: snap_create 2021-07-31T01:49:26.619 INFO:tasks.rados.rados.0.smithi045.stdout:1128: expect (ObjNum 323 snap 95 seq_num 323) 2021-07-31T01:49:27.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:26 smithi154 conmon[34553]: cluster 2021-07-31T01:49:25.535502+0000 mon.a (mon.0) 906 : 2021-07-31T01:49:27.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:26 smithi154 conmon[34553]: cluster [DBG] osdmap e266: 8 total, 8 up, 8 in 2021-07-31T01:49:27.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:26 smithi154 conmon[34553]: cluster 2021-07-31T01:49:25.821363+0000 mgr.x (mgr.24542) 90 : cluster [DBG] pgmap v179: 25 pgs: 25 active+clean; 128 MiB data, 744 MiB used, 714 GiB / 715 GiB avail; 380 KiB/s rd, 3.1 MiB/s wr, 6 op/s 2021-07-31T01:49:27.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:26 smithi045 conmon[33655]: cluster 2021-07-31T01:49:25.535502+0000 mon.a (mon.0) 906 : cluster 2021-07-31T01:49:27.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:26 smithi045 conmon[33655]: [DBG] osdmap e266: 8 total, 8 up, 8 in 2021-07-31T01:49:27.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:26 smithi045 conmon[33655]: cluster 2021-07-31T01:49:25.821363+0000 mgr.x (mgr.24542) 90 : cluster [DBG] pgmap v179: 25 pgs: 25 active+clean; 128 MiB data, 744 MiB used, 714 GiB / 715 GiB avail; 380 KiB/s rd, 3.1 MiB/s wr, 6 op/s 2021-07-31T01:49:27.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:26 smithi045 conmon[39718]: cluster 2021-07-31T01:49:25 2021-07-31T01:49:27.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:26 smithi045 conmon[39718]: .535502+0000 mon.a (mon.0) 906 : cluster [DBG] osdmap e266: 8 total, 8 up, 8 in 2021-07-31T01:49:27.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:26 smithi045 conmon[39718]: cluster 2021- 2021-07-31T01:49:27.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:26 smithi045 conmon[39718]: 07-31T01:49:25 2021-07-31T01:49:27.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:26 smithi045 conmon[39718]: .821363+0000 mgr.x (mgr. 2021-07-31T01:49:27.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:26 smithi045 conmon[39718]: 24542) 90 : cluster [DBG] pgmap v179: 25 pgs: 25 active+clean; 128 MiB data, 744 MiB used, 714 GiB / 715 GiB avail; 380 KiB/s rd, 3.1 MiB/s wr, 6 op/s 2021-07-31T01:49:27.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:26 smithi045 conmon[39718]: 2021-07-31T01:49:27.245 INFO:tasks.rados.rados.0.smithi045.stdout:1129: finishing write tid 1 to smithi04569269-44 2021-07-31T01:49:27.245 INFO:tasks.rados.rados.0.smithi045.stdout:1129: finishing write tid 2 to smithi04569269-44 2021-07-31T01:49:27.245 INFO:tasks.rados.rados.0.smithi045.stdout:1129: finishing write tid 3 to smithi04569269-44 2021-07-31T01:49:27.246 INFO:tasks.rados.rados.0.smithi045.stdout:1129: finishing write tid 4 to smithi04569269-44 2021-07-31T01:49:27.246 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 44 v 358 (ObjNum 351 snap 105 seq_num 351) dirty exists 2021-07-31T01:49:27.246 INFO:tasks.rados.rados.0.smithi045.stdout:1129: left oid 44 (ObjNum 351 snap 105 seq_num 351) 2021-07-31T01:49:27.553 INFO:tasks.rados.rados.0.smithi045.stdout:1128: done (2 left) 2021-07-31T01:49:27.554 INFO:tasks.rados.rados.0.smithi045.stdout:1129: done (1 left) 2021-07-31T01:49:27.554 INFO:tasks.rados.rados.0.smithi045.stdout:1130: done (0 left) 2021-07-31T01:49:27.555 INFO:tasks.rados.rados.0.smithi045.stdout:1131: read oid 49 snap -1 2021-07-31T01:49:27.555 INFO:tasks.rados.rados.0.smithi045.stdout:1131: expect (ObjNum 347 snap 103 seq_num 347) 2021-07-31T01:49:27.555 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 45 current snap is 106 2021-07-31T01:49:27.555 INFO:tasks.rados.rados.0.smithi045.stdout:1132: seq_num 352 ranges {0=704512,704512=401408} 2021-07-31T01:49:27.557 INFO:tasks.rados.rados.0.smithi045.stdout:1132: writing smithi04569269-45 from 0 to 704512 tid 1 2021-07-31T01:49:27.560 INFO:tasks.rados.rados.0.smithi045.stdout:1132: writing smithi04569269-45 from 704512 to 1105920 tid 2 2021-07-31T01:49:27.561 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 18 current snap is 106 2021-07-31T01:49:27.561 INFO:tasks.rados.rados.0.smithi045.stdout:1133: seq_num 353 ranges {483328=720896,1204224=401408} 2021-07-31T01:49:27.568 INFO:tasks.rados.rados.0.smithi045.stdout:1133: writing smithi04569269-18 from 483328 to 1204224 tid 1 2021-07-31T01:49:27.570 INFO:tasks.rados.rados.0.smithi045.stdout:1133: writing smithi04569269-18 from 1204224 to 1605632 tid 2 2021-07-31T01:49:27.570 INFO:tasks.rados.rados.0.smithi045.stdout:1134: rollback oid 27 current snap is 106 2021-07-31T01:49:27.570 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 27 to 105 2021-07-31T01:49:27.571 INFO:tasks.rados.rados.0.smithi045.stdout:1132: finishing write tid 1 to smithi04569269-45 2021-07-31T01:49:27.571 INFO:tasks.rados.rados.0.smithi045.stdout:1132: finishing write tid 2 to smithi04569269-45 2021-07-31T01:49:27.571 INFO:tasks.rados.rados.0.smithi045.stdout:1132: finishing write tid 3 to smithi04569269-45 2021-07-31T01:49:27.572 INFO:tasks.rados.rados.0.smithi045.stdout:1132: finishing write tid 4 to smithi04569269-45 2021-07-31T01:49:27.572 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 45 v 488 (ObjNum 352 snap 106 seq_num 352) dirty exists 2021-07-31T01:49:27.572 INFO:tasks.rados.rados.0.smithi045.stdout:1132: left oid 45 (ObjNum 352 snap 106 seq_num 352) 2021-07-31T01:49:27.572 INFO:tasks.rados.rados.0.smithi045.stdout:1131: expect (ObjNum 347 snap 103 seq_num 347) 2021-07-31T01:49:27.630 INFO:tasks.rados.rados.0.smithi045.stdout:1134: finishing rollback tid 1 to smithi04569269-27 2021-07-31T01:49:27.630 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 27 v 0 (ObjNum 288 snap 0 seq_num 273) dirty dne 2021-07-31T01:49:27.631 INFO:tasks.rados.rados.0.smithi045.stdout:1133: finishing write tid 1 to smithi04569269-18 2021-07-31T01:49:27.631 INFO:tasks.rados.rados.0.smithi045.stdout:1133: finishing write tid 2 to smithi04569269-18 2021-07-31T01:49:27.631 INFO:tasks.rados.rados.0.smithi045.stdout:1133: finishing write tid 3 to smithi04569269-18 2021-07-31T01:49:27.632 INFO:tasks.rados.rados.0.smithi045.stdout:1131: done (3 left) 2021-07-31T01:49:27.632 INFO:tasks.rados.rados.0.smithi045.stdout:1132: done (2 left) 2021-07-31T01:49:27.632 INFO:tasks.rados.rados.0.smithi045.stdout:1134: done (1 left) 2021-07-31T01:49:27.632 INFO:tasks.rados.rados.0.smithi045.stdout:1135: read oid 50 snap -1 2021-07-31T01:49:27.633 INFO:tasks.rados.rados.0.smithi045.stdout:1135: expect deleted 2021-07-31T01:49:27.633 INFO:tasks.rados.rados.0.smithi045.stdout:1136: read oid 22 snap -1 2021-07-31T01:49:27.633 INFO:tasks.rados.rados.0.smithi045.stdout:1136: expect (ObjNum 311 snap 93 seq_num 311) 2021-07-31T01:49:27.633 INFO:tasks.rados.rados.0.smithi045.stdout:1137: read oid 9 snap 103 2021-07-31T01:49:27.633 INFO:tasks.rados.rados.0.smithi045.stdout:1137: expect (ObjNum 172 snap 41 seq_num 172) 2021-07-31T01:49:27.634 INFO:tasks.rados.rados.0.smithi045.stdout:1138: rollback oid 3 current snap is 106 2021-07-31T01:49:27.634 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 3 to 102 2021-07-31T01:49:27.634 INFO:tasks.rados.rados.0.smithi045.stdout:1133: finishing write tid 4 to smithi04569269-18 2021-07-31T01:49:27.635 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 18 v 309 (ObjNum 353 snap 106 seq_num 353) dirty exists 2021-07-31T01:49:27.635 INFO:tasks.rados.rados.0.smithi045.stdout:1133: left oid 18 (ObjNum 353 snap 106 seq_num 353) 2021-07-31T01:49:27.635 INFO:tasks.rados.rados.0.smithi045.stdout:1133: done (4 left) 2021-07-31T01:49:27.636 INFO:tasks.rados.rados.0.smithi045.stdout:1139: rmattr oid 14 current snap is 106 2021-07-31T01:49:27.636 INFO:tasks.rados.rados.0.smithi045.stdout:1139: done (4 left) 2021-07-31T01:49:27.637 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 19 current snap is 106 2021-07-31T01:49:27.637 INFO:tasks.rados.rados.0.smithi045.stdout:1140: seq_num 354 ranges {2867200=737280,3604480=401408} 2021-07-31T01:49:27.648 INFO:tasks.rados.rados.0.smithi045.stdout:1140: writing smithi04569269-19 from 2867200 to 3604480 tid 1 2021-07-31T01:49:27.651 INFO:tasks.rados.rados.0.smithi045.stdout:1140: writing smithi04569269-19 from 3604480 to 4005888 tid 2 2021-07-31T01:49:27.651 INFO:tasks.rados.rados.0.smithi045.stdout:1141: read oid 26 snap -1 2021-07-31T01:49:27.652 INFO:tasks.rados.rados.0.smithi045.stdout:1141: expect deleted 2021-07-31T01:49:27.652 INFO:tasks.rados.rados.0.smithi045.stdout:1142: delete oid 25 current snap is 106 2021-07-31T01:49:27.652 INFO:tasks.rados.rados.0.smithi045.stdout:1138: finishing rollback tid 1 to smithi04569269-3 2021-07-31T01:49:27.653 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 3 v 0 (ObjNum 0 snap 0 seq_num 3188975360) dirty dne 2021-07-31T01:49:27.653 INFO:tasks.rados.rados.0.smithi045.stdout:1137: expect (ObjNum 172 snap 41 seq_num 172) 2021-07-31T01:49:27.667 INFO:tasks.rados.rados.0.smithi045.stdout:1136: expect (ObjNum 311 snap 93 seq_num 311) 2021-07-31T01:49:27.713 INFO:tasks.rados.rados.0.smithi045.stdout:1140: finishing write tid 1 to smithi04569269-19 2021-07-31T01:49:27.713 INFO:tasks.rados.rados.0.smithi045.stdout:1140: finishing write tid 2 to smithi04569269-19 2021-07-31T01:49:27.714 INFO:tasks.rados.rados.0.smithi045.stdout:1140: finishing write tid 3 to smithi04569269-19 2021-07-31T01:49:27.714 INFO:tasks.rados.rados.0.smithi045.stdout:1140: finishing write tid 4 to smithi04569269-19 2021-07-31T01:49:27.714 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 19 v 208 (ObjNum 354 snap 106 seq_num 354) dirty exists 2021-07-31T01:49:27.714 INFO:tasks.rados.rados.0.smithi045.stdout:1140: left oid 19 (ObjNum 354 snap 106 seq_num 354) 2021-07-31T01:49:27.714 INFO:tasks.rados.rados.0.smithi045.stdout:1135: done (6 left) 2021-07-31T01:49:27.715 INFO:tasks.rados.rados.0.smithi045.stdout:1136: done (5 left) 2021-07-31T01:49:27.715 INFO:tasks.rados.rados.0.smithi045.stdout:1137: done (4 left) 2021-07-31T01:49:27.715 INFO:tasks.rados.rados.0.smithi045.stdout:1138: done (3 left) 2021-07-31T01:49:27.715 INFO:tasks.rados.rados.0.smithi045.stdout:1140: done (2 left) 2021-07-31T01:49:27.716 INFO:tasks.rados.rados.0.smithi045.stdout:1141: done (1 left) 2021-07-31T01:49:27.716 INFO:tasks.rados.rados.0.smithi045.stdout:1142: done (0 left) 2021-07-31T01:49:27.716 INFO:tasks.rados.rados.0.smithi045.stdout:1143: rollback oid 34 current snap is 106 2021-07-31T01:49:27.716 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 34 to 103 2021-07-31T01:49:27.716 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 2 current snap is 106 2021-07-31T01:49:27.717 INFO:tasks.rados.rados.0.smithi045.stdout:1144: seq_num 356 ranges {0=770048,770048=401408} 2021-07-31T01:49:27.717 INFO:tasks.rados.rados.0.smithi045.stdout:1144: writing smithi04569269-2 from 0 to 770048 tid 1 2021-07-31T01:49:27.720 INFO:tasks.rados.rados.0.smithi045.stdout:1144: writing smithi04569269-2 from 770048 to 1171456 tid 2 2021-07-31T01:49:27.720 INFO:tasks.rados.rados.0.smithi045.stdout:1145: read oid 15 snap -1 2021-07-31T01:49:27.720 INFO:tasks.rados.rados.0.smithi045.stdout:1145: expect (ObjNum 323 snap 95 seq_num 323) 2021-07-31T01:49:27.720 INFO:tasks.rados.rados.0.smithi045.stdout:1146: read oid 31 snap 105 2021-07-31T01:49:27.720 INFO:tasks.rados.rados.0.smithi045.stdout:1146: expect (ObjNum 340 snap 100 seq_num 340) 2021-07-31T01:49:27.721 INFO:tasks.rados.rados.0.smithi045.stdout:1147: snap_remove snap 102 2021-07-31T01:49:28.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:27 smithi154 conmon[34553]: cluster 2021-07-31T01:49:26.546904+0000 mon.a (mon.0) 907 : cluster [DBG] osdmap e267: 8 total, 8 up, 8 in 2021-07-31T01:49:28.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:27 smithi045 conmon[33655]: cluster 2021-07-31T01:49:26.546904+0000 mon.a 2021-07-31T01:49:28.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:27 smithi045 conmon[33655]: (mon.0) 907 : cluster [DBG] osdmap e267: 8 total, 8 up, 8 in 2021-07-31T01:49:28.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:27 smithi045 conmon[39718]: cluster 2021-07-31T01:49:26.546904 2021-07-31T01:49:28.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:27 smithi045 conmon[39718]: +0000 mon.a (mon.0) 907 : cluster [DBG] osdmap e267: 8 total, 8 up, 8 in 2021-07-31T01:49:28.562 INFO:tasks.rados.rados.0.smithi045.stdout:1143: finishing rollback tid 0 to smithi04569269-34 2021-07-31T01:49:28.562 INFO:tasks.rados.rados.0.smithi045.stdout:1143: finishing rollback tid 1 to smithi04569269-34 2021-07-31T01:49:28.563 INFO:tasks.rados.rados.0.smithi045.stdout:1143: finishing rollback tid 2 to smithi04569269-34 2021-07-31T01:49:28.563 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 34 v 332 (ObjNum 341 snap 101 seq_num 341) dirty exists 2021-07-31T01:49:28.563 INFO:tasks.rados.rados.0.smithi045.stdout:1144: finishing write tid 1 to smithi04569269-2 2021-07-31T01:49:28.563 INFO:tasks.rados.rados.0.smithi045.stdout:1146: expect (ObjNum 340 snap 100 seq_num 340) 2021-07-31T01:49:28.588 INFO:tasks.rados.rados.0.smithi045.stdout:1144: finishing write tid 2 to smithi04569269-2 2021-07-31T01:49:28.589 INFO:tasks.rados.rados.0.smithi045.stdout:1144: finishing write tid 3 to smithi04569269-2 2021-07-31T01:49:28.589 INFO:tasks.rados.rados.0.smithi045.stdout:1144: finishing write tid 4 to smithi04569269-2 2021-07-31T01:49:28.589 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 2 v 268 (ObjNum 356 snap 106 seq_num 356) dirty exists 2021-07-31T01:49:28.589 INFO:tasks.rados.rados.0.smithi045.stdout:1144: left oid 2 (ObjNum 356 snap 106 seq_num 356) 2021-07-31T01:49:28.590 INFO:tasks.rados.rados.0.smithi045.stdout:1145: expect (ObjNum 323 snap 95 seq_num 323) 2021-07-31T01:49:28.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[34553]: cluster 2021-07-31T01:49:27.552503+0000 mon.a 2021-07-31T01:49:28.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[34553]: (mon.0) 908 : cluster [DBG] osdmap e268: 8 total, 8 up, 8 in 2021-07-31T01:49:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[34553]: cluster 2021-07-31T01:49:27.821793+0000 mgr.x ( 2021-07-31T01:49:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[34553]: mgr.24542) 91 : cluster [DBG] pgmap v182: 25 pgs: 25 active+clean; 128 MiB data, 744 MiB used, 714 GiB / 715 GiB avail; 380 KiB/s rd, 776 KiB/s wr, 3 op/s 2021-07-31T01:49:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[34553]: audit 2021-07-31T01:49:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[34553]: 2021-07-31T01:49 2021-07-31T01:49:28.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[34553]: :27.949687+0000 mon.b (mon.2) 72 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:49:28.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[34553]: audit 2021-07-31 2021-07-31T01:49:28.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[34553]: T01:49:27.950332+0000 mon.a (mon.0) 909 2021-07-31T01:49:28.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[34553]: : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:49:28.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[34553]: audit 2021-07-31T01:49:27.959760 2021-07-31T01:49:28.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[34553]: +0000 mon.b (mon.2) 73 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T01:49:28.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[34553]: audit 2021-07-31T01:49:28.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[34553]: 2021-07-31T01:49:27 2021-07-31T01:49:28.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[34553]: .960281+0000 mon.a (mon.0) 2021-07-31T01:49:28.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[34553]: 910 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T01:49:28.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[34553]: audit 2021-07-31T01:49:28.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[34553]: 2021-07-31T01:49 2021-07-31T01:49:28.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[34553]: :28.444889+0000 mon.b (mon.2) 2021-07-31T01:49:28.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[34553]: 74 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:49:28.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[34553]: audit 2021-07-31T 2021-07-31T01:49:28.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[34553]: 01:49:28.446275+0000 mon.b (mon.2) 2021-07-31T01:49:28.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[34553]: 75 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:49:28.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[34553]: audit 2021-07- 2021-07-31T01:49:28.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[34553]: 31T01:49:28.447256+0000 mon.b (mon.2) 76 : audit [INF] 2021-07-31T01:49:28.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[34553]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:49:28.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: debug 2021-07-31T01:49:28.491+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T01:49:28.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:49:28.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:49:28.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: yield 2021-07-31T01:49:28.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:49:28.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:49:28.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:49:28.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T01:49:28.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:49:28.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T01:49:28.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:49:28.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: await self._waiter 2021-07-31T01:49:28.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:49:28.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: task.result() 2021-07-31T01:49:28.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T01:49:28.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: debug 2021-07-31T01:49:28.492+0000 7fcb9215e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:49:28.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:49:28.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:49:28.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: yield 2021-07-31T01:49:28.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:49:28.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:49:28.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:49:28.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T01:49:28.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:49:28.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T01:49:28.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:49:28.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: await self._waiter 2021-07-31T01:49:28.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:49:28.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: task.result() 2021-07-31T01:49:28.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T01:49:28.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: 2021-07-31T01:49:28.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T01:49:28.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: 2021-07-31T01:49:28.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:49:28.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:49:28.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: return f(*arg) 2021-07-31T01:49:28.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:49:28.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:49:28.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:49:28.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:49:28.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:49:28.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:49:28.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:49:28.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T01:49:28.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:49:28.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: raise self._exception 2021-07-31T01:49:28.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:49:28.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:49:28.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:49:28.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T01:49:28.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:49:28.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: raise 2021-07-31T01:49:28.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:49:28.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T01:49:28.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:49:28.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:49:28.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T01:49:28.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: debug 2021-07-31T01:49:28.494+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T01:49:28.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:49:28.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:49:28.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: yield 2021-07-31T01:49:28.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:49:28.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:49:28.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:49:28.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T01:49:28.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:49:28.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T01:49:28.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:49:28.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: await self._waiter 2021-07-31T01:49:28.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:49:28.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: task.result() 2021-07-31T01:49:28.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T01:49:28.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: debug 2021-07-31T01:49:28.494+0000 7fcb9195d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:49:28.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:49:28.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:49:28.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: yield 2021-07-31T01:49:28.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:49:28.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:49:28.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:49:28.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T01:49:28.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:49:28.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T01:49:28.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:49:28.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: await self._waiter 2021-07-31T01:49:28.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:49:28.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: task.result() 2021-07-31T01:49:28.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T01:49:28.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: 2021-07-31T01:49:28.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T01:49:28.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: 2021-07-31T01:49:28.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:49:28.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:49:28.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: return f(*arg) 2021-07-31T01:49:28.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:49:28.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:49:28.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:49:28.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:49:28.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:49:28.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:49:28.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:49:28.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T01:49:28.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:49:28.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: raise self._exception 2021-07-31T01:49:28.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:49:28.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:49:28.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:49:28.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T01:49:28.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:49:28.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: raise 2021-07-31T01:49:28.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:49:28.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T01:49:28.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:49:28.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:49:28.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:49:28 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T01:49:28.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:28 smithi045 conmon[39718]: cluster 2021-07-31T01:49: 2021-07-31T01:49:28.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:28 smithi045 conmon[39718]: 27.552503+0000 mon.a (mon.0) 908 : cluster [DBG] osdmap e268: 8 total, 8 up, 8 in 2021-07-31T01:49:28.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:28 smithi045 conmon[39718]: cluster 2021-07-31T 2021-07-31T01:49:28.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:28 smithi045 conmon[39718]: 01:49:27.821793+0000 2021-07-31T01:49:28.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:28 smithi045 conmon[39718]: mgr.x (mgr.24542) 91 : cluster 2021-07-31T01:49:28.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:28 smithi045 conmon[39718]: [DBG] pgmap v182: 25 pgs: 25 active+clean; 128 MiB data, 744 MiB used, 714 GiB / 715 GiB avail; 380 KiB/s rd, 776 KiB/s wr, 3 op/s 2021-07-31T01:49:28.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:28 smithi045 conmon[39718]: audit 2021-07-31T01:49:28.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:28 smithi045 conmon[39718]: 2021-07-31T01: 2021-07-31T01:49:28.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:28 smithi045 conmon[39718]: 49:27.949687+0000 mon.b ( 2021-07-31T01:49:28.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:28 smithi045 conmon[39718]: mon.2) 72 : audit 2021-07-31T01:49:28.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:28 smithi045 conmon[39718]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:49:28.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:28 smithi045 conmon[39718]: audit 2021- 2021-07-31T01:49:28.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:28 smithi045 conmon[39718]: 07-31T01:49:27.950332 2021-07-31T01:49:28.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:28 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2021-07-31T01:49:28.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:28 smithi045 conmon[39718]: 909 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:49:28.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:28 smithi045 conmon[39718]: audit 2021- 2021-07-31T01:49:28.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:28 smithi045 conmon[39718]: 07-31T01:49:27. 2021-07-31T01:49:28.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:28 smithi045 conmon[39718]: 959760+0000 mon.b (mon.2) 2021-07-31T01:49:28.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:28 smithi045 conmon[39718]: 73 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T01:49:28.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:28 smithi045 conmon[39718]: audit 2021 2021-07-31T01:49:28.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:28 smithi045 conmon[39718]: -07-31T01:49: 2021-07-31T01:49:28.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:28 smithi045 conmon[39718]: 27.960281+0000 mon.a (mon 2021-07-31T01:49:28.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:28 smithi045 conmon[39718]: .0) 910 : audit [INF] 2021-07-31T01:49:28.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:28 smithi045 conmon[39718]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T01:49:28.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:28 smithi045 conmon[39718]: audit 2021-07-31T01:49:28.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:28 smithi045 conmon[39718]: 2021-07-31T01: 2021-07-31T01:49:28.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:28 smithi045 conmon[39718]: 49:28.444889+0000 mon.b 2021-07-31T01:49:28.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:28 smithi045 conmon[39718]: (mon.2) 74 : 2021-07-31T01:49:28.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:28 smithi045 conmon[39718]: audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:49:28.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:28 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T01:49:28.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:28 smithi045 conmon[39718]: T01:49:28.446275+0000 mon.b 2021-07-31T01:49:28.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:28 smithi045 conmon[39718]: (mon.2) 75 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:49:28.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:28 smithi045 conmon[39718]: 2021-07-31T01:49:28.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:28 smithi045 conmon[39718]: audit 2021-07 2021-07-31T01:49:28.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:28 smithi045 conmon[39718]: -31T01:49:28.447256+0000 mon.b (mon.2) 2021-07-31T01:49:28.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:28 smithi045 conmon[39718]: 76 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:49:28.795 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:28 smithi045 conmon[33655]: cluster 2021-07-31T01:49:27.552503+0000 mon.a (mon. 2021-07-31T01:49:28.796 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:28 smithi045 conmon[33655]: 0) 908 : cluster [DBG] osdmap e268: 8 total, 8 up, 8 in 2021-07-31T01:49:28.796 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:28 smithi045 conmon[33655]: cluster 2021-07-31T01:49:27.821793+0000 mgr.x (mgr.24542) 91 : cluster [DBG] pgmap v182: 25 pgs: 25 active+clean; 128 MiB data, 744 MiB used, 714 GiB / 715 GiB avail; 380 KiB/s rd, 776 KiB/s wr, 3 op/s 2021-07-31T01:49:28.796 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:28 smithi045 conmon[33655]: audit 2021-07-31T01:49:27.949687+0000 mon.b (mon.2) 72 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:49:28.796 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:28 smithi045 conmon[33655]: audit 2021-07-31T01:49:27. 2021-07-31T01:49:28.797 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:28 smithi045 conmon[33655]: 950332+0000 mon.a (mon.0) 909 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:49:28.797 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:28 smithi045 conmon[33655]: audit 2021-07-31T01:49:27.959760+0000 mon.b ( 2021-07-31T01:49:28.797 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:28 smithi045 conmon[33655]: mon.2) 73 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T01:49:28.797 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:28 smithi045 conmon[33655]: audit 2021- 2021-07-31T01:49:28.798 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:28 smithi045 conmon[33655]: 07-31T01:49:27.960281+0000 mon.a 2021-07-31T01:49:28.798 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:28 smithi045 conmon[33655]: (mon.0) 910 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T01:49:28.798 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:28 smithi045 conmon[33655]: audit 2021-07-31T01:49 2021-07-31T01:49:28.798 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:28 smithi045 conmon[33655]: :28.444889+0000 mon.b (mon.2) 74 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:49:28.799 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:28 smithi045 conmon[33655]: audit 2021-07-31T01:49:28.446275 2021-07-31T01:49:28.799 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:28 smithi045 conmon[33655]: +0000 mon.b (mon.2) 75 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:49:28.799 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:28 smithi045 conmon[33655]: audit 2021-07-31T01:49:28.447256+0000 mon.b (mon.2) 76 : 2021-07-31T01:49:28.799 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:28 smithi045 conmon[33655]: audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:49:28.845 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T01:49:29.124 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T01:49:29.125 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 4m ago 5m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T01:49:29.125 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 2m ago 4m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T01:49:29.125 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 2m ago 8m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T01:49:29.126 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 4m ago 10m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T01:49:29.126 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 4m ago 10m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T01:49:29.126 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 2m ago 8m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T01:49:29.126 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 4m ago 8m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T01:49:29.127 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 4m ago 5m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T01:49:29.127 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 2m ago 5m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T01:49:29.127 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 4m ago 8m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T01:49:29.127 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 4m ago 7m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T01:49:29.127 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 4m ago 7m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T01:49:29.128 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 4m ago 7m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T01:49:29.128 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 2m ago 6m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T01:49:29.128 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 2m ago 6m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T01:49:29.128 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 2m ago 6m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T01:49:29.128 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 2m ago 5m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T01:49:29.129 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 2m ago 5m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T01:49:29.219 INFO:tasks.rados.rados.0.smithi045.stdout:1143: done (4 left) 2021-07-31T01:49:29.219 INFO:tasks.rados.rados.0.smithi045.stdout:1144: done (3 left) 2021-07-31T01:49:29.219 INFO:tasks.rados.rados.0.smithi045.stdout:1145: done (2 left) 2021-07-31T01:49:29.219 INFO:tasks.rados.rados.0.smithi045.stdout:1146: done (1 left) 2021-07-31T01:49:29.220 INFO:tasks.rados.rados.0.smithi045.stdout:1147: done (0 left) 2021-07-31T01:49:29.220 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 2 current snap is 106 2021-07-31T01:49:29.220 INFO:tasks.rados.rados.0.smithi045.stdout:1148: seq_num 357 ranges {1171456=786432,1957888=401408} 2021-07-31T01:49:29.230 INFO:tasks.rados.rados.0.smithi045.stdout:1148: writing smithi04569269-2 from 1171456 to 1957888 tid 1 2021-07-31T01:49:29.233 INFO:tasks.rados.rados.0.smithi045.stdout:1148: writing smithi04569269-2 from 1957888 to 2359296 tid 2 2021-07-31T01:49:29.233 INFO:tasks.rados.rados.0.smithi045.stdout:1149: snap_create 2021-07-31T01:49:29.235 INFO:tasks.rados.rados.0.smithi045.stdout:1148: finishing write tid 1 to smithi04569269-2 2021-07-31T01:49:29.235 INFO:tasks.rados.rados.0.smithi045.stdout:1148: finishing write tid 2 to smithi04569269-2 2021-07-31T01:49:29.235 INFO:tasks.rados.rados.0.smithi045.stdout:1148: finishing write tid 3 to smithi04569269-2 2021-07-31T01:49:29.236 INFO:tasks.rados.rados.0.smithi045.stdout:1148: finishing write tid 4 to smithi04569269-2 2021-07-31T01:49:29.236 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 2 v 271 (ObjNum 357 snap 106 seq_num 357) dirty exists 2021-07-31T01:49:29.236 INFO:tasks.rados.rados.0.smithi045.stdout:1148: left oid 2 (ObjNum 357 snap 106 seq_num 357) 2021-07-31T01:49:29.464 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T01:49:29.464 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T01:49:29.464 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T01:49:29.465 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T01:49:29.465 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T01:49:29.465 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T01:49:29.465 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T01:49:29.465 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T01:49:29.466 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T01:49:29.466 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T01:49:29.466 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T01:49:29.466 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T01:49:29.467 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T01:49:29.467 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T01:49:29.467 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T01:49:29.468 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T01:49:29.468 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T01:49:29.563 INFO:tasks.rados.rados.0.smithi045.stdout:1148: done (1 left) 2021-07-31T01:49:29.563 INFO:tasks.rados.rados.0.smithi045.stdout:1149: done (0 left) 2021-07-31T01:49:29.564 INFO:tasks.rados.rados.0.smithi045.stdout:1150: snap_create 2021-07-31T01:49:30.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: cephadm 2021-07-31T01:49:28.447976+0000 mgr.x (mgr.24542) 92 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T01:49:30.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: cephadm 2021-07-31T01:49:28.448747+0000 mgr.x (mgr.24542) 93 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T01:49:30.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: cephadm 2021-07-31T01:49:28.449166+0000 mgr.x (mgr 2021-07-31T01:49:30.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: .24542) 94 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T01:49:30.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: cephadm 2021-07-31 2021-07-31T01:49:30.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: T01:49:28.492054+0000 mgr.x 2021-07-31T01:49:30.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: (mgr.24542) 95 : cephadm [ERR] 2021-07-31T01:49:30.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:49:30.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:49:30.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: yield 2021-07-31T01:49:30.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:49:30.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:49:30.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:49:30.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T01:49:30.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:49:30.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T01:49:30.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:49:30.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: await self._waiter 2021-07-31T01:49:30.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:49:30.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: task.result() 2021-07-31T01:49:30.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T01:49:30.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: 2021-07-31T01:49:30.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: cephadm 2021-07-31T01:49:28.492971+0000 mgr.x (mgr.24542 2021-07-31T01:49:30.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: ) 96 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:49:30.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:49:30.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:49:30.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: yield 2021-07-31T01:49:30.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:49:30.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:49:30.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:49:30.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T01:49:30.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:49:30.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T01:49:30.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:49:30.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: await self._waiter 2021-07-31T01:49:30.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:49:30.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: task.result() 2021-07-31T01:49:30.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T01:49:30.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: 2021-07-31T01:49:30.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T01:49:30.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: 2021-07-31T01:49:30.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:49:30.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:49:30.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: return f(*arg) 2021-07-31T01:49:30.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:49:30.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:49:30.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:49:30.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:49:30.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:49:30.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:49:30.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:49:30.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T01:49:30.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:49:30.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: raise self._exception 2021-07-31T01:49:30.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:49:30.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:49:30.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:49:30.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T01:49:30.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:49:30.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: raise 2021-07-31T01:49:30.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:49:30.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T01:49:30.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:49:30.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:49:30.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T01:49:30.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: cephadm 2021-07-31T01:49:28.494973+0000 mgr.x (mgr.24542) 97 : cephadm [ERR] 2021-07-31T01:49:30.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:49:30.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:49:30.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: yield 2021-07-31T01:49:30.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:49:30.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:49:30.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:49:30.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T01:49:30.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:49:30.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T01:49:30.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:49:30.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: await self._waiter 2021-07-31T01:49:30.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:49:30.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: task.result() 2021-07-31T01:49:30.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T01:49:30.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: cephadm 2021-07-31T01:49:28.495403+0000 mgr.x (mgr.24542) 98 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:49:30.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:49:30.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:49:30.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: yield 2021-07-31T01:49:30.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:49:30.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:49:30.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:49:30.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T01:49:30.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:49:30.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T01:49:30.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:49:30.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: await self._waiter 2021-07-31T01:49:30.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:49:30.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: task.result() 2021-07-31T01:49:30.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T01:49:30.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: 2021-07-31T01:49:30.035 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T01:49:30.035 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: 2021-07-31T01:49:30.035 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:49:30.035 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:49:30.036 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: return f(*arg) 2021-07-31T01:49:30.036 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:49:30.036 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:49:30.036 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:49:30.036 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:49:30.037 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:49:30.037 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:49:30.037 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:49:30.037 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T01:49:30.038 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:49:30.038 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: raise self._exception 2021-07-31T01:49:30.038 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:49:30.038 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:49:30.039 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:49:30.039 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T01:49:30.039 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:49:30.039 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: raise 2021-07-31T01:49:30.039 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:49:30.040 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T01:49:30.040 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:49:30.040 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:49:30.040 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T01:49:30.041 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: cluster 2021-07-31T01:49:28.558484+0000 mon.a (mon.0) 911 : cluster [DBG] osdmap e269: 8 total, 8 up, 8 in 2021-07-31T01:49:30.041 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: audit 2021-07-31T01:49 2021-07-31T01:49:30.041 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: :28.833200+0000 mgr.x (mgr.24542) 99 : audit [DBG] from='client.14784 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:49:30.041 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:29 smithi154 conmon[34553]: audit 2021-07-31T01:49:29.463710+0000 mon.c (mon.1) 29 : audit [DBG] from='client.? 172.21.15.45:0/2360645172' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T01:49:30.042 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: cephadm 2021-07-31T01:49:28.447976+0000 2021-07-31T01:49:30.042 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: mgr.x (mgr.24542) 92 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T01:49:30.043 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: cephadm 2021-07-31T01:49:28.448747+0000 mgr.x (mgr.24542) 93 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T01:49:30.043 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: 2021-07-31T01:49:30.043 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: cephadm 2021-07-31T01:49:28.449166+0000 mgr.x (mgr.24542) 2021-07-31T01:49:30.043 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: 94 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T01:49:30.043 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: cephadm 2021-07-31T01:49:30.044 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: 2021-07-31T01:49:28.492054+0000 mgr.x (mgr.24542) 95 : cephadm 2021-07-31T01:49:30.044 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: [ERR] 2021-07-31T01:49:30.044 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:49:30.044 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:49:30.045 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: yield 2021-07-31T01:49:30.045 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:49:30.045 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:49:30.045 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:49:30.046 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T01:49:30.046 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:49:30.046 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T01:49:30.046 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:49:30.046 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: await self._waiter 2021-07-31T01:49:30.047 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:49:30.047 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: task.result() 2021-07-31T01:49:30.047 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T01:49:30.047 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: cephadm 2021-07-31T01:49:28.492971+0000 mgr.x (mgr.24542) 96 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:49:30.048 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:49:30.048 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:49:30.048 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: yield 2021-07-31T01:49:30.048 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:49:30.049 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:49:30.049 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:49:30.049 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T01:49:30.049 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:49:30.049 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T01:49:30.050 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:49:30.050 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: await self._waiter 2021-07-31T01:49:30.050 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:49:30.050 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: task.result() 2021-07-31T01:49:30.051 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T01:49:30.051 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: 2021-07-31T01:49:30.051 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T01:49:30.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: cephadm 2021-07-31T01:49:28.447976 2021-07-31T01:49:30.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 92 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T01:49:30.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: cephadm 2021- 2021-07-31T01:49:30.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: 07-31T01:49:28.448747+0000 mgr.x (mgr.24542) 93 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T01:49:30.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: 2021-07-31T01:49:30.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: cephadm 2021 2021-07-31T01:49:30.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: -07-31T01:49:28.449166+0000 mgr.x (mgr.24542 2021-07-31T01:49:30.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: ) 94 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T01:49:30.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: cephadm 2021-07-31T01:49:30.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: 2021-07-31T01 2021-07-31T01:49:30.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: :49:28.492054+0000 mgr.x 2021-07-31T01:49:30.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: (mgr.24542) 95 : cephadm [ERR] 2021-07-31T01:49:30.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: 2021-07-31T01:49:30.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:49:30.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:49:30.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: yield 2021-07-31T01:49:30.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:49:30.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:49:30.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:49:30.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T01:49:30.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:49:30.057 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T01:49:30.057 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:49:30.057 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: await self._waiter 2021-07-31T01:49:30.057 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:49:30.057 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: task.result() 2021-07-31T01:49:30.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T01:49:30.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: cephadm 2021-07-31T01:49:28.492971+0000 mgr.x (mgr.24542) 96 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:49:30.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:49:30.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:49:30.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: yield 2021-07-31T01:49:30.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:49:30.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:49:30.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:49:30.060 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T01:49:30.060 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:49:30.060 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T01:49:30.060 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:49:30.060 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: await self._waiter 2021-07-31T01:49:30.061 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:49:30.061 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: task.result() 2021-07-31T01:49:30.061 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T01:49:30.062 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: 2021-07-31T01:49:30.062 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:49:30.062 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:49:30.062 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: return f(*arg) 2021-07-31T01:49:30.063 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:49:30.063 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:49:30.063 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:49:30.063 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:49:30.064 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:49:30.064 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:49:30.064 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:49:30.064 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T01:49:30.065 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:49:30.065 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: raise self._exception 2021-07-31T01:49:30.065 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:49:30.065 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:49:30.066 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:49:30.066 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T01:49:30.066 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:49:30.066 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: raise 2021-07-31T01:49:30.067 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:49:30.067 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T01:49:30.067 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:49:30.067 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:49:30.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T01:49:30.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: cephadm 2021-07-31T01:49:28.494973+0000 mgr.x (mgr.24542) 97 : cephadm [ERR] 2021-07-31T01:49:30.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: 2021-07-31T01:49:30.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:49:30.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:49:30.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: yield 2021-07-31T01:49:30.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:49:30.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:49:30.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:49:30.070 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T01:49:30.070 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:49:30.070 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T01:49:30.070 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:49:30.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: await self._waiter 2021-07-31T01:49:30.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:49:30.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: task.result() 2021-07-31T01:49:30.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T01:49:30.072 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: 2021-07-31T01:49:30.072 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T01:49:30.072 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: 2021-07-31T01:49:30.072 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:49:30.073 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:49:30.073 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: return f(*arg) 2021-07-31T01:49:30.073 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:49:30.073 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:49:30.074 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:49:30.074 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:49:30.074 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:49:30.074 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:49:30.075 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:49:30.075 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T01:49:30.075 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:49:30.075 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: raise self._exception 2021-07-31T01:49:30.075 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:49:30.076 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:49:30.076 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:49:30.076 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T01:49:30.076 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:49:30.077 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: raise 2021-07-31T01:49:30.077 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:49:30.077 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T01:49:30.077 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:49:30.077 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:49:30.078 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T01:49:30.078 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: cephadm 2021-07-31T01:49:28.494973+0000 mgr.x (mgr.24542) 97 : cephadm [ERR] 2021-07-31T01:49:30.078 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:49:30.078 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:49:30.079 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: yield 2021-07-31T01:49:30.079 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:49:30.079 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:49:30.079 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:49:30.080 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T01:49:30.080 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:49:30.080 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T01:49:30.080 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:49:30.080 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: await self._waiter 2021-07-31T01:49:30.081 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:49:30.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: cephadm 2021-07-31T01:49:28.495403+0000 mgr.x (mgr.24542) 98 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:49:30.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:49:30.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:49:30.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: yield 2021-07-31T01:49:30.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:49:30.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:49:30.084 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:49:30.084 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T01:49:30.084 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:49:30.084 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T01:49:30.085 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:49:30.085 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: await self._waiter 2021-07-31T01:49:30.085 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:49:30.085 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: task.result() 2021-07-31T01:49:30.085 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T01:49:30.086 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: 2021-07-31T01:49:30.086 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T01:49:30.086 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: 2021-07-31T01:49:30.086 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:49:30.087 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:49:30.087 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: return f(*arg) 2021-07-31T01:49:30.087 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:49:30.087 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:49:30.087 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:49:30.088 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:49:30.088 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:49:30.088 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:49:30.088 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:49:30.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T01:49:30.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:49:30.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: raise self._exception 2021-07-31T01:49:30.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:49:30.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:49:30.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:49:30.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T01:49:30.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:49:30.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: raise 2021-07-31T01:49:30.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:49:30.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T01:49:30.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:49:30.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:49:30.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T01:49:30.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: cluster 2021-07-31T01:49:28.558484+0000 mon.a (mon.0) 911 : cluster [DBG] osdmap e269: 8 total, 8 up, 8 in 2021-07-31T01:49:30.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: audit 2021-07-31T01:49:28.833200+0000 mgr.x (mgr.24542) 99 : audit [DBG] from='client.14784 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:49:30.093 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[33655]: audit 2021-07-31T01:49:29.463710+0000 mon.c (mon.1) 29 : audit [DBG] from='client.? 172.21.15.45:0/2360645172' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T01:49:30.093 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: task.result() 2021-07-31T01:49:30.094 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T01:49:30.094 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: cephadm 2021-07-31T01:49:28.495403+0000 mgr.x (mgr.24542) 98 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:49:30.094 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:49:30.094 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:49:30.094 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: yield 2021-07-31T01:49:30.095 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:49:30.095 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:49:30.095 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:49:30.095 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T01:49:30.096 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:49:30.096 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T01:49:30.096 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:49:30.096 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: await self._waiter 2021-07-31T01:49:30.096 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:49:30.097 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: task.result() 2021-07-31T01:49:30.097 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T01:49:30.097 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: 2021-07-31T01:49:30.097 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T01:49:30.098 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: 2021-07-31T01:49:30.098 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:49:30.098 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:49:30.098 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: return f(*arg) 2021-07-31T01:49:30.098 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:49:30.099 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:49:30.099 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:49:30.099 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:49:30.099 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:49:30.100 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:49:30.100 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:49:30.100 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T01:49:30.100 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:49:30.100 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: raise self._exception 2021-07-31T01:49:30.101 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:49:30.101 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:49:30.101 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:49:30.102 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T01:49:30.102 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:49:30.102 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: raise 2021-07-31T01:49:30.103 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:49:30.103 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T01:49:30.103 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:49:30.103 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:49:30.104 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T01:49:30.104 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: cluster 2021-07-31T01:49:28.558484+0000 mon.a (mon.0) 911 : cluster [DBG] osdmap e269: 8 total, 8 up, 8 in 2021-07-31T01:49:30.104 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: audit 2021-07-31T01:49:28.833200+0000 mgr.x (mgr.24542) 99 : audit [DBG] from='client.14784 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:49:30.105 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:29 smithi045 conmon[39718]: audit 2021-07-31T01:49:29.463710+0000 mon.c (mon.1) 29 : audit [DBG] from='client.? 172.21.15.45:0/2360645172' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T01:49:30.570 INFO:tasks.rados.rados.0.smithi045.stdout:1150: done (0 left) 2021-07-31T01:49:30.570 INFO:tasks.rados.rados.0.smithi045.stdout:1151: read oid 17 snap 106 2021-07-31T01:49:30.570 INFO:tasks.rados.rados.0.smithi045.stdout:1151: expect (ObjNum 196 snap 52 seq_num 196) 2021-07-31T01:49:30.571 INFO:tasks.rados.rados.0.smithi045.stdout:1152: delete oid 15 current snap is 108 2021-07-31T01:49:30.575 INFO:tasks.rados.rados.0.smithi045.stdout:1152: done (1 left) 2021-07-31T01:49:30.575 INFO:tasks.rados.rados.0.smithi045.stdout:1153: read oid 2 snap 105 2021-07-31T01:49:30.576 INFO:tasks.rados.rados.0.smithi045.stdout:1153: expect deleted 2021-07-31T01:49:30.576 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 8 current snap is 108 2021-07-31T01:49:30.576 INFO:tasks.rados.rados.0.smithi045.stdout:1154: seq_num 359 ranges {0=417792,417792=417792,835584=385024} 2021-07-31T01:49:30.578 INFO:tasks.rados.rados.0.smithi045.stdout:1154: writing smithi04569269-8 from 0 to 417792 tid 1 2021-07-31T01:49:30.580 INFO:tasks.rados.rados.0.smithi045.stdout:1154: writing smithi04569269-8 from 417792 to 835584 tid 2 2021-07-31T01:49:30.583 INFO:tasks.rados.rados.0.smithi045.stdout:1154: writing smithi04569269-8 from 835584 to 1220608 tid 3 2021-07-31T01:49:30.583 INFO:tasks.rados.rados.0.smithi045.stdout:1155: rollback oid 35 current snap is 108 2021-07-31T01:49:30.583 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 35 to 105 2021-07-31T01:49:30.584 INFO:tasks.rados.rados.0.smithi045.stdout:1153: done (3 left) 2021-07-31T01:49:30.584 INFO:tasks.rados.rados.0.smithi045.stdout:1156: rmattr oid 45 current snap is 108 2021-07-31T01:49:30.584 INFO:tasks.rados.rados.0.smithi045.stdout:1156: done (3 left) 2021-07-31T01:49:30.584 INFO:tasks.rados.rados.0.smithi045.stdout:1157: setattr oid 10 current snap is 108 2021-07-31T01:49:30.585 INFO:tasks.rados.rados.0.smithi045.stdout:1158: snap_remove snap 104 2021-07-31T01:49:31.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:30 smithi154 conmon[34553]: audit 2021-07-31T01:49:29.116351+0000 mgr.x (mgr.24542) 100 : audit [DBG] 2021-07-31T01:49:31.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:30 smithi154 conmon[34553]: from='client.14790 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:49:31.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:30 smithi154 conmon[34553]: cluster 2021-07-31T01:49:29.563078+0000 mon.a (mon.0) 912 : cluster [DBG] osdmap e270: 8 total, 8 up, 8 in 2021-07-31T01:49:31.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:30 smithi154 conmon[34553]: cluster 2021-07-31T01:49:29.822185+0000 mgr.x (mgr.24542) 101 : cluster [DBG] pgmap v185: 25 pgs: 25 active+clean; 128 MiB data, 771 MiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 2.1 MiB/s wr, 15 op/s 2021-07-31T01:49:31.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:30 smithi045 conmon[39718]: audit 2021-07-31T01:49:29.116351+0000 mgr.x (mgr.24542) 100 : audit 2021-07-31T01:49:31.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:30 smithi045 conmon[39718]: [DBG] from='client.14790 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:49:31.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:30 smithi045 conmon[39718]: cluster 2021-07-31T01:49:29.563078+0000 mon.a (mon.0) 912 : cluster [DBG] osdmap e270: 8 total, 8 up, 8 in 2021-07-31T01:49:31.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:30 smithi045 conmon[39718]: cluster 2021-07-31T01:49:29.822185+0000 mgr.x (mgr.24542) 101 : cluster [DBG] pgmap v185: 25 pgs: 25 active+clean; 128 MiB data, 771 MiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 2.1 MiB/s wr, 15 op/s 2021-07-31T01:49:31.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:30 smithi045 conmon[33655]: audit 2021-07-31T01:49:29.116351+0000 mgr.x (mgr.24542) 100 : audit [DBG] from='client.14790 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:49:31.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:30 smithi045 conmon[33655]: 2021-07-31T01:49:31.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:30 smithi045 conmon[33655]: cluster 2021-07-31T01:49:29.563078+0000 mon.a (mon.0) 912 : cluster [DBG] osdmap e270: 8 total, 8 up, 8 in 2021-07-31T01:49:31.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:30 smithi045 conmon[33655]: cluster 2021-07-31T01:49:29.822185+0000 mgr.x (mgr.24542) 101 : cluster 2021-07-31T01:49:31.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:30 smithi045 conmon[33655]: [DBG] pgmap v185: 25 pgs: 25 active+clean; 128 MiB data, 771 MiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 2.1 MiB/s wr, 15 op/s 2021-07-31T01:49:31.523 INFO:tasks.rados.rados.0.smithi045.stdout:1158: done (4 left) 2021-07-31T01:49:31.524 INFO:tasks.rados.rados.0.smithi045.stdout:1159: rmattr oid 11 current snap is 108 2021-07-31T01:49:31.524 INFO:tasks.rados.rados.0.smithi045.stdout:1159: done (4 left) 2021-07-31T01:49:31.524 INFO:tasks.rados.rados.0.smithi045.stdout:1160: snap_remove snap 107 2021-07-31T01:49:32.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:31 smithi154 conmon[34553]: cluster 2021-07-31T01:49:30.569105+0000 mon.a (mon.0) 913 : cluster [DBG] osdmap e271: 8 total, 8 up, 8 in 2021-07-31T01:49:32.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:31 smithi154 conmon[34553]: cluster 2021-07-31T01:49:31.523066+0000 mon.a (mon.0) 914 : cluster [DBG] osdmap e272: 8 total, 8 up, 8 in 2021-07-31T01:49:32.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:31 smithi045 conmon[33655]: cluster 2021-07-31T01:49 2021-07-31T01:49:32.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:31 smithi045 conmon[33655]: :30.569105+0000 mon.a (mon.0) 913 : cluster [DBG] osdmap e271: 8 total, 8 up, 8 in 2021-07-31T01:49:32.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:31 smithi045 conmon[33655]: cluster 2021-07-31T01:49:31.523066+0000 mon.a (mon.0) 914 : cluster [DBG] osdmap e272: 8 total, 8 up, 8 in 2021-07-31T01:49:32.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:31 smithi045 conmon[39718]: cluster 2021-07-31T01:49:30.569105+0000 mon.a (mon. 2021-07-31T01:49:32.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:31 smithi045 conmon[39718]: 0) 913 : cluster [DBG] osdmap e271: 8 total, 8 up, 8 in 2021-07-31T01:49:32.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:31 smithi045 conmon[39718]: cluster 2021-07-31T01:49:31.523066+0000 2021-07-31T01:49:32.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:31 smithi045 conmon[39718]: mon.a (mon.0) 914 : cluster [DBG] osdmap e272: 8 total, 8 up, 8 in 2021-07-31T01:49:32.530 INFO:tasks.rados.rados.0.smithi045.stdout:1160: done (4 left) 2021-07-31T01:49:32.530 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 7 current snap is 108 2021-07-31T01:49:32.531 INFO:tasks.rados.rados.0.smithi045.stdout:1154: finishing write tid 1 to smithi04569269-8 2021-07-31T01:49:32.531 INFO:tasks.rados.rados.0.smithi045.stdout:1161: seq_num 360 ranges {0=434176,434176=704512,1138688=98304} 2021-07-31T01:49:32.532 INFO:tasks.rados.rados.0.smithi045.stdout:1161: writing smithi04569269-7 from 0 to 434176 tid 1 2021-07-31T01:49:32.536 INFO:tasks.rados.rados.0.smithi045.stdout:1161: writing smithi04569269-7 from 434176 to 1138688 tid 2 2021-07-31T01:49:32.537 INFO:tasks.rados.rados.0.smithi045.stdout:1161: writing smithi04569269-7 from 1138688 to 1236992 tid 3 2021-07-31T01:49:32.537 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 36 current snap is 108 2021-07-31T01:49:32.537 INFO:tasks.rados.rados.0.smithi045.stdout:1154: finishing write tid 2 to smithi04569269-8 2021-07-31T01:49:32.538 INFO:tasks.rados.rados.0.smithi045.stdout:1162: seq_num 361 ranges {0=450560,450560=581632,1032192=221184} 2021-07-31T01:49:32.541 INFO:tasks.rados.rados.0.smithi045.stdout:1162: writing smithi04569269-36 from 0 to 450560 tid 1 2021-07-31T01:49:32.544 INFO:tasks.rados.rados.0.smithi045.stdout:1162: writing smithi04569269-36 from 450560 to 1032192 tid 2 2021-07-31T01:49:32.545 INFO:tasks.rados.rados.0.smithi045.stdout:1162: writing smithi04569269-36 from 1032192 to 1253376 tid 3 2021-07-31T01:49:32.546 INFO:tasks.rados.rados.0.smithi045.stdout:1163: rollback oid 32 current snap is 108 2021-07-31T01:49:32.546 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 32 to 106 2021-07-31T01:49:32.546 INFO:tasks.rados.rados.0.smithi045.stdout:1154: finishing write tid 3 to smithi04569269-8 2021-07-31T01:49:32.546 INFO:tasks.rados.rados.0.smithi045.stdout:1154: finishing write tid 4 to smithi04569269-8 2021-07-31T01:49:32.546 INFO:tasks.rados.rados.0.smithi045.stdout:1151: expect (ObjNum 196 snap 52 seq_num 196) 2021-07-31T01:49:32.773 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 10 v 494 (ObjNum 336 snap 99 seq_num 336) dirty exists 2021-07-31T01:49:32.774 INFO:tasks.rados.rados.0.smithi045.stdout:1155: finishing rollback tid 0 to smithi04569269-35 2021-07-31T01:49:32.774 INFO:tasks.rados.rados.0.smithi045.stdout:1155: finishing rollback tid 1 to smithi04569269-35 2021-07-31T01:49:32.774 INFO:tasks.rados.rados.0.smithi045.stdout:1151: done (6 left) 2021-07-31T01:49:32.775 INFO:tasks.rados.rados.0.smithi045.stdout:1157: done (5 left) 2021-07-31T01:49:32.775 INFO:tasks.rados.rados.0.smithi045.stdout:1164: rollback oid 38 current snap is 108 2021-07-31T01:49:32.775 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 38 to 103 2021-07-31T01:49:32.775 INFO:tasks.rados.rados.0.smithi045.stdout:1155: finishing rollback tid 2 to smithi04569269-35 2021-07-31T01:49:32.775 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 35 v 783 (ObjNum 311 snap 93 seq_num 311) dirty exists 2021-07-31T01:49:32.776 INFO:tasks.rados.rados.0.smithi045.stdout:1154: finishing write tid 5 to smithi04569269-8 2021-07-31T01:49:32.776 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 8 v 779 (ObjNum 359 snap 108 seq_num 359) dirty exists 2021-07-31T01:49:32.776 INFO:tasks.rados.rados.0.smithi045.stdout:1154: left oid 8 (ObjNum 359 snap 108 seq_num 359) 2021-07-31T01:49:32.776 INFO:tasks.rados.rados.0.smithi045.stdout:1161: finishing write tid 1 to smithi04569269-7 2021-07-31T01:49:32.777 INFO:tasks.rados.rados.0.smithi045.stdout:1161: finishing write tid 2 to smithi04569269-7 2021-07-31T01:49:32.777 INFO:tasks.rados.rados.0.smithi045.stdout:1161: finishing write tid 3 to smithi04569269-7 2021-07-31T01:49:32.777 INFO:tasks.rados.rados.0.smithi045.stdout:1161: finishing write tid 4 to smithi04569269-7 2021-07-31T01:49:32.777 INFO:tasks.rados.rados.0.smithi045.stdout:1162: finishing write tid 1 to smithi04569269-36 2021-07-31T01:49:32.777 INFO:tasks.rados.rados.0.smithi045.stdout:1161: finishing write tid 5 to smithi04569269-7 2021-07-31T01:49:32.778 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 7 v 515 (ObjNum 360 snap 108 seq_num 360) dirty exists 2021-07-31T01:49:32.778 INFO:tasks.rados.rados.0.smithi045.stdout:1161: left oid 7 (ObjNum 360 snap 108 seq_num 360) 2021-07-31T01:49:32.778 INFO:tasks.rados.rados.0.smithi045.stdout:1162: finishing write tid 2 to smithi04569269-36 2021-07-31T01:49:32.778 INFO:tasks.rados.rados.0.smithi045.stdout:1163: finishing rollback tid 0 to smithi04569269-32 2021-07-31T01:49:32.778 INFO:tasks.rados.rados.0.smithi045.stdout:1162: finishing write tid 3 to smithi04569269-36 2021-07-31T01:49:32.779 INFO:tasks.rados.rados.0.smithi045.stdout:1163: finishing rollback tid 1 to smithi04569269-32 2021-07-31T01:49:32.779 INFO:tasks.rados.rados.0.smithi045.stdout:1162: finishing write tid 4 to smithi04569269-36 2021-07-31T01:49:32.779 INFO:tasks.rados.rados.0.smithi045.stdout:1163: finishing rollback tid 2 to smithi04569269-32 2021-07-31T01:49:32.779 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 32 v 217 (ObjNum 330 snap 96 seq_num 330) dirty exists 2021-07-31T01:49:32.780 INFO:tasks.rados.rados.0.smithi045.stdout:1162: finishing write tid 5 to smithi04569269-36 2021-07-31T01:49:32.780 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 36 v 791 (ObjNum 361 snap 108 seq_num 361) dirty exists 2021-07-31T01:49:32.780 INFO:tasks.rados.rados.0.smithi045.stdout:1162: left oid 36 (ObjNum 361 snap 108 seq_num 361) 2021-07-31T01:49:32.780 INFO:tasks.rados.rados.0.smithi045.stdout:1154: done (5 left) 2021-07-31T01:49:32.780 INFO:tasks.rados.rados.0.smithi045.stdout:1155: done (4 left) 2021-07-31T01:49:32.781 INFO:tasks.rados.rados.0.smithi045.stdout:1161: done (3 left) 2021-07-31T01:49:32.781 INFO:tasks.rados.rados.0.smithi045.stdout:1162: done (2 left) 2021-07-31T01:49:32.781 INFO:tasks.rados.rados.0.smithi045.stdout:1163: done (1 left) 2021-07-31T01:49:32.781 INFO:tasks.rados.rados.0.smithi045.stdout:1165: rmattr oid 31 current snap is 108 2021-07-31T01:49:32.781 INFO:tasks.rados.rados.0.smithi045.stdout:1166: rmattr oid 26 current snap is 108 2021-07-31T01:49:32.782 INFO:tasks.rados.rados.0.smithi045.stdout:1166: done (2 left) 2021-07-31T01:49:32.782 INFO:tasks.rados.rados.0.smithi045.stdout:1167: delete oid 12 current snap is 108 2021-07-31T01:49:32.782 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 31 v 0 (ObjNum 340 snap 100 seq_num 340) dirty exists 2021-07-31T01:49:32.783 INFO:tasks.rados.rados.0.smithi045.stdout:1165: done (2 left) 2021-07-31T01:49:32.783 INFO:tasks.rados.rados.0.smithi045.stdout:1167: done (1 left) 2021-07-31T01:49:32.783 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 46 current snap is 108 2021-07-31T01:49:32.783 INFO:tasks.rados.rados.0.smithi045.stdout:1168: seq_num 363 ranges {4308992=483328,4792320=753664,5545984=49152} 2021-07-31T01:49:32.802 INFO:tasks.rados.rados.0.smithi045.stdout:1168: writing smithi04569269-46 from 4308992 to 4792320 tid 1 2021-07-31T01:49:32.806 INFO:tasks.rados.rados.0.smithi045.stdout:1168: writing smithi04569269-46 from 4792320 to 5545984 tid 2 2021-07-31T01:49:32.807 INFO:tasks.rados.rados.0.smithi045.stdout:1168: writing smithi04569269-46 from 5545984 to 5595136 tid 3 2021-07-31T01:49:32.807 INFO:tasks.rados.rados.0.smithi045.stdout:1169: snap_remove snap 105 2021-07-31T01:49:33.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:32 smithi154 conmon[34553]: cluster 2021-07-31T01:49:31.822627+0000 mgr.x (mgr.24542) 102 : cluster [DBG] 2021-07-31T01:49:33.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:32 smithi154 conmon[34553]: pgmap v188: 25 pgs: 25 active+clean; 128 MiB data, 771 MiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 2.1 MiB/s wr, 15 op/s 2021-07-31T01:49:33.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:32 smithi154 conmon[34553]: cluster 2021-07-31T01:49:32.529530+0000 mon.a (mon.0) 915 : cluster [DBG] osdmap e273: 8 total, 8 up, 8 in 2021-07-31T01:49:33.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:32 smithi045 conmon[39718]: cluster 2021-07-31T01:49:31 2021-07-31T01:49:33.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:32 smithi045 conmon[39718]: .822627+0000 mgr.x (mgr.24542) 102 : cluster [DBG] pgmap v188: 25 pgs: 25 active+clean; 128 MiB data, 771 MiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 2.1 MiB/s wr, 15 op/s 2021-07-31T01:49:33.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:32 smithi045 conmon[39718]: cluster 2021-07 2021-07-31T01:49:33.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:32 smithi045 conmon[39718]: -31T01:49:32.529530+0000 mon.a (mon.0 2021-07-31T01:49:33.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:32 smithi045 conmon[39718]: ) 915 : cluster [DBG] osdmap e273: 8 total, 8 up, 8 in 2021-07-31T01:49:33.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:32 smithi045 conmon[39718]: 2021-07-31T01:49:33.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:32 smithi045 conmon[33655]: cluster 2021-07-31T01:49:31. 2021-07-31T01:49:33.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:32 smithi045 conmon[33655]: 822627+0000 mgr.x (mgr.24542) 102 : cluster [DBG] pgmap v188: 25 pgs: 25 active+clean; 128 MiB data, 771 MiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 2.1 MiB/s wr, 15 op/s 2021-07-31T01:49:33.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:32 smithi045 conmon[33655]: cluster 2021- 2021-07-31T01:49:33.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:32 smithi045 conmon[33655]: 07-31T01:49:32.529530+0000 mon.a (mon.0) 915 2021-07-31T01:49:33.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:32 smithi045 conmon[33655]: : cluster [DBG] osdmap e273: 8 total, 8 up, 8 in 2021-07-31T01:49:33.578 INFO:tasks.rados.rados.0.smithi045.stdout:1169: done (2 left) 2021-07-31T01:49:33.579 INFO:tasks.rados.rados.0.smithi045.stdout:1170: read oid 39 snap -1 2021-07-31T01:49:33.579 INFO:tasks.rados.rados.0.smithi045.stdout:1170: expect (ObjNum 342 snap 102 seq_num 342) 2021-07-31T01:49:33.579 INFO:tasks.rados.rados.0.smithi045.stdout:1171: copy_from oid 13 from oid 43 current snap is 108 2021-07-31T01:49:33.579 INFO:tasks.rados.rados.0.smithi045.stdout:1172: delete oid 32 current snap is 108 2021-07-31T01:49:33.580 INFO:tasks.rados.rados.0.smithi045.stdout:1164: finishing rollback tid 0 to smithi04569269-38 2021-07-31T01:49:33.580 INFO:tasks.rados.rados.0.smithi045.stdout:1164: finishing rollback tid 1 to smithi04569269-38 2021-07-31T01:49:33.580 INFO:tasks.rados.rados.0.smithi045.stdout:1164: finishing rollback tid 2 to smithi04569269-38 2021-07-31T01:49:33.580 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 38 v 542 (ObjNum 244 snap 67 seq_num 244) dirty exists 2021-07-31T01:49:33.581 INFO:tasks.rados.rados.0.smithi045.stdout:1168: finishing write tid 1 to smithi04569269-46 2021-07-31T01:49:33.581 INFO:tasks.rados.rados.0.smithi045.stdout:1168: finishing write tid 2 to smithi04569269-46 2021-07-31T01:49:33.581 INFO:tasks.rados.rados.0.smithi045.stdout:1168: finishing write tid 3 to smithi04569269-46 2021-07-31T01:49:33.581 INFO:tasks.rados.rados.0.smithi045.stdout:1168: finishing write tid 4 to smithi04569269-46 2021-07-31T01:49:33.581 INFO:tasks.rados.rados.0.smithi045.stdout:1168: finishing write tid 5 to smithi04569269-46 2021-07-31T01:49:33.582 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 46 v 261 (ObjNum 363 snap 108 seq_num 363) dirty exists 2021-07-31T01:49:33.582 INFO:tasks.rados.rados.0.smithi045.stdout:1168: left oid 46 (ObjNum 363 snap 108 seq_num 363) 2021-07-31T01:49:33.584 INFO:tasks.rados.rados.0.smithi045.stdout:1164: done (4 left) 2021-07-31T01:49:33.584 INFO:tasks.rados.rados.0.smithi045.stdout:1168: done (3 left) 2021-07-31T01:49:33.585 INFO:tasks.rados.rados.0.smithi045.stdout:1172: done (2 left) 2021-07-31T01:49:33.585 INFO:tasks.rados.rados.0.smithi045.stdout:1173: copy_from oid 24 from oid 26 current snap is 108 2021-07-31T01:49:33.585 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 40 current snap is 108 2021-07-31T01:49:33.585 INFO:tasks.rados.rados.0.smithi045.stdout:1174: seq_num 365 ranges {6389760=516096,6905856=573440,7479296=229376} 2021-07-31T01:49:33.622 INFO:tasks.rados.rados.0.smithi045.stdout:1174: writing smithi04569269-40 from 6389760 to 6905856 tid 1 2021-07-31T01:49:33.625 INFO:tasks.rados.rados.0.smithi045.stdout:1174: writing smithi04569269-40 from 6905856 to 7479296 tid 2 2021-07-31T01:49:33.626 INFO:tasks.rados.rados.0.smithi045.stdout:1174: writing smithi04569269-40 from 7479296 to 7708672 tid 3 2021-07-31T01:49:33.627 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 36 current snap is 108 2021-07-31T01:49:33.627 INFO:tasks.rados.rados.0.smithi045.stdout:1173: finishing copy_from to smithi04569269-24 2021-07-31T01:49:33.627 INFO:tasks.rados.rados.0.smithi045.stdout:1173: got expected ENOENT (src dne) 2021-07-31T01:49:33.627 INFO:tasks.rados.rados.0.smithi045.stdout:1175: seq_num 366 ranges {1253376=532480,1785856=458752,2244608=344064} 2021-07-31T01:49:33.636 INFO:tasks.rados.rados.0.smithi045.stdout:1175: writing smithi04569269-36 from 1253376 to 1785856 tid 1 2021-07-31T01:49:33.638 INFO:tasks.rados.rados.0.smithi045.stdout:1175: writing smithi04569269-36 from 1785856 to 2244608 tid 2 2021-07-31T01:49:33.642 INFO:tasks.rados.rados.0.smithi045.stdout:1175: writing smithi04569269-36 from 2244608 to 2588672 tid 3 2021-07-31T01:49:33.642 INFO:tasks.rados.rados.0.smithi045.stdout:1176: read oid 19 snap -1 2021-07-31T01:49:33.643 INFO:tasks.rados.rados.0.smithi045.stdout:1176: expect (ObjNum 354 snap 106 seq_num 354) 2021-07-31T01:49:33.643 INFO:tasks.rados.rados.0.smithi045.stdout:1177: setattr oid 48 current snap is 108 2021-07-31T01:49:33.643 INFO:tasks.rados.rados.0.smithi045.stdout:1173: finishing copy_from racing read to smithi04569269-24 2021-07-31T01:49:33.644 INFO:tasks.rados.rados.0.smithi045.stdout:1171: finishing copy_from to smithi04569269-13 2021-07-31T01:49:33.644 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 13 v 346 (ObjNum 320 snap 95 seq_num 320) dirty exists 2021-07-31T01:49:33.644 INFO:tasks.rados.rados.0.smithi045.stdout:1171: finishing copy_from racing read to smithi04569269-13 2021-07-31T01:49:33.644 INFO:tasks.rados.rados.0.smithi045.stdout:1170: expect (ObjNum 342 snap 102 seq_num 342) 2021-07-31T01:49:33.668 INFO:tasks.rados.rados.0.smithi045.stdout:1174: finishing write tid 1 to smithi04569269-40 2021-07-31T01:49:33.668 INFO:tasks.rados.rados.0.smithi045.stdout:1174: finishing write tid 2 to smithi04569269-40 2021-07-31T01:49:33.669 INFO:tasks.rados.rados.0.smithi045.stdout:1174: finishing write tid 3 to smithi04569269-40 2021-07-31T01:49:33.669 INFO:tasks.rados.rados.0.smithi045.stdout:1174: finishing write tid 4 to smithi04569269-40 2021-07-31T01:49:33.669 INFO:tasks.rados.rados.0.smithi045.stdout:1174: finishing write tid 5 to smithi04569269-40 2021-07-31T01:49:33.669 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 40 v 549 (ObjNum 365 snap 108 seq_num 365) dirty exists 2021-07-31T01:49:33.670 INFO:tasks.rados.rados.0.smithi045.stdout:1174: left oid 40 (ObjNum 365 snap 108 seq_num 365) 2021-07-31T01:49:33.670 INFO:tasks.rados.rados.0.smithi045.stdout:1175: finishing write tid 1 to smithi04569269-36 2021-07-31T01:49:33.670 INFO:tasks.rados.rados.0.smithi045.stdout:1175: finishing write tid 2 to smithi04569269-36 2021-07-31T01:49:33.670 INFO:tasks.rados.rados.0.smithi045.stdout:1175: finishing write tid 3 to smithi04569269-36 2021-07-31T01:49:33.671 INFO:tasks.rados.rados.0.smithi045.stdout:1175: finishing write tid 4 to smithi04569269-36 2021-07-31T01:49:33.671 INFO:tasks.rados.rados.0.smithi045.stdout:1175: finishing write tid 5 to smithi04569269-36 2021-07-31T01:49:33.671 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 36 v 797 (ObjNum 366 snap 108 seq_num 366) dirty exists 2021-07-31T01:49:33.671 INFO:tasks.rados.rados.0.smithi045.stdout:1175: left oid 36 (ObjNum 366 snap 108 seq_num 366) 2021-07-31T01:49:33.671 INFO:tasks.rados.rados.0.smithi045.stdout:1176: expect (ObjNum 354 snap 106 seq_num 354) 2021-07-31T01:49:33.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:49:33 smithi045 conmon[61210]: level=error ts=2021-07-31T01:49:33.512Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:49:33.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:49:33 smithi045 conmon[61210]: level=error ts=2021-07-31T01:49:33.512Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:49:33.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:49:33 smithi045 conmon[61210]: level=error ts=2021-07-31T01:49:33.512Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T01:49:33.849 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 48 v 333 (ObjNum 275 snap 81 seq_num 275) dirty exists 2021-07-31T01:49:33.849 INFO:tasks.rados.rados.0.smithi045.stdout:1170: done (6 left) 2021-07-31T01:49:33.849 INFO:tasks.rados.rados.0.smithi045.stdout:1171: done (5 left) 2021-07-31T01:49:33.850 INFO:tasks.rados.rados.0.smithi045.stdout:1173: done (4 left) 2021-07-31T01:49:33.850 INFO:tasks.rados.rados.0.smithi045.stdout:1174: done (3 left) 2021-07-31T01:49:33.850 INFO:tasks.rados.rados.0.smithi045.stdout:1175: done (2 left) 2021-07-31T01:49:33.850 INFO:tasks.rados.rados.0.smithi045.stdout:1176: done (1 left) 2021-07-31T01:49:33.851 INFO:tasks.rados.rados.0.smithi045.stdout:1177: done (0 left) 2021-07-31T01:49:33.851 INFO:tasks.rados.rados.0.smithi045.stdout:1178: delete oid 26 current snap is 108 2021-07-31T01:49:33.852 INFO:tasks.rados.rados.0.smithi045.stdout:1178: done (0 left) 2021-07-31T01:49:33.853 INFO:tasks.rados.rados.0.smithi045.stdout:1179: delete oid 5 current snap is 108 2021-07-31T01:49:33.853 INFO:tasks.rados.rados.0.smithi045.stdout:1179: done (0 left) 2021-07-31T01:49:33.853 INFO:tasks.rados.rados.0.smithi045.stdout:1180: copy_from oid 21 from oid 29 current snap is 108 2021-07-31T01:49:33.853 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 3 current snap is 108 2021-07-31T01:49:33.853 INFO:tasks.rados.rados.0.smithi045.stdout:1181: seq_num 369 ranges {0=589824,589824=507904,1097728=294912} 2021-07-31T01:49:33.856 INFO:tasks.rados.rados.0.smithi045.stdout:1181: writing smithi04569269-3 from 0 to 589824 tid 1 2021-07-31T01:49:33.858 INFO:tasks.rados.rados.0.smithi045.stdout:1181: writing smithi04569269-3 from 589824 to 1097728 tid 2 2021-07-31T01:49:33.861 INFO:tasks.rados.rados.0.smithi045.stdout:1181: writing smithi04569269-3 from 1097728 to 1392640 tid 3 2021-07-31T01:49:33.861 INFO:tasks.rados.rados.0.smithi045.stdout:1182: copy_from oid 35 from oid 34 current snap is 108 2021-07-31T01:49:33.861 INFO:tasks.rados.rados.0.smithi045.stdout:1181: finishing write tid 1 to smithi04569269-3 2021-07-31T01:49:33.862 INFO:tasks.rados.rados.0.smithi045.stdout:1183: read oid 19 snap -1 2021-07-31T01:49:33.862 INFO:tasks.rados.rados.0.smithi045.stdout:1183: expect (ObjNum 354 snap 106 seq_num 354) 2021-07-31T01:49:33.862 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 41 current snap is 108 2021-07-31T01:49:33.862 INFO:tasks.rados.rados.0.smithi045.stdout:1184: seq_num 370 ranges {0=606208,606208=794624,1400832=8192} 2021-07-31T01:49:33.864 INFO:tasks.rados.rados.0.smithi045.stdout:1184: writing smithi04569269-41 from 0 to 606208 tid 1 2021-07-31T01:49:33.869 INFO:tasks.rados.rados.0.smithi045.stdout:1184: writing smithi04569269-41 from 606208 to 1400832 tid 2 2021-07-31T01:49:33.870 INFO:tasks.rados.rados.0.smithi045.stdout:1184: writing smithi04569269-41 from 1400832 to 1409024 tid 3 2021-07-31T01:49:33.870 INFO:tasks.rados.rados.0.smithi045.stdout:1185: copy_from oid 31 from oid 36 current snap is 108 2021-07-31T01:49:33.870 INFO:tasks.rados.rados.0.smithi045.stdout:1181: finishing write tid 2 to smithi04569269-3 2021-07-31T01:49:33.870 INFO:tasks.rados.rados.0.smithi045.stdout:1186: rmattr oid 44 current snap is 108 2021-07-31T01:49:33.871 INFO:tasks.rados.rados.0.smithi045.stdout:1181: finishing write tid 3 to smithi04569269-3 2021-07-31T01:49:33.871 INFO:tasks.rados.rados.0.smithi045.stdout:1181: finishing write tid 4 to smithi04569269-3 2021-07-31T01:49:33.871 INFO:tasks.rados.rados.0.smithi045.stdout:1181: finishing write tid 5 to smithi04569269-3 2021-07-31T01:49:33.871 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 3 v 801 (ObjNum 369 snap 108 seq_num 369) dirty exists 2021-07-31T01:49:33.872 INFO:tasks.rados.rados.0.smithi045.stdout:1181: left oid 3 (ObjNum 369 snap 108 seq_num 369) 2021-07-31T01:49:33.872 INFO:tasks.rados.rados.0.smithi045.stdout:1181: done (6 left) 2021-07-31T01:49:33.872 INFO:tasks.rados.rados.0.smithi045.stdout:1187: snap_remove snap 106 2021-07-31T01:49:34.584 INFO:tasks.rados.rados.0.smithi045.stdout:1187: done (6 left) 2021-07-31T01:49:34.584 INFO:tasks.rados.rados.0.smithi045.stdout:1188: copy_from oid 49 from oid 30 current snap is 108 2021-07-31T01:49:34.585 INFO:tasks.rados.rados.0.smithi045.stdout:1189: snap_create 2021-07-31T01:49:34.585 INFO:tasks.rados.rados.0.smithi045.stdout:1184: finishing write tid 1 to smithi04569269-41 2021-07-31T01:49:34.585 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 44 v 364 (ObjNum 351 snap 105 seq_num 351) dirty exists 2021-07-31T01:49:34.585 INFO:tasks.rados.rados.0.smithi045.stdout:1184: finishing write tid 2 to smithi04569269-41 2021-07-31T01:49:34.586 INFO:tasks.rados.rados.0.smithi045.stdout:1184: finishing write tid 3 to smithi04569269-41 2021-07-31T01:49:34.586 INFO:tasks.rados.rados.0.smithi045.stdout:1184: finishing write tid 4 to smithi04569269-41 2021-07-31T01:49:34.586 INFO:tasks.rados.rados.0.smithi045.stdout:1184: finishing write tid 5 to smithi04569269-41 2021-07-31T01:49:34.586 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 41 v 519 (ObjNum 370 snap 108 seq_num 370) dirty exists 2021-07-31T01:49:34.587 INFO:tasks.rados.rados.0.smithi045.stdout:1184: left oid 41 (ObjNum 370 snap 108 seq_num 370) 2021-07-31T01:49:34.587 INFO:tasks.rados.rados.0.smithi045.stdout:1183: expect (ObjNum 354 snap 106 seq_num 354) 2021-07-31T01:49:34.767 INFO:tasks.rados.rados.0.smithi045.stdout:1182: finishing copy_from to smithi04569269-35 2021-07-31T01:49:34.768 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 35 v 332 (ObjNum 341 snap 101 seq_num 341) dirty exists 2021-07-31T01:49:34.769 INFO:tasks.rados.rados.0.smithi045.stdout:1182: finishing copy_from racing read to smithi04569269-35 2021-07-31T01:49:34.769 INFO:tasks.rados.rados.0.smithi045.stdout:1180: finishing copy_from to smithi04569269-21 2021-07-31T01:49:34.769 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 21 v 187 (ObjNum 335 snap 99 seq_num 335) dirty exists 2021-07-31T01:49:34.769 INFO:tasks.rados.rados.0.smithi045.stdout:1180: finishing copy_from racing read to smithi04569269-21 2021-07-31T01:49:34.770 INFO:tasks.rados.rados.0.smithi045.stdout:1185: finishing copy_from to smithi04569269-31 2021-07-31T01:49:34.770 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 31 v 797 (ObjNum 366 snap 108 seq_num 366) dirty exists 2021-07-31T01:49:34.770 INFO:tasks.rados.rados.0.smithi045.stdout:1185: finishing copy_from racing read to smithi04569269-31 2021-07-31T01:49:34.770 INFO:tasks.rados.rados.0.smithi045.stdout:1188: finishing copy_from to smithi04569269-49 2021-07-31T01:49:34.771 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 49 v 310 (ObjNum 346 snap 103 seq_num 346) dirty exists 2021-07-31T01:49:34.771 INFO:tasks.rados.rados.0.smithi045.stdout:1188: finishing copy_from racing read to smithi04569269-49 2021-07-31T01:49:35.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:34 smithi154 conmon[34553]: cluster 2021-07-31T01:49:33.577461+0000 mon.a (mon.0) 916 2021-07-31T01:49:35.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:34 smithi154 conmon[34553]: : cluster [DBG] osdmap e274: 8 total, 8 up, 8 in 2021-07-31T01:49:35.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:34 smithi154 conmon[34553]: cluster 2021-07-31T01:49:33.823232+0000 mgr.x (mgr.24542) 103 : cluster [DBG] pgmap v191: 25 pgs: 25 active+clean; 133 MiB data, 785 MiB used, 714 GiB / 715 GiB avail; 4.2 MiB/s rd, 1.1 MiB/s wr, 18 op/s 2021-07-31T01:49:35.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:34 smithi045 conmon[33655]: cluster 2021-07-31T01:49:33.577461+0000 mon.a (mon.0) 2021-07-31T01:49:35.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:34 smithi045 conmon[33655]: 916 : cluster [DBG] osdmap e274: 8 total, 8 up, 8 in 2021-07-31T01:49:35.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:34 smithi045 conmon[33655]: cluster 2021-07-31T01:49:33.823232+0000 mgr.x (mgr.24542) 103 : cluster [DBG] pgmap v191: 25 pgs: 25 active+clean; 133 MiB data, 785 MiB used, 714 GiB / 715 GiB avail; 4.2 MiB/s rd, 1.1 MiB/s wr, 18 op/s 2021-07-31T01:49:35.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:34 smithi045 conmon[39718]: cluster 2021-07-31T01:49:33 2021-07-31T01:49:35.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:34 smithi045 conmon[39718]: .577461+0000 mon.a (mon.0) 916 : cluster [DBG] osdmap e274: 8 total, 8 up, 8 in 2021-07-31T01:49:35.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:34 smithi045 conmon[39718]: cluster 2021 2021-07-31T01:49:35.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:34 smithi045 conmon[39718]: -07-31T01:49:33.823232 2021-07-31T01:49:35.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:34 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 103 2021-07-31T01:49:35.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:34 smithi045 conmon[39718]: : cluster [DBG] pgmap v191: 25 pgs: 25 active+clean; 133 MiB data, 785 MiB used, 714 GiB / 715 GiB avail; 4.2 MiB/s rd, 1.1 MiB/s wr, 18 op/s 2021-07-31T01:49:35.585 INFO:tasks.rados.rados.0.smithi045.stdout:1180: done (7 left) 2021-07-31T01:49:35.585 INFO:tasks.rados.rados.0.smithi045.stdout:1182: done (6 left) 2021-07-31T01:49:35.585 INFO:tasks.rados.rados.0.smithi045.stdout:1183: done (5 left) 2021-07-31T01:49:35.586 INFO:tasks.rados.rados.0.smithi045.stdout:1184: done (4 left) 2021-07-31T01:49:35.586 INFO:tasks.rados.rados.0.smithi045.stdout:1185: done (3 left) 2021-07-31T01:49:35.586 INFO:tasks.rados.rados.0.smithi045.stdout:1186: done (2 left) 2021-07-31T01:49:35.586 INFO:tasks.rados.rados.0.smithi045.stdout:1188: done (1 left) 2021-07-31T01:49:35.586 INFO:tasks.rados.rados.0.smithi045.stdout:1189: done (0 left) 2021-07-31T01:49:35.587 INFO:tasks.rados.rados.0.smithi045.stdout:1190: read oid 31 snap 108 2021-07-31T01:49:35.587 INFO:tasks.rados.rados.0.smithi045.stdout:1190: expect (ObjNum 366 snap 108 seq_num 366) 2021-07-31T01:49:35.587 INFO:tasks.rados.rados.0.smithi045.stdout:1191: delete oid 4 current snap is 109 2021-07-31T01:49:35.590 INFO:tasks.rados.rados.0.smithi045.stdout:1191: done (1 left) 2021-07-31T01:49:35.590 INFO:tasks.rados.rados.0.smithi045.stdout:1192: snap_remove snap 103 2021-07-31T01:49:36.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:35 smithi154 conmon[34553]: cluster 2021-07-31T01:49:34.583272+0000 mon.a (mon.0) 917 : cluster [DBG] 2021-07-31T01:49:36.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:35 smithi154 conmon[34553]: osdmap e275: 8 total, 8 up, 8 in 2021-07-31T01:49:36.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:35 smithi045 conmon[33655]: cluster 2021-07-31T01:49:34. 2021-07-31T01:49:36.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:35 smithi045 conmon[33655]: 583272+0000 mon.a (mon.0) 917 : cluster [DBG] osdmap e275: 8 total, 8 up, 8 in 2021-07-31T01:49:36.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:35 smithi045 conmon[39718]: cluster 2021-07-31T01:49:34.583272+0000 mon.a 2021-07-31T01:49:36.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:35 smithi045 conmon[39718]: (mon.0) 917 : cluster [DBG] osdmap e275: 8 total, 8 up, 8 in 2021-07-31T01:49:36.596 INFO:tasks.rados.rados.0.smithi045.stdout:1192: done (1 left) 2021-07-31T01:49:36.597 INFO:tasks.rados.rados.0.smithi045.stdout:1193: read oid 9 snap -1 2021-07-31T01:49:36.597 INFO:tasks.rados.rados.0.smithi045.stdout:1193: expect (ObjNum 172 snap 41 seq_num 172) 2021-07-31T01:49:36.597 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 32 current snap is 109 2021-07-31T01:49:36.598 INFO:tasks.rados.rados.0.smithi045.stdout:1194: seq_num 372 ranges {0=638976,638976=565248,1204224=237568} 2021-07-31T01:49:36.600 INFO:tasks.rados.rados.0.smithi045.stdout:1194: writing smithi04569269-32 from 0 to 638976 tid 1 2021-07-31T01:49:36.604 INFO:tasks.rados.rados.0.smithi045.stdout:1194: writing smithi04569269-32 from 638976 to 1204224 tid 2 2021-07-31T01:49:36.606 INFO:tasks.rados.rados.0.smithi045.stdout:1194: writing smithi04569269-32 from 1204224 to 1441792 tid 3 2021-07-31T01:49:36.606 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 8 current snap is 109 2021-07-31T01:49:36.606 INFO:tasks.rados.rados.0.smithi045.stdout:1195: seq_num 373 ranges {1220608=655360,1875968=499712,2375680=303104} 2021-07-31T01:49:36.618 INFO:tasks.rados.rados.0.smithi045.stdout:1195: writing smithi04569269-8 from 1220608 to 1875968 tid 1 2021-07-31T01:49:36.621 INFO:tasks.rados.rados.0.smithi045.stdout:1195: writing smithi04569269-8 from 1875968 to 2375680 tid 2 2021-07-31T01:49:36.623 INFO:tasks.rados.rados.0.smithi045.stdout:1195: writing smithi04569269-8 from 2375680 to 2678784 tid 3 2021-07-31T01:49:36.624 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 27 current snap is 109 2021-07-31T01:49:36.624 INFO:tasks.rados.rados.0.smithi045.stdout:1190: expect (ObjNum 366 snap 108 seq_num 366) 2021-07-31T01:49:36.711 INFO:tasks.rados.rados.0.smithi045.stdout:1193: expect (ObjNum 172 snap 41 seq_num 172) 2021-07-31T01:49:36.728 INFO:tasks.rados.rados.0.smithi045.stdout:1194: finishing write tid 1 to smithi04569269-32 2021-07-31T01:49:36.728 INFO:tasks.rados.rados.0.smithi045.stdout:1194: finishing write tid 2 to smithi04569269-32 2021-07-31T01:49:36.728 INFO:tasks.rados.rados.0.smithi045.stdout:1194: finishing write tid 3 to smithi04569269-32 2021-07-31T01:49:36.728 INFO:tasks.rados.rados.0.smithi045.stdout:1194: finishing write tid 4 to smithi04569269-32 2021-07-31T01:49:36.729 INFO:tasks.rados.rados.0.smithi045.stdout:1194: finishing write tid 5 to smithi04569269-32 2021-07-31T01:49:36.729 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 32 v 222 (ObjNum 372 snap 109 seq_num 372) dirty exists 2021-07-31T01:49:36.729 INFO:tasks.rados.rados.0.smithi045.stdout:1194: left oid 32 (ObjNum 372 snap 109 seq_num 372) 2021-07-31T01:49:36.729 INFO:tasks.rados.rados.0.smithi045.stdout:1195: finishing write tid 1 to smithi04569269-8 2021-07-31T01:49:36.729 INFO:tasks.rados.rados.0.smithi045.stdout:1195: finishing write tid 2 to smithi04569269-8 2021-07-31T01:49:36.730 INFO:tasks.rados.rados.0.smithi045.stdout:1195: finishing write tid 3 to smithi04569269-8 2021-07-31T01:49:36.730 INFO:tasks.rados.rados.0.smithi045.stdout:1195: finishing write tid 4 to smithi04569269-8 2021-07-31T01:49:36.730 INFO:tasks.rados.rados.0.smithi045.stdout:1195: finishing write tid 5 to smithi04569269-8 2021-07-31T01:49:36.730 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 8 v 827 (ObjNum 373 snap 109 seq_num 373) dirty exists 2021-07-31T01:49:36.730 INFO:tasks.rados.rados.0.smithi045.stdout:1195: left oid 8 (ObjNum 373 snap 109 seq_num 373) 2021-07-31T01:49:36.731 INFO:tasks.rados.rados.0.smithi045.stdout:1196: seq_num 374 ranges {0=671744,671744=778240,1449984=24576} 2021-07-31T01:49:36.731 INFO:tasks.rados.rados.0.smithi045.stdout:1196: writing smithi04569269-27 from 0 to 671744 tid 1 2021-07-31T01:49:36.735 INFO:tasks.rados.rados.0.smithi045.stdout:1196: writing smithi04569269-27 from 671744 to 1449984 tid 2 2021-07-31T01:49:36.736 INFO:tasks.rados.rados.0.smithi045.stdout:1196: writing smithi04569269-27 from 1449984 to 1474560 tid 3 2021-07-31T01:49:36.736 INFO:tasks.rados.rados.0.smithi045.stdout:1196: finishing write tid 1 to smithi04569269-27 2021-07-31T01:49:36.736 INFO:tasks.rados.rados.0.smithi045.stdout:1190: done (4 left) 2021-07-31T01:49:36.736 INFO:tasks.rados.rados.0.smithi045.stdout:1193: done (3 left) 2021-07-31T01:49:36.736 INFO:tasks.rados.rados.0.smithi045.stdout:1194: done (2 left) 2021-07-31T01:49:36.737 INFO:tasks.rados.rados.0.smithi045.stdout:1195: done (1 left) 2021-07-31T01:49:36.737 INFO:tasks.rados.rados.0.smithi045.stdout:1197: snap_remove snap 108 2021-07-31T01:49:37.002 INFO:journalctl@ceph.osd.5.smithi154.stdout:Jul 31 01:49:36 smithi154 conmon[42050]: debug 2021-07-31T01:49:36.601+0000 7fc9b105c700 -1 osd.5 pg_epoch: 277 pg[3.cs0( v 277'474 (0'0,277'474] local-lis/les=56/57 n=6 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [5,3,6]p5(0) r=0 lpr=56 luod=275'472 crt=275'472 lcod 275'470 mlcod 275'470 active+clean+snaptrim trimq=[cf~1] ps=[d0~2,d3~2]] removing snap head 2021-07-31T01:49:37.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:36 smithi154 conmon[34553]: cluster 2021-07-31T01:49:37.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:36 smithi154 conmon[34553]: 2021-07-31T01:49:37.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:36 smithi154 conmon[34553]: 2021-07-31T01:49: 2021-07-31T01:49:37.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:36 smithi154 conmon[34553]: 35.584012+0000 mon.a (mon.0) 918 : cluster [DBG] osdmap e276: 8 total, 8 up, 8 in 2021-07-31T01:49:37.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:36 smithi154 conmon[34553]: cluster 2021-07-31T01:49:37.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:36 smithi154 conmon[34553]: 2021-07-31T01:49:35.823674+0000 2021-07-31T01:49:37.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:36 smithi154 conmon[34553]: mgr.x (mgr.24542) 104 : cluster [DBG] pgmap v194: 25 pgs: 25 active+clean; 137 MiB data, 790 MiB used, 714 GiB / 715 GiB avail; 7.1 MiB/s rd, 2.2 MiB/s wr, 24 op/s 2021-07-31T01:49:37.005 INFO:journalctl@ceph.osd.7.smithi154.stdout:Jul 31 01:49:36 smithi154 conmon[49149]: debug 2021-07-31T01:49:37.005 INFO:journalctl@ceph.osd.7.smithi154.stdout:Jul 31 01:49:36 smithi154 conmon[49149]: 2021-07-31T01:49:36.604+0000 7f8a719fc700 -1 osd.7 pg_epoch: 277 pg[3.ds0( v 275'504 (0'0,275'504] local-lis/les=56/57 n=8 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [7,5,3]p7(0) r=0 lpr=56 crt=275'504 lcod 274'502 mlcod 274'502 active+clean+snaptrim trimq=[cf~1] ps=[d0~2,d3~2]] removing snap head 2021-07-31T01:49:37.005 INFO:journalctl@ceph.osd.7.smithi154.stdout:Jul 31 01:49:36 smithi154 conmon[49149]: 2021-07-31T01:49:37.006 INFO:journalctl@ceph.osd.7.smithi154.stdout:Jul 31 01:49:36 smithi154 conmon[49149]: debug 2021-07-31T01:49:37.006 INFO:journalctl@ceph.osd.7.smithi154.stdout:Jul 31 01:49:36 smithi154 conmon[49149]: 2021-07-31T01:49:36.605+0000 7f8a711fb700 -1 osd.7 pg_epoch: 277 pg[3.es0( v 277'456 (0'0,277'456] local-lis/les=56/57 n=6 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [7,4,1]p7(0) r=0 lpr=56 luod=275'454 crt=275'454 lcod 275'452 mlcod 275'452 active+clean+snaptrim trimq=[cf~1] ps=[d0~2,d3~2]] removing snap head 2021-07-31T01:49:37.006 INFO:journalctl@ceph.osd.7.smithi154.stdout:Jul 31 01:49:36 smithi154 conmon[49149]: 2021-07-31T01:49:37.026 INFO:journalctl@ceph.osd.1.smithi045.stdout:Jul 31 01:49:36 smithi045 conmon[46771]: debug 2021-07-31T01:49:36.598+0000 7f41939a2700 -1 osd.1 pg_epoch: 277 pg[3.0s0( v 276'327 (0'0,276'327] local-lis/les=56/57 n=7 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [1,2,6]p1(0) r=0 lpr=56 crt=276'327 lcod 275'325 mlcod 275'325 active+clean+snaptrim trimq=[cf~1] ps=[d0~2,d3~2]] removing snap head 2021-07-31T01:49:37.026 INFO:journalctl@ceph.osd.3.smithi045.stdout:Jul 31 01:49:36 smithi045 conmon[54013]: debug 2021-07-31T01:49:36.594+0000 7f64fd0ce700 -1 osd.3 pg_epoch: 277 pg[3.2s0( v 274'201 (0'0,274'201] local-lis/les=56/57 n=3 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [3,5,6]p3(0) r=0 lpr=56 crt=274'201 lcod 272'199 mlcod 272'199 active+clean+snaptrim trimq=[cf~1] ps=[d0~2,d3~2]] removing snap head 2021-07-31T01:49:37.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:36 smithi045 conmon[39718]: cluster 2021-07-31T01:49:37.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:36 smithi045 conmon[39718]: 2021-07-31T01:49:35 2021-07-31T01:49:37.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:36 smithi045 conmon[39718]: .584012+0000 mon.a (mon.0) 918 : cluster [DBG] osdmap e276: 8 total, 8 up, 8 in 2021-07-31T01:49:37.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:36 smithi045 conmon[39718]: cluster 2021-07-31T01:49:35.823674+0000 mgr.x (mgr.24542) 104 : cluster [DBG] pgmap v194: 25 pgs: 25 active+clean; 137 MiB data, 790 MiB used, 714 GiB / 715 GiB avail; 7.1 MiB/s rd, 2.2 MiB/s wr, 24 op/s 2021-07-31T01:49:37.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:36 smithi045 conmon[33655]: cluster 2021-07-31T01:49:35.584012+0000 mon.a (mon.0) 918 : cluster 2021-07-31T01:49:37.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:36 smithi045 conmon[33655]: [DBG] osdmap e276: 8 total, 8 up, 8 in 2021-07-31T01:49:37.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:36 smithi045 conmon[33655]: cluster 2021-07-31T01:49:35.823674+0000 mgr.x (mgr.24542) 104 : cluster [DBG] pgmap v194: 25 pgs: 25 active+clean; 137 MiB data, 790 MiB used, 714 GiB / 715 GiB avail; 7.1 MiB/s rd, 2.2 MiB/s wr, 24 op/s 2021-07-31T01:49:37.600 INFO:tasks.rados.rados.0.smithi045.stdout:1197: done (1 left) 2021-07-31T01:49:37.601 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 4 current snap is 109 2021-07-31T01:49:37.601 INFO:tasks.rados.rados.0.smithi045.stdout:1196: finishing write tid 2 to smithi04569269-27 2021-07-31T01:49:37.601 INFO:tasks.rados.rados.0.smithi045.stdout:1198: seq_num 375 ranges {0=688128,688128=663552,1351680=139264} 2021-07-31T01:49:37.603 INFO:tasks.rados.rados.0.smithi045.stdout:1198: writing smithi04569269-4 from 0 to 688128 tid 1 2021-07-31T01:49:37.607 INFO:tasks.rados.rados.0.smithi045.stdout:1198: writing smithi04569269-4 from 688128 to 1351680 tid 2 2021-07-31T01:49:37.608 INFO:tasks.rados.rados.0.smithi045.stdout:1198: writing smithi04569269-4 from 1351680 to 1490944 tid 3 2021-07-31T01:49:37.608 INFO:tasks.rados.rados.0.smithi045.stdout:1199: setattr oid 31 current snap is 109 2021-07-31T01:49:37.608 INFO:tasks.rados.rados.0.smithi045.stdout:1196: finishing write tid 3 to smithi04569269-27 2021-07-31T01:49:37.609 INFO:tasks.rados.rados.0.smithi045.stdout:1196: finishing write tid 4 to smithi04569269-27 2021-07-31T01:49:37.609 INFO:tasks.rados.rados.0.smithi045.stdout:1196: finishing write tid 5 to smithi04569269-27 2021-07-31T01:49:37.609 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 27 v 372 (ObjNum 374 snap 109 seq_num 374) dirty exists 2021-07-31T01:49:37.609 INFO:tasks.rados.rados.0.smithi045.stdout:1196: left oid 27 (ObjNum 374 snap 109 seq_num 374) 2021-07-31T01:49:37.610 INFO:tasks.rados.rados.0.smithi045.stdout:1198: finishing write tid 1 to smithi04569269-4 2021-07-31T01:49:37.611 INFO:tasks.rados.rados.0.smithi045.stdout:1196: done (2 left) 2021-07-31T01:49:37.611 INFO:tasks.rados.rados.0.smithi045.stdout:1200: copy_from oid 47 from oid 9 current snap is 109 2021-07-31T01:49:37.611 INFO:tasks.rados.rados.0.smithi045.stdout:1201: snap_create 2021-07-31T01:49:37.612 INFO:tasks.rados.rados.0.smithi045.stdout:1198: finishing write tid 2 to smithi04569269-4 2021-07-31T01:49:37.613 INFO:tasks.rados.rados.0.smithi045.stdout:1198: finishing write tid 3 to smithi04569269-4 2021-07-31T01:49:37.614 INFO:tasks.rados.rados.0.smithi045.stdout:1198: finishing write tid 4 to smithi04569269-4 2021-07-31T01:49:37.617 INFO:tasks.rados.rados.0.smithi045.stdout:1198: finishing write tid 5 to smithi04569269-4 2021-07-31T01:49:37.617 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 4 v 339 (ObjNum 375 snap 109 seq_num 375) dirty exists 2021-07-31T01:49:37.617 INFO:tasks.rados.rados.0.smithi045.stdout:1198: left oid 4 (ObjNum 375 snap 109 seq_num 375) 2021-07-31T01:49:37.618 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 31 v 828 (ObjNum 366 snap 108 seq_num 366) dirty exists 2021-07-31T01:49:37.620 INFO:tasks.rados.rados.0.smithi045.stdout:1200: finishing copy_from to smithi04569269-47 2021-07-31T01:49:37.621 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 47 v 265 (ObjNum 172 snap 41 seq_num 172) dirty exists 2021-07-31T01:49:37.621 INFO:tasks.rados.rados.0.smithi045.stdout:1200: finishing copy_from racing read to smithi04569269-47 2021-07-31T01:49:38.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:37 smithi154 conmon[34553]: cluster 2021-07-31T01:49:36.593914+0000 mon.a (mon.0) 919 : 2021-07-31T01:49:38.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:37 smithi154 conmon[34553]: cluster [DBG] osdmap e277: 8 total, 8 up, 8 in 2021-07-31T01:49:38.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:37 smithi045 conmon[39718]: cluster 2021-07-31T01:49:36.593914+0000 2021-07-31T01:49:38.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:37 smithi045 conmon[39718]: mon.a (mon.0) 919 : cluster [DBG] osdmap e277: 8 total, 8 up, 8 in 2021-07-31T01:49:38.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:37 smithi045 conmon[33655]: cluster 2021-07-31T01:49:36.593914+0000 mon.a (mon.0) 919 : 2021-07-31T01:49:38.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:37 smithi045 conmon[33655]: cluster [DBG] osdmap e277: 8 total, 8 up, 8 in 2021-07-31T01:49:38.605 INFO:tasks.rados.rados.0.smithi045.stdout:1198: done (3 left) 2021-07-31T01:49:38.606 INFO:tasks.rados.rados.0.smithi045.stdout:1199: done (2 left) 2021-07-31T01:49:38.606 INFO:tasks.rados.rados.0.smithi045.stdout:1200: done (1 left) 2021-07-31T01:49:38.606 INFO:tasks.rados.rados.0.smithi045.stdout:1201: done (0 left) 2021-07-31T01:49:38.606 INFO:tasks.rados.rados.0.smithi045.stdout:1202: snap_remove snap 109 2021-07-31T01:49:39.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:38 smithi154 conmon[34553]: cluster 2021-07-31T01:49:37.599152+0000 mon.a (mon.0) 920 : cluster [DBG] osdmap e278: 8 total, 8 up, 8 in 2021-07-31T01:49:39.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:38 smithi154 conmon[34553]: cluster 2021-07-31T01:49:37.824108+0000 mgr.x (mgr.24542) 105 : cluster [DBG] pgmap v197: 25 pgs: 25 active+clean; 137 MiB data, 790 MiB used, 714 GiB / 715 GiB avail; 2.9 MiB/s rd, 1.2 MiB/s wr, 6 op/s 2021-07-31T01:49:39.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:38 smithi045 conmon[39718]: cluster 2021-07-31T01:49:37.599152+0000 mon.a (mon.0 2021-07-31T01:49:39.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:38 smithi045 conmon[39718]: ) 920 : cluster [DBG] osdmap e278: 8 total, 8 up, 8 in 2021-07-31T01:49:39.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:38 smithi045 conmon[39718]: cluster 2021-07-31T01:49:37.824108+0000 mgr.x (mgr. 2021-07-31T01:49:39.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:38 smithi045 conmon[39718]: 24542) 105 : cluster [DBG] pgmap v197: 25 pgs: 25 active+clean; 137 MiB data, 790 MiB used, 714 GiB / 715 GiB avail; 2.9 MiB/s rd, 1.2 MiB/s wr, 6 op/s 2021-07-31T01:49:39.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:38 smithi045 conmon[33655]: cluster 2021-07-31T01:49:37.599152+0000 mon.a (mon.0) 920 : cluster [DBG] 2021-07-31T01:49:39.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:38 smithi045 conmon[33655]: osdmap e278: 8 total, 8 up, 8 in 2021-07-31T01:49:39.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:38 smithi045 conmon[33655]: cluster 2021-07-31T01:49:37.824108+0000 mgr.x (mgr.24542) 105 : cluster [DBG] pgmap v197: 25 pgs: 25 active+clean; 137 MiB data, 790 MiB used, 714 GiB / 715 GiB avail; 2.9 MiB/s rd, 1.2 MiB/s wr, 6 op/s 2021-07-31T01:49:39.607 INFO:tasks.rados.rados.0.smithi045.stdout:1202: done (0 left) 2021-07-31T01:49:39.607 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 8 current snap is 110 2021-07-31T01:49:39.608 INFO:tasks.rados.rados.0.smithi045.stdout:1203: seq_num 376 ranges {2678784=704512,3383296=548864,3932160=253952} 2021-07-31T01:49:39.625 INFO:tasks.rados.rados.0.smithi045.stdout:1203: writing smithi04569269-8 from 2678784 to 3383296 tid 1 2021-07-31T01:49:39.629 INFO:tasks.rados.rados.0.smithi045.stdout:1203: writing smithi04569269-8 from 3383296 to 3932160 tid 2 2021-07-31T01:49:39.631 INFO:tasks.rados.rados.0.smithi045.stdout:1203: writing smithi04569269-8 from 3932160 to 4186112 tid 3 2021-07-31T01:49:39.631 INFO:tasks.rados.rados.0.smithi045.stdout:1204: rollback oid 31 current snap is 110 2021-07-31T01:49:39.631 INFO:tasks.rados.rados.0.smithi045.stdout:1204: done (1 left) 2021-07-31T01:49:39.632 INFO:tasks.rados.rados.0.smithi045.stdout:1205: copy_from oid 3 from oid 6 current snap is 110 2021-07-31T01:49:39.632 INFO:tasks.rados.rados.0.smithi045.stdout:1203: finishing write tid 1 to smithi04569269-8 2021-07-31T01:49:39.632 INFO:tasks.rados.rados.0.smithi045.stdout:1206: read oid 40 snap -1 2021-07-31T01:49:39.632 INFO:tasks.rados.rados.0.smithi045.stdout:1206: expect (ObjNum 365 snap 108 seq_num 365) 2021-07-31T01:49:39.632 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 2 current snap is 110 2021-07-31T01:49:39.633 INFO:tasks.rados.rados.0.smithi045.stdout:1207: seq_num 377 ranges {2359296=720896,3080192=434176,3514368=368640} 2021-07-31T01:49:39.648 INFO:tasks.rados.rados.0.smithi045.stdout:1207: writing smithi04569269-2 from 2359296 to 3080192 tid 1 2021-07-31T01:49:39.650 INFO:tasks.rados.rados.0.smithi045.stdout:1207: writing smithi04569269-2 from 3080192 to 3514368 tid 2 2021-07-31T01:49:39.653 INFO:tasks.rados.rados.0.smithi045.stdout:1207: writing smithi04569269-2 from 3514368 to 3883008 tid 3 2021-07-31T01:49:39.653 INFO:tasks.rados.rados.0.smithi045.stdout:1208: snap_create 2021-07-31T01:49:39.653 INFO:tasks.rados.rados.0.smithi045.stdout:1203: finishing write tid 2 to smithi04569269-8 2021-07-31T01:49:39.654 INFO:tasks.rados.rados.0.smithi045.stdout:1203: finishing write tid 3 to smithi04569269-8 2021-07-31T01:49:39.654 INFO:tasks.rados.rados.0.smithi045.stdout:1203: finishing write tid 4 to smithi04569269-8 2021-07-31T01:49:39.654 INFO:tasks.rados.rados.0.smithi045.stdout:1203: finishing write tid 5 to smithi04569269-8 2021-07-31T01:49:39.654 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 8 v 834 (ObjNum 376 snap 110 seq_num 376) dirty exists 2021-07-31T01:49:39.654 INFO:tasks.rados.rados.0.smithi045.stdout:1203: left oid 8 (ObjNum 376 snap 110 seq_num 376) 2021-07-31T01:49:39.655 INFO:tasks.rados.rados.0.smithi045.stdout:1207: finishing write tid 1 to smithi04569269-2 2021-07-31T01:49:39.655 INFO:tasks.rados.rados.0.smithi045.stdout:1207: finishing write tid 2 to smithi04569269-2 2021-07-31T01:49:39.656 INFO:tasks.rados.rados.0.smithi045.stdout:1207: finishing write tid 3 to smithi04569269-2 2021-07-31T01:49:39.656 INFO:tasks.rados.rados.0.smithi045.stdout:1207: finishing write tid 4 to smithi04569269-2 2021-07-31T01:49:39.657 INFO:tasks.rados.rados.0.smithi045.stdout:1207: finishing write tid 5 to smithi04569269-2 2021-07-31T01:49:39.658 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 2 v 275 (ObjNum 377 snap 110 seq_num 377) dirty exists 2021-07-31T01:49:39.658 INFO:tasks.rados.rados.0.smithi045.stdout:1207: left oid 2 (ObjNum 377 snap 110 seq_num 377) 2021-07-31T01:49:39.668 INFO:tasks.rados.rados.0.smithi045.stdout:1206: expect (ObjNum 365 snap 108 seq_num 365) 2021-07-31T01:49:40.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:39 smithi154 conmon[34553]: cluster 2021-07-31T01:49:38.604663+0000 mon.a (mon.0) 921 : cluster [DBG] osdmap e279: 8 total, 8 up, 8 in 2021-07-31T01:49:40.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:39 smithi045 conmon[39718]: cluster 2021-07-31T01:49:38.604663+0000 mon.a ( 2021-07-31T01:49:40.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:39 smithi045 conmon[39718]: mon.0) 921 : cluster [DBG] osdmap e279: 8 total, 8 up, 8 in 2021-07-31T01:49:40.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:39 smithi045 conmon[33655]: cluster 2021-07-31T01:49:38.604663+0000 2021-07-31T01:49:40.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:39 smithi045 conmon[33655]: mon.a (mon.0) 921 : cluster [DBG] osdmap e279: 8 total, 8 up, 8 in 2021-07-31T01:49:40.216 INFO:tasks.rados.rados.0.smithi045.stdout:1205: finishing copy_from to smithi04569269-3 2021-07-31T01:49:40.217 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 3 v 542 (ObjNum 215 snap 56 seq_num 215) dirty exists 2021-07-31T01:49:40.217 INFO:tasks.rados.rados.0.smithi045.stdout:1205: finishing copy_from racing read to smithi04569269-3 2021-07-31T01:49:40.615 INFO:tasks.rados.rados.0.smithi045.stdout:1203: done (4 left) 2021-07-31T01:49:40.616 INFO:tasks.rados.rados.0.smithi045.stdout:1205: done (3 left) 2021-07-31T01:49:40.616 INFO:tasks.rados.rados.0.smithi045.stdout:1206: done (2 left) 2021-07-31T01:49:40.616 INFO:tasks.rados.rados.0.smithi045.stdout:1207: done (1 left) 2021-07-31T01:49:40.617 INFO:tasks.rados.rados.0.smithi045.stdout:1208: done (0 left) 2021-07-31T01:49:40.617 INFO:tasks.rados.rados.0.smithi045.stdout:1209: snap_remove snap 110 2021-07-31T01:49:41.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:40 smithi154 conmon[34553]: cluster 2021-07-31T01:49:39.606472+0000 mon.a (mon.0) 922 : cluster [DBG] 2021-07-31T01:49:41.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:40 smithi154 conmon[34553]: osdmap e280: 8 total, 8 up, 8 in 2021-07-31T01:49:41.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:40 smithi154 conmon[34553]: cluster 2021-07-31T01:49:39.824498+0000 mgr.x (mgr.24542) 106 : cluster [DBG] pgmap v200: 25 pgs: 25 active+clean; 108 MiB data, 766 MiB used, 714 GiB / 715 GiB avail; 5.1 MiB/s rd, 5.1 MiB/s wr, 22 op/s 2021-07-31T01:49:41.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:40 smithi045 conmon[33655]: cluster 2021-07-31T01:49:39.606472+0000 mon.a (mon.0) 922 : cluster [DBG] osdmap e280: 8 total, 8 up, 8 in 2021-07-31T01:49:41.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:40 smithi045 conmon[33655]: cluster 2021-07-31T01:49:39.824498+0000 mgr.x (mgr.24542) 106 : cluster [DBG] pgmap v200: 25 pgs: 25 active+clean; 108 MiB data, 766 MiB used, 714 GiB / 715 GiB avail; 5.1 MiB/s rd, 5.1 MiB/s wr, 22 op/s 2021-07-31T01:49:41.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:40 smithi045 conmon[39718]: cluster 2021-07-31T01:49:39.606472+0000 2021-07-31T01:49:41.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:40 smithi045 conmon[39718]: mon.a (mon.0) 922 : cluster [DBG] osdmap e280: 8 total, 8 up, 8 in 2021-07-31T01:49:41.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:40 smithi045 conmon[39718]: cluster 2021-07-31 2021-07-31T01:49:41.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:40 smithi045 conmon[39718]: T01:49:39.824498+0000 mgr.x (mgr.24542) 106 : cluster [DBG] pgmap v200: 25 pgs: 25 active+clean; 108 MiB data, 766 MiB used, 714 GiB / 715 GiB avail; 5.1 MiB/s rd, 5.1 MiB/s wr, 22 op/s 2021-07-31T01:49:41.527 INFO:tasks.rados.rados.0.smithi045.stdout:1209: done (0 left) 2021-07-31T01:49:41.527 INFO:tasks.rados.rados.0.smithi045.stdout:1210: setattr oid 38 current snap is 111 2021-07-31T01:49:41.529 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 22 current snap is 111 2021-07-31T01:49:41.530 INFO:tasks.rados.rados.0.smithi045.stdout:1211: seq_num 378 ranges {1196032=737280,1933312=720896,2654208=81920} 2021-07-31T01:49:41.539 INFO:tasks.rados.rados.0.smithi045.stdout:1211: writing smithi04569269-22 from 1196032 to 1933312 tid 1 2021-07-31T01:49:41.543 INFO:tasks.rados.rados.0.smithi045.stdout:1211: writing smithi04569269-22 from 1933312 to 2654208 tid 2 2021-07-31T01:49:41.544 INFO:tasks.rados.rados.0.smithi045.stdout:1211: writing smithi04569269-22 from 2654208 to 2736128 tid 3 2021-07-31T01:49:41.544 INFO:tasks.rados.rados.0.smithi045.stdout:1212: rmattr oid 15 current snap is 111 2021-07-31T01:49:41.544 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 38 v 544 (ObjNum 244 snap 67 seq_num 244) dirty exists 2021-07-31T01:49:41.545 INFO:tasks.rados.rados.0.smithi045.stdout:1210: done (2 left) 2021-07-31T01:49:41.545 INFO:tasks.rados.rados.0.smithi045.stdout:1212: done (1 left) 2021-07-31T01:49:41.545 INFO:tasks.rados.rados.0.smithi045.stdout:1213: delete oid 23 current snap is 111 2021-07-31T01:49:41.546 INFO:tasks.rados.rados.0.smithi045.stdout:1211: finishing write tid 1 to smithi04569269-22 2021-07-31T01:49:41.546 INFO:tasks.rados.rados.0.smithi045.stdout:1213: done (1 left) 2021-07-31T01:49:41.546 INFO:tasks.rados.rados.0.smithi045.stdout:1214: rmattr oid 23 current snap is 111 2021-07-31T01:49:41.547 INFO:tasks.rados.rados.0.smithi045.stdout:1214: done (1 left) 2021-07-31T01:49:41.547 INFO:tasks.rados.rados.0.smithi045.stdout:1215: rmattr oid 41 current snap is 111 2021-07-31T01:49:41.547 INFO:tasks.rados.rados.0.smithi045.stdout:1215: done (1 left) 2021-07-31T01:49:41.547 INFO:tasks.rados.rados.0.smithi045.stdout:1216: snap_create 2021-07-31T01:49:41.548 INFO:tasks.rados.rados.0.smithi045.stdout:1211: finishing write tid 2 to smithi04569269-22 2021-07-31T01:49:41.548 INFO:tasks.rados.rados.0.smithi045.stdout:1211: finishing write tid 3 to smithi04569269-22 2021-07-31T01:49:41.549 INFO:tasks.rados.rados.0.smithi045.stdout:1211: finishing write tid 4 to smithi04569269-22 2021-07-31T01:49:41.549 INFO:tasks.rados.rados.0.smithi045.stdout:1211: finishing write tid 5 to smithi04569269-22 2021-07-31T01:49:41.549 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 22 v 220 (ObjNum 378 snap 111 seq_num 378) dirty exists 2021-07-31T01:49:41.550 INFO:tasks.rados.rados.0.smithi045.stdout:1211: left oid 22 (ObjNum 378 snap 111 seq_num 378) 2021-07-31T01:49:42.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:41 smithi154 conmon[34553]: cluster 2021-07-31T01:49:40.615172+0000 mon.a (mon.0) 923 : cluster [DBG] osdmap e281: 8 total, 8 up, 8 in 2021-07-31T01:49:42.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:41 smithi154 conmon[34553]: cluster 2021-07-31T01:49:41. 2021-07-31T01:49:42.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:41 smithi154 conmon[34553]: 526143+0000 mon.a (mon.0) 924 : cluster [DBG] osdmap e282: 8 total, 8 up, 8 in 2021-07-31T01:49:42.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:41 smithi045 conmon[39718]: cluster 2021-07-31T01:49:40 2021-07-31T01:49:42.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:41 smithi045 conmon[39718]: .615172+0000 mon.a (mon.0) 923 : cluster [DBG] osdmap e281: 8 total, 8 up, 8 in 2021-07-31T01:49:42.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:41 smithi045 conmon[39718]: cluster 2021-07-31T01:49:42.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:41 smithi045 conmon[39718]: 2021-07-31T01: 2021-07-31T01:49:42.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:41 smithi045 conmon[39718]: 49:41.526143+0000 mon.a 2021-07-31T01:49:42.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:41 smithi045 conmon[39718]: (mon.0) 924 : cluster 2021-07-31T01:49:42.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:41 smithi045 conmon[39718]: [DBG] osdmap e282: 8 total, 8 up, 8 in 2021-07-31T01:49:42.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:41 smithi045 conmon[33655]: cluster 2021-07-31T01:49:40.615172+0000 mon.a (mon 2021-07-31T01:49:42.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:41 smithi045 conmon[33655]: .0) 923 : cluster [DBG] osdmap e281: 8 total, 8 up, 8 in 2021-07-31T01:49:42.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:41 smithi045 conmon[33655]: cluster 2021-07-31T01:49:41.526143+0000 mon.a (mon.0 2021-07-31T01:49:42.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:41 smithi045 conmon[33655]: ) 924 : cluster [DBG] osdmap e282: 8 total, 8 up, 8 in 2021-07-31T01:49:42.533 INFO:tasks.rados.rados.0.smithi045.stdout:1211: done (1 left) 2021-07-31T01:49:42.533 INFO:tasks.rados.rados.0.smithi045.stdout:1216: done (0 left) 2021-07-31T01:49:42.534 INFO:tasks.rados.rados.0.smithi045.stdout:1217: rollback oid 39 current snap is 112 2021-07-31T01:49:42.534 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 39 to 111 2021-07-31T01:49:42.534 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 44 current snap is 112 2021-07-31T01:49:42.534 INFO:tasks.rados.rados.0.smithi045.stdout:1218: seq_num 380 ranges {5398528=770048,6168576=483328,6651904=319488} 2021-07-31T01:49:42.567 INFO:tasks.rados.rados.0.smithi045.stdout:1218: writing smithi04569269-44 from 5398528 to 6168576 tid 1 2021-07-31T01:49:42.569 INFO:tasks.rados.rados.0.smithi045.stdout:1218: writing smithi04569269-44 from 6168576 to 6651904 tid 2 2021-07-31T01:49:42.572 INFO:tasks.rados.rados.0.smithi045.stdout:1218: writing smithi04569269-44 from 6651904 to 6971392 tid 3 2021-07-31T01:49:42.572 INFO:tasks.rados.rados.0.smithi045.stdout:1219: read oid 16 snap 111 2021-07-31T01:49:42.572 INFO:tasks.rados.rados.0.smithi045.stdout:1219: expect (ObjNum 207 snap 53 seq_num 207) 2021-07-31T01:49:42.573 INFO:tasks.rados.rados.0.smithi045.stdout:1220: read oid 17 snap -1 2021-07-31T01:49:42.573 INFO:tasks.rados.rados.0.smithi045.stdout:1220: expect (ObjNum 196 snap 52 seq_num 196) 2021-07-31T01:49:42.573 INFO:tasks.rados.rados.0.smithi045.stdout:1221: copy_from oid 50 from oid 25 current snap is 112 2021-07-31T01:49:42.573 INFO:tasks.rados.rados.0.smithi045.stdout:1217: finishing rollback tid 0 to smithi04569269-39 2021-07-31T01:49:42.573 INFO:tasks.rados.rados.0.smithi045.stdout:1217: finishing rollback tid 1 to smithi04569269-39 2021-07-31T01:49:42.574 INFO:tasks.rados.rados.0.smithi045.stdout:1217: finishing rollback tid 2 to smithi04569269-39 2021-07-31T01:49:42.574 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 39 v 548 (ObjNum 342 snap 102 seq_num 342) dirty exists 2021-07-31T01:49:42.574 INFO:tasks.rados.rados.0.smithi045.stdout:1217: done (4 left) 2021-07-31T01:49:42.574 INFO:tasks.rados.rados.0.smithi045.stdout:1222: rollback oid 36 current snap is 112 2021-07-31T01:49:42.575 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 36 to 111 2021-07-31T01:49:42.575 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 26 current snap is 112 2021-07-31T01:49:42.575 INFO:tasks.rados.rados.0.smithi045.stdout:1223: seq_num 381 ranges {0=786432,786432=417792,1204224=385024} 2021-07-31T01:49:42.576 INFO:tasks.rados.rados.0.smithi045.stdout:1223: writing smithi04569269-26 from 0 to 786432 tid 1 2021-07-31T01:49:42.579 INFO:tasks.rados.rados.0.smithi045.stdout:1223: writing smithi04569269-26 from 786432 to 1204224 tid 2 2021-07-31T01:49:42.582 INFO:tasks.rados.rados.0.smithi045.stdout:1223: writing smithi04569269-26 from 1204224 to 1589248 tid 3 2021-07-31T01:49:42.582 INFO:tasks.rados.rados.0.smithi045.stdout:1224: read oid 22 snap -1 2021-07-31T01:49:42.582 INFO:tasks.rados.rados.0.smithi045.stdout:1224: expect (ObjNum 378 snap 111 seq_num 378) 2021-07-31T01:49:42.582 INFO:tasks.rados.rados.0.smithi045.stdout:1225: rmattr oid 45 current snap is 112 2021-07-31T01:49:42.583 INFO:tasks.rados.rados.0.smithi045.stdout:1218: finishing write tid 1 to smithi04569269-44 2021-07-31T01:49:42.583 INFO:tasks.rados.rados.0.smithi045.stdout:1225: done (7 left) 2021-07-31T01:49:42.583 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 23 current snap is 112 2021-07-31T01:49:42.583 INFO:tasks.rados.rados.0.smithi045.stdout:1218: finishing write tid 2 to smithi04569269-44 2021-07-31T01:49:42.584 INFO:tasks.rados.rados.0.smithi045.stdout:1218: finishing write tid 3 to smithi04569269-44 2021-07-31T01:49:42.584 INFO:tasks.rados.rados.0.smithi045.stdout:1218: finishing write tid 4 to smithi04569269-44 2021-07-31T01:49:42.584 INFO:tasks.rados.rados.0.smithi045.stdout:1226: seq_num 382 ranges {0=802816,802816=704512,1507328=98304} 2021-07-31T01:49:42.586 INFO:tasks.rados.rados.0.smithi045.stdout:1226: writing smithi04569269-23 from 0 to 802816 tid 1 2021-07-31T01:49:42.591 INFO:tasks.rados.rados.0.smithi045.stdout:1226: writing smithi04569269-23 from 802816 to 1507328 tid 2 2021-07-31T01:49:42.592 INFO:tasks.rados.rados.0.smithi045.stdout:1226: writing smithi04569269-23 from 1507328 to 1605632 tid 3 2021-07-31T01:49:42.592 INFO:tasks.rados.rados.0.smithi045.stdout:1227: copy_from oid 37 from oid 47 current snap is 112 2021-07-31T01:49:42.592 INFO:tasks.rados.rados.0.smithi045.stdout:1218: finishing write tid 5 to smithi04569269-44 2021-07-31T01:49:42.592 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 44 v 377 (ObjNum 380 snap 112 seq_num 380) dirty exists 2021-07-31T01:49:42.593 INFO:tasks.rados.rados.0.smithi045.stdout:1218: left oid 44 (ObjNum 380 snap 112 seq_num 380) 2021-07-31T01:49:42.593 INFO:tasks.rados.rados.0.smithi045.stdout:1222: finishing rollback tid 0 to smithi04569269-36 2021-07-31T01:49:42.593 INFO:tasks.rados.rados.0.smithi045.stdout:1219: expect (ObjNum 207 snap 53 seq_num 207) 2021-07-31T01:49:42.623 INFO:tasks.rados.rados.0.smithi045.stdout:1222: finishing rollback tid 1 to smithi04569269-36 2021-07-31T01:49:42.624 INFO:tasks.rados.rados.0.smithi045.stdout:1222: finishing rollback tid 2 to smithi04569269-36 2021-07-31T01:49:42.624 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 36 v 839 (ObjNum 366 snap 108 seq_num 366) dirty exists 2021-07-31T01:49:42.624 INFO:tasks.rados.rados.0.smithi045.stdout:1218: done (8 left) 2021-07-31T01:49:42.625 INFO:tasks.rados.rados.0.smithi045.stdout:1219: done (7 left) 2021-07-31T01:49:42.625 INFO:tasks.rados.rados.0.smithi045.stdout:1222: done (6 left) 2021-07-31T01:49:42.625 INFO:tasks.rados.rados.0.smithi045.stdout:1228: read oid 5 snap -1 2021-07-31T01:49:42.625 INFO:tasks.rados.rados.0.smithi045.stdout:1228: expect deleted 2021-07-31T01:49:42.626 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 20 current snap is 112 2021-07-31T01:49:42.626 INFO:tasks.rados.rados.0.smithi045.stdout:1226: finishing write tid 1 to smithi04569269-23 2021-07-31T01:49:42.626 INFO:tasks.rados.rados.0.smithi045.stdout:1226: finishing write tid 2 to smithi04569269-23 2021-07-31T01:49:42.626 INFO:tasks.rados.rados.0.smithi045.stdout:1229: seq_num 383 ranges {3088384=417792,3506176=589824,4096000=417792,4513792=196608} 2021-07-31T01:49:42.642 INFO:tasks.rados.rados.0.smithi045.stdout:1229: writing smithi04569269-20 from 3088384 to 3506176 tid 1 2021-07-31T01:49:42.645 INFO:tasks.rados.rados.0.smithi045.stdout:1229: writing smithi04569269-20 from 3506176 to 4096000 tid 2 2021-07-31T01:49:42.648 INFO:tasks.rados.rados.0.smithi045.stdout:1229: writing smithi04569269-20 from 4096000 to 4513792 tid 3 2021-07-31T01:49:42.649 INFO:tasks.rados.rados.0.smithi045.stdout:1229: writing smithi04569269-20 from 4513792 to 4710400 tid 4 2021-07-31T01:49:42.650 INFO:tasks.rados.rados.0.smithi045.stdout:1230: snap_create 2021-07-31T01:49:42.650 INFO:tasks.rados.rados.0.smithi045.stdout:1226: finishing write tid 3 to smithi04569269-23 2021-07-31T01:49:42.650 INFO:tasks.rados.rados.0.smithi045.stdout:1226: finishing write tid 4 to smithi04569269-23 2021-07-31T01:49:42.650 INFO:tasks.rados.rados.0.smithi045.stdout:1226: finishing write tid 5 to smithi04569269-23 2021-07-31T01:49:42.650 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 23 v 552 (ObjNum 382 snap 112 seq_num 382) dirty exists 2021-07-31T01:49:42.651 INFO:tasks.rados.rados.0.smithi045.stdout:1226: left oid 23 (ObjNum 382 snap 112 seq_num 382) 2021-07-31T01:49:42.651 INFO:tasks.rados.rados.0.smithi045.stdout:1221: finishing copy_from to smithi04569269-50 2021-07-31T01:49:42.651 INFO:tasks.rados.rados.0.smithi045.stdout:1221: got expected ENOENT (src dne) 2021-07-31T01:49:42.651 INFO:tasks.rados.rados.0.smithi045.stdout:1221: finishing copy_from racing read to smithi04569269-50 2021-07-31T01:49:42.652 INFO:tasks.rados.rados.0.smithi045.stdout:1224: expect (ObjNum 378 snap 111 seq_num 378) 2021-07-31T01:49:42.752 INFO:tasks.rados.rados.0.smithi045.stdout:1227: finishing copy_from to smithi04569269-37 2021-07-31T01:49:42.753 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 37 v 265 (ObjNum 172 snap 41 seq_num 172) dirty exists 2021-07-31T01:49:42.753 INFO:tasks.rados.rados.0.smithi045.stdout:1227: finishing copy_from racing read to smithi04569269-37 2021-07-31T01:49:42.753 INFO:tasks.rados.rados.0.smithi045.stdout:1220: expect (ObjNum 196 snap 52 seq_num 196) 2021-07-31T01:49:42.977 INFO:tasks.rados.rados.0.smithi045.stdout:1223: finishing write tid 1 to smithi04569269-26 2021-07-31T01:49:42.978 INFO:tasks.rados.rados.0.smithi045.stdout:1223: finishing write tid 2 to smithi04569269-26 2021-07-31T01:49:42.978 INFO:tasks.rados.rados.0.smithi045.stdout:1223: finishing write tid 3 to smithi04569269-26 2021-07-31T01:49:42.978 INFO:tasks.rados.rados.0.smithi045.stdout:1223: finishing write tid 4 to smithi04569269-26 2021-07-31T01:49:42.979 INFO:tasks.rados.rados.0.smithi045.stdout:1223: finishing write tid 5 to smithi04569269-26 2021-07-31T01:49:42.979 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 26 v 226 (ObjNum 381 snap 112 seq_num 381) dirty exists 2021-07-31T01:49:42.979 INFO:tasks.rados.rados.0.smithi045.stdout:1223: left oid 26 (ObjNum 381 snap 112 seq_num 381) 2021-07-31T01:49:42.979 INFO:tasks.rados.rados.0.smithi045.stdout:1229: finishing write tid 1 to smithi04569269-20 2021-07-31T01:49:42.980 INFO:tasks.rados.rados.0.smithi045.stdout:1229: finishing write tid 2 to smithi04569269-20 2021-07-31T01:49:42.980 INFO:tasks.rados.rados.0.smithi045.stdout:1229: finishing write tid 3 to smithi04569269-20 2021-07-31T01:49:42.980 INFO:tasks.rados.rados.0.smithi045.stdout:1229: finishing write tid 4 to smithi04569269-20 2021-07-31T01:49:42.980 INFO:tasks.rados.rados.0.smithi045.stdout:1229: finishing write tid 5 to smithi04569269-20 2021-07-31T01:49:42.981 INFO:tasks.rados.rados.0.smithi045.stdout:1229: finishing write tid 6 to smithi04569269-20 2021-07-31T01:49:42.981 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 20 v 845 (ObjNum 383 snap 112 seq_num 383) dirty exists 2021-07-31T01:49:42.981 INFO:tasks.rados.rados.0.smithi045.stdout:1229: left oid 20 (ObjNum 383 snap 112 seq_num 383) 2021-07-31T01:49:43.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:42 smithi154 conmon[34553]: cluster 2021-07-31T01:49:41.825060+0000 mgr.x (mgr.24542) 107 2021-07-31T01:49:43.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:42 smithi154 conmon[34553]: : cluster [DBG] pgmap v203: 25 pgs: 25 active+clean; 108 MiB data, 766 MiB used, 714 GiB / 715 GiB avail; 5.1 MiB/s rd, 5.1 MiB/s wr, 22 op/s 2021-07-31T01:49:43.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:42 smithi154 conmon[34553]: cluster 2021-07-31T01:49:42.532658+0000 mon.a (mon.0) 925 : cluster [DBG] osdmap e283: 8 total, 8 up, 8 in 2021-07-31T01:49:43.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:42 smithi045 conmon[33655]: cluster 2021-07-31T01:49:41.825060+0000 mgr.x (mgr.24542) 107 2021-07-31T01:49:43.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:42 smithi045 conmon[33655]: : cluster [DBG] pgmap v203: 25 pgs: 25 active+clean; 108 MiB data, 766 MiB used, 714 GiB / 715 GiB avail; 5.1 MiB/s rd, 5.1 MiB/s wr, 22 op/s 2021-07-31T01:49:43.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:42 smithi045 conmon[33655]: cluster 2021-07-31T01:49:42.532658+0000 mon.a (mon.0) 925 : cluster [DBG] osdmap e283: 8 total, 8 up, 8 in 2021-07-31T01:49:43.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:42 smithi045 conmon[39718]: cluster 2021-07-31T01:49:43.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:42 smithi045 conmon[39718]: 2021-07 2021-07-31T01:49:43.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:42 smithi045 conmon[39718]: -31T01:49:41.825060+0000 mgr.x (mgr.24542) 107 : cluster [DBG] pgmap v203: 25 pgs: 25 active+clean; 108 MiB data, 766 MiB used, 714 GiB / 715 GiB avail; 5.1 MiB/s rd, 5.1 MiB/s wr, 22 op/s 2021-07-31T01:49:43.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:42 smithi045 conmon[39718]: cluster 2021-07-31T01:49:43.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:42 smithi045 conmon[39718]: 2021-07-31T01:49:42.532658+0000 2021-07-31T01:49:43.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:42 smithi045 conmon[39718]: mon.a (mon.0) 925 : cluster [DBG] 2021-07-31T01:49:43.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:42 smithi045 conmon[39718]: osdmap e283: 8 total, 8 up, 8 in 2021-07-31T01:49:43.624 INFO:tasks.rados.rados.0.smithi045.stdout:1220: done (8 left) 2021-07-31T01:49:43.624 INFO:tasks.rados.rados.0.smithi045.stdout:1221: done (7 left) 2021-07-31T01:49:43.624 INFO:tasks.rados.rados.0.smithi045.stdout:1223: done (6 left) 2021-07-31T01:49:43.625 INFO:tasks.rados.rados.0.smithi045.stdout:1224: done (5 left) 2021-07-31T01:49:43.625 INFO:tasks.rados.rados.0.smithi045.stdout:1226: done (4 left) 2021-07-31T01:49:43.625 INFO:tasks.rados.rados.0.smithi045.stdout:1227: done (3 left) 2021-07-31T01:49:43.625 INFO:tasks.rados.rados.0.smithi045.stdout:1228: done (2 left) 2021-07-31T01:49:43.625 INFO:tasks.rados.rados.0.smithi045.stdout:1229: done (1 left) 2021-07-31T01:49:43.626 INFO:tasks.rados.rados.0.smithi045.stdout:1230: done (0 left) 2021-07-31T01:49:43.626 INFO:tasks.rados.rados.0.smithi045.stdout:1231: copy_from oid 49 from oid 24 current snap is 113 2021-07-31T01:49:43.626 INFO:tasks.rados.rados.0.smithi045.stdout:1232: rollback oid 15 current snap is 113 2021-07-31T01:49:43.626 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 15 to 111 2021-07-31T01:49:43.627 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 35 current snap is 113 2021-07-31T01:49:43.627 INFO:tasks.rados.rados.0.smithi045.stdout:1233: seq_num 384 ranges {3694592=434176,4128768=475136,4603904=630784,5234688=98304} 2021-07-31T01:49:43.647 INFO:tasks.rados.rados.0.smithi045.stdout:1233: writing smithi04569269-35 from 3694592 to 4128768 tid 1 2021-07-31T01:49:43.649 INFO:tasks.rados.rados.0.smithi045.stdout:1233: writing smithi04569269-35 from 4128768 to 4603904 tid 2 2021-07-31T01:49:43.653 INFO:tasks.rados.rados.0.smithi045.stdout:1233: writing smithi04569269-35 from 4603904 to 5234688 tid 3 2021-07-31T01:49:43.654 INFO:tasks.rados.rados.0.smithi045.stdout:1233: writing smithi04569269-35 from 5234688 to 5332992 tid 4 2021-07-31T01:49:43.654 INFO:tasks.rados.rados.0.smithi045.stdout:1234: snap_create 2021-07-31T01:49:43.654 INFO:tasks.rados.rados.0.smithi045.stdout:1232: finishing rollback tid 1 to smithi04569269-15 2021-07-31T01:49:43.654 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 15 v 0 (ObjNum 0 snap 32731 seq_num 3189076432) dirty dne 2021-07-31T01:49:43.656 INFO:tasks.rados.rados.0.smithi045.stdout:1231: finishing copy_from to smithi04569269-49 2021-07-31T01:49:43.657 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 49 v 307 (ObjNum 285 snap 83 seq_num 285) dirty exists 2021-07-31T01:49:43.658 INFO:tasks.rados.rados.0.smithi045.stdout:1231: finishing copy_from racing read to smithi04569269-49 2021-07-31T01:49:43.658 INFO:tasks.rados.rados.0.smithi045.stdout:1233: finishing write tid 1 to smithi04569269-35 2021-07-31T01:49:43.658 INFO:tasks.rados.rados.0.smithi045.stdout:1233: finishing write tid 2 to smithi04569269-35 2021-07-31T01:49:43.661 INFO:tasks.rados.rados.0.smithi045.stdout:1233: finishing write tid 3 to smithi04569269-35 2021-07-31T01:49:43.661 INFO:tasks.rados.rados.0.smithi045.stdout:1233: finishing write tid 4 to smithi04569269-35 2021-07-31T01:49:43.661 INFO:tasks.rados.rados.0.smithi045.stdout:1233: finishing write tid 5 to smithi04569269-35 2021-07-31T01:49:43.662 INFO:tasks.rados.rados.0.smithi045.stdout:1233: finishing write tid 6 to smithi04569269-35 2021-07-31T01:49:43.663 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 35 v 853 (ObjNum 384 snap 113 seq_num 384) dirty exists 2021-07-31T01:49:43.663 INFO:tasks.rados.rados.0.smithi045.stdout:1233: left oid 35 (ObjNum 384 snap 113 seq_num 384) 2021-07-31T01:49:43.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:49:43 smithi045 conmon[61210]: level=error ts=2021-07-31T01:49:43.512Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:49:43.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:49:43 smithi045 conmon[61210]: level=error ts=2021-07-31T01:49:43.512Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:49:43.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:49:43 smithi045 conmon[61210]: level=error ts=2021-07-31T01:49:43.512Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T01:49:44.629 INFO:tasks.rados.rados.0.smithi045.stdout:1231: done (3 left) 2021-07-31T01:49:44.629 INFO:tasks.rados.rados.0.smithi045.stdout:1232: done (2 left) 2021-07-31T01:49:44.629 INFO:tasks.rados.rados.0.smithi045.stdout:1233: done (1 left) 2021-07-31T01:49:44.630 INFO:tasks.rados.rados.0.smithi045.stdout:1234: done (0 left) 2021-07-31T01:49:44.630 INFO:tasks.rados.rados.0.smithi045.stdout:1235: rollback oid 1 current snap is 114 2021-07-31T01:49:44.630 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 1 to 113 2021-07-31T01:49:44.630 INFO:tasks.rados.rados.0.smithi045.stdout:1236: rollback oid 36 current snap is 114 2021-07-31T01:49:44.631 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 36 to 113 2021-07-31T01:49:44.631 INFO:tasks.rados.rados.0.smithi045.stdout:1237: snap_remove snap 111 2021-07-31T01:49:45.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:44 smithi154 conmon[34553]: cluster 2021-07-31T01:49:43.622834+0000 mon.a (mon.0) 926 : cluster [DBG] osdmap e284: 8 total, 8 up, 8 in 2021-07-31T01:49:45.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:44 smithi154 conmon[34553]: 2021-07-31T01:49:45.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:44 smithi154 conmon[34553]: cluster 2021-07-31T01:49:43.825580+0000 mgr.x (mgr.24542) 108 : cluster [DBG] pgmap v206: 25 pgs: 25 active+clean; 113 MiB data, 779 MiB used, 714 GiB / 715 GiB avail; 9.1 MiB/s rd, 1.3 MiB/s wr, 11 op/s 2021-07-31T01:49:45.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:44 smithi045 conmon[33655]: cluster 2021-07-31T01:49:43.622834+0000 mon.a (mon.0) 926 : cluster [DBG] 2021-07-31T01:49:45.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:44 smithi045 conmon[33655]: osdmap e284: 8 total, 8 up, 8 in 2021-07-31T01:49:45.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:44 smithi045 conmon[33655]: cluster 2021-07-31T01:49:43.825580+0000 mgr.x (mgr.24542) 108 : cluster [DBG] pgmap v206: 25 pgs: 25 active+clean; 113 MiB data, 779 MiB used, 714 GiB / 715 GiB avail; 9.1 MiB/s rd, 1.3 MiB/s wr, 11 op/s 2021-07-31T01:49:45.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:44 smithi045 conmon[39718]: cluster 2021-07-31T01:49:43.622834+0000 mon.a (mon.0) 926 : cluster 2021-07-31T01:49:45.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:44 smithi045 conmon[39718]: [DBG] osdmap e284: 8 total, 8 up, 8 in 2021-07-31T01:49:45.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:44 smithi045 conmon[39718]: cluster 2021-07-31T01:49:43.825580+0000 mgr.x (mgr.24542) 108 : cluster [DBG] pgmap v206: 25 pgs: 25 active+clean; 113 MiB data, 779 MiB used, 714 GiB / 715 GiB avail; 9.1 MiB/s rd, 1.3 MiB/s wr, 11 op/s 2021-07-31T01:49:45.631 INFO:tasks.rados.rados.0.smithi045.stdout:1236: finishing rollback tid 0 to smithi04569269-36 2021-07-31T01:49:45.631 INFO:tasks.rados.rados.0.smithi045.stdout:1236: finishing rollback tid 1 to smithi04569269-36 2021-07-31T01:49:45.632 INFO:tasks.rados.rados.0.smithi045.stdout:1236: finishing rollback tid 2 to smithi04569269-36 2021-07-31T01:49:45.632 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 36 v 857 (ObjNum 366 snap 108 seq_num 366) dirty exists 2021-07-31T01:49:45.633 INFO:tasks.rados.rados.0.smithi045.stdout:1235: finishing rollback tid 0 to smithi04569269-1 2021-07-31T01:49:45.633 INFO:tasks.rados.rados.0.smithi045.stdout:1235: finishing rollback tid 1 to smithi04569269-1 2021-07-31T01:49:45.633 INFO:tasks.rados.rados.0.smithi045.stdout:1235: finishing rollback tid 2 to smithi04569269-1 2021-07-31T01:49:45.633 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 1 v 336 (ObjNum 349 snap 105 seq_num 349) dirty exists 2021-07-31T01:49:45.634 INFO:tasks.rados.rados.0.smithi045.stdout:1235: done (2 left) 2021-07-31T01:49:45.634 INFO:tasks.rados.rados.0.smithi045.stdout:1236: done (1 left) 2021-07-31T01:49:45.634 INFO:tasks.rados.rados.0.smithi045.stdout:1237: done (0 left) 2021-07-31T01:49:45.634 INFO:tasks.rados.rados.0.smithi045.stdout:1238: snap_remove snap 113 2021-07-31T01:49:46.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:45 smithi154 conmon[34553]: cluster 2021-07-31T01:49:44.627916+0000 mon.a ( 2021-07-31T01:49:46.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:45 smithi154 conmon[34553]: mon.0) 927 : cluster [DBG] osdmap e285: 8 total, 8 up, 8 in 2021-07-31T01:49:46.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:45 smithi045 conmon[33655]: cluster 2021-07-31T01: 2021-07-31T01:49:46.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:45 smithi045 conmon[33655]: 49:44.627916+0000 mon.a (mon.0) 927 : cluster [DBG] osdmap e285: 8 total, 8 up, 8 in 2021-07-31T01:49:46.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:45 smithi045 conmon[39718]: cluster 2021-07-31T01:49:44.627916+0000 mon.a ( 2021-07-31T01:49:46.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:45 smithi045 conmon[39718]: mon.0) 927 : cluster [DBG] osdmap e285: 8 total, 8 up, 8 in 2021-07-31T01:49:46.632 INFO:tasks.rados.rados.0.smithi045.stdout:1238: done (0 left) 2021-07-31T01:49:46.633 INFO:tasks.rados.rados.0.smithi045.stdout:1239: snap_remove snap 112 2021-07-31T01:49:47.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:46 smithi154 conmon[34553]: cluster 2021-07-31T01:49:45.630372+0000 mon.a (mon.0) 928 : cluster [DBG] osdmap e286: 8 total, 8 up, 8 in 2021-07-31T01:49:47.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:46 smithi154 conmon[34553]: cluster 2021-07-31T01:49:45.826173+0000 mgr.x (mgr.24542) 109 : cluster [DBG] pgmap v209: 25 pgs: 25 active+clean; 127 MiB data, 788 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 5.2 MiB/s wr, 22 op/s 2021-07-31T01:49:47.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:46 smithi154 conmon[34553]: cluster 2021-07 2021-07-31T01:49:47.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:46 smithi154 conmon[34553]: -31T01:49:46.631829+0000 mon.a (mon.0) 929 : 2021-07-31T01:49:47.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:46 smithi154 conmon[34553]: cluster [DBG] osdmap e287: 8 total, 8 up, 8 in 2021-07-31T01:49:47.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:46 smithi045 conmon[33655]: cluster 2021-07-31T01:49:45.630372+0000 mon.a (mon.0 2021-07-31T01:49:47.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:46 smithi045 conmon[33655]: ) 928 : cluster [DBG] osdmap e286: 8 total, 8 up, 8 in 2021-07-31T01:49:47.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:46 smithi045 conmon[33655]: cluster 2021-07-31T01:49:45.826173+0000 mgr.x (mgr.24542) 109 : cluster [DBG] pgmap v209: 25 pgs: 25 active+clean; 127 MiB data, 788 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 5.2 MiB/s wr, 22 op/s 2021-07-31T01:49:47.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:46 smithi045 conmon[33655]: cluster 2021-07-31T01:49:46.631829+0000 mon.a (mon 2021-07-31T01:49:47.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:46 smithi045 conmon[33655]: .0) 929 : cluster [DBG] osdmap e287: 8 total, 8 up, 8 in 2021-07-31T01:49:47.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:46 smithi045 conmon[39718]: cluster 2021-07-31T01:49:45.630372 2021-07-31T01:49:47.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:46 smithi045 conmon[39718]: +0000 mon.a (mon.0) 928 : cluster [DBG] osdmap e286: 8 total, 8 up, 8 in 2021-07-31T01:49:47.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:46 smithi045 conmon[39718]: cluster 2021-07-31 2021-07-31T01:49:47.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:46 smithi045 conmon[39718]: T01:49:45.826173+0000 mgr.x (mgr.24542) 2021-07-31T01:49:47.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:46 smithi045 conmon[39718]: 109 : cluster [DBG] pgmap v209: 25 pgs: 25 active+clean; 127 MiB data, 788 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 5.2 MiB/s wr, 22 op/s 2021-07-31T01:49:47.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:46 smithi045 conmon[39718]: cluster 2021-07- 2021-07-31T01:49:47.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:46 smithi045 conmon[39718]: 31T01:49:46.631829+0000 mon.a (mon.0 2021-07-31T01:49:47.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:46 smithi045 conmon[39718]: ) 929 : cluster [DBG] osdmap e287: 8 total, 8 up, 8 in 2021-07-31T01:49:47.634 INFO:tasks.rados.rados.0.smithi045.stdout:1239: done (0 left) 2021-07-31T01:49:47.635 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 31 current snap is 114 2021-07-31T01:49:47.635 INFO:tasks.rados.rados.0.smithi045.stdout:1240: seq_num 385 ranges {2588672=450560,3039232=761856,3801088=442368} 2021-07-31T01:49:47.651 INFO:tasks.rados.rados.0.smithi045.stdout:1240: writing smithi04569269-31 from 2588672 to 3039232 tid 1 2021-07-31T01:49:47.655 INFO:tasks.rados.rados.0.smithi045.stdout:1240: writing smithi04569269-31 from 3039232 to 3801088 tid 2 2021-07-31T01:49:47.658 INFO:tasks.rados.rados.0.smithi045.stdout:1240: writing smithi04569269-31 from 3801088 to 4243456 tid 3 2021-07-31T01:49:47.658 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 40 current snap is 114 2021-07-31T01:49:47.659 INFO:tasks.rados.rados.0.smithi045.stdout:1241: seq_num 386 ranges {7708672=466944,8175616=638976,8814592=565248} 2021-07-31T01:49:47.703 INFO:tasks.rados.rados.0.smithi045.stdout:1241: writing smithi04569269-40 from 7708672 to 8175616 tid 1 2021-07-31T01:49:47.706 INFO:tasks.rados.rados.0.smithi045.stdout:1241: writing smithi04569269-40 from 8175616 to 8814592 tid 2 2021-07-31T01:49:47.711 INFO:tasks.rados.rados.0.smithi045.stdout:1241: writing smithi04569269-40 from 8814592 to 9379840 tid 3 2021-07-31T01:49:47.711 INFO:tasks.rados.rados.0.smithi045.stdout:1242: copy_from oid 19 from oid 18 current snap is 114 2021-07-31T01:49:47.711 INFO:tasks.rados.rados.0.smithi045.stdout:1240: finishing write tid 1 to smithi04569269-31 2021-07-31T01:49:47.711 INFO:tasks.rados.rados.0.smithi045.stdout:1240: finishing write tid 2 to smithi04569269-31 2021-07-31T01:49:47.712 INFO:tasks.rados.rados.0.smithi045.stdout:1240: finishing write tid 3 to smithi04569269-31 2021-07-31T01:49:47.712 INFO:tasks.rados.rados.0.smithi045.stdout:1240: finishing write tid 4 to smithi04569269-31 2021-07-31T01:49:47.712 INFO:tasks.rados.rados.0.smithi045.stdout:1240: finishing write tid 5 to smithi04569269-31 2021-07-31T01:49:47.712 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 31 v 875 (ObjNum 385 snap 114 seq_num 385) dirty exists 2021-07-31T01:49:47.713 INFO:tasks.rados.rados.0.smithi045.stdout:1240: left oid 31 (ObjNum 385 snap 114 seq_num 385) 2021-07-31T01:49:47.713 INFO:tasks.rados.rados.0.smithi045.stdout:1240: done (2 left) 2021-07-31T01:49:47.713 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 38 current snap is 114 2021-07-31T01:49:47.713 INFO:tasks.rados.rados.0.smithi045.stdout:1243: seq_num 387 ranges {1695744=483328,2179072=524288,2703360=679936} 2021-07-31T01:49:47.722 INFO:tasks.rados.rados.0.smithi045.stdout:1243: writing smithi04569269-38 from 1695744 to 2179072 tid 1 2021-07-31T01:49:47.725 INFO:tasks.rados.rados.0.smithi045.stdout:1243: writing smithi04569269-38 from 2179072 to 2703360 tid 2 2021-07-31T01:49:47.730 INFO:tasks.rados.rados.0.smithi045.stdout:1243: writing smithi04569269-38 from 2703360 to 3383296 tid 3 2021-07-31T01:49:47.730 INFO:tasks.rados.rados.0.smithi045.stdout:1244: read oid 45 snap -1 2021-07-31T01:49:47.731 INFO:tasks.rados.rados.0.smithi045.stdout:1244: expect (ObjNum 352 snap 106 seq_num 352) 2021-07-31T01:49:47.731 INFO:tasks.rados.rados.0.smithi045.stdout:1245: read oid 26 snap -1 2021-07-31T01:49:47.731 INFO:tasks.rados.rados.0.smithi045.stdout:1245: expect (ObjNum 381 snap 112 seq_num 381) 2021-07-31T01:49:47.731 INFO:tasks.rados.rados.0.smithi045.stdout:1246: snap_create 2021-07-31T01:49:47.731 INFO:tasks.rados.rados.0.smithi045.stdout:1241: finishing write tid 1 to smithi04569269-40 2021-07-31T01:49:47.732 INFO:tasks.rados.rados.0.smithi045.stdout:1241: finishing write tid 2 to smithi04569269-40 2021-07-31T01:49:47.732 INFO:tasks.rados.rados.0.smithi045.stdout:1241: finishing write tid 3 to smithi04569269-40 2021-07-31T01:49:47.732 INFO:tasks.rados.rados.0.smithi045.stdout:1241: finishing write tid 4 to smithi04569269-40 2021-07-31T01:49:47.732 INFO:tasks.rados.rados.0.smithi045.stdout:1241: finishing write tid 5 to smithi04569269-40 2021-07-31T01:49:47.733 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 40 v 553 (ObjNum 386 snap 114 seq_num 386) dirty exists 2021-07-31T01:49:47.733 INFO:tasks.rados.rados.0.smithi045.stdout:1241: left oid 40 (ObjNum 386 snap 114 seq_num 386) 2021-07-31T01:49:47.733 INFO:tasks.rados.rados.0.smithi045.stdout:1242: finishing copy_from to smithi04569269-19 2021-07-31T01:49:47.733 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 19 v 309 (ObjNum 353 snap 106 seq_num 353) dirty exists 2021-07-31T01:49:47.733 INFO:tasks.rados.rados.0.smithi045.stdout:1242: finishing copy_from racing read to smithi04569269-19 2021-07-31T01:49:47.734 INFO:tasks.rados.rados.0.smithi045.stdout:1243: finishing write tid 1 to smithi04569269-38 2021-07-31T01:49:47.734 INFO:tasks.rados.rados.0.smithi045.stdout:1243: finishing write tid 2 to smithi04569269-38 2021-07-31T01:49:47.735 INFO:tasks.rados.rados.0.smithi045.stdout:1243: finishing write tid 3 to smithi04569269-38 2021-07-31T01:49:47.735 INFO:tasks.rados.rados.0.smithi045.stdout:1243: finishing write tid 4 to smithi04569269-38 2021-07-31T01:49:47.737 INFO:tasks.rados.rados.0.smithi045.stdout:1243: finishing write tid 5 to smithi04569269-38 2021-07-31T01:49:47.737 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 38 v 556 (ObjNum 387 snap 114 seq_num 387) dirty exists 2021-07-31T01:49:47.737 INFO:tasks.rados.rados.0.smithi045.stdout:1243: left oid 38 (ObjNum 387 snap 114 seq_num 387) 2021-07-31T01:49:47.740 INFO:tasks.rados.rados.0.smithi045.stdout:1245: expect (ObjNum 381 snap 112 seq_num 381) 2021-07-31T01:49:47.781 INFO:tasks.rados.rados.0.smithi045.stdout:1244: expect (ObjNum 352 snap 106 seq_num 352) 2021-07-31T01:49:48.640 INFO:tasks.rados.rados.0.smithi045.stdout:1241: done (5 left) 2021-07-31T01:49:48.640 INFO:tasks.rados.rados.0.smithi045.stdout:1242: done (4 left) 2021-07-31T01:49:48.640 INFO:tasks.rados.rados.0.smithi045.stdout:1243: done (3 left) 2021-07-31T01:49:48.640 INFO:tasks.rados.rados.0.smithi045.stdout:1244: done (2 left) 2021-07-31T01:49:48.641 INFO:tasks.rados.rados.0.smithi045.stdout:1245: done (1 left) 2021-07-31T01:49:48.641 INFO:tasks.rados.rados.0.smithi045.stdout:1246: done (0 left) 2021-07-31T01:49:48.641 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 17 current snap is 115 2021-07-31T01:49:48.641 INFO:tasks.rados.rados.0.smithi045.stdout:1247: seq_num 388 ranges {4939776=507904,5447680=458752,5906432=606208,6512640=139264} 2021-07-31T01:49:48.669 INFO:tasks.rados.rados.0.smithi045.stdout:1247: writing smithi04569269-17 from 4939776 to 5447680 tid 1 2021-07-31T01:49:48.671 INFO:tasks.rados.rados.0.smithi045.stdout:1247: writing smithi04569269-17 from 5447680 to 5906432 tid 2 2021-07-31T01:49:48.675 INFO:tasks.rados.rados.0.smithi045.stdout:1247: writing smithi04569269-17 from 5906432 to 6512640 tid 3 2021-07-31T01:49:48.676 INFO:tasks.rados.rados.0.smithi045.stdout:1247: writing smithi04569269-17 from 6512640 to 6651904 tid 4 2021-07-31T01:49:48.676 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 35 current snap is 115 2021-07-31T01:49:48.676 INFO:tasks.rados.rados.0.smithi045.stdout:1247: finishing write tid 1 to smithi04569269-17 2021-07-31T01:49:48.677 INFO:tasks.rados.rados.0.smithi045.stdout:1248: seq_num 389 ranges {5332992=524288,5857280=745472,6602752=458752} 2021-07-31T01:49:48.708 INFO:tasks.rados.rados.0.smithi045.stdout:1248: writing smithi04569269-35 from 5332992 to 5857280 tid 1 2021-07-31T01:49:48.712 INFO:tasks.rados.rados.0.smithi045.stdout:1248: writing smithi04569269-35 from 5857280 to 6602752 tid 2 2021-07-31T01:49:48.715 INFO:tasks.rados.rados.0.smithi045.stdout:1248: writing smithi04569269-35 from 6602752 to 7061504 tid 3 2021-07-31T01:49:48.715 INFO:tasks.rados.rados.0.smithi045.stdout:1249: snap_create 2021-07-31T01:49:48.715 INFO:tasks.rados.rados.0.smithi045.stdout:1247: finishing write tid 2 to smithi04569269-17 2021-07-31T01:49:48.716 INFO:tasks.rados.rados.0.smithi045.stdout:1247: finishing write tid 3 to smithi04569269-17 2021-07-31T01:49:48.716 INFO:tasks.rados.rados.0.smithi045.stdout:1247: finishing write tid 4 to smithi04569269-17 2021-07-31T01:49:48.716 INFO:tasks.rados.rados.0.smithi045.stdout:1247: finishing write tid 5 to smithi04569269-17 2021-07-31T01:49:48.716 INFO:tasks.rados.rados.0.smithi045.stdout:1247: finishing write tid 6 to smithi04569269-17 2021-07-31T01:49:48.717 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 17 v 514 (ObjNum 388 snap 115 seq_num 388) dirty exists 2021-07-31T01:49:48.717 INFO:tasks.rados.rados.0.smithi045.stdout:1247: left oid 17 (ObjNum 388 snap 115 seq_num 388) 2021-07-31T01:49:48.717 INFO:tasks.rados.rados.0.smithi045.stdout:1248: finishing write tid 1 to smithi04569269-35 2021-07-31T01:49:48.718 INFO:tasks.rados.rados.0.smithi045.stdout:1248: finishing write tid 2 to smithi04569269-35 2021-07-31T01:49:48.718 INFO:tasks.rados.rados.0.smithi045.stdout:1248: finishing write tid 3 to smithi04569269-35 2021-07-31T01:49:48.718 INFO:tasks.rados.rados.0.smithi045.stdout:1248: finishing write tid 4 to smithi04569269-35 2021-07-31T01:49:48.719 INFO:tasks.rados.rados.0.smithi045.stdout:1248: finishing write tid 5 to smithi04569269-35 2021-07-31T01:49:48.719 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 35 v 882 (ObjNum 389 snap 115 seq_num 389) dirty exists 2021-07-31T01:49:48.720 INFO:tasks.rados.rados.0.smithi045.stdout:1248: left oid 35 (ObjNum 389 snap 115 seq_num 389) 2021-07-31T01:49:49.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:48 smithi154 conmon[34553]: cluster 2021-07-31T01:49:49.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:48 smithi154 conmon[34553]: 2021-07-31T01:49:47.633959 2021-07-31T01:49:49.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:48 smithi154 conmon[34553]: +0000 mon.a (mon.0) 930 : cluster [DBG] osdmap e288: 8 total, 8 up, 8 in 2021-07-31T01:49:49.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:48 smithi154 conmon[34553]: cluster 2021- 2021-07-31T01:49:49.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:48 smithi154 conmon[34553]: 07-31T01:49:47.826691+0000 mgr.x (mgr. 2021-07-31T01:49:49.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:48 smithi154 conmon[34553]: 24542) 110 : cluster [DBG] pgmap v212: 25 pgs: 25 active+clean; 127 MiB data, 788 MiB used, 714 GiB / 715 GiB avail; 2.7 MiB/s rd, 3.9 MiB/s wr, 10 op/s 2021-07-31T01:49:49.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:48 smithi045 conmon[33655]: cluster 2021-07-31T01:49:47.633959+0000 mon.a (mon.0) 930 : cluster [DBG] osdmap e288: 8 total, 8 up, 8 in 2021-07-31T01:49:49.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:48 smithi045 conmon[33655]: cluster 2021-07-31T01:49:47.826691+0000 mgr.x (mgr.24542) 110 : cluster [DBG] pgmap v212: 25 pgs: 25 active+clean; 127 MiB data, 788 MiB used, 714 GiB / 715 GiB avail; 2.7 MiB/s rd, 3.9 MiB/s wr, 10 op/s 2021-07-31T01:49:49.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:48 smithi045 conmon[39718]: cluster 2021-07-31T01:49:47.633959+0000 mon.a ( 2021-07-31T01:49:49.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:48 smithi045 conmon[39718]: mon.0) 930 : cluster [DBG] osdmap e288: 8 total, 8 up, 8 in 2021-07-31T01:49:49.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:48 smithi045 conmon[39718]: cluster 2021-07-31T01:49:47.826691+0000 mgr.x (mgr.24542) 110 : cluster [DBG] pgmap v212: 25 pgs: 25 active+clean; 127 MiB data, 788 MiB used, 714 GiB / 715 GiB avail; 2.7 MiB/s rd, 3.9 MiB/s wr, 10 op/s 2021-07-31T01:49:49.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:48 smithi045 conmon[39718]: 2021-07-31T01:49:49.644 INFO:tasks.rados.rados.0.smithi045.stdout:1247: done (2 left) 2021-07-31T01:49:49.645 INFO:tasks.rados.rados.0.smithi045.stdout:1248: done (1 left) 2021-07-31T01:49:49.645 INFO:tasks.rados.rados.0.smithi045.stdout:1249: done (0 left) 2021-07-31T01:49:49.645 INFO:tasks.rados.rados.0.smithi045.stdout:1250: delete oid 44 current snap is 116 2021-07-31T01:49:49.649 INFO:tasks.rados.rados.0.smithi045.stdout:1250: done (0 left) 2021-07-31T01:49:49.650 INFO:tasks.rados.rados.0.smithi045.stdout:1251: rmattr oid 21 current snap is 116 2021-07-31T01:49:49.650 INFO:tasks.rados.rados.0.smithi045.stdout:1251: done (0 left) 2021-07-31T01:49:49.651 INFO:tasks.rados.rados.0.smithi045.stdout:1252: read oid 50 snap -1 2021-07-31T01:49:49.651 INFO:tasks.rados.rados.0.smithi045.stdout:1252: expect deleted 2021-07-31T01:49:49.651 INFO:tasks.rados.rados.0.smithi045.stdout:1253: read oid 4 snap -1 2021-07-31T01:49:49.651 INFO:tasks.rados.rados.0.smithi045.stdout:1253: expect (ObjNum 375 snap 109 seq_num 375) 2021-07-31T01:49:49.652 INFO:tasks.rados.rados.0.smithi045.stdout:1254: snap_create 2021-07-31T01:49:49.655 INFO:tasks.rados.rados.0.smithi045.stdout:1253: expect (ObjNum 375 snap 109 seq_num 375) 2021-07-31T01:49:50.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:49 smithi154 conmon[34553]: cluster 2021-07-31T01:49:48.638869+0000 mon.a (mon.0) 931 : cluster [DBG] osdmap e289: 8 total, 8 up, 8 in 2021-07-31T01:49:50.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:49 smithi045 conmon[33655]: cluster 2021-07-31T01:49:48.638869+0000 mon.a (mon.0) 931 : 2021-07-31T01:49:50.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:49 smithi045 conmon[33655]: cluster [DBG] osdmap e289: 8 total, 8 up, 8 in 2021-07-31T01:49:50.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:49 smithi045 conmon[39718]: cluster 2021-07-31T01:49: 2021-07-31T01:49:50.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:49 smithi045 conmon[39718]: 48.638869+0000 mon.a (mon.0) 931 : cluster [DBG] osdmap e289: 8 total, 8 up, 8 in 2021-07-31T01:49:50.651 INFO:tasks.rados.rados.0.smithi045.stdout:1252: done (2 left) 2021-07-31T01:49:50.652 INFO:tasks.rados.rados.0.smithi045.stdout:1253: done (1 left) 2021-07-31T01:49:50.652 INFO:tasks.rados.rados.0.smithi045.stdout:1254: done (0 left) 2021-07-31T01:49:50.652 INFO:tasks.rados.rados.0.smithi045.stdout:1255: read oid 40 snap 116 2021-07-31T01:49:50.652 INFO:tasks.rados.rados.0.smithi045.stdout:1255: expect (ObjNum 386 snap 114 seq_num 386) 2021-07-31T01:49:50.653 INFO:tasks.rados.rados.0.smithi045.stdout:1256: copy_from oid 41 from oid 10 current snap is 117 2021-07-31T01:49:50.653 INFO:tasks.rados.rados.0.smithi045.stdout:1257: read oid 37 snap -1 2021-07-31T01:49:50.653 INFO:tasks.rados.rados.0.smithi045.stdout:1257: expect (ObjNum 172 snap 41 seq_num 172) 2021-07-31T01:49:50.653 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 8 current snap is 117 2021-07-31T01:49:50.653 INFO:tasks.rados.rados.0.smithi045.stdout:1258: seq_num 391 ranges {4186112=557056,4743168=516096,5259264=688128} 2021-07-31T01:49:50.679 INFO:tasks.rados.rados.0.smithi045.stdout:1258: writing smithi04569269-8 from 4186112 to 4743168 tid 1 2021-07-31T01:49:50.682 INFO:tasks.rados.rados.0.smithi045.stdout:1258: writing smithi04569269-8 from 4743168 to 5259264 tid 2 2021-07-31T01:49:50.688 INFO:tasks.rados.rados.0.smithi045.stdout:1258: writing smithi04569269-8 from 5259264 to 5947392 tid 3 2021-07-31T01:49:50.689 INFO:tasks.rados.rados.0.smithi045.stdout:1259: rmattr oid 49 current snap is 117 2021-07-31T01:49:50.689 INFO:tasks.rados.rados.0.smithi045.stdout:1260: delete oid 21 current snap is 117 2021-07-31T01:49:50.689 INFO:tasks.rados.rados.0.smithi045.stdout:1257: expect (ObjNum 172 snap 41 seq_num 172) 2021-07-31T01:49:50.707 INFO:tasks.rados.rados.0.smithi045.stdout:1258: finishing write tid 1 to smithi04569269-8 2021-07-31T01:49:50.708 INFO:tasks.rados.rados.0.smithi045.stdout:1258: finishing write tid 2 to smithi04569269-8 2021-07-31T01:49:50.708 INFO:tasks.rados.rados.0.smithi045.stdout:1258: finishing write tid 3 to smithi04569269-8 2021-07-31T01:49:50.708 INFO:tasks.rados.rados.0.smithi045.stdout:1258: finishing write tid 4 to smithi04569269-8 2021-07-31T01:49:50.709 INFO:tasks.rados.rados.0.smithi045.stdout:1258: finishing write tid 5 to smithi04569269-8 2021-07-31T01:49:50.709 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 8 v 887 (ObjNum 391 snap 117 seq_num 391) dirty exists 2021-07-31T01:49:50.710 INFO:tasks.rados.rados.0.smithi045.stdout:1258: left oid 8 (ObjNum 391 snap 117 seq_num 391) 2021-07-31T01:49:50.710 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 49 v 889 (ObjNum 285 snap 83 seq_num 285) dirty exists 2021-07-31T01:49:50.710 INFO:tasks.rados.rados.0.smithi045.stdout:1256: finishing copy_from to smithi04569269-41 2021-07-31T01:49:50.710 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 41 v 494 (ObjNum 336 snap 99 seq_num 336) dirty exists 2021-07-31T01:49:50.711 INFO:tasks.rados.rados.0.smithi045.stdout:1256: finishing copy_from racing read to smithi04569269-41 2021-07-31T01:49:50.711 INFO:tasks.rados.rados.0.smithi045.stdout:1255: expect (ObjNum 386 snap 114 seq_num 386) 2021-07-31T01:49:51.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:50 smithi154 conmon[34553]: cluster 2021-07-31T01:49:49.644200+0000 mon.a (mon.0) 932 : cluster [DBG] 2021-07-31T01:49:51.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:50 smithi154 conmon[34553]: osdmap e290: 8 total, 8 up, 8 in 2021-07-31T01:49:51.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:50 smithi154 conmon[34553]: cluster 2021-07-31T01:49:49.827206+0000 mgr.x (mgr.24542) 111 : cluster [DBG] pgmap v215: 25 pgs: 25 active+clean; 128 MiB data, 807 MiB used, 714 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.4 MiB/s wr, 9 op/s 2021-07-31T01:49:51.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:50 smithi045 conmon[39718]: cluster 2021-07-31T01:49:49.644200+0000 mon.a (mon.0) 932 2021-07-31T01:49:51.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:50 smithi045 conmon[39718]: : cluster [DBG] osdmap e290: 8 total, 8 up, 8 in 2021-07-31T01:49:51.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:50 smithi045 conmon[39718]: cluster 2021-07-31T01:49:49.827206+0000 mgr.x (mgr.24542) 111 : cluster [DBG] pgmap v215: 25 pgs: 25 active+clean; 128 MiB data, 807 MiB used, 714 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.4 MiB/s wr, 9 op/s 2021-07-31T01:49:51.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:50 smithi045 conmon[33655]: cluster 2021-07-31T01:49:49.644200+0000 mon.a (mon.0) 932 : cluster [DBG] 2021-07-31T01:49:51.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:50 smithi045 conmon[33655]: osdmap e290: 8 total, 8 up, 8 in 2021-07-31T01:49:51.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:50 smithi045 conmon[33655]: cluster 2021-07-31T01:49:49.827206+0000 mgr.x (mgr.24542) 111 : cluster [DBG] pgmap v215: 25 pgs: 25 active+clean; 128 MiB data, 807 MiB used, 714 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.4 MiB/s wr, 9 op/s 2021-07-31T01:49:51.423 INFO:tasks.rados.rados.0.smithi045.stdout:1255: done (5 left) 2021-07-31T01:49:51.424 INFO:tasks.rados.rados.0.smithi045.stdout:1256: done (4 left) 2021-07-31T01:49:51.424 INFO:tasks.rados.rados.0.smithi045.stdout:1257: done (3 left) 2021-07-31T01:49:51.424 INFO:tasks.rados.rados.0.smithi045.stdout:1258: done (2 left) 2021-07-31T01:49:51.424 INFO:tasks.rados.rados.0.smithi045.stdout:1259: done (1 left) 2021-07-31T01:49:51.425 INFO:tasks.rados.rados.0.smithi045.stdout:1260: done (0 left) 2021-07-31T01:49:51.425 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 48 current snap is 117 2021-07-31T01:49:51.425 INFO:tasks.rados.rados.0.smithi045.stdout:1261: seq_num 393 ranges {3883008=589824,4472832=679936,5152768=524288} 2021-07-31T01:49:51.446 INFO:tasks.rados.rados.0.smithi045.stdout:1261: writing smithi04569269-48 from 3883008 to 4472832 tid 1 2021-07-31T01:49:51.451 INFO:tasks.rados.rados.0.smithi045.stdout:1261: writing smithi04569269-48 from 4472832 to 5152768 tid 2 2021-07-31T01:49:51.454 INFO:tasks.rados.rados.0.smithi045.stdout:1261: writing smithi04569269-48 from 5152768 to 5677056 tid 3 2021-07-31T01:49:51.454 INFO:tasks.rados.rados.0.smithi045.stdout:1262: delete oid 20 current snap is 117 2021-07-31T01:49:51.454 INFO:tasks.rados.rados.0.smithi045.stdout:1261: finishing write tid 1 to smithi04569269-48 2021-07-31T01:49:51.455 INFO:tasks.rados.rados.0.smithi045.stdout:1261: finishing write tid 2 to smithi04569269-48 2021-07-31T01:49:51.457 INFO:tasks.rados.rados.0.smithi045.stdout:1262: done (1 left) 2021-07-31T01:49:51.457 INFO:tasks.rados.rados.0.smithi045.stdout:1263: setattr oid 17 current snap is 117 2021-07-31T01:49:51.457 INFO:tasks.rados.rados.0.smithi045.stdout:1261: finishing write tid 3 to smithi04569269-48 2021-07-31T01:49:51.457 INFO:tasks.rados.rados.0.smithi045.stdout:1261: finishing write tid 4 to smithi04569269-48 2021-07-31T01:49:51.459 INFO:tasks.rados.rados.0.smithi045.stdout:1261: finishing write tid 5 to smithi04569269-48 2021-07-31T01:49:51.459 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 48 v 340 (ObjNum 393 snap 117 seq_num 393) dirty exists 2021-07-31T01:49:51.459 INFO:tasks.rados.rados.0.smithi045.stdout:1261: left oid 48 (ObjNum 393 snap 117 seq_num 393) 2021-07-31T01:49:51.459 INFO:tasks.rados.rados.0.smithi045.stdout:1261: done (1 left) 2021-07-31T01:49:51.460 INFO:tasks.rados.rados.0.smithi045.stdout:1264: read oid 1 snap -1 2021-07-31T01:49:51.460 INFO:tasks.rados.rados.0.smithi045.stdout:1264: expect (ObjNum 349 snap 105 seq_num 349) 2021-07-31T01:49:51.460 INFO:tasks.rados.rados.0.smithi045.stdout:1265: snap_remove snap 115 2021-07-31T01:49:51.528 INFO:tasks.rados.rados.0.smithi045.stdout:1265: done (2 left) 2021-07-31T01:49:51.529 INFO:tasks.rados.rados.0.smithi045.stdout:1266: snap_remove snap 114 2021-07-31T01:49:52.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:51 smithi154 conmon[34553]: cluster 2021-07-31T01:49:50.649607+0000 mon.a (mon.0) 933 : cluster [DBG] osdmap e291: 8 total, 8 up, 8 in 2021-07-31T01:49:52.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:51 smithi154 conmon[34553]: cluster 2021-07-31T01:49:51.527149+0000 mon.a (mon.0) 934 : cluster [DBG] osdmap e292: 8 total, 8 up, 8 in 2021-07-31T01:49:52.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:51 smithi045 conmon[39718]: cluster 2021-07-31T01:49:50.649607+0000 2021-07-31T01:49:52.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:51 smithi045 conmon[39718]: mon.a (mon.0) 933 : cluster [DBG] osdmap e291: 8 total, 8 up, 8 in 2021-07-31T01:49:52.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:51 smithi045 conmon[39718]: cluster 2021-07-31T01:49:51 2021-07-31T01:49:52.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:51 smithi045 conmon[39718]: .527149+0000 mon.a (mon.0) 934 : cluster [DBG] osdmap e292: 8 total, 8 up, 8 in 2021-07-31T01:49:52.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:51 smithi045 conmon[33655]: cluster 2021-07-31T01:49:50.649607+0000 mon.a ( 2021-07-31T01:49:52.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:51 smithi045 conmon[33655]: mon.0) 933 : cluster [DBG] osdmap e291: 8 total, 8 up, 8 in 2021-07-31T01:49:52.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:51 smithi045 conmon[33655]: cluster 2021-07-31T01:49:51.527149+0000 mon.a (mon.0) 934 : cluster [DBG] osdmap e292: 8 total, 8 up, 8 in 2021-07-31T01:49:52.538 INFO:tasks.rados.rados.0.smithi045.stdout:1266: done (2 left) 2021-07-31T01:49:52.538 INFO:tasks.rados.rados.0.smithi045.stdout:1267: snap_create 2021-07-31T01:49:52.538 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 17 v 516 (ObjNum 388 snap 115 seq_num 388) dirty exists 2021-07-31T01:49:52.538 INFO:tasks.rados.rados.0.smithi045.stdout:1264: expect (ObjNum 349 snap 105 seq_num 349) 2021-07-31T01:49:53.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:52 smithi154 conmon[34553]: cluster 2021-07-31T01:49:51.827764+0000 mgr.x (mgr.24542) 112 : cluster [DBG] pgmap v218: 25 pgs: 25 active+clean; 128 MiB data, 807 MiB used, 714 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.4 MiB/s wr, 9 op/s 2021-07-31T01:49:53.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:52 smithi154 conmon[34553]: cluster 2021-07-31T01:49:52.536417 2021-07-31T01:49:53.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:52 smithi154 conmon[34553]: +0000 mon.a (mon.0) 935 : cluster [DBG] osdmap e293: 8 total, 8 up, 8 in 2021-07-31T01:49:53.004 INFO:journalctl@ceph.osd.6.smithi154.stdout:Jul 31 01:49:52 smithi154 conmon[45576]: debug 2021-07-31T01:49:52.539+0000 7f47c50ad700 -1 osd.6 pg_epoch: 293 pg[3.as0( v 292'383 (0'0,292'383] local-lis/les=56/57 n=5 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [6,4,1]p6(0) r=0 lpr=56 crt=292'383 lcod 290'381 mlcod 290'381 active+clean+snaptrim trimq=[e6~1] ps=[e7~1]] removing snap head 2021-07-31T01:49:53.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:52 smithi045 conmon[33655]: cluster 2021-07-31T01:49:51.827764+0000 mgr.x (mgr.24542) 112 : cluster [DBG] pgmap v218: 25 pgs: 25 active+clean; 128 MiB data, 807 MiB used, 714 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.4 MiB/s wr, 9 op/s 2021-07-31T01:49:53.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:52 smithi045 conmon[33655]: cluster 2021-07-31T01:49:52.536417+0000 mon.a (mon.0) 935 : cluster [DBG] osdmap e293: 8 total, 8 up, 8 in 2021-07-31T01:49:53.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:52 smithi045 conmon[39718]: cluster 2021-07-31T01:49:51.827764+0000 mgr.x (mgr 2021-07-31T01:49:53.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:52 smithi045 conmon[39718]: .24542) 112 : cluster [DBG] pgmap v218: 25 pgs: 25 active+clean; 128 MiB data, 807 MiB used, 714 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.4 MiB/s wr, 9 op/s 2021-07-31T01:49:53.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:52 smithi045 conmon[39718]: cluster 2021-07-31T01:49:52.536417+0000 mon.a (mon.0) 2021-07-31T01:49:53.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:52 smithi045 conmon[39718]: 935 : cluster [DBG] osdmap e293: 8 total, 8 up, 8 in 2021-07-31T01:49:53.538 INFO:tasks.rados.rados.0.smithi045.stdout:1263: done (2 left) 2021-07-31T01:49:53.538 INFO:tasks.rados.rados.0.smithi045.stdout:1264: done (1 left) 2021-07-31T01:49:53.539 INFO:tasks.rados.rados.0.smithi045.stdout:1267: done (0 left) 2021-07-31T01:49:53.539 INFO:tasks.rados.rados.0.smithi045.stdout:1268: rollback oid 14 current snap is 118 2021-07-31T01:49:53.539 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 14 to 116 2021-07-31T01:49:53.539 INFO:tasks.rados.rados.0.smithi045.stdout:1269: snap_remove snap 117 2021-07-31T01:49:53.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:49:53 smithi045 conmon[61210]: level=error ts=2021-07-31T01:49:53.513Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:49:53.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:49:53 smithi045 conmon[61210]: level=error ts=2021-07-31T01:49:53.513Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:49:53.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:49:53 smithi045 conmon[61210]: level=error ts=2021-07-31T01:49:53.513Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T01:49:54.544 INFO:tasks.rados.rados.0.smithi045.stdout:1269: done (1 left) 2021-07-31T01:49:54.545 INFO:tasks.rados.rados.0.smithi045.stdout:1270: snap_create 2021-07-31T01:49:54.545 INFO:tasks.rados.rados.0.smithi045.stdout:1268: finishing rollback tid 0 to smithi04569269-14 2021-07-31T01:49:54.546 INFO:tasks.rados.rados.0.smithi045.stdout:1268: finishing rollback tid 1 to smithi04569269-14 2021-07-31T01:49:54.546 INFO:tasks.rados.rados.0.smithi045.stdout:1268: finishing rollback tid 2 to smithi04569269-14 2021-07-31T01:49:54.547 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 14 v 389 (ObjNum 275 snap 81 seq_num 275) dirty exists 2021-07-31T01:49:55.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:54 smithi154 conmon[34553]: cluster 2021-07-31T01:49:53.537833+0000 mon.a (mon.0) 2021-07-31T01:49:55.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:54 smithi154 conmon[34553]: 936 : cluster [DBG] osdmap e294: 8 total, 8 up, 8 in 2021-07-31T01:49:55.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:54 smithi154 conmon[34553]: cluster 2021-07-31T01:49:53.828350+0000 mgr.x (mgr.24542) 113 : cluster [DBG] pgmap v221: 25 pgs: 25 active+clean; 128 MiB data, 816 MiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 1.6 MiB/s wr, 11 op/s 2021-07-31T01:49:55.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:54 smithi045 conmon[33655]: cluster 2021-07-31T01:49:53.537833+0000 2021-07-31T01:49:55.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:54 smithi045 conmon[33655]: mon.a (mon.0) 936 : cluster [DBG] osdmap e294: 8 total, 8 up, 8 in 2021-07-31T01:49:55.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:54 smithi045 conmon[33655]: cluster 2021-07-31T01:49:53.828350+0000 mgr.x (mgr.24542) 113 : cluster [DBG] pgmap v221: 25 pgs: 25 active+clean; 128 MiB data, 816 MiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 1.6 MiB/s wr, 11 op/s 2021-07-31T01:49:55.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:54 smithi045 conmon[39718]: cluster 2021-07-31T01:49:53.537833+0000 mon.a (mon.0 2021-07-31T01:49:55.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:54 smithi045 conmon[39718]: ) 936 : cluster [DBG] osdmap e294: 8 total, 8 up, 8 in 2021-07-31T01:49:55.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:54 smithi045 conmon[39718]: cluster 2021-07-31T01:49:53.828350+0000 mgr.x (mgr.24542 2021-07-31T01:49:55.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:54 smithi045 conmon[39718]: ) 113 : cluster [DBG] pgmap v221: 25 pgs: 25 active+clean; 128 MiB data, 816 MiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 1.6 MiB/s wr, 11 op/s 2021-07-31T01:49:55.550 INFO:tasks.rados.rados.0.smithi045.stdout:1268: done (1 left) 2021-07-31T01:49:55.550 INFO:tasks.rados.rados.0.smithi045.stdout:1270: done (0 left) 2021-07-31T01:49:55.550 INFO:tasks.rados.rados.0.smithi045.stdout:1271: rollback oid 16 current snap is 119 2021-07-31T01:49:55.551 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 16 to 118 2021-07-31T01:49:55.551 INFO:tasks.rados.rados.0.smithi045.stdout:1272: read oid 31 snap -1 2021-07-31T01:49:55.551 INFO:tasks.rados.rados.0.smithi045.stdout:1272: expect (ObjNum 385 snap 114 seq_num 385) 2021-07-31T01:49:55.551 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 13 current snap is 119 2021-07-31T01:49:55.552 INFO:tasks.rados.rados.0.smithi045.stdout:1273: seq_num 395 ranges {565248=622592,1187840=450560,1638400=753664} 2021-07-31T01:49:55.556 INFO:tasks.rados.rados.0.smithi045.stdout:1273: writing smithi04569269-13 from 565248 to 1187840 tid 1 2021-07-31T01:49:55.559 INFO:tasks.rados.rados.0.smithi045.stdout:1273: writing smithi04569269-13 from 1187840 to 1638400 tid 2 2021-07-31T01:49:55.564 INFO:tasks.rados.rados.0.smithi045.stdout:1273: writing smithi04569269-13 from 1638400 to 2392064 tid 3 2021-07-31T01:49:55.564 INFO:tasks.rados.rados.0.smithi045.stdout:1274: delete oid 10 current snap is 119 2021-07-31T01:49:55.565 INFO:tasks.rados.rados.0.smithi045.stdout:1271: finishing rollback tid 0 to smithi04569269-16 2021-07-31T01:49:55.565 INFO:tasks.rados.rados.0.smithi045.stdout:1271: finishing rollback tid 1 to smithi04569269-16 2021-07-31T01:49:55.565 INFO:tasks.rados.rados.0.smithi045.stdout:1271: finishing rollback tid 2 to smithi04569269-16 2021-07-31T01:49:55.565 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 16 v 312 (ObjNum 207 snap 53 seq_num 207) dirty exists 2021-07-31T01:49:55.565 INFO:tasks.rados.rados.0.smithi045.stdout:1273: finishing write tid 1 to smithi04569269-13 2021-07-31T01:49:55.566 INFO:tasks.rados.rados.0.smithi045.stdout:1273: finishing write tid 2 to smithi04569269-13 2021-07-31T01:49:55.566 INFO:tasks.rados.rados.0.smithi045.stdout:1272: expect (ObjNum 385 snap 114 seq_num 385) 2021-07-31T01:49:55.742 INFO:tasks.rados.rados.0.smithi045.stdout:1273: finishing write tid 3 to smithi04569269-13 2021-07-31T01:49:55.743 INFO:tasks.rados.rados.0.smithi045.stdout:1273: finishing write tid 4 to smithi04569269-13 2021-07-31T01:49:55.743 INFO:tasks.rados.rados.0.smithi045.stdout:1273: finishing write tid 5 to smithi04569269-13 2021-07-31T01:49:55.744 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 13 v 557 (ObjNum 395 snap 119 seq_num 395) dirty exists 2021-07-31T01:49:55.744 INFO:tasks.rados.rados.0.smithi045.stdout:1273: left oid 13 (ObjNum 395 snap 119 seq_num 395) 2021-07-31T01:49:55.744 INFO:tasks.rados.rados.0.smithi045.stdout:1271: done (3 left) 2021-07-31T01:49:55.744 INFO:tasks.rados.rados.0.smithi045.stdout:1272: done (2 left) 2021-07-31T01:49:55.745 INFO:tasks.rados.rados.0.smithi045.stdout:1273: done (1 left) 2021-07-31T01:49:55.745 INFO:tasks.rados.rados.0.smithi045.stdout:1274: done (0 left) 2021-07-31T01:49:55.745 INFO:tasks.rados.rados.0.smithi045.stdout:1275: read oid 41 snap -1 2021-07-31T01:49:55.745 INFO:tasks.rados.rados.0.smithi045.stdout:1275: expect (ObjNum 336 snap 99 seq_num 336) 2021-07-31T01:49:55.745 INFO:tasks.rados.rados.0.smithi045.stdout:1276: setattr oid 8 current snap is 119 2021-07-31T01:49:55.746 INFO:tasks.rados.rados.0.smithi045.stdout:1277: setattr oid 47 current snap is 119 2021-07-31T01:49:55.747 INFO:tasks.rados.rados.0.smithi045.stdout:1278: read oid 15 snap -1 2021-07-31T01:49:55.747 INFO:tasks.rados.rados.0.smithi045.stdout:1278: expect deleted 2021-07-31T01:49:55.747 INFO:tasks.rados.rados.0.smithi045.stdout:1279: copy_from oid 1 from oid 49 current snap is 119 2021-07-31T01:49:55.747 INFO:tasks.rados.rados.0.smithi045.stdout:1280: snap_create 2021-07-31T01:49:55.749 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 47 v 266 (ObjNum 172 snap 41 seq_num 172) dirty exists 2021-07-31T01:49:55.751 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 8 v 913 (ObjNum 391 snap 117 seq_num 391) dirty exists 2021-07-31T01:49:55.769 INFO:tasks.rados.rados.0.smithi045.stdout:1279: finishing copy_from to smithi04569269-1 2021-07-31T01:49:55.769 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 1 v 889 (ObjNum 285 snap 83 seq_num 285) dirty exists 2021-07-31T01:49:55.770 INFO:tasks.rados.rados.0.smithi045.stdout:1279: finishing copy_from racing read to smithi04569269-1 2021-07-31T01:49:55.780 INFO:tasks.rados.rados.0.smithi045.stdout:1275: expect (ObjNum 336 snap 99 seq_num 336) 2021-07-31T01:49:56.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:55 smithi154 conmon[34553]: cluster 2021-07-31T01:49:54.543116+0000 mon.a (mon.0) 937 : cluster [DBG] osdmap e295: 8 total, 8 up, 8 in 2021-07-31T01:49:56.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:55 smithi045 conmon[33655]: cluster 2021-07-31T01:49:54.543116+0000 mon.a (mon.0 2021-07-31T01:49:56.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:55 smithi045 conmon[33655]: ) 937 : cluster [DBG] osdmap e295: 8 total, 8 up, 8 in 2021-07-31T01:49:56.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:55 smithi045 conmon[39718]: cluster 2021-07-31T01:49:54.543116+0000 mon.a (mon.0) 2021-07-31T01:49:56.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:55 smithi045 conmon[39718]: 937 : cluster [DBG] osdmap e295: 8 total, 8 up, 8 in 2021-07-31T01:49:56.554 INFO:tasks.rados.rados.0.smithi045.stdout:1275: done (5 left) 2021-07-31T01:49:56.554 INFO:tasks.rados.rados.0.smithi045.stdout:1276: done (4 left) 2021-07-31T01:49:56.554 INFO:tasks.rados.rados.0.smithi045.stdout:1277: done (3 left) 2021-07-31T01:49:56.555 INFO:tasks.rados.rados.0.smithi045.stdout:1278: done (2 left) 2021-07-31T01:49:56.555 INFO:tasks.rados.rados.0.smithi045.stdout:1279: done (1 left) 2021-07-31T01:49:56.555 INFO:tasks.rados.rados.0.smithi045.stdout:1280: done (0 left) 2021-07-31T01:49:56.555 INFO:tasks.rados.rados.0.smithi045.stdout:1281: setattr oid 12 current snap is 120 2021-07-31T01:49:56.556 INFO:tasks.rados.rados.0.smithi045.stdout:1282: delete oid 11 current snap is 120 2021-07-31T01:49:56.559 INFO:tasks.rados.rados.0.smithi045.stdout:1282: done (1 left) 2021-07-31T01:49:56.559 INFO:tasks.rados.rados.0.smithi045.stdout:1283: snap_remove snap 118 2021-07-31T01:49:57.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:56 smithi154 conmon[34553]: cluster 2021-07-31T01:49:55.548488+0000 mon.a (mon.0) 938 : cluster [DBG] 2021-07-31T01:49:57.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:56 smithi154 conmon[34553]: osdmap e296: 8 total, 8 up, 8 in 2021-07-31T01:49:57.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:56 smithi154 conmon[34553]: cluster 2021-07-31T01:49:55.828828+0000 mgr.x (mgr.24542) 114 : cluster [DBG] pgmap v224: 25 pgs: 25 active+clean; 130 MiB data, 823 MiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 2.0 MiB/s wr, 14 op/s 2021-07-31T01:49:57.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:56 smithi045 conmon[39718]: cluster 2021-07-31T01:49:55.548488+0000 2021-07-31T01:49:57.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:56 smithi045 conmon[39718]: mon.a (mon.0) 938 : cluster [DBG] osdmap e296: 8 total, 8 up, 8 in 2021-07-31T01:49:57.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:56 smithi045 conmon[39718]: cluster 2021-07-31T01: 2021-07-31T01:49:57.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:56 smithi045 conmon[39718]: 49:55.828828+0000 mgr.x (mgr.24542) 114 : cluster [DBG] pgmap v224: 25 pgs: 25 active+clean; 130 MiB data, 823 MiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 2.0 MiB/s wr, 14 op/s 2021-07-31T01:49:57.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:56 smithi045 conmon[33655]: cluster 2021-07-31T01:49:55.548488+0000 mon.a (mon.0 2021-07-31T01:49:57.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:56 smithi045 conmon[33655]: ) 938 : cluster [DBG] osdmap e296: 8 total, 8 up, 8 in 2021-07-31T01:49:57.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:56 smithi045 conmon[33655]: cluster 2021-07-31T01:49:55.828828+0000 2021-07-31T01:49:57.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:56 smithi045 conmon[33655]: mgr.x (mgr.24542) 114 : cluster [DBG] pgmap v224: 25 pgs: 25 active+clean; 130 MiB data, 823 MiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 2.0 MiB/s wr, 14 op/s 2021-07-31T01:49:57.557 INFO:tasks.rados.rados.0.smithi045.stdout:1283: done (1 left) 2021-07-31T01:49:57.557 INFO:tasks.rados.rados.0.smithi045.stdout:1284: rmattr oid 47 current snap is 120 2021-07-31T01:49:57.558 INFO:tasks.rados.rados.0.smithi045.stdout:1285: read oid 5 snap 116 2021-07-31T01:49:57.558 INFO:tasks.rados.rados.0.smithi045.stdout:1285: expect deleted 2021-07-31T01:49:57.558 INFO:tasks.rados.rados.0.smithi045.stdout:1286: read oid 35 snap -1 2021-07-31T01:49:57.558 INFO:tasks.rados.rados.0.smithi045.stdout:1286: expect (ObjNum 389 snap 115 seq_num 389) 2021-07-31T01:49:57.559 INFO:tasks.rados.rados.0.smithi045.stdout:1287: delete oid 17 current snap is 120 2021-07-31T01:49:57.559 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 12 v 558 (ObjNum 1750883686 snap 1600345445 seq_num 1449613398) dirty exists 2021-07-31T01:49:57.563 INFO:tasks.rados.rados.0.smithi045.stdout:1281: done (4 left) 2021-07-31T01:49:57.564 INFO:tasks.rados.rados.0.smithi045.stdout:1285: done (3 left) 2021-07-31T01:49:57.564 INFO:tasks.rados.rados.0.smithi045.stdout:1287: done (2 left) 2021-07-31T01:49:57.564 INFO:tasks.rados.rados.0.smithi045.stdout:1288: snap_create 2021-07-31T01:49:57.566 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 47 v 267 (ObjNum 172 snap 41 seq_num 172) dirty exists 2021-07-31T01:49:57.587 INFO:tasks.rados.rados.0.smithi045.stdout:1286: expect (ObjNum 389 snap 115 seq_num 389) 2021-07-31T01:49:58.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:57 smithi154 conmon[34553]: cluster 2021-07-31T01:49:56.553139+0000 mon.a (mon.0) 939 : cluster [DBG] 2021-07-31T01:49:58.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:57 smithi154 conmon[34553]: osdmap e297: 8 total, 8 up, 8 in 2021-07-31T01:49:58.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:57 smithi045 conmon[33655]: cluster 2021-07-31T01:49:56.553139+0000 mon.a (mon.0) 939 : cluster 2021-07-31T01:49:58.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:57 smithi045 conmon[33655]: [DBG] osdmap e297: 8 total, 8 up, 8 in 2021-07-31T01:49:58.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:57 smithi045 conmon[39718]: cluster 2021-07-31T01:49:58.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:57 smithi045 conmon[39718]: 2021-07-31T01:49:56.553139+0000 2021-07-31T01:49:58.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:57 smithi045 conmon[39718]: mon.a (mon.0) 939 : cluster [DBG] osdmap e297: 8 total, 8 up, 8 in 2021-07-31T01:49:58.569 INFO:tasks.rados.rados.0.smithi045.stdout:1284: done (2 left) 2021-07-31T01:49:58.569 INFO:tasks.rados.rados.0.smithi045.stdout:1286: done (1 left) 2021-07-31T01:49:58.569 INFO:tasks.rados.rados.0.smithi045.stdout:1288: done (0 left) 2021-07-31T01:49:58.570 INFO:tasks.rados.rados.0.smithi045.stdout:1289: snap_remove snap 116 2021-07-31T01:49:59.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:58 smithi154 conmon[34553]: cluster 2021-07-31T01:49:57.555701+0000 mon.a (mon.0) 940 : cluster [DBG] osdmap e298: 8 total, 8 up, 8 in 2021-07-31T01:49:59.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:58 smithi154 conmon[34553]: 2021-07-31T01:49:59.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:58 smithi154 conmon[34553]: cluster 2021-07-31T01:49:57.829365+0000 mgr.x (mgr.24542) 115 : cluster [DBG] pgmap v227: 25 pgs: 25 active+clean; 130 MiB data, 823 MiB used, 714 GiB / 715 GiB avail; 255 B/s rd, 430 KiB/s wr, 2 op/s 2021-07-31T01:49:59.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:58 smithi045 conmon[39718]: cluster 2021-07-31T01:49:57 2021-07-31T01:49:59.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:58 smithi045 conmon[39718]: .555701+0000 mon.a (mon.0) 940 : cluster [DBG] osdmap e298: 8 total, 8 up, 8 in 2021-07-31T01:49:59.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:58 smithi045 conmon[39718]: cluster 2021-07-31 2021-07-31T01:49:59.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:58 smithi045 conmon[39718]: T01:49:57.829365+0000 mgr.x (mgr.24542) 115 : cluster [DBG] pgmap v227: 25 pgs: 25 active+clean; 130 MiB data, 823 MiB used, 714 GiB / 715 GiB avail; 255 B/s rd, 430 KiB/s wr, 2 op/s 2021-07-31T01:49:59.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:58 smithi045 conmon[33655]: cluster 2021-07-31T01:49:57.555701+0000 mon.a (mon.0) 940 : cluster 2021-07-31T01:49:59.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:58 smithi045 conmon[33655]: [DBG] osdmap e298: 8 total, 8 up, 8 in 2021-07-31T01:49:59.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:58 smithi045 conmon[33655]: cluster 2021-07-31T01:49:57.829365+0000 mgr.x (mgr.24542) 115 : cluster [DBG] pgmap v227: 25 pgs: 25 active+clean; 130 MiB data, 823 MiB used, 714 GiB / 715 GiB avail; 255 B/s rd, 430 KiB/s wr, 2 op/s 2021-07-31T01:49:59.571 INFO:tasks.rados.rados.0.smithi045.stdout:1289: done (0 left) 2021-07-31T01:49:59.572 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 33 current snap is 121 2021-07-31T01:49:59.572 INFO:tasks.rados.rados.0.smithi045.stdout:1290: seq_num 399 ranges {3883008=688128,4571136=434176,5005312=745472,5750784=24576} 2021-07-31T01:49:59.597 INFO:tasks.rados.rados.0.smithi045.stdout:1290: writing smithi04569269-33 from 3883008 to 4571136 tid 1 2021-07-31T01:49:59.599 INFO:tasks.rados.rados.0.smithi045.stdout:1290: writing smithi04569269-33 from 4571136 to 5005312 tid 2 2021-07-31T01:49:59.604 INFO:tasks.rados.rados.0.smithi045.stdout:1290: writing smithi04569269-33 from 5005312 to 5750784 tid 3 2021-07-31T01:49:59.604 INFO:tasks.rados.rados.0.smithi045.stdout:1290: writing smithi04569269-33 from 5750784 to 5775360 tid 4 2021-07-31T01:49:59.604 INFO:tasks.rados.rados.0.smithi045.stdout:1291: copy_from oid 22 from oid 50 current snap is 121 2021-07-31T01:49:59.604 INFO:tasks.rados.rados.0.smithi045.stdout:1290: finishing write tid 1 to smithi04569269-33 2021-07-31T01:49:59.605 INFO:tasks.rados.rados.0.smithi045.stdout:1290: finishing write tid 2 to smithi04569269-33 2021-07-31T01:49:59.605 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 38 current snap is 121 2021-07-31T01:49:59.605 INFO:tasks.rados.rados.0.smithi045.stdout:1292: seq_num 400 ranges {3383296=704512,4087808=720896,4808704=483328} 2021-07-31T01:49:59.626 INFO:tasks.rados.rados.0.smithi045.stdout:1292: writing smithi04569269-38 from 3383296 to 4087808 tid 1 2021-07-31T01:49:59.630 INFO:tasks.rados.rados.0.smithi045.stdout:1292: writing smithi04569269-38 from 4087808 to 4808704 tid 2 2021-07-31T01:49:59.634 INFO:tasks.rados.rados.0.smithi045.stdout:1292: writing smithi04569269-38 from 4808704 to 5292032 tid 3 2021-07-31T01:49:59.635 INFO:tasks.rados.rados.0.smithi045.stdout:1293: read oid 17 snap -1 2021-07-31T01:49:59.635 INFO:tasks.rados.rados.0.smithi045.stdout:1293: expect deleted 2021-07-31T01:49:59.636 INFO:tasks.rados.rados.0.smithi045.stdout:1294: read oid 43 snap -1 2021-07-31T01:49:59.636 INFO:tasks.rados.rados.0.smithi045.stdout:1294: expect (ObjNum 320 snap 95 seq_num 320) 2021-07-31T01:49:59.637 INFO:tasks.rados.rados.0.smithi045.stdout:1295: read oid 10 snap -1 2021-07-31T01:49:59.637 INFO:tasks.rados.rados.0.smithi045.stdout:1295: expect deleted 2021-07-31T01:49:59.638 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 15 current snap is 121 2021-07-31T01:49:59.639 INFO:tasks.rados.rados.0.smithi045.stdout:1291: finishing copy_from to smithi04569269-22 2021-07-31T01:49:59.639 INFO:tasks.rados.rados.0.smithi045.stdout:1291: got expected ENOENT (src dne) 2021-07-31T01:49:59.640 INFO:tasks.rados.rados.0.smithi045.stdout:1296: seq_num 401 ranges {0=720896,720896=606208,1327104=598016} 2021-07-31T01:49:59.641 INFO:tasks.rados.rados.0.smithi045.stdout:1296: writing smithi04569269-15 from 0 to 720896 tid 1 2021-07-31T01:49:59.641 INFO:tasks.rados.rados.0.smithi045.stdout:1296: writing smithi04569269-15 from 720896 to 1327104 tid 2 2021-07-31T01:49:59.646 INFO:tasks.rados.rados.0.smithi045.stdout:1296: writing smithi04569269-15 from 1327104 to 1925120 tid 3 2021-07-31T01:49:59.646 INFO:tasks.rados.rados.0.smithi045.stdout:1297: rollback oid 39 current snap is 121 2021-07-31T01:49:59.646 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 39 to 120 2021-07-31T01:49:59.646 INFO:tasks.rados.rados.0.smithi045.stdout:1291: finishing copy_from racing read to smithi04569269-22 2021-07-31T01:49:59.647 INFO:tasks.rados.rados.0.smithi045.stdout:1290: finishing write tid 3 to smithi04569269-33 2021-07-31T01:49:59.647 INFO:tasks.rados.rados.0.smithi045.stdout:1290: finishing write tid 4 to smithi04569269-33 2021-07-31T01:49:59.647 INFO:tasks.rados.rados.0.smithi045.stdout:1290: finishing write tid 5 to smithi04569269-33 2021-07-31T01:49:59.647 INFO:tasks.rados.rados.0.smithi045.stdout:1290: finishing write tid 6 to smithi04569269-33 2021-07-31T01:49:59.647 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 33 v 561 (ObjNum 399 snap 121 seq_num 399) dirty exists 2021-07-31T01:49:59.648 INFO:tasks.rados.rados.0.smithi045.stdout:1290: left oid 33 (ObjNum 399 snap 121 seq_num 399) 2021-07-31T01:49:59.648 INFO:tasks.rados.rados.0.smithi045.stdout:1292: finishing write tid 1 to smithi04569269-38 2021-07-31T01:49:59.649 INFO:tasks.rados.rados.0.smithi045.stdout:1290: done (7 left) 2021-07-31T01:49:59.649 INFO:tasks.rados.rados.0.smithi045.stdout:1291: done (6 left) 2021-07-31T01:49:59.650 INFO:tasks.rados.rados.0.smithi045.stdout:1298: read oid 4 snap -1 2021-07-31T01:49:59.650 INFO:tasks.rados.rados.0.smithi045.stdout:1298: expect (ObjNum 375 snap 109 seq_num 375) 2021-07-31T01:49:59.650 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 41 current snap is 121 2021-07-31T01:49:59.651 INFO:tasks.rados.rados.0.smithi045.stdout:1292: finishing write tid 2 to smithi04569269-38 2021-07-31T01:49:59.651 INFO:tasks.rados.rados.0.smithi045.stdout:1299: seq_num 402 ranges {4792320=737280,5529600=491520,6021120=516096,6537216=196608} 2021-07-31T01:49:59.674 INFO:tasks.rados.rados.0.smithi045.stdout:1299: writing smithi04569269-41 from 4792320 to 5529600 tid 1 2021-07-31T01:49:59.677 INFO:tasks.rados.rados.0.smithi045.stdout:1299: writing smithi04569269-41 from 5529600 to 6021120 tid 2 2021-07-31T01:49:59.680 INFO:tasks.rados.rados.0.smithi045.stdout:1299: writing smithi04569269-41 from 6021120 to 6537216 tid 3 2021-07-31T01:49:59.681 INFO:tasks.rados.rados.0.smithi045.stdout:1299: writing smithi04569269-41 from 6537216 to 6733824 tid 4 2021-07-31T01:49:59.681 INFO:tasks.rados.rados.0.smithi045.stdout:1293: done (7 left) 2021-07-31T01:49:59.682 INFO:tasks.rados.rados.0.smithi045.stdout:1300: snap_remove snap 119 2021-07-31T01:49:59.797 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T01:50:00.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:49:59 smithi154 conmon[34553]: cluster 2021-07-31T01:49:58.567762+0000 mon.a (mon.0) 941 : cluster [DBG] osdmap e299: 8 total, 8 up, 8 in 2021-07-31T01:50:00.003 INFO:journalctl@ceph.osd.7.smithi154.stdout:Jul 31 01:49:59 smithi154 conmon[49149]: debug 2021-07-31T01:50:00.003 INFO:journalctl@ceph.osd.7.smithi154.stdout:Jul 31 01:49:59 smithi154 conmon[49149]: 2021-07-31T01:49:59.574+0000 7f8a719fc700 -1 osd.7 pg_epoch: 300 pg[3.ds0( v 300'528 (0'0,300'528] local-lis/les=56/57 n=7 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [7,5,3]p7(0) r=0 lpr=56 luod=298'526 crt=298'526 lcod 298'524 mlcod 298'524 active+clean+snaptrim trimq=[e8~1] ps=[e6~2,eb~1,ed~1]] removing snap head 2021-07-31T01:50:00.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:59 smithi045 conmon[33655]: cluster 2021-07-31T01:49:58.567762+0000 mon.a (mon.0) 941 2021-07-31T01:50:00.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:49:59 smithi045 conmon[33655]: : cluster [DBG] osdmap e299: 8 total, 8 up, 8 in 2021-07-31T01:50:00.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:59 smithi045 conmon[39718]: cluster 2021-07-31T01 2021-07-31T01:50:00.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:49:59 smithi045 conmon[39718]: :49:58.567762+0000 mon.a (mon.0) 941 : cluster [DBG] osdmap e299: 8 total, 8 up, 8 in 2021-07-31T01:50:00.028 INFO:journalctl@ceph.osd.3.smithi045.stdout:Jul 31 01:49:59 smithi045 conmon[54013]: debug 2021-07-31T01:49:59.579+0000 7f64fc8cd700 -1 osd.3 pg_epoch: 300 pg[3.bs0( v 300'917 (0'0,300'917] local-lis/les=56/57 n=11 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [3,0,4]p3(0) r=0 lpr=56 luod=298'915 crt=298'915 lcod 296'913 mlcod 296'913 active+clean+snaptrim trimq=[e8~1] ps=[e6~2,eb~1,ed~1]] removing snap head 2021-07-31T01:50:00.029 INFO:journalctl@ceph.osd.3.smithi045.stdout:Jul 31 01:49:59 smithi045 conmon[54013]: debug 2021-07-31T01:49:59.584+0000 7f64fc8cd700 -1 osd.3 pg_epoch: 300 pg[3.bs0( v 300'921 (0'0,300'921] local-lis/les=56/57 n=8 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [3,0,4]p3(0) r=0 lpr=56 luod=300'919 crt=300'919 lcod 300'917 mlcod 300'917 active+clean+snaptrim trimq=[e8~1] ps=[e6~2,eb~1,ed~1]] removing snap head 2021-07-31T01:50:00.075 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T01:50:00.075 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 4m ago 5m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T01:50:00.075 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 2m ago 5m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T01:50:00.075 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 2m ago 8m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T01:50:00.076 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 4m ago 11m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T01:50:00.076 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 4m ago 11m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T01:50:00.076 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 2m ago 9m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T01:50:00.076 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 4m ago 9m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T01:50:00.077 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 4m ago 5m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T01:50:00.077 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 2m ago 5m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T01:50:00.077 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 4m ago 8m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T01:50:00.077 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 4m ago 8m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T01:50:00.078 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 4m ago 7m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T01:50:00.078 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 4m ago 7m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T01:50:00.078 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 2m ago 7m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T01:50:00.079 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 2m ago 6m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T01:50:00.079 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 2m ago 6m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T01:50:00.079 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 2m ago 6m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T01:50:00.080 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 2m ago 5m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T01:50:00.418 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T01:50:00.419 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T01:50:00.419 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T01:50:00.419 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T01:50:00.419 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T01:50:00.420 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T01:50:00.420 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T01:50:00.420 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T01:50:00.420 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T01:50:00.420 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T01:50:00.421 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T01:50:00.421 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T01:50:00.421 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T01:50:00.421 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T01:50:00.421 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T01:50:00.422 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T01:50:00.422 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T01:50:00.581 INFO:tasks.rados.rados.0.smithi045.stdout:1300: done (7 left) 2021-07-31T01:50:00.582 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 40 current snap is 121 2021-07-31T01:50:00.582 INFO:tasks.rados.rados.0.smithi045.stdout:1301: seq_num 403 ranges {9379840=753664,10133504=425984,10559488=720896,11280384=57344} 2021-07-31T01:50:00.636 INFO:tasks.rados.rados.0.smithi045.stdout:1301: writing smithi04569269-40 from 9379840 to 10133504 tid 1 2021-07-31T01:50:00.637 INFO:tasks.rados.rados.0.smithi045.stdout:1301: writing smithi04569269-40 from 10133504 to 10559488 tid 2 2021-07-31T01:50:00.642 INFO:tasks.rados.rados.0.smithi045.stdout:1301: writing smithi04569269-40 from 10559488 to 11280384 tid 3 2021-07-31T01:50:00.643 INFO:tasks.rados.rados.0.smithi045.stdout:1301: writing smithi04569269-40 from 11280384 to 11337728 tid 4 2021-07-31T01:50:00.643 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 46 current snap is 121 2021-07-31T01:50:00.644 INFO:tasks.rados.rados.0.smithi045.stdout:1292: finishing write tid 3 to smithi04569269-38 2021-07-31T01:50:00.644 INFO:tasks.rados.rados.0.smithi045.stdout:1292: finishing write tid 4 to smithi04569269-38 2021-07-31T01:50:00.644 INFO:tasks.rados.rados.0.smithi045.stdout:1292: finishing write tid 5 to smithi04569269-38 2021-07-31T01:50:00.644 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 38 v 565 (ObjNum 400 snap 121 seq_num 400) dirty exists 2021-07-31T01:50:00.644 INFO:tasks.rados.rados.0.smithi045.stdout:1292: left oid 38 (ObjNum 400 snap 121 seq_num 400) 2021-07-31T01:50:00.645 INFO:tasks.rados.rados.0.smithi045.stdout:1294: expect (ObjNum 320 snap 95 seq_num 320) 2021-07-31T01:50:00.659 INFO:tasks.rados.rados.0.smithi045.stdout:1296: finishing write tid 1 to smithi04569269-15 2021-07-31T01:50:00.659 INFO:tasks.rados.rados.0.smithi045.stdout:1296: finishing write tid 2 to smithi04569269-15 2021-07-31T01:50:00.660 INFO:tasks.rados.rados.0.smithi045.stdout:1297: finishing rollback tid 0 to smithi04569269-39 2021-07-31T01:50:00.660 INFO:tasks.rados.rados.0.smithi045.stdout:1297: finishing rollback tid 1 to smithi04569269-39 2021-07-31T01:50:00.660 INFO:tasks.rados.rados.0.smithi045.stdout:1297: finishing rollback tid 2 to smithi04569269-39 2021-07-31T01:50:00.660 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 39 v 568 (ObjNum 342 snap 102 seq_num 342) dirty exists 2021-07-31T01:50:00.660 INFO:tasks.rados.rados.0.smithi045.stdout:1296: finishing write tid 3 to smithi04569269-15 2021-07-31T01:50:00.661 INFO:tasks.rados.rados.0.smithi045.stdout:1302: seq_num 404 ranges {5595136=770048,6365184=712704,7077888=491520} 2021-07-31T01:50:00.694 INFO:tasks.rados.rados.0.smithi045.stdout:1302: writing smithi04569269-46 from 5595136 to 6365184 tid 1 2021-07-31T01:50:00.698 INFO:tasks.rados.rados.0.smithi045.stdout:1302: writing smithi04569269-46 from 6365184 to 7077888 tid 2 2021-07-31T01:50:00.703 INFO:tasks.rados.rados.0.smithi045.stdout:1302: writing smithi04569269-46 from 7077888 to 7569408 tid 3 2021-07-31T01:50:00.704 INFO:tasks.rados.rados.0.smithi045.stdout:1292: done (8 left) 2021-07-31T01:50:00.704 INFO:tasks.rados.rados.0.smithi045.stdout:1294: done (7 left) 2021-07-31T01:50:00.704 INFO:tasks.rados.rados.0.smithi045.stdout:1295: done (6 left) 2021-07-31T01:50:00.704 INFO:tasks.rados.rados.0.smithi045.stdout:1297: done (5 left) 2021-07-31T01:50:00.705 INFO:tasks.rados.rados.0.smithi045.stdout:1303: copy_from oid 1 from oid 31 current snap is 121 2021-07-31T01:50:00.705 INFO:tasks.rados.rados.0.smithi045.stdout:1296: finishing write tid 4 to smithi04569269-15 2021-07-31T01:50:00.705 INFO:tasks.rados.rados.0.smithi045.stdout:1296: finishing write tid 5 to smithi04569269-15 2021-07-31T01:50:00.705 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 15 v 347 (ObjNum 401 snap 121 seq_num 401) dirty exists 2021-07-31T01:50:00.706 INFO:tasks.rados.rados.0.smithi045.stdout:1296: left oid 15 (ObjNum 401 snap 121 seq_num 401) 2021-07-31T01:50:00.706 INFO:tasks.rados.rados.0.smithi045.stdout:1298: expect (ObjNum 375 snap 109 seq_num 375) 2021-07-31T01:50:00.741 INFO:tasks.rados.rados.0.smithi045.stdout:1299: finishing write tid 1 to smithi04569269-41 2021-07-31T01:50:00.742 INFO:tasks.rados.rados.0.smithi045.stdout:1299: finishing write tid 2 to smithi04569269-41 2021-07-31T01:50:00.742 INFO:tasks.rados.rados.0.smithi045.stdout:1299: finishing write tid 3 to smithi04569269-41 2021-07-31T01:50:00.742 INFO:tasks.rados.rados.0.smithi045.stdout:1299: finishing write tid 4 to smithi04569269-41 2021-07-31T01:50:00.743 INFO:tasks.rados.rados.0.smithi045.stdout:1299: finishing write tid 5 to smithi04569269-41 2021-07-31T01:50:00.743 INFO:tasks.rados.rados.0.smithi045.stdout:1299: finishing write tid 6 to smithi04569269-41 2021-07-31T01:50:00.743 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 41 v 524 (ObjNum 402 snap 121 seq_num 402) dirty exists 2021-07-31T01:50:00.743 INFO:tasks.rados.rados.0.smithi045.stdout:1299: left oid 41 (ObjNum 402 snap 121 seq_num 402) 2021-07-31T01:50:00.743 INFO:tasks.rados.rados.0.smithi045.stdout:1301: finishing write tid 1 to smithi04569269-40 2021-07-31T01:50:00.744 INFO:tasks.rados.rados.0.smithi045.stdout:1301: finishing write tid 2 to smithi04569269-40 2021-07-31T01:50:00.744 INFO:tasks.rados.rados.0.smithi045.stdout:1301: finishing write tid 3 to smithi04569269-40 2021-07-31T01:50:00.744 INFO:tasks.rados.rados.0.smithi045.stdout:1296: done (5 left) 2021-07-31T01:50:00.744 INFO:tasks.rados.rados.0.smithi045.stdout:1298: done (4 left) 2021-07-31T01:50:00.745 INFO:tasks.rados.rados.0.smithi045.stdout:1299: done (3 left) 2021-07-31T01:50:00.746 INFO:tasks.rados.rados.0.smithi045.stdout:1304: read oid 6 snap 120 2021-07-31T01:50:00.746 INFO:tasks.rados.rados.0.smithi045.stdout:1304: expect (ObjNum 215 snap 56 seq_num 215) 2021-07-31T01:50:00.746 INFO:tasks.rados.rados.0.smithi045.stdout:1305: read oid 27 snap 120 2021-07-31T01:50:00.747 INFO:tasks.rados.rados.0.smithi045.stdout:1305: expect (ObjNum 374 snap 109 seq_num 374) 2021-07-31T01:50:00.747 INFO:tasks.rados.rados.0.smithi045.stdout:1306: snap_create 2021-07-31T01:50:00.747 INFO:tasks.rados.rados.0.smithi045.stdout:1301: finishing write tid 4 to smithi04569269-40 2021-07-31T01:50:00.747 INFO:tasks.rados.rados.0.smithi045.stdout:1301: finishing write tid 5 to smithi04569269-40 2021-07-31T01:50:00.748 INFO:tasks.rados.rados.0.smithi045.stdout:1301: finishing write tid 6 to smithi04569269-40 2021-07-31T01:50:00.748 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 40 v 563 (ObjNum 403 snap 121 seq_num 403) dirty exists 2021-07-31T01:50:00.748 INFO:tasks.rados.rados.0.smithi045.stdout:1301: left oid 40 (ObjNum 403 snap 121 seq_num 403) 2021-07-31T01:50:00.748 INFO:tasks.rados.rados.0.smithi045.stdout:1302: finishing write tid 1 to smithi04569269-46 2021-07-31T01:50:00.749 INFO:tasks.rados.rados.0.smithi045.stdout:1302: finishing write tid 2 to smithi04569269-46 2021-07-31T01:50:00.749 INFO:tasks.rados.rados.0.smithi045.stdout:1302: finishing write tid 3 to smithi04569269-46 2021-07-31T01:50:00.749 INFO:tasks.rados.rados.0.smithi045.stdout:1302: finishing write tid 4 to smithi04569269-46 2021-07-31T01:50:00.749 INFO:tasks.rados.rados.0.smithi045.stdout:1302: finishing write tid 5 to smithi04569269-46 2021-07-31T01:50:00.750 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 46 v 265 (ObjNum 404 snap 121 seq_num 404) dirty exists 2021-07-31T01:50:00.750 INFO:tasks.rados.rados.0.smithi045.stdout:1302: left oid 46 (ObjNum 404 snap 121 seq_num 404) 2021-07-31T01:50:00.750 INFO:tasks.rados.rados.0.smithi045.stdout:1303: finishing copy_from to smithi04569269-1 2021-07-31T01:50:00.750 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 1 v 875 (ObjNum 385 snap 114 seq_num 385) dirty exists 2021-07-31T01:50:00.751 INFO:tasks.rados.rados.0.smithi045.stdout:1303: finishing copy_from racing read to smithi04569269-1 2021-07-31T01:50:00.753 INFO:tasks.rados.rados.0.smithi045.stdout:1305: expect (ObjNum 374 snap 109 seq_num 374) 2021-07-31T01:50:00.791 INFO:tasks.rados.rados.0.smithi045.stdout:1304: expect (ObjNum 215 snap 56 seq_num 215) 2021-07-31T01:50:01.004 INFO:journalctl@ceph.osd.7.smithi154.stdout:Jul 31 01:50:00 smithi154 conmon[49149]: debug 2021-07-31T01:50:00.637+0000 7f8a75a04700 -1 osd.7 pg_epoch: 301 pg[3.ds0( v 300'530 (0'0,300'530] local-lis/les=56/57 n=5 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [7,5,3]p7(0) r=0 lpr=56 crt=300'530 lcod 300'528 mlcod 300'528 active+clean+snaptrim trimq=[ee~1] ps=[e6~3,eb~1,ed~1]] removing snap head 2021-07-31T01:50:01.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:00 smithi154 conmon[34553]: cluster 2021-07-31T01:49:59.568768+0000 mon.a (mon.0) 942 : cluster [DBG] 2021-07-31T01:50:01.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:00 smithi154 conmon[34553]: osdmap e300: 8 total, 8 up, 8 in 2021-07-31T01:50:01.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:00 smithi154 conmon[34553]: audit 2021-07-31T01:49:59.784850+0000 mgr.x (mgr.24542) 116 : audit [DBG] from='client.24709 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:50:01.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:00 smithi154 conmon[34553]: cluster 2021-07-31T01:49:59.829880+0000 2021-07-31T01:50:01.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:00 smithi154 conmon[34553]: mgr.x (mgr.24542) 117 : cluster [DBG] pgmap v230: 25 pgs: 25 active+clean; 137 MiB data, 843 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 1.9 MiB/s wr, 25 op/s 2021-07-31T01:50:01.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:00 smithi154 conmon[34553]: cluster 2021-07-31T01:50:01.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:00 smithi154 conmon[34553]: 2021-07-31T01:50:00 2021-07-31T01:50:01.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:00 smithi154 conmon[34553]: .000097+0000 mon.a (mon.0 2021-07-31T01:50:01.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:00 smithi154 conmon[34553]: ) 943 : cluster [INF] overall HEALTH_OK 2021-07-31T01:50:01.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:00 smithi154 conmon[34553]: audit 2021-07-31T01: 2021-07-31T01:50:01.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:00 smithi154 conmon[34553]: 50:00.416522+0000 mon.a (mon.0) 944 : audit [DBG] from='client.? 172.21.15.45:0/2074719725' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T01:50:01.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:00 smithi045 conmon[33655]: cluster 2021-07-31T01:49:59.568768+0000 mon.a (mon.0) 942 : cluster [DBG] osdmap e300: 8 total, 8 up, 8 in 2021-07-31T01:50:01.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:00 smithi045 conmon[33655]: audit 2021-07-31 2021-07-31T01:50:01.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:00 smithi045 conmon[33655]: T01:49:59.784850+0000 mgr.x (mgr.24542) 116 : audit [DBG] from='client.24709 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:50:01.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:00 smithi045 conmon[33655]: cluster 2021-07-31T01:49:59.829880+0000 mgr.x (mgr.24542) 117 : cluster [DBG] pgmap v230: 25 pgs: 25 active+clean; 137 MiB data, 843 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 1.9 MiB/s wr, 25 op/s 2021-07-31T01:50:01.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:00 smithi045 conmon[33655]: cluster 2021-07-31T01:50:01.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:00 smithi045 conmon[33655]: 2021-07-31T01:50:00.000097+0000 mon.a (mon.0) 943 2021-07-31T01:50:01.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:00 smithi045 conmon[33655]: : cluster [INF] overall HEALTH_OK 2021-07-31T01:50:01.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:00 smithi045 conmon[33655]: audit 2021-07-31T01:50:01.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:00 smithi045 conmon[33655]: 2021-07-31T01:50:00.416522+0000 mon.a (mon.0) 944 : audit 2021-07-31T01:50:01.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:00 smithi045 conmon[33655]: [DBG] from='client.? 172.21.15.45:0/2074719725' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T01:50:01.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:00 smithi045 conmon[39718]: cluster 2021-07-31T01:49:59.568768 2021-07-31T01:50:01.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:00 smithi045 conmon[39718]: +0000 mon.a (mon.0) 942 : cluster [DBG] osdmap e300: 8 total, 8 up, 8 in 2021-07-31T01:50:01.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:00 smithi045 conmon[39718]: audit 2021-07-31T01 2021-07-31T01:50:01.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:00 smithi045 conmon[39718]: :49:59.784850+0000 mgr.x (mgr.24542) 116 : audit [DBG] from='client.24709 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:50:01.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:00 smithi045 conmon[39718]: cluster 2021-07-31T01:49:59.829880+0000 2021-07-31T01:50:01.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:00 smithi045 conmon[39718]: mgr.x (mgr.24542) 117 : cluster [DBG] pgmap v230: 25 pgs: 25 active+clean; 137 MiB data, 843 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 1.9 MiB/s wr, 25 op/s 2021-07-31T01:50:01.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:00 smithi045 conmon[39718]: cluster 2021-07-31T01:50 2021-07-31T01:50:01.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:00 smithi045 conmon[39718]: :00.000097+0000 mon.a (mon.0) 943 : cluster 2021-07-31T01:50:01.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:00 smithi045 conmon[39718]: [INF] overall HEALTH_OK 2021-07-31T01:50:01.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:00 smithi045 conmon[39718]: audit 2021-07-31T01:50:01.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:00 smithi045 conmon[39718]: 2021-07-31T01:50:00. 2021-07-31T01:50:01.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:00 smithi045 conmon[39718]: 416522+0000 mon.a (mon.0) 944 : audit [DBG] from='client.? 172.21.15.45:0/2074719725' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T01:50:01.531 INFO:tasks.rados.rados.0.smithi045.stdout:1301: done (5 left) 2021-07-31T01:50:01.532 INFO:tasks.rados.rados.0.smithi045.stdout:1302: done (4 left) 2021-07-31T01:50:01.532 INFO:tasks.rados.rados.0.smithi045.stdout:1303: done (3 left) 2021-07-31T01:50:01.532 INFO:tasks.rados.rados.0.smithi045.stdout:1304: done (2 left) 2021-07-31T01:50:01.532 INFO:tasks.rados.rados.0.smithi045.stdout:1305: done (1 left) 2021-07-31T01:50:01.533 INFO:tasks.rados.rados.0.smithi045.stdout:1306: done (0 left) 2021-07-31T01:50:01.533 INFO:tasks.rados.rados.0.smithi045.stdout:1307: rollback oid 19 current snap is 122 2021-07-31T01:50:01.533 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 19 to 121 2021-07-31T01:50:01.534 INFO:tasks.rados.rados.0.smithi045.stdout:1308: read oid 31 snap 120 2021-07-31T01:50:01.534 INFO:tasks.rados.rados.0.smithi045.stdout:1308: expect (ObjNum 385 snap 114 seq_num 385) 2021-07-31T01:50:01.534 INFO:tasks.rados.rados.0.smithi045.stdout:1309: copy_from oid 24 from oid 45 current snap is 122 2021-07-31T01:50:01.534 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 40 current snap is 122 2021-07-31T01:50:01.535 INFO:tasks.rados.rados.0.smithi045.stdout:1310: seq_num 405 ranges {11337728=786432,12124160=589824,12713984=614400} 2021-07-31T01:50:01.598 INFO:tasks.rados.rados.0.smithi045.stdout:1310: writing smithi04569269-40 from 11337728 to 12124160 tid 1 2021-07-31T01:50:01.601 INFO:tasks.rados.rados.0.smithi045.stdout:1310: writing smithi04569269-40 from 12124160 to 12713984 tid 2 2021-07-31T01:50:01.604 INFO:tasks.rados.rados.0.smithi045.stdout:1310: writing smithi04569269-40 from 12713984 to 13328384 tid 3 2021-07-31T01:50:01.605 INFO:tasks.rados.rados.0.smithi045.stdout:1311: copy_from oid 1 from oid 18 current snap is 122 2021-07-31T01:50:01.605 INFO:tasks.rados.rados.0.smithi045.stdout:1307: finishing rollback tid 0 to smithi04569269-19 2021-07-31T01:50:01.605 INFO:tasks.rados.rados.0.smithi045.stdout:1307: finishing rollback tid 1 to smithi04569269-19 2021-07-31T01:50:01.605 INFO:tasks.rados.rados.0.smithi045.stdout:1307: finishing rollback tid 2 to smithi04569269-19 2021-07-31T01:50:01.606 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 19 v 315 (ObjNum 353 snap 106 seq_num 353) dirty exists 2021-07-31T01:50:01.606 INFO:tasks.rados.rados.0.smithi045.stdout:1308: expect (ObjNum 385 snap 114 seq_num 385) 2021-07-31T01:50:01.782 INFO:tasks.rados.rados.0.smithi045.stdout:1309: finishing copy_from to smithi04569269-24 2021-07-31T01:50:01.782 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 24 v 488 (ObjNum 352 snap 106 seq_num 352) dirty exists 2021-07-31T01:50:01.783 INFO:tasks.rados.rados.0.smithi045.stdout:1309: finishing copy_from racing read to smithi04569269-24 2021-07-31T01:50:01.783 INFO:tasks.rados.rados.0.smithi045.stdout:1310: finishing write tid 1 to smithi04569269-40 2021-07-31T01:50:01.783 INFO:tasks.rados.rados.0.smithi045.stdout:1310: finishing write tid 2 to smithi04569269-40 2021-07-31T01:50:01.783 INFO:tasks.rados.rados.0.smithi045.stdout:1310: finishing write tid 3 to smithi04569269-40 2021-07-31T01:50:01.784 INFO:tasks.rados.rados.0.smithi045.stdout:1310: finishing write tid 4 to smithi04569269-40 2021-07-31T01:50:01.784 INFO:tasks.rados.rados.0.smithi045.stdout:1310: finishing write tid 5 to smithi04569269-40 2021-07-31T01:50:01.784 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 40 v 567 (ObjNum 405 snap 122 seq_num 405) dirty exists 2021-07-31T01:50:01.784 INFO:tasks.rados.rados.0.smithi045.stdout:1310: left oid 40 (ObjNum 405 snap 122 seq_num 405) 2021-07-31T01:50:01.784 INFO:tasks.rados.rados.0.smithi045.stdout:1311: finishing copy_from to smithi04569269-1 2021-07-31T01:50:01.785 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 1 v 309 (ObjNum 353 snap 106 seq_num 353) dirty exists 2021-07-31T01:50:01.785 INFO:tasks.rados.rados.0.smithi045.stdout:1311: finishing copy_from racing read to smithi04569269-1 2021-07-31T01:50:01.786 INFO:tasks.rados.rados.0.smithi045.stdout:1307: done (4 left) 2021-07-31T01:50:01.786 INFO:tasks.rados.rados.0.smithi045.stdout:1308: done (3 left) 2021-07-31T01:50:01.786 INFO:tasks.rados.rados.0.smithi045.stdout:1309: done (2 left) 2021-07-31T01:50:01.786 INFO:tasks.rados.rados.0.smithi045.stdout:1310: done (1 left) 2021-07-31T01:50:01.786 INFO:tasks.rados.rados.0.smithi045.stdout:1311: done (0 left) 2021-07-31T01:50:01.787 INFO:tasks.rados.rados.0.smithi045.stdout:1312: read oid 36 snap -1 2021-07-31T01:50:01.787 INFO:tasks.rados.rados.0.smithi045.stdout:1312: expect (ObjNum 366 snap 108 seq_num 366) 2021-07-31T01:50:01.787 INFO:tasks.rados.rados.0.smithi045.stdout:1313: delete oid 3 current snap is 122 2021-07-31T01:50:01.790 INFO:tasks.rados.rados.0.smithi045.stdout:1312: expect (ObjNum 366 snap 108 seq_num 366) 2021-07-31T01:50:01.879 INFO:tasks.rados.rados.0.smithi045.stdout:1312: done (1 left) 2021-07-31T01:50:01.879 INFO:tasks.rados.rados.0.smithi045.stdout:1313: done (0 left) 2021-07-31T01:50:01.880 INFO:tasks.rados.rados.0.smithi045.stdout:1314: copy_from oid 4 from oid 25 current snap is 122 2021-07-31T01:50:01.880 INFO:tasks.rados.rados.0.smithi045.stdout:1315: copy_from oid 1 from oid 11 current snap is 122 2021-07-31T01:50:01.881 INFO:tasks.rados.rados.0.smithi045.stdout:1316: copy_from oid 32 from oid 27 current snap is 122 2021-07-31T01:50:01.881 INFO:tasks.rados.rados.0.smithi045.stdout:1317: read oid 24 snap -1 2021-07-31T01:50:01.882 INFO:tasks.rados.rados.0.smithi045.stdout:1317: expect (ObjNum 352 snap 106 seq_num 352) 2021-07-31T01:50:01.882 INFO:tasks.rados.rados.0.smithi045.stdout:1318: read oid 41 snap -1 2021-07-31T01:50:01.882 INFO:tasks.rados.rados.0.smithi045.stdout:1318: expect (ObjNum 402 snap 121 seq_num 402) 2021-07-31T01:50:01.883 INFO:tasks.rados.rados.0.smithi045.stdout:1319: read oid 39 snap -1 2021-07-31T01:50:01.883 INFO:tasks.rados.rados.0.smithi045.stdout:1319: expect (ObjNum 342 snap 102 seq_num 342) 2021-07-31T01:50:01.884 INFO:tasks.rados.rados.0.smithi045.stdout:1320: setattr oid 29 current snap is 122 2021-07-31T01:50:01.884 INFO:tasks.rados.rados.0.smithi045.stdout:1314: finishing copy_from to smithi04569269-4 2021-07-31T01:50:01.885 INFO:tasks.rados.rados.0.smithi045.stdout:1314: got expected ENOENT (src dne) 2021-07-31T01:50:01.885 INFO:tasks.rados.rados.0.smithi045.stdout:1314: finishing copy_from racing read to smithi04569269-4 2021-07-31T01:50:01.885 INFO:tasks.rados.rados.0.smithi045.stdout:1314: done (6 left) 2021-07-31T01:50:01.885 INFO:tasks.rados.rados.0.smithi045.stdout:1321: snap_remove snap 120 2021-07-31T01:50:02.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:01 smithi154 conmon[34553]: audit 2021-07-31T01:50:00.066473+0000 mgr.x (mgr.24542) 118 : audit [DBG] from='client.14808 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:50:02.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:01 smithi154 conmon[34553]: 2021-07-31T01:50:02.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:01 smithi154 conmon[34553]: cluster 2021-07-31T01:50:00.579929+0000 mon.a (mon.0) 945 : cluster [DBG] osdmap e301: 8 total, 8 up, 8 in 2021-07-31T01:50:02.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:01 smithi154 conmon[34553]: cluster 2021-07-31T01:50:01.530271+0000 mon.a (mon.0) 2021-07-31T01:50:02.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:01 smithi154 conmon[34553]: 946 : cluster [DBG] osdmap e302: 8 total, 8 up, 8 in 2021-07-31T01:50:02.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:01 smithi045 conmon[33655]: audit 2021-07-31T01:50:00.066473+0000 mgr.x (mgr.24542 2021-07-31T01:50:02.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:01 smithi045 conmon[33655]: ) 118 : audit [DBG] from='client.14808 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:50:02.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:01 smithi045 conmon[33655]: cluster 2021-07-31T01:50:00.579929+0000 mon.a (mon.0) 945 : cluster [DBG] osdmap e301: 8 total, 8 up, 8 in 2021-07-31T01:50:02.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:01 smithi045 conmon[33655]: cluster 2021-07 2021-07-31T01:50:02.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:01 smithi045 conmon[33655]: -31T01:50:01.530271+0000 mon.a (mon.0) 946 : cluster [DBG] osdmap e302: 8 total, 8 up, 8 in 2021-07-31T01:50:02.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:01 smithi045 conmon[39718]: audit 2021-07-31T01:50:00.066473+0000 2021-07-31T01:50:02.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:01 smithi045 conmon[39718]: mgr.x (mgr.24542) 118 : audit [DBG] from='client.14808 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:50:02.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:01 smithi045 conmon[39718]: cluster 2021-07-31T01:50:02.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:01 smithi045 conmon[39718]: 2021-07-31T01:50:00.579929+0000 mon.a (mon.0) 945 2021-07-31T01:50:02.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:01 smithi045 conmon[39718]: : cluster [DBG] osdmap e301: 8 total, 8 up, 8 in 2021-07-31T01:50:02.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:01 smithi045 conmon[39718]: cluster 2021-07-31T01:50:02.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:01 smithi045 conmon[39718]: 2021-07-31T01: 2021-07-31T01:50:02.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:01 smithi045 conmon[39718]: 50:01.530271+0000 mon.a (mon.0) 2021-07-31T01:50:02.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:01 smithi045 conmon[39718]: 946 : cluster [DBG] osdmap e302: 8 total, 8 up, 8 in 2021-07-31T01:50:02.589 INFO:tasks.rados.rados.0.smithi045.stdout:1321: done (6 left) 2021-07-31T01:50:02.590 INFO:tasks.rados.rados.0.smithi045.stdout:1322: snap_remove snap 121 2021-07-31T01:50:03.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:02 smithi154 conmon[34553]: cluster 2021-07-31T01:50:01.830432+0000 2021-07-31T01:50:03.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:02 smithi154 conmon[34553]: mgr.x (mgr.24542) 119 : cluster [DBG] pgmap v233: 25 pgs: 25 active+clean; 137 MiB data, 843 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 1.9 MiB/s wr, 25 op/s 2021-07-31T01:50:03.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:02 smithi045 conmon[39718]: cluster 2021-07-31T01:50:01.830432+0000 mgr.x ( 2021-07-31T01:50:03.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:02 smithi045 conmon[39718]: mgr.24542) 119 : cluster [DBG] pgmap v233: 25 pgs: 25 active+clean; 137 MiB data, 843 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 1.9 MiB/s wr, 25 op/s 2021-07-31T01:50:03.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:02 smithi045 conmon[33655]: cluster 2021-07-31T01:50:01.830432+0000 mgr.x (mgr.24542) 119 : cluster 2021-07-31T01:50:03.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:02 smithi045 conmon[33655]: [DBG] pgmap v233: 25 pgs: 25 active+clean; 137 MiB data, 843 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 1.9 MiB/s wr, 25 op/s 2021-07-31T01:50:03.594 INFO:tasks.rados.rados.0.smithi045.stdout:1322: done (6 left) 2021-07-31T01:50:03.594 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 38 current snap is 122 2021-07-31T01:50:03.595 INFO:tasks.rados.rados.0.smithi045.stdout:1323: seq_num 407 ranges {5292032=425984,5718016=761856,6479872=704512,7184384=139264} 2021-07-31T01:50:03.625 INFO:tasks.rados.rados.0.smithi045.stdout:1323: writing smithi04569269-38 from 5292032 to 5718016 tid 1 2021-07-31T01:50:03.629 INFO:tasks.rados.rados.0.smithi045.stdout:1323: writing smithi04569269-38 from 5718016 to 6479872 tid 2 2021-07-31T01:50:03.634 INFO:tasks.rados.rados.0.smithi045.stdout:1323: writing smithi04569269-38 from 6479872 to 7184384 tid 3 2021-07-31T01:50:03.635 INFO:tasks.rados.rados.0.smithi045.stdout:1323: writing smithi04569269-38 from 7184384 to 7323648 tid 4 2021-07-31T01:50:03.635 INFO:tasks.rados.rados.0.smithi045.stdout:1324: rmattr oid 17 current snap is 122 2021-07-31T01:50:03.636 INFO:tasks.rados.rados.0.smithi045.stdout:1324: done (7 left) 2021-07-31T01:50:03.636 INFO:tasks.rados.rados.0.smithi045.stdout:1325: read oid 18 snap -1 2021-07-31T01:50:03.636 INFO:tasks.rados.rados.0.smithi045.stdout:1325: expect (ObjNum 353 snap 106 seq_num 353) 2021-07-31T01:50:03.636 INFO:tasks.rados.rados.0.smithi045.stdout:1326: delete oid 26 current snap is 122 2021-07-31T01:50:03.636 INFO:tasks.rados.rados.0.smithi045.stdout:1317: expect (ObjNum 352 snap 106 seq_num 352) 2021-07-31T01:50:03.667 INFO:tasks.rados.rados.0.smithi045.stdout:1315: finishing copy_from to smithi04569269-1 2021-07-31T01:50:03.667 INFO:tasks.rados.rados.0.smithi045.stdout:1315: got expected ENOENT (src dne) 2021-07-31T01:50:03.667 INFO:tasks.rados.rados.0.smithi045.stdout:1315: finishing copy_from racing read to smithi04569269-1 2021-07-31T01:50:03.667 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 29 v 188 (ObjNum 335 snap 99 seq_num 335) dirty exists 2021-07-31T01:50:03.668 INFO:tasks.rados.rados.0.smithi045.stdout:1319: expect (ObjNum 342 snap 102 seq_num 342) 2021-07-31T01:50:03.690 INFO:tasks.rados.rados.0.smithi045.stdout:1316: finishing copy_from to smithi04569269-32 2021-07-31T01:50:03.691 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 32 v 372 (ObjNum 374 snap 109 seq_num 374) dirty exists 2021-07-31T01:50:03.691 INFO:tasks.rados.rados.0.smithi045.stdout:1316: finishing copy_from racing read to smithi04569269-32 2021-07-31T01:50:03.691 INFO:tasks.rados.rados.0.smithi045.stdout:1318: expect (ObjNum 402 snap 121 seq_num 402) 2021-07-31T01:50:03.776 INFO:journalctl@ceph.osd.3.smithi045.stdout:Jul 31 01:50:03 smithi045 conmon[54013]: debug 2021-07-31T01:50:03.628+0000 7f64f88c5700 -1 osd.3 pg_epoch: 304 pg[3.bs0( v 303'927 (0'0,303'927] local-lis/les=56/57 n=7 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [3,0,4]p3(0) r=0 lpr=56 crt=303'927 lcod 302'925 mlcod 302'925 active+clean+snaptrim trimq=[f3~1] ps=[e8~1,ee~1,f0~1]] removing snap head 2021-07-31T01:50:03.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:03 smithi045 conmon[39718]: cluster 2021-07-31T01:50:02.587824+0000 2021-07-31T01:50:03.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:03 smithi045 conmon[39718]: mon.a (mon.0) 947 : cluster [DBG] osdmap e303: 8 total, 8 up, 8 in 2021-07-31T01:50:03.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:50:03 smithi045 conmon[61210]: level=error ts=2021-07-31T01:50:03.513Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:50:03.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:50:03 smithi045 conmon[61210]: level=error ts=2021-07-31T01:50:03.513Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:50:03.778 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:50:03 smithi045 conmon[61210]: level=error ts=2021-07-31T01:50:03.513Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T01:50:03.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:03 smithi045 conmon[33655]: cluster 2021-07-31T01:50:02.587824+0000 2021-07-31T01:50:03.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:03 smithi045 conmon[33655]: mon.a (mon.0) 947 : cluster [DBG] osdmap e303: 8 total, 8 up, 8 in 2021-07-31T01:50:04.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:03 smithi154 conmon[34553]: cluster 2021-07-31T01:50:02.587824+0000 mon.a (mon.0) 947 : cluster [DBG] osdmap e303: 8 total, 8 up, 8 in 2021-07-31T01:50:04.140 INFO:tasks.rados.rados.0.smithi045.stdout:1323: finishing write tid 1 to smithi04569269-38 2021-07-31T01:50:04.141 INFO:tasks.rados.rados.0.smithi045.stdout:1323: finishing write tid 2 to smithi04569269-38 2021-07-31T01:50:04.141 INFO:tasks.rados.rados.0.smithi045.stdout:1323: finishing write tid 3 to smithi04569269-38 2021-07-31T01:50:04.141 INFO:tasks.rados.rados.0.smithi045.stdout:1323: finishing write tid 4 to smithi04569269-38 2021-07-31T01:50:04.142 INFO:tasks.rados.rados.0.smithi045.stdout:1323: finishing write tid 5 to smithi04569269-38 2021-07-31T01:50:04.142 INFO:tasks.rados.rados.0.smithi045.stdout:1325: expect (ObjNum 353 snap 106 seq_num 353) 2021-07-31T01:50:04.194 INFO:tasks.rados.rados.0.smithi045.stdout:1323: finishing write tid 6 to smithi04569269-38 2021-07-31T01:50:04.194 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 38 v 573 (ObjNum 407 snap 122 seq_num 407) dirty exists 2021-07-31T01:50:04.194 INFO:tasks.rados.rados.0.smithi045.stdout:1323: left oid 38 (ObjNum 407 snap 122 seq_num 407) 2021-07-31T01:50:04.194 INFO:tasks.rados.rados.0.smithi045.stdout:1315: done (8 left) 2021-07-31T01:50:04.195 INFO:tasks.rados.rados.0.smithi045.stdout:1316: done (7 left) 2021-07-31T01:50:04.195 INFO:tasks.rados.rados.0.smithi045.stdout:1317: done (6 left) 2021-07-31T01:50:04.195 INFO:tasks.rados.rados.0.smithi045.stdout:1318: done (5 left) 2021-07-31T01:50:04.195 INFO:tasks.rados.rados.0.smithi045.stdout:1319: done (4 left) 2021-07-31T01:50:04.196 INFO:tasks.rados.rados.0.smithi045.stdout:1320: done (3 left) 2021-07-31T01:50:04.196 INFO:tasks.rados.rados.0.smithi045.stdout:1323: done (2 left) 2021-07-31T01:50:04.196 INFO:tasks.rados.rados.0.smithi045.stdout:1325: done (1 left) 2021-07-31T01:50:04.196 INFO:tasks.rados.rados.0.smithi045.stdout:1326: done (0 left) 2021-07-31T01:50:04.196 INFO:tasks.rados.rados.0.smithi045.stdout:1327: setattr oid 42 current snap is 122 2021-07-31T01:50:04.197 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 9 current snap is 122 2021-07-31T01:50:04.197 INFO:tasks.rados.rados.0.smithi045.stdout:1328: seq_num 409 ranges {483328=458752,942080=532480,1474560=720896,2195456=352256} 2021-07-31T01:50:04.199 INFO:tasks.rados.rados.0.smithi045.stdout:1328: writing smithi04569269-9 from 483328 to 942080 tid 1 2021-07-31T01:50:04.202 INFO:tasks.rados.rados.0.smithi045.stdout:1328: writing smithi04569269-9 from 942080 to 1474560 tid 2 2021-07-31T01:50:04.206 INFO:tasks.rados.rados.0.smithi045.stdout:1328: writing smithi04569269-9 from 1474560 to 2195456 tid 3 2021-07-31T01:50:04.210 INFO:tasks.rados.rados.0.smithi045.stdout:1328: writing smithi04569269-9 from 2195456 to 2547712 tid 4 2021-07-31T01:50:04.210 INFO:tasks.rados.rados.0.smithi045.stdout:1329: rollback oid 48 current snap is 122 2021-07-31T01:50:04.211 INFO:tasks.rados.rados.0.smithi045.stdout:1329: done (2 left) 2021-07-31T01:50:04.211 INFO:tasks.rados.rados.0.smithi045.stdout:1330: snap_create 2021-07-31T01:50:04.211 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 42 v 533 (ObjNum 176 snap 46 seq_num 176) dirty exists 2021-07-31T01:50:04.212 INFO:tasks.rados.rados.0.smithi045.stdout:1328: finishing write tid 1 to smithi04569269-9 2021-07-31T01:50:04.213 INFO:tasks.rados.rados.0.smithi045.stdout:1328: finishing write tid 2 to smithi04569269-9 2021-07-31T01:50:04.215 INFO:tasks.rados.rados.0.smithi045.stdout:1328: finishing write tid 3 to smithi04569269-9 2021-07-31T01:50:04.216 INFO:tasks.rados.rados.0.smithi045.stdout:1328: finishing write tid 4 to smithi04569269-9 2021-07-31T01:50:04.216 INFO:tasks.rados.rados.0.smithi045.stdout:1328: finishing write tid 5 to smithi04569269-9 2021-07-31T01:50:04.217 INFO:tasks.rados.rados.0.smithi045.stdout:1328: finishing write tid 6 to smithi04569269-9 2021-07-31T01:50:04.217 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 9 v 280 (ObjNum 409 snap 122 seq_num 409) dirty exists 2021-07-31T01:50:04.217 INFO:tasks.rados.rados.0.smithi045.stdout:1328: left oid 9 (ObjNum 409 snap 122 seq_num 409) 2021-07-31T01:50:04.599 INFO:tasks.rados.rados.0.smithi045.stdout:1327: done (2 left) 2021-07-31T01:50:04.599 INFO:tasks.rados.rados.0.smithi045.stdout:1328: done (1 left) 2021-07-31T01:50:04.600 INFO:tasks.rados.rados.0.smithi045.stdout:1330: done (0 left) 2021-07-31T01:50:04.600 INFO:tasks.rados.rados.0.smithi045.stdout:1331: snap_remove snap 122 2021-07-31T01:50:05.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:04 smithi154 conmon[34553]: cluster 2021-07-31T01:50:03.593204 2021-07-31T01:50:05.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:04 smithi154 conmon[34553]: +0000 mon.a (mon.0) 948 : cluster [DBG] osdmap e304: 8 total, 8 up, 8 in 2021-07-31T01:50:05.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:04 smithi154 conmon[34553]: cluster 2021-07-31T01:50:03.830982+0000 mgr.x (mgr.24542) 120 : cluster [DBG] pgmap v236: 25 pgs: 25 active+clean; 114 MiB data, 849 MiB used, 714 GiB / 715 GiB avail; 17 MiB/s rd, 5.7 MiB/s wr, 30 op/s 2021-07-31T01:50:05.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:04 smithi045 conmon[33655]: cluster 2021-07-31T01:50:03.593204+0000 2021-07-31T01:50:05.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:04 smithi045 conmon[33655]: mon.a (mon.0) 948 : cluster [DBG] osdmap e304: 8 total, 8 up, 8 in 2021-07-31T01:50:05.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:04 smithi045 conmon[33655]: cluster 2021-07-31T01:50:03.830982+0000 mgr.x (mgr.24542) 120 : cluster [DBG] 2021-07-31T01:50:05.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:04 smithi045 conmon[33655]: pgmap v236: 25 pgs: 25 active+clean; 114 MiB data, 849 MiB used, 714 GiB / 715 GiB avail; 17 MiB/s rd, 5.7 MiB/s wr, 30 op/s 2021-07-31T01:50:05.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:04 smithi045 conmon[39718]: cluster 2021-07-31T01:50:03. 2021-07-31T01:50:05.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:04 smithi045 conmon[39718]: 593204+0000 mon.a (mon.0) 948 : cluster [DBG] osdmap e304: 8 total, 8 up, 8 in 2021-07-31T01:50:05.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:04 smithi045 conmon[39718]: cluster 2021-07-31T01:50:03 2021-07-31T01:50:05.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:04 smithi045 conmon[39718]: .830982+0000 mgr.x (mgr.24542) 120 : cluster 2021-07-31T01:50:05.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:04 smithi045 conmon[39718]: [DBG] pgmap v236: 25 pgs: 25 active+clean; 114 MiB data, 849 MiB used, 714 GiB / 715 GiB avail; 17 MiB/s rd, 5.7 MiB/s wr, 30 op/s 2021-07-31T01:50:05.601 INFO:tasks.rados.rados.0.smithi045.stdout:1331: done (0 left) 2021-07-31T01:50:05.602 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 29 current snap is 123 2021-07-31T01:50:05.602 INFO:tasks.rados.rados.0.smithi045.stdout:1332: seq_num 410 ranges {2801664=475136,3276800=417792,3694592=475136,4169728=425984,4595712=286720} 2021-07-31T01:50:05.619 INFO:tasks.rados.rados.0.smithi045.stdout:1332: writing smithi04569269-29 from 2801664 to 3276800 tid 1 2021-07-31T01:50:05.621 INFO:tasks.rados.rados.0.smithi045.stdout:1332: writing smithi04569269-29 from 3276800 to 3694592 tid 2 2021-07-31T01:50:05.624 INFO:tasks.rados.rados.0.smithi045.stdout:1332: writing smithi04569269-29 from 3694592 to 4169728 tid 3 2021-07-31T01:50:05.627 INFO:tasks.rados.rados.0.smithi045.stdout:1332: writing smithi04569269-29 from 4169728 to 4595712 tid 4 2021-07-31T01:50:05.629 INFO:tasks.rados.rados.0.smithi045.stdout:1332: writing smithi04569269-29 from 4595712 to 4882432 tid 5 2021-07-31T01:50:05.630 INFO:tasks.rados.rados.0.smithi045.stdout:1333: copy_from oid 14 from oid 8 current snap is 123 2021-07-31T01:50:05.630 INFO:tasks.rados.rados.0.smithi045.stdout:1332: finishing write tid 1 to smithi04569269-29 2021-07-31T01:50:05.630 INFO:tasks.rados.rados.0.smithi045.stdout:1332: finishing write tid 2 to smithi04569269-29 2021-07-31T01:50:05.630 INFO:tasks.rados.rados.0.smithi045.stdout:1332: finishing write tid 3 to smithi04569269-29 2021-07-31T01:50:05.631 INFO:tasks.rados.rados.0.smithi045.stdout:1334: setattr oid 25 current snap is 123 2021-07-31T01:50:05.631 INFO:tasks.rados.rados.0.smithi045.stdout:1335: rollback oid 45 current snap is 123 2021-07-31T01:50:05.632 INFO:tasks.rados.rados.0.smithi045.stdout:1335: done (3 left) 2021-07-31T01:50:05.632 INFO:tasks.rados.rados.0.smithi045.stdout:1336: rollback oid 42 current snap is 123 2021-07-31T01:50:05.632 INFO:tasks.rados.rados.0.smithi045.stdout:1336: done (3 left) 2021-07-31T01:50:05.632 INFO:tasks.rados.rados.0.smithi045.stdout:1337: snap_create 2021-07-31T01:50:05.632 INFO:tasks.rados.rados.0.smithi045.stdout:1332: finishing write tid 4 to smithi04569269-29 2021-07-31T01:50:05.633 INFO:tasks.rados.rados.0.smithi045.stdout:1332: finishing write tid 5 to smithi04569269-29 2021-07-31T01:50:05.634 INFO:tasks.rados.rados.0.smithi045.stdout:1332: finishing write tid 6 to smithi04569269-29 2021-07-31T01:50:05.634 INFO:tasks.rados.rados.0.smithi045.stdout:1332: finishing write tid 7 to smithi04569269-29 2021-07-31T01:50:05.634 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 29 v 194 (ObjNum 410 snap 123 seq_num 410) dirty exists 2021-07-31T01:50:05.634 INFO:tasks.rados.rados.0.smithi045.stdout:1332: left oid 29 (ObjNum 410 snap 123 seq_num 410) 2021-07-31T01:50:05.637 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 25 v 268 (ObjNum 1953066355 snap 875587944 seq_num 842610229) dirty exists 2021-07-31T01:50:05.671 INFO:tasks.rados.rados.0.smithi045.stdout:1333: finishing copy_from to smithi04569269-14 2021-07-31T01:50:05.672 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 14 v 913 (ObjNum 391 snap 117 seq_num 391) dirty exists 2021-07-31T01:50:05.672 INFO:tasks.rados.rados.0.smithi045.stdout:1333: finishing copy_from racing read to smithi04569269-14 2021-07-31T01:50:06.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:05 smithi154 conmon[34553]: cluster 2021-07-31T01:50:04.598474+0000 mon.a (mon.0) 949 : cluster [DBG] osdmap e305: 8 total, 8 up, 8 in 2021-07-31T01:50:06.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:05 smithi045 conmon[33655]: cluster 2021-07-31T01:50:04.598474+0000 mon.a (mon.0) 2021-07-31T01:50:06.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:05 smithi045 conmon[33655]: 949 : cluster [DBG] osdmap e305: 8 total, 8 up, 8 in 2021-07-31T01:50:06.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:05 smithi045 conmon[39718]: cluster 2021-07-31T01:50:04.598474+0000 mon.a 2021-07-31T01:50:06.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:05 smithi045 conmon[39718]: (mon.0) 949 : cluster [DBG] osdmap e305: 8 total, 8 up, 8 in 2021-07-31T01:50:06.533 INFO:tasks.rados.rados.0.smithi045.stdout:1332: done (3 left) 2021-07-31T01:50:06.533 INFO:tasks.rados.rados.0.smithi045.stdout:1333: done (2 left) 2021-07-31T01:50:06.534 INFO:tasks.rados.rados.0.smithi045.stdout:1334: done (1 left) 2021-07-31T01:50:06.534 INFO:tasks.rados.rados.0.smithi045.stdout:1337: done (0 left) 2021-07-31T01:50:06.534 INFO:tasks.rados.rados.0.smithi045.stdout:1338: delete oid 14 current snap is 124 2021-07-31T01:50:06.541 INFO:tasks.rados.rados.0.smithi045.stdout:1338: done (0 left) 2021-07-31T01:50:06.541 INFO:tasks.rados.rados.0.smithi045.stdout:1339: read oid 24 snap -1 2021-07-31T01:50:06.541 INFO:tasks.rados.rados.0.smithi045.stdout:1339: expect (ObjNum 352 snap 106 seq_num 352) 2021-07-31T01:50:06.542 INFO:tasks.rados.rados.0.smithi045.stdout:1340: read oid 4 snap 123 2021-07-31T01:50:06.542 INFO:tasks.rados.rados.0.smithi045.stdout:1340: expect (ObjNum 375 snap 109 seq_num 375) 2021-07-31T01:50:06.542 INFO:tasks.rados.rados.0.smithi045.stdout:1341: read oid 26 snap 123 2021-07-31T01:50:06.542 INFO:tasks.rados.rados.0.smithi045.stdout:1341: expect deleted 2021-07-31T01:50:06.543 INFO:tasks.rados.rados.0.smithi045.stdout:1342: copy_from oid 2 from oid 27 current snap is 124 2021-07-31T01:50:06.543 INFO:tasks.rados.rados.0.smithi045.stdout:1343: copy_from oid 8 from oid 19 current snap is 124 2021-07-31T01:50:06.543 INFO:tasks.rados.rados.0.smithi045.stdout:1344: delete oid 49 current snap is 124 2021-07-31T01:50:06.550 INFO:tasks.rados.rados.0.smithi045.stdout:1340: expect (ObjNum 375 snap 109 seq_num 375) 2021-07-31T01:50:06.591 INFO:tasks.rados.rados.0.smithi045.stdout:1339: expect (ObjNum 352 snap 106 seq_num 352) 2021-07-31T01:50:06.622 INFO:tasks.rados.rados.0.smithi045.stdout:1342: finishing copy_from to smithi04569269-2 2021-07-31T01:50:06.623 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 2 v 372 (ObjNum 374 snap 109 seq_num 374) dirty exists 2021-07-31T01:50:06.623 INFO:tasks.rados.rados.0.smithi045.stdout:1342: finishing copy_from racing read to smithi04569269-2 2021-07-31T01:50:06.623 INFO:tasks.rados.rados.0.smithi045.stdout:1343: finishing copy_from to smithi04569269-8 2021-07-31T01:50:06.624 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 8 v 315 (ObjNum 353 snap 106 seq_num 353) dirty exists 2021-07-31T01:50:06.624 INFO:tasks.rados.rados.0.smithi045.stdout:1343: finishing copy_from racing read to smithi04569269-8 2021-07-31T01:50:06.624 INFO:tasks.rados.rados.0.smithi045.stdout:1339: done (5 left) 2021-07-31T01:50:06.624 INFO:tasks.rados.rados.0.smithi045.stdout:1340: done (4 left) 2021-07-31T01:50:06.625 INFO:tasks.rados.rados.0.smithi045.stdout:1341: done (3 left) 2021-07-31T01:50:06.625 INFO:tasks.rados.rados.0.smithi045.stdout:1342: done (2 left) 2021-07-31T01:50:06.626 INFO:tasks.rados.rados.0.smithi045.stdout:1343: done (1 left) 2021-07-31T01:50:06.626 INFO:tasks.rados.rados.0.smithi045.stdout:1344: done (0 left) 2021-07-31T01:50:06.626 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 20 current snap is 124 2021-07-31T01:50:06.626 INFO:tasks.rados.rados.0.smithi045.stdout:1345: seq_num 413 ranges {0=524288,524288=516096,1040384=696320,1736704=393216} 2021-07-31T01:50:06.627 INFO:tasks.rados.rados.0.smithi045.stdout:1345: writing smithi04569269-20 from 0 to 524288 tid 1 2021-07-31T01:50:06.628 INFO:tasks.rados.rados.0.smithi045.stdout:1345: writing smithi04569269-20 from 524288 to 1040384 tid 2 2021-07-31T01:50:06.632 INFO:tasks.rados.rados.0.smithi045.stdout:1345: writing smithi04569269-20 from 1040384 to 1736704 tid 3 2021-07-31T01:50:06.634 INFO:tasks.rados.rados.0.smithi045.stdout:1345: writing smithi04569269-20 from 1736704 to 2129920 tid 4 2021-07-31T01:50:06.635 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 23 current snap is 124 2021-07-31T01:50:06.635 INFO:tasks.rados.rados.0.smithi045.stdout:1345: finishing write tid 1 to smithi04569269-20 2021-07-31T01:50:06.635 INFO:tasks.rados.rados.0.smithi045.stdout:1346: seq_num 414 ranges {1605632=540672,2146304=802816,2949120=458752,3407872=344064} 2021-07-31T01:50:06.646 INFO:tasks.rados.rados.0.smithi045.stdout:1346: writing smithi04569269-23 from 1605632 to 2146304 tid 1 2021-07-31T01:50:06.650 INFO:tasks.rados.rados.0.smithi045.stdout:1346: writing smithi04569269-23 from 2146304 to 2949120 tid 2 2021-07-31T01:50:06.653 INFO:tasks.rados.rados.0.smithi045.stdout:1346: writing smithi04569269-23 from 2949120 to 3407872 tid 3 2021-07-31T01:50:06.656 INFO:tasks.rados.rados.0.smithi045.stdout:1346: writing smithi04569269-23 from 3407872 to 3751936 tid 4 2021-07-31T01:50:06.656 INFO:tasks.rados.rados.0.smithi045.stdout:1347: copy_from oid 35 from oid 5 current snap is 124 2021-07-31T01:50:06.656 INFO:tasks.rados.rados.0.smithi045.stdout:1345: finishing write tid 2 to smithi04569269-20 2021-07-31T01:50:06.656 INFO:tasks.rados.rados.0.smithi045.stdout:1345: finishing write tid 3 to smithi04569269-20 2021-07-31T01:50:06.657 INFO:tasks.rados.rados.0.smithi045.stdout:1345: finishing write tid 4 to smithi04569269-20 2021-07-31T01:50:06.657 INFO:tasks.rados.rados.0.smithi045.stdout:1345: finishing write tid 5 to smithi04569269-20 2021-07-31T01:50:06.657 INFO:tasks.rados.rados.0.smithi045.stdout:1345: finishing write tid 6 to smithi04569269-20 2021-07-31T01:50:06.658 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 20 v 938 (ObjNum 413 snap 124 seq_num 413) dirty exists 2021-07-31T01:50:06.658 INFO:tasks.rados.rados.0.smithi045.stdout:1345: left oid 20 (ObjNum 413 snap 124 seq_num 413) 2021-07-31T01:50:06.658 INFO:tasks.rados.rados.0.smithi045.stdout:1346: finishing write tid 1 to smithi04569269-23 2021-07-31T01:50:06.658 INFO:tasks.rados.rados.0.smithi045.stdout:1346: finishing write tid 2 to smithi04569269-23 2021-07-31T01:50:06.658 INFO:tasks.rados.rados.0.smithi045.stdout:1345: done (2 left) 2021-07-31T01:50:06.659 INFO:tasks.rados.rados.0.smithi045.stdout:1348: read oid 14 snap -1 2021-07-31T01:50:06.659 INFO:tasks.rados.rados.0.smithi045.stdout:1348: expect deleted 2021-07-31T01:50:06.659 INFO:tasks.rados.rados.0.smithi045.stdout:1349: snap_remove snap 123 2021-07-31T01:50:07.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:06 smithi154 conmon[34553]: cluster 2021-07-31T01:50:05.600625+0000 mon.a (mon.0) 950 : cluster 2021-07-31T01:50:07.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:06 smithi154 conmon[34553]: [DBG] osdmap e306: 8 total, 8 up, 8 in 2021-07-31T01:50:07.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:06 smithi154 conmon[34553]: cluster 2021-07-31T01:50:05.831617+0000 mgr.x (mgr.24542) 121 : cluster [DBG] pgmap v239: 25 pgs: 25 active+clean; 114 MiB data, 849 MiB used, 714 GiB / 715 GiB avail; 17 MiB/s rd, 5.7 MiB/s wr, 30 op/s 2021-07-31T01:50:07.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:06 smithi154 conmon[34553]: cluster 2021- 2021-07-31T01:50:07.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:06 smithi154 conmon[34553]: 07-31T01:50:06.532695+0000 mon.a (mon.0) 951 : cluster [DBG] osdmap e307: 8 total, 8 up, 8 in 2021-07-31T01:50:07.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:06 smithi045 conmon[33655]: cluster 2021-07-31T01:50:05. 2021-07-31T01:50:07.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:06 smithi045 conmon[33655]: 600625+0000 mon.a (mon.0) 950 : cluster [DBG] osdmap e306: 8 total, 8 up, 8 in 2021-07-31T01:50:07.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:06 smithi045 conmon[33655]: cluster 2021-07-31T01:50: 2021-07-31T01:50:07.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:06 smithi045 conmon[33655]: 05.831617+0000 mgr.x (mgr.24542) 121 : 2021-07-31T01:50:07.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:06 smithi045 conmon[33655]: cluster [DBG] pgmap v239: 25 pgs: 25 active+clean; 114 MiB data, 849 MiB used, 714 GiB / 715 GiB avail; 17 MiB/s rd, 5.7 MiB/s wr, 30 op/s 2021-07-31T01:50:07.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:06 smithi045 conmon[33655]: cluster 2021-07-31T01:50:07.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:06 smithi045 conmon[33655]: 2021-07-31T01 2021-07-31T01:50:07.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:06 smithi045 conmon[33655]: :50:06.532695+0000 2021-07-31T01:50:07.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:06 smithi045 conmon[33655]: mon.a (mon.0) 951 : 2021-07-31T01:50:07.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:06 smithi045 conmon[33655]: cluster [DBG] osdmap e307: 8 total, 8 up, 8 in 2021-07-31T01:50:07.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:06 smithi045 conmon[39718]: cluster 2021-07-31T01: 2021-07-31T01:50:07.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:06 smithi045 conmon[39718]: 50:05.600625+0000 mon.a (mon.0) 950 : cluster [DBG] osdmap e306: 8 total, 8 up, 8 in 2021-07-31T01:50:07.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:06 smithi045 conmon[39718]: cluster 2021-07-31T01:50:07.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:06 smithi045 conmon[39718]: 2021-07-31T01:50:05 2021-07-31T01:50:07.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:06 smithi045 conmon[39718]: .831617+0000 mgr.x (mgr.24542 2021-07-31T01:50:07.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:06 smithi045 conmon[39718]: ) 121 : cluster [DBG] pgmap v239: 25 pgs: 25 active+clean; 114 MiB data, 849 MiB used, 714 GiB / 715 GiB avail; 17 MiB/s rd, 5.7 MiB/s wr, 30 op/s 2021-07-31T01:50:07.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:06 smithi045 conmon[39718]: 2021-07-31T01:50:07.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:06 smithi045 conmon[39718]: cluster 2021-07-31 2021-07-31T01:50:07.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:06 smithi045 conmon[39718]: T01:50:06.532695 2021-07-31T01:50:07.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:06 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2021-07-31T01:50:07.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:06 smithi045 conmon[39718]: 951 : cluster [DBG] osdmap e307: 8 total, 8 up, 8 in 2021-07-31T01:50:07.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:06 smithi045 conmon[39718]: 2021-07-31T01:50:07.614 INFO:tasks.rados.rados.0.smithi045.stdout:1349: done (3 left) 2021-07-31T01:50:07.615 INFO:tasks.rados.rados.0.smithi045.stdout:1350: read oid 26 snap -1 2021-07-31T01:50:07.615 INFO:tasks.rados.rados.0.smithi045.stdout:1350: expect deleted 2021-07-31T01:50:07.616 INFO:tasks.rados.rados.0.smithi045.stdout:1351: read oid 2 snap -1 2021-07-31T01:50:07.616 INFO:tasks.rados.rados.0.smithi045.stdout:1351: expect (ObjNum 374 snap 109 seq_num 374) 2021-07-31T01:50:07.616 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 34 current snap is 124 2021-07-31T01:50:07.617 INFO:tasks.rados.rados.0.smithi045.stdout:1352: seq_num 415 ranges {3694592=557056,4251648=688128,4939776=663552,5603328=253952} 2021-07-31T01:50:07.637 INFO:tasks.rados.rados.0.smithi045.stdout:1352: writing smithi04569269-34 from 3694592 to 4251648 tid 1 2021-07-31T01:50:07.641 INFO:tasks.rados.rados.0.smithi045.stdout:1352: writing smithi04569269-34 from 4251648 to 4939776 tid 2 2021-07-31T01:50:07.645 INFO:tasks.rados.rados.0.smithi045.stdout:1352: writing smithi04569269-34 from 4939776 to 5603328 tid 3 2021-07-31T01:50:07.648 INFO:tasks.rados.rados.0.smithi045.stdout:1352: writing smithi04569269-34 from 5603328 to 5857280 tid 4 2021-07-31T01:50:07.648 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 32 current snap is 124 2021-07-31T01:50:07.648 INFO:tasks.rados.rados.0.smithi045.stdout:1347: finishing copy_from to smithi04569269-35 2021-07-31T01:50:07.649 INFO:tasks.rados.rados.0.smithi045.stdout:1347: got expected ENOENT (src dne) 2021-07-31T01:50:07.649 INFO:tasks.rados.rados.0.smithi045.stdout:1347: finishing copy_from racing read to smithi04569269-35 2021-07-31T01:50:07.650 INFO:tasks.rados.rados.0.smithi045.stdout:1346: finishing write tid 3 to smithi04569269-23 2021-07-31T01:50:07.651 INFO:tasks.rados.rados.0.smithi045.stdout:1346: finishing write tid 4 to smithi04569269-23 2021-07-31T01:50:07.651 INFO:tasks.rados.rados.0.smithi045.stdout:1346: finishing write tid 5 to smithi04569269-23 2021-07-31T01:50:07.651 INFO:tasks.rados.rados.0.smithi045.stdout:1346: finishing write tid 6 to smithi04569269-23 2021-07-31T01:50:07.652 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 23 v 578 (ObjNum 414 snap 124 seq_num 414) dirty exists 2021-07-31T01:50:07.652 INFO:tasks.rados.rados.0.smithi045.stdout:1346: left oid 23 (ObjNum 414 snap 124 seq_num 414) 2021-07-31T01:50:07.653 INFO:tasks.rados.rados.0.smithi045.stdout:1351: expect (ObjNum 374 snap 109 seq_num 374) 2021-07-31T01:50:07.686 INFO:tasks.rados.rados.0.smithi045.stdout:1352: finishing write tid 1 to smithi04569269-34 2021-07-31T01:50:07.686 INFO:tasks.rados.rados.0.smithi045.stdout:1352: finishing write tid 2 to smithi04569269-34 2021-07-31T01:50:07.686 INFO:tasks.rados.rados.0.smithi045.stdout:1352: finishing write tid 3 to smithi04569269-34 2021-07-31T01:50:07.687 INFO:tasks.rados.rados.0.smithi045.stdout:1352: finishing write tid 4 to smithi04569269-34 2021-07-31T01:50:07.687 INFO:tasks.rados.rados.0.smithi045.stdout:1352: finishing write tid 5 to smithi04569269-34 2021-07-31T01:50:07.687 INFO:tasks.rados.rados.0.smithi045.stdout:1352: finishing write tid 6 to smithi04569269-34 2021-07-31T01:50:07.687 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 34 v 894 (ObjNum 415 snap 124 seq_num 415) dirty exists 2021-07-31T01:50:07.687 INFO:tasks.rados.rados.0.smithi045.stdout:1352: left oid 34 (ObjNum 415 snap 124 seq_num 415) 2021-07-31T01:50:07.688 INFO:tasks.rados.rados.0.smithi045.stdout:1353: seq_num 416 ranges {1474560=573440,2048000=573440,2621440=475136,3096576=557056} 2021-07-31T01:50:07.696 INFO:tasks.rados.rados.0.smithi045.stdout:1353: writing smithi04569269-32 from 1474560 to 2048000 tid 1 2021-07-31T01:50:07.700 INFO:tasks.rados.rados.0.smithi045.stdout:1353: writing smithi04569269-32 from 2048000 to 2621440 tid 2 2021-07-31T01:50:07.703 INFO:tasks.rados.rados.0.smithi045.stdout:1353: writing smithi04569269-32 from 2621440 to 3096576 tid 3 2021-07-31T01:50:07.707 INFO:tasks.rados.rados.0.smithi045.stdout:1353: writing smithi04569269-32 from 3096576 to 3653632 tid 4 2021-07-31T01:50:07.707 INFO:tasks.rados.rados.0.smithi045.stdout:1346: done (6 left) 2021-07-31T01:50:07.707 INFO:tasks.rados.rados.0.smithi045.stdout:1347: done (5 left) 2021-07-31T01:50:07.708 INFO:tasks.rados.rados.0.smithi045.stdout:1348: done (4 left) 2021-07-31T01:50:07.708 INFO:tasks.rados.rados.0.smithi045.stdout:1350: done (3 left) 2021-07-31T01:50:07.708 INFO:tasks.rados.rados.0.smithi045.stdout:1351: done (2 left) 2021-07-31T01:50:07.708 INFO:tasks.rados.rados.0.smithi045.stdout:1352: done (1 left) 2021-07-31T01:50:07.708 INFO:tasks.rados.rados.0.smithi045.stdout:1354: rollback oid 9 current snap is 124 2021-07-31T01:50:07.709 INFO:tasks.rados.rados.0.smithi045.stdout:1354: done (1 left) 2021-07-31T01:50:07.709 INFO:tasks.rados.rados.0.smithi045.stdout:1355: read oid 38 snap -1 2021-07-31T01:50:07.709 INFO:tasks.rados.rados.0.smithi045.stdout:1355: expect (ObjNum 407 snap 122 seq_num 407) 2021-07-31T01:50:07.709 INFO:tasks.rados.rados.0.smithi045.stdout:1356: read oid 14 snap -1 2021-07-31T01:50:07.710 INFO:tasks.rados.rados.0.smithi045.stdout:1356: expect deleted 2021-07-31T01:50:07.710 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 30 current snap is 124 2021-07-31T01:50:07.710 INFO:tasks.rados.rados.0.smithi045.stdout:1353: finishing write tid 1 to smithi04569269-32 2021-07-31T01:50:07.710 INFO:tasks.rados.rados.0.smithi045.stdout:1357: seq_num 417 ranges {2973696=589824,3563520=458752,4022272=679936,4702208=466944} 2021-07-31T01:50:07.726 INFO:tasks.rados.rados.0.smithi045.stdout:1357: writing smithi04569269-30 from 2973696 to 3563520 tid 1 2021-07-31T01:50:07.729 INFO:tasks.rados.rados.0.smithi045.stdout:1357: writing smithi04569269-30 from 3563520 to 4022272 tid 2 2021-07-31T01:50:07.734 INFO:tasks.rados.rados.0.smithi045.stdout:1357: writing smithi04569269-30 from 4022272 to 4702208 tid 3 2021-07-31T01:50:07.738 INFO:tasks.rados.rados.0.smithi045.stdout:1357: writing smithi04569269-30 from 4702208 to 5169152 tid 4 2021-07-31T01:50:07.738 INFO:tasks.rados.rados.0.smithi045.stdout:1358: delete oid 11 current snap is 124 2021-07-31T01:50:07.739 INFO:tasks.rados.rados.0.smithi045.stdout:1353: finishing write tid 2 to smithi04569269-32 2021-07-31T01:50:07.739 INFO:tasks.rados.rados.0.smithi045.stdout:1353: finishing write tid 3 to smithi04569269-32 2021-07-31T01:50:07.740 INFO:tasks.rados.rados.0.smithi045.stdout:1353: finishing write tid 4 to smithi04569269-32 2021-07-31T01:50:07.740 INFO:tasks.rados.rados.0.smithi045.stdout:1353: finishing write tid 5 to smithi04569269-32 2021-07-31T01:50:07.740 INFO:tasks.rados.rados.0.smithi045.stdout:1353: finishing write tid 6 to smithi04569269-32 2021-07-31T01:50:07.741 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 32 v 378 (ObjNum 416 snap 124 seq_num 416) dirty exists 2021-07-31T01:50:07.741 INFO:tasks.rados.rados.0.smithi045.stdout:1353: left oid 32 (ObjNum 416 snap 124 seq_num 416) 2021-07-31T01:50:07.742 INFO:tasks.rados.rados.0.smithi045.stdout:1357: finishing write tid 1 to smithi04569269-30 2021-07-31T01:50:07.742 INFO:tasks.rados.rados.0.smithi045.stdout:1357: finishing write tid 2 to smithi04569269-30 2021-07-31T01:50:07.743 INFO:tasks.rados.rados.0.smithi045.stdout:1357: finishing write tid 3 to smithi04569269-30 2021-07-31T01:50:07.744 INFO:tasks.rados.rados.0.smithi045.stdout:1357: finishing write tid 4 to smithi04569269-30 2021-07-31T01:50:07.745 INFO:tasks.rados.rados.0.smithi045.stdout:1357: finishing write tid 5 to smithi04569269-30 2021-07-31T01:50:07.745 INFO:tasks.rados.rados.0.smithi045.stdout:1357: finishing write tid 6 to smithi04569269-30 2021-07-31T01:50:07.745 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 30 v 493 (ObjNum 417 snap 124 seq_num 417) dirty exists 2021-07-31T01:50:07.746 INFO:tasks.rados.rados.0.smithi045.stdout:1357: left oid 30 (ObjNum 417 snap 124 seq_num 417) 2021-07-31T01:50:07.746 INFO:tasks.rados.rados.0.smithi045.stdout:1355: expect (ObjNum 407 snap 122 seq_num 407) 2021-07-31T01:50:08.003 INFO:journalctl@ceph.osd.6.smithi154.stdout:Jul 31 01:50:07 smithi154 conmon[45576]: debug 2021-07-31T01:50:07.613+0000 7f47c10a5700 -1 osd.6 pg_epoch: 308 pg[3.as0( v 307'396 (0'0,307'396] local-lis/les=56/57 n=4 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [6,4,1]p6(0) r=0 lpr=56 crt=307'396 lcod 306'394 mlcod 306'394 active+clean+snaptrim trimq=[f8~1] ps=[f6~1]] removing snap head 2021-07-31T01:50:08.026 INFO:journalctl@ceph.osd.3.smithi045.stdout:Jul 31 01:50:07 smithi045 conmon[54013]: debug 2021-07-31T01:50:07.613+0000 7f64f88c5700 -1 osd.3 pg_epoch: 308 pg[3.bs0( v 307'938 (0'0,307'938] local-lis/les=56/57 n=8 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [3,0,4]p3(0) r=0 lpr=56 crt=307'938 lcod 307'937 mlcod 307'937 active+clean+snaptrim trimq=[f8~1] ps=[f6~1]] removing snap head 2021-07-31T01:50:08.102 INFO:tasks.rados.rados.0.smithi045.stdout:1353: done (4 left) 2021-07-31T01:50:08.103 INFO:tasks.rados.rados.0.smithi045.stdout:1355: done (3 left) 2021-07-31T01:50:08.103 INFO:tasks.rados.rados.0.smithi045.stdout:1356: done (2 left) 2021-07-31T01:50:08.104 INFO:tasks.rados.rados.0.smithi045.stdout:1357: done (1 left) 2021-07-31T01:50:08.104 INFO:tasks.rados.rados.0.smithi045.stdout:1358: done (0 left) 2021-07-31T01:50:08.104 INFO:tasks.rados.rados.0.smithi045.stdout:1359: snap_create 2021-07-31T01:50:08.619 INFO:tasks.rados.rados.0.smithi045.stdout:1359: done (0 left) 2021-07-31T01:50:08.619 INFO:tasks.rados.rados.0.smithi045.stdout:1360: rollback oid 30 current snap is 125 2021-07-31T01:50:08.619 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 30 to 124 2021-07-31T01:50:08.619 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 44 current snap is 125 2021-07-31T01:50:08.620 INFO:tasks.rados.rados.0.smithi045.stdout:1361: seq_num 419 ranges {0=622592,622592=671744,1294336=647168,1941504=286720} 2021-07-31T01:50:08.625 INFO:tasks.rados.rados.0.smithi045.stdout:1361: writing smithi04569269-44 from 0 to 622592 tid 1 2021-07-31T01:50:08.626 INFO:tasks.rados.rados.0.smithi045.stdout:1361: writing smithi04569269-44 from 622592 to 1294336 tid 2 2021-07-31T01:50:08.629 INFO:tasks.rados.rados.0.smithi045.stdout:1361: writing smithi04569269-44 from 1294336 to 1941504 tid 3 2021-07-31T01:50:08.632 INFO:tasks.rados.rados.0.smithi045.stdout:1361: writing smithi04569269-44 from 1941504 to 2228224 tid 4 2021-07-31T01:50:08.632 INFO:tasks.rados.rados.0.smithi045.stdout:1362: copy_from oid 19 from oid 45 current snap is 125 2021-07-31T01:50:08.632 INFO:tasks.rados.rados.0.smithi045.stdout:1360: finishing rollback tid 0 to smithi04569269-30 2021-07-31T01:50:08.632 INFO:tasks.rados.rados.0.smithi045.stdout:1360: finishing rollback tid 1 to smithi04569269-30 2021-07-31T01:50:08.633 INFO:tasks.rados.rados.0.smithi045.stdout:1360: finishing rollback tid 2 to smithi04569269-30 2021-07-31T01:50:08.633 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 30 v 496 (ObjNum 417 snap 124 seq_num 417) dirty exists 2021-07-31T01:50:08.633 INFO:tasks.rados.rados.0.smithi045.stdout:1361: finishing write tid 1 to smithi04569269-44 2021-07-31T01:50:08.633 INFO:tasks.rados.rados.0.smithi045.stdout:1361: finishing write tid 2 to smithi04569269-44 2021-07-31T01:50:08.634 INFO:tasks.rados.rados.0.smithi045.stdout:1360: done (2 left) 2021-07-31T01:50:08.634 INFO:tasks.rados.rados.0.smithi045.stdout:1363: read oid 36 snap -1 2021-07-31T01:50:08.634 INFO:tasks.rados.rados.0.smithi045.stdout:1363: expect (ObjNum 366 snap 108 seq_num 366) 2021-07-31T01:50:08.634 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 22 current snap is 125 2021-07-31T01:50:08.635 INFO:tasks.rados.rados.0.smithi045.stdout:1364: seq_num 420 ranges {2736128=638976,3375104=557056,3932160=450560,4382720=598016} 2021-07-31T01:50:08.650 INFO:tasks.rados.rados.0.smithi045.stdout:1364: writing smithi04569269-22 from 2736128 to 3375104 tid 1 2021-07-31T01:50:08.654 INFO:tasks.rados.rados.0.smithi045.stdout:1364: writing smithi04569269-22 from 3375104 to 3932160 tid 2 2021-07-31T01:50:08.656 INFO:tasks.rados.rados.0.smithi045.stdout:1364: writing smithi04569269-22 from 3932160 to 4382720 tid 3 2021-07-31T01:50:08.661 INFO:tasks.rados.rados.0.smithi045.stdout:1364: writing smithi04569269-22 from 4382720 to 4980736 tid 4 2021-07-31T01:50:08.661 INFO:tasks.rados.rados.0.smithi045.stdout:1365: read oid 48 snap -1 2021-07-31T01:50:08.661 INFO:tasks.rados.rados.0.smithi045.stdout:1365: expect (ObjNum 393 snap 117 seq_num 393) 2021-07-31T01:50:08.662 INFO:tasks.rados.rados.0.smithi045.stdout:1366: delete oid 20 current snap is 125 2021-07-31T01:50:08.662 INFO:tasks.rados.rados.0.smithi045.stdout:1361: finishing write tid 3 to smithi04569269-44 2021-07-31T01:50:08.662 INFO:tasks.rados.rados.0.smithi045.stdout:1361: finishing write tid 4 to smithi04569269-44 2021-07-31T01:50:08.663 INFO:tasks.rados.rados.0.smithi045.stdout:1361: finishing write tid 5 to smithi04569269-44 2021-07-31T01:50:08.663 INFO:tasks.rados.rados.0.smithi045.stdout:1361: finishing write tid 6 to smithi04569269-44 2021-07-31T01:50:08.663 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 44 v 919 (ObjNum 419 snap 125 seq_num 419) dirty exists 2021-07-31T01:50:08.663 INFO:tasks.rados.rados.0.smithi045.stdout:1361: left oid 44 (ObjNum 419 snap 125 seq_num 419) 2021-07-31T01:50:08.664 INFO:tasks.rados.rados.0.smithi045.stdout:1363: expect (ObjNum 366 snap 108 seq_num 366) 2021-07-31T01:50:08.751 INFO:tasks.rados.rados.0.smithi045.stdout:1362: finishing copy_from to smithi04569269-19 2021-07-31T01:50:08.751 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 19 v 488 (ObjNum 352 snap 106 seq_num 352) dirty exists 2021-07-31T01:50:08.751 INFO:tasks.rados.rados.0.smithi045.stdout:1362: finishing copy_from racing read to smithi04569269-19 2021-07-31T01:50:08.751 INFO:tasks.rados.rados.0.smithi045.stdout:1364: finishing write tid 1 to smithi04569269-22 2021-07-31T01:50:08.752 INFO:tasks.rados.rados.0.smithi045.stdout:1364: finishing write tid 2 to smithi04569269-22 2021-07-31T01:50:08.752 INFO:tasks.rados.rados.0.smithi045.stdout:1364: finishing write tid 3 to smithi04569269-22 2021-07-31T01:50:08.752 INFO:tasks.rados.rados.0.smithi045.stdout:1361: done (5 left) 2021-07-31T01:50:08.752 INFO:tasks.rados.rados.0.smithi045.stdout:1362: done (4 left) 2021-07-31T01:50:08.753 INFO:tasks.rados.rados.0.smithi045.stdout:1363: done (3 left) 2021-07-31T01:50:08.753 INFO:tasks.rados.rados.0.smithi045.stdout:1366: done (2 left) 2021-07-31T01:50:08.753 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 44 current snap is 125 2021-07-31T01:50:08.753 INFO:tasks.rados.rados.0.smithi045.stdout:1364: finishing write tid 4 to smithi04569269-22 2021-07-31T01:50:08.753 INFO:tasks.rados.rados.0.smithi045.stdout:1364: finishing write tid 5 to smithi04569269-22 2021-07-31T01:50:08.754 INFO:tasks.rados.rados.0.smithi045.stdout:1364: finishing write tid 6 to smithi04569269-22 2021-07-31T01:50:08.754 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 22 v 493 (ObjNum 420 snap 125 seq_num 420) dirty exists 2021-07-31T01:50:08.754 INFO:tasks.rados.rados.0.smithi045.stdout:1364: left oid 22 (ObjNum 420 snap 125 seq_num 420) 2021-07-31T01:50:08.754 INFO:tasks.rados.rados.0.smithi045.stdout:1365: expect (ObjNum 393 snap 117 seq_num 393) 2021-07-31T01:50:08.987 INFO:tasks.rados.rados.0.smithi045.stdout:1367: seq_num 422 ranges {2228224=671744,2899968=729088,3629056=417792,4046848=458752} 2021-07-31T01:50:09.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:08 smithi154 conmon[34553]: cluster 2021-07-31T01:50:07.612402+0000 mon.a (mon.0) 952 : cluster [DBG] osdmap e308: 8 total, 8 up, 8 in 2021-07-31T01:50:09.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:08 smithi154 conmon[34553]: cluster 2021-07-31T01:50:07.832019+0000 mgr.x (mgr.24542) 122 : cluster [DBG] pgmap v242: 25 pgs: 25 active+clean; 114 MiB data, 849 MiB used, 714 GiB / 715 GiB avail 2021-07-31T01:50:09.004 INFO:tasks.rados.rados.0.smithi045.stdout:1367: writing smithi04569269-44 from 2228224 to 2899968 tid 1 2021-07-31T01:50:09.007 INFO:tasks.rados.rados.0.smithi045.stdout:1367: writing smithi04569269-44 from 2899968 to 3629056 tid 2 2021-07-31T01:50:09.009 INFO:tasks.rados.rados.0.smithi045.stdout:1367: writing smithi04569269-44 from 3629056 to 4046848 tid 3 2021-07-31T01:50:09.012 INFO:tasks.rados.rados.0.smithi045.stdout:1367: writing smithi04569269-44 from 4046848 to 4505600 tid 4 2021-07-31T01:50:09.013 INFO:tasks.rados.rados.0.smithi045.stdout:1364: done (2 left) 2021-07-31T01:50:09.013 INFO:tasks.rados.rados.0.smithi045.stdout:1365: done (1 left) 2021-07-31T01:50:09.013 INFO:tasks.rados.rados.0.smithi045.stdout:1368: snap_create 2021-07-31T01:50:09.014 INFO:tasks.rados.rados.0.smithi045.stdout:1367: finishing write tid 1 to smithi04569269-44 2021-07-31T01:50:09.014 INFO:tasks.rados.rados.0.smithi045.stdout:1367: finishing write tid 2 to smithi04569269-44 2021-07-31T01:50:09.014 INFO:tasks.rados.rados.0.smithi045.stdout:1367: finishing write tid 3 to smithi04569269-44 2021-07-31T01:50:09.015 INFO:tasks.rados.rados.0.smithi045.stdout:1367: finishing write tid 4 to smithi04569269-44 2021-07-31T01:50:09.015 INFO:tasks.rados.rados.0.smithi045.stdout:1367: finishing write tid 5 to smithi04569269-44 2021-07-31T01:50:09.016 INFO:tasks.rados.rados.0.smithi045.stdout:1367: finishing write tid 6 to smithi04569269-44 2021-07-31T01:50:09.017 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 44 v 924 (ObjNum 422 snap 125 seq_num 422) dirty exists 2021-07-31T01:50:09.017 INFO:tasks.rados.rados.0.smithi045.stdout:1367: left oid 44 (ObjNum 422 snap 125 seq_num 422) 2021-07-31T01:50:09.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:08 smithi045 conmon[33655]: cluster 2021-07-31T01:50:07.612402+0000 mon.a (mon.0) 952 : cluster [DBG] 2021-07-31T01:50:09.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:08 smithi045 conmon[33655]: osdmap e308: 8 total, 8 up, 8 in 2021-07-31T01:50:09.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:08 smithi045 conmon[33655]: cluster 2021-07-31T01:50:07.832019+0000 mgr.x (mgr.24542) 122 : cluster [DBG] pgmap v242: 25 pgs: 25 active+clean; 114 MiB data, 849 MiB used, 714 GiB / 715 GiB avail 2021-07-31T01:50:09.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:08 smithi045 conmon[39718]: cluster 2021-07-31T01:50:07.612402+0000 mon.a (mon.0) 952 : cluster 2021-07-31T01:50:09.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:08 smithi045 conmon[39718]: [DBG] osdmap e308: 8 total, 8 up, 8 in 2021-07-31T01:50:09.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:08 smithi045 conmon[39718]: cluster 2021-07-31T01 2021-07-31T01:50:09.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:08 smithi045 conmon[39718]: :50:07.832019+0000 mgr.x (mgr.24542) 122 : cluster [DBG] pgmap v242: 25 pgs: 25 active+clean; 114 MiB data, 849 MiB used, 714 GiB / 715 GiB avail 2021-07-31T01:50:09.620 INFO:tasks.rados.rados.0.smithi045.stdout:1367: done (1 left) 2021-07-31T01:50:09.620 INFO:tasks.rados.rados.0.smithi045.stdout:1368: done (0 left) 2021-07-31T01:50:09.620 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 9 current snap is 126 2021-07-31T01:50:09.621 INFO:tasks.rados.rados.0.smithi045.stdout:1369: seq_num 423 ranges {2547712=688128,3235840=614400,3850240=622592,4472832=368640} 2021-07-31T01:50:09.637 INFO:tasks.rados.rados.0.smithi045.stdout:1369: writing smithi04569269-9 from 2547712 to 3235840 tid 1 2021-07-31T01:50:09.641 INFO:tasks.rados.rados.0.smithi045.stdout:1369: writing smithi04569269-9 from 3235840 to 3850240 tid 2 2021-07-31T01:50:09.645 INFO:tasks.rados.rados.0.smithi045.stdout:1369: writing smithi04569269-9 from 3850240 to 4472832 tid 3 2021-07-31T01:50:09.647 INFO:tasks.rados.rados.0.smithi045.stdout:1369: writing smithi04569269-9 from 4472832 to 4841472 tid 4 2021-07-31T01:50:09.647 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 10 current snap is 126 2021-07-31T01:50:09.648 INFO:tasks.rados.rados.0.smithi045.stdout:1369: finishing write tid 1 to smithi04569269-9 2021-07-31T01:50:09.648 INFO:tasks.rados.rados.0.smithi045.stdout:1370: seq_num 424 ranges {0=704512,704512=491520,1196032=434176,1630208=679936} 2021-07-31T01:50:09.651 INFO:tasks.rados.rados.0.smithi045.stdout:1370: writing smithi04569269-10 from 0 to 704512 tid 1 2021-07-31T01:50:09.654 INFO:tasks.rados.rados.0.smithi045.stdout:1370: writing smithi04569269-10 from 704512 to 1196032 tid 2 2021-07-31T01:50:09.656 INFO:tasks.rados.rados.0.smithi045.stdout:1370: writing smithi04569269-10 from 1196032 to 1630208 tid 3 2021-07-31T01:50:09.661 INFO:tasks.rados.rados.0.smithi045.stdout:1370: writing smithi04569269-10 from 1630208 to 2310144 tid 4 2021-07-31T01:50:09.662 INFO:tasks.rados.rados.0.smithi045.stdout:1371: copy_from oid 6 from oid 7 current snap is 126 2021-07-31T01:50:09.662 INFO:tasks.rados.rados.0.smithi045.stdout:1369: finishing write tid 2 to smithi04569269-9 2021-07-31T01:50:09.662 INFO:tasks.rados.rados.0.smithi045.stdout:1369: finishing write tid 3 to smithi04569269-9 2021-07-31T01:50:09.662 INFO:tasks.rados.rados.0.smithi045.stdout:1369: finishing write tid 4 to smithi04569269-9 2021-07-31T01:50:09.662 INFO:tasks.rados.rados.0.smithi045.stdout:1369: finishing write tid 5 to smithi04569269-9 2021-07-31T01:50:09.663 INFO:tasks.rados.rados.0.smithi045.stdout:1369: finishing write tid 6 to smithi04569269-9 2021-07-31T01:50:09.663 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 9 v 377 (ObjNum 423 snap 126 seq_num 423) dirty exists 2021-07-31T01:50:09.663 INFO:tasks.rados.rados.0.smithi045.stdout:1369: left oid 9 (ObjNum 423 snap 126 seq_num 423) 2021-07-31T01:50:09.663 INFO:tasks.rados.rados.0.smithi045.stdout:1370: finishing write tid 1 to smithi04569269-10 2021-07-31T01:50:09.664 INFO:tasks.rados.rados.0.smithi045.stdout:1370: finishing write tid 2 to smithi04569269-10 2021-07-31T01:50:09.664 INFO:tasks.rados.rados.0.smithi045.stdout:1370: finishing write tid 3 to smithi04569269-10 2021-07-31T01:50:09.664 INFO:tasks.rados.rados.0.smithi045.stdout:1369: done (2 left) 2021-07-31T01:50:09.664 INFO:tasks.rados.rados.0.smithi045.stdout:1372: rmattr oid 50 current snap is 126 2021-07-31T01:50:09.664 INFO:tasks.rados.rados.0.smithi045.stdout:1372: done (2 left) 2021-07-31T01:50:09.665 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 38 current snap is 126 2021-07-31T01:50:09.665 INFO:tasks.rados.rados.0.smithi045.stdout:1373: seq_num 425 ranges {7323648=720896,8044544=778240,8822784=638976,9461760=188416} 2021-07-31T01:50:09.705 INFO:tasks.rados.rados.0.smithi045.stdout:1373: writing smithi04569269-38 from 7323648 to 8044544 tid 1 2021-07-31T01:50:09.709 INFO:tasks.rados.rados.0.smithi045.stdout:1373: writing smithi04569269-38 from 8044544 to 8822784 tid 2 2021-07-31T01:50:09.712 INFO:tasks.rados.rados.0.smithi045.stdout:1373: writing smithi04569269-38 from 8822784 to 9461760 tid 3 2021-07-31T01:50:09.714 INFO:tasks.rados.rados.0.smithi045.stdout:1373: writing smithi04569269-38 from 9461760 to 9650176 tid 4 2021-07-31T01:50:09.714 INFO:tasks.rados.rados.0.smithi045.stdout:1374: delete oid 18 current snap is 126 2021-07-31T01:50:09.715 INFO:tasks.rados.rados.0.smithi045.stdout:1370: finishing write tid 4 to smithi04569269-10 2021-07-31T01:50:09.715 INFO:tasks.rados.rados.0.smithi045.stdout:1370: finishing write tid 5 to smithi04569269-10 2021-07-31T01:50:09.715 INFO:tasks.rados.rados.0.smithi045.stdout:1370: finishing write tid 6 to smithi04569269-10 2021-07-31T01:50:09.715 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 10 v 538 (ObjNum 424 snap 126 seq_num 424) dirty exists 2021-07-31T01:50:09.715 INFO:tasks.rados.rados.0.smithi045.stdout:1370: left oid 10 (ObjNum 424 snap 126 seq_num 424) 2021-07-31T01:50:09.716 INFO:tasks.rados.rados.0.smithi045.stdout:1371: finishing copy_from to smithi04569269-6 2021-07-31T01:50:09.716 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 6 v 515 (ObjNum 360 snap 108 seq_num 360) dirty exists 2021-07-31T01:50:09.716 INFO:tasks.rados.rados.0.smithi045.stdout:1371: finishing copy_from racing read to smithi04569269-6 2021-07-31T01:50:09.716 INFO:tasks.rados.rados.0.smithi045.stdout:1373: finishing write tid 1 to smithi04569269-38 2021-07-31T01:50:09.717 INFO:tasks.rados.rados.0.smithi045.stdout:1373: finishing write tid 2 to smithi04569269-38 2021-07-31T01:50:09.718 INFO:tasks.rados.rados.0.smithi045.stdout:1370: done (3 left) 2021-07-31T01:50:09.718 INFO:tasks.rados.rados.0.smithi045.stdout:1371: done (2 left) 2021-07-31T01:50:09.718 INFO:tasks.rados.rados.0.smithi045.stdout:1374: done (1 left) 2021-07-31T01:50:09.719 INFO:tasks.rados.rados.0.smithi045.stdout:1375: read oid 11 snap -1 2021-07-31T01:50:09.719 INFO:tasks.rados.rados.0.smithi045.stdout:1375: expect deleted 2021-07-31T01:50:09.719 INFO:tasks.rados.rados.0.smithi045.stdout:1376: snap_create 2021-07-31T01:50:09.719 INFO:tasks.rados.rados.0.smithi045.stdout:1373: finishing write tid 3 to smithi04569269-38 2021-07-31T01:50:09.720 INFO:tasks.rados.rados.0.smithi045.stdout:1373: finishing write tid 4 to smithi04569269-38 2021-07-31T01:50:09.720 INFO:tasks.rados.rados.0.smithi045.stdout:1373: finishing write tid 5 to smithi04569269-38 2021-07-31T01:50:09.720 INFO:tasks.rados.rados.0.smithi045.stdout:1373: finishing write tid 6 to smithi04569269-38 2021-07-31T01:50:09.721 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 38 v 583 (ObjNum 425 snap 126 seq_num 425) dirty exists 2021-07-31T01:50:09.721 INFO:tasks.rados.rados.0.smithi045.stdout:1373: left oid 38 (ObjNum 425 snap 126 seq_num 425) 2021-07-31T01:50:10.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:09 smithi154 conmon[34553]: cluster 2021-07-31T01:50:08.617786+0000 mon.a (mon.0) 953 : cluster [DBG] osdmap e309: 8 total, 8 up, 8 in 2021-07-31T01:50:10.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:09 smithi045 conmon[39718]: cluster 2021-07-31T01:50:08.617786+0000 mon.a 2021-07-31T01:50:10.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:09 smithi045 conmon[39718]: (mon.0) 953 : cluster [DBG] osdmap e309: 8 total, 8 up, 8 in 2021-07-31T01:50:10.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:09 smithi045 conmon[33655]: cluster 2021-07-31T01:50:08.617786+0000 2021-07-31T01:50:10.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:09 smithi045 conmon[33655]: mon.a (mon.0) 953 : cluster [DBG] osdmap e309: 8 total, 8 up, 8 in 2021-07-31T01:50:10.630 INFO:tasks.rados.rados.0.smithi045.stdout:1373: done (2 left) 2021-07-31T01:50:10.631 INFO:tasks.rados.rados.0.smithi045.stdout:1375: done (1 left) 2021-07-31T01:50:10.632 INFO:tasks.rados.rados.0.smithi045.stdout:1376: done (0 left) 2021-07-31T01:50:10.632 INFO:tasks.rados.rados.0.smithi045.stdout:1377: delete oid 24 current snap is 127 2021-07-31T01:50:10.634 INFO:tasks.rados.rados.0.smithi045.stdout:1377: done (0 left) 2021-07-31T01:50:10.634 INFO:tasks.rados.rados.0.smithi045.stdout:1378: snap_create 2021-07-31T01:50:11.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:10 smithi154 conmon[34553]: cluster 2021-07-31T01:50:09.619297+0000 mon.a (mon.0) 954 : cluster [DBG] 2021-07-31T01:50:11.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:10 smithi154 conmon[34553]: osdmap e310: 8 total, 8 up, 8 in 2021-07-31T01:50:11.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:10 smithi154 conmon[34553]: cluster 2021-07-31T01:50:09.832486+0000 mgr.x (mgr.24542) 123 : cluster [DBG] pgmap v245: 25 pgs: 25 active+clean; 117 MiB data, 841 MiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 7.0 MiB/s wr, 33 op/s 2021-07-31T01:50:11.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:10 smithi045 conmon[33655]: cluster 2021-07-31T01:50:09.619297+0000 mon.a (mon.0) 954 : cluster [DBG] osdmap e310: 8 total, 8 up, 8 in 2021-07-31T01:50:11.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:10 smithi045 conmon[33655]: cluster 2021-07-31T01:50:09.832486+0000 mgr.x (mgr.24542) 123 : cluster [DBG] pgmap v245: 25 pgs: 25 active+clean; 117 MiB data, 841 MiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 7.0 MiB/s wr, 33 op/s 2021-07-31T01:50:11.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:10 smithi045 conmon[39718]: cluster 2021-07-31T01:50:09.619297+0000 mon.a (mon.0) 954 : cluster [DBG] osdmap e310: 8 total, 8 up, 8 in 2021-07-31T01:50:11.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:10 smithi045 conmon[39718]: cluster 2021-07-31T01:50:09.832486+0000 mgr.x (mgr.24542) 123 : cluster [DBG] pgmap v245: 25 pgs: 25 active+clean; 117 MiB data, 841 MiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 7.0 MiB/s wr, 33 op/s 2021-07-31T01:50:11.534 INFO:tasks.rados.rados.0.smithi045.stdout:1378: done (0 left) 2021-07-31T01:50:11.534 INFO:tasks.rados.rados.0.smithi045.stdout:1379: snap_remove snap 124 2021-07-31T01:50:12.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:11 smithi154 conmon[34553]: cluster 2021-07-31T01:50:10.629754+0000 mon.a (mon.0) 955 : cluster [DBG] osdmap e311: 8 total, 8 up, 8 in 2021-07-31T01:50:12.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:11 smithi154 conmon[34553]: cluster 2021-07-31T01:50:11.533207+0000 mon.a (mon 2021-07-31T01:50:12.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:11 smithi154 conmon[34553]: .0) 956 : cluster [DBG] osdmap e312: 8 total, 8 up, 8 in 2021-07-31T01:50:12.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:11 smithi045 conmon[33655]: cluster 2021-07-31T01:50:10.629754+0000 2021-07-31T01:50:12.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:11 smithi045 conmon[33655]: mon.a (mon.0) 955 : cluster [DBG] osdmap e311: 8 total, 8 up, 8 in 2021-07-31T01:50:12.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:11 smithi045 conmon[33655]: cluster 2021-07-31T01:50:11.533207+0000 mon.a (mon.0) 956 : cluster [DBG] osdmap e312: 8 total, 8 up, 8 in 2021-07-31T01:50:12.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:11 smithi045 conmon[39718]: cluster 2021-07-31T01:50:10.629754+0000 mon.a (mon.0) 955 2021-07-31T01:50:12.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:11 smithi045 conmon[39718]: : cluster [DBG] osdmap e311: 8 total, 8 up, 8 in 2021-07-31T01:50:12.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:11 smithi045 conmon[39718]: cluster 2021-07-31T01:50:11.533207+0000 mon.a (mon.0) 956 : cluster [DBG] osdmap e312: 8 total, 8 up, 8 in 2021-07-31T01:50:12.540 INFO:tasks.rados.rados.0.smithi045.stdout:1379: done (0 left) 2021-07-31T01:50:12.540 INFO:tasks.rados.rados.0.smithi045.stdout:1380: snap_create 2021-07-31T01:50:13.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:12 smithi154 conmon[34553]: cluster 2021-07-31T01:50:11.832883+0000 mgr.x (mgr.24542) 124 : cluster [DBG] pgmap v248: 25 pgs: 25 active+clean; 117 MiB data, 841 MiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 7.0 MiB/s wr, 33 op/s 2021-07-31T01:50:13.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:12 smithi154 conmon[34553]: cluster 2021-07-31T01:50:12.539089+0000 mon.a (mon.0) 957 : cluster [DBG] osdmap e313: 8 total, 8 up, 8 in 2021-07-31T01:50:13.027 INFO:journalctl@ceph.osd.3.smithi045.stdout:Jul 31 01:50:12 smithi045 conmon[54013]: debug 2021-07-31T01:50:12.536+0000 7f64f88c5700 -1 osd.3 pg_epoch: 313 pg[3.bs0( v 309'944 (0'0,309'944] local-lis/les=56/57 n=6 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [3,0,4]p3(0) r=0 lpr=56 crt=309'944 lcod 308'942 mlcod 308'942 active+clean+snaptrim trimq=[fa~1]] removing snap head 2021-07-31T01:50:13.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:12 smithi045 conmon[39718]: cluster 2021-07-31T01:50:11.832883+0000 mgr.x ( 2021-07-31T01:50:13.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:12 smithi045 conmon[39718]: mgr.24542) 124 : cluster [DBG] pgmap v248: 25 pgs: 25 active+clean; 117 MiB data, 841 MiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 7.0 MiB/s wr, 33 op/s 2021-07-31T01:50:13.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:12 smithi045 conmon[39718]: cluster 2021-07-31T01:50:12.539089+0000 2021-07-31T01:50:13.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:12 smithi045 conmon[39718]: mon.a (mon.0) 957 : cluster [DBG] osdmap e313: 8 total, 8 up, 8 in 2021-07-31T01:50:13.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:12 smithi045 conmon[33655]: cluster 2021-07-31T01:50:11.832883+0000 mgr.x (mgr.24542) 124 : cluster 2021-07-31T01:50:13.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:12 smithi045 conmon[33655]: [DBG] pgmap v248: 25 pgs: 25 active+clean; 117 MiB data, 841 MiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 7.0 MiB/s wr, 33 op/s 2021-07-31T01:50:13.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:12 smithi045 conmon[33655]: cluster 2021-07-31T01:50:12.539089+0000 mon.a (mon.0) 957 : cluster [DBG] osdmap e313: 8 total, 8 up, 8 in 2021-07-31T01:50:13.541 INFO:tasks.rados.rados.0.smithi045.stdout:1380: done (0 left) 2021-07-31T01:50:13.541 INFO:tasks.rados.rados.0.smithi045.stdout:1381: read oid 11 snap -1 2021-07-31T01:50:13.541 INFO:tasks.rados.rados.0.smithi045.stdout:1381: expect deleted 2021-07-31T01:50:13.542 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 24 current snap is 129 2021-07-31T01:50:13.542 INFO:tasks.rados.rados.0.smithi045.stdout:1382: seq_num 428 ranges {0=778240,778240=483328,1261568=409600,1671168=417792,2088960=294912} 2021-07-31T01:50:13.545 INFO:tasks.rados.rados.0.smithi045.stdout:1382: writing smithi04569269-24 from 0 to 778240 tid 1 2021-07-31T01:50:13.548 INFO:tasks.rados.rados.0.smithi045.stdout:1382: writing smithi04569269-24 from 778240 to 1261568 tid 2 2021-07-31T01:50:13.550 INFO:tasks.rados.rados.0.smithi045.stdout:1382: writing smithi04569269-24 from 1261568 to 1671168 tid 3 2021-07-31T01:50:13.553 INFO:tasks.rados.rados.0.smithi045.stdout:1382: writing smithi04569269-24 from 1671168 to 2088960 tid 4 2021-07-31T01:50:13.555 INFO:tasks.rados.rados.0.smithi045.stdout:1382: writing smithi04569269-24 from 2088960 to 2383872 tid 5 2021-07-31T01:50:13.555 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 32 current snap is 129 2021-07-31T01:50:13.555 INFO:tasks.rados.rados.0.smithi045.stdout:1383: seq_num 429 ranges {3653632=794624,4448256=770048,5218304=622592,5840896=212992} 2021-07-31T01:50:13.579 INFO:tasks.rados.rados.0.smithi045.stdout:1383: writing smithi04569269-32 from 3653632 to 4448256 tid 1 2021-07-31T01:50:13.583 INFO:tasks.rados.rados.0.smithi045.stdout:1383: writing smithi04569269-32 from 4448256 to 5218304 tid 2 2021-07-31T01:50:13.587 INFO:tasks.rados.rados.0.smithi045.stdout:1383: writing smithi04569269-32 from 5218304 to 5840896 tid 3 2021-07-31T01:50:13.588 INFO:tasks.rados.rados.0.smithi045.stdout:1383: writing smithi04569269-32 from 5840896 to 6053888 tid 4 2021-07-31T01:50:13.589 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 36 current snap is 129 2021-07-31T01:50:13.589 INFO:tasks.rados.rados.0.smithi045.stdout:1384: seq_num 430 ranges {2588672=8192} 2021-07-31T01:50:13.603 INFO:tasks.rados.rados.0.smithi045.stdout:1384: writing smithi04569269-36 from 2588672 to 2596864 tid 1 2021-07-31T01:50:13.603 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 22 current snap is 129 2021-07-31T01:50:13.603 INFO:tasks.rados.rados.0.smithi045.stdout:1385: seq_num 431 ranges {4980736=24576} 2021-07-31T01:50:13.631 INFO:tasks.rados.rados.0.smithi045.stdout:1385: writing smithi04569269-22 from 4980736 to 5005312 tid 1 2021-07-31T01:50:13.631 INFO:tasks.rados.rados.0.smithi045.stdout:1386: snap_remove snap 126 2021-07-31T01:50:13.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:50:13 smithi045 conmon[61210]: level=error ts=2021-07-31T01:50:13.513Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:50:13.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:50:13 smithi045 conmon[61210]: level=error ts=2021-07-31T01:50:13.513Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:50:13.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:50:13 smithi045 conmon[61210]: level=error ts=2021-07-31T01:50:13.513Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T01:50:14.548 INFO:tasks.rados.rados.0.smithi045.stdout:1386: done (5 left) 2021-07-31T01:50:14.549 INFO:tasks.rados.rados.0.smithi045.stdout:1387: rollback oid 34 current snap is 129 2021-07-31T01:50:14.549 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 34 to 127 2021-07-31T01:50:14.549 INFO:tasks.rados.rados.0.smithi045.stdout:1382: finishing write tid 1 to smithi04569269-24 2021-07-31T01:50:14.549 INFO:tasks.rados.rados.0.smithi045.stdout:1382: finishing write tid 2 to smithi04569269-24 2021-07-31T01:50:14.550 INFO:tasks.rados.rados.0.smithi045.stdout:1382: finishing write tid 3 to smithi04569269-24 2021-07-31T01:50:14.550 INFO:tasks.rados.rados.0.smithi045.stdout:1382: finishing write tid 4 to smithi04569269-24 2021-07-31T01:50:14.550 INFO:tasks.rados.rados.0.smithi045.stdout:1382: finishing write tid 5 to smithi04569269-24 2021-07-31T01:50:14.550 INFO:tasks.rados.rados.0.smithi045.stdout:1382: finishing write tid 6 to smithi04569269-24 2021-07-31T01:50:14.551 INFO:tasks.rados.rados.0.smithi045.stdout:1382: finishing write tid 7 to smithi04569269-24 2021-07-31T01:50:14.551 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 24 v 503 (ObjNum 428 snap 129 seq_num 428) dirty exists 2021-07-31T01:50:14.551 INFO:tasks.rados.rados.0.smithi045.stdout:1382: left oid 24 (ObjNum 428 snap 129 seq_num 428) 2021-07-31T01:50:14.551 INFO:tasks.rados.rados.0.smithi045.stdout:1383: finishing write tid 1 to smithi04569269-32 2021-07-31T01:50:14.551 INFO:tasks.rados.rados.0.smithi045.stdout:1383: finishing write tid 2 to smithi04569269-32 2021-07-31T01:50:14.552 INFO:tasks.rados.rados.0.smithi045.stdout:1383: finishing write tid 3 to smithi04569269-32 2021-07-31T01:50:14.552 INFO:tasks.rados.rados.0.smithi045.stdout:1383: finishing write tid 4 to smithi04569269-32 2021-07-31T01:50:14.552 INFO:tasks.rados.rados.0.smithi045.stdout:1383: finishing write tid 5 to smithi04569269-32 2021-07-31T01:50:14.552 INFO:tasks.rados.rados.0.smithi045.stdout:1383: finishing write tid 6 to smithi04569269-32 2021-07-31T01:50:14.553 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 32 v 383 (ObjNum 429 snap 129 seq_num 429) dirty exists 2021-07-31T01:50:14.553 INFO:tasks.rados.rados.0.smithi045.stdout:1383: left oid 32 (ObjNum 429 snap 129 seq_num 429) 2021-07-31T01:50:14.553 INFO:tasks.rados.rados.0.smithi045.stdout:1384: finishing write tid 1 to smithi04569269-36 2021-07-31T01:50:14.553 INFO:tasks.rados.rados.0.smithi045.stdout:1384: finishing write tid 2 to smithi04569269-36 2021-07-31T01:50:14.553 INFO:tasks.rados.rados.0.smithi045.stdout:1384: finishing write tid 3 to smithi04569269-36 2021-07-31T01:50:14.554 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 36 v 949 (ObjNum 430 snap 129 seq_num 430) dirty exists 2021-07-31T01:50:14.554 INFO:tasks.rados.rados.0.smithi045.stdout:1384: left oid 36 (ObjNum 430 snap 129 seq_num 430) 2021-07-31T01:50:14.554 INFO:tasks.rados.rados.0.smithi045.stdout:1385: finishing write tid 1 to smithi04569269-22 2021-07-31T01:50:14.554 INFO:tasks.rados.rados.0.smithi045.stdout:1385: finishing write tid 2 to smithi04569269-22 2021-07-31T01:50:14.555 INFO:tasks.rados.rados.0.smithi045.stdout:1385: finishing write tid 3 to smithi04569269-22 2021-07-31T01:50:14.555 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 22 v 495 (ObjNum 431 snap 129 seq_num 431) dirty exists 2021-07-31T01:50:14.555 INFO:tasks.rados.rados.0.smithi045.stdout:1385: left oid 22 (ObjNum 431 snap 129 seq_num 431) 2021-07-31T01:50:14.555 INFO:tasks.rados.rados.0.smithi045.stdout:1381: done (5 left) 2021-07-31T01:50:14.555 INFO:tasks.rados.rados.0.smithi045.stdout:1382: done (4 left) 2021-07-31T01:50:14.556 INFO:tasks.rados.rados.0.smithi045.stdout:1383: done (3 left) 2021-07-31T01:50:14.556 INFO:tasks.rados.rados.0.smithi045.stdout:1384: done (2 left) 2021-07-31T01:50:14.556 INFO:tasks.rados.rados.0.smithi045.stdout:1385: done (1 left) 2021-07-31T01:50:14.556 INFO:tasks.rados.rados.0.smithi045.stdout:1388: snap_remove snap 128 2021-07-31T01:50:15.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:14 smithi154 conmon[34553]: cluster 2021-07-31T01:50:13.540393+0000 mon.a (mon.0) 958 : cluster [DBG] osdmap e314: 8 total, 8 up, 8 in 2021-07-31T01:50:15.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:14 smithi154 conmon[34553]: 2021-07-31T01:50:15.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:14 smithi154 conmon[34553]: cluster 2021-07-31T01:50:13.833421+0000 mgr.x (mgr.24542) 125 : cluster [DBG] pgmap v251: 25 pgs: 25 active+clean; 128 MiB data, 878 MiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 3.6 MiB/s wr, 12 op/s 2021-07-31T01:50:15.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:14 smithi045 conmon[33655]: cluster 2021-07-31T01:50:13.540393+0000 mon.a (mon.0 2021-07-31T01:50:15.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:14 smithi045 conmon[33655]: ) 958 : cluster [DBG] osdmap e314: 8 total, 8 up, 8 in 2021-07-31T01:50:15.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:14 smithi045 conmon[33655]: cluster 2021-07-31T01:50:13.833421+0000 mgr.x (mgr.24542) 125 : cluster 2021-07-31T01:50:15.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:14 smithi045 conmon[33655]: [DBG] pgmap v251: 25 pgs: 25 active+clean; 128 MiB data, 878 MiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 3.6 MiB/s wr, 12 op/s 2021-07-31T01:50:15.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:14 smithi045 conmon[39718]: cluster 2021-07-31T01:50:13 2021-07-31T01:50:15.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:14 smithi045 conmon[39718]: .540393+0000 mon.a (mon.0) 958 : cluster [DBG] osdmap e314: 8 total, 8 up, 8 in 2021-07-31T01:50:15.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:14 smithi045 conmon[39718]: cluster 2021-07-31T01:50:15.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:14 smithi045 conmon[39718]: 2021-07-31T01:50 2021-07-31T01:50:15.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:14 smithi045 conmon[39718]: :13.833421+0000 mgr.x ( 2021-07-31T01:50:15.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:14 smithi045 conmon[39718]: mgr.24542) 125 : cluster 2021-07-31T01:50:15.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:14 smithi045 conmon[39718]: [DBG] pgmap v251: 25 pgs: 25 active+clean; 128 MiB data, 878 MiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 3.6 MiB/s wr, 12 op/s 2021-07-31T01:50:15.549 INFO:tasks.rados.rados.0.smithi045.stdout:1388: done (1 left) 2021-07-31T01:50:15.549 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 23 current snap is 129 2021-07-31T01:50:15.549 INFO:tasks.rados.rados.0.smithi045.stdout:1387: finishing rollback tid 0 to smithi04569269-34 2021-07-31T01:50:15.550 INFO:tasks.rados.rados.0.smithi045.stdout:1387: finishing rollback tid 1 to smithi04569269-34 2021-07-31T01:50:15.550 INFO:tasks.rados.rados.0.smithi045.stdout:1387: finishing rollback tid 2 to smithi04569269-34 2021-07-31T01:50:15.550 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 34 v 897 (ObjNum 415 snap 124 seq_num 415) dirty exists 2021-07-31T01:50:15.550 INFO:tasks.rados.rados.0.smithi045.stdout:1389: seq_num 432 ranges {3751936=40960} 2021-07-31T01:50:15.570 INFO:tasks.rados.rados.0.smithi045.stdout:1389: writing smithi04569269-23 from 3751936 to 3792896 tid 1 2021-07-31T01:50:15.571 INFO:tasks.rados.rados.0.smithi045.stdout:1387: done (1 left) 2021-07-31T01:50:15.571 INFO:tasks.rados.rados.0.smithi045.stdout:1390: read oid 34 snap 125 2021-07-31T01:50:15.571 INFO:tasks.rados.rados.0.smithi045.stdout:1390: expect (ObjNum 415 snap 124 seq_num 415) 2021-07-31T01:50:15.571 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 24 current snap is 129 2021-07-31T01:50:15.572 INFO:tasks.rados.rados.0.smithi045.stdout:1391: seq_num 433 ranges {2383872=57344} 2021-07-31T01:50:15.584 INFO:tasks.rados.rados.0.smithi045.stdout:1391: writing smithi04569269-24 from 2383872 to 2441216 tid 1 2021-07-31T01:50:15.585 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 41 current snap is 129 2021-07-31T01:50:15.585 INFO:tasks.rados.rados.0.smithi045.stdout:1389: finishing write tid 1 to smithi04569269-23 2021-07-31T01:50:15.585 INFO:tasks.rados.rados.0.smithi045.stdout:1389: finishing write tid 2 to smithi04569269-23 2021-07-31T01:50:15.585 INFO:tasks.rados.rados.0.smithi045.stdout:1389: finishing write tid 3 to smithi04569269-23 2021-07-31T01:50:15.586 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 23 v 585 (ObjNum 432 snap 129 seq_num 432) dirty exists 2021-07-31T01:50:15.586 INFO:tasks.rados.rados.0.smithi045.stdout:1389: left oid 23 (ObjNum 432 snap 129 seq_num 432) 2021-07-31T01:50:15.586 INFO:tasks.rados.rados.0.smithi045.stdout:1392: seq_num 434 ranges {6733824=73728} 2021-07-31T01:50:15.622 INFO:tasks.rados.rados.0.smithi045.stdout:1392: writing smithi04569269-41 from 6733824 to 6807552 tid 1 2021-07-31T01:50:15.622 INFO:tasks.rados.rados.0.smithi045.stdout:1389: done (3 left) 2021-07-31T01:50:15.623 INFO:tasks.rados.rados.0.smithi045.stdout:1393: snap_remove snap 127 2021-07-31T01:50:16.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:15 smithi154 conmon[34553]: cluster 2021-07-31T01:50:14.546690+0000 mon.a (mon.0) 959 : cluster [DBG] osdmap e315: 8 total, 8 up, 8 in 2021-07-31T01:50:16.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:15 smithi045 conmon[39718]: cluster 2021-07-31T01:50:16.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:15 smithi045 conmon[39718]: 2021-07-31T01:50:16.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:15 smithi045 conmon[39718]: 2021 2021-07-31T01:50:16.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:15 smithi045 conmon[39718]: - 2021-07-31T01:50:16.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:15 smithi045 conmon[39718]: 07 2021-07-31T01:50:16.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:15 smithi045 conmon[39718]: - 2021-07-31T01:50:16.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:15 smithi045 conmon[39718]: 31 2021-07-31T01:50:16.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:15 smithi045 conmon[39718]: T 2021-07-31T01:50:16.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:15 smithi045 conmon[39718]: 01 2021-07-31T01:50:16.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:15 smithi045 conmon[39718]: : 2021-07-31T01:50:16.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:15 smithi045 conmon[39718]: 50 2021-07-31T01:50:16.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:15 smithi045 conmon[39718]: : 2021-07-31T01:50:16.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:15 smithi045 conmon[39718]: 14 2021-07-31T01:50:16.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:15 smithi045 conmon[39718]: . 2021-07-31T01:50:16.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:15 smithi045 conmon[39718]: 546690 2021-07-31T01:50:16.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:15 smithi045 conmon[39718]: +0000 2021-07-31T01:50:16.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:15 smithi045 conmon[39718]: 2021-07-31T01:50:16.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:15 smithi045 conmon[39718]: mon.a 2021-07-31T01:50:16.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:15 smithi045 conmon[39718]: ( 2021-07-31T01:50:16.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:15 smithi045 conmon[39718]: mon 2021-07-31T01:50:16.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:15 smithi045 conmon[39718]: . 2021-07-31T01:50:16.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:15 smithi045 conmon[39718]: 0 2021-07-31T01:50:16.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:15 smithi045 conmon[39718]: ) 2021-07-31T01:50:16.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:15 smithi045 conmon[39718]: 959 2021-07-31T01:50:16.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:15 smithi045 conmon[39718]: : 2021-07-31T01:50:16.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:15 smithi045 conmon[39718]: cluster 2021-07-31T01:50:16.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:15 smithi045 conmon[39718]: 2021-07-31T01:50:16.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:15 smithi045 conmon[39718]: [DBG] osdmap e315: 8 total, 8 up, 8 in 2021-07-31T01:50:16.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:15 smithi045 conmon[33655]: cluster 2021-07-31T01:50:16.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:15 smithi045 conmon[33655]: 2021- 2021-07-31T01:50:16.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:15 smithi045 conmon[33655]: 07-31T01:50:14.546690+0000 mon.a (mon.0) 959 : cluster [DBG] osdmap e315: 8 total, 8 up, 8 in 2021-07-31T01:50:16.535 INFO:tasks.rados.rados.0.smithi045.stdout:1393: done (3 left) 2021-07-31T01:50:16.535 INFO:tasks.rados.rados.0.smithi045.stdout:1394: delete oid 23 current snap is 129 2021-07-31T01:50:16.536 INFO:tasks.rados.rados.0.smithi045.stdout:1391: finishing write tid 1 to smithi04569269-24 2021-07-31T01:50:16.536 INFO:tasks.rados.rados.0.smithi045.stdout:1391: finishing write tid 2 to smithi04569269-24 2021-07-31T01:50:16.536 INFO:tasks.rados.rados.0.smithi045.stdout:1391: finishing write tid 3 to smithi04569269-24 2021-07-31T01:50:16.536 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 24 v 505 (ObjNum 433 snap 129 seq_num 433) dirty exists 2021-07-31T01:50:16.537 INFO:tasks.rados.rados.0.smithi045.stdout:1391: left oid 24 (ObjNum 433 snap 129 seq_num 433) 2021-07-31T01:50:16.537 INFO:tasks.rados.rados.0.smithi045.stdout:1390: expect (ObjNum 415 snap 124 seq_num 415) 2021-07-31T01:50:16.842 INFO:tasks.rados.rados.0.smithi045.stdout:1392: finishing write tid 1 to smithi04569269-41 2021-07-31T01:50:16.843 INFO:tasks.rados.rados.0.smithi045.stdout:1392: finishing write tid 2 to smithi04569269-41 2021-07-31T01:50:16.843 INFO:tasks.rados.rados.0.smithi045.stdout:1392: finishing write tid 3 to smithi04569269-41 2021-07-31T01:50:16.843 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 41 v 526 (ObjNum 434 snap 129 seq_num 434) dirty exists 2021-07-31T01:50:16.843 INFO:tasks.rados.rados.0.smithi045.stdout:1392: left oid 41 (ObjNum 434 snap 129 seq_num 434) 2021-07-31T01:50:16.844 INFO:tasks.rados.rados.0.smithi045.stdout:1390: done (3 left) 2021-07-31T01:50:16.844 INFO:tasks.rados.rados.0.smithi045.stdout:1391: done (2 left) 2021-07-31T01:50:16.844 INFO:tasks.rados.rados.0.smithi045.stdout:1392: done (1 left) 2021-07-31T01:50:16.845 INFO:tasks.rados.rados.0.smithi045.stdout:1394: done (0 left) 2021-07-31T01:50:16.845 INFO:tasks.rados.rados.0.smithi045.stdout:1395: delete oid 25 current snap is 129 2021-07-31T01:50:16.845 INFO:tasks.rados.rados.0.smithi045.stdout:1395: done (0 left) 2021-07-31T01:50:16.846 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 8 current snap is 129 2021-07-31T01:50:16.846 INFO:tasks.rados.rados.0.smithi045.stdout:1396: seq_num 437 ranges {1605632=122880} 2021-07-31T01:50:16.854 INFO:tasks.rados.rados.0.smithi045.stdout:1396: writing smithi04569269-8 from 1605632 to 1728512 tid 1 2021-07-31T01:50:16.854 INFO:tasks.rados.rados.0.smithi045.stdout:1397: delete oid 16 current snap is 129 2021-07-31T01:50:16.856 INFO:tasks.rados.rados.0.smithi045.stdout:1397: done (1 left) 2021-07-31T01:50:16.856 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 7 current snap is 129 2021-07-31T01:50:16.857 INFO:tasks.rados.rados.0.smithi045.stdout:1398: seq_num 439 ranges {1236992=155648} 2021-07-31T01:50:16.865 INFO:tasks.rados.rados.0.smithi045.stdout:1398: writing smithi04569269-7 from 1236992 to 1392640 tid 1 2021-07-31T01:50:16.865 INFO:tasks.rados.rados.0.smithi045.stdout:1399: rollback oid 47 current snap is 129 2021-07-31T01:50:16.865 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 47 to 125 2021-07-31T01:50:16.865 INFO:tasks.rados.rados.0.smithi045.stdout:1396: finishing write tid 1 to smithi04569269-8 2021-07-31T01:50:16.866 INFO:tasks.rados.rados.0.smithi045.stdout:1396: finishing write tid 2 to smithi04569269-8 2021-07-31T01:50:16.866 INFO:tasks.rados.rados.0.smithi045.stdout:1396: finishing write tid 3 to smithi04569269-8 2021-07-31T01:50:16.866 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 8 v 958 (ObjNum 437 snap 129 seq_num 437) dirty exists 2021-07-31T01:50:16.866 INFO:tasks.rados.rados.0.smithi045.stdout:1396: left oid 8 (ObjNum 437 snap 129 seq_num 437) 2021-07-31T01:50:16.866 INFO:tasks.rados.rados.0.smithi045.stdout:1396: done (2 left) 2021-07-31T01:50:16.867 INFO:tasks.rados.rados.0.smithi045.stdout:1400: copy_from oid 29 from oid 11 current snap is 129 2021-07-31T01:50:16.867 INFO:tasks.rados.rados.0.smithi045.stdout:1401: read oid 50 snap -1 2021-07-31T01:50:16.867 INFO:tasks.rados.rados.0.smithi045.stdout:1401: expect deleted 2021-07-31T01:50:16.867 INFO:tasks.rados.rados.0.smithi045.stdout:1402: rmattr oid 1 current snap is 129 2021-07-31T01:50:16.867 INFO:tasks.rados.rados.0.smithi045.stdout:1403: read oid 28 snap -1 2021-07-31T01:50:16.868 INFO:tasks.rados.rados.0.smithi045.stdout:1403: expect (ObjNum 333 snap 97 seq_num 333) 2021-07-31T01:50:16.868 INFO:tasks.rados.rados.0.smithi045.stdout:1404: delete oid 10 current snap is 129 2021-07-31T01:50:16.869 INFO:tasks.rados.rados.0.smithi045.stdout:1400: finishing copy_from to smithi04569269-29 2021-07-31T01:50:16.869 INFO:tasks.rados.rados.0.smithi045.stdout:1400: got expected ENOENT (src dne) 2021-07-31T01:50:16.870 INFO:tasks.rados.rados.0.smithi045.stdout:1400: finishing copy_from racing read to smithi04569269-29 2021-07-31T01:50:16.870 INFO:tasks.rados.rados.0.smithi045.stdout:1399: finishing rollback tid 0 to smithi04569269-47 2021-07-31T01:50:16.870 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 1 v 898 (ObjNum 353 snap 106 seq_num 353) dirty exists 2021-07-31T01:50:16.870 INFO:tasks.rados.rados.0.smithi045.stdout:1399: finishing rollback tid 1 to smithi04569269-47 2021-07-31T01:50:16.871 INFO:tasks.rados.rados.0.smithi045.stdout:1399: finishing rollback tid 2 to smithi04569269-47 2021-07-31T01:50:16.871 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 47 v 272 (ObjNum 172 snap 41 seq_num 172) dirty exists 2021-07-31T01:50:16.871 INFO:tasks.rados.rados.0.smithi045.stdout:1398: finishing write tid 1 to smithi04569269-7 2021-07-31T01:50:16.872 INFO:tasks.rados.rados.0.smithi045.stdout:1398: finishing write tid 2 to smithi04569269-7 2021-07-31T01:50:16.872 INFO:tasks.rados.rados.0.smithi045.stdout:1398: finishing write tid 3 to smithi04569269-7 2021-07-31T01:50:16.872 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 7 v 528 (ObjNum 439 snap 129 seq_num 439) dirty exists 2021-07-31T01:50:16.873 INFO:tasks.rados.rados.0.smithi045.stdout:1398: left oid 7 (ObjNum 439 snap 129 seq_num 439) 2021-07-31T01:50:16.873 INFO:tasks.rados.rados.0.smithi045.stdout:1403: expect (ObjNum 333 snap 97 seq_num 333) 2021-07-31T01:50:16.892 INFO:tasks.rados.rados.0.smithi045.stdout:1398: done (6 left) 2021-07-31T01:50:16.892 INFO:tasks.rados.rados.0.smithi045.stdout:1399: done (5 left) 2021-07-31T01:50:16.892 INFO:tasks.rados.rados.0.smithi045.stdout:1400: done (4 left) 2021-07-31T01:50:16.892 INFO:tasks.rados.rados.0.smithi045.stdout:1401: done (3 left) 2021-07-31T01:50:16.893 INFO:tasks.rados.rados.0.smithi045.stdout:1402: done (2 left) 2021-07-31T01:50:16.893 INFO:tasks.rados.rados.0.smithi045.stdout:1403: done (1 left) 2021-07-31T01:50:16.893 INFO:tasks.rados.rados.0.smithi045.stdout:1404: done (0 left) 2021-07-31T01:50:16.893 INFO:tasks.rados.rados.0.smithi045.stdout:1405: rollback oid 12 current snap is 129 2021-07-31T01:50:16.893 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 12 to 125 2021-07-31T01:50:16.894 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 11 current snap is 129 2021-07-31T01:50:16.894 INFO:tasks.rados.rados.0.smithi045.stdout:1406: seq_num 441 ranges {0=188416} 2021-07-31T01:50:16.895 INFO:tasks.rados.rados.0.smithi045.stdout:1406: writing smithi04569269-11 from 0 to 188416 tid 1 2021-07-31T01:50:16.895 INFO:tasks.rados.rados.0.smithi045.stdout:1407: setattr oid 13 current snap is 129 2021-07-31T01:50:16.895 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 41 current snap is 129 2021-07-31T01:50:16.895 INFO:tasks.rados.rados.0.smithi045.stdout:1408: seq_num 442 ranges {6807552=204800} 2021-07-31T01:50:16.935 INFO:tasks.rados.rados.0.smithi045.stdout:1408: writing smithi04569269-41 from 6807552 to 7012352 tid 1 2021-07-31T01:50:16.936 INFO:tasks.rados.rados.0.smithi045.stdout:1409: copy_from oid 31 from oid 10 current snap is 129 2021-07-31T01:50:16.936 INFO:tasks.rados.rados.0.smithi045.stdout:1405: finishing rollback tid 0 to smithi04569269-12 2021-07-31T01:50:16.936 INFO:tasks.rados.rados.0.smithi045.stdout:1410: read oid 40 snap 125 2021-07-31T01:50:16.936 INFO:tasks.rados.rados.0.smithi045.stdout:1410: expect (ObjNum 405 snap 122 seq_num 405) 2021-07-31T01:50:16.937 INFO:tasks.rados.rados.0.smithi045.stdout:1411: rollback oid 34 current snap is 129 2021-07-31T01:50:16.937 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 34 to 125 2021-07-31T01:50:16.937 INFO:tasks.rados.rados.0.smithi045.stdout:1406: finishing write tid 1 to smithi04569269-11 2021-07-31T01:50:16.937 INFO:tasks.rados.rados.0.smithi045.stdout:1406: finishing write tid 2 to smithi04569269-11 2021-07-31T01:50:16.937 INFO:tasks.rados.rados.0.smithi045.stdout:1406: finishing write tid 3 to smithi04569269-11 2021-07-31T01:50:16.938 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 11 v 588 (ObjNum 441 snap 129 seq_num 441) dirty exists 2021-07-31T01:50:16.938 INFO:tasks.rados.rados.0.smithi045.stdout:1406: left oid 11 (ObjNum 441 snap 129 seq_num 441) 2021-07-31T01:50:16.938 INFO:tasks.rados.rados.0.smithi045.stdout:1405: finishing rollback tid 1 to smithi04569269-12 2021-07-31T01:50:16.938 INFO:tasks.rados.rados.0.smithi045.stdout:1405: finishing rollback tid 2 to smithi04569269-12 2021-07-31T01:50:16.939 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 12 v 570 (ObjNum -1105611104 snap 21873 seq_num 3189348336) dirty exists 2021-07-31T01:50:16.939 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 13 v 571 (ObjNum 395 snap 119 seq_num 395) dirty exists 2021-07-31T01:50:16.939 INFO:tasks.rados.rados.0.smithi045.stdout:1405: done (6 left) 2021-07-31T01:50:16.939 INFO:tasks.rados.rados.0.smithi045.stdout:1406: done (5 left) 2021-07-31T01:50:16.939 INFO:tasks.rados.rados.0.smithi045.stdout:1407: done (4 left) 2021-07-31T01:50:16.940 INFO:tasks.rados.rados.0.smithi045.stdout:1412: rmattr oid 49 current snap is 129 2021-07-31T01:50:16.940 INFO:tasks.rados.rados.0.smithi045.stdout:1412: done (4 left) 2021-07-31T01:50:16.940 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 26 current snap is 129 2021-07-31T01:50:16.941 INFO:tasks.rados.rados.0.smithi045.stdout:1413: seq_num 443 ranges {0=221184} 2021-07-31T01:50:16.941 INFO:tasks.rados.rados.0.smithi045.stdout:1413: writing smithi04569269-26 from 0 to 221184 tid 1 2021-07-31T01:50:16.941 INFO:tasks.rados.rados.0.smithi045.stdout:1414: read oid 25 snap -1 2021-07-31T01:50:16.947 INFO:tasks.rados.rados.0.smithi045.stdout:1414: expect deleted 2021-07-31T01:50:16.947 INFO:tasks.rados.rados.0.smithi045.stdout:1415: rollback oid 7 current snap is 129 2021-07-31T01:50:16.947 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 7 to 125 2021-07-31T01:50:16.948 INFO:tasks.rados.rados.0.smithi045.stdout:1416: read oid 16 snap -1 2021-07-31T01:50:16.948 INFO:tasks.rados.rados.0.smithi045.stdout:1416: expect deleted 2021-07-31T01:50:16.948 INFO:tasks.rados.rados.0.smithi045.stdout:1417: snap_create 2021-07-31T01:50:16.948 INFO:tasks.rados.rados.0.smithi045.stdout:1409: finishing copy_from to smithi04569269-31 2021-07-31T01:50:16.948 INFO:tasks.rados.rados.0.smithi045.stdout:1409: got expected ENOENT (src dne) 2021-07-31T01:50:16.949 INFO:tasks.rados.rados.0.smithi045.stdout:1409: finishing copy_from racing read to smithi04569269-31 2021-07-31T01:50:16.951 INFO:tasks.rados.rados.0.smithi045.stdout:1411: finishing rollback tid 0 to smithi04569269-34 2021-07-31T01:50:16.951 INFO:tasks.rados.rados.0.smithi045.stdout:1411: finishing rollback tid 1 to smithi04569269-34 2021-07-31T01:50:16.951 INFO:tasks.rados.rados.0.smithi045.stdout:1411: finishing rollback tid 2 to smithi04569269-34 2021-07-31T01:50:16.952 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 34 v 901 (ObjNum 415 snap 124 seq_num 415) dirty exists 2021-07-31T01:50:16.952 INFO:tasks.rados.rados.0.smithi045.stdout:1408: finishing write tid 1 to smithi04569269-41 2021-07-31T01:50:16.952 INFO:tasks.rados.rados.0.smithi045.stdout:1408: finishing write tid 2 to smithi04569269-41 2021-07-31T01:50:16.952 INFO:tasks.rados.rados.0.smithi045.stdout:1415: finishing rollback tid 0 to smithi04569269-7 2021-07-31T01:50:16.953 INFO:tasks.rados.rados.0.smithi045.stdout:1408: finishing write tid 3 to smithi04569269-41 2021-07-31T01:50:16.953 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 41 v 530 (ObjNum 442 snap 129 seq_num 442) dirty exists 2021-07-31T01:50:16.953 INFO:tasks.rados.rados.0.smithi045.stdout:1408: left oid 41 (ObjNum 442 snap 129 seq_num 442) 2021-07-31T01:50:16.953 INFO:tasks.rados.rados.0.smithi045.stdout:1413: finishing write tid 1 to smithi04569269-26 2021-07-31T01:50:16.953 INFO:tasks.rados.rados.0.smithi045.stdout:1413: finishing write tid 2 to smithi04569269-26 2021-07-31T01:50:16.954 INFO:tasks.rados.rados.0.smithi045.stdout:1415: finishing rollback tid 1 to smithi04569269-7 2021-07-31T01:50:16.954 INFO:tasks.rados.rados.0.smithi045.stdout:1415: finishing rollback tid 2 to smithi04569269-7 2021-07-31T01:50:16.954 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 7 v 533 (ObjNum 360 snap 108 seq_num 360) dirty exists 2021-07-31T01:50:16.954 INFO:tasks.rados.rados.0.smithi045.stdout:1413: finishing write tid 3 to smithi04569269-26 2021-07-31T01:50:16.954 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 26 v 385 (ObjNum 443 snap 129 seq_num 443) dirty exists 2021-07-31T01:50:16.955 INFO:tasks.rados.rados.0.smithi045.stdout:1413: left oid 26 (ObjNum 443 snap 129 seq_num 443) 2021-07-31T01:50:16.993 INFO:tasks.rados.rados.0.smithi045.stdout:1410: expect (ObjNum 405 snap 122 seq_num 405) 2021-07-31T01:50:17.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:16 smithi154 conmon[34553]: cluster 2021-07-31T01:50:15.548199+0000 mon.a (mon.0) 960 : cluster 2021-07-31T01:50:17.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:16 smithi154 conmon[34553]: [DBG] osdmap e316: 8 total, 8 up, 8 in 2021-07-31T01:50:17.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:16 smithi154 conmon[34553]: cluster 2021-07-31T01:50:15.834036+0000 mgr.x (mgr.24542) 126 : cluster [DBG] pgmap v254: 25 pgs: 25 active+clean; 128 MiB data, 878 MiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 3.6 MiB/s wr, 12 op/s 2021-07-31T01:50:17.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:16 smithi154 conmon[34553]: cluster 2021-07-31T01: 2021-07-31T01:50:17.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:16 smithi154 conmon[34553]: 50:16.534648+0000 mon.a (mon.0) 961 : cluster [DBG] osdmap e317: 8 total, 8 up, 8 in 2021-07-31T01:50:17.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:16 smithi045 conmon[33655]: cluster 2021-07-31T01:50:15.548199+0000 mon.a (mon.0) 2021-07-31T01:50:17.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:16 smithi045 conmon[33655]: 960 : cluster [DBG] osdmap e316: 8 total, 8 up, 8 in 2021-07-31T01:50:17.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:16 smithi045 conmon[33655]: cluster 2021-07-31T01:50:15.834036+0000 mgr.x (mgr.24542) 126 : cluster [DBG] pgmap v254: 25 pgs: 25 active+clean; 128 MiB data, 878 MiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 3.6 MiB/s wr, 12 op/s 2021-07-31T01:50:17.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:16 smithi045 conmon[33655]: cluster 2021-07-31T01:50:16. 2021-07-31T01:50:17.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:16 smithi045 conmon[33655]: 534648+0000 mon.a (mon.0) 961 : cluster [DBG] osdmap e317: 8 total, 8 up, 8 in 2021-07-31T01:50:17.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:16 smithi045 conmon[33655]: 2021-07-31T01:50:17.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:16 smithi045 conmon[39718]: cluster 2021-07-31T01:50:15 2021-07-31T01:50:17.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:16 smithi045 conmon[39718]: .548199+0000 mon.a (mon.0) 960 : cluster [DBG] osdmap e316: 8 total, 8 up, 8 in 2021-07-31T01:50:17.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:16 smithi045 conmon[39718]: cluster 2021-07-31T01:50:17.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:16 smithi045 conmon[39718]: 2021-07-31T01:50 2021-07-31T01:50:17.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:16 smithi045 conmon[39718]: :15.834036+0000 mgr.x 2021-07-31T01:50:17.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:16 smithi045 conmon[39718]: (mgr.24542) 126 : cluster 2021-07-31T01:50:17.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:16 smithi045 conmon[39718]: [DBG] pgmap v254: 25 pgs: 25 active+clean; 128 MiB data, 878 MiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 3.6 MiB/s wr, 12 op/s 2021-07-31T01:50:17.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:16 smithi045 conmon[39718]: cluster 2021- 2021-07-31T01:50:17.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:16 smithi045 conmon[39718]: 07-31T01:50:16. 2021-07-31T01:50:17.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:16 smithi045 conmon[39718]: 534648+0000 mon.a (mon.0 2021-07-31T01:50:17.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:16 smithi045 conmon[39718]: ) 961 : cluster [DBG] osdmap e317: 8 total, 8 up, 8 in 2021-07-31T01:50:17.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:16 smithi045 conmon[39718]: 2021-07-31T01:50:18.151 INFO:tasks.rados.rados.0.smithi045.stdout:1408: done (8 left) 2021-07-31T01:50:18.151 INFO:tasks.rados.rados.0.smithi045.stdout:1409: done (7 left) 2021-07-31T01:50:18.152 INFO:tasks.rados.rados.0.smithi045.stdout:1410: done (6 left) 2021-07-31T01:50:18.152 INFO:tasks.rados.rados.0.smithi045.stdout:1411: done (5 left) 2021-07-31T01:50:18.152 INFO:tasks.rados.rados.0.smithi045.stdout:1413: done (4 left) 2021-07-31T01:50:18.152 INFO:tasks.rados.rados.0.smithi045.stdout:1414: done (3 left) 2021-07-31T01:50:18.153 INFO:tasks.rados.rados.0.smithi045.stdout:1415: done (2 left) 2021-07-31T01:50:18.153 INFO:tasks.rados.rados.0.smithi045.stdout:1416: done (1 left) 2021-07-31T01:50:18.153 INFO:tasks.rados.rados.0.smithi045.stdout:1417: done (0 left) 2021-07-31T01:50:18.153 INFO:tasks.rados.rados.0.smithi045.stdout:1418: delete oid 30 current snap is 130 2021-07-31T01:50:18.154 INFO:tasks.rados.rados.0.smithi045.stdout:1418: done (0 left) 2021-07-31T01:50:18.154 INFO:tasks.rados.rados.0.smithi045.stdout:1419: snap_create 2021-07-31T01:50:18.567 INFO:tasks.rados.rados.0.smithi045.stdout:1419: done (0 left) 2021-07-31T01:50:18.567 INFO:tasks.rados.rados.0.smithi045.stdout:1420: read oid 13 snap -1 2021-07-31T01:50:18.567 INFO:tasks.rados.rados.0.smithi045.stdout:1420: expect (ObjNum 395 snap 119 seq_num 395) 2021-07-31T01:50:18.567 INFO:tasks.rados.rados.0.smithi045.stdout:1421: read oid 34 snap -1 2021-07-31T01:50:18.568 INFO:tasks.rados.rados.0.smithi045.stdout:1421: expect (ObjNum 415 snap 124 seq_num 415) 2021-07-31T01:50:18.568 INFO:tasks.rados.rados.0.smithi045.stdout:1422: read oid 14 snap -1 2021-07-31T01:50:18.568 INFO:tasks.rados.rados.0.smithi045.stdout:1422: expect deleted 2021-07-31T01:50:18.568 INFO:tasks.rados.rados.0.smithi045.stdout:1423: snap_create 2021-07-31T01:50:18.581 INFO:tasks.rados.rados.0.smithi045.stdout:1420: expect (ObjNum 395 snap 119 seq_num 395) 2021-07-31T01:50:18.654 INFO:tasks.rados.rados.0.smithi045.stdout:1421: expect (ObjNum 415 snap 124 seq_num 415) 2021-07-31T01:50:19.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:18 smithi154 conmon[34553]: cluster 2021-07-31T01:50:17.559810+0000 mon.a (mon.0) 962 : cluster 2021-07-31T01:50:19.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:18 smithi154 conmon[34553]: [DBG] osdmap e318: 8 total, 8 up, 8 in 2021-07-31T01:50:19.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:18 smithi154 conmon[34553]: cluster 2021-07-31T01:50:17.834443+0000 mgr.x (mgr.24542) 127 : cluster [DBG] pgmap v257: 25 pgs: 25 active+clean; 128 MiB data, 878 MiB used, 714 GiB / 715 GiB avail 2021-07-31T01:50:19.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:18 smithi045 conmon[33655]: cluster 2021-07-31T01:50:17.559810+0000 2021-07-31T01:50:19.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:18 smithi045 conmon[33655]: mon.a (mon.0) 962 : cluster [DBG] osdmap e318: 8 total, 8 up, 8 in 2021-07-31T01:50:19.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:18 smithi045 conmon[33655]: cluster 2021-07-31T01:50:17.834443+0000 mgr.x (mgr.24542) 127 : cluster 2021-07-31T01:50:19.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:18 smithi045 conmon[33655]: [DBG] pgmap v257: 25 pgs: 25 active+clean; 128 MiB data, 878 MiB used, 714 GiB / 715 GiB avail 2021-07-31T01:50:19.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:18 smithi045 conmon[39718]: cluster 2021-07-31T01:50:17.559810+0000 mon.a (mon.0 2021-07-31T01:50:19.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:18 smithi045 conmon[39718]: ) 962 : cluster [DBG] osdmap e318: 8 total, 8 up, 8 in 2021-07-31T01:50:19.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:18 smithi045 conmon[39718]: cluster 2021-07-31T01:50:19.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:18 smithi045 conmon[39718]: 2021-07-31T01:50:17.834443+0000 mgr.x 2021-07-31T01:50:19.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:18 smithi045 conmon[39718]: (mgr.24542) 127 : cluster [DBG] pgmap v257: 25 pgs: 25 active+clean; 128 MiB data, 878 MiB used, 714 GiB / 715 GiB avail 2021-07-31T01:50:19.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:18 smithi045 conmon[39718]: 2021-07-31T01:50:19.569 INFO:tasks.rados.rados.0.smithi045.stdout:1420: done (3 left) 2021-07-31T01:50:19.570 INFO:tasks.rados.rados.0.smithi045.stdout:1421: done (2 left) 2021-07-31T01:50:19.570 INFO:tasks.rados.rados.0.smithi045.stdout:1422: done (1 left) 2021-07-31T01:50:19.571 INFO:tasks.rados.rados.0.smithi045.stdout:1423: done (0 left) 2021-07-31T01:50:19.571 INFO:tasks.rados.rados.0.smithi045.stdout:1424: read oid 35 snap -1 2021-07-31T01:50:19.572 INFO:tasks.rados.rados.0.smithi045.stdout:1424: expect (ObjNum 389 snap 115 seq_num 389) 2021-07-31T01:50:19.572 INFO:tasks.rados.rados.0.smithi045.stdout:1425: setattr oid 26 current snap is 132 2021-07-31T01:50:19.574 INFO:tasks.rados.rados.0.smithi045.stdout:1426: read oid 8 snap -1 2021-07-31T01:50:19.574 INFO:tasks.rados.rados.0.smithi045.stdout:1426: expect (ObjNum 437 snap 129 seq_num 437) 2021-07-31T01:50:19.575 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 17 current snap is 132 2021-07-31T01:50:19.575 INFO:tasks.rados.rados.0.smithi045.stdout:1427: seq_num 445 ranges {0=253952} 2021-07-31T01:50:19.575 INFO:tasks.rados.rados.0.smithi045.stdout:1427: writing smithi04569269-17 from 0 to 253952 tid 1 2021-07-31T01:50:19.576 INFO:tasks.rados.rados.0.smithi045.stdout:1428: copy_from oid 7 from oid 36 current snap is 132 2021-07-31T01:50:19.576 INFO:tasks.rados.rados.0.smithi045.stdout:1429: snap_remove snap 131 2021-07-31T01:50:20.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:19 smithi154 conmon[34553]: cluster 2021-07-31T01:50:18.566132+0000 mon.a (mon.0) 963 : cluster [DBG] osdmap e319: 8 total, 8 up, 8 in 2021-07-31T01:50:20.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:19 smithi045 conmon[39718]: cluster 2021-07-31T01:50:18.566132+0000 2021-07-31T01:50:20.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:19 smithi045 conmon[39718]: mon.a (mon.0) 963 : cluster [DBG] osdmap e319: 8 total, 8 up, 8 in 2021-07-31T01:50:20.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:19 smithi045 conmon[33655]: cluster 2021-07-31T01:50:18.566132+0000 mon.a (mon.0) 963 : 2021-07-31T01:50:20.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:19 smithi045 conmon[33655]: cluster [DBG] osdmap e319: 8 total, 8 up, 8 in 2021-07-31T01:50:20.580 INFO:tasks.rados.rados.0.smithi045.stdout:1429: done (5 left) 2021-07-31T01:50:20.580 INFO:tasks.rados.rados.0.smithi045.stdout:1430: delete oid 6 current snap is 132 2021-07-31T01:50:20.580 INFO:tasks.rados.rados.0.smithi045.stdout:1427: finishing write tid 1 to smithi04569269-17 2021-07-31T01:50:20.581 INFO:tasks.rados.rados.0.smithi045.stdout:1427: finishing write tid 2 to smithi04569269-17 2021-07-31T01:50:20.581 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 26 v 386 (ObjNum 443 snap 129 seq_num 443) dirty exists 2021-07-31T01:50:20.581 INFO:tasks.rados.rados.0.smithi045.stdout:1427: finishing write tid 3 to smithi04569269-17 2021-07-31T01:50:20.581 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 17 v 541 (ObjNum 445 snap 132 seq_num 445) dirty exists 2021-07-31T01:50:20.582 INFO:tasks.rados.rados.0.smithi045.stdout:1427: left oid 17 (ObjNum 445 snap 132 seq_num 445) 2021-07-31T01:50:20.582 INFO:tasks.rados.rados.0.smithi045.stdout:1424: expect (ObjNum 389 snap 115 seq_num 389) 2021-07-31T01:50:21.000 INFO:tasks.rados.rados.0.smithi045.stdout:1426: expect (ObjNum 437 snap 129 seq_num 437) 2021-07-31T01:50:21.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:20 smithi154 conmon[34553]: cluster 2021-07-31T01:50:19.568327+0000 mon.a (mon.0) 964 2021-07-31T01:50:21.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:20 smithi154 conmon[34553]: : cluster [DBG] osdmap e320: 8 total, 8 up, 8 in 2021-07-31T01:50:21.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:20 smithi154 conmon[34553]: cluster 2021-07-31T01:50:19.834946+0000 mgr.x (mgr.24542) 128 : cluster [DBG] pgmap v260: 25 pgs: 25 active+clean; 127 MiB data, 899 MiB used, 714 GiB / 715 GiB avail; 20 MiB/s rd, 260 KiB/s wr, 24 op/s 2021-07-31T01:50:21.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:20 smithi045 conmon[39718]: cluster 2021-07-31T01:50:19.568327+0000 mon.a ( 2021-07-31T01:50:21.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:20 smithi045 conmon[39718]: mon.0) 964 : cluster [DBG] osdmap e320: 8 total, 8 up, 8 in 2021-07-31T01:50:21.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:20 smithi045 conmon[39718]: cluster 2021-07-31T01:50:19.834946+0000 mgr.x ( 2021-07-31T01:50:21.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:20 smithi045 conmon[39718]: mgr.24542) 128 : cluster [DBG] pgmap v260: 25 pgs: 25 active+clean; 127 MiB data, 899 MiB used, 714 GiB / 715 GiB avail; 20 MiB/s rd, 260 KiB/s wr, 24 op/s 2021-07-31T01:50:21.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:20 smithi045 conmon[33655]: cluster 2021-07-31T01:50:19.568327+0000 mon.a (mon.0) 964 : cluster 2021-07-31T01:50:21.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:20 smithi045 conmon[33655]: [DBG] osdmap e320: 8 total, 8 up, 8 in 2021-07-31T01:50:21.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:20 smithi045 conmon[33655]: cluster 2021-07-31T01:50:19.834946+0000 mgr.x (mgr.24542) 128 : cluster [DBG] pgmap v260: 25 pgs: 25 active+clean; 127 MiB data, 899 MiB used, 714 GiB / 715 GiB avail; 20 MiB/s rd, 260 KiB/s wr, 24 op/s 2021-07-31T01:50:21.085 INFO:tasks.rados.rados.0.smithi045.stdout:1428: finishing copy_from to smithi04569269-7 2021-07-31T01:50:21.086 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 7 v 949 (ObjNum 430 snap 129 seq_num 430) dirty exists 2021-07-31T01:50:21.086 INFO:tasks.rados.rados.0.smithi045.stdout:1428: finishing copy_from racing read to smithi04569269-7 2021-07-31T01:50:21.086 INFO:tasks.rados.rados.0.smithi045.stdout:1424: done (5 left) 2021-07-31T01:50:21.087 INFO:tasks.rados.rados.0.smithi045.stdout:1425: done (4 left) 2021-07-31T01:50:21.087 INFO:tasks.rados.rados.0.smithi045.stdout:1426: done (3 left) 2021-07-31T01:50:21.087 INFO:tasks.rados.rados.0.smithi045.stdout:1427: done (2 left) 2021-07-31T01:50:21.087 INFO:tasks.rados.rados.0.smithi045.stdout:1428: done (1 left) 2021-07-31T01:50:21.088 INFO:tasks.rados.rados.0.smithi045.stdout:1430: done (0 left) 2021-07-31T01:50:21.088 INFO:tasks.rados.rados.0.smithi045.stdout:1431: read oid 24 snap -1 2021-07-31T01:50:21.088 INFO:tasks.rados.rados.0.smithi045.stdout:1431: expect (ObjNum 433 snap 129 seq_num 433) 2021-07-31T01:50:21.088 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 17 current snap is 132 2021-07-31T01:50:21.089 INFO:tasks.rados.rados.0.smithi045.stdout:1432: seq_num 447 ranges {253952=294912} 2021-07-31T01:50:21.089 INFO:tasks.rados.rados.0.smithi045.stdout:1432: writing smithi04569269-17 from 253952 to 548864 tid 1 2021-07-31T01:50:21.090 INFO:tasks.rados.rados.0.smithi045.stdout:1433: snap_remove snap 125 2021-07-31T01:50:21.541 INFO:tasks.rados.rados.0.smithi045.stdout:1431: expect (ObjNum 433 snap 129 seq_num 433) 2021-07-31T01:50:21.649 INFO:tasks.rados.rados.0.smithi045.stdout:1432: finishing write tid 1 to smithi04569269-17 2021-07-31T01:50:21.650 INFO:tasks.rados.rados.0.smithi045.stdout:1432: finishing write tid 2 to smithi04569269-17 2021-07-31T01:50:21.650 INFO:tasks.rados.rados.0.smithi045.stdout:1432: finishing write tid 3 to smithi04569269-17 2021-07-31T01:50:21.650 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 17 v 543 (ObjNum 447 snap 132 seq_num 447) dirty exists 2021-07-31T01:50:21.650 INFO:tasks.rados.rados.0.smithi045.stdout:1432: left oid 17 (ObjNum 447 snap 132 seq_num 447) 2021-07-31T01:50:21.651 INFO:tasks.rados.rados.0.smithi045.stdout:1431: done (2 left) 2021-07-31T01:50:21.651 INFO:tasks.rados.rados.0.smithi045.stdout:1432: done (1 left) 2021-07-31T01:50:21.651 INFO:tasks.rados.rados.0.smithi045.stdout:1433: done (0 left) 2021-07-31T01:50:21.651 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 35 current snap is 132 2021-07-31T01:50:21.652 INFO:tasks.rados.rados.0.smithi045.stdout:1434: seq_num 448 ranges {7061504=311296} 2021-07-31T01:50:21.689 INFO:tasks.rados.rados.0.smithi045.stdout:1434: writing smithi04569269-35 from 7061504 to 7372800 tid 1 2021-07-31T01:50:21.690 INFO:tasks.rados.rados.0.smithi045.stdout:1435: snap_create 2021-07-31T01:50:21.693 INFO:tasks.rados.rados.0.smithi045.stdout:1434: finishing write tid 1 to smithi04569269-35 2021-07-31T01:50:21.693 INFO:tasks.rados.rados.0.smithi045.stdout:1434: finishing write tid 2 to smithi04569269-35 2021-07-31T01:50:21.694 INFO:tasks.rados.rados.0.smithi045.stdout:1434: finishing write tid 3 to smithi04569269-35 2021-07-31T01:50:21.695 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 35 v 965 (ObjNum 448 snap 132 seq_num 448) dirty exists 2021-07-31T01:50:21.695 INFO:tasks.rados.rados.0.smithi045.stdout:1434: left oid 35 (ObjNum 448 snap 132 seq_num 448) 2021-07-31T01:50:22.003 INFO:journalctl@ceph.osd.5.smithi154.stdout:Jul 31 01:50:21 smithi154 conmon[42050]: debug 2021-07-31T01:50:21.542+0000 7fc9b5064700 -1 osd.5 pg_epoch: 322 pg[3.cs0( v 322'552 (0'0,322'552] local-lis/les=56/57 n=6 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [5,3,6]p5(0) r=0 lpr=56 luod=317'550 crt=317'550 lcod 317'549 mlcod 317'549 active+clean+snaptrim trimq=[fb~1] ps=[105~1]] removing snap head 2021-07-31T01:50:22.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:21 smithi154 conmon[34553]: cluster 2021-07-31T01:50:20.578410+0000 mon.a (mon.0) 965 : cluster [DBG] osdmap e321: 8 total, 8 up, 8 in 2021-07-31T01:50:22.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:21 smithi154 conmon[34553]: 2021-07-31T01:50:22.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:21 smithi154 conmon[34553]: cluster 2021-07-31T01:50:21.536245+0000 mon.a (mon.0) 966 : cluster [DBG] osdmap e322: 8 total, 8 up, 8 in 2021-07-31T01:50:22.026 INFO:journalctl@ceph.osd.0.smithi045.stdout:Jul 31 01:50:21 smithi045 conmon[43243]: debug 2021-07-31T01:50:21.549+0000 7fdf6ff98700 -1 osd.0 pg_epoch: 322 pg[3.1s0( v 322'262 (0'0,322'262] local-lis/les=56/57 n=4 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [0,4,3]p0(0) r=0 lpr=56 luod=317'260 crt=317'260 lcod 317'258 mlcod 317'258 active+clean+snaptrim trimq=[fb~1] ps=[105~1]] removing snap head 2021-07-31T01:50:22.026 INFO:journalctl@ceph.osd.1.smithi045.stdout:Jul 31 01:50:21 smithi045 conmon[46771]: debug 2021-07-31T01:50:22.027 INFO:journalctl@ceph.osd.1.smithi045.stdout:Jul 31 01:50:21 smithi045 conmon[46771]: 2021-07-31T01:50:21.538+0000 7f41979aa700 -1 osd.1 pg_epoch: 322 pg[3.0s0( v 313'351 (0'0,313'351] local-lis/les=56/57 n=5 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [1,2,6]p1(0) r=0 lpr=56 crt=313'351 lcod 310'349 mlcod 310'349 active+clean+snaptrim trimq=[fb~1] ps=[105~1]] removing snap head 2021-07-31T01:50:22.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:21 smithi045 conmon[33655]: cluster 2021-07-31T01:50:20.578410+0000 mon.a ( 2021-07-31T01:50:22.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:21 smithi045 conmon[33655]: mon.0) 965 : cluster [DBG] osdmap e321: 8 total, 8 up, 8 in 2021-07-31T01:50:22.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:21 smithi045 conmon[33655]: cluster 2021-07-31T01:50:21.536245+0000 mon.a (mon.0) 966 : cluster [DBG] osdmap e322: 8 total, 8 up, 8 in 2021-07-31T01:50:22.028 INFO:journalctl@ceph.osd.3.smithi045.stdout:Jul 31 01:50:21 smithi045 conmon[54013]: debug 2021-07-31T01:50:22.028 INFO:journalctl@ceph.osd.3.smithi045.stdout:Jul 31 01:50:21 smithi045 conmon[54013]: 2021-07-31T01:50:21.534+0000 7f64fd0ce700 -1 osd.3 pg_epoch: 322 pg[3.2s0( v 317'221 (0'0,317'221] local-lis/les=56/57 n=4 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [3,5,6]p3(0) r=0 lpr=56 crt=317'221 lcod 317'220 mlcod 317'220 active+clean+snaptrim trimq=[fb~1] ps=[105~1]] removing snap head 2021-07-31T01:50:22.029 INFO:journalctl@ceph.osd.3.smithi045.stdout:Jul 31 01:50:21 smithi045 conmon[54013]: 2021-07-31T01:50:22.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:21 smithi045 conmon[39718]: cluster 2021-07-31T01:50:20.578410 2021-07-31T01:50:22.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:21 smithi045 conmon[39718]: +0000 mon.a (mon.0) 965 : cluster [DBG] osdmap e321: 8 total, 8 up, 8 in 2021-07-31T01:50:22.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:21 smithi045 conmon[39718]: cluster 2021-07-31T01:50:22.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:21 smithi045 conmon[39718]: 2021-07-31T01:50:21.536245+0000 mon.a 2021-07-31T01:50:22.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:21 smithi045 conmon[39718]: (mon.0) 966 : cluster [DBG] osdmap e322: 8 total, 8 up, 8 in 2021-07-31T01:50:22.586 INFO:tasks.rados.rados.0.smithi045.stdout:1434: done (1 left) 2021-07-31T01:50:22.587 INFO:tasks.rados.rados.0.smithi045.stdout:1435: done (0 left) 2021-07-31T01:50:22.587 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 44 current snap is 133 2021-07-31T01:50:22.587 INFO:tasks.rados.rados.0.smithi045.stdout:1436: seq_num 449 ranges {4505600=327680} 2021-07-31T01:50:22.613 INFO:tasks.rados.rados.0.smithi045.stdout:1436: writing smithi04569269-44 from 4505600 to 4833280 tid 1 2021-07-31T01:50:22.613 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 48 current snap is 133 2021-07-31T01:50:22.613 INFO:tasks.rados.rados.0.smithi045.stdout:1437: seq_num 450 ranges {5677056=344064} 2021-07-31T01:50:22.646 INFO:tasks.rados.rados.0.smithi045.stdout:1437: writing smithi04569269-48 from 5677056 to 6021120 tid 1 2021-07-31T01:50:22.646 INFO:tasks.rados.rados.0.smithi045.stdout:1438: delete oid 25 current snap is 133 2021-07-31T01:50:22.647 INFO:tasks.rados.rados.0.smithi045.stdout:1436: finishing write tid 1 to smithi04569269-44 2021-07-31T01:50:22.647 INFO:tasks.rados.rados.0.smithi045.stdout:1436: finishing write tid 2 to smithi04569269-44 2021-07-31T01:50:22.647 INFO:tasks.rados.rados.0.smithi045.stdout:1436: finishing write tid 3 to smithi04569269-44 2021-07-31T01:50:22.648 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 44 v 926 (ObjNum 449 snap 133 seq_num 449) dirty exists 2021-07-31T01:50:22.648 INFO:tasks.rados.rados.0.smithi045.stdout:1436: left oid 44 (ObjNum 449 snap 133 seq_num 449) 2021-07-31T01:50:22.648 INFO:tasks.rados.rados.0.smithi045.stdout:1436: done (2 left) 2021-07-31T01:50:22.649 INFO:tasks.rados.rados.0.smithi045.stdout:1438: done (1 left) 2021-07-31T01:50:22.649 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 22 current snap is 133 2021-07-31T01:50:22.649 INFO:tasks.rados.rados.0.smithi045.stdout:1439: seq_num 452 ranges {5005312=376832} 2021-07-31T01:50:22.678 INFO:tasks.rados.rados.0.smithi045.stdout:1439: writing smithi04569269-22 from 5005312 to 5382144 tid 1 2021-07-31T01:50:22.679 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 49 current snap is 133 2021-07-31T01:50:22.679 INFO:tasks.rados.rados.0.smithi045.stdout:1440: seq_num 453 ranges {0=393216} 2021-07-31T01:50:22.680 INFO:tasks.rados.rados.0.smithi045.stdout:1440: writing smithi04569269-49 from 0 to 393216 tid 1 2021-07-31T01:50:22.681 INFO:tasks.rados.rados.0.smithi045.stdout:1441: read oid 44 snap 130 2021-07-31T01:50:22.681 INFO:tasks.rados.rados.0.smithi045.stdout:1441: expect (ObjNum 422 snap 125 seq_num 422) 2021-07-31T01:50:22.681 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 46 current snap is 133 2021-07-31T01:50:22.681 INFO:tasks.rados.rados.0.smithi045.stdout:1442: seq_num 454 ranges {7569408=409600} 2021-07-31T01:50:22.725 INFO:tasks.rados.rados.0.smithi045.stdout:1442: writing smithi04569269-46 from 7569408 to 7979008 tid 1 2021-07-31T01:50:22.726 INFO:tasks.rados.rados.0.smithi045.stdout:1443: delete oid 2 current snap is 133 2021-07-31T01:50:22.726 INFO:tasks.rados.rados.0.smithi045.stdout:1437: finishing write tid 1 to smithi04569269-48 2021-07-31T01:50:22.726 INFO:tasks.rados.rados.0.smithi045.stdout:1437: finishing write tid 2 to smithi04569269-48 2021-07-31T01:50:22.726 INFO:tasks.rados.rados.0.smithi045.stdout:1437: finishing write tid 3 to smithi04569269-48 2021-07-31T01:50:22.727 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 48 v 903 (ObjNum 450 snap 133 seq_num 450) dirty exists 2021-07-31T01:50:22.727 INFO:tasks.rados.rados.0.smithi045.stdout:1437: left oid 48 (ObjNum 450 snap 133 seq_num 450) 2021-07-31T01:50:22.727 INFO:tasks.rados.rados.0.smithi045.stdout:1439: finishing write tid 1 to smithi04569269-22 2021-07-31T01:50:22.727 INFO:tasks.rados.rados.0.smithi045.stdout:1439: finishing write tid 2 to smithi04569269-22 2021-07-31T01:50:22.727 INFO:tasks.rados.rados.0.smithi045.stdout:1440: finishing write tid 1 to smithi04569269-49 2021-07-31T01:50:22.728 INFO:tasks.rados.rados.0.smithi045.stdout:1440: finishing write tid 2 to smithi04569269-49 2021-07-31T01:50:22.728 INFO:tasks.rados.rados.0.smithi045.stdout:1439: finishing write tid 3 to smithi04569269-22 2021-07-31T01:50:22.728 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 22 v 498 (ObjNum 452 snap 133 seq_num 452) dirty exists 2021-07-31T01:50:22.728 INFO:tasks.rados.rados.0.smithi045.stdout:1439: left oid 22 (ObjNum 452 snap 133 seq_num 452) 2021-07-31T01:50:22.729 INFO:tasks.rados.rados.0.smithi045.stdout:1440: finishing write tid 3 to smithi04569269-49 2021-07-31T01:50:22.729 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 49 v 967 (ObjNum 453 snap 133 seq_num 453) dirty exists 2021-07-31T01:50:22.729 INFO:tasks.rados.rados.0.smithi045.stdout:1440: left oid 49 (ObjNum 453 snap 133 seq_num 453) 2021-07-31T01:50:22.729 INFO:tasks.rados.rados.0.smithi045.stdout:1441: expect (ObjNum 422 snap 125 seq_num 422) 2021-07-31T01:50:22.878 INFO:tasks.rados.rados.0.smithi045.stdout:1442: finishing write tid 1 to smithi04569269-46 2021-07-31T01:50:22.879 INFO:tasks.rados.rados.0.smithi045.stdout:1442: finishing write tid 2 to smithi04569269-46 2021-07-31T01:50:22.879 INFO:tasks.rados.rados.0.smithi045.stdout:1442: finishing write tid 3 to smithi04569269-46 2021-07-31T01:50:22.879 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 46 v 267 (ObjNum 454 snap 133 seq_num 454) dirty exists 2021-07-31T01:50:22.879 INFO:tasks.rados.rados.0.smithi045.stdout:1442: left oid 46 (ObjNum 454 snap 133 seq_num 454) 2021-07-31T01:50:22.880 INFO:tasks.rados.rados.0.smithi045.stdout:1437: done (5 left) 2021-07-31T01:50:22.880 INFO:tasks.rados.rados.0.smithi045.stdout:1439: done (4 left) 2021-07-31T01:50:22.880 INFO:tasks.rados.rados.0.smithi045.stdout:1440: done (3 left) 2021-07-31T01:50:22.880 INFO:tasks.rados.rados.0.smithi045.stdout:1441: done (2 left) 2021-07-31T01:50:22.881 INFO:tasks.rados.rados.0.smithi045.stdout:1442: done (1 left) 2021-07-31T01:50:22.881 INFO:tasks.rados.rados.0.smithi045.stdout:1443: done (0 left) 2021-07-31T01:50:22.881 INFO:tasks.rados.rados.0.smithi045.stdout:1444: setattr oid 28 current snap is 133 2021-07-31T01:50:22.882 INFO:tasks.rados.rados.0.smithi045.stdout:1445: snap_remove snap 132 2021-07-31T01:50:23.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:22 smithi154 conmon[34553]: cluster 2021-07-31T01:50:21.835368+0000 mgr.x (mgr. 2021-07-31T01:50:23.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:22 smithi154 conmon[34553]: 24542) 129 : cluster [DBG] pgmap v263: 25 pgs: 25 active+clean; 127 MiB data, 899 MiB used, 714 GiB / 715 GiB avail; 20 MiB/s rd, 260 KiB/s wr, 24 op/s 2021-07-31T01:50:23.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:22 smithi045 conmon[33655]: cluster 2021-07-31T01:50:21.835368+0000 mgr.x (mgr.24542) 129 : 2021-07-31T01:50:23.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:22 smithi045 conmon[33655]: cluster [DBG] pgmap v263: 25 pgs: 25 active+clean; 127 MiB data, 899 MiB used, 714 GiB / 715 GiB avail; 20 MiB/s rd, 260 KiB/s wr, 24 op/s 2021-07-31T01:50:23.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:22 smithi045 conmon[39718]: cluster 2021-07-31T01:50:21.835368+0000 mgr.x (mgr.24542) 129 : cluster 2021-07-31T01:50:23.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:22 smithi045 conmon[39718]: [DBG] pgmap v263: 25 pgs: 25 active+clean; 127 MiB data, 899 MiB used, 714 GiB / 715 GiB avail; 20 MiB/s rd, 260 KiB/s wr, 24 op/s 2021-07-31T01:50:23.593 INFO:tasks.rados.rados.0.smithi045.stdout:1445: done (1 left) 2021-07-31T01:50:23.593 INFO:tasks.rados.rados.0.smithi045.stdout:1446: setattr oid 50 current snap is 133 2021-07-31T01:50:23.593 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 28 v 545 (ObjNum 333 snap 97 seq_num 333) dirty exists 2021-07-31T01:50:23.595 INFO:tasks.rados.rados.0.smithi045.stdout:1444: done (1 left) 2021-07-31T01:50:23.595 INFO:tasks.rados.rados.0.smithi045.stdout:1447: rollback oid 32 current snap is 133 2021-07-31T01:50:23.595 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 32 to 130 2021-07-31T01:50:23.596 INFO:tasks.rados.rados.0.smithi045.stdout:1448: copy_from oid 19 from oid 46 current snap is 133 2021-07-31T01:50:23.596 INFO:tasks.rados.rados.0.smithi045.stdout:1449: delete oid 44 current snap is 133 2021-07-31T01:50:23.598 INFO:tasks.rados.rados.0.smithi045.stdout:1449: done (3 left) 2021-07-31T01:50:23.598 INFO:tasks.rados.rados.0.smithi045.stdout:1450: read oid 12 snap -1 2021-07-31T01:50:23.599 INFO:tasks.rados.rados.0.smithi045.stdout:1450: expect (ObjNum 0 snap 0 seq_num 0) 2021-07-31T01:50:23.600 INFO:tasks.rados.rados.0.smithi045.stdout:1451: snap_remove snap 129 2021-07-31T01:50:23.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:23 smithi045 conmon[39718]: cluster 2021-07-31T01:50:22.585265+0000 mon.a (mon.0) 967 2021-07-31T01:50:23.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:23 smithi045 conmon[39718]: : cluster [DBG] osdmap e323: 8 total, 8 up, 8 in 2021-07-31T01:50:23.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:50:23 smithi045 conmon[61210]: level=error ts=2021-07-31T01:50:23.514Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:50:23.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:50:23 smithi045 conmon[61210]: level=error ts=2021-07-31T01:50:23.514Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:50:23.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:50:23 smithi045 conmon[61210]: level=error ts=2021-07-31T01:50:23.514Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T01:50:23.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:23 smithi045 conmon[33655]: cluster 2021-07-31T01:50:22.585265+0000 mon.a ( 2021-07-31T01:50:23.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:23 smithi045 conmon[33655]: mon.0) 967 : cluster [DBG] osdmap e323: 8 total, 8 up, 8 in 2021-07-31T01:50:24.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:23 smithi154 conmon[34553]: cluster 2021-07-31T01:50:22.585265+0000 mon.a (mon.0) 967 : cluster [DBG] osdmap e323: 8 total, 8 up, 8 in 2021-07-31T01:50:24.597 INFO:tasks.rados.rados.0.smithi045.stdout:1451: done (4 left) 2021-07-31T01:50:24.598 INFO:tasks.rados.rados.0.smithi045.stdout:1452: copy_from oid 7 from oid 17 current snap is 133 2021-07-31T01:50:24.598 INFO:tasks.rados.rados.0.smithi045.stdout:1450: expect (ObjNum 0 snap 0 seq_num 0) 2021-07-31T01:50:24.599 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 50 v 546 (ObjNum -1105267744 snap 21873 seq_num 3190187248) dirty exists 2021-07-31T01:50:24.599 INFO:tasks.rados.rados.0.smithi045.stdout:1447: finishing rollback tid 0 to smithi04569269-32 2021-07-31T01:50:24.600 INFO:tasks.rados.rados.0.smithi045.stdout:1447: finishing rollback tid 1 to smithi04569269-32 2021-07-31T01:50:24.600 INFO:tasks.rados.rados.0.smithi045.stdout:1447: finishing rollback tid 2 to smithi04569269-32 2021-07-31T01:50:24.600 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 32 v 389 (ObjNum 429 snap 129 seq_num 429) dirty exists 2021-07-31T01:50:24.600 INFO:tasks.rados.rados.0.smithi045.stdout:1448: finishing copy_from to smithi04569269-19 2021-07-31T01:50:24.601 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 19 v 267 (ObjNum 454 snap 133 seq_num 454) dirty exists 2021-07-31T01:50:24.601 INFO:tasks.rados.rados.0.smithi045.stdout:1448: finishing copy_from racing read to smithi04569269-19 2021-07-31T01:50:24.601 INFO:tasks.rados.rados.0.smithi045.stdout:1446: done (4 left) 2021-07-31T01:50:24.601 INFO:tasks.rados.rados.0.smithi045.stdout:1447: done (3 left) 2021-07-31T01:50:24.601 INFO:tasks.rados.rados.0.smithi045.stdout:1448: done (2 left) 2021-07-31T01:50:24.602 INFO:tasks.rados.rados.0.smithi045.stdout:1450: done (1 left) 2021-07-31T01:50:24.602 INFO:tasks.rados.rados.0.smithi045.stdout:1453: snap_remove snap 130 2021-07-31T01:50:25.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:24 smithi154 conmon[34553]: cluster 2021-07-31T01:50:23.591087+0000 mon.a (mon.0) 968 : cluster [DBG] 2021-07-31T01:50:25.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:24 smithi154 conmon[34553]: osdmap e324: 8 total, 8 up, 8 in 2021-07-31T01:50:25.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:24 smithi154 conmon[34553]: cluster 2021-07-31T01:50:23.835841+0000 mgr.x (mgr.24542) 130 : cluster [DBG] pgmap v266: 25 pgs: 25 active+clean; 130 MiB data, 908 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 3.2 MiB/s wr, 28 op/s 2021-07-31T01:50:25.027 INFO:journalctl@ceph.osd.3.smithi045.stdout:Jul 31 01:50:24 smithi045 conmon[54013]: debug 2021-07-31T01:50:24.605+0000 7f64fa0c8700 -1 osd.3 pg_epoch: 325 pg[3.8s0( v 322'193 (0'0,322'193] local-lis/les=56/57 n=3 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [3,1,7]p3(0) r=0 lpr=56 crt=322'193 lcod 322'191 mlcod 322'191 active+clean+snaptrim trimq=[103~1] ps=[fb~1,105~1,108~1]] removing snap head 2021-07-31T01:50:25.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:24 smithi045 conmon[39718]: cluster 2021-07-31T01:50:23.591087+0000 2021-07-31T01:50:25.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:24 smithi045 conmon[39718]: mon.a (mon.0) 968 : cluster [DBG] osdmap e324: 8 total, 8 up, 8 in 2021-07-31T01:50:25.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:24 smithi045 conmon[39718]: cluster 2021-07-31T01:50: 2021-07-31T01:50:25.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:24 smithi045 conmon[39718]: 23.835841+0000 mgr.x (mgr.24542) 130 : cluster [DBG] pgmap v266: 25 pgs: 25 active+clean; 130 MiB data, 908 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 3.2 MiB/s wr, 28 op/s 2021-07-31T01:50:25.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:24 smithi045 conmon[33655]: cluster 2021-07-31T01:50:23.591087+0000 mon.a (mon.0 2021-07-31T01:50:25.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:24 smithi045 conmon[33655]: ) 968 : cluster [DBG] osdmap e324: 8 total, 8 up, 8 in 2021-07-31T01:50:25.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:24 smithi045 conmon[33655]: cluster 2021-07-31T01:50:23.835841+0000 mgr.x (mgr.24542) 130 : cluster [DBG] pgmap v266: 25 pgs: 25 active+clean; 130 MiB data, 908 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 3.2 MiB/s wr, 28 op/s 2021-07-31T01:50:25.603 INFO:tasks.rados.rados.0.smithi045.stdout:1453: done (1 left) 2021-07-31T01:50:25.604 INFO:tasks.rados.rados.0.smithi045.stdout:1454: read oid 32 snap -1 2021-07-31T01:50:25.604 INFO:tasks.rados.rados.0.smithi045.stdout:1454: expect (ObjNum 429 snap 129 seq_num 429) 2021-07-31T01:50:25.604 INFO:tasks.rados.rados.0.smithi045.stdout:1455: rollback oid 16 current snap is 133 2021-07-31T01:50:25.605 INFO:tasks.rados.rados.0.smithi045.stdout:1455: done (2 left) 2021-07-31T01:50:25.605 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 37 current snap is 133 2021-07-31T01:50:25.605 INFO:tasks.rados.rados.0.smithi045.stdout:1452: finishing copy_from to smithi04569269-7 2021-07-31T01:50:25.605 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 7 v 543 (ObjNum 447 snap 132 seq_num 447) dirty exists 2021-07-31T01:50:25.606 INFO:tasks.rados.rados.0.smithi045.stdout:1456: seq_num 457 ranges {483328=458752} 2021-07-31T01:50:25.609 INFO:tasks.rados.rados.0.smithi045.stdout:1456: writing smithi04569269-37 from 483328 to 942080 tid 1 2021-07-31T01:50:25.609 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 19 current snap is 133 2021-07-31T01:50:25.609 INFO:tasks.rados.rados.0.smithi045.stdout:1452: finishing copy_from racing read to smithi04569269-7 2021-07-31T01:50:25.609 INFO:tasks.rados.rados.0.smithi045.stdout:1457: seq_num 458 ranges {7979008=475136} 2021-07-31T01:50:25.657 INFO:tasks.rados.rados.0.smithi045.stdout:1457: writing smithi04569269-19 from 7979008 to 8454144 tid 1 2021-07-31T01:50:25.658 INFO:tasks.rados.rados.0.smithi045.stdout:1452: done (3 left) 2021-07-31T01:50:25.658 INFO:tasks.rados.rados.0.smithi045.stdout:1458: snap_create 2021-07-31T01:50:25.658 INFO:tasks.rados.rados.0.smithi045.stdout:1456: finishing write tid 1 to smithi04569269-37 2021-07-31T01:50:25.658 INFO:tasks.rados.rados.0.smithi045.stdout:1456: finishing write tid 2 to smithi04569269-37 2021-07-31T01:50:25.659 INFO:tasks.rados.rados.0.smithi045.stdout:1456: finishing write tid 3 to smithi04569269-37 2021-07-31T01:50:25.659 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 37 v 355 (ObjNum 457 snap 133 seq_num 457) dirty exists 2021-07-31T01:50:25.659 INFO:tasks.rados.rados.0.smithi045.stdout:1456: left oid 37 (ObjNum 457 snap 133 seq_num 457) 2021-07-31T01:50:25.659 INFO:tasks.rados.rados.0.smithi045.stdout:1454: expect (ObjNum 429 snap 129 seq_num 429) 2021-07-31T01:50:25.897 INFO:tasks.rados.rados.0.smithi045.stdout:1457: finishing write tid 1 to smithi04569269-19 2021-07-31T01:50:25.897 INFO:tasks.rados.rados.0.smithi045.stdout:1457: finishing write tid 2 to smithi04569269-19 2021-07-31T01:50:25.897 INFO:tasks.rados.rados.0.smithi045.stdout:1457: finishing write tid 3 to smithi04569269-19 2021-07-31T01:50:25.897 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 19 v 500 (ObjNum 458 snap 133 seq_num 458) dirty exists 2021-07-31T01:50:25.898 INFO:tasks.rados.rados.0.smithi045.stdout:1457: left oid 19 (ObjNum 458 snap 133 seq_num 458) 2021-07-31T01:50:26.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:25 smithi154 conmon[34553]: cluster 2021-07-31T01:50:24.596653+0000 mon.a (mon.0) 969 : cluster [DBG] osdmap e325: 8 total, 8 up, 8 in 2021-07-31T01:50:26.003 INFO:journalctl@ceph.osd.6.smithi154.stdout:Jul 31 01:50:25 smithi154 conmon[45576]: debug 2021-07-31T01:50:26.003 INFO:journalctl@ceph.osd.6.smithi154.stdout:Jul 31 01:50:25 smithi154 conmon[45576]: 2021-07-31T01:50:25.615+0000 7f47c10a5700 -1 osd.6 pg_epoch: 326 pg[3.as0( v 325'416 (0'0,325'416] local-lis/les=56/57 n=4 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [6,4,1]p6(0) r=0 lpr=56 crt=325'416 lcod 324'414 mlcod 324'414 active+clean+snaptrim trimq=[104~1] ps=[fb~1,103~1,105~1,108~1]] removing snap head 2021-07-31T01:50:26.004 INFO:journalctl@ceph.osd.7.smithi154.stdout:Jul 31 01:50:25 smithi154 conmon[49149]: debug 2021-07-31T01:50:26.004 INFO:journalctl@ceph.osd.7.smithi154.stdout:Jul 31 01:50:25 smithi154 conmon[49149]: 2021-07-31T01:50:25.609+0000 7f8a711fb700 -1 osd.7 pg_epoch: 326 pg[3.es0( v 325'509 (0'0,325'509] local-lis/les=56/57 n=5 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [7,4,1]p7(0) r=0 lpr=56 crt=325'509 lcod 322'507 mlcod 322'507 active+clean+snaptrim trimq=[104~1] ps=[fb~1,103~1,105~1,108~1]] removing snap head 2021-07-31T01:50:26.004 INFO:journalctl@ceph.osd.7.smithi154.stdout:Jul 31 01:50:25 smithi154 conmon[49149]: 2021-07-31T01:50:26.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:25 smithi045 conmon[39718]: cluster 2021-07-31T01:50:24.596653+0000 mon.a (mon.0) 969 : cluster [DBG] osdmap e325: 8 total, 8 up, 8 in 2021-07-31T01:50:26.026 INFO:journalctl@ceph.osd.0.smithi045.stdout:Jul 31 01:50:25 smithi045 conmon[43243]: debug 2021-07-31T01:50:26.027 INFO:journalctl@ceph.osd.0.smithi045.stdout:Jul 31 01:50:25 smithi045 conmon[43243]: 2021-07-31T01:50:25.609+0000 7fdf6d793700 -1 osd.0 pg_epoch: 326 pg[3.6s0( v 325'172 (0'0,325'172] local-lis/les=56/57 n=3 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [0,1,4]p0(0) r=0 lpr=56 crt=325'172 lcod 324'170 mlcod 324'170 active+clean+snaptrim trimq=[104~1] ps=[fb~1,103~1,105~1,108~1]] removing snap head 2021-07-31T01:50:26.027 INFO:journalctl@ceph.osd.0.smithi045.stdout:Jul 31 01:50:25 smithi045 conmon[43243]: 2021-07-31T01:50:26.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:25 smithi045 conmon[33655]: cluster 2021-07-31T01:50:24.596653+0000 mon.a (mon 2021-07-31T01:50:26.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:25 smithi045 conmon[33655]: .0) 969 : cluster [DBG] osdmap e325: 8 total, 8 up, 8 in 2021-07-31T01:50:26.539 INFO:tasks.rados.rados.0.smithi045.stdout:1454: done (3 left) 2021-07-31T01:50:26.539 INFO:tasks.rados.rados.0.smithi045.stdout:1456: done (2 left) 2021-07-31T01:50:26.539 INFO:tasks.rados.rados.0.smithi045.stdout:1457: done (1 left) 2021-07-31T01:50:26.540 INFO:tasks.rados.rados.0.smithi045.stdout:1458: done (0 left) 2021-07-31T01:50:26.540 INFO:tasks.rados.rados.0.smithi045.stdout:1459: read oid 26 snap -1 2021-07-31T01:50:26.540 INFO:tasks.rados.rados.0.smithi045.stdout:1459: expect (ObjNum 443 snap 129 seq_num 443) 2021-07-31T01:50:26.540 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 49 current snap is 134 2021-07-31T01:50:26.540 INFO:tasks.rados.rados.0.smithi045.stdout:1460: seq_num 459 ranges {393216=491520} 2021-07-31T01:50:26.543 INFO:tasks.rados.rados.0.smithi045.stdout:1460: writing smithi04569269-49 from 393216 to 884736 tid 1 2021-07-31T01:50:26.544 INFO:tasks.rados.rados.0.smithi045.stdout:1461: copy_from oid 36 from oid 47 current snap is 134 2021-07-31T01:50:26.544 INFO:tasks.rados.rados.0.smithi045.stdout:1462: rollback oid 32 current snap is 134 2021-07-31T01:50:26.544 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 32 to 133 2021-07-31T01:50:26.544 INFO:tasks.rados.rados.0.smithi045.stdout:1463: rollback oid 33 current snap is 134 2021-07-31T01:50:26.544 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 33 to 133 2021-07-31T01:50:26.545 INFO:tasks.rados.rados.0.smithi045.stdout:1464: delete oid 35 current snap is 134 2021-07-31T01:50:26.545 INFO:tasks.rados.rados.0.smithi045.stdout:1459: expect (ObjNum 443 snap 129 seq_num 443) 2021-07-31T01:50:26.557 INFO:tasks.rados.rados.0.smithi045.stdout:1459: done (5 left) 2021-07-31T01:50:26.558 INFO:tasks.rados.rados.0.smithi045.stdout:1464: done (4 left) 2021-07-31T01:50:26.558 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 40 current snap is 134 2021-07-31T01:50:26.558 INFO:tasks.rados.rados.0.smithi045.stdout:1462: finishing rollback tid 0 to smithi04569269-32 2021-07-31T01:50:26.558 INFO:tasks.rados.rados.0.smithi045.stdout:1465: seq_num 461 ranges {13328384=524288} 2021-07-31T01:50:26.633 INFO:tasks.rados.rados.0.smithi045.stdout:1465: writing smithi04569269-40 from 13328384 to 13852672 tid 1 2021-07-31T01:50:26.634 INFO:tasks.rados.rados.0.smithi045.stdout:1466: read oid 44 snap -1 2021-07-31T01:50:26.634 INFO:tasks.rados.rados.0.smithi045.stdout:1466: expect deleted 2021-07-31T01:50:26.634 INFO:tasks.rados.rados.0.smithi045.stdout:1467: read oid 23 snap 133 2021-07-31T01:50:26.634 INFO:tasks.rados.rados.0.smithi045.stdout:1467: expect deleted 2021-07-31T01:50:26.635 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 24 current snap is 134 2021-07-31T01:50:26.635 INFO:tasks.rados.rados.0.smithi045.stdout:1468: seq_num 462 ranges {2441216=540672} 2021-07-31T01:50:26.650 INFO:tasks.rados.rados.0.smithi045.stdout:1468: writing smithi04569269-24 from 2441216 to 2981888 tid 1 2021-07-31T01:50:26.650 INFO:tasks.rados.rados.0.smithi045.stdout:1469: delete oid 10 current snap is 134 2021-07-31T01:50:26.650 INFO:tasks.rados.rados.0.smithi045.stdout:1463: finishing rollback tid 0 to smithi04569269-33 2021-07-31T01:50:26.651 INFO:tasks.rados.rados.0.smithi045.stdout:1462: finishing rollback tid 1 to smithi04569269-32 2021-07-31T01:50:26.651 INFO:tasks.rados.rados.0.smithi045.stdout:1462: finishing rollback tid 2 to smithi04569269-32 2021-07-31T01:50:26.651 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 32 v 392 (ObjNum 429 snap 129 seq_num 429) dirty exists 2021-07-31T01:50:26.651 INFO:tasks.rados.rados.0.smithi045.stdout:1463: finishing rollback tid 1 to smithi04569269-33 2021-07-31T01:50:26.652 INFO:tasks.rados.rados.0.smithi045.stdout:1463: finishing rollback tid 2 to smithi04569269-33 2021-07-31T01:50:26.652 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 33 v 591 (ObjNum 399 snap 121 seq_num 399) dirty exists 2021-07-31T01:50:26.652 INFO:tasks.rados.rados.0.smithi045.stdout:1460: finishing write tid 1 to smithi04569269-49 2021-07-31T01:50:26.652 INFO:tasks.rados.rados.0.smithi045.stdout:1460: finishing write tid 2 to smithi04569269-49 2021-07-31T01:50:26.653 INFO:tasks.rados.rados.0.smithi045.stdout:1461: finishing copy_from to smithi04569269-36 2021-07-31T01:50:26.653 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 36 v 272 (ObjNum 172 snap 41 seq_num 172) dirty exists 2021-07-31T01:50:26.653 INFO:tasks.rados.rados.0.smithi045.stdout:1461: finishing copy_from racing read to smithi04569269-36 2021-07-31T01:50:26.653 INFO:tasks.rados.rados.0.smithi045.stdout:1460: finishing write tid 3 to smithi04569269-49 2021-07-31T01:50:26.653 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 49 v 974 (ObjNum 459 snap 134 seq_num 459) dirty exists 2021-07-31T01:50:26.654 INFO:tasks.rados.rados.0.smithi045.stdout:1460: left oid 49 (ObjNum 459 snap 134 seq_num 459) 2021-07-31T01:50:26.654 INFO:tasks.rados.rados.0.smithi045.stdout:1465: finishing write tid 1 to smithi04569269-40 2021-07-31T01:50:26.654 INFO:tasks.rados.rados.0.smithi045.stdout:1465: finishing write tid 2 to smithi04569269-40 2021-07-31T01:50:26.654 INFO:tasks.rados.rados.0.smithi045.stdout:1465: finishing write tid 3 to smithi04569269-40 2021-07-31T01:50:26.655 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 40 v 574 (ObjNum 461 snap 134 seq_num 461) dirty exists 2021-07-31T01:50:26.655 INFO:tasks.rados.rados.0.smithi045.stdout:1465: left oid 40 (ObjNum 461 snap 134 seq_num 461) 2021-07-31T01:50:26.655 INFO:tasks.rados.rados.0.smithi045.stdout:1460: done (8 left) 2021-07-31T01:50:26.656 INFO:tasks.rados.rados.0.smithi045.stdout:1461: done (7 left) 2021-07-31T01:50:26.656 INFO:tasks.rados.rados.0.smithi045.stdout:1462: done (6 left) 2021-07-31T01:50:26.656 INFO:tasks.rados.rados.0.smithi045.stdout:1463: done (5 left) 2021-07-31T01:50:26.656 INFO:tasks.rados.rados.0.smithi045.stdout:1465: done (4 left) 2021-07-31T01:50:26.656 INFO:tasks.rados.rados.0.smithi045.stdout:1466: done (3 left) 2021-07-31T01:50:26.657 INFO:tasks.rados.rados.0.smithi045.stdout:1467: done (2 left) 2021-07-31T01:50:26.657 INFO:tasks.rados.rados.0.smithi045.stdout:1469: done (1 left) 2021-07-31T01:50:26.657 INFO:tasks.rados.rados.0.smithi045.stdout:1470: snap_remove snap 133 2021-07-31T01:50:27.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:26 smithi154 conmon[34553]: cluster 2021-07-31T01:50:25.602308+0000 mon.a (mon.0) 970 : cluster [DBG] osdmap e326: 8 total, 8 up, 8 in 2021-07-31T01:50:27.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:26 smithi154 conmon[34553]: cluster 2021-07-31T01:50:25.836279+0000 mgr.x (mgr.24542) 131 : cluster [DBG] pgmap v269: 25 pgs: 25 active+clean; 130 MiB data, 908 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 3.2 MiB/s wr, 28 op/s 2021-07-31T01:50:27.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:26 smithi154 conmon[34553]: cluster 2021-07-31T01:50:26.537826+0000 mon.a (mon.0) 971 : cluster [DBG] osdmap e327: 8 total, 8 up, 8 in 2021-07-31T01:50:27.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:26 smithi045 conmon[33655]: cluster 2021-07-31T01:50:25. 2021-07-31T01:50:27.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:26 smithi045 conmon[33655]: 602308+0000 mon.a (mon.0) 970 : cluster [DBG] osdmap e326: 8 total, 8 up, 8 in 2021-07-31T01:50:27.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:26 smithi045 conmon[33655]: cluster 2021-07-31T01:50:25.836279+0000 mgr.x (mgr.24542) 131 2021-07-31T01:50:27.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:26 smithi045 conmon[33655]: : cluster [DBG] pgmap v269: 25 pgs: 25 active+clean; 130 MiB data, 908 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 3.2 MiB/s wr, 28 op/s 2021-07-31T01:50:27.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:26 smithi045 conmon[33655]: cluster 2021-07-31T01:50:26.537826+0000 2021-07-31T01:50:27.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:26 smithi045 conmon[33655]: mon.a (mon.0) 971 : cluster [DBG] osdmap e327: 8 total, 8 up, 8 in 2021-07-31T01:50:27.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:26 smithi045 conmon[39718]: cluster 2021-07-31T01:50:25.602308+0000 mon.a (mon.0 2021-07-31T01:50:27.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:26 smithi045 conmon[39718]: ) 970 : cluster [DBG] osdmap e326: 8 total, 8 up, 8 in 2021-07-31T01:50:27.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:26 smithi045 conmon[39718]: cluster 2021-07-31T01:50:25.836279+0000 mgr.x (mgr 2021-07-31T01:50:27.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:26 smithi045 conmon[39718]: .24542) 131 : cluster [DBG] pgmap v269: 25 pgs: 25 active+clean; 130 MiB data, 908 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 3.2 MiB/s wr, 28 op/s 2021-07-31T01:50:27.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:26 smithi045 conmon[39718]: cluster 2021-07-31T01:50:27.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:26 smithi045 conmon[39718]: 2021-07-31T01:50:26.537826+0000 mon.a (mon.0) 971 2021-07-31T01:50:27.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:26 smithi045 conmon[39718]: : cluster [DBG] osdmap e327: 8 total, 8 up, 8 in 2021-07-31T01:50:27.610 INFO:tasks.rados.rados.0.smithi045.stdout:1470: done (1 left) 2021-07-31T01:50:27.610 INFO:tasks.rados.rados.0.smithi045.stdout:1471: read oid 44 snap -1 2021-07-31T01:50:27.611 INFO:tasks.rados.rados.0.smithi045.stdout:1471: expect deleted 2021-07-31T01:50:27.611 INFO:tasks.rados.rados.0.smithi045.stdout:1472: setattr oid 28 current snap is 134 2021-07-31T01:50:27.611 INFO:tasks.rados.rados.0.smithi045.stdout:1468: finishing write tid 1 to smithi04569269-24 2021-07-31T01:50:27.611 INFO:tasks.rados.rados.0.smithi045.stdout:1468: finishing write tid 2 to smithi04569269-24 2021-07-31T01:50:27.612 INFO:tasks.rados.rados.0.smithi045.stdout:1468: finishing write tid 3 to smithi04569269-24 2021-07-31T01:50:27.612 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 24 v 508 (ObjNum 462 snap 134 seq_num 462) dirty exists 2021-07-31T01:50:27.612 INFO:tasks.rados.rados.0.smithi045.stdout:1468: left oid 24 (ObjNum 462 snap 134 seq_num 462) 2021-07-31T01:50:27.612 INFO:tasks.rados.rados.0.smithi045.stdout:1468: done (2 left) 2021-07-31T01:50:27.613 INFO:tasks.rados.rados.0.smithi045.stdout:1471: done (1 left) 2021-07-31T01:50:27.613 INFO:tasks.rados.rados.0.smithi045.stdout:1473: snap_create 2021-07-31T01:50:27.620 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 28 v 554 (ObjNum 333 snap 97 seq_num 333) dirty exists 2021-07-31T01:50:28.026 INFO:journalctl@ceph.osd.3.smithi045.stdout:Jul 31 01:50:27 smithi045 conmon[54013]: debug 2021-07-31T01:50:27.617+0000 7f64f88c5700 -1 osd.3 pg_epoch: 328 pg[3.bs0( v 327'978 (0'0,327'978] local-lis/les=56/57 n=8 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [3,0,4]p3(0) r=0 lpr=56 crt=327'978 lcod 327'976 mlcod 327'976 active+clean+snaptrim trimq=[10c~1] ps=[103~2]] removing snap head 2021-07-31T01:50:28.617 INFO:tasks.rados.rados.0.smithi045.stdout:1472: done (1 left) 2021-07-31T01:50:28.618 INFO:tasks.rados.rados.0.smithi045.stdout:1473: done (0 left) 2021-07-31T01:50:28.618 INFO:tasks.rados.rados.0.smithi045.stdout:1474: snap_remove snap 134 2021-07-31T01:50:28.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:28 smithi045 conmon[39718]: cluster 2021-07-31T01:50:27.609628+0000 mon.a (mon 2021-07-31T01:50:28.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:28 smithi045 conmon[39718]: .0) 972 : cluster [DBG] osdmap e328: 8 total, 8 up, 8 in 2021-07-31T01:50:28.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:28 smithi045 conmon[39718]: cluster 2021-07-31T01:50:27 2021-07-31T01:50:28.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:28 smithi045 conmon[39718]: .836656+0000 mgr.x (mgr.24542) 132 : cluster [DBG] pgmap v272: 25 pgs: 25 active+clean; 130 MiB data, 908 MiB used, 714 GiB / 715 GiB avail 2021-07-31T01:50:28.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:28 smithi045 conmon[39718]: audit 2021 2021-07-31T01:50:28.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:28 smithi045 conmon[39718]: -07-31T01:50:27.953192 2021-07-31T01:50:28.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:28 smithi045 conmon[39718]: +0000 mon.b (mon.2) 77 : audit 2021-07-31T01:50:28.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:28 smithi045 conmon[39718]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:50:28.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:28 smithi045 conmon[39718]: audit 2021-07-31T01:50:28.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:28 smithi045 conmon[39718]: 2021-07-31T01:50:27 2021-07-31T01:50:28.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:28 smithi045 conmon[39718]: .953911+0000 mon.a (mon 2021-07-31T01:50:28.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:28 smithi045 conmon[39718]: .0) 973 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:50:28.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:28 smithi045 conmon[39718]: audit 2021-07-31T01 2021-07-31T01:50:28.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:28 smithi045 conmon[39718]: :50:27.963771+0000 mon.b (mon.2) 2021-07-31T01:50:28.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:28 smithi045 conmon[39718]: 78 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T01:50:28.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:28 smithi045 conmon[39718]: audit 2021- 2021-07-31T01:50:28.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:28 smithi045 conmon[39718]: 07-31T01:50:27.964370 2021-07-31T01:50:28.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:28 smithi045 conmon[39718]: +0000 mon.a (mon.0) 974 : audit 2021-07-31T01:50:28.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:28 smithi045 conmon[39718]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T01:50:28.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:28 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T01:50:28.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:28 smithi045 conmon[39718]: T01:50:28.496120+0000 mon.b ( 2021-07-31T01:50:28.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:28 smithi045 conmon[39718]: mon.2) 79 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:50:28.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:28 smithi045 conmon[39718]: audit 2021 2021-07-31T01:50:28.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:28 smithi045 conmon[39718]: -07-31T01:50:28. 2021-07-31T01:50:28.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:28 smithi045 conmon[39718]: 497618+0000 mon.b (mon.2) 80 : 2021-07-31T01:50:28.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:28 smithi045 conmon[39718]: audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:50:28.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:28 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T01:50:28.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:28 smithi045 conmon[39718]: 01:50:28.498670+0000 mon.b (mon 2021-07-31T01:50:28.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:28 smithi045 conmon[39718]: .2) 81 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:50:28.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:28 smithi045 conmon[33655]: cluster 2021-07-31T01:50:27.609628+0000 mon.a (mon.0) 972 : cluster 2021-07-31T01:50:28.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:28 smithi045 conmon[33655]: [DBG] osdmap e328: 8 total, 8 up, 8 in 2021-07-31T01:50:28.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:28 smithi045 conmon[33655]: cluster 2021-07-31T01:50:27.836656+0000 mgr.x (mgr.24542) 132 : cluster [DBG] pgmap v272: 25 pgs: 25 active+clean; 130 MiB data, 908 MiB used, 714 GiB / 715 GiB avail 2021-07-31T01:50:28.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:28 smithi045 conmon[33655]: audit 2021-07-31T01:50:27.953192+0000 mon.b (mon.2) 77 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:50:28.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:28 smithi045 conmon[33655]: audit 2021-07-31T01:50:27. 2021-07-31T01:50:28.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:28 smithi045 conmon[33655]: 953911+0000 mon.a (mon.0) 973 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:50:28.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:28 smithi045 conmon[33655]: audit 2021-07-31 2021-07-31T01:50:28.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:28 smithi045 conmon[33655]: T01:50:27.963771+0000 mon.b (mon.2) 78 : audit 2021-07-31T01:50:28.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:28 smithi045 conmon[33655]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T01:50:28.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:28 smithi045 conmon[33655]: audit 2021-07-31T01:50:28.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:28 smithi045 conmon[33655]: 2021-07-31T01:50:27.964370+0000 mon.a (mon.0) 974 : audit 2021-07-31T01:50:28.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:28 smithi045 conmon[33655]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T01:50:28.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:28 smithi045 conmon[33655]: audit 2021-07- 2021-07-31T01:50:28.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:28 smithi045 conmon[33655]: 31T01:50:28.496120+0000 mon.b (mon.2) 79 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:50:28.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:28 smithi045 conmon[33655]: audit 2021-07-31T01 2021-07-31T01:50:28.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:28 smithi045 conmon[33655]: :50:28.497618+0000 mon.b (mon.2) 80 : audit 2021-07-31T01:50:28.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:28 smithi045 conmon[33655]: [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:50:28.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:28 smithi045 conmon[33655]: audit 2021-07-31T01:50:28.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:28 smithi045 conmon[33655]: 2021-07-31T01:50:28.498670+0000 mon.b (mon.2) 81 2021-07-31T01:50:28.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:28 smithi045 conmon[33655]: : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:50:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[34553]: cluster 2021-07-31T01:50:27.609628+0000 mon.a (mon.0) 972 : 2021-07-31T01:50:29.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[34553]: cluster [DBG] osdmap e328: 8 total, 8 up, 8 in 2021-07-31T01:50:29.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[34553]: cluster 2021-07-31T01:50:27.836656+0000 mgr.x (mgr.24542) 132 : cluster [DBG] pgmap v272: 25 pgs: 25 active+clean; 130 MiB data, 908 MiB used, 714 GiB / 715 GiB avail 2021-07-31T01:50:29.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[34553]: audit 2021-07-31T01:50:27.953192+0000 mon.b (mon.2) 77 : audit 2021-07-31T01:50:29.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[34553]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:50:29.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[34553]: audit 2021-07-31T01:50:29.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[34553]: 2021-07-31T01:50:27.953911+0000 2021-07-31T01:50:29.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[34553]: mon.a (mon.0) 973 : audit 2021-07-31T01:50:29.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[34553]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:50:29.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[34553]: audit 2021-07-31T01:50:27.963771+0000 mon.b ( 2021-07-31T01:50:29.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[34553]: mon.2) 78 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T01:50:29.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[34553]: audit 2021-07- 2021-07-31T01:50:29.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[34553]: 31T01:50:27.964370+0000 mon.a ( 2021-07-31T01:50:29.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[34553]: mon.0) 974 : audit [INF] 2021-07-31T01:50:29.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[34553]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T01:50:29.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[34553]: audit 2021-07-31T01:50:29.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[34553]: 2021-07-31T01:50:28 2021-07-31T01:50:29.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[34553]: .496120+0000 mon.b (mon.2) 79 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:50:29.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[34553]: audit 2021-07- 2021-07-31T01:50:29.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[34553]: 31T01:50:28.497618+0000 mon.b (mon.2) 80 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:50:29.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[34553]: 2021-07-31T01:50:29.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[34553]: audit 2021-07-31T01:50:29.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[34553]: 2021-07-31T01:50:28.498670+0000 mon.b (mon.2) 81 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:50:29.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[34553]: 2021-07-31T01:50:29.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: debug 2021-07-31T01:50:28.541+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T01:50:29.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:50:29.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:50:29.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: yield 2021-07-31T01:50:29.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:50:29.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:50:29.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:50:29.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T01:50:29.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:50:29.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T01:50:29.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:50:29.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: await self._waiter 2021-07-31T01:50:29.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:50:29.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: task.result() 2021-07-31T01:50:29.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T01:50:29.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: debug 2021-07-31T01:50:28.553+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T01:50:29.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:50:29.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:50:29.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: yield 2021-07-31T01:50:29.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:50:29.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:50:29.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:50:29.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T01:50:29.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:50:29.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T01:50:29.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:50:29.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: await self._waiter 2021-07-31T01:50:29.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:50:29.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: task.result() 2021-07-31T01:50:29.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T01:50:29.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: debug 2021-07-31T01:50:28.579+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T01:50:29.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:50:29.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:50:29.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: yield 2021-07-31T01:50:29.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:50:29.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:50:29.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:50:29.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T01:50:29.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:50:29.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T01:50:29.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:50:29.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: await self._waiter 2021-07-31T01:50:29.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:50:29.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: task.result() 2021-07-31T01:50:29.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T01:50:29.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: debug 2021-07-31T01:50:28.580+0000 7fcb9115c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:50:29.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:50:29.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:50:29.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: yield 2021-07-31T01:50:29.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:50:29.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:50:29.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:50:29.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T01:50:29.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:50:29.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T01:50:29.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:50:29.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: await self._waiter 2021-07-31T01:50:29.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:50:29.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: task.result() 2021-07-31T01:50:29.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T01:50:29.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: 2021-07-31T01:50:29.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T01:50:29.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: 2021-07-31T01:50:29.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:50:29.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:50:29.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: return f(*arg) 2021-07-31T01:50:29.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:50:29.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:50:29.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:50:29.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:50:29.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:50:29.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:50:29.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:50:29.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T01:50:29.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:50:29.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: raise self._exception 2021-07-31T01:50:29.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:50:29.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:50:29.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:50:29.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T01:50:29.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:50:29.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: raise 2021-07-31T01:50:29.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:50:29.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T01:50:29.035 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:50:29.035 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:50:29.035 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T01:50:29.035 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: debug 2021-07-31T01:50:28.588+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T01:50:29.035 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:50:29.036 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:50:29.036 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: yield 2021-07-31T01:50:29.036 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:50:29.036 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:50:29.037 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:50:29.037 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T01:50:29.037 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:50:29.037 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T01:50:29.037 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:50:29.038 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: await self._waiter 2021-07-31T01:50:29.038 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:50:29.038 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: task.result() 2021-07-31T01:50:29.038 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T01:50:29.039 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: debug 2021-07-31T01:50:28.589+0000 7fcb9095b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:50:29.039 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:50:29.039 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:50:29.039 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: yield 2021-07-31T01:50:29.039 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:50:29.040 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:50:29.040 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:50:29.040 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T01:50:29.040 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:50:29.040 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T01:50:29.041 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:50:29.041 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: await self._waiter 2021-07-31T01:50:29.041 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:50:29.041 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: task.result() 2021-07-31T01:50:29.042 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T01:50:29.042 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: 2021-07-31T01:50:29.042 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T01:50:29.042 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: 2021-07-31T01:50:29.043 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:50:29.043 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:50:29.043 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: return f(*arg) 2021-07-31T01:50:29.043 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:50:29.043 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:50:29.044 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:50:29.044 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:50:29.044 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:50:29.044 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:50:29.045 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:50:29.045 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T01:50:29.046 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:50:29.046 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: raise self._exception 2021-07-31T01:50:29.047 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:50:29.047 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:50:29.047 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:50:29.048 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T01:50:29.048 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:50:29.049 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: raise 2021-07-31T01:50:29.049 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:50:29.050 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T01:50:29.050 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:50:29.050 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:50:29.051 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:50:28 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T01:50:29.618 INFO:tasks.rados.rados.0.smithi045.stdout:1474: done (0 left) 2021-07-31T01:50:29.619 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 23 current snap is 135 2021-07-31T01:50:29.619 INFO:tasks.rados.rados.0.smithi045.stdout:1475: seq_num 464 ranges {0=573440} 2021-07-31T01:50:29.622 INFO:tasks.rados.rados.0.smithi045.stdout:1475: writing smithi04569269-23 from 0 to 573440 tid 1 2021-07-31T01:50:29.622 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 1 current snap is 135 2021-07-31T01:50:29.622 INFO:tasks.rados.rados.0.smithi045.stdout:1476: seq_num 465 ranges {1605632=598016} 2021-07-31T01:50:29.634 INFO:tasks.rados.rados.0.smithi045.stdout:1476: writing smithi04569269-1 from 1605632 to 2203648 tid 1 2021-07-31T01:50:29.635 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 32 current snap is 135 2021-07-31T01:50:29.636 INFO:tasks.rados.rados.0.smithi045.stdout:1475: finishing write tid 1 to smithi04569269-23 2021-07-31T01:50:29.636 INFO:tasks.rados.rados.0.smithi045.stdout:1477: seq_num 466 ranges {6053888=614400} 2021-07-31T01:50:29.672 INFO:tasks.rados.rados.0.smithi045.stdout:1477: writing smithi04569269-32 from 6053888 to 6668288 tid 1 2021-07-31T01:50:29.672 INFO:tasks.rados.rados.0.smithi045.stdout:1478: rollback oid 41 current snap is 135 2021-07-31T01:50:29.672 INFO:tasks.rados.rados.0.smithi045.stdout:1478: done (3 left) 2021-07-31T01:50:29.673 INFO:tasks.rados.rados.0.smithi045.stdout:1479: copy_from oid 28 from oid 44 current snap is 135 2021-07-31T01:50:29.673 INFO:tasks.rados.rados.0.smithi045.stdout:1475: finishing write tid 2 to smithi04569269-23 2021-07-31T01:50:29.673 INFO:tasks.rados.rados.0.smithi045.stdout:1475: finishing write tid 3 to smithi04569269-23 2021-07-31T01:50:29.673 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 23 v 593 (ObjNum 464 snap 135 seq_num 464) dirty exists 2021-07-31T01:50:29.674 INFO:tasks.rados.rados.0.smithi045.stdout:1475: left oid 23 (ObjNum 464 snap 135 seq_num 464) 2021-07-31T01:50:29.674 INFO:tasks.rados.rados.0.smithi045.stdout:1475: done (3 left) 2021-07-31T01:50:29.674 INFO:tasks.rados.rados.0.smithi045.stdout:1480: rollback oid 21 current snap is 135 2021-07-31T01:50:29.674 INFO:tasks.rados.rados.0.smithi045.stdout:1480: done (3 left) 2021-07-31T01:50:29.674 INFO:tasks.rados.rados.0.smithi045.stdout:1481: read oid 30 snap -1 2021-07-31T01:50:29.675 INFO:tasks.rados.rados.0.smithi045.stdout:1481: expect deleted 2021-07-31T01:50:29.675 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 4 current snap is 135 2021-07-31T01:50:29.675 INFO:tasks.rados.rados.0.smithi045.stdout:1476: finishing write tid 1 to smithi04569269-1 2021-07-31T01:50:29.675 INFO:tasks.rados.rados.0.smithi045.stdout:1476: finishing write tid 2 to smithi04569269-1 2021-07-31T01:50:29.676 INFO:tasks.rados.rados.0.smithi045.stdout:1476: finishing write tid 3 to smithi04569269-1 2021-07-31T01:50:29.676 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 1 v 905 (ObjNum 465 snap 135 seq_num 465) dirty exists 2021-07-31T01:50:29.676 INFO:tasks.rados.rados.0.smithi045.stdout:1476: left oid 1 (ObjNum 465 snap 135 seq_num 465) 2021-07-31T01:50:29.676 INFO:tasks.rados.rados.0.smithi045.stdout:1482: seq_num 467 ranges {1490944=630784} 2021-07-31T01:50:29.684 INFO:tasks.rados.rados.0.smithi045.stdout:1482: writing smithi04569269-4 from 1490944 to 2121728 tid 1 2021-07-31T01:50:29.684 INFO:tasks.rados.rados.0.smithi045.stdout:1476: done (4 left) 2021-07-31T01:50:29.685 INFO:tasks.rados.rados.0.smithi045.stdout:1483: read oid 34 snap -1 2021-07-31T01:50:29.685 INFO:tasks.rados.rados.0.smithi045.stdout:1483: expect (ObjNum 415 snap 124 seq_num 415) 2021-07-31T01:50:29.685 INFO:tasks.rados.rados.0.smithi045.stdout:1484: copy_from oid 3 from oid 17 current snap is 135 2021-07-31T01:50:29.685 INFO:tasks.rados.rados.0.smithi045.stdout:1479: finishing copy_from to smithi04569269-28 2021-07-31T01:50:29.685 INFO:tasks.rados.rados.0.smithi045.stdout:1479: got expected ENOENT (src dne) 2021-07-31T01:50:29.686 INFO:tasks.rados.rados.0.smithi045.stdout:1479: finishing copy_from racing read to smithi04569269-28 2021-07-31T01:50:29.686 INFO:tasks.rados.rados.0.smithi045.stdout:1477: finishing write tid 1 to smithi04569269-32 2021-07-31T01:50:29.686 INFO:tasks.rados.rados.0.smithi045.stdout:1477: finishing write tid 2 to smithi04569269-32 2021-07-31T01:50:29.686 INFO:tasks.rados.rados.0.smithi045.stdout:1477: finishing write tid 3 to smithi04569269-32 2021-07-31T01:50:29.687 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 32 v 394 (ObjNum 466 snap 135 seq_num 466) dirty exists 2021-07-31T01:50:29.687 INFO:tasks.rados.rados.0.smithi045.stdout:1477: left oid 32 (ObjNum 466 snap 135 seq_num 466) 2021-07-31T01:50:29.687 INFO:tasks.rados.rados.0.smithi045.stdout:1477: done (5 left) 2021-07-31T01:50:29.687 INFO:tasks.rados.rados.0.smithi045.stdout:1479: done (4 left) 2021-07-31T01:50:29.687 INFO:tasks.rados.rados.0.smithi045.stdout:1481: done (3 left) 2021-07-31T01:50:29.688 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 6 current snap is 135 2021-07-31T01:50:29.688 INFO:tasks.rados.rados.0.smithi045.stdout:1485: seq_num 468 ranges {0=647168} 2021-07-31T01:50:29.688 INFO:tasks.rados.rados.0.smithi045.stdout:1485: writing smithi04569269-6 from 0 to 647168 tid 1 2021-07-31T01:50:29.689 INFO:tasks.rados.rados.0.smithi045.stdout:1486: copy_from oid 28 from oid 44 current snap is 135 2021-07-31T01:50:29.689 INFO:tasks.rados.rados.0.smithi045.stdout:1487: snap_create 2021-07-31T01:50:29.690 INFO:tasks.rados.rados.0.smithi045.stdout:1486: finishing copy_from to smithi04569269-28 2021-07-31T01:50:29.690 INFO:tasks.rados.rados.0.smithi045.stdout:1486: got expected ENOENT (src dne) 2021-07-31T01:50:29.691 INFO:tasks.rados.rados.0.smithi045.stdout:1482: finishing write tid 1 to smithi04569269-4 2021-07-31T01:50:29.691 INFO:tasks.rados.rados.0.smithi045.stdout:1486: finishing copy_from racing read to smithi04569269-28 2021-07-31T01:50:29.691 INFO:tasks.rados.rados.0.smithi045.stdout:1484: finishing copy_from to smithi04569269-3 2021-07-31T01:50:29.692 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 3 v 543 (ObjNum 447 snap 132 seq_num 447) dirty exists 2021-07-31T01:50:29.693 INFO:tasks.rados.rados.0.smithi045.stdout:1484: finishing copy_from racing read to smithi04569269-3 2021-07-31T01:50:29.693 INFO:tasks.rados.rados.0.smithi045.stdout:1482: finishing write tid 2 to smithi04569269-4 2021-07-31T01:50:29.693 INFO:tasks.rados.rados.0.smithi045.stdout:1485: finishing write tid 1 to smithi04569269-6 2021-07-31T01:50:29.693 INFO:tasks.rados.rados.0.smithi045.stdout:1485: finishing write tid 2 to smithi04569269-6 2021-07-31T01:50:29.694 INFO:tasks.rados.rados.0.smithi045.stdout:1482: finishing write tid 3 to smithi04569269-4 2021-07-31T01:50:29.694 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 4 v 357 (ObjNum 467 snap 135 seq_num 467) dirty exists 2021-07-31T01:50:29.695 INFO:tasks.rados.rados.0.smithi045.stdout:1482: left oid 4 (ObjNum 467 snap 135 seq_num 467) 2021-07-31T01:50:29.695 INFO:tasks.rados.rados.0.smithi045.stdout:1485: finishing write tid 3 to smithi04569269-6 2021-07-31T01:50:29.695 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 6 v 576 (ObjNum 468 snap 135 seq_num 468) dirty exists 2021-07-31T01:50:29.695 INFO:tasks.rados.rados.0.smithi045.stdout:1485: left oid 6 (ObjNum 468 snap 135 seq_num 468) 2021-07-31T01:50:29.704 INFO:tasks.rados.rados.0.smithi045.stdout:1483: expect (ObjNum 415 snap 124 seq_num 415) 2021-07-31T01:50:30.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: cephadm 2021-07-31T01:50:28.499396+0000 mgr.x (mgr.24542) 133 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T01:50:30.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: cephadm 2021-07-31T01:50:28.542727+0000 mgr.x (mgr.24542) 134 : cephadm [ERR] 2021-07-31T01:50:30.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:50:30.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:50:30.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: yield 2021-07-31T01:50:30.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:50:30.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:50:30.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:50:30.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T01:50:30.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:50:30.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T01:50:30.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:50:30.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: await self._waiter 2021-07-31T01:50:30.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:50:30.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: task.result() 2021-07-31T01:50:30.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T01:50:30.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: cephadm 2021-07-31T01:50:28.543378+0000 mgr.x (mgr.24542) 135 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T01:50:30.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: cephadm 2021-07-31T01:50:30.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: 2021-07-31T01:50:28.554815+0000 mgr.x (mgr.24542) 136 : cephadm [ERR] 2021-07-31T01:50:30.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:50:30.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:50:30.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: yield 2021-07-31T01:50:30.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:50:30.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:50:30.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:50:30.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T01:50:30.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:50:30.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T01:50:30.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:50:30.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: await self._waiter 2021-07-31T01:50:30.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:50:30.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: task.result() 2021-07-31T01:50:30.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T01:50:30.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: cephadm 2021-07-31T01:50:28.555781+0000 mgr.x (mgr.24542) 137 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T01:50:30.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: cephadm 2021- 2021-07-31T01:50:30.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: 07-31T01:50:28.580763+0000 mgr.x (mgr.24542) 138 : cephadm [ERR] 2021-07-31T01:50:30.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:50:30.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:50:30.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: yield 2021-07-31T01:50:30.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:50:30.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:50:30.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:50:30.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T01:50:30.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:50:30.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T01:50:30.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:50:30.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: await self._waiter 2021-07-31T01:50:30.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:50:30.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: task.result() 2021-07-31T01:50:30.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T01:50:30.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: cephadm 2021-07-31T01:50:28.581227+0000 mgr.x (mgr.24542) 139 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:50:30.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:50:30.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:50:30.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: yield 2021-07-31T01:50:30.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:50:30.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:50:30.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:50:30.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T01:50:30.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:50:30.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T01:50:30.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:50:30.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: await self._waiter 2021-07-31T01:50:30.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:50:30.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: task.result() 2021-07-31T01:50:30.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T01:50:30.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: 2021-07-31T01:50:30.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T01:50:30.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: 2021-07-31T01:50:30.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:50:30.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:50:30.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: return f(*arg) 2021-07-31T01:50:30.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:50:30.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:50:30.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:50:30.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:50:30.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:50:30.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:50:30.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:50:30.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T01:50:30.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:50:30.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: raise self._exception 2021-07-31T01:50:30.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:50:30.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:50:30.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:50:30.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T01:50:30.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:50:30.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: raise 2021-07-31T01:50:30.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:50:30.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T01:50:30.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:50:30.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:50:30.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T01:50:30.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: cephadm 2021-07-31T01:50:28.589794+0000 mgr.x (mgr.24542 2021-07-31T01:50:30.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: ) 140 : cephadm [ERR] 2021-07-31T01:50:30.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:50:30.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:50:30.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: yield 2021-07-31T01:50:30.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:50:30.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:50:30.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:50:30.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T01:50:30.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:50:30.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T01:50:30.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:50:30.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: await self._waiter 2021-07-31T01:50:30.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:50:30.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: task.result() 2021-07-31T01:50:30.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T01:50:30.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: cephadm 2021-07-31T01:50:28.590240+0000 mgr.x (mgr.24542) 141 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:50:30.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:50:30.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:50:30.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: yield 2021-07-31T01:50:30.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:50:30.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:50:30.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:50:30.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T01:50:30.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:50:30.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T01:50:30.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:50:30.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: await self._waiter 2021-07-31T01:50:30.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:50:30.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: task.result() 2021-07-31T01:50:30.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T01:50:30.035 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: 2021-07-31T01:50:30.035 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T01:50:30.035 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: 2021-07-31T01:50:30.035 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:50:30.036 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:50:30.036 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: return f(*arg) 2021-07-31T01:50:30.036 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:50:30.036 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:50:30.036 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:50:30.037 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:50:30.037 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:50:30.037 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:50:30.037 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:50:30.038 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T01:50:30.038 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:50:30.038 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: raise self._exception 2021-07-31T01:50:30.038 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:50:30.039 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:50:30.039 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:50:30.039 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T01:50:30.039 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:50:30.040 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: raise 2021-07-31T01:50:30.040 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:50:30.040 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T01:50:30.040 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:50:30.041 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:50:30.041 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T01:50:30.041 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:29 smithi154 conmon[34553]: cluster 2021-07-31T01:50:28.616285+0000 mon.a (mon.0) 975 : cluster [DBG] osdmap e329: 8 total, 8 up, 8 in 2021-07-31T01:50:30.042 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: cephadm 2021-07-31T01:50:28.499396+0000 2021-07-31T01:50:30.042 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: mgr.x (mgr.24542) 133 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T01:50:30.042 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: cephadm 2021-07-31T01:50:28.542727+0000 mgr.x (mgr.24542) 134 : cephadm 2021-07-31T01:50:30.043 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: [ERR] 2021-07-31T01:50:30.043 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:50:30.043 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:50:30.043 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: yield 2021-07-31T01:50:30.044 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:50:30.044 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:50:30.044 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:50:30.044 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T01:50:30.044 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:50:30.045 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T01:50:30.045 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:50:30.045 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: await self._waiter 2021-07-31T01:50:30.045 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:50:30.045 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: task.result() 2021-07-31T01:50:30.046 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T01:50:30.046 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: cephadm 2021-07-31T01:50:28.543378+0000 mgr.x (mgr.24542) 135 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T01:50:30.046 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: cephadm 2021-07-31T 2021-07-31T01:50:30.046 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: 01:50:28.554815+0000 2021-07-31T01:50:30.047 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: mgr.x (mgr.24542) 136 2021-07-31T01:50:30.047 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: : cephadm [ERR] 2021-07-31T01:50:30.047 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:50:30.047 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:50:30.047 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: yield 2021-07-31T01:50:30.048 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:50:30.048 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:50:30.048 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:50:30.048 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T01:50:30.049 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:50:30.049 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T01:50:30.049 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:50:30.049 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: await self._waiter 2021-07-31T01:50:30.050 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:50:30.050 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: task.result() 2021-07-31T01:50:30.050 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T01:50:30.051 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: cephadm 2021-07-31T01:50:28.555781+0000 mgr.x (mgr.24542) 137 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T01:50:30.051 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: cephadm 2021-07- 2021-07-31T01:50:30.051 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: 31T01:50:28.580763+0000 mgr.x (mgr. 2021-07-31T01:50:30.051 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: 24542) 138 : cephadm [ERR] 2021-07-31T01:50:30.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: cephadm 2021-07-31T01:50:30.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: 2021-07 2021-07-31T01:50:30.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: -31T01:50:28.499396+0000 mgr.x (mgr.24542) 133 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T01:50:30.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: cephadm 2021-07 2021-07-31T01:50:30.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: -31T01:50:28.542727+0000 mgr.x (mgr.24542) 134 : cephadm [ERR] 2021-07-31T01:50:30.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: 2021-07-31T01:50:30.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:50:30.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:50:30.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: yield 2021-07-31T01:50:30.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:50:30.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:50:30.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:50:30.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T01:50:30.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:50:30.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T01:50:30.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:50:30.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: await self._waiter 2021-07-31T01:50:30.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:50:30.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: task.result() 2021-07-31T01:50:30.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T01:50:30.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: cephadm 2021-07-31T01:50:28.543378+0000 mgr.x (mgr.24542) 135 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T01:50:30.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: cephadm 2021-07-31 2021-07-31T01:50:30.057 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: T01:50:28.554815+0000 mgr.x (mgr.24542) 136 : cephadm [ERR] 2021-07-31T01:50:30.057 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:50:30.057 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:50:30.057 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: yield 2021-07-31T01:50:30.057 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:50:30.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:50:30.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:50:30.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T01:50:30.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:50:30.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T01:50:30.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:50:30.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: await self._waiter 2021-07-31T01:50:30.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:50:30.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: task.result() 2021-07-31T01:50:30.060 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T01:50:30.060 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: 2021-07-31T01:50:30.060 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: cephadm 2021-07-31T01:50:28.555781+0000 mgr.x (mgr.24542) 137 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T01:50:30.060 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: cephadm 2021-07-31T01 2021-07-31T01:50:30.060 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: :50:28.580763+0000 mgr.x (mgr.24542) 138 : cephadm [ERR] 2021-07-31T01:50:30.061 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:50:30.061 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:50:30.061 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: yield 2021-07-31T01:50:30.062 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:50:30.062 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:50:30.062 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:50:30.062 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T01:50:30.062 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:50:30.063 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T01:50:30.063 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:50:30.063 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: await self._waiter 2021-07-31T01:50:30.063 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:50:30.063 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: task.result() 2021-07-31T01:50:30.064 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T01:50:30.064 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: cephadm 2021-07-31T01:50:28.581227+0000 mgr.x (mgr.24542) 139 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:50:30.064 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:50:30.064 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:50:30.065 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: yield 2021-07-31T01:50:30.065 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:50:30.065 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:50:30.065 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:50:30.065 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T01:50:30.066 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:50:30.066 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T01:50:30.066 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:50:30.066 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: await self._waiter 2021-07-31T01:50:30.066 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:50:30.067 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: task.result() 2021-07-31T01:50:30.067 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T01:50:30.067 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: 2021-07-31T01:50:30.067 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T01:50:30.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: 2021-07-31T01:50:30.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:50:30.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:50:30.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: return f(*arg) 2021-07-31T01:50:30.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:50:30.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:50:30.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:50:30.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:50:30.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:50:30.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:50:30.070 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:50:30.070 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:50:30.070 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: yield 2021-07-31T01:50:30.071 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:50:30.071 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:50:30.071 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:50:30.071 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T01:50:30.071 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:50:30.072 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T01:50:30.072 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:50:30.072 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: await self._waiter 2021-07-31T01:50:30.072 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:50:30.073 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: task.result() 2021-07-31T01:50:30.073 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T01:50:30.073 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: cephadm 2021-07-31T01:50:28.581227+0000 mgr.x (mgr.24542) 139 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:50:30.073 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:50:30.074 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:50:30.074 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: yield 2021-07-31T01:50:30.074 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:50:30.074 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:50:30.075 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:50:30.075 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T01:50:30.075 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:50:30.075 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T01:50:30.075 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:50:30.076 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: await self._waiter 2021-07-31T01:50:30.076 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:50:30.076 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: task.result() 2021-07-31T01:50:30.076 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T01:50:30.076 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: 2021-07-31T01:50:30.077 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T01:50:30.077 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: 2021-07-31T01:50:30.077 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:50:30.077 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:50:30.077 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: return f(*arg) 2021-07-31T01:50:30.078 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:50:30.078 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:50:30.078 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:50:30.078 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:50:30.079 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:50:30.079 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:50:30.079 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:50:30.079 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T01:50:30.080 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:50:30.080 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: raise self._exception 2021-07-31T01:50:30.080 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:50:30.080 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:50:30.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:50:30.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T01:50:30.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:50:30.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: raise 2021-07-31T01:50:30.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:50:30.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T01:50:30.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:50:30.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:50:30.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T01:50:30.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: cephadm 2021- 2021-07-31T01:50:30.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: 07-31T01:50:28.589794+0000 mgr.x (mgr.24542) 140 : cephadm [ERR] 2021-07-31T01:50:30.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:50:30.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:50:30.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: yield 2021-07-31T01:50:30.084 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:50:30.084 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:50:30.084 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:50:30.084 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T01:50:30.084 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:50:30.085 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T01:50:30.085 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:50:30.085 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: await self._waiter 2021-07-31T01:50:30.085 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:50:30.086 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: task.result() 2021-07-31T01:50:30.086 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T01:50:30.086 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: cephadm 2021-07-31T01:50:28.590240+0000 mgr.x (mgr.24542) 141 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:50:30.086 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:50:30.086 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:50:30.087 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: yield 2021-07-31T01:50:30.087 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:50:30.087 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:50:30.087 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:50:30.087 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T01:50:30.088 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:50:30.088 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T01:50:30.088 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:50:30.089 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: raise self._exception 2021-07-31T01:50:30.089 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:50:30.089 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:50:30.089 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:50:30.089 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T01:50:30.090 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:50:30.090 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: raise 2021-07-31T01:50:30.090 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:50:30.090 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T01:50:30.091 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:50:30.091 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:50:30.091 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T01:50:30.091 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: cephadm 2021-07-31T01:50:28.589794+0000 mgr.x (mgr.24542) 140 : cephadm [ERR] 2021-07-31T01:50:30.091 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:50:30.092 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:50:30.092 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: yield 2021-07-31T01:50:30.092 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:50:30.092 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:50:30.092 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:50:30.093 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T01:50:30.093 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:50:30.093 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T01:50:30.093 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:50:30.093 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: await self._waiter 2021-07-31T01:50:30.094 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:50:30.094 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: task.result() 2021-07-31T01:50:30.094 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T01:50:30.094 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: cephadm 2021-07-31T01:50:28.590240+0000 mgr.x (mgr.24542) 141 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:50:30.095 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:50:30.095 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:50:30.095 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: yield 2021-07-31T01:50:30.095 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:50:30.095 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:50:30.096 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:50:30.096 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T01:50:30.096 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:50:30.097 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T01:50:30.097 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:50:30.097 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: await self._waiter 2021-07-31T01:50:30.097 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:50:30.097 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: task.result() 2021-07-31T01:50:30.098 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T01:50:30.098 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: 2021-07-31T01:50:30.098 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T01:50:30.098 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: 2021-07-31T01:50:30.098 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:50:30.099 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:50:30.099 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: return f(*arg) 2021-07-31T01:50:30.099 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:50:30.099 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:50:30.100 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:50:30.100 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:50:30.100 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:50:30.100 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:50:30.100 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:50:30.101 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T01:50:30.101 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:50:30.101 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: raise self._exception 2021-07-31T01:50:30.101 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:50:30.101 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:50:30.102 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:50:30.102 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T01:50:30.102 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:50:30.102 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: raise 2021-07-31T01:50:30.103 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:50:30.103 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T01:50:30.103 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:50:30.103 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:50:30.103 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T01:50:30.104 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[33655]: cluster 2021-07-31T01:50:28.616285+0000 mon.a (mon.0) 975 : cluster [DBG] osdmap e329: 8 total, 8 up, 8 in 2021-07-31T01:50:30.104 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:50:30.104 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T01:50:30.104 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:50:30.105 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: await self._waiter 2021-07-31T01:50:30.105 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:50:30.105 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: task.result() 2021-07-31T01:50:30.105 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T01:50:30.105 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: 2021-07-31T01:50:30.106 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T01:50:30.106 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: 2021-07-31T01:50:30.106 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:50:30.106 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:50:30.107 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: return f(*arg) 2021-07-31T01:50:30.107 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:50:30.107 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:50:30.107 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:50:30.107 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:50:30.108 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:50:30.108 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:50:30.108 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:50:30.108 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T01:50:30.108 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:50:30.109 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: raise self._exception 2021-07-31T01:50:30.109 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:50:30.109 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:50:30.109 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:50:30.109 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T01:50:30.110 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:50:30.110 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: raise 2021-07-31T01:50:30.110 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:50:30.110 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T01:50:30.111 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:50:30.111 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:50:30.111 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T01:50:30.111 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:29 smithi045 conmon[39718]: cluster 2021-07-31T01:50:28.616285+0000 mon.a (mon.0) 975 : cluster [DBG] osdmap e329: 8 total, 8 up, 8 in 2021-07-31T01:50:30.628 INFO:tasks.rados.rados.0.smithi045.stdout:1482: done (5 left) 2021-07-31T01:50:30.628 INFO:tasks.rados.rados.0.smithi045.stdout:1483: done (4 left) 2021-07-31T01:50:30.628 INFO:tasks.rados.rados.0.smithi045.stdout:1484: done (3 left) 2021-07-31T01:50:30.629 INFO:tasks.rados.rados.0.smithi045.stdout:1485: done (2 left) 2021-07-31T01:50:30.629 INFO:tasks.rados.rados.0.smithi045.stdout:1486: done (1 left) 2021-07-31T01:50:30.629 INFO:tasks.rados.rados.0.smithi045.stdout:1487: done (0 left) 2021-07-31T01:50:30.629 INFO:tasks.rados.rados.0.smithi045.stdout:1488: delete oid 36 current snap is 136 2021-07-31T01:50:30.632 INFO:tasks.rados.rados.0.smithi045.stdout:1488: done (0 left) 2021-07-31T01:50:30.632 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 4 current snap is 136 2021-07-31T01:50:30.632 INFO:tasks.rados.rados.0.smithi045.stdout:1489: seq_num 470 ranges {2121728=679936} 2021-07-31T01:50:30.647 INFO:tasks.rados.rados.0.smithi045.stdout:1489: writing smithi04569269-4 from 2121728 to 2801664 tid 1 2021-07-31T01:50:30.648 INFO:tasks.rados.rados.0.smithi045.stdout:1490: rollback oid 39 current snap is 136 2021-07-31T01:50:30.648 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 39 to 135 2021-07-31T01:50:30.648 INFO:tasks.rados.rados.0.smithi045.stdout:1491: read oid 48 snap 135 2021-07-31T01:50:30.648 INFO:tasks.rados.rados.0.smithi045.stdout:1491: expect (ObjNum 450 snap 133 seq_num 450) 2021-07-31T01:50:30.648 INFO:tasks.rados.rados.0.smithi045.stdout:1492: snap_create 2021-07-31T01:50:30.652 INFO:tasks.rados.rados.0.smithi045.stdout:1490: finishing rollback tid 0 to smithi04569269-39 2021-07-31T01:50:30.653 INFO:tasks.rados.rados.0.smithi045.stdout:1490: finishing rollback tid 1 to smithi04569269-39 2021-07-31T01:50:30.653 INFO:tasks.rados.rados.0.smithi045.stdout:1490: finishing rollback tid 2 to smithi04569269-39 2021-07-31T01:50:30.653 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 39 v 596 (ObjNum 342 snap 102 seq_num 342) dirty exists 2021-07-31T01:50:30.656 INFO:tasks.rados.rados.0.smithi045.stdout:1489: finishing write tid 1 to smithi04569269-4 2021-07-31T01:50:30.656 INFO:tasks.rados.rados.0.smithi045.stdout:1489: finishing write tid 2 to smithi04569269-4 2021-07-31T01:50:30.659 INFO:tasks.rados.rados.0.smithi045.stdout:1489: finishing write tid 3 to smithi04569269-4 2021-07-31T01:50:30.659 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 4 v 360 (ObjNum 470 snap 136 seq_num 470) dirty exists 2021-07-31T01:50:30.660 INFO:tasks.rados.rados.0.smithi045.stdout:1489: left oid 4 (ObjNum 470 snap 136 seq_num 470) 2021-07-31T01:50:30.665 INFO:tasks.rados.rados.0.smithi045.stdout:1491: expect (ObjNum 450 snap 133 seq_num 450) 2021-07-31T01:50:30.730 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T01:50:31.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:30 smithi154 conmon[34553]: cluster 2021-07-31T01:50:29.617487+0000 mon.a (mon. 2021-07-31T01:50:31.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:30 smithi154 conmon[34553]: 0) 976 : cluster [DBG] osdmap e330: 8 total, 8 up, 8 in 2021-07-31T01:50:31.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:30 smithi154 conmon[34553]: cluster 2021-07-31T01:50:29.837170+0000 mgr.x (mgr.24542) 142 : cluster [DBG] pgmap v275: 25 pgs: 25 active+clean; 107 MiB data, 885 MiB used, 714 GiB / 715 GiB avail; 4.5 MiB/s rd, 860 KiB/s wr, 12 op/s 2021-07-31T01:50:31.007 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T01:50:31.007 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 5m ago 6m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T01:50:31.008 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 3m ago 5m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T01:50:31.008 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 3m ago 9m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T01:50:31.008 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 5m ago 11m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T01:50:31.008 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 5m ago 11m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T01:50:31.009 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 3m ago 9m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T01:50:31.009 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 5m ago 9m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T01:50:31.009 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 5m ago 6m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T01:50:31.009 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 3m ago 6m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T01:50:31.009 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 5m ago 9m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T01:50:31.010 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 5m ago 8m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T01:50:31.010 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 5m ago 8m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T01:50:31.010 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 5m ago 8m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T01:50:31.010 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 3m ago 7m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T01:50:31.011 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 3m ago 7m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T01:50:31.011 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 3m ago 7m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T01:50:31.011 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 3m ago 6m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T01:50:31.011 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 3m ago 6m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T01:50:31.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:30 smithi045 conmon[33655]: cluster 2021-07-31T01:50: 2021-07-31T01:50:31.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:30 smithi045 conmon[33655]: 29.617487+0000 mon.a (mon.0) 976 : cluster [DBG] osdmap e330: 8 total, 8 up, 8 in 2021-07-31T01:50:31.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:30 smithi045 conmon[33655]: cluster 2021-07- 2021-07-31T01:50:31.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:30 smithi045 conmon[33655]: 31T01:50:29.837170+0000 mgr.x (mgr.24542 2021-07-31T01:50:31.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:30 smithi045 conmon[33655]: ) 142 : cluster [DBG] pgmap v275: 25 pgs: 25 active+clean; 107 MiB data, 885 MiB used, 714 GiB / 715 GiB avail; 4.5 MiB/s rd, 860 KiB/s wr, 12 op/s 2021-07-31T01:50:31.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:30 smithi045 conmon[39718]: cluster 2021-07-31T01:50:29.617487 2021-07-31T01:50:31.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:30 smithi045 conmon[39718]: +0000 mon.a (mon.0) 976 : cluster [DBG] osdmap e330: 8 total, 8 up, 8 in 2021-07-31T01:50:31.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:30 smithi045 conmon[39718]: cluster 2021-07-31T01:50 2021-07-31T01:50:31.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:30 smithi045 conmon[39718]: :29.837170+0000 mgr.x (mgr.24542) 142 2021-07-31T01:50:31.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:30 smithi045 conmon[39718]: : cluster [DBG] pgmap v275: 25 pgs: 25 active+clean; 107 MiB data, 885 MiB used, 714 GiB / 715 GiB avail; 4.5 MiB/s rd, 860 KiB/s wr, 12 op/s 2021-07-31T01:50:31.343 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T01:50:31.343 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T01:50:31.344 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T01:50:31.344 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T01:50:31.344 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T01:50:31.344 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T01:50:31.344 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T01:50:31.345 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T01:50:31.345 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T01:50:31.345 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T01:50:31.345 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T01:50:31.346 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T01:50:31.346 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T01:50:31.346 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T01:50:31.346 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T01:50:31.346 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T01:50:31.347 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T01:50:31.538 INFO:tasks.rados.rados.0.smithi045.stdout:1489: done (3 left) 2021-07-31T01:50:31.539 INFO:tasks.rados.rados.0.smithi045.stdout:1490: done (2 left) 2021-07-31T01:50:31.539 INFO:tasks.rados.rados.0.smithi045.stdout:1491: done (1 left) 2021-07-31T01:50:31.539 INFO:tasks.rados.rados.0.smithi045.stdout:1492: done (0 left) 2021-07-31T01:50:31.540 INFO:tasks.rados.rados.0.smithi045.stdout:1493: rmattr oid 13 current snap is 137 2021-07-31T01:50:31.540 INFO:tasks.rados.rados.0.smithi045.stdout:1494: snap_remove snap 136 2021-07-31T01:50:32.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:31 smithi154 conmon[34553]: cluster 2021-07-31T01:50:30.627920+0000 mon.a (mon.0) 977 : cluster [DBG] osdmap e331: 8 total, 8 up, 8 in 2021-07-31T01:50:32.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:31 smithi154 conmon[34553]: audit 2021-07-31T01:50:30.717867+0000 mgr.x (mgr.24542) 143 : audit [DBG] from='client.14820 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:50:32.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:31 smithi154 conmon[34553]: audit 2021-07-31T01:50:31.343065+0000 mon.c (mon.1) 30 : audit [DBG] from='client.? 172.21.15.45:0/1545438487' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T01:50:32.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:31 smithi154 conmon[34553]: cluster 2021-07-31T01:50:31.538036+0000 mon.a (mon.0) 978 : cluster [DBG] osdmap e332: 8 total, 8 up, 8 in 2021-07-31T01:50:32.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:31 smithi045 conmon[39718]: cluster 2021-07-31T01:50:30.627920+0000 mon.a (mon 2021-07-31T01:50:32.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:31 smithi045 conmon[39718]: .0) 977 : cluster [DBG] osdmap e331: 8 total, 8 up, 8 in 2021-07-31T01:50:32.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:31 smithi045 conmon[39718]: audit 2021- 2021-07-31T01:50:32.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:31 smithi045 conmon[39718]: 07-31T01:50:30.717867+0000 mgr.x (mgr.24542) 143 : audit [DBG] from='client.14820 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:50:32.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:31 smithi045 conmon[39718]: audit 2021-07-31T01:50: 2021-07-31T01:50:32.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:31 smithi045 conmon[39718]: 31.343065+0000 mon.c (mon.1) 30 : audit [DBG] from='client.? 172.21.15.45:0/1545438487' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T01:50:32.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:31 smithi045 conmon[39718]: cluster 2021-07-31T01:50:31.538036+0000 2021-07-31T01:50:32.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:31 smithi045 conmon[39718]: mon.a (mon.0) 978 : cluster [DBG] osdmap e332: 8 total, 8 up, 8 in 2021-07-31T01:50:32.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:31 smithi045 conmon[33655]: cluster 2021-07-31T01:50 2021-07-31T01:50:32.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:31 smithi045 conmon[33655]: :30.627920+0000 mon.a (mon.0) 977 : cluster [DBG] osdmap e331: 8 total, 8 up, 8 in 2021-07-31T01:50:32.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:31 smithi045 conmon[33655]: audit 2021-07-31T01:50:32.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:31 smithi045 conmon[33655]: 2021-07-31T01:50:30.717867+0000 mgr.x (mgr. 2021-07-31T01:50:32.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:31 smithi045 conmon[33655]: 24542) 143 : audit [DBG] from='client.14820 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:50:32.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:31 smithi045 conmon[33655]: audit 2021-07 2021-07-31T01:50:32.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:31 smithi045 conmon[33655]: -31T01:50:31.343065+0000 mon.c (mon.1 2021-07-31T01:50:32.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:31 smithi045 conmon[33655]: ) 30 : audit [DBG] from='client.? 172.21.15.45:0/1545438487' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T01:50:32.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:31 smithi045 conmon[33655]: cluster 2021-07-31T01:50:31.538036+0000 2021-07-31T01:50:32.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:31 smithi045 conmon[33655]: mon.a (mon.0) 978 : cluster [DBG] osdmap e332: 8 total, 8 up, 8 in 2021-07-31T01:50:32.545 INFO:tasks.rados.rados.0.smithi045.stdout:1494: done (1 left) 2021-07-31T01:50:32.545 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 25 current snap is 137 2021-07-31T01:50:32.546 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 13 v 577 (ObjNum 395 snap 119 seq_num 395) dirty exists 2021-07-31T01:50:32.546 INFO:tasks.rados.rados.0.smithi045.stdout:1495: seq_num 471 ranges {0=696320} 2021-07-31T01:50:32.549 INFO:tasks.rados.rados.0.smithi045.stdout:1495: writing smithi04569269-25 from 0 to 696320 tid 1 2021-07-31T01:50:32.549 INFO:tasks.rados.rados.0.smithi045.stdout:1493: done (1 left) 2021-07-31T01:50:32.549 INFO:tasks.rados.rados.0.smithi045.stdout:1496: snap_create 2021-07-31T01:50:32.553 INFO:tasks.rados.rados.0.smithi045.stdout:1495: finishing write tid 1 to smithi04569269-25 2021-07-31T01:50:32.554 INFO:tasks.rados.rados.0.smithi045.stdout:1495: finishing write tid 2 to smithi04569269-25 2021-07-31T01:50:32.555 INFO:tasks.rados.rados.0.smithi045.stdout:1495: finishing write tid 3 to smithi04569269-25 2021-07-31T01:50:32.555 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 25 v 274 (ObjNum 471 snap 137 seq_num 471) dirty exists 2021-07-31T01:50:32.556 INFO:tasks.rados.rados.0.smithi045.stdout:1495: left oid 25 (ObjNum 471 snap 137 seq_num 471) 2021-07-31T01:50:33.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:32 smithi154 conmon[34553]: audit 2021-07-31T01:50:30.999305+0000 mgr.x (mgr.24542) 144 : audit [DBG] from='client.14826 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:50:33.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:32 smithi154 conmon[34553]: cluster 2021-07-31T01:50:31.837620+0000 mgr.x (mgr.24542) 145 : cluster [DBG] pgmap v278: 25 pgs: 25 active+clean; 107 MiB data, 885 MiB used, 714 GiB / 715 GiB avail; 4.5 MiB/s rd, 860 KiB/s wr, 12 op/s 2021-07-31T01:50:33.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:32 smithi154 conmon[34553]: cluster 2021-07-31T01:50:32.544314+0000 mon.a (mon.0) 979 : cluster [DBG] 2021-07-31T01:50:33.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:32 smithi154 conmon[34553]: osdmap e333: 8 total, 8 up, 8 in 2021-07-31T01:50:33.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:32 smithi045 conmon[33655]: audit 2021-07-31T01:50:30.999305+0000 2021-07-31T01:50:33.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:32 smithi045 conmon[33655]: mgr.x (mgr.24542) 144 : audit [DBG] from='client.14826 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:50:33.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:32 smithi045 conmon[33655]: cluster 2021-07-31T01:50:31.837620+0000 mgr.x (mgr.24542) 145 : cluster [DBG] pgmap v278: 25 pgs: 25 active+clean; 107 MiB data, 885 MiB used, 714 GiB / 715 GiB avail; 4.5 MiB/s rd, 860 KiB/s wr, 12 op/s 2021-07-31T01:50:33.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:32 smithi045 conmon[33655]: cluster 2021-07-31T01:50:33.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:32 smithi045 conmon[33655]: 2021-07-31T01 2021-07-31T01:50:33.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:32 smithi045 conmon[33655]: :50:32.544314+0000 mon.a 2021-07-31T01:50:33.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:32 smithi045 conmon[33655]: (mon.0) 979 : cluster 2021-07-31T01:50:33.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:32 smithi045 conmon[33655]: [DBG] osdmap e333: 8 total, 8 up, 8 in 2021-07-31T01:50:33.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:32 smithi045 conmon[39718]: audit 2021-07-31T01:50:30.999305+0000 mgr.x (mgr.24542 2021-07-31T01:50:33.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:32 smithi045 conmon[39718]: ) 144 : audit [DBG] from='client.14826 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:50:33.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:32 smithi045 conmon[39718]: cluster 2021-07-31T01:50:31.837620+0000 mgr.x (mgr.24542) 145 : cluster [DBG] 2021-07-31T01:50:33.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:32 smithi045 conmon[39718]: pgmap v278: 25 pgs: 25 active+clean; 107 MiB data, 885 MiB used, 714 GiB / 715 GiB avail; 4.5 MiB/s rd, 860 KiB/s wr, 12 op/s 2021-07-31T01:50:33.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:32 smithi045 conmon[39718]: cluster 2021-07-31T01:50:32.544314+0000 mon.a (mon.0) 2021-07-31T01:50:33.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:32 smithi045 conmon[39718]: 979 : cluster [DBG] osdmap e333: 8 total, 8 up, 8 in 2021-07-31T01:50:33.547 INFO:tasks.rados.rados.0.smithi045.stdout:1495: done (1 left) 2021-07-31T01:50:33.547 INFO:tasks.rados.rados.0.smithi045.stdout:1496: done (0 left) 2021-07-31T01:50:33.547 INFO:tasks.rados.rados.0.smithi045.stdout:1497: read oid 27 snap -1 2021-07-31T01:50:33.548 INFO:tasks.rados.rados.0.smithi045.stdout:1497: expect (ObjNum 374 snap 109 seq_num 374) 2021-07-31T01:50:33.548 INFO:tasks.rados.rados.0.smithi045.stdout:1498: rmattr oid 22 current snap is 138 2021-07-31T01:50:33.548 INFO:tasks.rados.rados.0.smithi045.stdout:1498: done (1 left) 2021-07-31T01:50:33.548 INFO:tasks.rados.rados.0.smithi045.stdout:1499: read oid 21 snap 135 2021-07-31T01:50:33.548 INFO:tasks.rados.rados.0.smithi045.stdout:1499: expect deleted 2021-07-31T01:50:33.549 INFO:tasks.rados.rados.0.smithi045.stdout:1500: snap_create 2021-07-31T01:50:33.559 INFO:tasks.rados.rados.0.smithi045.stdout:1497: expect (ObjNum 374 snap 109 seq_num 374) 2021-07-31T01:50:33.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:50:33 smithi045 conmon[61210]: level=error ts=2021-07-31T01:50:33.514Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:50:33.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:50:33 smithi045 conmon[61210]: level=error ts=2021-07-31T01:50:33.514Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:50:33.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:50:33 smithi045 conmon[61210]: level=error ts=2021-07-31T01:50:33.514Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T01:50:34.552 INFO:tasks.rados.rados.0.smithi045.stdout:1497: done (2 left) 2021-07-31T01:50:34.552 INFO:tasks.rados.rados.0.smithi045.stdout:1499: done (1 left) 2021-07-31T01:50:34.553 INFO:tasks.rados.rados.0.smithi045.stdout:1500: done (0 left) 2021-07-31T01:50:34.553 INFO:tasks.rados.rados.0.smithi045.stdout:1501: setattr oid 18 current snap is 139 2021-07-31T01:50:34.554 INFO:tasks.rados.rados.0.smithi045.stdout:1502: setattr oid 47 current snap is 139 2021-07-31T01:50:34.557 INFO:tasks.rados.rados.0.smithi045.stdout:1503: delete oid 41 current snap is 139 2021-07-31T01:50:34.560 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 18 v 361 (ObjNum 1482060403 snap 1582788431 seq_num 1550410349) dirty exists 2021-07-31T01:50:34.562 INFO:tasks.rados.rados.0.smithi045.stdout:1501: done (2 left) 2021-07-31T01:50:34.562 INFO:tasks.rados.rados.0.smithi045.stdout:1503: done (1 left) 2021-07-31T01:50:34.562 INFO:tasks.rados.rados.0.smithi045.stdout:1504: snap_create 2021-07-31T01:50:34.563 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 47 v 275 (ObjNum 172 snap 41 seq_num 172) dirty exists 2021-07-31T01:50:35.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:34 smithi154 conmon[34553]: cluster 2021-07-31T01:50:33.545928+0000 mon.a (mon.0) 2021-07-31T01:50:35.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:34 smithi154 conmon[34553]: 980 : cluster [DBG] osdmap e334: 8 total, 8 up, 8 in 2021-07-31T01:50:35.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:34 smithi154 conmon[34553]: cluster 2021-07-31T01:50:33.838322+0000 mgr.x (mgr.24542) 146 : cluster [DBG] pgmap v281: 25 pgs: 25 active+clean; 112 MiB data, 883 MiB used, 714 GiB / 715 GiB avail; 9.6 MiB/s rd, 1.2 MiB/s wr, 11 op/s 2021-07-31T01:50:35.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:34 smithi045 conmon[33655]: cluster 2021-07-31T01:50:33.545928+0000 mon.a (mon.0) 980 : cluster [DBG] osdmap e334: 8 total, 8 up, 8 in 2021-07-31T01:50:35.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:34 smithi045 conmon[33655]: 2021-07-31T01:50:35.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:34 smithi045 conmon[33655]: cluster 2021-07-31T01:50:33.838322+0000 mgr.x (mgr.24542) 146 : cluster [DBG] pgmap v281: 25 pgs: 25 active+clean; 112 MiB data, 883 MiB used, 714 GiB / 715 GiB avail; 9.6 MiB/s rd, 1.2 MiB/s wr, 11 op/s 2021-07-31T01:50:35.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:34 smithi045 conmon[39718]: cluster 2021-07-31T01:50:33.545928+0000 mon.a (mon.0) 2021-07-31T01:50:35.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:34 smithi045 conmon[39718]: 980 : cluster [DBG] osdmap e334: 8 total, 8 up, 8 in 2021-07-31T01:50:35.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:34 smithi045 conmon[39718]: cluster 2021-07-31T01:50:33.838322+0000 mgr.x (mgr.24542) 146 : cluster [DBG] pgmap v281: 25 pgs: 25 active+clean; 112 MiB data, 883 MiB used, 714 GiB / 715 GiB avail; 9.6 MiB/s rd, 1.2 MiB/s wr, 11 op/s 2021-07-31T01:50:35.554 INFO:tasks.rados.rados.0.smithi045.stdout:1502: done (1 left) 2021-07-31T01:50:35.555 INFO:tasks.rados.rados.0.smithi045.stdout:1504: done (0 left) 2021-07-31T01:50:35.555 INFO:tasks.rados.rados.0.smithi045.stdout:1505: copy_from oid 49 from oid 48 current snap is 140 2021-07-31T01:50:35.555 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 15 current snap is 140 2021-07-31T01:50:35.555 INFO:tasks.rados.rados.0.smithi045.stdout:1506: seq_num 473 ranges {1925120=729088} 2021-07-31T01:50:35.568 INFO:tasks.rados.rados.0.smithi045.stdout:1506: writing smithi04569269-15 from 1925120 to 2654208 tid 1 2021-07-31T01:50:35.568 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 17 current snap is 140 2021-07-31T01:50:35.569 INFO:tasks.rados.rados.0.smithi045.stdout:1507: seq_num 474 ranges {548864=745472} 2021-07-31T01:50:35.576 INFO:tasks.rados.rados.0.smithi045.stdout:1507: writing smithi04569269-17 from 548864 to 1294336 tid 1 2021-07-31T01:50:35.577 INFO:tasks.rados.rados.0.smithi045.stdout:1508: rollback oid 28 current snap is 140 2021-07-31T01:50:35.577 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 28 to 139 2021-07-31T01:50:35.577 INFO:tasks.rados.rados.0.smithi045.stdout:1506: finishing write tid 1 to smithi04569269-15 2021-07-31T01:50:35.577 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 10 current snap is 140 2021-07-31T01:50:35.577 INFO:tasks.rados.rados.0.smithi045.stdout:1506: finishing write tid 2 to smithi04569269-15 2021-07-31T01:50:35.578 INFO:tasks.rados.rados.0.smithi045.stdout:1506: finishing write tid 3 to smithi04569269-15 2021-07-31T01:50:35.578 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 15 v 364 (ObjNum 473 snap 140 seq_num 473) dirty exists 2021-07-31T01:50:35.578 INFO:tasks.rados.rados.0.smithi045.stdout:1506: left oid 15 (ObjNum 473 snap 140 seq_num 473) 2021-07-31T01:50:35.578 INFO:tasks.rados.rados.0.smithi045.stdout:1509: seq_num 475 ranges {0=761856} 2021-07-31T01:50:35.580 INFO:tasks.rados.rados.0.smithi045.stdout:1509: writing smithi04569269-10 from 0 to 761856 tid 1 2021-07-31T01:50:35.581 INFO:tasks.rados.rados.0.smithi045.stdout:1506: done (4 left) 2021-07-31T01:50:35.581 INFO:tasks.rados.rados.0.smithi045.stdout:1510: delete oid 50 current snap is 140 2021-07-31T01:50:35.582 INFO:tasks.rados.rados.0.smithi045.stdout:1505: finishing copy_from to smithi04569269-49 2021-07-31T01:50:35.582 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 49 v 903 (ObjNum 450 snap 133 seq_num 450) dirty exists 2021-07-31T01:50:35.583 INFO:tasks.rados.rados.0.smithi045.stdout:1505: finishing copy_from racing read to smithi04569269-49 2021-07-31T01:50:35.585 INFO:tasks.rados.rados.0.smithi045.stdout:1507: finishing write tid 1 to smithi04569269-17 2021-07-31T01:50:35.585 INFO:tasks.rados.rados.0.smithi045.stdout:1507: finishing write tid 2 to smithi04569269-17 2021-07-31T01:50:35.586 INFO:tasks.rados.rados.0.smithi045.stdout:1508: finishing rollback tid 0 to smithi04569269-28 2021-07-31T01:50:35.586 INFO:tasks.rados.rados.0.smithi045.stdout:1507: finishing write tid 3 to smithi04569269-17 2021-07-31T01:50:35.587 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 17 v 557 (ObjNum 474 snap 140 seq_num 474) dirty exists 2021-07-31T01:50:35.587 INFO:tasks.rados.rados.0.smithi045.stdout:1507: left oid 17 (ObjNum 474 snap 140 seq_num 474) 2021-07-31T01:50:35.588 INFO:tasks.rados.rados.0.smithi045.stdout:1508: finishing rollback tid 1 to smithi04569269-28 2021-07-31T01:50:35.589 INFO:tasks.rados.rados.0.smithi045.stdout:1508: finishing rollback tid 2 to smithi04569269-28 2021-07-31T01:50:35.589 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 28 v 561 (ObjNum 333 snap 97 seq_num 333) dirty exists 2021-07-31T01:50:35.589 INFO:tasks.rados.rados.0.smithi045.stdout:1509: finishing write tid 1 to smithi04569269-10 2021-07-31T01:50:35.591 INFO:tasks.rados.rados.0.smithi045.stdout:1509: finishing write tid 2 to smithi04569269-10 2021-07-31T01:50:35.592 INFO:tasks.rados.rados.0.smithi045.stdout:1505: done (4 left) 2021-07-31T01:50:35.592 INFO:tasks.rados.rados.0.smithi045.stdout:1507: done (3 left) 2021-07-31T01:50:35.592 INFO:tasks.rados.rados.0.smithi045.stdout:1508: done (2 left) 2021-07-31T01:50:35.592 INFO:tasks.rados.rados.0.smithi045.stdout:1510: done (1 left) 2021-07-31T01:50:35.593 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 44 current snap is 140 2021-07-31T01:50:35.593 INFO:tasks.rados.rados.0.smithi045.stdout:1511: seq_num 477 ranges {0=794624} 2021-07-31T01:50:35.597 INFO:tasks.rados.rados.0.smithi045.stdout:1511: writing smithi04569269-44 from 0 to 794624 tid 1 2021-07-31T01:50:35.597 INFO:tasks.rados.rados.0.smithi045.stdout:1512: read oid 20 snap -1 2021-07-31T01:50:35.597 INFO:tasks.rados.rados.0.smithi045.stdout:1512: expect deleted 2021-07-31T01:50:35.597 INFO:tasks.rados.rados.0.smithi045.stdout:1513: snap_create 2021-07-31T01:50:35.597 INFO:tasks.rados.rados.0.smithi045.stdout:1509: finishing write tid 3 to smithi04569269-10 2021-07-31T01:50:35.598 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 10 v 563 (ObjNum 475 snap 140 seq_num 475) dirty exists 2021-07-31T01:50:35.598 INFO:tasks.rados.rados.0.smithi045.stdout:1509: left oid 10 (ObjNum 475 snap 140 seq_num 475) 2021-07-31T01:50:35.600 INFO:tasks.rados.rados.0.smithi045.stdout:1511: finishing write tid 1 to smithi04569269-44 2021-07-31T01:50:35.600 INFO:tasks.rados.rados.0.smithi045.stdout:1511: finishing write tid 2 to smithi04569269-44 2021-07-31T01:50:35.602 INFO:tasks.rados.rados.0.smithi045.stdout:1511: finishing write tid 3 to smithi04569269-44 2021-07-31T01:50:35.603 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 44 v 929 (ObjNum 477 snap 140 seq_num 477) dirty exists 2021-07-31T01:50:35.603 INFO:tasks.rados.rados.0.smithi045.stdout:1511: left oid 44 (ObjNum 477 snap 140 seq_num 477) 2021-07-31T01:50:36.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:35 smithi154 conmon[34553]: cluster 2021-07-31T01:50:34.551443+0000 mon.a (mon.0) 981 : cluster [DBG] osdmap e335: 8 total, 8 up, 8 in 2021-07-31T01:50:36.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:35 smithi045 conmon[39718]: cluster 2021-07-31T01:50: 2021-07-31T01:50:36.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:35 smithi045 conmon[39718]: 34.551443+0000 mon.a (mon.0) 981 : cluster [DBG] osdmap e335: 8 total, 8 up, 8 in 2021-07-31T01:50:36.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:35 smithi045 conmon[33655]: cluster 2021-07-31T01:50:34.551443+0000 mon.a (mon.0 2021-07-31T01:50:36.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:35 smithi045 conmon[33655]: ) 981 : cluster [DBG] osdmap e335: 8 total, 8 up, 8 in 2021-07-31T01:50:36.542 INFO:tasks.rados.rados.0.smithi045.stdout:1509: done (3 left) 2021-07-31T01:50:36.543 INFO:tasks.rados.rados.0.smithi045.stdout:1511: done (2 left) 2021-07-31T01:50:36.543 INFO:tasks.rados.rados.0.smithi045.stdout:1512: done (1 left) 2021-07-31T01:50:36.543 INFO:tasks.rados.rados.0.smithi045.stdout:1513: done (0 left) 2021-07-31T01:50:36.544 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 27 current snap is 141 2021-07-31T01:50:36.544 INFO:tasks.rados.rados.0.smithi045.stdout:1514: seq_num 478 ranges {1474560=409600,1884160=401408} 2021-07-31T01:50:36.552 INFO:tasks.rados.rados.0.smithi045.stdout:1514: writing smithi04569269-27 from 1474560 to 1884160 tid 1 2021-07-31T01:50:36.554 INFO:tasks.rados.rados.0.smithi045.stdout:1514: writing smithi04569269-27 from 1884160 to 2285568 tid 2 2021-07-31T01:50:36.555 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 15 current snap is 141 2021-07-31T01:50:36.555 INFO:tasks.rados.rados.0.smithi045.stdout:1515: seq_num 479 ranges {2654208=425984,3080192=401408} 2021-07-31T01:50:36.571 INFO:tasks.rados.rados.0.smithi045.stdout:1515: writing smithi04569269-15 from 2654208 to 3080192 tid 1 2021-07-31T01:50:36.575 INFO:tasks.rados.rados.0.smithi045.stdout:1515: writing smithi04569269-15 from 3080192 to 3481600 tid 2 2021-07-31T01:50:36.576 INFO:tasks.rados.rados.0.smithi045.stdout:1516: read oid 29 snap -1 2021-07-31T01:50:36.576 INFO:tasks.rados.rados.0.smithi045.stdout:1516: expect (ObjNum 410 snap 123 seq_num 410) 2021-07-31T01:50:36.576 INFO:tasks.rados.rados.0.smithi045.stdout:1517: read oid 9 snap 139 2021-07-31T01:50:36.577 INFO:tasks.rados.rados.0.smithi045.stdout:1517: expect (ObjNum 423 snap 126 seq_num 423) 2021-07-31T01:50:36.578 INFO:tasks.rados.rados.0.smithi045.stdout:1518: setattr oid 36 current snap is 141 2021-07-31T01:50:36.578 INFO:tasks.rados.rados.0.smithi045.stdout:1514: finishing write tid 1 to smithi04569269-27 2021-07-31T01:50:36.579 INFO:tasks.rados.rados.0.smithi045.stdout:1514: finishing write tid 2 to smithi04569269-27 2021-07-31T01:50:36.579 INFO:tasks.rados.rados.0.smithi045.stdout:1514: finishing write tid 3 to smithi04569269-27 2021-07-31T01:50:36.579 INFO:tasks.rados.rados.0.smithi045.stdout:1514: finishing write tid 4 to smithi04569269-27 2021-07-31T01:50:36.580 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 27 v 932 (ObjNum 478 snap 141 seq_num 478) dirty exists 2021-07-31T01:50:36.580 INFO:tasks.rados.rados.0.smithi045.stdout:1514: left oid 27 (ObjNum 478 snap 141 seq_num 478) 2021-07-31T01:50:36.581 INFO:tasks.rados.rados.0.smithi045.stdout:1515: finishing write tid 1 to smithi04569269-15 2021-07-31T01:50:36.582 INFO:tasks.rados.rados.0.smithi045.stdout:1514: done (4 left) 2021-07-31T01:50:36.582 INFO:tasks.rados.rados.0.smithi045.stdout:1519: snap_create 2021-07-31T01:50:36.583 INFO:tasks.rados.rados.0.smithi045.stdout:1515: finishing write tid 2 to smithi04569269-15 2021-07-31T01:50:36.583 INFO:tasks.rados.rados.0.smithi045.stdout:1515: finishing write tid 3 to smithi04569269-15 2021-07-31T01:50:36.584 INFO:tasks.rados.rados.0.smithi045.stdout:1515: finishing write tid 4 to smithi04569269-15 2021-07-31T01:50:36.585 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 15 v 368 (ObjNum 479 snap 141 seq_num 479) dirty exists 2021-07-31T01:50:36.585 INFO:tasks.rados.rados.0.smithi045.stdout:1515: left oid 15 (ObjNum 479 snap 141 seq_num 479) 2021-07-31T01:50:36.587 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 36 v 990 (ObjNum -1104778368 snap 21873 seq_num 3190178752) dirty exists 2021-07-31T01:50:36.594 INFO:tasks.rados.rados.0.smithi045.stdout:1517: expect (ObjNum 423 snap 126 seq_num 423) 2021-07-31T01:50:36.771 INFO:tasks.rados.rados.0.smithi045.stdout:1516: expect (ObjNum 410 snap 123 seq_num 410) 2021-07-31T01:50:37.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:36 smithi154 conmon[34553]: cluster 2021-07-31T01:50:35.552835+0000 mon.a (mon.0) 982 : cluster [DBG] osdmap e336: 8 total, 8 up, 8 in 2021-07-31T01:50:37.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:36 smithi154 conmon[34553]: cluster 2021-07-31T01:50:35.838761+0000 mgr.x (mgr.24542) 147 : cluster [DBG] pgmap v284: 25 pgs: 25 active+clean; 112 MiB data, 883 MiB used, 714 GiB / 715 GiB avail; 9.6 MiB/s rd, 1.2 MiB/s wr, 11 op/s 2021-07-31T01:50:37.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:36 smithi154 conmon[34553]: cluster 2021-07-31T01:50:36.541068+0000 mon.a (mon.0) 983 : cluster 2021-07-31T01:50:37.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:36 smithi154 conmon[34553]: [DBG] osdmap e337: 8 total, 8 up, 8 in 2021-07-31T01:50:37.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:36 smithi045 conmon[39718]: cluster 2021-07-31T01:50:35.552835+0000 2021-07-31T01:50:37.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:36 smithi045 conmon[39718]: mon.a (mon.0) 982 : cluster [DBG] osdmap e336: 8 total, 8 up, 8 in 2021-07-31T01:50:37.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:36 smithi045 conmon[39718]: cluster 2021-07 2021-07-31T01:50:37.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:36 smithi045 conmon[39718]: -31T01:50:35.838761+0000 mgr.x (mgr.24542) 147 : cluster [DBG] pgmap v284: 25 pgs: 25 active+clean; 112 MiB data, 883 MiB used, 714 GiB / 715 GiB avail; 9.6 MiB/s rd, 1.2 MiB/s wr, 11 op/s 2021-07-31T01:50:37.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:36 smithi045 conmon[39718]: cluster 2021-07-31T01:50:37.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:36 smithi045 conmon[39718]: 2021-07-31T01:50:36.541068+0000 mon.a ( 2021-07-31T01:50:37.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:36 smithi045 conmon[39718]: mon.0) 983 : cluster [DBG] osdmap e337: 8 total, 8 up, 8 in 2021-07-31T01:50:37.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:36 smithi045 conmon[33655]: cluster 2021-07-31T01 2021-07-31T01:50:37.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:36 smithi045 conmon[33655]: :50:35.552835+0000 mon.a (mon.0) 982 : cluster [DBG] osdmap e336: 8 total, 8 up, 8 in 2021-07-31T01:50:37.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:36 smithi045 conmon[33655]: cluster 2021-07-31T01:50:37.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:36 smithi045 conmon[33655]: 2021-07-31T01:50:35.838761+0000 mgr.x (mgr.24542) 147 : cluster [DBG] pgmap v284: 25 pgs: 25 active+clean; 112 MiB data, 883 MiB used, 714 GiB / 715 GiB avail; 9.6 MiB/s rd, 1.2 MiB/s wr, 11 op/s 2021-07-31T01:50:37.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:36 smithi045 conmon[33655]: cluster 2021-07-31 2021-07-31T01:50:37.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:36 smithi045 conmon[33655]: T01:50:36.541068+0000 mon.a (mon.0) 983 : cluster [DBG] 2021-07-31T01:50:37.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:36 smithi045 conmon[33655]: osdmap e337: 8 total, 8 up, 8 in 2021-07-31T01:50:37.566 INFO:tasks.rados.rados.0.smithi045.stdout:1515: done (4 left) 2021-07-31T01:50:37.567 INFO:tasks.rados.rados.0.smithi045.stdout:1516: done (3 left) 2021-07-31T01:50:37.567 INFO:tasks.rados.rados.0.smithi045.stdout:1517: done (2 left) 2021-07-31T01:50:37.568 INFO:tasks.rados.rados.0.smithi045.stdout:1518: done (1 left) 2021-07-31T01:50:37.568 INFO:tasks.rados.rados.0.smithi045.stdout:1519: done (0 left) 2021-07-31T01:50:37.569 INFO:tasks.rados.rados.0.smithi045.stdout:1520: delete oid 6 current snap is 142 2021-07-31T01:50:37.570 INFO:tasks.rados.rados.0.smithi045.stdout:1520: done (0 left) 2021-07-31T01:50:37.570 INFO:tasks.rados.rados.0.smithi045.stdout:1521: snap_create 2021-07-31T01:50:38.571 INFO:tasks.rados.rados.0.smithi045.stdout:1521: done (0 left) 2021-07-31T01:50:38.571 INFO:tasks.rados.rados.0.smithi045.stdout:1522: snap_remove snap 135 2021-07-31T01:50:39.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:38 smithi154 conmon[34553]: cluster 2021-07-31T01:50:37.565266+0000 mon.a (mon.0) 984 : 2021-07-31T01:50:39.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:38 smithi154 conmon[34553]: cluster [DBG] osdmap e338: 8 total, 8 up, 8 in 2021-07-31T01:50:39.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:38 smithi154 conmon[34553]: cluster 2021-07-31T01:50:37.839197+0000 mgr.x (mgr.24542) 148 : cluster [DBG] pgmap v287: 25 pgs: 25 active+clean; 112 MiB data, 883 MiB used, 714 GiB / 715 GiB avail 2021-07-31T01:50:39.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:38 smithi045 conmon[33655]: cluster 2021-07-31T01:50:37.565266+0000 2021-07-31T01:50:39.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:38 smithi045 conmon[33655]: mon.a (mon.0) 984 : cluster [DBG] osdmap e338: 8 total, 8 up, 8 in 2021-07-31T01:50:39.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:38 smithi045 conmon[33655]: cluster 2021-07-31T01:50:37.839197+0000 mgr.x (mgr.24542) 148 : cluster [DBG] pgmap v287: 25 pgs: 25 active+clean; 112 MiB data, 883 MiB used, 714 GiB / 715 GiB avail 2021-07-31T01:50:39.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:38 smithi045 conmon[39718]: cluster 2021-07-31T01:50:37.565266 2021-07-31T01:50:39.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:38 smithi045 conmon[39718]: +0000 mon.a (mon.0) 984 : cluster [DBG] osdmap e338: 8 total, 8 up, 8 in 2021-07-31T01:50:39.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:38 smithi045 conmon[39718]: cluster 2021-07-31T01:50:37.839197+0000 mgr.x (mgr.24542) 148 : cluster [DBG] pgmap v287: 25 pgs: 25 active+clean; 112 MiB data, 883 MiB used, 714 GiB / 715 GiB avail 2021-07-31T01:50:39.573 INFO:tasks.rados.rados.0.smithi045.stdout:1522: done (0 left) 2021-07-31T01:50:39.574 INFO:tasks.rados.rados.0.smithi045.stdout:1523: delete oid 32 current snap is 143 2021-07-31T01:50:39.579 INFO:tasks.rados.rados.0.smithi045.stdout:1523: done (0 left) 2021-07-31T01:50:39.580 INFO:tasks.rados.rados.0.smithi045.stdout:1524: delete oid 21 current snap is 143 2021-07-31T01:50:39.587 INFO:tasks.rados.rados.0.smithi045.stdout:1524: done (0 left) 2021-07-31T01:50:39.587 INFO:tasks.rados.rados.0.smithi045.stdout:1525: snap_remove snap 139 2021-07-31T01:50:40.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:39 smithi154 conmon[34553]: cluster 2021-07-31T01:50:38.570190+0000 mon.a (mon.0) 985 : cluster [DBG] osdmap e339: 8 total, 8 up, 8 in 2021-07-31T01:50:40.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:39 smithi045 conmon[39718]: cluster 2021-07-31T01:50:40.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:39 smithi045 conmon[39718]: 2021-07-31T01:50:40.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:39 smithi045 conmon[39718]: 2021-07-31T01: 2021-07-31T01:50:40.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:39 smithi045 conmon[39718]: 50:38.570190+0000 mon.a (mon.0) 985 : cluster [DBG] osdmap e339: 8 total, 8 up, 8 in 2021-07-31T01:50:40.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:39 smithi045 conmon[33655]: cluster 2021-07-31T01:50:38.570190+0000 mon.a (mon.0) 985 : cluster [DBG] 2021-07-31T01:50:40.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:39 smithi045 conmon[33655]: osdmap e339: 8 total, 8 up, 8 in 2021-07-31T01:50:40.584 INFO:tasks.rados.rados.0.smithi045.stdout:1525: done (0 left) 2021-07-31T01:50:40.585 INFO:tasks.rados.rados.0.smithi045.stdout:1526: snap_remove snap 140 2021-07-31T01:50:41.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:40 smithi154 conmon[34553]: cluster 2021-07-31T01:50:39.572310+0000 mon.a (mon.0) 986 : cluster 2021-07-31T01:50:41.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:40 smithi154 conmon[34553]: [DBG] osdmap e340: 8 total, 8 up, 8 in 2021-07-31T01:50:41.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:40 smithi154 conmon[34553]: cluster 2021-07-31T01:50:39.839727+0000 mgr.x (mgr.24542) 149 : cluster [DBG] pgmap v290: 25 pgs: 25 active+clean; 122 MiB data, 905 MiB used, 714 GiB / 715 GiB avail; 7.0 MiB/s rd, 2.5 MiB/s wr, 24 op/s 2021-07-31T01:50:41.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:40 smithi045 conmon[39718]: cluster 2021-07-31T01:50:39.572310+0000 mon.a (mon.0) 986 2021-07-31T01:50:41.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:40 smithi045 conmon[39718]: : cluster [DBG] osdmap e340: 8 total, 8 up, 8 in 2021-07-31T01:50:41.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:40 smithi045 conmon[39718]: cluster 2021-07-31T01:50:39.839727+0000 mgr.x (mgr.24542 2021-07-31T01:50:41.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:40 smithi045 conmon[39718]: ) 149 : cluster [DBG] pgmap v290: 25 pgs: 25 active+clean; 122 MiB data, 905 MiB used, 714 GiB / 715 GiB avail; 7.0 MiB/s rd, 2.5 MiB/s wr, 24 op/s 2021-07-31T01:50:41.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:40 smithi045 conmon[33655]: cluster 2021-07-31T01:50:39.572310+0000 mon.a (mon.0) 2021-07-31T01:50:41.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:40 smithi045 conmon[33655]: 986 : cluster [DBG] osdmap e340: 8 total, 8 up, 8 in 2021-07-31T01:50:41.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:40 smithi045 conmon[33655]: cluster 2021-07-31T01:50:39.839727+0000 mgr.x (mgr.24542) 149 : cluster 2021-07-31T01:50:41.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:40 smithi045 conmon[33655]: [DBG] pgmap v290: 25 pgs: 25 active+clean; 122 MiB data, 905 MiB used, 714 GiB / 715 GiB avail; 7.0 MiB/s rd, 2.5 MiB/s wr, 24 op/s 2021-07-31T01:50:41.589 INFO:tasks.rados.rados.0.smithi045.stdout:1526: done (0 left) 2021-07-31T01:50:41.590 INFO:tasks.rados.rados.0.smithi045.stdout:1527: copy_from oid 11 from oid 16 current snap is 143 2021-07-31T01:50:41.590 INFO:tasks.rados.rados.0.smithi045.stdout:1528: rollback oid 40 current snap is 143 2021-07-31T01:50:41.590 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 40 to 141 2021-07-31T01:50:41.590 INFO:tasks.rados.rados.0.smithi045.stdout:1529: read oid 17 snap -1 2021-07-31T01:50:41.590 INFO:tasks.rados.rados.0.smithi045.stdout:1529: expect (ObjNum 474 snap 140 seq_num 474) 2021-07-31T01:50:41.591 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 3 current snap is 143 2021-07-31T01:50:41.591 INFO:tasks.rados.rados.0.smithi045.stdout:1530: seq_num 483 ranges {548864=491520,1040384=401408} 2021-07-31T01:50:41.595 INFO:tasks.rados.rados.0.smithi045.stdout:1530: writing smithi04569269-3 from 548864 to 1040384 tid 1 2021-07-31T01:50:41.598 INFO:tasks.rados.rados.0.smithi045.stdout:1530: writing smithi04569269-3 from 1040384 to 1441792 tid 2 2021-07-31T01:50:41.599 INFO:tasks.rados.rados.0.smithi045.stdout:1531: copy_from oid 41 from oid 24 current snap is 143 2021-07-31T01:50:41.599 INFO:tasks.rados.rados.0.smithi045.stdout:1527: finishing copy_from to smithi04569269-11 2021-07-31T01:50:41.599 INFO:tasks.rados.rados.0.smithi045.stdout:1527: got expected ENOENT (src dne) 2021-07-31T01:50:41.599 INFO:tasks.rados.rados.0.smithi045.stdout:1527: finishing copy_from racing read to smithi04569269-11 2021-07-31T01:50:41.600 INFO:tasks.rados.rados.0.smithi045.stdout:1528: finishing rollback tid 0 to smithi04569269-40 2021-07-31T01:50:41.600 INFO:tasks.rados.rados.0.smithi045.stdout:1528: finishing rollback tid 1 to smithi04569269-40 2021-07-31T01:50:41.600 INFO:tasks.rados.rados.0.smithi045.stdout:1527: done (4 left) 2021-07-31T01:50:41.600 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 50 current snap is 143 2021-07-31T01:50:41.600 INFO:tasks.rados.rados.0.smithi045.stdout:1532: seq_num 484 ranges {0=507904,507904=401408} 2021-07-31T01:50:41.601 INFO:tasks.rados.rados.0.smithi045.stdout:1532: writing smithi04569269-50 from 0 to 507904 tid 1 2021-07-31T01:50:41.604 INFO:tasks.rados.rados.0.smithi045.stdout:1532: writing smithi04569269-50 from 507904 to 909312 tid 2 2021-07-31T01:50:41.604 INFO:tasks.rados.rados.0.smithi045.stdout:1533: copy_from oid 13 from oid 38 current snap is 143 2021-07-31T01:50:41.604 INFO:tasks.rados.rados.0.smithi045.stdout:1534: delete oid 43 current snap is 143 2021-07-31T01:50:41.605 INFO:tasks.rados.rados.0.smithi045.stdout:1528: finishing rollback tid 2 to smithi04569269-40 2021-07-31T01:50:41.605 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 40 v 581 (ObjNum 461 snap 134 seq_num 461) dirty exists 2021-07-31T01:50:41.605 INFO:tasks.rados.rados.0.smithi045.stdout:1529: expect (ObjNum 474 snap 140 seq_num 474) 2021-07-31T01:50:41.652 INFO:tasks.rados.rados.0.smithi045.stdout:1532: finishing write tid 1 to smithi04569269-50 2021-07-31T01:50:41.653 INFO:tasks.rados.rados.0.smithi045.stdout:1530: finishing write tid 1 to smithi04569269-3 2021-07-31T01:50:41.653 INFO:tasks.rados.rados.0.smithi045.stdout:1532: finishing write tid 2 to smithi04569269-50 2021-07-31T01:50:41.653 INFO:tasks.rados.rados.0.smithi045.stdout:1530: finishing write tid 2 to smithi04569269-3 2021-07-31T01:50:41.653 INFO:tasks.rados.rados.0.smithi045.stdout:1530: finishing write tid 3 to smithi04569269-3 2021-07-31T01:50:41.654 INFO:tasks.rados.rados.0.smithi045.stdout:1532: finishing write tid 3 to smithi04569269-50 2021-07-31T01:50:41.654 INFO:tasks.rados.rados.0.smithi045.stdout:1528: done (6 left) 2021-07-31T01:50:41.654 INFO:tasks.rados.rados.0.smithi045.stdout:1529: done (5 left) 2021-07-31T01:50:41.654 INFO:tasks.rados.rados.0.smithi045.stdout:1534: done (4 left) 2021-07-31T01:50:41.654 INFO:tasks.rados.rados.0.smithi045.stdout:1535: snap_remove snap 141 2021-07-31T01:50:42.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:41 smithi154 conmon[34553]: cluster 2021-07-31T01:50:40.583067+0000 mon.a (mon.0) 987 : cluster [DBG] osdmap e341: 8 total, 8 up, 8 in 2021-07-31T01:50:42.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:41 smithi154 conmon[34553]: 2021-07-31T01:50:42.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:41 smithi045 conmon[33655]: cluster 2021-07-31T01:50:40.583067+0000 mon.a (mon.0) 987 : 2021-07-31T01:50:42.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:41 smithi045 conmon[33655]: cluster [DBG] osdmap e341: 8 total, 8 up, 8 in 2021-07-31T01:50:42.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:41 smithi045 conmon[39718]: cluster 2021-07-31T01:50:40.583067+0000 mon.a (mon. 2021-07-31T01:50:42.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:41 smithi045 conmon[39718]: 0) 987 : cluster [DBG] osdmap e341: 8 total, 8 up, 8 in 2021-07-31T01:50:42.596 INFO:tasks.rados.rados.0.smithi045.stdout:1535: done (4 left) 2021-07-31T01:50:42.596 INFO:tasks.rados.rados.0.smithi045.stdout:1536: read oid 18 snap 142 2021-07-31T01:50:42.596 INFO:tasks.rados.rados.0.smithi045.stdout:1536: expect (ObjNum -1105261296 snap 21873 seq_num 3189482896) 2021-07-31T01:50:42.597 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 9 current snap is 143 2021-07-31T01:50:42.597 INFO:tasks.rados.rados.0.smithi045.stdout:1530: finishing write tid 4 to smithi04569269-3 2021-07-31T01:50:42.597 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 3 v 1001 (ObjNum 483 snap 143 seq_num 483) dirty exists 2021-07-31T01:50:42.597 INFO:tasks.rados.rados.0.smithi045.stdout:1530: left oid 3 (ObjNum 483 snap 143 seq_num 483) 2021-07-31T01:50:42.597 INFO:tasks.rados.rados.0.smithi045.stdout:1532: finishing write tid 4 to smithi04569269-50 2021-07-31T01:50:42.598 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 50 v 580 (ObjNum 484 snap 143 seq_num 484) dirty exists 2021-07-31T01:50:42.598 INFO:tasks.rados.rados.0.smithi045.stdout:1532: left oid 50 (ObjNum 484 snap 143 seq_num 484) 2021-07-31T01:50:42.599 INFO:tasks.rados.rados.0.smithi045.stdout:1531: finishing copy_from to smithi04569269-41 2021-07-31T01:50:42.599 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 41 v 508 (ObjNum 462 snap 134 seq_num 462) dirty exists 2021-07-31T01:50:42.599 INFO:tasks.rados.rados.0.smithi045.stdout:1531: finishing copy_from racing read to smithi04569269-41 2021-07-31T01:50:42.599 INFO:tasks.rados.rados.0.smithi045.stdout:1533: finishing copy_from to smithi04569269-13 2021-07-31T01:50:42.600 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 13 v 583 (ObjNum 425 snap 126 seq_num 425) dirty exists 2021-07-31T01:50:42.600 INFO:tasks.rados.rados.0.smithi045.stdout:1533: finishing copy_from racing read to smithi04569269-13 2021-07-31T01:50:42.600 INFO:tasks.rados.rados.0.smithi045.stdout:1537: seq_num 486 ranges {4841472=548864,5390336=401408} 2021-07-31T01:50:42.625 INFO:tasks.rados.rados.0.smithi045.stdout:1537: writing smithi04569269-9 from 4841472 to 5390336 tid 1 2021-07-31T01:50:42.628 INFO:tasks.rados.rados.0.smithi045.stdout:1537: writing smithi04569269-9 from 5390336 to 5791744 tid 2 2021-07-31T01:50:42.628 INFO:tasks.rados.rados.0.smithi045.stdout:1530: done (5 left) 2021-07-31T01:50:42.628 INFO:tasks.rados.rados.0.smithi045.stdout:1531: done (4 left) 2021-07-31T01:50:42.628 INFO:tasks.rados.rados.0.smithi045.stdout:1532: done (3 left) 2021-07-31T01:50:42.629 INFO:tasks.rados.rados.0.smithi045.stdout:1533: done (2 left) 2021-07-31T01:50:42.629 INFO:tasks.rados.rados.0.smithi045.stdout:1538: copy_from oid 13 from oid 43 current snap is 143 2021-07-31T01:50:42.630 INFO:tasks.rados.rados.0.smithi045.stdout:1536: expect (ObjNum -1105261296 snap 21873 seq_num 3189482896) 2021-07-31T01:50:42.633 INFO:tasks.rados.rados.0.smithi045.stdout:1537: finishing write tid 1 to smithi04569269-9 2021-07-31T01:50:42.633 INFO:tasks.rados.rados.0.smithi045.stdout:1537: finishing write tid 2 to smithi04569269-9 2021-07-31T01:50:42.633 INFO:tasks.rados.rados.0.smithi045.stdout:1537: finishing write tid 3 to smithi04569269-9 2021-07-31T01:50:42.634 INFO:tasks.rados.rados.0.smithi045.stdout:1536: done (2 left) 2021-07-31T01:50:42.634 INFO:tasks.rados.rados.0.smithi045.stdout:1539: delete oid 11 current snap is 143 2021-07-31T01:50:42.634 INFO:tasks.rados.rados.0.smithi045.stdout:1537: finishing write tid 4 to smithi04569269-9 2021-07-31T01:50:42.634 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 9 v 381 (ObjNum 486 snap 143 seq_num 486) dirty exists 2021-07-31T01:50:42.635 INFO:tasks.rados.rados.0.smithi045.stdout:1537: left oid 9 (ObjNum 486 snap 143 seq_num 486) 2021-07-31T01:50:42.635 INFO:tasks.rados.rados.0.smithi045.stdout:1538: finishing copy_from to smithi04569269-13 2021-07-31T01:50:42.635 INFO:tasks.rados.rados.0.smithi045.stdout:1538: got expected ENOENT (src dne) 2021-07-31T01:50:42.635 INFO:tasks.rados.rados.0.smithi045.stdout:1538: finishing copy_from racing read to smithi04569269-13 2021-07-31T01:50:42.638 INFO:tasks.rados.rados.0.smithi045.stdout:1537: done (2 left) 2021-07-31T01:50:42.638 INFO:tasks.rados.rados.0.smithi045.stdout:1538: done (1 left) 2021-07-31T01:50:42.638 INFO:tasks.rados.rados.0.smithi045.stdout:1539: done (0 left) 2021-07-31T01:50:42.638 INFO:tasks.rados.rados.0.smithi045.stdout:1540: delete oid 34 current snap is 143 2021-07-31T01:50:42.642 INFO:tasks.rados.rados.0.smithi045.stdout:1540: done (0 left) 2021-07-31T01:50:42.642 INFO:tasks.rados.rados.0.smithi045.stdout:1541: snap_create 2021-07-31T01:50:43.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:42 smithi154 conmon[34553]: cluster 2021-07-31T01:50:41.588274+0000 mon.a (mon 2021-07-31T01:50:43.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:42 smithi154 conmon[34553]: .0) 988 : cluster [DBG] osdmap e342: 8 total, 8 up, 8 in 2021-07-31T01:50:43.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:42 smithi154 conmon[34553]: cluster 2021-07-31T01:50:41.840113+0000 mgr.x (mgr.24542) 150 : cluster 2021-07-31T01:50:43.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:42 smithi154 conmon[34553]: [DBG] pgmap v293: 25 pgs: 25 active+clean; 122 MiB data, 905 MiB used, 714 GiB / 715 GiB avail; 7.0 MiB/s rd, 2.5 MiB/s wr, 24 op/s 2021-07-31T01:50:43.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:42 smithi045 conmon[33655]: cluster 2021-07-31T01:50:41.588274+0000 mon.a (mon.0) 988 2021-07-31T01:50:43.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:42 smithi045 conmon[33655]: : cluster [DBG] osdmap e342: 8 total, 8 up, 8 in 2021-07-31T01:50:43.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:42 smithi045 conmon[33655]: cluster 2021-07-31T01:50:41.840113+0000 mgr.x (mgr.24542) 150 : cluster [DBG] pgmap v293: 25 pgs: 25 active+clean; 122 MiB data, 905 MiB used, 714 GiB / 715 GiB avail; 7.0 MiB/s rd, 2.5 MiB/s wr, 24 op/s 2021-07-31T01:50:43.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:42 smithi045 conmon[39718]: cluster 2021-07-31T01:50:41.588274+0000 mon.a ( 2021-07-31T01:50:43.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:42 smithi045 conmon[39718]: mon.0) 988 : cluster [DBG] osdmap e342: 8 total, 8 up, 8 in 2021-07-31T01:50:43.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:42 smithi045 conmon[39718]: cluster 2021-07-31T01:50:41.840113+0000 mgr.x 2021-07-31T01:50:43.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:42 smithi045 conmon[39718]: (mgr.24542) 150 : cluster [DBG] pgmap v293: 25 pgs: 25 active+clean; 122 MiB data, 905 MiB used, 714 GiB / 715 GiB avail; 7.0 MiB/s rd, 2.5 MiB/s wr, 24 op/s 2021-07-31T01:50:43.597 INFO:tasks.rados.rados.0.smithi045.stdout:1541: done (0 left) 2021-07-31T01:50:43.597 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 49 current snap is 144 2021-07-31T01:50:43.597 INFO:tasks.rados.rados.0.smithi045.stdout:1542: seq_num 489 ranges {6021120=598016,6619136=401408} 2021-07-31T01:50:43.633 INFO:tasks.rados.rados.0.smithi045.stdout:1542: writing smithi04569269-49 from 6021120 to 6619136 tid 1 2021-07-31T01:50:43.635 INFO:tasks.rados.rados.0.smithi045.stdout:1542: writing smithi04569269-49 from 6619136 to 7020544 tid 2 2021-07-31T01:50:43.636 INFO:tasks.rados.rados.0.smithi045.stdout:1543: setattr oid 31 current snap is 144 2021-07-31T01:50:43.637 INFO:tasks.rados.rados.0.smithi045.stdout:1544: read oid 35 snap -1 2021-07-31T01:50:43.638 INFO:tasks.rados.rados.0.smithi045.stdout:1544: expect deleted 2021-07-31T01:50:43.638 INFO:tasks.rados.rados.0.smithi045.stdout:1545: snap_remove snap 143 2021-07-31T01:50:43.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:43 smithi045 conmon[39718]: cluster 2021-07-31T01:50:42.594427+0000 mon.a 2021-07-31T01:50:43.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:43 smithi045 conmon[39718]: (mon.0) 989 : cluster [DBG] osdmap e343: 8 total, 8 up, 8 in 2021-07-31T01:50:43.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:50:43 smithi045 conmon[61210]: level=error ts=2021-07-31T01:50:43.514Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:50:43.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:50:43 smithi045 conmon[61210]: level=error ts=2021-07-31T01:50:43.514Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:50:43.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:50:43 smithi045 conmon[61210]: level=error ts=2021-07-31T01:50:43.514Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T01:50:43.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:43 smithi045 conmon[33655]: cluster 2021-07-31T01:50:42.594427 2021-07-31T01:50:43.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:43 smithi045 conmon[33655]: +0000 mon.a (mon.0) 989 : cluster [DBG] osdmap e343: 8 total, 8 up, 8 in 2021-07-31T01:50:44.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:43 smithi154 conmon[34553]: cluster 2021-07-31T01:50:42.594427+0000 mon.a (mon.0) 989 : cluster 2021-07-31T01:50:44.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:43 smithi154 conmon[34553]: [DBG] osdmap e343: 8 total, 8 up, 8 in 2021-07-31T01:50:44.607 INFO:tasks.rados.rados.0.smithi045.stdout:1545: done (3 left) 2021-07-31T01:50:44.607 INFO:tasks.rados.rados.0.smithi045.stdout:1546: setattr oid 18 current snap is 144 2021-07-31T01:50:44.608 INFO:tasks.rados.rados.0.smithi045.stdout:1542: finishing write tid 1 to smithi04569269-49 2021-07-31T01:50:44.608 INFO:tasks.rados.rados.0.smithi045.stdout:1542: finishing write tid 2 to smithi04569269-49 2021-07-31T01:50:44.609 INFO:tasks.rados.rados.0.smithi045.stdout:1542: finishing write tid 3 to smithi04569269-49 2021-07-31T01:50:44.609 INFO:tasks.rados.rados.0.smithi045.stdout:1542: finishing write tid 4 to smithi04569269-49 2021-07-31T01:50:44.609 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 49 v 1007 (ObjNum 489 snap 144 seq_num 489) dirty exists 2021-07-31T01:50:44.610 INFO:tasks.rados.rados.0.smithi045.stdout:1542: left oid 49 (ObjNum 489 snap 144 seq_num 489) 2021-07-31T01:50:44.610 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 31 v 1009 (ObjNum 385 snap 114 seq_num 385) dirty exists 2021-07-31T01:50:44.611 INFO:tasks.rados.rados.0.smithi045.stdout:1542: done (3 left) 2021-07-31T01:50:44.611 INFO:tasks.rados.rados.0.smithi045.stdout:1543: done (2 left) 2021-07-31T01:50:44.612 INFO:tasks.rados.rados.0.smithi045.stdout:1544: done (1 left) 2021-07-31T01:50:44.612 INFO:tasks.rados.rados.0.smithi045.stdout:1547: snap_create 2021-07-31T01:50:44.614 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 18 v 378 (ObjNum -1104883040 snap 21873 seq_num 3189211904) dirty exists 2021-07-31T01:50:45.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:44 smithi154 conmon[34553]: cluster 2021-07-31T01:50:43.595736+0000 mon.a (mon.0) 990 : cluster 2021-07-31T01:50:45.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:44 smithi154 conmon[34553]: [DBG] osdmap e344: 8 total, 8 up, 8 in 2021-07-31T01:50:45.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:44 smithi154 conmon[34553]: cluster 2021-07-31T01:50:43.840623+0000 mgr.x (mgr.24542) 151 : cluster [DBG] pgmap v296: 25 pgs: 25 active+clean; 137 MiB data, 937 MiB used, 714 GiB / 715 GiB avail; 976 KiB/s rd, 3.9 MiB/s wr, 14 op/s 2021-07-31T01:50:45.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:44 smithi045 conmon[33655]: cluster 2021-07-31T01:50:43.595736+0000 mon.a (mon.0) 990 2021-07-31T01:50:45.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:44 smithi045 conmon[33655]: : cluster [DBG] osdmap e344: 8 total, 8 up, 8 in 2021-07-31T01:50:45.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:44 smithi045 conmon[33655]: cluster 2021-07-31T01:50:43.840623+0000 mgr.x (mgr.24542) 151 : cluster [DBG] pgmap v296: 25 pgs: 25 active+clean; 137 MiB data, 937 MiB used, 714 GiB / 715 GiB avail; 976 KiB/s rd, 3.9 MiB/s wr, 14 op/s 2021-07-31T01:50:45.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:44 smithi045 conmon[33655]: 2021-07-31T01:50:45.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:44 smithi045 conmon[39718]: cluster 2021-07-31T01:50:43.595736+0000 mon.a (mon. 2021-07-31T01:50:45.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:44 smithi045 conmon[39718]: 0) 990 : cluster [DBG] osdmap e344: 8 total, 8 up, 8 in 2021-07-31T01:50:45.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:44 smithi045 conmon[39718]: cluster 2021-07-31T01:50:43.840623+0000 mgr.x (mgr.24542) 2021-07-31T01:50:45.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:44 smithi045 conmon[39718]: 151 : cluster [DBG] pgmap v296: 25 pgs: 25 active+clean; 137 MiB data, 937 MiB used, 714 GiB / 715 GiB avail; 976 KiB/s rd, 3.9 MiB/s wr, 14 op/s 2021-07-31T01:50:45.608 INFO:tasks.rados.rados.0.smithi045.stdout:1546: done (1 left) 2021-07-31T01:50:45.608 INFO:tasks.rados.rados.0.smithi045.stdout:1547: done (0 left) 2021-07-31T01:50:45.608 INFO:tasks.rados.rados.0.smithi045.stdout:1548: snap_create 2021-07-31T01:50:46.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:45 smithi154 conmon[34553]: cluster 2021-07-31T01:50:44.606062+0000 mon.a (mon.0) 991 : cluster [DBG] 2021-07-31T01:50:46.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:45 smithi154 conmon[34553]: osdmap e345: 8 total, 8 up, 8 in 2021-07-31T01:50:46.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:45 smithi045 conmon[33655]: cluster 2021-07-31T01:50:44.606062+0000 2021-07-31T01:50:46.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:45 smithi045 conmon[33655]: mon.a (mon.0) 991 : cluster [DBG] osdmap e345: 8 total, 8 up, 8 in 2021-07-31T01:50:46.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:45 smithi045 conmon[39718]: cluster 2021-07-31T01:50:44.606062+0000 mon.a (mon 2021-07-31T01:50:46.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:45 smithi045 conmon[39718]: .0) 991 : cluster [DBG] osdmap e345: 8 total, 8 up, 8 in 2021-07-31T01:50:46.542 INFO:tasks.rados.rados.0.smithi045.stdout:1548: done (0 left) 2021-07-31T01:50:46.543 INFO:tasks.rados.rados.0.smithi045.stdout:1549: snap_remove snap 145 2021-07-31T01:50:47.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:46 smithi154 conmon[34553]: cluster 2021-07-31T01:50:45.607699+0000 mon.a (mon.0) 992 : cluster [DBG] 2021-07-31T01:50:47.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:46 smithi154 conmon[34553]: osdmap e346: 8 total, 8 up, 8 in 2021-07-31T01:50:47.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:46 smithi154 conmon[34553]: cluster 2021-07-31T01:50:45.841145+0000 mgr.x (mgr.24542) 152 : cluster [DBG] pgmap v299: 25 pgs: 25 active+clean; 137 MiB data, 937 MiB used, 714 GiB / 715 GiB avail; 976 KiB/s rd, 3.9 MiB/s wr, 14 op/s 2021-07-31T01:50:47.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:46 smithi154 conmon[34553]: cluster 2021-07-31T01:50:46.541828+0000 2021-07-31T01:50:47.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:46 smithi154 conmon[34553]: mon.a (mon.0) 993 : cluster [DBG] osdmap e347: 8 total, 8 up, 8 in 2021-07-31T01:50:47.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:46 smithi045 conmon[39718]: cluster 2021-07-31T01:50:45 2021-07-31T01:50:47.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:46 smithi045 conmon[39718]: .607699+0000 mon.a (mon.0) 992 : cluster [DBG] osdmap e346: 8 total, 8 up, 8 in 2021-07-31T01:50:47.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:46 smithi045 conmon[39718]: cluster 2021-07-31T01:50:47.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:46 smithi045 conmon[39718]: 2021-07-31 2021-07-31T01:50:47.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:46 smithi045 conmon[39718]: T01:50:45.841145 2021-07-31T01:50:47.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:46 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 152 2021-07-31T01:50:47.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:46 smithi045 conmon[39718]: : cluster [DBG] pgmap v299: 25 pgs: 25 active+clean; 137 MiB data, 937 MiB used, 714 GiB / 715 GiB avail; 976 KiB/s rd, 3.9 MiB/s wr, 14 op/s 2021-07-31T01:50:47.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:46 smithi045 conmon[39718]: cluster 2021- 2021-07-31T01:50:47.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:46 smithi045 conmon[39718]: 07-31T01:50:46.541828 2021-07-31T01:50:47.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:46 smithi045 conmon[39718]: +0000 mon.a (mon.0) 993 2021-07-31T01:50:47.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:46 smithi045 conmon[39718]: : cluster [DBG] osdmap e347: 8 total, 8 up, 8 in 2021-07-31T01:50:47.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:46 smithi045 conmon[33655]: cluster 2021-07-31T01:50:45.607699+0000 mon.a ( 2021-07-31T01:50:47.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:46 smithi045 conmon[33655]: mon.0) 992 : cluster [DBG] osdmap e346: 8 total, 8 up, 8 in 2021-07-31T01:50:47.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:46 smithi045 conmon[33655]: cluster 2021-07-31T01:50:45.841145+0000 mgr.x (mgr.24542) 152 : cluster [DBG] pgmap v299: 25 pgs: 25 active+clean; 137 MiB data, 937 MiB used, 714 GiB / 715 GiB avail; 976 KiB/s rd, 3.9 MiB/s wr, 14 op/s 2021-07-31T01:50:47.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:46 smithi045 conmon[33655]: cluster 2021-07-31T01:50:46.541828+0000 mon.a (mon.0) 993 : cluster [DBG] osdmap e347: 8 total, 8 up, 8 in 2021-07-31T01:50:47.552 INFO:tasks.rados.rados.0.smithi045.stdout:1549: done (0 left) 2021-07-31T01:50:47.552 INFO:tasks.rados.rados.0.smithi045.stdout:1550: rollback oid 7 current snap is 146 2021-07-31T01:50:47.552 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 7 to 137 2021-07-31T01:50:47.552 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 31 current snap is 146 2021-07-31T01:50:47.553 INFO:tasks.rados.rados.0.smithi045.stdout:1551: seq_num 490 ranges {4243456=614400,4857856=401408} 2021-07-31T01:50:47.578 INFO:tasks.rados.rados.0.smithi045.stdout:1551: writing smithi04569269-31 from 4243456 to 4857856 tid 1 2021-07-31T01:50:47.581 INFO:tasks.rados.rados.0.smithi045.stdout:1551: writing smithi04569269-31 from 4857856 to 5259264 tid 2 2021-07-31T01:50:47.581 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 50 current snap is 146 2021-07-31T01:50:47.582 INFO:tasks.rados.rados.0.smithi045.stdout:1550: finishing rollback tid 0 to smithi04569269-7 2021-07-31T01:50:47.582 INFO:tasks.rados.rados.0.smithi045.stdout:1550: finishing rollback tid 1 to smithi04569269-7 2021-07-31T01:50:47.583 INFO:tasks.rados.rados.0.smithi045.stdout:1552: seq_num 491 ranges {909312=630784,1540096=401408} 2021-07-31T01:50:47.589 INFO:tasks.rados.rados.0.smithi045.stdout:1552: writing smithi04569269-50 from 909312 to 1540096 tid 1 2021-07-31T01:50:47.592 INFO:tasks.rados.rados.0.smithi045.stdout:1552: writing smithi04569269-50 from 1540096 to 1941504 tid 2 2021-07-31T01:50:47.592 INFO:tasks.rados.rados.0.smithi045.stdout:1550: finishing rollback tid 2 to smithi04569269-7 2021-07-31T01:50:47.593 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 7 v 953 (ObjNum 447 snap 132 seq_num 447) dirty exists 2021-07-31T01:50:47.593 INFO:tasks.rados.rados.0.smithi045.stdout:1550: done (2 left) 2021-07-31T01:50:47.593 INFO:tasks.rados.rados.0.smithi045.stdout:1553: read oid 2 snap -1 2021-07-31T01:50:47.593 INFO:tasks.rados.rados.0.smithi045.stdout:1553: expect deleted 2021-07-31T01:50:47.594 INFO:tasks.rados.rados.0.smithi045.stdout:1554: snap_remove snap 142 2021-07-31T01:50:48.558 INFO:tasks.rados.rados.0.smithi045.stdout:1554: done (3 left) 2021-07-31T01:50:48.558 INFO:tasks.rados.rados.0.smithi045.stdout:1555: copy_from oid 36 from oid 30 current snap is 146 2021-07-31T01:50:48.559 INFO:tasks.rados.rados.0.smithi045.stdout:1551: finishing write tid 1 to smithi04569269-31 2021-07-31T01:50:48.559 INFO:tasks.rados.rados.0.smithi045.stdout:1551: finishing write tid 2 to smithi04569269-31 2021-07-31T01:50:48.559 INFO:tasks.rados.rados.0.smithi045.stdout:1551: finishing write tid 3 to smithi04569269-31 2021-07-31T01:50:48.559 INFO:tasks.rados.rados.0.smithi045.stdout:1551: finishing write tid 4 to smithi04569269-31 2021-07-31T01:50:48.560 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 31 v 1017 (ObjNum 490 snap 146 seq_num 490) dirty exists 2021-07-31T01:50:48.560 INFO:tasks.rados.rados.0.smithi045.stdout:1551: left oid 31 (ObjNum 490 snap 146 seq_num 490) 2021-07-31T01:50:48.560 INFO:tasks.rados.rados.0.smithi045.stdout:1552: finishing write tid 1 to smithi04569269-50 2021-07-31T01:50:48.560 INFO:tasks.rados.rados.0.smithi045.stdout:1552: finishing write tid 2 to smithi04569269-50 2021-07-31T01:50:48.560 INFO:tasks.rados.rados.0.smithi045.stdout:1552: finishing write tid 3 to smithi04569269-50 2021-07-31T01:50:48.561 INFO:tasks.rados.rados.0.smithi045.stdout:1552: finishing write tid 4 to smithi04569269-50 2021-07-31T01:50:48.561 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 50 v 584 (ObjNum 491 snap 146 seq_num 491) dirty exists 2021-07-31T01:50:48.561 INFO:tasks.rados.rados.0.smithi045.stdout:1552: left oid 50 (ObjNum 491 snap 146 seq_num 491) 2021-07-31T01:50:48.561 INFO:tasks.rados.rados.0.smithi045.stdout:1551: done (3 left) 2021-07-31T01:50:48.562 INFO:tasks.rados.rados.0.smithi045.stdout:1552: done (2 left) 2021-07-31T01:50:48.562 INFO:tasks.rados.rados.0.smithi045.stdout:1553: done (1 left) 2021-07-31T01:50:48.562 INFO:tasks.rados.rados.0.smithi045.stdout:1556: read oid 38 snap -1 2021-07-31T01:50:48.563 INFO:tasks.rados.rados.0.smithi045.stdout:1556: expect (ObjNum 425 snap 126 seq_num 425) 2021-07-31T01:50:48.563 INFO:tasks.rados.rados.0.smithi045.stdout:1557: copy_from oid 39 from oid 3 current snap is 146 2021-07-31T01:50:48.563 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 22 current snap is 146 2021-07-31T01:50:48.564 INFO:tasks.rados.rados.0.smithi045.stdout:1558: seq_num 492 ranges {5382144=647168,6029312=401408} 2021-07-31T01:50:48.592 INFO:tasks.rados.rados.0.smithi045.stdout:1558: writing smithi04569269-22 from 5382144 to 6029312 tid 1 2021-07-31T01:50:48.595 INFO:tasks.rados.rados.0.smithi045.stdout:1558: writing smithi04569269-22 from 6029312 to 6430720 tid 2 2021-07-31T01:50:48.596 INFO:tasks.rados.rados.0.smithi045.stdout:1559: snap_remove snap 138 2021-07-31T01:50:49.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:48 smithi154 conmon[34553]: cluster 2021-07-31T01:50:47.551160+0000 mon.a (mon.0) 994 : cluster [DBG] osdmap e348: 8 total, 8 up, 8 in 2021-07-31T01:50:49.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:48 smithi154 conmon[34553]: cluster 2021-07-31T01:50:47.841537+0000 mgr.x (mgr.24542) 153 : cluster [DBG] pgmap v302: 25 pgs: 25 active+clean; 137 MiB data, 937 MiB used, 714 GiB / 715 GiB avail 2021-07-31T01:50:49.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:48 smithi045 conmon[33655]: cluster 2021-07-31T01:50:47.551160+0000 2021-07-31T01:50:49.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:48 smithi045 conmon[33655]: mon.a (mon.0) 994 : cluster [DBG] osdmap e348: 8 total, 8 up, 8 in 2021-07-31T01:50:49.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:48 smithi045 conmon[33655]: cluster 2021-07-31T01: 2021-07-31T01:50:49.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:48 smithi045 conmon[33655]: 50:47.841537+0000 mgr.x (mgr.24542) 153 : cluster [DBG] 2021-07-31T01:50:49.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:48 smithi045 conmon[33655]: pgmap v302: 25 pgs: 25 active+clean; 137 MiB data, 937 MiB used, 714 GiB / 715 GiB avail 2021-07-31T01:50:49.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:48 smithi045 conmon[39718]: cluster 2021-07-31T01:50:47.551160+0000 2021-07-31T01:50:49.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:48 smithi045 conmon[39718]: mon.a (mon.0) 994 : cluster [DBG] osdmap e348: 8 total, 8 up, 8 in 2021-07-31T01:50:49.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:48 smithi045 conmon[39718]: cluster 2021-07-31T01 2021-07-31T01:50:49.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:48 smithi045 conmon[39718]: :50:47.841537+0000 mgr.x (mgr.24542) 153 : cluster 2021-07-31T01:50:49.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:48 smithi045 conmon[39718]: [DBG] pgmap v302: 25 pgs: 25 active+clean; 137 MiB data, 937 MiB used, 714 GiB / 715 GiB avail 2021-07-31T01:50:49.558 INFO:tasks.rados.rados.0.smithi045.stdout:1559: done (4 left) 2021-07-31T01:50:49.559 INFO:tasks.rados.rados.0.smithi045.stdout:1560: copy_from oid 17 from oid 15 current snap is 146 2021-07-31T01:50:49.559 INFO:tasks.rados.rados.0.smithi045.stdout:1555: finishing copy_from to smithi04569269-36 2021-07-31T01:50:49.559 INFO:tasks.rados.rados.0.smithi045.stdout:1555: got expected ENOENT (src dne) 2021-07-31T01:50:49.559 INFO:tasks.rados.rados.0.smithi045.stdout:1555: finishing copy_from racing read to smithi04569269-36 2021-07-31T01:50:49.560 INFO:tasks.rados.rados.0.smithi045.stdout:1558: finishing write tid 1 to smithi04569269-22 2021-07-31T01:50:49.560 INFO:tasks.rados.rados.0.smithi045.stdout:1558: finishing write tid 2 to smithi04569269-22 2021-07-31T01:50:49.560 INFO:tasks.rados.rados.0.smithi045.stdout:1558: finishing write tid 3 to smithi04569269-22 2021-07-31T01:50:49.560 INFO:tasks.rados.rados.0.smithi045.stdout:1558: finishing write tid 4 to smithi04569269-22 2021-07-31T01:50:49.560 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 22 v 503 (ObjNum 492 snap 146 seq_num 492) dirty exists 2021-07-31T01:50:49.561 INFO:tasks.rados.rados.0.smithi045.stdout:1558: left oid 22 (ObjNum 492 snap 146 seq_num 492) 2021-07-31T01:50:49.561 INFO:tasks.rados.rados.0.smithi045.stdout:1556: expect (ObjNum 425 snap 126 seq_num 425) 2021-07-31T01:50:50.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:49 smithi154 conmon[34553]: cluster 2021-07-31T01:50:48.556300+0000 mon.a ( 2021-07-31T01:50:50.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:49 smithi154 conmon[34553]: mon.0) 995 : cluster [DBG] osdmap e349: 8 total, 8 up, 8 in 2021-07-31T01:50:50.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:49 smithi045 conmon[33655]: cluster 2021-07-31T01:50:48.556300+0000 mon.a (mon.0 2021-07-31T01:50:50.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:49 smithi045 conmon[33655]: ) 995 : cluster [DBG] osdmap e349: 8 total, 8 up, 8 in 2021-07-31T01:50:50.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:49 smithi045 conmon[39718]: cluster 2021-07-31T01:50:48.556300+0000 2021-07-31T01:50:50.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:49 smithi045 conmon[39718]: mon.a (mon.0) 995 : cluster [DBG] osdmap e349: 8 total, 8 up, 8 in 2021-07-31T01:50:50.104 INFO:tasks.rados.rados.0.smithi045.stdout:1557: finishing copy_from to smithi04569269-39 2021-07-31T01:50:50.105 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 39 v 1001 (ObjNum 483 snap 143 seq_num 483) dirty exists 2021-07-31T01:50:50.105 INFO:tasks.rados.rados.0.smithi045.stdout:1557: finishing copy_from racing read to smithi04569269-39 2021-07-31T01:50:50.105 INFO:tasks.rados.rados.0.smithi045.stdout:1560: finishing copy_from to smithi04569269-17 2021-07-31T01:50:50.105 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 17 v 368 (ObjNum 479 snap 141 seq_num 479) dirty exists 2021-07-31T01:50:50.106 INFO:tasks.rados.rados.0.smithi045.stdout:1560: finishing copy_from racing read to smithi04569269-17 2021-07-31T01:50:50.106 INFO:tasks.rados.rados.0.smithi045.stdout:1555: done (4 left) 2021-07-31T01:50:50.106 INFO:tasks.rados.rados.0.smithi045.stdout:1556: done (3 left) 2021-07-31T01:50:50.106 INFO:tasks.rados.rados.0.smithi045.stdout:1557: done (2 left) 2021-07-31T01:50:50.107 INFO:tasks.rados.rados.0.smithi045.stdout:1558: done (1 left) 2021-07-31T01:50:50.107 INFO:tasks.rados.rados.0.smithi045.stdout:1560: done (0 left) 2021-07-31T01:50:50.107 INFO:tasks.rados.rados.0.smithi045.stdout:1561: snap_create 2021-07-31T01:50:50.570 INFO:tasks.rados.rados.0.smithi045.stdout:1561: done (0 left) 2021-07-31T01:50:50.571 INFO:tasks.rados.rados.0.smithi045.stdout:1562: delete oid 30 current snap is 147 2021-07-31T01:50:50.573 INFO:tasks.rados.rados.0.smithi045.stdout:1562: done (0 left) 2021-07-31T01:50:50.574 INFO:tasks.rados.rados.0.smithi045.stdout:1563: rollback oid 41 current snap is 147 2021-07-31T01:50:50.574 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 41 to 137 2021-07-31T01:50:50.574 INFO:tasks.rados.rados.0.smithi045.stdout:1564: read oid 42 snap -1 2021-07-31T01:50:50.575 INFO:tasks.rados.rados.0.smithi045.stdout:1564: expect (ObjNum 176 snap 46 seq_num 176) 2021-07-31T01:50:50.575 INFO:tasks.rados.rados.0.smithi045.stdout:1565: read oid 22 snap -1 2021-07-31T01:50:50.575 INFO:tasks.rados.rados.0.smithi045.stdout:1565: expect (ObjNum 492 snap 146 seq_num 492) 2021-07-31T01:50:50.575 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 12 current snap is 147 2021-07-31T01:50:50.576 INFO:tasks.rados.rados.0.smithi045.stdout:1566: seq_num 494 ranges {0=679936,679936=401408} 2021-07-31T01:50:50.578 INFO:tasks.rados.rados.0.smithi045.stdout:1566: writing smithi04569269-12 from 0 to 679936 tid 1 2021-07-31T01:50:50.580 INFO:tasks.rados.rados.0.smithi045.stdout:1566: writing smithi04569269-12 from 679936 to 1081344 tid 2 2021-07-31T01:50:50.580 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 43 current snap is 147 2021-07-31T01:50:50.581 INFO:tasks.rados.rados.0.smithi045.stdout:1567: seq_num 495 ranges {0=696320,696320=401408} 2021-07-31T01:50:50.584 INFO:tasks.rados.rados.0.smithi045.stdout:1567: writing smithi04569269-43 from 0 to 696320 tid 1 2021-07-31T01:50:50.587 INFO:tasks.rados.rados.0.smithi045.stdout:1567: writing smithi04569269-43 from 696320 to 1097728 tid 2 2021-07-31T01:50:50.588 INFO:tasks.rados.rados.0.smithi045.stdout:1568: snap_create 2021-07-31T01:50:50.588 INFO:tasks.rados.rados.0.smithi045.stdout:1564: expect (ObjNum 176 snap 46 seq_num 176) 2021-07-31T01:50:50.607 INFO:tasks.rados.rados.0.smithi045.stdout:1563: finishing rollback tid 0 to smithi04569269-41 2021-07-31T01:50:50.607 INFO:tasks.rados.rados.0.smithi045.stdout:1563: finishing rollback tid 1 to smithi04569269-41 2021-07-31T01:50:50.607 INFO:tasks.rados.rados.0.smithi045.stdout:1563: finishing rollback tid 2 to smithi04569269-41 2021-07-31T01:50:50.608 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 41 v 956 (ObjNum 442 snap 129 seq_num 442) dirty exists 2021-07-31T01:50:50.608 INFO:tasks.rados.rados.0.smithi045.stdout:1566: finishing write tid 1 to smithi04569269-12 2021-07-31T01:50:50.608 INFO:tasks.rados.rados.0.smithi045.stdout:1566: finishing write tid 2 to smithi04569269-12 2021-07-31T01:50:50.608 INFO:tasks.rados.rados.0.smithi045.stdout:1566: finishing write tid 3 to smithi04569269-12 2021-07-31T01:50:50.609 INFO:tasks.rados.rados.0.smithi045.stdout:1566: finishing write tid 4 to smithi04569269-12 2021-07-31T01:50:50.609 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 12 v 586 (ObjNum 494 snap 147 seq_num 494) dirty exists 2021-07-31T01:50:50.609 INFO:tasks.rados.rados.0.smithi045.stdout:1566: left oid 12 (ObjNum 494 snap 147 seq_num 494) 2021-07-31T01:50:50.609 INFO:tasks.rados.rados.0.smithi045.stdout:1567: finishing write tid 1 to smithi04569269-43 2021-07-31T01:50:50.609 INFO:tasks.rados.rados.0.smithi045.stdout:1567: finishing write tid 2 to smithi04569269-43 2021-07-31T01:50:50.610 INFO:tasks.rados.rados.0.smithi045.stdout:1567: finishing write tid 3 to smithi04569269-43 2021-07-31T01:50:50.610 INFO:tasks.rados.rados.0.smithi045.stdout:1567: finishing write tid 4 to smithi04569269-43 2021-07-31T01:50:50.610 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 43 v 936 (ObjNum 495 snap 147 seq_num 495) dirty exists 2021-07-31T01:50:50.610 INFO:tasks.rados.rados.0.smithi045.stdout:1567: left oid 43 (ObjNum 495 snap 147 seq_num 495) 2021-07-31T01:50:50.611 INFO:tasks.rados.rados.0.smithi045.stdout:1565: expect (ObjNum 492 snap 146 seq_num 492) 2021-07-31T01:50:51.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:50 smithi154 conmon[34553]: cluster 2021-07-31T01:50:49.557773+0000 mon.a (mon.0) 996 : cluster [DBG] osdmap e350: 8 total, 8 up, 8 in 2021-07-31T01:50:51.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:50 smithi154 conmon[34553]: cluster 2021-07-31T01:50:49.841998+0000 2021-07-31T01:50:51.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:50 smithi154 conmon[34553]: mgr.x (mgr.24542) 154 : cluster [DBG] pgmap v305: 25 pgs: 25 active+clean; 142 MiB data, 950 MiB used, 714 GiB / 715 GiB avail; 6.9 MiB/s rd, 1.1 MiB/s wr, 10 op/s 2021-07-31T01:50:51.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:50 smithi045 conmon[33655]: cluster 2021-07-31T01:50:49.557773+0000 mon.a (mon.0) 2021-07-31T01:50:51.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:50 smithi045 conmon[33655]: 996 : cluster [DBG] osdmap e350: 8 total, 8 up, 8 in 2021-07-31T01:50:51.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:50 smithi045 conmon[33655]: cluster 2021-07-31T01:50:49.841998+0000 mgr.x (mgr.24542 2021-07-31T01:50:51.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:50 smithi045 conmon[33655]: ) 154 : cluster [DBG] pgmap v305: 25 pgs: 25 active+clean; 142 MiB data, 950 MiB used, 714 GiB / 715 GiB avail; 6.9 MiB/s rd, 1.1 MiB/s wr, 10 op/s 2021-07-31T01:50:51.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:50 smithi045 conmon[39718]: cluster 2021-07-31T01:50 2021-07-31T01:50:51.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:50 smithi045 conmon[39718]: :49.557773+0000 mon.a (mon.0) 996 : cluster [DBG] osdmap e350: 8 total, 8 up, 8 in 2021-07-31T01:50:51.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:50 smithi045 conmon[39718]: cluster 2021 2021-07-31T01:50:51.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:50 smithi045 conmon[39718]: -07-31T01:50:49.841998+0000 2021-07-31T01:50:51.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:50 smithi045 conmon[39718]: mgr.x (mgr.24542) 154 : cluster [DBG] 2021-07-31T01:50:51.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:50 smithi045 conmon[39718]: pgmap v305: 25 pgs: 25 active+clean; 142 MiB data, 950 MiB used, 714 GiB / 715 GiB avail; 6.9 MiB/s rd, 1.1 MiB/s wr, 10 op/s 2021-07-31T01:50:51.544 INFO:tasks.rados.rados.0.smithi045.stdout:1563: done (5 left) 2021-07-31T01:50:51.545 INFO:tasks.rados.rados.0.smithi045.stdout:1564: done (4 left) 2021-07-31T01:50:51.545 INFO:tasks.rados.rados.0.smithi045.stdout:1565: done (3 left) 2021-07-31T01:50:51.545 INFO:tasks.rados.rados.0.smithi045.stdout:1566: done (2 left) 2021-07-31T01:50:51.545 INFO:tasks.rados.rados.0.smithi045.stdout:1567: done (1 left) 2021-07-31T01:50:51.546 INFO:tasks.rados.rados.0.smithi045.stdout:1568: done (0 left) 2021-07-31T01:50:51.546 INFO:tasks.rados.rados.0.smithi045.stdout:1569: rollback oid 13 current snap is 148 2021-07-31T01:50:51.546 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 13 to 147 2021-07-31T01:50:51.546 INFO:tasks.rados.rados.0.smithi045.stdout:1570: read oid 46 snap -1 2021-07-31T01:50:51.547 INFO:tasks.rados.rados.0.smithi045.stdout:1570: expect (ObjNum 454 snap 133 seq_num 454) 2021-07-31T01:50:51.547 INFO:tasks.rados.rados.0.smithi045.stdout:1571: snap_remove snap 137 2021-07-31T01:50:52.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:51 smithi154 conmon[34553]: cluster 2021-07-31T01:50:50.570286+0000 mon.a (mon.0) 997 : cluster 2021-07-31T01:50:52.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:51 smithi154 conmon[34553]: [DBG] osdmap e351: 8 total, 8 up, 8 in 2021-07-31T01:50:52.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:51 smithi154 conmon[34553]: cluster 2021-07-31T01:50:51.543522+0000 mon.a (mon.0) 998 : cluster [DBG] osdmap e352: 8 total, 8 up, 8 in 2021-07-31T01:50:52.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:51 smithi045 conmon[33655]: cluster 2021-07-31T01:50:50.570286+0000 mon.a (mon.0) 997 : cluster [DBG] 2021-07-31T01:50:52.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:51 smithi045 conmon[33655]: osdmap e351: 8 total, 8 up, 8 in 2021-07-31T01:50:52.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:51 smithi045 conmon[33655]: cluster 2021-07-31T01:50:51.543522+0000 mon.a (mon.0) 998 : cluster [DBG] osdmap e352: 8 total, 8 up, 8 in 2021-07-31T01:50:52.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:51 smithi045 conmon[39718]: cluster 2021-07-31T01:50 2021-07-31T01:50:52.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:51 smithi045 conmon[39718]: :50.570286+0000 mon.a (mon.0) 997 : cluster [DBG] osdmap e351: 8 total, 8 up, 8 in 2021-07-31T01:50:52.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:51 smithi045 conmon[39718]: cluster 2021-07-31T01:50:52.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:51 smithi045 conmon[39718]: 2021-07-31T01:50: 2021-07-31T01:50:52.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:51 smithi045 conmon[39718]: 51.543522+0000 mon.a (mon. 2021-07-31T01:50:52.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:51 smithi045 conmon[39718]: 0) 998 : cluster [DBG] 2021-07-31T01:50:52.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:51 smithi045 conmon[39718]: osdmap e352: 8 total, 8 up, 8 in 2021-07-31T01:50:52.552 INFO:tasks.rados.rados.0.smithi045.stdout:1571: done (2 left) 2021-07-31T01:50:52.552 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 44 current snap is 148 2021-07-31T01:50:52.553 INFO:tasks.rados.rados.0.smithi045.stdout:1569: finishing rollback tid 0 to smithi04569269-13 2021-07-31T01:50:52.553 INFO:tasks.rados.rados.0.smithi045.stdout:1572: seq_num 496 ranges {794624=712704,1507328=401408} 2021-07-31T01:50:52.560 INFO:tasks.rados.rados.0.smithi045.stdout:1572: writing smithi04569269-44 from 794624 to 1507328 tid 1 2021-07-31T01:50:52.563 INFO:tasks.rados.rados.0.smithi045.stdout:1572: writing smithi04569269-44 from 1507328 to 1908736 tid 2 2021-07-31T01:50:52.564 INFO:tasks.rados.rados.0.smithi045.stdout:1573: delete oid 8 current snap is 148 2021-07-31T01:50:52.564 INFO:tasks.rados.rados.0.smithi045.stdout:1569: finishing rollback tid 1 to smithi04569269-13 2021-07-31T01:50:52.564 INFO:tasks.rados.rados.0.smithi045.stdout:1569: finishing rollback tid 2 to smithi04569269-13 2021-07-31T01:50:52.564 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 13 v 589 (ObjNum 425 snap 126 seq_num 425) dirty exists 2021-07-31T01:50:52.564 INFO:tasks.rados.rados.0.smithi045.stdout:1570: expect (ObjNum 454 snap 133 seq_num 454) 2021-07-31T01:50:53.003 INFO:journalctl@ceph.osd.5.smithi154.stdout:Jul 31 01:50:52 smithi154 conmon[42050]: debug 2021-07-31T01:50:52.559+0000 7fc9b5064700 -1 osd.5 pg_epoch: 353 pg[3.cs0( v 353'580 (0'0,353'580] local-lis/les=56/57 n=7 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [5,3,6]p5(0) r=0 lpr=56 luod=350'578 crt=350'578 lcod 350'576 mlcod 350'576 active+clean+snaptrim trimq=[113~1] ps=[114~1]] removing snap head 2021-07-31T01:50:53.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:52 smithi154 conmon[34553]: cluster 2021-07-31T01:50:51.842382+0000 mgr.x (mgr.24542 2021-07-31T01:50:53.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:52 smithi154 conmon[34553]: ) 155 : cluster [DBG] pgmap v308: 25 pgs: 25 active+clean; 142 MiB data, 950 MiB used, 714 GiB / 715 GiB avail; 6.9 MiB/s rd, 1.1 MiB/s wr, 10 op/s 2021-07-31T01:50:53.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:52 smithi154 conmon[34553]: cluster 2021-07-31T01:50:52.551275+0000 mon.a (mon.0) 999 : cluster [DBG] osdmap e353: 8 total, 8 up, 8 in 2021-07-31T01:50:53.004 INFO:journalctl@ceph.osd.7.smithi154.stdout:Jul 31 01:50:52 smithi154 conmon[49149]: debug 2021-07-31T01:50:52.551+0000 7f8a75203700 -1 osd.7 pg_epoch: 353 pg[3.es0( v 352'562 (0'0,352'562] local-lis/les=56/57 n=9 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [7,4,1]p7(0) r=0 lpr=56 crt=352'562 lcod 352'561 mlcod 352'561 active+clean+snaptrim trimq=[113~1] ps=[114~1]] removing snap head 2021-07-31T01:50:53.004 INFO:journalctl@ceph.osd.7.smithi154.stdout:Jul 31 01:50:52 smithi154 conmon[49149]: debug 2021-07-31T01:50:52.569+0000 7f8a709fa700 -1 osd.7 pg_epoch: 353 pg[3.fs0( v 350'262 (0'0,350'262] local-lis/les=56/57 n=3 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [7,4,0]p7(0) r=0 lpr=56 crt=350'262 lcod 349'260 mlcod 349'260 active+clean+snaptrim trimq=[113~1] ps=[114~1]] removing snap head 2021-07-31T01:50:53.026 INFO:journalctl@ceph.osd.1.smithi045.stdout:Jul 31 01:50:52 smithi045 conmon[46771]: debug 2021-07-31T01:50:52.548+0000 7f41959a6700 -1 osd.1 pg_epoch: 353 pg[3.4s0( v 350'338 (0'0,350'338] local-lis/les=56/57 n=4 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [1,2,6]p1(0) r=0 lpr=56 crt=350'338 lcod 349'336 mlcod 349'336 active+clean+snaptrim trimq=[113~1] ps=[114~1]] removing snap head 2021-07-31T01:50:53.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:52 smithi045 conmon[33655]: cluster 2021-07-31T01:50:51.842382+0000 mgr.x (mgr.24542 2021-07-31T01:50:53.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:52 smithi045 conmon[33655]: ) 155 : cluster [DBG] pgmap v308: 25 pgs: 25 active+clean; 142 MiB data, 950 MiB used, 714 GiB / 715 GiB avail; 6.9 MiB/s rd, 1.1 MiB/s wr, 10 op/s 2021-07-31T01:50:53.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:52 smithi045 conmon[33655]: cluster 2021-07-31T01:50:52.551275+0000 mon.a (mon.0) 999 : cluster [DBG] osdmap e353: 8 total, 8 up, 8 in 2021-07-31T01:50:53.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:52 smithi045 conmon[39718]: cluster 2021-07-31T01:50:51.842382+0000 mgr.x ( 2021-07-31T01:50:53.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:52 smithi045 conmon[39718]: mgr.24542) 155 : cluster [DBG] pgmap v308: 25 pgs: 25 active+clean; 142 MiB data, 950 MiB used, 714 GiB / 715 GiB avail; 6.9 MiB/s rd, 1.1 MiB/s wr, 10 op/s 2021-07-31T01:50:53.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:52 smithi045 conmon[39718]: cluster 2021-07-31T01:50 2021-07-31T01:50:53.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:52 smithi045 conmon[39718]: :52.551275+0000 mon.a (mon.0) 999 : cluster [DBG] osdmap e353: 8 total, 8 up, 8 in 2021-07-31T01:50:53.090 INFO:tasks.rados.rados.0.smithi045.stdout:1572: finishing write tid 1 to smithi04569269-44 2021-07-31T01:50:53.091 INFO:tasks.rados.rados.0.smithi045.stdout:1572: finishing write tid 2 to smithi04569269-44 2021-07-31T01:50:53.091 INFO:tasks.rados.rados.0.smithi045.stdout:1572: finishing write tid 3 to smithi04569269-44 2021-07-31T01:50:53.091 INFO:tasks.rados.rados.0.smithi045.stdout:1572: finishing write tid 4 to smithi04569269-44 2021-07-31T01:50:53.091 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 44 v 939 (ObjNum 496 snap 148 seq_num 496) dirty exists 2021-07-31T01:50:53.092 INFO:tasks.rados.rados.0.smithi045.stdout:1572: left oid 44 (ObjNum 496 snap 148 seq_num 496) 2021-07-31T01:50:53.092 INFO:tasks.rados.rados.0.smithi045.stdout:1569: done (3 left) 2021-07-31T01:50:53.092 INFO:tasks.rados.rados.0.smithi045.stdout:1570: done (2 left) 2021-07-31T01:50:53.092 INFO:tasks.rados.rados.0.smithi045.stdout:1572: done (1 left) 2021-07-31T01:50:53.092 INFO:tasks.rados.rados.0.smithi045.stdout:1573: done (0 left) 2021-07-31T01:50:53.093 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 6 current snap is 148 2021-07-31T01:50:53.093 INFO:tasks.rados.rados.0.smithi045.stdout:1574: seq_num 498 ranges {0=745472,745472=401408} 2021-07-31T01:50:53.094 INFO:tasks.rados.rados.0.smithi045.stdout:1574: writing smithi04569269-6 from 0 to 745472 tid 1 2021-07-31T01:50:53.096 INFO:tasks.rados.rados.0.smithi045.stdout:1574: writing smithi04569269-6 from 745472 to 1146880 tid 2 2021-07-31T01:50:53.096 INFO:tasks.rados.rados.0.smithi045.stdout:1575: snap_create 2021-07-31T01:50:53.099 INFO:tasks.rados.rados.0.smithi045.stdout:1574: finishing write tid 1 to smithi04569269-6 2021-07-31T01:50:53.100 INFO:tasks.rados.rados.0.smithi045.stdout:1574: finishing write tid 2 to smithi04569269-6 2021-07-31T01:50:53.100 INFO:tasks.rados.rados.0.smithi045.stdout:1574: finishing write tid 3 to smithi04569269-6 2021-07-31T01:50:53.101 INFO:tasks.rados.rados.0.smithi045.stdout:1574: finishing write tid 4 to smithi04569269-6 2021-07-31T01:50:53.101 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 6 v 592 (ObjNum 498 snap 148 seq_num 498) dirty exists 2021-07-31T01:50:53.102 INFO:tasks.rados.rados.0.smithi045.stdout:1574: left oid 6 (ObjNum 498 snap 148 seq_num 498) 2021-07-31T01:50:53.580 INFO:tasks.rados.rados.0.smithi045.stdout:1574: done (1 left) 2021-07-31T01:50:53.580 INFO:tasks.rados.rados.0.smithi045.stdout:1575: done (0 left) 2021-07-31T01:50:53.581 INFO:tasks.rados.rados.0.smithi045.stdout:1576: rollback oid 21 current snap is 149 2021-07-31T01:50:53.581 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 21 to 148 2021-07-31T01:50:53.581 INFO:tasks.rados.rados.0.smithi045.stdout:1577: read oid 13 snap 146 2021-07-31T01:50:53.581 INFO:tasks.rados.rados.0.smithi045.stdout:1577: expect (ObjNum 425 snap 126 seq_num 425) 2021-07-31T01:50:53.582 INFO:tasks.rados.rados.0.smithi045.stdout:1578: rmattr oid 24 current snap is 149 2021-07-31T01:50:53.582 INFO:tasks.rados.rados.0.smithi045.stdout:1578: done (2 left) 2021-07-31T01:50:53.582 INFO:tasks.rados.rados.0.smithi045.stdout:1579: snap_remove snap 144 2021-07-31T01:50:53.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:50:53 smithi045 conmon[61210]: level=error ts=2021-07-31T01:50:53.515Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:50:53.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:50:53 smithi045 conmon[61210]: level=error ts=2021-07-31T01:50:53.515Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:50:53.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:50:53 smithi045 conmon[61210]: level=error ts=2021-07-31T01:50:53.515Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T01:50:54.588 INFO:tasks.rados.rados.0.smithi045.stdout:1579: done (2 left) 2021-07-31T01:50:54.589 INFO:tasks.rados.rados.0.smithi045.stdout:1580: setattr oid 33 current snap is 149 2021-07-31T01:50:54.590 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 44 current snap is 149 2021-07-31T01:50:54.591 INFO:tasks.rados.rados.0.smithi045.stdout:1581: seq_num 499 ranges {1908736=761856,2670592=401408} 2021-07-31T01:50:54.605 INFO:tasks.rados.rados.0.smithi045.stdout:1581: writing smithi04569269-44 from 1908736 to 2670592 tid 1 2021-07-31T01:50:54.608 INFO:tasks.rados.rados.0.smithi045.stdout:1581: writing smithi04569269-44 from 2670592 to 3072000 tid 2 2021-07-31T01:50:54.608 INFO:tasks.rados.rados.0.smithi045.stdout:1582: copy_from oid 7 from oid 32 current snap is 149 2021-07-31T01:50:54.609 INFO:tasks.rados.rados.0.smithi045.stdout:1576: finishing rollback tid 1 to smithi04569269-21 2021-07-31T01:50:54.609 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 21 v 0 (ObjNum 0 snap 0 seq_num 0) dirty dne 2021-07-31T01:50:54.609 INFO:tasks.rados.rados.0.smithi045.stdout:1576: done (4 left) 2021-07-31T01:50:54.609 INFO:tasks.rados.rados.0.smithi045.stdout:1583: rollback oid 34 current snap is 149 2021-07-31T01:50:54.609 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 34 to 147 2021-07-31T01:50:54.610 INFO:tasks.rados.rados.0.smithi045.stdout:1577: expect (ObjNum 425 snap 126 seq_num 425) 2021-07-31T01:50:55.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:54 smithi154 conmon[34553]: cluster 2021-07-31T01:50:53.579512+0000 mon.a (mon.0) 1000 : cluster [DBG] osdmap e354: 8 total, 8 up, 8 in 2021-07-31T01:50:55.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:54 smithi154 conmon[34553]: 2021-07-31T01:50:55.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:54 smithi154 conmon[34553]: cluster 2021-07-31T01:50:53.842976+0000 2021-07-31T01:50:55.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:54 smithi154 conmon[34553]: mgr.x (mgr.24542) 156 : cluster [DBG] pgmap v311: 25 pgs: 25 active+clean; 133 MiB data, 958 MiB used, 714 GiB / 715 GiB avail; 18 MiB/s rd, 1.9 MiB/s wr, 9 op/s 2021-07-31T01:50:55.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:54 smithi045 conmon[33655]: cluster 2021-07-31T01:50:53.579512+0000 mon.a (mon.0) 1000 2021-07-31T01:50:55.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:54 smithi045 conmon[33655]: : cluster [DBG] osdmap e354: 8 total, 8 up, 8 in 2021-07-31T01:50:55.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:54 smithi045 conmon[33655]: cluster 2021-07-31T01:50:53.842976+0000 mgr.x (mgr.24542) 156 : cluster [DBG] pgmap v311: 25 pgs: 25 active+clean; 133 MiB data, 958 MiB used, 714 GiB / 715 GiB avail; 18 MiB/s rd, 1.9 MiB/s wr, 9 op/s 2021-07-31T01:50:55.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:54 smithi045 conmon[39718]: cluster 2021-07-31T01:50:53.579512+0000 mon.a ( 2021-07-31T01:50:55.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:54 smithi045 conmon[39718]: mon.0) 1000 : cluster [DBG] osdmap e354: 8 total, 8 up, 8 in 2021-07-31T01:50:55.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:54 smithi045 conmon[39718]: cluster 2021-07-31T01:50:53.842976+0000 mgr.x (mgr. 2021-07-31T01:50:55.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:54 smithi045 conmon[39718]: 24542) 156 : cluster [DBG] pgmap v311: 25 pgs: 25 active+clean; 133 MiB data, 958 MiB used, 714 GiB / 715 GiB avail; 18 MiB/s rd, 1.9 MiB/s wr, 9 op/s 2021-07-31T01:50:55.149 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 33 v 1002 (ObjNum 399 snap 121 seq_num 399) dirty exists 2021-07-31T01:50:55.149 INFO:tasks.rados.rados.0.smithi045.stdout:1577: done (4 left) 2021-07-31T01:50:55.149 INFO:tasks.rados.rados.0.smithi045.stdout:1580: done (3 left) 2021-07-31T01:50:55.150 INFO:tasks.rados.rados.0.smithi045.stdout:1584: delete oid 27 current snap is 149 2021-07-31T01:50:55.150 INFO:tasks.rados.rados.0.smithi045.stdout:1583: finishing rollback tid 1 to smithi04569269-34 2021-07-31T01:50:55.151 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 34 v 0 (ObjNum -1106732768 snap 21873 seq_num 3189598416) dirty dne 2021-07-31T01:50:55.151 INFO:tasks.rados.rados.0.smithi045.stdout:1582: finishing copy_from to smithi04569269-7 2021-07-31T01:50:55.152 INFO:tasks.rados.rados.0.smithi045.stdout:1582: got expected ENOENT (src dne) 2021-07-31T01:50:55.152 INFO:tasks.rados.rados.0.smithi045.stdout:1582: finishing copy_from racing read to smithi04569269-7 2021-07-31T01:50:55.153 INFO:tasks.rados.rados.0.smithi045.stdout:1581: finishing write tid 1 to smithi04569269-44 2021-07-31T01:50:55.153 INFO:tasks.rados.rados.0.smithi045.stdout:1581: finishing write tid 2 to smithi04569269-44 2021-07-31T01:50:55.153 INFO:tasks.rados.rados.0.smithi045.stdout:1581: finishing write tid 3 to smithi04569269-44 2021-07-31T01:50:55.154 INFO:tasks.rados.rados.0.smithi045.stdout:1581: finishing write tid 4 to smithi04569269-44 2021-07-31T01:50:55.154 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 44 v 942 (ObjNum 499 snap 149 seq_num 499) dirty exists 2021-07-31T01:50:55.155 INFO:tasks.rados.rados.0.smithi045.stdout:1581: left oid 44 (ObjNum 499 snap 149 seq_num 499) 2021-07-31T01:50:55.158 INFO:tasks.rados.rados.0.smithi045.stdout:1581: done (3 left) 2021-07-31T01:50:55.158 INFO:tasks.rados.rados.0.smithi045.stdout:1582: done (2 left) 2021-07-31T01:50:55.159 INFO:tasks.rados.rados.0.smithi045.stdout:1583: done (1 left) 2021-07-31T01:50:55.159 INFO:tasks.rados.rados.0.smithi045.stdout:1584: done (0 left) 2021-07-31T01:50:55.159 INFO:tasks.rados.rados.0.smithi045.stdout:1585: copy_from oid 37 from oid 47 current snap is 149 2021-07-31T01:50:55.160 INFO:tasks.rados.rados.0.smithi045.stdout:1586: copy_from oid 17 from oid 36 current snap is 149 2021-07-31T01:50:55.160 INFO:tasks.rados.rados.0.smithi045.stdout:1587: read oid 27 snap -1 2021-07-31T01:50:55.160 INFO:tasks.rados.rados.0.smithi045.stdout:1587: expect deleted 2021-07-31T01:50:55.160 INFO:tasks.rados.rados.0.smithi045.stdout:1588: delete oid 23 current snap is 149 2021-07-31T01:50:55.160 INFO:tasks.rados.rados.0.smithi045.stdout:1587: done (3 left) 2021-07-31T01:50:55.161 INFO:tasks.rados.rados.0.smithi045.stdout:1588: done (2 left) 2021-07-31T01:50:55.161 INFO:tasks.rados.rados.0.smithi045.stdout:1589: rmattr oid 11 current snap is 149 2021-07-31T01:50:55.161 INFO:tasks.rados.rados.0.smithi045.stdout:1589: done (2 left) 2021-07-31T01:50:55.161 INFO:tasks.rados.rados.0.smithi045.stdout:1590: read oid 46 snap -1 2021-07-31T01:50:55.162 INFO:tasks.rados.rados.0.smithi045.stdout:1590: expect (ObjNum 454 snap 133 seq_num 454) 2021-07-31T01:50:55.162 INFO:tasks.rados.rados.0.smithi045.stdout:1591: snap_create 2021-07-31T01:50:55.162 INFO:tasks.rados.rados.0.smithi045.stdout:1586: finishing copy_from to smithi04569269-17 2021-07-31T01:50:55.163 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 17 v 990 (ObjNum 288 snap 0 seq_num 465) dirty exists 2021-07-31T01:50:55.163 INFO:tasks.rados.rados.0.smithi045.stdout:1586: finishing copy_from racing read to smithi04569269-17 2021-07-31T01:50:55.163 INFO:tasks.rados.rados.0.smithi045.stdout:1585: finishing copy_from to smithi04569269-37 2021-07-31T01:50:55.163 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 37 v 275 (ObjNum 172 snap 41 seq_num 172) dirty exists 2021-07-31T01:50:55.164 INFO:tasks.rados.rados.0.smithi045.stdout:1585: finishing copy_from racing read to smithi04569269-37 2021-07-31T01:50:55.188 INFO:tasks.rados.rados.0.smithi045.stdout:1590: expect (ObjNum 454 snap 133 seq_num 454) 2021-07-31T01:50:55.711 INFO:tasks.rados.rados.0.smithi045.stdout:1585: done (3 left) 2021-07-31T01:50:55.712 INFO:tasks.rados.rados.0.smithi045.stdout:1586: done (2 left) 2021-07-31T01:50:55.712 INFO:tasks.rados.rados.0.smithi045.stdout:1590: done (1 left) 2021-07-31T01:50:55.712 INFO:tasks.rados.rados.0.smithi045.stdout:1591: done (0 left) 2021-07-31T01:50:55.712 INFO:tasks.rados.rados.0.smithi045.stdout:1592: snap_create 2021-07-31T01:50:56.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:55 smithi154 conmon[34553]: cluster 2021-07-31T01:50:54.585975+0000 mon.a (mon.0) 1001 : cluster [DBG] osdmap e355: 8 total, 8 up, 8 in 2021-07-31T01:50:56.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:55 smithi045 conmon[33655]: cluster 2021-07-31T01:50:54.585975+0000 mon.a (mon.0) 1001 2021-07-31T01:50:56.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:55 smithi045 conmon[33655]: : cluster [DBG] osdmap e355: 8 total, 8 up, 8 in 2021-07-31T01:50:56.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:55 smithi045 conmon[39718]: cluster 2021-07-31T01:50:54.585975+0000 mon.a (mon. 2021-07-31T01:50:56.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:55 smithi045 conmon[39718]: 0) 1001 : cluster [DBG] osdmap e355: 8 total, 8 up, 8 in 2021-07-31T01:50:56.547 INFO:tasks.rados.rados.0.smithi045.stdout:1592: done (0 left) 2021-07-31T01:50:56.547 INFO:tasks.rados.rados.0.smithi045.stdout:1593: snap_remove snap 147 2021-07-31T01:50:57.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:56 smithi154 conmon[34553]: cluster 2021-07-31T01:50:55.590641+0000 mon.a (mon.0) 1002 : cluster [DBG] osdmap e356: 8 total, 8 up, 8 in 2021-07-31T01:50:57.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:56 smithi154 conmon[34553]: cluster 2021-07-31T01:50:55.843332+0000 mgr.x (mgr.24542) 157 : cluster [DBG] pgmap v314: 25 pgs: 25 active+clean; 133 MiB data, 958 MiB used, 714 GiB / 715 GiB avail; 18 MiB/s rd, 1.9 MiB/s wr, 9 op/s 2021-07-31T01:50:57.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:56 smithi154 conmon[34553]: cluster 2021-07-31T01:50:56.546538+0000 mon.a (mon.0) 1003 : cluster [DBG] osdmap e357: 8 total, 8 up, 8 in 2021-07-31T01:50:57.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:56 smithi045 conmon[39718]: cluster 2021-07-31T01:50:55.590641+0000 2021-07-31T01:50:57.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:56 smithi045 conmon[39718]: mon.a (mon.0) 1002 : cluster [DBG] osdmap e356: 8 total, 8 up, 8 in 2021-07-31T01:50:57.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:56 smithi045 conmon[39718]: cluster 2021-07-31T 2021-07-31T01:50:57.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:56 smithi045 conmon[39718]: 01:50:55.843332+0000 mgr.x (mgr.24542) 157 : cluster 2021-07-31T01:50:57.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:56 smithi045 conmon[39718]: [DBG] pgmap v314: 25 pgs: 25 active+clean; 133 MiB data, 958 MiB used, 714 GiB / 715 GiB avail; 18 MiB/s rd, 1.9 MiB/s wr, 9 op/s 2021-07-31T01:50:57.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:56 smithi045 conmon[39718]: cluster 2021-07-31T01:50:57.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:56 smithi045 conmon[39718]: 2021-07-31T01: 2021-07-31T01:50:57.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:56 smithi045 conmon[39718]: 50:56.546538+0000 mon.a 2021-07-31T01:50:57.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:56 smithi045 conmon[39718]: (mon.0) 1003 : cluster [DBG] 2021-07-31T01:50:57.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:56 smithi045 conmon[39718]: osdmap e357: 8 total, 8 up, 8 in 2021-07-31T01:50:57.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:56 smithi045 conmon[33655]: cluster 2021-07-31T01:50:55.590641+0000 mon.a (mon.0) 1002 : 2021-07-31T01:50:57.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:56 smithi045 conmon[33655]: cluster [DBG] osdmap e356: 8 total, 8 up, 8 in 2021-07-31T01:50:57.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:56 smithi045 conmon[33655]: cluster 2021-07-31T01:50:55.843332+0000 mgr.x (mgr.24542) 157 : cluster [DBG] pgmap v314: 25 pgs: 25 active+clean; 133 MiB data, 958 MiB used, 714 GiB / 715 GiB avail; 18 MiB/s rd, 1.9 MiB/s wr, 9 op/s 2021-07-31T01:50:57.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:56 smithi045 conmon[33655]: cluster 2021-07-31T01:50:56. 2021-07-31T01:50:57.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:56 smithi045 conmon[33655]: 546538+0000 mon.a (mon.0 2021-07-31T01:50:57.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:56 smithi045 conmon[33655]: ) 1003 : cluster [DBG] osdmap e357: 8 total, 8 up, 8 in 2021-07-31T01:50:57.553 INFO:tasks.rados.rados.0.smithi045.stdout:1593: done (0 left) 2021-07-31T01:50:57.554 INFO:tasks.rados.rados.0.smithi045.stdout:1594: rollback oid 35 current snap is 151 2021-07-31T01:50:57.554 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 35 to 148 2021-07-31T01:50:57.555 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 29 current snap is 151 2021-07-31T01:50:57.555 INFO:tasks.rados.rados.0.smithi045.stdout:1595: seq_num 502 ranges {4882432=409600,5292032=417792,5709824=385024} 2021-07-31T01:50:57.582 INFO:tasks.rados.rados.0.smithi045.stdout:1595: writing smithi04569269-29 from 4882432 to 5292032 tid 1 2021-07-31T01:50:57.584 INFO:tasks.rados.rados.0.smithi045.stdout:1595: writing smithi04569269-29 from 5292032 to 5709824 tid 2 2021-07-31T01:50:57.588 INFO:tasks.rados.rados.0.smithi045.stdout:1595: writing smithi04569269-29 from 5709824 to 6094848 tid 3 2021-07-31T01:50:57.588 INFO:tasks.rados.rados.0.smithi045.stdout:1596: delete oid 30 current snap is 151 2021-07-31T01:50:57.588 INFO:tasks.rados.rados.0.smithi045.stdout:1594: finishing rollback tid 1 to smithi04569269-35 2021-07-31T01:50:57.589 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 35 v 0 (ObjNum 1617261679 snap 1834898272 seq_num 1599498065) dirty dne 2021-07-31T01:50:57.589 INFO:tasks.rados.rados.0.smithi045.stdout:1595: finishing write tid 1 to smithi04569269-29 2021-07-31T01:50:57.590 INFO:tasks.rados.rados.0.smithi045.stdout:1594: done (2 left) 2021-07-31T01:50:57.590 INFO:tasks.rados.rados.0.smithi045.stdout:1596: done (1 left) 2021-07-31T01:50:57.591 INFO:tasks.rados.rados.0.smithi045.stdout:1597: delete oid 19 current snap is 151 2021-07-31T01:50:57.592 INFO:tasks.rados.rados.0.smithi045.stdout:1595: finishing write tid 2 to smithi04569269-29 2021-07-31T01:50:57.595 INFO:tasks.rados.rados.0.smithi045.stdout:1595: finishing write tid 3 to smithi04569269-29 2021-07-31T01:50:57.595 INFO:tasks.rados.rados.0.smithi045.stdout:1595: finishing write tid 4 to smithi04569269-29 2021-07-31T01:50:57.595 INFO:tasks.rados.rados.0.smithi045.stdout:1597: done (1 left) 2021-07-31T01:50:57.596 INFO:tasks.rados.rados.0.smithi045.stdout:1598: read oid 28 snap -1 2021-07-31T01:50:57.596 INFO:tasks.rados.rados.0.smithi045.stdout:1598: expect (ObjNum 333 snap 97 seq_num 333) 2021-07-31T01:50:57.596 INFO:tasks.rados.rados.0.smithi045.stdout:1599: read oid 4 snap -1 2021-07-31T01:50:57.597 INFO:tasks.rados.rados.0.smithi045.stdout:1599: expect (ObjNum 470 snap 136 seq_num 470) 2021-07-31T01:50:57.597 INFO:tasks.rados.rados.0.smithi045.stdout:1600: snap_create 2021-07-31T01:50:57.598 INFO:tasks.rados.rados.0.smithi045.stdout:1595: finishing write tid 5 to smithi04569269-29 2021-07-31T01:50:57.598 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 29 v 198 (ObjNum 502 snap 151 seq_num 502) dirty exists 2021-07-31T01:50:57.599 INFO:tasks.rados.rados.0.smithi045.stdout:1595: left oid 29 (ObjNum 502 snap 151 seq_num 502) 2021-07-31T01:50:57.600 INFO:tasks.rados.rados.0.smithi045.stdout:1598: expect (ObjNum 333 snap 97 seq_num 333) 2021-07-31T01:50:57.627 INFO:tasks.rados.rados.0.smithi045.stdout:1599: expect (ObjNum 470 snap 136 seq_num 470) 2021-07-31T01:50:58.559 INFO:tasks.rados.rados.0.smithi045.stdout:1595: done (3 left) 2021-07-31T01:50:58.559 INFO:tasks.rados.rados.0.smithi045.stdout:1598: done (2 left) 2021-07-31T01:50:58.560 INFO:tasks.rados.rados.0.smithi045.stdout:1599: done (1 left) 2021-07-31T01:50:58.560 INFO:tasks.rados.rados.0.smithi045.stdout:1600: done (0 left) 2021-07-31T01:50:58.560 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 16 current snap is 152 2021-07-31T01:50:58.561 INFO:tasks.rados.rados.0.smithi045.stdout:1601: seq_num 505 ranges {0=466944,466944=475136,942080=327680} 2021-07-31T01:50:58.562 INFO:tasks.rados.rados.0.smithi045.stdout:1601: writing smithi04569269-16 from 0 to 466944 tid 1 2021-07-31T01:50:58.564 INFO:tasks.rados.rados.0.smithi045.stdout:1601: writing smithi04569269-16 from 466944 to 942080 tid 2 2021-07-31T01:50:58.567 INFO:tasks.rados.rados.0.smithi045.stdout:1601: writing smithi04569269-16 from 942080 to 1269760 tid 3 2021-07-31T01:50:58.567 INFO:tasks.rados.rados.0.smithi045.stdout:1602: copy_from oid 50 from oid 48 current snap is 152 2021-07-31T01:50:58.567 INFO:tasks.rados.rados.0.smithi045.stdout:1603: copy_from oid 35 from oid 37 current snap is 152 2021-07-31T01:50:58.568 INFO:tasks.rados.rados.0.smithi045.stdout:1604: rollback oid 8 current snap is 152 2021-07-31T01:50:58.568 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 8 to 151 2021-07-31T01:50:58.568 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 12 current snap is 152 2021-07-31T01:50:58.569 INFO:tasks.rados.rados.0.smithi045.stdout:1605: seq_num 506 ranges {1081344=483328,1564672=761856,2326528=40960} 2021-07-31T01:50:58.576 INFO:tasks.rados.rados.0.smithi045.stdout:1605: writing smithi04569269-12 from 1081344 to 1564672 tid 1 2021-07-31T01:50:58.581 INFO:tasks.rados.rados.0.smithi045.stdout:1605: writing smithi04569269-12 from 1564672 to 2326528 tid 2 2021-07-31T01:50:58.581 INFO:tasks.rados.rados.0.smithi045.stdout:1605: writing smithi04569269-12 from 2326528 to 2367488 tid 3 2021-07-31T01:50:58.581 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 25 current snap is 152 2021-07-31T01:50:58.582 INFO:tasks.rados.rados.0.smithi045.stdout:1601: finishing write tid 1 to smithi04569269-16 2021-07-31T01:50:58.583 INFO:tasks.rados.rados.0.smithi045.stdout:1606: seq_num 507 ranges {696320=499712,1196032=638976,1835008=163840} 2021-07-31T01:50:58.587 INFO:tasks.rados.rados.0.smithi045.stdout:1606: writing smithi04569269-25 from 696320 to 1196032 tid 1 2021-07-31T01:50:58.590 INFO:tasks.rados.rados.0.smithi045.stdout:1606: writing smithi04569269-25 from 1196032 to 1835008 tid 2 2021-07-31T01:50:58.593 INFO:tasks.rados.rados.0.smithi045.stdout:1606: writing smithi04569269-25 from 1835008 to 1998848 tid 3 2021-07-31T01:50:58.594 INFO:tasks.rados.rados.0.smithi045.stdout:1607: delete oid 41 current snap is 152 2021-07-31T01:50:58.595 INFO:tasks.rados.rados.0.smithi045.stdout:1601: finishing write tid 2 to smithi04569269-16 2021-07-31T01:50:58.595 INFO:tasks.rados.rados.0.smithi045.stdout:1604: finishing rollback tid 1 to smithi04569269-8 2021-07-31T01:50:58.595 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 8 v 1035 (ObjNum -1126017128 snap 21873 seq_num 1) dirty dne 2021-07-31T01:50:58.595 INFO:tasks.rados.rados.0.smithi045.stdout:1601: finishing write tid 3 to smithi04569269-16 2021-07-31T01:50:58.595 INFO:tasks.rados.rados.0.smithi045.stdout:1601: finishing write tid 4 to smithi04569269-16 2021-07-31T01:50:58.596 INFO:tasks.rados.rados.0.smithi045.stdout:1601: finishing write tid 5 to smithi04569269-16 2021-07-31T01:50:58.596 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 16 v 508 (ObjNum 505 snap 152 seq_num 505) dirty exists 2021-07-31T01:50:58.596 INFO:tasks.rados.rados.0.smithi045.stdout:1601: left oid 16 (ObjNum 505 snap 152 seq_num 505) 2021-07-31T01:50:58.597 INFO:tasks.rados.rados.0.smithi045.stdout:1603: finishing copy_from to smithi04569269-35 2021-07-31T01:50:58.597 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 35 v 275 (ObjNum 172 snap 41 seq_num 172) dirty exists 2021-07-31T01:50:58.598 INFO:tasks.rados.rados.0.smithi045.stdout:1603: finishing copy_from racing read to smithi04569269-35 2021-07-31T01:50:58.598 INFO:tasks.rados.rados.0.smithi045.stdout:1605: finishing write tid 1 to smithi04569269-12 2021-07-31T01:50:58.598 INFO:tasks.rados.rados.0.smithi045.stdout:1605: finishing write tid 2 to smithi04569269-12 2021-07-31T01:50:58.598 INFO:tasks.rados.rados.0.smithi045.stdout:1605: finishing write tid 3 to smithi04569269-12 2021-07-31T01:50:58.599 INFO:tasks.rados.rados.0.smithi045.stdout:1605: finishing write tid 4 to smithi04569269-12 2021-07-31T01:50:58.599 INFO:tasks.rados.rados.0.smithi045.stdout:1605: finishing write tid 5 to smithi04569269-12 2021-07-31T01:50:58.599 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 12 v 596 (ObjNum 506 snap 152 seq_num 506) dirty exists 2021-07-31T01:50:58.600 INFO:tasks.rados.rados.0.smithi045.stdout:1605: left oid 12 (ObjNum 506 snap 152 seq_num 506) 2021-07-31T01:50:58.600 INFO:tasks.rados.rados.0.smithi045.stdout:1606: finishing write tid 1 to smithi04569269-25 2021-07-31T01:50:58.602 INFO:tasks.rados.rados.0.smithi045.stdout:1606: finishing write tid 2 to smithi04569269-25 2021-07-31T01:50:58.602 INFO:tasks.rados.rados.0.smithi045.stdout:1601: done (6 left) 2021-07-31T01:50:58.603 INFO:tasks.rados.rados.0.smithi045.stdout:1603: done (5 left) 2021-07-31T01:50:58.603 INFO:tasks.rados.rados.0.smithi045.stdout:1604: done (4 left) 2021-07-31T01:50:58.603 INFO:tasks.rados.rados.0.smithi045.stdout:1605: done (3 left) 2021-07-31T01:50:58.603 INFO:tasks.rados.rados.0.smithi045.stdout:1607: done (2 left) 2021-07-31T01:50:58.604 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 16 current snap is 152 2021-07-31T01:50:58.604 INFO:tasks.rados.rados.0.smithi045.stdout:1608: seq_num 509 ranges {1269760=532480,1802240=409600,2211840=393216} 2021-07-31T01:50:58.606 INFO:tasks.rados.rados.0.smithi045.stdout:1608: writing smithi04569269-16 from 1269760 to 1802240 tid 1 2021-07-31T01:50:58.609 INFO:tasks.rados.rados.0.smithi045.stdout:1608: writing smithi04569269-16 from 1802240 to 2211840 tid 2 2021-07-31T01:50:58.612 INFO:tasks.rados.rados.0.smithi045.stdout:1608: writing smithi04569269-16 from 2211840 to 2605056 tid 3 2021-07-31T01:50:58.612 INFO:tasks.rados.rados.0.smithi045.stdout:1609: rollback oid 29 current snap is 152 2021-07-31T01:50:58.612 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 29 to 149 2021-07-31T01:50:58.612 INFO:tasks.rados.rados.0.smithi045.stdout:1606: finishing write tid 3 to smithi04569269-25 2021-07-31T01:50:58.613 INFO:tasks.rados.rados.0.smithi045.stdout:1606: finishing write tid 4 to smithi04569269-25 2021-07-31T01:50:58.613 INFO:tasks.rados.rados.0.smithi045.stdout:1606: finishing write tid 5 to smithi04569269-25 2021-07-31T01:50:58.613 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 25 v 279 (ObjNum 507 snap 152 seq_num 507) dirty exists 2021-07-31T01:50:58.613 INFO:tasks.rados.rados.0.smithi045.stdout:1606: left oid 25 (ObjNum 507 snap 152 seq_num 507) 2021-07-31T01:50:58.613 INFO:tasks.rados.rados.0.smithi045.stdout:1602: finishing copy_from to smithi04569269-50 2021-07-31T01:50:58.614 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 50 v 903 (ObjNum 450 snap 133 seq_num 450) dirty exists 2021-07-31T01:50:58.614 INFO:tasks.rados.rados.0.smithi045.stdout:1602: finishing copy_from racing read to smithi04569269-50 2021-07-31T01:50:58.614 INFO:tasks.rados.rados.0.smithi045.stdout:1608: finishing write tid 1 to smithi04569269-16 2021-07-31T01:50:58.614 INFO:tasks.rados.rados.0.smithi045.stdout:1602: done (3 left) 2021-07-31T01:50:58.614 INFO:tasks.rados.rados.0.smithi045.stdout:1606: done (2 left) 2021-07-31T01:50:58.615 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 46 current snap is 152 2021-07-31T01:50:58.615 INFO:tasks.rados.rados.0.smithi045.stdout:1610: seq_num 510 ranges {7979008=548864,8527872=745472,9273344=57344} 2021-07-31T01:50:58.658 INFO:tasks.rados.rados.0.smithi045.stdout:1610: writing smithi04569269-46 from 7979008 to 8527872 tid 1 2021-07-31T01:50:58.662 INFO:tasks.rados.rados.0.smithi045.stdout:1610: writing smithi04569269-46 from 8527872 to 9273344 tid 2 2021-07-31T01:50:58.663 INFO:tasks.rados.rados.0.smithi045.stdout:1610: writing smithi04569269-46 from 9273344 to 9330688 tid 3 2021-07-31T01:50:58.664 INFO:tasks.rados.rados.0.smithi045.stdout:1611: setattr oid 13 current snap is 152 2021-07-31T01:50:58.664 INFO:tasks.rados.rados.0.smithi045.stdout:1608: finishing write tid 2 to smithi04569269-16 2021-07-31T01:50:58.664 INFO:tasks.rados.rados.0.smithi045.stdout:1609: finishing rollback tid 0 to smithi04569269-29 2021-07-31T01:50:58.664 INFO:tasks.rados.rados.0.smithi045.stdout:1608: finishing write tid 3 to smithi04569269-16 2021-07-31T01:50:58.665 INFO:tasks.rados.rados.0.smithi045.stdout:1609: finishing rollback tid 1 to smithi04569269-29 2021-07-31T01:50:58.665 INFO:tasks.rados.rados.0.smithi045.stdout:1608: finishing write tid 4 to smithi04569269-16 2021-07-31T01:50:58.665 INFO:tasks.rados.rados.0.smithi045.stdout:1609: finishing rollback tid 2 to smithi04569269-29 2021-07-31T01:50:58.665 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 29 v 201 (ObjNum 410 snap 123 seq_num 410) dirty exists 2021-07-31T01:50:58.666 INFO:tasks.rados.rados.0.smithi045.stdout:1608: finishing write tid 5 to smithi04569269-16 2021-07-31T01:50:58.666 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 16 v 512 (ObjNum 509 snap 152 seq_num 509) dirty exists 2021-07-31T01:50:58.666 INFO:tasks.rados.rados.0.smithi045.stdout:1608: left oid 16 (ObjNum 509 snap 152 seq_num 509) 2021-07-31T01:50:58.666 INFO:tasks.rados.rados.0.smithi045.stdout:1610: finishing write tid 1 to smithi04569269-46 2021-07-31T01:50:58.667 INFO:tasks.rados.rados.0.smithi045.stdout:1608: done (3 left) 2021-07-31T01:50:58.667 INFO:tasks.rados.rados.0.smithi045.stdout:1609: done (2 left) 2021-07-31T01:50:58.667 INFO:tasks.rados.rados.0.smithi045.stdout:1612: setattr oid 17 current snap is 152 2021-07-31T01:50:58.668 INFO:tasks.rados.rados.0.smithi045.stdout:1613: copy_from oid 43 from oid 39 current snap is 152 2021-07-31T01:50:58.668 INFO:tasks.rados.rados.0.smithi045.stdout:1614: read oid 38 snap -1 2021-07-31T01:50:58.668 INFO:tasks.rados.rados.0.smithi045.stdout:1614: expect (ObjNum 425 snap 126 seq_num 425) 2021-07-31T01:50:58.669 INFO:tasks.rados.rados.0.smithi045.stdout:1615: delete oid 32 current snap is 152 2021-07-31T01:50:58.669 INFO:tasks.rados.rados.0.smithi045.stdout:1615: done (5 left) 2021-07-31T01:50:58.669 INFO:tasks.rados.rados.0.smithi045.stdout:1616: rollback oid 10 current snap is 152 2021-07-31T01:50:58.669 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 10 to 149 2021-07-31T01:50:58.670 INFO:tasks.rados.rados.0.smithi045.stdout:1617: snap_remove snap 151 2021-07-31T01:50:59.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:58 smithi154 conmon[34553]: cluster 2021-07-31T01:50:57.553083+0000 mon.a (mon.0) 1004 : 2021-07-31T01:50:59.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:58 smithi154 conmon[34553]: cluster [DBG] osdmap e358: 8 total, 8 up, 8 in 2021-07-31T01:50:59.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:58 smithi154 conmon[34553]: cluster 2021-07-31T01:50:57.843795+0000 mgr.x (mgr.24542) 158 : cluster [DBG] pgmap v317: 25 pgs: 25 active+clean; 133 MiB data, 958 MiB used, 714 GiB / 715 GiB avail 2021-07-31T01:50:59.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:58 smithi045 conmon[39718]: cluster 2021-07-31T01:50:57.553083+0000 mon.a (mon.0 2021-07-31T01:50:59.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:58 smithi045 conmon[39718]: ) 1004 : cluster [DBG] osdmap e358: 8 total, 8 up, 8 in 2021-07-31T01:50:59.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:58 smithi045 conmon[39718]: cluster 2021-07-31T01:50:57.843795+0000 mgr.x (mgr.24542) 158 : cluster [DBG] 2021-07-31T01:50:59.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:58 smithi045 conmon[39718]: pgmap v317: 25 pgs: 25 active+clean; 133 MiB data, 958 MiB used, 714 GiB / 715 GiB avail 2021-07-31T01:50:59.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:58 smithi045 conmon[33655]: cluster 2021-07-31T01:50:57.553083+0000 mon.a (mon.0) 2021-07-31T01:50:59.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:58 smithi045 conmon[33655]: 1004 : cluster [DBG] osdmap e358: 8 total, 8 up, 8 in 2021-07-31T01:50:59.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:58 smithi045 conmon[33655]: cluster 2021-07-31T01:50:57.843795+0000 mgr.x (mgr.24542) 158 : cluster [DBG] pgmap v317: 25 pgs: 25 active+clean; 133 MiB data, 958 MiB used, 714 GiB / 715 GiB avail 2021-07-31T01:50:59.561 INFO:tasks.rados.rados.0.smithi045.stdout:1617: done (6 left) 2021-07-31T01:50:59.561 INFO:tasks.rados.rados.0.smithi045.stdout:1618: copy_from oid 27 from oid 25 current snap is 152 2021-07-31T01:50:59.561 INFO:tasks.rados.rados.0.smithi045.stdout:1619: snap_remove snap 150 2021-07-31T01:51:00.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:59 smithi154 conmon[34553]: cluster 2021-07-31T01:51:00.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:59 smithi154 conmon[34553]: 2021-07-31T01:51:00.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:59 smithi154 conmon[34553]: 2021 2021-07-31T01:51:00.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:59 smithi154 conmon[34553]: - 2021-07-31T01:51:00.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:59 smithi154 conmon[34553]: 07 2021-07-31T01:51:00.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:50:59 smithi154 conmon[34553]: -31T01:50:58.558300+0000 mon.a (mon.0) 1005 : cluster [DBG] osdmap e359: 8 total, 8 up, 8 in 2021-07-31T01:51:00.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:59 smithi045 conmon[33655]: cluster 2021-07-31T01:50:58.558300+0000 mon.a (mon. 2021-07-31T01:51:00.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:50:59 smithi045 conmon[33655]: 0) 1005 : cluster [DBG] osdmap e359: 8 total, 8 up, 8 in 2021-07-31T01:51:00.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:59 smithi045 conmon[39718]: cluster 2021-07-31T01:50:58.558300+0000 mon.a ( 2021-07-31T01:51:00.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:50:59 smithi045 conmon[39718]: mon.0) 1005 : cluster [DBG] osdmap e359: 8 total, 8 up, 8 in 2021-07-31T01:51:00.572 INFO:tasks.rados.rados.0.smithi045.stdout:1619: done (7 left) 2021-07-31T01:51:00.573 INFO:tasks.rados.rados.0.smithi045.stdout:1620: read oid 14 snap -1 2021-07-31T01:51:00.573 INFO:tasks.rados.rados.0.smithi045.stdout:1620: expect deleted 2021-07-31T01:51:00.573 INFO:tasks.rados.rados.0.smithi045.stdout:1621: copy_from oid 48 from oid 26 current snap is 152 2021-07-31T01:51:00.574 INFO:tasks.rados.rados.0.smithi045.stdout:1622: copy_from oid 29 from oid 9 current snap is 152 2021-07-31T01:51:00.578 INFO:tasks.rados.rados.0.smithi045.stdout:1623: read oid 6 snap 149 2021-07-31T01:51:00.578 INFO:tasks.rados.rados.0.smithi045.stdout:1623: expect (ObjNum 498 snap 148 seq_num 498) 2021-07-31T01:51:00.578 INFO:tasks.rados.rados.0.smithi045.stdout:1624: delete oid 18 current snap is 152 2021-07-31T01:51:00.578 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 13 v 597 (ObjNum 425 snap 126 seq_num 425) dirty exists 2021-07-31T01:51:00.579 INFO:tasks.rados.rados.0.smithi045.stdout:1610: finishing write tid 2 to smithi04569269-46 2021-07-31T01:51:00.579 INFO:tasks.rados.rados.0.smithi045.stdout:1610: finishing write tid 3 to smithi04569269-46 2021-07-31T01:51:00.579 INFO:tasks.rados.rados.0.smithi045.stdout:1610: finishing write tid 4 to smithi04569269-46 2021-07-31T01:51:00.579 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 17 v 991 (ObjNum 0 snap 32731 seq_num 3189720944) dirty exists 2021-07-31T01:51:00.579 INFO:tasks.rados.rados.0.smithi045.stdout:1616: finishing rollback tid 0 to smithi04569269-10 2021-07-31T01:51:00.580 INFO:tasks.rados.rados.0.smithi045.stdout:1616: finishing rollback tid 1 to smithi04569269-10 2021-07-31T01:51:00.580 INFO:tasks.rados.rados.0.smithi045.stdout:1616: finishing rollback tid 2 to smithi04569269-10 2021-07-31T01:51:00.580 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 10 v 994 (ObjNum 475 snap 140 seq_num 475) dirty exists 2021-07-31T01:51:00.581 INFO:tasks.rados.rados.0.smithi045.stdout:1610: finishing write tid 5 to smithi04569269-46 2021-07-31T01:51:00.581 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 46 v 271 (ObjNum 510 snap 152 seq_num 510) dirty exists 2021-07-31T01:51:00.581 INFO:tasks.rados.rados.0.smithi045.stdout:1610: left oid 46 (ObjNum 510 snap 152 seq_num 510) 2021-07-31T01:51:00.581 INFO:tasks.rados.rados.0.smithi045.stdout:1613: finishing copy_from to smithi04569269-43 2021-07-31T01:51:00.582 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 43 v 1001 (ObjNum 483 snap 143 seq_num 483) dirty exists 2021-07-31T01:51:00.582 INFO:tasks.rados.rados.0.smithi045.stdout:1613: finishing copy_from racing read to smithi04569269-43 2021-07-31T01:51:00.582 INFO:tasks.rados.rados.0.smithi045.stdout:1614: expect (ObjNum 425 snap 126 seq_num 425) 2021-07-31T01:51:01.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:00 smithi154 conmon[34553]: cluster 2021-07-31T01:50:59.560000+0000 mon.a (mon.0) 1006 : cluster [DBG] osdmap e360: 8 total, 8 up, 8 in 2021-07-31T01:51:01.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:00 smithi154 conmon[34553]: 2021-07-31T01:51:01.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:00 smithi154 conmon[34553]: cluster 2021-07-31T01:50:59.844303+0000 mgr.x (mgr.24542) 159 : cluster [DBG] pgmap v320: 25 pgs: 25 active+clean; 147 MiB data, 980 MiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 4.1 MiB/s wr, 32 op/s 2021-07-31T01:51:01.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:00 smithi045 conmon[33655]: cluster 2021-07-31T01:50:59.560000+0000 mon.a (mon.0) 1006 : 2021-07-31T01:51:01.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:00 smithi045 conmon[33655]: cluster [DBG] osdmap e360: 8 total, 8 up, 8 in 2021-07-31T01:51:01.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:00 smithi045 conmon[33655]: cluster 2021-07-31T01:50:59.844303+0000 mgr.x (mgr.24542) 159 : cluster [DBG] pgmap v320: 25 pgs: 25 active+clean; 147 MiB data, 980 MiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 4.1 MiB/s wr, 32 op/s 2021-07-31T01:51:01.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:00 smithi045 conmon[39718]: cluster 2021-07-31T01:50:59.560000+0000 mon.a 2021-07-31T01:51:01.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:00 smithi045 conmon[39718]: (mon.0) 1006 : cluster [DBG] osdmap e360: 8 total, 8 up, 8 in 2021-07-31T01:51:01.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:00 smithi045 conmon[39718]: cluster 2021-07-31T01:50:59.844303+0000 2021-07-31T01:51:01.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:00 smithi045 conmon[39718]: mgr.x (mgr.24542) 159 : cluster [DBG] pgmap v320: 25 pgs: 25 active+clean; 147 MiB data, 980 MiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 4.1 MiB/s wr, 32 op/s 2021-07-31T01:51:01.121 INFO:tasks.rados.rados.0.smithi045.stdout:1618: finishing copy_from to smithi04569269-27 2021-07-31T01:51:01.121 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 27 v 279 (ObjNum 507 snap 152 seq_num 507) dirty exists 2021-07-31T01:51:01.121 INFO:tasks.rados.rados.0.smithi045.stdout:1618: finishing copy_from racing read to smithi04569269-27 2021-07-31T01:51:01.121 INFO:tasks.rados.rados.0.smithi045.stdout:1610: done (11 left) 2021-07-31T01:51:01.122 INFO:tasks.rados.rados.0.smithi045.stdout:1611: done (10 left) 2021-07-31T01:51:01.122 INFO:tasks.rados.rados.0.smithi045.stdout:1612: done (9 left) 2021-07-31T01:51:01.122 INFO:tasks.rados.rados.0.smithi045.stdout:1613: done (8 left) 2021-07-31T01:51:01.122 INFO:tasks.rados.rados.0.smithi045.stdout:1614: done (7 left) 2021-07-31T01:51:01.123 INFO:tasks.rados.rados.0.smithi045.stdout:1616: done (6 left) 2021-07-31T01:51:01.123 INFO:tasks.rados.rados.0.smithi045.stdout:1618: done (5 left) 2021-07-31T01:51:01.123 INFO:tasks.rados.rados.0.smithi045.stdout:1624: done (4 left) 2021-07-31T01:51:01.123 INFO:tasks.rados.rados.0.smithi045.stdout:1625: read oid 10 snap -1 2021-07-31T01:51:01.124 INFO:tasks.rados.rados.0.smithi045.stdout:1625: expect (ObjNum 475 snap 140 seq_num 475) 2021-07-31T01:51:01.124 INFO:tasks.rados.rados.0.smithi045.stdout:1626: read oid 39 snap -1 2021-07-31T01:51:01.124 INFO:tasks.rados.rados.0.smithi045.stdout:1626: expect (ObjNum 483 snap 143 seq_num 483) 2021-07-31T01:51:01.124 INFO:tasks.rados.rados.0.smithi045.stdout:1627: copy_from oid 11 from oid 12 current snap is 152 2021-07-31T01:51:01.124 INFO:tasks.rados.rados.0.smithi045.stdout:1623: expect (ObjNum 498 snap 148 seq_num 498) 2021-07-31T01:51:01.150 INFO:tasks.rados.rados.0.smithi045.stdout:1621: finishing copy_from to smithi04569269-48 2021-07-31T01:51:01.151 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 48 v 386 (ObjNum 443 snap 129 seq_num 443) dirty exists 2021-07-31T01:51:01.151 INFO:tasks.rados.rados.0.smithi045.stdout:1621: finishing copy_from racing read to smithi04569269-48 2021-07-31T01:51:01.151 INFO:tasks.rados.rados.0.smithi045.stdout:1622: finishing copy_from to smithi04569269-29 2021-07-31T01:51:01.151 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 29 v 381 (ObjNum 486 snap 143 seq_num 486) dirty exists 2021-07-31T01:51:01.152 INFO:tasks.rados.rados.0.smithi045.stdout:1622: finishing copy_from racing read to smithi04569269-29 2021-07-31T01:51:01.152 INFO:tasks.rados.rados.0.smithi045.stdout:1620: done (6 left) 2021-07-31T01:51:01.152 INFO:tasks.rados.rados.0.smithi045.stdout:1621: done (5 left) 2021-07-31T01:51:01.152 INFO:tasks.rados.rados.0.smithi045.stdout:1622: done (4 left) 2021-07-31T01:51:01.152 INFO:tasks.rados.rados.0.smithi045.stdout:1623: done (3 left) 2021-07-31T01:51:01.153 INFO:tasks.rados.rados.0.smithi045.stdout:1628: snap_remove snap 149 2021-07-31T01:51:01.547 INFO:tasks.rados.rados.0.smithi045.stdout:1628: done (3 left) 2021-07-31T01:51:01.547 INFO:tasks.rados.rados.0.smithi045.stdout:1629: read oid 5 snap -1 2021-07-31T01:51:01.547 INFO:tasks.rados.rados.0.smithi045.stdout:1629: expect deleted 2021-07-31T01:51:01.548 INFO:tasks.rados.rados.0.smithi045.stdout:1630: delete oid 41 current snap is 152 2021-07-31T01:51:01.548 INFO:tasks.rados.rados.0.smithi045.stdout:1625: expect (ObjNum 475 snap 140 seq_num 475) 2021-07-31T01:51:01.586 INFO:tasks.rados.rados.0.smithi045.stdout:1626: expect (ObjNum 483 snap 143 seq_num 483) 2021-07-31T01:51:01.637 INFO:tasks.rados.rados.0.smithi045.stdout:1627: finishing copy_from to smithi04569269-11 2021-07-31T01:51:01.637 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 11 v 596 (ObjNum 506 snap 152 seq_num 506) dirty exists 2021-07-31T01:51:01.638 INFO:tasks.rados.rados.0.smithi045.stdout:1627: finishing copy_from racing read to smithi04569269-11 2021-07-31T01:51:01.638 INFO:tasks.rados.rados.0.smithi045.stdout:1625: done (4 left) 2021-07-31T01:51:01.638 INFO:tasks.rados.rados.0.smithi045.stdout:1626: done (3 left) 2021-07-31T01:51:01.638 INFO:tasks.rados.rados.0.smithi045.stdout:1627: done (2 left) 2021-07-31T01:51:01.639 INFO:tasks.rados.rados.0.smithi045.stdout:1629: done (1 left) 2021-07-31T01:51:01.639 INFO:tasks.rados.rados.0.smithi045.stdout:1630: done (0 left) 2021-07-31T01:51:01.639 INFO:tasks.rados.rados.0.smithi045.stdout:1631: read oid 32 snap -1 2021-07-31T01:51:01.639 INFO:tasks.rados.rados.0.smithi045.stdout:1631: expect deleted 2021-07-31T01:51:01.639 INFO:tasks.rados.rados.0.smithi045.stdout:1632: snap_remove snap 148 2021-07-31T01:51:01.665 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T01:51:01.942 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T01:51:01.943 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 5m ago 6m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T01:51:01.943 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 3m ago 6m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T01:51:01.943 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 3m ago 9m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T01:51:01.944 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 5m ago 12m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T01:51:01.944 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 5m ago 12m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T01:51:01.944 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 3m ago 10m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T01:51:01.945 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 5m ago 10m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T01:51:01.945 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 5m ago 6m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T01:51:01.946 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 3m ago 6m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T01:51:01.946 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 5m ago 9m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T01:51:01.947 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 5m ago 9m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T01:51:01.947 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 5m ago 8m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T01:51:01.947 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 5m ago 8m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T01:51:01.947 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 3m ago 8m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T01:51:01.948 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 3m ago 8m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T01:51:01.948 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 3m ago 7m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T01:51:01.948 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 3m ago 7m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T01:51:01.948 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 3m ago 7m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T01:51:02.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:01 smithi154 conmon[34553]: cluster 2021-07-31T01:51:00.571388+0000 mon.a (mon.0) 1007 : cluster [DBG] 2021-07-31T01:51:02.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:01 smithi154 conmon[34553]: osdmap e361: 8 total, 8 up, 8 in 2021-07-31T01:51:02.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:01 smithi154 conmon[34553]: cluster 2021-07-31T01:51:01.546450+0000 mon.a (mon.0) 1008 : cluster [DBG] osdmap e362: 8 total, 8 up, 8 in 2021-07-31T01:51:02.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:01 smithi045 conmon[33655]: cluster 2021-07-31T01:51:00.571388+0000 mon.a (mon 2021-07-31T01:51:02.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:01 smithi045 conmon[33655]: .0) 1007 : cluster [DBG] osdmap e361: 8 total, 8 up, 8 in 2021-07-31T01:51:02.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:01 smithi045 conmon[33655]: cluster 2021-07-31T01:51:01.546450+0000 mon.a (mon.0) 1008 : cluster [DBG] osdmap e362: 8 total, 8 up, 8 in 2021-07-31T01:51:02.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:01 smithi045 conmon[39718]: cluster 2021-07-31T01:51:00.571388+0000 2021-07-31T01:51:02.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:01 smithi045 conmon[39718]: mon.a (mon.0) 1007 : cluster [DBG] osdmap e361: 8 total, 8 up, 8 in 2021-07-31T01:51:02.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:01 smithi045 conmon[39718]: cluster 2021-07 2021-07-31T01:51:02.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:01 smithi045 conmon[39718]: -31T01:51:01.546450+0000 mon.a (mon.0) 1008 : cluster [DBG] 2021-07-31T01:51:02.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:01 smithi045 conmon[39718]: osdmap e362: 8 total, 8 up, 8 in 2021-07-31T01:51:02.282 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T01:51:02.283 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T01:51:02.283 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T01:51:02.283 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T01:51:02.284 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T01:51:02.284 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T01:51:02.284 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T01:51:02.284 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T01:51:02.284 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T01:51:02.285 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T01:51:02.285 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T01:51:02.285 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T01:51:02.285 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T01:51:02.286 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T01:51:02.286 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T01:51:02.286 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T01:51:02.286 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T01:51:02.578 INFO:tasks.rados.rados.0.smithi045.stdout:1632: done (1 left) 2021-07-31T01:51:02.579 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 6 current snap is 152 2021-07-31T01:51:02.579 INFO:tasks.rados.rados.0.smithi045.stdout:1633: seq_num 514 ranges {1146880=614400,1761280=679936,2441216=122880} 2021-07-31T01:51:02.589 INFO:tasks.rados.rados.0.smithi045.stdout:1633: writing smithi04569269-6 from 1146880 to 1761280 tid 1 2021-07-31T01:51:02.593 INFO:tasks.rados.rados.0.smithi045.stdout:1633: writing smithi04569269-6 from 1761280 to 2441216 tid 2 2021-07-31T01:51:02.594 INFO:tasks.rados.rados.0.smithi045.stdout:1633: writing smithi04569269-6 from 2441216 to 2564096 tid 3 2021-07-31T01:51:02.595 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 8 current snap is 152 2021-07-31T01:51:02.595 INFO:tasks.rados.rados.0.smithi045.stdout:1634: seq_num 515 ranges {0=630784,630784=565248,1196032=237568} 2021-07-31T01:51:02.598 INFO:tasks.rados.rados.0.smithi045.stdout:1634: writing smithi04569269-8 from 0 to 630784 tid 1 2021-07-31T01:51:02.601 INFO:tasks.rados.rados.0.smithi045.stdout:1634: writing smithi04569269-8 from 630784 to 1196032 tid 2 2021-07-31T01:51:02.603 INFO:tasks.rados.rados.0.smithi045.stdout:1634: writing smithi04569269-8 from 1196032 to 1433600 tid 3 2021-07-31T01:51:02.603 INFO:tasks.rados.rados.0.smithi045.stdout:1631: done (2 left) 2021-07-31T01:51:02.603 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 39 current snap is 152 2021-07-31T01:51:02.604 INFO:tasks.rados.rados.0.smithi045.stdout:1633: finishing write tid 1 to smithi04569269-6 2021-07-31T01:51:02.604 INFO:tasks.rados.rados.0.smithi045.stdout:1635: seq_num 516 ranges {1441792=647168,2088960=450560,2539520=352256} 2021-07-31T01:51:02.614 INFO:tasks.rados.rados.0.smithi045.stdout:1635: writing smithi04569269-39 from 1441792 to 2088960 tid 1 2021-07-31T01:51:02.617 INFO:tasks.rados.rados.0.smithi045.stdout:1635: writing smithi04569269-39 from 2088960 to 2539520 tid 2 2021-07-31T01:51:02.619 INFO:tasks.rados.rados.0.smithi045.stdout:1635: writing smithi04569269-39 from 2539520 to 2891776 tid 3 2021-07-31T01:51:02.620 INFO:tasks.rados.rados.0.smithi045.stdout:1636: snap_create 2021-07-31T01:51:02.620 INFO:tasks.rados.rados.0.smithi045.stdout:1633: finishing write tid 2 to smithi04569269-6 2021-07-31T01:51:02.620 INFO:tasks.rados.rados.0.smithi045.stdout:1633: finishing write tid 3 to smithi04569269-6 2021-07-31T01:51:02.620 INFO:tasks.rados.rados.0.smithi045.stdout:1633: finishing write tid 4 to smithi04569269-6 2021-07-31T01:51:02.621 INFO:tasks.rados.rados.0.smithi045.stdout:1633: finishing write tid 5 to smithi04569269-6 2021-07-31T01:51:02.621 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 6 v 606 (ObjNum 514 snap 152 seq_num 514) dirty exists 2021-07-31T01:51:02.621 INFO:tasks.rados.rados.0.smithi045.stdout:1633: left oid 6 (ObjNum 514 snap 152 seq_num 514) 2021-07-31T01:51:02.621 INFO:tasks.rados.rados.0.smithi045.stdout:1634: finishing write tid 1 to smithi04569269-8 2021-07-31T01:51:02.621 INFO:tasks.rados.rados.0.smithi045.stdout:1634: finishing write tid 2 to smithi04569269-8 2021-07-31T01:51:02.622 INFO:tasks.rados.rados.0.smithi045.stdout:1634: finishing write tid 3 to smithi04569269-8 2021-07-31T01:51:02.622 INFO:tasks.rados.rados.0.smithi045.stdout:1634: finishing write tid 4 to smithi04569269-8 2021-07-31T01:51:02.622 INFO:tasks.rados.rados.0.smithi045.stdout:1634: finishing write tid 5 to smithi04569269-8 2021-07-31T01:51:02.622 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 8 v 1049 (ObjNum 515 snap 152 seq_num 515) dirty exists 2021-07-31T01:51:02.623 INFO:tasks.rados.rados.0.smithi045.stdout:1634: left oid 8 (ObjNum 515 snap 152 seq_num 515) 2021-07-31T01:51:02.623 INFO:tasks.rados.rados.0.smithi045.stdout:1635: finishing write tid 1 to smithi04569269-39 2021-07-31T01:51:02.624 INFO:tasks.rados.rados.0.smithi045.stdout:1635: finishing write tid 2 to smithi04569269-39 2021-07-31T01:51:02.624 INFO:tasks.rados.rados.0.smithi045.stdout:1635: finishing write tid 3 to smithi04569269-39 2021-07-31T01:51:02.624 INFO:tasks.rados.rados.0.smithi045.stdout:1635: finishing write tid 4 to smithi04569269-39 2021-07-31T01:51:02.624 INFO:tasks.rados.rados.0.smithi045.stdout:1635: finishing write tid 5 to smithi04569269-39 2021-07-31T01:51:02.625 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 39 v 1007 (ObjNum 516 snap 152 seq_num 516) dirty exists 2021-07-31T01:51:02.625 INFO:tasks.rados.rados.0.smithi045.stdout:1635: left oid 39 (ObjNum 516 snap 152 seq_num 516) 2021-07-31T01:51:03.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:02 smithi154 conmon[34553]: audit 2021-07-31T01:51:01.652891+0000 mgr.x (mgr.24542) 2021-07-31T01:51:03.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:02 smithi154 conmon[34553]: 160 : audit [DBG] from='client.14838 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:51:03.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:02 smithi154 conmon[34553]: cluster 2021-07-31T01:51:01.844860+0000 mgr.x (mgr.24542) 161 : cluster [DBG] pgmap v323: 25 pgs: 25 active+clean; 147 MiB data, 980 MiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 4.1 MiB/s wr, 32 op/s 2021-07-31T01:51:03.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:02 smithi154 conmon[34553]: audit 2021-07-31T 2021-07-31T01:51:03.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:02 smithi154 conmon[34553]: 01:51:01.934602+0000 mgr.x (mgr.24542) 162 : audit [DBG] from='client.24751 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:51:03.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:02 smithi154 conmon[34553]: audit 2021-07-31T01:51:02.276869 2021-07-31T01:51:03.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:02 smithi154 conmon[34553]: +0000 mon.c (mon.1) 31 : audit 2021-07-31T01:51:03.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:02 smithi154 conmon[34553]: [DBG] from='client.? 172.21.15.45:0/2606830764' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T01:51:03.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:02 smithi045 conmon[39718]: audit 2021-07-31T01:51:01.652891 2021-07-31T01:51:03.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:02 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 160 : audit [DBG] from='client.14838 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:51:03.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:02 smithi045 conmon[39718]: cluster 2021 2021-07-31T01:51:03.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:02 smithi045 conmon[39718]: -07-31T01:51:01.844860+0000 mgr.x (mgr.24542) 161 : cluster [DBG] 2021-07-31T01:51:03.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:02 smithi045 conmon[39718]: pgmap v323: 25 pgs: 25 active+clean; 147 MiB data, 980 MiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 4.1 MiB/s wr, 32 op/s 2021-07-31T01:51:03.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:02 smithi045 conmon[39718]: audit 2021-07-31T01:51:01 2021-07-31T01:51:03.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:02 smithi045 conmon[39718]: .934602+0000 mgr.x (mgr.24542) 162 : audit 2021-07-31T01:51:03.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:02 smithi045 conmon[39718]: [DBG] from='client.24751 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:51:03.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:02 smithi045 conmon[39718]: audit 2021-07-31T01:51:02 2021-07-31T01:51:03.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:02 smithi045 conmon[39718]: .276869+0000 mon.c (mon.1) 31 : audit 2021-07-31T01:51:03.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:02 smithi045 conmon[39718]: [DBG] from='client.? 172.21.15.45:0/2606830764' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T01:51:03.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:02 smithi045 conmon[33655]: audit 2021-07-31T01:51:01. 2021-07-31T01:51:03.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:02 smithi045 conmon[33655]: 652891+0000 mgr.x (mgr.24542) 160 : audit [DBG] from='client.14838 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:51:03.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:02 smithi045 conmon[33655]: cluster 2021-07-31T01: 2021-07-31T01:51:03.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:02 smithi045 conmon[33655]: 51:01.844860+0000 mgr.x (mgr.24542) 161 : cluster [DBG] pgmap v323: 25 pgs: 25 active+clean; 147 MiB data, 980 MiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 4.1 MiB/s wr, 32 op/s 2021-07-31T01:51:03.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:02 smithi045 conmon[33655]: 2021-07-31T01:51:03.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:02 smithi045 conmon[33655]: audit 2021-07-31T01:51:03.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:02 smithi045 conmon[33655]: 2021-07-31T01:51:01.934602+0000 2021-07-31T01:51:03.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:02 smithi045 conmon[33655]: mgr.x (mgr.24542) 162 : audit 2021-07-31T01:51:03.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:02 smithi045 conmon[33655]: [DBG] from='client.24751 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:51:03.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:02 smithi045 conmon[33655]: audit 2021-07-31T01 2021-07-31T01:51:03.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:02 smithi045 conmon[33655]: :51:02.276869+0000 mon.c 2021-07-31T01:51:03.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:02 smithi045 conmon[33655]: (mon.1) 31 : audit 2021-07-31T01:51:03.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:02 smithi045 conmon[33655]: [DBG] from='client.? 172.21.15.45:0/2606830764' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T01:51:03.585 INFO:tasks.rados.rados.0.smithi045.stdout:1633: done (3 left) 2021-07-31T01:51:03.586 INFO:tasks.rados.rados.0.smithi045.stdout:1634: done (2 left) 2021-07-31T01:51:03.586 INFO:tasks.rados.rados.0.smithi045.stdout:1635: done (1 left) 2021-07-31T01:51:03.586 INFO:tasks.rados.rados.0.smithi045.stdout:1636: done (0 left) 2021-07-31T01:51:03.586 INFO:tasks.rados.rados.0.smithi045.stdout:1637: rmattr oid 13 current snap is 153 2021-07-31T01:51:03.587 INFO:tasks.rados.rados.0.smithi045.stdout:1638: read oid 26 snap 146 2021-07-31T01:51:03.587 INFO:tasks.rados.rados.0.smithi045.stdout:1638: expect (ObjNum 443 snap 129 seq_num 443) 2021-07-31T01:51:03.587 INFO:tasks.rados.rados.0.smithi045.stdout:1639: read oid 4 snap 146 2021-07-31T01:51:03.587 INFO:tasks.rados.rados.0.smithi045.stdout:1639: expect (ObjNum 470 snap 136 seq_num 470) 2021-07-31T01:51:03.587 INFO:tasks.rados.rados.0.smithi045.stdout:1640: rmattr oid 17 current snap is 153 2021-07-31T01:51:03.588 INFO:tasks.rados.rados.0.smithi045.stdout:1641: rollback oid 44 current snap is 153 2021-07-31T01:51:03.588 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 44 to 152 2021-07-31T01:51:03.588 INFO:tasks.rados.rados.0.smithi045.stdout:1642: setattr oid 21 current snap is 153 2021-07-31T01:51:03.589 INFO:tasks.rados.rados.0.smithi045.stdout:1643: read oid 39 snap -1 2021-07-31T01:51:03.589 INFO:tasks.rados.rados.0.smithi045.stdout:1643: expect (ObjNum 516 snap 152 seq_num 516) 2021-07-31T01:51:03.589 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 20 current snap is 153 2021-07-31T01:51:03.590 INFO:tasks.rados.rados.0.smithi045.stdout:1644: seq_num 517 ranges {0=663552,663552=786432,1449984=16384} 2021-07-31T01:51:03.592 INFO:tasks.rados.rados.0.smithi045.stdout:1644: writing smithi04569269-20 from 0 to 663552 tid 1 2021-07-31T01:51:03.598 INFO:tasks.rados.rados.0.smithi045.stdout:1644: writing smithi04569269-20 from 663552 to 1449984 tid 2 2021-07-31T01:51:03.599 INFO:tasks.rados.rados.0.smithi045.stdout:1644: writing smithi04569269-20 from 1449984 to 1466368 tid 3 2021-07-31T01:51:03.599 INFO:tasks.rados.rados.0.smithi045.stdout:1645: copy_from oid 36 from oid 10 current snap is 153 2021-07-31T01:51:03.599 INFO:tasks.rados.rados.0.smithi045.stdout:1638: expect (ObjNum 443 snap 129 seq_num 443) 2021-07-31T01:51:03.609 INFO:tasks.rados.rados.0.smithi045.stdout:1641: finishing rollback tid 0 to smithi04569269-44 2021-07-31T01:51:03.610 INFO:tasks.rados.rados.0.smithi045.stdout:1641: finishing rollback tid 1 to smithi04569269-44 2021-07-31T01:51:03.610 INFO:tasks.rados.rados.0.smithi045.stdout:1641: finishing rollback tid 2 to smithi04569269-44 2021-07-31T01:51:03.610 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 44 v 1004 (ObjNum 499 snap 149 seq_num 499) dirty exists 2021-07-31T01:51:03.610 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 17 v 995 (ObjNum 1953066355 snap 875587944 seq_num 842610229) dirty exists 2021-07-31T01:51:03.611 INFO:tasks.rados.rados.0.smithi045.stdout:1638: done (8 left) 2021-07-31T01:51:03.611 INFO:tasks.rados.rados.0.smithi045.stdout:1640: done (7 left) 2021-07-31T01:51:03.611 INFO:tasks.rados.rados.0.smithi045.stdout:1641: done (6 left) 2021-07-31T01:51:03.611 INFO:tasks.rados.rados.0.smithi045.stdout:1646: rollback oid 16 current snap is 153 2021-07-31T01:51:03.612 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 16 to 146 2021-07-31T01:51:03.612 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 13 v 608 (ObjNum 425 snap 126 seq_num 425) dirty exists 2021-07-31T01:51:03.612 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 21 v 1050 (ObjNum -1105697088 snap 21873 seq_num 3189853024) dirty exists 2021-07-31T01:51:03.612 INFO:tasks.rados.rados.0.smithi045.stdout:1644: finishing write tid 1 to smithi04569269-20 2021-07-31T01:51:03.612 INFO:tasks.rados.rados.0.smithi045.stdout:1644: finishing write tid 2 to smithi04569269-20 2021-07-31T01:51:03.613 INFO:tasks.rados.rados.0.smithi045.stdout:1639: expect (ObjNum 470 snap 136 seq_num 470) 2021-07-31T01:51:03.741 INFO:tasks.rados.rados.0.smithi045.stdout:1644: finishing write tid 3 to smithi04569269-20 2021-07-31T01:51:03.741 INFO:tasks.rados.rados.0.smithi045.stdout:1644: finishing write tid 4 to smithi04569269-20 2021-07-31T01:51:03.741 INFO:tasks.rados.rados.0.smithi045.stdout:1644: finishing write tid 5 to smithi04569269-20 2021-07-31T01:51:03.742 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 20 v 1054 (ObjNum 517 snap 153 seq_num 517) dirty exists 2021-07-31T01:51:03.743 INFO:tasks.rados.rados.0.smithi045.stdout:1644: left oid 20 (ObjNum 517 snap 153 seq_num 517) 2021-07-31T01:51:03.743 INFO:tasks.rados.rados.0.smithi045.stdout:1643: expect (ObjNum 516 snap 152 seq_num 516) 2021-07-31T01:51:03.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:03 smithi045 conmon[39718]: cluster 2021-07-31T01:51:02.578044+0000 mon.a 2021-07-31T01:51:03.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:03 smithi045 conmon[39718]: (mon.0) 1009 : cluster [DBG] osdmap e363: 8 total, 8 up, 8 in 2021-07-31T01:51:03.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:51:03 smithi045 conmon[61210]: level=error ts=2021-07-31T01:51:03.515Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:51:03.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:51:03 smithi045 conmon[61210]: level=error ts=2021-07-31T01:51:03.515Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:51:03.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:51:03 smithi045 conmon[61210]: level=error ts=2021-07-31T01:51:03.515Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T01:51:03.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:03 smithi045 conmon[33655]: cluster 2021-07-31T01:51:02 2021-07-31T01:51:03.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:03 smithi045 conmon[33655]: .578044+0000 mon.a (mon.0) 1009 : cluster [DBG] osdmap e363: 8 total, 8 up, 8 in 2021-07-31T01:51:03.852 INFO:tasks.rados.rados.0.smithi045.stdout:1646: finishing rollback tid 0 to smithi04569269-16 2021-07-31T01:51:03.852 INFO:tasks.rados.rados.0.smithi045.stdout:1645: finishing copy_from to smithi04569269-36 2021-07-31T01:51:03.852 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 36 v 994 (ObjNum 475 snap 140 seq_num 475) dirty exists 2021-07-31T01:51:03.852 INFO:tasks.rados.rados.0.smithi045.stdout:1646: finishing rollback tid 1 to smithi04569269-16 2021-07-31T01:51:03.853 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 16 v 514 (ObjNum 0 snap 32731 seq_num 3189783872) dirty dne 2021-07-31T01:51:03.853 INFO:tasks.rados.rados.0.smithi045.stdout:1645: finishing copy_from racing read to smithi04569269-36 2021-07-31T01:51:03.853 INFO:tasks.rados.rados.0.smithi045.stdout:1637: done (6 left) 2021-07-31T01:51:03.853 INFO:tasks.rados.rados.0.smithi045.stdout:1639: done (5 left) 2021-07-31T01:51:03.854 INFO:tasks.rados.rados.0.smithi045.stdout:1642: done (4 left) 2021-07-31T01:51:03.854 INFO:tasks.rados.rados.0.smithi045.stdout:1643: done (3 left) 2021-07-31T01:51:03.854 INFO:tasks.rados.rados.0.smithi045.stdout:1644: done (2 left) 2021-07-31T01:51:03.854 INFO:tasks.rados.rados.0.smithi045.stdout:1645: done (1 left) 2021-07-31T01:51:03.854 INFO:tasks.rados.rados.0.smithi045.stdout:1646: done (0 left) 2021-07-31T01:51:03.855 INFO:tasks.rados.rados.0.smithi045.stdout:1647: copy_from oid 43 from oid 3 current snap is 153 2021-07-31T01:51:03.855 INFO:tasks.rados.rados.0.smithi045.stdout:1648: read oid 47 snap 146 2021-07-31T01:51:03.855 INFO:tasks.rados.rados.0.smithi045.stdout:1648: expect (ObjNum 172 snap 41 seq_num 172) 2021-07-31T01:51:03.855 INFO:tasks.rados.rados.0.smithi045.stdout:1649: setattr oid 8 current snap is 153 2021-07-31T01:51:03.856 INFO:tasks.rados.rados.0.smithi045.stdout:1650: delete oid 41 current snap is 153 2021-07-31T01:51:03.856 INFO:tasks.rados.rados.0.smithi045.stdout:1648: expect (ObjNum 172 snap 41 seq_num 172) 2021-07-31T01:51:03.874 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 8 v 1058 (ObjNum 515 snap 152 seq_num 515) dirty exists 2021-07-31T01:51:03.874 INFO:tasks.rados.rados.0.smithi045.stdout:1647: finishing copy_from to smithi04569269-43 2021-07-31T01:51:03.874 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 43 v 1001 (ObjNum 483 snap 143 seq_num 483) dirty exists 2021-07-31T01:51:03.875 INFO:tasks.rados.rados.0.smithi045.stdout:1647: finishing copy_from racing read to smithi04569269-43 2021-07-31T01:51:03.875 INFO:tasks.rados.rados.0.smithi045.stdout:1647: done (3 left) 2021-07-31T01:51:03.875 INFO:tasks.rados.rados.0.smithi045.stdout:1648: done (2 left) 2021-07-31T01:51:03.875 INFO:tasks.rados.rados.0.smithi045.stdout:1649: done (1 left) 2021-07-31T01:51:03.876 INFO:tasks.rados.rados.0.smithi045.stdout:1650: done (0 left) 2021-07-31T01:51:03.876 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 19 current snap is 153 2021-07-31T01:51:03.876 INFO:tasks.rados.rados.0.smithi045.stdout:1651: seq_num 519 ranges {0=696320,696320=548864,1245184=253952} 2021-07-31T01:51:03.877 INFO:tasks.rados.rados.0.smithi045.stdout:1651: writing smithi04569269-19 from 0 to 696320 tid 1 2021-07-31T01:51:03.880 INFO:tasks.rados.rados.0.smithi045.stdout:1651: writing smithi04569269-19 from 696320 to 1245184 tid 2 2021-07-31T01:51:03.882 INFO:tasks.rados.rados.0.smithi045.stdout:1651: writing smithi04569269-19 from 1245184 to 1499136 tid 3 2021-07-31T01:51:03.883 INFO:tasks.rados.rados.0.smithi045.stdout:1652: rollback oid 40 current snap is 153 2021-07-31T01:51:03.883 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 40 to 146 2021-07-31T01:51:03.883 INFO:tasks.rados.rados.0.smithi045.stdout:1651: finishing write tid 1 to smithi04569269-19 2021-07-31T01:51:03.883 INFO:tasks.rados.rados.0.smithi045.stdout:1653: rmattr oid 1 current snap is 153 2021-07-31T01:51:03.884 INFO:tasks.rados.rados.0.smithi045.stdout:1654: delete oid 35 current snap is 153 2021-07-31T01:51:03.886 INFO:tasks.rados.rados.0.smithi045.stdout:1651: finishing write tid 2 to smithi04569269-19 2021-07-31T01:51:03.886 INFO:tasks.rados.rados.0.smithi045.stdout:1654: done (3 left) 2021-07-31T01:51:03.886 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 14 current snap is 153 2021-07-31T01:51:03.887 INFO:tasks.rados.rados.0.smithi045.stdout:1655: seq_num 521 ranges {0=729088,729088=720896,1449984=81920} 2021-07-31T01:51:03.890 INFO:tasks.rados.rados.0.smithi045.stdout:1655: writing smithi04569269-14 from 0 to 729088 tid 1 2021-07-31T01:51:03.894 INFO:tasks.rados.rados.0.smithi045.stdout:1655: writing smithi04569269-14 from 729088 to 1449984 tid 2 2021-07-31T01:51:03.896 INFO:tasks.rados.rados.0.smithi045.stdout:1655: writing smithi04569269-14 from 1449984 to 1531904 tid 3 2021-07-31T01:51:03.896 INFO:tasks.rados.rados.0.smithi045.stdout:1656: copy_from oid 44 from oid 31 current snap is 153 2021-07-31T01:51:03.896 INFO:tasks.rados.rados.0.smithi045.stdout:1652: finishing rollback tid 0 to smithi04569269-40 2021-07-31T01:51:03.896 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 1 v 907 (ObjNum 465 snap 135 seq_num 465) dirty exists 2021-07-31T01:51:03.897 INFO:tasks.rados.rados.0.smithi045.stdout:1651: finishing write tid 3 to smithi04569269-19 2021-07-31T01:51:03.897 INFO:tasks.rados.rados.0.smithi045.stdout:1652: finishing rollback tid 1 to smithi04569269-40 2021-07-31T01:51:03.897 INFO:tasks.rados.rados.0.smithi045.stdout:1652: finishing rollback tid 2 to smithi04569269-40 2021-07-31T01:51:03.897 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 40 v 612 (ObjNum 461 snap 134 seq_num 461) dirty exists 2021-07-31T01:51:03.898 INFO:tasks.rados.rados.0.smithi045.stdout:1651: finishing write tid 4 to smithi04569269-19 2021-07-31T01:51:03.898 INFO:tasks.rados.rados.0.smithi045.stdout:1651: finishing write tid 5 to smithi04569269-19 2021-07-31T01:51:03.898 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 19 v 518 (ObjNum 519 snap 153 seq_num 519) dirty exists 2021-07-31T01:51:03.898 INFO:tasks.rados.rados.0.smithi045.stdout:1651: left oid 19 (ObjNum 519 snap 153 seq_num 519) 2021-07-31T01:51:03.898 INFO:tasks.rados.rados.0.smithi045.stdout:1655: finishing write tid 1 to smithi04569269-14 2021-07-31T01:51:03.899 INFO:tasks.rados.rados.0.smithi045.stdout:1651: done (4 left) 2021-07-31T01:51:03.899 INFO:tasks.rados.rados.0.smithi045.stdout:1652: done (3 left) 2021-07-31T01:51:03.899 INFO:tasks.rados.rados.0.smithi045.stdout:1653: done (2 left) 2021-07-31T01:51:03.899 INFO:tasks.rados.rados.0.smithi045.stdout:1657: snap_create 2021-07-31T01:51:03.900 INFO:tasks.rados.rados.0.smithi045.stdout:1655: finishing write tid 2 to smithi04569269-14 2021-07-31T01:51:03.900 INFO:tasks.rados.rados.0.smithi045.stdout:1655: finishing write tid 3 to smithi04569269-14 2021-07-31T01:51:03.901 INFO:tasks.rados.rados.0.smithi045.stdout:1655: finishing write tid 4 to smithi04569269-14 2021-07-31T01:51:03.901 INFO:tasks.rados.rados.0.smithi045.stdout:1655: finishing write tid 5 to smithi04569269-14 2021-07-31T01:51:03.901 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 14 v 1008 (ObjNum 521 snap 153 seq_num 521) dirty exists 2021-07-31T01:51:03.901 INFO:tasks.rados.rados.0.smithi045.stdout:1655: left oid 14 (ObjNum 521 snap 153 seq_num 521) 2021-07-31T01:51:03.924 INFO:tasks.rados.rados.0.smithi045.stdout:1656: finishing copy_from to smithi04569269-44 2021-07-31T01:51:03.925 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 44 v 1017 (ObjNum 490 snap 146 seq_num 490) dirty exists 2021-07-31T01:51:03.925 INFO:tasks.rados.rados.0.smithi045.stdout:1656: finishing copy_from racing read to smithi04569269-44 2021-07-31T01:51:04.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:03 smithi154 conmon[34553]: cluster 2021-07-31T01:51:02.578044+0000 mon.a (mon.0) 1009 : cluster [DBG] osdmap e363: 8 total, 8 up, 8 in 2021-07-31T01:51:04.591 INFO:tasks.rados.rados.0.smithi045.stdout:1655: done (2 left) 2021-07-31T01:51:04.591 INFO:tasks.rados.rados.0.smithi045.stdout:1656: done (1 left) 2021-07-31T01:51:04.592 INFO:tasks.rados.rados.0.smithi045.stdout:1657: done (0 left) 2021-07-31T01:51:04.592 INFO:tasks.rados.rados.0.smithi045.stdout:1658: rollback oid 2 current snap is 154 2021-07-31T01:51:04.592 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 2 to 153 2021-07-31T01:51:04.592 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 11 current snap is 154 2021-07-31T01:51:04.593 INFO:tasks.rados.rados.0.smithi045.stdout:1659: seq_num 522 ranges {2367488=745472,3112960=606208,3719168=196608} 2021-07-31T01:51:04.607 INFO:tasks.rados.rados.0.smithi045.stdout:1659: writing smithi04569269-11 from 2367488 to 3112960 tid 1 2021-07-31T01:51:04.611 INFO:tasks.rados.rados.0.smithi045.stdout:1659: writing smithi04569269-11 from 3112960 to 3719168 tid 2 2021-07-31T01:51:04.612 INFO:tasks.rados.rados.0.smithi045.stdout:1659: writing smithi04569269-11 from 3719168 to 3915776 tid 3 2021-07-31T01:51:04.612 INFO:tasks.rados.rados.0.smithi045.stdout:1660: copy_from oid 49 from oid 36 current snap is 154 2021-07-31T01:51:04.613 INFO:tasks.rados.rados.0.smithi045.stdout:1658: finishing rollback tid 1 to smithi04569269-2 2021-07-31T01:51:04.613 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 2 v 0 (ObjNum 1 snap 32731 seq_num 3190190304) dirty dne 2021-07-31T01:51:04.613 INFO:tasks.rados.rados.0.smithi045.stdout:1658: done (2 left) 2021-07-31T01:51:04.613 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 43 current snap is 154 2021-07-31T01:51:04.613 INFO:tasks.rados.rados.0.smithi045.stdout:1661: seq_num 523 ranges {1441792=770048,2211840=491520,2703360=311296} 2021-07-31T01:51:04.625 INFO:tasks.rados.rados.0.smithi045.stdout:1661: writing smithi04569269-43 from 1441792 to 2211840 tid 1 2021-07-31T01:51:04.627 INFO:tasks.rados.rados.0.smithi045.stdout:1661: writing smithi04569269-43 from 2211840 to 2703360 tid 2 2021-07-31T01:51:04.630 INFO:tasks.rados.rados.0.smithi045.stdout:1661: writing smithi04569269-43 from 2703360 to 3014656 tid 3 2021-07-31T01:51:04.631 INFO:tasks.rados.rados.0.smithi045.stdout:1662: read oid 37 snap -1 2021-07-31T01:51:04.631 INFO:tasks.rados.rados.0.smithi045.stdout:1662: expect (ObjNum 172 snap 41 seq_num 172) 2021-07-31T01:51:04.631 INFO:tasks.rados.rados.0.smithi045.stdout:1663: copy_from oid 45 from oid 31 current snap is 154 2021-07-31T01:51:04.631 INFO:tasks.rados.rados.0.smithi045.stdout:1659: finishing write tid 1 to smithi04569269-11 2021-07-31T01:51:04.632 INFO:tasks.rados.rados.0.smithi045.stdout:1659: finishing write tid 2 to smithi04569269-11 2021-07-31T01:51:04.632 INFO:tasks.rados.rados.0.smithi045.stdout:1659: finishing write tid 3 to smithi04569269-11 2021-07-31T01:51:04.632 INFO:tasks.rados.rados.0.smithi045.stdout:1659: finishing write tid 4 to smithi04569269-11 2021-07-31T01:51:04.632 INFO:tasks.rados.rados.0.smithi045.stdout:1659: finishing write tid 5 to smithi04569269-11 2021-07-31T01:51:04.633 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 11 v 1011 (ObjNum 522 snap 154 seq_num 522) dirty exists 2021-07-31T01:51:04.633 INFO:tasks.rados.rados.0.smithi045.stdout:1659: left oid 11 (ObjNum 522 snap 154 seq_num 522) 2021-07-31T01:51:04.633 INFO:tasks.rados.rados.0.smithi045.stdout:1660: finishing copy_from to smithi04569269-49 2021-07-31T01:51:04.633 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 49 v 994 (ObjNum 475 snap 140 seq_num 475) dirty exists 2021-07-31T01:51:04.633 INFO:tasks.rados.rados.0.smithi045.stdout:1660: finishing copy_from racing read to smithi04569269-49 2021-07-31T01:51:04.634 INFO:tasks.rados.rados.0.smithi045.stdout:1661: finishing write tid 1 to smithi04569269-43 2021-07-31T01:51:04.634 INFO:tasks.rados.rados.0.smithi045.stdout:1659: done (4 left) 2021-07-31T01:51:04.634 INFO:tasks.rados.rados.0.smithi045.stdout:1660: done (3 left) 2021-07-31T01:51:04.634 INFO:tasks.rados.rados.0.smithi045.stdout:1664: rollback oid 38 current snap is 154 2021-07-31T01:51:04.635 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 38 to 146 2021-07-31T01:51:04.635 INFO:tasks.rados.rados.0.smithi045.stdout:1665: setattr oid 42 current snap is 154 2021-07-31T01:51:04.635 INFO:tasks.rados.rados.0.smithi045.stdout:1661: finishing write tid 2 to smithi04569269-43 2021-07-31T01:51:04.636 INFO:tasks.rados.rados.0.smithi045.stdout:1661: finishing write tid 3 to smithi04569269-43 2021-07-31T01:51:04.636 INFO:tasks.rados.rados.0.smithi045.stdout:1666: snap_create 2021-07-31T01:51:04.636 INFO:tasks.rados.rados.0.smithi045.stdout:1662: expect (ObjNum 172 snap 41 seq_num 172) 2021-07-31T01:51:04.656 INFO:tasks.rados.rados.0.smithi045.stdout:1664: finishing rollback tid 0 to smithi04569269-38 2021-07-31T01:51:04.656 INFO:tasks.rados.rados.0.smithi045.stdout:1661: finishing write tid 4 to smithi04569269-43 2021-07-31T01:51:04.656 INFO:tasks.rados.rados.0.smithi045.stdout:1664: finishing rollback tid 1 to smithi04569269-38 2021-07-31T01:51:04.656 INFO:tasks.rados.rados.0.smithi045.stdout:1664: finishing rollback tid 2 to smithi04569269-38 2021-07-31T01:51:04.657 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 38 v 1014 (ObjNum 425 snap 126 seq_num 425) dirty exists 2021-07-31T01:51:04.657 INFO:tasks.rados.rados.0.smithi045.stdout:1661: finishing write tid 5 to smithi04569269-43 2021-07-31T01:51:04.657 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 43 v 1021 (ObjNum 523 snap 154 seq_num 523) dirty exists 2021-07-31T01:51:04.657 INFO:tasks.rados.rados.0.smithi045.stdout:1661: left oid 43 (ObjNum 523 snap 154 seq_num 523) 2021-07-31T01:51:04.657 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 42 v 996 (ObjNum 176 snap 46 seq_num 176) dirty exists 2021-07-31T01:51:04.658 INFO:tasks.rados.rados.0.smithi045.stdout:1663: finishing copy_from to smithi04569269-45 2021-07-31T01:51:04.658 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 45 v 1017 (ObjNum 490 snap 146 seq_num 490) dirty exists 2021-07-31T01:51:04.659 INFO:tasks.rados.rados.0.smithi045.stdout:1663: finishing copy_from racing read to smithi04569269-45 2021-07-31T01:51:05.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:04 smithi154 conmon[34553]: cluster 2021-07-31T01:51:03.584644+0000 mon.a (mon.0) 1010 : cluster [DBG] osdmap e364: 8 total, 8 up, 8 in 2021-07-31T01:51:05.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:04 smithi154 conmon[34553]: cluster 2021-07-31T01:51:03.845380+0000 mgr.x (mgr.24542) 163 : cluster [DBG] pgmap v326: 25 pgs: 25 active+clean; 154 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 6.6 MiB/s rd, 4.0 MiB/s wr, 29 op/s 2021-07-31T01:51:05.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:04 smithi045 conmon[33655]: cluster 2021-07-31T01:51:03.584644+0000 mon.a (mon.0 2021-07-31T01:51:05.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:04 smithi045 conmon[33655]: ) 1010 : cluster [DBG] osdmap e364: 8 total, 8 up, 8 in 2021-07-31T01:51:05.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:04 smithi045 conmon[33655]: cluster 2021-07-31T01:51:03.845380+0000 mgr.x (mgr.24542) 163 : cluster 2021-07-31T01:51:05.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:04 smithi045 conmon[33655]: [DBG] pgmap v326: 25 pgs: 25 active+clean; 154 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 6.6 MiB/s rd, 4.0 MiB/s wr, 29 op/s 2021-07-31T01:51:05.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:04 smithi045 conmon[39718]: cluster 2021-07-31T01:51:03.584644+0000 mon.a (mon.0) 1010 : cluster 2021-07-31T01:51:05.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:04 smithi045 conmon[39718]: [DBG] osdmap e364: 8 total, 8 up, 8 in 2021-07-31T01:51:05.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:04 smithi045 conmon[39718]: cluster 2021-07-31T01:51:03.845380+0000 mgr.x (mgr.24542 2021-07-31T01:51:05.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:04 smithi045 conmon[39718]: ) 163 : cluster [DBG] pgmap v326: 25 pgs: 25 active+clean; 154 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 6.6 MiB/s rd, 4.0 MiB/s wr, 29 op/s 2021-07-31T01:51:05.596 INFO:tasks.rados.rados.0.smithi045.stdout:1661: done (5 left) 2021-07-31T01:51:05.596 INFO:tasks.rados.rados.0.smithi045.stdout:1662: done (4 left) 2021-07-31T01:51:05.596 INFO:tasks.rados.rados.0.smithi045.stdout:1663: done (3 left) 2021-07-31T01:51:05.597 INFO:tasks.rados.rados.0.smithi045.stdout:1664: done (2 left) 2021-07-31T01:51:05.597 INFO:tasks.rados.rados.0.smithi045.stdout:1665: done (1 left) 2021-07-31T01:51:05.598 INFO:tasks.rados.rados.0.smithi045.stdout:1666: done (0 left) 2021-07-31T01:51:05.598 INFO:tasks.rados.rados.0.smithi045.stdout:1667: rollback oid 31 current snap is 155 2021-07-31T01:51:05.598 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 31 to 154 2021-07-31T01:51:05.598 INFO:tasks.rados.rados.0.smithi045.stdout:1668: snap_remove snap 146 2021-07-31T01:51:06.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:05 smithi154 conmon[34553]: cluster 2021-07-31T01:51:04.589885+0000 mon.a (mon.0) 1011 : cluster 2021-07-31T01:51:06.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:05 smithi154 conmon[34553]: [DBG] osdmap e365: 8 total, 8 up, 8 in 2021-07-31T01:51:06.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:05 smithi045 conmon[39718]: cluster 2021-07-31T01:51:04.589885+0000 mon.a (mon.0) 1011 : cluster [DBG] osdmap e365: 8 total, 8 up, 8 in 2021-07-31T01:51:06.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:05 smithi045 conmon[33655]: cluster 2021-07-31T01:51:04.589885+0000 mon.a (mon.0) 1011 : cluster 2021-07-31T01:51:06.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:05 smithi045 conmon[33655]: [DBG] osdmap e365: 8 total, 8 up, 8 in 2021-07-31T01:51:06.549 INFO:tasks.rados.rados.0.smithi045.stdout:1668: done (1 left) 2021-07-31T01:51:06.550 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 25 current snap is 155 2021-07-31T01:51:06.550 INFO:tasks.rados.rados.0.smithi045.stdout:1669: seq_num 524 ranges {1998848=786432,2785280=778240,3563520=24576} 2021-07-31T01:51:06.566 INFO:tasks.rados.rados.0.smithi045.stdout:1669: writing smithi04569269-25 from 1998848 to 2785280 tid 1 2021-07-31T01:51:06.571 INFO:tasks.rados.rados.0.smithi045.stdout:1669: writing smithi04569269-25 from 2785280 to 3563520 tid 2 2021-07-31T01:51:06.571 INFO:tasks.rados.rados.0.smithi045.stdout:1669: writing smithi04569269-25 from 3563520 to 3588096 tid 3 2021-07-31T01:51:06.571 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 48 current snap is 155 2021-07-31T01:51:06.571 INFO:tasks.rados.rados.0.smithi045.stdout:1667: finishing rollback tid 0 to smithi04569269-31 2021-07-31T01:51:06.572 INFO:tasks.rados.rados.0.smithi045.stdout:1670: seq_num 525 ranges {221184=802816,1024000=712704,1736704=90112} 2021-07-31T01:51:06.577 INFO:tasks.rados.rados.0.smithi045.stdout:1670: writing smithi04569269-48 from 221184 to 1024000 tid 1 2021-07-31T01:51:06.581 INFO:tasks.rados.rados.0.smithi045.stdout:1670: writing smithi04569269-48 from 1024000 to 1736704 tid 2 2021-07-31T01:51:06.582 INFO:tasks.rados.rados.0.smithi045.stdout:1670: writing smithi04569269-48 from 1736704 to 1826816 tid 3 2021-07-31T01:51:06.582 INFO:tasks.rados.rados.0.smithi045.stdout:1671: read oid 6 snap -1 2021-07-31T01:51:06.583 INFO:tasks.rados.rados.0.smithi045.stdout:1671: expect (ObjNum 514 snap 152 seq_num 514) 2021-07-31T01:51:06.583 INFO:tasks.rados.rados.0.smithi045.stdout:1672: snap_create 2021-07-31T01:51:06.583 INFO:tasks.rados.rados.0.smithi045.stdout:1667: finishing rollback tid 1 to smithi04569269-31 2021-07-31T01:51:06.584 INFO:tasks.rados.rados.0.smithi045.stdout:1667: finishing rollback tid 2 to smithi04569269-31 2021-07-31T01:51:06.584 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 31 v 1066 (ObjNum 490 snap 146 seq_num 490) dirty exists 2021-07-31T01:51:06.584 INFO:tasks.rados.rados.0.smithi045.stdout:1669: finishing write tid 1 to smithi04569269-25 2021-07-31T01:51:06.584 INFO:tasks.rados.rados.0.smithi045.stdout:1669: finishing write tid 2 to smithi04569269-25 2021-07-31T01:51:06.584 INFO:tasks.rados.rados.0.smithi045.stdout:1669: finishing write tid 3 to smithi04569269-25 2021-07-31T01:51:06.585 INFO:tasks.rados.rados.0.smithi045.stdout:1669: finishing write tid 4 to smithi04569269-25 2021-07-31T01:51:06.585 INFO:tasks.rados.rados.0.smithi045.stdout:1669: finishing write tid 5 to smithi04569269-25 2021-07-31T01:51:06.585 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 25 v 283 (ObjNum 524 snap 155 seq_num 524) dirty exists 2021-07-31T01:51:06.586 INFO:tasks.rados.rados.0.smithi045.stdout:1669: left oid 25 (ObjNum 524 snap 155 seq_num 524) 2021-07-31T01:51:06.586 INFO:tasks.rados.rados.0.smithi045.stdout:1670: finishing write tid 1 to smithi04569269-48 2021-07-31T01:51:06.587 INFO:tasks.rados.rados.0.smithi045.stdout:1670: finishing write tid 2 to smithi04569269-48 2021-07-31T01:51:06.587 INFO:tasks.rados.rados.0.smithi045.stdout:1670: finishing write tid 3 to smithi04569269-48 2021-07-31T01:51:06.588 INFO:tasks.rados.rados.0.smithi045.stdout:1670: finishing write tid 4 to smithi04569269-48 2021-07-31T01:51:06.589 INFO:tasks.rados.rados.0.smithi045.stdout:1670: finishing write tid 5 to smithi04569269-48 2021-07-31T01:51:06.589 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 48 v 911 (ObjNum 525 snap 155 seq_num 525) dirty exists 2021-07-31T01:51:06.589 INFO:tasks.rados.rados.0.smithi045.stdout:1670: left oid 48 (ObjNum 525 snap 155 seq_num 525) 2021-07-31T01:51:06.596 INFO:tasks.rados.rados.0.smithi045.stdout:1671: expect (ObjNum 514 snap 152 seq_num 514) 2021-07-31T01:51:07.003 INFO:journalctl@ceph.osd.4.smithi154.stdout:Jul 31 01:51:06 smithi154 conmon[38506]: debug 2021-07-31T01:51:06.546+0000 7feae39cf700 -1 osd.4 pg_epoch: 367 pg[3.9s0( v 364'200 (0'0,364'200] local-lis/les=56/57 n=3 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [4,2,7]p4(0) r=0 lpr=56 crt=363'199 lcod 363'199 mlcod 363'199 active+clean+snaptrim trimq=[124~1]] removing snap head 2021-07-31T01:51:07.003 INFO:journalctl@ceph.osd.5.smithi154.stdout:Jul 31 01:51:06 smithi154 conmon[42050]: debug 2021-07-31T01:51:06.566+0000 7fc9b105c700 -1 osd.5 pg_epoch: 367 pg[3.cs0( v 367'617 (0'0,367'617] local-lis/les=56/57 n=8 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [5,3,6]p5(0) r=0 lpr=56 luod=367'615 crt=367'615 lcod 367'613 mlcod 367'613 active+clean+snaptrim trimq=[124~1]] removing snap head 2021-07-31T01:51:07.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:06 smithi154 conmon[34553]: cluster 2021-07-31T01:51:05.595396+0000 mon.a (mon.0) 1012 2021-07-31T01:51:07.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:06 smithi154 conmon[34553]: : cluster [DBG] osdmap e366: 8 total, 8 up, 8 in 2021-07-31T01:51:07.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:06 smithi154 conmon[34553]: cluster 2021-07-31T01:51:05.845819+0000 mgr.x (mgr.24542) 164 : cluster [DBG] pgmap v329: 25 pgs: 25 active+clean; 154 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 6.6 MiB/s rd, 4.0 MiB/s wr, 29 op/s 2021-07-31T01:51:07.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:06 smithi154 conmon[34553]: cluster 2021-07-31T01:51: 2021-07-31T01:51:07.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:06 smithi154 conmon[34553]: 06.548063+0000 mon.a (mon.0) 1013 : cluster [DBG] osdmap e367: 8 total, 8 up, 8 in 2021-07-31T01:51:07.026 INFO:journalctl@ceph.osd.1.smithi045.stdout:Jul 31 01:51:06 smithi045 conmon[46771]: debug 2021-07-31T01:51:07.027 INFO:journalctl@ceph.osd.1.smithi045.stdout:Jul 31 01:51:06 smithi045 conmon[46771]: 2021-07-31T01:51:06.553+0000 7f41979aa700 -1 osd.1 pg_epoch: 367 pg[3.0s0( v 367'398 (0'0,367'398] local-lis/les=56/57 n=5 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [1,2,6]p1(0) r=0 lpr=56 luod=363'396 crt=363'396 lcod 363'394 mlcod 363'394 active+clean+snaptrim trimq=[124~1]] removing snap head 2021-07-31T01:51:07.027 INFO:journalctl@ceph.osd.1.smithi045.stdout:Jul 31 01:51:06 smithi045 conmon[46771]: 2021-07-31T01:51:07.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:06 smithi045 conmon[33655]: cluster 2021-07-31T01:51:05 2021-07-31T01:51:07.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:06 smithi045 conmon[33655]: .595396+0000 mon.a (mon.0) 1012 : cluster [DBG] osdmap e366: 8 total, 8 up, 8 in 2021-07-31T01:51:07.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:06 smithi045 conmon[33655]: cluster 2021-07-31T01:51:07.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:06 smithi045 conmon[33655]: 2021-07-31T01:51:05.845819+0000 2021-07-31T01:51:07.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:06 smithi045 conmon[33655]: mgr.x (mgr.24542) 164 : 2021-07-31T01:51:07.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:06 smithi045 conmon[33655]: cluster [DBG] pgmap v329: 25 pgs: 25 active+clean; 154 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 6.6 MiB/s rd, 4.0 MiB/s wr, 29 op/s 2021-07-31T01:51:07.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:06 smithi045 conmon[33655]: cluster 2021-07-31T01:51:07.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:06 smithi045 conmon[33655]: 2021-07-31T01:51: 2021-07-31T01:51:07.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:06 smithi045 conmon[33655]: 06.548063+0000 mon.a ( 2021-07-31T01:51:07.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:06 smithi045 conmon[33655]: mon.0) 1013 : cluster 2021-07-31T01:51:07.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:06 smithi045 conmon[33655]: [DBG] osdmap e367: 8 total, 8 up, 8 in 2021-07-31T01:51:07.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:06 smithi045 conmon[39718]: cluster 2021-07-31T01:51:05.595396 2021-07-31T01:51:07.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:06 smithi045 conmon[39718]: +0000 mon.a (mon.0) 1012 : cluster [DBG] osdmap e366: 8 total, 8 up, 8 in 2021-07-31T01:51:07.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:06 smithi045 conmon[39718]: cluster 2021 2021-07-31T01:51:07.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:06 smithi045 conmon[39718]: -07-31T01:51:05.845819+0000 mgr.x (mgr.24542) 164 : 2021-07-31T01:51:07.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:06 smithi045 conmon[39718]: cluster [DBG] pgmap v329: 25 pgs: 25 active+clean; 154 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 6.6 MiB/s rd, 4.0 MiB/s wr, 29 op/s 2021-07-31T01:51:07.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:06 smithi045 conmon[39718]: cluster 2021-07- 2021-07-31T01:51:07.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:06 smithi045 conmon[39718]: 31T01:51:06.548063+0000 mon.a (mon.0 2021-07-31T01:51:07.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:06 smithi045 conmon[39718]: ) 1013 : cluster [DBG] osdmap e367: 8 total, 8 up, 8 in 2021-07-31T01:51:07.557 INFO:tasks.rados.rados.0.smithi045.stdout:1667: done (4 left) 2021-07-31T01:51:07.557 INFO:tasks.rados.rados.0.smithi045.stdout:1669: done (3 left) 2021-07-31T01:51:07.557 INFO:tasks.rados.rados.0.smithi045.stdout:1670: done (2 left) 2021-07-31T01:51:07.557 INFO:tasks.rados.rados.0.smithi045.stdout:1671: done (1 left) 2021-07-31T01:51:07.558 INFO:tasks.rados.rados.0.smithi045.stdout:1672: done (0 left) 2021-07-31T01:51:07.558 INFO:tasks.rados.rados.0.smithi045.stdout:1673: copy_from oid 23 from oid 48 current snap is 156 2021-07-31T01:51:07.558 INFO:tasks.rados.rados.0.smithi045.stdout:1674: delete oid 40 current snap is 156 2021-07-31T01:51:07.562 INFO:tasks.rados.rados.0.smithi045.stdout:1674: done (1 left) 2021-07-31T01:51:07.562 INFO:tasks.rados.rados.0.smithi045.stdout:1675: snap_remove snap 152 2021-07-31T01:51:08.562 INFO:tasks.rados.rados.0.smithi045.stdout:1675: done (1 left) 2021-07-31T01:51:08.563 INFO:tasks.rados.rados.0.smithi045.stdout:1676: setattr oid 37 current snap is 156 2021-07-31T01:51:08.563 INFO:tasks.rados.rados.0.smithi045.stdout:1673: finishing copy_from to smithi04569269-23 2021-07-31T01:51:08.564 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 23 v 911 (ObjNum 525 snap 155 seq_num 525) dirty exists 2021-07-31T01:51:08.564 INFO:tasks.rados.rados.0.smithi045.stdout:1673: finishing copy_from racing read to smithi04569269-23 2021-07-31T01:51:08.565 INFO:tasks.rados.rados.0.smithi045.stdout:1673: done (1 left) 2021-07-31T01:51:08.565 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 46 current snap is 156 2021-07-31T01:51:08.565 INFO:tasks.rados.rados.0.smithi045.stdout:1677: seq_num 527 ranges {9330688=434176,9764864=475136,10240000=565248,10805248=163840} 2021-07-31T01:51:08.617 INFO:tasks.rados.rados.0.smithi045.stdout:1677: writing smithi04569269-46 from 9330688 to 9764864 tid 1 2021-07-31T01:51:08.619 INFO:tasks.rados.rados.0.smithi045.stdout:1677: writing smithi04569269-46 from 9764864 to 10240000 tid 2 2021-07-31T01:51:08.623 INFO:tasks.rados.rados.0.smithi045.stdout:1677: writing smithi04569269-46 from 10240000 to 10805248 tid 3 2021-07-31T01:51:08.624 INFO:tasks.rados.rados.0.smithi045.stdout:1677: writing smithi04569269-46 from 10805248 to 10969088 tid 4 2021-07-31T01:51:08.624 INFO:tasks.rados.rados.0.smithi045.stdout:1678: delete oid 23 current snap is 156 2021-07-31T01:51:08.624 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 37 v 402 (ObjNum 172 snap 41 seq_num 172) dirty exists 2021-07-31T01:51:08.625 INFO:tasks.rados.rados.0.smithi045.stdout:1677: finishing write tid 1 to smithi04569269-46 2021-07-31T01:51:08.625 INFO:tasks.rados.rados.0.smithi045.stdout:1677: finishing write tid 2 to smithi04569269-46 2021-07-31T01:51:08.626 INFO:tasks.rados.rados.0.smithi045.stdout:1676: done (2 left) 2021-07-31T01:51:08.626 INFO:tasks.rados.rados.0.smithi045.stdout:1678: done (1 left) 2021-07-31T01:51:08.627 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 26 current snap is 156 2021-07-31T01:51:08.627 INFO:tasks.rados.rados.0.smithi045.stdout:1679: seq_num 529 ranges {221184=466944,688128=647168,1335296=532480,1867776=24576} 2021-07-31T01:51:08.630 INFO:tasks.rados.rados.0.smithi045.stdout:1679: writing smithi04569269-26 from 221184 to 688128 tid 1 2021-07-31T01:51:08.633 INFO:tasks.rados.rados.0.smithi045.stdout:1679: writing smithi04569269-26 from 688128 to 1335296 tid 2 2021-07-31T01:51:08.638 INFO:tasks.rados.rados.0.smithi045.stdout:1679: writing smithi04569269-26 from 1335296 to 1867776 tid 3 2021-07-31T01:51:08.638 INFO:tasks.rados.rados.0.smithi045.stdout:1679: writing smithi04569269-26 from 1867776 to 1892352 tid 4 2021-07-31T01:51:08.638 INFO:tasks.rados.rados.0.smithi045.stdout:1680: snap_remove snap 154 2021-07-31T01:51:09.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:08 smithi154 conmon[34553]: cluster 2021-07-31T01:51:07.556133+0000 mon.a (mon.0) 1014 2021-07-31T01:51:09.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:08 smithi154 conmon[34553]: : cluster [DBG] osdmap e368: 8 total, 8 up, 8 in 2021-07-31T01:51:09.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:08 smithi154 conmon[34553]: cluster 2021-07-31T01:51:07.846213+0000 mgr.x (mgr.24542) 165 : cluster [DBG] pgmap v332: 25 pgs: 25 active+clean; 154 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail 2021-07-31T01:51:09.026 INFO:journalctl@ceph.osd.0.smithi045.stdout:Jul 31 01:51:08 smithi045 conmon[43243]: debug 2021-07-31T01:51:08.576+0000 7fdf6bf90700 -1 osd.0 pg_epoch: 369 pg[3.1s0( v 367'316 (0'0,367'316] local-lis/les=56/57 n=4 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [0,4,3]p0(0) r=0 lpr=56 crt=367'316 lcod 364'314 mlcod 364'314 active+clean+snaptrim trimq=[131~1] ps=[124~1]] removing snap head 2021-07-31T01:51:09.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:08 smithi045 conmon[33655]: cluster 2021-07-31T01:51:07.556133+0000 2021-07-31T01:51:09.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:08 smithi045 conmon[33655]: mon.a (mon.0) 1014 : cluster [DBG] osdmap e368: 8 total, 8 up, 8 in 2021-07-31T01:51:09.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:08 smithi045 conmon[33655]: cluster 2021-07-31T01:51:07.846213+0000 mgr.x (mgr.24542) 165 : cluster [DBG] pgmap v332: 25 pgs: 25 active+clean; 154 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail 2021-07-31T01:51:09.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:08 smithi045 conmon[39718]: cluster 2021-07-31T01:51:07.556133+0000 mon.a (mon.0) 1014 : cluster 2021-07-31T01:51:09.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:08 smithi045 conmon[39718]: [DBG] osdmap e368: 8 total, 8 up, 8 in 2021-07-31T01:51:09.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:08 smithi045 conmon[39718]: cluster 2021-07-31T01:51:07.846213+0000 mgr.x (mgr.24542) 165 : cluster [DBG] pgmap v332: 25 pgs: 25 active+clean; 154 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail 2021-07-31T01:51:09.028 INFO:journalctl@ceph.osd.3.smithi045.stdout:Jul 31 01:51:08 smithi045 conmon[54013]: debug 2021-07-31T01:51:09.028 INFO:journalctl@ceph.osd.3.smithi045.stdout:Jul 31 01:51:08 smithi045 conmon[54013]: 2021-07-31T01:51:08.569+0000 7f64f88c5700 -1 osd.3 pg_epoch: 369 pg[3.bs0( v 367'1074 (0'0,367'1074] local-lis/les=56/57 n=13 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [3,0,4]p3(0) r=0 lpr=56 crt=367'1074 lcod 367'1072 mlcod 367'1072 active+clean+snaptrim trimq=[131~1] ps=[124~1]] removing snap head 2021-07-31T01:51:09.028 INFO:journalctl@ceph.osd.3.smithi045.stdout:Jul 31 01:51:08 smithi045 conmon[54013]: 2021-07-31T01:51:09.565 INFO:tasks.rados.rados.0.smithi045.stdout:1680: done (2 left) 2021-07-31T01:51:09.565 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 9 current snap is 156 2021-07-31T01:51:09.566 INFO:tasks.rados.rados.0.smithi045.stdout:1677: finishing write tid 3 to smithi04569269-46 2021-07-31T01:51:09.566 INFO:tasks.rados.rados.0.smithi045.stdout:1677: finishing write tid 4 to smithi04569269-46 2021-07-31T01:51:09.566 INFO:tasks.rados.rados.0.smithi045.stdout:1677: finishing write tid 5 to smithi04569269-46 2021-07-31T01:51:09.566 INFO:tasks.rados.rados.0.smithi045.stdout:1677: finishing write tid 6 to smithi04569269-46 2021-07-31T01:51:09.566 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 46 v 276 (ObjNum 527 snap 156 seq_num 527) dirty exists 2021-07-31T01:51:09.567 INFO:tasks.rados.rados.0.smithi045.stdout:1677: left oid 46 (ObjNum 527 snap 156 seq_num 527) 2021-07-31T01:51:09.567 INFO:tasks.rados.rados.0.smithi045.stdout:1679: finishing write tid 1 to smithi04569269-26 2021-07-31T01:51:09.567 INFO:tasks.rados.rados.0.smithi045.stdout:1679: finishing write tid 2 to smithi04569269-26 2021-07-31T01:51:09.567 INFO:tasks.rados.rados.0.smithi045.stdout:1679: finishing write tid 3 to smithi04569269-26 2021-07-31T01:51:09.568 INFO:tasks.rados.rados.0.smithi045.stdout:1679: finishing write tid 4 to smithi04569269-26 2021-07-31T01:51:09.568 INFO:tasks.rados.rados.0.smithi045.stdout:1679: finishing write tid 5 to smithi04569269-26 2021-07-31T01:51:09.568 INFO:tasks.rados.rados.0.smithi045.stdout:1679: finishing write tid 6 to smithi04569269-26 2021-07-31T01:51:09.568 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 26 v 400 (ObjNum 529 snap 156 seq_num 529) dirty exists 2021-07-31T01:51:09.568 INFO:tasks.rados.rados.0.smithi045.stdout:1679: left oid 26 (ObjNum 529 snap 156 seq_num 529) 2021-07-31T01:51:09.569 INFO:tasks.rados.rados.0.smithi045.stdout:1681: seq_num 530 ranges {5791744=483328,6275072=532480,6807552=671744} 2021-07-31T01:51:09.599 INFO:tasks.rados.rados.0.smithi045.stdout:1681: writing smithi04569269-9 from 5791744 to 6275072 tid 1 2021-07-31T01:51:09.602 INFO:tasks.rados.rados.0.smithi045.stdout:1681: writing smithi04569269-9 from 6275072 to 6807552 tid 2 2021-07-31T01:51:09.606 INFO:tasks.rados.rados.0.smithi045.stdout:1681: writing smithi04569269-9 from 6807552 to 7479296 tid 3 2021-07-31T01:51:09.607 INFO:tasks.rados.rados.0.smithi045.stdout:1677: done (2 left) 2021-07-31T01:51:09.607 INFO:tasks.rados.rados.0.smithi045.stdout:1679: done (1 left) 2021-07-31T01:51:09.607 INFO:tasks.rados.rados.0.smithi045.stdout:1682: snap_create 2021-07-31T01:51:09.607 INFO:tasks.rados.rados.0.smithi045.stdout:1681: finishing write tid 1 to smithi04569269-9 2021-07-31T01:51:09.608 INFO:tasks.rados.rados.0.smithi045.stdout:1681: finishing write tid 2 to smithi04569269-9 2021-07-31T01:51:09.610 INFO:tasks.rados.rados.0.smithi045.stdout:1681: finishing write tid 3 to smithi04569269-9 2021-07-31T01:51:09.610 INFO:tasks.rados.rados.0.smithi045.stdout:1681: finishing write tid 4 to smithi04569269-9 2021-07-31T01:51:09.612 INFO:tasks.rados.rados.0.smithi045.stdout:1681: finishing write tid 5 to smithi04569269-9 2021-07-31T01:51:09.612 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 9 v 385 (ObjNum 530 snap 156 seq_num 530) dirty exists 2021-07-31T01:51:09.612 INFO:tasks.rados.rados.0.smithi045.stdout:1681: left oid 9 (ObjNum 530 snap 156 seq_num 530) 2021-07-31T01:51:10.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:09 smithi154 conmon[34553]: cluster 2021-07-31T01:51:08.561689+0000 mon.a (mon.0) 1015 : cluster [DBG] osdmap e369: 8 total, 8 up, 8 in 2021-07-31T01:51:10.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:09 smithi045 conmon[39718]: cluster 2021-07-31T01:51:08.561689+0000 mon.a (mon 2021-07-31T01:51:10.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:09 smithi045 conmon[39718]: .0) 1015 : cluster [DBG] osdmap e369: 8 total, 8 up, 8 in 2021-07-31T01:51:10.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:09 smithi045 conmon[33655]: cluster 2021-07-31T01:51:08.561689+0000 mon.a (mon 2021-07-31T01:51:10.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:09 smithi045 conmon[33655]: .0) 1015 : cluster [DBG] osdmap e369: 8 total, 8 up, 8 in 2021-07-31T01:51:10.575 INFO:tasks.rados.rados.0.smithi045.stdout:1681: done (1 left) 2021-07-31T01:51:10.575 INFO:tasks.rados.rados.0.smithi045.stdout:1682: done (0 left) 2021-07-31T01:51:10.576 INFO:tasks.rados.rados.0.smithi045.stdout:1683: rollback oid 36 current snap is 157 2021-07-31T01:51:10.576 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 36 to 156 2021-07-31T01:51:10.576 INFO:tasks.rados.rados.0.smithi045.stdout:1684: snap_create 2021-07-31T01:51:10.580 INFO:tasks.rados.rados.0.smithi045.stdout:1683: finishing rollback tid 0 to smithi04569269-36 2021-07-31T01:51:10.581 INFO:tasks.rados.rados.0.smithi045.stdout:1683: finishing rollback tid 1 to smithi04569269-36 2021-07-31T01:51:10.581 INFO:tasks.rados.rados.0.smithi045.stdout:1683: finishing rollback tid 2 to smithi04569269-36 2021-07-31T01:51:10.581 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 36 v 1090 (ObjNum 475 snap 140 seq_num 475) dirty exists 2021-07-31T01:51:11.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:10 smithi154 conmon[34553]: cluster 2021-07-31T01:51:09.563902+0000 mon.a (mon.0) 1016 : 2021-07-31T01:51:11.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:10 smithi154 conmon[34553]: cluster [DBG] osdmap e370: 8 total, 8 up, 8 in 2021-07-31T01:51:11.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:10 smithi154 conmon[34553]: cluster 2021-07-31T01:51:09.846637+0000 mgr.x (mgr.24542) 166 : cluster [DBG] pgmap v335: 25 pgs: 25 active+clean; 140 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 2.6 MiB/s rd, 6.5 MiB/s wr, 35 op/s 2021-07-31T01:51:11.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:10 smithi045 conmon[33655]: cluster 2021-07-31T01:51:09.563902+0000 mon.a (mon.0) 1016 2021-07-31T01:51:11.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:10 smithi045 conmon[33655]: : cluster [DBG] osdmap e370: 8 total, 8 up, 8 in 2021-07-31T01:51:11.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:10 smithi045 conmon[33655]: cluster 2021-07-31T01:51:09.846637+0000 mgr.x (mgr.24542) 166 : cluster [DBG] pgmap v335: 25 pgs: 25 active+clean; 140 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 2.6 MiB/s rd, 6.5 MiB/s wr, 35 op/s 2021-07-31T01:51:11.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:10 smithi045 conmon[39718]: cluster 2021-07-31T01:51:09.563902+0000 2021-07-31T01:51:11.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:10 smithi045 conmon[39718]: mon.a (mon.0) 1016 : cluster [DBG] osdmap e370: 8 total, 8 up, 8 in 2021-07-31T01:51:11.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:10 smithi045 conmon[39718]: cluster 2021-07-31T01:51:09.846637 2021-07-31T01:51:11.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:10 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 166 : cluster [DBG] pgmap v335: 25 pgs: 25 active+clean; 140 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 2.6 MiB/s rd, 6.5 MiB/s wr, 35 op/s 2021-07-31T01:51:11.550 INFO:tasks.rados.rados.0.smithi045.stdout:1683: done (1 left) 2021-07-31T01:51:11.551 INFO:tasks.rados.rados.0.smithi045.stdout:1684: done (0 left) 2021-07-31T01:51:11.551 INFO:tasks.rados.rados.0.smithi045.stdout:1685: read oid 4 snap 156 2021-07-31T01:51:11.551 INFO:tasks.rados.rados.0.smithi045.stdout:1685: expect (ObjNum 470 snap 136 seq_num 470) 2021-07-31T01:51:11.551 INFO:tasks.rados.rados.0.smithi045.stdout:1686: snap_remove snap 155 2021-07-31T01:51:12.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:11 smithi154 conmon[34553]: cluster 2021-07-31T01:51:10.574218+0000 mon.a (mon.0) 1017 : cluster [DBG] osdmap e371: 8 total, 8 up, 8 in 2021-07-31T01:51:12.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:11 smithi154 conmon[34553]: 2021-07-31T01:51:12.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:11 smithi154 conmon[34553]: cluster 2021-07-31T01:51:11.549295+0000 mon.a (mon.0) 1018 : cluster [DBG] osdmap e372: 8 total, 8 up, 8 in 2021-07-31T01:51:12.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:11 smithi045 conmon[39718]: cluster 2021-07-31T01:51:10.574218+0000 mon.a (mon 2021-07-31T01:51:12.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:11 smithi045 conmon[39718]: .0) 1017 : cluster [DBG] osdmap e371: 8 total, 8 up, 8 in 2021-07-31T01:51:12.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:11 smithi045 conmon[39718]: cluster 2021-07-31T01:51:11 2021-07-31T01:51:12.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:11 smithi045 conmon[39718]: .549295+0000 mon.a (mon.0) 1018 : cluster [DBG] osdmap e372: 8 total, 8 up, 8 in 2021-07-31T01:51:12.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:11 smithi045 conmon[33655]: cluster 2021-07-31T01:51:10.574218+0000 mon.a (mon.0) 2021-07-31T01:51:12.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:11 smithi045 conmon[33655]: 1017 : cluster [DBG] osdmap e371: 8 total, 8 up, 8 in 2021-07-31T01:51:12.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:11 smithi045 conmon[33655]: cluster 2021-07-31T01:51:11.549295+0000 mon.a (mon.0) 1018 : cluster [DBG] osdmap e372: 8 total, 8 up, 8 in 2021-07-31T01:51:12.557 INFO:tasks.rados.rados.0.smithi045.stdout:1686: done (1 left) 2021-07-31T01:51:12.557 INFO:tasks.rados.rados.0.smithi045.stdout:1687: setattr oid 49 current snap is 158 2021-07-31T01:51:12.557 INFO:tasks.rados.rados.0.smithi045.stdout:1685: expect (ObjNum 470 snap 136 seq_num 470) 2021-07-31T01:51:12.689 INFO:tasks.rados.rados.0.smithi045.stdout:1685: done (1 left) 2021-07-31T01:51:12.689 INFO:tasks.rados.rados.0.smithi045.stdout:1688: snap_remove snap 153 2021-07-31T01:51:13.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:12 smithi154 conmon[34553]: cluster 2021-07-31T01:51:11.847040+0000 mgr.x (mgr.24542) 167 : cluster [DBG] pgmap v338: 25 pgs: 25 active+clean; 140 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 2.6 MiB/s rd, 6.5 MiB/s wr, 35 op/s 2021-07-31T01:51:13.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:12 smithi154 conmon[34553]: 2021-07-31T01:51:13.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:12 smithi154 conmon[34553]: cluster 2021-07-31T01:51:12.555628+0000 mon.a (mon.0) 1019 : cluster [DBG] osdmap e373: 8 total, 8 up, 8 in 2021-07-31T01:51:13.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:12 smithi045 conmon[39718]: cluster 2021-07-31T01:51:11.847040+0000 mgr.x (mgr.24542) 167 : cluster 2021-07-31T01:51:13.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:12 smithi045 conmon[39718]: [DBG] pgmap v338: 25 pgs: 25 active+clean; 140 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 2.6 MiB/s rd, 6.5 MiB/s wr, 35 op/s 2021-07-31T01:51:13.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:12 smithi045 conmon[39718]: cluster 2021-07-31T01:51:12.555628+0000 mon.a (mon.0) 1019 : cluster [DBG] osdmap e373: 8 total, 8 up, 8 in 2021-07-31T01:51:13.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:12 smithi045 conmon[33655]: cluster 2021-07-31T01:51:11.847040+0000 mgr.x (mgr.24542 2021-07-31T01:51:13.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:12 smithi045 conmon[33655]: ) 167 : cluster [DBG] pgmap v338: 25 pgs: 25 active+clean; 140 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 2.6 MiB/s rd, 6.5 MiB/s wr, 35 op/s 2021-07-31T01:51:13.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:12 smithi045 conmon[33655]: cluster 2021-07-31T01:51:12.555628+0000 mon.a (mon.0) 1019 : cluster [DBG] osdmap e373: 8 total, 8 up, 8 in 2021-07-31T01:51:13.585 INFO:tasks.rados.rados.0.smithi045.stdout:1688: done (1 left) 2021-07-31T01:51:13.585 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 41 current snap is 158 2021-07-31T01:51:13.585 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 49 v 1092 (ObjNum 475 snap 140 seq_num 475) dirty exists 2021-07-31T01:51:13.585 INFO:tasks.rados.rados.0.smithi045.stdout:1689: seq_num 531 ranges {0=499712,499712=417792,917504=548864,1466368=237568} 2021-07-31T01:51:13.587 INFO:tasks.rados.rados.0.smithi045.stdout:1689: writing smithi04569269-41 from 0 to 499712 tid 1 2021-07-31T01:51:13.590 INFO:tasks.rados.rados.0.smithi045.stdout:1689: writing smithi04569269-41 from 499712 to 917504 tid 2 2021-07-31T01:51:13.594 INFO:tasks.rados.rados.0.smithi045.stdout:1689: writing smithi04569269-41 from 917504 to 1466368 tid 3 2021-07-31T01:51:13.598 INFO:tasks.rados.rados.0.smithi045.stdout:1689: writing smithi04569269-41 from 1466368 to 1703936 tid 4 2021-07-31T01:51:13.599 INFO:tasks.rados.rados.0.smithi045.stdout:1687: done (1 left) 2021-07-31T01:51:13.599 INFO:tasks.rados.rados.0.smithi045.stdout:1690: copy_from oid 29 from oid 48 current snap is 158 2021-07-31T01:51:13.599 INFO:tasks.rados.rados.0.smithi045.stdout:1689: finishing write tid 1 to smithi04569269-41 2021-07-31T01:51:13.599 INFO:tasks.rados.rados.0.smithi045.stdout:1689: finishing write tid 2 to smithi04569269-41 2021-07-31T01:51:13.600 INFO:tasks.rados.rados.0.smithi045.stdout:1691: snap_remove snap 157 2021-07-31T01:51:13.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:51:13 smithi045 conmon[61210]: level=error ts=2021-07-31T01:51:13.515Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:51:13.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:51:13 smithi045 conmon[61210]: level=error ts=2021-07-31T01:51:13.515Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:51:13.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:51:13 smithi045 conmon[61210]: level=error ts=2021-07-31T01:51:13.515Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T01:51:14.003 INFO:journalctl@ceph.osd.7.smithi154.stdout:Jul 31 01:51:13 smithi154 conmon[49149]: debug 2021-07-31T01:51:13.598+0000 7f8a75203700 -1 osd.7 pg_epoch: 374 pg[3.es0( v 373'628 (0'0,373'628] local-lis/les=56/57 n=5 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [7,4,1]p7(0) r=0 lpr=56 crt=373'628 lcod 370'626 mlcod 370'626 active+clean+snaptrim trimq=[132~1] ps=[133~1,135~1]] removing snap head 2021-07-31T01:51:14.589 INFO:tasks.rados.rados.0.smithi045.stdout:1691: done (2 left) 2021-07-31T01:51:14.590 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 49 current snap is 158 2021-07-31T01:51:14.590 INFO:tasks.rados.rados.0.smithi045.stdout:1689: finishing write tid 3 to smithi04569269-41 2021-07-31T01:51:14.590 INFO:tasks.rados.rados.0.smithi045.stdout:1692: seq_num 532 ranges {761856=516096,1277952=696320,1974272=507904} 2021-07-31T01:51:14.596 INFO:tasks.rados.rados.0.smithi045.stdout:1692: writing smithi04569269-49 from 761856 to 1277952 tid 1 2021-07-31T01:51:14.600 INFO:tasks.rados.rados.0.smithi045.stdout:1692: writing smithi04569269-49 from 1277952 to 1974272 tid 2 2021-07-31T01:51:14.604 INFO:tasks.rados.rados.0.smithi045.stdout:1692: writing smithi04569269-49 from 1974272 to 2482176 tid 3 2021-07-31T01:51:14.604 INFO:tasks.rados.rados.0.smithi045.stdout:1693: read oid 30 snap -1 2021-07-31T01:51:14.605 INFO:tasks.rados.rados.0.smithi045.stdout:1693: expect deleted 2021-07-31T01:51:14.605 INFO:tasks.rados.rados.0.smithi045.stdout:1694: snap_create 2021-07-31T01:51:14.605 INFO:tasks.rados.rados.0.smithi045.stdout:1689: finishing write tid 4 to smithi04569269-41 2021-07-31T01:51:14.606 INFO:tasks.rados.rados.0.smithi045.stdout:1689: finishing write tid 5 to smithi04569269-41 2021-07-31T01:51:14.606 INFO:tasks.rados.rados.0.smithi045.stdout:1689: finishing write tid 6 to smithi04569269-41 2021-07-31T01:51:14.606 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 41 v 962 (ObjNum 531 snap 158 seq_num 531) dirty exists 2021-07-31T01:51:14.606 INFO:tasks.rados.rados.0.smithi045.stdout:1689: left oid 41 (ObjNum 531 snap 158 seq_num 531) 2021-07-31T01:51:14.607 INFO:tasks.rados.rados.0.smithi045.stdout:1690: finishing copy_from to smithi04569269-29 2021-07-31T01:51:14.607 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 29 v 911 (ObjNum 525 snap 155 seq_num 525) dirty exists 2021-07-31T01:51:14.607 INFO:tasks.rados.rados.0.smithi045.stdout:1690: finishing copy_from racing read to smithi04569269-29 2021-07-31T01:51:14.608 INFO:tasks.rados.rados.0.smithi045.stdout:1692: finishing write tid 1 to smithi04569269-49 2021-07-31T01:51:14.608 INFO:tasks.rados.rados.0.smithi045.stdout:1692: finishing write tid 2 to smithi04569269-49 2021-07-31T01:51:14.609 INFO:tasks.rados.rados.0.smithi045.stdout:1692: finishing write tid 3 to smithi04569269-49 2021-07-31T01:51:14.609 INFO:tasks.rados.rados.0.smithi045.stdout:1692: finishing write tid 4 to smithi04569269-49 2021-07-31T01:51:14.611 INFO:tasks.rados.rados.0.smithi045.stdout:1692: finishing write tid 5 to smithi04569269-49 2021-07-31T01:51:14.611 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 49 v 1106 (ObjNum 532 snap 158 seq_num 532) dirty exists 2021-07-31T01:51:14.611 INFO:tasks.rados.rados.0.smithi045.stdout:1692: left oid 49 (ObjNum 532 snap 158 seq_num 532) 2021-07-31T01:51:15.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:14 smithi154 conmon[34553]: cluster 2021-07-31T01:51:13.583056+0000 mon.a (mon.0 2021-07-31T01:51:15.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:14 smithi154 conmon[34553]: ) 1020 : cluster [DBG] osdmap e374: 8 total, 8 up, 8 in 2021-07-31T01:51:15.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:14 smithi154 conmon[34553]: cluster 2021-07-31T01:51:13.847436+0000 mgr.x (mgr.24542) 168 : cluster [DBG] pgmap v341: 25 pgs: 25 active+clean; 124 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 2.0 MiB/s rd, 1.2 MiB/s wr, 9 op/s 2021-07-31T01:51:15.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:14 smithi045 conmon[33655]: cluster 2021-07-31T01:51:13.583056+0000 mon.a (mon.0) 1020 2021-07-31T01:51:15.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:14 smithi045 conmon[33655]: : cluster [DBG] osdmap e374: 8 total, 8 up, 8 in 2021-07-31T01:51:15.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:14 smithi045 conmon[33655]: cluster 2021-07-31T01:51:13.847436+0000 mgr.x (mgr.24542) 168 : cluster [DBG] pgmap v341: 25 pgs: 25 active+clean; 124 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 2.0 MiB/s rd, 1.2 MiB/s wr, 9 op/s 2021-07-31T01:51:15.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:14 smithi045 conmon[39718]: cluster 2021-07-31T01:51:13.583056+0000 mon.a ( 2021-07-31T01:51:15.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:14 smithi045 conmon[39718]: mon.0) 1020 : cluster [DBG] osdmap e374: 8 total, 8 up, 8 in 2021-07-31T01:51:15.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:14 smithi045 conmon[39718]: cluster 2021-07-31T01:51:13.847436+0000 2021-07-31T01:51:15.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:14 smithi045 conmon[39718]: mgr.x (mgr.24542) 168 : cluster [DBG] pgmap v341: 25 pgs: 25 active+clean; 124 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 2.0 MiB/s rd, 1.2 MiB/s wr, 9 op/s 2021-07-31T01:51:15.594 INFO:tasks.rados.rados.0.smithi045.stdout:1689: done (4 left) 2021-07-31T01:51:15.595 INFO:tasks.rados.rados.0.smithi045.stdout:1690: done (3 left) 2021-07-31T01:51:15.595 INFO:tasks.rados.rados.0.smithi045.stdout:1692: done (2 left) 2021-07-31T01:51:15.595 INFO:tasks.rados.rados.0.smithi045.stdout:1693: done (1 left) 2021-07-31T01:51:15.595 INFO:tasks.rados.rados.0.smithi045.stdout:1694: done (0 left) 2021-07-31T01:51:15.596 INFO:tasks.rados.rados.0.smithi045.stdout:1695: setattr oid 37 current snap is 159 2021-07-31T01:51:15.597 INFO:tasks.rados.rados.0.smithi045.stdout:1696: copy_from oid 34 from oid 33 current snap is 159 2021-07-31T01:51:15.597 INFO:tasks.rados.rados.0.smithi045.stdout:1697: copy_from oid 16 from oid 43 current snap is 159 2021-07-31T01:51:15.597 INFO:tasks.rados.rados.0.smithi045.stdout:1698: rollback oid 13 current snap is 159 2021-07-31T01:51:15.597 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 13 to 156 2021-07-31T01:51:15.598 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 24 current snap is 159 2021-07-31T01:51:15.598 INFO:tasks.rados.rados.0.smithi045.stdout:1699: seq_num 533 ranges {2981888=532480,3514368=630784,4145152=516096,4661248=57344} 2021-07-31T01:51:15.616 INFO:tasks.rados.rados.0.smithi045.stdout:1699: writing smithi04569269-24 from 2981888 to 3514368 tid 1 2021-07-31T01:51:15.620 INFO:tasks.rados.rados.0.smithi045.stdout:1699: writing smithi04569269-24 from 3514368 to 4145152 tid 2 2021-07-31T01:51:15.623 INFO:tasks.rados.rados.0.smithi045.stdout:1699: writing smithi04569269-24 from 4145152 to 4661248 tid 3 2021-07-31T01:51:15.624 INFO:tasks.rados.rados.0.smithi045.stdout:1699: writing smithi04569269-24 from 4661248 to 4718592 tid 4 2021-07-31T01:51:15.625 INFO:tasks.rados.rados.0.smithi045.stdout:1700: copy_from oid 38 from oid 11 current snap is 159 2021-07-31T01:51:15.625 INFO:tasks.rados.rados.0.smithi045.stdout:1698: finishing rollback tid 0 to smithi04569269-13 2021-07-31T01:51:15.625 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 37 v 410 (ObjNum 172 snap 41 seq_num 172) dirty exists 2021-07-31T01:51:15.625 INFO:tasks.rados.rados.0.smithi045.stdout:1698: finishing rollback tid 1 to smithi04569269-13 2021-07-31T01:51:15.626 INFO:tasks.rados.rados.0.smithi045.stdout:1698: finishing rollback tid 2 to smithi04569269-13 2021-07-31T01:51:15.626 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 13 v 634 (ObjNum 425 snap 126 seq_num 425) dirty exists 2021-07-31T01:51:15.626 INFO:tasks.rados.rados.0.smithi045.stdout:1697: finishing copy_from to smithi04569269-16 2021-07-31T01:51:15.626 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 16 v 1021 (ObjNum 523 snap 154 seq_num 523) dirty exists 2021-07-31T01:51:15.627 INFO:tasks.rados.rados.0.smithi045.stdout:1697: finishing copy_from racing read to smithi04569269-16 2021-07-31T01:51:15.627 INFO:tasks.rados.rados.0.smithi045.stdout:1699: finishing write tid 1 to smithi04569269-24 2021-07-31T01:51:15.627 INFO:tasks.rados.rados.0.smithi045.stdout:1699: finishing write tid 2 to smithi04569269-24 2021-07-31T01:51:15.627 INFO:tasks.rados.rados.0.smithi045.stdout:1695: done (5 left) 2021-07-31T01:51:15.628 INFO:tasks.rados.rados.0.smithi045.stdout:1697: done (4 left) 2021-07-31T01:51:15.628 INFO:tasks.rados.rados.0.smithi045.stdout:1698: done (3 left) 2021-07-31T01:51:15.628 INFO:tasks.rados.rados.0.smithi045.stdout:1701: rollback oid 18 current snap is 159 2021-07-31T01:51:15.628 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 18 to 158 2021-07-31T01:51:15.628 INFO:tasks.rados.rados.0.smithi045.stdout:1702: snap_create 2021-07-31T01:51:15.629 INFO:tasks.rados.rados.0.smithi045.stdout:1701: finishing rollback tid 1 to smithi04569269-18 2021-07-31T01:51:15.629 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 18 v 0 (ObjNum 0 snap 0 seq_num 0) dirty dne 2021-07-31T01:51:15.629 INFO:tasks.rados.rados.0.smithi045.stdout:1699: finishing write tid 3 to smithi04569269-24 2021-07-31T01:51:15.631 INFO:tasks.rados.rados.0.smithi045.stdout:1699: finishing write tid 4 to smithi04569269-24 2021-07-31T01:51:15.631 INFO:tasks.rados.rados.0.smithi045.stdout:1699: finishing write tid 5 to smithi04569269-24 2021-07-31T01:51:15.633 INFO:tasks.rados.rados.0.smithi045.stdout:1699: finishing write tid 6 to smithi04569269-24 2021-07-31T01:51:15.633 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 24 v 513 (ObjNum 533 snap 159 seq_num 533) dirty exists 2021-07-31T01:51:15.633 INFO:tasks.rados.rados.0.smithi045.stdout:1699: left oid 24 (ObjNum 533 snap 159 seq_num 533) 2021-07-31T01:51:15.641 INFO:tasks.rados.rados.0.smithi045.stdout:1696: finishing copy_from to smithi04569269-34 2021-07-31T01:51:15.641 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 34 v 1002 (ObjNum 399 snap 121 seq_num 399) dirty exists 2021-07-31T01:51:15.641 INFO:tasks.rados.rados.0.smithi045.stdout:1696: finishing copy_from racing read to smithi04569269-34 2021-07-31T01:51:15.647 INFO:tasks.rados.rados.0.smithi045.stdout:1700: finishing copy_from to smithi04569269-38 2021-07-31T01:51:15.648 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 38 v 1011 (ObjNum 522 snap 154 seq_num 522) dirty exists 2021-07-31T01:51:15.648 INFO:tasks.rados.rados.0.smithi045.stdout:1700: finishing copy_from racing read to smithi04569269-38 2021-07-31T01:51:16.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:15 smithi154 conmon[34553]: cluster 2021-07-31T01:51:14.588842+0000 mon.a (mon.0) 1021 : cluster [DBG] 2021-07-31T01:51:16.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:15 smithi154 conmon[34553]: osdmap e375: 8 total, 8 up, 8 in 2021-07-31T01:51:16.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:15 smithi045 conmon[39718]: cluster 2021-07-31T01:51:14.588842+0000 mon.a (mon.0 2021-07-31T01:51:16.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:15 smithi045 conmon[39718]: ) 1021 : cluster [DBG] osdmap e375: 8 total, 8 up, 8 in 2021-07-31T01:51:16.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:15 smithi045 conmon[33655]: cluster 2021-07-31T01:51:14. 2021-07-31T01:51:16.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:15 smithi045 conmon[33655]: 588842+0000 mon.a (mon.0) 1021 : cluster [DBG] osdmap e375: 8 total, 8 up, 8 in 2021-07-31T01:51:16.552 INFO:tasks.rados.rados.0.smithi045.stdout:1696: done (4 left) 2021-07-31T01:51:16.552 INFO:tasks.rados.rados.0.smithi045.stdout:1699: done (3 left) 2021-07-31T01:51:16.553 INFO:tasks.rados.rados.0.smithi045.stdout:1700: done (2 left) 2021-07-31T01:51:16.553 INFO:tasks.rados.rados.0.smithi045.stdout:1701: done (1 left) 2021-07-31T01:51:16.553 INFO:tasks.rados.rados.0.smithi045.stdout:1702: done (0 left) 2021-07-31T01:51:16.553 INFO:tasks.rados.rados.0.smithi045.stdout:1703: rmattr oid 9 current snap is 160 2021-07-31T01:51:16.554 INFO:tasks.rados.rados.0.smithi045.stdout:1704: copy_from oid 11 from oid 13 current snap is 160 2021-07-31T01:51:16.554 INFO:tasks.rados.rados.0.smithi045.stdout:1705: delete oid 35 current snap is 160 2021-07-31T01:51:16.556 INFO:tasks.rados.rados.0.smithi045.stdout:1705: done (2 left) 2021-07-31T01:51:16.556 INFO:tasks.rados.rados.0.smithi045.stdout:1706: rollback oid 49 current snap is 160 2021-07-31T01:51:16.557 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 49 to 156 2021-07-31T01:51:16.557 INFO:tasks.rados.rados.0.smithi045.stdout:1707: snap_create 2021-07-31T01:51:16.560 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 9 v 386 (ObjNum 530 snap 156 seq_num 530) dirty exists 2021-07-31T01:51:16.561 INFO:tasks.rados.rados.0.smithi045.stdout:1706: finishing rollback tid 0 to smithi04569269-49 2021-07-31T01:51:16.562 INFO:tasks.rados.rados.0.smithi045.stdout:1706: finishing rollback tid 1 to smithi04569269-49 2021-07-31T01:51:16.562 INFO:tasks.rados.rados.0.smithi045.stdout:1706: finishing rollback tid 2 to smithi04569269-49 2021-07-31T01:51:16.562 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 49 v 1111 (ObjNum 475 snap 140 seq_num 475) dirty exists 2021-07-31T01:51:16.590 INFO:tasks.rados.rados.0.smithi045.stdout:1704: finishing copy_from to smithi04569269-11 2021-07-31T01:51:16.590 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 11 v 634 (ObjNum 425 snap 126 seq_num 425) dirty exists 2021-07-31T01:51:16.591 INFO:tasks.rados.rados.0.smithi045.stdout:1704: finishing copy_from racing read to smithi04569269-11 2021-07-31T01:51:17.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:16 smithi154 conmon[34553]: cluster 2021-07-31T01:51:15.594062+0000 mon.a (mon.0) 1022 : cluster [DBG] osdmap e376: 8 total, 8 up, 8 in 2021-07-31T01:51:17.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:16 smithi154 conmon[34553]: cluster 2021-07-31T01:51:15.847905+0000 mgr.x (mgr.24542) 169 : cluster [DBG] pgmap v344: 25 pgs: 25 active+clean; 124 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 2.0 MiB/s rd, 1.2 MiB/s wr, 9 op/s 2021-07-31T01:51:17.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:16 smithi154 conmon[34553]: cluster 2021-07-31T01:51:16.551370+0000 mon.a (mon.0) 1023 : cluster [DBG] osdmap e377: 8 total, 8 up, 8 in 2021-07-31T01:51:17.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:16 smithi154 conmon[34553]: 2021-07-31T01:51:17.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:16 smithi045 conmon[33655]: cluster 2021-07-31 2021-07-31T01:51:17.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:16 smithi045 conmon[33655]: T01:51:15.594062+0000 mon.a (mon.0) 1022 : cluster [DBG] osdmap e376: 8 total, 8 up, 8 in 2021-07-31T01:51:17.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:16 smithi045 conmon[33655]: cluster 2021-07-31T01:51:15.847905+0000 mgr.x (mgr.24542) 169 : cluster 2021-07-31T01:51:17.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:16 smithi045 conmon[33655]: [DBG] pgmap v344: 25 pgs: 25 active+clean; 124 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 2.0 MiB/s rd, 1.2 MiB/s wr, 9 op/s 2021-07-31T01:51:17.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:16 smithi045 conmon[33655]: cluster 2021-07- 2021-07-31T01:51:17.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:16 smithi045 conmon[33655]: 31T01:51:16.551370+0000 mon.a (mon.0) 1023 : cluster [DBG] osdmap e377: 8 total, 8 up, 8 in 2021-07-31T01:51:17.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:16 smithi045 conmon[33655]: 2021-07-31T01:51:17.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:16 smithi045 conmon[39718]: cluster 2021-07-31T01:51:15.594062+0000 mon.a 2021-07-31T01:51:17.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:16 smithi045 conmon[39718]: (mon.0) 1022 : cluster [DBG] osdmap e376: 8 total, 8 up, 8 in 2021-07-31T01:51:17.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:16 smithi045 conmon[39718]: cluster 2021-07-31T01 2021-07-31T01:51:17.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:16 smithi045 conmon[39718]: :51:15.847905+0000 mgr.x (mgr.24542) 169 : cluster [DBG] pgmap v344: 25 pgs: 25 active+clean; 124 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 2.0 MiB/s rd, 1.2 MiB/s wr, 9 op/s 2021-07-31T01:51:17.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:16 smithi045 conmon[39718]: cluster 2021-07-31T01:51:16. 2021-07-31T01:51:17.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:16 smithi045 conmon[39718]: 551370+0000 mon.a (mon.0) 1023 : cluster [DBG] osdmap e377: 8 total, 8 up, 8 in 2021-07-31T01:51:17.560 INFO:tasks.rados.rados.0.smithi045.stdout:1703: done (3 left) 2021-07-31T01:51:17.561 INFO:tasks.rados.rados.0.smithi045.stdout:1704: done (2 left) 2021-07-31T01:51:17.561 INFO:tasks.rados.rados.0.smithi045.stdout:1706: done (1 left) 2021-07-31T01:51:17.561 INFO:tasks.rados.rados.0.smithi045.stdout:1707: done (0 left) 2021-07-31T01:51:17.561 INFO:tasks.rados.rados.0.smithi045.stdout:1708: read oid 30 snap 158 2021-07-31T01:51:17.562 INFO:tasks.rados.rados.0.smithi045.stdout:1708: expect deleted 2021-07-31T01:51:17.562 INFO:tasks.rados.rados.0.smithi045.stdout:1709: rollback oid 17 current snap is 161 2021-07-31T01:51:17.562 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 17 to 159 2021-07-31T01:51:17.562 INFO:tasks.rados.rados.0.smithi045.stdout:1710: read oid 14 snap 159 2021-07-31T01:51:17.563 INFO:tasks.rados.rados.0.smithi045.stdout:1710: expect (ObjNum 521 snap 153 seq_num 521) 2021-07-31T01:51:17.563 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 13 current snap is 161 2021-07-31T01:51:17.563 INFO:tasks.rados.rados.0.smithi045.stdout:1711: seq_num 535 ranges {9650176=565248,10215424=802816,11018240=401408} 2021-07-31T01:51:17.616 INFO:tasks.rados.rados.0.smithi045.stdout:1711: writing smithi04569269-13 from 9650176 to 10215424 tid 1 2021-07-31T01:51:17.620 INFO:tasks.rados.rados.0.smithi045.stdout:1711: writing smithi04569269-13 from 10215424 to 11018240 tid 2 2021-07-31T01:51:17.623 INFO:tasks.rados.rados.0.smithi045.stdout:1711: writing smithi04569269-13 from 11018240 to 11419648 tid 3 2021-07-31T01:51:17.623 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 23 current snap is 161 2021-07-31T01:51:17.624 INFO:tasks.rados.rados.0.smithi045.stdout:1712: seq_num 536 ranges {0=581632,581632=688128,1269760=516096} 2021-07-31T01:51:17.626 INFO:tasks.rados.rados.0.smithi045.stdout:1712: writing smithi04569269-23 from 0 to 581632 tid 1 2021-07-31T01:51:17.630 INFO:tasks.rados.rados.0.smithi045.stdout:1712: writing smithi04569269-23 from 581632 to 1269760 tid 2 2021-07-31T01:51:17.634 INFO:tasks.rados.rados.0.smithi045.stdout:1712: writing smithi04569269-23 from 1269760 to 1785856 tid 3 2021-07-31T01:51:17.634 INFO:tasks.rados.rados.0.smithi045.stdout:1713: rollback oid 32 current snap is 161 2021-07-31T01:51:17.634 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 32 to 159 2021-07-31T01:51:17.635 INFO:tasks.rados.rados.0.smithi045.stdout:1709: finishing rollback tid 0 to smithi04569269-17 2021-07-31T01:51:17.636 INFO:tasks.rados.rados.0.smithi045.stdout:1709: finishing rollback tid 1 to smithi04569269-17 2021-07-31T01:51:17.636 INFO:tasks.rados.rados.0.smithi045.stdout:1709: finishing rollback tid 2 to smithi04569269-17 2021-07-31T01:51:17.636 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 17 v 1020 (ObjNum 288 snap 0 seq_num 65) dirty exists 2021-07-31T01:51:17.637 INFO:tasks.rados.rados.0.smithi045.stdout:1710: expect (ObjNum 521 snap 153 seq_num 521) 2021-07-31T01:51:17.674 INFO:tasks.rados.rados.0.smithi045.stdout:1711: finishing write tid 1 to smithi04569269-13 2021-07-31T01:51:17.674 INFO:tasks.rados.rados.0.smithi045.stdout:1711: finishing write tid 2 to smithi04569269-13 2021-07-31T01:51:17.674 INFO:tasks.rados.rados.0.smithi045.stdout:1711: finishing write tid 3 to smithi04569269-13 2021-07-31T01:51:17.675 INFO:tasks.rados.rados.0.smithi045.stdout:1711: finishing write tid 4 to smithi04569269-13 2021-07-31T01:51:17.675 INFO:tasks.rados.rados.0.smithi045.stdout:1711: finishing write tid 5 to smithi04569269-13 2021-07-31T01:51:17.675 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 13 v 639 (ObjNum 535 snap 161 seq_num 535) dirty exists 2021-07-31T01:51:17.675 INFO:tasks.rados.rados.0.smithi045.stdout:1711: left oid 13 (ObjNum 535 snap 161 seq_num 535) 2021-07-31T01:51:17.675 INFO:tasks.rados.rados.0.smithi045.stdout:1712: finishing write tid 1 to smithi04569269-23 2021-07-31T01:51:17.676 INFO:tasks.rados.rados.0.smithi045.stdout:1712: finishing write tid 2 to smithi04569269-23 2021-07-31T01:51:17.676 INFO:tasks.rados.rados.0.smithi045.stdout:1713: finishing rollback tid 1 to smithi04569269-32 2021-07-31T01:51:17.676 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 32 v 0 (ObjNum 0 snap 0 seq_num 0) dirty dne 2021-07-31T01:51:17.677 INFO:tasks.rados.rados.0.smithi045.stdout:1712: finishing write tid 3 to smithi04569269-23 2021-07-31T01:51:17.677 INFO:tasks.rados.rados.0.smithi045.stdout:1712: finishing write tid 4 to smithi04569269-23 2021-07-31T01:51:17.678 INFO:tasks.rados.rados.0.smithi045.stdout:1712: finishing write tid 5 to smithi04569269-23 2021-07-31T01:51:17.678 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 23 v 1019 (ObjNum 536 snap 161 seq_num 536) dirty exists 2021-07-31T01:51:17.678 INFO:tasks.rados.rados.0.smithi045.stdout:1712: left oid 23 (ObjNum 536 snap 161 seq_num 536) 2021-07-31T01:51:17.678 INFO:tasks.rados.rados.0.smithi045.stdout:1708: done (5 left) 2021-07-31T01:51:17.679 INFO:tasks.rados.rados.0.smithi045.stdout:1709: done (4 left) 2021-07-31T01:51:17.679 INFO:tasks.rados.rados.0.smithi045.stdout:1710: done (3 left) 2021-07-31T01:51:17.679 INFO:tasks.rados.rados.0.smithi045.stdout:1711: done (2 left) 2021-07-31T01:51:17.679 INFO:tasks.rados.rados.0.smithi045.stdout:1712: done (1 left) 2021-07-31T01:51:17.679 INFO:tasks.rados.rados.0.smithi045.stdout:1713: done (0 left) 2021-07-31T01:51:17.680 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 15 current snap is 161 2021-07-31T01:51:17.680 INFO:tasks.rados.rados.0.smithi045.stdout:1714: seq_num 537 ranges {3481600=598016,4079616=573440,4653056=491520,5144576=139264} 2021-07-31T01:51:17.695 INFO:tasks.rados.rados.0.smithi045.stdout:1714: writing smithi04569269-15 from 3481600 to 4079616 tid 1 2021-07-31T01:51:17.698 INFO:tasks.rados.rados.0.smithi045.stdout:1714: writing smithi04569269-15 from 4079616 to 4653056 tid 2 2021-07-31T01:51:17.701 INFO:tasks.rados.rados.0.smithi045.stdout:1714: writing smithi04569269-15 from 4653056 to 5144576 tid 3 2021-07-31T01:51:17.702 INFO:tasks.rados.rados.0.smithi045.stdout:1714: writing smithi04569269-15 from 5144576 to 5283840 tid 4 2021-07-31T01:51:17.702 INFO:tasks.rados.rados.0.smithi045.stdout:1715: read oid 16 snap -1 2021-07-31T01:51:17.703 INFO:tasks.rados.rados.0.smithi045.stdout:1715: expect (ObjNum 523 snap 154 seq_num 523) 2021-07-31T01:51:17.703 INFO:tasks.rados.rados.0.smithi045.stdout:1716: snap_remove snap 156 2021-07-31T01:51:18.568 INFO:tasks.rados.rados.0.smithi045.stdout:1716: done (2 left) 2021-07-31T01:51:18.568 INFO:tasks.rados.rados.0.smithi045.stdout:1717: rollback oid 10 current snap is 161 2021-07-31T01:51:18.568 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 10 to 160 2021-07-31T01:51:18.569 INFO:tasks.rados.rados.0.smithi045.stdout:1714: finishing write tid 1 to smithi04569269-15 2021-07-31T01:51:18.569 INFO:tasks.rados.rados.0.smithi045.stdout:1714: finishing write tid 2 to smithi04569269-15 2021-07-31T01:51:18.569 INFO:tasks.rados.rados.0.smithi045.stdout:1714: finishing write tid 3 to smithi04569269-15 2021-07-31T01:51:18.570 INFO:tasks.rados.rados.0.smithi045.stdout:1714: finishing write tid 4 to smithi04569269-15 2021-07-31T01:51:18.570 INFO:tasks.rados.rados.0.smithi045.stdout:1714: finishing write tid 5 to smithi04569269-15 2021-07-31T01:51:18.570 INFO:tasks.rados.rados.0.smithi045.stdout:1714: finishing write tid 6 to smithi04569269-15 2021-07-31T01:51:18.571 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 15 v 416 (ObjNum 537 snap 161 seq_num 537) dirty exists 2021-07-31T01:51:18.571 INFO:tasks.rados.rados.0.smithi045.stdout:1714: left oid 15 (ObjNum 537 snap 161 seq_num 537) 2021-07-31T01:51:18.571 INFO:tasks.rados.rados.0.smithi045.stdout:1715: expect (ObjNum 523 snap 154 seq_num 523) 2021-07-31T01:51:18.684 INFO:tasks.rados.rados.0.smithi045.stdout:1717: finishing rollback tid 0 to smithi04569269-10 2021-07-31T01:51:18.685 INFO:tasks.rados.rados.0.smithi045.stdout:1717: finishing rollback tid 1 to smithi04569269-10 2021-07-31T01:51:18.685 INFO:tasks.rados.rados.0.smithi045.stdout:1717: finishing rollback tid 2 to smithi04569269-10 2021-07-31T01:51:18.686 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 10 v 1023 (ObjNum 475 snap 140 seq_num 475) dirty exists 2021-07-31T01:51:18.688 INFO:tasks.rados.rados.0.smithi045.stdout:1714: done (2 left) 2021-07-31T01:51:18.688 INFO:tasks.rados.rados.0.smithi045.stdout:1715: done (1 left) 2021-07-31T01:51:18.689 INFO:tasks.rados.rados.0.smithi045.stdout:1717: done (0 left) 2021-07-31T01:51:18.689 INFO:tasks.rados.rados.0.smithi045.stdout:1718: read oid 1 snap 159 2021-07-31T01:51:18.689 INFO:tasks.rados.rados.0.smithi045.stdout:1718: expect (ObjNum 465 snap 135 seq_num 465) 2021-07-31T01:51:18.689 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 43 current snap is 161 2021-07-31T01:51:18.690 INFO:tasks.rados.rados.0.smithi045.stdout:1719: seq_num 538 ranges {3014656=614400,3629056=450560,4079616=704512,4784128=49152} 2021-07-31T01:51:18.704 INFO:tasks.rados.rados.0.smithi045.stdout:1719: writing smithi04569269-43 from 3014656 to 3629056 tid 1 2021-07-31T01:51:18.706 INFO:tasks.rados.rados.0.smithi045.stdout:1719: writing smithi04569269-43 from 3629056 to 4079616 tid 2 2021-07-31T01:51:18.711 INFO:tasks.rados.rados.0.smithi045.stdout:1719: writing smithi04569269-43 from 4079616 to 4784128 tid 3 2021-07-31T01:51:18.711 INFO:tasks.rados.rados.0.smithi045.stdout:1719: writing smithi04569269-43 from 4784128 to 4833280 tid 4 2021-07-31T01:51:18.711 INFO:tasks.rados.rados.0.smithi045.stdout:1720: read oid 21 snap 159 2021-07-31T01:51:18.712 INFO:tasks.rados.rados.0.smithi045.stdout:1720: expect (ObjNum -1105773728 snap 21873 seq_num 3190180080) 2021-07-31T01:51:18.712 INFO:tasks.rados.rados.0.smithi045.stdout:1721: copy_from oid 14 from oid 9 current snap is 161 2021-07-31T01:51:18.712 INFO:tasks.rados.rados.0.smithi045.stdout:1718: expect (ObjNum 465 snap 135 seq_num 465) 2021-07-31T01:51:18.804 INFO:tasks.rados.rados.0.smithi045.stdout:1719: finishing write tid 1 to smithi04569269-43 2021-07-31T01:51:18.804 INFO:tasks.rados.rados.0.smithi045.stdout:1718: done (3 left) 2021-07-31T01:51:18.805 INFO:tasks.rados.rados.0.smithi045.stdout:1722: snap_remove snap 158 2021-07-31T01:51:19.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:18 smithi154 conmon[34553]: cluster 2021-07-31T01:51:17.559533+0000 mon.a (mon.0) 2021-07-31T01:51:19.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:18 smithi154 conmon[34553]: 1024 : cluster [DBG] osdmap e378: 8 total, 8 up, 8 in 2021-07-31T01:51:19.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:18 smithi154 conmon[34553]: cluster 2021-07-31T01:51:17.848417+0000 mgr.x (mgr.24542) 170 : cluster [DBG] pgmap v347: 25 pgs: 25 active+clean; 124 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail 2021-07-31T01:51:19.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:18 smithi045 conmon[33655]: cluster 2021-07-31T01:51:17.559533+0000 mon.a (mon.0) 1024 2021-07-31T01:51:19.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:18 smithi045 conmon[33655]: : cluster [DBG] osdmap e378: 8 total, 8 up, 8 in 2021-07-31T01:51:19.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:18 smithi045 conmon[33655]: cluster 2021-07-31T01:51:17.848417+0000 mgr.x (mgr.24542) 170 : cluster [DBG] pgmap v347: 25 pgs: 25 active+clean; 124 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail 2021-07-31T01:51:19.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:18 smithi045 conmon[39718]: cluster 2021-07-31T01:51:17.559533+0000 mon.a ( 2021-07-31T01:51:19.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:18 smithi045 conmon[39718]: mon.0) 1024 : cluster [DBG] osdmap e378: 8 total, 8 up, 8 in 2021-07-31T01:51:19.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:18 smithi045 conmon[39718]: cluster 2021-07-31T01:51 2021-07-31T01:51:19.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:18 smithi045 conmon[39718]: :17.848417+0000 mgr.x (mgr.24542) 170 : cluster [DBG] pgmap v347: 25 pgs: 25 active+clean; 124 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail 2021-07-31T01:51:19.569 INFO:tasks.rados.rados.0.smithi045.stdout:1722: done (3 left) 2021-07-31T01:51:19.569 INFO:tasks.rados.rados.0.smithi045.stdout:1723: snap_create 2021-07-31T01:51:19.570 INFO:tasks.rados.rados.0.smithi045.stdout:1719: finishing write tid 2 to smithi04569269-43 2021-07-31T01:51:19.570 INFO:tasks.rados.rados.0.smithi045.stdout:1720: expect (ObjNum -1105773728 snap 21873 seq_num 3190180080) 2021-07-31T01:51:19.574 INFO:tasks.rados.rados.0.smithi045.stdout:1719: finishing write tid 3 to smithi04569269-43 2021-07-31T01:51:19.575 INFO:tasks.rados.rados.0.smithi045.stdout:1719: finishing write tid 4 to smithi04569269-43 2021-07-31T01:51:19.575 INFO:tasks.rados.rados.0.smithi045.stdout:1719: finishing write tid 5 to smithi04569269-43 2021-07-31T01:51:19.575 INFO:tasks.rados.rados.0.smithi045.stdout:1719: finishing write tid 6 to smithi04569269-43 2021-07-31T01:51:19.575 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 43 v 1026 (ObjNum 538 snap 161 seq_num 538) dirty exists 2021-07-31T01:51:19.576 INFO:tasks.rados.rados.0.smithi045.stdout:1719: left oid 43 (ObjNum 538 snap 161 seq_num 538) 2021-07-31T01:51:19.576 INFO:tasks.rados.rados.0.smithi045.stdout:1721: finishing copy_from to smithi04569269-14 2021-07-31T01:51:19.576 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 14 v 386 (ObjNum 530 snap 156 seq_num 530) dirty exists 2021-07-31T01:51:19.577 INFO:tasks.rados.rados.0.smithi045.stdout:1721: finishing copy_from racing read to smithi04569269-14 2021-07-31T01:51:20.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:19 smithi154 conmon[34553]: cluster 2021-07-31T01:51:20.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:19 smithi154 conmon[34553]: 2021-07-31T01:51:20.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:19 smithi154 conmon[34553]: 2021-07-31T01:51:18.565931+0000 2021-07-31T01:51:20.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:19 smithi154 conmon[34553]: mon.a (mon.0) 1025 : cluster [DBG] osdmap e379: 8 total, 8 up, 8 in 2021-07-31T01:51:20.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:19 smithi045 conmon[33655]: cluster 2021-07-31T01:51:18.565931+0000 mon.a 2021-07-31T01:51:20.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:19 smithi045 conmon[33655]: (mon.0) 1025 : cluster [DBG] osdmap e379: 8 total, 8 up, 8 in 2021-07-31T01:51:20.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:19 smithi045 conmon[39718]: cluster 2021-07-31T01:51:20.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:19 smithi045 conmon[39718]: 2021-07-31T01:51:18 2021-07-31T01:51:20.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:19 smithi045 conmon[39718]: .565931+0000 mon.a (mon.0) 1025 : cluster [DBG] osdmap e379: 8 total, 8 up, 8 in 2021-07-31T01:51:20.587 INFO:tasks.rados.rados.0.smithi045.stdout:1719: done (3 left) 2021-07-31T01:51:20.587 INFO:tasks.rados.rados.0.smithi045.stdout:1720: done (2 left) 2021-07-31T01:51:20.587 INFO:tasks.rados.rados.0.smithi045.stdout:1721: done (1 left) 2021-07-31T01:51:20.588 INFO:tasks.rados.rados.0.smithi045.stdout:1723: done (0 left) 2021-07-31T01:51:20.588 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 29 current snap is 162 2021-07-31T01:51:20.588 INFO:tasks.rados.rados.0.smithi045.stdout:1724: seq_num 539 ranges {1826816=630784,2457600=737280,3194880=466944} 2021-07-31T01:51:20.600 INFO:tasks.rados.rados.0.smithi045.stdout:1724: writing smithi04569269-29 from 1826816 to 2457600 tid 1 2021-07-31T01:51:20.604 INFO:tasks.rados.rados.0.smithi045.stdout:1724: writing smithi04569269-29 from 2457600 to 3194880 tid 2 2021-07-31T01:51:20.607 INFO:tasks.rados.rados.0.smithi045.stdout:1724: writing smithi04569269-29 from 3194880 to 3661824 tid 3 2021-07-31T01:51:20.607 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 46 current snap is 162 2021-07-31T01:51:20.608 INFO:tasks.rados.rados.0.smithi045.stdout:1724: finishing write tid 1 to smithi04569269-29 2021-07-31T01:51:20.608 INFO:tasks.rados.rados.0.smithi045.stdout:1725: seq_num 540 ranges {10969088=647168,11616256=671744,12288000=532480} 2021-07-31T01:51:20.671 INFO:tasks.rados.rados.0.smithi045.stdout:1725: writing smithi04569269-46 from 10969088 to 11616256 tid 1 2021-07-31T01:51:20.674 INFO:tasks.rados.rados.0.smithi045.stdout:1725: writing smithi04569269-46 from 11616256 to 12288000 tid 2 2021-07-31T01:51:20.677 INFO:tasks.rados.rados.0.smithi045.stdout:1725: writing smithi04569269-46 from 12288000 to 12820480 tid 3 2021-07-31T01:51:20.678 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 25 current snap is 162 2021-07-31T01:51:20.678 INFO:tasks.rados.rados.0.smithi045.stdout:1724: finishing write tid 2 to smithi04569269-29 2021-07-31T01:51:20.678 INFO:tasks.rados.rados.0.smithi045.stdout:1726: seq_num 541 ranges {3588096=663552,4251648=557056,4808704=475136,5283840=172032} 2021-07-31T01:51:20.701 INFO:tasks.rados.rados.0.smithi045.stdout:1726: writing smithi04569269-25 from 3588096 to 4251648 tid 1 2021-07-31T01:51:20.704 INFO:tasks.rados.rados.0.smithi045.stdout:1726: writing smithi04569269-25 from 4251648 to 4808704 tid 2 2021-07-31T01:51:20.706 INFO:tasks.rados.rados.0.smithi045.stdout:1726: writing smithi04569269-25 from 4808704 to 5283840 tid 3 2021-07-31T01:51:20.708 INFO:tasks.rados.rados.0.smithi045.stdout:1726: writing smithi04569269-25 from 5283840 to 5455872 tid 4 2021-07-31T01:51:20.709 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 2 current snap is 162 2021-07-31T01:51:20.709 INFO:tasks.rados.rados.0.smithi045.stdout:1724: finishing write tid 3 to smithi04569269-29 2021-07-31T01:51:20.709 INFO:tasks.rados.rados.0.smithi045.stdout:1724: finishing write tid 4 to smithi04569269-29 2021-07-31T01:51:20.709 INFO:tasks.rados.rados.0.smithi045.stdout:1724: finishing write tid 5 to smithi04569269-29 2021-07-31T01:51:20.710 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 29 v 915 (ObjNum 539 snap 162 seq_num 539) dirty exists 2021-07-31T01:51:20.710 INFO:tasks.rados.rados.0.smithi045.stdout:1724: left oid 29 (ObjNum 539 snap 162 seq_num 539) 2021-07-31T01:51:20.710 INFO:tasks.rados.rados.0.smithi045.stdout:1725: finishing write tid 1 to smithi04569269-46 2021-07-31T01:51:20.710 INFO:tasks.rados.rados.0.smithi045.stdout:1725: finishing write tid 2 to smithi04569269-46 2021-07-31T01:51:20.710 INFO:tasks.rados.rados.0.smithi045.stdout:1727: seq_num 542 ranges {0=679936,679936=442368,1122304=679936,1802240=81920} 2021-07-31T01:51:20.711 INFO:tasks.rados.rados.0.smithi045.stdout:1727: writing smithi04569269-2 from 0 to 679936 tid 1 2021-07-31T01:51:20.714 INFO:tasks.rados.rados.0.smithi045.stdout:1727: writing smithi04569269-2 from 679936 to 1122304 tid 2 2021-07-31T01:51:20.718 INFO:tasks.rados.rados.0.smithi045.stdout:1727: writing smithi04569269-2 from 1122304 to 1802240 tid 3 2021-07-31T01:51:20.719 INFO:tasks.rados.rados.0.smithi045.stdout:1727: writing smithi04569269-2 from 1802240 to 1884160 tid 4 2021-07-31T01:51:20.720 INFO:tasks.rados.rados.0.smithi045.stdout:1724: done (3 left) 2021-07-31T01:51:20.720 INFO:tasks.rados.rados.0.smithi045.stdout:1728: rmattr oid 18 current snap is 162 2021-07-31T01:51:20.720 INFO:tasks.rados.rados.0.smithi045.stdout:1728: done (3 left) 2021-07-31T01:51:20.720 INFO:tasks.rados.rados.0.smithi045.stdout:1729: setattr oid 33 current snap is 162 2021-07-31T01:51:20.720 INFO:tasks.rados.rados.0.smithi045.stdout:1725: finishing write tid 3 to smithi04569269-46 2021-07-31T01:51:20.721 INFO:tasks.rados.rados.0.smithi045.stdout:1725: finishing write tid 4 to smithi04569269-46 2021-07-31T01:51:20.721 INFO:tasks.rados.rados.0.smithi045.stdout:1725: finishing write tid 5 to smithi04569269-46 2021-07-31T01:51:20.721 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 46 v 280 (ObjNum 540 snap 162 seq_num 540) dirty exists 2021-07-31T01:51:20.721 INFO:tasks.rados.rados.0.smithi045.stdout:1725: left oid 46 (ObjNum 540 snap 162 seq_num 540) 2021-07-31T01:51:20.722 INFO:tasks.rados.rados.0.smithi045.stdout:1726: finishing write tid 1 to smithi04569269-25 2021-07-31T01:51:20.722 INFO:tasks.rados.rados.0.smithi045.stdout:1726: finishing write tid 2 to smithi04569269-25 2021-07-31T01:51:20.722 INFO:tasks.rados.rados.0.smithi045.stdout:1726: finishing write tid 3 to smithi04569269-25 2021-07-31T01:51:20.722 INFO:tasks.rados.rados.0.smithi045.stdout:1726: finishing write tid 4 to smithi04569269-25 2021-07-31T01:51:20.723 INFO:tasks.rados.rados.0.smithi045.stdout:1726: finishing write tid 5 to smithi04569269-25 2021-07-31T01:51:20.723 INFO:tasks.rados.rados.0.smithi045.stdout:1726: finishing write tid 6 to smithi04569269-25 2021-07-31T01:51:20.723 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 25 v 288 (ObjNum 541 snap 162 seq_num 541) dirty exists 2021-07-31T01:51:20.723 INFO:tasks.rados.rados.0.smithi045.stdout:1726: left oid 25 (ObjNum 541 snap 162 seq_num 541) 2021-07-31T01:51:20.723 INFO:tasks.rados.rados.0.smithi045.stdout:1727: finishing write tid 1 to smithi04569269-2 2021-07-31T01:51:20.724 INFO:tasks.rados.rados.0.smithi045.stdout:1727: finishing write tid 2 to smithi04569269-2 2021-07-31T01:51:20.724 INFO:tasks.rados.rados.0.smithi045.stdout:1727: finishing write tid 3 to smithi04569269-2 2021-07-31T01:51:20.725 INFO:tasks.rados.rados.0.smithi045.stdout:1725: done (3 left) 2021-07-31T01:51:20.725 INFO:tasks.rados.rados.0.smithi045.stdout:1726: done (2 left) 2021-07-31T01:51:20.725 INFO:tasks.rados.rados.0.smithi045.stdout:1730: rollback oid 24 current snap is 162 2021-07-31T01:51:20.725 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 24 to 161 2021-07-31T01:51:20.726 INFO:tasks.rados.rados.0.smithi045.stdout:1727: finishing write tid 4 to smithi04569269-2 2021-07-31T01:51:20.726 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 41 current snap is 162 2021-07-31T01:51:20.726 INFO:tasks.rados.rados.0.smithi045.stdout:1731: seq_num 543 ranges {1703936=704512,2408448=729088,3137536=475136} 2021-07-31T01:51:20.735 INFO:tasks.rados.rados.0.smithi045.stdout:1731: writing smithi04569269-41 from 1703936 to 2408448 tid 1 2021-07-31T01:51:20.738 INFO:tasks.rados.rados.0.smithi045.stdout:1731: writing smithi04569269-41 from 2408448 to 3137536 tid 2 2021-07-31T01:51:20.742 INFO:tasks.rados.rados.0.smithi045.stdout:1731: writing smithi04569269-41 from 3137536 to 3612672 tid 3 2021-07-31T01:51:20.742 INFO:tasks.rados.rados.0.smithi045.stdout:1732: rollback oid 43 current snap is 162 2021-07-31T01:51:20.742 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 43 to 161 2021-07-31T01:51:20.743 INFO:tasks.rados.rados.0.smithi045.stdout:1727: finishing write tid 5 to smithi04569269-2 2021-07-31T01:51:20.743 INFO:tasks.rados.rados.0.smithi045.stdout:1727: finishing write tid 6 to smithi04569269-2 2021-07-31T01:51:20.743 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 2 v 391 (ObjNum 542 snap 162 seq_num 542) dirty exists 2021-07-31T01:51:20.743 INFO:tasks.rados.rados.0.smithi045.stdout:1727: left oid 2 (ObjNum 542 snap 162 seq_num 542) 2021-07-31T01:51:20.744 INFO:tasks.rados.rados.0.smithi045.stdout:1730: finishing rollback tid 0 to smithi04569269-24 2021-07-31T01:51:20.744 INFO:tasks.rados.rados.0.smithi045.stdout:1727: done (4 left) 2021-07-31T01:51:20.744 INFO:tasks.rados.rados.0.smithi045.stdout:1733: copy_from oid 38 from oid 20 current snap is 162 2021-07-31T01:51:20.744 INFO:tasks.rados.rados.0.smithi045.stdout:1730: finishing rollback tid 1 to smithi04569269-24 2021-07-31T01:51:20.744 INFO:tasks.rados.rados.0.smithi045.stdout:1730: finishing rollback tid 2 to smithi04569269-24 2021-07-31T01:51:20.745 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 24 v 516 (ObjNum 533 snap 159 seq_num 533) dirty exists 2021-07-31T01:51:20.745 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 33 v 1020 (ObjNum 399 snap 121 seq_num 399) dirty exists 2021-07-31T01:51:20.745 INFO:tasks.rados.rados.0.smithi045.stdout:1731: finishing write tid 1 to smithi04569269-41 2021-07-31T01:51:20.745 INFO:tasks.rados.rados.0.smithi045.stdout:1729: done (4 left) 2021-07-31T01:51:20.746 INFO:tasks.rados.rados.0.smithi045.stdout:1730: done (3 left) 2021-07-31T01:51:20.746 INFO:tasks.rados.rados.0.smithi045.stdout:1734: delete oid 22 current snap is 162 2021-07-31T01:51:20.746 INFO:tasks.rados.rados.0.smithi045.stdout:1731: finishing write tid 2 to smithi04569269-41 2021-07-31T01:51:20.748 INFO:tasks.rados.rados.0.smithi045.stdout:1732: finishing rollback tid 0 to smithi04569269-43 2021-07-31T01:51:20.748 INFO:tasks.rados.rados.0.smithi045.stdout:1732: finishing rollback tid 1 to smithi04569269-43 2021-07-31T01:51:20.748 INFO:tasks.rados.rados.0.smithi045.stdout:1734: done (3 left) 2021-07-31T01:51:20.748 INFO:tasks.rados.rados.0.smithi045.stdout:1735: rmattr oid 27 current snap is 162 2021-07-31T01:51:20.749 INFO:tasks.rados.rados.0.smithi045.stdout:1735: done (3 left) 2021-07-31T01:51:20.749 INFO:tasks.rados.rados.0.smithi045.stdout:1736: copy_from oid 49 from oid 48 current snap is 162 2021-07-31T01:51:20.749 INFO:tasks.rados.rados.0.smithi045.stdout:1737: setattr oid 24 current snap is 162 2021-07-31T01:51:20.749 INFO:tasks.rados.rados.0.smithi045.stdout:1732: finishing rollback tid 2 to smithi04569269-43 2021-07-31T01:51:20.749 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 43 v 1029 (ObjNum 538 snap 161 seq_num 538) dirty exists 2021-07-31T01:51:20.750 INFO:tasks.rados.rados.0.smithi045.stdout:1731: finishing write tid 3 to smithi04569269-41 2021-07-31T01:51:20.750 INFO:tasks.rados.rados.0.smithi045.stdout:1731: finishing write tid 4 to smithi04569269-41 2021-07-31T01:51:20.750 INFO:tasks.rados.rados.0.smithi045.stdout:1731: finishing write tid 5 to smithi04569269-41 2021-07-31T01:51:20.750 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 41 v 966 (ObjNum 543 snap 162 seq_num 543) dirty exists 2021-07-31T01:51:20.751 INFO:tasks.rados.rados.0.smithi045.stdout:1731: left oid 41 (ObjNum 543 snap 162 seq_num 543) 2021-07-31T01:51:20.751 INFO:tasks.rados.rados.0.smithi045.stdout:1731: done (4 left) 2021-07-31T01:51:20.751 INFO:tasks.rados.rados.0.smithi045.stdout:1732: done (3 left) 2021-07-31T01:51:20.751 INFO:tasks.rados.rados.0.smithi045.stdout:1738: snap_create 2021-07-31T01:51:20.752 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 24 v 517 (ObjNum 533 snap 159 seq_num 533) dirty exists 2021-07-31T01:51:20.753 INFO:tasks.rados.rados.0.smithi045.stdout:1733: finishing copy_from to smithi04569269-38 2021-07-31T01:51:20.753 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 38 v 1054 (ObjNum 517 snap 153 seq_num 517) dirty exists 2021-07-31T01:51:20.754 INFO:tasks.rados.rados.0.smithi045.stdout:1733: finishing copy_from racing read to smithi04569269-38 2021-07-31T01:51:20.754 INFO:tasks.rados.rados.0.smithi045.stdout:1736: finishing copy_from to smithi04569269-49 2021-07-31T01:51:20.755 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 49 v 911 (ObjNum 525 snap 155 seq_num 525) dirty exists 2021-07-31T01:51:20.755 INFO:tasks.rados.rados.0.smithi045.stdout:1736: finishing copy_from racing read to smithi04569269-49 2021-07-31T01:51:21.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:20 smithi154 conmon[34553]: cluster 2021-07-31T01:51:19.568076+0000 mon.a (mon.0) 1026 : cluster 2021-07-31T01:51:21.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:20 smithi154 conmon[34553]: [DBG] osdmap e380: 8 total, 8 up, 8 in 2021-07-31T01:51:21.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:20 smithi154 conmon[34553]: cluster 2021-07-31T01:51:19.848807+0000 mgr.x (mgr.24542) 171 : cluster [DBG] pgmap v350: 25 pgs: 25 active+clean; 157 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 4.9 MiB/s rd, 9.6 MiB/s wr, 25 op/s 2021-07-31T01:51:21.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:20 smithi045 conmon[33655]: cluster 2021-07-31T01:51:19.568076 2021-07-31T01:51:21.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:20 smithi045 conmon[33655]: +0000 mon.a (mon.0) 1026 : cluster [DBG] osdmap e380: 8 total, 8 up, 8 in 2021-07-31T01:51:21.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:20 smithi045 conmon[33655]: cluster 2021-07-31T01:51:19.848807+0000 mgr.x (mgr.24542) 171 : cluster [DBG] pgmap v350: 25 pgs: 25 active+clean; 157 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 4.9 MiB/s rd, 9.6 MiB/s wr, 25 op/s 2021-07-31T01:51:21.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:20 smithi045 conmon[39718]: cluster 2021-07-31T01:51:19. 2021-07-31T01:51:21.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:20 smithi045 conmon[39718]: 568076+0000 mon.a (mon.0) 1026 : cluster [DBG] osdmap e380: 8 total, 8 up, 8 in 2021-07-31T01:51:21.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:20 smithi045 conmon[39718]: cluster 2021-07-31T01: 2021-07-31T01:51:21.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:20 smithi045 conmon[39718]: 51:19.848807+0000 mgr.x 2021-07-31T01:51:21.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:20 smithi045 conmon[39718]: (mgr.24542) 171 : cluster 2021-07-31T01:51:21.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:20 smithi045 conmon[39718]: [DBG] pgmap v350: 25 pgs: 25 active+clean; 157 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 4.9 MiB/s rd, 9.6 MiB/s wr, 25 op/s 2021-07-31T01:51:21.553 INFO:tasks.rados.rados.0.smithi045.stdout:1733: done (3 left) 2021-07-31T01:51:21.554 INFO:tasks.rados.rados.0.smithi045.stdout:1736: done (2 left) 2021-07-31T01:51:21.554 INFO:tasks.rados.rados.0.smithi045.stdout:1737: done (1 left) 2021-07-31T01:51:21.555 INFO:tasks.rados.rados.0.smithi045.stdout:1738: done (0 left) 2021-07-31T01:51:21.555 INFO:tasks.rados.rados.0.smithi045.stdout:1739: delete oid 24 current snap is 163 2021-07-31T01:51:21.562 INFO:tasks.rados.rados.0.smithi045.stdout:1739: done (0 left) 2021-07-31T01:51:21.562 INFO:tasks.rados.rados.0.smithi045.stdout:1740: rmattr oid 14 current snap is 163 2021-07-31T01:51:21.563 INFO:tasks.rados.rados.0.smithi045.stdout:1741: delete oid 7 current snap is 163 2021-07-31T01:51:21.563 INFO:tasks.rados.rados.0.smithi045.stdout:1741: done (1 left) 2021-07-31T01:51:21.564 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 29 current snap is 163 2021-07-31T01:51:21.564 INFO:tasks.rados.rados.0.smithi045.stdout:1742: seq_num 547 ranges {3661824=770048,4431872=663552,5095424=466944,5562368=73728} 2021-07-31T01:51:21.588 INFO:tasks.rados.rados.0.smithi045.stdout:1742: writing smithi04569269-29 from 3661824 to 4431872 tid 1 2021-07-31T01:51:21.591 INFO:tasks.rados.rados.0.smithi045.stdout:1742: writing smithi04569269-29 from 4431872 to 5095424 tid 2 2021-07-31T01:51:21.594 INFO:tasks.rados.rados.0.smithi045.stdout:1742: writing smithi04569269-29 from 5095424 to 5562368 tid 3 2021-07-31T01:51:21.595 INFO:tasks.rados.rados.0.smithi045.stdout:1742: writing smithi04569269-29 from 5562368 to 5636096 tid 4 2021-07-31T01:51:21.596 INFO:tasks.rados.rados.0.smithi045.stdout:1743: snap_create 2021-07-31T01:51:21.596 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 14 v 1030 (ObjNum 530 snap 156 seq_num 530) dirty exists 2021-07-31T01:51:21.596 INFO:tasks.rados.rados.0.smithi045.stdout:1742: finishing write tid 1 to smithi04569269-29 2021-07-31T01:51:21.596 INFO:tasks.rados.rados.0.smithi045.stdout:1742: finishing write tid 2 to smithi04569269-29 2021-07-31T01:51:21.599 INFO:tasks.rados.rados.0.smithi045.stdout:1742: finishing write tid 3 to smithi04569269-29 2021-07-31T01:51:21.599 INFO:tasks.rados.rados.0.smithi045.stdout:1742: finishing write tid 4 to smithi04569269-29 2021-07-31T01:51:21.599 INFO:tasks.rados.rados.0.smithi045.stdout:1742: finishing write tid 5 to smithi04569269-29 2021-07-31T01:51:21.600 INFO:tasks.rados.rados.0.smithi045.stdout:1742: finishing write tid 6 to smithi04569269-29 2021-07-31T01:51:21.600 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 29 v 920 (ObjNum 547 snap 163 seq_num 547) dirty exists 2021-07-31T01:51:21.601 INFO:tasks.rados.rados.0.smithi045.stdout:1742: left oid 29 (ObjNum 547 snap 163 seq_num 547) 2021-07-31T01:51:22.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:21 smithi154 conmon[34553]: cluster 2021-07-31T01:51:20.586332+0000 mon.a (mon.0) 1027 : cluster [DBG] osdmap e381: 8 total, 8 up, 8 in 2021-07-31T01:51:22.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:21 smithi154 conmon[34553]: cluster 2021-07-31T01:51:21.553120+0000 mon.a (mon.0) 1028 : cluster [DBG] osdmap e382: 8 total, 8 up, 8 in 2021-07-31T01:51:22.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:21 smithi045 conmon[33655]: cluster 2021-07-31T01:51:20.586332+0000 2021-07-31T01:51:22.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:21 smithi045 conmon[33655]: mon.a (mon.0) 1027 : cluster [DBG] osdmap e381: 8 total, 8 up, 8 in 2021-07-31T01:51:22.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:21 smithi045 conmon[33655]: cluster 2021-07-31T01:51:21.553120+0000 mon.a (mon.0) 1028 2021-07-31T01:51:22.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:21 smithi045 conmon[33655]: : cluster [DBG] osdmap e382: 8 total, 8 up, 8 in 2021-07-31T01:51:22.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:21 smithi045 conmon[39718]: cluster 2021-07-31T01:51:20.586332+0000 2021-07-31T01:51:22.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:21 smithi045 conmon[39718]: mon.a (mon.0) 1027 : cluster [DBG] osdmap e381: 8 total, 8 up, 8 in 2021-07-31T01:51:22.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:21 smithi045 conmon[39718]: cluster 2021- 2021-07-31T01:51:22.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:21 smithi045 conmon[39718]: 07-31T01:51:21.553120+0000 2021-07-31T01:51:22.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:21 smithi045 conmon[39718]: mon.a (mon.0) 1028 : cluster 2021-07-31T01:51:22.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:21 smithi045 conmon[39718]: [DBG] osdmap e382: 8 total, 8 up, 8 in 2021-07-31T01:51:22.593 INFO:tasks.rados.rados.0.smithi045.stdout:1740: done (2 left) 2021-07-31T01:51:22.594 INFO:tasks.rados.rados.0.smithi045.stdout:1742: done (1 left) 2021-07-31T01:51:22.594 INFO:tasks.rados.rados.0.smithi045.stdout:1743: done (0 left) 2021-07-31T01:51:22.594 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 45 current snap is 164 2021-07-31T01:51:22.595 INFO:tasks.rados.rados.0.smithi045.stdout:1744: seq_num 548 ranges {5259264=786432,6045696=598016,6643712=606208} 2021-07-31T01:51:22.626 INFO:tasks.rados.rados.0.smithi045.stdout:1744: writing smithi04569269-45 from 5259264 to 6045696 tid 1 2021-07-31T01:51:22.630 INFO:tasks.rados.rados.0.smithi045.stdout:1744: writing smithi04569269-45 from 6045696 to 6643712 tid 2 2021-07-31T01:51:22.633 INFO:tasks.rados.rados.0.smithi045.stdout:1744: writing smithi04569269-45 from 6643712 to 7249920 tid 3 2021-07-31T01:51:22.633 INFO:tasks.rados.rados.0.smithi045.stdout:1745: snap_remove snap 161 2021-07-31T01:51:23.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:22 smithi154 conmon[34553]: cluster 2021-07-31T01:51:21.849217+0000 mgr.x ( 2021-07-31T01:51:23.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:22 smithi154 conmon[34553]: mgr.24542) 172 : cluster [DBG] pgmap v353: 25 pgs: 25 active+clean; 157 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 4.9 MiB/s rd, 9.6 MiB/s wr, 25 op/s 2021-07-31T01:51:23.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:22 smithi045 conmon[33655]: cluster 2021-07-31T01:51:21.849217 2021-07-31T01:51:23.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:22 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 172 : cluster [DBG] pgmap v353: 25 pgs: 25 active+clean; 157 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 4.9 MiB/s rd, 9.6 MiB/s wr, 25 op/s 2021-07-31T01:51:23.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:22 smithi045 conmon[39718]: cluster 2021-07-31T01:51:21.849217+0000 mgr.x (mgr. 2021-07-31T01:51:23.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:22 smithi045 conmon[39718]: 24542) 172 : cluster [DBG] pgmap v353: 25 pgs: 25 active+clean; 157 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 4.9 MiB/s rd, 9.6 MiB/s wr, 25 op/s 2021-07-31T01:51:23.600 INFO:tasks.rados.rados.0.smithi045.stdout:1745: done (1 left) 2021-07-31T01:51:23.600 INFO:tasks.rados.rados.0.smithi045.stdout:1746: copy_from oid 25 from oid 40 current snap is 164 2021-07-31T01:51:23.600 INFO:tasks.rados.rados.0.smithi045.stdout:1744: finishing write tid 1 to smithi04569269-45 2021-07-31T01:51:23.601 INFO:tasks.rados.rados.0.smithi045.stdout:1744: finishing write tid 2 to smithi04569269-45 2021-07-31T01:51:23.601 INFO:tasks.rados.rados.0.smithi045.stdout:1744: finishing write tid 3 to smithi04569269-45 2021-07-31T01:51:23.601 INFO:tasks.rados.rados.0.smithi045.stdout:1744: finishing write tid 4 to smithi04569269-45 2021-07-31T01:51:23.601 INFO:tasks.rados.rados.0.smithi045.stdout:1744: finishing write tid 5 to smithi04569269-45 2021-07-31T01:51:23.601 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 45 v 1027 (ObjNum 548 snap 164 seq_num 548) dirty exists 2021-07-31T01:51:23.602 INFO:tasks.rados.rados.0.smithi045.stdout:1744: left oid 45 (ObjNum 548 snap 164 seq_num 548) 2021-07-31T01:51:23.602 INFO:tasks.rados.rados.0.smithi045.stdout:1744: done (1 left) 2021-07-31T01:51:23.602 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 32 current snap is 164 2021-07-31T01:51:23.602 INFO:tasks.rados.rados.0.smithi045.stdout:1747: seq_num 549 ranges {0=802816,802816=483328,1286144=434176,1720320=286720} 2021-07-31T01:51:23.605 INFO:tasks.rados.rados.0.smithi045.stdout:1747: writing smithi04569269-32 from 0 to 802816 tid 1 2021-07-31T01:51:23.607 INFO:tasks.rados.rados.0.smithi045.stdout:1747: writing smithi04569269-32 from 802816 to 1286144 tid 2 2021-07-31T01:51:23.609 INFO:tasks.rados.rados.0.smithi045.stdout:1747: writing smithi04569269-32 from 1286144 to 1720320 tid 3 2021-07-31T01:51:23.613 INFO:tasks.rados.rados.0.smithi045.stdout:1747: writing smithi04569269-32 from 1720320 to 2007040 tid 4 2021-07-31T01:51:23.613 INFO:tasks.rados.rados.0.smithi045.stdout:1746: finishing copy_from to smithi04569269-25 2021-07-31T01:51:23.614 INFO:tasks.rados.rados.0.smithi045.stdout:1746: got expected ENOENT (src dne) 2021-07-31T01:51:23.614 INFO:tasks.rados.rados.0.smithi045.stdout:1748: snap_remove snap 163 2021-07-31T01:51:23.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:23 smithi045 conmon[39718]: cluster 2021-07-31T01:51:22.592685+0000 mon.a ( 2021-07-31T01:51:23.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:23 smithi045 conmon[39718]: mon.0) 1029 : cluster [DBG] osdmap e383: 8 total, 8 up, 8 in 2021-07-31T01:51:23.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:51:23 smithi045 conmon[61210]: level=error ts=2021-07-31T01:51:23.515Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:51:23.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:51:23 smithi045 conmon[61210]: level=error ts=2021-07-31T01:51:23.515Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:51:23.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:51:23 smithi045 conmon[61210]: level=error ts=2021-07-31T01:51:23.516Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T01:51:23.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:23 smithi045 conmon[33655]: cluster 2021-07-31T01:51:22.592685+0000 mon.a (mon.0 2021-07-31T01:51:23.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:23 smithi045 conmon[33655]: ) 1029 : cluster [DBG] osdmap e383: 8 total, 8 up, 8 in 2021-07-31T01:51:24.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:23 smithi154 conmon[34553]: cluster 2021-07-31T01:51:22.592685+0000 mon.a (mon.0) 1029 : cluster [DBG] osdmap e383: 8 total, 8 up, 8 in 2021-07-31T01:51:24.606 INFO:tasks.rados.rados.0.smithi045.stdout:1748: done (2 left) 2021-07-31T01:51:24.606 INFO:tasks.rados.rados.0.smithi045.stdout:1749: delete oid 19 current snap is 164 2021-07-31T01:51:24.606 INFO:tasks.rados.rados.0.smithi045.stdout:1746: finishing copy_from racing read to smithi04569269-25 2021-07-31T01:51:24.607 INFO:tasks.rados.rados.0.smithi045.stdout:1747: finishing write tid 1 to smithi04569269-32 2021-07-31T01:51:24.607 INFO:tasks.rados.rados.0.smithi045.stdout:1747: finishing write tid 2 to smithi04569269-32 2021-07-31T01:51:24.607 INFO:tasks.rados.rados.0.smithi045.stdout:1747: finishing write tid 3 to smithi04569269-32 2021-07-31T01:51:24.608 INFO:tasks.rados.rados.0.smithi045.stdout:1747: finishing write tid 4 to smithi04569269-32 2021-07-31T01:51:24.608 INFO:tasks.rados.rados.0.smithi045.stdout:1747: finishing write tid 5 to smithi04569269-32 2021-07-31T01:51:24.608 INFO:tasks.rados.rados.0.smithi045.stdout:1747: finishing write tid 6 to smithi04569269-32 2021-07-31T01:51:24.608 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 32 v 405 (ObjNum 549 snap 164 seq_num 549) dirty exists 2021-07-31T01:51:24.609 INFO:tasks.rados.rados.0.smithi045.stdout:1747: left oid 32 (ObjNum 549 snap 164 seq_num 549) 2021-07-31T01:51:24.610 INFO:tasks.rados.rados.0.smithi045.stdout:1746: done (2 left) 2021-07-31T01:51:24.611 INFO:tasks.rados.rados.0.smithi045.stdout:1747: done (1 left) 2021-07-31T01:51:24.611 INFO:tasks.rados.rados.0.smithi045.stdout:1749: done (0 left) 2021-07-31T01:51:24.611 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 23 current snap is 164 2021-07-31T01:51:24.611 INFO:tasks.rados.rados.0.smithi045.stdout:1750: seq_num 551 ranges {1785856=434176,2220032=647168,2867200=450560,3317760=507904} 2021-07-31T01:51:24.621 INFO:tasks.rados.rados.0.smithi045.stdout:1750: writing smithi04569269-23 from 1785856 to 2220032 tid 1 2021-07-31T01:51:24.625 INFO:tasks.rados.rados.0.smithi045.stdout:1750: writing smithi04569269-23 from 2220032 to 2867200 tid 2 2021-07-31T01:51:24.627 INFO:tasks.rados.rados.0.smithi045.stdout:1750: writing smithi04569269-23 from 2867200 to 3317760 tid 3 2021-07-31T01:51:24.631 INFO:tasks.rados.rados.0.smithi045.stdout:1750: writing smithi04569269-23 from 3317760 to 3825664 tid 4 2021-07-31T01:51:24.631 INFO:tasks.rados.rados.0.smithi045.stdout:1751: snap_create 2021-07-31T01:51:24.631 INFO:tasks.rados.rados.0.smithi045.stdout:1750: finishing write tid 1 to smithi04569269-23 2021-07-31T01:51:24.631 INFO:tasks.rados.rados.0.smithi045.stdout:1750: finishing write tid 2 to smithi04569269-23 2021-07-31T01:51:24.632 INFO:tasks.rados.rados.0.smithi045.stdout:1750: finishing write tid 3 to smithi04569269-23 2021-07-31T01:51:24.633 INFO:tasks.rados.rados.0.smithi045.stdout:1750: finishing write tid 4 to smithi04569269-23 2021-07-31T01:51:24.634 INFO:tasks.rados.rados.0.smithi045.stdout:1750: finishing write tid 5 to smithi04569269-23 2021-07-31T01:51:24.635 INFO:tasks.rados.rados.0.smithi045.stdout:1750: finishing write tid 6 to smithi04569269-23 2021-07-31T01:51:24.635 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 23 v 1059 (ObjNum 551 snap 164 seq_num 551) dirty exists 2021-07-31T01:51:24.635 INFO:tasks.rados.rados.0.smithi045.stdout:1750: left oid 23 (ObjNum 551 snap 164 seq_num 551) 2021-07-31T01:51:25.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:24 smithi154 conmon[34553]: cluster 2021-07-31T01:51:23.598052+0000 mon.a (mon.0) 1030 : cluster [DBG] osdmap e384: 8 total, 8 up, 8 in 2021-07-31T01:51:25.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:24 smithi154 conmon[34553]: 2021-07-31T01:51:25.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:24 smithi154 conmon[34553]: cluster 2021-07-31T01:51:23.849849+0000 mgr.x (mgr.24542) 173 : cluster [DBG] pgmap v356: 25 pgs: 25 active+clean; 165 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 4.4 MiB/s wr, 24 op/s 2021-07-31T01:51:25.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:24 smithi045 conmon[39718]: cluster 2021-07-31T01:51:23.598052+0000 mon.a (mon.0) 1030 2021-07-31T01:51:25.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:24 smithi045 conmon[39718]: : cluster [DBG] osdmap e384: 8 total, 8 up, 8 in 2021-07-31T01:51:25.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:24 smithi045 conmon[39718]: cluster 2021-07-31T01:51:23.849849+0000 mgr.x (mgr 2021-07-31T01:51:25.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:24 smithi045 conmon[39718]: .24542) 173 : cluster [DBG] pgmap v356: 25 pgs: 25 active+clean; 165 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 4.4 MiB/s wr, 24 op/s 2021-07-31T01:51:25.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:24 smithi045 conmon[33655]: cluster 2021-07-31T01:51:23.598052+0000 2021-07-31T01:51:25.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:24 smithi045 conmon[33655]: mon.a (mon.0) 1030 : cluster [DBG] osdmap e384: 8 total, 8 up, 8 in 2021-07-31T01:51:25.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:24 smithi045 conmon[33655]: cluster 2021-07-31T01:51:23.849849+0000 mgr.x (mgr.24542) 173 : 2021-07-31T01:51:25.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:24 smithi045 conmon[33655]: cluster [DBG] pgmap v356: 25 pgs: 25 active+clean; 165 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 4.4 MiB/s wr, 24 op/s 2021-07-31T01:51:25.612 INFO:tasks.rados.rados.0.smithi045.stdout:1750: done (1 left) 2021-07-31T01:51:25.613 INFO:tasks.rados.rados.0.smithi045.stdout:1751: done (0 left) 2021-07-31T01:51:25.613 INFO:tasks.rados.rados.0.smithi045.stdout:1752: read oid 33 snap -1 2021-07-31T01:51:25.614 INFO:tasks.rados.rados.0.smithi045.stdout:1752: expect (ObjNum 399 snap 121 seq_num 399) 2021-07-31T01:51:25.614 INFO:tasks.rados.rados.0.smithi045.stdout:1753: read oid 5 snap -1 2021-07-31T01:51:25.614 INFO:tasks.rados.rados.0.smithi045.stdout:1753: expect deleted 2021-07-31T01:51:25.615 INFO:tasks.rados.rados.0.smithi045.stdout:1754: rmattr oid 16 current snap is 165 2021-07-31T01:51:25.615 INFO:tasks.rados.rados.0.smithi045.stdout:1754: done (2 left) 2021-07-31T01:51:25.616 INFO:tasks.rados.rados.0.smithi045.stdout:1755: read oid 44 snap -1 2021-07-31T01:51:25.616 INFO:tasks.rados.rados.0.smithi045.stdout:1755: expect (ObjNum 490 snap 146 seq_num 490) 2021-07-31T01:51:25.616 INFO:tasks.rados.rados.0.smithi045.stdout:1756: rollback oid 10 current snap is 165 2021-07-31T01:51:25.617 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 10 to 159 2021-07-31T01:51:25.617 INFO:tasks.rados.rados.0.smithi045.stdout:1757: snap_create 2021-07-31T01:51:25.618 INFO:tasks.rados.rados.0.smithi045.stdout:1756: finishing rollback tid 0 to smithi04569269-10 2021-07-31T01:51:25.619 INFO:tasks.rados.rados.0.smithi045.stdout:1756: finishing rollback tid 1 to smithi04569269-10 2021-07-31T01:51:25.620 INFO:tasks.rados.rados.0.smithi045.stdout:1756: finishing rollback tid 2 to smithi04569269-10 2021-07-31T01:51:25.620 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 10 v 1030 (ObjNum 475 snap 140 seq_num 475) dirty exists 2021-07-31T01:51:25.654 INFO:tasks.rados.rados.0.smithi045.stdout:1752: expect (ObjNum 399 snap 121 seq_num 399) 2021-07-31T01:51:25.899 INFO:tasks.rados.rados.0.smithi045.stdout:1755: expect (ObjNum 490 snap 146 seq_num 490) 2021-07-31T01:51:26.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:25 smithi154 conmon[34553]: cluster 2021-07-31T01:51:24.604989+0000 mon.a (mon.0) 1031 : cluster [DBG] osdmap e385: 8 total, 8 up, 8 in 2021-07-31T01:51:26.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:25 smithi045 conmon[39718]: cluster 2021-07-31T01:51:24.604989+0000 mon.a (mon.0) 1031 2021-07-31T01:51:26.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:25 smithi045 conmon[39718]: : cluster [DBG] osdmap e385: 8 total, 8 up, 8 in 2021-07-31T01:51:26.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:25 smithi045 conmon[33655]: cluster 2021-07-31T01:51:24.604989+0000 mon.a (mon.0) 2021-07-31T01:51:26.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:25 smithi045 conmon[33655]: 1031 : cluster [DBG] osdmap e385: 8 total, 8 up, 8 in 2021-07-31T01:51:26.554 INFO:tasks.rados.rados.0.smithi045.stdout:1752: done (4 left) 2021-07-31T01:51:26.555 INFO:tasks.rados.rados.0.smithi045.stdout:1753: done (3 left) 2021-07-31T01:51:26.555 INFO:tasks.rados.rados.0.smithi045.stdout:1755: done (2 left) 2021-07-31T01:51:26.555 INFO:tasks.rados.rados.0.smithi045.stdout:1756: done (1 left) 2021-07-31T01:51:26.555 INFO:tasks.rados.rados.0.smithi045.stdout:1757: done (0 left) 2021-07-31T01:51:26.556 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 41 current snap is 166 2021-07-31T01:51:26.556 INFO:tasks.rados.rados.0.smithi045.stdout:1758: seq_num 552 ranges {3612672=450560,4063232=532480,4595712=655360,5251072=417792} 2021-07-31T01:51:26.576 INFO:tasks.rados.rados.0.smithi045.stdout:1758: writing smithi04569269-41 from 3612672 to 4063232 tid 1 2021-07-31T01:51:26.579 INFO:tasks.rados.rados.0.smithi045.stdout:1758: writing smithi04569269-41 from 4063232 to 4595712 tid 2 2021-07-31T01:51:26.582 INFO:tasks.rados.rados.0.smithi045.stdout:1758: writing smithi04569269-41 from 4595712 to 5251072 tid 3 2021-07-31T01:51:26.585 INFO:tasks.rados.rados.0.smithi045.stdout:1758: writing smithi04569269-41 from 5251072 to 5668864 tid 4 2021-07-31T01:51:26.585 INFO:tasks.rados.rados.0.smithi045.stdout:1759: delete oid 38 current snap is 166 2021-07-31T01:51:26.585 INFO:tasks.rados.rados.0.smithi045.stdout:1758: finishing write tid 1 to smithi04569269-41 2021-07-31T01:51:26.586 INFO:tasks.rados.rados.0.smithi045.stdout:1758: finishing write tid 2 to smithi04569269-41 2021-07-31T01:51:26.586 INFO:tasks.rados.rados.0.smithi045.stdout:1758: finishing write tid 3 to smithi04569269-41 2021-07-31T01:51:26.588 INFO:tasks.rados.rados.0.smithi045.stdout:1758: finishing write tid 4 to smithi04569269-41 2021-07-31T01:51:26.589 INFO:tasks.rados.rados.0.smithi045.stdout:1758: finishing write tid 5 to smithi04569269-41 2021-07-31T01:51:26.589 INFO:tasks.rados.rados.0.smithi045.stdout:1758: finishing write tid 6 to smithi04569269-41 2021-07-31T01:51:26.590 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 41 v 972 (ObjNum 552 snap 166 seq_num 552) dirty exists 2021-07-31T01:51:26.590 INFO:tasks.rados.rados.0.smithi045.stdout:1758: left oid 41 (ObjNum 552 snap 166 seq_num 552) 2021-07-31T01:51:26.591 INFO:tasks.rados.rados.0.smithi045.stdout:1758: done (1 left) 2021-07-31T01:51:26.591 INFO:tasks.rados.rados.0.smithi045.stdout:1759: done (0 left) 2021-07-31T01:51:26.591 INFO:tasks.rados.rados.0.smithi045.stdout:1760: read oid 10 snap -1 2021-07-31T01:51:26.591 INFO:tasks.rados.rados.0.smithi045.stdout:1760: expect (ObjNum 475 snap 140 seq_num 475) 2021-07-31T01:51:26.592 INFO:tasks.rados.rados.0.smithi045.stdout:1761: delete oid 2 current snap is 166 2021-07-31T01:51:26.594 INFO:tasks.rados.rados.0.smithi045.stdout:1761: done (1 left) 2021-07-31T01:51:26.594 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 42 current snap is 166 2021-07-31T01:51:26.595 INFO:tasks.rados.rados.0.smithi045.stdout:1762: seq_num 555 ranges {557056=499712,1056768=638976,1695744=425984,2121728=540672} 2021-07-31T01:51:26.600 INFO:tasks.rados.rados.0.smithi045.stdout:1762: writing smithi04569269-42 from 557056 to 1056768 tid 1 2021-07-31T01:51:26.603 INFO:tasks.rados.rados.0.smithi045.stdout:1762: writing smithi04569269-42 from 1056768 to 1695744 tid 2 2021-07-31T01:51:26.606 INFO:tasks.rados.rados.0.smithi045.stdout:1762: writing smithi04569269-42 from 1695744 to 2121728 tid 3 2021-07-31T01:51:26.609 INFO:tasks.rados.rados.0.smithi045.stdout:1762: writing smithi04569269-42 from 2121728 to 2662400 tid 4 2021-07-31T01:51:26.609 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 20 current snap is 166 2021-07-31T01:51:26.610 INFO:tasks.rados.rados.0.smithi045.stdout:1763: seq_num 556 ranges {1466368=516096,1982464=524288,2506752=638976,3145728=442368} 2021-07-31T01:51:26.620 INFO:tasks.rados.rados.0.smithi045.stdout:1763: writing smithi04569269-20 from 1466368 to 1982464 tid 1 2021-07-31T01:51:26.623 INFO:tasks.rados.rados.0.smithi045.stdout:1763: writing smithi04569269-20 from 1982464 to 2506752 tid 2 2021-07-31T01:51:26.627 INFO:tasks.rados.rados.0.smithi045.stdout:1763: writing smithi04569269-20 from 2506752 to 3145728 tid 3 2021-07-31T01:51:26.629 INFO:tasks.rados.rados.0.smithi045.stdout:1763: writing smithi04569269-20 from 3145728 to 3588096 tid 4 2021-07-31T01:51:26.630 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 24 current snap is 166 2021-07-31T01:51:26.630 INFO:tasks.rados.rados.0.smithi045.stdout:1760: expect (ObjNum 475 snap 140 seq_num 475) 2021-07-31T01:51:26.650 INFO:tasks.rados.rados.0.smithi045.stdout:1762: finishing write tid 1 to smithi04569269-42 2021-07-31T01:51:26.651 INFO:tasks.rados.rados.0.smithi045.stdout:1762: finishing write tid 2 to smithi04569269-42 2021-07-31T01:51:26.651 INFO:tasks.rados.rados.0.smithi045.stdout:1762: finishing write tid 3 to smithi04569269-42 2021-07-31T01:51:26.651 INFO:tasks.rados.rados.0.smithi045.stdout:1762: finishing write tid 4 to smithi04569269-42 2021-07-31T01:51:26.652 INFO:tasks.rados.rados.0.smithi045.stdout:1762: finishing write tid 5 to smithi04569269-42 2021-07-31T01:51:26.652 INFO:tasks.rados.rados.0.smithi045.stdout:1762: finishing write tid 6 to smithi04569269-42 2021-07-31T01:51:26.652 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 42 v 1035 (ObjNum 555 snap 166 seq_num 555) dirty exists 2021-07-31T01:51:26.652 INFO:tasks.rados.rados.0.smithi045.stdout:1762: left oid 42 (ObjNum 555 snap 166 seq_num 555) 2021-07-31T01:51:26.652 INFO:tasks.rados.rados.0.smithi045.stdout:1763: finishing write tid 1 to smithi04569269-20 2021-07-31T01:51:26.653 INFO:tasks.rados.rados.0.smithi045.stdout:1763: finishing write tid 2 to smithi04569269-20 2021-07-31T01:51:26.653 INFO:tasks.rados.rados.0.smithi045.stdout:1763: finishing write tid 3 to smithi04569269-20 2021-07-31T01:51:26.653 INFO:tasks.rados.rados.0.smithi045.stdout:1763: finishing write tid 4 to smithi04569269-20 2021-07-31T01:51:26.653 INFO:tasks.rados.rados.0.smithi045.stdout:1763: finishing write tid 5 to smithi04569269-20 2021-07-31T01:51:26.654 INFO:tasks.rados.rados.0.smithi045.stdout:1763: finishing write tid 6 to smithi04569269-20 2021-07-31T01:51:26.654 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 20 v 1127 (ObjNum 556 snap 166 seq_num 556) dirty exists 2021-07-31T01:51:26.654 INFO:tasks.rados.rados.0.smithi045.stdout:1763: left oid 20 (ObjNum 556 snap 166 seq_num 556) 2021-07-31T01:51:26.654 INFO:tasks.rados.rados.0.smithi045.stdout:1764: seq_num 557 ranges {0=532480,532480=802816,1335296=794624,2129920=8192} 2021-07-31T01:51:26.655 INFO:tasks.rados.rados.0.smithi045.stdout:1764: writing smithi04569269-24 from 0 to 532480 tid 1 2021-07-31T01:51:26.657 INFO:tasks.rados.rados.0.smithi045.stdout:1764: writing smithi04569269-24 from 532480 to 1335296 tid 2 2021-07-31T01:51:26.663 INFO:tasks.rados.rados.0.smithi045.stdout:1764: writing smithi04569269-24 from 1335296 to 2129920 tid 3 2021-07-31T01:51:26.664 INFO:tasks.rados.rados.0.smithi045.stdout:1764: writing smithi04569269-24 from 2129920 to 2138112 tid 4 2021-07-31T01:51:26.664 INFO:tasks.rados.rados.0.smithi045.stdout:1760: done (3 left) 2021-07-31T01:51:26.664 INFO:tasks.rados.rados.0.smithi045.stdout:1762: done (2 left) 2021-07-31T01:51:26.664 INFO:tasks.rados.rados.0.smithi045.stdout:1763: done (1 left) 2021-07-31T01:51:26.664 INFO:tasks.rados.rados.0.smithi045.stdout:1765: read oid 5 snap -1 2021-07-31T01:51:26.665 INFO:tasks.rados.rados.0.smithi045.stdout:1765: expect deleted 2021-07-31T01:51:26.665 INFO:tasks.rados.rados.0.smithi045.stdout:1766: copy_from oid 11 from oid 46 current snap is 166 2021-07-31T01:51:26.666 INFO:tasks.rados.rados.0.smithi045.stdout:1764: finishing write tid 1 to smithi04569269-24 2021-07-31T01:51:26.666 INFO:tasks.rados.rados.0.smithi045.stdout:1764: finishing write tid 2 to smithi04569269-24 2021-07-31T01:51:26.667 INFO:tasks.rados.rados.0.smithi045.stdout:1767: read oid 35 snap -1 2021-07-31T01:51:26.667 INFO:tasks.rados.rados.0.smithi045.stdout:1767: expect deleted 2021-07-31T01:51:26.667 INFO:tasks.rados.rados.0.smithi045.stdout:1768: read oid 1 snap -1 2021-07-31T01:51:26.667 INFO:tasks.rados.rados.0.smithi045.stdout:1768: expect (ObjNum 465 snap 135 seq_num 465) 2021-07-31T01:51:26.668 INFO:tasks.rados.rados.0.smithi045.stdout:1769: delete oid 28 current snap is 166 2021-07-31T01:51:26.669 INFO:tasks.rados.rados.0.smithi045.stdout:1764: finishing write tid 3 to smithi04569269-24 2021-07-31T01:51:26.669 INFO:tasks.rados.rados.0.smithi045.stdout:1764: finishing write tid 4 to smithi04569269-24 2021-07-31T01:51:26.669 INFO:tasks.rados.rados.0.smithi045.stdout:1764: finishing write tid 5 to smithi04569269-24 2021-07-31T01:51:26.670 INFO:tasks.rados.rados.0.smithi045.stdout:1765: done (5 left) 2021-07-31T01:51:26.670 INFO:tasks.rados.rados.0.smithi045.stdout:1767: done (4 left) 2021-07-31T01:51:26.670 INFO:tasks.rados.rados.0.smithi045.stdout:1769: done (3 left) 2021-07-31T01:51:26.670 INFO:tasks.rados.rados.0.smithi045.stdout:1770: rmattr oid 31 current snap is 166 2021-07-31T01:51:26.670 INFO:tasks.rados.rados.0.smithi045.stdout:1771: delete oid 10 current snap is 166 2021-07-31T01:51:26.671 INFO:tasks.rados.rados.0.smithi045.stdout:1764: finishing write tid 6 to smithi04569269-24 2021-07-31T01:51:26.671 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 24 v 523 (ObjNum 557 snap 166 seq_num 557) dirty exists 2021-07-31T01:51:26.671 INFO:tasks.rados.rados.0.smithi045.stdout:1764: left oid 24 (ObjNum 557 snap 166 seq_num 557) 2021-07-31T01:51:26.672 INFO:tasks.rados.rados.0.smithi045.stdout:1764: done (4 left) 2021-07-31T01:51:26.673 INFO:tasks.rados.rados.0.smithi045.stdout:1771: done (3 left) 2021-07-31T01:51:26.673 INFO:tasks.rados.rados.0.smithi045.stdout:1772: snap_remove snap 159 2021-07-31T01:51:27.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:26 smithi154 conmon[34553]: cluster 2021-07-31T01:51:25.610020+0000 mon.a (mon.0) 1032 : cluster [DBG] osdmap e386: 8 total, 8 up, 8 in 2021-07-31T01:51:27.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:26 smithi154 conmon[34553]: cluster 2021-07-31T01:51:25.850376+0000 mgr.x (mgr.24542) 174 : cluster [DBG] pgmap v359: 25 pgs: 25 active+clean; 165 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 4.4 MiB/s wr, 24 op/s 2021-07-31T01:51:27.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:26 smithi154 conmon[34553]: cluster 2021-07-31T01:51:26.553367+0000 mon.a (mon.0) 1033 : cluster [DBG] osdmap e387: 8 total, 8 up, 8 in 2021-07-31T01:51:27.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:26 smithi045 conmon[33655]: cluster 2021-07-31T01:51:25. 2021-07-31T01:51:27.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:26 smithi045 conmon[33655]: 610020+0000 mon.a (mon.0) 1032 : cluster [DBG] osdmap e386: 8 total, 8 up, 8 in 2021-07-31T01:51:27.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:26 smithi045 conmon[33655]: cluster 2021 2021-07-31T01:51:27.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:26 smithi045 conmon[33655]: -07-31T01:51:25.850376+0000 mgr.x (mgr. 2021-07-31T01:51:27.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:26 smithi045 conmon[33655]: 24542) 174 : cluster [DBG] pgmap v359: 25 pgs: 25 active+clean; 165 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 4.4 MiB/s wr, 24 op/s 2021-07-31T01:51:27.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:26 smithi045 conmon[33655]: cluster 2021-07-31 2021-07-31T01:51:27.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:26 smithi045 conmon[33655]: T01:51:26.553367+0000 mon.a ( 2021-07-31T01:51:27.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:26 smithi045 conmon[33655]: mon.0) 1033 : cluster [DBG] osdmap e387: 8 total, 8 up, 8 in 2021-07-31T01:51:27.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:26 smithi045 conmon[39718]: cluster 2021-07-31T01:51:27.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:26 smithi045 conmon[39718]: 2021-07-31T01:51:25.610020 2021-07-31T01:51:27.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:26 smithi045 conmon[39718]: +0000 mon.a (mon.0) 1032 : cluster [DBG] osdmap e386: 8 total, 8 up, 8 in 2021-07-31T01:51:27.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:26 smithi045 conmon[39718]: cluster 2021-07-31T01:51:27.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:26 smithi045 conmon[39718]: 2021-07-31T01:51:25 2021-07-31T01:51:27.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:26 smithi045 conmon[39718]: .850376+0000 mgr.x (mgr.24542) 174 : cluster [DBG] pgmap v359: 25 pgs: 25 active+clean; 165 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 4.4 MiB/s wr, 24 op/s 2021-07-31T01:51:27.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:26 smithi045 conmon[39718]: 2021-07-31T01:51:27.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:26 smithi045 conmon[39718]: cluster 2021-07-31T01: 2021-07-31T01:51:27.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:26 smithi045 conmon[39718]: 51:26.553367+0000 mon.a (mon 2021-07-31T01:51:27.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:26 smithi045 conmon[39718]: .0) 1033 : cluster [DBG] osdmap e387: 8 total, 8 up, 8 in 2021-07-31T01:51:27.619 INFO:tasks.rados.rados.0.smithi045.stdout:1768: expect (ObjNum 465 snap 135 seq_num 465) 2021-07-31T01:51:27.718 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 31 v 1129 (ObjNum 490 snap 146 seq_num 490) dirty exists 2021-07-31T01:51:27.718 INFO:tasks.rados.rados.0.smithi045.stdout:1766: finishing copy_from to smithi04569269-11 2021-07-31T01:51:27.719 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 11 v 280 (ObjNum 540 snap 162 seq_num 540) dirty exists 2021-07-31T01:51:27.719 INFO:tasks.rados.rados.0.smithi045.stdout:1766: finishing copy_from racing read to smithi04569269-11 2021-07-31T01:51:27.719 INFO:tasks.rados.rados.0.smithi045.stdout:1766: done (3 left) 2021-07-31T01:51:27.719 INFO:tasks.rados.rados.0.smithi045.stdout:1768: done (2 left) 2021-07-31T01:51:27.720 INFO:tasks.rados.rados.0.smithi045.stdout:1770: done (1 left) 2021-07-31T01:51:27.720 INFO:tasks.rados.rados.0.smithi045.stdout:1772: done (0 left) 2021-07-31T01:51:27.720 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 23 current snap is 166 2021-07-31T01:51:27.720 INFO:tasks.rados.rados.0.smithi045.stdout:1773: seq_num 560 ranges {3825664=581632,4407296=458752,4866048=614400,5480448=532480} 2021-07-31T01:51:27.742 INFO:tasks.rados.rados.0.smithi045.stdout:1773: writing smithi04569269-23 from 3825664 to 4407296 tid 1 2021-07-31T01:51:27.744 INFO:tasks.rados.rados.0.smithi045.stdout:1773: writing smithi04569269-23 from 4407296 to 4866048 tid 2 2021-07-31T01:51:27.747 INFO:tasks.rados.rados.0.smithi045.stdout:1773: writing smithi04569269-23 from 4866048 to 5480448 tid 3 2021-07-31T01:51:27.751 INFO:tasks.rados.rados.0.smithi045.stdout:1773: writing smithi04569269-23 from 5480448 to 6012928 tid 4 2021-07-31T01:51:27.751 INFO:tasks.rados.rados.0.smithi045.stdout:1774: read oid 15 snap -1 2021-07-31T01:51:27.751 INFO:tasks.rados.rados.0.smithi045.stdout:1774: expect (ObjNum 537 snap 161 seq_num 537) 2021-07-31T01:51:27.752 INFO:tasks.rados.rados.0.smithi045.stdout:1775: setattr oid 36 current snap is 166 2021-07-31T01:51:27.752 INFO:tasks.rados.rados.0.smithi045.stdout:1773: finishing write tid 1 to smithi04569269-23 2021-07-31T01:51:27.752 INFO:tasks.rados.rados.0.smithi045.stdout:1773: finishing write tid 2 to smithi04569269-23 2021-07-31T01:51:27.752 INFO:tasks.rados.rados.0.smithi045.stdout:1773: finishing write tid 3 to smithi04569269-23 2021-07-31T01:51:27.753 INFO:tasks.rados.rados.0.smithi045.stdout:1776: snap_create 2021-07-31T01:51:27.754 INFO:tasks.rados.rados.0.smithi045.stdout:1773: finishing write tid 4 to smithi04569269-23 2021-07-31T01:51:27.755 INFO:tasks.rados.rados.0.smithi045.stdout:1773: finishing write tid 5 to smithi04569269-23 2021-07-31T01:51:27.755 INFO:tasks.rados.rados.0.smithi045.stdout:1773: finishing write tid 6 to smithi04569269-23 2021-07-31T01:51:27.756 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 23 v 1065 (ObjNum 560 snap 166 seq_num 560) dirty exists 2021-07-31T01:51:27.756 INFO:tasks.rados.rados.0.smithi045.stdout:1773: left oid 23 (ObjNum 560 snap 166 seq_num 560) 2021-07-31T01:51:27.756 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 36 v 1137 (ObjNum 475 snap 140 seq_num 475) dirty exists 2021-07-31T01:51:27.765 INFO:tasks.rados.rados.0.smithi045.stdout:1774: expect (ObjNum 537 snap 161 seq_num 537) 2021-07-31T01:51:28.626 INFO:tasks.rados.rados.0.smithi045.stdout:1773: done (3 left) 2021-07-31T01:51:28.626 INFO:tasks.rados.rados.0.smithi045.stdout:1774: done (2 left) 2021-07-31T01:51:28.627 INFO:tasks.rados.rados.0.smithi045.stdout:1775: done (1 left) 2021-07-31T01:51:28.627 INFO:tasks.rados.rados.0.smithi045.stdout:1776: done (0 left) 2021-07-31T01:51:28.627 INFO:tasks.rados.rados.0.smithi045.stdout:1777: delete oid 2 current snap is 167 2021-07-31T01:51:28.628 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:28 smithi045 conmon[39718]: cluster 2021-07-31T01:51:27.618470+0000 mon.a ( 2021-07-31T01:51:28.628 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:28 smithi045 conmon[39718]: mon.0) 1034 : cluster [DBG] osdmap e388: 8 total, 8 up, 8 in 2021-07-31T01:51:28.628 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:28 smithi045 conmon[39718]: cluster 2021-07-31T 2021-07-31T01:51:28.628 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:28 smithi045 conmon[39718]: 01:51:27.850951+0000 mgr.x (mgr.24542) 175 : cluster [DBG] pgmap v362: 25 pgs: 25 active+clean; 165 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2021-07-31T01:51:28.629 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:28 smithi045 conmon[39718]: audit 2021-07-31T01:51:28.629 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:28 smithi045 conmon[39718]: 2021-07-31T01:51:27.956341+0000 mon.b 2021-07-31T01:51:28.629 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:28 smithi045 conmon[33655]: cluster 2021-07-31T01:51:27.618470+0000 2021-07-31T01:51:28.629 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:28 smithi045 conmon[33655]: mon.a (mon.0) 1034 : cluster [DBG] osdmap e388: 8 total, 8 up, 8 in 2021-07-31T01:51:28.630 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:28 smithi045 conmon[33655]: cluster 2021-07-31T01:51:27.850951+0000 mgr.x (mgr 2021-07-31T01:51:28.630 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:28 smithi045 conmon[33655]: .24542) 175 : cluster [DBG] pgmap v362: 25 pgs: 25 active+clean; 165 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2021-07-31T01:51:28.630 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:28 smithi045 conmon[33655]: audit 2021-07-31T01:51:27.956341+0000 mon.b (mon.2) 82 : 2021-07-31T01:51:28.630 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:28 smithi045 conmon[33655]: audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:51:28.631 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:28 smithi045 conmon[33655]: audit 2021-07-31T01 2021-07-31T01:51:28.631 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:28 smithi045 conmon[33655]: :51:27.956810+0000 mon.a (mon.0) 1035 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:51:28.631 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:28 smithi045 conmon[33655]: audit 2021-07-31T01:51:28.631 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:28 smithi045 conmon[33655]: 2021-07-31T01:51:27.967732+0000 mon.b (mon.2) 83 : audit [INF] 2021-07-31T01:51:28.631 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:28 smithi045 conmon[33655]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T01:51:28.632 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:28 smithi045 conmon[33655]: audit 2021-07-31T01:51:28.632 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:28 smithi045 conmon[33655]: 2021-07-31T01:51:27.968082+0000 mon.a (mon.0) 1036 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T01:51:28.632 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:28 smithi045 conmon[33655]: audit 2021-07-31T01:51:28.632 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:28 smithi045 conmon[33655]: 2021-07-31T01:51:28.590979+0000 2021-07-31T01:51:28.633 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:28 smithi045 conmon[33655]: mon.b (mon.2) 84 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:51:28.633 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:28 smithi045 conmon[33655]: audit 2021 2021-07-31T01:51:28.633 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:28 smithi045 conmon[33655]: -07-31T01:51:28.592480+0000 2021-07-31T01:51:28.633 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:28 smithi045 conmon[33655]: mon.b (mon.2) 85 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:51:28.633 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:28 smithi045 conmon[33655]: audit 2021-07-31T01: 2021-07-31T01:51:28.634 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:28 smithi045 conmon[33655]: 51:28.593671+0000 mon.b (mon.2) 86 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:51:28.635 INFO:tasks.rados.rados.0.smithi045.stdout:1777: done (0 left) 2021-07-31T01:51:28.635 INFO:tasks.rados.rados.0.smithi045.stdout:1778: copy_from oid 29 from oid 30 current snap is 167 2021-07-31T01:51:28.636 INFO:tasks.rados.rados.0.smithi045.stdout:1779: read oid 1 snap 166 2021-07-31T01:51:28.636 INFO:tasks.rados.rados.0.smithi045.stdout:1779: expect (ObjNum 465 snap 135 seq_num 465) 2021-07-31T01:51:28.636 INFO:tasks.rados.rados.0.smithi045.stdout:1780: copy_from oid 2 from oid 27 current snap is 167 2021-07-31T01:51:28.636 INFO:tasks.rados.rados.0.smithi045.stdout:1781: read oid 41 snap 160 2021-07-31T01:51:28.637 INFO:tasks.rados.rados.0.smithi045.stdout:1781: expect (ObjNum 531 snap 158 seq_num 531) 2021-07-31T01:51:28.637 INFO:tasks.rados.rados.0.smithi045.stdout:1782: read oid 47 snap 166 2021-07-31T01:51:28.637 INFO:tasks.rados.rados.0.smithi045.stdout:1782: expect (ObjNum 172 snap 41 seq_num 172) 2021-07-31T01:51:28.637 INFO:tasks.rados.rados.0.smithi045.stdout:1783: read oid 28 snap -1 2021-07-31T01:51:28.637 INFO:tasks.rados.rados.0.smithi045.stdout:1783: expect deleted 2021-07-31T01:51:28.638 INFO:tasks.rados.rados.0.smithi045.stdout:1784: rollback oid 17 current snap is 167 2021-07-31T01:51:28.638 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 17 to 166 2021-07-31T01:51:28.638 INFO:tasks.rados.rados.0.smithi045.stdout:1785: copy_from oid 45 from oid 33 current snap is 167 2021-07-31T01:51:28.638 INFO:tasks.rados.rados.0.smithi045.stdout:1786: read oid 39 snap -1 2021-07-31T01:51:28.638 INFO:tasks.rados.rados.0.smithi045.stdout:1786: expect (ObjNum 516 snap 152 seq_num 516) 2021-07-31T01:51:28.639 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 43 current snap is 167 2021-07-31T01:51:28.639 INFO:tasks.rados.rados.0.smithi045.stdout:1787: seq_num 562 ranges {4833280=622592,5455872=630784,6086656=581632,6668288=393216} 2021-07-31T01:51:28.662 INFO:tasks.rados.rados.0.smithi045.stdout:1787: writing smithi04569269-43 from 4833280 to 5455872 tid 1 2021-07-31T01:51:28.665 INFO:tasks.rados.rados.0.smithi045.stdout:1787: writing smithi04569269-43 from 5455872 to 6086656 tid 2 2021-07-31T01:51:28.669 INFO:tasks.rados.rados.0.smithi045.stdout:1787: writing smithi04569269-43 from 6086656 to 6668288 tid 3 2021-07-31T01:51:28.672 INFO:tasks.rados.rados.0.smithi045.stdout:1787: writing smithi04569269-43 from 6668288 to 7061504 tid 4 2021-07-31T01:51:28.672 INFO:tasks.rados.rados.0.smithi045.stdout:1788: delete oid 8 current snap is 167 2021-07-31T01:51:28.672 INFO:tasks.rados.rados.0.smithi045.stdout:1778: finishing copy_from to smithi04569269-29 2021-07-31T01:51:28.673 INFO:tasks.rados.rados.0.smithi045.stdout:1778: got expected ENOENT (src dne) 2021-07-31T01:51:28.673 INFO:tasks.rados.rados.0.smithi045.stdout:1778: finishing copy_from racing read to smithi04569269-29 2021-07-31T01:51:28.673 INFO:tasks.rados.rados.0.smithi045.stdout:1784: finishing rollback tid 0 to smithi04569269-17 2021-07-31T01:51:28.673 INFO:tasks.rados.rados.0.smithi045.stdout:1784: finishing rollback tid 1 to smithi04569269-17 2021-07-31T01:51:28.674 INFO:tasks.rados.rados.0.smithi045.stdout:1784: finishing rollback tid 2 to smithi04569269-17 2021-07-31T01:51:28.674 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 17 v 1040 (ObjNum 288 snap 0 seq_num 33) dirty exists 2021-07-31T01:51:28.674 INFO:tasks.rados.rados.0.smithi045.stdout:1782: expect (ObjNum 172 snap 41 seq_num 172) 2021-07-31T01:51:28.694 INFO:tasks.rados.rados.0.smithi045.stdout:1779: expect (ObjNum 465 snap 135 seq_num 465) 2021-07-31T01:51:28.787 INFO:tasks.rados.rados.0.smithi045.stdout:1780: finishing copy_from to smithi04569269-2 2021-07-31T01:51:28.788 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 2 v 279 (ObjNum 507 snap 152 seq_num 507) dirty exists 2021-07-31T01:51:28.788 INFO:tasks.rados.rados.0.smithi045.stdout:1780: finishing copy_from racing read to smithi04569269-2 2021-07-31T01:51:28.788 INFO:tasks.rados.rados.0.smithi045.stdout:1781: expect (ObjNum 531 snap 158 seq_num 531) 2021-07-31T01:51:28.831 INFO:tasks.rados.rados.0.smithi045.stdout:1786: expect (ObjNum 516 snap 152 seq_num 516) 2021-07-31T01:51:28.942 INFO:tasks.rados.rados.0.smithi045.stdout:1787: finishing write tid 1 to smithi04569269-43 2021-07-31T01:51:28.942 INFO:tasks.rados.rados.0.smithi045.stdout:1785: finishing copy_from to smithi04569269-45 2021-07-31T01:51:28.943 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 45 v 1020 (ObjNum 399 snap 121 seq_num 399) dirty exists 2021-07-31T01:51:28.943 INFO:tasks.rados.rados.0.smithi045.stdout:1787: finishing write tid 2 to smithi04569269-43 2021-07-31T01:51:28.943 INFO:tasks.rados.rados.0.smithi045.stdout:1785: finishing copy_from racing read to smithi04569269-45 2021-07-31T01:51:28.943 INFO:tasks.rados.rados.0.smithi045.stdout:1787: finishing write tid 3 to smithi04569269-43 2021-07-31T01:51:28.943 INFO:tasks.rados.rados.0.smithi045.stdout:1778: done (10 left) 2021-07-31T01:51:28.944 INFO:tasks.rados.rados.0.smithi045.stdout:1779: done (9 left) 2021-07-31T01:51:28.944 INFO:tasks.rados.rados.0.smithi045.stdout:1780: done (8 left) 2021-07-31T01:51:28.944 INFO:tasks.rados.rados.0.smithi045.stdout:1781: done (7 left) 2021-07-31T01:51:28.944 INFO:tasks.rados.rados.0.smithi045.stdout:1782: done (6 left) 2021-07-31T01:51:28.944 INFO:tasks.rados.rados.0.smithi045.stdout:1783: done (5 left) 2021-07-31T01:51:28.945 INFO:tasks.rados.rados.0.smithi045.stdout:1784: done (4 left) 2021-07-31T01:51:28.945 INFO:tasks.rados.rados.0.smithi045.stdout:1785: done (3 left) 2021-07-31T01:51:28.945 INFO:tasks.rados.rados.0.smithi045.stdout:1786: done (2 left) 2021-07-31T01:51:28.945 INFO:tasks.rados.rados.0.smithi045.stdout:1788: done (1 left) 2021-07-31T01:51:28.946 INFO:tasks.rados.rados.0.smithi045.stdout:1789: delete oid 29 current snap is 167 2021-07-31T01:51:28.946 INFO:tasks.rados.rados.0.smithi045.stdout:1787: finishing write tid 4 to smithi04569269-43 2021-07-31T01:51:28.946 INFO:tasks.rados.rados.0.smithi045.stdout:1787: finishing write tid 5 to smithi04569269-43 2021-07-31T01:51:28.946 INFO:tasks.rados.rados.0.smithi045.stdout:1787: finishing write tid 6 to smithi04569269-43 2021-07-31T01:51:28.946 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 43 v 1035 (ObjNum 562 snap 167 seq_num 562) dirty exists 2021-07-31T01:51:28.947 INFO:tasks.rados.rados.0.smithi045.stdout:1787: left oid 43 (ObjNum 562 snap 167 seq_num 562) 2021-07-31T01:51:28.948 INFO:tasks.rados.rados.0.smithi045.stdout:1787: done (1 left) 2021-07-31T01:51:28.948 INFO:tasks.rados.rados.0.smithi045.stdout:1789: done (0 left) 2021-07-31T01:51:28.949 INFO:tasks.rados.rados.0.smithi045.stdout:1790: read oid 32 snap -1 2021-07-31T01:51:28.949 INFO:tasks.rados.rados.0.smithi045.stdout:1790: expect (ObjNum 549 snap 164 seq_num 549) 2021-07-31T01:51:28.949 INFO:tasks.rados.rados.0.smithi045.stdout:1791: copy_from oid 27 from oid 16 current snap is 167 2021-07-31T01:51:28.949 INFO:tasks.rados.rados.0.smithi045.stdout:1792: setattr oid 7 current snap is 167 2021-07-31T01:51:28.949 INFO:tasks.rados.rados.0.smithi045.stdout:1793: read oid 4 snap -1 2021-07-31T01:51:28.950 INFO:tasks.rados.rados.0.smithi045.stdout:1793: expect (ObjNum 470 snap 136 seq_num 470) 2021-07-31T01:51:28.950 INFO:tasks.rados.rados.0.smithi045.stdout:1794: read oid 34 snap -1 2021-07-31T01:51:28.950 INFO:tasks.rados.rados.0.smithi045.stdout:1794: expect (ObjNum 399 snap 121 seq_num 399) 2021-07-31T01:51:28.952 INFO:tasks.rados.rados.0.smithi045.stdout:1795: delete oid 36 current snap is 167 2021-07-31T01:51:28.953 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 7 v 973 (ObjNum 288 snap 0 seq_num 33) dirty exists 2021-07-31T01:51:28.953 INFO:tasks.rados.rados.0.smithi045.stdout:1792: done (5 left) 2021-07-31T01:51:28.954 INFO:tasks.rados.rados.0.smithi045.stdout:1795: done (4 left) 2021-07-31T01:51:28.954 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 17 current snap is 167 2021-07-31T01:51:28.954 INFO:tasks.rados.rados.0.smithi045.stdout:1796: seq_num 566 ranges {0=688128,688128=614400,1302528=565248,1867776=425984} 2021-07-31T01:51:28.957 INFO:tasks.rados.rados.0.smithi045.stdout:1796: writing smithi04569269-17 from 0 to 688128 tid 1 2021-07-31T01:51:28.963 INFO:tasks.rados.rados.0.smithi045.stdout:1796: writing smithi04569269-17 from 688128 to 1302528 tid 2 2021-07-31T01:51:28.964 INFO:tasks.rados.rados.0.smithi045.stdout:1796: writing smithi04569269-17 from 1302528 to 1867776 tid 3 2021-07-31T01:51:28.968 INFO:tasks.rados.rados.0.smithi045.stdout:1796: writing smithi04569269-17 from 1867776 to 2293760 tid 4 2021-07-31T01:51:28.968 INFO:tasks.rados.rados.0.smithi045.stdout:1797: snap_create 2021-07-31T01:51:28.968 INFO:tasks.rados.rados.0.smithi045.stdout:1790: expect (ObjNum 549 snap 164 seq_num 549) 2021-07-31T01:51:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[34553]: cluster 2021-07-31T01:51:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[34553]: 2021-07-31T01:51:27.618470+0000 2021-07-31T01:51:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[34553]: mon.a (mon.0) 1034 : cluster [DBG] osdmap e388: 8 total, 8 up, 8 in 2021-07-31T01:51:29.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[34553]: cluster 2021-07-31T01:51:27.850951+0000 mgr.x (mgr 2021-07-31T01:51:29.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[34553]: .24542) 175 : cluster [DBG] pgmap v362: 25 pgs: 25 active+clean; 165 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2021-07-31T01:51:29.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[34553]: audit 2021-07-31T01 2021-07-31T01:51:29.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[34553]: :51:27.956341+0000 mon.b (mon.2) 82 : audit [INF] 2021-07-31T01:51:29.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[34553]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:51:29.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[34553]: audit 2021-07-31T01:51:27.956810+0000 2021-07-31T01:51:29.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[34553]: mon.a (mon.0) 1035 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:51:29.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[34553]: audit 2021- 2021-07-31T01:51:29.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[34553]: 07-31T01:51:27.967732+0000 mon.b (mon.2) 83 : audit 2021-07-31T01:51:29.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[34553]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T01:51:29.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[34553]: audit 2021-07-31T01:51:27. 2021-07-31T01:51:29.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[34553]: 968082+0000 mon.a (mon.0) 1036 : audit 2021-07-31T01:51:29.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[34553]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T01:51:29.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[34553]: audit 2021-07-31T01:51:28.590979+0000 2021-07-31T01:51:29.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[34553]: mon.b (mon.2) 84 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:51:29.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[34553]: audit 2021-07 2021-07-31T01:51:29.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[34553]: -31T01:51:28.592480+0000 mon.b (mon.2) 85 : audit 2021-07-31T01:51:29.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[34553]: [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:51:29.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[34553]: audit 2021-07-31T 2021-07-31T01:51:29.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[34553]: 01:51:28.593671+0000 mon.b (mon.2) 86 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:51:29.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: debug 2021-07-31T01:51:28.643+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T01:51:29.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:51:29.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:51:29.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: yield 2021-07-31T01:51:29.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:51:29.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:51:29.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:51:29.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T01:51:29.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:51:29.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T01:51:29.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:51:29.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: await self._waiter 2021-07-31T01:51:29.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:51:29.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: task.result() 2021-07-31T01:51:29.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T01:51:29.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: debug 2021-07-31T01:51:28.645+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T01:51:29.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:51:29.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:51:29.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: yield 2021-07-31T01:51:29.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:51:29.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:51:29.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:51:29.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T01:51:29.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:51:29.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T01:51:29.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:51:29.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: await self._waiter 2021-07-31T01:51:29.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:51:29.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: task.result() 2021-07-31T01:51:29.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T01:51:29.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: debug 2021-07-31T01:51:28.689+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T01:51:29.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:51:29.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:51:29.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: yield 2021-07-31T01:51:29.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:51:29.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:51:29.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:51:29.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T01:51:29.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:51:29.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T01:51:29.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:51:29.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: await self._waiter 2021-07-31T01:51:29.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:51:29.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: task.result() 2021-07-31T01:51:29.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T01:51:29.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: debug 2021-07-31T01:51:28.689+0000 7fcb97969700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:51:29.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:51:29.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:51:29.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: yield 2021-07-31T01:51:29.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:51:29.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:51:29.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:51:29.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T01:51:29.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:51:29.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T01:51:29.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:51:29.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: await self._waiter 2021-07-31T01:51:29.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:51:29.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: task.result() 2021-07-31T01:51:29.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T01:51:29.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: 2021-07-31T01:51:29.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T01:51:29.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: 2021-07-31T01:51:29.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:51:29.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:51:29.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: return f(*arg) 2021-07-31T01:51:29.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:51:29.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:51:29.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:51:29.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:51:29.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:51:29.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:51:29.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:51:29.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T01:51:29.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:51:29.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: raise self._exception 2021-07-31T01:51:29.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:51:29.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:51:29.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:51:29.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T01:51:29.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:51:29.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: raise 2021-07-31T01:51:29.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:51:29.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T01:51:29.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:51:29.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:51:29.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T01:51:29.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: debug 2021-07-31T01:51:28.697+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T01:51:29.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:51:29.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:51:29.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: yield 2021-07-31T01:51:29.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:51:29.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:51:29.035 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:51:29.035 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T01:51:29.035 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:51:29.036 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T01:51:29.036 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:51:29.036 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: await self._waiter 2021-07-31T01:51:29.036 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:51:29.036 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: task.result() 2021-07-31T01:51:29.037 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T01:51:29.037 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: debug 2021-07-31T01:51:28.697+0000 7fcb9816a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:51:29.037 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:51:29.037 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:51:29.038 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: yield 2021-07-31T01:51:29.038 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:51:29.038 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:51:29.038 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:51:29.039 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T01:51:29.039 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:51:29.039 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T01:51:29.039 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:51:29.040 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: await self._waiter 2021-07-31T01:51:29.040 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:51:29.040 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: task.result() 2021-07-31T01:51:29.041 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T01:51:29.041 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: 2021-07-31T01:51:29.041 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T01:51:29.041 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: 2021-07-31T01:51:29.041 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:51:29.042 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:51:29.042 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: return f(*arg) 2021-07-31T01:51:29.042 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:51:29.042 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:51:29.043 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:51:29.043 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:51:29.043 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:51:29.043 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:51:29.043 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:51:29.044 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T01:51:29.044 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:51:29.044 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: raise self._exception 2021-07-31T01:51:29.044 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:51:29.044 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:51:29.045 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:51:29.045 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T01:51:29.045 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:51:29.045 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: raise 2021-07-31T01:51:29.046 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:51:29.046 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T01:51:29.046 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:51:29.046 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:51:29.047 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:51:28 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T01:51:29.047 INFO:tasks.rados.rados.0.smithi045.stdout:1793: expect (ObjNum 470 snap 136 seq_num 470) 2021-07-31T01:51:29.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:28 smithi045 conmon[39718]: (mon.2) 82 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:51:29.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:28 smithi045 conmon[39718]: audit 2021 2021-07-31T01:51:29.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:28 smithi045 conmon[39718]: -07-31T01:51:27.956810+0000 mon.a (mon.0) 1035 2021-07-31T01:51:29.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:28 smithi045 conmon[39718]: : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:51:29.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:28 smithi045 conmon[39718]: audit 2021-07-31T01:51:29.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:28 smithi045 conmon[39718]: 2021-07-31T01:51:27.967732+0000 2021-07-31T01:51:29.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:28 smithi045 conmon[39718]: mon.b (mon.2) 83 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T01:51:29.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:28 smithi045 conmon[39718]: 2021-07-31T01:51:29.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:28 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T01:51:29.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:28 smithi045 conmon[39718]: T01:51:27.968082+0000 mon.a (mon. 2021-07-31T01:51:29.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:28 smithi045 conmon[39718]: 0) 1036 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T01:51:29.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:28 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T01:51:29.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:28 smithi045 conmon[39718]: T01:51:28.590979+0000 mon.b (mon.2) 84 : audit 2021-07-31T01:51:29.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:28 smithi045 conmon[39718]: [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:51:29.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:28 smithi045 conmon[39718]: audit 2021-07-31T01:51:28.592480+0000 2021-07-31T01:51:29.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:28 smithi045 conmon[39718]: mon.b (mon.2) 85 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:51:29.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:28 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T01:51:29.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:28 smithi045 conmon[39718]: 31T01:51:28.593671+0000 mon.b (mon. 2021-07-31T01:51:29.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:28 smithi045 conmon[39718]: 2) 86 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:51:29.148 INFO:tasks.rados.rados.0.smithi045.stdout:1791: finishing copy_from to smithi04569269-27 2021-07-31T01:51:29.149 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 27 v 1021 (ObjNum 523 snap 154 seq_num 523) dirty exists 2021-07-31T01:51:29.149 INFO:tasks.rados.rados.0.smithi045.stdout:1791: finishing copy_from racing read to smithi04569269-27 2021-07-31T01:51:29.149 INFO:tasks.rados.rados.0.smithi045.stdout:1796: finishing write tid 1 to smithi04569269-17 2021-07-31T01:51:29.149 INFO:tasks.rados.rados.0.smithi045.stdout:1796: finishing write tid 2 to smithi04569269-17 2021-07-31T01:51:29.150 INFO:tasks.rados.rados.0.smithi045.stdout:1794: expect (ObjNum 399 snap 121 seq_num 399) 2021-07-31T01:51:29.390 INFO:tasks.rados.rados.0.smithi045.stdout:1796: finishing write tid 3 to smithi04569269-17 2021-07-31T01:51:29.391 INFO:tasks.rados.rados.0.smithi045.stdout:1796: finishing write tid 4 to smithi04569269-17 2021-07-31T01:51:29.391 INFO:tasks.rados.rados.0.smithi045.stdout:1796: finishing write tid 5 to smithi04569269-17 2021-07-31T01:51:29.391 INFO:tasks.rados.rados.0.smithi045.stdout:1796: finishing write tid 6 to smithi04569269-17 2021-07-31T01:51:29.391 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 17 v 1045 (ObjNum 566 snap 167 seq_num 566) dirty exists 2021-07-31T01:51:29.392 INFO:tasks.rados.rados.0.smithi045.stdout:1796: left oid 17 (ObjNum 566 snap 167 seq_num 566) 2021-07-31T01:51:29.629 INFO:tasks.rados.rados.0.smithi045.stdout:1790: done (5 left) 2021-07-31T01:51:29.629 INFO:tasks.rados.rados.0.smithi045.stdout:1791: done (4 left) 2021-07-31T01:51:29.630 INFO:tasks.rados.rados.0.smithi045.stdout:1793: done (3 left) 2021-07-31T01:51:29.630 INFO:tasks.rados.rados.0.smithi045.stdout:1794: done (2 left) 2021-07-31T01:51:29.630 INFO:tasks.rados.rados.0.smithi045.stdout:1796: done (1 left) 2021-07-31T01:51:29.630 INFO:tasks.rados.rados.0.smithi045.stdout:1797: done (0 left) 2021-07-31T01:51:29.631 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 42 current snap is 168 2021-07-31T01:51:29.631 INFO:tasks.rados.rados.0.smithi045.stdout:1798: seq_num 567 ranges {2662400=704512,3366912=499712,3866624=770048,4636672=335872} 2021-07-31T01:51:29.647 INFO:tasks.rados.rados.0.smithi045.stdout:1798: writing smithi04569269-42 from 2662400 to 3366912 tid 1 2021-07-31T01:51:29.650 INFO:tasks.rados.rados.0.smithi045.stdout:1798: writing smithi04569269-42 from 3366912 to 3866624 tid 2 2021-07-31T01:51:29.654 INFO:tasks.rados.rados.0.smithi045.stdout:1798: writing smithi04569269-42 from 3866624 to 4636672 tid 3 2021-07-31T01:51:29.657 INFO:tasks.rados.rados.0.smithi045.stdout:1798: writing smithi04569269-42 from 4636672 to 4972544 tid 4 2021-07-31T01:51:29.657 INFO:tasks.rados.rados.0.smithi045.stdout:1799: snap_create 2021-07-31T01:51:29.657 INFO:tasks.rados.rados.0.smithi045.stdout:1798: finishing write tid 1 to smithi04569269-42 2021-07-31T01:51:29.658 INFO:tasks.rados.rados.0.smithi045.stdout:1798: finishing write tid 2 to smithi04569269-42 2021-07-31T01:51:29.660 INFO:tasks.rados.rados.0.smithi045.stdout:1798: finishing write tid 3 to smithi04569269-42 2021-07-31T01:51:29.660 INFO:tasks.rados.rados.0.smithi045.stdout:1798: finishing write tid 4 to smithi04569269-42 2021-07-31T01:51:29.661 INFO:tasks.rados.rados.0.smithi045.stdout:1798: finishing write tid 5 to smithi04569269-42 2021-07-31T01:51:29.662 INFO:tasks.rados.rados.0.smithi045.stdout:1798: finishing write tid 6 to smithi04569269-42 2021-07-31T01:51:29.662 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 42 v 1050 (ObjNum 567 snap 168 seq_num 567) dirty exists 2021-07-31T01:51:29.663 INFO:tasks.rados.rados.0.smithi045.stdout:1798: left oid 42 (ObjNum 567 snap 168 seq_num 567) 2021-07-31T01:51:30.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: cephadm 2021-07-31T01:51:28.594475+0000 mgr.x (mgr.24542) 176 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T01:51:30.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: cluster 2021-07-31T01:51:28.624506+0000 mon.a (mon.0) 1037 : cluster [DBG] osdmap e389: 8 total, 8 up, 8 in 2021-07-31T01:51:30.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: cephadm 2021-07-31T01:51:28.644942+0000 mgr.x (mgr.24542) 177 : cephadm [ERR] 2021-07-31T01:51:30.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:51:30.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:51:30.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: yield 2021-07-31T01:51:30.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:51:30.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:51:30.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:51:30.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T01:51:30.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:51:30.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T01:51:30.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:51:30.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: await self._waiter 2021-07-31T01:51:30.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:51:30.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: task.result() 2021-07-31T01:51:30.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T01:51:30.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: cephadm 2021-07-31T01:51:28.646254+0000 mgr.x (mgr.24542) 178 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T01:51:30.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: cephadm 2021-07-31T01:51:28.647813+0000 mgr.x (mgr.24542) 179 : cephadm [ERR] 2021-07-31T01:51:30.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:51:30.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:51:30.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: yield 2021-07-31T01:51:30.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:51:30.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:51:30.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:51:30.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T01:51:30.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:51:30.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T01:51:30.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:51:30.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: await self._waiter 2021-07-31T01:51:30.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:51:30.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: task.result() 2021-07-31T01:51:30.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T01:51:30.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: cephadm 2021-07-31T01:51:30.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: 2021-07-31T01:51:28.659424+0000 mgr.x (mgr.24542) 180 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T01:51:30.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: cephadm 2021-07-31T01:51:28.690842+0000 mgr.x (mgr.24542) 181 : cephadm [ERR] 2021-07-31T01:51:30.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:51:30.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:51:30.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: yield 2021-07-31T01:51:30.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:51:30.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:51:30.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:51:30.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T01:51:30.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:51:30.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T01:51:30.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:51:30.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: await self._waiter 2021-07-31T01:51:30.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:51:30.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: task.result() 2021-07-31T01:51:30.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T01:51:30.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: cephadm 2021-07-31T01:51:28.691296+0000 mgr.x (mgr.24542) 182 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:51:30.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:51:30.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:51:30.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: yield 2021-07-31T01:51:30.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:51:30.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:51:30.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:51:30.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T01:51:30.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:51:30.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T01:51:30.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:51:30.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: await self._waiter 2021-07-31T01:51:30.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:51:30.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: task.result() 2021-07-31T01:51:30.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T01:51:30.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: 2021-07-31T01:51:30.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T01:51:30.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: 2021-07-31T01:51:30.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:51:30.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:51:30.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: return f(*arg) 2021-07-31T01:51:30.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:51:30.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:51:30.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:51:30.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:51:30.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:51:30.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:51:30.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:51:30.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T01:51:30.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:51:30.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: raise self._exception 2021-07-31T01:51:30.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:51:30.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:51:30.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:51:30.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T01:51:30.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:51:30.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: raise 2021-07-31T01:51:30.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:51:30.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T01:51:30.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:51:30.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:51:30.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T01:51:30.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: cephadm 2021-07-31T01:51:28.698850+0000 mgr.x (mgr.24542) 183 : cephadm [ERR] 2021-07-31T01:51:30.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: 2021-07-31T01:51:30.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:51:30.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:51:30.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: yield 2021-07-31T01:51:30.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:51:30.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:51:30.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:51:30.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T01:51:30.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:51:30.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T01:51:30.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:51:30.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: await self._waiter 2021-07-31T01:51:30.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:51:30.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: task.result() 2021-07-31T01:51:30.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T01:51:30.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: cephadm 2021-07-31T01:51:28.699255+0000 mgr.x (mgr.24542) 184 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:51:30.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:51:30.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:51:30.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: yield 2021-07-31T01:51:30.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:51:30.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:51:30.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:51:30.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T01:51:30.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:51:30.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T01:51:30.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:51:30.035 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: await self._waiter 2021-07-31T01:51:30.035 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:51:30.035 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: task.result() 2021-07-31T01:51:30.035 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T01:51:30.035 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: 2021-07-31T01:51:30.036 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T01:51:30.036 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: 2021-07-31T01:51:30.036 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:51:30.036 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:51:30.037 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: return f(*arg) 2021-07-31T01:51:30.037 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:51:30.037 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:51:30.037 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:51:30.038 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:51:30.038 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:51:30.038 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:51:30.039 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:51:30.039 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T01:51:30.040 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:51:30.040 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: raise self._exception 2021-07-31T01:51:30.040 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:51:30.040 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:51:30.040 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:51:30.041 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T01:51:30.041 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:51:30.041 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: raise 2021-07-31T01:51:30.041 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:51:30.042 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T01:51:30.042 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:51:30.042 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:51:30.042 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:29 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T01:51:30.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: cephadm 2021-07-31T01:51:28.594475+0000 2021-07-31T01:51:30.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: mgr.x (mgr.24542) 176 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T01:51:30.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: cluster 2021- 2021-07-31T01:51:30.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: 07-31T01:51:28.624506+0000 mon.a (mon.0) 1037 : cluster [DBG] osdmap e389: 8 total, 8 up, 8 in 2021-07-31T01:51:30.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: 2021-07-31T01:51:30.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: cephadm 2021-07-31T01: 2021-07-31T01:51:30.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: 51:28.644942+0000 mgr.x (mgr.24542 2021-07-31T01:51:30.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: ) 177 : cephadm [ERR] 2021-07-31T01:51:30.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:51:30.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:51:30.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: yield 2021-07-31T01:51:30.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:51:30.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:51:30.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:51:30.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T01:51:30.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:51:30.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T01:51:30.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:51:30.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: await self._waiter 2021-07-31T01:51:30.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:51:30.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: task.result() 2021-07-31T01:51:30.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T01:51:30.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: cephadm 2021-07-31T01:51:28.646254+0000 mgr.x (mgr.24542) 178 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T01:51:30.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: cephadm 2021-07-31T01: 2021-07-31T01:51:30.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: 51:28.647813+0000 mgr.x (mgr.24542) 179 : cephadm [ERR] 2021-07-31T01:51:30.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: 2021-07-31T01:51:30.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:51:30.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:51:30.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: yield 2021-07-31T01:51:30.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:51:30.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:51:30.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:51:30.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T01:51:30.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:51:30.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T01:51:30.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:51:30.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: await self._waiter 2021-07-31T01:51:30.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:51:30.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: task.result() 2021-07-31T01:51:30.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T01:51:30.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: cephadm 2021-07-31T01:51:28.659424+0000 mgr.x (mgr.24542) 180 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T01:51:30.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: cephadm 2021-07-31T01:51:28. 2021-07-31T01:51:30.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: 690842+0000 mgr.x (mgr.24542) 181 : cephadm [ERR] 2021-07-31T01:51:30.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:51:30.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:51:30.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: yield 2021-07-31T01:51:30.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:51:30.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:51:30.057 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:51:30.057 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T01:51:30.057 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:51:30.057 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T01:51:30.057 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:51:30.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: await self._waiter 2021-07-31T01:51:30.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:51:30.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: task.result() 2021-07-31T01:51:30.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T01:51:30.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: cephadm 2021-07-31T01:51:28.691296+0000 mgr.x (mgr.24542) 182 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:51:30.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:51:30.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:51:30.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: yield 2021-07-31T01:51:30.060 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:51:30.061 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:51:30.061 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:51:30.061 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T01:51:30.061 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:51:30.062 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T01:51:30.062 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:51:30.062 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: await self._waiter 2021-07-31T01:51:30.062 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:51:30.063 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: task.result() 2021-07-31T01:51:30.063 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T01:51:30.063 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: 2021-07-31T01:51:30.063 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T01:51:30.064 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: 2021-07-31T01:51:30.064 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:51:30.064 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:51:30.065 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: return f(*arg) 2021-07-31T01:51:30.065 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:51:30.065 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:51:30.065 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:51:30.066 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:51:30.066 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:51:30.066 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:51:30.066 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:51:30.067 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T01:51:30.067 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:51:30.067 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: raise self._exception 2021-07-31T01:51:30.067 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:51:30.067 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:51:30.068 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:51:30.068 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T01:51:30.068 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:51:30.068 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: raise 2021-07-31T01:51:30.068 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:51:30.069 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T01:51:30.069 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:51:30.069 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:51:30.069 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T01:51:30.070 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: cephadm 2021- 2021-07-31T01:51:30.070 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: 07-31T01:51:28.698850+0000 mgr.x (mgr.24542) 183 : cephadm [ERR] 2021-07-31T01:51:30.070 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:51:30.071 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:51:30.071 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: yield 2021-07-31T01:51:30.071 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:51:30.071 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:51:30.072 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:51:30.072 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T01:51:30.072 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:51:30.072 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T01:51:30.072 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:51:30.073 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: await self._waiter 2021-07-31T01:51:30.073 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:51:30.073 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: task.result() 2021-07-31T01:51:30.073 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T01:51:30.074 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: cephadm 2021-07-31T01:51:28.699255+0000 mgr.x (mgr.24542) 184 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:51:30.074 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:51:30.074 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:51:30.075 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: yield 2021-07-31T01:51:30.075 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:51:30.075 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:51:30.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: cephadm 2021-07-31T01:51: 2021-07-31T01:51:30.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: 28.594475+0000 mgr.x (mgr.24542) 176 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T01:51:30.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: cluster 2021-07-31T01:51:30.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: 2021-07-31T01:51:28.624506 2021-07-31T01:51:30.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: +0000 mon.a (mon.0) 1037 : cluster 2021-07-31T01:51:30.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: [DBG] osdmap e389: 8 total, 8 up, 8 in 2021-07-31T01:51:30.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: cephadm 2021-07- 2021-07-31T01:51:30.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: 31T01:51:28.644942+0000 2021-07-31T01:51:30.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: mgr.x (mgr.24542) 177 : cephadm [ERR] 2021-07-31T01:51:30.078 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: 2021-07-31T01:51:30.078 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:51:30.078 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:51:30.078 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: yield 2021-07-31T01:51:30.079 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:51:30.079 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:51:30.079 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:51:30.079 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T01:51:30.079 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:51:30.080 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T01:51:30.080 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:51:30.080 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: await self._waiter 2021-07-31T01:51:30.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:51:30.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: task.result() 2021-07-31T01:51:30.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T01:51:30.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: cephadm 2021-07-31T01:51:28.646254+0000 mgr.x (mgr.24542) 178 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T01:51:30.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: cephadm 2021-07-31T01:51:28.647813+0000 2021-07-31T01:51:30.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: mgr.x (mgr.24542) 179 : cephadm [ERR] 2021-07-31T01:51:30.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:51:30.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:51:30.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: yield 2021-07-31T01:51:30.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:51:30.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:51:30.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:51:30.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T01:51:30.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:51:30.084 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T01:51:30.084 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:51:30.084 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: await self._waiter 2021-07-31T01:51:30.084 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:51:30.085 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: task.result() 2021-07-31T01:51:30.085 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T01:51:30.086 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:51:30.086 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T01:51:30.086 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:51:30.087 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T01:51:30.087 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:51:30.087 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: await self._waiter 2021-07-31T01:51:30.087 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:51:30.087 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: task.result() 2021-07-31T01:51:30.088 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T01:51:30.088 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: 2021-07-31T01:51:30.088 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T01:51:30.088 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: 2021-07-31T01:51:30.088 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:51:30.089 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:51:30.089 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: return f(*arg) 2021-07-31T01:51:30.089 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:51:30.089 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:51:30.090 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:51:30.090 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:51:30.090 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:51:30.090 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:51:30.090 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:51:30.091 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T01:51:30.091 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:51:30.091 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: raise self._exception 2021-07-31T01:51:30.092 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:51:30.092 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:51:30.092 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:51:30.092 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T01:51:30.093 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:51:30.093 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: raise 2021-07-31T01:51:30.093 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:51:30.093 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T01:51:30.093 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:51:30.094 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:51:30.094 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T01:51:30.095 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: cephadm 2021-07-31T01:51:28.659424+0000 mgr.x (mgr.24542) 180 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T01:51:30.095 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: cephadm 2021-07-31T 2021-07-31T01:51:30.095 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: 01:51:28.690842+0000 mgr.x (mgr.24542) 2021-07-31T01:51:30.096 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: 181 : cephadm [ERR] 2021-07-31T01:51:30.096 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:51:30.096 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:51:30.096 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: yield 2021-07-31T01:51:30.097 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:51:30.097 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:51:30.097 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:51:30.097 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T01:51:30.097 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:51:30.098 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T01:51:30.098 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:51:30.098 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: await self._waiter 2021-07-31T01:51:30.098 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:51:30.099 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: task.result() 2021-07-31T01:51:30.099 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T01:51:30.100 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: cephadm 2021-07-31T01:51:28.691296+0000 mgr.x (mgr.24542) 182 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:51:30.100 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:51:30.100 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:51:30.101 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: yield 2021-07-31T01:51:30.101 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:51:30.101 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:51:30.101 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:51:30.102 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T01:51:30.102 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:51:30.102 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T01:51:30.102 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:51:30.103 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: await self._waiter 2021-07-31T01:51:30.103 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:51:30.103 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: task.result() 2021-07-31T01:51:30.103 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T01:51:30.103 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: 2021-07-31T01:51:30.104 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T01:51:30.104 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: 2021-07-31T01:51:30.104 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:51:30.104 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:51:30.104 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: return f(*arg) 2021-07-31T01:51:30.105 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:51:30.105 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:51:30.105 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:51:30.105 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:51:30.106 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:51:30.106 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:51:30.106 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:51:30.106 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T01:51:30.106 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:51:30.107 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: raise self._exception 2021-07-31T01:51:30.107 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:51:30.107 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:51:30.107 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:51:30.108 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T01:51:30.108 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:51:30.108 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: raise 2021-07-31T01:51:30.108 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:51:30.108 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T01:51:30.109 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:51:30.109 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:51:30.109 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T01:51:30.109 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: cephadm 2021 2021-07-31T01:51:30.109 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: -07-31T01:51:28.698850+0000 mgr.x (mgr.24542) 183 : cephadm [ERR] 2021-07-31T01:51:30.110 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:51:30.111 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:51:30.111 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: yield 2021-07-31T01:51:30.111 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:51:30.111 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:51:30.112 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:51:30.112 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T01:51:30.112 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:51:30.112 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T01:51:30.113 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:51:30.113 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: await self._waiter 2021-07-31T01:51:30.113 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:51:30.113 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: task.result() 2021-07-31T01:51:30.113 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T01:51:30.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: cephadm 2021-07-31T01:51:28.699255+0000 mgr.x (mgr.24542) 184 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:51:30.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:51:30.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:51:30.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: yield 2021-07-31T01:51:30.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:51:30.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:51:30.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:51:30.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T01:51:30.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:51:30.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T01:51:30.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:51:30.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: await self._waiter 2021-07-31T01:51:30.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:51:30.117 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: task.result() 2021-07-31T01:51:30.117 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T01:51:30.117 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: 2021-07-31T01:51:30.117 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T01:51:30.117 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: 2021-07-31T01:51:30.118 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:51:30.118 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:51:30.118 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: return f(*arg) 2021-07-31T01:51:30.118 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:51:30.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:51:30.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:51:30.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:51:30.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:51:30.120 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:51:30.120 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:51:30.120 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T01:51:30.120 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:51:30.120 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: raise self._exception 2021-07-31T01:51:30.121 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:51:30.121 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:51:30.121 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:51:30.121 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T01:51:30.121 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:51:30.122 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: raise 2021-07-31T01:51:30.122 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:51:30.122 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T01:51:30.122 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:51:30.123 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:51:30.123 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:29 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T01:51:30.636 INFO:tasks.rados.rados.0.smithi045.stdout:1798: done (1 left) 2021-07-31T01:51:30.637 INFO:tasks.rados.rados.0.smithi045.stdout:1799: done (0 left) 2021-07-31T01:51:30.637 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 18 current snap is 169 2021-07-31T01:51:30.637 INFO:tasks.rados.rados.0.smithi045.stdout:1800: seq_num 568 ranges {0=720896,720896=786432,1507328=581632,2088960=237568} 2021-07-31T01:51:30.640 INFO:tasks.rados.rados.0.smithi045.stdout:1800: writing smithi04569269-18 from 0 to 720896 tid 1 2021-07-31T01:51:30.645 INFO:tasks.rados.rados.0.smithi045.stdout:1800: writing smithi04569269-18 from 720896 to 1507328 tid 2 2021-07-31T01:51:30.648 INFO:tasks.rados.rados.0.smithi045.stdout:1800: writing smithi04569269-18 from 1507328 to 2088960 tid 3 2021-07-31T01:51:30.649 INFO:tasks.rados.rados.0.smithi045.stdout:1800: writing smithi04569269-18 from 2088960 to 2326528 tid 4 2021-07-31T01:51:30.650 INFO:tasks.rados.rados.0.smithi045.stdout:1801: snap_create 2021-07-31T01:51:30.650 INFO:tasks.rados.rados.0.smithi045.stdout:1800: finishing write tid 1 to smithi04569269-18 2021-07-31T01:51:30.650 INFO:tasks.rados.rados.0.smithi045.stdout:1800: finishing write tid 2 to smithi04569269-18 2021-07-31T01:51:30.652 INFO:tasks.rados.rados.0.smithi045.stdout:1800: finishing write tid 3 to smithi04569269-18 2021-07-31T01:51:30.653 INFO:tasks.rados.rados.0.smithi045.stdout:1800: finishing write tid 4 to smithi04569269-18 2021-07-31T01:51:30.653 INFO:tasks.rados.rados.0.smithi045.stdout:1800: finishing write tid 5 to smithi04569269-18 2021-07-31T01:51:30.654 INFO:tasks.rados.rados.0.smithi045.stdout:1800: finishing write tid 6 to smithi04569269-18 2021-07-31T01:51:30.654 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 18 v 431 (ObjNum 568 snap 169 seq_num 568) dirty exists 2021-07-31T01:51:30.655 INFO:tasks.rados.rados.0.smithi045.stdout:1800: left oid 18 (ObjNum 568 snap 169 seq_num 568) 2021-07-31T01:51:31.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:30 smithi154 conmon[34553]: cluster 2021-07-31T01:51:29.628729 2021-07-31T01:51:31.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:30 smithi154 conmon[34553]: +0000 mon.a (mon.0) 1038 : cluster [DBG] osdmap e390: 8 total, 8 up, 8 in 2021-07-31T01:51:31.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:30 smithi154 conmon[34553]: cluster 2021-07-31T01:51:29.851362+0000 mgr.x (mgr.24542) 185 : cluster [DBG] pgmap v365: 25 pgs: 25 active+clean; 194 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 19 MiB/s rd, 8.5 MiB/s wr, 33 op/s 2021-07-31T01:51:31.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:30 smithi045 conmon[33655]: cluster 2021-07-31T01:51:29.628729+0000 2021-07-31T01:51:31.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:30 smithi045 conmon[33655]: mon.a (mon.0) 1038 : cluster [DBG] osdmap e390: 8 total, 8 up, 8 in 2021-07-31T01:51:31.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:30 smithi045 conmon[33655]: cluster 2021-07-31T01:51:29.851362+0000 mgr.x (mgr.24542) 2021-07-31T01:51:31.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:30 smithi045 conmon[33655]: 185 : cluster [DBG] pgmap v365: 25 pgs: 25 active+clean; 194 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 19 MiB/s rd, 8.5 MiB/s wr, 33 op/s 2021-07-31T01:51:31.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:30 smithi045 conmon[39718]: cluster 2021-07-31T01:51:29.628729+0000 mon.a (mon. 2021-07-31T01:51:31.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:30 smithi045 conmon[39718]: 0) 1038 : cluster [DBG] osdmap e390: 8 total, 8 up, 8 in 2021-07-31T01:51:31.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:30 smithi045 conmon[39718]: cluster 2021-07-31T01:51:29.851362+0000 mgr.x (mgr.24542) 185 : cluster 2021-07-31T01:51:31.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:30 smithi045 conmon[39718]: [DBG] pgmap v365: 25 pgs: 25 active+clean; 194 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 19 MiB/s rd, 8.5 MiB/s wr, 33 op/s 2021-07-31T01:51:31.556 INFO:tasks.rados.rados.0.smithi045.stdout:1800: done (1 left) 2021-07-31T01:51:31.557 INFO:tasks.rados.rados.0.smithi045.stdout:1801: done (0 left) 2021-07-31T01:51:31.557 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 17 current snap is 170 2021-07-31T01:51:31.557 INFO:tasks.rados.rados.0.smithi045.stdout:1802: seq_num 569 ranges {2293760=737280,3031040=671744,3702784=737280,4440064=196608} 2021-07-31T01:51:31.573 INFO:tasks.rados.rados.0.smithi045.stdout:1802: writing smithi04569269-17 from 2293760 to 3031040 tid 1 2021-07-31T01:51:31.576 INFO:tasks.rados.rados.0.smithi045.stdout:1802: writing smithi04569269-17 from 3031040 to 3702784 tid 2 2021-07-31T01:51:31.581 INFO:tasks.rados.rados.0.smithi045.stdout:1802: writing smithi04569269-17 from 3702784 to 4440064 tid 3 2021-07-31T01:51:31.582 INFO:tasks.rados.rados.0.smithi045.stdout:1802: writing smithi04569269-17 from 4440064 to 4636672 tid 4 2021-07-31T01:51:31.582 INFO:tasks.rados.rados.0.smithi045.stdout:1803: rollback oid 22 current snap is 170 2021-07-31T01:51:31.583 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 22 to 167 2021-07-31T01:51:31.583 INFO:tasks.rados.rados.0.smithi045.stdout:1802: finishing write tid 1 to smithi04569269-17 2021-07-31T01:51:31.583 INFO:tasks.rados.rados.0.smithi045.stdout:1802: finishing write tid 2 to smithi04569269-17 2021-07-31T01:51:31.584 INFO:tasks.rados.rados.0.smithi045.stdout:1804: delete oid 39 current snap is 170 2021-07-31T01:51:31.587 INFO:tasks.rados.rados.0.smithi045.stdout:1804: done (2 left) 2021-07-31T01:51:31.587 INFO:tasks.rados.rados.0.smithi045.stdout:1805: snap_remove snap 165 2021-07-31T01:51:32.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:31 smithi154 conmon[34553]: cluster 2021-07-31 2021-07-31T01:51:32.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:31 smithi154 conmon[34553]: T01:51:30.635336+0000 mon.a (mon.0) 1039 : cluster [DBG] osdmap e391: 8 total, 8 up, 8 in 2021-07-31T01:51:32.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:31 smithi154 conmon[34553]: cluster 2021-07-31T01:51:31.555211+0000 mon.a (mon.0) 1040 : cluster [DBG] 2021-07-31T01:51:32.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:31 smithi154 conmon[34553]: osdmap e392: 8 total, 8 up, 8 in 2021-07-31T01:51:32.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:31 smithi045 conmon[39718]: cluster 2021-07-31T01:51:30.635336+0000 mon.a (mon 2021-07-31T01:51:32.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:31 smithi045 conmon[39718]: .0) 1039 : cluster [DBG] osdmap e391: 8 total, 8 up, 8 in 2021-07-31T01:51:32.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:31 smithi045 conmon[39718]: cluster 2021-07-31T01:51:31.555211+0000 mon.a 2021-07-31T01:51:32.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:31 smithi045 conmon[39718]: (mon.0) 1040 : cluster [DBG] osdmap e392: 8 total, 8 up, 8 in 2021-07-31T01:51:32.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:31 smithi045 conmon[33655]: cluster 2021-07-31T01:51:30.635336+0000 mon.a (mon. 2021-07-31T01:51:32.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:31 smithi045 conmon[33655]: 0) 1039 : cluster [DBG] osdmap e391: 8 total, 8 up, 8 in 2021-07-31T01:51:32.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:31 smithi045 conmon[33655]: cluster 2021-07-31T01:51:31.555211+0000 mon.a (mon.0) 1040 : cluster [DBG] osdmap e392: 8 total, 8 up, 8 in 2021-07-31T01:51:32.561 INFO:tasks.rados.rados.0.smithi045.stdout:1805: done (2 left) 2021-07-31T01:51:32.562 INFO:tasks.rados.rados.0.smithi045.stdout:1806: snap_create 2021-07-31T01:51:32.562 INFO:tasks.rados.rados.0.smithi045.stdout:1802: finishing write tid 3 to smithi04569269-17 2021-07-31T01:51:32.562 INFO:tasks.rados.rados.0.smithi045.stdout:1803: finishing rollback tid 1 to smithi04569269-22 2021-07-31T01:51:32.563 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 22 v 1022 (ObjNum -1104783568 snap 21873 seq_num 3189702464) dirty dne 2021-07-31T01:51:32.563 INFO:tasks.rados.rados.0.smithi045.stdout:1802: finishing write tid 4 to smithi04569269-17 2021-07-31T01:51:32.563 INFO:tasks.rados.rados.0.smithi045.stdout:1802: finishing write tid 5 to smithi04569269-17 2021-07-31T01:51:32.563 INFO:tasks.rados.rados.0.smithi045.stdout:1802: finishing write tid 6 to smithi04569269-17 2021-07-31T01:51:32.563 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 17 v 1055 (ObjNum 569 snap 170 seq_num 569) dirty exists 2021-07-31T01:51:32.564 INFO:tasks.rados.rados.0.smithi045.stdout:1802: left oid 17 (ObjNum 569 snap 170 seq_num 569) 2021-07-31T01:51:32.597 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T01:51:32.877 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T01:51:32.877 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 6m ago 7m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T01:51:32.877 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 4m ago 6m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T01:51:32.877 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 4m ago 10m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T01:51:32.878 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 6m ago 12m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T01:51:32.878 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 6m ago 12m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T01:51:32.878 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 4m ago 10m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T01:51:32.878 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 6m ago 10m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T01:51:32.878 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 6m ago 7m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T01:51:32.879 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 4m ago 7m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T01:51:32.879 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 6m ago 10m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T01:51:32.879 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 6m ago 9m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T01:51:32.879 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 6m ago 9m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T01:51:32.880 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 6m ago 9m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T01:51:32.880 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 4m ago 8m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T01:51:32.880 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 4m ago 8m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T01:51:32.880 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 4m ago 8m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T01:51:32.880 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 4m ago 7m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T01:51:32.881 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 4m ago 7m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T01:51:33.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:32 smithi154 conmon[34553]: cluster 2021-07-31T01:51:31.851771+0000 mgr.x (mgr.24542) 186 : cluster [DBG] 2021-07-31T01:51:33.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:32 smithi154 conmon[34553]: pgmap v368: 25 pgs: 25 active+clean; 194 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 19 MiB/s rd, 8.5 MiB/s wr, 33 op/s 2021-07-31T01:51:33.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:32 smithi154 conmon[34553]: cluster 2021-07-31T01:51:32.560923+0000 mon.a (mon.0) 1041 : cluster [DBG] osdmap e393: 8 total, 8 up, 8 in 2021-07-31T01:51:33.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:32 smithi045 conmon[33655]: cluster 2021-07-31T01:51:31.851771+0000 mgr.x ( 2021-07-31T01:51:33.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:32 smithi045 conmon[33655]: mgr.24542) 186 : cluster [DBG] pgmap v368: 25 pgs: 25 active+clean; 194 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 19 MiB/s rd, 8.5 MiB/s wr, 33 op/s 2021-07-31T01:51:33.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:32 smithi045 conmon[33655]: cluster 2021-07-31T01:51:32.560923+0000 mon.a (mon.0) 1041 2021-07-31T01:51:33.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:32 smithi045 conmon[33655]: : cluster [DBG] osdmap e393: 8 total, 8 up, 8 in 2021-07-31T01:51:33.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:32 smithi045 conmon[39718]: cluster 2021-07-31T01:51:31.851771+0000 mgr.x (mgr.24542) 186 : cluster 2021-07-31T01:51:33.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:32 smithi045 conmon[39718]: [DBG] pgmap v368: 25 pgs: 25 active+clean; 194 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 19 MiB/s rd, 8.5 MiB/s wr, 33 op/s 2021-07-31T01:51:33.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:32 smithi045 conmon[39718]: cluster 2021-07-31 2021-07-31T01:51:33.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:32 smithi045 conmon[39718]: T01:51:32.560923+0000 mon.a (mon 2021-07-31T01:51:33.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:32 smithi045 conmon[39718]: .0) 1041 : cluster [DBG] 2021-07-31T01:51:33.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:32 smithi045 conmon[39718]: osdmap e393: 8 total, 8 up, 8 in 2021-07-31T01:51:33.212 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T01:51:33.213 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T01:51:33.213 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T01:51:33.213 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T01:51:33.213 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T01:51:33.214 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T01:51:33.214 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T01:51:33.214 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T01:51:33.215 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T01:51:33.215 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T01:51:33.215 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T01:51:33.215 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T01:51:33.216 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T01:51:33.216 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T01:51:33.216 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T01:51:33.216 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T01:51:33.217 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T01:51:33.562 INFO:tasks.rados.rados.0.smithi045.stdout:1802: done (2 left) 2021-07-31T01:51:33.563 INFO:tasks.rados.rados.0.smithi045.stdout:1803: done (1 left) 2021-07-31T01:51:33.563 INFO:tasks.rados.rados.0.smithi045.stdout:1806: done (0 left) 2021-07-31T01:51:33.563 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 10 current snap is 171 2021-07-31T01:51:33.563 INFO:tasks.rados.rados.0.smithi045.stdout:1807: seq_num 571 ranges {0=770048,770048=483328,1253376=753664,2007040=368640} 2021-07-31T01:51:33.566 INFO:tasks.rados.rados.0.smithi045.stdout:1807: writing smithi04569269-10 from 0 to 770048 tid 1 2021-07-31T01:51:33.569 INFO:tasks.rados.rados.0.smithi045.stdout:1807: writing smithi04569269-10 from 770048 to 1253376 tid 2 2021-07-31T01:51:33.573 INFO:tasks.rados.rados.0.smithi045.stdout:1807: writing smithi04569269-10 from 1253376 to 2007040 tid 3 2021-07-31T01:51:33.575 INFO:tasks.rados.rados.0.smithi045.stdout:1807: writing smithi04569269-10 from 2007040 to 2375680 tid 4 2021-07-31T01:51:33.575 INFO:tasks.rados.rados.0.smithi045.stdout:1808: snap_create 2021-07-31T01:51:33.576 INFO:tasks.rados.rados.0.smithi045.stdout:1807: finishing write tid 1 to smithi04569269-10 2021-07-31T01:51:33.576 INFO:tasks.rados.rados.0.smithi045.stdout:1807: finishing write tid 2 to smithi04569269-10 2021-07-31T01:51:33.577 INFO:tasks.rados.rados.0.smithi045.stdout:1807: finishing write tid 3 to smithi04569269-10 2021-07-31T01:51:33.578 INFO:tasks.rados.rados.0.smithi045.stdout:1807: finishing write tid 4 to smithi04569269-10 2021-07-31T01:51:33.579 INFO:tasks.rados.rados.0.smithi045.stdout:1807: finishing write tid 5 to smithi04569269-10 2021-07-31T01:51:33.580 INFO:tasks.rados.rados.0.smithi045.stdout:1807: finishing write tid 6 to smithi04569269-10 2021-07-31T01:51:33.580 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 10 v 1060 (ObjNum 571 snap 171 seq_num 571) dirty exists 2021-07-31T01:51:33.580 INFO:tasks.rados.rados.0.smithi045.stdout:1807: left oid 10 (ObjNum 571 snap 171 seq_num 571) 2021-07-31T01:51:33.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:51:33 smithi045 conmon[61210]: level=error ts=2021-07-31T01:51:33.516Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:51:33.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:51:33 smithi045 conmon[61210]: level=error ts=2021-07-31T01:51:33.516Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:51:33.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:51:33 smithi045 conmon[61210]: level=error ts=2021-07-31T01:51:33.516Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T01:51:33.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:33 smithi045 conmon[33655]: audit 2021-07-31T01:51:32.585387+0000 mgr.x ( 2021-07-31T01:51:33.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:33 smithi045 conmon[33655]: mgr.24542) 187 : audit [DBG] from='client.24763 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:51:33.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:33 smithi045 conmon[33655]: audit 2021-07-31T01:51:32.868566+0000 mgr.x (mgr.24542) 188 : audit [DBG] from='client.14862 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:51:33.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:33 smithi045 conmon[33655]: audit 2021-07-31T 2021-07-31T01:51:33.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:33 smithi045 conmon[33655]: 01:51:33.211424+0000 mon.a (mon.0) 2021-07-31T01:51:33.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:33 smithi045 conmon[33655]: 1042 : audit [DBG] from='client.? 172.21.15.45:0/2587173188' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T01:51:33.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:33 smithi045 conmon[33655]: cluster 2021 2021-07-31T01:51:33.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:33 smithi045 conmon[33655]: -07-31T01:51:33.562191+0000 2021-07-31T01:51:33.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:33 smithi045 conmon[33655]: mon.a (mon.0) 1043 : cluster [DBG] osdmap e394: 8 total, 8 up, 8 in 2021-07-31T01:51:33.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:33 smithi045 conmon[33655]: 2021-07-31T01:51:33.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:33 smithi045 conmon[39718]: audit 2021-07-31T01:51:32.585387+0000 mgr.x ( 2021-07-31T01:51:33.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:33 smithi045 conmon[39718]: mgr.24542) 187 : audit [DBG] from='client.24763 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:51:33.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:33 smithi045 conmon[39718]: audit 2021-07-31T01:51:32.868566+0000 2021-07-31T01:51:33.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:33 smithi045 conmon[39718]: mgr.x (mgr.24542) 188 : audit [DBG] from='client.14862 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:51:33.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:33 smithi045 conmon[39718]: audit 2021-07-31T01:51:33.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:33 smithi045 conmon[39718]: 2021-07-31T01:51:33.211424+0000 mon.a (mon.0) 1042 : audit 2021-07-31T01:51:33.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:33 smithi045 conmon[39718]: [DBG] from='client.? 172.21.15.45:0/2587173188' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T01:51:33.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:33 smithi045 conmon[39718]: cluster 2021-07-31T01:51:33.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:33 smithi045 conmon[39718]: 2021-07-31T01:51:33.562191+0000 mon.a (mon.0) 1043 : cluster 2021-07-31T01:51:33.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:33 smithi045 conmon[39718]: [DBG] osdmap e394: 8 total, 8 up, 8 in 2021-07-31T01:51:34.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:33 smithi154 conmon[34553]: audit 2021-07-31T01:51:32.585387+0000 mgr.x (mgr.24542) 187 : audit 2021-07-31T01:51:34.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:33 smithi154 conmon[34553]: [DBG] from='client.24763 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:51:34.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:33 smithi154 conmon[34553]: audit 2021-07-31T01:51:32.868566+0000 mgr.x (mgr.24542) 188 : audit [DBG] from='client.14862 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:51:34.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:33 smithi154 conmon[34553]: audit 2021-07-31T01:51 2021-07-31T01:51:34.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:33 smithi154 conmon[34553]: :33.211424+0000 mon.a (mon.0) 1042 : audit [DBG] from='client.? 172.21.15.45:0/2587173188' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T01:51:34.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:33 smithi154 conmon[34553]: cluster 2021-07 2021-07-31T01:51:34.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:33 smithi154 conmon[34553]: -31T01:51:33.562191+0000 mon.a (mon. 2021-07-31T01:51:34.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:33 smithi154 conmon[34553]: 0) 1043 : cluster [DBG] osdmap e394: 8 total, 8 up, 8 in 2021-07-31T01:51:34.567 INFO:tasks.rados.rados.0.smithi045.stdout:1807: done (1 left) 2021-07-31T01:51:34.568 INFO:tasks.rados.rados.0.smithi045.stdout:1808: done (0 left) 2021-07-31T01:51:34.568 INFO:tasks.rados.rados.0.smithi045.stdout:1809: rollback oid 27 current snap is 172 2021-07-31T01:51:34.568 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 27 to 170 2021-07-31T01:51:34.568 INFO:tasks.rados.rados.0.smithi045.stdout:1810: delete oid 39 current snap is 172 2021-07-31T01:51:34.569 INFO:tasks.rados.rados.0.smithi045.stdout:1810: done (1 left) 2021-07-31T01:51:34.569 INFO:tasks.rados.rados.0.smithi045.stdout:1811: rollback oid 5 current snap is 172 2021-07-31T01:51:34.570 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 5 to 166 2021-07-31T01:51:34.570 INFO:tasks.rados.rados.0.smithi045.stdout:1812: copy_from oid 6 from oid 24 current snap is 172 2021-07-31T01:51:34.570 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 17 current snap is 172 2021-07-31T01:51:34.570 INFO:tasks.rados.rados.0.smithi045.stdout:1813: seq_num 573 ranges {4636672=802816,5439488=655360,6094848=720896,6815744=229376} 2021-07-31T01:51:34.599 INFO:tasks.rados.rados.0.smithi045.stdout:1813: writing smithi04569269-17 from 4636672 to 5439488 tid 1 2021-07-31T01:51:34.602 INFO:tasks.rados.rados.0.smithi045.stdout:1813: writing smithi04569269-17 from 5439488 to 6094848 tid 2 2021-07-31T01:51:34.607 INFO:tasks.rados.rados.0.smithi045.stdout:1813: writing smithi04569269-17 from 6094848 to 6815744 tid 3 2021-07-31T01:51:34.608 INFO:tasks.rados.rados.0.smithi045.stdout:1813: writing smithi04569269-17 from 6815744 to 7045120 tid 4 2021-07-31T01:51:34.609 INFO:tasks.rados.rados.0.smithi045.stdout:1814: snap_create 2021-07-31T01:51:34.609 INFO:tasks.rados.rados.0.smithi045.stdout:1811: finishing rollback tid 1 to smithi04569269-5 2021-07-31T01:51:34.609 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 5 v 0 (ObjNum -1105889024 snap 21873 seq_num 3189757952) dirty dne 2021-07-31T01:51:34.609 INFO:tasks.rados.rados.0.smithi045.stdout:1809: finishing rollback tid 0 to smithi04569269-27 2021-07-31T01:51:34.610 INFO:tasks.rados.rados.0.smithi045.stdout:1809: finishing rollback tid 1 to smithi04569269-27 2021-07-31T01:51:34.610 INFO:tasks.rados.rados.0.smithi045.stdout:1809: finishing rollback tid 2 to smithi04569269-27 2021-07-31T01:51:34.610 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 27 v 1038 (ObjNum 523 snap 154 seq_num 523) dirty exists 2021-07-31T01:51:34.610 INFO:tasks.rados.rados.0.smithi045.stdout:1812: finishing copy_from to smithi04569269-6 2021-07-31T01:51:34.611 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 6 v 523 (ObjNum 557 snap 166 seq_num 557) dirty exists 2021-07-31T01:51:34.611 INFO:tasks.rados.rados.0.smithi045.stdout:1812: finishing copy_from racing read to smithi04569269-6 2021-07-31T01:51:34.611 INFO:tasks.rados.rados.0.smithi045.stdout:1813: finishing write tid 1 to smithi04569269-17 2021-07-31T01:51:34.611 INFO:tasks.rados.rados.0.smithi045.stdout:1813: finishing write tid 2 to smithi04569269-17 2021-07-31T01:51:34.612 INFO:tasks.rados.rados.0.smithi045.stdout:1813: finishing write tid 3 to smithi04569269-17 2021-07-31T01:51:34.612 INFO:tasks.rados.rados.0.smithi045.stdout:1813: finishing write tid 4 to smithi04569269-17 2021-07-31T01:51:34.613 INFO:tasks.rados.rados.0.smithi045.stdout:1813: finishing write tid 5 to smithi04569269-17 2021-07-31T01:51:34.613 INFO:tasks.rados.rados.0.smithi045.stdout:1813: finishing write tid 6 to smithi04569269-17 2021-07-31T01:51:34.614 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 17 v 1065 (ObjNum 573 snap 172 seq_num 573) dirty exists 2021-07-31T01:51:34.614 INFO:tasks.rados.rados.0.smithi045.stdout:1813: left oid 17 (ObjNum 573 snap 172 seq_num 573) 2021-07-31T01:51:35.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:34 smithi154 conmon[34553]: cluster 2021-07-31T01:51:33.852346+0000 mgr.x (mgr.24542) 189 : cluster [DBG] pgmap v371: 25 pgs: 25 active+clean; 208 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 7.6 MiB/s rd, 3.5 MiB/s wr, 16 op/s 2021-07-31T01:51:35.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:34 smithi154 conmon[34553]: cluster 2021-07-31T01:51:34.564565+0000 mon.a (mon.0) 1044 : cluster [DBG] osdmap e395: 8 total, 8 up, 8 in 2021-07-31T01:51:35.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:34 smithi045 conmon[39718]: cluster 2021-07-31T01:51:33.852346+0000 2021-07-31T01:51:35.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:34 smithi045 conmon[39718]: mgr.x (mgr.24542) 189 : cluster [DBG] pgmap v371: 25 pgs: 25 active+clean; 208 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 7.6 MiB/s rd, 3.5 MiB/s wr, 16 op/s 2021-07-31T01:51:35.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:34 smithi045 conmon[39718]: cluster 2021-07-31T01:51:34. 2021-07-31T01:51:35.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:34 smithi045 conmon[39718]: 564565+0000 mon.a (mon.0) 1044 : cluster [DBG] osdmap e395: 8 total, 8 up, 8 in 2021-07-31T01:51:35.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:34 smithi045 conmon[33655]: cluster 2021-07-31T01:51:33.852346+0000 mgr.x (mgr.24542) 189 : 2021-07-31T01:51:35.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:34 smithi045 conmon[33655]: cluster [DBG] pgmap v371: 25 pgs: 25 active+clean; 208 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 7.6 MiB/s rd, 3.5 MiB/s wr, 16 op/s 2021-07-31T01:51:35.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:34 smithi045 conmon[33655]: cluster 2021-07-31T01:51:34.564565+0000 mon.a (mon.0) 1044 : cluster [DBG] osdmap e395: 8 total, 8 up, 8 in 2021-07-31T01:51:35.566 INFO:tasks.rados.rados.0.smithi045.stdout:1809: done (4 left) 2021-07-31T01:51:35.567 INFO:tasks.rados.rados.0.smithi045.stdout:1811: done (3 left) 2021-07-31T01:51:35.567 INFO:tasks.rados.rados.0.smithi045.stdout:1812: done (2 left) 2021-07-31T01:51:35.567 INFO:tasks.rados.rados.0.smithi045.stdout:1813: done (1 left) 2021-07-31T01:51:35.568 INFO:tasks.rados.rados.0.smithi045.stdout:1814: done (0 left) 2021-07-31T01:51:35.568 INFO:tasks.rados.rados.0.smithi045.stdout:1815: snap_remove snap 170 2021-07-31T01:51:36.557 INFO:tasks.rados.rados.0.smithi045.stdout:1815: done (0 left) 2021-07-31T01:51:36.558 INFO:tasks.rados.rados.0.smithi045.stdout:1816: read oid 42 snap 171 2021-07-31T01:51:36.558 INFO:tasks.rados.rados.0.smithi045.stdout:1816: expect (ObjNum 567 snap 168 seq_num 567) 2021-07-31T01:51:36.558 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 1 current snap is 173 2021-07-31T01:51:36.559 INFO:tasks.rados.rados.0.smithi045.stdout:1817: seq_num 574 ranges {2203648=16384} 2021-07-31T01:51:36.570 INFO:tasks.rados.rados.0.smithi045.stdout:1817: writing smithi04569269-1 from 2203648 to 2220032 tid 1 2021-07-31T01:51:36.570 INFO:tasks.rados.rados.0.smithi045.stdout:1818: snap_create 2021-07-31T01:51:36.573 INFO:tasks.rados.rados.0.smithi045.stdout:1817: finishing write tid 1 to smithi04569269-1 2021-07-31T01:51:36.574 INFO:tasks.rados.rados.0.smithi045.stdout:1817: finishing write tid 2 to smithi04569269-1 2021-07-31T01:51:36.576 INFO:tasks.rados.rados.0.smithi045.stdout:1817: finishing write tid 3 to smithi04569269-1 2021-07-31T01:51:36.576 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 1 v 1004 (ObjNum 574 snap 173 seq_num 574) dirty exists 2021-07-31T01:51:36.576 INFO:tasks.rados.rados.0.smithi045.stdout:1817: left oid 1 (ObjNum 574 snap 173 seq_num 574) 2021-07-31T01:51:36.594 INFO:tasks.rados.rados.0.smithi045.stdout:1816: expect (ObjNum 567 snap 168 seq_num 567) 2021-07-31T01:51:37.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:36 smithi154 conmon[34553]: cluster 2021-07-31T01:51:35.566224+0000 mon.a (mon.0) 2021-07-31T01:51:37.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:36 smithi154 conmon[34553]: 1045 : cluster [DBG] osdmap e396: 8 total, 8 up, 8 in 2021-07-31T01:51:37.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:36 smithi154 conmon[34553]: cluster 2021-07-31T01:51:35.852693+0000 mgr.x (mgr.24542) 190 : cluster [DBG] pgmap v374: 25 pgs: 25 active+clean; 208 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 7.6 MiB/s rd, 3.5 MiB/s wr, 16 op/s 2021-07-31T01:51:37.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:36 smithi154 conmon[34553]: cluster 2021-07-31T01:51:37.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:36 smithi154 conmon[34553]: 2021-07-31T01:51:36.556574+0000 mon.a (mon.0) 1046 : cluster [DBG] osdmap e397: 8 total, 8 up, 8 in 2021-07-31T01:51:37.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:36 smithi045 conmon[33655]: cluster 2021-07-31T01:51:35.566224+0000 mon.a ( 2021-07-31T01:51:37.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:36 smithi045 conmon[33655]: mon.0) 1045 : cluster [DBG] osdmap e396: 8 total, 8 up, 8 in 2021-07-31T01:51:37.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:36 smithi045 conmon[33655]: cluster 2021-07-31T01:51:35.852693+0000 mgr.x (mgr.24542) 190 : cluster [DBG] pgmap v374: 25 pgs: 25 active+clean; 208 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 7.6 MiB/s rd, 3.5 MiB/s wr, 16 op/s 2021-07-31T01:51:37.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:36 smithi045 conmon[33655]: cluster 2021-07-31T01:51:36.556574+0000 mon.a (mon.0) 1046 : cluster [DBG] osdmap e397: 8 total, 8 up, 8 in 2021-07-31T01:51:37.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:36 smithi045 conmon[33655]: 2021-07-31T01:51:37.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:36 smithi045 conmon[39718]: cluster 2021-07-31T01:51:37.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:36 smithi045 conmon[39718]: 2021-07-31T01:51: 2021-07-31T01:51:37.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:36 smithi045 conmon[39718]: 35.566224+0000 mon.a (mon.0) 1045 : cluster [DBG] osdmap e396: 8 total, 8 up, 8 in 2021-07-31T01:51:37.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:36 smithi045 conmon[39718]: cluster 2021-07-31T01 2021-07-31T01:51:37.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:36 smithi045 conmon[39718]: :51:35 2021-07-31T01:51:37.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:36 smithi045 conmon[39718]: .852693+0000 mgr.x (mgr.24542) 190 2021-07-31T01:51:37.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:36 smithi045 conmon[39718]: : cluster [DBG] pgmap v374: 25 pgs: 25 active+clean; 208 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 7.6 MiB/s rd, 3.5 MiB/s wr, 16 op/s 2021-07-31T01:51:37.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:36 smithi045 conmon[39718]: cluster 2021-07-31 2021-07-31T01:51:37.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:36 smithi045 conmon[39718]: T01:51:36.556574+0000 mon.a (mon 2021-07-31T01:51:37.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:36 smithi045 conmon[39718]: .0) 1046 : cluster [DBG] osdmap e397: 8 total, 8 up, 8 in 2021-07-31T01:51:37.576 INFO:tasks.rados.rados.0.smithi045.stdout:1816: done (2 left) 2021-07-31T01:51:37.576 INFO:tasks.rados.rados.0.smithi045.stdout:1817: done (1 left) 2021-07-31T01:51:37.576 INFO:tasks.rados.rados.0.smithi045.stdout:1818: done (0 left) 2021-07-31T01:51:37.577 INFO:tasks.rados.rados.0.smithi045.stdout:1819: read oid 48 snap -1 2021-07-31T01:51:37.577 INFO:tasks.rados.rados.0.smithi045.stdout:1819: expect (ObjNum 525 snap 155 seq_num 525) 2021-07-31T01:51:37.577 INFO:tasks.rados.rados.0.smithi045.stdout:1820: snap_create 2021-07-31T01:51:37.584 INFO:tasks.rados.rados.0.smithi045.stdout:1819: expect (ObjNum 525 snap 155 seq_num 525) 2021-07-31T01:51:38.583 INFO:tasks.rados.rados.0.smithi045.stdout:1819: done (1 left) 2021-07-31T01:51:38.584 INFO:tasks.rados.rados.0.smithi045.stdout:1820: done (0 left) 2021-07-31T01:51:38.584 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 22 current snap is 175 2021-07-31T01:51:38.584 INFO:tasks.rados.rados.0.smithi045.stdout:1821: seq_num 575 ranges {0=32768} 2021-07-31T01:51:38.585 INFO:tasks.rados.rados.0.smithi045.stdout:1821: writing smithi04569269-22 from 0 to 32768 tid 1 2021-07-31T01:51:38.585 INFO:tasks.rados.rados.0.smithi045.stdout:1822: copy_from oid 50 from oid 15 current snap is 175 2021-07-31T01:51:38.585 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 8 current snap is 175 2021-07-31T01:51:38.585 INFO:tasks.rados.rados.0.smithi045.stdout:1823: seq_num 576 ranges {0=49152} 2021-07-31T01:51:38.586 INFO:tasks.rados.rados.0.smithi045.stdout:1823: writing smithi04569269-8 from 0 to 49152 tid 1 2021-07-31T01:51:38.586 INFO:tasks.rados.rados.0.smithi045.stdout:1824: copy_from oid 31 from oid 27 current snap is 175 2021-07-31T01:51:38.586 INFO:tasks.rados.rados.0.smithi045.stdout:1825: snap_create 2021-07-31T01:51:38.588 INFO:tasks.rados.rados.0.smithi045.stdout:1823: finishing write tid 1 to smithi04569269-8 2021-07-31T01:51:38.588 INFO:tasks.rados.rados.0.smithi045.stdout:1821: finishing write tid 1 to smithi04569269-22 2021-07-31T01:51:38.589 INFO:tasks.rados.rados.0.smithi045.stdout:1823: finishing write tid 2 to smithi04569269-8 2021-07-31T01:51:38.589 INFO:tasks.rados.rados.0.smithi045.stdout:1821: finishing write tid 2 to smithi04569269-22 2021-07-31T01:51:38.589 INFO:tasks.rados.rados.0.smithi045.stdout:1823: finishing write tid 3 to smithi04569269-8 2021-07-31T01:51:38.590 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 8 v 1151 (ObjNum 576 snap 175 seq_num 576) dirty exists 2021-07-31T01:51:38.590 INFO:tasks.rados.rados.0.smithi045.stdout:1823: left oid 8 (ObjNum 576 snap 175 seq_num 576) 2021-07-31T01:51:38.590 INFO:tasks.rados.rados.0.smithi045.stdout:1821: finishing write tid 3 to smithi04569269-22 2021-07-31T01:51:38.591 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 22 v 1025 (ObjNum 575 snap 175 seq_num 575) dirty exists 2021-07-31T01:51:38.591 INFO:tasks.rados.rados.0.smithi045.stdout:1821: left oid 22 (ObjNum 575 snap 175 seq_num 575) 2021-07-31T01:51:38.606 INFO:tasks.rados.rados.0.smithi045.stdout:1824: finishing copy_from to smithi04569269-31 2021-07-31T01:51:38.606 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 31 v 1038 (ObjNum 523 snap 154 seq_num 523) dirty exists 2021-07-31T01:51:38.607 INFO:tasks.rados.rados.0.smithi045.stdout:1824: finishing copy_from racing read to smithi04569269-31 2021-07-31T01:51:38.625 INFO:tasks.rados.rados.0.smithi045.stdout:1822: finishing copy_from to smithi04569269-50 2021-07-31T01:51:38.626 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 50 v 416 (ObjNum 537 snap 161 seq_num 537) dirty exists 2021-07-31T01:51:38.626 INFO:tasks.rados.rados.0.smithi045.stdout:1822: finishing copy_from racing read to smithi04569269-50 2021-07-31T01:51:39.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:38 smithi154 conmon[34553]: cluster 2021-07-31T01:51:37.575255+0000 mon.a (mon.0) 1047 2021-07-31T01:51:39.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:38 smithi154 conmon[34553]: : cluster [DBG] osdmap e398: 8 total, 8 up, 8 in 2021-07-31T01:51:39.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:38 smithi154 conmon[34553]: cluster 2021-07-31T01:51:37.853141+0000 mgr.x (mgr.24542) 191 2021-07-31T01:51:39.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:38 smithi154 conmon[34553]: : cluster [DBG] pgmap v377: 25 pgs: 25 active+clean; 208 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2021-07-31T01:51:39.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:38 smithi045 conmon[39718]: cluster 2021-07-31T01:51:37.575255+0000 2021-07-31T01:51:39.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:38 smithi045 conmon[39718]: mon.a (mon.0) 1047 : cluster [DBG] osdmap e398: 8 total, 8 up, 8 in 2021-07-31T01:51:39.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:38 smithi045 conmon[39718]: cluster 2021-07 2021-07-31T01:51:39.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:38 smithi045 conmon[39718]: -31T01:51:37.853141+0000 mgr.x (mgr.24542) 191 : 2021-07-31T01:51:39.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:38 smithi045 conmon[39718]: cluster [DBG] pgmap v377: 25 pgs: 25 active+clean; 208 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2021-07-31T01:51:39.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:38 smithi045 conmon[33655]: cluster 2021-07-31T01:51:37.575255+0000 mon.a (mon.0) 1047 : cluster 2021-07-31T01:51:39.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:38 smithi045 conmon[33655]: [DBG] osdmap e398: 8 total, 8 up, 8 in 2021-07-31T01:51:39.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:38 smithi045 conmon[33655]: cluster 2021-07-31T01:51:37.853141+0000 mgr.x (mgr.24542) 191 : cluster [DBG] pgmap v377: 25 pgs: 25 active+clean; 208 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2021-07-31T01:51:39.582 INFO:tasks.rados.rados.0.smithi045.stdout:1821: done (4 left) 2021-07-31T01:51:39.582 INFO:tasks.rados.rados.0.smithi045.stdout:1822: done (3 left) 2021-07-31T01:51:39.582 INFO:tasks.rados.rados.0.smithi045.stdout:1823: done (2 left) 2021-07-31T01:51:39.582 INFO:tasks.rados.rados.0.smithi045.stdout:1824: done (1 left) 2021-07-31T01:51:39.583 INFO:tasks.rados.rados.0.smithi045.stdout:1825: done (0 left) 2021-07-31T01:51:39.583 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 43 current snap is 176 2021-07-31T01:51:39.583 INFO:tasks.rados.rados.0.smithi045.stdout:1826: seq_num 577 ranges {7061504=65536} 2021-07-31T01:51:39.620 INFO:tasks.rados.rados.0.smithi045.stdout:1826: writing smithi04569269-43 from 7061504 to 7127040 tid 1 2021-07-31T01:51:39.621 INFO:tasks.rados.rados.0.smithi045.stdout:1827: delete oid 18 current snap is 176 2021-07-31T01:51:39.625 INFO:tasks.rados.rados.0.smithi045.stdout:1827: done (1 left) 2021-07-31T01:51:39.625 INFO:tasks.rados.rados.0.smithi045.stdout:1828: read oid 13 snap -1 2021-07-31T01:51:39.625 INFO:tasks.rados.rados.0.smithi045.stdout:1828: expect (ObjNum 535 snap 161 seq_num 535) 2021-07-31T01:51:39.626 INFO:tasks.rados.rados.0.smithi045.stdout:1829: rollback oid 30 current snap is 176 2021-07-31T01:51:39.626 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 30 to 162 2021-07-31T01:51:39.626 INFO:tasks.rados.rados.0.smithi045.stdout:1830: read oid 45 snap 162 2021-07-31T01:51:39.626 INFO:tasks.rados.rados.0.smithi045.stdout:1830: expect (ObjNum 490 snap 146 seq_num 490) 2021-07-31T01:51:39.627 INFO:tasks.rados.rados.0.smithi045.stdout:1831: snap_create 2021-07-31T01:51:39.627 INFO:tasks.rados.rados.0.smithi045.stdout:1826: finishing write tid 1 to smithi04569269-43 2021-07-31T01:51:39.628 INFO:tasks.rados.rados.0.smithi045.stdout:1826: finishing write tid 2 to smithi04569269-43 2021-07-31T01:51:39.628 INFO:tasks.rados.rados.0.smithi045.stdout:1829: finishing rollback tid 1 to smithi04569269-30 2021-07-31T01:51:39.628 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 30 v 0 (ObjNum -1229693792 snap 32731 seq_num 0) dirty dne 2021-07-31T01:51:39.629 INFO:tasks.rados.rados.0.smithi045.stdout:1826: finishing write tid 3 to smithi04569269-43 2021-07-31T01:51:39.629 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 43 v 1040 (ObjNum 577 snap 176 seq_num 577) dirty exists 2021-07-31T01:51:39.629 INFO:tasks.rados.rados.0.smithi045.stdout:1826: left oid 43 (ObjNum 577 snap 176 seq_num 577) 2021-07-31T01:51:39.681 INFO:tasks.rados.rados.0.smithi045.stdout:1830: expect (ObjNum 490 snap 146 seq_num 490) 2021-07-31T01:51:39.958 INFO:tasks.rados.rados.0.smithi045.stdout:1828: expect (ObjNum 535 snap 161 seq_num 535) 2021-07-31T01:51:40.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:39 smithi154 conmon[34553]: cluster 2021-07-31T01:51:38.580042+0000 mon.a (mon.0) 1048 : cluster [DBG] osdmap e399: 8 total, 8 up, 8 in 2021-07-31T01:51:40.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:39 smithi045 conmon[39718]: cluster 2021-07-31T01:51:38.580042+0000 mon.a (mon.0 2021-07-31T01:51:40.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:39 smithi045 conmon[39718]: ) 1048 : cluster [DBG] osdmap e399: 8 total, 8 up, 8 in 2021-07-31T01:51:40.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:39 smithi045 conmon[33655]: cluster 2021-07-31T01:51:38.580042+0000 mon.a (mon.0) 1048 : cluster [DBG] osdmap e399: 8 total, 8 up, 8 in 2021-07-31T01:51:40.702 INFO:tasks.rados.rados.0.smithi045.stdout:1826: done (4 left) 2021-07-31T01:51:40.702 INFO:tasks.rados.rados.0.smithi045.stdout:1828: done (3 left) 2021-07-31T01:51:40.705 INFO:tasks.rados.rados.0.smithi045.stdout:1829: done (2 left) 2021-07-31T01:51:40.705 INFO:tasks.rados.rados.0.smithi045.stdout:1830: done (1 left) 2021-07-31T01:51:40.705 INFO:tasks.rados.rados.0.smithi045.stdout:1831: done (0 left) 2021-07-31T01:51:40.706 INFO:tasks.rados.rados.0.smithi045.stdout:1832: read oid 48 snap -1 2021-07-31T01:51:40.706 INFO:tasks.rados.rados.0.smithi045.stdout:1832: expect (ObjNum 525 snap 155 seq_num 525) 2021-07-31T01:51:40.707 INFO:tasks.rados.rados.0.smithi045.stdout:1833: read oid 25 snap 166 2021-07-31T01:51:40.707 INFO:tasks.rados.rados.0.smithi045.stdout:1833: expect (ObjNum 541 snap 162 seq_num 541) 2021-07-31T01:51:40.707 INFO:tasks.rados.rados.0.smithi045.stdout:1834: read oid 17 snap -1 2021-07-31T01:51:40.708 INFO:tasks.rados.rados.0.smithi045.stdout:1834: expect (ObjNum 573 snap 172 seq_num 573) 2021-07-31T01:51:40.708 INFO:tasks.rados.rados.0.smithi045.stdout:1835: snap_create 2021-07-31T01:51:40.710 INFO:tasks.rados.rados.0.smithi045.stdout:1832: expect (ObjNum 525 snap 155 seq_num 525) 2021-07-31T01:51:40.764 INFO:tasks.rados.rados.0.smithi045.stdout:1833: expect (ObjNum 541 snap 162 seq_num 541) 2021-07-31T01:51:41.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:40 smithi154 conmon[34553]: cluster 2021-07-31T01:51:39.581338+0000 mon.a (mon.0) 1049 : cluster [DBG] osdmap e400: 8 total, 8 up, 8 in 2021-07-31T01:51:41.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:40 smithi154 conmon[34553]: 2021-07-31T01:51:41.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:40 smithi154 conmon[34553]: cluster 2021-07-31T01:51:39.853624+0000 mgr.x (mgr.24542) 192 : cluster [DBG] pgmap v380: 25 pgs: 25 active+clean; 220 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 4.9 MiB/s rd, 3.1 MiB/s wr, 7 op/s 2021-07-31T01:51:41.005 INFO:tasks.rados.rados.0.smithi045.stdout:1834: expect (ObjNum 573 snap 172 seq_num 573) 2021-07-31T01:51:41.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:40 smithi045 conmon[39718]: cluster 2021-07-31T01:51:39. 2021-07-31T01:51:41.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:40 smithi045 conmon[39718]: 581338+0000 mon.a (mon.0) 1049 : cluster [DBG] osdmap e400: 8 total, 8 up, 8 in 2021-07-31T01:51:41.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:40 smithi045 conmon[39718]: cluster 2021-07-31T01:51:41.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:40 smithi045 conmon[39718]: 2021-07-31T01:51:39.853624+0000 mgr.x (mgr.24542 2021-07-31T01:51:41.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:40 smithi045 conmon[39718]: ) 192 : cluster [DBG] pgmap v380: 25 pgs: 25 active+clean; 220 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 4.9 MiB/s rd, 3.1 MiB/s wr, 7 op/s 2021-07-31T01:51:41.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:40 smithi045 conmon[33655]: cluster 2021-07-31T01:51:39.581338+0000 mon.a ( 2021-07-31T01:51:41.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:40 smithi045 conmon[33655]: mon.0) 1049 : cluster [DBG] osdmap e400: 8 total, 8 up, 8 in 2021-07-31T01:51:41.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:40 smithi045 conmon[33655]: cluster 2021-07-31T01:51:39.853624+0000 mgr.x (mgr.24542) 192 : cluster [DBG] pgmap v380: 25 pgs: 25 active+clean; 220 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 4.9 MiB/s rd, 3.1 MiB/s wr, 7 op/s 2021-07-31T01:51:41.558 INFO:tasks.rados.rados.0.smithi045.stdout:1832: done (3 left) 2021-07-31T01:51:41.559 INFO:tasks.rados.rados.0.smithi045.stdout:1833: done (2 left) 2021-07-31T01:51:41.559 INFO:tasks.rados.rados.0.smithi045.stdout:1834: done (1 left) 2021-07-31T01:51:41.560 INFO:tasks.rados.rados.0.smithi045.stdout:1835: done (0 left) 2021-07-31T01:51:41.560 INFO:tasks.rados.rados.0.smithi045.stdout:1836: delete oid 44 current snap is 178 2021-07-31T01:51:41.563 INFO:tasks.rados.rados.0.smithi045.stdout:1836: done (0 left) 2021-07-31T01:51:41.563 INFO:tasks.rados.rados.0.smithi045.stdout:1837: copy_from oid 39 from oid 22 current snap is 178 2021-07-31T01:51:41.564 INFO:tasks.rados.rados.0.smithi045.stdout:1838: snap_remove snap 168 2021-07-31T01:51:42.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:41 smithi154 conmon[34553]: cluster 2021-07-31T01:51:40.590197+0000 mon.a (mon.0) 1050 : cluster 2021-07-31T01:51:42.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:41 smithi154 conmon[34553]: [DBG] osdmap e401: 8 total, 8 up, 8 in 2021-07-31T01:51:42.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:41 smithi154 conmon[34553]: cluster 2021-07-31T01:51:41.556823+0000 mon.a (mon.0) 1051 : cluster [DBG] osdmap e402: 8 total, 8 up, 8 in 2021-07-31T01:51:42.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:41 smithi045 conmon[39718]: cluster 2021-07-31T01:51:40.590197 2021-07-31T01:51:42.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:41 smithi045 conmon[39718]: +0000 mon.a (mon.0) 1050 : cluster [DBG] osdmap e401: 8 total, 8 up, 8 in 2021-07-31T01:51:42.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:41 smithi045 conmon[39718]: cluster 2021-07-31T01:51:42.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:41 smithi045 conmon[39718]: 2021-07-31T01:51:41.556823+0000 mon.a (mon.0) 1051 : cluster 2021-07-31T01:51:42.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:41 smithi045 conmon[39718]: [DBG] osdmap e402: 8 total, 8 up, 8 in 2021-07-31T01:51:42.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:41 smithi045 conmon[33655]: cluster 2021-07-31T01:51:40.590197 2021-07-31T01:51:42.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:41 smithi045 conmon[33655]: +0000 mon.a (mon.0) 1050 : cluster [DBG] osdmap e401: 8 total, 8 up, 8 in 2021-07-31T01:51:42.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:41 smithi045 conmon[33655]: cluster 2021-07-31T01:51:41.556823+0000 mon.a (mon.0) 1051 : cluster [DBG] osdmap e402: 8 total, 8 up, 8 in 2021-07-31T01:51:42.564 INFO:tasks.rados.rados.0.smithi045.stdout:1838: done (1 left) 2021-07-31T01:51:42.565 INFO:tasks.rados.rados.0.smithi045.stdout:1839: read oid 14 snap -1 2021-07-31T01:51:42.565 INFO:tasks.rados.rados.0.smithi045.stdout:1839: expect (ObjNum 530 snap 156 seq_num 530) 2021-07-31T01:51:42.565 INFO:tasks.rados.rados.0.smithi045.stdout:1840: read oid 4 snap -1 2021-07-31T01:51:42.565 INFO:tasks.rados.rados.0.smithi045.stdout:1840: expect (ObjNum 470 snap 136 seq_num 470) 2021-07-31T01:51:42.566 INFO:tasks.rados.rados.0.smithi045.stdout:1841: setattr oid 46 current snap is 178 2021-07-31T01:51:42.566 INFO:tasks.rados.rados.0.smithi045.stdout:1837: finishing copy_from to smithi04569269-39 2021-07-31T01:51:42.566 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 39 v 1025 (ObjNum 575 snap 175 seq_num 575) dirty exists 2021-07-31T01:51:42.566 INFO:tasks.rados.rados.0.smithi045.stdout:1837: finishing copy_from racing read to smithi04569269-39 2021-07-31T01:51:42.567 INFO:tasks.rados.rados.0.smithi045.stdout:1837: done (3 left) 2021-07-31T01:51:42.568 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 29 current snap is 178 2021-07-31T01:51:42.568 INFO:tasks.rados.rados.0.smithi045.stdout:1842: seq_num 580 ranges {0=114688} 2021-07-31T01:51:42.569 INFO:tasks.rados.rados.0.smithi045.stdout:1842: writing smithi04569269-29 from 0 to 114688 tid 1 2021-07-31T01:51:42.569 INFO:tasks.rados.rados.0.smithi045.stdout:1843: snap_remove snap 160 2021-07-31T01:51:43.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:42 smithi154 conmon[34553]: cluster 2021-07-31T01:51:41.854036 2021-07-31T01:51:43.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:42 smithi154 conmon[34553]: +0000 mgr.x (mgr.24542) 193 : cluster [DBG] pgmap v383: 25 pgs: 25 active+clean; 220 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 4.9 MiB/s rd, 3.1 MiB/s wr, 7 op/s 2021-07-31T01:51:43.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:42 smithi154 conmon[34553]: cluster 2021-07-31T01:51:42.563102+0000 mon.a (mon.0) 1052 2021-07-31T01:51:43.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:42 smithi154 conmon[34553]: : cluster [DBG] osdmap e403: 8 total, 8 up, 8 in 2021-07-31T01:51:43.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:42 smithi045 conmon[39718]: cluster 2021-07-31T01:51:41.854036+0000 2021-07-31T01:51:43.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:42 smithi045 conmon[39718]: mgr.x (mgr.24542) 193 : cluster [DBG] pgmap v383: 25 pgs: 25 active+clean; 220 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 4.9 MiB/s rd, 3.1 MiB/s wr, 7 op/s 2021-07-31T01:51:43.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:42 smithi045 conmon[39718]: cluster 2021-07-31T01:51:43.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:42 smithi045 conmon[39718]: 2021-07-31T01:51 2021-07-31T01:51:43.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:42 smithi045 conmon[39718]: :42.563102+0000 mon.a (mon.0) 1052 : cluster 2021-07-31T01:51:43.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:42 smithi045 conmon[39718]: [DBG] osdmap e403: 8 total, 8 up, 8 in 2021-07-31T01:51:43.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:42 smithi045 conmon[33655]: cluster 2021-07-31T01:51:41.854036+0000 mgr.x ( 2021-07-31T01:51:43.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:42 smithi045 conmon[33655]: mgr.24542) 193 : cluster [DBG] pgmap v383: 25 pgs: 25 active+clean; 220 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 4.9 MiB/s rd, 3.1 MiB/s wr, 7 op/s 2021-07-31T01:51:43.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:42 smithi045 conmon[33655]: cluster 2021-07-31T01:51:42.563102+0000 mon.a (mon.0) 1052 : cluster [DBG] 2021-07-31T01:51:43.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:42 smithi045 conmon[33655]: osdmap e403: 8 total, 8 up, 8 in 2021-07-31T01:51:43.567 INFO:tasks.rados.rados.0.smithi045.stdout:1843: done (4 left) 2021-07-31T01:51:43.567 INFO:tasks.rados.rados.0.smithi045.stdout:1844: snap_create 2021-07-31T01:51:43.567 INFO:tasks.rados.rados.0.smithi045.stdout:1842: finishing write tid 1 to smithi04569269-29 2021-07-31T01:51:43.568 INFO:tasks.rados.rados.0.smithi045.stdout:1842: finishing write tid 2 to smithi04569269-29 2021-07-31T01:51:43.568 INFO:tasks.rados.rados.0.smithi045.stdout:1840: expect (ObjNum 470 snap 136 seq_num 470) 2021-07-31T01:51:43.705 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 46 v 281 (ObjNum 540 snap 162 seq_num 540) dirty exists 2021-07-31T01:51:43.706 INFO:tasks.rados.rados.0.smithi045.stdout:1842: finishing write tid 3 to smithi04569269-29 2021-07-31T01:51:43.706 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 29 v 923 (ObjNum 580 snap 178 seq_num 580) dirty exists 2021-07-31T01:51:43.706 INFO:tasks.rados.rados.0.smithi045.stdout:1842: left oid 29 (ObjNum 580 snap 178 seq_num 580) 2021-07-31T01:51:43.707 INFO:tasks.rados.rados.0.smithi045.stdout:1839: expect (ObjNum 530 snap 156 seq_num 530) 2021-07-31T01:51:43.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:51:43 smithi045 conmon[61210]: level=error ts=2021-07-31T01:51:43.516Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:51:43.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:51:43 smithi045 conmon[61210]: level=error ts=2021-07-31T01:51:43.516Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:51:43.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:51:43 smithi045 conmon[61210]: level=error ts=2021-07-31T01:51:43.516Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T01:51:44.571 INFO:tasks.rados.rados.0.smithi045.stdout:1839: done (4 left) 2021-07-31T01:51:44.572 INFO:tasks.rados.rados.0.smithi045.stdout:1840: done (3 left) 2021-07-31T01:51:44.572 INFO:tasks.rados.rados.0.smithi045.stdout:1841: done (2 left) 2021-07-31T01:51:44.572 INFO:tasks.rados.rados.0.smithi045.stdout:1842: done (1 left) 2021-07-31T01:51:44.573 INFO:tasks.rados.rados.0.smithi045.stdout:1844: done (0 left) 2021-07-31T01:51:44.573 INFO:tasks.rados.rados.0.smithi045.stdout:1845: delete oid 38 current snap is 179 2021-07-31T01:51:44.575 INFO:tasks.rados.rados.0.smithi045.stdout:1845: done (0 left) 2021-07-31T01:51:44.575 INFO:tasks.rados.rados.0.smithi045.stdout:1846: snap_remove snap 177 2021-07-31T01:51:45.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:44 smithi154 conmon[34553]: cluster 2021-07-31T01:51:43.565765+0000 mon.a (mon.0) 1053 : cluster [DBG] osdmap e404: 8 total, 8 up, 8 in 2021-07-31T01:51:45.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:44 smithi154 conmon[34553]: cluster 2021-07-31T01:51:43.854554+0000 mgr.x (mgr.24542) 194 : cluster [DBG] pgmap v386: 25 pgs: 25 active+clean; 208 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 30 MiB/s rd, 52 KiB/s wr, 14 op/s 2021-07-31T01:51:45.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:44 smithi045 conmon[33655]: cluster 2021-07-31T01:51:43.565765+0000 mon.a 2021-07-31T01:51:45.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:44 smithi045 conmon[33655]: (mon.0) 1053 : cluster [DBG] osdmap e404: 8 total, 8 up, 8 in 2021-07-31T01:51:45.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:44 smithi045 conmon[33655]: cluster 2021-07-31T01:51:43. 2021-07-31T01:51:45.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:44 smithi045 conmon[33655]: 854554+0000 mgr.x (mgr.24542) 194 : cluster [DBG] pgmap v386: 25 pgs: 25 active+clean; 208 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 30 MiB/s rd, 52 KiB/s wr, 14 op/s 2021-07-31T01:51:45.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:44 smithi045 conmon[39718]: cluster 2021-07-31T01:51:43.565765+0000 2021-07-31T01:51:45.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:44 smithi045 conmon[39718]: mon.a (mon.0) 1053 : cluster [DBG] osdmap e404: 8 total, 8 up, 8 in 2021-07-31T01:51:45.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:44 smithi045 conmon[39718]: cluster 2021-07-31T01:51: 2021-07-31T01:51:45.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:44 smithi045 conmon[39718]: 43.854554+0000 mgr.x (mgr.24542) 194 : cluster [DBG] pgmap v386: 25 pgs: 25 active+clean; 208 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 30 MiB/s rd, 52 KiB/s wr, 14 op/s 2021-07-31T01:51:45.576 INFO:tasks.rados.rados.0.smithi045.stdout:1846: done (0 left) 2021-07-31T01:51:45.577 INFO:tasks.rados.rados.0.smithi045.stdout:1847: rmattr oid 12 current snap is 179 2021-07-31T01:51:45.577 INFO:tasks.rados.rados.0.smithi045.stdout:1848: snap_remove snap 166 2021-07-31T01:51:46.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:45 smithi154 conmon[34553]: cluster 2021-07-31T01:51:44.570620+0000 mon.a (mon.0) 1054 : cluster [DBG] osdmap e405: 8 total, 8 up, 8 in 2021-07-31T01:51:46.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:45 smithi045 conmon[33655]: cluster 2021-07-31T01:51:44.570620+0000 mon.a (mon. 2021-07-31T01:51:46.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:45 smithi045 conmon[33655]: 0) 1054 : cluster [DBG] osdmap e405: 8 total, 8 up, 8 in 2021-07-31T01:51:46.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:45 smithi045 conmon[39718]: cluster 2021-07-31T01:51:44.570620+0000 mon.a (mon 2021-07-31T01:51:46.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:45 smithi045 conmon[39718]: .0) 1054 : cluster [DBG] osdmap e405: 8 total, 8 up, 8 in 2021-07-31T01:51:46.560 INFO:tasks.rados.rados.0.smithi045.stdout:1848: done (1 left) 2021-07-31T01:51:46.560 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 45 current snap is 179 2021-07-31T01:51:46.560 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 12 v 657 (ObjNum 506 snap 152 seq_num 506) dirty exists 2021-07-31T01:51:46.561 INFO:tasks.rados.rados.0.smithi045.stdout:1849: seq_num 582 ranges {5775360=155648} 2021-07-31T01:51:46.593 INFO:tasks.rados.rados.0.smithi045.stdout:1849: writing smithi04569269-45 from 5775360 to 5931008 tid 1 2021-07-31T01:51:46.594 INFO:tasks.rados.rados.0.smithi045.stdout:1847: done (1 left) 2021-07-31T01:51:46.594 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 37 current snap is 179 2021-07-31T01:51:46.594 INFO:tasks.rados.rados.0.smithi045.stdout:1850: seq_num 583 ranges {483328=172032} 2021-07-31T01:51:46.597 INFO:tasks.rados.rados.0.smithi045.stdout:1850: writing smithi04569269-37 from 483328 to 655360 tid 1 2021-07-31T01:51:46.597 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 1 current snap is 179 2021-07-31T01:51:46.597 INFO:tasks.rados.rados.0.smithi045.stdout:1851: seq_num 584 ranges {2220032=188416} 2021-07-31T01:51:46.611 INFO:tasks.rados.rados.0.smithi045.stdout:1851: writing smithi04569269-1 from 2220032 to 2408448 tid 1 2021-07-31T01:51:46.612 INFO:tasks.rados.rados.0.smithi045.stdout:1852: rollback oid 33 current snap is 179 2021-07-31T01:51:46.612 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 33 to 175 2021-07-31T01:51:46.612 INFO:tasks.rados.rados.0.smithi045.stdout:1849: finishing write tid 1 to smithi04569269-45 2021-07-31T01:51:46.613 INFO:tasks.rados.rados.0.smithi045.stdout:1849: finishing write tid 2 to smithi04569269-45 2021-07-31T01:51:46.613 INFO:tasks.rados.rados.0.smithi045.stdout:1850: finishing write tid 1 to smithi04569269-37 2021-07-31T01:51:46.613 INFO:tasks.rados.rados.0.smithi045.stdout:1853: rollback oid 7 current snap is 179 2021-07-31T01:51:46.613 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 7 to 178 2021-07-31T01:51:46.614 INFO:tasks.rados.rados.0.smithi045.stdout:1850: finishing write tid 2 to smithi04569269-37 2021-07-31T01:51:46.614 INFO:tasks.rados.rados.0.smithi045.stdout:1850: finishing write tid 3 to smithi04569269-37 2021-07-31T01:51:46.614 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 37 v 438 (ObjNum 583 snap 179 seq_num 583) dirty exists 2021-07-31T01:51:46.614 INFO:tasks.rados.rados.0.smithi045.stdout:1850: left oid 37 (ObjNum 583 snap 179 seq_num 583) 2021-07-31T01:51:46.615 INFO:tasks.rados.rados.0.smithi045.stdout:1849: finishing write tid 3 to smithi04569269-45 2021-07-31T01:51:46.615 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 45 v 1067 (ObjNum 582 snap 179 seq_num 582) dirty exists 2021-07-31T01:51:46.615 INFO:tasks.rados.rados.0.smithi045.stdout:1849: left oid 45 (ObjNum 582 snap 179 seq_num 582) 2021-07-31T01:51:46.617 INFO:tasks.rados.rados.0.smithi045.stdout:1849: done (4 left) 2021-07-31T01:51:46.617 INFO:tasks.rados.rados.0.smithi045.stdout:1850: done (3 left) 2021-07-31T01:51:46.617 INFO:tasks.rados.rados.0.smithi045.stdout:1854: copy_from oid 8 from oid 35 current snap is 179 2021-07-31T01:51:46.618 INFO:tasks.rados.rados.0.smithi045.stdout:1855: read oid 47 snap -1 2021-07-31T01:51:46.618 INFO:tasks.rados.rados.0.smithi045.stdout:1855: expect (ObjNum 172 snap 41 seq_num 172) 2021-07-31T01:51:46.619 INFO:tasks.rados.rados.0.smithi045.stdout:1856: rollback oid 9 current snap is 179 2021-07-31T01:51:46.619 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 9 to 171 2021-07-31T01:51:46.619 INFO:tasks.rados.rados.0.smithi045.stdout:1857: read oid 41 snap -1 2021-07-31T01:51:46.619 INFO:tasks.rados.rados.0.smithi045.stdout:1857: expect (ObjNum 552 snap 166 seq_num 552) 2021-07-31T01:51:46.619 INFO:tasks.rados.rados.0.smithi045.stdout:1858: copy_from oid 21 from oid 6 current snap is 179 2021-07-31T01:51:46.620 INFO:tasks.rados.rados.0.smithi045.stdout:1859: copy_from oid 32 from oid 39 current snap is 179 2021-07-31T01:51:46.620 INFO:tasks.rados.rados.0.smithi045.stdout:1860: snap_remove snap 173 2021-07-31T01:51:47.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:46 smithi154 conmon[34553]: cluster 2021-07-31T01:51:45.576186+0000 mon.a (mon.0) 1055 2021-07-31T01:51:47.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:46 smithi154 conmon[34553]: : cluster [DBG] osdmap e406: 8 total, 8 up, 8 in 2021-07-31T01:51:47.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:46 smithi154 conmon[34553]: cluster 2021-07-31T01:51:45.854998+0000 mgr.x (mgr.24542) 195 : cluster [DBG] pgmap v389: 25 pgs: 25 active+clean; 208 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 30 MiB/s rd, 52 KiB/s wr, 14 op/s 2021-07-31T01:51:47.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:46 smithi154 conmon[34553]: cluster 2021-07-31T 2021-07-31T01:51:47.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:46 smithi154 conmon[34553]: 01:51:46.559266+0000 mon.a (mon.0) 1056 : cluster [DBG] osdmap e407: 8 total, 8 up, 8 in 2021-07-31T01:51:47.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:46 smithi045 conmon[33655]: cluster 2021-07-31T01:51:45.576186+0000 mon.a (mon.0) 2021-07-31T01:51:47.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:46 smithi045 conmon[33655]: 1055 : cluster [DBG] osdmap e406: 8 total, 8 up, 8 in 2021-07-31T01:51:47.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:46 smithi045 conmon[33655]: cluster 2021-07-31T01:51:45.854998+0000 mgr.x (mgr.24542) 195 : cluster [DBG] pgmap v389: 25 pgs: 25 active+clean; 208 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 30 MiB/s rd, 52 KiB/s wr, 14 op/s 2021-07-31T01:51:47.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:46 smithi045 conmon[33655]: cluster 2021-07-31T01:51:47.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:46 smithi045 conmon[33655]: 2021-07-31T01:51:46.559266+0000 mon.a (mon.0) 1056 : cluster [DBG] osdmap e407: 8 total, 8 up, 8 in 2021-07-31T01:51:47.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:46 smithi045 conmon[39718]: cluster 2021-07-31T01:51:45.576186+0000 mon.a (mon. 2021-07-31T01:51:47.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:46 smithi045 conmon[39718]: 0) 1055 : cluster [DBG] osdmap e406: 8 total, 8 up, 8 in 2021-07-31T01:51:47.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:46 smithi045 conmon[39718]: cluster 2021-07-31T01:51:45.854998+0000 mgr.x 2021-07-31T01:51:47.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:46 smithi045 conmon[39718]: (mgr.24542) 195 : cluster [DBG] pgmap v389: 25 pgs: 25 active+clean; 208 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 30 MiB/s rd, 52 KiB/s wr, 14 op/s 2021-07-31T01:51:47.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:46 smithi045 conmon[39718]: cluster 2021-07-31T01: 2021-07-31T01:51:47.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:46 smithi045 conmon[39718]: 51:46.559266+0000 mon.a (mon.0) 1056 : cluster 2021-07-31T01:51:47.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:46 smithi045 conmon[39718]: [DBG] osdmap e407: 8 total, 8 up, 8 in 2021-07-31T01:51:47.587 INFO:tasks.rados.rados.0.smithi045.stdout:1860: done (9 left) 2021-07-31T01:51:47.587 INFO:tasks.rados.rados.0.smithi045.stdout:1861: read oid 16 snap -1 2021-07-31T01:51:47.587 INFO:tasks.rados.rados.0.smithi045.stdout:1861: expect (ObjNum 523 snap 154 seq_num 523) 2021-07-31T01:51:47.588 INFO:tasks.rados.rados.0.smithi045.stdout:1854: finishing copy_from to smithi04569269-8 2021-07-31T01:51:47.588 INFO:tasks.rados.rados.0.smithi045.stdout:1854: got expected ENOENT (src dne) 2021-07-31T01:51:47.588 INFO:tasks.rados.rados.0.smithi045.stdout:1854: finishing copy_from racing read to smithi04569269-8 2021-07-31T01:51:47.588 INFO:tasks.rados.rados.0.smithi045.stdout:1851: finishing write tid 1 to smithi04569269-1 2021-07-31T01:51:47.589 INFO:tasks.rados.rados.0.smithi045.stdout:1851: finishing write tid 2 to smithi04569269-1 2021-07-31T01:51:47.589 INFO:tasks.rados.rados.0.smithi045.stdout:1856: finishing rollback tid 0 to smithi04569269-9 2021-07-31T01:51:47.589 INFO:tasks.rados.rados.0.smithi045.stdout:1851: finishing write tid 3 to smithi04569269-1 2021-07-31T01:51:47.589 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 1 v 1006 (ObjNum 584 snap 179 seq_num 584) dirty exists 2021-07-31T01:51:47.589 INFO:tasks.rados.rados.0.smithi045.stdout:1851: left oid 1 (ObjNum 584 snap 179 seq_num 584) 2021-07-31T01:51:47.590 INFO:tasks.rados.rados.0.smithi045.stdout:1856: finishing rollback tid 1 to smithi04569269-9 2021-07-31T01:51:47.590 INFO:tasks.rados.rados.0.smithi045.stdout:1855: expect (ObjNum 172 snap 41 seq_num 172) 2021-07-31T01:51:47.612 INFO:tasks.rados.rados.0.smithi045.stdout:1856: finishing rollback tid 2 to smithi04569269-9 2021-07-31T01:51:47.612 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 9 v 395 (ObjNum 530 snap 156 seq_num 530) dirty exists 2021-07-31T01:51:47.613 INFO:tasks.rados.rados.0.smithi045.stdout:1852: finishing rollback tid 0 to smithi04569269-33 2021-07-31T01:51:47.613 INFO:tasks.rados.rados.0.smithi045.stdout:1862: read oid 3 snap 175 2021-07-31T01:51:47.613 INFO:tasks.rados.rados.0.smithi045.stdout:1862: expect (ObjNum 483 snap 143 seq_num 483) 2021-07-31T01:51:47.613 INFO:tasks.rados.rados.0.smithi045.stdout:1851: done (10 left) 2021-07-31T01:51:47.614 INFO:tasks.rados.rados.0.smithi045.stdout:1854: done (9 left) 2021-07-31T01:51:47.614 INFO:tasks.rados.rados.0.smithi045.stdout:1855: done (8 left) 2021-07-31T01:51:47.614 INFO:tasks.rados.rados.0.smithi045.stdout:1856: done (7 left) 2021-07-31T01:51:47.614 INFO:tasks.rados.rados.0.smithi045.stdout:1863: snap_create 2021-07-31T01:51:47.615 INFO:tasks.rados.rados.0.smithi045.stdout:1852: finishing rollback tid 1 to smithi04569269-33 2021-07-31T01:51:47.615 INFO:tasks.rados.rados.0.smithi045.stdout:1852: finishing rollback tid 2 to smithi04569269-33 2021-07-31T01:51:47.615 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 33 v 1069 (ObjNum 399 snap 121 seq_num 399) dirty exists 2021-07-31T01:51:47.615 INFO:tasks.rados.rados.0.smithi045.stdout:1859: finishing copy_from to smithi04569269-32 2021-07-31T01:51:47.615 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 32 v 1025 (ObjNum 575 snap 175 seq_num 575) dirty exists 2021-07-31T01:51:47.616 INFO:tasks.rados.rados.0.smithi045.stdout:1859: finishing copy_from racing read to smithi04569269-32 2021-07-31T01:51:47.616 INFO:tasks.rados.rados.0.smithi045.stdout:1858: finishing copy_from to smithi04569269-21 2021-07-31T01:51:47.616 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 21 v 523 (ObjNum 557 snap 166 seq_num 557) dirty exists 2021-07-31T01:51:47.616 INFO:tasks.rados.rados.0.smithi045.stdout:1858: finishing copy_from racing read to smithi04569269-21 2021-07-31T01:51:47.616 INFO:tasks.rados.rados.0.smithi045.stdout:1853: finishing rollback tid 0 to smithi04569269-7 2021-07-31T01:51:47.617 INFO:tasks.rados.rados.0.smithi045.stdout:1853: finishing rollback tid 1 to smithi04569269-7 2021-07-31T01:51:47.617 INFO:tasks.rados.rados.0.smithi045.stdout:1853: finishing rollback tid 2 to smithi04569269-7 2021-07-31T01:51:47.617 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 7 v 976 (ObjNum 1415537512 snap 1903317095 seq_num 1633837920) dirty exists 2021-07-31T01:51:47.617 INFO:tasks.rados.rados.0.smithi045.stdout:1857: expect (ObjNum 552 snap 166 seq_num 552) 2021-07-31T01:51:47.845 INFO:tasks.rados.rados.0.smithi045.stdout:1861: expect (ObjNum 523 snap 154 seq_num 523) 2021-07-31T01:51:47.958 INFO:tasks.rados.rados.0.smithi045.stdout:1862: expect (ObjNum 483 snap 143 seq_num 483) 2021-07-31T01:51:48.591 INFO:tasks.rados.rados.0.smithi045.stdout:1852: done (7 left) 2021-07-31T01:51:48.592 INFO:tasks.rados.rados.0.smithi045.stdout:1853: done (6 left) 2021-07-31T01:51:48.592 INFO:tasks.rados.rados.0.smithi045.stdout:1857: done (5 left) 2021-07-31T01:51:48.592 INFO:tasks.rados.rados.0.smithi045.stdout:1858: done (4 left) 2021-07-31T01:51:48.592 INFO:tasks.rados.rados.0.smithi045.stdout:1859: done (3 left) 2021-07-31T01:51:48.593 INFO:tasks.rados.rados.0.smithi045.stdout:1861: done (2 left) 2021-07-31T01:51:48.593 INFO:tasks.rados.rados.0.smithi045.stdout:1862: done (1 left) 2021-07-31T01:51:48.593 INFO:tasks.rados.rados.0.smithi045.stdout:1863: done (0 left) 2021-07-31T01:51:48.594 INFO:tasks.rados.rados.0.smithi045.stdout:1864: snap_remove snap 174 2021-07-31T01:51:49.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:48 smithi154 conmon[34553]: cluster 2021-07-31T01:51:47.585542+0000 mon.a (mon.0) 1057 : cluster [DBG] osdmap e408: 8 total, 8 up, 8 in 2021-07-31T01:51:49.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:48 smithi154 conmon[34553]: cluster 2021-07-31T01:51:47.855371+0000 mgr.x (mgr.24542) 196 : cluster [DBG] pgmap v392: 25 pgs: 25 active+clean; 208 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2021-07-31T01:51:49.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:48 smithi045 conmon[39718]: cluster 2021-07-31T01:51:47.585542+0000 mon.a 2021-07-31T01:51:49.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:48 smithi045 conmon[39718]: (mon.0) 1057 : cluster [DBG] osdmap e408: 8 total, 8 up, 8 in 2021-07-31T01:51:49.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:48 smithi045 conmon[39718]: cluster 2021-07-31T 2021-07-31T01:51:49.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:48 smithi045 conmon[39718]: 01:51:47.855371+0000 mgr.x (mgr.24542) 196 : cluster [DBG] pgmap v392: 25 pgs: 25 active+clean; 208 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2021-07-31T01:51:49.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:48 smithi045 conmon[33655]: cluster 2021-07-31T01:51:47.585542+0000 mon.a (mon.0 2021-07-31T01:51:49.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:48 smithi045 conmon[33655]: ) 1057 : cluster [DBG] osdmap e408: 8 total, 8 up, 8 in 2021-07-31T01:51:49.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:48 smithi045 conmon[33655]: cluster 2021-07-31T01:51:47.855371+0000 mgr.x (mgr.24542) 196 : cluster [DBG] pgmap v392: 25 pgs: 25 active+clean; 208 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2021-07-31T01:51:49.594 INFO:tasks.rados.rados.0.smithi045.stdout:1864: done (0 left) 2021-07-31T01:51:49.594 INFO:tasks.rados.rados.0.smithi045.stdout:1865: copy_from oid 3 from oid 28 current snap is 180 2021-07-31T01:51:49.595 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 45 current snap is 180 2021-07-31T01:51:49.595 INFO:tasks.rados.rados.0.smithi045.stdout:1866: seq_num 585 ranges {5931008=204800} 2021-07-31T01:51:49.629 INFO:tasks.rados.rados.0.smithi045.stdout:1866: writing smithi04569269-45 from 5931008 to 6135808 tid 1 2021-07-31T01:51:49.630 INFO:tasks.rados.rados.0.smithi045.stdout:1867: read oid 14 snap -1 2021-07-31T01:51:49.630 INFO:tasks.rados.rados.0.smithi045.stdout:1867: expect (ObjNum 530 snap 156 seq_num 530) 2021-07-31T01:51:49.630 INFO:tasks.rados.rados.0.smithi045.stdout:1865: finishing copy_from to smithi04569269-3 2021-07-31T01:51:49.631 INFO:tasks.rados.rados.0.smithi045.stdout:1865: got expected ENOENT (src dne) 2021-07-31T01:51:49.631 INFO:tasks.rados.rados.0.smithi045.stdout:1868: rollback oid 32 current snap is 180 2021-07-31T01:51:49.631 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 32 to 167 2021-07-31T01:51:49.632 INFO:tasks.rados.rados.0.smithi045.stdout:1865: finishing copy_from racing read to smithi04569269-3 2021-07-31T01:51:49.632 INFO:tasks.rados.rados.0.smithi045.stdout:1865: done (3 left) 2021-07-31T01:51:49.632 INFO:tasks.rados.rados.0.smithi045.stdout:1869: read oid 34 snap -1 2021-07-31T01:51:49.632 INFO:tasks.rados.rados.0.smithi045.stdout:1869: expect (ObjNum 399 snap 121 seq_num 399) 2021-07-31T01:51:49.632 INFO:tasks.rados.rados.0.smithi045.stdout:1870: read oid 22 snap -1 2021-07-31T01:51:49.633 INFO:tasks.rados.rados.0.smithi045.stdout:1870: expect (ObjNum 575 snap 175 seq_num 575) 2021-07-31T01:51:49.633 INFO:tasks.rados.rados.0.smithi045.stdout:1871: snap_remove snap 179 2021-07-31T01:51:50.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:49 smithi154 conmon[34553]: cluster 2021-07-31T01:51:48.589929+0000 mon.a (mon.0) 1058 : cluster [DBG] osdmap e409: 8 total, 8 up, 8 in 2021-07-31T01:51:50.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:49 smithi045 conmon[39718]: cluster 2021-07-31T01:51:48.589929+0000 mon.a (mon.0 2021-07-31T01:51:50.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:49 smithi045 conmon[39718]: ) 1058 : cluster [DBG] osdmap e409: 8 total, 8 up, 8 in 2021-07-31T01:51:50.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:49 smithi045 conmon[33655]: cluster 2021-07-31T01:51:50.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:49 smithi045 conmon[33655]: 2021-07- 2021-07-31T01:51:50.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:49 smithi045 conmon[33655]: 31T01:51:48.589929+0000 mon.a (mon.0) 1058 : cluster [DBG] osdmap e409: 8 total, 8 up, 8 in 2021-07-31T01:51:50.605 INFO:tasks.rados.rados.0.smithi045.stdout:1870: expect (ObjNum 575 snap 175 seq_num 575) 2021-07-31T01:51:50.607 INFO:tasks.rados.rados.0.smithi045.stdout:1868: finishing rollback tid 0 to smithi04569269-32 2021-07-31T01:51:50.607 INFO:tasks.rados.rados.0.smithi045.stdout:1868: finishing rollback tid 1 to smithi04569269-32 2021-07-31T01:51:50.607 INFO:tasks.rados.rados.0.smithi045.stdout:1866: finishing write tid 1 to smithi04569269-45 2021-07-31T01:51:50.607 INFO:tasks.rados.rados.0.smithi045.stdout:1868: finishing rollback tid 2 to smithi04569269-32 2021-07-31T01:51:50.608 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 32 v 1028 (ObjNum 549 snap 164 seq_num 549) dirty exists 2021-07-31T01:51:50.608 INFO:tasks.rados.rados.0.smithi045.stdout:1866: finishing write tid 2 to smithi04569269-45 2021-07-31T01:51:50.608 INFO:tasks.rados.rados.0.smithi045.stdout:1866: finishing write tid 3 to smithi04569269-45 2021-07-31T01:51:50.608 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 45 v 1069 (ObjNum 585 snap 180 seq_num 585) dirty exists 2021-07-31T01:51:50.608 INFO:tasks.rados.rados.0.smithi045.stdout:1866: left oid 45 (ObjNum 585 snap 180 seq_num 585) 2021-07-31T01:51:50.609 INFO:tasks.rados.rados.0.smithi045.stdout:1869: expect (ObjNum 399 snap 121 seq_num 399) 2021-07-31T01:51:50.848 INFO:tasks.rados.rados.0.smithi045.stdout:1867: expect (ObjNum 530 snap 156 seq_num 530) 2021-07-31T01:51:51.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:50 smithi154 conmon[34553]: cluster 2021-07-31T01:51:51.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:50 smithi154 conmon[34553]: 2021-07-31T01:51:49.592246 2021-07-31T01:51:51.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:50 smithi154 conmon[34553]: +0000 mon.a (mon.0) 1059 : cluster [DBG] osdmap e410: 8 total, 8 up, 8 in 2021-07-31T01:51:51.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:50 smithi154 conmon[34553]: cluster 2021-07-31T01:51:49.855736+0000 2021-07-31T01:51:51.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:50 smithi154 conmon[34553]: mgr.x (mgr.24542) 197 : cluster [DBG] pgmap v395: 25 pgs: 25 active+clean; 211 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 7.6 MiB/s rd, 656 KiB/s wr, 10 op/s 2021-07-31T01:51:51.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:50 smithi045 conmon[39718]: cluster 2021-07-31T01:51:49.592246+0000 mon.a (mon.0) 1059 : cluster 2021-07-31T01:51:51.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:50 smithi045 conmon[39718]: [DBG] osdmap e410: 8 total, 8 up, 8 in 2021-07-31T01:51:51.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:50 smithi045 conmon[39718]: cluster 2021-07-31T01:51:49.855736+0000 mgr.x (mgr.24542) 197 : cluster [DBG] pgmap v395: 25 pgs: 25 active+clean; 211 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 7.6 MiB/s rd, 656 KiB/s wr, 10 op/s 2021-07-31T01:51:51.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:50 smithi045 conmon[33655]: cluster 2021-07-31T01:51:49.592246+0000 mon.a (mon.0) 2021-07-31T01:51:51.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:50 smithi045 conmon[33655]: 1059 : cluster [DBG] osdmap e410: 8 total, 8 up, 8 in 2021-07-31T01:51:51.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:50 smithi045 conmon[33655]: cluster 2021-07-31T01:51:49.855736+0000 mgr.x (mgr.24542) 197 : cluster [DBG] pgmap v395: 25 pgs: 25 active+clean; 211 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 7.6 MiB/s rd, 656 KiB/s wr, 10 op/s 2021-07-31T01:51:51.263 INFO:tasks.rados.rados.0.smithi045.stdout:1866: done (5 left) 2021-07-31T01:51:51.263 INFO:tasks.rados.rados.0.smithi045.stdout:1867: done (4 left) 2021-07-31T01:51:51.264 INFO:tasks.rados.rados.0.smithi045.stdout:1868: done (3 left) 2021-07-31T01:51:51.264 INFO:tasks.rados.rados.0.smithi045.stdout:1869: done (2 left) 2021-07-31T01:51:51.264 INFO:tasks.rados.rados.0.smithi045.stdout:1870: done (1 left) 2021-07-31T01:51:51.264 INFO:tasks.rados.rados.0.smithi045.stdout:1871: done (0 left) 2021-07-31T01:51:51.265 INFO:tasks.rados.rados.0.smithi045.stdout:1872: rollback oid 36 current snap is 180 2021-07-31T01:51:51.265 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 36 to 171 2021-07-31T01:51:51.265 INFO:tasks.rados.rados.0.smithi045.stdout:1873: read oid 33 snap -1 2021-07-31T01:51:51.265 INFO:tasks.rados.rados.0.smithi045.stdout:1873: expect (ObjNum 399 snap 121 seq_num 399) 2021-07-31T01:51:51.265 INFO:tasks.rados.rados.0.smithi045.stdout:1874: copy_from oid 11 from oid 9 current snap is 180 2021-07-31T01:51:51.266 INFO:tasks.rados.rados.0.smithi045.stdout:1875: read oid 25 snap 176 2021-07-31T01:51:51.266 INFO:tasks.rados.rados.0.smithi045.stdout:1875: expect (ObjNum 541 snap 162 seq_num 541) 2021-07-31T01:51:51.266 INFO:tasks.rados.rados.0.smithi045.stdout:1876: rollback oid 31 current snap is 180 2021-07-31T01:51:51.266 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 31 to 171 2021-07-31T01:51:51.267 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 32 current snap is 180 2021-07-31T01:51:51.267 INFO:tasks.rados.rados.0.smithi045.stdout:1877: seq_num 586 ranges {2007040=221184} 2021-07-31T01:51:51.278 INFO:tasks.rados.rados.0.smithi045.stdout:1877: writing smithi04569269-32 from 2007040 to 2228224 tid 1 2021-07-31T01:51:51.278 INFO:tasks.rados.rados.0.smithi045.stdout:1878: delete oid 21 current snap is 180 2021-07-31T01:51:51.278 INFO:tasks.rados.rados.0.smithi045.stdout:1872: finishing rollback tid 1 to smithi04569269-36 2021-07-31T01:51:51.278 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 36 v 1141 (ObjNum -1106019584 snap 21873 seq_num 3189707632) dirty dne 2021-07-31T01:51:51.279 INFO:tasks.rados.rados.0.smithi045.stdout:1876: finishing rollback tid 0 to smithi04569269-31 2021-07-31T01:51:51.279 INFO:tasks.rados.rados.0.smithi045.stdout:1876: finishing rollback tid 1 to smithi04569269-31 2021-07-31T01:51:51.279 INFO:tasks.rados.rados.0.smithi045.stdout:1876: finishing rollback tid 2 to smithi04569269-31 2021-07-31T01:51:51.280 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 31 v 1186 (ObjNum 490 snap 146 seq_num 490) dirty exists 2021-07-31T01:51:51.282 INFO:tasks.rados.rados.0.smithi045.stdout:1877: finishing write tid 1 to smithi04569269-32 2021-07-31T01:51:51.283 INFO:tasks.rados.rados.0.smithi045.stdout:1877: finishing write tid 2 to smithi04569269-32 2021-07-31T01:51:51.283 INFO:tasks.rados.rados.0.smithi045.stdout:1877: finishing write tid 3 to smithi04569269-32 2021-07-31T01:51:51.284 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 32 v 1030 (ObjNum 586 snap 180 seq_num 586) dirty exists 2021-07-31T01:51:51.285 INFO:tasks.rados.rados.0.smithi045.stdout:1877: left oid 32 (ObjNum 586 snap 180 seq_num 586) 2021-07-31T01:51:51.285 INFO:tasks.rados.rados.0.smithi045.stdout:1875: expect (ObjNum 541 snap 162 seq_num 541) 2021-07-31T01:51:51.525 INFO:tasks.rados.rados.0.smithi045.stdout:1873: expect (ObjNum 399 snap 121 seq_num 399) 2021-07-31T01:51:51.771 INFO:tasks.rados.rados.0.smithi045.stdout:1874: finishing copy_from to smithi04569269-11 2021-07-31T01:51:51.771 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 11 v 395 (ObjNum 530 snap 156 seq_num 530) dirty exists 2021-07-31T01:51:51.772 INFO:tasks.rados.rados.0.smithi045.stdout:1874: finishing copy_from racing read to smithi04569269-11 2021-07-31T01:51:51.772 INFO:tasks.rados.rados.0.smithi045.stdout:1872: done (6 left) 2021-07-31T01:51:51.772 INFO:tasks.rados.rados.0.smithi045.stdout:1873: done (5 left) 2021-07-31T01:51:51.772 INFO:tasks.rados.rados.0.smithi045.stdout:1874: done (4 left) 2021-07-31T01:51:51.772 INFO:tasks.rados.rados.0.smithi045.stdout:1875: done (3 left) 2021-07-31T01:51:51.773 INFO:tasks.rados.rados.0.smithi045.stdout:1876: done (2 left) 2021-07-31T01:51:51.773 INFO:tasks.rados.rados.0.smithi045.stdout:1877: done (1 left) 2021-07-31T01:51:51.773 INFO:tasks.rados.rados.0.smithi045.stdout:1878: done (0 left) 2021-07-31T01:51:51.773 INFO:tasks.rados.rados.0.smithi045.stdout:1879: read oid 7 snap -1 2021-07-31T01:51:51.774 INFO:tasks.rados.rados.0.smithi045.stdout:1879: expect (ObjNum 0 snap 0 seq_num 0) 2021-07-31T01:51:51.774 INFO:tasks.rados.rados.0.smithi045.stdout:1880: snap_create 2021-07-31T01:51:51.774 INFO:tasks.rados.rados.0.smithi045.stdout:1879: expect (ObjNum 0 snap 0 seq_num 0) 2021-07-31T01:51:52.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:51 smithi154 conmon[34553]: cluster 2021-07-31T01:51:50.604101+0000 mon.a (mon.0) 1060 : cluster [DBG] osdmap e411: 8 total, 8 up, 8 in 2021-07-31T01:51:52.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:51 smithi154 conmon[34553]: cluster 2021-07-31T01:51:51. 2021-07-31T01:51:52.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:51 smithi154 conmon[34553]: 560042+0000 mon.a (mon.0) 1061 : cluster [DBG] osdmap e412: 8 total, 8 up, 8 in 2021-07-31T01:51:52.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:51 smithi045 conmon[39718]: cluster 2021-07-31T01:51:50.604101+0000 mon.a ( 2021-07-31T01:51:52.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:51 smithi045 conmon[39718]: mon.0) 1060 : cluster [DBG] osdmap e411: 8 total, 8 up, 8 in 2021-07-31T01:51:52.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:51 smithi045 conmon[39718]: cluster 2021-07-31T01:51:51.560042+0000 mon.a (mon.0) 1061 : cluster 2021-07-31T01:51:52.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:51 smithi045 conmon[39718]: [DBG] osdmap e412: 8 total, 8 up, 8 in 2021-07-31T01:51:52.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:51 smithi045 conmon[33655]: cluster 2021-07-31T01:51:50.604101+0000 mon.a (mon.0 2021-07-31T01:51:52.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:51 smithi045 conmon[33655]: ) 1060 : cluster [DBG] osdmap e411: 8 total, 8 up, 8 in 2021-07-31T01:51:52.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:51 smithi045 conmon[33655]: cluster 2021-07-31T01:51:51.560042+0000 mon.a (mon.0) 1061 : cluster [DBG] osdmap e412: 8 total, 8 up, 8 in 2021-07-31T01:51:52.612 INFO:tasks.rados.rados.0.smithi045.stdout:1879: done (1 left) 2021-07-31T01:51:52.612 INFO:tasks.rados.rados.0.smithi045.stdout:1880: done (0 left) 2021-07-31T01:51:52.613 INFO:tasks.rados.rados.0.smithi045.stdout:1881: rollback oid 42 current snap is 181 2021-07-31T01:51:52.613 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 42 to 171 2021-07-31T01:51:52.613 INFO:tasks.rados.rados.0.smithi045.stdout:1882: rollback oid 29 current snap is 181 2021-07-31T01:51:52.613 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 29 to 162 2021-07-31T01:51:52.614 INFO:tasks.rados.rados.0.smithi045.stdout:1883: snap_create 2021-07-31T01:51:52.619 INFO:tasks.rados.rados.0.smithi045.stdout:1882: finishing rollback tid 0 to smithi04569269-29 2021-07-31T01:51:52.619 INFO:tasks.rados.rados.0.smithi045.stdout:1882: finishing rollback tid 1 to smithi04569269-29 2021-07-31T01:51:52.620 INFO:tasks.rados.rados.0.smithi045.stdout:1882: finishing rollback tid 2 to smithi04569269-29 2021-07-31T01:51:52.620 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 29 v 926 (ObjNum 539 snap 162 seq_num 539) dirty exists 2021-07-31T01:51:52.620 INFO:tasks.rados.rados.0.smithi045.stdout:1881: finishing rollback tid 0 to smithi04569269-42 2021-07-31T01:51:52.621 INFO:tasks.rados.rados.0.smithi045.stdout:1881: finishing rollback tid 1 to smithi04569269-42 2021-07-31T01:51:52.621 INFO:tasks.rados.rados.0.smithi045.stdout:1881: finishing rollback tid 2 to smithi04569269-42 2021-07-31T01:51:52.621 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 42 v 1072 (ObjNum 567 snap 168 seq_num 567) dirty exists 2021-07-31T01:51:53.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:52 smithi154 conmon[34553]: cluster 2021-07-31T01:51:51.856116+0000 mgr.x (mgr.24542 2021-07-31T01:51:53.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:52 smithi154 conmon[34553]: ) 198 : cluster [DBG] pgmap v398: 25 pgs: 25 active+clean; 211 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 7.6 MiB/s rd, 656 KiB/s wr, 10 op/s 2021-07-31T01:51:53.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:52 smithi045 conmon[33655]: cluster 2021-07-31T01:51:51.856116+0000 mgr.x (mgr.24542) 198 : 2021-07-31T01:51:53.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:52 smithi045 conmon[33655]: cluster [DBG] pgmap v398: 25 pgs: 25 active+clean; 211 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 7.6 MiB/s rd, 656 KiB/s wr, 10 op/s 2021-07-31T01:51:53.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:52 smithi045 conmon[39718]: cluster 2021-07-31T01:51:51.856116+0000 mgr.x 2021-07-31T01:51:53.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:52 smithi045 conmon[39718]: (mgr.24542) 198 : cluster [DBG] pgmap v398: 25 pgs: 25 active+clean; 211 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 7.6 MiB/s rd, 656 KiB/s wr, 10 op/s 2021-07-31T01:51:53.613 INFO:tasks.rados.rados.0.smithi045.stdout:1881: done (2 left) 2021-07-31T01:51:53.613 INFO:tasks.rados.rados.0.smithi045.stdout:1882: done (1 left) 2021-07-31T01:51:53.614 INFO:tasks.rados.rados.0.smithi045.stdout:1883: done (0 left) 2021-07-31T01:51:53.614 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 50 current snap is 182 2021-07-31T01:51:53.614 INFO:tasks.rados.rados.0.smithi045.stdout:1884: seq_num 588 ranges {5283840=253952} 2021-07-31T01:51:53.644 INFO:tasks.rados.rados.0.smithi045.stdout:1884: writing smithi04569269-50 from 5283840 to 5537792 tid 1 2021-07-31T01:51:53.644 INFO:tasks.rados.rados.0.smithi045.stdout:1885: read oid 28 snap 175 2021-07-31T01:51:53.644 INFO:tasks.rados.rados.0.smithi045.stdout:1885: expect deleted 2021-07-31T01:51:53.645 INFO:tasks.rados.rados.0.smithi045.stdout:1886: snap_remove snap 178 2021-07-31T01:51:53.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:53 smithi045 conmon[33655]: cluster 2021-07-31T01:51:52.611115+0000 mon.a (mon.0) 1062 2021-07-31T01:51:53.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:53 smithi045 conmon[33655]: : cluster [DBG] osdmap e413: 8 total, 8 up, 8 in 2021-07-31T01:51:53.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:53 smithi045 conmon[39718]: cluster 2021-07-31T01:51:52.611115+0000 mon.a (mon.0 2021-07-31T01:51:53.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:53 smithi045 conmon[39718]: ) 1062 : cluster [DBG] osdmap e413: 8 total, 8 up, 8 in 2021-07-31T01:51:53.780 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:51:53 smithi045 conmon[61210]: level=error ts=2021-07-31T01:51:53.516Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:51:53.781 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:51:53 smithi045 conmon[61210]: level=error ts=2021-07-31T01:51:53.516Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:51:53.781 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:51:53 smithi045 conmon[61210]: level=error ts=2021-07-31T01:51:53.516Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T01:51:54.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:53 smithi154 conmon[34553]: cluster 2021-07-31T01:51:52.611115+0000 mon.a (mon.0) 1062 : cluster [DBG] 2021-07-31T01:51:54.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:53 smithi154 conmon[34553]: osdmap e413: 8 total, 8 up, 8 in 2021-07-31T01:51:54.627 INFO:tasks.rados.rados.0.smithi045.stdout:1886: done (2 left) 2021-07-31T01:51:54.628 INFO:tasks.rados.rados.0.smithi045.stdout:1887: read oid 36 snap -1 2021-07-31T01:51:54.628 INFO:tasks.rados.rados.0.smithi045.stdout:1887: expect deleted 2021-07-31T01:51:54.628 INFO:tasks.rados.rados.0.smithi045.stdout:1888: snap_remove snap 162 2021-07-31T01:51:55.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:54 smithi154 conmon[34553]: cluster 2021-07-31T01:51:53.612784+0000 mon.a (mon.0) 1063 : cluster [DBG] osdmap e414: 8 total, 8 up, 8 in 2021-07-31T01:51:55.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:54 smithi154 conmon[34553]: cluster 2021-07-31T01:51:53.856594+0000 mgr.x (mgr.24542) 199 : cluster [DBG] pgmap v401: 25 pgs: 25 active+clean; 227 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 18 MiB/s rd, 1.9 MiB/s wr, 11 op/s 2021-07-31T01:51:55.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:54 smithi045 conmon[39718]: cluster 2021-07-31T01:51:53.612784+0000 mon.a (mon.0) 2021-07-31T01:51:55.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:54 smithi045 conmon[39718]: 1063 : cluster [DBG] osdmap e414: 8 total, 8 up, 8 in 2021-07-31T01:51:55.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:54 smithi045 conmon[39718]: cluster 2021-07-31T01:51:53.856594+0000 mgr.x (mgr.24542) 199 2021-07-31T01:51:55.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:54 smithi045 conmon[39718]: : cluster [DBG] pgmap v401: 25 pgs: 25 active+clean; 227 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 18 MiB/s rd, 1.9 MiB/s wr, 11 op/s 2021-07-31T01:51:55.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:54 smithi045 conmon[33655]: cluster 2021-07-31T01:51:53.612784+0000 mon.a (mon.0) 1063 : cluster [DBG] 2021-07-31T01:51:55.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:54 smithi045 conmon[33655]: osdmap e414: 8 total, 8 up, 8 in 2021-07-31T01:51:55.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:54 smithi045 conmon[33655]: cluster 2021-07-31T01:51:53.856594+0000 mgr.x (mgr.24542) 199 : cluster [DBG] pgmap v401: 25 pgs: 25 active+clean; 227 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 18 MiB/s rd, 1.9 MiB/s wr, 11 op/s 2021-07-31T01:51:55.626 INFO:tasks.rados.rados.0.smithi045.stdout:1888: done (3 left) 2021-07-31T01:51:55.626 INFO:tasks.rados.rados.0.smithi045.stdout:1889: copy_from oid 49 from oid 25 current snap is 182 2021-07-31T01:51:55.626 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 21 current snap is 182 2021-07-31T01:51:55.627 INFO:tasks.rados.rados.0.smithi045.stdout:1890: seq_num 589 ranges {0=270336} 2021-07-31T01:51:55.630 INFO:tasks.rados.rados.0.smithi045.stdout:1890: writing smithi04569269-21 from 0 to 270336 tid 1 2021-07-31T01:51:55.630 INFO:tasks.rados.rados.0.smithi045.stdout:1891: rollback oid 12 current snap is 182 2021-07-31T01:51:55.630 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 12 to 167 2021-07-31T01:51:55.630 INFO:tasks.rados.rados.0.smithi045.stdout:1892: copy_from oid 43 from oid 16 current snap is 182 2021-07-31T01:51:55.631 INFO:tasks.rados.rados.0.smithi045.stdout:1893: copy_from oid 9 from oid 32 current snap is 182 2021-07-31T01:51:55.631 INFO:tasks.rados.rados.0.smithi045.stdout:1894: copy_from oid 2 from oid 34 current snap is 182 2021-07-31T01:51:55.631 INFO:tasks.rados.rados.0.smithi045.stdout:1895: copy_from oid 42 from oid 20 current snap is 182 2021-07-31T01:51:55.631 INFO:tasks.rados.rados.0.smithi045.stdout:1896: snap_create 2021-07-31T01:51:55.632 INFO:tasks.rados.rados.0.smithi045.stdout:1884: finishing write tid 1 to smithi04569269-50 2021-07-31T01:51:55.632 INFO:tasks.rados.rados.0.smithi045.stdout:1884: finishing write tid 2 to smithi04569269-50 2021-07-31T01:51:55.632 INFO:tasks.rados.rados.0.smithi045.stdout:1884: finishing write tid 3 to smithi04569269-50 2021-07-31T01:51:55.632 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 50 v 1074 (ObjNum 588 snap 182 seq_num 588) dirty exists 2021-07-31T01:51:55.633 INFO:tasks.rados.rados.0.smithi045.stdout:1884: left oid 50 (ObjNum 588 snap 182 seq_num 588) 2021-07-31T01:51:55.642 INFO:tasks.rados.rados.0.smithi045.stdout:1891: finishing rollback tid 0 to smithi04569269-12 2021-07-31T01:51:55.643 INFO:tasks.rados.rados.0.smithi045.stdout:1891: finishing rollback tid 1 to smithi04569269-12 2021-07-31T01:51:55.643 INFO:tasks.rados.rados.0.smithi045.stdout:1891: finishing rollback tid 2 to smithi04569269-12 2021-07-31T01:51:55.643 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 12 v 671 (ObjNum 506 snap 152 seq_num 506) dirty exists 2021-07-31T01:51:55.645 INFO:tasks.rados.rados.0.smithi045.stdout:1890: finishing write tid 1 to smithi04569269-21 2021-07-31T01:51:55.646 INFO:tasks.rados.rados.0.smithi045.stdout:1890: finishing write tid 2 to smithi04569269-21 2021-07-31T01:51:55.654 INFO:tasks.rados.rados.0.smithi045.stdout:1892: finishing copy_from to smithi04569269-43 2021-07-31T01:51:55.654 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 43 v 1021 (ObjNum 523 snap 154 seq_num 523) dirty exists 2021-07-31T01:51:55.655 INFO:tasks.rados.rados.0.smithi045.stdout:1892: finishing copy_from racing read to smithi04569269-43 2021-07-31T01:51:55.672 INFO:tasks.rados.rados.0.smithi045.stdout:1890: finishing write tid 3 to smithi04569269-21 2021-07-31T01:51:55.672 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 21 v 1193 (ObjNum 589 snap 182 seq_num 589) dirty exists 2021-07-31T01:51:55.672 INFO:tasks.rados.rados.0.smithi045.stdout:1890: left oid 21 (ObjNum 589 snap 182 seq_num 589) 2021-07-31T01:51:55.673 INFO:tasks.rados.rados.0.smithi045.stdout:1894: finishing copy_from to smithi04569269-2 2021-07-31T01:51:55.674 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 2 v 1002 (ObjNum 399 snap 121 seq_num 399) dirty exists 2021-07-31T01:51:55.674 INFO:tasks.rados.rados.0.smithi045.stdout:1894: finishing copy_from racing read to smithi04569269-2 2021-07-31T01:51:55.674 INFO:tasks.rados.rados.0.smithi045.stdout:1893: finishing copy_from to smithi04569269-9 2021-07-31T01:51:55.674 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 9 v 1030 (ObjNum 586 snap 180 seq_num 586) dirty exists 2021-07-31T01:51:55.676 INFO:tasks.rados.rados.0.smithi045.stdout:1893: finishing copy_from racing read to smithi04569269-9 2021-07-31T01:51:55.676 INFO:tasks.rados.rados.0.smithi045.stdout:1889: finishing copy_from to smithi04569269-49 2021-07-31T01:51:55.676 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 49 v 288 (ObjNum 541 snap 162 seq_num 541) dirty exists 2021-07-31T01:51:55.676 INFO:tasks.rados.rados.0.smithi045.stdout:1889: finishing copy_from racing read to smithi04569269-49 2021-07-31T01:51:55.684 INFO:tasks.rados.rados.0.smithi045.stdout:1895: finishing copy_from to smithi04569269-42 2021-07-31T01:51:55.684 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 42 v 1127 (ObjNum 556 snap 166 seq_num 556) dirty exists 2021-07-31T01:51:55.684 INFO:tasks.rados.rados.0.smithi045.stdout:1895: finishing copy_from racing read to smithi04569269-42 2021-07-31T01:51:56.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:55 smithi154 conmon[34553]: cluster 2021-07-31T01:51:56.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:55 smithi154 conmon[34553]: 2021-07-31T 2021-07-31T01:51:56.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:55 smithi154 conmon[34553]: 01:51:54.623042+0000 mon.a (mon.0) 1064 : cluster [DBG] osdmap e415: 8 total, 8 up, 8 in 2021-07-31T01:51:56.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:55 smithi045 conmon[39718]: cluster 2021-07-31T01:51:54.623042+0000 mon.a (mon.0) 1064 : cluster [DBG] osdmap e415: 8 total, 8 up, 8 in 2021-07-31T01:51:56.027 INFO:journalctl@ceph.osd.0.smithi045.stdout:Jul 31 01:51:55 smithi045 conmon[43243]: debug 2021-07-31T01:51:56.027 INFO:journalctl@ceph.osd.0.smithi045.stdout:Jul 31 01:51:55 smithi045 conmon[43243]: 2021-07-31T01:51:55.629+0000 7fdf6ff98700 -1 osd.0 pg_epoch: 416 pg[3.1s0( v 404'336 (0'0,404'336] local-lis/les=56/57 n=4 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [0,4,3]p0(0) r=0 lpr=56 crt=404'336 lcod 404'334 mlcod 404'334 active+clean+snaptrim trimq=[143~1] ps=[154~1,15a~1,15e~1]] removing snap head 2021-07-31T01:51:56.027 INFO:journalctl@ceph.osd.0.smithi045.stdout:Jul 31 01:51:55 smithi045 conmon[43243]: 2021-07-31T01:51:56.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:55 smithi045 conmon[33655]: cluster 2021-07-31T01:51: 2021-07-31T01:51:56.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:55 smithi045 conmon[33655]: 54.623042+0000 mon.a (mon.0) 1064 : cluster [DBG] osdmap e415: 8 total, 8 up, 8 in 2021-07-31T01:51:56.564 INFO:tasks.rados.rados.0.smithi045.stdout:1884: done (10 left) 2021-07-31T01:51:56.564 INFO:tasks.rados.rados.0.smithi045.stdout:1885: done (9 left) 2021-07-31T01:51:56.564 INFO:tasks.rados.rados.0.smithi045.stdout:1887: done (8 left) 2021-07-31T01:51:56.565 INFO:tasks.rados.rados.0.smithi045.stdout:1889: done (7 left) 2021-07-31T01:51:56.565 INFO:tasks.rados.rados.0.smithi045.stdout:1890: done (6 left) 2021-07-31T01:51:56.565 INFO:tasks.rados.rados.0.smithi045.stdout:1891: done (5 left) 2021-07-31T01:51:56.565 INFO:tasks.rados.rados.0.smithi045.stdout:1892: done (4 left) 2021-07-31T01:51:56.566 INFO:tasks.rados.rados.0.smithi045.stdout:1893: done (3 left) 2021-07-31T01:51:56.566 INFO:tasks.rados.rados.0.smithi045.stdout:1894: done (2 left) 2021-07-31T01:51:56.566 INFO:tasks.rados.rados.0.smithi045.stdout:1895: done (1 left) 2021-07-31T01:51:56.566 INFO:tasks.rados.rados.0.smithi045.stdout:1896: done (0 left) 2021-07-31T01:51:56.566 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 45 current snap is 183 2021-07-31T01:51:56.567 INFO:tasks.rados.rados.0.smithi045.stdout:1897: seq_num 590 ranges {6135808=286720} 2021-07-31T01:51:56.598 INFO:tasks.rados.rados.0.smithi045.stdout:1897: writing smithi04569269-45 from 6135808 to 6422528 tid 1 2021-07-31T01:51:56.599 INFO:tasks.rados.rados.0.smithi045.stdout:1898: rollback oid 15 current snap is 183 2021-07-31T01:51:56.599 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 15 to 171 2021-07-31T01:51:56.599 INFO:tasks.rados.rados.0.smithi045.stdout:1899: snap_remove snap 172 2021-07-31T01:51:57.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:56 smithi154 conmon[34553]: cluster 2021-07-31T01:51:55.625040+0000 mon.a (mon.0) 1065 : cluster [DBG] osdmap e416: 8 total, 8 up, 8 in 2021-07-31T01:51:57.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:56 smithi154 conmon[34553]: 2021-07-31T01:51:57.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:56 smithi154 conmon[34553]: cluster 2021-07-31T01:51:55.857075+0000 mgr.x (mgr.24542) 200 : cluster [DBG] pgmap v404: 25 pgs: 25 active+clean; 227 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 18 MiB/s rd, 1.9 MiB/s wr, 11 op/s 2021-07-31T01:51:57.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:56 smithi154 conmon[34553]: cluster 2021-07-31T01:51:56.562333+0000 mon.a (mon.0) 1066 : 2021-07-31T01:51:57.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:56 smithi154 conmon[34553]: cluster [DBG] osdmap e417: 8 total, 8 up, 8 in 2021-07-31T01:51:57.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:56 smithi045 conmon[33655]: cluster 2021-07-31T01:51 2021-07-31T01:51:57.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:56 smithi045 conmon[33655]: :55.625040+0000 mon.a (mon.0) 1065 : cluster [DBG] osdmap e416: 8 total, 8 up, 8 in 2021-07-31T01:51:57.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:56 smithi045 conmon[33655]: cluster 2021-07-31T01:51:55.857075+0000 2021-07-31T01:51:57.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:56 smithi045 conmon[33655]: mgr.x (mgr.24542) 200 : cluster [DBG] pgmap v404: 25 pgs: 25 active+clean; 227 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 18 MiB/s rd, 1.9 MiB/s wr, 11 op/s 2021-07-31T01:51:57.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:56 smithi045 conmon[33655]: cluster 2021-07-31T01:51:57.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:56 smithi045 conmon[33655]: 2021-07-31T01:51:56.562333+0000 mon.a ( 2021-07-31T01:51:57.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:56 smithi045 conmon[33655]: mon.0) 1066 : cluster [DBG] osdmap e417: 8 total, 8 up, 8 in 2021-07-31T01:51:57.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:56 smithi045 conmon[39718]: cluster 2021-07-31T01:51:55.625040 2021-07-31T01:51:57.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:56 smithi045 conmon[39718]: +0000 mon.a (mon.0) 1065 : cluster [DBG] osdmap e416: 8 total, 8 up, 8 in 2021-07-31T01:51:57.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:56 smithi045 conmon[39718]: cluster 2021-07- 2021-07-31T01:51:57.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:56 smithi045 conmon[39718]: 31T01:51:55.857075+0000 mgr.x (mgr 2021-07-31T01:51:57.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:56 smithi045 conmon[39718]: .24542) 200 : cluster [DBG] pgmap v404: 25 pgs: 25 active+clean; 227 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 18 MiB/s rd, 1.9 MiB/s wr, 11 op/s 2021-07-31T01:51:57.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:56 smithi045 conmon[39718]: cluster 2021-07-31T01:51:56. 2021-07-31T01:51:57.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:56 smithi045 conmon[39718]: 562333+0000 mon.a (mon.0) 1066 : cluster [DBG] osdmap e417: 8 total, 8 up, 8 in 2021-07-31T01:51:57.570 INFO:tasks.rados.rados.0.smithi045.stdout:1899: done (2 left) 2021-07-31T01:51:57.570 INFO:tasks.rados.rados.0.smithi045.stdout:1898: finishing rollback tid 0 to smithi04569269-15 2021-07-31T01:51:57.570 INFO:tasks.rados.rados.0.smithi045.stdout:1898: finishing rollback tid 1 to smithi04569269-15 2021-07-31T01:51:57.571 INFO:tasks.rados.rados.0.smithi045.stdout:1898: finishing rollback tid 2 to smithi04569269-15 2021-07-31T01:51:57.571 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 15 v 454 (ObjNum 537 snap 161 seq_num 537) dirty exists 2021-07-31T01:51:57.571 INFO:tasks.rados.rados.0.smithi045.stdout:1897: finishing write tid 1 to smithi04569269-45 2021-07-31T01:51:57.571 INFO:tasks.rados.rados.0.smithi045.stdout:1897: finishing write tid 2 to smithi04569269-45 2021-07-31T01:51:57.571 INFO:tasks.rados.rados.0.smithi045.stdout:1897: finishing write tid 3 to smithi04569269-45 2021-07-31T01:51:57.572 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 45 v 1129 (ObjNum 590 snap 183 seq_num 590) dirty exists 2021-07-31T01:51:57.572 INFO:tasks.rados.rados.0.smithi045.stdout:1897: left oid 45 (ObjNum 590 snap 183 seq_num 590) 2021-07-31T01:51:57.572 INFO:tasks.rados.rados.0.smithi045.stdout:1900: read oid 1 snap -1 2021-07-31T01:51:57.572 INFO:tasks.rados.rados.0.smithi045.stdout:1900: expect (ObjNum 584 snap 179 seq_num 584) 2021-07-31T01:51:57.573 INFO:tasks.rados.rados.0.smithi045.stdout:1897: done (2 left) 2021-07-31T01:51:57.573 INFO:tasks.rados.rados.0.smithi045.stdout:1898: done (1 left) 2021-07-31T01:51:57.573 INFO:tasks.rados.rados.0.smithi045.stdout:1901: copy_from oid 23 from oid 38 current snap is 183 2021-07-31T01:51:57.573 INFO:tasks.rados.rados.0.smithi045.stdout:1902: snap_remove snap 181 2021-07-31T01:51:58.576 INFO:tasks.rados.rados.0.smithi045.stdout:1902: done (2 left) 2021-07-31T01:51:58.576 INFO:tasks.rados.rados.0.smithi045.stdout:1903: read oid 40 snap 182 2021-07-31T01:51:58.577 INFO:tasks.rados.rados.0.smithi045.stdout:1903: expect deleted 2021-07-31T01:51:58.577 INFO:tasks.rados.rados.0.smithi045.stdout:1904: rollback oid 13 current snap is 183 2021-07-31T01:51:58.577 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 13 to 171 2021-07-31T01:51:58.578 INFO:tasks.rados.rados.0.smithi045.stdout:1901: finishing copy_from to smithi04569269-23 2021-07-31T01:51:58.578 INFO:tasks.rados.rados.0.smithi045.stdout:1901: got expected ENOENT (src dne) 2021-07-31T01:51:58.579 INFO:tasks.rados.rados.0.smithi045.stdout:1901: finishing copy_from racing read to smithi04569269-23 2021-07-31T01:51:58.579 INFO:tasks.rados.rados.0.smithi045.stdout:1900: expect (ObjNum 584 snap 179 seq_num 584) 2021-07-31T01:51:58.749 INFO:tasks.rados.rados.0.smithi045.stdout:1904: finishing rollback tid 0 to smithi04569269-13 2021-07-31T01:51:58.749 INFO:tasks.rados.rados.0.smithi045.stdout:1904: finishing rollback tid 1 to smithi04569269-13 2021-07-31T01:51:58.749 INFO:tasks.rados.rados.0.smithi045.stdout:1904: finishing rollback tid 2 to smithi04569269-13 2021-07-31T01:51:58.750 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 13 v 681 (ObjNum 535 snap 161 seq_num 535) dirty exists 2021-07-31T01:51:58.750 INFO:tasks.rados.rados.0.smithi045.stdout:1900: done (3 left) 2021-07-31T01:51:58.750 INFO:tasks.rados.rados.0.smithi045.stdout:1901: done (2 left) 2021-07-31T01:51:58.750 INFO:tasks.rados.rados.0.smithi045.stdout:1903: done (1 left) 2021-07-31T01:51:58.750 INFO:tasks.rados.rados.0.smithi045.stdout:1904: done (0 left) 2021-07-31T01:51:58.751 INFO:tasks.rados.rados.0.smithi045.stdout:1905: read oid 42 snap 164 2021-07-31T01:51:58.751 INFO:tasks.rados.rados.0.smithi045.stdout:1905: expect (ObjNum 176 snap 46 seq_num 176) 2021-07-31T01:51:58.751 INFO:tasks.rados.rados.0.smithi045.stdout:1906: setattr oid 41 current snap is 183 2021-07-31T01:51:58.752 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 1 current snap is 183 2021-07-31T01:51:58.752 INFO:tasks.rados.rados.0.smithi045.stdout:1907: seq_num 591 ranges {2408448=303104} 2021-07-31T01:51:58.765 INFO:tasks.rados.rados.0.smithi045.stdout:1907: writing smithi04569269-1 from 2408448 to 2711552 tid 1 2021-07-31T01:51:58.765 INFO:tasks.rados.rados.0.smithi045.stdout:1908: snap_create 2021-07-31T01:51:58.766 INFO:tasks.rados.rados.0.smithi045.stdout:1905: expect (ObjNum 176 snap 46 seq_num 176) 2021-07-31T01:51:58.788 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 41 v 977 (ObjNum 552 snap 166 seq_num 552) dirty exists 2021-07-31T01:51:58.788 INFO:tasks.rados.rados.0.smithi045.stdout:1907: finishing write tid 1 to smithi04569269-1 2021-07-31T01:51:58.788 INFO:tasks.rados.rados.0.smithi045.stdout:1907: finishing write tid 2 to smithi04569269-1 2021-07-31T01:51:58.788 INFO:tasks.rados.rados.0.smithi045.stdout:1907: finishing write tid 3 to smithi04569269-1 2021-07-31T01:51:58.789 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 1 v 1008 (ObjNum 591 snap 183 seq_num 591) dirty exists 2021-07-31T01:51:58.789 INFO:tasks.rados.rados.0.smithi045.stdout:1907: left oid 1 (ObjNum 591 snap 183 seq_num 591) 2021-07-31T01:51:59.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:58 smithi154 conmon[34553]: cluster 2021-07-31T01:51:57.568230+0000 mon.a (mon.0 2021-07-31T01:51:59.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:58 smithi154 conmon[34553]: ) 1067 : cluster [DBG] osdmap e418: 8 total, 8 up, 8 in 2021-07-31T01:51:59.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:58 smithi154 conmon[34553]: cluster 2021-07-31T01:51:57.857552+0000 mgr.x (mgr.24542) 201 : cluster [DBG] pgmap v407: 25 pgs: 25 active+clean; 227 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2021-07-31T01:51:59.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:58 smithi045 conmon[33655]: cluster 2021-07-31T01:51:57.568230+0000 mon.a (mon.0) 1067 : 2021-07-31T01:51:59.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:58 smithi045 conmon[33655]: cluster [DBG] osdmap e418: 8 total, 8 up, 8 in 2021-07-31T01:51:59.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:58 smithi045 conmon[33655]: cluster 2021-07-31T01:51:57.857552+0000 mgr.x (mgr.24542) 201 : cluster [DBG] pgmap v407: 25 pgs: 25 active+clean; 227 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2021-07-31T01:51:59.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:58 smithi045 conmon[39718]: cluster 2021-07-31T01:51:57.568230+0000 mon.a (mon.0) 1067 : cluster [DBG] osdmap e418: 8 total, 8 up, 8 in 2021-07-31T01:51:59.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:58 smithi045 conmon[39718]: cluster 2021-07-31T01:51:57.857552+0000 mgr.x (mgr.24542) 201 : cluster [DBG] pgmap v407: 25 pgs: 25 active+clean; 227 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2021-07-31T01:51:59.576 INFO:tasks.rados.rados.0.smithi045.stdout:1905: done (3 left) 2021-07-31T01:51:59.576 INFO:tasks.rados.rados.0.smithi045.stdout:1906: done (2 left) 2021-07-31T01:51:59.576 INFO:tasks.rados.rados.0.smithi045.stdout:1907: done (1 left) 2021-07-31T01:51:59.577 INFO:tasks.rados.rados.0.smithi045.stdout:1908: done (0 left) 2021-07-31T01:51:59.577 INFO:tasks.rados.rados.0.smithi045.stdout:1909: delete oid 9 current snap is 184 2021-07-31T01:51:59.580 INFO:tasks.rados.rados.0.smithi045.stdout:1909: done (0 left) 2021-07-31T01:51:59.580 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 35 current snap is 184 2021-07-31T01:51:59.581 INFO:tasks.rados.rados.0.smithi045.stdout:1910: seq_num 593 ranges {0=335872} 2021-07-31T01:51:59.582 INFO:tasks.rados.rados.0.smithi045.stdout:1910: writing smithi04569269-35 from 0 to 335872 tid 1 2021-07-31T01:51:59.582 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 6 current snap is 184 2021-07-31T01:51:59.583 INFO:tasks.rados.rados.0.smithi045.stdout:1911: seq_num 594 ranges {2138112=352256} 2021-07-31T01:51:59.596 INFO:tasks.rados.rados.0.smithi045.stdout:1911: writing smithi04569269-6 from 2138112 to 2490368 tid 1 2021-07-31T01:51:59.597 INFO:tasks.rados.rados.0.smithi045.stdout:1912: delete oid 37 current snap is 184 2021-07-31T01:51:59.597 INFO:tasks.rados.rados.0.smithi045.stdout:1910: finishing write tid 1 to smithi04569269-35 2021-07-31T01:51:59.597 INFO:tasks.rados.rados.0.smithi045.stdout:1910: finishing write tid 2 to smithi04569269-35 2021-07-31T01:51:59.597 INFO:tasks.rados.rados.0.smithi045.stdout:1910: finishing write tid 3 to smithi04569269-35 2021-07-31T01:51:59.598 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 35 v 1215 (ObjNum 593 snap 184 seq_num 593) dirty exists 2021-07-31T01:51:59.598 INFO:tasks.rados.rados.0.smithi045.stdout:1910: left oid 35 (ObjNum 593 snap 184 seq_num 593) 2021-07-31T01:51:59.607 INFO:tasks.rados.rados.0.smithi045.stdout:1911: finishing write tid 1 to smithi04569269-6 2021-07-31T01:51:59.607 INFO:tasks.rados.rados.0.smithi045.stdout:1911: finishing write tid 2 to smithi04569269-6 2021-07-31T01:51:59.608 INFO:tasks.rados.rados.0.smithi045.stdout:1910: done (2 left) 2021-07-31T01:51:59.608 INFO:tasks.rados.rados.0.smithi045.stdout:1912: done (1 left) 2021-07-31T01:51:59.609 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 40 current snap is 184 2021-07-31T01:51:59.609 INFO:tasks.rados.rados.0.smithi045.stdout:1913: seq_num 596 ranges {0=385024} 2021-07-31T01:51:59.610 INFO:tasks.rados.rados.0.smithi045.stdout:1913: writing smithi04569269-40 from 0 to 385024 tid 1 2021-07-31T01:51:59.611 INFO:tasks.rados.rados.0.smithi045.stdout:1914: copy_from oid 31 from oid 21 current snap is 184 2021-07-31T01:51:59.611 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 14 current snap is 184 2021-07-31T01:51:59.611 INFO:tasks.rados.rados.0.smithi045.stdout:1911: finishing write tid 3 to smithi04569269-6 2021-07-31T01:51:59.612 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 6 v 686 (ObjNum 594 snap 184 seq_num 594) dirty exists 2021-07-31T01:51:59.612 INFO:tasks.rados.rados.0.smithi045.stdout:1911: left oid 6 (ObjNum 594 snap 184 seq_num 594) 2021-07-31T01:51:59.613 INFO:tasks.rados.rados.0.smithi045.stdout:1915: seq_num 597 ranges {7479296=401408} 2021-07-31T01:51:59.655 INFO:tasks.rados.rados.0.smithi045.stdout:1915: writing smithi04569269-14 from 7479296 to 7880704 tid 1 2021-07-31T01:51:59.655 INFO:tasks.rados.rados.0.smithi045.stdout:1911: done (3 left) 2021-07-31T01:51:59.656 INFO:tasks.rados.rados.0.smithi045.stdout:1916: copy_from oid 19 from oid 47 current snap is 184 2021-07-31T01:51:59.656 INFO:tasks.rados.rados.0.smithi045.stdout:1913: finishing write tid 1 to smithi04569269-40 2021-07-31T01:51:59.656 INFO:tasks.rados.rados.0.smithi045.stdout:1913: finishing write tid 2 to smithi04569269-40 2021-07-31T01:51:59.657 INFO:tasks.rados.rados.0.smithi045.stdout:1913: finishing write tid 3 to smithi04569269-40 2021-07-31T01:51:59.657 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 40 v 688 (ObjNum 596 snap 184 seq_num 596) dirty exists 2021-07-31T01:51:59.657 INFO:tasks.rados.rados.0.smithi045.stdout:1913: left oid 40 (ObjNum 596 snap 184 seq_num 596) 2021-07-31T01:51:59.658 INFO:tasks.rados.rados.0.smithi045.stdout:1914: finishing copy_from to smithi04569269-31 2021-07-31T01:51:59.658 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 31 v 1193 (ObjNum 589 snap 182 seq_num 589) dirty exists 2021-07-31T01:51:59.658 INFO:tasks.rados.rados.0.smithi045.stdout:1914: finishing copy_from racing read to smithi04569269-31 2021-07-31T01:51:59.659 INFO:tasks.rados.rados.0.smithi045.stdout:1913: done (3 left) 2021-07-31T01:51:59.659 INFO:tasks.rados.rados.0.smithi045.stdout:1914: done (2 left) 2021-07-31T01:51:59.659 INFO:tasks.rados.rados.0.smithi045.stdout:1917: delete oid 31 current snap is 184 2021-07-31T01:51:59.660 INFO:tasks.rados.rados.0.smithi045.stdout:1917: done (2 left) 2021-07-31T01:51:59.660 INFO:tasks.rados.rados.0.smithi045.stdout:1918: snap_remove snap 171 2021-07-31T01:52:00.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:51:59 smithi154 conmon[34553]: cluster 2021-07-31T01:51:58.573806+0000 mon.a (mon.0) 1068 : cluster [DBG] osdmap e419: 8 total, 8 up, 8 in 2021-07-31T01:52:00.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:59 smithi045 conmon[33655]: cluster 2021-07-31T01:51:58.573806+0000 mon.a (mon.0) 1068 2021-07-31T01:52:00.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:51:59 smithi045 conmon[33655]: : cluster [DBG] osdmap e419: 8 total, 8 up, 8 in 2021-07-31T01:52:00.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:59 smithi045 conmon[39718]: cluster 2021-07-31T01:52:00.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:59 smithi045 conmon[39718]: 2021-07-31T01:51:58 2021-07-31T01:52:00.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:51:59 smithi045 conmon[39718]: .573806+0000 mon.a (mon.0) 1068 : cluster [DBG] osdmap e419: 8 total, 8 up, 8 in 2021-07-31T01:52:00.587 INFO:tasks.rados.rados.0.smithi045.stdout:1918: done (2 left) 2021-07-31T01:52:00.587 INFO:tasks.rados.rados.0.smithi045.stdout:1919: snap_remove snap 167 2021-07-31T01:52:01.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:00 smithi154 conmon[34553]: cluster 2021-07-31T01:51:59.575350+0000 mon.a (mon.0) 1069 : cluster [DBG] 2021-07-31T01:52:01.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:00 smithi154 conmon[34553]: osdmap e420: 8 total, 8 up, 8 in 2021-07-31T01:52:01.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:00 smithi154 conmon[34553]: cluster 2021-07-31T01:51:59.857975+0000 mgr.x (mgr.24542) 202 : cluster [DBG] pgmap v410: 25 pgs: 25 active+clean; 240 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 2.2 MiB/s rd, 5.0 MiB/s wr, 13 op/s 2021-07-31T01:52:01.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:00 smithi045 conmon[39718]: cluster 2021-07-31T01:51:59.575350+0000 mon.a (mon.0) 1069 : 2021-07-31T01:52:01.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:00 smithi045 conmon[39718]: cluster [DBG] osdmap e420: 8 total, 8 up, 8 in 2021-07-31T01:52:01.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:00 smithi045 conmon[39718]: cluster 2021-07-31T01:51:59.857975+0000 mgr.x (mgr.24542) 202 : cluster [DBG] pgmap v410: 25 pgs: 25 active+clean; 240 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 2.2 MiB/s rd, 5.0 MiB/s wr, 13 op/s 2021-07-31T01:52:01.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:00 smithi045 conmon[33655]: cluster 2021-07-31T01:51:59.575350+0000 mon.a 2021-07-31T01:52:01.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:00 smithi045 conmon[33655]: (mon.0) 1069 : cluster [DBG] osdmap e420: 8 total, 8 up, 8 in 2021-07-31T01:52:01.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:00 smithi045 conmon[33655]: cluster 2021-07-31T01:51:59.857975+0000 mgr.x (mgr.24542) 202 : cluster 2021-07-31T01:52:01.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:00 smithi045 conmon[33655]: [DBG] pgmap v410: 25 pgs: 25 active+clean; 240 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 2.2 MiB/s rd, 5.0 MiB/s wr, 13 op/s 2021-07-31T01:52:01.565 INFO:tasks.rados.rados.0.smithi045.stdout:1919: done (2 left) 2021-07-31T01:52:01.565 INFO:tasks.rados.rados.0.smithi045.stdout:1920: copy_from oid 30 from oid 31 current snap is 184 2021-07-31T01:52:01.566 INFO:tasks.rados.rados.0.smithi045.stdout:1915: finishing write tid 1 to smithi04569269-14 2021-07-31T01:52:01.566 INFO:tasks.rados.rados.0.smithi045.stdout:1915: finishing write tid 2 to smithi04569269-14 2021-07-31T01:52:01.566 INFO:tasks.rados.rados.0.smithi045.stdout:1916: finishing copy_from to smithi04569269-19 2021-07-31T01:52:01.566 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 19 v 275 (ObjNum 172 snap 41 seq_num 172) dirty exists 2021-07-31T01:52:01.566 INFO:tasks.rados.rados.0.smithi045.stdout:1916: finishing copy_from racing read to smithi04569269-19 2021-07-31T01:52:01.567 INFO:tasks.rados.rados.0.smithi045.stdout:1915: finishing write tid 3 to smithi04569269-14 2021-07-31T01:52:01.567 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 14 v 1043 (ObjNum 597 snap 184 seq_num 597) dirty exists 2021-07-31T01:52:01.567 INFO:tasks.rados.rados.0.smithi045.stdout:1915: left oid 14 (ObjNum 597 snap 184 seq_num 597) 2021-07-31T01:52:01.567 INFO:tasks.rados.rados.0.smithi045.stdout:1915: done (2 left) 2021-07-31T01:52:01.568 INFO:tasks.rados.rados.0.smithi045.stdout:1916: done (1 left) 2021-07-31T01:52:01.568 INFO:tasks.rados.rados.0.smithi045.stdout:1921: read oid 21 snap -1 2021-07-31T01:52:01.568 INFO:tasks.rados.rados.0.smithi045.stdout:1921: expect (ObjNum 589 snap 182 seq_num 589) 2021-07-31T01:52:01.568 INFO:tasks.rados.rados.0.smithi045.stdout:1922: setattr oid 28 current snap is 184 2021-07-31T01:52:01.569 INFO:tasks.rados.rados.0.smithi045.stdout:1923: snap_remove snap 180 2021-07-31T01:52:02.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:01 smithi154 conmon[34553]: cluster 2021-07-31T01:52:00.586012+0000 mon.a ( 2021-07-31T01:52:02.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:01 smithi154 conmon[34553]: mon.0) 1070 : cluster [DBG] osdmap e421: 8 total, 8 up, 8 in 2021-07-31T01:52:02.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:01 smithi154 conmon[34553]: cluster 2021-07-31T01:52:01.563512+0000 mon.a (mon.0) 1071 : cluster [DBG] osdmap e422: 8 total, 8 up, 8 in 2021-07-31T01:52:02.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:01 smithi045 conmon[39718]: cluster 2021-07-31T01:52:00.586012 2021-07-31T01:52:02.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:01 smithi045 conmon[39718]: +0000 mon.a (mon.0) 1070 : cluster [DBG] osdmap e421: 8 total, 8 up, 8 in 2021-07-31T01:52:02.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:01 smithi045 conmon[39718]: cluster 2021-07-31T01: 2021-07-31T01:52:02.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:01 smithi045 conmon[39718]: 52:01.563512+0000 mon.a (mon.0) 1071 : cluster [DBG] osdmap e422: 8 total, 8 up, 8 in 2021-07-31T01:52:02.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:01 smithi045 conmon[33655]: cluster 2021-07-31T01:52:02.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:01 smithi045 conmon[33655]: 2021-07-31T01:52:02.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:01 smithi045 conmon[33655]: 2021 2021-07-31T01:52:02.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:01 smithi045 conmon[33655]: - 2021-07-31T01:52:02.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:01 smithi045 conmon[33655]: 07 2021-07-31T01:52:02.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:01 smithi045 conmon[33655]: -31T01:52:00 2021-07-31T01:52:02.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:01 smithi045 conmon[33655]: .586012+0000 mon.a (mon.0) 1070 : cluster [DBG] osdmap e421: 8 total, 8 up, 8 in 2021-07-31T01:52:02.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:01 smithi045 conmon[33655]: cluster 2021-07-31T01:52:02.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:01 smithi045 conmon[33655]: 2021-07-31T 2021-07-31T01:52:02.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:01 smithi045 conmon[33655]: 01:52:01.563512+0000 mon.a (mon.0) 1071 : cluster [DBG] osdmap e422: 8 total, 8 up, 8 in 2021-07-31T01:52:02.570 INFO:tasks.rados.rados.0.smithi045.stdout:1923: done (3 left) 2021-07-31T01:52:02.571 INFO:tasks.rados.rados.0.smithi045.stdout:1924: read oid 42 snap 175 2021-07-31T01:52:02.571 INFO:tasks.rados.rados.0.smithi045.stdout:1924: expect (ObjNum 567 snap 168 seq_num 567) 2021-07-31T01:52:02.571 INFO:tasks.rados.rados.0.smithi045.stdout:1925: delete oid 27 current snap is 184 2021-07-31T01:52:02.571 INFO:tasks.rados.rados.0.smithi045.stdout:1921: expect (ObjNum 589 snap 182 seq_num 589) 2021-07-31T01:52:02.578 INFO:tasks.rados.rados.0.smithi045.stdout:1920: finishing copy_from to smithi04569269-30 2021-07-31T01:52:02.579 INFO:tasks.rados.rados.0.smithi045.stdout:1920: got expected ENOENT (src dne) 2021-07-31T01:52:02.579 INFO:tasks.rados.rados.0.smithi045.stdout:1920: finishing copy_from racing read to smithi04569269-30 2021-07-31T01:52:02.579 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 28 v 1130 (ObjNum 1953066355 snap 875587944 seq_num 842610229) dirty exists 2021-07-31T01:52:02.582 INFO:tasks.rados.rados.0.smithi045.stdout:1920: done (4 left) 2021-07-31T01:52:02.582 INFO:tasks.rados.rados.0.smithi045.stdout:1921: done (3 left) 2021-07-31T01:52:02.582 INFO:tasks.rados.rados.0.smithi045.stdout:1922: done (2 left) 2021-07-31T01:52:02.583 INFO:tasks.rados.rados.0.smithi045.stdout:1925: done (1 left) 2021-07-31T01:52:02.583 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 21 current snap is 184 2021-07-31T01:52:02.583 INFO:tasks.rados.rados.0.smithi045.stdout:1926: seq_num 600 ranges {270336=450560} 2021-07-31T01:52:02.587 INFO:tasks.rados.rados.0.smithi045.stdout:1926: writing smithi04569269-21 from 270336 to 720896 tid 1 2021-07-31T01:52:02.588 INFO:tasks.rados.rados.0.smithi045.stdout:1927: read oid 45 snap 164 2021-07-31T01:52:02.588 INFO:tasks.rados.rados.0.smithi045.stdout:1927: expect (ObjNum 548 snap 164 seq_num 548) 2021-07-31T01:52:02.588 INFO:tasks.rados.rados.0.smithi045.stdout:1928: copy_from oid 46 from oid 4 current snap is 184 2021-07-31T01:52:02.589 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 27 current snap is 184 2021-07-31T01:52:02.589 INFO:tasks.rados.rados.0.smithi045.stdout:1929: seq_num 601 ranges {0=475136} 2021-07-31T01:52:02.591 INFO:tasks.rados.rados.0.smithi045.stdout:1929: writing smithi04569269-27 from 0 to 475136 tid 1 2021-07-31T01:52:02.592 INFO:tasks.rados.rados.0.smithi045.stdout:1930: read oid 11 snap -1 2021-07-31T01:52:02.592 INFO:tasks.rados.rados.0.smithi045.stdout:1930: expect (ObjNum 530 snap 156 seq_num 530) 2021-07-31T01:52:02.592 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 28 current snap is 184 2021-07-31T01:52:02.592 INFO:tasks.rados.rados.0.smithi045.stdout:1931: seq_num 602 ranges {0=491520} 2021-07-31T01:52:02.594 INFO:tasks.rados.rados.0.smithi045.stdout:1931: writing smithi04569269-28 from 0 to 491520 tid 1 2021-07-31T01:52:02.594 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 13 current snap is 184 2021-07-31T01:52:02.595 INFO:tasks.rados.rados.0.smithi045.stdout:1932: seq_num 603 ranges {11419648=507904} 2021-07-31T01:52:02.660 INFO:tasks.rados.rados.0.smithi045.stdout:1932: writing smithi04569269-13 from 11419648 to 11927552 tid 1 2021-07-31T01:52:02.661 INFO:tasks.rados.rados.0.smithi045.stdout:1933: snap_remove snap 169 2021-07-31T01:52:03.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:02 smithi154 conmon[34553]: cluster 2021-07-31T01:52:01.858395+0000 mgr.x (mgr.24542) 203 : cluster 2021-07-31T01:52:03.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:02 smithi154 conmon[34553]: [DBG] pgmap v413: 25 pgs: 25 active+clean; 240 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 2.2 MiB/s rd, 5.0 MiB/s wr, 13 op/s 2021-07-31T01:52:03.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:02 smithi154 conmon[34553]: cluster 2021-07-31T01:52:02.569405+0000 mon.a (mon.0) 1072 : cluster [DBG] osdmap e423: 8 total, 8 up, 8 in 2021-07-31T01:52:03.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:02 smithi045 conmon[39718]: cluster 2021-07-31T01:52:01. 2021-07-31T01:52:03.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:02 smithi045 conmon[39718]: 858395+0000 mgr.x (mgr.24542) 203 : cluster [DBG] pgmap v413: 25 pgs: 25 active+clean; 240 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 2.2 MiB/s rd, 5.0 MiB/s wr, 13 op/s 2021-07-31T01:52:03.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:02 smithi045 conmon[39718]: cluster 2021-07-31T01 2021-07-31T01:52:03.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:02 smithi045 conmon[39718]: :52:02.569405+0000 mon.a (mon.0) 1072 : cluster [DBG] osdmap e423: 8 total, 8 up, 8 in 2021-07-31T01:52:03.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:02 smithi045 conmon[33655]: cluster 2021-07-31T01:52:01.858395+0000 mgr.x ( 2021-07-31T01:52:03.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:02 smithi045 conmon[33655]: mgr.24542) 203 : cluster [DBG] pgmap v413: 25 pgs: 25 active+clean; 240 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 2.2 MiB/s rd, 5.0 MiB/s wr, 13 op/s 2021-07-31T01:52:03.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:02 smithi045 conmon[33655]: cluster 2021-07-31T01:52:02.569405+0000 mon.a (mon.0) 1072 : cluster [DBG] osdmap e423: 8 total, 8 up, 8 in 2021-07-31T01:52:03.525 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T01:52:03.605 INFO:tasks.rados.rados.0.smithi045.stdout:1933: done (8 left) 2021-07-31T01:52:03.606 INFO:tasks.rados.rados.0.smithi045.stdout:1934: rollback oid 23 current snap is 184 2021-07-31T01:52:03.606 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 23 to 176 2021-07-31T01:52:03.606 INFO:tasks.rados.rados.0.smithi045.stdout:1929: finishing write tid 1 to smithi04569269-27 2021-07-31T01:52:03.608 INFO:tasks.rados.rados.0.smithi045.stdout:1929: finishing write tid 2 to smithi04569269-27 2021-07-31T01:52:03.608 INFO:tasks.rados.rados.0.smithi045.stdout:1924: expect (ObjNum 567 snap 168 seq_num 567) 2021-07-31T01:52:03.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:52:03 smithi045 conmon[61210]: level=error ts=2021-07-31T01:52:03.517Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:52:03.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:52:03 smithi045 conmon[61210]: level=error ts=2021-07-31T01:52:03.517Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:52:03.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:52:03 smithi045 conmon[61210]: level=error ts=2021-07-31T01:52:03.517Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T01:52:03.799 INFO:tasks.rados.rados.0.smithi045.stdout:1929: finishing write tid 3 to smithi04569269-27 2021-07-31T01:52:03.800 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 27 v 1046 (ObjNum 601 snap 184 seq_num 601) dirty exists 2021-07-31T01:52:03.800 INFO:tasks.rados.rados.0.smithi045.stdout:1929: left oid 27 (ObjNum 601 snap 184 seq_num 601) 2021-07-31T01:52:03.800 INFO:tasks.rados.rados.0.smithi045.stdout:1926: finishing write tid 1 to smithi04569269-21 2021-07-31T01:52:03.800 INFO:tasks.rados.rados.0.smithi045.stdout:1926: finishing write tid 2 to smithi04569269-21 2021-07-31T01:52:03.801 INFO:tasks.rados.rados.0.smithi045.stdout:1924: done (8 left) 2021-07-31T01:52:03.801 INFO:tasks.rados.rados.0.smithi045.stdout:1929: done (7 left) 2021-07-31T01:52:03.801 INFO:tasks.rados.rados.0.smithi045.stdout:1935: read oid 50 snap -1 2021-07-31T01:52:03.801 INFO:tasks.rados.rados.0.smithi045.stdout:1935: expect (ObjNum 588 snap 182 seq_num 588) 2021-07-31T01:52:03.802 INFO:tasks.rados.rados.0.smithi045.stdout:1936: snap_create 2021-07-31T01:52:03.802 INFO:tasks.rados.rados.0.smithi045.stdout:1926: finishing write tid 3 to smithi04569269-21 2021-07-31T01:52:03.802 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 21 v 1233 (ObjNum 600 snap 184 seq_num 600) dirty exists 2021-07-31T01:52:03.802 INFO:tasks.rados.rados.0.smithi045.stdout:1926: left oid 21 (ObjNum 600 snap 184 seq_num 600) 2021-07-31T01:52:03.803 INFO:tasks.rados.rados.0.smithi045.stdout:1928: finishing copy_from to smithi04569269-46 2021-07-31T01:52:03.803 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 46 v 360 (ObjNum 470 snap 136 seq_num 470) dirty exists 2021-07-31T01:52:03.803 INFO:tasks.rados.rados.0.smithi045.stdout:1928: finishing copy_from racing read to smithi04569269-46 2021-07-31T01:52:03.803 INFO:tasks.rados.rados.0.smithi045.stdout:1927: expect (ObjNum 548 snap 164 seq_num 548) 2021-07-31T01:52:03.813 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T01:52:03.814 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 6m ago 7m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T01:52:03.814 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 4m ago 7m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T01:52:03.814 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 4m ago 10m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T01:52:03.814 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 6m ago 13m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T01:52:03.815 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 6m ago 13m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T01:52:03.816 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 4m ago 11m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T01:52:03.816 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 6m ago 11m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T01:52:03.816 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 6m ago 7m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T01:52:03.816 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 4m ago 7m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T01:52:03.816 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 6m ago 10m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T01:52:03.817 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 6m ago 10m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T01:52:03.817 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 6m ago 10m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T01:52:03.817 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 6m ago 9m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T01:52:03.818 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 4m ago 9m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T01:52:03.818 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 4m ago 9m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T01:52:03.818 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 4m ago 8m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T01:52:03.818 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 4m ago 8m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T01:52:03.819 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 4m ago 8m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T01:52:04.155 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T01:52:04.155 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T01:52:04.156 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T01:52:04.156 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T01:52:04.156 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T01:52:04.156 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T01:52:04.157 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T01:52:04.157 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T01:52:04.157 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T01:52:04.157 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T01:52:04.158 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T01:52:04.158 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T01:52:04.158 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T01:52:04.158 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T01:52:04.158 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T01:52:04.159 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T01:52:04.159 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T01:52:04.211 INFO:tasks.rados.rados.0.smithi045.stdout:1931: finishing write tid 1 to smithi04569269-28 2021-07-31T01:52:04.211 INFO:tasks.rados.rados.0.smithi045.stdout:1931: finishing write tid 2 to smithi04569269-28 2021-07-31T01:52:04.212 INFO:tasks.rados.rados.0.smithi045.stdout:1931: finishing write tid 3 to smithi04569269-28 2021-07-31T01:52:04.212 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 28 v 1132 (ObjNum 602 snap 184 seq_num 602) dirty exists 2021-07-31T01:52:04.213 INFO:tasks.rados.rados.0.smithi045.stdout:1931: left oid 28 (ObjNum 602 snap 184 seq_num 602) 2021-07-31T01:52:04.213 INFO:tasks.rados.rados.0.smithi045.stdout:1930: expect (ObjNum 530 snap 156 seq_num 530) 2021-07-31T01:52:04.626 INFO:tasks.rados.rados.0.smithi045.stdout:1932: finishing write tid 1 to smithi04569269-13 2021-07-31T01:52:04.627 INFO:tasks.rados.rados.0.smithi045.stdout:1932: finishing write tid 2 to smithi04569269-13 2021-07-31T01:52:04.627 INFO:tasks.rados.rados.0.smithi045.stdout:1932: finishing write tid 3 to smithi04569269-13 2021-07-31T01:52:04.627 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 13 v 709 (ObjNum 603 snap 184 seq_num 603) dirty exists 2021-07-31T01:52:04.628 INFO:tasks.rados.rados.0.smithi045.stdout:1932: left oid 13 (ObjNum 603 snap 184 seq_num 603) 2021-07-31T01:52:04.628 INFO:tasks.rados.rados.0.smithi045.stdout:1934: finishing rollback tid 0 to smithi04569269-23 2021-07-31T01:52:04.628 INFO:tasks.rados.rados.0.smithi045.stdout:1934: finishing rollback tid 1 to smithi04569269-23 2021-07-31T01:52:04.628 INFO:tasks.rados.rados.0.smithi045.stdout:1934: finishing rollback tid 2 to smithi04569269-23 2021-07-31T01:52:04.629 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 23 v 1072 (ObjNum 560 snap 166 seq_num 560) dirty exists 2021-07-31T01:52:04.629 INFO:tasks.rados.rados.0.smithi045.stdout:1926: done (8 left) 2021-07-31T01:52:04.629 INFO:tasks.rados.rados.0.smithi045.stdout:1927: done (7 left) 2021-07-31T01:52:04.629 INFO:tasks.rados.rados.0.smithi045.stdout:1928: done (6 left) 2021-07-31T01:52:04.630 INFO:tasks.rados.rados.0.smithi045.stdout:1930: done (5 left) 2021-07-31T01:52:04.630 INFO:tasks.rados.rados.0.smithi045.stdout:1931: done (4 left) 2021-07-31T01:52:04.630 INFO:tasks.rados.rados.0.smithi045.stdout:1932: done (3 left) 2021-07-31T01:52:04.630 INFO:tasks.rados.rados.0.smithi045.stdout:1934: done (2 left) 2021-07-31T01:52:04.631 INFO:tasks.rados.rados.0.smithi045.stdout:1936: done (1 left) 2021-07-31T01:52:04.631 INFO:tasks.rados.rados.0.smithi045.stdout:1937: copy_from oid 44 from oid 5 current snap is 185 2021-07-31T01:52:04.631 INFO:tasks.rados.rados.0.smithi045.stdout:1935: expect (ObjNum 588 snap 182 seq_num 588) 2021-07-31T01:52:04.985 INFO:tasks.rados.rados.0.smithi045.stdout:1935: done (1 left) 2021-07-31T01:52:04.986 INFO:tasks.rados.rados.0.smithi045.stdout:1938: read oid 7 snap -1 2021-07-31T01:52:04.986 INFO:tasks.rados.rados.0.smithi045.stdout:1938: expect (ObjNum 0 snap 0 seq_num 0) 2021-07-31T01:52:04.986 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 9 current snap is 185 2021-07-31T01:52:04.987 INFO:tasks.rados.rados.0.smithi045.stdout:1937: finishing copy_from to smithi04569269-44 2021-07-31T01:52:04.987 INFO:tasks.rados.rados.0.smithi045.stdout:1937: got expected ENOENT (src dne) 2021-07-31T01:52:04.987 INFO:tasks.rados.rados.0.smithi045.stdout:1937: finishing copy_from racing read to smithi04569269-44 2021-07-31T01:52:04.987 INFO:tasks.rados.rados.0.smithi045.stdout:1939: seq_num 604 ranges {0=524288} 2021-07-31T01:52:04.988 INFO:tasks.rados.rados.0.smithi045.stdout:1939: writing smithi04569269-9 from 0 to 524288 tid 1 2021-07-31T01:52:04.988 INFO:tasks.rados.rados.0.smithi045.stdout:1937: done (2 left) 2021-07-31T01:52:04.989 INFO:tasks.rados.rados.0.smithi045.stdout:1940: snap_create 2021-07-31T01:52:04.989 INFO:tasks.rados.rados.0.smithi045.stdout:1938: expect (ObjNum 0 snap 0 seq_num 0) 2021-07-31T01:52:04.991 INFO:tasks.rados.rados.0.smithi045.stdout:1939: finishing write tid 1 to smithi04569269-9 2021-07-31T01:52:04.991 INFO:tasks.rados.rados.0.smithi045.stdout:1939: finishing write tid 2 to smithi04569269-9 2021-07-31T01:52:04.992 INFO:tasks.rados.rados.0.smithi045.stdout:1939: finishing write tid 3 to smithi04569269-9 2021-07-31T01:52:04.992 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 9 v 1033 (ObjNum 604 snap 185 seq_num 604) dirty exists 2021-07-31T01:52:04.992 INFO:tasks.rados.rados.0.smithi045.stdout:1939: left oid 9 (ObjNum 604 snap 185 seq_num 604) 2021-07-31T01:52:05.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:04 smithi154 conmon[34553]: audit 2021-07-31T01:52:03.512579+0000 mgr.x (mgr.24542) 2021-07-31T01:52:05.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:04 smithi154 conmon[34553]: 204 : audit [DBG] from='client.24781 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:52:05.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:04 smithi154 conmon[34553]: cluster 2021-07-31T01:52:03.603384+0000 mon.a (mon.0) 1073 : cluster [DBG] osdmap e424: 8 total, 8 up, 8 in 2021-07-31T01:52:05.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:04 smithi154 conmon[34553]: audit 2021-07-31T01:52 2021-07-31T01:52:05.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:04 smithi154 conmon[34553]: :03.805539+0000 mgr.x (mgr.24542) 205 : audit 2021-07-31T01:52:05.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:04 smithi154 conmon[34553]: [DBG] from='client.24787 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:52:05.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:04 smithi154 conmon[34553]: cluster 2021 2021-07-31T01:52:05.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:04 smithi154 conmon[34553]: -07-31T01:52:03.858833+0000 2021-07-31T01:52:05.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:04 smithi154 conmon[34553]: mgr.x (mgr.24542) 206 : cluster [DBG] pgmap v416: 25 pgs: 25 active+clean; 247 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 18 MiB/s rd, 1.7 MiB/s wr, 19 op/s 2021-07-31T01:52:05.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:04 smithi154 conmon[34553]: 2021-07-31T01:52:05.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:04 smithi154 conmon[34553]: audit 2021-07-31T01:52: 2021-07-31T01:52:05.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:04 smithi154 conmon[34553]: 04.154160+0000 mon.a (mon.0) 1074 : audit 2021-07-31T01:52:05.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:04 smithi154 conmon[34553]: [DBG] from='client.? 172.21.15.45:0/2839971497' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T01:52:05.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:04 smithi045 conmon[33655]: audit 2021-07-31T01:52:03 2021-07-31T01:52:05.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:04 smithi045 conmon[33655]: .512579+0000 mgr.x (mgr.24542) 204 : audit [DBG] from='client.24781 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:52:05.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:04 smithi045 conmon[33655]: cluster 2021-07-31T01: 2021-07-31T01:52:05.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:04 smithi045 conmon[33655]: 52:03.603384+0000 mon.a (mon.0) 1073 : cluster [DBG] osdmap e424: 8 total, 8 up, 8 in 2021-07-31T01:52:05.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:04 smithi045 conmon[33655]: audit 2021-07-31T01:52:05.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:04 smithi045 conmon[33655]: 2021-07-31T01:52:03.805539 2021-07-31T01:52:05.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:04 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 205 : audit [DBG] 2021-07-31T01:52:05.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:04 smithi045 conmon[33655]: from='client.24787 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:52:05.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:04 smithi045 conmon[33655]: cluster 2021-07- 2021-07-31T01:52:05.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:04 smithi045 conmon[33655]: 31T01:52:03.858833+0000 mgr.x ( 2021-07-31T01:52:05.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:04 smithi045 conmon[33655]: mgr.24542) 206 : cluster [DBG] pgmap v416: 25 pgs: 25 active+clean; 247 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 18 MiB/s rd, 1.7 MiB/s wr, 19 op/s 2021-07-31T01:52:05.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:04 smithi045 conmon[33655]: audit 2021-07-31T01:52:05.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:04 smithi045 conmon[33655]: 2021-07-31T01:52:04.154160+0000 mon.a 2021-07-31T01:52:05.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:04 smithi045 conmon[33655]: (mon.0) 1074 : audit [DBG] from='client.? 172.21.15.45:0/2839971497' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T01:52:05.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:04 smithi045 conmon[39718]: audit 2021-07-31T01:52:05.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:04 smithi045 conmon[39718]: 2021-07-31T01:52:03 2021-07-31T01:52:05.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:04 smithi045 conmon[39718]: .512579+0000 mgr.x (mgr.24542) 204 : audit [DBG] from='client.24781 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:52:05.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:04 smithi045 conmon[39718]: cluster 2021-07-31T01:52:05.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:04 smithi045 conmon[39718]: 2021-07-31T01:52:03.603384+0000 mon.a (mon 2021-07-31T01:52:05.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:04 smithi045 conmon[39718]: .0) 1073 : cluster [DBG] osdmap e424: 8 total, 8 up, 8 in 2021-07-31T01:52:05.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:04 smithi045 conmon[39718]: audit 2021-07-31T01: 2021-07-31T01:52:05.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:04 smithi045 conmon[39718]: 52:03.805539+0000 mgr.x (mgr.24542) 205 : audit [DBG] 2021-07-31T01:52:05.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:04 smithi045 conmon[39718]: from='client.24787 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:52:05.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:04 smithi045 conmon[39718]: cluster 2021-07-31T01:52:05.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:04 smithi045 conmon[39718]: 2021-07-31T01:52:03.858833+0000 mgr.x ( 2021-07-31T01:52:05.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:04 smithi045 conmon[39718]: mgr.24542) 206 : cluster [DBG] pgmap v416: 25 pgs: 25 active+clean; 247 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 18 MiB/s rd, 1.7 MiB/s wr, 19 op/s 2021-07-31T01:52:05.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:04 smithi045 conmon[39718]: audit 2021-07-31T01: 2021-07-31T01:52:05.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:04 smithi045 conmon[39718]: 52:04.154160+0000 mon.a (mon.0) 1074 : audit 2021-07-31T01:52:05.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:04 smithi045 conmon[39718]: [DBG] from='client.? 172.21.15.45:0/2839971497' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T01:52:05.610 INFO:tasks.rados.rados.0.smithi045.stdout:1938: done (2 left) 2021-07-31T01:52:05.611 INFO:tasks.rados.rados.0.smithi045.stdout:1939: done (1 left) 2021-07-31T01:52:05.611 INFO:tasks.rados.rados.0.smithi045.stdout:1940: done (0 left) 2021-07-31T01:52:05.611 INFO:tasks.rados.rados.0.smithi045.stdout:1941: snap_remove snap 185 2021-07-31T01:52:06.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:05 smithi154 conmon[34553]: cluster 2021-07-31T01:52:04.608366+0000 mon.a (mon.0) 1075 : cluster [DBG] osdmap e425: 8 total, 8 up, 8 in 2021-07-31T01:52:06.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:05 smithi045 conmon[33655]: cluster 2021-07-31T01:52:04.608366+0000 mon.a (mon.0 2021-07-31T01:52:06.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:05 smithi045 conmon[33655]: ) 1075 : cluster [DBG] osdmap e425: 8 total, 8 up, 8 in 2021-07-31T01:52:06.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:05 smithi045 conmon[39718]: cluster 2021-07-31T01:52:04.608366+0000 mon.a (mon 2021-07-31T01:52:06.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:05 smithi045 conmon[39718]: .0) 1075 : cluster [DBG] osdmap e425: 8 total, 8 up, 8 in 2021-07-31T01:52:06.566 INFO:tasks.rados.rados.0.smithi045.stdout:1941: done (0 left) 2021-07-31T01:52:06.567 INFO:tasks.rados.rados.0.smithi045.stdout:1942: read oid 39 snap -1 2021-07-31T01:52:06.567 INFO:tasks.rados.rados.0.smithi045.stdout:1942: expect (ObjNum 575 snap 175 seq_num 575) 2021-07-31T01:52:06.567 INFO:tasks.rados.rados.0.smithi045.stdout:1943: read oid 18 snap -1 2021-07-31T01:52:06.567 INFO:tasks.rados.rados.0.smithi045.stdout:1943: expect deleted 2021-07-31T01:52:06.568 INFO:tasks.rados.rados.0.smithi045.stdout:1944: delete oid 8 current snap is 186 2021-07-31T01:52:06.568 INFO:tasks.rados.rados.0.smithi045.stdout:1942: expect (ObjNum 575 snap 175 seq_num 575) 2021-07-31T01:52:06.573 INFO:tasks.rados.rados.0.smithi045.stdout:1942: done (2 left) 2021-07-31T01:52:06.574 INFO:tasks.rados.rados.0.smithi045.stdout:1943: done (1 left) 2021-07-31T01:52:06.574 INFO:tasks.rados.rados.0.smithi045.stdout:1944: done (0 left) 2021-07-31T01:52:06.574 INFO:tasks.rados.rados.0.smithi045.stdout:1945: copy_from oid 7 from oid 41 current snap is 186 2021-07-31T01:52:06.574 INFO:tasks.rados.rados.0.smithi045.stdout:1946: read oid 34 snap -1 2021-07-31T01:52:06.575 INFO:tasks.rados.rados.0.smithi045.stdout:1946: expect (ObjNum 399 snap 121 seq_num 399) 2021-07-31T01:52:06.575 INFO:tasks.rados.rados.0.smithi045.stdout:1947: delete oid 16 current snap is 186 2021-07-31T01:52:06.577 INFO:tasks.rados.rados.0.smithi045.stdout:1947: done (2 left) 2021-07-31T01:52:06.578 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 25 current snap is 186 2021-07-31T01:52:06.578 INFO:tasks.rados.rados.0.smithi045.stdout:1948: seq_num 607 ranges {5455872=573440} 2021-07-31T01:52:06.611 INFO:tasks.rados.rados.0.smithi045.stdout:1948: writing smithi04569269-25 from 5455872 to 6029312 tid 1 2021-07-31T01:52:06.612 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 1 current snap is 186 2021-07-31T01:52:06.612 INFO:tasks.rados.rados.0.smithi045.stdout:1949: seq_num 608 ranges {2711552=589824} 2021-07-31T01:52:06.629 INFO:tasks.rados.rados.0.smithi045.stdout:1949: writing smithi04569269-1 from 2711552 to 3301376 tid 1 2021-07-31T01:52:06.630 INFO:tasks.rados.rados.0.smithi045.stdout:1950: setattr oid 30 current snap is 186 2021-07-31T01:52:06.630 INFO:tasks.rados.rados.0.smithi045.stdout:1946: expect (ObjNum 399 snap 121 seq_num 399) 2021-07-31T01:52:06.871 INFO:tasks.rados.rados.0.smithi045.stdout:1945: finishing copy_from to smithi04569269-7 2021-07-31T01:52:06.872 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 7 v 977 (ObjNum 552 snap 166 seq_num 552) dirty exists 2021-07-31T01:52:06.872 INFO:tasks.rados.rados.0.smithi045.stdout:1945: finishing copy_from racing read to smithi04569269-7 2021-07-31T01:52:06.872 INFO:tasks.rados.rados.0.smithi045.stdout:1948: finishing write tid 1 to smithi04569269-25 2021-07-31T01:52:06.872 INFO:tasks.rados.rados.0.smithi045.stdout:1948: finishing write tid 2 to smithi04569269-25 2021-07-31T01:52:06.873 INFO:tasks.rados.rados.0.smithi045.stdout:1948: finishing write tid 3 to smithi04569269-25 2021-07-31T01:52:06.873 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 25 v 290 (ObjNum 607 snap 186 seq_num 607) dirty exists 2021-07-31T01:52:06.873 INFO:tasks.rados.rados.0.smithi045.stdout:1948: left oid 25 (ObjNum 607 snap 186 seq_num 607) 2021-07-31T01:52:06.873 INFO:tasks.rados.rados.0.smithi045.stdout:1949: finishing write tid 1 to smithi04569269-1 2021-07-31T01:52:06.874 INFO:tasks.rados.rados.0.smithi045.stdout:1949: finishing write tid 2 to smithi04569269-1 2021-07-31T01:52:06.874 INFO:tasks.rados.rados.0.smithi045.stdout:1949: finishing write tid 3 to smithi04569269-1 2021-07-31T01:52:06.874 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 1 v 1010 (ObjNum 608 snap 186 seq_num 608) dirty exists 2021-07-31T01:52:06.875 INFO:tasks.rados.rados.0.smithi045.stdout:1949: left oid 1 (ObjNum 608 snap 186 seq_num 608) 2021-07-31T01:52:06.875 INFO:tasks.rados.rados.0.smithi045.stdout:1945: done (4 left) 2021-07-31T01:52:06.875 INFO:tasks.rados.rados.0.smithi045.stdout:1946: done (3 left) 2021-07-31T01:52:06.875 INFO:tasks.rados.rados.0.smithi045.stdout:1948: done (2 left) 2021-07-31T01:52:06.876 INFO:tasks.rados.rados.0.smithi045.stdout:1949: done (1 left) 2021-07-31T01:52:06.876 INFO:tasks.rados.rados.0.smithi045.stdout:1951: delete oid 9 current snap is 186 2021-07-31T01:52:06.877 INFO:tasks.rados.rados.0.smithi045.stdout:1951: done (1 left) 2021-07-31T01:52:06.877 INFO:tasks.rados.rados.0.smithi045.stdout:1952: read oid 40 snap -1 2021-07-31T01:52:06.878 INFO:tasks.rados.rados.0.smithi045.stdout:1952: expect (ObjNum 596 snap 184 seq_num 596) 2021-07-31T01:52:06.878 INFO:tasks.rados.rados.0.smithi045.stdout:1953: read oid 2 snap 182 2021-07-31T01:52:06.878 INFO:tasks.rados.rados.0.smithi045.stdout:1953: expect (ObjNum 399 snap 121 seq_num 399) 2021-07-31T01:52:06.878 INFO:tasks.rados.rados.0.smithi045.stdout:1954: delete oid 47 current snap is 186 2021-07-31T01:52:06.879 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 30 v 524 (ObjNum -1229692432 snap 32731 seq_num 3065274864) dirty exists 2021-07-31T01:52:06.879 INFO:tasks.rados.rados.0.smithi045.stdout:1950: done (3 left) 2021-07-31T01:52:06.879 INFO:tasks.rados.rados.0.smithi045.stdout:1954: done (2 left) 2021-07-31T01:52:06.879 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 39 current snap is 186 2021-07-31T01:52:06.881 INFO:tasks.rados.rados.0.smithi045.stdout:1955: seq_num 611 ranges {32768=638976} 2021-07-31T01:52:06.882 INFO:tasks.rados.rados.0.smithi045.stdout:1955: writing smithi04569269-39 from 32768 to 671744 tid 1 2021-07-31T01:52:06.882 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 28 current snap is 186 2021-07-31T01:52:06.882 INFO:tasks.rados.rados.0.smithi045.stdout:1956: seq_num 612 ranges {491520=655360} 2021-07-31T01:52:06.887 INFO:tasks.rados.rados.0.smithi045.stdout:1956: writing smithi04569269-28 from 491520 to 1146880 tid 1 2021-07-31T01:52:06.887 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 11 current snap is 186 2021-07-31T01:52:06.888 INFO:tasks.rados.rados.0.smithi045.stdout:1957: seq_num 613 ranges {7479296=671744} 2021-07-31T01:52:06.932 INFO:tasks.rados.rados.0.smithi045.stdout:1957: writing smithi04569269-11 from 7479296 to 8151040 tid 1 2021-07-31T01:52:06.932 INFO:tasks.rados.rados.0.smithi045.stdout:1958: read oid 36 snap -1 2021-07-31T01:52:06.933 INFO:tasks.rados.rados.0.smithi045.stdout:1958: expect deleted 2021-07-31T01:52:06.933 INFO:tasks.rados.rados.0.smithi045.stdout:1959: snap_remove snap 176 2021-07-31T01:52:07.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:06 smithi154 conmon[34553]: cluster 2021-07-31T01:52:05.610239+0000 mon.a (mon.0) 1076 2021-07-31T01:52:07.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:06 smithi154 conmon[34553]: : cluster [DBG] osdmap e426: 8 total, 8 up, 8 in 2021-07-31T01:52:07.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:06 smithi154 conmon[34553]: cluster 2021-07-31T01:52:05.859365+0000 mgr.x (mgr.24542) 207 : cluster [DBG] pgmap v419: 25 pgs: 25 active+clean; 247 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 18 MiB/s rd, 1.7 MiB/s wr, 19 op/s 2021-07-31T01:52:07.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:06 smithi154 conmon[34553]: cluster 2021-07-31T01:52:06.564845 2021-07-31T01:52:07.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:06 smithi154 conmon[34553]: +0000 mon.a (mon.0) 1077 : cluster [DBG] osdmap e427: 8 total, 8 up, 8 in 2021-07-31T01:52:07.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:06 smithi045 conmon[39718]: cluster 2021-07-31T01:52:07.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:06 smithi045 conmon[39718]: 2021-07-31T01:52:07.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:06 smithi045 conmon[39718]: 2021-07-31T01:52:05 2021-07-31T01:52:07.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:06 smithi045 conmon[39718]: .610239+0000 mon.a (mon.0) 1076 : cluster [DBG] osdmap e426: 8 total, 8 up, 8 in 2021-07-31T01:52:07.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:06 smithi045 conmon[39718]: cluster 2021-07-31T01:52:07.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:06 smithi045 conmon[39718]: 2021-07-31T01:52:05.859365 2021-07-31T01:52:07.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:06 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 207 : cluster 2021-07-31T01:52:07.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:06 smithi045 conmon[39718]: [DBG] pgmap v419: 25 pgs: 25 active+clean; 247 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 18 MiB/s rd, 1.7 MiB/s wr, 19 op/s 2021-07-31T01:52:07.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:06 smithi045 conmon[39718]: cluster 2021-07-31T01: 2021-07-31T01:52:07.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:06 smithi045 conmon[39718]: 52:06.564845+0000 mon.a (mon.0) 2021-07-31T01:52:07.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:06 smithi045 conmon[39718]: 1077 : cluster [DBG] osdmap e427: 8 total, 8 up, 8 in 2021-07-31T01:52:07.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:06 smithi045 conmon[33655]: cluster 2021-07-31T01:52: 2021-07-31T01:52:07.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:06 smithi045 conmon[33655]: 05.610239+0000 mon.a (mon.0) 1076 : cluster [DBG] osdmap e426: 8 total, 8 up, 8 in 2021-07-31T01:52:07.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:06 smithi045 conmon[33655]: cluster 2021-07-31T01:52:05.859365+0000 mgr.x (mgr.24542) 207 : cluster [DBG] pgmap v419: 25 pgs: 25 active+clean; 247 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 18 MiB/s rd, 1.7 MiB/s wr, 19 op/s 2021-07-31T01:52:07.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:06 smithi045 conmon[33655]: cluster 2021-07-31T01:52:06.564845+0000 mon.a (mon.0) 1077 : cluster [DBG] osdmap e427: 8 total, 8 up, 8 in 2021-07-31T01:52:07.630 INFO:tasks.rados.rados.0.smithi045.stdout:1959: done (6 left) 2021-07-31T01:52:07.630 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 34 current snap is 186 2021-07-31T01:52:07.631 INFO:tasks.rados.rados.0.smithi045.stdout:1952: expect (ObjNum 596 snap 184 seq_num 596) 2021-07-31T01:52:07.644 INFO:tasks.rados.rados.0.smithi045.stdout:1960: seq_num 614 ranges {5775360=688128} 2021-07-31T01:52:07.680 INFO:tasks.rados.rados.0.smithi045.stdout:1960: writing smithi04569269-34 from 5775360 to 6463488 tid 1 2021-07-31T01:52:07.681 INFO:tasks.rados.rados.0.smithi045.stdout:1952: done (6 left) 2021-07-31T01:52:07.682 INFO:tasks.rados.rados.0.smithi045.stdout:1961: read oid 46 snap 183 2021-07-31T01:52:07.682 INFO:tasks.rados.rados.0.smithi045.stdout:1961: expect (ObjNum 540 snap 162 seq_num 540) 2021-07-31T01:52:07.683 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 18 current snap is 186 2021-07-31T01:52:07.683 INFO:tasks.rados.rados.0.smithi045.stdout:1955: finishing write tid 1 to smithi04569269-39 2021-07-31T01:52:07.683 INFO:tasks.rados.rados.0.smithi045.stdout:1955: finishing write tid 2 to smithi04569269-39 2021-07-31T01:52:07.683 INFO:tasks.rados.rados.0.smithi045.stdout:1955: finishing write tid 3 to smithi04569269-39 2021-07-31T01:52:07.684 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 39 v 1074 (ObjNum 611 snap 186 seq_num 611) dirty exists 2021-07-31T01:52:07.684 INFO:tasks.rados.rados.0.smithi045.stdout:1955: left oid 39 (ObjNum 611 snap 186 seq_num 611) 2021-07-31T01:52:07.684 INFO:tasks.rados.rados.0.smithi045.stdout:1962: seq_num 615 ranges {0=704512} 2021-07-31T01:52:07.685 INFO:tasks.rados.rados.0.smithi045.stdout:1962: writing smithi04569269-18 from 0 to 704512 tid 1 2021-07-31T01:52:07.685 INFO:tasks.rados.rados.0.smithi045.stdout:1955: done (7 left) 2021-07-31T01:52:07.685 INFO:tasks.rados.rados.0.smithi045.stdout:1963: snap_remove snap 164 2021-07-31T01:52:08.631 INFO:tasks.rados.rados.0.smithi045.stdout:1963: done (7 left) 2021-07-31T01:52:08.631 INFO:tasks.rados.rados.0.smithi045.stdout:1964: read oid 35 snap -1 2021-07-31T01:52:08.631 INFO:tasks.rados.rados.0.smithi045.stdout:1964: expect (ObjNum 593 snap 184 seq_num 593) 2021-07-31T01:52:08.632 INFO:tasks.rados.rados.0.smithi045.stdout:1965: snap_create 2021-07-31T01:52:08.632 INFO:tasks.rados.rados.0.smithi045.stdout:1953: expect (ObjNum 399 snap 121 seq_num 399) 2021-07-31T01:52:08.883 INFO:tasks.rados.rados.0.smithi045.stdout:1956: finishing write tid 1 to smithi04569269-28 2021-07-31T01:52:08.883 INFO:tasks.rados.rados.0.smithi045.stdout:1956: finishing write tid 2 to smithi04569269-28 2021-07-31T01:52:08.884 INFO:tasks.rados.rados.0.smithi045.stdout:1956: finishing write tid 3 to smithi04569269-28 2021-07-31T01:52:08.884 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 28 v 1134 (ObjNum 612 snap 186 seq_num 612) dirty exists 2021-07-31T01:52:08.884 INFO:tasks.rados.rados.0.smithi045.stdout:1956: left oid 28 (ObjNum 612 snap 186 seq_num 612) 2021-07-31T01:52:08.884 INFO:tasks.rados.rados.0.smithi045.stdout:1957: finishing write tid 1 to smithi04569269-11 2021-07-31T01:52:08.885 INFO:tasks.rados.rados.0.smithi045.stdout:1957: finishing write tid 2 to smithi04569269-11 2021-07-31T01:52:08.885 INFO:tasks.rados.rados.0.smithi045.stdout:1957: finishing write tid 3 to smithi04569269-11 2021-07-31T01:52:08.885 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 11 v 1076 (ObjNum 613 snap 186 seq_num 613) dirty exists 2021-07-31T01:52:08.885 INFO:tasks.rados.rados.0.smithi045.stdout:1957: left oid 11 (ObjNum 613 snap 186 seq_num 613) 2021-07-31T01:52:08.885 INFO:tasks.rados.rados.0.smithi045.stdout:1960: finishing write tid 1 to smithi04569269-34 2021-07-31T01:52:08.886 INFO:tasks.rados.rados.0.smithi045.stdout:1960: finishing write tid 2 to smithi04569269-34 2021-07-31T01:52:08.886 INFO:tasks.rados.rados.0.smithi045.stdout:1960: finishing write tid 3 to smithi04569269-34 2021-07-31T01:52:08.886 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 34 v 1012 (ObjNum 614 snap 186 seq_num 614) dirty exists 2021-07-31T01:52:08.886 INFO:tasks.rados.rados.0.smithi045.stdout:1960: left oid 34 (ObjNum 614 snap 186 seq_num 614) 2021-07-31T01:52:08.887 INFO:tasks.rados.rados.0.smithi045.stdout:1962: finishing write tid 1 to smithi04569269-18 2021-07-31T01:52:08.887 INFO:tasks.rados.rados.0.smithi045.stdout:1962: finishing write tid 2 to smithi04569269-18 2021-07-31T01:52:08.887 INFO:tasks.rados.rados.0.smithi045.stdout:1962: finishing write tid 3 to smithi04569269-18 2021-07-31T01:52:08.887 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 18 v 490 (ObjNum 615 snap 186 seq_num 615) dirty exists 2021-07-31T01:52:08.887 INFO:tasks.rados.rados.0.smithi045.stdout:1962: left oid 18 (ObjNum 615 snap 186 seq_num 615) 2021-07-31T01:52:08.888 INFO:tasks.rados.rados.0.smithi045.stdout:1961: expect (ObjNum 540 snap 162 seq_num 540) 2021-07-31T01:52:09.002 INFO:journalctl@ceph.osd.5.smithi154.stdout:Jul 31 01:52:08 smithi154 conmon[42050]: debug 2021-07-31T01:52:09.003 INFO:journalctl@ceph.osd.5.smithi154.stdout:Jul 31 01:52:08 smithi154 conmon[42050]: 2021-07-31T01:52:08.637+0000 7fc9b105c700 -1 osd.5 pg_epoch: 429 pg[3.cs0( v 429'762 (0'0,429'762] local-lis/les=56/57 n=14 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [5,3,6]p5(0) r=0 lpr=56 luod=428'760 crt=428'760 lcod 428'758 mlcod 428'758 active+clean+snaptrim trimq=[147~1] ps=[156~1,16e~1]] removing snap head 2021-07-31T01:52:09.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:08 smithi154 conmon[34553]: cluster 2021-07-31T01:52:07.625309+0000 mon.a (mon.0) 2021-07-31T01:52:09.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:08 smithi154 conmon[34553]: 1078 : cluster [DBG] osdmap e428: 8 total, 8 up, 8 in 2021-07-31T01:52:09.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:08 smithi154 conmon[34553]: cluster 2021-07-31T01:52:07.859751+0000 mgr.x (mgr.24542) 208 : 2021-07-31T01:52:09.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:08 smithi154 conmon[34553]: cluster [DBG] pgmap v422: 25 pgs: 25 active+clean; 247 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2021-07-31T01:52:09.026 INFO:journalctl@ceph.osd.3.smithi045.stdout:Jul 31 01:52:08 smithi045 conmon[54013]: debug 2021-07-31T01:52:08.659+0000 7f64fc8cd700 -1 osd.3 pg_epoch: 429 pg[3.bs0( v 429'1261 (0'0,429'1261] local-lis/les=56/57 n=16 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [3,0,4]p3(0) r=0 lpr=56 crt=429'1261 lcod 429'1259 mlcod 429'1259 active+clean+snaptrim trimq=[147~1] ps=[156~1,16e~1]] removing snap head 2021-07-31T01:52:09.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:08 smithi045 conmon[33655]: cluster 2021-07-31T01:52:07.625309+0000 mon.a (mon.0) 1078 : cluster 2021-07-31T01:52:09.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:08 smithi045 conmon[33655]: [DBG] osdmap e428: 8 total, 8 up, 8 in 2021-07-31T01:52:09.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:08 smithi045 conmon[33655]: cluster 2021-07-31T01:52:07.859751+0000 mgr.x (mgr.24542) 208 : cluster [DBG] pgmap v422: 25 pgs: 25 active+clean; 247 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2021-07-31T01:52:09.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:08 smithi045 conmon[39718]: cluster 2021-07-31T01:52:07.625309+0000 mon.a (mon.0 2021-07-31T01:52:09.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:08 smithi045 conmon[39718]: ) 1078 : cluster [DBG] osdmap e428: 8 total, 8 up, 8 in 2021-07-31T01:52:09.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:08 smithi045 conmon[39718]: cluster 2021-07-31T01:52:07 2021-07-31T01:52:09.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:08 smithi045 conmon[39718]: .859751+0000 mgr.x (mgr.24542) 208 : cluster [DBG] pgmap v422: 25 pgs: 25 active+clean; 247 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2021-07-31T01:52:09.995 INFO:tasks.rados.rados.0.smithi045.stdout:1964: expect (ObjNum 593 snap 184 seq_num 593) 2021-07-31T01:52:10.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:09 smithi154 conmon[34553]: cluster 2021-07-31T01:52:08.630200+0000 mon.a (mon.0) 1079 : cluster [DBG] osdmap e429: 8 total, 8 up, 8 in 2021-07-31T01:52:10.004 INFO:tasks.rados.rados.0.smithi045.stdout:1953: done (8 left) 2021-07-31T01:52:10.004 INFO:tasks.rados.rados.0.smithi045.stdout:1956: done (7 left) 2021-07-31T01:52:10.005 INFO:tasks.rados.rados.0.smithi045.stdout:1957: done (6 left) 2021-07-31T01:52:10.005 INFO:tasks.rados.rados.0.smithi045.stdout:1958: done (5 left) 2021-07-31T01:52:10.005 INFO:tasks.rados.rados.0.smithi045.stdout:1960: done (4 left) 2021-07-31T01:52:10.005 INFO:tasks.rados.rados.0.smithi045.stdout:1961: done (3 left) 2021-07-31T01:52:10.006 INFO:tasks.rados.rados.0.smithi045.stdout:1962: done (2 left) 2021-07-31T01:52:10.006 INFO:tasks.rados.rados.0.smithi045.stdout:1964: done (1 left) 2021-07-31T01:52:10.006 INFO:tasks.rados.rados.0.smithi045.stdout:1965: done (0 left) 2021-07-31T01:52:10.006 INFO:tasks.rados.rados.0.smithi045.stdout:1966: snap_remove snap 183 2021-07-31T01:52:10.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:09 smithi045 conmon[39718]: cluster 2021-07-31T01:52:08.630200 2021-07-31T01:52:10.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:09 smithi045 conmon[39718]: +0000 mon.a (mon.0) 1079 : cluster [DBG] osdmap e429: 8 total, 8 up, 8 in 2021-07-31T01:52:10.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:09 smithi045 conmon[33655]: cluster 2021-07-31T01:52:08.630200+0000 mon.a ( 2021-07-31T01:52:10.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:09 smithi045 conmon[33655]: mon.0) 1079 : cluster [DBG] osdmap e429: 8 total, 8 up, 8 in 2021-07-31T01:52:10.647 INFO:tasks.rados.rados.0.smithi045.stdout:1966: done (0 left) 2021-07-31T01:52:10.647 INFO:tasks.rados.rados.0.smithi045.stdout:1967: read oid 8 snap -1 2021-07-31T01:52:10.648 INFO:tasks.rados.rados.0.smithi045.stdout:1967: expect deleted 2021-07-31T01:52:10.648 INFO:tasks.rados.rados.0.smithi045.stdout:1968: read oid 35 snap -1 2021-07-31T01:52:10.648 INFO:tasks.rados.rados.0.smithi045.stdout:1968: expect (ObjNum 593 snap 184 seq_num 593) 2021-07-31T01:52:10.648 INFO:tasks.rados.rados.0.smithi045.stdout:1969: read oid 6 snap -1 2021-07-31T01:52:10.649 INFO:tasks.rados.rados.0.smithi045.stdout:1969: expect (ObjNum 594 snap 184 seq_num 594) 2021-07-31T01:52:10.649 INFO:tasks.rados.rados.0.smithi045.stdout:1970: snap_remove snap 184 2021-07-31T01:52:11.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:10 smithi154 conmon[34553]: cluster 2021-07-31T01:52:09.630360+0000 mon.a (mon.0) 1080 : cluster [DBG] 2021-07-31T01:52:11.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:10 smithi154 conmon[34553]: osdmap e430: 8 total, 8 up, 8 in 2021-07-31T01:52:11.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:10 smithi154 conmon[34553]: cluster 2021-07-31T01:52:09.860204+0000 mgr.x (mgr.24542) 209 : cluster [DBG] pgmap v425: 25 pgs: 25 active+clean; 210 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 18 MiB/s rd, 2.6 MiB/s wr, 19 op/s 2021-07-31T01:52:11.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:10 smithi045 conmon[39718]: cluster 2021-07-31T01:52:09.630360 2021-07-31T01:52:11.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:10 smithi045 conmon[39718]: +0000 mon.a (mon.0) 1080 : cluster [DBG] osdmap e430: 8 total, 8 up, 8 in 2021-07-31T01:52:11.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:10 smithi045 conmon[39718]: cluster 2021-07-31T01:52:11.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:10 smithi045 conmon[39718]: 2021-07 2021-07-31T01:52:11.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:10 smithi045 conmon[39718]: -31T01:52:09. 2021-07-31T01:52:11.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:10 smithi045 conmon[39718]: 860204+0000 mgr.x (mgr.24542 2021-07-31T01:52:11.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:10 smithi045 conmon[39718]: ) 209 : cluster [DBG] pgmap v425: 25 pgs: 25 active+clean; 210 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 18 MiB/s rd, 2.6 MiB/s wr, 19 op/s 2021-07-31T01:52:11.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:10 smithi045 conmon[39718]: 2021-07-31T01:52:11.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:10 smithi045 conmon[33655]: cluster 2021-07-31T01:52:09.630360+0000 mon.a (mon. 2021-07-31T01:52:11.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:10 smithi045 conmon[33655]: 0) 1080 : cluster [DBG] osdmap e430: 8 total, 8 up, 8 in 2021-07-31T01:52:11.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:10 smithi045 conmon[33655]: cluster 2021-07-31T01:52:09.860204+0000 mgr.x (mgr.24542) 209 : cluster [DBG] pgmap v425: 25 pgs: 25 active+clean; 210 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 18 MiB/s rd, 2.6 MiB/s wr, 19 op/s 2021-07-31T01:52:11.567 INFO:tasks.rados.rados.0.smithi045.stdout:1970: done (3 left) 2021-07-31T01:52:11.568 INFO:tasks.rados.rados.0.smithi045.stdout:1971: snap_create 2021-07-31T01:52:11.568 INFO:tasks.rados.rados.0.smithi045.stdout:1968: expect (ObjNum 593 snap 184 seq_num 593) 2021-07-31T01:52:11.581 INFO:tasks.rados.rados.0.smithi045.stdout:1969: expect (ObjNum 594 snap 184 seq_num 594) 2021-07-31T01:52:12.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:11 smithi154 conmon[34553]: cluster 2021-07-31T01:52:10.645648+0000 mon.a (mon.0) 1081 : cluster [DBG] 2021-07-31T01:52:12.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:11 smithi154 conmon[34553]: osdmap e431: 8 total, 8 up, 8 in 2021-07-31T01:52:12.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:11 smithi154 conmon[34553]: cluster 2021-07-31T01:52:11.566537+0000 mon.a (mon.0) 1082 : cluster [DBG] osdmap e432: 8 total, 8 up, 8 in 2021-07-31T01:52:12.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:11 smithi045 conmon[33655]: cluster 2021-07-31T01:52:10.645648+0000 2021-07-31T01:52:12.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:11 smithi045 conmon[33655]: mon.a (mon.0) 1081 : cluster [DBG] osdmap e431: 8 total, 8 up, 8 in 2021-07-31T01:52:12.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:11 smithi045 conmon[33655]: cluster 2021-07-31T01:52:11.566537+0000 mon.a (mon.0) 1082 : cluster [DBG] 2021-07-31T01:52:12.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:11 smithi045 conmon[33655]: osdmap e432: 8 total, 8 up, 8 in 2021-07-31T01:52:12.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:11 smithi045 conmon[39718]: cluster 2021-07-31T01:52 2021-07-31T01:52:12.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:11 smithi045 conmon[39718]: :10.645648+0000 mon.a (mon.0) 1081 : cluster [DBG] osdmap e431: 8 total, 8 up, 8 in 2021-07-31T01:52:12.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:11 smithi045 conmon[39718]: cluster 2021 2021-07-31T01:52:12.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:11 smithi045 conmon[39718]: -07-31T01:52:11.566537+0000 2021-07-31T01:52:12.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:11 smithi045 conmon[39718]: mon.a (mon.0) 1082 : cluster [DBG] 2021-07-31T01:52:12.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:11 smithi045 conmon[39718]: osdmap e432: 8 total, 8 up, 8 in 2021-07-31T01:52:12.575 INFO:tasks.rados.rados.0.smithi045.stdout:1967: done (3 left) 2021-07-31T01:52:12.575 INFO:tasks.rados.rados.0.smithi045.stdout:1968: done (2 left) 2021-07-31T01:52:12.576 INFO:tasks.rados.rados.0.smithi045.stdout:1969: done (1 left) 2021-07-31T01:52:12.576 INFO:tasks.rados.rados.0.smithi045.stdout:1971: done (0 left) 2021-07-31T01:52:12.576 INFO:tasks.rados.rados.0.smithi045.stdout:1972: snap_remove snap 187 2021-07-31T01:52:13.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:12 smithi154 conmon[34553]: cluster 2021-07-31T01:52:11.860787+0000 mgr.x (mgr.24542) 210 : cluster [DBG] pgmap v428: 25 pgs: 25 active+clean; 210 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 18 MiB/s rd, 2.6 MiB/s wr, 19 op/s 2021-07-31T01:52:13.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:12 smithi154 conmon[34553]: cluster 2021-07-31T01:52:12. 2021-07-31T01:52:13.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:12 smithi154 conmon[34553]: 574122+0000 mon.a (mon.0) 1083 : cluster [DBG] osdmap e433: 8 total, 8 up, 8 in 2021-07-31T01:52:13.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:12 smithi045 conmon[33655]: cluster 2021-07-31T01:52:11.860787+0000 mgr.x (mgr.24542) 210 : cluster 2021-07-31T01:52:13.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:12 smithi045 conmon[33655]: [DBG] pgmap v428: 25 pgs: 25 active+clean; 210 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 18 MiB/s rd, 2.6 MiB/s wr, 19 op/s 2021-07-31T01:52:13.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:12 smithi045 conmon[33655]: cluster 2021-07-31T01:52:12.574122+0000 mon.a (mon.0) 1083 : cluster [DBG] osdmap e433: 8 total, 8 up, 8 in 2021-07-31T01:52:13.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:12 smithi045 conmon[39718]: cluster 2021-07-31T01:52:11 2021-07-31T01:52:13.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:12 smithi045 conmon[39718]: .860787+0000 mgr.x (mgr.24542) 210 : cluster [DBG] pgmap v428: 25 pgs: 25 active+clean; 210 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 18 MiB/s rd, 2.6 MiB/s wr, 19 op/s 2021-07-31T01:52:13.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:12 smithi045 conmon[39718]: cluster 2021-07-31T01:52:12.574122+0000 2021-07-31T01:52:13.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:12 smithi045 conmon[39718]: mon.a (mon.0) 1083 : cluster [DBG] osdmap e433: 8 total, 8 up, 8 in 2021-07-31T01:52:13.576 INFO:tasks.rados.rados.0.smithi045.stdout:1972: done (0 left) 2021-07-31T01:52:13.577 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 6 current snap is 188 2021-07-31T01:52:13.577 INFO:tasks.rados.rados.0.smithi045.stdout:1973: seq_num 616 ranges {2490368=720896} 2021-07-31T01:52:13.595 INFO:tasks.rados.rados.0.smithi045.stdout:1973: writing smithi04569269-6 from 2490368 to 3211264 tid 1 2021-07-31T01:52:13.595 INFO:tasks.rados.rados.0.smithi045.stdout:1974: copy_from oid 20 from oid 45 current snap is 188 2021-07-31T01:52:13.595 INFO:tasks.rados.rados.0.smithi045.stdout:1975: rollback oid 39 current snap is 188 2021-07-31T01:52:13.595 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 39 to 186 2021-07-31T01:52:13.596 INFO:tasks.rados.rados.0.smithi045.stdout:1976: read oid 30 snap 182 2021-07-31T01:52:13.596 INFO:tasks.rados.rados.0.smithi045.stdout:1976: expect deleted 2021-07-31T01:52:13.596 INFO:tasks.rados.rados.0.smithi045.stdout:1977: read oid 26 snap 175 2021-07-31T01:52:13.596 INFO:tasks.rados.rados.0.smithi045.stdout:1977: expect (ObjNum 529 snap 156 seq_num 529) 2021-07-31T01:52:13.596 INFO:tasks.rados.rados.0.smithi045.stdout:1978: read oid 11 snap -1 2021-07-31T01:52:13.597 INFO:tasks.rados.rados.0.smithi045.stdout:1978: expect (ObjNum 613 snap 186 seq_num 613) 2021-07-31T01:52:13.597 INFO:tasks.rados.rados.0.smithi045.stdout:1979: setattr oid 46 current snap is 188 2021-07-31T01:52:13.597 INFO:tasks.rados.rados.0.smithi045.stdout:1976: done (6 left) 2021-07-31T01:52:13.598 INFO:tasks.rados.rados.0.smithi045.stdout:1980: rmattr oid 1 current snap is 188 2021-07-31T01:52:13.598 INFO:tasks.rados.rados.0.smithi045.stdout:1981: snap_create 2021-07-31T01:52:13.602 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 1 v 1013 (ObjNum 608 snap 186 seq_num 608) dirty exists 2021-07-31T01:52:13.603 INFO:tasks.rados.rados.0.smithi045.stdout:1975: finishing rollback tid 0 to smithi04569269-39 2021-07-31T01:52:13.604 INFO:tasks.rados.rados.0.smithi045.stdout:1975: finishing rollback tid 1 to smithi04569269-39 2021-07-31T01:52:13.604 INFO:tasks.rados.rados.0.smithi045.stdout:1975: finishing rollback tid 2 to smithi04569269-39 2021-07-31T01:52:13.604 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 39 v 1079 (ObjNum 611 snap 186 seq_num 611) dirty exists 2021-07-31T01:52:13.608 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 46 v 361 (ObjNum 470 snap 136 seq_num 470) dirty exists 2021-07-31T01:52:13.611 INFO:tasks.rados.rados.0.smithi045.stdout:1977: expect (ObjNum 529 snap 156 seq_num 529) 2021-07-31T01:52:13.667 INFO:tasks.rados.rados.0.smithi045.stdout:1973: finishing write tid 1 to smithi04569269-6 2021-07-31T01:52:13.667 INFO:tasks.rados.rados.0.smithi045.stdout:1973: finishing write tid 2 to smithi04569269-6 2021-07-31T01:52:13.668 INFO:tasks.rados.rados.0.smithi045.stdout:1973: finishing write tid 3 to smithi04569269-6 2021-07-31T01:52:13.668 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 6 v 734 (ObjNum 616 snap 188 seq_num 616) dirty exists 2021-07-31T01:52:13.668 INFO:tasks.rados.rados.0.smithi045.stdout:1973: left oid 6 (ObjNum 616 snap 188 seq_num 616) 2021-07-31T01:52:13.668 INFO:tasks.rados.rados.0.smithi045.stdout:1978: expect (ObjNum 613 snap 186 seq_num 613) 2021-07-31T01:52:13.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:52:13 smithi045 conmon[61210]: level=error ts=2021-07-31T01:52:13.517Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:52:13.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:52:13 smithi045 conmon[61210]: level=error ts=2021-07-31T01:52:13.517Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:52:13.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:52:13 smithi045 conmon[61210]: level=error ts=2021-07-31T01:52:13.517Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T01:52:14.222 INFO:tasks.rados.rados.0.smithi045.stdout:1974: finishing copy_from to smithi04569269-20 2021-07-31T01:52:14.223 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 20 v 1129 (ObjNum 590 snap 183 seq_num 590) dirty exists 2021-07-31T01:52:14.223 INFO:tasks.rados.rados.0.smithi045.stdout:1974: finishing copy_from racing read to smithi04569269-20 2021-07-31T01:52:14.582 INFO:tasks.rados.rados.0.smithi045.stdout:1973: done (7 left) 2021-07-31T01:52:14.582 INFO:tasks.rados.rados.0.smithi045.stdout:1974: done (6 left) 2021-07-31T01:52:14.583 INFO:tasks.rados.rados.0.smithi045.stdout:1975: done (5 left) 2021-07-31T01:52:14.583 INFO:tasks.rados.rados.0.smithi045.stdout:1977: done (4 left) 2021-07-31T01:52:14.583 INFO:tasks.rados.rados.0.smithi045.stdout:1978: done (3 left) 2021-07-31T01:52:14.583 INFO:tasks.rados.rados.0.smithi045.stdout:1979: done (2 left) 2021-07-31T01:52:14.583 INFO:tasks.rados.rados.0.smithi045.stdout:1980: done (1 left) 2021-07-31T01:52:14.584 INFO:tasks.rados.rados.0.smithi045.stdout:1981: done (0 left) 2021-07-31T01:52:14.584 INFO:tasks.rados.rados.0.smithi045.stdout:1982: read oid 44 snap 186 2021-07-31T01:52:14.584 INFO:tasks.rados.rados.0.smithi045.stdout:1982: expect deleted 2021-07-31T01:52:14.584 INFO:tasks.rados.rados.0.smithi045.stdout:1983: delete oid 39 current snap is 189 2021-07-31T01:52:14.587 INFO:tasks.rados.rados.0.smithi045.stdout:1982: done (1 left) 2021-07-31T01:52:14.588 INFO:tasks.rados.rados.0.smithi045.stdout:1983: done (0 left) 2021-07-31T01:52:14.588 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 7 current snap is 189 2021-07-31T01:52:14.588 INFO:tasks.rados.rados.0.smithi045.stdout:1984: seq_num 618 ranges {5668864=753664} 2021-07-31T01:52:14.623 INFO:tasks.rados.rados.0.smithi045.stdout:1984: writing smithi04569269-7 from 5668864 to 6422528 tid 1 2021-07-31T01:52:14.623 INFO:tasks.rados.rados.0.smithi045.stdout:1985: snap_remove snap 188 2021-07-31T01:52:15.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:14 smithi154 conmon[34553]: cluster 2021-07-31T01:52:13.576140+0000 mon.a (mon.0) 1084 : cluster [DBG] osdmap e434: 8 total, 8 up, 8 in 2021-07-31T01:52:15.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:14 smithi154 conmon[34553]: cluster 2021-07-31T01:52:13.861374+0000 mgr.x (mgr.24542) 211 : cluster [DBG] pgmap v431: 25 pgs: 25 active+clean; 217 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 9.2 MiB/s rd, 1.7 MiB/s wr, 13 op/s 2021-07-31T01:52:15.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:14 smithi045 conmon[33655]: cluster 2021-07-31T01:52:13.576140+0000 mon.a (mon.0 2021-07-31T01:52:15.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:14 smithi045 conmon[33655]: ) 1084 : cluster [DBG] osdmap e434: 8 total, 8 up, 8 in 2021-07-31T01:52:15.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:14 smithi045 conmon[33655]: cluster 2021-07-31T01:52:13.861374+0000 mgr.x (mgr.24542) 211 : cluster [DBG] pgmap v431: 25 pgs: 25 active+clean; 217 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 9.2 MiB/s rd, 1.7 MiB/s wr, 13 op/s 2021-07-31T01:52:15.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:14 smithi045 conmon[39718]: cluster 2021-07-31T01:52:13.576140 2021-07-31T01:52:15.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:14 smithi045 conmon[39718]: +0000 mon.a (mon.0) 1084 : cluster [DBG] osdmap e434: 8 total, 8 up, 8 in 2021-07-31T01:52:15.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:14 smithi045 conmon[39718]: cluster 2021-07-31T01:52:13 2021-07-31T01:52:15.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:14 smithi045 conmon[39718]: .861374+0000 mgr.x (mgr.24542) 211 : cluster 2021-07-31T01:52:15.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:14 smithi045 conmon[39718]: [DBG] pgmap v431: 25 pgs: 25 active+clean; 217 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 9.2 MiB/s rd, 1.7 MiB/s wr, 13 op/s 2021-07-31T01:52:15.588 INFO:tasks.rados.rados.0.smithi045.stdout:1985: done (1 left) 2021-07-31T01:52:15.589 INFO:tasks.rados.rados.0.smithi045.stdout:1986: read oid 48 snap -1 2021-07-31T01:52:15.589 INFO:tasks.rados.rados.0.smithi045.stdout:1986: expect (ObjNum 525 snap 155 seq_num 525) 2021-07-31T01:52:15.589 INFO:tasks.rados.rados.0.smithi045.stdout:1987: read oid 39 snap -1 2021-07-31T01:52:15.589 INFO:tasks.rados.rados.0.smithi045.stdout:1987: expect deleted 2021-07-31T01:52:15.590 INFO:tasks.rados.rados.0.smithi045.stdout:1988: rmattr oid 20 current snap is 189 2021-07-31T01:52:15.590 INFO:tasks.rados.rados.0.smithi045.stdout:1984: finishing write tid 1 to smithi04569269-7 2021-07-31T01:52:15.590 INFO:tasks.rados.rados.0.smithi045.stdout:1984: finishing write tid 2 to smithi04569269-7 2021-07-31T01:52:15.590 INFO:tasks.rados.rados.0.smithi045.stdout:1984: finishing write tid 3 to smithi04569269-7 2021-07-31T01:52:15.591 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 7 v 979 (ObjNum 618 snap 189 seq_num 618) dirty exists 2021-07-31T01:52:15.591 INFO:tasks.rados.rados.0.smithi045.stdout:1984: left oid 7 (ObjNum 618 snap 189 seq_num 618) 2021-07-31T01:52:15.591 INFO:tasks.rados.rados.0.smithi045.stdout:1984: done (3 left) 2021-07-31T01:52:15.591 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 5 current snap is 189 2021-07-31T01:52:15.592 INFO:tasks.rados.rados.0.smithi045.stdout:1989: seq_num 619 ranges {0=770048} 2021-07-31T01:52:15.594 INFO:tasks.rados.rados.0.smithi045.stdout:1989: writing smithi04569269-5 from 0 to 770048 tid 1 2021-07-31T01:52:15.595 INFO:tasks.rados.rados.0.smithi045.stdout:1990: delete oid 50 current snap is 189 2021-07-31T01:52:15.597 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 20 v 1276 (ObjNum 590 snap 183 seq_num 590) dirty exists 2021-07-31T01:52:15.598 INFO:tasks.rados.rados.0.smithi045.stdout:1986: expect (ObjNum 525 snap 155 seq_num 525) 2021-07-31T01:52:15.651 INFO:tasks.rados.rados.0.smithi045.stdout:1989: finishing write tid 1 to smithi04569269-5 2021-07-31T01:52:15.652 INFO:tasks.rados.rados.0.smithi045.stdout:1989: finishing write tid 2 to smithi04569269-5 2021-07-31T01:52:15.652 INFO:tasks.rados.rados.0.smithi045.stdout:1989: finishing write tid 3 to smithi04569269-5 2021-07-31T01:52:15.652 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 5 v 1082 (ObjNum 619 snap 189 seq_num 619) dirty exists 2021-07-31T01:52:15.652 INFO:tasks.rados.rados.0.smithi045.stdout:1989: left oid 5 (ObjNum 619 snap 189 seq_num 619) 2021-07-31T01:52:15.653 INFO:tasks.rados.rados.0.smithi045.stdout:1986: done (4 left) 2021-07-31T01:52:15.653 INFO:tasks.rados.rados.0.smithi045.stdout:1987: done (3 left) 2021-07-31T01:52:15.653 INFO:tasks.rados.rados.0.smithi045.stdout:1988: done (2 left) 2021-07-31T01:52:15.653 INFO:tasks.rados.rados.0.smithi045.stdout:1989: done (1 left) 2021-07-31T01:52:15.653 INFO:tasks.rados.rados.0.smithi045.stdout:1990: done (0 left) 2021-07-31T01:52:15.654 INFO:tasks.rados.rados.0.smithi045.stdout:1991: setattr oid 40 current snap is 189 2021-07-31T01:52:15.655 INFO:tasks.rados.rados.0.smithi045.stdout:1992: read oid 50 snap -1 2021-07-31T01:52:15.655 INFO:tasks.rados.rados.0.smithi045.stdout:1992: expect deleted 2021-07-31T01:52:15.655 INFO:tasks.rados.rados.0.smithi045.stdout:1993: copy_from oid 6 from oid 18 current snap is 189 2021-07-31T01:52:15.656 INFO:tasks.rados.rados.0.smithi045.stdout:1994: read oid 16 snap -1 2021-07-31T01:52:15.656 INFO:tasks.rados.rados.0.smithi045.stdout:1994: expect deleted 2021-07-31T01:52:15.656 INFO:tasks.rados.rados.0.smithi045.stdout:1995: snap_create 2021-07-31T01:52:15.656 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 40 v 736 (ObjNum 596 snap 184 seq_num 596) dirty exists 2021-07-31T01:52:15.659 INFO:tasks.rados.rados.0.smithi045.stdout:1993: finishing copy_from to smithi04569269-6 2021-07-31T01:52:15.659 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 6 v 490 (ObjNum 615 snap 186 seq_num 615) dirty exists 2021-07-31T01:52:15.660 INFO:tasks.rados.rados.0.smithi045.stdout:1993: finishing copy_from racing read to smithi04569269-6 2021-07-31T01:52:16.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:15 smithi154 conmon[34553]: cluster 2021-07-31T01:52:14.580683+0000 mon.a (mon.0) 1085 : cluster [DBG] 2021-07-31T01:52:16.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:15 smithi154 conmon[34553]: osdmap e435: 8 total, 8 up, 8 in 2021-07-31T01:52:16.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:15 smithi045 conmon[33655]: cluster 2021-07-31T01:52:14.580683+0000 mon.a (mon.0 2021-07-31T01:52:16.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:15 smithi045 conmon[33655]: ) 1085 : cluster [DBG] osdmap e435: 8 total, 8 up, 8 in 2021-07-31T01:52:16.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:15 smithi045 conmon[39718]: cluster 2021-07-31T01:52:14.580683+0000 mon.a (mon.0) 1085 : cluster [DBG] osdmap e435: 8 total, 8 up, 8 in 2021-07-31T01:52:16.568 INFO:tasks.rados.rados.0.smithi045.stdout:1991: done (4 left) 2021-07-31T01:52:16.568 INFO:tasks.rados.rados.0.smithi045.stdout:1992: done (3 left) 2021-07-31T01:52:16.569 INFO:tasks.rados.rados.0.smithi045.stdout:1993: done (2 left) 2021-07-31T01:52:16.569 INFO:tasks.rados.rados.0.smithi045.stdout:1994: done (1 left) 2021-07-31T01:52:16.569 INFO:tasks.rados.rados.0.smithi045.stdout:1995: done (0 left) 2021-07-31T01:52:16.569 INFO:tasks.rados.rados.0.smithi045.stdout:1996: snap_create 2021-07-31T01:52:17.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:16 smithi154 conmon[34553]: cluster 2021-07-31T01:52:15.587391+0000 mon.a (mon.0) 1086 : cluster 2021-07-31T01:52:17.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:16 smithi154 conmon[34553]: [DBG] osdmap e436: 8 total, 8 up, 8 in 2021-07-31T01:52:17.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:16 smithi154 conmon[34553]: cluster 2021-07-31T01:52:15.861883+0000 mgr.x (mgr.24542) 212 : cluster [DBG] pgmap v434: 25 pgs: 25 active+clean; 217 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 9.2 MiB/s rd, 1.7 MiB/s wr, 13 op/s 2021-07-31T01:52:17.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:16 smithi154 conmon[34553]: cluster 2021-07-31T01 2021-07-31T01:52:17.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:16 smithi154 conmon[34553]: :52:16.567813+0000 mon.a (mon.0) 1087 : cluster [DBG] osdmap e437: 8 total, 8 up, 8 in 2021-07-31T01:52:17.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:16 smithi045 conmon[33655]: cluster 2021-07-31T01:52:15.587391+0000 mon.a (mon.0) 1086 : cluster 2021-07-31T01:52:17.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:16 smithi045 conmon[33655]: [DBG] osdmap e436: 8 total, 8 up, 8 in 2021-07-31T01:52:17.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:16 smithi045 conmon[33655]: cluster 2021-07-31T01:52:15.861883+0000 mgr.x (mgr.24542) 212 : cluster [DBG] pgmap v434: 25 pgs: 25 active+clean; 217 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 9.2 MiB/s rd, 1.7 MiB/s wr, 13 op/s 2021-07-31T01:52:17.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:16 smithi045 conmon[33655]: cluster 2021-07-31T01:52:16 2021-07-31T01:52:17.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:16 smithi045 conmon[33655]: .567813+0000 mon.a (mon.0) 2021-07-31T01:52:17.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:16 smithi045 conmon[33655]: 1087 : cluster [DBG] osdmap e437: 8 total, 8 up, 8 in 2021-07-31T01:52:17.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:16 smithi045 conmon[39718]: cluster 2021-07-31T01:52:15.587391+0000 mon.a 2021-07-31T01:52:17.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:16 smithi045 conmon[39718]: (mon.0) 1086 : cluster [DBG] osdmap e436: 8 total, 8 up, 8 in 2021-07-31T01:52:17.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:16 smithi045 conmon[39718]: cluster 2021-07-31T01:52:15.861883+0000 mgr.x 2021-07-31T01:52:17.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:16 smithi045 conmon[39718]: (mgr.24542) 212 : cluster [DBG] pgmap v434: 25 pgs: 25 active+clean; 217 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 9.2 MiB/s rd, 1.7 MiB/s wr, 13 op/s 2021-07-31T01:52:17.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:16 smithi045 conmon[39718]: cluster 2021-07-31T01:52: 2021-07-31T01:52:17.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:16 smithi045 conmon[39718]: 16.567813+0000 mon.a (mon.0 2021-07-31T01:52:17.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:16 smithi045 conmon[39718]: ) 1087 : cluster [DBG] osdmap e437: 8 total, 8 up, 8 in 2021-07-31T01:52:17.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:16 smithi045 conmon[39718]: 2021-07-31T01:52:17.574 INFO:tasks.rados.rados.0.smithi045.stdout:1996: done (0 left) 2021-07-31T01:52:17.574 INFO:tasks.rados.rados.0.smithi045.stdout:1997: delete oid 45 current snap is 191 2021-07-31T01:52:17.579 INFO:tasks.rados.rados.0.smithi045.stdout:1997: done (0 left) 2021-07-31T01:52:17.580 INFO:tasks.rados.rados.0.smithi045.stdout:1998: read oid 40 snap -1 2021-07-31T01:52:17.580 INFO:tasks.rados.rados.0.smithi045.stdout:1998: expect (ObjNum 596 snap 184 seq_num 596) 2021-07-31T01:52:17.580 INFO:tasks.rados.rados.0.smithi045.stdout:1999: rmattr oid 10 current snap is 191 2021-07-31T01:52:17.580 INFO:tasks.rados.rados.0.smithi045.stdout:1999: done (1 left) 2021-07-31T01:52:17.581 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 15 current snap is 191 2021-07-31T01:52:17.581 INFO:tasks.rados.rados.0.smithi045.stdout:2000: seq_num 622 ranges {5283840=425984,5709824=401408} 2021-07-31T01:52:17.610 INFO:tasks.rados.rados.0.smithi045.stdout:2000: writing smithi04569269-15 from 5283840 to 5709824 tid 1 2021-07-31T01:52:17.613 INFO:tasks.rados.rados.0.smithi045.stdout:2000: writing smithi04569269-15 from 5709824 to 6111232 tid 2 2021-07-31T01:52:17.613 INFO:tasks.rados.rados.0.smithi045.stdout:2001: delete oid 42 current snap is 191 2021-07-31T01:52:17.613 INFO:tasks.rados.rados.0.smithi045.stdout:1998: expect (ObjNum 596 snap 184 seq_num 596) 2021-07-31T01:52:17.627 INFO:tasks.rados.rados.0.smithi045.stdout:2000: finishing write tid 1 to smithi04569269-15 2021-07-31T01:52:17.627 INFO:tasks.rados.rados.0.smithi045.stdout:2000: finishing write tid 2 to smithi04569269-15 2021-07-31T01:52:17.627 INFO:tasks.rados.rados.0.smithi045.stdout:2000: finishing write tid 3 to smithi04569269-15 2021-07-31T01:52:17.627 INFO:tasks.rados.rados.0.smithi045.stdout:2000: finishing write tid 4 to smithi04569269-15 2021-07-31T01:52:17.628 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 15 v 500 (ObjNum 622 snap 191 seq_num 622) dirty exists 2021-07-31T01:52:17.628 INFO:tasks.rados.rados.0.smithi045.stdout:2000: left oid 15 (ObjNum 622 snap 191 seq_num 622) 2021-07-31T01:52:17.629 INFO:tasks.rados.rados.0.smithi045.stdout:1998: done (2 left) 2021-07-31T01:52:17.629 INFO:tasks.rados.rados.0.smithi045.stdout:2000: done (1 left) 2021-07-31T01:52:17.629 INFO:tasks.rados.rados.0.smithi045.stdout:2001: done (0 left) 2021-07-31T01:52:17.629 INFO:tasks.rados.rados.0.smithi045.stdout:2002: read oid 7 snap -1 2021-07-31T01:52:17.630 INFO:tasks.rados.rados.0.smithi045.stdout:2002: expect (ObjNum 618 snap 189 seq_num 618) 2021-07-31T01:52:17.630 INFO:tasks.rados.rados.0.smithi045.stdout:2003: rollback oid 5 current snap is 191 2021-07-31T01:52:17.630 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 5 to 182 2021-07-31T01:52:17.630 INFO:tasks.rados.rados.0.smithi045.stdout:2004: snap_remove snap 175 2021-07-31T01:52:18.582 INFO:tasks.rados.rados.0.smithi045.stdout:2004: done (2 left) 2021-07-31T01:52:18.582 INFO:tasks.rados.rados.0.smithi045.stdout:2005: snap_remove snap 190 2021-07-31T01:52:19.003 INFO:journalctl@ceph.osd.6.smithi154.stdout:Jul 31 01:52:18 smithi154 conmon[45576]: debug 2021-07-31T01:52:18.595+0000 7f47c10a5700 -1 osd.6 pg_epoch: 439 pg[3.as0( v 439'658 (0'0,439'658] local-lis/les=56/57 n=7 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [6,4,1]p6(0) r=0 lpr=56 crt=439'658 lcod 439'656 mlcod 439'656 active+clean+snaptrim trimq=[155~1] ps=[177~1]] removing snap head 2021-07-31T01:52:19.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:18 smithi154 conmon[34553]: cluster 2021-07-31T01:52:17.573626+0000 mon.a (mon.0) 1088 : cluster [DBG] 2021-07-31T01:52:19.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:18 smithi154 conmon[34553]: osdmap e438: 8 total, 8 up, 8 in 2021-07-31T01:52:19.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:18 smithi154 conmon[34553]: cluster 2021-07-31T01:52:17.862312+0000 mgr.x (mgr.24542) 213 : cluster [DBG] pgmap v437: 25 pgs: 25 active+clean; 217 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2021-07-31T01:52:19.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:18 smithi045 conmon[33655]: cluster 2021-07-31T01:52:17.573626+0000 mon.a (mon.0) 1088 : cluster [DBG] osdmap e438: 8 total, 8 up, 8 in 2021-07-31T01:52:19.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:18 smithi045 conmon[33655]: 2021-07-31T01:52:19.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:18 smithi045 conmon[33655]: cluster 2021-07-31T01:52:17.862312+0000 mgr.x (mgr.24542) 213 : cluster [DBG] pgmap v437: 25 pgs: 25 active+clean; 217 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2021-07-31T01:52:19.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:18 smithi045 conmon[39718]: cluster 2021-07-31T01:52:17.573626+0000 2021-07-31T01:52:19.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:18 smithi045 conmon[39718]: mon.a (mon.0) 1088 : cluster [DBG] osdmap e438: 8 total, 8 up, 8 in 2021-07-31T01:52:19.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:18 smithi045 conmon[39718]: cluster 2021-07-31T01:52:17.862312+0000 mgr.x (mgr.24542) 213 2021-07-31T01:52:19.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:18 smithi045 conmon[39718]: : cluster [DBG] pgmap v437: 25 pgs: 25 active+clean; 217 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2021-07-31T01:52:19.581 INFO:tasks.rados.rados.0.smithi045.stdout:2005: done (2 left) 2021-07-31T01:52:19.582 INFO:tasks.rados.rados.0.smithi045.stdout:2006: copy_from oid 26 from oid 19 current snap is 191 2021-07-31T01:52:19.582 INFO:tasks.rados.rados.0.smithi045.stdout:2003: finishing rollback tid 0 to smithi04569269-5 2021-07-31T01:52:19.582 INFO:tasks.rados.rados.0.smithi045.stdout:2003: finishing rollback tid 1 to smithi04569269-5 2021-07-31T01:52:19.583 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 5 v 1084 (ObjNum 0 snap 21873 seq_num 3189000960) dirty dne 2021-07-31T01:52:19.583 INFO:tasks.rados.rados.0.smithi045.stdout:2002: expect (ObjNum 618 snap 189 seq_num 618) 2021-07-31T01:52:19.930 INFO:tasks.rados.rados.0.smithi045.stdout:2006: finishing copy_from to smithi04569269-26 2021-07-31T01:52:19.930 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 26 v 275 (ObjNum 172 snap 41 seq_num 172) dirty exists 2021-07-31T01:52:19.931 INFO:tasks.rados.rados.0.smithi045.stdout:2006: finishing copy_from racing read to smithi04569269-26 2021-07-31T01:52:19.931 INFO:tasks.rados.rados.0.smithi045.stdout:2002: done (2 left) 2021-07-31T01:52:19.931 INFO:tasks.rados.rados.0.smithi045.stdout:2003: done (1 left) 2021-07-31T01:52:19.931 INFO:tasks.rados.rados.0.smithi045.stdout:2006: done (0 left) 2021-07-31T01:52:19.931 INFO:tasks.rados.rados.0.smithi045.stdout:2007: rollback oid 29 current snap is 191 2021-07-31T01:52:19.932 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 29 to 186 2021-07-31T01:52:19.932 INFO:tasks.rados.rados.0.smithi045.stdout:2008: delete oid 23 current snap is 191 2021-07-31T01:52:19.933 INFO:tasks.rados.rados.0.smithi045.stdout:2008: done (1 left) 2021-07-31T01:52:19.933 INFO:tasks.rados.rados.0.smithi045.stdout:2009: rollback oid 20 current snap is 191 2021-07-31T01:52:19.933 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 20 to 182 2021-07-31T01:52:19.934 INFO:tasks.rados.rados.0.smithi045.stdout:2010: copy_from oid 38 from oid 28 current snap is 191 2021-07-31T01:52:19.934 INFO:tasks.rados.rados.0.smithi045.stdout:2011: delete oid 34 current snap is 191 2021-07-31T01:52:19.935 INFO:tasks.rados.rados.0.smithi045.stdout:2007: finishing rollback tid 0 to smithi04569269-29 2021-07-31T01:52:19.935 INFO:tasks.rados.rados.0.smithi045.stdout:2007: finishing rollback tid 1 to smithi04569269-29 2021-07-31T01:52:19.935 INFO:tasks.rados.rados.0.smithi045.stdout:2007: finishing rollback tid 2 to smithi04569269-29 2021-07-31T01:52:19.936 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 29 v 929 (ObjNum 539 snap 162 seq_num 539) dirty exists 2021-07-31T01:52:19.936 INFO:tasks.rados.rados.0.smithi045.stdout:2007: done (3 left) 2021-07-31T01:52:19.936 INFO:tasks.rados.rados.0.smithi045.stdout:2011: done (2 left) 2021-07-31T01:52:19.936 INFO:tasks.rados.rados.0.smithi045.stdout:2012: setattr oid 44 current snap is 191 2021-07-31T01:52:19.938 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 26 current snap is 191 2021-07-31T01:52:19.938 INFO:tasks.rados.rados.0.smithi045.stdout:2013: seq_num 626 ranges {483328=491520,974848=401408} 2021-07-31T01:52:19.943 INFO:tasks.rados.rados.0.smithi045.stdout:2013: writing smithi04569269-26 from 483328 to 974848 tid 1 2021-07-31T01:52:19.946 INFO:tasks.rados.rados.0.smithi045.stdout:2013: writing smithi04569269-26 from 974848 to 1376256 tid 2 2021-07-31T01:52:19.946 INFO:tasks.rados.rados.0.smithi045.stdout:2014: read oid 49 snap -1 2021-07-31T01:52:19.946 INFO:tasks.rados.rados.0.smithi045.stdout:2014: expect (ObjNum 541 snap 162 seq_num 541) 2021-07-31T01:52:19.947 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 13 current snap is 191 2021-07-31T01:52:19.947 INFO:tasks.rados.rados.0.smithi045.stdout:2009: finishing rollback tid 0 to smithi04569269-20 2021-07-31T01:52:19.947 INFO:tasks.rados.rados.0.smithi045.stdout:2009: finishing rollback tid 1 to smithi04569269-20 2021-07-31T01:52:19.947 INFO:tasks.rados.rados.0.smithi045.stdout:2009: finishing rollback tid 2 to smithi04569269-20 2021-07-31T01:52:19.948 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 20 v 1290 (ObjNum 556 snap 166 seq_num 556) dirty exists 2021-07-31T01:52:19.948 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 44 v 1047 (ObjNum 288 snap 0 seq_num 65) dirty exists 2021-07-31T01:52:19.948 INFO:tasks.rados.rados.0.smithi045.stdout:2010: finishing copy_from to smithi04569269-38 2021-07-31T01:52:19.949 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 38 v 1134 (ObjNum 612 snap 186 seq_num 612) dirty exists 2021-07-31T01:52:19.949 INFO:tasks.rados.rados.0.smithi045.stdout:2010: finishing copy_from racing read to smithi04569269-38 2021-07-31T01:52:19.949 INFO:tasks.rados.rados.0.smithi045.stdout:2015: seq_num 627 ranges {11927552=507904,12435456=401408} 2021-07-31T01:52:20.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:19 smithi154 conmon[34553]: cluster 2021-07-31T01:52:18.579622+0000 mon.a (mon.0) 1089 : cluster 2021-07-31T01:52:20.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:19 smithi154 conmon[34553]: [DBG] osdmap e439: 8 total, 8 up, 8 in 2021-07-31T01:52:20.013 INFO:tasks.rados.rados.0.smithi045.stdout:2015: writing smithi04569269-13 from 11927552 to 12435456 tid 1 2021-07-31T01:52:20.016 INFO:tasks.rados.rados.0.smithi045.stdout:2015: writing smithi04569269-13 from 12435456 to 12836864 tid 2 2021-07-31T01:52:20.016 INFO:tasks.rados.rados.0.smithi045.stdout:2009: done (5 left) 2021-07-31T01:52:20.016 INFO:tasks.rados.rados.0.smithi045.stdout:2010: done (4 left) 2021-07-31T01:52:20.017 INFO:tasks.rados.rados.0.smithi045.stdout:2012: done (3 left) 2021-07-31T01:52:20.017 INFO:tasks.rados.rados.0.smithi045.stdout:2016: snap_create 2021-07-31T01:52:20.017 INFO:tasks.rados.rados.0.smithi045.stdout:2013: finishing write tid 1 to smithi04569269-26 2021-07-31T01:52:20.017 INFO:tasks.rados.rados.0.smithi045.stdout:2013: finishing write tid 2 to smithi04569269-26 2021-07-31T01:52:20.018 INFO:tasks.rados.rados.0.smithi045.stdout:2013: finishing write tid 3 to smithi04569269-26 2021-07-31T01:52:20.018 INFO:tasks.rados.rados.0.smithi045.stdout:2013: finishing write tid 4 to smithi04569269-26 2021-07-31T01:52:20.018 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 26 v 1033 (ObjNum 626 snap 191 seq_num 626) dirty exists 2021-07-31T01:52:20.018 INFO:tasks.rados.rados.0.smithi045.stdout:2013: left oid 26 (ObjNum 626 snap 191 seq_num 626) 2021-07-31T01:52:20.018 INFO:tasks.rados.rados.0.smithi045.stdout:2014: expect (ObjNum 541 snap 162 seq_num 541) 2021-07-31T01:52:20.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:19 smithi045 conmon[33655]: cluster 2021-07-31T01:52:18.579622+0000 2021-07-31T01:52:20.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:19 smithi045 conmon[33655]: mon.a (mon.0) 1089 : cluster [DBG] osdmap e439: 8 total, 8 up, 8 in 2021-07-31T01:52:20.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:19 smithi045 conmon[39718]: cluster 2021-07-31T01:52:18.579622+0000 2021-07-31T01:52:20.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:19 smithi045 conmon[39718]: mon.a (mon.0) 1089 : cluster [DBG] osdmap e439: 8 total, 8 up, 8 in 2021-07-31T01:52:20.258 INFO:tasks.rados.rados.0.smithi045.stdout:2015: finishing write tid 1 to smithi04569269-13 2021-07-31T01:52:20.259 INFO:tasks.rados.rados.0.smithi045.stdout:2015: finishing write tid 2 to smithi04569269-13 2021-07-31T01:52:20.259 INFO:tasks.rados.rados.0.smithi045.stdout:2015: finishing write tid 3 to smithi04569269-13 2021-07-31T01:52:20.259 INFO:tasks.rados.rados.0.smithi045.stdout:2015: finishing write tid 4 to smithi04569269-13 2021-07-31T01:52:20.259 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 13 v 747 (ObjNum 627 snap 191 seq_num 627) dirty exists 2021-07-31T01:52:20.260 INFO:tasks.rados.rados.0.smithi045.stdout:2015: left oid 13 (ObjNum 627 snap 191 seq_num 627) 2021-07-31T01:52:20.591 INFO:tasks.rados.rados.0.smithi045.stdout:2013: done (3 left) 2021-07-31T01:52:20.591 INFO:tasks.rados.rados.0.smithi045.stdout:2014: done (2 left) 2021-07-31T01:52:20.591 INFO:tasks.rados.rados.0.smithi045.stdout:2015: done (1 left) 2021-07-31T01:52:20.592 INFO:tasks.rados.rados.0.smithi045.stdout:2016: done (0 left) 2021-07-31T01:52:20.592 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 6 current snap is 192 2021-07-31T01:52:20.592 INFO:tasks.rados.rados.0.smithi045.stdout:2017: seq_num 628 ranges {704512=524288,1228800=401408} 2021-07-31T01:52:20.597 INFO:tasks.rados.rados.0.smithi045.stdout:2017: writing smithi04569269-6 from 704512 to 1228800 tid 1 2021-07-31T01:52:20.600 INFO:tasks.rados.rados.0.smithi045.stdout:2017: writing smithi04569269-6 from 1228800 to 1630208 tid 2 2021-07-31T01:52:20.600 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 36 current snap is 192 2021-07-31T01:52:20.600 INFO:tasks.rados.rados.0.smithi045.stdout:2018: seq_num 629 ranges {0=540672,540672=401408} 2021-07-31T01:52:20.602 INFO:tasks.rados.rados.0.smithi045.stdout:2018: writing smithi04569269-36 from 0 to 540672 tid 1 2021-07-31T01:52:20.605 INFO:tasks.rados.rados.0.smithi045.stdout:2018: writing smithi04569269-36 from 540672 to 942080 tid 2 2021-07-31T01:52:20.605 INFO:tasks.rados.rados.0.smithi045.stdout:2019: snap_create 2021-07-31T01:52:20.605 INFO:tasks.rados.rados.0.smithi045.stdout:2017: finishing write tid 1 to smithi04569269-6 2021-07-31T01:52:20.605 INFO:tasks.rados.rados.0.smithi045.stdout:2017: finishing write tid 2 to smithi04569269-6 2021-07-31T01:52:20.606 INFO:tasks.rados.rados.0.smithi045.stdout:2017: finishing write tid 3 to smithi04569269-6 2021-07-31T01:52:20.606 INFO:tasks.rados.rados.0.smithi045.stdout:2017: finishing write tid 4 to smithi04569269-6 2021-07-31T01:52:20.606 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 6 v 751 (ObjNum 628 snap 192 seq_num 628) dirty exists 2021-07-31T01:52:20.607 INFO:tasks.rados.rados.0.smithi045.stdout:2017: left oid 6 (ObjNum 628 snap 192 seq_num 628) 2021-07-31T01:52:20.608 INFO:tasks.rados.rados.0.smithi045.stdout:2018: finishing write tid 1 to smithi04569269-36 2021-07-31T01:52:20.608 INFO:tasks.rados.rados.0.smithi045.stdout:2018: finishing write tid 2 to smithi04569269-36 2021-07-31T01:52:20.608 INFO:tasks.rados.rados.0.smithi045.stdout:2018: finishing write tid 3 to smithi04569269-36 2021-07-31T01:52:20.610 INFO:tasks.rados.rados.0.smithi045.stdout:2018: finishing write tid 4 to smithi04569269-36 2021-07-31T01:52:20.610 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 36 v 1293 (ObjNum 629 snap 192 seq_num 629) dirty exists 2021-07-31T01:52:20.611 INFO:tasks.rados.rados.0.smithi045.stdout:2018: left oid 36 (ObjNum 629 snap 192 seq_num 629) 2021-07-31T01:52:21.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:20 smithi154 conmon[34553]: cluster 2021-07-31T01:52:19.580706+0000 mon.a (mon.0) 1090 : cluster 2021-07-31T01:52:21.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:20 smithi154 conmon[34553]: [DBG] osdmap e440: 8 total, 8 up, 8 in 2021-07-31T01:52:21.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:20 smithi154 conmon[34553]: cluster 2021-07-31T01:52:19.862752+0000 mgr.x (mgr.24542) 214 : cluster [DBG] pgmap v440: 25 pgs: 25 active+clean; 173 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 6.2 MiB/s rd, 746 KiB/s wr, 22 op/s 2021-07-31T01:52:21.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:20 smithi045 conmon[33655]: cluster 2021-07-31T01:52:19.580706+0000 mon.a (mon.0) 1090 : cluster 2021-07-31T01:52:21.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:20 smithi045 conmon[33655]: [DBG] osdmap e440: 8 total, 8 up, 8 in 2021-07-31T01:52:21.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:20 smithi045 conmon[33655]: cluster 2021-07-31T01:52:19.862752+0000 mgr.x (mgr.24542) 214 : cluster [DBG] pgmap v440: 25 pgs: 25 active+clean; 173 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 6.2 MiB/s rd, 746 KiB/s wr, 22 op/s 2021-07-31T01:52:21.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:20 smithi045 conmon[39718]: cluster 2021-07-31T01:52:19.580706+0000 mon.a (mon 2021-07-31T01:52:21.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:20 smithi045 conmon[39718]: .0) 1090 : cluster [DBG] osdmap e440: 8 total, 8 up, 8 in 2021-07-31T01:52:21.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:20 smithi045 conmon[39718]: cluster 2021-07-31T01:52:19.862752 2021-07-31T01:52:21.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:20 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 214 : cluster [DBG] pgmap v440: 25 pgs: 25 active+clean; 173 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 6.2 MiB/s rd, 746 KiB/s wr, 22 op/s 2021-07-31T01:52:21.570 INFO:tasks.rados.rados.0.smithi045.stdout:2017: done (2 left) 2021-07-31T01:52:21.570 INFO:tasks.rados.rados.0.smithi045.stdout:2018: done (1 left) 2021-07-31T01:52:21.571 INFO:tasks.rados.rados.0.smithi045.stdout:2019: done (0 left) 2021-07-31T01:52:21.571 INFO:tasks.rados.rados.0.smithi045.stdout:2020: rmattr oid 11 current snap is 193 2021-07-31T01:52:21.571 INFO:tasks.rados.rados.0.smithi045.stdout:2021: read oid 20 snap -1 2021-07-31T01:52:21.571 INFO:tasks.rados.rados.0.smithi045.stdout:2021: expect (ObjNum 556 snap 166 seq_num 556) 2021-07-31T01:52:21.572 INFO:tasks.rados.rados.0.smithi045.stdout:2022: copy_from oid 23 from oid 37 current snap is 193 2021-07-31T01:52:21.572 INFO:tasks.rados.rados.0.smithi045.stdout:2023: snap_create 2021-07-31T01:52:21.579 INFO:tasks.rados.rados.0.smithi045.stdout:2022: finishing copy_from to smithi04569269-23 2021-07-31T01:52:21.579 INFO:tasks.rados.rados.0.smithi045.stdout:2022: got expected ENOENT (src dne) 2021-07-31T01:52:21.579 INFO:tasks.rados.rados.0.smithi045.stdout:2022: finishing copy_from racing read to smithi04569269-23 2021-07-31T01:52:21.580 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 11 v 1135 (ObjNum 613 snap 186 seq_num 613) dirty exists 2021-07-31T01:52:21.584 INFO:tasks.rados.rados.0.smithi045.stdout:2021: expect (ObjNum 556 snap 166 seq_num 556) 2021-07-31T01:52:22.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:21 smithi154 conmon[34553]: cluster 2021-07-31T01:52:20.590242+0000 mon.a (mon.0) 1091 : cluster [DBG] osdmap e441: 8 total, 8 up, 8 in 2021-07-31T01:52:22.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:21 smithi154 conmon[34553]: cluster 2021-07-31T01:52:21.569015+0000 2021-07-31T01:52:22.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:21 smithi154 conmon[34553]: mon.a (mon.0) 1092 : cluster [DBG] osdmap e442: 8 total, 8 up, 8 in 2021-07-31T01:52:22.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:21 smithi045 conmon[33655]: cluster 2021-07-31T01:52:20.590242+0000 mon.a (mon.0 2021-07-31T01:52:22.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:21 smithi045 conmon[33655]: ) 1091 : cluster [DBG] osdmap e441: 8 total, 8 up, 8 in 2021-07-31T01:52:22.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:21 smithi045 conmon[33655]: cluster 2021-07-31T01:52:21.569015+0000 mon.a (mon.0) 1092 : cluster [DBG] osdmap e442: 8 total, 8 up, 8 in 2021-07-31T01:52:22.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:21 smithi045 conmon[39718]: cluster 2021-07-31T01:52 2021-07-31T01:52:22.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:21 smithi045 conmon[39718]: :20.590242+0000 mon.a (mon.0) 1091 : cluster [DBG] osdmap e441: 8 total, 8 up, 8 in 2021-07-31T01:52:22.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:21 smithi045 conmon[39718]: cluster 2021- 2021-07-31T01:52:22.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:21 smithi045 conmon[39718]: 07-31T01:52:21.569015+0000 mon.a (mon.0) 1092 : cluster [DBG] osdmap e442: 8 total, 8 up, 8 in 2021-07-31T01:52:22.576 INFO:tasks.rados.rados.0.smithi045.stdout:2020: done (3 left) 2021-07-31T01:52:22.577 INFO:tasks.rados.rados.0.smithi045.stdout:2021: done (2 left) 2021-07-31T01:52:22.577 INFO:tasks.rados.rados.0.smithi045.stdout:2022: done (1 left) 2021-07-31T01:52:22.577 INFO:tasks.rados.rados.0.smithi045.stdout:2023: done (0 left) 2021-07-31T01:52:22.577 INFO:tasks.rados.rados.0.smithi045.stdout:2024: rollback oid 30 current snap is 194 2021-07-31T01:52:22.577 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 30 to 192 2021-07-31T01:52:22.578 INFO:tasks.rados.rados.0.smithi045.stdout:2025: rollback oid 36 current snap is 194 2021-07-31T01:52:22.578 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 36 to 193 2021-07-31T01:52:22.578 INFO:tasks.rados.rados.0.smithi045.stdout:2026: read oid 1 snap -1 2021-07-31T01:52:22.579 INFO:tasks.rados.rados.0.smithi045.stdout:2026: expect (ObjNum 608 snap 186 seq_num 608) 2021-07-31T01:52:22.579 INFO:tasks.rados.rados.0.smithi045.stdout:2027: delete oid 4 current snap is 194 2021-07-31T01:52:22.581 INFO:tasks.rados.rados.0.smithi045.stdout:2027: done (3 left) 2021-07-31T01:52:22.581 INFO:tasks.rados.rados.0.smithi045.stdout:2028: snap_remove snap 191 2021-07-31T01:52:23.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:22 smithi154 conmon[34553]: cluster 2021-07-31T01:52:21.863156+0000 mgr.x (mgr.24542) 215 : cluster [DBG] pgmap v443: 25 pgs: 25 active+clean; 173 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 6.2 MiB/s rd, 746 KiB/s wr, 22 op/s 2021-07-31T01:52:23.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:22 smithi154 conmon[34553]: cluster 2021-07-31T01:52:22.575320+0000 2021-07-31T01:52:23.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:22 smithi154 conmon[34553]: mon.a (mon.0) 1093 : cluster [DBG] osdmap e443: 8 total, 8 up, 8 in 2021-07-31T01:52:23.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:22 smithi045 conmon[39718]: cluster 2021-07-31T01:52:21.863156 2021-07-31T01:52:23.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:22 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 215 : cluster [DBG] pgmap v443: 25 pgs: 25 active+clean; 173 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 6.2 MiB/s rd, 746 KiB/s wr, 22 op/s 2021-07-31T01:52:23.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:22 smithi045 conmon[39718]: cluster 2021-07-31T01:52 2021-07-31T01:52:23.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:22 smithi045 conmon[39718]: :22.575320+0000 mon.a (mon.0) 1093 : cluster [DBG] osdmap e443: 8 total, 8 up, 8 in 2021-07-31T01:52:23.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:22 smithi045 conmon[33655]: cluster 2021-07-31T01:52: 2021-07-31T01:52:23.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:22 smithi045 conmon[33655]: 21.863156+0000 mgr.x (mgr.24542) 215 : cluster [DBG] pgmap v443: 25 pgs: 25 active+clean; 173 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 6.2 MiB/s rd, 746 KiB/s wr, 22 op/s 2021-07-31T01:52:23.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:22 smithi045 conmon[33655]: cluster 2021-07-31T01:52:22.575320+0000 mon.a (mon.0) 1093 : cluster [DBG] 2021-07-31T01:52:23.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:22 smithi045 conmon[33655]: osdmap e443: 8 total, 8 up, 8 in 2021-07-31T01:52:23.578 INFO:tasks.rados.rados.0.smithi045.stdout:2028: done (3 left) 2021-07-31T01:52:23.579 INFO:tasks.rados.rados.0.smithi045.stdout:2029: delete oid 21 current snap is 194 2021-07-31T01:52:23.579 INFO:tasks.rados.rados.0.smithi045.stdout:2024: finishing rollback tid 0 to smithi04569269-30 2021-07-31T01:52:23.580 INFO:tasks.rados.rados.0.smithi045.stdout:2024: finishing rollback tid 1 to smithi04569269-30 2021-07-31T01:52:23.580 INFO:tasks.rados.rados.0.smithi045.stdout:2024: finishing rollback tid 2 to smithi04569269-30 2021-07-31T01:52:23.580 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 30 v 527 (ObjNum -1105027632 snap 21873 seq_num 3189932112) dirty exists 2021-07-31T01:52:23.581 INFO:tasks.rados.rados.0.smithi045.stdout:2025: finishing rollback tid 0 to smithi04569269-36 2021-07-31T01:52:23.581 INFO:tasks.rados.rados.0.smithi045.stdout:2025: finishing rollback tid 1 to smithi04569269-36 2021-07-31T01:52:23.581 INFO:tasks.rados.rados.0.smithi045.stdout:2025: finishing rollback tid 2 to smithi04569269-36 2021-07-31T01:52:23.581 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 36 v 1297 (ObjNum 629 snap 192 seq_num 629) dirty exists 2021-07-31T01:52:23.582 INFO:tasks.rados.rados.0.smithi045.stdout:2026: expect (ObjNum 608 snap 186 seq_num 608) 2021-07-31T01:52:23.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:52:23 smithi045 conmon[61210]: level=error ts=2021-07-31T01:52:23.517Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:52:23.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:52:23 smithi045 conmon[61210]: level=error ts=2021-07-31T01:52:23.517Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:52:23.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:52:23 smithi045 conmon[61210]: level=error ts=2021-07-31T01:52:23.517Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T01:52:23.855 INFO:tasks.rados.rados.0.smithi045.stdout:2024: done (3 left) 2021-07-31T01:52:23.856 INFO:tasks.rados.rados.0.smithi045.stdout:2025: done (2 left) 2021-07-31T01:52:23.856 INFO:tasks.rados.rados.0.smithi045.stdout:2026: done (1 left) 2021-07-31T01:52:23.856 INFO:tasks.rados.rados.0.smithi045.stdout:2029: done (0 left) 2021-07-31T01:52:23.856 INFO:tasks.rados.rados.0.smithi045.stdout:2030: read oid 46 snap -1 2021-07-31T01:52:23.856 INFO:tasks.rados.rados.0.smithi045.stdout:2030: expect (ObjNum 470 snap 136 seq_num 470) 2021-07-31T01:52:23.857 INFO:tasks.rados.rados.0.smithi045.stdout:2031: copy_from oid 26 from oid 19 current snap is 194 2021-07-31T01:52:23.857 INFO:tasks.rados.rados.0.smithi045.stdout:2032: read oid 33 snap -1 2021-07-31T01:52:23.857 INFO:tasks.rados.rados.0.smithi045.stdout:2032: expect (ObjNum 399 snap 121 seq_num 399) 2021-07-31T01:52:23.857 INFO:tasks.rados.rados.0.smithi045.stdout:2033: rollback oid 31 current snap is 194 2021-07-31T01:52:23.858 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 31 to 193 2021-07-31T01:52:23.858 INFO:tasks.rados.rados.0.smithi045.stdout:2034: snap_create 2021-07-31T01:52:23.858 INFO:tasks.rados.rados.0.smithi045.stdout:2033: finishing rollback tid 1 to smithi04569269-31 2021-07-31T01:52:23.859 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 31 v 1218 (ObjNum -1105489568 snap 21873 seq_num 842610229) dirty dne 2021-07-31T01:52:23.864 INFO:tasks.rados.rados.0.smithi045.stdout:2031: finishing copy_from to smithi04569269-26 2021-07-31T01:52:23.864 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 26 v 275 (ObjNum 172 snap 41 seq_num 172) dirty exists 2021-07-31T01:52:23.865 INFO:tasks.rados.rados.0.smithi045.stdout:2031: finishing copy_from racing read to smithi04569269-26 2021-07-31T01:52:23.885 INFO:tasks.rados.rados.0.smithi045.stdout:2030: expect (ObjNum 470 snap 136 seq_num 470) 2021-07-31T01:52:24.018 INFO:tasks.rados.rados.0.smithi045.stdout:2032: expect (ObjNum 399 snap 121 seq_num 399) 2021-07-31T01:52:24.582 INFO:tasks.rados.rados.0.smithi045.stdout:2030: done (4 left) 2021-07-31T01:52:24.582 INFO:tasks.rados.rados.0.smithi045.stdout:2031: done (3 left) 2021-07-31T01:52:24.583 INFO:tasks.rados.rados.0.smithi045.stdout:2032: done (2 left) 2021-07-31T01:52:24.583 INFO:tasks.rados.rados.0.smithi045.stdout:2033: done (1 left) 2021-07-31T01:52:24.583 INFO:tasks.rados.rados.0.smithi045.stdout:2034: done (0 left) 2021-07-31T01:52:24.583 INFO:tasks.rados.rados.0.smithi045.stdout:2035: snap_create 2021-07-31T01:52:25.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:24 smithi154 conmon[34553]: cluster 2021-07-31T01:52:23.577202+0000 mon.a (mon.0) 2021-07-31T01:52:25.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:24 smithi154 conmon[34553]: 1094 : cluster [DBG] osdmap e444: 8 total, 8 up, 8 in 2021-07-31T01:52:25.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:24 smithi154 conmon[34553]: cluster 2021-07-31T01:52:23.863714+0000 mgr.x (mgr.24542) 216 : cluster [DBG] pgmap v446: 25 pgs: 25 active+clean; 181 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 8.8 MiB/s rd, 1.3 MiB/s wr, 17 op/s 2021-07-31T01:52:25.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:24 smithi045 conmon[39718]: cluster 2021-07-31T01:52:23.577202+0000 mon.a 2021-07-31T01:52:25.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:24 smithi045 conmon[39718]: (mon.0) 1094 : cluster [DBG] osdmap e444: 8 total, 8 up, 8 in 2021-07-31T01:52:25.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:24 smithi045 conmon[39718]: cluster 2021-07-31T01:52:23.863714+0000 mgr.x 2021-07-31T01:52:25.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:24 smithi045 conmon[39718]: (mgr.24542) 216 : cluster [DBG] pgmap v446: 25 pgs: 25 active+clean; 181 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 8.8 MiB/s rd, 1.3 MiB/s wr, 17 op/s 2021-07-31T01:52:25.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:24 smithi045 conmon[33655]: cluster 2021-07-31T01:52:23.577202+0000 mon.a (mon. 2021-07-31T01:52:25.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:24 smithi045 conmon[33655]: 0) 1094 : cluster [DBG] osdmap e444: 8 total, 8 up, 8 in 2021-07-31T01:52:25.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:24 smithi045 conmon[33655]: cluster 2021-07-31T01:52:23.863714+0000 mgr.x (mgr.24542 2021-07-31T01:52:25.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:24 smithi045 conmon[33655]: ) 216 : cluster [DBG] pgmap v446: 25 pgs: 25 active+clean; 181 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 8.8 MiB/s rd, 1.3 MiB/s wr, 17 op/s 2021-07-31T01:52:25.584 INFO:tasks.rados.rados.0.smithi045.stdout:2035: done (0 left) 2021-07-31T01:52:25.584 INFO:tasks.rados.rados.0.smithi045.stdout:2036: snap_create 2021-07-31T01:52:26.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:25 smithi154 conmon[34553]: cluster 2021-07-31T01:52:24.581472+0000 mon.a (mon.0) 1095 : cluster [DBG] osdmap e445: 8 total, 8 up, 8 in 2021-07-31T01:52:26.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:25 smithi045 conmon[33655]: cluster 2021-07-31T01:52:24.581472+0000 mon.a (mon.0) 1095 : 2021-07-31T01:52:26.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:25 smithi045 conmon[33655]: cluster [DBG] osdmap e445: 8 total, 8 up, 8 in 2021-07-31T01:52:26.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:25 smithi045 conmon[39718]: cluster 2021-07-31T01:52:24.581472+0000 2021-07-31T01:52:26.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:25 smithi045 conmon[39718]: mon.a (mon.0) 1095 : cluster [DBG] osdmap e445: 8 total, 8 up, 8 in 2021-07-31T01:52:26.571 INFO:tasks.rados.rados.0.smithi045.stdout:2036: done (0 left) 2021-07-31T01:52:26.572 INFO:tasks.rados.rados.0.smithi045.stdout:2037: read oid 21 snap -1 2021-07-31T01:52:26.572 INFO:tasks.rados.rados.0.smithi045.stdout:2037: expect deleted 2021-07-31T01:52:26.572 INFO:tasks.rados.rados.0.smithi045.stdout:2038: delete oid 10 current snap is 197 2021-07-31T01:52:26.577 INFO:tasks.rados.rados.0.smithi045.stdout:2037: done (1 left) 2021-07-31T01:52:26.578 INFO:tasks.rados.rados.0.smithi045.stdout:2038: done (0 left) 2021-07-31T01:52:26.578 INFO:tasks.rados.rados.0.smithi045.stdout:2039: rollback oid 38 current snap is 197 2021-07-31T01:52:26.578 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 38 to 189 2021-07-31T01:52:26.578 INFO:tasks.rados.rados.0.smithi045.stdout:2040: snap_remove snap 194 2021-07-31T01:52:27.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:26 smithi154 conmon[34553]: cluster 2021-07-31T01:52:25.583085+0000 mon.a (mon.0) 1096 : cluster 2021-07-31T01:52:27.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:26 smithi154 conmon[34553]: [DBG] osdmap e446: 8 total, 8 up, 8 in 2021-07-31T01:52:27.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:26 smithi154 conmon[34553]: cluster 2021-07-31T01:52:25.864107+0000 mgr.x (mgr.24542) 217 : cluster [DBG] pgmap v449: 25 pgs: 25 active+clean; 181 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 8.8 MiB/s rd, 1.3 MiB/s wr, 17 op/s 2021-07-31T01:52:27.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:26 smithi154 conmon[34553]: cluster 2021-07-31T01:52:26.570470+0000 mon.a (mon.0) 1097 : cluster [DBG] osdmap e447: 8 total, 8 up, 8 in 2021-07-31T01:52:27.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:26 smithi154 conmon[34553]: 2021-07-31T01:52:27.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:26 smithi045 conmon[39718]: cluster 2021-07-31T01:52:25.583085 2021-07-31T01:52:27.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:26 smithi045 conmon[39718]: +0000 mon.a (mon.0) 1096 : cluster [DBG] osdmap e446: 8 total, 8 up, 8 in 2021-07-31T01:52:27.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:26 smithi045 conmon[39718]: cluster 2021-07-31T01:52:25 2021-07-31T01:52:27.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:26 smithi045 conmon[39718]: .864107+0000 mgr.x (mgr.24542) 217 : cluster [DBG] 2021-07-31T01:52:27.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:26 smithi045 conmon[39718]: pgmap v449: 25 pgs: 25 active+clean; 181 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 8.8 MiB/s rd, 1.3 MiB/s wr, 17 op/s 2021-07-31T01:52:27.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:26 smithi045 conmon[39718]: cluster 2021-07-31T01:52: 2021-07-31T01:52:27.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:26 smithi045 conmon[39718]: 26.570470+0000 mon.a (mon.0) 1097 : 2021-07-31T01:52:27.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:26 smithi045 conmon[39718]: cluster [DBG] osdmap e447: 8 total, 8 up, 8 in 2021-07-31T01:52:27.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:26 smithi045 conmon[33655]: cluster 2021-07-31T01:52:25.583085+0000 mon.a ( 2021-07-31T01:52:27.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:26 smithi045 conmon[33655]: mon.0) 1096 : cluster [DBG] osdmap e446: 8 total, 8 up, 8 in 2021-07-31T01:52:27.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:26 smithi045 conmon[33655]: cluster 2021-07-31T01:52:25.864107+0000 mgr.x (mgr.24542) 217 : cluster [DBG] pgmap v449: 25 pgs: 25 active+clean; 181 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 8.8 MiB/s rd, 1.3 MiB/s wr, 17 op/s 2021-07-31T01:52:27.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:26 smithi045 conmon[33655]: cluster 2021-07-31T01:52:27.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:26 smithi045 conmon[33655]: 2021-07-31T01:52:26.570470 2021-07-31T01:52:27.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:26 smithi045 conmon[33655]: +0000 mon.a (mon.0) 2021-07-31T01:52:27.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:26 smithi045 conmon[33655]: 1097 : cluster [DBG] osdmap e447: 8 total, 8 up, 8 in 2021-07-31T01:52:27.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:26 smithi045 conmon[33655]: 2021-07-31T01:52:27.578 INFO:tasks.rados.rados.0.smithi045.stdout:2040: done (1 left) 2021-07-31T01:52:27.578 INFO:tasks.rados.rados.0.smithi045.stdout:2041: snap_create 2021-07-31T01:52:27.579 INFO:tasks.rados.rados.0.smithi045.stdout:2039: finishing rollback tid 0 to smithi04569269-38 2021-07-31T01:52:27.579 INFO:tasks.rados.rados.0.smithi045.stdout:2039: finishing rollback tid 1 to smithi04569269-38 2021-07-31T01:52:27.580 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 38 v 1137 (ObjNum 1 snap 32731 seq_num 3189023680) dirty dne 2021-07-31T01:52:28.583 INFO:tasks.rados.rados.0.smithi045.stdout:2039: done (1 left) 2021-07-31T01:52:28.583 INFO:tasks.rados.rados.0.smithi045.stdout:2041: done (0 left) 2021-07-31T01:52:28.584 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 40 current snap is 198 2021-07-31T01:52:28.584 INFO:tasks.rados.rados.0.smithi045.stdout:2042: seq_num 633 ranges {385024=606208,991232=401408} 2021-07-31T01:52:28.588 INFO:tasks.rados.rados.0.smithi045.stdout:2042: writing smithi04569269-40 from 385024 to 991232 tid 1 2021-07-31T01:52:28.590 INFO:tasks.rados.rados.0.smithi045.stdout:2042: writing smithi04569269-40 from 991232 to 1392640 tid 2 2021-07-31T01:52:28.590 INFO:tasks.rados.rados.0.smithi045.stdout:2043: setattr oid 18 current snap is 198 2021-07-31T01:52:28.591 INFO:tasks.rados.rados.0.smithi045.stdout:2044: setattr oid 7 current snap is 198 2021-07-31T01:52:28.592 INFO:tasks.rados.rados.0.smithi045.stdout:2045: rollback oid 12 current snap is 198 2021-07-31T01:52:28.593 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 12 to 193 2021-07-31T01:52:28.593 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 4 current snap is 198 2021-07-31T01:52:28.593 INFO:tasks.rados.rados.0.smithi045.stdout:2046: seq_num 634 ranges {0=622592,622592=401408} 2021-07-31T01:52:28.596 INFO:tasks.rados.rados.0.smithi045.stdout:2046: writing smithi04569269-4 from 0 to 622592 tid 1 2021-07-31T01:52:28.599 INFO:tasks.rados.rados.0.smithi045.stdout:2046: writing smithi04569269-4 from 622592 to 1024000 tid 2 2021-07-31T01:52:28.600 INFO:tasks.rados.rados.0.smithi045.stdout:2047: read oid 32 snap 192 2021-07-31T01:52:28.600 INFO:tasks.rados.rados.0.smithi045.stdout:2047: expect (ObjNum 586 snap 180 seq_num 586) 2021-07-31T01:52:28.600 INFO:tasks.rados.rados.0.smithi045.stdout:2048: copy_from oid 46 from oid 20 current snap is 198 2021-07-31T01:52:28.601 INFO:tasks.rados.rados.0.smithi045.stdout:2049: read oid 17 snap 193 2021-07-31T01:52:28.601 INFO:tasks.rados.rados.0.smithi045.stdout:2049: expect (ObjNum 573 snap 172 seq_num 573) 2021-07-31T01:52:28.601 INFO:tasks.rados.rados.0.smithi045.stdout:2050: copy_from oid 9 from oid 25 current snap is 198 2021-07-31T01:52:28.601 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 6 current snap is 198 2021-07-31T01:52:28.601 INFO:tasks.rados.rados.0.smithi045.stdout:2051: seq_num 635 ranges {1630208=638976,2269184=401408} 2021-07-31T01:52:28.612 INFO:tasks.rados.rados.0.smithi045.stdout:2051: writing smithi04569269-6 from 1630208 to 2269184 tid 1 2021-07-31T01:52:28.614 INFO:tasks.rados.rados.0.smithi045.stdout:2051: writing smithi04569269-6 from 2269184 to 2670592 tid 2 2021-07-31T01:52:28.615 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 48 current snap is 198 2021-07-31T01:52:28.615 INFO:tasks.rados.rados.0.smithi045.stdout:2042: finishing write tid 1 to smithi04569269-40 2021-07-31T01:52:28.615 INFO:tasks.rados.rados.0.smithi045.stdout:2052: seq_num 636 ranges {1826816=655360,2482176=401408} 2021-07-31T01:52:28.628 INFO:tasks.rados.rados.0.smithi045.stdout:2052: writing smithi04569269-48 from 1826816 to 2482176 tid 1 2021-07-31T01:52:28.631 INFO:tasks.rados.rados.0.smithi045.stdout:2052: writing smithi04569269-48 from 2482176 to 2883584 tid 2 2021-07-31T01:52:28.631 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 39 current snap is 198 2021-07-31T01:52:28.631 INFO:tasks.rados.rados.0.smithi045.stdout:2053: seq_num 637 ranges {0=671744,671744=401408} 2021-07-31T01:52:28.635 INFO:tasks.rados.rados.0.smithi045.stdout:2053: writing smithi04569269-39 from 0 to 671744 tid 1 2021-07-31T01:52:28.638 INFO:tasks.rados.rados.0.smithi045.stdout:2053: writing smithi04569269-39 from 671744 to 1073152 tid 2 2021-07-31T01:52:28.639 INFO:tasks.rados.rados.0.smithi045.stdout:2054: rollback oid 21 current snap is 198 2021-07-31T01:52:28.639 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 21 to 182 2021-07-31T01:52:28.639 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 18 v 514 (ObjNum 615 snap 186 seq_num 615) dirty exists 2021-07-31T01:52:28.639 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 7 v 980 (ObjNum 618 snap 189 seq_num 618) dirty exists 2021-07-31T01:52:28.640 INFO:tasks.rados.rados.0.smithi045.stdout:2042: finishing write tid 2 to smithi04569269-40 2021-07-31T01:52:28.640 INFO:tasks.rados.rados.0.smithi045.stdout:2042: finishing write tid 3 to smithi04569269-40 2021-07-31T01:52:28.640 INFO:tasks.rados.rados.0.smithi045.stdout:2045: finishing rollback tid 0 to smithi04569269-12 2021-07-31T01:52:28.640 INFO:tasks.rados.rados.0.smithi045.stdout:2046: finishing write tid 1 to smithi04569269-4 2021-07-31T01:52:28.641 INFO:tasks.rados.rados.0.smithi045.stdout:2045: finishing rollback tid 1 to smithi04569269-12 2021-07-31T01:52:28.641 INFO:tasks.rados.rados.0.smithi045.stdout:2042: finishing write tid 4 to smithi04569269-40 2021-07-31T01:52:28.641 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 40 v 757 (ObjNum 633 snap 198 seq_num 633) dirty exists 2021-07-31T01:52:28.641 INFO:tasks.rados.rados.0.smithi045.stdout:2042: left oid 40 (ObjNum 633 snap 198 seq_num 633) 2021-07-31T01:52:28.641 INFO:tasks.rados.rados.0.smithi045.stdout:2046: finishing write tid 2 to smithi04569269-4 2021-07-31T01:52:28.642 INFO:tasks.rados.rados.0.smithi045.stdout:2046: finishing write tid 3 to smithi04569269-4 2021-07-31T01:52:28.642 INFO:tasks.rados.rados.0.smithi045.stdout:2046: finishing write tid 4 to smithi04569269-4 2021-07-31T01:52:28.642 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 4 v 517 (ObjNum 634 snap 198 seq_num 634) dirty exists 2021-07-31T01:52:28.642 INFO:tasks.rados.rados.0.smithi045.stdout:2046: left oid 4 (ObjNum 634 snap 198 seq_num 634) 2021-07-31T01:52:28.643 INFO:tasks.rados.rados.0.smithi045.stdout:2045: finishing rollback tid 2 to smithi04569269-12 2021-07-31T01:52:28.643 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 12 v 761 (ObjNum 506 snap 152 seq_num 506) dirty exists 2021-07-31T01:52:28.643 INFO:tasks.rados.rados.0.smithi045.stdout:2047: expect (ObjNum 586 snap 180 seq_num 586) 2021-07-31T01:52:28.731 INFO:tasks.rados.rados.0.smithi045.stdout:2051: finishing write tid 1 to smithi04569269-6 2021-07-31T01:52:28.731 INFO:tasks.rados.rados.0.smithi045.stdout:2051: finishing write tid 2 to smithi04569269-6 2021-07-31T01:52:28.731 INFO:tasks.rados.rados.0.smithi045.stdout:2051: finishing write tid 3 to smithi04569269-6 2021-07-31T01:52:28.732 INFO:tasks.rados.rados.0.smithi045.stdout:2051: finishing write tid 4 to smithi04569269-6 2021-07-31T01:52:28.732 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 6 v 765 (ObjNum 635 snap 198 seq_num 635) dirty exists 2021-07-31T01:52:28.732 INFO:tasks.rados.rados.0.smithi045.stdout:2051: left oid 6 (ObjNum 635 snap 198 seq_num 635) 2021-07-31T01:52:28.732 INFO:tasks.rados.rados.0.smithi045.stdout:2048: finishing copy_from to smithi04569269-46 2021-07-31T01:52:28.732 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 46 v 1290 (ObjNum 556 snap 166 seq_num 556) dirty exists 2021-07-31T01:52:28.733 INFO:tasks.rados.rados.0.smithi045.stdout:2048: finishing copy_from racing read to smithi04569269-46 2021-07-31T01:52:28.733 INFO:tasks.rados.rados.0.smithi045.stdout:2052: finishing write tid 1 to smithi04569269-48 2021-07-31T01:52:28.733 INFO:tasks.rados.rados.0.smithi045.stdout:2052: finishing write tid 2 to smithi04569269-48 2021-07-31T01:52:28.733 INFO:tasks.rados.rados.0.smithi045.stdout:2042: done (12 left) 2021-07-31T01:52:28.734 INFO:tasks.rados.rados.0.smithi045.stdout:2043: done (11 left) 2021-07-31T01:52:28.734 INFO:tasks.rados.rados.0.smithi045.stdout:2044: done (10 left) 2021-07-31T01:52:28.734 INFO:tasks.rados.rados.0.smithi045.stdout:2045: done (9 left) 2021-07-31T01:52:28.734 INFO:tasks.rados.rados.0.smithi045.stdout:2046: done (8 left) 2021-07-31T01:52:28.734 INFO:tasks.rados.rados.0.smithi045.stdout:2047: done (7 left) 2021-07-31T01:52:28.735 INFO:tasks.rados.rados.0.smithi045.stdout:2048: done (6 left) 2021-07-31T01:52:28.735 INFO:tasks.rados.rados.0.smithi045.stdout:2051: done (5 left) 2021-07-31T01:52:28.735 INFO:tasks.rados.rados.0.smithi045.stdout:2055: rollback oid 4 current snap is 198 2021-07-31T01:52:28.735 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 4 to 196 2021-07-31T01:52:28.736 INFO:tasks.rados.rados.0.smithi045.stdout:2052: finishing write tid 3 to smithi04569269-48 2021-07-31T01:52:28.736 INFO:tasks.rados.rados.0.smithi045.stdout:2049: expect (ObjNum 573 snap 172 seq_num 573) 2021-07-31T01:52:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[34553]: cluster 2021-07-31T01:52:27.577238+0000 mon.a (mon.0) 1098 : cluster [DBG] osdmap e448: 8 total, 8 up, 8 in 2021-07-31T01:52:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[34553]: 2021-07-31T01:52:29.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[34553]: cluster 2021-07-31T01:52:27.864456+0000 mgr.x (mgr.24542) 218 : cluster [DBG] pgmap v452: 25 pgs: 25 active+clean; 181 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2021-07-31T01:52:29.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[34553]: audit 2021-07-31T01:52:27.959223+0000 mon.b (mon.2) 87 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:52:29.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[34553]: audit 2021-07-31T01:52:27.959688+0000 mon.a (mon.0) 1099 : audit [INF] 2021-07-31T01:52:29.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[34553]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:52:29.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[34553]: audit 2021-07 2021-07-31T01:52:29.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[34553]: -31T01:52:27.971347+0000 mon.b (mon.2 2021-07-31T01:52:29.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[34553]: ) 88 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T01:52:29.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[34553]: audit 2021-07 2021-07-31T01:52:29.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[34553]: -31T01:52:27.971917+0000 mon.a (mon.0) 1100 : 2021-07-31T01:52:29.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[34553]: audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T01:52:29.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: debug 2021-07-31T01:52:28.742+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T01:52:29.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:52:29.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:52:29.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: yield 2021-07-31T01:52:29.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:52:29.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:52:29.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:52:29.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T01:52:29.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:52:29.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T01:52:29.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:52:29.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: await self._waiter 2021-07-31T01:52:29.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:52:29.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: task.result() 2021-07-31T01:52:29.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T01:52:29.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: debug 2021-07-31T01:52:28.753+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T01:52:29.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:52:29.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:52:29.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: yield 2021-07-31T01:52:29.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:52:29.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:52:29.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:52:29.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T01:52:29.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:52:29.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T01:52:29.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:52:29.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: await self._waiter 2021-07-31T01:52:29.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:52:29.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: task.result() 2021-07-31T01:52:29.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T01:52:29.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: debug 2021-07-31T01:52:28.778+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T01:52:29.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:52:29.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:52:29.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: yield 2021-07-31T01:52:29.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:52:29.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:52:29.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:52:29.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T01:52:29.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:52:29.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T01:52:29.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:52:29.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: await self._waiter 2021-07-31T01:52:29.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:52:29.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: task.result() 2021-07-31T01:52:29.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T01:52:29.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: debug 2021-07-31T01:52:28.778+0000 7fcb93961700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:52:29.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:52:29.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:52:29.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: yield 2021-07-31T01:52:29.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:52:29.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:52:29.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:52:29.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T01:52:29.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:52:29.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T01:52:29.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:52:29.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: await self._waiter 2021-07-31T01:52:29.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:52:29.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: task.result() 2021-07-31T01:52:29.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T01:52:29.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: 2021-07-31T01:52:29.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T01:52:29.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: 2021-07-31T01:52:29.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:52:29.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:52:29.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: return f(*arg) 2021-07-31T01:52:29.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:52:29.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:52:29.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:52:29.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:52:29.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:52:29.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:52:29.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:52:29.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T01:52:29.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:52:29.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: raise self._exception 2021-07-31T01:52:29.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:52:29.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:52:29.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:52:29.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T01:52:29.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:52:29.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: raise 2021-07-31T01:52:29.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:52:29.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T01:52:29.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:52:29.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:52:29.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T01:52:29.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: debug 2021-07-31T01:52:28.787+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T01:52:29.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:52:29.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:52:29.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: yield 2021-07-31T01:52:29.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:52:29.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:52:29.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:52:29.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T01:52:29.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:52:29.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T01:52:29.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:52:29.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: await self._waiter 2021-07-31T01:52:29.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:52:29.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: task.result() 2021-07-31T01:52:29.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T01:52:29.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: debug 2021-07-31T01:52:28.787+0000 7fcb94162700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:52:29.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:52:29.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:52:29.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: yield 2021-07-31T01:52:29.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:52:29.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:52:29.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:52:29.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T01:52:29.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:52:29.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T01:52:29.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:52:29.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: await self._waiter 2021-07-31T01:52:29.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:52:29.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: task.result() 2021-07-31T01:52:29.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T01:52:29.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: 2021-07-31T01:52:29.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T01:52:29.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: 2021-07-31T01:52:29.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:52:29.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:52:29.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: return f(*arg) 2021-07-31T01:52:29.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:52:29.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:52:29.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:52:29.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:52:29.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:52:29.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:52:29.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:52:29.035 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T01:52:29.035 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:52:29.035 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: raise self._exception 2021-07-31T01:52:29.035 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:52:29.036 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:52:29.036 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:52:29.036 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T01:52:29.036 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:52:29.036 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: raise 2021-07-31T01:52:29.037 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:52:29.037 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T01:52:29.037 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:52:29.037 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:52:29.038 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:52:28 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T01:52:29.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:28 smithi045 conmon[39718]: cluster 2021-07-31T01:52:27.577238+0000 mon.a (mon.0) 1098 : cluster [DBG] 2021-07-31T01:52:29.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:28 smithi045 conmon[39718]: osdmap e448: 8 total, 8 up, 8 in 2021-07-31T01:52:29.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:28 smithi045 conmon[39718]: cluster 2021-07-31T01:52:27.864456+0000 mgr.x (mgr.24542) 218 : cluster [DBG] pgmap v452: 25 pgs: 25 active+clean; 181 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2021-07-31T01:52:29.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:28 smithi045 conmon[39718]: audit 2021-07-31T01:52:27.959223+0000 mon.b (mon.2) 87 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:52:29.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:28 smithi045 conmon[39718]: audit 2021-07-31T01:52:27.959688+0000 2021-07-31T01:52:29.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:28 smithi045 conmon[39718]: mon.a (mon.0) 1099 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:52:29.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:28 smithi045 conmon[39718]: audit 2021-07-31T01: 2021-07-31T01:52:29.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:28 smithi045 conmon[39718]: 52:27.971347+0000 mon.b (mon.2) 88 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T01:52:29.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:28 smithi045 conmon[39718]: audit 2021-07-31T01 2021-07-31T01:52:29.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:28 smithi045 conmon[39718]: :52:27.971917+0000 mon.a (mon.0) 1100 : audit 2021-07-31T01:52:29.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:28 smithi045 conmon[39718]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T01:52:29.041 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:28 smithi045 conmon[33655]: cluster 2021-07-31T01:52:27.577238+0000 mon.a (mon.0) 1098 : cluster [DBG] osdmap e448: 8 total, 8 up, 8 in 2021-07-31T01:52:29.041 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:28 smithi045 conmon[33655]: cluster 2021-07-31T01:52:29.041 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:28 smithi045 conmon[33655]: 2021-07-31T01:52:27.864456+0000 mgr.x (mgr.24542) 218 : cluster [DBG] pgmap v452: 25 pgs: 25 active+clean; 181 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2021-07-31T01:52:29.042 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:28 smithi045 conmon[33655]: audit 2021-07-31T01:52:27.959223+0000 mon.b (mon.2) 87 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:52:29.042 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:28 smithi045 conmon[33655]: audit 2021-07-31T01:52:29.042 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:28 smithi045 conmon[33655]: 2021-07-31T01:52:27.959688+0000 mon.a (mon.0) 1099 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:52:29.042 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:28 smithi045 conmon[33655]: audit 2021-07-31T 2021-07-31T01:52:29.042 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:28 smithi045 conmon[33655]: 01:52:27.971347+0000 mon.b (mon.2) 88 : audit 2021-07-31T01:52:29.043 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:28 smithi045 conmon[33655]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T01:52:29.043 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:28 smithi045 conmon[33655]: audit 2021-07-31T01:52:29.043 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:28 smithi045 conmon[33655]: 2021-07-31T01:52:27. 2021-07-31T01:52:29.043 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:28 smithi045 conmon[33655]: 971917+0000 mon.a (mon.0) 1100 : audit [INF] 2021-07-31T01:52:29.044 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:28 smithi045 conmon[33655]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T01:52:29.045 INFO:tasks.rados.rados.0.smithi045.stdout:2050: finishing copy_from to smithi04569269-9 2021-07-31T01:52:29.046 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 9 v 290 (ObjNum 607 snap 186 seq_num 607) dirty exists 2021-07-31T01:52:29.046 INFO:tasks.rados.rados.0.smithi045.stdout:2050: finishing copy_from racing read to smithi04569269-9 2021-07-31T01:52:29.046 INFO:tasks.rados.rados.0.smithi045.stdout:2053: finishing write tid 1 to smithi04569269-39 2021-07-31T01:52:29.046 INFO:tasks.rados.rados.0.smithi045.stdout:2049: done (5 left) 2021-07-31T01:52:29.047 INFO:tasks.rados.rados.0.smithi045.stdout:2050: done (4 left) 2021-07-31T01:52:29.047 INFO:tasks.rados.rados.0.smithi045.stdout:2056: read oid 5 snap -1 2021-07-31T01:52:29.047 INFO:tasks.rados.rados.0.smithi045.stdout:2056: expect deleted 2021-07-31T01:52:29.047 INFO:tasks.rados.rados.0.smithi045.stdout:2057: read oid 47 snap -1 2021-07-31T01:52:29.047 INFO:tasks.rados.rados.0.smithi045.stdout:2057: expect deleted 2021-07-31T01:52:29.048 INFO:tasks.rados.rados.0.smithi045.stdout:2058: setattr oid 37 current snap is 198 2021-07-31T01:52:29.048 INFO:tasks.rados.rados.0.smithi045.stdout:2054: finishing rollback tid 1 to smithi04569269-21 2021-07-31T01:52:29.048 INFO:tasks.rados.rados.0.smithi045.stdout:2052: finishing write tid 4 to smithi04569269-48 2021-07-31T01:52:29.048 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 48 v 1017 (ObjNum 636 snap 198 seq_num 636) dirty exists 2021-07-31T01:52:29.048 INFO:tasks.rados.rados.0.smithi045.stdout:2052: left oid 48 (ObjNum 636 snap 198 seq_num 636) 2021-07-31T01:52:29.049 INFO:tasks.rados.rados.0.smithi045.stdout:2053: finishing write tid 2 to smithi04569269-39 2021-07-31T01:52:29.049 INFO:tasks.rados.rados.0.smithi045.stdout:2054: finishing rollback tid 2 to smithi04569269-21 2021-07-31T01:52:29.049 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 21 v 1302 (ObjNum 589 snap 182 seq_num 589) dirty exists 2021-07-31T01:52:29.049 INFO:tasks.rados.rados.0.smithi045.stdout:2053: finishing write tid 3 to smithi04569269-39 2021-07-31T01:52:29.049 INFO:tasks.rados.rados.0.smithi045.stdout:2053: finishing write tid 4 to smithi04569269-39 2021-07-31T01:52:29.050 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 39 v 1140 (ObjNum 637 snap 198 seq_num 637) dirty exists 2021-07-31T01:52:29.050 INFO:tasks.rados.rados.0.smithi045.stdout:2053: left oid 39 (ObjNum 637 snap 198 seq_num 637) 2021-07-31T01:52:29.050 INFO:tasks.rados.rados.0.smithi045.stdout:2055: finishing rollback tid 0 to smithi04569269-4 2021-07-31T01:52:29.050 INFO:tasks.rados.rados.0.smithi045.stdout:2055: finishing rollback tid 1 to smithi04569269-4 2021-07-31T01:52:29.051 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 4 v 519 (ObjNum 288 snap 0 seq_num 65) dirty dne 2021-07-31T01:52:29.051 INFO:tasks.rados.rados.0.smithi045.stdout:2052: done (6 left) 2021-07-31T01:52:29.051 INFO:tasks.rados.rados.0.smithi045.stdout:2053: done (5 left) 2021-07-31T01:52:29.051 INFO:tasks.rados.rados.0.smithi045.stdout:2054: done (4 left) 2021-07-31T01:52:29.051 INFO:tasks.rados.rados.0.smithi045.stdout:2055: done (3 left) 2021-07-31T01:52:29.052 INFO:tasks.rados.rados.0.smithi045.stdout:2056: done (2 left) 2021-07-31T01:52:29.052 INFO:tasks.rados.rados.0.smithi045.stdout:2057: done (1 left) 2021-07-31T01:52:29.052 INFO:tasks.rados.rados.0.smithi045.stdout:2059: rollback oid 30 current snap is 198 2021-07-31T01:52:29.052 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 30 to 193 2021-07-31T01:52:29.052 INFO:tasks.rados.rados.0.smithi045.stdout:2060: snap_create 2021-07-31T01:52:29.053 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 37 v 520 (ObjNum 1953066355 snap 875587944 seq_num 842610229) dirty exists 2021-07-31T01:52:29.053 INFO:tasks.rados.rados.0.smithi045.stdout:2059: finishing rollback tid 0 to smithi04569269-30 2021-07-31T01:52:29.053 INFO:tasks.rados.rados.0.smithi045.stdout:2059: finishing rollback tid 1 to smithi04569269-30 2021-07-31T01:52:29.054 INFO:tasks.rados.rados.0.smithi045.stdout:2059: finishing rollback tid 2 to smithi04569269-30 2021-07-31T01:52:29.054 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 30 v 530 (ObjNum -1105886784 snap 21873 seq_num 3187445776) dirty exists 2021-07-31T01:52:29.585 INFO:tasks.rados.rados.0.smithi045.stdout:2058: done (2 left) 2021-07-31T01:52:29.585 INFO:tasks.rados.rados.0.smithi045.stdout:2059: done (1 left) 2021-07-31T01:52:29.586 INFO:tasks.rados.rados.0.smithi045.stdout:2060: done (0 left) 2021-07-31T01:52:29.586 INFO:tasks.rados.rados.0.smithi045.stdout:2061: read oid 7 snap -1 2021-07-31T01:52:29.586 INFO:tasks.rados.rados.0.smithi045.stdout:2061: expect (ObjNum 618 snap 189 seq_num 618) 2021-07-31T01:52:29.587 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 43 current snap is 199 2021-07-31T01:52:29.587 INFO:tasks.rados.rados.0.smithi045.stdout:2062: seq_num 638 ranges {3014656=688128,3702784=401408} 2021-07-31T01:52:29.605 INFO:tasks.rados.rados.0.smithi045.stdout:2062: writing smithi04569269-43 from 3014656 to 3702784 tid 1 2021-07-31T01:52:29.607 INFO:tasks.rados.rados.0.smithi045.stdout:2062: writing smithi04569269-43 from 3702784 to 4104192 tid 2 2021-07-31T01:52:29.608 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 41 current snap is 199 2021-07-31T01:52:29.608 INFO:tasks.rados.rados.0.smithi045.stdout:2063: seq_num 639 ranges {5668864=704512,6373376=401408} 2021-07-31T01:52:29.642 INFO:tasks.rados.rados.0.smithi045.stdout:2063: writing smithi04569269-41 from 5668864 to 6373376 tid 1 2021-07-31T01:52:29.644 INFO:tasks.rados.rados.0.smithi045.stdout:2063: writing smithi04569269-41 from 6373376 to 6774784 tid 2 2021-07-31T01:52:29.645 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 8 current snap is 199 2021-07-31T01:52:29.645 INFO:tasks.rados.rados.0.smithi045.stdout:2064: seq_num 640 ranges {0=729088,729088=401408} 2021-07-31T01:52:29.648 INFO:tasks.rados.rados.0.smithi045.stdout:2064: writing smithi04569269-8 from 0 to 729088 tid 1 2021-07-31T01:52:29.652 INFO:tasks.rados.rados.0.smithi045.stdout:2064: writing smithi04569269-8 from 729088 to 1130496 tid 2 2021-07-31T01:52:29.652 INFO:tasks.rados.rados.0.smithi045.stdout:2065: read oid 17 snap -1 2021-07-31T01:52:29.652 INFO:tasks.rados.rados.0.smithi045.stdout:2065: expect (ObjNum 573 snap 172 seq_num 573) 2021-07-31T01:52:29.652 INFO:tasks.rados.rados.0.smithi045.stdout:2066: read oid 16 snap 198 2021-07-31T01:52:29.652 INFO:tasks.rados.rados.0.smithi045.stdout:2066: expect deleted 2021-07-31T01:52:29.653 INFO:tasks.rados.rados.0.smithi045.stdout:2067: delete oid 35 current snap is 199 2021-07-31T01:52:29.654 INFO:tasks.rados.rados.0.smithi045.stdout:2062: finishing write tid 1 to smithi04569269-43 2021-07-31T01:52:29.654 INFO:tasks.rados.rados.0.smithi045.stdout:2062: finishing write tid 2 to smithi04569269-43 2021-07-31T01:52:29.654 INFO:tasks.rados.rados.0.smithi045.stdout:2062: finishing write tid 3 to smithi04569269-43 2021-07-31T01:52:29.654 INFO:tasks.rados.rados.0.smithi045.stdout:2061: expect (ObjNum 618 snap 189 seq_num 618) 2021-07-31T01:52:30.002 INFO:tasks.rados.rados.0.smithi045.stdout:2062: finishing write tid 4 to smithi04569269-43 2021-07-31T01:52:30.002 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 43 v 1050 (ObjNum 638 snap 199 seq_num 638) dirty exists 2021-07-31T01:52:30.002 INFO:tasks.rados.rados.0.smithi045.stdout:2062: left oid 43 (ObjNum 638 snap 199 seq_num 638) 2021-07-31T01:52:30.003 INFO:tasks.rados.rados.0.smithi045.stdout:2063: finishing write tid 1 to smithi04569269-41 2021-07-31T01:52:30.003 INFO:tasks.rados.rados.0.smithi045.stdout:2063: finishing write tid 2 to smithi04569269-41 2021-07-31T01:52:30.003 INFO:tasks.rados.rados.0.smithi045.stdout:2061: done (6 left) 2021-07-31T01:52:30.003 INFO:tasks.rados.rados.0.smithi045.stdout:2062: done (5 left) 2021-07-31T01:52:30.004 INFO:tasks.rados.rados.0.smithi045.stdout:2067: done (4 left) 2021-07-31T01:52:30.004 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 2 current snap is 199 2021-07-31T01:52:30.004 INFO:tasks.rados.rados.0.smithi045.stdout:2063: finishing write tid 3 to smithi04569269-41 2021-07-31T01:52:30.004 INFO:tasks.rados.rados.0.smithi045.stdout:2063: finishing write tid 4 to smithi04569269-41 2021-07-31T01:52:30.004 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 41 v 983 (ObjNum 639 snap 199 seq_num 639) dirty exists 2021-07-31T01:52:30.005 INFO:tasks.rados.rados.0.smithi045.stdout:2063: left oid 41 (ObjNum 639 snap 199 seq_num 639) 2021-07-31T01:52:30.005 INFO:tasks.rados.rados.0.smithi045.stdout:2064: finishing write tid 1 to smithi04569269-8 2021-07-31T01:52:30.005 INFO:tasks.rados.rados.0.smithi045.stdout:2064: finishing write tid 2 to smithi04569269-8 2021-07-31T01:52:30.005 INFO:tasks.rados.rados.0.smithi045.stdout:2068: seq_num 642 ranges {5775360=761856,6537216=401408} 2021-07-31T01:52:30.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: cluster 2021-07-31T01:52:28.582556+0000 mon.a (mon.0) 2021-07-31T01:52:30.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: 1101 : cluster [DBG] osdmap e449: 8 total, 8 up, 8 in 2021-07-31T01:52:30.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: audit 2021-07-31T01:52:28.700030+0000 mon.b (mon.2) 89 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:52:30.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: audit 2021-07-31T01:52:28.701181+0000 mon.b (mon.2) 90 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:52:30.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: 2021-07-31T01:52:30.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: audit 2021-07-31T01:52 2021-07-31T01:52:30.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: :28.701865+0000 mon.b (mon.2) 91 : audit 2021-07-31T01:52:30.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:52:30.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: cephadm 2021-07-31T01:52:30.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: 2021-07-31T01:52:28. 2021-07-31T01:52:30.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: 702458+0000 mgr.x (mgr.24542) 219 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T01:52:30.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: 2021-07-31T01:52:30.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: cephadm 2021-07-31T01:52: 2021-07-31T01:52:30.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: 28.743534+0000 mgr.x (mgr.24542) 220 : cephadm [ERR] 2021-07-31T01:52:30.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:52:30.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:52:30.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: yield 2021-07-31T01:52:30.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:52:30.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:52:30.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:52:30.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T01:52:30.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:52:30.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T01:52:30.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:52:30.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: await self._waiter 2021-07-31T01:52:30.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:52:30.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: task.result() 2021-07-31T01:52:30.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T01:52:30.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: cephadm 2021-07-31T01:52:28.743928+0000 mgr.x (mgr.24542) 221 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T01:52:30.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: cephadm 2021-07-31T01:52:28.754753+0000 mgr.x (mgr.24542) 222 : cephadm [ERR] 2021-07-31T01:52:30.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:52:30.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:52:30.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: yield 2021-07-31T01:52:30.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:52:30.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:52:30.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:52:30.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T01:52:30.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:52:30.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T01:52:30.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:52:30.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: await self._waiter 2021-07-31T01:52:30.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:52:30.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: task.result() 2021-07-31T01:52:30.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T01:52:30.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: cephadm 2021-07-31T01:52:28.755324+0000 mgr.x (mgr.24542) 223 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T01:52:30.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: cephadm 2021-07-31T01:52:28.779307 2021-07-31T01:52:30.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: +0000 mgr.x (mgr.24542) 224 : 2021-07-31T01:52:30.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: cephadm [ERR] 2021-07-31T01:52:30.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:52:30.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:52:30.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: yield 2021-07-31T01:52:30.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:52:30.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:52:30.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:52:30.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T01:52:30.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:52:30.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T01:52:30.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:52:30.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: await self._waiter 2021-07-31T01:52:30.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:52:30.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: task.result() 2021-07-31T01:52:30.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T01:52:30.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: cephadm 2021-07-31T01:52:28.779865+0000 mgr.x (mgr.24542) 225 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:52:30.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:52:30.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:52:30.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: yield 2021-07-31T01:52:30.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:52:30.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:52:30.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:52:30.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T01:52:30.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:52:30.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T01:52:30.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:52:30.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: await self._waiter 2021-07-31T01:52:30.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:52:30.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: task.result() 2021-07-31T01:52:30.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T01:52:30.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: 2021-07-31T01:52:30.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T01:52:30.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: 2021-07-31T01:52:30.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:52:30.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:52:30.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: return f(*arg) 2021-07-31T01:52:30.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:52:30.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:52:30.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:52:30.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:52:30.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:52:30.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:52:30.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:52:30.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T01:52:30.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:52:30.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: raise self._exception 2021-07-31T01:52:30.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:52:30.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:52:30.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:52:30.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T01:52:30.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:52:30.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: raise 2021-07-31T01:52:30.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:52:30.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T01:52:30.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:52:30.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:52:30.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T01:52:30.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: cephadm 2021-07-31T01:52:28.788392+0000 mgr.x (mgr.24542) 226 : cephadm [ERR] 2021-07-31T01:52:30.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:52:30.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:52:30.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: yield 2021-07-31T01:52:30.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:52:30.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:52:30.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:52:30.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T01:52:30.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:52:30.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T01:52:30.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:52:30.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: await self._waiter 2021-07-31T01:52:30.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:52:30.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: task.result() 2021-07-31T01:52:30.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T01:52:30.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: cephadm 2021-07-31T01:52:28.788747+0000 mgr.x (mgr.24542) 227 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:52:30.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:52:30.035 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:52:30.035 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: yield 2021-07-31T01:52:30.035 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:52:30.035 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:52:30.035 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:52:30.036 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T01:52:30.036 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:52:30.036 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T01:52:30.036 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:52:30.037 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: await self._waiter 2021-07-31T01:52:30.037 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:52:30.037 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: task.result() 2021-07-31T01:52:30.037 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T01:52:30.037 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: 2021-07-31T01:52:30.038 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T01:52:30.038 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: 2021-07-31T01:52:30.038 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:52:30.038 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:52:30.039 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: return f(*arg) 2021-07-31T01:52:30.039 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:52:30.039 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:52:30.039 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:52:30.039 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:52:30.040 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:52:30.040 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:52:30.040 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:52:30.040 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T01:52:30.040 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:52:30.041 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: raise self._exception 2021-07-31T01:52:30.041 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:52:30.041 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:52:30.041 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:52:30.041 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T01:52:30.042 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:52:30.042 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: raise 2021-07-31T01:52:30.042 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:52:30.042 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T01:52:30.043 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:52:30.043 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:52:30.043 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:29 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T01:52:30.044 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: cluster 2021-07-31T01:52:28. 2021-07-31T01:52:30.044 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: 582556+0000 mon.a (mon.0) 1101 : cluster [DBG] osdmap e449: 8 total, 8 up, 8 in 2021-07-31T01:52:30.044 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: audit 2021-07-31T01:52: 2021-07-31T01:52:30.044 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: 28.700030+0000 mon.b (mon.2) 89 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:52:30.045 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: audit 2021-07-31T01 2021-07-31T01:52:30.045 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: :52:28.701181+0000 mon.b (mon.2) 90 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:52:30.045 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: 2021-07-31T01:52:30.045 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: audit 2021-07-31T01 2021-07-31T01:52:30.045 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: :52:28.701865+0000 mon.b (mon.2) 91 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:52:30.046 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: cephadm 2021-07-31T01:52:28.702458 2021-07-31T01:52:30.046 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 219 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T01:52:30.046 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: cephadm 2021-07-31T01:52:28.743534 2021-07-31T01:52:30.046 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 220 : cephadm [ERR] 2021-07-31T01:52:30.046 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:52:30.047 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:52:30.047 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: yield 2021-07-31T01:52:30.047 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:52:30.047 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:52:30.048 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:52:30.048 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T01:52:30.048 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:52:30.048 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T01:52:30.049 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:52:30.049 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: await self._waiter 2021-07-31T01:52:30.049 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:52:30.049 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: task.result() 2021-07-31T01:52:30.049 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T01:52:30.050 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: cephadm 2021-07 2021-07-31T01:52:30.050 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: -31T01:52:28.743928+0000 mgr.x (mgr.24542) 221 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T01:52:30.050 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: cephadm 2021-07-31T01:52:28.754753+0000 mgr.x (mgr.24542) 222 : cephadm [ERR] 2021-07-31T01:52:30.050 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:52:30.051 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:52:30.051 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: yield 2021-07-31T01:52:30.051 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:52:30.051 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:52:30.051 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:52:30.052 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T01:52:30.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: cluster 2021-07-31T01:52:30.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: 2021-07-31T01:52:28.582556+0000 mon.a (mon.0) 1101 : cluster [DBG] osdmap e449: 8 total, 8 up, 8 in 2021-07-31T01:52:30.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: audit 2021-07-31T01:52:28.700030+0000 mon.b (mon.2) 89 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:52:30.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: audit 2021-07-31T01 2021-07-31T01:52:30.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: :52:28.701181+0000 mon.b (mon.2) 90 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:52:30.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: audit 2021-07-31T01:52:30.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: 2021-07-31T01:52:28.701865+0000 mon.b ( 2021-07-31T01:52:30.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: mon.2) 91 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:52:30.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: cephadm 2021-07-31T01:52:28.702458+0000 mgr.x (mgr.24542) 219 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T01:52:30.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: 2021-07-31T01:52:30.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: cephadm 2021-07-31T01:52:30.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: 2021-07-31T01:52:28.743534+0000 mgr.x (mgr.24542 2021-07-31T01:52:30.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: ) 220 : cephadm [ERR] 2021-07-31T01:52:30.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:52:30.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:52:30.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: yield 2021-07-31T01:52:30.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:52:30.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:52:30.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:52:30.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T01:52:30.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:52:30.057 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T01:52:30.057 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:52:30.057 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: await self._waiter 2021-07-31T01:52:30.057 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:52:30.057 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: task.result() 2021-07-31T01:52:30.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T01:52:30.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: cephadm 2021-07-31T01:52:28.743928+0000 mgr.x (mgr.24542) 221 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T01:52:30.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: cephadm 2021-07-31T01:52:30.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: 2021-07-31T01:52 2021-07-31T01:52:30.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: :28.754753+0000 mgr.x (mgr.24542) 222 2021-07-31T01:52:30.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: : cephadm [ERR] 2021-07-31T01:52:30.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:52:30.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:52:30.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: yield 2021-07-31T01:52:30.060 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:52:30.060 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:52:30.060 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:52:30.061 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:52:30.061 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T01:52:30.062 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:52:30.062 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: await self._waiter 2021-07-31T01:52:30.062 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:52:30.063 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: task.result() 2021-07-31T01:52:30.063 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T01:52:30.063 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: cephadm 2021-07-31T01:52:28.755324+0000 mgr.x (mgr.24542) 223 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T01:52:30.063 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: cephadm 2021-07-31T01:52:28.779307+0000 mgr.x (mgr.24542) 224 : cephadm [ERR] 2021-07-31T01:52:30.063 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:52:30.064 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:52:30.064 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: yield 2021-07-31T01:52:30.064 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:52:30.064 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:52:30.065 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:52:30.065 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T01:52:30.065 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:52:30.065 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T01:52:30.065 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:52:30.066 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: await self._waiter 2021-07-31T01:52:30.066 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:52:30.066 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: task.result() 2021-07-31T01:52:30.066 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T01:52:30.067 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: cephadm 2021-07-31T01:52:28.779865+0000 mgr.x (mgr.24542) 225 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:52:30.067 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:52:30.067 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:52:30.067 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: yield 2021-07-31T01:52:30.067 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:52:30.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:52:30.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:52:30.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T01:52:30.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:52:30.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T01:52:30.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:52:30.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: await self._waiter 2021-07-31T01:52:30.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:52:30.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: task.result() 2021-07-31T01:52:30.070 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T01:52:30.070 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: 2021-07-31T01:52:30.070 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T01:52:30.070 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: 2021-07-31T01:52:30.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:52:30.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:52:30.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: return f(*arg) 2021-07-31T01:52:30.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:52:30.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:52:30.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:52:30.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:52:30.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:52:30.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:52:30.073 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:52:30.073 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T01:52:30.073 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:52:30.073 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: raise self._exception 2021-07-31T01:52:30.073 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:52:30.074 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:52:30.074 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:52:30.074 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T01:52:30.074 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:52:30.074 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: raise 2021-07-31T01:52:30.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:52:30.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T01:52:30.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:52:30.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:52:30.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T01:52:30.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: cephadm 2021-07-31T01:52:28.788392+0000 mgr.x (mgr.24542) 226 : cephadm [ERR] 2021-07-31T01:52:30.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:52:30.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:52:30.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: yield 2021-07-31T01:52:30.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:52:30.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:52:30.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:52:30.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T01:52:30.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:52:30.078 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T01:52:30.078 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:52:30.078 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: await self._waiter 2021-07-31T01:52:30.078 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:52:30.079 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: task.result() 2021-07-31T01:52:30.079 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T01:52:30.079 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T01:52:30.080 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:52:30.080 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T01:52:30.080 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:52:30.080 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: await self._waiter 2021-07-31T01:52:30.081 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:52:30.081 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: task.result() 2021-07-31T01:52:30.081 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T01:52:30.081 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: cephadm 2021-07-31T01:52:28.755324+0000 mgr.x (mgr.24542) 223 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T01:52:30.081 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: cephadm 2021-07-31 2021-07-31T01:52:30.082 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: T01:52:28.779307+0000 mgr.x (mgr.24542) 224 : cephadm 2021-07-31T01:52:30.082 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: [ERR] 2021-07-31T01:52:30.082 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:52:30.082 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:52:30.082 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: yield 2021-07-31T01:52:30.083 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:52:30.083 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:52:30.083 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:52:30.084 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T01:52:30.084 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:52:30.084 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T01:52:30.084 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:52:30.084 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: await self._waiter 2021-07-31T01:52:30.085 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:52:30.085 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: task.result() 2021-07-31T01:52:30.085 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T01:52:30.085 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: cephadm 2021-07-31T01:52:28. 2021-07-31T01:52:30.085 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: 779865+0000 mgr.x (mgr.24542) 225 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:52:30.086 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:52:30.086 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:52:30.086 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: yield 2021-07-31T01:52:30.086 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:52:30.087 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:52:30.087 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:52:30.087 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T01:52:30.087 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:52:30.088 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T01:52:30.088 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:52:30.088 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: await self._waiter 2021-07-31T01:52:30.088 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:52:30.088 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: task.result() 2021-07-31T01:52:30.089 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T01:52:30.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: cephadm 2021-07-31T01:52:28.788747+0000 mgr.x (mgr.24542) 227 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:52:30.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:52:30.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:52:30.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: yield 2021-07-31T01:52:30.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:52:30.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:52:30.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:52:30.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T01:52:30.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:52:30.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T01:52:30.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:52:30.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: await self._waiter 2021-07-31T01:52:30.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:52:30.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: task.result() 2021-07-31T01:52:30.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T01:52:30.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: 2021-07-31T01:52:30.093 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T01:52:30.093 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: 2021-07-31T01:52:30.093 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:52:30.093 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:52:30.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: return f(*arg) 2021-07-31T01:52:30.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:52:30.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:52:30.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:52:30.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:52:30.095 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:52:30.095 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:52:30.095 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:52:30.095 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T01:52:30.095 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:52:30.096 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: raise self._exception 2021-07-31T01:52:30.096 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:52:30.096 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:52:30.096 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:52:30.096 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T01:52:30.097 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:52:30.097 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: raise 2021-07-31T01:52:30.097 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:52:30.097 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T01:52:30.098 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:52:30.098 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:52:30.098 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T01:52:30.099 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: 2021-07-31T01:52:30.099 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T01:52:30.099 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: 2021-07-31T01:52:30.099 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:52:30.100 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:52:30.100 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: return f(*arg) 2021-07-31T01:52:30.100 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:52:30.100 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:52:30.101 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:52:30.101 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:52:30.101 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:52:30.101 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:52:30.101 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:52:30.102 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T01:52:30.102 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:52:30.102 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: raise self._exception 2021-07-31T01:52:30.102 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:52:30.103 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:52:30.103 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:52:30.103 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T01:52:30.103 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:52:30.103 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: raise 2021-07-31T01:52:30.104 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:52:30.104 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T01:52:30.104 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:52:30.104 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:52:30.104 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T01:52:30.105 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: 2021-07-31T01:52:30.105 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: cephadm 2021-07-31T01:52:28.788392+0000 mgr.x (mgr.24542) 226 : cephadm [ERR] 2021-07-31T01:52:30.105 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:52:30.105 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:52:30.105 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: yield 2021-07-31T01:52:30.106 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:52:30.106 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:52:30.106 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:52:30.106 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T01:52:30.106 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:52:30.107 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T01:52:30.107 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:52:30.107 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: await self._waiter 2021-07-31T01:52:30.107 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:52:30.108 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: task.result() 2021-07-31T01:52:30.108 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T01:52:30.108 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: cephadm 2021-07-31T01:52:28.788747+0000 mgr.x (mgr.24542) 227 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:52:30.108 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:52:30.108 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:52:30.109 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: yield 2021-07-31T01:52:30.109 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:52:30.109 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:52:30.110 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:52:30.110 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T01:52:30.110 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:52:30.110 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T01:52:30.111 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:52:30.111 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: await self._waiter 2021-07-31T01:52:30.111 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:52:30.111 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: task.result() 2021-07-31T01:52:30.111 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T01:52:30.112 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: 2021-07-31T01:52:30.112 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T01:52:30.112 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: 2021-07-31T01:52:30.112 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:52:30.112 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:52:30.113 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: return f(*arg) 2021-07-31T01:52:30.113 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:52:30.113 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:52:30.113 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:52:30.114 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:52:30.114 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:52:30.114 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:52:30.114 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:52:30.114 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T01:52:30.115 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:52:30.115 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: raise self._exception 2021-07-31T01:52:30.115 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:52:30.115 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:52:30.115 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:52:30.116 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T01:52:30.116 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:52:30.116 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: raise 2021-07-31T01:52:30.116 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:52:30.116 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T01:52:30.117 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:52:30.117 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:52:30.117 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:29 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T01:52:30.118 INFO:tasks.rados.rados.0.smithi045.stdout:2068: writing smithi04569269-2 from 5775360 to 6537216 tid 1 2021-07-31T01:52:30.119 INFO:tasks.rados.rados.0.smithi045.stdout:2068: writing smithi04569269-2 from 6537216 to 6938624 tid 2 2021-07-31T01:52:30.119 INFO:tasks.rados.rados.0.smithi045.stdout:2063: done (4 left) 2021-07-31T01:52:30.119 INFO:tasks.rados.rados.0.smithi045.stdout:2066: done (3 left) 2021-07-31T01:52:30.119 INFO:tasks.rados.rados.0.smithi045.stdout:2069: rmattr oid 24 current snap is 199 2021-07-31T01:52:30.120 INFO:tasks.rados.rados.0.smithi045.stdout:2069: done (3 left) 2021-07-31T01:52:30.120 INFO:tasks.rados.rados.0.smithi045.stdout:2070: copy_from oid 42 from oid 3 current snap is 199 2021-07-31T01:52:30.120 INFO:tasks.rados.rados.0.smithi045.stdout:2064: finishing write tid 3 to smithi04569269-8 2021-07-31T01:52:30.120 INFO:tasks.rados.rados.0.smithi045.stdout:2064: finishing write tid 4 to smithi04569269-8 2021-07-31T01:52:30.120 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 8 v 1305 (ObjNum 640 snap 199 seq_num 640) dirty exists 2021-07-31T01:52:30.121 INFO:tasks.rados.rados.0.smithi045.stdout:2064: left oid 8 (ObjNum 640 snap 199 seq_num 640) 2021-07-31T01:52:30.121 INFO:tasks.rados.rados.0.smithi045.stdout:2065: expect (ObjNum 573 snap 172 seq_num 573) 2021-07-31T01:52:30.337 INFO:tasks.rados.rados.0.smithi045.stdout:2064: done (3 left) 2021-07-31T01:52:30.337 INFO:tasks.rados.rados.0.smithi045.stdout:2065: done (2 left) 2021-07-31T01:52:30.338 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 30 current snap is 199 2021-07-31T01:52:30.338 INFO:tasks.rados.rados.0.smithi045.stdout:2068: finishing write tid 1 to smithi04569269-2 2021-07-31T01:52:30.338 INFO:tasks.rados.rados.0.smithi045.stdout:2068: finishing write tid 2 to smithi04569269-2 2021-07-31T01:52:30.339 INFO:tasks.rados.rados.0.smithi045.stdout:2068: finishing write tid 3 to smithi04569269-2 2021-07-31T01:52:30.339 INFO:tasks.rados.rados.0.smithi045.stdout:2071: seq_num 643 ranges {0=778240,778240=401408} 2021-07-31T01:52:30.341 INFO:tasks.rados.rados.0.smithi045.stdout:2071: writing smithi04569269-30 from 0 to 778240 tid 1 2021-07-31T01:52:30.343 INFO:tasks.rados.rados.0.smithi045.stdout:2071: writing smithi04569269-30 from 778240 to 1179648 tid 2 2021-07-31T01:52:30.343 INFO:tasks.rados.rados.0.smithi045.stdout:2072: snap_remove snap 196 2021-07-31T01:52:30.594 INFO:tasks.rados.rados.0.smithi045.stdout:2072: done (3 left) 2021-07-31T01:52:30.594 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 24 current snap is 199 2021-07-31T01:52:30.595 INFO:tasks.rados.rados.0.smithi045.stdout:2068: finishing write tid 4 to smithi04569269-2 2021-07-31T01:52:30.595 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 2 v 1037 (ObjNum 642 snap 199 seq_num 642) dirty exists 2021-07-31T01:52:30.595 INFO:tasks.rados.rados.0.smithi045.stdout:2068: left oid 2 (ObjNum 642 snap 199 seq_num 642) 2021-07-31T01:52:30.595 INFO:tasks.rados.rados.0.smithi045.stdout:2070: finishing copy_from to smithi04569269-42 2021-07-31T01:52:30.595 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 42 v 1001 (ObjNum 483 snap 143 seq_num 483) dirty exists 2021-07-31T01:52:30.596 INFO:tasks.rados.rados.0.smithi045.stdout:2070: finishing copy_from racing read to smithi04569269-42 2021-07-31T01:52:30.596 INFO:tasks.rados.rados.0.smithi045.stdout:2071: finishing write tid 1 to smithi04569269-30 2021-07-31T01:52:30.596 INFO:tasks.rados.rados.0.smithi045.stdout:2071: finishing write tid 2 to smithi04569269-30 2021-07-31T01:52:30.596 INFO:tasks.rados.rados.0.smithi045.stdout:2071: finishing write tid 3 to smithi04569269-30 2021-07-31T01:52:30.596 INFO:tasks.rados.rados.0.smithi045.stdout:2071: finishing write tid 4 to smithi04569269-30 2021-07-31T01:52:30.597 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 30 v 533 (ObjNum 643 snap 199 seq_num 643) dirty exists 2021-07-31T01:52:30.597 INFO:tasks.rados.rados.0.smithi045.stdout:2071: left oid 30 (ObjNum 643 snap 199 seq_num 643) 2021-07-31T01:52:30.597 INFO:tasks.rados.rados.0.smithi045.stdout:2073: seq_num 644 ranges {2138112=794624,2932736=401408} 2021-07-31T01:52:30.610 INFO:tasks.rados.rados.0.smithi045.stdout:2073: writing smithi04569269-24 from 2138112 to 2932736 tid 1 2021-07-31T01:52:30.613 INFO:tasks.rados.rados.0.smithi045.stdout:2073: writing smithi04569269-24 from 2932736 to 3334144 tid 2 2021-07-31T01:52:30.613 INFO:tasks.rados.rados.0.smithi045.stdout:2068: done (3 left) 2021-07-31T01:52:30.614 INFO:tasks.rados.rados.0.smithi045.stdout:2070: done (2 left) 2021-07-31T01:52:30.614 INFO:tasks.rados.rados.0.smithi045.stdout:2071: done (1 left) 2021-07-31T01:52:30.614 INFO:tasks.rados.rados.0.smithi045.stdout:2074: rollback oid 5 current snap is 199 2021-07-31T01:52:30.614 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 5 to 189 2021-07-31T01:52:30.615 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 37 current snap is 199 2021-07-31T01:52:30.615 INFO:tasks.rados.rados.0.smithi045.stdout:2075: seq_num 645 ranges {0=409600,409600=778240,1187840=24576} 2021-07-31T01:52:30.615 INFO:tasks.rados.rados.0.smithi045.stdout:2075: writing smithi04569269-37 from 0 to 409600 tid 1 2021-07-31T01:52:30.621 INFO:tasks.rados.rados.0.smithi045.stdout:2075: writing smithi04569269-37 from 409600 to 1187840 tid 2 2021-07-31T01:52:30.621 INFO:tasks.rados.rados.0.smithi045.stdout:2075: writing smithi04569269-37 from 1187840 to 1212416 tid 3 2021-07-31T01:52:30.621 INFO:tasks.rados.rados.0.smithi045.stdout:2076: snap_create 2021-07-31T01:52:30.621 INFO:tasks.rados.rados.0.smithi045.stdout:2073: finishing write tid 1 to smithi04569269-24 2021-07-31T01:52:30.622 INFO:tasks.rados.rados.0.smithi045.stdout:2073: finishing write tid 2 to smithi04569269-24 2021-07-31T01:52:30.622 INFO:tasks.rados.rados.0.smithi045.stdout:2074: finishing rollback tid 1 to smithi04569269-5 2021-07-31T01:52:30.622 INFO:tasks.rados.rados.0.smithi045.stdout:2074: finishing rollback tid 2 to smithi04569269-5 2021-07-31T01:52:30.622 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 5 v 1142 (ObjNum 619 snap 189 seq_num 619) dirty exists 2021-07-31T01:52:30.623 INFO:tasks.rados.rados.0.smithi045.stdout:2073: finishing write tid 3 to smithi04569269-24 2021-07-31T01:52:30.623 INFO:tasks.rados.rados.0.smithi045.stdout:2073: finishing write tid 4 to smithi04569269-24 2021-07-31T01:52:30.623 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 24 v 536 (ObjNum 644 snap 199 seq_num 644) dirty exists 2021-07-31T01:52:30.623 INFO:tasks.rados.rados.0.smithi045.stdout:2073: left oid 24 (ObjNum 644 snap 199 seq_num 644) 2021-07-31T01:52:30.623 INFO:tasks.rados.rados.0.smithi045.stdout:2075: finishing write tid 1 to smithi04569269-37 2021-07-31T01:52:30.624 INFO:tasks.rados.rados.0.smithi045.stdout:2075: finishing write tid 2 to smithi04569269-37 2021-07-31T01:52:30.625 INFO:tasks.rados.rados.0.smithi045.stdout:2075: finishing write tid 3 to smithi04569269-37 2021-07-31T01:52:30.625 INFO:tasks.rados.rados.0.smithi045.stdout:2075: finishing write tid 4 to smithi04569269-37 2021-07-31T01:52:30.625 INFO:tasks.rados.rados.0.smithi045.stdout:2075: finishing write tid 5 to smithi04569269-37 2021-07-31T01:52:30.625 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 37 v 527 (ObjNum 645 snap 199 seq_num 645) dirty exists 2021-07-31T01:52:30.626 INFO:tasks.rados.rados.0.smithi045.stdout:2075: left oid 37 (ObjNum 645 snap 199 seq_num 645) 2021-07-31T01:52:31.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:30 smithi154 conmon[34553]: cluster 2021-07-31T01:52:31.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:30 smithi154 conmon[34553]: 2021-07-31T01:52:29.583484 2021-07-31T01:52:31.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:30 smithi154 conmon[34553]: +0000 mon.a (mon.0) 1102 : cluster [DBG] osdmap e450: 8 total, 8 up, 8 in 2021-07-31T01:52:31.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:30 smithi154 conmon[34553]: cluster 2021-07-31T 2021-07-31T01:52:31.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:30 smithi154 conmon[34553]: 01:52:29.865121+0000 mgr.x (mgr.24542) 228 : cluster 2021-07-31T01:52:31.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:30 smithi154 conmon[34553]: [DBG] pgmap v455: 25 pgs: 25 active+clean; 195 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 3.6 MiB/s wr, 22 op/s 2021-07-31T01:52:31.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:30 smithi045 conmon[33655]: cluster 2021-07-31T01:52:29.583484+0000 mon.a (mon 2021-07-31T01:52:31.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:30 smithi045 conmon[33655]: .0) 1102 : cluster [DBG] osdmap e450: 8 total, 8 up, 8 in 2021-07-31T01:52:31.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:30 smithi045 conmon[33655]: cluster 2021-07-31T01:52:29.865121+0000 mgr.x (mgr.24542) 228 : cluster [DBG] pgmap v455: 25 pgs: 25 active+clean; 195 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 3.6 MiB/s wr, 22 op/s 2021-07-31T01:52:31.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:30 smithi045 conmon[39718]: cluster 2021-07-31T01:52: 2021-07-31T01:52:31.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:30 smithi045 conmon[39718]: 29.583484+0000 mon.a (mon.0) 1102 : cluster [DBG] osdmap e450: 8 total, 8 up, 8 in 2021-07-31T01:52:31.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:30 smithi045 conmon[39718]: cluster 2021-07 2021-07-31T01:52:31.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:30 smithi045 conmon[39718]: -31T01:52:29.865121+0000 mgr.x (mgr.24542) 228 : cluster 2021-07-31T01:52:31.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:30 smithi045 conmon[39718]: [DBG] pgmap v455: 25 pgs: 25 active+clean; 195 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 3.6 MiB/s wr, 22 op/s 2021-07-31T01:52:31.573 INFO:tasks.rados.rados.0.smithi045.stdout:2073: done (3 left) 2021-07-31T01:52:31.573 INFO:tasks.rados.rados.0.smithi045.stdout:2074: done (2 left) 2021-07-31T01:52:31.574 INFO:tasks.rados.rados.0.smithi045.stdout:2075: done (1 left) 2021-07-31T01:52:31.574 INFO:tasks.rados.rados.0.smithi045.stdout:2076: done (0 left) 2021-07-31T01:52:31.574 INFO:tasks.rados.rados.0.smithi045.stdout:2077: snap_remove snap 195 2021-07-31T01:52:32.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:31 smithi154 conmon[34553]: cluster 2021-07-31T01:52:30.593061+0000 mon.a (mon.0) 1103 : cluster [DBG] osdmap e451: 8 total, 8 up, 8 in 2021-07-31T01:52:32.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:31 smithi154 conmon[34553]: cluster 2021-07-31T01:52:31.572833+0000 mon.a (mon.0) 1104 : cluster [DBG] osdmap e452: 8 total, 8 up, 8 in 2021-07-31T01:52:32.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:31 smithi045 conmon[39718]: cluster 2021-07-31T01:52:30.593061+0000 mon.a (mon.0) 1103 : cluster [DBG] osdmap e451: 8 total, 8 up, 8 in 2021-07-31T01:52:32.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:31 smithi045 conmon[39718]: cluster 2021-07-31T01:52:31.572833+0000 2021-07-31T01:52:32.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:31 smithi045 conmon[39718]: mon.a (mon.0) 1104 : cluster [DBG] 2021-07-31T01:52:32.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:31 smithi045 conmon[39718]: osdmap e452: 8 total, 8 up, 8 in 2021-07-31T01:52:32.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:31 smithi045 conmon[33655]: cluster 2021-07-31T01:52:30.593061+0000 mon.a (mon.0) 1103 : 2021-07-31T01:52:32.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:31 smithi045 conmon[33655]: cluster [DBG] osdmap e451: 8 total, 8 up, 8 in 2021-07-31T01:52:32.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:31 smithi045 conmon[33655]: cluster 2021-07-31T01:52:31.572833+0000 mon.a (mon.0) 1104 : cluster [DBG] osdmap e452: 8 total, 8 up, 8 in 2021-07-31T01:52:32.580 INFO:tasks.rados.rados.0.smithi045.stdout:2077: done (0 left) 2021-07-31T01:52:32.581 INFO:tasks.rados.rados.0.smithi045.stdout:2078: read oid 16 snap -1 2021-07-31T01:52:32.581 INFO:tasks.rados.rados.0.smithi045.stdout:2078: expect deleted 2021-07-31T01:52:32.582 INFO:tasks.rados.rados.0.smithi045.stdout:2079: read oid 15 snap -1 2021-07-31T01:52:32.582 INFO:tasks.rados.rados.0.smithi045.stdout:2079: expect (ObjNum 622 snap 191 seq_num 622) 2021-07-31T01:52:32.582 INFO:tasks.rados.rados.0.smithi045.stdout:2080: snap_create 2021-07-31T01:52:32.606 INFO:tasks.rados.rados.0.smithi045.stdout:2079: expect (ObjNum 622 snap 191 seq_num 622) 2021-07-31T01:52:33.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:32 smithi154 conmon[34553]: cluster 2021-07-31T01:52:31.865583+0000 mgr.x (mgr.24542) 229 : cluster 2021-07-31T01:52:33.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:32 smithi154 conmon[34553]: [DBG] pgmap v458: 25 pgs: 25 active+clean; 195 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 3.6 MiB/s wr, 22 op/s 2021-07-31T01:52:33.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:32 smithi154 conmon[34553]: cluster 2021-07-31T01:52:32.579372+0000 mon.a (mon.0) 1105 : cluster [DBG] osdmap e453: 8 total, 8 up, 8 in 2021-07-31T01:52:33.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:32 smithi045 conmon[39718]: cluster 2021-07-31T01:52:31.865583+0000 2021-07-31T01:52:33.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:32 smithi045 conmon[39718]: mgr.x (mgr.24542) 229 : cluster [DBG] pgmap v458: 25 pgs: 25 active+clean; 195 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 3.6 MiB/s wr, 22 op/s 2021-07-31T01:52:33.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:32 smithi045 conmon[39718]: cluster 2021-07-31T01:52:32.579372+0000 mon.a (mon.0) 1105 : cluster [DBG] osdmap e453: 8 total, 8 up, 8 in 2021-07-31T01:52:33.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:32 smithi045 conmon[33655]: cluster 2021-07-31T01:52:33.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:32 smithi045 conmon[33655]: 2021-07-31T01:52 2021-07-31T01:52:33.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:32 smithi045 conmon[33655]: :31.865583+0000 mgr.x (mgr.24542) 229 : cluster [DBG] pgmap v458: 25 pgs: 25 active+clean; 195 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 3.6 MiB/s wr, 22 op/s 2021-07-31T01:52:33.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:32 smithi045 conmon[33655]: cluster 2021-07-31T01:52:33.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:32 smithi045 conmon[33655]: 2021-07-31T01:52:32.579372+0000 mon.a 2021-07-31T01:52:33.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:32 smithi045 conmon[33655]: (mon.0) 1105 : cluster [DBG] osdmap e453: 8 total, 8 up, 8 in 2021-07-31T01:52:33.580 INFO:tasks.rados.rados.0.smithi045.stdout:2078: done (2 left) 2021-07-31T01:52:33.581 INFO:tasks.rados.rados.0.smithi045.stdout:2079: done (1 left) 2021-07-31T01:52:33.581 INFO:tasks.rados.rados.0.smithi045.stdout:2080: done (0 left) 2021-07-31T01:52:33.581 INFO:tasks.rados.rados.0.smithi045.stdout:2081: setattr oid 33 current snap is 201 2021-07-31T01:52:33.582 INFO:tasks.rados.rados.0.smithi045.stdout:2082: delete oid 35 current snap is 201 2021-07-31T01:52:33.586 INFO:tasks.rados.rados.0.smithi045.stdout:2082: done (1 left) 2021-07-31T01:52:33.586 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 49 current snap is 201 2021-07-31T01:52:33.587 INFO:tasks.rados.rados.0.smithi045.stdout:2083: seq_num 647 ranges {5455872=442368,5898240=589824,6488064=212992} 2021-07-31T01:52:33.618 INFO:tasks.rados.rados.0.smithi045.stdout:2083: writing smithi04569269-49 from 5455872 to 5898240 tid 1 2021-07-31T01:52:33.621 INFO:tasks.rados.rados.0.smithi045.stdout:2083: writing smithi04569269-49 from 5898240 to 6488064 tid 2 2021-07-31T01:52:33.623 INFO:tasks.rados.rados.0.smithi045.stdout:2083: writing smithi04569269-49 from 6488064 to 6701056 tid 3 2021-07-31T01:52:33.623 INFO:tasks.rados.rados.0.smithi045.stdout:2084: snap_create 2021-07-31T01:52:33.623 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 33 v 1143 (ObjNum 399 snap 121 seq_num 399) dirty exists 2021-07-31T01:52:33.623 INFO:tasks.rados.rados.0.smithi045.stdout:2083: finishing write tid 1 to smithi04569269-49 2021-07-31T01:52:33.627 INFO:tasks.rados.rados.0.smithi045.stdout:2083: finishing write tid 2 to smithi04569269-49 2021-07-31T01:52:33.627 INFO:tasks.rados.rados.0.smithi045.stdout:2083: finishing write tid 3 to smithi04569269-49 2021-07-31T01:52:33.627 INFO:tasks.rados.rados.0.smithi045.stdout:2083: finishing write tid 4 to smithi04569269-49 2021-07-31T01:52:33.628 INFO:tasks.rados.rados.0.smithi045.stdout:2083: finishing write tid 5 to smithi04569269-49 2021-07-31T01:52:33.628 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 49 v 1317 (ObjNum 647 snap 201 seq_num 647) dirty exists 2021-07-31T01:52:33.629 INFO:tasks.rados.rados.0.smithi045.stdout:2083: left oid 49 (ObjNum 647 snap 201 seq_num 647) 2021-07-31T01:52:33.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:52:33 smithi045 conmon[61210]: level=error ts=2021-07-31T01:52:33.517Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:52:33.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:52:33 smithi045 conmon[61210]: level=error ts=2021-07-31T01:52:33.518Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:52:33.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:52:33 smithi045 conmon[61210]: level=error ts=2021-07-31T01:52:33.518Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T01:52:34.469 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T01:52:34.585 INFO:tasks.rados.rados.0.smithi045.stdout:2081: done (2 left) 2021-07-31T01:52:34.586 INFO:tasks.rados.rados.0.smithi045.stdout:2083: done (1 left) 2021-07-31T01:52:34.586 INFO:tasks.rados.rados.0.smithi045.stdout:2084: done (0 left) 2021-07-31T01:52:34.586 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 32 current snap is 202 2021-07-31T01:52:34.586 INFO:tasks.rados.rados.0.smithi045.stdout:2085: seq_num 648 ranges {2228224=458752,2686976=475136,3162112=327680} 2021-07-31T01:52:34.600 INFO:tasks.rados.rados.0.smithi045.stdout:2085: writing smithi04569269-32 from 2228224 to 2686976 tid 1 2021-07-31T01:52:34.602 INFO:tasks.rados.rados.0.smithi045.stdout:2085: writing smithi04569269-32 from 2686976 to 3162112 tid 2 2021-07-31T01:52:34.605 INFO:tasks.rados.rados.0.smithi045.stdout:2085: writing smithi04569269-32 from 3162112 to 3489792 tid 3 2021-07-31T01:52:34.605 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 12 current snap is 202 2021-07-31T01:52:34.605 INFO:tasks.rados.rados.0.smithi045.stdout:2086: seq_num 649 ranges {2367488=475136,2842624=761856,3604480=40960} 2021-07-31T01:52:34.620 INFO:tasks.rados.rados.0.smithi045.stdout:2086: writing smithi04569269-12 from 2367488 to 2842624 tid 1 2021-07-31T01:52:34.624 INFO:tasks.rados.rados.0.smithi045.stdout:2086: writing smithi04569269-12 from 2842624 to 3604480 tid 2 2021-07-31T01:52:34.625 INFO:tasks.rados.rados.0.smithi045.stdout:2086: writing smithi04569269-12 from 3604480 to 3645440 tid 3 2021-07-31T01:52:34.626 INFO:tasks.rados.rados.0.smithi045.stdout:2087: rmattr oid 2 current snap is 202 2021-07-31T01:52:34.626 INFO:tasks.rados.rados.0.smithi045.stdout:2085: finishing write tid 1 to smithi04569269-32 2021-07-31T01:52:34.626 INFO:tasks.rados.rados.0.smithi045.stdout:2085: finishing write tid 2 to smithi04569269-32 2021-07-31T01:52:34.626 INFO:tasks.rados.rados.0.smithi045.stdout:2085: finishing write tid 3 to smithi04569269-32 2021-07-31T01:52:34.627 INFO:tasks.rados.rados.0.smithi045.stdout:2085: finishing write tid 4 to smithi04569269-32 2021-07-31T01:52:34.627 INFO:tasks.rados.rados.0.smithi045.stdout:2085: finishing write tid 5 to smithi04569269-32 2021-07-31T01:52:34.627 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 32 v 1037 (ObjNum 648 snap 202 seq_num 648) dirty exists 2021-07-31T01:52:34.627 INFO:tasks.rados.rados.0.smithi045.stdout:2085: left oid 32 (ObjNum 648 snap 202 seq_num 648) 2021-07-31T01:52:34.627 INFO:tasks.rados.rados.0.smithi045.stdout:2086: finishing write tid 1 to smithi04569269-12 2021-07-31T01:52:34.628 INFO:tasks.rados.rados.0.smithi045.stdout:2085: done (2 left) 2021-07-31T01:52:34.628 INFO:tasks.rados.rados.0.smithi045.stdout:2088: rollback oid 46 current snap is 202 2021-07-31T01:52:34.628 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 46 to 197 2021-07-31T01:52:34.628 INFO:tasks.rados.rados.0.smithi045.stdout:2089: delete oid 16 current snap is 202 2021-07-31T01:52:34.631 INFO:tasks.rados.rados.0.smithi045.stdout:2089: done (3 left) 2021-07-31T01:52:34.631 INFO:tasks.rados.rados.0.smithi045.stdout:2090: setattr oid 49 current snap is 202 2021-07-31T01:52:34.631 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 2 v 1038 (ObjNum 642 snap 199 seq_num 642) dirty exists 2021-07-31T01:52:34.631 INFO:tasks.rados.rados.0.smithi045.stdout:2087: done (3 left) 2021-07-31T01:52:34.632 INFO:tasks.rados.rados.0.smithi045.stdout:2091: rmattr oid 41 current snap is 202 2021-07-31T01:52:34.632 INFO:tasks.rados.rados.0.smithi045.stdout:2092: read oid 36 snap -1 2021-07-31T01:52:34.632 INFO:tasks.rados.rados.0.smithi045.stdout:2092: expect (ObjNum 629 snap 192 seq_num 629) 2021-07-31T01:52:34.632 INFO:tasks.rados.rados.0.smithi045.stdout:2093: copy_from oid 18 from oid 24 current snap is 202 2021-07-31T01:52:34.633 INFO:tasks.rados.rados.0.smithi045.stdout:2094: setattr oid 15 current snap is 202 2021-07-31T01:52:34.635 INFO:tasks.rados.rados.0.smithi045.stdout:2095: rollback oid 2 current snap is 202 2021-07-31T01:52:34.635 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 2 to 200 2021-07-31T01:52:34.635 INFO:tasks.rados.rados.0.smithi045.stdout:2096: rollback oid 8 current snap is 202 2021-07-31T01:52:34.636 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 8 to 186 2021-07-31T01:52:34.636 INFO:tasks.rados.rados.0.smithi045.stdout:2097: copy_from oid 48 from oid 16 current snap is 202 2021-07-31T01:52:34.636 INFO:tasks.rados.rados.0.smithi045.stdout:2098: read oid 20 snap 199 2021-07-31T01:52:34.636 INFO:tasks.rados.rados.0.smithi045.stdout:2098: expect (ObjNum 556 snap 166 seq_num 556) 2021-07-31T01:52:34.636 INFO:tasks.rados.rados.0.smithi045.stdout:2099: delete oid 47 current snap is 202 2021-07-31T01:52:34.637 INFO:tasks.rados.rados.0.smithi045.stdout:2088: finishing rollback tid 0 to smithi04569269-46 2021-07-31T01:52:34.637 INFO:tasks.rados.rados.0.smithi045.stdout:2088: finishing rollback tid 1 to smithi04569269-46 2021-07-31T01:52:34.637 INFO:tasks.rados.rados.0.smithi045.stdout:2088: finishing rollback tid 2 to smithi04569269-46 2021-07-31T01:52:34.637 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 46 v 1293 (ObjNum 470 snap 136 seq_num 470) dirty exists 2021-07-31T01:52:34.637 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 41 v 984 (ObjNum 639 snap 199 seq_num 639) dirty exists 2021-07-31T01:52:34.638 INFO:tasks.rados.rados.0.smithi045.stdout:2095: finishing rollback tid 0 to smithi04569269-2 2021-07-31T01:52:34.638 INFO:tasks.rados.rados.0.smithi045.stdout:2095: finishing rollback tid 1 to smithi04569269-2 2021-07-31T01:52:34.638 INFO:tasks.rados.rados.0.smithi045.stdout:2088: done (11 left) 2021-07-31T01:52:34.638 INFO:tasks.rados.rados.0.smithi045.stdout:2091: done (10 left) 2021-07-31T01:52:34.639 INFO:tasks.rados.rados.0.smithi045.stdout:2099: done (9 left) 2021-07-31T01:52:34.639 INFO:tasks.rados.rados.0.smithi045.stdout:2100: rmattr oid 5 current snap is 202 2021-07-31T01:52:34.639 INFO:tasks.rados.rados.0.smithi045.stdout:2100: done (9 left) 2021-07-31T01:52:34.639 INFO:tasks.rados.rados.0.smithi045.stdout:2101: delete oid 1 current snap is 202 2021-07-31T01:52:34.639 INFO:tasks.rados.rados.0.smithi045.stdout:2095: finishing rollback tid 2 to smithi04569269-2 2021-07-31T01:52:34.640 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 2 v 1041 (ObjNum 642 snap 199 seq_num 642) dirty exists 2021-07-31T01:52:34.641 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 49 v 1319 (ObjNum 647 snap 201 seq_num 647) dirty exists 2021-07-31T01:52:34.641 INFO:tasks.rados.rados.0.smithi045.stdout:2086: finishing write tid 2 to smithi04569269-12 2021-07-31T01:52:34.642 INFO:tasks.rados.rados.0.smithi045.stdout:2086: finishing write tid 3 to smithi04569269-12 2021-07-31T01:52:34.642 INFO:tasks.rados.rados.0.smithi045.stdout:2086: finishing write tid 4 to smithi04569269-12 2021-07-31T01:52:34.642 INFO:tasks.rados.rados.0.smithi045.stdout:2086: finishing write tid 5 to smithi04569269-12 2021-07-31T01:52:34.642 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 12 v 782 (ObjNum 649 snap 202 seq_num 649) dirty exists 2021-07-31T01:52:34.643 INFO:tasks.rados.rados.0.smithi045.stdout:2086: left oid 12 (ObjNum 649 snap 202 seq_num 649) 2021-07-31T01:52:34.643 INFO:tasks.rados.rados.0.smithi045.stdout:2097: finishing copy_from to smithi04569269-48 2021-07-31T01:52:34.643 INFO:tasks.rados.rados.0.smithi045.stdout:2097: got expected ENOENT (src dne) 2021-07-31T01:52:34.643 INFO:tasks.rados.rados.0.smithi045.stdout:2097: finishing copy_from racing read to smithi04569269-48 2021-07-31T01:52:34.644 INFO:tasks.rados.rados.0.smithi045.stdout:2092: expect (ObjNum 629 snap 192 seq_num 629) 2021-07-31T01:52:34.669 INFO:tasks.rados.rados.0.smithi045.stdout:2096: finishing rollback tid 0 to smithi04569269-8 2021-07-31T01:52:34.670 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 15 v 531 (ObjNum 622 snap 191 seq_num 622) dirty exists 2021-07-31T01:52:34.670 INFO:tasks.rados.rados.0.smithi045.stdout:2086: done (9 left) 2021-07-31T01:52:34.670 INFO:tasks.rados.rados.0.smithi045.stdout:2090: done (8 left) 2021-07-31T01:52:34.671 INFO:tasks.rados.rados.0.smithi045.stdout:2092: done (7 left) 2021-07-31T01:52:34.671 INFO:tasks.rados.rados.0.smithi045.stdout:2094: done (6 left) 2021-07-31T01:52:34.671 INFO:tasks.rados.rados.0.smithi045.stdout:2095: done (5 left) 2021-07-31T01:52:34.671 INFO:tasks.rados.rados.0.smithi045.stdout:2097: done (4 left) 2021-07-31T01:52:34.672 INFO:tasks.rados.rados.0.smithi045.stdout:2101: done (3 left) 2021-07-31T01:52:34.672 INFO:tasks.rados.rados.0.smithi045.stdout:2102: copy_from oid 1 from oid 42 current snap is 202 2021-07-31T01:52:34.672 INFO:tasks.rados.rados.0.smithi045.stdout:2093: finishing copy_from to smithi04569269-18 2021-07-31T01:52:34.672 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 18 v 536 (ObjNum 644 snap 199 seq_num 644) dirty exists 2021-07-31T01:52:34.673 INFO:tasks.rados.rados.0.smithi045.stdout:2093: finishing copy_from racing read to smithi04569269-18 2021-07-31T01:52:34.673 INFO:tasks.rados.rados.0.smithi045.stdout:2098: expect (ObjNum 556 snap 166 seq_num 556) 2021-07-31T01:52:34.763 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T01:52:34.763 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 7m ago 8m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T01:52:34.763 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 5m ago 7m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T01:52:34.763 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 5m ago 11m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T01:52:34.764 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 7m ago 13m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T01:52:34.764 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 7m ago 13m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T01:52:34.764 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 5m ago 11m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T01:52:34.764 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 7m ago 11m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T01:52:34.764 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 7m ago 8m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T01:52:34.765 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 5m ago 8m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T01:52:34.765 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 7m ago 11m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T01:52:34.765 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 7m ago 10m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T01:52:34.765 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 7m ago 10m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T01:52:34.766 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 7m ago 10m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T01:52:34.766 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 5m ago 9m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T01:52:34.766 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 5m ago 9m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T01:52:34.766 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 5m ago 9m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T01:52:34.767 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 5m ago 8m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T01:52:34.770 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 5m ago 8m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T01:52:34.786 INFO:tasks.rados.rados.0.smithi045.stdout:2096: finishing rollback tid 1 to smithi04569269-8 2021-07-31T01:52:34.786 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 8 v 1322 (ObjNum 0 snap 0 seq_num 0) dirty dne 2021-07-31T01:52:34.786 INFO:tasks.rados.rados.0.smithi045.stdout:2102: finishing copy_from to smithi04569269-1 2021-07-31T01:52:34.787 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 1 v 1001 (ObjNum 483 snap 143 seq_num 483) dirty exists 2021-07-31T01:52:34.787 INFO:tasks.rados.rados.0.smithi045.stdout:2102: finishing copy_from racing read to smithi04569269-1 2021-07-31T01:52:34.787 INFO:tasks.rados.rados.0.smithi045.stdout:2093: done (3 left) 2021-07-31T01:52:34.787 INFO:tasks.rados.rados.0.smithi045.stdout:2096: done (2 left) 2021-07-31T01:52:34.788 INFO:tasks.rados.rados.0.smithi045.stdout:2098: done (1 left) 2021-07-31T01:52:34.788 INFO:tasks.rados.rados.0.smithi045.stdout:2102: done (0 left) 2021-07-31T01:52:34.788 INFO:tasks.rados.rados.0.smithi045.stdout:2103: read oid 6 snap 199 2021-07-31T01:52:34.788 INFO:tasks.rados.rados.0.smithi045.stdout:2103: expect (ObjNum 635 snap 198 seq_num 635) 2021-07-31T01:52:34.789 INFO:tasks.rados.rados.0.smithi045.stdout:2104: read oid 35 snap -1 2021-07-31T01:52:34.789 INFO:tasks.rados.rados.0.smithi045.stdout:2104: expect deleted 2021-07-31T01:52:34.789 INFO:tasks.rados.rados.0.smithi045.stdout:2105: snap_create 2021-07-31T01:52:34.798 INFO:tasks.rados.rados.0.smithi045.stdout:2103: expect (ObjNum 635 snap 198 seq_num 635) 2021-07-31T01:52:35.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:34 smithi154 conmon[34553]: cluster 2021-07-31T01:52:33.580295+0000 mon.a (mon.0) 1106 : cluster [DBG] osdmap e454: 8 total, 8 up, 8 in 2021-07-31T01:52:35.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:34 smithi154 conmon[34553]: cluster 2021-07-31T01:52:33.866059+0000 mgr.x (mgr. 2021-07-31T01:52:35.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:34 smithi154 conmon[34553]: 24542) 230 : cluster [DBG] pgmap v461: 25 pgs: 25 active+clean; 206 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 2.6 MiB/s wr, 22 op/s 2021-07-31T01:52:35.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:34 smithi045 conmon[33655]: cluster 2021-07-31T01:52:33.580295+0000 mon.a (mon 2021-07-31T01:52:35.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:34 smithi045 conmon[33655]: .0) 1106 : cluster [DBG] osdmap e454: 8 total, 8 up, 8 in 2021-07-31T01:52:35.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:34 smithi045 conmon[33655]: cluster 2021-07-31T01:52:33.866059+0000 mgr.x ( 2021-07-31T01:52:35.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:34 smithi045 conmon[33655]: mgr.24542) 230 : cluster [DBG] pgmap v461: 25 pgs: 25 active+clean; 206 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 2.6 MiB/s wr, 22 op/s 2021-07-31T01:52:35.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:34 smithi045 conmon[39718]: cluster 2021-07-31T01:52:33.580295+0000 mon.a (mon 2021-07-31T01:52:35.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:34 smithi045 conmon[39718]: .0) 1106 : cluster [DBG] osdmap e454: 8 total, 8 up, 8 in 2021-07-31T01:52:35.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:34 smithi045 conmon[39718]: cluster 2021- 2021-07-31T01:52:35.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:34 smithi045 conmon[39718]: 07-31T01:52:33.866059+0000 mgr.x (mgr.24542) 230 : cluster 2021-07-31T01:52:35.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:34 smithi045 conmon[39718]: [DBG] pgmap v461: 25 pgs: 25 active+clean; 206 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 2.6 MiB/s wr, 22 op/s 2021-07-31T01:52:35.113 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T01:52:35.113 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T01:52:35.113 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T01:52:35.114 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T01:52:35.114 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T01:52:35.114 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T01:52:35.114 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T01:52:35.115 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T01:52:35.115 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T01:52:35.115 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T01:52:35.115 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T01:52:35.115 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T01:52:35.116 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T01:52:35.116 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T01:52:35.116 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T01:52:35.116 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T01:52:35.117 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T01:52:35.591 INFO:tasks.rados.rados.0.smithi045.stdout:2103: done (2 left) 2021-07-31T01:52:35.591 INFO:tasks.rados.rados.0.smithi045.stdout:2104: done (1 left) 2021-07-31T01:52:35.592 INFO:tasks.rados.rados.0.smithi045.stdout:2105: done (0 left) 2021-07-31T01:52:35.592 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 24 current snap is 203 2021-07-31T01:52:35.592 INFO:tasks.rados.rados.0.smithi045.stdout:2106: seq_num 653 ranges {3334144=540672,3874816=696320,4571136=106496} 2021-07-31T01:52:35.612 INFO:tasks.rados.rados.0.smithi045.stdout:2106: writing smithi04569269-24 from 3334144 to 3874816 tid 1 2021-07-31T01:52:35.616 INFO:tasks.rados.rados.0.smithi045.stdout:2106: writing smithi04569269-24 from 3874816 to 4571136 tid 2 2021-07-31T01:52:35.617 INFO:tasks.rados.rados.0.smithi045.stdout:2106: writing smithi04569269-24 from 4571136 to 4677632 tid 3 2021-07-31T01:52:35.618 INFO:tasks.rados.rados.0.smithi045.stdout:2107: delete oid 9 current snap is 203 2021-07-31T01:52:35.618 INFO:tasks.rados.rados.0.smithi045.stdout:2106: finishing write tid 1 to smithi04569269-24 2021-07-31T01:52:35.619 INFO:tasks.rados.rados.0.smithi045.stdout:2106: finishing write tid 2 to smithi04569269-24 2021-07-31T01:52:35.621 INFO:tasks.rados.rados.0.smithi045.stdout:2106: finishing write tid 3 to smithi04569269-24 2021-07-31T01:52:35.621 INFO:tasks.rados.rados.0.smithi045.stdout:2107: done (1 left) 2021-07-31T01:52:35.622 INFO:tasks.rados.rados.0.smithi045.stdout:2108: read oid 27 snap -1 2021-07-31T01:52:35.622 INFO:tasks.rados.rados.0.smithi045.stdout:2108: expect (ObjNum 601 snap 184 seq_num 601) 2021-07-31T01:52:35.622 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 12 current snap is 203 2021-07-31T01:52:35.622 INFO:tasks.rados.rados.0.smithi045.stdout:2109: seq_num 655 ranges {3645440=573440,4218880=516096,4734976=286720} 2021-07-31T01:52:35.644 INFO:tasks.rados.rados.0.smithi045.stdout:2109: writing smithi04569269-12 from 3645440 to 4218880 tid 1 2021-07-31T01:52:35.647 INFO:tasks.rados.rados.0.smithi045.stdout:2109: writing smithi04569269-12 from 4218880 to 4734976 tid 2 2021-07-31T01:52:35.651 INFO:tasks.rados.rados.0.smithi045.stdout:2109: writing smithi04569269-12 from 4734976 to 5021696 tid 3 2021-07-31T01:52:35.651 INFO:tasks.rados.rados.0.smithi045.stdout:2110: rollback oid 30 current snap is 203 2021-07-31T01:52:35.651 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 30 to 198 2021-07-31T01:52:35.652 INFO:tasks.rados.rados.0.smithi045.stdout:2106: finishing write tid 4 to smithi04569269-24 2021-07-31T01:52:35.652 INFO:tasks.rados.rados.0.smithi045.stdout:2106: finishing write tid 5 to smithi04569269-24 2021-07-31T01:52:35.652 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 24 v 540 (ObjNum 653 snap 203 seq_num 653) dirty exists 2021-07-31T01:52:35.652 INFO:tasks.rados.rados.0.smithi045.stdout:2106: left oid 24 (ObjNum 653 snap 203 seq_num 653) 2021-07-31T01:52:35.652 INFO:tasks.rados.rados.0.smithi045.stdout:2108: expect (ObjNum 601 snap 184 seq_num 601) 2021-07-31T01:52:35.664 INFO:tasks.rados.rados.0.smithi045.stdout:2109: finishing write tid 1 to smithi04569269-12 2021-07-31T01:52:35.664 INFO:tasks.rados.rados.0.smithi045.stdout:2110: finishing rollback tid 0 to smithi04569269-30 2021-07-31T01:52:35.664 INFO:tasks.rados.rados.0.smithi045.stdout:2109: finishing write tid 2 to smithi04569269-12 2021-07-31T01:52:35.665 INFO:tasks.rados.rados.0.smithi045.stdout:2110: finishing rollback tid 1 to smithi04569269-30 2021-07-31T01:52:35.665 INFO:tasks.rados.rados.0.smithi045.stdout:2109: finishing write tid 3 to smithi04569269-12 2021-07-31T01:52:35.665 INFO:tasks.rados.rados.0.smithi045.stdout:2110: finishing rollback tid 2 to smithi04569269-30 2021-07-31T01:52:35.665 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 30 v 543 (ObjNum 1 snap 32731 seq_num 3190566792) dirty exists 2021-07-31T01:52:35.666 INFO:tasks.rados.rados.0.smithi045.stdout:2109: finishing write tid 4 to smithi04569269-12 2021-07-31T01:52:35.666 INFO:tasks.rados.rados.0.smithi045.stdout:2109: finishing write tid 5 to smithi04569269-12 2021-07-31T01:52:35.666 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 12 v 787 (ObjNum 655 snap 203 seq_num 655) dirty exists 2021-07-31T01:52:35.666 INFO:tasks.rados.rados.0.smithi045.stdout:2109: left oid 12 (ObjNum 655 snap 203 seq_num 655) 2021-07-31T01:52:35.667 INFO:tasks.rados.rados.0.smithi045.stdout:2106: done (3 left) 2021-07-31T01:52:35.667 INFO:tasks.rados.rados.0.smithi045.stdout:2108: done (2 left) 2021-07-31T01:52:35.667 INFO:tasks.rados.rados.0.smithi045.stdout:2109: done (1 left) 2021-07-31T01:52:35.668 INFO:tasks.rados.rados.0.smithi045.stdout:2110: done (0 left) 2021-07-31T01:52:35.668 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 23 current snap is 203 2021-07-31T01:52:35.668 INFO:tasks.rados.rados.0.smithi045.stdout:2111: seq_num 656 ranges {0=589824,589824=802816} 2021-07-31T01:52:35.669 INFO:tasks.rados.rados.0.smithi045.stdout:2111: writing smithi04569269-23 from 0 to 589824 tid 1 2021-07-31T01:52:35.670 INFO:tasks.rados.rados.0.smithi045.stdout:2111: writing smithi04569269-23 from 589824 to 1392640 tid 2 2021-07-31T01:52:35.671 INFO:tasks.rados.rados.0.smithi045.stdout:2112: copy_from oid 31 from oid 42 current snap is 203 2021-07-31T01:52:35.671 INFO:tasks.rados.rados.0.smithi045.stdout:2111: finishing write tid 1 to smithi04569269-23 2021-07-31T01:52:35.671 INFO:tasks.rados.rados.0.smithi045.stdout:2113: rollback oid 12 current snap is 203 2021-07-31T01:52:35.671 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 12 to 186 2021-07-31T01:52:35.672 INFO:tasks.rados.rados.0.smithi045.stdout:2114: delete oid 38 current snap is 203 2021-07-31T01:52:35.675 INFO:tasks.rados.rados.0.smithi045.stdout:2113: finishing rollback tid 0 to smithi04569269-12 2021-07-31T01:52:35.676 INFO:tasks.rados.rados.0.smithi045.stdout:2111: finishing write tid 2 to smithi04569269-23 2021-07-31T01:52:35.676 INFO:tasks.rados.rados.0.smithi045.stdout:2111: finishing write tid 3 to smithi04569269-23 2021-07-31T01:52:35.678 INFO:tasks.rados.rados.0.smithi045.stdout:2113: finishing rollback tid 1 to smithi04569269-12 2021-07-31T01:52:35.678 INFO:tasks.rados.rados.0.smithi045.stdout:2113: finishing rollback tid 2 to smithi04569269-12 2021-07-31T01:52:35.678 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 12 v 790 (ObjNum 506 snap 152 seq_num 506) dirty exists 2021-07-31T01:52:35.679 INFO:tasks.rados.rados.0.smithi045.stdout:2113: done (3 left) 2021-07-31T01:52:35.679 INFO:tasks.rados.rados.0.smithi045.stdout:2114: done (2 left) 2021-07-31T01:52:35.679 INFO:tasks.rados.rados.0.smithi045.stdout:2115: read oid 29 snap 200 2021-07-31T01:52:35.679 INFO:tasks.rados.rados.0.smithi045.stdout:2115: expect (ObjNum 539 snap 162 seq_num 539) 2021-07-31T01:52:35.680 INFO:tasks.rados.rados.0.smithi045.stdout:2116: delete oid 18 current snap is 203 2021-07-31T01:52:35.680 INFO:tasks.rados.rados.0.smithi045.stdout:2111: finishing write tid 4 to smithi04569269-23 2021-07-31T01:52:35.680 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 23 v 1146 (ObjNum 656 snap 203 seq_num 656) dirty exists 2021-07-31T01:52:35.681 INFO:tasks.rados.rados.0.smithi045.stdout:2111: left oid 23 (ObjNum 656 snap 203 seq_num 656) 2021-07-31T01:52:35.681 INFO:tasks.rados.rados.0.smithi045.stdout:2111: done (3 left) 2021-07-31T01:52:35.681 INFO:tasks.rados.rados.0.smithi045.stdout:2116: done (2 left) 2021-07-31T01:52:35.682 INFO:tasks.rados.rados.0.smithi045.stdout:2117: snap_remove snap 202 2021-07-31T01:52:36.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:35 smithi154 conmon[34553]: audit 2021-07-31T01:52:34.454986+0000 mgr.x (mgr.24542) 231 : audit [DBG] from='client.14892 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:52:36.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:35 smithi154 conmon[34553]: cluster 2021-07-31T01:52:34.584719+0000 mon.a (mon.0) 1107 : cluster [DBG] osdmap e455: 8 total, 8 up, 8 in 2021-07-31T01:52:36.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:35 smithi154 conmon[34553]: audit 2021-07-31T01:52:34.754797+0000 mgr.x (mgr.24542) 232 : audit [DBG] from='client.14898 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:52:36.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:35 smithi154 conmon[34553]: audit 2021-07-31T01:52:35.112771+0000 mon.a (mon.0) 1108 : audit 2021-07-31T01:52:36.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:35 smithi154 conmon[34553]: [DBG] from='client.? 172.21.15.45:0/530520841' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T01:52:36.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:35 smithi045 conmon[33655]: audit 2021-07-31T01:52:34.454986+0000 2021-07-31T01:52:36.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:35 smithi045 conmon[33655]: mgr.x (mgr.24542) 231 : audit [DBG] from='client.14892 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:52:36.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:35 smithi045 conmon[33655]: cluster 2021-07-31T01:52:34.584719+0000 mon.a (mon.0) 1107 : cluster 2021-07-31T01:52:36.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:35 smithi045 conmon[33655]: [DBG] osdmap e455: 8 total, 8 up, 8 in 2021-07-31T01:52:36.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:35 smithi045 conmon[33655]: audit 2021-07-31T01:52:34 2021-07-31T01:52:36.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:35 smithi045 conmon[33655]: .754797+0000 mgr.x (mgr.24542) 232 : audit [DBG] from='client.14898 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:52:36.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:35 smithi045 conmon[33655]: audit 2021-07-31 2021-07-31T01:52:36.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:35 smithi045 conmon[33655]: T01:52:35.112771+0000 mon.a (mon. 2021-07-31T01:52:36.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:35 smithi045 conmon[33655]: 0) 1108 : audit [DBG] from='client.? 172.21.15.45:0/530520841' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T01:52:36.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:35 smithi045 conmon[39718]: audit 2021-07-31T01:52:34 2021-07-31T01:52:36.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:35 smithi045 conmon[39718]: .454986+0000 mgr.x (mgr.24542) 231 : audit [DBG] from='client.14892 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:52:36.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:35 smithi045 conmon[39718]: cluster 2021-07 2021-07-31T01:52:36.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:35 smithi045 conmon[39718]: -31T01:52:34. 2021-07-31T01:52:36.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:35 smithi045 conmon[39718]: 584719+0000 mon.a (mon. 2021-07-31T01:52:36.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:35 smithi045 conmon[39718]: 0) 1107 : cluster [DBG] 2021-07-31T01:52:36.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:35 smithi045 conmon[39718]: osdmap e455: 8 total, 8 up, 8 in 2021-07-31T01:52:36.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:35 smithi045 conmon[39718]: audit 2021- 2021-07-31T01:52:36.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:35 smithi045 conmon[39718]: 07-31T01:52:34.754797 2021-07-31T01:52:36.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:35 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 2021-07-31T01:52:36.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:35 smithi045 conmon[39718]: 232 : audit [DBG] from='client.14898 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:52:36.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:35 smithi045 conmon[39718]: audit 2021 2021-07-31T01:52:36.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:35 smithi045 conmon[39718]: -07-31T01:52:35 2021-07-31T01:52:36.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:35 smithi045 conmon[39718]: .112771+0000 mon.a (mon. 2021-07-31T01:52:36.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:35 smithi045 conmon[39718]: 0) 1108 : audit [DBG] 2021-07-31T01:52:36.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:35 smithi045 conmon[39718]: from='client.? 172.21.15.45:0/530520841' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T01:52:36.575 INFO:tasks.rados.rados.0.smithi045.stdout:2117: done (2 left) 2021-07-31T01:52:36.575 INFO:tasks.rados.rados.0.smithi045.stdout:2118: read oid 1 snap -1 2021-07-31T01:52:36.576 INFO:tasks.rados.rados.0.smithi045.stdout:2118: expect (ObjNum 483 snap 143 seq_num 483) 2021-07-31T01:52:36.576 INFO:tasks.rados.rados.0.smithi045.stdout:2119: read oid 2 snap -1 2021-07-31T01:52:36.576 INFO:tasks.rados.rados.0.smithi045.stdout:2119: expect (ObjNum 642 snap 199 seq_num 642) 2021-07-31T01:52:36.576 INFO:tasks.rados.rados.0.smithi045.stdout:2120: snap_create 2021-07-31T01:52:36.577 INFO:tasks.rados.rados.0.smithi045.stdout:2112: finishing copy_from to smithi04569269-31 2021-07-31T01:52:36.577 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 31 v 1001 (ObjNum 483 snap 143 seq_num 483) dirty exists 2021-07-31T01:52:36.577 INFO:tasks.rados.rados.0.smithi045.stdout:2112: finishing copy_from racing read to smithi04569269-31 2021-07-31T01:52:36.577 INFO:tasks.rados.rados.0.smithi045.stdout:2115: expect (ObjNum 539 snap 162 seq_num 539) 2021-07-31T01:52:36.709 INFO:tasks.rados.rados.0.smithi045.stdout:2118: expect (ObjNum 483 snap 143 seq_num 483) 2021-07-31T01:52:36.759 INFO:tasks.rados.rados.0.smithi045.stdout:2119: expect (ObjNum 642 snap 199 seq_num 642) 2021-07-31T01:52:37.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:36 smithi154 conmon[34553]: cluster 2021-07-31T01:52:35.590395+0000 mon.a (mon.0) 1109 : cluster [DBG] osdmap e456: 8 total, 8 up, 8 in 2021-07-31T01:52:37.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:36 smithi154 conmon[34553]: cluster 2021-07-31T01:52:35.866620 2021-07-31T01:52:37.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:36 smithi154 conmon[34553]: +0000 mgr.x (mgr.24542) 233 : cluster [DBG] pgmap v464: 25 pgs: 25 active+clean; 206 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 2.6 MiB/s wr, 22 op/s 2021-07-31T01:52:37.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:36 smithi154 conmon[34553]: cluster 2021-07-31T01:52:36.573696+0000 mon.a (mon.0) 1110 : cluster [DBG] osdmap e457: 8 total, 8 up, 8 in 2021-07-31T01:52:37.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: cluster 2021-07-31T01:52:37.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: 2021-07-31T01:52:37.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: 2021 2021-07-31T01:52:37.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: - 2021-07-31T01:52:37.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: 07 2021-07-31T01:52:37.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: - 2021-07-31T01:52:37.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: 31 2021-07-31T01:52:37.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: T 2021-07-31T01:52:37.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: 01 2021-07-31T01:52:37.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: : 2021-07-31T01:52:37.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: 52 2021-07-31T01:52:37.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: : 2021-07-31T01:52:37.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: 35 2021-07-31T01:52:37.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: . 2021-07-31T01:52:37.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: 590395 2021-07-31T01:52:37.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: +0000 2021-07-31T01:52:37.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: 2021-07-31T01:52:37.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: mon.a 2021-07-31T01:52:37.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: ( 2021-07-31T01:52:37.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: mon 2021-07-31T01:52:37.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: . 2021-07-31T01:52:37.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: 0 2021-07-31T01:52:37.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: ) 2021-07-31T01:52:37.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: 1109 2021-07-31T01:52:37.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: : 2021-07-31T01:52:37.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: cluster 2021-07-31T01:52:37.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: 2021-07-31T01:52:37.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: [DBG] 2021-07-31T01:52:37.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: 2021-07-31T01:52:37.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: osdmap e456: 8 total, 8 up, 8 in 2021-07-31T01:52:37.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: 2021-07-31T01:52:37.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: cluster 2021-07-31T01:52:37.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: 2021-07-31T01:52:37.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: 2021 2021-07-31T01:52:37.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: - 2021-07-31T01:52:37.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: 07 2021-07-31T01:52:37.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: - 2021-07-31T01:52:37.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: 31 2021-07-31T01:52:37.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: T 2021-07-31T01:52:37.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: 01 2021-07-31T01:52:37.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: : 2021-07-31T01:52:37.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: 52 2021-07-31T01:52:37.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: : 2021-07-31T01:52:37.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: 35 2021-07-31T01:52:37.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: . 2021-07-31T01:52:37.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: 866620 2021-07-31T01:52:37.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: +0000 2021-07-31T01:52:37.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: 2021-07-31T01:52:37.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: mgr.x 2021-07-31T01:52:37.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: ( 2021-07-31T01:52:37.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: mgr 2021-07-31T01:52:37.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: . 2021-07-31T01:52:37.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: 24542 2021-07-31T01:52:37.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: ) 2021-07-31T01:52:37.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: 233 2021-07-31T01:52:37.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: : 2021-07-31T01:52:37.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: cluster 2021-07-31T01:52:37.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: 2021-07-31T01:52:37.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: [DBG] 2021-07-31T01:52:37.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: 2021-07-31T01:52:37.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: pgmap v464: 25 pgs: 25 active+clean; 206 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 2.6 MiB/s wr, 22 op/s 2021-07-31T01:52:37.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: 2021-07-31T01:52:37.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: cluster 2021-07-31T01:52:37.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: 2021-07-31T01:52:37.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: 2021 2021-07-31T01:52:37.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: - 2021-07-31T01:52:37.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: 07 2021-07-31T01:52:37.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: - 2021-07-31T01:52:37.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: 31 2021-07-31T01:52:37.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: T 2021-07-31T01:52:37.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: 01 2021-07-31T01:52:37.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: : 2021-07-31T01:52:37.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: 52 2021-07-31T01:52:37.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: : 2021-07-31T01:52:37.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: 36 2021-07-31T01:52:37.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: . 2021-07-31T01:52:37.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: 573696 2021-07-31T01:52:37.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: +0000 2021-07-31T01:52:37.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: 2021-07-31T01:52:37.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: mon.a 2021-07-31T01:52:37.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: ( 2021-07-31T01:52:37.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: mon 2021-07-31T01:52:37.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: . 2021-07-31T01:52:37.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: 0 2021-07-31T01:52:37.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: ) 2021-07-31T01:52:37.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: 1110 2021-07-31T01:52:37.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: : 2021-07-31T01:52:37.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: cluster 2021-07-31T01:52:37.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: 2021-07-31T01:52:37.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: [DBG] 2021-07-31T01:52:37.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: 2021-07-31T01:52:37.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: osdmap e457: 8 total, 8 up, 8 in 2021-07-31T01:52:37.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[39718]: 2021-07-31T01:52:37.047 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[33655]: cluster 2021-07-31T01:52:35. 2021-07-31T01:52:37.047 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[33655]: 590395+0000 mon.a (mon.0) 1109 : cluster [DBG] osdmap e456: 8 total, 8 up, 8 in 2021-07-31T01:52:37.047 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[33655]: cluster 2021-07-31T01:52:35.866620+0000 mgr.x ( 2021-07-31T01:52:37.047 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[33655]: mgr.24542) 233 : cluster [DBG] pgmap v464: 25 pgs: 25 active+clean; 206 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 2.6 MiB/s wr, 22 op/s 2021-07-31T01:52:37.047 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[33655]: cluster 2021-07-31T01 2021-07-31T01:52:37.048 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[33655]: :52:36.573696+0000 mon.a (mon. 2021-07-31T01:52:37.048 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:36 smithi045 conmon[33655]: 0) 1110 : cluster [DBG] osdmap e457: 8 total, 8 up, 8 in 2021-07-31T01:52:37.580 INFO:tasks.rados.rados.0.smithi045.stdout:2112: done (4 left) 2021-07-31T01:52:37.580 INFO:tasks.rados.rados.0.smithi045.stdout:2115: done (3 left) 2021-07-31T01:52:37.580 INFO:tasks.rados.rados.0.smithi045.stdout:2118: done (2 left) 2021-07-31T01:52:37.581 INFO:tasks.rados.rados.0.smithi045.stdout:2119: done (1 left) 2021-07-31T01:52:37.581 INFO:tasks.rados.rados.0.smithi045.stdout:2120: done (0 left) 2021-07-31T01:52:37.581 INFO:tasks.rados.rados.0.smithi045.stdout:2121: snap_remove snap 199 2021-07-31T01:52:38.586 INFO:tasks.rados.rados.0.smithi045.stdout:2121: done (0 left) 2021-07-31T01:52:38.586 INFO:tasks.rados.rados.0.smithi045.stdout:2122: snap_create 2021-07-31T01:52:39.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:38 smithi154 conmon[34553]: cluster 2021-07-31T01:52:37.579185+0000 mon.a (mon.0) 2021-07-31T01:52:39.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:38 smithi154 conmon[34553]: 1111 : cluster [DBG] osdmap e458: 8 total, 8 up, 8 in 2021-07-31T01:52:39.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:38 smithi154 conmon[34553]: cluster 2021-07-31T01:52:37.867209+0000 mgr.x (mgr.24542) 234 : cluster [DBG] pgmap v467: 25 pgs: 25 active+clean; 206 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2021-07-31T01:52:39.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:38 smithi045 conmon[33655]: cluster 2021-07-31T01:52:37.579185+0000 mon.a (mon.0) 1111 : cluster [DBG] osdmap e458: 8 total, 8 up, 8 in 2021-07-31T01:52:39.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:38 smithi045 conmon[33655]: cluster 2021-07-31T01:52:37.867209+0000 mgr.x (mgr.24542) 234 : cluster [DBG] pgmap v467: 25 pgs: 25 active+clean; 206 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2021-07-31T01:52:39.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:38 smithi045 conmon[39718]: cluster 2021-07-31T01:52:37.579185 2021-07-31T01:52:39.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:38 smithi045 conmon[39718]: +0000 mon.a (mon.0) 1111 : cluster [DBG] osdmap e458: 8 total, 8 up, 8 in 2021-07-31T01:52:39.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:38 smithi045 conmon[39718]: cluster 2021-07-31T 2021-07-31T01:52:39.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:38 smithi045 conmon[39718]: 01:52:37.867209+0000 mgr.x (mgr.24542) 234 : cluster [DBG] pgmap v467: 25 pgs: 25 active+clean; 206 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2021-07-31T01:52:39.587 INFO:tasks.rados.rados.0.smithi045.stdout:2122: done (0 left) 2021-07-31T01:52:39.588 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 14 current snap is 205 2021-07-31T01:52:39.588 INFO:tasks.rados.rados.0.smithi045.stdout:2123: seq_num 659 ranges {7880704=647168,8527872=450560,8978432=352256} 2021-07-31T01:52:39.634 INFO:tasks.rados.rados.0.smithi045.stdout:2123: writing smithi04569269-14 from 7880704 to 8527872 tid 1 2021-07-31T01:52:39.636 INFO:tasks.rados.rados.0.smithi045.stdout:2123: writing smithi04569269-14 from 8527872 to 8978432 tid 2 2021-07-31T01:52:39.639 INFO:tasks.rados.rados.0.smithi045.stdout:2123: writing smithi04569269-14 from 8978432 to 9330688 tid 3 2021-07-31T01:52:39.639 INFO:tasks.rados.rados.0.smithi045.stdout:2124: read oid 25 snap -1 2021-07-31T01:52:39.639 INFO:tasks.rados.rados.0.smithi045.stdout:2124: expect (ObjNum 607 snap 186 seq_num 607) 2021-07-31T01:52:39.640 INFO:tasks.rados.rados.0.smithi045.stdout:2125: read oid 47 snap -1 2021-07-31T01:52:39.640 INFO:tasks.rados.rados.0.smithi045.stdout:2125: expect deleted 2021-07-31T01:52:39.640 INFO:tasks.rados.rados.0.smithi045.stdout:2126: rmattr oid 36 current snap is 205 2021-07-31T01:52:39.640 INFO:tasks.rados.rados.0.smithi045.stdout:2127: rollback oid 10 current snap is 205 2021-07-31T01:52:39.640 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 10 to 203 2021-07-31T01:52:39.641 INFO:tasks.rados.rados.0.smithi045.stdout:2128: delete oid 23 current snap is 205 2021-07-31T01:52:39.641 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 36 v 0 (ObjNum 629 snap 192 seq_num 629) dirty exists 2021-07-31T01:52:39.643 INFO:tasks.rados.rados.0.smithi045.stdout:2123: finishing write tid 1 to smithi04569269-14 2021-07-31T01:52:39.643 INFO:tasks.rados.rados.0.smithi045.stdout:2123: finishing write tid 2 to smithi04569269-14 2021-07-31T01:52:39.644 INFO:tasks.rados.rados.0.smithi045.stdout:2127: finishing rollback tid 1 to smithi04569269-10 2021-07-31T01:52:39.644 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 10 v 1138 (ObjNum -1105318832 snap 21873 seq_num 3189626160) dirty dne 2021-07-31T01:52:39.645 INFO:tasks.rados.rados.0.smithi045.stdout:2123: finishing write tid 3 to smithi04569269-14 2021-07-31T01:52:39.645 INFO:tasks.rados.rados.0.smithi045.stdout:2123: finishing write tid 4 to smithi04569269-14 2021-07-31T01:52:39.646 INFO:tasks.rados.rados.0.smithi045.stdout:2125: done (5 left) 2021-07-31T01:52:39.646 INFO:tasks.rados.rados.0.smithi045.stdout:2126: done (4 left) 2021-07-31T01:52:39.647 INFO:tasks.rados.rados.0.smithi045.stdout:2127: done (3 left) 2021-07-31T01:52:39.647 INFO:tasks.rados.rados.0.smithi045.stdout:2128: done (2 left) 2021-07-31T01:52:39.647 INFO:tasks.rados.rados.0.smithi045.stdout:2129: rollback oid 31 current snap is 205 2021-07-31T01:52:39.647 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 31 to 198 2021-07-31T01:52:39.648 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 43 current snap is 205 2021-07-31T01:52:39.648 INFO:tasks.rados.rados.0.smithi045.stdout:2130: seq_num 661 ranges {4104192=679936,4784128=622592,5406720=180224} 2021-07-31T01:52:39.673 INFO:tasks.rados.rados.0.smithi045.stdout:2130: writing smithi04569269-43 from 4104192 to 4784128 tid 1 2021-07-31T01:52:39.676 INFO:tasks.rados.rados.0.smithi045.stdout:2130: writing smithi04569269-43 from 4784128 to 5406720 tid 2 2021-07-31T01:52:39.678 INFO:tasks.rados.rados.0.smithi045.stdout:2130: writing smithi04569269-43 from 5406720 to 5586944 tid 3 2021-07-31T01:52:39.678 INFO:tasks.rados.rados.0.smithi045.stdout:2131: read oid 23 snap 186 2021-07-31T01:52:39.678 INFO:tasks.rados.rados.0.smithi045.stdout:2131: expect (ObjNum 560 snap 166 seq_num 560) 2021-07-31T01:52:39.679 INFO:tasks.rados.rados.0.smithi045.stdout:2132: rmattr oid 27 current snap is 205 2021-07-31T01:52:39.679 INFO:tasks.rados.rados.0.smithi045.stdout:2132: done (5 left) 2021-07-31T01:52:39.679 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 13 current snap is 205 2021-07-31T01:52:39.679 INFO:tasks.rados.rados.0.smithi045.stdout:2123: finishing write tid 5 to smithi04569269-14 2021-07-31T01:52:39.679 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 14 v 1054 (ObjNum 659 snap 205 seq_num 659) dirty exists 2021-07-31T01:52:39.680 INFO:tasks.rados.rados.0.smithi045.stdout:2123: left oid 14 (ObjNum 659 snap 205 seq_num 659) 2021-07-31T01:52:39.680 INFO:tasks.rados.rados.0.smithi045.stdout:2129: finishing rollback tid 0 to smithi04569269-31 2021-07-31T01:52:39.680 INFO:tasks.rados.rados.0.smithi045.stdout:2133: seq_num 662 ranges {12836864=696320,13533184=557056,14090240=245760} 2021-07-31T01:52:39.753 INFO:tasks.rados.rados.0.smithi045.stdout:2133: writing smithi04569269-13 from 12836864 to 13533184 tid 1 2021-07-31T01:52:39.756 INFO:tasks.rados.rados.0.smithi045.stdout:2133: writing smithi04569269-13 from 13533184 to 14090240 tid 2 2021-07-31T01:52:39.758 INFO:tasks.rados.rados.0.smithi045.stdout:2133: writing smithi04569269-13 from 14090240 to 14336000 tid 3 2021-07-31T01:52:39.758 INFO:tasks.rados.rados.0.smithi045.stdout:2123: done (5 left) 2021-07-31T01:52:39.759 INFO:tasks.rados.rados.0.smithi045.stdout:2134: snap_create 2021-07-31T01:52:39.760 INFO:tasks.rados.rados.0.smithi045.stdout:2129: finishing rollback tid 1 to smithi04569269-31 2021-07-31T01:52:39.761 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 31 v 1330 (ObjNum -1105278656 snap 21873 seq_num 0) dirty dne 2021-07-31T01:52:39.761 INFO:tasks.rados.rados.0.smithi045.stdout:2124: expect (ObjNum 607 snap 186 seq_num 607) 2021-07-31T01:52:40.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:39 smithi154 conmon[34553]: cluster 2021-07-31T01:52:38.585052+0000 mon.a (mon.0) 1112 : cluster [DBG] osdmap e459: 8 total, 8 up, 8 in 2021-07-31T01:52:40.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:39 smithi154 conmon[34553]: 2021-07-31T01:52:40.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:39 smithi045 conmon[39718]: cluster 2021-07-31T01:52:38.585052+0000 mon.a (mon.0) 1112 2021-07-31T01:52:40.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:39 smithi045 conmon[39718]: : cluster [DBG] osdmap e459: 8 total, 8 up, 8 in 2021-07-31T01:52:40.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:39 smithi045 conmon[33655]: cluster 2021-07-31T01:52:38.585052+0000 mon.a (mon.0) 1112 : cluster 2021-07-31T01:52:40.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:39 smithi045 conmon[33655]: [DBG] osdmap e459: 8 total, 8 up, 8 in 2021-07-31T01:52:40.105 INFO:tasks.rados.rados.0.smithi045.stdout:2130: finishing write tid 1 to smithi04569269-43 2021-07-31T01:52:40.105 INFO:tasks.rados.rados.0.smithi045.stdout:2130: finishing write tid 2 to smithi04569269-43 2021-07-31T01:52:40.105 INFO:tasks.rados.rados.0.smithi045.stdout:2130: finishing write tid 3 to smithi04569269-43 2021-07-31T01:52:40.106 INFO:tasks.rados.rados.0.smithi045.stdout:2130: finishing write tid 4 to smithi04569269-43 2021-07-31T01:52:40.106 INFO:tasks.rados.rados.0.smithi045.stdout:2130: finishing write tid 5 to smithi04569269-43 2021-07-31T01:52:40.106 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 43 v 1058 (ObjNum 661 snap 205 seq_num 661) dirty exists 2021-07-31T01:52:40.106 INFO:tasks.rados.rados.0.smithi045.stdout:2130: left oid 43 (ObjNum 661 snap 205 seq_num 661) 2021-07-31T01:52:40.106 INFO:tasks.rados.rados.0.smithi045.stdout:2131: expect (ObjNum 560 snap 166 seq_num 560) 2021-07-31T01:52:40.350 INFO:tasks.rados.rados.0.smithi045.stdout:2133: finishing write tid 1 to smithi04569269-13 2021-07-31T01:52:40.350 INFO:tasks.rados.rados.0.smithi045.stdout:2133: finishing write tid 2 to smithi04569269-13 2021-07-31T01:52:40.351 INFO:tasks.rados.rados.0.smithi045.stdout:2133: finishing write tid 3 to smithi04569269-13 2021-07-31T01:52:40.351 INFO:tasks.rados.rados.0.smithi045.stdout:2133: finishing write tid 4 to smithi04569269-13 2021-07-31T01:52:40.352 INFO:tasks.rados.rados.0.smithi045.stdout:2133: finishing write tid 5 to smithi04569269-13 2021-07-31T01:52:40.352 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 13 v 799 (ObjNum 662 snap 205 seq_num 662) dirty exists 2021-07-31T01:52:40.352 INFO:tasks.rados.rados.0.smithi045.stdout:2133: left oid 13 (ObjNum 662 snap 205 seq_num 662) 2021-07-31T01:52:40.598 INFO:tasks.rados.rados.0.smithi045.stdout:2124: done (5 left) 2021-07-31T01:52:40.599 INFO:tasks.rados.rados.0.smithi045.stdout:2129: done (4 left) 2021-07-31T01:52:40.599 INFO:tasks.rados.rados.0.smithi045.stdout:2130: done (3 left) 2021-07-31T01:52:40.601 INFO:tasks.rados.rados.0.smithi045.stdout:2131: done (2 left) 2021-07-31T01:52:40.602 INFO:tasks.rados.rados.0.smithi045.stdout:2133: done (1 left) 2021-07-31T01:52:40.602 INFO:tasks.rados.rados.0.smithi045.stdout:2134: done (0 left) 2021-07-31T01:52:40.603 INFO:tasks.rados.rados.0.smithi045.stdout:2135: read oid 13 snap 198 2021-07-31T01:52:40.603 INFO:tasks.rados.rados.0.smithi045.stdout:2135: expect (ObjNum 627 snap 191 seq_num 627) 2021-07-31T01:52:40.603 INFO:tasks.rados.rados.0.smithi045.stdout:2136: snap_create 2021-07-31T01:52:40.661 INFO:tasks.rados.rados.0.smithi045.stdout:2135: expect (ObjNum 627 snap 191 seq_num 627) 2021-07-31T01:52:41.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:40 smithi154 conmon[34553]: cluster 2021-07-31T01:52:39.586417 2021-07-31T01:52:41.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:40 smithi154 conmon[34553]: +0000 mon.a (mon.0) 1113 : cluster [DBG] osdmap e460: 8 total, 8 up, 8 in 2021-07-31T01:52:41.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:40 smithi154 conmon[34553]: cluster 2021-07-31T01:52:39.867623+0000 mgr.x (mgr.24542) 235 : cluster [DBG] pgmap v470: 25 pgs: 25 active+clean; 215 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 3.1 MiB/s wr, 31 op/s 2021-07-31T01:52:41.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:40 smithi045 conmon[33655]: cluster 2021-07-31T01:52:39. 2021-07-31T01:52:41.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:40 smithi045 conmon[33655]: 586417+0000 mon.a (mon.0) 1113 : cluster [DBG] osdmap e460: 8 total, 8 up, 8 in 2021-07-31T01:52:41.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:40 smithi045 conmon[33655]: cluster 2021-07-31T01:52:39.867623+0000 mgr.x (mgr 2021-07-31T01:52:41.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:40 smithi045 conmon[33655]: .24542) 235 : cluster [DBG] pgmap v470: 25 pgs: 25 active+clean; 215 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 3.1 MiB/s wr, 31 op/s 2021-07-31T01:52:41.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:40 smithi045 conmon[39718]: cluster 2021-07-31T01:52 2021-07-31T01:52:41.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:40 smithi045 conmon[39718]: :39.586417+0000 mon.a (mon.0) 1113 : cluster [DBG] osdmap e460: 8 total, 8 up, 8 in 2021-07-31T01:52:41.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:40 smithi045 conmon[39718]: cluster 2021- 2021-07-31T01:52:41.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:40 smithi045 conmon[39718]: 07-31T01:52:39. 2021-07-31T01:52:41.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:40 smithi045 conmon[39718]: 867623+0000 mgr.x (mgr.24542 2021-07-31T01:52:41.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:40 smithi045 conmon[39718]: ) 235 : cluster [DBG] pgmap v470: 25 pgs: 25 active+clean; 215 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 3.1 MiB/s wr, 31 op/s 2021-07-31T01:52:41.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:40 smithi045 conmon[39718]: 2021-07-31T01:52:41.823 INFO:tasks.rados.rados.0.smithi045.stdout:2135: done (1 left) 2021-07-31T01:52:41.824 INFO:tasks.rados.rados.0.smithi045.stdout:2136: done (0 left) 2021-07-31T01:52:41.824 INFO:tasks.rados.rados.0.smithi045.stdout:2137: snap_create 2021-07-31T01:52:42.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:41 smithi154 conmon[34553]: cluster 2021-07-31T01:52:40.596376+0000 mon.a (mon.0) 1114 : cluster [DBG] osdmap e461: 8 total, 8 up, 8 in 2021-07-31T01:52:42.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:41 smithi154 conmon[34553]: 2021-07-31T01:52:42.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:41 smithi154 conmon[34553]: cluster 2021-07-31T01:52:41.573847+0000 mon.a (mon.0) 1115 : cluster [DBG] osdmap e462: 8 total, 8 up, 8 in 2021-07-31T01:52:42.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:41 smithi045 conmon[33655]: cluster 2021-07-31T01:52:40.596376 2021-07-31T01:52:42.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:41 smithi045 conmon[33655]: +0000 mon.a (mon.0) 1114 : cluster [DBG] osdmap e461: 8 total, 8 up, 8 in 2021-07-31T01:52:42.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:41 smithi045 conmon[33655]: cluster 2021-07-31T01:52:41.573847 2021-07-31T01:52:42.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:41 smithi045 conmon[33655]: +0000 mon.a (mon.0) 1115 : cluster [DBG] osdmap e462: 8 total, 8 up, 8 in 2021-07-31T01:52:42.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:41 smithi045 conmon[33655]: 2021-07-31T01:52:42.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:41 smithi045 conmon[39718]: cluster 2021-07-31T01:52:40.596376+0000 2021-07-31T01:52:42.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:41 smithi045 conmon[39718]: mon.a (mon.0) 1114 : cluster [DBG] osdmap e461: 8 total, 8 up, 8 in 2021-07-31T01:52:42.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:41 smithi045 conmon[39718]: cluster 2021-07-31T01:52:42.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:41 smithi045 conmon[39718]: 2021-07-31T01:52:41.573847 2021-07-31T01:52:42.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:41 smithi045 conmon[39718]: +0000 mon.a (mon.0) 1115 : cluster [DBG] 2021-07-31T01:52:42.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:41 smithi045 conmon[39718]: osdmap e462: 8 total, 8 up, 8 in 2021-07-31T01:52:42.605 INFO:tasks.rados.rados.0.smithi045.stdout:2137: done (0 left) 2021-07-31T01:52:42.606 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 13 current snap is 208 2021-07-31T01:52:42.606 INFO:tasks.rados.rados.0.smithi045.stdout:2138: seq_num 663 ranges {14336000=712704,15048704=442368,15491072=360448} 2021-07-31T01:52:42.687 INFO:tasks.rados.rados.0.smithi045.stdout:2138: writing smithi04569269-13 from 14336000 to 15048704 tid 1 2021-07-31T01:52:42.689 INFO:tasks.rados.rados.0.smithi045.stdout:2138: writing smithi04569269-13 from 15048704 to 15491072 tid 2 2021-07-31T01:52:42.691 INFO:tasks.rados.rados.0.smithi045.stdout:2138: writing smithi04569269-13 from 15491072 to 15851520 tid 3 2021-07-31T01:52:42.692 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 17 current snap is 208 2021-07-31T01:52:42.692 INFO:tasks.rados.rados.0.smithi045.stdout:2139: seq_num 664 ranges {7045120=729088,7774208=729088,8503296=73728} 2021-07-31T01:52:42.735 INFO:tasks.rados.rados.0.smithi045.stdout:2139: writing smithi04569269-17 from 7045120 to 7774208 tid 1 2021-07-31T01:52:42.738 INFO:tasks.rados.rados.0.smithi045.stdout:2139: writing smithi04569269-17 from 7774208 to 8503296 tid 2 2021-07-31T01:52:42.740 INFO:tasks.rados.rados.0.smithi045.stdout:2139: writing smithi04569269-17 from 8503296 to 8577024 tid 3 2021-07-31T01:52:42.740 INFO:tasks.rados.rados.0.smithi045.stdout:2138: finishing write tid 1 to smithi04569269-13 2021-07-31T01:52:42.740 INFO:tasks.rados.rados.0.smithi045.stdout:2138: finishing write tid 2 to smithi04569269-13 2021-07-31T01:52:42.740 INFO:tasks.rados.rados.0.smithi045.stdout:2138: finishing write tid 3 to smithi04569269-13 2021-07-31T01:52:42.742 INFO:tasks.rados.rados.0.smithi045.stdout:2138: finishing write tid 4 to smithi04569269-13 2021-07-31T01:52:42.743 INFO:tasks.rados.rados.0.smithi045.stdout:2140: read oid 30 snap -1 2021-07-31T01:52:42.743 INFO:tasks.rados.rados.0.smithi045.stdout:2140: expect (ObjNum 0 snap 0 seq_num 0) 2021-07-31T01:52:42.743 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 46 current snap is 208 2021-07-31T01:52:42.743 INFO:tasks.rados.rados.0.smithi045.stdout:2138: finishing write tid 5 to smithi04569269-13 2021-07-31T01:52:42.744 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 13 v 804 (ObjNum 663 snap 208 seq_num 663) dirty exists 2021-07-31T01:52:42.744 INFO:tasks.rados.rados.0.smithi045.stdout:2138: left oid 13 (ObjNum 663 snap 208 seq_num 663) 2021-07-31T01:52:42.744 INFO:tasks.rados.rados.0.smithi045.stdout:2139: finishing write tid 1 to smithi04569269-17 2021-07-31T01:52:42.744 INFO:tasks.rados.rados.0.smithi045.stdout:2141: seq_num 665 ranges {2801664=745472,3547136=606208,4153344=196608} 2021-07-31T01:52:42.759 INFO:tasks.rados.rados.0.smithi045.stdout:2141: writing smithi04569269-46 from 2801664 to 3547136 tid 1 2021-07-31T01:52:42.761 INFO:tasks.rados.rados.0.smithi045.stdout:2141: writing smithi04569269-46 from 3547136 to 4153344 tid 2 2021-07-31T01:52:42.763 INFO:tasks.rados.rados.0.smithi045.stdout:2141: writing smithi04569269-46 from 4153344 to 4349952 tid 3 2021-07-31T01:52:42.763 INFO:tasks.rados.rados.0.smithi045.stdout:2138: done (3 left) 2021-07-31T01:52:42.763 INFO:tasks.rados.rados.0.smithi045.stdout:2142: snap_remove snap 186 2021-07-31T01:52:43.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:42 smithi154 conmon[34553]: cluster 2021-07-31T01:52:41.868106+0000 mgr.x (mgr.24542) 236 : 2021-07-31T01:52:43.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:42 smithi154 conmon[34553]: cluster [DBG] pgmap v473: 25 pgs: 25 active+clean; 215 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 3.1 MiB/s wr, 31 op/s 2021-07-31T01:52:43.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:42 smithi045 conmon[39718]: cluster 2021-07-31T01:52:41.868106+0000 mgr.x 2021-07-31T01:52:43.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:42 smithi045 conmon[39718]: (mgr.24542) 236 : cluster [DBG] pgmap v473: 25 pgs: 25 active+clean; 215 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 3.1 MiB/s wr, 31 op/s 2021-07-31T01:52:43.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:42 smithi045 conmon[33655]: cluster 2021-07-31T01:52:41.868106+0000 2021-07-31T01:52:43.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:42 smithi045 conmon[33655]: mgr.x (mgr.24542) 236 : cluster [DBG] pgmap v473: 25 pgs: 25 active+clean; 215 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 3.1 MiB/s wr, 31 op/s 2021-07-31T01:52:43.608 INFO:tasks.rados.rados.0.smithi045.stdout:2142: done (3 left) 2021-07-31T01:52:43.608 INFO:tasks.rados.rados.0.smithi045.stdout:2143: copy_from oid 50 from oid 6 current snap is 208 2021-07-31T01:52:43.608 INFO:tasks.rados.rados.0.smithi045.stdout:2144: copy_from oid 16 from oid 5 current snap is 208 2021-07-31T01:52:43.609 INFO:tasks.rados.rados.0.smithi045.stdout:2145: delete oid 28 current snap is 208 2021-07-31T01:52:43.609 INFO:tasks.rados.rados.0.smithi045.stdout:2140: expect (ObjNum 0 snap 0 seq_num 0) 2021-07-31T01:52:43.613 INFO:tasks.rados.rados.0.smithi045.stdout:2139: finishing write tid 2 to smithi04569269-17 2021-07-31T01:52:43.613 INFO:tasks.rados.rados.0.smithi045.stdout:2139: finishing write tid 3 to smithi04569269-17 2021-07-31T01:52:43.613 INFO:tasks.rados.rados.0.smithi045.stdout:2139: finishing write tid 4 to smithi04569269-17 2021-07-31T01:52:43.614 INFO:tasks.rados.rados.0.smithi045.stdout:2139: finishing write tid 5 to smithi04569269-17 2021-07-31T01:52:43.614 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 17 v 1142 (ObjNum 664 snap 208 seq_num 664) dirty exists 2021-07-31T01:52:43.614 INFO:tasks.rados.rados.0.smithi045.stdout:2139: left oid 17 (ObjNum 664 snap 208 seq_num 664) 2021-07-31T01:52:43.614 INFO:tasks.rados.rados.0.smithi045.stdout:2141: finishing write tid 1 to smithi04569269-46 2021-07-31T01:52:43.615 INFO:tasks.rados.rados.0.smithi045.stdout:2141: finishing write tid 2 to smithi04569269-46 2021-07-31T01:52:43.615 INFO:tasks.rados.rados.0.smithi045.stdout:2141: finishing write tid 3 to smithi04569269-46 2021-07-31T01:52:43.615 INFO:tasks.rados.rados.0.smithi045.stdout:2141: finishing write tid 4 to smithi04569269-46 2021-07-31T01:52:43.615 INFO:tasks.rados.rados.0.smithi045.stdout:2141: finishing write tid 5 to smithi04569269-46 2021-07-31T01:52:43.615 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 46 v 1297 (ObjNum 665 snap 208 seq_num 665) dirty exists 2021-07-31T01:52:43.616 INFO:tasks.rados.rados.0.smithi045.stdout:2141: left oid 46 (ObjNum 665 snap 208 seq_num 665) 2021-07-31T01:52:43.621 INFO:tasks.rados.rados.0.smithi045.stdout:2139: done (5 left) 2021-07-31T01:52:43.621 INFO:tasks.rados.rados.0.smithi045.stdout:2140: done (4 left) 2021-07-31T01:52:43.621 INFO:tasks.rados.rados.0.smithi045.stdout:2141: done (3 left) 2021-07-31T01:52:43.621 INFO:tasks.rados.rados.0.smithi045.stdout:2145: done (2 left) 2021-07-31T01:52:43.622 INFO:tasks.rados.rados.0.smithi045.stdout:2146: read oid 3 snap -1 2021-07-31T01:52:43.622 INFO:tasks.rados.rados.0.smithi045.stdout:2146: expect (ObjNum 483 snap 143 seq_num 483) 2021-07-31T01:52:43.622 INFO:tasks.rados.rados.0.smithi045.stdout:2147: snap_remove snap 193 2021-07-31T01:52:43.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:52:43 smithi045 conmon[61210]: level=error ts=2021-07-31T01:52:43.518Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:52:43.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:52:43 smithi045 conmon[61210]: level=error ts=2021-07-31T01:52:43.518Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:52:43.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:52:43 smithi045 conmon[61210]: level=error ts=2021-07-31T01:52:43.518Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T01:52:43.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:43 smithi045 conmon[39718]: cluster 2021-07-31T01:52:43.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:43 smithi045 conmon[39718]: 2021-07-31T01:52:43.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:43 smithi045 conmon[39718]: 2021-07-31T01: 2021-07-31T01:52:43.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:43 smithi045 conmon[39718]: 52:42.604431+0000 mon.a (mon.0) 1116 : cluster [DBG] osdmap e463: 8 total, 8 up, 8 in 2021-07-31T01:52:43.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:43 smithi045 conmon[33655]: cluster 2021-07-31T01:52:42.604431+0000 mon.a (mon.0) 1116 : cluster [DBG] osdmap e463: 8 total, 8 up, 8 in 2021-07-31T01:52:44.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:43 smithi154 conmon[34553]: cluster 2021-07-31T01:52:44.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:43 smithi154 conmon[34553]: 2021-07-31T01:52:44.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:43 smithi154 conmon[34553]: 2021 2021-07-31T01:52:44.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:43 smithi154 conmon[34553]: - 2021-07-31T01:52:44.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:43 smithi154 conmon[34553]: 07 2021-07-31T01:52:44.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:43 smithi154 conmon[34553]: - 2021-07-31T01:52:44.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:43 smithi154 conmon[34553]: 31 2021-07-31T01:52:44.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:43 smithi154 conmon[34553]: T 2021-07-31T01:52:44.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:43 smithi154 conmon[34553]: 01 2021-07-31T01:52:44.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:43 smithi154 conmon[34553]: : 2021-07-31T01:52:44.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:43 smithi154 conmon[34553]: 52 2021-07-31T01:52:44.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:43 smithi154 conmon[34553]: : 2021-07-31T01:52:44.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:43 smithi154 conmon[34553]: 42 2021-07-31T01:52:44.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:43 smithi154 conmon[34553]: . 2021-07-31T01:52:44.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:43 smithi154 conmon[34553]: 604431 2021-07-31T01:52:44.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:43 smithi154 conmon[34553]: +0000 2021-07-31T01:52:44.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:43 smithi154 conmon[34553]: 2021-07-31T01:52:44.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:43 smithi154 conmon[34553]: mon.a 2021-07-31T01:52:44.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:43 smithi154 conmon[34553]: ( 2021-07-31T01:52:44.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:43 smithi154 conmon[34553]: mon 2021-07-31T01:52:44.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:43 smithi154 conmon[34553]: . 2021-07-31T01:52:44.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:43 smithi154 conmon[34553]: 0 2021-07-31T01:52:44.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:43 smithi154 conmon[34553]: ) 2021-07-31T01:52:44.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:43 smithi154 conmon[34553]: 1116 2021-07-31T01:52:44.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:43 smithi154 conmon[34553]: : 2021-07-31T01:52:44.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:43 smithi154 conmon[34553]: cluster 2021-07-31T01:52:44.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:43 smithi154 conmon[34553]: 2021-07-31T01:52:44.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:43 smithi154 conmon[34553]: [DBG] 2021-07-31T01:52:44.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:43 smithi154 conmon[34553]: 2021-07-31T01:52:44.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:43 smithi154 conmon[34553]: osdmap e463: 8 total, 8 up, 8 in 2021-07-31T01:52:44.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:43 smithi154 conmon[34553]: 2021-07-31T01:52:44.618 INFO:tasks.rados.rados.0.smithi045.stdout:2147: done (3 left) 2021-07-31T01:52:44.619 INFO:tasks.rados.rados.0.smithi045.stdout:2148: copy_from oid 34 from oid 41 current snap is 208 2021-07-31T01:52:44.619 INFO:tasks.rados.rados.0.smithi045.stdout:2146: expect (ObjNum 483 snap 143 seq_num 483) 2021-07-31T01:52:44.679 INFO:tasks.rados.rados.0.smithi045.stdout:2144: finishing copy_from to smithi04569269-16 2021-07-31T01:52:44.679 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 16 v 1142 (ObjNum 619 snap 189 seq_num 619) dirty exists 2021-07-31T01:52:44.680 INFO:tasks.rados.rados.0.smithi045.stdout:2144: finishing copy_from racing read to smithi04569269-16 2021-07-31T01:52:44.680 INFO:tasks.rados.rados.0.smithi045.stdout:2143: finishing copy_from to smithi04569269-50 2021-07-31T01:52:44.680 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 50 v 765 (ObjNum 635 snap 198 seq_num 635) dirty exists 2021-07-31T01:52:44.680 INFO:tasks.rados.rados.0.smithi045.stdout:2143: finishing copy_from racing read to smithi04569269-50 2021-07-31T01:52:44.681 INFO:tasks.rados.rados.0.smithi045.stdout:2148: finishing copy_from to smithi04569269-34 2021-07-31T01:52:44.681 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 34 v 984 (ObjNum 639 snap 199 seq_num 639) dirty exists 2021-07-31T01:52:44.681 INFO:tasks.rados.rados.0.smithi045.stdout:2148: finishing copy_from racing read to smithi04569269-34 2021-07-31T01:52:44.681 INFO:tasks.rados.rados.0.smithi045.stdout:2143: done (3 left) 2021-07-31T01:52:44.681 INFO:tasks.rados.rados.0.smithi045.stdout:2144: done (2 left) 2021-07-31T01:52:44.682 INFO:tasks.rados.rados.0.smithi045.stdout:2146: done (1 left) 2021-07-31T01:52:44.682 INFO:tasks.rados.rados.0.smithi045.stdout:2148: done (0 left) 2021-07-31T01:52:44.682 INFO:tasks.rados.rados.0.smithi045.stdout:2149: delete oid 26 current snap is 208 2021-07-31T01:52:44.683 INFO:tasks.rados.rados.0.smithi045.stdout:2149: done (0 left) 2021-07-31T01:52:44.683 INFO:tasks.rados.rados.0.smithi045.stdout:2150: read oid 42 snap -1 2021-07-31T01:52:44.683 INFO:tasks.rados.rados.0.smithi045.stdout:2150: expect (ObjNum 483 snap 143 seq_num 483) 2021-07-31T01:52:44.683 INFO:tasks.rados.rados.0.smithi045.stdout:2151: snap_remove snap 207 2021-07-31T01:52:45.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:44 smithi154 conmon[34553]: cluster 2021-07-31T01: 2021-07-31T01:52:45.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:44 smithi154 conmon[34553]: 52:43.605796+0000 mon.a (mon.0) 1117 : cluster [DBG] osdmap e464: 8 total, 8 up, 8 in 2021-07-31T01:52:45.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:44 smithi154 conmon[34553]: cluster 2021-07-31T01:52:43.868540+0000 mgr.x (mgr.24542) 237 : cluster [DBG] pgmap v476: 25 pgs: 25 active+clean; 226 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 19 MiB/s rd, 3.0 MiB/s wr, 14 op/s 2021-07-31T01:52:45.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:44 smithi045 conmon[39718]: cluster 2021-07-31T01:52:43.605796+0000 mon.a (mon.0) 1117 2021-07-31T01:52:45.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:44 smithi045 conmon[39718]: : cluster [DBG] osdmap e464: 8 total, 8 up, 8 in 2021-07-31T01:52:45.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:44 smithi045 conmon[39718]: cluster 2021-07-31T01:52:43.868540+0000 mgr.x (mgr.24542) 237 : cluster [DBG] 2021-07-31T01:52:45.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:44 smithi045 conmon[39718]: pgmap v476: 25 pgs: 25 active+clean; 226 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 19 MiB/s rd, 3.0 MiB/s wr, 14 op/s 2021-07-31T01:52:45.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:44 smithi045 conmon[33655]: cluster 2021-07-31T01:52:43.605796+0000 2021-07-31T01:52:45.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:44 smithi045 conmon[33655]: mon.a (mon.0) 1117 : cluster [DBG] osdmap e464: 8 total, 8 up, 8 in 2021-07-31T01:52:45.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:44 smithi045 conmon[33655]: cluster 2021-07-31T01:52:43.868540 2021-07-31T01:52:45.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:44 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 237 : cluster [DBG] pgmap v476: 25 pgs: 25 active+clean; 226 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 19 MiB/s rd, 3.0 MiB/s wr, 14 op/s 2021-07-31T01:52:45.624 INFO:tasks.rados.rados.0.smithi045.stdout:2151: done (1 left) 2021-07-31T01:52:45.625 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 12 current snap is 208 2021-07-31T01:52:45.625 INFO:tasks.rados.rados.0.smithi045.stdout:2152: seq_num 668 ranges {2367488=794624,3162112=663552,3825664=139264} 2021-07-31T01:52:45.642 INFO:tasks.rados.rados.0.smithi045.stdout:2152: writing smithi04569269-12 from 2367488 to 3162112 tid 1 2021-07-31T01:52:45.645 INFO:tasks.rados.rados.0.smithi045.stdout:2152: writing smithi04569269-12 from 3162112 to 3825664 tid 2 2021-07-31T01:52:45.646 INFO:tasks.rados.rados.0.smithi045.stdout:2152: writing smithi04569269-12 from 3825664 to 3964928 tid 3 2021-07-31T01:52:45.646 INFO:tasks.rados.rados.0.smithi045.stdout:2153: rmattr oid 9 current snap is 208 2021-07-31T01:52:45.647 INFO:tasks.rados.rados.0.smithi045.stdout:2153: done (2 left) 2021-07-31T01:52:45.647 INFO:tasks.rados.rados.0.smithi045.stdout:2154: read oid 32 snap 205 2021-07-31T01:52:45.647 INFO:tasks.rados.rados.0.smithi045.stdout:2154: expect (ObjNum 648 snap 202 seq_num 648) 2021-07-31T01:52:45.647 INFO:tasks.rados.rados.0.smithi045.stdout:2155: read oid 18 snap 203 2021-07-31T01:52:45.647 INFO:tasks.rados.rados.0.smithi045.stdout:2155: expect deleted 2021-07-31T01:52:45.648 INFO:tasks.rados.rados.0.smithi045.stdout:2156: delete oid 44 current snap is 208 2021-07-31T01:52:45.648 INFO:tasks.rados.rados.0.smithi045.stdout:2150: expect (ObjNum 483 snap 143 seq_num 483) 2021-07-31T01:52:45.700 INFO:tasks.rados.rados.0.smithi045.stdout:2152: finishing write tid 1 to smithi04569269-12 2021-07-31T01:52:45.700 INFO:tasks.rados.rados.0.smithi045.stdout:2152: finishing write tid 2 to smithi04569269-12 2021-07-31T01:52:45.700 INFO:tasks.rados.rados.0.smithi045.stdout:2152: finishing write tid 3 to smithi04569269-12 2021-07-31T01:52:45.701 INFO:tasks.rados.rados.0.smithi045.stdout:2152: finishing write tid 4 to smithi04569269-12 2021-07-31T01:52:45.701 INFO:tasks.rados.rados.0.smithi045.stdout:2152: finishing write tid 5 to smithi04569269-12 2021-07-31T01:52:45.701 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 12 v 825 (ObjNum 668 snap 208 seq_num 668) dirty exists 2021-07-31T01:52:45.701 INFO:tasks.rados.rados.0.smithi045.stdout:2152: left oid 12 (ObjNum 668 snap 208 seq_num 668) 2021-07-31T01:52:45.702 INFO:tasks.rados.rados.0.smithi045.stdout:2154: expect (ObjNum 648 snap 202 seq_num 648) 2021-07-31T01:52:45.858 INFO:tasks.rados.rados.0.smithi045.stdout:2150: done (4 left) 2021-07-31T01:52:45.859 INFO:tasks.rados.rados.0.smithi045.stdout:2152: done (3 left) 2021-07-31T01:52:45.859 INFO:tasks.rados.rados.0.smithi045.stdout:2154: done (2 left) 2021-07-31T01:52:45.859 INFO:tasks.rados.rados.0.smithi045.stdout:2155: done (1 left) 2021-07-31T01:52:45.859 INFO:tasks.rados.rados.0.smithi045.stdout:2156: done (0 left) 2021-07-31T01:52:45.860 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 24 current snap is 208 2021-07-31T01:52:45.860 INFO:tasks.rados.rados.0.smithi045.stdout:2157: seq_num 670 ranges {4677632=425984,5103616=483328,5586944=507904,6094848=212992} 2021-07-31T01:52:45.886 INFO:tasks.rados.rados.0.smithi045.stdout:2157: writing smithi04569269-24 from 4677632 to 5103616 tid 1 2021-07-31T01:52:45.888 INFO:tasks.rados.rados.0.smithi045.stdout:2157: writing smithi04569269-24 from 5103616 to 5586944 tid 2 2021-07-31T01:52:45.891 INFO:tasks.rados.rados.0.smithi045.stdout:2157: writing smithi04569269-24 from 5586944 to 6094848 tid 3 2021-07-31T01:52:45.893 INFO:tasks.rados.rados.0.smithi045.stdout:2157: writing smithi04569269-24 from 6094848 to 6307840 tid 4 2021-07-31T01:52:45.893 INFO:tasks.rados.rados.0.smithi045.stdout:2158: delete oid 29 current snap is 208 2021-07-31T01:52:45.893 INFO:tasks.rados.rados.0.smithi045.stdout:2157: finishing write tid 1 to smithi04569269-24 2021-07-31T01:52:45.893 INFO:tasks.rados.rados.0.smithi045.stdout:2157: finishing write tid 2 to smithi04569269-24 2021-07-31T01:52:45.894 INFO:tasks.rados.rados.0.smithi045.stdout:2157: finishing write tid 3 to smithi04569269-24 2021-07-31T01:52:45.895 INFO:tasks.rados.rados.0.smithi045.stdout:2157: finishing write tid 4 to smithi04569269-24 2021-07-31T01:52:45.896 INFO:tasks.rados.rados.0.smithi045.stdout:2157: finishing write tid 5 to smithi04569269-24 2021-07-31T01:52:45.896 INFO:tasks.rados.rados.0.smithi045.stdout:2158: done (1 left) 2021-07-31T01:52:45.896 INFO:tasks.rados.rados.0.smithi045.stdout:2159: snap_create 2021-07-31T01:52:45.897 INFO:tasks.rados.rados.0.smithi045.stdout:2157: finishing write tid 6 to smithi04569269-24 2021-07-31T01:52:45.897 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 24 v 548 (ObjNum 670 snap 208 seq_num 670) dirty exists 2021-07-31T01:52:45.897 INFO:tasks.rados.rados.0.smithi045.stdout:2157: left oid 24 (ObjNum 670 snap 208 seq_num 670) 2021-07-31T01:52:46.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:45 smithi154 conmon[34553]: cluster 2021-07-31T01:52:44.617478+0000 mon.a (mon.0) 1118 : cluster [DBG] osdmap e465: 8 total, 8 up, 8 in 2021-07-31T01:52:46.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:45 smithi045 conmon[39718]: cluster 2021-07-31T01:52:44.617478+0000 mon.a (mon 2021-07-31T01:52:46.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:45 smithi045 conmon[39718]: .0) 1118 : cluster [DBG] osdmap e465: 8 total, 8 up, 8 in 2021-07-31T01:52:46.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:45 smithi045 conmon[33655]: cluster 2021-07-31T01:52:44.617478+0000 2021-07-31T01:52:46.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:45 smithi045 conmon[33655]: mon.a (mon.0) 1118 : cluster [DBG] osdmap e465: 8 total, 8 up, 8 in 2021-07-31T01:52:46.576 INFO:tasks.rados.rados.0.smithi045.stdout:2157: done (1 left) 2021-07-31T01:52:46.576 INFO:tasks.rados.rados.0.smithi045.stdout:2159: done (0 left) 2021-07-31T01:52:46.577 INFO:tasks.rados.rados.0.smithi045.stdout:2160: snap_remove snap 201 2021-07-31T01:52:47.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:46 smithi154 conmon[34553]: cluster 2021-07-31T01:52:45.623859+0000 mon.a (mon.0) 1119 : cluster [DBG] osdmap e466: 8 total, 8 up, 8 in 2021-07-31T01:52:47.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:46 smithi154 conmon[34553]: cluster 2021-07-31T01:52:45.869066+0000 mgr.x (mgr.24542) 238 : cluster [DBG] pgmap v479: 25 pgs: 25 active+clean; 226 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 19 MiB/s rd, 3.0 MiB/s wr, 14 op/s 2021-07-31T01:52:47.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:46 smithi154 conmon[34553]: cluster 2021-07-31T01:52:46.575696+0000 mon.a (mon.0) 1120 : cluster 2021-07-31T01:52:47.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:46 smithi154 conmon[34553]: [DBG] osdmap e467: 8 total, 8 up, 8 in 2021-07-31T01:52:47.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:46 smithi045 conmon[39718]: cluster 2021-07-31T01:52:45.623859+0000 2021-07-31T01:52:47.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:46 smithi045 conmon[39718]: mon.a (mon.0) 1119 : cluster [DBG] osdmap e466: 8 total, 8 up, 8 in 2021-07-31T01:52:47.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:46 smithi045 conmon[39718]: cluster 2021-07-31T01:52:45. 2021-07-31T01:52:47.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:46 smithi045 conmon[39718]: 869066+0000 mgr.x (mgr.24542) 238 : cluster [DBG] pgmap v479: 25 pgs: 25 active+clean; 226 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 19 MiB/s rd, 3.0 MiB/s wr, 14 op/s 2021-07-31T01:52:47.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:46 smithi045 conmon[39718]: cluster 2021 2021-07-31T01:52:47.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:46 smithi045 conmon[39718]: -07-31T01:52:46.575696+0000 mon.a (mon.0) 1120 : cluster [DBG] 2021-07-31T01:52:47.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:46 smithi045 conmon[39718]: osdmap e467: 8 total, 8 up, 8 in 2021-07-31T01:52:47.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:46 smithi045 conmon[33655]: cluster 2021-07-31T01:52:45.623859+0000 mon.a (mon 2021-07-31T01:52:47.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:46 smithi045 conmon[33655]: .0) 1119 : cluster [DBG] osdmap e466: 8 total, 8 up, 8 in 2021-07-31T01:52:47.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:46 smithi045 conmon[33655]: cluster 2021-07-31T01:52:45.869066+0000 mgr.x (mgr.24542) 238 : cluster [DBG] pgmap v479: 25 pgs: 25 active+clean; 226 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 19 MiB/s rd, 3.0 MiB/s wr, 14 op/s 2021-07-31T01:52:47.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:46 smithi045 conmon[33655]: cluster 2021-07-31T01:52:46. 2021-07-31T01:52:47.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:46 smithi045 conmon[33655]: 575696+0000 mon.a (mon.0) 1120 : cluster [DBG] 2021-07-31T01:52:47.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:46 smithi045 conmon[33655]: osdmap e467: 8 total, 8 up, 8 in 2021-07-31T01:52:47.586 INFO:tasks.rados.rados.0.smithi045.stdout:2160: done (0 left) 2021-07-31T01:52:47.586 INFO:tasks.rados.rados.0.smithi045.stdout:2161: copy_from oid 33 from oid 14 current snap is 209 2021-07-31T01:52:47.586 INFO:tasks.rados.rados.0.smithi045.stdout:2162: rmattr oid 20 current snap is 209 2021-07-31T01:52:47.586 INFO:tasks.rados.rados.0.smithi045.stdout:2162: done (1 left) 2021-07-31T01:52:47.587 INFO:tasks.rados.rados.0.smithi045.stdout:2163: snap_remove snap 200 2021-07-31T01:52:48.591 INFO:tasks.rados.rados.0.smithi045.stdout:2163: done (1 left) 2021-07-31T01:52:48.591 INFO:tasks.rados.rados.0.smithi045.stdout:2164: snap_create 2021-07-31T01:52:48.592 INFO:tasks.rados.rados.0.smithi045.stdout:2161: finishing copy_from to smithi04569269-33 2021-07-31T01:52:48.592 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 33 v 1054 (ObjNum 659 snap 205 seq_num 659) dirty exists 2021-07-31T01:52:48.592 INFO:tasks.rados.rados.0.smithi045.stdout:2161: finishing copy_from racing read to smithi04569269-33 2021-07-31T01:52:49.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:48 smithi154 conmon[34553]: cluster 2021-07-31T01:52:47.584257+0000 mon.a (mon.0) 1121 : 2021-07-31T01:52:49.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:48 smithi154 conmon[34553]: cluster [DBG] osdmap e468: 8 total, 8 up, 8 in 2021-07-31T01:52:49.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:48 smithi154 conmon[34553]: cluster 2021-07-31T01:52:47.869603+0000 mgr.x (mgr.24542) 239 : cluster [DBG] pgmap v482: 25 pgs: 25 active+clean; 226 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2021-07-31T01:52:49.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:48 smithi045 conmon[33655]: cluster 2021-07-31T01:52:47.584257+0000 mon.a 2021-07-31T01:52:49.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:48 smithi045 conmon[33655]: (mon.0) 1121 : cluster [DBG] osdmap e468: 8 total, 8 up, 8 in 2021-07-31T01:52:49.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:48 smithi045 conmon[33655]: cluster 2021-07-31T01:52:47.869603+0000 mgr.x (mgr.24542) 239 : cluster [DBG] pgmap v482: 25 pgs: 25 active+clean; 226 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2021-07-31T01:52:49.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:48 smithi045 conmon[39718]: cluster 2021-07-31T01:52:47.584257+0000 mon.a ( 2021-07-31T01:52:49.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:48 smithi045 conmon[39718]: mon.0) 1121 : cluster [DBG] osdmap e468: 8 total, 8 up, 8 in 2021-07-31T01:52:49.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:48 smithi045 conmon[39718]: cluster 2021-07-31T01:52:47 2021-07-31T01:52:49.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:48 smithi045 conmon[39718]: .869603+0000 mgr.x (mgr.24542) 239 : cluster [DBG] pgmap v482: 25 pgs: 25 active+clean; 226 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2021-07-31T01:52:49.596 INFO:tasks.rados.rados.0.smithi045.stdout:2161: done (1 left) 2021-07-31T01:52:49.596 INFO:tasks.rados.rados.0.smithi045.stdout:2164: done (0 left) 2021-07-31T01:52:49.597 INFO:tasks.rados.rados.0.smithi045.stdout:2165: read oid 47 snap -1 2021-07-31T01:52:49.597 INFO:tasks.rados.rados.0.smithi045.stdout:2165: expect deleted 2021-07-31T01:52:49.599 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 12 current snap is 210 2021-07-31T01:52:49.599 INFO:tasks.rados.rados.0.smithi045.stdout:2166: seq_num 672 ranges {3964928=458752,4423680=647168,5070848=466944,5537792=90112} 2021-07-31T01:52:49.620 INFO:tasks.rados.rados.0.smithi045.stdout:2166: writing smithi04569269-12 from 3964928 to 4423680 tid 1 2021-07-31T01:52:49.623 INFO:tasks.rados.rados.0.smithi045.stdout:2166: writing smithi04569269-12 from 4423680 to 5070848 tid 2 2021-07-31T01:52:49.626 INFO:tasks.rados.rados.0.smithi045.stdout:2166: writing smithi04569269-12 from 5070848 to 5537792 tid 3 2021-07-31T01:52:49.627 INFO:tasks.rados.rados.0.smithi045.stdout:2166: writing smithi04569269-12 from 5537792 to 5627904 tid 4 2021-07-31T01:52:49.627 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 8 current snap is 210 2021-07-31T01:52:49.627 INFO:tasks.rados.rados.0.smithi045.stdout:2167: seq_num 673 ranges {0=475136,475136=532480,1007616=671744} 2021-07-31T01:52:49.629 INFO:tasks.rados.rados.0.smithi045.stdout:2167: writing smithi04569269-8 from 0 to 475136 tid 1 2021-07-31T01:52:49.632 INFO:tasks.rados.rados.0.smithi045.stdout:2167: writing smithi04569269-8 from 475136 to 1007616 tid 2 2021-07-31T01:52:49.637 INFO:tasks.rados.rados.0.smithi045.stdout:2167: writing smithi04569269-8 from 1007616 to 1679360 tid 3 2021-07-31T01:52:49.637 INFO:tasks.rados.rados.0.smithi045.stdout:2168: delete oid 31 current snap is 210 2021-07-31T01:52:49.637 INFO:tasks.rados.rados.0.smithi045.stdout:2166: finishing write tid 1 to smithi04569269-12 2021-07-31T01:52:49.638 INFO:tasks.rados.rados.0.smithi045.stdout:2166: finishing write tid 2 to smithi04569269-12 2021-07-31T01:52:49.638 INFO:tasks.rados.rados.0.smithi045.stdout:2166: finishing write tid 3 to smithi04569269-12 2021-07-31T01:52:49.638 INFO:tasks.rados.rados.0.smithi045.stdout:2166: finishing write tid 4 to smithi04569269-12 2021-07-31T01:52:49.638 INFO:tasks.rados.rados.0.smithi045.stdout:2166: finishing write tid 5 to smithi04569269-12 2021-07-31T01:52:49.638 INFO:tasks.rados.rados.0.smithi045.stdout:2166: finishing write tid 6 to smithi04569269-12 2021-07-31T01:52:49.639 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 12 v 841 (ObjNum 672 snap 210 seq_num 672) dirty exists 2021-07-31T01:52:49.639 INFO:tasks.rados.rados.0.smithi045.stdout:2166: left oid 12 (ObjNum 672 snap 210 seq_num 672) 2021-07-31T01:52:49.639 INFO:tasks.rados.rados.0.smithi045.stdout:2167: finishing write tid 1 to smithi04569269-8 2021-07-31T01:52:49.640 INFO:tasks.rados.rados.0.smithi045.stdout:2167: finishing write tid 2 to smithi04569269-8 2021-07-31T01:52:49.640 INFO:tasks.rados.rados.0.smithi045.stdout:2167: finishing write tid 3 to smithi04569269-8 2021-07-31T01:52:49.641 INFO:tasks.rados.rados.0.smithi045.stdout:2167: finishing write tid 4 to smithi04569269-8 2021-07-31T01:52:49.642 INFO:tasks.rados.rados.0.smithi045.stdout:2167: finishing write tid 5 to smithi04569269-8 2021-07-31T01:52:49.642 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 8 v 1358 (ObjNum 673 snap 210 seq_num 673) dirty exists 2021-07-31T01:52:49.642 INFO:tasks.rados.rados.0.smithi045.stdout:2167: left oid 8 (ObjNum 673 snap 210 seq_num 673) 2021-07-31T01:52:49.643 INFO:tasks.rados.rados.0.smithi045.stdout:2165: done (3 left) 2021-07-31T01:52:49.643 INFO:tasks.rados.rados.0.smithi045.stdout:2166: done (2 left) 2021-07-31T01:52:49.644 INFO:tasks.rados.rados.0.smithi045.stdout:2167: done (1 left) 2021-07-31T01:52:49.644 INFO:tasks.rados.rados.0.smithi045.stdout:2168: done (0 left) 2021-07-31T01:52:49.644 INFO:tasks.rados.rados.0.smithi045.stdout:2169: delete oid 43 current snap is 210 2021-07-31T01:52:49.647 INFO:tasks.rados.rados.0.smithi045.stdout:2169: done (0 left) 2021-07-31T01:52:49.647 INFO:tasks.rados.rados.0.smithi045.stdout:2170: read oid 13 snap -1 2021-07-31T01:52:49.647 INFO:tasks.rados.rados.0.smithi045.stdout:2170: expect (ObjNum 663 snap 208 seq_num 663) 2021-07-31T01:52:49.648 INFO:tasks.rados.rados.0.smithi045.stdout:2171: rmattr oid 41 current snap is 210 2021-07-31T01:52:49.648 INFO:tasks.rados.rados.0.smithi045.stdout:2172: snap_remove snap 192 2021-07-31T01:52:50.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:49 smithi154 conmon[34553]: cluster 2021-07-31T01:52:48.589879+0000 mon.a (mon.0) 1122 : cluster [DBG] 2021-07-31T01:52:50.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:49 smithi154 conmon[34553]: osdmap e469: 8 total, 8 up, 8 in 2021-07-31T01:52:50.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:49 smithi045 conmon[33655]: cluster 2021-07-31T01:52:48.589879+0000 mon.a (mon. 2021-07-31T01:52:50.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:49 smithi045 conmon[33655]: 0) 1122 : cluster [DBG] osdmap e469: 8 total, 8 up, 8 in 2021-07-31T01:52:50.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:49 smithi045 conmon[39718]: cluster 2021-07-31T01:52:48.589879+0000 mon.a (mon 2021-07-31T01:52:50.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:49 smithi045 conmon[39718]: .0) 1122 : cluster [DBG] osdmap e469: 8 total, 8 up, 8 in 2021-07-31T01:52:50.605 INFO:tasks.rados.rados.0.smithi045.stdout:2172: done (2 left) 2021-07-31T01:52:50.606 INFO:tasks.rados.rados.0.smithi045.stdout:2173: rollback oid 2 current snap is 210 2021-07-31T01:52:50.606 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 2 to 203 2021-07-31T01:52:50.607 INFO:tasks.rados.rados.0.smithi045.stdout:2174: copy_from oid 36 from oid 37 current snap is 210 2021-07-31T01:52:50.607 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 41 v 985 (ObjNum 639 snap 199 seq_num 639) dirty exists 2021-07-31T01:52:50.607 INFO:tasks.rados.rados.0.smithi045.stdout:2170: expect (ObjNum 663 snap 208 seq_num 663) 2021-07-31T01:52:51.003 INFO:journalctl@ceph.osd.5.smithi154.stdout:Jul 31 01:52:50 smithi154 conmon[42050]: debug 2021-07-31T01:52:50.612+0000 7fc9b105c700 -1 osd.5 pg_epoch: 471 pg[3.cs0( v 471'850 (0'0,471'850] local-lis/les=56/57 n=16 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [5,3,6]p5(0) r=0 lpr=56 luod=469'848 crt=469'848 lcod 468'846 mlcod 468'846 active+clean+snaptrim trimq=[17e~1] ps=[17f~1,18a~2,193~1]] removing snap head 2021-07-31T01:52:51.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:50 smithi154 conmon[34553]: cluster 2021-07-31T01:52:49.595320+0000 mon.a (mon.0) 2021-07-31T01:52:51.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:50 smithi154 conmon[34553]: 1123 : cluster [DBG] osdmap e470: 8 total, 8 up, 8 in 2021-07-31T01:52:51.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:50 smithi154 conmon[34553]: cluster 2021-07-31T01:52:49.870106+0000 mgr.x (mgr.24542) 240 : cluster [DBG] pgmap v485: 25 pgs: 25 active+clean; 243 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.5 MiB/s rd, 4.6 MiB/s wr, 6 op/s 2021-07-31T01:52:51.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:50 smithi045 conmon[33655]: cluster 2021-07-31T01:52:49.595320 2021-07-31T01:52:51.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:50 smithi045 conmon[33655]: +0000 mon.a (mon.0) 1123 : cluster [DBG] osdmap e470: 8 total, 8 up, 8 in 2021-07-31T01:52:51.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:50 smithi045 conmon[33655]: cluster 2021-07-31T01:52:49.870106+0000 mgr.x (mgr.24542) 240 : cluster 2021-07-31T01:52:51.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:50 smithi045 conmon[33655]: [DBG] pgmap v485: 25 pgs: 25 active+clean; 243 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.5 MiB/s rd, 4.6 MiB/s wr, 6 op/s 2021-07-31T01:52:51.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:50 smithi045 conmon[39718]: cluster 2021-07-31T01:52:49.595320 2021-07-31T01:52:51.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:50 smithi045 conmon[39718]: +0000 mon.a (mon.0) 1123 : cluster [DBG] osdmap e470: 8 total, 8 up, 8 in 2021-07-31T01:52:51.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:50 smithi045 conmon[39718]: cluster 2021-07-31T01: 2021-07-31T01:52:51.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:50 smithi045 conmon[39718]: 52:49.870106+0000 mgr.x (mgr.24542) 240 : cluster [DBG] 2021-07-31T01:52:51.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:50 smithi045 conmon[39718]: pgmap v485: 25 pgs: 25 active+clean; 243 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.5 MiB/s rd, 4.6 MiB/s wr, 6 op/s 2021-07-31T01:52:52.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:51 smithi154 conmon[34553]: cluster 2021-07-31T01:52:50.603248+0000 mon.a (mon.0) 1124 : cluster [DBG] osdmap e471: 8 total, 8 up, 8 in 2021-07-31T01:52:52.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:51 smithi154 conmon[34553]: cluster 2021-07-31T01:52:51.576487+0000 mon.a (mon.0) 1125 : cluster [DBG] osdmap e472: 8 total, 8 up, 8 in 2021-07-31T01:52:52.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:51 smithi045 conmon[33655]: cluster 2021-07-31T01:52:50.603248 2021-07-31T01:52:52.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:51 smithi045 conmon[33655]: +0000 mon.a (mon.0) 1124 : cluster [DBG] osdmap e471: 8 total, 8 up, 8 in 2021-07-31T01:52:52.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:51 smithi045 conmon[33655]: cluster 2021-07-31T01:52:51.576487+0000 mon.a (mon.0) 1125 : cluster [DBG] osdmap e472: 8 total, 8 up, 8 in 2021-07-31T01:52:52.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:51 smithi045 conmon[39718]: cluster 2021-07-31T01:52:50.603248+0000 2021-07-31T01:52:52.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:51 smithi045 conmon[39718]: mon.a (mon.0) 1124 : cluster [DBG] osdmap e471: 8 total, 8 up, 8 in 2021-07-31T01:52:52.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:51 smithi045 conmon[39718]: cluster 2021-07-31T01:52:52.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:51 smithi045 conmon[39718]: 2021-07-31T01:52:51.576487+0000 mon.a (mon.0 2021-07-31T01:52:52.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:51 smithi045 conmon[39718]: ) 1125 : cluster [DBG] osdmap e472: 8 total, 8 up, 8 in 2021-07-31T01:52:52.294 INFO:tasks.rados.rados.0.smithi045.stdout:2173: finishing rollback tid 0 to smithi04569269-2 2021-07-31T01:52:52.295 INFO:tasks.rados.rados.0.smithi045.stdout:2173: finishing rollback tid 1 to smithi04569269-2 2021-07-31T01:52:52.295 INFO:tasks.rados.rados.0.smithi045.stdout:2173: finishing rollback tid 2 to smithi04569269-2 2021-07-31T01:52:52.295 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 2 v 1045 (ObjNum 642 snap 199 seq_num 642) dirty exists 2021-07-31T01:52:52.295 INFO:tasks.rados.rados.0.smithi045.stdout:2174: finishing copy_from to smithi04569269-36 2021-07-31T01:52:52.296 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 36 v 527 (ObjNum 645 snap 199 seq_num 645) dirty exists 2021-07-31T01:52:52.296 INFO:tasks.rados.rados.0.smithi045.stdout:2174: finishing copy_from racing read to smithi04569269-36 2021-07-31T01:52:52.296 INFO:tasks.rados.rados.0.smithi045.stdout:2170: done (3 left) 2021-07-31T01:52:52.296 INFO:tasks.rados.rados.0.smithi045.stdout:2171: done (2 left) 2021-07-31T01:52:52.296 INFO:tasks.rados.rados.0.smithi045.stdout:2173: done (1 left) 2021-07-31T01:52:52.297 INFO:tasks.rados.rados.0.smithi045.stdout:2174: done (0 left) 2021-07-31T01:52:52.297 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 44 current snap is 210 2021-07-31T01:52:52.297 INFO:tasks.rados.rados.0.smithi045.stdout:2175: seq_num 676 ranges {0=524288,524288=589824,1114112=450560,1564672=163840} 2021-07-31T01:52:52.298 INFO:tasks.rados.rados.0.smithi045.stdout:2175: writing smithi04569269-44 from 0 to 524288 tid 1 2021-07-31T01:52:52.300 INFO:tasks.rados.rados.0.smithi045.stdout:2175: writing smithi04569269-44 from 524288 to 1114112 tid 2 2021-07-31T01:52:52.303 INFO:tasks.rados.rados.0.smithi045.stdout:2175: writing smithi04569269-44 from 1114112 to 1564672 tid 3 2021-07-31T01:52:52.304 INFO:tasks.rados.rados.0.smithi045.stdout:2175: writing smithi04569269-44 from 1564672 to 1728512 tid 4 2021-07-31T01:52:52.304 INFO:tasks.rados.rados.0.smithi045.stdout:2176: snap_create 2021-07-31T01:52:52.304 INFO:tasks.rados.rados.0.smithi045.stdout:2175: finishing write tid 1 to smithi04569269-44 2021-07-31T01:52:52.305 INFO:tasks.rados.rados.0.smithi045.stdout:2175: finishing write tid 2 to smithi04569269-44 2021-07-31T01:52:52.307 INFO:tasks.rados.rados.0.smithi045.stdout:2175: finishing write tid 3 to smithi04569269-44 2021-07-31T01:52:52.307 INFO:tasks.rados.rados.0.smithi045.stdout:2175: finishing write tid 4 to smithi04569269-44 2021-07-31T01:52:52.308 INFO:tasks.rados.rados.0.smithi045.stdout:2175: finishing write tid 5 to smithi04569269-44 2021-07-31T01:52:52.309 INFO:tasks.rados.rados.0.smithi045.stdout:2175: finishing write tid 6 to smithi04569269-44 2021-07-31T01:52:52.309 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 44 v 1065 (ObjNum 676 snap 210 seq_num 676) dirty exists 2021-07-31T01:52:52.309 INFO:tasks.rados.rados.0.smithi045.stdout:2175: left oid 44 (ObjNum 676 snap 210 seq_num 676) 2021-07-31T01:52:52.611 INFO:tasks.rados.rados.0.smithi045.stdout:2175: done (1 left) 2021-07-31T01:52:52.612 INFO:tasks.rados.rados.0.smithi045.stdout:2176: done (0 left) 2021-07-31T01:52:52.612 INFO:tasks.rados.rados.0.smithi045.stdout:2177: read oid 28 snap -1 2021-07-31T01:52:52.612 INFO:tasks.rados.rados.0.smithi045.stdout:2177: expect deleted 2021-07-31T01:52:52.613 INFO:tasks.rados.rados.0.smithi045.stdout:2178: read oid 50 snap -1 2021-07-31T01:52:52.613 INFO:tasks.rados.rados.0.smithi045.stdout:2178: expect (ObjNum 635 snap 198 seq_num 635) 2021-07-31T01:52:52.613 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 15 current snap is 211 2021-07-31T01:52:52.613 INFO:tasks.rados.rados.0.smithi045.stdout:2179: seq_num 677 ranges {6111232=540672,6651904=516096,7168000=655360,7823360=32768} 2021-07-31T01:52:52.647 INFO:tasks.rados.rados.0.smithi045.stdout:2179: writing smithi04569269-15 from 6111232 to 6651904 tid 1 2021-07-31T01:52:52.649 INFO:tasks.rados.rados.0.smithi045.stdout:2179: writing smithi04569269-15 from 6651904 to 7168000 tid 2 2021-07-31T01:52:52.654 INFO:tasks.rados.rados.0.smithi045.stdout:2179: writing smithi04569269-15 from 7168000 to 7823360 tid 3 2021-07-31T01:52:52.654 INFO:tasks.rados.rados.0.smithi045.stdout:2179: writing smithi04569269-15 from 7823360 to 7856128 tid 4 2021-07-31T01:52:52.654 INFO:tasks.rados.rados.0.smithi045.stdout:2180: snap_remove snap 209 2021-07-31T01:52:53.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:52 smithi154 conmon[34553]: cluster 2021-07-31T01:52:51.870613+0000 mgr.x (mgr.24542 2021-07-31T01:52:53.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:52 smithi154 conmon[34553]: ) 241 : cluster [DBG] pgmap v488: 25 pgs: 25 active+clean; 243 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.5 MiB/s rd, 4.6 MiB/s wr, 6 op/s 2021-07-31T01:52:53.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:52 smithi045 conmon[33655]: cluster 2021-07-31T01:52:51 2021-07-31T01:52:53.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:52 smithi045 conmon[33655]: .870613+0000 mgr.x (mgr.24542) 241 : cluster [DBG] pgmap v488: 25 pgs: 25 active+clean; 243 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.5 MiB/s rd, 4.6 MiB/s wr, 6 op/s 2021-07-31T01:52:53.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:52 smithi045 conmon[39718]: cluster 2021-07-31T01:52: 2021-07-31T01:52:53.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:52 smithi045 conmon[39718]: 51.870613+0000 mgr.x (mgr.24542) 241 : cluster [DBG] pgmap v488: 25 pgs: 25 active+clean; 243 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.5 MiB/s rd, 4.6 MiB/s wr, 6 op/s 2021-07-31T01:52:53.617 INFO:tasks.rados.rados.0.smithi045.stdout:2180: done (3 left) 2021-07-31T01:52:53.618 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 25 current snap is 211 2021-07-31T01:52:53.618 INFO:tasks.rados.rados.0.smithi045.stdout:2181: seq_num 678 ranges {6029312=565248,6594560=802816,7397376=401408} 2021-07-31T01:52:53.654 INFO:tasks.rados.rados.0.smithi045.stdout:2181: writing smithi04569269-25 from 6029312 to 6594560 tid 1 2021-07-31T01:52:53.659 INFO:tasks.rados.rados.0.smithi045.stdout:2181: writing smithi04569269-25 from 6594560 to 7397376 tid 2 2021-07-31T01:52:53.660 INFO:tasks.rados.rados.0.smithi045.stdout:2181: writing smithi04569269-25 from 7397376 to 7798784 tid 3 2021-07-31T01:52:53.661 INFO:tasks.rados.rados.0.smithi045.stdout:2182: rollback oid 23 current snap is 211 2021-07-31T01:52:53.661 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 23 to 203 2021-07-31T01:52:53.662 INFO:tasks.rados.rados.0.smithi045.stdout:2178: expect (ObjNum 635 snap 198 seq_num 635) 2021-07-31T01:52:53.770 INFO:tasks.rados.rados.0.smithi045.stdout:2179: finishing write tid 1 to smithi04569269-15 2021-07-31T01:52:53.770 INFO:tasks.rados.rados.0.smithi045.stdout:2179: finishing write tid 2 to smithi04569269-15 2021-07-31T01:52:53.771 INFO:tasks.rados.rados.0.smithi045.stdout:2179: finishing write tid 3 to smithi04569269-15 2021-07-31T01:52:53.771 INFO:tasks.rados.rados.0.smithi045.stdout:2179: finishing write tid 4 to smithi04569269-15 2021-07-31T01:52:53.771 INFO:tasks.rados.rados.0.smithi045.stdout:2179: finishing write tid 5 to smithi04569269-15 2021-07-31T01:52:53.771 INFO:tasks.rados.rados.0.smithi045.stdout:2179: finishing write tid 6 to smithi04569269-15 2021-07-31T01:52:53.772 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 15 v 573 (ObjNum 677 snap 211 seq_num 677) dirty exists 2021-07-31T01:52:53.772 INFO:tasks.rados.rados.0.smithi045.stdout:2179: left oid 15 (ObjNum 677 snap 211 seq_num 677) 2021-07-31T01:52:53.772 INFO:tasks.rados.rados.0.smithi045.stdout:2181: finishing write tid 1 to smithi04569269-25 2021-07-31T01:52:53.772 INFO:tasks.rados.rados.0.smithi045.stdout:2181: finishing write tid 2 to smithi04569269-25 2021-07-31T01:52:53.772 INFO:tasks.rados.rados.0.smithi045.stdout:2177: done (4 left) 2021-07-31T01:52:53.773 INFO:tasks.rados.rados.0.smithi045.stdout:2178: done (3 left) 2021-07-31T01:52:53.773 INFO:tasks.rados.rados.0.smithi045.stdout:2179: done (2 left) 2021-07-31T01:52:53.773 INFO:tasks.rados.rados.0.smithi045.stdout:2183: read oid 26 snap 198 2021-07-31T01:52:53.773 INFO:tasks.rados.rados.0.smithi045.stdout:2183: expect (ObjNum 172 snap 41 seq_num 172) 2021-07-31T01:52:53.774 INFO:tasks.rados.rados.0.smithi045.stdout:2184: copy_from oid 3 from oid 49 current snap is 211 2021-07-31T01:52:53.774 INFO:tasks.rados.rados.0.smithi045.stdout:2181: finishing write tid 3 to smithi04569269-25 2021-07-31T01:52:53.774 INFO:tasks.rados.rados.0.smithi045.stdout:2181: finishing write tid 4 to smithi04569269-25 2021-07-31T01:52:53.775 INFO:tasks.rados.rados.0.smithi045.stdout:2182: finishing rollback tid 1 to smithi04569269-23 2021-07-31T01:52:53.775 INFO:tasks.rados.rados.0.smithi045.stdout:2182: finishing rollback tid 2 to smithi04569269-23 2021-07-31T01:52:53.776 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 23 v 1149 (ObjNum 656 snap 203 seq_num 656) dirty exists 2021-07-31T01:52:53.776 INFO:tasks.rados.rados.0.smithi045.stdout:2181: finishing write tid 5 to smithi04569269-25 2021-07-31T01:52:53.776 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 25 v 305 (ObjNum 678 snap 211 seq_num 678) dirty exists 2021-07-31T01:52:53.777 INFO:tasks.rados.rados.0.smithi045.stdout:2181: left oid 25 (ObjNum 678 snap 211 seq_num 678) 2021-07-31T01:52:53.777 INFO:tasks.rados.rados.0.smithi045.stdout:2181: done (3 left) 2021-07-31T01:52:53.777 INFO:tasks.rados.rados.0.smithi045.stdout:2182: done (2 left) 2021-07-31T01:52:53.777 INFO:tasks.rados.rados.0.smithi045.stdout:2185: read oid 29 snap -1 2021-07-31T01:52:53.778 INFO:tasks.rados.rados.0.smithi045.stdout:2185: expect deleted 2021-07-31T01:52:53.778 INFO:tasks.rados.rados.0.smithi045.stdout:2186: copy_from oid 5 from oid 19 current snap is 211 2021-07-31T01:52:53.778 INFO:tasks.rados.rados.0.smithi045.stdout:2187: rmattr oid 21 current snap is 211 2021-07-31T01:52:53.779 INFO:tasks.rados.rados.0.smithi045.stdout:2187: done (4 left) 2021-07-31T01:52:53.779 INFO:tasks.rados.rados.0.smithi045.stdout:2188: read oid 38 snap 210 2021-07-31T01:52:53.779 INFO:tasks.rados.rados.0.smithi045.stdout:2188: expect deleted 2021-07-31T01:52:53.780 INFO:tasks.rados.rados.0.smithi045.stdout:2189: rollback oid 21 current snap is 211 2021-07-31T01:52:53.780 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 21 to 203 2021-07-31T01:52:53.780 INFO:tasks.rados.rados.0.smithi045.stdout:2190: rollback oid 23 current snap is 211 2021-07-31T01:52:53.780 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 23 to 182 2021-07-31T01:52:53.781 INFO:tasks.rados.rados.0.smithi045.stdout:2191: snap_create 2021-07-31T01:52:53.781 INFO:tasks.rados.rados.0.smithi045.stdout:2183: expect (ObjNum 172 snap 41 seq_num 172) 2021-07-31T01:52:53.782 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:52:53 smithi045 conmon[61210]: level=error ts=2021-07-31T01:52:53.518Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:52:53.782 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:52:53 smithi045 conmon[61210]: level=error ts=2021-07-31T01:52:53.518Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:52:53.782 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:52:53 smithi045 conmon[61210]: level=error ts=2021-07-31T01:52:53.518Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T01:52:53.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:53 smithi045 conmon[33655]: cluster 2021-07-31T01:52:52.609326+0000 mon.a (mon.0) 1126 : cluster 2021-07-31T01:52:53.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:53 smithi045 conmon[33655]: [DBG] osdmap e473: 8 total, 8 up, 8 in 2021-07-31T01:52:53.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:53 smithi045 conmon[39718]: cluster 2021-07-31T01:52:52.609326+0000 mon.a ( 2021-07-31T01:52:53.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:53 smithi045 conmon[39718]: mon.0) 1126 : cluster [DBG] osdmap e473: 8 total, 8 up, 8 in 2021-07-31T01:52:53.795 INFO:tasks.rados.rados.0.smithi045.stdout:2189: finishing rollback tid 0 to smithi04569269-21 2021-07-31T01:52:53.796 INFO:tasks.rados.rados.0.smithi045.stdout:2190: finishing rollback tid 0 to smithi04569269-23 2021-07-31T01:52:53.796 INFO:tasks.rados.rados.0.smithi045.stdout:2190: finishing rollback tid 1 to smithi04569269-23 2021-07-31T01:52:53.796 INFO:tasks.rados.rados.0.smithi045.stdout:2190: finishing rollback tid 2 to smithi04569269-23 2021-07-31T01:52:53.796 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 23 v 1152 (ObjNum 560 snap 166 seq_num 560) dirty exists 2021-07-31T01:52:53.797 INFO:tasks.rados.rados.0.smithi045.stdout:2186: finishing copy_from to smithi04569269-5 2021-07-31T01:52:53.798 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 5 v 275 (ObjNum 172 snap 41 seq_num 172) dirty exists 2021-07-31T01:52:53.798 INFO:tasks.rados.rados.0.smithi045.stdout:2186: finishing copy_from racing read to smithi04569269-5 2021-07-31T01:52:53.799 INFO:tasks.rados.rados.0.smithi045.stdout:2189: finishing rollback tid 1 to smithi04569269-21 2021-07-31T01:52:53.799 INFO:tasks.rados.rados.0.smithi045.stdout:2189: finishing rollback tid 2 to smithi04569269-21 2021-07-31T01:52:53.799 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 21 v 1375 (ObjNum 589 snap 182 seq_num 589) dirty exists 2021-07-31T01:52:53.801 INFO:tasks.rados.rados.0.smithi045.stdout:2184: finishing copy_from to smithi04569269-3 2021-07-31T01:52:53.801 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 3 v 1319 (ObjNum 647 snap 201 seq_num 647) dirty exists 2021-07-31T01:52:53.801 INFO:tasks.rados.rados.0.smithi045.stdout:2184: finishing copy_from racing read to smithi04569269-3 2021-07-31T01:52:54.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:53 smithi154 conmon[34553]: cluster 2021-07-31T01:52:52.609326+0000 mon.a (mon.0) 1126 : cluster [DBG] osdmap e473: 8 total, 8 up, 8 in 2021-07-31T01:52:54.623 INFO:tasks.rados.rados.0.smithi045.stdout:2183: done (7 left) 2021-07-31T01:52:54.624 INFO:tasks.rados.rados.0.smithi045.stdout:2184: done (6 left) 2021-07-31T01:52:54.624 INFO:tasks.rados.rados.0.smithi045.stdout:2185: done (5 left) 2021-07-31T01:52:54.624 INFO:tasks.rados.rados.0.smithi045.stdout:2186: done (4 left) 2021-07-31T01:52:54.625 INFO:tasks.rados.rados.0.smithi045.stdout:2188: done (3 left) 2021-07-31T01:52:54.625 INFO:tasks.rados.rados.0.smithi045.stdout:2189: done (2 left) 2021-07-31T01:52:54.625 INFO:tasks.rados.rados.0.smithi045.stdout:2190: done (1 left) 2021-07-31T01:52:54.625 INFO:tasks.rados.rados.0.smithi045.stdout:2191: done (0 left) 2021-07-31T01:52:54.625 INFO:tasks.rados.rados.0.smithi045.stdout:2192: copy_from oid 12 from oid 43 current snap is 212 2021-07-31T01:52:54.626 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 49 current snap is 212 2021-07-31T01:52:54.626 INFO:tasks.rados.rados.0.smithi045.stdout:2193: seq_num 679 ranges {6701056=581632,7282688=688128,7970816=516096} 2021-07-31T01:52:54.663 INFO:tasks.rados.rados.0.smithi045.stdout:2193: writing smithi04569269-49 from 6701056 to 7282688 tid 1 2021-07-31T01:52:54.667 INFO:tasks.rados.rados.0.smithi045.stdout:2193: writing smithi04569269-49 from 7282688 to 7970816 tid 2 2021-07-31T01:52:54.670 INFO:tasks.rados.rados.0.smithi045.stdout:2193: writing smithi04569269-49 from 7970816 to 8486912 tid 3 2021-07-31T01:52:54.671 INFO:tasks.rados.rados.0.smithi045.stdout:2194: copy_from oid 6 from oid 21 current snap is 212 2021-07-31T01:52:54.671 INFO:tasks.rados.rados.0.smithi045.stdout:2192: finishing copy_from to smithi04569269-12 2021-07-31T01:52:54.671 INFO:tasks.rados.rados.0.smithi045.stdout:2192: got expected ENOENT (src dne) 2021-07-31T01:52:54.671 INFO:tasks.rados.rados.0.smithi045.stdout:2192: finishing copy_from racing read to smithi04569269-12 2021-07-31T01:52:54.671 INFO:tasks.rados.rados.0.smithi045.stdout:2192: done (2 left) 2021-07-31T01:52:54.672 INFO:tasks.rados.rados.0.smithi045.stdout:2195: snap_remove snap 198 2021-07-31T01:52:55.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:54 smithi154 conmon[34553]: cluster 2021-07-31T01:52:53.616308+0000 mon.a (mon.0) 1127 : cluster [DBG] 2021-07-31T01:52:55.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:54 smithi154 conmon[34553]: osdmap e474: 8 total, 8 up, 8 in 2021-07-31T01:52:55.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:54 smithi154 conmon[34553]: cluster 2021-07-31T01:52:53.871198+0000 mgr.x (mgr.24542) 242 : cluster [DBG] pgmap v491: 25 pgs: 25 active+clean; 262 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 4.0 MiB/s wr, 14 op/s 2021-07-31T01:52:55.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:54 smithi045 conmon[33655]: cluster 2021-07-31T01:52:53.616308+0000 mon.a (mon.0) 1127 2021-07-31T01:52:55.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:54 smithi045 conmon[33655]: : cluster [DBG] osdmap e474: 8 total, 8 up, 8 in 2021-07-31T01:52:55.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:54 smithi045 conmon[33655]: cluster 2021-07-31T01:52:53.871198+0000 mgr.x (mgr.24542) 242 : cluster [DBG] pgmap v491: 25 pgs: 25 active+clean; 262 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 4.0 MiB/s wr, 14 op/s 2021-07-31T01:52:55.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:54 smithi045 conmon[39718]: cluster 2021-07-31T01:52:53.616308+0000 2021-07-31T01:52:55.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:54 smithi045 conmon[39718]: mon.a (mon.0) 1127 : cluster [DBG] osdmap e474: 8 total, 8 up, 8 in 2021-07-31T01:52:55.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:54 smithi045 conmon[39718]: cluster 2021-07-31T01:52:53.871198+0000 2021-07-31T01:52:55.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:54 smithi045 conmon[39718]: mgr.x (mgr.24542) 242 : cluster [DBG] pgmap v491: 25 pgs: 25 active+clean; 262 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 4.0 MiB/s wr, 14 op/s 2021-07-31T01:52:55.630 INFO:tasks.rados.rados.0.smithi045.stdout:2195: done (2 left) 2021-07-31T01:52:55.630 INFO:tasks.rados.rados.0.smithi045.stdout:2196: snap_remove snap 210 2021-07-31T01:52:56.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:55 smithi154 conmon[34553]: cluster 2021-07-31T01:52:54.622489+0000 mon.a (mon.0) 1128 : cluster [DBG] osdmap e475: 8 total, 8 up, 8 in 2021-07-31T01:52:56.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:55 smithi045 conmon[39718]: cluster 2021-07-31T01:52:54.622489+0000 mon.a ( 2021-07-31T01:52:56.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:55 smithi045 conmon[39718]: mon.0) 1128 : cluster [DBG] osdmap e475: 8 total, 8 up, 8 in 2021-07-31T01:52:56.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:55 smithi045 conmon[33655]: cluster 2021-07-31T01:52:54.622489 2021-07-31T01:52:56.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:55 smithi045 conmon[33655]: +0000 mon.a (mon.0) 1128 : cluster [DBG] osdmap e475: 8 total, 8 up, 8 in 2021-07-31T01:52:56.579 INFO:tasks.rados.rados.0.smithi045.stdout:2196: done (2 left) 2021-07-31T01:52:56.580 INFO:tasks.rados.rados.0.smithi045.stdout:2197: read oid 13 snap 211 2021-07-31T01:52:56.580 INFO:tasks.rados.rados.0.smithi045.stdout:2197: expect (ObjNum 663 snap 208 seq_num 663) 2021-07-31T01:52:56.580 INFO:tasks.rados.rados.0.smithi045.stdout:2198: read oid 42 snap -1 2021-07-31T01:52:56.580 INFO:tasks.rados.rados.0.smithi045.stdout:2198: expect (ObjNum 483 snap 143 seq_num 483) 2021-07-31T01:52:56.581 INFO:tasks.rados.rados.0.smithi045.stdout:2199: snap_remove snap 203 2021-07-31T01:52:57.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:56 smithi154 conmon[34553]: cluster 2021-07-31T01:52:55.628150+0000 mon.a (mon.0) 2021-07-31T01:52:57.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:56 smithi154 conmon[34553]: 1129 : cluster [DBG] osdmap e476: 8 total, 8 up, 8 in 2021-07-31T01:52:57.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:56 smithi154 conmon[34553]: cluster 2021-07-31T01:52:55.871793+0000 mgr.x (mgr.24542) 243 : cluster [DBG] pgmap v494: 25 pgs: 25 active+clean; 262 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 4.0 MiB/s wr, 14 op/s 2021-07-31T01:52:57.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:56 smithi154 conmon[34553]: cluster 2021- 2021-07-31T01:52:57.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:56 smithi154 conmon[34553]: 07-31T01:52:56.578509+0000 mon.a (mon.0) 1130 : cluster [DBG] osdmap e477: 8 total, 8 up, 8 in 2021-07-31T01:52:57.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:56 smithi045 conmon[39718]: cluster 2021-07-31T01:52:55.628150+0000 2021-07-31T01:52:57.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:56 smithi045 conmon[39718]: mon.a (mon.0) 1129 : cluster [DBG] osdmap e476: 8 total, 8 up, 8 in 2021-07-31T01:52:57.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:56 smithi045 conmon[39718]: cluster 2021-07-31T01:52:55.871793+0000 mgr.x (mgr.24542) 243 : cluster [DBG] 2021-07-31T01:52:57.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:56 smithi045 conmon[39718]: pgmap v494: 25 pgs: 25 active+clean; 262 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 4.0 MiB/s wr, 14 op/s 2021-07-31T01:52:57.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:56 smithi045 conmon[39718]: cluster 2021- 2021-07-31T01:52:57.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:56 smithi045 conmon[39718]: 07-31T01:52:56.578509+0000 mon.a (mon.0) 1130 : cluster [DBG] osdmap e477: 8 total, 8 up, 8 in 2021-07-31T01:52:57.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:56 smithi045 conmon[33655]: cluster 2021-07-31T01:52:55.628150+0000 2021-07-31T01:52:57.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:56 smithi045 conmon[33655]: mon.a (mon.0) 1129 : cluster [DBG] osdmap e476: 8 total, 8 up, 8 in 2021-07-31T01:52:57.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:56 smithi045 conmon[33655]: cluster 2021-07-31T01:52:55.871793+0000 mgr.x (mgr.24542) 243 : cluster [DBG] pgmap v494: 25 pgs: 25 active+clean; 262 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 4.0 MiB/s wr, 14 op/s 2021-07-31T01:52:57.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:56 smithi045 conmon[33655]: cluster 2021-07-31T01:52:56.578509+0000 mon.a (mon.0) 1130 : cluster [DBG] osdmap e477: 8 total, 8 up, 8 in 2021-07-31T01:52:57.589 INFO:tasks.rados.rados.0.smithi045.stdout:2199: done (4 left) 2021-07-31T01:52:57.589 INFO:tasks.rados.rados.0.smithi045.stdout:2200: delete oid 35 current snap is 212 2021-07-31T01:52:57.590 INFO:tasks.rados.rados.0.smithi045.stdout:2193: finishing write tid 1 to smithi04569269-49 2021-07-31T01:52:57.590 INFO:tasks.rados.rados.0.smithi045.stdout:2193: finishing write tid 2 to smithi04569269-49 2021-07-31T01:52:57.591 INFO:tasks.rados.rados.0.smithi045.stdout:2193: finishing write tid 3 to smithi04569269-49 2021-07-31T01:52:57.591 INFO:tasks.rados.rados.0.smithi045.stdout:2193: finishing write tid 4 to smithi04569269-49 2021-07-31T01:52:57.591 INFO:tasks.rados.rados.0.smithi045.stdout:2193: finishing write tid 5 to smithi04569269-49 2021-07-31T01:52:57.592 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 49 v 1382 (ObjNum 679 snap 212 seq_num 679) dirty exists 2021-07-31T01:52:57.593 INFO:tasks.rados.rados.0.smithi045.stdout:2193: left oid 49 (ObjNum 679 snap 212 seq_num 679) 2021-07-31T01:52:57.594 INFO:tasks.rados.rados.0.smithi045.stdout:2194: finishing copy_from to smithi04569269-6 2021-07-31T01:52:57.594 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 6 v 1375 (ObjNum 589 snap 182 seq_num 589) dirty exists 2021-07-31T01:52:57.594 INFO:tasks.rados.rados.0.smithi045.stdout:2194: finishing copy_from racing read to smithi04569269-6 2021-07-31T01:52:57.595 INFO:tasks.rados.rados.0.smithi045.stdout:2198: expect (ObjNum 483 snap 143 seq_num 483) 2021-07-31T01:52:57.643 INFO:tasks.rados.rados.0.smithi045.stdout:2197: expect (ObjNum 663 snap 208 seq_num 663) 2021-07-31T01:52:59.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:58 smithi154 conmon[34553]: cluster 2021-07-31T01:52:57.584579+0000 mon.a (mon.0) 1131 : cluster [DBG] osdmap e478: 8 total, 8 up, 8 in 2021-07-31T01:52:59.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:52:58 smithi154 conmon[34553]: cluster 2021-07-31T01:52:57.872363+0000 mgr.x (mgr.24542) 244 : cluster [DBG] pgmap v497: 25 pgs: 25 active+clean; 262 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2021-07-31T01:52:59.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:58 smithi045 conmon[33655]: cluster 2021-07-31T01:52:57.584579+0000 mon.a 2021-07-31T01:52:59.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:58 smithi045 conmon[33655]: (mon.0) 1131 : cluster [DBG] osdmap e478: 8 total, 8 up, 8 in 2021-07-31T01:52:59.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:58 smithi045 conmon[33655]: cluster 2021-07-31T01:52:57.872363+0000 mgr.x 2021-07-31T01:52:59.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:52:58 smithi045 conmon[33655]: (mgr.24542) 244 : cluster [DBG] pgmap v497: 25 pgs: 25 active+clean; 262 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2021-07-31T01:52:59.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:58 smithi045 conmon[39718]: cluster 2021-07-31T01: 2021-07-31T01:52:59.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:58 smithi045 conmon[39718]: 52:57.584579+0000 mon.a (mon.0) 1131 : cluster [DBG] osdmap e478: 8 total, 8 up, 8 in 2021-07-31T01:52:59.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:58 smithi045 conmon[39718]: cluster 2021-07-31T01:52:57. 2021-07-31T01:52:59.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:52:58 smithi045 conmon[39718]: 872363+0000 mgr.x (mgr.24542) 244 : cluster [DBG] pgmap v497: 25 pgs: 25 active+clean; 262 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2021-07-31T01:52:59.321 INFO:tasks.rados.rados.0.smithi045.stdout:2193: done (4 left) 2021-07-31T01:52:59.321 INFO:tasks.rados.rados.0.smithi045.stdout:2194: done (3 left) 2021-07-31T01:52:59.321 INFO:tasks.rados.rados.0.smithi045.stdout:2197: done (2 left) 2021-07-31T01:52:59.321 INFO:tasks.rados.rados.0.smithi045.stdout:2198: done (1 left) 2021-07-31T01:52:59.322 INFO:tasks.rados.rados.0.smithi045.stdout:2200: done (0 left) 2021-07-31T01:52:59.322 INFO:tasks.rados.rados.0.smithi045.stdout:2201: copy_from oid 37 from oid 34 current snap is 212 2021-07-31T01:52:59.322 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 9 current snap is 212 2021-07-31T01:52:59.322 INFO:tasks.rados.rados.0.smithi045.stdout:2202: seq_num 681 ranges {0=614400,614400=458752,1073152=638976,1712128=106496} 2021-07-31T01:52:59.323 INFO:tasks.rados.rados.0.smithi045.stdout:2202: writing smithi04569269-9 from 0 to 614400 tid 1 2021-07-31T01:52:59.326 INFO:tasks.rados.rados.0.smithi045.stdout:2202: writing smithi04569269-9 from 614400 to 1073152 tid 2 2021-07-31T01:52:59.330 INFO:tasks.rados.rados.0.smithi045.stdout:2202: writing smithi04569269-9 from 1073152 to 1712128 tid 3 2021-07-31T01:52:59.331 INFO:tasks.rados.rados.0.smithi045.stdout:2202: writing smithi04569269-9 from 1712128 to 1818624 tid 4 2021-07-31T01:52:59.331 INFO:tasks.rados.rados.0.smithi045.stdout:2203: rollback oid 27 current snap is 212 2021-07-31T01:52:59.331 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 27 to 182 2021-07-31T01:52:59.332 INFO:tasks.rados.rados.0.smithi045.stdout:2202: finishing write tid 1 to smithi04569269-9 2021-07-31T01:52:59.332 INFO:tasks.rados.rados.0.smithi045.stdout:2202: finishing write tid 2 to smithi04569269-9 2021-07-31T01:52:59.332 INFO:tasks.rados.rados.0.smithi045.stdout:2204: delete oid 28 current snap is 212 2021-07-31T01:52:59.334 INFO:tasks.rados.rados.0.smithi045.stdout:2204: done (3 left) 2021-07-31T01:52:59.335 INFO:tasks.rados.rados.0.smithi045.stdout:2205: delete oid 35 current snap is 212 2021-07-31T01:52:59.335 INFO:tasks.rados.rados.0.smithi045.stdout:2202: finishing write tid 3 to smithi04569269-9 2021-07-31T01:52:59.335 INFO:tasks.rados.rados.0.smithi045.stdout:2202: finishing write tid 4 to smithi04569269-9 2021-07-31T01:52:59.336 INFO:tasks.rados.rados.0.smithi045.stdout:2202: finishing write tid 5 to smithi04569269-9 2021-07-31T01:52:59.336 INFO:tasks.rados.rados.0.smithi045.stdout:2203: finishing rollback tid 0 to smithi04569269-27 2021-07-31T01:52:59.337 INFO:tasks.rados.rados.0.smithi045.stdout:2203: finishing rollback tid 1 to smithi04569269-27 2021-07-31T01:52:59.337 INFO:tasks.rados.rados.0.smithi045.stdout:2203: finishing rollback tid 2 to smithi04569269-27 2021-07-31T01:52:59.337 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 27 v 1068 (ObjNum 523 snap 154 seq_num 523) dirty exists 2021-07-31T01:52:59.338 INFO:tasks.rados.rados.0.smithi045.stdout:2202: finishing write tid 6 to smithi04569269-9 2021-07-31T01:52:59.338 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 9 v 1050 (ObjNum 681 snap 212 seq_num 681) dirty exists 2021-07-31T01:52:59.338 INFO:tasks.rados.rados.0.smithi045.stdout:2202: left oid 9 (ObjNum 681 snap 212 seq_num 681) 2021-07-31T01:52:59.339 INFO:tasks.rados.rados.0.smithi045.stdout:2202: done (3 left) 2021-07-31T01:52:59.339 INFO:tasks.rados.rados.0.smithi045.stdout:2203: done (2 left) 2021-07-31T01:52:59.339 INFO:tasks.rados.rados.0.smithi045.stdout:2205: done (1 left) 2021-07-31T01:52:59.339 INFO:tasks.rados.rados.0.smithi045.stdout:2206: snap_create 2021-07-31T01:52:59.348 INFO:tasks.rados.rados.0.smithi045.stdout:2201: finishing copy_from to smithi04569269-37 2021-07-31T01:52:59.348 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 37 v 984 (ObjNum 639 snap 199 seq_num 639) dirty exists 2021-07-31T01:52:59.349 INFO:tasks.rados.rados.0.smithi045.stdout:2201: finishing copy_from racing read to smithi04569269-37 2021-07-31T01:52:59.590 INFO:tasks.rados.rados.0.smithi045.stdout:2201: done (1 left) 2021-07-31T01:52:59.591 INFO:tasks.rados.rados.0.smithi045.stdout:2206: done (0 left) 2021-07-31T01:52:59.591 INFO:tasks.rados.rados.0.smithi045.stdout:2207: snap_remove snap 208 2021-07-31T01:53:00.600 INFO:tasks.rados.rados.0.smithi045.stdout:2207: done (0 left) 2021-07-31T01:53:00.600 INFO:tasks.rados.rados.0.smithi045.stdout:2208: snap_create 2021-07-31T01:53:01.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:00 smithi154 conmon[34553]: cluster 2021-07-31T01:52:59.590296+0000 mon.a (mon.0) 1132 : cluster [DBG] osdmap e479: 8 total, 8 up, 8 in 2021-07-31T01:53:01.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:00 smithi154 conmon[34553]: cluster 2021-07-31T01:52:59.872977+0000 mgr.x (mgr.24542) 245 : cluster [DBG] pgmap v499: 25 pgs: 25 active+clean; 255 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 473 KiB/s wr, 3 op/s 2021-07-31T01:53:01.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:00 smithi045 conmon[33655]: cluster 2021-07-31T01:52:59.590296+0000 mon.a (mon.0) 1132 : cluster [DBG] 2021-07-31T01:53:01.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:00 smithi045 conmon[33655]: osdmap e479: 8 total, 8 up, 8 in 2021-07-31T01:53:01.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:00 smithi045 conmon[33655]: cluster 2021-07-31T01:52:59.872977+0000 mgr.x (mgr.24542) 245 : cluster [DBG] pgmap v499: 25 pgs: 25 active+clean; 255 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 473 KiB/s wr, 3 op/s 2021-07-31T01:53:01.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:00 smithi045 conmon[39718]: cluster 2021-07-31T01:52:59.590296+0000 mon.a (mon. 2021-07-31T01:53:01.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:00 smithi045 conmon[39718]: 0) 1132 : cluster [DBG] osdmap e479: 8 total, 8 up, 8 in 2021-07-31T01:53:01.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:00 smithi045 conmon[39718]: cluster 2021-07-31T01:52:59.872977+0000 mgr.x (mgr.24542) 245 : cluster 2021-07-31T01:53:01.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:00 smithi045 conmon[39718]: [DBG] pgmap v499: 25 pgs: 25 active+clean; 255 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 473 KiB/s wr, 3 op/s 2021-07-31T01:53:01.581 INFO:tasks.rados.rados.0.smithi045.stdout:2208: done (0 left) 2021-07-31T01:53:01.581 INFO:tasks.rados.rados.0.smithi045.stdout:2209: snap_remove snap 204 2021-07-31T01:53:02.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:01 smithi154 conmon[34553]: cluster 2021-07-31T01:53:00.595960+0000 mon.a (mon.0) 1133 : cluster [DBG] osdmap e480: 8 total, 8 up, 8 in 2021-07-31T01:53:02.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:01 smithi154 conmon[34553]: cluster 2021-07-31T01:53:01.580075 2021-07-31T01:53:02.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:01 smithi154 conmon[34553]: +0000 mon.a (mon.0) 1134 : cluster [DBG] osdmap e481: 8 total, 8 up, 8 in 2021-07-31T01:53:02.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:01 smithi045 conmon[39718]: cluster 2021-07-31T01:53:00.595960+0000 mon.a 2021-07-31T01:53:02.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:01 smithi045 conmon[39718]: (mon.0) 1133 : cluster [DBG] osdmap e480: 8 total, 8 up, 8 in 2021-07-31T01:53:02.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:01 smithi045 conmon[39718]: cluster 2021-07-31T01:53: 2021-07-31T01:53:02.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:01 smithi045 conmon[39718]: 01.580075+0000 mon.a (mon.0) 1134 : cluster [DBG] osdmap e481: 8 total, 8 up, 8 in 2021-07-31T01:53:02.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:01 smithi045 conmon[33655]: cluster 2021-07-31T01:53:00.595960+0000 mon.a (mon.0) 1133 : cluster 2021-07-31T01:53:02.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:01 smithi045 conmon[33655]: [DBG] osdmap e480: 8 total, 8 up, 8 in 2021-07-31T01:53:02.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:01 smithi045 conmon[33655]: cluster 2021-07-31T01:53:01.580075+0000 mon.a (mon.0) 1134 : cluster [DBG] osdmap e481: 8 total, 8 up, 8 in 2021-07-31T01:53:02.589 INFO:tasks.rados.rados.0.smithi045.stdout:2209: done (0 left) 2021-07-31T01:53:02.590 INFO:tasks.rados.rados.0.smithi045.stdout:2210: setattr oid 43 current snap is 214 2021-07-31T01:53:02.590 INFO:tasks.rados.rados.0.smithi045.stdout:2211: setattr oid 29 current snap is 214 2021-07-31T01:53:02.595 INFO:tasks.rados.rados.0.smithi045.stdout:2212: rmattr oid 12 current snap is 214 2021-07-31T01:53:02.595 INFO:tasks.rados.rados.0.smithi045.stdout:2213: read oid 6 snap -1 2021-07-31T01:53:02.595 INFO:tasks.rados.rados.0.smithi045.stdout:2213: expect (ObjNum 589 snap 182 seq_num 589) 2021-07-31T01:53:02.596 INFO:tasks.rados.rados.0.smithi045.stdout:2214: copy_from oid 42 from oid 26 current snap is 214 2021-07-31T01:53:02.596 INFO:tasks.rados.rados.0.smithi045.stdout:2215: read oid 46 snap 197 2021-07-31T01:53:02.597 INFO:tasks.rados.rados.0.smithi045.stdout:2215: expect (ObjNum 470 snap 136 seq_num 470) 2021-07-31T01:53:02.597 INFO:tasks.rados.rados.0.smithi045.stdout:2216: read oid 50 snap -1 2021-07-31T01:53:02.597 INFO:tasks.rados.rados.0.smithi045.stdout:2216: expect (ObjNum 635 snap 198 seq_num 635) 2021-07-31T01:53:02.597 INFO:tasks.rados.rados.0.smithi045.stdout:2217: read oid 8 snap -1 2021-07-31T01:53:02.598 INFO:tasks.rados.rados.0.smithi045.stdout:2217: expect (ObjNum 673 snap 210 seq_num 673) 2021-07-31T01:53:02.598 INFO:tasks.rados.rados.0.smithi045.stdout:2218: rollback oid 35 current snap is 214 2021-07-31T01:53:02.598 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 35 to 197 2021-07-31T01:53:02.598 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 16 current snap is 214 2021-07-31T01:53:02.599 INFO:tasks.rados.rados.0.smithi045.stdout:2219: seq_num 684 ranges {770048=663552,1433600=507904,1941504=409600,2351104=286720} 2021-07-31T01:53:02.602 INFO:tasks.rados.rados.0.smithi045.stdout:2219: writing smithi04569269-16 from 770048 to 1433600 tid 1 2021-07-31T01:53:02.606 INFO:tasks.rados.rados.0.smithi045.stdout:2219: writing smithi04569269-16 from 1433600 to 1941504 tid 2 2021-07-31T01:53:02.608 INFO:tasks.rados.rados.0.smithi045.stdout:2219: writing smithi04569269-16 from 1941504 to 2351104 tid 3 2021-07-31T01:53:02.611 INFO:tasks.rados.rados.0.smithi045.stdout:2219: writing smithi04569269-16 from 2351104 to 2637824 tid 4 2021-07-31T01:53:02.611 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 28 current snap is 214 2021-07-31T01:53:02.611 INFO:tasks.rados.rados.0.smithi045.stdout:2220: seq_num 685 ranges {0=679936,679936=442368,1122304=622592,1744896=139264} 2021-07-31T01:53:02.617 INFO:tasks.rados.rados.0.smithi045.stdout:2220: writing smithi04569269-28 from 0 to 679936 tid 1 2021-07-31T01:53:02.619 INFO:tasks.rados.rados.0.smithi045.stdout:2220: writing smithi04569269-28 from 679936 to 1122304 tid 2 2021-07-31T01:53:02.623 INFO:tasks.rados.rados.0.smithi045.stdout:2220: writing smithi04569269-28 from 1122304 to 1744896 tid 3 2021-07-31T01:53:02.625 INFO:tasks.rados.rados.0.smithi045.stdout:2220: writing smithi04569269-28 from 1744896 to 1884160 tid 4 2021-07-31T01:53:02.625 INFO:tasks.rados.rados.0.smithi045.stdout:2214: finishing copy_from to smithi04569269-42 2021-07-31T01:53:02.626 INFO:tasks.rados.rados.0.smithi045.stdout:2214: got expected ENOENT (src dne) 2021-07-31T01:53:02.626 INFO:tasks.rados.rados.0.smithi045.stdout:2221: read oid 14 snap -1 2021-07-31T01:53:02.626 INFO:tasks.rados.rados.0.smithi045.stdout:2221: expect (ObjNum 659 snap 205 seq_num 659) 2021-07-31T01:53:02.626 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 45 current snap is 214 2021-07-31T01:53:02.627 INFO:tasks.rados.rados.0.smithi045.stdout:2214: finishing copy_from racing read to smithi04569269-42 2021-07-31T01:53:02.627 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 29 v 931 (ObjNum 0 snap 0 seq_num 0) dirty exists 2021-07-31T01:53:02.627 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 43 v 1069 (ObjNum -1 snap 32731 seq_num 3065274944) dirty exists 2021-07-31T01:53:02.628 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 12 v 1376 (ObjNum 672 snap 210 seq_num 672) dirty exists 2021-07-31T01:53:02.628 INFO:tasks.rados.rados.0.smithi045.stdout:2222: seq_num 686 ranges {0=696320,696320=729088,1425408=425984,1851392=49152} 2021-07-31T01:53:02.629 INFO:tasks.rados.rados.0.smithi045.stdout:2222: writing smithi04569269-45 from 0 to 696320 tid 1 2021-07-31T01:53:02.634 INFO:tasks.rados.rados.0.smithi045.stdout:2222: writing smithi04569269-45 from 696320 to 1425408 tid 2 2021-07-31T01:53:02.636 INFO:tasks.rados.rados.0.smithi045.stdout:2222: writing smithi04569269-45 from 1425408 to 1851392 tid 3 2021-07-31T01:53:02.637 INFO:tasks.rados.rados.0.smithi045.stdout:2222: writing smithi04569269-45 from 1851392 to 1900544 tid 4 2021-07-31T01:53:02.637 INFO:tasks.rados.rados.0.smithi045.stdout:2210: done (12 left) 2021-07-31T01:53:02.637 INFO:tasks.rados.rados.0.smithi045.stdout:2211: done (11 left) 2021-07-31T01:53:02.638 INFO:tasks.rados.rados.0.smithi045.stdout:2212: done (10 left) 2021-07-31T01:53:02.638 INFO:tasks.rados.rados.0.smithi045.stdout:2214: done (9 left) 2021-07-31T01:53:02.638 INFO:tasks.rados.rados.0.smithi045.stdout:2223: snap_remove snap 213 2021-07-31T01:53:03.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:02 smithi154 conmon[34553]: cluster 2021-07-31T01:53:03.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:02 smithi154 conmon[34553]: 2021-07-31T01:53:03.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:02 smithi154 conmon[34553]: 2021 2021-07-31T01:53:03.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:02 smithi154 conmon[34553]: - 2021-07-31T01:53:03.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:02 smithi154 conmon[34553]: 07 2021-07-31T01:53:03.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:02 smithi154 conmon[34553]: - 2021-07-31T01:53:03.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:02 smithi154 conmon[34553]: 31 2021-07-31T01:53:03.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:02 smithi154 conmon[34553]: T 2021-07-31T01:53:03.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:02 smithi154 conmon[34553]: 01 2021-07-31T01:53:03.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:02 smithi154 conmon[34553]: : 2021-07-31T01:53:03.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:02 smithi154 conmon[34553]: 53 2021-07-31T01:53:03.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:02 smithi154 conmon[34553]: : 2021-07-31T01:53:03.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:02 smithi154 conmon[34553]: 01.873431+0000 mgr.x (mgr.24542) 246 : cluster [DBG] 2021-07-31T01:53:03.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:02 smithi154 conmon[34553]: pgmap v502: 25 pgs: 25 active+clean; 255 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 468 KiB/s wr, 3 op/s 2021-07-31T01:53:03.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:02 smithi154 conmon[34553]: cluster 2021 2021-07-31T01:53:03.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:02 smithi154 conmon[34553]: -07-31T01:53:02.588592+0000 mon.a (mon.0) 1135 : cluster [DBG] osdmap e482: 8 total, 8 up, 8 in 2021-07-31T01:53:03.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:02 smithi045 conmon[39718]: cluster 2021-07-31T01:53:01.873431+0000 mgr.x (mgr.24542) 246 : cluster [DBG] pgmap v502: 25 pgs: 25 active+clean; 255 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 468 KiB/s wr, 3 op/s 2021-07-31T01:53:03.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:02 smithi045 conmon[39718]: cluster 2021-07-31T01:53:02.588592+0000 mon.a (mon.0) 1135 : cluster [DBG] osdmap e482: 8 total, 8 up, 8 in 2021-07-31T01:53:03.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:02 smithi045 conmon[33655]: cluster 2021-07-31T01:53:03.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:02 smithi045 conmon[33655]: 2021-07-31T01:53:03.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:02 smithi045 conmon[33655]: 2021 2021-07-31T01:53:03.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:02 smithi045 conmon[33655]: - 2021-07-31T01:53:03.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:02 smithi045 conmon[33655]: 07 2021-07-31T01:53:03.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:02 smithi045 conmon[33655]: - 2021-07-31T01:53:03.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:02 smithi045 conmon[33655]: 31 2021-07-31T01:53:03.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:02 smithi045 conmon[33655]: T 2021-07-31T01:53:03.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:02 smithi045 conmon[33655]: 01 2021-07-31T01:53:03.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:02 smithi045 conmon[33655]: : 2021-07-31T01:53:03.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:02 smithi045 conmon[33655]: 53 2021-07-31T01:53:03.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:02 smithi045 conmon[33655]: : 2021-07-31T01:53:03.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:02 smithi045 conmon[33655]: 01 2021-07-31T01:53:03.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:02 smithi045 conmon[33655]: . 2021-07-31T01:53:03.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:02 smithi045 conmon[33655]: 873431 2021-07-31T01:53:03.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:02 smithi045 conmon[33655]: +0000 2021-07-31T01:53:03.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:02 smithi045 conmon[33655]: 2021-07-31T01:53:03.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:02 smithi045 conmon[33655]: mgr.x 2021-07-31T01:53:03.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:02 smithi045 conmon[33655]: ( 2021-07-31T01:53:03.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:02 smithi045 conmon[33655]: mgr 2021-07-31T01:53:03.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:02 smithi045 conmon[33655]: . 2021-07-31T01:53:03.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:02 smithi045 conmon[33655]: 24542 2021-07-31T01:53:03.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:02 smithi045 conmon[33655]: ) 2021-07-31T01:53:03.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:02 smithi045 conmon[33655]: 246 2021-07-31T01:53:03.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:02 smithi045 conmon[33655]: : 2021-07-31T01:53:03.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:02 smithi045 conmon[33655]: cluster 2021-07-31T01:53:03.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:02 smithi045 conmon[33655]: 2021-07-31T01:53:03.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:02 smithi045 conmon[33655]: [DBG] 2021-07-31T01:53:03.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:02 smithi045 conmon[33655]: 2021-07-31T01:53:03.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:02 smithi045 conmon[33655]: pgmap v502: 25 pgs: 25 active+clean; 255 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 468 KiB/s wr, 3 op/s 2021-07-31T01:53:03.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:02 smithi045 conmon[33655]: 2021-07-31T01:53:03.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:02 smithi045 conmon[33655]: cluster 2021-07-31T01:53:03.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:02 smithi045 conmon[33655]: 2021-07-31T01:53:03.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:02 smithi045 conmon[33655]: 2021 2021-07-31T01:53:03.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:02 smithi045 conmon[33655]: - 2021-07-31T01:53:03.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:02 smithi045 conmon[33655]: 07 2021-07-31T01:53:03.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:02 smithi045 conmon[33655]: - 2021-07-31T01:53:03.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:02 smithi045 conmon[33655]: 31 2021-07-31T01:53:03.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:02 smithi045 conmon[33655]: T 2021-07-31T01:53:03.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:02 smithi045 conmon[33655]: 01 2021-07-31T01:53:03.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:02 smithi045 conmon[33655]: : 2021-07-31T01:53:03.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:02 smithi045 conmon[33655]: 53 2021-07-31T01:53:03.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:02 smithi045 conmon[33655]: : 2021-07-31T01:53:03.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:02 smithi045 conmon[33655]: 02 2021-07-31T01:53:03.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:02 smithi045 conmon[33655]: . 2021-07-31T01:53:03.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:02 smithi045 conmon[33655]: 588592 2021-07-31T01:53:03.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:02 smithi045 conmon[33655]: +0000 2021-07-31T01:53:03.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:02 smithi045 conmon[33655]: 2021-07-31T01:53:03.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:02 smithi045 conmon[33655]: mon.a 2021-07-31T01:53:03.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:02 smithi045 conmon[33655]: ( 2021-07-31T01:53:03.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:02 smithi045 conmon[33655]: mon 2021-07-31T01:53:03.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:02 smithi045 conmon[33655]: . 2021-07-31T01:53:03.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:02 smithi045 conmon[33655]: 0 2021-07-31T01:53:03.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:02 smithi045 conmon[33655]: ) 2021-07-31T01:53:03.040 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:02 smithi045 conmon[33655]: 1135 2021-07-31T01:53:03.040 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:02 smithi045 conmon[33655]: : 2021-07-31T01:53:03.040 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:02 smithi045 conmon[33655]: cluster [DBG] osdmap e482: 8 total, 8 up, 8 in 2021-07-31T01:53:03.614 INFO:tasks.rados.rados.0.smithi045.stdout:2223: done (9 left) 2021-07-31T01:53:03.614 INFO:tasks.rados.rados.0.smithi045.stdout:2224: read oid 37 snap 197 2021-07-31T01:53:03.615 INFO:tasks.rados.rados.0.smithi045.stdout:2224: expect deleted 2021-07-31T01:53:03.615 INFO:tasks.rados.rados.0.smithi045.stdout:2219: finishing write tid 1 to smithi04569269-16 2021-07-31T01:53:03.616 INFO:tasks.rados.rados.0.smithi045.stdout:2216: expect (ObjNum 635 snap 198 seq_num 635) 2021-07-31T01:53:03.721 INFO:tasks.rados.rados.0.smithi045.stdout:2213: expect (ObjNum 589 snap 182 seq_num 589) 2021-07-31T01:53:03.728 INFO:tasks.rados.rados.0.smithi045.stdout:2219: finishing write tid 2 to smithi04569269-16 2021-07-31T01:53:03.729 INFO:tasks.rados.rados.0.smithi045.stdout:2219: finishing write tid 3 to smithi04569269-16 2021-07-31T01:53:03.729 INFO:tasks.rados.rados.0.smithi045.stdout:2215: expect (ObjNum 470 snap 136 seq_num 470) 2021-07-31T01:53:03.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:53:03 smithi045 conmon[61210]: level=error ts=2021-07-31T01:53:03.519Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:53:03.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:53:03 smithi045 conmon[61210]: level=error ts=2021-07-31T01:53:03.519Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:53:03.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:53:03 smithi045 conmon[61210]: level=error ts=2021-07-31T01:53:03.519Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T01:53:03.862 INFO:tasks.rados.rados.0.smithi045.stdout:2217: expect (ObjNum 673 snap 210 seq_num 673) 2021-07-31T01:53:03.906 INFO:tasks.rados.rados.0.smithi045.stdout:2218: finishing rollback tid 1 to smithi04569269-35 2021-07-31T01:53:03.907 INFO:tasks.rados.rados.0.smithi045.stdout:2218: finishing rollback tid 2 to smithi04569269-35 2021-07-31T01:53:03.907 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 35 v 1420 (ObjNum 593 snap 184 seq_num 593) dirty exists 2021-07-31T01:53:03.907 INFO:tasks.rados.rados.0.smithi045.stdout:2219: finishing write tid 4 to smithi04569269-16 2021-07-31T01:53:03.907 INFO:tasks.rados.rados.0.smithi045.stdout:2219: finishing write tid 5 to smithi04569269-16 2021-07-31T01:53:03.908 INFO:tasks.rados.rados.0.smithi045.stdout:2225: read oid 17 snap -1 2021-07-31T01:53:03.908 INFO:tasks.rados.rados.0.smithi045.stdout:2225: expect (ObjNum 664 snap 208 seq_num 664) 2021-07-31T01:53:03.908 INFO:tasks.rados.rados.0.smithi045.stdout:2213: done (10 left) 2021-07-31T01:53:03.908 INFO:tasks.rados.rados.0.smithi045.stdout:2215: done (9 left) 2021-07-31T01:53:03.908 INFO:tasks.rados.rados.0.smithi045.stdout:2216: done (8 left) 2021-07-31T01:53:03.909 INFO:tasks.rados.rados.0.smithi045.stdout:2217: done (7 left) 2021-07-31T01:53:03.909 INFO:tasks.rados.rados.0.smithi045.stdout:2218: done (6 left) 2021-07-31T01:53:03.909 INFO:tasks.rados.rados.0.smithi045.stdout:2226: read oid 5 snap -1 2021-07-31T01:53:03.909 INFO:tasks.rados.rados.0.smithi045.stdout:2226: expect (ObjNum 172 snap 41 seq_num 172) 2021-07-31T01:53:03.910 INFO:tasks.rados.rados.0.smithi045.stdout:2227: read oid 38 snap -1 2021-07-31T01:53:03.910 INFO:tasks.rados.rados.0.smithi045.stdout:2227: expect deleted 2021-07-31T01:53:03.910 INFO:tasks.rados.rados.0.smithi045.stdout:2228: rollback oid 43 current snap is 214 2021-07-31T01:53:03.910 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 43 to 197 2021-07-31T01:53:03.910 INFO:tasks.rados.rados.0.smithi045.stdout:2219: finishing write tid 6 to smithi04569269-16 2021-07-31T01:53:03.911 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 16 v 1147 (ObjNum 684 snap 214 seq_num 684) dirty exists 2021-07-31T01:53:03.911 INFO:tasks.rados.rados.0.smithi045.stdout:2219: left oid 16 (ObjNum 684 snap 214 seq_num 684) 2021-07-31T01:53:03.911 INFO:tasks.rados.rados.0.smithi045.stdout:2220: finishing write tid 1 to smithi04569269-28 2021-07-31T01:53:03.911 INFO:tasks.rados.rados.0.smithi045.stdout:2220: finishing write tid 2 to smithi04569269-28 2021-07-31T01:53:03.912 INFO:tasks.rados.rados.0.smithi045.stdout:2220: finishing write tid 3 to smithi04569269-28 2021-07-31T01:53:03.912 INFO:tasks.rados.rados.0.smithi045.stdout:2220: finishing write tid 4 to smithi04569269-28 2021-07-31T01:53:03.912 INFO:tasks.rados.rados.0.smithi045.stdout:2220: finishing write tid 5 to smithi04569269-28 2021-07-31T01:53:03.912 INFO:tasks.rados.rados.0.smithi045.stdout:2222: finishing write tid 1 to smithi04569269-45 2021-07-31T01:53:03.912 INFO:tasks.rados.rados.0.smithi045.stdout:2220: finishing write tid 6 to smithi04569269-28 2021-07-31T01:53:03.913 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 28 v 1148 (ObjNum 685 snap 214 seq_num 685) dirty exists 2021-07-31T01:53:03.913 INFO:tasks.rados.rados.0.smithi045.stdout:2220: left oid 28 (ObjNum 685 snap 214 seq_num 685) 2021-07-31T01:53:03.913 INFO:tasks.rados.rados.0.smithi045.stdout:2222: finishing write tid 2 to smithi04569269-45 2021-07-31T01:53:03.913 INFO:tasks.rados.rados.0.smithi045.stdout:2222: finishing write tid 3 to smithi04569269-45 2021-07-31T01:53:03.913 INFO:tasks.rados.rados.0.smithi045.stdout:2222: finishing write tid 4 to smithi04569269-45 2021-07-31T01:53:03.914 INFO:tasks.rados.rados.0.smithi045.stdout:2222: finishing write tid 5 to smithi04569269-45 2021-07-31T01:53:03.914 INFO:tasks.rados.rados.0.smithi045.stdout:2222: finishing write tid 6 to smithi04569269-45 2021-07-31T01:53:03.915 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 45 v 1153 (ObjNum 686 snap 214 seq_num 686) dirty exists 2021-07-31T01:53:03.915 INFO:tasks.rados.rados.0.smithi045.stdout:2222: left oid 45 (ObjNum 686 snap 214 seq_num 686) 2021-07-31T01:53:03.915 INFO:tasks.rados.rados.0.smithi045.stdout:2219: done (8 left) 2021-07-31T01:53:03.915 INFO:tasks.rados.rados.0.smithi045.stdout:2220: done (7 left) 2021-07-31T01:53:03.916 INFO:tasks.rados.rados.0.smithi045.stdout:2222: done (6 left) 2021-07-31T01:53:03.916 INFO:tasks.rados.rados.0.smithi045.stdout:2229: copy_from oid 21 from oid 12 current snap is 214 2021-07-31T01:53:03.916 INFO:tasks.rados.rados.0.smithi045.stdout:2221: expect (ObjNum 659 snap 205 seq_num 659) 2021-07-31T01:53:04.623 INFO:tasks.rados.rados.0.smithi045.stdout:2226: expect (ObjNum 172 snap 41 seq_num 172) 2021-07-31T01:53:04.642 INFO:tasks.rados.rados.0.smithi045.stdout:2228: finishing rollback tid 0 to smithi04569269-43 2021-07-31T01:53:04.642 INFO:tasks.rados.rados.0.smithi045.stdout:2228: finishing rollback tid 1 to smithi04569269-43 2021-07-31T01:53:04.643 INFO:tasks.rados.rados.0.smithi045.stdout:2228: finishing rollback tid 2 to smithi04569269-43 2021-07-31T01:53:04.643 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 43 v 1072 (ObjNum 523 snap 154 seq_num 523) dirty exists 2021-07-31T01:53:04.643 INFO:tasks.rados.rados.0.smithi045.stdout:2225: expect (ObjNum 664 snap 208 seq_num 664) 2021-07-31T01:53:05.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:04 smithi154 conmon[34553]: cluster 2021-07-31T01:53:03.612805+0000 mon.a (mon.0) 1136 : cluster [DBG] osdmap e483: 8 total, 8 up, 8 in 2021-07-31T01:53:05.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:04 smithi154 conmon[34553]: cluster 2021-07-31T01:53:03.874126+0000 mgr.x (mgr.24542) 247 : cluster [DBG] pgmap v505: 25 pgs: 2 active+clean+snaptrim, 22 active+clean, 1 active+clean+snaptrim_wait; 269 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 3.4 MiB/s wr, 24 op/s 2021-07-31T01:53:05.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:04 smithi045 conmon[39718]: cluster 2021-07-31T01:53: 2021-07-31T01:53:05.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:04 smithi045 conmon[39718]: 03.612805+0000 mon.a (mon.0) 1136 : cluster [DBG] osdmap e483: 8 total, 8 up, 8 in 2021-07-31T01:53:05.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:04 smithi045 conmon[39718]: cluster 2021-07-31T01:53:03.874126 2021-07-31T01:53:05.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:04 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 247 : cluster [DBG] pgmap v505: 25 pgs: 2 active+clean+snaptrim, 22 active+clean, 1 active+clean+snaptrim_wait; 269 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 3.4 MiB/s wr, 24 op/s 2021-07-31T01:53:05.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:04 smithi045 conmon[33655]: cluster 2021-07-31T01:53:03.612805+0000 2021-07-31T01:53:05.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:04 smithi045 conmon[33655]: mon.a (mon.0) 1136 : cluster [DBG] osdmap e483: 8 total, 8 up, 8 in 2021-07-31T01:53:05.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:04 smithi045 conmon[33655]: cluster 2021-07-31T01:53:03.874126+0000 mgr.x (mgr.24542) 247 : cluster [DBG] pgmap v505: 25 pgs: 2 active+clean+snaptrim, 22 active+clean, 1 active+clean+snaptrim_wait; 269 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 3.4 MiB/s wr, 24 op/s 2021-07-31T01:53:05.093 INFO:tasks.rados.rados.0.smithi045.stdout:2229: finishing copy_from to smithi04569269-21 2021-07-31T01:53:05.093 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 21 v 1376 (ObjNum 672 snap 210 seq_num 672) dirty exists 2021-07-31T01:53:05.093 INFO:tasks.rados.rados.0.smithi045.stdout:2229: finishing copy_from racing read to smithi04569269-21 2021-07-31T01:53:05.094 INFO:tasks.rados.rados.0.smithi045.stdout:2221: done (6 left) 2021-07-31T01:53:05.094 INFO:tasks.rados.rados.0.smithi045.stdout:2224: done (5 left) 2021-07-31T01:53:05.094 INFO:tasks.rados.rados.0.smithi045.stdout:2225: done (4 left) 2021-07-31T01:53:05.094 INFO:tasks.rados.rados.0.smithi045.stdout:2226: done (3 left) 2021-07-31T01:53:05.094 INFO:tasks.rados.rados.0.smithi045.stdout:2227: done (2 left) 2021-07-31T01:53:05.095 INFO:tasks.rados.rados.0.smithi045.stdout:2228: done (1 left) 2021-07-31T01:53:05.095 INFO:tasks.rados.rados.0.smithi045.stdout:2229: done (0 left) 2021-07-31T01:53:05.095 INFO:tasks.rados.rados.0.smithi045.stdout:2230: snap_remove snap 182 2021-07-31T01:53:05.413 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T01:53:05.620 INFO:tasks.rados.rados.0.smithi045.stdout:2230: done (0 left) 2021-07-31T01:53:05.620 INFO:tasks.rados.rados.0.smithi045.stdout:2231: rollback oid 15 current snap is 214 2021-07-31T01:53:05.621 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 15 to 212 2021-07-31T01:53:05.621 INFO:tasks.rados.rados.0.smithi045.stdout:2232: read oid 5 snap 212 2021-07-31T01:53:05.621 INFO:tasks.rados.rados.0.smithi045.stdout:2232: expect (ObjNum 172 snap 41 seq_num 172) 2021-07-31T01:53:05.621 INFO:tasks.rados.rados.0.smithi045.stdout:2233: snap_remove snap 189 2021-07-31T01:53:05.707 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T01:53:05.708 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 7m ago 8m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T01:53:05.708 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 5m ago 8m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T01:53:05.708 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 5m ago 11m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T01:53:05.709 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 7m ago 14m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T01:53:05.709 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 7m ago 14m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T01:53:05.709 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 5m ago 12m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T01:53:05.709 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 7m ago 12m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T01:53:05.709 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 7m ago 8m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T01:53:05.710 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 5m ago 8m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T01:53:05.710 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 7m ago 11m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T01:53:05.710 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 7m ago 11m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T01:53:05.711 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 7m ago 11m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T01:53:05.712 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 7m ago 10m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T01:53:05.712 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 5m ago 10m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T01:53:05.712 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 5m ago 10m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T01:53:05.713 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 5m ago 9m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T01:53:05.714 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 5m ago 9m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T01:53:05.714 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 5m ago 9m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T01:53:06.026 INFO:journalctl@ceph.osd.3.smithi045.stdout:Jul 31 01:53:05 smithi045 conmon[54013]: debug 2021-07-31T01:53:06.026 INFO:journalctl@ceph.osd.3.smithi045.stdout:Jul 31 01:53:05 smithi045 conmon[54013]: 2021-07-31T01:53:05.625+0000 7f64f90c6700 -1 osd.3 pg_epoch: 484 pg[3.2s0( v 484'317 (0'0,484'317] local-lis/les=56/57 n=4 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [3,5,6]p3(0) r=0 lpr=56 luod=482'315 crt=482'315 lcod 480'313 mlcod 480'313 active+clean+snaptrim trimq=[165~1] ps=[190~1,197~1,1a4~1]] removing snap head 2021-07-31T01:53:06.026 INFO:journalctl@ceph.osd.3.smithi045.stdout:Jul 31 01:53:05 smithi045 conmon[54013]: 2021-07-31T01:53:06.027 INFO:journalctl@ceph.osd.3.smithi045.stdout:Jul 31 01:53:05 smithi045 conmon[54013]: debug 2021-07-31T01:53:05.637+0000 7f64f88c5700 -1 osd.3 pg_epoch: 484 pg[3.bs0( v 484'1438 (0'0,484'1438] local-lis/les=56/57 n=19 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [3,0,4]p3(0) r=0 lpr=56 luod=484'1436 crt=484'1436 lcod 484'1434 mlcod 484'1434 active+clean+snaptrim trimq=[165~1] ps=[190~1,197~1,1a4~1]] removing snap head 2021-07-31T01:53:06.042 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T01:53:06.043 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T01:53:06.043 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T01:53:06.044 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T01:53:06.044 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T01:53:06.044 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T01:53:06.044 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T01:53:06.045 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T01:53:06.045 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T01:53:06.045 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T01:53:06.045 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T01:53:06.046 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T01:53:06.046 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T01:53:06.046 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T01:53:06.047 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T01:53:06.047 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T01:53:06.047 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T01:53:06.584 INFO:tasks.rados.rados.0.smithi045.stdout:2233: done (2 left) 2021-07-31T01:53:06.585 INFO:tasks.rados.rados.0.smithi045.stdout:2234: copy_from oid 36 from oid 44 current snap is 214 2021-07-31T01:53:06.585 INFO:tasks.rados.rados.0.smithi045.stdout:2231: finishing rollback tid 0 to smithi04569269-15 2021-07-31T01:53:06.585 INFO:tasks.rados.rados.0.smithi045.stdout:2232: expect (ObjNum 172 snap 41 seq_num 172) 2021-07-31T01:53:06.613 INFO:tasks.rados.rados.0.smithi045.stdout:2231: finishing rollback tid 1 to smithi04569269-15 2021-07-31T01:53:06.613 INFO:tasks.rados.rados.0.smithi045.stdout:2231: finishing rollback tid 2 to smithi04569269-15 2021-07-31T01:53:06.614 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 15 v 987 (ObjNum 677 snap 211 seq_num 677) dirty exists 2021-07-31T01:53:06.615 INFO:tasks.rados.rados.0.smithi045.stdout:2231: done (2 left) 2021-07-31T01:53:06.615 INFO:tasks.rados.rados.0.smithi045.stdout:2232: done (1 left) 2021-07-31T01:53:06.615 INFO:tasks.rados.rados.0.smithi045.stdout:2235: read oid 10 snap -1 2021-07-31T01:53:06.616 INFO:tasks.rados.rados.0.smithi045.stdout:2235: expect deleted 2021-07-31T01:53:06.616 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 32 current snap is 214 2021-07-31T01:53:06.616 INFO:tasks.rados.rados.0.smithi045.stdout:2234: finishing copy_from to smithi04569269-36 2021-07-31T01:53:06.616 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 36 v 1065 (ObjNum 676 snap 210 seq_num 676) dirty exists 2021-07-31T01:53:06.617 INFO:tasks.rados.rados.0.smithi045.stdout:2234: finishing copy_from racing read to smithi04569269-36 2021-07-31T01:53:06.617 INFO:tasks.rados.rados.0.smithi045.stdout:2236: seq_num 687 ranges {3489792=712704,4202496=614400,4816896=589824} 2021-07-31T01:53:06.637 INFO:tasks.rados.rados.0.smithi045.stdout:2236: writing smithi04569269-32 from 3489792 to 4202496 tid 1 2021-07-31T01:53:06.639 INFO:tasks.rados.rados.0.smithi045.stdout:2236: writing smithi04569269-32 from 4202496 to 4816896 tid 2 2021-07-31T01:53:06.643 INFO:tasks.rados.rados.0.smithi045.stdout:2236: writing smithi04569269-32 from 4816896 to 5406720 tid 3 2021-07-31T01:53:06.644 INFO:tasks.rados.rados.0.smithi045.stdout:2234: done (2 left) 2021-07-31T01:53:06.644 INFO:tasks.rados.rados.0.smithi045.stdout:2237: rollback oid 8 current snap is 214 2021-07-31T01:53:06.644 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 8 to 212 2021-07-31T01:53:06.644 INFO:tasks.rados.rados.0.smithi045.stdout:2236: finishing write tid 1 to smithi04569269-32 2021-07-31T01:53:06.646 INFO:tasks.rados.rados.0.smithi045.stdout:2236: finishing write tid 2 to smithi04569269-32 2021-07-31T01:53:06.646 INFO:tasks.rados.rados.0.smithi045.stdout:2235: done (2 left) 2021-07-31T01:53:06.647 INFO:tasks.rados.rados.0.smithi045.stdout:2238: read oid 46 snap -1 2021-07-31T01:53:06.647 INFO:tasks.rados.rados.0.smithi045.stdout:2238: expect (ObjNum 665 snap 208 seq_num 665) 2021-07-31T01:53:06.647 INFO:tasks.rados.rados.0.smithi045.stdout:2239: setattr oid 9 current snap is 214 2021-07-31T01:53:06.648 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 34 current snap is 214 2021-07-31T01:53:06.648 INFO:tasks.rados.rados.0.smithi045.stdout:2240: seq_num 688 ranges {6774784=729088,7503872=499712,8003584=704512} 2021-07-31T01:53:06.686 INFO:tasks.rados.rados.0.smithi045.stdout:2240: writing smithi04569269-34 from 6774784 to 7503872 tid 1 2021-07-31T01:53:06.688 INFO:tasks.rados.rados.0.smithi045.stdout:2240: writing smithi04569269-34 from 7503872 to 8003584 tid 2 2021-07-31T01:53:06.693 INFO:tasks.rados.rados.0.smithi045.stdout:2240: writing smithi04569269-34 from 8003584 to 8708096 tid 3 2021-07-31T01:53:06.694 INFO:tasks.rados.rados.0.smithi045.stdout:2241: delete oid 23 current snap is 214 2021-07-31T01:53:06.694 INFO:tasks.rados.rados.0.smithi045.stdout:2236: finishing write tid 3 to smithi04569269-32 2021-07-31T01:53:06.694 INFO:tasks.rados.rados.0.smithi045.stdout:2236: finishing write tid 4 to smithi04569269-32 2021-07-31T01:53:06.695 INFO:tasks.rados.rados.0.smithi045.stdout:2237: finishing rollback tid 0 to smithi04569269-8 2021-07-31T01:53:06.697 INFO:tasks.rados.rados.0.smithi045.stdout:2236: finishing write tid 5 to smithi04569269-32 2021-07-31T01:53:06.698 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 32 v 1042 (ObjNum 687 snap 214 seq_num 687) dirty exists 2021-07-31T01:53:06.698 INFO:tasks.rados.rados.0.smithi045.stdout:2236: left oid 32 (ObjNum 687 snap 214 seq_num 687) 2021-07-31T01:53:06.699 INFO:tasks.rados.rados.0.smithi045.stdout:2237: finishing rollback tid 1 to smithi04569269-8 2021-07-31T01:53:06.700 INFO:tasks.rados.rados.0.smithi045.stdout:2237: finishing rollback tid 2 to smithi04569269-8 2021-07-31T01:53:06.700 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 8 v 1460 (ObjNum 673 snap 210 seq_num 673) dirty exists 2021-07-31T01:53:06.701 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 9 v 1051 (ObjNum 681 snap 212 seq_num 681) dirty exists 2021-07-31T01:53:06.701 INFO:tasks.rados.rados.0.smithi045.stdout:2238: expect (ObjNum 665 snap 208 seq_num 665) 2021-07-31T01:53:06.913 INFO:tasks.rados.rados.0.smithi045.stdout:2240: finishing write tid 1 to smithi04569269-34 2021-07-31T01:53:06.914 INFO:tasks.rados.rados.0.smithi045.stdout:2240: finishing write tid 2 to smithi04569269-34 2021-07-31T01:53:06.914 INFO:tasks.rados.rados.0.smithi045.stdout:2240: finishing write tid 3 to smithi04569269-34 2021-07-31T01:53:06.914 INFO:tasks.rados.rados.0.smithi045.stdout:2240: finishing write tid 4 to smithi04569269-34 2021-07-31T01:53:06.915 INFO:tasks.rados.rados.0.smithi045.stdout:2240: finishing write tid 5 to smithi04569269-34 2021-07-31T01:53:06.915 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 34 v 1022 (ObjNum 688 snap 214 seq_num 688) dirty exists 2021-07-31T01:53:06.915 INFO:tasks.rados.rados.0.smithi045.stdout:2240: left oid 34 (ObjNum 688 snap 214 seq_num 688) 2021-07-31T01:53:06.916 INFO:tasks.rados.rados.0.smithi045.stdout:2236: done (5 left) 2021-07-31T01:53:06.916 INFO:tasks.rados.rados.0.smithi045.stdout:2237: done (4 left) 2021-07-31T01:53:06.917 INFO:tasks.rados.rados.0.smithi045.stdout:2238: done (3 left) 2021-07-31T01:53:06.917 INFO:tasks.rados.rados.0.smithi045.stdout:2239: done (2 left) 2021-07-31T01:53:06.918 INFO:tasks.rados.rados.0.smithi045.stdout:2240: done (1 left) 2021-07-31T01:53:06.918 INFO:tasks.rados.rados.0.smithi045.stdout:2241: done (0 left) 2021-07-31T01:53:06.918 INFO:tasks.rados.rados.0.smithi045.stdout:2242: setattr oid 7 current snap is 214 2021-07-31T01:53:06.919 INFO:tasks.rados.rados.0.smithi045.stdout:2243: copy_from oid 43 from oid 12 current snap is 214 2021-07-31T01:53:06.919 INFO:tasks.rados.rados.0.smithi045.stdout:2244: read oid 28 snap -1 2021-07-31T01:53:06.919 INFO:tasks.rados.rados.0.smithi045.stdout:2244: expect (ObjNum 685 snap 214 seq_num 685) 2021-07-31T01:53:06.920 INFO:tasks.rados.rados.0.smithi045.stdout:2245: rollback oid 47 current snap is 214 2021-07-31T01:53:06.920 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 47 to 211 2021-07-31T01:53:06.921 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 10 current snap is 214 2021-07-31T01:53:06.921 INFO:tasks.rados.rados.0.smithi045.stdout:2246: seq_num 690 ranges {0=761856,761856=663552,1425408=409600,1835008=131072} 2021-07-31T01:53:06.923 INFO:tasks.rados.rados.0.smithi045.stdout:2246: writing smithi04569269-10 from 0 to 761856 tid 1 2021-07-31T01:53:06.923 INFO:tasks.rados.rados.0.smithi045.stdout:2246: writing smithi04569269-10 from 761856 to 1425408 tid 2 2021-07-31T01:53:06.923 INFO:tasks.rados.rados.0.smithi045.stdout:2246: writing smithi04569269-10 from 1425408 to 1835008 tid 3 2021-07-31T01:53:06.925 INFO:tasks.rados.rados.0.smithi045.stdout:2246: writing smithi04569269-10 from 1835008 to 1966080 tid 4 2021-07-31T01:53:06.925 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 34 current snap is 214 2021-07-31T01:53:06.925 INFO:tasks.rados.rados.0.smithi045.stdout:2245: finishing rollback tid 1 to smithi04569269-47 2021-07-31T01:53:06.926 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 47 v 0 (ObjNum 418 snap 0 seq_num 414) dirty dne 2021-07-31T01:53:06.926 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 7 v 986 (ObjNum 618 snap 189 seq_num 618) dirty exists 2021-07-31T01:53:06.927 INFO:tasks.rados.rados.0.smithi045.stdout:2247: seq_num 691 ranges {8708096=778240,9486336=548864,10035200=614400,10649600=40960} 2021-07-31T01:53:06.976 INFO:tasks.rados.rados.0.smithi045.stdout:2247: writing smithi04569269-34 from 8708096 to 9486336 tid 1 2021-07-31T01:53:06.979 INFO:tasks.rados.rados.0.smithi045.stdout:2247: writing smithi04569269-34 from 9486336 to 10035200 tid 2 2021-07-31T01:53:06.984 INFO:tasks.rados.rados.0.smithi045.stdout:2247: writing smithi04569269-34 from 10035200 to 10649600 tid 3 2021-07-31T01:53:06.984 INFO:tasks.rados.rados.0.smithi045.stdout:2247: writing smithi04569269-34 from 10649600 to 10690560 tid 4 2021-07-31T01:53:06.984 INFO:tasks.rados.rados.0.smithi045.stdout:2242: done (5 left) 2021-07-31T01:53:06.985 INFO:tasks.rados.rados.0.smithi045.stdout:2245: done (4 left) 2021-07-31T01:53:06.985 INFO:tasks.rados.rados.0.smithi045.stdout:2248: snap_create 2021-07-31T01:53:06.985 INFO:tasks.rados.rados.0.smithi045.stdout:2244: expect (ObjNum 685 snap 214 seq_num 685) 2021-07-31T01:53:07.003 INFO:journalctl@ceph.osd.7.smithi154.stdout:Jul 31 01:53:06 smithi154 conmon[49149]: debug 2021-07-31T01:53:07.003 INFO:journalctl@ceph.osd.7.smithi154.stdout:Jul 31 01:53:06 smithi154 conmon[49149]: 2021-07-31T01:53:06.606+0000 7f8a719fc700 -1 osd.7 pg_epoch: 485 pg[3.ds0( v 485'1006 (0'0,485'1006] local-lis/les=56/57 n=9 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [7,5,3]p7(0) r=0 lpr=56 crt=485'1006 lcod 485'1004 mlcod 485'1004 active+clean+snaptrim trimq=[179~1] ps=[165~1,1a4~1]] removing snap head 2021-07-31T01:53:07.003 INFO:journalctl@ceph.osd.7.smithi154.stdout:Jul 31 01:53:06 smithi154 conmon[49149]: 2021-07-31T01:53:07.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:06 smithi154 conmon[34553]: audit 2021-07-31T01:53:05.401637+0000 mgr.x (mgr.24542) 248 : audit 2021-07-31T01:53:07.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:06 smithi154 conmon[34553]: [DBG] from='client.14910 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:53:07.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:06 smithi154 conmon[34553]: cluster 2021-07-31T01:53:05.618886+0000 mon.a (mon.0) 1137 : cluster [DBG] osdmap e484: 8 total, 8 up, 8 in 2021-07-31T01:53:07.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:06 smithi154 conmon[34553]: audit 2021-07-31T01:53:05.699608+0000 mgr.x (mgr.24542) 249 : audit 2021-07-31T01:53:07.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:06 smithi154 conmon[34553]: [DBG] from='client.14916 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:53:07.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:06 smithi154 conmon[34553]: cluster 2021-07-31T01:53:05.874554+0000 mgr.x (mgr.24542) 250 : cluster [DBG] pgmap v507: 25 pgs: 2 active+clean+snaptrim, 22 active+clean, 1 active+clean+snaptrim_wait; 269 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 3.2 MiB/s wr, 22 op/s 2021-07-31T01:53:07.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:06 smithi154 conmon[34553]: audit 2021-07-31T01:53:06.042107+0000 mon.c (mon 2021-07-31T01:53:07.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:06 smithi154 conmon[34553]: .1) 32 : audit [DBG] from='client.? 172.21.15.45:0/342376039' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T01:53:07.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:06 smithi154 conmon[34553]: cluster 2021-07-31T01:53:07.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:06 smithi154 conmon[34553]: 2021-07-31T01:53:06.581340+0000 mon.a (mon.0) 1138 : cluster [DBG] 2021-07-31T01:53:07.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:06 smithi154 conmon[34553]: osdmap e485: 8 total, 8 up, 8 in 2021-07-31T01:53:07.026 INFO:journalctl@ceph.osd.1.smithi045.stdout:Jul 31 01:53:06 smithi045 conmon[46771]: debug 2021-07-31T01:53:07.026 INFO:journalctl@ceph.osd.1.smithi045.stdout:Jul 31 01:53:06 smithi045 conmon[46771]: 2021-07-31T01:53:06.596+0000 7f41979aa700 -1 osd.1 pg_epoch: 485 pg[3.0s0( v 485'609 (0'0,485'609] local-lis/les=56/57 n=10 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [1,2,6]p1(0) r=0 lpr=56 crt=485'609 lcod 485'607 mlcod 485'607 active+clean+snaptrim trimq=[179~1] ps=[165~1,1a4~1]] removing snap head 2021-07-31T01:53:07.026 INFO:journalctl@ceph.osd.1.smithi045.stdout:Jul 31 01:53:06 smithi045 conmon[46771]: 2021-07-31T01:53:07.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:06 smithi045 conmon[33655]: audit 2021-07-31T01:53:05.401637+0000 mgr.x (mgr.24542) 248 : audit [DBG] from='client.14910 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:53:07.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:06 smithi045 conmon[33655]: 2021-07-31T01:53:07.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:06 smithi045 conmon[33655]: cluster 2021-07-31T01:53:05.618886+0000 mon.a (mon.0) 1137 : cluster [DBG] osdmap e484: 8 total, 8 up, 8 in 2021-07-31T01:53:07.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:06 smithi045 conmon[33655]: audit 2021-07-31T01:53:05.699608 2021-07-31T01:53:07.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:06 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 249 : audit [DBG] from='client.14916 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:53:07.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:06 smithi045 conmon[33655]: cluster 2021-07-31T01:53:07.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:06 smithi045 conmon[33655]: 2021-07-31T01:53:05.874554+0000 mgr.x (mgr 2021-07-31T01:53:07.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:06 smithi045 conmon[33655]: .24542) 250 : cluster [DBG] pgmap v507: 25 pgs: 2 active+clean+snaptrim, 22 active+clean, 1 active+clean+snaptrim_wait; 269 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 3.2 MiB/s wr, 22 op/s 2021-07-31T01:53:07.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:06 smithi045 conmon[33655]: audit 2021-07 2021-07-31T01:53:07.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:06 smithi045 conmon[33655]: -31T01:53:06.042107+0000 2021-07-31T01:53:07.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:06 smithi045 conmon[33655]: mon.c (mon.1) 32 : audit [DBG] from='client.? 172.21.15.45:0/342376039' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T01:53:07.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:06 smithi045 conmon[33655]: cluster 2021-07-31T01:53:07.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:06 smithi045 conmon[33655]: 2021-07-31T01:53: 2021-07-31T01:53:07.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:06 smithi045 conmon[33655]: 06.581340+0000 mon.a (mon.0) 1138 : cluster 2021-07-31T01:53:07.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:06 smithi045 conmon[33655]: [DBG] osdmap e485: 8 total, 8 up, 8 in 2021-07-31T01:53:07.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:06 smithi045 conmon[39718]: audit 2021-07-31T01:53 2021-07-31T01:53:07.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:06 smithi045 conmon[39718]: :05.401637+0000 mgr.x (mgr.24542) 248 : audit [DBG] from='client.14910 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:53:07.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:06 smithi045 conmon[39718]: cluster 2021-07-31T01:53:07.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:06 smithi045 conmon[39718]: 2021-07-31T01:53: 2021-07-31T01:53:07.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:06 smithi045 conmon[39718]: 05.618886+0000 mon.a (mon 2021-07-31T01:53:07.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:06 smithi045 conmon[39718]: .0) 1137 : cluster [DBG] 2021-07-31T01:53:07.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:06 smithi045 conmon[39718]: osdmap e484: 8 total, 8 up, 8 in 2021-07-31T01:53:07.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:06 smithi045 conmon[39718]: audit 2021 2021-07-31T01:53:07.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:06 smithi045 conmon[39718]: -07-31T01:53: 2021-07-31T01:53:07.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:06 smithi045 conmon[39718]: 05.699608+0000 mgr.x (mgr.24542) 249 : audit 2021-07-31T01:53:07.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:06 smithi045 conmon[39718]: [DBG] from='client.14916 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:53:07.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:06 smithi045 conmon[39718]: cluster 2021-07-31T01:53:07.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:06 smithi045 conmon[39718]: 2021-07-31T01 2021-07-31T01:53:07.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:06 smithi045 conmon[39718]: :53:05.874554+0000 2021-07-31T01:53:07.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:06 smithi045 conmon[39718]: mgr.x (mgr.24542) 250 : 2021-07-31T01:53:07.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:06 smithi045 conmon[39718]: cluster [DBG] pgmap v507: 25 pgs: 2 active+clean+snaptrim, 22 active+clean, 1 active+clean+snaptrim_wait; 269 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 3.2 MiB/s wr, 22 op/s 2021-07-31T01:53:07.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:06 smithi045 conmon[39718]: audit 2021 2021-07-31T01:53:07.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:06 smithi045 conmon[39718]: -07-31T01:53:06 2021-07-31T01:53:07.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:06 smithi045 conmon[39718]: .042107+0000 mon.c (mon. 2021-07-31T01:53:07.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:06 smithi045 conmon[39718]: 1) 32 : audit [DBG] 2021-07-31T01:53:07.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:06 smithi045 conmon[39718]: from='client.? 172.21.15.45:0/342376039' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T01:53:07.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:06 smithi045 conmon[39718]: cluster 2021 2021-07-31T01:53:07.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:06 smithi045 conmon[39718]: -07-31T01:53 2021-07-31T01:53:07.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:06 smithi045 conmon[39718]: :06.581340+0000 mon.a 2021-07-31T01:53:07.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:06 smithi045 conmon[39718]: (mon.0) 1138 : cluster 2021-07-31T01:53:07.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:06 smithi045 conmon[39718]: [DBG] osdmap e485: 8 total, 8 up, 8 in 2021-07-31T01:53:07.040 INFO:tasks.rados.rados.0.smithi045.stdout:2246: finishing write tid 1 to smithi04569269-10 2021-07-31T01:53:07.041 INFO:tasks.rados.rados.0.smithi045.stdout:2246: finishing write tid 2 to smithi04569269-10 2021-07-31T01:53:07.041 INFO:tasks.rados.rados.0.smithi045.stdout:2246: finishing write tid 3 to smithi04569269-10 2021-07-31T01:53:07.041 INFO:tasks.rados.rados.0.smithi045.stdout:2246: finishing write tid 4 to smithi04569269-10 2021-07-31T01:53:07.041 INFO:tasks.rados.rados.0.smithi045.stdout:2246: finishing write tid 5 to smithi04569269-10 2021-07-31T01:53:07.042 INFO:tasks.rados.rados.0.smithi045.stdout:2246: finishing write tid 6 to smithi04569269-10 2021-07-31T01:53:07.042 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 10 v 1158 (ObjNum 690 snap 214 seq_num 690) dirty exists 2021-07-31T01:53:07.042 INFO:tasks.rados.rados.0.smithi045.stdout:2246: left oid 10 (ObjNum 690 snap 214 seq_num 690) 2021-07-31T01:53:07.042 INFO:tasks.rados.rados.0.smithi045.stdout:2243: finishing copy_from to smithi04569269-43 2021-07-31T01:53:07.043 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 43 v 1376 (ObjNum 672 snap 210 seq_num 672) dirty exists 2021-07-31T01:53:07.043 INFO:tasks.rados.rados.0.smithi045.stdout:2243: finishing copy_from racing read to smithi04569269-43 2021-07-31T01:53:07.043 INFO:tasks.rados.rados.0.smithi045.stdout:2247: finishing write tid 1 to smithi04569269-34 2021-07-31T01:53:07.043 INFO:tasks.rados.rados.0.smithi045.stdout:2247: finishing write tid 2 to smithi04569269-34 2021-07-31T01:53:07.043 INFO:tasks.rados.rados.0.smithi045.stdout:2247: finishing write tid 3 to smithi04569269-34 2021-07-31T01:53:07.044 INFO:tasks.rados.rados.0.smithi045.stdout:2247: finishing write tid 4 to smithi04569269-34 2021-07-31T01:53:07.044 INFO:tasks.rados.rados.0.smithi045.stdout:2247: finishing write tid 5 to smithi04569269-34 2021-07-31T01:53:07.044 INFO:tasks.rados.rados.0.smithi045.stdout:2247: finishing write tid 6 to smithi04569269-34 2021-07-31T01:53:07.045 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 34 v 1027 (ObjNum 691 snap 214 seq_num 691) dirty exists 2021-07-31T01:53:07.045 INFO:tasks.rados.rados.0.smithi045.stdout:2247: left oid 34 (ObjNum 691 snap 214 seq_num 691) 2021-07-31T01:53:07.628 INFO:tasks.rados.rados.0.smithi045.stdout:2243: done (4 left) 2021-07-31T01:53:07.628 INFO:tasks.rados.rados.0.smithi045.stdout:2244: done (3 left) 2021-07-31T01:53:07.628 INFO:tasks.rados.rados.0.smithi045.stdout:2246: done (2 left) 2021-07-31T01:53:07.629 INFO:tasks.rados.rados.0.smithi045.stdout:2247: done (1 left) 2021-07-31T01:53:07.629 INFO:tasks.rados.rados.0.smithi045.stdout:2248: done (0 left) 2021-07-31T01:53:07.629 INFO:tasks.rados.rados.0.smithi045.stdout:2249: rollback oid 45 current snap is 215 2021-07-31T01:53:07.629 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 45 to 214 2021-07-31T01:53:07.630 INFO:tasks.rados.rados.0.smithi045.stdout:2250: read oid 14 snap -1 2021-07-31T01:53:07.630 INFO:tasks.rados.rados.0.smithi045.stdout:2250: expect (ObjNum 659 snap 205 seq_num 659) 2021-07-31T01:53:07.630 INFO:tasks.rados.rados.0.smithi045.stdout:2251: snap_remove snap 212 2021-07-31T01:53:08.635 INFO:tasks.rados.rados.0.smithi045.stdout:2251: done (2 left) 2021-07-31T01:53:08.635 INFO:tasks.rados.rados.0.smithi045.stdout:2252: copy_from oid 36 from oid 42 current snap is 215 2021-07-31T01:53:08.635 INFO:tasks.rados.rados.0.smithi045.stdout:2253: delete oid 38 current snap is 215 2021-07-31T01:53:08.636 INFO:tasks.rados.rados.0.smithi045.stdout:2249: finishing rollback tid 0 to smithi04569269-45 2021-07-31T01:53:08.636 INFO:tasks.rados.rados.0.smithi045.stdout:2249: finishing rollback tid 1 to smithi04569269-45 2021-07-31T01:53:08.636 INFO:tasks.rados.rados.0.smithi045.stdout:2249: finishing rollback tid 2 to smithi04569269-45 2021-07-31T01:53:08.637 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 45 v 1161 (ObjNum 686 snap 214 seq_num 686) dirty exists 2021-07-31T01:53:08.637 INFO:tasks.rados.rados.0.smithi045.stdout:2250: expect (ObjNum 659 snap 205 seq_num 659) 2021-07-31T01:53:09.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:08 smithi154 conmon[34553]: cluster 2021-07-31T01:53:07.626977+0000 mon.a (mon.0) 1139 : 2021-07-31T01:53:09.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:08 smithi154 conmon[34553]: cluster [DBG] osdmap e486: 8 total, 8 up, 8 in 2021-07-31T01:53:09.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:08 smithi154 conmon[34553]: cluster 2021-07-31T01:53:07.875115+0000 mgr.x (mgr.24542) 251 : cluster [DBG] pgmap v510: 25 pgs: 2 active+clean+snaptrim, 22 active+clean, 1 active+clean+snaptrim_wait; 269 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 3.2 MiB/s wr, 22 op/s 2021-07-31T01:53:09.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:08 smithi045 conmon[33655]: cluster 2021-07-31T01:53:07.626977+0000 mon.a (mon.0) 2021-07-31T01:53:09.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:08 smithi045 conmon[33655]: 1139 : cluster [DBG] osdmap e486: 8 total, 8 up, 8 in 2021-07-31T01:53:09.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:08 smithi045 conmon[33655]: cluster 2021-07-31T01:53:07.875115+0000 mgr.x ( 2021-07-31T01:53:09.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:08 smithi045 conmon[33655]: mgr.24542) 251 : cluster [DBG] pgmap v510: 25 pgs: 2 active+clean+snaptrim, 22 active+clean, 1 active+clean+snaptrim_wait; 269 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 3.2 MiB/s wr, 22 op/s 2021-07-31T01:53:09.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:08 smithi045 conmon[39718]: cluster 2021-07-31T01:53:07.626977+0000 mon.a 2021-07-31T01:53:09.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:08 smithi045 conmon[39718]: (mon.0) 1139 : cluster [DBG] osdmap e486: 8 total, 8 up, 8 in 2021-07-31T01:53:09.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:08 smithi045 conmon[39718]: cluster 2021-07-31T01:53:09.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:08 smithi045 conmon[39718]: 2021-07-31T01:53:07.875115+0000 2021-07-31T01:53:09.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:08 smithi045 conmon[39718]: mgr.x (mgr.24542) 251 : cluster [DBG] 2021-07-31T01:53:09.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:08 smithi045 conmon[39718]: pgmap v510: 25 pgs: 2 active+clean+snaptrim, 22 active+clean, 1 active+clean+snaptrim_wait; 269 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 3.2 MiB/s wr, 22 op/s 2021-07-31T01:53:09.351 INFO:tasks.rados.rados.0.smithi045.stdout:2252: finishing copy_from to smithi04569269-36 2021-07-31T01:53:09.352 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 36 v 1001 (ObjNum 483 snap 143 seq_num 483) dirty exists 2021-07-31T01:53:09.352 INFO:tasks.rados.rados.0.smithi045.stdout:2252: finishing copy_from racing read to smithi04569269-36 2021-07-31T01:53:09.352 INFO:tasks.rados.rados.0.smithi045.stdout:2249: done (3 left) 2021-07-31T01:53:09.352 INFO:tasks.rados.rados.0.smithi045.stdout:2250: done (2 left) 2021-07-31T01:53:09.352 INFO:tasks.rados.rados.0.smithi045.stdout:2252: done (1 left) 2021-07-31T01:53:09.353 INFO:tasks.rados.rados.0.smithi045.stdout:2253: done (0 left) 2021-07-31T01:53:09.353 INFO:tasks.rados.rados.0.smithi045.stdout:2254: delete oid 41 current snap is 215 2021-07-31T01:53:09.354 INFO:tasks.rados.rados.0.smithi045.stdout:2254: done (0 left) 2021-07-31T01:53:09.355 INFO:tasks.rados.rados.0.smithi045.stdout:2255: read oid 40 snap -1 2021-07-31T01:53:09.355 INFO:tasks.rados.rados.0.smithi045.stdout:2255: expect (ObjNum 633 snap 198 seq_num 633) 2021-07-31T01:53:09.355 INFO:tasks.rados.rados.0.smithi045.stdout:2256: snap_create 2021-07-31T01:53:09.362 INFO:tasks.rados.rados.0.smithi045.stdout:2255: expect (ObjNum 633 snap 198 seq_num 633) 2021-07-31T01:53:09.633 INFO:tasks.rados.rados.0.smithi045.stdout:2255: done (1 left) 2021-07-31T01:53:09.634 INFO:tasks.rados.rados.0.smithi045.stdout:2256: done (0 left) 2021-07-31T01:53:09.634 INFO:tasks.rados.rados.0.smithi045.stdout:2257: snap_create 2021-07-31T01:53:10.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:09 smithi154 conmon[34553]: cluster 2021-07-31T01:53:08.633207+0000 mon.a (mon.0) 1140 : cluster [DBG] osdmap e487: 8 total, 8 up, 8 in 2021-07-31T01:53:10.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:09 smithi045 conmon[39718]: cluster 2021-07-31T01:53:08.633207+0000 2021-07-31T01:53:10.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:09 smithi045 conmon[39718]: mon.a (mon.0) 1140 : cluster [DBG] osdmap e487: 8 total, 8 up, 8 in 2021-07-31T01:53:10.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:09 smithi045 conmon[33655]: cluster 2021-07-31T01:53:08.633207 2021-07-31T01:53:10.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:09 smithi045 conmon[33655]: +0000 mon.a (mon.0) 1140 : cluster [DBG] osdmap e487: 8 total, 8 up, 8 in 2021-07-31T01:53:10.646 INFO:tasks.rados.rados.0.smithi045.stdout:2257: done (0 left) 2021-07-31T01:53:10.646 INFO:tasks.rados.rados.0.smithi045.stdout:2258: delete oid 40 current snap is 217 2021-07-31T01:53:10.653 INFO:tasks.rados.rados.0.smithi045.stdout:2258: done (0 left) 2021-07-31T01:53:10.653 INFO:tasks.rados.rados.0.smithi045.stdout:2259: snap_remove snap 197 2021-07-31T01:53:11.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:10 smithi154 conmon[34553]: cluster 2021-07-31T01:53:09.633404+0000 mon.a (mon.0) 1141 : 2021-07-31T01:53:11.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:10 smithi154 conmon[34553]: cluster [DBG] osdmap e488: 8 total, 8 up, 8 in 2021-07-31T01:53:11.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:10 smithi154 conmon[34553]: cluster 2021-07-31T01:53:09.875566+0000 mgr.x (mgr.24542) 252 : cluster [DBG] pgmap v513: 25 pgs: 25 active+clean; 217 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 18 MiB/s rd, 5.3 MiB/s wr, 25 op/s 2021-07-31T01:53:11.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:10 smithi045 conmon[33655]: cluster 2021-07-31T01:53:09.633404+0000 mon.a (mon.0) 2021-07-31T01:53:11.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:10 smithi045 conmon[33655]: 1141 : cluster [DBG] osdmap e488: 8 total, 8 up, 8 in 2021-07-31T01:53:11.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:10 smithi045 conmon[33655]: cluster 2021-07-31T01:53:09.875566+0000 mgr.x ( 2021-07-31T01:53:11.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:10 smithi045 conmon[33655]: mgr.24542) 252 : cluster [DBG] pgmap v513: 25 pgs: 25 active+clean; 217 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 18 MiB/s rd, 5.3 MiB/s wr, 25 op/s 2021-07-31T01:53:11.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:10 smithi045 conmon[39718]: cluster 2021-07-31T01:53:09.633404+0000 mon.a (mon. 2021-07-31T01:53:11.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:10 smithi045 conmon[39718]: 0) 1141 : cluster [DBG] osdmap e488: 8 total, 8 up, 8 in 2021-07-31T01:53:11.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:10 smithi045 conmon[39718]: cluster 2021-07-31T01:53:09.875566+0000 mgr.x (mgr.24542) 252 : cluster 2021-07-31T01:53:11.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:10 smithi045 conmon[39718]: [DBG] pgmap v513: 25 pgs: 25 active+clean; 217 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 18 MiB/s rd, 5.3 MiB/s wr, 25 op/s 2021-07-31T01:53:11.584 INFO:tasks.rados.rados.0.smithi045.stdout:2259: done (0 left) 2021-07-31T01:53:11.585 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 5 current snap is 217 2021-07-31T01:53:11.585 INFO:tasks.rados.rados.0.smithi045.stdout:2260: seq_num 695 ranges {483328=442368,925696=540672,1466368=598016,2064384=466944} 2021-07-31T01:53:11.589 INFO:tasks.rados.rados.0.smithi045.stdout:2260: writing smithi04569269-5 from 483328 to 925696 tid 1 2021-07-31T01:53:11.593 INFO:tasks.rados.rados.0.smithi045.stdout:2260: writing smithi04569269-5 from 925696 to 1466368 tid 2 2021-07-31T01:53:11.597 INFO:tasks.rados.rados.0.smithi045.stdout:2260: writing smithi04569269-5 from 1466368 to 2064384 tid 3 2021-07-31T01:53:11.600 INFO:tasks.rados.rados.0.smithi045.stdout:2260: writing smithi04569269-5 from 2064384 to 2531328 tid 4 2021-07-31T01:53:11.601 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 24 current snap is 217 2021-07-31T01:53:11.601 INFO:tasks.rados.rados.0.smithi045.stdout:2261: seq_num 696 ranges {6307840=458752,6766592=417792,7184384=753664,7938048=434176} 2021-07-31T01:53:11.639 INFO:tasks.rados.rados.0.smithi045.stdout:2261: writing smithi04569269-24 from 6307840 to 6766592 tid 1 2021-07-31T01:53:11.641 INFO:tasks.rados.rados.0.smithi045.stdout:2261: writing smithi04569269-24 from 6766592 to 7184384 tid 2 2021-07-31T01:53:11.646 INFO:tasks.rados.rados.0.smithi045.stdout:2261: writing smithi04569269-24 from 7184384 to 7938048 tid 3 2021-07-31T01:53:11.654 INFO:tasks.rados.rados.0.smithi045.stdout:2261: writing smithi04569269-24 from 7938048 to 8372224 tid 4 2021-07-31T01:53:11.655 INFO:tasks.rados.rados.0.smithi045.stdout:2262: rollback oid 13 current snap is 217 2021-07-31T01:53:11.655 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 13 to 206 2021-07-31T01:53:11.655 INFO:tasks.rados.rados.0.smithi045.stdout:2260: finishing write tid 1 to smithi04569269-5 2021-07-31T01:53:11.655 INFO:tasks.rados.rados.0.smithi045.stdout:2260: finishing write tid 2 to smithi04569269-5 2021-07-31T01:53:11.656 INFO:tasks.rados.rados.0.smithi045.stdout:2260: finishing write tid 3 to smithi04569269-5 2021-07-31T01:53:11.656 INFO:tasks.rados.rados.0.smithi045.stdout:2260: finishing write tid 4 to smithi04569269-5 2021-07-31T01:53:11.656 INFO:tasks.rados.rados.0.smithi045.stdout:2260: finishing write tid 5 to smithi04569269-5 2021-07-31T01:53:11.656 INFO:tasks.rados.rados.0.smithi045.stdout:2260: finishing write tid 6 to smithi04569269-5 2021-07-31T01:53:11.656 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 5 v 1158 (ObjNum 695 snap 217 seq_num 695) dirty exists 2021-07-31T01:53:11.657 INFO:tasks.rados.rados.0.smithi045.stdout:2260: left oid 5 (ObjNum 695 snap 217 seq_num 695) 2021-07-31T01:53:11.657 INFO:tasks.rados.rados.0.smithi045.stdout:2261: finishing write tid 1 to smithi04569269-24 2021-07-31T01:53:11.657 INFO:tasks.rados.rados.0.smithi045.stdout:2261: finishing write tid 2 to smithi04569269-24 2021-07-31T01:53:11.657 INFO:tasks.rados.rados.0.smithi045.stdout:2260: done (2 left) 2021-07-31T01:53:11.657 INFO:tasks.rados.rados.0.smithi045.stdout:2263: delete oid 45 current snap is 217 2021-07-31T01:53:11.658 INFO:tasks.rados.rados.0.smithi045.stdout:2261: finishing write tid 3 to smithi04569269-24 2021-07-31T01:53:11.658 INFO:tasks.rados.rados.0.smithi045.stdout:2261: finishing write tid 4 to smithi04569269-24 2021-07-31T01:53:11.658 INFO:tasks.rados.rados.0.smithi045.stdout:2263: done (2 left) 2021-07-31T01:53:11.658 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 9 current snap is 217 2021-07-31T01:53:11.659 INFO:tasks.rados.rados.0.smithi045.stdout:2264: seq_num 698 ranges {1818624=499712,2318336=589824,2908160=770048,3678208=245760} 2021-07-31T01:53:11.664 INFO:tasks.rados.rados.0.smithi045.stdout:2264: writing smithi04569269-9 from 1818624 to 2318336 tid 1 2021-07-31T01:53:11.667 INFO:tasks.rados.rados.0.smithi045.stdout:2264: writing smithi04569269-9 from 2318336 to 2908160 tid 2 2021-07-31T01:53:11.671 INFO:tasks.rados.rados.0.smithi045.stdout:2264: writing smithi04569269-9 from 2908160 to 3678208 tid 3 2021-07-31T01:53:11.673 INFO:tasks.rados.rados.0.smithi045.stdout:2264: writing smithi04569269-9 from 3678208 to 3923968 tid 4 2021-07-31T01:53:11.673 INFO:tasks.rados.rados.0.smithi045.stdout:2265: delete oid 44 current snap is 217 2021-07-31T01:53:11.674 INFO:tasks.rados.rados.0.smithi045.stdout:2261: finishing write tid 5 to smithi04569269-24 2021-07-31T01:53:11.674 INFO:tasks.rados.rados.0.smithi045.stdout:2261: finishing write tid 6 to smithi04569269-24 2021-07-31T01:53:11.674 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 24 v 553 (ObjNum 696 snap 217 seq_num 696) dirty exists 2021-07-31T01:53:11.674 INFO:tasks.rados.rados.0.smithi045.stdout:2261: left oid 24 (ObjNum 696 snap 217 seq_num 696) 2021-07-31T01:53:11.675 INFO:tasks.rados.rados.0.smithi045.stdout:2262: finishing rollback tid 0 to smithi04569269-13 2021-07-31T01:53:11.677 INFO:tasks.rados.rados.0.smithi045.stdout:2262: finishing rollback tid 1 to smithi04569269-13 2021-07-31T01:53:11.678 INFO:tasks.rados.rados.0.smithi045.stdout:2262: finishing rollback tid 2 to smithi04569269-13 2021-07-31T01:53:11.678 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 13 v 1380 (ObjNum 662 snap 205 seq_num 662) dirty exists 2021-07-31T01:53:11.678 INFO:tasks.rados.rados.0.smithi045.stdout:2264: finishing write tid 1 to smithi04569269-9 2021-07-31T01:53:11.678 INFO:tasks.rados.rados.0.smithi045.stdout:2264: finishing write tid 2 to smithi04569269-9 2021-07-31T01:53:11.679 INFO:tasks.rados.rados.0.smithi045.stdout:2261: done (3 left) 2021-07-31T01:53:11.679 INFO:tasks.rados.rados.0.smithi045.stdout:2262: done (2 left) 2021-07-31T01:53:11.679 INFO:tasks.rados.rados.0.smithi045.stdout:2265: done (1 left) 2021-07-31T01:53:11.680 INFO:tasks.rados.rados.0.smithi045.stdout:2266: snap_remove snap 214 2021-07-31T01:53:12.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:11 smithi154 conmon[34553]: cluster 2021-07-31T01:53:10.644993+0000 mon.a (mon.0) 1142 : cluster [DBG] osdmap e489: 8 total, 8 up, 8 in 2021-07-31T01:53:12.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:11 smithi154 conmon[34553]: cluster 2021-07-31T01:53:11.582898+0000 mon.a (mon.0) 1143 : cluster [DBG] osdmap e490: 8 total, 8 up, 8 in 2021-07-31T01:53:12.026 INFO:journalctl@ceph.osd.1.smithi045.stdout:Jul 31 01:53:11 smithi045 conmon[46771]: debug 2021-07-31T01:53:11.588+0000 7f41939a2700 -1 osd.1 pg_epoch: 490 pg[3.0s0( v 490'615 (0'0,490'615] local-lis/les=56/57 n=7 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [1,2,6]p1(0) r=0 lpr=56 luod=487'613 crt=487'613 lcod 485'611 mlcod 485'611 active+clean+snaptrim trimq=[185~1]] removing snap head 2021-07-31T01:53:12.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:11 smithi045 conmon[39718]: cluster 2021-07-31T01:53:10.644993+0000 mon.a (mon.0) 1142 2021-07-31T01:53:12.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:11 smithi045 conmon[39718]: : cluster [DBG] osdmap e489: 8 total, 8 up, 8 in 2021-07-31T01:53:12.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:11 smithi045 conmon[39718]: cluster 2021-07-31T01:53:11. 2021-07-31T01:53:12.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:11 smithi045 conmon[39718]: 582898+0000 mon.a (mon.0) 1143 : cluster [DBG] osdmap e490: 8 total, 8 up, 8 in 2021-07-31T01:53:12.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:11 smithi045 conmon[33655]: cluster 2021-07-31T01:53:12.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:11 smithi045 conmon[33655]: 2021-07-31T01:53:12.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:11 smithi045 conmon[33655]: 2021-07-31T01:53:10.644993 2021-07-31T01:53:12.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:11 smithi045 conmon[33655]: +0000 mon.a (mon.0) 1142 : cluster [DBG] osdmap e489: 8 total, 8 up, 8 in 2021-07-31T01:53:12.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:11 smithi045 conmon[33655]: cluster 2021-07-31T01:53:12.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:11 smithi045 conmon[33655]: 2021-07-31T01:53:11.582898+0000 2021-07-31T01:53:12.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:11 smithi045 conmon[33655]: mon.a (mon.0) 1143 : cluster [DBG] osdmap e490: 8 total, 8 up, 8 in 2021-07-31T01:53:12.653 INFO:tasks.rados.rados.0.smithi045.stdout:2266: done (1 left) 2021-07-31T01:53:12.654 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 38 current snap is 217 2021-07-31T01:53:12.654 INFO:tasks.rados.rados.0.smithi045.stdout:2267: seq_num 700 ranges {0=532480,532480=409600,942080=737280,1679360=458752} 2021-07-31T01:53:12.656 INFO:tasks.rados.rados.0.smithi045.stdout:2267: writing smithi04569269-38 from 0 to 532480 tid 1 2021-07-31T01:53:12.659 INFO:tasks.rados.rados.0.smithi045.stdout:2267: writing smithi04569269-38 from 532480 to 942080 tid 2 2021-07-31T01:53:12.664 INFO:tasks.rados.rados.0.smithi045.stdout:2267: writing smithi04569269-38 from 942080 to 1679360 tid 3 2021-07-31T01:53:12.666 INFO:tasks.rados.rados.0.smithi045.stdout:2267: writing smithi04569269-38 from 1679360 to 2138112 tid 4 2021-07-31T01:53:12.667 INFO:tasks.rados.rados.0.smithi045.stdout:2268: snap_remove snap 211 2021-07-31T01:53:13.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:12 smithi154 conmon[34553]: cluster 2021-07-31T01:53:11.876037+0000 mgr.x 2021-07-31T01:53:13.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:12 smithi154 conmon[34553]: (mgr.24542) 253 : cluster [DBG] pgmap v516: 25 pgs: 25 active+clean; 217 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 18 MiB/s rd, 5.3 MiB/s wr, 25 op/s 2021-07-31T01:53:13.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:12 smithi045 conmon[33655]: cluster 2021-07-31T01:53:11.876037+0000 mgr.x (mgr 2021-07-31T01:53:13.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:12 smithi045 conmon[33655]: .24542) 253 : cluster [DBG] pgmap v516: 25 pgs: 25 active+clean; 217 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 18 MiB/s rd, 5.3 MiB/s wr, 25 op/s 2021-07-31T01:53:13.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:12 smithi045 conmon[39718]: cluster 2021-07-31T01:53:11.876037 2021-07-31T01:53:13.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:12 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 253 : cluster [DBG] pgmap v516: 25 pgs: 25 active+clean; 217 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 18 MiB/s rd, 5.3 MiB/s wr, 25 op/s 2021-07-31T01:53:13.659 INFO:tasks.rados.rados.0.smithi045.stdout:2268: done (2 left) 2021-07-31T01:53:13.660 INFO:tasks.rados.rados.0.smithi045.stdout:2269: copy_from oid 29 from oid 2 current snap is 217 2021-07-31T01:53:13.660 INFO:tasks.rados.rados.0.smithi045.stdout:2264: finishing write tid 3 to smithi04569269-9 2021-07-31T01:53:13.660 INFO:tasks.rados.rados.0.smithi045.stdout:2264: finishing write tid 4 to smithi04569269-9 2021-07-31T01:53:13.660 INFO:tasks.rados.rados.0.smithi045.stdout:2264: finishing write tid 5 to smithi04569269-9 2021-07-31T01:53:13.661 INFO:tasks.rados.rados.0.smithi045.stdout:2264: finishing write tid 6 to smithi04569269-9 2021-07-31T01:53:13.661 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 9 v 1056 (ObjNum 698 snap 217 seq_num 698) dirty exists 2021-07-31T01:53:13.661 INFO:tasks.rados.rados.0.smithi045.stdout:2264: left oid 9 (ObjNum 698 snap 217 seq_num 698) 2021-07-31T01:53:13.661 INFO:tasks.rados.rados.0.smithi045.stdout:2267: finishing write tid 1 to smithi04569269-38 2021-07-31T01:53:13.662 INFO:tasks.rados.rados.0.smithi045.stdout:2267: finishing write tid 2 to smithi04569269-38 2021-07-31T01:53:13.662 INFO:tasks.rados.rados.0.smithi045.stdout:2267: finishing write tid 3 to smithi04569269-38 2021-07-31T01:53:13.662 INFO:tasks.rados.rados.0.smithi045.stdout:2267: finishing write tid 4 to smithi04569269-38 2021-07-31T01:53:13.662 INFO:tasks.rados.rados.0.smithi045.stdout:2267: finishing write tid 5 to smithi04569269-38 2021-07-31T01:53:13.662 INFO:tasks.rados.rados.0.smithi045.stdout:2267: finishing write tid 6 to smithi04569269-38 2021-07-31T01:53:13.663 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 38 v 1163 (ObjNum 700 snap 217 seq_num 700) dirty exists 2021-07-31T01:53:13.663 INFO:tasks.rados.rados.0.smithi045.stdout:2267: left oid 38 (ObjNum 700 snap 217 seq_num 700) 2021-07-31T01:53:13.664 INFO:tasks.rados.rados.0.smithi045.stdout:2264: done (2 left) 2021-07-31T01:53:13.664 INFO:tasks.rados.rados.0.smithi045.stdout:2267: done (1 left) 2021-07-31T01:53:13.664 INFO:tasks.rados.rados.0.smithi045.stdout:2270: read oid 42 snap 215 2021-07-31T01:53:13.664 INFO:tasks.rados.rados.0.smithi045.stdout:2270: expect (ObjNum 483 snap 143 seq_num 483) 2021-07-31T01:53:13.664 INFO:tasks.rados.rados.0.smithi045.stdout:2271: copy_from oid 10 from oid 27 current snap is 217 2021-07-31T01:53:13.665 INFO:tasks.rados.rados.0.smithi045.stdout:2272: rmattr oid 37 current snap is 217 2021-07-31T01:53:13.665 INFO:tasks.rados.rados.0.smithi045.stdout:2273: rmattr oid 18 current snap is 217 2021-07-31T01:53:13.666 INFO:tasks.rados.rados.0.smithi045.stdout:2273: done (4 left) 2021-07-31T01:53:13.667 INFO:tasks.rados.rados.0.smithi045.stdout:2274: snap_create 2021-07-31T01:53:13.671 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 37 v 988 (ObjNum 639 snap 199 seq_num 639) dirty exists 2021-07-31T01:53:13.673 INFO:tasks.rados.rados.0.smithi045.stdout:2270: expect (ObjNum 483 snap 143 seq_num 483) 2021-07-31T01:53:13.727 INFO:tasks.rados.rados.0.smithi045.stdout:2271: finishing copy_from to smithi04569269-10 2021-07-31T01:53:13.727 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 10 v 1068 (ObjNum 523 snap 154 seq_num 523) dirty exists 2021-07-31T01:53:13.727 INFO:tasks.rados.rados.0.smithi045.stdout:2271: finishing copy_from racing read to smithi04569269-10 2021-07-31T01:53:13.728 INFO:tasks.rados.rados.0.smithi045.stdout:2269: finishing copy_from to smithi04569269-29 2021-07-31T01:53:13.728 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 29 v 1045 (ObjNum 642 snap 199 seq_num 642) dirty exists 2021-07-31T01:53:13.728 INFO:tasks.rados.rados.0.smithi045.stdout:2269: finishing copy_from racing read to smithi04569269-29 2021-07-31T01:53:13.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:13 smithi045 conmon[39718]: cluster 2021-07-31T01:53:12.652595+0000 mon.a (mon.0) 2021-07-31T01:53:13.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:13 smithi045 conmon[39718]: 1144 : cluster [DBG] osdmap e491: 8 total, 8 up, 8 in 2021-07-31T01:53:13.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:53:13 smithi045 conmon[61210]: level=error ts=2021-07-31T01:53:13.519Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:53:13.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:53:13 smithi045 conmon[61210]: level=error ts=2021-07-31T01:53:13.519Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:53:13.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:53:13 smithi045 conmon[61210]: level=error ts=2021-07-31T01:53:13.519Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T01:53:13.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:13 smithi045 conmon[33655]: cluster 2021-07-31T01:53:12.652595+0000 mon.a (mon.0) 1144 2021-07-31T01:53:13.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:13 smithi045 conmon[33655]: : cluster [DBG] osdmap e491: 8 total, 8 up, 8 in 2021-07-31T01:53:14.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:13 smithi154 conmon[34553]: cluster 2021-07-31T01:53:12.652595+0000 mon.a (mon.0) 1144 : cluster [DBG] osdmap e491: 8 total, 8 up, 8 in 2021-07-31T01:53:14.003 INFO:journalctl@ceph.osd.5.smithi154.stdout:Jul 31 01:53:13 smithi154 conmon[42050]: debug 2021-07-31T01:53:13.664+0000 7fc9b105c700 -1 osd.5 pg_epoch: 492 pg[3.cs0( v 492'919 (0'0,492'919] local-lis/les=56/57 n=10 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [5,3,6]p5(0) r=0 lpr=56 luod=491'917 crt=491'917 lcod 491'916 mlcod 491'916 active+clean+snaptrim trimq=[19e~1] ps=[185~1,1a9~1]] removing snap head 2021-07-31T01:53:14.665 INFO:tasks.rados.rados.0.smithi045.stdout:2269: done (4 left) 2021-07-31T01:53:14.665 INFO:tasks.rados.rados.0.smithi045.stdout:2270: done (3 left) 2021-07-31T01:53:14.666 INFO:tasks.rados.rados.0.smithi045.stdout:2271: done (2 left) 2021-07-31T01:53:14.666 INFO:tasks.rados.rados.0.smithi045.stdout:2272: done (1 left) 2021-07-31T01:53:14.667 INFO:tasks.rados.rados.0.smithi045.stdout:2274: done (0 left) 2021-07-31T01:53:14.667 INFO:tasks.rados.rados.0.smithi045.stdout:2275: read oid 19 snap -1 2021-07-31T01:53:14.668 INFO:tasks.rados.rados.0.smithi045.stdout:2275: expect (ObjNum 172 snap 41 seq_num 172) 2021-07-31T01:53:14.668 INFO:tasks.rados.rados.0.smithi045.stdout:2276: setattr oid 2 current snap is 218 2021-07-31T01:53:14.669 INFO:tasks.rados.rados.0.smithi045.stdout:2277: read oid 20 snap -1 2021-07-31T01:53:14.670 INFO:tasks.rados.rados.0.smithi045.stdout:2277: expect (ObjNum 556 snap 166 seq_num 556) 2021-07-31T01:53:14.670 INFO:tasks.rados.rados.0.smithi045.stdout:2278: snap_remove snap 205 2021-07-31T01:53:15.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:14 smithi154 conmon[34553]: cluster 2021-07-31T01:53:13.657714+0000 mon.a (mon.0) 1145 : cluster 2021-07-31T01:53:15.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:14 smithi154 conmon[34553]: [DBG] osdmap e492: 8 total, 8 up, 8 in 2021-07-31T01:53:15.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:14 smithi154 conmon[34553]: cluster 2021-07-31T01:53:13.876473+0000 mgr.x (mgr.24542) 254 : cluster [DBG] pgmap v519: 25 pgs: 25 active+clean; 219 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 2.0 MiB/s rd, 4.4 MiB/s wr, 11 op/s 2021-07-31T01:53:15.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:14 smithi045 conmon[33655]: cluster 2021-07-31T01:53:13.657714+0000 mon.a (mon.0) 1145 : cluster 2021-07-31T01:53:15.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:14 smithi045 conmon[33655]: [DBG] osdmap e492: 8 total, 8 up, 8 in 2021-07-31T01:53:15.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:14 smithi045 conmon[33655]: cluster 2021-07-31T01:53:13.876473+0000 mgr.x (mgr.24542) 254 : cluster [DBG] pgmap v519: 25 pgs: 25 active+clean; 219 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 2.0 MiB/s rd, 4.4 MiB/s wr, 11 op/s 2021-07-31T01:53:15.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:14 smithi045 conmon[39718]: cluster 2021-07-31T01:53:13.657714+0000 mon.a (mon.0 2021-07-31T01:53:15.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:14 smithi045 conmon[39718]: ) 1145 : cluster [DBG] osdmap e492: 8 total, 8 up, 8 in 2021-07-31T01:53:15.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:14 smithi045 conmon[39718]: cluster 2021-07-31T01:53:13.876473+0000 2021-07-31T01:53:15.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:14 smithi045 conmon[39718]: mgr.x (mgr.24542) 254 : cluster [DBG] pgmap v519: 25 pgs: 25 active+clean; 219 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 2.0 MiB/s rd, 4.4 MiB/s wr, 11 op/s 2021-07-31T01:53:15.667 INFO:tasks.rados.rados.0.smithi045.stdout:2278: done (3 left) 2021-07-31T01:53:15.667 INFO:tasks.rados.rados.0.smithi045.stdout:2279: copy_from oid 34 from oid 29 current snap is 218 2021-07-31T01:53:15.667 INFO:tasks.rados.rados.0.smithi045.stdout:2275: expect (ObjNum 172 snap 41 seq_num 172) 2021-07-31T01:53:15.695 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 2 v 1057 (ObjNum 642 snap 199 seq_num 642) dirty exists 2021-07-31T01:53:15.695 INFO:tasks.rados.rados.0.smithi045.stdout:2277: expect (ObjNum 556 snap 166 seq_num 556) 2021-07-31T01:53:15.822 INFO:tasks.rados.rados.0.smithi045.stdout:2279: finishing copy_from to smithi04569269-34 2021-07-31T01:53:15.823 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 34 v 1045 (ObjNum 642 snap 199 seq_num 642) dirty exists 2021-07-31T01:53:15.823 INFO:tasks.rados.rados.0.smithi045.stdout:2279: finishing copy_from racing read to smithi04569269-34 2021-07-31T01:53:15.823 INFO:tasks.rados.rados.0.smithi045.stdout:2275: done (3 left) 2021-07-31T01:53:15.823 INFO:tasks.rados.rados.0.smithi045.stdout:2276: done (2 left) 2021-07-31T01:53:15.824 INFO:tasks.rados.rados.0.smithi045.stdout:2277: done (1 left) 2021-07-31T01:53:15.824 INFO:tasks.rados.rados.0.smithi045.stdout:2279: done (0 left) 2021-07-31T01:53:15.824 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 8 current snap is 218 2021-07-31T01:53:15.825 INFO:tasks.rados.rados.0.smithi045.stdout:2280: seq_num 701 ranges {1679360=548864,2228224=696320,2924544=540672,3465216=368640} 2021-07-31T01:53:15.833 INFO:tasks.rados.rados.0.smithi045.stdout:2280: writing smithi04569269-8 from 1679360 to 2228224 tid 1 2021-07-31T01:53:15.837 INFO:tasks.rados.rados.0.smithi045.stdout:2280: writing smithi04569269-8 from 2228224 to 2924544 tid 2 2021-07-31T01:53:15.841 INFO:tasks.rados.rados.0.smithi045.stdout:2280: writing smithi04569269-8 from 2924544 to 3465216 tid 3 2021-07-31T01:53:15.843 INFO:tasks.rados.rados.0.smithi045.stdout:2280: writing smithi04569269-8 from 3465216 to 3833856 tid 4 2021-07-31T01:53:15.844 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 30 current snap is 218 2021-07-31T01:53:15.844 INFO:tasks.rados.rados.0.smithi045.stdout:2280: finishing write tid 1 to smithi04569269-8 2021-07-31T01:53:15.845 INFO:tasks.rados.rados.0.smithi045.stdout:2280: finishing write tid 2 to smithi04569269-8 2021-07-31T01:53:15.845 INFO:tasks.rados.rados.0.smithi045.stdout:2280: finishing write tid 3 to smithi04569269-8 2021-07-31T01:53:15.845 INFO:tasks.rados.rados.0.smithi045.stdout:2281: seq_num 702 ranges {0=565248,565248=581632,1146880=745472,1892352=278528} 2021-07-31T01:53:15.846 INFO:tasks.rados.rados.0.smithi045.stdout:2281: writing smithi04569269-30 from 0 to 565248 tid 1 2021-07-31T01:53:15.850 INFO:tasks.rados.rados.0.smithi045.stdout:2281: writing smithi04569269-30 from 565248 to 1146880 tid 2 2021-07-31T01:53:15.854 INFO:tasks.rados.rados.0.smithi045.stdout:2281: writing smithi04569269-30 from 1146880 to 1892352 tid 3 2021-07-31T01:53:15.856 INFO:tasks.rados.rados.0.smithi045.stdout:2281: writing smithi04569269-30 from 1892352 to 2170880 tid 4 2021-07-31T01:53:15.856 INFO:tasks.rados.rados.0.smithi045.stdout:2282: read oid 44 snap -1 2021-07-31T01:53:15.856 INFO:tasks.rados.rados.0.smithi045.stdout:2282: expect deleted 2021-07-31T01:53:15.857 INFO:tasks.rados.rados.0.smithi045.stdout:2283: snap_remove snap 217 2021-07-31T01:53:16.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:15 smithi154 conmon[34553]: cluster 2021-07-31T01:53:16.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:15 smithi154 conmon[34553]: 2021-07-31T01:53:16.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:15 smithi154 conmon[34553]: 2021-07-31T01:53:14.663744+0000 mon.a (mon 2021-07-31T01:53:16.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:15 smithi154 conmon[34553]: .0) 1146 : cluster [DBG] osdmap e493: 8 total, 8 up, 8 in 2021-07-31T01:53:16.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:15 smithi045 conmon[33655]: cluster 2021-07-31T01:53:16.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:15 smithi045 conmon[33655]: 2021-07-31T01:53:14.663744 2021-07-31T01:53:16.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:15 smithi045 conmon[33655]: +0000 mon.a (mon.0) 1146 : cluster [DBG] osdmap e493: 8 total, 8 up, 8 in 2021-07-31T01:53:16.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:15 smithi045 conmon[39718]: cluster 2021-07-31T01:53:16.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:15 smithi045 conmon[39718]: 2021-07-31T01:53:16.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:15 smithi045 conmon[39718]: 2021 2021-07-31T01:53:16.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:15 smithi045 conmon[39718]: - 2021-07-31T01:53:16.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:15 smithi045 conmon[39718]: 07 2021-07-31T01:53:16.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:15 smithi045 conmon[39718]: - 2021-07-31T01:53:16.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:15 smithi045 conmon[39718]: 31 2021-07-31T01:53:16.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:15 smithi045 conmon[39718]: T 2021-07-31T01:53:16.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:15 smithi045 conmon[39718]: 01 2021-07-31T01:53:16.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:15 smithi045 conmon[39718]: : 2021-07-31T01:53:16.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:15 smithi045 conmon[39718]: 53 2021-07-31T01:53:16.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:15 smithi045 conmon[39718]: : 2021-07-31T01:53:16.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:15 smithi045 conmon[39718]: 14 2021-07-31T01:53:16.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:15 smithi045 conmon[39718]: . 2021-07-31T01:53:16.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:15 smithi045 conmon[39718]: 663744 2021-07-31T01:53:16.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:15 smithi045 conmon[39718]: +0000 2021-07-31T01:53:16.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:15 smithi045 conmon[39718]: 2021-07-31T01:53:16.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:15 smithi045 conmon[39718]: mon.a 2021-07-31T01:53:16.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:15 smithi045 conmon[39718]: ( 2021-07-31T01:53:16.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:15 smithi045 conmon[39718]: mon 2021-07-31T01:53:16.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:15 smithi045 conmon[39718]: . 2021-07-31T01:53:16.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:15 smithi045 conmon[39718]: 0 2021-07-31T01:53:16.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:15 smithi045 conmon[39718]: ) 2021-07-31T01:53:16.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:15 smithi045 conmon[39718]: 1146 2021-07-31T01:53:16.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:15 smithi045 conmon[39718]: : 2021-07-31T01:53:16.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:15 smithi045 conmon[39718]: cluster 2021-07-31T01:53:16.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:15 smithi045 conmon[39718]: 2021-07-31T01:53:16.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:15 smithi045 conmon[39718]: [DBG] 2021-07-31T01:53:16.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:15 smithi045 conmon[39718]: 2021-07-31T01:53:16.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:15 smithi045 conmon[39718]: osdmap e493: 8 total, 8 up, 8 in 2021-07-31T01:53:16.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:15 smithi045 conmon[39718]: 2021-07-31T01:53:16.588 INFO:tasks.rados.rados.0.smithi045.stdout:2283: done (3 left) 2021-07-31T01:53:16.589 INFO:tasks.rados.rados.0.smithi045.stdout:2284: read oid 39 snap 215 2021-07-31T01:53:16.589 INFO:tasks.rados.rados.0.smithi045.stdout:2284: expect (ObjNum 637 snap 198 seq_num 637) 2021-07-31T01:53:16.589 INFO:tasks.rados.rados.0.smithi045.stdout:2285: snap_create 2021-07-31T01:53:16.589 INFO:tasks.rados.rados.0.smithi045.stdout:2280: finishing write tid 4 to smithi04569269-8 2021-07-31T01:53:16.590 INFO:tasks.rados.rados.0.smithi045.stdout:2280: finishing write tid 5 to smithi04569269-8 2021-07-31T01:53:16.590 INFO:tasks.rados.rados.0.smithi045.stdout:2280: finishing write tid 6 to smithi04569269-8 2021-07-31T01:53:16.590 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 8 v 1500 (ObjNum 701 snap 218 seq_num 701) dirty exists 2021-07-31T01:53:16.590 INFO:tasks.rados.rados.0.smithi045.stdout:2280: left oid 8 (ObjNum 701 snap 218 seq_num 701) 2021-07-31T01:53:16.591 INFO:tasks.rados.rados.0.smithi045.stdout:2281: finishing write tid 1 to smithi04569269-30 2021-07-31T01:53:16.591 INFO:tasks.rados.rados.0.smithi045.stdout:2281: finishing write tid 2 to smithi04569269-30 2021-07-31T01:53:16.591 INFO:tasks.rados.rados.0.smithi045.stdout:2281: finishing write tid 3 to smithi04569269-30 2021-07-31T01:53:16.591 INFO:tasks.rados.rados.0.smithi045.stdout:2281: finishing write tid 4 to smithi04569269-30 2021-07-31T01:53:16.591 INFO:tasks.rados.rados.0.smithi045.stdout:2281: finishing write tid 5 to smithi04569269-30 2021-07-31T01:53:16.592 INFO:tasks.rados.rados.0.smithi045.stdout:2281: finishing write tid 6 to smithi04569269-30 2021-07-31T01:53:16.592 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 30 v 558 (ObjNum 702 snap 218 seq_num 702) dirty exists 2021-07-31T01:53:16.592 INFO:tasks.rados.rados.0.smithi045.stdout:2281: left oid 30 (ObjNum 702 snap 218 seq_num 702) 2021-07-31T01:53:16.599 INFO:tasks.rados.rados.0.smithi045.stdout:2284: expect (ObjNum 637 snap 198 seq_num 637) 2021-07-31T01:53:17.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:16 smithi154 conmon[34553]: cluster 2021-07-31T01:53:15.665148+0000 mon.a (mon.0) 1147 : cluster [DBG] osdmap e494: 8 total, 8 up, 8 in 2021-07-31T01:53:17.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:16 smithi154 conmon[34553]: cluster 2021-07-31T01:53: 2021-07-31T01:53:17.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:16 smithi154 conmon[34553]: 15.876903+0000 mgr.x (mgr.24542) 255 : cluster [DBG] pgmap v522: 25 pgs: 25 active+clean; 219 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 2.0 MiB/s rd, 4.4 MiB/s wr, 11 op/s 2021-07-31T01:53:17.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:16 smithi154 conmon[34553]: cluster 2021-07-31T01:53:16.586962+0000 mon.a (mon.0) 1148 : cluster [DBG] osdmap e495: 8 total, 8 up, 8 in 2021-07-31T01:53:17.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:16 smithi045 conmon[33655]: cluster 2021-07-31T01:53:15.665148+0000 mon.a (mon. 2021-07-31T01:53:17.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:16 smithi045 conmon[33655]: 0) 1147 : cluster [DBG] osdmap e494: 8 total, 8 up, 8 in 2021-07-31T01:53:17.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:16 smithi045 conmon[33655]: cluster 2021-07-31T01:53:15.876903+0000 mgr.x (mgr.24542) 255 : cluster [DBG] pgmap v522: 25 pgs: 25 active+clean; 219 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 2.0 MiB/s rd, 4.4 MiB/s wr, 11 op/s 2021-07-31T01:53:17.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:16 smithi045 conmon[33655]: cluster 2021-07-31T01:53:16 2021-07-31T01:53:17.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:16 smithi045 conmon[33655]: .586962+0000 mon.a (mon.0 2021-07-31T01:53:17.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:16 smithi045 conmon[33655]: ) 1148 : cluster [DBG] 2021-07-31T01:53:17.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:16 smithi045 conmon[33655]: osdmap e495: 8 total, 8 up, 8 in 2021-07-31T01:53:17.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:16 smithi045 conmon[39718]: cluster 2021-07-31T01:53 2021-07-31T01:53:17.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:16 smithi045 conmon[39718]: :15.665148+0000 mon.a (mon.0) 1147 : cluster [DBG] osdmap e494: 8 total, 8 up, 8 in 2021-07-31T01:53:17.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:16 smithi045 conmon[39718]: cluster 2021 2021-07-31T01:53:17.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:16 smithi045 conmon[39718]: -07-31T01:53:15.876903 2021-07-31T01:53:17.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:16 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 2021-07-31T01:53:17.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:16 smithi045 conmon[39718]: 255 : cluster [DBG] pgmap v522: 25 pgs: 25 active+clean; 219 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 2.0 MiB/s rd, 4.4 MiB/s wr, 11 op/s 2021-07-31T01:53:17.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:16 smithi045 conmon[39718]: cluster 2021-07-31T01:53:17.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:16 smithi045 conmon[39718]: 2021-07-31T01:53 2021-07-31T01:53:17.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:16 smithi045 conmon[39718]: :16.586962+0000 mon.a ( 2021-07-31T01:53:17.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:16 smithi045 conmon[39718]: mon.0) 1148 : cluster 2021-07-31T01:53:17.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:16 smithi045 conmon[39718]: [DBG] osdmap e495: 8 total, 8 up, 8 in 2021-07-31T01:53:17.594 INFO:tasks.rados.rados.0.smithi045.stdout:2280: done (4 left) 2021-07-31T01:53:17.594 INFO:tasks.rados.rados.0.smithi045.stdout:2281: done (3 left) 2021-07-31T01:53:17.595 INFO:tasks.rados.rados.0.smithi045.stdout:2282: done (2 left) 2021-07-31T01:53:17.595 INFO:tasks.rados.rados.0.smithi045.stdout:2284: done (1 left) 2021-07-31T01:53:17.595 INFO:tasks.rados.rados.0.smithi045.stdout:2285: done (0 left) 2021-07-31T01:53:17.595 INFO:tasks.rados.rados.0.smithi045.stdout:2286: setattr oid 44 current snap is 219 2021-07-31T01:53:17.596 INFO:tasks.rados.rados.0.smithi045.stdout:2287: rollback oid 17 current snap is 219 2021-07-31T01:53:17.596 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 17 to 218 2021-07-31T01:53:17.596 INFO:tasks.rados.rados.0.smithi045.stdout:2288: snap_remove snap 206 2021-07-31T01:53:18.601 INFO:tasks.rados.rados.0.smithi045.stdout:2288: done (2 left) 2021-07-31T01:53:18.601 INFO:tasks.rados.rados.0.smithi045.stdout:2289: delete oid 48 current snap is 219 2021-07-31T01:53:18.601 INFO:tasks.rados.rados.0.smithi045.stdout:2287: finishing rollback tid 0 to smithi04569269-17 2021-07-31T01:53:18.602 INFO:tasks.rados.rados.0.smithi045.stdout:2287: finishing rollback tid 1 to smithi04569269-17 2021-07-31T01:53:18.602 INFO:tasks.rados.rados.0.smithi045.stdout:2287: finishing rollback tid 2 to smithi04569269-17 2021-07-31T01:53:18.603 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 17 v 1165 (ObjNum 664 snap 208 seq_num 664) dirty exists 2021-07-31T01:53:18.603 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 44 v 1378 (ObjNum 0 snap 32731 seq_num 3189385504) dirty exists 2021-07-31T01:53:18.610 INFO:tasks.rados.rados.0.smithi045.stdout:2286: done (2 left) 2021-07-31T01:53:18.610 INFO:tasks.rados.rados.0.smithi045.stdout:2287: done (1 left) 2021-07-31T01:53:18.610 INFO:tasks.rados.rados.0.smithi045.stdout:2289: done (0 left) 2021-07-31T01:53:18.611 INFO:tasks.rados.rados.0.smithi045.stdout:2290: setattr oid 14 current snap is 219 2021-07-31T01:53:18.614 INFO:tasks.rados.rados.0.smithi045.stdout:2291: snap_remove snap 216 2021-07-31T01:53:19.003 INFO:journalctl@ceph.osd.4.smithi154.stdout:Jul 31 01:53:18 smithi154 conmon[38506]: debug 2021-07-31T01:53:18.602+0000 7feae39cf700 -1 osd.4 pg_epoch: 497 pg[3.9s0( v 494'362 (0'0,494'362] local-lis/les=56/57 n=4 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [4,2,7]p4(0) r=0 lpr=56 crt=494'362 lcod 494'360 mlcod 494'360 active+clean+snaptrim trimq=[192~1] ps=[191~1,1b0~1]] removing snap head 2021-07-31T01:53:19.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:18 smithi154 conmon[34553]: cluster 2021-07-31T01:53:17.593370+0000 mon.a (mon.0) 1149 : 2021-07-31T01:53:19.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:18 smithi154 conmon[34553]: cluster [DBG] osdmap e496: 8 total, 8 up, 8 in 2021-07-31T01:53:19.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:18 smithi154 conmon[34553]: cluster 2021-07-31T01:53:17.877369+0000 mgr.x (mgr.24542) 256 : cluster [DBG] pgmap v525: 25 pgs: 25 active+clean; 219 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail 2021-07-31T01:53:19.004 INFO:journalctl@ceph.osd.7.smithi154.stdout:Jul 31 01:53:18 smithi154 conmon[49149]: debug 2021-07-31T01:53:19.004 INFO:journalctl@ceph.osd.7.smithi154.stdout:Jul 31 01:53:18 smithi154 conmon[49149]: 2021-07-31T01:53:18.613+0000 7f8a709fa700 -1 osd.7 pg_epoch: 497 pg[3.fs0( v 497'385 (0'0,497'385] local-lis/les=56/57 n=3 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [7,4,0]p7(0) r=0 lpr=56 luod=494'383 crt=494'383 lcod 494'381 mlcod 494'381 active+clean+snaptrim trimq=[192~1] ps=[191~1,1b0~1]] removing snap head 2021-07-31T01:53:19.005 INFO:journalctl@ceph.osd.7.smithi154.stdout:Jul 31 01:53:18 smithi154 conmon[49149]: 2021-07-31T01:53:19.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:18 smithi045 conmon[33655]: cluster 2021-07-31T01:53:17.593370+0000 mon.a ( 2021-07-31T01:53:19.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:18 smithi045 conmon[33655]: mon.0) 1149 : cluster [DBG] osdmap e496: 8 total, 8 up, 8 in 2021-07-31T01:53:19.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:18 smithi045 conmon[33655]: cluster 2021-07-31T01:53:17.877369+0000 mgr.x (mgr.24542) 256 : cluster [DBG] 2021-07-31T01:53:19.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:18 smithi045 conmon[33655]: pgmap v525: 25 pgs: 25 active+clean; 219 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail 2021-07-31T01:53:19.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:18 smithi045 conmon[39718]: cluster 2021-07-31T01:53:17.593370+0000 mon.a ( 2021-07-31T01:53:19.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:18 smithi045 conmon[39718]: mon.0) 1149 : cluster [DBG] osdmap e496: 8 total, 8 up, 8 in 2021-07-31T01:53:19.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:18 smithi045 conmon[39718]: cluster 2021-07- 2021-07-31T01:53:19.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:18 smithi045 conmon[39718]: 31T01:53:17.877369+0000 mgr.x (mgr.24542) 256 : cluster [DBG] pgmap v525: 25 pgs: 25 active+clean; 219 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail 2021-07-31T01:53:19.605 INFO:tasks.rados.rados.0.smithi045.stdout:2291: done (1 left) 2021-07-31T01:53:19.605 INFO:tasks.rados.rados.0.smithi045.stdout:2292: rollback oid 37 current snap is 219 2021-07-31T01:53:19.606 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 37 to 218 2021-07-31T01:53:19.606 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 14 v 1379 (ObjNum 659 snap 205 seq_num 659) dirty exists 2021-07-31T01:53:19.606 INFO:tasks.rados.rados.0.smithi045.stdout:2290: done (1 left) 2021-07-31T01:53:19.606 INFO:tasks.rados.rados.0.smithi045.stdout:2293: snap_remove snap 215 2021-07-31T01:53:20.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:19 smithi154 conmon[34553]: cluster 2021-07-31T01:53:18.599327+0000 mon.a (mon.0) 1150 : 2021-07-31T01:53:20.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:19 smithi154 conmon[34553]: cluster [DBG] osdmap e497: 8 total, 8 up, 8 in 2021-07-31T01:53:20.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:19 smithi045 conmon[39718]: cluster 2021-07-31T01:53:18.599327+0000 mon.a (mon. 2021-07-31T01:53:20.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:19 smithi045 conmon[39718]: 0) 1150 : cluster [DBG] osdmap e497: 8 total, 8 up, 8 in 2021-07-31T01:53:20.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:19 smithi045 conmon[33655]: cluster 2021-07-31T01:53:18.599327+0000 mon.a (mon.0) 1150 : cluster 2021-07-31T01:53:20.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:19 smithi045 conmon[33655]: [DBG] osdmap e497: 8 total, 8 up, 8 in 2021-07-31T01:53:20.612 INFO:tasks.rados.rados.0.smithi045.stdout:2293: done (1 left) 2021-07-31T01:53:20.612 INFO:tasks.rados.rados.0.smithi045.stdout:2294: copy_from oid 41 from oid 7 current snap is 219 2021-07-31T01:53:20.613 INFO:tasks.rados.rados.0.smithi045.stdout:2295: read oid 3 snap -1 2021-07-31T01:53:20.613 INFO:tasks.rados.rados.0.smithi045.stdout:2295: expect (ObjNum 647 snap 201 seq_num 647) 2021-07-31T01:53:20.613 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 42 current snap is 219 2021-07-31T01:53:20.613 INFO:tasks.rados.rados.0.smithi045.stdout:2296: seq_num 704 ranges {1441792=598016,2039808=745472,2785280=712704,3497984=147456} 2021-07-31T01:53:20.625 INFO:tasks.rados.rados.0.smithi045.stdout:2296: writing smithi04569269-42 from 1441792 to 2039808 tid 1 2021-07-31T01:53:20.629 INFO:tasks.rados.rados.0.smithi045.stdout:2296: writing smithi04569269-42 from 2039808 to 2785280 tid 2 2021-07-31T01:53:20.633 INFO:tasks.rados.rados.0.smithi045.stdout:2296: writing smithi04569269-42 from 2785280 to 3497984 tid 3 2021-07-31T01:53:20.635 INFO:tasks.rados.rados.0.smithi045.stdout:2296: writing smithi04569269-42 from 3497984 to 3645440 tid 4 2021-07-31T01:53:20.635 INFO:tasks.rados.rados.0.smithi045.stdout:2297: snap_create 2021-07-31T01:53:20.635 INFO:tasks.rados.rados.0.smithi045.stdout:2292: finishing rollback tid 0 to smithi04569269-37 2021-07-31T01:53:20.636 INFO:tasks.rados.rados.0.smithi045.stdout:2292: finishing rollback tid 1 to smithi04569269-37 2021-07-31T01:53:20.636 INFO:tasks.rados.rados.0.smithi045.stdout:2292: finishing rollback tid 2 to smithi04569269-37 2021-07-31T01:53:20.637 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 37 v 991 (ObjNum 639 snap 199 seq_num 639) dirty exists 2021-07-31T01:53:20.637 INFO:tasks.rados.rados.0.smithi045.stdout:2296: finishing write tid 1 to smithi04569269-42 2021-07-31T01:53:20.637 INFO:tasks.rados.rados.0.smithi045.stdout:2296: finishing write tid 2 to smithi04569269-42 2021-07-31T01:53:20.640 INFO:tasks.rados.rados.0.smithi045.stdout:2296: finishing write tid 3 to smithi04569269-42 2021-07-31T01:53:20.641 INFO:tasks.rados.rados.0.smithi045.stdout:2296: finishing write tid 4 to smithi04569269-42 2021-07-31T01:53:20.641 INFO:tasks.rados.rados.0.smithi045.stdout:2296: finishing write tid 5 to smithi04569269-42 2021-07-31T01:53:20.644 INFO:tasks.rados.rados.0.smithi045.stdout:2296: finishing write tid 6 to smithi04569269-42 2021-07-31T01:53:20.644 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 42 v 1170 (ObjNum 704 snap 219 seq_num 704) dirty exists 2021-07-31T01:53:20.644 INFO:tasks.rados.rados.0.smithi045.stdout:2296: left oid 42 (ObjNum 704 snap 219 seq_num 704) 2021-07-31T01:53:20.647 INFO:tasks.rados.rados.0.smithi045.stdout:2295: expect (ObjNum 647 snap 201 seq_num 647) 2021-07-31T01:53:21.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:20 smithi154 conmon[34553]: cluster 2021-07-31T01:53:21.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:20 smithi154 conmon[34553]: 2021-07-31T01:53:19.604352 2021-07-31T01:53:21.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:20 smithi154 conmon[34553]: +0000 mon.a (mon.0) 1151 : cluster [DBG] osdmap e498: 8 total, 8 up, 8 in 2021-07-31T01:53:21.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:20 smithi154 conmon[34553]: cluster 2021-07-31T01:53:21.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:20 smithi154 conmon[34553]: 2021-07-31T01:53:19.877819+0000 mgr.x (mgr.24542) 257 : cluster [DBG] pgmap v528: 25 pgs: 25 active+clean; 200 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.7 MiB/s rd, 2.7 MiB/s wr, 22 op/s 2021-07-31T01:53:21.004 INFO:journalctl@ceph.osd.7.smithi154.stdout:Jul 31 01:53:20 smithi154 conmon[49149]: debug 2021-07-31T01:53:21.004 INFO:journalctl@ceph.osd.7.smithi154.stdout:Jul 31 01:53:20 smithi154 conmon[49149]: 2021-07-31T01:53:20.620+0000 7f8a711fb700 -1 osd.7 pg_epoch: 499 pg[3.es0( v 498'943 (0'0,498'943] local-lis/les=56/57 n=6 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [7,4,1]p7(0) r=0 lpr=56 crt=498'943 lcod 498'941 mlcod 498'941 active+clean+snaptrim trimq=[1ab~1] ps=[191~2,1ac~1,1b0~1]] removing snap head 2021-07-31T01:53:21.005 INFO:journalctl@ceph.osd.7.smithi154.stdout:Jul 31 01:53:20 smithi154 conmon[49149]: 2021-07-31T01:53:21.005 INFO:journalctl@ceph.osd.7.smithi154.stdout:Jul 31 01:53:20 smithi154 conmon[49149]: debug 2021-07-31T01:53:20.621+0000 7f8a75a04700 -1 osd.7 pg_epoch: 499 pg[3.ds0( v 498'1045 (0'0,498'1045] local-lis/les=56/57 n=9 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [7,5,3]p7(0) r=0 lpr=56 crt=498'1045 lcod 498'1043 mlcod 498'1043 active+clean+snaptrim trimq=[1ab~1] ps=[191~2,1ac~1,1b0~1]] removing snap head 2021-07-31T01:53:21.015 INFO:tasks.rados.rados.0.smithi045.stdout:2294: finishing copy_from to smithi04569269-41 2021-07-31T01:53:21.015 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 41 v 986 (ObjNum 618 snap 189 seq_num 618) dirty exists 2021-07-31T01:53:21.015 INFO:tasks.rados.rados.0.smithi045.stdout:2294: finishing copy_from racing read to smithi04569269-41 2021-07-31T01:53:21.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:20 smithi045 conmon[33655]: cluster 2021-07-31T01:53:19.604352+0000 mon.a (mon.0) 1151 : cluster [DBG] osdmap e498: 8 total, 8 up, 8 in 2021-07-31T01:53:21.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:20 smithi045 conmon[33655]: cluster 2021-07-31T01:53:19.877819+0000 mgr.x (mgr.24542) 257 : cluster [DBG] pgmap v528: 25 pgs: 25 active+clean; 200 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.7 MiB/s rd, 2.7 MiB/s wr, 22 op/s 2021-07-31T01:53:21.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:20 smithi045 conmon[39718]: cluster 2021-07-31T01:53:19.604352+0000 2021-07-31T01:53:21.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:20 smithi045 conmon[39718]: mon.a (mon.0) 1151 : cluster [DBG] osdmap e498: 8 total, 8 up, 8 in 2021-07-31T01:53:21.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:20 smithi045 conmon[39718]: cluster 2021-07-31T01:53:21.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:20 smithi045 conmon[39718]: 2021-07-31T01:53:19.877819+0000 mgr.x (mgr.24542) 257 : cluster [DBG] pgmap v528: 25 pgs: 25 active+clean; 200 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.7 MiB/s rd, 2.7 MiB/s wr, 22 op/s 2021-07-31T01:53:21.589 INFO:tasks.rados.rados.0.smithi045.stdout:2292: done (4 left) 2021-07-31T01:53:21.590 INFO:tasks.rados.rados.0.smithi045.stdout:2294: done (3 left) 2021-07-31T01:53:21.590 INFO:tasks.rados.rados.0.smithi045.stdout:2295: done (2 left) 2021-07-31T01:53:21.590 INFO:tasks.rados.rados.0.smithi045.stdout:2296: done (1 left) 2021-07-31T01:53:21.590 INFO:tasks.rados.rados.0.smithi045.stdout:2297: done (0 left) 2021-07-31T01:53:21.590 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 48 current snap is 220 2021-07-31T01:53:21.591 INFO:tasks.rados.rados.0.smithi045.stdout:2298: seq_num 705 ranges {0=614400,614400=630784,1245184=524288,1769472=450560} 2021-07-31T01:53:21.592 INFO:tasks.rados.rados.0.smithi045.stdout:2298: writing smithi04569269-48 from 0 to 614400 tid 1 2021-07-31T01:53:21.596 INFO:tasks.rados.rados.0.smithi045.stdout:2298: writing smithi04569269-48 from 614400 to 1245184 tid 2 2021-07-31T01:53:21.599 INFO:tasks.rados.rados.0.smithi045.stdout:2298: writing smithi04569269-48 from 1245184 to 1769472 tid 3 2021-07-31T01:53:21.602 INFO:tasks.rados.rados.0.smithi045.stdout:2298: writing smithi04569269-48 from 1769472 to 2220032 tid 4 2021-07-31T01:53:21.602 INFO:tasks.rados.rados.0.smithi045.stdout:2299: delete oid 9 current snap is 220 2021-07-31T01:53:21.602 INFO:tasks.rados.rados.0.smithi045.stdout:2298: finishing write tid 1 to smithi04569269-48 2021-07-31T01:53:21.602 INFO:tasks.rados.rados.0.smithi045.stdout:2298: finishing write tid 2 to smithi04569269-48 2021-07-31T01:53:21.604 INFO:tasks.rados.rados.0.smithi045.stdout:2298: finishing write tid 3 to smithi04569269-48 2021-07-31T01:53:21.605 INFO:tasks.rados.rados.0.smithi045.stdout:2298: finishing write tid 4 to smithi04569269-48 2021-07-31T01:53:21.605 INFO:tasks.rados.rados.0.smithi045.stdout:2298: finishing write tid 5 to smithi04569269-48 2021-07-31T01:53:21.608 INFO:tasks.rados.rados.0.smithi045.stdout:2299: done (1 left) 2021-07-31T01:53:21.608 INFO:tasks.rados.rados.0.smithi045.stdout:2300: read oid 25 snap -1 2021-07-31T01:53:21.608 INFO:tasks.rados.rados.0.smithi045.stdout:2300: expect (ObjNum 678 snap 211 seq_num 678) 2021-07-31T01:53:21.608 INFO:tasks.rados.rados.0.smithi045.stdout:2301: read oid 18 snap -1 2021-07-31T01:53:21.609 INFO:tasks.rados.rados.0.smithi045.stdout:2301: expect deleted 2021-07-31T01:53:21.609 INFO:tasks.rados.rados.0.smithi045.stdout:2302: copy_from oid 45 from oid 1 current snap is 220 2021-07-31T01:53:21.609 INFO:tasks.rados.rados.0.smithi045.stdout:2298: finishing write tid 6 to smithi04569269-48 2021-07-31T01:53:21.609 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 48 v 1051 (ObjNum 705 snap 220 seq_num 705) dirty exists 2021-07-31T01:53:21.610 INFO:tasks.rados.rados.0.smithi045.stdout:2298: left oid 48 (ObjNum 705 snap 220 seq_num 705) 2021-07-31T01:53:21.610 INFO:tasks.rados.rados.0.smithi045.stdout:2298: done (3 left) 2021-07-31T01:53:21.610 INFO:tasks.rados.rados.0.smithi045.stdout:2303: read oid 8 snap -1 2021-07-31T01:53:21.610 INFO:tasks.rados.rados.0.smithi045.stdout:2303: expect (ObjNum 701 snap 218 seq_num 701) 2021-07-31T01:53:21.611 INFO:tasks.rados.rados.0.smithi045.stdout:2304: rollback oid 38 current snap is 220 2021-07-31T01:53:21.611 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 38 to 218 2021-07-31T01:53:21.611 INFO:tasks.rados.rados.0.smithi045.stdout:2305: snap_create 2021-07-31T01:53:21.614 INFO:tasks.rados.rados.0.smithi045.stdout:2304: finishing rollback tid 0 to smithi04569269-38 2021-07-31T01:53:21.614 INFO:tasks.rados.rados.0.smithi045.stdout:2304: finishing rollback tid 1 to smithi04569269-38 2021-07-31T01:53:21.615 INFO:tasks.rados.rados.0.smithi045.stdout:2304: finishing rollback tid 2 to smithi04569269-38 2021-07-31T01:53:21.615 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 38 v 1166 (ObjNum 700 snap 217 seq_num 700) dirty exists 2021-07-31T01:53:21.622 INFO:tasks.rados.rados.0.smithi045.stdout:2302: finishing copy_from to smithi04569269-45 2021-07-31T01:53:21.623 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 45 v 1001 (ObjNum 483 snap 143 seq_num 483) dirty exists 2021-07-31T01:53:21.623 INFO:tasks.rados.rados.0.smithi045.stdout:2302: finishing copy_from racing read to smithi04569269-45 2021-07-31T01:53:21.627 INFO:tasks.rados.rados.0.smithi045.stdout:2303: expect (ObjNum 701 snap 218 seq_num 701) 2021-07-31T01:53:21.759 INFO:tasks.rados.rados.0.smithi045.stdout:2300: expect (ObjNum 678 snap 211 seq_num 678) 2021-07-31T01:53:22.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:21 smithi154 conmon[34553]: cluster 2021-07-31T01:53:20.609858+0000 mon.a (mon.0) 1152 : cluster [DBG] 2021-07-31T01:53:22.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:21 smithi154 conmon[34553]: osdmap e499: 8 total, 8 up, 8 in 2021-07-31T01:53:22.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:21 smithi154 conmon[34553]: cluster 2021-07-31T01:53:21.587826+0000 mon.a (mon.0) 1153 : cluster [DBG] osdmap e500: 8 total, 8 up, 8 in 2021-07-31T01:53:22.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:21 smithi045 conmon[33655]: cluster 2021-07-31T01:53:20.609858+0000 mon.a (mon.0) 1152 : cluster [DBG] osdmap e499: 8 total, 8 up, 8 in 2021-07-31T01:53:22.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:21 smithi045 conmon[33655]: cluster 2021-07-31T01:53:21.587826+0000 mon.a (mon.0) 1153 : cluster [DBG] osdmap e500: 8 total, 8 up, 8 in 2021-07-31T01:53:22.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:21 smithi045 conmon[39718]: cluster 2021-07-31T01:53:20.609858+0000 2021-07-31T01:53:22.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:21 smithi045 conmon[39718]: mon.a (mon.0) 1152 : cluster [DBG] osdmap e499: 8 total, 8 up, 8 in 2021-07-31T01:53:22.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:21 smithi045 conmon[39718]: cluster 2021 2021-07-31T01:53:22.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:21 smithi045 conmon[39718]: -07-31T01:53:21.587826+0000 mon.a (mon.0) 1153 : cluster 2021-07-31T01:53:22.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:21 smithi045 conmon[39718]: [DBG] osdmap e500: 8 total, 8 up, 8 in 2021-07-31T01:53:22.617 INFO:tasks.rados.rados.0.smithi045.stdout:2300: done (5 left) 2021-07-31T01:53:22.617 INFO:tasks.rados.rados.0.smithi045.stdout:2301: done (4 left) 2021-07-31T01:53:22.617 INFO:tasks.rados.rados.0.smithi045.stdout:2302: done (3 left) 2021-07-31T01:53:22.618 INFO:tasks.rados.rados.0.smithi045.stdout:2303: done (2 left) 2021-07-31T01:53:22.618 INFO:tasks.rados.rados.0.smithi045.stdout:2304: done (1 left) 2021-07-31T01:53:22.618 INFO:tasks.rados.rados.0.smithi045.stdout:2305: done (0 left) 2021-07-31T01:53:22.618 INFO:tasks.rados.rados.0.smithi045.stdout:2306: snap_remove snap 218 2021-07-31T01:53:23.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:22 smithi154 conmon[34553]: cluster 2021-07-31T01:53:21.878389+0000 mgr.x (mgr. 2021-07-31T01:53:23.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:22 smithi154 conmon[34553]: 24542) 258 : cluster [DBG] pgmap v531: 25 pgs: 25 active+clean; 200 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.7 MiB/s rd, 2.7 MiB/s wr, 22 op/s 2021-07-31T01:53:23.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:22 smithi045 conmon[39718]: cluster 2021-07-31T01:53:21.878389+0000 mgr.x (mgr.24542) 2021-07-31T01:53:23.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:22 smithi045 conmon[39718]: 258 : cluster [DBG] pgmap v531: 25 pgs: 25 active+clean; 200 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.7 MiB/s rd, 2.7 MiB/s wr, 22 op/s 2021-07-31T01:53:23.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:22 smithi045 conmon[33655]: cluster 2021-07-31T01:53:21.878389+0000 mgr.x (mgr.24542 2021-07-31T01:53:23.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:22 smithi045 conmon[33655]: ) 258 : cluster [DBG] pgmap v531: 25 pgs: 25 active+clean; 200 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.7 MiB/s rd, 2.7 MiB/s wr, 22 op/s 2021-07-31T01:53:23.619 INFO:tasks.rados.rados.0.smithi045.stdout:2306: done (0 left) 2021-07-31T01:53:23.619 INFO:tasks.rados.rados.0.smithi045.stdout:2307: delete oid 47 current snap is 221 2021-07-31T01:53:23.627 INFO:tasks.rados.rados.0.smithi045.stdout:2307: done (0 left) 2021-07-31T01:53:23.627 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 27 current snap is 221 2021-07-31T01:53:23.627 INFO:tasks.rados.rados.0.smithi045.stdout:2308: seq_num 708 ranges {3014656=663552,3678208=737280,4415488=696320,5111808=172032} 2021-07-31T01:53:23.647 INFO:tasks.rados.rados.0.smithi045.stdout:2308: writing smithi04569269-27 from 3014656 to 3678208 tid 1 2021-07-31T01:53:23.651 INFO:tasks.rados.rados.0.smithi045.stdout:2308: writing smithi04569269-27 from 3678208 to 4415488 tid 2 2021-07-31T01:53:23.655 INFO:tasks.rados.rados.0.smithi045.stdout:2308: writing smithi04569269-27 from 4415488 to 5111808 tid 3 2021-07-31T01:53:23.656 INFO:tasks.rados.rados.0.smithi045.stdout:2308: writing smithi04569269-27 from 5111808 to 5283840 tid 4 2021-07-31T01:53:23.657 INFO:tasks.rados.rados.0.smithi045.stdout:2309: read oid 18 snap -1 2021-07-31T01:53:23.657 INFO:tasks.rados.rados.0.smithi045.stdout:2309: expect deleted 2021-07-31T01:53:23.657 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 43 current snap is 221 2021-07-31T01:53:23.657 INFO:tasks.rados.rados.0.smithi045.stdout:2308: finishing write tid 1 to smithi04569269-27 2021-07-31T01:53:23.657 INFO:tasks.rados.rados.0.smithi045.stdout:2310: seq_num 709 ranges {5627904=679936,6307840=614400,6922240=499712,7421952=491520} 2021-07-31T01:53:23.691 INFO:tasks.rados.rados.0.smithi045.stdout:2310: writing smithi04569269-43 from 5627904 to 6307840 tid 1 2021-07-31T01:53:23.694 INFO:tasks.rados.rados.0.smithi045.stdout:2310: writing smithi04569269-43 from 6307840 to 6922240 tid 2 2021-07-31T01:53:23.697 INFO:tasks.rados.rados.0.smithi045.stdout:2310: writing smithi04569269-43 from 6922240 to 7421952 tid 3 2021-07-31T01:53:23.700 INFO:tasks.rados.rados.0.smithi045.stdout:2310: writing smithi04569269-43 from 7421952 to 7913472 tid 4 2021-07-31T01:53:23.701 INFO:tasks.rados.rados.0.smithi045.stdout:2311: read oid 8 snap 219 2021-07-31T01:53:23.701 INFO:tasks.rados.rados.0.smithi045.stdout:2311: expect (ObjNum 701 snap 218 seq_num 701) 2021-07-31T01:53:23.701 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 13 current snap is 221 2021-07-31T01:53:23.702 INFO:tasks.rados.rados.0.smithi045.stdout:2308: finishing write tid 2 to smithi04569269-27 2021-07-31T01:53:23.702 INFO:tasks.rados.rados.0.smithi045.stdout:2312: seq_num 710 ranges {14336000=696320,15032320=499712,15532032=712704,16244736=393216} 2021-07-31T01:53:23.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:23 smithi045 conmon[39718]: cluster 2021-07-31T01:53:22.616525 2021-07-31T01:53:23.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:23 smithi045 conmon[39718]: +0000 mon.a (mon.0) 1154 : cluster [DBG] osdmap e501: 8 total, 8 up, 8 in 2021-07-31T01:53:23.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:53:23 smithi045 conmon[61210]: level=error ts=2021-07-31T01:53:23.519Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:53:23.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:53:23 smithi045 conmon[61210]: level=error ts=2021-07-31T01:53:23.519Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:53:23.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:53:23 smithi045 conmon[61210]: level=error ts=2021-07-31T01:53:23.519Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T01:53:23.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:23 smithi045 conmon[33655]: cluster 2021-07-31T01:53:23.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:23 smithi045 conmon[33655]: 2021-07-31T01:53:23.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:23 smithi045 conmon[33655]: 2021 2021-07-31T01:53:23.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:23 smithi045 conmon[33655]: -07-31T01:53:22.616525+0000 mon.a 2021-07-31T01:53:23.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:23 smithi045 conmon[33655]: (mon.0) 1154 : cluster [DBG] osdmap e501: 8 total, 8 up, 8 in 2021-07-31T01:53:23.783 INFO:tasks.rados.rados.0.smithi045.stdout:2312: writing smithi04569269-13 from 14336000 to 15032320 tid 1 2021-07-31T01:53:23.785 INFO:tasks.rados.rados.0.smithi045.stdout:2312: writing smithi04569269-13 from 15032320 to 15532032 tid 2 2021-07-31T01:53:23.789 INFO:tasks.rados.rados.0.smithi045.stdout:2312: writing smithi04569269-13 from 15532032 to 16244736 tid 3 2021-07-31T01:53:23.791 INFO:tasks.rados.rados.0.smithi045.stdout:2312: writing smithi04569269-13 from 16244736 to 16637952 tid 4 2021-07-31T01:53:23.791 INFO:tasks.rados.rados.0.smithi045.stdout:2313: snap_remove snap 220 2021-07-31T01:53:24.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:23 smithi154 conmon[34553]: cluster 2021-07-31T01:53:22.616525+0000 mon.a 2021-07-31T01:53:24.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:23 smithi154 conmon[34553]: (mon.0) 1154 : cluster [DBG] osdmap e501: 8 total, 8 up, 8 in 2021-07-31T01:53:24.634 INFO:tasks.rados.rados.0.smithi045.stdout:2313: done (5 left) 2021-07-31T01:53:24.634 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 34 current snap is 221 2021-07-31T01:53:24.634 INFO:tasks.rados.rados.0.smithi045.stdout:2314: seq_num 711 ranges {6938624=712704,7651328=786432,8437760=516096,8953856=303104} 2021-07-31T01:53:24.675 INFO:tasks.rados.rados.0.smithi045.stdout:2314: writing smithi04569269-34 from 6938624 to 7651328 tid 1 2021-07-31T01:53:24.680 INFO:tasks.rados.rados.0.smithi045.stdout:2314: writing smithi04569269-34 from 7651328 to 8437760 tid 2 2021-07-31T01:53:24.683 INFO:tasks.rados.rados.0.smithi045.stdout:2314: writing smithi04569269-34 from 8437760 to 8953856 tid 3 2021-07-31T01:53:24.686 INFO:tasks.rados.rados.0.smithi045.stdout:2314: writing smithi04569269-34 from 8953856 to 9256960 tid 4 2021-07-31T01:53:24.686 INFO:tasks.rados.rados.0.smithi045.stdout:2315: read oid 33 snap -1 2021-07-31T01:53:24.686 INFO:tasks.rados.rados.0.smithi045.stdout:2315: expect (ObjNum 659 snap 205 seq_num 659) 2021-07-31T01:53:24.688 INFO:tasks.rados.rados.0.smithi045.stdout:2316: read oid 39 snap 219 2021-07-31T01:53:24.688 INFO:tasks.rados.rados.0.smithi045.stdout:2316: expect (ObjNum 637 snap 198 seq_num 637) 2021-07-31T01:53:24.688 INFO:tasks.rados.rados.0.smithi045.stdout:2317: rmattr oid 28 current snap is 221 2021-07-31T01:53:24.688 INFO:tasks.rados.rados.0.smithi045.stdout:2317: done (8 left) 2021-07-31T01:53:24.689 INFO:tasks.rados.rados.0.smithi045.stdout:2318: copy_from oid 3 from oid 41 current snap is 221 2021-07-31T01:53:24.689 INFO:tasks.rados.rados.0.smithi045.stdout:2308: finishing write tid 3 to smithi04569269-27 2021-07-31T01:53:24.689 INFO:tasks.rados.rados.0.smithi045.stdout:2308: finishing write tid 4 to smithi04569269-27 2021-07-31T01:53:24.689 INFO:tasks.rados.rados.0.smithi045.stdout:2308: finishing write tid 5 to smithi04569269-27 2021-07-31T01:53:24.690 INFO:tasks.rados.rados.0.smithi045.stdout:2308: finishing write tid 6 to smithi04569269-27 2021-07-31T01:53:24.690 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 27 v 1384 (ObjNum 708 snap 221 seq_num 708) dirty exists 2021-07-31T01:53:24.690 INFO:tasks.rados.rados.0.smithi045.stdout:2308: left oid 27 (ObjNum 708 snap 221 seq_num 708) 2021-07-31T01:53:24.690 INFO:tasks.rados.rados.0.smithi045.stdout:2310: finishing write tid 1 to smithi04569269-43 2021-07-31T01:53:24.691 INFO:tasks.rados.rados.0.smithi045.stdout:2308: done (8 left) 2021-07-31T01:53:24.691 INFO:tasks.rados.rados.0.smithi045.stdout:2309: done (7 left) 2021-07-31T01:53:24.691 INFO:tasks.rados.rados.0.smithi045.stdout:2319: delete oid 20 current snap is 221 2021-07-31T01:53:24.692 INFO:tasks.rados.rados.0.smithi045.stdout:2310: finishing write tid 2 to smithi04569269-43 2021-07-31T01:53:24.692 INFO:tasks.rados.rados.0.smithi045.stdout:2310: finishing write tid 3 to smithi04569269-43 2021-07-31T01:53:24.693 INFO:tasks.rados.rados.0.smithi045.stdout:2310: finishing write tid 4 to smithi04569269-43 2021-07-31T01:53:24.693 INFO:tasks.rados.rados.0.smithi045.stdout:2310: finishing write tid 5 to smithi04569269-43 2021-07-31T01:53:24.693 INFO:tasks.rados.rados.0.smithi045.stdout:2310: finishing write tid 6 to smithi04569269-43 2021-07-31T01:53:24.693 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 43 v 1389 (ObjNum 709 snap 221 seq_num 709) dirty exists 2021-07-31T01:53:24.694 INFO:tasks.rados.rados.0.smithi045.stdout:2310: left oid 43 (ObjNum 709 snap 221 seq_num 709) 2021-07-31T01:53:24.694 INFO:tasks.rados.rados.0.smithi045.stdout:2311: expect (ObjNum 701 snap 218 seq_num 701) 2021-07-31T01:53:24.823 INFO:tasks.rados.rados.0.smithi045.stdout:2312: finishing write tid 1 to smithi04569269-13 2021-07-31T01:53:24.823 INFO:tasks.rados.rados.0.smithi045.stdout:2312: finishing write tid 2 to smithi04569269-13 2021-07-31T01:53:24.824 INFO:tasks.rados.rados.0.smithi045.stdout:2312: finishing write tid 3 to smithi04569269-13 2021-07-31T01:53:24.824 INFO:tasks.rados.rados.0.smithi045.stdout:2312: finishing write tid 4 to smithi04569269-13 2021-07-31T01:53:24.824 INFO:tasks.rados.rados.0.smithi045.stdout:2312: finishing write tid 5 to smithi04569269-13 2021-07-31T01:53:24.825 INFO:tasks.rados.rados.0.smithi045.stdout:2312: finishing write tid 6 to smithi04569269-13 2021-07-31T01:53:24.825 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 13 v 1385 (ObjNum 710 snap 221 seq_num 710) dirty exists 2021-07-31T01:53:24.825 INFO:tasks.rados.rados.0.smithi045.stdout:2312: left oid 13 (ObjNum 710 snap 221 seq_num 710) 2021-07-31T01:53:24.826 INFO:tasks.rados.rados.0.smithi045.stdout:2314: finishing write tid 1 to smithi04569269-34 2021-07-31T01:53:24.826 INFO:tasks.rados.rados.0.smithi045.stdout:2314: finishing write tid 2 to smithi04569269-34 2021-07-31T01:53:24.826 INFO:tasks.rados.rados.0.smithi045.stdout:2310: done (7 left) 2021-07-31T01:53:24.826 INFO:tasks.rados.rados.0.smithi045.stdout:2311: done (6 left) 2021-07-31T01:53:24.826 INFO:tasks.rados.rados.0.smithi045.stdout:2312: done (5 left) 2021-07-31T01:53:24.827 INFO:tasks.rados.rados.0.smithi045.stdout:2319: done (4 left) 2021-07-31T01:53:24.827 INFO:tasks.rados.rados.0.smithi045.stdout:2320: setattr oid 45 current snap is 221 2021-07-31T01:53:24.827 INFO:tasks.rados.rados.0.smithi045.stdout:2314: finishing write tid 3 to smithi04569269-34 2021-07-31T01:53:24.827 INFO:tasks.rados.rados.0.smithi045.stdout:2314: finishing write tid 4 to smithi04569269-34 2021-07-31T01:53:24.827 INFO:tasks.rados.rados.0.smithi045.stdout:2314: finishing write tid 5 to smithi04569269-34 2021-07-31T01:53:24.828 INFO:tasks.rados.rados.0.smithi045.stdout:2314: finishing write tid 6 to smithi04569269-34 2021-07-31T01:53:24.828 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 34 v 1056 (ObjNum 711 snap 221 seq_num 711) dirty exists 2021-07-31T01:53:24.828 INFO:tasks.rados.rados.0.smithi045.stdout:2314: left oid 34 (ObjNum 711 snap 221 seq_num 711) 2021-07-31T01:53:24.828 INFO:tasks.rados.rados.0.smithi045.stdout:2315: expect (ObjNum 659 snap 205 seq_num 659) 2021-07-31T01:53:25.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:24 smithi154 conmon[34553]: cluster 2021-07-31T01:53:23.618100+0000 mon.a (mon.0) 1155 : cluster [DBG] osdmap e502: 8 total, 8 up, 8 in 2021-07-31T01:53:25.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:24 smithi154 conmon[34553]: cluster 2021-07-31T01:53:23.879062+0000 mgr.x (mgr.24542) 259 : cluster [DBG] pgmap v534: 25 pgs: 25 active+clean; 198 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 4.6 MiB/s wr, 13 op/s 2021-07-31T01:53:25.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:24 smithi045 conmon[39718]: cluster 2021-07-31T01:53:23.618100+0000 2021-07-31T01:53:25.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:24 smithi045 conmon[39718]: mon.a (mon.0) 1155 : cluster [DBG] osdmap e502: 8 total, 8 up, 8 in 2021-07-31T01:53:25.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:24 smithi045 conmon[39718]: cluster 2021-07-31 2021-07-31T01:53:25.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:24 smithi045 conmon[39718]: T01:53:23.879062+0000 mgr.x (mgr.24542) 259 : cluster 2021-07-31T01:53:25.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:24 smithi045 conmon[39718]: [DBG] pgmap v534: 25 pgs: 25 active+clean; 198 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 4.6 MiB/s wr, 13 op/s 2021-07-31T01:53:25.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:24 smithi045 conmon[33655]: cluster 2021-07-31T01:53:23.618100+0000 mon.a (mon.0 2021-07-31T01:53:25.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:24 smithi045 conmon[33655]: ) 1155 : cluster [DBG] osdmap e502: 8 total, 8 up, 8 in 2021-07-31T01:53:25.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:24 smithi045 conmon[33655]: cluster 2021-07-31T01:53:23.879062+0000 mgr.x (mgr.24542) 259 : 2021-07-31T01:53:25.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:24 smithi045 conmon[33655]: cluster [DBG] pgmap v534: 25 pgs: 25 active+clean; 198 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 4.6 MiB/s wr, 13 op/s 2021-07-31T01:53:25.536 INFO:tasks.rados.rados.0.smithi045.stdout:2316: expect (ObjNum 637 snap 198 seq_num 637) 2021-07-31T01:53:25.563 INFO:tasks.rados.rados.0.smithi045.stdout:2318: finishing copy_from to smithi04569269-3 2021-07-31T01:53:25.563 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 3 v 986 (ObjNum 618 snap 189 seq_num 618) dirty exists 2021-07-31T01:53:25.564 INFO:tasks.rados.rados.0.smithi045.stdout:2318: finishing copy_from racing read to smithi04569269-3 2021-07-31T01:53:25.564 INFO:tasks.rados.rados.0.smithi045.stdout:2314: done (4 left) 2021-07-31T01:53:25.564 INFO:tasks.rados.rados.0.smithi045.stdout:2315: done (3 left) 2021-07-31T01:53:25.564 INFO:tasks.rados.rados.0.smithi045.stdout:2316: done (2 left) 2021-07-31T01:53:25.565 INFO:tasks.rados.rados.0.smithi045.stdout:2318: done (1 left) 2021-07-31T01:53:25.565 INFO:tasks.rados.rados.0.smithi045.stdout:2321: read oid 10 snap -1 2021-07-31T01:53:25.565 INFO:tasks.rados.rados.0.smithi045.stdout:2321: expect (ObjNum 523 snap 154 seq_num 523) 2021-07-31T01:53:25.565 INFO:tasks.rados.rados.0.smithi045.stdout:2322: setattr oid 35 current snap is 221 2021-07-31T01:53:25.566 INFO:tasks.rados.rados.0.smithi045.stdout:2323: rmattr oid 8 current snap is 221 2021-07-31T01:53:25.566 INFO:tasks.rados.rados.0.smithi045.stdout:2323: done (3 left) 2021-07-31T01:53:25.567 INFO:tasks.rados.rados.0.smithi045.stdout:2324: read oid 15 snap -1 2021-07-31T01:53:25.567 INFO:tasks.rados.rados.0.smithi045.stdout:2324: expect (ObjNum 677 snap 211 seq_num 677) 2021-07-31T01:53:25.567 INFO:tasks.rados.rados.0.smithi045.stdout:2325: snap_create 2021-07-31T01:53:25.568 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 35 v 1520 (ObjNum 593 snap 184 seq_num 593) dirty exists 2021-07-31T01:53:25.574 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 45 v 1171 (ObjNum 483 snap 143 seq_num 483) dirty exists 2021-07-31T01:53:25.585 INFO:tasks.rados.rados.0.smithi045.stdout:2321: expect (ObjNum 523 snap 154 seq_num 523) 2021-07-31T01:53:25.700 INFO:tasks.rados.rados.0.smithi045.stdout:2324: expect (ObjNum 677 snap 211 seq_num 677) 2021-07-31T01:53:26.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:25 smithi154 conmon[34553]: cluster 2021-07-31T01:53:24.633496+0000 mon.a (mon.0) 1156 : cluster [DBG] osdmap e503: 8 total, 8 up, 8 in 2021-07-31T01:53:26.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:25 smithi045 conmon[39718]: cluster 2021-07-31T01 2021-07-31T01:53:26.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:25 smithi045 conmon[39718]: :53:24.633496+0000 mon.a (mon.0) 1156 : cluster [DBG] osdmap e503: 8 total, 8 up, 8 in 2021-07-31T01:53:26.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:25 smithi045 conmon[33655]: cluster 2021-07-31T01:53:24.633496 2021-07-31T01:53:26.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:25 smithi045 conmon[33655]: +0000 mon.a (mon.0) 1156 : cluster [DBG] osdmap e503: 8 total, 8 up, 8 in 2021-07-31T01:53:26.218 INFO:tasks.rados.rados.0.smithi045.stdout:2320: done (4 left) 2021-07-31T01:53:26.219 INFO:tasks.rados.rados.0.smithi045.stdout:2321: done (3 left) 2021-07-31T01:53:26.219 INFO:tasks.rados.rados.0.smithi045.stdout:2322: done (2 left) 2021-07-31T01:53:26.219 INFO:tasks.rados.rados.0.smithi045.stdout:2324: done (1 left) 2021-07-31T01:53:26.219 INFO:tasks.rados.rados.0.smithi045.stdout:2325: done (0 left) 2021-07-31T01:53:26.219 INFO:tasks.rados.rados.0.smithi045.stdout:2326: snap_remove snap 221 2021-07-31T01:53:26.590 INFO:tasks.rados.rados.0.smithi045.stdout:2326: done (0 left) 2021-07-31T01:53:26.590 INFO:tasks.rados.rados.0.smithi045.stdout:2327: rmattr oid 26 current snap is 222 2021-07-31T01:53:26.591 INFO:tasks.rados.rados.0.smithi045.stdout:2327: done (0 left) 2021-07-31T01:53:26.591 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 1 current snap is 222 2021-07-31T01:53:26.591 INFO:tasks.rados.rados.0.smithi045.stdout:2328: seq_num 713 ranges {1441792=745472,2187264=557056,2744320=483328,3227648=565248} 2021-07-31T01:53:26.602 INFO:tasks.rados.rados.0.smithi045.stdout:2328: writing smithi04569269-1 from 1441792 to 2187264 tid 1 2021-07-31T01:53:26.605 INFO:tasks.rados.rados.0.smithi045.stdout:2328: writing smithi04569269-1 from 2187264 to 2744320 tid 2 2021-07-31T01:53:26.608 INFO:tasks.rados.rados.0.smithi045.stdout:2328: writing smithi04569269-1 from 2744320 to 3227648 tid 3 2021-07-31T01:53:26.612 INFO:tasks.rados.rados.0.smithi045.stdout:2328: writing smithi04569269-1 from 3227648 to 3792896 tid 4 2021-07-31T01:53:26.612 INFO:tasks.rados.rados.0.smithi045.stdout:2329: copy_from oid 15 from oid 9 current snap is 222 2021-07-31T01:53:26.613 INFO:tasks.rados.rados.0.smithi045.stdout:2328: finishing write tid 1 to smithi04569269-1 2021-07-31T01:53:26.613 INFO:tasks.rados.rados.0.smithi045.stdout:2328: finishing write tid 2 to smithi04569269-1 2021-07-31T01:53:26.613 INFO:tasks.rados.rados.0.smithi045.stdout:2328: finishing write tid 3 to smithi04569269-1 2021-07-31T01:53:26.613 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 41 current snap is 222 2021-07-31T01:53:26.614 INFO:tasks.rados.rados.0.smithi045.stdout:2330: seq_num 714 ranges {6422528=761856,7184384=442368,7626752=688128,8314880=475136} 2021-07-31T01:53:26.653 INFO:tasks.rados.rados.0.smithi045.stdout:2330: writing smithi04569269-41 from 6422528 to 7184384 tid 1 2021-07-31T01:53:26.654 INFO:tasks.rados.rados.0.smithi045.stdout:2330: writing smithi04569269-41 from 7184384 to 7626752 tid 2 2021-07-31T01:53:26.662 INFO:tasks.rados.rados.0.smithi045.stdout:2330: writing smithi04569269-41 from 7626752 to 8314880 tid 3 2021-07-31T01:53:26.663 INFO:tasks.rados.rados.0.smithi045.stdout:2330: writing smithi04569269-41 from 8314880 to 8790016 tid 4 2021-07-31T01:53:26.663 INFO:tasks.rados.rados.0.smithi045.stdout:2331: delete oid 38 current snap is 222 2021-07-31T01:53:26.664 INFO:tasks.rados.rados.0.smithi045.stdout:2328: finishing write tid 4 to smithi04569269-1 2021-07-31T01:53:26.664 INFO:tasks.rados.rados.0.smithi045.stdout:2328: finishing write tid 5 to smithi04569269-1 2021-07-31T01:53:26.664 INFO:tasks.rados.rados.0.smithi045.stdout:2329: finishing copy_from to smithi04569269-15 2021-07-31T01:53:26.664 INFO:tasks.rados.rados.0.smithi045.stdout:2329: got expected ENOENT (src dne) 2021-07-31T01:53:26.665 INFO:tasks.rados.rados.0.smithi045.stdout:2329: finishing copy_from racing read to smithi04569269-15 2021-07-31T01:53:26.665 INFO:tasks.rados.rados.0.smithi045.stdout:2328: finishing write tid 6 to smithi04569269-1 2021-07-31T01:53:26.665 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 1 v 1061 (ObjNum 713 snap 222 seq_num 713) dirty exists 2021-07-31T01:53:26.665 INFO:tasks.rados.rados.0.smithi045.stdout:2328: left oid 1 (ObjNum 713 snap 222 seq_num 713) 2021-07-31T01:53:26.665 INFO:tasks.rados.rados.0.smithi045.stdout:2330: finishing write tid 1 to smithi04569269-41 2021-07-31T01:53:26.666 INFO:tasks.rados.rados.0.smithi045.stdout:2330: finishing write tid 2 to smithi04569269-41 2021-07-31T01:53:26.667 INFO:tasks.rados.rados.0.smithi045.stdout:2330: finishing write tid 3 to smithi04569269-41 2021-07-31T01:53:26.667 INFO:tasks.rados.rados.0.smithi045.stdout:2330: finishing write tid 4 to smithi04569269-41 2021-07-31T01:53:26.668 INFO:tasks.rados.rados.0.smithi045.stdout:2330: finishing write tid 5 to smithi04569269-41 2021-07-31T01:53:26.668 INFO:tasks.rados.rados.0.smithi045.stdout:2328: done (3 left) 2021-07-31T01:53:26.668 INFO:tasks.rados.rados.0.smithi045.stdout:2329: done (2 left) 2021-07-31T01:53:26.668 INFO:tasks.rados.rados.0.smithi045.stdout:2331: done (1 left) 2021-07-31T01:53:26.668 INFO:tasks.rados.rados.0.smithi045.stdout:2332: rollback oid 42 current snap is 222 2021-07-31T01:53:26.669 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 42 to 219 2021-07-31T01:53:26.669 INFO:tasks.rados.rados.0.smithi045.stdout:2333: rollback oid 35 current snap is 222 2021-07-31T01:53:26.669 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 35 to 219 2021-07-31T01:53:26.670 INFO:tasks.rados.rados.0.smithi045.stdout:2334: rollback oid 19 current snap is 222 2021-07-31T01:53:26.670 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 19 to 219 2021-07-31T01:53:26.670 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 4 current snap is 222 2021-07-31T01:53:26.671 INFO:tasks.rados.rados.0.smithi045.stdout:2335: seq_num 716 ranges {0=794624,794624=655360,1449984=704512,2154496=245760} 2021-07-31T01:53:26.677 INFO:tasks.rados.rados.0.smithi045.stdout:2335: writing smithi04569269-4 from 0 to 794624 tid 1 2021-07-31T01:53:26.678 INFO:tasks.rados.rados.0.smithi045.stdout:2335: writing smithi04569269-4 from 794624 to 1449984 tid 2 2021-07-31T01:53:26.682 INFO:tasks.rados.rados.0.smithi045.stdout:2335: writing smithi04569269-4 from 1449984 to 2154496 tid 3 2021-07-31T01:53:26.682 INFO:tasks.rados.rados.0.smithi045.stdout:2335: writing smithi04569269-4 from 2154496 to 2400256 tid 4 2021-07-31T01:53:26.683 INFO:tasks.rados.rados.0.smithi045.stdout:2336: snap_create 2021-07-31T01:53:26.683 INFO:tasks.rados.rados.0.smithi045.stdout:2330: finishing write tid 6 to smithi04569269-41 2021-07-31T01:53:26.683 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 41 v 992 (ObjNum 714 snap 222 seq_num 714) dirty exists 2021-07-31T01:53:26.684 INFO:tasks.rados.rados.0.smithi045.stdout:2330: left oid 41 (ObjNum 714 snap 222 seq_num 714) 2021-07-31T01:53:26.684 INFO:tasks.rados.rados.0.smithi045.stdout:2332: finishing rollback tid 0 to smithi04569269-42 2021-07-31T01:53:26.684 INFO:tasks.rados.rados.0.smithi045.stdout:2334: finishing rollback tid 0 to smithi04569269-19 2021-07-31T01:53:26.684 INFO:tasks.rados.rados.0.smithi045.stdout:2332: finishing rollback tid 1 to smithi04569269-42 2021-07-31T01:53:26.685 INFO:tasks.rados.rados.0.smithi045.stdout:2332: finishing rollback tid 2 to smithi04569269-42 2021-07-31T01:53:26.685 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 42 v 1174 (ObjNum 704 snap 219 seq_num 704) dirty exists 2021-07-31T01:53:26.687 INFO:tasks.rados.rados.0.smithi045.stdout:2334: finishing rollback tid 1 to smithi04569269-19 2021-07-31T01:53:26.687 INFO:tasks.rados.rados.0.smithi045.stdout:2333: finishing rollback tid 0 to smithi04569269-35 2021-07-31T01:53:26.687 INFO:tasks.rados.rados.0.smithi045.stdout:2333: finishing rollback tid 1 to smithi04569269-35 2021-07-31T01:53:26.688 INFO:tasks.rados.rados.0.smithi045.stdout:2334: finishing rollback tid 2 to smithi04569269-19 2021-07-31T01:53:26.688 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 19 v 1150 (ObjNum 172 snap 41 seq_num 172) dirty exists 2021-07-31T01:53:26.688 INFO:tasks.rados.rados.0.smithi045.stdout:2333: finishing rollback tid 2 to smithi04569269-35 2021-07-31T01:53:26.688 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 35 v 1523 (ObjNum 593 snap 184 seq_num 593) dirty exists 2021-07-31T01:53:26.688 INFO:tasks.rados.rados.0.smithi045.stdout:2335: finishing write tid 1 to smithi04569269-4 2021-07-31T01:53:26.689 INFO:tasks.rados.rados.0.smithi045.stdout:2335: finishing write tid 2 to smithi04569269-4 2021-07-31T01:53:26.690 INFO:tasks.rados.rados.0.smithi045.stdout:2335: finishing write tid 3 to smithi04569269-4 2021-07-31T01:53:26.690 INFO:tasks.rados.rados.0.smithi045.stdout:2335: finishing write tid 4 to smithi04569269-4 2021-07-31T01:53:26.690 INFO:tasks.rados.rados.0.smithi045.stdout:2335: finishing write tid 5 to smithi04569269-4 2021-07-31T01:53:26.691 INFO:tasks.rados.rados.0.smithi045.stdout:2335: finishing write tid 6 to smithi04569269-4 2021-07-31T01:53:26.691 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 4 v 996 (ObjNum 716 snap 222 seq_num 716) dirty exists 2021-07-31T01:53:26.692 INFO:tasks.rados.rados.0.smithi045.stdout:2335: left oid 4 (ObjNum 716 snap 222 seq_num 716) 2021-07-31T01:53:27.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:26 smithi154 conmon[34553]: cluster 2021-07-31T01:53:25.633030+0000 mon.a (mon.0) 1157 : cluster [DBG] osdmap e504: 8 total, 8 up, 8 in 2021-07-31T01:53:27.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:26 smithi154 conmon[34553]: cluster 2021-07-31T01:53:25.879515+0000 mgr.x (mgr.24542) 260 : cluster [DBG] pgmap v537: 25 pgs: 25 active+clean; 198 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 4.6 MiB/s wr, 13 op/s 2021-07-31T01:53:27.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:26 smithi154 conmon[34553]: cluster 2021-07-31T01:53:26.589576+0000 mon.a (mon.0) 1158 : cluster [DBG] osdmap e505: 8 total, 8 up, 8 in 2021-07-31T01:53:27.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:26 smithi045 conmon[39718]: cluster 2021-07-31T01:53: 2021-07-31T01:53:27.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:26 smithi045 conmon[39718]: 25.633030+0000 mon.a (mon.0) 1157 : cluster [DBG] osdmap e504: 8 total, 8 up, 8 in 2021-07-31T01:53:27.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:26 smithi045 conmon[39718]: cluster 2021 2021-07-31T01:53:27.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:26 smithi045 conmon[39718]: -07-31T01:53: 2021-07-31T01:53:27.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:26 smithi045 conmon[39718]: 25.879515+0000 mgr.x (mgr. 2021-07-31T01:53:27.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:26 smithi045 conmon[39718]: 24542) 260 : cluster [DBG] 2021-07-31T01:53:27.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:26 smithi045 conmon[39718]: pgmap v537: 25 pgs: 25 active+clean; 198 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 4.6 MiB/s wr, 13 op/s 2021-07-31T01:53:27.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:26 smithi045 conmon[39718]: cluster 2021-07 2021-07-31T01:53:27.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:26 smithi045 conmon[39718]: -31T01:53:26. 2021-07-31T01:53:27.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:26 smithi045 conmon[39718]: 589576+0000 mon.a (mon.0 2021-07-31T01:53:27.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:26 smithi045 conmon[39718]: ) 1158 : cluster [DBG] osdmap e505: 8 total, 8 up, 8 in 2021-07-31T01:53:27.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:26 smithi045 conmon[33655]: cluster 2021-07-31T01:53: 2021-07-31T01:53:27.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:26 smithi045 conmon[33655]: 25.633030+0000 mon.a (mon.0) 1157 : cluster [DBG] osdmap e504: 8 total, 8 up, 8 in 2021-07-31T01:53:27.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:26 smithi045 conmon[33655]: cluster 2021- 2021-07-31T01:53:27.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:26 smithi045 conmon[33655]: 07-31T01:53:25.879515+0000 mgr.x ( 2021-07-31T01:53:27.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:26 smithi045 conmon[33655]: mgr.24542) 260 : cluster [DBG] pgmap v537: 25 pgs: 25 active+clean; 198 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 4.6 MiB/s wr, 13 op/s 2021-07-31T01:53:27.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:26 smithi045 conmon[33655]: cluster 2021-07-31 2021-07-31T01:53:27.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:26 smithi045 conmon[33655]: T01:53:26.589576+0000 mon.a (mon.0) 1158 2021-07-31T01:53:27.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:26 smithi045 conmon[33655]: : cluster [DBG] osdmap e505: 8 total, 8 up, 8 in 2021-07-31T01:53:27.646 INFO:tasks.rados.rados.0.smithi045.stdout:2330: done (5 left) 2021-07-31T01:53:27.646 INFO:tasks.rados.rados.0.smithi045.stdout:2332: done (4 left) 2021-07-31T01:53:27.646 INFO:tasks.rados.rados.0.smithi045.stdout:2333: done (3 left) 2021-07-31T01:53:27.646 INFO:tasks.rados.rados.0.smithi045.stdout:2334: done (2 left) 2021-07-31T01:53:27.647 INFO:tasks.rados.rados.0.smithi045.stdout:2335: done (1 left) 2021-07-31T01:53:27.647 INFO:tasks.rados.rados.0.smithi045.stdout:2336: done (0 left) 2021-07-31T01:53:27.647 INFO:tasks.rados.rados.0.smithi045.stdout:2337: read oid 19 snap -1 2021-07-31T01:53:27.648 INFO:tasks.rados.rados.0.smithi045.stdout:2337: expect (ObjNum 172 snap 41 seq_num 172) 2021-07-31T01:53:27.648 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 20 current snap is 223 2021-07-31T01:53:27.648 INFO:tasks.rados.rados.0.smithi045.stdout:2338: seq_num 717 ranges {0=16384} 2021-07-31T01:53:27.648 INFO:tasks.rados.rados.0.smithi045.stdout:2338: writing smithi04569269-20 from 0 to 16384 tid 1 2021-07-31T01:53:27.649 INFO:tasks.rados.rados.0.smithi045.stdout:2339: delete oid 3 current snap is 223 2021-07-31T01:53:27.650 INFO:tasks.rados.rados.0.smithi045.stdout:2337: expect (ObjNum 172 snap 41 seq_num 172) 2021-07-31T01:53:27.671 INFO:tasks.rados.rados.0.smithi045.stdout:2338: finishing write tid 1 to smithi04569269-20 2021-07-31T01:53:27.671 INFO:tasks.rados.rados.0.smithi045.stdout:2338: finishing write tid 2 to smithi04569269-20 2021-07-31T01:53:27.672 INFO:tasks.rados.rados.0.smithi045.stdout:2338: finishing write tid 3 to smithi04569269-20 2021-07-31T01:53:27.672 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 20 v 1525 (ObjNum 717 snap 223 seq_num 717) dirty exists 2021-07-31T01:53:27.672 INFO:tasks.rados.rados.0.smithi045.stdout:2338: left oid 20 (ObjNum 717 snap 223 seq_num 717) 2021-07-31T01:53:27.672 INFO:tasks.rados.rados.0.smithi045.stdout:2337: done (2 left) 2021-07-31T01:53:27.673 INFO:tasks.rados.rados.0.smithi045.stdout:2338: done (1 left) 2021-07-31T01:53:27.673 INFO:tasks.rados.rados.0.smithi045.stdout:2339: done (0 left) 2021-07-31T01:53:27.673 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 30 current snap is 223 2021-07-31T01:53:27.673 INFO:tasks.rados.rados.0.smithi045.stdout:2340: seq_num 719 ranges {2170880=49152} 2021-07-31T01:53:27.683 INFO:tasks.rados.rados.0.smithi045.stdout:2340: writing smithi04569269-30 from 2170880 to 2220032 tid 1 2021-07-31T01:53:27.683 INFO:tasks.rados.rados.0.smithi045.stdout:2341: delete oid 35 current snap is 223 2021-07-31T01:53:27.685 INFO:tasks.rados.rados.0.smithi045.stdout:2341: done (1 left) 2021-07-31T01:53:27.686 INFO:tasks.rados.rados.0.smithi045.stdout:2342: rollback oid 39 current snap is 223 2021-07-31T01:53:27.686 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 39 to 219 2021-07-31T01:53:27.686 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 47 current snap is 223 2021-07-31T01:53:27.686 INFO:tasks.rados.rados.0.smithi045.stdout:2343: seq_num 721 ranges {0=81920} 2021-07-31T01:53:27.688 INFO:tasks.rados.rados.0.smithi045.stdout:2343: writing smithi04569269-47 from 0 to 81920 tid 1 2021-07-31T01:53:27.688 INFO:tasks.rados.rados.0.smithi045.stdout:2344: read oid 7 snap -1 2021-07-31T01:53:27.688 INFO:tasks.rados.rados.0.smithi045.stdout:2344: expect (ObjNum 618 snap 189 seq_num 618) 2021-07-31T01:53:27.689 INFO:tasks.rados.rados.0.smithi045.stdout:2345: snap_create 2021-07-31T01:53:27.689 INFO:tasks.rados.rados.0.smithi045.stdout:2340: finishing write tid 1 to smithi04569269-30 2021-07-31T01:53:27.689 INFO:tasks.rados.rados.0.smithi045.stdout:2340: finishing write tid 2 to smithi04569269-30 2021-07-31T01:53:27.690 INFO:tasks.rados.rados.0.smithi045.stdout:2340: finishing write tid 3 to smithi04569269-30 2021-07-31T01:53:27.690 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 30 v 560 (ObjNum 719 snap 223 seq_num 719) dirty exists 2021-07-31T01:53:27.690 INFO:tasks.rados.rados.0.smithi045.stdout:2340: left oid 30 (ObjNum 719 snap 223 seq_num 719) 2021-07-31T01:53:27.691 INFO:tasks.rados.rados.0.smithi045.stdout:2343: finishing write tid 1 to smithi04569269-47 2021-07-31T01:53:27.691 INFO:tasks.rados.rados.0.smithi045.stdout:2343: finishing write tid 2 to smithi04569269-47 2021-07-31T01:53:27.691 INFO:tasks.rados.rados.0.smithi045.stdout:2343: finishing write tid 3 to smithi04569269-47 2021-07-31T01:53:27.692 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 47 v 330 (ObjNum 721 snap 223 seq_num 721) dirty exists 2021-07-31T01:53:27.692 INFO:tasks.rados.rados.0.smithi045.stdout:2343: left oid 47 (ObjNum 721 snap 223 seq_num 721) 2021-07-31T01:53:27.692 INFO:tasks.rados.rados.0.smithi045.stdout:2342: finishing rollback tid 0 to smithi04569269-39 2021-07-31T01:53:27.692 INFO:tasks.rados.rados.0.smithi045.stdout:2342: finishing rollback tid 1 to smithi04569269-39 2021-07-31T01:53:27.693 INFO:tasks.rados.rados.0.smithi045.stdout:2342: finishing rollback tid 2 to smithi04569269-39 2021-07-31T01:53:27.693 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 39 v 1170 (ObjNum 637 snap 198 seq_num 637) dirty exists 2021-07-31T01:53:27.718 INFO:tasks.rados.rados.0.smithi045.stdout:2344: expect (ObjNum 618 snap 189 seq_num 618) 2021-07-31T01:53:28.652 INFO:tasks.rados.rados.0.smithi045.stdout:2340: done (4 left) 2021-07-31T01:53:28.653 INFO:tasks.rados.rados.0.smithi045.stdout:2342: done (3 left) 2021-07-31T01:53:28.653 INFO:tasks.rados.rados.0.smithi045.stdout:2343: done (2 left) 2021-07-31T01:53:28.654 INFO:tasks.rados.rados.0.smithi045.stdout:2344: done (1 left) 2021-07-31T01:53:28.654 INFO:tasks.rados.rados.0.smithi045.stdout:2345: done (0 left) 2021-07-31T01:53:28.654 INFO:tasks.rados.rados.0.smithi045.stdout:2346: rmattr oid 16 current snap is 224 2021-07-31T01:53:28.654 INFO:tasks.rados.rados.0.smithi045.stdout:2346: done (0 left) 2021-07-31T01:53:28.654 INFO:tasks.rados.rados.0.smithi045.stdout:2347: rollback oid 26 current snap is 224 2021-07-31T01:53:28.655 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 26 to 219 2021-07-31T01:53:28.655 INFO:tasks.rados.rados.0.smithi045.stdout:2348: rollback oid 36 current snap is 224 2021-07-31T01:53:28.655 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 36 to 222 2021-07-31T01:53:28.655 INFO:tasks.rados.rados.0.smithi045.stdout:2349: rollback oid 20 current snap is 224 2021-07-31T01:53:28.656 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 20 to 219 2021-07-31T01:53:28.656 INFO:tasks.rados.rados.0.smithi045.stdout:2350: read oid 46 snap -1 2021-07-31T01:53:28.656 INFO:tasks.rados.rados.0.smithi045.stdout:2350: expect (ObjNum 665 snap 208 seq_num 665) 2021-07-31T01:53:28.656 INFO:tasks.rados.rados.0.smithi045.stdout:2351: rollback oid 25 current snap is 224 2021-07-31T01:53:28.656 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 25 to 219 2021-07-31T01:53:28.657 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 10 current snap is 224 2021-07-31T01:53:28.657 INFO:tasks.rados.rados.0.smithi045.stdout:2352: seq_num 722 ranges {3014656=98304} 2021-07-31T01:53:28.671 INFO:tasks.rados.rados.0.smithi045.stdout:2352: writing smithi04569269-10 from 3014656 to 3112960 tid 1 2021-07-31T01:53:28.671 INFO:tasks.rados.rados.0.smithi045.stdout:2353: read oid 34 snap -1 2021-07-31T01:53:28.672 INFO:tasks.rados.rados.0.smithi045.stdout:2353: expect (ObjNum 711 snap 221 seq_num 711) 2021-07-31T01:53:28.672 INFO:tasks.rados.rados.0.smithi045.stdout:2354: snap_create 2021-07-31T01:53:28.672 INFO:tasks.rados.rados.0.smithi045.stdout:2347: finishing rollback tid 1 to smithi04569269-26 2021-07-31T01:53:28.672 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 26 v 0 (ObjNum -1126017128 snap 21873 seq_num 1) dirty dne 2021-07-31T01:53:28.673 INFO:tasks.rados.rados.0.smithi045.stdout:2351: finishing rollback tid 0 to smithi04569269-25 2021-07-31T01:53:28.673 INFO:tasks.rados.rados.0.smithi045.stdout:2351: finishing rollback tid 1 to smithi04569269-25 2021-07-31T01:53:28.673 INFO:tasks.rados.rados.0.smithi045.stdout:2351: finishing rollback tid 2 to smithi04569269-25 2021-07-31T01:53:28.673 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 25 v 334 (ObjNum 678 snap 211 seq_num 678) dirty exists 2021-07-31T01:53:28.673 INFO:tasks.rados.rados.0.smithi045.stdout:2348: finishing rollback tid 0 to smithi04569269-36 2021-07-31T01:53:28.674 INFO:tasks.rados.rados.0.smithi045.stdout:2348: finishing rollback tid 1 to smithi04569269-36 2021-07-31T01:53:28.674 INFO:tasks.rados.rados.0.smithi045.stdout:2348: finishing rollback tid 2 to smithi04569269-36 2021-07-31T01:53:28.674 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 36 v 1533 (ObjNum 483 snap 143 seq_num 483) dirty exists 2021-07-31T01:53:28.675 INFO:tasks.rados.rados.0.smithi045.stdout:2349: finishing rollback tid 0 to smithi04569269-20 2021-07-31T01:53:28.675 INFO:tasks.rados.rados.0.smithi045.stdout:2349: finishing rollback tid 1 to smithi04569269-20 2021-07-31T01:53:28.675 INFO:tasks.rados.rados.0.smithi045.stdout:2349: finishing rollback tid 2 to smithi04569269-20 2021-07-31T01:53:28.675 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 20 v 1537 (ObjNum 556 snap 166 seq_num 556) dirty exists 2021-07-31T01:53:28.676 INFO:tasks.rados.rados.0.smithi045.stdout:2350: expect (ObjNum 665 snap 208 seq_num 665) 2021-07-31T01:53:28.897 INFO:tasks.rados.rados.0.smithi045.stdout:2352: finishing write tid 1 to smithi04569269-10 2021-07-31T01:53:28.897 INFO:tasks.rados.rados.0.smithi045.stdout:2352: finishing write tid 2 to smithi04569269-10 2021-07-31T01:53:28.897 INFO:tasks.rados.rados.0.smithi045.stdout:2352: finishing write tid 3 to smithi04569269-10 2021-07-31T01:53:28.897 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 10 v 1176 (ObjNum 722 snap 224 seq_num 722) dirty exists 2021-07-31T01:53:28.898 INFO:tasks.rados.rados.0.smithi045.stdout:2352: left oid 10 (ObjNum 722 snap 224 seq_num 722) 2021-07-31T01:53:28.898 INFO:tasks.rados.rados.0.smithi045.stdout:2353: expect (ObjNum 711 snap 221 seq_num 711) 2021-07-31T01:53:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[34553]: cluster 2021-07-31T01:53:27.644571+0000 2021-07-31T01:53:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[34553]: mon.a (mon.0) 1159 : cluster [DBG] osdmap e506: 8 total, 8 up, 8 in 2021-07-31T01:53:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[34553]: cluster 2021-07-31T01:53:27.880010+0000 mgr.x (mgr.24542) 261 : cluster [DBG] pgmap v540: 25 pgs: 25 active+clean; 198 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail 2021-07-31T01:53:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[34553]: audit 2021-07-31T01:53:29.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[34553]: 2021-07-31T01:53:27.962169+0000 mon.b (mon.2) 92 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:53:29.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[34553]: audit 2021-07-31T01:53: 2021-07-31T01:53:29.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[34553]: 27.962526+0000 mon.a (mon.0) 1160 : audit [INF] 2021-07-31T01:53:29.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[34553]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:53:29.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[34553]: audit 2021-07-31T01:53:29.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[34553]: 2021-07-31T01:53: 2021-07-31T01:53:29.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[34553]: 27.975661+0000 mon.b (mon.2) 93 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T01:53:29.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[34553]: audit 2021-07- 2021-07-31T01:53:29.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[34553]: 31T01:53:27.975938+0000 mon.a (mon.0) 1161 : audit 2021-07-31T01:53:29.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[34553]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T01:53:29.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: debug 2021-07-31T01:53:28.831+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T01:53:29.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:53:29.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:53:29.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: yield 2021-07-31T01:53:29.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:53:29.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:53:29.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:53:29.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T01:53:29.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:53:29.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T01:53:29.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:53:29.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: await self._waiter 2021-07-31T01:53:29.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:53:29.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: task.result() 2021-07-31T01:53:29.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T01:53:29.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: debug 2021-07-31T01:53:28.843+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T01:53:29.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:53:29.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:53:29.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: yield 2021-07-31T01:53:29.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:53:29.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:53:29.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:53:29.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T01:53:29.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:53:29.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T01:53:29.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:53:29.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: await self._waiter 2021-07-31T01:53:29.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:53:29.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: task.result() 2021-07-31T01:53:29.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T01:53:29.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: debug 2021-07-31T01:53:28.868+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T01:53:29.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:53:29.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:53:29.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: yield 2021-07-31T01:53:29.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:53:29.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:53:29.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:53:29.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T01:53:29.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:53:29.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T01:53:29.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:53:29.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: await self._waiter 2021-07-31T01:53:29.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:53:29.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: task.result() 2021-07-31T01:53:29.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T01:53:29.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: debug 2021-07-31T01:53:28.868+0000 7fcb93160700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:53:29.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:53:29.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:53:29.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: yield 2021-07-31T01:53:29.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:53:29.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:53:29.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:53:29.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T01:53:29.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:53:29.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T01:53:29.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:53:29.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: await self._waiter 2021-07-31T01:53:29.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:53:29.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: task.result() 2021-07-31T01:53:29.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T01:53:29.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: 2021-07-31T01:53:29.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T01:53:29.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: 2021-07-31T01:53:29.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:53:29.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:53:29.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: return f(*arg) 2021-07-31T01:53:29.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:53:29.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:53:29.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:53:29.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:53:29.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:53:29.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:53:29.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:53:29.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T01:53:29.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:53:29.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: raise self._exception 2021-07-31T01:53:29.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:53:29.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:53:29.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:53:29.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T01:53:29.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:53:29.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: raise 2021-07-31T01:53:29.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:53:29.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T01:53:29.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:53:29.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:53:29.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T01:53:29.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: debug 2021-07-31T01:53:28.877+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T01:53:29.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:53:29.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:53:29.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: yield 2021-07-31T01:53:29.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:53:29.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:53:29.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:53:29.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T01:53:29.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:53:29.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T01:53:29.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:53:29.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: await self._waiter 2021-07-31T01:53:29.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:53:29.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: task.result() 2021-07-31T01:53:29.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T01:53:29.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: debug 2021-07-31T01:53:28.877+0000 7fcb9295f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:53:29.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:53:29.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:53:29.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: yield 2021-07-31T01:53:29.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:53:29.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:53:29.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:53:29.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T01:53:29.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:53:29.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T01:53:29.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:53:29.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: await self._waiter 2021-07-31T01:53:29.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:53:29.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: task.result() 2021-07-31T01:53:29.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T01:53:29.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: 2021-07-31T01:53:29.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T01:53:29.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: 2021-07-31T01:53:29.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:53:29.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:53:29.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: return f(*arg) 2021-07-31T01:53:29.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:53:29.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:53:29.035 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:53:29.035 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:53:29.035 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:53:29.035 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:53:29.036 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:53:29.036 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T01:53:29.036 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:53:29.036 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: raise self._exception 2021-07-31T01:53:29.037 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:53:29.037 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:53:29.037 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:53:29.037 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T01:53:29.038 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:53:29.038 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: raise 2021-07-31T01:53:29.038 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:53:29.038 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T01:53:29.038 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:53:29.039 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:53:29.039 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:53:28 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T01:53:29.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:28 smithi045 conmon[39718]: cluster 2021-07-31T01:53:27. 2021-07-31T01:53:29.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:28 smithi045 conmon[39718]: 644571+0000 mon.a (mon.0) 1159 : cluster [DBG] osdmap e506: 8 total, 8 up, 8 in 2021-07-31T01:53:29.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:28 smithi045 conmon[39718]: cluster 2021-07-31T01 2021-07-31T01:53:29.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:28 smithi045 conmon[39718]: :53:27.880010+0000 mgr.x (mgr.24542) 261 : cluster [DBG] 2021-07-31T01:53:29.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:28 smithi045 conmon[39718]: pgmap v540: 25 pgs: 25 active+clean; 198 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail 2021-07-31T01:53:29.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:28 smithi045 conmon[39718]: audit 2021- 2021-07-31T01:53:29.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:28 smithi045 conmon[39718]: 07-31T01:53:27.962169+0000 mon.b (mon 2021-07-31T01:53:29.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:28 smithi045 conmon[39718]: .2) 92 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:53:29.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:28 smithi045 conmon[39718]: audit 2021-07 2021-07-31T01:53:29.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:28 smithi045 conmon[39718]: -31T01:53:27.962526+0000 mon.a (mon.0) 2021-07-31T01:53:29.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:28 smithi045 conmon[39718]: 1160 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:53:29.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:28 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T01:53:29.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:28 smithi045 conmon[39718]: 31T01:53:27.975661+0000 mon.b (mon.2) 93 2021-07-31T01:53:29.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:28 smithi045 conmon[39718]: : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T01:53:29.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:28 smithi045 conmon[39718]: audit 2021-07 2021-07-31T01:53:29.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:28 smithi045 conmon[39718]: -31T01:53:27.975938+0000 mon.a (mon. 2021-07-31T01:53:29.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:28 smithi045 conmon[39718]: 0) 1161 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T01:53:29.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:28 smithi045 conmon[39718]: 2021-07-31T01:53:29.044 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:28 smithi045 conmon[33655]: cluster 2021-07-31T01:53:27.644571+0000 mon.a (mon. 2021-07-31T01:53:29.044 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:28 smithi045 conmon[33655]: 0) 1159 : cluster [DBG] osdmap e506: 8 total, 8 up, 8 in 2021-07-31T01:53:29.044 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:28 smithi045 conmon[33655]: cluster 2021-07-31T01:53:27.880010+0000 mgr.x (mgr.24542) 261 : cluster [DBG] pgmap v540: 25 pgs: 25 active+clean; 198 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail 2021-07-31T01:53:29.045 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:28 smithi045 conmon[33655]: audit 2021- 2021-07-31T01:53:29.045 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:28 smithi045 conmon[33655]: 07-31T01:53:27 2021-07-31T01:53:29.045 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:28 smithi045 conmon[33655]: .962169+0000 mon.b (mon.2) 2021-07-31T01:53:29.045 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:28 smithi045 conmon[33655]: 92 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:53:29.045 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:28 smithi045 conmon[33655]: audit 2021-07-31T01 2021-07-31T01:53:29.046 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:28 smithi045 conmon[33655]: :53:27.962526+0000 mon.a 2021-07-31T01:53:29.046 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:28 smithi045 conmon[33655]: (mon.0) 1160 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:53:29.046 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:28 smithi045 conmon[33655]: 2021-07-31T01:53:29.046 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:28 smithi045 conmon[33655]: audit 2021-07-31 2021-07-31T01:53:29.047 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:28 smithi045 conmon[33655]: T01:53:27.975661+0000 2021-07-31T01:53:29.047 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:28 smithi045 conmon[33655]: mon.b (mon.2) 93 : audit [INF] 2021-07-31T01:53:29.047 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:28 smithi045 conmon[33655]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T01:53:29.047 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:28 smithi045 conmon[33655]: audit 2021 2021-07-31T01:53:29.047 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:28 smithi045 conmon[33655]: -07-31T01:53: 2021-07-31T01:53:29.048 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:28 smithi045 conmon[33655]: 27.975938+0000 mon.a (mon. 2021-07-31T01:53:29.048 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:28 smithi045 conmon[33655]: 0) 1161 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T01:53:29.655 INFO:tasks.rados.rados.0.smithi045.stdout:2347: done (7 left) 2021-07-31T01:53:29.656 INFO:tasks.rados.rados.0.smithi045.stdout:2348: done (6 left) 2021-07-31T01:53:29.656 INFO:tasks.rados.rados.0.smithi045.stdout:2349: done (5 left) 2021-07-31T01:53:29.656 INFO:tasks.rados.rados.0.smithi045.stdout:2350: done (4 left) 2021-07-31T01:53:29.657 INFO:tasks.rados.rados.0.smithi045.stdout:2351: done (3 left) 2021-07-31T01:53:29.657 INFO:tasks.rados.rados.0.smithi045.stdout:2352: done (2 left) 2021-07-31T01:53:29.657 INFO:tasks.rados.rados.0.smithi045.stdout:2353: done (1 left) 2021-07-31T01:53:29.657 INFO:tasks.rados.rados.0.smithi045.stdout:2354: done (0 left) 2021-07-31T01:53:29.658 INFO:tasks.rados.rados.0.smithi045.stdout:2355: rollback oid 6 current snap is 225 2021-07-31T01:53:29.658 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 6 to 223 2021-07-31T01:53:29.658 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 9 current snap is 225 2021-07-31T01:53:29.658 INFO:tasks.rados.rados.0.smithi045.stdout:2356: seq_num 723 ranges {0=114688} 2021-07-31T01:53:29.659 INFO:tasks.rados.rados.0.smithi045.stdout:2356: writing smithi04569269-9 from 0 to 114688 tid 1 2021-07-31T01:53:29.659 INFO:tasks.rados.rados.0.smithi045.stdout:2357: copy_from oid 19 from oid 31 current snap is 225 2021-07-31T01:53:29.659 INFO:tasks.rados.rados.0.smithi045.stdout:2358: rollback oid 17 current snap is 225 2021-07-31T01:53:29.659 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 17 to 222 2021-07-31T01:53:29.659 INFO:tasks.rados.rados.0.smithi045.stdout:2359: delete oid 21 current snap is 225 2021-07-31T01:53:29.661 INFO:tasks.rados.rados.0.smithi045.stdout:2355: finishing rollback tid 0 to smithi04569269-6 2021-07-31T01:53:29.661 INFO:tasks.rados.rados.0.smithi045.stdout:2358: finishing rollback tid 0 to smithi04569269-17 2021-07-31T01:53:29.664 INFO:tasks.rados.rados.0.smithi045.stdout:2358: finishing rollback tid 1 to smithi04569269-17 2021-07-31T01:53:29.664 INFO:tasks.rados.rados.0.smithi045.stdout:2355: finishing rollback tid 1 to smithi04569269-6 2021-07-31T01:53:29.665 INFO:tasks.rados.rados.0.smithi045.stdout:2355: finishing rollback tid 2 to smithi04569269-6 2021-07-31T01:53:29.665 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 6 v 1388 (ObjNum 589 snap 182 seq_num 589) dirty exists 2021-07-31T01:53:29.665 INFO:tasks.rados.rados.0.smithi045.stdout:2357: finishing copy_from to smithi04569269-19 2021-07-31T01:53:29.665 INFO:tasks.rados.rados.0.smithi045.stdout:2357: got expected ENOENT (src dne) 2021-07-31T01:53:29.666 INFO:tasks.rados.rados.0.smithi045.stdout:2356: finishing write tid 1 to smithi04569269-9 2021-07-31T01:53:29.666 INFO:tasks.rados.rados.0.smithi045.stdout:2357: finishing copy_from racing read to smithi04569269-19 2021-07-31T01:53:29.666 INFO:tasks.rados.rados.0.smithi045.stdout:2356: finishing write tid 2 to smithi04569269-9 2021-07-31T01:53:29.666 INFO:tasks.rados.rados.0.smithi045.stdout:2358: finishing rollback tid 2 to smithi04569269-17 2021-07-31T01:53:29.667 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 17 v 1179 (ObjNum 664 snap 208 seq_num 664) dirty exists 2021-07-31T01:53:29.667 INFO:tasks.rados.rados.0.smithi045.stdout:2355: done (4 left) 2021-07-31T01:53:29.667 INFO:tasks.rados.rados.0.smithi045.stdout:2357: done (3 left) 2021-07-31T01:53:29.667 INFO:tasks.rados.rados.0.smithi045.stdout:2358: done (2 left) 2021-07-31T01:53:29.667 INFO:tasks.rados.rados.0.smithi045.stdout:2359: done (1 left) 2021-07-31T01:53:29.668 INFO:tasks.rados.rados.0.smithi045.stdout:2360: read oid 22 snap -1 2021-07-31T01:53:29.668 INFO:tasks.rados.rados.0.smithi045.stdout:2360: expect (ObjNum 575 snap 175 seq_num 575) 2021-07-31T01:53:29.668 INFO:tasks.rados.rados.0.smithi045.stdout:2361: read oid 50 snap -1 2021-07-31T01:53:29.668 INFO:tasks.rados.rados.0.smithi045.stdout:2361: expect (ObjNum 635 snap 198 seq_num 635) 2021-07-31T01:53:29.668 INFO:tasks.rados.rados.0.smithi045.stdout:2362: delete oid 11 current snap is 225 2021-07-31T01:53:29.670 INFO:tasks.rados.rados.0.smithi045.stdout:2356: finishing write tid 3 to smithi04569269-9 2021-07-31T01:53:29.670 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 9 v 1060 (ObjNum 723 snap 225 seq_num 723) dirty exists 2021-07-31T01:53:29.670 INFO:tasks.rados.rados.0.smithi045.stdout:2356: left oid 9 (ObjNum 723 snap 225 seq_num 723) 2021-07-31T01:53:29.670 INFO:tasks.rados.rados.0.smithi045.stdout:2360: expect (ObjNum 575 snap 175 seq_num 575) 2021-07-31T01:53:29.671 INFO:tasks.rados.rados.0.smithi045.stdout:2356: done (3 left) 2021-07-31T01:53:29.671 INFO:tasks.rados.rados.0.smithi045.stdout:2360: done (2 left) 2021-07-31T01:53:29.671 INFO:tasks.rados.rados.0.smithi045.stdout:2362: done (1 left) 2021-07-31T01:53:29.671 INFO:tasks.rados.rados.0.smithi045.stdout:2363: read oid 41 snap -1 2021-07-31T01:53:29.672 INFO:tasks.rados.rados.0.smithi045.stdout:2363: expect (ObjNum 714 snap 222 seq_num 714) 2021-07-31T01:53:29.672 INFO:tasks.rados.rados.0.smithi045.stdout:2364: snap_remove snap 223 2021-07-31T01:53:30.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: cluster 2021-07-31T01:53:28.650463+0000 mon.a (mon.0) 1162 : cluster 2021-07-31T01:53:30.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: [DBG] osdmap e507: 8 total, 8 up, 8 in 2021-07-31T01:53:30.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: audit 2021-07-31T01:53:28.789562+0000 mon.b (mon.2) 94 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:53:30.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: audit 2021-07-31T01: 2021-07-31T01:53:30.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: 53:28.790813+0000 mon.b (mon.2) 95 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:53:30.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: audit 2021-07- 2021-07-31T01:53:30.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: 31T01:53:28.791489+0000 mon.b (mon.2) 96 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:53:30.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: cephadm 2021-07-31T01:53:28.792094+0000 mgr.x ( 2021-07-31T01:53:30.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: mgr.24542) 262 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T01:53:30.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: cephadm 2021-07-31T01:53: 2021-07-31T01:53:30.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: 28.832892+0000 mgr.x (mgr.24542) 263 : cephadm [ERR] 2021-07-31T01:53:30.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:53:30.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:53:30.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: yield 2021-07-31T01:53:30.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:53:30.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:53:30.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:53:30.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T01:53:30.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:53:30.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T01:53:30.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:53:30.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: await self._waiter 2021-07-31T01:53:30.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:53:30.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: task.result() 2021-07-31T01:53:30.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T01:53:30.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: cephadm 2021-07-31T01:53:28.833263+0000 mgr.x (mgr.24542) 264 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T01:53:30.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: cephadm 2021-07-31T01:53:28.844869+0000 mgr.x (mgr.24542) 265 : cephadm [ERR] 2021-07-31T01:53:30.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:53:30.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:53:30.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: yield 2021-07-31T01:53:30.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:53:30.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:53:30.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:53:30.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T01:53:30.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:53:30.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T01:53:30.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:53:30.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: await self._waiter 2021-07-31T01:53:30.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:53:30.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: task.result() 2021-07-31T01:53:30.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T01:53:30.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: cephadm 2021 2021-07-31T01:53:30.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: -07-31T01:53:28.845620+0000 mgr.x (mgr.24542) 266 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T01:53:30.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: cephadm 2021-07-31T01:53:28.869132+0000 mgr.x (mgr.24542) 267 : cephadm [ERR] 2021-07-31T01:53:30.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:53:30.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:53:30.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: yield 2021-07-31T01:53:30.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:53:30.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:53:30.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:53:30.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T01:53:30.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:53:30.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T01:53:30.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:53:30.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: await self._waiter 2021-07-31T01:53:30.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:53:30.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: task.result() 2021-07-31T01:53:30.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T01:53:30.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: cephadm 2021-07-31 2021-07-31T01:53:30.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: T01:53:28.869532+0000 mgr.x (mgr.24542) 268 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:53:30.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:53:30.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:53:30.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: yield 2021-07-31T01:53:30.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:53:30.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:53:30.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:53:30.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T01:53:30.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:53:30.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T01:53:30.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:53:30.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: await self._waiter 2021-07-31T01:53:30.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:53:30.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: task.result() 2021-07-31T01:53:30.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T01:53:30.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: 2021-07-31T01:53:30.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T01:53:30.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: 2021-07-31T01:53:30.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:53:30.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:53:30.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: return f(*arg) 2021-07-31T01:53:30.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:53:30.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:53:30.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:53:30.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:53:30.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:53:30.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:53:30.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:53:30.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T01:53:30.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:53:30.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: raise self._exception 2021-07-31T01:53:30.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:53:30.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:53:30.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:53:30.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T01:53:30.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:53:30.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: raise 2021-07-31T01:53:30.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:53:30.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T01:53:30.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:53:30.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:53:30.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T01:53:30.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: cephadm 2021-07-31T01:53:28.878147+0000 mgr.x (mgr.24542) 269 : cephadm [ERR] 2021-07-31T01:53:30.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:53:30.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:53:30.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: yield 2021-07-31T01:53:30.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:53:30.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:53:30.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:53:30.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T01:53:30.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:53:30.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T01:53:30.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:53:30.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: await self._waiter 2021-07-31T01:53:30.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:53:30.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: task.result() 2021-07-31T01:53:30.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T01:53:30.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: cephadm 2021-07-31T01:53:28.878500+0000 mgr.x (mgr.24542) 270 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:53:30.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:53:30.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:53:30.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: yield 2021-07-31T01:53:30.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:53:30.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:53:30.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:53:30.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T01:53:30.035 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:53:30.035 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T01:53:30.035 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:53:30.035 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: await self._waiter 2021-07-31T01:53:30.036 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:53:30.036 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: task.result() 2021-07-31T01:53:30.036 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T01:53:30.036 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: 2021-07-31T01:53:30.037 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T01:53:30.037 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: 2021-07-31T01:53:30.037 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:53:30.037 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:53:30.037 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: return f(*arg) 2021-07-31T01:53:30.038 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:53:30.038 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:53:30.038 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:53:30.038 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:53:30.038 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:53:30.039 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:53:30.039 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:53:30.039 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T01:53:30.039 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:53:30.039 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: raise self._exception 2021-07-31T01:53:30.040 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:53:30.040 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:53:30.040 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:53:30.040 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T01:53:30.041 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:53:30.041 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: raise 2021-07-31T01:53:30.041 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:53:30.041 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T01:53:30.042 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:53:30.042 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:53:30.042 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T01:53:30.042 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:29 smithi154 conmon[34553]: 2021-07-31T01:53:30.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: cluster 2021-07-31T01:53:28.650463+0000 2021-07-31T01:53:30.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: mon.a (mon.0) 1162 : cluster [DBG] osdmap e507: 8 total, 8 up, 8 in 2021-07-31T01:53:30.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T01:53:30.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: 31T01:53:28.789562+0000 mon.b (mon 2021-07-31T01:53:30.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: .2) 94 : audit [DBG] 2021-07-31T01:53:30.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:53:30.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: audit 2021-07-31T01:53:30.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: 2021-07-31T01: 2021-07-31T01:53:30.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: 53:28.790813+0000 2021-07-31T01:53:30.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: mon.b (mon.2) 95 : 2021-07-31T01:53:30.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:53:30.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: audit 2021-07-31T01:53:30.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: 2021-07-31 2021-07-31T01:53:30.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: T01:53:28.791489+0000 mon.b (mon. 2021-07-31T01:53:30.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: 2) 96 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:53:30.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: cephadm 2021- 2021-07-31T01:53:30.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: 07-31T01:53:28 2021-07-31T01:53:30.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: .792094+0000 mgr.x (mgr.24542) 2021-07-31T01:53:30.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: 262 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T01:53:30.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: cephadm 2021-07-31T01:53:30.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: 2021-07 2021-07-31T01:53:30.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: -31T01:53:28.832892 2021-07-31T01:53:30.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 263 : cephadm 2021-07-31T01:53:30.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: [ERR] 2021-07-31T01:53:30.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:53:30.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:53:30.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: yield 2021-07-31T01:53:30.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:53:30.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:53:30.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:53:30.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T01:53:30.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:53:30.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T01:53:30.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:53:30.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: await self._waiter 2021-07-31T01:53:30.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:53:30.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: task.result() 2021-07-31T01:53:30.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T01:53:30.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: cephadm 2021-07-31T01:53:28.833263+0000 mgr.x (mgr.24542) 264 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T01:53:30.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: cephadm 2021-07-31T01 2021-07-31T01:53:30.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: :53:28.844869+0000 mgr.x (mgr.24542) 265 : 2021-07-31T01:53:30.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: cephadm [ERR] 2021-07-31T01:53:30.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:53:30.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:53:30.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: yield 2021-07-31T01:53:30.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:53:30.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:53:30.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:53:30.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T01:53:30.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:53:30.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T01:53:30.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:53:30.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: await self._waiter 2021-07-31T01:53:30.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:53:30.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: task.result() 2021-07-31T01:53:30.057 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T01:53:30.057 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: cephadm 2021-07-31T01:53:28.845620+0000 mgr.x (mgr.24542) 266 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T01:53:30.057 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: cephadm 2021 2021-07-31T01:53:30.057 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: -07-31T01:53: 2021-07-31T01:53:30.057 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: 28.869132+0000 mgr.x ( 2021-07-31T01:53:30.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: mgr.24542) 267 : cephadm [ERR] 2021-07-31T01:53:30.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:53:30.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:53:30.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: yield 2021-07-31T01:53:30.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:53:30.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:53:30.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:53:30.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T01:53:30.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:53:30.060 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T01:53:30.060 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:53:30.060 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: await self._waiter 2021-07-31T01:53:30.060 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:53:30.060 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: task.result() 2021-07-31T01:53:30.061 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T01:53:30.061 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: 2021-07-31T01:53:30.061 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: cephadm 2021-07-31T01:53:28.869532+0000 mgr.x (mgr.24542) 268 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:53:30.061 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:53:30.061 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:53:30.062 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: yield 2021-07-31T01:53:30.062 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:53:30.062 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:53:30.062 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:53:30.063 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T01:53:30.063 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:53:30.063 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T01:53:30.064 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: cluster 2021-07-31T01:53:28.650463+0000 mon.a (mon 2021-07-31T01:53:30.064 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: .0) 1162 : cluster [DBG] osdmap e507: 8 total, 8 up, 8 in 2021-07-31T01:53:30.064 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: audit 2021-07-31T01:53:28.789562+0000 mon.b (mon.2) 94 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:53:30.064 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: audit 2021-07-31T01:53:28.790813+0000 mon.b (mon.2 2021-07-31T01:53:30.065 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: ) 95 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:53:30.065 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: audit 2021-07-31T01:53:28.791489 2021-07-31T01:53:30.065 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: +0000 mon.b (mon.2) 96 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:53:30.065 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: cephadm 2021-07-31 2021-07-31T01:53:30.066 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: T01:53:28.792094+0000 mgr.x (mgr.24542) 2021-07-31T01:53:30.066 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: 262 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T01:53:30.066 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: cephadm 2021-07-31T01:53:28.832892+0000 mgr.x 2021-07-31T01:53:30.066 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: (mgr.24542) 263 : cephadm [ERR] 2021-07-31T01:53:30.067 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:53:30.067 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:53:30.067 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: yield 2021-07-31T01:53:30.067 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:53:30.067 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:53:30.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:53:30.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T01:53:30.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:53:30.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T01:53:30.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:53:30.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: await self._waiter 2021-07-31T01:53:30.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:53:30.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: task.result() 2021-07-31T01:53:30.070 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T01:53:30.070 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: cephadm 2021-07-31T01:53:28.833263+0000 mgr.x (mgr.24542) 264 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T01:53:30.070 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: cephadm 2021-07-31T01:53:28.844869+0000 mgr.x (mgr.24542) 265 : cephadm [ERR] 2021-07-31T01:53:30.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:53:30.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:53:30.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: yield 2021-07-31T01:53:30.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:53:30.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:53:30.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:53:30.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T01:53:30.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:53:30.073 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:53:30.073 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: await self._waiter 2021-07-31T01:53:30.073 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:53:30.074 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: task.result() 2021-07-31T01:53:30.074 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T01:53:30.074 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: 2021-07-31T01:53:30.074 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T01:53:30.075 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: 2021-07-31T01:53:30.075 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:53:30.075 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:53:30.075 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: return f(*arg) 2021-07-31T01:53:30.075 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:53:30.076 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:53:30.076 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:53:30.076 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:53:30.076 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:53:30.076 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:53:30.077 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:53:30.077 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T01:53:30.077 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:53:30.077 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: raise self._exception 2021-07-31T01:53:30.077 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:53:30.078 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:53:30.078 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:53:30.078 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T01:53:30.078 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:53:30.078 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: raise 2021-07-31T01:53:30.079 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:53:30.079 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T01:53:30.079 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:53:30.079 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:53:30.080 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T01:53:30.080 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: cephadm 2021-07-31T01:53:28.878147+0000 mgr.x (mgr.24542) 269 : cephadm [ERR] 2021-07-31T01:53:30.080 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:53:30.080 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:53:30.080 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: yield 2021-07-31T01:53:30.081 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:53:30.081 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:53:30.081 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:53:30.081 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T01:53:30.081 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:53:30.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T01:53:30.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:53:30.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: await self._waiter 2021-07-31T01:53:30.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:53:30.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: task.result() 2021-07-31T01:53:30.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T01:53:30.085 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: cephadm 2021-07-31T 2021-07-31T01:53:30.085 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: 01:53:28.845620+0000 mgr.x (mgr.24542) 266 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T01:53:30.086 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: cephadm 2021-07-31T01:53:28.869132+0000 mgr.x (mgr.24542) 267 : cephadm [ERR] 2021-07-31T01:53:30.086 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:53:30.086 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:53:30.086 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: yield 2021-07-31T01:53:30.087 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:53:30.087 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:53:30.087 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:53:30.087 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T01:53:30.087 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:53:30.088 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T01:53:30.088 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:53:30.088 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: await self._waiter 2021-07-31T01:53:30.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:53:30.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: task.result() 2021-07-31T01:53:30.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T01:53:30.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: cephadm 2021-07-31T01:53: 2021-07-31T01:53:30.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: 28.869532+0000 mgr.x (mgr.24542) 268 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:53:30.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:53:30.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:53:30.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: yield 2021-07-31T01:53:30.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:53:30.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:53:30.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:53:30.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T01:53:30.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:53:30.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T01:53:30.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:53:30.093 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: await self._waiter 2021-07-31T01:53:30.093 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:53:30.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: task.result() 2021-07-31T01:53:30.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T01:53:30.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: 2021-07-31T01:53:30.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T01:53:30.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: 2021-07-31T01:53:30.095 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T01:53:30.095 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:53:30.096 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: await self._waiter 2021-07-31T01:53:30.096 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:53:30.096 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: task.result() 2021-07-31T01:53:30.096 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T01:53:30.097 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: cephadm 2021-07-31T01:53:28.878500+0000 mgr.x (mgr.24542) 270 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:53:30.097 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:53:30.097 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:53:30.097 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: yield 2021-07-31T01:53:30.098 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:53:30.098 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:53:30.098 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:53:30.098 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T01:53:30.098 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:53:30.099 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T01:53:30.099 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:53:30.099 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: await self._waiter 2021-07-31T01:53:30.099 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:53:30.100 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: task.result() 2021-07-31T01:53:30.100 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T01:53:30.100 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: 2021-07-31T01:53:30.100 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T01:53:30.100 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: 2021-07-31T01:53:30.101 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:53:30.101 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:53:30.101 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: return f(*arg) 2021-07-31T01:53:30.101 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:53:30.102 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:53:30.102 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:53:30.102 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:53:30.102 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:53:30.102 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:53:30.103 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:53:30.103 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T01:53:30.103 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:53:30.103 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: raise self._exception 2021-07-31T01:53:30.104 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:53:30.104 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:53:30.104 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:53:30.104 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T01:53:30.104 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:53:30.105 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: raise 2021-07-31T01:53:30.105 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:53:30.105 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T01:53:30.105 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:53:30.105 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:53:30.106 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T01:53:30.106 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:53:30.107 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:53:30.107 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: return f(*arg) 2021-07-31T01:53:30.107 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:53:30.107 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:53:30.108 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:53:30.108 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:53:30.108 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:53:30.108 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:53:30.108 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:53:30.109 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T01:53:30.109 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:53:30.109 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: raise self._exception 2021-07-31T01:53:30.109 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:53:30.110 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:53:30.110 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:53:30.110 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T01:53:30.110 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:53:30.110 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: raise 2021-07-31T01:53:30.111 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:53:30.111 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T01:53:30.111 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:53:30.111 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:53:30.112 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T01:53:30.112 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: cephadm 2021-07-31T01:53:28.878147+0000 mgr.x (mgr.24542) 269 : cephadm [ERR] 2021-07-31T01:53:30.112 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:53:30.112 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:53:30.112 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: yield 2021-07-31T01:53:30.113 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:53:30.113 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:53:30.113 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:53:30.113 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T01:53:30.113 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:53:30.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T01:53:30.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:53:30.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: await self._waiter 2021-07-31T01:53:30.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:53:30.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: task.result() 2021-07-31T01:53:30.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T01:53:30.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: cephadm 2021-07-31T01:53:28.878500+0000 mgr.x (mgr.24542) 270 : 2021-07-31T01:53:30.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:53:30.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:53:30.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:53:30.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: yield 2021-07-31T01:53:30.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:53:30.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:53:30.117 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:53:30.117 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T01:53:30.117 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:53:30.117 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T01:53:30.117 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:53:30.118 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: await self._waiter 2021-07-31T01:53:30.118 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:53:30.118 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: task.result() 2021-07-31T01:53:30.118 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T01:53:30.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: 2021-07-31T01:53:30.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T01:53:30.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: 2021-07-31T01:53:30.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:53:30.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:53:30.120 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: return f(*arg) 2021-07-31T01:53:30.120 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:53:30.120 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:53:30.120 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:53:30.120 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:53:30.121 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:53:30.121 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:53:30.121 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:53:30.121 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T01:53:30.122 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:53:30.122 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: raise self._exception 2021-07-31T01:53:30.122 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:53:30.122 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:53:30.122 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:53:30.123 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T01:53:30.123 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:53:30.123 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: raise 2021-07-31T01:53:30.123 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:53:30.123 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T01:53:30.124 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:53:30.124 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:53:30.124 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:29 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T01:53:30.662 INFO:tasks.rados.rados.0.smithi045.stdout:2364: done (2 left) 2021-07-31T01:53:30.663 INFO:tasks.rados.rados.0.smithi045.stdout:2365: snap_create 2021-07-31T01:53:30.663 INFO:tasks.rados.rados.0.smithi045.stdout:2361: expect (ObjNum 635 snap 198 seq_num 635) 2021-07-31T01:53:30.769 INFO:tasks.rados.rados.0.smithi045.stdout:2363: expect (ObjNum 714 snap 222 seq_num 714) 2021-07-31T01:53:31.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:30 smithi154 conmon[34553]: cluster 2021-07-31T01:53:29.654880+0000 mon.a (mon.0) 1163 : cluster [DBG] 2021-07-31T01:53:31.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:30 smithi154 conmon[34553]: osdmap e508: 8 total, 8 up, 8 in 2021-07-31T01:53:31.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:30 smithi154 conmon[34553]: cluster 2021-07-31T01:53:29.880559+0000 mgr.x (mgr.24542) 271 : cluster [DBG] pgmap v543: 25 pgs: 25 active+clean; 216 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 30 MiB/s rd, 3.8 MiB/s wr, 32 op/s 2021-07-31T01:53:31.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:30 smithi045 conmon[33655]: cluster 2021-07-31T01:53:29.654880+0000 mon.a (mon.0) 1163 : cluster [DBG] 2021-07-31T01:53:31.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:30 smithi045 conmon[33655]: osdmap e508: 8 total, 8 up, 8 in 2021-07-31T01:53:31.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:30 smithi045 conmon[33655]: cluster 2021-07-31T01:53:29.880559+0000 mgr.x (mgr.24542) 271 : cluster [DBG] pgmap v543: 25 pgs: 25 active+clean; 216 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 30 MiB/s rd, 3.8 MiB/s wr, 32 op/s 2021-07-31T01:53:31.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:30 smithi045 conmon[39718]: cluster 2021-07-31T01:53:29.654880+0000 mon.a (mon. 2021-07-31T01:53:31.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:30 smithi045 conmon[39718]: 0) 1163 : cluster [DBG] osdmap e508: 8 total, 8 up, 8 in 2021-07-31T01:53:31.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:30 smithi045 conmon[39718]: cluster 2021-07-31T01:53:29.880559+0000 mgr.x (mgr. 2021-07-31T01:53:31.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:30 smithi045 conmon[39718]: 24542) 271 : cluster [DBG] pgmap v543: 25 pgs: 25 active+clean; 216 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 30 MiB/s rd, 3.8 MiB/s wr, 32 op/s 2021-07-31T01:53:31.590 INFO:tasks.rados.rados.0.smithi045.stdout:2361: done (2 left) 2021-07-31T01:53:31.591 INFO:tasks.rados.rados.0.smithi045.stdout:2363: done (1 left) 2021-07-31T01:53:31.591 INFO:tasks.rados.rados.0.smithi045.stdout:2365: done (0 left) 2021-07-31T01:53:31.591 INFO:tasks.rados.rados.0.smithi045.stdout:2366: copy_from oid 3 from oid 15 current snap is 226 2021-07-31T01:53:31.591 INFO:tasks.rados.rados.0.smithi045.stdout:2367: read oid 33 snap -1 2021-07-31T01:53:31.592 INFO:tasks.rados.rados.0.smithi045.stdout:2367: expect (ObjNum 659 snap 205 seq_num 659) 2021-07-31T01:53:31.592 INFO:tasks.rados.rados.0.smithi045.stdout:2368: rmattr oid 5 current snap is 226 2021-07-31T01:53:31.592 INFO:tasks.rados.rados.0.smithi045.stdout:2369: copy_from oid 30 from oid 25 current snap is 226 2021-07-31T01:53:31.592 INFO:tasks.rados.rados.0.smithi045.stdout:2370: snap_remove snap 219 2021-07-31T01:53:32.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:31 smithi154 conmon[34553]: cluster 2021-07-31T01:53:30.661700+0000 mon.a (mon.0) 1164 : 2021-07-31T01:53:32.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:31 smithi154 conmon[34553]: cluster [DBG] osdmap e509: 8 total, 8 up, 8 in 2021-07-31T01:53:32.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:31 smithi154 conmon[34553]: cluster 2021-07-31T01:53:31.589347+0000 mon.a (mon.0) 1165 : cluster [DBG] osdmap e510: 8 total, 8 up, 8 in 2021-07-31T01:53:32.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:31 smithi045 conmon[33655]: cluster 2021-07-31T01:53:30.661700+0000 mon.a (mon 2021-07-31T01:53:32.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:31 smithi045 conmon[33655]: .0) 1164 : cluster [DBG] osdmap e509: 8 total, 8 up, 8 in 2021-07-31T01:53:32.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:31 smithi045 conmon[33655]: cluster 2021-07-31T01:53:31.589347+0000 mon.a (mon.0) 1165 : cluster [DBG] osdmap e510: 8 total, 8 up, 8 in 2021-07-31T01:53:32.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:31 smithi045 conmon[39718]: cluster 2021-07-31T01:53:30 2021-07-31T01:53:32.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:31 smithi045 conmon[39718]: .661700+0000 mon.a (mon.0) 1164 : cluster [DBG] osdmap e509: 8 total, 8 up, 8 in 2021-07-31T01:53:32.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:31 smithi045 conmon[39718]: cluster 2021- 2021-07-31T01:53:32.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:31 smithi045 conmon[39718]: 07-31T01:53: 2021-07-31T01:53:32.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:31 smithi045 conmon[39718]: 31.589347+0000 mon.a (mon.0 2021-07-31T01:53:32.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:31 smithi045 conmon[39718]: ) 1165 : cluster [DBG] osdmap e510: 8 total, 8 up, 8 in 2021-07-31T01:53:32.598 INFO:tasks.rados.rados.0.smithi045.stdout:2370: done (4 left) 2021-07-31T01:53:32.598 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 31 current snap is 226 2021-07-31T01:53:32.598 INFO:tasks.rados.rados.0.smithi045.stdout:2371: seq_num 726 ranges {0=163840} 2021-07-31T01:53:32.600 INFO:tasks.rados.rados.0.smithi045.stdout:2371: writing smithi04569269-31 from 0 to 163840 tid 1 2021-07-31T01:53:32.600 INFO:tasks.rados.rados.0.smithi045.stdout:2372: delete oid 9 current snap is 226 2021-07-31T01:53:32.601 INFO:tasks.rados.rados.0.smithi045.stdout:2366: finishing copy_from to smithi04569269-3 2021-07-31T01:53:32.601 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 3 v 987 (ObjNum 677 snap 211 seq_num 677) dirty exists 2021-07-31T01:53:32.601 INFO:tasks.rados.rados.0.smithi045.stdout:2366: finishing copy_from racing read to smithi04569269-3 2021-07-31T01:53:32.602 INFO:tasks.rados.rados.0.smithi045.stdout:2367: expect (ObjNum 659 snap 205 seq_num 659) 2021-07-31T01:53:33.003 INFO:journalctl@ceph.osd.5.smithi154.stdout:Jul 31 01:53:32 smithi154 conmon[42050]: debug 2021-07-31T01:53:32.603+0000 7fc9b5064700 -1 osd.5 pg_epoch: 511 pg[3.cs0( v 510'950 (0'0,510'950] local-lis/les=56/57 n=9 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [5,3,6]p5(0) r=0 lpr=56 crt=510'950 lcod 509'948 mlcod 509'948 active+clean+snaptrim trimq=[1b7~1] ps=[1be~1]] removing snap head 2021-07-31T01:53:33.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:32 smithi154 conmon[34553]: cluster 2021-07-31T01:53:31.881075+0000 mgr.x (mgr.24542) 272 : 2021-07-31T01:53:33.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:32 smithi154 conmon[34553]: cluster [DBG] pgmap v546: 25 pgs: 25 active+clean; 216 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 30 MiB/s rd, 3.8 MiB/s wr, 32 op/s 2021-07-31T01:53:33.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:32 smithi154 conmon[34553]: cluster 2021-07-31T01:53:32.597090+0000 mon.a (mon.0) 1166 : cluster [DBG] osdmap e511: 8 total, 8 up, 8 in 2021-07-31T01:53:33.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:32 smithi045 conmon[33655]: cluster 2021-07-31T01:53:31.881075+0000 mgr.x (mgr.24542) 272 : cluster [DBG] pgmap v546: 25 pgs: 25 active+clean; 216 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 30 MiB/s rd, 3.8 MiB/s wr, 32 op/s 2021-07-31T01:53:33.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:32 smithi045 conmon[33655]: cluster 2021-07-31T01:53:32.597090+0000 mon.a (mon.0) 1166 : cluster [DBG] osdmap e511: 8 total, 8 up, 8 in 2021-07-31T01:53:33.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:32 smithi045 conmon[39718]: cluster 2021-07-31T01:53:31.881075+0000 2021-07-31T01:53:33.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:32 smithi045 conmon[39718]: mgr.x (mgr.24542) 272 : cluster [DBG] pgmap v546: 25 pgs: 25 active+clean; 216 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 30 MiB/s rd, 3.8 MiB/s wr, 32 op/s 2021-07-31T01:53:33.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:32 smithi045 conmon[39718]: cluster 2021-07-31T01: 2021-07-31T01:53:33.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:32 smithi045 conmon[39718]: 53:32.597090+0000 mon.a (mon.0) 1166 : cluster 2021-07-31T01:53:33.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:32 smithi045 conmon[39718]: [DBG] osdmap e511: 8 total, 8 up, 8 in 2021-07-31T01:53:33.319 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 5 v 1172 (ObjNum 695 snap 217 seq_num 695) dirty exists 2021-07-31T01:53:33.319 INFO:tasks.rados.rados.0.smithi045.stdout:2369: finishing copy_from to smithi04569269-30 2021-07-31T01:53:33.319 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 30 v 334 (ObjNum 678 snap 211 seq_num 678) dirty exists 2021-07-31T01:53:33.320 INFO:tasks.rados.rados.0.smithi045.stdout:2369: finishing copy_from racing read to smithi04569269-30 2021-07-31T01:53:33.320 INFO:tasks.rados.rados.0.smithi045.stdout:2371: finishing write tid 1 to smithi04569269-31 2021-07-31T01:53:33.320 INFO:tasks.rados.rados.0.smithi045.stdout:2366: done (5 left) 2021-07-31T01:53:33.320 INFO:tasks.rados.rados.0.smithi045.stdout:2367: done (4 left) 2021-07-31T01:53:33.321 INFO:tasks.rados.rados.0.smithi045.stdout:2368: done (3 left) 2021-07-31T01:53:33.321 INFO:tasks.rados.rados.0.smithi045.stdout:2369: done (2 left) 2021-07-31T01:53:33.321 INFO:tasks.rados.rados.0.smithi045.stdout:2372: done (1 left) 2021-07-31T01:53:33.321 INFO:tasks.rados.rados.0.smithi045.stdout:2373: delete oid 28 current snap is 226 2021-07-31T01:53:33.321 INFO:tasks.rados.rados.0.smithi045.stdout:2371: finishing write tid 2 to smithi04569269-31 2021-07-31T01:53:33.322 INFO:tasks.rados.rados.0.smithi045.stdout:2371: finishing write tid 3 to smithi04569269-31 2021-07-31T01:53:33.322 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 31 v 1548 (ObjNum 726 snap 226 seq_num 726) dirty exists 2021-07-31T01:53:33.322 INFO:tasks.rados.rados.0.smithi045.stdout:2371: left oid 31 (ObjNum 726 snap 226 seq_num 726) 2021-07-31T01:53:33.323 INFO:tasks.rados.rados.0.smithi045.stdout:2371: done (1 left) 2021-07-31T01:53:33.323 INFO:tasks.rados.rados.0.smithi045.stdout:2373: done (0 left) 2021-07-31T01:53:33.323 INFO:tasks.rados.rados.0.smithi045.stdout:2374: snap_remove snap 224 2021-07-31T01:53:33.670 INFO:tasks.rados.rados.0.smithi045.stdout:2374: done (0 left) 2021-07-31T01:53:33.670 INFO:tasks.rados.rados.0.smithi045.stdout:2375: snap_create 2021-07-31T01:53:33.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:53:33 smithi045 conmon[61210]: level=error ts=2021-07-31T01:53:33.520Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:53:33.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:53:33 smithi045 conmon[61210]: level=error ts=2021-07-31T01:53:33.520Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:53:33.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:53:33 smithi045 conmon[61210]: level=error ts=2021-07-31T01:53:33.520Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T01:53:34.676 INFO:tasks.rados.rados.0.smithi045.stdout:2375: done (0 left) 2021-07-31T01:53:34.676 INFO:tasks.rados.rados.0.smithi045.stdout:2376: rmattr oid 49 current snap is 227 2021-07-31T01:53:34.677 INFO:tasks.rados.rados.0.smithi045.stdout:2377: snap_remove snap 225 2021-07-31T01:53:35.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:34 smithi154 conmon[34553]: cluster 2021-07-31T01:53:33.669515+0000 mon.a (mon.0) 2021-07-31T01:53:35.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:34 smithi154 conmon[34553]: 1167 : cluster [DBG] osdmap e512: 8 total, 8 up, 8 in 2021-07-31T01:53:35.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:34 smithi154 conmon[34553]: cluster 2021-07-31T01:53:33.881698+0000 mgr.x (mgr.24542) 273 : cluster [DBG] pgmap v549: 25 pgs: 25 active+clean; 216 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 3.8 MiB/s wr, 11 op/s 2021-07-31T01:53:35.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:34 smithi045 conmon[33655]: cluster 2021-07-31T01:53:33.669515+0000 mon.a (mon.0) 1167 : 2021-07-31T01:53:35.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:34 smithi045 conmon[33655]: cluster [DBG] osdmap e512: 8 total, 8 up, 8 in 2021-07-31T01:53:35.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:34 smithi045 conmon[33655]: cluster 2021-07-31T01:53:33.881698+0000 mgr.x (mgr.24542) 273 : cluster [DBG] pgmap v549: 25 pgs: 25 active+clean; 216 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 3.8 MiB/s wr, 11 op/s 2021-07-31T01:53:35.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:34 smithi045 conmon[39718]: cluster 2021-07-31T01:53:33.669515+0000 mon.a ( 2021-07-31T01:53:35.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:34 smithi045 conmon[39718]: mon.0) 1167 : cluster [DBG] osdmap e512: 8 total, 8 up, 8 in 2021-07-31T01:53:35.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:34 smithi045 conmon[39718]: cluster 2021-07-31T01:53 2021-07-31T01:53:35.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:34 smithi045 conmon[39718]: :33.881698+0000 mgr.x (mgr.24542) 273 : cluster [DBG] pgmap v549: 25 pgs: 25 active+clean; 216 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 3.8 MiB/s wr, 11 op/s 2021-07-31T01:53:35.683 INFO:tasks.rados.rados.0.smithi045.stdout:2377: done (1 left) 2021-07-31T01:53:35.684 INFO:tasks.rados.rados.0.smithi045.stdout:2378: rollback oid 14 current snap is 227 2021-07-31T01:53:35.684 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 14 to 226 2021-07-31T01:53:35.684 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 49 v 1562 (ObjNum 679 snap 212 seq_num 679) dirty exists 2021-07-31T01:53:35.684 INFO:tasks.rados.rados.0.smithi045.stdout:2376: done (1 left) 2021-07-31T01:53:35.685 INFO:tasks.rados.rados.0.smithi045.stdout:2379: delete oid 36 current snap is 227 2021-07-31T01:53:35.691 INFO:tasks.rados.rados.0.smithi045.stdout:2378: finishing rollback tid 0 to smithi04569269-14 2021-07-31T01:53:35.692 INFO:tasks.rados.rados.0.smithi045.stdout:2379: done (1 left) 2021-07-31T01:53:35.693 INFO:tasks.rados.rados.0.smithi045.stdout:2380: copy_from oid 1 from oid 9 current snap is 227 2021-07-31T01:53:35.693 INFO:tasks.rados.rados.0.smithi045.stdout:2381: snap_remove snap 222 2021-07-31T01:53:36.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:35 smithi154 conmon[34553]: cluster 2021-07-31T01:53:34.675296+0000 mon.a (mon.0) 1168 : cluster [DBG] osdmap e513: 8 total, 8 up, 8 in 2021-07-31T01:53:36.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:35 smithi045 conmon[39718]: cluster 2021-07-31T01:53:34.675296+0000 mon.a ( 2021-07-31T01:53:36.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:35 smithi045 conmon[39718]: mon.0) 1168 : cluster [DBG] osdmap e513: 8 total, 8 up, 8 in 2021-07-31T01:53:36.027 INFO:journalctl@ceph.osd.0.smithi045.stdout:Jul 31 01:53:35 smithi045 conmon[43243]: debug 2021-07-31T01:53:35.687+0000 7fdf6d793700 -1 osd.0 pg_epoch: 514 pg[3.6s0( v 511'376 (0'0,511'376] local-lis/les=56/57 n=3 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [0,1,4]p0(0) r=0 lpr=56 crt=511'376 lcod 511'374 mlcod 511'374 active+clean+snaptrim trimq=[1c1~1] ps=[1b7~1,1be~2]] removing snap head 2021-07-31T01:53:36.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:35 smithi045 conmon[33655]: cluster 2021-07-31T01:53:34.675296+0000 mon.a (mon.0) 1168 : cluster [DBG] 2021-07-31T01:53:36.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:35 smithi045 conmon[33655]: osdmap e513: 8 total, 8 up, 8 in 2021-07-31T01:53:36.360 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T01:53:36.592 INFO:tasks.rados.rados.0.smithi045.stdout:2381: done (2 left) 2021-07-31T01:53:36.592 INFO:tasks.rados.rados.0.smithi045.stdout:2382: setattr oid 41 current snap is 227 2021-07-31T01:53:36.593 INFO:tasks.rados.rados.0.smithi045.stdout:2378: finishing rollback tid 1 to smithi04569269-14 2021-07-31T01:53:36.593 INFO:tasks.rados.rados.0.smithi045.stdout:2378: finishing rollback tid 2 to smithi04569269-14 2021-07-31T01:53:36.593 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 14 v 1392 (ObjNum 659 snap 205 seq_num 659) dirty exists 2021-07-31T01:53:36.593 INFO:tasks.rados.rados.0.smithi045.stdout:2380: finishing copy_from to smithi04569269-1 2021-07-31T01:53:36.594 INFO:tasks.rados.rados.0.smithi045.stdout:2380: got expected ENOENT (src dne) 2021-07-31T01:53:36.594 INFO:tasks.rados.rados.0.smithi045.stdout:2380: finishing copy_from racing read to smithi04569269-1 2021-07-31T01:53:36.594 INFO:tasks.rados.rados.0.smithi045.stdout:2378: done (2 left) 2021-07-31T01:53:36.595 INFO:tasks.rados.rados.0.smithi045.stdout:2380: done (1 left) 2021-07-31T01:53:36.595 INFO:tasks.rados.rados.0.smithi045.stdout:2383: snap_remove snap 226 2021-07-31T01:53:36.645 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T01:53:36.645 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 8m ago 9m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T01:53:36.645 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 6m ago 8m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T01:53:36.646 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 6m ago 12m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T01:53:36.646 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 8m ago 14m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T01:53:36.646 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 8m ago 14m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T01:53:36.646 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 6m ago 12m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T01:53:36.647 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 8m ago 12m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T01:53:36.647 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 8m ago 9m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T01:53:36.648 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 6m ago 9m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T01:53:36.648 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 8m ago 12m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T01:53:36.648 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 8m ago 11m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T01:53:36.648 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 8m ago 11m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T01:53:36.649 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 8m ago 11m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T01:53:36.649 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 6m ago 10m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T01:53:36.649 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 6m ago 10m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T01:53:36.649 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 6m ago 10m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T01:53:36.649 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 6m ago 9m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T01:53:36.650 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 6m ago 9m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T01:53:36.987 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T01:53:36.987 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T01:53:36.987 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T01:53:36.988 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T01:53:36.988 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T01:53:36.988 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T01:53:36.988 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T01:53:36.988 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T01:53:36.989 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T01:53:36.989 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T01:53:36.990 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T01:53:36.990 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T01:53:36.990 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T01:53:36.990 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T01:53:36.991 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T01:53:36.991 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T01:53:36.991 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T01:53:37.002 INFO:journalctl@ceph.osd.7.smithi154.stdout:Jul 31 01:53:36 smithi154 conmon[49149]: debug 2021-07-31T01:53:36.596+0000 7f8a75a04700 -1 osd.7 pg_epoch: 515 pg[3.ds0( v 514'1092 (0'0,514'1092] local-lis/les=56/57 n=9 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [7,5,3]p7(0) r=0 lpr=56 crt=514'1092 lcod 512'1090 mlcod 512'1090 active+clean+snaptrim trimq=[1bd~1] ps=[1c1~1]] removing snap head 2021-07-31T01:53:37.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:36 smithi154 conmon[34553]: cluster 2021-07-31T01:53:35.682712+0000 mon.a (mon.0) 1169 : cluster [DBG] osdmap e514: 8 total, 8 up, 8 in 2021-07-31T01:53:37.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:36 smithi154 conmon[34553]: cluster 2021-07-31T01:53:35.882188+0000 mgr.x (mgr.24542) 274 : cluster [DBG] pgmap v552: 25 pgs: 25 active+clean; 216 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 3.8 MiB/s wr, 11 op/s 2021-07-31T01:53:37.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:36 smithi154 conmon[34553]: cluster 2021-07-31T01:53:36.591511+0000 mon.a (mon.0) 1170 : cluster [DBG] osdmap e515: 8 total, 8 up, 8 in 2021-07-31T01:53:37.004 INFO:journalctl@ceph.osd.5.smithi154.stdout:Jul 31 01:53:36 smithi154 conmon[42050]: debug 2021-07-31T01:53:36.609+0000 7fc9b105c700 -1 osd.5 pg_epoch: 515 pg[3.cs0( v 515'968 (0'0,515'968] local-lis/les=56/57 n=5 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [5,3,6]p5(0) r=0 lpr=56 crt=515'968 lcod 515'966 mlcod 515'966 active+clean+snaptrim trimq=[1bd~1] ps=[1c1~1]] removing snap head 2021-07-31T01:53:37.026 INFO:journalctl@ceph.osd.3.smithi045.stdout:Jul 31 01:53:36 smithi045 conmon[54013]: debug 2021-07-31T01:53:37.026 INFO:journalctl@ceph.osd.3.smithi045.stdout:Jul 31 01:53:36 smithi045 conmon[54013]: 2021-07-31T01:53:36.602+0000 7f64fc8cd700 -1 osd.3 pg_epoch: 515 pg[3.bs0( v 514'1566 (0'0,514'1566] local-lis/les=56/57 n=14 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [3,0,4]p3(0) r=0 lpr=56 crt=514'1566 lcod 514'1564 mlcod 514'1564 active+clean+snaptrim trimq=[1bd~1] ps=[1c1~1]] removing snap head 2021-07-31T01:53:37.026 INFO:journalctl@ceph.osd.3.smithi045.stdout:Jul 31 01:53:36 smithi045 conmon[54013]: debug 2021-07-31T01:53:36.609+0000 7f64f88c5700 -1 osd.3 pg_epoch: 515 pg[3.bs0( v 515'1570 (0'0,515'1570] local-lis/les=56/57 n=12 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [3,0,4]p3(0) r=0 lpr=56 crt=515'1570 lcod 515'1568 mlcod 515'1568 active+clean+snaptrim trimq=[1bd~1] ps=[1c1~1]] removing snap head 2021-07-31T01:53:37.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:36 smithi045 conmon[39718]: cluster 2021-07-31T01:53:35.682712+0000 2021-07-31T01:53:37.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:36 smithi045 conmon[39718]: mon.a (mon.0) 1169 : cluster [DBG] osdmap e514: 8 total, 8 up, 8 in 2021-07-31T01:53:37.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:36 smithi045 conmon[39718]: cluster 2021-07-31T01:53:37.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:36 smithi045 conmon[39718]: 2021-07-31T01:53:35.882188+0000 mgr.x (mgr. 2021-07-31T01:53:37.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:36 smithi045 conmon[39718]: 24542) 274 : cluster [DBG] pgmap v552: 25 pgs: 25 active+clean; 216 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 3.8 MiB/s wr, 11 op/s 2021-07-31T01:53:37.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:36 smithi045 conmon[39718]: cluster 2021 2021-07-31T01:53:37.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:36 smithi045 conmon[39718]: -07-31T01:53:36.591511+0000 2021-07-31T01:53:37.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:36 smithi045 conmon[39718]: mon.a (mon.0) 1170 : cluster [DBG] osdmap e515: 8 total, 8 up, 8 in 2021-07-31T01:53:37.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:36 smithi045 conmon[39718]: 2021-07-31T01:53:37.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:36 smithi045 conmon[33655]: cluster 2021-07-31T01:53:35.682712+0000 2021-07-31T01:53:37.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:36 smithi045 conmon[33655]: mon.a (mon.0) 1169 : cluster [DBG] osdmap e514: 8 total, 8 up, 8 in 2021-07-31T01:53:37.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:36 smithi045 conmon[33655]: cluster 2021-07-31T01:53:35.882188+0000 mgr.x (mgr. 2021-07-31T01:53:37.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:36 smithi045 conmon[33655]: 24542) 274 : cluster [DBG] pgmap v552: 25 pgs: 25 active+clean; 216 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 3.8 MiB/s wr, 11 op/s 2021-07-31T01:53:37.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:36 smithi045 conmon[33655]: cluster 2021-07-31 2021-07-31T01:53:37.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:36 smithi045 conmon[33655]: T01:53:36.591511+0000 mon.a (mon.0) 1170 : cluster [DBG] osdmap e515: 8 total, 8 up, 8 in 2021-07-31T01:53:37.597 INFO:tasks.rados.rados.0.smithi045.stdout:2383: done (1 left) 2021-07-31T01:53:37.598 INFO:tasks.rados.rados.0.smithi045.stdout:2384: delete oid 35 current snap is 227 2021-07-31T01:53:37.598 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 41 v 993 (ObjNum 714 snap 222 seq_num 714) dirty exists 2021-07-31T01:53:37.603 INFO:tasks.rados.rados.0.smithi045.stdout:2382: done (1 left) 2021-07-31T01:53:37.604 INFO:tasks.rados.rados.0.smithi045.stdout:2384: done (0 left) 2021-07-31T01:53:37.604 INFO:tasks.rados.rados.0.smithi045.stdout:2385: copy_from oid 14 from oid 22 current snap is 227 2021-07-31T01:53:37.604 INFO:tasks.rados.rados.0.smithi045.stdout:2386: delete oid 40 current snap is 227 2021-07-31T01:53:37.606 INFO:tasks.rados.rados.0.smithi045.stdout:2386: done (1 left) 2021-07-31T01:53:37.607 INFO:tasks.rados.rados.0.smithi045.stdout:2387: delete oid 21 current snap is 227 2021-07-31T01:53:37.610 INFO:tasks.rados.rados.0.smithi045.stdout:2387: done (1 left) 2021-07-31T01:53:37.610 INFO:tasks.rados.rados.0.smithi045.stdout:2388: rmattr oid 28 current snap is 227 2021-07-31T01:53:37.610 INFO:tasks.rados.rados.0.smithi045.stdout:2388: done (1 left) 2021-07-31T01:53:37.610 INFO:tasks.rados.rados.0.smithi045.stdout:2389: read oid 7 snap -1 2021-07-31T01:53:37.611 INFO:tasks.rados.rados.0.smithi045.stdout:2389: expect (ObjNum 618 snap 189 seq_num 618) 2021-07-31T01:53:37.611 INFO:tasks.rados.rados.0.smithi045.stdout:2390: delete oid 18 current snap is 227 2021-07-31T01:53:37.611 INFO:tasks.rados.rados.0.smithi045.stdout:2385: finishing copy_from to smithi04569269-14 2021-07-31T01:53:37.612 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 14 v 1025 (ObjNum 575 snap 175 seq_num 575) dirty exists 2021-07-31T01:53:37.613 INFO:tasks.rados.rados.0.smithi045.stdout:2385: finishing copy_from racing read to smithi04569269-14 2021-07-31T01:53:37.613 INFO:tasks.rados.rados.0.smithi045.stdout:2385: done (2 left) 2021-07-31T01:53:37.613 INFO:tasks.rados.rados.0.smithi045.stdout:2390: done (1 left) 2021-07-31T01:53:37.614 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 48 current snap is 227 2021-07-31T01:53:37.614 INFO:tasks.rados.rados.0.smithi045.stdout:2391: seq_num 734 ranges {2220032=294912} 2021-07-31T01:53:37.627 INFO:tasks.rados.rados.0.smithi045.stdout:2391: writing smithi04569269-48 from 2220032 to 2514944 tid 1 2021-07-31T01:53:37.627 INFO:tasks.rados.rados.0.smithi045.stdout:2392: setattr oid 47 current snap is 227 2021-07-31T01:53:37.629 INFO:tasks.rados.rados.0.smithi045.stdout:2393: snap_create 2021-07-31T01:53:37.630 INFO:tasks.rados.rados.0.smithi045.stdout:2391: finishing write tid 1 to smithi04569269-48 2021-07-31T01:53:37.631 INFO:tasks.rados.rados.0.smithi045.stdout:2391: finishing write tid 2 to smithi04569269-48 2021-07-31T01:53:37.632 INFO:tasks.rados.rados.0.smithi045.stdout:2391: finishing write tid 3 to smithi04569269-48 2021-07-31T01:53:37.633 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 48 v 1063 (ObjNum 734 snap 227 seq_num 734) dirty exists 2021-07-31T01:53:37.633 INFO:tasks.rados.rados.0.smithi045.stdout:2391: left oid 48 (ObjNum 734 snap 227 seq_num 734) 2021-07-31T01:53:37.634 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 47 v 341 (ObjNum 721 snap 223 seq_num 721) dirty exists 2021-07-31T01:53:37.639 INFO:tasks.rados.rados.0.smithi045.stdout:2389: expect (ObjNum 618 snap 189 seq_num 618) 2021-07-31T01:53:38.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:37 smithi154 conmon[34553]: audit 2021-07-31T01:53:36.347445+0000 mgr.x (mgr.24542) 275 : audit [DBG] 2021-07-31T01:53:38.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:37 smithi154 conmon[34553]: from='client.24835 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:53:38.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:37 smithi154 conmon[34553]: audit 2021-07-31T01:53:36.636693+0000 mgr.x (mgr.24542) 276 : audit [DBG] from='client.14934 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:53:38.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:37 smithi154 conmon[34553]: audit 2021-07-31T01:53:36.986433+0000 mon.a (mon.0) 1171 : audit [DBG] from='client.? 172.21.15.45:0/3435614656' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T01:53:38.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:37 smithi154 conmon[34553]: cluster 2021-07-31T01:53:37.597258+0000 mon.a (mon.0) 1172 : cluster [DBG] osdmap e516: 8 total, 8 up, 8 in 2021-07-31T01:53:38.026 INFO:journalctl@ceph.osd.3.smithi045.stdout:Jul 31 01:53:37 smithi045 conmon[54013]: debug 2021-07-31T01:53:37.601+0000 7f64f88c5700 -1 osd.3 pg_epoch: 516 pg[3.bs0( v 516'1579 (0'0,516'1579] local-lis/les=56/57 n=7 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [3,0,4]p3(0) r=0 lpr=56 luod=515'1576 crt=515'1576 lcod 515'1574 mlcod 515'1574 active+clean+snaptrim trimq=[1c4~1] ps=[1bd~1,1c1~1]] removing snap head 2021-07-31T01:53:38.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:37 smithi045 conmon[39718]: audit 2021-07-31T01:53:36 2021-07-31T01:53:38.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:37 smithi045 conmon[39718]: .347445+0000 mgr.x (mgr.24542) 275 : audit [DBG] from='client.24835 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:53:38.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:37 smithi045 conmon[39718]: audit 2021-07-31T01:53:36 2021-07-31T01:53:38.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:37 smithi045 conmon[39718]: .636693+0000 mgr.x (mgr.24542) 276 : 2021-07-31T01:53:38.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:37 smithi045 conmon[39718]: audit [DBG] from='client.14934 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:53:38.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:37 smithi045 conmon[39718]: audit 2021- 2021-07-31T01:53:38.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:37 smithi045 conmon[39718]: 07-31T01:53:36.986433+0000 mon.a (mon 2021-07-31T01:53:38.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:37 smithi045 conmon[39718]: .0) 1171 : audit [DBG] from='client.? 172.21.15.45:0/3435614656' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T01:53:38.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:37 smithi045 conmon[39718]: cluster 2021-07 2021-07-31T01:53:38.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:37 smithi045 conmon[39718]: -31T01:53:37.597258+0000 mon.a (mon 2021-07-31T01:53:38.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:37 smithi045 conmon[39718]: .0) 1172 : cluster [DBG] osdmap e516: 8 total, 8 up, 8 in 2021-07-31T01:53:38.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:37 smithi045 conmon[33655]: audit 2021-07-31T01:53:36.347445+0000 mgr.x ( 2021-07-31T01:53:38.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:37 smithi045 conmon[33655]: mgr.24542) 275 : audit [DBG] from='client.24835 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:53:38.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:37 smithi045 conmon[33655]: audit 2021-07-31T01:53:36.636693+0000 mgr.x (mgr 2021-07-31T01:53:38.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:37 smithi045 conmon[33655]: .24542) 276 : audit [DBG] from='client.14934 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:53:38.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:37 smithi045 conmon[33655]: audit 2021- 2021-07-31T01:53:38.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:37 smithi045 conmon[33655]: 07-31T01:53:36.986433+0000 mon.a (mon.0) 1171 : audit [DBG] from='client.? 172.21.15.45:0/3435614656' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T01:53:38.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:37 smithi045 conmon[33655]: cluster 2021-07-31T01:53:37. 2021-07-31T01:53:38.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:37 smithi045 conmon[33655]: 597258+0000 mon.a (mon.0) 1172 : cluster [DBG] osdmap e516: 8 total, 8 up, 8 in 2021-07-31T01:53:38.599 INFO:tasks.rados.rados.0.smithi045.stdout:2389: done (3 left) 2021-07-31T01:53:38.600 INFO:tasks.rados.rados.0.smithi045.stdout:2391: done (2 left) 2021-07-31T01:53:38.600 INFO:tasks.rados.rados.0.smithi045.stdout:2392: done (1 left) 2021-07-31T01:53:38.600 INFO:tasks.rados.rados.0.smithi045.stdout:2393: done (0 left) 2021-07-31T01:53:38.600 INFO:tasks.rados.rados.0.smithi045.stdout:2394: delete oid 28 current snap is 228 2021-07-31T01:53:38.603 INFO:tasks.rados.rados.0.smithi045.stdout:2394: done (0 left) 2021-07-31T01:53:38.603 INFO:tasks.rados.rados.0.smithi045.stdout:2395: delete oid 19 current snap is 228 2021-07-31T01:53:38.609 INFO:tasks.rados.rados.0.smithi045.stdout:2395: done (0 left) 2021-07-31T01:53:38.610 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 18 current snap is 228 2021-07-31T01:53:38.610 INFO:tasks.rados.rados.0.smithi045.stdout:2396: seq_num 737 ranges {0=352256} 2021-07-31T01:53:38.611 INFO:tasks.rados.rados.0.smithi045.stdout:2396: writing smithi04569269-18 from 0 to 352256 tid 1 2021-07-31T01:53:38.611 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 14 current snap is 228 2021-07-31T01:53:38.612 INFO:tasks.rados.rados.0.smithi045.stdout:2397: seq_num 738 ranges {32768=368640} 2021-07-31T01:53:38.613 INFO:tasks.rados.rados.0.smithi045.stdout:2397: writing smithi04569269-14 from 32768 to 401408 tid 1 2021-07-31T01:53:38.613 INFO:tasks.rados.rados.0.smithi045.stdout:2398: delete oid 35 current snap is 228 2021-07-31T01:53:38.614 INFO:tasks.rados.rados.0.smithi045.stdout:2396: finishing write tid 1 to smithi04569269-18 2021-07-31T01:53:38.614 INFO:tasks.rados.rados.0.smithi045.stdout:2396: finishing write tid 2 to smithi04569269-18 2021-07-31T01:53:38.616 INFO:tasks.rados.rados.0.smithi045.stdout:2396: finishing write tid 3 to smithi04569269-18 2021-07-31T01:53:38.617 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 18 v 998 (ObjNum 737 snap 228 seq_num 737) dirty exists 2021-07-31T01:53:38.617 INFO:tasks.rados.rados.0.smithi045.stdout:2396: left oid 18 (ObjNum 737 snap 228 seq_num 737) 2021-07-31T01:53:38.617 INFO:tasks.rados.rados.0.smithi045.stdout:2396: done (2 left) 2021-07-31T01:53:38.618 INFO:tasks.rados.rados.0.smithi045.stdout:2398: done (1 left) 2021-07-31T01:53:38.618 INFO:tasks.rados.rados.0.smithi045.stdout:2399: setattr oid 35 current snap is 228 2021-07-31T01:53:38.620 INFO:tasks.rados.rados.0.smithi045.stdout:2397: finishing write tid 1 to smithi04569269-14 2021-07-31T01:53:38.621 INFO:tasks.rados.rados.0.smithi045.stdout:2397: finishing write tid 2 to smithi04569269-14 2021-07-31T01:53:38.621 INFO:tasks.rados.rados.0.smithi045.stdout:2400: copy_from oid 26 from oid 1 current snap is 228 2021-07-31T01:53:38.622 INFO:tasks.rados.rados.0.smithi045.stdout:2401: copy_from oid 17 from oid 29 current snap is 228 2021-07-31T01:53:38.622 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 27 current snap is 228 2021-07-31T01:53:38.622 INFO:tasks.rados.rados.0.smithi045.stdout:2402: seq_num 740 ranges {5283840=401408} 2021-07-31T01:53:38.651 INFO:tasks.rados.rados.0.smithi045.stdout:2402: writing smithi04569269-27 from 5283840 to 5685248 tid 1 2021-07-31T01:53:38.652 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 2 current snap is 228 2021-07-31T01:53:38.652 INFO:tasks.rados.rados.0.smithi045.stdout:2397: finishing write tid 3 to smithi04569269-14 2021-07-31T01:53:38.652 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 14 v 1394 (ObjNum 738 snap 228 seq_num 738) dirty exists 2021-07-31T01:53:38.652 INFO:tasks.rados.rados.0.smithi045.stdout:2397: left oid 14 (ObjNum 738 snap 228 seq_num 738) 2021-07-31T01:53:38.653 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 35 v 1584 (ObjNum 0 snap 0 seq_num 0) dirty exists 2021-07-31T01:53:38.653 INFO:tasks.rados.rados.0.smithi045.stdout:2400: finishing copy_from to smithi04569269-26 2021-07-31T01:53:38.653 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 26 v 1061 (ObjNum 713 snap 222 seq_num 713) dirty exists 2021-07-31T01:53:38.653 INFO:tasks.rados.rados.0.smithi045.stdout:2400: finishing copy_from racing read to smithi04569269-26 2021-07-31T01:53:38.654 INFO:tasks.rados.rados.0.smithi045.stdout:2403: seq_num 741 ranges {6938624=417792} 2021-07-31T01:53:38.693 INFO:tasks.rados.rados.0.smithi045.stdout:2403: writing smithi04569269-2 from 6938624 to 7356416 tid 1 2021-07-31T01:53:38.694 INFO:tasks.rados.rados.0.smithi045.stdout:2397: done (5 left) 2021-07-31T01:53:38.694 INFO:tasks.rados.rados.0.smithi045.stdout:2399: done (4 left) 2021-07-31T01:53:38.695 INFO:tasks.rados.rados.0.smithi045.stdout:2400: done (3 left) 2021-07-31T01:53:38.697 INFO:tasks.rados.rados.0.smithi045.stdout:2404: read oid 45 snap -1 2021-07-31T01:53:38.697 INFO:tasks.rados.rados.0.smithi045.stdout:2404: expect (ObjNum 483 snap 143 seq_num 483) 2021-07-31T01:53:38.698 INFO:tasks.rados.rados.0.smithi045.stdout:2405: rollback oid 10 current snap is 228 2021-07-31T01:53:38.699 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 10 to 227 2021-07-31T01:53:38.700 INFO:tasks.rados.rados.0.smithi045.stdout:2402: finishing write tid 1 to smithi04569269-27 2021-07-31T01:53:38.701 INFO:tasks.rados.rados.0.smithi045.stdout:2402: finishing write tid 2 to smithi04569269-27 2021-07-31T01:53:38.702 INFO:tasks.rados.rados.0.smithi045.stdout:2401: finishing copy_from to smithi04569269-17 2021-07-31T01:53:38.702 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 17 v 1045 (ObjNum 642 snap 199 seq_num 642) dirty exists 2021-07-31T01:53:38.702 INFO:tasks.rados.rados.0.smithi045.stdout:2402: finishing write tid 3 to smithi04569269-27 2021-07-31T01:53:38.703 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 27 v 1396 (ObjNum 740 snap 228 seq_num 740) dirty exists 2021-07-31T01:53:38.703 INFO:tasks.rados.rados.0.smithi045.stdout:2402: left oid 27 (ObjNum 740 snap 228 seq_num 740) 2021-07-31T01:53:38.703 INFO:tasks.rados.rados.0.smithi045.stdout:2401: finishing copy_from racing read to smithi04569269-17 2021-07-31T01:53:38.703 INFO:tasks.rados.rados.0.smithi045.stdout:2401: done (4 left) 2021-07-31T01:53:38.703 INFO:tasks.rados.rados.0.smithi045.stdout:2402: done (3 left) 2021-07-31T01:53:38.704 INFO:tasks.rados.rados.0.smithi045.stdout:2406: read oid 15 snap -1 2021-07-31T01:53:38.704 INFO:tasks.rados.rados.0.smithi045.stdout:2406: expect (ObjNum 677 snap 211 seq_num 677) 2021-07-31T01:53:38.704 INFO:tasks.rados.rados.0.smithi045.stdout:2407: rollback oid 33 current snap is 228 2021-07-31T01:53:38.704 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 33 to 227 2021-07-31T01:53:38.705 INFO:tasks.rados.rados.0.smithi045.stdout:2408: snap_create 2021-07-31T01:53:38.706 INFO:tasks.rados.rados.0.smithi045.stdout:2407: finishing rollback tid 0 to smithi04569269-33 2021-07-31T01:53:38.706 INFO:tasks.rados.rados.0.smithi045.stdout:2407: finishing rollback tid 1 to smithi04569269-33 2021-07-31T01:53:38.706 INFO:tasks.rados.rados.0.smithi045.stdout:2407: finishing rollback tid 2 to smithi04569269-33 2021-07-31T01:53:38.706 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 33 v 1175 (ObjNum 659 snap 205 seq_num 659) dirty exists 2021-07-31T01:53:38.706 INFO:tasks.rados.rados.0.smithi045.stdout:2404: expect (ObjNum 483 snap 143 seq_num 483) 2021-07-31T01:53:38.755 INFO:tasks.rados.rados.0.smithi045.stdout:2405: finishing rollback tid 0 to smithi04569269-10 2021-07-31T01:53:38.756 INFO:tasks.rados.rados.0.smithi045.stdout:2405: finishing rollback tid 1 to smithi04569269-10 2021-07-31T01:53:38.756 INFO:tasks.rados.rados.0.smithi045.stdout:2405: finishing rollback tid 2 to smithi04569269-10 2021-07-31T01:53:38.756 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 10 v 1183 (ObjNum 722 snap 224 seq_num 722) dirty exists 2021-07-31T01:53:38.756 INFO:tasks.rados.rados.0.smithi045.stdout:2403: finishing write tid 1 to smithi04569269-2 2021-07-31T01:53:38.756 INFO:tasks.rados.rados.0.smithi045.stdout:2403: finishing write tid 2 to smithi04569269-2 2021-07-31T01:53:38.757 INFO:tasks.rados.rados.0.smithi045.stdout:2403: finishing write tid 3 to smithi04569269-2 2021-07-31T01:53:38.758 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 2 v 1063 (ObjNum 741 snap 228 seq_num 741) dirty exists 2021-07-31T01:53:38.758 INFO:tasks.rados.rados.0.smithi045.stdout:2403: left oid 2 (ObjNum 741 snap 228 seq_num 741) 2021-07-31T01:53:38.759 INFO:tasks.rados.rados.0.smithi045.stdout:2406: expect (ObjNum 677 snap 211 seq_num 677) 2021-07-31T01:53:39.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:38 smithi154 conmon[34553]: cluster 2021-07-31T01:53:37.882736+0000 mgr.x (mgr. 2021-07-31T01:53:39.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:38 smithi154 conmon[34553]: 24542) 277 : cluster [DBG] pgmap v555: 25 pgs: 25 active+clean; 216 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail 2021-07-31T01:53:39.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:38 smithi154 conmon[34553]: cluster 2021-07-31T01:53:38.598774+0000 mon.a (mon.0) 1173 : cluster [DBG] osdmap e517: 8 total, 8 up, 8 in 2021-07-31T01:53:39.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:38 smithi045 conmon[33655]: cluster 2021-07-31T01:53:37.882736 2021-07-31T01:53:39.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:38 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 277 : cluster [DBG] pgmap v555: 25 pgs: 25 active+clean; 216 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail 2021-07-31T01:53:39.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:38 smithi045 conmon[33655]: cluster 2021-07-31T01:53:38.598774+0000 mon.a (mon.0) 1173 : cluster [DBG] osdmap e517: 8 total, 8 up, 8 in 2021-07-31T01:53:39.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:38 smithi045 conmon[39718]: cluster 2021-07-31T01:53:37.882736+0000 mgr.x (mgr.24542) 277 : cluster [DBG] pgmap v555: 25 pgs: 25 active+clean; 216 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail 2021-07-31T01:53:39.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:38 smithi045 conmon[39718]: cluster 2021-07-31T01:53:38.598774+0000 mon.a (mon.0) 1173 : cluster [DBG] osdmap e517: 8 total, 8 up, 8 in 2021-07-31T01:53:39.696 INFO:tasks.rados.rados.0.smithi045.stdout:2403: done (5 left) 2021-07-31T01:53:39.696 INFO:tasks.rados.rados.0.smithi045.stdout:2404: done (4 left) 2021-07-31T01:53:39.697 INFO:tasks.rados.rados.0.smithi045.stdout:2405: done (3 left) 2021-07-31T01:53:39.697 INFO:tasks.rados.rados.0.smithi045.stdout:2406: done (2 left) 2021-07-31T01:53:39.697 INFO:tasks.rados.rados.0.smithi045.stdout:2407: done (1 left) 2021-07-31T01:53:39.697 INFO:tasks.rados.rados.0.smithi045.stdout:2408: done (0 left) 2021-07-31T01:53:39.698 INFO:tasks.rados.rados.0.smithi045.stdout:2409: read oid 49 snap -1 2021-07-31T01:53:39.698 INFO:tasks.rados.rados.0.smithi045.stdout:2409: expect (ObjNum 679 snap 212 seq_num 679) 2021-07-31T01:53:39.698 INFO:tasks.rados.rados.0.smithi045.stdout:2410: read oid 19 snap -1 2021-07-31T01:53:39.698 INFO:tasks.rados.rados.0.smithi045.stdout:2410: expect deleted 2021-07-31T01:53:39.699 INFO:tasks.rados.rados.0.smithi045.stdout:2411: snap_create 2021-07-31T01:53:39.725 INFO:tasks.rados.rados.0.smithi045.stdout:2409: expect (ObjNum 679 snap 212 seq_num 679) 2021-07-31T01:53:40.702 INFO:tasks.rados.rados.0.smithi045.stdout:2409: done (2 left) 2021-07-31T01:53:40.702 INFO:tasks.rados.rados.0.smithi045.stdout:2410: done (1 left) 2021-07-31T01:53:40.702 INFO:tasks.rados.rados.0.smithi045.stdout:2411: done (0 left) 2021-07-31T01:53:40.703 INFO:tasks.rados.rados.0.smithi045.stdout:2412: rollback oid 30 current snap is 230 2021-07-31T01:53:40.703 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 30 to 229 2021-07-31T01:53:40.703 INFO:tasks.rados.rados.0.smithi045.stdout:2413: copy_from oid 10 from oid 49 current snap is 230 2021-07-31T01:53:40.703 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 50 current snap is 230 2021-07-31T01:53:40.704 INFO:tasks.rados.rados.0.smithi045.stdout:2414: seq_num 742 ranges {2670592=434176} 2021-07-31T01:53:40.719 INFO:tasks.rados.rados.0.smithi045.stdout:2414: writing smithi04569269-50 from 2670592 to 3104768 tid 1 2021-07-31T01:53:40.719 INFO:tasks.rados.rados.0.smithi045.stdout:2415: snap_remove snap 228 2021-07-31T01:53:41.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:40 smithi154 conmon[34553]: cluster 2021-07-31T01:53:39.695220+0000 mon.a (mon.0) 1174 : cluster [DBG] osdmap e518: 8 total, 8 up, 8 in 2021-07-31T01:53:41.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:40 smithi154 conmon[34553]: cluster 2021-07-31T01:53:39.883303+0000 mgr.x (mgr.24542) 278 : cluster [DBG] pgmap v558: 25 pgs: 25 active+clean; 194 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 3.0 MiB/s wr, 26 op/s 2021-07-31T01:53:41.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:40 smithi045 conmon[33655]: cluster 2021-07-31T01:53:39.695220+0000 mon.a (mon 2021-07-31T01:53:41.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:40 smithi045 conmon[33655]: .0) 1174 : cluster [DBG] osdmap e518: 8 total, 8 up, 8 in 2021-07-31T01:53:41.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:40 smithi045 conmon[33655]: cluster 2021-07-31T01:53:39.883303+0000 mgr.x 2021-07-31T01:53:41.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:40 smithi045 conmon[33655]: (mgr.24542) 278 : cluster [DBG] pgmap v558: 25 pgs: 25 active+clean; 194 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 3.0 MiB/s wr, 26 op/s 2021-07-31T01:53:41.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:40 smithi045 conmon[39718]: cluster 2021-07-31T01:53:39.695220+0000 2021-07-31T01:53:41.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:40 smithi045 conmon[39718]: mon.a (mon.0) 1174 : cluster [DBG] osdmap e518: 8 total, 8 up, 8 in 2021-07-31T01:53:41.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:40 smithi045 conmon[39718]: cluster 2021-07-31T01:53:41.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:40 smithi045 conmon[39718]: 2021-07-31T01:53:39. 2021-07-31T01:53:41.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:40 smithi045 conmon[39718]: 883303+0000 mgr.x (mgr.24542) 278 : 2021-07-31T01:53:41.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:40 smithi045 conmon[39718]: cluster [DBG] pgmap v558: 25 pgs: 25 active+clean; 194 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 3.0 MiB/s wr, 26 op/s 2021-07-31T01:53:41.594 INFO:tasks.rados.rados.0.smithi045.stdout:2415: done (3 left) 2021-07-31T01:53:41.594 INFO:tasks.rados.rados.0.smithi045.stdout:2416: delete oid 33 current snap is 230 2021-07-31T01:53:41.595 INFO:tasks.rados.rados.0.smithi045.stdout:2412: finishing rollback tid 0 to smithi04569269-30 2021-07-31T01:53:41.595 INFO:tasks.rados.rados.0.smithi045.stdout:2412: finishing rollback tid 1 to smithi04569269-30 2021-07-31T01:53:41.595 INFO:tasks.rados.rados.0.smithi045.stdout:2412: finishing rollback tid 2 to smithi04569269-30 2021-07-31T01:53:41.595 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 30 v 563 (ObjNum 678 snap 211 seq_num 678) dirty exists 2021-07-31T01:53:41.595 INFO:tasks.rados.rados.0.smithi045.stdout:2414: finishing write tid 1 to smithi04569269-50 2021-07-31T01:53:41.596 INFO:tasks.rados.rados.0.smithi045.stdout:2414: finishing write tid 2 to smithi04569269-50 2021-07-31T01:53:41.596 INFO:tasks.rados.rados.0.smithi045.stdout:2414: finishing write tid 3 to smithi04569269-50 2021-07-31T01:53:41.596 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 50 v 1185 (ObjNum 742 snap 230 seq_num 742) dirty exists 2021-07-31T01:53:41.596 INFO:tasks.rados.rados.0.smithi045.stdout:2414: left oid 50 (ObjNum 742 snap 230 seq_num 742) 2021-07-31T01:53:41.597 INFO:tasks.rados.rados.0.smithi045.stdout:2413: finishing copy_from to smithi04569269-10 2021-07-31T01:53:41.597 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 10 v 1562 (ObjNum 679 snap 212 seq_num 679) dirty exists 2021-07-31T01:53:41.597 INFO:tasks.rados.rados.0.smithi045.stdout:2413: finishing copy_from racing read to smithi04569269-10 2021-07-31T01:53:41.603 INFO:tasks.rados.rados.0.smithi045.stdout:2412: done (3 left) 2021-07-31T01:53:41.603 INFO:tasks.rados.rados.0.smithi045.stdout:2413: done (2 left) 2021-07-31T01:53:41.603 INFO:tasks.rados.rados.0.smithi045.stdout:2414: done (1 left) 2021-07-31T01:53:41.603 INFO:tasks.rados.rados.0.smithi045.stdout:2416: done (0 left) 2021-07-31T01:53:41.604 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 43 current snap is 230 2021-07-31T01:53:41.604 INFO:tasks.rados.rados.0.smithi045.stdout:2417: seq_num 744 ranges {7913472=466944} 2021-07-31T01:53:41.649 INFO:tasks.rados.rados.0.smithi045.stdout:2417: writing smithi04569269-43 from 7913472 to 8380416 tid 1 2021-07-31T01:53:41.649 INFO:tasks.rados.rados.0.smithi045.stdout:2418: snap_remove snap 229 2021-07-31T01:53:42.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:41 smithi154 conmon[34553]: cluster 2021-07-31T01:53:40.700876+0000 mon.a (mon.0) 1175 : cluster [DBG] osdmap e519: 8 total, 8 up, 8 in 2021-07-31T01:53:42.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:41 smithi154 conmon[34553]: 2021-07-31T01:53:42.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:41 smithi154 conmon[34553]: cluster 2021-07-31T01:53:41.593414+0000 mon.a (mon.0) 1176 : cluster [DBG] osdmap e520: 8 total, 8 up, 8 in 2021-07-31T01:53:42.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:41 smithi045 conmon[39718]: cluster 2021-07-31T01:53:40.700876+0000 2021-07-31T01:53:42.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:41 smithi045 conmon[39718]: mon.a (mon.0) 1175 : cluster [DBG] osdmap e519: 8 total, 8 up, 8 in 2021-07-31T01:53:42.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:41 smithi045 conmon[39718]: cluster 2021- 2021-07-31T01:53:42.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:41 smithi045 conmon[39718]: 07-31T01:53:41.593414+0000 mon.a 2021-07-31T01:53:42.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:41 smithi045 conmon[39718]: (mon.0) 1176 : cluster 2021-07-31T01:53:42.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:41 smithi045 conmon[39718]: [DBG] osdmap e520: 8 total, 8 up, 8 in 2021-07-31T01:53:42.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:41 smithi045 conmon[33655]: cluster 2021-07-31T01:53:40.700876+0000 2021-07-31T01:53:42.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:41 smithi045 conmon[33655]: mon.a (mon.0) 1175 : cluster [DBG] osdmap e519: 8 total, 8 up, 8 in 2021-07-31T01:53:42.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:41 smithi045 conmon[33655]: cluster 2021-07-31T01:53:41.593414+0000 mon.a (mon.0) 1176 : cluster 2021-07-31T01:53:42.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:41 smithi045 conmon[33655]: [DBG] osdmap e520: 8 total, 8 up, 8 in 2021-07-31T01:53:42.602 INFO:tasks.rados.rados.0.smithi045.stdout:2418: done (1 left) 2021-07-31T01:53:42.602 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 21 current snap is 230 2021-07-31T01:53:42.602 INFO:tasks.rados.rados.0.smithi045.stdout:2417: finishing write tid 1 to smithi04569269-43 2021-07-31T01:53:42.603 INFO:tasks.rados.rados.0.smithi045.stdout:2419: seq_num 745 ranges {0=483328} 2021-07-31T01:53:42.604 INFO:tasks.rados.rados.0.smithi045.stdout:2419: writing smithi04569269-21 from 0 to 483328 tid 1 2021-07-31T01:53:42.604 INFO:tasks.rados.rados.0.smithi045.stdout:2420: rmattr oid 39 current snap is 230 2021-07-31T01:53:42.605 INFO:tasks.rados.rados.0.smithi045.stdout:2420: done (2 left) 2021-07-31T01:53:42.605 INFO:tasks.rados.rados.0.smithi045.stdout:2421: snap_remove snap 227 2021-07-31T01:53:43.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:42 smithi154 conmon[34553]: cluster 2021-07-31T01:53:41.883730+0000 mgr.x (mgr.24542) 279 : cluster [DBG] pgmap v561: 25 pgs: 25 active+clean; 194 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 3.0 MiB/s wr, 26 op/s 2021-07-31T01:53:43.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:42 smithi154 conmon[34553]: 2021-07-31T01:53:43.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:42 smithi154 conmon[34553]: cluster 2021-07-31T01:53:42.601261+0000 mon.a (mon.0) 1177 : cluster [DBG] osdmap e521: 8 total, 8 up, 8 in 2021-07-31T01:53:43.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:42 smithi045 conmon[33655]: cluster 2021-07-31T01:53:41.883730+0000 mgr.x (mgr.24542) 279 : cluster [DBG] pgmap v561: 25 pgs: 25 active+clean; 194 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 3.0 MiB/s wr, 26 op/s 2021-07-31T01:53:43.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:42 smithi045 conmon[33655]: cluster 2021-07-31T01:53:42.601261+0000 mon.a (mon.0) 1177 : cluster [DBG] osdmap e521: 8 total, 8 up, 8 in 2021-07-31T01:53:43.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:42 smithi045 conmon[39718]: cluster 2021-07-31T01:53:41.883730+0000 mgr.x 2021-07-31T01:53:43.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:42 smithi045 conmon[39718]: (mgr.24542) 279 : cluster [DBG] pgmap v561: 25 pgs: 25 active+clean; 194 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 3.0 MiB/s wr, 26 op/s 2021-07-31T01:53:43.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:42 smithi045 conmon[39718]: cluster 2021-07-31T01:53:42. 2021-07-31T01:53:43.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:42 smithi045 conmon[39718]: 601261+0000 mon.a (mon.0) 1177 : cluster [DBG] osdmap e521: 8 total, 8 up, 8 in 2021-07-31T01:53:43.604 INFO:tasks.rados.rados.0.smithi045.stdout:2421: done (2 left) 2021-07-31T01:53:43.604 INFO:tasks.rados.rados.0.smithi045.stdout:2422: rmattr oid 29 current snap is 230 2021-07-31T01:53:43.605 INFO:tasks.rados.rados.0.smithi045.stdout:2417: finishing write tid 2 to smithi04569269-43 2021-07-31T01:53:43.605 INFO:tasks.rados.rados.0.smithi045.stdout:2417: finishing write tid 3 to smithi04569269-43 2021-07-31T01:53:43.605 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 43 v 1398 (ObjNum 744 snap 230 seq_num 744) dirty exists 2021-07-31T01:53:43.605 INFO:tasks.rados.rados.0.smithi045.stdout:2417: left oid 43 (ObjNum 744 snap 230 seq_num 744) 2021-07-31T01:53:43.606 INFO:tasks.rados.rados.0.smithi045.stdout:2419: finishing write tid 1 to smithi04569269-21 2021-07-31T01:53:43.606 INFO:tasks.rados.rados.0.smithi045.stdout:2419: finishing write tid 2 to smithi04569269-21 2021-07-31T01:53:43.606 INFO:tasks.rados.rados.0.smithi045.stdout:2419: finishing write tid 3 to smithi04569269-21 2021-07-31T01:53:43.606 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 21 v 1586 (ObjNum 745 snap 230 seq_num 745) dirty exists 2021-07-31T01:53:43.607 INFO:tasks.rados.rados.0.smithi045.stdout:2419: left oid 21 (ObjNum 745 snap 230 seq_num 745) 2021-07-31T01:53:43.607 INFO:tasks.rados.rados.0.smithi045.stdout:2417: done (2 left) 2021-07-31T01:53:43.607 INFO:tasks.rados.rados.0.smithi045.stdout:2419: done (1 left) 2021-07-31T01:53:43.607 INFO:tasks.rados.rados.0.smithi045.stdout:2423: read oid 37 snap -1 2021-07-31T01:53:43.607 INFO:tasks.rados.rados.0.smithi045.stdout:2423: expect (ObjNum 639 snap 199 seq_num 639) 2021-07-31T01:53:43.608 INFO:tasks.rados.rados.0.smithi045.stdout:2424: setattr oid 14 current snap is 230 2021-07-31T01:53:43.608 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 18 current snap is 230 2021-07-31T01:53:43.608 INFO:tasks.rados.rados.0.smithi045.stdout:2425: seq_num 746 ranges {352256=499712} 2021-07-31T01:53:43.613 INFO:tasks.rados.rados.0.smithi045.stdout:2425: writing smithi04569269-18 from 352256 to 851968 tid 1 2021-07-31T01:53:43.614 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 45 current snap is 230 2021-07-31T01:53:43.614 INFO:tasks.rados.rados.0.smithi045.stdout:2426: seq_num 747 ranges {1441792=516096} 2021-07-31T01:53:43.623 INFO:tasks.rados.rados.0.smithi045.stdout:2426: writing smithi04569269-45 from 1441792 to 1957888 tid 1 2021-07-31T01:53:43.623 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 41 current snap is 230 2021-07-31T01:53:43.624 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 29 v 1046 (ObjNum 642 snap 199 seq_num 642) dirty exists 2021-07-31T01:53:43.624 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 14 v 1399 (ObjNum 738 snap 228 seq_num 738) dirty exists 2021-07-31T01:53:43.624 INFO:tasks.rados.rados.0.smithi045.stdout:2427: seq_num 748 ranges {8790016=532480} 2021-07-31T01:53:43.675 INFO:tasks.rados.rados.0.smithi045.stdout:2427: writing smithi04569269-41 from 8790016 to 9322496 tid 1 2021-07-31T01:53:43.676 INFO:tasks.rados.rados.0.smithi045.stdout:2422: done (5 left) 2021-07-31T01:53:43.677 INFO:tasks.rados.rados.0.smithi045.stdout:2424: done (4 left) 2021-07-31T01:53:43.678 INFO:tasks.rados.rados.0.smithi045.stdout:2428: rmattr oid 25 current snap is 230 2021-07-31T01:53:43.678 INFO:tasks.rados.rados.0.smithi045.stdout:2428: done (4 left) 2021-07-31T01:53:43.678 INFO:tasks.rados.rados.0.smithi045.stdout:2429: delete oid 39 current snap is 230 2021-07-31T01:53:43.678 INFO:tasks.rados.rados.0.smithi045.stdout:2426: finishing write tid 1 to smithi04569269-45 2021-07-31T01:53:43.679 INFO:tasks.rados.rados.0.smithi045.stdout:2426: finishing write tid 2 to smithi04569269-45 2021-07-31T01:53:43.679 INFO:tasks.rados.rados.0.smithi045.stdout:2426: finishing write tid 3 to smithi04569269-45 2021-07-31T01:53:43.680 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 45 v 1564 (ObjNum 747 snap 230 seq_num 747) dirty exists 2021-07-31T01:53:43.680 INFO:tasks.rados.rados.0.smithi045.stdout:2426: left oid 45 (ObjNum 747 snap 230 seq_num 747) 2021-07-31T01:53:43.680 INFO:tasks.rados.rados.0.smithi045.stdout:2423: expect (ObjNum 639 snap 199 seq_num 639) 2021-07-31T01:53:43.776 INFO:journalctl@ceph.osd.0.smithi045.stdout:Jul 31 01:53:43 smithi045 conmon[43243]: debug 2021-07-31T01:53:43.776 INFO:journalctl@ceph.osd.0.smithi045.stdout:Jul 31 01:53:43 smithi045 conmon[43243]: 2021-07-31T01:53:43.608+0000 7fdf6bf90700 -1 osd.0 pg_epoch: 522 pg[3.1s0( v 517'375 (0'0,517'375] local-lis/les=56/57 n=4 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [0,4,3]p0(0) r=0 lpr=56 crt=517'375 lcod 511'373 mlcod 511'373 active+clean+snaptrim trimq=[1c8~1] ps=[1c9~2]] removing snap head 2021-07-31T01:53:43.776 INFO:journalctl@ceph.osd.0.smithi045.stdout:Jul 31 01:53:43 smithi045 conmon[43243]: 2021-07-31T01:53:43.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:53:43 smithi045 conmon[61210]: level=error ts=2021-07-31T01:53:43.520Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:53:43.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:53:43 smithi045 conmon[61210]: level=error ts=2021-07-31T01:53:43.520Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:53:43.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:53:43 smithi045 conmon[61210]: level=error ts=2021-07-31T01:53:43.520Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T01:53:44.003 INFO:journalctl@ceph.osd.5.smithi154.stdout:Jul 31 01:53:43 smithi154 conmon[42050]: debug 2021-07-31T01:53:43.602+0000 7fc9b105c700 -1 osd.5 pg_epoch: 522 pg[3.cs0( v 521'978 (0'0,521'978] local-lis/les=56/57 n=4 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [5,3,6]p5(0) r=0 lpr=56 crt=521'978 lcod 520'976 mlcod 520'976 active+clean+snaptrim trimq=[1c8~1] ps=[1c9~2]] removing snap head 2021-07-31T01:53:44.030 INFO:tasks.rados.rados.0.smithi045.stdout:2425: finishing write tid 1 to smithi04569269-18 2021-07-31T01:53:44.030 INFO:tasks.rados.rados.0.smithi045.stdout:2425: finishing write tid 2 to smithi04569269-18 2021-07-31T01:53:44.031 INFO:tasks.rados.rados.0.smithi045.stdout:2425: finishing write tid 3 to smithi04569269-18 2021-07-31T01:53:44.031 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 18 v 1000 (ObjNum 746 snap 230 seq_num 746) dirty exists 2021-07-31T01:53:44.031 INFO:tasks.rados.rados.0.smithi045.stdout:2425: left oid 18 (ObjNum 746 snap 230 seq_num 746) 2021-07-31T01:53:44.031 INFO:tasks.rados.rados.0.smithi045.stdout:2427: finishing write tid 1 to smithi04569269-41 2021-07-31T01:53:44.032 INFO:tasks.rados.rados.0.smithi045.stdout:2427: finishing write tid 2 to smithi04569269-41 2021-07-31T01:53:44.032 INFO:tasks.rados.rados.0.smithi045.stdout:2423: done (4 left) 2021-07-31T01:53:44.032 INFO:tasks.rados.rados.0.smithi045.stdout:2425: done (3 left) 2021-07-31T01:53:44.033 INFO:tasks.rados.rados.0.smithi045.stdout:2426: done (2 left) 2021-07-31T01:53:44.033 INFO:tasks.rados.rados.0.smithi045.stdout:2429: done (1 left) 2021-07-31T01:53:44.033 INFO:tasks.rados.rados.0.smithi045.stdout:2430: snap_create 2021-07-31T01:53:44.034 INFO:tasks.rados.rados.0.smithi045.stdout:2427: finishing write tid 3 to smithi04569269-41 2021-07-31T01:53:44.034 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 41 v 995 (ObjNum 748 snap 230 seq_num 748) dirty exists 2021-07-31T01:53:44.034 INFO:tasks.rados.rados.0.smithi045.stdout:2427: left oid 41 (ObjNum 748 snap 230 seq_num 748) 2021-07-31T01:53:44.612 INFO:tasks.rados.rados.0.smithi045.stdout:2427: done (1 left) 2021-07-31T01:53:44.613 INFO:tasks.rados.rados.0.smithi045.stdout:2430: done (0 left) 2021-07-31T01:53:44.613 INFO:tasks.rados.rados.0.smithi045.stdout:2431: rollback oid 20 current snap is 231 2021-07-31T01:53:44.614 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 20 to 230 2021-07-31T01:53:44.614 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 35 current snap is 231 2021-07-31T01:53:44.614 INFO:tasks.rados.rados.0.smithi045.stdout:2432: seq_num 750 ranges {0=565248} 2021-07-31T01:53:44.616 INFO:tasks.rados.rados.0.smithi045.stdout:2432: writing smithi04569269-35 from 0 to 565248 tid 1 2021-07-31T01:53:44.616 INFO:tasks.rados.rados.0.smithi045.stdout:2433: delete oid 5 current snap is 231 2021-07-31T01:53:44.617 INFO:tasks.rados.rados.0.smithi045.stdout:2431: finishing rollback tid 0 to smithi04569269-20 2021-07-31T01:53:44.619 INFO:tasks.rados.rados.0.smithi045.stdout:2433: done (2 left) 2021-07-31T01:53:44.620 INFO:tasks.rados.rados.0.smithi045.stdout:2434: setattr oid 15 current snap is 231 2021-07-31T01:53:44.621 INFO:tasks.rados.rados.0.smithi045.stdout:2431: finishing rollback tid 1 to smithi04569269-20 2021-07-31T01:53:44.622 INFO:tasks.rados.rados.0.smithi045.stdout:2431: finishing rollback tid 2 to smithi04569269-20 2021-07-31T01:53:44.622 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 20 v 1590 (ObjNum 556 snap 166 seq_num 556) dirty exists 2021-07-31T01:53:44.623 INFO:tasks.rados.rados.0.smithi045.stdout:2432: finishing write tid 1 to smithi04569269-35 2021-07-31T01:53:44.623 INFO:tasks.rados.rados.0.smithi045.stdout:2431: done (2 left) 2021-07-31T01:53:44.624 INFO:tasks.rados.rados.0.smithi045.stdout:2435: rmattr oid 7 current snap is 231 2021-07-31T01:53:44.624 INFO:tasks.rados.rados.0.smithi045.stdout:2432: finishing write tid 2 to smithi04569269-35 2021-07-31T01:53:44.624 INFO:tasks.rados.rados.0.smithi045.stdout:2436: rmattr oid 1 current snap is 231 2021-07-31T01:53:44.624 INFO:tasks.rados.rados.0.smithi045.stdout:2436: done (3 left) 2021-07-31T01:53:44.625 INFO:tasks.rados.rados.0.smithi045.stdout:2437: read oid 2 snap 230 2021-07-31T01:53:44.625 INFO:tasks.rados.rados.0.smithi045.stdout:2437: expect (ObjNum 741 snap 228 seq_num 741) 2021-07-31T01:53:44.625 INFO:tasks.rados.rados.0.smithi045.stdout:2438: rollback oid 19 current snap is 231 2021-07-31T01:53:44.625 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 19 to 230 2021-07-31T01:53:44.626 INFO:tasks.rados.rados.0.smithi045.stdout:2439: read oid 22 snap -1 2021-07-31T01:53:44.626 INFO:tasks.rados.rados.0.smithi045.stdout:2439: expect (ObjNum 575 snap 175 seq_num 575) 2021-07-31T01:53:44.626 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 41 current snap is 231 2021-07-31T01:53:44.626 INFO:tasks.rados.rados.0.smithi045.stdout:2440: seq_num 752 ranges {9322496=598016} 2021-07-31T01:53:44.678 INFO:tasks.rados.rados.0.smithi045.stdout:2440: writing smithi04569269-41 from 9322496 to 9920512 tid 1 2021-07-31T01:53:44.678 INFO:tasks.rados.rados.0.smithi045.stdout:2441: rollback oid 42 current snap is 231 2021-07-31T01:53:44.678 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 42 to 230 2021-07-31T01:53:44.679 INFO:tasks.rados.rados.0.smithi045.stdout:2432: finishing write tid 3 to smithi04569269-35 2021-07-31T01:53:44.679 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 35 v 1593 (ObjNum 750 snap 231 seq_num 750) dirty exists 2021-07-31T01:53:44.679 INFO:tasks.rados.rados.0.smithi045.stdout:2432: left oid 35 (ObjNum 750 snap 231 seq_num 750) 2021-07-31T01:53:44.679 INFO:tasks.rados.rados.0.smithi045.stdout:2438: finishing rollback tid 1 to smithi04569269-19 2021-07-31T01:53:44.680 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 19 v 0 (ObjNum -1106013808 snap 21873 seq_num 3189312784) dirty dne 2021-07-31T01:53:44.680 INFO:tasks.rados.rados.0.smithi045.stdout:2439: expect (ObjNum 575 snap 175 seq_num 575) 2021-07-31T01:53:44.680 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 7 v 996 (ObjNum 618 snap 189 seq_num 618) dirty exists 2021-07-31T01:53:44.680 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 15 v 1001 (ObjNum 677 snap 211 seq_num 677) dirty exists 2021-07-31T01:53:44.681 INFO:tasks.rados.rados.0.smithi045.stdout:2437: expect (ObjNum 741 snap 228 seq_num 741) 2021-07-31T01:53:45.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:44 smithi154 conmon[34553]: cluster 2021-07-31T01:53:43.602942+0000 mon.a (mon. 2021-07-31T01:53:45.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:44 smithi154 conmon[34553]: 0) 1178 : cluster [DBG] osdmap e522: 8 total, 8 up, 8 in 2021-07-31T01:53:45.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:44 smithi154 conmon[34553]: cluster 2021-07-31T01:53:43.884435+0000 mgr.x (mgr.24542) 280 : cluster [DBG] pgmap v564: 25 pgs: 25 active+clean; 184 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 2.7 MiB/s wr, 14 op/s 2021-07-31T01:53:45.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:44 smithi045 conmon[33655]: cluster 2021-07-31T01:53:43.602942+0000 2021-07-31T01:53:45.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:44 smithi045 conmon[33655]: mon.a (mon.0) 1178 : cluster [DBG] osdmap e522: 8 total, 8 up, 8 in 2021-07-31T01:53:45.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:44 smithi045 conmon[33655]: cluster 2021-07-31T01:53:43.884435+0000 mgr.x (mgr.24542) 280 : cluster [DBG] pgmap v564: 25 pgs: 25 active+clean; 184 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 2.7 MiB/s wr, 14 op/s 2021-07-31T01:53:45.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:44 smithi045 conmon[39718]: cluster 2021-07-31T01:53:43 2021-07-31T01:53:45.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:44 smithi045 conmon[39718]: .602942+0000 mon.a (mon.0) 1178 : cluster [DBG] osdmap e522: 8 total, 8 up, 8 in 2021-07-31T01:53:45.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:44 smithi045 conmon[39718]: cluster 2021-07- 2021-07-31T01:53:45.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:44 smithi045 conmon[39718]: 31T01:53:43.884435+0000 mgr.x (mgr.24542) 280 : cluster [DBG] pgmap v564: 25 pgs: 25 active+clean; 184 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 2.7 MiB/s wr, 14 op/s 2021-07-31T01:53:45.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:44 smithi045 conmon[39718]: 2021-07-31T01:53:45.173 INFO:tasks.rados.rados.0.smithi045.stdout:2432: done (7 left) 2021-07-31T01:53:45.174 INFO:tasks.rados.rados.0.smithi045.stdout:2434: done (6 left) 2021-07-31T01:53:45.174 INFO:tasks.rados.rados.0.smithi045.stdout:2435: done (5 left) 2021-07-31T01:53:45.175 INFO:tasks.rados.rados.0.smithi045.stdout:2437: done (4 left) 2021-07-31T01:53:45.175 INFO:tasks.rados.rados.0.smithi045.stdout:2438: done (3 left) 2021-07-31T01:53:45.176 INFO:tasks.rados.rados.0.smithi045.stdout:2439: done (2 left) 2021-07-31T01:53:45.176 INFO:tasks.rados.rados.0.smithi045.stdout:2442: rmattr oid 49 current snap is 231 2021-07-31T01:53:45.177 INFO:tasks.rados.rados.0.smithi045.stdout:2441: finishing rollback tid 0 to smithi04569269-42 2021-07-31T01:53:45.177 INFO:tasks.rados.rados.0.smithi045.stdout:2441: finishing rollback tid 1 to smithi04569269-42 2021-07-31T01:53:45.177 INFO:tasks.rados.rados.0.smithi045.stdout:2440: finishing write tid 1 to smithi04569269-41 2021-07-31T01:53:45.177 INFO:tasks.rados.rados.0.smithi045.stdout:2440: finishing write tid 2 to smithi04569269-41 2021-07-31T01:53:45.178 INFO:tasks.rados.rados.0.smithi045.stdout:2441: finishing rollback tid 2 to smithi04569269-42 2021-07-31T01:53:45.178 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 42 v 1567 (ObjNum 704 snap 219 seq_num 704) dirty exists 2021-07-31T01:53:45.178 INFO:tasks.rados.rados.0.smithi045.stdout:2440: finishing write tid 3 to smithi04569269-41 2021-07-31T01:53:45.178 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 41 v 998 (ObjNum 752 snap 231 seq_num 752) dirty exists 2021-07-31T01:53:45.179 INFO:tasks.rados.rados.0.smithi045.stdout:2440: left oid 41 (ObjNum 752 snap 231 seq_num 752) 2021-07-31T01:53:45.179 INFO:tasks.rados.rados.0.smithi045.stdout:2440: done (2 left) 2021-07-31T01:53:45.180 INFO:tasks.rados.rados.0.smithi045.stdout:2441: done (1 left) 2021-07-31T01:53:45.180 INFO:tasks.rados.rados.0.smithi045.stdout:2443: read oid 39 snap 230 2021-07-31T01:53:45.180 INFO:tasks.rados.rados.0.smithi045.stdout:2443: expect deleted 2021-07-31T01:53:45.180 INFO:tasks.rados.rados.0.smithi045.stdout:2444: delete oid 5 current snap is 231 2021-07-31T01:53:45.181 INFO:tasks.rados.rados.0.smithi045.stdout:2443: done (2 left) 2021-07-31T01:53:45.182 INFO:tasks.rados.rados.0.smithi045.stdout:2444: done (1 left) 2021-07-31T01:53:45.182 INFO:tasks.rados.rados.0.smithi045.stdout:2445: copy_from oid 22 from oid 45 current snap is 231 2021-07-31T01:53:45.182 INFO:tasks.rados.rados.0.smithi045.stdout:2446: read oid 15 snap -1 2021-07-31T01:53:45.182 INFO:tasks.rados.rados.0.smithi045.stdout:2446: expect (ObjNum 677 snap 211 seq_num 677) 2021-07-31T01:53:45.183 INFO:tasks.rados.rados.0.smithi045.stdout:2447: snap_create 2021-07-31T01:53:45.183 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 49 v 1595 (ObjNum 679 snap 212 seq_num 679) dirty exists 2021-07-31T01:53:45.185 INFO:tasks.rados.rados.0.smithi045.stdout:2445: finishing copy_from to smithi04569269-22 2021-07-31T01:53:45.185 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 22 v 1564 (ObjNum 747 snap 230 seq_num 747) dirty exists 2021-07-31T01:53:45.186 INFO:tasks.rados.rados.0.smithi045.stdout:2445: finishing copy_from racing read to smithi04569269-22 2021-07-31T01:53:45.193 INFO:tasks.rados.rados.0.smithi045.stdout:2446: expect (ObjNum 677 snap 211 seq_num 677) 2021-07-31T01:53:45.717 INFO:tasks.rados.rados.0.smithi045.stdout:2442: done (3 left) 2021-07-31T01:53:45.718 INFO:tasks.rados.rados.0.smithi045.stdout:2445: done (2 left) 2021-07-31T01:53:45.718 INFO:tasks.rados.rados.0.smithi045.stdout:2446: done (1 left) 2021-07-31T01:53:45.718 INFO:tasks.rados.rados.0.smithi045.stdout:2447: done (0 left) 2021-07-31T01:53:45.718 INFO:tasks.rados.rados.0.smithi045.stdout:2448: read oid 49 snap -1 2021-07-31T01:53:45.719 INFO:tasks.rados.rados.0.smithi045.stdout:2448: expect (ObjNum 679 snap 212 seq_num 679) 2021-07-31T01:53:45.719 INFO:tasks.rados.rados.0.smithi045.stdout:2449: snap_create 2021-07-31T01:53:45.738 INFO:tasks.rados.rados.0.smithi045.stdout:2448: expect (ObjNum 679 snap 212 seq_num 679) 2021-07-31T01:53:46.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:45 smithi154 conmon[34553]: cluster 2021-07-31T01:53:44.610447+0000 mon.a (mon.0) 1179 : cluster [DBG] osdmap e523: 8 total, 8 up, 8 in 2021-07-31T01:53:46.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:45 smithi154 conmon[34553]: 2021-07-31T01:53:46.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:45 smithi045 conmon[33655]: cluster 2021-07-31T01:53:44.610447+0000 mon.a (mon.0) 2021-07-31T01:53:46.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:45 smithi045 conmon[33655]: 1179 : cluster [DBG] osdmap e523: 8 total, 8 up, 8 in 2021-07-31T01:53:46.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:45 smithi045 conmon[39718]: cluster 2021-07-31T01:53:44.610447+0000 2021-07-31T01:53:46.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:45 smithi045 conmon[39718]: mon.a (mon.0) 1179 : cluster [DBG] osdmap e523: 8 total, 8 up, 8 in 2021-07-31T01:53:46.594 INFO:tasks.rados.rados.0.smithi045.stdout:2448: done (1 left) 2021-07-31T01:53:46.595 INFO:tasks.rados.rados.0.smithi045.stdout:2449: done (0 left) 2021-07-31T01:53:46.595 INFO:tasks.rados.rados.0.smithi045.stdout:2450: setattr oid 29 current snap is 233 2021-07-31T01:53:46.596 INFO:tasks.rados.rados.0.smithi045.stdout:2451: read oid 12 snap -1 2021-07-31T01:53:46.596 INFO:tasks.rados.rados.0.smithi045.stdout:2451: expect (ObjNum 672 snap 210 seq_num 672) 2021-07-31T01:53:46.597 INFO:tasks.rados.rados.0.smithi045.stdout:2452: read oid 4 snap 232 2021-07-31T01:53:46.597 INFO:tasks.rados.rados.0.smithi045.stdout:2452: expect (ObjNum 716 snap 222 seq_num 716) 2021-07-31T01:53:46.597 INFO:tasks.rados.rados.0.smithi045.stdout:2453: rmattr oid 38 current snap is 233 2021-07-31T01:53:46.597 INFO:tasks.rados.rados.0.smithi045.stdout:2453: done (3 left) 2021-07-31T01:53:46.598 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 43 current snap is 233 2021-07-31T01:53:46.598 INFO:tasks.rados.rados.0.smithi045.stdout:2454: seq_num 754 ranges {8380416=630784} 2021-07-31T01:53:46.648 INFO:tasks.rados.rados.0.smithi045.stdout:2454: writing smithi04569269-43 from 8380416 to 9011200 tid 1 2021-07-31T01:53:46.648 INFO:tasks.rados.rados.0.smithi045.stdout:2455: delete oid 36 current snap is 233 2021-07-31T01:53:46.648 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 29 v 1047 (ObjNum 642 snap 199 seq_num 642) dirty exists 2021-07-31T01:53:46.648 INFO:tasks.rados.rados.0.smithi045.stdout:2452: expect (ObjNum 716 snap 222 seq_num 716) 2021-07-31T01:53:46.709 INFO:tasks.rados.rados.0.smithi045.stdout:2451: expect (ObjNum 672 snap 210 seq_num 672) 2021-07-31T01:53:46.975 INFO:tasks.rados.rados.0.smithi045.stdout:2454: finishing write tid 1 to smithi04569269-43 2021-07-31T01:53:46.975 INFO:tasks.rados.rados.0.smithi045.stdout:2454: finishing write tid 2 to smithi04569269-43 2021-07-31T01:53:46.975 INFO:tasks.rados.rados.0.smithi045.stdout:2454: finishing write tid 3 to smithi04569269-43 2021-07-31T01:53:46.976 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 43 v 1401 (ObjNum 754 snap 233 seq_num 754) dirty exists 2021-07-31T01:53:46.976 INFO:tasks.rados.rados.0.smithi045.stdout:2454: left oid 43 (ObjNum 754 snap 233 seq_num 754) 2021-07-31T01:53:46.976 INFO:tasks.rados.rados.0.smithi045.stdout:2450: done (4 left) 2021-07-31T01:53:46.976 INFO:tasks.rados.rados.0.smithi045.stdout:2451: done (3 left) 2021-07-31T01:53:46.977 INFO:tasks.rados.rados.0.smithi045.stdout:2452: done (2 left) 2021-07-31T01:53:46.977 INFO:tasks.rados.rados.0.smithi045.stdout:2454: done (1 left) 2021-07-31T01:53:46.977 INFO:tasks.rados.rados.0.smithi045.stdout:2455: done (0 left) 2021-07-31T01:53:46.977 INFO:tasks.rados.rados.0.smithi045.stdout:2456: read oid 29 snap -1 2021-07-31T01:53:46.977 INFO:tasks.rados.rados.0.smithi045.stdout:2456: expect (ObjNum 642 snap 199 seq_num 642) 2021-07-31T01:53:46.978 INFO:tasks.rados.rados.0.smithi045.stdout:2457: read oid 8 snap -1 2021-07-31T01:53:46.978 INFO:tasks.rados.rados.0.smithi045.stdout:2457: expect (ObjNum 701 snap 218 seq_num 701) 2021-07-31T01:53:46.978 INFO:tasks.rados.rados.0.smithi045.stdout:2458: copy_from oid 5 from oid 43 current snap is 233 2021-07-31T01:53:46.978 INFO:tasks.rados.rados.0.smithi045.stdout:2459: rollback oid 17 current snap is 233 2021-07-31T01:53:46.979 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 17 to 230 2021-07-31T01:53:46.979 INFO:tasks.rados.rados.0.smithi045.stdout:2460: read oid 4 snap -1 2021-07-31T01:53:46.979 INFO:tasks.rados.rados.0.smithi045.stdout:2460: expect (ObjNum 716 snap 222 seq_num 716) 2021-07-31T01:53:46.979 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 39 current snap is 233 2021-07-31T01:53:46.979 INFO:tasks.rados.rados.0.smithi045.stdout:2461: seq_num 756 ranges {0=671744} 2021-07-31T01:53:46.981 INFO:tasks.rados.rados.0.smithi045.stdout:2461: writing smithi04569269-39 from 0 to 671744 tid 1 2021-07-31T01:53:46.981 INFO:tasks.rados.rados.0.smithi045.stdout:2462: copy_from oid 25 from oid 9 current snap is 233 2021-07-31T01:53:46.981 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 10 current snap is 233 2021-07-31T01:53:46.983 INFO:tasks.rados.rados.0.smithi045.stdout:2463: seq_num 757 ranges {8486912=688128} 2021-07-31T01:53:47.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:46 smithi154 conmon[34553]: cluster 2021-07-31T01:53:45.614511+0000 mon.a (mon.0) 1180 : cluster [DBG] osdmap e524: 8 total, 8 up, 8 in 2021-07-31T01:53:47.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:46 smithi154 conmon[34553]: cluster 2021-07-31T01:53:45.884906+0000 mgr.x (mgr.24542) 281 : cluster [DBG] pgmap v567: 25 pgs: 25 active+clean; 184 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 2.7 MiB/s wr, 14 op/s 2021-07-31T01:53:47.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:46 smithi154 conmon[34553]: cluster 2021-07-31T01:53:46.593904+0000 mon.a (mon.0) 2021-07-31T01:53:47.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:46 smithi154 conmon[34553]: 1181 : cluster [DBG] osdmap e525: 8 total, 8 up, 8 in 2021-07-31T01:53:47.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:46 smithi045 conmon[39718]: cluster 2021-07-31T01:53:45.614511+0000 mon.a ( 2021-07-31T01:53:47.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:46 smithi045 conmon[39718]: mon.0) 1180 : cluster [DBG] osdmap e524: 8 total, 8 up, 8 in 2021-07-31T01:53:47.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:46 smithi045 conmon[39718]: cluster 2021-07-31T01:53:45.884906+0000 mgr.x ( 2021-07-31T01:53:47.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:46 smithi045 conmon[39718]: mgr.24542) 281 : cluster [DBG] pgmap v567: 25 pgs: 25 active+clean; 184 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 2.7 MiB/s wr, 14 op/s 2021-07-31T01:53:47.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:46 smithi045 conmon[39718]: cluster 2021-07-31T01:53:46. 2021-07-31T01:53:47.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:46 smithi045 conmon[39718]: 593904+0000 mon.a (mon.0) 1181 : cluster [DBG] 2021-07-31T01:53:47.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:46 smithi045 conmon[39718]: osdmap e525: 8 total, 8 up, 8 in 2021-07-31T01:53:47.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:46 smithi045 conmon[33655]: cluster 2021-07-31T01:53:45.614511 2021-07-31T01:53:47.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:46 smithi045 conmon[33655]: +0000 mon.a (mon.0) 1180 : cluster [DBG] osdmap e524: 8 total, 8 up, 8 in 2021-07-31T01:53:47.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:46 smithi045 conmon[33655]: cluster 2021-07-31T01:53:45.884906 2021-07-31T01:53:47.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:46 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 281 : cluster [DBG] pgmap v567: 25 pgs: 25 active+clean; 184 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 2.7 MiB/s wr, 14 op/s 2021-07-31T01:53:47.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:46 smithi045 conmon[33655]: cluster 2021-07 2021-07-31T01:53:47.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:46 smithi045 conmon[33655]: -31T01:53:46.593904+0000 mon.a (mon 2021-07-31T01:53:47.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:46 smithi045 conmon[33655]: .0) 1181 : cluster [DBG] 2021-07-31T01:53:47.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:46 smithi045 conmon[33655]: osdmap e525: 8 total, 8 up, 8 in 2021-07-31T01:53:47.033 INFO:tasks.rados.rados.0.smithi045.stdout:2463: writing smithi04569269-10 from 8486912 to 9175040 tid 1 2021-07-31T01:53:47.033 INFO:tasks.rados.rados.0.smithi045.stdout:2464: read oid 36 snap -1 2021-07-31T01:53:47.034 INFO:tasks.rados.rados.0.smithi045.stdout:2464: expect deleted 2021-07-31T01:53:47.034 INFO:tasks.rados.rados.0.smithi045.stdout:2465: copy_from oid 27 from oid 33 current snap is 233 2021-07-31T01:53:47.035 INFO:tasks.rados.rados.0.smithi045.stdout:2462: finishing copy_from to smithi04569269-25 2021-07-31T01:53:47.035 INFO:tasks.rados.rados.0.smithi045.stdout:2462: got expected ENOENT (src dne) 2021-07-31T01:53:47.035 INFO:tasks.rados.rados.0.smithi045.stdout:2462: finishing copy_from racing read to smithi04569269-25 2021-07-31T01:53:47.035 INFO:tasks.rados.rados.0.smithi045.stdout:2462: done (9 left) 2021-07-31T01:53:47.035 INFO:tasks.rados.rados.0.smithi045.stdout:2466: read oid 45 snap -1 2021-07-31T01:53:47.036 INFO:tasks.rados.rados.0.smithi045.stdout:2466: expect (ObjNum 747 snap 230 seq_num 747) 2021-07-31T01:53:47.036 INFO:tasks.rados.rados.0.smithi045.stdout:2467: copy_from oid 47 from oid 50 current snap is 233 2021-07-31T01:53:47.036 INFO:tasks.rados.rados.0.smithi045.stdout:2459: finishing rollback tid 0 to smithi04569269-17 2021-07-31T01:53:47.036 INFO:tasks.rados.rados.0.smithi045.stdout:2459: finishing rollback tid 1 to smithi04569269-17 2021-07-31T01:53:47.036 INFO:tasks.rados.rados.0.smithi045.stdout:2459: finishing rollback tid 2 to smithi04569269-17 2021-07-31T01:53:47.037 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 17 v 1570 (ObjNum 642 snap 199 seq_num 642) dirty exists 2021-07-31T01:53:47.038 INFO:tasks.rados.rados.0.smithi045.stdout:2460: expect (ObjNum 716 snap 222 seq_num 716) 2021-07-31T01:53:47.092 INFO:tasks.rados.rados.0.smithi045.stdout:2461: finishing write tid 1 to smithi04569269-39 2021-07-31T01:53:47.092 INFO:tasks.rados.rados.0.smithi045.stdout:2461: finishing write tid 2 to smithi04569269-39 2021-07-31T01:53:47.092 INFO:tasks.rados.rados.0.smithi045.stdout:2461: finishing write tid 3 to smithi04569269-39 2021-07-31T01:53:47.093 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 39 v 1180 (ObjNum 756 snap 233 seq_num 756) dirty exists 2021-07-31T01:53:47.093 INFO:tasks.rados.rados.0.smithi045.stdout:2461: left oid 39 (ObjNum 756 snap 233 seq_num 756) 2021-07-31T01:53:47.093 INFO:tasks.rados.rados.0.smithi045.stdout:2457: expect (ObjNum 701 snap 218 seq_num 701) 2021-07-31T01:53:47.216 INFO:tasks.rados.rados.0.smithi045.stdout:2456: expect (ObjNum 642 snap 199 seq_num 642) 2021-07-31T01:53:47.582 INFO:tasks.rados.rados.0.smithi045.stdout:2458: finishing copy_from to smithi04569269-5 2021-07-31T01:53:47.582 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 5 v 1401 (ObjNum 754 snap 233 seq_num 754) dirty exists 2021-07-31T01:53:47.583 INFO:tasks.rados.rados.0.smithi045.stdout:2458: finishing copy_from racing read to smithi04569269-5 2021-07-31T01:53:47.583 INFO:tasks.rados.rados.0.smithi045.stdout:2465: finishing copy_from to smithi04569269-27 2021-07-31T01:53:47.583 INFO:tasks.rados.rados.0.smithi045.stdout:2465: got expected ENOENT (src dne) 2021-07-31T01:53:47.584 INFO:tasks.rados.rados.0.smithi045.stdout:2465: finishing copy_from racing read to smithi04569269-27 2021-07-31T01:53:47.584 INFO:tasks.rados.rados.0.smithi045.stdout:2463: finishing write tid 1 to smithi04569269-10 2021-07-31T01:53:47.584 INFO:tasks.rados.rados.0.smithi045.stdout:2463: finishing write tid 2 to smithi04569269-10 2021-07-31T01:53:47.584 INFO:tasks.rados.rados.0.smithi045.stdout:2456: done (10 left) 2021-07-31T01:53:47.584 INFO:tasks.rados.rados.0.smithi045.stdout:2457: done (9 left) 2021-07-31T01:53:47.585 INFO:tasks.rados.rados.0.smithi045.stdout:2458: done (8 left) 2021-07-31T01:53:47.585 INFO:tasks.rados.rados.0.smithi045.stdout:2459: done (7 left) 2021-07-31T01:53:47.586 INFO:tasks.rados.rados.0.smithi045.stdout:2460: done (6 left) 2021-07-31T01:53:47.587 INFO:tasks.rados.rados.0.smithi045.stdout:2461: done (5 left) 2021-07-31T01:53:47.587 INFO:tasks.rados.rados.0.smithi045.stdout:2464: done (4 left) 2021-07-31T01:53:47.587 INFO:tasks.rados.rados.0.smithi045.stdout:2465: done (3 left) 2021-07-31T01:53:47.587 INFO:tasks.rados.rados.0.smithi045.stdout:2468: copy_from oid 18 from oid 4 current snap is 233 2021-07-31T01:53:47.588 INFO:tasks.rados.rados.0.smithi045.stdout:2463: finishing write tid 3 to smithi04569269-10 2021-07-31T01:53:47.588 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 10 v 1572 (ObjNum 757 snap 233 seq_num 757) dirty exists 2021-07-31T01:53:47.588 INFO:tasks.rados.rados.0.smithi045.stdout:2463: left oid 10 (ObjNum 757 snap 233 seq_num 757) 2021-07-31T01:53:47.588 INFO:tasks.rados.rados.0.smithi045.stdout:2466: expect (ObjNum 747 snap 230 seq_num 747) 2021-07-31T01:53:47.671 INFO:tasks.rados.rados.0.smithi045.stdout:2467: finishing copy_from to smithi04569269-47 2021-07-31T01:53:47.671 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 47 v 1185 (ObjNum 742 snap 230 seq_num 742) dirty exists 2021-07-31T01:53:47.672 INFO:tasks.rados.rados.0.smithi045.stdout:2467: finishing copy_from racing read to smithi04569269-47 2021-07-31T01:53:47.672 INFO:tasks.rados.rados.0.smithi045.stdout:2468: finishing copy_from to smithi04569269-18 2021-07-31T01:53:47.672 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 18 v 996 (ObjNum 716 snap 222 seq_num 716) dirty exists 2021-07-31T01:53:47.672 INFO:tasks.rados.rados.0.smithi045.stdout:2468: finishing copy_from racing read to smithi04569269-18 2021-07-31T01:53:47.673 INFO:tasks.rados.rados.0.smithi045.stdout:2463: done (3 left) 2021-07-31T01:53:47.673 INFO:tasks.rados.rados.0.smithi045.stdout:2466: done (2 left) 2021-07-31T01:53:47.673 INFO:tasks.rados.rados.0.smithi045.stdout:2467: done (1 left) 2021-07-31T01:53:47.673 INFO:tasks.rados.rados.0.smithi045.stdout:2468: done (0 left) 2021-07-31T01:53:47.674 INFO:tasks.rados.rados.0.smithi045.stdout:2469: snap_remove snap 230 2021-07-31T01:53:47.726 INFO:tasks.rados.rados.0.smithi045.stdout:2469: done (0 left) 2021-07-31T01:53:47.726 INFO:tasks.rados.rados.0.smithi045.stdout:2470: delete oid 29 current snap is 233 2021-07-31T01:53:47.735 INFO:tasks.rados.rados.0.smithi045.stdout:2470: done (0 left) 2021-07-31T01:53:47.736 INFO:tasks.rados.rados.0.smithi045.stdout:2471: delete oid 21 current snap is 233 2021-07-31T01:53:47.741 INFO:tasks.rados.rados.0.smithi045.stdout:2471: done (0 left) 2021-07-31T01:53:47.741 INFO:tasks.rados.rados.0.smithi045.stdout:2472: copy_from oid 13 from oid 17 current snap is 233 2021-07-31T01:53:47.742 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 26 current snap is 233 2021-07-31T01:53:47.742 INFO:tasks.rados.rados.0.smithi045.stdout:2473: seq_num 760 ranges {3792896=737280} 2021-07-31T01:53:47.768 INFO:tasks.rados.rados.0.smithi045.stdout:2473: writing smithi04569269-26 from 3792896 to 4530176 tid 1 2021-07-31T01:53:47.768 INFO:tasks.rados.rados.0.smithi045.stdout:2474: snap_create 2021-07-31T01:53:47.775 INFO:tasks.rados.rados.0.smithi045.stdout:2473: finishing write tid 1 to smithi04569269-26 2021-07-31T01:53:47.775 INFO:tasks.rados.rados.0.smithi045.stdout:2473: finishing write tid 2 to smithi04569269-26 2021-07-31T01:53:47.776 INFO:tasks.rados.rados.0.smithi045.stdout:2472: finishing copy_from to smithi04569269-13 2021-07-31T01:53:47.776 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 13 v 1570 (ObjNum 642 snap 199 seq_num 642) dirty exists 2021-07-31T01:53:47.776 INFO:tasks.rados.rados.0.smithi045.stdout:2472: finishing copy_from racing read to smithi04569269-13 2021-07-31T01:53:47.777 INFO:tasks.rados.rados.0.smithi045.stdout:2473: finishing write tid 3 to smithi04569269-26 2021-07-31T01:53:47.778 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 26 v 1063 (ObjNum 760 snap 233 seq_num 760) dirty exists 2021-07-31T01:53:47.778 INFO:tasks.rados.rados.0.smithi045.stdout:2473: left oid 26 (ObjNum 760 snap 233 seq_num 760) 2021-07-31T01:53:48.734 INFO:tasks.rados.rados.0.smithi045.stdout:2472: done (2 left) 2021-07-31T01:53:48.734 INFO:tasks.rados.rados.0.smithi045.stdout:2473: done (1 left) 2021-07-31T01:53:48.734 INFO:tasks.rados.rados.0.smithi045.stdout:2474: done (0 left) 2021-07-31T01:53:48.735 INFO:tasks.rados.rados.0.smithi045.stdout:2475: copy_from oid 24 from oid 35 current snap is 234 2021-07-31T01:53:48.735 INFO:tasks.rados.rados.0.smithi045.stdout:2476: read oid 32 snap 231 2021-07-31T01:53:48.735 INFO:tasks.rados.rados.0.smithi045.stdout:2476: expect (ObjNum 687 snap 214 seq_num 687) 2021-07-31T01:53:48.735 INFO:tasks.rados.rados.0.smithi045.stdout:2477: read oid 38 snap -1 2021-07-31T01:53:48.736 INFO:tasks.rados.rados.0.smithi045.stdout:2477: expect deleted 2021-07-31T01:53:48.736 INFO:tasks.rados.rados.0.smithi045.stdout:2478: read oid 50 snap 232 2021-07-31T01:53:48.736 INFO:tasks.rados.rados.0.smithi045.stdout:2478: expect (ObjNum 742 snap 230 seq_num 742) 2021-07-31T01:53:48.736 INFO:tasks.rados.rados.0.smithi045.stdout:2479: snap_remove snap 233 2021-07-31T01:53:49.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:48 smithi154 conmon[34553]: cluster 2021-07-31T01:53:49.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:48 smithi154 conmon[34553]: 2021-07-31T01:53:49.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:48 smithi154 conmon[34553]: 2021 2021-07-31T01:53:49.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:48 smithi154 conmon[34553]: - 2021-07-31T01:53:49.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:48 smithi154 conmon[34553]: 07 2021-07-31T01:53:49.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:48 smithi154 conmon[34553]: - 2021-07-31T01:53:49.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:48 smithi154 conmon[34553]: 31 2021-07-31T01:53:49.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:48 smithi154 conmon[34553]: T 2021-07-31T01:53:49.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:48 smithi154 conmon[34553]: 01:53:47.725844+0000 mon.a (mon.0) 1182 : 2021-07-31T01:53:49.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:48 smithi154 conmon[34553]: cluster [DBG] osdmap e526: 8 total, 8 up, 8 in 2021-07-31T01:53:49.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:48 smithi154 conmon[34553]: cluster 2021-07- 2021-07-31T01:53:49.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:48 smithi154 conmon[34553]: 31T01:53:47.885324+0000 mgr.x (mgr.24542) 282 : cluster [DBG] pgmap v570: 25 pgs: 25 active+clean; 184 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail 2021-07-31T01:53:49.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:48 smithi045 conmon[33655]: cluster 2021-07-31T01:53:47.725844+0000 mon.a (mon.0) 1182 : cluster [DBG] osdmap e526: 8 total, 8 up, 8 in 2021-07-31T01:53:49.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:48 smithi045 conmon[33655]: 2021-07-31T01:53:49.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:48 smithi045 conmon[33655]: cluster 2021-07-31T01:53:47.885324+0000 mgr.x (mgr.24542) 282 : cluster [DBG] pgmap v570: 25 pgs: 25 active+clean; 184 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail 2021-07-31T01:53:49.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:48 smithi045 conmon[39718]: cluster 2021-07-31T01:53:47.725844+0000 mon.a (mon.0) 2021-07-31T01:53:49.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:48 smithi045 conmon[39718]: 1182 : cluster [DBG] osdmap e526: 8 total, 8 up, 8 in 2021-07-31T01:53:49.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:48 smithi045 conmon[39718]: cluster 2021-07-31T01:53:47.885324+0000 mgr.x (mgr.24542) 282 2021-07-31T01:53:49.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:48 smithi045 conmon[39718]: : cluster [DBG] pgmap v570: 25 pgs: 25 active+clean; 184 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail 2021-07-31T01:53:49.739 INFO:tasks.rados.rados.0.smithi045.stdout:2479: done (4 left) 2021-07-31T01:53:49.739 INFO:tasks.rados.rados.0.smithi045.stdout:2480: delete oid 6 current snap is 234 2021-07-31T01:53:49.740 INFO:tasks.rados.rados.0.smithi045.stdout:2475: finishing copy_from to smithi04569269-24 2021-07-31T01:53:49.740 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 24 v 1593 (ObjNum 750 snap 231 seq_num 750) dirty exists 2021-07-31T01:53:49.740 INFO:tasks.rados.rados.0.smithi045.stdout:2475: finishing copy_from racing read to smithi04569269-24 2021-07-31T01:53:49.740 INFO:tasks.rados.rados.0.smithi045.stdout:2478: expect (ObjNum 742 snap 230 seq_num 742) 2021-07-31T01:53:49.901 INFO:tasks.rados.rados.0.smithi045.stdout:2476: expect (ObjNum 687 snap 214 seq_num 687) 2021-07-31T01:53:50.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:49 smithi154 conmon[34553]: cluster 2021-07-31T01:53:48.732883+0000 mon.a (mon.0) 1183 : cluster [DBG] osdmap e527: 8 total, 8 up, 8 in 2021-07-31T01:53:50.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:49 smithi154 conmon[34553]: 2021-07-31T01:53:50.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:49 smithi045 conmon[39718]: cluster 2021-07-31T01:53:48.732883+0000 mon.a (mon.0) 1183 : cluster [DBG] osdmap e527: 8 total, 8 up, 8 in 2021-07-31T01:53:50.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:49 smithi045 conmon[33655]: cluster 2021-07-31T01:53 2021-07-31T01:53:50.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:49 smithi045 conmon[33655]: :48.732883+0000 mon.a (mon.0) 1183 : cluster [DBG] osdmap e527: 8 total, 8 up, 8 in 2021-07-31T01:53:50.166 INFO:tasks.rados.rados.0.smithi045.stdout:2475: done (4 left) 2021-07-31T01:53:50.166 INFO:tasks.rados.rados.0.smithi045.stdout:2476: done (3 left) 2021-07-31T01:53:50.166 INFO:tasks.rados.rados.0.smithi045.stdout:2477: done (2 left) 2021-07-31T01:53:50.167 INFO:tasks.rados.rados.0.smithi045.stdout:2478: done (1 left) 2021-07-31T01:53:50.167 INFO:tasks.rados.rados.0.smithi045.stdout:2480: done (0 left) 2021-07-31T01:53:50.167 INFO:tasks.rados.rados.0.smithi045.stdout:2481: rollback oid 31 current snap is 234 2021-07-31T01:53:50.169 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 31 to 232 2021-07-31T01:53:50.169 INFO:tasks.rados.rados.0.smithi045.stdout:2482: snap_remove snap 231 2021-07-31T01:53:50.745 INFO:tasks.rados.rados.0.smithi045.stdout:2482: done (1 left) 2021-07-31T01:53:50.746 INFO:tasks.rados.rados.0.smithi045.stdout:2483: read oid 24 snap -1 2021-07-31T01:53:50.746 INFO:tasks.rados.rados.0.smithi045.stdout:2483: expect (ObjNum 750 snap 231 seq_num 750) 2021-07-31T01:53:50.746 INFO:tasks.rados.rados.0.smithi045.stdout:2484: rollback oid 13 current snap is 234 2021-07-31T01:53:50.747 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 13 to 232 2021-07-31T01:53:50.747 INFO:tasks.rados.rados.0.smithi045.stdout:2481: finishing rollback tid 0 to smithi04569269-31 2021-07-31T01:53:50.747 INFO:tasks.rados.rados.0.smithi045.stdout:2481: finishing rollback tid 1 to smithi04569269-31 2021-07-31T01:53:50.747 INFO:tasks.rados.rados.0.smithi045.stdout:2481: finishing rollback tid 2 to smithi04569269-31 2021-07-31T01:53:50.747 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 31 v 1608 (ObjNum 726 snap 226 seq_num 726) dirty exists 2021-07-31T01:53:50.748 INFO:tasks.rados.rados.0.smithi045.stdout:2481: done (2 left) 2021-07-31T01:53:50.748 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 18 current snap is 234 2021-07-31T01:53:50.748 INFO:tasks.rados.rados.0.smithi045.stdout:2485: seq_num 762 ranges {2400256=770048} 2021-07-31T01:53:50.764 INFO:tasks.rados.rados.0.smithi045.stdout:2485: writing smithi04569269-18 from 2400256 to 3170304 tid 1 2021-07-31T01:53:50.764 INFO:tasks.rados.rados.0.smithi045.stdout:2486: read oid 48 snap -1 2021-07-31T01:53:50.765 INFO:tasks.rados.rados.0.smithi045.stdout:2486: expect (ObjNum 734 snap 227 seq_num 734) 2021-07-31T01:53:50.768 INFO:tasks.rados.rados.0.smithi045.stdout:2487: read oid 9 snap -1 2021-07-31T01:53:50.768 INFO:tasks.rados.rados.0.smithi045.stdout:2487: expect deleted 2021-07-31T01:53:50.769 INFO:tasks.rados.rados.0.smithi045.stdout:2488: rollback oid 2 current snap is 234 2021-07-31T01:53:50.769 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 2 to 232 2021-07-31T01:53:50.769 INFO:tasks.rados.rados.0.smithi045.stdout:2483: expect (ObjNum 750 snap 231 seq_num 750) 2021-07-31T01:53:50.787 INFO:tasks.rados.rados.0.smithi045.stdout:2484: finishing rollback tid 0 to smithi04569269-13 2021-07-31T01:53:50.787 INFO:tasks.rados.rados.0.smithi045.stdout:2484: finishing rollback tid 1 to smithi04569269-13 2021-07-31T01:53:50.787 INFO:tasks.rados.rados.0.smithi045.stdout:2484: finishing rollback tid 2 to smithi04569269-13 2021-07-31T01:53:50.788 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 13 v 1574 (ObjNum 710 snap 221 seq_num 710) dirty exists 2021-07-31T01:53:50.788 INFO:tasks.rados.rados.0.smithi045.stdout:2483: done (5 left) 2021-07-31T01:53:50.788 INFO:tasks.rados.rados.0.smithi045.stdout:2484: done (4 left) 2021-07-31T01:53:50.788 INFO:tasks.rados.rados.0.smithi045.stdout:2489: delete oid 41 current snap is 234 2021-07-31T01:53:50.788 INFO:tasks.rados.rados.0.smithi045.stdout:2485: finishing write tid 1 to smithi04569269-18 2021-07-31T01:53:50.789 INFO:tasks.rados.rados.0.smithi045.stdout:2485: finishing write tid 2 to smithi04569269-18 2021-07-31T01:53:50.789 INFO:tasks.rados.rados.0.smithi045.stdout:2488: finishing rollback tid 0 to smithi04569269-2 2021-07-31T01:53:50.789 INFO:tasks.rados.rados.0.smithi045.stdout:2488: finishing rollback tid 1 to smithi04569269-2 2021-07-31T01:53:50.789 INFO:tasks.rados.rados.0.smithi045.stdout:2488: finishing rollback tid 2 to smithi04569269-2 2021-07-31T01:53:50.790 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 2 v 1066 (ObjNum 741 snap 228 seq_num 741) dirty exists 2021-07-31T01:53:50.790 INFO:tasks.rados.rados.0.smithi045.stdout:2486: expect (ObjNum 734 snap 227 seq_num 734) 2021-07-31T01:53:50.888 INFO:tasks.rados.rados.0.smithi045.stdout:2485: finishing write tid 3 to smithi04569269-18 2021-07-31T01:53:50.888 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 18 v 1003 (ObjNum 762 snap 234 seq_num 762) dirty exists 2021-07-31T01:53:50.888 INFO:tasks.rados.rados.0.smithi045.stdout:2485: left oid 18 (ObjNum 762 snap 234 seq_num 762) 2021-07-31T01:53:50.889 INFO:tasks.rados.rados.0.smithi045.stdout:2485: done (4 left) 2021-07-31T01:53:50.889 INFO:tasks.rados.rados.0.smithi045.stdout:2486: done (3 left) 2021-07-31T01:53:50.889 INFO:tasks.rados.rados.0.smithi045.stdout:2487: done (2 left) 2021-07-31T01:53:50.889 INFO:tasks.rados.rados.0.smithi045.stdout:2488: done (1 left) 2021-07-31T01:53:50.890 INFO:tasks.rados.rados.0.smithi045.stdout:2489: done (0 left) 2021-07-31T01:53:50.890 INFO:tasks.rados.rados.0.smithi045.stdout:2490: setattr oid 22 current snap is 234 2021-07-31T01:53:50.890 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 36 current snap is 234 2021-07-31T01:53:50.891 INFO:tasks.rados.rados.0.smithi045.stdout:2491: seq_num 764 ranges {0=802816} 2021-07-31T01:53:50.894 INFO:tasks.rados.rados.0.smithi045.stdout:2491: writing smithi04569269-36 from 0 to 802816 tid 1 2021-07-31T01:53:50.894 INFO:tasks.rados.rados.0.smithi045.stdout:2492: copy_from oid 7 from oid 37 current snap is 234 2021-07-31T01:53:50.894 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 22 v 1565 (ObjNum 747 snap 230 seq_num 747) dirty exists 2021-07-31T01:53:50.894 INFO:tasks.rados.rados.0.smithi045.stdout:2490: done (2 left) 2021-07-31T01:53:50.895 INFO:tasks.rados.rados.0.smithi045.stdout:2493: rollback oid 11 current snap is 234 2021-07-31T01:53:50.895 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 11 to 232 2021-07-31T01:53:50.895 INFO:tasks.rados.rados.0.smithi045.stdout:2494: setattr oid 8 current snap is 234 2021-07-31T01:53:50.896 INFO:tasks.rados.rados.0.smithi045.stdout:2493: finishing rollback tid 1 to smithi04569269-11 2021-07-31T01:53:50.896 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 11 v 0 (ObjNum 1315332980 snap 1818518616 seq_num 1499100264) dirty dne 2021-07-31T01:53:50.896 INFO:tasks.rados.rados.0.smithi045.stdout:2493: done (3 left) 2021-07-31T01:53:50.896 INFO:tasks.rados.rados.0.smithi045.stdout:2495: snap_create 2021-07-31T01:53:50.900 INFO:tasks.rados.rados.0.smithi045.stdout:2491: finishing write tid 1 to smithi04569269-36 2021-07-31T01:53:50.900 INFO:tasks.rados.rados.0.smithi045.stdout:2491: finishing write tid 2 to smithi04569269-36 2021-07-31T01:53:50.901 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 8 v 1616 (ObjNum 701 snap 218 seq_num 701) dirty exists 2021-07-31T01:53:50.901 INFO:tasks.rados.rados.0.smithi045.stdout:2491: finishing write tid 3 to smithi04569269-36 2021-07-31T01:53:50.902 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 36 v 1614 (ObjNum 764 snap 234 seq_num 764) dirty exists 2021-07-31T01:53:50.902 INFO:tasks.rados.rados.0.smithi045.stdout:2491: left oid 36 (ObjNum 764 snap 234 seq_num 764) 2021-07-31T01:53:50.927 INFO:tasks.rados.rados.0.smithi045.stdout:2492: finishing copy_from to smithi04569269-7 2021-07-31T01:53:50.928 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 7 v 991 (ObjNum 639 snap 199 seq_num 639) dirty exists 2021-07-31T01:53:50.928 INFO:tasks.rados.rados.0.smithi045.stdout:2492: finishing copy_from racing read to smithi04569269-7 2021-07-31T01:53:51.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:50 smithi154 conmon[34553]: cluster 2021-07-31T01:53:49.738388+0000 mon.a (mon.0 2021-07-31T01:53:51.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:50 smithi154 conmon[34553]: ) 1184 : cluster [DBG] osdmap e528: 8 total, 8 up, 8 in 2021-07-31T01:53:51.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:50 smithi154 conmon[34553]: cluster 2021-07-31T01:53:49.885778+0000 mgr.x (mgr.24542) 283 : cluster [DBG] pgmap v573: 25 pgs: 25 active+clean; 208 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 40 MiB/s rd, 6.6 MiB/s wr, 39 op/s 2021-07-31T01:53:51.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:50 smithi045 conmon[33655]: cluster 2021-07-31T01:53:49.738388+0000 mon.a (mon.0) 2021-07-31T01:53:51.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:50 smithi045 conmon[33655]: 1184 : cluster [DBG] osdmap e528: 8 total, 8 up, 8 in 2021-07-31T01:53:51.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:50 smithi045 conmon[33655]: cluster 2021-07-31T01:53:49.885778+0000 mgr.x (mgr.24542) 283 : cluster [DBG] pgmap v573: 25 pgs: 25 active+clean; 208 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 40 MiB/s rd, 6.6 MiB/s wr, 39 op/s 2021-07-31T01:53:51.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:50 smithi045 conmon[39718]: cluster 2021-07-31T01:53:49.738388+0000 mon.a (mon. 2021-07-31T01:53:51.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:50 smithi045 conmon[39718]: 0) 1184 : cluster [DBG] osdmap e528: 8 total, 8 up, 8 in 2021-07-31T01:53:51.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:50 smithi045 conmon[39718]: cluster 2021-07-31T01:53:49.885778+0000 mgr.x (mgr.24542) 283 : cluster 2021-07-31T01:53:51.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:50 smithi045 conmon[39718]: [DBG] pgmap v573: 25 pgs: 25 active+clean; 208 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 40 MiB/s rd, 6.6 MiB/s wr, 39 op/s 2021-07-31T01:53:51.597 INFO:tasks.rados.rados.0.smithi045.stdout:2491: done (3 left) 2021-07-31T01:53:51.597 INFO:tasks.rados.rados.0.smithi045.stdout:2492: done (2 left) 2021-07-31T01:53:51.597 INFO:tasks.rados.rados.0.smithi045.stdout:2494: done (1 left) 2021-07-31T01:53:51.598 INFO:tasks.rados.rados.0.smithi045.stdout:2495: done (0 left) 2021-07-31T01:53:51.598 INFO:tasks.rados.rados.0.smithi045.stdout:2496: read oid 18 snap -1 2021-07-31T01:53:51.598 INFO:tasks.rados.rados.0.smithi045.stdout:2496: expect (ObjNum 762 snap 234 seq_num 762) 2021-07-31T01:53:51.598 INFO:tasks.rados.rados.0.smithi045.stdout:2497: read oid 38 snap -1 2021-07-31T01:53:51.598 INFO:tasks.rados.rados.0.smithi045.stdout:2497: expect deleted 2021-07-31T01:53:51.599 INFO:tasks.rados.rados.0.smithi045.stdout:2498: read oid 10 snap -1 2021-07-31T01:53:51.599 INFO:tasks.rados.rados.0.smithi045.stdout:2498: expect (ObjNum 757 snap 233 seq_num 757) 2021-07-31T01:53:51.599 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 34 current snap is 235 2021-07-31T01:53:51.599 INFO:tasks.rados.rados.0.smithi045.stdout:2499: seq_num 765 ranges {9256960=417792,9674752=401408} 2021-07-31T01:53:51.650 INFO:tasks.rados.rados.0.smithi045.stdout:2499: writing smithi04569269-34 from 9256960 to 9674752 tid 1 2021-07-31T01:53:51.652 INFO:tasks.rados.rados.0.smithi045.stdout:2499: writing smithi04569269-34 from 9674752 to 10076160 tid 2 2021-07-31T01:53:51.652 INFO:tasks.rados.rados.0.smithi045.stdout:2500: snap_remove snap 232 2021-07-31T01:53:52.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:51 smithi154 conmon[34553]: cluster 2021-07-31T01:53:50.744283+0000 mon.a (mon.0) 1185 : cluster [DBG] osdmap e529: 8 total, 8 up, 8 in 2021-07-31T01:53:52.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:51 smithi154 conmon[34553]: cluster 2021-07-31T01:53:51.595830+0000 mon.a (mon.0) 1186 : cluster [DBG] osdmap e530: 8 total, 8 up, 8 in 2021-07-31T01:53:52.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:51 smithi045 conmon[33655]: cluster 2021-07-31T01:53:50.744283+0000 mon.a (mon 2021-07-31T01:53:52.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:51 smithi045 conmon[33655]: .0) 1185 : cluster [DBG] osdmap e529: 8 total, 8 up, 8 in 2021-07-31T01:53:52.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:51 smithi045 conmon[33655]: cluster 2021-07-31T01:53:51.595830+0000 mon.a (mon.0) 1186 : cluster [DBG] osdmap e530: 8 total, 8 up, 8 in 2021-07-31T01:53:52.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:51 smithi045 conmon[39718]: cluster 2021-07-31T01:53:50.744283 2021-07-31T01:53:52.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:51 smithi045 conmon[39718]: +0000 mon.a (mon.0) 1185 : cluster [DBG] osdmap e529: 8 total, 8 up, 8 in 2021-07-31T01:53:52.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:51 smithi045 conmon[39718]: cluster 2021-07 2021-07-31T01:53:52.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:51 smithi045 conmon[39718]: -31T01:53:51.595830+0000 mon.a (mon.0) 1186 : cluster [DBG] osdmap e530: 8 total, 8 up, 8 in 2021-07-31T01:53:52.604 INFO:tasks.rados.rados.0.smithi045.stdout:2500: done (4 left) 2021-07-31T01:53:52.604 INFO:tasks.rados.rados.0.smithi045.stdout:2501: setattr oid 46 current snap is 235 2021-07-31T01:53:52.605 INFO:tasks.rados.rados.0.smithi045.stdout:2502: read oid 32 snap -1 2021-07-31T01:53:52.606 INFO:tasks.rados.rados.0.smithi045.stdout:2502: expect (ObjNum 687 snap 214 seq_num 687) 2021-07-31T01:53:52.606 INFO:tasks.rados.rados.0.smithi045.stdout:2503: rollback oid 29 current snap is 235 2021-07-31T01:53:52.606 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 29 to 234 2021-07-31T01:53:52.606 INFO:tasks.rados.rados.0.smithi045.stdout:2504: delete oid 37 current snap is 235 2021-07-31T01:53:52.606 INFO:tasks.rados.rados.0.smithi045.stdout:2496: expect (ObjNum 762 snap 234 seq_num 762) 2021-07-31T01:53:52.733 INFO:tasks.rados.rados.0.smithi045.stdout:2498: expect (ObjNum 757 snap 233 seq_num 757) 2021-07-31T01:53:53.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:52 smithi154 conmon[34553]: cluster 2021-07-31T01:53:51.886162+0000 mgr.x (mgr. 2021-07-31T01:53:53.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:52 smithi154 conmon[34553]: 24542) 284 : cluster [DBG] pgmap v576: 25 pgs: 25 active+clean; 208 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 40 MiB/s rd, 6.6 MiB/s wr, 39 op/s 2021-07-31T01:53:53.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:52 smithi154 conmon[34553]: cluster 2021-07-31T01:53:52.602267+0000 mon.a (mon.0) 1187 : cluster [DBG] osdmap e531: 8 total, 8 up, 8 in 2021-07-31T01:53:53.004 INFO:journalctl@ceph.osd.4.smithi154.stdout:Jul 31 01:53:52 smithi154 conmon[38506]: debug 2021-07-31T01:53:52.601+0000 7feae39cf700 -1 osd.4 pg_epoch: 531 pg[3.9s0( v 529'394 (0'0,529'394] local-lis/les=56/57 n=4 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [4,2,7]p4(0) r=0 lpr=56 crt=529'394 lcod 529'392 mlcod 529'392 active+clean+snaptrim trimq=[1d0~1] ps=[1cf~1,1d2~1]] removing snap head 2021-07-31T01:53:53.004 INFO:journalctl@ceph.osd.4.smithi154.stdout:Jul 31 01:53:52 smithi154 conmon[38506]: debug 2021-07-31T01:53:52.601+0000 7feae69d5700 -1 osd.4 pg_epoch: 531 pg[3.3s0( v 529'221 (0'0,529'221] local-lis/les=56/57 n=2 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [4,1,0]p4(0) r=0 lpr=56 crt=529'221 lcod 526'219 mlcod 526'219 active+clean+snaptrim trimq=[1d0~1] ps=[1cf~1,1d2~1]] removing snap head 2021-07-31T01:53:53.005 INFO:journalctl@ceph.osd.7.smithi154.stdout:Jul 31 01:53:52 smithi154 conmon[49149]: debug 2021-07-31T01:53:52.604+0000 7f8a75203700 -1 osd.7 pg_epoch: 531 pg[3.es0( v 529'981 (0'0,529'981] local-lis/les=56/57 n=5 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [7,4,1]p7(0) r=0 lpr=56 crt=529'981 lcod 529'979 mlcod 529'979 active+clean+snaptrim trimq=[1d0~1] ps=[1cf~1,1d2~1]] removing snap head 2021-07-31T01:53:53.026 INFO:journalctl@ceph.osd.3.smithi045.stdout:Jul 31 01:53:52 smithi045 conmon[54013]: debug 2021-07-31T01:53:52.614+0000 7f64fc8cd700 -1 osd.3 pg_epoch: 531 pg[3.bs0( v 529'1616 (0'0,529'1616] local-lis/les=56/57 n=11 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [3,0,4]p3(0) r=0 lpr=56 crt=529'1616 lcod 529'1614 mlcod 529'1614 active+clean+snaptrim trimq=[1d0~1] ps=[1cf~1,1d2~1]] removing snap head 2021-07-31T01:53:53.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:52 smithi045 conmon[39718]: cluster 2021-07-31T01:53:51.886162+0000 mgr.x (mgr. 2021-07-31T01:53:53.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:52 smithi045 conmon[39718]: 24542) 284 : cluster [DBG] pgmap v576: 25 pgs: 25 active+clean; 208 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 40 MiB/s rd, 6.6 MiB/s wr, 39 op/s 2021-07-31T01:53:53.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:52 smithi045 conmon[39718]: cluster 2021-07-31T01:53:52.602267+0000 mon.a 2021-07-31T01:53:53.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:52 smithi045 conmon[39718]: (mon.0) 1187 : cluster [DBG] osdmap e531: 8 total, 8 up, 8 in 2021-07-31T01:53:53.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:52 smithi045 conmon[33655]: cluster 2021-07-31T01:53:51.886162+0000 mgr.x 2021-07-31T01:53:53.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:52 smithi045 conmon[33655]: (mgr.24542) 284 : cluster [DBG] pgmap v576: 25 pgs: 25 active+clean; 208 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 40 MiB/s rd, 6.6 MiB/s wr, 39 op/s 2021-07-31T01:53:53.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:52 smithi045 conmon[33655]: cluster 2021-07-31T01:53:52.602267+0000 mon.a (mon. 2021-07-31T01:53:53.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:52 smithi045 conmon[33655]: 0) 1187 : cluster [DBG] osdmap e531: 8 total, 8 up, 8 in 2021-07-31T01:53:53.410 INFO:tasks.rados.rados.0.smithi045.stdout:2499: finishing write tid 1 to smithi04569269-34 2021-07-31T01:53:53.410 INFO:tasks.rados.rados.0.smithi045.stdout:2499: finishing write tid 2 to smithi04569269-34 2021-07-31T01:53:53.411 INFO:tasks.rados.rados.0.smithi045.stdout:2499: finishing write tid 3 to smithi04569269-34 2021-07-31T01:53:53.411 INFO:tasks.rados.rados.0.smithi045.stdout:2499: finishing write tid 4 to smithi04569269-34 2021-07-31T01:53:53.411 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 34 v 1066 (ObjNum 765 snap 235 seq_num 765) dirty exists 2021-07-31T01:53:53.411 INFO:tasks.rados.rados.0.smithi045.stdout:2499: left oid 34 (ObjNum 765 snap 235 seq_num 765) 2021-07-31T01:53:53.412 INFO:tasks.rados.rados.0.smithi045.stdout:2496: done (7 left) 2021-07-31T01:53:53.412 INFO:tasks.rados.rados.0.smithi045.stdout:2497: done (6 left) 2021-07-31T01:53:53.412 INFO:tasks.rados.rados.0.smithi045.stdout:2498: done (5 left) 2021-07-31T01:53:53.413 INFO:tasks.rados.rados.0.smithi045.stdout:2499: done (4 left) 2021-07-31T01:53:53.413 INFO:tasks.rados.rados.0.smithi045.stdout:2504: done (3 left) 2021-07-31T01:53:53.413 INFO:tasks.rados.rados.0.smithi045.stdout:2505: rollback oid 6 current snap is 235 2021-07-31T01:53:53.413 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 6 to 234 2021-07-31T01:53:53.413 INFO:tasks.rados.rados.0.smithi045.stdout:2503: finishing rollback tid 1 to smithi04569269-29 2021-07-31T01:53:53.414 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 29 v 0 (ObjNum 0 snap 0 seq_num 0) dirty dne 2021-07-31T01:53:53.414 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 46 v 1298 (ObjNum 665 snap 208 seq_num 665) dirty exists 2021-07-31T01:53:53.414 INFO:tasks.rados.rados.0.smithi045.stdout:2502: expect (ObjNum 687 snap 214 seq_num 687) 2021-07-31T01:53:53.673 INFO:tasks.rados.rados.0.smithi045.stdout:2505: finishing rollback tid 1 to smithi04569269-6 2021-07-31T01:53:53.674 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 6 v 0 (ObjNum -1105764752 snap 21873 seq_num 3189064368) dirty dne 2021-07-31T01:53:53.674 INFO:tasks.rados.rados.0.smithi045.stdout:2501: done (3 left) 2021-07-31T01:53:53.674 INFO:tasks.rados.rados.0.smithi045.stdout:2502: done (2 left) 2021-07-31T01:53:53.674 INFO:tasks.rados.rados.0.smithi045.stdout:2503: done (1 left) 2021-07-31T01:53:53.674 INFO:tasks.rados.rados.0.smithi045.stdout:2505: done (0 left) 2021-07-31T01:53:53.675 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 7 current snap is 235 2021-07-31T01:53:53.675 INFO:tasks.rados.rados.0.smithi045.stdout:2506: seq_num 767 ranges {6774784=450560,7225344=401408} 2021-07-31T01:53:53.713 INFO:tasks.rados.rados.0.smithi045.stdout:2506: writing smithi04569269-7 from 6774784 to 7225344 tid 1 2021-07-31T01:53:53.714 INFO:tasks.rados.rados.0.smithi045.stdout:2506: writing smithi04569269-7 from 7225344 to 7626752 tid 2 2021-07-31T01:53:53.715 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 10 current snap is 235 2021-07-31T01:53:53.715 INFO:tasks.rados.rados.0.smithi045.stdout:2507: seq_num 768 ranges {9175040=466944,9641984=401408} 2021-07-31T01:53:53.768 INFO:tasks.rados.rados.0.smithi045.stdout:2507: writing smithi04569269-10 from 9175040 to 9641984 tid 1 2021-07-31T01:53:53.769 INFO:tasks.rados.rados.0.smithi045.stdout:2507: writing smithi04569269-10 from 9641984 to 10043392 tid 2 2021-07-31T01:53:53.769 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 18 current snap is 235 2021-07-31T01:53:53.770 INFO:tasks.rados.rados.0.smithi045.stdout:2508: seq_num 769 ranges {3170304=483328,3653632=401408} 2021-07-31T01:53:53.775 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:53:53 smithi045 conmon[61210]: level=error ts=2021-07-31T01:53:53.520Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:53:53.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:53:53 smithi045 conmon[61210]: level=error ts=2021-07-31T01:53:53.520Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:53:53.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:53:53 smithi045 conmon[61210]: level=error ts=2021-07-31T01:53:53.520Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T01:53:53.789 INFO:tasks.rados.rados.0.smithi045.stdout:2508: writing smithi04569269-18 from 3170304 to 3653632 tid 1 2021-07-31T01:53:53.792 INFO:tasks.rados.rados.0.smithi045.stdout:2508: writing smithi04569269-18 from 3653632 to 4055040 tid 2 2021-07-31T01:53:53.793 INFO:tasks.rados.rados.0.smithi045.stdout:2509: snap_create 2021-07-31T01:53:53.793 INFO:tasks.rados.rados.0.smithi045.stdout:2506: finishing write tid 1 to smithi04569269-7 2021-07-31T01:53:53.793 INFO:tasks.rados.rados.0.smithi045.stdout:2506: finishing write tid 2 to smithi04569269-7 2021-07-31T01:53:53.794 INFO:tasks.rados.rados.0.smithi045.stdout:2506: finishing write tid 3 to smithi04569269-7 2021-07-31T01:53:53.794 INFO:tasks.rados.rados.0.smithi045.stdout:2506: finishing write tid 4 to smithi04569269-7 2021-07-31T01:53:53.794 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 7 v 1002 (ObjNum 767 snap 235 seq_num 767) dirty exists 2021-07-31T01:53:53.794 INFO:tasks.rados.rados.0.smithi045.stdout:2506: left oid 7 (ObjNum 767 snap 235 seq_num 767) 2021-07-31T01:53:53.794 INFO:tasks.rados.rados.0.smithi045.stdout:2507: finishing write tid 1 to smithi04569269-10 2021-07-31T01:53:53.795 INFO:tasks.rados.rados.0.smithi045.stdout:2507: finishing write tid 2 to smithi04569269-10 2021-07-31T01:53:53.795 INFO:tasks.rados.rados.0.smithi045.stdout:2507: finishing write tid 3 to smithi04569269-10 2021-07-31T01:53:53.795 INFO:tasks.rados.rados.0.smithi045.stdout:2507: finishing write tid 4 to smithi04569269-10 2021-07-31T01:53:53.795 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 10 v 1575 (ObjNum 768 snap 235 seq_num 768) dirty exists 2021-07-31T01:53:53.796 INFO:tasks.rados.rados.0.smithi045.stdout:2507: left oid 10 (ObjNum 768 snap 235 seq_num 768) 2021-07-31T01:53:53.797 INFO:tasks.rados.rados.0.smithi045.stdout:2508: finishing write tid 1 to smithi04569269-18 2021-07-31T01:53:53.797 INFO:tasks.rados.rados.0.smithi045.stdout:2508: finishing write tid 2 to smithi04569269-18 2021-07-31T01:53:53.797 INFO:tasks.rados.rados.0.smithi045.stdout:2508: finishing write tid 3 to smithi04569269-18 2021-07-31T01:53:53.797 INFO:tasks.rados.rados.0.smithi045.stdout:2508: finishing write tid 4 to smithi04569269-18 2021-07-31T01:53:53.797 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 18 v 1007 (ObjNum 769 snap 235 seq_num 769) dirty exists 2021-07-31T01:53:53.798 INFO:tasks.rados.rados.0.smithi045.stdout:2508: left oid 18 (ObjNum 769 snap 235 seq_num 769) 2021-07-31T01:53:53.847 INFO:tasks.rados.rados.0.smithi045.stdout:2506: done (3 left) 2021-07-31T01:53:53.847 INFO:tasks.rados.rados.0.smithi045.stdout:2507: done (2 left) 2021-07-31T01:53:53.847 INFO:tasks.rados.rados.0.smithi045.stdout:2508: done (1 left) 2021-07-31T01:53:53.848 INFO:tasks.rados.rados.0.smithi045.stdout:2509: done (0 left) 2021-07-31T01:53:53.848 INFO:tasks.rados.rados.0.smithi045.stdout:2510: copy_from oid 48 from oid 43 current snap is 236 2021-07-31T01:53:53.848 INFO:tasks.rados.rados.0.smithi045.stdout:2511: setattr oid 7 current snap is 236 2021-07-31T01:53:53.849 INFO:tasks.rados.rados.0.smithi045.stdout:2512: read oid 36 snap 234 2021-07-31T01:53:53.849 INFO:tasks.rados.rados.0.smithi045.stdout:2512: expect (ObjNum 764 snap 234 seq_num 764) 2021-07-31T01:53:53.849 INFO:tasks.rados.rados.0.smithi045.stdout:2513: read oid 25 snap -1 2021-07-31T01:53:53.850 INFO:tasks.rados.rados.0.smithi045.stdout:2513: expect (ObjNum 678 snap 211 seq_num 678) 2021-07-31T01:53:53.850 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 44 current snap is 236 2021-07-31T01:53:53.850 INFO:tasks.rados.rados.0.smithi045.stdout:2514: seq_num 770 ranges {0=499712,499712=401408} 2021-07-31T01:53:53.852 INFO:tasks.rados.rados.0.smithi045.stdout:2514: writing smithi04569269-44 from 0 to 499712 tid 1 2021-07-31T01:53:53.855 INFO:tasks.rados.rados.0.smithi045.stdout:2514: writing smithi04569269-44 from 499712 to 901120 tid 2 2021-07-31T01:53:53.855 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 24 current snap is 236 2021-07-31T01:53:53.855 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 7 v 1003 (ObjNum 767 snap 235 seq_num 767) dirty exists 2021-07-31T01:53:53.855 INFO:tasks.rados.rados.0.smithi045.stdout:2515: seq_num 771 ranges {565248=516096,1081344=401408} 2021-07-31T01:53:53.861 INFO:tasks.rados.rados.0.smithi045.stdout:2515: writing smithi04569269-24 from 565248 to 1081344 tid 1 2021-07-31T01:53:53.864 INFO:tasks.rados.rados.0.smithi045.stdout:2515: writing smithi04569269-24 from 1081344 to 1482752 tid 2 2021-07-31T01:53:53.865 INFO:tasks.rados.rados.0.smithi045.stdout:2511: done (5 left) 2021-07-31T01:53:53.866 INFO:tasks.rados.rados.0.smithi045.stdout:2516: rmattr oid 10 current snap is 236 2021-07-31T01:53:53.866 INFO:tasks.rados.rados.0.smithi045.stdout:2512: expect (ObjNum 764 snap 234 seq_num 764) 2021-07-31T01:53:53.894 INFO:tasks.rados.rados.0.smithi045.stdout:2514: finishing write tid 1 to smithi04569269-44 2021-07-31T01:53:53.894 INFO:tasks.rados.rados.0.smithi045.stdout:2514: finishing write tid 2 to smithi04569269-44 2021-07-31T01:53:53.895 INFO:tasks.rados.rados.0.smithi045.stdout:2514: finishing write tid 3 to smithi04569269-44 2021-07-31T01:53:53.895 INFO:tasks.rados.rados.0.smithi045.stdout:2514: finishing write tid 4 to smithi04569269-44 2021-07-31T01:53:53.895 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 44 v 1404 (ObjNum 770 snap 236 seq_num 770) dirty exists 2021-07-31T01:53:53.895 INFO:tasks.rados.rados.0.smithi045.stdout:2514: left oid 44 (ObjNum 770 snap 236 seq_num 770) 2021-07-31T01:53:53.896 INFO:tasks.rados.rados.0.smithi045.stdout:2512: done (5 left) 2021-07-31T01:53:53.896 INFO:tasks.rados.rados.0.smithi045.stdout:2514: done (4 left) 2021-07-31T01:53:53.896 INFO:tasks.rados.rados.0.smithi045.stdout:2517: rmattr oid 5 current snap is 236 2021-07-31T01:53:53.896 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 10 v 1576 (ObjNum 768 snap 235 seq_num 768) dirty exists 2021-07-31T01:53:53.896 INFO:tasks.rados.rados.0.smithi045.stdout:2515: finishing write tid 1 to smithi04569269-24 2021-07-31T01:53:53.897 INFO:tasks.rados.rados.0.smithi045.stdout:2515: finishing write tid 2 to smithi04569269-24 2021-07-31T01:53:53.897 INFO:tasks.rados.rados.0.smithi045.stdout:2515: finishing write tid 3 to smithi04569269-24 2021-07-31T01:53:53.897 INFO:tasks.rados.rados.0.smithi045.stdout:2515: finishing write tid 4 to smithi04569269-24 2021-07-31T01:53:53.897 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 24 v 1596 (ObjNum 771 snap 236 seq_num 771) dirty exists 2021-07-31T01:53:53.898 INFO:tasks.rados.rados.0.smithi045.stdout:2515: left oid 24 (ObjNum 771 snap 236 seq_num 771) 2021-07-31T01:53:53.898 INFO:tasks.rados.rados.0.smithi045.stdout:2513: expect (ObjNum 678 snap 211 seq_num 678) 2021-07-31T01:53:54.382 INFO:tasks.rados.rados.0.smithi045.stdout:2510: finishing copy_from to smithi04569269-48 2021-07-31T01:53:54.382 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 48 v 1401 (ObjNum 754 snap 233 seq_num 754) dirty exists 2021-07-31T01:53:54.382 INFO:tasks.rados.rados.0.smithi045.stdout:2510: finishing copy_from racing read to smithi04569269-48 2021-07-31T01:53:54.382 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 5 v 1402 (ObjNum 754 snap 233 seq_num 754) dirty exists 2021-07-31T01:53:54.383 INFO:tasks.rados.rados.0.smithi045.stdout:2510: done (4 left) 2021-07-31T01:53:54.383 INFO:tasks.rados.rados.0.smithi045.stdout:2513: done (3 left) 2021-07-31T01:53:54.383 INFO:tasks.rados.rados.0.smithi045.stdout:2515: done (2 left) 2021-07-31T01:53:54.383 INFO:tasks.rados.rados.0.smithi045.stdout:2516: done (1 left) 2021-07-31T01:53:54.384 INFO:tasks.rados.rados.0.smithi045.stdout:2517: done (0 left) 2021-07-31T01:53:54.384 INFO:tasks.rados.rados.0.smithi045.stdout:2518: read oid 19 snap 235 2021-07-31T01:53:54.384 INFO:tasks.rados.rados.0.smithi045.stdout:2518: expect deleted 2021-07-31T01:53:54.384 INFO:tasks.rados.rados.0.smithi045.stdout:2519: delete oid 18 current snap is 236 2021-07-31T01:53:54.385 INFO:tasks.rados.rados.0.smithi045.stdout:2518: done (1 left) 2021-07-31T01:53:54.385 INFO:tasks.rados.rados.0.smithi045.stdout:2519: done (0 left) 2021-07-31T01:53:54.386 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 14 current snap is 236 2021-07-31T01:53:54.386 INFO:tasks.rados.rados.0.smithi045.stdout:2520: seq_num 773 ranges {401408=548864,950272=401408} 2021-07-31T01:53:54.389 INFO:tasks.rados.rados.0.smithi045.stdout:2520: writing smithi04569269-14 from 401408 to 950272 tid 1 2021-07-31T01:53:54.391 INFO:tasks.rados.rados.0.smithi045.stdout:2520: writing smithi04569269-14 from 950272 to 1351680 tid 2 2021-07-31T01:53:54.391 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 15 current snap is 236 2021-07-31T01:53:54.392 INFO:tasks.rados.rados.0.smithi045.stdout:2521: seq_num 774 ranges {7856128=565248,8421376=401408} 2021-07-31T01:53:54.437 INFO:tasks.rados.rados.0.smithi045.stdout:2521: writing smithi04569269-15 from 7856128 to 8421376 tid 1 2021-07-31T01:53:54.440 INFO:tasks.rados.rados.0.smithi045.stdout:2521: writing smithi04569269-15 from 8421376 to 8822784 tid 2 2021-07-31T01:53:54.440 INFO:tasks.rados.rados.0.smithi045.stdout:2522: read oid 21 snap 235 2021-07-31T01:53:54.441 INFO:tasks.rados.rados.0.smithi045.stdout:2522: expect deleted 2021-07-31T01:53:54.441 INFO:tasks.rados.rados.0.smithi045.stdout:2523: snap_remove snap 234 2021-07-31T01:53:54.853 INFO:tasks.rados.rados.0.smithi045.stdout:2523: done (3 left) 2021-07-31T01:53:54.854 INFO:tasks.rados.rados.0.smithi045.stdout:2524: delete oid 33 current snap is 236 2021-07-31T01:53:54.854 INFO:tasks.rados.rados.0.smithi045.stdout:2520: finishing write tid 1 to smithi04569269-14 2021-07-31T01:53:54.854 INFO:tasks.rados.rados.0.smithi045.stdout:2520: finishing write tid 2 to smithi04569269-14 2021-07-31T01:53:54.855 INFO:tasks.rados.rados.0.smithi045.stdout:2520: finishing write tid 3 to smithi04569269-14 2021-07-31T01:53:54.855 INFO:tasks.rados.rados.0.smithi045.stdout:2520: finishing write tid 4 to smithi04569269-14 2021-07-31T01:53:54.855 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 14 v 1407 (ObjNum 773 snap 236 seq_num 773) dirty exists 2021-07-31T01:53:54.855 INFO:tasks.rados.rados.0.smithi045.stdout:2520: left oid 14 (ObjNum 773 snap 236 seq_num 773) 2021-07-31T01:53:54.856 INFO:tasks.rados.rados.0.smithi045.stdout:2521: finishing write tid 1 to smithi04569269-15 2021-07-31T01:53:54.856 INFO:tasks.rados.rados.0.smithi045.stdout:2521: finishing write tid 2 to smithi04569269-15 2021-07-31T01:53:54.856 INFO:tasks.rados.rados.0.smithi045.stdout:2521: finishing write tid 3 to smithi04569269-15 2021-07-31T01:53:54.856 INFO:tasks.rados.rados.0.smithi045.stdout:2521: finishing write tid 4 to smithi04569269-15 2021-07-31T01:53:54.856 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 15 v 1011 (ObjNum 774 snap 236 seq_num 774) dirty exists 2021-07-31T01:53:54.857 INFO:tasks.rados.rados.0.smithi045.stdout:2521: left oid 15 (ObjNum 774 snap 236 seq_num 774) 2021-07-31T01:53:54.862 INFO:tasks.rados.rados.0.smithi045.stdout:2520: done (3 left) 2021-07-31T01:53:54.863 INFO:tasks.rados.rados.0.smithi045.stdout:2521: done (2 left) 2021-07-31T01:53:54.863 INFO:tasks.rados.rados.0.smithi045.stdout:2522: done (1 left) 2021-07-31T01:53:54.863 INFO:tasks.rados.rados.0.smithi045.stdout:2524: done (0 left) 2021-07-31T01:53:54.863 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 44 current snap is 236 2021-07-31T01:53:54.863 INFO:tasks.rados.rados.0.smithi045.stdout:2525: seq_num 776 ranges {901120=606208,1507328=401408} 2021-07-31T01:53:54.870 INFO:tasks.rados.rados.0.smithi045.stdout:2525: writing smithi04569269-44 from 901120 to 1507328 tid 1 2021-07-31T01:53:54.872 INFO:tasks.rados.rados.0.smithi045.stdout:2525: writing smithi04569269-44 from 1507328 to 1908736 tid 2 2021-07-31T01:53:54.873 INFO:tasks.rados.rados.0.smithi045.stdout:2526: snap_create 2021-07-31T01:53:54.875 INFO:tasks.rados.rados.0.smithi045.stdout:2525: finishing write tid 1 to smithi04569269-44 2021-07-31T01:53:54.876 INFO:tasks.rados.rados.0.smithi045.stdout:2525: finishing write tid 2 to smithi04569269-44 2021-07-31T01:53:54.876 INFO:tasks.rados.rados.0.smithi045.stdout:2525: finishing write tid 3 to smithi04569269-44 2021-07-31T01:53:54.877 INFO:tasks.rados.rados.0.smithi045.stdout:2525: finishing write tid 4 to smithi04569269-44 2021-07-31T01:53:54.877 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 44 v 1410 (ObjNum 776 snap 236 seq_num 776) dirty exists 2021-07-31T01:53:54.878 INFO:tasks.rados.rados.0.smithi045.stdout:2525: left oid 44 (ObjNum 776 snap 236 seq_num 776) 2021-07-31T01:53:55.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:54 smithi154 conmon[34553]: cluster 2021-07-31T01:53:53.846614+0000 mon.a (mon.0) 1188 : cluster [DBG] osdmap e532: 8 total, 8 up, 8 in 2021-07-31T01:53:55.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:54 smithi154 conmon[34553]: cluster 2021-07-31T01:53:53.886532+0000 mgr.x (mgr.24542) 285 : cluster [DBG] pgmap v579: 25 pgs: 25 active+clean; 180 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 2.8 MiB/s wr, 28 op/s 2021-07-31T01:53:55.276 INFO:journalctl@ceph.osd.1.smithi045.stdout:Jul 31 01:53:54 smithi045 conmon[46771]: debug 2021-07-31T01:53:54.857+0000 7f41939a2700 -1 osd.1 pg_epoch: 533 pg[3.0s0( v 532'677 (0'0,532'677] local-lis/les=56/57 n=8 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [1,2,6]p1(0) r=0 lpr=56 crt=532'677 lcod 532'676 mlcod 532'676 active+clean+snaptrim trimq=[1d5~1] ps=[1cf~2,1d2~1]] removing snap head 2021-07-31T01:53:55.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:54 smithi045 conmon[33655]: cluster 2021-07-31T01:53:53.846614+0000 mon.a (mon. 2021-07-31T01:53:55.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:54 smithi045 conmon[33655]: 0) 1188 : cluster [DBG] osdmap e532: 8 total, 8 up, 8 in 2021-07-31T01:53:55.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:54 smithi045 conmon[33655]: cluster 2021-07-31T01:53:53.886532+0000 mgr.x (mgr.24542) 285 : cluster [DBG] pgmap v579: 25 pgs: 25 active+clean; 180 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 2.8 MiB/s wr, 28 op/s 2021-07-31T01:53:55.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:54 smithi045 conmon[39718]: cluster 2021-07-31T01:53: 2021-07-31T01:53:55.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:54 smithi045 conmon[39718]: 53.846614+0000 mon.a (mon.0) 1188 : cluster [DBG] osdmap e532: 8 total, 8 up, 8 in 2021-07-31T01:53:55.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:54 smithi045 conmon[39718]: cluster 2021-07-31T 2021-07-31T01:53:55.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:54 smithi045 conmon[39718]: 01:53:53.886532+0000 mgr.x (mgr.24542) 2021-07-31T01:53:55.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:54 smithi045 conmon[39718]: 285 : cluster [DBG] pgmap v579: 25 pgs: 25 active+clean; 180 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 2.8 MiB/s wr, 28 op/s 2021-07-31T01:53:55.859 INFO:tasks.rados.rados.0.smithi045.stdout:2525: done (1 left) 2021-07-31T01:53:55.860 INFO:tasks.rados.rados.0.smithi045.stdout:2526: done (0 left) 2021-07-31T01:53:55.860 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 13 current snap is 237 2021-07-31T01:53:55.860 INFO:tasks.rados.rados.0.smithi045.stdout:2527: seq_num 777 ranges {16637952=622592,17260544=401408} 2021-07-31T01:53:55.954 INFO:tasks.rados.rados.0.smithi045.stdout:2527: writing smithi04569269-13 from 16637952 to 17260544 tid 1 2021-07-31T01:53:55.956 INFO:tasks.rados.rados.0.smithi045.stdout:2527: writing smithi04569269-13 from 17260544 to 17661952 tid 2 2021-07-31T01:53:55.956 INFO:tasks.rados.rados.0.smithi045.stdout:2528: copy_from oid 41 from oid 46 current snap is 237 2021-07-31T01:53:55.957 INFO:tasks.rados.rados.0.smithi045.stdout:2529: snap_remove snap 235 2021-07-31T01:53:56.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:55 smithi154 conmon[34553]: cluster 2021-07-31T01:53:54.852949+0000 mon.a (mon.0) 1189 : cluster [DBG] osdmap e533: 8 total, 8 up, 8 in 2021-07-31T01:53:56.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:55 smithi045 conmon[39718]: cluster 2021-07-31T01:53:54.852949+0000 mon.a (mon.0) 1189 : cluster [DBG] osdmap e533: 8 total, 8 up, 8 in 2021-07-31T01:53:56.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:55 smithi045 conmon[33655]: cluster 2021-07-31T01:53:54.852949+0000 2021-07-31T01:53:56.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:55 smithi045 conmon[33655]: mon.a (mon.0) 1189 : cluster [DBG] osdmap e533: 8 total, 8 up, 8 in 2021-07-31T01:53:56.598 INFO:tasks.rados.rados.0.smithi045.stdout:2529: done (2 left) 2021-07-31T01:53:56.598 INFO:tasks.rados.rados.0.smithi045.stdout:2530: read oid 23 snap -1 2021-07-31T01:53:56.599 INFO:tasks.rados.rados.0.smithi045.stdout:2530: expect deleted 2021-07-31T01:53:56.599 INFO:tasks.rados.rados.0.smithi045.stdout:2531: rollback oid 28 current snap is 237 2021-07-31T01:53:56.599 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 28 to 236 2021-07-31T01:53:56.600 INFO:tasks.rados.rados.0.smithi045.stdout:2527: finishing write tid 1 to smithi04569269-13 2021-07-31T01:53:56.600 INFO:tasks.rados.rados.0.smithi045.stdout:2527: finishing write tid 2 to smithi04569269-13 2021-07-31T01:53:56.601 INFO:tasks.rados.rados.0.smithi045.stdout:2527: finishing write tid 3 to smithi04569269-13 2021-07-31T01:53:56.601 INFO:tasks.rados.rados.0.smithi045.stdout:2527: finishing write tid 4 to smithi04569269-13 2021-07-31T01:53:56.601 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 13 v 1577 (ObjNum 777 snap 237 seq_num 777) dirty exists 2021-07-31T01:53:56.601 INFO:tasks.rados.rados.0.smithi045.stdout:2527: left oid 13 (ObjNum 777 snap 237 seq_num 777) 2021-07-31T01:53:56.602 INFO:tasks.rados.rados.0.smithi045.stdout:2528: finishing copy_from to smithi04569269-41 2021-07-31T01:53:56.602 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 41 v 1298 (ObjNum 665 snap 208 seq_num 665) dirty exists 2021-07-31T01:53:56.602 INFO:tasks.rados.rados.0.smithi045.stdout:2527: done (3 left) 2021-07-31T01:53:56.602 INFO:tasks.rados.rados.0.smithi045.stdout:2532: setattr oid 4 current snap is 237 2021-07-31T01:53:56.603 INFO:tasks.rados.rados.0.smithi045.stdout:2528: finishing copy_from racing read to smithi04569269-41 2021-07-31T01:53:56.604 INFO:tasks.rados.rados.0.smithi045.stdout:2531: finishing rollback tid 1 to smithi04569269-28 2021-07-31T01:53:56.604 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 28 v 0 (ObjNum -1126011800 snap 21873 seq_num 1) dirty dne 2021-07-31T01:53:56.604 INFO:tasks.rados.rados.0.smithi045.stdout:2528: done (3 left) 2021-07-31T01:53:56.604 INFO:tasks.rados.rados.0.smithi045.stdout:2531: done (2 left) 2021-07-31T01:53:56.605 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 40 current snap is 237 2021-07-31T01:53:56.605 INFO:tasks.rados.rados.0.smithi045.stdout:2533: seq_num 778 ranges {0=638976,638976=401408} 2021-07-31T01:53:56.605 INFO:tasks.rados.rados.0.smithi045.stdout:2533: writing smithi04569269-40 from 0 to 638976 tid 1 2021-07-31T01:53:56.608 INFO:tasks.rados.rados.0.smithi045.stdout:2533: writing smithi04569269-40 from 638976 to 1040384 tid 2 2021-07-31T01:53:56.609 INFO:tasks.rados.rados.0.smithi045.stdout:2534: snap_remove snap 236 2021-07-31T01:53:56.862 INFO:journalctl@ceph.osd.1.smithi045.stdout:Jul 31 01:53:56 smithi045 conmon[46771]: debug 2021-07-31T01:53:56.862 INFO:journalctl@ceph.osd.1.smithi045.stdout:Jul 31 01:53:56 smithi045 conmon[46771]: 2021-07-31T01:53:56.609+0000 7f41979aa700 -1 osd.1 pg_epoch: 535 pg[3.0s0( v 535'687 (0'0,535'687] local-lis/les=56/57 n=5 ec=56/56 lis/c=56/56 les/c/f=57/57/0 sis=56) [1,2,6]p1(0) r=0 lpr=56 luod=533'683 crt=533'683 lcod 533'681 mlcod 533'681 active+clean+snaptrim trimq=[1d7~1] ps=[1d5~1]] removing snap head 2021-07-31T01:53:56.863 INFO:journalctl@ceph.osd.1.smithi045.stdout:Jul 31 01:53:56 smithi045 conmon[46771]: 2021-07-31T01:53:57.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:56 smithi154 conmon[34553]: cluster 2021-07-31T01:53:55.859029+0000 mon.a (mon 2021-07-31T01:53:57.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:56 smithi154 conmon[34553]: .0) 1190 : cluster [DBG] osdmap e534: 8 total, 8 up, 8 in 2021-07-31T01:53:57.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:56 smithi154 conmon[34553]: cluster 2021-07-31T01:53:55.886876+0000 mgr.x (mgr.24542) 286 : cluster [DBG] pgmap v582: 25 pgs: 25 active+clean; 180 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 2.8 MiB/s wr, 28 op/s 2021-07-31T01:53:57.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:56 smithi154 conmon[34553]: cluster 2021-07-31T01:53:57.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:56 smithi154 conmon[34553]: 2021-07-31T01:53:56.596801+0000 mon.a (mon.0) 1191 : cluster [DBG] osdmap e535: 8 total, 8 up, 8 in 2021-07-31T01:53:57.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:56 smithi045 conmon[39718]: cluster 2021-07-31T01:53:55.859029+0000 mon.a (mon 2021-07-31T01:53:57.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:56 smithi045 conmon[39718]: .0) 1190 : cluster [DBG] osdmap e534: 8 total, 8 up, 8 in 2021-07-31T01:53:57.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:56 smithi045 conmon[39718]: cluster 2021-07-31T01:53:55.886876+0000 2021-07-31T01:53:57.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:56 smithi045 conmon[39718]: mgr.x (mgr.24542) 286 : cluster [DBG] pgmap v582: 25 pgs: 25 active+clean; 180 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 2.8 MiB/s wr, 28 op/s 2021-07-31T01:53:57.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:56 smithi045 conmon[39718]: cluster 2021-07- 2021-07-31T01:53:57.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:56 smithi045 conmon[39718]: 31T01:53:56.596801+0000 mon.a 2021-07-31T01:53:57.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:56 smithi045 conmon[39718]: (mon.0) 1191 : cluster [DBG] 2021-07-31T01:53:57.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:56 smithi045 conmon[39718]: osdmap e535: 8 total, 8 up, 8 in 2021-07-31T01:53:57.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:56 smithi045 conmon[33655]: cluster 2021-07-31T01:53:55.859029+0000 mon.a (mon.0) 1190 : cluster [DBG] 2021-07-31T01:53:57.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:56 smithi045 conmon[33655]: osdmap e534: 8 total, 8 up, 8 in 2021-07-31T01:53:57.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:56 smithi045 conmon[33655]: cluster 2021-07-31T01:53:55.886876+0000 mgr.x (mgr.24542) 286 : cluster [DBG] pgmap v582: 25 pgs: 25 active+clean; 180 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 2.8 MiB/s wr, 28 op/s 2021-07-31T01:53:57.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:56 smithi045 conmon[33655]: cluster 2021-07-31T01:53:57.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:56 smithi045 conmon[33655]: 2021-07-31T01:53:56.596801+0000 mon.a (mon.0) 1191 : cluster [DBG] osdmap e535: 8 total, 8 up, 8 in 2021-07-31T01:53:57.604 INFO:tasks.rados.rados.0.smithi045.stdout:2534: done (3 left) 2021-07-31T01:53:57.605 INFO:tasks.rados.rados.0.smithi045.stdout:2535: delete oid 2 current snap is 237 2021-07-31T01:53:57.605 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 4 v 1012 (ObjNum 716 snap 222 seq_num 716) dirty exists 2021-07-31T01:53:57.605 INFO:tasks.rados.rados.0.smithi045.stdout:2533: finishing write tid 1 to smithi04569269-40 2021-07-31T01:53:57.605 INFO:tasks.rados.rados.0.smithi045.stdout:2533: finishing write tid 2 to smithi04569269-40 2021-07-31T01:53:57.606 INFO:tasks.rados.rados.0.smithi045.stdout:2533: finishing write tid 3 to smithi04569269-40 2021-07-31T01:53:57.606 INFO:tasks.rados.rados.0.smithi045.stdout:2533: finishing write tid 4 to smithi04569269-40 2021-07-31T01:53:57.606 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 40 v 1580 (ObjNum 778 snap 237 seq_num 778) dirty exists 2021-07-31T01:53:57.606 INFO:tasks.rados.rados.0.smithi045.stdout:2533: left oid 40 (ObjNum 778 snap 237 seq_num 778) 2021-07-31T01:53:57.607 INFO:tasks.rados.rados.0.smithi045.stdout:2530: done (3 left) 2021-07-31T01:53:57.608 INFO:tasks.rados.rados.0.smithi045.stdout:2532: done (2 left) 2021-07-31T01:53:57.608 INFO:tasks.rados.rados.0.smithi045.stdout:2533: done (1 left) 2021-07-31T01:53:57.608 INFO:tasks.rados.rados.0.smithi045.stdout:2535: done (0 left) 2021-07-31T01:53:57.610 INFO:tasks.rados.rados.0.smithi045.stdout:2536: snap_create 2021-07-31T01:53:58.609 INFO:tasks.rados.rados.0.smithi045.stdout:2536: done (0 left) 2021-07-31T01:53:58.610 INFO:tasks.rados.rados.0.smithi045.stdout:2537: read oid 43 snap 237 2021-07-31T01:53:58.610 INFO:tasks.rados.rados.0.smithi045.stdout:2537: expect (ObjNum 754 snap 233 seq_num 754) 2021-07-31T01:53:58.610 INFO:tasks.rados.rados.0.smithi045.stdout:2538: snap_create 2021-07-31T01:53:58.649 INFO:tasks.rados.rados.0.smithi045.stdout:2537: expect (ObjNum 754 snap 233 seq_num 754) 2021-07-31T01:53:59.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:58 smithi154 conmon[34553]: cluster 2021-07-31T01:53:57.603780+0000 mon.a (mon.0) 1192 : cluster 2021-07-31T01:53:59.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:58 smithi154 conmon[34553]: [DBG] osdmap e536: 8 total, 8 up, 8 in 2021-07-31T01:53:59.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:58 smithi154 conmon[34553]: cluster 2021-07-31T01:53:57.887281+0000 mgr.x (mgr.24542) 287 : cluster [DBG] pgmap v585: 25 pgs: 25 active+clean; 180 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail 2021-07-31T01:53:59.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:58 smithi045 conmon[39718]: cluster 2021-07-31T01:53:57.603780+0000 mon.a (mon.0) 2021-07-31T01:53:59.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:58 smithi045 conmon[39718]: 1192 : cluster [DBG] osdmap e536: 8 total, 8 up, 8 in 2021-07-31T01:53:59.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:58 smithi045 conmon[39718]: cluster 2021-07-31T01:53:57.887281+0000 mgr.x (mgr.24542) 287 : cluster [DBG] pgmap v585: 25 pgs: 25 active+clean; 180 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail 2021-07-31T01:53:59.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:58 smithi045 conmon[33655]: cluster 2021-07-31T01:53:57.603780+0000 mon.a (mon.0) 2021-07-31T01:53:59.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:58 smithi045 conmon[33655]: 1192 : cluster [DBG] osdmap e536: 8 total, 8 up, 8 in 2021-07-31T01:53:59.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:58 smithi045 conmon[33655]: cluster 2021-07-31T01:53:57.887281+0000 mgr.x (mgr.24542) 287 : cluster [DBG] pgmap v585: 25 pgs: 25 active+clean; 180 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail 2021-07-31T01:53:59.611 INFO:tasks.rados.rados.0.smithi045.stdout:2537: done (1 left) 2021-07-31T01:53:59.611 INFO:tasks.rados.rados.0.smithi045.stdout:2538: done (0 left) 2021-07-31T01:53:59.611 INFO:tasks.rados.rados.0.smithi045.stdout:2539: copy_from oid 50 from oid 8 current snap is 239 2021-07-31T01:53:59.612 INFO:tasks.rados.rados.0.smithi045.stdout:2540: copy_from oid 29 from oid 43 current snap is 239 2021-07-31T01:53:59.612 INFO:tasks.rados.rados.0.smithi045.stdout:2541: read oid 40 snap -1 2021-07-31T01:53:59.612 INFO:tasks.rados.rados.0.smithi045.stdout:2541: expect (ObjNum 778 snap 237 seq_num 778) 2021-07-31T01:53:59.612 INFO:tasks.rados.rados.0.smithi045.stdout:2542: read oid 20 snap -1 2021-07-31T01:53:59.613 INFO:tasks.rados.rados.0.smithi045.stdout:2542: expect (ObjNum 556 snap 166 seq_num 556) 2021-07-31T01:53:59.613 INFO:tasks.rados.rados.0.smithi045.stdout:2543: read oid 26 snap -1 2021-07-31T01:53:59.613 INFO:tasks.rados.rados.0.smithi045.stdout:2543: expect (ObjNum 760 snap 233 seq_num 760) 2021-07-31T01:53:59.614 INFO:tasks.rados.rados.0.smithi045.stdout:2544: snap_remove snap 237 2021-07-31T01:54:00.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:59 smithi154 conmon[34553]: cluster 2021-07-31T01:54:00.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:59 smithi154 conmon[34553]: 2021-07-31T01:54:00.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:59 smithi154 conmon[34553]: 2021 2021-07-31T01:54:00.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:59 smithi154 conmon[34553]: -07-31T01:53:58.608642+0000 mon.a ( 2021-07-31T01:54:00.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:53:59 smithi154 conmon[34553]: mon.0) 1193 : cluster [DBG] osdmap e537: 8 total, 8 up, 8 in 2021-07-31T01:54:00.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:59 smithi045 conmon[33655]: cluster 2021-07-31T01:53:58.608642+0000 mon.a (mon.0) 1193 : 2021-07-31T01:54:00.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:53:59 smithi045 conmon[33655]: cluster [DBG] osdmap e537: 8 total, 8 up, 8 in 2021-07-31T01:54:00.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:59 smithi045 conmon[39718]: cluster 2021-07-31T01:54:00.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:59 smithi045 conmon[39718]: 2021-07-31T01: 2021-07-31T01:54:00.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:53:59 smithi045 conmon[39718]: 53:58.608642+0000 mon.a (mon.0) 1193 : cluster [DBG] osdmap e537: 8 total, 8 up, 8 in 2021-07-31T01:54:00.622 INFO:tasks.rados.rados.0.smithi045.stdout:2544: done (5 left) 2021-07-31T01:54:00.622 INFO:tasks.rados.rados.0.smithi045.stdout:2545: delete oid 9 current snap is 239 2021-07-31T01:54:00.622 INFO:tasks.rados.rados.0.smithi045.stdout:2541: expect (ObjNum 778 snap 237 seq_num 778) 2021-07-31T01:54:00.649 INFO:tasks.rados.rados.0.smithi045.stdout:2542: expect (ObjNum 556 snap 166 seq_num 556) 2021-07-31T01:54:00.765 INFO:tasks.rados.rados.0.smithi045.stdout:2543: expect (ObjNum 760 snap 233 seq_num 760) 2021-07-31T01:54:00.980 INFO:tasks.rados.rados.0.smithi045.stdout:2539: finishing copy_from to smithi04569269-50 2021-07-31T01:54:00.980 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 50 v 1616 (ObjNum 701 snap 218 seq_num 701) dirty exists 2021-07-31T01:54:00.980 INFO:tasks.rados.rados.0.smithi045.stdout:2539: finishing copy_from racing read to smithi04569269-50 2021-07-31T01:54:00.981 INFO:tasks.rados.rados.0.smithi045.stdout:2540: finishing copy_from to smithi04569269-29 2021-07-31T01:54:00.981 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 29 v 1401 (ObjNum 754 snap 233 seq_num 754) dirty exists 2021-07-31T01:54:00.981 INFO:tasks.rados.rados.0.smithi045.stdout:2540: finishing copy_from racing read to smithi04569269-29 2021-07-31T01:54:00.981 INFO:tasks.rados.rados.0.smithi045.stdout:2539: done (5 left) 2021-07-31T01:54:00.982 INFO:tasks.rados.rados.0.smithi045.stdout:2540: done (4 left) 2021-07-31T01:54:00.982 INFO:tasks.rados.rados.0.smithi045.stdout:2541: done (3 left) 2021-07-31T01:54:00.982 INFO:tasks.rados.rados.0.smithi045.stdout:2542: done (2 left) 2021-07-31T01:54:00.982 INFO:tasks.rados.rados.0.smithi045.stdout:2543: done (1 left) 2021-07-31T01:54:00.983 INFO:tasks.rados.rados.0.smithi045.stdout:2545: done (0 left) 2021-07-31T01:54:00.983 INFO:tasks.rados.rados.0.smithi045.stdout:2546: snap_remove snap 238 2021-07-31T01:54:01.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:00 smithi154 conmon[34553]: cluster 2021-07-31T01:53:59.610327+0000 mon.a (mon 2021-07-31T01:54:01.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:00 smithi154 conmon[34553]: .0) 1194 : cluster [DBG] osdmap e538: 8 total, 8 up, 8 in 2021-07-31T01:54:01.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:00 smithi154 conmon[34553]: cluster 2021-07-31T01:53:59.887835+0000 mgr.x (mgr.24542) 288 : cluster [DBG] pgmap v588: 25 pgs: 25 active+clean; 180 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 4.8 MiB/s wr, 23 op/s 2021-07-31T01:54:01.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:00 smithi045 conmon[39718]: cluster 2021-07-31T01:53:59.610327+0000 mon.a (mon.0) 1194 : cluster [DBG] 2021-07-31T01:54:01.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:00 smithi045 conmon[39718]: osdmap e538: 8 total, 8 up, 8 in 2021-07-31T01:54:01.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:00 smithi045 conmon[39718]: cluster 2021-07-31T01:53:59.887835+0000 mgr.x (mgr. 2021-07-31T01:54:01.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:00 smithi045 conmon[39718]: 24542) 288 : cluster [DBG] pgmap v588: 25 pgs: 25 active+clean; 180 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 4.8 MiB/s wr, 23 op/s 2021-07-31T01:54:01.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:00 smithi045 conmon[33655]: cluster 2021-07-31T01:53:59.610327+0000 mon.a (mon.0) 1194 2021-07-31T01:54:01.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:00 smithi045 conmon[33655]: : cluster [DBG] osdmap e538: 8 total, 8 up, 8 in 2021-07-31T01:54:01.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:00 smithi045 conmon[33655]: cluster 2021-07-31T01:53:59.887835+0000 mgr.x (mgr.24542) 288 : cluster [DBG] pgmap v588: 25 pgs: 25 active+clean; 180 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 4.8 MiB/s wr, 23 op/s 2021-07-31T01:54:01.599 INFO:tasks.rados.rados.0.smithi045.stdout:2546: done (0 left) 2021-07-31T01:54:01.599 INFO:tasks.rados.rados.0.smithi045.stdout:2547: read oid 45 snap -1 2021-07-31T01:54:01.600 INFO:tasks.rados.rados.0.smithi045.stdout:2547: expect (ObjNum 747 snap 230 seq_num 747) 2021-07-31T01:54:01.600 INFO:tasks.rados.rados.0.smithi045.stdout:2548: copy_from oid 41 from oid 8 current snap is 239 2021-07-31T01:54:01.600 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 15 current snap is 239 2021-07-31T01:54:01.600 INFO:tasks.rados.rados.0.smithi045.stdout:2549: seq_num 781 ranges {8822784=688128,9510912=401408} 2021-07-31T01:54:01.651 INFO:tasks.rados.rados.0.smithi045.stdout:2549: writing smithi04569269-15 from 8822784 to 9510912 tid 1 2021-07-31T01:54:01.654 INFO:tasks.rados.rados.0.smithi045.stdout:2549: writing smithi04569269-15 from 9510912 to 9912320 tid 2 2021-07-31T01:54:01.654 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 11 current snap is 239 2021-07-31T01:54:01.655 INFO:tasks.rados.rados.0.smithi045.stdout:2550: seq_num 782 ranges {0=704512,704512=401408} 2021-07-31T01:54:01.658 INFO:tasks.rados.rados.0.smithi045.stdout:2550: writing smithi04569269-11 from 0 to 704512 tid 1 2021-07-31T01:54:01.661 INFO:tasks.rados.rados.0.smithi045.stdout:2550: writing smithi04569269-11 from 704512 to 1105920 tid 2 2021-07-31T01:54:01.661 INFO:tasks.rados.rados.0.smithi045.stdout:2551: rollback oid 50 current snap is 239 2021-07-31T01:54:01.661 INFO:tasks.rados.rados.0.smithi045.stdout:2551: done (4 left) 2021-07-31T01:54:01.662 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 21 current snap is 239 2021-07-31T01:54:01.662 INFO:tasks.rados.rados.0.smithi045.stdout:2552: seq_num 783 ranges {0=720896,720896=401408} 2021-07-31T01:54:01.665 INFO:tasks.rados.rados.0.smithi045.stdout:2552: writing smithi04569269-21 from 0 to 720896 tid 1 2021-07-31T01:54:01.667 INFO:tasks.rados.rados.0.smithi045.stdout:2552: writing smithi04569269-21 from 720896 to 1122304 tid 2 2021-07-31T01:54:01.667 INFO:tasks.rados.rados.0.smithi045.stdout:2553: rollback oid 25 current snap is 239 2021-07-31T01:54:01.668 INFO:tasks.rados.rados.0.smithi045.stdout:2553: done (5 left) 2021-07-31T01:54:01.668 INFO:tasks.rados.rados.0.smithi045.stdout:2554: copy_from oid 1 from oid 38 current snap is 239 2021-07-31T01:54:01.668 INFO:tasks.rados.rados.0.smithi045.stdout:2547: expect (ObjNum 747 snap 230 seq_num 747) 2021-07-31T01:54:01.758 INFO:tasks.rados.rados.0.smithi045.stdout:2548: finishing copy_from to smithi04569269-41 2021-07-31T01:54:01.759 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 41 v 1616 (ObjNum 701 snap 218 seq_num 701) dirty exists 2021-07-31T01:54:01.759 INFO:tasks.rados.rados.0.smithi045.stdout:2548: finishing copy_from racing read to smithi04569269-41 2021-07-31T01:54:01.759 INFO:tasks.rados.rados.0.smithi045.stdout:2549: finishing write tid 1 to smithi04569269-15 2021-07-31T01:54:01.759 INFO:tasks.rados.rados.0.smithi045.stdout:2549: finishing write tid 2 to smithi04569269-15 2021-07-31T01:54:01.760 INFO:tasks.rados.rados.0.smithi045.stdout:2549: finishing write tid 3 to smithi04569269-15 2021-07-31T01:54:01.760 INFO:tasks.rados.rados.0.smithi045.stdout:2547: done (5 left) 2021-07-31T01:54:01.761 INFO:tasks.rados.rados.0.smithi045.stdout:2548: done (4 left) 2021-07-31T01:54:01.761 INFO:tasks.rados.rados.0.smithi045.stdout:2555: read oid 22 snap -1 2021-07-31T01:54:01.761 INFO:tasks.rados.rados.0.smithi045.stdout:2555: expect (ObjNum 747 snap 230 seq_num 747) 2021-07-31T01:54:01.761 INFO:tasks.rados.rados.0.smithi045.stdout:2556: copy_from oid 32 from oid 29 current snap is 239 2021-07-31T01:54:01.761 INFO:tasks.rados.rados.0.smithi045.stdout:2549: finishing write tid 4 to smithi04569269-15 2021-07-31T01:54:01.762 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 15 v 1015 (ObjNum 781 snap 239 seq_num 781) dirty exists 2021-07-31T01:54:01.762 INFO:tasks.rados.rados.0.smithi045.stdout:2549: left oid 15 (ObjNum 781 snap 239 seq_num 781) 2021-07-31T01:54:01.762 INFO:tasks.rados.rados.0.smithi045.stdout:2550: finishing write tid 1 to smithi04569269-11 2021-07-31T01:54:01.762 INFO:tasks.rados.rados.0.smithi045.stdout:2550: finishing write tid 2 to smithi04569269-11 2021-07-31T01:54:01.763 INFO:tasks.rados.rados.0.smithi045.stdout:2550: finishing write tid 3 to smithi04569269-11 2021-07-31T01:54:01.763 INFO:tasks.rados.rados.0.smithi045.stdout:2550: finishing write tid 4 to smithi04569269-11 2021-07-31T01:54:01.763 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 11 v 1405 (ObjNum 782 snap 239 seq_num 782) dirty exists 2021-07-31T01:54:01.763 INFO:tasks.rados.rados.0.smithi045.stdout:2550: left oid 11 (ObjNum 782 snap 239 seq_num 782) 2021-07-31T01:54:01.763 INFO:tasks.rados.rados.0.smithi045.stdout:2554: finishing copy_from to smithi04569269-1 2021-07-31T01:54:01.764 INFO:tasks.rados.rados.0.smithi045.stdout:2554: got expected ENOENT (src dne) 2021-07-31T01:54:01.764 INFO:tasks.rados.rados.0.smithi045.stdout:2549: done (5 left) 2021-07-31T01:54:01.764 INFO:tasks.rados.rados.0.smithi045.stdout:2550: done (4 left) 2021-07-31T01:54:01.764 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 13 current snap is 239 2021-07-31T01:54:01.765 INFO:tasks.rados.rados.0.smithi045.stdout:2554: finishing copy_from racing read to smithi04569269-1 2021-07-31T01:54:01.765 INFO:tasks.rados.rados.0.smithi045.stdout:2557: seq_num 784 ranges {17661952=737280,18399232=401408} 2021-07-31T01:54:01.858 INFO:tasks.rados.rados.0.smithi045.stdout:2557: writing smithi04569269-13 from 17661952 to 18399232 tid 1 2021-07-31T01:54:01.861 INFO:tasks.rados.rados.0.smithi045.stdout:2557: writing smithi04569269-13 from 18399232 to 18800640 tid 2 2021-07-31T01:54:01.861 INFO:tasks.rados.rados.0.smithi045.stdout:2554: done (4 left) 2021-07-31T01:54:01.861 INFO:tasks.rados.rados.0.smithi045.stdout:2558: snap_create 2021-07-31T01:54:01.862 INFO:tasks.rados.rados.0.smithi045.stdout:2552: finishing write tid 1 to smithi04569269-21 2021-07-31T01:54:01.862 INFO:tasks.rados.rados.0.smithi045.stdout:2552: finishing write tid 2 to smithi04569269-21 2021-07-31T01:54:01.862 INFO:tasks.rados.rados.0.smithi045.stdout:2552: finishing write tid 3 to smithi04569269-21 2021-07-31T01:54:01.863 INFO:tasks.rados.rados.0.smithi045.stdout:2552: finishing write tid 4 to smithi04569269-21 2021-07-31T01:54:01.863 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 21 v 1625 (ObjNum 783 snap 239 seq_num 783) dirty exists 2021-07-31T01:54:01.863 INFO:tasks.rados.rados.0.smithi045.stdout:2552: left oid 21 (ObjNum 783 snap 239 seq_num 783) 2021-07-31T01:54:01.863 INFO:tasks.rados.rados.0.smithi045.stdout:2555: expect (ObjNum 747 snap 230 seq_num 747) 2021-07-31T01:54:01.953 INFO:tasks.rados.rados.0.smithi045.stdout:2556: finishing copy_from to smithi04569269-32 2021-07-31T01:54:01.953 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 32 v 1401 (ObjNum 754 snap 233 seq_num 754) dirty exists 2021-07-31T01:54:01.953 INFO:tasks.rados.rados.0.smithi045.stdout:2556: finishing copy_from racing read to smithi04569269-32 2021-07-31T01:54:01.954 INFO:tasks.rados.rados.0.smithi045.stdout:2557: finishing write tid 1 to smithi04569269-13 2021-07-31T01:54:01.954 INFO:tasks.rados.rados.0.smithi045.stdout:2557: finishing write tid 2 to smithi04569269-13 2021-07-31T01:54:01.954 INFO:tasks.rados.rados.0.smithi045.stdout:2557: finishing write tid 3 to smithi04569269-13 2021-07-31T01:54:01.954 INFO:tasks.rados.rados.0.smithi045.stdout:2557: finishing write tid 4 to smithi04569269-13 2021-07-31T01:54:01.955 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 13 v 1583 (ObjNum 784 snap 239 seq_num 784) dirty exists 2021-07-31T01:54:01.955 INFO:tasks.rados.rados.0.smithi045.stdout:2557: left oid 13 (ObjNum 784 snap 239 seq_num 784) 2021-07-31T01:54:02.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:01 smithi154 conmon[34553]: cluster 2021-07-31T01:54:00.620926+0000 mon.a (mon.0) 1195 : cluster [DBG] osdmap e539: 8 total, 8 up, 8 in 2021-07-31T01:54:02.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:01 smithi154 conmon[34553]: cluster 2021-07-31T01:54:01.598406+0000 mon.a (mon.0) 1196 : cluster [DBG] osdmap e540: 8 total, 8 up, 8 in 2021-07-31T01:54:02.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:01 smithi045 conmon[39718]: cluster 2021-07-31T01:54:00. 2021-07-31T01:54:02.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:01 smithi045 conmon[39718]: 620926+0000 mon.a (mon.0) 1195 : cluster [DBG] osdmap e539: 8 total, 8 up, 8 in 2021-07-31T01:54:02.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:01 smithi045 conmon[39718]: cluster 2021-07-31T01:54:01.598406+0000 mon.a (mon.0) 1196 2021-07-31T01:54:02.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:01 smithi045 conmon[39718]: : cluster [DBG] osdmap e540: 8 total, 8 up, 8 in 2021-07-31T01:54:02.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:01 smithi045 conmon[33655]: cluster 2021-07-31T01:54:00.620926+0000 mon.a (mon.0) 1195 2021-07-31T01:54:02.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:01 smithi045 conmon[33655]: : cluster [DBG] osdmap e539: 8 total, 8 up, 8 in 2021-07-31T01:54:02.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:01 smithi045 conmon[33655]: cluster 2021-07-31T01:54:01.598406+0000 mon.a (mon.0) 1196 : cluster [DBG] osdmap e540: 8 total, 8 up, 8 in 2021-07-31T01:54:02.628 INFO:tasks.rados.rados.0.smithi045.stdout:2552: done (4 left) 2021-07-31T01:54:02.629 INFO:tasks.rados.rados.0.smithi045.stdout:2555: done (3 left) 2021-07-31T01:54:02.629 INFO:tasks.rados.rados.0.smithi045.stdout:2556: done (2 left) 2021-07-31T01:54:02.629 INFO:tasks.rados.rados.0.smithi045.stdout:2557: done (1 left) 2021-07-31T01:54:02.629 INFO:tasks.rados.rados.0.smithi045.stdout:2558: done (0 left) 2021-07-31T01:54:02.630 INFO:tasks.rados.rados.0.smithi045.stdout:2559: rollback oid 12 current snap is 240 2021-07-31T01:54:02.630 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 12 to 239 2021-07-31T01:54:02.630 INFO:tasks.rados.rados.0.smithi045.stdout:2560: rmattr oid 23 current snap is 240 2021-07-31T01:54:02.630 INFO:tasks.rados.rados.0.smithi045.stdout:2560: done (1 left) 2021-07-31T01:54:02.630 INFO:tasks.rados.rados.0.smithi045.stdout:2561: read oid 2 snap -1 2021-07-31T01:54:02.631 INFO:tasks.rados.rados.0.smithi045.stdout:2561: expect deleted 2021-07-31T01:54:02.631 INFO:tasks.rados.rados.0.smithi045.stdout:2562: read oid 34 snap -1 2021-07-31T01:54:02.631 INFO:tasks.rados.rados.0.smithi045.stdout:2562: expect (ObjNum 765 snap 235 seq_num 765) 2021-07-31T01:54:02.631 INFO:tasks.rados.rados.0.smithi045.stdout:2563: setattr oid 30 current snap is 240 2021-07-31T01:54:02.639 INFO:tasks.rados.rados.0.smithi045.stdout:2564: rollback oid 9 current snap is 240 2021-07-31T01:54:02.639 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 9 to 239 2021-07-31T01:54:02.639 INFO:tasks.rados.rados.0.smithi045.stdout:2565: rmattr oid 44 current snap is 240 2021-07-31T01:54:02.640 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 11 current snap is 240 2021-07-31T01:54:02.640 INFO:tasks.rados.rados.0.smithi045.stdout:2566: seq_num 785 ranges {1105920=753664,1859584=401408} 2021-07-31T01:54:02.641 INFO:tasks.rados.rados.0.smithi045.stdout:2566: writing smithi04569269-11 from 1105920 to 1859584 tid 1 2021-07-31T01:54:02.646 INFO:tasks.rados.rados.0.smithi045.stdout:2566: writing smithi04569269-11 from 1859584 to 2260992 tid 2 2021-07-31T01:54:02.646 INFO:tasks.rados.rados.0.smithi045.stdout:2567: delete oid 38 current snap is 240 2021-07-31T01:54:02.646 INFO:tasks.rados.rados.0.smithi045.stdout:2564: finishing rollback tid 1 to smithi04569269-9 2021-07-31T01:54:02.646 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 9 v 0 (ObjNum 288 snap 0 seq_num 33) dirty dne 2021-07-31T01:54:02.647 INFO:tasks.rados.rados.0.smithi045.stdout:2559: finishing rollback tid 0 to smithi04569269-12 2021-07-31T01:54:02.647 INFO:tasks.rados.rados.0.smithi045.stdout:2559: finishing rollback tid 1 to smithi04569269-12 2021-07-31T01:54:02.647 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 44 v 1411 (ObjNum 776 snap 236 seq_num 776) dirty exists 2021-07-31T01:54:02.648 INFO:tasks.rados.rados.0.smithi045.stdout:2559: finishing rollback tid 2 to smithi04569269-12 2021-07-31T01:54:02.649 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 12 v 1586 (ObjNum 672 snap 210 seq_num 672) dirty exists 2021-07-31T01:54:02.649 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 30 v 1597 (ObjNum 678 snap 211 seq_num 678) dirty exists 2021-07-31T01:54:02.650 INFO:tasks.rados.rados.0.smithi045.stdout:2566: finishing write tid 1 to smithi04569269-11 2021-07-31T01:54:02.650 INFO:tasks.rados.rados.0.smithi045.stdout:2566: finishing write tid 2 to smithi04569269-11 2021-07-31T01:54:02.650 INFO:tasks.rados.rados.0.smithi045.stdout:2566: finishing write tid 3 to smithi04569269-11 2021-07-31T01:54:02.651 INFO:tasks.rados.rados.0.smithi045.stdout:2559: done (7 left) 2021-07-31T01:54:02.652 INFO:tasks.rados.rados.0.smithi045.stdout:2561: done (6 left) 2021-07-31T01:54:02.652 INFO:tasks.rados.rados.0.smithi045.stdout:2563: done (5 left) 2021-07-31T01:54:02.652 INFO:tasks.rados.rados.0.smithi045.stdout:2564: done (4 left) 2021-07-31T01:54:02.652 INFO:tasks.rados.rados.0.smithi045.stdout:2565: done (3 left) 2021-07-31T01:54:02.653 INFO:tasks.rados.rados.0.smithi045.stdout:2567: done (2 left) 2021-07-31T01:54:02.653 INFO:tasks.rados.rados.0.smithi045.stdout:2568: copy_from oid 33 from oid 31 current snap is 240 2021-07-31T01:54:02.653 INFO:tasks.rados.rados.0.smithi045.stdout:2569: delete oid 49 current snap is 240 2021-07-31T01:54:02.653 INFO:tasks.rados.rados.0.smithi045.stdout:2566: finishing write tid 4 to smithi04569269-11 2021-07-31T01:54:02.653 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 11 v 1408 (ObjNum 785 snap 240 seq_num 785) dirty exists 2021-07-31T01:54:02.654 INFO:tasks.rados.rados.0.smithi045.stdout:2566: left oid 11 (ObjNum 785 snap 240 seq_num 785) 2021-07-31T01:54:02.655 INFO:tasks.rados.rados.0.smithi045.stdout:2566: done (3 left) 2021-07-31T01:54:02.655 INFO:tasks.rados.rados.0.smithi045.stdout:2569: done (2 left) 2021-07-31T01:54:02.655 INFO:tasks.rados.rados.0.smithi045.stdout:2570: setattr oid 3 current snap is 240 2021-07-31T01:54:02.661 INFO:tasks.rados.rados.0.smithi045.stdout:2571: rollback oid 26 current snap is 240 2021-07-31T01:54:02.662 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 26 to 239 2021-07-31T01:54:02.662 INFO:tasks.rados.rados.0.smithi045.stdout:2572: copy_from oid 20 from oid 48 current snap is 240 2021-07-31T01:54:02.662 INFO:tasks.rados.rados.0.smithi045.stdout:2573: rollback oid 22 current snap is 240 2021-07-31T01:54:02.663 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 22 to 239 2021-07-31T01:54:02.663 INFO:tasks.rados.rados.0.smithi045.stdout:2574: read oid 30 snap -1 2021-07-31T01:54:02.664 INFO:tasks.rados.rados.0.smithi045.stdout:2574: expect (ObjNum 678 snap 211 seq_num 678) 2021-07-31T01:54:02.664 INFO:tasks.rados.rados.0.smithi045.stdout:2575: read oid 12 snap -1 2021-07-31T01:54:02.665 INFO:tasks.rados.rados.0.smithi045.stdout:2575: expect (ObjNum 672 snap 210 seq_num 672) 2021-07-31T01:54:02.665 INFO:tasks.rados.rados.0.smithi045.stdout:2576: rmattr oid 37 current snap is 240 2021-07-31T01:54:02.666 INFO:tasks.rados.rados.0.smithi045.stdout:2576: done (8 left) 2021-07-31T01:54:02.667 INFO:tasks.rados.rados.0.smithi045.stdout:2577: snap_create 2021-07-31T01:54:02.667 INFO:tasks.rados.rados.0.smithi045.stdout:2568: finishing copy_from to smithi04569269-33 2021-07-31T01:54:02.667 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 33 v 1608 (ObjNum 726 snap 226 seq_num 726) dirty exists 2021-07-31T01:54:02.667 INFO:tasks.rados.rados.0.smithi045.stdout:2568: finishing copy_from racing read to smithi04569269-33 2021-07-31T01:54:02.668 INFO:tasks.rados.rados.0.smithi045.stdout:2571: finishing rollback tid 0 to smithi04569269-26 2021-07-31T01:54:02.668 INFO:tasks.rados.rados.0.smithi045.stdout:2562: expect (ObjNum 765 snap 235 seq_num 765) 2021-07-31T01:54:03.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:02 smithi154 conmon[34553]: cluster 2021-07-31T01:54:01.888199+0000 mgr.x (mgr.24542) 289 : cluster [DBG] 2021-07-31T01:54:03.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:02 smithi154 conmon[34553]: pgmap v591: 25 pgs: 25 active+clean; 180 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 4.8 MiB/s wr, 23 op/s 2021-07-31T01:54:03.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:02 smithi045 conmon[33655]: cluster 2021-07-31T01:54:01.888199+0000 mgr.x (mgr.24542) 289 : cluster [DBG] 2021-07-31T01:54:03.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:02 smithi045 conmon[33655]: pgmap v591: 25 pgs: 25 active+clean; 180 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 4.8 MiB/s wr, 23 op/s 2021-07-31T01:54:03.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:02 smithi045 conmon[39718]: cluster 2021-07-31T01:54:01.888199+0000 mgr.x (mgr.24542) 289 : cluster 2021-07-31T01:54:03.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:02 smithi045 conmon[39718]: [DBG] pgmap v591: 25 pgs: 25 active+clean; 180 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 4.8 MiB/s wr, 23 op/s 2021-07-31T01:54:03.378 INFO:tasks.rados.rados.0.smithi045.stdout:2573: finishing rollback tid 0 to smithi04569269-22 2021-07-31T01:54:03.378 INFO:tasks.rados.rados.0.smithi045.stdout:2573: finishing rollback tid 1 to smithi04569269-22 2021-07-31T01:54:03.378 INFO:tasks.rados.rados.0.smithi045.stdout:2573: finishing rollback tid 2 to smithi04569269-22 2021-07-31T01:54:03.379 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 22 v 1568 (ObjNum 747 snap 230 seq_num 747) dirty exists 2021-07-31T01:54:03.379 INFO:tasks.rados.rados.0.smithi045.stdout:2571: finishing rollback tid 1 to smithi04569269-26 2021-07-31T01:54:03.379 INFO:tasks.rados.rados.0.smithi045.stdout:2571: finishing rollback tid 2 to smithi04569269-26 2021-07-31T01:54:03.379 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 26 v 1404 (ObjNum 760 snap 233 seq_num 760) dirty exists 2021-07-31T01:54:03.380 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 3 v 1629 (ObjNum 677 snap 211 seq_num 677) dirty exists 2021-07-31T01:54:03.380 INFO:tasks.rados.rados.0.smithi045.stdout:2575: expect (ObjNum 672 snap 210 seq_num 672) 2021-07-31T01:54:03.644 INFO:tasks.rados.rados.0.smithi045.stdout:2574: expect (ObjNum 678 snap 211 seq_num 678) 2021-07-31T01:54:03.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:03 smithi045 conmon[33655]: cluster 2021-07-31T01:54:02.627311 2021-07-31T01:54:03.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:03 smithi045 conmon[33655]: +0000 mon.a (mon.0) 1197 : cluster [DBG] osdmap e541: 8 total, 8 up, 8 in 2021-07-31T01:54:03.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:03 smithi045 conmon[39718]: cluster 2021-07-31T01:54:02.627311 2021-07-31T01:54:03.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:03 smithi045 conmon[39718]: +0000 mon.a (mon.0) 1197 : cluster [DBG] osdmap e541: 8 total, 8 up, 8 in 2021-07-31T01:54:03.779 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:54:03 smithi045 conmon[61210]: level=error ts=2021-07-31T01:54:03.520Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:54:03.780 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:54:03 smithi045 conmon[61210]: level=error ts=2021-07-31T01:54:03.520Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:54:03.780 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:54:03 smithi045 conmon[61210]: level=error ts=2021-07-31T01:54:03.520Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T01:54:04.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:03 smithi154 conmon[34553]: cluster 2021-07-31T01:54:02.627311+0000 mon.a (mon.0) 1197 : cluster [DBG] osdmap e541: 8 total, 8 up, 8 in 2021-07-31T01:54:04.136 INFO:tasks.rados.rados.0.smithi045.stdout:2572: finishing copy_from to smithi04569269-20 2021-07-31T01:54:04.137 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 20 v 1401 (ObjNum 754 snap 233 seq_num 754) dirty exists 2021-07-31T01:54:04.137 INFO:tasks.rados.rados.0.smithi045.stdout:2572: finishing copy_from racing read to smithi04569269-20 2021-07-31T01:54:04.138 INFO:tasks.rados.rados.0.smithi045.stdout:2562: done (8 left) 2021-07-31T01:54:04.138 INFO:tasks.rados.rados.0.smithi045.stdout:2568: done (7 left) 2021-07-31T01:54:04.138 INFO:tasks.rados.rados.0.smithi045.stdout:2570: done (6 left) 2021-07-31T01:54:04.138 INFO:tasks.rados.rados.0.smithi045.stdout:2571: done (5 left) 2021-07-31T01:54:04.138 INFO:tasks.rados.rados.0.smithi045.stdout:2572: done (4 left) 2021-07-31T01:54:04.139 INFO:tasks.rados.rados.0.smithi045.stdout:2573: done (3 left) 2021-07-31T01:54:04.139 INFO:tasks.rados.rados.0.smithi045.stdout:2574: done (2 left) 2021-07-31T01:54:04.139 INFO:tasks.rados.rados.0.smithi045.stdout:2575: done (1 left) 2021-07-31T01:54:04.139 INFO:tasks.rados.rados.0.smithi045.stdout:2577: done (0 left) 2021-07-31T01:54:04.140 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 25 current snap is 241 2021-07-31T01:54:04.140 INFO:tasks.rados.rados.0.smithi045.stdout:2578: seq_num 788 ranges {7798784=802816,8601600=401408} 2021-07-31T01:54:04.183 INFO:tasks.rados.rados.0.smithi045.stdout:2578: writing smithi04569269-25 from 7798784 to 8601600 tid 1 2021-07-31T01:54:04.185 INFO:tasks.rados.rados.0.smithi045.stdout:2578: writing smithi04569269-25 from 8601600 to 9003008 tid 2 2021-07-31T01:54:04.187 INFO:tasks.rados.rados.0.smithi045.stdout:2579: rmattr oid 5 current snap is 241 2021-07-31T01:54:04.187 INFO:tasks.rados.rados.0.smithi045.stdout:2580: snap_create 2021-07-31T01:54:04.188 INFO:tasks.rados.rados.0.smithi045.stdout:2578: finishing write tid 1 to smithi04569269-25 2021-07-31T01:54:04.188 INFO:tasks.rados.rados.0.smithi045.stdout:2578: finishing write tid 2 to smithi04569269-25 2021-07-31T01:54:04.188 INFO:tasks.rados.rados.0.smithi045.stdout:2578: finishing write tid 3 to smithi04569269-25 2021-07-31T01:54:04.189 INFO:tasks.rados.rados.0.smithi045.stdout:2578: finishing write tid 4 to smithi04569269-25 2021-07-31T01:54:04.189 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 25 v 1188 (ObjNum 788 snap 241 seq_num 788) dirty exists 2021-07-31T01:54:04.189 INFO:tasks.rados.rados.0.smithi045.stdout:2578: left oid 25 (ObjNum 788 snap 241 seq_num 788) 2021-07-31T01:54:04.190 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 5 v 1609 (ObjNum 754 snap 233 seq_num 754) dirty exists 2021-07-31T01:54:04.638 INFO:tasks.rados.rados.0.smithi045.stdout:2578: done (2 left) 2021-07-31T01:54:04.639 INFO:tasks.rados.rados.0.smithi045.stdout:2579: done (1 left) 2021-07-31T01:54:04.639 INFO:tasks.rados.rados.0.smithi045.stdout:2580: done (0 left) 2021-07-31T01:54:04.639 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 40 current snap is 242 2021-07-31T01:54:04.640 INFO:tasks.rados.rados.0.smithi045.stdout:2581: seq_num 789 ranges {1040384=417792,1458176=663552,2121728=139264} 2021-07-31T01:54:04.646 INFO:tasks.rados.rados.0.smithi045.stdout:2581: writing smithi04569269-40 from 1040384 to 1458176 tid 1 2021-07-31T01:54:04.650 INFO:tasks.rados.rados.0.smithi045.stdout:2581: writing smithi04569269-40 from 1458176 to 2121728 tid 2 2021-07-31T01:54:04.651 INFO:tasks.rados.rados.0.smithi045.stdout:2581: writing smithi04569269-40 from 2121728 to 2260992 tid 3 2021-07-31T01:54:04.651 INFO:tasks.rados.rados.0.smithi045.stdout:2582: snap_create 2021-07-31T01:54:04.654 INFO:tasks.rados.rados.0.smithi045.stdout:2581: finishing write tid 1 to smithi04569269-40 2021-07-31T01:54:04.655 INFO:tasks.rados.rados.0.smithi045.stdout:2581: finishing write tid 2 to smithi04569269-40 2021-07-31T01:54:04.656 INFO:tasks.rados.rados.0.smithi045.stdout:2581: finishing write tid 3 to smithi04569269-40 2021-07-31T01:54:04.656 INFO:tasks.rados.rados.0.smithi045.stdout:2581: finishing write tid 4 to smithi04569269-40 2021-07-31T01:54:04.657 INFO:tasks.rados.rados.0.smithi045.stdout:2581: finishing write tid 5 to smithi04569269-40 2021-07-31T01:54:04.657 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 40 v 1590 (ObjNum 789 snap 242 seq_num 789) dirty exists 2021-07-31T01:54:04.658 INFO:tasks.rados.rados.0.smithi045.stdout:2581: left oid 40 (ObjNum 789 snap 242 seq_num 789) 2021-07-31T01:54:05.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:04 smithi154 conmon[34553]: cluster 2021-07-31T01:54:03.632340+0000 mon.a (mon.0) 1198 : cluster [DBG] osdmap e542: 8 total, 8 up, 8 in 2021-07-31T01:54:05.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:04 smithi154 conmon[34553]: 2021-07-31T01:54:05.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:04 smithi154 conmon[34553]: cluster 2021-07-31T01:54:03.888803+0000 mgr.x (mgr.24542) 290 : cluster [DBG] pgmap v594: 25 pgs: 25 active+clean; 206 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 26 MiB/s rd, 9.6 MiB/s wr, 29 op/s 2021-07-31T01:54:05.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:04 smithi045 conmon[33655]: cluster 2021-07-31T01:54:03.632340+0000 mon.a (mon.0) 1198 : cluster 2021-07-31T01:54:05.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:04 smithi045 conmon[33655]: [DBG] osdmap e542: 8 total, 8 up, 8 in 2021-07-31T01:54:05.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:04 smithi045 conmon[33655]: cluster 2021-07-31T01:54:03.888803+0000 mgr.x (mgr.24542) 290 : cluster [DBG] pgmap v594: 25 pgs: 25 active+clean; 206 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 26 MiB/s rd, 9.6 MiB/s wr, 29 op/s 2021-07-31T01:54:05.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:04 smithi045 conmon[39718]: cluster 2021-07-31T01:54:03.632340+0000 mon.a (mon. 2021-07-31T01:54:05.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:04 smithi045 conmon[39718]: 0) 1198 : cluster [DBG] osdmap e542: 8 total, 8 up, 8 in 2021-07-31T01:54:05.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:04 smithi045 conmon[39718]: cluster 2021-07-31T01:54:03.888803+0000 mgr.x (mgr.24542) 290 : cluster [DBG] pgmap v594: 25 pgs: 25 active+clean; 206 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 26 MiB/s rd, 9.6 MiB/s wr, 29 op/s 2021-07-31T01:54:05.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:04 smithi045 conmon[39718]: 2021-07-31T01:54:05.644 INFO:tasks.rados.rados.0.smithi045.stdout:2581: done (1 left) 2021-07-31T01:54:05.644 INFO:tasks.rados.rados.0.smithi045.stdout:2582: done (0 left) 2021-07-31T01:54:05.645 INFO:tasks.rados.rados.0.smithi045.stdout:2583: snap_create 2021-07-31T01:54:06.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:05 smithi154 conmon[34553]: cluster 2021-07-31T01:54:04.638175+0000 mon.a (mon.0) 1199 : cluster [DBG] osdmap e543: 8 total, 8 up, 8 in 2021-07-31T01:54:06.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:05 smithi045 conmon[33655]: cluster 2021-07-31T01:54:04.638175+0000 mon.a (mon.0 2021-07-31T01:54:06.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:05 smithi045 conmon[33655]: ) 1199 : cluster [DBG] osdmap e543: 8 total, 8 up, 8 in 2021-07-31T01:54:06.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:05 smithi045 conmon[39718]: cluster 2021-07-31T01:54:04.638175+0000 mon.a 2021-07-31T01:54:06.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:05 smithi045 conmon[39718]: (mon.0) 1199 : cluster [DBG] osdmap e543: 8 total, 8 up, 8 in 2021-07-31T01:54:06.600 INFO:tasks.rados.rados.0.smithi045.stdout:2583: done (0 left) 2021-07-31T01:54:06.601 INFO:tasks.rados.rados.0.smithi045.stdout:2584: setattr oid 37 current snap is 244 2021-07-31T01:54:06.602 INFO:tasks.rados.rados.0.smithi045.stdout:2585: copy_from oid 13 from oid 1 current snap is 244 2021-07-31T01:54:06.602 INFO:tasks.rados.rados.0.smithi045.stdout:2586: delete oid 27 current snap is 244 2021-07-31T01:54:06.609 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 37 v 1016 (ObjNum -1126011800 snap 21873 seq_num 1) dirty exists 2021-07-31T01:54:06.610 INFO:tasks.rados.rados.0.smithi045.stdout:2584: done (2 left) 2021-07-31T01:54:06.610 INFO:tasks.rados.rados.0.smithi045.stdout:2586: done (1 left) 2021-07-31T01:54:06.610 INFO:tasks.rados.rados.0.smithi045.stdout:2587: copy_from oid 18 from oid 34 current snap is 244 2021-07-31T01:54:06.610 INFO:tasks.rados.rados.0.smithi045.stdout:2588: snap_remove snap 243 2021-07-31T01:54:07.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:06 smithi154 conmon[34553]: cluster 2021-07-31T01:54:05.643995+0000 mon.a (mon.0) 1200 : cluster [DBG] osdmap e544: 8 total, 8 up, 8 in 2021-07-31T01:54:07.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:06 smithi154 conmon[34553]: cluster 2021-07-31T01:54 2021-07-31T01:54:07.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:06 smithi154 conmon[34553]: :05.889243+0000 mgr.x (mgr.24542) 291 : cluster [DBG] pgmap v597: 25 pgs: 25 active+clean; 206 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 26 MiB/s rd, 9.6 MiB/s wr, 29 op/s 2021-07-31T01:54:07.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:06 smithi154 conmon[34553]: cluster 2021-07-31T01:54:06.599869+0000 mon.a (mon.0) 1201 : cluster [DBG] osdmap e545: 8 total, 8 up, 8 in 2021-07-31T01:54:07.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:06 smithi045 conmon[33655]: cluster 2021-07-31T01:54:05.643995+0000 mon.a (mon 2021-07-31T01:54:07.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:06 smithi045 conmon[33655]: .0) 1200 : cluster [DBG] osdmap e544: 8 total, 8 up, 8 in 2021-07-31T01:54:07.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:06 smithi045 conmon[33655]: cluster 2021-07-31T01:54:05.889243+0000 mgr.x (mgr.24542) 291 : cluster [DBG] pgmap v597: 25 pgs: 25 active+clean; 206 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 26 MiB/s rd, 9.6 MiB/s wr, 29 op/s 2021-07-31T01:54:07.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:06 smithi045 conmon[33655]: cluster 2021-07-31T01:54:06.599869 2021-07-31T01:54:07.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:06 smithi045 conmon[33655]: +0000 mon.a (mon.0) 1201 : cluster 2021-07-31T01:54:07.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:06 smithi045 conmon[33655]: [DBG] osdmap e545: 8 total, 8 up, 8 in 2021-07-31T01:54:07.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:06 smithi045 conmon[39718]: cluster 2021-07-31T01 2021-07-31T01:54:07.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:06 smithi045 conmon[39718]: :54:05.643995+0000 mon.a (mon.0) 1200 : cluster [DBG] osdmap e544: 8 total, 8 up, 8 in 2021-07-31T01:54:07.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:06 smithi045 conmon[39718]: cluster 2021 2021-07-31T01:54:07.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:06 smithi045 conmon[39718]: -07-31T01:54:05 2021-07-31T01:54:07.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:06 smithi045 conmon[39718]: .889243+0000 mgr.x (mgr 2021-07-31T01:54:07.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:06 smithi045 conmon[39718]: .24542) 291 : cluster [DBG] 2021-07-31T01:54:07.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:06 smithi045 conmon[39718]: pgmap v597: 25 pgs: 25 active+clean; 206 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 26 MiB/s rd, 9.6 MiB/s wr, 29 op/s 2021-07-31T01:54:07.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:06 smithi045 conmon[39718]: cluster 2021-07-31T01:54:07.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:06 smithi045 conmon[39718]: 2021-07-31T01:54 2021-07-31T01:54:07.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:06 smithi045 conmon[39718]: :06.599869+0000 mon.a 2021-07-31T01:54:07.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:06 smithi045 conmon[39718]: (mon.0) 1201 : cluster 2021-07-31T01:54:07.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:06 smithi045 conmon[39718]: [DBG] osdmap e545: 8 total, 8 up, 8 in 2021-07-31T01:54:07.299 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T01:54:07.576 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T01:54:07.577 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 9m ago 9m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T01:54:07.577 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 6m ago 9m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T01:54:07.577 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 6m ago 13m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T01:54:07.578 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 9m ago 15m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T01:54:07.578 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 9m ago 15m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T01:54:07.578 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 6m ago 13m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T01:54:07.578 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 9m ago 13m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T01:54:07.579 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 9m ago 9m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T01:54:07.579 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 6m ago 9m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T01:54:07.579 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 9m ago 12m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T01:54:07.580 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 9m ago 12m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T01:54:07.580 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 9m ago 12m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T01:54:07.580 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 9m ago 11m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T01:54:07.580 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 6m ago 11m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T01:54:07.580 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 6m ago 11m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T01:54:07.581 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 6m ago 10m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T01:54:07.581 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 6m ago 10m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T01:54:07.581 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 6m ago 10m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T01:54:07.605 INFO:tasks.rados.rados.0.smithi045.stdout:2588: done (2 left) 2021-07-31T01:54:07.606 INFO:tasks.rados.rados.0.smithi045.stdout:2589: rmattr oid 15 current snap is 244 2021-07-31T01:54:07.606 INFO:tasks.rados.rados.0.smithi045.stdout:2585: finishing copy_from to smithi04569269-13 2021-07-31T01:54:07.606 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 13 v 1061 (ObjNum 713 snap 222 seq_num 713) dirty exists 2021-07-31T01:54:07.606 INFO:tasks.rados.rados.0.smithi045.stdout:2585: finishing copy_from racing read to smithi04569269-13 2021-07-31T01:54:07.607 INFO:tasks.rados.rados.0.smithi045.stdout:2587: finishing copy_from to smithi04569269-18 2021-07-31T01:54:07.607 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 18 v 1066 (ObjNum 765 snap 235 seq_num 765) dirty exists 2021-07-31T01:54:07.607 INFO:tasks.rados.rados.0.smithi045.stdout:2587: finishing copy_from racing read to smithi04569269-18 2021-07-31T01:54:07.607 INFO:tasks.rados.rados.0.smithi045.stdout:2585: done (2 left) 2021-07-31T01:54:07.607 INFO:tasks.rados.rados.0.smithi045.stdout:2587: done (1 left) 2021-07-31T01:54:07.608 INFO:tasks.rados.rados.0.smithi045.stdout:2590: setattr oid 48 current snap is 244 2021-07-31T01:54:07.608 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 27 current snap is 244 2021-07-31T01:54:07.608 INFO:tasks.rados.rados.0.smithi045.stdout:2591: seq_num 791 ranges {0=450560,450560=475136,925696=327680} 2021-07-31T01:54:07.610 INFO:tasks.rados.rados.0.smithi045.stdout:2591: writing smithi04569269-27 from 0 to 450560 tid 1 2021-07-31T01:54:07.613 INFO:tasks.rados.rados.0.smithi045.stdout:2591: writing smithi04569269-27 from 450560 to 925696 tid 2 2021-07-31T01:54:07.616 INFO:tasks.rados.rados.0.smithi045.stdout:2591: writing smithi04569269-27 from 925696 to 1253376 tid 3 2021-07-31T01:54:07.617 INFO:tasks.rados.rados.0.smithi045.stdout:2592: read oid 32 snap -1 2021-07-31T01:54:07.617 INFO:tasks.rados.rados.0.smithi045.stdout:2592: expect (ObjNum 754 snap 233 seq_num 754) 2021-07-31T01:54:07.617 INFO:tasks.rados.rados.0.smithi045.stdout:2593: read oid 8 snap -1 2021-07-31T01:54:07.617 INFO:tasks.rados.rados.0.smithi045.stdout:2593: expect (ObjNum 701 snap 218 seq_num 701) 2021-07-31T01:54:07.618 INFO:tasks.rados.rados.0.smithi045.stdout:2594: delete oid 36 current snap is 244 2021-07-31T01:54:07.618 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 15 v 1067 (ObjNum 781 snap 239 seq_num 781) dirty exists 2021-07-31T01:54:07.618 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 48 v 1402 (ObjNum 754 snap 233 seq_num 754) dirty exists 2021-07-31T01:54:07.618 INFO:tasks.rados.rados.0.smithi045.stdout:2591: finishing write tid 1 to smithi04569269-27 2021-07-31T01:54:07.619 INFO:tasks.rados.rados.0.smithi045.stdout:2591: finishing write tid 2 to smithi04569269-27 2021-07-31T01:54:07.621 INFO:tasks.rados.rados.0.smithi045.stdout:2591: finishing write tid 3 to smithi04569269-27 2021-07-31T01:54:07.622 INFO:tasks.rados.rados.0.smithi045.stdout:2591: finishing write tid 4 to smithi04569269-27 2021-07-31T01:54:07.625 INFO:tasks.rados.rados.0.smithi045.stdout:2591: finishing write tid 5 to smithi04569269-27 2021-07-31T01:54:07.625 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 27 v 1416 (ObjNum 791 snap 244 seq_num 791) dirty exists 2021-07-31T01:54:07.625 INFO:tasks.rados.rados.0.smithi045.stdout:2591: left oid 27 (ObjNum 791 snap 244 seq_num 791) 2021-07-31T01:54:07.627 INFO:tasks.rados.rados.0.smithi045.stdout:2593: expect (ObjNum 701 snap 218 seq_num 701) 2021-07-31T01:54:07.757 INFO:tasks.rados.rados.0.smithi045.stdout:2592: expect (ObjNum 754 snap 233 seq_num 754) 2021-07-31T01:54:07.929 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T01:54:07.930 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T01:54:07.930 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T01:54:07.930 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T01:54:07.930 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T01:54:07.931 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T01:54:07.931 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T01:54:07.931 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T01:54:07.931 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T01:54:07.931 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T01:54:07.932 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T01:54:07.932 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T01:54:07.932 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T01:54:07.932 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T01:54:07.932 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T01:54:07.933 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T01:54:07.933 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T01:54:08.468 INFO:tasks.rados.rados.0.smithi045.stdout:2589: done (5 left) 2021-07-31T01:54:08.468 INFO:tasks.rados.rados.0.smithi045.stdout:2590: done (4 left) 2021-07-31T01:54:08.468 INFO:tasks.rados.rados.0.smithi045.stdout:2591: done (3 left) 2021-07-31T01:54:08.469 INFO:tasks.rados.rados.0.smithi045.stdout:2592: done (2 left) 2021-07-31T01:54:08.469 INFO:tasks.rados.rados.0.smithi045.stdout:2593: done (1 left) 2021-07-31T01:54:08.469 INFO:tasks.rados.rados.0.smithi045.stdout:2594: done (0 left) 2021-07-31T01:54:08.469 INFO:tasks.rados.rados.0.smithi045.stdout:2595: rollback oid 41 current snap is 244 2021-07-31T01:54:08.469 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 41 to 242 2021-07-31T01:54:08.470 INFO:tasks.rados.rados.0.smithi045.stdout:2596: read oid 26 snap -1 2021-07-31T01:54:08.470 INFO:tasks.rados.rados.0.smithi045.stdout:2596: expect (ObjNum 760 snap 233 seq_num 760) 2021-07-31T01:54:08.470 INFO:tasks.rados.rados.0.smithi045.stdout:2597: read oid 2 snap 242 2021-07-31T01:54:08.470 INFO:tasks.rados.rados.0.smithi045.stdout:2597: expect deleted 2021-07-31T01:54:08.471 INFO:tasks.rados.rados.0.smithi045.stdout:2598: rollback oid 40 current snap is 244 2021-07-31T01:54:08.471 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 40 to 239 2021-07-31T01:54:08.471 INFO:tasks.rados.rados.0.smithi045.stdout:2599: read oid 13 snap -1 2021-07-31T01:54:08.471 INFO:tasks.rados.rados.0.smithi045.stdout:2599: expect (ObjNum 713 snap 222 seq_num 713) 2021-07-31T01:54:08.471 INFO:tasks.rados.rados.0.smithi045.stdout:2600: copy_from oid 15 from oid 10 current snap is 244 2021-07-31T01:54:08.472 INFO:tasks.rados.rados.0.smithi045.stdout:2601: read oid 23 snap 242 2021-07-31T01:54:08.472 INFO:tasks.rados.rados.0.smithi045.stdout:2601: expect deleted 2021-07-31T01:54:08.472 INFO:tasks.rados.rados.0.smithi045.stdout:2602: copy_from oid 47 from oid 21 current snap is 244 2021-07-31T01:54:08.472 INFO:tasks.rados.rados.0.smithi045.stdout:2603: snap_remove snap 240 2021-07-31T01:54:08.612 INFO:tasks.rados.rados.0.smithi045.stdout:2603: done (8 left) 2021-07-31T01:54:08.612 INFO:tasks.rados.rados.0.smithi045.stdout:2604: setattr oid 50 current snap is 244 2021-07-31T01:54:08.612 INFO:tasks.rados.rados.0.smithi045.stdout:2602: finishing copy_from to smithi04569269-47 2021-07-31T01:54:08.612 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 47 v 1625 (ObjNum 783 snap 239 seq_num 783) dirty exists 2021-07-31T01:54:08.613 INFO:tasks.rados.rados.0.smithi045.stdout:2595: finishing rollback tid 0 to smithi04569269-41 2021-07-31T01:54:08.613 INFO:tasks.rados.rados.0.smithi045.stdout:2602: finishing copy_from racing read to smithi04569269-47 2021-07-31T01:54:08.613 INFO:tasks.rados.rados.0.smithi045.stdout:2595: finishing rollback tid 1 to smithi04569269-41 2021-07-31T01:54:08.613 INFO:tasks.rados.rados.0.smithi045.stdout:2595: finishing rollback tid 2 to smithi04569269-41 2021-07-31T01:54:08.613 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 41 v 1619 (ObjNum 701 snap 218 seq_num 701) dirty exists 2021-07-31T01:54:08.614 INFO:tasks.rados.rados.0.smithi045.stdout:2598: finishing rollback tid 0 to smithi04569269-40 2021-07-31T01:54:08.614 INFO:tasks.rados.rados.0.smithi045.stdout:2598: finishing rollback tid 1 to smithi04569269-40 2021-07-31T01:54:08.615 INFO:tasks.rados.rados.0.smithi045.stdout:2598: finishing rollback tid 2 to smithi04569269-40 2021-07-31T01:54:08.615 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 40 v 1593 (ObjNum 778 snap 237 seq_num 778) dirty exists 2021-07-31T01:54:08.615 INFO:tasks.rados.rados.0.smithi045.stdout:2596: expect (ObjNum 760 snap 233 seq_num 760) 2021-07-31T01:54:08.828 INFO:tasks.rados.rados.0.smithi045.stdout:2599: expect (ObjNum 713 snap 222 seq_num 713) 2021-07-31T01:54:08.964 INFO:tasks.rados.rados.0.smithi045.stdout:2600: finishing copy_from to smithi04569269-15 2021-07-31T01:54:08.964 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 15 v 1576 (ObjNum 768 snap 235 seq_num 768) dirty exists 2021-07-31T01:54:08.965 INFO:tasks.rados.rados.0.smithi045.stdout:2600: finishing copy_from racing read to smithi04569269-15 2021-07-31T01:54:08.965 INFO:tasks.rados.rados.0.smithi045.stdout:2595: done (8 left) 2021-07-31T01:54:08.965 INFO:tasks.rados.rados.0.smithi045.stdout:2596: done (7 left) 2021-07-31T01:54:08.966 INFO:tasks.rados.rados.0.smithi045.stdout:2597: done (6 left) 2021-07-31T01:54:08.966 INFO:tasks.rados.rados.0.smithi045.stdout:2598: done (5 left) 2021-07-31T01:54:08.967 INFO:tasks.rados.rados.0.smithi045.stdout:2599: done (4 left) 2021-07-31T01:54:08.967 INFO:tasks.rados.rados.0.smithi045.stdout:2600: done (3 left) 2021-07-31T01:54:08.968 INFO:tasks.rados.rados.0.smithi045.stdout:2601: done (2 left) 2021-07-31T01:54:08.968 INFO:tasks.rados.rados.0.smithi045.stdout:2602: done (1 left) 2021-07-31T01:54:08.969 INFO:tasks.rados.rados.0.smithi045.stdout:2605: copy_from oid 37 from oid 5 current snap is 244 2021-07-31T01:54:08.969 INFO:tasks.rados.rados.0.smithi045.stdout:2606: read oid 1 snap 242 2021-07-31T01:54:08.970 INFO:tasks.rados.rados.0.smithi045.stdout:2606: expect (ObjNum 713 snap 222 seq_num 713) 2021-07-31T01:54:08.970 INFO:tasks.rados.rados.0.smithi045.stdout:2607: read oid 4 snap 241 2021-07-31T01:54:08.971 INFO:tasks.rados.rados.0.smithi045.stdout:2607: expect (ObjNum 716 snap 222 seq_num 716) 2021-07-31T01:54:08.971 INFO:tasks.rados.rados.0.smithi045.stdout:2608: read oid 31 snap -1 2021-07-31T01:54:08.972 INFO:tasks.rados.rados.0.smithi045.stdout:2608: expect (ObjNum 726 snap 226 seq_num 726) 2021-07-31T01:54:08.972 INFO:tasks.rados.rados.0.smithi045.stdout:2609: delete oid 24 current snap is 244 2021-07-31T01:54:08.975 INFO:tasks.rados.rados.0.smithi045.stdout:2608: expect (ObjNum 726 snap 226 seq_num 726) 2021-07-31T01:54:08.975 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 50 v 1617 (ObjNum 701 snap 218 seq_num 701) dirty exists 2021-07-31T01:54:08.976 INFO:tasks.rados.rados.0.smithi045.stdout:2604: done (5 left) 2021-07-31T01:54:08.976 INFO:tasks.rados.rados.0.smithi045.stdout:2608: done (4 left) 2021-07-31T01:54:08.976 INFO:tasks.rados.rados.0.smithi045.stdout:2609: done (3 left) 2021-07-31T01:54:08.977 INFO:tasks.rados.rados.0.smithi045.stdout:2610: read oid 16 snap -1 2021-07-31T01:54:08.978 INFO:tasks.rados.rados.0.smithi045.stdout:2610: expect (ObjNum 684 snap 214 seq_num 684) 2021-07-31T01:54:08.978 INFO:tasks.rados.rados.0.smithi045.stdout:2611: read oid 22 snap 241 2021-07-31T01:54:08.978 INFO:tasks.rados.rados.0.smithi045.stdout:2611: expect (ObjNum 747 snap 230 seq_num 747) 2021-07-31T01:54:08.979 INFO:tasks.rados.rados.0.smithi045.stdout:2612: read oid 21 snap -1 2021-07-31T01:54:08.979 INFO:tasks.rados.rados.0.smithi045.stdout:2612: expect (ObjNum 783 snap 239 seq_num 783) 2021-07-31T01:54:08.980 INFO:tasks.rados.rados.0.smithi045.stdout:2613: snap_create 2021-07-31T01:54:08.981 INFO:tasks.rados.rados.0.smithi045.stdout:2607: expect (ObjNum 716 snap 222 seq_num 716) 2021-07-31T01:54:09.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:08 smithi154 conmon[34553]: audit 2021-07-31T01:54:07.287321+0000 mgr.x (mgr.24542) 292 : audit [DBG] from='client.24853 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:54:09.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:08 smithi154 conmon[34553]: audit 2021-07-31T01:54:07.568725+0000 mgr.x (mgr.24542) 293 : audit [DBG] from='client.14952 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:54:09.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:08 smithi154 conmon[34553]: cluster 2021-07-31T01:54:09.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:08 smithi154 conmon[34553]: 2021-07-31T01:54:07.604884+0000 2021-07-31T01:54:09.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:08 smithi154 conmon[34553]: mon.a (mon.0) 1202 : cluster [DBG] 2021-07-31T01:54:09.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:08 smithi154 conmon[34553]: osdmap e546: 8 total, 8 up, 8 in 2021-07-31T01:54:09.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:08 smithi154 conmon[34553]: cluster 2021-07-31T01 2021-07-31T01:54:09.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:08 smithi154 conmon[34553]: :54:07.889630+0000 mgr.x (mgr.24542) 294 : cluster 2021-07-31T01:54:09.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:08 smithi154 conmon[34553]: [DBG] pgmap v600: 25 pgs: 25 active+clean; 206 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail 2021-07-31T01:54:09.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:08 smithi154 conmon[34553]: audit 2021 2021-07-31T01:54:09.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:08 smithi154 conmon[34553]: -07-31T01:54:07.928231+0000 2021-07-31T01:54:09.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:08 smithi154 conmon[34553]: mon.a (mon.0) 1203 : audit [DBG] from='client.? 172.21.15.45:0/2828497773' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T01:54:09.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:08 smithi045 conmon[39718]: audit 2021-07-31T01:54:07.287321+0000 2021-07-31T01:54:09.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:08 smithi045 conmon[39718]: mgr.x (mgr.24542) 292 : audit [DBG] from='client.24853 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:54:09.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:08 smithi045 conmon[39718]: audit 2021-07-31T01: 2021-07-31T01:54:09.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:08 smithi045 conmon[39718]: 54:07.568725+0000 mgr.x (mgr.24542) 293 : audit [DBG] from='client.14952 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:54:09.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:08 smithi045 conmon[39718]: cluster 2021-07-31T01:54:07.604884+0000 2021-07-31T01:54:09.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:08 smithi045 conmon[39718]: mon.a (mon.0) 1202 : cluster [DBG] osdmap e546: 8 total, 8 up, 8 in 2021-07-31T01:54:09.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:08 smithi045 conmon[39718]: cluster 2021-07-31T01:54:07 2021-07-31T01:54:09.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:08 smithi045 conmon[39718]: .889630+0000 mgr.x (mgr.24542) 294 : cluster [DBG] pgmap v600: 25 pgs: 25 active+clean; 206 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail 2021-07-31T01:54:09.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:08 smithi045 conmon[39718]: audit 2021-07-31T01 2021-07-31T01:54:09.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:08 smithi045 conmon[39718]: :54:07.928231+0000 mon.a (mon.0) 1203 : audit 2021-07-31T01:54:09.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:08 smithi045 conmon[39718]: [DBG] from='client.? 172.21.15.45:0/2828497773' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T01:54:09.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:08 smithi045 conmon[33655]: audit 2021-07-31T01:54:07.287321 2021-07-31T01:54:09.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:08 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 292 : audit [DBG] from='client.24853 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:54:09.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:08 smithi045 conmon[33655]: audit 2021-07-31T01:54:07.568725+0000 mgr.x (mgr.24542 2021-07-31T01:54:09.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:08 smithi045 conmon[33655]: ) 293 : audit [DBG] from='client.14952 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:54:09.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:08 smithi045 conmon[33655]: cluster 2021-07-31T01:54:09.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:08 smithi045 conmon[33655]: 2021-07-31T01:54:07 2021-07-31T01:54:09.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:08 smithi045 conmon[33655]: .604884+0000 mon.a (mon.0) 1202 2021-07-31T01:54:09.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:08 smithi045 conmon[33655]: : cluster [DBG] osdmap e546: 8 total, 8 up, 8 in 2021-07-31T01:54:09.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:08 smithi045 conmon[33655]: cluster 2021 2021-07-31T01:54:09.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:08 smithi045 conmon[33655]: -07-31T01:54 2021-07-31T01:54:09.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:08 smithi045 conmon[33655]: :07.889630+0000 mgr.x (mgr 2021-07-31T01:54:09.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:08 smithi045 conmon[33655]: .24542) 294 : cluster [DBG] pgmap v600: 25 pgs: 25 active+clean; 206 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail 2021-07-31T01:54:09.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:08 smithi045 conmon[33655]: audit 2021 2021-07-31T01:54:09.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:08 smithi045 conmon[33655]: -07-31T01:54:07.928231+0000 mon.a 2021-07-31T01:54:09.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:08 smithi045 conmon[33655]: (mon.0) 1203 : audit [DBG] from='client.? 172.21.15.45:0/2828497773' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T01:54:09.042 INFO:tasks.rados.rados.0.smithi045.stdout:2612: expect (ObjNum 783 snap 239 seq_num 783) 2021-07-31T01:54:09.071 INFO:tasks.rados.rados.0.smithi045.stdout:2606: expect (ObjNum 713 snap 222 seq_num 713) 2021-07-31T01:54:09.204 INFO:tasks.rados.rados.0.smithi045.stdout:2610: expect (ObjNum 684 snap 214 seq_num 684) 2021-07-31T01:54:09.283 INFO:tasks.rados.rados.0.smithi045.stdout:2611: expect (ObjNum 747 snap 230 seq_num 747) 2021-07-31T01:54:09.374 INFO:tasks.rados.rados.0.smithi045.stdout:2605: finishing copy_from to smithi04569269-37 2021-07-31T01:54:09.374 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 37 v 1609 (ObjNum 754 snap 233 seq_num 754) dirty exists 2021-07-31T01:54:09.375 INFO:tasks.rados.rados.0.smithi045.stdout:2605: finishing copy_from racing read to smithi04569269-37 2021-07-31T01:54:09.615 INFO:tasks.rados.rados.0.smithi045.stdout:2605: done (6 left) 2021-07-31T01:54:09.616 INFO:tasks.rados.rados.0.smithi045.stdout:2606: done (5 left) 2021-07-31T01:54:09.616 INFO:tasks.rados.rados.0.smithi045.stdout:2607: done (4 left) 2021-07-31T01:54:09.616 INFO:tasks.rados.rados.0.smithi045.stdout:2610: done (3 left) 2021-07-31T01:54:09.616 INFO:tasks.rados.rados.0.smithi045.stdout:2611: done (2 left) 2021-07-31T01:54:09.617 INFO:tasks.rados.rados.0.smithi045.stdout:2612: done (1 left) 2021-07-31T01:54:09.617 INFO:tasks.rados.rados.0.smithi045.stdout:2613: done (0 left) 2021-07-31T01:54:09.617 INFO:tasks.rados.rados.0.smithi045.stdout:2614: delete oid 12 current snap is 245 2021-07-31T01:54:09.619 INFO:tasks.rados.rados.0.smithi045.stdout:2614: done (0 left) 2021-07-31T01:54:09.619 INFO:tasks.rados.rados.0.smithi045.stdout:2615: rmattr oid 25 current snap is 245 2021-07-31T01:54:09.620 INFO:tasks.rados.rados.0.smithi045.stdout:2615: done (0 left) 2021-07-31T01:54:09.620 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 40 current snap is 245 2021-07-31T01:54:09.620 INFO:tasks.rados.rados.0.smithi045.stdout:2616: seq_num 795 ranges {1040384=524288,1564672=417792,1982464=385024} 2021-07-31T01:54:09.627 INFO:tasks.rados.rados.0.smithi045.stdout:2616: writing smithi04569269-40 from 1040384 to 1564672 tid 1 2021-07-31T01:54:09.630 INFO:tasks.rados.rados.0.smithi045.stdout:2616: writing smithi04569269-40 from 1564672 to 1982464 tid 2 2021-07-31T01:54:09.632 INFO:tasks.rados.rados.0.smithi045.stdout:2616: writing smithi04569269-40 from 1982464 to 2367488 tid 3 2021-07-31T01:54:09.632 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 29 current snap is 245 2021-07-31T01:54:09.632 INFO:tasks.rados.rados.0.smithi045.stdout:2616: finishing write tid 1 to smithi04569269-40 2021-07-31T01:54:09.633 INFO:tasks.rados.rados.0.smithi045.stdout:2617: seq_num 796 ranges {9011200=540672,9551872=704512,10256384=98304} 2021-07-31T01:54:09.684 INFO:tasks.rados.rados.0.smithi045.stdout:2617: writing smithi04569269-29 from 9011200 to 9551872 tid 1 2021-07-31T01:54:09.688 INFO:tasks.rados.rados.0.smithi045.stdout:2617: writing smithi04569269-29 from 9551872 to 10256384 tid 2 2021-07-31T01:54:09.689 INFO:tasks.rados.rados.0.smithi045.stdout:2617: writing smithi04569269-29 from 10256384 to 10354688 tid 3 2021-07-31T01:54:09.689 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 20 current snap is 245 2021-07-31T01:54:09.689 INFO:tasks.rados.rados.0.smithi045.stdout:2616: finishing write tid 2 to smithi04569269-40 2021-07-31T01:54:09.689 INFO:tasks.rados.rados.0.smithi045.stdout:2618: seq_num 797 ranges {9011200=557056,9568256=589824,10158080=212992} 2021-07-31T01:54:09.740 INFO:tasks.rados.rados.0.smithi045.stdout:2618: writing smithi04569269-20 from 9011200 to 9568256 tid 1 2021-07-31T01:54:09.744 INFO:tasks.rados.rados.0.smithi045.stdout:2618: writing smithi04569269-20 from 9568256 to 10158080 tid 2 2021-07-31T01:54:09.746 INFO:tasks.rados.rados.0.smithi045.stdout:2618: writing smithi04569269-20 from 10158080 to 10371072 tid 3 2021-07-31T01:54:09.746 INFO:tasks.rados.rados.0.smithi045.stdout:2619: copy_from oid 5 from oid 36 current snap is 245 2021-07-31T01:54:09.746 INFO:tasks.rados.rados.0.smithi045.stdout:2616: finishing write tid 3 to smithi04569269-40 2021-07-31T01:54:09.747 INFO:tasks.rados.rados.0.smithi045.stdout:2616: finishing write tid 4 to smithi04569269-40 2021-07-31T01:54:09.747 INFO:tasks.rados.rados.0.smithi045.stdout:2616: finishing write tid 5 to smithi04569269-40 2021-07-31T01:54:09.747 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 40 v 1598 (ObjNum 795 snap 245 seq_num 795) dirty exists 2021-07-31T01:54:09.747 INFO:tasks.rados.rados.0.smithi045.stdout:2616: left oid 40 (ObjNum 795 snap 245 seq_num 795) 2021-07-31T01:54:09.747 INFO:tasks.rados.rados.0.smithi045.stdout:2617: finishing write tid 1 to smithi04569269-29 2021-07-31T01:54:09.748 INFO:tasks.rados.rados.0.smithi045.stdout:2617: finishing write tid 2 to smithi04569269-29 2021-07-31T01:54:09.748 INFO:tasks.rados.rados.0.smithi045.stdout:2617: finishing write tid 3 to smithi04569269-29 2021-07-31T01:54:09.748 INFO:tasks.rados.rados.0.smithi045.stdout:2617: finishing write tid 4 to smithi04569269-29 2021-07-31T01:54:09.748 INFO:tasks.rados.rados.0.smithi045.stdout:2617: finishing write tid 5 to smithi04569269-29 2021-07-31T01:54:09.749 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 29 v 1405 (ObjNum 796 snap 245 seq_num 796) dirty exists 2021-07-31T01:54:09.749 INFO:tasks.rados.rados.0.smithi045.stdout:2617: left oid 29 (ObjNum 796 snap 245 seq_num 796) 2021-07-31T01:54:09.749 INFO:tasks.rados.rados.0.smithi045.stdout:2616: done (3 left) 2021-07-31T01:54:09.749 INFO:tasks.rados.rados.0.smithi045.stdout:2617: done (2 left) 2021-07-31T01:54:09.749 INFO:tasks.rados.rados.0.smithi045.stdout:2620: snap_create 2021-07-31T01:54:09.751 INFO:tasks.rados.rados.0.smithi045.stdout:2618: finishing write tid 1 to smithi04569269-20 2021-07-31T01:54:09.751 INFO:tasks.rados.rados.0.smithi045.stdout:2618: finishing write tid 2 to smithi04569269-20 2021-07-31T01:54:09.751 INFO:tasks.rados.rados.0.smithi045.stdout:2618: finishing write tid 3 to smithi04569269-20 2021-07-31T01:54:09.752 INFO:tasks.rados.rados.0.smithi045.stdout:2619: finishing copy_from to smithi04569269-5 2021-07-31T01:54:09.752 INFO:tasks.rados.rados.0.smithi045.stdout:2619: got expected ENOENT (src dne) 2021-07-31T01:54:09.752 INFO:tasks.rados.rados.0.smithi045.stdout:2618: finishing write tid 4 to smithi04569269-20 2021-07-31T01:54:09.752 INFO:tasks.rados.rados.0.smithi045.stdout:2619: finishing copy_from racing read to smithi04569269-5 2021-07-31T01:54:09.753 INFO:tasks.rados.rados.0.smithi045.stdout:2618: finishing write tid 5 to smithi04569269-20 2021-07-31T01:54:09.753 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 20 v 1640 (ObjNum 797 snap 245 seq_num 797) dirty exists 2021-07-31T01:54:09.753 INFO:tasks.rados.rados.0.smithi045.stdout:2618: left oid 20 (ObjNum 797 snap 245 seq_num 797) 2021-07-31T01:54:10.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:09 smithi154 conmon[34553]: cluster 2021-07-31T01:54:08.609394+0000 mon.a (mon.0) 1204 : cluster [DBG] osdmap e547: 8 total, 8 up, 8 in 2021-07-31T01:54:10.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:09 smithi045 conmon[39718]: cluster 2021-07-31T01:54:08.609394+0000 2021-07-31T01:54:10.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:09 smithi045 conmon[39718]: mon.a (mon.0) 1204 : cluster [DBG] osdmap e547: 8 total, 8 up, 8 in 2021-07-31T01:54:10.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:09 smithi045 conmon[33655]: cluster 2021-07-31T01:54:08.609394+0000 mon.a (mon. 2021-07-31T01:54:10.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:09 smithi045 conmon[33655]: 0) 1204 : cluster [DBG] osdmap e547: 8 total, 8 up, 8 in 2021-07-31T01:54:10.622 INFO:tasks.rados.rados.0.smithi045.stdout:2618: done (2 left) 2021-07-31T01:54:10.623 INFO:tasks.rados.rados.0.smithi045.stdout:2619: done (1 left) 2021-07-31T01:54:10.623 INFO:tasks.rados.rados.0.smithi045.stdout:2620: done (0 left) 2021-07-31T01:54:10.623 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 13 current snap is 246 2021-07-31T01:54:10.623 INFO:tasks.rados.rados.0.smithi045.stdout:2621: seq_num 798 ranges {3792896=573440,4366336=466944,4833280=335872} 2021-07-31T01:54:10.646 INFO:tasks.rados.rados.0.smithi045.stdout:2621: writing smithi04569269-13 from 3792896 to 4366336 tid 1 2021-07-31T01:54:10.648 INFO:tasks.rados.rados.0.smithi045.stdout:2621: writing smithi04569269-13 from 4366336 to 4833280 tid 2 2021-07-31T01:54:10.650 INFO:tasks.rados.rados.0.smithi045.stdout:2621: writing smithi04569269-13 from 4833280 to 5169152 tid 3 2021-07-31T01:54:10.651 INFO:tasks.rados.rados.0.smithi045.stdout:2622: rollback oid 43 current snap is 246 2021-07-31T01:54:10.651 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 43 to 241 2021-07-31T01:54:10.651 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 30 current snap is 246 2021-07-31T01:54:10.652 INFO:tasks.rados.rados.0.smithi045.stdout:2623: seq_num 799 ranges {7798784=589824,8388608=802816} 2021-07-31T01:54:10.696 INFO:tasks.rados.rados.0.smithi045.stdout:2623: writing smithi04569269-30 from 7798784 to 8388608 tid 1 2021-07-31T01:54:10.702 INFO:tasks.rados.rados.0.smithi045.stdout:2623: writing smithi04569269-30 from 8388608 to 9191424 tid 2 2021-07-31T01:54:10.702 INFO:tasks.rados.rados.0.smithi045.stdout:2624: snap_create 2021-07-31T01:54:10.702 INFO:tasks.rados.rados.0.smithi045.stdout:2621: finishing write tid 1 to smithi04569269-13 2021-07-31T01:54:10.702 INFO:tasks.rados.rados.0.smithi045.stdout:2621: finishing write tid 2 to smithi04569269-13 2021-07-31T01:54:10.703 INFO:tasks.rados.rados.0.smithi045.stdout:2622: finishing rollback tid 0 to smithi04569269-43 2021-07-31T01:54:10.703 INFO:tasks.rados.rados.0.smithi045.stdout:2621: finishing write tid 3 to smithi04569269-13 2021-07-31T01:54:10.703 INFO:tasks.rados.rados.0.smithi045.stdout:2621: finishing write tid 4 to smithi04569269-13 2021-07-31T01:54:10.703 INFO:tasks.rados.rados.0.smithi045.stdout:2622: finishing rollback tid 1 to smithi04569269-43 2021-07-31T01:54:10.704 INFO:tasks.rados.rados.0.smithi045.stdout:2622: finishing rollback tid 2 to smithi04569269-43 2021-07-31T01:54:10.704 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 43 v 1419 (ObjNum 754 snap 233 seq_num 754) dirty exists 2021-07-31T01:54:10.704 INFO:tasks.rados.rados.0.smithi045.stdout:2621: finishing write tid 5 to smithi04569269-13 2021-07-31T01:54:10.704 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 13 v 1602 (ObjNum 798 snap 246 seq_num 798) dirty exists 2021-07-31T01:54:10.704 INFO:tasks.rados.rados.0.smithi045.stdout:2621: left oid 13 (ObjNum 798 snap 246 seq_num 798) 2021-07-31T01:54:10.705 INFO:tasks.rados.rados.0.smithi045.stdout:2623: finishing write tid 1 to smithi04569269-30 2021-07-31T01:54:10.706 INFO:tasks.rados.rados.0.smithi045.stdout:2623: finishing write tid 2 to smithi04569269-30 2021-07-31T01:54:10.706 INFO:tasks.rados.rados.0.smithi045.stdout:2623: finishing write tid 3 to smithi04569269-30 2021-07-31T01:54:10.707 INFO:tasks.rados.rados.0.smithi045.stdout:2623: finishing write tid 4 to smithi04569269-30 2021-07-31T01:54:10.707 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 30 v 1601 (ObjNum 799 snap 246 seq_num 799) dirty exists 2021-07-31T01:54:10.707 INFO:tasks.rados.rados.0.smithi045.stdout:2623: left oid 30 (ObjNum 799 snap 246 seq_num 799) 2021-07-31T01:54:11.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:10 smithi154 conmon[34553]: cluster 2021-07-31T01:54:11.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:10 smithi154 conmon[34553]: 2021-07-31T01:54:09. 2021-07-31T01:54:11.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:10 smithi154 conmon[34553]: 614688+0000 mon.a (mon.0) 1205 : cluster [DBG] osdmap e548: 8 total, 8 up, 8 in 2021-07-31T01:54:11.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:10 smithi154 conmon[34553]: cluster 2021-07-31T01:54:11.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:10 smithi154 conmon[34553]: 2021-07-31T01:54:09.890045+0000 2021-07-31T01:54:11.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:10 smithi154 conmon[34553]: mgr.x (mgr.24542) 295 : cluster [DBG] pgmap v603: 25 pgs: 25 active+clean; 234 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 15 MiB/s rd, 6.8 MiB/s wr, 25 op/s 2021-07-31T01:54:11.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:10 smithi045 conmon[33655]: cluster 2021-07-31T01:54:09.614688+0000 2021-07-31T01:54:11.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:10 smithi045 conmon[33655]: mon.a (mon.0) 1205 : cluster [DBG] osdmap e548: 8 total, 8 up, 8 in 2021-07-31T01:54:11.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:10 smithi045 conmon[33655]: cluster 2021-07-31T01:54:09.890045+0000 mgr.x (mgr.24542) 295 : cluster [DBG] pgmap v603: 25 pgs: 25 active+clean; 234 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 15 MiB/s rd, 6.8 MiB/s wr, 25 op/s 2021-07-31T01:54:11.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:10 smithi045 conmon[39718]: cluster 2021-07-31T01:54:09.614688+0000 mon.a (mon.0) 2021-07-31T01:54:11.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:10 smithi045 conmon[39718]: 1205 : cluster [DBG] osdmap e548: 8 total, 8 up, 8 in 2021-07-31T01:54:11.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:10 smithi045 conmon[39718]: cluster 2021-07-31T01:54:09.890045+0000 mgr.x (mgr.24542) 295 : cluster [DBG] pgmap v603: 25 pgs: 25 active+clean; 234 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 15 MiB/s rd, 6.8 MiB/s wr, 25 op/s 2021-07-31T01:54:11.602 INFO:tasks.rados.rados.0.smithi045.stdout:2621: done (3 left) 2021-07-31T01:54:11.603 INFO:tasks.rados.rados.0.smithi045.stdout:2622: done (2 left) 2021-07-31T01:54:11.603 INFO:tasks.rados.rados.0.smithi045.stdout:2623: done (1 left) 2021-07-31T01:54:11.603 INFO:tasks.rados.rados.0.smithi045.stdout:2624: done (0 left) 2021-07-31T01:54:11.603 INFO:tasks.rados.rados.0.smithi045.stdout:2625: snap_remove snap 245 2021-07-31T01:54:12.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:11 smithi154 conmon[34553]: cluster 2021-07-31T01:54:10.621782+0000 mon.a (mon.0) 1206 : cluster [DBG] osdmap e549: 8 total, 8 up, 8 in 2021-07-31T01:54:12.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:11 smithi154 conmon[34553]: cluster 2021-07-31T01:54:11.601060+0000 mon.a (mon.0) 1207 : cluster [DBG] osdmap e550: 8 total, 8 up, 8 in 2021-07-31T01:54:12.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:11 smithi045 conmon[33655]: cluster 2021-07-31T01:54:10.621782+0000 mon.a (mon.0) 1206 : cluster 2021-07-31T01:54:12.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:11 smithi045 conmon[33655]: [DBG] osdmap e549: 8 total, 8 up, 8 in 2021-07-31T01:54:12.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:11 smithi045 conmon[33655]: cluster 2021-07-31T01:54:11.601060+0000 mon.a (mon.0) 1207 : cluster [DBG] osdmap e550: 8 total, 8 up, 8 in 2021-07-31T01:54:12.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:11 smithi045 conmon[39718]: cluster 2021-07-31T01:54:10.621782+0000 mon.a 2021-07-31T01:54:12.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:11 smithi045 conmon[39718]: (mon.0) 1206 : cluster [DBG] osdmap e549: 8 total, 8 up, 8 in 2021-07-31T01:54:12.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:11 smithi045 conmon[39718]: cluster 2021 2021-07-31T01:54:12.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:11 smithi045 conmon[39718]: -07-31T01:54:11.601060+0000 mon.a ( 2021-07-31T01:54:12.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:11 smithi045 conmon[39718]: mon.0) 1207 : cluster [DBG] osdmap e550: 8 total, 8 up, 8 in 2021-07-31T01:54:12.607 INFO:tasks.rados.rados.0.smithi045.stdout:2625: done (0 left) 2021-07-31T01:54:12.608 INFO:tasks.rados.rados.0.smithi045.stdout:2626: setattr oid 40 current snap is 247 2021-07-31T01:54:12.609 INFO:tasks.rados.rados.0.smithi045.stdout:2627: read oid 41 snap 241 2021-07-31T01:54:12.609 INFO:tasks.rados.rados.0.smithi045.stdout:2627: expect (ObjNum 701 snap 218 seq_num 701) 2021-07-31T01:54:12.609 INFO:tasks.rados.rados.0.smithi045.stdout:2628: read oid 47 snap -1 2021-07-31T01:54:12.609 INFO:tasks.rados.rados.0.smithi045.stdout:2628: expect (ObjNum 783 snap 239 seq_num 783) 2021-07-31T01:54:12.609 INFO:tasks.rados.rados.0.smithi045.stdout:2629: copy_from oid 12 from oid 49 current snap is 247 2021-07-31T01:54:12.610 INFO:tasks.rados.rados.0.smithi045.stdout:2630: copy_from oid 16 from oid 14 current snap is 247 2021-07-31T01:54:12.610 INFO:tasks.rados.rados.0.smithi045.stdout:2631: snap_create 2021-07-31T01:54:12.616 INFO:tasks.rados.rados.0.smithi045.stdout:2629: finishing copy_from to smithi04569269-12 2021-07-31T01:54:12.616 INFO:tasks.rados.rados.0.smithi045.stdout:2629: got expected ENOENT (src dne) 2021-07-31T01:54:12.617 INFO:tasks.rados.rados.0.smithi045.stdout:2629: finishing copy_from racing read to smithi04569269-12 2021-07-31T01:54:12.617 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 40 v 1603 (ObjNum 795 snap 245 seq_num 795) dirty exists 2021-07-31T01:54:12.619 INFO:tasks.rados.rados.0.smithi045.stdout:2628: expect (ObjNum 783 snap 239 seq_num 783) 2021-07-31T01:54:12.653 INFO:tasks.rados.rados.0.smithi045.stdout:2627: expect (ObjNum 701 snap 218 seq_num 701) 2021-07-31T01:54:12.781 INFO:tasks.rados.rados.0.smithi045.stdout:2630: finishing copy_from to smithi04569269-16 2021-07-31T01:54:12.781 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 16 v 1407 (ObjNum 773 snap 236 seq_num 773) dirty exists 2021-07-31T01:54:12.782 INFO:tasks.rados.rados.0.smithi045.stdout:2630: finishing copy_from racing read to smithi04569269-16 2021-07-31T01:54:13.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:12 smithi154 conmon[34553]: cluster 2021-07-31T01:54:11.890602+0000 mgr.x (mgr.24542) 296 2021-07-31T01:54:13.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:12 smithi154 conmon[34553]: : cluster [DBG] pgmap v606: 25 pgs: 25 active+clean; 234 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 15 MiB/s rd, 6.8 MiB/s wr, 25 op/s 2021-07-31T01:54:13.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:12 smithi154 conmon[34553]: cluster 2021-07-31T01:54:12.607177+0000 mon.a (mon.0) 1208 : cluster [DBG] osdmap e551: 8 total, 8 up, 8 in 2021-07-31T01:54:13.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:12 smithi045 conmon[33655]: cluster 2021-07-31T01:54:11.890602+0000 mgr.x (mgr. 2021-07-31T01:54:13.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:12 smithi045 conmon[33655]: 24542) 296 : cluster [DBG] pgmap v606: 25 pgs: 25 active+clean; 234 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 15 MiB/s rd, 6.8 MiB/s wr, 25 op/s 2021-07-31T01:54:13.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:12 smithi045 conmon[33655]: cluster 2021-07-31T01:54:12.607177+0000 mon.a (mon.0) 1208 : cluster [DBG] osdmap e551: 8 total, 8 up, 8 in 2021-07-31T01:54:13.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:12 smithi045 conmon[39718]: cluster 2021-07-31T01:54:11.890602+0000 mgr.x ( 2021-07-31T01:54:13.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:12 smithi045 conmon[39718]: mgr.24542) 296 : cluster [DBG] pgmap v606: 25 pgs: 25 active+clean; 234 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 15 MiB/s rd, 6.8 MiB/s wr, 25 op/s 2021-07-31T01:54:13.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:12 smithi045 conmon[39718]: cluster 2021-07-31T01:54:12. 2021-07-31T01:54:13.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:12 smithi045 conmon[39718]: 607177+0000 mon.a (mon.0) 1208 : cluster [DBG] osdmap e551: 8 total, 8 up, 8 in 2021-07-31T01:54:13.609 INFO:tasks.rados.rados.0.smithi045.stdout:2626: done (5 left) 2021-07-31T01:54:13.609 INFO:tasks.rados.rados.0.smithi045.stdout:2627: done (4 left) 2021-07-31T01:54:13.610 INFO:tasks.rados.rados.0.smithi045.stdout:2628: done (3 left) 2021-07-31T01:54:13.610 INFO:tasks.rados.rados.0.smithi045.stdout:2629: done (2 left) 2021-07-31T01:54:13.610 INFO:tasks.rados.rados.0.smithi045.stdout:2630: done (1 left) 2021-07-31T01:54:13.610 INFO:tasks.rados.rados.0.smithi045.stdout:2631: done (0 left) 2021-07-31T01:54:13.611 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 18 current snap is 248 2021-07-31T01:54:13.611 INFO:tasks.rados.rados.0.smithi045.stdout:2632: seq_num 800 ranges {10076160=606208,10682368=688128,11370496=114688} 2021-07-31T01:54:13.667 INFO:tasks.rados.rados.0.smithi045.stdout:2632: writing smithi04569269-18 from 10076160 to 10682368 tid 1 2021-07-31T01:54:13.671 INFO:tasks.rados.rados.0.smithi045.stdout:2632: writing smithi04569269-18 from 10682368 to 11370496 tid 2 2021-07-31T01:54:13.672 INFO:tasks.rados.rados.0.smithi045.stdout:2632: writing smithi04569269-18 from 11370496 to 11485184 tid 3 2021-07-31T01:54:13.672 INFO:tasks.rados.rados.0.smithi045.stdout:2633: snap_create 2021-07-31T01:54:13.675 INFO:tasks.rados.rados.0.smithi045.stdout:2632: finishing write tid 1 to smithi04569269-18 2021-07-31T01:54:13.675 INFO:tasks.rados.rados.0.smithi045.stdout:2632: finishing write tid 2 to smithi04569269-18 2021-07-31T01:54:13.676 INFO:tasks.rados.rados.0.smithi045.stdout:2632: finishing write tid 3 to smithi04569269-18 2021-07-31T01:54:13.677 INFO:tasks.rados.rados.0.smithi045.stdout:2632: finishing write tid 4 to smithi04569269-18 2021-07-31T01:54:13.678 INFO:tasks.rados.rados.0.smithi045.stdout:2632: finishing write tid 5 to smithi04569269-18 2021-07-31T01:54:13.678 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 18 v 1613 (ObjNum 800 snap 248 seq_num 800) dirty exists 2021-07-31T01:54:13.679 INFO:tasks.rados.rados.0.smithi045.stdout:2632: left oid 18 (ObjNum 800 snap 248 seq_num 800) 2021-07-31T01:54:13.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:54:13 smithi045 conmon[61210]: level=error ts=2021-07-31T01:54:13.521Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:54:13.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:54:13 smithi045 conmon[61210]: level=error ts=2021-07-31T01:54:13.521Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:54:13.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:54:13 smithi045 conmon[61210]: level=error ts=2021-07-31T01:54:13.521Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T01:54:14.615 INFO:tasks.rados.rados.0.smithi045.stdout:2632: done (1 left) 2021-07-31T01:54:14.616 INFO:tasks.rados.rados.0.smithi045.stdout:2633: done (0 left) 2021-07-31T01:54:14.616 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 45 current snap is 249 2021-07-31T01:54:14.616 INFO:tasks.rados.rados.0.smithi045.stdout:2634: seq_num 801 ranges {1957888=622592,2580480=573440,3153920=229376} 2021-07-31T01:54:14.629 INFO:tasks.rados.rados.0.smithi045.stdout:2634: writing smithi04569269-45 from 1957888 to 2580480 tid 1 2021-07-31T01:54:14.632 INFO:tasks.rados.rados.0.smithi045.stdout:2634: writing smithi04569269-45 from 2580480 to 3153920 tid 2 2021-07-31T01:54:14.634 INFO:tasks.rados.rados.0.smithi045.stdout:2634: writing smithi04569269-45 from 3153920 to 3383296 tid 3 2021-07-31T01:54:14.634 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 17 current snap is 249 2021-07-31T01:54:14.634 INFO:tasks.rados.rados.0.smithi045.stdout:2635: seq_num 802 ranges {6938624=638976,7577600=458752,8036352=344064} 2021-07-31T01:54:14.675 INFO:tasks.rados.rados.0.smithi045.stdout:2635: writing smithi04569269-17 from 6938624 to 7577600 tid 1 2021-07-31T01:54:14.677 INFO:tasks.rados.rados.0.smithi045.stdout:2635: writing smithi04569269-17 from 7577600 to 8036352 tid 2 2021-07-31T01:54:14.680 INFO:tasks.rados.rados.0.smithi045.stdout:2635: writing smithi04569269-17 from 8036352 to 8380416 tid 3 2021-07-31T01:54:14.680 INFO:tasks.rados.rados.0.smithi045.stdout:2636: snap_create 2021-07-31T01:54:14.680 INFO:tasks.rados.rados.0.smithi045.stdout:2634: finishing write tid 1 to smithi04569269-45 2021-07-31T01:54:14.680 INFO:tasks.rados.rados.0.smithi045.stdout:2634: finishing write tid 2 to smithi04569269-45 2021-07-31T01:54:14.680 INFO:tasks.rados.rados.0.smithi045.stdout:2634: finishing write tid 3 to smithi04569269-45 2021-07-31T01:54:14.681 INFO:tasks.rados.rados.0.smithi045.stdout:2634: finishing write tid 4 to smithi04569269-45 2021-07-31T01:54:14.681 INFO:tasks.rados.rados.0.smithi045.stdout:2634: finishing write tid 5 to smithi04569269-45 2021-07-31T01:54:14.681 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 45 v 1621 (ObjNum 801 snap 249 seq_num 801) dirty exists 2021-07-31T01:54:14.681 INFO:tasks.rados.rados.0.smithi045.stdout:2634: left oid 45 (ObjNum 801 snap 249 seq_num 801) 2021-07-31T01:54:14.682 INFO:tasks.rados.rados.0.smithi045.stdout:2635: finishing write tid 1 to smithi04569269-17 2021-07-31T01:54:14.682 INFO:tasks.rados.rados.0.smithi045.stdout:2635: finishing write tid 2 to smithi04569269-17 2021-07-31T01:54:14.683 INFO:tasks.rados.rados.0.smithi045.stdout:2635: finishing write tid 3 to smithi04569269-17 2021-07-31T01:54:14.683 INFO:tasks.rados.rados.0.smithi045.stdout:2635: finishing write tid 4 to smithi04569269-17 2021-07-31T01:54:14.685 INFO:tasks.rados.rados.0.smithi045.stdout:2635: finishing write tid 5 to smithi04569269-17 2021-07-31T01:54:14.685 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 17 v 1625 (ObjNum 802 snap 249 seq_num 802) dirty exists 2021-07-31T01:54:14.686 INFO:tasks.rados.rados.0.smithi045.stdout:2635: left oid 17 (ObjNum 802 snap 249 seq_num 802) 2021-07-31T01:54:15.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:14 smithi154 conmon[34553]: cluster 2021-07-31T01:54:13.608471+0000 mon.a (mon.0) 1209 : cluster [DBG] osdmap e552: 8 total, 8 up, 8 in 2021-07-31T01:54:15.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:14 smithi154 conmon[34553]: cluster 2021-07-31T01:54:13.891212+0000 mgr.x (mgr.24542) 297 : cluster [DBG] pgmap v609: 25 pgs: 25 active+clean; 251 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 4.4 MiB/s wr, 27 op/s 2021-07-31T01:54:15.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:14 smithi045 conmon[39718]: cluster 2021-07-31T01:54:13.608471+0000 mon.a (mon.0) 2021-07-31T01:54:15.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:14 smithi045 conmon[39718]: 1209 : cluster [DBG] osdmap e552: 8 total, 8 up, 8 in 2021-07-31T01:54:15.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:14 smithi045 conmon[39718]: cluster 2021-07-31T01:54:13.891212+0000 mgr.x (mgr.24542) 297 : cluster [DBG] pgmap v609: 25 pgs: 25 active+clean; 251 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 4.4 MiB/s wr, 27 op/s 2021-07-31T01:54:15.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:14 smithi045 conmon[39718]: 2021-07-31T01:54:15.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:14 smithi045 conmon[33655]: cluster 2021-07-31T01:54:13.608471+0000 mon.a (mon.0) 1209 : 2021-07-31T01:54:15.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:14 smithi045 conmon[33655]: cluster [DBG] osdmap e552: 8 total, 8 up, 8 in 2021-07-31T01:54:15.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:14 smithi045 conmon[33655]: cluster 2021-07-31T01:54:13.891212+0000 mgr.x (mgr.24542) 297 : cluster [DBG] pgmap v609: 25 pgs: 25 active+clean; 251 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 4.4 MiB/s wr, 27 op/s 2021-07-31T01:54:15.621 INFO:tasks.rados.rados.0.smithi045.stdout:2634: done (2 left) 2021-07-31T01:54:15.621 INFO:tasks.rados.rados.0.smithi045.stdout:2635: done (1 left) 2021-07-31T01:54:15.621 INFO:tasks.rados.rados.0.smithi045.stdout:2636: done (0 left) 2021-07-31T01:54:15.622 INFO:tasks.rados.rados.0.smithi045.stdout:2637: rmattr oid 43 current snap is 250 2021-07-31T01:54:15.622 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 49 current snap is 250 2021-07-31T01:54:15.623 INFO:tasks.rados.rados.0.smithi045.stdout:2638: seq_num 803 ranges {0=655360,655360=745472,1400832=57344} 2021-07-31T01:54:15.624 INFO:tasks.rados.rados.0.smithi045.stdout:2638: writing smithi04569269-49 from 0 to 655360 tid 1 2021-07-31T01:54:15.628 INFO:tasks.rados.rados.0.smithi045.stdout:2638: writing smithi04569269-49 from 655360 to 1400832 tid 2 2021-07-31T01:54:15.631 INFO:tasks.rados.rados.0.smithi045.stdout:2638: writing smithi04569269-49 from 1400832 to 1458176 tid 3 2021-07-31T01:54:15.631 INFO:tasks.rados.rados.0.smithi045.stdout:2639: read oid 14 snap -1 2021-07-31T01:54:15.631 INFO:tasks.rados.rados.0.smithi045.stdout:2639: expect (ObjNum 773 snap 236 seq_num 773) 2021-07-31T01:54:15.631 INFO:tasks.rados.rados.0.smithi045.stdout:2640: rmattr oid 17 current snap is 250 2021-07-31T01:54:15.632 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 43 v 1420 (ObjNum 754 snap 233 seq_num 754) dirty exists 2021-07-31T01:54:15.632 INFO:tasks.rados.rados.0.smithi045.stdout:2637: done (3 left) 2021-07-31T01:54:15.633 INFO:tasks.rados.rados.0.smithi045.stdout:2641: read oid 3 snap 239 2021-07-31T01:54:15.633 INFO:tasks.rados.rados.0.smithi045.stdout:2641: expect (ObjNum 677 snap 211 seq_num 677) 2021-07-31T01:54:15.633 INFO:tasks.rados.rados.0.smithi045.stdout:2642: read oid 11 snap -1 2021-07-31T01:54:15.633 INFO:tasks.rados.rados.0.smithi045.stdout:2642: expect (ObjNum 785 snap 240 seq_num 785) 2021-07-31T01:54:15.634 INFO:tasks.rados.rados.0.smithi045.stdout:2643: rmattr oid 12 current snap is 250 2021-07-31T01:54:15.634 INFO:tasks.rados.rados.0.smithi045.stdout:2643: done (5 left) 2021-07-31T01:54:15.634 INFO:tasks.rados.rados.0.smithi045.stdout:2644: read oid 9 snap -1 2021-07-31T01:54:15.634 INFO:tasks.rados.rados.0.smithi045.stdout:2644: expect deleted 2021-07-31T01:54:15.634 INFO:tasks.rados.rados.0.smithi045.stdout:2645: copy_from oid 20 from oid 46 current snap is 250 2021-07-31T01:54:15.635 INFO:tasks.rados.rados.0.smithi045.stdout:2646: setattr oid 34 current snap is 250 2021-07-31T01:54:15.636 INFO:tasks.rados.rados.0.smithi045.stdout:2644: done (7 left) 2021-07-31T01:54:15.636 INFO:tasks.rados.rados.0.smithi045.stdout:2647: read oid 15 snap 249 2021-07-31T01:54:15.637 INFO:tasks.rados.rados.0.smithi045.stdout:2647: expect (ObjNum 768 snap 235 seq_num 768) 2021-07-31T01:54:15.637 INFO:tasks.rados.rados.0.smithi045.stdout:2648: setattr oid 36 current snap is 250 2021-07-31T01:54:15.637 INFO:tasks.rados.rados.0.smithi045.stdout:2638: finishing write tid 1 to smithi04569269-49 2021-07-31T01:54:15.638 INFO:tasks.rados.rados.0.smithi045.stdout:2649: setattr oid 27 current snap is 250 2021-07-31T01:54:15.640 INFO:tasks.rados.rados.0.smithi045.stdout:2639: expect (ObjNum 773 snap 236 seq_num 773) 2021-07-31T01:54:15.697 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 17 v 1626 (ObjNum 802 snap 249 seq_num 802) dirty exists 2021-07-31T01:54:15.697 INFO:tasks.rados.rados.0.smithi045.stdout:2638: finishing write tid 2 to smithi04569269-49 2021-07-31T01:54:15.697 INFO:tasks.rados.rados.0.smithi045.stdout:2638: finishing write tid 3 to smithi04569269-49 2021-07-31T01:54:15.697 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 34 v 1403 (ObjNum 765 snap 235 seq_num 765) dirty exists 2021-07-31T01:54:15.698 INFO:tasks.rados.rados.0.smithi045.stdout:2638: finishing write tid 4 to smithi04569269-49 2021-07-31T01:54:15.698 INFO:tasks.rados.rados.0.smithi045.stdout:2642: expect (ObjNum 785 snap 240 seq_num 785) 2021-07-31T01:54:15.772 INFO:tasks.rados.rados.0.smithi045.stdout:2638: finishing write tid 5 to smithi04569269-49 2021-07-31T01:54:15.773 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 49 v 1644 (ObjNum 803 snap 250 seq_num 803) dirty exists 2021-07-31T01:54:15.773 INFO:tasks.rados.rados.0.smithi045.stdout:2638: left oid 49 (ObjNum 803 snap 250 seq_num 803) 2021-07-31T01:54:15.773 INFO:tasks.rados.rados.0.smithi045.stdout:2647: expect (ObjNum 768 snap 235 seq_num 768) 2021-07-31T01:54:16.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:15 smithi154 conmon[34553]: cluster 2021-07-31T01:54:14.614425+0000 mon.a (mon.0) 1210 : cluster [DBG] osdmap e553: 8 total, 8 up, 8 in 2021-07-31T01:54:16.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:15 smithi045 conmon[39718]: cluster 2021-07-31T01:54:14.614425+0000 mon.a (mon.0) 1210 : cluster [DBG] 2021-07-31T01:54:16.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:15 smithi045 conmon[39718]: osdmap e553: 8 total, 8 up, 8 in 2021-07-31T01:54:16.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:15 smithi045 conmon[33655]: cluster 2021-07-31T01:54:14.614425+0000 mon.a (mon.0) 1210 : cluster [DBG] osdmap e553: 8 total, 8 up, 8 in 2021-07-31T01:54:16.620 INFO:tasks.rados.rados.0.smithi045.stdout:2641: expect (ObjNum 677 snap 211 seq_num 677) 2021-07-31T01:54:17.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:16 smithi154 conmon[34553]: cluster 2021-07-31T01:54:15.620062+0000 mon.a (mon.0) 1211 : cluster [DBG] osdmap e554: 8 total, 8 up, 8 in 2021-07-31T01:54:17.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:16 smithi154 conmon[34553]: cluster 2021-07-31T01:54:15 2021-07-31T01:54:17.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:16 smithi154 conmon[34553]: .891692+0000 mgr.x (mgr.24542) 298 : cluster [DBG] pgmap v612: 25 pgs: 25 active+clean; 251 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 4.4 MiB/s wr, 27 op/s 2021-07-31T01:54:17.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:16 smithi154 conmon[34553]: cluster 2021-07-31T01:54:16.601011+0000 mon.a (mon.0) 1212 : cluster [DBG] osdmap e555: 8 total, 8 up, 8 in 2021-07-31T01:54:17.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:16 smithi045 conmon[39718]: cluster 2021-07-31T01:54:15.620062 2021-07-31T01:54:17.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:16 smithi045 conmon[39718]: +0000 mon.a (mon.0) 1211 : cluster [DBG] osdmap e554: 8 total, 8 up, 8 in 2021-07-31T01:54:17.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:16 smithi045 conmon[39718]: cluster 2021-07-31T01:54:17.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:16 smithi045 conmon[39718]: 2021-07-31T01:54:15.891692+0000 mgr.x 2021-07-31T01:54:17.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:16 smithi045 conmon[39718]: (mgr.24542) 298 : cluster [DBG] pgmap v612: 25 pgs: 25 active+clean; 251 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 4.4 MiB/s wr, 27 op/s 2021-07-31T01:54:17.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:16 smithi045 conmon[39718]: cluster 2021-07-31T01:54:17.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:16 smithi045 conmon[39718]: 2021-07-31T01:54: 2021-07-31T01:54:17.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:16 smithi045 conmon[39718]: 16.601011+0000 mon.a (mon.0) 1212 : cluster 2021-07-31T01:54:17.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:16 smithi045 conmon[39718]: [DBG] osdmap e555: 8 total, 8 up, 8 in 2021-07-31T01:54:17.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:16 smithi045 conmon[33655]: cluster 2021-07-31T01:54:15.620062+0000 mon.a (mon.0) 2021-07-31T01:54:17.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:16 smithi045 conmon[33655]: 1211 : cluster [DBG] osdmap e554: 8 total, 8 up, 8 in 2021-07-31T01:54:17.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:16 smithi045 conmon[33655]: cluster 2021-07-31T01:54:15.891692+0000 mgr.x (mgr.24542) 298 : cluster [DBG] pgmap v612: 25 pgs: 25 active+clean; 251 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 4.4 MiB/s wr, 27 op/s 2021-07-31T01:54:17.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:16 smithi045 conmon[33655]: cluster 2021-07 2021-07-31T01:54:17.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:16 smithi045 conmon[33655]: -31T01:54:16.601011+0000 mon.a ( 2021-07-31T01:54:17.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:16 smithi045 conmon[33655]: mon.0) 1212 : cluster [DBG] osdmap e555: 8 total, 8 up, 8 in 2021-07-31T01:54:17.143 INFO:tasks.rados.rados.0.smithi045.stdout:2645: finishing copy_from to smithi04569269-20 2021-07-31T01:54:17.144 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 20 v 1298 (ObjNum 665 snap 208 seq_num 665) dirty exists 2021-07-31T01:54:17.144 INFO:tasks.rados.rados.0.smithi045.stdout:2645: finishing copy_from racing read to smithi04569269-20 2021-07-31T01:54:17.144 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 36 v 1647 (ObjNum -1104788000 snap 21873 seq_num 3189788928) dirty exists 2021-07-31T01:54:17.146 INFO:tasks.rados.rados.0.smithi045.stdout:2638: done (9 left) 2021-07-31T01:54:17.146 INFO:tasks.rados.rados.0.smithi045.stdout:2639: done (8 left) 2021-07-31T01:54:17.146 INFO:tasks.rados.rados.0.smithi045.stdout:2640: done (7 left) 2021-07-31T01:54:17.146 INFO:tasks.rados.rados.0.smithi045.stdout:2641: done (6 left) 2021-07-31T01:54:17.147 INFO:tasks.rados.rados.0.smithi045.stdout:2642: done (5 left) 2021-07-31T01:54:17.147 INFO:tasks.rados.rados.0.smithi045.stdout:2645: done (4 left) 2021-07-31T01:54:17.147 INFO:tasks.rados.rados.0.smithi045.stdout:2646: done (3 left) 2021-07-31T01:54:17.147 INFO:tasks.rados.rados.0.smithi045.stdout:2647: done (2 left) 2021-07-31T01:54:17.147 INFO:tasks.rados.rados.0.smithi045.stdout:2648: done (1 left) 2021-07-31T01:54:17.148 INFO:tasks.rados.rados.0.smithi045.stdout:2650: read oid 24 snap 244 2021-07-31T01:54:17.148 INFO:tasks.rados.rados.0.smithi045.stdout:2650: expect deleted 2021-07-31T01:54:17.148 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 16 current snap is 250 2021-07-31T01:54:17.148 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 27 v 1421 (ObjNum 791 snap 244 seq_num 791) dirty exists 2021-07-31T01:54:17.148 INFO:tasks.rados.rados.0.smithi045.stdout:2651: seq_num 804 ranges {1351680=671744,2023424=622592,2646016=180224} 2021-07-31T01:54:17.153 INFO:tasks.rados.rados.0.smithi045.stdout:2651: writing smithi04569269-16 from 1351680 to 2023424 tid 1 2021-07-31T01:54:17.157 INFO:tasks.rados.rados.0.smithi045.stdout:2651: writing smithi04569269-16 from 2023424 to 2646016 tid 2 2021-07-31T01:54:17.159 INFO:tasks.rados.rados.0.smithi045.stdout:2651: writing smithi04569269-16 from 2646016 to 2826240 tid 3 2021-07-31T01:54:17.159 INFO:tasks.rados.rados.0.smithi045.stdout:2649: done (2 left) 2021-07-31T01:54:17.160 INFO:tasks.rados.rados.0.smithi045.stdout:2652: read oid 35 snap -1 2021-07-31T01:54:17.160 INFO:tasks.rados.rados.0.smithi045.stdout:2652: expect (ObjNum 750 snap 231 seq_num 750) 2021-07-31T01:54:17.160 INFO:tasks.rados.rados.0.smithi045.stdout:2653: rmattr oid 12 current snap is 250 2021-07-31T01:54:17.160 INFO:tasks.rados.rados.0.smithi045.stdout:2653: done (3 left) 2021-07-31T01:54:17.160 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 49 current snap is 250 2021-07-31T01:54:17.161 INFO:tasks.rados.rados.0.smithi045.stdout:2654: seq_num 805 ranges {1458176=688128,2146304=507904,2654208=294912} 2021-07-31T01:54:17.172 INFO:tasks.rados.rados.0.smithi045.stdout:2654: writing smithi04569269-49 from 1458176 to 2146304 tid 1 2021-07-31T01:54:17.176 INFO:tasks.rados.rados.0.smithi045.stdout:2654: writing smithi04569269-49 from 2146304 to 2654208 tid 2 2021-07-31T01:54:17.176 INFO:tasks.rados.rados.0.smithi045.stdout:2654: writing smithi04569269-49 from 2654208 to 2949120 tid 3 2021-07-31T01:54:17.176 INFO:tasks.rados.rados.0.smithi045.stdout:2655: snap_remove snap 241 2021-07-31T01:54:17.631 INFO:tasks.rados.rados.0.smithi045.stdout:2651: finishing write tid 1 to smithi04569269-16 2021-07-31T01:54:17.631 INFO:tasks.rados.rados.0.smithi045.stdout:2652: expect (ObjNum 750 snap 231 seq_num 750) 2021-07-31T01:54:17.653 INFO:tasks.rados.rados.0.smithi045.stdout:2651: finishing write tid 2 to smithi04569269-16 2021-07-31T01:54:17.654 INFO:tasks.rados.rados.0.smithi045.stdout:2651: finishing write tid 3 to smithi04569269-16 2021-07-31T01:54:17.654 INFO:tasks.rados.rados.0.smithi045.stdout:2651: finishing write tid 4 to smithi04569269-16 2021-07-31T01:54:17.654 INFO:tasks.rados.rados.0.smithi045.stdout:2651: finishing write tid 5 to smithi04569269-16 2021-07-31T01:54:17.654 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 16 v 1572 (ObjNum 804 snap 250 seq_num 804) dirty exists 2021-07-31T01:54:17.655 INFO:tasks.rados.rados.0.smithi045.stdout:2651: left oid 16 (ObjNum 804 snap 250 seq_num 804) 2021-07-31T01:54:17.655 INFO:tasks.rados.rados.0.smithi045.stdout:2654: finishing write tid 1 to smithi04569269-49 2021-07-31T01:54:17.655 INFO:tasks.rados.rados.0.smithi045.stdout:2654: finishing write tid 2 to smithi04569269-49 2021-07-31T01:54:17.655 INFO:tasks.rados.rados.0.smithi045.stdout:2654: finishing write tid 3 to smithi04569269-49 2021-07-31T01:54:17.656 INFO:tasks.rados.rados.0.smithi045.stdout:2654: finishing write tid 4 to smithi04569269-49 2021-07-31T01:54:17.656 INFO:tasks.rados.rados.0.smithi045.stdout:2654: finishing write tid 5 to smithi04569269-49 2021-07-31T01:54:17.656 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 49 v 1651 (ObjNum 805 snap 250 seq_num 805) dirty exists 2021-07-31T01:54:17.656 INFO:tasks.rados.rados.0.smithi045.stdout:2654: left oid 49 (ObjNum 805 snap 250 seq_num 805) 2021-07-31T01:54:17.656 INFO:tasks.rados.rados.0.smithi045.stdout:2650: done (4 left) 2021-07-31T01:54:17.657 INFO:tasks.rados.rados.0.smithi045.stdout:2651: done (3 left) 2021-07-31T01:54:17.657 INFO:tasks.rados.rados.0.smithi045.stdout:2652: done (2 left) 2021-07-31T01:54:17.658 INFO:tasks.rados.rados.0.smithi045.stdout:2654: done (1 left) 2021-07-31T01:54:17.658 INFO:tasks.rados.rados.0.smithi045.stdout:2655: done (0 left) 2021-07-31T01:54:17.658 INFO:tasks.rados.rados.0.smithi045.stdout:2656: read oid 12 snap -1 2021-07-31T01:54:17.658 INFO:tasks.rados.rados.0.smithi045.stdout:2656: expect deleted 2021-07-31T01:54:17.659 INFO:tasks.rados.rados.0.smithi045.stdout:2657: copy_from oid 24 from oid 35 current snap is 250 2021-07-31T01:54:17.659 INFO:tasks.rados.rados.0.smithi045.stdout:2658: delete oid 25 current snap is 250 2021-07-31T01:54:17.660 INFO:tasks.rados.rados.0.smithi045.stdout:2656: done (2 left) 2021-07-31T01:54:17.660 INFO:tasks.rados.rados.0.smithi045.stdout:2658: done (1 left) 2021-07-31T01:54:17.660 INFO:tasks.rados.rados.0.smithi045.stdout:2659: snap_create 2021-07-31T01:54:17.660 INFO:tasks.rados.rados.0.smithi045.stdout:2657: finishing copy_from to smithi04569269-24 2021-07-31T01:54:17.661 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 24 v 1593 (ObjNum 750 snap 231 seq_num 750) dirty exists 2021-07-31T01:54:17.661 INFO:tasks.rados.rados.0.smithi045.stdout:2657: finishing copy_from racing read to smithi04569269-24 2021-07-31T01:54:18.633 INFO:tasks.rados.rados.0.smithi045.stdout:2657: done (1 left) 2021-07-31T01:54:18.634 INFO:tasks.rados.rados.0.smithi045.stdout:2659: done (0 left) 2021-07-31T01:54:18.635 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 29 current snap is 251 2021-07-31T01:54:18.635 INFO:tasks.rados.rados.0.smithi045.stdout:2660: seq_num 807 ranges {10354688=720896,11075584=729088,11804672=73728} 2021-07-31T01:54:18.693 INFO:tasks.rados.rados.0.smithi045.stdout:2660: writing smithi04569269-29 from 10354688 to 11075584 tid 1 2021-07-31T01:54:18.697 INFO:tasks.rados.rados.0.smithi045.stdout:2660: writing smithi04569269-29 from 11075584 to 11804672 tid 2 2021-07-31T01:54:18.698 INFO:tasks.rados.rados.0.smithi045.stdout:2660: writing smithi04569269-29 from 11804672 to 11878400 tid 3 2021-07-31T01:54:18.698 INFO:tasks.rados.rados.0.smithi045.stdout:2661: read oid 15 snap -1 2021-07-31T01:54:18.698 INFO:tasks.rados.rados.0.smithi045.stdout:2661: expect (ObjNum 768 snap 235 seq_num 768) 2021-07-31T01:54:18.699 INFO:tasks.rados.rados.0.smithi045.stdout:2662: setattr oid 30 current snap is 251 2021-07-31T01:54:18.700 INFO:tasks.rados.rados.0.smithi045.stdout:2663: rollback oid 16 current snap is 251 2021-07-31T01:54:18.700 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 16 to 246 2021-07-31T01:54:18.700 INFO:tasks.rados.rados.0.smithi045.stdout:2664: rmattr oid 2 current snap is 251 2021-07-31T01:54:18.701 INFO:tasks.rados.rados.0.smithi045.stdout:2664: done (4 left) 2021-07-31T01:54:18.701 INFO:tasks.rados.rados.0.smithi045.stdout:2665: copy_from oid 49 from oid 10 current snap is 251 2021-07-31T01:54:18.701 INFO:tasks.rados.rados.0.smithi045.stdout:2666: snap_create 2021-07-31T01:54:18.701 INFO:tasks.rados.rados.0.smithi045.stdout:2660: finishing write tid 1 to smithi04569269-29 2021-07-31T01:54:18.704 INFO:tasks.rados.rados.0.smithi045.stdout:2660: finishing write tid 2 to smithi04569269-29 2021-07-31T01:54:18.704 INFO:tasks.rados.rados.0.smithi045.stdout:2660: finishing write tid 3 to smithi04569269-29 2021-07-31T01:54:18.705 INFO:tasks.rados.rados.0.smithi045.stdout:2660: finishing write tid 4 to smithi04569269-29 2021-07-31T01:54:18.706 INFO:tasks.rados.rados.0.smithi045.stdout:2663: finishing rollback tid 0 to smithi04569269-16 2021-07-31T01:54:18.706 INFO:tasks.rados.rados.0.smithi045.stdout:2660: finishing write tid 5 to smithi04569269-29 2021-07-31T01:54:18.707 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 29 v 1409 (ObjNum 807 snap 251 seq_num 807) dirty exists 2021-07-31T01:54:18.707 INFO:tasks.rados.rados.0.smithi045.stdout:2660: left oid 29 (ObjNum 807 snap 251 seq_num 807) 2021-07-31T01:54:18.708 INFO:tasks.rados.rados.0.smithi045.stdout:2663: finishing rollback tid 1 to smithi04569269-16 2021-07-31T01:54:18.708 INFO:tasks.rados.rados.0.smithi045.stdout:2663: finishing rollback tid 2 to smithi04569269-16 2021-07-31T01:54:18.708 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 16 v 1575 (ObjNum 684 snap 214 seq_num 684) dirty exists 2021-07-31T01:54:18.729 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 30 v 1602 (ObjNum 799 snap 246 seq_num 799) dirty exists 2021-07-31T01:54:18.736 INFO:tasks.rados.rados.0.smithi045.stdout:2661: expect (ObjNum 768 snap 235 seq_num 768) 2021-07-31T01:54:19.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:18 smithi154 conmon[34553]: cluster 2021-07-31T01:54:17.625722+0000 mon.a (mon.0) 1213 : cluster [DBG] osdmap e556: 8 total, 8 up, 8 in 2021-07-31T01:54:19.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:18 smithi154 conmon[34553]: 2021-07-31T01:54:19.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:18 smithi154 conmon[34553]: cluster 2021-07-31T01:54:17.892124+0000 mgr.x (mgr.24542) 299 : cluster [DBG] pgmap v615: 25 pgs: 25 active+clean; 251 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:54:19.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:18 smithi045 conmon[33655]: cluster 2021-07-31T01:54:17.625722+0000 mon.a (mon.0) 1213 : cluster [DBG] osdmap e556: 8 total, 8 up, 8 in 2021-07-31T01:54:19.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:18 smithi045 conmon[33655]: cluster 2021-07-31T01:54:17.892124+0000 mgr.x (mgr.24542) 299 : cluster [DBG] pgmap v615: 25 pgs: 25 active+clean; 251 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:54:19.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:18 smithi045 conmon[39718]: cluster 2021-07-31T01:54:17.625722+0000 mon.a (mon.0) 1213 : cluster 2021-07-31T01:54:19.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:18 smithi045 conmon[39718]: [DBG] osdmap e556: 8 total, 8 up, 8 in 2021-07-31T01:54:19.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:18 smithi045 conmon[39718]: cluster 2021-07-31T01:54:17.892124+0000 mgr.x (mgr.24542) 299 : cluster [DBG] pgmap v615: 25 pgs: 25 active+clean; 251 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:54:19.583 INFO:tasks.rados.rados.0.smithi045.stdout:2665: finishing copy_from to smithi04569269-49 2021-07-31T01:54:19.583 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 49 v 1576 (ObjNum 768 snap 235 seq_num 768) dirty exists 2021-07-31T01:54:19.584 INFO:tasks.rados.rados.0.smithi045.stdout:2665: finishing copy_from racing read to smithi04569269-49 2021-07-31T01:54:19.646 INFO:tasks.rados.rados.0.smithi045.stdout:2660: done (5 left) 2021-07-31T01:54:19.646 INFO:tasks.rados.rados.0.smithi045.stdout:2661: done (4 left) 2021-07-31T01:54:19.646 INFO:tasks.rados.rados.0.smithi045.stdout:2662: done (3 left) 2021-07-31T01:54:19.646 INFO:tasks.rados.rados.0.smithi045.stdout:2663: done (2 left) 2021-07-31T01:54:19.647 INFO:tasks.rados.rados.0.smithi045.stdout:2665: done (1 left) 2021-07-31T01:54:19.647 INFO:tasks.rados.rados.0.smithi045.stdout:2666: done (0 left) 2021-07-31T01:54:19.647 INFO:tasks.rados.rados.0.smithi045.stdout:2667: read oid 2 snap 248 2021-07-31T01:54:19.647 INFO:tasks.rados.rados.0.smithi045.stdout:2667: expect deleted 2021-07-31T01:54:19.648 INFO:tasks.rados.rados.0.smithi045.stdout:2668: copy_from oid 8 from oid 20 current snap is 252 2021-07-31T01:54:19.648 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 5 current snap is 252 2021-07-31T01:54:19.648 INFO:tasks.rados.rados.0.smithi045.stdout:2669: seq_num 808 ranges {9011200=737280,9748480=614400,10362880=188416} 2021-07-31T01:54:19.691 INFO:tasks.rados.rados.0.smithi045.stdout:2669: writing smithi04569269-5 from 9011200 to 9748480 tid 1 2021-07-31T01:54:19.694 INFO:tasks.rados.rados.0.smithi045.stdout:2669: writing smithi04569269-5 from 9748480 to 10362880 tid 2 2021-07-31T01:54:19.695 INFO:tasks.rados.rados.0.smithi045.stdout:2669: writing smithi04569269-5 from 10362880 to 10551296 tid 3 2021-07-31T01:54:19.696 INFO:tasks.rados.rados.0.smithi045.stdout:2670: read oid 7 snap -1 2021-07-31T01:54:19.696 INFO:tasks.rados.rados.0.smithi045.stdout:2670: expect (ObjNum 767 snap 235 seq_num 767) 2021-07-31T01:54:19.696 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 50 current snap is 252 2021-07-31T01:54:19.696 INFO:tasks.rados.rados.0.smithi045.stdout:2671: seq_num 809 ranges {3833856=753664,4587520=499712,5087232=303104} 2021-07-31T01:54:19.721 INFO:tasks.rados.rados.0.smithi045.stdout:2671: writing smithi04569269-50 from 3833856 to 4587520 tid 1 2021-07-31T01:54:19.723 INFO:tasks.rados.rados.0.smithi045.stdout:2671: writing smithi04569269-50 from 4587520 to 5087232 tid 2 2021-07-31T01:54:19.726 INFO:tasks.rados.rados.0.smithi045.stdout:2671: writing smithi04569269-50 from 5087232 to 5390336 tid 3 2021-07-31T01:54:19.726 INFO:tasks.rados.rados.0.smithi045.stdout:2672: read oid 41 snap -1 2021-07-31T01:54:19.726 INFO:tasks.rados.rados.0.smithi045.stdout:2672: expect (ObjNum 701 snap 218 seq_num 701) 2021-07-31T01:54:19.727 INFO:tasks.rados.rados.0.smithi045.stdout:2673: rollback oid 15 current snap is 252 2021-07-31T01:54:19.727 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 15 to 244 2021-07-31T01:54:19.727 INFO:tasks.rados.rados.0.smithi045.stdout:2668: finishing copy_from to smithi04569269-8 2021-07-31T01:54:19.728 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 8 v 1298 (ObjNum 665 snap 208 seq_num 665) dirty exists 2021-07-31T01:54:19.728 INFO:tasks.rados.rados.0.smithi045.stdout:2668: finishing copy_from racing read to smithi04569269-8 2021-07-31T01:54:19.729 INFO:tasks.rados.rados.0.smithi045.stdout:2669: finishing write tid 1 to smithi04569269-5 2021-07-31T01:54:19.729 INFO:tasks.rados.rados.0.smithi045.stdout:2669: finishing write tid 2 to smithi04569269-5 2021-07-31T01:54:19.729 INFO:tasks.rados.rados.0.smithi045.stdout:2667: done (6 left) 2021-07-31T01:54:19.729 INFO:tasks.rados.rados.0.smithi045.stdout:2668: done (5 left) 2021-07-31T01:54:19.730 INFO:tasks.rados.rados.0.smithi045.stdout:2674: rmattr oid 22 current snap is 252 2021-07-31T01:54:19.730 INFO:tasks.rados.rados.0.smithi045.stdout:2675: snap_create 2021-07-31T01:54:19.730 INFO:tasks.rados.rados.0.smithi045.stdout:2669: finishing write tid 3 to smithi04569269-5 2021-07-31T01:54:19.730 INFO:tasks.rados.rados.0.smithi045.stdout:2669: finishing write tid 4 to smithi04569269-5 2021-07-31T01:54:19.730 INFO:tasks.rados.rados.0.smithi045.stdout:2669: finishing write tid 5 to smithi04569269-5 2021-07-31T01:54:19.731 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 5 v 1613 (ObjNum 808 snap 252 seq_num 808) dirty exists 2021-07-31T01:54:19.731 INFO:tasks.rados.rados.0.smithi045.stdout:2669: left oid 5 (ObjNum 808 snap 252 seq_num 808) 2021-07-31T01:54:19.732 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 22 v 1576 (ObjNum 747 snap 230 seq_num 747) dirty exists 2021-07-31T01:54:19.732 INFO:tasks.rados.rados.0.smithi045.stdout:2673: finishing rollback tid 0 to smithi04569269-15 2021-07-31T01:54:19.732 INFO:tasks.rados.rados.0.smithi045.stdout:2673: finishing rollback tid 1 to smithi04569269-15 2021-07-31T01:54:19.733 INFO:tasks.rados.rados.0.smithi045.stdout:2673: finishing rollback tid 2 to smithi04569269-15 2021-07-31T01:54:19.733 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 15 v 1616 (ObjNum 768 snap 235 seq_num 768) dirty exists 2021-07-31T01:54:19.734 INFO:tasks.rados.rados.0.smithi045.stdout:2670: expect (ObjNum 767 snap 235 seq_num 767) 2021-07-31T01:54:20.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:19 smithi154 conmon[34553]: cluster 2021-07-31T01:54:18.631612+0000 mon.a (mon.0) 1214 : cluster [DBG] 2021-07-31T01:54:20.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:19 smithi154 conmon[34553]: osdmap e557: 8 total, 8 up, 8 in 2021-07-31T01:54:20.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:19 smithi045 conmon[33655]: cluster 2021-07-31T01:54:18.631612+0000 mon.a ( 2021-07-31T01:54:20.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:19 smithi045 conmon[33655]: mon.0) 1214 : cluster [DBG] osdmap e557: 8 total, 8 up, 8 in 2021-07-31T01:54:20.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:19 smithi045 conmon[39718]: cluster 2021-07-31T01 2021-07-31T01:54:20.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:19 smithi045 conmon[39718]: :54:18.631612+0000 mon.a (mon.0) 1214 : cluster [DBG] osdmap e557: 8 total, 8 up, 8 in 2021-07-31T01:54:20.221 INFO:tasks.rados.rados.0.smithi045.stdout:2671: finishing write tid 1 to smithi04569269-50 2021-07-31T01:54:20.222 INFO:tasks.rados.rados.0.smithi045.stdout:2671: finishing write tid 2 to smithi04569269-50 2021-07-31T01:54:20.222 INFO:tasks.rados.rados.0.smithi045.stdout:2671: finishing write tid 3 to smithi04569269-50 2021-07-31T01:54:20.223 INFO:tasks.rados.rados.0.smithi045.stdout:2671: finishing write tid 4 to smithi04569269-50 2021-07-31T01:54:20.223 INFO:tasks.rados.rados.0.smithi045.stdout:2671: finishing write tid 5 to smithi04569269-50 2021-07-31T01:54:20.224 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 50 v 1630 (ObjNum 809 snap 252 seq_num 809) dirty exists 2021-07-31T01:54:20.224 INFO:tasks.rados.rados.0.smithi045.stdout:2671: left oid 50 (ObjNum 809 snap 252 seq_num 809) 2021-07-31T01:54:20.224 INFO:tasks.rados.rados.0.smithi045.stdout:2672: expect (ObjNum 701 snap 218 seq_num 701) 2021-07-31T01:54:20.643 INFO:tasks.rados.rados.0.smithi045.stdout:2669: done (6 left) 2021-07-31T01:54:20.643 INFO:tasks.rados.rados.0.smithi045.stdout:2670: done (5 left) 2021-07-31T01:54:20.643 INFO:tasks.rados.rados.0.smithi045.stdout:2671: done (4 left) 2021-07-31T01:54:20.644 INFO:tasks.rados.rados.0.smithi045.stdout:2672: done (3 left) 2021-07-31T01:54:20.644 INFO:tasks.rados.rados.0.smithi045.stdout:2673: done (2 left) 2021-07-31T01:54:20.644 INFO:tasks.rados.rados.0.smithi045.stdout:2674: done (1 left) 2021-07-31T01:54:20.644 INFO:tasks.rados.rados.0.smithi045.stdout:2675: done (0 left) 2021-07-31T01:54:20.644 INFO:tasks.rados.rados.0.smithi045.stdout:2676: snap_create 2021-07-31T01:54:21.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:20 smithi154 conmon[34553]: cluster 2021-07-31T01:54:19.637028+0000 mon.a (mon.0) 1215 : cluster [DBG] 2021-07-31T01:54:21.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:20 smithi154 conmon[34553]: osdmap e558: 8 total, 8 up, 8 in 2021-07-31T01:54:21.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:20 smithi154 conmon[34553]: cluster 2021-07-31T01:54:19.892478+0000 mgr.x (mgr.24542) 300 : cluster [DBG] pgmap v618: 25 pgs: 25 active+clean; 277 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 23 MiB/s rd, 5.7 MiB/s wr, 38 op/s 2021-07-31T01:54:21.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:20 smithi045 conmon[39718]: cluster 2021-07-31T01:54:19.637028+0000 2021-07-31T01:54:21.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:20 smithi045 conmon[39718]: mon.a (mon.0) 1215 : cluster [DBG] osdmap e558: 8 total, 8 up, 8 in 2021-07-31T01:54:21.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:20 smithi045 conmon[39718]: cluster 2021-07-31T01:54:21.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:20 smithi045 conmon[39718]: 2021-07-31T01:54:19.892478+0000 mgr.x (mgr. 2021-07-31T01:54:21.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:20 smithi045 conmon[39718]: 24542) 300 : cluster [DBG] pgmap v618: 25 pgs: 25 active+clean; 277 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 23 MiB/s rd, 5.7 MiB/s wr, 38 op/s 2021-07-31T01:54:21.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:20 smithi045 conmon[33655]: cluster 2021-07-31T01:54:19.637028+0000 mon.a (mon.0 2021-07-31T01:54:21.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:20 smithi045 conmon[33655]: ) 1215 : cluster [DBG] osdmap e558: 8 total, 8 up, 8 in 2021-07-31T01:54:21.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:20 smithi045 conmon[33655]: cluster 2021-07-31T01:54:19.892478+0000 2021-07-31T01:54:21.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:20 smithi045 conmon[33655]: mgr.x (mgr.24542) 300 : cluster [DBG] pgmap v618: 25 pgs: 25 active+clean; 277 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 23 MiB/s rd, 5.7 MiB/s wr, 38 op/s 2021-07-31T01:54:21.604 INFO:tasks.rados.rados.0.smithi045.stdout:2676: done (0 left) 2021-07-31T01:54:21.604 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 15 current snap is 254 2021-07-31T01:54:21.605 INFO:tasks.rados.rados.0.smithi045.stdout:2677: seq_num 810 ranges {10043392=770048,10813440=778240,11591680=24576} 2021-07-31T01:54:21.663 INFO:tasks.rados.rados.0.smithi045.stdout:2677: writing smithi04569269-15 from 10043392 to 10813440 tid 1 2021-07-31T01:54:21.671 INFO:tasks.rados.rados.0.smithi045.stdout:2677: writing smithi04569269-15 from 10813440 to 11591680 tid 2 2021-07-31T01:54:21.671 INFO:tasks.rados.rados.0.smithi045.stdout:2677: writing smithi04569269-15 from 11591680 to 11616256 tid 3 2021-07-31T01:54:21.671 INFO:tasks.rados.rados.0.smithi045.stdout:2678: rollback oid 3 current snap is 254 2021-07-31T01:54:21.672 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 3 to 251 2021-07-31T01:54:21.672 INFO:tasks.rados.rados.0.smithi045.stdout:2677: finishing write tid 1 to smithi04569269-15 2021-07-31T01:54:21.672 INFO:tasks.rados.rados.0.smithi045.stdout:2679: read oid 50 snap 242 2021-07-31T01:54:21.672 INFO:tasks.rados.rados.0.smithi045.stdout:2679: expect (ObjNum 701 snap 218 seq_num 701) 2021-07-31T01:54:21.673 INFO:tasks.rados.rados.0.smithi045.stdout:2680: read oid 24 snap 253 2021-07-31T01:54:21.673 INFO:tasks.rados.rados.0.smithi045.stdout:2680: expect (ObjNum 750 snap 231 seq_num 750) 2021-07-31T01:54:21.673 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 1 current snap is 254 2021-07-31T01:54:21.673 INFO:tasks.rados.rados.0.smithi045.stdout:2681: seq_num 811 ranges {3792896=786432,4579328=663552,5242880=139264} 2021-07-31T01:54:21.694 INFO:tasks.rados.rados.0.smithi045.stdout:2681: writing smithi04569269-1 from 3792896 to 4579328 tid 1 2021-07-31T01:54:21.697 INFO:tasks.rados.rados.0.smithi045.stdout:2681: writing smithi04569269-1 from 4579328 to 5242880 tid 2 2021-07-31T01:54:21.698 INFO:tasks.rados.rados.0.smithi045.stdout:2681: writing smithi04569269-1 from 5242880 to 5382144 tid 3 2021-07-31T01:54:21.699 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 11 current snap is 254 2021-07-31T01:54:21.700 INFO:tasks.rados.rados.0.smithi045.stdout:2677: finishing write tid 2 to smithi04569269-15 2021-07-31T01:54:21.700 INFO:tasks.rados.rados.0.smithi045.stdout:2682: seq_num 812 ranges {2260992=802816,3063808=548864,3612672=253952} 2021-07-31T01:54:21.715 INFO:tasks.rados.rados.0.smithi045.stdout:2682: writing smithi04569269-11 from 2260992 to 3063808 tid 1 2021-07-31T01:54:21.718 INFO:tasks.rados.rados.0.smithi045.stdout:2682: writing smithi04569269-11 from 3063808 to 3612672 tid 2 2021-07-31T01:54:21.721 INFO:tasks.rados.rados.0.smithi045.stdout:2682: writing smithi04569269-11 from 3612672 to 3866624 tid 3 2021-07-31T01:54:21.721 INFO:tasks.rados.rados.0.smithi045.stdout:2683: read oid 46 snap 251 2021-07-31T01:54:21.721 INFO:tasks.rados.rados.0.smithi045.stdout:2683: expect (ObjNum 665 snap 208 seq_num 665) 2021-07-31T01:54:21.722 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 31 current snap is 254 2021-07-31T01:54:21.722 INFO:tasks.rados.rados.0.smithi045.stdout:2677: finishing write tid 3 to smithi04569269-15 2021-07-31T01:54:21.722 INFO:tasks.rados.rados.0.smithi045.stdout:2677: finishing write tid 4 to smithi04569269-15 2021-07-31T01:54:21.722 INFO:tasks.rados.rados.0.smithi045.stdout:2677: finishing write tid 5 to smithi04569269-15 2021-07-31T01:54:21.725 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 15 v 1620 (ObjNum 810 snap 254 seq_num 810) dirty exists 2021-07-31T01:54:21.725 INFO:tasks.rados.rados.0.smithi045.stdout:2677: left oid 15 (ObjNum 810 snap 254 seq_num 810) 2021-07-31T01:54:21.726 INFO:tasks.rados.rados.0.smithi045.stdout:2678: finishing rollback tid 0 to smithi04569269-3 2021-07-31T01:54:21.726 INFO:tasks.rados.rados.0.smithi045.stdout:2680: expect (ObjNum 750 snap 231 seq_num 750) 2021-07-31T01:54:21.743 INFO:tasks.rados.rados.0.smithi045.stdout:2678: finishing rollback tid 1 to smithi04569269-3 2021-07-31T01:54:21.743 INFO:tasks.rados.rados.0.smithi045.stdout:2678: finishing rollback tid 2 to smithi04569269-3 2021-07-31T01:54:21.743 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 3 v 1663 (ObjNum 677 snap 211 seq_num 677) dirty exists 2021-07-31T01:54:21.743 INFO:tasks.rados.rados.0.smithi045.stdout:2684: seq_num 813 ranges {163840=417792,581632=434176,1015808=442368,1458176=327680} 2021-07-31T01:54:21.745 INFO:tasks.rados.rados.0.smithi045.stdout:2684: writing smithi04569269-31 from 163840 to 581632 tid 1 2021-07-31T01:54:21.748 INFO:tasks.rados.rados.0.smithi045.stdout:2684: writing smithi04569269-31 from 581632 to 1015808 tid 2 2021-07-31T01:54:21.750 INFO:tasks.rados.rados.0.smithi045.stdout:2684: writing smithi04569269-31 from 1015808 to 1458176 tid 3 2021-07-31T01:54:21.752 INFO:tasks.rados.rados.0.smithi045.stdout:2684: writing smithi04569269-31 from 1458176 to 1785856 tid 4 2021-07-31T01:54:21.753 INFO:tasks.rados.rados.0.smithi045.stdout:2677: done (7 left) 2021-07-31T01:54:21.753 INFO:tasks.rados.rados.0.smithi045.stdout:2678: done (6 left) 2021-07-31T01:54:21.753 INFO:tasks.rados.rados.0.smithi045.stdout:2680: done (5 left) 2021-07-31T01:54:21.753 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 35 current snap is 254 2021-07-31T01:54:21.754 INFO:tasks.rados.rados.0.smithi045.stdout:2685: seq_num 814 ranges {565248=442368,1007616=770048,1777664=434176} 2021-07-31T01:54:21.758 INFO:tasks.rados.rados.0.smithi045.stdout:2685: writing smithi04569269-35 from 565248 to 1007616 tid 1 2021-07-31T01:54:21.762 INFO:tasks.rados.rados.0.smithi045.stdout:2685: writing smithi04569269-35 from 1007616 to 1777664 tid 2 2021-07-31T01:54:21.765 INFO:tasks.rados.rados.0.smithi045.stdout:2685: writing smithi04569269-35 from 1777664 to 2211840 tid 3 2021-07-31T01:54:21.765 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 6 current snap is 254 2021-07-31T01:54:21.765 INFO:tasks.rados.rados.0.smithi045.stdout:2679: expect (ObjNum 701 snap 218 seq_num 701) 2021-07-31T01:54:21.894 INFO:tasks.rados.rados.0.smithi045.stdout:2681: finishing write tid 1 to smithi04569269-1 2021-07-31T01:54:21.895 INFO:tasks.rados.rados.0.smithi045.stdout:2681: finishing write tid 2 to smithi04569269-1 2021-07-31T01:54:21.895 INFO:tasks.rados.rados.0.smithi045.stdout:2681: finishing write tid 3 to smithi04569269-1 2021-07-31T01:54:21.895 INFO:tasks.rados.rados.0.smithi045.stdout:2681: finishing write tid 4 to smithi04569269-1 2021-07-31T01:54:21.895 INFO:tasks.rados.rados.0.smithi045.stdout:2681: finishing write tid 5 to smithi04569269-1 2021-07-31T01:54:21.896 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 1 v 1407 (ObjNum 811 snap 254 seq_num 811) dirty exists 2021-07-31T01:54:21.896 INFO:tasks.rados.rados.0.smithi045.stdout:2681: left oid 1 (ObjNum 811 snap 254 seq_num 811) 2021-07-31T01:54:21.896 INFO:tasks.rados.rados.0.smithi045.stdout:2682: finishing write tid 1 to smithi04569269-11 2021-07-31T01:54:21.896 INFO:tasks.rados.rados.0.smithi045.stdout:2682: finishing write tid 2 to smithi04569269-11 2021-07-31T01:54:21.897 INFO:tasks.rados.rados.0.smithi045.stdout:2682: finishing write tid 3 to smithi04569269-11 2021-07-31T01:54:21.897 INFO:tasks.rados.rados.0.smithi045.stdout:2682: finishing write tid 4 to smithi04569269-11 2021-07-31T01:54:21.897 INFO:tasks.rados.rados.0.smithi045.stdout:2682: finishing write tid 5 to smithi04569269-11 2021-07-31T01:54:21.897 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 11 v 1617 (ObjNum 812 snap 254 seq_num 812) dirty exists 2021-07-31T01:54:21.897 INFO:tasks.rados.rados.0.smithi045.stdout:2682: left oid 11 (ObjNum 812 snap 254 seq_num 812) 2021-07-31T01:54:21.898 INFO:tasks.rados.rados.0.smithi045.stdout:2683: expect (ObjNum 665 snap 208 seq_num 665) 2021-07-31T01:54:22.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:21 smithi154 conmon[34553]: cluster 2021-07-31T01:54:20.642001+0000 mon.a (mon.0) 1216 : cluster [DBG] osdmap e559: 8 total, 8 up, 8 in 2021-07-31T01:54:22.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:21 smithi154 conmon[34553]: cluster 2021-07-31T01:54:21.603498+0000 mon.a (mon.0) 1217 : cluster [DBG] osdmap e560: 8 total, 8 up, 8 in 2021-07-31T01:54:22.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:21 smithi045 conmon[39718]: cluster 2021-07-31T01:54 2021-07-31T01:54:22.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:21 smithi045 conmon[39718]: :20.642001+0000 mon.a (mon.0) 1216 : cluster [DBG] osdmap e559: 8 total, 8 up, 8 in 2021-07-31T01:54:22.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:21 smithi045 conmon[39718]: cluster 2021- 2021-07-31T01:54:22.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:21 smithi045 conmon[39718]: 07-31T01:54:21.603498 2021-07-31T01:54:22.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:21 smithi045 conmon[39718]: +0000 mon.a (mon.0) 1217 2021-07-31T01:54:22.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:21 smithi045 conmon[39718]: : cluster [DBG] osdmap e560: 8 total, 8 up, 8 in 2021-07-31T01:54:22.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:21 smithi045 conmon[33655]: cluster 2021-07-31T01:54:20.642001+0000 mon.a (mon. 2021-07-31T01:54:22.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:21 smithi045 conmon[33655]: 0) 1216 : cluster [DBG] osdmap e559: 8 total, 8 up, 8 in 2021-07-31T01:54:22.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:21 smithi045 conmon[33655]: cluster 2021-07-31T01:54:21.603498+0000 mon.a (mon.0) 1217 : cluster [DBG] osdmap e560: 8 total, 8 up, 8 in 2021-07-31T01:54:22.113 INFO:tasks.rados.rados.0.smithi045.stdout:2684: finishing write tid 1 to smithi04569269-31 2021-07-31T01:54:22.114 INFO:tasks.rados.rados.0.smithi045.stdout:2684: finishing write tid 2 to smithi04569269-31 2021-07-31T01:54:22.114 INFO:tasks.rados.rados.0.smithi045.stdout:2684: finishing write tid 3 to smithi04569269-31 2021-07-31T01:54:22.114 INFO:tasks.rados.rados.0.smithi045.stdout:2684: finishing write tid 4 to smithi04569269-31 2021-07-31T01:54:22.115 INFO:tasks.rados.rados.0.smithi045.stdout:2684: finishing write tid 5 to smithi04569269-31 2021-07-31T01:54:22.115 INFO:tasks.rados.rados.0.smithi045.stdout:2684: finishing write tid 6 to smithi04569269-31 2021-07-31T01:54:22.115 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 31 v 1669 (ObjNum 813 snap 254 seq_num 813) dirty exists 2021-07-31T01:54:22.115 INFO:tasks.rados.rados.0.smithi045.stdout:2684: left oid 31 (ObjNum 813 snap 254 seq_num 813) 2021-07-31T01:54:22.116 INFO:tasks.rados.rados.0.smithi045.stdout:2685: finishing write tid 1 to smithi04569269-35 2021-07-31T01:54:22.116 INFO:tasks.rados.rados.0.smithi045.stdout:2685: finishing write tid 2 to smithi04569269-35 2021-07-31T01:54:22.116 INFO:tasks.rados.rados.0.smithi045.stdout:2685: finishing write tid 3 to smithi04569269-35 2021-07-31T01:54:22.116 INFO:tasks.rados.rados.0.smithi045.stdout:2685: finishing write tid 4 to smithi04569269-35 2021-07-31T01:54:22.117 INFO:tasks.rados.rados.0.smithi045.stdout:2686: seq_num 815 ranges {0=458752,458752=655360,1114112=409600,1523712=139264} 2021-07-31T01:54:22.117 INFO:tasks.rados.rados.0.smithi045.stdout:2686: writing smithi04569269-6 from 0 to 458752 tid 1 2021-07-31T01:54:22.118 INFO:tasks.rados.rados.0.smithi045.stdout:2686: writing smithi04569269-6 from 458752 to 1114112 tid 2 2021-07-31T01:54:22.120 INFO:tasks.rados.rados.0.smithi045.stdout:2686: writing smithi04569269-6 from 1114112 to 1523712 tid 3 2021-07-31T01:54:22.122 INFO:tasks.rados.rados.0.smithi045.stdout:2686: writing smithi04569269-6 from 1523712 to 1662976 tid 4 2021-07-31T01:54:22.122 INFO:tasks.rados.rados.0.smithi045.stdout:2679: done (6 left) 2021-07-31T01:54:22.122 INFO:tasks.rados.rados.0.smithi045.stdout:2681: done (5 left) 2021-07-31T01:54:22.123 INFO:tasks.rados.rados.0.smithi045.stdout:2682: done (4 left) 2021-07-31T01:54:22.123 INFO:tasks.rados.rados.0.smithi045.stdout:2683: done (3 left) 2021-07-31T01:54:22.123 INFO:tasks.rados.rados.0.smithi045.stdout:2684: done (2 left) 2021-07-31T01:54:22.123 INFO:tasks.rados.rados.0.smithi045.stdout:2687: snap_create 2021-07-31T01:54:22.124 INFO:tasks.rados.rados.0.smithi045.stdout:2685: finishing write tid 5 to smithi04569269-35 2021-07-31T01:54:22.124 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 35 v 1674 (ObjNum 814 snap 254 seq_num 814) dirty exists 2021-07-31T01:54:22.124 INFO:tasks.rados.rados.0.smithi045.stdout:2685: left oid 35 (ObjNum 814 snap 254 seq_num 814) 2021-07-31T01:54:22.124 INFO:tasks.rados.rados.0.smithi045.stdout:2686: finishing write tid 1 to smithi04569269-6 2021-07-31T01:54:22.125 INFO:tasks.rados.rados.0.smithi045.stdout:2686: finishing write tid 2 to smithi04569269-6 2021-07-31T01:54:22.126 INFO:tasks.rados.rados.0.smithi045.stdout:2686: finishing write tid 3 to smithi04569269-6 2021-07-31T01:54:22.126 INFO:tasks.rados.rados.0.smithi045.stdout:2686: finishing write tid 4 to smithi04569269-6 2021-07-31T01:54:22.126 INFO:tasks.rados.rados.0.smithi045.stdout:2686: finishing write tid 5 to smithi04569269-6 2021-07-31T01:54:22.127 INFO:tasks.rados.rados.0.smithi045.stdout:2686: finishing write tid 6 to smithi04569269-6 2021-07-31T01:54:22.127 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 6 v 1608 (ObjNum 815 snap 254 seq_num 815) dirty exists 2021-07-31T01:54:22.127 INFO:tasks.rados.rados.0.smithi045.stdout:2686: left oid 6 (ObjNum 815 snap 254 seq_num 815) 2021-07-31T01:54:22.650 INFO:tasks.rados.rados.0.smithi045.stdout:2685: done (2 left) 2021-07-31T01:54:22.651 INFO:tasks.rados.rados.0.smithi045.stdout:2686: done (1 left) 2021-07-31T01:54:22.651 INFO:tasks.rados.rados.0.smithi045.stdout:2687: done (0 left) 2021-07-31T01:54:22.651 INFO:tasks.rados.rados.0.smithi045.stdout:2688: rmattr oid 44 current snap is 255 2021-07-31T01:54:22.652 INFO:tasks.rados.rados.0.smithi045.stdout:2689: rmattr oid 36 current snap is 255 2021-07-31T01:54:22.652 INFO:tasks.rados.rados.0.smithi045.stdout:2690: read oid 23 snap -1 2021-07-31T01:54:22.652 INFO:tasks.rados.rados.0.smithi045.stdout:2690: expect deleted 2021-07-31T01:54:22.652 INFO:tasks.rados.rados.0.smithi045.stdout:2691: rmattr oid 5 current snap is 255 2021-07-31T01:54:22.653 INFO:tasks.rados.rados.0.smithi045.stdout:2692: snap_create 2021-07-31T01:54:22.655 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 36 v 1676 (ObjNum -1470214144 snap 32731 seq_num 3187445776) dirty exists 2021-07-31T01:54:22.658 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 44 v 1422 (ObjNum 776 snap 236 seq_num 776) dirty exists 2021-07-31T01:54:22.659 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 5 v 1618 (ObjNum 808 snap 252 seq_num 808) dirty exists 2021-07-31T01:54:23.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:22 smithi154 conmon[34553]: cluster 2021-07-31T01:54:21.892964+0000 mgr.x (mgr.24542) 301 : cluster [DBG] pgmap v621: 25 pgs: 25 active+clean; 277 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 23 MiB/s rd, 5.7 MiB/s wr, 38 op/s 2021-07-31T01:54:23.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:22 smithi045 conmon[33655]: cluster 2021-07-31T01:54:21.892964+0000 mgr.x (mgr.24542) 301 : cluster 2021-07-31T01:54:23.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:22 smithi045 conmon[33655]: [DBG] pgmap v621: 25 pgs: 25 active+clean; 277 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 23 MiB/s rd, 5.7 MiB/s wr, 38 op/s 2021-07-31T01:54:23.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:22 smithi045 conmon[39718]: cluster 2021-07-31T01:54:21.892964+0000 mgr.x ( 2021-07-31T01:54:23.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:22 smithi045 conmon[39718]: mgr.24542) 301 : cluster [DBG] pgmap v621: 25 pgs: 25 active+clean; 277 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 23 MiB/s rd, 5.7 MiB/s wr, 38 op/s 2021-07-31T01:54:23.652 INFO:tasks.rados.rados.0.smithi045.stdout:2688: done (4 left) 2021-07-31T01:54:23.652 INFO:tasks.rados.rados.0.smithi045.stdout:2689: done (3 left) 2021-07-31T01:54:23.652 INFO:tasks.rados.rados.0.smithi045.stdout:2690: done (2 left) 2021-07-31T01:54:23.653 INFO:tasks.rados.rados.0.smithi045.stdout:2691: done (1 left) 2021-07-31T01:54:23.653 INFO:tasks.rados.rados.0.smithi045.stdout:2692: done (0 left) 2021-07-31T01:54:23.653 INFO:tasks.rados.rados.0.smithi045.stdout:2693: snap_remove snap 254 2021-07-31T01:54:23.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:23 smithi045 conmon[33655]: cluster 2021-07-31T01:54:22.649606+0000 mon.a ( 2021-07-31T01:54:23.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:23 smithi045 conmon[33655]: mon.0) 1218 : cluster [DBG] osdmap e561: 8 total, 8 up, 8 in 2021-07-31T01:54:23.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:23 smithi045 conmon[39718]: cluster 2021-07-31T01:54:22.649606+0000 mon.a ( 2021-07-31T01:54:23.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:23 smithi045 conmon[39718]: mon.0) 1218 : cluster [DBG] osdmap e561: 8 total, 8 up, 8 in 2021-07-31T01:54:23.778 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:54:23 smithi045 conmon[61210]: level=error ts=2021-07-31T01:54:23.521Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:54:23.779 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:54:23 smithi045 conmon[61210]: level=error ts=2021-07-31T01:54:23.521Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:54:23.779 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:54:23 smithi045 conmon[61210]: level=error ts=2021-07-31T01:54:23.521Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T01:54:24.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:23 smithi154 conmon[34553]: cluster 2021-07-31T01:54:22.649606+0000 mon.a (mon.0) 1218 : cluster [DBG] 2021-07-31T01:54:24.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:23 smithi154 conmon[34553]: osdmap e561: 8 total, 8 up, 8 in 2021-07-31T01:54:24.665 INFO:tasks.rados.rados.0.smithi045.stdout:2693: done (0 left) 2021-07-31T01:54:24.665 INFO:tasks.rados.rados.0.smithi045.stdout:2694: read oid 1 snap -1 2021-07-31T01:54:24.665 INFO:tasks.rados.rados.0.smithi045.stdout:2694: expect (ObjNum 811 snap 254 seq_num 811) 2021-07-31T01:54:24.666 INFO:tasks.rados.rados.0.smithi045.stdout:2695: copy_from oid 12 from oid 10 current snap is 256 2021-07-31T01:54:24.666 INFO:tasks.rados.rados.0.smithi045.stdout:2696: read oid 45 snap -1 2021-07-31T01:54:24.666 INFO:tasks.rados.rados.0.smithi045.stdout:2696: expect (ObjNum 801 snap 249 seq_num 801) 2021-07-31T01:54:24.666 INFO:tasks.rados.rados.0.smithi045.stdout:2697: rmattr oid 23 current snap is 256 2021-07-31T01:54:24.667 INFO:tasks.rados.rados.0.smithi045.stdout:2697: done (3 left) 2021-07-31T01:54:24.667 INFO:tasks.rados.rados.0.smithi045.stdout:2698: snap_remove snap 239 2021-07-31T01:54:25.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:24 smithi154 conmon[34553]: cluster 2021-07-31T01:54:23.651212+0000 mon.a (mon.0) 1219 : cluster 2021-07-31T01:54:25.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:24 smithi154 conmon[34553]: [DBG] osdmap e562: 8 total, 8 up, 8 in 2021-07-31T01:54:25.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:24 smithi154 conmon[34553]: cluster 2021-07-31T01:54:23.893598+0000 mgr.x (mgr.24542) 302 : cluster [DBG] pgmap v624: 25 pgs: 25 active+clean; 293 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 15 MiB/s rd, 4.1 MiB/s wr, 24 op/s 2021-07-31T01:54:25.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:24 smithi045 conmon[33655]: cluster 2021-07-31T01:54:23.651212+0000 2021-07-31T01:54:25.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:24 smithi045 conmon[33655]: mon.a (mon.0) 1219 : cluster [DBG] osdmap e562: 8 total, 8 up, 8 in 2021-07-31T01:54:25.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:24 smithi045 conmon[33655]: cluster 2021-07-31T01:54:23.893598 2021-07-31T01:54:25.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:24 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 302 : cluster [DBG] pgmap v624: 25 pgs: 25 active+clean; 293 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 15 MiB/s rd, 4.1 MiB/s wr, 24 op/s 2021-07-31T01:54:25.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:24 smithi045 conmon[33655]: 2021-07-31T01:54:25.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:24 smithi045 conmon[39718]: cluster 2021-07-31T01:54:23 2021-07-31T01:54:25.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:24 smithi045 conmon[39718]: .651212+0000 mon.a (mon.0) 1219 : cluster [DBG] osdmap e562: 8 total, 8 up, 8 in 2021-07-31T01:54:25.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:24 smithi045 conmon[39718]: cluster 2021-07 2021-07-31T01:54:25.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:24 smithi045 conmon[39718]: -31T01:54:23.893598+0000 mgr.x ( 2021-07-31T01:54:25.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:24 smithi045 conmon[39718]: mgr.24542) 302 : cluster [DBG] pgmap v624: 25 pgs: 25 active+clean; 293 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 15 MiB/s rd, 4.1 MiB/s wr, 24 op/s 2021-07-31T01:54:25.673 INFO:tasks.rados.rados.0.smithi045.stdout:2698: done (3 left) 2021-07-31T01:54:25.673 INFO:tasks.rados.rados.0.smithi045.stdout:2699: snap_remove snap 247 2021-07-31T01:54:26.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:25 smithi154 conmon[34553]: cluster 2021-07-31T01:54:24.664204+0000 mon.a (mon.0) 1220 : cluster [DBG] osdmap e563: 8 total, 8 up, 8 in 2021-07-31T01:54:26.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:25 smithi045 conmon[33655]: cluster 2021-07-31T 2021-07-31T01:54:26.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:25 smithi045 conmon[33655]: 01:54:24.664204+0000 mon.a (mon.0) 1220 : cluster [DBG] osdmap e563: 8 total, 8 up, 8 in 2021-07-31T01:54:26.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:25 smithi045 conmon[39718]: cluster 2021-07-31T01:54:24.664204+0000 mon.a (mon.0 2021-07-31T01:54:26.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:25 smithi045 conmon[39718]: ) 1220 : cluster [DBG] osdmap e563: 8 total, 8 up, 8 in 2021-07-31T01:54:26.677 INFO:tasks.rados.rados.0.smithi045.stdout:2699: done (3 left) 2021-07-31T01:54:26.677 INFO:tasks.rados.rados.0.smithi045.stdout:2700: read oid 16 snap 253 2021-07-31T01:54:26.677 INFO:tasks.rados.rados.0.smithi045.stdout:2700: expect (ObjNum 684 snap 214 seq_num 684) 2021-07-31T01:54:26.677 INFO:tasks.rados.rados.0.smithi045.stdout:2701: rollback oid 24 current snap is 256 2021-07-31T01:54:26.678 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 24 to 251 2021-07-31T01:54:26.678 INFO:tasks.rados.rados.0.smithi045.stdout:2694: expect (ObjNum 811 snap 254 seq_num 811) 2021-07-31T01:54:26.920 INFO:tasks.rados.rados.0.smithi045.stdout:2696: expect (ObjNum 801 snap 249 seq_num 801) 2021-07-31T01:54:27.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:26 smithi154 conmon[34553]: cluster 2021-07-31T01:54:25.669749+0000 mon.a (mon.0) 1221 : cluster [DBG] osdmap e564: 8 total, 8 up, 8 in 2021-07-31T01:54:27.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:26 smithi154 conmon[34553]: cluster 2021-07-31T01:54:25.894077+0000 mgr.x (mgr.24542) 303 : cluster [DBG] pgmap v627: 25 pgs: 25 active+clean; 293 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 15 MiB/s rd, 4.1 MiB/s wr, 24 op/s 2021-07-31T01:54:27.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:26 smithi045 conmon[39718]: cluster 2021-07-31T01:54:25.669749+0000 mon.a (mon.0) 1221 : 2021-07-31T01:54:27.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:26 smithi045 conmon[39718]: cluster [DBG] osdmap e564: 8 total, 8 up, 8 in 2021-07-31T01:54:27.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:26 smithi045 conmon[39718]: cluster 2021-07-31T01:54:25.894077+0000 mgr.x (mgr.24542) 303 : cluster [DBG] pgmap v627: 25 pgs: 25 active+clean; 293 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 15 MiB/s rd, 4.1 MiB/s wr, 24 op/s 2021-07-31T01:54:27.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:26 smithi045 conmon[33655]: cluster 2021-07-31T01:54:25.669749+0000 mon.a (mon.0) 1221 : cluster 2021-07-31T01:54:27.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:26 smithi045 conmon[33655]: [DBG] osdmap e564: 8 total, 8 up, 8 in 2021-07-31T01:54:27.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:26 smithi045 conmon[33655]: cluster 2021-07-31T01:54:25.894077+0000 mgr.x (mgr.24542) 303 : cluster [DBG] pgmap v627: 25 pgs: 25 active+clean; 293 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 15 MiB/s rd, 4.1 MiB/s wr, 24 op/s 2021-07-31T01:54:27.078 INFO:tasks.rados.rados.0.smithi045.stdout:2695: finishing copy_from to smithi04569269-12 2021-07-31T01:54:27.078 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 12 v 1576 (ObjNum 768 snap 235 seq_num 768) dirty exists 2021-07-31T01:54:27.078 INFO:tasks.rados.rados.0.smithi045.stdout:2695: finishing copy_from racing read to smithi04569269-12 2021-07-31T01:54:27.078 INFO:tasks.rados.rados.0.smithi045.stdout:2694: done (4 left) 2021-07-31T01:54:27.079 INFO:tasks.rados.rados.0.smithi045.stdout:2695: done (3 left) 2021-07-31T01:54:27.079 INFO:tasks.rados.rados.0.smithi045.stdout:2696: done (2 left) 2021-07-31T01:54:27.079 INFO:tasks.rados.rados.0.smithi045.stdout:2702: snap_create 2021-07-31T01:54:27.079 INFO:tasks.rados.rados.0.smithi045.stdout:2700: expect (ObjNum 684 snap 214 seq_num 684) 2021-07-31T01:54:27.156 INFO:tasks.rados.rados.0.smithi045.stdout:2701: finishing rollback tid 0 to smithi04569269-24 2021-07-31T01:54:27.157 INFO:tasks.rados.rados.0.smithi045.stdout:2701: finishing rollback tid 1 to smithi04569269-24 2021-07-31T01:54:27.157 INFO:tasks.rados.rados.0.smithi045.stdout:2701: finishing rollback tid 2 to smithi04569269-24 2021-07-31T01:54:27.157 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 24 v 1605 (ObjNum 750 snap 231 seq_num 750) dirty exists 2021-07-31T01:54:27.678 INFO:tasks.rados.rados.0.smithi045.stdout:2700: done (2 left) 2021-07-31T01:54:27.678 INFO:tasks.rados.rados.0.smithi045.stdout:2701: done (1 left) 2021-07-31T01:54:27.678 INFO:tasks.rados.rados.0.smithi045.stdout:2702: done (0 left) 2021-07-31T01:54:27.678 INFO:tasks.rados.rados.0.smithi045.stdout:2703: read oid 13 snap -1 2021-07-31T01:54:27.679 INFO:tasks.rados.rados.0.smithi045.stdout:2703: expect (ObjNum 798 snap 246 seq_num 798) 2021-07-31T01:54:27.679 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 14 current snap is 257 2021-07-31T01:54:27.679 INFO:tasks.rados.rados.0.smithi045.stdout:2704: seq_num 816 ranges {1351680=475136,1826816=540672,2367488=614400,2981888=49152} 2021-07-31T01:54:27.688 INFO:tasks.rados.rados.0.smithi045.stdout:2704: writing smithi04569269-14 from 1351680 to 1826816 tid 1 2021-07-31T01:54:27.690 INFO:tasks.rados.rados.0.smithi045.stdout:2704: writing smithi04569269-14 from 1826816 to 2367488 tid 2 2021-07-31T01:54:27.695 INFO:tasks.rados.rados.0.smithi045.stdout:2704: writing smithi04569269-14 from 2367488 to 2981888 tid 3 2021-07-31T01:54:27.695 INFO:tasks.rados.rados.0.smithi045.stdout:2704: writing smithi04569269-14 from 2981888 to 3031040 tid 4 2021-07-31T01:54:27.696 INFO:tasks.rados.rados.0.smithi045.stdout:2705: rmattr oid 44 current snap is 257 2021-07-31T01:54:27.696 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 5 current snap is 257 2021-07-31T01:54:27.697 INFO:tasks.rados.rados.0.smithi045.stdout:2706: seq_num 817 ranges {10551296=491520,11042816=417792,11460608=425984,11886592=360448} 2021-07-31T01:54:27.755 INFO:tasks.rados.rados.0.smithi045.stdout:2706: writing smithi04569269-5 from 10551296 to 11042816 tid 1 2021-07-31T01:54:27.757 INFO:tasks.rados.rados.0.smithi045.stdout:2706: writing smithi04569269-5 from 11042816 to 11460608 tid 2 2021-07-31T01:54:27.759 INFO:tasks.rados.rados.0.smithi045.stdout:2706: writing smithi04569269-5 from 11460608 to 11886592 tid 3 2021-07-31T01:54:27.762 INFO:tasks.rados.rados.0.smithi045.stdout:2706: writing smithi04569269-5 from 11886592 to 12247040 tid 4 2021-07-31T01:54:27.762 INFO:tasks.rados.rados.0.smithi045.stdout:2707: rollback oid 4 current snap is 257 2021-07-31T01:54:27.762 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 4 to 242 2021-07-31T01:54:27.763 INFO:tasks.rados.rados.0.smithi045.stdout:2704: finishing write tid 1 to smithi04569269-14 2021-07-31T01:54:27.763 INFO:tasks.rados.rados.0.smithi045.stdout:2704: finishing write tid 2 to smithi04569269-14 2021-07-31T01:54:27.763 INFO:tasks.rados.rados.0.smithi045.stdout:2704: finishing write tid 3 to smithi04569269-14 2021-07-31T01:54:27.763 INFO:tasks.rados.rados.0.smithi045.stdout:2704: finishing write tid 4 to smithi04569269-14 2021-07-31T01:54:27.763 INFO:tasks.rados.rados.0.smithi045.stdout:2704: finishing write tid 5 to smithi04569269-14 2021-07-31T01:54:27.764 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 44 v 1428 (ObjNum 776 snap 236 seq_num 776) dirty exists 2021-07-31T01:54:27.764 INFO:tasks.rados.rados.0.smithi045.stdout:2704: finishing write tid 6 to smithi04569269-14 2021-07-31T01:54:27.764 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 14 v 1427 (ObjNum 816 snap 257 seq_num 816) dirty exists 2021-07-31T01:54:27.764 INFO:tasks.rados.rados.0.smithi045.stdout:2704: left oid 14 (ObjNum 816 snap 257 seq_num 816) 2021-07-31T01:54:27.764 INFO:tasks.rados.rados.0.smithi045.stdout:2703: expect (ObjNum 798 snap 246 seq_num 798) 2021-07-31T01:54:27.993 INFO:tasks.rados.rados.0.smithi045.stdout:2706: finishing write tid 1 to smithi04569269-5 2021-07-31T01:54:27.993 INFO:tasks.rados.rados.0.smithi045.stdout:2706: finishing write tid 2 to smithi04569269-5 2021-07-31T01:54:27.994 INFO:tasks.rados.rados.0.smithi045.stdout:2706: finishing write tid 3 to smithi04569269-5 2021-07-31T01:54:27.994 INFO:tasks.rados.rados.0.smithi045.stdout:2706: finishing write tid 4 to smithi04569269-5 2021-07-31T01:54:27.994 INFO:tasks.rados.rados.0.smithi045.stdout:2706: finishing write tid 5 to smithi04569269-5 2021-07-31T01:54:27.995 INFO:tasks.rados.rados.0.smithi045.stdout:2707: finishing rollback tid 0 to smithi04569269-4 2021-07-31T01:54:27.995 INFO:tasks.rados.rados.0.smithi045.stdout:2703: done (4 left) 2021-07-31T01:54:27.995 INFO:tasks.rados.rados.0.smithi045.stdout:2704: done (3 left) 2021-07-31T01:54:27.995 INFO:tasks.rados.rados.0.smithi045.stdout:2705: done (2 left) 2021-07-31T01:54:27.995 INFO:tasks.rados.rados.0.smithi045.stdout:2708: read oid 24 snap -1 2021-07-31T01:54:27.996 INFO:tasks.rados.rados.0.smithi045.stdout:2708: expect (ObjNum 750 snap 231 seq_num 750) 2021-07-31T01:54:27.996 INFO:tasks.rados.rados.0.smithi045.stdout:2709: delete oid 13 current snap is 257 2021-07-31T01:54:27.996 INFO:tasks.rados.rados.0.smithi045.stdout:2707: finishing rollback tid 1 to smithi04569269-4 2021-07-31T01:54:27.997 INFO:tasks.rados.rados.0.smithi045.stdout:2707: finishing rollback tid 2 to smithi04569269-4 2021-07-31T01:54:27.997 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 4 v 1623 (ObjNum 716 snap 222 seq_num 716) dirty exists 2021-07-31T01:54:27.997 INFO:tasks.rados.rados.0.smithi045.stdout:2706: finishing write tid 6 to smithi04569269-5 2021-07-31T01:54:27.998 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 5 v 1623 (ObjNum 817 snap 257 seq_num 817) dirty exists 2021-07-31T01:54:27.998 INFO:tasks.rados.rados.0.smithi045.stdout:2706: left oid 5 (ObjNum 817 snap 257 seq_num 817) 2021-07-31T01:54:27.999 INFO:tasks.rados.rados.0.smithi045.stdout:2706: done (3 left) 2021-07-31T01:54:27.999 INFO:tasks.rados.rados.0.smithi045.stdout:2707: done (2 left) 2021-07-31T01:54:27.999 INFO:tasks.rados.rados.0.smithi045.stdout:2709: done (1 left) 2021-07-31T01:54:28.000 INFO:tasks.rados.rados.0.smithi045.stdout:2710: read oid 9 snap -1 2021-07-31T01:54:28.001 INFO:tasks.rados.rados.0.smithi045.stdout:2710: expect deleted 2021-07-31T01:54:28.001 INFO:tasks.rados.rados.0.smithi045.stdout:2711: rmattr oid 47 current snap is 257 2021-07-31T01:54:28.001 INFO:tasks.rados.rados.0.smithi045.stdout:2711: done (2 left) 2021-07-31T01:54:28.001 INFO:tasks.rados.rados.0.smithi045.stdout:2712: snap_create 2021-07-31T01:54:28.002 INFO:tasks.rados.rados.0.smithi045.stdout:2708: expect (ObjNum 750 snap 231 seq_num 750) 2021-07-31T01:54:28.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:27 smithi154 conmon[34553]: cluster 2021-07-31T01:54:28.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:27 smithi154 conmon[34553]: 2021-07-31T01:54:26.675396+0000 2021-07-31T01:54:28.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:27 smithi154 conmon[34553]: mon.a (mon.0) 1222 : cluster [DBG] osdmap e565: 8 total, 8 up, 8 in 2021-07-31T01:54:28.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:27 smithi045 conmon[33655]: cluster 2021-07-31T01:54:26.675396+0000 mon.a (mon.0) 2021-07-31T01:54:28.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:27 smithi045 conmon[33655]: 1222 : cluster [DBG] osdmap e565: 8 total, 8 up, 8 in 2021-07-31T01:54:28.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:27 smithi045 conmon[39718]: cluster 2021-07-31T01:54:26.675396+0000 2021-07-31T01:54:28.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:27 smithi045 conmon[39718]: mon.a (mon.0) 1222 : cluster [DBG] osdmap e565: 8 total, 8 up, 8 in 2021-07-31T01:54:28.691 INFO:tasks.rados.rados.0.smithi045.stdout:2708: done (2 left) 2021-07-31T01:54:28.692 INFO:tasks.rados.rados.0.smithi045.stdout:2710: done (1 left) 2021-07-31T01:54:28.692 INFO:tasks.rados.rados.0.smithi045.stdout:2712: done (0 left) 2021-07-31T01:54:28.692 INFO:tasks.rados.rados.0.smithi045.stdout:2713: read oid 49 snap -1 2021-07-31T01:54:28.693 INFO:tasks.rados.rados.0.smithi045.stdout:2713: expect (ObjNum 768 snap 235 seq_num 768) 2021-07-31T01:54:28.693 INFO:tasks.rados.rados.0.smithi045.stdout:2714: rollback oid 9 current snap is 258 2021-07-31T01:54:28.693 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 9 to 246 2021-07-31T01:54:28.693 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 7 current snap is 258 2021-07-31T01:54:28.694 INFO:tasks.rados.rados.0.smithi045.stdout:2715: seq_num 819 ranges {7626752=524288,8151040=589824,8740864=614400} 2021-07-31T01:54:28.735 INFO:tasks.rados.rados.0.smithi045.stdout:2715: writing smithi04569269-7 from 7626752 to 8151040 tid 1 2021-07-31T01:54:28.739 INFO:tasks.rados.rados.0.smithi045.stdout:2715: writing smithi04569269-7 from 8151040 to 8740864 tid 2 2021-07-31T01:54:28.742 INFO:tasks.rados.rados.0.smithi045.stdout:2715: writing smithi04569269-7 from 8740864 to 9355264 tid 3 2021-07-31T01:54:28.742 INFO:tasks.rados.rados.0.smithi045.stdout:2716: snap_remove snap 252 2021-07-31T01:54:28.939 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:28 smithi045 conmon[39718]: cluster 2021-07-31T01:54:27.676993+0000 mon.a ( 2021-07-31T01:54:28.940 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:28 smithi045 conmon[39718]: mon.0) 1223 : cluster [DBG] osdmap e566: 8 total, 8 up, 8 in 2021-07-31T01:54:28.940 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:28 smithi045 conmon[39718]: cluster 2021-07-31T01 2021-07-31T01:54:28.940 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:28 smithi045 conmon[39718]: :54:27.894382+0000 mgr.x (mgr.24542) 304 : cluster 2021-07-31T01:54:28.941 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:28 smithi045 conmon[39718]: [DBG] pgmap v630: 25 pgs: 25 active+clean; 293 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:54:28.941 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:28 smithi045 conmon[39718]: audit 2021-07-31T01:54:27.964945+0000 mon.b 2021-07-31T01:54:28.941 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:28 smithi045 conmon[39718]: (mon.2) 97 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:54:28.941 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:28 smithi045 conmon[39718]: audit 2021-07-31T01:54:27.965317+0000 2021-07-31T01:54:28.942 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:28 smithi045 conmon[39718]: mon.a (mon.0) 1224 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:54:28.942 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:28 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T01:54:28.942 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:28 smithi045 conmon[39718]: T01:54:27.978845+0000 mon.b (mon. 2021-07-31T01:54:28.942 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:28 smithi045 conmon[39718]: 2) 98 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T01:54:28.942 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:28 smithi045 conmon[39718]: audit 2021-07-31T01:54: 2021-07-31T01:54:28.943 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:28 smithi045 conmon[39718]: 27.979139+0000 mon.a (mon.0) 1225 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T01:54:28.943 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:28 smithi045 conmon[33655]: cluster 2021-07-31T01:54:27.676993+0000 mon.a (mon.0) 2021-07-31T01:54:28.943 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:28 smithi045 conmon[33655]: 1223 : cluster [DBG] osdmap e566: 8 total, 8 up, 8 in 2021-07-31T01:54:28.943 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:28 smithi045 conmon[33655]: cluster 2021-07-31T01:54:27.894382+0000 mgr.x (mgr.24542) 304 : cluster [DBG] pgmap v630: 25 pgs: 25 active+clean; 293 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:54:28.944 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:28 smithi045 conmon[33655]: audit 2021-07-31T01:54:27.964945+0000 mon.b (mon.2) 2021-07-31T01:54:28.944 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:28 smithi045 conmon[33655]: 97 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:54:28.944 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:28 smithi045 conmon[33655]: audit 2021-07-31T01:54:28.944 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:28 smithi045 conmon[33655]: 2021-07-31T01: 2021-07-31T01:54:28.945 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:28 smithi045 conmon[33655]: 54:27.965317+0000 mon.a 2021-07-31T01:54:28.945 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:28 smithi045 conmon[33655]: (mon.0) 1224 : audit 2021-07-31T01:54:28.945 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:28 smithi045 conmon[33655]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:54:28.945 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:28 smithi045 conmon[33655]: audit 2021- 2021-07-31T01:54:28.945 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:28 smithi045 conmon[33655]: 07-31T01:54:27.978845 2021-07-31T01:54:28.946 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:28 smithi045 conmon[33655]: +0000 mon.b (mon.2) 2021-07-31T01:54:28.946 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:28 smithi045 conmon[33655]: 98 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T01:54:28.946 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:28 smithi045 conmon[33655]: audit 2021-07 2021-07-31T01:54:28.946 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:28 smithi045 conmon[33655]: -31T01:54:27.979139+0000 mon.a (mon 2021-07-31T01:54:28.947 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:28 smithi045 conmon[33655]: .0) 1225 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T01:54:28.947 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:28 smithi045 conmon[33655]: 2021-07-31T01:54:28.950 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[34553]: cluster 2021-07-31T01:54:27.676993+0000 mon.a (mon.0) 1223 : cluster [DBG] osdmap e566: 8 total, 8 up, 8 in 2021-07-31T01:54:28.951 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[34553]: cluster 2021-07-31T01:54:27.894382+0000 mgr.x (mgr.24542) 304 : cluster [DBG] pgmap v630: 25 pgs: 25 active+clean; 293 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:54:28.951 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[34553]: audit 2021-07-31T01:54:27.964945+0000 mon.b (mon.2) 97 : audit 2021-07-31T01:54:28.951 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[34553]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:54:28.951 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[34553]: audit 2021-07-31T01:54:27.965317+0000 2021-07-31T01:54:28.952 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[34553]: mon.a (mon.0) 1224 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:54:28.952 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[34553]: audit 2021- 2021-07-31T01:54:28.952 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[34553]: 07-31T01:54:27.978845+0000 mon.b ( 2021-07-31T01:54:28.952 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[34553]: mon.2) 98 : audit [INF] 2021-07-31T01:54:28.953 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[34553]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T01:54:28.953 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[34553]: audit 2021-07-31T01:54:27 2021-07-31T01:54:28.953 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[34553]: .979139+0000 mon.a (mon.0) 1225 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T01:54:28.953 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: debug 2021-07-31T01:54:28.923+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T01:54:28.954 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:54:28.954 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:54:28.954 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: yield 2021-07-31T01:54:28.954 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:54:28.954 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:54:28.955 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:54:28.955 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T01:54:28.955 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:54:28.955 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T01:54:28.956 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:54:28.956 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: await self._waiter 2021-07-31T01:54:28.956 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:54:28.956 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: task.result() 2021-07-31T01:54:28.956 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T01:54:28.957 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: debug 2021-07-31T01:54:28.934+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T01:54:28.957 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:54:28.957 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:54:28.957 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: yield 2021-07-31T01:54:28.958 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:54:28.958 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:54:28.958 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:54:28.958 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T01:54:28.958 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:54:28.959 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T01:54:28.959 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:54:28.959 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: await self._waiter 2021-07-31T01:54:28.959 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:54:28.960 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: task.result() 2021-07-31T01:54:28.960 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T01:54:29.253 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: debug 2021-07-31T01:54:28.959+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T01:54:29.253 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:54:29.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:54:29.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: yield 2021-07-31T01:54:29.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:54:29.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:54:29.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:54:29.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T01:54:29.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:54:29.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T01:54:29.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:54:29.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: await self._waiter 2021-07-31T01:54:29.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:54:29.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: task.result() 2021-07-31T01:54:29.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T01:54:29.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: debug 2021-07-31T01:54:28.960+0000 7fcb9195d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:54:29.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:54:29.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:54:29.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: yield 2021-07-31T01:54:29.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:54:29.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:54:29.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:54:29.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T01:54:29.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:54:29.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T01:54:29.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:54:29.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: await self._waiter 2021-07-31T01:54:29.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:54:29.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: task.result() 2021-07-31T01:54:29.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T01:54:29.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: 2021-07-31T01:54:29.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T01:54:29.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: 2021-07-31T01:54:29.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:54:29.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:54:29.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: return f(*arg) 2021-07-31T01:54:29.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:54:29.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:54:29.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:54:29.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:54:29.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:54:29.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:54:29.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:54:29.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T01:54:29.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:54:29.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: raise self._exception 2021-07-31T01:54:29.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:54:29.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:54:29.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:54:29.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T01:54:29.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:54:29.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: raise 2021-07-31T01:54:29.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:54:29.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T01:54:29.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:54:29.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:54:29.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T01:54:29.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: debug 2021-07-31T01:54:28.968+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T01:54:29.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:54:29.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:54:29.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: yield 2021-07-31T01:54:29.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:54:29.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:54:29.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:54:29.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T01:54:29.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:54:29.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T01:54:29.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:54:29.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: await self._waiter 2021-07-31T01:54:29.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:54:29.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: task.result() 2021-07-31T01:54:29.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T01:54:29.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: debug 2021-07-31T01:54:28.969+0000 7fcb9215e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:54:29.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:54:29.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:54:29.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: yield 2021-07-31T01:54:29.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:54:29.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:54:29.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:54:29.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T01:54:29.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:54:29.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T01:54:29.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:54:29.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: await self._waiter 2021-07-31T01:54:29.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:54:29.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: task.result() 2021-07-31T01:54:29.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T01:54:29.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: 2021-07-31T01:54:29.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T01:54:29.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: 2021-07-31T01:54:29.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:54:29.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:54:29.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: return f(*arg) 2021-07-31T01:54:29.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:54:29.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:54:29.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:54:29.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:54:29.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:54:29.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:54:29.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:54:29.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T01:54:29.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:54:29.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: raise self._exception 2021-07-31T01:54:29.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:54:29.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:54:29.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:54:29.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T01:54:29.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:54:29.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: raise 2021-07-31T01:54:29.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:54:29.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T01:54:29.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:54:29.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:54:29.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:54:28 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T01:54:29.696 INFO:tasks.rados.rados.0.smithi045.stdout:2716: done (3 left) 2021-07-31T01:54:29.696 INFO:tasks.rados.rados.0.smithi045.stdout:2717: delete oid 14 current snap is 258 2021-07-31T01:54:29.696 INFO:tasks.rados.rados.0.smithi045.stdout:2714: finishing rollback tid 1 to smithi04569269-9 2021-07-31T01:54:29.696 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 9 v 0 (ObjNum -1104831712 snap 21873 seq_num 0) dirty dne 2021-07-31T01:54:29.697 INFO:tasks.rados.rados.0.smithi045.stdout:2713: expect (ObjNum 768 snap 235 seq_num 768) 2021-07-31T01:54:30.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: cluster 2021-07-31T01:54:28.690118+0000 mon.a (mon.0) 1226 : cluster [DBG] osdmap e567: 8 total, 8 up, 8 in 2021-07-31T01:54:30.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: audit 2021-07-31T01:54:28.879234+0000 mon.b (mon.2) 99 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:54:30.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: audit 2021-07-31T01:54:28.880563+0000 mon.b (mon.2) 100 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:54:30.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: audit 2021-07-31T01:54: 2021-07-31T01:54:30.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: 28.881586+0000 mon.b (mon.2) 101 : audit 2021-07-31T01:54:30.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:54:30.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: cephadm 2021-07-31 2021-07-31T01:54:30.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: T01:54:28.882287+0000 mgr.x (mgr 2021-07-31T01:54:30.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: .24542) 305 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T01:54:30.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: cephadm 2021-07-31T 2021-07-31T01:54:30.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: 01:54:28.924301+0000 mgr.x (mgr.24542) 306 : 2021-07-31T01:54:30.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: cephadm [ERR] 2021-07-31T01:54:30.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:54:30.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:54:30.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: yield 2021-07-31T01:54:30.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:54:30.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:54:30.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:54:30.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T01:54:30.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:54:30.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T01:54:30.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:54:30.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: await self._waiter 2021-07-31T01:54:30.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:54:30.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: task.result() 2021-07-31T01:54:30.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T01:54:30.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: cephadm 2021-07-31T01:54:28.924874+0000 mgr.x (mgr.24542) 307 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T01:54:30.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: cephadm 2021-07-31T01:54:28.935795+0000 mgr.x (mgr.24542) 308 : cephadm [ERR] 2021-07-31T01:54:30.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:54:30.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:54:30.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: yield 2021-07-31T01:54:30.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:54:30.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:54:30.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:54:30.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T01:54:30.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:54:30.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T01:54:30.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:54:30.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: await self._waiter 2021-07-31T01:54:30.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:54:30.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: task.result() 2021-07-31T01:54:30.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T01:54:30.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: cephadm 2021-07-31T01:54:28.936621+0000 mgr.x (mgr.24542) 309 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T01:54:30.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: cephadm 2021-07-31T01:54:28.960788+0000 mgr.x (mgr.24542) 310 : cephadm 2021-07-31T01:54:30.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: [ERR] 2021-07-31T01:54:30.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:54:30.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:54:30.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: yield 2021-07-31T01:54:30.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:54:30.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:54:30.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:54:30.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T01:54:30.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:54:30.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T01:54:30.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:54:30.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: await self._waiter 2021-07-31T01:54:30.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:54:30.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: task.result() 2021-07-31T01:54:30.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T01:54:30.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: cephadm 2021-07-31T01:54:28.961243+0000 mgr.x (mgr.24542) 311 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:54:30.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:54:30.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:54:30.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: yield 2021-07-31T01:54:30.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:54:30.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:54:30.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:54:30.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T01:54:30.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:54:30.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T01:54:30.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:54:30.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: await self._waiter 2021-07-31T01:54:30.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:54:30.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: task.result() 2021-07-31T01:54:30.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T01:54:30.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: 2021-07-31T01:54:30.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T01:54:30.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: 2021-07-31T01:54:30.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:54:30.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:54:30.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: return f(*arg) 2021-07-31T01:54:30.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:54:30.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:54:30.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:54:30.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:54:30.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:54:30.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:54:30.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:54:30.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T01:54:30.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:54:30.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: raise self._exception 2021-07-31T01:54:30.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:54:30.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:54:30.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:54:30.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T01:54:30.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:54:30.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: raise 2021-07-31T01:54:30.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:54:30.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T01:54:30.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:54:30.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:54:30.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T01:54:30.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: cephadm 2021-07-31T01:54:28.969846+0000 mgr.x ( 2021-07-31T01:54:30.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: mgr.24542) 312 : cephadm [ERR] 2021-07-31T01:54:30.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:54:30.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:54:30.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: yield 2021-07-31T01:54:30.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:54:30.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:54:30.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:54:30.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T01:54:30.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:54:30.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T01:54:30.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:54:30.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: await self._waiter 2021-07-31T01:54:30.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:54:30.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: task.result() 2021-07-31T01:54:30.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T01:54:30.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: cluster 2021-07-31T01:54:28.690118+0000 mon.a (mon.0) 1226 : cluster 2021-07-31T01:54:30.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: [DBG] osdmap e567: 8 total, 8 up, 8 in 2021-07-31T01:54:30.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: audit 2021-07-31T01:54:28.879234+0000 mon.b (mon.2) 99 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:54:30.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: audit 2021-07-31T01:54:28.880563+0000 mon.b (mon.2) 2021-07-31T01:54:30.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: 100 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:54:30.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: audit 2021-07-31T01:54: 2021-07-31T01:54:30.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: 28.881586+0000 mon.b (mon.2) 101 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:54:30.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: 2021-07-31T01:54:30.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: cephadm 2021-07-31T01:54:30.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: 2021-07-31T01:54:28.882287+0000 mgr.x (mgr.24542) 305 : 2021-07-31T01:54:30.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T01:54:30.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: cephadm 2021-07-31T01:54:28.924301+0000 mgr.x (mgr 2021-07-31T01:54:30.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: .24542) 306 : cephadm [ERR] 2021-07-31T01:54:30.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:54:30.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:54:30.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: yield 2021-07-31T01:54:30.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:54:30.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:54:30.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:54:30.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T01:54:30.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:54:30.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T01:54:30.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:54:30.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: await self._waiter 2021-07-31T01:54:30.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:54:30.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: task.result() 2021-07-31T01:54:30.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T01:54:30.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: cephadm 2021-07-31T01:54:28.924874+0000 mgr.x (mgr.24542) 307 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T01:54:30.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: cephadm 2021-07-31T01:54:28.935795+0000 mgr.x (mgr.24542) 308 : cephadm [ERR] 2021-07-31T01:54:30.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:54:30.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:54:30.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: yield 2021-07-31T01:54:30.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:54:30.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:54:30.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:54:30.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T01:54:30.038 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: cephadm 2021-07-31T01:54:28.970209+0000 mgr.x (mgr.24542) 313 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:54:30.039 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:54:30.039 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:54:30.039 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: yield 2021-07-31T01:54:30.039 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:54:30.039 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:54:30.040 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:54:30.040 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T01:54:30.040 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:54:30.040 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T01:54:30.040 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:54:30.041 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: await self._waiter 2021-07-31T01:54:30.041 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:54:30.041 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: task.result() 2021-07-31T01:54:30.041 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T01:54:30.042 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: 2021-07-31T01:54:30.042 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T01:54:30.042 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: 2021-07-31T01:54:30.042 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:54:30.042 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:54:30.043 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: return f(*arg) 2021-07-31T01:54:30.043 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:54:30.043 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:54:30.043 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:54:30.043 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:54:30.044 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:54:30.044 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:54:30.044 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:54:30.044 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T01:54:30.045 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:54:30.045 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: raise self._exception 2021-07-31T01:54:30.045 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:54:30.045 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:54:30.045 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:54:30.046 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T01:54:30.046 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:54:30.046 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: raise 2021-07-31T01:54:30.046 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:54:30.047 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T01:54:30.047 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:54:30.047 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:54:30.047 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:29 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T01:54:30.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: cluster 2021-07-31T01:54:28.690118+0000 mon.a ( 2021-07-31T01:54:30.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: mon.0) 1226 : cluster [DBG] osdmap e567: 8 total, 8 up, 8 in 2021-07-31T01:54:30.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: audit 2021-07-31T01:54:28.879234+0000 2021-07-31T01:54:30.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: mon.b (mon.2) 99 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:54:30.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: audit 2021-07-31T01:54:28.880563+0000 2021-07-31T01:54:30.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: mon.b (mon.2) 100 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:54:30.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: 2021-07-31T01:54:30.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: audit 2021-07-31T01:54:30.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: 2021-07-31T01:54:28.881586+0000 mon.b ( 2021-07-31T01:54:30.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: mon.2) 101 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:54:30.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: cephadm 2021- 2021-07-31T01:54:30.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: 07-31T01:54:28.882287+0000 2021-07-31T01:54:30.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: mgr.x (mgr.24542) 305 : cephadm [WRN] 2021-07-31T01:54:30.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T01:54:30.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: cephadm 2021-07-31T01 2021-07-31T01:54:30.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: :54:28.924301+0000 mgr.x (mgr. 2021-07-31T01:54:30.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: 24542) 306 : cephadm [ERR] 2021-07-31T01:54:30.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:54:30.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:54:30.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: yield 2021-07-31T01:54:30.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:54:30.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:54:30.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:54:30.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T01:54:30.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:54:30.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T01:54:30.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:54:30.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: await self._waiter 2021-07-31T01:54:30.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:54:30.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: task.result() 2021-07-31T01:54:30.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T01:54:30.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: cephadm 2021-07-31T01:54:28.924874+0000 mgr.x (mgr.24542) 307 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T01:54:30.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: cephadm 2021-07-31 2021-07-31T01:54:30.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: T01:54:28.935795+0000 mgr.x (mgr.24542) 308 : cephadm [ERR] 2021-07-31T01:54:30.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:54:30.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:54:30.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: yield 2021-07-31T01:54:30.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:54:30.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:54:30.056 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:54:30.056 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T01:54:30.057 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:54:30.057 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: await self._waiter 2021-07-31T01:54:30.057 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:54:30.057 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: task.result() 2021-07-31T01:54:30.058 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T01:54:30.058 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: cephadm 2021-07-31T01 2021-07-31T01:54:30.058 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: :54:28.936621+0000 mgr.x (mgr.24542) 309 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T01:54:30.058 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: cephadm 2021-07-31T01:54:28.960788+0000 mgr.x (mgr.24542) 310 : cephadm [ERR] 2021-07-31T01:54:30.058 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:54:30.059 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:54:30.059 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: yield 2021-07-31T01:54:30.059 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:54:30.059 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:54:30.059 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:54:30.060 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T01:54:30.060 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:54:30.060 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T01:54:30.060 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:54:30.060 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: await self._waiter 2021-07-31T01:54:30.061 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:54:30.061 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: task.result() 2021-07-31T01:54:30.061 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T01:54:30.062 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: cephadm 2021-07-31T01:54:28.961243+0000 mgr.x (mgr.24542) 311 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:54:30.062 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:54:30.062 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:54:30.062 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: yield 2021-07-31T01:54:30.062 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:54:30.063 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:54:30.063 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:54:30.063 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T01:54:30.063 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:54:30.064 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T01:54:30.064 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:54:30.064 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: await self._waiter 2021-07-31T01:54:30.064 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:54:30.064 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: task.result() 2021-07-31T01:54:30.065 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T01:54:30.065 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: 2021-07-31T01:54:30.065 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T01:54:30.065 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: 2021-07-31T01:54:30.066 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:54:30.066 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T01:54:30.066 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:54:30.066 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T01:54:30.067 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:54:30.067 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: await self._waiter 2021-07-31T01:54:30.067 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:54:30.067 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: task.result() 2021-07-31T01:54:30.067 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T01:54:30.068 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: cephadm 2021-07-31T01:54:28.936621+0000 mgr.x (mgr.24542) 309 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T01:54:30.068 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: cephadm 2021-07-31 2021-07-31T01:54:30.068 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: T01:54:28.960788+0000 mgr.x (mgr.24542 2021-07-31T01:54:30.068 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: ) 310 : cephadm [ERR] 2021-07-31T01:54:30.069 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:54:30.069 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:54:30.069 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: yield 2021-07-31T01:54:30.069 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:54:30.069 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:54:30.070 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:54:30.070 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T01:54:30.070 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:54:30.070 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T01:54:30.070 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:54:30.071 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: await self._waiter 2021-07-31T01:54:30.071 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:54:30.071 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: task.result() 2021-07-31T01:54:30.071 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T01:54:30.072 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: cephadm 2021-07-31T01:54:28.961243+0000 mgr.x (mgr.24542) 311 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:54:30.072 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:54:30.072 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:54:30.072 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: yield 2021-07-31T01:54:30.072 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:54:30.073 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:54:30.073 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:54:30.073 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T01:54:30.073 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:54:30.073 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T01:54:30.074 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:54:30.074 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: await self._waiter 2021-07-31T01:54:30.074 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:54:30.074 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: task.result() 2021-07-31T01:54:30.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:54:30.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:54:30.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: return f(*arg) 2021-07-31T01:54:30.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:54:30.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:54:30.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:54:30.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:54:30.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:54:30.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:54:30.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:54:30.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T01:54:30.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:54:30.078 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: raise self._exception 2021-07-31T01:54:30.078 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:54:30.078 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:54:30.078 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:54:30.079 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T01:54:30.079 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:54:30.079 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: raise 2021-07-31T01:54:30.079 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:54:30.079 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T01:54:30.080 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:54:30.080 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:54:30.080 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T01:54:30.080 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: cephadm 2021-07-31T01:54:28.969846+0000 mgr.x (mgr.24542) 312 : cephadm [ERR] 2021-07-31T01:54:30.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:54:30.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:54:30.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: yield 2021-07-31T01:54:30.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:54:30.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:54:30.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:54:30.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T01:54:30.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:54:30.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T01:54:30.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:54:30.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: await self._waiter 2021-07-31T01:54:30.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:54:30.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: task.result() 2021-07-31T01:54:30.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T01:54:30.084 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: cephadm 2021-07-31T01:54:28.970209+0000 mgr.x (mgr.24542) 313 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:54:30.084 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:54:30.084 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:54:30.084 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: yield 2021-07-31T01:54:30.084 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:54:30.085 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:54:30.085 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:54:30.085 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T01:54:30.085 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:54:30.086 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T01:54:30.086 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:54:30.086 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: await self._waiter 2021-07-31T01:54:30.086 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:54:30.087 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: task.result() 2021-07-31T01:54:30.087 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T01:54:30.087 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: 2021-07-31T01:54:30.087 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T01:54:30.087 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: 2021-07-31T01:54:30.088 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:54:30.088 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:54:30.088 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: return f(*arg) 2021-07-31T01:54:30.088 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:54:30.088 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:54:30.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:54:30.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:54:30.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:54:30.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:54:30.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:54:30.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T01:54:30.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:54:30.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: raise self._exception 2021-07-31T01:54:30.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:54:30.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:54:30.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:54:30.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T01:54:30.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:54:30.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: raise 2021-07-31T01:54:30.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:54:30.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T01:54:30.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:54:30.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:54:30.093 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T01:54:30.093 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T01:54:30.094 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: 2021-07-31T01:54:30.094 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T01:54:30.094 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: 2021-07-31T01:54:30.094 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:54:30.095 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:54:30.095 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: return f(*arg) 2021-07-31T01:54:30.095 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:54:30.095 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:54:30.095 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:54:30.096 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:54:30.096 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:54:30.096 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:54:30.096 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:54:30.096 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T01:54:30.097 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:54:30.097 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: raise self._exception 2021-07-31T01:54:30.097 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:54:30.098 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:54:30.098 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:54:30.098 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T01:54:30.098 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:54:30.098 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: raise 2021-07-31T01:54:30.099 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:54:30.099 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T01:54:30.099 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:54:30.099 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:54:30.100 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T01:54:30.100 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: cephadm 2021-07-31T 2021-07-31T01:54:30.100 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: 01:54:28.969846+0000 mgr.x (mgr.24542) 312 : cephadm [ERR] 2021-07-31T01:54:30.100 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:54:30.101 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:54:30.101 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: yield 2021-07-31T01:54:30.101 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:54:30.101 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:54:30.101 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:54:30.102 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T01:54:30.102 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:54:30.102 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T01:54:30.102 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:54:30.103 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: await self._waiter 2021-07-31T01:54:30.103 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:54:30.103 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: task.result() 2021-07-31T01:54:30.103 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T01:54:30.103 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: cephadm 2021-07-31T01:54:28.970209+0000 mgr.x (mgr.24542) 313 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:54:30.104 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:54:30.104 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:54:30.104 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: yield 2021-07-31T01:54:30.104 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:54:30.104 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:54:30.105 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:54:30.105 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T01:54:30.105 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:54:30.105 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T01:54:30.106 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:54:30.106 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: await self._waiter 2021-07-31T01:54:30.106 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:54:30.106 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: task.result() 2021-07-31T01:54:30.106 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T01:54:30.107 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: 2021-07-31T01:54:30.107 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T01:54:30.107 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: 2021-07-31T01:54:30.107 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:54:30.107 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:54:30.108 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: return f(*arg) 2021-07-31T01:54:30.108 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:54:30.108 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:54:30.108 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:54:30.109 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:54:30.109 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:54:30.109 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:54:30.109 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:54:30.109 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T01:54:30.110 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:54:30.110 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: raise self._exception 2021-07-31T01:54:30.110 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:54:30.110 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:54:30.110 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:54:30.111 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T01:54:30.111 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:54:30.111 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: raise 2021-07-31T01:54:30.111 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:54:30.112 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T01:54:30.112 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:54:30.112 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:54:30.112 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:29 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T01:54:30.550 INFO:tasks.rados.rados.0.smithi045.stdout:2715: finishing write tid 1 to smithi04569269-7 2021-07-31T01:54:30.550 INFO:tasks.rados.rados.0.smithi045.stdout:2715: finishing write tid 2 to smithi04569269-7 2021-07-31T01:54:30.550 INFO:tasks.rados.rados.0.smithi045.stdout:2715: finishing write tid 3 to smithi04569269-7 2021-07-31T01:54:30.551 INFO:tasks.rados.rados.0.smithi045.stdout:2715: finishing write tid 4 to smithi04569269-7 2021-07-31T01:54:30.551 INFO:tasks.rados.rados.0.smithi045.stdout:2715: finishing write tid 5 to smithi04569269-7 2021-07-31T01:54:30.551 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 7 v 1623 (ObjNum 819 snap 258 seq_num 819) dirty exists 2021-07-31T01:54:30.551 INFO:tasks.rados.rados.0.smithi045.stdout:2715: left oid 7 (ObjNum 819 snap 258 seq_num 819) 2021-07-31T01:54:30.552 INFO:tasks.rados.rados.0.smithi045.stdout:2713: done (3 left) 2021-07-31T01:54:30.552 INFO:tasks.rados.rados.0.smithi045.stdout:2714: done (2 left) 2021-07-31T01:54:30.552 INFO:tasks.rados.rados.0.smithi045.stdout:2715: done (1 left) 2021-07-31T01:54:30.552 INFO:tasks.rados.rados.0.smithi045.stdout:2717: done (0 left) 2021-07-31T01:54:30.553 INFO:tasks.rados.rados.0.smithi045.stdout:2718: delete oid 29 current snap is 258 2021-07-31T01:54:30.553 INFO:tasks.rados.rados.0.smithi045.stdout:2718: done (0 left) 2021-07-31T01:54:30.554 INFO:tasks.rados.rados.0.smithi045.stdout:2719: rmattr oid 32 current snap is 258 2021-07-31T01:54:30.554 INFO:tasks.rados.rados.0.smithi045.stdout:2720: snap_remove snap 253 2021-07-31T01:54:30.703 INFO:tasks.rados.rados.0.smithi045.stdout:2720: done (1 left) 2021-07-31T01:54:30.703 INFO:tasks.rados.rados.0.smithi045.stdout:2721: read oid 29 snap -1 2021-07-31T01:54:30.703 INFO:tasks.rados.rados.0.smithi045.stdout:2721: expect deleted 2021-07-31T01:54:30.704 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 6 current snap is 258 2021-07-31T01:54:30.704 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 32 v 1405 (ObjNum 754 snap 233 seq_num 754) dirty exists 2021-07-31T01:54:30.704 INFO:tasks.rados.rados.0.smithi045.stdout:2722: seq_num 822 ranges {1662976=573440,2236416=696320,2932736=507904} 2021-07-31T01:54:30.715 INFO:tasks.rados.rados.0.smithi045.stdout:2722: writing smithi04569269-6 from 1662976 to 2236416 tid 1 2021-07-31T01:54:30.719 INFO:tasks.rados.rados.0.smithi045.stdout:2722: writing smithi04569269-6 from 2236416 to 2932736 tid 2 2021-07-31T01:54:30.722 INFO:tasks.rados.rados.0.smithi045.stdout:2722: writing smithi04569269-6 from 2932736 to 3440640 tid 3 2021-07-31T01:54:30.722 INFO:tasks.rados.rados.0.smithi045.stdout:2719: done (2 left) 2021-07-31T01:54:30.723 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 2 current snap is 258 2021-07-31T01:54:30.723 INFO:tasks.rados.rados.0.smithi045.stdout:2723: seq_num 823 ranges {0=589824,589824=573440,1163264=630784} 2021-07-31T01:54:30.725 INFO:tasks.rados.rados.0.smithi045.stdout:2723: writing smithi04569269-2 from 0 to 589824 tid 1 2021-07-31T01:54:30.728 INFO:tasks.rados.rados.0.smithi045.stdout:2723: writing smithi04569269-2 from 589824 to 1163264 tid 2 2021-07-31T01:54:30.732 INFO:tasks.rados.rados.0.smithi045.stdout:2723: writing smithi04569269-2 from 1163264 to 1794048 tid 3 2021-07-31T01:54:30.733 INFO:tasks.rados.rados.0.smithi045.stdout:2724: read oid 8 snap -1 2021-07-31T01:54:30.733 INFO:tasks.rados.rados.0.smithi045.stdout:2724: expect (ObjNum 665 snap 208 seq_num 665) 2021-07-31T01:54:30.733 INFO:tasks.rados.rados.0.smithi045.stdout:2725: snap_remove snap 250 2021-07-31T01:54:31.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:30 smithi154 conmon[34553]: cluster 2021-07-31T01:54:29.694826+0000 mon.a (mon.0) 1227 : cluster 2021-07-31T01:54:31.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:30 smithi154 conmon[34553]: [DBG] osdmap e568: 8 total, 8 up, 8 in 2021-07-31T01:54:31.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:30 smithi154 conmon[34553]: cluster 2021-07-31T01:54:29.894889+0000 mgr.x (mgr.24542) 314 : cluster [DBG] pgmap v633: 25 pgs: 25 active+clean; 296 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 19 MiB/s rd, 3.6 MiB/s wr, 13 op/s 2021-07-31T01:54:31.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:30 smithi045 conmon[33655]: cluster 2021-07-31T01:54:29.694826+0000 mon.a (mon.0) 2021-07-31T01:54:31.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:30 smithi045 conmon[33655]: 1227 : cluster [DBG] osdmap e568: 8 total, 8 up, 8 in 2021-07-31T01:54:31.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:30 smithi045 conmon[33655]: cluster 2021-07-31T01:54:29.894889+0000 mgr.x (mgr.24542) 314 : cluster [DBG] pgmap v633: 25 pgs: 25 active+clean; 296 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 19 MiB/s rd, 3.6 MiB/s wr, 13 op/s 2021-07-31T01:54:31.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:30 smithi045 conmon[39718]: cluster 2021-07-31T01:54:29 2021-07-31T01:54:31.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:30 smithi045 conmon[39718]: .694826+0000 mon.a (mon.0) 1227 : cluster [DBG] osdmap e568: 8 total, 8 up, 8 in 2021-07-31T01:54:31.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:30 smithi045 conmon[39718]: cluster 2021-07-31T01: 2021-07-31T01:54:31.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:30 smithi045 conmon[39718]: 54:29.894889+0000 mgr.x (mgr.24542 2021-07-31T01:54:31.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:30 smithi045 conmon[39718]: ) 314 : cluster [DBG] pgmap v633: 25 pgs: 25 active+clean; 296 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 19 MiB/s rd, 3.6 MiB/s wr, 13 op/s 2021-07-31T01:54:31.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:30 smithi045 conmon[39718]: 2021-07-31T01:54:31.608 INFO:tasks.rados.rados.0.smithi045.stdout:2725: done (4 left) 2021-07-31T01:54:31.608 INFO:tasks.rados.rados.0.smithi045.stdout:2726: read oid 45 snap 248 2021-07-31T01:54:31.608 INFO:tasks.rados.rados.0.smithi045.stdout:2726: expect (ObjNum 747 snap 230 seq_num 747) 2021-07-31T01:54:31.609 INFO:tasks.rados.rados.0.smithi045.stdout:2727: read oid 38 snap 249 2021-07-31T01:54:31.609 INFO:tasks.rados.rados.0.smithi045.stdout:2727: expect deleted 2021-07-31T01:54:31.609 INFO:tasks.rados.rados.0.smithi045.stdout:2728: copy_from oid 15 from oid 21 current snap is 258 2021-07-31T01:54:31.609 INFO:tasks.rados.rados.0.smithi045.stdout:2722: finishing write tid 1 to smithi04569269-6 2021-07-31T01:54:31.610 INFO:tasks.rados.rados.0.smithi045.stdout:2722: finishing write tid 2 to smithi04569269-6 2021-07-31T01:54:31.610 INFO:tasks.rados.rados.0.smithi045.stdout:2722: finishing write tid 3 to smithi04569269-6 2021-07-31T01:54:31.610 INFO:tasks.rados.rados.0.smithi045.stdout:2722: finishing write tid 4 to smithi04569269-6 2021-07-31T01:54:31.610 INFO:tasks.rados.rados.0.smithi045.stdout:2722: finishing write tid 5 to smithi04569269-6 2021-07-31T01:54:31.611 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 6 v 1613 (ObjNum 822 snap 258 seq_num 822) dirty exists 2021-07-31T01:54:31.611 INFO:tasks.rados.rados.0.smithi045.stdout:2722: left oid 6 (ObjNum 822 snap 258 seq_num 822) 2021-07-31T01:54:31.611 INFO:tasks.rados.rados.0.smithi045.stdout:2723: finishing write tid 1 to smithi04569269-2 2021-07-31T01:54:31.611 INFO:tasks.rados.rados.0.smithi045.stdout:2723: finishing write tid 2 to smithi04569269-2 2021-07-31T01:54:31.611 INFO:tasks.rados.rados.0.smithi045.stdout:2723: finishing write tid 3 to smithi04569269-2 2021-07-31T01:54:31.612 INFO:tasks.rados.rados.0.smithi045.stdout:2721: done (6 left) 2021-07-31T01:54:31.612 INFO:tasks.rados.rados.0.smithi045.stdout:2722: done (5 left) 2021-07-31T01:54:31.613 INFO:tasks.rados.rados.0.smithi045.stdout:2729: delete oid 26 current snap is 258 2021-07-31T01:54:31.613 INFO:tasks.rados.rados.0.smithi045.stdout:2723: finishing write tid 4 to smithi04569269-2 2021-07-31T01:54:31.613 INFO:tasks.rados.rados.0.smithi045.stdout:2723: finishing write tid 5 to smithi04569269-2 2021-07-31T01:54:31.613 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 2 v 1071 (ObjNum 823 snap 258 seq_num 823) dirty exists 2021-07-31T01:54:31.614 INFO:tasks.rados.rados.0.smithi045.stdout:2723: left oid 2 (ObjNum 823 snap 258 seq_num 823) 2021-07-31T01:54:31.614 INFO:tasks.rados.rados.0.smithi045.stdout:2724: expect (ObjNum 665 snap 208 seq_num 665) 2021-07-31T01:54:31.834 INFO:tasks.rados.rados.0.smithi045.stdout:2726: expect (ObjNum 747 snap 230 seq_num 747) 2021-07-31T01:54:31.923 INFO:tasks.rados.rados.0.smithi045.stdout:2728: finishing copy_from to smithi04569269-15 2021-07-31T01:54:31.924 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 15 v 1625 (ObjNum 783 snap 239 seq_num 783) dirty exists 2021-07-31T01:54:31.924 INFO:tasks.rados.rados.0.smithi045.stdout:2728: finishing copy_from racing read to smithi04569269-15 2021-07-31T01:54:31.924 INFO:tasks.rados.rados.0.smithi045.stdout:2723: done (5 left) 2021-07-31T01:54:31.925 INFO:tasks.rados.rados.0.smithi045.stdout:2724: done (4 left) 2021-07-31T01:54:31.925 INFO:tasks.rados.rados.0.smithi045.stdout:2726: done (3 left) 2021-07-31T01:54:31.925 INFO:tasks.rados.rados.0.smithi045.stdout:2727: done (2 left) 2021-07-31T01:54:31.925 INFO:tasks.rados.rados.0.smithi045.stdout:2728: done (1 left) 2021-07-31T01:54:31.925 INFO:tasks.rados.rados.0.smithi045.stdout:2729: done (0 left) 2021-07-31T01:54:31.926 INFO:tasks.rados.rados.0.smithi045.stdout:2730: snap_remove snap 256 2021-07-31T01:54:32.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:31 smithi154 conmon[34553]: cluster 2021-07-31T01:54:30.701819+0000 mon.a (mon.0) 1228 : cluster [DBG] osdmap e569: 8 total, 8 up, 8 in 2021-07-31T01:54:32.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:31 smithi154 conmon[34553]: cluster 2021-07-31T01:54 2021-07-31T01:54:32.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:31 smithi154 conmon[34553]: :31.606042+0000 mon.a (mon.0) 1229 : cluster [DBG] osdmap e570: 8 total, 8 up, 8 in 2021-07-31T01:54:32.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:31 smithi045 conmon[33655]: cluster 2021-07-31T01:54:30.701819+0000 mon.a ( 2021-07-31T01:54:32.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:31 smithi045 conmon[33655]: mon.0) 1228 : cluster [DBG] osdmap e569: 8 total, 8 up, 8 in 2021-07-31T01:54:32.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:31 smithi045 conmon[33655]: cluster 2021-07-31T01:54:31.606042+0000 mon.a (mon.0) 1229 : cluster [DBG] osdmap e570: 8 total, 8 up, 8 in 2021-07-31T01:54:32.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:31 smithi045 conmon[39718]: cluster 2021-07-31T01:54:30.701819 2021-07-31T01:54:32.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:31 smithi045 conmon[39718]: +0000 mon.a (mon.0) 1228 : cluster [DBG] osdmap e569: 8 total, 8 up, 8 in 2021-07-31T01:54:32.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:31 smithi045 conmon[39718]: cluster 2021-07 2021-07-31T01:54:32.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:31 smithi045 conmon[39718]: -31T01:54:31.606042+0000 mon.a (mon.0) 1229 : cluster [DBG] osdmap e570: 8 total, 8 up, 8 in 2021-07-31T01:54:32.711 INFO:tasks.rados.rados.0.smithi045.stdout:2730: done (0 left) 2021-07-31T01:54:32.711 INFO:tasks.rados.rados.0.smithi045.stdout:2731: read oid 6 snap 246 2021-07-31T01:54:32.711 INFO:tasks.rados.rados.0.smithi045.stdout:2731: expect deleted 2021-07-31T01:54:32.712 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 49 current snap is 258 2021-07-31T01:54:32.712 INFO:tasks.rados.rados.0.smithi045.stdout:2732: seq_num 825 ranges {10043392=622592,10665984=745472,11411456=458752} 2021-07-31T01:54:32.770 INFO:tasks.rados.rados.0.smithi045.stdout:2732: writing smithi04569269-49 from 10043392 to 10665984 tid 1 2021-07-31T01:54:32.774 INFO:tasks.rados.rados.0.smithi045.stdout:2732: writing smithi04569269-49 from 10665984 to 11411456 tid 2 2021-07-31T01:54:32.777 INFO:tasks.rados.rados.0.smithi045.stdout:2732: writing smithi04569269-49 from 11411456 to 11870208 tid 3 2021-07-31T01:54:32.777 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 46 current snap is 258 2021-07-31T01:54:32.777 INFO:tasks.rados.rados.0.smithi045.stdout:2733: seq_num 826 ranges {4349952=638976,4988928=630784,5619712=573440} 2021-07-31T01:54:32.804 INFO:tasks.rados.rados.0.smithi045.stdout:2733: writing smithi04569269-46 from 4349952 to 4988928 tid 1 2021-07-31T01:54:32.808 INFO:tasks.rados.rados.0.smithi045.stdout:2733: writing smithi04569269-46 from 4988928 to 5619712 tid 2 2021-07-31T01:54:32.811 INFO:tasks.rados.rados.0.smithi045.stdout:2733: writing smithi04569269-46 from 5619712 to 6193152 tid 3 2021-07-31T01:54:32.811 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 39 current snap is 258 2021-07-31T01:54:32.812 INFO:tasks.rados.rados.0.smithi045.stdout:2734: seq_num 827 ranges {671744=655360,1327104=516096,1843200=688128} 2021-07-31T01:54:32.818 INFO:tasks.rados.rados.0.smithi045.stdout:2734: writing smithi04569269-39 from 671744 to 1327104 tid 1 2021-07-31T01:54:32.821 INFO:tasks.rados.rados.0.smithi045.stdout:2734: writing smithi04569269-39 from 1327104 to 1843200 tid 2 2021-07-31T01:54:32.826 INFO:tasks.rados.rados.0.smithi045.stdout:2734: writing smithi04569269-39 from 1843200 to 2531328 tid 3 2021-07-31T01:54:32.826 INFO:tasks.rados.rados.0.smithi045.stdout:2735: delete oid 35 current snap is 258 2021-07-31T01:54:32.826 INFO:tasks.rados.rados.0.smithi045.stdout:2732: finishing write tid 1 to smithi04569269-49 2021-07-31T01:54:32.827 INFO:tasks.rados.rados.0.smithi045.stdout:2732: finishing write tid 2 to smithi04569269-49 2021-07-31T01:54:32.827 INFO:tasks.rados.rados.0.smithi045.stdout:2732: finishing write tid 3 to smithi04569269-49 2021-07-31T01:54:32.827 INFO:tasks.rados.rados.0.smithi045.stdout:2732: finishing write tid 4 to smithi04569269-49 2021-07-31T01:54:32.827 INFO:tasks.rados.rados.0.smithi045.stdout:2732: finishing write tid 5 to smithi04569269-49 2021-07-31T01:54:32.828 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 49 v 1735 (ObjNum 825 snap 258 seq_num 825) dirty exists 2021-07-31T01:54:32.828 INFO:tasks.rados.rados.0.smithi045.stdout:2732: left oid 49 (ObjNum 825 snap 258 seq_num 825) 2021-07-31T01:54:32.828 INFO:tasks.rados.rados.0.smithi045.stdout:2733: finishing write tid 1 to smithi04569269-46 2021-07-31T01:54:32.828 INFO:tasks.rados.rados.0.smithi045.stdout:2733: finishing write tid 2 to smithi04569269-46 2021-07-31T01:54:32.828 INFO:tasks.rados.rados.0.smithi045.stdout:2733: finishing write tid 3 to smithi04569269-46 2021-07-31T01:54:32.829 INFO:tasks.rados.rados.0.smithi045.stdout:2733: finishing write tid 4 to smithi04569269-46 2021-07-31T01:54:32.829 INFO:tasks.rados.rados.0.smithi045.stdout:2733: finishing write tid 5 to smithi04569269-46 2021-07-31T01:54:32.829 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 46 v 1302 (ObjNum 826 snap 258 seq_num 826) dirty exists 2021-07-31T01:54:32.829 INFO:tasks.rados.rados.0.smithi045.stdout:2733: left oid 46 (ObjNum 826 snap 258 seq_num 826) 2021-07-31T01:54:32.830 INFO:tasks.rados.rados.0.smithi045.stdout:2734: finishing write tid 1 to smithi04569269-39 2021-07-31T01:54:32.830 INFO:tasks.rados.rados.0.smithi045.stdout:2734: finishing write tid 2 to smithi04569269-39 2021-07-31T01:54:32.831 INFO:tasks.rados.rados.0.smithi045.stdout:2731: done (4 left) 2021-07-31T01:54:32.831 INFO:tasks.rados.rados.0.smithi045.stdout:2732: done (3 left) 2021-07-31T01:54:32.832 INFO:tasks.rados.rados.0.smithi045.stdout:2733: done (2 left) 2021-07-31T01:54:32.832 INFO:tasks.rados.rados.0.smithi045.stdout:2735: done (1 left) 2021-07-31T01:54:32.832 INFO:tasks.rados.rados.0.smithi045.stdout:2736: snap_create 2021-07-31T01:54:32.832 INFO:tasks.rados.rados.0.smithi045.stdout:2734: finishing write tid 3 to smithi04569269-39 2021-07-31T01:54:32.833 INFO:tasks.rados.rados.0.smithi045.stdout:2734: finishing write tid 4 to smithi04569269-39 2021-07-31T01:54:32.833 INFO:tasks.rados.rados.0.smithi045.stdout:2734: finishing write tid 5 to smithi04569269-39 2021-07-31T01:54:32.833 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 39 v 1627 (ObjNum 827 snap 258 seq_num 827) dirty exists 2021-07-31T01:54:32.833 INFO:tasks.rados.rados.0.smithi045.stdout:2734: left oid 39 (ObjNum 827 snap 258 seq_num 827) 2021-07-31T01:54:33.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:32 smithi154 conmon[34553]: cluster 2021-07-31T01:54:31.895416+0000 mgr.x (mgr.24542) 2021-07-31T01:54:33.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:32 smithi154 conmon[34553]: 315 : cluster [DBG] pgmap v636: 25 pgs: 25 active+clean; 296 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 19 MiB/s rd, 3.6 MiB/s wr, 13 op/s 2021-07-31T01:54:33.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:32 smithi045 conmon[39718]: cluster 2021-07-31T01:54:31.895416+0000 2021-07-31T01:54:33.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:32 smithi045 conmon[39718]: mgr.x (mgr.24542) 315 : cluster [DBG] pgmap v636: 25 pgs: 25 active+clean; 296 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 19 MiB/s rd, 3.6 MiB/s wr, 13 op/s 2021-07-31T01:54:33.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:32 smithi045 conmon[33655]: cluster 2021-07-31T01:54:31.895416+0000 mgr.x (mgr.24542) 315 : 2021-07-31T01:54:33.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:32 smithi045 conmon[33655]: cluster [DBG] pgmap v636: 25 pgs: 25 active+clean; 296 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 19 MiB/s rd, 3.6 MiB/s wr, 13 op/s 2021-07-31T01:54:33.716 INFO:tasks.rados.rados.0.smithi045.stdout:2734: done (1 left) 2021-07-31T01:54:33.716 INFO:tasks.rados.rados.0.smithi045.stdout:2736: done (0 left) 2021-07-31T01:54:33.717 INFO:tasks.rados.rados.0.smithi045.stdout:2737: snap_create 2021-07-31T01:54:33.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:33 smithi045 conmon[39718]: cluster 2021-07-31T01:54:32.709599+0000 mon.a 2021-07-31T01:54:33.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:33 smithi045 conmon[39718]: (mon.0) 1230 : cluster [DBG] osdmap e571: 8 total, 8 up, 8 in 2021-07-31T01:54:33.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:54:33 smithi045 conmon[61210]: level=error ts=2021-07-31T01:54:33.521Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:54:33.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:54:33 smithi045 conmon[61210]: level=error ts=2021-07-31T01:54:33.521Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:54:33.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:54:33 smithi045 conmon[61210]: level=error ts=2021-07-31T01:54:33.521Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T01:54:33.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:33 smithi045 conmon[33655]: cluster 2021-07-31T01:54:32.709599+0000 mon.a (mon.0) 1230 : cluster 2021-07-31T01:54:33.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:33 smithi045 conmon[33655]: [DBG] osdmap e571: 8 total, 8 up, 8 in 2021-07-31T01:54:34.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:33 smithi154 conmon[34553]: cluster 2021-07-31T01:54:32.709599+0000 mon.a (mon.0) 1230 : cluster [DBG] osdmap e571: 8 total, 8 up, 8 in 2021-07-31T01:54:34.724 INFO:tasks.rados.rados.0.smithi045.stdout:2737: done (0 left) 2021-07-31T01:54:34.725 INFO:tasks.rados.rados.0.smithi045.stdout:2738: snap_create 2021-07-31T01:54:35.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:34 smithi154 conmon[34553]: cluster 2021-07-31T01:54:33.714901+0000 mon.a ( 2021-07-31T01:54:35.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:34 smithi154 conmon[34553]: mon.0) 1231 : cluster [DBG] osdmap e572: 8 total, 8 up, 8 in 2021-07-31T01:54:35.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:34 smithi154 conmon[34553]: cluster 2021-07-31T01:54:33.895874+0000 mgr.x (mgr.24542) 316 : cluster [DBG] pgmap v639: 25 pgs: 25 active+clean; 301 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 4.6 MiB/s rd, 2.4 MiB/s wr, 10 op/s 2021-07-31T01:54:35.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:34 smithi045 conmon[33655]: cluster 2021-07-31T01:54:33.714901+0000 mon.a (mon.0) 1231 2021-07-31T01:54:35.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:34 smithi045 conmon[33655]: : cluster [DBG] osdmap e572: 8 total, 8 up, 8 in 2021-07-31T01:54:35.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:34 smithi045 conmon[33655]: cluster 2021-07-31T01:54:33.895874+0000 mgr.x (mgr.24542) 316 : cluster [DBG] pgmap v639: 25 pgs: 25 active+clean; 301 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 4.6 MiB/s rd, 2.4 MiB/s wr, 10 op/s 2021-07-31T01:54:35.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:34 smithi045 conmon[39718]: cluster 2021-07-31T01:54:33.714901+0000 mon.a (mon.0) 2021-07-31T01:54:35.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:34 smithi045 conmon[39718]: 1231 : cluster [DBG] osdmap e572: 8 total, 8 up, 8 in 2021-07-31T01:54:35.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:34 smithi045 conmon[39718]: cluster 2021-07-31T01:54:33.895874+0000 mgr.x ( 2021-07-31T01:54:35.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:34 smithi045 conmon[39718]: mgr.24542) 316 : cluster [DBG] pgmap v639: 25 pgs: 25 active+clean; 301 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 4.6 MiB/s rd, 2.4 MiB/s wr, 10 op/s 2021-07-31T01:54:35.734 INFO:tasks.rados.rados.0.smithi045.stdout:2738: done (0 left) 2021-07-31T01:54:35.735 INFO:tasks.rados.rados.0.smithi045.stdout:2739: snap_create 2021-07-31T01:54:36.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:35 smithi154 conmon[34553]: cluster 2021-07-31T01:54:34.721365+0000 mon.a (mon.0) 1232 : cluster [DBG] osdmap e573: 8 total, 8 up, 8 in 2021-07-31T01:54:36.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:35 smithi045 conmon[39718]: cluster 2021-07-31T01:54:34.721365+0000 mon.a (mon.0) 1232 : cluster [DBG] osdmap e573: 8 total, 8 up, 8 in 2021-07-31T01:54:36.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:35 smithi045 conmon[33655]: cluster 2021-07-31T01:54:34.721365+0000 mon.a (mon.0 2021-07-31T01:54:36.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:35 smithi045 conmon[33655]: ) 1232 : cluster [DBG] osdmap e573: 8 total, 8 up, 8 in 2021-07-31T01:54:36.608 INFO:tasks.rados.rados.0.smithi045.stdout:2739: done (0 left) 2021-07-31T01:54:36.609 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 17 current snap is 262 2021-07-31T01:54:36.609 INFO:tasks.rados.rados.0.smithi045.stdout:2740: seq_num 829 ranges {8380416=688128,9068544=729088,9797632=475136} 2021-07-31T01:54:36.657 INFO:tasks.rados.rados.0.smithi045.stdout:2740: writing smithi04569269-17 from 8380416 to 9068544 tid 1 2021-07-31T01:54:36.661 INFO:tasks.rados.rados.0.smithi045.stdout:2740: writing smithi04569269-17 from 9068544 to 9797632 tid 2 2021-07-31T01:54:36.665 INFO:tasks.rados.rados.0.smithi045.stdout:2740: writing smithi04569269-17 from 9797632 to 10272768 tid 3 2021-07-31T01:54:36.665 INFO:tasks.rados.rados.0.smithi045.stdout:2741: rollback oid 40 current snap is 262 2021-07-31T01:54:36.665 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 40 to 260 2021-07-31T01:54:36.665 INFO:tasks.rados.rados.0.smithi045.stdout:2740: finishing write tid 1 to smithi04569269-17 2021-07-31T01:54:36.666 INFO:tasks.rados.rados.0.smithi045.stdout:2742: snap_remove snap 258 2021-07-31T01:54:37.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:36 smithi154 conmon[34553]: cluster 2021-07-31T01:54:35.728160+0000 mon.a (mon.0) 1233 : cluster [DBG] 2021-07-31T01:54:37.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:36 smithi154 conmon[34553]: osdmap e574: 8 total, 8 up, 8 in 2021-07-31T01:54:37.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:36 smithi154 conmon[34553]: cluster 2021-07-31T01:54:35.896219+0000 mgr.x (mgr.24542) 317 : cluster [DBG] pgmap v642: 25 pgs: 25 active+clean; 301 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 4.6 MiB/s rd, 2.4 MiB/s wr, 10 op/s 2021-07-31T01:54:37.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:36 smithi154 conmon[34553]: cluster 2021-07-31T01:54:36.607548+0000 mon.a (mon.0) 2021-07-31T01:54:37.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:36 smithi154 conmon[34553]: 1234 : cluster [DBG] osdmap e575: 8 total, 8 up, 8 in 2021-07-31T01:54:37.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:36 smithi045 conmon[33655]: cluster 2021-07-31T01:54:35.728160+0000 mon.a (mon.0 2021-07-31T01:54:37.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:36 smithi045 conmon[33655]: ) 1233 : cluster [DBG] osdmap e574: 8 total, 8 up, 8 in 2021-07-31T01:54:37.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:36 smithi045 conmon[33655]: cluster 2021-07-31T01:54:35.896219+0000 mgr.x (mgr.24542) 317 : cluster [DBG] pgmap v642: 25 pgs: 25 active+clean; 301 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 4.6 MiB/s rd, 2.4 MiB/s wr, 10 op/s 2021-07-31T01:54:37.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:36 smithi045 conmon[33655]: cluster 2021-07-31T01:54:36.607548+0000 mon.a (mon.0) 1234 : cluster [DBG] osdmap e575: 8 total, 8 up, 8 in 2021-07-31T01:54:37.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:36 smithi045 conmon[33655]: 2021-07-31T01:54:37.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:36 smithi045 conmon[39718]: cluster 2021-07-31T01:54:35.728160+0000 2021-07-31T01:54:37.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:36 smithi045 conmon[39718]: mon.a (mon.0) 1233 : cluster [DBG] osdmap e574: 8 total, 8 up, 8 in 2021-07-31T01:54:37.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:36 smithi045 conmon[39718]: cluster 2021-07-31T01:54: 2021-07-31T01:54:37.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:36 smithi045 conmon[39718]: 35.896219+0000 mgr.x (mgr.24542) 317 : cluster [DBG] pgmap v642: 25 pgs: 25 active+clean; 301 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 4.6 MiB/s rd, 2.4 MiB/s wr, 10 op/s 2021-07-31T01:54:37.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:36 smithi045 conmon[39718]: cluster 2021-07-31 2021-07-31T01:54:37.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:36 smithi045 conmon[39718]: T01:54:36.607548+0000 mon.a (mon.0) 1234 : cluster [DBG] osdmap e575: 8 total, 8 up, 8 in 2021-07-31T01:54:37.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:36 smithi045 conmon[39718]: 2021-07-31T01:54:37.617 INFO:tasks.rados.rados.0.smithi045.stdout:2742: done (2 left) 2021-07-31T01:54:37.618 INFO:tasks.rados.rados.0.smithi045.stdout:2743: read oid 29 snap 248 2021-07-31T01:54:37.618 INFO:tasks.rados.rados.0.smithi045.stdout:2743: expect (ObjNum 796 snap 245 seq_num 796) 2021-07-31T01:54:37.618 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 24 current snap is 262 2021-07-31T01:54:37.619 INFO:tasks.rados.rados.0.smithi045.stdout:2740: finishing write tid 2 to smithi04569269-17 2021-07-31T01:54:37.619 INFO:tasks.rados.rados.0.smithi045.stdout:2744: seq_num 830 ranges {565248=704512,1269760=614400,1884160=573440,2457600=16384} 2021-07-31T01:54:37.624 INFO:tasks.rados.rados.0.smithi045.stdout:2744: writing smithi04569269-24 from 565248 to 1269760 tid 1 2021-07-31T01:54:37.628 INFO:tasks.rados.rados.0.smithi045.stdout:2744: writing smithi04569269-24 from 1269760 to 1884160 tid 2 2021-07-31T01:54:37.632 INFO:tasks.rados.rados.0.smithi045.stdout:2744: writing smithi04569269-24 from 1884160 to 2457600 tid 3 2021-07-31T01:54:37.633 INFO:tasks.rados.rados.0.smithi045.stdout:2744: writing smithi04569269-24 from 2457600 to 2473984 tid 4 2021-07-31T01:54:37.634 INFO:tasks.rados.rados.0.smithi045.stdout:2745: snap_create 2021-07-31T01:54:37.634 INFO:tasks.rados.rados.0.smithi045.stdout:2740: finishing write tid 3 to smithi04569269-17 2021-07-31T01:54:37.634 INFO:tasks.rados.rados.0.smithi045.stdout:2740: finishing write tid 4 to smithi04569269-17 2021-07-31T01:54:37.635 INFO:tasks.rados.rados.0.smithi045.stdout:2741: finishing rollback tid 0 to smithi04569269-40 2021-07-31T01:54:37.635 INFO:tasks.rados.rados.0.smithi045.stdout:2741: finishing rollback tid 1 to smithi04569269-40 2021-07-31T01:54:37.636 INFO:tasks.rados.rados.0.smithi045.stdout:2741: finishing rollback tid 2 to smithi04569269-40 2021-07-31T01:54:37.636 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 40 v 1616 (ObjNum 795 snap 245 seq_num 795) dirty exists 2021-07-31T01:54:37.637 INFO:tasks.rados.rados.0.smithi045.stdout:2740: finishing write tid 5 to smithi04569269-17 2021-07-31T01:54:37.637 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 17 v 1634 (ObjNum 829 snap 262 seq_num 829) dirty exists 2021-07-31T01:54:37.637 INFO:tasks.rados.rados.0.smithi045.stdout:2740: left oid 17 (ObjNum 829 snap 262 seq_num 829) 2021-07-31T01:54:37.647 INFO:tasks.rados.rados.0.smithi045.stdout:2744: finishing write tid 1 to smithi04569269-24 2021-07-31T01:54:37.648 INFO:tasks.rados.rados.0.smithi045.stdout:2744: finishing write tid 2 to smithi04569269-24 2021-07-31T01:54:37.648 INFO:tasks.rados.rados.0.smithi045.stdout:2744: finishing write tid 3 to smithi04569269-24 2021-07-31T01:54:37.648 INFO:tasks.rados.rados.0.smithi045.stdout:2744: finishing write tid 4 to smithi04569269-24 2021-07-31T01:54:37.648 INFO:tasks.rados.rados.0.smithi045.stdout:2744: finishing write tid 5 to smithi04569269-24 2021-07-31T01:54:37.649 INFO:tasks.rados.rados.0.smithi045.stdout:2744: finishing write tid 6 to smithi04569269-24 2021-07-31T01:54:37.649 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 24 v 1610 (ObjNum 830 snap 262 seq_num 830) dirty exists 2021-07-31T01:54:37.649 INFO:tasks.rados.rados.0.smithi045.stdout:2744: left oid 24 (ObjNum 830 snap 262 seq_num 830) 2021-07-31T01:54:37.671 INFO:tasks.rados.rados.0.smithi045.stdout:2743: expect (ObjNum 796 snap 245 seq_num 796) 2021-07-31T01:54:38.232 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T01:54:38.517 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T01:54:38.518 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 9m ago 10m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T01:54:38.518 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 7m ago 10m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T01:54:38.518 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 7m ago 13m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T01:54:38.519 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 9m ago 15m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T01:54:38.519 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 9m ago 15m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T01:54:38.520 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 7m ago 13m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T01:54:38.520 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 9m ago 13m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T01:54:38.520 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 9m ago 10m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T01:54:38.521 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 7m ago 10m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T01:54:38.521 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 9m ago 13m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T01:54:38.521 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 9m ago 12m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T01:54:38.521 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 9m ago 12m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T01:54:38.522 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 9m ago 12m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T01:54:38.522 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 7m ago 11m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T01:54:38.522 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 7m ago 11m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T01:54:38.522 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 7m ago 11m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T01:54:38.522 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 7m ago 10m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T01:54:38.523 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 7m ago 10m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T01:54:38.621 INFO:tasks.rados.rados.0.smithi045.stdout:2740: done (4 left) 2021-07-31T01:54:38.621 INFO:tasks.rados.rados.0.smithi045.stdout:2741: done (3 left) 2021-07-31T01:54:38.622 INFO:tasks.rados.rados.0.smithi045.stdout:2743: done (2 left) 2021-07-31T01:54:38.622 INFO:tasks.rados.rados.0.smithi045.stdout:2744: done (1 left) 2021-07-31T01:54:38.622 INFO:tasks.rados.rados.0.smithi045.stdout:2745: done (0 left) 2021-07-31T01:54:38.622 INFO:tasks.rados.rados.0.smithi045.stdout:2746: copy_from oid 24 from oid 15 current snap is 263 2021-07-31T01:54:38.622 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 8 current snap is 263 2021-07-31T01:54:38.623 INFO:tasks.rados.rados.0.smithi045.stdout:2747: seq_num 831 ranges {4349952=720896,5070848=499712,5570560=704512} 2021-07-31T01:54:38.649 INFO:tasks.rados.rados.0.smithi045.stdout:2747: writing smithi04569269-8 from 4349952 to 5070848 tid 1 2021-07-31T01:54:38.652 INFO:tasks.rados.rados.0.smithi045.stdout:2747: writing smithi04569269-8 from 5070848 to 5570560 tid 2 2021-07-31T01:54:38.656 INFO:tasks.rados.rados.0.smithi045.stdout:2747: writing smithi04569269-8 from 5570560 to 6275072 tid 3 2021-07-31T01:54:38.656 INFO:tasks.rados.rados.0.smithi045.stdout:2748: delete oid 50 current snap is 263 2021-07-31T01:54:38.657 INFO:tasks.rados.rados.0.smithi045.stdout:2746: finishing copy_from to smithi04569269-24 2021-07-31T01:54:38.657 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 24 v 1625 (ObjNum 783 snap 239 seq_num 783) dirty exists 2021-07-31T01:54:38.657 INFO:tasks.rados.rados.0.smithi045.stdout:2746: finishing copy_from racing read to smithi04569269-24 2021-07-31T01:54:38.657 INFO:tasks.rados.rados.0.smithi045.stdout:2747: finishing write tid 1 to smithi04569269-8 2021-07-31T01:54:38.658 INFO:tasks.rados.rados.0.smithi045.stdout:2747: finishing write tid 2 to smithi04569269-8 2021-07-31T01:54:38.661 INFO:tasks.rados.rados.0.smithi045.stdout:2747: finishing write tid 3 to smithi04569269-8 2021-07-31T01:54:38.661 INFO:tasks.rados.rados.0.smithi045.stdout:2746: done (2 left) 2021-07-31T01:54:38.661 INFO:tasks.rados.rados.0.smithi045.stdout:2748: done (1 left) 2021-07-31T01:54:38.662 INFO:tasks.rados.rados.0.smithi045.stdout:2749: read oid 33 snap -1 2021-07-31T01:54:38.662 INFO:tasks.rados.rados.0.smithi045.stdout:2749: expect (ObjNum 726 snap 226 seq_num 726) 2021-07-31T01:54:38.662 INFO:tasks.rados.rados.0.smithi045.stdout:2750: copy_from oid 39 from oid 47 current snap is 263 2021-07-31T01:54:38.662 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 7 current snap is 263 2021-07-31T01:54:38.662 INFO:tasks.rados.rados.0.smithi045.stdout:2751: seq_num 833 ranges {9355264=761856,10117120=671744,10788864=532480} 2021-07-31T01:54:38.720 INFO:tasks.rados.rados.0.smithi045.stdout:2751: writing smithi04569269-7 from 9355264 to 10117120 tid 1 2021-07-31T01:54:38.720 INFO:tasks.rados.rados.0.smithi045.stdout:2751: writing smithi04569269-7 from 10117120 to 10788864 tid 2 2021-07-31T01:54:38.723 INFO:tasks.rados.rados.0.smithi045.stdout:2751: writing smithi04569269-7 from 10788864 to 11321344 tid 3 2021-07-31T01:54:38.723 INFO:tasks.rados.rados.0.smithi045.stdout:2752: read oid 24 snap -1 2021-07-31T01:54:38.724 INFO:tasks.rados.rados.0.smithi045.stdout:2752: expect (ObjNum 783 snap 239 seq_num 783) 2021-07-31T01:54:38.724 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 38 current snap is 263 2021-07-31T01:54:38.724 INFO:tasks.rados.rados.0.smithi045.stdout:2747: finishing write tid 4 to smithi04569269-8 2021-07-31T01:54:38.724 INFO:tasks.rados.rados.0.smithi045.stdout:2753: seq_num 834 ranges {0=778240,778240=557056,1335296=548864,1884160=98304} 2021-07-31T01:54:38.728 INFO:tasks.rados.rados.0.smithi045.stdout:2753: writing smithi04569269-38 from 0 to 778240 tid 1 2021-07-31T01:54:38.731 INFO:tasks.rados.rados.0.smithi045.stdout:2753: writing smithi04569269-38 from 778240 to 1335296 tid 2 2021-07-31T01:54:38.738 INFO:tasks.rados.rados.0.smithi045.stdout:2753: writing smithi04569269-38 from 1335296 to 1884160 tid 3 2021-07-31T01:54:38.738 INFO:tasks.rados.rados.0.smithi045.stdout:2753: writing smithi04569269-38 from 1884160 to 1982464 tid 4 2021-07-31T01:54:38.739 INFO:tasks.rados.rados.0.smithi045.stdout:2747: finishing write tid 5 to smithi04569269-8 2021-07-31T01:54:38.739 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 8 v 1742 (ObjNum 831 snap 263 seq_num 831) dirty exists 2021-07-31T01:54:38.740 INFO:tasks.rados.rados.0.smithi045.stdout:2747: left oid 8 (ObjNum 831 snap 263 seq_num 831) 2021-07-31T01:54:38.740 INFO:tasks.rados.rados.0.smithi045.stdout:2754: read oid 10 snap -1 2021-07-31T01:54:38.740 INFO:tasks.rados.rados.0.smithi045.stdout:2754: expect (ObjNum 768 snap 235 seq_num 768) 2021-07-31T01:54:38.740 INFO:tasks.rados.rados.0.smithi045.stdout:2747: done (6 left) 2021-07-31T01:54:38.740 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 36 current snap is 263 2021-07-31T01:54:38.741 INFO:tasks.rados.rados.0.smithi045.stdout:2749: expect (ObjNum 726 snap 226 seq_num 726) 2021-07-31T01:54:38.741 INFO:tasks.rados.rados.0.smithi045.stdout:2750: finishing copy_from to smithi04569269-39 2021-07-31T01:54:38.742 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 39 v 1625 (ObjNum 783 snap 239 seq_num 783) dirty exists 2021-07-31T01:54:38.742 INFO:tasks.rados.rados.0.smithi045.stdout:2750: finishing copy_from racing read to smithi04569269-39 2021-07-31T01:54:38.742 INFO:tasks.rados.rados.0.smithi045.stdout:2751: finishing write tid 1 to smithi04569269-7 2021-07-31T01:54:38.742 INFO:tasks.rados.rados.0.smithi045.stdout:2751: finishing write tid 2 to smithi04569269-7 2021-07-31T01:54:38.742 INFO:tasks.rados.rados.0.smithi045.stdout:2752: expect (ObjNum 783 snap 239 seq_num 783) 2021-07-31T01:54:38.768 INFO:tasks.rados.rados.0.smithi045.stdout:2751: finishing write tid 3 to smithi04569269-7 2021-07-31T01:54:38.769 INFO:tasks.rados.rados.0.smithi045.stdout:2751: finishing write tid 4 to smithi04569269-7 2021-07-31T01:54:38.769 INFO:tasks.rados.rados.0.smithi045.stdout:2755: seq_num 835 ranges {0=794624,794624=442368,1236992=712704,1949696=49152} 2021-07-31T01:54:38.772 INFO:tasks.rados.rados.0.smithi045.stdout:2755: writing smithi04569269-36 from 0 to 794624 tid 1 2021-07-31T01:54:38.776 INFO:tasks.rados.rados.0.smithi045.stdout:2755: writing smithi04569269-36 from 794624 to 1236992 tid 2 2021-07-31T01:54:38.780 INFO:tasks.rados.rados.0.smithi045.stdout:2755: writing smithi04569269-36 from 1236992 to 1949696 tid 3 2021-07-31T01:54:38.781 INFO:tasks.rados.rados.0.smithi045.stdout:2755: writing smithi04569269-36 from 1949696 to 1998848 tid 4 2021-07-31T01:54:38.781 INFO:tasks.rados.rados.0.smithi045.stdout:2749: done (6 left) 2021-07-31T01:54:38.781 INFO:tasks.rados.rados.0.smithi045.stdout:2750: done (5 left) 2021-07-31T01:54:38.781 INFO:tasks.rados.rados.0.smithi045.stdout:2752: done (4 left) 2021-07-31T01:54:38.782 INFO:tasks.rados.rados.0.smithi045.stdout:2756: read oid 4 snap -1 2021-07-31T01:54:38.782 INFO:tasks.rados.rados.0.smithi045.stdout:2756: expect (ObjNum 716 snap 222 seq_num 716) 2021-07-31T01:54:38.782 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 9 current snap is 263 2021-07-31T01:54:38.782 INFO:tasks.rados.rados.0.smithi045.stdout:2757: seq_num 836 ranges {0=409600,409600=720896,1130496=516096,1646592=368640} 2021-07-31T01:54:38.783 INFO:tasks.rados.rados.0.smithi045.stdout:2757: writing smithi04569269-9 from 0 to 409600 tid 1 2021-07-31T01:54:38.789 INFO:tasks.rados.rados.0.smithi045.stdout:2757: writing smithi04569269-9 from 409600 to 1130496 tid 2 2021-07-31T01:54:38.791 INFO:tasks.rados.rados.0.smithi045.stdout:2757: writing smithi04569269-9 from 1130496 to 1646592 tid 3 2021-07-31T01:54:38.794 INFO:tasks.rados.rados.0.smithi045.stdout:2757: writing smithi04569269-9 from 1646592 to 2015232 tid 4 2021-07-31T01:54:38.795 INFO:tasks.rados.rados.0.smithi045.stdout:2758: copy_from oid 14 from oid 5 current snap is 263 2021-07-31T01:54:38.795 INFO:tasks.rados.rados.0.smithi045.stdout:2751: finishing write tid 5 to smithi04569269-7 2021-07-31T01:54:38.795 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 7 v 1627 (ObjNum 833 snap 263 seq_num 833) dirty exists 2021-07-31T01:54:38.795 INFO:tasks.rados.rados.0.smithi045.stdout:2751: left oid 7 (ObjNum 833 snap 263 seq_num 833) 2021-07-31T01:54:38.796 INFO:tasks.rados.rados.0.smithi045.stdout:2751: done (6 left) 2021-07-31T01:54:38.796 INFO:tasks.rados.rados.0.smithi045.stdout:2759: rmattr oid 37 current snap is 263 2021-07-31T01:54:38.796 INFO:tasks.rados.rados.0.smithi045.stdout:2753: finishing write tid 1 to smithi04569269-38 2021-07-31T01:54:38.796 INFO:tasks.rados.rados.0.smithi045.stdout:2753: finishing write tid 2 to smithi04569269-38 2021-07-31T01:54:38.796 INFO:tasks.rados.rados.0.smithi045.stdout:2753: finishing write tid 3 to smithi04569269-38 2021-07-31T01:54:38.797 INFO:tasks.rados.rados.0.smithi045.stdout:2753: finishing write tid 4 to smithi04569269-38 2021-07-31T01:54:38.797 INFO:tasks.rados.rados.0.smithi045.stdout:2753: finishing write tid 5 to smithi04569269-38 2021-07-31T01:54:38.798 INFO:tasks.rados.rados.0.smithi045.stdout:2753: finishing write tid 6 to smithi04569269-38 2021-07-31T01:54:38.798 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 38 v 1632 (ObjNum 834 snap 263 seq_num 834) dirty exists 2021-07-31T01:54:38.798 INFO:tasks.rados.rados.0.smithi045.stdout:2753: left oid 38 (ObjNum 834 snap 263 seq_num 834) 2021-07-31T01:54:38.799 INFO:tasks.rados.rados.0.smithi045.stdout:2755: finishing write tid 1 to smithi04569269-36 2021-07-31T01:54:38.799 INFO:tasks.rados.rados.0.smithi045.stdout:2755: finishing write tid 2 to smithi04569269-36 2021-07-31T01:54:38.799 INFO:tasks.rados.rados.0.smithi045.stdout:2754: expect (ObjNum 768 snap 235 seq_num 768) 2021-07-31T01:54:38.895 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T01:54:38.895 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T01:54:38.896 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T01:54:38.896 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T01:54:38.896 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T01:54:38.897 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T01:54:38.897 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T01:54:38.898 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T01:54:38.898 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T01:54:38.898 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T01:54:38.900 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T01:54:38.900 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T01:54:38.900 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T01:54:38.901 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T01:54:38.901 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T01:54:38.901 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T01:54:38.901 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T01:54:39.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:38 smithi154 conmon[34553]: cluster 2021-07-31T01:54:37.616008+0000 mon.a (mon.0 2021-07-31T01:54:39.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:38 smithi154 conmon[34553]: ) 1235 : cluster [DBG] osdmap e576: 8 total, 8 up, 8 in 2021-07-31T01:54:39.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:38 smithi154 conmon[34553]: cluster 2021-07-31T01:54:37.896559+0000 mgr.x (mgr.24542) 318 : cluster [DBG] pgmap v645: 25 pgs: 25 active+clean; 301 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:54:39.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:38 smithi045 conmon[33655]: cluster 2021-07-31T01:54:37.616008+0000 mon.a ( 2021-07-31T01:54:39.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:38 smithi045 conmon[33655]: mon.0) 1235 : cluster [DBG] osdmap e576: 8 total, 8 up, 8 in 2021-07-31T01:54:39.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:38 smithi045 conmon[33655]: cluster 2021-07-31T01:54:37.896559+0000 mgr.x (mgr.24542) 318 : cluster [DBG] pgmap v645: 25 pgs: 25 active+clean; 301 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:54:39.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:38 smithi045 conmon[39718]: cluster 2021-07-31T01:54:37.616008+0000 2021-07-31T01:54:39.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:38 smithi045 conmon[39718]: mon.a (mon.0) 1235 : cluster [DBG] osdmap e576: 8 total, 8 up, 8 in 2021-07-31T01:54:39.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:38 smithi045 conmon[39718]: cluster 2021-07-31T01 2021-07-31T01:54:39.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:38 smithi045 conmon[39718]: :54:37.896559+0000 mgr.x (mgr.24542) 318 : cluster [DBG] pgmap v645: 25 pgs: 25 active+clean; 301 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:54:39.653 INFO:tasks.rados.rados.0.smithi045.stdout:2757: finishing write tid 1 to smithi04569269-9 2021-07-31T01:54:39.653 INFO:tasks.rados.rados.0.smithi045.stdout:2755: finishing write tid 3 to smithi04569269-36 2021-07-31T01:54:39.653 INFO:tasks.rados.rados.0.smithi045.stdout:2755: finishing write tid 4 to smithi04569269-36 2021-07-31T01:54:39.654 INFO:tasks.rados.rados.0.smithi045.stdout:2753: done (6 left) 2021-07-31T01:54:39.654 INFO:tasks.rados.rados.0.smithi045.stdout:2754: done (5 left) 2021-07-31T01:54:39.654 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 50 current snap is 263 2021-07-31T01:54:39.654 INFO:tasks.rados.rados.0.smithi045.stdout:2755: finishing write tid 5 to smithi04569269-36 2021-07-31T01:54:39.655 INFO:tasks.rados.rados.0.smithi045.stdout:2760: seq_num 837 ranges {0=425984,425984=655360,1081344=729088,1810432=221184} 2021-07-31T01:54:39.655 INFO:tasks.rados.rados.0.smithi045.stdout:2760: writing smithi04569269-50 from 0 to 425984 tid 1 2021-07-31T01:54:39.658 INFO:tasks.rados.rados.0.smithi045.stdout:2760: writing smithi04569269-50 from 425984 to 1081344 tid 2 2021-07-31T01:54:39.663 INFO:tasks.rados.rados.0.smithi045.stdout:2760: writing smithi04569269-50 from 1081344 to 1810432 tid 3 2021-07-31T01:54:39.664 INFO:tasks.rados.rados.0.smithi045.stdout:2760: writing smithi04569269-50 from 1810432 to 2031616 tid 4 2021-07-31T01:54:39.664 INFO:tasks.rados.rados.0.smithi045.stdout:2761: snap_create 2021-07-31T01:54:39.665 INFO:tasks.rados.rados.0.smithi045.stdout:2756: expect (ObjNum 716 snap 222 seq_num 716) 2021-07-31T01:54:39.728 INFO:tasks.rados.rados.0.smithi045.stdout:2755: finishing write tid 6 to smithi04569269-36 2021-07-31T01:54:39.729 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 36 v 1748 (ObjNum 835 snap 263 seq_num 835) dirty exists 2021-07-31T01:54:39.729 INFO:tasks.rados.rados.0.smithi045.stdout:2755: left oid 36 (ObjNum 835 snap 263 seq_num 835) 2021-07-31T01:54:39.730 INFO:tasks.rados.rados.0.smithi045.stdout:2757: finishing write tid 2 to smithi04569269-9 2021-07-31T01:54:39.730 INFO:tasks.rados.rados.0.smithi045.stdout:2757: finishing write tid 3 to smithi04569269-9 2021-07-31T01:54:39.730 INFO:tasks.rados.rados.0.smithi045.stdout:2757: finishing write tid 4 to smithi04569269-9 2021-07-31T01:54:39.730 INFO:tasks.rados.rados.0.smithi045.stdout:2757: finishing write tid 5 to smithi04569269-9 2021-07-31T01:54:39.731 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 37 v 1626 (ObjNum 754 snap 233 seq_num 754) dirty exists 2021-07-31T01:54:39.731 INFO:tasks.rados.rados.0.smithi045.stdout:2757: finishing write tid 6 to smithi04569269-9 2021-07-31T01:54:39.731 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 9 v 1076 (ObjNum 836 snap 263 seq_num 836) dirty exists 2021-07-31T01:54:39.732 INFO:tasks.rados.rados.0.smithi045.stdout:2757: left oid 9 (ObjNum 836 snap 263 seq_num 836) 2021-07-31T01:54:39.732 INFO:tasks.rados.rados.0.smithi045.stdout:2758: finishing copy_from to smithi04569269-14 2021-07-31T01:54:39.733 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 14 v 1623 (ObjNum 817 snap 257 seq_num 817) dirty exists 2021-07-31T01:54:39.733 INFO:tasks.rados.rados.0.smithi045.stdout:2758: finishing copy_from racing read to smithi04569269-14 2021-07-31T01:54:39.734 INFO:tasks.rados.rados.0.smithi045.stdout:2760: finishing write tid 1 to smithi04569269-50 2021-07-31T01:54:39.734 INFO:tasks.rados.rados.0.smithi045.stdout:2760: finishing write tid 2 to smithi04569269-50 2021-07-31T01:54:39.734 INFO:tasks.rados.rados.0.smithi045.stdout:2760: finishing write tid 3 to smithi04569269-50 2021-07-31T01:54:39.735 INFO:tasks.rados.rados.0.smithi045.stdout:2760: finishing write tid 4 to smithi04569269-50 2021-07-31T01:54:39.735 INFO:tasks.rados.rados.0.smithi045.stdout:2760: finishing write tid 5 to smithi04569269-50 2021-07-31T01:54:39.736 INFO:tasks.rados.rados.0.smithi045.stdout:2760: finishing write tid 6 to smithi04569269-50 2021-07-31T01:54:39.736 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 50 v 1640 (ObjNum 837 snap 263 seq_num 837) dirty exists 2021-07-31T01:54:39.737 INFO:tasks.rados.rados.0.smithi045.stdout:2760: left oid 50 (ObjNum 837 snap 263 seq_num 837) 2021-07-31T01:54:40.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:39 smithi154 conmon[34553]: audit 2021-07-31T01:54:38.220352+0000 mgr.x (mgr.24542) 319 : audit [DBG] from='client.24871 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:54:40.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:39 smithi154 conmon[34553]: 2021-07-31T01:54:40.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:39 smithi154 conmon[34553]: audit 2021-07-31T01:54:38.509965+0000 mgr.x (mgr.24542) 320 : audit [DBG] from='client.14970 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:54:40.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:39 smithi154 conmon[34553]: cluster 2021-07-31T01:54:38.620101+0000 mon.a (mon.0 2021-07-31T01:54:40.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:39 smithi154 conmon[34553]: ) 1236 : cluster [DBG] osdmap e577: 8 total, 8 up, 8 in 2021-07-31T01:54:40.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:39 smithi154 conmon[34553]: audit 2021-07 2021-07-31T01:54:40.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:39 smithi154 conmon[34553]: -31T01:54:38.894743+0000 mon.c (mon.1) 33 : 2021-07-31T01:54:40.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:39 smithi154 conmon[34553]: audit [DBG] from='client.? 172.21.15.45:0/3287673211' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T01:54:40.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:39 smithi045 conmon[33655]: audit 2021-07-31T01:54:38.220352+0000 mgr.x (mgr.24542) 2021-07-31T01:54:40.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:39 smithi045 conmon[33655]: 319 : audit [DBG] from='client.24871 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:54:40.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:39 smithi045 conmon[33655]: audit 2021-07-31T01:54:38.509965+0000 mgr.x (mgr.24542) 320 : audit [DBG] from='client.14970 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:54:40.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:39 smithi045 conmon[33655]: cluster 2021-07-31T01:54:38. 2021-07-31T01:54:40.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:39 smithi045 conmon[33655]: 620101+0000 mon.a (mon.0) 1236 : cluster [DBG] 2021-07-31T01:54:40.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:39 smithi045 conmon[33655]: osdmap e577: 8 total, 8 up, 8 in 2021-07-31T01:54:40.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:39 smithi045 conmon[33655]: audit 2021-07-31T 2021-07-31T01:54:40.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:39 smithi045 conmon[33655]: 01:54:38.894743+0000 mon.c 2021-07-31T01:54:40.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:39 smithi045 conmon[33655]: (mon.1) 33 : audit [DBG] from='client.? 172.21.15.45:0/3287673211' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T01:54:40.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:39 smithi045 conmon[33655]: 2021-07-31T01:54:40.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:39 smithi045 conmon[39718]: audit 2021-07-31T01:54:38.220352+0000 2021-07-31T01:54:40.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:39 smithi045 conmon[39718]: mgr.x (mgr.24542) 319 : audit [DBG] from='client.24871 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:54:40.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:39 smithi045 conmon[39718]: audit 2021 2021-07-31T01:54:40.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:39 smithi045 conmon[39718]: -07-31T01:54:38.509965+0000 mgr.x (mgr.24542) 320 : audit 2021-07-31T01:54:40.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:39 smithi045 conmon[39718]: [DBG] from='client.14970 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:54:40.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:39 smithi045 conmon[39718]: cluster 2021-07-31T01:54:38.620101 2021-07-31T01:54:40.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:39 smithi045 conmon[39718]: +0000 mon.a (mon.0) 1236 : cluster [DBG] osdmap e577: 8 total, 8 up, 8 in 2021-07-31T01:54:40.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:39 smithi045 conmon[39718]: 2021-07-31T01:54:40.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:39 smithi045 conmon[39718]: audit 2021- 2021-07-31T01:54:40.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:39 smithi045 conmon[39718]: 07-31T01:54:38.894743+0000 mon.c (mon.1) 33 2021-07-31T01:54:40.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:39 smithi045 conmon[39718]: : audit [DBG] from='client.? 172.21.15.45:0/3287673211' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T01:54:40.630 INFO:tasks.rados.rados.0.smithi045.stdout:2755: done (6 left) 2021-07-31T01:54:40.630 INFO:tasks.rados.rados.0.smithi045.stdout:2756: done (5 left) 2021-07-31T01:54:40.631 INFO:tasks.rados.rados.0.smithi045.stdout:2757: done (4 left) 2021-07-31T01:54:40.631 INFO:tasks.rados.rados.0.smithi045.stdout:2758: done (3 left) 2021-07-31T01:54:40.631 INFO:tasks.rados.rados.0.smithi045.stdout:2759: done (2 left) 2021-07-31T01:54:40.631 INFO:tasks.rados.rados.0.smithi045.stdout:2760: done (1 left) 2021-07-31T01:54:40.632 INFO:tasks.rados.rados.0.smithi045.stdout:2761: done (0 left) 2021-07-31T01:54:40.632 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 39 current snap is 264 2021-07-31T01:54:40.632 INFO:tasks.rados.rados.0.smithi045.stdout:2762: seq_num 838 ranges {1122304=442368,1564672=540672,2105344=532480,2637824=442368,3080192=90112} 2021-07-31T01:54:40.637 INFO:tasks.rados.rados.0.smithi045.stdout:2762: writing smithi04569269-39 from 1122304 to 1564672 tid 1 2021-07-31T01:54:40.640 INFO:tasks.rados.rados.0.smithi045.stdout:2762: writing smithi04569269-39 from 1564672 to 2105344 tid 2 2021-07-31T01:54:40.644 INFO:tasks.rados.rados.0.smithi045.stdout:2762: writing smithi04569269-39 from 2105344 to 2637824 tid 3 2021-07-31T01:54:40.645 INFO:tasks.rados.rados.0.smithi045.stdout:2762: writing smithi04569269-39 from 2637824 to 3080192 tid 4 2021-07-31T01:54:40.646 INFO:tasks.rados.rados.0.smithi045.stdout:2762: writing smithi04569269-39 from 3080192 to 3170304 tid 5 2021-07-31T01:54:40.647 INFO:tasks.rados.rados.0.smithi045.stdout:2763: read oid 17 snap -1 2021-07-31T01:54:40.647 INFO:tasks.rados.rados.0.smithi045.stdout:2763: expect (ObjNum 829 snap 262 seq_num 829) 2021-07-31T01:54:40.647 INFO:tasks.rados.rados.0.smithi045.stdout:2764: snap_create 2021-07-31T01:54:40.647 INFO:tasks.rados.rados.0.smithi045.stdout:2762: finishing write tid 1 to smithi04569269-39 2021-07-31T01:54:40.647 INFO:tasks.rados.rados.0.smithi045.stdout:2762: finishing write tid 2 to smithi04569269-39 2021-07-31T01:54:40.649 INFO:tasks.rados.rados.0.smithi045.stdout:2762: finishing write tid 3 to smithi04569269-39 2021-07-31T01:54:40.650 INFO:tasks.rados.rados.0.smithi045.stdout:2762: finishing write tid 4 to smithi04569269-39 2021-07-31T01:54:40.651 INFO:tasks.rados.rados.0.smithi045.stdout:2762: finishing write tid 5 to smithi04569269-39 2021-07-31T01:54:40.653 INFO:tasks.rados.rados.0.smithi045.stdout:2762: finishing write tid 6 to smithi04569269-39 2021-07-31T01:54:40.654 INFO:tasks.rados.rados.0.smithi045.stdout:2762: finishing write tid 7 to smithi04569269-39 2021-07-31T01:54:40.654 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 39 v 1638 (ObjNum 838 snap 264 seq_num 838) dirty exists 2021-07-31T01:54:40.655 INFO:tasks.rados.rados.0.smithi045.stdout:2762: left oid 39 (ObjNum 838 snap 264 seq_num 838) 2021-07-31T01:54:40.700 INFO:tasks.rados.rados.0.smithi045.stdout:2763: expect (ObjNum 829 snap 262 seq_num 829) 2021-07-31T01:54:41.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:40 smithi154 conmon[34553]: cluster 2021-07-31T01:54:39.897166+0000 mgr.x (mgr.24542) 321 : cluster 2021-07-31T01:54:41.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:40 smithi154 conmon[34553]: [DBG] pgmap v647: 25 pgs: 25 active+clean; 328 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 17 MiB/s rd, 6.4 MiB/s wr, 17 op/s 2021-07-31T01:54:41.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:40 smithi045 conmon[39718]: cluster 2021-07-31T01:54:39.897166+0000 mgr.x ( 2021-07-31T01:54:41.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:40 smithi045 conmon[39718]: mgr.24542) 321 : cluster [DBG] pgmap v647: 25 pgs: 25 active+clean; 328 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 17 MiB/s rd, 6.4 MiB/s wr, 17 op/s 2021-07-31T01:54:41.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:40 smithi045 conmon[33655]: cluster 2021-07-31T01:54:39.897166+0000 mgr.x (mgr.24542) 321 : cluster 2021-07-31T01:54:41.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:40 smithi045 conmon[33655]: [DBG] pgmap v647: 25 pgs: 25 active+clean; 328 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 17 MiB/s rd, 6.4 MiB/s wr, 17 op/s 2021-07-31T01:54:41.609 INFO:tasks.rados.rados.0.smithi045.stdout:2762: done (2 left) 2021-07-31T01:54:41.609 INFO:tasks.rados.rados.0.smithi045.stdout:2763: done (1 left) 2021-07-31T01:54:41.609 INFO:tasks.rados.rados.0.smithi045.stdout:2764: done (0 left) 2021-07-31T01:54:41.610 INFO:tasks.rados.rados.0.smithi045.stdout:2765: rmattr oid 10 current snap is 265 2021-07-31T01:54:41.610 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 15 current snap is 265 2021-07-31T01:54:41.610 INFO:tasks.rados.rados.0.smithi045.stdout:2766: seq_num 839 ranges {1122304=458752,1581056=425984,2007040=688128,2695168=491520} 2021-07-31T01:54:41.617 INFO:tasks.rados.rados.0.smithi045.stdout:2766: writing smithi04569269-15 from 1122304 to 1581056 tid 1 2021-07-31T01:54:41.619 INFO:tasks.rados.rados.0.smithi045.stdout:2766: writing smithi04569269-15 from 1581056 to 2007040 tid 2 2021-07-31T01:54:41.624 INFO:tasks.rados.rados.0.smithi045.stdout:2766: writing smithi04569269-15 from 2007040 to 2695168 tid 3 2021-07-31T01:54:41.627 INFO:tasks.rados.rados.0.smithi045.stdout:2766: writing smithi04569269-15 from 2695168 to 3186688 tid 4 2021-07-31T01:54:41.628 INFO:tasks.rados.rados.0.smithi045.stdout:2767: read oid 32 snap -1 2021-07-31T01:54:41.628 INFO:tasks.rados.rados.0.smithi045.stdout:2767: expect (ObjNum 754 snap 233 seq_num 754) 2021-07-31T01:54:41.628 INFO:tasks.rados.rados.0.smithi045.stdout:2768: copy_from oid 34 from oid 22 current snap is 265 2021-07-31T01:54:41.628 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 10 v 1641 (ObjNum 768 snap 235 seq_num 768) dirty exists 2021-07-31T01:54:41.629 INFO:tasks.rados.rados.0.smithi045.stdout:2766: finishing write tid 1 to smithi04569269-15 2021-07-31T01:54:41.629 INFO:tasks.rados.rados.0.smithi045.stdout:2766: finishing write tid 2 to smithi04569269-15 2021-07-31T01:54:41.629 INFO:tasks.rados.rados.0.smithi045.stdout:2766: finishing write tid 3 to smithi04569269-15 2021-07-31T01:54:41.629 INFO:tasks.rados.rados.0.smithi045.stdout:2765: done (3 left) 2021-07-31T01:54:41.629 INFO:tasks.rados.rados.0.smithi045.stdout:2769: copy_from oid 44 from oid 42 current snap is 265 2021-07-31T01:54:41.630 INFO:tasks.rados.rados.0.smithi045.stdout:2770: snap_remove snap 257 2021-07-31T01:54:42.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:41 smithi154 conmon[34553]: cluster 2021-07-31T01:54:42.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:41 smithi154 conmon[34553]: 2021-07-31T01:54:42.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:41 smithi154 conmon[34553]: 2021 2021-07-31T01:54:42.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:41 smithi154 conmon[34553]: - 2021-07-31T01:54:42.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:41 smithi154 conmon[34553]: 07 2021-07-31T01:54:42.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:41 smithi154 conmon[34553]: - 2021-07-31T01:54:42.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:41 smithi154 conmon[34553]: 31 2021-07-31T01:54:42.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:41 smithi154 conmon[34553]: T 2021-07-31T01:54:42.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:41 smithi154 conmon[34553]: 01 2021-07-31T01:54:42.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:41 smithi154 conmon[34553]: : 2021-07-31T01:54:42.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:41 smithi154 conmon[34553]: 54 2021-07-31T01:54:42.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:41 smithi154 conmon[34553]: : 2021-07-31T01:54:42.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:41 smithi154 conmon[34553]: 40.627487+0000 mon.a 2021-07-31T01:54:42.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:41 smithi154 conmon[34553]: (mon.0) 1237 : cluster [DBG] osdmap e578: 8 total, 8 up, 8 in 2021-07-31T01:54:42.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:41 smithi154 conmon[34553]: cluster 2021-07-31T01:54:41.608245+0000 mon.a (mon.0) 1238 : cluster [DBG] 2021-07-31T01:54:42.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:41 smithi154 conmon[34553]: osdmap e579: 8 total, 8 up, 8 in 2021-07-31T01:54:42.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:41 smithi045 conmon[33655]: cluster 2021-07-31T01:54:40.627487+0000 2021-07-31T01:54:42.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:41 smithi045 conmon[33655]: mon.a (mon.0) 1237 : cluster [DBG] osdmap e578: 8 total, 8 up, 8 in 2021-07-31T01:54:42.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:41 smithi045 conmon[33655]: cluster 2021-07-31T01:54:41.608245+0000 mon.a (mon.0) 1238 : cluster 2021-07-31T01:54:42.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:41 smithi045 conmon[33655]: [DBG] osdmap e579: 8 total, 8 up, 8 in 2021-07-31T01:54:42.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:41 smithi045 conmon[39718]: cluster 2021-07-31T01:54:42.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:41 smithi045 conmon[39718]: 2021-07-31T01 2021-07-31T01:54:42.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:41 smithi045 conmon[39718]: :54:40.627487+0000 mon.a (mon.0) 1237 : cluster [DBG] osdmap e578: 8 total, 8 up, 8 in 2021-07-31T01:54:42.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:41 smithi045 conmon[39718]: cluster 2021-07-31T01:54:42.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:41 smithi045 conmon[39718]: 2021-07-31T01:54:41 2021-07-31T01:54:42.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:41 smithi045 conmon[39718]: .608245+0000 mon.a (mon.0) 1238 2021-07-31T01:54:42.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:41 smithi045 conmon[39718]: : cluster [DBG] osdmap e579: 8 total, 8 up, 8 in 2021-07-31T01:54:42.639 INFO:tasks.rados.rados.0.smithi045.stdout:2770: done (4 left) 2021-07-31T01:54:42.639 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 9 current snap is 265 2021-07-31T01:54:42.639 INFO:tasks.rados.rados.0.smithi045.stdout:2766: finishing write tid 4 to smithi04569269-15 2021-07-31T01:54:42.640 INFO:tasks.rados.rados.0.smithi045.stdout:2771: seq_num 840 ranges {2015232=475136,2490368=712704,3203072=499712,3702784=393216} 2021-07-31T01:54:42.655 INFO:tasks.rados.rados.0.smithi045.stdout:2771: writing smithi04569269-9 from 2015232 to 2490368 tid 1 2021-07-31T01:54:42.657 INFO:tasks.rados.rados.0.smithi045.stdout:2771: writing smithi04569269-9 from 2490368 to 3203072 tid 2 2021-07-31T01:54:42.660 INFO:tasks.rados.rados.0.smithi045.stdout:2771: writing smithi04569269-9 from 3203072 to 3702784 tid 3 2021-07-31T01:54:42.663 INFO:tasks.rados.rados.0.smithi045.stdout:2771: writing smithi04569269-9 from 3702784 to 4096000 tid 4 2021-07-31T01:54:42.663 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 14 current snap is 265 2021-07-31T01:54:42.663 INFO:tasks.rados.rados.0.smithi045.stdout:2766: finishing write tid 5 to smithi04569269-15 2021-07-31T01:54:42.664 INFO:tasks.rados.rados.0.smithi045.stdout:2766: finishing write tid 6 to smithi04569269-15 2021-07-31T01:54:42.664 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 15 v 1631 (ObjNum 839 snap 265 seq_num 839) dirty exists 2021-07-31T01:54:42.664 INFO:tasks.rados.rados.0.smithi045.stdout:2766: left oid 15 (ObjNum 839 snap 265 seq_num 839) 2021-07-31T01:54:42.664 INFO:tasks.rados.rados.0.smithi045.stdout:2768: finishing copy_from to smithi04569269-34 2021-07-31T01:54:42.665 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 34 v 1576 (ObjNum 747 snap 230 seq_num 747) dirty exists 2021-07-31T01:54:42.665 INFO:tasks.rados.rados.0.smithi045.stdout:2768: finishing copy_from racing read to smithi04569269-34 2021-07-31T01:54:42.665 INFO:tasks.rados.rados.0.smithi045.stdout:2769: finishing copy_from to smithi04569269-44 2021-07-31T01:54:42.666 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 44 v 1567 (ObjNum 704 snap 219 seq_num 704) dirty exists 2021-07-31T01:54:42.666 INFO:tasks.rados.rados.0.smithi045.stdout:2769: finishing copy_from racing read to smithi04569269-44 2021-07-31T01:54:42.666 INFO:tasks.rados.rados.0.smithi045.stdout:2767: expect (ObjNum 754 snap 233 seq_num 754) 2021-07-31T01:54:43.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:42 smithi154 conmon[34553]: cluster 2021-07-31T01:54:41.897573+0000 mgr.x (mgr.24542 2021-07-31T01:54:43.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:42 smithi154 conmon[34553]: ) 322 : cluster [DBG] pgmap v650: 25 pgs: 25 active+clean; 328 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 16 MiB/s rd, 6.3 MiB/s wr, 16 op/s 2021-07-31T01:54:43.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:42 smithi045 conmon[33655]: cluster 2021-07-31T01:54:41.897573+0000 mgr.x (mgr.24542) 322 2021-07-31T01:54:43.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:42 smithi045 conmon[33655]: : cluster [DBG] pgmap v650: 25 pgs: 25 active+clean; 328 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 16 MiB/s rd, 6.3 MiB/s wr, 16 op/s 2021-07-31T01:54:43.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:42 smithi045 conmon[39718]: cluster 2021-07-31T01:54:41.897573+0000 mgr.x (mgr.24542) 2021-07-31T01:54:43.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:42 smithi045 conmon[39718]: 322 : cluster [DBG] pgmap v650: 25 pgs: 25 active+clean; 328 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 16 MiB/s rd, 6.3 MiB/s wr, 16 op/s 2021-07-31T01:54:43.373 INFO:tasks.rados.rados.0.smithi045.stdout:2771: finishing write tid 1 to smithi04569269-9 2021-07-31T01:54:43.373 INFO:tasks.rados.rados.0.smithi045.stdout:2771: finishing write tid 2 to smithi04569269-9 2021-07-31T01:54:43.373 INFO:tasks.rados.rados.0.smithi045.stdout:2771: finishing write tid 3 to smithi04569269-9 2021-07-31T01:54:43.373 INFO:tasks.rados.rados.0.smithi045.stdout:2771: finishing write tid 4 to smithi04569269-9 2021-07-31T01:54:43.374 INFO:tasks.rados.rados.0.smithi045.stdout:2771: finishing write tid 5 to smithi04569269-9 2021-07-31T01:54:43.374 INFO:tasks.rados.rados.0.smithi045.stdout:2771: finishing write tid 6 to smithi04569269-9 2021-07-31T01:54:43.374 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 9 v 1081 (ObjNum 840 snap 265 seq_num 840) dirty exists 2021-07-31T01:54:43.374 INFO:tasks.rados.rados.0.smithi045.stdout:2771: left oid 9 (ObjNum 840 snap 265 seq_num 840) 2021-07-31T01:54:43.375 INFO:tasks.rados.rados.0.smithi045.stdout:2772: seq_num 841 ranges {12247040=491520,12738560=598016,13336576=704512,14041088=303104} 2021-07-31T01:54:43.442 INFO:tasks.rados.rados.0.smithi045.stdout:2772: writing smithi04569269-14 from 12247040 to 12738560 tid 1 2021-07-31T01:54:43.445 INFO:tasks.rados.rados.0.smithi045.stdout:2772: writing smithi04569269-14 from 12738560 to 13336576 tid 2 2021-07-31T01:54:43.449 INFO:tasks.rados.rados.0.smithi045.stdout:2772: writing smithi04569269-14 from 13336576 to 14041088 tid 3 2021-07-31T01:54:43.451 INFO:tasks.rados.rados.0.smithi045.stdout:2772: writing smithi04569269-14 from 14041088 to 14344192 tid 4 2021-07-31T01:54:43.451 INFO:tasks.rados.rados.0.smithi045.stdout:2766: done (5 left) 2021-07-31T01:54:43.452 INFO:tasks.rados.rados.0.smithi045.stdout:2767: done (4 left) 2021-07-31T01:54:43.452 INFO:tasks.rados.rados.0.smithi045.stdout:2768: done (3 left) 2021-07-31T01:54:43.452 INFO:tasks.rados.rados.0.smithi045.stdout:2769: done (2 left) 2021-07-31T01:54:43.452 INFO:tasks.rados.rados.0.smithi045.stdout:2771: done (1 left) 2021-07-31T01:54:43.452 INFO:tasks.rados.rados.0.smithi045.stdout:2773: setattr oid 17 current snap is 265 2021-07-31T01:54:43.453 INFO:tasks.rados.rados.0.smithi045.stdout:2772: finishing write tid 1 to smithi04569269-14 2021-07-31T01:54:43.453 INFO:tasks.rados.rados.0.smithi045.stdout:2772: finishing write tid 2 to smithi04569269-14 2021-07-31T01:54:43.453 INFO:tasks.rados.rados.0.smithi045.stdout:2774: snap_remove snap 260 2021-07-31T01:54:43.636 INFO:tasks.rados.rados.0.smithi045.stdout:2774: done (2 left) 2021-07-31T01:54:43.637 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 38 current snap is 265 2021-07-31T01:54:43.637 INFO:tasks.rados.rados.0.smithi045.stdout:2772: finishing write tid 3 to smithi04569269-14 2021-07-31T01:54:43.637 INFO:tasks.rados.rados.0.smithi045.stdout:2775: seq_num 842 ranges {1982464=507904,2490368=475136,2965504=516096,3481600=499712,3981312=114688} 2021-07-31T01:54:43.651 INFO:tasks.rados.rados.0.smithi045.stdout:2775: writing smithi04569269-38 from 1982464 to 2490368 tid 1 2021-07-31T01:54:43.653 INFO:tasks.rados.rados.0.smithi045.stdout:2775: writing smithi04569269-38 from 2490368 to 2965504 tid 2 2021-07-31T01:54:43.656 INFO:tasks.rados.rados.0.smithi045.stdout:2775: writing smithi04569269-38 from 2965504 to 3481600 tid 3 2021-07-31T01:54:43.659 INFO:tasks.rados.rados.0.smithi045.stdout:2775: writing smithi04569269-38 from 3481600 to 3981312 tid 4 2021-07-31T01:54:43.660 INFO:tasks.rados.rados.0.smithi045.stdout:2775: writing smithi04569269-38 from 3981312 to 4096000 tid 5 2021-07-31T01:54:43.661 INFO:tasks.rados.rados.0.smithi045.stdout:2776: read oid 10 snap -1 2021-07-31T01:54:43.661 INFO:tasks.rados.rados.0.smithi045.stdout:2776: expect (ObjNum 768 snap 235 seq_num 768) 2021-07-31T01:54:43.661 INFO:tasks.rados.rados.0.smithi045.stdout:2777: delete oid 16 current snap is 265 2021-07-31T01:54:43.661 INFO:tasks.rados.rados.0.smithi045.stdout:2772: finishing write tid 4 to smithi04569269-14 2021-07-31T01:54:43.662 INFO:tasks.rados.rados.0.smithi045.stdout:2772: finishing write tid 5 to smithi04569269-14 2021-07-31T01:54:43.662 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 17 v 1642 (ObjNum 829 snap 262 seq_num 829) dirty exists 2021-07-31T01:54:43.662 INFO:tasks.rados.rados.0.smithi045.stdout:2772: finishing write tid 6 to smithi04569269-14 2021-07-31T01:54:43.662 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 14 v 1628 (ObjNum 841 snap 265 seq_num 841) dirty exists 2021-07-31T01:54:43.663 INFO:tasks.rados.rados.0.smithi045.stdout:2772: left oid 14 (ObjNum 841 snap 265 seq_num 841) 2021-07-31T01:54:43.663 INFO:tasks.rados.rados.0.smithi045.stdout:2775: finishing write tid 1 to smithi04569269-38 2021-07-31T01:54:43.663 INFO:tasks.rados.rados.0.smithi045.stdout:2775: finishing write tid 2 to smithi04569269-38 2021-07-31T01:54:43.663 INFO:tasks.rados.rados.0.smithi045.stdout:2775: finishing write tid 3 to smithi04569269-38 2021-07-31T01:54:43.664 INFO:tasks.rados.rados.0.smithi045.stdout:2772: done (4 left) 2021-07-31T01:54:43.664 INFO:tasks.rados.rados.0.smithi045.stdout:2773: done (3 left) 2021-07-31T01:54:43.664 INFO:tasks.rados.rados.0.smithi045.stdout:2777: done (2 left) 2021-07-31T01:54:43.665 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 37 current snap is 265 2021-07-31T01:54:43.665 INFO:tasks.rados.rados.0.smithi045.stdout:2778: seq_num 844 ranges {9011200=540672,9551872=696320,10248192=475136,10723328=434176} 2021-07-31T01:54:43.718 INFO:tasks.rados.rados.0.smithi045.stdout:2778: writing smithi04569269-37 from 9011200 to 9551872 tid 1 2021-07-31T01:54:43.722 INFO:tasks.rados.rados.0.smithi045.stdout:2778: writing smithi04569269-37 from 9551872 to 10248192 tid 2 2021-07-31T01:54:43.725 INFO:tasks.rados.rados.0.smithi045.stdout:2778: writing smithi04569269-37 from 10248192 to 10723328 tid 3 2021-07-31T01:54:43.728 INFO:tasks.rados.rados.0.smithi045.stdout:2778: writing smithi04569269-37 from 10723328 to 11157504 tid 4 2021-07-31T01:54:43.728 INFO:tasks.rados.rados.0.smithi045.stdout:2779: read oid 23 snap -1 2021-07-31T01:54:43.728 INFO:tasks.rados.rados.0.smithi045.stdout:2779: expect deleted 2021-07-31T01:54:43.729 INFO:tasks.rados.rados.0.smithi045.stdout:2780: snap_remove snap 246 2021-07-31T01:54:43.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:43 smithi045 conmon[33655]: cluster 2021 2021-07-31T01:54:43.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:43 smithi045 conmon[33655]: -07-31T01:54:42.635480+0000 mon.a (mon.0) 1239 : cluster [DBG] osdmap e580: 8 total, 8 up, 8 in 2021-07-31T01:54:43.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:43 smithi045 conmon[39718]: cluster 2021-07-31T01:54:43.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:43 smithi045 conmon[39718]: 2021-07-31T01:54:43.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:43 smithi045 conmon[39718]: 2021-07-31T01:54: 2021-07-31T01:54:43.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:43 smithi045 conmon[39718]: 42.635480+0000 mon.a (mon.0) 1239 : cluster [DBG] osdmap e580: 8 total, 8 up, 8 in 2021-07-31T01:54:43.778 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:54:43 smithi045 conmon[61210]: level=error ts=2021-07-31T01:54:43.522Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:54:43.778 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:54:43 smithi045 conmon[61210]: level=error ts=2021-07-31T01:54:43.522Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:54:43.778 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:54:43 smithi045 conmon[61210]: level=error ts=2021-07-31T01:54:43.522Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T01:54:44.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:43 smithi154 conmon[34553]: cluster 2021-07-31T01:54:42.635480+0000 mon.a (mon.0) 1239 : cluster [DBG] osdmap e580: 8 total, 8 up, 8 in 2021-07-31T01:54:44.655 INFO:tasks.rados.rados.0.smithi045.stdout:2780: done (4 left) 2021-07-31T01:54:44.655 INFO:tasks.rados.rados.0.smithi045.stdout:2781: read oid 1 snap -1 2021-07-31T01:54:44.655 INFO:tasks.rados.rados.0.smithi045.stdout:2781: expect (ObjNum 811 snap 254 seq_num 811) 2021-07-31T01:54:44.656 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 40 current snap is 265 2021-07-31T01:54:44.656 INFO:tasks.rados.rados.0.smithi045.stdout:2782: seq_num 845 ranges {2367488=557056,2924544=581632,3506176=688128,4194304=335872} 2021-07-31T01:54:44.669 INFO:tasks.rados.rados.0.smithi045.stdout:2782: writing smithi04569269-40 from 2367488 to 2924544 tid 1 2021-07-31T01:54:44.672 INFO:tasks.rados.rados.0.smithi045.stdout:2782: writing smithi04569269-40 from 2924544 to 3506176 tid 2 2021-07-31T01:54:44.680 INFO:tasks.rados.rados.0.smithi045.stdout:2782: writing smithi04569269-40 from 3506176 to 4194304 tid 3 2021-07-31T01:54:44.681 INFO:tasks.rados.rados.0.smithi045.stdout:2782: writing smithi04569269-40 from 4194304 to 4530176 tid 4 2021-07-31T01:54:44.681 INFO:tasks.rados.rados.0.smithi045.stdout:2783: copy_from oid 20 from oid 19 current snap is 265 2021-07-31T01:54:44.681 INFO:tasks.rados.rados.0.smithi045.stdout:2775: finishing write tid 4 to smithi04569269-38 2021-07-31T01:54:44.681 INFO:tasks.rados.rados.0.smithi045.stdout:2775: finishing write tid 5 to smithi04569269-38 2021-07-31T01:54:44.682 INFO:tasks.rados.rados.0.smithi045.stdout:2775: finishing write tid 6 to smithi04569269-38 2021-07-31T01:54:44.682 INFO:tasks.rados.rados.0.smithi045.stdout:2775: finishing write tid 7 to smithi04569269-38 2021-07-31T01:54:44.682 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 38 v 1644 (ObjNum 842 snap 265 seq_num 842) dirty exists 2021-07-31T01:54:44.682 INFO:tasks.rados.rados.0.smithi045.stdout:2775: left oid 38 (ObjNum 842 snap 265 seq_num 842) 2021-07-31T01:54:44.683 INFO:tasks.rados.rados.0.smithi045.stdout:2776: expect (ObjNum 768 snap 235 seq_num 768) 2021-07-31T01:54:45.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:44 smithi154 conmon[34553]: cluster 2021-07-31T01:54:43.636084+0000 mon.a (mon.0) 1240 : cluster [DBG] osdmap e581: 8 total, 8 up, 8 in 2021-07-31T01:54:45.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:44 smithi154 conmon[34553]: cluster 2021-07-31T01:54:43.897990+0000 mgr.x (mgr.24542) 323 : cluster [DBG] pgmap v653: 25 pgs: 25 active+clean; 346 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 21 MiB/s rd, 4.8 MiB/s wr, 21 op/s 2021-07-31T01:54:45.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:44 smithi045 conmon[39718]: cluster 2021-07-31T01:54:43.636084+0000 mon.a ( 2021-07-31T01:54:45.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:44 smithi045 conmon[39718]: mon.0) 1240 : cluster [DBG] osdmap e581: 8 total, 8 up, 8 in 2021-07-31T01:54:45.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:44 smithi045 conmon[39718]: cluster 2021-07-31T01:54:43.897990+0000 mgr.x (mgr.24542 2021-07-31T01:54:45.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:44 smithi045 conmon[39718]: ) 323 : cluster [DBG] pgmap v653: 25 pgs: 25 active+clean; 346 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 21 MiB/s rd, 4.8 MiB/s wr, 21 op/s 2021-07-31T01:54:45.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:44 smithi045 conmon[33655]: cluster 2021-07-31T01:54:43.636084 2021-07-31T01:54:45.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:44 smithi045 conmon[33655]: +0000 mon.a (mon.0) 1240 : cluster [DBG] osdmap e581: 8 total, 8 up, 8 in 2021-07-31T01:54:45.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:44 smithi045 conmon[33655]: cluster 2021-07-31T01:54:43.897990+0000 mgr.x (mgr 2021-07-31T01:54:45.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:44 smithi045 conmon[33655]: .24542) 323 : cluster [DBG] pgmap v653: 25 pgs: 25 active+clean; 346 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 21 MiB/s rd, 4.8 MiB/s wr, 21 op/s 2021-07-31T01:54:45.526 INFO:tasks.rados.rados.0.smithi045.stdout:2778: finishing write tid 1 to smithi04569269-37 2021-07-31T01:54:45.527 INFO:tasks.rados.rados.0.smithi045.stdout:2778: finishing write tid 2 to smithi04569269-37 2021-07-31T01:54:45.527 INFO:tasks.rados.rados.0.smithi045.stdout:2778: finishing write tid 3 to smithi04569269-37 2021-07-31T01:54:45.527 INFO:tasks.rados.rados.0.smithi045.stdout:2778: finishing write tid 4 to smithi04569269-37 2021-07-31T01:54:45.527 INFO:tasks.rados.rados.0.smithi045.stdout:2778: finishing write tid 5 to smithi04569269-37 2021-07-31T01:54:45.528 INFO:tasks.rados.rados.0.smithi045.stdout:2778: finishing write tid 6 to smithi04569269-37 2021-07-31T01:54:45.528 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 37 v 1636 (ObjNum 844 snap 265 seq_num 844) dirty exists 2021-07-31T01:54:45.528 INFO:tasks.rados.rados.0.smithi045.stdout:2778: left oid 37 (ObjNum 844 snap 265 seq_num 844) 2021-07-31T01:54:45.528 INFO:tasks.rados.rados.0.smithi045.stdout:2775: done (6 left) 2021-07-31T01:54:45.529 INFO:tasks.rados.rados.0.smithi045.stdout:2776: done (5 left) 2021-07-31T01:54:45.529 INFO:tasks.rados.rados.0.smithi045.stdout:2778: done (4 left) 2021-07-31T01:54:45.529 INFO:tasks.rados.rados.0.smithi045.stdout:2779: done (3 left) 2021-07-31T01:54:45.529 INFO:tasks.rados.rados.0.smithi045.stdout:2784: read oid 24 snap -1 2021-07-31T01:54:45.529 INFO:tasks.rados.rados.0.smithi045.stdout:2784: expect (ObjNum 783 snap 239 seq_num 783) 2021-07-31T01:54:45.530 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 29 current snap is 265 2021-07-31T01:54:45.530 INFO:tasks.rados.rados.0.smithi045.stdout:2781: expect (ObjNum 811 snap 254 seq_num 811) 2021-07-31T01:54:45.764 INFO:tasks.rados.rados.0.smithi045.stdout:2783: finishing copy_from to smithi04569269-20 2021-07-31T01:54:45.765 INFO:tasks.rados.rados.0.smithi045.stdout:2783: got expected ENOENT (src dne) 2021-07-31T01:54:45.765 INFO:tasks.rados.rados.0.smithi045.stdout:2783: finishing copy_from racing read to smithi04569269-20 2021-07-31T01:54:45.765 INFO:tasks.rados.rados.0.smithi045.stdout:2782: finishing write tid 1 to smithi04569269-40 2021-07-31T01:54:45.765 INFO:tasks.rados.rados.0.smithi045.stdout:2782: finishing write tid 2 to smithi04569269-40 2021-07-31T01:54:45.766 INFO:tasks.rados.rados.0.smithi045.stdout:2782: finishing write tid 3 to smithi04569269-40 2021-07-31T01:54:45.766 INFO:tasks.rados.rados.0.smithi045.stdout:2782: finishing write tid 4 to smithi04569269-40 2021-07-31T01:54:45.766 INFO:tasks.rados.rados.0.smithi045.stdout:2782: finishing write tid 5 to smithi04569269-40 2021-07-31T01:54:45.766 INFO:tasks.rados.rados.0.smithi045.stdout:2782: finishing write tid 6 to smithi04569269-40 2021-07-31T01:54:45.767 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 40 v 1621 (ObjNum 845 snap 265 seq_num 845) dirty exists 2021-07-31T01:54:45.767 INFO:tasks.rados.rados.0.smithi045.stdout:2782: left oid 40 (ObjNum 845 snap 265 seq_num 845) 2021-07-31T01:54:45.767 INFO:tasks.rados.rados.0.smithi045.stdout:2785: seq_num 846 ranges {0=573440,573440=466944,1040384=491520,1531904=548864,2080768=98304} 2021-07-31T01:54:45.768 INFO:tasks.rados.rados.0.smithi045.stdout:2785: writing smithi04569269-29 from 0 to 573440 tid 1 2021-07-31T01:54:45.768 INFO:tasks.rados.rados.0.smithi045.stdout:2785: writing smithi04569269-29 from 573440 to 1040384 tid 2 2021-07-31T01:54:45.770 INFO:tasks.rados.rados.0.smithi045.stdout:2785: writing smithi04569269-29 from 1040384 to 1531904 tid 3 2021-07-31T01:54:45.773 INFO:tasks.rados.rados.0.smithi045.stdout:2785: writing smithi04569269-29 from 1531904 to 2080768 tid 4 2021-07-31T01:54:45.775 INFO:tasks.rados.rados.0.smithi045.stdout:2785: writing smithi04569269-29 from 2080768 to 2179072 tid 5 2021-07-31T01:54:45.775 INFO:tasks.rados.rados.0.smithi045.stdout:2781: done (4 left) 2021-07-31T01:54:45.775 INFO:tasks.rados.rados.0.smithi045.stdout:2782: done (3 left) 2021-07-31T01:54:45.775 INFO:tasks.rados.rados.0.smithi045.stdout:2783: done (2 left) 2021-07-31T01:54:45.775 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 10 current snap is 265 2021-07-31T01:54:45.776 INFO:tasks.rados.rados.0.smithi045.stdout:2784: expect (ObjNum 783 snap 239 seq_num 783) 2021-07-31T01:54:45.811 INFO:tasks.rados.rados.0.smithi045.stdout:2785: finishing write tid 1 to smithi04569269-29 2021-07-31T01:54:45.811 INFO:tasks.rados.rados.0.smithi045.stdout:2785: finishing write tid 2 to smithi04569269-29 2021-07-31T01:54:45.812 INFO:tasks.rados.rados.0.smithi045.stdout:2785: finishing write tid 3 to smithi04569269-29 2021-07-31T01:54:45.812 INFO:tasks.rados.rados.0.smithi045.stdout:2785: finishing write tid 4 to smithi04569269-29 2021-07-31T01:54:45.813 INFO:tasks.rados.rados.0.smithi045.stdout:2785: finishing write tid 5 to smithi04569269-29 2021-07-31T01:54:45.813 INFO:tasks.rados.rados.0.smithi045.stdout:2785: finishing write tid 6 to smithi04569269-29 2021-07-31T01:54:45.814 INFO:tasks.rados.rados.0.smithi045.stdout:2785: finishing write tid 7 to smithi04569269-29 2021-07-31T01:54:45.814 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 29 v 1416 (ObjNum 846 snap 265 seq_num 846) dirty exists 2021-07-31T01:54:45.814 INFO:tasks.rados.rados.0.smithi045.stdout:2785: left oid 29 (ObjNum 846 snap 265 seq_num 846) 2021-07-31T01:54:45.815 INFO:tasks.rados.rados.0.smithi045.stdout:2786: seq_num 847 ranges {10043392=589824,10633216=753664,11386880=655360,12042240=196608} 2021-07-31T01:54:45.861 INFO:tasks.rados.rados.0.smithi045.stdout:2786: writing smithi04569269-10 from 10043392 to 10633216 tid 1 2021-07-31T01:54:45.865 INFO:tasks.rados.rados.0.smithi045.stdout:2786: writing smithi04569269-10 from 10633216 to 11386880 tid 2 2021-07-31T01:54:45.869 INFO:tasks.rados.rados.0.smithi045.stdout:2786: writing smithi04569269-10 from 11386880 to 12042240 tid 3 2021-07-31T01:54:45.870 INFO:tasks.rados.rados.0.smithi045.stdout:2786: writing smithi04569269-10 from 12042240 to 12238848 tid 4 2021-07-31T01:54:45.871 INFO:tasks.rados.rados.0.smithi045.stdout:2784: done (2 left) 2021-07-31T01:54:45.871 INFO:tasks.rados.rados.0.smithi045.stdout:2785: done (1 left) 2021-07-31T01:54:45.872 INFO:tasks.rados.rados.0.smithi045.stdout:2787: rollback oid 25 current snap is 265 2021-07-31T01:54:45.872 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 25 to 249 2021-07-31T01:54:45.872 INFO:tasks.rados.rados.0.smithi045.stdout:2786: finishing write tid 1 to smithi04569269-10 2021-07-31T01:54:45.872 INFO:tasks.rados.rados.0.smithi045.stdout:2786: finishing write tid 2 to smithi04569269-10 2021-07-31T01:54:45.872 INFO:tasks.rados.rados.0.smithi045.stdout:2788: read oid 30 snap -1 2021-07-31T01:54:45.873 INFO:tasks.rados.rados.0.smithi045.stdout:2788: expect (ObjNum 799 snap 246 seq_num 799) 2021-07-31T01:54:45.873 INFO:tasks.rados.rados.0.smithi045.stdout:2789: setattr oid 33 current snap is 265 2021-07-31T01:54:45.874 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 35 current snap is 265 2021-07-31T01:54:45.875 INFO:tasks.rados.rados.0.smithi045.stdout:2790: seq_num 848 ranges {0=606208,606208=630784,1236992=458752,1695744=516096} 2021-07-31T01:54:45.876 INFO:tasks.rados.rados.0.smithi045.stdout:2790: writing smithi04569269-35 from 0 to 606208 tid 1 2021-07-31T01:54:45.879 INFO:tasks.rados.rados.0.smithi045.stdout:2790: writing smithi04569269-35 from 606208 to 1236992 tid 2 2021-07-31T01:54:45.882 INFO:tasks.rados.rados.0.smithi045.stdout:2790: writing smithi04569269-35 from 1236992 to 1695744 tid 3 2021-07-31T01:54:45.887 INFO:tasks.rados.rados.0.smithi045.stdout:2790: writing smithi04569269-35 from 1695744 to 2211840 tid 4 2021-07-31T01:54:45.887 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 18 current snap is 265 2021-07-31T01:54:45.888 INFO:tasks.rados.rados.0.smithi045.stdout:2786: finishing write tid 3 to smithi04569269-10 2021-07-31T01:54:45.888 INFO:tasks.rados.rados.0.smithi045.stdout:2786: finishing write tid 4 to smithi04569269-10 2021-07-31T01:54:45.888 INFO:tasks.rados.rados.0.smithi045.stdout:2786: finishing write tid 5 to smithi04569269-10 2021-07-31T01:54:45.888 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 33 v 1645 (ObjNum 726 snap 226 seq_num 726) dirty exists 2021-07-31T01:54:45.889 INFO:tasks.rados.rados.0.smithi045.stdout:2786: finishing write tid 6 to smithi04569269-10 2021-07-31T01:54:45.889 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 10 v 1647 (ObjNum 847 snap 265 seq_num 847) dirty exists 2021-07-31T01:54:45.889 INFO:tasks.rados.rados.0.smithi045.stdout:2786: left oid 10 (ObjNum 847 snap 265 seq_num 847) 2021-07-31T01:54:45.889 INFO:tasks.rados.rados.0.smithi045.stdout:2787: finishing rollback tid 1 to smithi04569269-25 2021-07-31T01:54:45.889 INFO:tasks.rados.rados.0.smithi045.stdout:2787: finishing rollback tid 2 to smithi04569269-25 2021-07-31T01:54:45.890 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 25 v 1628 (ObjNum 788 snap 241 seq_num 788) dirty exists 2021-07-31T01:54:45.890 INFO:tasks.rados.rados.0.smithi045.stdout:2791: seq_num 849 ranges {11485184=622592,12107776=516096,12623872=663552,13287424=425984} 2021-07-31T01:54:45.952 INFO:tasks.rados.rados.0.smithi045.stdout:2791: writing smithi04569269-18 from 11485184 to 12107776 tid 1 2021-07-31T01:54:45.955 INFO:tasks.rados.rados.0.smithi045.stdout:2791: writing smithi04569269-18 from 12107776 to 12623872 tid 2 2021-07-31T01:54:45.959 INFO:tasks.rados.rados.0.smithi045.stdout:2791: writing smithi04569269-18 from 12623872 to 13287424 tid 3 2021-07-31T01:54:45.962 INFO:tasks.rados.rados.0.smithi045.stdout:2791: writing smithi04569269-18 from 13287424 to 13713408 tid 4 2021-07-31T01:54:45.962 INFO:tasks.rados.rados.0.smithi045.stdout:2786: done (5 left) 2021-07-31T01:54:45.962 INFO:tasks.rados.rados.0.smithi045.stdout:2787: done (4 left) 2021-07-31T01:54:45.963 INFO:tasks.rados.rados.0.smithi045.stdout:2789: done (3 left) 2021-07-31T01:54:45.963 INFO:tasks.rados.rados.0.smithi045.stdout:2792: snap_remove snap 242 2021-07-31T01:54:46.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:45 smithi154 conmon[34553]: cluster 2021-07-31T01:54:44.649368+0000 mon.a (mon.0) 1241 : cluster [DBG] osdmap e582: 8 total, 8 up, 8 in 2021-07-31T01:54:46.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:45 smithi154 conmon[34553]: 2021-07-31T01:54:46.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:45 smithi045 conmon[39718]: cluster 2021-07-31T01:54:44. 2021-07-31T01:54:46.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:45 smithi045 conmon[39718]: 649368+0000 mon.a (mon.0) 1241 : cluster [DBG] osdmap e582: 8 total, 8 up, 8 in 2021-07-31T01:54:46.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:45 smithi045 conmon[33655]: cluster 2021-07-31T01:54:44.649368+0000 mon.a (mon.0) 1241 : cluster [DBG] osdmap e582: 8 total, 8 up, 8 in 2021-07-31T01:54:46.617 INFO:tasks.rados.rados.0.smithi045.stdout:2792: done (3 left) 2021-07-31T01:54:46.617 INFO:tasks.rados.rados.0.smithi045.stdout:2793: read oid 32 snap -1 2021-07-31T01:54:46.618 INFO:tasks.rados.rados.0.smithi045.stdout:2793: expect (ObjNum 754 snap 233 seq_num 754) 2021-07-31T01:54:46.618 INFO:tasks.rados.rados.0.smithi045.stdout:2794: snap_remove snap 262 2021-07-31T01:54:47.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:46 smithi154 conmon[34553]: cluster 2021-07-31T01:54:45.898471+0000 mgr.x (mgr.24542) 2021-07-31T01:54:47.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:46 smithi154 conmon[34553]: 324 : cluster [DBG] pgmap v655: 25 pgs: 25 active+clean; 346 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 20 MiB/s rd, 4.5 MiB/s wr, 20 op/s 2021-07-31T01:54:47.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:46 smithi154 conmon[34553]: cluster 2021-07-31T01:54:46.610335+0000 mon.a (mon.0) 1242 : cluster [DBG] osdmap e583: 8 total, 8 up, 8 in 2021-07-31T01:54:47.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:46 smithi045 conmon[39718]: cluster 2021-07-31T01:54:45.898471+0000 mgr.x ( 2021-07-31T01:54:47.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:46 smithi045 conmon[39718]: mgr.24542) 324 : cluster [DBG] pgmap v655: 25 pgs: 25 active+clean; 346 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 20 MiB/s rd, 4.5 MiB/s wr, 20 op/s 2021-07-31T01:54:47.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:46 smithi045 conmon[39718]: cluster 2021-07- 2021-07-31T01:54:47.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:46 smithi045 conmon[39718]: 31T01:54:46.610335+0000 mon.a (mon.0) 1242 : cluster [DBG] osdmap e583: 8 total, 8 up, 8 in 2021-07-31T01:54:47.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:46 smithi045 conmon[33655]: cluster 2021-07-31T01 2021-07-31T01:54:47.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:46 smithi045 conmon[33655]: :54:45.898471+0000 mgr.x (mgr.24542) 324 : cluster [DBG] pgmap v655: 25 pgs: 25 active+clean; 346 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 20 MiB/s rd, 4.5 MiB/s wr, 20 op/s 2021-07-31T01:54:47.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:46 smithi045 conmon[33655]: cluster 2021-07 2021-07-31T01:54:47.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:46 smithi045 conmon[33655]: -31T01:54:46.610335+0000 mon.a (mon.0 2021-07-31T01:54:47.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:46 smithi045 conmon[33655]: ) 1242 : cluster [DBG] osdmap e583: 8 total, 8 up, 8 in 2021-07-31T01:54:47.621 INFO:tasks.rados.rados.0.smithi045.stdout:2794: done (4 left) 2021-07-31T01:54:47.622 INFO:tasks.rados.rados.0.smithi045.stdout:2795: read oid 33 snap -1 2021-07-31T01:54:47.622 INFO:tasks.rados.rados.0.smithi045.stdout:2795: expect (ObjNum 726 snap 226 seq_num 726) 2021-07-31T01:54:47.624 INFO:tasks.rados.rados.0.smithi045.stdout:2796: snap_remove snap 264 2021-07-31T01:54:48.626 INFO:tasks.rados.rados.0.smithi045.stdout:2796: done (5 left) 2021-07-31T01:54:48.626 INFO:tasks.rados.rados.0.smithi045.stdout:2797: snap_create 2021-07-31T01:54:48.626 INFO:tasks.rados.rados.0.smithi045.stdout:2790: finishing write tid 1 to smithi04569269-35 2021-07-31T01:54:48.627 INFO:tasks.rados.rados.0.smithi045.stdout:2790: finishing write tid 2 to smithi04569269-35 2021-07-31T01:54:48.627 INFO:tasks.rados.rados.0.smithi045.stdout:2790: finishing write tid 3 to smithi04569269-35 2021-07-31T01:54:48.627 INFO:tasks.rados.rados.0.smithi045.stdout:2790: finishing write tid 4 to smithi04569269-35 2021-07-31T01:54:48.628 INFO:tasks.rados.rados.0.smithi045.stdout:2790: finishing write tid 5 to smithi04569269-35 2021-07-31T01:54:48.628 INFO:tasks.rados.rados.0.smithi045.stdout:2790: finishing write tid 6 to smithi04569269-35 2021-07-31T01:54:48.628 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 35 v 1775 (ObjNum 848 snap 265 seq_num 848) dirty exists 2021-07-31T01:54:48.628 INFO:tasks.rados.rados.0.smithi045.stdout:2790: left oid 35 (ObjNum 848 snap 265 seq_num 848) 2021-07-31T01:54:48.629 INFO:tasks.rados.rados.0.smithi045.stdout:2788: expect (ObjNum 799 snap 246 seq_num 799) 2021-07-31T01:54:49.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:48 smithi154 conmon[34553]: cluster 2021-07-31T01:54:47.619996+0000 mon.a (mon.0) 1243 : cluster [DBG] osdmap e584: 8 total, 8 up, 8 in 2021-07-31T01:54:49.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:48 smithi154 conmon[34553]: cluster 2021-07-31T01:54:47.898912+0000 mgr.x (mgr.24542) 325 : cluster [DBG] pgmap v658: 25 pgs: 25 active+clean; 346 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 20 MiB/s rd, 4.5 MiB/s wr, 20 op/s 2021-07-31T01:54:49.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:48 smithi045 conmon[33655]: cluster 2021-07-31T01:54:47.619996 2021-07-31T01:54:49.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:48 smithi045 conmon[33655]: +0000 mon.a (mon.0) 1243 : cluster [DBG] osdmap e584: 8 total, 8 up, 8 in 2021-07-31T01:54:49.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:48 smithi045 conmon[33655]: cluster 2021-07-31T01:54:47.898912+0000 mgr.x (mgr.24542) 325 : cluster [DBG] pgmap v658: 25 pgs: 25 active+clean; 346 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 20 MiB/s rd, 4.5 MiB/s wr, 20 op/s 2021-07-31T01:54:49.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:48 smithi045 conmon[39718]: cluster 2021-07-31T01:54:47 2021-07-31T01:54:49.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:48 smithi045 conmon[39718]: .619996+0000 mon.a (mon.0) 1243 : cluster [DBG] osdmap e584: 8 total, 8 up, 8 in 2021-07-31T01:54:49.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:48 smithi045 conmon[39718]: cluster 2021- 2021-07-31T01:54:49.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:48 smithi045 conmon[39718]: 07-31T01:54:47.898912+0000 mgr.x (mgr.24542) 325 : cluster [DBG] pgmap v658: 25 pgs: 25 active+clean; 346 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 20 MiB/s rd, 4.5 MiB/s wr, 20 op/s 2021-07-31T01:54:49.283 INFO:tasks.rados.rados.0.smithi045.stdout:2791: finishing write tid 1 to smithi04569269-18 2021-07-31T01:54:49.284 INFO:tasks.rados.rados.0.smithi045.stdout:2791: finishing write tid 2 to smithi04569269-18 2021-07-31T01:54:49.284 INFO:tasks.rados.rados.0.smithi045.stdout:2791: finishing write tid 3 to smithi04569269-18 2021-07-31T01:54:49.284 INFO:tasks.rados.rados.0.smithi045.stdout:2791: finishing write tid 4 to smithi04569269-18 2021-07-31T01:54:49.284 INFO:tasks.rados.rados.0.smithi045.stdout:2791: finishing write tid 5 to smithi04569269-18 2021-07-31T01:54:49.284 INFO:tasks.rados.rados.0.smithi045.stdout:2791: finishing write tid 6 to smithi04569269-18 2021-07-31T01:54:49.285 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 18 v 1641 (ObjNum 849 snap 265 seq_num 849) dirty exists 2021-07-31T01:54:49.285 INFO:tasks.rados.rados.0.smithi045.stdout:2791: left oid 18 (ObjNum 849 snap 265 seq_num 849) 2021-07-31T01:54:49.285 INFO:tasks.rados.rados.0.smithi045.stdout:2793: expect (ObjNum 754 snap 233 seq_num 754) 2021-07-31T01:54:49.993 INFO:tasks.rados.rados.0.smithi045.stdout:2795: expect (ObjNum 726 snap 226 seq_num 726) 2021-07-31T01:54:50.000 INFO:tasks.rados.rados.0.smithi045.stdout:2788: done (5 left) 2021-07-31T01:54:50.001 INFO:tasks.rados.rados.0.smithi045.stdout:2790: done (4 left) 2021-07-31T01:54:50.001 INFO:tasks.rados.rados.0.smithi045.stdout:2791: done (3 left) 2021-07-31T01:54:50.001 INFO:tasks.rados.rados.0.smithi045.stdout:2793: done (2 left) 2021-07-31T01:54:50.001 INFO:tasks.rados.rados.0.smithi045.stdout:2795: done (1 left) 2021-07-31T01:54:50.002 INFO:tasks.rados.rados.0.smithi045.stdout:2797: done (0 left) 2021-07-31T01:54:50.002 INFO:tasks.rados.rados.0.smithi045.stdout:2798: snap_create 2021-07-31T01:54:50.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:49 smithi154 conmon[34553]: cluster 2021-07-31T01:54:48.624968+0000 mon.a (mon.0) 1244 : cluster [DBG] osdmap e585: 8 total, 8 up, 8 in 2021-07-31T01:54:50.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:49 smithi045 conmon[39718]: cluster 2021-07-31T01:54:48.624968+0000 2021-07-31T01:54:50.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:49 smithi045 conmon[39718]: mon.a (mon.0) 1244 : cluster [DBG] osdmap e585: 8 total, 8 up, 8 in 2021-07-31T01:54:50.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:49 smithi045 conmon[33655]: cluster 2021-07-31T01:54:48.624968+0000 mon.a (mon. 2021-07-31T01:54:50.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:49 smithi045 conmon[33655]: 0) 1244 : cluster [DBG] osdmap e585: 8 total, 8 up, 8 in 2021-07-31T01:54:50.637 INFO:tasks.rados.rados.0.smithi045.stdout:2798: done (0 left) 2021-07-31T01:54:50.637 INFO:tasks.rados.rados.0.smithi045.stdout:2799: rmattr oid 31 current snap is 267 2021-07-31T01:54:50.637 INFO:tasks.rados.rados.0.smithi045.stdout:2799: done (0 left) 2021-07-31T01:54:50.638 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 24 current snap is 267 2021-07-31T01:54:50.638 INFO:tasks.rados.rados.0.smithi045.stdout:2800: seq_num 850 ranges {1122304=638976,1761280=802816,2564096=475136,3039232=327680} 2021-07-31T01:54:50.646 INFO:tasks.rados.rados.0.smithi045.stdout:2800: writing smithi04569269-24 from 1122304 to 1761280 tid 1 2021-07-31T01:54:50.651 INFO:tasks.rados.rados.0.smithi045.stdout:2800: writing smithi04569269-24 from 1761280 to 2564096 tid 2 2021-07-31T01:54:50.654 INFO:tasks.rados.rados.0.smithi045.stdout:2800: writing smithi04569269-24 from 2564096 to 3039232 tid 3 2021-07-31T01:54:50.656 INFO:tasks.rados.rados.0.smithi045.stdout:2800: writing smithi04569269-24 from 3039232 to 3366912 tid 4 2021-07-31T01:54:50.657 INFO:tasks.rados.rados.0.smithi045.stdout:2801: copy_from oid 13 from oid 50 current snap is 267 2021-07-31T01:54:50.657 INFO:tasks.rados.rados.0.smithi045.stdout:2800: finishing write tid 1 to smithi04569269-24 2021-07-31T01:54:50.657 INFO:tasks.rados.rados.0.smithi045.stdout:2800: finishing write tid 2 to smithi04569269-24 2021-07-31T01:54:50.657 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 6 current snap is 267 2021-07-31T01:54:50.658 INFO:tasks.rados.rados.0.smithi045.stdout:2802: seq_num 851 ranges {3440640=655360,4096000=688128,4784128=679936,5464064=237568} 2021-07-31T01:54:50.679 INFO:tasks.rados.rados.0.smithi045.stdout:2802: writing smithi04569269-6 from 3440640 to 4096000 tid 1 2021-07-31T01:54:50.682 INFO:tasks.rados.rados.0.smithi045.stdout:2802: writing smithi04569269-6 from 4096000 to 4784128 tid 2 2021-07-31T01:54:50.687 INFO:tasks.rados.rados.0.smithi045.stdout:2802: writing smithi04569269-6 from 4784128 to 5464064 tid 3 2021-07-31T01:54:50.689 INFO:tasks.rados.rados.0.smithi045.stdout:2802: writing smithi04569269-6 from 5464064 to 5701632 tid 4 2021-07-31T01:54:50.690 INFO:tasks.rados.rados.0.smithi045.stdout:2803: copy_from oid 2 from oid 10 current snap is 267 2021-07-31T01:54:50.690 INFO:tasks.rados.rados.0.smithi045.stdout:2800: finishing write tid 3 to smithi04569269-24 2021-07-31T01:54:50.690 INFO:tasks.rados.rados.0.smithi045.stdout:2800: finishing write tid 4 to smithi04569269-24 2021-07-31T01:54:50.691 INFO:tasks.rados.rados.0.smithi045.stdout:2800: finishing write tid 5 to smithi04569269-24 2021-07-31T01:54:50.691 INFO:tasks.rados.rados.0.smithi045.stdout:2800: finishing write tid 6 to smithi04569269-24 2021-07-31T01:54:50.691 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 24 v 1630 (ObjNum 850 snap 267 seq_num 850) dirty exists 2021-07-31T01:54:50.692 INFO:tasks.rados.rados.0.smithi045.stdout:2800: left oid 24 (ObjNum 850 snap 267 seq_num 850) 2021-07-31T01:54:50.692 INFO:tasks.rados.rados.0.smithi045.stdout:2801: finishing copy_from to smithi04569269-13 2021-07-31T01:54:50.692 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 13 v 1640 (ObjNum 837 snap 263 seq_num 837) dirty exists 2021-07-31T01:54:50.692 INFO:tasks.rados.rados.0.smithi045.stdout:2801: finishing copy_from racing read to smithi04569269-13 2021-07-31T01:54:50.692 INFO:tasks.rados.rados.0.smithi045.stdout:2802: finishing write tid 1 to smithi04569269-6 2021-07-31T01:54:50.693 INFO:tasks.rados.rados.0.smithi045.stdout:2802: finishing write tid 2 to smithi04569269-6 2021-07-31T01:54:50.693 INFO:tasks.rados.rados.0.smithi045.stdout:2800: done (3 left) 2021-07-31T01:54:50.693 INFO:tasks.rados.rados.0.smithi045.stdout:2801: done (2 left) 2021-07-31T01:54:50.693 INFO:tasks.rados.rados.0.smithi045.stdout:2804: read oid 20 snap -1 2021-07-31T01:54:50.693 INFO:tasks.rados.rados.0.smithi045.stdout:2804: expect (ObjNum 665 snap 208 seq_num 665) 2021-07-31T01:54:50.694 INFO:tasks.rados.rados.0.smithi045.stdout:2805: snap_create 2021-07-31T01:54:50.695 INFO:tasks.rados.rados.0.smithi045.stdout:2802: finishing write tid 3 to smithi04569269-6 2021-07-31T01:54:50.695 INFO:tasks.rados.rados.0.smithi045.stdout:2802: finishing write tid 4 to smithi04569269-6 2021-07-31T01:54:50.695 INFO:tasks.rados.rados.0.smithi045.stdout:2802: finishing write tid 5 to smithi04569269-6 2021-07-31T01:54:50.696 INFO:tasks.rados.rados.0.smithi045.stdout:2802: finishing write tid 6 to smithi04569269-6 2021-07-31T01:54:50.696 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 6 v 1645 (ObjNum 851 snap 267 seq_num 851) dirty exists 2021-07-31T01:54:50.696 INFO:tasks.rados.rados.0.smithi045.stdout:2802: left oid 6 (ObjNum 851 snap 267 seq_num 851) 2021-07-31T01:54:50.702 INFO:tasks.rados.rados.0.smithi045.stdout:2804: expect (ObjNum 665 snap 208 seq_num 665) 2021-07-31T01:54:50.923 INFO:tasks.rados.rados.0.smithi045.stdout:2803: finishing copy_from to smithi04569269-2 2021-07-31T01:54:50.923 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 2 v 1647 (ObjNum 847 snap 265 seq_num 847) dirty exists 2021-07-31T01:54:50.924 INFO:tasks.rados.rados.0.smithi045.stdout:2803: finishing copy_from racing read to smithi04569269-2 2021-07-31T01:54:51.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:50 smithi154 conmon[34553]: cluster 2021-07-31T01:54:49.629760+0000 mon.a (mon. 2021-07-31T01:54:51.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:50 smithi154 conmon[34553]: 0) 1245 : cluster [DBG] osdmap e586: 8 total, 8 up, 8 in 2021-07-31T01:54:51.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:50 smithi154 conmon[34553]: cluster 2021-07-31T01:54:49.899470+0000 mgr.x (mgr.24542) 326 : cluster [DBG] pgmap v661: 25 pgs: 25 active+clean; 324 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 18 MiB/s rd, 2.6 MiB/s wr, 18 op/s 2021-07-31T01:54:51.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:50 smithi045 conmon[39718]: cluster 2021-07-31T01:54:49.629760+0000 mon.a (mon. 2021-07-31T01:54:51.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:50 smithi045 conmon[39718]: 0) 1245 : cluster [DBG] osdmap e586: 8 total, 8 up, 8 in 2021-07-31T01:54:51.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:50 smithi045 conmon[39718]: cluster 2021-07-31T01:54:49.899470 2021-07-31T01:54:51.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:50 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 326 : cluster [DBG] pgmap v661: 25 pgs: 25 active+clean; 324 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 18 MiB/s rd, 2.6 MiB/s wr, 18 op/s 2021-07-31T01:54:51.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:50 smithi045 conmon[33655]: cluster 2021-07-31T01:54:49.629760+0000 mon.a (mon.0) 1245 : cluster [DBG] 2021-07-31T01:54:51.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:50 smithi045 conmon[33655]: osdmap e586: 8 total, 8 up, 8 in 2021-07-31T01:54:51.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:50 smithi045 conmon[33655]: cluster 2021-07-31T01:54:49.899470+0000 mgr.x (mgr.24542) 326 : cluster [DBG] pgmap v661: 25 pgs: 25 active+clean; 324 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 18 MiB/s rd, 2.6 MiB/s wr, 18 op/s 2021-07-31T01:54:51.612 INFO:tasks.rados.rados.0.smithi045.stdout:2802: done (3 left) 2021-07-31T01:54:51.613 INFO:tasks.rados.rados.0.smithi045.stdout:2803: done (2 left) 2021-07-31T01:54:51.613 INFO:tasks.rados.rados.0.smithi045.stdout:2804: done (1 left) 2021-07-31T01:54:51.613 INFO:tasks.rados.rados.0.smithi045.stdout:2805: done (0 left) 2021-07-31T01:54:51.613 INFO:tasks.rados.rados.0.smithi045.stdout:2806: read oid 36 snap 244 2021-07-31T01:54:51.614 INFO:tasks.rados.rados.0.smithi045.stdout:2806: expect deleted 2021-07-31T01:54:51.614 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 44 current snap is 268 2021-07-31T01:54:51.614 INFO:tasks.rados.rados.0.smithi045.stdout:2807: seq_num 852 ranges {3645440=671744,4317184=622592,4939776=442368,5382144=540672} 2021-07-31T01:54:51.636 INFO:tasks.rados.rados.0.smithi045.stdout:2807: writing smithi04569269-44 from 3645440 to 4317184 tid 1 2021-07-31T01:54:51.639 INFO:tasks.rados.rados.0.smithi045.stdout:2807: writing smithi04569269-44 from 4317184 to 4939776 tid 2 2021-07-31T01:54:51.642 INFO:tasks.rados.rados.0.smithi045.stdout:2807: writing smithi04569269-44 from 4939776 to 5382144 tid 3 2021-07-31T01:54:51.645 INFO:tasks.rados.rados.0.smithi045.stdout:2807: writing smithi04569269-44 from 5382144 to 5922816 tid 4 2021-07-31T01:54:51.645 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 42 current snap is 268 2021-07-31T01:54:51.645 INFO:tasks.rados.rados.0.smithi045.stdout:2808: seq_num 853 ranges {3645440=696320,4341760=507904,4849664=647168,5496832=450560} 2021-07-31T01:54:51.669 INFO:tasks.rados.rados.0.smithi045.stdout:2808: writing smithi04569269-42 from 3645440 to 4341760 tid 1 2021-07-31T01:54:51.671 INFO:tasks.rados.rados.0.smithi045.stdout:2808: writing smithi04569269-42 from 4341760 to 4849664 tid 2 2021-07-31T01:54:51.675 INFO:tasks.rados.rados.0.smithi045.stdout:2808: writing smithi04569269-42 from 4849664 to 5496832 tid 3 2021-07-31T01:54:51.678 INFO:tasks.rados.rados.0.smithi045.stdout:2808: writing smithi04569269-42 from 5496832 to 5947392 tid 4 2021-07-31T01:54:51.678 INFO:tasks.rados.rados.0.smithi045.stdout:2809: rollback oid 22 current snap is 268 2021-07-31T01:54:51.679 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 22 to 261 2021-07-31T01:54:51.679 INFO:tasks.rados.rados.0.smithi045.stdout:2807: finishing write tid 1 to smithi04569269-44 2021-07-31T01:54:51.679 INFO:tasks.rados.rados.0.smithi045.stdout:2807: finishing write tid 2 to smithi04569269-44 2021-07-31T01:54:51.679 INFO:tasks.rados.rados.0.smithi045.stdout:2807: finishing write tid 3 to smithi04569269-44 2021-07-31T01:54:51.680 INFO:tasks.rados.rados.0.smithi045.stdout:2807: finishing write tid 4 to smithi04569269-44 2021-07-31T01:54:51.680 INFO:tasks.rados.rados.0.smithi045.stdout:2807: finishing write tid 5 to smithi04569269-44 2021-07-31T01:54:51.680 INFO:tasks.rados.rados.0.smithi045.stdout:2807: finishing write tid 6 to smithi04569269-44 2021-07-31T01:54:51.680 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 44 v 1633 (ObjNum 852 snap 268 seq_num 852) dirty exists 2021-07-31T01:54:51.681 INFO:tasks.rados.rados.0.smithi045.stdout:2807: left oid 44 (ObjNum 852 snap 268 seq_num 852) 2021-07-31T01:54:51.681 INFO:tasks.rados.rados.0.smithi045.stdout:2808: finishing write tid 1 to smithi04569269-42 2021-07-31T01:54:51.681 INFO:tasks.rados.rados.0.smithi045.stdout:2806: done (3 left) 2021-07-31T01:54:51.681 INFO:tasks.rados.rados.0.smithi045.stdout:2807: done (2 left) 2021-07-31T01:54:51.682 INFO:tasks.rados.rados.0.smithi045.stdout:2810: read oid 16 snap -1 2021-07-31T01:54:51.682 INFO:tasks.rados.rados.0.smithi045.stdout:2810: expect deleted 2021-07-31T01:54:51.682 INFO:tasks.rados.rados.0.smithi045.stdout:2811: snap_remove snap 267 2021-07-31T01:54:52.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:51 smithi154 conmon[34553]: cluster 2021-07-31T01:54:50.636132+0000 mon.a (mon.0) 1246 : cluster [DBG] osdmap e587: 8 total, 8 up, 8 in 2021-07-31T01:54:52.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:51 smithi154 conmon[34553]: cluster 2021-07-31T01:54:51.611462+0000 2021-07-31T01:54:52.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:51 smithi154 conmon[34553]: mon.a (mon.0) 1247 : cluster [DBG] osdmap e588: 8 total, 8 up, 8 in 2021-07-31T01:54:52.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:51 smithi045 conmon[33655]: cluster 2021-07-31T01:54:50. 2021-07-31T01:54:52.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:51 smithi045 conmon[33655]: 636132+0000 mon.a (mon.0) 1246 : cluster [DBG] osdmap e587: 8 total, 8 up, 8 in 2021-07-31T01:54:52.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:51 smithi045 conmon[33655]: cluster 2021-07-31T01:54: 2021-07-31T01:54:52.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:51 smithi045 conmon[33655]: 51.611462+0000 mon.a (mon.0) 1247 : cluster [DBG] osdmap e588: 8 total, 8 up, 8 in 2021-07-31T01:54:52.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:51 smithi045 conmon[39718]: cluster 2021-07-31T01:54: 2021-07-31T01:54:52.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:51 smithi045 conmon[39718]: 50.636132+0000 mon.a (mon.0) 1246 : cluster [DBG] osdmap e587: 8 total, 8 up, 8 in 2021-07-31T01:54:52.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:51 smithi045 conmon[39718]: cluster 2021-07-31T01:54:52.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:51 smithi045 conmon[39718]: 2021-07-31T01:54 2021-07-31T01:54:52.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:51 smithi045 conmon[39718]: :51.611462+0000 mon.a ( 2021-07-31T01:54:52.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:51 smithi045 conmon[39718]: mon.0) 1247 : cluster [DBG] 2021-07-31T01:54:52.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:51 smithi045 conmon[39718]: osdmap e588: 8 total, 8 up, 8 in 2021-07-31T01:54:52.645 INFO:tasks.rados.rados.0.smithi045.stdout:2811: done (3 left) 2021-07-31T01:54:52.645 INFO:tasks.rados.rados.0.smithi045.stdout:2812: delete oid 13 current snap is 268 2021-07-31T01:54:52.645 INFO:tasks.rados.rados.0.smithi045.stdout:2808: finishing write tid 2 to smithi04569269-42 2021-07-31T01:54:52.646 INFO:tasks.rados.rados.0.smithi045.stdout:2808: finishing write tid 3 to smithi04569269-42 2021-07-31T01:54:52.646 INFO:tasks.rados.rados.0.smithi045.stdout:2808: finishing write tid 4 to smithi04569269-42 2021-07-31T01:54:52.646 INFO:tasks.rados.rados.0.smithi045.stdout:2808: finishing write tid 5 to smithi04569269-42 2021-07-31T01:54:52.646 INFO:tasks.rados.rados.0.smithi045.stdout:2809: finishing rollback tid 0 to smithi04569269-22 2021-07-31T01:54:52.647 INFO:tasks.rados.rados.0.smithi045.stdout:2809: finishing rollback tid 1 to smithi04569269-22 2021-07-31T01:54:52.647 INFO:tasks.rados.rados.0.smithi045.stdout:2809: finishing rollback tid 2 to smithi04569269-22 2021-07-31T01:54:52.647 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 22 v 1580 (ObjNum 747 snap 230 seq_num 747) dirty exists 2021-07-31T01:54:52.647 INFO:tasks.rados.rados.0.smithi045.stdout:2808: finishing write tid 6 to smithi04569269-42 2021-07-31T01:54:52.648 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 42 v 1652 (ObjNum 853 snap 268 seq_num 853) dirty exists 2021-07-31T01:54:52.648 INFO:tasks.rados.rados.0.smithi045.stdout:2808: left oid 42 (ObjNum 853 snap 268 seq_num 853) 2021-07-31T01:54:52.649 INFO:tasks.rados.rados.0.smithi045.stdout:2808: done (3 left) 2021-07-31T01:54:52.649 INFO:tasks.rados.rados.0.smithi045.stdout:2809: done (2 left) 2021-07-31T01:54:52.649 INFO:tasks.rados.rados.0.smithi045.stdout:2810: done (1 left) 2021-07-31T01:54:52.649 INFO:tasks.rados.rados.0.smithi045.stdout:2812: done (0 left) 2021-07-31T01:54:52.650 INFO:tasks.rados.rados.0.smithi045.stdout:2813: rollback oid 21 current snap is 268 2021-07-31T01:54:52.650 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 21 to 244 2021-07-31T01:54:52.650 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 49 current snap is 268 2021-07-31T01:54:52.650 INFO:tasks.rados.rados.0.smithi045.stdout:2814: seq_num 855 ranges {11870208=729088,12599296=671744,13271040=663552,13934592=270336} 2021-07-31T01:54:52.717 INFO:tasks.rados.rados.0.smithi045.stdout:2814: writing smithi04569269-49 from 11870208 to 12599296 tid 1 2021-07-31T01:54:52.721 INFO:tasks.rados.rados.0.smithi045.stdout:2814: writing smithi04569269-49 from 12599296 to 13271040 tid 2 2021-07-31T01:54:52.724 INFO:tasks.rados.rados.0.smithi045.stdout:2814: writing smithi04569269-49 from 13271040 to 13934592 tid 3 2021-07-31T01:54:52.727 INFO:tasks.rados.rados.0.smithi045.stdout:2814: writing smithi04569269-49 from 13934592 to 14204928 tid 4 2021-07-31T01:54:52.727 INFO:tasks.rados.rados.0.smithi045.stdout:2815: read oid 4 snap 249 2021-07-31T01:54:52.727 INFO:tasks.rados.rados.0.smithi045.stdout:2815: expect (ObjNum 716 snap 222 seq_num 716) 2021-07-31T01:54:52.727 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 39 current snap is 268 2021-07-31T01:54:52.728 INFO:tasks.rados.rados.0.smithi045.stdout:2813: finishing rollback tid 0 to smithi04569269-21 2021-07-31T01:54:52.728 INFO:tasks.rados.rados.0.smithi045.stdout:2816: seq_num 856 ranges {3170304=745472,3915776=557056,4472832=417792,4890624=434176,5324800=196608} 2021-07-31T01:54:52.748 INFO:tasks.rados.rados.0.smithi045.stdout:2816: writing smithi04569269-39 from 3170304 to 3915776 tid 1 2021-07-31T01:54:52.751 INFO:tasks.rados.rados.0.smithi045.stdout:2816: writing smithi04569269-39 from 3915776 to 4472832 tid 2 2021-07-31T01:54:52.754 INFO:tasks.rados.rados.0.smithi045.stdout:2816: writing smithi04569269-39 from 4472832 to 4890624 tid 3 2021-07-31T01:54:52.756 INFO:tasks.rados.rados.0.smithi045.stdout:2816: writing smithi04569269-39 from 4890624 to 5324800 tid 4 2021-07-31T01:54:52.757 INFO:tasks.rados.rados.0.smithi045.stdout:2816: writing smithi04569269-39 from 5324800 to 5521408 tid 5 2021-07-31T01:54:52.758 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 20 current snap is 268 2021-07-31T01:54:52.758 INFO:tasks.rados.rados.0.smithi045.stdout:2813: finishing rollback tid 1 to smithi04569269-21 2021-07-31T01:54:52.758 INFO:tasks.rados.rados.0.smithi045.stdout:2813: finishing rollback tid 2 to smithi04569269-21 2021-07-31T01:54:52.758 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 21 v 1795 (ObjNum 783 snap 239 seq_num 783) dirty exists 2021-07-31T01:54:52.759 INFO:tasks.rados.rados.0.smithi045.stdout:2814: finishing write tid 1 to smithi04569269-49 2021-07-31T01:54:52.759 INFO:tasks.rados.rados.0.smithi045.stdout:2814: finishing write tid 2 to smithi04569269-49 2021-07-31T01:54:52.759 INFO:tasks.rados.rados.0.smithi045.stdout:2817: seq_num 857 ranges {4349952=761856,5111808=442368,5554176=630784,6184960=532480} 2021-07-31T01:54:52.785 INFO:tasks.rados.rados.0.smithi045.stdout:2817: writing smithi04569269-20 from 4349952 to 5111808 tid 1 2021-07-31T01:54:52.788 INFO:tasks.rados.rados.0.smithi045.stdout:2817: writing smithi04569269-20 from 5111808 to 5554176 tid 2 2021-07-31T01:54:52.792 INFO:tasks.rados.rados.0.smithi045.stdout:2817: writing smithi04569269-20 from 5554176 to 6184960 tid 3 2021-07-31T01:54:52.795 INFO:tasks.rados.rados.0.smithi045.stdout:2817: writing smithi04569269-20 from 6184960 to 6717440 tid 4 2021-07-31T01:54:52.795 INFO:tasks.rados.rados.0.smithi045.stdout:2813: done (4 left) 2021-07-31T01:54:52.795 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 41 current snap is 268 2021-07-31T01:54:52.796 INFO:tasks.rados.rados.0.smithi045.stdout:2818: seq_num 858 ranges {3833856=778240,4612096=729088,5341184=434176,5775360=442368} 2021-07-31T01:54:52.820 INFO:tasks.rados.rados.0.smithi045.stdout:2818: writing smithi04569269-41 from 3833856 to 4612096 tid 1 2021-07-31T01:54:52.824 INFO:tasks.rados.rados.0.smithi045.stdout:2818: writing smithi04569269-41 from 4612096 to 5341184 tid 2 2021-07-31T01:54:52.826 INFO:tasks.rados.rados.0.smithi045.stdout:2818: writing smithi04569269-41 from 5341184 to 5775360 tid 3 2021-07-31T01:54:52.829 INFO:tasks.rados.rados.0.smithi045.stdout:2818: writing smithi04569269-41 from 5775360 to 6217728 tid 4 2021-07-31T01:54:52.829 INFO:tasks.rados.rados.0.smithi045.stdout:2819: delete oid 12 current snap is 268 2021-07-31T01:54:52.829 INFO:tasks.rados.rados.0.smithi045.stdout:2814: finishing write tid 3 to smithi04569269-49 2021-07-31T01:54:52.830 INFO:tasks.rados.rados.0.smithi045.stdout:2814: finishing write tid 4 to smithi04569269-49 2021-07-31T01:54:52.830 INFO:tasks.rados.rados.0.smithi045.stdout:2814: finishing write tid 5 to smithi04569269-49 2021-07-31T01:54:52.830 INFO:tasks.rados.rados.0.smithi045.stdout:2815: expect (ObjNum 716 snap 222 seq_num 716) 2021-07-31T01:54:52.895 INFO:tasks.rados.rados.0.smithi045.stdout:2814: finishing write tid 6 to smithi04569269-49 2021-07-31T01:54:52.895 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 49 v 1801 (ObjNum 855 snap 268 seq_num 855) dirty exists 2021-07-31T01:54:52.896 INFO:tasks.rados.rados.0.smithi045.stdout:2814: left oid 49 (ObjNum 855 snap 268 seq_num 855) 2021-07-31T01:54:52.896 INFO:tasks.rados.rados.0.smithi045.stdout:2816: finishing write tid 1 to smithi04569269-39 2021-07-31T01:54:52.896 INFO:tasks.rados.rados.0.smithi045.stdout:2816: finishing write tid 2 to smithi04569269-39 2021-07-31T01:54:52.896 INFO:tasks.rados.rados.0.smithi045.stdout:2814: done (5 left) 2021-07-31T01:54:52.896 INFO:tasks.rados.rados.0.smithi045.stdout:2815: done (4 left) 2021-07-31T01:54:52.897 INFO:tasks.rados.rados.0.smithi045.stdout:2819: done (3 left) 2021-07-31T01:54:52.897 INFO:tasks.rados.rados.0.smithi045.stdout:2820: snap_create 2021-07-31T01:54:52.897 INFO:tasks.rados.rados.0.smithi045.stdout:2816: finishing write tid 3 to smithi04569269-39 2021-07-31T01:54:52.897 INFO:tasks.rados.rados.0.smithi045.stdout:2816: finishing write tid 4 to smithi04569269-39 2021-07-31T01:54:52.898 INFO:tasks.rados.rados.0.smithi045.stdout:2816: finishing write tid 5 to smithi04569269-39 2021-07-31T01:54:52.898 INFO:tasks.rados.rados.0.smithi045.stdout:2816: finishing write tid 6 to smithi04569269-39 2021-07-31T01:54:52.898 INFO:tasks.rados.rados.0.smithi045.stdout:2816: finishing write tid 7 to smithi04569269-39 2021-07-31T01:54:52.898 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 39 v 1651 (ObjNum 856 snap 268 seq_num 856) dirty exists 2021-07-31T01:54:52.898 INFO:tasks.rados.rados.0.smithi045.stdout:2816: left oid 39 (ObjNum 856 snap 268 seq_num 856) 2021-07-31T01:54:52.899 INFO:tasks.rados.rados.0.smithi045.stdout:2817: finishing write tid 1 to smithi04569269-20 2021-07-31T01:54:52.899 INFO:tasks.rados.rados.0.smithi045.stdout:2817: finishing write tid 2 to smithi04569269-20 2021-07-31T01:54:52.899 INFO:tasks.rados.rados.0.smithi045.stdout:2817: finishing write tid 3 to smithi04569269-20 2021-07-31T01:54:52.899 INFO:tasks.rados.rados.0.smithi045.stdout:2817: finishing write tid 4 to smithi04569269-20 2021-07-31T01:54:52.900 INFO:tasks.rados.rados.0.smithi045.stdout:2817: finishing write tid 5 to smithi04569269-20 2021-07-31T01:54:52.900 INFO:tasks.rados.rados.0.smithi045.stdout:2817: finishing write tid 6 to smithi04569269-20 2021-07-31T01:54:52.900 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 20 v 1807 (ObjNum 857 snap 268 seq_num 857) dirty exists 2021-07-31T01:54:52.900 INFO:tasks.rados.rados.0.smithi045.stdout:2817: left oid 20 (ObjNum 857 snap 268 seq_num 857) 2021-07-31T01:54:52.900 INFO:tasks.rados.rados.0.smithi045.stdout:2818: finishing write tid 1 to smithi04569269-41 2021-07-31T01:54:52.901 INFO:tasks.rados.rados.0.smithi045.stdout:2818: finishing write tid 2 to smithi04569269-41 2021-07-31T01:54:52.901 INFO:tasks.rados.rados.0.smithi045.stdout:2818: finishing write tid 3 to smithi04569269-41 2021-07-31T01:54:52.901 INFO:tasks.rados.rados.0.smithi045.stdout:2818: finishing write tid 4 to smithi04569269-41 2021-07-31T01:54:52.901 INFO:tasks.rados.rados.0.smithi045.stdout:2818: finishing write tid 5 to smithi04569269-41 2021-07-31T01:54:52.901 INFO:tasks.rados.rados.0.smithi045.stdout:2818: finishing write tid 6 to smithi04569269-41 2021-07-31T01:54:52.902 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 41 v 1632 (ObjNum 858 snap 268 seq_num 858) dirty exists 2021-07-31T01:54:52.902 INFO:tasks.rados.rados.0.smithi045.stdout:2818: left oid 41 (ObjNum 858 snap 268 seq_num 858) 2021-07-31T01:54:53.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:52 smithi154 conmon[34553]: cluster 2021-07-31T01:54:51.900104+0000 mgr.x (mgr.24542) 327 : cluster [DBG] pgmap v664: 25 pgs: 25 active+clean; 324 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 18 MiB/s rd, 2.6 MiB/s wr, 18 op/s 2021-07-31T01:54:53.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:52 smithi045 conmon[39718]: cluster 2021-07-31T01:54:51.900104+0000 mgr.x (mgr.24542) 327 : cluster 2021-07-31T01:54:53.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:52 smithi045 conmon[39718]: [DBG] pgmap v664: 25 pgs: 25 active+clean; 324 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 18 MiB/s rd, 2.6 MiB/s wr, 18 op/s 2021-07-31T01:54:53.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:52 smithi045 conmon[33655]: cluster 2021-07-31T01:54:51.900104+0000 mgr.x ( 2021-07-31T01:54:53.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:52 smithi045 conmon[33655]: mgr.24542) 327 : cluster [DBG] pgmap v664: 25 pgs: 25 active+clean; 324 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 18 MiB/s rd, 2.6 MiB/s wr, 18 op/s 2021-07-31T01:54:53.646 INFO:tasks.rados.rados.0.smithi045.stdout:2816: done (3 left) 2021-07-31T01:54:53.646 INFO:tasks.rados.rados.0.smithi045.stdout:2817: done (2 left) 2021-07-31T01:54:53.647 INFO:tasks.rados.rados.0.smithi045.stdout:2818: done (1 left) 2021-07-31T01:54:53.647 INFO:tasks.rados.rados.0.smithi045.stdout:2820: done (0 left) 2021-07-31T01:54:53.647 INFO:tasks.rados.rados.0.smithi045.stdout:2821: read oid 21 snap 263 2021-07-31T01:54:53.647 INFO:tasks.rados.rados.0.smithi045.stdout:2821: expect (ObjNum 783 snap 239 seq_num 783) 2021-07-31T01:54:53.648 INFO:tasks.rados.rados.0.smithi045.stdout:2822: delete oid 3 current snap is 269 2021-07-31T01:54:53.651 INFO:tasks.rados.rados.0.smithi045.stdout:2821: expect (ObjNum 783 snap 239 seq_num 783) 2021-07-31T01:54:53.680 INFO:tasks.rados.rados.0.smithi045.stdout:2821: done (1 left) 2021-07-31T01:54:53.681 INFO:tasks.rados.rados.0.smithi045.stdout:2822: done (0 left) 2021-07-31T01:54:53.681 INFO:tasks.rados.rados.0.smithi045.stdout:2823: read oid 23 snap 244 2021-07-31T01:54:53.681 INFO:tasks.rados.rados.0.smithi045.stdout:2823: expect deleted 2021-07-31T01:54:53.681 INFO:tasks.rados.rados.0.smithi045.stdout:2824: setattr oid 29 current snap is 269 2021-07-31T01:54:53.682 INFO:tasks.rados.rados.0.smithi045.stdout:2825: copy_from oid 21 from oid 16 current snap is 269 2021-07-31T01:54:53.682 INFO:tasks.rados.rados.0.smithi045.stdout:2826: delete oid 4 current snap is 269 2021-07-31T01:54:53.683 INFO:tasks.rados.rados.0.smithi045.stdout:2825: finishing copy_from to smithi04569269-21 2021-07-31T01:54:53.683 INFO:tasks.rados.rados.0.smithi045.stdout:2825: got expected ENOENT (src dne) 2021-07-31T01:54:53.683 INFO:tasks.rados.rados.0.smithi045.stdout:2825: finishing copy_from racing read to smithi04569269-21 2021-07-31T01:54:53.685 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 29 v 1417 (ObjNum 846 snap 265 seq_num 846) dirty exists 2021-07-31T01:54:53.686 INFO:tasks.rados.rados.0.smithi045.stdout:2823: done (3 left) 2021-07-31T01:54:53.686 INFO:tasks.rados.rados.0.smithi045.stdout:2824: done (2 left) 2021-07-31T01:54:53.686 INFO:tasks.rados.rados.0.smithi045.stdout:2825: done (1 left) 2021-07-31T01:54:53.687 INFO:tasks.rados.rados.0.smithi045.stdout:2826: done (0 left) 2021-07-31T01:54:53.687 INFO:tasks.rados.rados.0.smithi045.stdout:2827: snap_create 2021-07-31T01:54:53.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:53 smithi045 conmon[39718]: cluster 2021-07-31T01:54 2021-07-31T01:54:53.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:53 smithi045 conmon[39718]: :52.643804+0000 mon.a (mon.0) 1248 : cluster [DBG] osdmap e589: 8 total, 8 up, 8 in 2021-07-31T01:54:53.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:54:53 smithi045 conmon[61210]: level=error ts=2021-07-31T01:54:53.522Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:54:53.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:54:53 smithi045 conmon[61210]: level=error ts=2021-07-31T01:54:53.522Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:54:53.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:54:53 smithi045 conmon[61210]: level=error ts=2021-07-31T01:54:53.522Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T01:54:53.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:53 smithi045 conmon[33655]: cluster 2021-07-31T01:54:52.643804+0000 mon.a ( 2021-07-31T01:54:53.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:53 smithi045 conmon[33655]: mon.0) 1248 : cluster [DBG] osdmap e589: 8 total, 8 up, 8 in 2021-07-31T01:54:54.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:53 smithi154 conmon[34553]: cluster 2021-07-31T01:54:52.643804+0000 mon.a (mon.0) 1248 : cluster [DBG] 2021-07-31T01:54:54.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:53 smithi154 conmon[34553]: osdmap e589: 8 total, 8 up, 8 in 2021-07-31T01:54:54.656 INFO:tasks.rados.rados.0.smithi045.stdout:2827: done (0 left) 2021-07-31T01:54:54.657 INFO:tasks.rados.rados.0.smithi045.stdout:2828: copy_from oid 14 from oid 19 current snap is 270 2021-07-31T01:54:54.657 INFO:tasks.rados.rados.0.smithi045.stdout:2829: rmattr oid 49 current snap is 270 2021-07-31T01:54:54.657 INFO:tasks.rados.rados.0.smithi045.stdout:2830: delete oid 31 current snap is 270 2021-07-31T01:54:54.661 INFO:tasks.rados.rados.0.smithi045.stdout:2828: finishing copy_from to smithi04569269-14 2021-07-31T01:54:54.662 INFO:tasks.rados.rados.0.smithi045.stdout:2828: got expected ENOENT (src dne) 2021-07-31T01:54:54.663 INFO:tasks.rados.rados.0.smithi045.stdout:2828: finishing copy_from racing read to smithi04569269-14 2021-07-31T01:54:54.664 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 49 v 1811 (ObjNum 855 snap 268 seq_num 855) dirty exists 2021-07-31T01:54:54.665 INFO:tasks.rados.rados.0.smithi045.stdout:2828: done (2 left) 2021-07-31T01:54:54.666 INFO:tasks.rados.rados.0.smithi045.stdout:2829: done (1 left) 2021-07-31T01:54:54.666 INFO:tasks.rados.rados.0.smithi045.stdout:2830: done (0 left) 2021-07-31T01:54:54.667 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 39 current snap is 270 2021-07-31T01:54:54.668 INFO:tasks.rados.rados.0.smithi045.stdout:2831: seq_num 863 ranges {5521408=57344} 2021-07-31T01:54:54.701 INFO:tasks.rados.rados.0.smithi045.stdout:2831: writing smithi04569269-39 from 5521408 to 5578752 tid 1 2021-07-31T01:54:54.707 INFO:tasks.rados.rados.0.smithi045.stdout:2832: delete oid 2 current snap is 270 2021-07-31T01:54:54.707 INFO:tasks.rados.rados.0.smithi045.stdout:2832: done (1 left) 2021-07-31T01:54:54.708 INFO:tasks.rados.rados.0.smithi045.stdout:2833: rmattr oid 47 current snap is 270 2021-07-31T01:54:54.708 INFO:tasks.rados.rados.0.smithi045.stdout:2833: done (1 left) 2021-07-31T01:54:54.708 INFO:tasks.rados.rados.0.smithi045.stdout:2834: read oid 47 snap 265 2021-07-31T01:54:54.709 INFO:tasks.rados.rados.0.smithi045.stdout:2834: expect (ObjNum 783 snap 239 seq_num 783) 2021-07-31T01:54:54.711 INFO:tasks.rados.rados.0.smithi045.stdout:2835: read oid 15 snap -1 2021-07-31T01:54:54.711 INFO:tasks.rados.rados.0.smithi045.stdout:2835: expect (ObjNum 839 snap 265 seq_num 839) 2021-07-31T01:54:54.711 INFO:tasks.rados.rados.0.smithi045.stdout:2836: rollback oid 37 current snap is 270 2021-07-31T01:54:54.711 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 37 to 266 2021-07-31T01:54:54.712 INFO:tasks.rados.rados.0.smithi045.stdout:2837: copy_from oid 49 from oid 14 current snap is 270 2021-07-31T01:54:54.712 INFO:tasks.rados.rados.0.smithi045.stdout:2838: read oid 40 snap 266 2021-07-31T01:54:54.712 INFO:tasks.rados.rados.0.smithi045.stdout:2838: expect (ObjNum 845 snap 265 seq_num 845) 2021-07-31T01:54:54.712 INFO:tasks.rados.rados.0.smithi045.stdout:2839: delete oid 45 current snap is 270 2021-07-31T01:54:54.713 INFO:tasks.rados.rados.0.smithi045.stdout:2831: finishing write tid 1 to smithi04569269-39 2021-07-31T01:54:54.713 INFO:tasks.rados.rados.0.smithi045.stdout:2834: expect (ObjNum 783 snap 239 seq_num 783) 2021-07-31T01:54:54.735 INFO:tasks.rados.rados.0.smithi045.stdout:2831: finishing write tid 2 to smithi04569269-39 2021-07-31T01:54:54.735 INFO:tasks.rados.rados.0.smithi045.stdout:2831: finishing write tid 3 to smithi04569269-39 2021-07-31T01:54:54.735 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 39 v 1653 (ObjNum 863 snap 270 seq_num 863) dirty exists 2021-07-31T01:54:54.736 INFO:tasks.rados.rados.0.smithi045.stdout:2831: left oid 39 (ObjNum 863 snap 270 seq_num 863) 2021-07-31T01:54:54.736 INFO:tasks.rados.rados.0.smithi045.stdout:2835: expect (ObjNum 839 snap 265 seq_num 839) 2021-07-31T01:54:54.835 INFO:tasks.rados.rados.0.smithi045.stdout:2836: finishing rollback tid 0 to smithi04569269-37 2021-07-31T01:54:54.835 INFO:tasks.rados.rados.0.smithi045.stdout:2836: finishing rollback tid 1 to smithi04569269-37 2021-07-31T01:54:54.836 INFO:tasks.rados.rados.0.smithi045.stdout:2836: finishing rollback tid 2 to smithi04569269-37 2021-07-31T01:54:54.836 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 37 v 1645 (ObjNum 844 snap 265 seq_num 844) dirty exists 2021-07-31T01:54:54.836 INFO:tasks.rados.rados.0.smithi045.stdout:2838: expect (ObjNum 845 snap 265 seq_num 845) 2021-07-31T01:54:55.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:54 smithi154 conmon[34553]: cluster 2021-07-31T01:54:53.644828+0000 mon.a (mon.0) 1249 : cluster [DBG] 2021-07-31T01:54:55.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:54 smithi154 conmon[34553]: osdmap e590: 8 total, 8 up, 8 in 2021-07-31T01:54:55.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:54 smithi154 conmon[34553]: cluster 2021-07-31T01:54:53.900717+0000 mgr.x (mgr.24542) 328 : cluster [DBG] pgmap v667: 25 pgs: 25 active+clean; 353 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 5.7 MiB/s rd, 7.8 MiB/s wr, 23 op/s 2021-07-31T01:54:55.008 INFO:tasks.rados.rados.0.smithi045.stdout:2837: finishing copy_from to smithi04569269-49 2021-07-31T01:54:55.008 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 49 v 1628 (ObjNum 841 snap 265 seq_num 841) dirty exists 2021-07-31T01:54:55.008 INFO:tasks.rados.rados.0.smithi045.stdout:2837: finishing copy_from racing read to smithi04569269-49 2021-07-31T01:54:55.009 INFO:tasks.rados.rados.0.smithi045.stdout:2831: done (6 left) 2021-07-31T01:54:55.009 INFO:tasks.rados.rados.0.smithi045.stdout:2834: done (5 left) 2021-07-31T01:54:55.009 INFO:tasks.rados.rados.0.smithi045.stdout:2835: done (4 left) 2021-07-31T01:54:55.010 INFO:tasks.rados.rados.0.smithi045.stdout:2836: done (3 left) 2021-07-31T01:54:55.010 INFO:tasks.rados.rados.0.smithi045.stdout:2837: done (2 left) 2021-07-31T01:54:55.011 INFO:tasks.rados.rados.0.smithi045.stdout:2838: done (1 left) 2021-07-31T01:54:55.011 INFO:tasks.rados.rados.0.smithi045.stdout:2839: done (0 left) 2021-07-31T01:54:55.011 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 21 current snap is 270 2021-07-31T01:54:55.011 INFO:tasks.rados.rados.0.smithi045.stdout:2840: seq_num 866 ranges {1122304=106496} 2021-07-31T01:54:55.015 INFO:tasks.rados.rados.0.smithi045.stdout:2840: writing smithi04569269-21 from 1122304 to 1228800 tid 1 2021-07-31T01:54:55.015 INFO:tasks.rados.rados.0.smithi045.stdout:2841: snap_create 2021-07-31T01:54:55.017 INFO:tasks.rados.rados.0.smithi045.stdout:2840: finishing write tid 1 to smithi04569269-21 2021-07-31T01:54:55.018 INFO:tasks.rados.rados.0.smithi045.stdout:2840: finishing write tid 2 to smithi04569269-21 2021-07-31T01:54:55.019 INFO:tasks.rados.rados.0.smithi045.stdout:2840: finishing write tid 3 to smithi04569269-21 2021-07-31T01:54:55.020 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 21 v 1817 (ObjNum 866 snap 270 seq_num 866) dirty exists 2021-07-31T01:54:55.020 INFO:tasks.rados.rados.0.smithi045.stdout:2840: left oid 21 (ObjNum 866 snap 270 seq_num 866) 2021-07-31T01:54:55.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:54 smithi045 conmon[39718]: cluster 2021-07-31T01:54:53.644828+0000 mon.a (mon.0) 2021-07-31T01:54:55.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:54 smithi045 conmon[39718]: 1249 : cluster [DBG] osdmap e590: 8 total, 8 up, 8 in 2021-07-31T01:54:55.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:54 smithi045 conmon[39718]: cluster 2021-07-31T01:54:53.900717+0000 mgr.x (mgr.24542) 328 : cluster [DBG] pgmap v667: 25 pgs: 25 active+clean; 353 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 5.7 MiB/s rd, 7.8 MiB/s wr, 23 op/s 2021-07-31T01:54:55.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:54 smithi045 conmon[33655]: cluster 2021-07-31T01:54:53.644828+0000 mon.a (mon.0) 1249 2021-07-31T01:54:55.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:54 smithi045 conmon[33655]: : cluster [DBG] osdmap e590: 8 total, 8 up, 8 in 2021-07-31T01:54:55.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:54 smithi045 conmon[33655]: cluster 2021-07-31T01:54:53.900717+0000 mgr.x (mgr.24542) 328 : cluster [DBG] pgmap v667: 25 pgs: 25 active+clean; 353 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 5.7 MiB/s rd, 7.8 MiB/s wr, 23 op/s 2021-07-31T01:54:55.663 INFO:tasks.rados.rados.0.smithi045.stdout:2840: done (1 left) 2021-07-31T01:54:55.663 INFO:tasks.rados.rados.0.smithi045.stdout:2841: done (0 left) 2021-07-31T01:54:55.663 INFO:tasks.rados.rados.0.smithi045.stdout:2842: snap_create 2021-07-31T01:54:56.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:55 smithi154 conmon[34553]: cluster 2021-07-31T01:54:54.655517+0000 mon.a (mon.0) 1250 : cluster [DBG] osdmap e591: 8 total, 8 up, 8 in 2021-07-31T01:54:56.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:55 smithi045 conmon[33655]: cluster 2021-07-31T01:54:54.655517+0000 mon.a (mon.0) 1250 : 2021-07-31T01:54:56.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:55 smithi045 conmon[33655]: cluster [DBG] osdmap e591: 8 total, 8 up, 8 in 2021-07-31T01:54:56.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:55 smithi045 conmon[39718]: cluster 2021-07-31T01:54: 2021-07-31T01:54:56.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:55 smithi045 conmon[39718]: 54.655517+0000 mon.a (mon.0) 1250 : cluster [DBG] osdmap e591: 8 total, 8 up, 8 in 2021-07-31T01:54:56.613 INFO:tasks.rados.rados.0.smithi045.stdout:2842: done (0 left) 2021-07-31T01:54:56.614 INFO:tasks.rados.rados.0.smithi045.stdout:2843: delete oid 42 current snap is 272 2021-07-31T01:54:56.619 INFO:tasks.rados.rados.0.smithi045.stdout:2843: done (0 left) 2021-07-31T01:54:56.619 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 26 current snap is 272 2021-07-31T01:54:56.619 INFO:tasks.rados.rados.0.smithi045.stdout:2844: seq_num 868 ranges {0=139264} 2021-07-31T01:54:56.621 INFO:tasks.rados.rados.0.smithi045.stdout:2844: writing smithi04569269-26 from 0 to 139264 tid 1 2021-07-31T01:54:56.621 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 42 current snap is 272 2021-07-31T01:54:56.621 INFO:tasks.rados.rados.0.smithi045.stdout:2845: seq_num 869 ranges {0=155648} 2021-07-31T01:54:56.622 INFO:tasks.rados.rados.0.smithi045.stdout:2845: writing smithi04569269-42 from 0 to 155648 tid 1 2021-07-31T01:54:56.622 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 22 current snap is 272 2021-07-31T01:54:56.622 INFO:tasks.rados.rados.0.smithi045.stdout:2846: seq_num 870 ranges {1957888=172032} 2021-07-31T01:54:56.633 INFO:tasks.rados.rados.0.smithi045.stdout:2846: writing smithi04569269-22 from 1957888 to 2129920 tid 1 2021-07-31T01:54:56.633 INFO:tasks.rados.rados.0.smithi045.stdout:2847: delete oid 44 current snap is 272 2021-07-31T01:54:56.633 INFO:tasks.rados.rados.0.smithi045.stdout:2844: finishing write tid 1 to smithi04569269-26 2021-07-31T01:54:56.634 INFO:tasks.rados.rados.0.smithi045.stdout:2844: finishing write tid 2 to smithi04569269-26 2021-07-31T01:54:56.634 INFO:tasks.rados.rados.0.smithi045.stdout:2845: finishing write tid 1 to smithi04569269-42 2021-07-31T01:54:56.634 INFO:tasks.rados.rados.0.smithi045.stdout:2845: finishing write tid 2 to smithi04569269-42 2021-07-31T01:54:56.634 INFO:tasks.rados.rados.0.smithi045.stdout:2844: finishing write tid 3 to smithi04569269-26 2021-07-31T01:54:56.634 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 26 v 1408 (ObjNum 868 snap 272 seq_num 868) dirty exists 2021-07-31T01:54:56.635 INFO:tasks.rados.rados.0.smithi045.stdout:2844: left oid 26 (ObjNum 868 snap 272 seq_num 868) 2021-07-31T01:54:56.635 INFO:tasks.rados.rados.0.smithi045.stdout:2845: finishing write tid 3 to smithi04569269-42 2021-07-31T01:54:56.635 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 42 v 1656 (ObjNum 869 snap 272 seq_num 869) dirty exists 2021-07-31T01:54:56.635 INFO:tasks.rados.rados.0.smithi045.stdout:2845: left oid 42 (ObjNum 869 snap 272 seq_num 869) 2021-07-31T01:54:56.636 INFO:tasks.rados.rados.0.smithi045.stdout:2846: finishing write tid 1 to smithi04569269-22 2021-07-31T01:54:56.636 INFO:tasks.rados.rados.0.smithi045.stdout:2846: finishing write tid 2 to smithi04569269-22 2021-07-31T01:54:56.637 INFO:tasks.rados.rados.0.smithi045.stdout:2846: finishing write tid 3 to smithi04569269-22 2021-07-31T01:54:56.638 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 22 v 1582 (ObjNum 870 snap 272 seq_num 870) dirty exists 2021-07-31T01:54:56.638 INFO:tasks.rados.rados.0.smithi045.stdout:2846: left oid 22 (ObjNum 870 snap 272 seq_num 870) 2021-07-31T01:54:56.639 INFO:tasks.rados.rados.0.smithi045.stdout:2844: done (3 left) 2021-07-31T01:54:56.639 INFO:tasks.rados.rados.0.smithi045.stdout:2845: done (2 left) 2021-07-31T01:54:56.640 INFO:tasks.rados.rados.0.smithi045.stdout:2846: done (1 left) 2021-07-31T01:54:56.640 INFO:tasks.rados.rados.0.smithi045.stdout:2847: done (0 left) 2021-07-31T01:54:56.640 INFO:tasks.rados.rados.0.smithi045.stdout:2848: read oid 38 snap -1 2021-07-31T01:54:56.640 INFO:tasks.rados.rados.0.smithi045.stdout:2848: expect (ObjNum 842 snap 265 seq_num 842) 2021-07-31T01:54:56.640 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 48 current snap is 272 2021-07-31T01:54:56.641 INFO:tasks.rados.rados.0.smithi045.stdout:2849: seq_num 872 ranges {9011200=212992} 2021-07-31T01:54:56.690 INFO:tasks.rados.rados.0.smithi045.stdout:2849: writing smithi04569269-48 from 9011200 to 9224192 tid 1 2021-07-31T01:54:56.690 INFO:tasks.rados.rados.0.smithi045.stdout:2850: read oid 5 snap -1 2021-07-31T01:54:56.691 INFO:tasks.rados.rados.0.smithi045.stdout:2850: expect (ObjNum 817 snap 257 seq_num 817) 2021-07-31T01:54:56.691 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 13 current snap is 272 2021-07-31T01:54:56.692 INFO:tasks.rados.rados.0.smithi045.stdout:2851: seq_num 873 ranges {0=229376} 2021-07-31T01:54:56.694 INFO:tasks.rados.rados.0.smithi045.stdout:2851: writing smithi04569269-13 from 0 to 229376 tid 1 2021-07-31T01:54:56.694 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 34 current snap is 272 2021-07-31T01:54:56.695 INFO:tasks.rados.rados.0.smithi045.stdout:2852: seq_num 874 ranges {1957888=245760} 2021-07-31T01:54:56.703 INFO:tasks.rados.rados.0.smithi045.stdout:2852: writing smithi04569269-34 from 1957888 to 2203648 tid 1 2021-07-31T01:54:56.703 INFO:tasks.rados.rados.0.smithi045.stdout:2853: rmattr oid 14 current snap is 272 2021-07-31T01:54:56.703 INFO:tasks.rados.rados.0.smithi045.stdout:2848: expect (ObjNum 842 snap 265 seq_num 842) 2021-07-31T01:54:56.849 INFO:tasks.rados.rados.0.smithi045.stdout:2849: finishing write tid 1 to smithi04569269-48 2021-07-31T01:54:56.849 INFO:tasks.rados.rados.0.smithi045.stdout:2849: finishing write tid 2 to smithi04569269-48 2021-07-31T01:54:56.849 INFO:tasks.rados.rados.0.smithi045.stdout:2851: finishing write tid 1 to smithi04569269-13 2021-07-31T01:54:56.849 INFO:tasks.rados.rados.0.smithi045.stdout:2851: finishing write tid 2 to smithi04569269-13 2021-07-31T01:54:56.850 INFO:tasks.rados.rados.0.smithi045.stdout:2849: finishing write tid 3 to smithi04569269-48 2021-07-31T01:54:56.850 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 48 v 1578 (ObjNum 872 snap 272 seq_num 872) dirty exists 2021-07-31T01:54:56.850 INFO:tasks.rados.rados.0.smithi045.stdout:2849: left oid 48 (ObjNum 872 snap 272 seq_num 872) 2021-07-31T01:54:56.850 INFO:tasks.rados.rados.0.smithi045.stdout:2851: finishing write tid 3 to smithi04569269-13 2021-07-31T01:54:56.851 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 13 v 1649 (ObjNum 873 snap 272 seq_num 873) dirty exists 2021-07-31T01:54:56.851 INFO:tasks.rados.rados.0.smithi045.stdout:2851: left oid 13 (ObjNum 873 snap 272 seq_num 873) 2021-07-31T01:54:56.851 INFO:tasks.rados.rados.0.smithi045.stdout:2852: finishing write tid 1 to smithi04569269-34 2021-07-31T01:54:56.851 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 14 v 1635 (ObjNum 841 snap 265 seq_num 841) dirty exists 2021-07-31T01:54:56.851 INFO:tasks.rados.rados.0.smithi045.stdout:2852: finishing write tid 2 to smithi04569269-34 2021-07-31T01:54:56.852 INFO:tasks.rados.rados.0.smithi045.stdout:2852: finishing write tid 3 to smithi04569269-34 2021-07-31T01:54:56.852 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 34 v 1580 (ObjNum 874 snap 272 seq_num 874) dirty exists 2021-07-31T01:54:56.852 INFO:tasks.rados.rados.0.smithi045.stdout:2852: left oid 34 (ObjNum 874 snap 272 seq_num 874) 2021-07-31T01:54:56.852 INFO:tasks.rados.rados.0.smithi045.stdout:2848: done (5 left) 2021-07-31T01:54:56.853 INFO:tasks.rados.rados.0.smithi045.stdout:2849: done (4 left) 2021-07-31T01:54:56.853 INFO:tasks.rados.rados.0.smithi045.stdout:2851: done (3 left) 2021-07-31T01:54:56.853 INFO:tasks.rados.rados.0.smithi045.stdout:2852: done (2 left) 2021-07-31T01:54:56.853 INFO:tasks.rados.rados.0.smithi045.stdout:2853: done (1 left) 2021-07-31T01:54:56.853 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 7 current snap is 272 2021-07-31T01:54:56.854 INFO:tasks.rados.rados.0.smithi045.stdout:2850: expect (ObjNum 817 snap 257 seq_num 817) 2021-07-31T01:54:57.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:56 smithi154 conmon[34553]: cluster 2021-07-31T01:54:55.662582+0000 mon.a (mon.0) 1251 : cluster 2021-07-31T01:54:57.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:56 smithi154 conmon[34553]: [DBG] osdmap e592: 8 total, 8 up, 8 in 2021-07-31T01:54:57.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:56 smithi154 conmon[34553]: cluster 2021-07-31T01:54:55.901295+0000 mgr.x (mgr.24542) 329 : cluster [DBG] pgmap v670: 25 pgs: 25 active+clean; 353 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 5.7 MiB/s rd, 7.8 MiB/s wr, 23 op/s 2021-07-31T01:54:57.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:56 smithi154 conmon[34553]: cluster 2021-07-31T01:54:56.613104+0000 mon.a (mon.0 2021-07-31T01:54:57.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:56 smithi154 conmon[34553]: ) 1252 : cluster [DBG] osdmap e593: 8 total, 8 up, 8 in 2021-07-31T01:54:57.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:56 smithi045 conmon[33655]: cluster 2021-07-31T01:54:55.662582+0000 mon.a (mon.0) 1251 : cluster [DBG] osdmap e592: 8 total, 8 up, 8 in 2021-07-31T01:54:57.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:56 smithi045 conmon[33655]: cluster 2021-07-31T01:54:55.901295+0000 mgr.x (mgr.24542) 329 : cluster [DBG] pgmap v670: 25 pgs: 25 active+clean; 353 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 5.7 MiB/s rd, 7.8 MiB/s wr, 23 op/s 2021-07-31T01:54:57.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:56 smithi045 conmon[33655]: cluster 2021-07-31T01:54:56.613104+0000 mon.a (mon.0) 1252 : cluster [DBG] osdmap e593: 8 total, 8 up, 8 in 2021-07-31T01:54:57.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:56 smithi045 conmon[39718]: cluster 2021-07-31T01:54:55.662582+0000 2021-07-31T01:54:57.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:56 smithi045 conmon[39718]: mon.a (mon.0) 1251 : cluster [DBG] osdmap e592: 8 total, 8 up, 8 in 2021-07-31T01:54:57.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:56 smithi045 conmon[39718]: cluster 2021-07-31T01:54:57.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:56 smithi045 conmon[39718]: 2021-07-31T01:54:55.901295+0000 mgr.x 2021-07-31T01:54:57.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:56 smithi045 conmon[39718]: (mgr.24542) 329 : cluster [DBG] pgmap v670: 25 pgs: 25 active+clean; 353 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 5.7 MiB/s rd, 7.8 MiB/s wr, 23 op/s 2021-07-31T01:54:57.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:56 smithi045 conmon[39718]: cluster 2021-07- 2021-07-31T01:54:57.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:56 smithi045 conmon[39718]: 31T01:54:56.613104+0000 2021-07-31T01:54:57.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:56 smithi045 conmon[39718]: mon.a (mon.0) 1252 : cluster 2021-07-31T01:54:57.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:56 smithi045 conmon[39718]: [DBG] osdmap e593: 8 total, 8 up, 8 in 2021-07-31T01:54:57.958 INFO:tasks.rados.rados.0.smithi045.stdout:2854: seq_num 875 ranges {11321344=262144} 2021-07-31T01:54:58.021 INFO:tasks.rados.rados.0.smithi045.stdout:2854: writing smithi04569269-7 from 11321344 to 11583488 tid 1 2021-07-31T01:54:58.021 INFO:tasks.rados.rados.0.smithi045.stdout:2850: done (1 left) 2021-07-31T01:54:58.021 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 47 current snap is 272 2021-07-31T01:54:58.021 INFO:tasks.rados.rados.0.smithi045.stdout:2855: seq_num 876 ranges {1122304=278528} 2021-07-31T01:54:58.029 INFO:tasks.rados.rados.0.smithi045.stdout:2855: writing smithi04569269-47 from 1122304 to 1400832 tid 1 2021-07-31T01:54:58.029 INFO:tasks.rados.rados.0.smithi045.stdout:2856: read oid 46 snap 265 2021-07-31T01:54:58.029 INFO:tasks.rados.rados.0.smithi045.stdout:2856: expect (ObjNum 826 snap 258 seq_num 826) 2021-07-31T01:54:58.029 INFO:tasks.rados.rados.0.smithi045.stdout:2857: rollback oid 23 current snap is 272 2021-07-31T01:54:58.030 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 23 to 259 2021-07-31T01:54:58.030 INFO:tasks.rados.rados.0.smithi045.stdout:2854: finishing write tid 1 to smithi04569269-7 2021-07-31T01:54:58.030 INFO:tasks.rados.rados.0.smithi045.stdout:2854: finishing write tid 2 to smithi04569269-7 2021-07-31T01:54:58.030 INFO:tasks.rados.rados.0.smithi045.stdout:2854: finishing write tid 3 to smithi04569269-7 2021-07-31T01:54:58.031 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 7 v 1634 (ObjNum 875 snap 272 seq_num 875) dirty exists 2021-07-31T01:54:58.031 INFO:tasks.rados.rados.0.smithi045.stdout:2854: left oid 7 (ObjNum 875 snap 272 seq_num 875) 2021-07-31T01:54:58.031 INFO:tasks.rados.rados.0.smithi045.stdout:2854: done (3 left) 2021-07-31T01:54:58.031 INFO:tasks.rados.rados.0.smithi045.stdout:2858: read oid 26 snap -1 2021-07-31T01:54:58.031 INFO:tasks.rados.rados.0.smithi045.stdout:2858: expect (ObjNum 868 snap 272 seq_num 868) 2021-07-31T01:54:58.032 INFO:tasks.rados.rados.0.smithi045.stdout:2859: rollback oid 27 current snap is 272 2021-07-31T01:54:58.032 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 27 to 271 2021-07-31T01:54:58.032 INFO:tasks.rados.rados.0.smithi045.stdout:2860: delete oid 12 current snap is 272 2021-07-31T01:54:58.033 INFO:tasks.rados.rados.0.smithi045.stdout:2860: done (5 left) 2021-07-31T01:54:58.033 INFO:tasks.rados.rados.0.smithi045.stdout:2861: read oid 35 snap -1 2021-07-31T01:54:58.034 INFO:tasks.rados.rados.0.smithi045.stdout:2861: expect (ObjNum 848 snap 265 seq_num 848) 2021-07-31T01:54:58.034 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 22 current snap is 272 2021-07-31T01:54:58.034 INFO:tasks.rados.rados.0.smithi045.stdout:2862: seq_num 878 ranges {2129920=311296} 2021-07-31T01:54:58.047 INFO:tasks.rados.rados.0.smithi045.stdout:2862: writing smithi04569269-22 from 2129920 to 2441216 tid 1 2021-07-31T01:54:58.047 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 40 current snap is 272 2021-07-31T01:54:58.047 INFO:tasks.rados.rados.0.smithi045.stdout:2857: finishing rollback tid 1 to smithi04569269-23 2021-07-31T01:54:58.047 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 23 v 0 (ObjNum 0 snap 0 seq_num 0) dirty dne 2021-07-31T01:54:58.048 INFO:tasks.rados.rados.0.smithi045.stdout:2863: seq_num 879 ranges {4530176=327680} 2021-07-31T01:54:58.073 INFO:tasks.rados.rados.0.smithi045.stdout:2863: writing smithi04569269-40 from 4530176 to 4857856 tid 1 2021-07-31T01:54:58.074 INFO:tasks.rados.rados.0.smithi045.stdout:2857: done (7 left) 2021-07-31T01:54:58.074 INFO:tasks.rados.rados.0.smithi045.stdout:2864: copy_from oid 43 from oid 48 current snap is 272 2021-07-31T01:54:58.074 INFO:tasks.rados.rados.0.smithi045.stdout:2858: expect (ObjNum 868 snap 272 seq_num 868) 2021-07-31T01:54:58.078 INFO:tasks.rados.rados.0.smithi045.stdout:2859: finishing rollback tid 0 to smithi04569269-27 2021-07-31T01:54:58.078 INFO:tasks.rados.rados.0.smithi045.stdout:2859: finishing rollback tid 1 to smithi04569269-27 2021-07-31T01:54:58.079 INFO:tasks.rados.rados.0.smithi045.stdout:2859: finishing rollback tid 2 to smithi04569269-27 2021-07-31T01:54:58.079 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 27 v 1638 (ObjNum 791 snap 244 seq_num 791) dirty exists 2021-07-31T01:54:58.080 INFO:tasks.rados.rados.0.smithi045.stdout:2855: finishing write tid 1 to smithi04569269-47 2021-07-31T01:54:58.080 INFO:tasks.rados.rados.0.smithi045.stdout:2855: finishing write tid 2 to smithi04569269-47 2021-07-31T01:54:58.080 INFO:tasks.rados.rados.0.smithi045.stdout:2858: done (7 left) 2021-07-31T01:54:58.081 INFO:tasks.rados.rados.0.smithi045.stdout:2859: done (6 left) 2021-07-31T01:54:58.081 INFO:tasks.rados.rados.0.smithi045.stdout:2865: snap_remove snap 261 2021-07-31T01:54:58.963 INFO:tasks.rados.rados.0.smithi045.stdout:2865: done (6 left) 2021-07-31T01:54:58.964 INFO:tasks.rados.rados.0.smithi045.stdout:2866: copy_from oid 2 from oid 50 current snap is 272 2021-07-31T01:54:58.964 INFO:tasks.rados.rados.0.smithi045.stdout:2861: expect (ObjNum 848 snap 265 seq_num 848) 2021-07-31T01:54:59.032 INFO:tasks.rados.rados.0.smithi045.stdout:2855: finishing write tid 3 to smithi04569269-47 2021-07-31T01:54:59.032 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 47 v 1630 (ObjNum 876 snap 272 seq_num 876) dirty exists 2021-07-31T01:54:59.032 INFO:tasks.rados.rados.0.smithi045.stdout:2855: left oid 47 (ObjNum 876 snap 272 seq_num 876) 2021-07-31T01:54:59.033 INFO:tasks.rados.rados.0.smithi045.stdout:2855: done (6 left) 2021-07-31T01:54:59.033 INFO:tasks.rados.rados.0.smithi045.stdout:2861: done (5 left) 2021-07-31T01:54:59.033 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 32 current snap is 272 2021-07-31T01:54:59.033 INFO:tasks.rados.rados.0.smithi045.stdout:2862: finishing write tid 1 to smithi04569269-22 2021-07-31T01:54:59.034 INFO:tasks.rados.rados.0.smithi045.stdout:2862: finishing write tid 2 to smithi04569269-22 2021-07-31T01:54:59.034 INFO:tasks.rados.rados.0.smithi045.stdout:2862: finishing write tid 3 to smithi04569269-22 2021-07-31T01:54:59.034 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 22 v 1584 (ObjNum 878 snap 272 seq_num 878) dirty exists 2021-07-31T01:54:59.035 INFO:tasks.rados.rados.0.smithi045.stdout:2862: left oid 22 (ObjNum 878 snap 272 seq_num 878) 2021-07-31T01:54:59.035 INFO:tasks.rados.rados.0.smithi045.stdout:2856: expect (ObjNum 826 snap 258 seq_num 826) 2021-07-31T01:54:59.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:58 smithi154 conmon[34553]: cluster 2021-07-31T01:54:57.901782+0000 mgr.x (mgr.24542) 330 : cluster [DBG] 2021-07-31T01:54:59.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:58 smithi154 conmon[34553]: pgmap v672: 25 pgs: 25 active+clean; 353 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 5.3 MiB/s rd, 7.3 MiB/s wr, 22 op/s 2021-07-31T01:54:59.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:58 smithi045 conmon[39718]: cluster 2021-07-31T01:54:57.901782+0000 mgr.x (mgr.24542) 330 : cluster [DBG] pgmap v672: 25 pgs: 25 active+clean; 353 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 5.3 MiB/s rd, 7.3 MiB/s wr, 22 op/s 2021-07-31T01:54:59.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:58 smithi045 conmon[33655]: cluster 2021-07-31T01:54:57.901782+0000 mgr.x (mgr.24542) 330 : cluster [DBG] pgmap v672: 25 pgs: 25 active+clean; 353 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 5.3 MiB/s rd, 7.3 MiB/s wr, 22 op/s 2021-07-31T01:54:59.370 INFO:tasks.rados.rados.0.smithi045.stdout:2863: finishing write tid 1 to smithi04569269-40 2021-07-31T01:54:59.370 INFO:tasks.rados.rados.0.smithi045.stdout:2863: finishing write tid 2 to smithi04569269-40 2021-07-31T01:54:59.370 INFO:tasks.rados.rados.0.smithi045.stdout:2863: finishing write tid 3 to smithi04569269-40 2021-07-31T01:54:59.371 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 40 v 1651 (ObjNum 879 snap 272 seq_num 879) dirty exists 2021-07-31T01:54:59.371 INFO:tasks.rados.rados.0.smithi045.stdout:2863: left oid 40 (ObjNum 879 snap 272 seq_num 879) 2021-07-31T01:54:59.371 INFO:tasks.rados.rados.0.smithi045.stdout:2864: finishing copy_from to smithi04569269-43 2021-07-31T01:54:59.371 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 43 v 1578 (ObjNum 872 snap 272 seq_num 872) dirty exists 2021-07-31T01:54:59.372 INFO:tasks.rados.rados.0.smithi045.stdout:2864: finishing copy_from racing read to smithi04569269-43 2021-07-31T01:54:59.372 INFO:tasks.rados.rados.0.smithi045.stdout:2866: finishing copy_from to smithi04569269-2 2021-07-31T01:54:59.372 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 2 v 1640 (ObjNum 837 snap 263 seq_num 837) dirty exists 2021-07-31T01:54:59.373 INFO:tasks.rados.rados.0.smithi045.stdout:2866: finishing copy_from racing read to smithi04569269-2 2021-07-31T01:54:59.373 INFO:tasks.rados.rados.0.smithi045.stdout:2867: seq_num 880 ranges {9011200=344064} 2021-07-31T01:54:59.421 INFO:tasks.rados.rados.0.smithi045.stdout:2867: writing smithi04569269-32 from 9011200 to 9355264 tid 1 2021-07-31T01:54:59.421 INFO:tasks.rados.rados.0.smithi045.stdout:2856: done (5 left) 2021-07-31T01:54:59.421 INFO:tasks.rados.rados.0.smithi045.stdout:2862: done (4 left) 2021-07-31T01:54:59.422 INFO:tasks.rados.rados.0.smithi045.stdout:2863: done (3 left) 2021-07-31T01:54:59.422 INFO:tasks.rados.rados.0.smithi045.stdout:2864: done (2 left) 2021-07-31T01:54:59.422 INFO:tasks.rados.rados.0.smithi045.stdout:2866: done (1 left) 2021-07-31T01:54:59.422 INFO:tasks.rados.rados.0.smithi045.stdout:2868: rollback oid 37 current snap is 272 2021-07-31T01:54:59.422 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 37 to 270 2021-07-31T01:54:59.423 INFO:tasks.rados.rados.0.smithi045.stdout:2869: rollback oid 31 current snap is 272 2021-07-31T01:54:59.423 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 31 to 268 2021-07-31T01:54:59.423 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 48 current snap is 272 2021-07-31T01:54:59.423 INFO:tasks.rados.rados.0.smithi045.stdout:2870: seq_num 881 ranges {9224192=360448} 2021-07-31T01:54:59.473 INFO:tasks.rados.rados.0.smithi045.stdout:2870: writing smithi04569269-48 from 9224192 to 9584640 tid 1 2021-07-31T01:54:59.473 INFO:tasks.rados.rados.0.smithi045.stdout:2871: delete oid 25 current snap is 272 2021-07-31T01:54:59.473 INFO:tasks.rados.rados.0.smithi045.stdout:2869: finishing rollback tid 1 to smithi04569269-31 2021-07-31T01:54:59.476 INFO:tasks.rados.rados.0.smithi045.stdout:2868: finishing rollback tid 0 to smithi04569269-37 2021-07-31T01:54:59.476 INFO:tasks.rados.rados.0.smithi045.stdout:2869: finishing rollback tid 2 to smithi04569269-31 2021-07-31T01:54:59.476 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 31 v 1833 (ObjNum 813 snap 254 seq_num 813) dirty exists 2021-07-31T01:54:59.477 INFO:tasks.rados.rados.0.smithi045.stdout:2868: finishing rollback tid 1 to smithi04569269-37 2021-07-31T01:54:59.477 INFO:tasks.rados.rados.0.smithi045.stdout:2867: finishing write tid 1 to smithi04569269-32 2021-07-31T01:54:59.477 INFO:tasks.rados.rados.0.smithi045.stdout:2867: finishing write tid 2 to smithi04569269-32 2021-07-31T01:54:59.477 INFO:tasks.rados.rados.0.smithi045.stdout:2868: finishing rollback tid 2 to smithi04569269-37 2021-07-31T01:54:59.478 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 37 v 1648 (ObjNum 844 snap 265 seq_num 844) dirty exists 2021-07-31T01:54:59.478 INFO:tasks.rados.rados.0.smithi045.stdout:2867: finishing write tid 3 to smithi04569269-32 2021-07-31T01:54:59.478 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 32 v 1410 (ObjNum 880 snap 272 seq_num 880) dirty exists 2021-07-31T01:54:59.478 INFO:tasks.rados.rados.0.smithi045.stdout:2867: left oid 32 (ObjNum 880 snap 272 seq_num 880) 2021-07-31T01:54:59.480 INFO:tasks.rados.rados.0.smithi045.stdout:2867: done (4 left) 2021-07-31T01:54:59.480 INFO:tasks.rados.rados.0.smithi045.stdout:2868: done (3 left) 2021-07-31T01:54:59.480 INFO:tasks.rados.rados.0.smithi045.stdout:2869: done (2 left) 2021-07-31T01:54:59.481 INFO:tasks.rados.rados.0.smithi045.stdout:2871: done (1 left) 2021-07-31T01:54:59.481 INFO:tasks.rados.rados.0.smithi045.stdout:2872: read oid 16 snap -1 2021-07-31T01:54:59.481 INFO:tasks.rados.rados.0.smithi045.stdout:2872: expect deleted 2021-07-31T01:54:59.481 INFO:tasks.rados.rados.0.smithi045.stdout:2873: delete oid 30 current snap is 272 2021-07-31T01:54:59.482 INFO:tasks.rados.rados.0.smithi045.stdout:2870: finishing write tid 1 to smithi04569269-48 2021-07-31T01:54:59.482 INFO:tasks.rados.rados.0.smithi045.stdout:2870: finishing write tid 2 to smithi04569269-48 2021-07-31T01:54:59.482 INFO:tasks.rados.rados.0.smithi045.stdout:2870: finishing write tid 3 to smithi04569269-48 2021-07-31T01:54:59.482 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 48 v 1582 (ObjNum 881 snap 272 seq_num 881) dirty exists 2021-07-31T01:54:59.482 INFO:tasks.rados.rados.0.smithi045.stdout:2870: left oid 48 (ObjNum 881 snap 272 seq_num 881) 2021-07-31T01:54:59.483 INFO:tasks.rados.rados.0.smithi045.stdout:2870: done (2 left) 2021-07-31T01:54:59.483 INFO:tasks.rados.rados.0.smithi045.stdout:2872: done (1 left) 2021-07-31T01:54:59.483 INFO:tasks.rados.rados.0.smithi045.stdout:2873: done (0 left) 2021-07-31T01:54:59.483 INFO:tasks.rados.rados.0.smithi045.stdout:2874: copy_from oid 16 from oid 14 current snap is 272 2021-07-31T01:54:59.484 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 1 current snap is 272 2021-07-31T01:54:59.484 INFO:tasks.rados.rados.0.smithi045.stdout:2875: seq_num 884 ranges {5382144=409600} 2021-07-31T01:54:59.510 INFO:tasks.rados.rados.0.smithi045.stdout:2875: writing smithi04569269-1 from 5382144 to 5791744 tid 1 2021-07-31T01:54:59.511 INFO:tasks.rados.rados.0.smithi045.stdout:2876: read oid 27 snap -1 2021-07-31T01:54:59.511 INFO:tasks.rados.rados.0.smithi045.stdout:2876: expect (ObjNum 791 snap 244 seq_num 791) 2021-07-31T01:54:59.511 INFO:tasks.rados.rados.0.smithi045.stdout:2877: read oid 23 snap 266 2021-07-31T01:54:59.512 INFO:tasks.rados.rados.0.smithi045.stdout:2877: expect deleted 2021-07-31T01:54:59.512 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 37 current snap is 272 2021-07-31T01:54:59.512 INFO:tasks.rados.rados.0.smithi045.stdout:2878: seq_num 885 ranges {11157504=425984} 2021-07-31T01:54:59.574 INFO:tasks.rados.rados.0.smithi045.stdout:2878: writing smithi04569269-37 from 11157504 to 11583488 tid 1 2021-07-31T01:54:59.574 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 30 current snap is 272 2021-07-31T01:54:59.574 INFO:tasks.rados.rados.0.smithi045.stdout:2875: finishing write tid 1 to smithi04569269-1 2021-07-31T01:54:59.575 INFO:tasks.rados.rados.0.smithi045.stdout:2875: finishing write tid 2 to smithi04569269-1 2021-07-31T01:54:59.575 INFO:tasks.rados.rados.0.smithi045.stdout:2875: finishing write tid 3 to smithi04569269-1 2021-07-31T01:54:59.575 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 1 v 1584 (ObjNum 884 snap 272 seq_num 884) dirty exists 2021-07-31T01:54:59.575 INFO:tasks.rados.rados.0.smithi045.stdout:2875: left oid 1 (ObjNum 884 snap 272 seq_num 884) 2021-07-31T01:54:59.576 INFO:tasks.rados.rados.0.smithi045.stdout:2876: expect (ObjNum 791 snap 244 seq_num 791) 2021-07-31T01:54:59.609 INFO:tasks.rados.rados.0.smithi045.stdout:2874: finishing copy_from to smithi04569269-16 2021-07-31T01:54:59.610 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 16 v 1635 (ObjNum 841 snap 265 seq_num 841) dirty exists 2021-07-31T01:54:59.610 INFO:tasks.rados.rados.0.smithi045.stdout:2874: finishing copy_from racing read to smithi04569269-16 2021-07-31T01:54:59.610 INFO:tasks.rados.rados.0.smithi045.stdout:2879: seq_num 886 ranges {0=442368} 2021-07-31T01:54:59.612 INFO:tasks.rados.rados.0.smithi045.stdout:2879: writing smithi04569269-30 from 0 to 442368 tid 1 2021-07-31T01:54:59.612 INFO:tasks.rados.rados.0.smithi045.stdout:2874: done (5 left) 2021-07-31T01:54:59.613 INFO:tasks.rados.rados.0.smithi045.stdout:2875: done (4 left) 2021-07-31T01:54:59.613 INFO:tasks.rados.rados.0.smithi045.stdout:2876: done (3 left) 2021-07-31T01:54:59.613 INFO:tasks.rados.rados.0.smithi045.stdout:2877: done (2 left) 2021-07-31T01:54:59.613 INFO:tasks.rados.rados.0.smithi045.stdout:2880: read oid 10 snap -1 2021-07-31T01:54:59.614 INFO:tasks.rados.rados.0.smithi045.stdout:2880: expect (ObjNum 847 snap 265 seq_num 847) 2021-07-31T01:54:59.614 INFO:tasks.rados.rados.0.smithi045.stdout:2881: setattr oid 18 current snap is 272 2021-07-31T01:54:59.614 INFO:tasks.rados.rados.0.smithi045.stdout:2878: finishing write tid 1 to smithi04569269-37 2021-07-31T01:54:59.614 INFO:tasks.rados.rados.0.smithi045.stdout:2878: finishing write tid 2 to smithi04569269-37 2021-07-31T01:54:59.615 INFO:tasks.rados.rados.0.smithi045.stdout:2878: finishing write tid 3 to smithi04569269-37 2021-07-31T01:54:59.615 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 37 v 1650 (ObjNum 885 snap 272 seq_num 885) dirty exists 2021-07-31T01:54:59.615 INFO:tasks.rados.rados.0.smithi045.stdout:2878: left oid 37 (ObjNum 885 snap 272 seq_num 885) 2021-07-31T01:54:59.616 INFO:tasks.rados.rados.0.smithi045.stdout:2878: done (3 left) 2021-07-31T01:54:59.616 INFO:tasks.rados.rados.0.smithi045.stdout:2882: delete oid 7 current snap is 272 2021-07-31T01:54:59.616 INFO:tasks.rados.rados.0.smithi045.stdout:2879: finishing write tid 1 to smithi04569269-30 2021-07-31T01:54:59.616 INFO:tasks.rados.rados.0.smithi045.stdout:2879: finishing write tid 2 to smithi04569269-30 2021-07-31T01:54:59.617 INFO:tasks.rados.rados.0.smithi045.stdout:2879: finishing write tid 3 to smithi04569269-30 2021-07-31T01:54:59.617 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 30 v 1633 (ObjNum 886 snap 272 seq_num 886) dirty exists 2021-07-31T01:54:59.618 INFO:tasks.rados.rados.0.smithi045.stdout:2879: left oid 30 (ObjNum 886 snap 272 seq_num 886) 2021-07-31T01:54:59.618 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 18 v 1651 (ObjNum 849 snap 265 seq_num 849) dirty exists 2021-07-31T01:54:59.628 INFO:tasks.rados.rados.0.smithi045.stdout:2879: done (3 left) 2021-07-31T01:54:59.628 INFO:tasks.rados.rados.0.smithi045.stdout:2881: done (2 left) 2021-07-31T01:54:59.628 INFO:tasks.rados.rados.0.smithi045.stdout:2882: done (1 left) 2021-07-31T01:54:59.628 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 43 current snap is 272 2021-07-31T01:54:59.629 INFO:tasks.rados.rados.0.smithi045.stdout:2883: seq_num 888 ranges {9224192=475136} 2021-07-31T01:54:59.681 INFO:tasks.rados.rados.0.smithi045.stdout:2883: writing smithi04569269-43 from 9224192 to 9699328 tid 1 2021-07-31T01:54:59.681 INFO:tasks.rados.rados.0.smithi045.stdout:2884: delete oid 15 current snap is 272 2021-07-31T01:54:59.681 INFO:tasks.rados.rados.0.smithi045.stdout:2880: expect (ObjNum 847 snap 265 seq_num 847) 2021-07-31T01:55:00.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:54:59 smithi154 conmon[34553]: cluster 2021-07-31T01:54:58.962683+0000 mon.a (mon.0) 1253 : cluster [DBG] osdmap e594: 8 total, 8 up, 8 in 2021-07-31T01:55:00.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:59 smithi045 conmon[39718]: cluster 2021-07-31T01:54:58. 2021-07-31T01:55:00.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:54:59 smithi045 conmon[39718]: 962683+0000 mon.a (mon.0) 1253 : cluster [DBG] osdmap e594: 8 total, 8 up, 8 in 2021-07-31T01:55:00.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:59 smithi045 conmon[33655]: cluster 2021-07-31T01:54:58.962683+0000 mon.a 2021-07-31T01:55:00.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:54:59 smithi045 conmon[33655]: (mon.0) 1253 : cluster [DBG] osdmap e594: 8 total, 8 up, 8 in 2021-07-31T01:55:00.747 INFO:tasks.rados.rados.0.smithi045.stdout:2883: finishing write tid 1 to smithi04569269-43 2021-07-31T01:55:00.748 INFO:tasks.rados.rados.0.smithi045.stdout:2883: finishing write tid 2 to smithi04569269-43 2021-07-31T01:55:00.748 INFO:tasks.rados.rados.0.smithi045.stdout:2883: finishing write tid 3 to smithi04569269-43 2021-07-31T01:55:00.748 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 43 v 1640 (ObjNum 888 snap 272 seq_num 888) dirty exists 2021-07-31T01:55:00.748 INFO:tasks.rados.rados.0.smithi045.stdout:2883: left oid 43 (ObjNum 888 snap 272 seq_num 888) 2021-07-31T01:55:00.748 INFO:tasks.rados.rados.0.smithi045.stdout:2880: done (2 left) 2021-07-31T01:55:00.749 INFO:tasks.rados.rados.0.smithi045.stdout:2883: done (1 left) 2021-07-31T01:55:00.749 INFO:tasks.rados.rados.0.smithi045.stdout:2884: done (0 left) 2021-07-31T01:55:00.749 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 49 current snap is 272 2021-07-31T01:55:00.749 INFO:tasks.rados.rados.0.smithi045.stdout:2885: seq_num 890 ranges {14344192=507904} 2021-07-31T01:55:00.828 INFO:tasks.rados.rados.0.smithi045.stdout:2885: writing smithi04569269-49 from 14344192 to 14852096 tid 1 2021-07-31T01:55:00.828 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 14 current snap is 272 2021-07-31T01:55:00.828 INFO:tasks.rados.rados.0.smithi045.stdout:2886: seq_num 891 ranges {14344192=532480} 2021-07-31T01:55:00.909 INFO:tasks.rados.rados.0.smithi045.stdout:2886: writing smithi04569269-14 from 14344192 to 14876672 tid 1 2021-07-31T01:55:00.909 INFO:tasks.rados.rados.0.smithi045.stdout:2887: rollback oid 26 current snap is 272 2021-07-31T01:55:00.910 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 26 to 244 2021-07-31T01:55:00.910 INFO:tasks.rados.rados.0.smithi045.stdout:2885: finishing write tid 1 to smithi04569269-49 2021-07-31T01:55:00.910 INFO:tasks.rados.rados.0.smithi045.stdout:2888: delete oid 38 current snap is 272 2021-07-31T01:55:00.910 INFO:tasks.rados.rados.0.smithi045.stdout:2885: finishing write tid 2 to smithi04569269-49 2021-07-31T01:55:00.911 INFO:tasks.rados.rados.0.smithi045.stdout:2885: finishing write tid 3 to smithi04569269-49 2021-07-31T01:55:00.911 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 49 v 1836 (ObjNum 890 snap 272 seq_num 890) dirty exists 2021-07-31T01:55:00.911 INFO:tasks.rados.rados.0.smithi045.stdout:2885: left oid 49 (ObjNum 890 snap 272 seq_num 890) 2021-07-31T01:55:00.912 INFO:tasks.rados.rados.0.smithi045.stdout:2885: done (3 left) 2021-07-31T01:55:00.912 INFO:tasks.rados.rados.0.smithi045.stdout:2888: done (2 left) 2021-07-31T01:55:00.912 INFO:tasks.rados.rados.0.smithi045.stdout:2889: delete oid 34 current snap is 272 2021-07-31T01:55:00.913 INFO:tasks.rados.rados.0.smithi045.stdout:2887: finishing rollback tid 0 to smithi04569269-26 2021-07-31T01:55:00.914 INFO:tasks.rados.rados.0.smithi045.stdout:2887: finishing rollback tid 1 to smithi04569269-26 2021-07-31T01:55:00.915 INFO:tasks.rados.rados.0.smithi045.stdout:2886: finishing write tid 1 to smithi04569269-14 2021-07-31T01:55:00.915 INFO:tasks.rados.rados.0.smithi045.stdout:2887: finishing rollback tid 2 to smithi04569269-26 2021-07-31T01:55:00.915 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 26 v 1413 (ObjNum 760 snap 233 seq_num 760) dirty exists 2021-07-31T01:55:00.915 INFO:tasks.rados.rados.0.smithi045.stdout:2887: done (2 left) 2021-07-31T01:55:00.916 INFO:tasks.rados.rados.0.smithi045.stdout:2889: done (1 left) 2021-07-31T01:55:00.916 INFO:tasks.rados.rados.0.smithi045.stdout:2890: read oid 24 snap 244 2021-07-31T01:55:00.916 INFO:tasks.rados.rados.0.smithi045.stdout:2890: expect deleted 2021-07-31T01:55:00.916 INFO:tasks.rados.rados.0.smithi045.stdout:2891: snap_remove snap 270 2021-07-31T01:55:00.973 INFO:tasks.rados.rados.0.smithi045.stdout:2891: done (2 left) 2021-07-31T01:55:00.973 INFO:tasks.rados.rados.0.smithi045.stdout:2892: delete oid 8 current snap is 272 2021-07-31T01:55:00.974 INFO:tasks.rados.rados.0.smithi045.stdout:2886: finishing write tid 2 to smithi04569269-14 2021-07-31T01:55:00.974 INFO:tasks.rados.rados.0.smithi045.stdout:2886: finishing write tid 3 to smithi04569269-14 2021-07-31T01:55:00.974 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 14 v 1642 (ObjNum 891 snap 272 seq_num 891) dirty exists 2021-07-31T01:55:00.974 INFO:tasks.rados.rados.0.smithi045.stdout:2886: left oid 14 (ObjNum 891 snap 272 seq_num 891) 2021-07-31T01:55:00.985 INFO:tasks.rados.rados.0.smithi045.stdout:2886: done (2 left) 2021-07-31T01:55:00.985 INFO:tasks.rados.rados.0.smithi045.stdout:2890: done (1 left) 2021-07-31T01:55:00.986 INFO:tasks.rados.rados.0.smithi045.stdout:2892: done (0 left) 2021-07-31T01:55:00.986 INFO:tasks.rados.rados.0.smithi045.stdout:2893: delete oid 22 current snap is 272 2021-07-31T01:55:00.988 INFO:tasks.rados.rados.0.smithi045.stdout:2893: done (0 left) 2021-07-31T01:55:00.988 INFO:tasks.rados.rados.0.smithi045.stdout:2894: rollback oid 41 current snap is 272 2021-07-31T01:55:00.989 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 41 to 265 2021-07-31T01:55:00.989 INFO:tasks.rados.rados.0.smithi045.stdout:2895: rollback oid 19 current snap is 272 2021-07-31T01:55:00.989 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 19 to 271 2021-07-31T01:55:00.990 INFO:tasks.rados.rados.0.smithi045.stdout:2896: rollback oid 24 current snap is 272 2021-07-31T01:55:00.991 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 24 to 249 2021-07-31T01:55:00.991 INFO:tasks.rados.rados.0.smithi045.stdout:2897: snap_create 2021-07-31T01:55:00.992 INFO:tasks.rados.rados.0.smithi045.stdout:2896: finishing rollback tid 0 to smithi04569269-24 2021-07-31T01:55:00.992 INFO:tasks.rados.rados.0.smithi045.stdout:2896: finishing rollback tid 1 to smithi04569269-24 2021-07-31T01:55:00.992 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 24 v 1635 (ObjNum -1104917680 snap 21873 seq_num 3190516608) dirty dne 2021-07-31T01:55:00.992 INFO:tasks.rados.rados.0.smithi045.stdout:2894: finishing rollback tid 0 to smithi04569269-41 2021-07-31T01:55:00.993 INFO:tasks.rados.rados.0.smithi045.stdout:2894: finishing rollback tid 1 to smithi04569269-41 2021-07-31T01:55:00.993 INFO:tasks.rados.rados.0.smithi045.stdout:2895: finishing rollback tid 1 to smithi04569269-19 2021-07-31T01:55:00.994 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 19 v 0 (ObjNum 1953066355 snap 875587944 seq_num 842610229) dirty dne 2021-07-31T01:55:00.994 INFO:tasks.rados.rados.0.smithi045.stdout:2894: finishing rollback tid 2 to smithi04569269-41 2021-07-31T01:55:00.994 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 41 v 1638 (ObjNum 701 snap 218 seq_num 701) dirty exists 2021-07-31T01:55:01.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:00 smithi154 conmon[34553]: cluster 2021-07-31T01:54:59.902148+0000 mgr.x (mgr.24542) 331 : cluster [DBG] pgmap v674: 25 pgs: 25 active+clean; 378 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 18 MiB/s rd, 4.7 MiB/s wr, 16 op/s 2021-07-31T01:55:01.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:00 smithi045 conmon[39718]: cluster 2021-07-31T01:54:59. 2021-07-31T01:55:01.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:00 smithi045 conmon[39718]: 902148+0000 mgr.x (mgr.24542) 331 : cluster [DBG] pgmap v674: 25 pgs: 25 active+clean; 378 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 18 MiB/s rd, 4.7 MiB/s wr, 16 op/s 2021-07-31T01:55:01.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:00 smithi045 conmon[33655]: cluster 2021-07-31T01:54:59.902148+0000 mgr.x (mgr 2021-07-31T01:55:01.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:00 smithi045 conmon[33655]: .24542) 331 : cluster [DBG] pgmap v674: 25 pgs: 25 active+clean; 378 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 18 MiB/s rd, 4.7 MiB/s wr, 16 op/s 2021-07-31T01:55:01.977 INFO:tasks.rados.rados.0.smithi045.stdout:2894: done (3 left) 2021-07-31T01:55:01.978 INFO:tasks.rados.rados.0.smithi045.stdout:2895: done (2 left) 2021-07-31T01:55:01.978 INFO:tasks.rados.rados.0.smithi045.stdout:2896: done (1 left) 2021-07-31T01:55:01.978 INFO:tasks.rados.rados.0.smithi045.stdout:2897: done (0 left) 2021-07-31T01:55:01.978 INFO:tasks.rados.rados.0.smithi045.stdout:2898: read oid 49 snap -1 2021-07-31T01:55:01.979 INFO:tasks.rados.rados.0.smithi045.stdout:2898: expect (ObjNum 890 snap 272 seq_num 890) 2021-07-31T01:55:01.979 INFO:tasks.rados.rados.0.smithi045.stdout:2899: snap_remove snap 269 2021-07-31T01:55:02.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:01 smithi154 conmon[34553]: cluster 2021-07-31T01:55:00.971436+0000 mon.a (mon.0) 1254 : cluster [DBG] osdmap e595: 8 total, 8 up, 8 in 2021-07-31T01:55:02.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:01 smithi045 conmon[39718]: cluster 2021-07-31T01:55:00.971436+0000 mon.a (mon 2021-07-31T01:55:02.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:01 smithi045 conmon[39718]: .0) 1254 : cluster [DBG] osdmap e595: 8 total, 8 up, 8 in 2021-07-31T01:55:02.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:01 smithi045 conmon[33655]: cluster 2021-07-31T01:55:00.971436+0000 mon.a (mon.0) 1254 2021-07-31T01:55:02.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:01 smithi045 conmon[33655]: : cluster [DBG] osdmap e595: 8 total, 8 up, 8 in 2021-07-31T01:55:02.985 INFO:tasks.rados.rados.0.smithi045.stdout:2899: done (1 left) 2021-07-31T01:55:02.985 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 30 current snap is 273 2021-07-31T01:55:02.985 INFO:tasks.rados.rados.0.smithi045.stdout:2900: seq_num 896 ranges {442368=614400} 2021-07-31T01:55:02.991 INFO:tasks.rados.rados.0.smithi045.stdout:2900: writing smithi04569269-30 from 442368 to 1056768 tid 1 2021-07-31T01:55:02.991 INFO:tasks.rados.rados.0.smithi045.stdout:2901: rollback oid 19 current snap is 273 2021-07-31T01:55:02.992 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 19 to 263 2021-07-31T01:55:02.992 INFO:tasks.rados.rados.0.smithi045.stdout:2902: snap_remove snap 265 2021-07-31T01:55:03.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:02 smithi154 conmon[34553]: cluster 2021-07-31T01:55:01.902714+0000 mgr.x (mgr.24542) 332 : cluster [DBG] 2021-07-31T01:55:03.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:02 smithi154 conmon[34553]: pgmap v676: 25 pgs: 25 active+clean; 378 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 16 MiB/s rd, 4.1 MiB/s wr, 14 op/s 2021-07-31T01:55:03.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:02 smithi154 conmon[34553]: cluster 2021-07-31T01:55:01.976815+0000 mon.a (mon.0) 1255 : cluster [DBG] osdmap e596: 8 total, 8 up, 8 in 2021-07-31T01:55:03.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:02 smithi045 conmon[33655]: cluster 2021-07-31T01:55:01.902714+0000 2021-07-31T01:55:03.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:02 smithi045 conmon[33655]: mgr.x (mgr.24542) 332 : cluster [DBG] pgmap v676: 25 pgs: 25 active+clean; 378 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 16 MiB/s rd, 4.1 MiB/s wr, 14 op/s 2021-07-31T01:55:03.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:02 smithi045 conmon[33655]: cluster 2021-07-31T01:55:01.976815+0000 mon.a (mon.0) 2021-07-31T01:55:03.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:02 smithi045 conmon[33655]: 1255 : cluster [DBG] osdmap e596: 8 total, 8 up, 8 in 2021-07-31T01:55:03.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:02 smithi045 conmon[39718]: cluster 2021-07-31T01:55:01.902714+0000 mgr.x (mgr.24542) 2021-07-31T01:55:03.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:02 smithi045 conmon[39718]: 332 : cluster [DBG] pgmap v676: 25 pgs: 25 active+clean; 378 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 16 MiB/s rd, 4.1 MiB/s wr, 14 op/s 2021-07-31T01:55:03.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:02 smithi045 conmon[39718]: cluster 2021-07-31T01:55:01.976815+0000 mon.a (mon.0) 1255 : cluster [DBG] osdmap e596: 8 total, 8 up, 8 in 2021-07-31T01:55:03.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:02 smithi045 conmon[39718]: 2021-07-31T01:55:03.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:55:03 smithi045 conmon[61210]: level=error ts=2021-07-31T01:55:03.522Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:55:03.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:55:03 smithi045 conmon[61210]: level=error ts=2021-07-31T01:55:03.522Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:55:03.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:55:03 smithi045 conmon[61210]: level=error ts=2021-07-31T01:55:03.522Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T01:55:03.990 INFO:tasks.rados.rados.0.smithi045.stdout:2902: done (3 left) 2021-07-31T01:55:03.990 INFO:tasks.rados.rados.0.smithi045.stdout:2903: rmattr oid 44 current snap is 273 2021-07-31T01:55:03.990 INFO:tasks.rados.rados.0.smithi045.stdout:2903: done (3 left) 2021-07-31T01:55:03.990 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 28 current snap is 273 2021-07-31T01:55:03.991 INFO:tasks.rados.rados.0.smithi045.stdout:2904: seq_num 897 ranges {0=630784} 2021-07-31T01:55:03.994 INFO:tasks.rados.rados.0.smithi045.stdout:2904: writing smithi04569269-28 from 0 to 630784 tid 1 2021-07-31T01:55:03.994 INFO:tasks.rados.rados.0.smithi045.stdout:2905: snap_remove snap 249 2021-07-31T01:55:04.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:03 smithi154 conmon[34553]: cluster 2021-07-31T01:55:02.983904+0000 mon.a (mon.0) 1256 : cluster [DBG] osdmap e597: 8 total, 8 up, 8 in 2021-07-31T01:55:04.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:03 smithi045 conmon[39718]: cluster 2021-07-31T01:55:02.983904+0000 mon.a ( 2021-07-31T01:55:04.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:03 smithi045 conmon[39718]: mon.0) 1256 : cluster [DBG] osdmap e597: 8 total, 8 up, 8 in 2021-07-31T01:55:04.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:03 smithi045 conmon[33655]: cluster 2021-07-31T01:55:02.983904+0000 2021-07-31T01:55:04.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:03 smithi045 conmon[33655]: mon.a (mon.0) 1256 : cluster [DBG] osdmap e597: 8 total, 8 up, 8 in 2021-07-31T01:55:04.995 INFO:tasks.rados.rados.0.smithi045.stdout:2905: done (4 left) 2021-07-31T01:55:04.995 INFO:tasks.rados.rados.0.smithi045.stdout:2906: copy_from oid 9 from oid 15 current snap is 273 2021-07-31T01:55:04.996 INFO:tasks.rados.rados.0.smithi045.stdout:2898: expect (ObjNum 890 snap 272 seq_num 890) 2021-07-31T01:55:05.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:04 smithi154 conmon[34553]: cluster 2021-07-31T01:55:03.903273+0000 mgr.x (mgr.24542) 333 : cluster [DBG] pgmap v679: 25 pgs: 25 active+clean; 403 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 16 MiB/s rd, 4.0 MiB/s wr, 16 op/s 2021-07-31T01:55:05.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:04 smithi154 conmon[34553]: 2021-07-31T01:55:05.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:04 smithi154 conmon[34553]: cluster 2021-07-31T01:55:03.988930+0000 mon.a (mon.0) 1257 : cluster [DBG] osdmap e598: 8 total, 8 up, 8 in 2021-07-31T01:55:05.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:04 smithi045 conmon[39718]: cluster 2021-07-31T01:55 2021-07-31T01:55:05.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:04 smithi045 conmon[39718]: :03.903273+0000 mgr.x (mgr.24542) 333 : cluster [DBG] pgmap v679: 25 pgs: 25 active+clean; 403 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 16 MiB/s rd, 4.0 MiB/s wr, 16 op/s 2021-07-31T01:55:05.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:04 smithi045 conmon[39718]: cluster 2021-07 2021-07-31T01:55:05.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:04 smithi045 conmon[39718]: -31T01:55:03.988930+0000 mon.a (mon.0) 1257 2021-07-31T01:55:05.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:04 smithi045 conmon[39718]: : cluster [DBG] osdmap e598: 8 total, 8 up, 8 in 2021-07-31T01:55:05.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:04 smithi045 conmon[33655]: cluster 2021-07-31T01:55:03.903273+0000 mgr.x (mgr.24542) 2021-07-31T01:55:05.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:04 smithi045 conmon[33655]: 333 : cluster [DBG] pgmap v679: 25 pgs: 25 active+clean; 403 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 16 MiB/s rd, 4.0 MiB/s wr, 16 op/s 2021-07-31T01:55:05.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:04 smithi045 conmon[33655]: cluster 2021-07-31T01:55:03.988930+0000 mon.a (mon.0) 1257 : cluster [DBG] osdmap e598: 8 total, 8 up, 8 in 2021-07-31T01:55:06.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:05 smithi154 conmon[34553]: cluster 2021-07-31T01:55:04.994611+0000 mon.a (mon.0) 1258 : cluster [DBG] osdmap e599: 8 total, 8 up, 8 in 2021-07-31T01:55:06.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:05 smithi045 conmon[39718]: cluster 2021-07-31T01:55:04. 2021-07-31T01:55:06.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:05 smithi045 conmon[39718]: 994611+0000 mon.a (mon.0) 1258 : cluster [DBG] osdmap e599: 8 total, 8 up, 8 in 2021-07-31T01:55:06.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:05 smithi045 conmon[33655]: cluster 2021-07-31T01:55:04.994611+0000 mon.a (mon.0 2021-07-31T01:55:06.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:05 smithi045 conmon[33655]: ) 1258 : cluster [DBG] osdmap e599: 8 total, 8 up, 8 in 2021-07-31T01:55:06.616 INFO:tasks.rados.rados.0.smithi045.stdout:2901: finishing rollback tid 1 to smithi04569269-19 2021-07-31T01:55:06.617 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 19 v 0 (ObjNum 0 snap 32731 seq_num 3189759008) dirty dne 2021-07-31T01:55:06.617 INFO:tasks.rados.rados.0.smithi045.stdout:2900: finishing write tid 1 to smithi04569269-30 2021-07-31T01:55:06.617 INFO:tasks.rados.rados.0.smithi045.stdout:2898: done (4 left) 2021-07-31T01:55:06.618 INFO:tasks.rados.rados.0.smithi045.stdout:2901: done (3 left) 2021-07-31T01:55:06.618 INFO:tasks.rados.rados.0.smithi045.stdout:2907: copy_from oid 25 from oid 1 current snap is 273 2021-07-31T01:55:06.618 INFO:tasks.rados.rados.0.smithi045.stdout:2900: finishing write tid 2 to smithi04569269-30 2021-07-31T01:55:06.618 INFO:tasks.rados.rados.0.smithi045.stdout:2900: finishing write tid 3 to smithi04569269-30 2021-07-31T01:55:06.619 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 30 v 1637 (ObjNum 896 snap 273 seq_num 896) dirty exists 2021-07-31T01:55:06.619 INFO:tasks.rados.rados.0.smithi045.stdout:2900: left oid 30 (ObjNum 896 snap 273 seq_num 896) 2021-07-31T01:55:06.619 INFO:tasks.rados.rados.0.smithi045.stdout:2904: finishing write tid 1 to smithi04569269-28 2021-07-31T01:55:06.619 INFO:tasks.rados.rados.0.smithi045.stdout:2904: finishing write tid 2 to smithi04569269-28 2021-07-31T01:55:06.620 INFO:tasks.rados.rados.0.smithi045.stdout:2904: finishing write tid 3 to smithi04569269-28 2021-07-31T01:55:06.620 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 28 v 1658 (ObjNum 897 snap 273 seq_num 897) dirty exists 2021-07-31T01:55:06.620 INFO:tasks.rados.rados.0.smithi045.stdout:2904: left oid 28 (ObjNum 897 snap 273 seq_num 897) 2021-07-31T01:55:06.621 INFO:tasks.rados.rados.0.smithi045.stdout:2906: finishing copy_from to smithi04569269-9 2021-07-31T01:55:06.621 INFO:tasks.rados.rados.0.smithi045.stdout:2906: got expected ENOENT (src dne) 2021-07-31T01:55:06.621 INFO:tasks.rados.rados.0.smithi045.stdout:2906: finishing copy_from racing read to smithi04569269-9 2021-07-31T01:55:06.622 INFO:tasks.rados.rados.0.smithi045.stdout:2900: done (3 left) 2021-07-31T01:55:06.622 INFO:tasks.rados.rados.0.smithi045.stdout:2904: done (2 left) 2021-07-31T01:55:06.622 INFO:tasks.rados.rados.0.smithi045.stdout:2906: done (1 left) 2021-07-31T01:55:06.622 INFO:tasks.rados.rados.0.smithi045.stdout:2908: setattr oid 22 current snap is 273 2021-07-31T01:55:06.623 INFO:tasks.rados.rados.0.smithi045.stdout:2909: delete oid 14 current snap is 273 2021-07-31T01:55:06.624 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 22 v 1637 (ObjNum 0 snap 0 seq_num 0) dirty exists 2021-07-31T01:55:06.624 INFO:tasks.rados.rados.0.smithi045.stdout:2908: done (2 left) 2021-07-31T01:55:06.624 INFO:tasks.rados.rados.0.smithi045.stdout:2909: done (1 left) 2021-07-31T01:55:06.624 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 45 current snap is 273 2021-07-31T01:55:06.625 INFO:tasks.rados.rados.0.smithi045.stdout:2910: seq_num 899 ranges {0=663552} 2021-07-31T01:55:06.625 INFO:tasks.rados.rados.0.smithi045.stdout:2910: writing smithi04569269-45 from 0 to 663552 tid 1 2021-07-31T01:55:06.626 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 7 current snap is 273 2021-07-31T01:55:06.626 INFO:tasks.rados.rados.0.smithi045.stdout:2911: seq_num 900 ranges {0=679936} 2021-07-31T01:55:06.630 INFO:tasks.rados.rados.0.smithi045.stdout:2911: writing smithi04569269-7 from 0 to 679936 tid 1 2021-07-31T01:55:06.630 INFO:tasks.rados.rados.0.smithi045.stdout:2912: setattr oid 18 current snap is 273 2021-07-31T01:55:06.630 INFO:tasks.rados.rados.0.smithi045.stdout:2910: finishing write tid 1 to smithi04569269-45 2021-07-31T01:55:06.631 INFO:tasks.rados.rados.0.smithi045.stdout:2910: finishing write tid 2 to smithi04569269-45 2021-07-31T01:55:06.632 INFO:tasks.rados.rados.0.smithi045.stdout:2913: rollback oid 44 current snap is 273 2021-07-31T01:55:06.632 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 44 to 266 2021-07-31T01:55:06.632 INFO:tasks.rados.rados.0.smithi045.stdout:2914: setattr oid 33 current snap is 273 2021-07-31T01:55:06.632 INFO:tasks.rados.rados.0.smithi045.stdout:2910: finishing write tid 3 to smithi04569269-45 2021-07-31T01:55:06.633 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 45 v 1660 (ObjNum 899 snap 273 seq_num 899) dirty exists 2021-07-31T01:55:06.633 INFO:tasks.rados.rados.0.smithi045.stdout:2910: left oid 45 (ObjNum 899 snap 273 seq_num 899) 2021-07-31T01:55:06.633 INFO:tasks.rados.rados.0.smithi045.stdout:2910: done (5 left) 2021-07-31T01:55:06.633 INFO:tasks.rados.rados.0.smithi045.stdout:2915: snap_create 2021-07-31T01:55:06.635 INFO:tasks.rados.rados.0.smithi045.stdout:2913: finishing rollback tid 1 to smithi04569269-44 2021-07-31T01:55:06.635 INFO:tasks.rados.rados.0.smithi045.stdout:2913: finishing rollback tid 2 to smithi04569269-44 2021-07-31T01:55:06.636 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 44 v 1645 (ObjNum 704 snap 219 seq_num 704) dirty exists 2021-07-31T01:55:06.637 INFO:tasks.rados.rados.0.smithi045.stdout:2911: finishing write tid 1 to smithi04569269-7 2021-07-31T01:55:06.638 INFO:tasks.rados.rados.0.smithi045.stdout:2911: finishing write tid 2 to smithi04569269-7 2021-07-31T01:55:06.638 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 18 v 1653 (ObjNum 849 snap 265 seq_num 849) dirty exists 2021-07-31T01:55:06.641 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 33 v 1655 (ObjNum 726 snap 226 seq_num 726) dirty exists 2021-07-31T01:55:06.641 INFO:tasks.rados.rados.0.smithi045.stdout:2911: finishing write tid 3 to smithi04569269-7 2021-07-31T01:55:06.641 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 7 v 1640 (ObjNum 900 snap 273 seq_num 900) dirty exists 2021-07-31T01:55:06.642 INFO:tasks.rados.rados.0.smithi045.stdout:2911: left oid 7 (ObjNum 900 snap 273 seq_num 900) 2021-07-31T01:55:06.643 INFO:tasks.rados.rados.0.smithi045.stdout:2907: finishing copy_from to smithi04569269-25 2021-07-31T01:55:06.644 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 25 v 1584 (ObjNum 884 snap 272 seq_num 884) dirty exists 2021-07-31T01:55:06.644 INFO:tasks.rados.rados.0.smithi045.stdout:2907: finishing copy_from racing read to smithi04569269-25 2021-07-31T01:55:07.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:06 smithi154 conmon[34553]: cluster 2021-07-31T01:55:05.903856+0000 mgr.x (mgr.24542) 334 : cluster 2021-07-31T01:55:07.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:06 smithi154 conmon[34553]: [DBG] pgmap v682: 25 pgs: 25 active+clean; 403 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 20 MiB/s rd, 4.9 MiB/s wr, 20 op/s 2021-07-31T01:55:07.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:06 smithi154 conmon[34553]: cluster 2021-07-31T01:55:06.625141+0000 mon.a (mon.0) 1259 : cluster [DBG] osdmap e600: 8 total, 8 up, 8 in 2021-07-31T01:55:07.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:06 smithi045 conmon[39718]: cluster 2021-07-31T01:55:05.903856 2021-07-31T01:55:07.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:06 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 334 : cluster [DBG] pgmap v682: 25 pgs: 25 active+clean; 403 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 20 MiB/s rd, 4.9 MiB/s wr, 20 op/s 2021-07-31T01:55:07.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:06 smithi045 conmon[39718]: cluster 2021-07-31T01:55:07.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:06 smithi045 conmon[39718]: 2021-07-31T01:55:06 2021-07-31T01:55:07.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:06 smithi045 conmon[39718]: .625141+0000 mon.a (mon.0) 2021-07-31T01:55:07.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:06 smithi045 conmon[39718]: 1259 : cluster [DBG] osdmap e600: 8 total, 8 up, 8 in 2021-07-31T01:55:07.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:06 smithi045 conmon[33655]: cluster 2021-07-31T01:55:05.903856+0000 mgr.x (mgr 2021-07-31T01:55:07.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:06 smithi045 conmon[33655]: .24542) 334 : cluster [DBG] pgmap v682: 25 pgs: 25 active+clean; 403 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 20 MiB/s rd, 4.9 MiB/s wr, 20 op/s 2021-07-31T01:55:07.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:06 smithi045 conmon[33655]: cluster 2021-07 2021-07-31T01:55:07.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:06 smithi045 conmon[33655]: -31T01:55:06.625141+0000 mon.a (mon.0) 1259 : cluster 2021-07-31T01:55:07.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:06 smithi045 conmon[33655]: [DBG] osdmap e600: 8 total, 8 up, 8 in 2021-07-31T01:55:07.632 INFO:tasks.rados.rados.0.smithi045.stdout:2907: done (5 left) 2021-07-31T01:55:07.632 INFO:tasks.rados.rados.0.smithi045.stdout:2911: done (4 left) 2021-07-31T01:55:07.633 INFO:tasks.rados.rados.0.smithi045.stdout:2912: done (3 left) 2021-07-31T01:55:07.633 INFO:tasks.rados.rados.0.smithi045.stdout:2913: done (2 left) 2021-07-31T01:55:07.633 INFO:tasks.rados.rados.0.smithi045.stdout:2914: done (1 left) 2021-07-31T01:55:07.634 INFO:tasks.rados.rados.0.smithi045.stdout:2915: done (0 left) 2021-07-31T01:55:07.634 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 45 current snap is 274 2021-07-31T01:55:07.635 INFO:tasks.rados.rados.0.smithi045.stdout:2916: seq_num 901 ranges {663552=696320} 2021-07-31T01:55:07.640 INFO:tasks.rados.rados.0.smithi045.stdout:2916: writing smithi04569269-45 from 663552 to 1359872 tid 1 2021-07-31T01:55:07.640 INFO:tasks.rados.rados.0.smithi045.stdout:2917: read oid 41 snap 255 2021-07-31T01:55:07.640 INFO:tasks.rados.rados.0.smithi045.stdout:2917: expect (ObjNum 701 snap 218 seq_num 701) 2021-07-31T01:55:07.641 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 35 current snap is 274 2021-07-31T01:55:07.641 INFO:tasks.rados.rados.0.smithi045.stdout:2918: seq_num 902 ranges {2211840=712704} 2021-07-31T01:55:07.656 INFO:tasks.rados.rados.0.smithi045.stdout:2918: writing smithi04569269-35 from 2211840 to 2924544 tid 1 2021-07-31T01:55:07.657 INFO:tasks.rados.rados.0.smithi045.stdout:2919: delete oid 40 current snap is 274 2021-07-31T01:55:07.657 INFO:tasks.rados.rados.0.smithi045.stdout:2916: finishing write tid 1 to smithi04569269-45 2021-07-31T01:55:07.657 INFO:tasks.rados.rados.0.smithi045.stdout:2916: finishing write tid 2 to smithi04569269-45 2021-07-31T01:55:07.657 INFO:tasks.rados.rados.0.smithi045.stdout:2916: finishing write tid 3 to smithi04569269-45 2021-07-31T01:55:07.658 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 45 v 1662 (ObjNum 901 snap 274 seq_num 901) dirty exists 2021-07-31T01:55:07.658 INFO:tasks.rados.rados.0.smithi045.stdout:2916: left oid 45 (ObjNum 901 snap 274 seq_num 901) 2021-07-31T01:55:07.661 INFO:tasks.rados.rados.0.smithi045.stdout:2916: done (3 left) 2021-07-31T01:55:07.662 INFO:tasks.rados.rados.0.smithi045.stdout:2919: done (2 left) 2021-07-31T01:55:07.662 INFO:tasks.rados.rados.0.smithi045.stdout:2920: copy_from oid 10 from oid 50 current snap is 274 2021-07-31T01:55:07.662 INFO:tasks.rados.rados.0.smithi045.stdout:2921: setattr oid 26 current snap is 274 2021-07-31T01:55:07.662 INFO:tasks.rados.rados.0.smithi045.stdout:2917: expect (ObjNum 701 snap 218 seq_num 701) 2021-07-31T01:55:07.791 INFO:tasks.rados.rados.0.smithi045.stdout:2918: finishing write tid 1 to smithi04569269-35 2021-07-31T01:55:07.791 INFO:tasks.rados.rados.0.smithi045.stdout:2918: finishing write tid 2 to smithi04569269-35 2021-07-31T01:55:07.791 INFO:tasks.rados.rados.0.smithi045.stdout:2918: finishing write tid 3 to smithi04569269-35 2021-07-31T01:55:07.791 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 35 v 1875 (ObjNum 902 snap 274 seq_num 902) dirty exists 2021-07-31T01:55:07.792 INFO:tasks.rados.rados.0.smithi045.stdout:2918: left oid 35 (ObjNum 902 snap 274 seq_num 902) 2021-07-31T01:55:07.792 INFO:tasks.rados.rados.0.smithi045.stdout:2920: finishing copy_from to smithi04569269-10 2021-07-31T01:55:07.792 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 10 v 1640 (ObjNum 837 snap 263 seq_num 837) dirty exists 2021-07-31T01:55:07.792 INFO:tasks.rados.rados.0.smithi045.stdout:2920: finishing copy_from racing read to smithi04569269-10 2021-07-31T01:55:07.793 INFO:tasks.rados.rados.0.smithi045.stdout:2917: done (3 left) 2021-07-31T01:55:07.793 INFO:tasks.rados.rados.0.smithi045.stdout:2918: done (2 left) 2021-07-31T01:55:07.793 INFO:tasks.rados.rados.0.smithi045.stdout:2920: done (1 left) 2021-07-31T01:55:07.793 INFO:tasks.rados.rados.0.smithi045.stdout:2922: rollback oid 47 current snap is 274 2021-07-31T01:55:07.793 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 47 to 244 2021-07-31T01:55:07.794 INFO:tasks.rados.rados.0.smithi045.stdout:2923: rmattr oid 23 current snap is 274 2021-07-31T01:55:07.794 INFO:tasks.rados.rados.0.smithi045.stdout:2923: done (2 left) 2021-07-31T01:55:07.794 INFO:tasks.rados.rados.0.smithi045.stdout:2924: rollback oid 32 current snap is 274 2021-07-31T01:55:07.794 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 32 to 271 2021-07-31T01:55:07.795 INFO:tasks.rados.rados.0.smithi045.stdout:2925: copy_from oid 27 from oid 29 current snap is 274 2021-07-31T01:55:07.795 INFO:tasks.rados.rados.0.smithi045.stdout:2926: rollback oid 4 current snap is 274 2021-07-31T01:55:07.795 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 4 to 259 2021-07-31T01:55:07.795 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 1 current snap is 274 2021-07-31T01:55:07.796 INFO:tasks.rados.rados.0.smithi045.stdout:2927: seq_num 904 ranges {5791744=745472} 2021-07-31T01:55:07.827 INFO:tasks.rados.rados.0.smithi045.stdout:2927: writing smithi04569269-1 from 5791744 to 6537216 tid 1 2021-07-31T01:55:07.827 INFO:tasks.rados.rados.0.smithi045.stdout:2928: read oid 19 snap 263 2021-07-31T01:55:07.828 INFO:tasks.rados.rados.0.smithi045.stdout:2928: expect deleted 2021-07-31T01:55:07.828 INFO:tasks.rados.rados.0.smithi045.stdout:2929: read oid 21 snap -1 2021-07-31T01:55:07.828 INFO:tasks.rados.rados.0.smithi045.stdout:2929: expect (ObjNum 866 snap 270 seq_num 866) 2021-07-31T01:55:07.828 INFO:tasks.rados.rados.0.smithi045.stdout:2930: delete oid 11 current snap is 274 2021-07-31T01:55:07.828 INFO:tasks.rados.rados.0.smithi045.stdout:2922: finishing rollback tid 0 to smithi04569269-47 2021-07-31T01:55:07.829 INFO:tasks.rados.rados.0.smithi045.stdout:2926: finishing rollback tid 1 to smithi04569269-4 2021-07-31T01:55:07.829 INFO:tasks.rados.rados.0.smithi045.stdout:2922: finishing rollback tid 1 to smithi04569269-47 2021-07-31T01:55:07.829 INFO:tasks.rados.rados.0.smithi045.stdout:2926: finishing rollback tid 2 to smithi04569269-4 2021-07-31T01:55:07.829 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 4 v 1655 (ObjNum 716 snap 222 seq_num 716) dirty exists 2021-07-31T01:55:07.830 INFO:tasks.rados.rados.0.smithi045.stdout:2922: finishing rollback tid 2 to smithi04569269-47 2021-07-31T01:55:07.830 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 47 v 1634 (ObjNum 783 snap 239 seq_num 783) dirty exists 2021-07-31T01:55:07.830 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 26 v 1414 (ObjNum 760 snap 233 seq_num 760) dirty exists 2021-07-31T01:55:07.830 INFO:tasks.rados.rados.0.smithi045.stdout:2924: finishing rollback tid 0 to smithi04569269-32 2021-07-31T01:55:07.831 INFO:tasks.rados.rados.0.smithi045.stdout:2924: finishing rollback tid 1 to smithi04569269-32 2021-07-31T01:55:07.831 INFO:tasks.rados.rados.0.smithi045.stdout:2924: finishing rollback tid 2 to smithi04569269-32 2021-07-31T01:55:07.831 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 32 v 1417 (ObjNum 754 snap 233 seq_num 754) dirty exists 2021-07-31T01:55:07.831 INFO:tasks.rados.rados.0.smithi045.stdout:2925: finishing copy_from to smithi04569269-27 2021-07-31T01:55:07.831 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 27 v 1417 (ObjNum 846 snap 265 seq_num 846) dirty exists 2021-07-31T01:55:07.832 INFO:tasks.rados.rados.0.smithi045.stdout:2925: finishing copy_from racing read to smithi04569269-27 2021-07-31T01:55:07.833 INFO:tasks.rados.rados.0.smithi045.stdout:2927: finishing write tid 1 to smithi04569269-1 2021-07-31T01:55:07.833 INFO:tasks.rados.rados.0.smithi045.stdout:2927: finishing write tid 2 to smithi04569269-1 2021-07-31T01:55:07.833 INFO:tasks.rados.rados.0.smithi045.stdout:2921: done (8 left) 2021-07-31T01:55:07.833 INFO:tasks.rados.rados.0.smithi045.stdout:2922: done (7 left) 2021-07-31T01:55:07.833 INFO:tasks.rados.rados.0.smithi045.stdout:2924: done (6 left) 2021-07-31T01:55:07.834 INFO:tasks.rados.rados.0.smithi045.stdout:2925: done (5 left) 2021-07-31T01:55:07.834 INFO:tasks.rados.rados.0.smithi045.stdout:2926: done (4 left) 2021-07-31T01:55:07.834 INFO:tasks.rados.rados.0.smithi045.stdout:2928: done (3 left) 2021-07-31T01:55:07.834 INFO:tasks.rados.rados.0.smithi045.stdout:2930: done (2 left) 2021-07-31T01:55:07.835 INFO:tasks.rados.rados.0.smithi045.stdout:2931: delete oid 47 current snap is 274 2021-07-31T01:55:07.835 INFO:tasks.rados.rados.0.smithi045.stdout:2929: expect (ObjNum 866 snap 270 seq_num 866) 2021-07-31T01:55:07.881 INFO:tasks.rados.rados.0.smithi045.stdout:2927: finishing write tid 3 to smithi04569269-1 2021-07-31T01:55:07.882 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 1 v 1587 (ObjNum 904 snap 274 seq_num 904) dirty exists 2021-07-31T01:55:07.882 INFO:tasks.rados.rados.0.smithi045.stdout:2927: left oid 1 (ObjNum 904 snap 274 seq_num 904) 2021-07-31T01:55:07.882 INFO:tasks.rados.rados.0.smithi045.stdout:2927: done (2 left) 2021-07-31T01:55:07.883 INFO:tasks.rados.rados.0.smithi045.stdout:2929: done (1 left) 2021-07-31T01:55:07.883 INFO:tasks.rados.rados.0.smithi045.stdout:2931: done (0 left) 2021-07-31T01:55:07.883 INFO:tasks.rados.rados.0.smithi045.stdout:2932: read oid 45 snap -1 2021-07-31T01:55:07.883 INFO:tasks.rados.rados.0.smithi045.stdout:2932: expect (ObjNum 901 snap 274 seq_num 901) 2021-07-31T01:55:07.884 INFO:tasks.rados.rados.0.smithi045.stdout:2933: copy_from oid 44 from oid 31 current snap is 274 2021-07-31T01:55:07.884 INFO:tasks.rados.rados.0.smithi045.stdout:2934: rollback oid 4 current snap is 274 2021-07-31T01:55:07.884 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 4 to 268 2021-07-31T01:55:07.884 INFO:tasks.rados.rados.0.smithi045.stdout:2935: snap_create 2021-07-31T01:55:07.885 INFO:tasks.rados.rados.0.smithi045.stdout:2934: finishing rollback tid 0 to smithi04569269-4 2021-07-31T01:55:07.886 INFO:tasks.rados.rados.0.smithi045.stdout:2934: finishing rollback tid 1 to smithi04569269-4 2021-07-31T01:55:07.886 INFO:tasks.rados.rados.0.smithi045.stdout:2934: finishing rollback tid 2 to smithi04569269-4 2021-07-31T01:55:07.886 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 4 v 1658 (ObjNum 716 snap 222 seq_num 716) dirty exists 2021-07-31T01:55:07.889 INFO:tasks.rados.rados.0.smithi045.stdout:2932: expect (ObjNum 901 snap 274 seq_num 901) 2021-07-31T01:55:07.934 INFO:tasks.rados.rados.0.smithi045.stdout:2933: finishing copy_from to smithi04569269-44 2021-07-31T01:55:07.934 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 44 v 1833 (ObjNum 813 snap 254 seq_num 813) dirty exists 2021-07-31T01:55:07.935 INFO:tasks.rados.rados.0.smithi045.stdout:2933: finishing copy_from racing read to smithi04569269-44 2021-07-31T01:55:08.636 INFO:tasks.rados.rados.0.smithi045.stdout:2932: done (3 left) 2021-07-31T01:55:08.637 INFO:tasks.rados.rados.0.smithi045.stdout:2933: done (2 left) 2021-07-31T01:55:08.637 INFO:tasks.rados.rados.0.smithi045.stdout:2934: done (1 left) 2021-07-31T01:55:08.637 INFO:tasks.rados.rados.0.smithi045.stdout:2935: done (0 left) 2021-07-31T01:55:08.637 INFO:tasks.rados.rados.0.smithi045.stdout:2936: snap_create 2021-07-31T01:55:09.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:08 smithi154 conmon[34553]: cluster 2021-07-31T01:55:07.631484+0000 mon.a (mon.0) 1260 : cluster [DBG] osdmap e601: 8 total, 8 up, 8 in 2021-07-31T01:55:09.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:08 smithi154 conmon[34553]: cluster 2021-07-31T01:55:07.904427+0000 mgr.x (mgr.24542) 335 : cluster [DBG] pgmap v685: 25 pgs: 25 active+clean; 403 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:55:09.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:08 smithi045 conmon[39718]: cluster 2021-07-31T01:55:07.631484+0000 mon.a (mon.0 2021-07-31T01:55:09.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:08 smithi045 conmon[39718]: ) 1260 : cluster [DBG] osdmap e601: 8 total, 8 up, 8 in 2021-07-31T01:55:09.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:08 smithi045 conmon[39718]: cluster 2021-07-31T01:55:07.904427+0000 mgr.x (mgr.24542 2021-07-31T01:55:09.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:08 smithi045 conmon[39718]: ) 335 : cluster [DBG] pgmap v685: 25 pgs: 25 active+clean; 403 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:55:09.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:08 smithi045 conmon[33655]: cluster 2021-07-31T01:55:07.631484+0000 mon.a (mon.0 2021-07-31T01:55:09.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:08 smithi045 conmon[33655]: ) 1260 : cluster [DBG] osdmap e601: 8 total, 8 up, 8 in 2021-07-31T01:55:09.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:08 smithi045 conmon[33655]: cluster 2021-07-31T01:55:07.904427+0000 mgr.x (mgr.24542) 335 : cluster [DBG] pgmap v685: 25 pgs: 25 active+clean; 403 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:55:09.208 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T01:55:09.485 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T01:55:09.486 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 10m ago 10m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T01:55:09.486 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 7m ago 10m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T01:55:09.486 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 7m ago 14m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T01:55:09.486 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 10m ago 16m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T01:55:09.487 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 10m ago 16m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T01:55:09.487 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 7m ago 14m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T01:55:09.487 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 10m ago 14m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T01:55:09.487 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 10m ago 11m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T01:55:09.487 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 7m ago 10m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T01:55:09.488 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 10m ago 13m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T01:55:09.488 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 10m ago 13m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T01:55:09.488 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 10m ago 13m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T01:55:09.488 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 10m ago 12m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T01:55:09.489 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 7m ago 12m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T01:55:09.489 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 7m ago 12m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T01:55:09.489 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 7m ago 11m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T01:55:09.489 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 7m ago 11m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T01:55:09.489 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 7m ago 11m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T01:55:09.638 INFO:tasks.rados.rados.0.smithi045.stdout:2936: done (0 left) 2021-07-31T01:55:09.638 INFO:tasks.rados.rados.0.smithi045.stdout:2937: delete oid 22 current snap is 276 2021-07-31T01:55:09.643 INFO:tasks.rados.rados.0.smithi045.stdout:2937: done (0 left) 2021-07-31T01:55:09.644 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 47 current snap is 276 2021-07-31T01:55:09.644 INFO:tasks.rados.rados.0.smithi045.stdout:2938: seq_num 908 ranges {0=409600,409600=401408} 2021-07-31T01:55:09.645 INFO:tasks.rados.rados.0.smithi045.stdout:2938: writing smithi04569269-47 from 0 to 409600 tid 1 2021-07-31T01:55:09.648 INFO:tasks.rados.rados.0.smithi045.stdout:2938: writing smithi04569269-47 from 409600 to 811008 tid 2 2021-07-31T01:55:09.648 INFO:tasks.rados.rados.0.smithi045.stdout:2939: setattr oid 7 current snap is 276 2021-07-31T01:55:09.650 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 2 current snap is 276 2021-07-31T01:55:09.650 INFO:tasks.rados.rados.0.smithi045.stdout:2940: seq_num 909 ranges {2031616=425984,2457600=401408} 2021-07-31T01:55:09.664 INFO:tasks.rados.rados.0.smithi045.stdout:2940: writing smithi04569269-2 from 2031616 to 2457600 tid 1 2021-07-31T01:55:09.667 INFO:tasks.rados.rados.0.smithi045.stdout:2940: writing smithi04569269-2 from 2457600 to 2859008 tid 2 2021-07-31T01:55:09.667 INFO:tasks.rados.rados.0.smithi045.stdout:2941: rmattr oid 28 current snap is 276 2021-07-31T01:55:09.667 INFO:tasks.rados.rados.0.smithi045.stdout:2941: done (3 left) 2021-07-31T01:55:09.667 INFO:tasks.rados.rados.0.smithi045.stdout:2942: snap_create 2021-07-31T01:55:09.668 INFO:tasks.rados.rados.0.smithi045.stdout:2938: finishing write tid 1 to smithi04569269-47 2021-07-31T01:55:09.668 INFO:tasks.rados.rados.0.smithi045.stdout:2938: finishing write tid 2 to smithi04569269-47 2021-07-31T01:55:09.668 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 7 v 1641 (ObjNum 900 snap 273 seq_num 900) dirty exists 2021-07-31T01:55:09.668 INFO:tasks.rados.rados.0.smithi045.stdout:2938: finishing write tid 3 to smithi04569269-47 2021-07-31T01:55:09.668 INFO:tasks.rados.rados.0.smithi045.stdout:2938: finishing write tid 4 to smithi04569269-47 2021-07-31T01:55:09.669 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 47 v 1638 (ObjNum 908 snap 276 seq_num 908) dirty exists 2021-07-31T01:55:09.669 INFO:tasks.rados.rados.0.smithi045.stdout:2938: left oid 47 (ObjNum 908 snap 276 seq_num 908) 2021-07-31T01:55:09.670 INFO:tasks.rados.rados.0.smithi045.stdout:2940: finishing write tid 1 to smithi04569269-2 2021-07-31T01:55:09.670 INFO:tasks.rados.rados.0.smithi045.stdout:2940: finishing write tid 2 to smithi04569269-2 2021-07-31T01:55:09.670 INFO:tasks.rados.rados.0.smithi045.stdout:2940: finishing write tid 3 to smithi04569269-2 2021-07-31T01:55:09.671 INFO:tasks.rados.rados.0.smithi045.stdout:2940: finishing write tid 4 to smithi04569269-2 2021-07-31T01:55:09.672 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 2 v 1651 (ObjNum 909 snap 276 seq_num 909) dirty exists 2021-07-31T01:55:09.673 INFO:tasks.rados.rados.0.smithi045.stdout:2940: left oid 2 (ObjNum 909 snap 276 seq_num 909) 2021-07-31T01:55:09.835 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T01:55:09.836 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T01:55:09.836 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T01:55:09.836 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T01:55:09.836 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T01:55:09.836 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T01:55:09.837 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T01:55:09.837 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T01:55:09.837 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T01:55:09.838 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T01:55:09.838 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T01:55:09.838 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T01:55:09.838 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T01:55:09.839 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T01:55:09.839 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T01:55:09.839 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T01:55:09.839 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T01:55:10.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:09 smithi154 conmon[34553]: cluster 2021-07-31T01:55:08.635808+0000 mon.a (mon.0) 1261 : cluster [DBG] osdmap e602: 8 total, 8 up, 8 in 2021-07-31T01:55:10.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:09 smithi154 conmon[34553]: 2021-07-31T01:55:10.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:09 smithi045 conmon[33655]: cluster 2021-07-31T01:55:08.635808 2021-07-31T01:55:10.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:09 smithi045 conmon[33655]: +0000 mon.a (mon.0) 1261 : cluster [DBG] osdmap e602: 8 total, 8 up, 8 in 2021-07-31T01:55:10.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:09 smithi045 conmon[39718]: cluster 2021-07-31T01:55:08. 2021-07-31T01:55:10.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:09 smithi045 conmon[39718]: 635808+0000 mon.a (mon.0) 1261 : cluster [DBG] osdmap e602: 8 total, 8 up, 8 in 2021-07-31T01:55:10.650 INFO:tasks.rados.rados.0.smithi045.stdout:2938: done (3 left) 2021-07-31T01:55:10.650 INFO:tasks.rados.rados.0.smithi045.stdout:2939: done (2 left) 2021-07-31T01:55:10.650 INFO:tasks.rados.rados.0.smithi045.stdout:2940: done (1 left) 2021-07-31T01:55:10.650 INFO:tasks.rados.rados.0.smithi045.stdout:2942: done (0 left) 2021-07-31T01:55:10.651 INFO:tasks.rados.rados.0.smithi045.stdout:2943: copy_from oid 26 from oid 49 current snap is 277 2021-07-31T01:55:10.651 INFO:tasks.rados.rados.0.smithi045.stdout:2944: setattr oid 24 current snap is 277 2021-07-31T01:55:10.652 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 16 current snap is 277 2021-07-31T01:55:10.652 INFO:tasks.rados.rados.0.smithi045.stdout:2945: seq_num 910 ranges {14344192=442368,14786560=401408} 2021-07-31T01:55:10.733 INFO:tasks.rados.rados.0.smithi045.stdout:2945: writing smithi04569269-16 from 14344192 to 14786560 tid 1 2021-07-31T01:55:10.735 INFO:tasks.rados.rados.0.smithi045.stdout:2945: writing smithi04569269-16 from 14786560 to 15187968 tid 2 2021-07-31T01:55:10.735 INFO:tasks.rados.rados.0.smithi045.stdout:2946: delete oid 41 current snap is 277 2021-07-31T01:55:10.735 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 24 v 1638 (ObjNum -1104988560 snap 21873 seq_num 3190504016) dirty exists 2021-07-31T01:55:10.735 INFO:tasks.rados.rados.0.smithi045.stdout:2943: finishing copy_from to smithi04569269-26 2021-07-31T01:55:10.736 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 26 v 1836 (ObjNum 890 snap 272 seq_num 890) dirty exists 2021-07-31T01:55:10.736 INFO:tasks.rados.rados.0.smithi045.stdout:2943: finishing copy_from racing read to smithi04569269-26 2021-07-31T01:55:10.738 INFO:tasks.rados.rados.0.smithi045.stdout:2945: finishing write tid 1 to smithi04569269-16 2021-07-31T01:55:10.738 INFO:tasks.rados.rados.0.smithi045.stdout:2943: done (3 left) 2021-07-31T01:55:10.738 INFO:tasks.rados.rados.0.smithi045.stdout:2944: done (2 left) 2021-07-31T01:55:10.738 INFO:tasks.rados.rados.0.smithi045.stdout:2946: done (1 left) 2021-07-31T01:55:10.739 INFO:tasks.rados.rados.0.smithi045.stdout:2947: rmattr oid 10 current snap is 277 2021-07-31T01:55:10.739 INFO:tasks.rados.rados.0.smithi045.stdout:2947: done (1 left) 2021-07-31T01:55:10.739 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 5 current snap is 277 2021-07-31T01:55:10.739 INFO:tasks.rados.rados.0.smithi045.stdout:2948: seq_num 912 ranges {12247040=483328,12730368=401408} 2021-07-31T01:55:10.807 INFO:tasks.rados.rados.0.smithi045.stdout:2948: writing smithi04569269-5 from 12247040 to 12730368 tid 1 2021-07-31T01:55:10.810 INFO:tasks.rados.rados.0.smithi045.stdout:2948: writing smithi04569269-5 from 12730368 to 13131776 tid 2 2021-07-31T01:55:10.810 INFO:tasks.rados.rados.0.smithi045.stdout:2949: snap_create 2021-07-31T01:55:10.810 INFO:tasks.rados.rados.0.smithi045.stdout:2945: finishing write tid 2 to smithi04569269-16 2021-07-31T01:55:10.811 INFO:tasks.rados.rados.0.smithi045.stdout:2945: finishing write tid 3 to smithi04569269-16 2021-07-31T01:55:10.811 INFO:tasks.rados.rados.0.smithi045.stdout:2945: finishing write tid 4 to smithi04569269-16 2021-07-31T01:55:10.811 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 16 v 1641 (ObjNum 910 snap 277 seq_num 910) dirty exists 2021-07-31T01:55:10.812 INFO:tasks.rados.rados.0.smithi045.stdout:2945: left oid 16 (ObjNum 910 snap 277 seq_num 910) 2021-07-31T01:55:10.814 INFO:tasks.rados.rados.0.smithi045.stdout:2948: finishing write tid 1 to smithi04569269-5 2021-07-31T01:55:10.814 INFO:tasks.rados.rados.0.smithi045.stdout:2948: finishing write tid 2 to smithi04569269-5 2021-07-31T01:55:10.814 INFO:tasks.rados.rados.0.smithi045.stdout:2948: finishing write tid 3 to smithi04569269-5 2021-07-31T01:55:10.815 INFO:tasks.rados.rados.0.smithi045.stdout:2948: finishing write tid 4 to smithi04569269-5 2021-07-31T01:55:10.816 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 5 v 1659 (ObjNum 912 snap 277 seq_num 912) dirty exists 2021-07-31T01:55:10.816 INFO:tasks.rados.rados.0.smithi045.stdout:2948: left oid 5 (ObjNum 912 snap 277 seq_num 912) 2021-07-31T01:55:11.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:10 smithi154 conmon[34553]: audit 2021-07-31T01:55:09.195473+0000 mgr.x (mgr.24542) 336 : audit [DBG] from='client.14982 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:55:11.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:10 smithi154 conmon[34553]: audit 2021-07-31T01:55:09.477402+0000 mgr.x (mgr.24542) 337 : audit [DBG] from='client.24895 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:55:11.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:10 smithi154 conmon[34553]: cluster 2021-07-31T01:55:09.636870+0000 mon.a (mon.0) 1262 : cluster 2021-07-31T01:55:11.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:10 smithi154 conmon[34553]: [DBG] osdmap e603: 8 total, 8 up, 8 in 2021-07-31T01:55:11.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:10 smithi154 conmon[34553]: audit 2021-07 2021-07-31T01:55:11.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:10 smithi154 conmon[34553]: -31T01:55:09.835258+0000 mon.a (mon.0) 1263 : audit [DBG] 2021-07-31T01:55:11.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:10 smithi154 conmon[34553]: from='client.? 172.21.15.45:0/3951452204' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T01:55:11.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:10 smithi154 conmon[34553]: cluster 2021-07-31T01:55:09 2021-07-31T01:55:11.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:10 smithi154 conmon[34553]: .904836+0000 mgr.x (mgr.24542) 338 : cluster [DBG] pgmap v688: 25 pgs: 25 active+clean; 424 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 4.6 MiB/s rd, 3.8 MiB/s wr, 33 op/s 2021-07-31T01:55:11.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:10 smithi154 conmon[34553]: 2021-07-31T01:55:11.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:10 smithi045 conmon[33655]: audit 2021-07-31T01:55:09.195473+0000 mgr.x (mgr. 2021-07-31T01:55:11.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:10 smithi045 conmon[33655]: 24542) 336 : audit [DBG] from='client.14982 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:55:11.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:10 smithi045 conmon[33655]: audit 2021-07-31T01:55:09.477402+0000 mgr.x (mgr.24542) 2021-07-31T01:55:11.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:10 smithi045 conmon[33655]: 337 : audit [DBG] from='client.24895 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:55:11.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:10 smithi045 conmon[33655]: cluster 2021-07-31T01 2021-07-31T01:55:11.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:10 smithi045 conmon[33655]: :55:09.636870+0000 mon.a (mon.0) 1262 : cluster [DBG] osdmap e603: 8 total, 8 up, 8 in 2021-07-31T01:55:11.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:10 smithi045 conmon[33655]: 2021-07-31T01:55:11.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:10 smithi045 conmon[33655]: audit 2021-07-31 2021-07-31T01:55:11.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:10 smithi045 conmon[33655]: T01:55:09.835258+0000 mon.a (mon.0) 1263 : 2021-07-31T01:55:11.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:10 smithi045 conmon[33655]: audit [DBG] from='client.? 172.21.15.45:0/3951452204' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T01:55:11.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:10 smithi045 conmon[33655]: cluster 2021-07-31T01:55:09.904836+0000 mgr.x ( 2021-07-31T01:55:11.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:10 smithi045 conmon[33655]: mgr.24542) 338 : cluster [DBG] pgmap v688: 25 pgs: 25 active+clean; 424 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 4.6 MiB/s rd, 3.8 MiB/s wr, 33 op/s 2021-07-31T01:55:11.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:10 smithi045 conmon[39718]: audit 2021-07-31T01:55:09 2021-07-31T01:55:11.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:10 smithi045 conmon[39718]: .195473+0000 mgr.x (mgr.24542) 336 : audit [DBG] from='client.14982 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:55:11.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:10 smithi045 conmon[39718]: audit 2021-07-31T01:55:11.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:10 smithi045 conmon[39718]: 2021-07-31T01:55:09 2021-07-31T01:55:11.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:10 smithi045 conmon[39718]: .477402+0000 mgr.x (mgr.24542) 337 : 2021-07-31T01:55:11.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:10 smithi045 conmon[39718]: audit [DBG] from='client.24895 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:55:11.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:10 smithi045 conmon[39718]: cluster 2021-07-31T01:55:11.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:10 smithi045 conmon[39718]: 2021-07-31T01 2021-07-31T01:55:11.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:10 smithi045 conmon[39718]: :55:09.636870+0000 mon.a (mon.0) 1262 : cluster [DBG] osdmap e603: 8 total, 8 up, 8 in 2021-07-31T01:55:11.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:10 smithi045 conmon[39718]: 2021-07-31T01:55:11.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:10 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T01:55:11.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:10 smithi045 conmon[39718]: 01:55:09.835258+0000 mon.a (mon.0) 1263 : 2021-07-31T01:55:11.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:10 smithi045 conmon[39718]: audit [DBG] from='client.? 172.21.15.45:0/3951452204' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T01:55:11.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:10 smithi045 conmon[39718]: cluster 2021- 2021-07-31T01:55:11.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:10 smithi045 conmon[39718]: 07-31T01:55:09. 2021-07-31T01:55:11.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:10 smithi045 conmon[39718]: 904836+0000 mgr.x (mgr. 2021-07-31T01:55:11.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:10 smithi045 conmon[39718]: 24542) 338 : cluster [DBG] 2021-07-31T01:55:11.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:10 smithi045 conmon[39718]: pgmap v688: 25 pgs: 25 active+clean; 424 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 4.6 MiB/s rd, 3.8 MiB/s wr, 33 op/s 2021-07-31T01:55:11.628 INFO:tasks.rados.rados.0.smithi045.stdout:2945: done (2 left) 2021-07-31T01:55:11.628 INFO:tasks.rados.rados.0.smithi045.stdout:2948: done (1 left) 2021-07-31T01:55:11.628 INFO:tasks.rados.rados.0.smithi045.stdout:2949: done (0 left) 2021-07-31T01:55:11.629 INFO:tasks.rados.rados.0.smithi045.stdout:2950: delete oid 25 current snap is 278 2021-07-31T01:55:11.634 INFO:tasks.rados.rados.0.smithi045.stdout:2950: done (0 left) 2021-07-31T01:55:11.634 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 45 current snap is 278 2021-07-31T01:55:11.635 INFO:tasks.rados.rados.0.smithi045.stdout:2951: seq_num 914 ranges {1359872=516096,1875968=401408} 2021-07-31T01:55:11.644 INFO:tasks.rados.rados.0.smithi045.stdout:2951: writing smithi04569269-45 from 1359872 to 1875968 tid 1 2021-07-31T01:55:11.647 INFO:tasks.rados.rados.0.smithi045.stdout:2951: writing smithi04569269-45 from 1875968 to 2277376 tid 2 2021-07-31T01:55:11.647 INFO:tasks.rados.rados.0.smithi045.stdout:2952: read oid 17 snap -1 2021-07-31T01:55:11.648 INFO:tasks.rados.rados.0.smithi045.stdout:2952: expect (ObjNum 829 snap 262 seq_num 829) 2021-07-31T01:55:11.648 INFO:tasks.rados.rados.0.smithi045.stdout:2953: copy_from oid 6 from oid 31 current snap is 278 2021-07-31T01:55:11.648 INFO:tasks.rados.rados.0.smithi045.stdout:2954: read oid 50 snap -1 2021-07-31T01:55:11.648 INFO:tasks.rados.rados.0.smithi045.stdout:2954: expect (ObjNum 837 snap 263 seq_num 837) 2021-07-31T01:55:11.649 INFO:tasks.rados.rados.0.smithi045.stdout:2955: rollback oid 46 current snap is 278 2021-07-31T01:55:11.649 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 46 to 277 2021-07-31T01:55:11.649 INFO:tasks.rados.rados.0.smithi045.stdout:2956: setattr oid 21 current snap is 278 2021-07-31T01:55:11.650 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 25 current snap is 278 2021-07-31T01:55:11.650 INFO:tasks.rados.rados.0.smithi045.stdout:2957: seq_num 915 ranges {0=532480,532480=401408} 2021-07-31T01:55:11.652 INFO:tasks.rados.rados.0.smithi045.stdout:2957: writing smithi04569269-25 from 0 to 532480 tid 1 2021-07-31T01:55:11.655 INFO:tasks.rados.rados.0.smithi045.stdout:2957: writing smithi04569269-25 from 532480 to 933888 tid 2 2021-07-31T01:55:11.655 INFO:tasks.rados.rados.0.smithi045.stdout:2958: read oid 7 snap -1 2021-07-31T01:55:11.655 INFO:tasks.rados.rados.0.smithi045.stdout:2958: expect (ObjNum 900 snap 273 seq_num 900) 2021-07-31T01:55:11.655 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 9 current snap is 278 2021-07-31T01:55:11.656 INFO:tasks.rados.rados.0.smithi045.stdout:2955: finishing rollback tid 0 to smithi04569269-46 2021-07-31T01:55:11.656 INFO:tasks.rados.rados.0.smithi045.stdout:2955: finishing rollback tid 1 to smithi04569269-46 2021-07-31T01:55:11.656 INFO:tasks.rados.rados.0.smithi045.stdout:2959: seq_num 916 ranges {4096000=548864,4644864=401408} 2021-07-31T01:55:11.681 INFO:tasks.rados.rados.0.smithi045.stdout:2959: writing smithi04569269-9 from 4096000 to 4644864 tid 1 2021-07-31T01:55:11.684 INFO:tasks.rados.rados.0.smithi045.stdout:2959: writing smithi04569269-9 from 4644864 to 5046272 tid 2 2021-07-31T01:55:11.684 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 22 current snap is 278 2021-07-31T01:55:11.685 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 21 v 1877 (ObjNum 866 snap 270 seq_num 866) dirty exists 2021-07-31T01:55:11.685 INFO:tasks.rados.rados.0.smithi045.stdout:2960: seq_num 917 ranges {0=565248,565248=401408} 2021-07-31T01:55:11.688 INFO:tasks.rados.rados.0.smithi045.stdout:2960: writing smithi04569269-22 from 0 to 565248 tid 1 2021-07-31T01:55:11.691 INFO:tasks.rados.rados.0.smithi045.stdout:2960: writing smithi04569269-22 from 565248 to 966656 tid 2 2021-07-31T01:55:11.691 INFO:tasks.rados.rados.0.smithi045.stdout:2956: done (9 left) 2021-07-31T01:55:11.691 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 42 current snap is 278 2021-07-31T01:55:11.691 INFO:tasks.rados.rados.0.smithi045.stdout:2957: finishing write tid 1 to smithi04569269-25 2021-07-31T01:55:11.692 INFO:tasks.rados.rados.0.smithi045.stdout:2957: finishing write tid 2 to smithi04569269-25 2021-07-31T01:55:11.692 INFO:tasks.rados.rados.0.smithi045.stdout:2957: finishing write tid 3 to smithi04569269-25 2021-07-31T01:55:11.692 INFO:tasks.rados.rados.0.smithi045.stdout:2955: finishing rollback tid 2 to smithi04569269-46 2021-07-31T01:55:11.692 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 46 v 1305 (ObjNum 826 snap 258 seq_num 826) dirty exists 2021-07-31T01:55:11.693 INFO:tasks.rados.rados.0.smithi045.stdout:2957: finishing write tid 4 to smithi04569269-25 2021-07-31T01:55:11.693 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 25 v 1642 (ObjNum 915 snap 278 seq_num 915) dirty exists 2021-07-31T01:55:11.693 INFO:tasks.rados.rados.0.smithi045.stdout:2957: left oid 25 (ObjNum 915 snap 278 seq_num 915) 2021-07-31T01:55:11.693 INFO:tasks.rados.rados.0.smithi045.stdout:2951: finishing write tid 1 to smithi04569269-45 2021-07-31T01:55:11.693 INFO:tasks.rados.rados.0.smithi045.stdout:2951: finishing write tid 2 to smithi04569269-45 2021-07-31T01:55:11.694 INFO:tasks.rados.rados.0.smithi045.stdout:2958: expect (ObjNum 900 snap 273 seq_num 900) 2021-07-31T01:55:11.711 INFO:tasks.rados.rados.0.smithi045.stdout:2953: finishing copy_from to smithi04569269-6 2021-07-31T01:55:11.711 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 6 v 1833 (ObjNum 813 snap 254 seq_num 813) dirty exists 2021-07-31T01:55:11.712 INFO:tasks.rados.rados.0.smithi045.stdout:2953: finishing copy_from racing read to smithi04569269-6 2021-07-31T01:55:11.712 INFO:tasks.rados.rados.0.smithi045.stdout:2961: seq_num 918 ranges {155648=581632,737280=401408} 2021-07-31T01:55:11.715 INFO:tasks.rados.rados.0.smithi045.stdout:2961: writing smithi04569269-42 from 155648 to 737280 tid 1 2021-07-31T01:55:11.718 INFO:tasks.rados.rados.0.smithi045.stdout:2961: writing smithi04569269-42 from 737280 to 1138688 tid 2 2021-07-31T01:55:11.718 INFO:tasks.rados.rados.0.smithi045.stdout:2953: done (9 left) 2021-07-31T01:55:11.718 INFO:tasks.rados.rados.0.smithi045.stdout:2955: done (8 left) 2021-07-31T01:55:11.719 INFO:tasks.rados.rados.0.smithi045.stdout:2957: done (7 left) 2021-07-31T01:55:11.719 INFO:tasks.rados.rados.0.smithi045.stdout:2958: done (6 left) 2021-07-31T01:55:11.719 INFO:tasks.rados.rados.0.smithi045.stdout:2962: setattr oid 43 current snap is 278 2021-07-31T01:55:11.719 INFO:tasks.rados.rados.0.smithi045.stdout:2951: finishing write tid 3 to smithi04569269-45 2021-07-31T01:55:11.720 INFO:tasks.rados.rados.0.smithi045.stdout:2959: finishing write tid 1 to smithi04569269-9 2021-07-31T01:55:11.720 INFO:tasks.rados.rados.0.smithi045.stdout:2959: finishing write tid 2 to smithi04569269-9 2021-07-31T01:55:11.720 INFO:tasks.rados.rados.0.smithi045.stdout:2959: finishing write tid 3 to smithi04569269-9 2021-07-31T01:55:11.720 INFO:tasks.rados.rados.0.smithi045.stdout:2959: finishing write tid 4 to smithi04569269-9 2021-07-31T01:55:11.720 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 9 v 1654 (ObjNum 916 snap 278 seq_num 916) dirty exists 2021-07-31T01:55:11.721 INFO:tasks.rados.rados.0.smithi045.stdout:2959: left oid 9 (ObjNum 916 snap 278 seq_num 916) 2021-07-31T01:55:11.721 INFO:tasks.rados.rados.0.smithi045.stdout:2960: finishing write tid 1 to smithi04569269-22 2021-07-31T01:55:11.721 INFO:tasks.rados.rados.0.smithi045.stdout:2960: finishing write tid 2 to smithi04569269-22 2021-07-31T01:55:11.721 INFO:tasks.rados.rados.0.smithi045.stdout:2960: finishing write tid 3 to smithi04569269-22 2021-07-31T01:55:11.722 INFO:tasks.rados.rados.0.smithi045.stdout:2960: finishing write tid 4 to smithi04569269-22 2021-07-31T01:55:11.722 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 22 v 1644 (ObjNum 917 snap 278 seq_num 917) dirty exists 2021-07-31T01:55:11.722 INFO:tasks.rados.rados.0.smithi045.stdout:2960: left oid 22 (ObjNum 917 snap 278 seq_num 917) 2021-07-31T01:55:11.722 INFO:tasks.rados.rados.0.smithi045.stdout:2951: finishing write tid 4 to smithi04569269-45 2021-07-31T01:55:11.722 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 45 v 1665 (ObjNum 914 snap 278 seq_num 914) dirty exists 2021-07-31T01:55:11.723 INFO:tasks.rados.rados.0.smithi045.stdout:2951: left oid 45 (ObjNum 914 snap 278 seq_num 914) 2021-07-31T01:55:11.723 INFO:tasks.rados.rados.0.smithi045.stdout:2952: expect (ObjNum 829 snap 262 seq_num 829) 2021-07-31T01:55:12.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:11 smithi154 conmon[34553]: cluster 2021-07-31T01:55:12.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:11 smithi154 conmon[34553]: 2021-07-31T01:55:12.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:11 smithi154 conmon[34553]: 2021 2021-07-31T01:55:12.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:11 smithi154 conmon[34553]: - 2021-07-31T01:55:12.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:11 smithi154 conmon[34553]: 07 2021-07-31T01:55:12.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:11 smithi154 conmon[34553]: -31T01:55:10.648284+0000 mon.a 2021-07-31T01:55:12.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:11 smithi154 conmon[34553]: (mon.0) 1264 : cluster [DBG] osdmap e604: 8 total, 8 up, 8 in 2021-07-31T01:55:12.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:11 smithi154 conmon[34553]: cluster 2021-07-31T01:55:12.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:11 smithi154 conmon[34553]: 2021-07-31T01:55:11.627287+0000 mon.a (mon.0) 1265 : cluster [DBG] osdmap e605: 8 total, 8 up, 8 in 2021-07-31T01:55:12.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:11 smithi045 conmon[33655]: cluster 2021-07-31T01:55:10.648284+0000 mon.a ( 2021-07-31T01:55:12.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:11 smithi045 conmon[33655]: mon.0) 1264 : cluster [DBG] osdmap e604: 8 total, 8 up, 8 in 2021-07-31T01:55:12.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:11 smithi045 conmon[33655]: cluster 2021-07-31T01:55:11.627287+0000 mon.a (mon. 2021-07-31T01:55:12.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:11 smithi045 conmon[33655]: 0) 1265 : cluster [DBG] osdmap e605: 8 total, 8 up, 8 in 2021-07-31T01:55:12.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:11 smithi045 conmon[39718]: cluster 2021-07-31T01: 2021-07-31T01:55:12.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:11 smithi045 conmon[39718]: 55:10.648284+0000 mon.a (mon.0) 1264 : cluster [DBG] osdmap e604: 8 total, 8 up, 8 in 2021-07-31T01:55:12.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:11 smithi045 conmon[39718]: cluster 2021-07-31T01: 2021-07-31T01:55:12.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:11 smithi045 conmon[39718]: 55:11.627287+0000 mon.a (mon.0) 1265 2021-07-31T01:55:12.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:11 smithi045 conmon[39718]: : cluster [DBG] osdmap e605: 8 total, 8 up, 8 in 2021-07-31T01:55:12.418 INFO:tasks.rados.rados.0.smithi045.stdout:2954: expect (ObjNum 837 snap 263 seq_num 837) 2021-07-31T01:55:12.470 INFO:tasks.rados.rados.0.smithi045.stdout:2961: finishing write tid 1 to smithi04569269-42 2021-07-31T01:55:12.471 INFO:tasks.rados.rados.0.smithi045.stdout:2961: finishing write tid 2 to smithi04569269-42 2021-07-31T01:55:12.471 INFO:tasks.rados.rados.0.smithi045.stdout:2961: finishing write tid 3 to smithi04569269-42 2021-07-31T01:55:12.471 INFO:tasks.rados.rados.0.smithi045.stdout:2961: finishing write tid 4 to smithi04569269-42 2021-07-31T01:55:12.471 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 42 v 1668 (ObjNum 918 snap 278 seq_num 918) dirty exists 2021-07-31T01:55:12.472 INFO:tasks.rados.rados.0.smithi045.stdout:2961: left oid 42 (ObjNum 918 snap 278 seq_num 918) 2021-07-31T01:55:12.472 INFO:tasks.rados.rados.0.smithi045.stdout:2951: done (6 left) 2021-07-31T01:55:12.472 INFO:tasks.rados.rados.0.smithi045.stdout:2952: done (5 left) 2021-07-31T01:55:12.472 INFO:tasks.rados.rados.0.smithi045.stdout:2954: done (4 left) 2021-07-31T01:55:12.472 INFO:tasks.rados.rados.0.smithi045.stdout:2959: done (3 left) 2021-07-31T01:55:12.473 INFO:tasks.rados.rados.0.smithi045.stdout:2960: done (2 left) 2021-07-31T01:55:12.473 INFO:tasks.rados.rados.0.smithi045.stdout:2961: done (1 left) 2021-07-31T01:55:12.473 INFO:tasks.rados.rados.0.smithi045.stdout:2963: rmattr oid 23 current snap is 278 2021-07-31T01:55:12.473 INFO:tasks.rados.rados.0.smithi045.stdout:2963: done (1 left) 2021-07-31T01:55:12.473 INFO:tasks.rados.rados.0.smithi045.stdout:2964: copy_from oid 17 from oid 4 current snap is 278 2021-07-31T01:55:12.474 INFO:tasks.rados.rados.0.smithi045.stdout:2965: read oid 49 snap -1 2021-07-31T01:55:12.474 INFO:tasks.rados.rados.0.smithi045.stdout:2965: expect (ObjNum 890 snap 272 seq_num 890) 2021-07-31T01:55:12.474 INFO:tasks.rados.rados.0.smithi045.stdout:2966: delete oid 45 current snap is 278 2021-07-31T01:55:12.475 INFO:tasks.rados.rados.0.smithi045.stdout:2966: done (3 left) 2021-07-31T01:55:12.475 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 11 current snap is 278 2021-07-31T01:55:12.475 INFO:tasks.rados.rados.0.smithi045.stdout:2967: seq_num 920 ranges {0=614400,614400=401408} 2021-07-31T01:55:12.477 INFO:tasks.rados.rados.0.smithi045.stdout:2967: writing smithi04569269-11 from 0 to 614400 tid 1 2021-07-31T01:55:12.480 INFO:tasks.rados.rados.0.smithi045.stdout:2967: writing smithi04569269-11 from 614400 to 1015808 tid 2 2021-07-31T01:55:12.480 INFO:tasks.rados.rados.0.smithi045.stdout:2968: read oid 36 snap -1 2021-07-31T01:55:12.481 INFO:tasks.rados.rados.0.smithi045.stdout:2968: expect (ObjNum 835 snap 263 seq_num 835) 2021-07-31T01:55:12.481 INFO:tasks.rados.rados.0.smithi045.stdout:2969: rmattr oid 22 current snap is 278 2021-07-31T01:55:12.481 INFO:tasks.rados.rados.0.smithi045.stdout:2969: done (5 left) 2021-07-31T01:55:12.481 INFO:tasks.rados.rados.0.smithi045.stdout:2970: snap_remove snap 273 2021-07-31T01:55:12.667 INFO:tasks.rados.rados.0.smithi045.stdout:2970: done (5 left) 2021-07-31T01:55:12.668 INFO:tasks.rados.rados.0.smithi045.stdout:2971: read oid 22 snap -1 2021-07-31T01:55:12.668 INFO:tasks.rados.rados.0.smithi045.stdout:2971: expect (ObjNum 917 snap 278 seq_num 917) 2021-07-31T01:55:12.668 INFO:tasks.rados.rados.0.smithi045.stdout:2972: snap_create 2021-07-31T01:55:12.668 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 43 v 1834 (ObjNum 888 snap 272 seq_num 888) dirty exists 2021-07-31T01:55:12.669 INFO:tasks.rados.rados.0.smithi045.stdout:2964: finishing copy_from to smithi04569269-17 2021-07-31T01:55:12.669 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 17 v 1658 (ObjNum 716 snap 222 seq_num 716) dirty exists 2021-07-31T01:55:12.669 INFO:tasks.rados.rados.0.smithi045.stdout:2964: finishing copy_from racing read to smithi04569269-17 2021-07-31T01:55:12.669 INFO:tasks.rados.rados.0.smithi045.stdout:2967: finishing write tid 1 to smithi04569269-11 2021-07-31T01:55:12.669 INFO:tasks.rados.rados.0.smithi045.stdout:2967: finishing write tid 2 to smithi04569269-11 2021-07-31T01:55:12.670 INFO:tasks.rados.rados.0.smithi045.stdout:2967: finishing write tid 3 to smithi04569269-11 2021-07-31T01:55:12.670 INFO:tasks.rados.rados.0.smithi045.stdout:2967: finishing write tid 4 to smithi04569269-11 2021-07-31T01:55:12.670 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 11 v 1662 (ObjNum 920 snap 278 seq_num 920) dirty exists 2021-07-31T01:55:12.671 INFO:tasks.rados.rados.0.smithi045.stdout:2967: left oid 11 (ObjNum 920 snap 278 seq_num 920) 2021-07-31T01:55:12.671 INFO:tasks.rados.rados.0.smithi045.stdout:2965: expect (ObjNum 890 snap 272 seq_num 890) 2021-07-31T01:55:13.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:12 smithi154 conmon[34553]: cluster 2021-07-31T01:55:13.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:12 smithi154 conmon[34553]: 2021-07-31T01:55:13.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:12 smithi154 conmon[34553]: 2021 2021-07-31T01:55:13.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:12 smithi154 conmon[34553]: - 2021-07-31T01:55:13.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:12 smithi154 conmon[34553]: 07-31T01:55:11.905203+0000 mgr.x (mgr.24542) 339 : cluster 2021-07-31T01:55:13.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:12 smithi154 conmon[34553]: [DBG] pgmap v691: 25 pgs: 25 active+clean; 424 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 4.6 MiB/s rd, 3.8 MiB/s wr, 33 op/s 2021-07-31T01:55:13.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:12 smithi045 conmon[39718]: cluster 2021-07-31T01:55: 2021-07-31T01:55:13.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:12 smithi045 conmon[39718]: 11.905203+0000 mgr.x (mgr.24542) 339 : cluster [DBG] pgmap v691: 25 pgs: 25 active+clean; 424 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 4.6 MiB/s rd, 3.8 MiB/s wr, 33 op/s 2021-07-31T01:55:13.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:12 smithi045 conmon[33655]: cluster 2021-07-31T01:55:11.905203+0000 2021-07-31T01:55:13.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:12 smithi045 conmon[33655]: mgr.x (mgr.24542) 339 : cluster [DBG] pgmap v691: 25 pgs: 25 active+clean; 424 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 4.6 MiB/s rd, 3.8 MiB/s wr, 33 op/s 2021-07-31T01:55:13.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:13 smithi045 conmon[39718]: cluster 2021-07-31T01: 2021-07-31T01:55:13.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:13 smithi045 conmon[39718]: 55:12.654254+0000 mon.a (mon.0) 1266 : cluster [DBG] osdmap e606: 8 total, 8 up, 8 in 2021-07-31T01:55:13.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:55:13 smithi045 conmon[61210]: level=error ts=2021-07-31T01:55:13.523Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:55:13.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:55:13 smithi045 conmon[61210]: level=error ts=2021-07-31T01:55:13.523Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:55:13.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:55:13 smithi045 conmon[61210]: level=error ts=2021-07-31T01:55:13.523Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T01:55:13.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:13 smithi045 conmon[33655]: cluster 2021-07-31T01:55:12.654254+0000 mon.a (mon. 2021-07-31T01:55:13.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:13 smithi045 conmon[33655]: 0) 1266 : cluster [DBG] osdmap e606: 8 total, 8 up, 8 in 2021-07-31T01:55:14.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:13 smithi154 conmon[34553]: cluster 2021-07-31T01:55:12.654254+0000 mon.a (mon.0) 1266 : cluster [DBG] osdmap e606: 8 total, 8 up, 8 in 2021-07-31T01:55:14.270 INFO:tasks.rados.rados.0.smithi045.stdout:2968: expect (ObjNum 835 snap 263 seq_num 835) 2021-07-31T01:55:14.323 INFO:tasks.rados.rados.0.smithi045.stdout:2971: expect (ObjNum 917 snap 278 seq_num 917) 2021-07-31T01:55:14.348 INFO:tasks.rados.rados.0.smithi045.stdout:2962: done (6 left) 2021-07-31T01:55:14.348 INFO:tasks.rados.rados.0.smithi045.stdout:2964: done (5 left) 2021-07-31T01:55:14.348 INFO:tasks.rados.rados.0.smithi045.stdout:2965: done (4 left) 2021-07-31T01:55:14.348 INFO:tasks.rados.rados.0.smithi045.stdout:2967: done (3 left) 2021-07-31T01:55:14.349 INFO:tasks.rados.rados.0.smithi045.stdout:2968: done (2 left) 2021-07-31T01:55:14.349 INFO:tasks.rados.rados.0.smithi045.stdout:2971: done (1 left) 2021-07-31T01:55:14.349 INFO:tasks.rados.rados.0.smithi045.stdout:2972: done (0 left) 2021-07-31T01:55:14.349 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 18 current snap is 279 2021-07-31T01:55:14.350 INFO:tasks.rados.rados.0.smithi045.stdout:2973: seq_num 921 ranges {13713408=630784,14344192=401408} 2021-07-31T01:55:14.425 INFO:tasks.rados.rados.0.smithi045.stdout:2973: writing smithi04569269-18 from 13713408 to 14344192 tid 1 2021-07-31T01:55:14.429 INFO:tasks.rados.rados.0.smithi045.stdout:2973: writing smithi04569269-18 from 14344192 to 14745600 tid 2 2021-07-31T01:55:14.429 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 5 current snap is 279 2021-07-31T01:55:14.429 INFO:tasks.rados.rados.0.smithi045.stdout:2974: seq_num 922 ranges {13131776=647168,13778944=401408} 2021-07-31T01:55:14.503 INFO:tasks.rados.rados.0.smithi045.stdout:2974: writing smithi04569269-5 from 13131776 to 13778944 tid 1 2021-07-31T01:55:14.505 INFO:tasks.rados.rados.0.smithi045.stdout:2974: writing smithi04569269-5 from 13778944 to 14180352 tid 2 2021-07-31T01:55:14.506 INFO:tasks.rados.rados.0.smithi045.stdout:2975: delete oid 35 current snap is 279 2021-07-31T01:55:14.506 INFO:tasks.rados.rados.0.smithi045.stdout:2973: finishing write tid 1 to smithi04569269-18 2021-07-31T01:55:14.506 INFO:tasks.rados.rados.0.smithi045.stdout:2973: finishing write tid 2 to smithi04569269-18 2021-07-31T01:55:14.506 INFO:tasks.rados.rados.0.smithi045.stdout:2973: finishing write tid 3 to smithi04569269-18 2021-07-31T01:55:14.507 INFO:tasks.rados.rados.0.smithi045.stdout:2973: finishing write tid 4 to smithi04569269-18 2021-07-31T01:55:14.507 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 18 v 1661 (ObjNum 921 snap 279 seq_num 921) dirty exists 2021-07-31T01:55:14.507 INFO:tasks.rados.rados.0.smithi045.stdout:2973: left oid 18 (ObjNum 921 snap 279 seq_num 921) 2021-07-31T01:55:14.509 INFO:tasks.rados.rados.0.smithi045.stdout:2973: done (2 left) 2021-07-31T01:55:14.510 INFO:tasks.rados.rados.0.smithi045.stdout:2975: done (1 left) 2021-07-31T01:55:14.510 INFO:tasks.rados.rados.0.smithi045.stdout:2976: delete oid 33 current snap is 279 2021-07-31T01:55:14.510 INFO:tasks.rados.rados.0.smithi045.stdout:2974: finishing write tid 1 to smithi04569269-5 2021-07-31T01:55:14.513 INFO:tasks.rados.rados.0.smithi045.stdout:2974: finishing write tid 2 to smithi04569269-5 2021-07-31T01:55:14.513 INFO:tasks.rados.rados.0.smithi045.stdout:2974: finishing write tid 3 to smithi04569269-5 2021-07-31T01:55:14.514 INFO:tasks.rados.rados.0.smithi045.stdout:2974: finishing write tid 4 to smithi04569269-5 2021-07-31T01:55:14.515 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 5 v 1665 (ObjNum 922 snap 279 seq_num 922) dirty exists 2021-07-31T01:55:14.515 INFO:tasks.rados.rados.0.smithi045.stdout:2974: left oid 5 (ObjNum 922 snap 279 seq_num 922) 2021-07-31T01:55:14.515 INFO:tasks.rados.rados.0.smithi045.stdout:2974: done (1 left) 2021-07-31T01:55:14.515 INFO:tasks.rados.rados.0.smithi045.stdout:2976: done (0 left) 2021-07-31T01:55:14.516 INFO:tasks.rados.rados.0.smithi045.stdout:2977: snap_create 2021-07-31T01:55:14.665 INFO:tasks.rados.rados.0.smithi045.stdout:2977: done (0 left) 2021-07-31T01:55:14.666 INFO:tasks.rados.rados.0.smithi045.stdout:2978: copy_from oid 23 from oid 22 current snap is 280 2021-07-31T01:55:14.666 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 33 current snap is 280 2021-07-31T01:55:14.666 INFO:tasks.rados.rados.0.smithi045.stdout:2979: seq_num 925 ranges {0=696320,696320=401408} 2021-07-31T01:55:14.669 INFO:tasks.rados.rados.0.smithi045.stdout:2979: writing smithi04569269-33 from 0 to 696320 tid 1 2021-07-31T01:55:14.671 INFO:tasks.rados.rados.0.smithi045.stdout:2979: writing smithi04569269-33 from 696320 to 1097728 tid 2 2021-07-31T01:55:14.672 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 13 current snap is 280 2021-07-31T01:55:14.672 INFO:tasks.rados.rados.0.smithi045.stdout:2980: seq_num 926 ranges {229376=712704,942080=401408} 2021-07-31T01:55:14.676 INFO:tasks.rados.rados.0.smithi045.stdout:2980: writing smithi04569269-13 from 229376 to 942080 tid 1 2021-07-31T01:55:14.679 INFO:tasks.rados.rados.0.smithi045.stdout:2980: writing smithi04569269-13 from 942080 to 1343488 tid 2 2021-07-31T01:55:14.680 INFO:tasks.rados.rados.0.smithi045.stdout:2981: rmattr oid 16 current snap is 280 2021-07-31T01:55:14.680 INFO:tasks.rados.rados.0.smithi045.stdout:2981: done (3 left) 2021-07-31T01:55:14.680 INFO:tasks.rados.rados.0.smithi045.stdout:2982: snap_create 2021-07-31T01:55:14.680 INFO:tasks.rados.rados.0.smithi045.stdout:2979: finishing write tid 1 to smithi04569269-33 2021-07-31T01:55:14.681 INFO:tasks.rados.rados.0.smithi045.stdout:2978: finishing copy_from to smithi04569269-23 2021-07-31T01:55:14.681 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 23 v 1644 (ObjNum 917 snap 278 seq_num 917) dirty exists 2021-07-31T01:55:14.682 INFO:tasks.rados.rados.0.smithi045.stdout:2978: finishing copy_from racing read to smithi04569269-23 2021-07-31T01:55:14.683 INFO:tasks.rados.rados.0.smithi045.stdout:2979: finishing write tid 2 to smithi04569269-33 2021-07-31T01:55:14.683 INFO:tasks.rados.rados.0.smithi045.stdout:2979: finishing write tid 3 to smithi04569269-33 2021-07-31T01:55:14.685 INFO:tasks.rados.rados.0.smithi045.stdout:2979: finishing write tid 4 to smithi04569269-33 2021-07-31T01:55:14.685 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 33 v 1669 (ObjNum 925 snap 280 seq_num 925) dirty exists 2021-07-31T01:55:14.685 INFO:tasks.rados.rados.0.smithi045.stdout:2979: left oid 33 (ObjNum 925 snap 280 seq_num 925) 2021-07-31T01:55:14.685 INFO:tasks.rados.rados.0.smithi045.stdout:2980: finishing write tid 1 to smithi04569269-13 2021-07-31T01:55:14.686 INFO:tasks.rados.rados.0.smithi045.stdout:2980: finishing write tid 2 to smithi04569269-13 2021-07-31T01:55:14.686 INFO:tasks.rados.rados.0.smithi045.stdout:2980: finishing write tid 3 to smithi04569269-13 2021-07-31T01:55:14.686 INFO:tasks.rados.rados.0.smithi045.stdout:2980: finishing write tid 4 to smithi04569269-13 2021-07-31T01:55:14.686 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 13 v 1836 (ObjNum 926 snap 280 seq_num 926) dirty exists 2021-07-31T01:55:14.686 INFO:tasks.rados.rados.0.smithi045.stdout:2980: left oid 13 (ObjNum 926 snap 280 seq_num 926) 2021-07-31T01:55:15.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:14 smithi154 conmon[34553]: cluster 2021-07-31T01:55:13.656079+0000 mon.a (mon.0) 1267 : cluster 2021-07-31T01:55:15.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:14 smithi154 conmon[34553]: [DBG] osdmap e607: 8 total, 8 up, 8 in 2021-07-31T01:55:15.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:14 smithi154 conmon[34553]: cluster 2021-07-31T01:55:13.905678+0000 mgr.x (mgr.24542) 340 : cluster [DBG] pgmap v694: 25 pgs: 25 active+clean; 446 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 22 MiB/s rd, 6.7 MiB/s wr, 39 op/s 2021-07-31T01:55:15.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:14 smithi045 conmon[33655]: cluster 2021-07-31T01:55:13.656079+0000 mon.a ( 2021-07-31T01:55:15.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:14 smithi045 conmon[33655]: mon.0) 1267 : cluster [DBG] osdmap e607: 8 total, 8 up, 8 in 2021-07-31T01:55:15.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:14 smithi045 conmon[33655]: cluster 2021-07-31T01:55:13.905678+0000 mgr.x (mgr.24542) 340 : cluster [DBG] pgmap v694: 25 pgs: 25 active+clean; 446 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 22 MiB/s rd, 6.7 MiB/s wr, 39 op/s 2021-07-31T01:55:15.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:14 smithi045 conmon[39718]: cluster 2021-07- 2021-07-31T01:55:15.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:14 smithi045 conmon[39718]: 31T01:55:13.656079+0000 mon.a (mon.0) 1267 : cluster [DBG] osdmap e607: 8 total, 8 up, 8 in 2021-07-31T01:55:15.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:14 smithi045 conmon[39718]: cluster 2021- 2021-07-31T01:55:15.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:14 smithi045 conmon[39718]: 07-31T01:55:13. 2021-07-31T01:55:15.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:14 smithi045 conmon[39718]: 905678+0000 mgr.x (mgr.24542) 340 2021-07-31T01:55:15.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:14 smithi045 conmon[39718]: : cluster [DBG] pgmap v694: 25 pgs: 25 active+clean; 446 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 22 MiB/s rd, 6.7 MiB/s wr, 39 op/s 2021-07-31T01:55:15.671 INFO:tasks.rados.rados.0.smithi045.stdout:2978: done (3 left) 2021-07-31T01:55:15.671 INFO:tasks.rados.rados.0.smithi045.stdout:2979: done (2 left) 2021-07-31T01:55:15.672 INFO:tasks.rados.rados.0.smithi045.stdout:2980: done (1 left) 2021-07-31T01:55:15.672 INFO:tasks.rados.rados.0.smithi045.stdout:2982: done (0 left) 2021-07-31T01:55:15.672 INFO:tasks.rados.rados.0.smithi045.stdout:2983: delete oid 7 current snap is 281 2021-07-31T01:55:15.675 INFO:tasks.rados.rados.0.smithi045.stdout:2983: done (0 left) 2021-07-31T01:55:15.675 INFO:tasks.rados.rados.0.smithi045.stdout:2984: snap_remove snap 259 2021-07-31T01:55:16.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:15 smithi154 conmon[34553]: cluster 2021-07-31T01:55:14.664617+0000 mon.a (mon.0) 1268 : cluster [DBG] osdmap e608: 8 total, 8 up, 8 in 2021-07-31T01:55:16.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:15 smithi045 conmon[39718]: cluster 2021-07-31T01:55:14 2021-07-31T01:55:16.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:15 smithi045 conmon[39718]: .664617+0000 mon.a (mon.0) 1268 : cluster [DBG] osdmap e608: 8 total, 8 up, 8 in 2021-07-31T01:55:16.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:15 smithi045 conmon[33655]: cluster 2021-07-31T01:55:14.664617+0000 mon.a (mon.0 2021-07-31T01:55:16.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:15 smithi045 conmon[33655]: ) 1268 : cluster [DBG] osdmap e608: 8 total, 8 up, 8 in 2021-07-31T01:55:16.632 INFO:tasks.rados.rados.0.smithi045.stdout:2984: done (0 left) 2021-07-31T01:55:16.632 INFO:tasks.rados.rados.0.smithi045.stdout:2985: read oid 20 snap -1 2021-07-31T01:55:16.633 INFO:tasks.rados.rados.0.smithi045.stdout:2985: expect (ObjNum 857 snap 268 seq_num 857) 2021-07-31T01:55:16.633 INFO:tasks.rados.rados.0.smithi045.stdout:2986: rollback oid 46 current snap is 281 2021-07-31T01:55:16.633 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 46 to 279 2021-07-31T01:55:16.633 INFO:tasks.rados.rados.0.smithi045.stdout:2987: read oid 21 snap -1 2021-07-31T01:55:16.634 INFO:tasks.rados.rados.0.smithi045.stdout:2987: expect (ObjNum 866 snap 270 seq_num 866) 2021-07-31T01:55:16.634 INFO:tasks.rados.rados.0.smithi045.stdout:2988: copy_from oid 26 from oid 48 current snap is 281 2021-07-31T01:55:16.634 INFO:tasks.rados.rados.0.smithi045.stdout:2989: snap_remove snap 278 2021-07-31T01:55:17.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:16 smithi154 conmon[34553]: cluster 2021-07-31T01:55:15.670908+0000 mon.a (mon.0) 1269 : cluster [DBG] 2021-07-31T01:55:17.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:16 smithi154 conmon[34553]: osdmap e609: 8 total, 8 up, 8 in 2021-07-31T01:55:17.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:16 smithi154 conmon[34553]: cluster 2021-07-31T01:55:15.906123+0000 mgr.x (mgr.24542) 341 : cluster [DBG] pgmap v697: 25 pgs: 25 active+clean; 446 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 22 MiB/s rd, 6.7 MiB/s wr, 39 op/s 2021-07-31T01:55:17.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:16 smithi154 conmon[34553]: cluster 2021-07-31T01:55:16.629440+0000 mon.a (mon.0) 1270 : cluster [DBG] osdmap e610: 8 total, 8 up, 8 in 2021-07-31T01:55:17.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:16 smithi045 conmon[33655]: cluster 2021-07-31T01:55:15.670908+0000 mon.a (mon.0) 2021-07-31T01:55:17.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:16 smithi045 conmon[33655]: 1269 : cluster [DBG] osdmap e609: 8 total, 8 up, 8 in 2021-07-31T01:55:17.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:16 smithi045 conmon[33655]: cluster 2021-07-31T01:55:15.906123+0000 mgr.x (mgr.24542) 341 : cluster [DBG] pgmap v697: 25 pgs: 25 active+clean; 446 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 22 MiB/s rd, 6.7 MiB/s wr, 39 op/s 2021-07-31T01:55:17.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:16 smithi045 conmon[33655]: cluster 2021-07-31T01:55:16.629440+0000 mon.a (mon.0) 1270 : cluster [DBG] osdmap e610: 8 total, 8 up, 8 in 2021-07-31T01:55:17.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:16 smithi045 conmon[39718]: cluster 2021-07-31T01:55 2021-07-31T01:55:17.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:16 smithi045 conmon[39718]: :15.670908+0000 mon.a (mon.0) 1269 : cluster [DBG] osdmap e609: 8 total, 8 up, 8 in 2021-07-31T01:55:17.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:16 smithi045 conmon[39718]: cluster 2021-07-31 2021-07-31T01:55:17.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:16 smithi045 conmon[39718]: T01:55:15.906123+0000 mgr.x (mgr.24542 2021-07-31T01:55:17.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:16 smithi045 conmon[39718]: ) 341 : cluster [DBG] pgmap v697: 25 pgs: 25 active+clean; 446 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 22 MiB/s rd, 6.7 MiB/s wr, 39 op/s 2021-07-31T01:55:17.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:16 smithi045 conmon[39718]: cluster 2021-07-31T01 2021-07-31T01:55:17.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:16 smithi045 conmon[39718]: :55:16.629440+0000 mon.a (mon.0 2021-07-31T01:55:17.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:16 smithi045 conmon[39718]: ) 1270 : cluster [DBG] osdmap e610: 8 total, 8 up, 8 in 2021-07-31T01:55:17.638 INFO:tasks.rados.rados.0.smithi045.stdout:2989: done (4 left) 2021-07-31T01:55:17.638 INFO:tasks.rados.rados.0.smithi045.stdout:2990: read oid 36 snap 268 2021-07-31T01:55:17.638 INFO:tasks.rados.rados.0.smithi045.stdout:2990: expect (ObjNum 835 snap 263 seq_num 835) 2021-07-31T01:55:17.639 INFO:tasks.rados.rados.0.smithi045.stdout:2991: copy_from oid 19 from oid 34 current snap is 281 2021-07-31T01:55:17.639 INFO:tasks.rados.rados.0.smithi045.stdout:2986: finishing rollback tid 0 to smithi04569269-46 2021-07-31T01:55:17.639 INFO:tasks.rados.rados.0.smithi045.stdout:2986: finishing rollback tid 1 to smithi04569269-46 2021-07-31T01:55:17.639 INFO:tasks.rados.rados.0.smithi045.stdout:2986: finishing rollback tid 2 to smithi04569269-46 2021-07-31T01:55:17.640 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 46 v 1308 (ObjNum 826 snap 258 seq_num 826) dirty exists 2021-07-31T01:55:17.640 INFO:tasks.rados.rados.0.smithi045.stdout:2985: expect (ObjNum 857 snap 268 seq_num 857) 2021-07-31T01:55:17.991 INFO:tasks.rados.rados.0.smithi045.stdout:2987: expect (ObjNum 866 snap 270 seq_num 866) 2021-07-31T01:55:18.043 INFO:tasks.rados.rados.0.smithi045.stdout:2988: finishing copy_from to smithi04569269-26 2021-07-31T01:55:18.043 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 26 v 1582 (ObjNum 881 snap 272 seq_num 881) dirty exists 2021-07-31T01:55:18.043 INFO:tasks.rados.rados.0.smithi045.stdout:2988: finishing copy_from racing read to smithi04569269-26 2021-07-31T01:55:18.044 INFO:tasks.rados.rados.0.smithi045.stdout:2985: done (5 left) 2021-07-31T01:55:18.044 INFO:tasks.rados.rados.0.smithi045.stdout:2986: done (4 left) 2021-07-31T01:55:18.044 INFO:tasks.rados.rados.0.smithi045.stdout:2987: done (3 left) 2021-07-31T01:55:18.044 INFO:tasks.rados.rados.0.smithi045.stdout:2988: done (2 left) 2021-07-31T01:55:18.045 INFO:tasks.rados.rados.0.smithi045.stdout:2992: snap_remove snap 277 2021-07-31T01:55:18.642 INFO:tasks.rados.rados.0.smithi045.stdout:2992: done (2 left) 2021-07-31T01:55:18.642 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 18 current snap is 281 2021-07-31T01:55:18.642 INFO:tasks.rados.rados.0.smithi045.stdout:2991: finishing copy_from to smithi04569269-19 2021-07-31T01:55:18.643 INFO:tasks.rados.rados.0.smithi045.stdout:2991: got expected ENOENT (src dne) 2021-07-31T01:55:18.643 INFO:tasks.rados.rados.0.smithi045.stdout:2991: finishing copy_from racing read to smithi04569269-19 2021-07-31T01:55:18.643 INFO:tasks.rados.rados.0.smithi045.stdout:2990: expect (ObjNum 835 snap 263 seq_num 835) 2021-07-31T01:55:18.700 INFO:tasks.rados.rados.0.smithi045.stdout:2993: seq_num 928 ranges {14745600=745472,15491072=401408} 2021-07-31T01:55:18.784 INFO:tasks.rados.rados.0.smithi045.stdout:2993: writing smithi04569269-18 from 14745600 to 15491072 tid 1 2021-07-31T01:55:18.786 INFO:tasks.rados.rados.0.smithi045.stdout:2993: writing smithi04569269-18 from 15491072 to 15892480 tid 2 2021-07-31T01:55:18.786 INFO:tasks.rados.rados.0.smithi045.stdout:2990: done (2 left) 2021-07-31T01:55:18.787 INFO:tasks.rados.rados.0.smithi045.stdout:2991: done (1 left) 2021-07-31T01:55:18.787 INFO:tasks.rados.rados.0.smithi045.stdout:2994: snap_create 2021-07-31T01:55:18.789 INFO:tasks.rados.rados.0.smithi045.stdout:2993: finishing write tid 1 to smithi04569269-18 2021-07-31T01:55:18.790 INFO:tasks.rados.rados.0.smithi045.stdout:2993: finishing write tid 2 to smithi04569269-18 2021-07-31T01:55:18.791 INFO:tasks.rados.rados.0.smithi045.stdout:2993: finishing write tid 3 to smithi04569269-18 2021-07-31T01:55:18.791 INFO:tasks.rados.rados.0.smithi045.stdout:2993: finishing write tid 4 to smithi04569269-18 2021-07-31T01:55:18.792 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 18 v 1664 (ObjNum 928 snap 281 seq_num 928) dirty exists 2021-07-31T01:55:18.792 INFO:tasks.rados.rados.0.smithi045.stdout:2993: left oid 18 (ObjNum 928 snap 281 seq_num 928) 2021-07-31T01:55:19.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:18 smithi154 conmon[34553]: cluster 2021-07-31T01:55:17.635840+0000 mon.a (mon.0) 2021-07-31T01:55:19.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:18 smithi154 conmon[34553]: 1271 : cluster [DBG] osdmap e611: 8 total, 8 up, 8 in 2021-07-31T01:55:19.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:18 smithi154 conmon[34553]: cluster 2021-07-31T01:55:17.906503+0000 mgr.x (mgr.24542) 342 : cluster [DBG] pgmap v700: 25 pgs: 25 active+clean; 446 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:55:19.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:18 smithi045 conmon[33655]: cluster 2021-07-31T01:55:17.635840+0000 mon.a (mon.0 2021-07-31T01:55:19.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:18 smithi045 conmon[33655]: ) 1271 : cluster [DBG] osdmap e611: 8 total, 8 up, 8 in 2021-07-31T01:55:19.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:18 smithi045 conmon[33655]: cluster 2021-07-31T01:55:17.906503+0000 mgr.x (mgr.24542) 342 : cluster [DBG] pgmap v700: 25 pgs: 25 active+clean; 446 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:55:19.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:18 smithi045 conmon[39718]: cluster 2021-07-31T01:55:17.635840+0000 mon.a ( 2021-07-31T01:55:19.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:18 smithi045 conmon[39718]: mon.0) 1271 : cluster [DBG] osdmap e611: 8 total, 8 up, 8 in 2021-07-31T01:55:19.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:18 smithi045 conmon[39718]: cluster 2021-07-31T01:55: 2021-07-31T01:55:19.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:18 smithi045 conmon[39718]: 17.906503+0000 mgr.x (mgr.24542) 342 : cluster [DBG] pgmap v700: 25 pgs: 25 active+clean; 446 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:55:19.643 INFO:tasks.rados.rados.0.smithi045.stdout:2993: done (1 left) 2021-07-31T01:55:19.644 INFO:tasks.rados.rados.0.smithi045.stdout:2994: done (0 left) 2021-07-31T01:55:19.644 INFO:tasks.rados.rados.0.smithi045.stdout:2995: rmattr oid 43 current snap is 282 2021-07-31T01:55:19.645 INFO:tasks.rados.rados.0.smithi045.stdout:2996: read oid 38 snap 263 2021-07-31T01:55:19.645 INFO:tasks.rados.rados.0.smithi045.stdout:2996: expect (ObjNum 834 snap 263 seq_num 834) 2021-07-31T01:55:19.646 INFO:tasks.rados.rados.0.smithi045.stdout:2997: delete oid 13 current snap is 282 2021-07-31T01:55:19.650 INFO:tasks.rados.rados.0.smithi045.stdout:2997: done (2 left) 2021-07-31T01:55:19.650 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 35 current snap is 282 2021-07-31T01:55:19.650 INFO:tasks.rados.rados.0.smithi045.stdout:2998: seq_num 930 ranges {0=786432,786432=401408} 2021-07-31T01:55:19.654 INFO:tasks.rados.rados.0.smithi045.stdout:2998: writing smithi04569269-35 from 0 to 786432 tid 1 2021-07-31T01:55:19.657 INFO:tasks.rados.rados.0.smithi045.stdout:2998: writing smithi04569269-35 from 786432 to 1187840 tid 2 2021-07-31T01:55:19.657 INFO:tasks.rados.rados.0.smithi045.stdout:2999: copy_from oid 40 from oid 33 current snap is 282 2021-07-31T01:55:19.658 INFO:tasks.rados.rados.0.smithi045.stdout:2996: expect (ObjNum 834 snap 263 seq_num 834) 2021-07-31T01:55:19.711 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 43 v 1835 (ObjNum 888 snap 272 seq_num 888) dirty exists 2021-07-31T01:55:19.711 INFO:tasks.rados.rados.0.smithi045.stdout:2998: finishing write tid 1 to smithi04569269-35 2021-07-31T01:55:19.711 INFO:tasks.rados.rados.0.smithi045.stdout:2998: finishing write tid 2 to smithi04569269-35 2021-07-31T01:55:19.712 INFO:tasks.rados.rados.0.smithi045.stdout:2998: finishing write tid 3 to smithi04569269-35 2021-07-31T01:55:19.712 INFO:tasks.rados.rados.0.smithi045.stdout:2999: finishing copy_from to smithi04569269-40 2021-07-31T01:55:19.712 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 40 v 1669 (ObjNum 925 snap 280 seq_num 925) dirty exists 2021-07-31T01:55:19.713 INFO:tasks.rados.rados.0.smithi045.stdout:2999: finishing copy_from racing read to smithi04569269-40 2021-07-31T01:55:19.713 INFO:tasks.rados.rados.0.smithi045.stdout:2998: finishing write tid 4 to smithi04569269-35 2021-07-31T01:55:19.713 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 35 v 1906 (ObjNum 930 snap 282 seq_num 930) dirty exists 2021-07-31T01:55:19.713 INFO:tasks.rados.rados.0.smithi045.stdout:2998: left oid 35 (ObjNum 930 snap 282 seq_num 930) 2021-07-31T01:55:19.714 INFO:tasks.rados.rados.0.smithi045.stdout:2995: done (3 left) 2021-07-31T01:55:19.714 INFO:tasks.rados.rados.0.smithi045.stdout:2996: done (2 left) 2021-07-31T01:55:19.714 INFO:tasks.rados.rados.0.smithi045.stdout:2998: done (1 left) 2021-07-31T01:55:19.714 INFO:tasks.rados.rados.0.smithi045.stdout:2999: done (0 left) 2021-07-31T01:55:19.715 INFO:tasks.rados.rados.0.smithi045.stdout:3000: delete oid 41 current snap is 282 2021-07-31T01:55:19.715 INFO:tasks.rados.rados.0.smithi045.stdout:3000: done (0 left) 2021-07-31T01:55:19.716 INFO:tasks.rados.rados.0.smithi045.stdout:3001: read oid 12 snap 275 2021-07-31T01:55:19.716 INFO:tasks.rados.rados.0.smithi045.stdout:3001: expect deleted 2021-07-31T01:55:19.716 INFO:tasks.rados.rados.0.smithi045.stdout:3002: snap_remove snap 280 2021-07-31T01:55:20.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:19 smithi154 conmon[34553]: cluster 2021-07-31T01:55:18.640898+0000 mon.a (mon.0) 1272 : cluster [DBG] osdmap e612: 8 total, 8 up, 8 in 2021-07-31T01:55:20.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:19 smithi045 conmon[33655]: cluster 2021-07-31T01:55:18.640898+0000 mon.a ( 2021-07-31T01:55:20.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:19 smithi045 conmon[33655]: mon.0) 1272 : cluster [DBG] osdmap e612: 8 total, 8 up, 8 in 2021-07-31T01:55:20.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:19 smithi045 conmon[39718]: cluster 2021-07-31T01:55:18.640898+0000 mon.a (mon. 2021-07-31T01:55:20.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:19 smithi045 conmon[39718]: 0) 1272 : cluster [DBG] osdmap e612: 8 total, 8 up, 8 in 2021-07-31T01:55:20.655 INFO:tasks.rados.rados.0.smithi045.stdout:3002: done (1 left) 2021-07-31T01:55:20.655 INFO:tasks.rados.rados.0.smithi045.stdout:3003: snap_create 2021-07-31T01:55:21.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:20 smithi154 conmon[34553]: cluster 2021-07-31T01:55:19.642369+0000 mon.a (mon.0 2021-07-31T01:55:21.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:20 smithi154 conmon[34553]: ) 1273 : cluster [DBG] osdmap e613: 8 total, 8 up, 8 in 2021-07-31T01:55:21.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:20 smithi154 conmon[34553]: cluster 2021-07-31T01:55:19.907101+0000 mgr.x (mgr.24542) 343 : cluster [DBG] pgmap v703: 25 pgs: 25 active+clean; 460 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 7.2 MiB/s rd, 3.8 MiB/s wr, 10 op/s 2021-07-31T01:55:21.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:20 smithi045 conmon[33655]: cluster 2021-07-31T01:55:19.642369+0000 mon.a (mon.0) 1273 2021-07-31T01:55:21.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:20 smithi045 conmon[33655]: : cluster [DBG] osdmap e613: 8 total, 8 up, 8 in 2021-07-31T01:55:21.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:20 smithi045 conmon[33655]: cluster 2021-07-31T01:55:19.907101+0000 mgr.x (mgr.24542) 343 : cluster [DBG] pgmap v703: 25 pgs: 25 active+clean; 460 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 7.2 MiB/s rd, 3.8 MiB/s wr, 10 op/s 2021-07-31T01:55:21.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:20 smithi045 conmon[39718]: cluster 2021-07-31T01:55:19. 2021-07-31T01:55:21.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:20 smithi045 conmon[39718]: 642369+0000 mon.a (mon.0) 1273 : cluster [DBG] osdmap e613: 8 total, 8 up, 8 in 2021-07-31T01:55:21.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:20 smithi045 conmon[39718]: cluster 2021-07-31T01:55:21.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:20 smithi045 conmon[39718]: 2021-07-31T01:55:19.907101+0000 mgr.x (mgr.24542) 2021-07-31T01:55:21.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:20 smithi045 conmon[39718]: 343 : cluster [DBG] pgmap v703: 25 pgs: 25 active+clean; 460 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 7.2 MiB/s rd, 3.8 MiB/s wr, 10 op/s 2021-07-31T01:55:21.631 INFO:tasks.rados.rados.0.smithi045.stdout:3001: done (1 left) 2021-07-31T01:55:21.632 INFO:tasks.rados.rados.0.smithi045.stdout:3003: done (0 left) 2021-07-31T01:55:21.632 INFO:tasks.rados.rados.0.smithi045.stdout:3004: read oid 37 snap -1 2021-07-31T01:55:21.632 INFO:tasks.rados.rados.0.smithi045.stdout:3004: expect (ObjNum 885 snap 272 seq_num 885) 2021-07-31T01:55:21.633 INFO:tasks.rados.rados.0.smithi045.stdout:3005: snap_create 2021-07-31T01:55:21.677 INFO:tasks.rados.rados.0.smithi045.stdout:3004: expect (ObjNum 885 snap 272 seq_num 885) 2021-07-31T01:55:22.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:21 smithi154 conmon[34553]: cluster 2021-07-31T01:55:20.654372+0000 mon.a (mon.0) 1274 : cluster [DBG] osdmap e614: 8 total, 8 up, 8 in 2021-07-31T01:55:22.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:21 smithi154 conmon[34553]: cluster 2021-07-31T01:55:21.630916+0000 mon.a (mon.0) 1275 : cluster [DBG] osdmap e615: 8 total, 8 up, 8 in 2021-07-31T01:55:22.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:21 smithi045 conmon[33655]: cluster 2021-07-31T01:55:20.654372+0000 mon.a (mon.0) 1274 : cluster 2021-07-31T01:55:22.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:21 smithi045 conmon[33655]: [DBG] osdmap e614: 8 total, 8 up, 8 in 2021-07-31T01:55:22.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:21 smithi045 conmon[33655]: cluster 2021-07-31T01:55:21.630916+0000 mon.a (mon.0) 1275 : cluster [DBG] osdmap e615: 8 total, 8 up, 8 in 2021-07-31T01:55:22.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:21 smithi045 conmon[39718]: cluster 2021-07-31T01:55:20.654372+0000 mon.a (mon.0 2021-07-31T01:55:22.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:21 smithi045 conmon[39718]: ) 1274 : cluster [DBG] osdmap e614: 8 total, 8 up, 8 in 2021-07-31T01:55:22.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:21 smithi045 conmon[39718]: cluster 2021-07-31T01:55:21.630916+0000 mon.a (mon.0) 1275 : cluster [DBG] osdmap e615: 8 total, 8 up, 8 in 2021-07-31T01:55:22.783 INFO:tasks.rados.rados.0.smithi045.stdout:3004: done (1 left) 2021-07-31T01:55:22.783 INFO:tasks.rados.rados.0.smithi045.stdout:3005: done (0 left) 2021-07-31T01:55:22.783 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 7 current snap is 284 2021-07-31T01:55:22.783 INFO:tasks.rados.rados.0.smithi045.stdout:3006: seq_num 932 ranges {0=417792,417792=663552,1081344=139264} 2021-07-31T01:55:22.784 INFO:tasks.rados.rados.0.smithi045.stdout:3006: writing smithi04569269-7 from 0 to 417792 tid 1 2021-07-31T01:55:22.788 INFO:tasks.rados.rados.0.smithi045.stdout:3006: writing smithi04569269-7 from 417792 to 1081344 tid 2 2021-07-31T01:55:22.790 INFO:tasks.rados.rados.0.smithi045.stdout:3006: writing smithi04569269-7 from 1081344 to 1220608 tid 3 2021-07-31T01:55:22.790 INFO:tasks.rados.rados.0.smithi045.stdout:3007: delete oid 47 current snap is 284 2021-07-31T01:55:22.790 INFO:tasks.rados.rados.0.smithi045.stdout:3006: finishing write tid 1 to smithi04569269-7 2021-07-31T01:55:22.793 INFO:tasks.rados.rados.0.smithi045.stdout:3006: finishing write tid 2 to smithi04569269-7 2021-07-31T01:55:22.794 INFO:tasks.rados.rados.0.smithi045.stdout:3007: done (1 left) 2021-07-31T01:55:22.794 INFO:tasks.rados.rados.0.smithi045.stdout:3008: read oid 21 snap -1 2021-07-31T01:55:22.794 INFO:tasks.rados.rados.0.smithi045.stdout:3008: expect (ObjNum 866 snap 270 seq_num 866) 2021-07-31T01:55:22.795 INFO:tasks.rados.rados.0.smithi045.stdout:3009: read oid 31 snap -1 2021-07-31T01:55:22.795 INFO:tasks.rados.rados.0.smithi045.stdout:3009: expect (ObjNum 813 snap 254 seq_num 813) 2021-07-31T01:55:22.796 INFO:tasks.rados.rados.0.smithi045.stdout:3010: delete oid 16 current snap is 284 2021-07-31T01:55:22.797 INFO:tasks.rados.rados.0.smithi045.stdout:3006: finishing write tid 3 to smithi04569269-7 2021-07-31T01:55:22.797 INFO:tasks.rados.rados.0.smithi045.stdout:3006: finishing write tid 4 to smithi04569269-7 2021-07-31T01:55:22.798 INFO:tasks.rados.rados.0.smithi045.stdout:3006: finishing write tid 5 to smithi04569269-7 2021-07-31T01:55:22.799 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 7 v 1647 (ObjNum 932 snap 284 seq_num 932) dirty exists 2021-07-31T01:55:22.799 INFO:tasks.rados.rados.0.smithi045.stdout:3006: left oid 7 (ObjNum 932 snap 284 seq_num 932) 2021-07-31T01:55:22.801 INFO:tasks.rados.rados.0.smithi045.stdout:3006: done (3 left) 2021-07-31T01:55:22.802 INFO:tasks.rados.rados.0.smithi045.stdout:3010: done (2 left) 2021-07-31T01:55:22.802 INFO:tasks.rados.rados.0.smithi045.stdout:3011: read oid 43 snap -1 2021-07-31T01:55:22.803 INFO:tasks.rados.rados.0.smithi045.stdout:3011: expect (ObjNum 888 snap 272 seq_num 888) 2021-07-31T01:55:22.803 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 15 current snap is 284 2021-07-31T01:55:22.804 INFO:tasks.rados.rados.0.smithi045.stdout:3012: seq_num 935 ranges {0=466944,466944=720896,1187840=81920} 2021-07-31T01:55:22.804 INFO:tasks.rados.rados.0.smithi045.stdout:3012: writing smithi04569269-15 from 0 to 466944 tid 1 2021-07-31T01:55:22.806 INFO:tasks.rados.rados.0.smithi045.stdout:3012: writing smithi04569269-15 from 466944 to 1187840 tid 2 2021-07-31T01:55:22.806 INFO:tasks.rados.rados.0.smithi045.stdout:3012: writing smithi04569269-15 from 1187840 to 1269760 tid 3 2021-07-31T01:55:22.806 INFO:tasks.rados.rados.0.smithi045.stdout:3013: rmattr oid 17 current snap is 284 2021-07-31T01:55:22.807 INFO:tasks.rados.rados.0.smithi045.stdout:3008: expect (ObjNum 866 snap 270 seq_num 866) 2021-07-31T01:55:22.858 INFO:tasks.rados.rados.0.smithi045.stdout:3012: finishing write tid 1 to smithi04569269-15 2021-07-31T01:55:22.859 INFO:tasks.rados.rados.0.smithi045.stdout:3009: expect (ObjNum 813 snap 254 seq_num 813) 2021-07-31T01:55:22.906 INFO:tasks.rados.rados.0.smithi045.stdout:3012: finishing write tid 2 to smithi04569269-15 2021-07-31T01:55:22.907 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 17 v 1670 (ObjNum 716 snap 222 seq_num 716) dirty exists 2021-07-31T01:55:22.907 INFO:tasks.rados.rados.0.smithi045.stdout:3012: finishing write tid 3 to smithi04569269-15 2021-07-31T01:55:22.908 INFO:tasks.rados.rados.0.smithi045.stdout:3012: finishing write tid 4 to smithi04569269-15 2021-07-31T01:55:22.908 INFO:tasks.rados.rados.0.smithi045.stdout:3012: finishing write tid 5 to smithi04569269-15 2021-07-31T01:55:22.908 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 15 v 1668 (ObjNum 935 snap 284 seq_num 935) dirty exists 2021-07-31T01:55:22.909 INFO:tasks.rados.rados.0.smithi045.stdout:3012: left oid 15 (ObjNum 935 snap 284 seq_num 935) 2021-07-31T01:55:22.909 INFO:tasks.rados.rados.0.smithi045.stdout:3011: expect (ObjNum 888 snap 272 seq_num 888) 2021-07-31T01:55:23.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:22 smithi154 conmon[34553]: cluster 2021-07-31T01:55:21.907546+0000 mgr.x (mgr.24542) 344 : cluster [DBG] 2021-07-31T01:55:23.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:22 smithi154 conmon[34553]: pgmap v706: 25 pgs: 25 active+clean; 460 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 7.2 MiB/s rd, 3.8 MiB/s wr, 10 op/s 2021-07-31T01:55:23.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:22 smithi154 conmon[34553]: cluster 2021-07-31T01:55:22.635865+0000 mon.a (mon.0) 1276 : cluster [DBG] osdmap e616: 8 total, 8 up, 8 in 2021-07-31T01:55:23.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:22 smithi045 conmon[33655]: cluster 2021-07-31T01:55:21.907546+0000 mgr.x (mgr.24542) 344 2021-07-31T01:55:23.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:22 smithi045 conmon[33655]: : cluster [DBG] pgmap v706: 25 pgs: 25 active+clean; 460 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 7.2 MiB/s rd, 3.8 MiB/s wr, 10 op/s 2021-07-31T01:55:23.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:22 smithi045 conmon[33655]: cluster 2021-07-31T01:55:22.635865+0000 mon.a (mon.0) 1276 : cluster [DBG] osdmap e616: 8 total, 8 up, 8 in 2021-07-31T01:55:23.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:22 smithi045 conmon[39718]: cluster 2021-07-31T01:55:21.907546 2021-07-31T01:55:23.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:22 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 344 : cluster [DBG] pgmap v706: 25 pgs: 25 active+clean; 460 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 7.2 MiB/s rd, 3.8 MiB/s wr, 10 op/s 2021-07-31T01:55:23.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:22 smithi045 conmon[39718]: cluster 2021-07-31T01:55:23.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:22 smithi045 conmon[39718]: 2021-07-31T01:55:22 2021-07-31T01:55:23.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:22 smithi045 conmon[39718]: .635865+0000 mon.a (mon. 2021-07-31T01:55:23.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:22 smithi045 conmon[39718]: 0) 1276 : cluster [DBG] 2021-07-31T01:55:23.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:22 smithi045 conmon[39718]: osdmap e616: 8 total, 8 up, 8 in 2021-07-31T01:55:23.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:55:23 smithi045 conmon[61210]: level=error ts=2021-07-31T01:55:23.523Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:55:23.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:55:23 smithi045 conmon[61210]: level=error ts=2021-07-31T01:55:23.523Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:55:23.778 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:55:23 smithi045 conmon[61210]: level=error ts=2021-07-31T01:55:23.523Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T01:55:23.936 INFO:tasks.rados.rados.0.smithi045.stdout:3008: done (4 left) 2021-07-31T01:55:23.936 INFO:tasks.rados.rados.0.smithi045.stdout:3009: done (3 left) 2021-07-31T01:55:23.937 INFO:tasks.rados.rados.0.smithi045.stdout:3011: done (2 left) 2021-07-31T01:55:23.937 INFO:tasks.rados.rados.0.smithi045.stdout:3012: done (1 left) 2021-07-31T01:55:23.937 INFO:tasks.rados.rados.0.smithi045.stdout:3013: done (0 left) 2021-07-31T01:55:23.938 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 30 current snap is 284 2021-07-31T01:55:23.938 INFO:tasks.rados.rados.0.smithi045.stdout:3014: seq_num 936 ranges {1056768=483328,1540096=655360,2195456=147456} 2021-07-31T01:55:23.943 INFO:tasks.rados.rados.0.smithi045.stdout:3014: writing smithi04569269-30 from 1056768 to 1540096 tid 1 2021-07-31T01:55:23.946 INFO:tasks.rados.rados.0.smithi045.stdout:3014: writing smithi04569269-30 from 1540096 to 2195456 tid 2 2021-07-31T01:55:23.948 INFO:tasks.rados.rados.0.smithi045.stdout:3014: writing smithi04569269-30 from 2195456 to 2342912 tid 3 2021-07-31T01:55:23.948 INFO:tasks.rados.rados.0.smithi045.stdout:3015: copy_from oid 36 from oid 5 current snap is 284 2021-07-31T01:55:23.949 INFO:tasks.rados.rados.0.smithi045.stdout:3014: finishing write tid 1 to smithi04569269-30 2021-07-31T01:55:23.949 INFO:tasks.rados.rados.0.smithi045.stdout:3016: delete oid 29 current snap is 284 2021-07-31T01:55:23.951 INFO:tasks.rados.rados.0.smithi045.stdout:3016: done (2 left) 2021-07-31T01:55:23.952 INFO:tasks.rados.rados.0.smithi045.stdout:3017: copy_from oid 39 from oid 16 current snap is 284 2021-07-31T01:55:23.952 INFO:tasks.rados.rados.0.smithi045.stdout:3014: finishing write tid 2 to smithi04569269-30 2021-07-31T01:55:23.952 INFO:tasks.rados.rados.0.smithi045.stdout:3018: read oid 18 snap -1 2021-07-31T01:55:23.952 INFO:tasks.rados.rados.0.smithi045.stdout:3018: expect (ObjNum 928 snap 281 seq_num 928) 2021-07-31T01:55:23.953 INFO:tasks.rados.rados.0.smithi045.stdout:3019: read oid 45 snap 244 2021-07-31T01:55:23.953 INFO:tasks.rados.rados.0.smithi045.stdout:3019: expect (ObjNum 747 snap 230 seq_num 747) 2021-07-31T01:55:23.953 INFO:tasks.rados.rados.0.smithi045.stdout:3020: copy_from oid 42 from oid 28 current snap is 284 2021-07-31T01:55:23.953 INFO:tasks.rados.rados.0.smithi045.stdout:3021: read oid 15 snap 279 2021-07-31T01:55:23.953 INFO:tasks.rados.rados.0.smithi045.stdout:3021: expect deleted 2021-07-31T01:55:23.954 INFO:tasks.rados.rados.0.smithi045.stdout:3022: snap_remove snap 251 2021-07-31T01:55:24.009 INFO:tasks.rados.rados.0.smithi045.stdout:3022: done (7 left) 2021-07-31T01:55:24.010 INFO:tasks.rados.rados.0.smithi045.stdout:3023: copy_from oid 8 from oid 6 current snap is 284 2021-07-31T01:55:24.013 INFO:tasks.rados.rados.0.smithi045.stdout:3024: read oid 22 snap -1 2021-07-31T01:55:24.014 INFO:tasks.rados.rados.0.smithi045.stdout:3024: expect (ObjNum 917 snap 278 seq_num 917) 2021-07-31T01:55:24.015 INFO:tasks.rados.rados.0.smithi045.stdout:3025: copy_from oid 27 from oid 10 current snap is 284 2021-07-31T01:55:24.015 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 3 current snap is 284 2021-07-31T01:55:24.015 INFO:tasks.rados.rados.0.smithi045.stdout:3026: seq_num 938 ranges {0=516096,516096=417792,933888=385024} 2021-07-31T01:55:24.015 INFO:tasks.rados.rados.0.smithi045.stdout:3026: writing smithi04569269-3 from 0 to 516096 tid 1 2021-07-31T01:55:24.016 INFO:tasks.rados.rados.0.smithi045.stdout:3026: writing smithi04569269-3 from 516096 to 933888 tid 2 2021-07-31T01:55:24.019 INFO:tasks.rados.rados.0.smithi045.stdout:3026: writing smithi04569269-3 from 933888 to 1318912 tid 3 2021-07-31T01:55:24.019 INFO:tasks.rados.rados.0.smithi045.stdout:3027: rollback oid 37 current snap is 284 2021-07-31T01:55:24.019 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 37 to 275 2021-07-31T01:55:24.019 INFO:tasks.rados.rados.0.smithi045.stdout:3014: finishing write tid 3 to smithi04569269-30 2021-07-31T01:55:24.019 INFO:tasks.rados.rados.0.smithi045.stdout:3014: finishing write tid 4 to smithi04569269-30 2021-07-31T01:55:24.020 INFO:tasks.rados.rados.0.smithi045.stdout:3017: finishing copy_from to smithi04569269-39 2021-07-31T01:55:24.020 INFO:tasks.rados.rados.0.smithi045.stdout:3017: got expected ENOENT (src dne) 2021-07-31T01:55:24.020 INFO:tasks.rados.rados.0.smithi045.stdout:3017: finishing copy_from racing read to smithi04569269-39 2021-07-31T01:55:24.020 INFO:tasks.rados.rados.0.smithi045.stdout:3019: expect (ObjNum 747 snap 230 seq_num 747) 2021-07-31T01:55:24.118 INFO:tasks.rados.rados.0.smithi045.stdout:3014: finishing write tid 5 to smithi04569269-30 2021-07-31T01:55:24.119 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 30 v 1642 (ObjNum 936 snap 284 seq_num 936) dirty exists 2021-07-31T01:55:24.119 INFO:tasks.rados.rados.0.smithi045.stdout:3014: left oid 30 (ObjNum 936 snap 284 seq_num 936) 2021-07-31T01:55:24.119 INFO:tasks.rados.rados.0.smithi045.stdout:3020: finishing copy_from to smithi04569269-42 2021-07-31T01:55:24.120 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 42 v 1658 (ObjNum 897 snap 273 seq_num 897) dirty exists 2021-07-31T01:55:24.120 INFO:tasks.rados.rados.0.smithi045.stdout:3020: finishing copy_from racing read to smithi04569269-42 2021-07-31T01:55:24.120 INFO:tasks.rados.rados.0.smithi045.stdout:3018: expect (ObjNum 928 snap 281 seq_num 928) 2021-07-31T01:55:25.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:25 smithi045 conmon[39718]: cluster 2021-07-31T01:55:23.908152+0000 2021-07-31T01:55:25.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:25 smithi045 conmon[39718]: mgr.x (mgr.24542) 345 : cluster [DBG] pgmap v708: 25 pgs: 25 active+clean; 464 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 18 MiB/s rd, 1.1 MiB/s wr, 14 op/s 2021-07-31T01:55:25.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:25 smithi045 conmon[39718]: cluster 2021-07-31 2021-07-31T01:55:25.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:25 smithi045 conmon[39718]: T01:55:24.007768+0000 mon.a (mon. 2021-07-31T01:55:25.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:25 smithi045 conmon[39718]: 0) 1277 : cluster [DBG] osdmap e617: 8 total, 8 up, 8 in 2021-07-31T01:55:25.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:25 smithi045 conmon[39718]: 2021-07-31T01:55:25.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:25 smithi045 conmon[33655]: cluster 2021-07-31T01:55:23.908152+0000 mgr.x ( 2021-07-31T01:55:25.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:25 smithi045 conmon[33655]: mgr.24542) 345 : cluster [DBG] pgmap v708: 25 pgs: 25 active+clean; 464 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 18 MiB/s rd, 1.1 MiB/s wr, 14 op/s 2021-07-31T01:55:25.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:25 smithi045 conmon[33655]: cluster 2021-07-31T01:55:24.007768+0000 mon.a (mon.0) 1277 : cluster [DBG] osdmap e617: 8 total, 8 up, 8 in 2021-07-31T01:55:25.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:25 smithi154 conmon[34553]: cluster 2021-07-31T01:55:23.908152+0000 mgr.x (mgr.24542) 345 : cluster 2021-07-31T01:55:25.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:25 smithi154 conmon[34553]: [DBG] pgmap v708: 25 pgs: 25 active+clean; 464 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 18 MiB/s rd, 1.1 MiB/s wr, 14 op/s 2021-07-31T01:55:25.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:25 smithi154 conmon[34553]: cluster 2021-07-31T01:55:24.007768+0000 mon.a (mon.0) 1277 : cluster [DBG] osdmap e617: 8 total, 8 up, 8 in 2021-07-31T01:55:25.793 INFO:tasks.rados.rados.0.smithi045.stdout:3024: expect (ObjNum 917 snap 278 seq_num 917) 2021-07-31T01:55:25.819 INFO:tasks.rados.rados.0.smithi045.stdout:3025: finishing copy_from to smithi04569269-27 2021-07-31T01:55:25.820 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 27 v 1640 (ObjNum 837 snap 263 seq_num 837) dirty exists 2021-07-31T01:55:25.820 INFO:tasks.rados.rados.0.smithi045.stdout:3025: finishing copy_from racing read to smithi04569269-27 2021-07-31T01:55:25.820 INFO:tasks.rados.rados.0.smithi045.stdout:3027: finishing rollback tid 0 to smithi04569269-37 2021-07-31T01:55:25.820 INFO:tasks.rados.rados.0.smithi045.stdout:3027: finishing rollback tid 1 to smithi04569269-37 2021-07-31T01:55:25.821 INFO:tasks.rados.rados.0.smithi045.stdout:3027: finishing rollback tid 2 to smithi04569269-37 2021-07-31T01:55:25.821 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 37 v 1671 (ObjNum 885 snap 272 seq_num 885) dirty exists 2021-07-31T01:55:25.821 INFO:tasks.rados.rados.0.smithi045.stdout:3015: finishing copy_from to smithi04569269-36 2021-07-31T01:55:25.821 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 36 v 1665 (ObjNum 922 snap 279 seq_num 922) dirty exists 2021-07-31T01:55:25.822 INFO:tasks.rados.rados.0.smithi045.stdout:3026: finishing write tid 1 to smithi04569269-3 2021-07-31T01:55:25.822 INFO:tasks.rados.rados.0.smithi045.stdout:3015: finishing copy_from racing read to smithi04569269-36 2021-07-31T01:55:25.822 INFO:tasks.rados.rados.0.smithi045.stdout:3026: finishing write tid 2 to smithi04569269-3 2021-07-31T01:55:25.822 INFO:tasks.rados.rados.0.smithi045.stdout:3026: finishing write tid 3 to smithi04569269-3 2021-07-31T01:55:25.822 INFO:tasks.rados.rados.0.smithi045.stdout:3026: finishing write tid 4 to smithi04569269-3 2021-07-31T01:55:25.823 INFO:tasks.rados.rados.0.smithi045.stdout:3026: finishing write tid 5 to smithi04569269-3 2021-07-31T01:55:25.823 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 3 v 1912 (ObjNum 938 snap 284 seq_num 938) dirty exists 2021-07-31T01:55:25.823 INFO:tasks.rados.rados.0.smithi045.stdout:3026: left oid 3 (ObjNum 938 snap 284 seq_num 938) 2021-07-31T01:55:25.823 INFO:tasks.rados.rados.0.smithi045.stdout:3014: done (11 left) 2021-07-31T01:55:25.824 INFO:tasks.rados.rados.0.smithi045.stdout:3015: done (10 left) 2021-07-31T01:55:25.824 INFO:tasks.rados.rados.0.smithi045.stdout:3017: done (9 left) 2021-07-31T01:55:25.824 INFO:tasks.rados.rados.0.smithi045.stdout:3018: done (8 left) 2021-07-31T01:55:25.824 INFO:tasks.rados.rados.0.smithi045.stdout:3019: done (7 left) 2021-07-31T01:55:25.824 INFO:tasks.rados.rados.0.smithi045.stdout:3020: done (6 left) 2021-07-31T01:55:25.825 INFO:tasks.rados.rados.0.smithi045.stdout:3021: done (5 left) 2021-07-31T01:55:25.825 INFO:tasks.rados.rados.0.smithi045.stdout:3024: done (4 left) 2021-07-31T01:55:25.825 INFO:tasks.rados.rados.0.smithi045.stdout:3025: done (3 left) 2021-07-31T01:55:25.825 INFO:tasks.rados.rados.0.smithi045.stdout:3026: done (2 left) 2021-07-31T01:55:25.825 INFO:tasks.rados.rados.0.smithi045.stdout:3027: done (1 left) 2021-07-31T01:55:25.826 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 39 current snap is 284 2021-07-31T01:55:25.826 INFO:tasks.rados.rados.0.smithi045.stdout:3023: finishing copy_from to smithi04569269-8 2021-07-31T01:55:25.826 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 8 v 1833 (ObjNum 813 snap 254 seq_num 813) dirty exists 2021-07-31T01:55:25.826 INFO:tasks.rados.rados.0.smithi045.stdout:3023: finishing copy_from racing read to smithi04569269-8 2021-07-31T01:55:25.827 INFO:tasks.rados.rados.0.smithi045.stdout:3028: seq_num 939 ranges {5578752=532480,6111232=704512,6815744=98304} 2021-07-31T01:55:25.851 INFO:tasks.rados.rados.0.smithi045.stdout:3028: writing smithi04569269-39 from 5578752 to 6111232 tid 1 2021-07-31T01:55:25.855 INFO:tasks.rados.rados.0.smithi045.stdout:3028: writing smithi04569269-39 from 6111232 to 6815744 tid 2 2021-07-31T01:55:25.856 INFO:tasks.rados.rados.0.smithi045.stdout:3028: writing smithi04569269-39 from 6815744 to 6914048 tid 3 2021-07-31T01:55:25.857 INFO:tasks.rados.rados.0.smithi045.stdout:3023: done (1 left) 2021-07-31T01:55:25.857 INFO:tasks.rados.rados.0.smithi045.stdout:3029: snap_create 2021-07-31T01:55:25.857 INFO:tasks.rados.rados.0.smithi045.stdout:3028: finishing write tid 1 to smithi04569269-39 2021-07-31T01:55:25.860 INFO:tasks.rados.rados.0.smithi045.stdout:3028: finishing write tid 2 to smithi04569269-39 2021-07-31T01:55:25.861 INFO:tasks.rados.rados.0.smithi045.stdout:3028: finishing write tid 3 to smithi04569269-39 2021-07-31T01:55:25.861 INFO:tasks.rados.rados.0.smithi045.stdout:3028: finishing write tid 4 to smithi04569269-39 2021-07-31T01:55:25.862 INFO:tasks.rados.rados.0.smithi045.stdout:3028: finishing write tid 5 to smithi04569269-39 2021-07-31T01:55:25.863 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 39 v 1673 (ObjNum 939 snap 284 seq_num 939) dirty exists 2021-07-31T01:55:25.863 INFO:tasks.rados.rados.0.smithi045.stdout:3028: left oid 39 (ObjNum 939 snap 284 seq_num 939) 2021-07-31T01:55:26.014 INFO:tasks.rados.rados.0.smithi045.stdout:3028: done (1 left) 2021-07-31T01:55:26.015 INFO:tasks.rados.rados.0.smithi045.stdout:3029: done (0 left) 2021-07-31T01:55:26.015 INFO:tasks.rados.rados.0.smithi045.stdout:3030: rmattr oid 26 current snap is 285 2021-07-31T01:55:26.015 INFO:tasks.rados.rados.0.smithi045.stdout:3031: rollback oid 10 current snap is 285 2021-07-31T01:55:26.015 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 10 to 275 2021-07-31T01:55:26.015 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 33 current snap is 285 2021-07-31T01:55:26.016 INFO:tasks.rados.rados.0.smithi045.stdout:3032: seq_num 940 ranges {1097728=548864,1646592=589824,2236416=212992} 2021-07-31T01:55:26.022 INFO:tasks.rados.rados.0.smithi045.stdout:3032: writing smithi04569269-33 from 1097728 to 1646592 tid 1 2021-07-31T01:55:26.026 INFO:tasks.rados.rados.0.smithi045.stdout:3032: writing smithi04569269-33 from 1646592 to 2236416 tid 2 2021-07-31T01:55:26.027 INFO:tasks.rados.rados.0.smithi045.stdout:3032: writing smithi04569269-33 from 2236416 to 2449408 tid 3 2021-07-31T01:55:26.027 INFO:tasks.rados.rados.0.smithi045.stdout:3033: snap_remove snap 272 2021-07-31T01:55:26.632 INFO:tasks.rados.rados.0.smithi045.stdout:3033: done (3 left) 2021-07-31T01:55:26.632 INFO:tasks.rados.rados.0.smithi045.stdout:3034: setattr oid 16 current snap is 285 2021-07-31T01:55:26.632 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 26 v 1837 (ObjNum 881 snap 272 seq_num 881) dirty exists 2021-07-31T01:55:26.633 INFO:tasks.rados.rados.0.smithi045.stdout:3031: finishing rollback tid 0 to smithi04569269-10 2021-07-31T01:55:26.633 INFO:tasks.rados.rados.0.smithi045.stdout:3031: finishing rollback tid 1 to smithi04569269-10 2021-07-31T01:55:26.633 INFO:tasks.rados.rados.0.smithi045.stdout:3031: finishing rollback tid 2 to smithi04569269-10 2021-07-31T01:55:26.633 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 10 v 1673 (ObjNum 837 snap 263 seq_num 837) dirty exists 2021-07-31T01:55:26.634 INFO:tasks.rados.rados.0.smithi045.stdout:3032: finishing write tid 1 to smithi04569269-33 2021-07-31T01:55:26.634 INFO:tasks.rados.rados.0.smithi045.stdout:3032: finishing write tid 2 to smithi04569269-33 2021-07-31T01:55:26.634 INFO:tasks.rados.rados.0.smithi045.stdout:3032: finishing write tid 3 to smithi04569269-33 2021-07-31T01:55:26.634 INFO:tasks.rados.rados.0.smithi045.stdout:3032: finishing write tid 4 to smithi04569269-33 2021-07-31T01:55:26.634 INFO:tasks.rados.rados.0.smithi045.stdout:3032: finishing write tid 5 to smithi04569269-33 2021-07-31T01:55:26.635 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 33 v 1677 (ObjNum 940 snap 285 seq_num 940) dirty exists 2021-07-31T01:55:26.635 INFO:tasks.rados.rados.0.smithi045.stdout:3032: left oid 33 (ObjNum 940 snap 285 seq_num 940) 2021-07-31T01:55:26.636 INFO:tasks.rados.rados.0.smithi045.stdout:3030: done (3 left) 2021-07-31T01:55:26.636 INFO:tasks.rados.rados.0.smithi045.stdout:3031: done (2 left) 2021-07-31T01:55:26.637 INFO:tasks.rados.rados.0.smithi045.stdout:3032: done (1 left) 2021-07-31T01:55:26.637 INFO:tasks.rados.rados.0.smithi045.stdout:3035: read oid 44 snap -1 2021-07-31T01:55:26.637 INFO:tasks.rados.rados.0.smithi045.stdout:3035: expect (ObjNum 813 snap 254 seq_num 813) 2021-07-31T01:55:26.637 INFO:tasks.rados.rados.0.smithi045.stdout:3036: read oid 50 snap -1 2021-07-31T01:55:26.638 INFO:tasks.rados.rados.0.smithi045.stdout:3036: expect (ObjNum 837 snap 263 seq_num 837) 2021-07-31T01:55:26.638 INFO:tasks.rados.rados.0.smithi045.stdout:3037: copy_from oid 42 from oid 11 current snap is 285 2021-07-31T01:55:26.639 INFO:tasks.rados.rados.0.smithi045.stdout:3038: rmattr oid 2 current snap is 285 2021-07-31T01:55:26.639 INFO:tasks.rados.rados.0.smithi045.stdout:3038: done (4 left) 2021-07-31T01:55:26.640 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 12 current snap is 285 2021-07-31T01:55:26.640 INFO:tasks.rados.rados.0.smithi045.stdout:3039: seq_num 941 ranges {0=565248,565248=475136,1040384=327680} 2021-07-31T01:55:26.641 INFO:tasks.rados.rados.0.smithi045.stdout:3039: writing smithi04569269-12 from 0 to 565248 tid 1 2021-07-31T01:55:26.641 INFO:tasks.rados.rados.0.smithi045.stdout:3039: writing smithi04569269-12 from 565248 to 1040384 tid 2 2021-07-31T01:55:26.642 INFO:tasks.rados.rados.0.smithi045.stdout:3039: writing smithi04569269-12 from 1040384 to 1368064 tid 3 2021-07-31T01:55:26.642 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 28 current snap is 285 2021-07-31T01:55:26.642 INFO:tasks.rados.rados.0.smithi045.stdout:3040: seq_num 942 ranges {630784=581632,1212416=761856,1974272=40960} 2021-07-31T01:55:26.649 INFO:tasks.rados.rados.0.smithi045.stdout:3040: writing smithi04569269-28 from 630784 to 1212416 tid 1 2021-07-31T01:55:26.654 INFO:tasks.rados.rados.0.smithi045.stdout:3040: writing smithi04569269-28 from 1212416 to 1974272 tid 2 2021-07-31T01:55:26.655 INFO:tasks.rados.rados.0.smithi045.stdout:3040: writing smithi04569269-28 from 1974272 to 2015232 tid 3 2021-07-31T01:55:26.655 INFO:tasks.rados.rados.0.smithi045.stdout:3041: snap_remove snap 275 2021-07-31T01:55:27.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:27 smithi045 conmon[39718]: cluster 2021-07-31T01:55:25.908618+0000 mgr.x (mgr 2021-07-31T01:55:27.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:27 smithi045 conmon[39718]: .24542) 346 : cluster [DBG] pgmap v710: 25 pgs: 25 active+clean; 464 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 14 MiB/s rd, 888 KiB/s wr, 11 op/s 2021-07-31T01:55:27.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:27 smithi045 conmon[39718]: cluster 2021-07-31T01:55:26.013215+0000 2021-07-31T01:55:27.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:27 smithi045 conmon[39718]: mon.a (mon.0) 1278 : cluster [DBG] osdmap e618: 8 total, 8 up, 8 in 2021-07-31T01:55:27.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:27 smithi045 conmon[39718]: cluster 2021-07-31 2021-07-31T01:55:27.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:27 smithi045 conmon[39718]: T01:55:26.631290+0000 2021-07-31T01:55:27.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:27 smithi045 conmon[39718]: mon.a (mon.0) 1279 : cluster 2021-07-31T01:55:27.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:27 smithi045 conmon[39718]: [DBG] osdmap e619: 8 total, 8 up, 8 in 2021-07-31T01:55:27.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:27 smithi045 conmon[33655]: cluster 2021-07-31T01:55:25.908618+0000 mgr.x (mgr.24542) 346 : 2021-07-31T01:55:27.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:27 smithi045 conmon[33655]: cluster [DBG] pgmap v710: 25 pgs: 25 active+clean; 464 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 14 MiB/s rd, 888 KiB/s wr, 11 op/s 2021-07-31T01:55:27.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:27 smithi045 conmon[33655]: cluster 2021-07-31T01:55:26.013215+0000 mon.a (mon.0) 1278 : cluster [DBG] osdmap e618: 8 total, 8 up, 8 in 2021-07-31T01:55:27.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:27 smithi045 conmon[33655]: cluster 2021-07-31T01:55:26.631290+0000 mon.a (mon.0) 1279 : cluster [DBG] osdmap e619: 8 total, 8 up, 8 in 2021-07-31T01:55:27.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:27 smithi154 conmon[34553]: cluster 2021-07-31T01:55:25.908618+0000 mgr.x (mgr.24542) 346 : cluster [DBG] pgmap v710: 25 pgs: 25 active+clean; 464 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 14 MiB/s rd, 888 KiB/s wr, 11 op/s 2021-07-31T01:55:27.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:27 smithi154 conmon[34553]: cluster 2021-07-31T01:55:26.013215+0000 mon.a (mon.0) 1278 : cluster [DBG] osdmap e618: 8 total, 8 up, 8 in 2021-07-31T01:55:27.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:27 smithi154 conmon[34553]: cluster 2021-07-31T01:55:26.631290+0000 mon.a (mon.0) 1279 2021-07-31T01:55:27.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:27 smithi154 conmon[34553]: : cluster [DBG] osdmap e619: 8 total, 8 up, 8 in 2021-07-31T01:55:27.650 INFO:tasks.rados.rados.0.smithi045.stdout:3041: done (6 left) 2021-07-31T01:55:27.651 INFO:tasks.rados.rados.0.smithi045.stdout:3042: snap_create 2021-07-31T01:55:27.652 INFO:tasks.rados.rados.0.smithi045.stdout:3036: expect (ObjNum 837 snap 263 seq_num 837) 2021-07-31T01:55:27.711 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 16 v 1646 (ObjNum -1229692672 snap 32731 seq_num 3065274624) dirty exists 2021-07-31T01:55:27.712 INFO:tasks.rados.rados.0.smithi045.stdout:3039: finishing write tid 1 to smithi04569269-12 2021-07-31T01:55:27.712 INFO:tasks.rados.rados.0.smithi045.stdout:3039: finishing write tid 2 to smithi04569269-12 2021-07-31T01:55:27.712 INFO:tasks.rados.rados.0.smithi045.stdout:3039: finishing write tid 3 to smithi04569269-12 2021-07-31T01:55:27.712 INFO:tasks.rados.rados.0.smithi045.stdout:3039: finishing write tid 4 to smithi04569269-12 2021-07-31T01:55:27.715 INFO:tasks.rados.rados.0.smithi045.stdout:3039: finishing write tid 5 to smithi04569269-12 2021-07-31T01:55:27.715 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 12 v 1841 (ObjNum 941 snap 285 seq_num 941) dirty exists 2021-07-31T01:55:27.716 INFO:tasks.rados.rados.0.smithi045.stdout:3039: left oid 12 (ObjNum 941 snap 285 seq_num 941) 2021-07-31T01:55:27.716 INFO:tasks.rados.rados.0.smithi045.stdout:3035: expect (ObjNum 813 snap 254 seq_num 813) 2021-07-31T01:55:27.758 INFO:tasks.rados.rados.0.smithi045.stdout:3040: finishing write tid 1 to smithi04569269-28 2021-07-31T01:55:27.758 INFO:tasks.rados.rados.0.smithi045.stdout:3040: finishing write tid 2 to smithi04569269-28 2021-07-31T01:55:27.759 INFO:tasks.rados.rados.0.smithi045.stdout:3040: finishing write tid 3 to smithi04569269-28 2021-07-31T01:55:27.759 INFO:tasks.rados.rados.0.smithi045.stdout:3037: finishing copy_from to smithi04569269-42 2021-07-31T01:55:27.759 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 42 v 1662 (ObjNum 920 snap 278 seq_num 920) dirty exists 2021-07-31T01:55:27.759 INFO:tasks.rados.rados.0.smithi045.stdout:3037: finishing copy_from racing read to smithi04569269-42 2021-07-31T01:55:27.760 INFO:tasks.rados.rados.0.smithi045.stdout:3040: finishing write tid 4 to smithi04569269-28 2021-07-31T01:55:27.760 INFO:tasks.rados.rados.0.smithi045.stdout:3040: finishing write tid 5 to smithi04569269-28 2021-07-31T01:55:27.760 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 28 v 1677 (ObjNum 942 snap 285 seq_num 942) dirty exists 2021-07-31T01:55:27.760 INFO:tasks.rados.rados.0.smithi045.stdout:3040: left oid 28 (ObjNum 942 snap 285 seq_num 942) 2021-07-31T01:55:28.655 INFO:tasks.rados.rados.0.smithi045.stdout:3034: done (6 left) 2021-07-31T01:55:28.655 INFO:tasks.rados.rados.0.smithi045.stdout:3035: done (5 left) 2021-07-31T01:55:28.656 INFO:tasks.rados.rados.0.smithi045.stdout:3036: done (4 left) 2021-07-31T01:55:28.656 INFO:tasks.rados.rados.0.smithi045.stdout:3037: done (3 left) 2021-07-31T01:55:28.656 INFO:tasks.rados.rados.0.smithi045.stdout:3039: done (2 left) 2021-07-31T01:55:28.656 INFO:tasks.rados.rados.0.smithi045.stdout:3040: done (1 left) 2021-07-31T01:55:28.657 INFO:tasks.rados.rados.0.smithi045.stdout:3042: done (0 left) 2021-07-31T01:55:28.657 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 17 current snap is 286 2021-07-31T01:55:28.657 INFO:tasks.rados.rados.0.smithi045.stdout:3043: seq_num 943 ranges {2400256=598016,2998272=688128,3686400=114688} 2021-07-31T01:55:28.673 INFO:tasks.rados.rados.0.smithi045.stdout:3043: writing smithi04569269-17 from 2400256 to 2998272 tid 1 2021-07-31T01:55:28.674 INFO:tasks.rados.rados.0.smithi045.stdout:3043: writing smithi04569269-17 from 2998272 to 3686400 tid 2 2021-07-31T01:55:28.675 INFO:tasks.rados.rados.0.smithi045.stdout:3043: writing smithi04569269-17 from 3686400 to 3801088 tid 3 2021-07-31T01:55:28.676 INFO:tasks.rados.rados.0.smithi045.stdout:3044: read oid 7 snap -1 2021-07-31T01:55:28.676 INFO:tasks.rados.rados.0.smithi045.stdout:3044: expect (ObjNum 932 snap 284 seq_num 932) 2021-07-31T01:55:28.676 INFO:tasks.rados.rados.0.smithi045.stdout:3045: snap_remove snap 282 2021-07-31T01:55:28.999 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:28 smithi154 conmon[34553]: cluster 2021-07-31T01:55:27.647433+0000 mon.a (mon.0) 1280 : cluster [DBG] osdmap e620: 8 total, 8 up, 8 in 2021-07-31T01:55:28.999 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:28 smithi154 conmon[34553]: cluster 2021-07-31T01:55:27.909014+0000 mgr.x (mgr.24542) 347 : cluster [DBG] pgmap v714: 25 pgs: 25 active+clean; 464 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:55:29.000 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:28 smithi154 conmon[34553]: audit 2021-07-31T01:55:27.967679+0000 mon.b (mon.2) 102 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:55:29.000 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:28 smithi154 conmon[34553]: audit 2021-07-31T01:55:27.968293+0000 mon.a (mon.0) 1281 : 2021-07-31T01:55:29.000 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:28 smithi154 conmon[34553]: audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:55:29.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:28 smithi154 conmon[34553]: audit 2021-07-31T01:55:29.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:28 smithi154 conmon[34553]: 2021-07-31T01:55:27.982298+0000 mon.b (mon.2) 103 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T01:55:29.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:28 smithi154 conmon[34553]: audit 2021 2021-07-31T01:55:29.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:28 smithi154 conmon[34553]: -07-31T01:55:27.982804+0000 2021-07-31T01:55:29.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:28 smithi154 conmon[34553]: mon.a (mon.0) 1282 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T01:55:29.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:28 smithi154 conmon[34553]: 2021-07-31T01:55:29.002 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:28 smithi045 conmon[39718]: cluster 2021-07-31T01:55:27.647433+0000 2021-07-31T01:55:29.003 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:28 smithi045 conmon[39718]: mon.a (mon.0) 1280 : cluster [DBG] osdmap e620: 8 total, 8 up, 8 in 2021-07-31T01:55:29.003 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:28 smithi045 conmon[39718]: cluster 2021 2021-07-31T01:55:29.003 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:28 smithi045 conmon[39718]: -07-31T01:55: 2021-07-31T01:55:29.003 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:28 smithi045 conmon[39718]: 27.909014+0000 mgr.x (mgr 2021-07-31T01:55:29.004 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:28 smithi045 conmon[39718]: .24542) 347 : cluster [DBG] 2021-07-31T01:55:29.004 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:28 smithi045 conmon[39718]: pgmap v714: 25 pgs: 25 active+clean; 464 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:55:29.004 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:28 smithi045 conmon[39718]: audit 2021 2021-07-31T01:55:29.004 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:28 smithi045 conmon[39718]: -07-31T01:55:27 2021-07-31T01:55:29.005 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:28 smithi045 conmon[39718]: .967679+0000 mon.b (mon. 2021-07-31T01:55:29.005 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:28 smithi045 conmon[39718]: 2) 102 : audit [INF] 2021-07-31T01:55:29.005 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:28 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:55:29.006 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:28 smithi045 conmon[39718]: audit 2021 2021-07-31T01:55:29.006 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:28 smithi045 conmon[39718]: -07-31T01:55:27 2021-07-31T01:55:29.006 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:28 smithi045 conmon[39718]: .968293+0000 mon.a (mon. 2021-07-31T01:55:29.007 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:28 smithi045 conmon[39718]: 0) 1281 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:55:29.007 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:28 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T01:55:29.007 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:28 smithi045 conmon[39718]: 31T01:55:27.982298+0000 mon.b (mon.2) 103 2021-07-31T01:55:29.007 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:28 smithi045 conmon[39718]: : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T01:55:29.007 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:28 smithi045 conmon[39718]: audit 2021-07-31T01:55:29.008 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:28 smithi045 conmon[39718]: 2021-07-31T01:55:27 2021-07-31T01:55:29.008 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:28 smithi045 conmon[39718]: .982804+0000 mon.a (mon.0) 1282 : audit [INF] 2021-07-31T01:55:29.008 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:28 smithi045 conmon[39718]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T01:55:29.008 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:28 smithi045 conmon[33655]: cluster 2021-07-31T01:55:27.647433+0000 mon.a (mon.0) 2021-07-31T01:55:29.009 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:28 smithi045 conmon[33655]: 1280 : cluster [DBG] osdmap e620: 8 total, 8 up, 8 in 2021-07-31T01:55:29.009 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:28 smithi045 conmon[33655]: cluster 2021-07-31T01:55:27.909014+0000 mgr.x (mgr.24542) 347 : cluster [DBG] pgmap v714: 25 pgs: 25 active+clean; 464 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:55:29.009 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:28 smithi045 conmon[33655]: audit 2021-07-31T01:55:27.967679+0000 mon.b (mon.2) 102 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:55:29.009 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:28 smithi045 conmon[33655]: audit 2021-07-31T01:55:27.968293+0000 mon.a (mon.0) 1281 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:55:29.010 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:28 smithi045 conmon[33655]: audit 2021-07-31T01:55:27.982298+0000 mon.b (mon.2) 103 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T01:55:29.010 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:28 smithi045 conmon[33655]: audit 2021-07-31T01:55:27.982804+0000 mon.a (mon.0) 1282 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T01:55:29.253 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: debug 2021-07-31T01:55:29.017+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T01:55:29.253 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:55:29.253 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:55:29.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: yield 2021-07-31T01:55:29.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:55:29.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:55:29.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:55:29.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T01:55:29.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:55:29.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T01:55:29.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:55:29.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T01:55:29.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:55:29.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: task.result() 2021-07-31T01:55:29.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T01:55:29.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: debug 2021-07-31T01:55:29.028+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T01:55:29.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:55:29.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:55:29.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: yield 2021-07-31T01:55:29.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:55:29.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:55:29.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:55:29.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T01:55:29.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:55:29.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T01:55:29.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:55:29.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T01:55:29.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:55:29.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: task.result() 2021-07-31T01:55:29.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T01:55:29.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: debug 2021-07-31T01:55:29.055+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T01:55:29.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:55:29.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:55:29.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: yield 2021-07-31T01:55:29.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:55:29.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:55:29.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:55:29.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T01:55:29.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:55:29.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T01:55:29.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:55:29.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T01:55:29.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:55:29.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: task.result() 2021-07-31T01:55:29.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T01:55:29.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: debug 2021-07-31T01:55:29.056+0000 7fcb9115c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:55:29.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:55:29.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:55:29.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: yield 2021-07-31T01:55:29.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:55:29.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:55:29.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:55:29.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T01:55:29.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:55:29.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T01:55:29.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:55:29.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T01:55:29.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:55:29.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: task.result() 2021-07-31T01:55:29.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T01:55:29.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: 2021-07-31T01:55:29.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T01:55:29.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: 2021-07-31T01:55:29.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:55:29.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:55:29.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: return f(*arg) 2021-07-31T01:55:29.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:55:29.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:55:29.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:55:29.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:55:29.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:55:29.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:55:29.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:55:29.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T01:55:29.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:55:29.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: raise self._exception 2021-07-31T01:55:29.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:55:29.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:55:29.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:55:29.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T01:55:29.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:55:29.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: raise 2021-07-31T01:55:29.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:55:29.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T01:55:29.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:55:29.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:55:29.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T01:55:29.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: debug 2021-07-31T01:55:29.064+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T01:55:29.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:55:29.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:55:29.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: yield 2021-07-31T01:55:29.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:55:29.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:55:29.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:55:29.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T01:55:29.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:55:29.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T01:55:29.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:55:29.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T01:55:29.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:55:29.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: task.result() 2021-07-31T01:55:29.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T01:55:29.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: debug 2021-07-31T01:55:29.064+0000 7fcb9095b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:55:29.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:55:29.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:55:29.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: yield 2021-07-31T01:55:29.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:55:29.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:55:29.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:55:29.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T01:55:29.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:55:29.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T01:55:29.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:55:29.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T01:55:29.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:55:29.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: task.result() 2021-07-31T01:55:29.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T01:55:29.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: 2021-07-31T01:55:29.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T01:55:29.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: 2021-07-31T01:55:29.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:55:29.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:55:29.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: return f(*arg) 2021-07-31T01:55:29.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:55:29.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:55:29.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:55:29.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:55:29.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:55:29.284 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:55:29.284 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:55:29.284 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T01:55:29.284 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:55:29.285 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: raise self._exception 2021-07-31T01:55:29.285 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:55:29.285 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:55:29.285 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:55:29.285 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T01:55:29.286 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:55:29.286 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: raise 2021-07-31T01:55:29.286 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:55:29.286 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T01:55:29.287 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:55:29.287 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:55:29.287 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T01:55:29.656 INFO:tasks.rados.rados.0.smithi045.stdout:3045: done (2 left) 2021-07-31T01:55:29.656 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 16 current snap is 286 2021-07-31T01:55:29.656 INFO:tasks.rados.rados.0.smithi045.stdout:3046: seq_num 944 ranges {0=614400,614400=573440,1187840=229376} 2021-07-31T01:55:29.660 INFO:tasks.rados.rados.0.smithi045.stdout:3046: writing smithi04569269-16 from 0 to 614400 tid 1 2021-07-31T01:55:29.663 INFO:tasks.rados.rados.0.smithi045.stdout:3046: writing smithi04569269-16 from 614400 to 1187840 tid 2 2021-07-31T01:55:29.666 INFO:tasks.rados.rados.0.smithi045.stdout:3046: writing smithi04569269-16 from 1187840 to 1417216 tid 3 2021-07-31T01:55:29.666 INFO:tasks.rados.rados.0.smithi045.stdout:3047: rmattr oid 29 current snap is 286 2021-07-31T01:55:29.666 INFO:tasks.rados.rados.0.smithi045.stdout:3047: done (3 left) 2021-07-31T01:55:29.667 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 10 current snap is 286 2021-07-31T01:55:29.667 INFO:tasks.rados.rados.0.smithi045.stdout:3048: seq_num 945 ranges {2031616=630784,2662400=458752,3121152=344064} 2021-07-31T01:55:29.680 INFO:tasks.rados.rados.0.smithi045.stdout:3048: writing smithi04569269-10 from 2031616 to 2662400 tid 1 2021-07-31T01:55:29.683 INFO:tasks.rados.rados.0.smithi045.stdout:3048: writing smithi04569269-10 from 2662400 to 3121152 tid 2 2021-07-31T01:55:29.685 INFO:tasks.rados.rados.0.smithi045.stdout:3048: writing smithi04569269-10 from 3121152 to 3465216 tid 3 2021-07-31T01:55:29.685 INFO:tasks.rados.rados.0.smithi045.stdout:3049: delete oid 19 current snap is 286 2021-07-31T01:55:29.686 INFO:tasks.rados.rados.0.smithi045.stdout:3043: finishing write tid 1 to smithi04569269-17 2021-07-31T01:55:29.686 INFO:tasks.rados.rados.0.smithi045.stdout:3043: finishing write tid 2 to smithi04569269-17 2021-07-31T01:55:29.686 INFO:tasks.rados.rados.0.smithi045.stdout:3043: finishing write tid 3 to smithi04569269-17 2021-07-31T01:55:29.686 INFO:tasks.rados.rados.0.smithi045.stdout:3043: finishing write tid 4 to smithi04569269-17 2021-07-31T01:55:29.687 INFO:tasks.rados.rados.0.smithi045.stdout:3044: expect (ObjNum 932 snap 284 seq_num 932) 2021-07-31T01:55:29.717 INFO:tasks.rados.rados.0.smithi045.stdout:3043: finishing write tid 5 to smithi04569269-17 2021-07-31T01:55:29.718 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 17 v 1681 (ObjNum 943 snap 286 seq_num 943) dirty exists 2021-07-31T01:55:29.718 INFO:tasks.rados.rados.0.smithi045.stdout:3043: left oid 17 (ObjNum 943 snap 286 seq_num 943) 2021-07-31T01:55:29.718 INFO:tasks.rados.rados.0.smithi045.stdout:3046: finishing write tid 1 to smithi04569269-16 2021-07-31T01:55:29.719 INFO:tasks.rados.rados.0.smithi045.stdout:3046: finishing write tid 2 to smithi04569269-16 2021-07-31T01:55:29.719 INFO:tasks.rados.rados.0.smithi045.stdout:3046: finishing write tid 3 to smithi04569269-16 2021-07-31T01:55:29.719 INFO:tasks.rados.rados.0.smithi045.stdout:3046: finishing write tid 4 to smithi04569269-16 2021-07-31T01:55:29.719 INFO:tasks.rados.rados.0.smithi045.stdout:3043: done (4 left) 2021-07-31T01:55:29.720 INFO:tasks.rados.rados.0.smithi045.stdout:3044: done (3 left) 2021-07-31T01:55:29.720 INFO:tasks.rados.rados.0.smithi045.stdout:3049: done (2 left) 2021-07-31T01:55:29.720 INFO:tasks.rados.rados.0.smithi045.stdout:3050: copy_from oid 9 from oid 26 current snap is 286 2021-07-31T01:55:29.720 INFO:tasks.rados.rados.0.smithi045.stdout:3046: finishing write tid 5 to smithi04569269-16 2021-07-31T01:55:29.720 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 16 v 1650 (ObjNum 944 snap 286 seq_num 944) dirty exists 2021-07-31T01:55:29.721 INFO:tasks.rados.rados.0.smithi045.stdout:3046: left oid 16 (ObjNum 944 snap 286 seq_num 944) 2021-07-31T01:55:29.721 INFO:tasks.rados.rados.0.smithi045.stdout:3048: finishing write tid 1 to smithi04569269-10 2021-07-31T01:55:29.721 INFO:tasks.rados.rados.0.smithi045.stdout:3048: finishing write tid 2 to smithi04569269-10 2021-07-31T01:55:29.721 INFO:tasks.rados.rados.0.smithi045.stdout:3048: finishing write tid 3 to smithi04569269-10 2021-07-31T01:55:29.721 INFO:tasks.rados.rados.0.smithi045.stdout:3048: finishing write tid 4 to smithi04569269-10 2021-07-31T01:55:29.722 INFO:tasks.rados.rados.0.smithi045.stdout:3048: finishing write tid 5 to smithi04569269-10 2021-07-31T01:55:29.722 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 10 v 1685 (ObjNum 945 snap 286 seq_num 945) dirty exists 2021-07-31T01:55:29.722 INFO:tasks.rados.rados.0.smithi045.stdout:3048: left oid 10 (ObjNum 945 snap 286 seq_num 945) 2021-07-31T01:55:29.722 INFO:tasks.rados.rados.0.smithi045.stdout:3046: done (2 left) 2021-07-31T01:55:29.723 INFO:tasks.rados.rados.0.smithi045.stdout:3048: done (1 left) 2021-07-31T01:55:29.723 INFO:tasks.rados.rados.0.smithi045.stdout:3051: rmattr oid 5 current snap is 286 2021-07-31T01:55:29.723 INFO:tasks.rados.rados.0.smithi045.stdout:3051: done (1 left) 2021-07-31T01:55:29.723 INFO:tasks.rados.rados.0.smithi045.stdout:3052: snap_remove snap 281 2021-07-31T01:55:30.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[34553]: cluster 2021-07-31T01:55:30.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[34553]: 2021- 2021-07-31T01:55:30.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[34553]: 07-31T01:55:28.653213+0000 mon.a (mon.0) 1283 : cluster [DBG] osdmap e621: 8 total, 8 up, 8 in 2021-07-31T01:55:30.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[34553]: audit 2021-07-31T 2021-07-31T01:55:30.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[34553]: 01:55:28.971063+0000 mon.b (mon.2) 104 : audit [DBG] 2021-07-31T01:55:30.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[34553]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:55:30.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[34553]: audit 2021-07-31T01:55:28. 2021-07-31T01:55:30.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[34553]: 972541+0000 mon.b (mon.2) 105 : audit [DBG] 2021-07-31T01:55:30.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[34553]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:55:30.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[34553]: audit 2021-07-31T01:55:28. 2021-07-31T01:55:30.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[34553]: 973552+0000 mon.b (mon.2) 106 : audit [INF] 2021-07-31T01:55:30.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[34553]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:55:30.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[34553]: cephadm 2021-07-31T01:55:28 2021-07-31T01:55:30.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[34553]: .974273+0000 mgr.x (mgr.24542) 348 : cephadm 2021-07-31T01:55:30.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:29 smithi154 conmon[34553]: [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T01:55:30.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:29 smithi045 conmon[33655]: cluster 2021-07-31T01:55:30.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:29 smithi045 conmon[33655]: 2021-07-31T01:55 2021-07-31T01:55:30.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:29 smithi045 conmon[33655]: :28.653213+0000 mon.a (mon.0) 1283 : cluster [DBG] osdmap e621: 8 total, 8 up, 8 in 2021-07-31T01:55:30.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:29 smithi045 conmon[33655]: audit 2021-07-31T01:55:28.971063+0000 mon.b (mon.2) 104 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:55:30.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:29 smithi045 conmon[33655]: audit 2021-07-31T01:55:28.972541+0000 mon.b (mon.2 2021-07-31T01:55:30.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:29 smithi045 conmon[33655]: ) 105 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:55:30.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:29 smithi045 conmon[33655]: audit 2021-07-31T01:55:28.973552+0000 mon.b (mon.2) 106 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:55:30.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:29 smithi045 conmon[33655]: cephadm 2021-07-31T01:55:28.974273+0000 mgr.x (mgr.24542) 348 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T01:55:30.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:29 smithi045 conmon[39718]: cluster 2021-07-31T01:55:30.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:29 smithi045 conmon[39718]: 2021-07-31T01:55:30.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:29 smithi045 conmon[39718]: 2021-07-31T01:55:28.653213+0000 2021-07-31T01:55:30.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:29 smithi045 conmon[39718]: mon.a (mon.0) 1283 : cluster [DBG] osdmap e621: 8 total, 8 up, 8 in 2021-07-31T01:55:30.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:29 smithi045 conmon[39718]: audit 2021-07-31T01 2021-07-31T01:55:30.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:29 smithi045 conmon[39718]: :55:28.971063+0000 2021-07-31T01:55:30.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:29 smithi045 conmon[39718]: mon.b (mon.2) 104 : 2021-07-31T01:55:30.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:29 smithi045 conmon[39718]: audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:55:30.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:29 smithi045 conmon[39718]: audit 2021-07 2021-07-31T01:55:30.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:29 smithi045 conmon[39718]: -31T01:55:28.972541+0000 2021-07-31T01:55:30.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:29 smithi045 conmon[39718]: mon.b (mon.2) 105 2021-07-31T01:55:30.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:29 smithi045 conmon[39718]: : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:55:30.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:29 smithi045 conmon[39718]: audit 2021 2021-07-31T01:55:30.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:29 smithi045 conmon[39718]: -07-31T01:55: 2021-07-31T01:55:30.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:29 smithi045 conmon[39718]: 28.973552+0000 mon.b (mon 2021-07-31T01:55:30.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:29 smithi045 conmon[39718]: .2) 106 : audit [INF] 2021-07-31T01:55:30.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:29 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:55:30.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:29 smithi045 conmon[39718]: cephadm 2021 2021-07-31T01:55:30.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:29 smithi045 conmon[39718]: -07-31T01:55: 2021-07-31T01:55:30.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:29 smithi045 conmon[39718]: 28.974273+0000 mgr.x (mgr 2021-07-31T01:55:30.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:29 smithi045 conmon[39718]: .24542) 348 : cephadm [WRN] 2021-07-31T01:55:30.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:29 smithi045 conmon[39718]: unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T01:55:30.678 INFO:tasks.rados.rados.0.smithi045.stdout:3052: done (1 left) 2021-07-31T01:55:30.679 INFO:tasks.rados.rados.0.smithi045.stdout:3053: setattr oid 41 current snap is 286 2021-07-31T01:55:30.679 INFO:tasks.rados.rados.0.smithi045.stdout:3050: finishing copy_from to smithi04569269-9 2021-07-31T01:55:30.679 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 9 v 1837 (ObjNum 881 snap 272 seq_num 881) dirty exists 2021-07-31T01:55:30.680 INFO:tasks.rados.rados.0.smithi045.stdout:3050: finishing copy_from racing read to smithi04569269-9 2021-07-31T01:55:30.680 INFO:tasks.rados.rados.0.smithi045.stdout:3050: done (1 left) 2021-07-31T01:55:30.680 INFO:tasks.rados.rados.0.smithi045.stdout:3054: copy_from oid 3 from oid 9 current snap is 286 2021-07-31T01:55:30.680 INFO:tasks.rados.rados.0.smithi045.stdout:3055: read oid 44 snap -1 2021-07-31T01:55:30.680 INFO:tasks.rados.rados.0.smithi045.stdout:3055: expect (ObjNum 813 snap 254 seq_num 813) 2021-07-31T01:55:30.681 INFO:tasks.rados.rados.0.smithi045.stdout:3056: read oid 49 snap 271 2021-07-31T01:55:30.681 INFO:tasks.rados.rados.0.smithi045.stdout:3056: expect (ObjNum 841 snap 265 seq_num 841) 2021-07-31T01:55:30.681 INFO:tasks.rados.rados.0.smithi045.stdout:3057: delete oid 6 current snap is 286 2021-07-31T01:55:30.682 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 41 v 1648 (ObjNum 1 snap 21873 seq_num 3190699888) dirty exists 2021-07-31T01:55:30.685 INFO:tasks.rados.rados.0.smithi045.stdout:3053: done (4 left) 2021-07-31T01:55:30.685 INFO:tasks.rados.rados.0.smithi045.stdout:3057: done (3 left) 2021-07-31T01:55:30.685 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 41 current snap is 286 2021-07-31T01:55:30.685 INFO:tasks.rados.rados.0.smithi045.stdout:3058: seq_num 948 ranges {0=679936,679936=516096,1196032=286720} 2021-07-31T01:55:30.689 INFO:tasks.rados.rados.0.smithi045.stdout:3058: writing smithi04569269-41 from 0 to 679936 tid 1 2021-07-31T01:55:30.692 INFO:tasks.rados.rados.0.smithi045.stdout:3058: writing smithi04569269-41 from 679936 to 1196032 tid 2 2021-07-31T01:55:30.694 INFO:tasks.rados.rados.0.smithi045.stdout:3058: writing smithi04569269-41 from 1196032 to 1482752 tid 3 2021-07-31T01:55:30.695 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 39 current snap is 286 2021-07-31T01:55:30.695 INFO:tasks.rados.rados.0.smithi045.stdout:3059: seq_num 949 ranges {6914048=704512,7618560=794624,8413184=8192} 2021-07-31T01:55:30.738 INFO:tasks.rados.rados.0.smithi045.stdout:3059: writing smithi04569269-39 from 6914048 to 7618560 tid 1 2021-07-31T01:55:30.743 INFO:tasks.rados.rados.0.smithi045.stdout:3059: writing smithi04569269-39 from 7618560 to 8413184 tid 2 2021-07-31T01:55:30.744 INFO:tasks.rados.rados.0.smithi045.stdout:3059: writing smithi04569269-39 from 8413184 to 8421376 tid 3 2021-07-31T01:55:30.744 INFO:tasks.rados.rados.0.smithi045.stdout:3060: read oid 40 snap -1 2021-07-31T01:55:30.745 INFO:tasks.rados.rados.0.smithi045.stdout:3060: expect (ObjNum 925 snap 280 seq_num 925) 2021-07-31T01:55:30.745 INFO:tasks.rados.rados.0.smithi045.stdout:3061: read oid 24 snap -1 2021-07-31T01:55:30.745 INFO:tasks.rados.rados.0.smithi045.stdout:3061: expect (ObjNum 0 snap 0 seq_num 0) 2021-07-31T01:55:30.745 INFO:tasks.rados.rados.0.smithi045.stdout:3062: read oid 22 snap -1 2021-07-31T01:55:30.746 INFO:tasks.rados.rados.0.smithi045.stdout:3062: expect (ObjNum 917 snap 278 seq_num 917) 2021-07-31T01:55:30.746 INFO:tasks.rados.rados.0.smithi045.stdout:3063: delete oid 28 current snap is 286 2021-07-31T01:55:30.746 INFO:tasks.rados.rados.0.smithi045.stdout:3055: expect (ObjNum 813 snap 254 seq_num 813) 2021-07-31T01:55:30.794 INFO:tasks.rados.rados.0.smithi045.stdout:3058: finishing write tid 1 to smithi04569269-41 2021-07-31T01:55:30.795 INFO:tasks.rados.rados.0.smithi045.stdout:3058: finishing write tid 2 to smithi04569269-41 2021-07-31T01:55:30.795 INFO:tasks.rados.rados.0.smithi045.stdout:3058: finishing write tid 3 to smithi04569269-41 2021-07-31T01:55:30.795 INFO:tasks.rados.rados.0.smithi045.stdout:3055: done (8 left) 2021-07-31T01:55:30.795 INFO:tasks.rados.rados.0.smithi045.stdout:3063: done (7 left) 2021-07-31T01:55:30.795 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 43 current snap is 286 2021-07-31T01:55:30.796 INFO:tasks.rados.rados.0.smithi045.stdout:3058: finishing write tid 4 to smithi04569269-41 2021-07-31T01:55:30.796 INFO:tasks.rados.rados.0.smithi045.stdout:3058: finishing write tid 5 to smithi04569269-41 2021-07-31T01:55:30.796 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 41 v 1652 (ObjNum 948 snap 286 seq_num 948) dirty exists 2021-07-31T01:55:30.797 INFO:tasks.rados.rados.0.smithi045.stdout:3058: left oid 41 (ObjNum 948 snap 286 seq_num 948) 2021-07-31T01:55:30.797 INFO:tasks.rados.rados.0.smithi045.stdout:3059: finishing write tid 1 to smithi04569269-39 2021-07-31T01:55:30.797 INFO:tasks.rados.rados.0.smithi045.stdout:3056: expect (ObjNum 841 snap 265 seq_num 841) 2021-07-31T01:55:31.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: cephadm 2021-07-31T01:55:29.017982+0000 mgr.x (mgr.24542) 349 : cephadm [ERR] 2021-07-31T01:55:31.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:55:31.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:55:31.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: yield 2021-07-31T01:55:31.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:55:31.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:55:31.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:55:31.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T01:55:31.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:55:31.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T01:55:31.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:55:31.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T01:55:31.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:55:31.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: task.result() 2021-07-31T01:55:31.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T01:55:31.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: 2021-07-31T01:55:31.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: cephadm 2021-07-31T01:55:29.018413+0000 mgr.x (mgr.24542) 350 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T01:55:31.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: cephadm 2021-07-31T01:55:29.029479+0000 mgr.x (mgr.24542) 351 : cephadm [ERR] 2021-07-31T01:55:31.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:55:31.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:55:31.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: yield 2021-07-31T01:55:31.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:55:31.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:55:31.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:55:31.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T01:55:31.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:55:31.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T01:55:31.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:55:31.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T01:55:31.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:55:31.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: task.result() 2021-07-31T01:55:31.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T01:55:31.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: cephadm 2021-07-31T01:55:29.030282+0000 mgr.x (mgr.24542) 352 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T01:55:31.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: cephadm 2021 2021-07-31T01:55:31.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: -07-31T01:55:29.056406+0000 mgr.x (mgr.24542) 353 2021-07-31T01:55:31.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: : cephadm [ERR] 2021-07-31T01:55:31.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:55:31.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:55:31.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: yield 2021-07-31T01:55:31.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:55:31.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:55:31.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:55:31.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T01:55:31.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:55:31.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T01:55:31.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:55:31.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T01:55:31.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:55:31.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: task.result() 2021-07-31T01:55:31.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T01:55:31.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: cephadm 2021-07-31T01:55:29.057014+0000 mgr.x (mgr.24542) 354 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:55:31.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:55:31.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:55:31.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: yield 2021-07-31T01:55:31.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:55:31.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:55:31.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:55:31.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T01:55:31.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:55:31.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T01:55:31.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:55:31.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T01:55:31.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:55:31.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: task.result() 2021-07-31T01:55:31.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T01:55:31.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: 2021-07-31T01:55:31.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T01:55:31.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: 2021-07-31T01:55:31.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:55:31.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:55:31.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: return f(*arg) 2021-07-31T01:55:31.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:55:31.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:55:31.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:55:31.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:55:31.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:55:31.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:55:31.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:55:31.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T01:55:31.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:55:31.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: raise self._exception 2021-07-31T01:55:31.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:55:31.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:55:31.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:55:31.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T01:55:31.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:55:31.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: raise 2021-07-31T01:55:31.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:55:31.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T01:55:31.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:55:31.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:55:31.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T01:55:31.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: cephadm 2021-07-31T01 2021-07-31T01:55:31.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: :55:29.065263+0000 mgr.x (mgr.24542) 355 : cephadm [ERR] 2021-07-31T01:55:31.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:55:31.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:55:31.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: yield 2021-07-31T01:55:31.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:55:31.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:55:31.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:55:31.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T01:55:31.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:55:31.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T01:55:31.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:55:31.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T01:55:31.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:55:31.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: task.result() 2021-07-31T01:55:31.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T01:55:31.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: cephadm 2021-07-31T01:55:29.065622+0000 mgr.x (mgr.24542) 356 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:55:31.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:55:31.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:55:31.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: yield 2021-07-31T01:55:31.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:55:31.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:55:31.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:55:31.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T01:55:31.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:55:31.035 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T01:55:31.035 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:55:31.035 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T01:55:31.035 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:55:31.036 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: task.result() 2021-07-31T01:55:31.036 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T01:55:31.036 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: 2021-07-31T01:55:31.036 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T01:55:31.036 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: 2021-07-31T01:55:31.037 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:55:31.037 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:55:31.037 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: return f(*arg) 2021-07-31T01:55:31.037 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:55:31.037 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:55:31.038 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:55:31.038 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:55:31.038 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:55:31.038 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:55:31.039 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:55:31.039 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T01:55:31.039 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:55:31.039 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: raise self._exception 2021-07-31T01:55:31.039 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:55:31.040 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:55:31.040 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:55:31.040 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T01:55:31.040 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:55:31.040 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: raise 2021-07-31T01:55:31.041 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:55:31.041 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T01:55:31.041 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:55:31.041 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:55:31.041 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T01:55:31.042 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: cluster 2021-07-31T01:55:29.655368+0000 mon.a (mon.0) 1284 : cluster [DBG] osdmap e622: 8 total, 8 up, 8 in 2021-07-31T01:55:31.042 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:30 smithi154 conmon[34553]: cluster 2021-07-31T01:55:29.909411+0000 mgr.x (mgr.24542) 357 : cluster [DBG] pgmap v717: 25 pgs: 25 active+clean; 491 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 18 MiB/s rd, 7.1 MiB/s wr, 26 op/s 2021-07-31T01:55:31.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: cephadm 2021-07-31T01:55:29.017982+0000 2021-07-31T01:55:31.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: mgr.x (mgr.24542) 349 : cephadm [ERR] 2021-07-31T01:55:31.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:55:31.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:55:31.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: yield 2021-07-31T01:55:31.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:55:31.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:55:31.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:55:31.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T01:55:31.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:55:31.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T01:55:31.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:55:31.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T01:55:31.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:55:31.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: task.result() 2021-07-31T01:55:31.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T01:55:31.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: cephadm 2021-07-31T 2021-07-31T01:55:31.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: 01:55:29.018413+0000 mgr.x (mgr.24542) 350 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T01:55:31.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: cephadm 2021-07-31T01:55:29.029479+0000 mgr.x (mgr.24542) 351 : cephadm [ERR] 2021-07-31T01:55:31.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:55:31.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:55:31.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: yield 2021-07-31T01:55:31.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:55:31.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:55:31.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:55:31.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T01:55:31.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:55:31.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T01:55:31.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:55:31.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T01:55:31.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:55:31.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: task.result() 2021-07-31T01:55:31.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T01:55:31.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: cephadm 2021-07-31T01:55:29.030282+0000 mgr.x (mgr.24542) 352 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T01:55:31.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: cephadm 2021-07-31T01:55:29.056406+0000 mgr.x (mgr.24542) 353 : cephadm [ERR] 2021-07-31T01:55:31.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: 2021-07-31T01:55:31.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:55:31.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:55:31.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: yield 2021-07-31T01:55:31.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:55:31.052 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: cephadm 2021-07-31T01:55:29. 2021-07-31T01:55:31.052 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: 017982+0000 mgr.x (mgr.24542) 349 : cephadm [ERR] 2021-07-31T01:55:31.052 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:55:31.052 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:55:31.053 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: yield 2021-07-31T01:55:31.053 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:55:31.053 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:55:31.053 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:55:31.053 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T01:55:31.054 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:55:31.054 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T01:55:31.054 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:55:31.054 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T01:55:31.055 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:55:31.055 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: task.result() 2021-07-31T01:55:31.055 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T01:55:31.055 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: cephadm 2021- 2021-07-31T01:55:31.055 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: 07-31T01:55:29.018413+0000 mgr.x (mgr.24542) 350 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T01:55:31.056 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: cephadm 2021-07-31T01:55:31.056 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: 2021-07-31 2021-07-31T01:55:31.056 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: T01:55:29.029479+0000 mgr.x 2021-07-31T01:55:31.056 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: (mgr.24542) 351 : cephadm 2021-07-31T01:55:31.056 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: [ERR] 2021-07-31T01:55:31.057 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:55:31.057 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:55:31.057 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: yield 2021-07-31T01:55:31.057 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:55:31.058 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:55:31.058 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:55:31.058 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T01:55:31.058 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:55:31.058 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T01:55:31.059 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:55:31.059 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T01:55:31.059 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:55:31.059 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: task.result() 2021-07-31T01:55:31.059 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T01:55:31.060 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: cephadm 2021-07-31T01:55:29.030282+0000 mgr.x (mgr.24542) 352 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T01:55:31.060 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: cephadm 2021- 2021-07-31T01:55:31.060 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: 07-31T01:55:29 2021-07-31T01:55:31.061 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: .056406+0000 mgr.x (mgr.24542 2021-07-31T01:55:31.061 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: ) 353 : cephadm [ERR] 2021-07-31T01:55:31.061 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:55:31.061 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:55:31.062 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:55:31.062 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:55:31.062 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T01:55:31.063 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:55:31.063 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T01:55:31.063 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:55:31.063 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T01:55:31.063 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:55:31.064 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: task.result() 2021-07-31T01:55:31.064 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T01:55:31.064 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: cephadm 2021-07-31T01:55:31.064 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: 2021-07-31T01:55:29.057014+0000 mgr.x (mgr.24542) 354 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:55:31.064 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:55:31.065 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:55:31.065 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: yield 2021-07-31T01:55:31.065 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:55:31.065 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:55:31.066 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:55:31.066 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T01:55:31.066 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:55:31.066 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T01:55:31.066 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:55:31.067 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T01:55:31.067 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:55:31.067 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: task.result() 2021-07-31T01:55:31.067 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T01:55:31.067 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: 2021-07-31T01:55:31.068 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T01:55:31.068 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: 2021-07-31T01:55:31.068 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:55:31.068 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:55:31.068 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: return f(*arg) 2021-07-31T01:55:31.069 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:55:31.069 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:55:31.069 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:55:31.069 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:55:31.070 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:55:31.070 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:55:31.070 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:55:31.070 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T01:55:31.070 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:55:31.071 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: raise self._exception 2021-07-31T01:55:31.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: yield 2021-07-31T01:55:31.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:55:31.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:55:31.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:55:31.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T01:55:31.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:55:31.073 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T01:55:31.073 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:55:31.073 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T01:55:31.073 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:55:31.074 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: task.result() 2021-07-31T01:55:31.074 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T01:55:31.074 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: 2021-07-31T01:55:31.074 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: cephadm 2021-07-31T01:55:29.057014+0000 mgr.x (mgr.24542) 2021-07-31T01:55:31.074 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: 354 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:55:31.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:55:31.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:55:31.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: yield 2021-07-31T01:55:31.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:55:31.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:55:31.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:55:31.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T01:55:31.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:55:31.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T01:55:31.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:55:31.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T01:55:31.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:55:31.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: task.result() 2021-07-31T01:55:31.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T01:55:31.078 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: 2021-07-31T01:55:31.078 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T01:55:31.078 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: 2021-07-31T01:55:31.078 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:55:31.078 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:55:31.079 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: return f(*arg) 2021-07-31T01:55:31.079 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:55:31.079 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:55:31.079 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:55:31.079 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:55:31.080 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:55:31.080 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:55:31.080 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:55:31.080 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T01:55:31.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:55:31.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: raise self._exception 2021-07-31T01:55:31.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:55:31.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:55:31.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:55:31.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T01:55:31.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:55:31.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: raise 2021-07-31T01:55:31.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:55:31.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T01:55:31.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:55:31.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:55:31.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T01:55:31.084 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: cephadm 2021-07-31T01:55:29.065263+0000 mgr.x (mgr.24542) 355 : cephadm [ERR] 2021-07-31T01:55:31.084 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:55:31.084 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:55:31.084 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: yield 2021-07-31T01:55:31.084 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:55:31.085 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:55:31.085 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:55:31.085 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T01:55:31.085 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:55:31.086 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T01:55:31.086 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:55:31.086 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T01:55:31.086 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:55:31.086 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: task.result() 2021-07-31T01:55:31.087 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T01:55:31.087 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: cephadm 2021-07-31T01:55:29. 2021-07-31T01:55:31.087 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: 065622+0000 mgr.x (mgr.24542) 356 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:55:31.087 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:55:31.087 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:55:31.088 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: yield 2021-07-31T01:55:31.088 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:55:31.088 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:55:31.088 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:55:31.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T01:55:31.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:55:31.089 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:55:31.090 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:55:31.090 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:55:31.090 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T01:55:31.090 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:55:31.091 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: raise 2021-07-31T01:55:31.091 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:55:31.091 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T01:55:31.091 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:55:31.091 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:55:31.094 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T01:55:31.094 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: cephadm 2021-07-31T01:55:29.065263+0000 mgr.x (mgr.24542) 355 : cephadm [ERR] 2021-07-31T01:55:31.094 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:55:31.095 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:55:31.095 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: yield 2021-07-31T01:55:31.095 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:55:31.095 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:55:31.096 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:55:31.096 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T01:55:31.096 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:55:31.096 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T01:55:31.096 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:55:31.097 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T01:55:31.097 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:55:31.097 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: task.result() 2021-07-31T01:55:31.097 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T01:55:31.098 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: cephadm 2021-07-31T01:55:29.065622+0000 mgr.x (mgr.24542) 356 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:55:31.098 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:55:31.098 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:55:31.098 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: yield 2021-07-31T01:55:31.098 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:55:31.099 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:55:31.099 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:55:31.099 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T01:55:31.099 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:55:31.100 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T01:55:31.100 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:55:31.100 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T01:55:31.101 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T01:55:31.101 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:55:31.101 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T01:55:31.101 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:55:31.102 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: task.result() 2021-07-31T01:55:31.102 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T01:55:31.102 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: 2021-07-31T01:55:31.102 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T01:55:31.102 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: 2021-07-31T01:55:31.103 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:55:31.103 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:55:31.103 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: return f(*arg) 2021-07-31T01:55:31.103 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:55:31.104 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:55:31.104 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:55:31.104 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:55:31.104 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:55:31.104 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:55:31.105 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:55:31.105 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T01:55:31.105 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:55:31.105 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: raise self._exception 2021-07-31T01:55:31.106 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:55:31.106 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:55:31.106 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:55:31.106 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T01:55:31.107 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:55:31.107 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: raise 2021-07-31T01:55:31.107 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:55:31.107 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T01:55:31.108 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:55:31.108 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:55:31.108 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T01:55:31.108 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: cluster 2021-07-31T01:55:29.655368+0000 mon.a (mon.0) 1284 : cluster [DBG] osdmap e622: 8 total, 8 up, 8 in 2021-07-31T01:55:31.109 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[33655]: cluster 2021-07-31T01:55:29.909411+0000 mgr.x (mgr.24542) 357 : cluster [DBG] pgmap v717: 25 pgs: 25 active+clean; 491 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 18 MiB/s rd, 7.1 MiB/s wr, 26 op/s 2021-07-31T01:55:31.109 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:55:31.109 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: task.result() 2021-07-31T01:55:31.109 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T01:55:31.110 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: 2021-07-31T01:55:31.110 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T01:55:31.110 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: 2021-07-31T01:55:31.110 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:55:31.111 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:55:31.111 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: return f(*arg) 2021-07-31T01:55:31.111 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:55:31.111 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:55:31.111 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:55:31.112 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:55:31.112 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:55:31.112 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:55:31.112 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:55:31.112 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T01:55:31.113 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:55:31.113 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: raise self._exception 2021-07-31T01:55:31.113 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:55:31.113 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:55:31.113 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:55:31.114 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T01:55:31.114 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:55:31.114 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: raise 2021-07-31T01:55:31.114 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:55:31.115 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T01:55:31.115 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:55:31.115 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:55:31.115 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T01:55:31.115 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: cluster 2021-07-31T01:55:29.655368+0000 mon.a (mon.0) 1284 : cluster [DBG] osdmap e622: 8 total, 8 up, 8 in 2021-07-31T01:55:31.116 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:30 smithi045 conmon[39718]: cluster 2021-07-31T01:55:29.909411+0000 mgr.x (mgr.24542) 357 : cluster [DBG] pgmap v717: 25 pgs: 25 active+clean; 491 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 18 MiB/s rd, 7.1 MiB/s wr, 26 op/s 2021-07-31T01:55:32.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:31 smithi154 conmon[34553]: cluster 2021-07-31T01:55:30.667862+0000 mon.a (mon.0) 1285 : cluster [DBG] osdmap e623: 8 total, 8 up, 8 in 2021-07-31T01:55:32.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:31 smithi154 conmon[34553]: cluster 2021-07-31T01:55:31.631927+0000 mon.a (mon.0) 1286 : cluster [DBG] osdmap e624: 8 total, 8 up, 8 in 2021-07-31T01:55:32.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:31 smithi045 conmon[33655]: cluster 2021-07-31T01:55:30.667862+0000 mon.a (mon.0) 1285 : cluster 2021-07-31T01:55:32.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:31 smithi045 conmon[33655]: [DBG] osdmap e623: 8 total, 8 up, 8 in 2021-07-31T01:55:32.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:31 smithi045 conmon[33655]: cluster 2021-07-31T01:55:31.631927+0000 mon.a (mon 2021-07-31T01:55:32.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:31 smithi045 conmon[33655]: .0) 1286 : cluster [DBG] osdmap e624: 8 total, 8 up, 8 in 2021-07-31T01:55:32.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:31 smithi045 conmon[39718]: cluster 2021-07-31T01:55:30. 2021-07-31T01:55:32.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:31 smithi045 conmon[39718]: 667862+0000 mon.a (mon.0) 1285 : cluster [DBG] osdmap e623: 8 total, 8 up, 8 in 2021-07-31T01:55:32.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:31 smithi045 conmon[39718]: cluster 2021-07-31T01:55: 2021-07-31T01:55:32.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:31 smithi045 conmon[39718]: 31.631927+0000 mon.a (mon.0) 2021-07-31T01:55:32.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:31 smithi045 conmon[39718]: 1286 : cluster [DBG] osdmap e624: 8 total, 8 up, 8 in 2021-07-31T01:55:32.159 INFO:tasks.rados.rados.0.smithi045.stdout:3054: finishing copy_from to smithi04569269-3 2021-07-31T01:55:32.159 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 3 v 1837 (ObjNum 881 snap 272 seq_num 881) dirty exists 2021-07-31T01:55:32.160 INFO:tasks.rados.rados.0.smithi045.stdout:3054: finishing copy_from racing read to smithi04569269-3 2021-07-31T01:55:32.160 INFO:tasks.rados.rados.0.smithi045.stdout:3061: expect (ObjNum 0 snap 0 seq_num 0) 2021-07-31T01:55:32.161 INFO:tasks.rados.rados.0.smithi045.stdout:3059: finishing write tid 2 to smithi04569269-39 2021-07-31T01:55:32.162 INFO:tasks.rados.rados.0.smithi045.stdout:3062: expect (ObjNum 917 snap 278 seq_num 917) 2021-07-31T01:55:32.186 INFO:tasks.rados.rados.0.smithi045.stdout:3059: finishing write tid 3 to smithi04569269-39 2021-07-31T01:55:32.186 INFO:tasks.rados.rados.0.smithi045.stdout:3059: finishing write tid 4 to smithi04569269-39 2021-07-31T01:55:32.187 INFO:tasks.rados.rados.0.smithi045.stdout:3059: finishing write tid 5 to smithi04569269-39 2021-07-31T01:55:32.187 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 39 v 1681 (ObjNum 949 snap 286 seq_num 949) dirty exists 2021-07-31T01:55:32.187 INFO:tasks.rados.rados.0.smithi045.stdout:3059: left oid 39 (ObjNum 949 snap 286 seq_num 949) 2021-07-31T01:55:32.188 INFO:tasks.rados.rados.0.smithi045.stdout:3060: expect (ObjNum 925 snap 280 seq_num 925) 2021-07-31T01:55:32.214 INFO:tasks.rados.rados.0.smithi045.stdout:3064: seq_num 951 ranges {9699328=737280,10436608=614400,11051008=188416} 2021-07-31T01:55:32.271 INFO:tasks.rados.rados.0.smithi045.stdout:3064: writing smithi04569269-43 from 9699328 to 10436608 tid 1 2021-07-31T01:55:32.274 INFO:tasks.rados.rados.0.smithi045.stdout:3064: writing smithi04569269-43 from 10436608 to 11051008 tid 2 2021-07-31T01:55:32.276 INFO:tasks.rados.rados.0.smithi045.stdout:3064: writing smithi04569269-43 from 11051008 to 11239424 tid 3 2021-07-31T01:55:32.277 INFO:tasks.rados.rados.0.smithi045.stdout:3054: done (7 left) 2021-07-31T01:55:32.277 INFO:tasks.rados.rados.0.smithi045.stdout:3056: done (6 left) 2021-07-31T01:55:32.277 INFO:tasks.rados.rados.0.smithi045.stdout:3058: done (5 left) 2021-07-31T01:55:32.278 INFO:tasks.rados.rados.0.smithi045.stdout:3059: done (4 left) 2021-07-31T01:55:32.278 INFO:tasks.rados.rados.0.smithi045.stdout:3060: done (3 left) 2021-07-31T01:55:32.278 INFO:tasks.rados.rados.0.smithi045.stdout:3061: done (2 left) 2021-07-31T01:55:32.278 INFO:tasks.rados.rados.0.smithi045.stdout:3062: done (1 left) 2021-07-31T01:55:32.279 INFO:tasks.rados.rados.0.smithi045.stdout:3065: snap_create 2021-07-31T01:55:32.280 INFO:tasks.rados.rados.0.smithi045.stdout:3064: finishing write tid 1 to smithi04569269-43 2021-07-31T01:55:32.280 INFO:tasks.rados.rados.0.smithi045.stdout:3064: finishing write tid 2 to smithi04569269-43 2021-07-31T01:55:32.281 INFO:tasks.rados.rados.0.smithi045.stdout:3064: finishing write tid 3 to smithi04569269-43 2021-07-31T01:55:32.281 INFO:tasks.rados.rados.0.smithi045.stdout:3064: finishing write tid 4 to smithi04569269-43 2021-07-31T01:55:32.283 INFO:tasks.rados.rados.0.smithi045.stdout:3064: finishing write tid 5 to smithi04569269-43 2021-07-31T01:55:32.283 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 43 v 1839 (ObjNum 951 snap 286 seq_num 951) dirty exists 2021-07-31T01:55:32.283 INFO:tasks.rados.rados.0.smithi045.stdout:3064: left oid 43 (ObjNum 951 snap 286 seq_num 951) 2021-07-31T01:55:32.673 INFO:tasks.rados.rados.0.smithi045.stdout:3064: done (1 left) 2021-07-31T01:55:32.673 INFO:tasks.rados.rados.0.smithi045.stdout:3065: done (0 left) 2021-07-31T01:55:32.673 INFO:tasks.rados.rados.0.smithi045.stdout:3066: rmattr oid 21 current snap is 287 2021-07-31T01:55:32.674 INFO:tasks.rados.rados.0.smithi045.stdout:3067: snap_remove snap 268 2021-07-31T01:55:33.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:32 smithi154 conmon[34553]: cluster 2021-07-31T01:55:31.909964+0000 mgr.x (mgr.24542) 358 : cluster 2021-07-31T01:55:33.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:32 smithi154 conmon[34553]: [DBG] pgmap v720: 25 pgs: 25 active+clean; 491 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 17 MiB/s rd, 6.9 MiB/s wr, 25 op/s 2021-07-31T01:55:33.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:32 smithi045 conmon[39718]: cluster 2021-07-31T01:55:31.909964+0000 mgr.x (mgr 2021-07-31T01:55:33.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:32 smithi045 conmon[39718]: .24542) 358 : cluster [DBG] pgmap v720: 25 pgs: 25 active+clean; 491 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 17 MiB/s rd, 6.9 MiB/s wr, 25 op/s 2021-07-31T01:55:33.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:32 smithi045 conmon[33655]: cluster 2021-07-31T01:55:31.909964+0000 mgr.x (mgr.24542) 358 : cluster [DBG] 2021-07-31T01:55:33.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:32 smithi045 conmon[33655]: pgmap v720: 25 pgs: 25 active+clean; 491 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 17 MiB/s rd, 6.9 MiB/s wr, 25 op/s 2021-07-31T01:55:33.681 INFO:tasks.rados.rados.0.smithi045.stdout:3067: done (1 left) 2021-07-31T01:55:33.681 INFO:tasks.rados.rados.0.smithi045.stdout:3068: rmattr oid 15 current snap is 287 2021-07-31T01:55:33.681 INFO:tasks.rados.rados.0.smithi045.stdout:3068: done (1 left) 2021-07-31T01:55:33.682 INFO:tasks.rados.rados.0.smithi045.stdout:3069: setattr oid 4 current snap is 287 2021-07-31T01:55:33.682 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 21 v 1949 (ObjNum 866 snap 270 seq_num 866) dirty exists 2021-07-31T01:55:33.683 INFO:tasks.rados.rados.0.smithi045.stdout:3066: done (1 left) 2021-07-31T01:55:33.683 INFO:tasks.rados.rados.0.smithi045.stdout:3070: read oid 11 snap 274 2021-07-31T01:55:33.683 INFO:tasks.rados.rados.0.smithi045.stdout:3070: expect deleted 2021-07-31T01:55:33.684 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 36 current snap is 287 2021-07-31T01:55:33.684 INFO:tasks.rados.rados.0.smithi045.stdout:3071: seq_num 952 ranges {14180352=753664,14934016=499712,15433728=303104} 2021-07-31T01:55:33.767 INFO:tasks.rados.rados.0.smithi045.stdout:3071: writing smithi04569269-36 from 14180352 to 14934016 tid 1 2021-07-31T01:55:33.769 INFO:tasks.rados.rados.0.smithi045.stdout:3071: writing smithi04569269-36 from 14934016 to 15433728 tid 2 2021-07-31T01:55:33.772 INFO:tasks.rados.rados.0.smithi045.stdout:3071: writing smithi04569269-36 from 15433728 to 15736832 tid 3 2021-07-31T01:55:33.772 INFO:tasks.rados.rados.0.smithi045.stdout:3072: snap_create 2021-07-31T01:55:33.772 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 4 v 1672 (ObjNum 716 snap 222 seq_num 716) dirty exists 2021-07-31T01:55:33.772 INFO:tasks.rados.rados.0.smithi045.stdout:3071: finishing write tid 1 to smithi04569269-36 2021-07-31T01:55:33.775 INFO:tasks.rados.rados.0.smithi045.stdout:3071: finishing write tid 2 to smithi04569269-36 2021-07-31T01:55:33.775 INFO:tasks.rados.rados.0.smithi045.stdout:3071: finishing write tid 3 to smithi04569269-36 2021-07-31T01:55:33.776 INFO:tasks.rados.rados.0.smithi045.stdout:3071: finishing write tid 4 to smithi04569269-36 2021-07-31T01:55:33.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:33 smithi045 conmon[39718]: cluster 2021-07-31T01:55:32.672282+0000 mon.a (mon 2021-07-31T01:55:33.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:33 smithi045 conmon[39718]: .0) 1287 : cluster [DBG] osdmap e625: 8 total, 8 up, 8 in 2021-07-31T01:55:33.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:55:33 smithi045 conmon[61210]: level=error ts=2021-07-31T01:55:33.523Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:55:33.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:55:33 smithi045 conmon[61210]: level=error ts=2021-07-31T01:55:33.523Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:55:33.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:55:33 smithi045 conmon[61210]: level=error ts=2021-07-31T01:55:33.523Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T01:55:33.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:33 smithi045 conmon[33655]: cluster 2021-07-31T01:55:32 2021-07-31T01:55:33.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:33 smithi045 conmon[33655]: .672282+0000 mon.a (mon.0) 1287 : cluster [DBG] osdmap e625: 8 total, 8 up, 8 in 2021-07-31T01:55:33.779 INFO:tasks.rados.rados.0.smithi045.stdout:3071: finishing write tid 5 to smithi04569269-36 2021-07-31T01:55:33.779 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 36 v 1968 (ObjNum 952 snap 287 seq_num 952) dirty exists 2021-07-31T01:55:33.779 INFO:tasks.rados.rados.0.smithi045.stdout:3071: left oid 36 (ObjNum 952 snap 287 seq_num 952) 2021-07-31T01:55:34.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:33 smithi154 conmon[34553]: cluster 2021-07-31T01:55:32.672282+0000 mon.a (mon.0) 1287 : cluster [DBG] osdmap e625: 8 total, 8 up, 8 in 2021-07-31T01:55:34.685 INFO:tasks.rados.rados.0.smithi045.stdout:3069: done (3 left) 2021-07-31T01:55:34.685 INFO:tasks.rados.rados.0.smithi045.stdout:3070: done (2 left) 2021-07-31T01:55:34.686 INFO:tasks.rados.rados.0.smithi045.stdout:3071: done (1 left) 2021-07-31T01:55:34.686 INFO:tasks.rados.rados.0.smithi045.stdout:3072: done (0 left) 2021-07-31T01:55:34.686 INFO:tasks.rados.rados.0.smithi045.stdout:3073: snap_remove snap 287 2021-07-31T01:55:35.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:34 smithi154 conmon[34553]: cluster 2021-07-31T01:55:33.678672+0000 mon.a (mon.0) 1288 : 2021-07-31T01:55:35.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:34 smithi154 conmon[34553]: cluster [DBG] osdmap e626: 8 total, 8 up, 8 in 2021-07-31T01:55:35.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:34 smithi154 conmon[34553]: cluster 2021-07-31T01:55:33.910426+0000 mgr.x (mgr.24542) 359 : cluster [DBG] pgmap v723: 25 pgs: 25 active+clean; 502 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 13 MiB/s rd, 6.7 MiB/s wr, 25 op/s 2021-07-31T01:55:35.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:34 smithi154 conmon[34553]: 2021-07-31T01:55:35.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:34 smithi045 conmon[33655]: cluster 2021-07-31T01:55:33.678672+0000 mon.a (mon.0) 2021-07-31T01:55:35.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:34 smithi045 conmon[33655]: 1288 : cluster [DBG] osdmap e626: 8 total, 8 up, 8 in 2021-07-31T01:55:35.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:34 smithi045 conmon[33655]: cluster 2021-07-31T01:55:33.910426+0000 mgr.x (mgr.24542) 359 : cluster [DBG] pgmap v723: 25 pgs: 25 active+clean; 502 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 13 MiB/s rd, 6.7 MiB/s wr, 25 op/s 2021-07-31T01:55:35.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:34 smithi045 conmon[39718]: cluster 2021-07-31T01:55:33.678672+0000 mon.a ( 2021-07-31T01:55:35.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:34 smithi045 conmon[39718]: mon.0) 1288 : cluster [DBG] osdmap e626: 8 total, 8 up, 8 in 2021-07-31T01:55:35.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:34 smithi045 conmon[39718]: cluster 2021-07-31T01:55:33.910426+0000 mgr.x (mgr 2021-07-31T01:55:35.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:34 smithi045 conmon[39718]: .24542) 359 : cluster [DBG] pgmap v723: 25 pgs: 25 active+clean; 502 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 13 MiB/s rd, 6.7 MiB/s wr, 25 op/s 2021-07-31T01:55:35.690 INFO:tasks.rados.rados.0.smithi045.stdout:3073: done (0 left) 2021-07-31T01:55:35.691 INFO:tasks.rados.rados.0.smithi045.stdout:3074: snap_create 2021-07-31T01:55:36.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:35 smithi154 conmon[34553]: cluster 2021-07-31T01:55:34.684550+0000 mon.a (mon.0) 1289 : cluster [DBG] osdmap e627: 8 total, 8 up, 8 in 2021-07-31T01:55:36.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:35 smithi045 conmon[39718]: cluster 2021-07-31T01:55:34.684550+0000 mon.a ( 2021-07-31T01:55:36.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:35 smithi045 conmon[39718]: mon.0) 1289 : cluster [DBG] osdmap e627: 8 total, 8 up, 8 in 2021-07-31T01:55:36.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:35 smithi045 conmon[33655]: cluster 2021-07-31T01:55:34.684550+0000 mon.a (mon. 2021-07-31T01:55:36.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:35 smithi045 conmon[33655]: 0) 1289 : cluster [DBG] osdmap e627: 8 total, 8 up, 8 in 2021-07-31T01:55:36.635 INFO:tasks.rados.rados.0.smithi045.stdout:3074: done (0 left) 2021-07-31T01:55:36.635 INFO:tasks.rados.rados.0.smithi045.stdout:3075: read oid 11 snap -1 2021-07-31T01:55:36.635 INFO:tasks.rados.rados.0.smithi045.stdout:3075: expect (ObjNum 920 snap 278 seq_num 920) 2021-07-31T01:55:36.636 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 28 current snap is 289 2021-07-31T01:55:36.636 INFO:tasks.rados.rados.0.smithi045.stdout:3076: seq_num 953 ranges {0=770048,770048=786432,1556480=16384} 2021-07-31T01:55:36.639 INFO:tasks.rados.rados.0.smithi045.stdout:3076: writing smithi04569269-28 from 0 to 770048 tid 1 2021-07-31T01:55:36.644 INFO:tasks.rados.rados.0.smithi045.stdout:3076: writing smithi04569269-28 from 770048 to 1556480 tid 2 2021-07-31T01:55:36.644 INFO:tasks.rados.rados.0.smithi045.stdout:3076: writing smithi04569269-28 from 1556480 to 1572864 tid 3 2021-07-31T01:55:36.644 INFO:tasks.rados.rados.0.smithi045.stdout:3077: delete oid 46 current snap is 289 2021-07-31T01:55:36.645 INFO:tasks.rados.rados.0.smithi045.stdout:3075: expect (ObjNum 920 snap 278 seq_num 920) 2021-07-31T01:55:36.672 INFO:tasks.rados.rados.0.smithi045.stdout:3076: finishing write tid 1 to smithi04569269-28 2021-07-31T01:55:36.673 INFO:tasks.rados.rados.0.smithi045.stdout:3076: finishing write tid 2 to smithi04569269-28 2021-07-31T01:55:36.673 INFO:tasks.rados.rados.0.smithi045.stdout:3076: finishing write tid 3 to smithi04569269-28 2021-07-31T01:55:36.673 INFO:tasks.rados.rados.0.smithi045.stdout:3076: finishing write tid 4 to smithi04569269-28 2021-07-31T01:55:36.673 INFO:tasks.rados.rados.0.smithi045.stdout:3076: finishing write tid 5 to smithi04569269-28 2021-07-31T01:55:36.674 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 28 v 1690 (ObjNum 953 snap 289 seq_num 953) dirty exists 2021-07-31T01:55:36.674 INFO:tasks.rados.rados.0.smithi045.stdout:3076: left oid 28 (ObjNum 953 snap 289 seq_num 953) 2021-07-31T01:55:36.674 INFO:tasks.rados.rados.0.smithi045.stdout:3075: done (2 left) 2021-07-31T01:55:36.674 INFO:tasks.rados.rados.0.smithi045.stdout:3076: done (1 left) 2021-07-31T01:55:36.674 INFO:tasks.rados.rados.0.smithi045.stdout:3077: done (0 left) 2021-07-31T01:55:36.675 INFO:tasks.rados.rados.0.smithi045.stdout:3078: read oid 9 snap -1 2021-07-31T01:55:36.675 INFO:tasks.rados.rados.0.smithi045.stdout:3078: expect (ObjNum 881 snap 272 seq_num 881) 2021-07-31T01:55:36.675 INFO:tasks.rados.rados.0.smithi045.stdout:3079: read oid 43 snap -1 2021-07-31T01:55:36.675 INFO:tasks.rados.rados.0.smithi045.stdout:3079: expect (ObjNum 951 snap 286 seq_num 951) 2021-07-31T01:55:36.676 INFO:tasks.rados.rados.0.smithi045.stdout:3080: copy_from oid 2 from oid 1 current snap is 289 2021-07-31T01:55:36.676 INFO:tasks.rados.rados.0.smithi045.stdout:3081: copy_from oid 25 from oid 49 current snap is 289 2021-07-31T01:55:36.676 INFO:tasks.rados.rados.0.smithi045.stdout:3082: read oid 11 snap -1 2021-07-31T01:55:36.676 INFO:tasks.rados.rados.0.smithi045.stdout:3082: expect (ObjNum 920 snap 278 seq_num 920) 2021-07-31T01:55:36.676 INFO:tasks.rados.rados.0.smithi045.stdout:3083: snap_remove snap 288 2021-07-31T01:55:37.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:36 smithi154 conmon[34553]: cluster 2021-07-31T01:55:35.689898+0000 mon.a (mon.0) 1290 2021-07-31T01:55:37.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:36 smithi154 conmon[34553]: : cluster [DBG] osdmap e628: 8 total, 8 up, 8 in 2021-07-31T01:55:37.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:36 smithi154 conmon[34553]: cluster 2021-07-31T01:55:35.910894+0000 mgr.x (mgr.24542) 360 : cluster [DBG] pgmap v726: 25 pgs: 25 active+clean; 502 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 13 MiB/s rd, 6.7 MiB/s wr, 25 op/s 2021-07-31T01:55:37.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:36 smithi154 conmon[34553]: cluster 2021-07-31T01:55:36.634337+0000 mon.a ( 2021-07-31T01:55:37.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:36 smithi154 conmon[34553]: mon.0) 1291 : cluster [DBG] osdmap e629: 8 total, 8 up, 8 in 2021-07-31T01:55:37.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:36 smithi045 conmon[39718]: cluster 2021-07-31T01:55:35.689898 2021-07-31T01:55:37.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:36 smithi045 conmon[39718]: +0000 mon.a (mon.0) 1290 : cluster [DBG] osdmap e628: 8 total, 8 up, 8 in 2021-07-31T01:55:37.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:36 smithi045 conmon[39718]: cluster 2021-07-31T01:55:35. 2021-07-31T01:55:37.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:36 smithi045 conmon[39718]: 910894+0000 mgr.x (mgr.24542) 360 : cluster [DBG] pgmap v726: 25 pgs: 25 active+clean; 502 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 13 MiB/s rd, 6.7 MiB/s wr, 25 op/s 2021-07-31T01:55:37.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:36 smithi045 conmon[39718]: cluster 2021 2021-07-31T01:55:37.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:36 smithi045 conmon[39718]: -07-31T01:55:36.634337+0000 2021-07-31T01:55:37.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:36 smithi045 conmon[39718]: mon.a (mon.0) 1291 : cluster [DBG] 2021-07-31T01:55:37.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:36 smithi045 conmon[39718]: osdmap e629: 8 total, 8 up, 8 in 2021-07-31T01:55:37.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:36 smithi045 conmon[33655]: cluster 2021-07-31T01:55:35. 2021-07-31T01:55:37.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:36 smithi045 conmon[33655]: 689898+0000 mon.a (mon.0) 1290 : cluster [DBG] osdmap e628: 8 total, 8 up, 8 in 2021-07-31T01:55:37.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:36 smithi045 conmon[33655]: cluster 2021-07-31T01:55:35.910894+0000 mgr.x (mgr 2021-07-31T01:55:37.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:36 smithi045 conmon[33655]: .24542) 360 : cluster [DBG] pgmap v726: 25 pgs: 25 active+clean; 502 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 13 MiB/s rd, 6.7 MiB/s wr, 25 op/s 2021-07-31T01:55:37.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:36 smithi045 conmon[33655]: cluster 2021-07-31T 2021-07-31T01:55:37.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:36 smithi045 conmon[33655]: 01:55:36.634337+0000 mon.a (mon.0) 1291 : cluster [DBG] 2021-07-31T01:55:37.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:36 smithi045 conmon[33655]: osdmap e629: 8 total, 8 up, 8 in 2021-07-31T01:55:37.641 INFO:tasks.rados.rados.0.smithi045.stdout:3083: done (5 left) 2021-07-31T01:55:37.642 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 44 current snap is 289 2021-07-31T01:55:37.643 INFO:tasks.rados.rados.0.smithi045.stdout:3084: seq_num 955 ranges {1785856=802816,2588672=557056,3145728=245760} 2021-07-31T01:55:37.656 INFO:tasks.rados.rados.0.smithi045.stdout:3084: writing smithi04569269-44 from 1785856 to 2588672 tid 1 2021-07-31T01:55:37.659 INFO:tasks.rados.rados.0.smithi045.stdout:3084: writing smithi04569269-44 from 2588672 to 3145728 tid 2 2021-07-31T01:55:37.661 INFO:tasks.rados.rados.0.smithi045.stdout:3084: writing smithi04569269-44 from 3145728 to 3391488 tid 3 2021-07-31T01:55:37.661 INFO:tasks.rados.rados.0.smithi045.stdout:3085: read oid 46 snap 286 2021-07-31T01:55:37.661 INFO:tasks.rados.rados.0.smithi045.stdout:3085: expect (ObjNum 826 snap 258 seq_num 826) 2021-07-31T01:55:37.661 INFO:tasks.rados.rados.0.smithi045.stdout:3086: snap_create 2021-07-31T01:55:37.662 INFO:tasks.rados.rados.0.smithi045.stdout:3082: expect (ObjNum 920 snap 278 seq_num 920) 2021-07-31T01:55:37.687 INFO:tasks.rados.rados.0.smithi045.stdout:3079: expect (ObjNum 951 snap 286 seq_num 951) 2021-07-31T01:55:38.914 INFO:tasks.rados.rados.0.smithi045.stdout:3078: expect (ObjNum 881 snap 272 seq_num 881) 2021-07-31T01:55:39.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:38 smithi154 conmon[34553]: cluster 2021-07-31T01:55:37.641135+0000 mon.a (mon.0) 1292 : cluster 2021-07-31T01:55:39.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:38 smithi154 conmon[34553]: [DBG] osdmap e630: 8 total, 8 up, 8 in 2021-07-31T01:55:39.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:38 smithi154 conmon[34553]: cluster 2021-07-31T01:55:37.911351+0000 mgr.x (mgr.24542) 361 : cluster [DBG] pgmap v729: 25 pgs: 25 active+clean; 502 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:55:39.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:38 smithi045 conmon[33655]: cluster 2021-07-31T01:55:37 2021-07-31T01:55:39.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:38 smithi045 conmon[33655]: .641135+0000 mon.a (mon.0) 1292 : cluster [DBG] osdmap e630: 8 total, 8 up, 8 in 2021-07-31T01:55:39.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:38 smithi045 conmon[33655]: cluster 2021-07-31T01:55:37.911351+0000 mgr.x (mgr.24542) 361 : cluster [DBG] pgmap v729: 25 pgs: 25 active+clean; 502 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:55:39.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:38 smithi045 conmon[39718]: cluster 2021-07-31T01 2021-07-31T01:55:39.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:38 smithi045 conmon[39718]: :55:37.641135+0000 mon.a (mon.0) 1292 : cluster [DBG] osdmap e630: 8 total, 8 up, 8 in 2021-07-31T01:55:39.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:38 smithi045 conmon[39718]: cluster 2021-07-31T01:55:39.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:38 smithi045 conmon[39718]: 2021-07-31T01:55: 2021-07-31T01:55:39.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:38 smithi045 conmon[39718]: 37.911351+0000 mgr.x (mgr.24542) 2021-07-31T01:55:39.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:38 smithi045 conmon[39718]: 361 : cluster [DBG] pgmap v729: 25 pgs: 25 active+clean; 502 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:55:39.936 INFO:tasks.rados.rados.0.smithi045.stdout:3080: finishing copy_from to smithi04569269-2 2021-07-31T01:55:39.937 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 2 v 1587 (ObjNum 904 snap 274 seq_num 904) dirty exists 2021-07-31T01:55:39.937 INFO:tasks.rados.rados.0.smithi045.stdout:3080: finishing copy_from racing read to smithi04569269-2 2021-07-31T01:55:39.937 INFO:tasks.rados.rados.0.smithi045.stdout:3081: finishing copy_from to smithi04569269-25 2021-07-31T01:55:39.937 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 25 v 1836 (ObjNum 890 snap 272 seq_num 890) dirty exists 2021-07-31T01:55:39.938 INFO:tasks.rados.rados.0.smithi045.stdout:3081: finishing copy_from racing read to smithi04569269-25 2021-07-31T01:55:39.938 INFO:tasks.rados.rados.0.smithi045.stdout:3084: finishing write tid 1 to smithi04569269-44 2021-07-31T01:55:39.938 INFO:tasks.rados.rados.0.smithi045.stdout:3084: finishing write tid 2 to smithi04569269-44 2021-07-31T01:55:39.938 INFO:tasks.rados.rados.0.smithi045.stdout:3084: finishing write tid 3 to smithi04569269-44 2021-07-31T01:55:39.939 INFO:tasks.rados.rados.0.smithi045.stdout:3084: finishing write tid 4 to smithi04569269-44 2021-07-31T01:55:39.939 INFO:tasks.rados.rados.0.smithi045.stdout:3084: finishing write tid 5 to smithi04569269-44 2021-07-31T01:55:39.939 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 44 v 1843 (ObjNum 955 snap 289 seq_num 955) dirty exists 2021-07-31T01:55:39.939 INFO:tasks.rados.rados.0.smithi045.stdout:3084: left oid 44 (ObjNum 955 snap 289 seq_num 955) 2021-07-31T01:55:39.940 INFO:tasks.rados.rados.0.smithi045.stdout:3085: expect (ObjNum 826 snap 258 seq_num 826) 2021-07-31T01:55:40.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:39 smithi154 conmon[34553]: cluster 2021-07-31T01:55:38.645526+0000 mon.a (mon.0) 2021-07-31T01:55:40.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:39 smithi154 conmon[34553]: 1293 : cluster [DBG] osdmap e631: 8 total, 8 up, 8 in 2021-07-31T01:55:40.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:39 smithi045 conmon[39718]: cluster 2021-07-31T01:55:38.645526 2021-07-31T01:55:40.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:39 smithi045 conmon[39718]: +0000 mon.a (mon.0) 1293 : cluster [DBG] osdmap e631: 8 total, 8 up, 8 in 2021-07-31T01:55:40.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:39 smithi045 conmon[33655]: cluster 2021-07-31T01:55:38.645526+0000 mon.a (mon 2021-07-31T01:55:40.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:39 smithi045 conmon[33655]: .0) 1293 : cluster [DBG] osdmap e631: 8 total, 8 up, 8 in 2021-07-31T01:55:40.152 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T01:55:40.291 INFO:tasks.rados.rados.0.smithi045.stdout:3078: done (7 left) 2021-07-31T01:55:40.291 INFO:tasks.rados.rados.0.smithi045.stdout:3079: done (6 left) 2021-07-31T01:55:40.291 INFO:tasks.rados.rados.0.smithi045.stdout:3080: done (5 left) 2021-07-31T01:55:40.292 INFO:tasks.rados.rados.0.smithi045.stdout:3081: done (4 left) 2021-07-31T01:55:40.292 INFO:tasks.rados.rados.0.smithi045.stdout:3082: done (3 left) 2021-07-31T01:55:40.292 INFO:tasks.rados.rados.0.smithi045.stdout:3084: done (2 left) 2021-07-31T01:55:40.292 INFO:tasks.rados.rados.0.smithi045.stdout:3085: done (1 left) 2021-07-31T01:55:40.292 INFO:tasks.rados.rados.0.smithi045.stdout:3086: done (0 left) 2021-07-31T01:55:40.293 INFO:tasks.rados.rados.0.smithi045.stdout:3087: snap_create 2021-07-31T01:55:40.446 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T01:55:40.447 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 10m ago 11m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T01:55:40.447 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 8m ago 11m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T01:55:40.447 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 8m ago 14m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T01:55:40.448 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 10m ago 16m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T01:55:40.448 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 10m ago 16m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T01:55:40.448 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 8m ago 14m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T01:55:40.448 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 10m ago 15m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T01:55:40.448 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 10m ago 11m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T01:55:40.449 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 8m ago 11m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T01:55:40.449 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 10m ago 14m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T01:55:40.449 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 10m ago 13m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T01:55:40.449 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 10m ago 13m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T01:55:40.450 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 10m ago 13m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T01:55:40.450 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 8m ago 12m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T01:55:40.450 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 8m ago 12m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T01:55:40.450 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 8m ago 12m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T01:55:40.451 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 8m ago 11m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T01:55:40.451 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 8m ago 11m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T01:55:40.652 INFO:tasks.rados.rados.0.smithi045.stdout:3087: done (0 left) 2021-07-31T01:55:40.653 INFO:tasks.rados.rados.0.smithi045.stdout:3088: snap_create 2021-07-31T01:55:40.783 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T01:55:40.783 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T01:55:40.784 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T01:55:40.784 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T01:55:40.785 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T01:55:40.785 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T01:55:40.785 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T01:55:40.786 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T01:55:40.787 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T01:55:40.787 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T01:55:40.788 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T01:55:40.788 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T01:55:40.789 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T01:55:40.789 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T01:55:40.790 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T01:55:40.790 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T01:55:40.791 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T01:55:41.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:40 smithi154 conmon[34553]: cluster 2021-07-31T01:55:39.912067+0000 mgr.x (mgr.24542) 362 : cluster [DBG] 2021-07-31T01:55:41.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:40 smithi154 conmon[34553]: pgmap v731: 25 pgs: 25 active+clean; 526 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 20 MiB/s rd, 5.6 MiB/s wr, 8 op/s 2021-07-31T01:55:41.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:40 smithi045 conmon[33655]: cluster 2021-07-31T01:55:39.912067+0000 mgr.x (mgr. 2021-07-31T01:55:41.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:40 smithi045 conmon[33655]: 24542) 362 : cluster [DBG] pgmap v731: 25 pgs: 25 active+clean; 526 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 20 MiB/s rd, 5.6 MiB/s wr, 8 op/s 2021-07-31T01:55:41.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:40 smithi045 conmon[39718]: cluster 2021-07-31T01:55:39.912067 2021-07-31T01:55:41.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:40 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 362 : cluster [DBG] pgmap v731: 25 pgs: 25 active+clean; 526 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 20 MiB/s rd, 5.6 MiB/s wr, 8 op/s 2021-07-31T01:55:41.636 INFO:tasks.rados.rados.0.smithi045.stdout:3088: done (0 left) 2021-07-31T01:55:41.637 INFO:tasks.rados.rados.0.smithi045.stdout:3089: copy_from oid 38 from oid 6 current snap is 292 2021-07-31T01:55:41.637 INFO:tasks.rados.rados.0.smithi045.stdout:3090: setattr oid 26 current snap is 292 2021-07-31T01:55:41.638 INFO:tasks.rados.rados.0.smithi045.stdout:3091: snap_create 2021-07-31T01:55:41.642 INFO:tasks.rados.rados.0.smithi045.stdout:3089: finishing copy_from to smithi04569269-38 2021-07-31T01:55:41.642 INFO:tasks.rados.rados.0.smithi045.stdout:3089: got expected ENOENT (src dne) 2021-07-31T01:55:41.643 INFO:tasks.rados.rados.0.smithi045.stdout:3089: finishing copy_from racing read to smithi04569269-38 2021-07-31T01:55:41.646 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 26 v 1838 (ObjNum 881 snap 272 seq_num 881) dirty exists 2021-07-31T01:55:42.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:41 smithi154 conmon[34553]: audit 2021-07-31T01:55:40.139937+0000 mgr.x (mgr.24542) 363 : audit [DBG] from='client.24907 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:55:42.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:41 smithi154 conmon[34553]: audit 2021-07-31T01:55:40.438542+0000 mgr.x (mgr.24542) 364 : audit [DBG] from='client.24913 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:55:42.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:41 smithi154 conmon[34553]: cluster 2021-07-31T01:55:40.651293+0000 mon.a (mon.0) 2021-07-31T01:55:42.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:41 smithi154 conmon[34553]: 1294 : cluster [DBG] osdmap e632: 8 total, 8 up, 8 in 2021-07-31T01:55:42.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:41 smithi154 conmon[34553]: audit 2021- 2021-07-31T01:55:42.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:41 smithi154 conmon[34553]: 07-31T01:55:40.782648 2021-07-31T01:55:42.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:41 smithi154 conmon[34553]: +0000 mon.a (mon.0) 1295 2021-07-31T01:55:42.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:41 smithi154 conmon[34553]: : audit [DBG] from='client.? 172.21.15.45:0/2590233991' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T01:55:42.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:41 smithi154 conmon[34553]: cluster 2021-07 2021-07-31T01:55:42.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:41 smithi154 conmon[34553]: -31T01:55:41.635882+0000 mon.a 2021-07-31T01:55:42.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:41 smithi154 conmon[34553]: (mon.0) 1296 : cluster [DBG] 2021-07-31T01:55:42.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:41 smithi154 conmon[34553]: osdmap e633: 8 total, 8 up, 8 in 2021-07-31T01:55:42.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:41 smithi045 conmon[33655]: audit 2021-07-31T01:55:40.139937+0000 mgr.x (mgr.24542) 2021-07-31T01:55:42.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:41 smithi045 conmon[33655]: 363 : audit [DBG] from='client.24907 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:55:42.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:41 smithi045 conmon[33655]: audit 2021-07-31T01:55:40.438542+0000 mgr.x (mgr.24542) 364 : audit [DBG] from='client.24913 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:55:42.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:41 smithi045 conmon[33655]: cluster 2021-07-31T01:55:40.651293+0000 mon.a (mon.0) 1294 : cluster [DBG] 2021-07-31T01:55:42.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:41 smithi045 conmon[33655]: osdmap e632: 8 total, 8 up, 8 in 2021-07-31T01:55:42.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:41 smithi045 conmon[33655]: audit 2021-07-31T01:55:40.782648+0000 mon.a (mon.0) 1295 : audit [DBG] from='client.? 172.21.15.45:0/2590233991' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T01:55:42.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:41 smithi045 conmon[33655]: cluster 2021-07-31T01:55:41.635882 2021-07-31T01:55:42.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:41 smithi045 conmon[33655]: +0000 mon.a (mon.0) 1296 : cluster [DBG] osdmap e633: 8 total, 8 up, 8 in 2021-07-31T01:55:42.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:41 smithi045 conmon[39718]: audit 2021-07-31T01:55:40.139937+0000 mgr.x 2021-07-31T01:55:42.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:41 smithi045 conmon[39718]: (mgr.24542) 363 : audit [DBG] from='client.24907 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:55:42.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:41 smithi045 conmon[39718]: audit 2021-07-31T01:55 2021-07-31T01:55:42.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:41 smithi045 conmon[39718]: :40.438542+0000 mgr.x (mgr.24542) 364 : audit [DBG] from='client.24913 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:55:42.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:41 smithi045 conmon[39718]: cluster 2021-07-31T01:55:40.651293+0000 mon.a (mon.0 2021-07-31T01:55:42.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:41 smithi045 conmon[39718]: ) 1294 : cluster [DBG] osdmap e632: 8 total, 8 up, 8 in 2021-07-31T01:55:42.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:41 smithi045 conmon[39718]: audit 2021-07-31T01:55:40.782648+0000 2021-07-31T01:55:42.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:41 smithi045 conmon[39718]: mon.a (mon.0) 1295 : audit [DBG] from='client.? 172.21.15.45:0/2590233991' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T01:55:42.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:41 smithi045 conmon[39718]: cluster 2021-07-31T01:55:41.635882+0000 mon.a (mon.0 2021-07-31T01:55:42.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:41 smithi045 conmon[39718]: ) 1296 : cluster [DBG] osdmap e633: 8 total, 8 up, 8 in 2021-07-31T01:55:42.646 INFO:tasks.rados.rados.0.smithi045.stdout:3089: done (2 left) 2021-07-31T01:55:42.646 INFO:tasks.rados.rados.0.smithi045.stdout:3090: done (1 left) 2021-07-31T01:55:42.647 INFO:tasks.rados.rados.0.smithi045.stdout:3091: done (0 left) 2021-07-31T01:55:42.647 INFO:tasks.rados.rados.0.smithi045.stdout:3092: delete oid 34 current snap is 293 2021-07-31T01:55:42.647 INFO:tasks.rados.rados.0.smithi045.stdout:3092: done (0 left) 2021-07-31T01:55:42.647 INFO:tasks.rados.rados.0.smithi045.stdout:3093: rmattr oid 6 current snap is 293 2021-07-31T01:55:42.648 INFO:tasks.rados.rados.0.smithi045.stdout:3093: done (0 left) 2021-07-31T01:55:42.648 INFO:tasks.rados.rados.0.smithi045.stdout:3094: read oid 36 snap -1 2021-07-31T01:55:42.648 INFO:tasks.rados.rados.0.smithi045.stdout:3094: expect (ObjNum 952 snap 287 seq_num 952) 2021-07-31T01:55:42.648 INFO:tasks.rados.rados.0.smithi045.stdout:3095: delete oid 5 current snap is 293 2021-07-31T01:55:42.649 INFO:tasks.rados.rados.0.smithi045.stdout:3095: done (1 left) 2021-07-31T01:55:42.649 INFO:tasks.rados.rados.0.smithi045.stdout:3096: snap_remove snap 271 2021-07-31T01:55:43.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:42 smithi154 conmon[34553]: cluster 2021-07-31T01:55:41.912478+0000 mgr.x (mgr.24542) 365 : cluster 2021-07-31T01:55:43.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:42 smithi154 conmon[34553]: [DBG] pgmap v734: 25 pgs: 25 active+clean; 526 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 20 MiB/s rd, 5.5 MiB/s wr, 8 op/s 2021-07-31T01:55:43.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:42 smithi154 conmon[34553]: cluster 2021-07-31T01:55:42.641586+0000 mon.a (mon.0) 1297 : cluster [DBG] osdmap e634: 8 total, 8 up, 8 in 2021-07-31T01:55:43.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:42 smithi045 conmon[39718]: cluster 2021-07-31T01:55:43.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:42 smithi045 conmon[39718]: 2021-07-31T01:55:41.912478 2021-07-31T01:55:43.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:42 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 365 : cluster [DBG] pgmap v734: 25 pgs: 25 active+clean; 526 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 20 MiB/s rd, 5.5 MiB/s wr, 8 op/s 2021-07-31T01:55:43.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:42 smithi045 conmon[39718]: cluster 2021-07-31T01:55:42.641586+0000 mon.a 2021-07-31T01:55:43.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:42 smithi045 conmon[39718]: (mon.0) 1297 : cluster [DBG] osdmap e634: 8 total, 8 up, 8 in 2021-07-31T01:55:43.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:42 smithi045 conmon[33655]: cluster 2021-07-31T01:55:43.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:42 smithi045 conmon[33655]: 2021-07-31T01:55:43.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:42 smithi045 conmon[33655]: 2021 2021-07-31T01:55:43.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:42 smithi045 conmon[33655]: - 2021-07-31T01:55:43.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:42 smithi045 conmon[33655]: 07 2021-07-31T01:55:43.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:42 smithi045 conmon[33655]: - 2021-07-31T01:55:43.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:42 smithi045 conmon[33655]: 31 2021-07-31T01:55:43.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:42 smithi045 conmon[33655]: T 2021-07-31T01:55:43.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:42 smithi045 conmon[33655]: 01 2021-07-31T01:55:43.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:42 smithi045 conmon[33655]: : 2021-07-31T01:55:43.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:42 smithi045 conmon[33655]: 55 2021-07-31T01:55:43.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:42 smithi045 conmon[33655]: : 2021-07-31T01:55:43.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:42 smithi045 conmon[33655]: 41 2021-07-31T01:55:43.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:42 smithi045 conmon[33655]: . 2021-07-31T01:55:43.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:42 smithi045 conmon[33655]: 912478 2021-07-31T01:55:43.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:42 smithi045 conmon[33655]: +0000 2021-07-31T01:55:43.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:42 smithi045 conmon[33655]: 2021-07-31T01:55:43.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:42 smithi045 conmon[33655]: mgr.x 2021-07-31T01:55:43.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:42 smithi045 conmon[33655]: ( 2021-07-31T01:55:43.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:42 smithi045 conmon[33655]: mgr 2021-07-31T01:55:43.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:42 smithi045 conmon[33655]: . 2021-07-31T01:55:43.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:42 smithi045 conmon[33655]: 24542 2021-07-31T01:55:43.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:42 smithi045 conmon[33655]: ) 2021-07-31T01:55:43.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:42 smithi045 conmon[33655]: 365 2021-07-31T01:55:43.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:42 smithi045 conmon[33655]: : 2021-07-31T01:55:43.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:42 smithi045 conmon[33655]: cluster 2021-07-31T01:55:43.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:42 smithi045 conmon[33655]: 2021-07-31T01:55:43.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:42 smithi045 conmon[33655]: [DBG] 2021-07-31T01:55:43.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:42 smithi045 conmon[33655]: 2021-07-31T01:55:43.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:42 smithi045 conmon[33655]: pgmap v734: 25 pgs: 25 active+clean; 526 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 20 MiB/s rd, 5.5 MiB/s wr, 8 op/s 2021-07-31T01:55:43.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:42 smithi045 conmon[33655]: 2021-07-31T01:55:43.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:42 smithi045 conmon[33655]: cluster 2021-07-31T01:55:43.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:42 smithi045 conmon[33655]: 2021-07-31T01:55:43.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:42 smithi045 conmon[33655]: 2021 2021-07-31T01:55:43.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:42 smithi045 conmon[33655]: - 2021-07-31T01:55:43.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:42 smithi045 conmon[33655]: 07 2021-07-31T01:55:43.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:42 smithi045 conmon[33655]: - 2021-07-31T01:55:43.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:42 smithi045 conmon[33655]: 31 2021-07-31T01:55:43.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:42 smithi045 conmon[33655]: T 2021-07-31T01:55:43.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:42 smithi045 conmon[33655]: 01 2021-07-31T01:55:43.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:42 smithi045 conmon[33655]: : 2021-07-31T01:55:43.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:42 smithi045 conmon[33655]: 55 2021-07-31T01:55:43.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:42 smithi045 conmon[33655]: : 2021-07-31T01:55:43.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:42 smithi045 conmon[33655]: 42 2021-07-31T01:55:43.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:42 smithi045 conmon[33655]: . 2021-07-31T01:55:43.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:42 smithi045 conmon[33655]: 641586 2021-07-31T01:55:43.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:42 smithi045 conmon[33655]: +0000 2021-07-31T01:55:43.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:42 smithi045 conmon[33655]: 2021-07-31T01:55:43.040 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:42 smithi045 conmon[33655]: mon.a 2021-07-31T01:55:43.040 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:42 smithi045 conmon[33655]: ( 2021-07-31T01:55:43.040 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:42 smithi045 conmon[33655]: mon 2021-07-31T01:55:43.040 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:42 smithi045 conmon[33655]: . 2021-07-31T01:55:43.041 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:42 smithi045 conmon[33655]: 0 2021-07-31T01:55:43.041 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:42 smithi045 conmon[33655]: ) 2021-07-31T01:55:43.041 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:42 smithi045 conmon[33655]: 1297 2021-07-31T01:55:43.042 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:42 smithi045 conmon[33655]: : 2021-07-31T01:55:43.042 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:42 smithi045 conmon[33655]: cluster 2021-07-31T01:55:43.042 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:42 smithi045 conmon[33655]: 2021-07-31T01:55:43.042 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:42 smithi045 conmon[33655]: [DBG] 2021-07-31T01:55:43.042 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:42 smithi045 conmon[33655]: 2021-07-31T01:55:43.043 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:42 smithi045 conmon[33655]: osdmap e634: 8 total, 8 up, 8 in 2021-07-31T01:55:43.043 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:42 smithi045 conmon[33655]: 2021-07-31T01:55:43.645 INFO:tasks.rados.rados.0.smithi045.stdout:3094: expect (ObjNum 952 snap 287 seq_num 952) 2021-07-31T01:55:43.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:55:43 smithi045 conmon[61210]: level=error ts=2021-07-31T01:55:43.523Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:55:43.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:55:43 smithi045 conmon[61210]: level=error ts=2021-07-31T01:55:43.523Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:55:43.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:55:43 smithi045 conmon[61210]: level=error ts=2021-07-31T01:55:43.523Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T01:55:45.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:44 smithi154 conmon[34553]: cluster 2021-07-31T01:55:43.643434+0000 mon.a (mon.0) 1298 : cluster [DBG] osdmap e635: 8 total, 8 up, 8 in 2021-07-31T01:55:45.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:44 smithi154 conmon[34553]: cluster 2021-07-31T01:55:43.912985+0000 mgr.x (mgr.24542) 366 : cluster [DBG] pgmap v737: 25 pgs: 25 active+clean; 517 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 0 B/s wr, 6 op/s 2021-07-31T01:55:45.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:44 smithi045 conmon[39718]: cluster 2021-07-31T01:55:43.643434+0000 mon.a (mon 2021-07-31T01:55:45.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:44 smithi045 conmon[39718]: .0) 1298 : cluster [DBG] osdmap e635: 8 total, 8 up, 8 in 2021-07-31T01:55:45.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:44 smithi045 conmon[39718]: cluster 2021-07-31T01:55:43.912985+0000 mgr.x (mgr.24542) 366 : cluster [DBG] pgmap v737: 25 pgs: 25 active+clean; 517 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 0 B/s wr, 6 op/s 2021-07-31T01:55:45.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:44 smithi045 conmon[33655]: cluster 2021-07-31T01:55:43.643434+0000 mon.a ( 2021-07-31T01:55:45.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:44 smithi045 conmon[33655]: mon.0) 1298 : cluster [DBG] osdmap e635: 8 total, 8 up, 8 in 2021-07-31T01:55:45.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:44 smithi045 conmon[33655]: cluster 2021-07-31T01:55:43.912985+0000 mgr.x (mgr.24542) 366 : cluster [DBG] pgmap v737: 25 pgs: 25 active+clean; 517 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 0 B/s wr, 6 op/s 2021-07-31T01:55:45.496 INFO:tasks.rados.rados.0.smithi045.stdout:3094: done (1 left) 2021-07-31T01:55:45.496 INFO:tasks.rados.rados.0.smithi045.stdout:3096: done (0 left) 2021-07-31T01:55:45.496 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 46 current snap is 293 2021-07-31T01:55:45.497 INFO:tasks.rados.rados.0.smithi045.stdout:3097: seq_num 958 ranges {0=450560,450560=655360,1105920=548864} 2021-07-31T01:55:45.498 INFO:tasks.rados.rados.0.smithi045.stdout:3097: writing smithi04569269-46 from 0 to 450560 tid 1 2021-07-31T01:55:45.502 INFO:tasks.rados.rados.0.smithi045.stdout:3097: writing smithi04569269-46 from 450560 to 1105920 tid 2 2021-07-31T01:55:45.505 INFO:tasks.rados.rados.0.smithi045.stdout:3097: writing smithi04569269-46 from 1105920 to 1654784 tid 3 2021-07-31T01:55:45.506 INFO:tasks.rados.rados.0.smithi045.stdout:3098: copy_from oid 36 from oid 23 current snap is 293 2021-07-31T01:55:45.506 INFO:tasks.rados.rados.0.smithi045.stdout:3097: finishing write tid 1 to smithi04569269-46 2021-07-31T01:55:45.506 INFO:tasks.rados.rados.0.smithi045.stdout:3099: read oid 33 snap -1 2021-07-31T01:55:45.506 INFO:tasks.rados.rados.0.smithi045.stdout:3099: expect (ObjNum 940 snap 285 seq_num 940) 2021-07-31T01:55:45.507 INFO:tasks.rados.rados.0.smithi045.stdout:3100: snap_remove snap 291 2021-07-31T01:55:45.648 INFO:tasks.rados.rados.0.smithi045.stdout:3100: done (3 left) 2021-07-31T01:55:45.649 INFO:tasks.rados.rados.0.smithi045.stdout:3101: snap_remove snap 244 2021-07-31T01:55:46.638 INFO:tasks.rados.rados.0.smithi045.stdout:3101: done (3 left) 2021-07-31T01:55:46.638 INFO:tasks.rados.rados.0.smithi045.stdout:3102: copy_from oid 25 from oid 27 current snap is 293 2021-07-31T01:55:46.639 INFO:tasks.rados.rados.0.smithi045.stdout:3097: finishing write tid 2 to smithi04569269-46 2021-07-31T01:55:46.639 INFO:tasks.rados.rados.0.smithi045.stdout:3097: finishing write tid 3 to smithi04569269-46 2021-07-31T01:55:46.639 INFO:tasks.rados.rados.0.smithi045.stdout:3097: finishing write tid 4 to smithi04569269-46 2021-07-31T01:55:46.640 INFO:tasks.rados.rados.0.smithi045.stdout:3097: finishing write tid 5 to smithi04569269-46 2021-07-31T01:55:46.640 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 46 v 1313 (ObjNum 958 snap 293 seq_num 958) dirty exists 2021-07-31T01:55:46.640 INFO:tasks.rados.rados.0.smithi045.stdout:3097: left oid 46 (ObjNum 958 snap 293 seq_num 958) 2021-07-31T01:55:46.640 INFO:tasks.rados.rados.0.smithi045.stdout:3098: finishing copy_from to smithi04569269-36 2021-07-31T01:55:46.641 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 36 v 1644 (ObjNum 917 snap 278 seq_num 917) dirty exists 2021-07-31T01:55:46.641 INFO:tasks.rados.rados.0.smithi045.stdout:3099: expect (ObjNum 940 snap 285 seq_num 940) 2021-07-31T01:55:46.733 INFO:tasks.rados.rados.0.smithi045.stdout:3098: finishing copy_from racing read to smithi04569269-36 2021-07-31T01:55:46.734 INFO:tasks.rados.rados.0.smithi045.stdout:3097: done (3 left) 2021-07-31T01:55:46.734 INFO:tasks.rados.rados.0.smithi045.stdout:3098: done (2 left) 2021-07-31T01:55:46.735 INFO:tasks.rados.rados.0.smithi045.stdout:3099: done (1 left) 2021-07-31T01:55:46.735 INFO:tasks.rados.rados.0.smithi045.stdout:3103: read oid 48 snap -1 2021-07-31T01:55:46.735 INFO:tasks.rados.rados.0.smithi045.stdout:3103: expect (ObjNum 881 snap 272 seq_num 881) 2021-07-31T01:55:46.736 INFO:tasks.rados.rados.0.smithi045.stdout:3102: finishing copy_from to smithi04569269-25 2021-07-31T01:55:46.736 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 25 v 1640 (ObjNum 837 snap 263 seq_num 837) dirty exists 2021-07-31T01:55:46.737 INFO:tasks.rados.rados.0.smithi045.stdout:3102: finishing copy_from racing read to smithi04569269-25 2021-07-31T01:55:46.737 INFO:tasks.rados.rados.0.smithi045.stdout:3104: read oid 13 snap -1 2021-07-31T01:55:46.737 INFO:tasks.rados.rados.0.smithi045.stdout:3104: expect deleted 2021-07-31T01:55:46.738 INFO:tasks.rados.rados.0.smithi045.stdout:3102: done (2 left) 2021-07-31T01:55:46.738 INFO:tasks.rados.rados.0.smithi045.stdout:3105: copy_from oid 30 from oid 19 current snap is 293 2021-07-31T01:55:46.739 INFO:tasks.rados.rados.0.smithi045.stdout:3106: rollback oid 40 current snap is 293 2021-07-31T01:55:46.739 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 40 to 266 2021-07-31T01:55:46.740 INFO:tasks.rados.rados.0.smithi045.stdout:3107: snap_create 2021-07-31T01:55:46.740 INFO:tasks.rados.rados.0.smithi045.stdout:3105: finishing copy_from to smithi04569269-30 2021-07-31T01:55:46.740 INFO:tasks.rados.rados.0.smithi045.stdout:3105: got expected ENOENT (src dne) 2021-07-31T01:55:46.741 INFO:tasks.rados.rados.0.smithi045.stdout:3105: finishing copy_from racing read to smithi04569269-30 2021-07-31T01:55:46.742 INFO:tasks.rados.rados.0.smithi045.stdout:3106: finishing rollback tid 0 to smithi04569269-40 2021-07-31T01:55:46.743 INFO:tasks.rados.rados.0.smithi045.stdout:3106: finishing rollback tid 1 to smithi04569269-40 2021-07-31T01:55:46.743 INFO:tasks.rados.rados.0.smithi045.stdout:3106: finishing rollback tid 2 to smithi04569269-40 2021-07-31T01:55:46.744 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 40 v 1845 (ObjNum 845 snap 265 seq_num 845) dirty exists 2021-07-31T01:55:46.761 INFO:tasks.rados.rados.0.smithi045.stdout:3103: expect (ObjNum 881 snap 272 seq_num 881) 2021-07-31T01:55:47.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:46 smithi154 conmon[34553]: cluster 2021-07-31T01:55:45.648385+0000 2021-07-31T01:55:47.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:46 smithi154 conmon[34553]: mon.a (mon.0) 1299 : cluster [DBG] osdmap e636: 8 total, 8 up, 8 in 2021-07-31T01:55:47.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:46 smithi154 conmon[34553]: cluster 2021-07-31T01:55:45.913416+0000 mgr.x (mgr.24542) 367 : cluster [DBG] pgmap v739: 25 pgs: 25 active+clean; 517 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 0 B/s wr, 6 op/s 2021-07-31T01:55:47.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:46 smithi154 conmon[34553]: cluster 2021-07-31T01:55:46.636863+0000 mon.a (mon.0) 1300 : cluster [DBG] osdmap e637: 8 total, 8 up, 8 in 2021-07-31T01:55:47.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:46 smithi045 conmon[33655]: cluster 2021-07-31T01:55:45.648385+0000 mon.a (mon.0) 1299 : cluster [DBG] osdmap e636: 8 total, 8 up, 8 in 2021-07-31T01:55:47.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:46 smithi045 conmon[33655]: cluster 2021-07-31T01:55:45.913416+0000 mgr.x (mgr.24542) 367 : cluster [DBG] pgmap v739: 25 pgs: 25 active+clean; 517 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 0 B/s wr, 6 op/s 2021-07-31T01:55:47.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:46 smithi045 conmon[33655]: cluster 2021-07-31T01:55:46.636863+0000 mon.a (mon.0) 1300 : cluster [DBG] osdmap e637: 8 total, 8 up, 8 in 2021-07-31T01:55:47.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:46 smithi045 conmon[39718]: cluster 2021-07-31T01:55:45.648385+0000 2021-07-31T01:55:47.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:46 smithi045 conmon[39718]: mon.a (mon.0) 1299 : cluster [DBG] osdmap e636: 8 total, 8 up, 8 in 2021-07-31T01:55:47.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:46 smithi045 conmon[39718]: cluster 2021- 2021-07-31T01:55:47.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:46 smithi045 conmon[39718]: 07-31T01:55:45.913416+0000 mgr.x 2021-07-31T01:55:47.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:46 smithi045 conmon[39718]: (mgr.24542) 367 : cluster [DBG] pgmap v739: 25 pgs: 25 active+clean; 517 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 0 B/s wr, 6 op/s 2021-07-31T01:55:47.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:46 smithi045 conmon[39718]: 2021-07-31T01:55:47.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:46 smithi045 conmon[39718]: cluster 2021- 2021-07-31T01:55:47.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:46 smithi045 conmon[39718]: 07-31T01:55:46.636863+0000 2021-07-31T01:55:47.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:46 smithi045 conmon[39718]: mon.a (mon.0) 1300 : cluster 2021-07-31T01:55:47.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:46 smithi045 conmon[39718]: [DBG] osdmap e637: 8 total, 8 up, 8 in 2021-07-31T01:55:47.778 INFO:tasks.rados.rados.0.smithi045.stdout:3103: done (4 left) 2021-07-31T01:55:47.778 INFO:tasks.rados.rados.0.smithi045.stdout:3104: done (3 left) 2021-07-31T01:55:47.779 INFO:tasks.rados.rados.0.smithi045.stdout:3105: done (2 left) 2021-07-31T01:55:47.779 INFO:tasks.rados.rados.0.smithi045.stdout:3106: done (1 left) 2021-07-31T01:55:47.779 INFO:tasks.rados.rados.0.smithi045.stdout:3107: done (0 left) 2021-07-31T01:55:47.779 INFO:tasks.rados.rados.0.smithi045.stdout:3108: rollback oid 13 current snap is 294 2021-07-31T01:55:47.780 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 13 to 293 2021-07-31T01:55:47.780 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 42 current snap is 294 2021-07-31T01:55:47.780 INFO:tasks.rados.rados.0.smithi045.stdout:3109: seq_num 959 ranges {1015808=466944,1482752=540672,2023424=557056,2580480=106496} 2021-07-31T01:55:47.786 INFO:tasks.rados.rados.0.smithi045.stdout:3109: writing smithi04569269-42 from 1015808 to 1482752 tid 1 2021-07-31T01:55:47.789 INFO:tasks.rados.rados.0.smithi045.stdout:3109: writing smithi04569269-42 from 1482752 to 2023424 tid 2 2021-07-31T01:55:47.792 INFO:tasks.rados.rados.0.smithi045.stdout:3109: writing smithi04569269-42 from 2023424 to 2580480 tid 3 2021-07-31T01:55:47.794 INFO:tasks.rados.rados.0.smithi045.stdout:3109: writing smithi04569269-42 from 2580480 to 2686976 tid 4 2021-07-31T01:55:47.794 INFO:tasks.rados.rados.0.smithi045.stdout:3110: copy_from oid 47 from oid 38 current snap is 294 2021-07-31T01:55:47.795 INFO:tasks.rados.rados.0.smithi045.stdout:3108: finishing rollback tid 1 to smithi04569269-13 2021-07-31T01:55:47.795 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 13 v 1837 (ObjNum 1 snap 21873 seq_num 3189086192) dirty dne 2021-07-31T01:55:47.795 INFO:tasks.rados.rados.0.smithi045.stdout:3109: finishing write tid 1 to smithi04569269-42 2021-07-31T01:55:47.796 INFO:tasks.rados.rados.0.smithi045.stdout:3109: finishing write tid 2 to smithi04569269-42 2021-07-31T01:55:47.796 INFO:tasks.rados.rados.0.smithi045.stdout:3108: done (2 left) 2021-07-31T01:55:47.797 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 6 current snap is 294 2021-07-31T01:55:47.797 INFO:tasks.rados.rados.0.smithi045.stdout:3111: seq_num 960 ranges {0=483328,483328=425984,909312=761856,1671168=16384} 2021-07-31T01:55:47.798 INFO:tasks.rados.rados.0.smithi045.stdout:3111: writing smithi04569269-6 from 0 to 483328 tid 1 2021-07-31T01:55:47.799 INFO:tasks.rados.rados.0.smithi045.stdout:3111: writing smithi04569269-6 from 483328 to 909312 tid 2 2021-07-31T01:55:47.803 INFO:tasks.rados.rados.0.smithi045.stdout:3111: writing smithi04569269-6 from 909312 to 1671168 tid 3 2021-07-31T01:55:47.804 INFO:tasks.rados.rados.0.smithi045.stdout:3111: writing smithi04569269-6 from 1671168 to 1687552 tid 4 2021-07-31T01:55:47.804 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 12 current snap is 294 2021-07-31T01:55:47.804 INFO:tasks.rados.rados.0.smithi045.stdout:3110: finishing copy_from to smithi04569269-47 2021-07-31T01:55:47.804 INFO:tasks.rados.rados.0.smithi045.stdout:3110: got expected ENOENT (src dne) 2021-07-31T01:55:47.804 INFO:tasks.rados.rados.0.smithi045.stdout:3110: finishing copy_from racing read to smithi04569269-47 2021-07-31T01:55:47.805 INFO:tasks.rados.rados.0.smithi045.stdout:3109: finishing write tid 3 to smithi04569269-42 2021-07-31T01:55:47.805 INFO:tasks.rados.rados.0.smithi045.stdout:3109: finishing write tid 4 to smithi04569269-42 2021-07-31T01:55:47.805 INFO:tasks.rados.rados.0.smithi045.stdout:3109: finishing write tid 5 to smithi04569269-42 2021-07-31T01:55:47.805 INFO:tasks.rados.rados.0.smithi045.stdout:3111: finishing write tid 1 to smithi04569269-6 2021-07-31T01:55:47.806 INFO:tasks.rados.rados.0.smithi045.stdout:3109: finishing write tid 6 to smithi04569269-42 2021-07-31T01:55:47.806 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 42 v 1695 (ObjNum 959 snap 294 seq_num 959) dirty exists 2021-07-31T01:55:47.806 INFO:tasks.rados.rados.0.smithi045.stdout:3109: left oid 42 (ObjNum 959 snap 294 seq_num 959) 2021-07-31T01:55:47.806 INFO:tasks.rados.rados.0.smithi045.stdout:3111: finishing write tid 2 to smithi04569269-6 2021-07-31T01:55:47.807 INFO:tasks.rados.rados.0.smithi045.stdout:3112: seq_num 961 ranges {1368064=499712,1867776=712704,2580480=491520} 2021-07-31T01:55:47.813 INFO:tasks.rados.rados.0.smithi045.stdout:3112: writing smithi04569269-12 from 1368064 to 1867776 tid 1 2021-07-31T01:55:47.817 INFO:tasks.rados.rados.0.smithi045.stdout:3112: writing smithi04569269-12 from 1867776 to 2580480 tid 2 2021-07-31T01:55:47.820 INFO:tasks.rados.rados.0.smithi045.stdout:3112: writing smithi04569269-12 from 2580480 to 3072000 tid 3 2021-07-31T01:55:47.820 INFO:tasks.rados.rados.0.smithi045.stdout:3109: done (3 left) 2021-07-31T01:55:47.821 INFO:tasks.rados.rados.0.smithi045.stdout:3110: done (2 left) 2021-07-31T01:55:47.821 INFO:tasks.rados.rados.0.smithi045.stdout:3113: snap_create 2021-07-31T01:55:47.821 INFO:tasks.rados.rados.0.smithi045.stdout:3111: finishing write tid 3 to smithi04569269-6 2021-07-31T01:55:47.821 INFO:tasks.rados.rados.0.smithi045.stdout:3111: finishing write tid 4 to smithi04569269-6 2021-07-31T01:55:47.821 INFO:tasks.rados.rados.0.smithi045.stdout:3111: finishing write tid 5 to smithi04569269-6 2021-07-31T01:55:47.822 INFO:tasks.rados.rados.0.smithi045.stdout:3111: finishing write tid 6 to smithi04569269-6 2021-07-31T01:55:47.822 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 6 v 1850 (ObjNum 960 snap 294 seq_num 960) dirty exists 2021-07-31T01:55:47.822 INFO:tasks.rados.rados.0.smithi045.stdout:3111: left oid 6 (ObjNum 960 snap 294 seq_num 960) 2021-07-31T01:55:47.823 INFO:tasks.rados.rados.0.smithi045.stdout:3112: finishing write tid 1 to smithi04569269-12 2021-07-31T01:55:47.823 INFO:tasks.rados.rados.0.smithi045.stdout:3112: finishing write tid 2 to smithi04569269-12 2021-07-31T01:55:47.824 INFO:tasks.rados.rados.0.smithi045.stdout:3112: finishing write tid 3 to smithi04569269-12 2021-07-31T01:55:47.824 INFO:tasks.rados.rados.0.smithi045.stdout:3112: finishing write tid 4 to smithi04569269-12 2021-07-31T01:55:47.825 INFO:tasks.rados.rados.0.smithi045.stdout:3112: finishing write tid 5 to smithi04569269-12 2021-07-31T01:55:47.825 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 12 v 1854 (ObjNum 961 snap 294 seq_num 961) dirty exists 2021-07-31T01:55:47.825 INFO:tasks.rados.rados.0.smithi045.stdout:3112: left oid 12 (ObjNum 961 snap 294 seq_num 961) 2021-07-31T01:55:48.671 INFO:tasks.rados.rados.0.smithi045.stdout:3111: done (2 left) 2021-07-31T01:55:48.671 INFO:tasks.rados.rados.0.smithi045.stdout:3112: done (1 left) 2021-07-31T01:55:48.671 INFO:tasks.rados.rados.0.smithi045.stdout:3113: done (0 left) 2021-07-31T01:55:48.672 INFO:tasks.rados.rados.0.smithi045.stdout:3114: read oid 35 snap 293 2021-07-31T01:55:48.672 INFO:tasks.rados.rados.0.smithi045.stdout:3114: expect (ObjNum 930 snap 282 seq_num 930) 2021-07-31T01:55:48.672 INFO:tasks.rados.rados.0.smithi045.stdout:3115: read oid 5 snap 266 2021-07-31T01:55:48.672 INFO:tasks.rados.rados.0.smithi045.stdout:3115: expect (ObjNum 817 snap 257 seq_num 817) 2021-07-31T01:55:48.673 INFO:tasks.rados.rados.0.smithi045.stdout:3116: copy_from oid 4 from oid 3 current snap is 295 2021-07-31T01:55:48.675 INFO:tasks.rados.rados.0.smithi045.stdout:3117: delete oid 27 current snap is 295 2021-07-31T01:55:48.676 INFO:tasks.rados.rados.0.smithi045.stdout:3117: done (3 left) 2021-07-31T01:55:48.676 INFO:tasks.rados.rados.0.smithi045.stdout:3118: delete oid 47 current snap is 295 2021-07-31T01:55:48.678 INFO:tasks.rados.rados.0.smithi045.stdout:3118: done (3 left) 2021-07-31T01:55:48.679 INFO:tasks.rados.rados.0.smithi045.stdout:3119: read oid 46 snap 286 2021-07-31T01:55:48.679 INFO:tasks.rados.rados.0.smithi045.stdout:3119: expect (ObjNum 826 snap 258 seq_num 826) 2021-07-31T01:55:48.679 INFO:tasks.rados.rados.0.smithi045.stdout:3120: read oid 17 snap -1 2021-07-31T01:55:48.679 INFO:tasks.rados.rados.0.smithi045.stdout:3120: expect (ObjNum 943 snap 286 seq_num 943) 2021-07-31T01:55:48.679 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 24 current snap is 295 2021-07-31T01:55:48.680 INFO:tasks.rados.rados.0.smithi045.stdout:3121: seq_num 964 ranges {0=548864,548864=761856,1310720=442368} 2021-07-31T01:55:48.682 INFO:tasks.rados.rados.0.smithi045.stdout:3121: writing smithi04569269-24 from 0 to 548864 tid 1 2021-07-31T01:55:48.686 INFO:tasks.rados.rados.0.smithi045.stdout:3121: writing smithi04569269-24 from 548864 to 1310720 tid 2 2021-07-31T01:55:48.689 INFO:tasks.rados.rados.0.smithi045.stdout:3121: writing smithi04569269-24 from 1310720 to 1753088 tid 3 2021-07-31T01:55:48.690 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 13 current snap is 295 2021-07-31T01:55:48.690 INFO:tasks.rados.rados.0.smithi045.stdout:3122: seq_num 965 ranges {0=565248,565248=647168,1212416=548864,1761280=8192} 2021-07-31T01:55:48.692 INFO:tasks.rados.rados.0.smithi045.stdout:3122: writing smithi04569269-13 from 0 to 565248 tid 1 2021-07-31T01:55:48.697 INFO:tasks.rados.rados.0.smithi045.stdout:3122: writing smithi04569269-13 from 565248 to 1212416 tid 2 2021-07-31T01:55:48.700 INFO:tasks.rados.rados.0.smithi045.stdout:3122: writing smithi04569269-13 from 1212416 to 1761280 tid 3 2021-07-31T01:55:48.701 INFO:tasks.rados.rados.0.smithi045.stdout:3122: writing smithi04569269-13 from 1761280 to 1769472 tid 4 2021-07-31T01:55:48.701 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 15 current snap is 295 2021-07-31T01:55:48.701 INFO:tasks.rados.rados.0.smithi045.stdout:3114: expect (ObjNum 930 snap 282 seq_num 930) 2021-07-31T01:55:48.738 INFO:tasks.rados.rados.0.smithi045.stdout:3121: finishing write tid 1 to smithi04569269-24 2021-07-31T01:55:48.739 INFO:tasks.rados.rados.0.smithi045.stdout:3123: seq_num 966 ranges {1269760=581632,1851392=581632,2433024=622592} 2021-07-31T01:55:48.749 INFO:tasks.rados.rados.0.smithi045.stdout:3123: writing smithi04569269-15 from 1269760 to 1851392 tid 1 2021-07-31T01:55:48.752 INFO:tasks.rados.rados.0.smithi045.stdout:3123: writing smithi04569269-15 from 1851392 to 2433024 tid 2 2021-07-31T01:55:48.756 INFO:tasks.rados.rados.0.smithi045.stdout:3123: writing smithi04569269-15 from 2433024 to 3055616 tid 3 2021-07-31T01:55:48.757 INFO:tasks.rados.rados.0.smithi045.stdout:3114: done (7 left) 2021-07-31T01:55:48.757 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 12 current snap is 295 2021-07-31T01:55:48.757 INFO:tasks.rados.rados.0.smithi045.stdout:3121: finishing write tid 2 to smithi04569269-24 2021-07-31T01:55:48.757 INFO:tasks.rados.rados.0.smithi045.stdout:3121: finishing write tid 3 to smithi04569269-24 2021-07-31T01:55:48.758 INFO:tasks.rados.rados.0.smithi045.stdout:3121: finishing write tid 4 to smithi04569269-24 2021-07-31T01:55:48.758 INFO:tasks.rados.rados.0.smithi045.stdout:3121: finishing write tid 5 to smithi04569269-24 2021-07-31T01:55:48.758 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 24 v 1646 (ObjNum 964 snap 295 seq_num 964) dirty exists 2021-07-31T01:55:48.759 INFO:tasks.rados.rados.0.smithi045.stdout:3121: left oid 24 (ObjNum 964 snap 295 seq_num 964) 2021-07-31T01:55:48.759 INFO:tasks.rados.rados.0.smithi045.stdout:3120: expect (ObjNum 943 snap 286 seq_num 943) 2021-07-31T01:55:48.895 INFO:tasks.rados.rados.0.smithi045.stdout:3122: finishing write tid 1 to smithi04569269-13 2021-07-31T01:55:48.895 INFO:tasks.rados.rados.0.smithi045.stdout:3122: finishing write tid 2 to smithi04569269-13 2021-07-31T01:55:48.896 INFO:tasks.rados.rados.0.smithi045.stdout:3122: finishing write tid 3 to smithi04569269-13 2021-07-31T01:55:48.896 INFO:tasks.rados.rados.0.smithi045.stdout:3122: finishing write tid 4 to smithi04569269-13 2021-07-31T01:55:48.896 INFO:tasks.rados.rados.0.smithi045.stdout:3122: finishing write tid 5 to smithi04569269-13 2021-07-31T01:55:48.898 INFO:tasks.rados.rados.0.smithi045.stdout:3122: finishing write tid 6 to smithi04569269-13 2021-07-31T01:55:48.899 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 13 v 1859 (ObjNum 965 snap 295 seq_num 965) dirty exists 2021-07-31T01:55:48.899 INFO:tasks.rados.rados.0.smithi045.stdout:3122: left oid 13 (ObjNum 965 snap 295 seq_num 965) 2021-07-31T01:55:48.899 INFO:tasks.rados.rados.0.smithi045.stdout:3116: finishing copy_from to smithi04569269-4 2021-07-31T01:55:48.899 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 4 v 1837 (ObjNum 881 snap 272 seq_num 881) dirty exists 2021-07-31T01:55:48.899 INFO:tasks.rados.rados.0.smithi045.stdout:3116: finishing copy_from racing read to smithi04569269-4 2021-07-31T01:55:48.900 INFO:tasks.rados.rados.0.smithi045.stdout:3124: seq_num 967 ranges {3072000=598016,3670016=466944,4136960=516096,4653056=221184} 2021-07-31T01:55:48.914 INFO:tasks.rados.rados.0.smithi045.stdout:3124: writing smithi04569269-12 from 3072000 to 3670016 tid 1 2021-07-31T01:55:48.917 INFO:tasks.rados.rados.0.smithi045.stdout:3124: writing smithi04569269-12 from 3670016 to 4136960 tid 2 2021-07-31T01:55:48.920 INFO:tasks.rados.rados.0.smithi045.stdout:3124: writing smithi04569269-12 from 4136960 to 4653056 tid 3 2021-07-31T01:55:48.921 INFO:tasks.rados.rados.0.smithi045.stdout:3124: writing smithi04569269-12 from 4653056 to 4874240 tid 4 2021-07-31T01:55:48.921 INFO:tasks.rados.rados.0.smithi045.stdout:3116: done (7 left) 2021-07-31T01:55:48.922 INFO:tasks.rados.rados.0.smithi045.stdout:3120: done (6 left) 2021-07-31T01:55:48.922 INFO:tasks.rados.rados.0.smithi045.stdout:3121: done (5 left) 2021-07-31T01:55:48.922 INFO:tasks.rados.rados.0.smithi045.stdout:3122: done (4 left) 2021-07-31T01:55:48.923 INFO:tasks.rados.rados.0.smithi045.stdout:3125: snap_remove snap 284 2021-07-31T01:55:49.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:48 smithi154 conmon[34553]: cluster 2021-07-31T01:55:49.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:48 smithi154 conmon[34553]: 2021-07-31T01:55:47.661731+0000 2021-07-31T01:55:49.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:48 smithi154 conmon[34553]: mon.a (mon.0) 1301 : cluster [DBG] osdmap e638: 8 total, 8 up, 8 in 2021-07-31T01:55:49.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:48 smithi154 conmon[34553]: cluster 2021-07-31T 2021-07-31T01:55:49.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:48 smithi154 conmon[34553]: 01:55:47.913780+0000 mgr.x (mgr.24542) 368 : cluster [DBG] pgmap v742: 25 pgs: 25 active+clean; 517 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 0 B/s wr, 6 op/s 2021-07-31T01:55:49.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:48 smithi045 conmon[33655]: cluster 2021-07-31T01:55:47.661731+0000 mon.a (mon.0) 1301 : cluster 2021-07-31T01:55:49.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:48 smithi045 conmon[33655]: [DBG] osdmap e638: 8 total, 8 up, 8 in 2021-07-31T01:55:49.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:48 smithi045 conmon[33655]: cluster 2021-07-31T01:55:47.913780+0000 mgr.x (mgr.24542) 368 : cluster [DBG] pgmap v742: 25 pgs: 25 active+clean; 517 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 0 B/s wr, 6 op/s 2021-07-31T01:55:49.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:48 smithi045 conmon[39718]: cluster 2021-07-31T01:55:47. 2021-07-31T01:55:49.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:48 smithi045 conmon[39718]: 661731+0000 mon.a (mon.0) 1301 : cluster [DBG] osdmap e638: 8 total, 8 up, 8 in 2021-07-31T01:55:49.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:48 smithi045 conmon[39718]: cluster 2021-07- 2021-07-31T01:55:49.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:48 smithi045 conmon[39718]: 31T01:55:47.913780+0000 mgr.x (mgr.24542) 368 : cluster 2021-07-31T01:55:49.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:48 smithi045 conmon[39718]: [DBG] pgmap v742: 25 pgs: 25 active+clean; 517 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 0 B/s wr, 6 op/s 2021-07-31T01:55:49.675 INFO:tasks.rados.rados.0.smithi045.stdout:3125: done (4 left) 2021-07-31T01:55:49.676 INFO:tasks.rados.rados.0.smithi045.stdout:3126: snap_remove snap 292 2021-07-31T01:55:50.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:49 smithi154 conmon[34553]: cluster 2021-07-31T01:55:48.668705+0000 mon.a (mon.0) 1302 : cluster [DBG] osdmap e639: 8 total, 8 up, 8 in 2021-07-31T01:55:50.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:49 smithi045 conmon[33655]: cluster 2021-07-31T01:55:48.668705+0000 2021-07-31T01:55:50.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:49 smithi045 conmon[33655]: mon.a (mon.0) 1302 : cluster [DBG] osdmap e639: 8 total, 8 up, 8 in 2021-07-31T01:55:50.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:49 smithi045 conmon[39718]: cluster 2021-07-31T01:55:48.668705+0000 mon.a ( 2021-07-31T01:55:50.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:49 smithi045 conmon[39718]: mon.0) 1302 : cluster [DBG] osdmap e639: 8 total, 8 up, 8 in 2021-07-31T01:55:50.682 INFO:tasks.rados.rados.0.smithi045.stdout:3126: done (4 left) 2021-07-31T01:55:50.683 INFO:tasks.rados.rados.0.smithi045.stdout:3127: rollback oid 37 current snap is 295 2021-07-31T01:55:50.683 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 37 to 285 2021-07-31T01:55:50.683 INFO:tasks.rados.rados.0.smithi045.stdout:3119: expect (ObjNum 826 snap 258 seq_num 826) 2021-07-31T01:55:51.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:50 smithi154 conmon[34553]: cluster 2021-07-31T01:55:49.674210+0000 mon.a (mon.0) 1303 : cluster [DBG] osdmap e640: 8 total, 8 up, 8 in 2021-07-31T01:55:51.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:50 smithi154 conmon[34553]: cluster 2021-07-31T01:55:49.914192+0000 mgr.x (mgr.24542) 369 : cluster [DBG] pgmap v745: 25 pgs: 25 active+clean; 537 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 25 MiB/s rd, 5.9 MiB/s wr, 17 op/s 2021-07-31T01:55:51.024 INFO:tasks.rados.rados.0.smithi045.stdout:3123: finishing write tid 1 to smithi04569269-15 2021-07-31T01:55:51.024 INFO:tasks.rados.rados.0.smithi045.stdout:3115: expect (ObjNum 817 snap 257 seq_num 817) 2021-07-31T01:55:51.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:50 smithi045 conmon[33655]: cluster 2021-07-31T01:55:49.674210+0000 mon.a (mon.0) 2021-07-31T01:55:51.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:50 smithi045 conmon[33655]: 1303 : cluster [DBG] osdmap e640: 8 total, 8 up, 8 in 2021-07-31T01:55:51.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:50 smithi045 conmon[33655]: cluster 2021-07-31T01:55:49.914192+0000 mgr.x (mgr.24542) 369 : cluster [DBG] pgmap v745: 25 pgs: 25 active+clean; 537 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 25 MiB/s rd, 5.9 MiB/s wr, 17 op/s 2021-07-31T01:55:51.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:50 smithi045 conmon[39718]: cluster 2021-07-31T01:55: 2021-07-31T01:55:51.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:50 smithi045 conmon[39718]: 49.674210+0000 mon.a (mon.0) 1303 : cluster [DBG] osdmap e640: 8 total, 8 up, 8 in 2021-07-31T01:55:51.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:50 smithi045 conmon[39718]: cluster 2021-07-31T01:55:49.914192 2021-07-31T01:55:51.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:50 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 369 : cluster [DBG] pgmap v745: 25 pgs: 25 active+clean; 537 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 25 MiB/s rd, 5.9 MiB/s wr, 17 op/s 2021-07-31T01:55:51.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:50 smithi045 conmon[39718]: 2021-07-31T01:55:52.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:51 smithi154 conmon[34553]: cluster 2021-07-31T01:55:50.681662+0000 mon.a (mon.0) 1304 : cluster [DBG] osdmap e641: 8 total, 8 up, 8 in 2021-07-31T01:55:52.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:51 smithi154 conmon[34553]: cluster 2021-07-31T01:55:51.637473+0000 mon.a (mon.0) 1305 : cluster [DBG] osdmap e642: 8 total, 8 up, 8 in 2021-07-31T01:55:52.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:51 smithi045 conmon[39718]: cluster 2021-07-31T01:55:50.681662+0000 2021-07-31T01:55:52.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:51 smithi045 conmon[39718]: mon.a (mon.0) 1304 : cluster [DBG] osdmap e641: 8 total, 8 up, 8 in 2021-07-31T01:55:52.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:51 smithi045 conmon[39718]: cluster 2021-07- 2021-07-31T01:55:52.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:51 smithi045 conmon[39718]: 31T01:55:51.637473+0000 mon.a (mon.0) 1305 : 2021-07-31T01:55:52.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:51 smithi045 conmon[39718]: cluster [DBG] osdmap e642: 8 total, 8 up, 8 in 2021-07-31T01:55:52.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:51 smithi045 conmon[33655]: cluster 2021-07-31T01:55:50. 2021-07-31T01:55:52.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:51 smithi045 conmon[33655]: 681662+0000 mon.a (mon.0) 1304 : cluster [DBG] osdmap e641: 8 total, 8 up, 8 in 2021-07-31T01:55:52.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:51 smithi045 conmon[33655]: cluster 2021- 2021-07-31T01:55:52.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:51 smithi045 conmon[33655]: 07-31T01:55:51.637473+0000 mon.a (mon. 2021-07-31T01:55:52.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:51 smithi045 conmon[33655]: 0) 1305 : cluster [DBG] osdmap e642: 8 total, 8 up, 8 in 2021-07-31T01:55:52.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:51 smithi045 conmon[33655]: 2021-07-31T01:55:52.135 INFO:tasks.rados.rados.0.smithi045.stdout:3123: finishing write tid 2 to smithi04569269-15 2021-07-31T01:55:52.136 INFO:tasks.rados.rados.0.smithi045.stdout:3123: finishing write tid 3 to smithi04569269-15 2021-07-31T01:55:52.136 INFO:tasks.rados.rados.0.smithi045.stdout:3123: finishing write tid 4 to smithi04569269-15 2021-07-31T01:55:52.136 INFO:tasks.rados.rados.0.smithi045.stdout:3123: finishing write tid 5 to smithi04569269-15 2021-07-31T01:55:52.137 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 15 v 1841 (ObjNum 966 snap 295 seq_num 966) dirty exists 2021-07-31T01:55:52.137 INFO:tasks.rados.rados.0.smithi045.stdout:3123: left oid 15 (ObjNum 966 snap 295 seq_num 966) 2021-07-31T01:55:52.137 INFO:tasks.rados.rados.0.smithi045.stdout:3115: done (4 left) 2021-07-31T01:55:52.138 INFO:tasks.rados.rados.0.smithi045.stdout:3119: done (3 left) 2021-07-31T01:55:52.138 INFO:tasks.rados.rados.0.smithi045.stdout:3123: done (2 left) 2021-07-31T01:55:52.138 INFO:tasks.rados.rados.0.smithi045.stdout:3128: rollback oid 43 current snap is 295 2021-07-31T01:55:52.138 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 43 to 279 2021-07-31T01:55:52.138 INFO:tasks.rados.rados.0.smithi045.stdout:3124: finishing write tid 1 to smithi04569269-12 2021-07-31T01:55:52.139 INFO:tasks.rados.rados.0.smithi045.stdout:3124: finishing write tid 2 to smithi04569269-12 2021-07-31T01:55:52.139 INFO:tasks.rados.rados.0.smithi045.stdout:3124: finishing write tid 3 to smithi04569269-12 2021-07-31T01:55:52.139 INFO:tasks.rados.rados.0.smithi045.stdout:3124: finishing write tid 4 to smithi04569269-12 2021-07-31T01:55:52.140 INFO:tasks.rados.rados.0.smithi045.stdout:3124: finishing write tid 5 to smithi04569269-12 2021-07-31T01:55:52.140 INFO:tasks.rados.rados.0.smithi045.stdout:3124: finishing write tid 6 to smithi04569269-12 2021-07-31T01:55:52.140 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 12 v 1864 (ObjNum 967 snap 295 seq_num 967) dirty exists 2021-07-31T01:55:52.140 INFO:tasks.rados.rados.0.smithi045.stdout:3124: left oid 12 (ObjNum 967 snap 295 seq_num 967) 2021-07-31T01:55:52.141 INFO:tasks.rados.rados.0.smithi045.stdout:3127: finishing rollback tid 0 to smithi04569269-37 2021-07-31T01:55:52.141 INFO:tasks.rados.rados.0.smithi045.stdout:3127: finishing rollback tid 1 to smithi04569269-37 2021-07-31T01:55:52.141 INFO:tasks.rados.rados.0.smithi045.stdout:3127: finishing rollback tid 2 to smithi04569269-37 2021-07-31T01:55:52.141 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 37 v 1844 (ObjNum 885 snap 272 seq_num 885) dirty exists 2021-07-31T01:55:52.142 INFO:tasks.rados.rados.0.smithi045.stdout:3124: done (2 left) 2021-07-31T01:55:52.142 INFO:tasks.rados.rados.0.smithi045.stdout:3127: done (1 left) 2021-07-31T01:55:52.142 INFO:tasks.rados.rados.0.smithi045.stdout:3129: read oid 49 snap -1 2021-07-31T01:55:52.142 INFO:tasks.rados.rados.0.smithi045.stdout:3129: expect (ObjNum 890 snap 272 seq_num 890) 2021-07-31T01:55:52.142 INFO:tasks.rados.rados.0.smithi045.stdout:3130: snap_create 2021-07-31T01:55:52.143 INFO:tasks.rados.rados.0.smithi045.stdout:3128: finishing rollback tid 0 to smithi04569269-43 2021-07-31T01:55:52.143 INFO:tasks.rados.rados.0.smithi045.stdout:3128: finishing rollback tid 1 to smithi04569269-43 2021-07-31T01:55:52.144 INFO:tasks.rados.rados.0.smithi045.stdout:3128: finishing rollback tid 2 to smithi04569269-43 2021-07-31T01:55:52.144 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 43 v 1847 (ObjNum 888 snap 272 seq_num 888) dirty exists 2021-07-31T01:55:52.179 INFO:tasks.rados.rados.0.smithi045.stdout:3129: expect (ObjNum 890 snap 272 seq_num 890) 2021-07-31T01:55:53.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:52 smithi154 conmon[34553]: cluster 2021-07-31T01:55:51.914675+0000 mgr.x (mgr.24542) 370 : cluster [DBG] 2021-07-31T01:55:53.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:52 smithi154 conmon[34553]: pgmap v748: 25 pgs: 25 active+clean; 537 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 25 MiB/s rd, 5.9 MiB/s wr, 17 op/s 2021-07-31T01:55:53.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:52 smithi045 conmon[39718]: cluster 2021-07-31T01:55: 2021-07-31T01:55:53.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:52 smithi045 conmon[39718]: 51.914675+0000 mgr.x (mgr.24542) 370 : cluster [DBG] pgmap v748: 25 pgs: 25 active+clean; 537 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 25 MiB/s rd, 5.9 MiB/s wr, 17 op/s 2021-07-31T01:55:53.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:52 smithi045 conmon[33655]: cluster 2021-07-31T01:55:51.914675+0000 2021-07-31T01:55:53.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:52 smithi045 conmon[33655]: mgr.x (mgr.24542) 370 : cluster [DBG] pgmap v748: 25 pgs: 25 active+clean; 537 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 25 MiB/s rd, 5.9 MiB/s wr, 17 op/s 2021-07-31T01:55:53.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:53 smithi045 conmon[39718]: cluster 2021-07-31T01:55: 2021-07-31T01:55:53.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:53 smithi045 conmon[39718]: 52.686470+0000 mon.a (mon.0) 1306 : cluster [DBG] osdmap e643: 8 total, 8 up, 8 in 2021-07-31T01:55:53.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:53 smithi045 conmon[33655]: cluster 2021-07-31T01:55:52.686470+0000 2021-07-31T01:55:53.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:53 smithi045 conmon[33655]: mon.a (mon.0) 1306 : cluster [DBG] osdmap e643: 8 total, 8 up, 8 in 2021-07-31T01:55:53.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:55:53 smithi045 conmon[61210]: level=error ts=2021-07-31T01:55:53.524Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:55:53.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:55:53 smithi045 conmon[61210]: level=error ts=2021-07-31T01:55:53.524Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:55:53.778 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:55:53 smithi045 conmon[61210]: level=error ts=2021-07-31T01:55:53.524Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T01:55:53.790 INFO:tasks.rados.rados.0.smithi045.stdout:3128: done (2 left) 2021-07-31T01:55:53.791 INFO:tasks.rados.rados.0.smithi045.stdout:3129: done (1 left) 2021-07-31T01:55:53.791 INFO:tasks.rados.rados.0.smithi045.stdout:3130: done (0 left) 2021-07-31T01:55:53.791 INFO:tasks.rados.rados.0.smithi045.stdout:3131: rollback oid 31 current snap is 296 2021-07-31T01:55:53.791 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 31 to 290 2021-07-31T01:55:53.792 INFO:tasks.rados.rados.0.smithi045.stdout:3132: read oid 13 snap 266 2021-07-31T01:55:53.792 INFO:tasks.rados.rados.0.smithi045.stdout:3132: expect deleted 2021-07-31T01:55:53.792 INFO:tasks.rados.rados.0.smithi045.stdout:3133: delete oid 29 current snap is 296 2021-07-31T01:55:53.793 INFO:tasks.rados.rados.0.smithi045.stdout:3132: done (2 left) 2021-07-31T01:55:53.793 INFO:tasks.rados.rados.0.smithi045.stdout:3133: done (1 left) 2021-07-31T01:55:53.793 INFO:tasks.rados.rados.0.smithi045.stdout:3134: copy_from oid 48 from oid 44 current snap is 296 2021-07-31T01:55:53.793 INFO:tasks.rados.rados.0.smithi045.stdout:3135: delete oid 14 current snap is 296 2021-07-31T01:55:53.794 INFO:tasks.rados.rados.0.smithi045.stdout:3131: finishing rollback tid 0 to smithi04569269-31 2021-07-31T01:55:53.794 INFO:tasks.rados.rados.0.smithi045.stdout:3131: finishing rollback tid 1 to smithi04569269-31 2021-07-31T01:55:53.795 INFO:tasks.rados.rados.0.smithi045.stdout:3131: finishing rollback tid 2 to smithi04569269-31 2021-07-31T01:55:53.795 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 31 v 2006 (ObjNum 813 snap 254 seq_num 813) dirty exists 2021-07-31T01:55:53.798 INFO:tasks.rados.rados.0.smithi045.stdout:3131: done (2 left) 2021-07-31T01:55:53.798 INFO:tasks.rados.rados.0.smithi045.stdout:3135: done (1 left) 2021-07-31T01:55:53.799 INFO:tasks.rados.rados.0.smithi045.stdout:3136: snap_create 2021-07-31T01:55:53.812 INFO:tasks.rados.rados.0.smithi045.stdout:3134: finishing copy_from to smithi04569269-48 2021-07-31T01:55:53.813 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 48 v 1843 (ObjNum 955 snap 289 seq_num 955) dirty exists 2021-07-31T01:55:53.813 INFO:tasks.rados.rados.0.smithi045.stdout:3134: finishing copy_from racing read to smithi04569269-48 2021-07-31T01:55:54.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:53 smithi154 conmon[34553]: cluster 2021-07-31T01:55:52.686470+0000 mon.a (mon.0) 1306 : cluster [DBG] 2021-07-31T01:55:54.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:53 smithi154 conmon[34553]: osdmap e643: 8 total, 8 up, 8 in 2021-07-31T01:55:54.696 INFO:tasks.rados.rados.0.smithi045.stdout:3134: done (1 left) 2021-07-31T01:55:54.696 INFO:tasks.rados.rados.0.smithi045.stdout:3136: done (0 left) 2021-07-31T01:55:54.697 INFO:tasks.rados.rados.0.smithi045.stdout:3137: snap_remove snap 274 2021-07-31T01:55:55.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:54 smithi154 conmon[34553]: cluster 2021-07-31T01:55:53.915355+0000 mgr.x (mgr.24542 2021-07-31T01:55:55.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:54 smithi154 conmon[34553]: ) 371 : cluster [DBG] pgmap v750: 25 pgs: 25 active+clean; 541 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 10 MiB/s rd, 1.2 MiB/s wr, 4 op/s 2021-07-31T01:55:55.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:54 smithi045 conmon[33655]: cluster 2021-07-31T01:55:53.915355+0000 mgr.x (mgr.24542) 371 : 2021-07-31T01:55:55.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:54 smithi045 conmon[33655]: cluster [DBG] pgmap v750: 25 pgs: 25 active+clean; 541 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 10 MiB/s rd, 1.2 MiB/s wr, 4 op/s 2021-07-31T01:55:55.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:54 smithi045 conmon[39718]: cluster 2021-07-31T01:55:53. 2021-07-31T01:55:55.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:54 smithi045 conmon[39718]: 915355+0000 mgr.x (mgr.24542) 371 : cluster [DBG] pgmap v750: 25 pgs: 25 active+clean; 541 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 10 MiB/s rd, 1.2 MiB/s wr, 4 op/s 2021-07-31T01:55:55.698 INFO:tasks.rados.rados.0.smithi045.stdout:3137: done (0 left) 2021-07-31T01:55:55.699 INFO:tasks.rados.rados.0.smithi045.stdout:3138: read oid 8 snap -1 2021-07-31T01:55:55.699 INFO:tasks.rados.rados.0.smithi045.stdout:3138: expect (ObjNum 813 snap 254 seq_num 813) 2021-07-31T01:55:55.699 INFO:tasks.rados.rados.0.smithi045.stdout:3139: copy_from oid 2 from oid 5 current snap is 297 2021-07-31T01:55:55.700 INFO:tasks.rados.rados.0.smithi045.stdout:3140: copy_from oid 15 from oid 11 current snap is 297 2021-07-31T01:55:55.700 INFO:tasks.rados.rados.0.smithi045.stdout:3141: delete oid 22 current snap is 297 2021-07-31T01:55:55.713 INFO:tasks.rados.rados.0.smithi045.stdout:3141: done (3 left) 2021-07-31T01:55:55.714 INFO:tasks.rados.rados.0.smithi045.stdout:3142: rollback oid 24 current snap is 297 2021-07-31T01:55:55.714 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 24 to 285 2021-07-31T01:55:55.714 INFO:tasks.rados.rados.0.smithi045.stdout:3143: rollback oid 1 current snap is 297 2021-07-31T01:55:55.714 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 1 to 248 2021-07-31T01:55:55.715 INFO:tasks.rados.rados.0.smithi045.stdout:3144: delete oid 7 current snap is 297 2021-07-31T01:55:55.718 INFO:tasks.rados.rados.0.smithi045.stdout:3139: finishing copy_from to smithi04569269-2 2021-07-31T01:55:55.719 INFO:tasks.rados.rados.0.smithi045.stdout:3139: got expected ENOENT (src dne) 2021-07-31T01:55:55.720 INFO:tasks.rados.rados.0.smithi045.stdout:3143: finishing rollback tid 0 to smithi04569269-1 2021-07-31T01:55:55.721 INFO:tasks.rados.rados.0.smithi045.stdout:3144: done (5 left) 2021-07-31T01:55:55.721 INFO:tasks.rados.rados.0.smithi045.stdout:3145: copy_from oid 46 from oid 48 current snap is 297 2021-07-31T01:55:55.721 INFO:tasks.rados.rados.0.smithi045.stdout:3139: finishing copy_from racing read to smithi04569269-2 2021-07-31T01:55:55.722 INFO:tasks.rados.rados.0.smithi045.stdout:3139: done (5 left) 2021-07-31T01:55:55.722 INFO:tasks.rados.rados.0.smithi045.stdout:3146: snap_create 2021-07-31T01:55:55.722 INFO:tasks.rados.rados.0.smithi045.stdout:3138: expect (ObjNum 813 snap 254 seq_num 813) 2021-07-31T01:55:55.776 INFO:tasks.rados.rados.0.smithi045.stdout:3143: finishing rollback tid 1 to smithi04569269-1 2021-07-31T01:55:55.776 INFO:tasks.rados.rados.0.smithi045.stdout:3143: finishing rollback tid 2 to smithi04569269-1 2021-07-31T01:55:55.776 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 1 v 1846 (ObjNum 713 snap 222 seq_num 713) dirty exists 2021-07-31T01:55:55.777 INFO:tasks.rados.rados.0.smithi045.stdout:3140: finishing copy_from to smithi04569269-15 2021-07-31T01:55:55.777 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 15 v 1662 (ObjNum 920 snap 278 seq_num 920) dirty exists 2021-07-31T01:55:55.777 INFO:tasks.rados.rados.0.smithi045.stdout:3140: finishing copy_from racing read to smithi04569269-15 2021-07-31T01:55:55.777 INFO:tasks.rados.rados.0.smithi045.stdout:3142: finishing rollback tid 0 to smithi04569269-24 2021-07-31T01:55:55.778 INFO:tasks.rados.rados.0.smithi045.stdout:3142: finishing rollback tid 1 to smithi04569269-24 2021-07-31T01:55:55.778 INFO:tasks.rados.rados.0.smithi045.stdout:3142: finishing rollback tid 2 to smithi04569269-24 2021-07-31T01:55:55.778 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 24 v 1649 (ObjNum 1 snap 32731 seq_num 3189903696) dirty exists 2021-07-31T01:55:55.778 INFO:tasks.rados.rados.0.smithi045.stdout:3145: finishing copy_from to smithi04569269-46 2021-07-31T01:55:55.778 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 46 v 1843 (ObjNum 955 snap 289 seq_num 955) dirty exists 2021-07-31T01:55:55.779 INFO:tasks.rados.rados.0.smithi045.stdout:3145: finishing copy_from racing read to smithi04569269-46 2021-07-31T01:55:56.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:55 smithi154 conmon[34553]: cluster 2021-07-31T01:55:54.695000+0000 mon.a (mon.0) 2021-07-31T01:55:56.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:55 smithi154 conmon[34553]: 1307 : cluster [DBG] osdmap e644: 8 total, 8 up, 8 in 2021-07-31T01:55:56.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:55 smithi045 conmon[33655]: cluster 2021-07-31T01:55:56.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:55 smithi045 conmon[33655]: 2021-07-31T01:55:56.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:55 smithi045 conmon[33655]: 2021 2021-07-31T01:55:56.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:55 smithi045 conmon[33655]: - 2021-07-31T01:55:56.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:55 smithi045 conmon[33655]: 07 2021-07-31T01:55:56.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:55 smithi045 conmon[33655]: - 2021-07-31T01:55:56.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:55 smithi045 conmon[33655]: 31 2021-07-31T01:55:56.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:55 smithi045 conmon[33655]: T 2021-07-31T01:55:56.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:55 smithi045 conmon[33655]: 01 2021-07-31T01:55:56.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:55 smithi045 conmon[33655]: :55:54.695000+0000 2021-07-31T01:55:56.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:55 smithi045 conmon[33655]: mon.a (mon.0) 1307 : cluster [DBG] osdmap e644: 8 total, 8 up, 8 in 2021-07-31T01:55:56.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:55 smithi045 conmon[39718]: cluster 2021-07-31T01:55:56.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:55 smithi045 conmon[39718]: 2021-07-31T01:55:56.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:55 smithi045 conmon[39718]: 2021 2021-07-31T01:55:56.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:55 smithi045 conmon[39718]: - 2021-07-31T01:55:56.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:55 smithi045 conmon[39718]: 07 2021-07-31T01:55:56.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:55 smithi045 conmon[39718]: - 2021-07-31T01:55:56.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:55 smithi045 conmon[39718]: 31 2021-07-31T01:55:56.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:55 smithi045 conmon[39718]: T 2021-07-31T01:55:56.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:55 smithi045 conmon[39718]: 01 2021-07-31T01:55:56.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:55 smithi045 conmon[39718]: : 2021-07-31T01:55:56.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:55 smithi045 conmon[39718]: 55 2021-07-31T01:55:56.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:55 smithi045 conmon[39718]: : 2021-07-31T01:55:56.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:55 smithi045 conmon[39718]: 54 2021-07-31T01:55:56.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:55 smithi045 conmon[39718]: . 2021-07-31T01:55:56.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:55 smithi045 conmon[39718]: 695000 2021-07-31T01:55:56.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:55 smithi045 conmon[39718]: +0000 2021-07-31T01:55:56.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:55 smithi045 conmon[39718]: 2021-07-31T01:55:56.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:55 smithi045 conmon[39718]: mon.a 2021-07-31T01:55:56.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:55 smithi045 conmon[39718]: ( 2021-07-31T01:55:56.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:55 smithi045 conmon[39718]: mon 2021-07-31T01:55:56.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:55 smithi045 conmon[39718]: . 2021-07-31T01:55:56.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:55 smithi045 conmon[39718]: 0 2021-07-31T01:55:56.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:55 smithi045 conmon[39718]: ) 2021-07-31T01:55:56.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:55 smithi045 conmon[39718]: 1307 : cluster [DBG] osdmap e644: 8 total, 8 up, 8 in 2021-07-31T01:55:56.640 INFO:tasks.rados.rados.0.smithi045.stdout:3138: done (5 left) 2021-07-31T01:55:56.641 INFO:tasks.rados.rados.0.smithi045.stdout:3140: done (4 left) 2021-07-31T01:55:56.641 INFO:tasks.rados.rados.0.smithi045.stdout:3142: done (3 left) 2021-07-31T01:55:56.641 INFO:tasks.rados.rados.0.smithi045.stdout:3143: done (2 left) 2021-07-31T01:55:56.641 INFO:tasks.rados.rados.0.smithi045.stdout:3145: done (1 left) 2021-07-31T01:55:56.641 INFO:tasks.rados.rados.0.smithi045.stdout:3146: done (0 left) 2021-07-31T01:55:56.642 INFO:tasks.rados.rados.0.smithi045.stdout:3147: delete oid 13 current snap is 298 2021-07-31T01:55:56.646 INFO:tasks.rados.rados.0.smithi045.stdout:3147: done (0 left) 2021-07-31T01:55:56.646 INFO:tasks.rados.rados.0.smithi045.stdout:3148: delete oid 2 current snap is 298 2021-07-31T01:55:56.648 INFO:tasks.rados.rados.0.smithi045.stdout:3148: done (0 left) 2021-07-31T01:55:56.648 INFO:tasks.rados.rados.0.smithi045.stdout:3149: snap_create 2021-07-31T01:55:57.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:56 smithi154 conmon[34553]: cluster 2021-07-31T01:55:55.696701+0000 mon.a (mon.0) 1308 : cluster [DBG] osdmap e645: 8 total, 8 up, 8 in 2021-07-31T01:55:57.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:56 smithi154 conmon[34553]: cluster 2021-07-31T01:55:55.915957+0000 mgr.x (mgr.24542) 372 : cluster [DBG] pgmap v753: 25 pgs: 25 active+clean; 541 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 9.9 MiB/s rd, 1.2 MiB/s wr, 4 op/s 2021-07-31T01:55:57.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:56 smithi154 conmon[34553]: cluster 2021-07-31T01:55:56.639243+0000 mon.a (mon.0) 1309 : cluster [DBG] osdmap e646: 8 total, 8 up, 8 in 2021-07-31T01:55:57.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:56 smithi045 conmon[39718]: cluster 2021-07-31T01:55:55.696701+0000 mon.a 2021-07-31T01:55:57.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:56 smithi045 conmon[39718]: (mon.0) 1308 : cluster [DBG] osdmap e645: 8 total, 8 up, 8 in 2021-07-31T01:55:57.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:56 smithi045 conmon[39718]: cluster 2021-07-31T 2021-07-31T01:55:57.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:56 smithi045 conmon[39718]: 01:55:55.915957+0000 mgr.x (mgr.24542) 372 : cluster 2021-07-31T01:55:57.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:56 smithi045 conmon[39718]: [DBG] pgmap v753: 25 pgs: 25 active+clean; 541 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 9.9 MiB/s rd, 1.2 MiB/s wr, 4 op/s 2021-07-31T01:55:57.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:56 smithi045 conmon[39718]: cluster 2021-07-31T01:55:57.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:56 smithi045 conmon[39718]: 2021-07-31 2021-07-31T01:55:57.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:56 smithi045 conmon[39718]: T01:55:56.639243+0000 2021-07-31T01:55:57.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:56 smithi045 conmon[39718]: mon.a (mon.0) 1309 2021-07-31T01:55:57.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:56 smithi045 conmon[39718]: : cluster [DBG] osdmap e646: 8 total, 8 up, 8 in 2021-07-31T01:55:57.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:56 smithi045 conmon[33655]: cluster 2021-07-31T01:55:55.696701+0000 mon.a (mon.0) 1308 : cluster [DBG] osdmap e645: 8 total, 8 up, 8 in 2021-07-31T01:55:57.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:56 smithi045 conmon[33655]: cluster 2021-07-31T01:55:55.915957+0000 mgr.x (mgr.24542) 372 : cluster [DBG] pgmap v753: 25 pgs: 25 active+clean; 541 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 9.9 MiB/s rd, 1.2 MiB/s wr, 4 op/s 2021-07-31T01:55:57.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:56 smithi045 conmon[33655]: cluster 2021-07-31T01:55:56.639243+0000 mon.a (mon.0) 1309 : cluster [DBG] osdmap e646: 8 total, 8 up, 8 in 2021-07-31T01:55:57.647 INFO:tasks.rados.rados.0.smithi045.stdout:3149: done (0 left) 2021-07-31T01:55:57.647 INFO:tasks.rados.rados.0.smithi045.stdout:3150: read oid 49 snap -1 2021-07-31T01:55:57.647 INFO:tasks.rados.rados.0.smithi045.stdout:3150: expect (ObjNum 890 snap 272 seq_num 890) 2021-07-31T01:55:57.648 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 37 current snap is 299 2021-07-31T01:55:57.648 INFO:tasks.rados.rados.0.smithi045.stdout:3151: seq_num 974 ranges {11583488=720896,12304384=507904,12812288=671744,13484032=24576} 2021-07-31T01:55:57.715 INFO:tasks.rados.rados.0.smithi045.stdout:3151: writing smithi04569269-37 from 11583488 to 12304384 tid 1 2021-07-31T01:55:57.717 INFO:tasks.rados.rados.0.smithi045.stdout:3151: writing smithi04569269-37 from 12304384 to 12812288 tid 2 2021-07-31T01:55:57.722 INFO:tasks.rados.rados.0.smithi045.stdout:3151: writing smithi04569269-37 from 12812288 to 13484032 tid 3 2021-07-31T01:55:57.722 INFO:tasks.rados.rados.0.smithi045.stdout:3151: writing smithi04569269-37 from 13484032 to 13508608 tid 4 2021-07-31T01:55:57.722 INFO:tasks.rados.rados.0.smithi045.stdout:3152: copy_from oid 14 from oid 27 current snap is 299 2021-07-31T01:55:57.722 INFO:tasks.rados.rados.0.smithi045.stdout:3150: expect (ObjNum 890 snap 272 seq_num 890) 2021-07-31T01:55:59.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:58 smithi154 conmon[34553]: cluster 2021-07-31T01:55:57.645915+0000 mon.a (mon.0) 1310 : cluster [DBG] 2021-07-31T01:55:59.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:58 smithi154 conmon[34553]: osdmap e647: 8 total, 8 up, 8 in 2021-07-31T01:55:59.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:55:58 smithi154 conmon[34553]: cluster 2021-07-31T01:55:57.916382+0000 mgr.x (mgr.24542) 373 : cluster [DBG] pgmap v756: 25 pgs: 25 active+clean; 541 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:55:59.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:58 smithi045 conmon[39718]: cluster 2021-07-31T01: 2021-07-31T01:55:59.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:58 smithi045 conmon[39718]: 55:57.645915+0000 mon.a (mon.0) 1310 : cluster [DBG] osdmap e647: 8 total, 8 up, 8 in 2021-07-31T01:55:59.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:58 smithi045 conmon[39718]: cluster 2021-07 2021-07-31T01:55:59.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:58 smithi045 conmon[39718]: -31T01:55:57.916382+0000 2021-07-31T01:55:59.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:58 smithi045 conmon[39718]: mgr.x (mgr.24542) 373 : cluster [DBG] pgmap v756: 25 pgs: 25 active+clean; 541 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:55:59.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:55:58 smithi045 conmon[39718]: 2021-07-31T01:55:59.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:58 smithi045 conmon[33655]: cluster 2021-07-31T01:55:57.645915+0000 mon.a (mon.0) 1310 : cluster 2021-07-31T01:55:59.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:58 smithi045 conmon[33655]: [DBG] osdmap e647: 8 total, 8 up, 8 in 2021-07-31T01:55:59.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:58 smithi045 conmon[33655]: cluster 2021-07-31T01:55:57.916382+0000 mgr.x (mgr.24542) 373 : cluster 2021-07-31T01:55:59.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:55:58 smithi045 conmon[33655]: [DBG] pgmap v756: 25 pgs: 25 active+clean; 541 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:55:59.333 INFO:tasks.rados.rados.0.smithi045.stdout:3151: finishing write tid 1 to smithi04569269-37 2021-07-31T01:55:59.333 INFO:tasks.rados.rados.0.smithi045.stdout:3151: finishing write tid 2 to smithi04569269-37 2021-07-31T01:55:59.334 INFO:tasks.rados.rados.0.smithi045.stdout:3152: finishing copy_from to smithi04569269-14 2021-07-31T01:55:59.334 INFO:tasks.rados.rados.0.smithi045.stdout:3152: got expected ENOENT (src dne) 2021-07-31T01:55:59.334 INFO:tasks.rados.rados.0.smithi045.stdout:3152: finishing copy_from racing read to smithi04569269-14 2021-07-31T01:55:59.334 INFO:tasks.rados.rados.0.smithi045.stdout:3150: done (2 left) 2021-07-31T01:55:59.335 INFO:tasks.rados.rados.0.smithi045.stdout:3152: done (1 left) 2021-07-31T01:55:59.335 INFO:tasks.rados.rados.0.smithi045.stdout:3153: read oid 29 snap 286 2021-07-31T01:55:59.335 INFO:tasks.rados.rados.0.smithi045.stdout:3153: expect deleted 2021-07-31T01:55:59.335 INFO:tasks.rados.rados.0.smithi045.stdout:3154: read oid 14 snap -1 2021-07-31T01:55:59.335 INFO:tasks.rados.rados.0.smithi045.stdout:3154: expect deleted 2021-07-31T01:55:59.336 INFO:tasks.rados.rados.0.smithi045.stdout:3155: delete oid 35 current snap is 299 2021-07-31T01:55:59.336 INFO:tasks.rados.rados.0.smithi045.stdout:3151: finishing write tid 3 to smithi04569269-37 2021-07-31T01:55:59.336 INFO:tasks.rados.rados.0.smithi045.stdout:3151: finishing write tid 4 to smithi04569269-37 2021-07-31T01:55:59.336 INFO:tasks.rados.rados.0.smithi045.stdout:3151: finishing write tid 5 to smithi04569269-37 2021-07-31T01:55:59.337 INFO:tasks.rados.rados.0.smithi045.stdout:3151: finishing write tid 6 to smithi04569269-37 2021-07-31T01:55:59.337 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 37 v 1849 (ObjNum 974 snap 299 seq_num 974) dirty exists 2021-07-31T01:55:59.337 INFO:tasks.rados.rados.0.smithi045.stdout:3151: left oid 37 (ObjNum 974 snap 299 seq_num 974) 2021-07-31T01:55:59.338 INFO:tasks.rados.rados.0.smithi045.stdout:3151: done (3 left) 2021-07-31T01:55:59.338 INFO:tasks.rados.rados.0.smithi045.stdout:3153: done (2 left) 2021-07-31T01:55:59.339 INFO:tasks.rados.rados.0.smithi045.stdout:3154: done (1 left) 2021-07-31T01:55:59.339 INFO:tasks.rados.rados.0.smithi045.stdout:3155: done (0 left) 2021-07-31T01:55:59.339 INFO:tasks.rados.rados.0.smithi045.stdout:3156: rmattr oid 46 current snap is 299 2021-07-31T01:55:59.339 INFO:tasks.rados.rados.0.smithi045.stdout:3156: done (0 left) 2021-07-31T01:55:59.339 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 11 current snap is 299 2021-07-31T01:55:59.340 INFO:tasks.rados.rados.0.smithi045.stdout:3157: seq_num 976 ranges {1015808=753664,1769472=671744,2441216=532480} 2021-07-31T01:55:59.345 INFO:tasks.rados.rados.0.smithi045.stdout:3157: writing smithi04569269-11 from 1015808 to 1769472 tid 1 2021-07-31T01:55:59.349 INFO:tasks.rados.rados.0.smithi045.stdout:3157: writing smithi04569269-11 from 1769472 to 2441216 tid 2 2021-07-31T01:55:59.352 INFO:tasks.rados.rados.0.smithi045.stdout:3157: writing smithi04569269-11 from 2441216 to 2973696 tid 3 2021-07-31T01:55:59.352 INFO:tasks.rados.rados.0.smithi045.stdout:3158: rollback oid 13 current snap is 299 2021-07-31T01:55:59.353 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 13 to 294 2021-07-31T01:55:59.354 INFO:tasks.rados.rados.0.smithi045.stdout:3157: finishing write tid 1 to smithi04569269-11 2021-07-31T01:55:59.354 INFO:tasks.rados.rados.0.smithi045.stdout:3159: rmattr oid 6 current snap is 299 2021-07-31T01:55:59.354 INFO:tasks.rados.rados.0.smithi045.stdout:3159: done (2 left) 2021-07-31T01:55:59.354 INFO:tasks.rados.rados.0.smithi045.stdout:3160: read oid 19 snap -1 2021-07-31T01:55:59.355 INFO:tasks.rados.rados.0.smithi045.stdout:3160: expect deleted 2021-07-31T01:55:59.355 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 48 current snap is 299 2021-07-31T01:55:59.355 INFO:tasks.rados.rados.0.smithi045.stdout:3161: seq_num 977 ranges {3391488=770048,4161536=557056,4718592=491520,5210112=155648} 2021-07-31T01:55:59.374 INFO:tasks.rados.rados.0.smithi045.stdout:3161: writing smithi04569269-48 from 3391488 to 4161536 tid 1 2021-07-31T01:55:59.377 INFO:tasks.rados.rados.0.smithi045.stdout:3161: writing smithi04569269-48 from 4161536 to 4718592 tid 2 2021-07-31T01:55:59.382 INFO:tasks.rados.rados.0.smithi045.stdout:3161: writing smithi04569269-48 from 4718592 to 5210112 tid 3 2021-07-31T01:55:59.383 INFO:tasks.rados.rados.0.smithi045.stdout:3161: writing smithi04569269-48 from 5210112 to 5365760 tid 4 2021-07-31T01:55:59.383 INFO:tasks.rados.rados.0.smithi045.stdout:3162: snap_create 2021-07-31T01:55:59.385 INFO:tasks.rados.rados.0.smithi045.stdout:3157: finishing write tid 2 to smithi04569269-11 2021-07-31T01:55:59.385 INFO:tasks.rados.rados.0.smithi045.stdout:3158: finishing rollback tid 1 to smithi04569269-13 2021-07-31T01:55:59.386 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 13 v 1865 (ObjNum 1953066355 snap 875587944 seq_num 842610229) dirty dne 2021-07-31T01:55:59.386 INFO:tasks.rados.rados.0.smithi045.stdout:3157: finishing write tid 3 to smithi04569269-11 2021-07-31T01:55:59.386 INFO:tasks.rados.rados.0.smithi045.stdout:3157: finishing write tid 4 to smithi04569269-11 2021-07-31T01:55:59.386 INFO:tasks.rados.rados.0.smithi045.stdout:3157: finishing write tid 5 to smithi04569269-11 2021-07-31T01:55:59.386 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 11 v 1686 (ObjNum 976 snap 299 seq_num 976) dirty exists 2021-07-31T01:55:59.387 INFO:tasks.rados.rados.0.smithi045.stdout:3157: left oid 11 (ObjNum 976 snap 299 seq_num 976) 2021-07-31T01:55:59.387 INFO:tasks.rados.rados.0.smithi045.stdout:3161: finishing write tid 1 to smithi04569269-48 2021-07-31T01:55:59.387 INFO:tasks.rados.rados.0.smithi045.stdout:3161: finishing write tid 2 to smithi04569269-48 2021-07-31T01:55:59.388 INFO:tasks.rados.rados.0.smithi045.stdout:3161: finishing write tid 3 to smithi04569269-48 2021-07-31T01:55:59.388 INFO:tasks.rados.rados.0.smithi045.stdout:3161: finishing write tid 4 to smithi04569269-48 2021-07-31T01:55:59.389 INFO:tasks.rados.rados.0.smithi045.stdout:3161: finishing write tid 5 to smithi04569269-48 2021-07-31T01:55:59.389 INFO:tasks.rados.rados.0.smithi045.stdout:3161: finishing write tid 6 to smithi04569269-48 2021-07-31T01:55:59.389 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 48 v 1851 (ObjNum 977 snap 299 seq_num 977) dirty exists 2021-07-31T01:55:59.389 INFO:tasks.rados.rados.0.smithi045.stdout:3161: left oid 48 (ObjNum 977 snap 299 seq_num 977) 2021-07-31T01:55:59.652 INFO:tasks.rados.rados.0.smithi045.stdout:3157: done (4 left) 2021-07-31T01:55:59.652 INFO:tasks.rados.rados.0.smithi045.stdout:3158: done (3 left) 2021-07-31T01:55:59.653 INFO:tasks.rados.rados.0.smithi045.stdout:3160: done (2 left) 2021-07-31T01:55:59.653 INFO:tasks.rados.rados.0.smithi045.stdout:3161: done (1 left) 2021-07-31T01:55:59.653 INFO:tasks.rados.rados.0.smithi045.stdout:3162: done (0 left) 2021-07-31T01:55:59.653 INFO:tasks.rados.rados.0.smithi045.stdout:3163: snap_create 2021-07-31T01:56:00.658 INFO:tasks.rados.rados.0.smithi045.stdout:3163: done (0 left) 2021-07-31T01:56:00.659 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 32 current snap is 301 2021-07-31T01:56:00.659 INFO:tasks.rados.rados.0.smithi045.stdout:3164: seq_num 978 ranges {9011200=786432,9797632=442368,10240000=647168,10887168=114688} 2021-07-31T01:56:00.711 INFO:tasks.rados.rados.0.smithi045.stdout:3164: writing smithi04569269-32 from 9011200 to 9797632 tid 1 2021-07-31T01:56:00.714 INFO:tasks.rados.rados.0.smithi045.stdout:3164: writing smithi04569269-32 from 9797632 to 10240000 tid 2 2021-07-31T01:56:00.717 INFO:tasks.rados.rados.0.smithi045.stdout:3164: writing smithi04569269-32 from 10240000 to 10887168 tid 3 2021-07-31T01:56:00.718 INFO:tasks.rados.rados.0.smithi045.stdout:3164: writing smithi04569269-32 from 10887168 to 11001856 tid 4 2021-07-31T01:56:00.719 INFO:tasks.rados.rados.0.smithi045.stdout:3165: read oid 50 snap 286 2021-07-31T01:56:00.719 INFO:tasks.rados.rados.0.smithi045.stdout:3165: expect (ObjNum 837 snap 263 seq_num 837) 2021-07-31T01:56:00.719 INFO:tasks.rados.rados.0.smithi045.stdout:3166: copy_from oid 45 from oid 16 current snap is 301 2021-07-31T01:56:00.719 INFO:tasks.rados.rados.0.smithi045.stdout:3164: finishing write tid 1 to smithi04569269-32 2021-07-31T01:56:00.719 INFO:tasks.rados.rados.0.smithi045.stdout:3164: finishing write tid 2 to smithi04569269-32 2021-07-31T01:56:00.720 INFO:tasks.rados.rados.0.smithi045.stdout:3167: snap_create 2021-07-31T01:56:00.722 INFO:tasks.rados.rados.0.smithi045.stdout:3164: finishing write tid 3 to smithi04569269-32 2021-07-31T01:56:00.726 INFO:tasks.rados.rados.0.smithi045.stdout:3164: finishing write tid 4 to smithi04569269-32 2021-07-31T01:56:00.726 INFO:tasks.rados.rados.0.smithi045.stdout:3164: finishing write tid 5 to smithi04569269-32 2021-07-31T01:56:00.727 INFO:tasks.rados.rados.0.smithi045.stdout:3164: finishing write tid 6 to smithi04569269-32 2021-07-31T01:56:00.727 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 32 v 1843 (ObjNum 978 snap 301 seq_num 978) dirty exists 2021-07-31T01:56:00.728 INFO:tasks.rados.rados.0.smithi045.stdout:3164: left oid 32 (ObjNum 978 snap 301 seq_num 978) 2021-07-31T01:56:00.729 INFO:tasks.rados.rados.0.smithi045.stdout:3165: expect (ObjNum 837 snap 263 seq_num 837) 2021-07-31T01:56:00.783 INFO:tasks.rados.rados.0.smithi045.stdout:3166: finishing copy_from to smithi04569269-45 2021-07-31T01:56:00.783 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 45 v 1650 (ObjNum 944 snap 286 seq_num 944) dirty exists 2021-07-31T01:56:00.783 INFO:tasks.rados.rados.0.smithi045.stdout:3166: finishing copy_from racing read to smithi04569269-45 2021-07-31T01:56:01.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:00 smithi154 conmon[34553]: cluster 2021-07-31T01:56:01.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:00 smithi154 conmon[34553]: 2021-07-31T01:55:59 2021-07-31T01:56:01.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:00 smithi154 conmon[34553]: .652202+0000 mon.a (mon.0) 1311 : cluster [DBG] osdmap e648: 8 total, 8 up, 8 in 2021-07-31T01:56:01.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:00 smithi154 conmon[34553]: cluster 2021-07-31T01:55:59.917084+0000 mgr.x 2021-07-31T01:56:01.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:00 smithi154 conmon[34553]: (mgr.24542) 374 : cluster [DBG] pgmap v758: 25 pgs: 25 active+clean; 547 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 1.4 MiB/s wr, 6 op/s 2021-07-31T01:56:01.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:00 smithi045 conmon[39718]: cluster 2021-07-31T01:55:59.652202+0000 mon.a (mon 2021-07-31T01:56:01.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:00 smithi045 conmon[39718]: .0) 1311 : cluster [DBG] osdmap e648: 8 total, 8 up, 8 in 2021-07-31T01:56:01.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:00 smithi045 conmon[39718]: cluster 2021-07-31T01:55:59 2021-07-31T01:56:01.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:00 smithi045 conmon[39718]: .917084+0000 mgr.x (mgr.24542) 374 : cluster [DBG] pgmap v758: 25 pgs: 25 active+clean; 547 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 1.4 MiB/s wr, 6 op/s 2021-07-31T01:56:01.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:00 smithi045 conmon[33655]: cluster 2021-07-31T01:55:59.652202+0000 mon.a (mon.0) 1311 : 2021-07-31T01:56:01.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:00 smithi045 conmon[33655]: cluster [DBG] osdmap e648: 8 total, 8 up, 8 in 2021-07-31T01:56:01.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:00 smithi045 conmon[33655]: cluster 2021-07-31T01:55:59.917084+0000 mgr.x (mgr.24542) 374 : cluster [DBG] pgmap v758: 25 pgs: 25 active+clean; 547 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 1.4 MiB/s wr, 6 op/s 2021-07-31T01:56:01.641 INFO:tasks.rados.rados.0.smithi045.stdout:3164: done (3 left) 2021-07-31T01:56:01.642 INFO:tasks.rados.rados.0.smithi045.stdout:3165: done (2 left) 2021-07-31T01:56:01.643 INFO:tasks.rados.rados.0.smithi045.stdout:3166: done (1 left) 2021-07-31T01:56:01.643 INFO:tasks.rados.rados.0.smithi045.stdout:3167: done (0 left) 2021-07-31T01:56:01.643 INFO:tasks.rados.rados.0.smithi045.stdout:3168: read oid 34 snap -1 2021-07-31T01:56:01.643 INFO:tasks.rados.rados.0.smithi045.stdout:3168: expect deleted 2021-07-31T01:56:01.643 INFO:tasks.rados.rados.0.smithi045.stdout:3169: copy_from oid 11 from oid 38 current snap is 302 2021-07-31T01:56:01.644 INFO:tasks.rados.rados.0.smithi045.stdout:3170: snap_create 2021-07-31T01:56:01.645 INFO:tasks.rados.rados.0.smithi045.stdout:3169: finishing copy_from to smithi04569269-11 2021-07-31T01:56:01.646 INFO:tasks.rados.rados.0.smithi045.stdout:3169: got expected ENOENT (src dne) 2021-07-31T01:56:01.646 INFO:tasks.rados.rados.0.smithi045.stdout:3169: finishing copy_from racing read to smithi04569269-11 2021-07-31T01:56:02.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:01 smithi154 conmon[34553]: cluster 2021-07-31T01:56:00.657399+0000 mon.a (mon.0) 1312 : cluster [DBG] osdmap e649: 8 total, 8 up, 8 in 2021-07-31T01:56:02.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:01 smithi154 conmon[34553]: cluster 2021-07-31T01:56:01.640568+0000 mon.a (mon.0) 1313 : cluster [DBG] osdmap e650: 8 total, 8 up, 8 in 2021-07-31T01:56:02.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:01 smithi045 conmon[39718]: cluster 2021-07-31T01:56:00.657399+0000 mon.a (mon.0) 2021-07-31T01:56:02.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:01 smithi045 conmon[39718]: 1312 : cluster [DBG] osdmap e649: 8 total, 8 up, 8 in 2021-07-31T01:56:02.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:01 smithi045 conmon[39718]: cluster 2021-07-31T01:56:01.640568+0000 mon.a (mon.0) 1313 : cluster 2021-07-31T01:56:02.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:01 smithi045 conmon[39718]: [DBG] osdmap e650: 8 total, 8 up, 8 in 2021-07-31T01:56:02.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:01 smithi045 conmon[33655]: cluster 2021-07-31T01:56:00.657399+0000 mon.a (mon. 2021-07-31T01:56:02.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:01 smithi045 conmon[33655]: 0) 1312 : cluster [DBG] osdmap e649: 8 total, 8 up, 8 in 2021-07-31T01:56:02.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:01 smithi045 conmon[33655]: cluster 2021-07-31T01:56:01.640568+0000 mon.a (mon.0) 1313 : cluster 2021-07-31T01:56:02.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:01 smithi045 conmon[33655]: [DBG] osdmap e650: 8 total, 8 up, 8 in 2021-07-31T01:56:02.647 INFO:tasks.rados.rados.0.smithi045.stdout:3168: done (2 left) 2021-07-31T01:56:02.648 INFO:tasks.rados.rados.0.smithi045.stdout:3169: done (1 left) 2021-07-31T01:56:02.648 INFO:tasks.rados.rados.0.smithi045.stdout:3170: done (0 left) 2021-07-31T01:56:02.648 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 26 current snap is 303 2021-07-31T01:56:02.648 INFO:tasks.rados.rados.0.smithi045.stdout:3171: seq_num 979 ranges {9584640=802816,10387456=729088,11116544=458752,11575296=16384} 2021-07-31T01:56:02.704 INFO:tasks.rados.rados.0.smithi045.stdout:3171: writing smithi04569269-26 from 9584640 to 10387456 tid 1 2021-07-31T01:56:02.708 INFO:tasks.rados.rados.0.smithi045.stdout:3171: writing smithi04569269-26 from 10387456 to 11116544 tid 2 2021-07-31T01:56:02.711 INFO:tasks.rados.rados.0.smithi045.stdout:3171: writing smithi04569269-26 from 11116544 to 11575296 tid 3 2021-07-31T01:56:02.711 INFO:tasks.rados.rados.0.smithi045.stdout:3171: writing smithi04569269-26 from 11575296 to 11591680 tid 4 2021-07-31T01:56:02.712 INFO:tasks.rados.rados.0.smithi045.stdout:3172: rollback oid 10 current snap is 303 2021-07-31T01:56:02.712 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 10 to 248 2021-07-31T01:56:02.712 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 49 current snap is 303 2021-07-31T01:56:02.712 INFO:tasks.rados.rados.0.smithi045.stdout:3173: seq_num 980 ranges {14852096=417792,15269888=614400,15884288=663552,16547840=327680} 2021-07-31T01:56:02.795 INFO:tasks.rados.rados.0.smithi045.stdout:3173: writing smithi04569269-49 from 14852096 to 15269888 tid 1 2021-07-31T01:56:02.798 INFO:tasks.rados.rados.0.smithi045.stdout:3173: writing smithi04569269-49 from 15269888 to 15884288 tid 2 2021-07-31T01:56:02.802 INFO:tasks.rados.rados.0.smithi045.stdout:3173: writing smithi04569269-49 from 15884288 to 16547840 tid 3 2021-07-31T01:56:02.804 INFO:tasks.rados.rados.0.smithi045.stdout:3173: writing smithi04569269-49 from 16547840 to 16875520 tid 4 2021-07-31T01:56:02.804 INFO:tasks.rados.rados.0.smithi045.stdout:3174: read oid 25 snap -1 2021-07-31T01:56:02.805 INFO:tasks.rados.rados.0.smithi045.stdout:3174: expect (ObjNum 837 snap 263 seq_num 837) 2021-07-31T01:56:02.805 INFO:tasks.rados.rados.0.smithi045.stdout:3175: read oid 24 snap -1 2021-07-31T01:56:02.805 INFO:tasks.rados.rados.0.smithi045.stdout:3175: expect (ObjNum 0 snap 0 seq_num 0) 2021-07-31T01:56:02.805 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 43 current snap is 303 2021-07-31T01:56:02.806 INFO:tasks.rados.rados.0.smithi045.stdout:3171: finishing write tid 1 to smithi04569269-26 2021-07-31T01:56:02.806 INFO:tasks.rados.rados.0.smithi045.stdout:3176: seq_num 981 ranges {9699328=434176,10133504=540672,10674176=466944,11141120=524288,11665408=73728} 2021-07-31T01:56:02.860 INFO:tasks.rados.rados.0.smithi045.stdout:3176: writing smithi04569269-43 from 9699328 to 10133504 tid 1 2021-07-31T01:56:02.862 INFO:tasks.rados.rados.0.smithi045.stdout:3176: writing smithi04569269-43 from 10133504 to 10674176 tid 2 2021-07-31T01:56:02.865 INFO:tasks.rados.rados.0.smithi045.stdout:3176: writing smithi04569269-43 from 10674176 to 11141120 tid 3 2021-07-31T01:56:02.868 INFO:tasks.rados.rados.0.smithi045.stdout:3176: writing smithi04569269-43 from 11141120 to 11665408 tid 4 2021-07-31T01:56:02.869 INFO:tasks.rados.rados.0.smithi045.stdout:3176: writing smithi04569269-43 from 11665408 to 11739136 tid 5 2021-07-31T01:56:02.869 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 17 current snap is 303 2021-07-31T01:56:02.870 INFO:tasks.rados.rados.0.smithi045.stdout:3171: finishing write tid 2 to smithi04569269-26 2021-07-31T01:56:02.870 INFO:tasks.rados.rados.0.smithi045.stdout:3171: finishing write tid 3 to smithi04569269-26 2021-07-31T01:56:02.871 INFO:tasks.rados.rados.0.smithi045.stdout:3171: finishing write tid 4 to smithi04569269-26 2021-07-31T01:56:02.871 INFO:tasks.rados.rados.0.smithi045.stdout:3177: seq_num 982 ranges {3801088=450560,4251648=425984,4677632=679936,5357568=466944,5824512=32768} 2021-07-31T01:56:02.892 INFO:tasks.rados.rados.0.smithi045.stdout:3177: writing smithi04569269-17 from 3801088 to 4251648 tid 1 2021-07-31T01:56:02.894 INFO:tasks.rados.rados.0.smithi045.stdout:3177: writing smithi04569269-17 from 4251648 to 4677632 tid 2 2021-07-31T01:56:02.898 INFO:tasks.rados.rados.0.smithi045.stdout:3177: writing smithi04569269-17 from 4677632 to 5357568 tid 3 2021-07-31T01:56:02.902 INFO:tasks.rados.rados.0.smithi045.stdout:3177: writing smithi04569269-17 from 5357568 to 5824512 tid 4 2021-07-31T01:56:02.903 INFO:tasks.rados.rados.0.smithi045.stdout:3177: writing smithi04569269-17 from 5824512 to 5857280 tid 5 2021-07-31T01:56:02.903 INFO:tasks.rados.rados.0.smithi045.stdout:3178: setattr oid 27 current snap is 303 2021-07-31T01:56:02.903 INFO:tasks.rados.rados.0.smithi045.stdout:3172: finishing rollback tid 0 to smithi04569269-10 2021-07-31T01:56:02.904 INFO:tasks.rados.rados.0.smithi045.stdout:3172: finishing rollback tid 1 to smithi04569269-10 2021-07-31T01:56:02.904 INFO:tasks.rados.rados.0.smithi045.stdout:3171: finishing write tid 5 to smithi04569269-26 2021-07-31T01:56:02.904 INFO:tasks.rados.rados.0.smithi045.stdout:3172: finishing rollback tid 2 to smithi04569269-10 2021-07-31T01:56:02.904 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 10 v 1698 (ObjNum 768 snap 235 seq_num 768) dirty exists 2021-07-31T01:56:02.904 INFO:tasks.rados.rados.0.smithi045.stdout:3171: finishing write tid 6 to smithi04569269-26 2021-07-31T01:56:02.905 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 26 v 1848 (ObjNum 979 snap 303 seq_num 979) dirty exists 2021-07-31T01:56:02.905 INFO:tasks.rados.rados.0.smithi045.stdout:3171: left oid 26 (ObjNum 979 snap 303 seq_num 979) 2021-07-31T01:56:02.905 INFO:tasks.rados.rados.0.smithi045.stdout:3173: finishing write tid 1 to smithi04569269-49 2021-07-31T01:56:02.905 INFO:tasks.rados.rados.0.smithi045.stdout:3173: finishing write tid 2 to smithi04569269-49 2021-07-31T01:56:02.906 INFO:tasks.rados.rados.0.smithi045.stdout:3175: expect (ObjNum 0 snap 0 seq_num 0) 2021-07-31T01:56:02.906 INFO:tasks.rados.rados.0.smithi045.stdout:3173: finishing write tid 3 to smithi04569269-49 2021-07-31T01:56:02.907 INFO:tasks.rados.rados.0.smithi045.stdout:3173: finishing write tid 4 to smithi04569269-49 2021-07-31T01:56:02.907 INFO:tasks.rados.rados.0.smithi045.stdout:3173: finishing write tid 5 to smithi04569269-49 2021-07-31T01:56:02.907 INFO:tasks.rados.rados.0.smithi045.stdout:3173: finishing write tid 6 to smithi04569269-49 2021-07-31T01:56:02.907 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 49 v 2022 (ObjNum 980 snap 303 seq_num 980) dirty exists 2021-07-31T01:56:02.907 INFO:tasks.rados.rados.0.smithi045.stdout:3173: left oid 49 (ObjNum 980 snap 303 seq_num 980) 2021-07-31T01:56:02.908 INFO:tasks.rados.rados.0.smithi045.stdout:3174: expect (ObjNum 837 snap 263 seq_num 837) 2021-07-31T01:56:02.959 INFO:tasks.rados.rados.0.smithi045.stdout:3176: finishing write tid 1 to smithi04569269-43 2021-07-31T01:56:02.959 INFO:tasks.rados.rados.0.smithi045.stdout:3176: finishing write tid 2 to smithi04569269-43 2021-07-31T01:56:02.960 INFO:tasks.rados.rados.0.smithi045.stdout:3176: finishing write tid 3 to smithi04569269-43 2021-07-31T01:56:02.960 INFO:tasks.rados.rados.0.smithi045.stdout:3176: finishing write tid 4 to smithi04569269-43 2021-07-31T01:56:02.960 INFO:tasks.rados.rados.0.smithi045.stdout:3171: done (7 left) 2021-07-31T01:56:02.961 INFO:tasks.rados.rados.0.smithi045.stdout:3172: done (6 left) 2021-07-31T01:56:02.961 INFO:tasks.rados.rados.0.smithi045.stdout:3173: done (5 left) 2021-07-31T01:56:02.961 INFO:tasks.rados.rados.0.smithi045.stdout:3174: done (4 left) 2021-07-31T01:56:02.962 INFO:tasks.rados.rados.0.smithi045.stdout:3175: done (3 left) 2021-07-31T01:56:02.962 INFO:tasks.rados.rados.0.smithi045.stdout:3179: copy_from oid 34 from oid 16 current snap is 303 2021-07-31T01:56:02.962 INFO:tasks.rados.rados.0.smithi045.stdout:3176: finishing write tid 5 to smithi04569269-43 2021-07-31T01:56:02.962 INFO:tasks.rados.rados.0.smithi045.stdout:3176: finishing write tid 6 to smithi04569269-43 2021-07-31T01:56:02.963 INFO:tasks.rados.rados.0.smithi045.stdout:3176: finishing write tid 7 to smithi04569269-43 2021-07-31T01:56:02.963 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 43 v 1853 (ObjNum 981 snap 303 seq_num 981) dirty exists 2021-07-31T01:56:02.963 INFO:tasks.rados.rados.0.smithi045.stdout:3176: left oid 43 (ObjNum 981 snap 303 seq_num 981) 2021-07-31T01:56:02.963 INFO:tasks.rados.rados.0.smithi045.stdout:3177: finishing write tid 1 to smithi04569269-17 2021-07-31T01:56:02.964 INFO:tasks.rados.rados.0.smithi045.stdout:3177: finishing write tid 2 to smithi04569269-17 2021-07-31T01:56:02.964 INFO:tasks.rados.rados.0.smithi045.stdout:3177: finishing write tid 3 to smithi04569269-17 2021-07-31T01:56:02.964 INFO:tasks.rados.rados.0.smithi045.stdout:3177: finishing write tid 4 to smithi04569269-17 2021-07-31T01:56:02.964 INFO:tasks.rados.rados.0.smithi045.stdout:3176: done (3 left) 2021-07-31T01:56:02.965 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 42 current snap is 303 2021-07-31T01:56:02.965 INFO:tasks.rados.rados.0.smithi045.stdout:3177: finishing write tid 5 to smithi04569269-17 2021-07-31T01:56:02.966 INFO:tasks.rados.rados.0.smithi045.stdout:3177: finishing write tid 6 to smithi04569269-17 2021-07-31T01:56:02.966 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 27 v 1854 (ObjNum 1953066355 snap 875587944 seq_num 842610229) dirty exists 2021-07-31T01:56:02.966 INFO:tasks.rados.rados.0.smithi045.stdout:3177: finishing write tid 7 to smithi04569269-17 2021-07-31T01:56:02.966 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 17 v 1704 (ObjNum 982 snap 303 seq_num 982) dirty exists 2021-07-31T01:56:02.967 INFO:tasks.rados.rados.0.smithi045.stdout:3177: left oid 17 (ObjNum 982 snap 303 seq_num 982) 2021-07-31T01:56:02.967 INFO:tasks.rados.rados.0.smithi045.stdout:3180: seq_num 983 ranges {2686976=466944,3153920=712704,3866624=434176,4300800=458752} 2021-07-31T01:56:02.974 INFO:tasks.rados.rados.0.smithi045.stdout:3180: writing smithi04569269-42 from 2686976 to 3153920 tid 1 2021-07-31T01:56:02.978 INFO:tasks.rados.rados.0.smithi045.stdout:3180: writing smithi04569269-42 from 3153920 to 3866624 tid 2 2021-07-31T01:56:02.980 INFO:tasks.rados.rados.0.smithi045.stdout:3180: writing smithi04569269-42 from 3866624 to 4300800 tid 3 2021-07-31T01:56:02.983 INFO:tasks.rados.rados.0.smithi045.stdout:3180: writing smithi04569269-42 from 4300800 to 4759552 tid 4 2021-07-31T01:56:02.983 INFO:tasks.rados.rados.0.smithi045.stdout:3177: done (3 left) 2021-07-31T01:56:02.983 INFO:tasks.rados.rados.0.smithi045.stdout:3178: done (2 left) 2021-07-31T01:56:02.984 INFO:tasks.rados.rados.0.smithi045.stdout:3181: delete oid 15 current snap is 303 2021-07-31T01:56:02.984 INFO:tasks.rados.rados.0.smithi045.stdout:3179: finishing copy_from to smithi04569269-34 2021-07-31T01:56:02.984 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 34 v 1650 (ObjNum 944 snap 286 seq_num 944) dirty exists 2021-07-31T01:56:02.984 INFO:tasks.rados.rados.0.smithi045.stdout:3179: finishing copy_from racing read to smithi04569269-34 2021-07-31T01:56:02.985 INFO:tasks.rados.rados.0.smithi045.stdout:3180: finishing write tid 1 to smithi04569269-42 2021-07-31T01:56:02.985 INFO:tasks.rados.rados.0.smithi045.stdout:3180: finishing write tid 2 to smithi04569269-42 2021-07-31T01:56:02.986 INFO:tasks.rados.rados.0.smithi045.stdout:3180: finishing write tid 3 to smithi04569269-42 2021-07-31T01:56:02.987 INFO:tasks.rados.rados.0.smithi045.stdout:3179: done (2 left) 2021-07-31T01:56:02.987 INFO:tasks.rados.rados.0.smithi045.stdout:3181: done (1 left) 2021-07-31T01:56:02.988 INFO:tasks.rados.rados.0.smithi045.stdout:3182: read oid 44 snap -1 2021-07-31T01:56:02.988 INFO:tasks.rados.rados.0.smithi045.stdout:3182: expect (ObjNum 955 snap 289 seq_num 955) 2021-07-31T01:56:02.988 INFO:tasks.rados.rados.0.smithi045.stdout:3183: read oid 10 snap -1 2021-07-31T01:56:02.989 INFO:tasks.rados.rados.0.smithi045.stdout:3183: expect (ObjNum 768 snap 235 seq_num 768) 2021-07-31T01:56:02.989 INFO:tasks.rados.rados.0.smithi045.stdout:3184: snap_remove snap 276 2021-07-31T01:56:03.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:02 smithi154 conmon[34553]: cluster 2021-07-31T01:56:01.917503+0000 mgr.x (mgr.24542) 375 : cluster [DBG] pgmap v761: 25 pgs: 25 active+clean; 547 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 1.4 MiB/s wr, 6 op/s 2021-07-31T01:56:03.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:02 smithi154 conmon[34553]: cluster 2021-07-31T01:56:02.646606+0000 2021-07-31T01:56:03.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:02 smithi154 conmon[34553]: mon.a (mon.0) 1314 : cluster [DBG] osdmap e651: 8 total, 8 up, 8 in 2021-07-31T01:56:03.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:02 smithi045 conmon[33655]: cluster 2021-07-31T01:56:01.917503+0000 mgr.x (mgr 2021-07-31T01:56:03.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:02 smithi045 conmon[33655]: .24542) 375 : cluster [DBG] pgmap v761: 25 pgs: 25 active+clean; 547 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 1.4 MiB/s wr, 6 op/s 2021-07-31T01:56:03.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:02 smithi045 conmon[33655]: cluster 2021-07-31T01:56:02.646606+0000 mon.a (mon.0) 1314 : cluster [DBG] osdmap e651: 8 total, 8 up, 8 in 2021-07-31T01:56:03.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:02 smithi045 conmon[39718]: cluster 2021-07-31T01:56:01 2021-07-31T01:56:03.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:02 smithi045 conmon[39718]: .917503+0000 mgr.x (mgr.24542) 375 : cluster [DBG] pgmap v761: 25 pgs: 25 active+clean; 547 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 1.4 MiB/s wr, 6 op/s 2021-07-31T01:56:03.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:02 smithi045 conmon[39718]: cluster 2021-07-31T01: 2021-07-31T01:56:03.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:02 smithi045 conmon[39718]: 56:02.646606+0000 mon.a ( 2021-07-31T01:56:03.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:02 smithi045 conmon[39718]: mon.0) 1314 : cluster [DBG] 2021-07-31T01:56:03.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:02 smithi045 conmon[39718]: osdmap e651: 8 total, 8 up, 8 in 2021-07-31T01:56:03.667 INFO:tasks.rados.rados.0.smithi045.stdout:3184: done (3 left) 2021-07-31T01:56:03.668 INFO:tasks.rados.rados.0.smithi045.stdout:3185: copy_from oid 26 from oid 1 current snap is 303 2021-07-31T01:56:03.669 INFO:tasks.rados.rados.0.smithi045.stdout:3180: finishing write tid 4 to smithi04569269-42 2021-07-31T01:56:03.669 INFO:tasks.rados.rados.0.smithi045.stdout:3180: finishing write tid 5 to smithi04569269-42 2021-07-31T01:56:03.669 INFO:tasks.rados.rados.0.smithi045.stdout:3182: expect (ObjNum 955 snap 289 seq_num 955) 2021-07-31T01:56:03.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:56:03 smithi045 conmon[61210]: level=error ts=2021-07-31T01:56:03.524Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:56:03.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:56:03 smithi045 conmon[61210]: level=error ts=2021-07-31T01:56:03.524Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:56:03.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:56:03 smithi045 conmon[61210]: level=error ts=2021-07-31T01:56:03.524Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T01:56:03.795 INFO:tasks.rados.rados.0.smithi045.stdout:3180: finishing write tid 6 to smithi04569269-42 2021-07-31T01:56:03.795 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 42 v 1709 (ObjNum 983 snap 303 seq_num 983) dirty exists 2021-07-31T01:56:03.796 INFO:tasks.rados.rados.0.smithi045.stdout:3180: left oid 42 (ObjNum 983 snap 303 seq_num 983) 2021-07-31T01:56:03.796 INFO:tasks.rados.rados.0.smithi045.stdout:3183: expect (ObjNum 768 snap 235 seq_num 768) 2021-07-31T01:56:04.641 INFO:tasks.rados.rados.0.smithi045.stdout:3185: finishing copy_from to smithi04569269-26 2021-07-31T01:56:04.641 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 26 v 1846 (ObjNum 713 snap 222 seq_num 713) dirty exists 2021-07-31T01:56:04.642 INFO:tasks.rados.rados.0.smithi045.stdout:3185: finishing copy_from racing read to smithi04569269-26 2021-07-31T01:56:04.642 INFO:tasks.rados.rados.0.smithi045.stdout:3180: done (3 left) 2021-07-31T01:56:04.642 INFO:tasks.rados.rados.0.smithi045.stdout:3182: done (2 left) 2021-07-31T01:56:04.642 INFO:tasks.rados.rados.0.smithi045.stdout:3183: done (1 left) 2021-07-31T01:56:04.643 INFO:tasks.rados.rados.0.smithi045.stdout:3185: done (0 left) 2021-07-31T01:56:04.643 INFO:tasks.rados.rados.0.smithi045.stdout:3186: rmattr oid 6 current snap is 303 2021-07-31T01:56:04.643 INFO:tasks.rados.rados.0.smithi045.stdout:3186: done (0 left) 2021-07-31T01:56:04.643 INFO:tasks.rados.rados.0.smithi045.stdout:3187: read oid 27 snap 293 2021-07-31T01:56:04.644 INFO:tasks.rados.rados.0.smithi045.stdout:3187: expect (ObjNum 837 snap 263 seq_num 837) 2021-07-31T01:56:04.644 INFO:tasks.rados.rados.0.smithi045.stdout:3188: copy_from oid 41 from oid 11 current snap is 303 2021-07-31T01:56:04.644 INFO:tasks.rados.rados.0.smithi045.stdout:3189: snap_create 2021-07-31T01:56:04.651 INFO:tasks.rados.rados.0.smithi045.stdout:3187: expect (ObjNum 837 snap 263 seq_num 837) 2021-07-31T01:56:04.703 INFO:tasks.rados.rados.0.smithi045.stdout:3188: finishing copy_from to smithi04569269-41 2021-07-31T01:56:04.704 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 41 v 1686 (ObjNum 976 snap 299 seq_num 976) dirty exists 2021-07-31T01:56:04.704 INFO:tasks.rados.rados.0.smithi045.stdout:3188: finishing copy_from racing read to smithi04569269-41 2021-07-31T01:56:04.704 INFO:tasks.rados.rados.0.smithi045.stdout:3187: done (2 left) 2021-07-31T01:56:04.705 INFO:tasks.rados.rados.0.smithi045.stdout:3188: done (1 left) 2021-07-31T01:56:04.705 INFO:tasks.rados.rados.0.smithi045.stdout:3189: done (0 left) 2021-07-31T01:56:04.705 INFO:tasks.rados.rados.0.smithi045.stdout:3190: snap_create 2021-07-31T01:56:05.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:04 smithi154 conmon[34553]: cluster 2021-07-31T01:56:03.666481+0000 mon.a (mon.0) 2021-07-31T01:56:05.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:04 smithi154 conmon[34553]: 1315 : cluster [DBG] osdmap e652: 8 total, 8 up, 8 in 2021-07-31T01:56:05.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:04 smithi154 conmon[34553]: cluster 2021-07-31T01:56:03.918153+0000 mgr.x (mgr.24542) 376 : cluster [DBG] pgmap v764: 25 pgs: 25 active+clean; 563 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 13 MiB/s rd, 5.4 MiB/s wr, 24 op/s 2021-07-31T01:56:05.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:04 smithi045 conmon[33655]: cluster 2021-07-31T01:56:03.666481+0000 2021-07-31T01:56:05.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:04 smithi045 conmon[33655]: mon.a (mon.0) 1315 : cluster [DBG] osdmap e652: 8 total, 8 up, 8 in 2021-07-31T01:56:05.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:04 smithi045 conmon[33655]: cluster 2021-07-31T01:56:03.918153+0000 mgr.x (mgr.24542) 376 : cluster [DBG] pgmap v764: 25 pgs: 25 active+clean; 563 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 13 MiB/s rd, 5.4 MiB/s wr, 24 op/s 2021-07-31T01:56:05.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:04 smithi045 conmon[39718]: cluster 2021-07-31T01:56:03.666481+0000 2021-07-31T01:56:05.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:04 smithi045 conmon[39718]: mon.a (mon.0) 1315 : cluster [DBG] osdmap e652: 8 total, 8 up, 8 in 2021-07-31T01:56:05.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:04 smithi045 conmon[39718]: cluster 2021-07-31T01:56:03.918153+0000 mgr.x 2021-07-31T01:56:05.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:04 smithi045 conmon[39718]: (mgr.24542) 376 : cluster [DBG] pgmap v764: 25 pgs: 25 active+clean; 563 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 13 MiB/s rd, 5.4 MiB/s wr, 24 op/s 2021-07-31T01:56:05.675 INFO:tasks.rados.rados.0.smithi045.stdout:3190: done (0 left) 2021-07-31T01:56:05.675 INFO:tasks.rados.rados.0.smithi045.stdout:3191: rollback oid 9 current snap is 305 2021-07-31T01:56:05.675 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 9 to 290 2021-07-31T01:56:05.676 INFO:tasks.rados.rados.0.smithi045.stdout:3192: copy_from oid 11 from oid 31 current snap is 305 2021-07-31T01:56:05.676 INFO:tasks.rados.rados.0.smithi045.stdout:3193: read oid 13 snap -1 2021-07-31T01:56:05.676 INFO:tasks.rados.rados.0.smithi045.stdout:3193: expect deleted 2021-07-31T01:56:05.676 INFO:tasks.rados.rados.0.smithi045.stdout:3194: snap_create 2021-07-31T01:56:05.679 INFO:tasks.rados.rados.0.smithi045.stdout:3191: finishing rollback tid 0 to smithi04569269-9 2021-07-31T01:56:05.679 INFO:tasks.rados.rados.0.smithi045.stdout:3191: finishing rollback tid 1 to smithi04569269-9 2021-07-31T01:56:05.680 INFO:tasks.rados.rados.0.smithi045.stdout:3191: finishing rollback tid 2 to smithi04569269-9 2021-07-31T01:56:05.680 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 9 v 1841 (ObjNum 881 snap 272 seq_num 881) dirty exists 2021-07-31T01:56:05.690 INFO:tasks.rados.rados.0.smithi045.stdout:3192: finishing copy_from to smithi04569269-11 2021-07-31T01:56:05.691 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 11 v 2006 (ObjNum 813 snap 254 seq_num 813) dirty exists 2021-07-31T01:56:05.692 INFO:tasks.rados.rados.0.smithi045.stdout:3192: finishing copy_from racing read to smithi04569269-11 2021-07-31T01:56:06.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:05 smithi154 conmon[34553]: cluster 2021-07-31T01:56:04.670506+0000 mon.a (mon.0) 1316 : cluster 2021-07-31T01:56:06.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:05 smithi154 conmon[34553]: [DBG] osdmap e653: 8 total, 8 up, 8 in 2021-07-31T01:56:06.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:05 smithi045 conmon[39718]: cluster 2021-07-31T01:56:04.670506 2021-07-31T01:56:06.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:05 smithi045 conmon[39718]: +0000 mon.a (mon.0) 1316 : cluster [DBG] osdmap e653: 8 total, 8 up, 8 in 2021-07-31T01:56:06.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:05 smithi045 conmon[33655]: cluster 2021-07-31T01:56:06.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:05 smithi045 conmon[33655]: 2021-07-31T01:56:04 2021-07-31T01:56:06.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:05 smithi045 conmon[33655]: .670506+0000 mon.a (mon.0) 1316 : cluster [DBG] osdmap e653: 8 total, 8 up, 8 in 2021-07-31T01:56:06.643 INFO:tasks.rados.rados.0.smithi045.stdout:3191: done (3 left) 2021-07-31T01:56:06.644 INFO:tasks.rados.rados.0.smithi045.stdout:3192: done (2 left) 2021-07-31T01:56:06.644 INFO:tasks.rados.rados.0.smithi045.stdout:3193: done (1 left) 2021-07-31T01:56:06.644 INFO:tasks.rados.rados.0.smithi045.stdout:3194: done (0 left) 2021-07-31T01:56:06.644 INFO:tasks.rados.rados.0.smithi045.stdout:3195: rollback oid 20 current snap is 306 2021-07-31T01:56:06.644 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 20 to 305 2021-07-31T01:56:06.645 INFO:tasks.rados.rados.0.smithi045.stdout:3196: rmattr oid 23 current snap is 306 2021-07-31T01:56:06.645 INFO:tasks.rados.rados.0.smithi045.stdout:3196: done (1 left) 2021-07-31T01:56:06.645 INFO:tasks.rados.rados.0.smithi045.stdout:3197: rollback oid 31 current snap is 306 2021-07-31T01:56:06.646 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 31 to 296 2021-07-31T01:56:06.646 INFO:tasks.rados.rados.0.smithi045.stdout:3198: delete oid 39 current snap is 306 2021-07-31T01:56:06.650 INFO:tasks.rados.rados.0.smithi045.stdout:3195: finishing rollback tid 0 to smithi04569269-20 2021-07-31T01:56:06.651 INFO:tasks.rados.rados.0.smithi045.stdout:3198: done (2 left) 2021-07-31T01:56:06.651 INFO:tasks.rados.rados.0.smithi045.stdout:3199: delete oid 11 current snap is 306 2021-07-31T01:56:06.652 INFO:tasks.rados.rados.0.smithi045.stdout:3195: finishing rollback tid 1 to smithi04569269-20 2021-07-31T01:56:06.653 INFO:tasks.rados.rados.0.smithi045.stdout:3195: finishing rollback tid 2 to smithi04569269-20 2021-07-31T01:56:06.653 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 20 v 2036 (ObjNum 857 snap 268 seq_num 857) dirty exists 2021-07-31T01:56:06.654 INFO:tasks.rados.rados.0.smithi045.stdout:3195: done (2 left) 2021-07-31T01:56:06.654 INFO:tasks.rados.rados.0.smithi045.stdout:3199: done (1 left) 2021-07-31T01:56:06.654 INFO:tasks.rados.rados.0.smithi045.stdout:3200: snap_remove snap 248 2021-07-31T01:56:07.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:06 smithi154 conmon[34553]: cluster 2021-07-31T01:56:05.672882+0000 mon.a (mon.0) 1317 : cluster [DBG] osdmap e654: 8 total, 8 up, 8 in 2021-07-31T01:56:07.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:06 smithi154 conmon[34553]: cluster 2021-07-31T01:56:05.918611+0000 mgr.x (mgr.24542) 377 : cluster [DBG] pgmap v767: 25 pgs: 25 active+clean; 563 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 13 MiB/s rd, 5.4 MiB/s wr, 24 op/s 2021-07-31T01:56:07.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:06 smithi154 conmon[34553]: cluster 2021-07-31T01:56:06.642111+0000 mon.a (mon.0) 1318 : cluster [DBG] osdmap e655: 8 total, 8 up, 8 in 2021-07-31T01:56:07.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:06 smithi045 conmon[33655]: cluster 2021-07-31T01:56:05.672882+0000 mon.a (mon.0) 1317 : 2021-07-31T01:56:07.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:06 smithi045 conmon[33655]: cluster [DBG] osdmap e654: 8 total, 8 up, 8 in 2021-07-31T01:56:07.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:06 smithi045 conmon[33655]: cluster 2021-07-31T01:56:05.918611+0000 mgr.x (mgr.24542) 377 : cluster [DBG] pgmap v767: 25 pgs: 25 active+clean; 563 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 13 MiB/s rd, 5.4 MiB/s wr, 24 op/s 2021-07-31T01:56:07.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:06 smithi045 conmon[33655]: cluster 2021-07-31T01:56:06.642111+0000 mon.a ( 2021-07-31T01:56:07.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:06 smithi045 conmon[33655]: mon.0) 1318 : cluster [DBG] osdmap e655: 8 total, 8 up, 8 in 2021-07-31T01:56:07.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:06 smithi045 conmon[39718]: cluster 2021-07-31T01:56:05.672882+0000 mon.a ( 2021-07-31T01:56:07.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:06 smithi045 conmon[39718]: mon.0) 1317 : cluster [DBG] osdmap e654: 8 total, 8 up, 8 in 2021-07-31T01:56:07.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:06 smithi045 conmon[39718]: cluster 2021-07-31T01:56:05 2021-07-31T01:56:07.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:06 smithi045 conmon[39718]: .918611+0000 mgr.x (mgr.24542) 377 : cluster [DBG] 2021-07-31T01:56:07.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:06 smithi045 conmon[39718]: pgmap v767: 25 pgs: 25 active+clean; 563 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 13 MiB/s rd, 5.4 MiB/s wr, 24 op/s 2021-07-31T01:56:07.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:06 smithi045 conmon[39718]: cluster 2021-07-31T01: 2021-07-31T01:56:07.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:06 smithi045 conmon[39718]: 56:06.642111+0000 mon.a (mon.0) 1318 : cluster [DBG] osdmap e655: 8 total, 8 up, 8 in 2021-07-31T01:56:07.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:06 smithi045 conmon[39718]: 2021-07-31T01:56:07.650 INFO:tasks.rados.rados.0.smithi045.stdout:3200: done (1 left) 2021-07-31T01:56:07.650 INFO:tasks.rados.rados.0.smithi045.stdout:3201: snap_create 2021-07-31T01:56:07.651 INFO:tasks.rados.rados.0.smithi045.stdout:3197: finishing rollback tid 0 to smithi04569269-31 2021-07-31T01:56:07.651 INFO:tasks.rados.rados.0.smithi045.stdout:3197: finishing rollback tid 1 to smithi04569269-31 2021-07-31T01:56:07.651 INFO:tasks.rados.rados.0.smithi045.stdout:3197: finishing rollback tid 2 to smithi04569269-31 2021-07-31T01:56:07.651 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 31 v 2040 (ObjNum 813 snap 254 seq_num 813) dirty exists 2021-07-31T01:56:08.655 INFO:tasks.rados.rados.0.smithi045.stdout:3197: done (1 left) 2021-07-31T01:56:08.655 INFO:tasks.rados.rados.0.smithi045.stdout:3201: done (0 left) 2021-07-31T01:56:08.656 INFO:tasks.rados.rados.0.smithi045.stdout:3202: rollback oid 11 current snap is 307 2021-07-31T01:56:08.656 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 11 to 289 2021-07-31T01:56:08.656 INFO:tasks.rados.rados.0.smithi045.stdout:3203: snap_remove snap 285 2021-07-31T01:56:09.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:08 smithi154 conmon[34553]: cluster 2021-07-31T01:56:07.648368+0000 mon.a (mon.0) 1319 2021-07-31T01:56:09.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:08 smithi154 conmon[34553]: : cluster [DBG] osdmap e656: 8 total, 8 up, 8 in 2021-07-31T01:56:09.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:08 smithi154 conmon[34553]: cluster 2021-07-31T01:56:07.919053+0000 mgr.x (mgr.24542) 378 : cluster [DBG] pgmap v770: 25 pgs: 25 active+clean; 563 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:56:09.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:08 smithi045 conmon[33655]: cluster 2021-07-31T01:56:07.648368+0000 mon.a (mon 2021-07-31T01:56:09.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:08 smithi045 conmon[33655]: .0) 1319 : cluster [DBG] osdmap e656: 8 total, 8 up, 8 in 2021-07-31T01:56:09.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:08 smithi045 conmon[33655]: cluster 2021-07-31T01:56:07.919053+0000 mgr.x (mgr.24542) 378 : cluster [DBG] pgmap v770: 25 pgs: 25 active+clean; 563 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:56:09.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:08 smithi045 conmon[39718]: cluster 2021-07-31T01:56:07.648368+0000 mon.a ( 2021-07-31T01:56:09.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:08 smithi045 conmon[39718]: mon.0) 1319 : cluster [DBG] osdmap e656: 8 total, 8 up, 8 in 2021-07-31T01:56:09.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:08 smithi045 conmon[39718]: cluster 2021-07-31T01:56 2021-07-31T01:56:09.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:08 smithi045 conmon[39718]: :07.919053+0000 mgr.x (mgr.24542) 378 : cluster 2021-07-31T01:56:09.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:08 smithi045 conmon[39718]: [DBG] pgmap v770: 25 pgs: 25 active+clean; 563 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:56:09.661 INFO:tasks.rados.rados.0.smithi045.stdout:3203: done (1 left) 2021-07-31T01:56:09.661 INFO:tasks.rados.rados.0.smithi045.stdout:3204: delete oid 34 current snap is 307 2021-07-31T01:56:09.662 INFO:tasks.rados.rados.0.smithi045.stdout:3202: finishing rollback tid 1 to smithi04569269-11 2021-07-31T01:56:09.662 INFO:tasks.rados.rados.0.smithi045.stdout:3202: finishing rollback tid 2 to smithi04569269-11 2021-07-31T01:56:09.662 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 11 v 2010 (ObjNum 920 snap 278 seq_num 920) dirty exists 2021-07-31T01:56:09.675 INFO:tasks.rados.rados.0.smithi045.stdout:3202: done (1 left) 2021-07-31T01:56:09.675 INFO:tasks.rados.rados.0.smithi045.stdout:3204: done (0 left) 2021-07-31T01:56:09.675 INFO:tasks.rados.rados.0.smithi045.stdout:3205: snap_remove snap 297 2021-07-31T01:56:10.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:09 smithi154 conmon[34553]: cluster 2021-07-31T01:56:08.654189+0000 mon.a (mon.0) 1320 : cluster [DBG] osdmap e657: 8 total, 8 up, 8 in 2021-07-31T01:56:10.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:09 smithi045 conmon[33655]: cluster 2021-07-31T01:56:08.654189+0000 mon.a (mon 2021-07-31T01:56:10.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:09 smithi045 conmon[33655]: .0) 1320 : cluster [DBG] osdmap e657: 8 total, 8 up, 8 in 2021-07-31T01:56:10.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:09 smithi045 conmon[39718]: cluster 2021-07-31T01:56:08.654189+0000 2021-07-31T01:56:10.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:09 smithi045 conmon[39718]: mon.a (mon.0) 1320 : cluster [DBG] osdmap e657: 8 total, 8 up, 8 in 2021-07-31T01:56:10.667 INFO:tasks.rados.rados.0.smithi045.stdout:3205: done (0 left) 2021-07-31T01:56:10.668 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 33 current snap is 307 2021-07-31T01:56:10.668 INFO:tasks.rados.rados.0.smithi045.stdout:3206: seq_num 988 ranges {2449408=557056,3006464=532480,3538944=622592,4161536=450560} 2021-07-31T01:56:10.685 INFO:tasks.rados.rados.0.smithi045.stdout:3206: writing smithi04569269-33 from 2449408 to 3006464 tid 1 2021-07-31T01:56:10.688 INFO:tasks.rados.rados.0.smithi045.stdout:3206: writing smithi04569269-33 from 3006464 to 3538944 tid 2 2021-07-31T01:56:10.691 INFO:tasks.rados.rados.0.smithi045.stdout:3206: writing smithi04569269-33 from 3538944 to 4161536 tid 3 2021-07-31T01:56:10.695 INFO:tasks.rados.rados.0.smithi045.stdout:3206: writing smithi04569269-33 from 4161536 to 4612096 tid 4 2021-07-31T01:56:10.695 INFO:tasks.rados.rados.0.smithi045.stdout:3207: rollback oid 19 current snap is 307 2021-07-31T01:56:10.695 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 19 to 255 2021-07-31T01:56:10.696 INFO:tasks.rados.rados.0.smithi045.stdout:3206: finishing write tid 1 to smithi04569269-33 2021-07-31T01:56:10.696 INFO:tasks.rados.rados.0.smithi045.stdout:3206: finishing write tid 2 to smithi04569269-33 2021-07-31T01:56:10.696 INFO:tasks.rados.rados.0.smithi045.stdout:3208: delete oid 4 current snap is 307 2021-07-31T01:56:10.696 INFO:tasks.rados.rados.0.smithi045.stdout:3206: finishing write tid 3 to smithi04569269-33 2021-07-31T01:56:10.697 INFO:tasks.rados.rados.0.smithi045.stdout:3207: finishing rollback tid 1 to smithi04569269-19 2021-07-31T01:56:10.697 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 19 v 0 (ObjNum 0 snap 32731 seq_num 3189749568) dirty dne 2021-07-31T01:56:10.698 INFO:tasks.rados.rados.0.smithi045.stdout:3206: finishing write tid 4 to smithi04569269-33 2021-07-31T01:56:10.699 INFO:tasks.rados.rados.0.smithi045.stdout:3207: done (2 left) 2021-07-31T01:56:10.699 INFO:tasks.rados.rados.0.smithi045.stdout:3208: done (1 left) 2021-07-31T01:56:10.700 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 9 current snap is 307 2021-07-31T01:56:10.700 INFO:tasks.rados.rados.0.smithi045.stdout:3209: seq_num 990 ranges {9584640=589824,10174464=753664,10928128=638976,11567104=212992} 2021-07-31T01:56:10.754 INFO:tasks.rados.rados.0.smithi045.stdout:3209: writing smithi04569269-9 from 9584640 to 10174464 tid 1 2021-07-31T01:56:10.758 INFO:tasks.rados.rados.0.smithi045.stdout:3209: writing smithi04569269-9 from 10174464 to 10928128 tid 2 2021-07-31T01:56:10.762 INFO:tasks.rados.rados.0.smithi045.stdout:3209: writing smithi04569269-9 from 10928128 to 11567104 tid 3 2021-07-31T01:56:10.763 INFO:tasks.rados.rados.0.smithi045.stdout:3209: writing smithi04569269-9 from 11567104 to 11780096 tid 4 2021-07-31T01:56:10.764 INFO:tasks.rados.rados.0.smithi045.stdout:3210: snap_create 2021-07-31T01:56:10.764 INFO:tasks.rados.rados.0.smithi045.stdout:3206: finishing write tid 5 to smithi04569269-33 2021-07-31T01:56:10.764 INFO:tasks.rados.rados.0.smithi045.stdout:3206: finishing write tid 6 to smithi04569269-33 2021-07-31T01:56:10.764 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 33 v 2015 (ObjNum 988 snap 307 seq_num 988) dirty exists 2021-07-31T01:56:10.764 INFO:tasks.rados.rados.0.smithi045.stdout:3206: left oid 33 (ObjNum 988 snap 307 seq_num 988) 2021-07-31T01:56:10.765 INFO:tasks.rados.rados.0.smithi045.stdout:3209: finishing write tid 1 to smithi04569269-9 2021-07-31T01:56:10.765 INFO:tasks.rados.rados.0.smithi045.stdout:3209: finishing write tid 2 to smithi04569269-9 2021-07-31T01:56:10.766 INFO:tasks.rados.rados.0.smithi045.stdout:3209: finishing write tid 3 to smithi04569269-9 2021-07-31T01:56:10.766 INFO:tasks.rados.rados.0.smithi045.stdout:3209: finishing write tid 4 to smithi04569269-9 2021-07-31T01:56:10.766 INFO:tasks.rados.rados.0.smithi045.stdout:3209: finishing write tid 5 to smithi04569269-9 2021-07-31T01:56:10.767 INFO:tasks.rados.rados.0.smithi045.stdout:3209: finishing write tid 6 to smithi04569269-9 2021-07-31T01:56:10.767 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 9 v 1846 (ObjNum 990 snap 307 seq_num 990) dirty exists 2021-07-31T01:56:10.768 INFO:tasks.rados.rados.0.smithi045.stdout:3209: left oid 9 (ObjNum 990 snap 307 seq_num 990) 2021-07-31T01:56:11.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:10 smithi154 conmon[34553]: cluster 2021-07-31T01:56:09.660207+0000 mon.a (mon.0) 1321 : cluster [DBG] 2021-07-31T01:56:11.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:10 smithi154 conmon[34553]: osdmap e658: 8 total, 8 up, 8 in 2021-07-31T01:56:11.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:10 smithi154 conmon[34553]: cluster 2021-07-31T01:56:09.919606+0000 mgr.x (mgr.24542) 379 : cluster [DBG] pgmap v773: 25 pgs: 25 active+clean; 545 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.5 MiB/s rd, 1.1 MiB/s wr, 4 op/s 2021-07-31T01:56:11.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:10 smithi045 conmon[39718]: cluster 2021-07-31T01:56:09.660207+0000 mon.a (mon 2021-07-31T01:56:11.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:10 smithi045 conmon[39718]: .0) 1321 : cluster [DBG] osdmap e658: 8 total, 8 up, 8 in 2021-07-31T01:56:11.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:10 smithi045 conmon[39718]: cluster 2021-07-31T01:56:09.919606+0000 mgr.x (mgr.24542) 379 : cluster 2021-07-31T01:56:11.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:10 smithi045 conmon[39718]: [DBG] pgmap v773: 25 pgs: 25 active+clean; 545 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.5 MiB/s rd, 1.1 MiB/s wr, 4 op/s 2021-07-31T01:56:11.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:10 smithi045 conmon[33655]: cluster 2021-07-31T01:56:09.660207+0000 mon.a (mon.0) 1321 2021-07-31T01:56:11.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:10 smithi045 conmon[33655]: : cluster [DBG] osdmap e658: 8 total, 8 up, 8 in 2021-07-31T01:56:11.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:10 smithi045 conmon[33655]: cluster 2021-07-31T01:56:09.919606+0000 mgr.x (mgr.24542) 379 : cluster [DBG] pgmap v773: 25 pgs: 25 active+clean; 545 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.5 MiB/s rd, 1.1 MiB/s wr, 4 op/s 2021-07-31T01:56:11.084 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T01:56:11.359 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T01:56:11.360 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 11m ago 11m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T01:56:11.360 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 8m ago 11m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T01:56:11.360 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 8m ago 15m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T01:56:11.360 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 11m ago 17m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T01:56:11.361 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 11m ago 17m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T01:56:11.361 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 8m ago 15m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T01:56:11.361 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 11m ago 15m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T01:56:11.361 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 11m ago 12m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T01:56:11.361 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 8m ago 12m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T01:56:11.362 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 11m ago 14m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T01:56:11.362 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 11m ago 14m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T01:56:11.362 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 11m ago 14m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T01:56:11.362 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 11m ago 13m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T01:56:11.362 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 8m ago 13m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T01:56:11.363 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 8m ago 13m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T01:56:11.363 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 8m ago 12m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T01:56:11.363 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 8m ago 12m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T01:56:11.363 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 8m ago 12m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T01:56:11.643 INFO:tasks.rados.rados.0.smithi045.stdout:3206: done (2 left) 2021-07-31T01:56:11.644 INFO:tasks.rados.rados.0.smithi045.stdout:3209: done (1 left) 2021-07-31T01:56:11.644 INFO:tasks.rados.rados.0.smithi045.stdout:3210: done (0 left) 2021-07-31T01:56:11.644 INFO:tasks.rados.rados.0.smithi045.stdout:3211: read oid 31 snap -1 2021-07-31T01:56:11.645 INFO:tasks.rados.rados.0.smithi045.stdout:3211: expect (ObjNum 813 snap 254 seq_num 813) 2021-07-31T01:56:11.645 INFO:tasks.rados.rados.0.smithi045.stdout:3212: read oid 45 snap -1 2021-07-31T01:56:11.645 INFO:tasks.rados.rados.0.smithi045.stdout:3212: expect (ObjNum 944 snap 286 seq_num 944) 2021-07-31T01:56:11.645 INFO:tasks.rados.rados.0.smithi045.stdout:3213: rollback oid 32 current snap is 308 2021-07-31T01:56:11.645 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 32 to 306 2021-07-31T01:56:11.646 INFO:tasks.rados.rados.0.smithi045.stdout:3214: delete oid 48 current snap is 308 2021-07-31T01:56:11.649 INFO:tasks.rados.rados.0.smithi045.stdout:3214: done (3 left) 2021-07-31T01:56:11.650 INFO:tasks.rados.rados.0.smithi045.stdout:3215: rollback oid 43 current snap is 308 2021-07-31T01:56:11.650 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 43 to 279 2021-07-31T01:56:11.650 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 3 current snap is 308 2021-07-31T01:56:11.650 INFO:tasks.rados.rados.0.smithi045.stdout:3216: seq_num 992 ranges {9584640=622592,10207232=524288,10731520=606208,11337728=475136} 2021-07-31T01:56:11.703 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T01:56:11.704 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T01:56:11.704 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T01:56:11.704 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T01:56:11.704 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T01:56:11.705 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T01:56:11.705 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T01:56:11.705 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T01:56:11.705 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T01:56:11.705 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T01:56:11.706 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T01:56:11.706 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T01:56:11.706 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T01:56:11.706 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T01:56:11.706 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T01:56:11.707 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T01:56:11.707 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T01:56:11.708 INFO:tasks.rados.rados.0.smithi045.stdout:3216: writing smithi04569269-3 from 9584640 to 10207232 tid 1 2021-07-31T01:56:11.708 INFO:tasks.rados.rados.0.smithi045.stdout:3216: writing smithi04569269-3 from 10207232 to 10731520 tid 2 2021-07-31T01:56:11.712 INFO:tasks.rados.rados.0.smithi045.stdout:3216: writing smithi04569269-3 from 10731520 to 11337728 tid 3 2021-07-31T01:56:11.715 INFO:tasks.rados.rados.0.smithi045.stdout:3216: writing smithi04569269-3 from 11337728 to 11812864 tid 4 2021-07-31T01:56:11.715 INFO:tasks.rados.rados.0.smithi045.stdout:3217: snap_create 2021-07-31T01:56:11.716 INFO:tasks.rados.rados.0.smithi045.stdout:3211: expect (ObjNum 813 snap 254 seq_num 813) 2021-07-31T01:56:11.765 INFO:tasks.rados.rados.0.smithi045.stdout:3213: finishing rollback tid 0 to smithi04569269-32 2021-07-31T01:56:11.766 INFO:tasks.rados.rados.0.smithi045.stdout:3213: finishing rollback tid 1 to smithi04569269-32 2021-07-31T01:56:11.766 INFO:tasks.rados.rados.0.smithi045.stdout:3215: finishing rollback tid 0 to smithi04569269-43 2021-07-31T01:56:11.766 INFO:tasks.rados.rados.0.smithi045.stdout:3215: finishing rollback tid 1 to smithi04569269-43 2021-07-31T01:56:11.767 INFO:tasks.rados.rados.0.smithi045.stdout:3215: finishing rollback tid 2 to smithi04569269-43 2021-07-31T01:56:11.767 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 43 v 1857 (ObjNum 888 snap 272 seq_num 888) dirty exists 2021-07-31T01:56:11.767 INFO:tasks.rados.rados.0.smithi045.stdout:3213: finishing rollback tid 2 to smithi04569269-32 2021-07-31T01:56:11.767 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 32 v 1851 (ObjNum 978 snap 301 seq_num 978) dirty exists 2021-07-31T01:56:11.767 INFO:tasks.rados.rados.0.smithi045.stdout:3212: expect (ObjNum 944 snap 286 seq_num 944) 2021-07-31T01:56:11.803 INFO:tasks.rados.rados.0.smithi045.stdout:3216: finishing write tid 1 to smithi04569269-3 2021-07-31T01:56:11.804 INFO:tasks.rados.rados.0.smithi045.stdout:3216: finishing write tid 2 to smithi04569269-3 2021-07-31T01:56:11.804 INFO:tasks.rados.rados.0.smithi045.stdout:3216: finishing write tid 3 to smithi04569269-3 2021-07-31T01:56:11.804 INFO:tasks.rados.rados.0.smithi045.stdout:3216: finishing write tid 4 to smithi04569269-3 2021-07-31T01:56:11.805 INFO:tasks.rados.rados.0.smithi045.stdout:3216: finishing write tid 5 to smithi04569269-3 2021-07-31T01:56:11.805 INFO:tasks.rados.rados.0.smithi045.stdout:3216: finishing write tid 6 to smithi04569269-3 2021-07-31T01:56:11.805 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 3 v 2080 (ObjNum 992 snap 308 seq_num 992) dirty exists 2021-07-31T01:56:11.805 INFO:tasks.rados.rados.0.smithi045.stdout:3216: left oid 3 (ObjNum 992 snap 308 seq_num 992) 2021-07-31T01:56:12.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:11 smithi154 conmon[34553]: cluster 2021-07-31T01:56:10.666789+0000 mon.a (mon.0) 1322 : cluster [DBG] osdmap e659: 8 total, 8 up, 8 in 2021-07-31T01:56:12.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:11 smithi154 conmon[34553]: cluster 2021-07-31T01:56:11.642849+0000 mon.a (mon.0) 1323 : cluster [DBG] osdmap e660: 8 total, 8 up, 8 in 2021-07-31T01:56:12.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:11 smithi045 conmon[39718]: cluster 2021-07-31T01:56:10.666789+0000 2021-07-31T01:56:12.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:11 smithi045 conmon[39718]: mon.a (mon.0) 1322 : cluster [DBG] osdmap e659: 8 total, 8 up, 8 in 2021-07-31T01:56:12.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:11 smithi045 conmon[39718]: cluster 2021- 2021-07-31T01:56:12.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:11 smithi045 conmon[39718]: 07-31T01:56:11.642849+0000 mon.a (mon.0) 1323 : 2021-07-31T01:56:12.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:11 smithi045 conmon[39718]: cluster [DBG] osdmap e660: 8 total, 8 up, 8 in 2021-07-31T01:56:12.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:11 smithi045 conmon[33655]: cluster 2021-07-31T01:56:12.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:11 smithi045 conmon[33655]: 2021-07-31T 2021-07-31T01:56:12.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:11 smithi045 conmon[33655]: 01:56:10.666789+0000 mon.a (mon.0) 1322 : cluster [DBG] osdmap e659: 8 total, 8 up, 8 in 2021-07-31T01:56:12.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:11 smithi045 conmon[33655]: cluster 2021-07 2021-07-31T01:56:12.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:11 smithi045 conmon[33655]: -31T01:56:11.642849+0000 2021-07-31T01:56:12.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:11 smithi045 conmon[33655]: mon.a (mon.0) 1323 : cluster [DBG] 2021-07-31T01:56:12.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:11 smithi045 conmon[33655]: osdmap e660: 8 total, 8 up, 8 in 2021-07-31T01:56:12.674 INFO:tasks.rados.rados.0.smithi045.stdout:3211: done (5 left) 2021-07-31T01:56:12.674 INFO:tasks.rados.rados.0.smithi045.stdout:3212: done (4 left) 2021-07-31T01:56:12.675 INFO:tasks.rados.rados.0.smithi045.stdout:3213: done (3 left) 2021-07-31T01:56:12.675 INFO:tasks.rados.rados.0.smithi045.stdout:3215: done (2 left) 2021-07-31T01:56:12.675 INFO:tasks.rados.rados.0.smithi045.stdout:3216: done (1 left) 2021-07-31T01:56:12.675 INFO:tasks.rados.rados.0.smithi045.stdout:3217: done (0 left) 2021-07-31T01:56:12.676 INFO:tasks.rados.rados.0.smithi045.stdout:3218: rollback oid 16 current snap is 309 2021-07-31T01:56:12.676 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 16 to 308 2021-07-31T01:56:12.676 INFO:tasks.rados.rados.0.smithi045.stdout:3219: setattr oid 43 current snap is 309 2021-07-31T01:56:12.676 INFO:tasks.rados.rados.0.smithi045.stdout:3220: delete oid 40 current snap is 309 2021-07-31T01:56:12.680 INFO:tasks.rados.rados.0.smithi045.stdout:3218: finishing rollback tid 0 to smithi04569269-16 2021-07-31T01:56:12.680 INFO:tasks.rados.rados.0.smithi045.stdout:3218: finishing rollback tid 1 to smithi04569269-16 2021-07-31T01:56:12.680 INFO:tasks.rados.rados.0.smithi045.stdout:3218: finishing rollback tid 2 to smithi04569269-16 2021-07-31T01:56:12.681 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 16 v 1654 (ObjNum 944 snap 286 seq_num 944) dirty exists 2021-07-31T01:56:12.681 INFO:tasks.rados.rados.0.smithi045.stdout:3218: done (2 left) 2021-07-31T01:56:12.682 INFO:tasks.rados.rados.0.smithi045.stdout:3220: done (1 left) 2021-07-31T01:56:12.682 INFO:tasks.rados.rados.0.smithi045.stdout:3221: copy_from oid 6 from oid 22 current snap is 309 2021-07-31T01:56:12.682 INFO:tasks.rados.rados.0.smithi045.stdout:3222: setattr oid 40 current snap is 309 2021-07-31T01:56:12.683 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 43 v 1858 (ObjNum 888 snap 272 seq_num 888) dirty exists 2021-07-31T01:56:12.684 INFO:tasks.rados.rados.0.smithi045.stdout:3221: finishing copy_from to smithi04569269-6 2021-07-31T01:56:12.684 INFO:tasks.rados.rados.0.smithi045.stdout:3221: got expected ENOENT (src dne) 2021-07-31T01:56:12.684 INFO:tasks.rados.rados.0.smithi045.stdout:3221: finishing copy_from racing read to smithi04569269-6 2021-07-31T01:56:12.684 INFO:tasks.rados.rados.0.smithi045.stdout:3219: done (2 left) 2021-07-31T01:56:12.684 INFO:tasks.rados.rados.0.smithi045.stdout:3221: done (1 left) 2021-07-31T01:56:12.685 INFO:tasks.rados.rados.0.smithi045.stdout:3223: copy_from oid 24 from oid 4 current snap is 309 2021-07-31T01:56:12.685 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 8 current snap is 309 2021-07-31T01:56:12.685 INFO:tasks.rados.rados.0.smithi045.stdout:3224: seq_num 994 ranges {1785856=655360,2441216=688128,3129344=622592,3751936=294912} 2021-07-31T01:56:12.696 INFO:tasks.rados.rados.0.smithi045.stdout:3224: writing smithi04569269-8 from 1785856 to 2441216 tid 1 2021-07-31T01:56:12.700 INFO:tasks.rados.rados.0.smithi045.stdout:3224: writing smithi04569269-8 from 2441216 to 3129344 tid 2 2021-07-31T01:56:12.704 INFO:tasks.rados.rados.0.smithi045.stdout:3224: writing smithi04569269-8 from 3129344 to 3751936 tid 3 2021-07-31T01:56:12.706 INFO:tasks.rados.rados.0.smithi045.stdout:3224: writing smithi04569269-8 from 3751936 to 4046848 tid 4 2021-07-31T01:56:12.706 INFO:tasks.rados.rados.0.smithi045.stdout:3225: snap_create 2021-07-31T01:56:12.707 INFO:tasks.rados.rados.0.smithi045.stdout:3223: finishing copy_from to smithi04569269-24 2021-07-31T01:56:12.707 INFO:tasks.rados.rados.0.smithi045.stdout:3223: got expected ENOENT (src dne) 2021-07-31T01:56:12.707 INFO:tasks.rados.rados.0.smithi045.stdout:3223: finishing copy_from racing read to smithi04569269-24 2021-07-31T01:56:12.707 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 40 v 1867 (ObjNum 1953066355 snap 875587944 seq_num 842610229) dirty exists 2021-07-31T01:56:12.708 INFO:tasks.rados.rados.0.smithi045.stdout:3224: finishing write tid 1 to smithi04569269-8 2021-07-31T01:56:12.708 INFO:tasks.rados.rados.0.smithi045.stdout:3224: finishing write tid 2 to smithi04569269-8 2021-07-31T01:56:12.709 INFO:tasks.rados.rados.0.smithi045.stdout:3224: finishing write tid 3 to smithi04569269-8 2021-07-31T01:56:12.710 INFO:tasks.rados.rados.0.smithi045.stdout:3224: finishing write tid 4 to smithi04569269-8 2021-07-31T01:56:12.710 INFO:tasks.rados.rados.0.smithi045.stdout:3224: finishing write tid 5 to smithi04569269-8 2021-07-31T01:56:12.711 INFO:tasks.rados.rados.0.smithi045.stdout:3224: finishing write tid 6 to smithi04569269-8 2021-07-31T01:56:12.712 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 8 v 2086 (ObjNum 994 snap 309 seq_num 994) dirty exists 2021-07-31T01:56:12.712 INFO:tasks.rados.rados.0.smithi045.stdout:3224: left oid 8 (ObjNum 994 snap 309 seq_num 994) 2021-07-31T01:56:13.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:12 smithi154 conmon[34553]: audit 2021-07-31T01:56:11.071700+0000 mgr.x (mgr.24542) 380 : audit [DBG] 2021-07-31T01:56:13.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:12 smithi154 conmon[34553]: from='client.24925 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:56:13.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:12 smithi154 conmon[34553]: audit 2021-07-31T01:56:11.351586+0000 mgr.x (mgr.24542) 381 : audit [DBG] from='client.15024 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:56:13.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:12 smithi154 conmon[34553]: audit 2021-07-31T01:56:11.702946+0000 mon.c (mon.1) 2021-07-31T01:56:13.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:12 smithi154 conmon[34553]: 34 : audit [DBG] from='client.? 172.21.15.45:0/1337026546' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T01:56:13.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:12 smithi154 conmon[34553]: cluster 2021-07-31T01:56:11.920205+0000 mgr.x (mgr.24542) 2021-07-31T01:56:13.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:12 smithi154 conmon[34553]: 382 : cluster [DBG] pgmap v776: 25 pgs: 25 active+clean; 545 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.5 MiB/s rd, 1.1 MiB/s wr, 4 op/s 2021-07-31T01:56:13.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:12 smithi045 conmon[39718]: audit 2021-07-31T01:56:11.071700+0000 mgr.x (mgr.24542) 380 : audit 2021-07-31T01:56:13.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:12 smithi045 conmon[39718]: [DBG] from='client.24925 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:56:13.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:12 smithi045 conmon[39718]: audit 2021-07-31T01:56:11.351586+0000 mgr.x (mgr.24542) 381 : audit [DBG] from='client.15024 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:56:13.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:12 smithi045 conmon[39718]: audit 2021-07-31T01:56:11 2021-07-31T01:56:13.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:12 smithi045 conmon[39718]: .702946+0000 mon.c (mon.1) 34 : audit [DBG] 2021-07-31T01:56:13.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:12 smithi045 conmon[39718]: from='client.? 172.21.15.45:0/1337026546' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T01:56:13.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:12 smithi045 conmon[39718]: cluster 2021-07-31T01:56:11.920205+0000 mgr.x (mgr 2021-07-31T01:56:13.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:12 smithi045 conmon[39718]: .24542) 382 : cluster [DBG] pgmap v776: 25 pgs: 25 active+clean; 545 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.5 MiB/s rd, 1.1 MiB/s wr, 4 op/s 2021-07-31T01:56:13.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:12 smithi045 conmon[33655]: audit 2021-07-31T01:56:11.071700+0000 mgr.x (mgr.24542) 2021-07-31T01:56:13.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:12 smithi045 conmon[33655]: 380 : audit [DBG] from='client.24925 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:56:13.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:12 smithi045 conmon[33655]: audit 2021-07-31T01:56:11.351586+0000 mgr.x (mgr.24542) 381 : audit [DBG] from='client.15024 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:56:13.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:12 smithi045 conmon[33655]: audit 2021-07-31T01:56:11.702946+0000 mon.c (mon.1) 34 : 2021-07-31T01:56:13.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:12 smithi045 conmon[33655]: audit [DBG] from='client.? 172.21.15.45:0/1337026546' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T01:56:13.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:12 smithi045 conmon[33655]: cluster 2021-07-31T01: 2021-07-31T01:56:13.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:12 smithi045 conmon[33655]: 56:11.920205+0000 mgr.x (mgr.24542) 382 : cluster [DBG] pgmap v776: 25 pgs: 25 active+clean; 545 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.5 MiB/s rd, 1.1 MiB/s wr, 4 op/s 2021-07-31T01:56:13.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:12 smithi045 conmon[33655]: 2021-07-31T01:56:13.676 INFO:tasks.rados.rados.0.smithi045.stdout:3222: done (3 left) 2021-07-31T01:56:13.677 INFO:tasks.rados.rados.0.smithi045.stdout:3223: done (2 left) 2021-07-31T01:56:13.677 INFO:tasks.rados.rados.0.smithi045.stdout:3224: done (1 left) 2021-07-31T01:56:13.677 INFO:tasks.rados.rados.0.smithi045.stdout:3225: done (0 left) 2021-07-31T01:56:13.677 INFO:tasks.rados.rados.0.smithi045.stdout:3226: read oid 10 snap -1 2021-07-31T01:56:13.678 INFO:tasks.rados.rados.0.smithi045.stdout:3226: expect (ObjNum 768 snap 235 seq_num 768) 2021-07-31T01:56:13.678 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 24 current snap is 310 2021-07-31T01:56:13.678 INFO:tasks.rados.rados.0.smithi045.stdout:3227: seq_num 995 ranges {0=671744,671744=573440,1245184=778240,2023424=253952} 2021-07-31T01:56:13.679 INFO:tasks.rados.rados.0.smithi045.stdout:3227: writing smithi04569269-24 from 0 to 671744 tid 1 2021-07-31T01:56:13.683 INFO:tasks.rados.rados.0.smithi045.stdout:3227: writing smithi04569269-24 from 671744 to 1245184 tid 2 2021-07-31T01:56:13.687 INFO:tasks.rados.rados.0.smithi045.stdout:3227: writing smithi04569269-24 from 1245184 to 2023424 tid 3 2021-07-31T01:56:13.689 INFO:tasks.rados.rados.0.smithi045.stdout:3227: writing smithi04569269-24 from 2023424 to 2277376 tid 4 2021-07-31T01:56:13.690 INFO:tasks.rados.rados.0.smithi045.stdout:3228: snap_remove snap 305 2021-07-31T01:56:13.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:13 smithi045 conmon[33655]: cluster 2021-07-31T01:56:12.673421+0000 mon.a (mon.0) 2021-07-31T01:56:13.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:13 smithi045 conmon[33655]: 1324 : cluster [DBG] osdmap e661: 8 total, 8 up, 8 in 2021-07-31T01:56:13.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:13 smithi045 conmon[39718]: cluster 2021-07-31T01:56:12.673421+0000 mon.a (mon. 2021-07-31T01:56:13.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:13 smithi045 conmon[39718]: 0) 1324 : cluster [DBG] osdmap e661: 8 total, 8 up, 8 in 2021-07-31T01:56:13.778 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:56:13 smithi045 conmon[61210]: level=error ts=2021-07-31T01:56:13.524Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:56:13.778 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:56:13 smithi045 conmon[61210]: level=error ts=2021-07-31T01:56:13.524Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:56:13.778 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:56:13 smithi045 conmon[61210]: level=error ts=2021-07-31T01:56:13.524Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T01:56:14.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:13 smithi154 conmon[34553]: cluster 2021-07-31T01:56:12.673421+0000 mon.a (mon.0) 1324 : cluster 2021-07-31T01:56:14.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:13 smithi154 conmon[34553]: [DBG] osdmap e661: 8 total, 8 up, 8 in 2021-07-31T01:56:14.696 INFO:tasks.rados.rados.0.smithi045.stdout:3228: done (2 left) 2021-07-31T01:56:14.696 INFO:tasks.rados.rados.0.smithi045.stdout:3229: rollback oid 21 current snap is 310 2021-07-31T01:56:14.697 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 21 to 302 2021-07-31T01:56:14.697 INFO:tasks.rados.rados.0.smithi045.stdout:3227: finishing write tid 1 to smithi04569269-24 2021-07-31T01:56:14.697 INFO:tasks.rados.rados.0.smithi045.stdout:3227: finishing write tid 2 to smithi04569269-24 2021-07-31T01:56:14.697 INFO:tasks.rados.rados.0.smithi045.stdout:3227: finishing write tid 3 to smithi04569269-24 2021-07-31T01:56:14.697 INFO:tasks.rados.rados.0.smithi045.stdout:3227: finishing write tid 4 to smithi04569269-24 2021-07-31T01:56:14.698 INFO:tasks.rados.rados.0.smithi045.stdout:3227: finishing write tid 5 to smithi04569269-24 2021-07-31T01:56:14.698 INFO:tasks.rados.rados.0.smithi045.stdout:3227: finishing write tid 6 to smithi04569269-24 2021-07-31T01:56:14.698 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 24 v 1654 (ObjNum 995 snap 310 seq_num 995) dirty exists 2021-07-31T01:56:14.698 INFO:tasks.rados.rados.0.smithi045.stdout:3227: left oid 24 (ObjNum 995 snap 310 seq_num 995) 2021-07-31T01:56:14.699 INFO:tasks.rados.rados.0.smithi045.stdout:3226: expect (ObjNum 768 snap 235 seq_num 768) 2021-07-31T01:56:15.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:14 smithi154 conmon[34553]: cluster 2021-07-31T01:56:13.675015+0000 mon.a (mon.0) 1325 : cluster [DBG] osdmap e662: 8 total, 8 up, 8 in 2021-07-31T01:56:15.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:14 smithi154 conmon[34553]: cluster 2021-07-31T01:56:13.920862+0000 mgr.x (mgr.24542) 383 : cluster [DBG] pgmap v779: 25 pgs: 25 active+clean; 553 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 9.5 MiB/s rd, 2.7 MiB/s wr, 23 op/s 2021-07-31T01:56:15.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:14 smithi045 conmon[39718]: cluster 2021-07-31T01:56:13.675015+0000 mon.a 2021-07-31T01:56:15.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:14 smithi045 conmon[39718]: (mon.0) 1325 : cluster [DBG] osdmap e662: 8 total, 8 up, 8 in 2021-07-31T01:56:15.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:14 smithi045 conmon[39718]: cluster 2021-07-31T01:56: 2021-07-31T01:56:15.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:14 smithi045 conmon[39718]: 13.920862+0000 mgr.x (mgr.24542) 383 : cluster 2021-07-31T01:56:15.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:14 smithi045 conmon[39718]: [DBG] pgmap v779: 25 pgs: 25 active+clean; 553 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 9.5 MiB/s rd, 2.7 MiB/s wr, 23 op/s 2021-07-31T01:56:15.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:14 smithi045 conmon[33655]: cluster 2021-07-31T01:56:13.675015+0000 mon.a (mon.0) 2021-07-31T01:56:15.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:14 smithi045 conmon[33655]: 1325 : cluster [DBG] osdmap e662: 8 total, 8 up, 8 in 2021-07-31T01:56:15.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:14 smithi045 conmon[33655]: cluster 2021-07-31T01:56:13.920862+0000 mgr.x (mgr.24542) 383 : cluster [DBG] pgmap v779: 25 pgs: 25 active+clean; 553 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 9.5 MiB/s rd, 2.7 MiB/s wr, 23 op/s 2021-07-31T01:56:15.543 INFO:tasks.rados.rados.0.smithi045.stdout:3229: finishing rollback tid 0 to smithi04569269-21 2021-07-31T01:56:15.543 INFO:tasks.rados.rados.0.smithi045.stdout:3229: finishing rollback tid 1 to smithi04569269-21 2021-07-31T01:56:15.543 INFO:tasks.rados.rados.0.smithi045.stdout:3229: finishing rollback tid 2 to smithi04569269-21 2021-07-31T01:56:15.544 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 21 v 2094 (ObjNum 866 snap 270 seq_num 866) dirty exists 2021-07-31T01:56:15.544 INFO:tasks.rados.rados.0.smithi045.stdout:3226: done (2 left) 2021-07-31T01:56:15.544 INFO:tasks.rados.rados.0.smithi045.stdout:3227: done (1 left) 2021-07-31T01:56:15.544 INFO:tasks.rados.rados.0.smithi045.stdout:3229: done (0 left) 2021-07-31T01:56:15.545 INFO:tasks.rados.rados.0.smithi045.stdout:3230: snap_create 2021-07-31T01:56:15.689 INFO:tasks.rados.rados.0.smithi045.stdout:3230: done (0 left) 2021-07-31T01:56:15.689 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 4 current snap is 311 2021-07-31T01:56:15.690 INFO:tasks.rados.rados.0.smithi045.stdout:3231: seq_num 996 ranges {0=688128,688128=507904,1196032=581632,1777664=516096} 2021-07-31T01:56:15.692 INFO:tasks.rados.rados.0.smithi045.stdout:3231: writing smithi04569269-4 from 0 to 688128 tid 1 2021-07-31T01:56:15.695 INFO:tasks.rados.rados.0.smithi045.stdout:3231: writing smithi04569269-4 from 688128 to 1196032 tid 2 2021-07-31T01:56:15.699 INFO:tasks.rados.rados.0.smithi045.stdout:3231: writing smithi04569269-4 from 1196032 to 1777664 tid 3 2021-07-31T01:56:15.702 INFO:tasks.rados.rados.0.smithi045.stdout:3231: writing smithi04569269-4 from 1777664 to 2293760 tid 4 2021-07-31T01:56:15.702 INFO:tasks.rados.rados.0.smithi045.stdout:3232: snap_remove snap 308 2021-07-31T01:56:16.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:15 smithi154 conmon[34553]: cluster 2021-07-31T01:56:14.689008+0000 mon.a (mon.0) 1326 : cluster [DBG] osdmap e663: 8 total, 8 up, 8 in 2021-07-31T01:56:16.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:15 smithi045 conmon[39718]: cluster 2021-07-31T01:56:14.689008 2021-07-31T01:56:16.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:15 smithi045 conmon[39718]: +0000 mon.a (mon.0) 1326 : cluster [DBG] osdmap e663: 8 total, 8 up, 8 in 2021-07-31T01:56:16.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:15 smithi045 conmon[33655]: cluster 2021-07-31T01:56:14.689008+0000 mon.a (mon.0) 1326 : cluster [DBG] osdmap e663: 8 total, 8 up, 8 in 2021-07-31T01:56:16.646 INFO:tasks.rados.rados.0.smithi045.stdout:3232: done (1 left) 2021-07-31T01:56:16.646 INFO:tasks.rados.rados.0.smithi045.stdout:3233: snap_remove snap 306 2021-07-31T01:56:17.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:16 smithi154 conmon[34553]: cluster 2021-07-31T01:56:15.688820+0000 mon.a (mon.0) 1327 : cluster [DBG] osdmap e664: 8 total, 8 up, 8 in 2021-07-31T01:56:17.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:16 smithi154 conmon[34553]: cluster 2021-07-31T01:56:15.921428+0000 mgr.x (mgr.24542) 384 : cluster [DBG] pgmap v782: 25 pgs: 25 active+clean; 553 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 9.5 MiB/s rd, 2.7 MiB/s wr, 23 op/s 2021-07-31T01:56:17.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:16 smithi154 conmon[34553]: cluster 2021-07-31T01:56:16.644537+0000 mon.a (mon.0) 1328 : cluster [DBG] osdmap e665: 8 total, 8 up, 8 in 2021-07-31T01:56:17.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:16 smithi045 conmon[39718]: cluster 2021-07-31T01:56: 2021-07-31T01:56:17.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:16 smithi045 conmon[39718]: 15.688820+0000 mon.a (mon.0) 1327 : cluster [DBG] osdmap e664: 8 total, 8 up, 8 in 2021-07-31T01:56:17.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:16 smithi045 conmon[39718]: cluster 2021-07-31T01:56:17.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:16 smithi045 conmon[39718]: 2021-07-31T01:56:15.921428+0000 mgr.x ( 2021-07-31T01:56:17.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:16 smithi045 conmon[39718]: mgr.24542) 384 : cluster [DBG] 2021-07-31T01:56:17.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:16 smithi045 conmon[39718]: pgmap v782: 25 pgs: 25 active+clean; 553 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 9.5 MiB/s rd, 2.7 MiB/s wr, 23 op/s 2021-07-31T01:56:17.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:16 smithi045 conmon[39718]: cluster 2021- 2021-07-31T01:56:17.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:16 smithi045 conmon[39718]: 07-31T01:56:16. 2021-07-31T01:56:17.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:16 smithi045 conmon[39718]: 644537+0000 mon.a (mon.0 2021-07-31T01:56:17.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:16 smithi045 conmon[39718]: ) 1328 : cluster [DBG] osdmap e665: 8 total, 8 up, 8 in 2021-07-31T01:56:17.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:16 smithi045 conmon[39718]: 2021-07-31T01:56:17.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:16 smithi045 conmon[33655]: cluster 2021-07-31T01:56:15.688820+0000 mon.a (mon.0) 1327 : cluster [DBG] 2021-07-31T01:56:17.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:16 smithi045 conmon[33655]: osdmap e664: 8 total, 8 up, 8 in 2021-07-31T01:56:17.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:16 smithi045 conmon[33655]: cluster 2021-07-31T01:56:15.921428+0000 2021-07-31T01:56:17.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:16 smithi045 conmon[33655]: mgr.x (mgr.24542) 384 : cluster [DBG] pgmap v782: 25 pgs: 25 active+clean; 553 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 9.5 MiB/s rd, 2.7 MiB/s wr, 23 op/s 2021-07-31T01:56:17.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:16 smithi045 conmon[33655]: cluster 2021-07-31T01:56:17.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:16 smithi045 conmon[33655]: 2021 2021-07-31T01:56:17.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:16 smithi045 conmon[33655]: -07-31T01:56: 2021-07-31T01:56:17.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:16 smithi045 conmon[33655]: 16.644537+0000 mon.a ( 2021-07-31T01:56:17.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:16 smithi045 conmon[33655]: mon.0) 1328 : cluster 2021-07-31T01:56:17.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:16 smithi045 conmon[33655]: [DBG] osdmap e665: 8 total, 8 up, 8 in 2021-07-31T01:56:17.653 INFO:tasks.rados.rados.0.smithi045.stdout:3233: done (1 left) 2021-07-31T01:56:17.653 INFO:tasks.rados.rados.0.smithi045.stdout:3234: delete oid 26 current snap is 311 2021-07-31T01:56:17.653 INFO:tasks.rados.rados.0.smithi045.stdout:3231: finishing write tid 1 to smithi04569269-4 2021-07-31T01:56:17.654 INFO:tasks.rados.rados.0.smithi045.stdout:3231: finishing write tid 2 to smithi04569269-4 2021-07-31T01:56:17.654 INFO:tasks.rados.rados.0.smithi045.stdout:3231: finishing write tid 3 to smithi04569269-4 2021-07-31T01:56:17.654 INFO:tasks.rados.rados.0.smithi045.stdout:3231: finishing write tid 4 to smithi04569269-4 2021-07-31T01:56:17.654 INFO:tasks.rados.rados.0.smithi045.stdout:3231: finishing write tid 5 to smithi04569269-4 2021-07-31T01:56:17.655 INFO:tasks.rados.rados.0.smithi045.stdout:3231: finishing write tid 6 to smithi04569269-4 2021-07-31T01:56:17.655 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 4 v 1856 (ObjNum 996 snap 311 seq_num 996) dirty exists 2021-07-31T01:56:17.655 INFO:tasks.rados.rados.0.smithi045.stdout:3231: left oid 4 (ObjNum 996 snap 311 seq_num 996) 2021-07-31T01:56:17.659 INFO:tasks.rados.rados.0.smithi045.stdout:3231: done (1 left) 2021-07-31T01:56:17.660 INFO:tasks.rados.rados.0.smithi045.stdout:3234: done (0 left) 2021-07-31T01:56:17.660 INFO:tasks.rados.rados.0.smithi045.stdout:3235: snap_remove snap 309 2021-07-31T01:56:18.659 INFO:tasks.rados.rados.0.smithi045.stdout:3235: done (0 left) 2021-07-31T01:56:18.659 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 50 current snap is 311 2021-07-31T01:56:18.659 INFO:tasks.rados.rados.0.smithi045.stdout:3236: seq_num 998 ranges {2031616=720896,2752512=679936,3432448=598016,4030464=327680} 2021-07-31T01:56:18.674 INFO:tasks.rados.rados.0.smithi045.stdout:3236: writing smithi04569269-50 from 2031616 to 2752512 tid 1 2021-07-31T01:56:18.678 INFO:tasks.rados.rados.0.smithi045.stdout:3236: writing smithi04569269-50 from 2752512 to 3432448 tid 2 2021-07-31T01:56:18.681 INFO:tasks.rados.rados.0.smithi045.stdout:3236: writing smithi04569269-50 from 3432448 to 4030464 tid 3 2021-07-31T01:56:18.683 INFO:tasks.rados.rados.0.smithi045.stdout:3236: writing smithi04569269-50 from 4030464 to 4358144 tid 4 2021-07-31T01:56:18.683 INFO:tasks.rados.rados.0.smithi045.stdout:3237: delete oid 42 current snap is 311 2021-07-31T01:56:18.683 INFO:tasks.rados.rados.0.smithi045.stdout:3236: finishing write tid 1 to smithi04569269-50 2021-07-31T01:56:18.684 INFO:tasks.rados.rados.0.smithi045.stdout:3236: finishing write tid 2 to smithi04569269-50 2021-07-31T01:56:18.686 INFO:tasks.rados.rados.0.smithi045.stdout:3236: finishing write tid 3 to smithi04569269-50 2021-07-31T01:56:18.686 INFO:tasks.rados.rados.0.smithi045.stdout:3236: finishing write tid 4 to smithi04569269-50 2021-07-31T01:56:18.687 INFO:tasks.rados.rados.0.smithi045.stdout:3236: finishing write tid 5 to smithi04569269-50 2021-07-31T01:56:18.690 INFO:tasks.rados.rados.0.smithi045.stdout:3236: finishing write tid 6 to smithi04569269-50 2021-07-31T01:56:18.690 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 50 v 1714 (ObjNum 998 snap 311 seq_num 998) dirty exists 2021-07-31T01:56:18.690 INFO:tasks.rados.rados.0.smithi045.stdout:3236: left oid 50 (ObjNum 998 snap 311 seq_num 998) 2021-07-31T01:56:18.690 INFO:tasks.rados.rados.0.smithi045.stdout:3236: done (1 left) 2021-07-31T01:56:18.691 INFO:tasks.rados.rados.0.smithi045.stdout:3237: done (0 left) 2021-07-31T01:56:18.691 INFO:tasks.rados.rados.0.smithi045.stdout:3238: snap_create 2021-07-31T01:56:19.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:18 smithi154 conmon[34553]: cluster 2021-07-31T01:56:17.652201+0000 mon.a (mon.0 2021-07-31T01:56:19.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:18 smithi154 conmon[34553]: ) 1329 : cluster [DBG] osdmap e666: 8 total, 8 up, 8 in 2021-07-31T01:56:19.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:18 smithi154 conmon[34553]: cluster 2021-07-31T01:56:17.922016+0000 mgr.x (mgr.24542) 385 : cluster [DBG] pgmap v785: 25 pgs: 25 active+clean; 553 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:56:19.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:18 smithi045 conmon[39718]: cluster 2021-07-31T01:56:17.652201+0000 mon.a 2021-07-31T01:56:19.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:18 smithi045 conmon[39718]: (mon.0) 1329 : cluster [DBG] osdmap e666: 8 total, 8 up, 8 in 2021-07-31T01:56:19.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:18 smithi045 conmon[39718]: cluster 2021-07-31T01:56:17.922016+0000 2021-07-31T01:56:19.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:18 smithi045 conmon[39718]: mgr.x (mgr.24542) 385 : cluster [DBG] pgmap v785: 25 pgs: 25 active+clean; 553 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:56:19.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:18 smithi045 conmon[33655]: cluster 2021-07-31T01:56:17.652201+0000 mon.a (mon.0 2021-07-31T01:56:19.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:18 smithi045 conmon[33655]: ) 1329 : cluster [DBG] osdmap e666: 8 total, 8 up, 8 in 2021-07-31T01:56:19.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:18 smithi045 conmon[33655]: cluster 2021-07-31T01:56:17.922016+0000 mgr.x (mgr.24542) 385 : cluster [DBG] 2021-07-31T01:56:19.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:18 smithi045 conmon[33655]: pgmap v785: 25 pgs: 25 active+clean; 553 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:56:19.663 INFO:tasks.rados.rados.0.smithi045.stdout:3238: done (0 left) 2021-07-31T01:56:19.663 INFO:tasks.rados.rados.0.smithi045.stdout:3239: snap_create 2021-07-31T01:56:20.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:19 smithi154 conmon[34553]: cluster 2021-07-31T01:56:18.657706+0000 mon.a (mon.0) 1330 : cluster [DBG] osdmap e667: 8 total, 8 up, 8 in 2021-07-31T01:56:20.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:19 smithi045 conmon[39718]: cluster 2021-07-31T01:56:18.657706+0000 mon.a 2021-07-31T01:56:20.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:19 smithi045 conmon[39718]: (mon.0) 1330 : cluster [DBG] osdmap e667: 8 total, 8 up, 8 in 2021-07-31T01:56:20.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:19 smithi045 conmon[33655]: cluster 2021-07-31T01:56:18.657706+0000 mon.a (mon.0) 1330 2021-07-31T01:56:20.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:19 smithi045 conmon[33655]: : cluster [DBG] osdmap e667: 8 total, 8 up, 8 in 2021-07-31T01:56:20.671 INFO:tasks.rados.rados.0.smithi045.stdout:3239: done (0 left) 2021-07-31T01:56:20.671 INFO:tasks.rados.rados.0.smithi045.stdout:3240: rollback oid 11 current snap is 313 2021-07-31T01:56:20.672 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 11 to 303 2021-07-31T01:56:20.672 INFO:tasks.rados.rados.0.smithi045.stdout:3241: rollback oid 32 current snap is 313 2021-07-31T01:56:20.672 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 32 to 263 2021-07-31T01:56:20.672 INFO:tasks.rados.rados.0.smithi045.stdout:3242: delete oid 15 current snap is 313 2021-07-31T01:56:20.676 INFO:tasks.rados.rados.0.smithi045.stdout:3242: done (2 left) 2021-07-31T01:56:20.676 INFO:tasks.rados.rados.0.smithi045.stdout:3243: rollback oid 41 current snap is 313 2021-07-31T01:56:20.676 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 41 to 300 2021-07-31T01:56:20.677 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 46 current snap is 313 2021-07-31T01:56:20.677 INFO:tasks.rados.rados.0.smithi045.stdout:3244: seq_num 1001 ranges {3391488=770048,4161536=729088,4890624=770048,5660672=106496} 2021-07-31T01:56:20.699 INFO:tasks.rados.rados.0.smithi045.stdout:3244: writing smithi04569269-46 from 3391488 to 4161536 tid 1 2021-07-31T01:56:20.703 INFO:tasks.rados.rados.0.smithi045.stdout:3244: writing smithi04569269-46 from 4161536 to 4890624 tid 2 2021-07-31T01:56:20.707 INFO:tasks.rados.rados.0.smithi045.stdout:3244: writing smithi04569269-46 from 4890624 to 5660672 tid 3 2021-07-31T01:56:20.708 INFO:tasks.rados.rados.0.smithi045.stdout:3244: writing smithi04569269-46 from 5660672 to 5767168 tid 4 2021-07-31T01:56:20.709 INFO:tasks.rados.rados.0.smithi045.stdout:3245: snap_create 2021-07-31T01:56:20.709 INFO:tasks.rados.rados.0.smithi045.stdout:3240: finishing rollback tid 0 to smithi04569269-11 2021-07-31T01:56:20.709 INFO:tasks.rados.rados.0.smithi045.stdout:3240: finishing rollback tid 1 to smithi04569269-11 2021-07-31T01:56:20.709 INFO:tasks.rados.rados.0.smithi045.stdout:3240: finishing rollback tid 2 to smithi04569269-11 2021-07-31T01:56:20.709 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 11 v 2018 (ObjNum 976 snap 299 seq_num 976) dirty exists 2021-07-31T01:56:20.711 INFO:tasks.rados.rados.0.smithi045.stdout:3241: finishing rollback tid 0 to smithi04569269-32 2021-07-31T01:56:20.712 INFO:tasks.rados.rados.0.smithi045.stdout:3241: finishing rollback tid 1 to smithi04569269-32 2021-07-31T01:56:20.712 INFO:tasks.rados.rados.0.smithi045.stdout:3243: finishing rollback tid 0 to smithi04569269-41 2021-07-31T01:56:20.712 INFO:tasks.rados.rados.0.smithi045.stdout:3241: finishing rollback tid 2 to smithi04569269-32 2021-07-31T01:56:20.712 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 32 v 1855 (ObjNum 754 snap 233 seq_num 754) dirty exists 2021-07-31T01:56:20.712 INFO:tasks.rados.rados.0.smithi045.stdout:3243: finishing rollback tid 1 to smithi04569269-41 2021-07-31T01:56:20.713 INFO:tasks.rados.rados.0.smithi045.stdout:3243: finishing rollback tid 2 to smithi04569269-41 2021-07-31T01:56:20.713 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 41 v 1689 (ObjNum 948 snap 286 seq_num 948) dirty exists 2021-07-31T01:56:20.713 INFO:tasks.rados.rados.0.smithi045.stdout:3244: finishing write tid 1 to smithi04569269-46 2021-07-31T01:56:20.713 INFO:tasks.rados.rados.0.smithi045.stdout:3244: finishing write tid 2 to smithi04569269-46 2021-07-31T01:56:20.716 INFO:tasks.rados.rados.0.smithi045.stdout:3244: finishing write tid 3 to smithi04569269-46 2021-07-31T01:56:20.716 INFO:tasks.rados.rados.0.smithi045.stdout:3244: finishing write tid 4 to smithi04569269-46 2021-07-31T01:56:20.716 INFO:tasks.rados.rados.0.smithi045.stdout:3244: finishing write tid 5 to smithi04569269-46 2021-07-31T01:56:20.716 INFO:tasks.rados.rados.0.smithi045.stdout:3244: finishing write tid 6 to smithi04569269-46 2021-07-31T01:56:20.716 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 46 v 1848 (ObjNum 1001 snap 313 seq_num 1001) dirty exists 2021-07-31T01:56:20.717 INFO:tasks.rados.rados.0.smithi045.stdout:3244: left oid 46 (ObjNum 1001 snap 313 seq_num 1001) 2021-07-31T01:56:21.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:20 smithi154 conmon[34553]: cluster 2021-07-31T01:56:19.662694+0000 mon.a (mon.0) 1331 : cluster 2021-07-31T01:56:21.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:20 smithi154 conmon[34553]: [DBG] osdmap e668: 8 total, 8 up, 8 in 2021-07-31T01:56:21.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:20 smithi154 conmon[34553]: cluster 2021-07-31T01:56:19.922467+0000 mgr.x (mgr.24542) 386 : cluster [DBG] pgmap v788: 25 pgs: 25 active+clean; 556 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 1.1 MiB/s wr, 4 op/s 2021-07-31T01:56:21.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:20 smithi045 conmon[33655]: cluster 2021-07-31T01:56:19.662694 2021-07-31T01:56:21.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:20 smithi045 conmon[33655]: +0000 mon.a (mon.0) 1331 : cluster [DBG] osdmap e668: 8 total, 8 up, 8 in 2021-07-31T01:56:21.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:20 smithi045 conmon[33655]: cluster 2021-07-31T01:56:19 2021-07-31T01:56:21.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:20 smithi045 conmon[33655]: .922467+0000 mgr.x (mgr.24542) 386 : cluster [DBG] pgmap v788: 25 pgs: 25 active+clean; 556 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 1.1 MiB/s wr, 4 op/s 2021-07-31T01:56:21.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:20 smithi045 conmon[39718]: cluster 2021-07-31T01:56:19 2021-07-31T01:56:21.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:20 smithi045 conmon[39718]: .662694+0000 mon.a (mon.0) 1331 : cluster [DBG] osdmap e668: 8 total, 8 up, 8 in 2021-07-31T01:56:21.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:20 smithi045 conmon[39718]: cluster 2021-07-31T01:56:21.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:20 smithi045 conmon[39718]: 2021-07-31T01:56:19. 2021-07-31T01:56:21.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:20 smithi045 conmon[39718]: 922467+0000 mgr.x (mgr.24542) 386 2021-07-31T01:56:21.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:20 smithi045 conmon[39718]: : cluster [DBG] pgmap v788: 25 pgs: 25 active+clean; 556 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 1.1 MiB/s wr, 4 op/s 2021-07-31T01:56:21.646 INFO:tasks.rados.rados.0.smithi045.stdout:3240: done (4 left) 2021-07-31T01:56:21.647 INFO:tasks.rados.rados.0.smithi045.stdout:3241: done (3 left) 2021-07-31T01:56:21.647 INFO:tasks.rados.rados.0.smithi045.stdout:3243: done (2 left) 2021-07-31T01:56:21.647 INFO:tasks.rados.rados.0.smithi045.stdout:3244: done (1 left) 2021-07-31T01:56:21.647 INFO:tasks.rados.rados.0.smithi045.stdout:3245: done (0 left) 2021-07-31T01:56:21.648 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 13 current snap is 314 2021-07-31T01:56:21.648 INFO:tasks.rados.rados.0.smithi045.stdout:3246: seq_num 1002 ranges {0=786432,786432=614400,1400832=581632,1982464=409600} 2021-07-31T01:56:21.651 INFO:tasks.rados.rados.0.smithi045.stdout:3246: writing smithi04569269-13 from 0 to 786432 tid 1 2021-07-31T01:56:21.654 INFO:tasks.rados.rados.0.smithi045.stdout:3246: writing smithi04569269-13 from 786432 to 1400832 tid 2 2021-07-31T01:56:21.658 INFO:tasks.rados.rados.0.smithi045.stdout:3246: writing smithi04569269-13 from 1400832 to 1982464 tid 3 2021-07-31T01:56:21.660 INFO:tasks.rados.rados.0.smithi045.stdout:3246: writing smithi04569269-13 from 1982464 to 2392064 tid 4 2021-07-31T01:56:21.661 INFO:tasks.rados.rados.0.smithi045.stdout:3247: rollback oid 30 current snap is 314 2021-07-31T01:56:21.661 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 30 to 303 2021-07-31T01:56:21.661 INFO:tasks.rados.rados.0.smithi045.stdout:3246: finishing write tid 1 to smithi04569269-13 2021-07-31T01:56:21.661 INFO:tasks.rados.rados.0.smithi045.stdout:3248: snap_create 2021-07-31T01:56:21.662 INFO:tasks.rados.rados.0.smithi045.stdout:3246: finishing write tid 2 to smithi04569269-13 2021-07-31T01:56:21.664 INFO:tasks.rados.rados.0.smithi045.stdout:3246: finishing write tid 3 to smithi04569269-13 2021-07-31T01:56:21.664 INFO:tasks.rados.rados.0.smithi045.stdout:3246: finishing write tid 4 to smithi04569269-13 2021-07-31T01:56:21.665 INFO:tasks.rados.rados.0.smithi045.stdout:3246: finishing write tid 5 to smithi04569269-13 2021-07-31T01:56:21.665 INFO:tasks.rados.rados.0.smithi045.stdout:3246: finishing write tid 6 to smithi04569269-13 2021-07-31T01:56:21.666 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 13 v 1872 (ObjNum 1002 snap 314 seq_num 1002) dirty exists 2021-07-31T01:56:21.666 INFO:tasks.rados.rados.0.smithi045.stdout:3246: left oid 13 (ObjNum 1002 snap 314 seq_num 1002) 2021-07-31T01:56:21.667 INFO:tasks.rados.rados.0.smithi045.stdout:3247: finishing rollback tid 0 to smithi04569269-30 2021-07-31T01:56:21.667 INFO:tasks.rados.rados.0.smithi045.stdout:3247: finishing rollback tid 1 to smithi04569269-30 2021-07-31T01:56:21.667 INFO:tasks.rados.rados.0.smithi045.stdout:3247: finishing rollback tid 2 to smithi04569269-30 2021-07-31T01:56:21.667 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 30 v 1657 (ObjNum 936 snap 284 seq_num 936) dirty exists 2021-07-31T01:56:22.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:21 smithi154 conmon[34553]: cluster 2021-07-31T01:56:20.670141+0000 mon.a (mon.0) 1332 : cluster 2021-07-31T01:56:22.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:21 smithi154 conmon[34553]: [DBG] osdmap e669: 8 total, 8 up, 8 in 2021-07-31T01:56:22.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:21 smithi154 conmon[34553]: cluster 2021-07-31T01:56:21.645698+0000 mon.a (mon.0) 1333 : cluster [DBG] osdmap e670: 8 total, 8 up, 8 in 2021-07-31T01:56:22.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:21 smithi045 conmon[33655]: cluster 2021-07-31T01:56:20.670141+0000 mon.a (mon.0) 1332 : cluster [DBG] osdmap e669: 8 total, 8 up, 8 in 2021-07-31T01:56:22.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:21 smithi045 conmon[33655]: 2021-07-31T01:56:22.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:21 smithi045 conmon[33655]: cluster 2021-07-31T01:56:21.645698+0000 mon.a (mon.0) 1333 : cluster [DBG] osdmap e670: 8 total, 8 up, 8 in 2021-07-31T01:56:22.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:21 smithi045 conmon[39718]: cluster 2021-07-31T01:56:20.670141+0000 mon.a 2021-07-31T01:56:22.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:21 smithi045 conmon[39718]: (mon.0) 1332 : cluster [DBG] osdmap e669: 8 total, 8 up, 8 in 2021-07-31T01:56:22.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:21 smithi045 conmon[39718]: cluster 2021-07 2021-07-31T01:56:22.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:21 smithi045 conmon[39718]: -31T01:56:21.645698+0000 mon.a (mon.0 2021-07-31T01:56:22.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:21 smithi045 conmon[39718]: ) 1333 : cluster [DBG] osdmap e670: 8 total, 8 up, 8 in 2021-07-31T01:56:22.654 INFO:tasks.rados.rados.0.smithi045.stdout:3246: done (2 left) 2021-07-31T01:56:22.655 INFO:tasks.rados.rados.0.smithi045.stdout:3247: done (1 left) 2021-07-31T01:56:22.655 INFO:tasks.rados.rados.0.smithi045.stdout:3248: done (0 left) 2021-07-31T01:56:22.655 INFO:tasks.rados.rados.0.smithi045.stdout:3249: read oid 41 snap -1 2021-07-31T01:56:22.655 INFO:tasks.rados.rados.0.smithi045.stdout:3249: expect (ObjNum 948 snap 286 seq_num 948) 2021-07-31T01:56:22.656 INFO:tasks.rados.rados.0.smithi045.stdout:3250: delete oid 28 current snap is 315 2021-07-31T01:56:22.659 INFO:tasks.rados.rados.0.smithi045.stdout:3250: done (1 left) 2021-07-31T01:56:22.660 INFO:tasks.rados.rados.0.smithi045.stdout:3251: snap_create 2021-07-31T01:56:22.666 INFO:tasks.rados.rados.0.smithi045.stdout:3249: expect (ObjNum 948 snap 286 seq_num 948) 2021-07-31T01:56:23.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:22 smithi154 conmon[34553]: cluster 2021-07-31T01:56:21.922966+0000 mgr.x (mgr.24542) 387 : cluster 2021-07-31T01:56:23.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:22 smithi154 conmon[34553]: [DBG] pgmap v791: 25 pgs: 25 active+clean; 556 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 1.1 MiB/s wr, 4 op/s 2021-07-31T01:56:23.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:22 smithi154 conmon[34553]: cluster 2021-07-31T01:56:22.653749+0000 mon.a (mon.0) 1334 : cluster [DBG] osdmap e671: 8 total, 8 up, 8 in 2021-07-31T01:56:23.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:22 smithi045 conmon[33655]: cluster 2021-07-31T01:56:21.922966+0000 mgr.x ( 2021-07-31T01:56:23.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:22 smithi045 conmon[33655]: mgr.24542) 387 : cluster [DBG] pgmap v791: 25 pgs: 25 active+clean; 556 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 1.1 MiB/s wr, 4 op/s 2021-07-31T01:56:23.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:22 smithi045 conmon[33655]: cluster 2021-07-31T01:56:22.653749+0000 mon.a (mon.0) 1334 : cluster [DBG] osdmap e671: 8 total, 8 up, 8 in 2021-07-31T01:56:23.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:22 smithi045 conmon[39718]: cluster 2021-07-31T01:56:21.922966+0000 2021-07-31T01:56:23.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:22 smithi045 conmon[39718]: mgr.x (mgr.24542) 387 : cluster [DBG] pgmap v791: 25 pgs: 25 active+clean; 556 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 1.1 MiB/s wr, 4 op/s 2021-07-31T01:56:23.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:22 smithi045 conmon[39718]: cluster 2021-07-31T01:56:23.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:22 smithi045 conmon[39718]: 2021-07-31T01:56:22.653749+0000 mon.a (mon.0) 1334 : cluster [DBG] 2021-07-31T01:56:23.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:22 smithi045 conmon[39718]: osdmap e671: 8 total, 8 up, 8 in 2021-07-31T01:56:23.656 INFO:tasks.rados.rados.0.smithi045.stdout:3249: done (1 left) 2021-07-31T01:56:23.656 INFO:tasks.rados.rados.0.smithi045.stdout:3251: done (0 left) 2021-07-31T01:56:23.656 INFO:tasks.rados.rados.0.smithi045.stdout:3252: snap_remove snap 283 2021-07-31T01:56:23.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:56:23 smithi045 conmon[61210]: level=error ts=2021-07-31T01:56:23.524Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:56:23.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:56:23 smithi045 conmon[61210]: level=error ts=2021-07-31T01:56:23.524Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:56:23.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:56:23 smithi045 conmon[61210]: level=error ts=2021-07-31T01:56:23.524Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T01:56:24.664 INFO:tasks.rados.rados.0.smithi045.stdout:3252: done (0 left) 2021-07-31T01:56:24.665 INFO:tasks.rados.rados.0.smithi045.stdout:3253: snap_create 2021-07-31T01:56:25.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:24 smithi154 conmon[34553]: cluster 2021-07-31T01:56:25.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:24 smithi154 conmon[34553]: 2021-07-31T01:56:25.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:24 smithi154 conmon[34553]: 2021 2021-07-31T01:56:25.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:24 smithi154 conmon[34553]: - 2021-07-31T01:56:25.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:24 smithi154 conmon[34553]: 07 2021-07-31T01:56:25.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:24 smithi154 conmon[34553]: - 2021-07-31T01:56:25.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:24 smithi154 conmon[34553]: 31 2021-07-31T01:56:25.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:24 smithi154 conmon[34553]: T 2021-07-31T01:56:25.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:24 smithi154 conmon[34553]: 01 2021-07-31T01:56:25.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:24 smithi154 conmon[34553]: :56:23.655245+0000 mon.a (mon.0) 1335 : cluster [DBG] osdmap e672: 8 total, 8 up, 8 in 2021-07-31T01:56:25.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:24 smithi154 conmon[34553]: cluster 2021-07-31T01:56:23. 2021-07-31T01:56:25.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:24 smithi154 conmon[34553]: 923479+0000 mgr.x (mgr.24542) 388 : cluster [DBG] pgmap v794: 25 pgs: 25 active+clean; 564 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 6 op/s 2021-07-31T01:56:25.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:24 smithi045 conmon[39718]: cluster 2021-07-31T01:56:23.655245+0000 mon.a (mon. 2021-07-31T01:56:25.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:24 smithi045 conmon[39718]: 0) 1335 : cluster [DBG] osdmap e672: 8 total, 8 up, 8 in 2021-07-31T01:56:25.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:24 smithi045 conmon[39718]: cluster 2021-07-31T01:56: 2021-07-31T01:56:25.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:24 smithi045 conmon[39718]: 23.923479+0000 mgr.x (mgr.24542) 388 : cluster [DBG] pgmap v794: 25 pgs: 25 active+clean; 564 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 6 op/s 2021-07-31T01:56:25.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:24 smithi045 conmon[33655]: cluster 2021-07-31T01:56:23.655245+0000 mon.a (mon.0) 1335 : cluster 2021-07-31T01:56:25.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:24 smithi045 conmon[33655]: [DBG] osdmap e672: 8 total, 8 up, 8 in 2021-07-31T01:56:25.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:24 smithi045 conmon[33655]: cluster 2021-07-31T01:56:23.923479+0000 mgr.x (mgr.24542) 388 : cluster [DBG] pgmap v794: 25 pgs: 25 active+clean; 564 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 6 op/s 2021-07-31T01:56:25.663 INFO:tasks.rados.rados.0.smithi045.stdout:3253: done (0 left) 2021-07-31T01:56:25.663 INFO:tasks.rados.rados.0.smithi045.stdout:3254: copy_from oid 13 from oid 23 current snap is 317 2021-07-31T01:56:25.663 INFO:tasks.rados.rados.0.smithi045.stdout:3255: delete oid 7 current snap is 317 2021-07-31T01:56:25.665 INFO:tasks.rados.rados.0.smithi045.stdout:3255: done (1 left) 2021-07-31T01:56:25.665 INFO:tasks.rados.rados.0.smithi045.stdout:3256: snap_create 2021-07-31T01:56:25.676 INFO:tasks.rados.rados.0.smithi045.stdout:3254: finishing copy_from to smithi04569269-13 2021-07-31T01:56:25.676 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 13 v 1644 (ObjNum 917 snap 278 seq_num 917) dirty exists 2021-07-31T01:56:25.676 INFO:tasks.rados.rados.0.smithi045.stdout:3254: finishing copy_from racing read to smithi04569269-13 2021-07-31T01:56:26.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:25 smithi154 conmon[34553]: cluster 2021-07-31T01:56:24.661111+0000 mon.a (mon.0 2021-07-31T01:56:26.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:25 smithi154 conmon[34553]: ) 1336 : cluster [DBG] osdmap e673: 8 total, 8 up, 8 in 2021-07-31T01:56:26.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:25 smithi045 conmon[33655]: cluster 2021-07-31T01:56 2021-07-31T01:56:26.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:25 smithi045 conmon[33655]: :24.661111+0000 mon.a (mon.0) 1336 : cluster [DBG] osdmap e673: 8 total, 8 up, 8 in 2021-07-31T01:56:26.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:25 smithi045 conmon[39718]: cluster 2021-07-31T01:56:24.661111+0000 mon.a (mon.0) 1336 2021-07-31T01:56:26.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:25 smithi045 conmon[39718]: : cluster [DBG] osdmap e673: 8 total, 8 up, 8 in 2021-07-31T01:56:26.648 INFO:tasks.rados.rados.0.smithi045.stdout:3254: done (1 left) 2021-07-31T01:56:26.648 INFO:tasks.rados.rados.0.smithi045.stdout:3256: done (0 left) 2021-07-31T01:56:26.648 INFO:tasks.rados.rados.0.smithi045.stdout:3257: snap_create 2021-07-31T01:56:27.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:26 smithi154 conmon[34553]: cluster 2021-07-31T01:56:25.662156+0000 mon.a (mon.0) 1337 : cluster 2021-07-31T01:56:27.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:26 smithi154 conmon[34553]: [DBG] osdmap e674: 8 total, 8 up, 8 in 2021-07-31T01:56:27.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:26 smithi154 conmon[34553]: cluster 2021-07-31T01:56:25.924014+0000 mgr.x (mgr.24542) 389 : cluster [DBG] pgmap v797: 25 pgs: 25 active+clean; 564 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 6 op/s 2021-07-31T01:56:27.036 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:26 smithi154 conmon[34553]: cluster 2021-07-31T01 2021-07-31T01:56:27.037 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:26 smithi154 conmon[34553]: :56:26.647425+0000 mon.a (mon.0) 1338 : cluster [DBG] osdmap e675: 8 total, 8 up, 8 in 2021-07-31T01:56:27.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:26 smithi045 conmon[33655]: cluster 2021-07-31T01:56:25.662156+0000 mon.a 2021-07-31T01:56:27.040 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:26 smithi045 conmon[33655]: (mon.0) 1337 : cluster [DBG] osdmap e674: 8 total, 8 up, 8 in 2021-07-31T01:56:27.041 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:26 smithi045 conmon[33655]: cluster 2021-07-31T01:56:25.924014+0000 mgr.x (mgr.24542) 389 : cluster [DBG] pgmap v797: 25 pgs: 25 active+clean; 564 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 6 op/s 2021-07-31T01:56:27.041 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:26 smithi045 conmon[33655]: cluster 2021-07-31T01:56:27.042 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:26 smithi045 conmon[33655]: 2021-07-31T01 2021-07-31T01:56:27.042 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:26 smithi045 conmon[33655]: :56:26.647425+0000 mon.a 2021-07-31T01:56:27.043 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:26 smithi045 conmon[33655]: (mon.0) 1338 : cluster [DBG] 2021-07-31T01:56:27.044 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:26 smithi045 conmon[33655]: osdmap e675: 8 total, 8 up, 8 in 2021-07-31T01:56:27.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:26 smithi045 conmon[39718]: cluster 2021-07-31T01:56:25.662156+0000 mon.a (mon.0) 1337 2021-07-31T01:56:27.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:26 smithi045 conmon[39718]: : cluster [DBG] osdmap e674: 8 total, 8 up, 8 in 2021-07-31T01:56:27.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:26 smithi045 conmon[39718]: cluster 2021-07-31T01:56:27.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:26 smithi045 conmon[39718]: 2021-07-31T01:56:25.924014+0000 2021-07-31T01:56:27.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:26 smithi045 conmon[39718]: mgr.x (mgr.24542) 389 : cluster [DBG] pgmap v797: 25 pgs: 25 active+clean; 564 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 6 op/s 2021-07-31T01:56:27.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:26 smithi045 conmon[39718]: 2021-07-31T01:56:27.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:26 smithi045 conmon[39718]: cluster 2021-07-31T01 2021-07-31T01:56:27.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:26 smithi045 conmon[39718]: :56:26 2021-07-31T01:56:27.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:26 smithi045 conmon[39718]: .647425+0000 mon.a (mon.0) 1338 2021-07-31T01:56:27.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:26 smithi045 conmon[39718]: : cluster [DBG] osdmap e675: 8 total, 8 up, 8 in 2021-07-31T01:56:27.654 INFO:tasks.rados.rados.0.smithi045.stdout:3257: done (0 left) 2021-07-31T01:56:27.654 INFO:tasks.rados.rados.0.smithi045.stdout:3258: snap_create 2021-07-31T01:56:28.660 INFO:tasks.rados.rados.0.smithi045.stdout:3258: done (0 left) 2021-07-31T01:56:28.661 INFO:tasks.rados.rados.0.smithi045.stdout:3259: snap_create 2021-07-31T01:56:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:28 smithi154 conmon[34553]: cluster 2021-07-31T01:56:27.653608+0000 mon.a (mon.0) 1339 : 2021-07-31T01:56:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:28 smithi154 conmon[34553]: cluster [DBG] osdmap e676: 8 total, 8 up, 8 in 2021-07-31T01:56:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:28 smithi154 conmon[34553]: cluster 2021-07-31T01:56:27.924426+0000 mgr.x (mgr.24542) 390 : cluster [DBG] pgmap v800: 25 pgs: 25 active+clean; 564 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:56:29.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:28 smithi154 conmon[34553]: audit 2021-07-31T01 2021-07-31T01:56:29.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:28 smithi154 conmon[34553]: :56:27.970764+0000 mon.b (mon.2 2021-07-31T01:56:29.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:28 smithi154 conmon[34553]: ) 107 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:56:29.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:28 smithi154 conmon[34553]: audit 2021-07-31 2021-07-31T01:56:29.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:28 smithi154 conmon[34553]: T01:56:27.971350+0000 mon.a (mon.0) 1340 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:56:29.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:28 smithi154 conmon[34553]: 2021-07-31T01:56:29.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:28 smithi154 conmon[34553]: audit 2021-07-31T01:56:29.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:28 smithi154 conmon[34553]: 2021-07-31T01:56:27.986083+0000 mon.b (mon.2) 108 : 2021-07-31T01:56:29.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:28 smithi154 conmon[34553]: audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T01:56:29.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:28 smithi154 conmon[34553]: audit 2021-07-31T01:56:29.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:28 smithi154 conmon[34553]: 2021-07-31T01:56:27.986553+0000 mon.a (mon.0) 1341 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T01:56:29.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:28 smithi045 conmon[33655]: cluster 2021-07-31T01:56:27.653608+0000 mon.a (mon.0) 1339 : 2021-07-31T01:56:29.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:28 smithi045 conmon[33655]: cluster [DBG] osdmap e676: 8 total, 8 up, 8 in 2021-07-31T01:56:29.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:28 smithi045 conmon[33655]: cluster 2021-07-31T01:56:27.924426+0000 mgr.x (mgr.24542) 390 : cluster [DBG] pgmap v800: 25 pgs: 25 active+clean; 564 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:56:29.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:28 smithi045 conmon[33655]: audit 2021-07-31T01:56:27.970764+0000 mon.b (mon.2) 107 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:56:29.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:28 smithi045 conmon[33655]: 2021-07-31T01:56:29.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:28 smithi045 conmon[33655]: audit 2021-07-31T01: 2021-07-31T01:56:29.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:28 smithi045 conmon[33655]: 56:27.971350+0000 mon.a (mon.0 2021-07-31T01:56:29.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:28 smithi045 conmon[33655]: ) 1340 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:56:29.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:28 smithi045 conmon[33655]: audit 2021- 2021-07-31T01:56:29.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:28 smithi045 conmon[33655]: 07-31T01:56:27.986083+0000 2021-07-31T01:56:29.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:28 smithi045 conmon[33655]: mon.b (mon.2) 108 : audit [INF] 2021-07-31T01:56:29.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:28 smithi045 conmon[33655]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T01:56:29.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:28 smithi045 conmon[33655]: audit 2021-07- 2021-07-31T01:56:29.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:28 smithi045 conmon[33655]: 31T01:56:27.986553+0000 mon.a 2021-07-31T01:56:29.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:28 smithi045 conmon[33655]: (mon.0) 1341 : audit [INF] 2021-07-31T01:56:29.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:28 smithi045 conmon[33655]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T01:56:29.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:28 smithi045 conmon[39718]: cluster 2021-07-31T01:56:27.653608 2021-07-31T01:56:29.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:28 smithi045 conmon[39718]: +0000 mon.a (mon.0) 1339 : cluster [DBG] osdmap e676: 8 total, 8 up, 8 in 2021-07-31T01:56:29.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:28 smithi045 conmon[39718]: cluster 2021-07-31T01:56:29.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:28 smithi045 conmon[39718]: 2021-07-31T01:56:27.924426+0000 mgr.x (mgr.24542) 390 : cluster [DBG] pgmap v800: 25 pgs: 25 active+clean; 564 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:56:29.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:28 smithi045 conmon[39718]: 2021-07-31T01:56:29.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:28 smithi045 conmon[39718]: audit 2021-07-31T01:56:27.970764+0000 mon.b ( 2021-07-31T01:56:29.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:28 smithi045 conmon[39718]: mon.2) 107 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:56:29.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:28 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T01:56:29.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:28 smithi045 conmon[39718]: 31T01:56:27.971350+0000 mon.a (mon.0) 1340 2021-07-31T01:56:29.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:28 smithi045 conmon[39718]: : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:56:29.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:28 smithi045 conmon[39718]: audit 2021- 2021-07-31T01:56:29.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:28 smithi045 conmon[39718]: 07-31T01:56:27.986083+0000 mon.b (mon.2) 108 : 2021-07-31T01:56:29.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:28 smithi045 conmon[39718]: audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T01:56:29.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:28 smithi045 conmon[39718]: audit 2021-07-31T01:56:27.986553+0000 mon.a ( 2021-07-31T01:56:29.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:28 smithi045 conmon[39718]: mon.0) 1341 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T01:56:29.503 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: debug 2021-07-31T01:56:29.113+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T01:56:29.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:56:29.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:56:29.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: yield 2021-07-31T01:56:29.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:56:29.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:56:29.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:56:29.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T01:56:29.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:56:29.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T01:56:29.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:56:29.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T01:56:29.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:56:29.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: task.result() 2021-07-31T01:56:29.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T01:56:29.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: debug 2021-07-31T01:56:29.124+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T01:56:29.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:56:29.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:56:29.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: yield 2021-07-31T01:56:29.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:56:29.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:56:29.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:56:29.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T01:56:29.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:56:29.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T01:56:29.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:56:29.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T01:56:29.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:56:29.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: task.result() 2021-07-31T01:56:29.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T01:56:29.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: debug 2021-07-31T01:56:29.150+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T01:56:29.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:56:29.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:56:29.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: yield 2021-07-31T01:56:29.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:56:29.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:56:29.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:56:29.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T01:56:29.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:56:29.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T01:56:29.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:56:29.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T01:56:29.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:56:29.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: task.result() 2021-07-31T01:56:29.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T01:56:29.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: debug 2021-07-31T01:56:29.152+0000 7fcb97969700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:56:29.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:56:29.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:56:29.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: yield 2021-07-31T01:56:29.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:56:29.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:56:29.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:56:29.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T01:56:29.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:56:29.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T01:56:29.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:56:29.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T01:56:29.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:56:29.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: task.result() 2021-07-31T01:56:29.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T01:56:29.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: 2021-07-31T01:56:29.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T01:56:29.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: 2021-07-31T01:56:29.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:56:29.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:56:29.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: return f(*arg) 2021-07-31T01:56:29.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:56:29.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:56:29.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:56:29.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:56:29.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:56:29.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:56:29.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:56:29.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T01:56:29.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:56:29.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: raise self._exception 2021-07-31T01:56:29.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:56:29.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:56:29.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:56:29.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T01:56:29.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:56:29.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: raise 2021-07-31T01:56:29.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:56:29.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T01:56:29.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:56:29.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:56:29.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T01:56:29.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: debug 2021-07-31T01:56:29.159+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T01:56:29.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:56:29.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:56:29.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: yield 2021-07-31T01:56:29.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:56:29.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:56:29.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:56:29.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T01:56:29.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:56:29.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T01:56:29.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:56:29.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T01:56:29.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:56:29.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: task.result() 2021-07-31T01:56:29.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T01:56:29.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: debug 2021-07-31T01:56:29.159+0000 7fcb9816a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:56:29.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:56:29.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:56:29.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: yield 2021-07-31T01:56:29.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:56:29.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:56:29.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:56:29.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T01:56:29.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:56:29.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T01:56:29.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:56:29.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T01:56:29.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:56:29.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: task.result() 2021-07-31T01:56:29.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T01:56:29.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: 2021-07-31T01:56:29.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T01:56:29.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: 2021-07-31T01:56:29.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:56:29.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:56:29.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: return f(*arg) 2021-07-31T01:56:29.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:56:29.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:56:29.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:56:29.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:56:29.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:56:29.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:56:29.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:56:29.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T01:56:29.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:56:29.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: raise self._exception 2021-07-31T01:56:29.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:56:29.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:56:29.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:56:29.535 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T01:56:29.535 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:56:29.535 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: raise 2021-07-31T01:56:29.535 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:56:29.535 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T01:56:29.536 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:56:29.536 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:56:29.536 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T01:56:29.663 INFO:tasks.rados.rados.0.smithi045.stdout:3259: done (0 left) 2021-07-31T01:56:29.663 INFO:tasks.rados.rados.0.smithi045.stdout:3260: read oid 3 snap 319 2021-07-31T01:56:29.663 INFO:tasks.rados.rados.0.smithi045.stdout:3260: expect (ObjNum 992 snap 308 seq_num 992) 2021-07-31T01:56:29.664 INFO:tasks.rados.rados.0.smithi045.stdout:3261: delete oid 11 current snap is 321 2021-07-31T01:56:29.669 INFO:tasks.rados.rados.0.smithi045.stdout:3261: done (1 left) 2021-07-31T01:56:29.669 INFO:tasks.rados.rados.0.smithi045.stdout:3262: snap_remove snap 311 2021-07-31T01:56:30.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[34553]: cluster 2021-07-31T01:56:28.659534+0000 mon.a (mon.0) 1342 : 2021-07-31T01:56:30.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[34553]: cluster [DBG] osdmap e677: 8 total, 8 up, 8 in 2021-07-31T01:56:30.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[34553]: audit 2021-07-31T01:56:29.066508+0000 mon.b (mon.2) 109 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:56:30.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[34553]: audit 2021-07-31T01:56:30.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[34553]: 2021-07-31T01:56:29.068024+0000 mon.b (mon.2) 110 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:56:30.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[34553]: audit 2021-07-31T01:56: 2021-07-31T01:56:30.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[34553]: 29.068890+0000 mon.b (mon.2) 111 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:56:30.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:29 smithi154 conmon[34553]: 2021-07-31T01:56:30.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:29 smithi045 conmon[33655]: cluster 2021-07-31T01:56:28.659534+0000 2021-07-31T01:56:30.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:29 smithi045 conmon[33655]: mon.a (mon.0) 1342 : cluster [DBG] osdmap e677: 8 total, 8 up, 8 in 2021-07-31T01:56:30.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:29 smithi045 conmon[33655]: audit 2021-07-31T01:56:29.066508+0000 mon.b (mon.2) 109 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:56:30.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:29 smithi045 conmon[33655]: audit 2021-07-31T01:56:29.068024+0000 mon.b (mon.2) 110 : audit 2021-07-31T01:56:30.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:29 smithi045 conmon[33655]: [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:56:30.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:29 smithi045 conmon[33655]: audit 2021-07-31T01:56:30.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:29 smithi045 conmon[33655]: 2021-07-31T01:56:29.068890+0000 mon.b (mon.2) 111 : audit [INF] 2021-07-31T01:56:30.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:29 smithi045 conmon[33655]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:56:30.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:29 smithi045 conmon[39718]: cluster 2021-07-31T01:56:30.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:29 smithi045 conmon[39718]: 2021-07-31T01:56:30.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:29 smithi045 conmon[39718]: 2021 2021-07-31T01:56:30.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:29 smithi045 conmon[39718]: - 2021-07-31T01:56:30.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:29 smithi045 conmon[39718]: 07 2021-07-31T01:56:30.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:29 smithi045 conmon[39718]: -31T01:56:28.659534+0000 mon.a ( 2021-07-31T01:56:30.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:29 smithi045 conmon[39718]: mon.0) 1342 : cluster [DBG] osdmap e677: 8 total, 8 up, 8 in 2021-07-31T01:56:30.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:29 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T01:56:30.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:29 smithi045 conmon[39718]: 31T01:56:29.066508+0000 mon.b (mon. 2021-07-31T01:56:30.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:29 smithi045 conmon[39718]: 2) 109 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:56:30.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:29 smithi045 conmon[39718]: 2021-07-31T01:56:30.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:29 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T01:56:30.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:29 smithi045 conmon[39718]: T01:56:29.068024+0000 mon.b (mon.2) 110 : audit 2021-07-31T01:56:30.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:29 smithi045 conmon[39718]: [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:56:30.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:29 smithi045 conmon[39718]: audit 2021-07-31T01:56 2021-07-31T01:56:30.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:29 smithi045 conmon[39718]: :29.068890+0000 mon.b (mon.2) 111 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:56:30.674 INFO:tasks.rados.rados.0.smithi045.stdout:3262: done (1 left) 2021-07-31T01:56:30.675 INFO:tasks.rados.rados.0.smithi045.stdout:3263: read oid 2 snap -1 2021-07-31T01:56:30.675 INFO:tasks.rados.rados.0.smithi045.stdout:3263: expect deleted 2021-07-31T01:56:30.675 INFO:tasks.rados.rados.0.smithi045.stdout:3264: snap_create 2021-07-31T01:56:30.675 INFO:tasks.rados.rados.0.smithi045.stdout:3260: expect (ObjNum 992 snap 308 seq_num 992) 2021-07-31T01:56:31.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: cephadm 2021-07-31T01:56:29.069639+0000 mgr.x (mgr.24542) 391 : cephadm 2021-07-31T01:56:31.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T01:56:31.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: cephadm 2021-07-31T01:56:29.114057+0000 mgr.x (mgr.24542) 392 : cephadm [ERR] 2021-07-31T01:56:31.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:56:31.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:56:31.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: yield 2021-07-31T01:56:31.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:56:31.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:56:31.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:56:31.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T01:56:31.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:56:31.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T01:56:31.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:56:31.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T01:56:31.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:56:31.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: task.result() 2021-07-31T01:56:31.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T01:56:31.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: cephadm 2021-07-31T01:56:29.114538+0000 mgr.x (mgr.24542) 393 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T01:56:31.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: cephadm 2021-07-31T01:56:29.125526+0000 mgr.x (mgr.24542) 394 : cephadm [ERR] 2021-07-31T01:56:31.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:56:31.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:56:31.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: yield 2021-07-31T01:56:31.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:56:31.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:56:31.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:56:31.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T01:56:31.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:56:31.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T01:56:31.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:56:31.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T01:56:31.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:56:31.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: task.result() 2021-07-31T01:56:31.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T01:56:31.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: cephadm 2021-07-31T01:56:29.126395+0000 mgr.x (mgr.24542) 395 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T01:56:31.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: cephadm 2021- 2021-07-31T01:56:31.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: 07-31T01:56:29.151438+0000 mgr.x (mgr.24542) 396 : cephadm [ERR] 2021-07-31T01:56:31.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:56:31.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:56:31.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: yield 2021-07-31T01:56:31.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:56:31.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:56:31.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:56:31.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T01:56:31.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:56:31.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T01:56:31.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:56:31.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T01:56:31.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:56:31.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: task.result() 2021-07-31T01:56:31.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T01:56:31.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: cephadm 2021-07-31T01:56:29.153116+0000 mgr.x (mgr.24542) 397 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:56:31.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:56:31.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:56:31.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: yield 2021-07-31T01:56:31.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:56:31.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:56:31.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:56:31.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T01:56:31.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:56:31.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T01:56:31.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:56:31.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T01:56:31.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:56:31.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: task.result() 2021-07-31T01:56:31.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T01:56:31.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: 2021-07-31T01:56:31.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T01:56:31.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: 2021-07-31T01:56:31.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:56:31.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:56:31.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: return f(*arg) 2021-07-31T01:56:31.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:56:31.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:56:31.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:56:31.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:56:31.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:56:31.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:56:31.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:56:31.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T01:56:31.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:56:31.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: raise self._exception 2021-07-31T01:56:31.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:56:31.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:56:31.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:56:31.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T01:56:31.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:56:31.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: raise 2021-07-31T01:56:31.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:56:31.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T01:56:31.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:56:31.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:56:31.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T01:56:31.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: cephadm 2021-07-31T01:56:29.160263+0000 mgr.x (mgr 2021-07-31T01:56:31.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: .24542) 398 : cephadm [ERR] 2021-07-31T01:56:31.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:56:31.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:56:31.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: yield 2021-07-31T01:56:31.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:56:31.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:56:31.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:56:31.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T01:56:31.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:56:31.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T01:56:31.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:56:31.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T01:56:31.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:56:31.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: task.result() 2021-07-31T01:56:31.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T01:56:31.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: cephadm 2021-07-31T01:56:29.160615+0000 mgr.x (mgr.24542) 399 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:56:31.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:56:31.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:56:31.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: yield 2021-07-31T01:56:31.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:56:31.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:56:31.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:56:31.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T01:56:31.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:56:31.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T01:56:31.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: cephadm 2021-07-31T01:56:29.069639+0000 mgr.x (mgr 2021-07-31T01:56:31.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: .24542) 391 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T01:56:31.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: cephadm 2021-07-31T01:56:29.114057+0000 mgr.x (mgr.24542) 392 : cephadm [ERR] 2021-07-31T01:56:31.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:56:31.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:56:31.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: yield 2021-07-31T01:56:31.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:56:31.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:56:31.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:56:31.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T01:56:31.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:56:31.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T01:56:31.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:56:31.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T01:56:31.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:56:31.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: task.result() 2021-07-31T01:56:31.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T01:56:31.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: 2021-07-31T01:56:31.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: cephadm 2021-07-31T01:56:29.114538+0000 mgr.x (mgr.24542) 393 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T01:56:31.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: cephadm 2021-07-31 2021-07-31T01:56:31.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: T01:56:29.125526+0000 mgr.x (mgr.24542) 394 : cephadm 2021-07-31T01:56:31.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: [ERR] 2021-07-31T01:56:31.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:56:31.040 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:56:31.040 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: yield 2021-07-31T01:56:31.040 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:56:31.040 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:56:31.041 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:56:31.041 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T01:56:31.041 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:56:31.041 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T01:56:31.042 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:56:31.042 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T01:56:31.043 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:56:31.043 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: task.result() 2021-07-31T01:56:31.044 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T01:56:31.044 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: cephadm 2021-07-31T01:56:29.126395+0000 mgr.x (mgr.24542) 395 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T01:56:31.044 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: cephadm 2021-07-31 2021-07-31T01:56:31.044 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: T01:56:29.151438+0000 mgr.x (mgr.24542) 396 : cephadm 2021-07-31T01:56:31.044 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: [ERR] 2021-07-31T01:56:31.045 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:56:31.046 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T01:56:31.046 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:56:31.046 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: task.result() 2021-07-31T01:56:31.046 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T01:56:31.047 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: 2021-07-31T01:56:31.047 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T01:56:31.047 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: 2021-07-31T01:56:31.047 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:56:31.047 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:56:31.048 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: return f(*arg) 2021-07-31T01:56:31.048 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:56:31.048 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:56:31.048 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:56:31.049 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:56:31.049 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:56:31.049 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:56:31.049 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:56:31.050 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T01:56:31.050 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:56:31.050 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: raise self._exception 2021-07-31T01:56:31.051 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:56:31.051 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:56:31.051 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:56:31.051 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T01:56:31.051 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:56:31.052 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: raise 2021-07-31T01:56:31.052 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:56:31.052 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T01:56:31.052 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:56:31.053 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:56:31.053 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T01:56:31.053 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: cluster 2021-07-31T01:56:29.660688+0000 mon.a (mon.0) 1343 : cluster [DBG] osdmap e678: 8 total, 8 up, 8 in 2021-07-31T01:56:31.054 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:30 smithi154 conmon[34553]: cluster 2021-07-31T01:56:29.924843+0000 mgr.x (mgr.24542) 400 : cluster [DBG] pgmap v803: 25 pgs: 25 active+clean; 565 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 0 B/s rd, 236 KiB/s wr, 0 op/s 2021-07-31T01:56:31.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: cephadm 2021-07-31T01:56:29 2021-07-31T01:56:31.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: .069639+0000 mgr.x (mgr.24542) 391 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T01:56:31.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: cephadm 2021-07-31T01 2021-07-31T01:56:31.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: :56:29.114057+0000 mgr.x (mgr 2021-07-31T01:56:31.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: .24542) 392 : cephadm [ERR] 2021-07-31T01:56:31.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:56:31.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:56:31.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: yield 2021-07-31T01:56:31.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:56:31.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:56:31.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:56:31.057 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T01:56:31.057 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:56:31.057 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T01:56:31.057 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:56:31.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T01:56:31.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:56:31.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: task.result() 2021-07-31T01:56:31.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T01:56:31.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: cephadm 2021-07-31T01:56:29.114538+0000 mgr.x (mgr.24542) 393 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T01:56:31.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: cephadm 2021- 2021-07-31T01:56:31.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: 07-31T01:56:29.125526+0000 mgr.x (mgr.24542) 394 2021-07-31T01:56:31.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: : cephadm [ERR] 2021-07-31T01:56:31.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:56:31.060 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:56:31.060 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: yield 2021-07-31T01:56:31.060 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:56:31.060 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:56:31.060 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:56:31.061 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T01:56:31.061 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:56:31.061 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T01:56:31.061 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:56:31.062 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T01:56:31.062 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:56:31.062 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: task.result() 2021-07-31T01:56:31.062 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T01:56:31.062 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: cephadm 2021-07-31T01:56:29.126395+0000 mgr.x (mgr.24542) 395 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T01:56:31.063 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: cephadm 2021-07-31T01:56:29.151438+0000 2021-07-31T01:56:31.063 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: mgr.x (mgr.24542) 396 : cephadm [ERR] 2021-07-31T01:56:31.064 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:56:31.064 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:56:31.064 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: yield 2021-07-31T01:56:31.065 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:56:31.065 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:56:31.065 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:56:31.065 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T01:56:31.065 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:56:31.066 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T01:56:31.066 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:56:31.066 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T01:56:31.066 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:56:31.067 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: task.result() 2021-07-31T01:56:31.067 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T01:56:31.067 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: cephadm 2021-07-31T01:56:29.153116+0000 mgr.x (mgr.24542) 397 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:56:31.067 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:56:31.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:56:31.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: yield 2021-07-31T01:56:31.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:56:31.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:56:31.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:56:31.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T01:56:31.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:56:31.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T01:56:31.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:56:31.070 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T01:56:31.070 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:56:31.070 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: task.result() 2021-07-31T01:56:31.070 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T01:56:31.070 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: 2021-07-31T01:56:31.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T01:56:31.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: 2021-07-31T01:56:31.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:56:31.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:56:31.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: return f(*arg) 2021-07-31T01:56:31.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:56:31.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:56:31.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:56:31.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:56:31.073 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:56:31.073 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:56:31.073 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:56:31.074 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:56:31.074 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: yield 2021-07-31T01:56:31.075 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:56:31.075 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:56:31.075 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:56:31.075 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T01:56:31.076 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:56:31.076 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T01:56:31.076 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:56:31.076 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T01:56:31.076 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:56:31.077 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: task.result() 2021-07-31T01:56:31.077 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T01:56:31.077 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: cephadm 2021-07-31T01:56:29.153116+0000 mgr.x (mgr.24542) 397 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:56:31.078 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:56:31.078 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:56:31.078 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: yield 2021-07-31T01:56:31.079 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:56:31.079 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:56:31.079 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:56:31.080 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T01:56:31.080 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:56:31.080 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T01:56:31.081 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:56:31.081 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T01:56:31.081 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:56:31.082 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: task.result() 2021-07-31T01:56:31.082 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T01:56:31.082 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: 2021-07-31T01:56:31.083 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T01:56:31.083 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: 2021-07-31T01:56:31.084 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:56:31.084 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:56:31.085 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: return f(*arg) 2021-07-31T01:56:31.085 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:56:31.085 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:56:31.085 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:56:31.086 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:56:31.086 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:56:31.086 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:56:31.087 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:56:31.087 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T01:56:31.087 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:56:31.087 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: raise self._exception 2021-07-31T01:56:31.088 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:56:31.088 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:56:31.088 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:56:31.088 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T01:56:31.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:56:31.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: raise 2021-07-31T01:56:31.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:56:31.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T01:56:31.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:56:31.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:56:31.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T01:56:31.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: cephadm 2021-07-31T01:56:31.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: 2021-07-31T01:56:29.160263+0000 mgr.x (mgr.24542) 398 : cephadm [ERR] 2021-07-31T01:56:31.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:56:31.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:56:31.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: yield 2021-07-31T01:56:31.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:56:31.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:56:31.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:56:31.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T01:56:31.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:56:31.093 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T01:56:31.093 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:56:31.093 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T01:56:31.093 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:56:31.093 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: task.result() 2021-07-31T01:56:31.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T01:56:31.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: cephadm 2021-07-31T01:56:29.160615+0000 mgr.x (mgr.24542) 399 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:56:31.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:56:31.095 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:56:31.095 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: yield 2021-07-31T01:56:31.095 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:56:31.095 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:56:31.096 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:56:31.096 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T01:56:31.096 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:56:31.097 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T01:56:31.097 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:56:31.097 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: raise self._exception 2021-07-31T01:56:31.097 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:56:31.098 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:56:31.098 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:56:31.098 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T01:56:31.098 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:56:31.099 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: raise 2021-07-31T01:56:31.099 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:56:31.099 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T01:56:31.099 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:56:31.100 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:56:31.100 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T01:56:31.100 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: cephadm 2021- 2021-07-31T01:56:31.100 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: 07-31T01:56:29.160263+0000 mgr.x (mgr.24542) 398 : cephadm [ERR] 2021-07-31T01:56:31.100 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:56:31.101 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:56:31.101 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: yield 2021-07-31T01:56:31.101 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:56:31.101 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:56:31.101 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:56:31.102 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T01:56:31.102 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:56:31.102 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T01:56:31.102 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:56:31.103 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T01:56:31.103 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:56:31.103 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: task.result() 2021-07-31T01:56:31.103 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T01:56:31.103 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: cephadm 2021-07-31T01:56:29.160615+0000 mgr.x (mgr.24542) 399 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:56:31.104 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:56:31.104 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:56:31.104 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: yield 2021-07-31T01:56:31.104 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:56:31.104 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:56:31.105 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:56:31.105 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T01:56:31.106 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:56:31.106 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T01:56:31.106 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:56:31.106 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T01:56:31.107 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:56:31.107 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: task.result() 2021-07-31T01:56:31.107 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T01:56:31.107 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: 2021-07-31T01:56:31.108 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T01:56:31.108 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: 2021-07-31T01:56:31.108 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:56:31.109 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:56:31.109 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: return f(*arg) 2021-07-31T01:56:31.109 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:56:31.109 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:56:31.109 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:56:31.110 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:56:31.110 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:56:31.110 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:56:31.110 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:56:31.111 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T01:56:31.111 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:56:31.111 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: raise self._exception 2021-07-31T01:56:31.111 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:56:31.111 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:56:31.112 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:56:31.112 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T01:56:31.112 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:56:31.112 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: raise 2021-07-31T01:56:31.112 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:56:31.113 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T01:56:31.113 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:56:31.113 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:56:31.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T01:56:31.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: cluster 2021-07-31T01:56:29.660688+0000 mon.a (mon.0) 1343 : cluster [DBG] osdmap e678: 8 total, 8 up, 8 in 2021-07-31T01:56:31.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: cluster 2021-07-31T01:56 2021-07-31T01:56:31.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[33655]: :29.924843+0000 mgr.x (mgr.24542) 400 : cluster [DBG] pgmap v803: 25 pgs: 25 active+clean; 565 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 0 B/s rd, 236 KiB/s wr, 0 op/s 2021-07-31T01:56:31.115 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:56:31.115 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T01:56:31.115 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:56:31.116 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T01:56:31.116 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:56:31.116 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: task.result() 2021-07-31T01:56:31.116 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T01:56:31.117 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: 2021-07-31T01:56:31.117 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T01:56:31.117 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: 2021-07-31T01:56:31.117 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:56:31.117 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:56:31.118 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: return f(*arg) 2021-07-31T01:56:31.118 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:56:31.118 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:56:31.118 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:56:31.118 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:56:31.119 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:56:31.119 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:56:31.119 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:56:31.119 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T01:56:31.120 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:56:31.120 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: raise self._exception 2021-07-31T01:56:31.120 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:56:31.120 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:56:31.120 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:56:31.121 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T01:56:31.121 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:56:31.121 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: raise 2021-07-31T01:56:31.121 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:56:31.121 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T01:56:31.122 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:56:31.122 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:56:31.122 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T01:56:31.122 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: cluster 2021-07-31T01:56:29.660688+0000 mon.a (mon.0) 1343 : cluster [DBG] osdmap e678: 8 total, 8 up, 8 in 2021-07-31T01:56:31.123 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: cluster 2021-07-31T01:56:29.924843+0000 mgr.x (mgr.24542 2021-07-31T01:56:31.123 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:30 smithi045 conmon[39718]: ) 400 : cluster [DBG] pgmap v803: 25 pgs: 25 active+clean; 565 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 0 B/s rd, 236 KiB/s wr, 0 op/s 2021-07-31T01:56:31.914 INFO:tasks.rados.rados.0.smithi045.stdout:3260: done (2 left) 2021-07-31T01:56:31.915 INFO:tasks.rados.rados.0.smithi045.stdout:3263: done (1 left) 2021-07-31T01:56:31.915 INFO:tasks.rados.rados.0.smithi045.stdout:3264: done (0 left) 2021-07-31T01:56:31.916 INFO:tasks.rados.rados.0.smithi045.stdout:3265: read oid 48 snap -1 2021-07-31T01:56:31.916 INFO:tasks.rados.rados.0.smithi045.stdout:3265: expect deleted 2021-07-31T01:56:31.916 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 31 current snap is 322 2021-07-31T01:56:31.916 INFO:tasks.rados.rados.0.smithi045.stdout:3266: seq_num 1006 ranges {1785856=49152} 2021-07-31T01:56:31.924 INFO:tasks.rados.rados.0.smithi045.stdout:3266: writing smithi04569269-31 from 1785856 to 1835008 tid 1 2021-07-31T01:56:31.925 INFO:tasks.rados.rados.0.smithi045.stdout:3267: read oid 49 snap -1 2021-07-31T01:56:31.925 INFO:tasks.rados.rados.0.smithi045.stdout:3267: expect (ObjNum 980 snap 303 seq_num 980) 2021-07-31T01:56:31.925 INFO:tasks.rados.rados.0.smithi045.stdout:3268: read oid 32 snap -1 2021-07-31T01:56:31.925 INFO:tasks.rados.rados.0.smithi045.stdout:3268: expect (ObjNum 754 snap 233 seq_num 754) 2021-07-31T01:56:31.926 INFO:tasks.rados.rados.0.smithi045.stdout:3269: delete oid 6 current snap is 322 2021-07-31T01:56:31.931 INFO:tasks.rados.rados.0.smithi045.stdout:3265: done (4 left) 2021-07-31T01:56:31.931 INFO:tasks.rados.rados.0.smithi045.stdout:3269: done (3 left) 2021-07-31T01:56:31.931 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 50 current snap is 322 2021-07-31T01:56:31.932 INFO:tasks.rados.rados.0.smithi045.stdout:3270: seq_num 1008 ranges {4358144=90112} 2021-07-31T01:56:31.956 INFO:tasks.rados.rados.0.smithi045.stdout:3270: writing smithi04569269-50 from 4358144 to 4448256 tid 1 2021-07-31T01:56:31.956 INFO:tasks.rados.rados.0.smithi045.stdout:3271: snap_create 2021-07-31T01:56:31.957 INFO:tasks.rados.rados.0.smithi045.stdout:3266: finishing write tid 1 to smithi04569269-31 2021-07-31T01:56:31.957 INFO:tasks.rados.rados.0.smithi045.stdout:3266: finishing write tid 2 to smithi04569269-31 2021-07-31T01:56:31.968 INFO:tasks.rados.rados.0.smithi045.stdout:3268: expect (ObjNum 754 snap 233 seq_num 754) 2021-07-31T01:56:32.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:31 smithi154 conmon[34553]: cluster 2021-07-31T01:56:30.673521+0000 mon.a (mon.0) 1344 : cluster [DBG] osdmap e679: 8 total, 8 up, 8 in 2021-07-31T01:56:32.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:31 smithi154 conmon[34553]: cluster 2021-07-31T01:56:31.647982+0000 mon.a (mon.0) 1345 : cluster [DBG] osdmap e680: 8 total, 8 up, 8 in 2021-07-31T01:56:32.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:31 smithi045 conmon[39718]: cluster 2021-07-31T01:56:30.673521+0000 2021-07-31T01:56:32.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:31 smithi045 conmon[39718]: mon.a (mon.0) 1344 : cluster [DBG] osdmap e679: 8 total, 8 up, 8 in 2021-07-31T01:56:32.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:31 smithi045 conmon[39718]: cluster 2021-07-31T 2021-07-31T01:56:32.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:31 smithi045 conmon[39718]: 01:56:31.647982+0000 mon.a (mon.0) 1345 : cluster [DBG] osdmap e680: 8 total, 8 up, 8 in 2021-07-31T01:56:32.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:31 smithi045 conmon[33655]: cluster 2021-07-31T01:56:30.673521+0000 mon.a (mon. 2021-07-31T01:56:32.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:31 smithi045 conmon[33655]: 0) 1344 : cluster [DBG] osdmap e679: 8 total, 8 up, 8 in 2021-07-31T01:56:32.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:31 smithi045 conmon[33655]: cluster 2021-07-31T01:56:31.647982+0000 mon.a (mon.0) 1345 : cluster [DBG] osdmap e680: 8 total, 8 up, 8 in 2021-07-31T01:56:32.682 INFO:tasks.rados.rados.0.smithi045.stdout:3270: finishing write tid 1 to smithi04569269-50 2021-07-31T01:56:32.683 INFO:tasks.rados.rados.0.smithi045.stdout:3270: finishing write tid 2 to smithi04569269-50 2021-07-31T01:56:32.683 INFO:tasks.rados.rados.0.smithi045.stdout:3270: finishing write tid 3 to smithi04569269-50 2021-07-31T01:56:32.683 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 50 v 1718 (ObjNum 1008 snap 322 seq_num 1008) dirty exists 2021-07-31T01:56:32.683 INFO:tasks.rados.rados.0.smithi045.stdout:3270: left oid 50 (ObjNum 1008 snap 322 seq_num 1008) 2021-07-31T01:56:32.684 INFO:tasks.rados.rados.0.smithi045.stdout:3267: expect (ObjNum 980 snap 303 seq_num 980) 2021-07-31T01:56:33.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:32 smithi154 conmon[34553]: cluster 2021-07-31T01:56:31.925367+0000 mgr.x (mgr.24542) 2021-07-31T01:56:33.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:32 smithi154 conmon[34553]: 401 : cluster [DBG] pgmap v806: 25 pgs: 25 active+clean; 565 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 0 B/s rd, 236 KiB/s wr, 0 op/s 2021-07-31T01:56:33.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:32 smithi045 conmon[39718]: cluster 2021-07-31T01:56:31.925367+0000 mgr.x (mgr.24542 2021-07-31T01:56:33.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:32 smithi045 conmon[39718]: ) 401 : cluster [DBG] pgmap v806: 25 pgs: 25 active+clean; 565 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 0 B/s rd, 236 KiB/s wr, 0 op/s 2021-07-31T01:56:33.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:32 smithi045 conmon[33655]: cluster 2021-07-31T01:56:31.925367+0000 mgr.x (mgr.24542) 2021-07-31T01:56:33.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:32 smithi045 conmon[33655]: 401 : cluster [DBG] pgmap v806: 25 pgs: 25 active+clean; 565 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 0 B/s rd, 236 KiB/s wr, 0 op/s 2021-07-31T01:56:33.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:33 smithi045 conmon[39718]: cluster 2021-07-31T01:56:32.677399 2021-07-31T01:56:33.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:33 smithi045 conmon[39718]: +0000 mon.a (mon.0) 1346 : cluster [DBG] osdmap e681: 8 total, 8 up, 8 in 2021-07-31T01:56:33.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:56:33 smithi045 conmon[61210]: level=error ts=2021-07-31T01:56:33.525Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:56:33.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:56:33 smithi045 conmon[61210]: level=error ts=2021-07-31T01:56:33.525Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:56:33.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:56:33 smithi045 conmon[61210]: level=error ts=2021-07-31T01:56:33.525Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T01:56:33.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:33 smithi045 conmon[33655]: cluster 2021-07-31T01:56:32. 2021-07-31T01:56:33.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:33 smithi045 conmon[33655]: 677399+0000 mon.a (mon.0) 1346 : cluster [DBG] osdmap e681: 8 total, 8 up, 8 in 2021-07-31T01:56:34.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:33 smithi154 conmon[34553]: cluster 2021-07-31T01:56:32.677399+0000 mon.a (mon.0) 1346 : cluster [DBG] osdmap e681: 8 total, 8 up, 8 in 2021-07-31T01:56:34.587 INFO:tasks.rados.rados.0.smithi045.stdout:3266: finishing write tid 3 to smithi04569269-31 2021-07-31T01:56:34.587 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 31 v 2125 (ObjNum 1006 snap 322 seq_num 1006) dirty exists 2021-07-31T01:56:34.587 INFO:tasks.rados.rados.0.smithi045.stdout:3266: left oid 31 (ObjNum 1006 snap 322 seq_num 1006) 2021-07-31T01:56:34.588 INFO:tasks.rados.rados.0.smithi045.stdout:3266: done (4 left) 2021-07-31T01:56:34.588 INFO:tasks.rados.rados.0.smithi045.stdout:3267: done (3 left) 2021-07-31T01:56:34.588 INFO:tasks.rados.rados.0.smithi045.stdout:3268: done (2 left) 2021-07-31T01:56:34.588 INFO:tasks.rados.rados.0.smithi045.stdout:3270: done (1 left) 2021-07-31T01:56:34.589 INFO:tasks.rados.rados.0.smithi045.stdout:3271: done (0 left) 2021-07-31T01:56:34.589 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 23 current snap is 323 2021-07-31T01:56:34.589 INFO:tasks.rados.rados.0.smithi045.stdout:3272: seq_num 1009 ranges {966656=106496} 2021-07-31T01:56:34.592 INFO:tasks.rados.rados.0.smithi045.stdout:3272: writing smithi04569269-23 from 966656 to 1073152 tid 1 2021-07-31T01:56:34.592 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 3 current snap is 323 2021-07-31T01:56:34.593 INFO:tasks.rados.rados.0.smithi045.stdout:3273: seq_num 1010 ranges {11812864=122880} 2021-07-31T01:56:34.658 INFO:tasks.rados.rados.0.smithi045.stdout:3273: writing smithi04569269-3 from 11812864 to 11935744 tid 1 2021-07-31T01:56:34.658 INFO:tasks.rados.rados.0.smithi045.stdout:3274: read oid 34 snap -1 2021-07-31T01:56:34.658 INFO:tasks.rados.rados.0.smithi045.stdout:3274: expect deleted 2021-07-31T01:56:34.659 INFO:tasks.rados.rados.0.smithi045.stdout:3275: rollback oid 19 current snap is 323 2021-07-31T01:56:34.659 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 19 to 294 2021-07-31T01:56:34.659 INFO:tasks.rados.rados.0.smithi045.stdout:3272: finishing write tid 1 to smithi04569269-23 2021-07-31T01:56:34.659 INFO:tasks.rados.rados.0.smithi045.stdout:3276: delete oid 30 current snap is 323 2021-07-31T01:56:34.660 INFO:tasks.rados.rados.0.smithi045.stdout:3272: finishing write tid 2 to smithi04569269-23 2021-07-31T01:56:34.660 INFO:tasks.rados.rados.0.smithi045.stdout:3272: finishing write tid 3 to smithi04569269-23 2021-07-31T01:56:34.660 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 23 v 2021 (ObjNum 1009 snap 323 seq_num 1009) dirty exists 2021-07-31T01:56:34.661 INFO:tasks.rados.rados.0.smithi045.stdout:3272: left oid 23 (ObjNum 1009 snap 323 seq_num 1009) 2021-07-31T01:56:34.662 INFO:tasks.rados.rados.0.smithi045.stdout:3275: finishing rollback tid 1 to smithi04569269-19 2021-07-31T01:56:34.662 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 19 v 0 (ObjNum -1104671312 snap 21873 seq_num 3190241824) dirty dne 2021-07-31T01:56:34.662 INFO:tasks.rados.rados.0.smithi045.stdout:3272: done (4 left) 2021-07-31T01:56:34.662 INFO:tasks.rados.rados.0.smithi045.stdout:3274: done (3 left) 2021-07-31T01:56:34.663 INFO:tasks.rados.rados.0.smithi045.stdout:3275: done (2 left) 2021-07-31T01:56:34.663 INFO:tasks.rados.rados.0.smithi045.stdout:3276: done (1 left) 2021-07-31T01:56:34.663 INFO:tasks.rados.rados.0.smithi045.stdout:3277: rollback oid 11 current snap is 323 2021-07-31T01:56:34.663 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 11 to 294 2021-07-31T01:56:34.663 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 25 current snap is 323 2021-07-31T01:56:34.664 INFO:tasks.rados.rados.0.smithi045.stdout:3278: seq_num 1012 ranges {2031616=155648} 2021-07-31T01:56:34.677 INFO:tasks.rados.rados.0.smithi045.stdout:3278: writing smithi04569269-25 from 2031616 to 2187264 tid 1 2021-07-31T01:56:34.679 INFO:tasks.rados.rados.0.smithi045.stdout:3279: rollback oid 30 current snap is 323 2021-07-31T01:56:34.680 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 30 to 255 2021-07-31T01:56:34.680 INFO:tasks.rados.rados.0.smithi045.stdout:3273: finishing write tid 1 to smithi04569269-3 2021-07-31T01:56:34.680 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 21 current snap is 323 2021-07-31T01:56:34.680 INFO:tasks.rados.rados.0.smithi045.stdout:3273: finishing write tid 2 to smithi04569269-3 2021-07-31T01:56:34.681 INFO:tasks.rados.rados.0.smithi045.stdout:3280: seq_num 1013 ranges {1228800=172032} 2021-07-31T01:56:34.682 INFO:tasks.rados.rados.0.smithi045.stdout:3280: writing smithi04569269-21 from 1228800 to 1400832 tid 1 2021-07-31T01:56:34.682 INFO:tasks.rados.rados.0.smithi045.stdout:3281: snap_create 2021-07-31T01:56:34.682 INFO:tasks.rados.rados.0.smithi045.stdout:3277: finishing rollback tid 1 to smithi04569269-11 2021-07-31T01:56:34.682 INFO:tasks.rados.rados.0.smithi045.stdout:3277: finishing rollback tid 2 to smithi04569269-11 2021-07-31T01:56:34.683 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 11 v 2023 (ObjNum 920 snap 278 seq_num 920) dirty exists 2021-07-31T01:56:34.683 INFO:tasks.rados.rados.0.smithi045.stdout:3273: finishing write tid 3 to smithi04569269-3 2021-07-31T01:56:34.683 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 3 v 2128 (ObjNum 1010 snap 323 seq_num 1010) dirty exists 2021-07-31T01:56:34.683 INFO:tasks.rados.rados.0.smithi045.stdout:3273: left oid 3 (ObjNum 1010 snap 323 seq_num 1010) 2021-07-31T01:56:34.686 INFO:tasks.rados.rados.0.smithi045.stdout:3279: finishing rollback tid 1 to smithi04569269-30 2021-07-31T01:56:34.686 INFO:tasks.rados.rados.0.smithi045.stdout:3278: finishing write tid 1 to smithi04569269-25 2021-07-31T01:56:34.686 INFO:tasks.rados.rados.0.smithi045.stdout:3278: finishing write tid 2 to smithi04569269-25 2021-07-31T01:56:34.686 INFO:tasks.rados.rados.0.smithi045.stdout:3279: finishing rollback tid 2 to smithi04569269-30 2021-07-31T01:56:34.687 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 30 v 1660 (ObjNum 799 snap 246 seq_num 799) dirty exists 2021-07-31T01:56:34.687 INFO:tasks.rados.rados.0.smithi045.stdout:3278: finishing write tid 3 to smithi04569269-25 2021-07-31T01:56:34.688 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 25 v 1838 (ObjNum 1012 snap 323 seq_num 1012) dirty exists 2021-07-31T01:56:34.688 INFO:tasks.rados.rados.0.smithi045.stdout:3278: left oid 25 (ObjNum 1012 snap 323 seq_num 1012) 2021-07-31T01:56:34.689 INFO:tasks.rados.rados.0.smithi045.stdout:3280: finishing write tid 1 to smithi04569269-21 2021-07-31T01:56:34.689 INFO:tasks.rados.rados.0.smithi045.stdout:3280: finishing write tid 2 to smithi04569269-21 2021-07-31T01:56:34.689 INFO:tasks.rados.rados.0.smithi045.stdout:3280: finishing write tid 3 to smithi04569269-21 2021-07-31T01:56:34.690 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 21 v 2131 (ObjNum 1013 snap 323 seq_num 1013) dirty exists 2021-07-31T01:56:34.690 INFO:tasks.rados.rados.0.smithi045.stdout:3280: left oid 21 (ObjNum 1013 snap 323 seq_num 1013) 2021-07-31T01:56:35.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:34 smithi154 conmon[34553]: cluster 2021-07-31T01:56:33.925980+0000 mgr.x (mgr.24542) 402 : cluster [DBG] pgmap v808: 25 pgs: 25 active+clean; 565 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 25 MiB/s rd, 32 KiB/s wr, 4 op/s 2021-07-31T01:56:35.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:34 smithi045 conmon[39718]: cluster 2021-07-31T01: 2021-07-31T01:56:35.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:34 smithi045 conmon[39718]: 56:33.925980+0000 mgr.x (mgr.24542) 402 : cluster [DBG] pgmap v808: 25 pgs: 25 active+clean; 565 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 25 MiB/s rd, 32 KiB/s wr, 4 op/s 2021-07-31T01:56:35.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:34 smithi045 conmon[33655]: cluster 2021-07-31T01 2021-07-31T01:56:35.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:34 smithi045 conmon[33655]: :56:33.925980+0000 mgr.x (mgr.24542) 402 : cluster [DBG] pgmap v808: 25 pgs: 25 active+clean; 565 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 25 MiB/s rd, 32 KiB/s wr, 4 op/s 2021-07-31T01:56:35.684 INFO:tasks.rados.rados.0.smithi045.stdout:3273: done (5 left) 2021-07-31T01:56:35.684 INFO:tasks.rados.rados.0.smithi045.stdout:3277: done (4 left) 2021-07-31T01:56:35.684 INFO:tasks.rados.rados.0.smithi045.stdout:3278: done (3 left) 2021-07-31T01:56:35.685 INFO:tasks.rados.rados.0.smithi045.stdout:3279: done (2 left) 2021-07-31T01:56:35.685 INFO:tasks.rados.rados.0.smithi045.stdout:3280: done (1 left) 2021-07-31T01:56:35.685 INFO:tasks.rados.rados.0.smithi045.stdout:3281: done (0 left) 2021-07-31T01:56:35.685 INFO:tasks.rados.rados.0.smithi045.stdout:3282: delete oid 49 current snap is 324 2021-07-31T01:56:35.687 INFO:tasks.rados.rados.0.smithi045.stdout:3282: done (0 left) 2021-07-31T01:56:35.687 INFO:tasks.rados.rados.0.smithi045.stdout:3283: snap_remove snap 313 2021-07-31T01:56:36.650 INFO:tasks.rados.rados.0.smithi045.stdout:3283: done (0 left) 2021-07-31T01:56:36.651 INFO:tasks.rados.rados.0.smithi045.stdout:3284: rollback oid 20 current snap is 324 2021-07-31T01:56:36.651 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 20 to 310 2021-07-31T01:56:36.651 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 10 current snap is 324 2021-07-31T01:56:36.652 INFO:tasks.rados.rados.0.smithi045.stdout:3285: seq_num 1015 ranges {10043392=204800} 2021-07-31T01:56:36.707 INFO:tasks.rados.rados.0.smithi045.stdout:3285: writing smithi04569269-10 from 10043392 to 10248192 tid 1 2021-07-31T01:56:36.709 INFO:tasks.rados.rados.0.smithi045.stdout:3286: copy_from oid 24 from oid 49 current snap is 324 2021-07-31T01:56:36.709 INFO:tasks.rados.rados.0.smithi045.stdout:3284: finishing rollback tid 0 to smithi04569269-20 2021-07-31T01:56:36.709 INFO:tasks.rados.rados.0.smithi045.stdout:3284: finishing rollback tid 1 to smithi04569269-20 2021-07-31T01:56:36.709 INFO:tasks.rados.rados.0.smithi045.stdout:3284: finishing rollback tid 2 to smithi04569269-20 2021-07-31T01:56:36.709 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 20 v 2137 (ObjNum 857 snap 268 seq_num 857) dirty exists 2021-07-31T01:56:36.710 INFO:tasks.rados.rados.0.smithi045.stdout:3284: done (2 left) 2021-07-31T01:56:36.710 INFO:tasks.rados.rados.0.smithi045.stdout:3287: rollback oid 22 current snap is 324 2021-07-31T01:56:36.710 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 22 to 289 2021-07-31T01:56:36.710 INFO:tasks.rados.rados.0.smithi045.stdout:3288: snap_remove snap 315 2021-07-31T01:56:37.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:36 smithi154 conmon[34553]: cluster 2021-07-31T01:56:35.682970+0000 mon.a (mon.0) 1347 : cluster [DBG] osdmap e682: 8 total, 8 up, 8 in 2021-07-31T01:56:37.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:36 smithi154 conmon[34553]: cluster 2021-07-31T01:56:35 2021-07-31T01:56:37.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:36 smithi154 conmon[34553]: .926474+0000 mgr.x (mgr.24542) 403 : cluster [DBG] pgmap v810: 25 pgs: 25 active+clean; 565 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 21 MiB/s rd, 26 KiB/s wr, 3 op/s 2021-07-31T01:56:37.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:36 smithi154 conmon[34553]: cluster 2021-07-31T01:56:36.649304+0000 mon.a (mon.0) 1348 : cluster [DBG] osdmap e683: 8 total, 8 up, 8 in 2021-07-31T01:56:37.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:36 smithi045 conmon[39718]: cluster 2021-07-31T01:56:35.682970+0000 mon.a (mon. 2021-07-31T01:56:37.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:36 smithi045 conmon[39718]: 0) 1347 : cluster [DBG] osdmap e682: 8 total, 8 up, 8 in 2021-07-31T01:56:37.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:36 smithi045 conmon[39718]: cluster 2021-07-31T01:56:35.926474+0000 mgr.x (mgr.24542) 403 : cluster 2021-07-31T01:56:37.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:36 smithi045 conmon[39718]: [DBG] pgmap v810: 25 pgs: 25 active+clean; 565 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 21 MiB/s rd, 26 KiB/s wr, 3 op/s 2021-07-31T01:56:37.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:36 smithi045 conmon[39718]: cluster 2021- 2021-07-31T01:56:37.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:36 smithi045 conmon[39718]: 07-31T01:56:36.649304+0000 mon.a (mon. 2021-07-31T01:56:37.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:36 smithi045 conmon[39718]: 0) 1348 : cluster [DBG] osdmap e683: 8 total, 8 up, 8 in 2021-07-31T01:56:37.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:36 smithi045 conmon[33655]: cluster 2021-07-31T01:56 2021-07-31T01:56:37.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:36 smithi045 conmon[33655]: :35.682970+0000 mon.a (mon.0) 1347 : cluster [DBG] osdmap e682: 8 total, 8 up, 8 in 2021-07-31T01:56:37.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:36 smithi045 conmon[33655]: cluster 2021-07-31T01:56:37.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:36 smithi045 conmon[33655]: 2021-07-31T01:56:35 2021-07-31T01:56:37.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:36 smithi045 conmon[33655]: .926474+0000 mgr.x (mgr.24542) 2021-07-31T01:56:37.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:36 smithi045 conmon[33655]: 403 : cluster [DBG] pgmap v810: 25 pgs: 25 active+clean; 565 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 21 MiB/s rd, 26 KiB/s wr, 3 op/s 2021-07-31T01:56:37.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:36 smithi045 conmon[33655]: cluster 2021-07-31T01:56:37.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:36 smithi045 conmon[33655]: 2021-07-31T01:56:36. 2021-07-31T01:56:37.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:36 smithi045 conmon[33655]: 649304+0000 mon.a (mon.0) 1348 : cluster [DBG] osdmap e683: 8 total, 8 up, 8 in 2021-07-31T01:56:37.692 INFO:tasks.rados.rados.0.smithi045.stdout:3288: done (3 left) 2021-07-31T01:56:37.693 INFO:tasks.rados.rados.0.smithi045.stdout:3289: delete oid 30 current snap is 324 2021-07-31T01:56:37.693 INFO:tasks.rados.rados.0.smithi045.stdout:3286: finishing copy_from to smithi04569269-24 2021-07-31T01:56:37.693 INFO:tasks.rados.rados.0.smithi045.stdout:3286: got expected ENOENT (src dne) 2021-07-31T01:56:37.693 INFO:tasks.rados.rados.0.smithi045.stdout:3286: finishing copy_from racing read to smithi04569269-24 2021-07-31T01:56:37.693 INFO:tasks.rados.rados.0.smithi045.stdout:3287: finishing rollback tid 1 to smithi04569269-22 2021-07-31T01:56:37.694 INFO:tasks.rados.rados.0.smithi045.stdout:3287: finishing rollback tid 2 to smithi04569269-22 2021-07-31T01:56:37.694 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 22 v 1656 (ObjNum 917 snap 278 seq_num 917) dirty exists 2021-07-31T01:56:37.695 INFO:tasks.rados.rados.0.smithi045.stdout:3285: finishing write tid 1 to smithi04569269-10 2021-07-31T01:56:37.695 INFO:tasks.rados.rados.0.smithi045.stdout:3285: finishing write tid 2 to smithi04569269-10 2021-07-31T01:56:37.695 INFO:tasks.rados.rados.0.smithi045.stdout:3285: finishing write tid 3 to smithi04569269-10 2021-07-31T01:56:37.695 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 10 v 1720 (ObjNum 1015 snap 324 seq_num 1015) dirty exists 2021-07-31T01:56:37.695 INFO:tasks.rados.rados.0.smithi045.stdout:3285: left oid 10 (ObjNum 1015 snap 324 seq_num 1015) 2021-07-31T01:56:37.698 INFO:tasks.rados.rados.0.smithi045.stdout:3285: done (3 left) 2021-07-31T01:56:37.699 INFO:tasks.rados.rados.0.smithi045.stdout:3286: done (2 left) 2021-07-31T01:56:37.699 INFO:tasks.rados.rados.0.smithi045.stdout:3287: done (1 left) 2021-07-31T01:56:37.699 INFO:tasks.rados.rados.0.smithi045.stdout:3289: done (0 left) 2021-07-31T01:56:37.699 INFO:tasks.rados.rados.0.smithi045.stdout:3290: read oid 27 snap -1 2021-07-31T01:56:37.700 INFO:tasks.rados.rados.0.smithi045.stdout:3290: expect (ObjNum 0 snap 0 seq_num 0) 2021-07-31T01:56:37.700 INFO:tasks.rados.rados.0.smithi045.stdout:3291: copy_from oid 37 from oid 10 current snap is 324 2021-07-31T01:56:37.700 INFO:tasks.rados.rados.0.smithi045.stdout:3292: snap_create 2021-07-31T01:56:37.703 INFO:tasks.rados.rados.0.smithi045.stdout:3290: expect (ObjNum 0 snap 0 seq_num 0) 2021-07-31T01:56:37.751 INFO:tasks.rados.rados.0.smithi045.stdout:3291: finishing copy_from to smithi04569269-37 2021-07-31T01:56:37.751 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 37 v 1720 (ObjNum 1015 snap 324 seq_num 1015) dirty exists 2021-07-31T01:56:37.752 INFO:tasks.rados.rados.0.smithi045.stdout:3291: finishing copy_from racing read to smithi04569269-37 2021-07-31T01:56:38.699 INFO:tasks.rados.rados.0.smithi045.stdout:3290: done (2 left) 2021-07-31T01:56:38.699 INFO:tasks.rados.rados.0.smithi045.stdout:3291: done (1 left) 2021-07-31T01:56:38.699 INFO:tasks.rados.rados.0.smithi045.stdout:3292: done (0 left) 2021-07-31T01:56:38.699 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 43 current snap is 325 2021-07-31T01:56:38.700 INFO:tasks.rados.rados.0.smithi045.stdout:3293: seq_num 1017 ranges {9699328=237568} 2021-07-31T01:56:38.754 INFO:tasks.rados.rados.0.smithi045.stdout:3293: writing smithi04569269-43 from 9699328 to 9936896 tid 1 2021-07-31T01:56:38.754 INFO:tasks.rados.rados.0.smithi045.stdout:3294: setattr oid 29 current snap is 325 2021-07-31T01:56:38.755 INFO:tasks.rados.rados.0.smithi045.stdout:3295: snap_create 2021-07-31T01:56:38.760 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 29 v 1419 (ObjNum 288 snap 0 seq_num 161) dirty exists 2021-07-31T01:56:38.760 INFO:tasks.rados.rados.0.smithi045.stdout:3293: finishing write tid 1 to smithi04569269-43 2021-07-31T01:56:38.761 INFO:tasks.rados.rados.0.smithi045.stdout:3293: finishing write tid 2 to smithi04569269-43 2021-07-31T01:56:38.766 INFO:tasks.rados.rados.0.smithi045.stdout:3293: finishing write tid 3 to smithi04569269-43 2021-07-31T01:56:38.766 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 43 v 1860 (ObjNum 1017 snap 325 seq_num 1017) dirty exists 2021-07-31T01:56:38.766 INFO:tasks.rados.rados.0.smithi045.stdout:3293: left oid 43 (ObjNum 1017 snap 325 seq_num 1017) 2021-07-31T01:56:39.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:38 smithi154 conmon[34553]: cluster 2021-07-31T01:56:37.691609+0000 mon.a (mon.0 2021-07-31T01:56:39.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:38 smithi154 conmon[34553]: ) 1349 : cluster [DBG] osdmap e684: 8 total, 8 up, 8 in 2021-07-31T01:56:39.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:38 smithi154 conmon[34553]: cluster 2021-07-31T01:56:37.927061+0000 mgr.x (mgr.24542) 404 : cluster 2021-07-31T01:56:39.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:38 smithi154 conmon[34553]: [DBG] pgmap v813: 25 pgs: 25 active+clean; 565 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 21 MiB/s rd, 26 KiB/s wr, 3 op/s 2021-07-31T01:56:39.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:38 smithi045 conmon[33655]: cluster 2021-07-31T01:56:37.691609+0000 2021-07-31T01:56:39.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:38 smithi045 conmon[33655]: mon.a (mon.0) 1349 : cluster [DBG] osdmap e684: 8 total, 8 up, 8 in 2021-07-31T01:56:39.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:38 smithi045 conmon[33655]: cluster 2021-07-31 2021-07-31T01:56:39.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:38 smithi045 conmon[33655]: T01:56:37.927061+0000 mgr.x (mgr.24542) 404 : cluster [DBG] pgmap v813: 25 pgs: 25 active+clean; 565 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 21 MiB/s rd, 26 KiB/s wr, 3 op/s 2021-07-31T01:56:39.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:38 smithi045 conmon[39718]: cluster 2021-07-31T01:56:37.691609+0000 mon.a 2021-07-31T01:56:39.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:38 smithi045 conmon[39718]: (mon.0) 1349 : cluster [DBG] osdmap e684: 8 total, 8 up, 8 in 2021-07-31T01:56:39.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:38 smithi045 conmon[39718]: cluster 2021-07-31T01:56:39.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:38 smithi045 conmon[39718]: 2021-07-31T01:56:37.927061+0000 mgr.x (mgr.24542) 404 : 2021-07-31T01:56:39.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:38 smithi045 conmon[39718]: cluster [DBG] pgmap v813: 25 pgs: 25 active+clean; 565 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 21 MiB/s rd, 26 KiB/s wr, 3 op/s 2021-07-31T01:56:39.699 INFO:tasks.rados.rados.0.smithi045.stdout:3293: done (2 left) 2021-07-31T01:56:39.699 INFO:tasks.rados.rados.0.smithi045.stdout:3294: done (1 left) 2021-07-31T01:56:39.699 INFO:tasks.rados.rados.0.smithi045.stdout:3295: done (0 left) 2021-07-31T01:56:39.700 INFO:tasks.rados.rados.0.smithi045.stdout:3296: snap_create 2021-07-31T01:56:40.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:39 smithi154 conmon[34553]: cluster 2021-07-31T01:56:38.698011+0000 mon.a (mon.0) 1350 : cluster [DBG] osdmap e685: 8 total, 8 up, 8 in 2021-07-31T01:56:40.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:39 smithi045 conmon[33655]: cluster 2021-07-31T01:56:38.698011+0000 mon.a (mon.0) 2021-07-31T01:56:40.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:39 smithi045 conmon[33655]: 1350 : cluster [DBG] osdmap e685: 8 total, 8 up, 8 in 2021-07-31T01:56:40.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:39 smithi045 conmon[39718]: cluster 2021-07-31T01:56:38.698011+0000 mon.a ( 2021-07-31T01:56:40.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:39 smithi045 conmon[39718]: mon.0) 1350 : cluster [DBG] osdmap e685: 8 total, 8 up, 8 in 2021-07-31T01:56:40.709 INFO:tasks.rados.rados.0.smithi045.stdout:3296: done (0 left) 2021-07-31T01:56:40.710 INFO:tasks.rados.rados.0.smithi045.stdout:3297: delete oid 2 current snap is 327 2021-07-31T01:56:40.713 INFO:tasks.rados.rados.0.smithi045.stdout:3297: done (0 left) 2021-07-31T01:56:40.713 INFO:tasks.rados.rados.0.smithi045.stdout:3298: snap_remove snap 323 2021-07-31T01:56:41.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:40 smithi154 conmon[34553]: cluster 2021-07-31T01:56:39.698591+0000 mon.a (mon.0 2021-07-31T01:56:41.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:40 smithi154 conmon[34553]: ) 1351 : cluster [DBG] osdmap e686: 8 total, 8 up, 8 in 2021-07-31T01:56:41.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:40 smithi154 conmon[34553]: cluster 2021-07-31T01:56:39.927660+0000 mgr.x (mgr.24542) 405 : cluster [DBG] pgmap v816: 25 pgs: 25 active+clean; 587 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 2.7 MiB/s wr, 13 op/s 2021-07-31T01:56:41.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:40 smithi045 conmon[33655]: cluster 2021-07-31T01:56:39.698591+0000 mon.a (mon.0 2021-07-31T01:56:41.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:40 smithi045 conmon[33655]: ) 1351 : cluster [DBG] osdmap e686: 8 total, 8 up, 8 in 2021-07-31T01:56:41.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:40 smithi045 conmon[33655]: cluster 2021-07-31T01:56:39.927660+0000 mgr.x (mgr.24542) 405 : cluster [DBG] pgmap v816: 25 pgs: 25 active+clean; 587 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 2.7 MiB/s wr, 13 op/s 2021-07-31T01:56:41.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:40 smithi045 conmon[39718]: cluster 2021-07-31T01:56:39.698591+0000 mon.a (mon 2021-07-31T01:56:41.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:40 smithi045 conmon[39718]: .0) 1351 : cluster [DBG] osdmap e686: 8 total, 8 up, 8 in 2021-07-31T01:56:41.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:40 smithi045 conmon[39718]: cluster 2021-07-31 2021-07-31T01:56:41.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:40 smithi045 conmon[39718]: T01:56:39.927660+0000 mgr.x (mgr.24542 2021-07-31T01:56:41.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:40 smithi045 conmon[39718]: ) 405 : cluster [DBG] pgmap v816: 25 pgs: 25 active+clean; 587 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 2.7 MiB/s wr, 13 op/s 2021-07-31T01:56:41.654 INFO:tasks.rados.rados.0.smithi045.stdout:3298: done (0 left) 2021-07-31T01:56:41.654 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 15 current snap is 327 2021-07-31T01:56:41.655 INFO:tasks.rados.rados.0.smithi045.stdout:3299: seq_num 1019 ranges {0=270336} 2021-07-31T01:56:41.656 INFO:tasks.rados.rados.0.smithi045.stdout:3299: writing smithi04569269-15 from 0 to 270336 tid 1 2021-07-31T01:56:41.657 INFO:tasks.rados.rados.0.smithi045.stdout:3300: read oid 39 snap -1 2021-07-31T01:56:41.657 INFO:tasks.rados.rados.0.smithi045.stdout:3300: expect deleted 2021-07-31T01:56:41.657 INFO:tasks.rados.rados.0.smithi045.stdout:3301: rollback oid 17 current snap is 327 2021-07-31T01:56:41.657 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 17 to 312 2021-07-31T01:56:41.658 INFO:tasks.rados.rados.0.smithi045.stdout:3302: read oid 19 snap -1 2021-07-31T01:56:41.658 INFO:tasks.rados.rados.0.smithi045.stdout:3302: expect deleted 2021-07-31T01:56:41.659 INFO:tasks.rados.rados.0.smithi045.stdout:3303: rollback oid 24 current snap is 327 2021-07-31T01:56:41.659 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 24 to 286 2021-07-31T01:56:41.660 INFO:tasks.rados.rados.0.smithi045.stdout:3304: copy_from oid 10 from oid 32 current snap is 327 2021-07-31T01:56:41.660 INFO:tasks.rados.rados.0.smithi045.stdout:3305: copy_from oid 27 from oid 30 current snap is 327 2021-07-31T01:56:41.660 INFO:tasks.rados.rados.0.smithi045.stdout:3306: delete oid 37 current snap is 327 2021-07-31T01:56:41.662 INFO:tasks.rados.rados.0.smithi045.stdout:3299: finishing write tid 1 to smithi04569269-15 2021-07-31T01:56:41.663 INFO:tasks.rados.rados.0.smithi045.stdout:3299: finishing write tid 2 to smithi04569269-15 2021-07-31T01:56:41.664 INFO:tasks.rados.rados.0.smithi045.stdout:3305: finishing copy_from to smithi04569269-27 2021-07-31T01:56:41.664 INFO:tasks.rados.rados.0.smithi045.stdout:3305: got expected ENOENT (src dne) 2021-07-31T01:56:41.664 INFO:tasks.rados.rados.0.smithi045.stdout:3305: finishing copy_from racing read to smithi04569269-27 2021-07-31T01:56:41.665 INFO:tasks.rados.rados.0.smithi045.stdout:3300: done (7 left) 2021-07-31T01:56:41.665 INFO:tasks.rados.rados.0.smithi045.stdout:3302: done (6 left) 2021-07-31T01:56:41.665 INFO:tasks.rados.rados.0.smithi045.stdout:3305: done (5 left) 2021-07-31T01:56:41.666 INFO:tasks.rados.rados.0.smithi045.stdout:3306: done (4 left) 2021-07-31T01:56:41.666 INFO:tasks.rados.rados.0.smithi045.stdout:3307: read oid 35 snap -1 2021-07-31T01:56:41.667 INFO:tasks.rados.rados.0.smithi045.stdout:3307: expect deleted 2021-07-31T01:56:41.668 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 33 current snap is 327 2021-07-31T01:56:41.668 INFO:tasks.rados.rados.0.smithi045.stdout:3308: seq_num 1021 ranges {4612096=303104} 2021-07-31T01:56:41.694 INFO:tasks.rados.rados.0.smithi045.stdout:3308: writing smithi04569269-33 from 4612096 to 4915200 tid 1 2021-07-31T01:56:41.694 INFO:tasks.rados.rados.0.smithi045.stdout:3309: rmattr oid 13 current snap is 327 2021-07-31T01:56:41.694 INFO:tasks.rados.rados.0.smithi045.stdout:3309: done (6 left) 2021-07-31T01:56:41.694 INFO:tasks.rados.rados.0.smithi045.stdout:3310: delete oid 22 current snap is 327 2021-07-31T01:56:41.695 INFO:tasks.rados.rados.0.smithi045.stdout:3303: finishing rollback tid 0 to smithi04569269-24 2021-07-31T01:56:41.695 INFO:tasks.rados.rados.0.smithi045.stdout:3303: finishing rollback tid 1 to smithi04569269-24 2021-07-31T01:56:41.695 INFO:tasks.rados.rados.0.smithi045.stdout:3303: finishing rollback tid 2 to smithi04569269-24 2021-07-31T01:56:41.695 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 24 v 1664 (ObjNum 1 snap 32731 seq_num 3189517592) dirty exists 2021-07-31T01:56:41.696 INFO:tasks.rados.rados.0.smithi045.stdout:3299: finishing write tid 3 to smithi04569269-15 2021-07-31T01:56:41.696 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 15 v 1858 (ObjNum 1019 snap 327 seq_num 1019) dirty exists 2021-07-31T01:56:41.696 INFO:tasks.rados.rados.0.smithi045.stdout:3299: left oid 15 (ObjNum 1019 snap 327 seq_num 1019) 2021-07-31T01:56:41.696 INFO:tasks.rados.rados.0.smithi045.stdout:3301: finishing rollback tid 0 to smithi04569269-17 2021-07-31T01:56:41.697 INFO:tasks.rados.rados.0.smithi045.stdout:3301: finishing rollback tid 1 to smithi04569269-17 2021-07-31T01:56:41.697 INFO:tasks.rados.rados.0.smithi045.stdout:3301: finishing rollback tid 2 to smithi04569269-17 2021-07-31T01:56:41.697 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 17 v 1723 (ObjNum 982 snap 303 seq_num 982) dirty exists 2021-07-31T01:56:41.700 INFO:tasks.rados.rados.0.smithi045.stdout:3299: done (6 left) 2021-07-31T01:56:41.700 INFO:tasks.rados.rados.0.smithi045.stdout:3301: done (5 left) 2021-07-31T01:56:41.700 INFO:tasks.rados.rados.0.smithi045.stdout:3303: done (4 left) 2021-07-31T01:56:41.701 INFO:tasks.rados.rados.0.smithi045.stdout:3307: done (3 left) 2021-07-31T01:56:41.701 INFO:tasks.rados.rados.0.smithi045.stdout:3310: done (2 left) 2021-07-31T01:56:41.702 INFO:tasks.rados.rados.0.smithi045.stdout:3311: rmattr oid 19 current snap is 327 2021-07-31T01:56:41.702 INFO:tasks.rados.rados.0.smithi045.stdout:3311: done (2 left) 2021-07-31T01:56:41.702 INFO:tasks.rados.rados.0.smithi045.stdout:3312: copy_from oid 40 from oid 14 current snap is 327 2021-07-31T01:56:41.703 INFO:tasks.rados.rados.0.smithi045.stdout:3313: rollback oid 3 current snap is 327 2021-07-31T01:56:41.703 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 3 to 299 2021-07-31T01:56:41.703 INFO:tasks.rados.rados.0.smithi045.stdout:3314: delete oid 20 current snap is 327 2021-07-31T01:56:41.704 INFO:tasks.rados.rados.0.smithi045.stdout:3308: finishing write tid 1 to smithi04569269-33 2021-07-31T01:56:41.704 INFO:tasks.rados.rados.0.smithi045.stdout:3308: finishing write tid 2 to smithi04569269-33 2021-07-31T01:56:41.704 INFO:tasks.rados.rados.0.smithi045.stdout:3312: finishing copy_from to smithi04569269-40 2021-07-31T01:56:41.705 INFO:tasks.rados.rados.0.smithi045.stdout:3312: got expected ENOENT (src dne) 2021-07-31T01:56:41.705 INFO:tasks.rados.rados.0.smithi045.stdout:3312: finishing copy_from racing read to smithi04569269-40 2021-07-31T01:56:41.705 INFO:tasks.rados.rados.0.smithi045.stdout:3304: finishing copy_from to smithi04569269-10 2021-07-31T01:56:41.706 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 10 v 1855 (ObjNum 754 snap 233 seq_num 754) dirty exists 2021-07-31T01:56:41.716 INFO:tasks.rados.rados.0.smithi045.stdout:3308: finishing write tid 3 to smithi04569269-33 2021-07-31T01:56:41.716 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 33 v 2025 (ObjNum 1021 snap 327 seq_num 1021) dirty exists 2021-07-31T01:56:41.716 INFO:tasks.rados.rados.0.smithi045.stdout:3308: left oid 33 (ObjNum 1021 snap 327 seq_num 1021) 2021-07-31T01:56:41.716 INFO:tasks.rados.rados.0.smithi045.stdout:3304: finishing copy_from racing read to smithi04569269-10 2021-07-31T01:56:41.717 INFO:tasks.rados.rados.0.smithi045.stdout:3313: finishing rollback tid 0 to smithi04569269-3 2021-07-31T01:56:41.717 INFO:tasks.rados.rados.0.smithi045.stdout:3313: finishing rollback tid 1 to smithi04569269-3 2021-07-31T01:56:41.717 INFO:tasks.rados.rados.0.smithi045.stdout:3313: finishing rollback tid 2 to smithi04569269-3 2021-07-31T01:56:41.717 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 3 v 2163 (ObjNum 881 snap 272 seq_num 881) dirty exists 2021-07-31T01:56:41.718 INFO:tasks.rados.rados.0.smithi045.stdout:3304: done (4 left) 2021-07-31T01:56:41.718 INFO:tasks.rados.rados.0.smithi045.stdout:3308: done (3 left) 2021-07-31T01:56:41.718 INFO:tasks.rados.rados.0.smithi045.stdout:3312: done (2 left) 2021-07-31T01:56:41.718 INFO:tasks.rados.rados.0.smithi045.stdout:3313: done (1 left) 2021-07-31T01:56:41.718 INFO:tasks.rados.rados.0.smithi045.stdout:3314: done (0 left) 2021-07-31T01:56:41.719 INFO:tasks.rados.rados.0.smithi045.stdout:3315: read oid 22 snap 300 2021-07-31T01:56:41.719 INFO:tasks.rados.rados.0.smithi045.stdout:3315: expect deleted 2021-07-31T01:56:41.719 INFO:tasks.rados.rados.0.smithi045.stdout:3316: setattr oid 42 current snap is 327 2021-07-31T01:56:41.719 INFO:tasks.rados.rados.0.smithi045.stdout:3315: done (1 left) 2021-07-31T01:56:41.720 INFO:tasks.rados.rados.0.smithi045.stdout:3317: snap_create 2021-07-31T01:56:41.720 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 42 v 1856 (ObjNum -1105182832 snap 21873 seq_num 3187445776) dirty exists 2021-07-31T01:56:42.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:41 smithi154 conmon[34553]: cluster 2021-07-31T01:56:40.708717+0000 mon.a (mon.0) 1352 : 2021-07-31T01:56:42.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:41 smithi154 conmon[34553]: cluster [DBG] osdmap e687: 8 total, 8 up, 8 in 2021-07-31T01:56:42.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:41 smithi154 conmon[34553]: cluster 2021-07-31T01:56:41.651139+0000 mon.a (mon.0) 1353 : cluster [DBG] osdmap e688: 8 total, 8 up, 8 in 2021-07-31T01:56:42.009 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T01:56:42.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:41 smithi045 conmon[33655]: cluster 2021-07-31T 2021-07-31T01:56:42.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:41 smithi045 conmon[33655]: 01:56:40.708717+0000 mon.a (mon.0) 1352 : cluster [DBG] osdmap e687: 8 total, 8 up, 8 in 2021-07-31T01:56:42.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:41 smithi045 conmon[33655]: cluster 2021-07-31T01:56:41.651139+0000 mon.a (mon.0) 1353 : cluster [DBG] 2021-07-31T01:56:42.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:41 smithi045 conmon[33655]: osdmap e688: 8 total, 8 up, 8 in 2021-07-31T01:56:42.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:41 smithi045 conmon[39718]: cluster 2021-07-31T01:56 2021-07-31T01:56:42.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:41 smithi045 conmon[39718]: :40.708717+0000 mon.a (mon.0) 1352 : cluster [DBG] osdmap e687: 8 total, 8 up, 8 in 2021-07-31T01:56:42.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:41 smithi045 conmon[39718]: cluster 2021-07-31T01:56:42.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:41 smithi045 conmon[39718]: 2021-07-31T01:56:41 2021-07-31T01:56:42.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:41 smithi045 conmon[39718]: .651139+0000 mon.a (mon. 2021-07-31T01:56:42.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:41 smithi045 conmon[39718]: 0) 1353 : cluster [DBG] 2021-07-31T01:56:42.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:41 smithi045 conmon[39718]: osdmap e688: 8 total, 8 up, 8 in 2021-07-31T01:56:42.287 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T01:56:42.288 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 11m ago 12m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T01:56:42.288 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 9m ago 12m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T01:56:42.288 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 9m ago 15m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T01:56:42.288 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 11m ago 17m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T01:56:42.289 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 11m ago 17m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T01:56:42.289 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 9m ago 15m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T01:56:42.289 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 11m ago 16m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T01:56:42.289 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 11m ago 12m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T01:56:42.289 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 9m ago 12m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T01:56:42.290 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 11m ago 15m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T01:56:42.290 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 11m ago 15m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T01:56:42.290 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 11m ago 14m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T01:56:42.290 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 11m ago 14m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T01:56:42.290 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 9m ago 14m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T01:56:42.291 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 9m ago 13m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T01:56:42.291 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 9m ago 13m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T01:56:42.291 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 9m ago 13m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T01:56:42.291 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 9m ago 12m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T01:56:42.622 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T01:56:42.622 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T01:56:42.623 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T01:56:42.623 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T01:56:42.623 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T01:56:42.623 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T01:56:42.624 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T01:56:42.624 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T01:56:42.624 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T01:56:42.624 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T01:56:42.624 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T01:56:42.625 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T01:56:42.625 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T01:56:42.625 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T01:56:42.625 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T01:56:42.626 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T01:56:42.626 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T01:56:42.716 INFO:tasks.rados.rados.0.smithi045.stdout:3316: done (1 left) 2021-07-31T01:56:42.716 INFO:tasks.rados.rados.0.smithi045.stdout:3317: done (0 left) 2021-07-31T01:56:42.716 INFO:tasks.rados.rados.0.smithi045.stdout:3318: rmattr oid 41 current snap is 328 2021-07-31T01:56:42.716 INFO:tasks.rados.rados.0.smithi045.stdout:3319: read oid 24 snap -1 2021-07-31T01:56:42.717 INFO:tasks.rados.rados.0.smithi045.stdout:3319: expect (ObjNum 0 snap 0 seq_num 0) 2021-07-31T01:56:42.718 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 29 current snap is 328 2021-07-31T01:56:42.718 INFO:tasks.rados.rados.0.smithi045.stdout:3320: seq_num 1024 ranges {0=352256} 2021-07-31T01:56:42.719 INFO:tasks.rados.rados.0.smithi045.stdout:3320: writing smithi04569269-29 from 0 to 352256 tid 1 2021-07-31T01:56:42.719 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 5 current snap is 328 2021-07-31T01:56:42.720 INFO:tasks.rados.rados.0.smithi045.stdout:3321: seq_num 1025 ranges {0=368640} 2021-07-31T01:56:42.720 INFO:tasks.rados.rados.0.smithi045.stdout:3321: writing smithi04569269-5 from 0 to 368640 tid 1 2021-07-31T01:56:42.721 INFO:tasks.rados.rados.0.smithi045.stdout:3322: rollback oid 22 current snap is 328 2021-07-31T01:56:42.721 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 22 to 255 2021-07-31T01:56:42.722 INFO:tasks.rados.rados.0.smithi045.stdout:3319: expect (ObjNum 0 snap 0 seq_num 0) 2021-07-31T01:56:42.723 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 41 v 1690 (ObjNum 948 snap 286 seq_num 948) dirty exists 2021-07-31T01:56:42.723 INFO:tasks.rados.rados.0.smithi045.stdout:3322: finishing rollback tid 1 to smithi04569269-22 2021-07-31T01:56:42.724 INFO:tasks.rados.rados.0.smithi045.stdout:3322: finishing rollback tid 2 to smithi04569269-22 2021-07-31T01:56:42.724 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 22 v 1659 (ObjNum 747 snap 230 seq_num 747) dirty exists 2021-07-31T01:56:42.724 INFO:tasks.rados.rados.0.smithi045.stdout:3318: done (4 left) 2021-07-31T01:56:42.725 INFO:tasks.rados.rados.0.smithi045.stdout:3319: done (3 left) 2021-07-31T01:56:42.725 INFO:tasks.rados.rados.0.smithi045.stdout:3322: done (2 left) 2021-07-31T01:56:42.726 INFO:tasks.rados.rados.0.smithi045.stdout:3323: snap_create 2021-07-31T01:56:42.727 INFO:tasks.rados.rados.0.smithi045.stdout:3320: finishing write tid 1 to smithi04569269-29 2021-07-31T01:56:42.727 INFO:tasks.rados.rados.0.smithi045.stdout:3320: finishing write tid 2 to smithi04569269-29 2021-07-31T01:56:42.727 INFO:tasks.rados.rados.0.smithi045.stdout:3321: finishing write tid 1 to smithi04569269-5 2021-07-31T01:56:42.728 INFO:tasks.rados.rados.0.smithi045.stdout:3321: finishing write tid 2 to smithi04569269-5 2021-07-31T01:56:42.728 INFO:tasks.rados.rados.0.smithi045.stdout:3320: finishing write tid 3 to smithi04569269-29 2021-07-31T01:56:42.728 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 29 v 1421 (ObjNum 1024 snap 328 seq_num 1024) dirty exists 2021-07-31T01:56:42.728 INFO:tasks.rados.rados.0.smithi045.stdout:3320: left oid 29 (ObjNum 1024 snap 328 seq_num 1024) 2021-07-31T01:56:42.729 INFO:tasks.rados.rados.0.smithi045.stdout:3321: finishing write tid 3 to smithi04569269-5 2021-07-31T01:56:42.729 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 5 v 2027 (ObjNum 1025 snap 328 seq_num 1025) dirty exists 2021-07-31T01:56:42.729 INFO:tasks.rados.rados.0.smithi045.stdout:3321: left oid 5 (ObjNum 1025 snap 328 seq_num 1025) 2021-07-31T01:56:43.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:42 smithi154 conmon[34553]: cluster 2021-07-31T01:56:41.928004+0000 mgr.x (mgr.24542) 406 : cluster [DBG] 2021-07-31T01:56:43.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:42 smithi154 conmon[34553]: pgmap v819: 25 pgs: 25 active+clean; 587 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 2.7 MiB/s wr, 13 op/s 2021-07-31T01:56:43.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:42 smithi154 conmon[34553]: audit 2021-07-31T01:56:41.998072+0000 mgr.x (mgr.24542) 407 : audit [DBG] from='client.24943 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:56:43.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:42 smithi154 conmon[34553]: audit 2021-07-31T01 2021-07-31T01:56:43.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:42 smithi154 conmon[34553]: :56:42.620726+0000 mon.a (mon.0) 1354 : audit [DBG] from='client.? 172.21.15.45:0/874820707' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T01:56:43.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:42 smithi154 conmon[34553]: 2021-07-31T01:56:43.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:42 smithi045 conmon[33655]: cluster 2021-07-31T01:56:41.928004+0000 mgr.x (mgr. 2021-07-31T01:56:43.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:42 smithi045 conmon[33655]: 24542) 406 : cluster [DBG] pgmap v819: 25 pgs: 25 active+clean; 587 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 2.7 MiB/s wr, 13 op/s 2021-07-31T01:56:43.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:42 smithi045 conmon[33655]: audit 2021-07-31T01:56:41.998072+0000 mgr.x (mgr.24542) 407 : audit [DBG] from='client.24943 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:56:43.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:42 smithi045 conmon[33655]: audit 2021-07- 2021-07-31T01:56:43.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:42 smithi045 conmon[33655]: 31T01:56:42.620726+0000 mon.a (mon.0) 1354 : audit [DBG] from='client.? 172.21.15.45:0/874820707' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T01:56:43.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:42 smithi045 conmon[39718]: cluster 2021-07-31T01:56:41.928004+0000 mgr.x 2021-07-31T01:56:43.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:42 smithi045 conmon[39718]: (mgr.24542) 406 : cluster [DBG] pgmap v819: 25 pgs: 25 active+clean; 587 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 2.7 MiB/s wr, 13 op/s 2021-07-31T01:56:43.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:42 smithi045 conmon[39718]: audit 2021-07-31T01:56:41.998072+0000 mgr.x ( 2021-07-31T01:56:43.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:42 smithi045 conmon[39718]: mgr.24542) 407 : audit [DBG] from='client.24943 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:56:43.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:42 smithi045 conmon[39718]: audit 2021- 2021-07-31T01:56:43.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:42 smithi045 conmon[39718]: 07-31T01:56:42.620726+0000 mon.a (mon.0) 1354 : audit [DBG] 2021-07-31T01:56:43.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:42 smithi045 conmon[39718]: from='client.? 172.21.15.45:0/874820707' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T01:56:43.722 INFO:tasks.rados.rados.0.smithi045.stdout:3320: done (2 left) 2021-07-31T01:56:43.722 INFO:tasks.rados.rados.0.smithi045.stdout:3321: done (1 left) 2021-07-31T01:56:43.722 INFO:tasks.rados.rados.0.smithi045.stdout:3323: done (0 left) 2021-07-31T01:56:43.723 INFO:tasks.rados.rados.0.smithi045.stdout:3324: rollback oid 6 current snap is 329 2021-07-31T01:56:43.723 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 6 to 324 2021-07-31T01:56:43.723 INFO:tasks.rados.rados.0.smithi045.stdout:3325: read oid 16 snap -1 2021-07-31T01:56:43.723 INFO:tasks.rados.rados.0.smithi045.stdout:3325: expect (ObjNum 944 snap 286 seq_num 944) 2021-07-31T01:56:43.723 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 35 current snap is 329 2021-07-31T01:56:43.724 INFO:tasks.rados.rados.0.smithi045.stdout:3326: seq_num 1026 ranges {0=385024} 2021-07-31T01:56:43.724 INFO:tasks.rados.rados.0.smithi045.stdout:3326: writing smithi04569269-35 from 0 to 385024 tid 1 2021-07-31T01:56:43.725 INFO:tasks.rados.rados.0.smithi045.stdout:3327: snap_create 2021-07-31T01:56:43.725 INFO:tasks.rados.rados.0.smithi045.stdout:3324: finishing rollback tid 1 to smithi04569269-6 2021-07-31T01:56:43.726 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 6 v 1873 (ObjNum 1953066355 snap 875587944 seq_num 842610229) dirty dne 2021-07-31T01:56:43.730 INFO:tasks.rados.rados.0.smithi045.stdout:3325: expect (ObjNum 944 snap 286 seq_num 944) 2021-07-31T01:56:43.768 INFO:tasks.rados.rados.0.smithi045.stdout:3326: finishing write tid 1 to smithi04569269-35 2021-07-31T01:56:43.768 INFO:tasks.rados.rados.0.smithi045.stdout:3326: finishing write tid 2 to smithi04569269-35 2021-07-31T01:56:43.769 INFO:tasks.rados.rados.0.smithi045.stdout:3326: finishing write tid 3 to smithi04569269-35 2021-07-31T01:56:43.769 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 35 v 2167 (ObjNum 1026 snap 329 seq_num 1026) dirty exists 2021-07-31T01:56:43.769 INFO:tasks.rados.rados.0.smithi045.stdout:3326: left oid 35 (ObjNum 1026 snap 329 seq_num 1026) 2021-07-31T01:56:43.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:43 smithi045 conmon[39718]: audit 2021-07-31T01:56:42.279204+0000 mgr.x (mgr.24542) 408 : audit [DBG] from='client.15042 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:56:43.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:43 smithi045 conmon[39718]: 2021-07-31T01:56:43.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:43 smithi045 conmon[39718]: cluster 2021-07-31T01:56:42.714923+0000 mon.a (mon.0) 1355 : cluster [DBG] osdmap e689: 8 total, 8 up, 8 in 2021-07-31T01:56:43.786 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:56:43 smithi045 conmon[61210]: level=error ts=2021-07-31T01:56:43.525Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:56:43.786 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:56:43 smithi045 conmon[61210]: level=error ts=2021-07-31T01:56:43.525Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:56:43.787 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:56:43 smithi045 conmon[61210]: level=error ts=2021-07-31T01:56:43.525Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T01:56:43.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:43 smithi045 conmon[33655]: audit 2021-07-31T01:56:42.279204+0000 mgr.x (mgr.24542) 408 : audit [DBG] from='client.15042 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:56:43.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:43 smithi045 conmon[33655]: cluster 2021-07-31T01:56:42.714923+0000 mon.a (mon.0) 1355 : cluster [DBG] osdmap e689: 8 total, 8 up, 8 in 2021-07-31T01:56:44.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:43 smithi154 conmon[34553]: audit 2021-07-31T01:56:42.279204+0000 mgr.x (mgr.24542) 408 : audit [DBG] from='client.15042 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:56:44.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:43 smithi154 conmon[34553]: cluster 2021-07-31T01: 2021-07-31T01:56:44.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:43 smithi154 conmon[34553]: 56:42.714923+0000 mon.a (mon.0) 1355 : cluster [DBG] osdmap e689: 8 total, 8 up, 8 in 2021-07-31T01:56:44.729 INFO:tasks.rados.rados.0.smithi045.stdout:3324: done (3 left) 2021-07-31T01:56:44.729 INFO:tasks.rados.rados.0.smithi045.stdout:3325: done (2 left) 2021-07-31T01:56:44.729 INFO:tasks.rados.rados.0.smithi045.stdout:3326: done (1 left) 2021-07-31T01:56:44.730 INFO:tasks.rados.rados.0.smithi045.stdout:3327: done (0 left) 2021-07-31T01:56:44.730 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 44 current snap is 330 2021-07-31T01:56:44.730 INFO:tasks.rados.rados.0.smithi045.stdout:3328: seq_num 1027 ranges {3391488=409600} 2021-07-31T01:56:44.751 INFO:tasks.rados.rados.0.smithi045.stdout:3328: writing smithi04569269-44 from 3391488 to 3801088 tid 1 2021-07-31T01:56:44.751 INFO:tasks.rados.rados.0.smithi045.stdout:3329: snap_remove snap 314 2021-07-31T01:56:45.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:44 smithi154 conmon[34553]: cluster 2021-07-31T01:56:43.721122+0000 mon.a (mon.0) 1356 : cluster [DBG] osdmap e690: 8 total, 8 up, 8 in 2021-07-31T01:56:45.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:44 smithi154 conmon[34553]: cluster 2021-07-31T01:56:43.928495+0000 mgr.x (mgr. 2021-07-31T01:56:45.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:44 smithi154 conmon[34553]: 24542) 409 : cluster [DBG] pgmap v822: 25 pgs: 25 active+clean; 590 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.0 MiB/s rd, 2.5 MiB/s wr, 16 op/s 2021-07-31T01:56:45.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:44 smithi045 conmon[39718]: cluster 2021-07-31T01:56:43 2021-07-31T01:56:45.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:44 smithi045 conmon[39718]: .721122+0000 mon.a (mon.0) 1356 : cluster [DBG] osdmap e690: 8 total, 8 up, 8 in 2021-07-31T01:56:45.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:44 smithi045 conmon[39718]: cluster 2021-07-31T01:56:45.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:44 smithi045 conmon[39718]: 2021-07-31T01:56:43 2021-07-31T01:56:45.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:44 smithi045 conmon[39718]: .928495+0000 mgr.x (mgr.24542 2021-07-31T01:56:45.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:44 smithi045 conmon[39718]: ) 409 : cluster [DBG] pgmap v822: 25 pgs: 25 active+clean; 590 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.0 MiB/s rd, 2.5 MiB/s wr, 16 op/s 2021-07-31T01:56:45.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:44 smithi045 conmon[33655]: cluster 2021-07-31T01:56:43.721122+0000 mon.a (mon.0) 1356 : cluster [DBG] osdmap e690: 8 total, 8 up, 8 in 2021-07-31T01:56:45.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:44 smithi045 conmon[33655]: cluster 2021-07-31T01:56:45.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:44 smithi045 conmon[33655]: 2021-07-31T01:56:43.928495+0000 mgr.x (mgr.24542) 409 : cluster [DBG] pgmap v822: 25 pgs: 25 active+clean; 590 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.0 MiB/s rd, 2.5 MiB/s wr, 16 op/s 2021-07-31T01:56:45.731 INFO:tasks.rados.rados.0.smithi045.stdout:3329: done (1 left) 2021-07-31T01:56:45.731 INFO:tasks.rados.rados.0.smithi045.stdout:3330: delete oid 42 current snap is 330 2021-07-31T01:56:45.732 INFO:tasks.rados.rados.0.smithi045.stdout:3328: finishing write tid 1 to smithi04569269-44 2021-07-31T01:56:45.732 INFO:tasks.rados.rados.0.smithi045.stdout:3328: finishing write tid 2 to smithi04569269-44 2021-07-31T01:56:45.733 INFO:tasks.rados.rados.0.smithi045.stdout:3328: finishing write tid 3 to smithi04569269-44 2021-07-31T01:56:45.733 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 44 v 1862 (ObjNum 1027 snap 330 seq_num 1027) dirty exists 2021-07-31T01:56:45.733 INFO:tasks.rados.rados.0.smithi045.stdout:3328: left oid 44 (ObjNum 1027 snap 330 seq_num 1027) 2021-07-31T01:56:45.744 INFO:tasks.rados.rados.0.smithi045.stdout:3328: done (1 left) 2021-07-31T01:56:45.744 INFO:tasks.rados.rados.0.smithi045.stdout:3330: done (0 left) 2021-07-31T01:56:45.745 INFO:tasks.rados.rados.0.smithi045.stdout:3331: setattr oid 19 current snap is 330 2021-07-31T01:56:45.745 INFO:tasks.rados.rados.0.smithi045.stdout:3332: snap_create 2021-07-31T01:56:45.748 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 19 v 1660 (ObjNum 1 snap 32731 seq_num 3190508048) dirty exists 2021-07-31T01:56:46.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:45 smithi154 conmon[34553]: cluster 2021-07-31T01:56:46.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:45 smithi154 conmon[34553]: 2021-07-31T01:56:44.728277+0000 mon.a (mon.0) 1357 2021-07-31T01:56:46.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:45 smithi154 conmon[34553]: : cluster [DBG] osdmap e691: 8 total, 8 up, 8 in 2021-07-31T01:56:46.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:45 smithi045 conmon[39718]: cluster 2021-07-31T01:56:44 2021-07-31T01:56:46.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:45 smithi045 conmon[39718]: .728277+0000 mon.a (mon.0) 1357 : cluster [DBG] osdmap e691: 8 total, 8 up, 8 in 2021-07-31T01:56:46.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:45 smithi045 conmon[33655]: cluster 2021-07-31T01:56:46.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:45 smithi045 conmon[33655]: 2021-07-31T01:56:46.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:45 smithi045 conmon[33655]: 2021-07-31T01:56:44.728277 2021-07-31T01:56:46.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:45 smithi045 conmon[33655]: +0000 mon.a (mon.0) 1357 : cluster [DBG] osdmap e691: 8 total, 8 up, 8 in 2021-07-31T01:56:46.654 INFO:tasks.rados.rados.0.smithi045.stdout:3331: done (1 left) 2021-07-31T01:56:46.654 INFO:tasks.rados.rados.0.smithi045.stdout:3332: done (0 left) 2021-07-31T01:56:46.654 INFO:tasks.rados.rados.0.smithi045.stdout:3333: rollback oid 1 current snap is 331 2021-07-31T01:56:46.655 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 1 to 329 2021-07-31T01:56:46.656 INFO:tasks.rados.rados.0.smithi045.stdout:3334: rollback oid 18 current snap is 331 2021-07-31T01:56:46.656 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 18 to 325 2021-07-31T01:56:46.656 INFO:tasks.rados.rados.0.smithi045.stdout:3335: copy_from oid 6 from oid 4 current snap is 331 2021-07-31T01:56:46.656 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 15 current snap is 331 2021-07-31T01:56:46.657 INFO:tasks.rados.rados.0.smithi045.stdout:3336: seq_num 1029 ranges {270336=442368} 2021-07-31T01:56:46.658 INFO:tasks.rados.rados.0.smithi045.stdout:3336: writing smithi04569269-15 from 270336 to 712704 tid 1 2021-07-31T01:56:46.658 INFO:tasks.rados.rados.0.smithi045.stdout:3337: read oid 39 snap 303 2021-07-31T01:56:46.658 INFO:tasks.rados.rados.0.smithi045.stdout:3337: expect (ObjNum 949 snap 286 seq_num 949) 2021-07-31T01:56:46.659 INFO:tasks.rados.rados.0.smithi045.stdout:3338: snap_create 2021-07-31T01:56:46.662 INFO:tasks.rados.rados.0.smithi045.stdout:3333: finishing rollback tid 0 to smithi04569269-1 2021-07-31T01:56:46.664 INFO:tasks.rados.rados.0.smithi045.stdout:3333: finishing rollback tid 1 to smithi04569269-1 2021-07-31T01:56:46.667 INFO:tasks.rados.rados.0.smithi045.stdout:3334: finishing rollback tid 0 to smithi04569269-18 2021-07-31T01:56:46.668 INFO:tasks.rados.rados.0.smithi045.stdout:3333: finishing rollback tid 2 to smithi04569269-1 2021-07-31T01:56:46.669 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 1 v 1856 (ObjNum 713 snap 222 seq_num 713) dirty exists 2021-07-31T01:56:46.673 INFO:tasks.rados.rados.0.smithi045.stdout:3334: finishing rollback tid 1 to smithi04569269-18 2021-07-31T01:56:46.673 INFO:tasks.rados.rados.0.smithi045.stdout:3334: finishing rollback tid 2 to smithi04569269-18 2021-07-31T01:56:46.673 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 18 v 1862 (ObjNum 928 snap 281 seq_num 928) dirty exists 2021-07-31T01:56:46.674 INFO:tasks.rados.rados.0.smithi045.stdout:3336: finishing write tid 1 to smithi04569269-15 2021-07-31T01:56:46.674 INFO:tasks.rados.rados.0.smithi045.stdout:3336: finishing write tid 2 to smithi04569269-15 2021-07-31T01:56:46.675 INFO:tasks.rados.rados.0.smithi045.stdout:3336: finishing write tid 3 to smithi04569269-15 2021-07-31T01:56:46.675 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 15 v 1864 (ObjNum 1029 snap 331 seq_num 1029) dirty exists 2021-07-31T01:56:46.676 INFO:tasks.rados.rados.0.smithi045.stdout:3336: left oid 15 (ObjNum 1029 snap 331 seq_num 1029) 2021-07-31T01:56:46.687 INFO:tasks.rados.rados.0.smithi045.stdout:3335: finishing copy_from to smithi04569269-6 2021-07-31T01:56:46.688 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 6 v 1856 (ObjNum 996 snap 311 seq_num 996) dirty exists 2021-07-31T01:56:46.688 INFO:tasks.rados.rados.0.smithi045.stdout:3335: finishing copy_from racing read to smithi04569269-6 2021-07-31T01:56:46.703 INFO:tasks.rados.rados.0.smithi045.stdout:3337: expect (ObjNum 949 snap 286 seq_num 949) 2021-07-31T01:56:47.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:46 smithi154 conmon[34553]: cluster 2021-07-31T01:56:45.729909+0000 mon.a (mon.0) 1358 : 2021-07-31T01:56:47.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:46 smithi154 conmon[34553]: cluster [DBG] osdmap e692: 8 total, 8 up, 8 in 2021-07-31T01:56:47.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:46 smithi154 conmon[34553]: cluster 2021-07-31T01:56:45.928901+0000 mgr.x (mgr.24542) 410 : cluster [DBG] pgmap v825: 25 pgs: 25 active+clean; 590 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.0 MiB/s rd, 2.5 MiB/s wr, 16 op/s 2021-07-31T01:56:47.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:46 smithi154 conmon[34553]: cluster 2021-07-31T01:56:46.652946+0000 mon.a (mon.0) 1359 : cluster [DBG] osdmap e693: 8 total, 8 up, 8 in 2021-07-31T01:56:47.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:46 smithi045 conmon[39718]: cluster 2021-07-31T01:56:45.729909+0000 mon.a (mon 2021-07-31T01:56:47.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:46 smithi045 conmon[39718]: .0) 1358 : cluster [DBG] osdmap e692: 8 total, 8 up, 8 in 2021-07-31T01:56:47.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:46 smithi045 conmon[39718]: cluster 2021-07-31T01:56:45.928901+0000 mgr.x (mgr 2021-07-31T01:56:47.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:46 smithi045 conmon[39718]: .24542) 410 : cluster [DBG] pgmap v825: 25 pgs: 25 active+clean; 590 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.0 MiB/s rd, 2.5 MiB/s wr, 16 op/s 2021-07-31T01:56:47.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:46 smithi045 conmon[39718]: cluster 2021-07-31T01:56 2021-07-31T01:56:47.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:46 smithi045 conmon[39718]: :46.652946+0000 mon.a (mon.0) 1359 : cluster [DBG] osdmap e693: 8 total, 8 up, 8 in 2021-07-31T01:56:47.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:46 smithi045 conmon[33655]: cluster 2021-07-31T01:56:45.729909+0000 mon.a (mon. 2021-07-31T01:56:47.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:46 smithi045 conmon[33655]: 0) 1358 : cluster [DBG] osdmap e692: 8 total, 8 up, 8 in 2021-07-31T01:56:47.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:46 smithi045 conmon[33655]: cluster 2021-07-31T01:56:45.928901+0000 mgr.x (mgr.24542) 410 : cluster [DBG] pgmap v825: 25 pgs: 25 active+clean; 590 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.0 MiB/s rd, 2.5 MiB/s wr, 16 op/s 2021-07-31T01:56:47.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:46 smithi045 conmon[33655]: cluster 2021-07-31T 2021-07-31T01:56:47.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:46 smithi045 conmon[33655]: 01:56:46.652946+0000 mon.a (mon.0) 1359 : cluster [DBG] osdmap e693: 8 total, 8 up, 8 in 2021-07-31T01:56:47.659 INFO:tasks.rados.rados.0.smithi045.stdout:3333: done (5 left) 2021-07-31T01:56:47.659 INFO:tasks.rados.rados.0.smithi045.stdout:3334: done (4 left) 2021-07-31T01:56:47.659 INFO:tasks.rados.rados.0.smithi045.stdout:3335: done (3 left) 2021-07-31T01:56:47.660 INFO:tasks.rados.rados.0.smithi045.stdout:3336: done (2 left) 2021-07-31T01:56:47.660 INFO:tasks.rados.rados.0.smithi045.stdout:3337: done (1 left) 2021-07-31T01:56:47.660 INFO:tasks.rados.rados.0.smithi045.stdout:3338: done (0 left) 2021-07-31T01:56:47.660 INFO:tasks.rados.rados.0.smithi045.stdout:3339: rmattr oid 27 current snap is 332 2021-07-31T01:56:47.660 INFO:tasks.rados.rados.0.smithi045.stdout:3340: copy_from oid 28 from oid 23 current snap is 332 2021-07-31T01:56:47.661 INFO:tasks.rados.rados.0.smithi045.stdout:3341: setattr oid 29 current snap is 332 2021-07-31T01:56:47.661 INFO:tasks.rados.rados.0.smithi045.stdout:3342: read oid 46 snap -1 2021-07-31T01:56:47.662 INFO:tasks.rados.rados.0.smithi045.stdout:3342: expect (ObjNum 1001 snap 313 seq_num 1001) 2021-07-31T01:56:47.662 INFO:tasks.rados.rados.0.smithi045.stdout:3343: rollback oid 4 current snap is 332 2021-07-31T01:56:47.662 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 4 to 307 2021-07-31T01:56:47.662 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 37 current snap is 332 2021-07-31T01:56:47.663 INFO:tasks.rados.rados.0.smithi045.stdout:3344: seq_num 1030 ranges {0=458752} 2021-07-31T01:56:47.663 INFO:tasks.rados.rados.0.smithi045.stdout:3344: writing smithi04569269-37 from 0 to 458752 tid 1 2021-07-31T01:56:47.664 INFO:tasks.rados.rados.0.smithi045.stdout:3345: rollback oid 22 current snap is 332 2021-07-31T01:56:47.664 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 22 to 307 2021-07-31T01:56:47.664 INFO:tasks.rados.rados.0.smithi045.stdout:3346: setattr oid 35 current snap is 332 2021-07-31T01:56:47.666 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 29 v 1422 (ObjNum 1024 snap 328 seq_num 1024) dirty exists 2021-07-31T01:56:47.666 INFO:tasks.rados.rados.0.smithi045.stdout:3341: done (7 left) 2021-07-31T01:56:47.666 INFO:tasks.rados.rados.0.smithi045.stdout:3347: snap_create 2021-07-31T01:56:47.667 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 27 v 1863 (ObjNum 288 snap 0 seq_num 65) dirty exists 2021-07-31T01:56:47.668 INFO:tasks.rados.rados.0.smithi045.stdout:3345: finishing rollback tid 0 to smithi04569269-22 2021-07-31T01:56:47.669 INFO:tasks.rados.rados.0.smithi045.stdout:3345: finishing rollback tid 1 to smithi04569269-22 2021-07-31T01:56:47.669 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 22 v 1662 (ObjNum 0 snap 0 seq_num 0) dirty dne 2021-07-31T01:56:47.671 INFO:tasks.rados.rados.0.smithi045.stdout:3343: finishing rollback tid 0 to smithi04569269-4 2021-07-31T01:56:47.672 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 35 v 2179 (ObjNum 1026 snap 329 seq_num 1026) dirty exists 2021-07-31T01:56:47.672 INFO:tasks.rados.rados.0.smithi045.stdout:3343: finishing rollback tid 1 to smithi04569269-4 2021-07-31T01:56:47.672 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 4 v 1866 (ObjNum -1104322544 snap 21873 seq_num 3190593424) dirty dne 2021-07-31T01:56:47.673 INFO:tasks.rados.rados.0.smithi045.stdout:3344: finishing write tid 1 to smithi04569269-37 2021-07-31T01:56:47.673 INFO:tasks.rados.rados.0.smithi045.stdout:3344: finishing write tid 2 to smithi04569269-37 2021-07-31T01:56:47.675 INFO:tasks.rados.rados.0.smithi045.stdout:3344: finishing write tid 3 to smithi04569269-37 2021-07-31T01:56:47.675 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 37 v 1868 (ObjNum 1030 snap 332 seq_num 1030) dirty exists 2021-07-31T01:56:47.675 INFO:tasks.rados.rados.0.smithi045.stdout:3344: left oid 37 (ObjNum 1030 snap 332 seq_num 1030) 2021-07-31T01:56:47.679 INFO:tasks.rados.rados.0.smithi045.stdout:3340: finishing copy_from to smithi04569269-28 2021-07-31T01:56:47.679 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 28 v 2021 (ObjNum 1009 snap 323 seq_num 1009) dirty exists 2021-07-31T01:56:47.680 INFO:tasks.rados.rados.0.smithi045.stdout:3340: finishing copy_from racing read to smithi04569269-28 2021-07-31T01:56:47.691 INFO:tasks.rados.rados.0.smithi045.stdout:3342: expect (ObjNum 1001 snap 313 seq_num 1001) 2021-07-31T01:56:48.665 INFO:tasks.rados.rados.0.smithi045.stdout:3339: done (7 left) 2021-07-31T01:56:48.666 INFO:tasks.rados.rados.0.smithi045.stdout:3340: done (6 left) 2021-07-31T01:56:48.666 INFO:tasks.rados.rados.0.smithi045.stdout:3342: done (5 left) 2021-07-31T01:56:48.667 INFO:tasks.rados.rados.0.smithi045.stdout:3343: done (4 left) 2021-07-31T01:56:48.667 INFO:tasks.rados.rados.0.smithi045.stdout:3344: done (3 left) 2021-07-31T01:56:48.667 INFO:tasks.rados.rados.0.smithi045.stdout:3345: done (2 left) 2021-07-31T01:56:48.668 INFO:tasks.rados.rados.0.smithi045.stdout:3346: done (1 left) 2021-07-31T01:56:48.668 INFO:tasks.rados.rados.0.smithi045.stdout:3347: done (0 left) 2021-07-31T01:56:48.669 INFO:tasks.rados.rados.0.smithi045.stdout:3348: read oid 20 snap -1 2021-07-31T01:56:48.669 INFO:tasks.rados.rados.0.smithi045.stdout:3348: expect deleted 2021-07-31T01:56:48.669 INFO:tasks.rados.rados.0.smithi045.stdout:3349: snap_create 2021-07-31T01:56:49.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:48 smithi154 conmon[34553]: cluster 2021-07-31T01:56:47.658016+0000 mon.a (mon 2021-07-31T01:56:49.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:48 smithi154 conmon[34553]: .0) 1360 : cluster [DBG] osdmap e694: 8 total, 8 up, 8 in 2021-07-31T01:56:49.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:48 smithi154 conmon[34553]: cluster 2021-07-31T01:56:47.929334+0000 mgr.x (mgr.24542) 411 : cluster [DBG] pgmap v828: 25 pgs: 25 active+clean; 590 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:56:49.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:48 smithi045 conmon[39718]: cluster 2021-07-31T01:56:47.658016+0000 mon.a (mon.0) 1360 : cluster [DBG] osdmap e694: 8 total, 8 up, 8 in 2021-07-31T01:56:49.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:48 smithi045 conmon[39718]: cluster 2021-07-31T01:56:47.929334+0000 mgr.x (mgr.24542) 411 : cluster [DBG] pgmap v828: 25 pgs: 25 active+clean; 590 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:56:49.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:48 smithi045 conmon[33655]: cluster 2021-07-31T01:56:47.658016+0000 mon.a (mon.0) 1360 : cluster [DBG] osdmap e694: 8 total, 8 up, 8 in 2021-07-31T01:56:49.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:48 smithi045 conmon[33655]: cluster 2021-07-31T01:56:47.929334+0000 mgr.x (mgr.24542) 411 : cluster [DBG] pgmap v828: 25 pgs: 25 active+clean; 590 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:56:49.667 INFO:tasks.rados.rados.0.smithi045.stdout:3348: done (1 left) 2021-07-31T01:56:49.667 INFO:tasks.rados.rados.0.smithi045.stdout:3349: done (0 left) 2021-07-31T01:56:49.667 INFO:tasks.rados.rados.0.smithi045.stdout:3350: snap_create 2021-07-31T01:56:50.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:49 smithi154 conmon[34553]: cluster 2021-07-31T01:56:48.664236+0000 mon.a (mon.0) 1361 : cluster [DBG] osdmap e695: 8 total, 8 up, 8 in 2021-07-31T01:56:50.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:49 smithi045 conmon[39718]: cluster 2021-07-31T01:56:48.664236+0000 mon.a (mon.0) 1361 : cluster 2021-07-31T01:56:50.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:49 smithi045 conmon[39718]: [DBG] osdmap e695: 8 total, 8 up, 8 in 2021-07-31T01:56:50.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:49 smithi045 conmon[33655]: cluster 2021-07-31T01:56:48.664236+0000 mon.a (mon.0) 1361 2021-07-31T01:56:50.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:49 smithi045 conmon[33655]: : cluster [DBG] osdmap e695: 8 total, 8 up, 8 in 2021-07-31T01:56:50.678 INFO:tasks.rados.rados.0.smithi045.stdout:3350: done (0 left) 2021-07-31T01:56:50.678 INFO:tasks.rados.rados.0.smithi045.stdout:3351: rmattr oid 8 current snap is 335 2021-07-31T01:56:50.679 INFO:tasks.rados.rados.0.smithi045.stdout:3352: copy_from oid 30 from oid 7 current snap is 335 2021-07-31T01:56:50.679 INFO:tasks.rados.rados.0.smithi045.stdout:3353: setattr oid 47 current snap is 335 2021-07-31T01:56:50.679 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 8 v 0 (ObjNum 994 snap 309 seq_num 994) dirty exists 2021-07-31T01:56:50.680 INFO:tasks.rados.rados.0.smithi045.stdout:3351: done (2 left) 2021-07-31T01:56:50.680 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 25 current snap is 335 2021-07-31T01:56:50.680 INFO:tasks.rados.rados.0.smithi045.stdout:3354: seq_num 1031 ranges {2187264=475136} 2021-07-31T01:56:50.695 INFO:tasks.rados.rados.0.smithi045.stdout:3354: writing smithi04569269-25 from 2187264 to 2662400 tid 1 2021-07-31T01:56:50.695 INFO:tasks.rados.rados.0.smithi045.stdout:3355: snap_remove snap 279 2021-07-31T01:56:51.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:50 smithi154 conmon[34553]: cluster 2021-07-31T01:56:49.666536+0000 mon.a (mon.0) 1362 : cluster [DBG] 2021-07-31T01:56:51.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:50 smithi154 conmon[34553]: osdmap e696: 8 total, 8 up, 8 in 2021-07-31T01:56:51.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:50 smithi154 conmon[34553]: cluster 2021-07-31T01:56:49.929874+0000 mgr.x (mgr.24542) 412 : cluster [DBG] pgmap v831: 25 pgs: 25 active+clean; 595 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 10 MiB/s rd, 1.1 MiB/s wr, 24 op/s 2021-07-31T01:56:51.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:50 smithi045 conmon[39718]: cluster 2021-07-31T01:56:49.666536+0000 mon.a ( 2021-07-31T01:56:51.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:50 smithi045 conmon[39718]: mon.0) 1362 : cluster [DBG] osdmap e696: 8 total, 8 up, 8 in 2021-07-31T01:56:51.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:50 smithi045 conmon[39718]: cluster 2021-07-31T01:56:49.929874+0000 mgr.x (mgr 2021-07-31T01:56:51.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:50 smithi045 conmon[39718]: .24542) 412 : cluster [DBG] pgmap v831: 25 pgs: 25 active+clean; 595 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 10 MiB/s rd, 1.1 MiB/s wr, 24 op/s 2021-07-31T01:56:51.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:50 smithi045 conmon[33655]: cluster 2021-07-31T01:56:49.666536+0000 mon.a (mon.0) 1362 : cluster 2021-07-31T01:56:51.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:50 smithi045 conmon[33655]: [DBG] osdmap e696: 8 total, 8 up, 8 in 2021-07-31T01:56:51.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:50 smithi045 conmon[33655]: cluster 2021-07-31T01:56:49.929874+0000 mgr.x (mgr.24542) 412 : cluster [DBG] pgmap v831: 25 pgs: 25 active+clean; 595 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 10 MiB/s rd, 1.1 MiB/s wr, 24 op/s 2021-07-31T01:56:51.655 INFO:tasks.rados.rados.0.smithi045.stdout:3355: done (3 left) 2021-07-31T01:56:51.655 INFO:tasks.rados.rados.0.smithi045.stdout:3356: read oid 4 snap 295 2021-07-31T01:56:51.656 INFO:tasks.rados.rados.0.smithi045.stdout:3356: expect (ObjNum 881 snap 272 seq_num 881) 2021-07-31T01:56:51.656 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 43 current snap is 335 2021-07-31T01:56:51.657 INFO:tasks.rados.rados.0.smithi045.stdout:3352: finishing copy_from to smithi04569269-30 2021-07-31T01:56:51.657 INFO:tasks.rados.rados.0.smithi045.stdout:3352: got expected ENOENT (src dne) 2021-07-31T01:56:51.657 INFO:tasks.rados.rados.0.smithi045.stdout:3357: seq_num 1032 ranges {9936896=491520} 2021-07-31T01:56:51.715 INFO:tasks.rados.rados.0.smithi045.stdout:3357: writing smithi04569269-43 from 9936896 to 10428416 tid 1 2021-07-31T01:56:51.715 INFO:tasks.rados.rados.0.smithi045.stdout:3358: delete oid 26 current snap is 335 2021-07-31T01:56:51.716 INFO:tasks.rados.rados.0.smithi045.stdout:3352: finishing copy_from racing read to smithi04569269-30 2021-07-31T01:56:51.716 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 47 v 1839 (ObjNum 1953066355 snap 875587944 seq_num 842610229) dirty exists 2021-07-31T01:56:51.716 INFO:tasks.rados.rados.0.smithi045.stdout:3354: finishing write tid 1 to smithi04569269-25 2021-07-31T01:56:51.716 INFO:tasks.rados.rados.0.smithi045.stdout:3354: finishing write tid 2 to smithi04569269-25 2021-07-31T01:56:51.717 INFO:tasks.rados.rados.0.smithi045.stdout:3354: finishing write tid 3 to smithi04569269-25 2021-07-31T01:56:51.717 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 25 v 1841 (ObjNum 1031 snap 335 seq_num 1031) dirty exists 2021-07-31T01:56:51.717 INFO:tasks.rados.rados.0.smithi045.stdout:3354: left oid 25 (ObjNum 1031 snap 335 seq_num 1031) 2021-07-31T01:56:51.717 INFO:tasks.rados.rados.0.smithi045.stdout:3356: expect (ObjNum 881 snap 272 seq_num 881) 2021-07-31T01:56:52.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:51 smithi154 conmon[34553]: cluster 2021-07-31T01:56:52.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:51 smithi154 conmon[34553]: 2021-07-31T01:56:52.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:51 smithi154 conmon[34553]: 2021 2021-07-31T01:56:52.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:51 smithi154 conmon[34553]: - 2021-07-31T01:56:52.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:51 smithi154 conmon[34553]: 07 2021-07-31T01:56:52.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:51 smithi154 conmon[34553]: - 2021-07-31T01:56:52.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:51 smithi154 conmon[34553]: 31T01:56:50.676814+0000 mon.a (mon. 2021-07-31T01:56:52.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:51 smithi154 conmon[34553]: 0) 1363 : cluster [DBG] osdmap e697: 8 total, 8 up, 8 in 2021-07-31T01:56:52.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:51 smithi154 conmon[34553]: cluster 2021-07-31T01:56:51.653237 2021-07-31T01:56:52.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:51 smithi154 conmon[34553]: +0000 mon.a (mon.0) 1364 : cluster [DBG] osdmap e698: 8 total, 8 up, 8 in 2021-07-31T01:56:52.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:51 smithi045 conmon[39718]: cluster 2021-07-31T01:56:52.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:51 smithi045 conmon[39718]: 2021-07-31T01:56:52.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:51 smithi045 conmon[39718]: 2021 2021-07-31T01:56:52.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:51 smithi045 conmon[39718]: - 2021-07-31T01:56:52.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:51 smithi045 conmon[39718]: 07 2021-07-31T01:56:52.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:51 smithi045 conmon[39718]: - 2021-07-31T01:56:52.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:51 smithi045 conmon[39718]: 31 2021-07-31T01:56:52.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:51 smithi045 conmon[39718]: T 2021-07-31T01:56:52.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:51 smithi045 conmon[39718]: 01 2021-07-31T01:56:52.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:51 smithi045 conmon[39718]: : 2021-07-31T01:56:52.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:51 smithi045 conmon[39718]: 56 2021-07-31T01:56:52.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:51 smithi045 conmon[39718]: : 2021-07-31T01:56:52.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:51 smithi045 conmon[39718]: 50 2021-07-31T01:56:52.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:51 smithi045 conmon[39718]: . 2021-07-31T01:56:52.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:51 smithi045 conmon[39718]: 676814 2021-07-31T01:56:52.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:51 smithi045 conmon[39718]: +0000 2021-07-31T01:56:52.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:51 smithi045 conmon[39718]: 2021-07-31T01:56:52.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:51 smithi045 conmon[39718]: mon.a 2021-07-31T01:56:52.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:51 smithi045 conmon[39718]: ( 2021-07-31T01:56:52.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:51 smithi045 conmon[39718]: mon 2021-07-31T01:56:52.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:51 smithi045 conmon[39718]: . 2021-07-31T01:56:52.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:51 smithi045 conmon[39718]: 0 2021-07-31T01:56:52.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:51 smithi045 conmon[39718]: ) 2021-07-31T01:56:52.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:51 smithi045 conmon[39718]: 1363 2021-07-31T01:56:52.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:51 smithi045 conmon[39718]: : 2021-07-31T01:56:52.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:51 smithi045 conmon[39718]: cluster 2021-07-31T01:56:52.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:51 smithi045 conmon[39718]: 2021-07-31T01:56:52.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:51 smithi045 conmon[39718]: [DBG] 2021-07-31T01:56:52.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:51 smithi045 conmon[39718]: 2021-07-31T01:56:52.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:51 smithi045 conmon[39718]: osdmap e697: 8 total, 8 up, 8 in 2021-07-31T01:56:52.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:51 smithi045 conmon[39718]: 2021-07-31T01:56:52.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:51 smithi045 conmon[39718]: cluster 2021-07-31T01:56:52.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:51 smithi045 conmon[39718]: 2021-07-31T01:56:52.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:51 smithi045 conmon[39718]: 2021 2021-07-31T01:56:52.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:51 smithi045 conmon[39718]: - 2021-07-31T01:56:52.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:51 smithi045 conmon[39718]: 07 2021-07-31T01:56:52.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:51 smithi045 conmon[39718]: - 2021-07-31T01:56:52.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:51 smithi045 conmon[39718]: 31 2021-07-31T01:56:52.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:51 smithi045 conmon[39718]: T 2021-07-31T01:56:52.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:51 smithi045 conmon[39718]: 01 2021-07-31T01:56:52.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:51 smithi045 conmon[39718]: : 2021-07-31T01:56:52.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:51 smithi045 conmon[39718]: 56 2021-07-31T01:56:52.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:51 smithi045 conmon[39718]: : 2021-07-31T01:56:52.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:51 smithi045 conmon[39718]: 51 2021-07-31T01:56:52.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:51 smithi045 conmon[39718]: . 2021-07-31T01:56:52.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:51 smithi045 conmon[39718]: 653237 2021-07-31T01:56:52.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:51 smithi045 conmon[39718]: +0000 2021-07-31T01:56:52.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:51 smithi045 conmon[39718]: 2021-07-31T01:56:52.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:51 smithi045 conmon[39718]: mon.a 2021-07-31T01:56:52.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:51 smithi045 conmon[39718]: ( 2021-07-31T01:56:52.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:51 smithi045 conmon[39718]: mon 2021-07-31T01:56:52.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:51 smithi045 conmon[39718]: . 2021-07-31T01:56:52.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:51 smithi045 conmon[39718]: 0 2021-07-31T01:56:52.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:51 smithi045 conmon[39718]: ) 2021-07-31T01:56:52.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:51 smithi045 conmon[39718]: 1364 2021-07-31T01:56:52.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:51 smithi045 conmon[39718]: : 2021-07-31T01:56:52.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:51 smithi045 conmon[39718]: cluster 2021-07-31T01:56:52.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:51 smithi045 conmon[39718]: 2021-07-31T01:56:52.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:51 smithi045 conmon[39718]: [DBG] 2021-07-31T01:56:52.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:51 smithi045 conmon[39718]: 2021-07-31T01:56:52.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:51 smithi045 conmon[39718]: osdmap e698: 8 total, 8 up, 8 in 2021-07-31T01:56:52.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:51 smithi045 conmon[39718]: 2021-07-31T01:56:52.050 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:51 smithi045 conmon[33655]: cluster 2021-07-31T01:56:52.050 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:51 smithi045 conmon[33655]: 2021-07-31T01:56:52.051 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:51 smithi045 conmon[33655]: 2021 2021-07-31T01:56:52.051 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:51 smithi045 conmon[33655]: - 2021-07-31T01:56:52.051 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:51 smithi045 conmon[33655]: 07 2021-07-31T01:56:52.051 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:51 smithi045 conmon[33655]: - 2021-07-31T01:56:52.052 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:51 smithi045 conmon[33655]: 31T01:56:50.676814+0000 2021-07-31T01:56:52.052 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:51 smithi045 conmon[33655]: mon.a (mon.0) 1363 : cluster [DBG] osdmap e697: 8 total, 8 up, 8 in 2021-07-31T01:56:52.052 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:51 smithi045 conmon[33655]: cluster 2021-07-31T01:56:52.053 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:51 smithi045 conmon[33655]: 2021-07-31T01:56:51.653237+0000 mon.a (mon.0) 1364 : cluster [DBG] osdmap e698: 8 total, 8 up, 8 in 2021-07-31T01:56:52.739 INFO:tasks.rados.rados.0.smithi045.stdout:3357: finishing write tid 1 to smithi04569269-43 2021-07-31T01:56:52.740 INFO:tasks.rados.rados.0.smithi045.stdout:3357: finishing write tid 2 to smithi04569269-43 2021-07-31T01:56:52.740 INFO:tasks.rados.rados.0.smithi045.stdout:3357: finishing write tid 3 to smithi04569269-43 2021-07-31T01:56:52.740 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 43 v 1865 (ObjNum 1032 snap 335 seq_num 1032) dirty exists 2021-07-31T01:56:52.742 INFO:tasks.rados.rados.0.smithi045.stdout:3357: left oid 43 (ObjNum 1032 snap 335 seq_num 1032) 2021-07-31T01:56:52.742 INFO:tasks.rados.rados.0.smithi045.stdout:3352: done (5 left) 2021-07-31T01:56:52.742 INFO:tasks.rados.rados.0.smithi045.stdout:3353: done (4 left) 2021-07-31T01:56:52.743 INFO:tasks.rados.rados.0.smithi045.stdout:3354: done (3 left) 2021-07-31T01:56:52.743 INFO:tasks.rados.rados.0.smithi045.stdout:3356: done (2 left) 2021-07-31T01:56:52.743 INFO:tasks.rados.rados.0.smithi045.stdout:3357: done (1 left) 2021-07-31T01:56:52.743 INFO:tasks.rados.rados.0.smithi045.stdout:3358: done (0 left) 2021-07-31T01:56:52.743 INFO:tasks.rados.rados.0.smithi045.stdout:3359: delete oid 48 current snap is 335 2021-07-31T01:56:52.744 INFO:tasks.rados.rados.0.smithi045.stdout:3359: done (0 left) 2021-07-31T01:56:52.744 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 37 current snap is 335 2021-07-31T01:56:52.745 INFO:tasks.rados.rados.0.smithi045.stdout:3360: seq_num 1035 ranges {458752=540672} 2021-07-31T01:56:52.745 INFO:tasks.rados.rados.0.smithi045.stdout:3360: writing smithi04569269-37 from 458752 to 999424 tid 1 2021-07-31T01:56:52.745 INFO:tasks.rados.rados.0.smithi045.stdout:3361: snap_create 2021-07-31T01:56:52.749 INFO:tasks.rados.rados.0.smithi045.stdout:3360: finishing write tid 1 to smithi04569269-37 2021-07-31T01:56:52.749 INFO:tasks.rados.rados.0.smithi045.stdout:3360: finishing write tid 2 to smithi04569269-37 2021-07-31T01:56:52.749 INFO:tasks.rados.rados.0.smithi045.stdout:3360: finishing write tid 3 to smithi04569269-37 2021-07-31T01:56:52.750 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 37 v 1870 (ObjNum 1035 snap 335 seq_num 1035) dirty exists 2021-07-31T01:56:52.750 INFO:tasks.rados.rados.0.smithi045.stdout:3360: left oid 37 (ObjNum 1035 snap 335 seq_num 1035) 2021-07-31T01:56:53.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:52 smithi154 conmon[34553]: cluster 2021-07-31T01:56:51.930259+0000 mgr.x (mgr.24542) 413 : cluster [DBG] pgmap v834: 25 pgs: 25 active+clean; 595 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 10 MiB/s rd, 1.1 MiB/s wr, 24 op/s 2021-07-31T01:56:53.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:52 smithi045 conmon[33655]: cluster 2021-07-31T01:56 2021-07-31T01:56:53.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:52 smithi045 conmon[33655]: :51.930259+0000 mgr.x (mgr.24542) 413 : cluster [DBG] pgmap v834: 25 pgs: 25 active+clean; 595 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 10 MiB/s rd, 1.1 MiB/s wr, 24 op/s 2021-07-31T01:56:53.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:52 smithi045 conmon[39718]: cluster 2021-07-31T01:56:51.930259+0000 2021-07-31T01:56:53.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:52 smithi045 conmon[39718]: mgr.x (mgr.24542) 413 : cluster [DBG] pgmap v834: 25 pgs: 25 active+clean; 595 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 10 MiB/s rd, 1.1 MiB/s wr, 24 op/s 2021-07-31T01:56:53.688 INFO:tasks.rados.rados.0.smithi045.stdout:3360: done (1 left) 2021-07-31T01:56:53.688 INFO:tasks.rados.rados.0.smithi045.stdout:3361: done (0 left) 2021-07-31T01:56:53.689 INFO:tasks.rados.rados.0.smithi045.stdout:3362: read oid 25 snap -1 2021-07-31T01:56:53.689 INFO:tasks.rados.rados.0.smithi045.stdout:3362: expect (ObjNum 1031 snap 335 seq_num 1031) 2021-07-31T01:56:53.689 INFO:tasks.rados.rados.0.smithi045.stdout:3363: rollback oid 21 current snap is 336 2021-07-31T01:56:53.690 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 21 to 301 2021-07-31T01:56:53.690 INFO:tasks.rados.rados.0.smithi045.stdout:3364: read oid 35 snap 293 2021-07-31T01:56:53.691 INFO:tasks.rados.rados.0.smithi045.stdout:3364: expect (ObjNum 930 snap 282 seq_num 930) 2021-07-31T01:56:53.691 INFO:tasks.rados.rados.0.smithi045.stdout:3365: snap_create 2021-07-31T01:56:53.697 INFO:tasks.rados.rados.0.smithi045.stdout:3362: expect (ObjNum 1031 snap 335 seq_num 1031) 2021-07-31T01:56:53.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:56:53 smithi045 conmon[61210]: level=error ts=2021-07-31T01:56:53.525Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:56:53.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:56:53 smithi045 conmon[61210]: level=error ts=2021-07-31T01:56:53.525Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:56:53.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:56:53 smithi045 conmon[61210]: level=error ts=2021-07-31T01:56:53.525Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T01:56:53.839 INFO:tasks.rados.rados.0.smithi045.stdout:3363: finishing rollback tid 0 to smithi04569269-21 2021-07-31T01:56:53.839 INFO:tasks.rados.rados.0.smithi045.stdout:3363: finishing rollback tid 1 to smithi04569269-21 2021-07-31T01:56:53.839 INFO:tasks.rados.rados.0.smithi045.stdout:3363: finishing rollback tid 2 to smithi04569269-21 2021-07-31T01:56:53.840 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 21 v 2193 (ObjNum 866 snap 270 seq_num 866) dirty exists 2021-07-31T01:56:53.840 INFO:tasks.rados.rados.0.smithi045.stdout:3364: expect (ObjNum 930 snap 282 seq_num 930) 2021-07-31T01:56:54.692 INFO:tasks.rados.rados.0.smithi045.stdout:3362: done (3 left) 2021-07-31T01:56:54.693 INFO:tasks.rados.rados.0.smithi045.stdout:3363: done (2 left) 2021-07-31T01:56:54.693 INFO:tasks.rados.rados.0.smithi045.stdout:3364: done (1 left) 2021-07-31T01:56:54.693 INFO:tasks.rados.rados.0.smithi045.stdout:3365: done (0 left) 2021-07-31T01:56:54.693 INFO:tasks.rados.rados.0.smithi045.stdout:3366: setattr oid 4 current snap is 337 2021-07-31T01:56:54.694 INFO:tasks.rados.rados.0.smithi045.stdout:3367: setattr oid 39 current snap is 337 2021-07-31T01:56:54.696 INFO:tasks.rados.rados.0.smithi045.stdout:3368: snap_create 2021-07-31T01:56:54.699 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 4 v 1871 (ObjNum -1105033696 snap 21873 seq_num 0) dirty exists 2021-07-31T01:56:54.701 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 39 v 2028 (ObjNum 1566727013 snap 1599956054 seq_num 1517056871) dirty exists 2021-07-31T01:56:55.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:54 smithi154 conmon[34553]: cluster 2021-07-31T01:56:53.686553+0000 mon.a (mon.0) 2021-07-31T01:56:55.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:54 smithi154 conmon[34553]: 1365 : cluster [DBG] osdmap e699: 8 total, 8 up, 8 in 2021-07-31T01:56:55.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:54 smithi154 conmon[34553]: cluster 2021-07-31T01:56:53.930764+0000 mgr.x ( 2021-07-31T01:56:55.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:54 smithi154 conmon[34553]: mgr.24542) 414 : cluster [DBG] pgmap v836: 25 pgs: 25 active+clean; 559 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 9.0 MiB/s rd, 345 KiB/s wr, 10 op/s 2021-07-31T01:56:55.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:54 smithi045 conmon[33655]: cluster 2021-07-31T01:56:53.686553 2021-07-31T01:56:55.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:54 smithi045 conmon[33655]: +0000 mon.a (mon.0) 1365 : cluster [DBG] osdmap e699: 8 total, 8 up, 8 in 2021-07-31T01:56:55.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:54 smithi045 conmon[33655]: cluster 2021-07-31T01:56:53.930764+0000 mgr.x (mgr. 2021-07-31T01:56:55.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:54 smithi045 conmon[33655]: 24542) 414 : cluster [DBG] pgmap v836: 25 pgs: 25 active+clean; 559 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 9.0 MiB/s rd, 345 KiB/s wr, 10 op/s 2021-07-31T01:56:55.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:54 smithi045 conmon[39718]: cluster 2021-07-31T01:56:53.686553+0000 2021-07-31T01:56:55.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:54 smithi045 conmon[39718]: mon.a (mon.0) 1365 : cluster [DBG] osdmap e699: 8 total, 8 up, 8 in 2021-07-31T01:56:55.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:54 smithi045 conmon[39718]: cluster 2021-07-31T01: 2021-07-31T01:56:55.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:54 smithi045 conmon[39718]: 56:53.930764+0000 mgr.x (mgr.24542) 414 : cluster [DBG] pgmap v836: 25 pgs: 25 active+clean; 559 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 9.0 MiB/s rd, 345 KiB/s wr, 10 op/s 2021-07-31T01:56:55.698 INFO:tasks.rados.rados.0.smithi045.stdout:3366: done (2 left) 2021-07-31T01:56:55.700 INFO:tasks.rados.rados.0.smithi045.stdout:3367: done (1 left) 2021-07-31T01:56:55.700 INFO:tasks.rados.rados.0.smithi045.stdout:3368: done (0 left) 2021-07-31T01:56:55.701 INFO:tasks.rados.rados.0.smithi045.stdout:3369: rollback oid 29 current snap is 338 2021-07-31T01:56:55.701 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 29 to 325 2021-07-31T01:56:55.701 INFO:tasks.rados.rados.0.smithi045.stdout:3370: rollback oid 14 current snap is 338 2021-07-31T01:56:55.702 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 14 to 289 2021-07-31T01:56:55.702 INFO:tasks.rados.rados.0.smithi045.stdout:3371: snap_remove snap 319 2021-07-31T01:56:56.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:55 smithi154 conmon[34553]: cluster 2021-07-31T01:56:54.691692+0000 mon.a ( 2021-07-31T01:56:56.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:55 smithi154 conmon[34553]: mon.0) 1366 : cluster [DBG] osdmap e700: 8 total, 8 up, 8 in 2021-07-31T01:56:56.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:55 smithi045 conmon[39718]: cluster 2021-07-31T01:56:54.691692+0000 mon.a (mon. 2021-07-31T01:56:56.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:55 smithi045 conmon[39718]: 0) 1366 : cluster [DBG] osdmap e700: 8 total, 8 up, 8 in 2021-07-31T01:56:56.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:55 smithi045 conmon[33655]: cluster 2021-07-31T01:56:54.691692+0000 mon.a (mon. 2021-07-31T01:56:56.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:55 smithi045 conmon[33655]: 0) 1366 : cluster [DBG] osdmap e700: 8 total, 8 up, 8 in 2021-07-31T01:56:56.660 INFO:tasks.rados.rados.0.smithi045.stdout:3371: done (2 left) 2021-07-31T01:56:56.660 INFO:tasks.rados.rados.0.smithi045.stdout:3372: rmattr oid 43 current snap is 338 2021-07-31T01:56:56.661 INFO:tasks.rados.rados.0.smithi045.stdout:3370: finishing rollback tid 1 to smithi04569269-14 2021-07-31T01:56:56.661 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 14 v 1643 (ObjNum -1105824320 snap 21873 seq_num 3190169296) dirty dne 2021-07-31T01:56:56.661 INFO:tasks.rados.rados.0.smithi045.stdout:3369: finishing rollback tid 0 to smithi04569269-29 2021-07-31T01:56:56.661 INFO:tasks.rados.rados.0.smithi045.stdout:3369: finishing rollback tid 1 to smithi04569269-29 2021-07-31T01:56:56.661 INFO:tasks.rados.rados.0.smithi045.stdout:3369: finishing rollback tid 2 to smithi04569269-29 2021-07-31T01:56:56.662 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 29 v 1425 (ObjNum 0 snap 32731 seq_num 3189378992) dirty exists 2021-07-31T01:56:56.663 INFO:tasks.rados.rados.0.smithi045.stdout:3369: done (2 left) 2021-07-31T01:56:56.663 INFO:tasks.rados.rados.0.smithi045.stdout:3370: done (1 left) 2021-07-31T01:56:56.663 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 4 current snap is 338 2021-07-31T01:56:56.663 INFO:tasks.rados.rados.0.smithi045.stdout:3373: seq_num 1036 ranges {0=557056} 2021-07-31T01:56:56.664 INFO:tasks.rados.rados.0.smithi045.stdout:3373: writing smithi04569269-4 from 0 to 557056 tid 1 2021-07-31T01:56:56.664 INFO:tasks.rados.rados.0.smithi045.stdout:3374: read oid 23 snap -1 2021-07-31T01:56:56.664 INFO:tasks.rados.rados.0.smithi045.stdout:3374: expect (ObjNum 1009 snap 323 seq_num 1009) 2021-07-31T01:56:56.664 INFO:tasks.rados.rados.0.smithi045.stdout:3375: copy_from oid 11 from oid 36 current snap is 338 2021-07-31T01:56:56.665 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 38 current snap is 338 2021-07-31T01:56:56.665 INFO:tasks.rados.rados.0.smithi045.stdout:3376: seq_num 1037 ranges {0=573440} 2021-07-31T01:56:56.666 INFO:tasks.rados.rados.0.smithi045.stdout:3376: writing smithi04569269-38 from 0 to 573440 tid 1 2021-07-31T01:56:56.666 INFO:tasks.rados.rados.0.smithi045.stdout:3377: copy_from oid 16 from oid 25 current snap is 338 2021-07-31T01:56:56.666 INFO:tasks.rados.rados.0.smithi045.stdout:3378: read oid 28 snap -1 2021-07-31T01:56:56.667 INFO:tasks.rados.rados.0.smithi045.stdout:3378: expect (ObjNum 1009 snap 323 seq_num 1009) 2021-07-31T01:56:56.667 INFO:tasks.rados.rados.0.smithi045.stdout:3379: setattr oid 35 current snap is 338 2021-07-31T01:56:56.667 INFO:tasks.rados.rados.0.smithi045.stdout:3380: delete oid 7 current snap is 338 2021-07-31T01:56:56.668 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 43 v 1866 (ObjNum 1032 snap 335 seq_num 1032) dirty exists 2021-07-31T01:56:56.672 INFO:tasks.rados.rados.0.smithi045.stdout:3373: finishing write tid 1 to smithi04569269-4 2021-07-31T01:56:56.672 INFO:tasks.rados.rados.0.smithi045.stdout:3373: finishing write tid 2 to smithi04569269-4 2021-07-31T01:56:56.673 INFO:tasks.rados.rados.0.smithi045.stdout:3372: done (8 left) 2021-07-31T01:56:56.674 INFO:tasks.rados.rados.0.smithi045.stdout:3380: done (7 left) 2021-07-31T01:56:56.674 INFO:tasks.rados.rados.0.smithi045.stdout:3381: read oid 43 snap -1 2021-07-31T01:56:56.674 INFO:tasks.rados.rados.0.smithi045.stdout:3381: expect (ObjNum 1032 snap 335 seq_num 1032) 2021-07-31T01:56:56.674 INFO:tasks.rados.rados.0.smithi045.stdout:3382: snap_remove snap 334 2021-07-31T01:56:57.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:56 smithi154 conmon[34553]: cluster 2021-07-31T01:56:55.692986+0000 mon.a (mon.0) 1367 : cluster [DBG] osdmap e701: 8 total, 8 up, 8 in 2021-07-31T01:56:57.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:56 smithi154 conmon[34553]: 2021-07-31T01:56:57.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:56 smithi154 conmon[34553]: cluster 2021-07-31T01:56:55.931224+0000 mgr.x (mgr.24542) 415 : cluster [DBG] pgmap v839: 25 pgs: 25 active+clean; 559 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 9.0 MiB/s rd, 344 KiB/s wr, 10 op/s 2021-07-31T01:56:57.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:56 smithi154 conmon[34553]: cluster 2021-07-31T01:56:56.655004+0000 mon.a (mon.0) 1368 : cluster [DBG] osdmap e702: 8 total, 8 up, 8 in 2021-07-31T01:56:57.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:56 smithi045 conmon[39718]: cluster 2021-07 2021-07-31T01:56:57.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:56 smithi045 conmon[39718]: -31T01:56:55.692986+0000 mon.a (mon.0) 1367 : cluster [DBG] osdmap e701: 8 total, 8 up, 8 in 2021-07-31T01:56:57.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:56 smithi045 conmon[39718]: cluster 2021-07-31T01:56:57.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:56 smithi045 conmon[39718]: 2021 2021-07-31T01:56:57.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:56 smithi045 conmon[39718]: -07-31T01:56 2021-07-31T01:56:57.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:56 smithi045 conmon[39718]: :55.931224+0000 2021-07-31T01:56:57.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:56 smithi045 conmon[39718]: mgr.x 2021-07-31T01:56:57.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:56 smithi045 conmon[39718]: (mgr.24542) 415 : cluster 2021-07-31T01:56:57.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:56 smithi045 conmon[39718]: [DBG] pgmap v839: 25 pgs: 25 active+clean; 559 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 9.0 MiB/s rd, 344 KiB/s wr, 10 op/s 2021-07-31T01:56:57.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:56 smithi045 conmon[39718]: cluster 2021- 2021-07-31T01:56:57.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:56 smithi045 conmon[39718]: 07-31T01:56 2021-07-31T01:56:57.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:56 smithi045 conmon[39718]: :56.655004+0000 mon.a ( 2021-07-31T01:56:57.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:56 smithi045 conmon[39718]: mon.0) 1368 : cluster 2021-07-31T01:56:57.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:56 smithi045 conmon[39718]: [DBG] osdmap e702: 8 total, 8 up, 8 in 2021-07-31T01:56:57.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:56 smithi045 conmon[33655]: cluster 2021-07-31T01:56:55.692986+0000 mon.a (mon 2021-07-31T01:56:57.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:56 smithi045 conmon[33655]: .0) 1367 : cluster [DBG] osdmap e701: 8 total, 8 up, 8 in 2021-07-31T01:56:57.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:56 smithi045 conmon[33655]: cluster 2021-07-31T01:56:55.931224+0000 mgr.x (mgr.24542) 415 : cluster [DBG] pgmap v839: 25 pgs: 25 active+clean; 559 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 9.0 MiB/s rd, 344 KiB/s wr, 10 op/s 2021-07-31T01:56:57.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:56 smithi045 conmon[33655]: cluster 2021 2021-07-31T01:56:57.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:56 smithi045 conmon[33655]: -07-31T01:56:56.655004+0000 mon.a (mon.0) 1368 : cluster [DBG] osdmap e702: 8 total, 8 up, 8 in 2021-07-31T01:56:57.663 INFO:tasks.rados.rados.0.smithi045.stdout:3382: done (8 left) 2021-07-31T01:56:57.664 INFO:tasks.rados.rados.0.smithi045.stdout:3383: rollback oid 48 current snap is 338 2021-07-31T01:56:57.664 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 48 to 330 2021-07-31T01:56:57.664 INFO:tasks.rados.rados.0.smithi045.stdout:3384: delete oid 18 current snap is 338 2021-07-31T01:56:57.665 INFO:tasks.rados.rados.0.smithi045.stdout:3373: finishing write tid 3 to smithi04569269-4 2021-07-31T01:56:57.666 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 4 v 1873 (ObjNum 1036 snap 338 seq_num 1036) dirty exists 2021-07-31T01:56:57.666 INFO:tasks.rados.rados.0.smithi045.stdout:3373: left oid 4 (ObjNum 1036 snap 338 seq_num 1036) 2021-07-31T01:56:57.666 INFO:tasks.rados.rados.0.smithi045.stdout:3378: expect (ObjNum 1009 snap 323 seq_num 1009) 2021-07-31T01:56:57.710 INFO:tasks.rados.rados.0.smithi045.stdout:3374: expect (ObjNum 1009 snap 323 seq_num 1009) 2021-07-31T01:56:57.754 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 35 v 2199 (ObjNum 1026 snap 329 seq_num 1026) dirty exists 2021-07-31T01:56:57.755 INFO:tasks.rados.rados.0.smithi045.stdout:3376: finishing write tid 1 to smithi04569269-38 2021-07-31T01:56:57.756 INFO:tasks.rados.rados.0.smithi045.stdout:3376: finishing write tid 2 to smithi04569269-38 2021-07-31T01:56:57.758 INFO:tasks.rados.rados.0.smithi045.stdout:3377: finishing copy_from to smithi04569269-16 2021-07-31T01:56:57.758 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 16 v 1841 (ObjNum 1031 snap 335 seq_num 1031) dirty exists 2021-07-31T01:56:57.759 INFO:tasks.rados.rados.0.smithi045.stdout:3377: finishing copy_from racing read to smithi04569269-16 2021-07-31T01:56:57.760 INFO:tasks.rados.rados.0.smithi045.stdout:3376: finishing write tid 3 to smithi04569269-38 2021-07-31T01:56:57.761 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 38 v 2030 (ObjNum 1037 snap 338 seq_num 1037) dirty exists 2021-07-31T01:56:57.761 INFO:tasks.rados.rados.0.smithi045.stdout:3376: left oid 38 (ObjNum 1037 snap 338 seq_num 1037) 2021-07-31T01:56:57.762 INFO:tasks.rados.rados.0.smithi045.stdout:3375: finishing copy_from to smithi04569269-11 2021-07-31T01:56:57.762 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 11 v 1644 (ObjNum 917 snap 278 seq_num 917) dirty exists 2021-07-31T01:56:57.763 INFO:tasks.rados.rados.0.smithi045.stdout:3375: finishing copy_from racing read to smithi04569269-11 2021-07-31T01:56:57.764 INFO:tasks.rados.rados.0.smithi045.stdout:3381: expect (ObjNum 1032 snap 335 seq_num 1032) 2021-07-31T01:56:59.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:58 smithi154 conmon[34553]: cluster 2021-07-31T01:56:57.662010+0000 mon.a (mon.0) 1369 : cluster 2021-07-31T01:56:59.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:58 smithi154 conmon[34553]: [DBG] osdmap e703: 8 total, 8 up, 8 in 2021-07-31T01:56:59.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:56:58 smithi154 conmon[34553]: cluster 2021-07-31T01:56:57.931644+0000 mgr.x (mgr.24542) 416 : cluster [DBG] pgmap v842: 25 pgs: 25 active+clean; 559 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:56:59.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:58 smithi045 conmon[33655]: cluster 2021-07-31T01:56:57.662010+0000 mon.a 2021-07-31T01:56:59.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:58 smithi045 conmon[33655]: (mon.0) 1369 : cluster [DBG] osdmap e703: 8 total, 8 up, 8 in 2021-07-31T01:56:59.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:58 smithi045 conmon[33655]: cluster 2021-07-31T01:56:57.931644+0000 mgr.x 2021-07-31T01:56:59.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:56:58 smithi045 conmon[33655]: (mgr.24542) 416 : cluster [DBG] pgmap v842: 25 pgs: 25 active+clean; 559 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:56:59.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:58 smithi045 conmon[39718]: cluster 2021-07-31T01:56:57.662010 2021-07-31T01:56:59.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:58 smithi045 conmon[39718]: +0000 mon.a (mon.0) 1369 : cluster [DBG] osdmap e703: 8 total, 8 up, 8 in 2021-07-31T01:56:59.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:58 smithi045 conmon[39718]: cluster 2021-07-31 2021-07-31T01:56:59.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:58 smithi045 conmon[39718]: T01:56:57.931644+0000 mgr.x (mgr.24542) 416 : cluster [DBG] 2021-07-31T01:56:59.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:56:58 smithi045 conmon[39718]: pgmap v842: 25 pgs: 25 active+clean; 559 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:56:59.123 INFO:tasks.rados.rados.0.smithi045.stdout:3383: finishing rollback tid 1 to smithi04569269-48 2021-07-31T01:56:59.123 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 48 v 1853 (ObjNum 1 snap 32731 seq_num 2605350152) dirty dne 2021-07-31T01:56:59.124 INFO:tasks.rados.rados.0.smithi045.stdout:3373: done (9 left) 2021-07-31T01:56:59.124 INFO:tasks.rados.rados.0.smithi045.stdout:3374: done (8 left) 2021-07-31T01:56:59.124 INFO:tasks.rados.rados.0.smithi045.stdout:3375: done (7 left) 2021-07-31T01:56:59.125 INFO:tasks.rados.rados.0.smithi045.stdout:3376: done (6 left) 2021-07-31T01:56:59.125 INFO:tasks.rados.rados.0.smithi045.stdout:3377: done (5 left) 2021-07-31T01:56:59.125 INFO:tasks.rados.rados.0.smithi045.stdout:3378: done (4 left) 2021-07-31T01:56:59.125 INFO:tasks.rados.rados.0.smithi045.stdout:3379: done (3 left) 2021-07-31T01:56:59.126 INFO:tasks.rados.rados.0.smithi045.stdout:3381: done (2 left) 2021-07-31T01:56:59.126 INFO:tasks.rados.rados.0.smithi045.stdout:3383: done (1 left) 2021-07-31T01:56:59.126 INFO:tasks.rados.rados.0.smithi045.stdout:3384: done (0 left) 2021-07-31T01:56:59.127 INFO:tasks.rados.rados.0.smithi045.stdout:3385: delete oid 28 current snap is 338 2021-07-31T01:56:59.127 INFO:tasks.rados.rados.0.smithi045.stdout:3385: done (0 left) 2021-07-31T01:56:59.128 INFO:tasks.rados.rados.0.smithi045.stdout:3386: rmattr oid 44 current snap is 338 2021-07-31T01:56:59.128 INFO:tasks.rados.rados.0.smithi045.stdout:3386: done (0 left) 2021-07-31T01:56:59.128 INFO:tasks.rados.rados.0.smithi045.stdout:3387: read oid 7 snap -1 2021-07-31T01:56:59.128 INFO:tasks.rados.rados.0.smithi045.stdout:3387: expect deleted 2021-07-31T01:56:59.129 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 46 current snap is 338 2021-07-31T01:56:59.129 INFO:tasks.rados.rados.0.smithi045.stdout:3388: seq_num 1041 ranges {5767168=638976} 2021-07-31T01:56:59.161 INFO:tasks.rados.rados.0.smithi045.stdout:3388: writing smithi04569269-46 from 5767168 to 6406144 tid 1 2021-07-31T01:56:59.161 INFO:tasks.rados.rados.0.smithi045.stdout:3389: copy_from oid 36 from oid 4 current snap is 338 2021-07-31T01:56:59.162 INFO:tasks.rados.rados.0.smithi045.stdout:3387: done (2 left) 2021-07-31T01:56:59.162 INFO:tasks.rados.rados.0.smithi045.stdout:3390: read oid 12 snap -1 2021-07-31T01:56:59.163 INFO:tasks.rados.rados.0.smithi045.stdout:3390: expect (ObjNum 967 snap 295 seq_num 967) 2021-07-31T01:56:59.163 INFO:tasks.rados.rados.0.smithi045.stdout:3391: read oid 8 snap -1 2021-07-31T01:56:59.163 INFO:tasks.rados.rados.0.smithi045.stdout:3391: expect (ObjNum 994 snap 309 seq_num 994) 2021-07-31T01:56:59.164 INFO:tasks.rados.rados.0.smithi045.stdout:3392: copy_from oid 9 from oid 14 current snap is 338 2021-07-31T01:56:59.164 INFO:tasks.rados.rados.0.smithi045.stdout:3393: delete oid 30 current snap is 338 2021-07-31T01:56:59.165 INFO:tasks.rados.rados.0.smithi045.stdout:3392: finishing copy_from to smithi04569269-9 2021-07-31T01:56:59.165 INFO:tasks.rados.rados.0.smithi045.stdout:3392: got expected ENOENT (src dne) 2021-07-31T01:56:59.165 INFO:tasks.rados.rados.0.smithi045.stdout:3392: finishing copy_from racing read to smithi04569269-9 2021-07-31T01:56:59.165 INFO:tasks.rados.rados.0.smithi045.stdout:3392: done (5 left) 2021-07-31T01:56:59.166 INFO:tasks.rados.rados.0.smithi045.stdout:3393: done (4 left) 2021-07-31T01:56:59.166 INFO:tasks.rados.rados.0.smithi045.stdout:3394: snap_create 2021-07-31T01:56:59.168 INFO:tasks.rados.rados.0.smithi045.stdout:3388: finishing write tid 1 to smithi04569269-46 2021-07-31T01:56:59.168 INFO:tasks.rados.rados.0.smithi045.stdout:3388: finishing write tid 2 to smithi04569269-46 2021-07-31T01:56:59.169 INFO:tasks.rados.rados.0.smithi045.stdout:3388: finishing write tid 3 to smithi04569269-46 2021-07-31T01:56:59.170 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 46 v 1850 (ObjNum 1041 snap 338 seq_num 1041) dirty exists 2021-07-31T01:56:59.170 INFO:tasks.rados.rados.0.smithi045.stdout:3388: left oid 46 (ObjNum 1041 snap 338 seq_num 1041) 2021-07-31T01:56:59.176 INFO:tasks.rados.rados.0.smithi045.stdout:3391: expect (ObjNum 994 snap 309 seq_num 994) 2021-07-31T01:56:59.311 INFO:tasks.rados.rados.0.smithi045.stdout:3389: finishing copy_from to smithi04569269-36 2021-07-31T01:56:59.311 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 36 v 1873 (ObjNum 1036 snap 338 seq_num 1036) dirty exists 2021-07-31T01:56:59.311 INFO:tasks.rados.rados.0.smithi045.stdout:3389: finishing copy_from racing read to smithi04569269-36 2021-07-31T01:56:59.311 INFO:tasks.rados.rados.0.smithi045.stdout:3390: expect (ObjNum 967 snap 295 seq_num 967) 2021-07-31T01:56:59.668 INFO:tasks.rados.rados.0.smithi045.stdout:3388: done (4 left) 2021-07-31T01:56:59.668 INFO:tasks.rados.rados.0.smithi045.stdout:3389: done (3 left) 2021-07-31T01:56:59.668 INFO:tasks.rados.rados.0.smithi045.stdout:3390: done (2 left) 2021-07-31T01:56:59.669 INFO:tasks.rados.rados.0.smithi045.stdout:3391: done (1 left) 2021-07-31T01:56:59.669 INFO:tasks.rados.rados.0.smithi045.stdout:3394: done (0 left) 2021-07-31T01:56:59.669 INFO:tasks.rados.rados.0.smithi045.stdout:3395: read oid 47 snap 337 2021-07-31T01:56:59.669 INFO:tasks.rados.rados.0.smithi045.stdout:3395: expect (ObjNum -1104838976 snap 21873 seq_num 3189123776) 2021-07-31T01:56:59.669 INFO:tasks.rados.rados.0.smithi045.stdout:3396: copy_from oid 43 from oid 38 current snap is 339 2021-07-31T01:56:59.670 INFO:tasks.rados.rados.0.smithi045.stdout:3397: read oid 2 snap -1 2021-07-31T01:56:59.670 INFO:tasks.rados.rados.0.smithi045.stdout:3397: expect deleted 2021-07-31T01:56:59.670 INFO:tasks.rados.rados.0.smithi045.stdout:3398: read oid 10 snap 318 2021-07-31T01:56:59.670 INFO:tasks.rados.rados.0.smithi045.stdout:3398: expect (ObjNum 768 snap 235 seq_num 768) 2021-07-31T01:56:59.671 INFO:tasks.rados.rados.0.smithi045.stdout:3399: delete oid 14 current snap is 339 2021-07-31T01:56:59.671 INFO:tasks.rados.rados.0.smithi045.stdout:3395: expect (ObjNum -1104838976 snap 21873 seq_num 3189123776) 2021-07-31T01:56:59.673 INFO:tasks.rados.rados.0.smithi045.stdout:3395: done (4 left) 2021-07-31T01:56:59.673 INFO:tasks.rados.rados.0.smithi045.stdout:3397: done (3 left) 2021-07-31T01:56:59.673 INFO:tasks.rados.rados.0.smithi045.stdout:3399: done (2 left) 2021-07-31T01:56:59.673 INFO:tasks.rados.rados.0.smithi045.stdout:3400: snap_create 2021-07-31T01:56:59.680 INFO:tasks.rados.rados.0.smithi045.stdout:3396: finishing copy_from to smithi04569269-43 2021-07-31T01:56:59.681 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 43 v 2030 (ObjNum 1037 snap 338 seq_num 1037) dirty exists 2021-07-31T01:56:59.681 INFO:tasks.rados.rados.0.smithi045.stdout:3396: finishing copy_from racing read to smithi04569269-43 2021-07-31T01:56:59.713 INFO:tasks.rados.rados.0.smithi045.stdout:3398: expect (ObjNum 768 snap 235 seq_num 768) 2021-07-31T01:57:00.679 INFO:tasks.rados.rados.0.smithi045.stdout:3396: done (2 left) 2021-07-31T01:57:00.679 INFO:tasks.rados.rados.0.smithi045.stdout:3398: done (1 left) 2021-07-31T01:57:00.679 INFO:tasks.rados.rados.0.smithi045.stdout:3400: done (0 left) 2021-07-31T01:57:00.680 INFO:tasks.rados.rados.0.smithi045.stdout:3401: read oid 4 snap -1 2021-07-31T01:57:00.680 INFO:tasks.rados.rados.0.smithi045.stdout:3401: expect (ObjNum 1036 snap 338 seq_num 1036) 2021-07-31T01:57:00.680 INFO:tasks.rados.rados.0.smithi045.stdout:3402: rmattr oid 46 current snap is 340 2021-07-31T01:57:00.680 INFO:tasks.rados.rados.0.smithi045.stdout:3402: done (1 left) 2021-07-31T01:57:00.685 INFO:tasks.rados.rados.0.smithi045.stdout:3403: delete oid 37 current snap is 340 2021-07-31T01:57:00.686 INFO:tasks.rados.rados.0.smithi045.stdout:3401: expect (ObjNum 1036 snap 338 seq_num 1036) 2021-07-31T01:57:00.692 INFO:tasks.rados.rados.0.smithi045.stdout:3401: done (1 left) 2021-07-31T01:57:00.693 INFO:tasks.rados.rados.0.smithi045.stdout:3403: done (0 left) 2021-07-31T01:57:00.693 INFO:tasks.rados.rados.0.smithi045.stdout:3404: rollback oid 10 current snap is 340 2021-07-31T01:57:00.693 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 10 to 337 2021-07-31T01:57:00.694 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 1 current snap is 340 2021-07-31T01:57:00.694 INFO:tasks.rados.rados.0.smithi045.stdout:3405: seq_num 1045 ranges {3792896=704512} 2021-07-31T01:57:00.717 INFO:tasks.rados.rados.0.smithi045.stdout:3405: writing smithi04569269-1 from 3792896 to 4497408 tid 1 2021-07-31T01:57:00.718 INFO:tasks.rados.rados.0.smithi045.stdout:3406: rmattr oid 15 current snap is 340 2021-07-31T01:57:00.718 INFO:tasks.rados.rados.0.smithi045.stdout:3406: done (2 left) 2021-07-31T01:57:00.718 INFO:tasks.rados.rados.0.smithi045.stdout:3407: copy_from oid 35 from oid 3 current snap is 340 2021-07-31T01:57:00.718 INFO:tasks.rados.rados.0.smithi045.stdout:3404: finishing rollback tid 0 to smithi04569269-10 2021-07-31T01:57:00.719 INFO:tasks.rados.rados.0.smithi045.stdout:3404: finishing rollback tid 1 to smithi04569269-10 2021-07-31T01:57:00.719 INFO:tasks.rados.rados.0.smithi045.stdout:3404: finishing rollback tid 2 to smithi04569269-10 2021-07-31T01:57:00.719 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 10 v 2025 (ObjNum 754 snap 233 seq_num 754) dirty exists 2021-07-31T01:57:00.719 INFO:tasks.rados.rados.0.smithi045.stdout:3404: done (2 left) 2021-07-31T01:57:00.720 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 20 current snap is 340 2021-07-31T01:57:00.720 INFO:tasks.rados.rados.0.smithi045.stdout:3408: seq_num 1046 ranges {0=729088} 2021-07-31T01:57:00.722 INFO:tasks.rados.rados.0.smithi045.stdout:3408: writing smithi04569269-20 from 0 to 729088 tid 1 2021-07-31T01:57:00.722 INFO:tasks.rados.rados.0.smithi045.stdout:3409: read oid 30 snap -1 2021-07-31T01:57:00.722 INFO:tasks.rados.rados.0.smithi045.stdout:3409: expect deleted 2021-07-31T01:57:00.723 INFO:tasks.rados.rados.0.smithi045.stdout:3410: rmattr oid 16 current snap is 340 2021-07-31T01:57:00.723 INFO:tasks.rados.rados.0.smithi045.stdout:3410: done (4 left) 2021-07-31T01:57:00.723 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 42 current snap is 340 2021-07-31T01:57:00.723 INFO:tasks.rados.rados.0.smithi045.stdout:3411: seq_num 1047 ranges {0=745472} 2021-07-31T01:57:00.726 INFO:tasks.rados.rados.0.smithi045.stdout:3411: writing smithi04569269-42 from 0 to 745472 tid 1 2021-07-31T01:57:00.727 INFO:tasks.rados.rados.0.smithi045.stdout:3412: rmattr oid 11 current snap is 340 2021-07-31T01:57:00.727 INFO:tasks.rados.rados.0.smithi045.stdout:3412: done (5 left) 2021-07-31T01:57:00.727 INFO:tasks.rados.rados.0.smithi045.stdout:3413: snap_create 2021-07-31T01:57:00.728 INFO:tasks.rados.rados.0.smithi045.stdout:3405: finishing write tid 1 to smithi04569269-1 2021-07-31T01:57:00.729 INFO:tasks.rados.rados.0.smithi045.stdout:3405: finishing write tid 2 to smithi04569269-1 2021-07-31T01:57:00.729 INFO:tasks.rados.rados.0.smithi045.stdout:3405: finishing write tid 3 to smithi04569269-1 2021-07-31T01:57:00.729 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 1 v 1858 (ObjNum 1045 snap 340 seq_num 1045) dirty exists 2021-07-31T01:57:00.729 INFO:tasks.rados.rados.0.smithi045.stdout:3405: left oid 1 (ObjNum 1045 snap 340 seq_num 1045) 2021-07-31T01:57:00.731 INFO:tasks.rados.rados.0.smithi045.stdout:3408: finishing write tid 1 to smithi04569269-20 2021-07-31T01:57:00.731 INFO:tasks.rados.rados.0.smithi045.stdout:3408: finishing write tid 2 to smithi04569269-20 2021-07-31T01:57:00.732 INFO:tasks.rados.rados.0.smithi045.stdout:3411: finishing write tid 1 to smithi04569269-42 2021-07-31T01:57:00.734 INFO:tasks.rados.rados.0.smithi045.stdout:3411: finishing write tid 2 to smithi04569269-42 2021-07-31T01:57:00.735 INFO:tasks.rados.rados.0.smithi045.stdout:3411: finishing write tid 3 to smithi04569269-42 2021-07-31T01:57:00.735 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 42 v 2027 (ObjNum 1047 snap 340 seq_num 1047) dirty exists 2021-07-31T01:57:00.736 INFO:tasks.rados.rados.0.smithi045.stdout:3411: left oid 42 (ObjNum 1047 snap 340 seq_num 1047) 2021-07-31T01:57:00.746 INFO:tasks.rados.rados.0.smithi045.stdout:3408: finishing write tid 3 to smithi04569269-20 2021-07-31T01:57:00.746 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 20 v 2213 (ObjNum 1046 snap 340 seq_num 1046) dirty exists 2021-07-31T01:57:00.747 INFO:tasks.rados.rados.0.smithi045.stdout:3408: left oid 20 (ObjNum 1046 snap 340 seq_num 1046) 2021-07-31T01:57:00.756 INFO:tasks.rados.rados.0.smithi045.stdout:3407: finishing copy_from to smithi04569269-35 2021-07-31T01:57:00.756 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 35 v 2163 (ObjNum 881 snap 272 seq_num 881) dirty exists 2021-07-31T01:57:00.756 INFO:tasks.rados.rados.0.smithi045.stdout:3407: finishing copy_from racing read to smithi04569269-35 2021-07-31T01:57:01.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:00 smithi154 conmon[34553]: cluster 2021-07-31T01:56:59.666936+0000 mon.a (mon.0) 1370 : cluster [DBG] osdmap e704: 8 total, 8 up, 8 in 2021-07-31T01:57:01.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:00 smithi154 conmon[34553]: cluster 2021-07-31T01:56:59.932148+0000 mgr.x (mgr.24542) 417 : cluster [DBG] pgmap v844: 25 pgs: 25 active+clean; 564 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 8.5 MiB/s rd, 1.1 MiB/s wr, 26 op/s 2021-07-31T01:57:01.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:00 smithi045 conmon[33655]: cluster 2021-07-31T01:56:59.666936+0000 mon.a (mon.0) 1370 : cluster 2021-07-31T01:57:01.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:00 smithi045 conmon[33655]: [DBG] osdmap e704: 8 total, 8 up, 8 in 2021-07-31T01:57:01.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:00 smithi045 conmon[33655]: cluster 2021-07-31T01:56:59.932148+0000 mgr.x (mgr.24542) 417 : cluster [DBG] pgmap v844: 25 pgs: 25 active+clean; 564 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 8.5 MiB/s rd, 1.1 MiB/s wr, 26 op/s 2021-07-31T01:57:01.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:00 smithi045 conmon[39718]: cluster 2021-07-31T01:56:59.666936+0000 mon.a (mon.0) 2021-07-31T01:57:01.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:00 smithi045 conmon[39718]: 1370 : cluster [DBG] osdmap e704: 8 total, 8 up, 8 in 2021-07-31T01:57:01.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:00 smithi045 conmon[39718]: cluster 2021-07-31T01:56:59.932148+0000 mgr.x (mgr.24542) 417 : cluster [DBG] pgmap v844: 25 pgs: 25 active+clean; 564 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 8.5 MiB/s rd, 1.1 MiB/s wr, 26 op/s 2021-07-31T01:57:01.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:00 smithi045 conmon[39718]: 2021-07-31T01:57:01.658 INFO:tasks.rados.rados.0.smithi045.stdout:3405: done (5 left) 2021-07-31T01:57:01.659 INFO:tasks.rados.rados.0.smithi045.stdout:3407: done (4 left) 2021-07-31T01:57:01.659 INFO:tasks.rados.rados.0.smithi045.stdout:3408: done (3 left) 2021-07-31T01:57:01.660 INFO:tasks.rados.rados.0.smithi045.stdout:3409: done (2 left) 2021-07-31T01:57:01.660 INFO:tasks.rados.rados.0.smithi045.stdout:3411: done (1 left) 2021-07-31T01:57:01.660 INFO:tasks.rados.rados.0.smithi045.stdout:3413: done (0 left) 2021-07-31T01:57:01.660 INFO:tasks.rados.rados.0.smithi045.stdout:3414: rmattr oid 16 current snap is 341 2021-07-31T01:57:01.660 INFO:tasks.rados.rados.0.smithi045.stdout:3414: done (0 left) 2021-07-31T01:57:01.661 INFO:tasks.rados.rados.0.smithi045.stdout:3415: read oid 23 snap 332 2021-07-31T01:57:01.661 INFO:tasks.rados.rados.0.smithi045.stdout:3415: expect (ObjNum 1009 snap 323 seq_num 1009) 2021-07-31T01:57:01.661 INFO:tasks.rados.rados.0.smithi045.stdout:3416: delete oid 47 current snap is 341 2021-07-31T01:57:01.668 INFO:tasks.rados.rados.0.smithi045.stdout:3415: expect (ObjNum 1009 snap 323 seq_num 1009) 2021-07-31T01:57:01.715 INFO:tasks.rados.rados.0.smithi045.stdout:3415: done (1 left) 2021-07-31T01:57:01.715 INFO:tasks.rados.rados.0.smithi045.stdout:3416: done (0 left) 2021-07-31T01:57:01.715 INFO:tasks.rados.rados.0.smithi045.stdout:3417: setattr oid 8 current snap is 341 2021-07-31T01:57:01.716 INFO:tasks.rados.rados.0.smithi045.stdout:3418: rollback oid 26 current snap is 341 2021-07-31T01:57:01.716 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 26 to 318 2021-07-31T01:57:01.716 INFO:tasks.rados.rados.0.smithi045.stdout:3419: snap_create 2021-07-31T01:57:01.720 INFO:tasks.rados.rados.0.smithi045.stdout:3418: finishing rollback tid 1 to smithi04569269-26 2021-07-31T01:57:01.721 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 26 v 1852 (ObjNum -1229693936 snap 32731 seq_num 3189932240) dirty dne 2021-07-31T01:57:01.721 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 8 v 2217 (ObjNum 994 snap 309 seq_num 994) dirty exists 2021-07-31T01:57:02.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:01 smithi154 conmon[34553]: cluster 2021-07-31T01:57:00.672383+0000 mon.a (mon.0) 1371 : cluster [DBG] osdmap e705: 8 total, 8 up, 8 in 2021-07-31T01:57:02.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:01 smithi154 conmon[34553]: cluster 2021-07-31T 2021-07-31T01:57:02.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:01 smithi154 conmon[34553]: 01:57:01.657462+0000 mon.a (mon.0) 1372 : cluster [DBG] osdmap e706: 8 total, 8 up, 8 in 2021-07-31T01:57:02.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:01 smithi045 conmon[33655]: cluster 2021-07-31T01:57:00.672383+0000 mon.a (mon.0) 1371 : cluster [DBG] osdmap e705: 8 total, 8 up, 8 in 2021-07-31T01:57:02.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:01 smithi045 conmon[33655]: cluster 2021-07-31T01:57:01.657462+0000 mon.a (mon.0) 1372 : cluster [DBG] osdmap e706: 8 total, 8 up, 8 in 2021-07-31T01:57:02.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:01 smithi045 conmon[39718]: cluster 2021-07-31T01:57:00.672383+0000 mon.a (mon 2021-07-31T01:57:02.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:01 smithi045 conmon[39718]: .0) 1371 : cluster [DBG] osdmap e705: 8 total, 8 up, 8 in 2021-07-31T01:57:02.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:01 smithi045 conmon[39718]: cluster 2021-07-31T01:57:01.657462+0000 2021-07-31T01:57:02.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:01 smithi045 conmon[39718]: mon.a (mon.0) 1372 : cluster [DBG] osdmap e706: 8 total, 8 up, 8 in 2021-07-31T01:57:02.694 INFO:tasks.rados.rados.0.smithi045.stdout:3417: done (2 left) 2021-07-31T01:57:02.694 INFO:tasks.rados.rados.0.smithi045.stdout:3418: done (1 left) 2021-07-31T01:57:02.695 INFO:tasks.rados.rados.0.smithi045.stdout:3419: done (0 left) 2021-07-31T01:57:02.695 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 23 current snap is 342 2021-07-31T01:57:02.695 INFO:tasks.rados.rados.0.smithi045.stdout:3420: seq_num 1049 ranges {1073152=778240} 2021-07-31T01:57:02.705 INFO:tasks.rados.rados.0.smithi045.stdout:3420: writing smithi04569269-23 from 1073152 to 1851392 tid 1 2021-07-31T01:57:02.706 INFO:tasks.rados.rados.0.smithi045.stdout:3421: delete oid 13 current snap is 342 2021-07-31T01:57:02.712 INFO:tasks.rados.rados.0.smithi045.stdout:3421: done (1 left) 2021-07-31T01:57:02.712 INFO:tasks.rados.rados.0.smithi045.stdout:3422: copy_from oid 14 from oid 37 current snap is 342 2021-07-31T01:57:02.712 INFO:tasks.rados.rados.0.smithi045.stdout:3423: snap_remove snap 299 2021-07-31T01:57:03.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:02 smithi154 conmon[34553]: cluster 2021-07-31T01:57:01.932680+0000 mgr.x (mgr.24542) 418 : 2021-07-31T01:57:03.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:02 smithi154 conmon[34553]: cluster [DBG] pgmap v847: 25 pgs: 25 active+clean; 564 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 8.5 MiB/s rd, 1.1 MiB/s wr, 26 op/s 2021-07-31T01:57:03.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:02 smithi045 conmon[33655]: cluster 2021-07-31T01:57:01.932680+0000 mgr.x (mgr.24542) 418 : cluster 2021-07-31T01:57:03.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:02 smithi045 conmon[33655]: [DBG] pgmap v847: 25 pgs: 25 active+clean; 564 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 8.5 MiB/s rd, 1.1 MiB/s wr, 26 op/s 2021-07-31T01:57:03.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:02 smithi045 conmon[39718]: cluster 2021-07-31T01:57:01.932680+0000 mgr.x (mgr. 2021-07-31T01:57:03.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:02 smithi045 conmon[39718]: 24542) 418 : cluster [DBG] pgmap v847: 25 pgs: 25 active+clean; 564 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 8.5 MiB/s rd, 1.1 MiB/s wr, 26 op/s 2021-07-31T01:57:03.701 INFO:tasks.rados.rados.0.smithi045.stdout:3423: done (2 left) 2021-07-31T01:57:03.701 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 19 current snap is 342 2021-07-31T01:57:03.702 INFO:tasks.rados.rados.0.smithi045.stdout:3420: finishing write tid 1 to smithi04569269-23 2021-07-31T01:57:03.702 INFO:tasks.rados.rados.0.smithi045.stdout:3424: seq_num 1051 ranges {0=409600,409600=401408} 2021-07-31T01:57:03.703 INFO:tasks.rados.rados.0.smithi045.stdout:3424: writing smithi04569269-19 from 0 to 409600 tid 1 2021-07-31T01:57:03.707 INFO:tasks.rados.rados.0.smithi045.stdout:3424: writing smithi04569269-19 from 409600 to 811008 tid 2 2021-07-31T01:57:03.707 INFO:tasks.rados.rados.0.smithi045.stdout:3425: snap_create 2021-07-31T01:57:03.707 INFO:tasks.rados.rados.0.smithi045.stdout:3420: finishing write tid 2 to smithi04569269-23 2021-07-31T01:57:03.708 INFO:tasks.rados.rados.0.smithi045.stdout:3422: finishing copy_from to smithi04569269-14 2021-07-31T01:57:03.708 INFO:tasks.rados.rados.0.smithi045.stdout:3422: got expected ENOENT (src dne) 2021-07-31T01:57:03.708 INFO:tasks.rados.rados.0.smithi045.stdout:3422: finishing copy_from racing read to smithi04569269-14 2021-07-31T01:57:03.708 INFO:tasks.rados.rados.0.smithi045.stdout:3420: finishing write tid 3 to smithi04569269-23 2021-07-31T01:57:03.709 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 23 v 2032 (ObjNum 1049 snap 342 seq_num 1049) dirty exists 2021-07-31T01:57:03.709 INFO:tasks.rados.rados.0.smithi045.stdout:3420: left oid 23 (ObjNum 1049 snap 342 seq_num 1049) 2021-07-31T01:57:03.719 INFO:tasks.rados.rados.0.smithi045.stdout:3424: finishing write tid 1 to smithi04569269-19 2021-07-31T01:57:03.722 INFO:tasks.rados.rados.0.smithi045.stdout:3424: finishing write tid 2 to smithi04569269-19 2021-07-31T01:57:03.722 INFO:tasks.rados.rados.0.smithi045.stdout:3424: finishing write tid 3 to smithi04569269-19 2021-07-31T01:57:03.724 INFO:tasks.rados.rados.0.smithi045.stdout:3424: finishing write tid 4 to smithi04569269-19 2021-07-31T01:57:03.724 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 19 v 1844 (ObjNum 1051 snap 342 seq_num 1051) dirty exists 2021-07-31T01:57:03.724 INFO:tasks.rados.rados.0.smithi045.stdout:3424: left oid 19 (ObjNum 1051 snap 342 seq_num 1051) 2021-07-31T01:57:04.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:03 smithi154 conmon[34553]: cluster 2021-07-31T01:57:02.693722+0000 mon.a (mon.0) 1373 : cluster [DBG] osdmap e707: 8 total, 8 up, 8 in 2021-07-31T01:57:04.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:03 smithi045 conmon[39718]: cluster 2021-07-31T01:57:02.693722+0000 mon.a (mon.0) 1373 : cluster [DBG] osdmap e707: 8 total, 8 up, 8 in 2021-07-31T01:57:04.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:03 smithi045 conmon[33655]: cluster 2021-07-31T01:57:02.693722+0000 mon.a (mon. 2021-07-31T01:57:04.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:03 smithi045 conmon[33655]: 0) 1373 : cluster [DBG] osdmap e707: 8 total, 8 up, 8 in 2021-07-31T01:57:04.027 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:57:03 smithi045 conmon[61210]: level=error ts=2021-07-31T01:57:03.525Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:57:04.027 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:57:03 smithi045 conmon[61210]: level=error ts=2021-07-31T01:57:03.526Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:57:04.027 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:57:03 smithi045 conmon[61210]: level=error ts=2021-07-31T01:57:03.526Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T01:57:04.707 INFO:tasks.rados.rados.0.smithi045.stdout:3420: done (3 left) 2021-07-31T01:57:04.708 INFO:tasks.rados.rados.0.smithi045.stdout:3422: done (2 left) 2021-07-31T01:57:04.708 INFO:tasks.rados.rados.0.smithi045.stdout:3424: done (1 left) 2021-07-31T01:57:04.708 INFO:tasks.rados.rados.0.smithi045.stdout:3425: done (0 left) 2021-07-31T01:57:04.709 INFO:tasks.rados.rados.0.smithi045.stdout:3426: copy_from oid 7 from oid 49 current snap is 343 2021-07-31T01:57:04.709 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 37 current snap is 343 2021-07-31T01:57:04.710 INFO:tasks.rados.rados.0.smithi045.stdout:3427: seq_num 1052 ranges {0=425984,425984=401408} 2021-07-31T01:57:04.710 INFO:tasks.rados.rados.0.smithi045.stdout:3427: writing smithi04569269-37 from 0 to 425984 tid 1 2021-07-31T01:57:04.712 INFO:tasks.rados.rados.0.smithi045.stdout:3427: writing smithi04569269-37 from 425984 to 827392 tid 2 2021-07-31T01:57:04.712 INFO:tasks.rados.rados.0.smithi045.stdout:3428: copy_from oid 5 from oid 6 current snap is 343 2021-07-31T01:57:04.713 INFO:tasks.rados.rados.0.smithi045.stdout:3426: finishing copy_from to smithi04569269-7 2021-07-31T01:57:04.713 INFO:tasks.rados.rados.0.smithi045.stdout:3426: got expected ENOENT (src dne) 2021-07-31T01:57:04.713 INFO:tasks.rados.rados.0.smithi045.stdout:3426: finishing copy_from racing read to smithi04569269-7 2021-07-31T01:57:04.713 INFO:tasks.rados.rados.0.smithi045.stdout:3426: done (2 left) 2021-07-31T01:57:04.714 INFO:tasks.rados.rados.0.smithi045.stdout:3429: rollback oid 46 current snap is 343 2021-07-31T01:57:04.714 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 46 to 342 2021-07-31T01:57:04.714 INFO:tasks.rados.rados.0.smithi045.stdout:3430: rollback oid 33 current snap is 343 2021-07-31T01:57:04.714 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 33 to 307 2021-07-31T01:57:04.714 INFO:tasks.rados.rados.0.smithi045.stdout:3431: copy_from oid 34 from oid 48 current snap is 343 2021-07-31T01:57:04.715 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 26 current snap is 343 2021-07-31T01:57:04.715 INFO:tasks.rados.rados.0.smithi045.stdout:3432: seq_num 1053 ranges {0=442368,442368=401408} 2021-07-31T01:57:04.715 INFO:tasks.rados.rados.0.smithi045.stdout:3432: writing smithi04569269-26 from 0 to 442368 tid 1 2021-07-31T01:57:04.717 INFO:tasks.rados.rados.0.smithi045.stdout:3432: writing smithi04569269-26 from 442368 to 843776 tid 2 2021-07-31T01:57:04.718 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 43 current snap is 343 2021-07-31T01:57:04.718 INFO:tasks.rados.rados.0.smithi045.stdout:3431: finishing copy_from to smithi04569269-34 2021-07-31T01:57:04.718 INFO:tasks.rados.rados.0.smithi045.stdout:3431: got expected ENOENT (src dne) 2021-07-31T01:57:04.719 INFO:tasks.rados.rados.0.smithi045.stdout:3433: seq_num 1054 ranges {573440=458752,1032192=401408} 2021-07-31T01:57:04.723 INFO:tasks.rados.rados.0.smithi045.stdout:3433: writing smithi04569269-43 from 573440 to 1032192 tid 1 2021-07-31T01:57:04.726 INFO:tasks.rados.rados.0.smithi045.stdout:3433: writing smithi04569269-43 from 1032192 to 1433600 tid 2 2021-07-31T01:57:04.727 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 44 current snap is 343 2021-07-31T01:57:04.727 INFO:tasks.rados.rados.0.smithi045.stdout:3431: finishing copy_from racing read to smithi04569269-34 2021-07-31T01:57:04.727 INFO:tasks.rados.rados.0.smithi045.stdout:3427: finishing write tid 1 to smithi04569269-37 2021-07-31T01:57:04.728 INFO:tasks.rados.rados.0.smithi045.stdout:3427: finishing write tid 2 to smithi04569269-37 2021-07-31T01:57:04.728 INFO:tasks.rados.rados.0.smithi045.stdout:3434: seq_num 1055 ranges {3801088=475136,4276224=401408} 2021-07-31T01:57:04.749 INFO:tasks.rados.rados.0.smithi045.stdout:3434: writing smithi04569269-44 from 3801088 to 4276224 tid 1 2021-07-31T01:57:04.752 INFO:tasks.rados.rados.0.smithi045.stdout:3434: writing smithi04569269-44 from 4276224 to 4677632 tid 2 2021-07-31T01:57:04.752 INFO:tasks.rados.rados.0.smithi045.stdout:3431: done (7 left) 2021-07-31T01:57:04.753 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 9 current snap is 343 2021-07-31T01:57:04.753 INFO:tasks.rados.rados.0.smithi045.stdout:3429: finishing rollback tid 0 to smithi04569269-46 2021-07-31T01:57:04.753 INFO:tasks.rados.rados.0.smithi045.stdout:3429: finishing rollback tid 1 to smithi04569269-46 2021-07-31T01:57:04.753 INFO:tasks.rados.rados.0.smithi045.stdout:3427: finishing write tid 3 to smithi04569269-37 2021-07-31T01:57:04.753 INFO:tasks.rados.rados.0.smithi045.stdout:3429: finishing rollback tid 2 to smithi04569269-46 2021-07-31T01:57:04.754 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 46 v 1853 (ObjNum 1041 snap 338 seq_num 1041) dirty exists 2021-07-31T01:57:04.754 INFO:tasks.rados.rados.0.smithi045.stdout:3430: finishing rollback tid 0 to smithi04569269-33 2021-07-31T01:57:04.754 INFO:tasks.rados.rados.0.smithi045.stdout:3430: finishing rollback tid 1 to smithi04569269-33 2021-07-31T01:57:04.754 INFO:tasks.rados.rados.0.smithi045.stdout:3427: finishing write tid 4 to smithi04569269-37 2021-07-31T01:57:04.755 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 37 v 1878 (ObjNum 1052 snap 343 seq_num 1052) dirty exists 2021-07-31T01:57:04.755 INFO:tasks.rados.rados.0.smithi045.stdout:3427: left oid 37 (ObjNum 1052 snap 343 seq_num 1052) 2021-07-31T01:57:04.755 INFO:tasks.rados.rados.0.smithi045.stdout:3430: finishing rollback tid 2 to smithi04569269-33 2021-07-31T01:57:04.755 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 33 v 2035 (ObjNum 988 snap 307 seq_num 988) dirty exists 2021-07-31T01:57:04.755 INFO:tasks.rados.rados.0.smithi045.stdout:3432: finishing write tid 1 to smithi04569269-26 2021-07-31T01:57:04.756 INFO:tasks.rados.rados.0.smithi045.stdout:3435: seq_num 1056 ranges {11780096=491520,12271616=401408} 2021-07-31T01:57:04.819 INFO:tasks.rados.rados.0.smithi045.stdout:3435: writing smithi04569269-9 from 11780096 to 12271616 tid 1 2021-07-31T01:57:04.822 INFO:tasks.rados.rados.0.smithi045.stdout:3435: writing smithi04569269-9 from 12271616 to 12673024 tid 2 2021-07-31T01:57:04.822 INFO:tasks.rados.rados.0.smithi045.stdout:3427: done (7 left) 2021-07-31T01:57:04.822 INFO:tasks.rados.rados.0.smithi045.stdout:3429: done (6 left) 2021-07-31T01:57:04.823 INFO:tasks.rados.rados.0.smithi045.stdout:3430: done (5 left) 2021-07-31T01:57:04.823 INFO:tasks.rados.rados.0.smithi045.stdout:3436: read oid 16 snap -1 2021-07-31T01:57:04.823 INFO:tasks.rados.rados.0.smithi045.stdout:3436: expect (ObjNum 1031 snap 335 seq_num 1031) 2021-07-31T01:57:04.823 INFO:tasks.rados.rados.0.smithi045.stdout:3437: copy_from oid 33 from oid 34 current snap is 343 2021-07-31T01:57:04.823 INFO:tasks.rados.rados.0.smithi045.stdout:3432: finishing write tid 2 to smithi04569269-26 2021-07-31T01:57:04.824 INFO:tasks.rados.rados.0.smithi045.stdout:3432: finishing write tid 3 to smithi04569269-26 2021-07-31T01:57:04.824 INFO:tasks.rados.rados.0.smithi045.stdout:3432: finishing write tid 4 to smithi04569269-26 2021-07-31T01:57:04.824 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 26 v 1858 (ObjNum 1053 snap 343 seq_num 1053) dirty exists 2021-07-31T01:57:04.826 INFO:tasks.rados.rados.0.smithi045.stdout:3432: left oid 26 (ObjNum 1053 snap 343 seq_num 1053) 2021-07-31T01:57:04.826 INFO:tasks.rados.rados.0.smithi045.stdout:3432: done (6 left) 2021-07-31T01:57:04.826 INFO:tasks.rados.rados.0.smithi045.stdout:3438: snap_remove snap 321 2021-07-31T01:57:05.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:04 smithi154 conmon[34553]: cluster 2021-07-31T01: 2021-07-31T01:57:05.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:04 smithi154 conmon[34553]: 57:03.699494+0000 mon.a (mon.0) 1374 : cluster [DBG] osdmap e708: 8 total, 8 up, 8 in 2021-07-31T01:57:05.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:04 smithi154 conmon[34553]: cluster 2021-07-31T 2021-07-31T01:57:05.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:04 smithi154 conmon[34553]: 01:57:03.933316+0000 mgr.x (mgr.24542) 419 : cluster [DBG] pgmap v850: 25 pgs: 25 active+clean; 578 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 15 MiB/s rd, 3.6 MiB/s wr, 19 op/s 2021-07-31T01:57:05.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:04 smithi045 conmon[39718]: cluster 2021-07-31T01:57:03.699494 2021-07-31T01:57:05.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:04 smithi045 conmon[39718]: +0000 mon.a (mon.0) 1374 : cluster [DBG] osdmap e708: 8 total, 8 up, 8 in 2021-07-31T01:57:05.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:04 smithi045 conmon[39718]: cluster 2021-07-31 2021-07-31T01:57:05.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:04 smithi045 conmon[39718]: T01:57:03.933316+0000 2021-07-31T01:57:05.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:04 smithi045 conmon[39718]: mgr.x (mgr.24542) 419 : cluster 2021-07-31T01:57:05.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:04 smithi045 conmon[39718]: [DBG] pgmap v850: 25 pgs: 25 active+clean; 578 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 15 MiB/s rd, 3.6 MiB/s wr, 19 op/s 2021-07-31T01:57:05.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:04 smithi045 conmon[33655]: cluster 2021-07-31T01:57:03.699494+0000 mon.a (mon.0 2021-07-31T01:57:05.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:04 smithi045 conmon[33655]: ) 1374 : cluster [DBG] osdmap e708: 8 total, 8 up, 8 in 2021-07-31T01:57:05.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:04 smithi045 conmon[33655]: cluster 2021-07-31T01:57:03.933316+0000 mgr.x (mgr.24542) 419 : cluster [DBG] pgmap v850: 25 pgs: 25 active+clean; 578 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 15 MiB/s rd, 3.6 MiB/s wr, 19 op/s 2021-07-31T01:57:05.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:04 smithi045 conmon[33655]: 2021-07-31T01:57:05.715 INFO:tasks.rados.rados.0.smithi045.stdout:3433: finishing write tid 1 to smithi04569269-43 2021-07-31T01:57:05.715 INFO:tasks.rados.rados.0.smithi045.stdout:3433: finishing write tid 2 to smithi04569269-43 2021-07-31T01:57:05.716 INFO:tasks.rados.rados.0.smithi045.stdout:3433: finishing write tid 3 to smithi04569269-43 2021-07-31T01:57:05.716 INFO:tasks.rados.rados.0.smithi045.stdout:3428: finishing copy_from to smithi04569269-5 2021-07-31T01:57:05.717 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 5 v 1856 (ObjNum 996 snap 311 seq_num 996) dirty exists 2021-07-31T01:57:05.717 INFO:tasks.rados.rados.0.smithi045.stdout:3428: finishing copy_from racing read to smithi04569269-5 2021-07-31T01:57:05.718 INFO:tasks.rados.rados.0.smithi045.stdout:3433: finishing write tid 4 to smithi04569269-43 2021-07-31T01:57:05.718 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 43 v 2033 (ObjNum 1054 snap 343 seq_num 1054) dirty exists 2021-07-31T01:57:05.719 INFO:tasks.rados.rados.0.smithi045.stdout:3433: left oid 43 (ObjNum 1054 snap 343 seq_num 1054) 2021-07-31T01:57:05.719 INFO:tasks.rados.rados.0.smithi045.stdout:3434: finishing write tid 1 to smithi04569269-44 2021-07-31T01:57:05.720 INFO:tasks.rados.rados.0.smithi045.stdout:3434: finishing write tid 2 to smithi04569269-44 2021-07-31T01:57:05.720 INFO:tasks.rados.rados.0.smithi045.stdout:3434: finishing write tid 3 to smithi04569269-44 2021-07-31T01:57:05.721 INFO:tasks.rados.rados.0.smithi045.stdout:3434: finishing write tid 4 to smithi04569269-44 2021-07-31T01:57:05.721 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 44 v 2036 (ObjNum 1055 snap 343 seq_num 1055) dirty exists 2021-07-31T01:57:05.722 INFO:tasks.rados.rados.0.smithi045.stdout:3434: left oid 44 (ObjNum 1055 snap 343 seq_num 1055) 2021-07-31T01:57:05.722 INFO:tasks.rados.rados.0.smithi045.stdout:3437: finishing copy_from to smithi04569269-33 2021-07-31T01:57:05.722 INFO:tasks.rados.rados.0.smithi045.stdout:3437: got expected ENOENT (src dne) 2021-07-31T01:57:05.723 INFO:tasks.rados.rados.0.smithi045.stdout:3437: finishing copy_from racing read to smithi04569269-33 2021-07-31T01:57:05.723 INFO:tasks.rados.rados.0.smithi045.stdout:3435: finishing write tid 1 to smithi04569269-9 2021-07-31T01:57:05.723 INFO:tasks.rados.rados.0.smithi045.stdout:3435: finishing write tid 2 to smithi04569269-9 2021-07-31T01:57:05.724 INFO:tasks.rados.rados.0.smithi045.stdout:3435: finishing write tid 3 to smithi04569269-9 2021-07-31T01:57:05.724 INFO:tasks.rados.rados.0.smithi045.stdout:3435: finishing write tid 4 to smithi04569269-9 2021-07-31T01:57:05.725 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 9 v 1849 (ObjNum 1056 snap 343 seq_num 1056) dirty exists 2021-07-31T01:57:05.725 INFO:tasks.rados.rados.0.smithi045.stdout:3435: left oid 9 (ObjNum 1056 snap 343 seq_num 1056) 2021-07-31T01:57:05.725 INFO:tasks.rados.rados.0.smithi045.stdout:3436: expect (ObjNum 1031 snap 335 seq_num 1031) 2021-07-31T01:57:05.857 INFO:tasks.rados.rados.0.smithi045.stdout:3428: done (6 left) 2021-07-31T01:57:05.858 INFO:tasks.rados.rados.0.smithi045.stdout:3433: done (5 left) 2021-07-31T01:57:05.858 INFO:tasks.rados.rados.0.smithi045.stdout:3434: done (4 left) 2021-07-31T01:57:05.858 INFO:tasks.rados.rados.0.smithi045.stdout:3435: done (3 left) 2021-07-31T01:57:05.858 INFO:tasks.rados.rados.0.smithi045.stdout:3436: done (2 left) 2021-07-31T01:57:05.858 INFO:tasks.rados.rados.0.smithi045.stdout:3437: done (1 left) 2021-07-31T01:57:05.859 INFO:tasks.rados.rados.0.smithi045.stdout:3438: done (0 left) 2021-07-31T01:57:05.859 INFO:tasks.rados.rados.0.smithi045.stdout:3439: copy_from oid 45 from oid 30 current snap is 343 2021-07-31T01:57:05.880 INFO:tasks.rados.rados.0.smithi045.stdout:3440: rollback oid 35 current snap is 343 2021-07-31T01:57:05.881 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 35 to 322 2021-07-31T01:57:05.883 INFO:tasks.rados.rados.0.smithi045.stdout:3441: copy_from oid 11 from oid 41 current snap is 343 2021-07-31T01:57:05.883 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 14 current snap is 343 2021-07-31T01:57:05.883 INFO:tasks.rados.rados.0.smithi045.stdout:3442: seq_num 1057 ranges {0=507904,507904=401408} 2021-07-31T01:57:05.890 INFO:tasks.rados.rados.0.smithi045.stdout:3442: writing smithi04569269-14 from 0 to 507904 tid 1 2021-07-31T01:57:05.890 INFO:tasks.rados.rados.0.smithi045.stdout:3442: writing smithi04569269-14 from 507904 to 909312 tid 2 2021-07-31T01:57:05.891 INFO:tasks.rados.rados.0.smithi045.stdout:3443: copy_from oid 32 from oid 5 current snap is 343 2021-07-31T01:57:05.891 INFO:tasks.rados.rados.0.smithi045.stdout:3439: finishing copy_from to smithi04569269-45 2021-07-31T01:57:05.891 INFO:tasks.rados.rados.0.smithi045.stdout:3439: got expected ENOENT (src dne) 2021-07-31T01:57:05.892 INFO:tasks.rados.rados.0.smithi045.stdout:3439: finishing copy_from racing read to smithi04569269-45 2021-07-31T01:57:05.892 INFO:tasks.rados.rados.0.smithi045.stdout:3440: finishing rollback tid 0 to smithi04569269-35 2021-07-31T01:57:05.893 INFO:tasks.rados.rados.0.smithi045.stdout:3440: finishing rollback tid 1 to smithi04569269-35 2021-07-31T01:57:05.893 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 35 v 2248 (ObjNum 0 snap 0 seq_num 0) dirty dne 2021-07-31T01:57:05.894 INFO:tasks.rados.rados.0.smithi045.stdout:3439: done (4 left) 2021-07-31T01:57:05.894 INFO:tasks.rados.rados.0.smithi045.stdout:3440: done (3 left) 2021-07-31T01:57:05.895 INFO:tasks.rados.rados.0.smithi045.stdout:3444: snap_remove snap 318 2021-07-31T01:57:06.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:05 smithi154 conmon[34553]: cluster 2021-07-31T01:57:04.706255+0000 mon.a (mon. 2021-07-31T01:57:06.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:05 smithi154 conmon[34553]: 0) 1375 : cluster [DBG] osdmap e709: 8 total, 8 up, 8 in 2021-07-31T01:57:06.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:05 smithi045 conmon[39718]: cluster 2021-07-31T01:57:06.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:05 smithi045 conmon[39718]: 2021-07-31T01:57:06.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:05 smithi045 conmon[39718]: 2021 2021-07-31T01:57:06.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:05 smithi045 conmon[39718]: - 2021-07-31T01:57:06.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:05 smithi045 conmon[39718]: 07 2021-07-31T01:57:06.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:05 smithi045 conmon[39718]: - 2021-07-31T01:57:06.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:05 smithi045 conmon[39718]: 31 2021-07-31T01:57:06.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:05 smithi045 conmon[39718]: T 2021-07-31T01:57:06.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:05 smithi045 conmon[39718]: 01 2021-07-31T01:57:06.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:05 smithi045 conmon[39718]: :57:04.706255+0000 mon.a (mon.0) 2021-07-31T01:57:06.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:05 smithi045 conmon[39718]: 1375 : cluster [DBG] osdmap e709: 8 total, 8 up, 8 in 2021-07-31T01:57:06.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:05 smithi045 conmon[33655]: cluster 2021-07-31T01:57:04.706255+0000 mon.a (mon.0) 1375 2021-07-31T01:57:06.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:05 smithi045 conmon[33655]: : cluster [DBG] osdmap e709: 8 total, 8 up, 8 in 2021-07-31T01:57:06.662 INFO:tasks.rados.rados.0.smithi045.stdout:3444: done (3 left) 2021-07-31T01:57:06.662 INFO:tasks.rados.rados.0.smithi045.stdout:3445: snap_create 2021-07-31T01:57:06.663 INFO:tasks.rados.rados.0.smithi045.stdout:3442: finishing write tid 1 to smithi04569269-14 2021-07-31T01:57:06.663 INFO:tasks.rados.rados.0.smithi045.stdout:3442: finishing write tid 2 to smithi04569269-14 2021-07-31T01:57:06.664 INFO:tasks.rados.rados.0.smithi045.stdout:3442: finishing write tid 3 to smithi04569269-14 2021-07-31T01:57:06.664 INFO:tasks.rados.rados.0.smithi045.stdout:3442: finishing write tid 4 to smithi04569269-14 2021-07-31T01:57:06.664 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 14 v 2039 (ObjNum 1057 snap 343 seq_num 1057) dirty exists 2021-07-31T01:57:06.664 INFO:tasks.rados.rados.0.smithi045.stdout:3442: left oid 14 (ObjNum 1057 snap 343 seq_num 1057) 2021-07-31T01:57:06.664 INFO:tasks.rados.rados.0.smithi045.stdout:3441: finishing copy_from to smithi04569269-11 2021-07-31T01:57:06.665 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 11 v 1690 (ObjNum 948 snap 286 seq_num 948) dirty exists 2021-07-31T01:57:06.665 INFO:tasks.rados.rados.0.smithi045.stdout:3441: finishing copy_from racing read to smithi04569269-11 2021-07-31T01:57:06.666 INFO:tasks.rados.rados.0.smithi045.stdout:3443: finishing copy_from to smithi04569269-32 2021-07-31T01:57:06.666 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 32 v 1856 (ObjNum 996 snap 311 seq_num 996) dirty exists 2021-07-31T01:57:06.668 INFO:tasks.rados.rados.0.smithi045.stdout:3443: finishing copy_from racing read to smithi04569269-32 2021-07-31T01:57:07.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:06 smithi154 conmon[34553]: cluster 2021-07-31T01:57:05.707895+0000 mon.a (mon.0) 1376 : cluster 2021-07-31T01:57:07.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:06 smithi154 conmon[34553]: [DBG] osdmap e710: 8 total, 8 up, 8 in 2021-07-31T01:57:07.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:06 smithi154 conmon[34553]: cluster 2021-07-31T01:57:05.933669+0000 mgr.x (mgr.24542) 420 : cluster [DBG] pgmap v853: 25 pgs: 25 active+clean; 578 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 15 MiB/s rd, 3.6 MiB/s wr, 19 op/s 2021-07-31T01:57:07.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:06 smithi154 conmon[34553]: cluster 2021-07-31T01: 2021-07-31T01:57:07.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:06 smithi154 conmon[34553]: 57:06.660398+0000 mon.a (mon.0) 1377 : cluster [DBG] osdmap e711: 8 total, 8 up, 8 in 2021-07-31T01:57:07.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:06 smithi045 conmon[33655]: cluster 2021-07-31T01:57: 2021-07-31T01:57:07.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:06 smithi045 conmon[33655]: 05.707895+0000 mon.a (mon.0) 1376 : cluster [DBG] osdmap e710: 8 total, 8 up, 8 in 2021-07-31T01:57:07.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:06 smithi045 conmon[33655]: cluster 2021-07-31T01:57:07.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:06 smithi045 conmon[33655]: 2021-07-31T01:57:05.933669+0000 mgr.x ( 2021-07-31T01:57:07.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:06 smithi045 conmon[33655]: mgr.24542) 420 : cluster [DBG] 2021-07-31T01:57:07.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:06 smithi045 conmon[33655]: pgmap v853: 25 pgs: 25 active+clean; 578 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 15 MiB/s rd, 3.6 MiB/s wr, 19 op/s 2021-07-31T01:57:07.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:06 smithi045 conmon[33655]: cluster 2021-07-31 2021-07-31T01:57:07.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:06 smithi045 conmon[33655]: T01:57:06.660398+0000 mon.a (mon.0) 1377 : cluster [DBG] osdmap e711: 8 total, 8 up, 8 in 2021-07-31T01:57:07.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:06 smithi045 conmon[33655]: 2021-07-31T01:57:07.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:06 smithi045 conmon[39718]: cluster 2021-07-31T01: 2021-07-31T01:57:07.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:06 smithi045 conmon[39718]: 57:05.707895+0000 mon.a (mon.0) 1376 : cluster [DBG] osdmap e710: 8 total, 8 up, 8 in 2021-07-31T01:57:07.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:06 smithi045 conmon[39718]: cluster 2021 2021-07-31T01:57:07.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:06 smithi045 conmon[39718]: -07-31T01:57:05 2021-07-31T01:57:07.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:06 smithi045 conmon[39718]: .933669+0000 mgr.x (mgr. 2021-07-31T01:57:07.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:06 smithi045 conmon[39718]: 24542) 420 : cluster [DBG] 2021-07-31T01:57:07.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:06 smithi045 conmon[39718]: pgmap v853: 25 pgs: 25 active+clean; 578 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 15 MiB/s rd, 3.6 MiB/s wr, 19 op/s 2021-07-31T01:57:07.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:06 smithi045 conmon[39718]: cluster 2021-07-31T01:57:07.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:06 smithi045 conmon[39718]: 2021-07-31T01:57: 2021-07-31T01:57:07.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:06 smithi045 conmon[39718]: 06.660398+0000 mon.a ( 2021-07-31T01:57:07.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:06 smithi045 conmon[39718]: mon.0) 1377 : cluster 2021-07-31T01:57:07.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:06 smithi045 conmon[39718]: [DBG] osdmap e711: 8 total, 8 up, 8 in 2021-07-31T01:57:07.670 INFO:tasks.rados.rados.0.smithi045.stdout:3441: done (3 left) 2021-07-31T01:57:07.671 INFO:tasks.rados.rados.0.smithi045.stdout:3442: done (2 left) 2021-07-31T01:57:07.671 INFO:tasks.rados.rados.0.smithi045.stdout:3443: done (1 left) 2021-07-31T01:57:07.671 INFO:tasks.rados.rados.0.smithi045.stdout:3445: done (0 left) 2021-07-31T01:57:07.672 INFO:tasks.rados.rados.0.smithi045.stdout:3446: rollback oid 42 current snap is 344 2021-07-31T01:57:07.672 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 42 to 307 2021-07-31T01:57:07.672 INFO:tasks.rados.rados.0.smithi045.stdout:3447: read oid 27 snap -1 2021-07-31T01:57:07.673 INFO:tasks.rados.rados.0.smithi045.stdout:3447: expect (ObjNum 0 snap 0 seq_num 0) 2021-07-31T01:57:07.673 INFO:tasks.rados.rados.0.smithi045.stdout:3448: copy_from oid 12 from oid 26 current snap is 344 2021-07-31T01:57:07.674 INFO:tasks.rados.rados.0.smithi045.stdout:3449: rmattr oid 33 current snap is 344 2021-07-31T01:57:07.674 INFO:tasks.rados.rados.0.smithi045.stdout:3449: done (3 left) 2021-07-31T01:57:07.675 INFO:tasks.rados.rados.0.smithi045.stdout:3450: read oid 37 snap -1 2021-07-31T01:57:07.676 INFO:tasks.rados.rados.0.smithi045.stdout:3450: expect (ObjNum 1052 snap 343 seq_num 1052) 2021-07-31T01:57:07.676 INFO:tasks.rados.rados.0.smithi045.stdout:3451: setattr oid 15 current snap is 344 2021-07-31T01:57:07.677 INFO:tasks.rados.rados.0.smithi045.stdout:3452: delete oid 7 current snap is 344 2021-07-31T01:57:07.677 INFO:tasks.rados.rados.0.smithi045.stdout:3447: expect (ObjNum 0 snap 0 seq_num 0) 2021-07-31T01:57:07.678 INFO:tasks.rados.rados.0.smithi045.stdout:3447: done (5 left) 2021-07-31T01:57:07.678 INFO:tasks.rados.rados.0.smithi045.stdout:3452: done (4 left) 2021-07-31T01:57:07.678 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 28 current snap is 344 2021-07-31T01:57:07.678 INFO:tasks.rados.rados.0.smithi045.stdout:3453: seq_num 1059 ranges {0=540672,540672=401408} 2021-07-31T01:57:07.680 INFO:tasks.rados.rados.0.smithi045.stdout:3453: writing smithi04569269-28 from 0 to 540672 tid 1 2021-07-31T01:57:07.682 INFO:tasks.rados.rados.0.smithi045.stdout:3453: writing smithi04569269-28 from 540672 to 942080 tid 2 2021-07-31T01:57:07.683 INFO:tasks.rados.rados.0.smithi045.stdout:3454: delete oid 36 current snap is 344 2021-07-31T01:57:07.683 INFO:tasks.rados.rados.0.smithi045.stdout:3450: expect (ObjNum 1052 snap 343 seq_num 1052) 2021-07-31T01:57:07.706 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 15 v 1879 (ObjNum 1029 snap 331 seq_num 1029) dirty exists 2021-07-31T01:57:07.706 INFO:tasks.rados.rados.0.smithi045.stdout:3446: finishing rollback tid 0 to smithi04569269-42 2021-07-31T01:57:07.706 INFO:tasks.rados.rados.0.smithi045.stdout:3446: finishing rollback tid 1 to smithi04569269-42 2021-07-31T01:57:07.706 INFO:tasks.rados.rados.0.smithi045.stdout:3450: done (5 left) 2021-07-31T01:57:07.707 INFO:tasks.rados.rados.0.smithi045.stdout:3451: done (4 left) 2021-07-31T01:57:07.707 INFO:tasks.rados.rados.0.smithi045.stdout:3454: done (3 left) 2021-07-31T01:57:07.707 INFO:tasks.rados.rados.0.smithi045.stdout:3455: snap_remove snap 320 2021-07-31T01:57:08.676 INFO:tasks.rados.rados.0.smithi045.stdout:3455: done (3 left) 2021-07-31T01:57:08.677 INFO:tasks.rados.rados.0.smithi045.stdout:3456: read oid 29 snap -1 2021-07-31T01:57:08.677 INFO:tasks.rados.rados.0.smithi045.stdout:3456: expect (ObjNum 0 snap 0 seq_num 0) 2021-07-31T01:57:08.677 INFO:tasks.rados.rados.0.smithi045.stdout:3457: snap_remove snap 286 2021-07-31T01:57:09.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:08 smithi154 conmon[34553]: cluster 2021-07-31T01:57:07.669293+0000 mon.a (mon.0) 1378 : cluster [DBG] 2021-07-31T01:57:09.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:08 smithi154 conmon[34553]: osdmap e712: 8 total, 8 up, 8 in 2021-07-31T01:57:09.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:08 smithi154 conmon[34553]: cluster 2021-07-31T01:57:07.934078+0000 mgr.x (mgr.24542) 421 : cluster [DBG] pgmap v856: 25 pgs: 25 active+clean; 578 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:57:09.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:08 smithi045 conmon[33655]: cluster 2021-07-31T01:57:07.669293+0000 mon.a (mon.0 2021-07-31T01:57:09.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:08 smithi045 conmon[33655]: ) 1378 : cluster [DBG] osdmap e712: 8 total, 8 up, 8 in 2021-07-31T01:57:09.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:08 smithi045 conmon[33655]: cluster 2021-07-31T01:57:07.934078+0000 mgr.x (mgr.24542) 421 : cluster [DBG] pgmap v856: 25 pgs: 25 active+clean; 578 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:57:09.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:08 smithi045 conmon[39718]: cluster 2021-07-31T01:57:07.669293+0000 mon.a (mon 2021-07-31T01:57:09.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:08 smithi045 conmon[39718]: .0) 1378 : cluster [DBG] osdmap e712: 8 total, 8 up, 8 in 2021-07-31T01:57:09.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:08 smithi045 conmon[39718]: cluster 2021-07-31T01:57:07.934078+0000 mgr.x (mgr.24542) 421 : cluster [DBG] pgmap v856: 25 pgs: 25 active+clean; 578 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:57:09.681 INFO:tasks.rados.rados.0.smithi045.stdout:3457: done (4 left) 2021-07-31T01:57:09.681 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 40 current snap is 344 2021-07-31T01:57:09.681 INFO:tasks.rados.rados.0.smithi045.stdout:3446: finishing rollback tid 2 to smithi04569269-42 2021-07-31T01:57:09.682 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 42 v 2030 (ObjNum 983 snap 303 seq_num 983) dirty exists 2021-07-31T01:57:09.682 INFO:tasks.rados.rados.0.smithi045.stdout:3458: seq_num 1061 ranges {0=573440,573440=401408} 2021-07-31T01:57:09.684 INFO:tasks.rados.rados.0.smithi045.stdout:3458: writing smithi04569269-40 from 0 to 573440 tid 1 2021-07-31T01:57:09.686 INFO:tasks.rados.rados.0.smithi045.stdout:3458: writing smithi04569269-40 from 573440 to 974848 tid 2 2021-07-31T01:57:09.687 INFO:tasks.rados.rados.0.smithi045.stdout:3446: done (4 left) 2021-07-31T01:57:09.688 INFO:tasks.rados.rados.0.smithi045.stdout:3459: snap_remove snap 337 2021-07-31T01:57:10.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:09 smithi154 conmon[34553]: cluster 2021-07-31T01:57:08.675245+0000 mon.a (mon.0) 1379 : cluster [DBG] 2021-07-31T01:57:10.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:09 smithi154 conmon[34553]: osdmap e713: 8 total, 8 up, 8 in 2021-07-31T01:57:10.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:09 smithi045 conmon[39718]: cluster 2021-07-31T01:57:08.675245+0000 mon.a (mon. 2021-07-31T01:57:10.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:09 smithi045 conmon[39718]: 0) 1379 : cluster [DBG] osdmap e713: 8 total, 8 up, 8 in 2021-07-31T01:57:10.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:09 smithi045 conmon[33655]: cluster 2021-07-31T01:57:08.675245+0000 mon.a (mon.0 2021-07-31T01:57:10.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:09 smithi045 conmon[33655]: ) 1379 : cluster [DBG] osdmap e713: 8 total, 8 up, 8 in 2021-07-31T01:57:10.689 INFO:tasks.rados.rados.0.smithi045.stdout:3459: done (4 left) 2021-07-31T01:57:10.690 INFO:tasks.rados.rados.0.smithi045.stdout:3460: read oid 25 snap -1 2021-07-31T01:57:10.690 INFO:tasks.rados.rados.0.smithi045.stdout:3460: expect (ObjNum 1031 snap 335 seq_num 1031) 2021-07-31T01:57:10.690 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 37 current snap is 344 2021-07-31T01:57:10.691 INFO:tasks.rados.rados.0.smithi045.stdout:3461: seq_num 1062 ranges {827392=589824,1417216=401408} 2021-07-31T01:57:10.698 INFO:tasks.rados.rados.0.smithi045.stdout:3461: writing smithi04569269-37 from 827392 to 1417216 tid 1 2021-07-31T01:57:10.701 INFO:tasks.rados.rados.0.smithi045.stdout:3461: writing smithi04569269-37 from 1417216 to 1818624 tid 2 2021-07-31T01:57:10.701 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 34 current snap is 344 2021-07-31T01:57:10.702 INFO:tasks.rados.rados.0.smithi045.stdout:3448: finishing copy_from to smithi04569269-12 2021-07-31T01:57:10.703 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 12 v 1858 (ObjNum 1053 snap 343 seq_num 1053) dirty exists 2021-07-31T01:57:10.703 INFO:tasks.rados.rados.0.smithi045.stdout:3448: finishing copy_from racing read to smithi04569269-12 2021-07-31T01:57:10.703 INFO:tasks.rados.rados.0.smithi045.stdout:3453: finishing write tid 1 to smithi04569269-28 2021-07-31T01:57:10.703 INFO:tasks.rados.rados.0.smithi045.stdout:3453: finishing write tid 2 to smithi04569269-28 2021-07-31T01:57:10.704 INFO:tasks.rados.rados.0.smithi045.stdout:3453: finishing write tid 3 to smithi04569269-28 2021-07-31T01:57:10.704 INFO:tasks.rados.rados.0.smithi045.stdout:3453: finishing write tid 4 to smithi04569269-28 2021-07-31T01:57:10.704 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 28 v 2033 (ObjNum 1059 snap 344 seq_num 1059) dirty exists 2021-07-31T01:57:10.704 INFO:tasks.rados.rados.0.smithi045.stdout:3453: left oid 28 (ObjNum 1059 snap 344 seq_num 1059) 2021-07-31T01:57:10.705 INFO:tasks.rados.rados.0.smithi045.stdout:3456: expect (ObjNum 0 snap 0 seq_num 0) 2021-07-31T01:57:10.705 INFO:tasks.rados.rados.0.smithi045.stdout:3458: finishing write tid 1 to smithi04569269-40 2021-07-31T01:57:10.706 INFO:tasks.rados.rados.0.smithi045.stdout:3458: finishing write tid 2 to smithi04569269-40 2021-07-31T01:57:10.706 INFO:tasks.rados.rados.0.smithi045.stdout:3458: finishing write tid 3 to smithi04569269-40 2021-07-31T01:57:10.706 INFO:tasks.rados.rados.0.smithi045.stdout:3458: finishing write tid 4 to smithi04569269-40 2021-07-31T01:57:10.706 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 40 v 1877 (ObjNum 1061 snap 344 seq_num 1061) dirty exists 2021-07-31T01:57:10.707 INFO:tasks.rados.rados.0.smithi045.stdout:3458: left oid 40 (ObjNum 1061 snap 344 seq_num 1061) 2021-07-31T01:57:10.707 INFO:tasks.rados.rados.0.smithi045.stdout:3462: seq_num 1063 ranges {0=606208,606208=401408} 2021-07-31T01:57:10.708 INFO:tasks.rados.rados.0.smithi045.stdout:3462: writing smithi04569269-34 from 0 to 606208 tid 1 2021-07-31T01:57:10.711 INFO:tasks.rados.rados.0.smithi045.stdout:3462: writing smithi04569269-34 from 606208 to 1007616 tid 2 2021-07-31T01:57:10.715 INFO:tasks.rados.rados.0.smithi045.stdout:3460: expect (ObjNum 1031 snap 335 seq_num 1031) 2021-07-31T01:57:10.853 INFO:tasks.rados.rados.0.smithi045.stdout:3462: finishing write tid 1 to smithi04569269-34 2021-07-31T01:57:10.854 INFO:tasks.rados.rados.0.smithi045.stdout:3461: finishing write tid 1 to smithi04569269-37 2021-07-31T01:57:10.854 INFO:tasks.rados.rados.0.smithi045.stdout:3461: finishing write tid 2 to smithi04569269-37 2021-07-31T01:57:10.854 INFO:tasks.rados.rados.0.smithi045.stdout:3462: finishing write tid 2 to smithi04569269-34 2021-07-31T01:57:10.855 INFO:tasks.rados.rados.0.smithi045.stdout:3461: finishing write tid 3 to smithi04569269-37 2021-07-31T01:57:10.855 INFO:tasks.rados.rados.0.smithi045.stdout:3448: done (6 left) 2021-07-31T01:57:10.855 INFO:tasks.rados.rados.0.smithi045.stdout:3453: done (5 left) 2021-07-31T01:57:10.856 INFO:tasks.rados.rados.0.smithi045.stdout:3456: done (4 left) 2021-07-31T01:57:10.856 INFO:tasks.rados.rados.0.smithi045.stdout:3458: done (3 left) 2021-07-31T01:57:10.856 INFO:tasks.rados.rados.0.smithi045.stdout:3460: done (2 left) 2021-07-31T01:57:10.856 INFO:tasks.rados.rados.0.smithi045.stdout:3463: setattr oid 32 current snap is 344 2021-07-31T01:57:10.857 INFO:tasks.rados.rados.0.smithi045.stdout:3462: finishing write tid 3 to smithi04569269-34 2021-07-31T01:57:10.857 INFO:tasks.rados.rados.0.smithi045.stdout:3461: finishing write tid 4 to smithi04569269-37 2021-07-31T01:57:10.857 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 37 v 1882 (ObjNum 1062 snap 344 seq_num 1062) dirty exists 2021-07-31T01:57:10.857 INFO:tasks.rados.rados.0.smithi045.stdout:3461: left oid 37 (ObjNum 1062 snap 344 seq_num 1062) 2021-07-31T01:57:10.857 INFO:tasks.rados.rados.0.smithi045.stdout:3462: finishing write tid 4 to smithi04569269-34 2021-07-31T01:57:10.858 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 34 v 1861 (ObjNum 1063 snap 344 seq_num 1063) dirty exists 2021-07-31T01:57:10.858 INFO:tasks.rados.rados.0.smithi045.stdout:3462: left oid 34 (ObjNum 1063 snap 344 seq_num 1063) 2021-07-31T01:57:10.858 INFO:tasks.rados.rados.0.smithi045.stdout:3461: done (2 left) 2021-07-31T01:57:10.858 INFO:tasks.rados.rados.0.smithi045.stdout:3462: done (1 left) 2021-07-31T01:57:10.858 INFO:tasks.rados.rados.0.smithi045.stdout:3464: rollback oid 30 current snap is 344 2021-07-31T01:57:10.859 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 30 to 312 2021-07-31T01:57:10.859 INFO:tasks.rados.rados.0.smithi045.stdout:3465: snap_remove snap 327 2021-07-31T01:57:11.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:10 smithi154 conmon[34553]: cluster 2021-07-31T01:57:11.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:10 smithi154 conmon[34553]: 2021-07-31T01:57:11.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:10 smithi154 conmon[34553]: 2021 2021-07-31T01:57:11.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:10 smithi154 conmon[34553]: - 2021-07-31T01:57:11.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:10 smithi154 conmon[34553]: 07 2021-07-31T01:57:11.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:10 smithi154 conmon[34553]: - 2021-07-31T01:57:11.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:10 smithi154 conmon[34553]: 31T01:57:09.680212+0000 mon.a (mon.0) 1380 2021-07-31T01:57:11.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:10 smithi154 conmon[34553]: : cluster [DBG] osdmap e714: 8 total, 8 up, 8 in 2021-07-31T01:57:11.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:10 smithi154 conmon[34553]: cluster 2021- 2021-07-31T01:57:11.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:10 smithi154 conmon[34553]: 07-31T01:57:09.934501+0000 mgr.x (mgr.24542) 422 : cluster [DBG] pgmap v859: 25 pgs: 25 active+clean; 595 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 3.1 MiB/s wr, 20 op/s 2021-07-31T01:57:11.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:10 smithi045 conmon[33655]: cluster 2021-07-31T01:57:09.680212+0000 2021-07-31T01:57:11.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:10 smithi045 conmon[33655]: mon.a (mon.0) 1380 : cluster [DBG] osdmap e714: 8 total, 8 up, 8 in 2021-07-31T01:57:11.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:10 smithi045 conmon[33655]: cluster 2021-07-31T01:57:09.934501+0000 mgr.x (mgr.24542) 422 : cluster [DBG] 2021-07-31T01:57:11.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:10 smithi045 conmon[33655]: pgmap v859: 25 pgs: 25 active+clean; 595 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 3.1 MiB/s wr, 20 op/s 2021-07-31T01:57:11.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:10 smithi045 conmon[39718]: cluster 2021-07-31T01:57:09.680212+0000 mon.a (mon 2021-07-31T01:57:11.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:10 smithi045 conmon[39718]: .0) 1380 : cluster [DBG] osdmap e714: 8 total, 8 up, 8 in 2021-07-31T01:57:11.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:10 smithi045 conmon[39718]: cluster 2021-07-31T01:57:09. 2021-07-31T01:57:11.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:10 smithi045 conmon[39718]: 934501+0000 mgr.x (mgr.24542) 422 : cluster [DBG] 2021-07-31T01:57:11.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:10 smithi045 conmon[39718]: pgmap v859: 25 pgs: 25 active+clean; 595 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 3.1 MiB/s wr, 20 op/s 2021-07-31T01:57:11.663 INFO:tasks.rados.rados.0.smithi045.stdout:3464: finishing rollback tid 1 to smithi04569269-30 2021-07-31T01:57:11.663 INFO:tasks.rados.rados.0.smithi045.stdout:3464: finishing rollback tid 2 to smithi04569269-30 2021-07-31T01:57:11.664 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 30 v 1666 (ObjNum 936 snap 284 seq_num 936) dirty exists 2021-07-31T01:57:11.664 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 32 v 1859 (ObjNum 996 snap 311 seq_num 996) dirty exists 2021-07-31T01:57:11.664 INFO:tasks.rados.rados.0.smithi045.stdout:3463: done (2 left) 2021-07-31T01:57:11.664 INFO:tasks.rados.rados.0.smithi045.stdout:3464: done (1 left) 2021-07-31T01:57:11.665 INFO:tasks.rados.rados.0.smithi045.stdout:3465: done (0 left) 2021-07-31T01:57:11.665 INFO:tasks.rados.rados.0.smithi045.stdout:3466: rmattr oid 22 current snap is 344 2021-07-31T01:57:11.665 INFO:tasks.rados.rados.0.smithi045.stdout:3466: done (0 left) 2021-07-31T01:57:11.665 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 28 current snap is 344 2021-07-31T01:57:11.666 INFO:tasks.rados.rados.0.smithi045.stdout:3467: seq_num 1064 ranges {942080=622592,1564672=401408} 2021-07-31T01:57:11.672 INFO:tasks.rados.rados.0.smithi045.stdout:3467: writing smithi04569269-28 from 942080 to 1564672 tid 1 2021-07-31T01:57:11.675 INFO:tasks.rados.rados.0.smithi045.stdout:3467: writing smithi04569269-28 from 1564672 to 1966080 tid 2 2021-07-31T01:57:11.676 INFO:tasks.rados.rados.0.smithi045.stdout:3468: rmattr oid 41 current snap is 344 2021-07-31T01:57:11.676 INFO:tasks.rados.rados.0.smithi045.stdout:3469: read oid 24 snap 329 2021-07-31T01:57:11.676 INFO:tasks.rados.rados.0.smithi045.stdout:3469: expect (ObjNum -1104806576 snap 21873 seq_num 3189982256) 2021-07-31T01:57:11.676 INFO:tasks.rados.rados.0.smithi045.stdout:3470: copy_from oid 37 from oid 1 current snap is 344 2021-07-31T01:57:11.676 INFO:tasks.rados.rados.0.smithi045.stdout:3471: read oid 46 snap -1 2021-07-31T01:57:11.677 INFO:tasks.rados.rados.0.smithi045.stdout:3471: expect (ObjNum 1041 snap 338 seq_num 1041) 2021-07-31T01:57:11.677 INFO:tasks.rados.rados.0.smithi045.stdout:3472: snap_remove snap 289 2021-07-31T01:57:12.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:11 smithi154 conmon[34553]: cluster 2021-07-31T01:57:10.687960+0000 mon.a ( 2021-07-31T01:57:12.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:11 smithi154 conmon[34553]: mon.0) 1381 : cluster [DBG] osdmap e715: 8 total, 8 up, 8 in 2021-07-31T01:57:12.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:11 smithi154 conmon[34553]: cluster 2021-07-31T01:57:11.661960+0000 mon.a (mon.0) 1382 : cluster [DBG] osdmap e716: 8 total, 8 up, 8 in 2021-07-31T01:57:12.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:11 smithi045 conmon[39718]: cluster 2021-07-31T01:57:10.687960 2021-07-31T01:57:12.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:11 smithi045 conmon[39718]: +0000 mon.a (mon.0) 1381 : cluster [DBG] osdmap e715: 8 total, 8 up, 8 in 2021-07-31T01:57:12.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:11 smithi045 conmon[39718]: cluster 2021-07-31T 2021-07-31T01:57:12.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:11 smithi045 conmon[39718]: 01:57:11.661960+0000 mon.a (mon.0) 1382 : cluster [DBG] osdmap e716: 8 total, 8 up, 8 in 2021-07-31T01:57:12.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:11 smithi045 conmon[33655]: cluster 2021-07-31T01: 2021-07-31T01:57:12.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:11 smithi045 conmon[33655]: 57:10.687960+0000 mon.a (mon.0) 1381 : cluster [DBG] osdmap e715: 8 total, 8 up, 8 in 2021-07-31T01:57:12.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:11 smithi045 conmon[33655]: cluster 2021-07-31T01:57:11.661960+0000 mon.a (mon.0) 1382 : cluster [DBG] osdmap e716: 8 total, 8 up, 8 in 2021-07-31T01:57:12.683 INFO:tasks.rados.rados.0.smithi045.stdout:3472: done (5 left) 2021-07-31T01:57:12.684 INFO:tasks.rados.rados.0.smithi045.stdout:3473: rollback oid 48 current snap is 344 2021-07-31T01:57:12.684 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 48 to 338 2021-07-31T01:57:12.684 INFO:tasks.rados.rados.0.smithi045.stdout:3469: expect (ObjNum -1104806576 snap 21873 seq_num 3189982256) 2021-07-31T01:57:12.686 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 41 v 1691 (ObjNum 948 snap 286 seq_num 948) dirty exists 2021-07-31T01:57:12.686 INFO:tasks.rados.rados.0.smithi045.stdout:3467: finishing write tid 1 to smithi04569269-28 2021-07-31T01:57:12.686 INFO:tasks.rados.rados.0.smithi045.stdout:3467: finishing write tid 2 to smithi04569269-28 2021-07-31T01:57:12.687 INFO:tasks.rados.rados.0.smithi045.stdout:3467: finishing write tid 3 to smithi04569269-28 2021-07-31T01:57:12.687 INFO:tasks.rados.rados.0.smithi045.stdout:3467: finishing write tid 4 to smithi04569269-28 2021-07-31T01:57:12.687 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 28 v 2036 (ObjNum 1064 snap 344 seq_num 1064) dirty exists 2021-07-31T01:57:12.687 INFO:tasks.rados.rados.0.smithi045.stdout:3467: left oid 28 (ObjNum 1064 snap 344 seq_num 1064) 2021-07-31T01:57:12.687 INFO:tasks.rados.rados.0.smithi045.stdout:3470: finishing copy_from to smithi04569269-37 2021-07-31T01:57:12.688 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 37 v 1858 (ObjNum 1045 snap 340 seq_num 1045) dirty exists 2021-07-31T01:57:12.688 INFO:tasks.rados.rados.0.smithi045.stdout:3470: finishing copy_from racing read to smithi04569269-37 2021-07-31T01:57:12.688 INFO:tasks.rados.rados.0.smithi045.stdout:3471: expect (ObjNum 1041 snap 338 seq_num 1041) 2021-07-31T01:57:12.953 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T01:57:13.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:12 smithi154 conmon[34553]: cluster 2021-07-31T01:57:11.934960+0000 mgr.x (mgr.24542) 423 : cluster 2021-07-31T01:57:13.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:12 smithi154 conmon[34553]: [DBG] pgmap v862: 25 pgs: 25 active+clean; 595 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 3.1 MiB/s wr, 20 op/s 2021-07-31T01:57:13.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:12 smithi154 conmon[34553]: cluster 2021-07-31T01:57:12.681791+0000 mon.a (mon.0) 1383 : cluster [DBG] osdmap e717: 8 total, 8 up, 8 in 2021-07-31T01:57:13.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:12 smithi045 conmon[33655]: cluster 2021-07-31T01:57:11.934960+0000 mgr.x (mgr 2021-07-31T01:57:13.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:12 smithi045 conmon[33655]: .24542) 423 : cluster [DBG] pgmap v862: 25 pgs: 25 active+clean; 595 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 3.1 MiB/s wr, 20 op/s 2021-07-31T01:57:13.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:12 smithi045 conmon[33655]: cluster 2021-07-31T01:57:12.681791+0000 mon.a (mon.0) 1383 : cluster [DBG] osdmap e717: 8 total, 8 up, 8 in 2021-07-31T01:57:13.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:12 smithi045 conmon[39718]: cluster 2021-07-31T01:57:13.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:12 smithi045 conmon[39718]: 2021-07-31T01:57:13.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:12 smithi045 conmon[39718]: 2021 2021-07-31T01:57:13.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:12 smithi045 conmon[39718]: - 2021-07-31T01:57:13.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:12 smithi045 conmon[39718]: 07 2021-07-31T01:57:13.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:12 smithi045 conmon[39718]: - 2021-07-31T01:57:13.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:12 smithi045 conmon[39718]: 31 2021-07-31T01:57:13.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:12 smithi045 conmon[39718]: T 2021-07-31T01:57:13.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:12 smithi045 conmon[39718]: 01 2021-07-31T01:57:13.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:12 smithi045 conmon[39718]: : 2021-07-31T01:57:13.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:12 smithi045 conmon[39718]: 57 2021-07-31T01:57:13.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:12 smithi045 conmon[39718]: : 2021-07-31T01:57:13.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:12 smithi045 conmon[39718]: 11 2021-07-31T01:57:13.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:12 smithi045 conmon[39718]: . 2021-07-31T01:57:13.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:12 smithi045 conmon[39718]: 934960 2021-07-31T01:57:13.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:12 smithi045 conmon[39718]: +0000 2021-07-31T01:57:13.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:12 smithi045 conmon[39718]: 2021-07-31T01:57:13.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:12 smithi045 conmon[39718]: mgr.x 2021-07-31T01:57:13.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:12 smithi045 conmon[39718]: ( 2021-07-31T01:57:13.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:12 smithi045 conmon[39718]: mgr 2021-07-31T01:57:13.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:12 smithi045 conmon[39718]: . 2021-07-31T01:57:13.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:12 smithi045 conmon[39718]: 24542 2021-07-31T01:57:13.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:12 smithi045 conmon[39718]: ) 2021-07-31T01:57:13.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:12 smithi045 conmon[39718]: 423 2021-07-31T01:57:13.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:12 smithi045 conmon[39718]: : 2021-07-31T01:57:13.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:12 smithi045 conmon[39718]: cluster 2021-07-31T01:57:13.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:12 smithi045 conmon[39718]: 2021-07-31T01:57:13.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:12 smithi045 conmon[39718]: [DBG] 2021-07-31T01:57:13.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:12 smithi045 conmon[39718]: 2021-07-31T01:57:13.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:12 smithi045 conmon[39718]: pgmap v862: 25 pgs: 25 active+clean; 595 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 3.1 MiB/s wr, 20 op/s 2021-07-31T01:57:13.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:12 smithi045 conmon[39718]: 2021-07-31T01:57:13.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:12 smithi045 conmon[39718]: cluster 2021-07-31T01:57:13.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:12 smithi045 conmon[39718]: 2021-07-31T01:57:13.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:12 smithi045 conmon[39718]: 2021 2021-07-31T01:57:13.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:12 smithi045 conmon[39718]: - 2021-07-31T01:57:13.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:12 smithi045 conmon[39718]: 07 2021-07-31T01:57:13.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:12 smithi045 conmon[39718]: - 2021-07-31T01:57:13.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:12 smithi045 conmon[39718]: 31 2021-07-31T01:57:13.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:12 smithi045 conmon[39718]: T 2021-07-31T01:57:13.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:12 smithi045 conmon[39718]: 01 2021-07-31T01:57:13.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:12 smithi045 conmon[39718]: : 2021-07-31T01:57:13.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:12 smithi045 conmon[39718]: 57 2021-07-31T01:57:13.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:12 smithi045 conmon[39718]: : 2021-07-31T01:57:13.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:12 smithi045 conmon[39718]: 12 2021-07-31T01:57:13.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:12 smithi045 conmon[39718]: . 2021-07-31T01:57:13.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:12 smithi045 conmon[39718]: 681791 2021-07-31T01:57:13.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:12 smithi045 conmon[39718]: +0000 2021-07-31T01:57:13.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:12 smithi045 conmon[39718]: 2021-07-31T01:57:13.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:12 smithi045 conmon[39718]: mon.a 2021-07-31T01:57:13.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:12 smithi045 conmon[39718]: ( 2021-07-31T01:57:13.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:12 smithi045 conmon[39718]: mon 2021-07-31T01:57:13.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:12 smithi045 conmon[39718]: . 2021-07-31T01:57:13.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:12 smithi045 conmon[39718]: 0 2021-07-31T01:57:13.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:12 smithi045 conmon[39718]: ) 2021-07-31T01:57:13.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:12 smithi045 conmon[39718]: 1383 2021-07-31T01:57:13.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:12 smithi045 conmon[39718]: : 2021-07-31T01:57:13.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:12 smithi045 conmon[39718]: cluster 2021-07-31T01:57:13.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:12 smithi045 conmon[39718]: 2021-07-31T01:57:13.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:12 smithi045 conmon[39718]: [DBG] 2021-07-31T01:57:13.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:12 smithi045 conmon[39718]: 2021-07-31T01:57:13.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:12 smithi045 conmon[39718]: osdmap e717: 8 total, 8 up, 8 in 2021-07-31T01:57:13.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:12 smithi045 conmon[39718]: 2021-07-31T01:57:13.047 INFO:tasks.rados.rados.0.smithi045.stdout:3467: done (5 left) 2021-07-31T01:57:13.047 INFO:tasks.rados.rados.0.smithi045.stdout:3468: done (4 left) 2021-07-31T01:57:13.048 INFO:tasks.rados.rados.0.smithi045.stdout:3469: done (3 left) 2021-07-31T01:57:13.048 INFO:tasks.rados.rados.0.smithi045.stdout:3470: done (2 left) 2021-07-31T01:57:13.048 INFO:tasks.rados.rados.0.smithi045.stdout:3471: done (1 left) 2021-07-31T01:57:13.049 INFO:tasks.rados.rados.0.smithi045.stdout:3474: rollback oid 21 current snap is 344 2021-07-31T01:57:13.049 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 21 to 330 2021-07-31T01:57:13.049 INFO:tasks.rados.rados.0.smithi045.stdout:3473: finishing rollback tid 1 to smithi04569269-48 2021-07-31T01:57:13.049 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 48 v 1853 (ObjNum -1104207232 snap 21873 seq_num 3189797584) dirty dne 2021-07-31T01:57:13.050 INFO:tasks.rados.rados.0.smithi045.stdout:3473: done (1 left) 2021-07-31T01:57:13.050 INFO:tasks.rados.rados.0.smithi045.stdout:3475: copy_from oid 17 from oid 49 current snap is 344 2021-07-31T01:57:13.052 INFO:tasks.rados.rados.0.smithi045.stdout:3476: setattr oid 33 current snap is 344 2021-07-31T01:57:13.053 INFO:tasks.rados.rados.0.smithi045.stdout:3477: rollback oid 28 current snap is 344 2021-07-31T01:57:13.053 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 28 to 339 2021-07-31T01:57:13.054 INFO:tasks.rados.rados.0.smithi045.stdout:3478: read oid 15 snap -1 2021-07-31T01:57:13.054 INFO:tasks.rados.rados.0.smithi045.stdout:3478: expect (ObjNum 1029 snap 331 seq_num 1029) 2021-07-31T01:57:13.054 INFO:tasks.rados.rados.0.smithi045.stdout:3479: setattr oid 1 current snap is 344 2021-07-31T01:57:13.055 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 19 current snap is 344 2021-07-31T01:57:13.056 INFO:tasks.rados.rados.0.smithi045.stdout:3480: seq_num 1065 ranges {811008=638976,1449984=401408} 2021-07-31T01:57:13.057 INFO:tasks.rados.rados.0.smithi045.stdout:3480: writing smithi04569269-19 from 811008 to 1449984 tid 1 2021-07-31T01:57:13.060 INFO:tasks.rados.rados.0.smithi045.stdout:3480: writing smithi04569269-19 from 1449984 to 1851392 tid 2 2021-07-31T01:57:13.061 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 29 current snap is 344 2021-07-31T01:57:13.062 INFO:tasks.rados.rados.0.smithi045.stdout:3475: finishing copy_from to smithi04569269-17 2021-07-31T01:57:13.062 INFO:tasks.rados.rados.0.smithi045.stdout:3475: got expected ENOENT (src dne) 2021-07-31T01:57:13.063 INFO:tasks.rados.rados.0.smithi045.stdout:3481: seq_num 1066 ranges {0=663552,663552=401408} 2021-07-31T01:57:13.063 INFO:tasks.rados.rados.0.smithi045.stdout:3481: writing smithi04569269-29 from 0 to 663552 tid 1 2021-07-31T01:57:13.066 INFO:tasks.rados.rados.0.smithi045.stdout:3481: writing smithi04569269-29 from 663552 to 1064960 tid 2 2021-07-31T01:57:13.067 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 40 current snap is 344 2021-07-31T01:57:13.067 INFO:tasks.rados.rados.0.smithi045.stdout:3475: finishing copy_from racing read to smithi04569269-17 2021-07-31T01:57:13.067 INFO:tasks.rados.rados.0.smithi045.stdout:3474: finishing rollback tid 0 to smithi04569269-21 2021-07-31T01:57:13.068 INFO:tasks.rados.rados.0.smithi045.stdout:3482: seq_num 1067 ranges {974848=679936,1654784=401408} 2021-07-31T01:57:13.075 INFO:tasks.rados.rados.0.smithi045.stdout:3482: writing smithi04569269-40 from 974848 to 1654784 tid 1 2021-07-31T01:57:13.078 INFO:tasks.rados.rados.0.smithi045.stdout:3482: writing smithi04569269-40 from 1654784 to 2056192 tid 2 2021-07-31T01:57:13.079 INFO:tasks.rados.rados.0.smithi045.stdout:3475: done (8 left) 2021-07-31T01:57:13.079 INFO:tasks.rados.rados.0.smithi045.stdout:3483: read oid 18 snap -1 2021-07-31T01:57:13.080 INFO:tasks.rados.rados.0.smithi045.stdout:3483: expect deleted 2021-07-31T01:57:13.081 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 26 current snap is 344 2021-07-31T01:57:13.081 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 33 v 2036 (ObjNum 988 snap 307 seq_num 988) dirty exists 2021-07-31T01:57:13.082 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 1 v 1862 (ObjNum 1045 snap 340 seq_num 1045) dirty exists 2021-07-31T01:57:13.082 INFO:tasks.rados.rados.0.smithi045.stdout:3484: seq_num 1068 ranges {843776=696320,1540096=401408} 2021-07-31T01:57:13.086 INFO:tasks.rados.rados.0.smithi045.stdout:3484: writing smithi04569269-26 from 843776 to 1540096 tid 1 2021-07-31T01:57:13.088 INFO:tasks.rados.rados.0.smithi045.stdout:3484: writing smithi04569269-26 from 1540096 to 1941504 tid 2 2021-07-31T01:57:13.088 INFO:tasks.rados.rados.0.smithi045.stdout:3476: done (9 left) 2021-07-31T01:57:13.089 INFO:tasks.rados.rados.0.smithi045.stdout:3479: done (8 left) 2021-07-31T01:57:13.089 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 49 current snap is 344 2021-07-31T01:57:13.089 INFO:tasks.rados.rados.0.smithi045.stdout:3474: finishing rollback tid 1 to smithi04569269-21 2021-07-31T01:57:13.089 INFO:tasks.rados.rados.0.smithi045.stdout:3478: expect (ObjNum 1029 snap 331 seq_num 1029) 2021-07-31T01:57:13.120 INFO:tasks.rados.rados.0.smithi045.stdout:3474: finishing rollback tid 2 to smithi04569269-21 2021-07-31T01:57:13.120 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 21 v 2326 (ObjNum 1013 snap 323 seq_num 1013) dirty exists 2021-07-31T01:57:13.121 INFO:tasks.rados.rados.0.smithi045.stdout:3477: finishing rollback tid 0 to smithi04569269-28 2021-07-31T01:57:13.121 INFO:tasks.rados.rados.0.smithi045.stdout:3477: finishing rollback tid 1 to smithi04569269-28 2021-07-31T01:57:13.121 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 28 v 2038 (ObjNum 1 snap 32731 seq_num 3190296000) dirty dne 2021-07-31T01:57:13.121 INFO:tasks.rados.rados.0.smithi045.stdout:3485: seq_num 1069 ranges {0=712704,712704=401408} 2021-07-31T01:57:13.123 INFO:tasks.rados.rados.0.smithi045.stdout:3485: writing smithi04569269-49 from 0 to 712704 tid 1 2021-07-31T01:57:13.126 INFO:tasks.rados.rados.0.smithi045.stdout:3485: writing smithi04569269-49 from 712704 to 1114112 tid 2 2021-07-31T01:57:13.126 INFO:tasks.rados.rados.0.smithi045.stdout:3474: done (8 left) 2021-07-31T01:57:13.127 INFO:tasks.rados.rados.0.smithi045.stdout:3477: done (7 left) 2021-07-31T01:57:13.127 INFO:tasks.rados.rados.0.smithi045.stdout:3478: done (6 left) 2021-07-31T01:57:13.127 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 17 current snap is 344 2021-07-31T01:57:13.128 INFO:tasks.rados.rados.0.smithi045.stdout:3480: finishing write tid 1 to smithi04569269-19 2021-07-31T01:57:13.128 INFO:tasks.rados.rados.0.smithi045.stdout:3486: seq_num 1070 ranges {5857280=729088,6586368=401408} 2021-07-31T01:57:13.162 INFO:tasks.rados.rados.0.smithi045.stdout:3486: writing smithi04569269-17 from 5857280 to 6586368 tid 1 2021-07-31T01:57:13.165 INFO:tasks.rados.rados.0.smithi045.stdout:3486: writing smithi04569269-17 from 6586368 to 6987776 tid 2 2021-07-31T01:57:13.165 INFO:tasks.rados.rados.0.smithi045.stdout:3487: rmattr oid 9 current snap is 344 2021-07-31T01:57:13.165 INFO:tasks.rados.rados.0.smithi045.stdout:3480: finishing write tid 2 to smithi04569269-19 2021-07-31T01:57:13.166 INFO:tasks.rados.rados.0.smithi045.stdout:3480: finishing write tid 3 to smithi04569269-19 2021-07-31T01:57:13.166 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 10 current snap is 344 2021-07-31T01:57:13.166 INFO:tasks.rados.rados.0.smithi045.stdout:3480: finishing write tid 4 to smithi04569269-19 2021-07-31T01:57:13.166 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 19 v 1847 (ObjNum 1065 snap 344 seq_num 1065) dirty exists 2021-07-31T01:57:13.166 INFO:tasks.rados.rados.0.smithi045.stdout:3480: left oid 19 (ObjNum 1065 snap 344 seq_num 1065) 2021-07-31T01:57:13.167 INFO:tasks.rados.rados.0.smithi045.stdout:3481: finishing write tid 1 to smithi04569269-29 2021-07-31T01:57:13.167 INFO:tasks.rados.rados.0.smithi045.stdout:3481: finishing write tid 2 to smithi04569269-29 2021-07-31T01:57:13.167 INFO:tasks.rados.rados.0.smithi045.stdout:3488: seq_num 1071 ranges {9011200=745472,9756672=401408} 2021-07-31T01:57:13.218 INFO:tasks.rados.rados.0.smithi045.stdout:3488: writing smithi04569269-10 from 9011200 to 9756672 tid 1 2021-07-31T01:57:13.222 INFO:tasks.rados.rados.0.smithi045.stdout:3488: writing smithi04569269-10 from 9756672 to 10158080 tid 2 2021-07-31T01:57:13.222 INFO:tasks.rados.rados.0.smithi045.stdout:3480: done (8 left) 2021-07-31T01:57:13.222 INFO:tasks.rados.rados.0.smithi045.stdout:3489: snap_create 2021-07-31T01:57:13.223 INFO:tasks.rados.rados.0.smithi045.stdout:3481: finishing write tid 3 to smithi04569269-29 2021-07-31T01:57:13.223 INFO:tasks.rados.rados.0.smithi045.stdout:3481: finishing write tid 4 to smithi04569269-29 2021-07-31T01:57:13.223 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 29 v 1428 (ObjNum 1066 snap 344 seq_num 1066) dirty exists 2021-07-31T01:57:13.223 INFO:tasks.rados.rados.0.smithi045.stdout:3481: left oid 29 (ObjNum 1066 snap 344 seq_num 1066) 2021-07-31T01:57:13.224 INFO:tasks.rados.rados.0.smithi045.stdout:3482: finishing write tid 1 to smithi04569269-40 2021-07-31T01:57:13.224 INFO:tasks.rados.rados.0.smithi045.stdout:3482: finishing write tid 2 to smithi04569269-40 2021-07-31T01:57:13.224 INFO:tasks.rados.rados.0.smithi045.stdout:3482: finishing write tid 3 to smithi04569269-40 2021-07-31T01:57:13.224 INFO:tasks.rados.rados.0.smithi045.stdout:3482: finishing write tid 4 to smithi04569269-40 2021-07-31T01:57:13.224 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 40 v 1880 (ObjNum 1067 snap 344 seq_num 1067) dirty exists 2021-07-31T01:57:13.225 INFO:tasks.rados.rados.0.smithi045.stdout:3482: left oid 40 (ObjNum 1067 snap 344 seq_num 1067) 2021-07-31T01:57:13.225 INFO:tasks.rados.rados.0.smithi045.stdout:3484: finishing write tid 1 to smithi04569269-26 2021-07-31T01:57:13.225 INFO:tasks.rados.rados.0.smithi045.stdout:3484: finishing write tid 2 to smithi04569269-26 2021-07-31T01:57:13.225 INFO:tasks.rados.rados.0.smithi045.stdout:3484: finishing write tid 3 to smithi04569269-26 2021-07-31T01:57:13.225 INFO:tasks.rados.rados.0.smithi045.stdout:3484: finishing write tid 4 to smithi04569269-26 2021-07-31T01:57:13.226 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 26 v 1862 (ObjNum 1068 snap 344 seq_num 1068) dirty exists 2021-07-31T01:57:13.226 INFO:tasks.rados.rados.0.smithi045.stdout:3484: left oid 26 (ObjNum 1068 snap 344 seq_num 1068) 2021-07-31T01:57:13.226 INFO:tasks.rados.rados.0.smithi045.stdout:3485: finishing write tid 1 to smithi04569269-49 2021-07-31T01:57:13.226 INFO:tasks.rados.rados.0.smithi045.stdout:3485: finishing write tid 2 to smithi04569269-49 2021-07-31T01:57:13.227 INFO:tasks.rados.rados.0.smithi045.stdout:3485: finishing write tid 3 to smithi04569269-49 2021-07-31T01:57:13.229 INFO:tasks.rados.rados.0.smithi045.stdout:3485: finishing write tid 4 to smithi04569269-49 2021-07-31T01:57:13.229 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 49 v 2329 (ObjNum 1069 snap 344 seq_num 1069) dirty exists 2021-07-31T01:57:13.229 INFO:tasks.rados.rados.0.smithi045.stdout:3485: left oid 49 (ObjNum 1069 snap 344 seq_num 1069) 2021-07-31T01:57:13.230 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 9 v 1850 (ObjNum 1056 snap 343 seq_num 1056) dirty exists 2021-07-31T01:57:13.231 INFO:tasks.rados.rados.0.smithi045.stdout:3486: finishing write tid 1 to smithi04569269-17 2021-07-31T01:57:13.231 INFO:tasks.rados.rados.0.smithi045.stdout:3486: finishing write tid 2 to smithi04569269-17 2021-07-31T01:57:13.231 INFO:tasks.rados.rados.0.smithi045.stdout:3486: finishing write tid 3 to smithi04569269-17 2021-07-31T01:57:13.231 INFO:tasks.rados.rados.0.smithi045.stdout:3486: finishing write tid 4 to smithi04569269-17 2021-07-31T01:57:13.232 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 17 v 2041 (ObjNum 1070 snap 344 seq_num 1070) dirty exists 2021-07-31T01:57:13.232 INFO:tasks.rados.rados.0.smithi045.stdout:3486: left oid 17 (ObjNum 1070 snap 344 seq_num 1070) 2021-07-31T01:57:13.233 INFO:tasks.rados.rados.0.smithi045.stdout:3488: finishing write tid 1 to smithi04569269-10 2021-07-31T01:57:13.233 INFO:tasks.rados.rados.0.smithi045.stdout:3488: finishing write tid 2 to smithi04569269-10 2021-07-31T01:57:13.233 INFO:tasks.rados.rados.0.smithi045.stdout:3488: finishing write tid 3 to smithi04569269-10 2021-07-31T01:57:13.234 INFO:tasks.rados.rados.0.smithi045.stdout:3488: finishing write tid 4 to smithi04569269-10 2021-07-31T01:57:13.234 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 10 v 2044 (ObjNum 1071 snap 344 seq_num 1071) dirty exists 2021-07-31T01:57:13.234 INFO:tasks.rados.rados.0.smithi045.stdout:3488: left oid 10 (ObjNum 1071 snap 344 seq_num 1071) 2021-07-31T01:57:13.252 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T01:57:13.252 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 12m ago 12m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T01:57:13.252 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 9m ago 12m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T01:57:13.253 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 9m ago 16m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T01:57:13.253 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 12m ago 18m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T01:57:13.253 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 12m ago 18m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T01:57:13.254 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 9m ago 16m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T01:57:13.255 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 12m ago 16m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T01:57:13.255 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 12m ago 13m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T01:57:13.255 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 9m ago 13m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T01:57:13.255 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 12m ago 15m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T01:57:13.256 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 12m ago 15m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T01:57:13.256 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 12m ago 15m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T01:57:13.256 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 12m ago 14m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T01:57:13.256 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 9m ago 14m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T01:57:13.257 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 9m ago 14m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T01:57:13.257 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 9m ago 13m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T01:57:13.257 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 9m ago 13m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T01:57:13.258 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 9m ago 13m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T01:57:13.589 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T01:57:13.589 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T01:57:13.589 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T01:57:13.590 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T01:57:13.590 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T01:57:13.590 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T01:57:13.590 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T01:57:13.591 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T01:57:13.591 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T01:57:13.591 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T01:57:13.591 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T01:57:13.591 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T01:57:13.592 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T01:57:13.592 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T01:57:13.592 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T01:57:13.592 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T01:57:13.592 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T01:57:13.701 INFO:tasks.rados.rados.0.smithi045.stdout:3481: done (8 left) 2021-07-31T01:57:13.701 INFO:tasks.rados.rados.0.smithi045.stdout:3482: done (7 left) 2021-07-31T01:57:13.701 INFO:tasks.rados.rados.0.smithi045.stdout:3483: done (6 left) 2021-07-31T01:57:13.702 INFO:tasks.rados.rados.0.smithi045.stdout:3484: done (5 left) 2021-07-31T01:57:13.702 INFO:tasks.rados.rados.0.smithi045.stdout:3485: done (4 left) 2021-07-31T01:57:13.702 INFO:tasks.rados.rados.0.smithi045.stdout:3486: done (3 left) 2021-07-31T01:57:13.702 INFO:tasks.rados.rados.0.smithi045.stdout:3487: done (2 left) 2021-07-31T01:57:13.702 INFO:tasks.rados.rados.0.smithi045.stdout:3488: done (1 left) 2021-07-31T01:57:13.703 INFO:tasks.rados.rados.0.smithi045.stdout:3489: done (0 left) 2021-07-31T01:57:13.703 INFO:tasks.rados.rados.0.smithi045.stdout:3490: snap_remove snap 328 2021-07-31T01:57:14.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:13 smithi154 conmon[34553]: audit 2021-07-31T01:57:12.941318+0000 mgr.x (mgr.24542) 424 : audit 2021-07-31T01:57:14.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:13 smithi154 conmon[34553]: [DBG] from='client.15054 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:57:14.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:13 smithi154 conmon[34553]: audit 2021-07-31T01:57:13.588903+0000 mon.a (mon.0) 1384 : audit [DBG] from='client.? 172.21.15.45:0/1722413168' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T01:57:14.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:57:13 smithi045 conmon[61210]: level=error ts=2021-07-31T01:57:13.526Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:57:14.027 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:57:13 smithi045 conmon[61210]: level=error ts=2021-07-31T01:57:13.526Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:57:14.027 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:57:13 smithi045 conmon[61210]: level=error ts=2021-07-31T01:57:13.526Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T01:57:14.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:13 smithi045 conmon[39718]: audit 2021-07-31T01:57:12.941318 2021-07-31T01:57:14.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:13 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 424 : audit [DBG] from='client.15054 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:57:14.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:13 smithi045 conmon[39718]: audit 2021-07 2021-07-31T01:57:14.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:13 smithi045 conmon[39718]: -31T01:57:13.588903+0000 mon.a (mon. 2021-07-31T01:57:14.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:13 smithi045 conmon[39718]: 0) 1384 : audit [DBG] from='client.? 172.21.15.45:0/1722413168' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T01:57:14.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:13 smithi045 conmon[33655]: audit 2021-07-31T01:57:12.941318+0000 2021-07-31T01:57:14.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:13 smithi045 conmon[33655]: mgr.x (mgr.24542) 424 : audit [DBG] from='client.15054 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:57:14.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:13 smithi045 conmon[33655]: audit 2021-07-31T01:57:13 2021-07-31T01:57:14.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:13 smithi045 conmon[33655]: .588903+0000 mon.a (mon.0) 1384 : audit [DBG] from='client.? 172.21.15.45:0/1722413168' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T01:57:14.714 INFO:tasks.rados.rados.0.smithi045.stdout:3490: done (0 left) 2021-07-31T01:57:14.714 INFO:tasks.rados.rados.0.smithi045.stdout:3491: copy_from oid 49 from oid 2 current snap is 345 2021-07-31T01:57:14.714 INFO:tasks.rados.rados.0.smithi045.stdout:3492: read oid 18 snap -1 2021-07-31T01:57:14.714 INFO:tasks.rados.rados.0.smithi045.stdout:3492: expect deleted 2021-07-31T01:57:14.715 INFO:tasks.rados.rados.0.smithi045.stdout:3493: copy_from oid 12 from oid 4 current snap is 345 2021-07-31T01:57:14.715 INFO:tasks.rados.rados.0.smithi045.stdout:3494: rollback oid 25 current snap is 345 2021-07-31T01:57:14.716 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 25 to 342 2021-07-31T01:57:14.716 INFO:tasks.rados.rados.0.smithi045.stdout:3495: snap_create 2021-07-31T01:57:14.719 INFO:tasks.rados.rados.0.smithi045.stdout:3494: finishing rollback tid 0 to smithi04569269-25 2021-07-31T01:57:14.722 INFO:tasks.rados.rados.0.smithi045.stdout:3494: finishing rollback tid 1 to smithi04569269-25 2021-07-31T01:57:14.722 INFO:tasks.rados.rados.0.smithi045.stdout:3494: finishing rollback tid 2 to smithi04569269-25 2021-07-31T01:57:14.723 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 25 v 1845 (ObjNum 1031 snap 335 seq_num 1031) dirty exists 2021-07-31T01:57:14.729 INFO:tasks.rados.rados.0.smithi045.stdout:3491: finishing copy_from to smithi04569269-49 2021-07-31T01:57:14.730 INFO:tasks.rados.rados.0.smithi045.stdout:3491: got expected ENOENT (src dne) 2021-07-31T01:57:14.730 INFO:tasks.rados.rados.0.smithi045.stdout:3491: finishing copy_from racing read to smithi04569269-49 2021-07-31T01:57:14.736 INFO:tasks.rados.rados.0.smithi045.stdout:3493: finishing copy_from to smithi04569269-12 2021-07-31T01:57:14.736 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 12 v 1873 (ObjNum 1036 snap 338 seq_num 1036) dirty exists 2021-07-31T01:57:14.737 INFO:tasks.rados.rados.0.smithi045.stdout:3493: finishing copy_from racing read to smithi04569269-12 2021-07-31T01:57:15.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:14 smithi154 conmon[34553]: audit 2021-07-31T01:57:13.244332+0000 mgr.x (mgr.24542) 425 2021-07-31T01:57:15.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:14 smithi154 conmon[34553]: : audit [DBG] from='client.15060 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:57:15.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:14 smithi154 conmon[34553]: cluster 2021-07-31T01:57:13.700660+0000 mon.a (mon.0) 1385 : cluster [DBG] osdmap e718: 8 total, 8 up, 8 in 2021-07-31T01:57:15.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:14 smithi154 conmon[34553]: cluster 2021-07-31T01:57:13.935451+0000 mgr.x (mgr.24542) 426 : cluster [DBG] pgmap v865: 25 pgs: 25 active+clean; 607 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 7.0 MiB/s rd, 3.9 MiB/s wr, 36 op/s 2021-07-31T01:57:15.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:14 smithi045 conmon[39718]: audit 2021-07-31T01:57:13.244332 2021-07-31T01:57:15.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:14 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 425 : audit [DBG] from='client.15060 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:57:15.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:14 smithi045 conmon[39718]: cluster 2021-07-31T01:57:15.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:14 smithi045 conmon[39718]: 2021-07 2021-07-31T01:57:15.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:14 smithi045 conmon[39718]: -31T01:57:13. 2021-07-31T01:57:15.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:14 smithi045 conmon[39718]: 700660+0000 mon.a (mon.0 2021-07-31T01:57:15.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:14 smithi045 conmon[39718]: ) 1385 : cluster [DBG] osdmap e718: 8 total, 8 up, 8 in 2021-07-31T01:57:15.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:14 smithi045 conmon[39718]: 2021-07-31T01:57:15.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:14 smithi045 conmon[39718]: cluster 2021-07 2021-07-31T01:57:15.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:14 smithi045 conmon[39718]: -31T01:57:13.935451+0000 2021-07-31T01:57:15.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:14 smithi045 conmon[39718]: mgr.x (mgr.24542) 426 2021-07-31T01:57:15.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:14 smithi045 conmon[39718]: : cluster [DBG] pgmap v865: 25 pgs: 25 active+clean; 607 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 7.0 MiB/s rd, 3.9 MiB/s wr, 36 op/s 2021-07-31T01:57:15.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:14 smithi045 conmon[33655]: audit 2021-07-31T01:57:13.244332+0000 mgr.x (mgr.24542) 425 : audit [DBG] 2021-07-31T01:57:15.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:14 smithi045 conmon[33655]: from='client.15060 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:57:15.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:14 smithi045 conmon[33655]: cluster 2021-07-31T01:57:13.700660+0000 mon.a (mon.0) 1385 : cluster [DBG] osdmap e718: 8 total, 8 up, 8 in 2021-07-31T01:57:15.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:14 smithi045 conmon[33655]: cluster 2021-07-31T01:57:13.935451+0000 mgr.x (mgr.24542) 426 : cluster [DBG] pgmap v865: 25 pgs: 25 active+clean; 607 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 7.0 MiB/s rd, 3.9 MiB/s wr, 36 op/s 2021-07-31T01:57:15.716 INFO:tasks.rados.rados.0.smithi045.stdout:3491: done (4 left) 2021-07-31T01:57:15.716 INFO:tasks.rados.rados.0.smithi045.stdout:3492: done (3 left) 2021-07-31T01:57:15.717 INFO:tasks.rados.rados.0.smithi045.stdout:3493: done (2 left) 2021-07-31T01:57:15.717 INFO:tasks.rados.rados.0.smithi045.stdout:3494: done (1 left) 2021-07-31T01:57:15.718 INFO:tasks.rados.rados.0.smithi045.stdout:3495: done (0 left) 2021-07-31T01:57:15.718 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 32 current snap is 346 2021-07-31T01:57:15.719 INFO:tasks.rados.rados.0.smithi045.stdout:3496: seq_num 1072 ranges {2293760=761856,3055616=401408} 2021-07-31T01:57:15.730 INFO:tasks.rados.rados.0.smithi045.stdout:3496: writing smithi04569269-32 from 2293760 to 3055616 tid 1 2021-07-31T01:57:15.732 INFO:tasks.rados.rados.0.smithi045.stdout:3496: writing smithi04569269-32 from 3055616 to 3457024 tid 2 2021-07-31T01:57:15.733 INFO:tasks.rados.rados.0.smithi045.stdout:3497: snap_remove snap 324 2021-07-31T01:57:16.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:15 smithi154 conmon[34553]: cluster 2021-07-31T01:57:14.711267+0000 mon.a (mon.0) 1386 : cluster [DBG] osdmap e719: 8 total, 8 up, 8 in 2021-07-31T01:57:16.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:15 smithi045 conmon[39718]: cluster 2021-07-31T01:57:14.711267+0000 mon.a ( 2021-07-31T01:57:16.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:15 smithi045 conmon[39718]: mon.0) 1386 : cluster [DBG] osdmap e719: 8 total, 8 up, 8 in 2021-07-31T01:57:16.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:15 smithi045 conmon[33655]: cluster 2021-07-31T01:57:14.711267+0000 2021-07-31T01:57:16.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:15 smithi045 conmon[33655]: mon.a (mon.0) 1386 : cluster [DBG] osdmap e719: 8 total, 8 up, 8 in 2021-07-31T01:57:16.666 INFO:tasks.rados.rados.0.smithi045.stdout:3496: finishing write tid 1 to smithi04569269-32 2021-07-31T01:57:16.666 INFO:tasks.rados.rados.0.smithi045.stdout:3496: finishing write tid 2 to smithi04569269-32 2021-07-31T01:57:16.666 INFO:tasks.rados.rados.0.smithi045.stdout:3496: finishing write tid 3 to smithi04569269-32 2021-07-31T01:57:16.667 INFO:tasks.rados.rados.0.smithi045.stdout:3496: finishing write tid 4 to smithi04569269-32 2021-07-31T01:57:16.667 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 32 v 1865 (ObjNum 1072 snap 346 seq_num 1072) dirty exists 2021-07-31T01:57:16.667 INFO:tasks.rados.rados.0.smithi045.stdout:3496: left oid 32 (ObjNum 1072 snap 346 seq_num 1072) 2021-07-31T01:57:16.668 INFO:tasks.rados.rados.0.smithi045.stdout:3496: done (1 left) 2021-07-31T01:57:16.668 INFO:tasks.rados.rados.0.smithi045.stdout:3497: done (0 left) 2021-07-31T01:57:16.668 INFO:tasks.rados.rados.0.smithi045.stdout:3498: copy_from oid 9 from oid 3 current snap is 346 2021-07-31T01:57:16.668 INFO:tasks.rados.rados.0.smithi045.stdout:3499: read oid 40 snap -1 2021-07-31T01:57:16.669 INFO:tasks.rados.rados.0.smithi045.stdout:3499: expect (ObjNum 1067 snap 344 seq_num 1067) 2021-07-31T01:57:16.669 INFO:tasks.rados.rados.0.smithi045.stdout:3500: read oid 37 snap -1 2021-07-31T01:57:16.669 INFO:tasks.rados.rados.0.smithi045.stdout:3500: expect (ObjNum 1045 snap 340 seq_num 1045) 2021-07-31T01:57:16.669 INFO:tasks.rados.rados.0.smithi045.stdout:3501: copy_from oid 50 from oid 21 current snap is 346 2021-07-31T01:57:16.669 INFO:tasks.rados.rados.0.smithi045.stdout:3502: setattr oid 11 current snap is 346 2021-07-31T01:57:16.670 INFO:tasks.rados.rados.0.smithi045.stdout:3503: delete oid 32 current snap is 346 2021-07-31T01:57:16.680 INFO:tasks.rados.rados.0.smithi045.stdout:3503: done (5 left) 2021-07-31T01:57:16.680 INFO:tasks.rados.rados.0.smithi045.stdout:3504: copy_from oid 47 from oid 16 current snap is 346 2021-07-31T01:57:16.681 INFO:tasks.rados.rados.0.smithi045.stdout:3505: copy_from oid 48 from oid 12 current snap is 346 2021-07-31T01:57:16.681 INFO:tasks.rados.rados.0.smithi045.stdout:3506: delete oid 36 current snap is 346 2021-07-31T01:57:16.685 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 11 v 2037 (ObjNum 948 snap 286 seq_num 948) dirty exists 2021-07-31T01:57:16.688 INFO:tasks.rados.rados.0.smithi045.stdout:3499: expect (ObjNum 1067 snap 344 seq_num 1067) 2021-07-31T01:57:16.771 INFO:tasks.rados.rados.0.smithi045.stdout:3505: finishing copy_from to smithi04569269-48 2021-07-31T01:57:16.772 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 48 v 1873 (ObjNum 1036 snap 338 seq_num 1036) dirty exists 2021-07-31T01:57:16.772 INFO:tasks.rados.rados.0.smithi045.stdout:3505: finishing copy_from racing read to smithi04569269-48 2021-07-31T01:57:16.772 INFO:tasks.rados.rados.0.smithi045.stdout:3500: expect (ObjNum 1045 snap 340 seq_num 1045) 2021-07-31T01:57:16.985 INFO:tasks.rados.rados.0.smithi045.stdout:3504: finishing copy_from to smithi04569269-47 2021-07-31T01:57:16.985 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 47 v 1841 (ObjNum 1031 snap 335 seq_num 1031) dirty exists 2021-07-31T01:57:16.986 INFO:tasks.rados.rados.0.smithi045.stdout:3504: finishing copy_from racing read to smithi04569269-47 2021-07-31T01:57:16.986 INFO:tasks.rados.rados.0.smithi045.stdout:3501: finishing copy_from to smithi04569269-50 2021-07-31T01:57:16.986 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 50 v 2326 (ObjNum 1013 snap 323 seq_num 1013) dirty exists 2021-07-31T01:57:16.986 INFO:tasks.rados.rados.0.smithi045.stdout:3501: finishing copy_from racing read to smithi04569269-50 2021-07-31T01:57:16.987 INFO:tasks.rados.rados.0.smithi045.stdout:3498: finishing copy_from to smithi04569269-9 2021-07-31T01:57:16.987 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 9 v 2163 (ObjNum 881 snap 272 seq_num 881) dirty exists 2021-07-31T01:57:16.987 INFO:tasks.rados.rados.0.smithi045.stdout:3498: finishing copy_from racing read to smithi04569269-9 2021-07-31T01:57:16.987 INFO:tasks.rados.rados.0.smithi045.stdout:3498: done (7 left) 2021-07-31T01:57:16.988 INFO:tasks.rados.rados.0.smithi045.stdout:3499: done (6 left) 2021-07-31T01:57:16.988 INFO:tasks.rados.rados.0.smithi045.stdout:3500: done (5 left) 2021-07-31T01:57:16.988 INFO:tasks.rados.rados.0.smithi045.stdout:3501: done (4 left) 2021-07-31T01:57:16.989 INFO:tasks.rados.rados.0.smithi045.stdout:3502: done (3 left) 2021-07-31T01:57:16.989 INFO:tasks.rados.rados.0.smithi045.stdout:3504: done (2 left) 2021-07-31T01:57:16.989 INFO:tasks.rados.rados.0.smithi045.stdout:3505: done (1 left) 2021-07-31T01:57:16.989 INFO:tasks.rados.rados.0.smithi045.stdout:3506: done (0 left) 2021-07-31T01:57:16.990 INFO:tasks.rados.rados.0.smithi045.stdout:3507: snap_remove snap 312 2021-07-31T01:57:17.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:16 smithi154 conmon[34553]: cluster 2021-07-31T01:57:15.712803+0000 mon.a (mon.0) 1387 2021-07-31T01:57:17.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:16 smithi154 conmon[34553]: : cluster [DBG] osdmap e720: 8 total, 8 up, 8 in 2021-07-31T01:57:17.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:16 smithi154 conmon[34553]: cluster 2021-07-31T01:57:15.935870+0000 mgr.x (mgr.24542) 427 : cluster [DBG] pgmap v868: 25 pgs: 25 active+clean; 607 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 7.0 MiB/s rd, 3.9 MiB/s wr, 36 op/s 2021-07-31T01:57:17.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:16 smithi154 conmon[34553]: cluster 2021- 2021-07-31T01:57:17.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:16 smithi154 conmon[34553]: 07-31T01:57:16.663490+0000 mon.a (mon.0) 1388 : cluster 2021-07-31T01:57:17.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:16 smithi154 conmon[34553]: [DBG] osdmap e721: 8 total, 8 up, 8 in 2021-07-31T01:57:17.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:16 smithi045 conmon[39718]: cluster 2021-07-31T01:57:15. 2021-07-31T01:57:17.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:16 smithi045 conmon[39718]: 712803+0000 mon.a (mon.0) 1387 : cluster [DBG] osdmap e720: 8 total, 8 up, 8 in 2021-07-31T01:57:17.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:16 smithi045 conmon[39718]: cluster 2021-07-31T01:57 2021-07-31T01:57:17.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:16 smithi045 conmon[39718]: :15.935870+0000 mgr.x (mgr.24542) 427 2021-07-31T01:57:17.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:16 smithi045 conmon[39718]: : cluster [DBG] pgmap v868: 25 pgs: 25 active+clean; 607 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 7.0 MiB/s rd, 3.9 MiB/s wr, 36 op/s 2021-07-31T01:57:17.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:16 smithi045 conmon[39718]: cluster 2021-07-31T 2021-07-31T01:57:17.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:16 smithi045 conmon[39718]: 01:57:16.663490+0000 mon.a (mon. 2021-07-31T01:57:17.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:16 smithi045 conmon[39718]: 0) 1388 : cluster [DBG] osdmap e721: 8 total, 8 up, 8 in 2021-07-31T01:57:17.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:16 smithi045 conmon[33655]: cluster 2021-07-31T01:57:17.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:16 smithi045 conmon[33655]: 2021-07-31T01 2021-07-31T01:57:17.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:16 smithi045 conmon[33655]: :57:15.712803+0000 mon.a (mon.0) 1387 : cluster [DBG] osdmap e720: 8 total, 8 up, 8 in 2021-07-31T01:57:17.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:16 smithi045 conmon[33655]: cluster 2021-07-31T01 2021-07-31T01:57:17.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:16 smithi045 conmon[33655]: :57:15.935870+0000 mgr.x (mgr.24542) 427 : 2021-07-31T01:57:17.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:16 smithi045 conmon[33655]: cluster [DBG] pgmap v868: 25 pgs: 25 active+clean; 607 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 7.0 MiB/s rd, 3.9 MiB/s wr, 36 op/s 2021-07-31T01:57:17.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:16 smithi045 conmon[33655]: cluster 2021-07-31T01:57:17.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:16 smithi045 conmon[33655]: 2021-07-31T01:57:16.663490+0000 mon.a (mon.0) 2021-07-31T01:57:17.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:16 smithi045 conmon[33655]: 1388 : cluster [DBG] osdmap e721: 8 total, 8 up, 8 in 2021-07-31T01:57:17.733 INFO:tasks.rados.rados.0.smithi045.stdout:3507: done (0 left) 2021-07-31T01:57:17.734 INFO:tasks.rados.rados.0.smithi045.stdout:3508: setattr oid 36 current snap is 346 2021-07-31T01:57:17.736 INFO:tasks.rados.rados.0.smithi045.stdout:3509: delete oid 33 current snap is 346 2021-07-31T01:57:17.745 INFO:tasks.rados.rados.0.smithi045.stdout:3509: done (1 left) 2021-07-31T01:57:17.745 INFO:tasks.rados.rados.0.smithi045.stdout:3510: read oid 49 snap -1 2021-07-31T01:57:17.745 INFO:tasks.rados.rados.0.smithi045.stdout:3510: expect (ObjNum 1069 snap 344 seq_num 1069) 2021-07-31T01:57:17.746 INFO:tasks.rados.rados.0.smithi045.stdout:3511: rollback oid 21 current snap is 346 2021-07-31T01:57:17.746 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 21 to 345 2021-07-31T01:57:17.747 INFO:tasks.rados.rados.0.smithi045.stdout:3512: read oid 4 snap -1 2021-07-31T01:57:17.747 INFO:tasks.rados.rados.0.smithi045.stdout:3512: expect (ObjNum 1036 snap 338 seq_num 1036) 2021-07-31T01:57:17.747 INFO:tasks.rados.rados.0.smithi045.stdout:3513: read oid 40 snap -1 2021-07-31T01:57:17.747 INFO:tasks.rados.rados.0.smithi045.stdout:3513: expect (ObjNum 1067 snap 344 seq_num 1067) 2021-07-31T01:57:17.748 INFO:tasks.rados.rados.0.smithi045.stdout:3514: delete oid 41 current snap is 346 2021-07-31T01:57:17.748 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 36 v 2347 (ObjNum -1104427824 snap 21873 seq_num 3192290000) dirty exists 2021-07-31T01:57:17.749 INFO:tasks.rados.rados.0.smithi045.stdout:3508: done (5 left) 2021-07-31T01:57:17.749 INFO:tasks.rados.rados.0.smithi045.stdout:3514: done (4 left) 2021-07-31T01:57:17.750 INFO:tasks.rados.rados.0.smithi045.stdout:3515: copy_from oid 2 from oid 16 current snap is 346 2021-07-31T01:57:17.750 INFO:tasks.rados.rados.0.smithi045.stdout:3516: rmattr oid 47 current snap is 346 2021-07-31T01:57:17.750 INFO:tasks.rados.rados.0.smithi045.stdout:3516: done (5 left) 2021-07-31T01:57:17.751 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 28 current snap is 346 2021-07-31T01:57:17.752 INFO:tasks.rados.rados.0.smithi045.stdout:3517: seq_num 1077 ranges {0=442368,442368=434176,876544=368640} 2021-07-31T01:57:17.753 INFO:tasks.rados.rados.0.smithi045.stdout:3517: writing smithi04569269-28 from 0 to 442368 tid 1 2021-07-31T01:57:17.755 INFO:tasks.rados.rados.0.smithi045.stdout:3517: writing smithi04569269-28 from 442368 to 876544 tid 2 2021-07-31T01:57:17.757 INFO:tasks.rados.rados.0.smithi045.stdout:3517: writing smithi04569269-28 from 876544 to 1245184 tid 3 2021-07-31T01:57:17.758 INFO:tasks.rados.rados.0.smithi045.stdout:3518: copy_from oid 50 from oid 3 current snap is 346 2021-07-31T01:57:17.759 INFO:tasks.rados.rados.0.smithi045.stdout:3519: setattr oid 44 current snap is 346 2021-07-31T01:57:17.761 INFO:tasks.rados.rados.0.smithi045.stdout:3520: rollback oid 14 current snap is 346 2021-07-31T01:57:17.761 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 14 to 338 2021-07-31T01:57:17.762 INFO:tasks.rados.rados.0.smithi045.stdout:3521: read oid 18 snap 304 2021-07-31T01:57:17.762 INFO:tasks.rados.rados.0.smithi045.stdout:3521: expect (ObjNum 928 snap 281 seq_num 928) 2021-07-31T01:57:17.762 INFO:tasks.rados.rados.0.smithi045.stdout:3522: read oid 13 snap -1 2021-07-31T01:57:17.763 INFO:tasks.rados.rados.0.smithi045.stdout:3522: expect deleted 2021-07-31T01:57:17.763 INFO:tasks.rados.rados.0.smithi045.stdout:3523: delete oid 42 current snap is 346 2021-07-31T01:57:17.763 INFO:tasks.rados.rados.0.smithi045.stdout:3512: expect (ObjNum 1036 snap 338 seq_num 1036) 2021-07-31T01:57:17.787 INFO:tasks.rados.rados.0.smithi045.stdout:3512: done (11 left) 2021-07-31T01:57:17.787 INFO:tasks.rados.rados.0.smithi045.stdout:3523: done (10 left) 2021-07-31T01:57:17.787 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 26 current snap is 346 2021-07-31T01:57:17.788 INFO:tasks.rados.rados.0.smithi045.stdout:3524: seq_num 1079 ranges {1941504=475136,2416640=606208,3022848=196608} 2021-07-31T01:57:17.800 INFO:tasks.rados.rados.0.smithi045.stdout:3524: writing smithi04569269-26 from 1941504 to 2416640 tid 1 2021-07-31T01:57:17.803 INFO:tasks.rados.rados.0.smithi045.stdout:3524: writing smithi04569269-26 from 2416640 to 3022848 tid 2 2021-07-31T01:57:17.805 INFO:tasks.rados.rados.0.smithi045.stdout:3524: writing smithi04569269-26 from 3022848 to 3219456 tid 3 2021-07-31T01:57:17.805 INFO:tasks.rados.rados.0.smithi045.stdout:3525: rollback oid 25 current snap is 346 2021-07-31T01:57:17.805 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 25 to 344 2021-07-31T01:57:17.806 INFO:tasks.rados.rados.0.smithi045.stdout:3510: expect (ObjNum 1069 snap 344 seq_num 1069) 2021-07-31T01:57:17.839 INFO:tasks.rados.rados.0.smithi045.stdout:3511: finishing rollback tid 0 to smithi04569269-21 2021-07-31T01:57:17.840 INFO:tasks.rados.rados.0.smithi045.stdout:3513: expect (ObjNum 1067 snap 344 seq_num 1067) 2021-07-31T01:57:17.911 INFO:tasks.rados.rados.0.smithi045.stdout:3517: finishing write tid 1 to smithi04569269-28 2021-07-31T01:57:17.911 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 44 v 2040 (ObjNum 1055 snap 343 seq_num 1055) dirty exists 2021-07-31T01:57:17.911 INFO:tasks.rados.rados.0.smithi045.stdout:3520: finishing rollback tid 0 to smithi04569269-14 2021-07-31T01:57:17.912 INFO:tasks.rados.rados.0.smithi045.stdout:3510: done (11 left) 2021-07-31T01:57:17.912 INFO:tasks.rados.rados.0.smithi045.stdout:3513: done (10 left) 2021-07-31T01:57:17.912 INFO:tasks.rados.rados.0.smithi045.stdout:3519: done (9 left) 2021-07-31T01:57:17.912 INFO:tasks.rados.rados.0.smithi045.stdout:3522: done (8 left) 2021-07-31T01:57:17.913 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 43 current snap is 346 2021-07-31T01:57:17.913 INFO:tasks.rados.rados.0.smithi045.stdout:3520: finishing rollback tid 1 to smithi04569269-14 2021-07-31T01:57:17.913 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 14 v 2042 (ObjNum 0 snap 0 seq_num 0) dirty dne 2021-07-31T01:57:17.913 INFO:tasks.rados.rados.0.smithi045.stdout:3515: finishing copy_from to smithi04569269-2 2021-07-31T01:57:17.913 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 2 v 1841 (ObjNum 1031 snap 335 seq_num 1031) dirty exists 2021-07-31T01:57:17.914 INFO:tasks.rados.rados.0.smithi045.stdout:3517: finishing write tid 2 to smithi04569269-28 2021-07-31T01:57:17.914 INFO:tasks.rados.rados.0.smithi045.stdout:3515: finishing copy_from racing read to smithi04569269-2 2021-07-31T01:57:17.914 INFO:tasks.rados.rados.0.smithi045.stdout:3517: finishing write tid 3 to smithi04569269-28 2021-07-31T01:57:17.914 INFO:tasks.rados.rados.0.smithi045.stdout:3517: finishing write tid 4 to smithi04569269-28 2021-07-31T01:57:17.915 INFO:tasks.rados.rados.0.smithi045.stdout:3526: seq_num 1080 ranges {1433600=491520,1925120=540672,2465792=262144} 2021-07-31T01:57:17.921 INFO:tasks.rados.rados.0.smithi045.stdout:3526: writing smithi04569269-43 from 1433600 to 1925120 tid 1 2021-07-31T01:57:17.924 INFO:tasks.rados.rados.0.smithi045.stdout:3526: writing smithi04569269-43 from 1925120 to 2465792 tid 2 2021-07-31T01:57:17.926 INFO:tasks.rados.rados.0.smithi045.stdout:3526: writing smithi04569269-43 from 2465792 to 2727936 tid 3 2021-07-31T01:57:17.926 INFO:tasks.rados.rados.0.smithi045.stdout:3515: done (8 left) 2021-07-31T01:57:17.927 INFO:tasks.rados.rados.0.smithi045.stdout:3520: done (7 left) 2021-07-31T01:57:17.927 INFO:tasks.rados.rados.0.smithi045.stdout:3527: read oid 38 snap -1 2021-07-31T01:57:17.927 INFO:tasks.rados.rados.0.smithi045.stdout:3527: expect (ObjNum 1037 snap 338 seq_num 1037) 2021-07-31T01:57:17.927 INFO:tasks.rados.rados.0.smithi045.stdout:3528: snap_remove snap 298 2021-07-31T01:57:18.739 INFO:tasks.rados.rados.0.smithi045.stdout:3528: done (8 left) 2021-07-31T01:57:18.739 INFO:tasks.rados.rados.0.smithi045.stdout:3529: read oid 22 snap -1 2021-07-31T01:57:18.740 INFO:tasks.rados.rados.0.smithi045.stdout:3529: expect deleted 2021-07-31T01:57:18.740 INFO:tasks.rados.rados.0.smithi045.stdout:3530: copy_from oid 9 from oid 36 current snap is 346 2021-07-31T01:57:18.740 INFO:tasks.rados.rados.0.smithi045.stdout:3517: finishing write tid 5 to smithi04569269-28 2021-07-31T01:57:18.740 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 28 v 2330 (ObjNum 1077 snap 346 seq_num 1077) dirty exists 2021-07-31T01:57:18.741 INFO:tasks.rados.rados.0.smithi045.stdout:3517: left oid 28 (ObjNum 1077 snap 346 seq_num 1077) 2021-07-31T01:57:18.741 INFO:tasks.rados.rados.0.smithi045.stdout:3511: finishing rollback tid 1 to smithi04569269-21 2021-07-31T01:57:18.741 INFO:tasks.rados.rados.0.smithi045.stdout:3511: finishing rollback tid 2 to smithi04569269-21 2021-07-31T01:57:18.741 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 21 v 2355 (ObjNum 1013 snap 323 seq_num 1013) dirty exists 2021-07-31T01:57:18.742 INFO:tasks.rados.rados.0.smithi045.stdout:3524: finishing write tid 1 to smithi04569269-26 2021-07-31T01:57:18.742 INFO:tasks.rados.rados.0.smithi045.stdout:3524: finishing write tid 2 to smithi04569269-26 2021-07-31T01:57:18.742 INFO:tasks.rados.rados.0.smithi045.stdout:3524: finishing write tid 3 to smithi04569269-26 2021-07-31T01:57:18.742 INFO:tasks.rados.rados.0.smithi045.stdout:3524: finishing write tid 4 to smithi04569269-26 2021-07-31T01:57:18.742 INFO:tasks.rados.rados.0.smithi045.stdout:3525: finishing rollback tid 0 to smithi04569269-25 2021-07-31T01:57:18.743 INFO:tasks.rados.rados.0.smithi045.stdout:3525: finishing rollback tid 1 to smithi04569269-25 2021-07-31T01:57:18.743 INFO:tasks.rados.rados.0.smithi045.stdout:3525: finishing rollback tid 2 to smithi04569269-25 2021-07-31T01:57:18.743 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 25 v 1848 (ObjNum 1031 snap 335 seq_num 1031) dirty exists 2021-07-31T01:57:18.743 INFO:tasks.rados.rados.0.smithi045.stdout:3524: finishing write tid 5 to smithi04569269-26 2021-07-31T01:57:18.744 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 26 v 1870 (ObjNum 1079 snap 346 seq_num 1079) dirty exists 2021-07-31T01:57:18.744 INFO:tasks.rados.rados.0.smithi045.stdout:3524: left oid 26 (ObjNum 1079 snap 346 seq_num 1079) 2021-07-31T01:57:18.744 INFO:tasks.rados.rados.0.smithi045.stdout:3518: finishing copy_from to smithi04569269-50 2021-07-31T01:57:18.744 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 50 v 2163 (ObjNum 881 snap 272 seq_num 881) dirty exists 2021-07-31T01:57:18.745 INFO:tasks.rados.rados.0.smithi045.stdout:3518: finishing copy_from racing read to smithi04569269-50 2021-07-31T01:57:18.745 INFO:tasks.rados.rados.0.smithi045.stdout:3521: expect (ObjNum 928 snap 281 seq_num 928) 2021-07-31T01:57:19.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:18 smithi154 conmon[34553]: cluster 2021-07-31T01:57:17.731758+0000 mon.a (mon.0) 1389 : 2021-07-31T01:57:19.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:18 smithi154 conmon[34553]: cluster [DBG] osdmap e722: 8 total, 8 up, 8 in 2021-07-31T01:57:19.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:18 smithi154 conmon[34553]: cluster 2021-07-31T01:57:17.936285+0000 mgr.x (mgr.24542) 428 : cluster [DBG] pgmap v871: 25 pgs: 25 active+clean; 607 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail 2021-07-31T01:57:19.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:18 smithi045 conmon[39718]: cluster 2021-07-31T01:57:17.731758+0000 mon.a (mon 2021-07-31T01:57:19.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:18 smithi045 conmon[39718]: .0) 1389 : cluster [DBG] osdmap e722: 8 total, 8 up, 8 in 2021-07-31T01:57:19.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:18 smithi045 conmon[39718]: cluster 2021-07-31T01:57:17.936285+0000 mgr.x (mgr.24542) 428 : cluster 2021-07-31T01:57:19.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:18 smithi045 conmon[39718]: [DBG] pgmap v871: 25 pgs: 25 active+clean; 607 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail 2021-07-31T01:57:19.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:18 smithi045 conmon[33655]: cluster 2021-07-31T01:57:17.731758+0000 mon.a (mon.0) 2021-07-31T01:57:19.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:18 smithi045 conmon[33655]: 1389 : cluster [DBG] osdmap e722: 8 total, 8 up, 8 in 2021-07-31T01:57:19.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:18 smithi045 conmon[33655]: cluster 2021-07-31T01:57:17.936285+0000 mgr.x (mgr.24542) 428 : cluster [DBG] pgmap v871: 25 pgs: 25 active+clean; 607 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail 2021-07-31T01:57:20.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:19 smithi154 conmon[34553]: cluster 2021-07-31T01:57:18.737446+0000 mon.a (mon.0) 1390 : cluster [DBG] osdmap e723: 8 total, 8 up, 8 in 2021-07-31T01:57:20.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:19 smithi045 conmon[39718]: cluster 2021-07-31T01:57:18.737446+0000 mon.a (mon.0) 1390 : cluster [DBG] osdmap e723: 8 total, 8 up, 8 in 2021-07-31T01:57:20.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:19 smithi045 conmon[33655]: cluster 2021-07-31T01:57:18.737446+0000 mon.a (mon.0) 1390 2021-07-31T01:57:20.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:19 smithi045 conmon[33655]: : cluster [DBG] osdmap e723: 8 total, 8 up, 8 in 2021-07-31T01:57:20.424 INFO:tasks.rados.rados.0.smithi045.stdout:3526: finishing write tid 1 to smithi04569269-43 2021-07-31T01:57:20.424 INFO:tasks.rados.rados.0.smithi045.stdout:3527: expect (ObjNum 1037 snap 338 seq_num 1037) 2021-07-31T01:57:20.439 INFO:tasks.rados.rados.0.smithi045.stdout:3526: finishing write tid 2 to smithi04569269-43 2021-07-31T01:57:20.440 INFO:tasks.rados.rados.0.smithi045.stdout:3526: finishing write tid 3 to smithi04569269-43 2021-07-31T01:57:20.440 INFO:tasks.rados.rados.0.smithi045.stdout:3526: finishing write tid 4 to smithi04569269-43 2021-07-31T01:57:20.440 INFO:tasks.rados.rados.0.smithi045.stdout:3526: finishing write tid 5 to smithi04569269-43 2021-07-31T01:57:20.440 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 43 v 2046 (ObjNum 1080 snap 346 seq_num 1080) dirty exists 2021-07-31T01:57:20.441 INFO:tasks.rados.rados.0.smithi045.stdout:3526: left oid 43 (ObjNum 1080 snap 346 seq_num 1080) 2021-07-31T01:57:20.441 INFO:tasks.rados.rados.0.smithi045.stdout:3511: done (9 left) 2021-07-31T01:57:20.441 INFO:tasks.rados.rados.0.smithi045.stdout:3517: done (8 left) 2021-07-31T01:57:20.441 INFO:tasks.rados.rados.0.smithi045.stdout:3518: done (7 left) 2021-07-31T01:57:20.441 INFO:tasks.rados.rados.0.smithi045.stdout:3521: done (6 left) 2021-07-31T01:57:20.442 INFO:tasks.rados.rados.0.smithi045.stdout:3524: done (5 left) 2021-07-31T01:57:20.442 INFO:tasks.rados.rados.0.smithi045.stdout:3525: done (4 left) 2021-07-31T01:57:20.442 INFO:tasks.rados.rados.0.smithi045.stdout:3526: done (3 left) 2021-07-31T01:57:20.442 INFO:tasks.rados.rados.0.smithi045.stdout:3527: done (2 left) 2021-07-31T01:57:20.443 INFO:tasks.rados.rados.0.smithi045.stdout:3531: read oid 32 snap -1 2021-07-31T01:57:20.443 INFO:tasks.rados.rados.0.smithi045.stdout:3531: expect deleted 2021-07-31T01:57:20.443 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 35 current snap is 346 2021-07-31T01:57:20.443 INFO:tasks.rados.rados.0.smithi045.stdout:3530: finishing copy_from to smithi04569269-9 2021-07-31T01:57:20.443 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 9 v 2347 (ObjNum 1953066355 snap 875587944 seq_num 842610229) dirty exists 2021-07-31T01:57:20.444 INFO:tasks.rados.rados.0.smithi045.stdout:3530: finishing copy_from racing read to smithi04569269-9 2021-07-31T01:57:20.444 INFO:tasks.rados.rados.0.smithi045.stdout:3532: seq_num 1081 ranges {0=507904,507904=425984,933888=376832} 2021-07-31T01:57:20.445 INFO:tasks.rados.rados.0.smithi045.stdout:3532: writing smithi04569269-35 from 0 to 507904 tid 1 2021-07-31T01:57:20.445 INFO:tasks.rados.rados.0.smithi045.stdout:3532: writing smithi04569269-35 from 507904 to 933888 tid 2 2021-07-31T01:57:20.446 INFO:tasks.rados.rados.0.smithi045.stdout:3532: writing smithi04569269-35 from 933888 to 1310720 tid 3 2021-07-31T01:57:20.446 INFO:tasks.rados.rados.0.smithi045.stdout:3529: done (3 left) 2021-07-31T01:57:20.446 INFO:tasks.rados.rados.0.smithi045.stdout:3530: done (2 left) 2021-07-31T01:57:20.446 INFO:tasks.rados.rados.0.smithi045.stdout:3533: snap_create 2021-07-31T01:57:20.447 INFO:tasks.rados.rados.0.smithi045.stdout:3532: finishing write tid 1 to smithi04569269-35 2021-07-31T01:57:20.449 INFO:tasks.rados.rados.0.smithi045.stdout:3532: finishing write tid 2 to smithi04569269-35 2021-07-31T01:57:20.449 INFO:tasks.rados.rados.0.smithi045.stdout:3532: finishing write tid 3 to smithi04569269-35 2021-07-31T01:57:20.450 INFO:tasks.rados.rados.0.smithi045.stdout:3532: finishing write tid 4 to smithi04569269-35 2021-07-31T01:57:20.450 INFO:tasks.rados.rados.0.smithi045.stdout:3532: finishing write tid 5 to smithi04569269-35 2021-07-31T01:57:20.451 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 35 v 2385 (ObjNum 1081 snap 346 seq_num 1081) dirty exists 2021-07-31T01:57:20.451 INFO:tasks.rados.rados.0.smithi045.stdout:3532: left oid 35 (ObjNum 1081 snap 346 seq_num 1081) 2021-07-31T01:57:20.749 INFO:tasks.rados.rados.0.smithi045.stdout:3531: done (2 left) 2021-07-31T01:57:20.749 INFO:tasks.rados.rados.0.smithi045.stdout:3532: done (1 left) 2021-07-31T01:57:20.750 INFO:tasks.rados.rados.0.smithi045.stdout:3533: done (0 left) 2021-07-31T01:57:20.750 INFO:tasks.rados.rados.0.smithi045.stdout:3534: delete oid 30 current snap is 347 2021-07-31T01:57:20.751 INFO:tasks.rados.rados.0.smithi045.stdout:3534: done (0 left) 2021-07-31T01:57:20.751 INFO:tasks.rados.rados.0.smithi045.stdout:3535: read oid 9 snap 333 2021-07-31T01:57:20.751 INFO:tasks.rados.rados.0.smithi045.stdout:3535: expect (ObjNum 990 snap 307 seq_num 990) 2021-07-31T01:57:20.752 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 32 current snap is 347 2021-07-31T01:57:20.752 INFO:tasks.rados.rados.0.smithi045.stdout:3536: seq_num 1083 ranges {0=540672,540672=589824,1130496=212992} 2021-07-31T01:57:20.753 INFO:tasks.rados.rados.0.smithi045.stdout:3536: writing smithi04569269-32 from 0 to 540672 tid 1 2021-07-31T01:57:20.756 INFO:tasks.rados.rados.0.smithi045.stdout:3536: writing smithi04569269-32 from 540672 to 1130496 tid 2 2021-07-31T01:57:20.758 INFO:tasks.rados.rados.0.smithi045.stdout:3536: writing smithi04569269-32 from 1130496 to 1343488 tid 3 2021-07-31T01:57:20.758 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 20 current snap is 347 2021-07-31T01:57:20.758 INFO:tasks.rados.rados.0.smithi045.stdout:3537: seq_num 1084 ranges {729088=557056,1286144=475136,1761280=327680} 2021-07-31T01:57:20.765 INFO:tasks.rados.rados.0.smithi045.stdout:3537: writing smithi04569269-20 from 729088 to 1286144 tid 1 2021-07-31T01:57:20.768 INFO:tasks.rados.rados.0.smithi045.stdout:3537: writing smithi04569269-20 from 1286144 to 1761280 tid 2 2021-07-31T01:57:20.771 INFO:tasks.rados.rados.0.smithi045.stdout:3537: writing smithi04569269-20 from 1761280 to 2088960 tid 3 2021-07-31T01:57:20.771 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 4 current snap is 347 2021-07-31T01:57:20.771 INFO:tasks.rados.rados.0.smithi045.stdout:3538: seq_num 1085 ranges {557056=581632,1138688=761856,1900544=40960} 2021-07-31T01:57:20.778 INFO:tasks.rados.rados.0.smithi045.stdout:3538: writing smithi04569269-4 from 557056 to 1138688 tid 1 2021-07-31T01:57:20.783 INFO:tasks.rados.rados.0.smithi045.stdout:3538: writing smithi04569269-4 from 1138688 to 1900544 tid 2 2021-07-31T01:57:20.783 INFO:tasks.rados.rados.0.smithi045.stdout:3538: writing smithi04569269-4 from 1900544 to 1941504 tid 3 2021-07-31T01:57:20.784 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 26 current snap is 347 2021-07-31T01:57:20.784 INFO:tasks.rados.rados.0.smithi045.stdout:3539: seq_num 1086 ranges {3219456=598016,3817472=647168,4464640=155648} 2021-07-31T01:57:20.807 INFO:tasks.rados.rados.0.smithi045.stdout:3539: writing smithi04569269-26 from 3219456 to 3817472 tid 1 2021-07-31T01:57:20.808 INFO:tasks.rados.rados.0.smithi045.stdout:3539: writing smithi04569269-26 from 3817472 to 4464640 tid 2 2021-07-31T01:57:20.811 INFO:tasks.rados.rados.0.smithi045.stdout:3539: writing smithi04569269-26 from 4464640 to 4620288 tid 3 2021-07-31T01:57:20.812 INFO:tasks.rados.rados.0.smithi045.stdout:3540: snap_create 2021-07-31T01:57:20.812 INFO:tasks.rados.rados.0.smithi045.stdout:3536: finishing write tid 1 to smithi04569269-32 2021-07-31T01:57:20.812 INFO:tasks.rados.rados.0.smithi045.stdout:3536: finishing write tid 2 to smithi04569269-32 2021-07-31T01:57:20.812 INFO:tasks.rados.rados.0.smithi045.stdout:3536: finishing write tid 3 to smithi04569269-32 2021-07-31T01:57:20.813 INFO:tasks.rados.rados.0.smithi045.stdout:3536: finishing write tid 4 to smithi04569269-32 2021-07-31T01:57:20.813 INFO:tasks.rados.rados.0.smithi045.stdout:3536: finishing write tid 5 to smithi04569269-32 2021-07-31T01:57:20.813 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 32 v 1874 (ObjNum 1083 snap 347 seq_num 1083) dirty exists 2021-07-31T01:57:20.813 INFO:tasks.rados.rados.0.smithi045.stdout:3536: left oid 32 (ObjNum 1083 snap 347 seq_num 1083) 2021-07-31T01:57:20.814 INFO:tasks.rados.rados.0.smithi045.stdout:3537: finishing write tid 1 to smithi04569269-20 2021-07-31T01:57:20.814 INFO:tasks.rados.rados.0.smithi045.stdout:3537: finishing write tid 2 to smithi04569269-20 2021-07-31T01:57:20.814 INFO:tasks.rados.rados.0.smithi045.stdout:3537: finishing write tid 3 to smithi04569269-20 2021-07-31T01:57:20.814 INFO:tasks.rados.rados.0.smithi045.stdout:3537: finishing write tid 4 to smithi04569269-20 2021-07-31T01:57:20.814 INFO:tasks.rados.rados.0.smithi045.stdout:3537: finishing write tid 5 to smithi04569269-20 2021-07-31T01:57:20.815 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 20 v 2390 (ObjNum 1084 snap 347 seq_num 1084) dirty exists 2021-07-31T01:57:20.815 INFO:tasks.rados.rados.0.smithi045.stdout:3537: left oid 20 (ObjNum 1084 snap 347 seq_num 1084) 2021-07-31T01:57:20.815 INFO:tasks.rados.rados.0.smithi045.stdout:3538: finishing write tid 1 to smithi04569269-4 2021-07-31T01:57:20.815 INFO:tasks.rados.rados.0.smithi045.stdout:3538: finishing write tid 2 to smithi04569269-4 2021-07-31T01:57:20.815 INFO:tasks.rados.rados.0.smithi045.stdout:3538: finishing write tid 3 to smithi04569269-4 2021-07-31T01:57:20.816 INFO:tasks.rados.rados.0.smithi045.stdout:3538: finishing write tid 4 to smithi04569269-4 2021-07-31T01:57:20.816 INFO:tasks.rados.rados.0.smithi045.stdout:3535: expect (ObjNum 990 snap 307 seq_num 990) 2021-07-31T01:57:21.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:20 smithi154 conmon[34553]: cluster 2021-07-31T01:57:19.936824+0000 mgr.x (mgr.24542) 2021-07-31T01:57:21.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:20 smithi154 conmon[34553]: 429 : cluster [DBG] pgmap v873: 25 pgs: 25 active+clean; 626 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 18 MiB/s rd, 7.2 MiB/s wr, 32 op/s 2021-07-31T01:57:21.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:20 smithi045 conmon[39718]: cluster 2021-07-31T01:57:19.936824+0000 mgr.x (mgr.24542 2021-07-31T01:57:21.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:20 smithi045 conmon[39718]: ) 429 : cluster [DBG] pgmap v873: 25 pgs: 25 active+clean; 626 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 18 MiB/s rd, 7.2 MiB/s wr, 32 op/s 2021-07-31T01:57:21.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:20 smithi045 conmon[33655]: cluster 2021-07-31T01:57:19.936824+0000 mgr.x (mgr.24542) 429 : 2021-07-31T01:57:21.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:20 smithi045 conmon[33655]: cluster [DBG] pgmap v873: 25 pgs: 25 active+clean; 626 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 18 MiB/s rd, 7.2 MiB/s wr, 32 op/s 2021-07-31T01:57:22.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:21 smithi154 conmon[34553]: cluster 2021-07-31T01:57:20.745452+0000 mon.a (mon.0) 1391 : cluster [DBG] osdmap e724: 8 total, 8 up, 8 in 2021-07-31T01:57:22.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:21 smithi154 conmon[34553]: cluster 2021-07-31T01:57:21.663954+0000 mon.a (mon.0) 1392 : cluster [DBG] osdmap e725: 8 total, 8 up, 8 in 2021-07-31T01:57:22.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:21 smithi045 conmon[39718]: cluster 2021-07-31T01:57:20.745452 2021-07-31T01:57:22.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:21 smithi045 conmon[39718]: +0000 mon.a (mon.0) 1391 : cluster [DBG] osdmap e724: 8 total, 8 up, 8 in 2021-07-31T01:57:22.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:21 smithi045 conmon[39718]: cluster 2021-07-31T01:57:22.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:21 smithi045 conmon[39718]: 2021-07-31T01:57:21.663954+0000 mon.a ( 2021-07-31T01:57:22.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:21 smithi045 conmon[39718]: mon.0) 1392 : cluster [DBG] osdmap e725: 8 total, 8 up, 8 in 2021-07-31T01:57:22.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:21 smithi045 conmon[33655]: cluster 2021-07-31T01:57:20.745452+0000 2021-07-31T01:57:22.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:21 smithi045 conmon[33655]: mon.a (mon.0) 1391 : cluster [DBG] osdmap e724: 8 total, 8 up, 8 in 2021-07-31T01:57:22.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:21 smithi045 conmon[33655]: cluster 2021-07-31T01:57:21.663954+0000 mon.a (mon.0 2021-07-31T01:57:22.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:21 smithi045 conmon[33655]: ) 1392 : cluster [DBG] osdmap e725: 8 total, 8 up, 8 in 2021-07-31T01:57:22.044 INFO:tasks.rados.rados.0.smithi045.stdout:3538: finishing write tid 5 to smithi04569269-4 2021-07-31T01:57:22.044 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 4 v 1886 (ObjNum 1085 snap 347 seq_num 1085) dirty exists 2021-07-31T01:57:22.045 INFO:tasks.rados.rados.0.smithi045.stdout:3538: left oid 4 (ObjNum 1085 snap 347 seq_num 1085) 2021-07-31T01:57:22.045 INFO:tasks.rados.rados.0.smithi045.stdout:3539: finishing write tid 1 to smithi04569269-26 2021-07-31T01:57:22.045 INFO:tasks.rados.rados.0.smithi045.stdout:3539: finishing write tid 2 to smithi04569269-26 2021-07-31T01:57:22.045 INFO:tasks.rados.rados.0.smithi045.stdout:3539: finishing write tid 3 to smithi04569269-26 2021-07-31T01:57:22.046 INFO:tasks.rados.rados.0.smithi045.stdout:3539: finishing write tid 4 to smithi04569269-26 2021-07-31T01:57:22.046 INFO:tasks.rados.rados.0.smithi045.stdout:3539: finishing write tid 5 to smithi04569269-26 2021-07-31T01:57:22.046 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 26 v 1878 (ObjNum 1086 snap 347 seq_num 1086) dirty exists 2021-07-31T01:57:22.047 INFO:tasks.rados.rados.0.smithi045.stdout:3539: left oid 26 (ObjNum 1086 snap 347 seq_num 1086) 2021-07-31T01:57:22.047 INFO:tasks.rados.rados.0.smithi045.stdout:3535: done (5 left) 2021-07-31T01:57:22.047 INFO:tasks.rados.rados.0.smithi045.stdout:3536: done (4 left) 2021-07-31T01:57:22.047 INFO:tasks.rados.rados.0.smithi045.stdout:3537: done (3 left) 2021-07-31T01:57:22.047 INFO:tasks.rados.rados.0.smithi045.stdout:3538: done (2 left) 2021-07-31T01:57:22.048 INFO:tasks.rados.rados.0.smithi045.stdout:3539: done (1 left) 2021-07-31T01:57:22.048 INFO:tasks.rados.rados.0.smithi045.stdout:3540: done (0 left) 2021-07-31T01:57:22.048 INFO:tasks.rados.rados.0.smithi045.stdout:3541: read oid 22 snap -1 2021-07-31T01:57:22.048 INFO:tasks.rados.rados.0.smithi045.stdout:3541: expect deleted 2021-07-31T01:57:22.048 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 2 current snap is 348 2021-07-31T01:57:22.049 INFO:tasks.rados.rados.0.smithi045.stdout:3542: seq_num 1087 ranges {2662400=614400,3276800=532480,3809280=270336} 2021-07-31T01:57:22.061 INFO:tasks.rados.rados.0.smithi045.stdout:3542: writing smithi04569269-2 from 2662400 to 3276800 tid 1 2021-07-31T01:57:22.064 INFO:tasks.rados.rados.0.smithi045.stdout:3542: writing smithi04569269-2 from 3276800 to 3809280 tid 2 2021-07-31T01:57:22.066 INFO:tasks.rados.rados.0.smithi045.stdout:3542: writing smithi04569269-2 from 3809280 to 4079616 tid 3 2021-07-31T01:57:22.066 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 44 current snap is 348 2021-07-31T01:57:22.067 INFO:tasks.rados.rados.0.smithi045.stdout:3543: seq_num 1088 ranges {4677632=630784,5308416=466944,5775360=335872} 2021-07-31T01:57:22.095 INFO:tasks.rados.rados.0.smithi045.stdout:3543: writing smithi04569269-44 from 4677632 to 5308416 tid 1 2021-07-31T01:57:22.097 INFO:tasks.rados.rados.0.smithi045.stdout:3543: writing smithi04569269-44 from 5308416 to 5775360 tid 2 2021-07-31T01:57:22.101 INFO:tasks.rados.rados.0.smithi045.stdout:3543: writing smithi04569269-44 from 5775360 to 6111232 tid 3 2021-07-31T01:57:22.101 INFO:tasks.rados.rados.0.smithi045.stdout:3544: copy_from oid 50 from oid 29 current snap is 348 2021-07-31T01:57:22.101 INFO:tasks.rados.rados.0.smithi045.stdout:3542: finishing write tid 1 to smithi04569269-2 2021-07-31T01:57:22.101 INFO:tasks.rados.rados.0.smithi045.stdout:3542: finishing write tid 2 to smithi04569269-2 2021-07-31T01:57:22.101 INFO:tasks.rados.rados.0.smithi045.stdout:3542: finishing write tid 3 to smithi04569269-2 2021-07-31T01:57:22.102 INFO:tasks.rados.rados.0.smithi045.stdout:3542: finishing write tid 4 to smithi04569269-2 2021-07-31T01:57:22.102 INFO:tasks.rados.rados.0.smithi045.stdout:3542: finishing write tid 5 to smithi04569269-2 2021-07-31T01:57:22.102 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 2 v 2351 (ObjNum 1087 snap 348 seq_num 1087) dirty exists 2021-07-31T01:57:22.102 INFO:tasks.rados.rados.0.smithi045.stdout:3542: left oid 2 (ObjNum 1087 snap 348 seq_num 1087) 2021-07-31T01:57:22.103 INFO:tasks.rados.rados.0.smithi045.stdout:3541: done (3 left) 2021-07-31T01:57:22.103 INFO:tasks.rados.rados.0.smithi045.stdout:3542: done (2 left) 2021-07-31T01:57:22.104 INFO:tasks.rados.rados.0.smithi045.stdout:3545: read oid 41 snap 342 2021-07-31T01:57:22.104 INFO:tasks.rados.rados.0.smithi045.stdout:3545: expect (ObjNum 948 snap 286 seq_num 948) 2021-07-31T01:57:22.105 INFO:tasks.rados.rados.0.smithi045.stdout:3546: snap_create 2021-07-31T01:57:22.106 INFO:tasks.rados.rados.0.smithi045.stdout:3543: finishing write tid 1 to smithi04569269-44 2021-07-31T01:57:22.106 INFO:tasks.rados.rados.0.smithi045.stdout:3543: finishing write tid 2 to smithi04569269-44 2021-07-31T01:57:22.108 INFO:tasks.rados.rados.0.smithi045.stdout:3543: finishing write tid 3 to smithi04569269-44 2021-07-31T01:57:22.109 INFO:tasks.rados.rados.0.smithi045.stdout:3543: finishing write tid 4 to smithi04569269-44 2021-07-31T01:57:22.109 INFO:tasks.rados.rados.0.smithi045.stdout:3545: expect (ObjNum 948 snap 286 seq_num 948) 2021-07-31T01:57:22.148 INFO:tasks.rados.rados.0.smithi045.stdout:3543: finishing write tid 5 to smithi04569269-44 2021-07-31T01:57:22.149 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 44 v 2050 (ObjNum 1088 snap 348 seq_num 1088) dirty exists 2021-07-31T01:57:22.149 INFO:tasks.rados.rados.0.smithi045.stdout:3543: left oid 44 (ObjNum 1088 snap 348 seq_num 1088) 2021-07-31T01:57:22.149 INFO:tasks.rados.rados.0.smithi045.stdout:3544: finishing copy_from to smithi04569269-50 2021-07-31T01:57:22.150 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 50 v 1428 (ObjNum 1066 snap 344 seq_num 1066) dirty exists 2021-07-31T01:57:22.150 INFO:tasks.rados.rados.0.smithi045.stdout:3544: finishing copy_from racing read to smithi04569269-50 2021-07-31T01:57:22.754 INFO:tasks.rados.rados.0.smithi045.stdout:3543: done (3 left) 2021-07-31T01:57:22.754 INFO:tasks.rados.rados.0.smithi045.stdout:3544: done (2 left) 2021-07-31T01:57:22.755 INFO:tasks.rados.rados.0.smithi045.stdout:3545: done (1 left) 2021-07-31T01:57:22.755 INFO:tasks.rados.rados.0.smithi045.stdout:3546: done (0 left) 2021-07-31T01:57:22.755 INFO:tasks.rados.rados.0.smithi045.stdout:3547: snap_remove snap 307 2021-07-31T01:57:23.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:22 smithi154 conmon[34553]: cluster 2021-07-31T01:57:21.937246+0000 mgr.x (mgr.24542) 430 : cluster 2021-07-31T01:57:23.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:22 smithi154 conmon[34553]: [DBG] pgmap v876: 25 pgs: 25 active+clean; 626 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 18 MiB/s rd, 7.3 MiB/s wr, 33 op/s 2021-07-31T01:57:23.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:22 smithi045 conmon[33655]: cluster 2021-07-31T01:57:21.937246+0000 mgr.x (mgr.24542) 430 : cluster [DBG] 2021-07-31T01:57:23.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:22 smithi045 conmon[33655]: pgmap v876: 25 pgs: 25 active+clean; 626 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 18 MiB/s rd, 7.3 MiB/s wr, 33 op/s 2021-07-31T01:57:23.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:22 smithi045 conmon[39718]: cluster 2021-07-31T01:57:21.937246+0000 mgr.x (mgr.24542 2021-07-31T01:57:23.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:22 smithi045 conmon[39718]: ) 430 : cluster [DBG] pgmap v876: 25 pgs: 25 active+clean; 626 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 18 MiB/s rd, 7.3 MiB/s wr, 33 op/s 2021-07-31T01:57:23.756 INFO:tasks.rados.rados.0.smithi045.stdout:3547: done (0 left) 2021-07-31T01:57:23.756 INFO:tasks.rados.rados.0.smithi045.stdout:3548: setattr oid 44 current snap is 349 2021-07-31T01:57:23.759 INFO:tasks.rados.rados.0.smithi045.stdout:3549: copy_from oid 35 from oid 34 current snap is 349 2021-07-31T01:57:23.759 INFO:tasks.rados.rados.0.smithi045.stdout:3550: read oid 22 snap -1 2021-07-31T01:57:23.759 INFO:tasks.rados.rados.0.smithi045.stdout:3550: expect deleted 2021-07-31T01:57:23.760 INFO:tasks.rados.rados.0.smithi045.stdout:3551: snap_remove snap 322 2021-07-31T01:57:24.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:23 smithi045 conmon[33655]: cluster 2021-07-31T01:57:22.753083+0000 mon.a (mon. 2021-07-31T01:57:24.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:23 smithi045 conmon[33655]: 0) 1393 : cluster [DBG] osdmap e726: 8 total, 8 up, 8 in 2021-07-31T01:57:24.027 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:57:23 smithi045 conmon[61210]: level=error ts=2021-07-31T01:57:23.526Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:57:24.027 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:57:23 smithi045 conmon[61210]: level=error ts=2021-07-31T01:57:23.526Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:57:24.027 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:57:23 smithi045 conmon[61210]: level=error ts=2021-07-31T01:57:23.526Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T01:57:24.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:23 smithi045 conmon[39718]: cluster 2021-07-31T01:57:22.753083+0000 mon.a (mon.0 2021-07-31T01:57:24.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:23 smithi045 conmon[39718]: ) 1393 : cluster [DBG] osdmap e726: 8 total, 8 up, 8 in 2021-07-31T01:57:24.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:23 smithi154 conmon[34553]: cluster 2021-07-31T01:57:22.753083+0000 mon.a (mon.0) 1393 : cluster 2021-07-31T01:57:24.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:23 smithi154 conmon[34553]: [DBG] osdmap e726: 8 total, 8 up, 8 in 2021-07-31T01:57:24.772 INFO:tasks.rados.rados.0.smithi045.stdout:3551: done (3 left) 2021-07-31T01:57:24.772 INFO:tasks.rados.rados.0.smithi045.stdout:3552: rollback oid 12 current snap is 349 2021-07-31T01:57:24.772 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 12 to 340 2021-07-31T01:57:24.772 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 44 v 2051 (ObjNum 1088 snap 348 seq_num 1088) dirty exists 2021-07-31T01:57:24.773 INFO:tasks.rados.rados.0.smithi045.stdout:3549: finishing copy_from to smithi04569269-35 2021-07-31T01:57:24.773 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 35 v 1861 (ObjNum 1063 snap 344 seq_num 1063) dirty exists 2021-07-31T01:57:24.773 INFO:tasks.rados.rados.0.smithi045.stdout:3549: finishing copy_from racing read to smithi04569269-35 2021-07-31T01:57:24.773 INFO:tasks.rados.rados.0.smithi045.stdout:3548: done (3 left) 2021-07-31T01:57:24.774 INFO:tasks.rados.rados.0.smithi045.stdout:3549: done (2 left) 2021-07-31T01:57:24.774 INFO:tasks.rados.rados.0.smithi045.stdout:3550: done (1 left) 2021-07-31T01:57:24.774 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 2 current snap is 349 2021-07-31T01:57:24.774 INFO:tasks.rados.rados.0.smithi045.stdout:3553: seq_num 1089 ranges {4079616=647168,4726784=745472,5472256=57344} 2021-07-31T01:57:24.799 INFO:tasks.rados.rados.0.smithi045.stdout:3553: writing smithi04569269-2 from 4079616 to 4726784 tid 1 2021-07-31T01:57:24.803 INFO:tasks.rados.rados.0.smithi045.stdout:3553: writing smithi04569269-2 from 4726784 to 5472256 tid 2 2021-07-31T01:57:24.804 INFO:tasks.rados.rados.0.smithi045.stdout:3553: writing smithi04569269-2 from 5472256 to 5529600 tid 3 2021-07-31T01:57:24.804 INFO:tasks.rados.rados.0.smithi045.stdout:3554: snap_remove snap 345 2021-07-31T01:57:25.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:24 smithi045 conmon[33655]: cluster 2021-07-31T01:57:23.755463+0000 2021-07-31T01:57:25.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:24 smithi045 conmon[33655]: mon.a (mon.0) 1394 : cluster [DBG] osdmap e727: 8 total, 8 up, 8 in 2021-07-31T01:57:25.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:24 smithi045 conmon[33655]: cluster 2021-07-31T01:57:23.937724+0000 mgr.x (mgr.24542) 431 : cluster 2021-07-31T01:57:25.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:24 smithi045 conmon[33655]: [DBG] pgmap v879: 25 pgs: 25 active+clean; 637 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 9.5 MiB/s rd, 2.8 MiB/s wr, 17 op/s 2021-07-31T01:57:25.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:24 smithi045 conmon[39718]: cluster 2021-07-31T01:57:23.755463+0000 mon.a ( 2021-07-31T01:57:25.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:24 smithi045 conmon[39718]: mon.0) 1394 : cluster [DBG] osdmap e727: 8 total, 8 up, 8 in 2021-07-31T01:57:25.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:24 smithi045 conmon[39718]: cluster 2021-07-31T01:57:25.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:24 smithi045 conmon[39718]: 2021-07-31T01:57:23.937724+0000 mgr.x (mgr. 2021-07-31T01:57:25.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:24 smithi045 conmon[39718]: 24542) 431 : cluster [DBG] pgmap v879: 25 pgs: 25 active+clean; 637 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 9.5 MiB/s rd, 2.8 MiB/s wr, 17 op/s 2021-07-31T01:57:25.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:24 smithi154 conmon[34553]: cluster 2021-07-31T01:57:23.755463+0000 mon.a (mon.0) 1394 : cluster 2021-07-31T01:57:25.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:24 smithi154 conmon[34553]: [DBG] osdmap e727: 8 total, 8 up, 8 in 2021-07-31T01:57:25.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:24 smithi154 conmon[34553]: cluster 2021-07-31T01:57:23.937724+0000 mgr.x (mgr.24542) 431 : cluster [DBG] pgmap v879: 25 pgs: 25 active+clean; 637 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 9.5 MiB/s rd, 2.8 MiB/s wr, 17 op/s 2021-07-31T01:57:25.776 INFO:tasks.rados.rados.0.smithi045.stdout:3554: done (2 left) 2021-07-31T01:57:25.777 INFO:tasks.rados.rados.0.smithi045.stdout:3555: read oid 18 snap -1 2021-07-31T01:57:25.777 INFO:tasks.rados.rados.0.smithi045.stdout:3555: expect deleted 2021-07-31T01:57:25.777 INFO:tasks.rados.rados.0.smithi045.stdout:3556: setattr oid 20 current snap is 349 2021-07-31T01:57:25.777 INFO:tasks.rados.rados.0.smithi045.stdout:3552: finishing rollback tid 0 to smithi04569269-12 2021-07-31T01:57:25.778 INFO:tasks.rados.rados.0.smithi045.stdout:3552: finishing rollback tid 1 to smithi04569269-12 2021-07-31T01:57:25.778 INFO:tasks.rados.rados.0.smithi045.stdout:3552: finishing rollback tid 2 to smithi04569269-12 2021-07-31T01:57:25.778 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 12 v 1883 (ObjNum 967 snap 295 seq_num 967) dirty exists 2021-07-31T01:57:25.778 INFO:tasks.rados.rados.0.smithi045.stdout:3553: finishing write tid 1 to smithi04569269-2 2021-07-31T01:57:25.778 INFO:tasks.rados.rados.0.smithi045.stdout:3553: finishing write tid 2 to smithi04569269-2 2021-07-31T01:57:25.779 INFO:tasks.rados.rados.0.smithi045.stdout:3553: finishing write tid 3 to smithi04569269-2 2021-07-31T01:57:25.779 INFO:tasks.rados.rados.0.smithi045.stdout:3553: finishing write tid 4 to smithi04569269-2 2021-07-31T01:57:25.779 INFO:tasks.rados.rados.0.smithi045.stdout:3553: finishing write tid 5 to smithi04569269-2 2021-07-31T01:57:25.779 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 2 v 2355 (ObjNum 1089 snap 349 seq_num 1089) dirty exists 2021-07-31T01:57:25.780 INFO:tasks.rados.rados.0.smithi045.stdout:3553: left oid 2 (ObjNum 1089 snap 349 seq_num 1089) 2021-07-31T01:57:25.780 INFO:tasks.rados.rados.0.smithi045.stdout:3552: done (3 left) 2021-07-31T01:57:25.781 INFO:tasks.rados.rados.0.smithi045.stdout:3553: done (2 left) 2021-07-31T01:57:25.781 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 49 current snap is 349 2021-07-31T01:57:25.781 INFO:tasks.rados.rados.0.smithi045.stdout:3557: seq_num 1090 ranges {1114112=663552,1777664=630784,2408448=172032} 2021-07-31T01:57:25.788 INFO:tasks.rados.rados.0.smithi045.stdout:3557: writing smithi04569269-49 from 1114112 to 1777664 tid 1 2021-07-31T01:57:25.792 INFO:tasks.rados.rados.0.smithi045.stdout:3557: writing smithi04569269-49 from 1777664 to 2408448 tid 2 2021-07-31T01:57:25.793 INFO:tasks.rados.rados.0.smithi045.stdout:3557: writing smithi04569269-49 from 2408448 to 2580480 tid 3 2021-07-31T01:57:25.793 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 28 current snap is 349 2021-07-31T01:57:25.794 INFO:tasks.rados.rados.0.smithi045.stdout:3558: seq_num 1091 ranges {1245184=679936,1925120=516096,2441216=286720} 2021-07-31T01:57:25.804 INFO:tasks.rados.rados.0.smithi045.stdout:3558: writing smithi04569269-28 from 1245184 to 1925120 tid 1 2021-07-31T01:57:25.806 INFO:tasks.rados.rados.0.smithi045.stdout:3558: writing smithi04569269-28 from 1925120 to 2441216 tid 2 2021-07-31T01:57:25.809 INFO:tasks.rados.rados.0.smithi045.stdout:3558: writing smithi04569269-28 from 2441216 to 2727936 tid 3 2021-07-31T01:57:25.809 INFO:tasks.rados.rados.0.smithi045.stdout:3559: copy_from oid 36 from oid 5 current snap is 349 2021-07-31T01:57:25.809 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 20 v 2420 (ObjNum 1084 snap 347 seq_num 1084) dirty exists 2021-07-31T01:57:25.809 INFO:tasks.rados.rados.0.smithi045.stdout:3557: finishing write tid 1 to smithi04569269-49 2021-07-31T01:57:25.810 INFO:tasks.rados.rados.0.smithi045.stdout:3557: finishing write tid 2 to smithi04569269-49 2021-07-31T01:57:25.810 INFO:tasks.rados.rados.0.smithi045.stdout:3557: finishing write tid 3 to smithi04569269-49 2021-07-31T01:57:25.810 INFO:tasks.rados.rados.0.smithi045.stdout:3557: finishing write tid 4 to smithi04569269-49 2021-07-31T01:57:25.810 INFO:tasks.rados.rados.0.smithi045.stdout:3557: finishing write tid 5 to smithi04569269-49 2021-07-31T01:57:25.811 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 49 v 2429 (ObjNum 1090 snap 349 seq_num 1090) dirty exists 2021-07-31T01:57:25.811 INFO:tasks.rados.rados.0.smithi045.stdout:3557: left oid 49 (ObjNum 1090 snap 349 seq_num 1090) 2021-07-31T01:57:25.811 INFO:tasks.rados.rados.0.smithi045.stdout:3555: done (4 left) 2021-07-31T01:57:25.811 INFO:tasks.rados.rados.0.smithi045.stdout:3556: done (3 left) 2021-07-31T01:57:25.811 INFO:tasks.rados.rados.0.smithi045.stdout:3557: done (2 left) 2021-07-31T01:57:25.812 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 22 current snap is 349 2021-07-31T01:57:25.812 INFO:tasks.rados.rados.0.smithi045.stdout:3560: seq_num 1092 ranges {0=696320,696320=802816} 2021-07-31T01:57:25.812 INFO:tasks.rados.rados.0.smithi045.stdout:3560: writing smithi04569269-22 from 0 to 696320 tid 1 2021-07-31T01:57:25.818 INFO:tasks.rados.rados.0.smithi045.stdout:3560: writing smithi04569269-22 from 696320 to 1499136 tid 2 2021-07-31T01:57:25.818 INFO:tasks.rados.rados.0.smithi045.stdout:3561: copy_from oid 27 from oid 24 current snap is 349 2021-07-31T01:57:25.818 INFO:tasks.rados.rados.0.smithi045.stdout:3558: finishing write tid 1 to smithi04569269-28 2021-07-31T01:57:25.818 INFO:tasks.rados.rados.0.smithi045.stdout:3558: finishing write tid 2 to smithi04569269-28 2021-07-31T01:57:25.819 INFO:tasks.rados.rados.0.smithi045.stdout:3558: finishing write tid 3 to smithi04569269-28 2021-07-31T01:57:25.819 INFO:tasks.rados.rados.0.smithi045.stdout:3558: finishing write tid 4 to smithi04569269-28 2021-07-31T01:57:25.819 INFO:tasks.rados.rados.0.smithi045.stdout:3562: snap_remove snap 342 2021-07-31T01:57:26.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:25 smithi045 conmon[33655]: cluster 2021-07-31T01:57:24.770464+0000 mon.a (mon.0) 1395 : cluster [DBG] osdmap e728: 8 total, 8 up, 8 in 2021-07-31T01:57:26.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:25 smithi045 conmon[39718]: cluster 2021-07-31T01:57:24.770464+0000 mon.a (mon. 2021-07-31T01:57:26.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:25 smithi045 conmon[39718]: 0) 1395 : cluster [DBG] osdmap e728: 8 total, 8 up, 8 in 2021-07-31T01:57:26.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:25 smithi154 conmon[34553]: cluster 2021-07-31T01:57:24.770464+0000 mon.a (mon.0) 1395 : cluster 2021-07-31T01:57:26.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:25 smithi154 conmon[34553]: [DBG] osdmap e728: 8 total, 8 up, 8 in 2021-07-31T01:57:26.667 INFO:tasks.rados.rados.0.smithi045.stdout:3562: done (4 left) 2021-07-31T01:57:26.668 INFO:tasks.rados.rados.0.smithi045.stdout:3563: rollback oid 16 current snap is 349 2021-07-31T01:57:26.668 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 16 to 295 2021-07-31T01:57:26.668 INFO:tasks.rados.rados.0.smithi045.stdout:3558: finishing write tid 5 to smithi04569269-28 2021-07-31T01:57:26.669 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 28 v 2335 (ObjNum 1091 snap 349 seq_num 1091) dirty exists 2021-07-31T01:57:26.669 INFO:tasks.rados.rados.0.smithi045.stdout:3558: left oid 28 (ObjNum 1091 snap 349 seq_num 1091) 2021-07-31T01:57:26.669 INFO:tasks.rados.rados.0.smithi045.stdout:3560: finishing write tid 1 to smithi04569269-22 2021-07-31T01:57:26.669 INFO:tasks.rados.rados.0.smithi045.stdout:3561: finishing copy_from to smithi04569269-27 2021-07-31T01:57:26.669 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 27 v 1664 (ObjNum 288 snap 0 seq_num 273) dirty exists 2021-07-31T01:57:26.670 INFO:tasks.rados.rados.0.smithi045.stdout:3560: finishing write tid 2 to smithi04569269-22 2021-07-31T01:57:26.670 INFO:tasks.rados.rados.0.smithi045.stdout:3559: finishing copy_from to smithi04569269-36 2021-07-31T01:57:26.670 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 36 v 1856 (ObjNum 996 snap 311 seq_num 996) dirty exists 2021-07-31T01:57:26.670 INFO:tasks.rados.rados.0.smithi045.stdout:3560: finishing write tid 3 to smithi04569269-22 2021-07-31T01:57:26.671 INFO:tasks.rados.rados.0.smithi045.stdout:3561: finishing copy_from racing read to smithi04569269-27 2021-07-31T01:57:26.671 INFO:tasks.rados.rados.0.smithi045.stdout:3559: finishing copy_from racing read to smithi04569269-36 2021-07-31T01:57:26.671 INFO:tasks.rados.rados.0.smithi045.stdout:3560: finishing write tid 4 to smithi04569269-22 2021-07-31T01:57:26.672 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 22 v 1850 (ObjNum 1092 snap 349 seq_num 1092) dirty exists 2021-07-31T01:57:26.672 INFO:tasks.rados.rados.0.smithi045.stdout:3560: left oid 22 (ObjNum 1092 snap 349 seq_num 1092) 2021-07-31T01:57:26.672 INFO:tasks.rados.rados.0.smithi045.stdout:3558: done (4 left) 2021-07-31T01:57:26.672 INFO:tasks.rados.rados.0.smithi045.stdout:3559: done (3 left) 2021-07-31T01:57:26.672 INFO:tasks.rados.rados.0.smithi045.stdout:3560: done (2 left) 2021-07-31T01:57:26.673 INFO:tasks.rados.rados.0.smithi045.stdout:3561: done (1 left) 2021-07-31T01:57:26.673 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 33 current snap is 349 2021-07-31T01:57:26.673 INFO:tasks.rados.rados.0.smithi045.stdout:3564: seq_num 1093 ranges {0=712704,712704=688128,1400832=114688} 2021-07-31T01:57:26.674 INFO:tasks.rados.rados.0.smithi045.stdout:3564: writing smithi04569269-33 from 0 to 712704 tid 1 2021-07-31T01:57:26.675 INFO:tasks.rados.rados.0.smithi045.stdout:3564: writing smithi04569269-33 from 712704 to 1400832 tid 2 2021-07-31T01:57:26.677 INFO:tasks.rados.rados.0.smithi045.stdout:3564: writing smithi04569269-33 from 1400832 to 1515520 tid 3 2021-07-31T01:57:26.677 INFO:tasks.rados.rados.0.smithi045.stdout:3565: read oid 7 snap 339 2021-07-31T01:57:26.677 INFO:tasks.rados.rados.0.smithi045.stdout:3565: expect deleted 2021-07-31T01:57:26.678 INFO:tasks.rados.rados.0.smithi045.stdout:3566: snap_remove snap 290 2021-07-31T01:57:27.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:26 smithi154 conmon[34553]: cluster 2021-07-31T01:57:25.775098+0000 mon.a (mon.0) 1396 : cluster [DBG] 2021-07-31T01:57:27.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:26 smithi154 conmon[34553]: osdmap e729: 8 total, 8 up, 8 in 2021-07-31T01:57:27.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:26 smithi154 conmon[34553]: cluster 2021-07-31T01:57:25.938138+0000 mgr.x (mgr.24542) 432 : cluster [DBG] pgmap v882: 25 pgs: 25 active+clean; 637 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 9.5 MiB/s rd, 2.8 MiB/s wr, 17 op/s 2021-07-31T01:57:27.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:26 smithi154 conmon[34553]: cluster 2021-07-31T01:57:26.665379+0000 mon.a (mon.0) 1397 : cluster 2021-07-31T01:57:27.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:26 smithi154 conmon[34553]: [DBG] osdmap e730: 8 total, 8 up, 8 in 2021-07-31T01:57:27.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:26 smithi045 conmon[33655]: cluster 2021-07-31T01:57:25 2021-07-31T01:57:27.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:26 smithi045 conmon[33655]: .775098+0000 mon.a (mon.0) 1396 : cluster [DBG] osdmap e729: 8 total, 8 up, 8 in 2021-07-31T01:57:27.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:26 smithi045 conmon[33655]: cluster 2021-07-31T01:57:25 2021-07-31T01:57:27.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:26 smithi045 conmon[33655]: .938138+0000 mgr.x (mgr.24542) 432 : cluster [DBG] pgmap v882: 25 pgs: 25 active+clean; 637 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 9.5 MiB/s rd, 2.8 MiB/s wr, 17 op/s 2021-07-31T01:57:27.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:26 smithi045 conmon[33655]: cluster 2021-07-31T01:57:27.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:26 smithi045 conmon[33655]: 2021-07-31T 2021-07-31T01:57:27.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:26 smithi045 conmon[33655]: 01:57:26.665379+0000 mon.a 2021-07-31T01:57:27.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:26 smithi045 conmon[33655]: (mon.0) 1397 : cluster [DBG] 2021-07-31T01:57:27.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:26 smithi045 conmon[33655]: osdmap e730: 8 total, 8 up, 8 in 2021-07-31T01:57:27.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:26 smithi045 conmon[39718]: cluster 2021-07-31T01:57:25.775098+0000 2021-07-31T01:57:27.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:26 smithi045 conmon[39718]: mon.a (mon.0) 1396 : cluster [DBG] osdmap e729: 8 total, 8 up, 8 in 2021-07-31T01:57:27.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:26 smithi045 conmon[39718]: cluster 2021- 2021-07-31T01:57:27.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:26 smithi045 conmon[39718]: 07-31T01:57:25.938138+0000 mgr.x (mgr.24542) 432 2021-07-31T01:57:27.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:26 smithi045 conmon[39718]: : cluster [DBG] pgmap v882: 25 pgs: 25 active+clean; 637 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 9.5 MiB/s rd, 2.8 MiB/s wr, 17 op/s 2021-07-31T01:57:27.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:26 smithi045 conmon[39718]: cluster 2021-07-31 2021-07-31T01:57:27.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:26 smithi045 conmon[39718]: T01:57:26.665379+0000 mon.a (mon.0) 1397 : cluster [DBG] 2021-07-31T01:57:27.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:26 smithi045 conmon[39718]: osdmap e730: 8 total, 8 up, 8 in 2021-07-31T01:57:27.677 INFO:tasks.rados.rados.0.smithi045.stdout:3566: done (3 left) 2021-07-31T01:57:27.677 INFO:tasks.rados.rados.0.smithi045.stdout:3567: copy_from oid 17 from oid 31 current snap is 349 2021-07-31T01:57:27.678 INFO:tasks.rados.rados.0.smithi045.stdout:3563: finishing rollback tid 0 to smithi04569269-16 2021-07-31T01:57:27.678 INFO:tasks.rados.rados.0.smithi045.stdout:3563: finishing rollback tid 1 to smithi04569269-16 2021-07-31T01:57:27.678 INFO:tasks.rados.rados.0.smithi045.stdout:3563: finishing rollback tid 2 to smithi04569269-16 2021-07-31T01:57:27.678 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 16 v 1853 (ObjNum 944 snap 286 seq_num 944) dirty exists 2021-07-31T01:57:27.679 INFO:tasks.rados.rados.0.smithi045.stdout:3564: finishing write tid 1 to smithi04569269-33 2021-07-31T01:57:27.679 INFO:tasks.rados.rados.0.smithi045.stdout:3564: finishing write tid 2 to smithi04569269-33 2021-07-31T01:57:27.679 INFO:tasks.rados.rados.0.smithi045.stdout:3564: finishing write tid 3 to smithi04569269-33 2021-07-31T01:57:27.679 INFO:tasks.rados.rados.0.smithi045.stdout:3564: finishing write tid 4 to smithi04569269-33 2021-07-31T01:57:27.680 INFO:tasks.rados.rados.0.smithi045.stdout:3564: finishing write tid 5 to smithi04569269-33 2021-07-31T01:57:27.680 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 33 v 2042 (ObjNum 1093 snap 349 seq_num 1093) dirty exists 2021-07-31T01:57:27.680 INFO:tasks.rados.rados.0.smithi045.stdout:3564: left oid 33 (ObjNum 1093 snap 349 seq_num 1093) 2021-07-31T01:57:27.680 INFO:tasks.rados.rados.0.smithi045.stdout:3563: done (3 left) 2021-07-31T01:57:27.681 INFO:tasks.rados.rados.0.smithi045.stdout:3564: done (2 left) 2021-07-31T01:57:27.681 INFO:tasks.rados.rados.0.smithi045.stdout:3565: done (1 left) 2021-07-31T01:57:27.681 INFO:tasks.rados.rados.0.smithi045.stdout:3568: snap_remove snap 294 2021-07-31T01:57:28.683 INFO:tasks.rados.rados.0.smithi045.stdout:3568: done (1 left) 2021-07-31T01:57:28.684 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 43 current snap is 349 2021-07-31T01:57:28.684 INFO:tasks.rados.rados.0.smithi045.stdout:3567: finishing copy_from to smithi04569269-17 2021-07-31T01:57:28.684 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 17 v 2125 (ObjNum 1006 snap 322 seq_num 1006) dirty exists 2021-07-31T01:57:28.684 INFO:tasks.rados.rados.0.smithi045.stdout:3569: seq_num 1094 ranges {2727936=729088,3457024=573440,4030464=229376} 2021-07-31T01:57:28.704 INFO:tasks.rados.rados.0.smithi045.stdout:3569: writing smithi04569269-43 from 2727936 to 3457024 tid 1 2021-07-31T01:57:28.708 INFO:tasks.rados.rados.0.smithi045.stdout:3569: writing smithi04569269-43 from 3457024 to 4030464 tid 2 2021-07-31T01:57:28.710 INFO:tasks.rados.rados.0.smithi045.stdout:3569: writing smithi04569269-43 from 4030464 to 4259840 tid 3 2021-07-31T01:57:28.710 INFO:tasks.rados.rados.0.smithi045.stdout:3570: snap_create 2021-07-31T01:57:28.710 INFO:tasks.rados.rados.0.smithi045.stdout:3567: finishing copy_from racing read to smithi04569269-17 2021-07-31T01:57:28.715 INFO:tasks.rados.rados.0.smithi045.stdout:3569: finishing write tid 1 to smithi04569269-43 2021-07-31T01:57:28.715 INFO:tasks.rados.rados.0.smithi045.stdout:3569: finishing write tid 2 to smithi04569269-43 2021-07-31T01:57:28.716 INFO:tasks.rados.rados.0.smithi045.stdout:3569: finishing write tid 3 to smithi04569269-43 2021-07-31T01:57:28.716 INFO:tasks.rados.rados.0.smithi045.stdout:3569: finishing write tid 4 to smithi04569269-43 2021-07-31T01:57:28.717 INFO:tasks.rados.rados.0.smithi045.stdout:3569: finishing write tid 5 to smithi04569269-43 2021-07-31T01:57:28.718 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 43 v 2055 (ObjNum 1094 snap 349 seq_num 1094) dirty exists 2021-07-31T01:57:28.718 INFO:tasks.rados.rados.0.smithi045.stdout:3569: left oid 43 (ObjNum 1094 snap 349 seq_num 1094) 2021-07-31T01:57:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:28 smithi154 conmon[34553]: cluster 2021-07-31T01:57:27.676174+0000 mon.a (mon.0) 1398 2021-07-31T01:57:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:28 smithi154 conmon[34553]: : cluster [DBG] osdmap e731: 8 total, 8 up, 8 in 2021-07-31T01:57:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:28 smithi154 conmon[34553]: cluster 2021-07-31T01:57:27.938592+0000 mgr.x (mgr.24542) 433 : cluster [DBG] pgmap v885: 25 pgs: 25 active+clean; 637 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T01:57:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:28 smithi154 conmon[34553]: audit 2021-07-31T 2021-07-31T01:57:29.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:28 smithi154 conmon[34553]: 01:57:27.973747+0000 mon.b (mon.2) 112 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:57:29.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:28 smithi154 conmon[34553]: audit 2021-07-31T01:57:29.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:28 smithi154 conmon[34553]: 2021-07-31T01:57:27.974348+0000 mon.a (mon.0) 1399 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:57:29.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:28 smithi154 conmon[34553]: audit 2021-07 2021-07-31T01:57:29.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:28 smithi154 conmon[34553]: -31T01:57:27.989951+0000 mon.b (mon.2) 113 : audit 2021-07-31T01:57:29.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:28 smithi154 conmon[34553]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T01:57:29.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:28 smithi154 conmon[34553]: audit 2021-07-31T 2021-07-31T01:57:29.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:28 smithi154 conmon[34553]: 01:57:27.990250+0000 mon.a (mon.0) 1400 : audit 2021-07-31T01:57:29.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:28 smithi154 conmon[34553]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T01:57:29.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:28 smithi045 conmon[33655]: cluster 2021-07-31T01:57:27.676174+0000 mon.a (mon.0) 2021-07-31T01:57:29.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:28 smithi045 conmon[33655]: 1398 : cluster [DBG] osdmap e731: 8 total, 8 up, 8 in 2021-07-31T01:57:29.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:28 smithi045 conmon[33655]: cluster 2021-07-31T01:57:27.938592+0000 mgr.x (mgr.24542) 433 : cluster [DBG] pgmap v885: 25 pgs: 25 active+clean; 637 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T01:57:29.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:28 smithi045 conmon[33655]: audit 2021-07-31T01:57:27.973747+0000 mon.b (mon.2) 112 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:57:29.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:28 smithi045 conmon[33655]: audit 2021-07-31T01:57:27.974348+0000 mon.a (mon.0) 1399 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:57:29.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:28 smithi045 conmon[33655]: 2021-07-31T01:57:29.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:28 smithi045 conmon[33655]: audit 2021 2021-07-31T01:57:29.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:28 smithi045 conmon[33655]: -07-31T01:57:27.989951+0000 mon.b ( 2021-07-31T01:57:29.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:28 smithi045 conmon[33655]: mon.2) 113 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T01:57:29.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:28 smithi045 conmon[33655]: audit 2021-07-31 2021-07-31T01:57:29.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:28 smithi045 conmon[33655]: T01:57:27.990250+0000 mon.a (mon.0) 1400 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T01:57:29.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:28 smithi045 conmon[39718]: cluster 2021-07-31T01:57:27.676174+0000 mon.a 2021-07-31T01:57:29.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:28 smithi045 conmon[39718]: (mon.0) 1398 : cluster [DBG] osdmap e731: 8 total, 8 up, 8 in 2021-07-31T01:57:29.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:28 smithi045 conmon[39718]: cluster 2021-07-31T01:57 2021-07-31T01:57:29.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:28 smithi045 conmon[39718]: :27.938592+0000 mgr.x (mgr.24542) 433 : cluster [DBG] pgmap v885: 25 pgs: 25 active+clean; 637 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T01:57:29.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:28 smithi045 conmon[39718]: audit 2021 2021-07-31T01:57:29.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:28 smithi045 conmon[39718]: -07-31T01:57:27.973747+0000 mon.b (mon 2021-07-31T01:57:29.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:28 smithi045 conmon[39718]: .2) 112 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:57:29.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:28 smithi045 conmon[39718]: audit 2021-07-31T01 2021-07-31T01:57:29.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:28 smithi045 conmon[39718]: :57:27.974348+0000 mon.a (mon.0) 1399 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:57:29.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:28 smithi045 conmon[39718]: 2021-07-31T01:57:29.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:28 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T01:57:29.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:28 smithi045 conmon[39718]: T01:57:27.989951+0000 mon.b (mon. 2021-07-31T01:57:29.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:28 smithi045 conmon[39718]: 2) 113 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T01:57:29.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:28 smithi045 conmon[39718]: audit 2021-07-31T01:57:29.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:28 smithi045 conmon[39718]: 2021-07-31T01 2021-07-31T01:57:29.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:28 smithi045 conmon[39718]: :57:27.990250+0000 mon.a (mon.0 2021-07-31T01:57:29.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:28 smithi045 conmon[39718]: ) 1400 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T01:57:29.503 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: debug 2021-07-31T01:57:29.207+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T01:57:29.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:57:29.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:57:29.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: yield 2021-07-31T01:57:29.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:57:29.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:57:29.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:57:29.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T01:57:29.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:57:29.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T01:57:29.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:57:29.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T01:57:29.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:57:29.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: task.result() 2021-07-31T01:57:29.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T01:57:29.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: debug 2021-07-31T01:57:29.218+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T01:57:29.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:57:29.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:57:29.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: yield 2021-07-31T01:57:29.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:57:29.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:57:29.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:57:29.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T01:57:29.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:57:29.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T01:57:29.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:57:29.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T01:57:29.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:57:29.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: task.result() 2021-07-31T01:57:29.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T01:57:29.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: debug 2021-07-31T01:57:29.244+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T01:57:29.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:57:29.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:57:29.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: yield 2021-07-31T01:57:29.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:57:29.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:57:29.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:57:29.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T01:57:29.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:57:29.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T01:57:29.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:57:29.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T01:57:29.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:57:29.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: task.result() 2021-07-31T01:57:29.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T01:57:29.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: debug 2021-07-31T01:57:29.244+0000 7fcb93961700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:57:29.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:57:29.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:57:29.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: yield 2021-07-31T01:57:29.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:57:29.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:57:29.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:57:29.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T01:57:29.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:57:29.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T01:57:29.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:57:29.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T01:57:29.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:57:29.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: task.result() 2021-07-31T01:57:29.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T01:57:29.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: 2021-07-31T01:57:29.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T01:57:29.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: 2021-07-31T01:57:29.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:57:29.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:57:29.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: return f(*arg) 2021-07-31T01:57:29.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:57:29.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:57:29.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:57:29.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:57:29.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:57:29.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:57:29.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:57:29.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T01:57:29.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:57:29.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: raise self._exception 2021-07-31T01:57:29.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:57:29.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:57:29.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:57:29.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T01:57:29.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:57:29.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: raise 2021-07-31T01:57:29.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:57:29.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T01:57:29.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:57:29.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:57:29.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T01:57:29.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: debug 2021-07-31T01:57:29.254+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T01:57:29.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:57:29.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:57:29.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: yield 2021-07-31T01:57:29.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:57:29.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:57:29.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:57:29.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T01:57:29.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:57:29.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T01:57:29.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:57:29.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T01:57:29.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:57:29.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: task.result() 2021-07-31T01:57:29.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T01:57:29.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: debug 2021-07-31T01:57:29.254+0000 7fcb94162700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:57:29.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:57:29.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:57:29.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: yield 2021-07-31T01:57:29.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:57:29.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:57:29.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:57:29.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T01:57:29.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:57:29.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T01:57:29.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:57:29.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T01:57:29.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:57:29.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: task.result() 2021-07-31T01:57:29.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T01:57:29.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: 2021-07-31T01:57:29.535 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T01:57:29.535 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: 2021-07-31T01:57:29.535 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:57:29.535 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:57:29.536 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: return f(*arg) 2021-07-31T01:57:29.536 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:57:29.536 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:57:29.536 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:57:29.537 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:57:29.537 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:57:29.537 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:57:29.537 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:57:29.538 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T01:57:29.538 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:57:29.538 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: raise self._exception 2021-07-31T01:57:29.538 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:57:29.539 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:57:29.539 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:57:29.539 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T01:57:29.539 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:57:29.540 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: raise 2021-07-31T01:57:29.540 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:57:29.540 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T01:57:29.540 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:57:29.541 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:57:29.541 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T01:57:29.688 INFO:tasks.rados.rados.0.smithi045.stdout:3567: done (2 left) 2021-07-31T01:57:29.689 INFO:tasks.rados.rados.0.smithi045.stdout:3569: done (1 left) 2021-07-31T01:57:29.689 INFO:tasks.rados.rados.0.smithi045.stdout:3570: done (0 left) 2021-07-31T01:57:29.689 INFO:tasks.rados.rados.0.smithi045.stdout:3571: snap_remove snap 330 2021-07-31T01:57:30.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[34553]: cluster 2021-07-31T01:57:28.682520+0000 mon.a (mon.0) 1401 : cluster [DBG] osdmap e732: 8 total, 8 up, 8 in 2021-07-31T01:57:30.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[34553]: audit 2021-07-31T01:57:29.161465+0000 mon.b (mon.2) 114 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:57:30.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[34553]: audit 2021-07-31T01:57:29.162967 2021-07-31T01:57:30.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[34553]: +0000 mon.b (mon.2) 115 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:57:30.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[34553]: audit 2021-07-31T01:57:29.163819+0000 mon.b (mon.2) 116 : audit 2021-07-31T01:57:30.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:29 smithi154 conmon[34553]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:57:30.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:29 smithi045 conmon[33655]: cluster 2021-07-31T01:57:28.682520+0000 mon.a (mon.0) 1401 : cluster 2021-07-31T01:57:30.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:29 smithi045 conmon[33655]: [DBG] osdmap e732: 8 total, 8 up, 8 in 2021-07-31T01:57:30.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:29 smithi045 conmon[33655]: audit 2021-07-31T01:57:29.161465+0000 mon.b (mon.2) 114 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:57:30.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:29 smithi045 conmon[33655]: audit 2021-07-31T01:57 2021-07-31T01:57:30.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:29 smithi045 conmon[33655]: :29.162967+0000 mon.b (mon.2) 115 : audit [DBG] 2021-07-31T01:57:30.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:29 smithi045 conmon[33655]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:57:30.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:29 smithi045 conmon[33655]: audit 2021-07- 2021-07-31T01:57:30.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:29 smithi045 conmon[33655]: 31T01:57:29.163819+0000 mon.b (mon 2021-07-31T01:57:30.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:29 smithi045 conmon[33655]: .2) 116 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:57:30.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:29 smithi045 conmon[39718]: cluster 2021-07-31T01:57:28.682520+0000 mon.a (mon. 2021-07-31T01:57:30.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:29 smithi045 conmon[39718]: 0) 1401 : cluster [DBG] osdmap e732: 8 total, 8 up, 8 in 2021-07-31T01:57:30.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:29 smithi045 conmon[39718]: audit 2021-07-31T01:57:29.161465+0000 mon.b (mon.2) 114 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:57:30.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:29 smithi045 conmon[39718]: 2021-07-31T01:57:30.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:29 smithi045 conmon[39718]: audit 2021-07-31T01:57:29. 2021-07-31T01:57:30.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:29 smithi045 conmon[39718]: 162967+0000 mon.b (mon.2) 115 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:57:30.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:29 smithi045 conmon[39718]: 2021-07-31T01:57:30.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:29 smithi045 conmon[39718]: audit 2021-07-31T01:57: 2021-07-31T01:57:30.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:29 smithi045 conmon[39718]: 29.163819+0000 mon.b (mon.2) 116 2021-07-31T01:57:30.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:29 smithi045 conmon[39718]: : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:57:30.698 INFO:tasks.rados.rados.0.smithi045.stdout:3571: done (0 left) 2021-07-31T01:57:30.699 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 20 current snap is 350 2021-07-31T01:57:30.699 INFO:tasks.rados.rados.0.smithi045.stdout:3572: seq_num 1095 ranges {2088960=745472,2834432=499712,3334144=303104} 2021-07-31T01:57:30.716 INFO:tasks.rados.rados.0.smithi045.stdout:3572: writing smithi04569269-20 from 2088960 to 2834432 tid 1 2021-07-31T01:57:30.719 INFO:tasks.rados.rados.0.smithi045.stdout:3572: writing smithi04569269-20 from 2834432 to 3334144 tid 2 2021-07-31T01:57:30.723 INFO:tasks.rados.rados.0.smithi045.stdout:3572: writing smithi04569269-20 from 3334144 to 3637248 tid 3 2021-07-31T01:57:30.723 INFO:tasks.rados.rados.0.smithi045.stdout:3573: read oid 45 snap -1 2021-07-31T01:57:30.723 INFO:tasks.rados.rados.0.smithi045.stdout:3573: expect (ObjNum 944 snap 286 seq_num 944) 2021-07-31T01:57:30.724 INFO:tasks.rados.rados.0.smithi045.stdout:3574: read oid 26 snap -1 2021-07-31T01:57:30.724 INFO:tasks.rados.rados.0.smithi045.stdout:3574: expect (ObjNum 1086 snap 347 seq_num 1086) 2021-07-31T01:57:30.725 INFO:tasks.rados.rados.0.smithi045.stdout:3575: snap_remove snap 346 2021-07-31T01:57:31.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: cephadm 2021-07-31T01:57:29.164573+0000 mgr.x (mgr.24542) 434 : cephadm 2021-07-31T01:57:31.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T01:57:31.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: cephadm 2021-07-31T01:57:29.208072+0000 mgr.x (mgr.24542) 435 : cephadm [ERR] 2021-07-31T01:57:31.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:57:31.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:57:31.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: yield 2021-07-31T01:57:31.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:57:31.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:57:31.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:57:31.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T01:57:31.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:57:31.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T01:57:31.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:57:31.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T01:57:31.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:57:31.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: task.result() 2021-07-31T01:57:31.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T01:57:31.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: cephadm 2021- 2021-07-31T01:57:31.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: 07-31T01:57:29.208479+0000 mgr.x (mgr.24542) 436 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T01:57:31.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: cephadm 2021-07-31T01:57:29.219554+0000 mgr.x (mgr.24542) 437 : cephadm [ERR] 2021-07-31T01:57:31.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:57:31.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:57:31.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: yield 2021-07-31T01:57:31.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:57:31.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:57:31.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:57:31.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T01:57:31.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:57:31.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T01:57:31.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:57:31.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T01:57:31.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:57:31.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: task.result() 2021-07-31T01:57:31.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T01:57:31.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: cephadm 2021-07-31T01:57:29.220410+0000 mgr.x (mgr.24542) 438 2021-07-31T01:57:31.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T01:57:31.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: cephadm 2021-07-31T01:57:29.245069+0000 mgr.x (mgr.24542) 439 : cephadm [ERR] 2021-07-31T01:57:31.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:57:31.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:57:31.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: yield 2021-07-31T01:57:31.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:57:31.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:57:31.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:57:31.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T01:57:31.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:57:31.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T01:57:31.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:57:31.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T01:57:31.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:57:31.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: task.result() 2021-07-31T01:57:31.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T01:57:31.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: cephadm 2021-07-31T01:57: 2021-07-31T01:57:31.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: 29.245598+0000 mgr.x (mgr.24542) 440 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:57:31.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:57:31.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:57:31.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: yield 2021-07-31T01:57:31.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:57:31.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:57:31.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:57:31.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T01:57:31.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:57:31.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T01:57:31.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:57:31.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T01:57:31.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:57:31.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: task.result() 2021-07-31T01:57:31.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T01:57:31.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: 2021-07-31T01:57:31.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T01:57:31.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: 2021-07-31T01:57:31.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:57:31.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:57:31.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: return f(*arg) 2021-07-31T01:57:31.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:57:31.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:57:31.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:57:31.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:57:31.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:57:31.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:57:31.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:57:31.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T01:57:31.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:57:31.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: raise self._exception 2021-07-31T01:57:31.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:57:31.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:57:31.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:57:31.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T01:57:31.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:57:31.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: raise 2021-07-31T01:57:31.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:57:31.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T01:57:31.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:57:31.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:57:31.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T01:57:31.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: cephadm 2021-07-31T01:57:29.254969+0000 mgr.x (mgr.24542) 441 : cephadm [ERR] 2021-07-31T01:57:31.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:57:31.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:57:31.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: yield 2021-07-31T01:57:31.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:57:31.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:57:31.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:57:31.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T01:57:31.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:57:31.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T01:57:31.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:57:31.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T01:57:31.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:57:31.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: task.result() 2021-07-31T01:57:31.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T01:57:31.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: cephadm 2021-07-31T01:57:29.255368+0000 mgr.x (mgr.24542) 2021-07-31T01:57:31.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: 442 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:57:31.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:57:31.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:57:31.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: yield 2021-07-31T01:57:31.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:57:31.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:57:31.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:57:31.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T01:57:31.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:57:31.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T01:57:31.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:57:31.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T01:57:31.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:57:31.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: task.result() 2021-07-31T01:57:31.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T01:57:31.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: 2021-07-31T01:57:31.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T01:57:31.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: 2021-07-31T01:57:31.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:57:31.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:57:31.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: return f(*arg) 2021-07-31T01:57:31.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:57:31.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:57:31.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:57:31.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:57:31.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:57:31.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:57:31.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:57:31.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T01:57:31.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:57:31.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: raise self._exception 2021-07-31T01:57:31.035 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:57:31.035 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:57:31.035 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:57:31.035 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T01:57:31.035 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:57:31.036 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: raise 2021-07-31T01:57:31.036 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:57:31.036 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T01:57:31.036 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:57:31.036 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:57:31.037 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T01:57:31.037 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: cluster 2021-07-31T01:57:29.688047+0000 mon.a (mon.0) 1402 : cluster [DBG] osdmap e733: 8 total, 8 up, 8 in 2021-07-31T01:57:31.037 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:30 smithi154 conmon[34553]: cluster 2021-07-31T01:57:29.939071+0000 mgr.x (mgr.24542) 443 : cluster [DBG] pgmap v888: 25 pgs: 25 active+clean; 623 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.5 KiB/s rd, 3.1 MiB/s wr, 14 op/s 2021-07-31T01:57:31.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: cephadm 2021-07-31T01:57:29.164573+0000 mgr.x (mgr.24542) 434 : cephadm 2021-07-31T01:57:31.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T01:57:31.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: cephadm 2021-07-31T01:57:29.208072+0000 mgr.x (mgr.24542) 435 : cephadm [ERR] 2021-07-31T01:57:31.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:57:31.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:57:31.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: yield 2021-07-31T01:57:31.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:57:31.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:57:31.040 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:57:31.040 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T01:57:31.040 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:57:31.040 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T01:57:31.040 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:57:31.041 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T01:57:31.041 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:57:31.041 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: task.result() 2021-07-31T01:57:31.041 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T01:57:31.042 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: cephadm 2021-07-31T01:57:31.042 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: 2021-07-31T01:57:29.208479+0000 mgr.x (mgr.24542) 436 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T01:57:31.042 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: cephadm 2021-07-31T01:57:29.219554+0000 mgr.x (mgr.24542) 437 : cephadm [ERR] 2021-07-31T01:57:31.042 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:57:31.042 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:57:31.043 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: yield 2021-07-31T01:57:31.043 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:57:31.043 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:57:31.043 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:57:31.043 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T01:57:31.044 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:57:31.044 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T01:57:31.044 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:57:31.044 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T01:57:31.044 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:57:31.045 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: task.result() 2021-07-31T01:57:31.045 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T01:57:31.045 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: cephadm 2021-07-31T01:57:29.220410+0000 mgr.x (mgr.24542) 438 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T01:57:31.045 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: cephadm 2021-07-31T01:57:29.245069+0000 mgr.x 2021-07-31T01:57:31.046 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: (mgr.24542) 439 : cephadm [ERR] 2021-07-31T01:57:31.046 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:57:31.046 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:57:31.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: cephadm 2021-07-31T01:57:29 2021-07-31T01:57:31.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: .164573+0000 mgr.x (mgr.24542) 434 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T01:57:31.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: cephadm 2021-07-31T01:57:31.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: 2021-07-31T01 2021-07-31T01:57:31.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: :57:29.208072+0000 2021-07-31T01:57:31.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: mgr.x (mgr.24542) 435 2021-07-31T01:57:31.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: : cephadm [ERR] 2021-07-31T01:57:31.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:57:31.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:57:31.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: yield 2021-07-31T01:57:31.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:57:31.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:57:31.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:57:31.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T01:57:31.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:57:31.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T01:57:31.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:57:31.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T01:57:31.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:57:31.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: task.result() 2021-07-31T01:57:31.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T01:57:31.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: cephadm 2021-07-31T01:57:29.208479+0000 mgr.x (mgr. 2021-07-31T01:57:31.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: 24542) 436 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T01:57:31.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: 2021-07-31T01:57:31.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: cephadm 2021 2021-07-31T01:57:31.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: -07-31T01:57:29. 2021-07-31T01:57:31.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: 219554+0000 mgr.x (mgr. 2021-07-31T01:57:31.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: 24542) 437 : cephadm [ERR] 2021-07-31T01:57:31.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: 2021-07-31T01:57:31.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:57:31.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:57:31.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: yield 2021-07-31T01:57:31.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:57:31.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:57:31.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:57:31.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T01:57:31.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:57:31.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T01:57:31.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:57:31.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T01:57:31.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:57:31.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: task.result() 2021-07-31T01:57:31.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T01:57:31.056 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: yield 2021-07-31T01:57:31.056 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:57:31.056 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:57:31.056 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:57:31.057 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T01:57:31.057 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:57:31.057 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T01:57:31.057 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:57:31.057 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T01:57:31.058 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:57:31.058 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: task.result() 2021-07-31T01:57:31.058 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T01:57:31.058 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: cephadm 2021-07-31T01:57:29.245598+0000 mgr.x (mgr.24542) 440 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:57:31.059 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:57:31.059 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:57:31.059 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: yield 2021-07-31T01:57:31.059 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:57:31.059 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:57:31.060 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:57:31.060 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T01:57:31.060 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:57:31.060 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T01:57:31.060 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:57:31.061 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T01:57:31.061 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:57:31.061 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: task.result() 2021-07-31T01:57:31.061 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T01:57:31.061 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: 2021-07-31T01:57:31.062 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T01:57:31.062 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: 2021-07-31T01:57:31.062 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:57:31.062 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:57:31.062 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: return f(*arg) 2021-07-31T01:57:31.063 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:57:31.063 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:57:31.063 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:57:31.063 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:57:31.063 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:57:31.064 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:57:31.064 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:57:31.064 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: cephadm 2021-07-31T01:57:29.220410+0000 mgr.x (mgr.24542) 438 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T01:57:31.065 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: 2021-07-31T01:57:31.065 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: cephadm 2021-07-31T01:57:31.065 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: 2021-07-31T01:57:29.245069+0000 mgr.x (mgr.24542) 2021-07-31T01:57:31.065 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: 439 : cephadm [ERR] 2021-07-31T01:57:31.066 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:57:31.066 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:57:31.066 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: yield 2021-07-31T01:57:31.066 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:57:31.066 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:57:31.067 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:57:31.067 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T01:57:31.067 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:57:31.067 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T01:57:31.067 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:57:31.068 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T01:57:31.068 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:57:31.068 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: task.result() 2021-07-31T01:57:31.068 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T01:57:31.068 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: cephadm 2021-07-31T01:57:29.245598+0000 mgr.x (mgr.24542) 440 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:57:31.069 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:57:31.069 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:57:31.069 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: yield 2021-07-31T01:57:31.069 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:57:31.069 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:57:31.070 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:57:31.070 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T01:57:31.070 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:57:31.070 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T01:57:31.071 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:57:31.071 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T01:57:31.071 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:57:31.071 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: task.result() 2021-07-31T01:57:31.071 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T01:57:31.072 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: 2021-07-31T01:57:31.072 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T01:57:31.072 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: 2021-07-31T01:57:31.072 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:57:31.072 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:57:31.073 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: return f(*arg) 2021-07-31T01:57:31.073 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:57:31.073 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:57:31.073 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:57:31.074 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T01:57:31.074 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:57:31.074 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: raise self._exception 2021-07-31T01:57:31.074 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:57:31.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:57:31.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:57:31.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T01:57:31.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:57:31.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: raise 2021-07-31T01:57:31.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:57:31.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T01:57:31.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:57:31.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:57:31.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T01:57:31.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: cephadm 2021-07-31T01:57:29.254969+0000 mgr.x (mgr.24542) 441 : cephadm [ERR] 2021-07-31T01:57:31.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:57:31.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:57:31.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: yield 2021-07-31T01:57:31.078 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:57:31.078 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:57:31.078 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:57:31.078 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T01:57:31.078 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:57:31.079 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T01:57:31.079 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:57:31.079 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T01:57:31.079 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:57:31.079 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: task.result() 2021-07-31T01:57:31.080 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T01:57:31.080 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: cephadm 2021-07-31T01:57:29.255368+0000 mgr.x (mgr.24542) 442 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:57:31.080 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:57:31.080 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:57:31.080 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: yield 2021-07-31T01:57:31.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:57:31.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:57:31.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:57:31.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T01:57:31.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:57:31.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T01:57:31.082 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:57:31.082 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:57:31.083 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:57:31.083 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:57:31.083 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T01:57:31.083 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:57:31.083 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: raise self._exception 2021-07-31T01:57:31.084 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:57:31.084 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:57:31.084 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:57:31.084 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T01:57:31.084 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:57:31.085 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: raise 2021-07-31T01:57:31.085 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:57:31.085 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T01:57:31.085 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:57:31.086 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:57:31.086 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T01:57:31.086 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: cephadm 2021-07-31T01:57:29.254969+0000 mgr.x (mgr.24542) 441 : cephadm [ERR] 2021-07-31T01:57:31.086 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:57:31.086 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:57:31.087 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: yield 2021-07-31T01:57:31.087 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:57:31.087 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:57:31.087 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:57:31.088 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T01:57:31.088 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:57:31.088 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T01:57:31.088 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:57:31.088 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T01:57:31.089 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:57:31.089 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: task.result() 2021-07-31T01:57:31.089 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T01:57:31.089 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: cephadm 2021-07-31T01:57:29.255368+0000 mgr.x (mgr.24542) 442 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:57:31.089 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:57:31.090 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:57:31.090 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: yield 2021-07-31T01:57:31.090 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:57:31.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:57:31.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T01:57:31.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:57:31.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: task.result() 2021-07-31T01:57:31.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T01:57:31.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: 2021-07-31T01:57:31.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T01:57:31.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: 2021-07-31T01:57:31.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:57:31.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:57:31.093 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: return f(*arg) 2021-07-31T01:57:31.093 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:57:31.093 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:57:31.093 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:57:31.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:57:31.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:57:31.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:57:31.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:57:31.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T01:57:31.095 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:57:31.095 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: raise self._exception 2021-07-31T01:57:31.095 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:57:31.095 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:57:31.095 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:57:31.096 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T01:57:31.096 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:57:31.096 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: raise 2021-07-31T01:57:31.096 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:57:31.096 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T01:57:31.097 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:57:31.097 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:57:31.097 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T01:57:31.097 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: cluster 2021-07-31T01:57:29.688047+0000 mon.a (mon.0) 1402 : cluster [DBG] osdmap e733: 8 total, 8 up, 8 in 2021-07-31T01:57:31.097 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[33655]: cluster 2021-07-31T01:57:29.939071+0000 mgr.x (mgr.24542) 443 : cluster [DBG] pgmap v888: 25 pgs: 25 active+clean; 623 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.5 KiB/s rd, 3.1 MiB/s wr, 14 op/s 2021-07-31T01:57:31.098 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:57:31.098 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:57:31.099 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T01:57:31.099 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:57:31.099 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T01:57:31.099 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:57:31.099 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T01:57:31.100 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:57:31.100 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: task.result() 2021-07-31T01:57:31.100 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T01:57:31.100 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: 2021-07-31T01:57:31.101 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T01:57:31.101 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: 2021-07-31T01:57:31.101 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:57:31.101 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:57:31.101 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: return f(*arg) 2021-07-31T01:57:31.102 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:57:31.102 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:57:31.102 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:57:31.102 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:57:31.102 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:57:31.103 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:57:31.103 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:57:31.103 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T01:57:31.103 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:57:31.103 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: raise self._exception 2021-07-31T01:57:31.104 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:57:31.104 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:57:31.104 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:57:31.104 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T01:57:31.104 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:57:31.105 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: raise 2021-07-31T01:57:31.105 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:57:31.105 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T01:57:31.105 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:57:31.105 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:57:31.106 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T01:57:31.106 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: cluster 2021-07-31T01:57:29.688047+0000 mon.a (mon.0) 1402 : cluster [DBG] osdmap e733: 8 total, 8 up, 8 in 2021-07-31T01:57:31.106 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:30 smithi045 conmon[39718]: cluster 2021-07-31T01:57:29.939071+0000 mgr.x (mgr.24542) 443 : cluster [DBG] pgmap v888: 25 pgs: 25 active+clean; 623 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.5 KiB/s rd, 3.1 MiB/s wr, 14 op/s 2021-07-31T01:57:31.668 INFO:tasks.rados.rados.0.smithi045.stdout:3575: done (3 left) 2021-07-31T01:57:31.668 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 9 current snap is 350 2021-07-31T01:57:31.669 INFO:tasks.rados.rados.0.smithi045.stdout:3576: seq_num 1096 ranges {0=761856,761856=786432,1548288=16384} 2021-07-31T01:57:31.673 INFO:tasks.rados.rados.0.smithi045.stdout:3576: writing smithi04569269-9 from 0 to 761856 tid 1 2021-07-31T01:57:31.678 INFO:tasks.rados.rados.0.smithi045.stdout:3576: writing smithi04569269-9 from 761856 to 1548288 tid 2 2021-07-31T01:57:31.678 INFO:tasks.rados.rados.0.smithi045.stdout:3576: writing smithi04569269-9 from 1548288 to 1564672 tid 3 2021-07-31T01:57:31.679 INFO:tasks.rados.rados.0.smithi045.stdout:3577: read oid 10 snap 335 2021-07-31T01:57:31.679 INFO:tasks.rados.rados.0.smithi045.stdout:3577: expect (ObjNum 754 snap 233 seq_num 754) 2021-07-31T01:57:31.679 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 28 current snap is 350 2021-07-31T01:57:31.679 INFO:tasks.rados.rados.0.smithi045.stdout:3578: seq_num 1097 ranges {2727936=778240,3506176=671744,4177920=131072} 2021-07-31T01:57:31.699 INFO:tasks.rados.rados.0.smithi045.stdout:3578: writing smithi04569269-28 from 2727936 to 3506176 tid 1 2021-07-31T01:57:31.702 INFO:tasks.rados.rados.0.smithi045.stdout:3578: writing smithi04569269-28 from 3506176 to 4177920 tid 2 2021-07-31T01:57:31.703 INFO:tasks.rados.rados.0.smithi045.stdout:3578: writing smithi04569269-28 from 4177920 to 4308992 tid 3 2021-07-31T01:57:31.704 INFO:tasks.rados.rados.0.smithi045.stdout:3579: rmattr oid 46 current snap is 350 2021-07-31T01:57:31.704 INFO:tasks.rados.rados.0.smithi045.stdout:3579: done (6 left) 2021-07-31T01:57:31.704 INFO:tasks.rados.rados.0.smithi045.stdout:3580: setattr oid 25 current snap is 350 2021-07-31T01:57:31.704 INFO:tasks.rados.rados.0.smithi045.stdout:3572: finishing write tid 1 to smithi04569269-20 2021-07-31T01:57:31.705 INFO:tasks.rados.rados.0.smithi045.stdout:3572: finishing write tid 2 to smithi04569269-20 2021-07-31T01:57:31.705 INFO:tasks.rados.rados.0.smithi045.stdout:3572: finishing write tid 3 to smithi04569269-20 2021-07-31T01:57:31.705 INFO:tasks.rados.rados.0.smithi045.stdout:3572: finishing write tid 4 to smithi04569269-20 2021-07-31T01:57:31.705 INFO:tasks.rados.rados.0.smithi045.stdout:3572: finishing write tid 5 to smithi04569269-20 2021-07-31T01:57:31.705 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 20 v 2484 (ObjNum 1095 snap 350 seq_num 1095) dirty exists 2021-07-31T01:57:31.706 INFO:tasks.rados.rados.0.smithi045.stdout:3572: left oid 20 (ObjNum 1095 snap 350 seq_num 1095) 2021-07-31T01:57:31.706 INFO:tasks.rados.rados.0.smithi045.stdout:3573: expect (ObjNum 944 snap 286 seq_num 944) 2021-07-31T01:57:31.746 INFO:tasks.rados.rados.0.smithi045.stdout:3574: expect (ObjNum 1086 snap 347 seq_num 1086) 2021-07-31T01:57:31.960 INFO:tasks.rados.rados.0.smithi045.stdout:3576: finishing write tid 1 to smithi04569269-9 2021-07-31T01:57:31.960 INFO:tasks.rados.rados.0.smithi045.stdout:3576: finishing write tid 2 to smithi04569269-9 2021-07-31T01:57:31.961 INFO:tasks.rados.rados.0.smithi045.stdout:3576: finishing write tid 3 to smithi04569269-9 2021-07-31T01:57:31.961 INFO:tasks.rados.rados.0.smithi045.stdout:3576: finishing write tid 4 to smithi04569269-9 2021-07-31T01:57:31.961 INFO:tasks.rados.rados.0.smithi045.stdout:3576: finishing write tid 5 to smithi04569269-9 2021-07-31T01:57:31.961 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 9 v 2359 (ObjNum 1096 snap 350 seq_num 1096) dirty exists 2021-07-31T01:57:31.962 INFO:tasks.rados.rados.0.smithi045.stdout:3576: left oid 9 (ObjNum 1096 snap 350 seq_num 1096) 2021-07-31T01:57:31.962 INFO:tasks.rados.rados.0.smithi045.stdout:3577: expect (ObjNum 754 snap 233 seq_num 754) 2021-07-31T01:57:32.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:31 smithi154 conmon[34553]: cluster 2021-07-31T01:57:30.697037+0000 mon.a (mon.0) 1403 : cluster 2021-07-31T01:57:32.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:31 smithi154 conmon[34553]: [DBG] osdmap e734: 8 total, 8 up, 8 in 2021-07-31T01:57:32.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:31 smithi154 conmon[34553]: cluster 2021-07-31T01:57:31.667100+0000 mon.a (mon.0) 1404 : cluster [DBG] osdmap e735: 8 total, 8 up, 8 in 2021-07-31T01:57:32.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:31 smithi045 conmon[39718]: cluster 2021-07-31T01:57:30.697037+0000 mon.a ( 2021-07-31T01:57:32.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:31 smithi045 conmon[39718]: mon.0) 1403 : cluster [DBG] osdmap e734: 8 total, 8 up, 8 in 2021-07-31T01:57:32.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:31 smithi045 conmon[39718]: cluster 2021-07-31T01:57 2021-07-31T01:57:32.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:31 smithi045 conmon[39718]: :31.667100+0000 mon.a (mon.0) 1404 : cluster [DBG] osdmap e735: 8 total, 8 up, 8 in 2021-07-31T01:57:32.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:31 smithi045 conmon[33655]: cluster 2021-07-31T01:57:32.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:31 smithi045 conmon[33655]: 2021-07-31T01:57 2021-07-31T01:57:32.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:31 smithi045 conmon[33655]: :30.697037+0000 mon.a (mon.0) 1403 : cluster [DBG] osdmap e734: 8 total, 8 up, 8 in 2021-07-31T01:57:32.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:31 smithi045 conmon[33655]: cluster 2021-07 2021-07-31T01:57:32.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:31 smithi045 conmon[33655]: -31T01:57:31.667100+0000 mon.a (mon.0 2021-07-31T01:57:32.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:31 smithi045 conmon[33655]: ) 1404 : cluster [DBG] osdmap e735: 8 total, 8 up, 8 in 2021-07-31T01:57:32.669 INFO:tasks.rados.rados.0.smithi045.stdout:3578: finishing write tid 1 to smithi04569269-28 2021-07-31T01:57:32.670 INFO:tasks.rados.rados.0.smithi045.stdout:3578: finishing write tid 2 to smithi04569269-28 2021-07-31T01:57:32.670 INFO:tasks.rados.rados.0.smithi045.stdout:3578: finishing write tid 3 to smithi04569269-28 2021-07-31T01:57:32.670 INFO:tasks.rados.rados.0.smithi045.stdout:3578: finishing write tid 4 to smithi04569269-28 2021-07-31T01:57:32.671 INFO:tasks.rados.rados.0.smithi045.stdout:3578: finishing write tid 5 to smithi04569269-28 2021-07-31T01:57:32.671 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 28 v 2339 (ObjNum 1097 snap 350 seq_num 1097) dirty exists 2021-07-31T01:57:32.671 INFO:tasks.rados.rados.0.smithi045.stdout:3578: left oid 28 (ObjNum 1097 snap 350 seq_num 1097) 2021-07-31T01:57:32.671 INFO:tasks.rados.rados.0.smithi045.stdout:3572: done (6 left) 2021-07-31T01:57:32.671 INFO:tasks.rados.rados.0.smithi045.stdout:3573: done (5 left) 2021-07-31T01:57:32.672 INFO:tasks.rados.rados.0.smithi045.stdout:3574: done (4 left) 2021-07-31T01:57:32.672 INFO:tasks.rados.rados.0.smithi045.stdout:3576: done (3 left) 2021-07-31T01:57:32.674 INFO:tasks.rados.rados.0.smithi045.stdout:3577: done (2 left) 2021-07-31T01:57:32.674 INFO:tasks.rados.rados.0.smithi045.stdout:3578: done (1 left) 2021-07-31T01:57:32.674 INFO:tasks.rados.rados.0.smithi045.stdout:3581: read oid 21 snap -1 2021-07-31T01:57:32.674 INFO:tasks.rados.rados.0.smithi045.stdout:3581: expect (ObjNum 1013 snap 323 seq_num 1013) 2021-07-31T01:57:32.675 INFO:tasks.rados.rados.0.smithi045.stdout:3582: rollback oid 24 current snap is 350 2021-07-31T01:57:32.675 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 24 to 293 2021-07-31T01:57:32.675 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 13 current snap is 350 2021-07-31T01:57:32.675 INFO:tasks.rados.rados.0.smithi045.stdout:3583: seq_num 1098 ranges {0=794624,794624=557056,1351680=245760} 2021-07-31T01:57:32.676 INFO:tasks.rados.rados.0.smithi045.stdout:3583: writing smithi04569269-13 from 0 to 794624 tid 1 2021-07-31T01:57:32.677 INFO:tasks.rados.rados.0.smithi045.stdout:3583: writing smithi04569269-13 from 794624 to 1351680 tid 2 2021-07-31T01:57:32.679 INFO:tasks.rados.rados.0.smithi045.stdout:3583: writing smithi04569269-13 from 1351680 to 1597440 tid 3 2021-07-31T01:57:32.679 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 14 current snap is 350 2021-07-31T01:57:32.679 INFO:tasks.rados.rados.0.smithi045.stdout:3584: seq_num 1099 ranges {0=409600,409600=442368,851968=720896,1572864=40960} 2021-07-31T01:57:32.681 INFO:tasks.rados.rados.0.smithi045.stdout:3584: writing smithi04569269-14 from 0 to 409600 tid 1 2021-07-31T01:57:32.683 INFO:tasks.rados.rados.0.smithi045.stdout:3584: writing smithi04569269-14 from 409600 to 851968 tid 2 2021-07-31T01:57:32.687 INFO:tasks.rados.rados.0.smithi045.stdout:3584: writing smithi04569269-14 from 851968 to 1572864 tid 3 2021-07-31T01:57:32.688 INFO:tasks.rados.rados.0.smithi045.stdout:3584: writing smithi04569269-14 from 1572864 to 1613824 tid 4 2021-07-31T01:57:32.688 INFO:tasks.rados.rados.0.smithi045.stdout:3585: snap_remove snap 333 2021-07-31T01:57:32.702 INFO:tasks.rados.rados.0.smithi045.stdout:3585: done (5 left) 2021-07-31T01:57:32.703 INFO:tasks.rados.rados.0.smithi045.stdout:3586: rollback oid 41 current snap is 350 2021-07-31T01:57:32.703 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 41 to 326 2021-07-31T01:57:32.703 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 25 v 1849 (ObjNum 1031 snap 335 seq_num 1031) dirty exists 2021-07-31T01:57:32.703 INFO:tasks.rados.rados.0.smithi045.stdout:3582: finishing rollback tid 0 to smithi04569269-24 2021-07-31T01:57:32.704 INFO:tasks.rados.rados.0.smithi045.stdout:3581: expect (ObjNum 1013 snap 323 seq_num 1013) 2021-07-31T01:57:32.784 INFO:tasks.rados.rados.0.smithi045.stdout:3582: finishing rollback tid 1 to smithi04569269-24 2021-07-31T01:57:32.784 INFO:tasks.rados.rados.0.smithi045.stdout:3582: finishing rollback tid 2 to smithi04569269-24 2021-07-31T01:57:32.784 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 24 v 1670 (ObjNum -1105631600 snap 21873 seq_num 2818574544) dirty exists 2021-07-31T01:57:32.785 INFO:tasks.rados.rados.0.smithi045.stdout:3580: done (5 left) 2021-07-31T01:57:32.785 INFO:tasks.rados.rados.0.smithi045.stdout:3581: done (4 left) 2021-07-31T01:57:32.785 INFO:tasks.rados.rados.0.smithi045.stdout:3582: done (3 left) 2021-07-31T01:57:32.785 INFO:tasks.rados.rados.0.smithi045.stdout:3587: rollback oid 23 current snap is 350 2021-07-31T01:57:32.786 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 23 to 348 2021-07-31T01:57:32.786 INFO:tasks.rados.rados.0.smithi045.stdout:3583: finishing write tid 1 to smithi04569269-13 2021-07-31T01:57:32.786 INFO:tasks.rados.rados.0.smithi045.stdout:3583: finishing write tid 2 to smithi04569269-13 2021-07-31T01:57:32.786 INFO:tasks.rados.rados.0.smithi045.stdout:3583: finishing write tid 3 to smithi04569269-13 2021-07-31T01:57:32.787 INFO:tasks.rados.rados.0.smithi045.stdout:3583: finishing write tid 4 to smithi04569269-13 2021-07-31T01:57:32.787 INFO:tasks.rados.rados.0.smithi045.stdout:3583: finishing write tid 5 to smithi04569269-13 2021-07-31T01:57:32.787 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 13 v 1887 (ObjNum 1098 snap 350 seq_num 1098) dirty exists 2021-07-31T01:57:32.787 INFO:tasks.rados.rados.0.smithi045.stdout:3583: left oid 13 (ObjNum 1098 snap 350 seq_num 1098) 2021-07-31T01:57:32.788 INFO:tasks.rados.rados.0.smithi045.stdout:3584: finishing write tid 1 to smithi04569269-14 2021-07-31T01:57:32.788 INFO:tasks.rados.rados.0.smithi045.stdout:3584: finishing write tid 2 to smithi04569269-14 2021-07-31T01:57:32.788 INFO:tasks.rados.rados.0.smithi045.stdout:3584: finishing write tid 3 to smithi04569269-14 2021-07-31T01:57:32.788 INFO:tasks.rados.rados.0.smithi045.stdout:3584: finishing write tid 4 to smithi04569269-14 2021-07-31T01:57:32.788 INFO:tasks.rados.rados.0.smithi045.stdout:3584: finishing write tid 5 to smithi04569269-14 2021-07-31T01:57:32.789 INFO:tasks.rados.rados.0.smithi045.stdout:3584: finishing write tid 6 to smithi04569269-14 2021-07-31T01:57:32.789 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 14 v 2060 (ObjNum 1099 snap 350 seq_num 1099) dirty exists 2021-07-31T01:57:32.789 INFO:tasks.rados.rados.0.smithi045.stdout:3584: left oid 14 (ObjNum 1099 snap 350 seq_num 1099) 2021-07-31T01:57:32.789 INFO:tasks.rados.rados.0.smithi045.stdout:3586: finishing rollback tid 1 to smithi04569269-41 2021-07-31T01:57:32.790 INFO:tasks.rados.rados.0.smithi045.stdout:3586: finishing rollback tid 2 to smithi04569269-41 2021-07-31T01:57:32.790 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 41 v 1694 (ObjNum 948 snap 286 seq_num 948) dirty exists 2021-07-31T01:57:32.790 INFO:tasks.rados.rados.0.smithi045.stdout:3583: done (3 left) 2021-07-31T01:57:32.790 INFO:tasks.rados.rados.0.smithi045.stdout:3584: done (2 left) 2021-07-31T01:57:32.791 INFO:tasks.rados.rados.0.smithi045.stdout:3586: done (1 left) 2021-07-31T01:57:32.792 INFO:tasks.rados.rados.0.smithi045.stdout:3588: delete oid 11 current snap is 350 2021-07-31T01:57:32.793 INFO:tasks.rados.rados.0.smithi045.stdout:3587: finishing rollback tid 0 to smithi04569269-23 2021-07-31T01:57:32.794 INFO:tasks.rados.rados.0.smithi045.stdout:3587: finishing rollback tid 1 to smithi04569269-23 2021-07-31T01:57:32.794 INFO:tasks.rados.rados.0.smithi045.stdout:3587: finishing rollback tid 2 to smithi04569269-23 2021-07-31T01:57:32.794 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 23 v 2045 (ObjNum 1049 snap 342 seq_num 1049) dirty exists 2021-07-31T01:57:32.795 INFO:tasks.rados.rados.0.smithi045.stdout:3587: done (1 left) 2021-07-31T01:57:32.795 INFO:tasks.rados.rados.0.smithi045.stdout:3588: done (0 left) 2021-07-31T01:57:32.795 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 20 current snap is 350 2021-07-31T01:57:32.795 INFO:tasks.rados.rados.0.smithi045.stdout:3589: seq_num 1101 ranges {3637248=442368,4079616=606208,4685824=598016} 2021-07-31T01:57:32.812 INFO:tasks.rados.rados.0.smithi045.stdout:3589: writing smithi04569269-20 from 3637248 to 4079616 tid 1 2021-07-31T01:57:32.816 INFO:tasks.rados.rados.0.smithi045.stdout:3589: writing smithi04569269-20 from 4079616 to 4685824 tid 2 2021-07-31T01:57:32.819 INFO:tasks.rados.rados.0.smithi045.stdout:3589: writing smithi04569269-20 from 4685824 to 5283840 tid 3 2021-07-31T01:57:32.819 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 28 current snap is 350 2021-07-31T01:57:32.819 INFO:tasks.rados.rados.0.smithi045.stdout:3589: finishing write tid 1 to smithi04569269-20 2021-07-31T01:57:32.820 INFO:tasks.rados.rados.0.smithi045.stdout:3590: seq_num 1102 ranges {4308992=458752,4767744=491520,5259264=491520,5750784=221184} 2021-07-31T01:57:32.845 INFO:tasks.rados.rados.0.smithi045.stdout:3590: writing smithi04569269-28 from 4308992 to 4767744 tid 1 2021-07-31T01:57:32.847 INFO:tasks.rados.rados.0.smithi045.stdout:3590: writing smithi04569269-28 from 4767744 to 5259264 tid 2 2021-07-31T01:57:32.850 INFO:tasks.rados.rados.0.smithi045.stdout:3590: writing smithi04569269-28 from 5259264 to 5750784 tid 3 2021-07-31T01:57:32.852 INFO:tasks.rados.rados.0.smithi045.stdout:3590: writing smithi04569269-28 from 5750784 to 5971968 tid 4 2021-07-31T01:57:32.852 INFO:tasks.rados.rados.0.smithi045.stdout:3591: snap_remove snap 332 2021-07-31T01:57:33.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:32 smithi154 conmon[34553]: cluster 2021-07-31T01:57:31.939463+0000 mgr.x (mgr.24542) 2021-07-31T01:57:33.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:32 smithi154 conmon[34553]: 444 : cluster [DBG] pgmap v891: 25 pgs: 25 active+clean; 623 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.5 KiB/s rd, 3.1 MiB/s wr, 14 op/s 2021-07-31T01:57:33.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:32 smithi045 conmon[39718]: cluster 2021-07-31T01:57:31.939463+0000 2021-07-31T01:57:33.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:32 smithi045 conmon[39718]: mgr.x (mgr.24542) 444 : cluster [DBG] pgmap v891: 25 pgs: 25 active+clean; 623 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.5 KiB/s rd, 3.1 MiB/s wr, 14 op/s 2021-07-31T01:57:33.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:32 smithi045 conmon[33655]: cluster 2021-07-31T01:57:31.939463+0000 2021-07-31T01:57:33.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:32 smithi045 conmon[33655]: mgr.x (mgr.24542) 444 : cluster [DBG] pgmap v891: 25 pgs: 25 active+clean; 623 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.5 KiB/s rd, 3.1 MiB/s wr, 14 op/s 2021-07-31T01:57:33.709 INFO:tasks.rados.rados.0.smithi045.stdout:3591: done (2 left) 2021-07-31T01:57:33.710 INFO:tasks.rados.rados.0.smithi045.stdout:3592: delete oid 7 current snap is 350 2021-07-31T01:57:33.710 INFO:tasks.rados.rados.0.smithi045.stdout:3589: finishing write tid 2 to smithi04569269-20 2021-07-31T01:57:33.710 INFO:tasks.rados.rados.0.smithi045.stdout:3589: finishing write tid 3 to smithi04569269-20 2021-07-31T01:57:33.711 INFO:tasks.rados.rados.0.smithi045.stdout:3589: finishing write tid 4 to smithi04569269-20 2021-07-31T01:57:33.711 INFO:tasks.rados.rados.0.smithi045.stdout:3589: finishing write tid 5 to smithi04569269-20 2021-07-31T01:57:33.711 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 20 v 2504 (ObjNum 1101 snap 350 seq_num 1101) dirty exists 2021-07-31T01:57:33.711 INFO:tasks.rados.rados.0.smithi045.stdout:3589: left oid 20 (ObjNum 1101 snap 350 seq_num 1101) 2021-07-31T01:57:33.711 INFO:tasks.rados.rados.0.smithi045.stdout:3590: finishing write tid 1 to smithi04569269-28 2021-07-31T01:57:33.712 INFO:tasks.rados.rados.0.smithi045.stdout:3590: finishing write tid 2 to smithi04569269-28 2021-07-31T01:57:33.712 INFO:tasks.rados.rados.0.smithi045.stdout:3590: finishing write tid 3 to smithi04569269-28 2021-07-31T01:57:33.712 INFO:tasks.rados.rados.0.smithi045.stdout:3590: finishing write tid 4 to smithi04569269-28 2021-07-31T01:57:33.712 INFO:tasks.rados.rados.0.smithi045.stdout:3590: finishing write tid 5 to smithi04569269-28 2021-07-31T01:57:33.712 INFO:tasks.rados.rados.0.smithi045.stdout:3590: finishing write tid 6 to smithi04569269-28 2021-07-31T01:57:33.713 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 28 v 2344 (ObjNum 1102 snap 350 seq_num 1102) dirty exists 2021-07-31T01:57:33.713 INFO:tasks.rados.rados.0.smithi045.stdout:3590: left oid 28 (ObjNum 1102 snap 350 seq_num 1102) 2021-07-31T01:57:33.722 INFO:tasks.rados.rados.0.smithi045.stdout:3589: done (2 left) 2021-07-31T01:57:33.723 INFO:tasks.rados.rados.0.smithi045.stdout:3590: done (1 left) 2021-07-31T01:57:33.723 INFO:tasks.rados.rados.0.smithi045.stdout:3592: done (0 left) 2021-07-31T01:57:33.723 INFO:tasks.rados.rados.0.smithi045.stdout:3593: read oid 22 snap -1 2021-07-31T01:57:33.724 INFO:tasks.rados.rados.0.smithi045.stdout:3593: expect (ObjNum 1092 snap 349 seq_num 1092) 2021-07-31T01:57:33.724 INFO:tasks.rados.rados.0.smithi045.stdout:3594: rollback oid 32 current snap is 350 2021-07-31T01:57:33.724 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 32 to 304 2021-07-31T01:57:33.724 INFO:tasks.rados.rados.0.smithi045.stdout:3595: rmattr oid 10 current snap is 350 2021-07-31T01:57:33.724 INFO:tasks.rados.rados.0.smithi045.stdout:3596: rollback oid 25 current snap is 350 2021-07-31T01:57:33.725 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 25 to 301 2021-07-31T01:57:33.725 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 36 current snap is 350 2021-07-31T01:57:33.725 INFO:tasks.rados.rados.0.smithi045.stdout:3597: seq_num 1104 ranges {2293760=499712,2793472=712704,3506176=491520} 2021-07-31T01:57:33.740 INFO:tasks.rados.rados.0.smithi045.stdout:3597: writing smithi04569269-36 from 2293760 to 2793472 tid 1 2021-07-31T01:57:33.744 INFO:tasks.rados.rados.0.smithi045.stdout:3597: writing smithi04569269-36 from 2793472 to 3506176 tid 2 2021-07-31T01:57:33.746 INFO:tasks.rados.rados.0.smithi045.stdout:3597: writing smithi04569269-36 from 3506176 to 3997696 tid 3 2021-07-31T01:57:33.747 INFO:tasks.rados.rados.0.smithi045.stdout:3598: read oid 27 snap -1 2021-07-31T01:57:33.748 INFO:tasks.rados.rados.0.smithi045.stdout:3598: expect (ObjNum 0 snap 0 seq_num 0) 2021-07-31T01:57:33.748 INFO:tasks.rados.rados.0.smithi045.stdout:3599: rmattr oid 42 current snap is 350 2021-07-31T01:57:33.748 INFO:tasks.rados.rados.0.smithi045.stdout:3599: done (6 left) 2021-07-31T01:57:33.748 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 1 current snap is 350 2021-07-31T01:57:33.748 INFO:tasks.rados.rados.0.smithi045.stdout:3600: seq_num 1105 ranges {4497408=516096,5013504=598016,5611520=606208} 2021-07-31T01:57:33.774 INFO:tasks.rados.rados.0.smithi045.stdout:3600: writing smithi04569269-1 from 4497408 to 5013504 tid 1 2021-07-31T01:57:33.777 INFO:tasks.rados.rados.0.smithi045.stdout:3600: writing smithi04569269-1 from 5013504 to 5611520 tid 2 2021-07-31T01:57:33.781 INFO:tasks.rados.rados.0.smithi045.stdout:3600: writing smithi04569269-1 from 5611520 to 6217728 tid 3 2021-07-31T01:57:33.782 INFO:tasks.rados.rados.0.smithi045.stdout:3601: read oid 5 snap -1 2021-07-31T01:57:33.782 INFO:tasks.rados.rados.0.smithi045.stdout:3601: expect (ObjNum 996 snap 311 seq_num 996) 2021-07-31T01:57:33.782 INFO:tasks.rados.rados.0.smithi045.stdout:3602: snap_remove snap 349 2021-07-31T01:57:34.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:33 smithi154 conmon[34553]: cluster 2021-07-31T01:57:32.701575+0000 mon.a (mon.0) 1405 : cluster [DBG] osdmap e736: 8 total, 8 up, 8 in 2021-07-31T01:57:34.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:33 smithi154 conmon[34553]: 2021-07-31T01:57:34.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:33 smithi045 conmon[33655]: cluster 2021-07-31T01:57:32.701575+0000 mon.a ( 2021-07-31T01:57:34.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:33 smithi045 conmon[33655]: mon.0) 1405 : cluster [DBG] osdmap e736: 8 total, 8 up, 8 in 2021-07-31T01:57:34.027 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:57:33 smithi045 conmon[61210]: level=error ts=2021-07-31T01:57:33.526Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:57:34.027 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:57:33 smithi045 conmon[61210]: level=error ts=2021-07-31T01:57:33.526Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:57:34.027 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:57:33 smithi045 conmon[61210]: level=error ts=2021-07-31T01:57:33.526Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T01:57:34.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:33 smithi045 conmon[39718]: cluster 2021-07-31T01:57:32.701575+0000 mon.a (mon 2021-07-31T01:57:34.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:33 smithi045 conmon[39718]: .0) 1405 : cluster [DBG] osdmap e736: 8 total, 8 up, 8 in 2021-07-31T01:57:34.715 INFO:tasks.rados.rados.0.smithi045.stdout:3602: done (8 left) 2021-07-31T01:57:34.715 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 17 current snap is 350 2021-07-31T01:57:34.715 INFO:tasks.rados.rados.0.smithi045.stdout:3603: seq_num 1106 ranges {1835008=532480,2367488=483328,2850816=475136,3325952=245760} 2021-07-31T01:57:34.728 INFO:tasks.rados.rados.0.smithi045.stdout:3603: writing smithi04569269-17 from 1835008 to 2367488 tid 1 2021-07-31T01:57:34.731 INFO:tasks.rados.rados.0.smithi045.stdout:3603: writing smithi04569269-17 from 2367488 to 2850816 tid 2 2021-07-31T01:57:34.734 INFO:tasks.rados.rados.0.smithi045.stdout:3603: writing smithi04569269-17 from 2850816 to 3325952 tid 3 2021-07-31T01:57:34.736 INFO:tasks.rados.rados.0.smithi045.stdout:3603: writing smithi04569269-17 from 3325952 to 3571712 tid 4 2021-07-31T01:57:34.736 INFO:tasks.rados.rados.0.smithi045.stdout:3604: copy_from oid 39 from oid 4 current snap is 350 2021-07-31T01:57:34.736 INFO:tasks.rados.rados.0.smithi045.stdout:3596: finishing rollback tid 0 to smithi04569269-25 2021-07-31T01:57:34.736 INFO:tasks.rados.rados.0.smithi045.stdout:3596: finishing rollback tid 1 to smithi04569269-25 2021-07-31T01:57:34.737 INFO:tasks.rados.rados.0.smithi045.stdout:3596: finishing rollback tid 2 to smithi04569269-25 2021-07-31T01:57:34.737 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 25 v 1852 (ObjNum 837 snap 263 seq_num 837) dirty exists 2021-07-31T01:57:34.737 INFO:tasks.rados.rados.0.smithi045.stdout:3593: expect (ObjNum 1092 snap 349 seq_num 1092) 2021-07-31T01:57:34.776 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 10 v 2345 (ObjNum 1071 snap 344 seq_num 1071) dirty exists 2021-07-31T01:57:34.776 INFO:tasks.rados.rados.0.smithi045.stdout:3594: finishing rollback tid 0 to smithi04569269-32 2021-07-31T01:57:34.777 INFO:tasks.rados.rados.0.smithi045.stdout:3594: finishing rollback tid 1 to smithi04569269-32 2021-07-31T01:57:34.777 INFO:tasks.rados.rados.0.smithi045.stdout:3594: finishing rollback tid 2 to smithi04569269-32 2021-07-31T01:57:34.777 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 32 v 1881 (ObjNum 978 snap 301 seq_num 978) dirty exists 2021-07-31T01:57:34.778 INFO:tasks.rados.rados.0.smithi045.stdout:3597: finishing write tid 1 to smithi04569269-36 2021-07-31T01:57:34.779 INFO:tasks.rados.rados.0.smithi045.stdout:3597: finishing write tid 2 to smithi04569269-36 2021-07-31T01:57:34.779 INFO:tasks.rados.rados.0.smithi045.stdout:3598: expect (ObjNum 0 snap 0 seq_num 0) 2021-07-31T01:57:34.780 INFO:tasks.rados.rados.0.smithi045.stdout:3597: finishing write tid 3 to smithi04569269-36 2021-07-31T01:57:34.780 INFO:tasks.rados.rados.0.smithi045.stdout:3597: finishing write tid 4 to smithi04569269-36 2021-07-31T01:57:34.780 INFO:tasks.rados.rados.0.smithi045.stdout:3597: finishing write tid 5 to smithi04569269-36 2021-07-31T01:57:34.781 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 36 v 2517 (ObjNum 1104 snap 350 seq_num 1104) dirty exists 2021-07-31T01:57:34.781 INFO:tasks.rados.rados.0.smithi045.stdout:3597: left oid 36 (ObjNum 1104 snap 350 seq_num 1104) 2021-07-31T01:57:34.781 INFO:tasks.rados.rados.0.smithi045.stdout:3600: finishing write tid 1 to smithi04569269-1 2021-07-31T01:57:34.781 INFO:tasks.rados.rados.0.smithi045.stdout:3600: finishing write tid 2 to smithi04569269-1 2021-07-31T01:57:34.782 INFO:tasks.rados.rados.0.smithi045.stdout:3600: finishing write tid 3 to smithi04569269-1 2021-07-31T01:57:34.782 INFO:tasks.rados.rados.0.smithi045.stdout:3600: finishing write tid 4 to smithi04569269-1 2021-07-31T01:57:34.782 INFO:tasks.rados.rados.0.smithi045.stdout:3600: finishing write tid 5 to smithi04569269-1 2021-07-31T01:57:34.782 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 1 v 1877 (ObjNum 1105 snap 350 seq_num 1105) dirty exists 2021-07-31T01:57:34.783 INFO:tasks.rados.rados.0.smithi045.stdout:3600: left oid 1 (ObjNum 1105 snap 350 seq_num 1105) 2021-07-31T01:57:34.783 INFO:tasks.rados.rados.0.smithi045.stdout:3601: expect (ObjNum 996 snap 311 seq_num 996) 2021-07-31T01:57:34.838 INFO:tasks.rados.rados.0.smithi045.stdout:3603: finishing write tid 1 to smithi04569269-17 2021-07-31T01:57:34.838 INFO:tasks.rados.rados.0.smithi045.stdout:3603: finishing write tid 2 to smithi04569269-17 2021-07-31T01:57:34.839 INFO:tasks.rados.rados.0.smithi045.stdout:3603: finishing write tid 3 to smithi04569269-17 2021-07-31T01:57:34.839 INFO:tasks.rados.rados.0.smithi045.stdout:3603: finishing write tid 4 to smithi04569269-17 2021-07-31T01:57:34.839 INFO:tasks.rados.rados.0.smithi045.stdout:3603: finishing write tid 5 to smithi04569269-17 2021-07-31T01:57:34.840 INFO:tasks.rados.rados.0.smithi045.stdout:3603: finishing write tid 6 to smithi04569269-17 2021-07-31T01:57:34.840 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 17 v 2350 (ObjNum 1106 snap 350 seq_num 1106) dirty exists 2021-07-31T01:57:34.840 INFO:tasks.rados.rados.0.smithi045.stdout:3603: left oid 17 (ObjNum 1106 snap 350 seq_num 1106) 2021-07-31T01:57:34.840 INFO:tasks.rados.rados.0.smithi045.stdout:3604: finishing copy_from to smithi04569269-39 2021-07-31T01:57:34.841 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 39 v 1886 (ObjNum 1085 snap 347 seq_num 1085) dirty exists 2021-07-31T01:57:34.841 INFO:tasks.rados.rados.0.smithi045.stdout:3604: finishing copy_from racing read to smithi04569269-39 2021-07-31T01:57:34.841 INFO:tasks.rados.rados.0.smithi045.stdout:3593: done (9 left) 2021-07-31T01:57:34.841 INFO:tasks.rados.rados.0.smithi045.stdout:3594: done (8 left) 2021-07-31T01:57:34.841 INFO:tasks.rados.rados.0.smithi045.stdout:3595: done (7 left) 2021-07-31T01:57:34.842 INFO:tasks.rados.rados.0.smithi045.stdout:3596: done (6 left) 2021-07-31T01:57:34.842 INFO:tasks.rados.rados.0.smithi045.stdout:3597: done (5 left) 2021-07-31T01:57:34.842 INFO:tasks.rados.rados.0.smithi045.stdout:3598: done (4 left) 2021-07-31T01:57:34.842 INFO:tasks.rados.rados.0.smithi045.stdout:3600: done (3 left) 2021-07-31T01:57:34.843 INFO:tasks.rados.rados.0.smithi045.stdout:3601: done (2 left) 2021-07-31T01:57:34.843 INFO:tasks.rados.rados.0.smithi045.stdout:3603: done (1 left) 2021-07-31T01:57:34.843 INFO:tasks.rados.rados.0.smithi045.stdout:3604: done (0 left) 2021-07-31T01:57:34.843 INFO:tasks.rados.rados.0.smithi045.stdout:3605: rmattr oid 28 current snap is 350 2021-07-31T01:57:34.844 INFO:tasks.rados.rados.0.smithi045.stdout:3605: done (0 left) 2021-07-31T01:57:34.844 INFO:tasks.rados.rados.0.smithi045.stdout:3606: snap_remove snap 335 2021-07-31T01:57:35.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:34 smithi154 conmon[34553]: cluster 2021-07-31T01:57:33.707557+0000 mon.a (mon.0) 1406 : cluster [DBG] osdmap e737: 8 total, 8 up, 8 in 2021-07-31T01:57:35.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:34 smithi154 conmon[34553]: cluster 2021-07-31T01:57:33.940047+0000 mgr.x (mgr.24542) 445 : cluster [DBG] pgmap v894: 25 pgs: 25 active+clean; 649 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 15 MiB/s rd, 3.5 MiB/s wr, 26 op/s 2021-07-31T01:57:35.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:34 smithi045 conmon[33655]: cluster 2021-07-31T01:57:33.707557+0000 mon.a (mon.0 2021-07-31T01:57:35.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:34 smithi045 conmon[33655]: ) 1406 : cluster [DBG] osdmap e737: 8 total, 8 up, 8 in 2021-07-31T01:57:35.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:34 smithi045 conmon[33655]: cluster 2021-07-31T01:57:33.940047+0000 mgr.x (mgr.24542) 445 : cluster [DBG] pgmap v894: 25 pgs: 25 active+clean; 649 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 15 MiB/s rd, 3.5 MiB/s wr, 26 op/s 2021-07-31T01:57:35.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:34 smithi045 conmon[39718]: cluster 2021-07-31T01:57:33.707557+0000 mon.a ( 2021-07-31T01:57:35.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:34 smithi045 conmon[39718]: mon.0) 1406 : cluster [DBG] osdmap e737: 8 total, 8 up, 8 in 2021-07-31T01:57:35.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:34 smithi045 conmon[39718]: cluster 2021-07-31T01:57: 2021-07-31T01:57:35.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:34 smithi045 conmon[39718]: 33.940047+0000 mgr.x (mgr.24542) 445 : cluster [DBG] pgmap v894: 25 pgs: 25 active+clean; 649 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 15 MiB/s rd, 3.5 MiB/s wr, 26 op/s 2021-07-31T01:57:35.716 INFO:tasks.rados.rados.0.smithi045.stdout:3606: done (0 left) 2021-07-31T01:57:35.717 INFO:tasks.rados.rados.0.smithi045.stdout:3607: rmattr oid 28 current snap is 350 2021-07-31T01:57:35.717 INFO:tasks.rados.rados.0.smithi045.stdout:3607: done (0 left) 2021-07-31T01:57:35.717 INFO:tasks.rados.rados.0.smithi045.stdout:3608: rollback oid 9 current snap is 350 2021-07-31T01:57:35.717 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 9 to 300 2021-07-31T01:57:35.718 INFO:tasks.rados.rados.0.smithi045.stdout:3609: read oid 41 snap -1 2021-07-31T01:57:35.719 INFO:tasks.rados.rados.0.smithi045.stdout:3609: expect (ObjNum 948 snap 286 seq_num 948) 2021-07-31T01:57:35.719 INFO:tasks.rados.rados.0.smithi045.stdout:3610: snap_create 2021-07-31T01:57:35.730 INFO:tasks.rados.rados.0.smithi045.stdout:3608: finishing rollback tid 0 to smithi04569269-9 2021-07-31T01:57:35.730 INFO:tasks.rados.rados.0.smithi045.stdout:3608: finishing rollback tid 1 to smithi04569269-9 2021-07-31T01:57:35.730 INFO:tasks.rados.rados.0.smithi045.stdout:3608: finishing rollback tid 2 to smithi04569269-9 2021-07-31T01:57:35.730 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 9 v 2362 (ObjNum 881 snap 272 seq_num 881) dirty exists 2021-07-31T01:57:35.733 INFO:tasks.rados.rados.0.smithi045.stdout:3609: expect (ObjNum 948 snap 286 seq_num 948) 2021-07-31T01:57:36.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:35 smithi154 conmon[34553]: cluster 2021-07-31T01:57:36.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:35 smithi154 conmon[34553]: 2021-07-31T01:57:36.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:35 smithi154 conmon[34553]: 2021 2021-07-31T01:57:36.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:35 smithi154 conmon[34553]: - 2021-07-31T01:57:36.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:35 smithi154 conmon[34553]: 07 2021-07-31T01:57:36.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:35 smithi154 conmon[34553]: - 2021-07-31T01:57:36.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:35 smithi154 conmon[34553]: 31 2021-07-31T01:57:36.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:35 smithi154 conmon[34553]: T 2021-07-31T01:57:36.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:35 smithi154 conmon[34553]: 01 2021-07-31T01:57:36.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:35 smithi154 conmon[34553]: : 2021-07-31T01:57:36.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:35 smithi154 conmon[34553]: 57 2021-07-31T01:57:36.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:35 smithi154 conmon[34553]: : 2021-07-31T01:57:36.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:35 smithi154 conmon[34553]: 34 2021-07-31T01:57:36.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:35 smithi154 conmon[34553]: . 2021-07-31T01:57:36.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:35 smithi154 conmon[34553]: 713782 2021-07-31T01:57:36.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:35 smithi154 conmon[34553]: +0000 2021-07-31T01:57:36.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:35 smithi154 conmon[34553]: 2021-07-31T01:57:36.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:35 smithi154 conmon[34553]: mon.a 2021-07-31T01:57:36.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:35 smithi154 conmon[34553]: ( 2021-07-31T01:57:36.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:35 smithi154 conmon[34553]: mon 2021-07-31T01:57:36.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:35 smithi154 conmon[34553]: . 2021-07-31T01:57:36.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:35 smithi154 conmon[34553]: 0 2021-07-31T01:57:36.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:35 smithi154 conmon[34553]: ) 2021-07-31T01:57:36.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:35 smithi154 conmon[34553]: 1407 : cluster [DBG] osdmap e738: 8 total, 8 up, 8 in 2021-07-31T01:57:36.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:35 smithi154 conmon[34553]: 2021-07-31T01:57:36.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:35 smithi045 conmon[33655]: cluster 2021-07-31T01:57:36.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:35 smithi045 conmon[33655]: 2021-07-31T01: 2021-07-31T01:57:36.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:35 smithi045 conmon[33655]: 57:34.713782+0000 mon.a (mon.0) 1407 : cluster [DBG] osdmap e738: 8 total, 8 up, 8 in 2021-07-31T01:57:36.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:35 smithi045 conmon[39718]: cluster 2021-07-31T01:57:36.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:35 smithi045 conmon[39718]: 2021-07-31T01:57:36.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:35 smithi045 conmon[39718]: 2021 2021-07-31T01:57:36.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:35 smithi045 conmon[39718]: - 2021-07-31T01:57:36.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:35 smithi045 conmon[39718]: 07 2021-07-31T01:57:36.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:35 smithi045 conmon[39718]: - 2021-07-31T01:57:36.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:35 smithi045 conmon[39718]: 31 2021-07-31T01:57:36.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:35 smithi045 conmon[39718]: T 2021-07-31T01:57:36.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:35 smithi045 conmon[39718]: 01 2021-07-31T01:57:36.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:35 smithi045 conmon[39718]: : 2021-07-31T01:57:36.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:35 smithi045 conmon[39718]: 57 2021-07-31T01:57:36.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:35 smithi045 conmon[39718]: : 2021-07-31T01:57:36.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:35 smithi045 conmon[39718]: 34 2021-07-31T01:57:36.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:35 smithi045 conmon[39718]: . 2021-07-31T01:57:36.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:35 smithi045 conmon[39718]: 713782 2021-07-31T01:57:36.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:35 smithi045 conmon[39718]: +0000 2021-07-31T01:57:36.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:35 smithi045 conmon[39718]: 2021-07-31T01:57:36.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:35 smithi045 conmon[39718]: mon.a 2021-07-31T01:57:36.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:35 smithi045 conmon[39718]: ( 2021-07-31T01:57:36.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:35 smithi045 conmon[39718]: mon 2021-07-31T01:57:36.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:35 smithi045 conmon[39718]: . 2021-07-31T01:57:36.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:35 smithi045 conmon[39718]: 0 2021-07-31T01:57:36.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:35 smithi045 conmon[39718]: ) 2021-07-31T01:57:36.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:35 smithi045 conmon[39718]: 1407 2021-07-31T01:57:36.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:35 smithi045 conmon[39718]: : 2021-07-31T01:57:36.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:35 smithi045 conmon[39718]: cluster 2021-07-31T01:57:36.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:35 smithi045 conmon[39718]: 2021-07-31T01:57:36.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:35 smithi045 conmon[39718]: [DBG] 2021-07-31T01:57:36.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:35 smithi045 conmon[39718]: 2021-07-31T01:57:36.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:35 smithi045 conmon[39718]: osdmap e738: 8 total, 8 up, 8 in 2021-07-31T01:57:36.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:35 smithi045 conmon[39718]: 2021-07-31T01:57:36.669 INFO:tasks.rados.rados.0.smithi045.stdout:3608: done (2 left) 2021-07-31T01:57:36.670 INFO:tasks.rados.rados.0.smithi045.stdout:3609: done (1 left) 2021-07-31T01:57:36.670 INFO:tasks.rados.rados.0.smithi045.stdout:3610: done (0 left) 2021-07-31T01:57:36.670 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 34 current snap is 351 2021-07-31T01:57:36.670 INFO:tasks.rados.rados.0.smithi045.stdout:3611: seq_num 1107 ranges {1007616=548864,1556480=770048,2326528=434176} 2021-07-31T01:57:36.678 INFO:tasks.rados.rados.0.smithi045.stdout:3611: writing smithi04569269-34 from 1007616 to 1556480 tid 1 2021-07-31T01:57:36.682 INFO:tasks.rados.rados.0.smithi045.stdout:3611: writing smithi04569269-34 from 1556480 to 2326528 tid 2 2021-07-31T01:57:36.685 INFO:tasks.rados.rados.0.smithi045.stdout:3611: writing smithi04569269-34 from 2326528 to 2760704 tid 3 2021-07-31T01:57:36.685 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 21 current snap is 351 2021-07-31T01:57:36.685 INFO:tasks.rados.rados.0.smithi045.stdout:3612: seq_num 1108 ranges {1400832=565248,1966080=647168,2613248=483328,3096576=73728} 2021-07-31T01:57:36.696 INFO:tasks.rados.rados.0.smithi045.stdout:3612: writing smithi04569269-21 from 1400832 to 1966080 tid 1 2021-07-31T01:57:36.699 INFO:tasks.rados.rados.0.smithi045.stdout:3612: writing smithi04569269-21 from 1966080 to 2613248 tid 2 2021-07-31T01:57:36.703 INFO:tasks.rados.rados.0.smithi045.stdout:3612: writing smithi04569269-21 from 2613248 to 3096576 tid 3 2021-07-31T01:57:36.704 INFO:tasks.rados.rados.0.smithi045.stdout:3612: writing smithi04569269-21 from 3096576 to 3170304 tid 4 2021-07-31T01:57:36.704 INFO:tasks.rados.rados.0.smithi045.stdout:3613: read oid 36 snap 255 2021-07-31T01:57:36.704 INFO:tasks.rados.rados.0.smithi045.stdout:3613: expect (ObjNum -1105914768 snap 21873 seq_num 2820255792) 2021-07-31T01:57:36.704 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 14 current snap is 351 2021-07-31T01:57:36.705 INFO:tasks.rados.rados.0.smithi045.stdout:3611: finishing write tid 1 to smithi04569269-34 2021-07-31T01:57:36.705 INFO:tasks.rados.rados.0.smithi045.stdout:3614: seq_num 1109 ranges {1613824=581632,2195456=532480,2727936=647168,3375104=24576} 2021-07-31T01:57:36.715 INFO:tasks.rados.rados.0.smithi045.stdout:3614: writing smithi04569269-14 from 1613824 to 2195456 tid 1 2021-07-31T01:57:36.718 INFO:tasks.rados.rados.0.smithi045.stdout:3614: writing smithi04569269-14 from 2195456 to 2727936 tid 2 2021-07-31T01:57:36.723 INFO:tasks.rados.rados.0.smithi045.stdout:3614: writing smithi04569269-14 from 2727936 to 3375104 tid 3 2021-07-31T01:57:36.723 INFO:tasks.rados.rados.0.smithi045.stdout:3614: writing smithi04569269-14 from 3375104 to 3399680 tid 4 2021-07-31T01:57:36.723 INFO:tasks.rados.rados.0.smithi045.stdout:3615: snap_remove snap 301 2021-07-31T01:57:37.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:36 smithi154 conmon[34553]: cluster 2021-07-31T01:57:35.715253 2021-07-31T01:57:37.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:36 smithi154 conmon[34553]: +0000 mon.a (mon.0) 1408 : cluster [DBG] osdmap e739: 8 total, 8 up, 8 in 2021-07-31T01:57:37.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:36 smithi154 conmon[34553]: cluster 2021-07-31T01:57:35. 2021-07-31T01:57:37.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:36 smithi154 conmon[34553]: 940485+0000 mgr.x (mgr.24542) 446 : cluster [DBG] pgmap v897: 25 pgs: 25 active+clean; 649 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 15 MiB/s rd, 3.5 MiB/s wr, 26 op/s 2021-07-31T01:57:37.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:36 smithi154 conmon[34553]: 2021-07-31T01:57:37.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:36 smithi154 conmon[34553]: cluster 2021 2021-07-31T01:57:37.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:36 smithi154 conmon[34553]: -07-31T01:57:36 2021-07-31T01:57:37.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:36 smithi154 conmon[34553]: .668667+0000 mon.a (mon. 2021-07-31T01:57:37.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:36 smithi154 conmon[34553]: 0) 1409 : cluster [DBG] 2021-07-31T01:57:37.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:36 smithi154 conmon[34553]: osdmap e740: 8 total, 8 up, 8 in 2021-07-31T01:57:37.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:36 smithi045 conmon[33655]: cluster 2021-07-31T01:57:35.715253+0000 mon.a (mon. 2021-07-31T01:57:37.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:36 smithi045 conmon[33655]: 0) 1408 : cluster [DBG] osdmap e739: 8 total, 8 up, 8 in 2021-07-31T01:57:37.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:36 smithi045 conmon[33655]: cluster 2021-07-31T01:57:35.940485+0000 mgr.x (mgr.24542) 446 : cluster [DBG] pgmap v897: 25 pgs: 25 active+clean; 649 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 15 MiB/s rd, 3.5 MiB/s wr, 26 op/s 2021-07-31T01:57:37.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:36 smithi045 conmon[33655]: cluster 2021-07-31T01:57:36.668667 2021-07-31T01:57:37.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:36 smithi045 conmon[33655]: +0000 mon.a (mon.0) 1409 : cluster [DBG] osdmap e740: 8 total, 8 up, 8 in 2021-07-31T01:57:37.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:36 smithi045 conmon[39718]: cluster 2021-07-31T01:57:35.715253+0000 2021-07-31T01:57:37.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:36 smithi045 conmon[39718]: mon.a (mon.0) 1408 : cluster [DBG] osdmap e739: 8 total, 8 up, 8 in 2021-07-31T01:57:37.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:36 smithi045 conmon[39718]: cluster 2021 2021-07-31T01:57:37.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:36 smithi045 conmon[39718]: -07-31T01:57:35.940485+0000 mgr.x (mgr 2021-07-31T01:57:37.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:36 smithi045 conmon[39718]: .24542) 446 : cluster [DBG] pgmap v897: 25 pgs: 25 active+clean; 649 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 15 MiB/s rd, 3.5 MiB/s wr, 26 op/s 2021-07-31T01:57:37.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:36 smithi045 conmon[39718]: cluster 2021-07-31T01:57 2021-07-31T01:57:37.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:36 smithi045 conmon[39718]: :36.668667+0000 mon.a (mon.0) 2021-07-31T01:57:37.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:36 smithi045 conmon[39718]: 1409 : cluster [DBG] osdmap e740: 8 total, 8 up, 8 in 2021-07-31T01:57:37.730 INFO:tasks.rados.rados.0.smithi045.stdout:3615: done (4 left) 2021-07-31T01:57:37.731 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 17 current snap is 351 2021-07-31T01:57:37.731 INFO:tasks.rados.rados.0.smithi045.stdout:3616: seq_num 1110 ranges {3571712=598016,4169728=466944,4636672=450560,5087232=286720} 2021-07-31T01:57:37.755 INFO:tasks.rados.rados.0.smithi045.stdout:3616: writing smithi04569269-17 from 3571712 to 4169728 tid 1 2021-07-31T01:57:37.757 INFO:tasks.rados.rados.0.smithi045.stdout:3616: writing smithi04569269-17 from 4169728 to 4636672 tid 2 2021-07-31T01:57:37.760 INFO:tasks.rados.rados.0.smithi045.stdout:3616: writing smithi04569269-17 from 4636672 to 5087232 tid 3 2021-07-31T01:57:37.762 INFO:tasks.rados.rados.0.smithi045.stdout:3616: writing smithi04569269-17 from 5087232 to 5373952 tid 4 2021-07-31T01:57:37.763 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 10 current snap is 351 2021-07-31T01:57:37.763 INFO:tasks.rados.rados.0.smithi045.stdout:3611: finishing write tid 2 to smithi04569269-34 2021-07-31T01:57:37.763 INFO:tasks.rados.rados.0.smithi045.stdout:3611: finishing write tid 3 to smithi04569269-34 2021-07-31T01:57:37.763 INFO:tasks.rados.rados.0.smithi045.stdout:3611: finishing write tid 4 to smithi04569269-34 2021-07-31T01:57:37.764 INFO:tasks.rados.rados.0.smithi045.stdout:3617: seq_num 1111 ranges {10158080=614400,10772480=753664,11526144=450560} 2021-07-31T01:57:37.821 INFO:tasks.rados.rados.0.smithi045.stdout:3617: writing smithi04569269-10 from 10158080 to 10772480 tid 1 2021-07-31T01:57:37.825 INFO:tasks.rados.rados.0.smithi045.stdout:3617: writing smithi04569269-10 from 10772480 to 11526144 tid 2 2021-07-31T01:57:37.828 INFO:tasks.rados.rados.0.smithi045.stdout:3617: writing smithi04569269-10 from 11526144 to 11976704 tid 3 2021-07-31T01:57:37.829 INFO:tasks.rados.rados.0.smithi045.stdout:3618: copy_from oid 42 from oid 48 current snap is 351 2021-07-31T01:57:37.829 INFO:tasks.rados.rados.0.smithi045.stdout:3611: finishing write tid 5 to smithi04569269-34 2021-07-31T01:57:37.829 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 34 v 1881 (ObjNum 1107 snap 351 seq_num 1107) dirty exists 2021-07-31T01:57:37.829 INFO:tasks.rados.rados.0.smithi045.stdout:3611: left oid 34 (ObjNum 1107 snap 351 seq_num 1107) 2021-07-31T01:57:37.830 INFO:tasks.rados.rados.0.smithi045.stdout:3611: done (6 left) 2021-07-31T01:57:37.830 INFO:tasks.rados.rados.0.smithi045.stdout:3619: delete oid 41 current snap is 351 2021-07-31T01:57:37.830 INFO:tasks.rados.rados.0.smithi045.stdout:3612: finishing write tid 1 to smithi04569269-21 2021-07-31T01:57:37.831 INFO:tasks.rados.rados.0.smithi045.stdout:3612: finishing write tid 2 to smithi04569269-21 2021-07-31T01:57:37.832 INFO:tasks.rados.rados.0.smithi045.stdout:3612: finishing write tid 3 to smithi04569269-21 2021-07-31T01:57:37.832 INFO:tasks.rados.rados.0.smithi045.stdout:3612: finishing write tid 4 to smithi04569269-21 2021-07-31T01:57:37.832 INFO:tasks.rados.rados.0.smithi045.stdout:3612: finishing write tid 5 to smithi04569269-21 2021-07-31T01:57:37.832 INFO:tasks.rados.rados.0.smithi045.stdout:3613: expect (ObjNum -1105914768 snap 21873 seq_num 2820255792) 2021-07-31T01:57:37.834 INFO:tasks.rados.rados.0.smithi045.stdout:3612: finishing write tid 6 to smithi04569269-21 2021-07-31T01:57:37.834 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 21 v 2535 (ObjNum 1108 snap 351 seq_num 1108) dirty exists 2021-07-31T01:57:37.835 INFO:tasks.rados.rados.0.smithi045.stdout:3612: left oid 21 (ObjNum 1108 snap 351 seq_num 1108) 2021-07-31T01:57:37.835 INFO:tasks.rados.rados.0.smithi045.stdout:3614: finishing write tid 1 to smithi04569269-14 2021-07-31T01:57:37.835 INFO:tasks.rados.rados.0.smithi045.stdout:3614: finishing write tid 2 to smithi04569269-14 2021-07-31T01:57:37.835 INFO:tasks.rados.rados.0.smithi045.stdout:3614: finishing write tid 3 to smithi04569269-14 2021-07-31T01:57:37.835 INFO:tasks.rados.rados.0.smithi045.stdout:3614: finishing write tid 4 to smithi04569269-14 2021-07-31T01:57:37.836 INFO:tasks.rados.rados.0.smithi045.stdout:3614: finishing write tid 5 to smithi04569269-14 2021-07-31T01:57:37.836 INFO:tasks.rados.rados.0.smithi045.stdout:3614: finishing write tid 6 to smithi04569269-14 2021-07-31T01:57:37.836 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 14 v 2065 (ObjNum 1109 snap 351 seq_num 1109) dirty exists 2021-07-31T01:57:37.836 INFO:tasks.rados.rados.0.smithi045.stdout:3614: left oid 14 (ObjNum 1109 snap 351 seq_num 1109) 2021-07-31T01:57:37.837 INFO:tasks.rados.rados.0.smithi045.stdout:3616: finishing write tid 1 to smithi04569269-17 2021-07-31T01:57:37.837 INFO:tasks.rados.rados.0.smithi045.stdout:3616: finishing write tid 2 to smithi04569269-17 2021-07-31T01:57:37.837 INFO:tasks.rados.rados.0.smithi045.stdout:3616: finishing write tid 3 to smithi04569269-17 2021-07-31T01:57:37.837 INFO:tasks.rados.rados.0.smithi045.stdout:3616: finishing write tid 4 to smithi04569269-17 2021-07-31T01:57:37.837 INFO:tasks.rados.rados.0.smithi045.stdout:3616: finishing write tid 5 to smithi04569269-17 2021-07-31T01:57:37.838 INFO:tasks.rados.rados.0.smithi045.stdout:3616: finishing write tid 6 to smithi04569269-17 2021-07-31T01:57:37.838 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 17 v 2355 (ObjNum 1110 snap 351 seq_num 1110) dirty exists 2021-07-31T01:57:37.838 INFO:tasks.rados.rados.0.smithi045.stdout:3616: left oid 17 (ObjNum 1110 snap 351 seq_num 1110) 2021-07-31T01:57:37.838 INFO:tasks.rados.rados.0.smithi045.stdout:3617: finishing write tid 1 to smithi04569269-10 2021-07-31T01:57:37.839 INFO:tasks.rados.rados.0.smithi045.stdout:3617: finishing write tid 2 to smithi04569269-10 2021-07-31T01:57:37.839 INFO:tasks.rados.rados.0.smithi045.stdout:3612: done (6 left) 2021-07-31T01:57:37.839 INFO:tasks.rados.rados.0.smithi045.stdout:3613: done (5 left) 2021-07-31T01:57:37.839 INFO:tasks.rados.rados.0.smithi045.stdout:3614: done (4 left) 2021-07-31T01:57:37.839 INFO:tasks.rados.rados.0.smithi045.stdout:3616: done (3 left) 2021-07-31T01:57:37.840 INFO:tasks.rados.rados.0.smithi045.stdout:3619: done (2 left) 2021-07-31T01:57:37.840 INFO:tasks.rados.rados.0.smithi045.stdout:3620: copy_from oid 13 from oid 17 current snap is 351 2021-07-31T01:57:37.840 INFO:tasks.rados.rados.0.smithi045.stdout:3621: copy_from oid 35 from oid 41 current snap is 351 2021-07-31T01:57:37.840 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 43 current snap is 351 2021-07-31T01:57:37.840 INFO:tasks.rados.rados.0.smithi045.stdout:3622: seq_num 1113 ranges {4259840=647168,4907008=524288,5431296=622592,6053888=57344} 2021-07-31T01:57:37.858 INFO:tasks.rados.rados.0.smithi045.stdout:3622: writing smithi04569269-43 from 4259840 to 4907008 tid 1 2021-07-31T01:57:37.861 INFO:tasks.rados.rados.0.smithi045.stdout:3622: writing smithi04569269-43 from 4907008 to 5431296 tid 2 2021-07-31T01:57:37.865 INFO:tasks.rados.rados.0.smithi045.stdout:3622: writing smithi04569269-43 from 5431296 to 6053888 tid 3 2021-07-31T01:57:37.866 INFO:tasks.rados.rados.0.smithi045.stdout:3622: writing smithi04569269-43 from 6053888 to 6111232 tid 4 2021-07-31T01:57:37.866 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 19 current snap is 351 2021-07-31T01:57:37.866 INFO:tasks.rados.rados.0.smithi045.stdout:3621: finishing copy_from to smithi04569269-35 2021-07-31T01:57:37.867 INFO:tasks.rados.rados.0.smithi045.stdout:3621: got expected ENOENT (src dne) 2021-07-31T01:57:37.867 INFO:tasks.rados.rados.0.smithi045.stdout:3621: finishing copy_from racing read to smithi04569269-35 2021-07-31T01:57:37.867 INFO:tasks.rados.rados.0.smithi045.stdout:3617: finishing write tid 3 to smithi04569269-10 2021-07-31T01:57:37.867 INFO:tasks.rados.rados.0.smithi045.stdout:3617: finishing write tid 4 to smithi04569269-10 2021-07-31T01:57:37.868 INFO:tasks.rados.rados.0.smithi045.stdout:3623: seq_num 1114 ranges {1851392=663552,2514944=802816,3317760=401408} 2021-07-31T01:57:37.879 INFO:tasks.rados.rados.0.smithi045.stdout:3623: writing smithi04569269-19 from 1851392 to 2514944 tid 1 2021-07-31T01:57:37.884 INFO:tasks.rados.rados.0.smithi045.stdout:3623: writing smithi04569269-19 from 2514944 to 3317760 tid 2 2021-07-31T01:57:37.886 INFO:tasks.rados.rados.0.smithi045.stdout:3623: writing smithi04569269-19 from 3317760 to 3719168 tid 3 2021-07-31T01:57:37.887 INFO:tasks.rados.rados.0.smithi045.stdout:3621: done (5 left) 2021-07-31T01:57:37.887 INFO:tasks.rados.rados.0.smithi045.stdout:3624: snap_remove snap 296 2021-07-31T01:57:38.740 INFO:tasks.rados.rados.0.smithi045.stdout:3624: done (5 left) 2021-07-31T01:57:38.740 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 14 current snap is 351 2021-07-31T01:57:38.741 INFO:tasks.rados.rados.0.smithi045.stdout:3625: seq_num 1115 ranges {3399680=679936,4079616=688128,4767744=516096} 2021-07-31T01:57:38.762 INFO:tasks.rados.rados.0.smithi045.stdout:3625: writing smithi04569269-14 from 3399680 to 4079616 tid 1 2021-07-31T01:57:38.766 INFO:tasks.rados.rados.0.smithi045.stdout:3625: writing smithi04569269-14 from 4079616 to 4767744 tid 2 2021-07-31T01:57:38.770 INFO:tasks.rados.rados.0.smithi045.stdout:3625: writing smithi04569269-14 from 4767744 to 5283840 tid 3 2021-07-31T01:57:38.771 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 1 current snap is 351 2021-07-31T01:57:38.771 INFO:tasks.rados.rados.0.smithi045.stdout:3617: finishing write tid 5 to smithi04569269-10 2021-07-31T01:57:38.771 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 10 v 2359 (ObjNum 1111 snap 351 seq_num 1111) dirty exists 2021-07-31T01:57:38.771 INFO:tasks.rados.rados.0.smithi045.stdout:3617: left oid 10 (ObjNum 1111 snap 351 seq_num 1111) 2021-07-31T01:57:38.772 INFO:tasks.rados.rados.0.smithi045.stdout:3626: seq_num 1116 ranges {6217728=696320,6914048=573440,7487488=450560,7938048=180224} 2021-07-31T01:57:38.808 INFO:tasks.rados.rados.0.smithi045.stdout:3626: writing smithi04569269-1 from 6217728 to 6914048 tid 1 2021-07-31T01:57:38.811 INFO:tasks.rados.rados.0.smithi045.stdout:3626: writing smithi04569269-1 from 6914048 to 7487488 tid 2 2021-07-31T01:57:38.814 INFO:tasks.rados.rados.0.smithi045.stdout:3626: writing smithi04569269-1 from 7487488 to 7938048 tid 3 2021-07-31T01:57:38.817 INFO:tasks.rados.rados.0.smithi045.stdout:3626: writing smithi04569269-1 from 7938048 to 8118272 tid 4 2021-07-31T01:57:38.818 INFO:tasks.rados.rados.0.smithi045.stdout:3617: done (6 left) 2021-07-31T01:57:38.818 INFO:tasks.rados.rados.0.smithi045.stdout:3627: rollback oid 6 current snap is 351 2021-07-31T01:57:38.818 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 6 to 266 2021-07-31T01:57:38.818 INFO:tasks.rados.rados.0.smithi045.stdout:3618: finishing copy_from to smithi04569269-42 2021-07-31T01:57:38.819 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 42 v 1873 (ObjNum 1036 snap 338 seq_num 1036) dirty exists 2021-07-31T01:57:38.819 INFO:tasks.rados.rados.0.smithi045.stdout:3618: finishing copy_from racing read to smithi04569269-42 2021-07-31T01:57:38.819 INFO:tasks.rados.rados.0.smithi045.stdout:3620: finishing copy_from to smithi04569269-13 2021-07-31T01:57:38.819 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 13 v 2355 (ObjNum 1110 snap 351 seq_num 1110) dirty exists 2021-07-31T01:57:38.820 INFO:tasks.rados.rados.0.smithi045.stdout:3620: finishing copy_from racing read to smithi04569269-13 2021-07-31T01:57:38.820 INFO:tasks.rados.rados.0.smithi045.stdout:3622: finishing write tid 1 to smithi04569269-43 2021-07-31T01:57:38.820 INFO:tasks.rados.rados.0.smithi045.stdout:3622: finishing write tid 2 to smithi04569269-43 2021-07-31T01:57:38.820 INFO:tasks.rados.rados.0.smithi045.stdout:3622: finishing write tid 3 to smithi04569269-43 2021-07-31T01:57:38.820 INFO:tasks.rados.rados.0.smithi045.stdout:3618: done (6 left) 2021-07-31T01:57:38.821 INFO:tasks.rados.rados.0.smithi045.stdout:3620: done (5 left) 2021-07-31T01:57:38.821 INFO:tasks.rados.rados.0.smithi045.stdout:3628: read oid 23 snap -1 2021-07-31T01:57:38.821 INFO:tasks.rados.rados.0.smithi045.stdout:3628: expect (ObjNum 1049 snap 342 seq_num 1049) 2021-07-31T01:57:38.821 INFO:tasks.rados.rados.0.smithi045.stdout:3629: read oid 30 snap -1 2021-07-31T01:57:38.822 INFO:tasks.rados.rados.0.smithi045.stdout:3629: expect deleted 2021-07-31T01:57:38.822 INFO:tasks.rados.rados.0.smithi045.stdout:3630: rollback oid 31 current snap is 351 2021-07-31T01:57:38.822 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 31 to 331 2021-07-31T01:57:38.822 INFO:tasks.rados.rados.0.smithi045.stdout:3622: finishing write tid 4 to smithi04569269-43 2021-07-31T01:57:38.823 INFO:tasks.rados.rados.0.smithi045.stdout:3622: finishing write tid 5 to smithi04569269-43 2021-07-31T01:57:38.823 INFO:tasks.rados.rados.0.smithi045.stdout:3622: finishing write tid 6 to smithi04569269-43 2021-07-31T01:57:38.823 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 43 v 2070 (ObjNum 1113 snap 351 seq_num 1113) dirty exists 2021-07-31T01:57:38.823 INFO:tasks.rados.rados.0.smithi045.stdout:3622: left oid 43 (ObjNum 1113 snap 351 seq_num 1113) 2021-07-31T01:57:38.824 INFO:tasks.rados.rados.0.smithi045.stdout:3623: finishing write tid 1 to smithi04569269-19 2021-07-31T01:57:38.824 INFO:tasks.rados.rados.0.smithi045.stdout:3623: finishing write tid 2 to smithi04569269-19 2021-07-31T01:57:38.824 INFO:tasks.rados.rados.0.smithi045.stdout:3623: finishing write tid 3 to smithi04569269-19 2021-07-31T01:57:38.824 INFO:tasks.rados.rados.0.smithi045.stdout:3623: finishing write tid 4 to smithi04569269-19 2021-07-31T01:57:38.824 INFO:tasks.rados.rados.0.smithi045.stdout:3623: finishing write tid 5 to smithi04569269-19 2021-07-31T01:57:38.825 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 19 v 1857 (ObjNum 1114 snap 351 seq_num 1114) dirty exists 2021-07-31T01:57:38.825 INFO:tasks.rados.rados.0.smithi045.stdout:3623: left oid 19 (ObjNum 1114 snap 351 seq_num 1114) 2021-07-31T01:57:38.825 INFO:tasks.rados.rados.0.smithi045.stdout:3622: done (7 left) 2021-07-31T01:57:38.825 INFO:tasks.rados.rados.0.smithi045.stdout:3623: done (6 left) 2021-07-31T01:57:38.826 INFO:tasks.rados.rados.0.smithi045.stdout:3631: rollback oid 20 current snap is 351 2021-07-31T01:57:38.826 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 20 to 329 2021-07-31T01:57:38.826 INFO:tasks.rados.rados.0.smithi045.stdout:3625: finishing write tid 1 to smithi04569269-14 2021-07-31T01:57:38.826 INFO:tasks.rados.rados.0.smithi045.stdout:3625: finishing write tid 2 to smithi04569269-14 2021-07-31T01:57:38.827 INFO:tasks.rados.rados.0.smithi045.stdout:3625: finishing write tid 3 to smithi04569269-14 2021-07-31T01:57:38.827 INFO:tasks.rados.rados.0.smithi045.stdout:3625: finishing write tid 4 to smithi04569269-14 2021-07-31T01:57:38.827 INFO:tasks.rados.rados.0.smithi045.stdout:3625: finishing write tid 5 to smithi04569269-14 2021-07-31T01:57:38.827 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 14 v 2074 (ObjNum 1115 snap 351 seq_num 1115) dirty exists 2021-07-31T01:57:38.828 INFO:tasks.rados.rados.0.smithi045.stdout:3625: left oid 14 (ObjNum 1115 snap 351 seq_num 1115) 2021-07-31T01:57:38.828 INFO:tasks.rados.rados.0.smithi045.stdout:3625: done (6 left) 2021-07-31T01:57:38.828 INFO:tasks.rados.rados.0.smithi045.stdout:3632: delete oid 10 current snap is 351 2021-07-31T01:57:38.828 INFO:tasks.rados.rados.0.smithi045.stdout:3626: finishing write tid 1 to smithi04569269-1 2021-07-31T01:57:38.828 INFO:tasks.rados.rados.0.smithi045.stdout:3626: finishing write tid 2 to smithi04569269-1 2021-07-31T01:57:38.830 INFO:tasks.rados.rados.0.smithi045.stdout:3626: finishing write tid 3 to smithi04569269-1 2021-07-31T01:57:38.830 INFO:tasks.rados.rados.0.smithi045.stdout:3626: finishing write tid 4 to smithi04569269-1 2021-07-31T01:57:38.830 INFO:tasks.rados.rados.0.smithi045.stdout:3629: done (6 left) 2021-07-31T01:57:38.830 INFO:tasks.rados.rados.0.smithi045.stdout:3632: done (5 left) 2021-07-31T01:57:38.831 INFO:tasks.rados.rados.0.smithi045.stdout:3633: rmattr oid 12 current snap is 351 2021-07-31T01:57:38.831 INFO:tasks.rados.rados.0.smithi045.stdout:3633: done (5 left) 2021-07-31T01:57:38.831 INFO:tasks.rados.rados.0.smithi045.stdout:3634: rollback oid 38 current snap is 351 2021-07-31T01:57:38.831 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 38 to 343 2021-07-31T01:57:38.832 INFO:tasks.rados.rados.0.smithi045.stdout:3635: snap_remove snap 347 2021-07-31T01:57:39.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:38 smithi154 conmon[34553]: cluster 2021-07-31T01:57:39.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:38 smithi154 conmon[34553]: 2021-07-31T01:57:37.729141+0000 2021-07-31T01:57:39.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:38 smithi154 conmon[34553]: mon.a (mon.0) 1410 : cluster [DBG] osdmap e741: 8 total, 8 up, 8 in 2021-07-31T01:57:39.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:38 smithi154 conmon[34553]: cluster 2021-07-31T01:57:39.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:38 smithi154 conmon[34553]: 2021-07-31T01:57:37.940857+0000 mgr.x (mgr.24542) 447 : cluster [DBG] pgmap v900: 25 pgs: 25 active+clean; 649 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2021-07-31T01:57:39.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:38 smithi154 conmon[34553]: 2021-07-31T01:57:39.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:38 smithi045 conmon[33655]: cluster 2021-07-31T01:57:37.729141+0000 mon.a (mon.0) 1410 2021-07-31T01:57:39.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:38 smithi045 conmon[33655]: : cluster [DBG] osdmap e741: 8 total, 8 up, 8 in 2021-07-31T01:57:39.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:38 smithi045 conmon[33655]: cluster 2021-07-31T01:57:37.940857+0000 mgr.x (mgr.24542) 447 : cluster [DBG] pgmap v900: 25 pgs: 25 active+clean; 649 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2021-07-31T01:57:39.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:38 smithi045 conmon[39718]: cluster 2021-07-31T01:57:37 2021-07-31T01:57:39.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:38 smithi045 conmon[39718]: .729141+0000 mon.a (mon.0) 1410 : cluster [DBG] osdmap e741: 8 total, 8 up, 8 in 2021-07-31T01:57:39.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:38 smithi045 conmon[39718]: cluster 2021-07-31T01:57:37 2021-07-31T01:57:39.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:38 smithi045 conmon[39718]: .940857+0000 mgr.x (mgr.24542) 447 : cluster [DBG] 2021-07-31T01:57:39.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:38 smithi045 conmon[39718]: pgmap v900: 25 pgs: 25 active+clean; 649 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2021-07-31T01:57:39.738 INFO:tasks.rados.rados.0.smithi045.stdout:3635: done (6 left) 2021-07-31T01:57:39.740 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 36 current snap is 351 2021-07-31T01:57:39.740 INFO:tasks.rados.rados.0.smithi045.stdout:3627: finishing rollback tid 0 to smithi04569269-6 2021-07-31T01:57:39.740 INFO:tasks.rados.rados.0.smithi045.stdout:3627: finishing rollback tid 1 to smithi04569269-6 2021-07-31T01:57:39.741 INFO:tasks.rados.rados.0.smithi045.stdout:3636: seq_num 1118 ranges {3997696=729088,4726784=794624,5521408=409600} 2021-07-31T01:57:39.763 INFO:tasks.rados.rados.0.smithi045.stdout:3636: writing smithi04569269-36 from 3997696 to 4726784 tid 1 2021-07-31T01:57:39.768 INFO:tasks.rados.rados.0.smithi045.stdout:3636: writing smithi04569269-36 from 4726784 to 5521408 tid 2 2021-07-31T01:57:39.771 INFO:tasks.rados.rados.0.smithi045.stdout:3636: writing smithi04569269-36 from 5521408 to 5931008 tid 3 2021-07-31T01:57:39.771 INFO:tasks.rados.rados.0.smithi045.stdout:3637: snap_create 2021-07-31T01:57:39.772 INFO:tasks.rados.rados.0.smithi045.stdout:3626: finishing write tid 5 to smithi04569269-1 2021-07-31T01:57:39.772 INFO:tasks.rados.rados.0.smithi045.stdout:3630: finishing rollback tid 0 to smithi04569269-31 2021-07-31T01:57:39.773 INFO:tasks.rados.rados.0.smithi045.stdout:3627: finishing rollback tid 2 to smithi04569269-6 2021-07-31T01:57:39.773 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 6 v 2358 (ObjNum 822 snap 258 seq_num 822) dirty exists 2021-07-31T01:57:39.773 INFO:tasks.rados.rados.0.smithi045.stdout:3626: finishing write tid 6 to smithi04569269-1 2021-07-31T01:57:39.773 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 1 v 1886 (ObjNum 1116 snap 351 seq_num 1116) dirty exists 2021-07-31T01:57:39.773 INFO:tasks.rados.rados.0.smithi045.stdout:3626: left oid 1 (ObjNum 1116 snap 351 seq_num 1116) 2021-07-31T01:57:39.774 INFO:tasks.rados.rados.0.smithi045.stdout:3630: finishing rollback tid 1 to smithi04569269-31 2021-07-31T01:57:39.774 INFO:tasks.rados.rados.0.smithi045.stdout:3630: finishing rollback tid 2 to smithi04569269-31 2021-07-31T01:57:39.774 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 31 v 2569 (ObjNum 1006 snap 322 seq_num 1006) dirty exists 2021-07-31T01:57:39.774 INFO:tasks.rados.rados.0.smithi045.stdout:3628: expect (ObjNum 1049 snap 342 seq_num 1049) 2021-07-31T01:57:39.859 INFO:tasks.rados.rados.0.smithi045.stdout:3634: finishing rollback tid 0 to smithi04569269-38 2021-07-31T01:57:39.860 INFO:tasks.rados.rados.0.smithi045.stdout:3634: finishing rollback tid 1 to smithi04569269-38 2021-07-31T01:57:39.861 INFO:tasks.rados.rados.0.smithi045.stdout:3631: finishing rollback tid 0 to smithi04569269-20 2021-07-31T01:57:39.861 INFO:tasks.rados.rados.0.smithi045.stdout:3634: finishing rollback tid 2 to smithi04569269-38 2021-07-31T01:57:39.862 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 38 v 2049 (ObjNum 1037 snap 338 seq_num 1037) dirty exists 2021-07-31T01:57:39.862 INFO:tasks.rados.rados.0.smithi045.stdout:3631: finishing rollback tid 1 to smithi04569269-20 2021-07-31T01:57:39.862 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 20 v 2572 (ObjNum 1 snap 32731 seq_num 3188952672) dirty dne 2021-07-31T01:57:39.862 INFO:tasks.rados.rados.0.smithi045.stdout:3636: finishing write tid 1 to smithi04569269-36 2021-07-31T01:57:39.863 INFO:tasks.rados.rados.0.smithi045.stdout:3636: finishing write tid 2 to smithi04569269-36 2021-07-31T01:57:39.863 INFO:tasks.rados.rados.0.smithi045.stdout:3636: finishing write tid 3 to smithi04569269-36 2021-07-31T01:57:39.863 INFO:tasks.rados.rados.0.smithi045.stdout:3636: finishing write tid 4 to smithi04569269-36 2021-07-31T01:57:39.863 INFO:tasks.rados.rados.0.smithi045.stdout:3636: finishing write tid 5 to smithi04569269-36 2021-07-31T01:57:39.863 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 36 v 2577 (ObjNum 1118 snap 351 seq_num 1118) dirty exists 2021-07-31T01:57:39.864 INFO:tasks.rados.rados.0.smithi045.stdout:3636: left oid 36 (ObjNum 1118 snap 351 seq_num 1118) 2021-07-31T01:57:40.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:39 smithi154 conmon[34553]: cluster 2021-07-31T01:57:38.736112+0000 2021-07-31T01:57:40.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:39 smithi154 conmon[34553]: mon.a (mon.0) 1411 : cluster [DBG] osdmap e742: 8 total, 8 up, 8 in 2021-07-31T01:57:40.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:39 smithi045 conmon[39718]: cluster 2021-07-31T01:57:38.736112 2021-07-31T01:57:40.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:39 smithi045 conmon[39718]: +0000 mon.a (mon.0) 1411 : cluster [DBG] osdmap e742: 8 total, 8 up, 8 in 2021-07-31T01:57:40.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:39 smithi045 conmon[33655]: cluster 2021-07-31T01:57:38.736112+0000 mon.a (mon.0) 1411 : cluster 2021-07-31T01:57:40.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:39 smithi045 conmon[33655]: [DBG] osdmap e742: 8 total, 8 up, 8 in 2021-07-31T01:57:40.751 INFO:tasks.rados.rados.0.smithi045.stdout:3626: done (7 left) 2021-07-31T01:57:40.751 INFO:tasks.rados.rados.0.smithi045.stdout:3627: done (6 left) 2021-07-31T01:57:40.751 INFO:tasks.rados.rados.0.smithi045.stdout:3628: done (5 left) 2021-07-31T01:57:40.751 INFO:tasks.rados.rados.0.smithi045.stdout:3630: done (4 left) 2021-07-31T01:57:40.752 INFO:tasks.rados.rados.0.smithi045.stdout:3631: done (3 left) 2021-07-31T01:57:40.752 INFO:tasks.rados.rados.0.smithi045.stdout:3634: done (2 left) 2021-07-31T01:57:40.752 INFO:tasks.rados.rados.0.smithi045.stdout:3636: done (1 left) 2021-07-31T01:57:40.752 INFO:tasks.rados.rados.0.smithi045.stdout:3637: done (0 left) 2021-07-31T01:57:40.752 INFO:tasks.rados.rados.0.smithi045.stdout:3638: snap_create 2021-07-31T01:57:41.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:40 smithi154 conmon[34553]: cluster 2021-07-31T01:57:39.736774+0000 mon.a (mon.0) 2021-07-31T01:57:41.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:40 smithi154 conmon[34553]: 1412 : cluster [DBG] osdmap e743: 8 total, 8 up, 8 in 2021-07-31T01:57:41.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:40 smithi154 conmon[34553]: cluster 2021-07-31T01:57:39.941257+0000 mgr.x (mgr.24542) 448 : cluster [DBG] pgmap v903: 25 pgs: 25 active+clean; 683 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 2.4 MiB/s rd, 6.2 MiB/s wr, 21 op/s 2021-07-31T01:57:41.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:40 smithi045 conmon[39718]: cluster 2021-07-31T01:57:39.736774+0000 2021-07-31T01:57:41.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:40 smithi045 conmon[39718]: mon.a (mon.0) 1412 : cluster [DBG] osdmap e743: 8 total, 8 up, 8 in 2021-07-31T01:57:41.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:40 smithi045 conmon[39718]: cluster 2021-07- 2021-07-31T01:57:41.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:40 smithi045 conmon[39718]: 31T01:57:39.941257+0000 mgr.x (mgr.24542) 448 : cluster 2021-07-31T01:57:41.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:40 smithi045 conmon[39718]: [DBG] pgmap v903: 25 pgs: 25 active+clean; 683 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 2.4 MiB/s rd, 6.2 MiB/s wr, 21 op/s 2021-07-31T01:57:41.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:40 smithi045 conmon[33655]: cluster 2021-07-31T01:57:39.736774+0000 mon.a (mon.0) 1412 : cluster 2021-07-31T01:57:41.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:40 smithi045 conmon[33655]: [DBG] osdmap e743: 8 total, 8 up, 8 in 2021-07-31T01:57:41.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:40 smithi045 conmon[33655]: cluster 2021-07-31T01:57:39.941257+0000 mgr.x (mgr.24542) 448 : cluster [DBG] pgmap v903: 25 pgs: 25 active+clean; 683 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 2.4 MiB/s rd, 6.2 MiB/s wr, 21 op/s 2021-07-31T01:57:41.670 INFO:tasks.rados.rados.0.smithi045.stdout:3638: done (0 left) 2021-07-31T01:57:41.671 INFO:tasks.rados.rados.0.smithi045.stdout:3639: snap_remove snap 302 2021-07-31T01:57:42.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:41 smithi154 conmon[34553]: cluster 2021-07-31T01:57:40.749519+0000 mon.a (mon.0) 1413 : cluster [DBG] osdmap e744: 8 total, 8 up, 8 in 2021-07-31T01:57:42.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:41 smithi154 conmon[34553]: cluster 2021-07-31T01:57:41.670024+0000 mon.a (mon.0) 1414 : cluster [DBG] osdmap e745: 8 total, 8 up, 8 in 2021-07-31T01:57:42.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:41 smithi045 conmon[39718]: cluster 2021-07-31T01:57:40 2021-07-31T01:57:42.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:41 smithi045 conmon[39718]: .749519+0000 mon.a (mon.0) 1413 : cluster [DBG] osdmap e744: 8 total, 8 up, 8 in 2021-07-31T01:57:42.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:41 smithi045 conmon[39718]: cluster 2021-07-31T01:57:42.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:41 smithi045 conmon[39718]: 2021-07- 2021-07-31T01:57:42.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:41 smithi045 conmon[39718]: 31T01:57:41. 2021-07-31T01:57:42.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:41 smithi045 conmon[39718]: 670024+0000 mon.a (mon.0 2021-07-31T01:57:42.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:41 smithi045 conmon[39718]: ) 1414 : cluster [DBG] osdmap e745: 8 total, 8 up, 8 in 2021-07-31T01:57:42.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:41 smithi045 conmon[39718]: 2021-07-31T01:57:42.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:41 smithi045 conmon[33655]: cluster 2021-07-31T01:57:40.749519+0000 mon.a (mon.0) 1413 : cluster [DBG] 2021-07-31T01:57:42.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:41 smithi045 conmon[33655]: osdmap e744: 8 total, 8 up, 8 in 2021-07-31T01:57:42.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:41 smithi045 conmon[33655]: cluster 2021-07-31T01:57:41.670024+0000 mon.a (mon.0) 1414 : cluster [DBG] osdmap e745: 8 total, 8 up, 8 in 2021-07-31T01:57:42.680 INFO:tasks.rados.rados.0.smithi045.stdout:3639: done (0 left) 2021-07-31T01:57:42.680 INFO:tasks.rados.rados.0.smithi045.stdout:3640: snap_remove snap 263 2021-07-31T01:57:43.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:42 smithi045 conmon[39718]: cluster 2021-07-31T01:57:41.941716+0000 mgr.x ( 2021-07-31T01:57:43.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:42 smithi045 conmon[39718]: mgr.24542) 449 : cluster [DBG] pgmap v906: 25 pgs: 25 active+clean; 683 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 2.4 MiB/s rd, 6.2 MiB/s wr, 21 op/s 2021-07-31T01:57:43.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:42 smithi045 conmon[39718]: cluster 2021-07-31T 2021-07-31T01:57:43.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:42 smithi045 conmon[39718]: 01:57:42.678362+0000 mon.a (mon.0) 1415 : cluster [DBG] osdmap e746: 8 total, 8 up, 8 in 2021-07-31T01:57:43.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:42 smithi045 conmon[33655]: cluster 2021-07-31T01:57:41.941716+0000 mgr.x (mgr.24542) 449 : cluster [DBG] 2021-07-31T01:57:43.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:42 smithi045 conmon[33655]: pgmap v906: 25 pgs: 25 active+clean; 683 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 2.4 MiB/s rd, 6.2 MiB/s wr, 21 op/s 2021-07-31T01:57:43.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:42 smithi045 conmon[33655]: cluster 2021-07-31T01:57:42.678362+0000 mon.a (mon.0) 1415 : cluster [DBG] osdmap e746: 8 total, 8 up, 8 in 2021-07-31T01:57:43.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:42 smithi154 conmon[34553]: cluster 2021-07-31T01:57:41.941716+0000 mgr.x (mgr.24542) 449 : cluster [DBG] pgmap v906: 25 pgs: 25 active+clean; 683 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 2.4 MiB/s rd, 6.2 MiB/s wr, 21 op/s 2021-07-31T01:57:43.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:42 smithi154 conmon[34553]: cluster 2021-07-31T01:57:42.678362+0000 mon.a (mon.0) 1415 : cluster [DBG] osdmap e746: 8 total, 8 up, 8 in 2021-07-31T01:57:43.680 INFO:tasks.rados.rados.0.smithi045.stdout:3640: done (0 left) 2021-07-31T01:57:43.680 INFO:tasks.rados.rados.0.smithi045.stdout:3641: rmattr oid 1 current snap is 353 2021-07-31T01:57:43.681 INFO:tasks.rados.rados.0.smithi045.stdout:3642: snap_remove snap 255 2021-07-31T01:57:43.917 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T01:57:44.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:57:43 smithi045 conmon[61210]: level=error ts=2021-07-31T01:57:43.527Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:57:44.027 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:57:43 smithi045 conmon[61210]: level=error ts=2021-07-31T01:57:43.527Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:57:44.027 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:57:43 smithi045 conmon[61210]: level=error ts=2021-07-31T01:57:43.527Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T01:57:44.192 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T01:57:44.192 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 12m ago 13m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T01:57:44.193 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 10m ago 13m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T01:57:44.193 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 10m ago 16m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T01:57:44.193 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 12m ago 18m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T01:57:44.193 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 12m ago 18m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T01:57:44.194 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 10m ago 16m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T01:57:44.194 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 12m ago 17m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T01:57:44.194 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 12m ago 13m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T01:57:44.194 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 10m ago 13m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T01:57:44.194 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 12m ago 16m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T01:57:44.195 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 12m ago 16m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T01:57:44.195 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 12m ago 15m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T01:57:44.195 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 12m ago 15m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T01:57:44.195 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 10m ago 15m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T01:57:44.196 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 10m ago 14m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T01:57:44.196 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 10m ago 14m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T01:57:44.196 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 10m ago 14m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T01:57:44.196 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 10m ago 13m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T01:57:44.527 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T01:57:44.528 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T01:57:44.528 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T01:57:44.528 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T01:57:44.529 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T01:57:44.529 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T01:57:44.529 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T01:57:44.529 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T01:57:44.529 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T01:57:44.530 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T01:57:44.530 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T01:57:44.530 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T01:57:44.530 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T01:57:44.531 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T01:57:44.531 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T01:57:44.531 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T01:57:44.531 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T01:57:44.685 INFO:tasks.rados.rados.0.smithi045.stdout:3642: done (1 left) 2021-07-31T01:57:44.686 INFO:tasks.rados.rados.0.smithi045.stdout:3643: delete oid 14 current snap is 353 2021-07-31T01:57:44.686 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 1 v 1887 (ObjNum 1116 snap 351 seq_num 1116) dirty exists 2021-07-31T01:57:44.699 INFO:tasks.rados.rados.0.smithi045.stdout:3641: done (1 left) 2021-07-31T01:57:44.700 INFO:tasks.rados.rados.0.smithi045.stdout:3643: done (0 left) 2021-07-31T01:57:44.700 INFO:tasks.rados.rados.0.smithi045.stdout:3644: setattr oid 31 current snap is 353 2021-07-31T01:57:44.701 INFO:tasks.rados.rados.0.smithi045.stdout:3645: copy_from oid 2 from oid 7 current snap is 353 2021-07-31T01:57:44.701 INFO:tasks.rados.rados.0.smithi045.stdout:3646: read oid 50 snap -1 2021-07-31T01:57:44.702 INFO:tasks.rados.rados.0.smithi045.stdout:3646: expect (ObjNum 1066 snap 344 seq_num 1066) 2021-07-31T01:57:44.702 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 18 current snap is 353 2021-07-31T01:57:44.702 INFO:tasks.rados.rados.0.smithi045.stdout:3647: seq_num 1120 ranges {0=761856,761856=557056,1318912=425984,1744896=221184} 2021-07-31T01:57:44.706 INFO:tasks.rados.rados.0.smithi045.stdout:3647: writing smithi04569269-18 from 0 to 761856 tid 1 2021-07-31T01:57:44.710 INFO:tasks.rados.rados.0.smithi045.stdout:3647: writing smithi04569269-18 from 761856 to 1318912 tid 2 2021-07-31T01:57:44.712 INFO:tasks.rados.rados.0.smithi045.stdout:3647: writing smithi04569269-18 from 1318912 to 1744896 tid 3 2021-07-31T01:57:44.715 INFO:tasks.rados.rados.0.smithi045.stdout:3647: writing smithi04569269-18 from 1744896 to 1966080 tid 4 2021-07-31T01:57:44.715 INFO:tasks.rados.rados.0.smithi045.stdout:3648: read oid 21 snap -1 2021-07-31T01:57:44.715 INFO:tasks.rados.rados.0.smithi045.stdout:3648: expect (ObjNum 1108 snap 351 seq_num 1108) 2021-07-31T01:57:44.715 INFO:tasks.rados.rados.0.smithi045.stdout:3649: read oid 39 snap -1 2021-07-31T01:57:44.716 INFO:tasks.rados.rados.0.smithi045.stdout:3649: expect (ObjNum 1085 snap 347 seq_num 1085) 2021-07-31T01:57:44.716 INFO:tasks.rados.rados.0.smithi045.stdout:3650: snap_remove snap 317 2021-07-31T01:57:45.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:44 smithi154 conmon[34553]: cluster 2021-07-31T01:57:45.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:44 smithi154 conmon[34553]: 2021-07-31T01:57:45.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:44 smithi154 conmon[34553]: 2021-07-31T01:57 2021-07-31T01:57:45.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:44 smithi154 conmon[34553]: :43.679082+0000 mon.a (mon.0) 1416 : cluster [DBG] osdmap e747: 8 total, 8 up, 8 in 2021-07-31T01:57:45.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:44 smithi154 conmon[34553]: audit 2021-07 2021-07-31T01:57:45.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:44 smithi154 conmon[34553]: -31T01:57:43.903611+0000 mgr.x (mgr.24542) 450 : audit [DBG] 2021-07-31T01:57:45.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:44 smithi154 conmon[34553]: from='client.24979 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:57:45.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:44 smithi154 conmon[34553]: cluster 2021-07-31T01 2021-07-31T01:57:45.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:44 smithi154 conmon[34553]: :57:43.942354+0000 mgr.x (mgr.24542) 451 2021-07-31T01:57:45.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:44 smithi154 conmon[34553]: : cluster [DBG] pgmap v909: 25 pgs: 25 active+clean; 676 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 255 B/s rd, 472 KiB/s wr, 3 op/s 2021-07-31T01:57:45.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:44 smithi154 conmon[34553]: audit 2021-07-31T01:57:45.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:44 smithi154 conmon[34553]: 2021-07-31T01:57:44.527450+0000 2021-07-31T01:57:45.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:44 smithi154 conmon[34553]: mon.a (mon.0) 1417 : audit [DBG] from='client.? 172.21.15.45:0/2539186441' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T01:57:45.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:44 smithi045 conmon[33655]: cluster 2021-07-31T01:57:43.679082+0000 mon.a ( 2021-07-31T01:57:45.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:44 smithi045 conmon[33655]: mon.0) 1416 : cluster [DBG] osdmap e747: 8 total, 8 up, 8 in 2021-07-31T01:57:45.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:44 smithi045 conmon[33655]: audit 2021-07-31T01:57:43.903611+0000 mgr.x (mgr.24542) 450 : audit [DBG] from='client.24979 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:57:45.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:44 smithi045 conmon[33655]: cluster 2021-07-31T01:57:43.942354+0000 mgr.x ( 2021-07-31T01:57:45.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:44 smithi045 conmon[33655]: mgr.24542) 451 : cluster [DBG] pgmap v909: 25 pgs: 25 active+clean; 676 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 255 B/s rd, 472 KiB/s wr, 3 op/s 2021-07-31T01:57:45.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:44 smithi045 conmon[33655]: audit 2021-07-31T01: 2021-07-31T01:57:45.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:44 smithi045 conmon[33655]: 57:44.527450+0000 mon.a (mon.0) 1417 : audit 2021-07-31T01:57:45.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:44 smithi045 conmon[33655]: [DBG] from='client.? 172.21.15.45:0/2539186441' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T01:57:45.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:44 smithi045 conmon[39718]: cluster 2021-07-31T01:57:43. 2021-07-31T01:57:45.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:44 smithi045 conmon[39718]: 679082+0000 mon.a (mon.0) 1416 : cluster [DBG] osdmap e747: 8 total, 8 up, 8 in 2021-07-31T01:57:45.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:44 smithi045 conmon[39718]: audit 2021-07-31T01:57:45.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:44 smithi045 conmon[39718]: 2021-07-31T01:57:43.903611+0000 2021-07-31T01:57:45.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:44 smithi045 conmon[39718]: mgr.x (mgr.24542) 450 : audit [DBG] 2021-07-31T01:57:45.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:44 smithi045 conmon[39718]: from='client.24979 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:57:45.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:44 smithi045 conmon[39718]: cluster 2021-07-31T01:57:43 2021-07-31T01:57:45.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:44 smithi045 conmon[39718]: .942354+0000 mgr.x (mgr.24542) 451 2021-07-31T01:57:45.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:44 smithi045 conmon[39718]: : cluster [DBG] pgmap v909: 25 pgs: 25 active+clean; 676 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 255 B/s rd, 472 KiB/s wr, 3 op/s 2021-07-31T01:57:45.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:44 smithi045 conmon[39718]: audit 2021- 2021-07-31T01:57:45.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:44 smithi045 conmon[39718]: 07-31T01:57:44.527450 2021-07-31T01:57:45.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:44 smithi045 conmon[39718]: +0000 mon.a (mon.0) 1417 : audit 2021-07-31T01:57:45.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:44 smithi045 conmon[39718]: [DBG] from='client.? 172.21.15.45:0/2539186441' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T01:57:45.693 INFO:tasks.rados.rados.0.smithi045.stdout:3650: done (6 left) 2021-07-31T01:57:45.694 INFO:tasks.rados.rados.0.smithi045.stdout:3651: read oid 11 snap -1 2021-07-31T01:57:45.694 INFO:tasks.rados.rados.0.smithi045.stdout:3651: expect deleted 2021-07-31T01:57:45.694 INFO:tasks.rados.rados.0.smithi045.stdout:3652: snap_create 2021-07-31T01:57:45.694 INFO:tasks.rados.rados.0.smithi045.stdout:3645: finishing copy_from to smithi04569269-2 2021-07-31T01:57:45.695 INFO:tasks.rados.rados.0.smithi045.stdout:3645: got expected ENOENT (src dne) 2021-07-31T01:57:45.695 INFO:tasks.rados.rados.0.smithi045.stdout:3645: finishing copy_from racing read to smithi04569269-2 2021-07-31T01:57:45.695 INFO:tasks.rados.rados.0.smithi045.stdout:3646: expect (ObjNum 1066 snap 344 seq_num 1066) 2021-07-31T01:57:45.732 INFO:tasks.rados.rados.0.smithi045.stdout:3647: finishing write tid 1 to smithi04569269-18 2021-07-31T01:57:45.732 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 31 v 2623 (ObjNum 1006 snap 322 seq_num 1006) dirty exists 2021-07-31T01:57:45.733 INFO:tasks.rados.rados.0.smithi045.stdout:3647: finishing write tid 2 to smithi04569269-18 2021-07-31T01:57:45.733 INFO:tasks.rados.rados.0.smithi045.stdout:3647: finishing write tid 3 to smithi04569269-18 2021-07-31T01:57:45.733 INFO:tasks.rados.rados.0.smithi045.stdout:3647: finishing write tid 4 to smithi04569269-18 2021-07-31T01:57:45.733 INFO:tasks.rados.rados.0.smithi045.stdout:3647: finishing write tid 5 to smithi04569269-18 2021-07-31T01:57:45.733 INFO:tasks.rados.rados.0.smithi045.stdout:3647: finishing write tid 6 to smithi04569269-18 2021-07-31T01:57:45.734 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 18 v 1891 (ObjNum 1120 snap 353 seq_num 1120) dirty exists 2021-07-31T01:57:45.734 INFO:tasks.rados.rados.0.smithi045.stdout:3647: left oid 18 (ObjNum 1120 snap 353 seq_num 1120) 2021-07-31T01:57:45.734 INFO:tasks.rados.rados.0.smithi045.stdout:3649: expect (ObjNum 1085 snap 347 seq_num 1085) 2021-07-31T01:57:45.794 INFO:tasks.rados.rados.0.smithi045.stdout:3648: expect (ObjNum 1108 snap 351 seq_num 1108) 2021-07-31T01:57:46.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:45 smithi154 conmon[34553]: audit 2021-07-31T01:57:44.183910+0000 mgr.x (mgr.24542) 452 : audit [DBG] from='client.15078 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:57:46.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:45 smithi154 conmon[34553]: 2021-07-31T01:57:46.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:45 smithi154 conmon[34553]: cluster 2021-07-31T01:57:44.684029+0000 mon.a (mon.0) 1418 : cluster [DBG] osdmap e748: 8 total, 8 up, 8 in 2021-07-31T01:57:46.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:45 smithi045 conmon[33655]: audit 2021-07-31T01: 2021-07-31T01:57:46.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:45 smithi045 conmon[33655]: 57:44.183910+0000 mgr.x (mgr.24542) 452 : audit [DBG] from='client.15078 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:57:46.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:45 smithi045 conmon[33655]: cluster 2021-07-31T01:57 2021-07-31T01:57:46.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:45 smithi045 conmon[33655]: :44.684029+0000 mon.a (mon.0) 1418 : cluster [DBG] osdmap e748: 8 total, 8 up, 8 in 2021-07-31T01:57:46.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:45 smithi045 conmon[39718]: audit 2021-07-31T01:57:44.183910+0000 mgr.x ( 2021-07-31T01:57:46.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:45 smithi045 conmon[39718]: mgr.24542) 452 : audit [DBG] from='client.15078 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:57:46.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:45 smithi045 conmon[39718]: cluster 2021-07-31T01:57:44.684029+0000 mon.a 2021-07-31T01:57:46.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:45 smithi045 conmon[39718]: (mon.0) 1418 : cluster [DBG] osdmap e748: 8 total, 8 up, 8 in 2021-07-31T01:57:46.672 INFO:tasks.rados.rados.0.smithi045.stdout:3644: done (7 left) 2021-07-31T01:57:46.673 INFO:tasks.rados.rados.0.smithi045.stdout:3645: done (6 left) 2021-07-31T01:57:46.673 INFO:tasks.rados.rados.0.smithi045.stdout:3646: done (5 left) 2021-07-31T01:57:46.673 INFO:tasks.rados.rados.0.smithi045.stdout:3647: done (4 left) 2021-07-31T01:57:46.673 INFO:tasks.rados.rados.0.smithi045.stdout:3648: done (3 left) 2021-07-31T01:57:46.674 INFO:tasks.rados.rados.0.smithi045.stdout:3649: done (2 left) 2021-07-31T01:57:46.674 INFO:tasks.rados.rados.0.smithi045.stdout:3651: done (1 left) 2021-07-31T01:57:46.674 INFO:tasks.rados.rados.0.smithi045.stdout:3652: done (0 left) 2021-07-31T01:57:46.674 INFO:tasks.rados.rados.0.smithi045.stdout:3653: rollback oid 46 current snap is 354 2021-07-31T01:57:46.674 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 46 to 295 2021-07-31T01:57:46.675 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 48 current snap is 354 2021-07-31T01:57:46.675 INFO:tasks.rados.rados.0.smithi045.stdout:3654: seq_num 1121 ranges {557056=778240,1335296=442368,1777664=638976,2416640=122880} 2021-07-31T01:57:46.679 INFO:tasks.rados.rados.0.smithi045.stdout:3654: writing smithi04569269-48 from 557056 to 1335296 tid 1 2021-07-31T01:57:46.682 INFO:tasks.rados.rados.0.smithi045.stdout:3654: writing smithi04569269-48 from 1335296 to 1777664 tid 2 2021-07-31T01:57:46.686 INFO:tasks.rados.rados.0.smithi045.stdout:3654: writing smithi04569269-48 from 1777664 to 2416640 tid 3 2021-07-31T01:57:46.687 INFO:tasks.rados.rados.0.smithi045.stdout:3654: writing smithi04569269-48 from 2416640 to 2539520 tid 4 2021-07-31T01:57:46.687 INFO:tasks.rados.rados.0.smithi045.stdout:3655: read oid 33 snap -1 2021-07-31T01:57:46.688 INFO:tasks.rados.rados.0.smithi045.stdout:3655: expect (ObjNum 1093 snap 349 seq_num 1093) 2021-07-31T01:57:46.688 INFO:tasks.rados.rados.0.smithi045.stdout:3656: setattr oid 14 current snap is 354 2021-07-31T01:57:46.688 INFO:tasks.rados.rados.0.smithi045.stdout:3653: finishing rollback tid 0 to smithi04569269-46 2021-07-31T01:57:46.688 INFO:tasks.rados.rados.0.smithi045.stdout:3653: finishing rollback tid 1 to smithi04569269-46 2021-07-31T01:57:46.688 INFO:tasks.rados.rados.0.smithi045.stdout:3653: finishing rollback tid 2 to smithi04569269-46 2021-07-31T01:57:46.689 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 46 v 1856 (ObjNum 958 snap 293 seq_num 958) dirty exists 2021-07-31T01:57:46.689 INFO:tasks.rados.rados.0.smithi045.stdout:3654: finishing write tid 1 to smithi04569269-48 2021-07-31T01:57:46.689 INFO:tasks.rados.rados.0.smithi045.stdout:3654: finishing write tid 2 to smithi04569269-48 2021-07-31T01:57:46.690 INFO:tasks.rados.rados.0.smithi045.stdout:3653: done (3 left) 2021-07-31T01:57:46.690 INFO:tasks.rados.rados.0.smithi045.stdout:3657: read oid 3 snap -1 2021-07-31T01:57:46.691 INFO:tasks.rados.rados.0.smithi045.stdout:3657: expect (ObjNum 881 snap 272 seq_num 881) 2021-07-31T01:57:46.691 INFO:tasks.rados.rados.0.smithi045.stdout:3658: read oid 4 snap -1 2021-07-31T01:57:46.691 INFO:tasks.rados.rados.0.smithi045.stdout:3658: expect (ObjNum 1085 snap 347 seq_num 1085) 2021-07-31T01:57:46.691 INFO:tasks.rados.rados.0.smithi045.stdout:3659: copy_from oid 45 from oid 16 current snap is 354 2021-07-31T01:57:46.691 INFO:tasks.rados.rados.0.smithi045.stdout:3660: snap_create 2021-07-31T01:57:46.692 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 14 v 2076 (ObjNum 1953066355 snap 875587944 seq_num 842610229) dirty exists 2021-07-31T01:57:46.699 INFO:tasks.rados.rados.0.smithi045.stdout:3655: expect (ObjNum 1093 snap 349 seq_num 1093) 2021-07-31T01:57:46.748 INFO:tasks.rados.rados.0.smithi045.stdout:3654: finishing write tid 3 to smithi04569269-48 2021-07-31T01:57:46.748 INFO:tasks.rados.rados.0.smithi045.stdout:3654: finishing write tid 4 to smithi04569269-48 2021-07-31T01:57:46.748 INFO:tasks.rados.rados.0.smithi045.stdout:3654: finishing write tid 5 to smithi04569269-48 2021-07-31T01:57:46.749 INFO:tasks.rados.rados.0.smithi045.stdout:3654: finishing write tid 6 to smithi04569269-48 2021-07-31T01:57:46.749 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 48 v 1892 (ObjNum 1121 snap 354 seq_num 1121) dirty exists 2021-07-31T01:57:46.749 INFO:tasks.rados.rados.0.smithi045.stdout:3654: left oid 48 (ObjNum 1121 snap 354 seq_num 1121) 2021-07-31T01:57:46.749 INFO:tasks.rados.rados.0.smithi045.stdout:3658: expect (ObjNum 1085 snap 347 seq_num 1085) 2021-07-31T01:57:46.809 INFO:tasks.rados.rados.0.smithi045.stdout:3659: finishing copy_from to smithi04569269-45 2021-07-31T01:57:46.809 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 45 v 1853 (ObjNum 944 snap 286 seq_num 944) dirty exists 2021-07-31T01:57:46.809 INFO:tasks.rados.rados.0.smithi045.stdout:3659: finishing copy_from racing read to smithi04569269-45 2021-07-31T01:57:46.810 INFO:tasks.rados.rados.0.smithi045.stdout:3657: expect (ObjNum 881 snap 272 seq_num 881) 2021-07-31T01:57:47.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:46 smithi154 conmon[34553]: cluster 2021-07-31T01:57:45.690709+0000 mon.a (mon.0) 1419 : cluster 2021-07-31T01:57:47.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:46 smithi154 conmon[34553]: [DBG] osdmap e749: 8 total, 8 up, 8 in 2021-07-31T01:57:47.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:46 smithi154 conmon[34553]: cluster 2021-07-31T01:57:45.942900+0000 mgr.x (mgr.24542) 453 : cluster [DBG] pgmap v912: 25 pgs: 25 active+clean; 676 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 255 B/s rd, 472 KiB/s wr, 3 op/s 2021-07-31T01:57:47.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:46 smithi154 conmon[34553]: cluster 2021-07-31T01:57:46.671426+0000 2021-07-31T01:57:47.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:46 smithi154 conmon[34553]: mon.a (mon.0) 1420 : cluster [DBG] osdmap e750: 8 total, 8 up, 8 in 2021-07-31T01:57:47.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:46 smithi045 conmon[39718]: cluster 2021-07-31T01:57:45.690709+0000 mon.a 2021-07-31T01:57:47.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:46 smithi045 conmon[39718]: (mon.0) 1419 : cluster [DBG] osdmap e749: 8 total, 8 up, 8 in 2021-07-31T01:57:47.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:46 smithi045 conmon[39718]: cluster 2021-07-31T01:57:45.942900+0000 mgr.x (mgr.24542) 453 : cluster [DBG] pgmap v912: 25 pgs: 25 active+clean; 676 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 255 B/s rd, 472 KiB/s wr, 3 op/s 2021-07-31T01:57:47.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:46 smithi045 conmon[39718]: cluster 2021-07-31T01:57:46.671426+0000 mon.a (mon.0) 1420 : cluster [DBG] osdmap e750: 8 total, 8 up, 8 in 2021-07-31T01:57:47.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:46 smithi045 conmon[33655]: cluster 2021-07-31T01:57:45.690709+0000 mon.a ( 2021-07-31T01:57:47.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:46 smithi045 conmon[33655]: mon.0) 1419 : cluster [DBG] osdmap e749: 8 total, 8 up, 8 in 2021-07-31T01:57:47.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:46 smithi045 conmon[33655]: cluster 2021-07-31T01:57:45.942900+0000 mgr.x (mgr.24542) 453 : cluster [DBG] pgmap v912: 25 pgs: 25 active+clean; 676 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 255 B/s rd, 472 KiB/s wr, 3 op/s 2021-07-31T01:57:47.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:46 smithi045 conmon[33655]: cluster 2021-07-31T01:57: 2021-07-31T01:57:47.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:46 smithi045 conmon[33655]: 46.671426+0000 mon.a (mon.0) 1420 : cluster [DBG] osdmap e750: 8 total, 8 up, 8 in 2021-07-31T01:57:47.831 INFO:tasks.rados.rados.0.smithi045.stdout:3654: done (6 left) 2021-07-31T01:57:47.832 INFO:tasks.rados.rados.0.smithi045.stdout:3655: done (5 left) 2021-07-31T01:57:47.832 INFO:tasks.rados.rados.0.smithi045.stdout:3656: done (4 left) 2021-07-31T01:57:47.832 INFO:tasks.rados.rados.0.smithi045.stdout:3657: done (3 left) 2021-07-31T01:57:47.832 INFO:tasks.rados.rados.0.smithi045.stdout:3658: done (2 left) 2021-07-31T01:57:47.833 INFO:tasks.rados.rados.0.smithi045.stdout:3659: done (1 left) 2021-07-31T01:57:47.833 INFO:tasks.rados.rados.0.smithi045.stdout:3660: done (0 left) 2021-07-31T01:57:47.833 INFO:tasks.rados.rados.0.smithi045.stdout:3661: rollback oid 12 current snap is 355 2021-07-31T01:57:47.833 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 12 to 295 2021-07-31T01:57:47.833 INFO:tasks.rados.rados.0.smithi045.stdout:3662: read oid 10 snap -1 2021-07-31T01:57:47.834 INFO:tasks.rados.rados.0.smithi045.stdout:3662: expect deleted 2021-07-31T01:57:47.834 INFO:tasks.rados.rados.0.smithi045.stdout:3663: read oid 1 snap 310 2021-07-31T01:57:47.834 INFO:tasks.rados.rados.0.smithi045.stdout:3663: expect (ObjNum 713 snap 222 seq_num 713) 2021-07-31T01:57:47.834 INFO:tasks.rados.rados.0.smithi045.stdout:3664: snap_remove snap 293 2021-07-31T01:57:48.702 INFO:tasks.rados.rados.0.smithi045.stdout:3664: done (3 left) 2021-07-31T01:57:48.702 INFO:tasks.rados.rados.0.smithi045.stdout:3665: copy_from oid 22 from oid 46 current snap is 355 2021-07-31T01:57:48.702 INFO:tasks.rados.rados.0.smithi045.stdout:3661: finishing rollback tid 0 to smithi04569269-12 2021-07-31T01:57:48.703 INFO:tasks.rados.rados.0.smithi045.stdout:3661: finishing rollback tid 1 to smithi04569269-12 2021-07-31T01:57:48.703 INFO:tasks.rados.rados.0.smithi045.stdout:3661: finishing rollback tid 2 to smithi04569269-12 2021-07-31T01:57:48.703 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 12 v 2361 (ObjNum 967 snap 295 seq_num 967) dirty exists 2021-07-31T01:57:48.703 INFO:tasks.rados.rados.0.smithi045.stdout:3663: expect (ObjNum 713 snap 222 seq_num 713) 2021-07-31T01:57:48.845 INFO:tasks.rados.rados.0.smithi045.stdout:3661: done (3 left) 2021-07-31T01:57:48.846 INFO:tasks.rados.rados.0.smithi045.stdout:3662: done (2 left) 2021-07-31T01:57:48.846 INFO:tasks.rados.rados.0.smithi045.stdout:3663: done (1 left) 2021-07-31T01:57:48.846 INFO:tasks.rados.rados.0.smithi045.stdout:3666: copy_from oid 41 from oid 3 current snap is 355 2021-07-31T01:57:48.847 INFO:tasks.rados.rados.0.smithi045.stdout:3665: finishing copy_from to smithi04569269-22 2021-07-31T01:57:48.847 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 22 v 1856 (ObjNum 958 snap 293 seq_num 958) dirty exists 2021-07-31T01:57:48.848 INFO:tasks.rados.rados.0.smithi045.stdout:3665: finishing copy_from racing read to smithi04569269-22 2021-07-31T01:57:48.848 INFO:tasks.rados.rados.0.smithi045.stdout:3665: done (1 left) 2021-07-31T01:57:48.848 INFO:tasks.rados.rados.0.smithi045.stdout:3667: delete oid 13 current snap is 355 2021-07-31T01:57:48.849 INFO:tasks.rados.rados.0.smithi045.stdout:3667: done (1 left) 2021-07-31T01:57:48.849 INFO:tasks.rados.rados.0.smithi045.stdout:3668: snap_create 2021-07-31T01:57:48.882 INFO:tasks.rados.rados.0.smithi045.stdout:3666: finishing copy_from to smithi04569269-41 2021-07-31T01:57:48.882 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 41 v 2163 (ObjNum 881 snap 272 seq_num 881) dirty exists 2021-07-31T01:57:48.883 INFO:tasks.rados.rados.0.smithi045.stdout:3666: finishing copy_from racing read to smithi04569269-41 2021-07-31T01:57:49.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:48 smithi154 conmon[34553]: cluster 2021-07-31T01:57:49.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:48 smithi154 conmon[34553]: 2021-07-31T01:57:49.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:48 smithi154 conmon[34553]: 2021-07-31T01:57:47.693525+0000 mon.a (mon.0) 1421 2021-07-31T01:57:49.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:48 smithi154 conmon[34553]: : cluster [DBG] osdmap e751: 8 total, 8 up, 8 in 2021-07-31T01:57:49.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:48 smithi154 conmon[34553]: cluster 2021-07-31T01:57:49.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:48 smithi154 conmon[34553]: 2021-07-31T01:57:47.943384+0000 mgr.x (mgr.24542) 454 : cluster [DBG] pgmap v915: 25 pgs: 25 active+clean; 676 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2021-07-31T01:57:49.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:48 smithi045 conmon[33655]: cluster 2021-07-31T01:57:47.693525+0000 mon.a (mon 2021-07-31T01:57:49.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:48 smithi045 conmon[33655]: .0) 1421 : cluster [DBG] osdmap e751: 8 total, 8 up, 8 in 2021-07-31T01:57:49.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:48 smithi045 conmon[33655]: cluster 2021-07-31T01:57:47.943384+0000 mgr.x (mgr.24542) 454 : cluster [DBG] pgmap v915: 25 pgs: 25 active+clean; 676 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2021-07-31T01:57:49.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:48 smithi045 conmon[39718]: cluster 2021-07-31T01:57:47.693525+0000 2021-07-31T01:57:49.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:48 smithi045 conmon[39718]: mon.a (mon.0) 1421 : cluster [DBG] osdmap e751: 8 total, 8 up, 8 in 2021-07-31T01:57:49.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:48 smithi045 conmon[39718]: cluster 2021 2021-07-31T01:57:49.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:48 smithi045 conmon[39718]: -07-31T01:57:47.943384+0000 mgr.x (mgr.24542) 454 : cluster 2021-07-31T01:57:49.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:48 smithi045 conmon[39718]: [DBG] pgmap v915: 25 pgs: 25 active+clean; 676 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2021-07-31T01:57:49.701 INFO:tasks.rados.rados.0.smithi045.stdout:3666: done (1 left) 2021-07-31T01:57:49.702 INFO:tasks.rados.rados.0.smithi045.stdout:3668: done (0 left) 2021-07-31T01:57:49.702 INFO:tasks.rados.rados.0.smithi045.stdout:3669: snap_remove snap 300 2021-07-31T01:57:50.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:49 smithi154 conmon[34553]: cluster 2021-07-31T01:57:48.700780+0000 mon.a (mon.0) 1422 : cluster [DBG] osdmap e752: 8 total, 8 up, 8 in 2021-07-31T01:57:50.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:49 smithi045 conmon[39718]: cluster 2021-07-31T01:57:48.700780+0000 mon.a (mon.0) 1422 : 2021-07-31T01:57:50.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:49 smithi045 conmon[39718]: cluster [DBG] osdmap e752: 8 total, 8 up, 8 in 2021-07-31T01:57:50.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:49 smithi045 conmon[33655]: cluster 2021-07-31T01:57:48.700780+0000 mon.a (mon.0) 1422 : cluster [DBG] osdmap e752: 8 total, 8 up, 8 in 2021-07-31T01:57:50.713 INFO:tasks.rados.rados.0.smithi045.stdout:3669: done (0 left) 2021-07-31T01:57:50.714 INFO:tasks.rados.rados.0.smithi045.stdout:3670: delete oid 36 current snap is 356 2021-07-31T01:57:50.727 INFO:tasks.rados.rados.0.smithi045.stdout:3670: done (0 left) 2021-07-31T01:57:50.728 INFO:tasks.rados.rados.0.smithi045.stdout:3671: rmattr oid 37 current snap is 356 2021-07-31T01:57:50.728 INFO:tasks.rados.rados.0.smithi045.stdout:3671: done (0 left) 2021-07-31T01:57:50.728 INFO:tasks.rados.rados.0.smithi045.stdout:3672: read oid 40 snap -1 2021-07-31T01:57:50.728 INFO:tasks.rados.rados.0.smithi045.stdout:3672: expect (ObjNum 1067 snap 344 seq_num 1067) 2021-07-31T01:57:50.729 INFO:tasks.rados.rados.0.smithi045.stdout:3673: snap_create 2021-07-31T01:57:50.739 INFO:tasks.rados.rados.0.smithi045.stdout:3672: expect (ObjNum 1067 snap 344 seq_num 1067) 2021-07-31T01:57:51.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:50 smithi154 conmon[34553]: cluster 2021-07-31T01:57:49.701196+0000 mon.a (mon.0) 1423 2021-07-31T01:57:51.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:50 smithi154 conmon[34553]: : cluster [DBG] osdmap e753: 8 total, 8 up, 8 in 2021-07-31T01:57:51.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:50 smithi154 conmon[34553]: cluster 2021-07-31T01:57:49.943826+0000 mgr.x (mgr.24542) 455 : cluster [DBG] pgmap v918: 25 pgs: 25 active+clean; 628 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 17 MiB/s rd, 1.7 MiB/s wr, 22 op/s 2021-07-31T01:57:51.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:50 smithi045 conmon[39718]: cluster 2021-07-31T01:57:49 2021-07-31T01:57:51.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:50 smithi045 conmon[39718]: .701196+0000 mon.a (mon.0) 1423 : cluster [DBG] osdmap e753: 8 total, 8 up, 8 in 2021-07-31T01:57:51.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:50 smithi045 conmon[39718]: cluster 2021-07-31T01:57:51.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:50 smithi045 conmon[39718]: 2021-07-31T01:57:49.943826+0000 mgr.x (mgr.24542) 455 2021-07-31T01:57:51.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:50 smithi045 conmon[39718]: : cluster [DBG] pgmap v918: 25 pgs: 25 active+clean; 628 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 17 MiB/s rd, 1.7 MiB/s wr, 22 op/s 2021-07-31T01:57:51.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:50 smithi045 conmon[33655]: cluster 2021-07-31T01:57:49.701196+0000 mon.a (mon.0) 2021-07-31T01:57:51.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:50 smithi045 conmon[33655]: 1423 : cluster [DBG] osdmap e753: 8 total, 8 up, 8 in 2021-07-31T01:57:51.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:50 smithi045 conmon[33655]: cluster 2021-07-31T01:57:49.943826+0000 mgr.x (mgr.24542) 455 : cluster [DBG] pgmap v918: 25 pgs: 25 active+clean; 628 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 17 MiB/s rd, 1.7 MiB/s wr, 22 op/s 2021-07-31T01:57:51.673 INFO:tasks.rados.rados.0.smithi045.stdout:3672: done (1 left) 2021-07-31T01:57:51.674 INFO:tasks.rados.rados.0.smithi045.stdout:3673: done (0 left) 2021-07-31T01:57:51.674 INFO:tasks.rados.rados.0.smithi045.stdout:3674: read oid 39 snap 341 2021-07-31T01:57:51.674 INFO:tasks.rados.rados.0.smithi045.stdout:3674: expect (ObjNum -1104481840 snap 21873 seq_num 2751505664) 2021-07-31T01:57:51.674 INFO:tasks.rados.rados.0.smithi045.stdout:3675: rollback oid 44 current snap is 357 2021-07-31T01:57:51.675 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 44 to 356 2021-07-31T01:57:51.675 INFO:tasks.rados.rados.0.smithi045.stdout:3676: snap_remove snap 353 2021-07-31T01:57:52.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:51 smithi154 conmon[34553]: cluster 2021-07-31T01:57:50.712435+0000 mon.a (mon.0) 1424 : cluster [DBG] osdmap e754: 8 total, 8 up, 8 in 2021-07-31T01:57:52.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:51 smithi154 conmon[34553]: cluster 2021-07-31T01:57:51.672630+0000 mon.a (mon.0) 1425 : cluster [DBG] osdmap e755: 8 total, 8 up, 8 in 2021-07-31T01:57:52.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:51 smithi045 conmon[33655]: cluster 2021-07-31T01:57:50.712435+0000 mon.a (mon.0) 1424 : cluster [DBG] 2021-07-31T01:57:52.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:51 smithi045 conmon[33655]: osdmap e754: 8 total, 8 up, 8 in 2021-07-31T01:57:52.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:51 smithi045 conmon[33655]: cluster 2021-07-31T01:57:51.672630+0000 mon.a (mon.0) 1425 : cluster [DBG] osdmap e755: 8 total, 8 up, 8 in 2021-07-31T01:57:52.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:51 smithi045 conmon[39718]: cluster 2021-07-31T01:57:50.712435+0000 mon.a ( 2021-07-31T01:57:52.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:51 smithi045 conmon[39718]: mon.0) 1424 : cluster [DBG] osdmap e754: 8 total, 8 up, 8 in 2021-07-31T01:57:52.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:51 smithi045 conmon[39718]: cluster 2021-07-31T01:57:51.672630+0000 mon.a (mon.0) 2021-07-31T01:57:52.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:51 smithi045 conmon[39718]: 1425 : cluster [DBG] osdmap e755: 8 total, 8 up, 8 in 2021-07-31T01:57:52.680 INFO:tasks.rados.rados.0.smithi045.stdout:3676: done (2 left) 2021-07-31T01:57:52.680 INFO:tasks.rados.rados.0.smithi045.stdout:3677: read oid 10 snap -1 2021-07-31T01:57:52.681 INFO:tasks.rados.rados.0.smithi045.stdout:3677: expect deleted 2021-07-31T01:57:52.681 INFO:tasks.rados.rados.0.smithi045.stdout:3678: delete oid 19 current snap is 357 2021-07-31T01:57:52.681 INFO:tasks.rados.rados.0.smithi045.stdout:3674: expect (ObjNum -1104481840 snap 21873 seq_num 2751505664) 2021-07-31T01:57:52.684 INFO:tasks.rados.rados.0.smithi045.stdout:3675: finishing rollback tid 0 to smithi04569269-44 2021-07-31T01:57:52.684 INFO:tasks.rados.rados.0.smithi045.stdout:3675: finishing rollback tid 1 to smithi04569269-44 2021-07-31T01:57:52.684 INFO:tasks.rados.rados.0.smithi045.stdout:3675: finishing rollback tid 2 to smithi04569269-44 2021-07-31T01:57:52.685 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 44 v 2079 (ObjNum 1088 snap 348 seq_num 1088) dirty exists 2021-07-31T01:57:52.686 INFO:tasks.rados.rados.0.smithi045.stdout:3674: done (3 left) 2021-07-31T01:57:52.686 INFO:tasks.rados.rados.0.smithi045.stdout:3675: done (2 left) 2021-07-31T01:57:52.686 INFO:tasks.rados.rados.0.smithi045.stdout:3677: done (1 left) 2021-07-31T01:57:52.686 INFO:tasks.rados.rados.0.smithi045.stdout:3678: done (0 left) 2021-07-31T01:57:52.687 INFO:tasks.rados.rados.0.smithi045.stdout:3679: rollback oid 11 current snap is 357 2021-07-31T01:57:52.687 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 11 to 351 2021-07-31T01:57:52.687 INFO:tasks.rados.rados.0.smithi045.stdout:3680: delete oid 24 current snap is 357 2021-07-31T01:57:52.689 INFO:tasks.rados.rados.0.smithi045.stdout:3679: finishing rollback tid 1 to smithi04569269-11 2021-07-31T01:57:52.689 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 11 v 2046 (ObjNum 1936482661 snap 1533891683 seq_num 1517316208) dirty dne 2021-07-31T01:57:52.691 INFO:tasks.rados.rados.0.smithi045.stdout:3679: done (1 left) 2021-07-31T01:57:52.691 INFO:tasks.rados.rados.0.smithi045.stdout:3680: done (0 left) 2021-07-31T01:57:52.691 INFO:tasks.rados.rados.0.smithi045.stdout:3681: snap_remove snap 329 2021-07-31T01:57:53.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:52 smithi154 conmon[34553]: cluster 2021-07-31T01:57:51.944370+0000 mgr.x (mgr.24542) 456 : 2021-07-31T01:57:53.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:52 smithi154 conmon[34553]: cluster [DBG] pgmap v921: 25 pgs: 25 active+clean; 628 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 17 MiB/s rd, 1.7 MiB/s wr, 22 op/s 2021-07-31T01:57:53.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:52 smithi154 conmon[34553]: cluster 2021-07-31T01:57:52.678814+0000 mon.a (mon.0) 1426 : cluster [DBG] osdmap e756: 8 total, 8 up, 8 in 2021-07-31T01:57:53.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:52 smithi045 conmon[33655]: cluster 2021-07-31T01:57:51.944370+0000 mgr.x (mgr.24542 2021-07-31T01:57:53.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:52 smithi045 conmon[33655]: ) 456 : cluster [DBG] pgmap v921: 25 pgs: 25 active+clean; 628 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 17 MiB/s rd, 1.7 MiB/s wr, 22 op/s 2021-07-31T01:57:53.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:52 smithi045 conmon[33655]: cluster 2021-07-31T01:57:52.678814+0000 mon.a (mon.0) 1426 : cluster [DBG] osdmap e756: 8 total, 8 up, 8 in 2021-07-31T01:57:53.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:52 smithi045 conmon[39718]: cluster 2021-07-31T01:57 2021-07-31T01:57:53.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:52 smithi045 conmon[39718]: :51.944370+0000 mgr.x (mgr.24542) 456 : cluster [DBG] pgmap v921: 25 pgs: 25 active+clean; 628 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 17 MiB/s rd, 1.7 MiB/s wr, 22 op/s 2021-07-31T01:57:53.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:52 smithi045 conmon[39718]: cluster 2021-07-31T01:57:53.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:52 smithi045 conmon[39718]: 2021-07-31T01:57:52.678814+0000 2021-07-31T01:57:53.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:52 smithi045 conmon[39718]: mon.a (mon.0) 1426 : cluster [DBG] osdmap e756: 8 total, 8 up, 8 in 2021-07-31T01:57:53.681 INFO:tasks.rados.rados.0.smithi045.stdout:3681: done (0 left) 2021-07-31T01:57:53.682 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 46 current snap is 357 2021-07-31T01:57:53.682 INFO:tasks.rados.rados.0.smithi045.stdout:3682: seq_num 1126 ranges {1654784=466944,2121728=720896,2842624=778240,3620864=106496} 2021-07-31T01:57:53.694 INFO:tasks.rados.rados.0.smithi045.stdout:3682: writing smithi04569269-46 from 1654784 to 2121728 tid 1 2021-07-31T01:57:53.699 INFO:tasks.rados.rados.0.smithi045.stdout:3682: writing smithi04569269-46 from 2121728 to 2842624 tid 2 2021-07-31T01:57:53.704 INFO:tasks.rados.rados.0.smithi045.stdout:3682: writing smithi04569269-46 from 2842624 to 3620864 tid 3 2021-07-31T01:57:53.705 INFO:tasks.rados.rados.0.smithi045.stdout:3682: writing smithi04569269-46 from 3620864 to 3727360 tid 4 2021-07-31T01:57:53.705 INFO:tasks.rados.rados.0.smithi045.stdout:3683: rmattr oid 27 current snap is 357 2021-07-31T01:57:53.705 INFO:tasks.rados.rados.0.smithi045.stdout:3684: delete oid 11 current snap is 357 2021-07-31T01:57:53.708 INFO:tasks.rados.rados.0.smithi045.stdout:3682: finishing write tid 1 to smithi04569269-46 2021-07-31T01:57:53.709 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 27 v 2080 (ObjNum 288 snap 0 seq_num 161) dirty exists 2021-07-31T01:57:53.710 INFO:tasks.rados.rados.0.smithi045.stdout:3682: finishing write tid 2 to smithi04569269-46 2021-07-31T01:57:53.711 INFO:tasks.rados.rados.0.smithi045.stdout:3683: done (2 left) 2021-07-31T01:57:53.711 INFO:tasks.rados.rados.0.smithi045.stdout:3684: done (1 left) 2021-07-31T01:57:53.711 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 10 current snap is 357 2021-07-31T01:57:53.712 INFO:tasks.rados.rados.0.smithi045.stdout:3685: seq_num 1128 ranges {0=499712,499712=483328,983040=786432,1769472=335872} 2021-07-31T01:57:53.714 INFO:tasks.rados.rados.0.smithi045.stdout:3685: writing smithi04569269-10 from 0 to 499712 tid 1 2021-07-31T01:57:53.716 INFO:tasks.rados.rados.0.smithi045.stdout:3685: writing smithi04569269-10 from 499712 to 983040 tid 2 2021-07-31T01:57:53.721 INFO:tasks.rados.rados.0.smithi045.stdout:3685: writing smithi04569269-10 from 983040 to 1769472 tid 3 2021-07-31T01:57:53.724 INFO:tasks.rados.rados.0.smithi045.stdout:3685: writing smithi04569269-10 from 1769472 to 2105344 tid 4 2021-07-31T01:57:53.724 INFO:tasks.rados.rados.0.smithi045.stdout:3686: read oid 23 snap 338 2021-07-31T01:57:53.724 INFO:tasks.rados.rados.0.smithi045.stdout:3686: expect (ObjNum 1009 snap 323 seq_num 1009) 2021-07-31T01:57:53.725 INFO:tasks.rados.rados.0.smithi045.stdout:3687: read oid 22 snap -1 2021-07-31T01:57:53.725 INFO:tasks.rados.rados.0.smithi045.stdout:3687: expect (ObjNum 958 snap 293 seq_num 958) 2021-07-31T01:57:53.725 INFO:tasks.rados.rados.0.smithi045.stdout:3688: read oid 49 snap -1 2021-07-31T01:57:53.726 INFO:tasks.rados.rados.0.smithi045.stdout:3688: expect (ObjNum 1090 snap 349 seq_num 1090) 2021-07-31T01:57:53.726 INFO:tasks.rados.rados.0.smithi045.stdout:3689: read oid 7 snap -1 2021-07-31T01:57:53.726 INFO:tasks.rados.rados.0.smithi045.stdout:3689: expect deleted 2021-07-31T01:57:53.727 INFO:tasks.rados.rados.0.smithi045.stdout:3690: setattr oid 16 current snap is 357 2021-07-31T01:57:53.727 INFO:tasks.rados.rados.0.smithi045.stdout:3682: finishing write tid 3 to smithi04569269-46 2021-07-31T01:57:53.727 INFO:tasks.rados.rados.0.smithi045.stdout:3682: finishing write tid 4 to smithi04569269-46 2021-07-31T01:57:53.727 INFO:tasks.rados.rados.0.smithi045.stdout:3682: finishing write tid 5 to smithi04569269-46 2021-07-31T01:57:53.728 INFO:tasks.rados.rados.0.smithi045.stdout:3682: finishing write tid 6 to smithi04569269-46 2021-07-31T01:57:53.728 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 46 v 1861 (ObjNum 1126 snap 357 seq_num 1126) dirty exists 2021-07-31T01:57:53.729 INFO:tasks.rados.rados.0.smithi045.stdout:3682: left oid 46 (ObjNum 1126 snap 357 seq_num 1126) 2021-07-31T01:57:53.729 INFO:tasks.rados.rados.0.smithi045.stdout:3685: finishing write tid 1 to smithi04569269-10 2021-07-31T01:57:53.729 INFO:tasks.rados.rados.0.smithi045.stdout:3685: finishing write tid 2 to smithi04569269-10 2021-07-31T01:57:53.730 INFO:tasks.rados.rados.0.smithi045.stdout:3682: done (6 left) 2021-07-31T01:57:53.731 INFO:tasks.rados.rados.0.smithi045.stdout:3689: done (5 left) 2021-07-31T01:57:53.731 INFO:tasks.rados.rados.0.smithi045.stdout:3691: read oid 35 snap -1 2021-07-31T01:57:53.731 INFO:tasks.rados.rados.0.smithi045.stdout:3691: expect (ObjNum 1063 snap 344 seq_num 1063) 2021-07-31T01:57:53.731 INFO:tasks.rados.rados.0.smithi045.stdout:3692: snap_create 2021-07-31T01:57:53.732 INFO:tasks.rados.rados.0.smithi045.stdout:3685: finishing write tid 3 to smithi04569269-10 2021-07-31T01:57:53.732 INFO:tasks.rados.rados.0.smithi045.stdout:3685: finishing write tid 4 to smithi04569269-10 2021-07-31T01:57:53.733 INFO:tasks.rados.rados.0.smithi045.stdout:3685: finishing write tid 5 to smithi04569269-10 2021-07-31T01:57:53.733 INFO:tasks.rados.rados.0.smithi045.stdout:3686: expect (ObjNum 1009 snap 323 seq_num 1009) 2021-07-31T01:57:53.777 INFO:tasks.rados.rados.0.smithi045.stdout:3685: finishing write tid 6 to smithi04569269-10 2021-07-31T01:57:53.777 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 10 v 2365 (ObjNum 1128 snap 357 seq_num 1128) dirty exists 2021-07-31T01:57:53.777 INFO:tasks.rados.rados.0.smithi045.stdout:3685: left oid 10 (ObjNum 1128 snap 357 seq_num 1128) 2021-07-31T01:57:53.778 INFO:tasks.rados.rados.0.smithi045.stdout:3687: expect (ObjNum 958 snap 293 seq_num 958) 2021-07-31T01:57:53.818 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 16 v 1859 (ObjNum 944 snap 286 seq_num 944) dirty exists 2021-07-31T01:57:53.819 INFO:tasks.rados.rados.0.smithi045.stdout:3688: expect (ObjNum 1090 snap 349 seq_num 1090) 2021-07-31T01:57:53.903 INFO:tasks.rados.rados.0.smithi045.stdout:3691: expect (ObjNum 1063 snap 344 seq_num 1063) 2021-07-31T01:57:54.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:57:53 smithi045 conmon[61210]: level=error ts=2021-07-31T01:57:53.527Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:57:54.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:57:53 smithi045 conmon[61210]: level=error ts=2021-07-31T01:57:53.527Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:57:54.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:57:53 smithi045 conmon[61210]: level=error ts=2021-07-31T01:57:53.527Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T01:57:54.687 INFO:tasks.rados.rados.0.smithi045.stdout:3685: done (6 left) 2021-07-31T01:57:54.688 INFO:tasks.rados.rados.0.smithi045.stdout:3686: done (5 left) 2021-07-31T01:57:54.688 INFO:tasks.rados.rados.0.smithi045.stdout:3687: done (4 left) 2021-07-31T01:57:54.688 INFO:tasks.rados.rados.0.smithi045.stdout:3688: done (3 left) 2021-07-31T01:57:54.688 INFO:tasks.rados.rados.0.smithi045.stdout:3690: done (2 left) 2021-07-31T01:57:54.689 INFO:tasks.rados.rados.0.smithi045.stdout:3691: done (1 left) 2021-07-31T01:57:54.689 INFO:tasks.rados.rados.0.smithi045.stdout:3692: done (0 left) 2021-07-31T01:57:54.689 INFO:tasks.rados.rados.0.smithi045.stdout:3693: delete oid 12 current snap is 358 2021-07-31T01:57:54.691 INFO:tasks.rados.rados.0.smithi045.stdout:3693: done (0 left) 2021-07-31T01:57:54.691 INFO:tasks.rados.rados.0.smithi045.stdout:3694: snap_create 2021-07-31T01:57:55.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:54 smithi154 conmon[34553]: cluster 2021-07-31T01:57:53.680598+0000 mon.a (mon.0 2021-07-31T01:57:55.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:54 smithi154 conmon[34553]: ) 1427 : cluster [DBG] osdmap e757: 8 total, 8 up, 8 in 2021-07-31T01:57:55.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:54 smithi154 conmon[34553]: cluster 2021-07-31T01:57:53.944999+0000 mgr.x (mgr.24542) 457 : cluster [DBG] pgmap v924: 25 pgs: 25 active+clean; 640 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.0 MiB/s rd, 3.3 MiB/s wr, 17 op/s 2021-07-31T01:57:55.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:54 smithi045 conmon[39718]: cluster 2021-07-31T01:57:53.680598+0000 mon.a (mon.0) 2021-07-31T01:57:55.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:54 smithi045 conmon[39718]: 1427 : cluster [DBG] osdmap e757: 8 total, 8 up, 8 in 2021-07-31T01:57:55.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:54 smithi045 conmon[39718]: cluster 2021-07-31T01:57:53.944999+0000 mgr.x (mgr.24542) 457 : cluster [DBG] pgmap v924: 25 pgs: 25 active+clean; 640 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.0 MiB/s rd, 3.3 MiB/s wr, 17 op/s 2021-07-31T01:57:55.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:54 smithi045 conmon[33655]: cluster 2021-07-31T01:57:53.680598+0000 mon.a (mon.0) 1427 : 2021-07-31T01:57:55.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:54 smithi045 conmon[33655]: cluster [DBG] osdmap e757: 8 total, 8 up, 8 in 2021-07-31T01:57:55.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:54 smithi045 conmon[33655]: cluster 2021-07-31T01:57:53.944999+0000 mgr.x (mgr.24542) 457 : cluster [DBG] pgmap v924: 25 pgs: 25 active+clean; 640 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.0 MiB/s rd, 3.3 MiB/s wr, 17 op/s 2021-07-31T01:57:55.688 INFO:tasks.rados.rados.0.smithi045.stdout:3694: done (0 left) 2021-07-31T01:57:55.689 INFO:tasks.rados.rados.0.smithi045.stdout:3695: delete oid 41 current snap is 359 2021-07-31T01:57:55.691 INFO:tasks.rados.rados.0.smithi045.stdout:3695: done (0 left) 2021-07-31T01:57:55.692 INFO:tasks.rados.rados.0.smithi045.stdout:3696: read oid 22 snap 336 2021-07-31T01:57:55.692 INFO:tasks.rados.rados.0.smithi045.stdout:3696: expect deleted 2021-07-31T01:57:55.692 INFO:tasks.rados.rados.0.smithi045.stdout:3697: snap_create 2021-07-31T01:57:56.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:55 smithi154 conmon[34553]: cluster 2021-07-31T01:57:54.685959+0000 mon.a (mon 2021-07-31T01:57:56.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:55 smithi154 conmon[34553]: .0) 1428 : cluster [DBG] osdmap e758: 8 total, 8 up, 8 in 2021-07-31T01:57:56.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:55 smithi045 conmon[39718]: cluster 2021-07-31T01:57:54.685959+0000 2021-07-31T01:57:56.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:55 smithi045 conmon[39718]: mon.a (mon.0) 1428 : cluster [DBG] osdmap e758: 8 total, 8 up, 8 in 2021-07-31T01:57:56.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:55 smithi045 conmon[33655]: cluster 2021-07-31T 2021-07-31T01:57:56.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:55 smithi045 conmon[33655]: 01:57:54.685959+0000 mon.a (mon.0) 1428 : cluster [DBG] osdmap e758: 8 total, 8 up, 8 in 2021-07-31T01:57:56.675 INFO:tasks.rados.rados.0.smithi045.stdout:3696: done (1 left) 2021-07-31T01:57:56.675 INFO:tasks.rados.rados.0.smithi045.stdout:3697: done (0 left) 2021-07-31T01:57:56.676 INFO:tasks.rados.rados.0.smithi045.stdout:3698: read oid 42 snap -1 2021-07-31T01:57:56.676 INFO:tasks.rados.rados.0.smithi045.stdout:3698: expect (ObjNum 1036 snap 338 seq_num 1036) 2021-07-31T01:57:56.676 INFO:tasks.rados.rados.0.smithi045.stdout:3699: rollback oid 40 current snap is 360 2021-07-31T01:57:56.676 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 40 to 336 2021-07-31T01:57:56.677 INFO:tasks.rados.rados.0.smithi045.stdout:3700: read oid 5 snap -1 2021-07-31T01:57:56.677 INFO:tasks.rados.rados.0.smithi045.stdout:3700: expect (ObjNum 996 snap 311 seq_num 996) 2021-07-31T01:57:56.677 INFO:tasks.rados.rados.0.smithi045.stdout:3701: read oid 29 snap -1 2021-07-31T01:57:56.677 INFO:tasks.rados.rados.0.smithi045.stdout:3701: expect (ObjNum 1066 snap 344 seq_num 1066) 2021-07-31T01:57:56.677 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 8 current snap is 360 2021-07-31T01:57:56.678 INFO:tasks.rados.rados.0.smithi045.stdout:3702: seq_num 1131 ranges {4046848=548864,4595712=540672,5136384=565248,5701632=425984,6127616=73728} 2021-07-31T01:57:56.701 INFO:tasks.rados.rados.0.smithi045.stdout:3702: writing smithi04569269-8 from 4046848 to 4595712 tid 1 2021-07-31T01:57:56.704 INFO:tasks.rados.rados.0.smithi045.stdout:3702: writing smithi04569269-8 from 4595712 to 5136384 tid 2 2021-07-31T01:57:56.707 INFO:tasks.rados.rados.0.smithi045.stdout:3702: writing smithi04569269-8 from 5136384 to 5701632 tid 3 2021-07-31T01:57:56.710 INFO:tasks.rados.rados.0.smithi045.stdout:3702: writing smithi04569269-8 from 5701632 to 6127616 tid 4 2021-07-31T01:57:56.711 INFO:tasks.rados.rados.0.smithi045.stdout:3702: writing smithi04569269-8 from 6127616 to 6201344 tid 5 2021-07-31T01:57:56.711 INFO:tasks.rados.rados.0.smithi045.stdout:3703: setattr oid 39 current snap is 360 2021-07-31T01:57:56.712 INFO:tasks.rados.rados.0.smithi045.stdout:3700: expect (ObjNum 996 snap 311 seq_num 996) 2021-07-31T01:57:56.771 INFO:tasks.rados.rados.0.smithi045.stdout:3699: finishing rollback tid 0 to smithi04569269-40 2021-07-31T01:57:56.772 INFO:tasks.rados.rados.0.smithi045.stdout:3699: finishing rollback tid 1 to smithi04569269-40 2021-07-31T01:57:56.772 INFO:tasks.rados.rados.0.smithi045.stdout:3701: expect (ObjNum 1066 snap 344 seq_num 1066) 2021-07-31T01:57:56.800 INFO:tasks.rados.rados.0.smithi045.stdout:3699: finishing rollback tid 2 to smithi04569269-40 2021-07-31T01:57:56.801 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 40 v 2366 (ObjNum 1 snap 32731 seq_num 3190901576) dirty exists 2021-07-31T01:57:56.801 INFO:tasks.rados.rados.0.smithi045.stdout:3698: expect (ObjNum 1036 snap 338 seq_num 1036) 2021-07-31T01:57:56.818 INFO:tasks.rados.rados.0.smithi045.stdout:3702: finishing write tid 1 to smithi04569269-8 2021-07-31T01:57:56.818 INFO:tasks.rados.rados.0.smithi045.stdout:3702: finishing write tid 2 to smithi04569269-8 2021-07-31T01:57:56.818 INFO:tasks.rados.rados.0.smithi045.stdout:3702: finishing write tid 3 to smithi04569269-8 2021-07-31T01:57:56.819 INFO:tasks.rados.rados.0.smithi045.stdout:3702: finishing write tid 4 to smithi04569269-8 2021-07-31T01:57:56.819 INFO:tasks.rados.rados.0.smithi045.stdout:3702: finishing write tid 5 to smithi04569269-8 2021-07-31T01:57:56.819 INFO:tasks.rados.rados.0.smithi045.stdout:3702: finishing write tid 6 to smithi04569269-8 2021-07-31T01:57:56.819 INFO:tasks.rados.rados.0.smithi045.stdout:3702: finishing write tid 7 to smithi04569269-8 2021-07-31T01:57:56.819 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 8 v 2700 (ObjNum 1131 snap 360 seq_num 1131) dirty exists 2021-07-31T01:57:56.820 INFO:tasks.rados.rados.0.smithi045.stdout:3702: left oid 8 (ObjNum 1131 snap 360 seq_num 1131) 2021-07-31T01:57:56.820 INFO:tasks.rados.rados.0.smithi045.stdout:3698: done (5 left) 2021-07-31T01:57:56.820 INFO:tasks.rados.rados.0.smithi045.stdout:3699: done (4 left) 2021-07-31T01:57:56.820 INFO:tasks.rados.rados.0.smithi045.stdout:3700: done (3 left) 2021-07-31T01:57:56.821 INFO:tasks.rados.rados.0.smithi045.stdout:3701: done (2 left) 2021-07-31T01:57:56.821 INFO:tasks.rados.rados.0.smithi045.stdout:3702: done (1 left) 2021-07-31T01:57:56.821 INFO:tasks.rados.rados.0.smithi045.stdout:3704: read oid 42 snap -1 2021-07-31T01:57:56.821 INFO:tasks.rados.rados.0.smithi045.stdout:3704: expect (ObjNum 1036 snap 338 seq_num 1036) 2021-07-31T01:57:56.821 INFO:tasks.rados.rados.0.smithi045.stdout:3705: read oid 36 snap -1 2021-07-31T01:57:56.822 INFO:tasks.rados.rados.0.smithi045.stdout:3705: expect deleted 2021-07-31T01:57:56.822 INFO:tasks.rados.rados.0.smithi045.stdout:3706: rmattr oid 24 current snap is 360 2021-07-31T01:57:56.822 INFO:tasks.rados.rados.0.smithi045.stdout:3706: done (3 left) 2021-07-31T01:57:56.822 INFO:tasks.rados.rados.0.smithi045.stdout:3707: read oid 23 snap -1 2021-07-31T01:57:56.822 INFO:tasks.rados.rados.0.smithi045.stdout:3707: expect (ObjNum 1049 snap 342 seq_num 1049) 2021-07-31T01:57:56.823 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 48 current snap is 360 2021-07-31T01:57:56.823 INFO:tasks.rados.rados.0.smithi045.stdout:3708: seq_num 1132 ranges {2539520=565248,3104768=425984,3530752=770048,4300800=409600} 2021-07-31T01:57:56.834 INFO:tasks.rados.rados.0.smithi045.stdout:3708: writing smithi04569269-48 from 2539520 to 3104768 tid 1 2021-07-31T01:57:56.837 INFO:tasks.rados.rados.0.smithi045.stdout:3708: writing smithi04569269-48 from 3104768 to 3530752 tid 2 2021-07-31T01:57:56.841 INFO:tasks.rados.rados.0.smithi045.stdout:3708: writing smithi04569269-48 from 3530752 to 4300800 tid 3 2021-07-31T01:57:56.844 INFO:tasks.rados.rados.0.smithi045.stdout:3708: writing smithi04569269-48 from 4300800 to 4710400 tid 4 2021-07-31T01:57:56.844 INFO:tasks.rados.rados.0.smithi045.stdout:3709: snap_remove snap 326 2021-07-31T01:57:57.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:56 smithi154 conmon[34553]: cluster 2021-07-31T01:57:55.687223+0000 mon.a (mon.0) 1429 2021-07-31T01:57:57.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:56 smithi154 conmon[34553]: : cluster [DBG] osdmap e759: 8 total, 8 up, 8 in 2021-07-31T01:57:57.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:56 smithi154 conmon[34553]: cluster 2021-07-31T01:57:55.945520+0000 mgr.x (mgr.24542) 458 : cluster [DBG] pgmap v927: 25 pgs: 25 active+clean; 640 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.0 MiB/s rd, 3.3 MiB/s wr, 17 op/s 2021-07-31T01:57:57.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:56 smithi154 conmon[34553]: cluster 2021-07-31T01:57:56.673950+0000 2021-07-31T01:57:57.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:56 smithi154 conmon[34553]: mon.a (mon.0) 1430 : cluster [DBG] osdmap e760: 8 total, 8 up, 8 in 2021-07-31T01:57:57.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:56 smithi045 conmon[33655]: cluster 2021-07-31T01:57:55.687223+0000 mon.a ( 2021-07-31T01:57:57.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:56 smithi045 conmon[33655]: mon.0) 1429 : cluster [DBG] osdmap e759: 8 total, 8 up, 8 in 2021-07-31T01:57:57.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:56 smithi045 conmon[33655]: cluster 2021-07-31T01:57:55.945520+0000 mgr.x (mgr.24542) 458 : cluster [DBG] pgmap v927: 25 pgs: 25 active+clean; 640 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.0 MiB/s rd, 3.3 MiB/s wr, 17 op/s 2021-07-31T01:57:57.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:56 smithi045 conmon[33655]: cluster 2021-07 2021-07-31T01:57:57.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:56 smithi045 conmon[33655]: -31T01:57:56. 2021-07-31T01:57:57.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:56 smithi045 conmon[33655]: 673950+0000 mon.a (mon.0) 1430 : cluster [DBG] osdmap e760: 8 total, 8 up, 8 in 2021-07-31T01:57:57.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:56 smithi045 conmon[39718]: cluster 2021-07-31T01:57:55.687223+0000 mon.a ( 2021-07-31T01:57:57.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:56 smithi045 conmon[39718]: mon.0) 1429 : cluster [DBG] osdmap e759: 8 total, 8 up, 8 in 2021-07-31T01:57:57.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:56 smithi045 conmon[39718]: cluster 2021- 2021-07-31T01:57:57.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:56 smithi045 conmon[39718]: 07-31T01:57:55.945520+0000 mgr.x (mgr.24542) 458 : cluster [DBG] pgmap v927: 25 pgs: 25 active+clean; 640 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.0 MiB/s rd, 3.3 MiB/s wr, 17 op/s 2021-07-31T01:57:57.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:56 smithi045 conmon[39718]: cluster 2021-07- 2021-07-31T01:57:57.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:56 smithi045 conmon[39718]: 31T01:57:56.673950+0000 mon.a (mon.0 2021-07-31T01:57:57.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:56 smithi045 conmon[39718]: ) 1430 : cluster [DBG] osdmap e760: 8 total, 8 up, 8 in 2021-07-31T01:57:57.704 INFO:tasks.rados.rados.0.smithi045.stdout:3704: expect (ObjNum 1036 snap 338 seq_num 1036) 2021-07-31T01:57:57.730 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 39 v 2050 (ObjNum 1085 snap 347 seq_num 1085) dirty exists 2021-07-31T01:57:57.730 INFO:tasks.rados.rados.0.smithi045.stdout:3707: expect (ObjNum 1049 snap 342 seq_num 1049) 2021-07-31T01:57:57.813 INFO:tasks.rados.rados.0.smithi045.stdout:3708: finishing write tid 1 to smithi04569269-48 2021-07-31T01:57:57.813 INFO:tasks.rados.rados.0.smithi045.stdout:3708: finishing write tid 2 to smithi04569269-48 2021-07-31T01:57:57.813 INFO:tasks.rados.rados.0.smithi045.stdout:3708: finishing write tid 3 to smithi04569269-48 2021-07-31T01:57:57.813 INFO:tasks.rados.rados.0.smithi045.stdout:3708: finishing write tid 4 to smithi04569269-48 2021-07-31T01:57:57.814 INFO:tasks.rados.rados.0.smithi045.stdout:3708: finishing write tid 5 to smithi04569269-48 2021-07-31T01:57:57.814 INFO:tasks.rados.rados.0.smithi045.stdout:3708: finishing write tid 6 to smithi04569269-48 2021-07-31T01:57:57.814 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 48 v 1897 (ObjNum 1132 snap 360 seq_num 1132) dirty exists 2021-07-31T01:57:57.814 INFO:tasks.rados.rados.0.smithi045.stdout:3708: left oid 48 (ObjNum 1132 snap 360 seq_num 1132) 2021-07-31T01:57:57.814 INFO:tasks.rados.rados.0.smithi045.stdout:3703: done (5 left) 2021-07-31T01:57:57.815 INFO:tasks.rados.rados.0.smithi045.stdout:3704: done (4 left) 2021-07-31T01:57:57.815 INFO:tasks.rados.rados.0.smithi045.stdout:3705: done (3 left) 2021-07-31T01:57:57.815 INFO:tasks.rados.rados.0.smithi045.stdout:3707: done (2 left) 2021-07-31T01:57:57.815 INFO:tasks.rados.rados.0.smithi045.stdout:3708: done (1 left) 2021-07-31T01:57:57.816 INFO:tasks.rados.rados.0.smithi045.stdout:3709: done (0 left) 2021-07-31T01:57:57.816 INFO:tasks.rados.rados.0.smithi045.stdout:3710: rollback oid 48 current snap is 360 2021-07-31T01:57:57.816 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 48 to 356 2021-07-31T01:57:57.816 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 2 current snap is 360 2021-07-31T01:57:57.817 INFO:tasks.rados.rados.0.smithi045.stdout:3711: seq_num 1133 ranges {5529600=581632,6111232=753664,6864896=573440,7438336=278528} 2021-07-31T01:57:57.846 INFO:tasks.rados.rados.0.smithi045.stdout:3711: writing smithi04569269-2 from 5529600 to 6111232 tid 1 2021-07-31T01:57:57.849 INFO:tasks.rados.rados.0.smithi045.stdout:3711: writing smithi04569269-2 from 6111232 to 6864896 tid 2 2021-07-31T01:57:57.852 INFO:tasks.rados.rados.0.smithi045.stdout:3711: writing smithi04569269-2 from 6864896 to 7438336 tid 3 2021-07-31T01:57:57.855 INFO:tasks.rados.rados.0.smithi045.stdout:3711: writing smithi04569269-2 from 7438336 to 7716864 tid 4 2021-07-31T01:57:57.855 INFO:tasks.rados.rados.0.smithi045.stdout:3712: rollback oid 44 current snap is 360 2021-07-31T01:57:57.855 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 44 to 331 2021-07-31T01:57:57.856 INFO:tasks.rados.rados.0.smithi045.stdout:3710: finishing rollback tid 0 to smithi04569269-48 2021-07-31T01:57:57.856 INFO:tasks.rados.rados.0.smithi045.stdout:3710: finishing rollback tid 1 to smithi04569269-48 2021-07-31T01:57:57.856 INFO:tasks.rados.rados.0.smithi045.stdout:3710: finishing rollback tid 2 to smithi04569269-48 2021-07-31T01:57:57.857 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 48 v 1900 (ObjNum 1121 snap 354 seq_num 1121) dirty exists 2021-07-31T01:57:57.857 INFO:tasks.rados.rados.0.smithi045.stdout:3711: finishing write tid 1 to smithi04569269-2 2021-07-31T01:57:57.857 INFO:tasks.rados.rados.0.smithi045.stdout:3711: finishing write tid 2 to smithi04569269-2 2021-07-31T01:57:57.857 INFO:tasks.rados.rados.0.smithi045.stdout:3710: done (2 left) 2021-07-31T01:57:57.858 INFO:tasks.rados.rados.0.smithi045.stdout:3713: snap_remove snap 359 2021-07-31T01:57:58.709 INFO:tasks.rados.rados.0.smithi045.stdout:3713: done (2 left) 2021-07-31T01:57:58.709 INFO:tasks.rados.rados.0.smithi045.stdout:3714: delete oid 36 current snap is 360 2021-07-31T01:57:58.709 INFO:tasks.rados.rados.0.smithi045.stdout:3711: finishing write tid 3 to smithi04569269-2 2021-07-31T01:57:58.709 INFO:tasks.rados.rados.0.smithi045.stdout:3711: finishing write tid 4 to smithi04569269-2 2021-07-31T01:57:58.710 INFO:tasks.rados.rados.0.smithi045.stdout:3711: finishing write tid 5 to smithi04569269-2 2021-07-31T01:57:58.710 INFO:tasks.rados.rados.0.smithi045.stdout:3711: finishing write tid 6 to smithi04569269-2 2021-07-31T01:57:58.710 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 2 v 2367 (ObjNum 1133 snap 360 seq_num 1133) dirty exists 2021-07-31T01:57:58.710 INFO:tasks.rados.rados.0.smithi045.stdout:3711: left oid 2 (ObjNum 1133 snap 360 seq_num 1133) 2021-07-31T01:57:58.711 INFO:tasks.rados.rados.0.smithi045.stdout:3712: finishing rollback tid 0 to smithi04569269-44 2021-07-31T01:57:58.711 INFO:tasks.rados.rados.0.smithi045.stdout:3712: finishing rollback tid 1 to smithi04569269-44 2021-07-31T01:57:58.711 INFO:tasks.rados.rados.0.smithi045.stdout:3712: finishing rollback tid 2 to smithi04569269-44 2021-07-31T01:57:58.711 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 44 v 2083 (ObjNum 1027 snap 330 seq_num 1027) dirty exists 2021-07-31T01:57:58.716 INFO:tasks.rados.rados.0.smithi045.stdout:3711: done (2 left) 2021-07-31T01:57:58.717 INFO:tasks.rados.rados.0.smithi045.stdout:3712: done (1 left) 2021-07-31T01:57:58.717 INFO:tasks.rados.rados.0.smithi045.stdout:3714: done (0 left) 2021-07-31T01:57:58.717 INFO:tasks.rados.rados.0.smithi045.stdout:3715: read oid 6 snap 357 2021-07-31T01:57:58.717 INFO:tasks.rados.rados.0.smithi045.stdout:3715: expect (ObjNum 822 snap 258 seq_num 822) 2021-07-31T01:57:58.718 INFO:tasks.rados.rados.0.smithi045.stdout:3716: rollback oid 19 current snap is 360 2021-07-31T01:57:58.718 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 19 to 304 2021-07-31T01:57:58.718 INFO:tasks.rados.rados.0.smithi045.stdout:3717: delete oid 40 current snap is 360 2021-07-31T01:57:58.719 INFO:tasks.rados.rados.0.smithi045.stdout:3716: finishing rollback tid 1 to smithi04569269-19 2021-07-31T01:57:58.719 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 19 v 1858 (ObjNum -1126011944 snap 21873 seq_num 1654784) dirty dne 2021-07-31T01:57:58.738 INFO:tasks.rados.rados.0.smithi045.stdout:3715: expect (ObjNum 822 snap 258 seq_num 822) 2021-07-31T01:57:58.854 INFO:tasks.rados.rados.0.smithi045.stdout:3715: done (2 left) 2021-07-31T01:57:58.855 INFO:tasks.rados.rados.0.smithi045.stdout:3716: done (1 left) 2021-07-31T01:57:58.855 INFO:tasks.rados.rados.0.smithi045.stdout:3717: done (0 left) 2021-07-31T01:57:58.855 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 5 current snap is 360 2021-07-31T01:57:58.856 INFO:tasks.rados.rados.0.smithi045.stdout:3718: seq_num 1136 ranges {2293760=630784,2924544=409600,3334144=753664,4087808=417792,4505600=24576} 2021-07-31T01:57:58.869 INFO:tasks.rados.rados.0.smithi045.stdout:3718: writing smithi04569269-5 from 2293760 to 2924544 tid 1 2021-07-31T01:57:58.871 INFO:tasks.rados.rados.0.smithi045.stdout:3718: writing smithi04569269-5 from 2924544 to 3334144 tid 2 2021-07-31T01:57:58.876 INFO:tasks.rados.rados.0.smithi045.stdout:3718: writing smithi04569269-5 from 3334144 to 4087808 tid 3 2021-07-31T01:57:58.879 INFO:tasks.rados.rados.0.smithi045.stdout:3718: writing smithi04569269-5 from 4087808 to 4505600 tid 4 2021-07-31T01:57:58.879 INFO:tasks.rados.rados.0.smithi045.stdout:3718: writing smithi04569269-5 from 4505600 to 4530176 tid 5 2021-07-31T01:57:58.879 INFO:tasks.rados.rados.0.smithi045.stdout:3719: snap_create 2021-07-31T01:57:58.880 INFO:tasks.rados.rados.0.smithi045.stdout:3718: finishing write tid 1 to smithi04569269-5 2021-07-31T01:57:58.880 INFO:tasks.rados.rados.0.smithi045.stdout:3718: finishing write tid 2 to smithi04569269-5 2021-07-31T01:57:58.883 INFO:tasks.rados.rados.0.smithi045.stdout:3718: finishing write tid 3 to smithi04569269-5 2021-07-31T01:57:58.883 INFO:tasks.rados.rados.0.smithi045.stdout:3718: finishing write tid 4 to smithi04569269-5 2021-07-31T01:57:58.883 INFO:tasks.rados.rados.0.smithi045.stdout:3718: finishing write tid 5 to smithi04569269-5 2021-07-31T01:57:58.885 INFO:tasks.rados.rados.0.smithi045.stdout:3718: finishing write tid 6 to smithi04569269-5 2021-07-31T01:57:58.886 INFO:tasks.rados.rados.0.smithi045.stdout:3718: finishing write tid 7 to smithi04569269-5 2021-07-31T01:57:58.886 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 5 v 2056 (ObjNum 1136 snap 360 seq_num 1136) dirty exists 2021-07-31T01:57:58.887 INFO:tasks.rados.rados.0.smithi045.stdout:3718: left oid 5 (ObjNum 1136 snap 360 seq_num 1136) 2021-07-31T01:57:59.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:58 smithi154 conmon[34553]: cluster 2021-07-31T01:57:57.702923+0000 mon.a (mon.0) 1431 : cluster [DBG] osdmap e761: 8 total, 8 up, 8 in 2021-07-31T01:57:59.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:58 smithi154 conmon[34553]: cluster 2021-07-31T01:57:57.945965+0000 mgr.x (mgr.24542) 459 : cluster [DBG] pgmap v930: 25 pgs: 25 active+clean; 640 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2021-07-31T01:57:59.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:58 smithi045 conmon[39718]: cluster 2021-07-31T01:57:57.702923+0000 mon.a (mon.0) 1431 : cluster 2021-07-31T01:57:59.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:58 smithi045 conmon[39718]: [DBG] osdmap e761: 8 total, 8 up, 8 in 2021-07-31T01:57:59.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:58 smithi045 conmon[39718]: cluster 2021-07-31T01:57:57.945965+0000 mgr.x (mgr.24542) 459 : cluster [DBG] 2021-07-31T01:57:59.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:58 smithi045 conmon[39718]: pgmap v930: 25 pgs: 25 active+clean; 640 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2021-07-31T01:57:59.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:58 smithi045 conmon[33655]: cluster 2021-07-31T01:57:57.702923+0000 mon.a (mon.0) 1431 : cluster 2021-07-31T01:57:59.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:58 smithi045 conmon[33655]: [DBG] osdmap e761: 8 total, 8 up, 8 in 2021-07-31T01:57:59.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:58 smithi045 conmon[33655]: cluster 2021-07-31T01:57:57.945965+0000 mgr.x (mgr.24542) 459 : cluster [DBG] pgmap v930: 25 pgs: 25 active+clean; 640 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2021-07-31T01:57:59.710 INFO:tasks.rados.rados.0.smithi045.stdout:3718: done (1 left) 2021-07-31T01:57:59.710 INFO:tasks.rados.rados.0.smithi045.stdout:3719: done (0 left) 2021-07-31T01:57:59.710 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 42 current snap is 361 2021-07-31T01:57:59.711 INFO:tasks.rados.rados.0.smithi045.stdout:3720: seq_num 1137 ranges {557056=647168,1204224=696320,1900544=557056,2457600=352256} 2021-07-31T01:57:59.716 INFO:tasks.rados.rados.0.smithi045.stdout:3720: writing smithi04569269-42 from 557056 to 1204224 tid 1 2021-07-31T01:57:59.720 INFO:tasks.rados.rados.0.smithi045.stdout:3720: writing smithi04569269-42 from 1204224 to 1900544 tid 2 2021-07-31T01:57:59.724 INFO:tasks.rados.rados.0.smithi045.stdout:3720: writing smithi04569269-42 from 1900544 to 2457600 tid 3 2021-07-31T01:57:59.726 INFO:tasks.rados.rados.0.smithi045.stdout:3720: writing smithi04569269-42 from 2457600 to 2809856 tid 4 2021-07-31T01:57:59.726 INFO:tasks.rados.rados.0.smithi045.stdout:3721: read oid 41 snap -1 2021-07-31T01:57:59.726 INFO:tasks.rados.rados.0.smithi045.stdout:3721: expect deleted 2021-07-31T01:57:59.726 INFO:tasks.rados.rados.0.smithi045.stdout:3722: snap_create 2021-07-31T01:57:59.727 INFO:tasks.rados.rados.0.smithi045.stdout:3720: finishing write tid 1 to smithi04569269-42 2021-07-31T01:57:59.727 INFO:tasks.rados.rados.0.smithi045.stdout:3720: finishing write tid 2 to smithi04569269-42 2021-07-31T01:57:59.731 INFO:tasks.rados.rados.0.smithi045.stdout:3720: finishing write tid 3 to smithi04569269-42 2021-07-31T01:57:59.731 INFO:tasks.rados.rados.0.smithi045.stdout:3720: finishing write tid 4 to smithi04569269-42 2021-07-31T01:57:59.731 INFO:tasks.rados.rados.0.smithi045.stdout:3720: finishing write tid 5 to smithi04569269-42 2021-07-31T01:57:59.732 INFO:tasks.rados.rados.0.smithi045.stdout:3720: finishing write tid 6 to smithi04569269-42 2021-07-31T01:57:59.733 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 42 v 2370 (ObjNum 1137 snap 361 seq_num 1137) dirty exists 2021-07-31T01:57:59.733 INFO:tasks.rados.rados.0.smithi045.stdout:3720: left oid 42 (ObjNum 1137 snap 361 seq_num 1137) 2021-07-31T01:58:00.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:57:59 smithi154 conmon[34553]: cluster 2021-07-31T01:57:58.707724+0000 mon.a (mon.0) 1432 : cluster [DBG] osdmap e762: 8 total, 8 up, 8 in 2021-07-31T01:58:00.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:59 smithi045 conmon[39718]: cluster 2021-07-31T01:57:58.707724+0000 mon.a 2021-07-31T01:58:00.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:57:59 smithi045 conmon[39718]: (mon.0) 1432 : cluster [DBG] osdmap e762: 8 total, 8 up, 8 in 2021-07-31T01:58:00.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:59 smithi045 conmon[33655]: cluster 2021-07-31T01:57:58.707724+0000 2021-07-31T01:58:00.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:57:59 smithi045 conmon[33655]: mon.a (mon.0) 1432 : cluster [DBG] osdmap e762: 8 total, 8 up, 8 in 2021-07-31T01:58:00.721 INFO:tasks.rados.rados.0.smithi045.stdout:3720: done (2 left) 2021-07-31T01:58:00.721 INFO:tasks.rados.rados.0.smithi045.stdout:3721: done (1 left) 2021-07-31T01:58:00.722 INFO:tasks.rados.rados.0.smithi045.stdout:3722: done (0 left) 2021-07-31T01:58:00.722 INFO:tasks.rados.rados.0.smithi045.stdout:3723: copy_from oid 39 from oid 30 current snap is 362 2021-07-31T01:58:00.722 INFO:tasks.rados.rados.0.smithi045.stdout:3724: read oid 40 snap -1 2021-07-31T01:58:00.722 INFO:tasks.rados.rados.0.smithi045.stdout:3724: expect deleted 2021-07-31T01:58:00.723 INFO:tasks.rados.rados.0.smithi045.stdout:3725: read oid 43 snap -1 2021-07-31T01:58:00.723 INFO:tasks.rados.rados.0.smithi045.stdout:3725: expect (ObjNum 1113 snap 351 seq_num 1113) 2021-07-31T01:58:00.723 INFO:tasks.rados.rados.0.smithi045.stdout:3726: delete oid 25 current snap is 362 2021-07-31T01:58:00.724 INFO:tasks.rados.rados.0.smithi045.stdout:3723: finishing copy_from to smithi04569269-39 2021-07-31T01:58:00.724 INFO:tasks.rados.rados.0.smithi045.stdout:3723: got expected ENOENT (src dne) 2021-07-31T01:58:00.724 INFO:tasks.rados.rados.0.smithi045.stdout:3723: finishing copy_from racing read to smithi04569269-39 2021-07-31T01:58:00.726 INFO:tasks.rados.rados.0.smithi045.stdout:3723: done (3 left) 2021-07-31T01:58:00.726 INFO:tasks.rados.rados.0.smithi045.stdout:3726: done (2 left) 2021-07-31T01:58:00.726 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 2 current snap is 362 2021-07-31T01:58:00.727 INFO:tasks.rados.rados.0.smithi045.stdout:3727: seq_num 1139 ranges {7716864=679936,8396800=458752,8855552=524288,9379840=540672,9920512=81920} 2021-07-31T01:58:00.772 INFO:tasks.rados.rados.0.smithi045.stdout:3727: writing smithi04569269-2 from 7716864 to 8396800 tid 1 2021-07-31T01:58:00.775 INFO:tasks.rados.rados.0.smithi045.stdout:3727: writing smithi04569269-2 from 8396800 to 8855552 tid 2 2021-07-31T01:58:00.778 INFO:tasks.rados.rados.0.smithi045.stdout:3727: writing smithi04569269-2 from 8855552 to 9379840 tid 3 2021-07-31T01:58:00.782 INFO:tasks.rados.rados.0.smithi045.stdout:3727: writing smithi04569269-2 from 9379840 to 9920512 tid 4 2021-07-31T01:58:00.783 INFO:tasks.rados.rados.0.smithi045.stdout:3727: writing smithi04569269-2 from 9920512 to 10002432 tid 5 2021-07-31T01:58:00.783 INFO:tasks.rados.rados.0.smithi045.stdout:3728: setattr oid 20 current snap is 362 2021-07-31T01:58:00.783 INFO:tasks.rados.rados.0.smithi045.stdout:3725: expect (ObjNum 1113 snap 351 seq_num 1113) 2021-07-31T01:58:01.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:00 smithi154 conmon[34553]: cluster 2021-07-31T01:57:59.709457+0000 mon.a (mon.0) 1433 : cluster 2021-07-31T01:58:01.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:00 smithi154 conmon[34553]: [DBG] osdmap e763: 8 total, 8 up, 8 in 2021-07-31T01:58:01.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:00 smithi154 conmon[34553]: cluster 2021-07-31T01:57:59.946421+0000 mgr.x (mgr.24542) 460 : cluster [DBG] pgmap v933: 25 pgs: 25 active+clean; 644 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 7.0 MiB/s rd, 1.6 MiB/s wr, 17 op/s 2021-07-31T01:58:01.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:00 smithi045 conmon[39718]: cluster 2021-07-31T01:57:59.709457+0000 2021-07-31T01:58:01.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:00 smithi045 conmon[39718]: mon.a (mon.0) 1433 : cluster [DBG] osdmap e763: 8 total, 8 up, 8 in 2021-07-31T01:58:01.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:00 smithi045 conmon[39718]: cluster 2021-07-31T01: 2021-07-31T01:58:01.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:00 smithi045 conmon[39718]: 57:59.946421+0000 mgr.x (mgr.24542) 460 : cluster [DBG] pgmap v933: 25 pgs: 25 active+clean; 644 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 7.0 MiB/s rd, 1.6 MiB/s wr, 17 op/s 2021-07-31T01:58:01.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:00 smithi045 conmon[33655]: cluster 2021-07-31T01:57:59.709457+0000 mon.a (mon.0) 1433 : cluster [DBG] osdmap e763: 8 total, 8 up, 8 in 2021-07-31T01:58:01.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:00 smithi045 conmon[33655]: cluster 2021-07-31T01:57:59.946421+0000 mgr.x (mgr.24542) 460 : cluster [DBG] pgmap v933: 25 pgs: 25 active+clean; 644 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 7.0 MiB/s rd, 1.6 MiB/s wr, 17 op/s 2021-07-31T01:58:01.088 INFO:tasks.rados.rados.0.smithi045.stdout:3727: finishing write tid 1 to smithi04569269-2 2021-07-31T01:58:01.088 INFO:tasks.rados.rados.0.smithi045.stdout:3727: finishing write tid 2 to smithi04569269-2 2021-07-31T01:58:01.089 INFO:tasks.rados.rados.0.smithi045.stdout:3727: finishing write tid 3 to smithi04569269-2 2021-07-31T01:58:01.089 INFO:tasks.rados.rados.0.smithi045.stdout:3727: finishing write tid 4 to smithi04569269-2 2021-07-31T01:58:01.089 INFO:tasks.rados.rados.0.smithi045.stdout:3727: finishing write tid 5 to smithi04569269-2 2021-07-31T01:58:01.089 INFO:tasks.rados.rados.0.smithi045.stdout:3727: finishing write tid 6 to smithi04569269-2 2021-07-31T01:58:01.090 INFO:tasks.rados.rados.0.smithi045.stdout:3727: finishing write tid 7 to smithi04569269-2 2021-07-31T01:58:01.090 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 2 v 2373 (ObjNum 1139 snap 362 seq_num 1139) dirty exists 2021-07-31T01:58:01.090 INFO:tasks.rados.rados.0.smithi045.stdout:3727: left oid 2 (ObjNum 1139 snap 362 seq_num 1139) 2021-07-31T01:58:01.090 INFO:tasks.rados.rados.0.smithi045.stdout:3724: done (3 left) 2021-07-31T01:58:01.090 INFO:tasks.rados.rados.0.smithi045.stdout:3725: done (2 left) 2021-07-31T01:58:01.091 INFO:tasks.rados.rados.0.smithi045.stdout:3727: done (1 left) 2021-07-31T01:58:01.091 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 49 current snap is 362 2021-07-31T01:58:01.091 INFO:tasks.rados.rados.0.smithi045.stdout:3729: seq_num 1140 ranges {2580480=696320,3276800=745472,4022272=729088,4751360=131072} 2021-07-31T01:58:01.105 INFO:tasks.rados.rados.0.smithi045.stdout:3729: writing smithi04569269-49 from 2580480 to 3276800 tid 1 2021-07-31T01:58:01.109 INFO:tasks.rados.rados.0.smithi045.stdout:3729: writing smithi04569269-49 from 3276800 to 4022272 tid 2 2021-07-31T01:58:01.113 INFO:tasks.rados.rados.0.smithi045.stdout:3729: writing smithi04569269-49 from 4022272 to 4751360 tid 3 2021-07-31T01:58:01.115 INFO:tasks.rados.rados.0.smithi045.stdout:3729: writing smithi04569269-49 from 4751360 to 4882432 tid 4 2021-07-31T01:58:01.115 INFO:tasks.rados.rados.0.smithi045.stdout:3730: read oid 48 snap -1 2021-07-31T01:58:01.115 INFO:tasks.rados.rados.0.smithi045.stdout:3730: expect (ObjNum 1121 snap 354 seq_num 1121) 2021-07-31T01:58:01.116 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 44 current snap is 362 2021-07-31T01:58:01.116 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 20 v 2716 (ObjNum 0 snap 0 seq_num 0) dirty exists 2021-07-31T01:58:01.116 INFO:tasks.rados.rados.0.smithi045.stdout:3731: seq_num 1141 ranges {3801088=712704,4513792=679936,5193728=540672,5734400=385024} 2021-07-31T01:58:01.139 INFO:tasks.rados.rados.0.smithi045.stdout:3731: writing smithi04569269-44 from 3801088 to 4513792 tid 1 2021-07-31T01:58:01.143 INFO:tasks.rados.rados.0.smithi045.stdout:3731: writing smithi04569269-44 from 4513792 to 5193728 tid 2 2021-07-31T01:58:01.146 INFO:tasks.rados.rados.0.smithi045.stdout:3731: writing smithi04569269-44 from 5193728 to 5734400 tid 3 2021-07-31T01:58:01.148 INFO:tasks.rados.rados.0.smithi045.stdout:3731: writing smithi04569269-44 from 5734400 to 6119424 tid 4 2021-07-31T01:58:01.149 INFO:tasks.rados.rados.0.smithi045.stdout:3728: done (3 left) 2021-07-31T01:58:01.149 INFO:tasks.rados.rados.0.smithi045.stdout:3732: rmattr oid 1 current snap is 362 2021-07-31T01:58:01.149 INFO:tasks.rados.rados.0.smithi045.stdout:3729: finishing write tid 1 to smithi04569269-49 2021-07-31T01:58:01.150 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 3 current snap is 362 2021-07-31T01:58:01.150 INFO:tasks.rados.rados.0.smithi045.stdout:3729: finishing write tid 2 to smithi04569269-49 2021-07-31T01:58:01.150 INFO:tasks.rados.rados.0.smithi045.stdout:3733: seq_num 1142 ranges {9584640=729088,10313728=565248,10878976=696320,11575296=344064} 2021-07-31T01:58:01.204 INFO:tasks.rados.rados.0.smithi045.stdout:3733: writing smithi04569269-3 from 9584640 to 10313728 tid 1 2021-07-31T01:58:01.207 INFO:tasks.rados.rados.0.smithi045.stdout:3733: writing smithi04569269-3 from 10313728 to 10878976 tid 2 2021-07-31T01:58:01.211 INFO:tasks.rados.rados.0.smithi045.stdout:3733: writing smithi04569269-3 from 10878976 to 11575296 tid 3 2021-07-31T01:58:01.214 INFO:tasks.rados.rados.0.smithi045.stdout:3733: writing smithi04569269-3 from 11575296 to 11919360 tid 4 2021-07-31T01:58:01.214 INFO:tasks.rados.rados.0.smithi045.stdout:3734: delete oid 30 current snap is 362 2021-07-31T01:58:01.214 INFO:tasks.rados.rados.0.smithi045.stdout:3729: finishing write tid 3 to smithi04569269-49 2021-07-31T01:58:01.215 INFO:tasks.rados.rados.0.smithi045.stdout:3729: finishing write tid 4 to smithi04569269-49 2021-07-31T01:58:01.215 INFO:tasks.rados.rados.0.smithi045.stdout:3729: finishing write tid 5 to smithi04569269-49 2021-07-31T01:58:01.215 INFO:tasks.rados.rados.0.smithi045.stdout:3729: finishing write tid 6 to smithi04569269-49 2021-07-31T01:58:01.216 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 49 v 2722 (ObjNum 1140 snap 362 seq_num 1140) dirty exists 2021-07-31T01:58:01.216 INFO:tasks.rados.rados.0.smithi045.stdout:3729: left oid 49 (ObjNum 1140 snap 362 seq_num 1140) 2021-07-31T01:58:01.216 INFO:tasks.rados.rados.0.smithi045.stdout:3730: expect (ObjNum 1121 snap 354 seq_num 1121) 2021-07-31T01:58:01.292 INFO:tasks.rados.rados.0.smithi045.stdout:3731: finishing write tid 1 to smithi04569269-44 2021-07-31T01:58:01.292 INFO:tasks.rados.rados.0.smithi045.stdout:3731: finishing write tid 2 to smithi04569269-44 2021-07-31T01:58:01.293 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 1 v 1901 (ObjNum 1116 snap 351 seq_num 1116) dirty exists 2021-07-31T01:58:01.293 INFO:tasks.rados.rados.0.smithi045.stdout:3729: done (5 left) 2021-07-31T01:58:01.293 INFO:tasks.rados.rados.0.smithi045.stdout:3730: done (4 left) 2021-07-31T01:58:01.293 INFO:tasks.rados.rados.0.smithi045.stdout:3732: done (3 left) 2021-07-31T01:58:01.294 INFO:tasks.rados.rados.0.smithi045.stdout:3734: done (2 left) 2021-07-31T01:58:01.294 INFO:tasks.rados.rados.0.smithi045.stdout:3735: snap_remove snap 358 2021-07-31T01:58:01.675 INFO:tasks.rados.rados.0.smithi045.stdout:3735: done (2 left) 2021-07-31T01:58:01.676 INFO:tasks.rados.rados.0.smithi045.stdout:3736: rmattr oid 28 current snap is 362 2021-07-31T01:58:01.676 INFO:tasks.rados.rados.0.smithi045.stdout:3736: done (2 left) 2021-07-31T01:58:01.676 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 15 current snap is 362 2021-07-31T01:58:01.677 INFO:tasks.rados.rados.0.smithi045.stdout:3731: finishing write tid 3 to smithi04569269-44 2021-07-31T01:58:01.677 INFO:tasks.rados.rados.0.smithi045.stdout:3737: seq_num 1144 ranges {712704=770048,1482752=737280,2220032=712704,2932736=155648} 2021-07-31T01:58:01.684 INFO:tasks.rados.rados.0.smithi045.stdout:3737: writing smithi04569269-15 from 712704 to 1482752 tid 1 2021-07-31T01:58:01.689 INFO:tasks.rados.rados.0.smithi045.stdout:3737: writing smithi04569269-15 from 1482752 to 2220032 tid 2 2021-07-31T01:58:01.693 INFO:tasks.rados.rados.0.smithi045.stdout:3737: writing smithi04569269-15 from 2220032 to 2932736 tid 3 2021-07-31T01:58:01.696 INFO:tasks.rados.rados.0.smithi045.stdout:3737: writing smithi04569269-15 from 2932736 to 3088384 tid 4 2021-07-31T01:58:01.696 INFO:tasks.rados.rados.0.smithi045.stdout:3738: rollback oid 49 current snap is 362 2021-07-31T01:58:01.697 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 49 to 361 2021-07-31T01:58:01.697 INFO:tasks.rados.rados.0.smithi045.stdout:3731: finishing write tid 4 to smithi04569269-44 2021-07-31T01:58:01.698 INFO:tasks.rados.rados.0.smithi045.stdout:3731: finishing write tid 5 to smithi04569269-44 2021-07-31T01:58:01.698 INFO:tasks.rados.rados.0.smithi045.stdout:3731: finishing write tid 6 to smithi04569269-44 2021-07-31T01:58:01.698 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 44 v 2088 (ObjNum 1141 snap 362 seq_num 1141) dirty exists 2021-07-31T01:58:01.698 INFO:tasks.rados.rados.0.smithi045.stdout:3731: left oid 44 (ObjNum 1141 snap 362 seq_num 1141) 2021-07-31T01:58:01.698 INFO:tasks.rados.rados.0.smithi045.stdout:3733: finishing write tid 1 to smithi04569269-3 2021-07-31T01:58:01.699 INFO:tasks.rados.rados.0.smithi045.stdout:3733: finishing write tid 2 to smithi04569269-3 2021-07-31T01:58:01.699 INFO:tasks.rados.rados.0.smithi045.stdout:3733: finishing write tid 3 to smithi04569269-3 2021-07-31T01:58:01.699 INFO:tasks.rados.rados.0.smithi045.stdout:3733: finishing write tid 4 to smithi04569269-3 2021-07-31T01:58:01.699 INFO:tasks.rados.rados.0.smithi045.stdout:3733: finishing write tid 5 to smithi04569269-3 2021-07-31T01:58:01.700 INFO:tasks.rados.rados.0.smithi045.stdout:3733: finishing write tid 6 to smithi04569269-3 2021-07-31T01:58:01.700 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 3 v 2728 (ObjNum 1142 snap 362 seq_num 1142) dirty exists 2021-07-31T01:58:01.700 INFO:tasks.rados.rados.0.smithi045.stdout:3733: left oid 3 (ObjNum 1142 snap 362 seq_num 1142) 2021-07-31T01:58:01.700 INFO:tasks.rados.rados.0.smithi045.stdout:3731: done (3 left) 2021-07-31T01:58:01.700 INFO:tasks.rados.rados.0.smithi045.stdout:3733: done (2 left) 2021-07-31T01:58:01.701 INFO:tasks.rados.rados.0.smithi045.stdout:3739: copy_from oid 6 from oid 35 current snap is 362 2021-07-31T01:58:01.701 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 28 current snap is 362 2021-07-31T01:58:01.701 INFO:tasks.rados.rados.0.smithi045.stdout:3740: seq_num 1145 ranges {5971968=786432,6758400=622592,7380992=516096,7897088=417792,8314880=49152} 2021-07-31T01:58:01.734 INFO:tasks.rados.rados.0.smithi045.stdout:3740: writing smithi04569269-28 from 5971968 to 6758400 tid 1 2021-07-31T01:58:01.736 INFO:tasks.rados.rados.0.smithi045.stdout:3740: writing smithi04569269-28 from 6758400 to 7380992 tid 2 2021-07-31T01:58:01.739 INFO:tasks.rados.rados.0.smithi045.stdout:3740: writing smithi04569269-28 from 7380992 to 7897088 tid 3 2021-07-31T01:58:01.742 INFO:tasks.rados.rados.0.smithi045.stdout:3740: writing smithi04569269-28 from 7897088 to 8314880 tid 4 2021-07-31T01:58:01.742 INFO:tasks.rados.rados.0.smithi045.stdout:3740: writing smithi04569269-28 from 8314880 to 8364032 tid 5 2021-07-31T01:58:01.742 INFO:tasks.rados.rados.0.smithi045.stdout:3741: delete oid 8 current snap is 362 2021-07-31T01:58:01.743 INFO:tasks.rados.rados.0.smithi045.stdout:3737: finishing write tid 1 to smithi04569269-15 2021-07-31T01:58:01.743 INFO:tasks.rados.rados.0.smithi045.stdout:3737: finishing write tid 2 to smithi04569269-15 2021-07-31T01:58:01.743 INFO:tasks.rados.rados.0.smithi045.stdout:3737: finishing write tid 3 to smithi04569269-15 2021-07-31T01:58:01.743 INFO:tasks.rados.rados.0.smithi045.stdout:3738: finishing rollback tid 0 to smithi04569269-49 2021-07-31T01:58:01.744 INFO:tasks.rados.rados.0.smithi045.stdout:3737: finishing write tid 4 to smithi04569269-15 2021-07-31T01:58:01.744 INFO:tasks.rados.rados.0.smithi045.stdout:3738: finishing rollback tid 1 to smithi04569269-49 2021-07-31T01:58:01.744 INFO:tasks.rados.rados.0.smithi045.stdout:3737: finishing write tid 5 to smithi04569269-15 2021-07-31T01:58:01.744 INFO:tasks.rados.rados.0.smithi045.stdout:3738: finishing rollback tid 2 to smithi04569269-49 2021-07-31T01:58:01.744 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 49 v 2735 (ObjNum 1090 snap 349 seq_num 1090) dirty exists 2021-07-31T01:58:01.745 INFO:tasks.rados.rados.0.smithi045.stdout:3737: finishing write tid 6 to smithi04569269-15 2021-07-31T01:58:01.745 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 15 v 1896 (ObjNum 1144 snap 362 seq_num 1144) dirty exists 2021-07-31T01:58:01.745 INFO:tasks.rados.rados.0.smithi045.stdout:3737: left oid 15 (ObjNum 1144 snap 362 seq_num 1144) 2021-07-31T01:58:01.745 INFO:tasks.rados.rados.0.smithi045.stdout:3739: finishing copy_from to smithi04569269-6 2021-07-31T01:58:01.745 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 6 v 1861 (ObjNum 1063 snap 344 seq_num 1063) dirty exists 2021-07-31T01:58:01.746 INFO:tasks.rados.rados.0.smithi045.stdout:3739: finishing copy_from racing read to smithi04569269-6 2021-07-31T01:58:01.746 INFO:tasks.rados.rados.0.smithi045.stdout:3740: finishing write tid 1 to smithi04569269-28 2021-07-31T01:58:01.746 INFO:tasks.rados.rados.0.smithi045.stdout:3740: finishing write tid 2 to smithi04569269-28 2021-07-31T01:58:01.748 INFO:tasks.rados.rados.0.smithi045.stdout:3737: done (4 left) 2021-07-31T01:58:01.748 INFO:tasks.rados.rados.0.smithi045.stdout:3738: done (3 left) 2021-07-31T01:58:01.748 INFO:tasks.rados.rados.0.smithi045.stdout:3739: done (2 left) 2021-07-31T01:58:01.748 INFO:tasks.rados.rados.0.smithi045.stdout:3741: done (1 left) 2021-07-31T01:58:01.749 INFO:tasks.rados.rados.0.smithi045.stdout:3742: copy_from oid 9 from oid 37 current snap is 362 2021-07-31T01:58:01.749 INFO:tasks.rados.rados.0.smithi045.stdout:3743: snap_create 2021-07-31T01:58:01.749 INFO:tasks.rados.rados.0.smithi045.stdout:3740: finishing write tid 3 to smithi04569269-28 2021-07-31T01:58:01.749 INFO:tasks.rados.rados.0.smithi045.stdout:3740: finishing write tid 4 to smithi04569269-28 2021-07-31T01:58:01.749 INFO:tasks.rados.rados.0.smithi045.stdout:3740: finishing write tid 5 to smithi04569269-28 2021-07-31T01:58:01.750 INFO:tasks.rados.rados.0.smithi045.stdout:3740: finishing write tid 6 to smithi04569269-28 2021-07-31T01:58:01.750 INFO:tasks.rados.rados.0.smithi045.stdout:3740: finishing write tid 7 to smithi04569269-28 2021-07-31T01:58:01.751 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 28 v 2376 (ObjNum 1145 snap 362 seq_num 1145) dirty exists 2021-07-31T01:58:01.751 INFO:tasks.rados.rados.0.smithi045.stdout:3740: left oid 28 (ObjNum 1145 snap 362 seq_num 1145) 2021-07-31T01:58:01.763 INFO:tasks.rados.rados.0.smithi045.stdout:3742: finishing copy_from to smithi04569269-9 2021-07-31T01:58:01.764 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 9 v 1858 (ObjNum 1045 snap 340 seq_num 1045) dirty exists 2021-07-31T01:58:01.764 INFO:tasks.rados.rados.0.smithi045.stdout:3742: finishing copy_from racing read to smithi04569269-9 2021-07-31T01:58:02.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:01 smithi154 conmon[34553]: cluster 2021-07-31T01:58:00.720201+0000 mon.a (mon.0) 1434 : cluster [DBG] osdmap e764: 8 total, 8 up, 8 in 2021-07-31T01:58:02.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:01 smithi154 conmon[34553]: cluster 2021-07- 2021-07-31T01:58:02.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:01 smithi154 conmon[34553]: 31T01:58:01.675089+0000 mon.a (mon.0) 1435 : cluster [DBG] osdmap e765: 8 total, 8 up, 8 in 2021-07-31T01:58:02.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:01 smithi045 conmon[39718]: cluster 2021-07-31T01:58:00.720201+0000 mon.a 2021-07-31T01:58:02.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:01 smithi045 conmon[39718]: (mon.0) 1434 : cluster [DBG] osdmap e764: 8 total, 8 up, 8 in 2021-07-31T01:58:02.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:01 smithi045 conmon[39718]: cluster 2021-07-31 2021-07-31T01:58:02.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:01 smithi045 conmon[39718]: T01:58:01.675089+0000 mon.a (mon.0) 1435 : cluster [DBG] osdmap e765: 8 total, 8 up, 8 in 2021-07-31T01:58:02.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:01 smithi045 conmon[33655]: cluster 2021-07-31T01:58:00.720201+0000 mon.a (mon.0 2021-07-31T01:58:02.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:01 smithi045 conmon[33655]: ) 1434 : cluster [DBG] osdmap e764: 8 total, 8 up, 8 in 2021-07-31T01:58:02.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:01 smithi045 conmon[33655]: cluster 2021-07-31T01:58:01.675089+0000 mon.a (mon.0) 1435 : cluster [DBG] osdmap e765: 8 total, 8 up, 8 in 2021-07-31T01:58:02.728 INFO:tasks.rados.rados.0.smithi045.stdout:3740: done (2 left) 2021-07-31T01:58:02.729 INFO:tasks.rados.rados.0.smithi045.stdout:3742: done (1 left) 2021-07-31T01:58:02.729 INFO:tasks.rados.rados.0.smithi045.stdout:3743: done (0 left) 2021-07-31T01:58:02.729 INFO:tasks.rados.rados.0.smithi045.stdout:3744: rollback oid 35 current snap is 363 2021-07-31T01:58:02.730 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 35 to 356 2021-07-31T01:58:02.730 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 45 current snap is 363 2021-07-31T01:58:02.730 INFO:tasks.rados.rados.0.smithi045.stdout:3745: seq_num 1147 ranges {1417216=16384} 2021-07-31T01:58:02.737 INFO:tasks.rados.rados.0.smithi045.stdout:3745: writing smithi04569269-45 from 1417216 to 1433600 tid 1 2021-07-31T01:58:02.737 INFO:tasks.rados.rados.0.smithi045.stdout:3746: snap_remove snap 310 2021-07-31T01:58:03.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:02 smithi154 conmon[34553]: cluster 2021-07-31T01:58:01.947036+0000 mgr.x (mgr.24542 2021-07-31T01:58:03.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:02 smithi154 conmon[34553]: ) 461 : cluster [DBG] pgmap v936: 25 pgs: 25 active+clean; 644 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 7.0 MiB/s rd, 1.6 MiB/s wr, 17 op/s 2021-07-31T01:58:03.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:02 smithi045 conmon[39718]: cluster 2021-07-31T01:58:01.947036+0000 mgr.x (mgr. 2021-07-31T01:58:03.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:02 smithi045 conmon[39718]: 24542) 461 : cluster [DBG] pgmap v936: 25 pgs: 25 active+clean; 644 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 7.0 MiB/s rd, 1.6 MiB/s wr, 17 op/s 2021-07-31T01:58:03.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:02 smithi045 conmon[33655]: cluster 2021-07-31T01:58:01.947036+0000 mgr.x (mgr.24542) 461 2021-07-31T01:58:03.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:02 smithi045 conmon[33655]: : cluster [DBG] pgmap v936: 25 pgs: 25 active+clean; 644 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 7.0 MiB/s rd, 1.6 MiB/s wr, 17 op/s 2021-07-31T01:58:03.736 INFO:tasks.rados.rados.0.smithi045.stdout:3746: done (2 left) 2021-07-31T01:58:03.737 INFO:tasks.rados.rados.0.smithi045.stdout:3747: snap_create 2021-07-31T01:58:03.737 INFO:tasks.rados.rados.0.smithi045.stdout:3744: finishing rollback tid 0 to smithi04569269-35 2021-07-31T01:58:03.737 INFO:tasks.rados.rados.0.smithi045.stdout:3744: finishing rollback tid 1 to smithi04569269-35 2021-07-31T01:58:03.738 INFO:tasks.rados.rados.0.smithi045.stdout:3744: finishing rollback tid 2 to smithi04569269-35 2021-07-31T01:58:03.738 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 35 v 2743 (ObjNum 1063 snap 344 seq_num 1063) dirty exists 2021-07-31T01:58:03.738 INFO:tasks.rados.rados.0.smithi045.stdout:3745: finishing write tid 1 to smithi04569269-45 2021-07-31T01:58:03.738 INFO:tasks.rados.rados.0.smithi045.stdout:3745: finishing write tid 2 to smithi04569269-45 2021-07-31T01:58:03.739 INFO:tasks.rados.rados.0.smithi045.stdout:3745: finishing write tid 3 to smithi04569269-45 2021-07-31T01:58:03.739 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 45 v 2378 (ObjNum 1147 snap 363 seq_num 1147) dirty exists 2021-07-31T01:58:03.739 INFO:tasks.rados.rados.0.smithi045.stdout:3745: left oid 45 (ObjNum 1147 snap 363 seq_num 1147) 2021-07-31T01:58:04.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:03 smithi154 conmon[34553]: cluster 2021-07-31T01:58:02.727461+0000 mon.a (mon.0) 1436 : cluster [DBG] osdmap e766: 8 total, 8 up, 8 in 2021-07-31T01:58:04.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:03 smithi045 conmon[39718]: cluster 2021-07-31T01:58:02.727461+0000 mon.a (mon.0) 1436 : cluster [DBG] osdmap e766: 8 total, 8 up, 8 in 2021-07-31T01:58:04.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:58:03 smithi045 conmon[61210]: level=error ts=2021-07-31T01:58:03.527Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:58:04.027 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:58:03 smithi045 conmon[61210]: level=error ts=2021-07-31T01:58:03.527Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:58:04.027 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:58:03 smithi045 conmon[61210]: level=error ts=2021-07-31T01:58:03.527Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T01:58:04.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:03 smithi045 conmon[33655]: cluster 2021-07-31T01:58:02.727461+0000 mon.a (mon.0) 1436 : cluster [DBG] osdmap e766: 8 total, 8 up, 8 in 2021-07-31T01:58:04.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:03 smithi045 conmon[33655]: 2021-07-31T01:58:04.740 INFO:tasks.rados.rados.0.smithi045.stdout:3744: done (2 left) 2021-07-31T01:58:04.741 INFO:tasks.rados.rados.0.smithi045.stdout:3745: done (1 left) 2021-07-31T01:58:04.741 INFO:tasks.rados.rados.0.smithi045.stdout:3747: done (0 left) 2021-07-31T01:58:04.741 INFO:tasks.rados.rados.0.smithi045.stdout:3748: snap_remove snap 336 2021-07-31T01:58:05.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:04 smithi154 conmon[34553]: cluster 2021-07-31T01:58:03.734102+0000 mon.a (mon.0) 1437 : cluster 2021-07-31T01:58:05.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:04 smithi154 conmon[34553]: [DBG] osdmap e767: 8 total, 8 up, 8 in 2021-07-31T01:58:05.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:04 smithi154 conmon[34553]: cluster 2021-07-31T01:58:03.947497+0000 mgr.x (mgr.24542) 462 : cluster [DBG] pgmap v939: 25 pgs: 25 active+clean; 664 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.2 MiB/s rd, 5.7 MiB/s wr, 24 op/s 2021-07-31T01:58:05.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:04 smithi045 conmon[33655]: cluster 2021-07-31T01:58:03.734102+0000 mon.a (mon.0) 1437 : cluster [DBG] 2021-07-31T01:58:05.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:04 smithi045 conmon[33655]: osdmap e767: 8 total, 8 up, 8 in 2021-07-31T01:58:05.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:04 smithi045 conmon[33655]: cluster 2021-07-31T01:58:03.947497+0000 mgr.x (mgr.24542) 462 : cluster [DBG] pgmap v939: 25 pgs: 25 active+clean; 664 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.2 MiB/s rd, 5.7 MiB/s wr, 24 op/s 2021-07-31T01:58:05.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:04 smithi045 conmon[39718]: cluster 2021-07-31T01:58:03.734102+0000 mon.a (mon 2021-07-31T01:58:05.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:04 smithi045 conmon[39718]: .0) 1437 : cluster [DBG] osdmap e767: 8 total, 8 up, 8 in 2021-07-31T01:58:05.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:04 smithi045 conmon[39718]: cluster 2021-07-31T01:58:03.947497+0000 mgr.x (mgr.24542) 462 : 2021-07-31T01:58:05.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:04 smithi045 conmon[39718]: cluster [DBG] pgmap v939: 25 pgs: 25 active+clean; 664 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.2 MiB/s rd, 5.7 MiB/s wr, 24 op/s 2021-07-31T01:58:05.744 INFO:tasks.rados.rados.0.smithi045.stdout:3748: done (0 left) 2021-07-31T01:58:05.744 INFO:tasks.rados.rados.0.smithi045.stdout:3749: rollback oid 37 current snap is 364 2021-07-31T01:58:05.745 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 37 to 344 2021-07-31T01:58:05.745 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 36 current snap is 364 2021-07-31T01:58:05.745 INFO:tasks.rados.rados.0.smithi045.stdout:3750: seq_num 1148 ranges {0=32768} 2021-07-31T01:58:05.745 INFO:tasks.rados.rados.0.smithi045.stdout:3750: writing smithi04569269-36 from 0 to 32768 tid 1 2021-07-31T01:58:05.745 INFO:tasks.rados.rados.0.smithi045.stdout:3751: read oid 21 snap -1 2021-07-31T01:58:05.746 INFO:tasks.rados.rados.0.smithi045.stdout:3751: expect (ObjNum 1108 snap 351 seq_num 1108) 2021-07-31T01:58:05.746 INFO:tasks.rados.rados.0.smithi045.stdout:3752: copy_from oid 6 from oid 43 current snap is 364 2021-07-31T01:58:05.746 INFO:tasks.rados.rados.0.smithi045.stdout:3753: snap_create 2021-07-31T01:58:05.757 INFO:tasks.rados.rados.0.smithi045.stdout:3749: finishing rollback tid 0 to smithi04569269-37 2021-07-31T01:58:05.757 INFO:tasks.rados.rados.0.smithi045.stdout:3749: finishing rollback tid 1 to smithi04569269-37 2021-07-31T01:58:05.757 INFO:tasks.rados.rados.0.smithi045.stdout:3749: finishing rollback tid 2 to smithi04569269-37 2021-07-31T01:58:05.757 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 37 v 1899 (ObjNum 1045 snap 340 seq_num 1045) dirty exists 2021-07-31T01:58:05.771 INFO:tasks.rados.rados.0.smithi045.stdout:3750: finishing write tid 1 to smithi04569269-36 2021-07-31T01:58:05.772 INFO:tasks.rados.rados.0.smithi045.stdout:3750: finishing write tid 2 to smithi04569269-36 2021-07-31T01:58:05.774 INFO:tasks.rados.rados.0.smithi045.stdout:3751: expect (ObjNum 1108 snap 351 seq_num 1108) 2021-07-31T01:58:05.920 INFO:tasks.rados.rados.0.smithi045.stdout:3750: finishing write tid 3 to smithi04569269-36 2021-07-31T01:58:05.920 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 36 v 2759 (ObjNum 1148 snap 364 seq_num 1148) dirty exists 2021-07-31T01:58:05.920 INFO:tasks.rados.rados.0.smithi045.stdout:3750: left oid 36 (ObjNum 1148 snap 364 seq_num 1148) 2021-07-31T01:58:05.920 INFO:tasks.rados.rados.0.smithi045.stdout:3752: finishing copy_from to smithi04569269-6 2021-07-31T01:58:05.921 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 6 v 2070 (ObjNum 1113 snap 351 seq_num 1113) dirty exists 2021-07-31T01:58:05.921 INFO:tasks.rados.rados.0.smithi045.stdout:3752: finishing copy_from racing read to smithi04569269-6 2021-07-31T01:58:06.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:05 smithi154 conmon[34553]: cluster 2021-07-31T01:58:04.739731+0000 mon.a (mon.0) 2021-07-31T01:58:06.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:05 smithi154 conmon[34553]: 1438 : cluster [DBG] osdmap e768: 8 total, 8 up, 8 in 2021-07-31T01:58:06.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:05 smithi045 conmon[33655]: cluster 2021-07-31T01:58:04.739731+0000 2021-07-31T01:58:06.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:05 smithi045 conmon[33655]: mon.a (mon.0) 1438 : cluster [DBG] osdmap e768: 8 total, 8 up, 8 in 2021-07-31T01:58:06.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:05 smithi045 conmon[39718]: cluster 2021-07-31T01:58:04.739731+0000 mon.a (mon.0) 1438 : cluster [DBG] osdmap e768: 8 total, 8 up, 8 in 2021-07-31T01:58:06.678 INFO:tasks.rados.rados.0.smithi045.stdout:3749: done (4 left) 2021-07-31T01:58:06.678 INFO:tasks.rados.rados.0.smithi045.stdout:3750: done (3 left) 2021-07-31T01:58:06.678 INFO:tasks.rados.rados.0.smithi045.stdout:3751: done (2 left) 2021-07-31T01:58:06.678 INFO:tasks.rados.rados.0.smithi045.stdout:3752: done (1 left) 2021-07-31T01:58:06.679 INFO:tasks.rados.rados.0.smithi045.stdout:3753: done (0 left) 2021-07-31T01:58:06.679 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 20 current snap is 365 2021-07-31T01:58:06.679 INFO:tasks.rados.rados.0.smithi045.stdout:3754: seq_num 1149 ranges {0=49152} 2021-07-31T01:58:06.679 INFO:tasks.rados.rados.0.smithi045.stdout:3754: writing smithi04569269-20 from 0 to 49152 tid 1 2021-07-31T01:58:06.680 INFO:tasks.rados.rados.0.smithi045.stdout:3755: read oid 30 snap -1 2021-07-31T01:58:06.680 INFO:tasks.rados.rados.0.smithi045.stdout:3755: expect deleted 2021-07-31T01:58:06.680 INFO:tasks.rados.rados.0.smithi045.stdout:3756: snap_remove snap 361 2021-07-31T01:58:07.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:06 smithi154 conmon[34553]: cluster 2021-07-31T01:58:05.741513+0000 mon.a (mon.0) 1439 : cluster [DBG] osdmap e769: 8 total, 8 up, 8 in 2021-07-31T01:58:07.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:06 smithi154 conmon[34553]: cluster 2021-07-31T01:58:05.947976+0000 mgr.x (mgr.24542) 463 : cluster [DBG] pgmap v942: 25 pgs: 25 active+clean; 664 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.2 MiB/s rd, 5.7 MiB/s wr, 24 op/s 2021-07-31T01:58:07.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:06 smithi154 conmon[34553]: cluster 2021-07-31T01:58:06.676419+0000 mon.a (mon.0) 1440 : cluster [DBG] osdmap e770: 8 total, 8 up, 8 in 2021-07-31T01:58:07.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:06 smithi045 conmon[39718]: cluster 2021-07-31T01:58:05.741513+0000 mon.a (mon.0) 1439 : cluster [DBG] osdmap e769: 8 total, 8 up, 8 in 2021-07-31T01:58:07.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:06 smithi045 conmon[39718]: cluster 2021-07-31T01:58:05.947976+0000 mgr.x (mgr.24542) 463 : cluster [DBG] pgmap v942: 25 pgs: 25 active+clean; 664 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.2 MiB/s rd, 5.7 MiB/s wr, 24 op/s 2021-07-31T01:58:07.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:06 smithi045 conmon[39718]: cluster 2021-07-31T01:58:06.676419+0000 mon.a (mon 2021-07-31T01:58:07.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:06 smithi045 conmon[39718]: .0) 1440 : cluster [DBG] osdmap e770: 8 total, 8 up, 8 in 2021-07-31T01:58:07.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:06 smithi045 conmon[33655]: cluster 2021-07-31T01:58:05.741513+0000 mon.a (mon.0) 1439 : 2021-07-31T01:58:07.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:06 smithi045 conmon[33655]: cluster [DBG] osdmap e769: 8 total, 8 up, 8 in 2021-07-31T01:58:07.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:06 smithi045 conmon[33655]: cluster 2021-07-31T01:58:05.947976+0000 mgr.x (mgr.24542) 463 : cluster [DBG] pgmap v942: 25 pgs: 25 active+clean; 664 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.2 MiB/s rd, 5.7 MiB/s wr, 24 op/s 2021-07-31T01:58:07.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:06 smithi045 conmon[33655]: cluster 2021-07-31T01:58:07.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:06 smithi045 conmon[33655]: 2021-07-31T01:58:06.676419+0000 mon.a (mon.0) 1440 : cluster [DBG] osdmap e770: 8 total, 8 up, 8 in 2021-07-31T01:58:07.686 INFO:tasks.rados.rados.0.smithi045.stdout:3756: done (2 left) 2021-07-31T01:58:07.686 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 29 current snap is 365 2021-07-31T01:58:07.687 INFO:tasks.rados.rados.0.smithi045.stdout:3757: seq_num 1150 ranges {1064960=65536} 2021-07-31T01:58:07.693 INFO:tasks.rados.rados.0.smithi045.stdout:3757: writing smithi04569269-29 from 1064960 to 1130496 tid 1 2021-07-31T01:58:07.694 INFO:tasks.rados.rados.0.smithi045.stdout:3758: read oid 18 snap -1 2021-07-31T01:58:07.694 INFO:tasks.rados.rados.0.smithi045.stdout:3758: expect (ObjNum 1120 snap 353 seq_num 1120) 2021-07-31T01:58:07.694 INFO:tasks.rados.rados.0.smithi045.stdout:3759: read oid 14 snap -1 2021-07-31T01:58:07.694 INFO:tasks.rados.rados.0.smithi045.stdout:3759: expect (ObjNum 0 snap 0 seq_num 0) 2021-07-31T01:58:07.695 INFO:tasks.rados.rados.0.smithi045.stdout:3760: rollback oid 19 current snap is 365 2021-07-31T01:58:07.696 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 19 to 304 2021-07-31T01:58:07.696 INFO:tasks.rados.rados.0.smithi045.stdout:3754: finishing write tid 1 to smithi04569269-20 2021-07-31T01:58:07.697 INFO:tasks.rados.rados.0.smithi045.stdout:3754: finishing write tid 2 to smithi04569269-20 2021-07-31T01:58:07.697 INFO:tasks.rados.rados.0.smithi045.stdout:3754: finishing write tid 3 to smithi04569269-20 2021-07-31T01:58:07.698 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 20 v 2774 (ObjNum 1149 snap 365 seq_num 1149) dirty exists 2021-07-31T01:58:07.699 INFO:tasks.rados.rados.0.smithi045.stdout:3754: left oid 20 (ObjNum 1149 snap 365 seq_num 1149) 2021-07-31T01:58:07.699 INFO:tasks.rados.rados.0.smithi045.stdout:3754: done (5 left) 2021-07-31T01:58:07.700 INFO:tasks.rados.rados.0.smithi045.stdout:3755: done (4 left) 2021-07-31T01:58:07.700 INFO:tasks.rados.rados.0.smithi045.stdout:3761: rollback oid 44 current snap is 365 2021-07-31T01:58:07.700 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 44 to 303 2021-07-31T01:58:07.700 INFO:tasks.rados.rados.0.smithi045.stdout:3762: setattr oid 34 current snap is 365 2021-07-31T01:58:07.701 INFO:tasks.rados.rados.0.smithi045.stdout:3759: expect (ObjNum 0 snap 0 seq_num 0) 2021-07-31T01:58:07.702 INFO:tasks.rados.rados.0.smithi045.stdout:3759: done (5 left) 2021-07-31T01:58:07.702 INFO:tasks.rados.rados.0.smithi045.stdout:3763: rollback oid 3 current snap is 365 2021-07-31T01:58:07.702 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 3 to 352 2021-07-31T01:58:07.702 INFO:tasks.rados.rados.0.smithi045.stdout:3764: snap_remove snap 344 2021-07-31T01:58:08.696 INFO:tasks.rados.rados.0.smithi045.stdout:3757: finishing write tid 1 to smithi04569269-29 2021-07-31T01:58:08.696 INFO:tasks.rados.rados.0.smithi045.stdout:3761: finishing rollback tid 0 to smithi04569269-44 2021-07-31T01:58:08.696 INFO:tasks.rados.rados.0.smithi045.stdout:3761: finishing rollback tid 1 to smithi04569269-44 2021-07-31T01:58:08.697 INFO:tasks.rados.rados.0.smithi045.stdout:3757: finishing write tid 2 to smithi04569269-29 2021-07-31T01:58:08.697 INFO:tasks.rados.rados.0.smithi045.stdout:3757: finishing write tid 3 to smithi04569269-29 2021-07-31T01:58:08.697 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 29 v 1430 (ObjNum 1150 snap 365 seq_num 1150) dirty exists 2021-07-31T01:58:08.697 INFO:tasks.rados.rados.0.smithi045.stdout:3757: left oid 29 (ObjNum 1150 snap 365 seq_num 1150) 2021-07-31T01:58:08.697 INFO:tasks.rados.rados.0.smithi045.stdout:3760: finishing rollback tid 1 to smithi04569269-19 2021-07-31T01:58:08.698 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 19 v 1858 (ObjNum 1667788657 snap 1751077968 seq_num 1801807700) dirty dne 2021-07-31T01:58:08.698 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 34 v 1902 (ObjNum 1107 snap 351 seq_num 1107) dirty exists 2021-07-31T01:58:08.698 INFO:tasks.rados.rados.0.smithi045.stdout:3761: finishing rollback tid 2 to smithi04569269-44 2021-07-31T01:58:08.698 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 44 v 2091 (ObjNum 955 snap 289 seq_num 955) dirty exists 2021-07-31T01:58:08.699 INFO:tasks.rados.rados.0.smithi045.stdout:3758: expect (ObjNum 1120 snap 353 seq_num 1120) 2021-07-31T01:58:08.754 INFO:tasks.rados.rados.0.smithi045.stdout:3763: finishing rollback tid 0 to smithi04569269-3 2021-07-31T01:58:08.755 INFO:tasks.rados.rados.0.smithi045.stdout:3763: finishing rollback tid 1 to smithi04569269-3 2021-07-31T01:58:08.755 INFO:tasks.rados.rados.0.smithi045.stdout:3763: finishing rollback tid 2 to smithi04569269-3 2021-07-31T01:58:08.756 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 3 v 2778 (ObjNum 881 snap 272 seq_num 881) dirty exists 2021-07-31T01:58:08.756 INFO:tasks.rados.rados.0.smithi045.stdout:3757: done (6 left) 2021-07-31T01:58:08.756 INFO:tasks.rados.rados.0.smithi045.stdout:3758: done (5 left) 2021-07-31T01:58:08.756 INFO:tasks.rados.rados.0.smithi045.stdout:3760: done (4 left) 2021-07-31T01:58:08.757 INFO:tasks.rados.rados.0.smithi045.stdout:3761: done (3 left) 2021-07-31T01:58:08.757 INFO:tasks.rados.rados.0.smithi045.stdout:3762: done (2 left) 2021-07-31T01:58:08.757 INFO:tasks.rados.rados.0.smithi045.stdout:3763: done (1 left) 2021-07-31T01:58:08.757 INFO:tasks.rados.rados.0.smithi045.stdout:3764: done (0 left) 2021-07-31T01:58:08.757 INFO:tasks.rados.rados.0.smithi045.stdout:3765: read oid 11 snap 348 2021-07-31T01:58:08.758 INFO:tasks.rados.rados.0.smithi045.stdout:3765: expect (ObjNum 948 snap 286 seq_num 948) 2021-07-31T01:58:08.758 INFO:tasks.rados.rados.0.smithi045.stdout:3766: snap_create 2021-07-31T01:58:08.762 INFO:tasks.rados.rados.0.smithi045.stdout:3765: expect (ObjNum 948 snap 286 seq_num 948) 2021-07-31T01:58:09.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:08 smithi154 conmon[34553]: cluster 2021-07-31T01:58:07.685577+0000 mon.a (mon.0) 1441 : cluster 2021-07-31T01:58:09.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:08 smithi154 conmon[34553]: [DBG] osdmap e771: 8 total, 8 up, 8 in 2021-07-31T01:58:09.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:08 smithi154 conmon[34553]: cluster 2021-07-31T01:58:07.948463+0000 mgr.x (mgr.24542) 464 : cluster [DBG] pgmap v945: 25 pgs: 25 active+clean; 664 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2021-07-31T01:58:09.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:08 smithi045 conmon[33655]: cluster 2021-07-31T01:58:07.685577+0000 2021-07-31T01:58:09.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:08 smithi045 conmon[33655]: mon.a (mon.0) 1441 : cluster [DBG] osdmap e771: 8 total, 8 up, 8 in 2021-07-31T01:58:09.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:08 smithi045 conmon[33655]: cluster 2021-07-31T01:58:07.948463+0000 2021-07-31T01:58:09.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:08 smithi045 conmon[33655]: mgr.x (mgr.24542) 464 : cluster [DBG] pgmap v945: 25 pgs: 25 active+clean; 664 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2021-07-31T01:58:09.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:08 smithi045 conmon[39718]: cluster 2021-07-31T01:58:09.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:08 smithi045 conmon[39718]: 2021-07-31T01:58:07 2021-07-31T01:58:09.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:08 smithi045 conmon[39718]: .685577+0000 mon.a (mon.0) 1441 : cluster [DBG] osdmap e771: 8 total, 8 up, 8 in 2021-07-31T01:58:09.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:08 smithi045 conmon[39718]: cluster 2021-07-31T 2021-07-31T01:58:09.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:08 smithi045 conmon[39718]: 01:58:07.948463+0000 mgr.x (mgr.24542) 464 2021-07-31T01:58:09.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:08 smithi045 conmon[39718]: : cluster [DBG] pgmap v945: 25 pgs: 25 active+clean; 664 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2021-07-31T01:58:09.694 INFO:tasks.rados.rados.0.smithi045.stdout:3765: done (1 left) 2021-07-31T01:58:09.694 INFO:tasks.rados.rados.0.smithi045.stdout:3766: done (0 left) 2021-07-31T01:58:09.695 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 31 current snap is 366 2021-07-31T01:58:09.695 INFO:tasks.rados.rados.0.smithi045.stdout:3767: seq_num 1151 ranges {1835008=81920} 2021-07-31T01:58:09.705 INFO:tasks.rados.rados.0.smithi045.stdout:3767: writing smithi04569269-31 from 1835008 to 1916928 tid 1 2021-07-31T01:58:09.705 INFO:tasks.rados.rados.0.smithi045.stdout:3768: copy_from oid 50 from oid 15 current snap is 366 2021-07-31T01:58:09.705 INFO:tasks.rados.rados.0.smithi045.stdout:3769: snap_create 2021-07-31T01:58:09.710 INFO:tasks.rados.rados.0.smithi045.stdout:3767: finishing write tid 1 to smithi04569269-31 2021-07-31T01:58:09.711 INFO:tasks.rados.rados.0.smithi045.stdout:3767: finishing write tid 2 to smithi04569269-31 2021-07-31T01:58:09.715 INFO:tasks.rados.rados.0.smithi045.stdout:3767: finishing write tid 3 to smithi04569269-31 2021-07-31T01:58:09.715 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 31 v 2801 (ObjNum 1151 snap 366 seq_num 1151) dirty exists 2021-07-31T01:58:09.715 INFO:tasks.rados.rados.0.smithi045.stdout:3767: left oid 31 (ObjNum 1151 snap 366 seq_num 1151) 2021-07-31T01:58:09.728 INFO:tasks.rados.rados.0.smithi045.stdout:3768: finishing copy_from to smithi04569269-50 2021-07-31T01:58:09.728 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 50 v 1896 (ObjNum 1144 snap 362 seq_num 1144) dirty exists 2021-07-31T01:58:09.729 INFO:tasks.rados.rados.0.smithi045.stdout:3768: finishing copy_from racing read to smithi04569269-50 2021-07-31T01:58:10.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:09 smithi154 conmon[34553]: cluster 2021-07-31T01:58:08.692924+0000 mon.a (mon.0) 1442 : cluster [DBG] osdmap e772: 8 total, 8 up, 8 in 2021-07-31T01:58:10.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:09 smithi045 conmon[39718]: cluster 2021-07-31T01:58:08.692924 2021-07-31T01:58:10.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:09 smithi045 conmon[39718]: +0000 mon.a (mon.0) 1442 : cluster [DBG] osdmap e772: 8 total, 8 up, 8 in 2021-07-31T01:58:10.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:09 smithi045 conmon[33655]: cluster 2021-07-31T01:58:08.692924+0000 mon.a (mon.0 2021-07-31T01:58:10.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:09 smithi045 conmon[33655]: ) 1442 : cluster [DBG] osdmap e772: 8 total, 8 up, 8 in 2021-07-31T01:58:10.705 INFO:tasks.rados.rados.0.smithi045.stdout:3767: done (2 left) 2021-07-31T01:58:10.706 INFO:tasks.rados.rados.0.smithi045.stdout:3768: done (1 left) 2021-07-31T01:58:10.707 INFO:tasks.rados.rados.0.smithi045.stdout:3769: done (0 left) 2021-07-31T01:58:10.707 INFO:tasks.rados.rados.0.smithi045.stdout:3770: rollback oid 31 current snap is 367 2021-07-31T01:58:10.707 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 31 to 341 2021-07-31T01:58:10.707 INFO:tasks.rados.rados.0.smithi045.stdout:3771: read oid 29 snap -1 2021-07-31T01:58:10.707 INFO:tasks.rados.rados.0.smithi045.stdout:3771: expect (ObjNum 1150 snap 365 seq_num 1150) 2021-07-31T01:58:10.708 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 8 current snap is 367 2021-07-31T01:58:10.708 INFO:tasks.rados.rados.0.smithi045.stdout:3772: seq_num 1152 ranges {0=98304} 2021-07-31T01:58:10.708 INFO:tasks.rados.rados.0.smithi045.stdout:3772: writing smithi04569269-8 from 0 to 98304 tid 1 2021-07-31T01:58:10.708 INFO:tasks.rados.rados.0.smithi045.stdout:3773: rmattr oid 6 current snap is 367 2021-07-31T01:58:10.709 INFO:tasks.rados.rados.0.smithi045.stdout:3773: done (3 left) 2021-07-31T01:58:10.709 INFO:tasks.rados.rados.0.smithi045.stdout:3774: copy_from oid 4 from oid 21 current snap is 367 2021-07-31T01:58:10.709 INFO:tasks.rados.rados.0.smithi045.stdout:3775: setattr oid 45 current snap is 367 2021-07-31T01:58:10.711 INFO:tasks.rados.rados.0.smithi045.stdout:3776: rollback oid 9 current snap is 367 2021-07-31T01:58:10.711 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 9 to 340 2021-07-31T01:58:10.711 INFO:tasks.rados.rados.0.smithi045.stdout:3777: read oid 46 snap -1 2021-07-31T01:58:10.711 INFO:tasks.rados.rados.0.smithi045.stdout:3777: expect (ObjNum 1126 snap 357 seq_num 1126) 2021-07-31T01:58:10.712 INFO:tasks.rados.rados.0.smithi045.stdout:3778: rollback oid 23 current snap is 367 2021-07-31T01:58:10.712 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 23 to 364 2021-07-31T01:58:10.712 INFO:tasks.rados.rados.0.smithi045.stdout:3779: rollback oid 43 current snap is 367 2021-07-31T01:58:10.712 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 43 to 352 2021-07-31T01:58:10.712 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 11 current snap is 367 2021-07-31T01:58:10.713 INFO:tasks.rados.rados.0.smithi045.stdout:3780: seq_num 1153 ranges {0=114688} 2021-07-31T01:58:10.713 INFO:tasks.rados.rados.0.smithi045.stdout:3780: writing smithi04569269-11 from 0 to 114688 tid 1 2021-07-31T01:58:10.713 INFO:tasks.rados.rados.0.smithi045.stdout:3781: snap_create 2021-07-31T01:58:10.714 INFO:tasks.rados.rados.0.smithi045.stdout:3771: expect (ObjNum 1150 snap 365 seq_num 1150) 2021-07-31T01:58:10.768 INFO:tasks.rados.rados.0.smithi045.stdout:3776: finishing rollback tid 0 to smithi04569269-9 2021-07-31T01:58:10.768 INFO:tasks.rados.rados.0.smithi045.stdout:3776: finishing rollback tid 1 to smithi04569269-9 2021-07-31T01:58:10.769 INFO:tasks.rados.rados.0.smithi045.stdout:3776: finishing rollback tid 2 to smithi04569269-9 2021-07-31T01:58:10.769 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 9 v 2376 (ObjNum 990 snap 307 seq_num 990) dirty exists 2021-07-31T01:58:10.769 INFO:tasks.rados.rados.0.smithi045.stdout:3778: finishing rollback tid 0 to smithi04569269-23 2021-07-31T01:58:10.769 INFO:tasks.rados.rados.0.smithi045.stdout:3778: finishing rollback tid 1 to smithi04569269-23 2021-07-31T01:58:10.770 INFO:tasks.rados.rados.0.smithi045.stdout:3779: finishing rollback tid 0 to smithi04569269-43 2021-07-31T01:58:10.770 INFO:tasks.rados.rados.0.smithi045.stdout:3779: finishing rollback tid 1 to smithi04569269-43 2021-07-31T01:58:10.770 INFO:tasks.rados.rados.0.smithi045.stdout:3779: finishing rollback tid 2 to smithi04569269-43 2021-07-31T01:58:10.770 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 43 v 2094 (ObjNum 1113 snap 351 seq_num 1113) dirty exists 2021-07-31T01:58:10.771 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 45 v 2379 (ObjNum 1147 snap 363 seq_num 1147) dirty exists 2021-07-31T01:58:10.771 INFO:tasks.rados.rados.0.smithi045.stdout:3770: finishing rollback tid 0 to smithi04569269-31 2021-07-31T01:58:10.771 INFO:tasks.rados.rados.0.smithi045.stdout:3770: finishing rollback tid 1 to smithi04569269-31 2021-07-31T01:58:10.771 INFO:tasks.rados.rados.0.smithi045.stdout:3778: finishing rollback tid 2 to smithi04569269-23 2021-07-31T01:58:10.771 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 23 v 2059 (ObjNum 1049 snap 342 seq_num 1049) dirty exists 2021-07-31T01:58:10.772 INFO:tasks.rados.rados.0.smithi045.stdout:3770: finishing rollback tid 2 to smithi04569269-31 2021-07-31T01:58:10.772 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 31 v 2805 (ObjNum 1006 snap 322 seq_num 1006) dirty exists 2021-07-31T01:58:10.772 INFO:tasks.rados.rados.0.smithi045.stdout:3772: finishing write tid 1 to smithi04569269-8 2021-07-31T01:58:10.772 INFO:tasks.rados.rados.0.smithi045.stdout:3772: finishing write tid 2 to smithi04569269-8 2021-07-31T01:58:10.773 INFO:tasks.rados.rados.0.smithi045.stdout:3780: finishing write tid 1 to smithi04569269-11 2021-07-31T01:58:10.773 INFO:tasks.rados.rados.0.smithi045.stdout:3780: finishing write tid 2 to smithi04569269-11 2021-07-31T01:58:10.773 INFO:tasks.rados.rados.0.smithi045.stdout:3780: finishing write tid 3 to smithi04569269-11 2021-07-31T01:58:10.773 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 11 v 2061 (ObjNum 1153 snap 367 seq_num 1153) dirty exists 2021-07-31T01:58:10.773 INFO:tasks.rados.rados.0.smithi045.stdout:3780: left oid 11 (ObjNum 1153 snap 367 seq_num 1153) 2021-07-31T01:58:10.774 INFO:tasks.rados.rados.0.smithi045.stdout:3772: finishing write tid 3 to smithi04569269-8 2021-07-31T01:58:10.774 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 8 v 2807 (ObjNum 1152 snap 367 seq_num 1152) dirty exists 2021-07-31T01:58:10.774 INFO:tasks.rados.rados.0.smithi045.stdout:3772: left oid 8 (ObjNum 1152 snap 367 seq_num 1152) 2021-07-31T01:58:10.774 INFO:tasks.rados.rados.0.smithi045.stdout:3777: expect (ObjNum 1126 snap 357 seq_num 1126) 2021-07-31T01:58:10.889 INFO:tasks.rados.rados.0.smithi045.stdout:3774: finishing copy_from to smithi04569269-4 2021-07-31T01:58:10.890 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 4 v 2535 (ObjNum 1108 snap 351 seq_num 1108) dirty exists 2021-07-31T01:58:10.890 INFO:tasks.rados.rados.0.smithi045.stdout:3774: finishing copy_from racing read to smithi04569269-4 2021-07-31T01:58:11.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:10 smithi154 conmon[34553]: cluster 2021-07-31T01:58:09.693521+0000 mon.a (mon.0) 2021-07-31T01:58:11.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:10 smithi154 conmon[34553]: 1443 : cluster [DBG] osdmap e773: 8 total, 8 up, 8 in 2021-07-31T01:58:11.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:10 smithi154 conmon[34553]: cluster 2021-07-31T01:58:09.948908+0000 mgr.x (mgr.24542) 465 : cluster [DBG] pgmap v948: 25 pgs: 25 active+clean; 661 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 4.8 MiB/s rd, 1.5 MiB/s wr, 13 op/s 2021-07-31T01:58:11.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:10 smithi045 conmon[33655]: cluster 2021-07-31T01:58:09.693521+0000 mon.a (mon.0) 1443 : cluster 2021-07-31T01:58:11.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:10 smithi045 conmon[33655]: [DBG] osdmap e773: 8 total, 8 up, 8 in 2021-07-31T01:58:11.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:10 smithi045 conmon[33655]: cluster 2021-07-31T01:58:09.948908+0000 mgr.x (mgr.24542) 465 : cluster [DBG] pgmap v948: 25 pgs: 25 active+clean; 661 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 4.8 MiB/s rd, 1.5 MiB/s wr, 13 op/s 2021-07-31T01:58:11.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:10 smithi045 conmon[39718]: cluster 2021-07-31T01:58:09.693521+0000 2021-07-31T01:58:11.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:10 smithi045 conmon[39718]: mon.a (mon.0) 1443 : cluster [DBG] osdmap e773: 8 total, 8 up, 8 in 2021-07-31T01:58:11.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:10 smithi045 conmon[39718]: cluster 2021-07-31T01:58:11.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:10 smithi045 conmon[39718]: 2021-07-31T01:58:09.948908+0000 2021-07-31T01:58:11.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:10 smithi045 conmon[39718]: mgr.x (mgr.24542) 465 : 2021-07-31T01:58:11.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:10 smithi045 conmon[39718]: cluster [DBG] pgmap v948: 25 pgs: 25 active+clean; 661 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 4.8 MiB/s rd, 1.5 MiB/s wr, 13 op/s 2021-07-31T01:58:11.679 INFO:tasks.rados.rados.0.smithi045.stdout:3770: done (10 left) 2021-07-31T01:58:11.680 INFO:tasks.rados.rados.0.smithi045.stdout:3771: done (9 left) 2021-07-31T01:58:11.680 INFO:tasks.rados.rados.0.smithi045.stdout:3772: done (8 left) 2021-07-31T01:58:11.681 INFO:tasks.rados.rados.0.smithi045.stdout:3774: done (7 left) 2021-07-31T01:58:11.681 INFO:tasks.rados.rados.0.smithi045.stdout:3775: done (6 left) 2021-07-31T01:58:11.681 INFO:tasks.rados.rados.0.smithi045.stdout:3776: done (5 left) 2021-07-31T01:58:11.681 INFO:tasks.rados.rados.0.smithi045.stdout:3777: done (4 left) 2021-07-31T01:58:11.682 INFO:tasks.rados.rados.0.smithi045.stdout:3778: done (3 left) 2021-07-31T01:58:11.682 INFO:tasks.rados.rados.0.smithi045.stdout:3779: done (2 left) 2021-07-31T01:58:11.682 INFO:tasks.rados.rados.0.smithi045.stdout:3780: done (1 left) 2021-07-31T01:58:11.683 INFO:tasks.rados.rados.0.smithi045.stdout:3781: done (0 left) 2021-07-31T01:58:11.683 INFO:tasks.rados.rados.0.smithi045.stdout:3782: snap_create 2021-07-31T01:58:12.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:11 smithi154 conmon[34553]: cluster 2021-07-31T01:58:10.704330+0000 mon.a (mon.0) 2021-07-31T01:58:12.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:11 smithi154 conmon[34553]: 1444 : cluster [DBG] osdmap e774: 8 total, 8 up, 8 in 2021-07-31T01:58:12.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:11 smithi154 conmon[34553]: cluster 2021-07-31T01:58:11.678249+0000 mon.a (mon.0) 1445 : cluster [DBG] osdmap e775: 8 total, 8 up, 8 in 2021-07-31T01:58:12.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:11 smithi045 conmon[33655]: cluster 2021-07-31T01:58:10.704330+0000 mon.a (mon.0) 2021-07-31T01:58:12.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:11 smithi045 conmon[33655]: 1444 : cluster [DBG] osdmap e774: 8 total, 8 up, 8 in 2021-07-31T01:58:12.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:11 smithi045 conmon[33655]: cluster 2021-07-31T01:58:11.678249+0000 mon.a (mon.0) 1445 : cluster [DBG] osdmap e775: 8 total, 8 up, 8 in 2021-07-31T01:58:12.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:11 smithi045 conmon[39718]: cluster 2021-07-31T01:58:10.704330+0000 mon.a (mon. 2021-07-31T01:58:12.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:11 smithi045 conmon[39718]: 0) 1444 : cluster [DBG] osdmap e774: 8 total, 8 up, 8 in 2021-07-31T01:58:12.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:11 smithi045 conmon[39718]: cluster 2021-07-31T01:58:11.678249+0000 mon.a (mon.0) 1445 : cluster 2021-07-31T01:58:12.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:11 smithi045 conmon[39718]: [DBG] osdmap e775: 8 total, 8 up, 8 in 2021-07-31T01:58:12.692 INFO:tasks.rados.rados.0.smithi045.stdout:3782: done (0 left) 2021-07-31T01:58:12.692 INFO:tasks.rados.rados.0.smithi045.stdout:3783: copy_from oid 12 from oid 13 current snap is 369 2021-07-31T01:58:12.693 INFO:tasks.rados.rados.0.smithi045.stdout:3784: snap_create 2021-07-31T01:58:12.693 INFO:tasks.rados.rados.0.smithi045.stdout:3783: finishing copy_from to smithi04569269-12 2021-07-31T01:58:12.693 INFO:tasks.rados.rados.0.smithi045.stdout:3783: got expected ENOENT (src dne) 2021-07-31T01:58:12.693 INFO:tasks.rados.rados.0.smithi045.stdout:3783: finishing copy_from racing read to smithi04569269-12 2021-07-31T01:58:13.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:12 smithi154 conmon[34553]: cluster 2021-07-31T01:58:11.949432+0000 mgr.x (mgr.24542) 466 : cluster [DBG] pgmap v951: 25 pgs: 25 active+clean; 661 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 4.8 MiB/s rd, 1.5 MiB/s wr, 13 op/s 2021-07-31T01:58:13.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:12 smithi154 conmon[34553]: cluster 2021-07-31T 2021-07-31T01:58:13.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:12 smithi154 conmon[34553]: 01:58:12.686536+0000 mon.a (mon.0) 1446 : cluster [DBG] osdmap e776: 8 total, 8 up, 8 in 2021-07-31T01:58:13.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:12 smithi045 conmon[39718]: cluster 2021-07-31T01:58:11.949432+0000 mgr.x (mgr.24542 2021-07-31T01:58:13.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:12 smithi045 conmon[39718]: ) 466 : cluster [DBG] pgmap v951: 25 pgs: 25 active+clean; 661 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 4.8 MiB/s rd, 1.5 MiB/s wr, 13 op/s 2021-07-31T01:58:13.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:12 smithi045 conmon[39718]: cluster 2021-07-31T01:58:12.686536+0000 mon.a (mon.0) 1446 : cluster 2021-07-31T01:58:13.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:12 smithi045 conmon[39718]: [DBG] osdmap e776: 8 total, 8 up, 8 in 2021-07-31T01:58:13.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:12 smithi045 conmon[33655]: cluster 2021-07-31T01:58:11.949432+0000 mgr.x (mgr 2021-07-31T01:58:13.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:12 smithi045 conmon[33655]: .24542) 466 : cluster [DBG] pgmap v951: 25 pgs: 25 active+clean; 661 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 4.8 MiB/s rd, 1.5 MiB/s wr, 13 op/s 2021-07-31T01:58:13.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:12 smithi045 conmon[33655]: cluster 2021-07-31T01:58:12.686536+0000 mon.a (mon.0) 1446 : cluster [DBG] osdmap e776: 8 total, 8 up, 8 in 2021-07-31T01:58:13.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:12 smithi045 conmon[33655]: 2021-07-31T01:58:13.689 INFO:tasks.rados.rados.0.smithi045.stdout:3783: done (1 left) 2021-07-31T01:58:13.689 INFO:tasks.rados.rados.0.smithi045.stdout:3784: done (0 left) 2021-07-31T01:58:13.689 INFO:tasks.rados.rados.0.smithi045.stdout:3785: rollback oid 8 current snap is 370 2021-07-31T01:58:13.689 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 8 to 360 2021-07-31T01:58:13.690 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 32 current snap is 370 2021-07-31T01:58:13.690 INFO:tasks.rados.rados.0.smithi045.stdout:3786: seq_num 1154 ranges {11001856=131072} 2021-07-31T01:58:13.751 INFO:tasks.rados.rados.0.smithi045.stdout:3786: writing smithi04569269-32 from 11001856 to 11132928 tid 1 2021-07-31T01:58:13.751 INFO:tasks.rados.rados.0.smithi045.stdout:3787: snap_create 2021-07-31T01:58:13.751 INFO:tasks.rados.rados.0.smithi045.stdout:3785: finishing rollback tid 0 to smithi04569269-8 2021-07-31T01:58:13.751 INFO:tasks.rados.rados.0.smithi045.stdout:3785: finishing rollback tid 1 to smithi04569269-8 2021-07-31T01:58:13.752 INFO:tasks.rados.rados.0.smithi045.stdout:3785: finishing rollback tid 2 to smithi04569269-8 2021-07-31T01:58:13.752 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 8 v 2811 (ObjNum 1131 snap 360 seq_num 1131) dirty exists 2021-07-31T01:58:13.756 INFO:tasks.rados.rados.0.smithi045.stdout:3786: finishing write tid 1 to smithi04569269-32 2021-07-31T01:58:13.757 INFO:tasks.rados.rados.0.smithi045.stdout:3786: finishing write tid 2 to smithi04569269-32 2021-07-31T01:58:13.758 INFO:tasks.rados.rados.0.smithi045.stdout:3786: finishing write tid 3 to smithi04569269-32 2021-07-31T01:58:13.758 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 32 v 1883 (ObjNum 1154 snap 370 seq_num 1154) dirty exists 2021-07-31T01:58:13.758 INFO:tasks.rados.rados.0.smithi045.stdout:3786: left oid 32 (ObjNum 1154 snap 370 seq_num 1154) 2021-07-31T01:58:14.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:58:13 smithi045 conmon[61210]: level=error ts=2021-07-31T01:58:13.528Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:58:14.027 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:58:13 smithi045 conmon[61210]: level=error ts=2021-07-31T01:58:13.528Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:58:14.027 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:58:13 smithi045 conmon[61210]: level=error ts=2021-07-31T01:58:13.528Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T01:58:14.693 INFO:tasks.rados.rados.0.smithi045.stdout:3785: done (2 left) 2021-07-31T01:58:14.693 INFO:tasks.rados.rados.0.smithi045.stdout:3786: done (1 left) 2021-07-31T01:58:14.694 INFO:tasks.rados.rados.0.smithi045.stdout:3787: done (0 left) 2021-07-31T01:58:14.694 INFO:tasks.rados.rados.0.smithi045.stdout:3788: rmattr oid 22 current snap is 371 2021-07-31T01:58:14.694 INFO:tasks.rados.rados.0.smithi045.stdout:3788: done (0 left) 2021-07-31T01:58:14.694 INFO:tasks.rados.rados.0.smithi045.stdout:3789: read oid 10 snap -1 2021-07-31T01:58:14.695 INFO:tasks.rados.rados.0.smithi045.stdout:3789: expect (ObjNum 1128 snap 357 seq_num 1128) 2021-07-31T01:58:14.695 INFO:tasks.rados.rados.0.smithi045.stdout:3790: delete oid 6 current snap is 371 2021-07-31T01:58:14.706 INFO:tasks.rados.rados.0.smithi045.stdout:3790: done (1 left) 2021-07-31T01:58:14.707 INFO:tasks.rados.rados.0.smithi045.stdout:3791: snap_remove snap 304 2021-07-31T01:58:14.843 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T01:58:15.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:14 smithi154 conmon[34553]: cluster 2021-07-31T01:58:13.687915+0000 mon.a (mon.0) 1447 2021-07-31T01:58:15.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:14 smithi154 conmon[34553]: : cluster [DBG] osdmap e777: 8 total, 8 up, 8 in 2021-07-31T01:58:15.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:14 smithi154 conmon[34553]: cluster 2021-07-31T01:58:13.950085+0000 mgr.x (mgr.24542) 467 : cluster [DBG] pgmap v954: 25 pgs: 25 active+clean; 685 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 3.5 MiB/s rd, 1.6 MiB/s wr, 13 op/s 2021-07-31T01:58:15.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:14 smithi045 conmon[33655]: cluster 2021-07-31T01:58:13.687915+0000 mon.a (mon.0) 2021-07-31T01:58:15.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:14 smithi045 conmon[33655]: 1447 : cluster [DBG] osdmap e777: 8 total, 8 up, 8 in 2021-07-31T01:58:15.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:14 smithi045 conmon[33655]: cluster 2021-07-31T01:58:13.950085+0000 mgr.x (mgr.24542) 467 : cluster [DBG] pgmap v954: 25 pgs: 25 active+clean; 685 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 3.5 MiB/s rd, 1.6 MiB/s wr, 13 op/s 2021-07-31T01:58:15.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:14 smithi045 conmon[39718]: cluster 2021-07-31T01:58:13.687915+0000 2021-07-31T01:58:15.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:14 smithi045 conmon[39718]: mon.a (mon.0) 1447 : cluster [DBG] osdmap e777: 8 total, 8 up, 8 in 2021-07-31T01:58:15.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:14 smithi045 conmon[39718]: cluster 2021-07-31 2021-07-31T01:58:15.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:14 smithi045 conmon[39718]: T01:58:13.950085+0000 mgr.x (mgr.24542) 467 : cluster [DBG] pgmap v954: 25 pgs: 25 active+clean; 685 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 3.5 MiB/s rd, 1.6 MiB/s wr, 13 op/s 2021-07-31T01:58:15.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:14 smithi045 conmon[39718]: 2021-07-31T01:58:15.119 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T01:58:15.120 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 13m ago 13m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T01:58:15.120 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 10m ago 13m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T01:58:15.120 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 10m ago 17m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T01:58:15.120 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 13m ago 19m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T01:58:15.121 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 13m ago 19m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T01:58:15.121 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 10m ago 17m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T01:58:15.121 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 13m ago 17m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T01:58:15.121 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 13m ago 14m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T01:58:15.122 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 10m ago 14m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T01:58:15.122 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 13m ago 16m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T01:58:15.122 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 13m ago 16m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T01:58:15.122 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 13m ago 16m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T01:58:15.123 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 13m ago 15m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T01:58:15.123 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 10m ago 15m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T01:58:15.123 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 10m ago 15m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T01:58:15.124 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 10m ago 14m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T01:58:15.124 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 10m ago 14m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T01:58:15.125 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 10m ago 14m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T01:58:15.455 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T01:58:15.455 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T01:58:15.455 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T01:58:15.456 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T01:58:15.456 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T01:58:15.456 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T01:58:15.456 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T01:58:15.457 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T01:58:15.457 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T01:58:15.457 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T01:58:15.457 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T01:58:15.458 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T01:58:15.458 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T01:58:15.458 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T01:58:15.458 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T01:58:15.459 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T01:58:15.459 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T01:58:15.699 INFO:tasks.rados.rados.0.smithi045.stdout:3791: done (1 left) 2021-07-31T01:58:15.700 INFO:tasks.rados.rados.0.smithi045.stdout:3792: snap_remove snap 365 2021-07-31T01:58:16.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:15 smithi154 conmon[34553]: cluster 2021-07-31T01:58:14.692281+0000 mon.a (mon.0) 1448 : cluster [DBG] osdmap e778: 8 total, 8 up, 8 in 2021-07-31T01:58:16.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:15 smithi154 conmon[34553]: audit 2021-07-31T01:58:14.831740+0000 mgr.x (mgr.24542) 468 : audit [DBG] from='client.15090 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:58:16.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:15 smithi154 conmon[34553]: audit 2021-07-31T01:58:15.454917+0000 mon.a (mon.0) 1449 : audit [DBG] 2021-07-31T01:58:16.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:15 smithi154 conmon[34553]: from='client.? 172.21.15.45:0/2065499049' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T01:58:16.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:15 smithi045 conmon[33655]: cluster 2021-07-31T01:58: 2021-07-31T01:58:16.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:15 smithi045 conmon[33655]: 14.692281+0000 mon.a (mon.0) 1448 : cluster [DBG] osdmap e778: 8 total, 8 up, 8 in 2021-07-31T01:58:16.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:15 smithi045 conmon[33655]: audit 2021-07-31T01:58: 2021-07-31T01:58:16.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:15 smithi045 conmon[33655]: 14.831740+0000 mgr.x (mgr.24542) 468 : audit [DBG] from='client.15090 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:58:16.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:15 smithi045 conmon[33655]: audit 2021-07-31T01:58:16.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:15 smithi045 conmon[33655]: 2021-07-31T01:58:15. 2021-07-31T01:58:16.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:15 smithi045 conmon[33655]: 454917+0000 mon.a (mon.0) 1449 2021-07-31T01:58:16.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:15 smithi045 conmon[33655]: : audit [DBG] from='client.? 172.21.15.45:0/2065499049' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T01:58:16.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:15 smithi045 conmon[39718]: cluster 2021-07-31T01:58:14 2021-07-31T01:58:16.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:15 smithi045 conmon[39718]: .692281+0000 mon.a (mon.0) 1448 : cluster [DBG] osdmap e778: 8 total, 8 up, 8 in 2021-07-31T01:58:16.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:15 smithi045 conmon[39718]: audit 2021- 2021-07-31T01:58:16.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:15 smithi045 conmon[39718]: 07-31T01:58:14 2021-07-31T01:58:16.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:15 smithi045 conmon[39718]: .831740+0000 mgr.x (mgr 2021-07-31T01:58:16.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:15 smithi045 conmon[39718]: .24542) 468 : audit [DBG] 2021-07-31T01:58:16.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:15 smithi045 conmon[39718]: from='client.15090 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:58:16.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:15 smithi045 conmon[39718]: audit 2021- 2021-07-31T01:58:16.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:15 smithi045 conmon[39718]: 07-31T01:58:15. 2021-07-31T01:58:16.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:15 smithi045 conmon[39718]: 454917+0000 mon.a (mon. 2021-07-31T01:58:16.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:15 smithi045 conmon[39718]: 0) 1449 : audit [DBG] 2021-07-31T01:58:16.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:15 smithi045 conmon[39718]: from='client.? 172.21.15.45:0/2065499049' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T01:58:16.714 INFO:tasks.rados.rados.0.smithi045.stdout:3792: done (1 left) 2021-07-31T01:58:16.714 INFO:tasks.rados.rados.0.smithi045.stdout:3793: read oid 44 snap -1 2021-07-31T01:58:16.715 INFO:tasks.rados.rados.0.smithi045.stdout:3793: expect (ObjNum 955 snap 289 seq_num 955) 2021-07-31T01:58:16.715 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 4 current snap is 371 2021-07-31T01:58:16.715 INFO:tasks.rados.rados.0.smithi045.stdout:3794: seq_num 1156 ranges {3170304=163840} 2021-07-31T01:58:16.733 INFO:tasks.rados.rados.0.smithi045.stdout:3794: writing smithi04569269-4 from 3170304 to 3334144 tid 1 2021-07-31T01:58:16.734 INFO:tasks.rados.rados.0.smithi045.stdout:3795: snap_create 2021-07-31T01:58:16.734 INFO:tasks.rados.rados.0.smithi045.stdout:3789: expect (ObjNum 1128 snap 357 seq_num 1128) 2021-07-31T01:58:16.790 INFO:tasks.rados.rados.0.smithi045.stdout:3793: expect (ObjNum 955 snap 289 seq_num 955) 2021-07-31T01:58:16.909 INFO:tasks.rados.rados.0.smithi045.stdout:3794: finishing write tid 1 to smithi04569269-4 2021-07-31T01:58:16.909 INFO:tasks.rados.rados.0.smithi045.stdout:3794: finishing write tid 2 to smithi04569269-4 2021-07-31T01:58:16.909 INFO:tasks.rados.rados.0.smithi045.stdout:3794: finishing write tid 3 to smithi04569269-4 2021-07-31T01:58:16.909 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 4 v 2537 (ObjNum 1156 snap 371 seq_num 1156) dirty exists 2021-07-31T01:58:16.910 INFO:tasks.rados.rados.0.smithi045.stdout:3794: left oid 4 (ObjNum 1156 snap 371 seq_num 1156) 2021-07-31T01:58:17.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:16 smithi154 conmon[34553]: audit 2021-07-31T01:58:15.111318+0000 mgr.x (mgr 2021-07-31T01:58:17.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:16 smithi154 conmon[34553]: .24542) 469 : audit [DBG] from='client.15096 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:58:17.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:16 smithi154 conmon[34553]: cluster 2021-07-31T01:58:15.697934+0000 mon.a (mon.0) 1450 : cluster [DBG] osdmap e779: 8 total, 8 up, 8 in 2021-07-31T01:58:17.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:16 smithi154 conmon[34553]: cluster 2021-07-31T01:58:17.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:16 smithi154 conmon[34553]: 2021-07-31T01:58:15.950631 2021-07-31T01:58:17.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:16 smithi154 conmon[34553]: +0000 mgr.x (mgr.24542) 470 : cluster 2021-07-31T01:58:17.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:16 smithi154 conmon[34553]: [DBG] pgmap v957: 25 pgs: 25 active+clean; 685 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 3.5 MiB/s rd, 1.6 MiB/s wr, 13 op/s 2021-07-31T01:58:17.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:16 smithi045 conmon[33655]: audit 2021-07-31T01:58:17.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:16 smithi045 conmon[33655]: 2021-07-31T01:58: 2021-07-31T01:58:17.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:16 smithi045 conmon[33655]: 15.111318+0000 mgr.x (mgr.24542) 469 : audit [DBG] from='client.15096 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:58:17.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:16 smithi045 conmon[33655]: cluster 2021-07-31T01:58 2021-07-31T01:58:17.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:16 smithi045 conmon[33655]: :15.697934+0000 mon.a (mon.0) 1450 : cluster [DBG] osdmap e779: 8 total, 8 up, 8 in 2021-07-31T01:58:17.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:16 smithi045 conmon[33655]: cluster 2021-07-31T01:58:15.950631 2021-07-31T01:58:17.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:16 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 470 : cluster [DBG] pgmap v957: 25 pgs: 25 active+clean; 685 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 3.5 MiB/s rd, 1.6 MiB/s wr, 13 op/s 2021-07-31T01:58:17.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:16 smithi045 conmon[39718]: audit 2021-07-31T01:58:17.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:16 smithi045 conmon[39718]: 2021-07-31T01:58:17.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:16 smithi045 conmon[39718]: 2021 2021-07-31T01:58:17.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:16 smithi045 conmon[39718]: - 2021-07-31T01:58:17.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:16 smithi045 conmon[39718]: 07 2021-07-31T01:58:17.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:16 smithi045 conmon[39718]: - 2021-07-31T01:58:17.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:16 smithi045 conmon[39718]: 31 2021-07-31T01:58:17.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:16 smithi045 conmon[39718]: T 2021-07-31T01:58:17.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:16 smithi045 conmon[39718]: 01 2021-07-31T01:58:17.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:16 smithi045 conmon[39718]: :58:15.111318+0000 mgr.x (mgr.24542) 469 2021-07-31T01:58:17.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:16 smithi045 conmon[39718]: : audit [DBG] from='client.15096 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:58:17.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:16 smithi045 conmon[39718]: cluster 2021-07-31T01:58:17.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:16 smithi045 conmon[39718]: 2021-07-31T01:58:15.697934+0000 mon.a (mon.0) 1450 : cluster [DBG] 2021-07-31T01:58:17.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:16 smithi045 conmon[39718]: osdmap e779: 8 total, 8 up, 8 in 2021-07-31T01:58:17.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:16 smithi045 conmon[39718]: cluster 2021-07-31T01:58:17.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:16 smithi045 conmon[39718]: 2021-07-31T01:58:15 2021-07-31T01:58:17.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:16 smithi045 conmon[39718]: .950631+0000 mgr.x (mgr.24542) 470 : cluster [DBG] pgmap v957: 25 pgs: 25 active+clean; 685 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 3.5 MiB/s rd, 1.6 MiB/s wr, 13 op/s 2021-07-31T01:58:17.717 INFO:tasks.rados.rados.0.smithi045.stdout:3789: done (3 left) 2021-07-31T01:58:17.718 INFO:tasks.rados.rados.0.smithi045.stdout:3793: done (2 left) 2021-07-31T01:58:17.718 INFO:tasks.rados.rados.0.smithi045.stdout:3794: done (1 left) 2021-07-31T01:58:17.719 INFO:tasks.rados.rados.0.smithi045.stdout:3795: done (0 left) 2021-07-31T01:58:17.719 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 25 current snap is 372 2021-07-31T01:58:17.719 INFO:tasks.rados.rados.0.smithi045.stdout:3796: seq_num 1157 ranges {0=180224} 2021-07-31T01:58:17.720 INFO:tasks.rados.rados.0.smithi045.stdout:3796: writing smithi04569269-25 from 0 to 180224 tid 1 2021-07-31T01:58:17.720 INFO:tasks.rados.rados.0.smithi045.stdout:3797: read oid 17 snap -1 2021-07-31T01:58:17.721 INFO:tasks.rados.rados.0.smithi045.stdout:3797: expect (ObjNum 1110 snap 351 seq_num 1110) 2021-07-31T01:58:17.721 INFO:tasks.rados.rados.0.smithi045.stdout:3798: snap_remove snap 364 2021-07-31T01:58:18.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:17 smithi154 conmon[34553]: cluster 2021-07-31T01:58:16.712159+0000 mon.a (mon.0) 1451 : cluster [DBG] osdmap e780: 8 total, 8 up, 8 in 2021-07-31T01:58:18.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:17 smithi045 conmon[39718]: cluster 2021-07-31T01:58:16.712159+0000 mon.a ( 2021-07-31T01:58:18.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:17 smithi045 conmon[39718]: mon.0) 1451 : cluster [DBG] osdmap e780: 8 total, 8 up, 8 in 2021-07-31T01:58:18.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:17 smithi045 conmon[33655]: cluster 2021-07-31T01:58:16.712159+0000 mon.a (mon.0) 1451 : cluster [DBG] 2021-07-31T01:58:18.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:17 smithi045 conmon[33655]: osdmap e780: 8 total, 8 up, 8 in 2021-07-31T01:58:18.724 INFO:tasks.rados.rados.0.smithi045.stdout:3798: done (2 left) 2021-07-31T01:58:18.725 INFO:tasks.rados.rados.0.smithi045.stdout:3799: read oid 44 snap -1 2021-07-31T01:58:18.725 INFO:tasks.rados.rados.0.smithi045.stdout:3799: expect (ObjNum 955 snap 289 seq_num 955) 2021-07-31T01:58:18.725 INFO:tasks.rados.rados.0.smithi045.stdout:3800: setattr oid 12 current snap is 372 2021-07-31T01:58:18.725 INFO:tasks.rados.rados.0.smithi045.stdout:3796: finishing write tid 1 to smithi04569269-25 2021-07-31T01:58:18.726 INFO:tasks.rados.rados.0.smithi045.stdout:3796: finishing write tid 2 to smithi04569269-25 2021-07-31T01:58:18.726 INFO:tasks.rados.rados.0.smithi045.stdout:3796: finishing write tid 3 to smithi04569269-25 2021-07-31T01:58:18.726 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 25 v 1855 (ObjNum 1157 snap 372 seq_num 1157) dirty exists 2021-07-31T01:58:18.726 INFO:tasks.rados.rados.0.smithi045.stdout:3796: left oid 25 (ObjNum 1157 snap 372 seq_num 1157) 2021-07-31T01:58:18.727 INFO:tasks.rados.rados.0.smithi045.stdout:3797: expect (ObjNum 1110 snap 351 seq_num 1110) 2021-07-31T01:58:18.989 INFO:tasks.rados.rados.0.smithi045.stdout:3799: expect (ObjNum 955 snap 289 seq_num 955) 2021-07-31T01:58:19.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:18 smithi154 conmon[34553]: cluster 2021-07-31T01:58:17.717105+0000 mon.a ( 2021-07-31T01:58:19.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:18 smithi154 conmon[34553]: mon.0) 1452 : cluster [DBG] osdmap e781: 8 total, 8 up, 8 in 2021-07-31T01:58:19.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:18 smithi154 conmon[34553]: cluster 2021-07-31T01:58:17.950996+0000 mgr.x (mgr.24542) 471 : cluster [DBG] pgmap v960: 25 pgs: 25 active+clean; 685 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2021-07-31T01:58:19.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:18 smithi045 conmon[33655]: cluster 2021-07-31T01:58:17.717105+0000 mon.a (mon.0) 1452 : cluster 2021-07-31T01:58:19.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:18 smithi045 conmon[33655]: [DBG] osdmap e781: 8 total, 8 up, 8 in 2021-07-31T01:58:19.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:18 smithi045 conmon[33655]: cluster 2021-07-31T01:58:17.950996+0000 mgr.x (mgr.24542) 471 : cluster [DBG] pgmap v960: 25 pgs: 25 active+clean; 685 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2021-07-31T01:58:19.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:18 smithi045 conmon[39718]: cluster 2021-07-31T01:58:17.717105+0000 mon.a (mon 2021-07-31T01:58:19.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:18 smithi045 conmon[39718]: .0) 1452 : cluster [DBG] osdmap e781: 8 total, 8 up, 8 in 2021-07-31T01:58:19.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:18 smithi045 conmon[39718]: cluster 2021-07-31T01:58: 2021-07-31T01:58:19.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:18 smithi045 conmon[39718]: 17.950996+0000 mgr.x (mgr.24542) 471 : cluster [DBG] pgmap v960: 25 pgs: 25 active+clean; 685 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2021-07-31T01:58:19.107 INFO:tasks.rados.rados.0.smithi045.stdout:3796: done (3 left) 2021-07-31T01:58:19.108 INFO:tasks.rados.rados.0.smithi045.stdout:3797: done (2 left) 2021-07-31T01:58:19.108 INFO:tasks.rados.rados.0.smithi045.stdout:3799: done (1 left) 2021-07-31T01:58:19.108 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 32 current snap is 372 2021-07-31T01:58:19.108 INFO:tasks.rados.rados.0.smithi045.stdout:3801: seq_num 1158 ranges {11132928=196608} 2021-07-31T01:58:19.169 INFO:tasks.rados.rados.0.smithi045.stdout:3801: writing smithi04569269-32 from 11132928 to 11329536 tid 1 2021-07-31T01:58:19.170 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 46 current snap is 372 2021-07-31T01:58:19.170 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 12 v 2369 (ObjNum -1104372512 snap 21873 seq_num 0) dirty exists 2021-07-31T01:58:19.170 INFO:tasks.rados.rados.0.smithi045.stdout:3802: seq_num 1159 ranges {3727360=212992} 2021-07-31T01:58:19.191 INFO:tasks.rados.rados.0.smithi045.stdout:3802: writing smithi04569269-46 from 3727360 to 3940352 tid 1 2021-07-31T01:58:19.191 INFO:tasks.rados.rados.0.smithi045.stdout:3800: done (2 left) 2021-07-31T01:58:19.191 INFO:tasks.rados.rados.0.smithi045.stdout:3803: copy_from oid 20 from oid 16 current snap is 372 2021-07-31T01:58:19.192 INFO:tasks.rados.rados.0.smithi045.stdout:3801: finishing write tid 1 to smithi04569269-32 2021-07-31T01:58:19.192 INFO:tasks.rados.rados.0.smithi045.stdout:3801: finishing write tid 2 to smithi04569269-32 2021-07-31T01:58:19.192 INFO:tasks.rados.rados.0.smithi045.stdout:3801: finishing write tid 3 to smithi04569269-32 2021-07-31T01:58:19.192 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 32 v 1885 (ObjNum 1158 snap 372 seq_num 1158) dirty exists 2021-07-31T01:58:19.192 INFO:tasks.rados.rados.0.smithi045.stdout:3801: left oid 32 (ObjNum 1158 snap 372 seq_num 1158) 2021-07-31T01:58:19.193 INFO:tasks.rados.rados.0.smithi045.stdout:3801: done (2 left) 2021-07-31T01:58:19.193 INFO:tasks.rados.rados.0.smithi045.stdout:3804: snap_create 2021-07-31T01:58:19.194 INFO:tasks.rados.rados.0.smithi045.stdout:3802: finishing write tid 1 to smithi04569269-46 2021-07-31T01:58:19.194 INFO:tasks.rados.rados.0.smithi045.stdout:3802: finishing write tid 2 to smithi04569269-46 2021-07-31T01:58:19.195 INFO:tasks.rados.rados.0.smithi045.stdout:3802: finishing write tid 3 to smithi04569269-46 2021-07-31T01:58:19.196 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 46 v 1863 (ObjNum 1159 snap 372 seq_num 1159) dirty exists 2021-07-31T01:58:19.196 INFO:tasks.rados.rados.0.smithi045.stdout:3802: left oid 46 (ObjNum 1159 snap 372 seq_num 1159) 2021-07-31T01:58:19.200 INFO:tasks.rados.rados.0.smithi045.stdout:3803: finishing copy_from to smithi04569269-20 2021-07-31T01:58:19.201 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 20 v 1859 (ObjNum 944 snap 286 seq_num 944) dirty exists 2021-07-31T01:58:19.202 INFO:tasks.rados.rados.0.smithi045.stdout:3803: finishing copy_from racing read to smithi04569269-20 2021-07-31T01:58:19.726 INFO:tasks.rados.rados.0.smithi045.stdout:3802: done (2 left) 2021-07-31T01:58:19.727 INFO:tasks.rados.rados.0.smithi045.stdout:3803: done (1 left) 2021-07-31T01:58:19.727 INFO:tasks.rados.rados.0.smithi045.stdout:3804: done (0 left) 2021-07-31T01:58:19.727 INFO:tasks.rados.rados.0.smithi045.stdout:3805: read oid 12 snap -1 2021-07-31T01:58:19.727 INFO:tasks.rados.rados.0.smithi045.stdout:3805: expect (ObjNum 0 snap 0 seq_num 0) 2021-07-31T01:58:19.728 INFO:tasks.rados.rados.0.smithi045.stdout:3806: read oid 10 snap -1 2021-07-31T01:58:19.728 INFO:tasks.rados.rados.0.smithi045.stdout:3806: expect (ObjNum 1128 snap 357 seq_num 1128) 2021-07-31T01:58:19.728 INFO:tasks.rados.rados.0.smithi045.stdout:3807: read oid 25 snap -1 2021-07-31T01:58:19.728 INFO:tasks.rados.rados.0.smithi045.stdout:3807: expect (ObjNum 1157 snap 372 seq_num 1157) 2021-07-31T01:58:19.728 INFO:tasks.rados.rados.0.smithi045.stdout:3808: delete oid 41 current snap is 373 2021-07-31T01:58:19.730 INFO:tasks.rados.rados.0.smithi045.stdout:3805: expect (ObjNum 0 snap 0 seq_num 0) 2021-07-31T01:58:19.731 INFO:tasks.rados.rados.0.smithi045.stdout:3807: expect (ObjNum 1157 snap 372 seq_num 1157) 2021-07-31T01:58:19.736 INFO:tasks.rados.rados.0.smithi045.stdout:3805: done (3 left) 2021-07-31T01:58:19.737 INFO:tasks.rados.rados.0.smithi045.stdout:3807: done (2 left) 2021-07-31T01:58:19.737 INFO:tasks.rados.rados.0.smithi045.stdout:3808: done (1 left) 2021-07-31T01:58:19.737 INFO:tasks.rados.rados.0.smithi045.stdout:3809: read oid 22 snap -1 2021-07-31T01:58:19.737 INFO:tasks.rados.rados.0.smithi045.stdout:3809: expect (ObjNum 958 snap 293 seq_num 958) 2021-07-31T01:58:19.738 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 24 current snap is 373 2021-07-31T01:58:19.738 INFO:tasks.rados.rados.0.smithi045.stdout:3810: seq_num 1161 ranges {0=245760} 2021-07-31T01:58:19.739 INFO:tasks.rados.rados.0.smithi045.stdout:3810: writing smithi04569269-24 from 0 to 245760 tid 1 2021-07-31T01:58:19.739 INFO:tasks.rados.rados.0.smithi045.stdout:3811: rmattr oid 23 current snap is 373 2021-07-31T01:58:19.739 INFO:tasks.rados.rados.0.smithi045.stdout:3811: done (3 left) 2021-07-31T01:58:19.740 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 3 current snap is 373 2021-07-31T01:58:19.740 INFO:tasks.rados.rados.0.smithi045.stdout:3812: seq_num 1162 ranges {9584640=270336} 2021-07-31T01:58:19.792 INFO:tasks.rados.rados.0.smithi045.stdout:3812: writing smithi04569269-3 from 9584640 to 9854976 tid 1 2021-07-31T01:58:19.793 INFO:tasks.rados.rados.0.smithi045.stdout:3813: read oid 9 snap -1 2021-07-31T01:58:19.793 INFO:tasks.rados.rados.0.smithi045.stdout:3813: expect (ObjNum 990 snap 307 seq_num 990) 2021-07-31T01:58:19.794 INFO:tasks.rados.rados.0.smithi045.stdout:3814: setattr oid 47 current snap is 373 2021-07-31T01:58:19.794 INFO:tasks.rados.rados.0.smithi045.stdout:3806: expect (ObjNum 1128 snap 357 seq_num 1128) 2021-07-31T01:58:19.849 INFO:tasks.rados.rados.0.smithi045.stdout:3810: finishing write tid 1 to smithi04569269-24 2021-07-31T01:58:19.849 INFO:tasks.rados.rados.0.smithi045.stdout:3810: finishing write tid 2 to smithi04569269-24 2021-07-31T01:58:19.849 INFO:tasks.rados.rados.0.smithi045.stdout:3809: expect (ObjNum 958 snap 293 seq_num 958) 2021-07-31T01:58:19.892 INFO:tasks.rados.rados.0.smithi045.stdout:3810: finishing write tid 3 to smithi04569269-24 2021-07-31T01:58:19.892 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 24 v 1673 (ObjNum 1161 snap 373 seq_num 1161) dirty exists 2021-07-31T01:58:19.892 INFO:tasks.rados.rados.0.smithi045.stdout:3810: left oid 24 (ObjNum 1161 snap 373 seq_num 1161) 2021-07-31T01:58:19.893 INFO:tasks.rados.rados.0.smithi045.stdout:3812: finishing write tid 1 to smithi04569269-3 2021-07-31T01:58:19.893 INFO:tasks.rados.rados.0.smithi045.stdout:3812: finishing write tid 2 to smithi04569269-3 2021-07-31T01:58:19.893 INFO:tasks.rados.rados.0.smithi045.stdout:3812: finishing write tid 3 to smithi04569269-3 2021-07-31T01:58:19.893 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 3 v 2838 (ObjNum 1162 snap 373 seq_num 1162) dirty exists 2021-07-31T01:58:19.894 INFO:tasks.rados.rados.0.smithi045.stdout:3812: left oid 3 (ObjNum 1162 snap 373 seq_num 1162) 2021-07-31T01:58:19.894 INFO:tasks.rados.rados.0.smithi045.stdout:3813: expect (ObjNum 990 snap 307 seq_num 990) 2021-07-31T01:58:20.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:19 smithi154 conmon[34553]: cluster 2021-07-31T01:58:18.723450+0000 mon.a (mon.0) 2021-07-31T01:58:20.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:19 smithi154 conmon[34553]: 1453 : cluster [DBG] osdmap e782: 8 total, 8 up, 8 in 2021-07-31T01:58:20.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:19 smithi045 conmon[33655]: cluster 2021-07-31T01:58:18.723450+0000 mon.a (mon.0) 2021-07-31T01:58:20.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:19 smithi045 conmon[33655]: 1453 : cluster [DBG] osdmap e782: 8 total, 8 up, 8 in 2021-07-31T01:58:20.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:19 smithi045 conmon[39718]: cluster 2021-07-31T01:58:18.723450 2021-07-31T01:58:20.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:19 smithi045 conmon[39718]: +0000 mon.a (mon.0) 1453 : cluster [DBG] osdmap e782: 8 total, 8 up, 8 in 2021-07-31T01:58:21.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:20 smithi154 conmon[34553]: cluster 2021-07-31T01:58:19.725264+0000 mon.a (mon.0) 1454 : cluster [DBG] osdmap e783: 8 total, 8 up, 8 in 2021-07-31T01:58:21.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:20 smithi154 conmon[34553]: 2021-07-31T01:58:21.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:20 smithi154 conmon[34553]: cluster 2021-07-31T01:58:19.951547+0000 mgr.x (mgr.24542) 472 : cluster [DBG] pgmap v963: 25 pgs: 25 active+clean; 685 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 10 MiB/s rd, 84 KiB/s wr, 5 op/s 2021-07-31T01:58:21.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:20 smithi045 conmon[33655]: cluster 2021-07-31T01:58 2021-07-31T01:58:21.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:20 smithi045 conmon[33655]: :19.725264+0000 mon.a (mon.0) 1454 : cluster [DBG] osdmap e783: 8 total, 8 up, 8 in 2021-07-31T01:58:21.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:20 smithi045 conmon[33655]: cluster 2021-07-31T01:58:21.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:20 smithi045 conmon[33655]: 2021-07-31T01:58:19.951547+0000 mgr.x ( 2021-07-31T01:58:21.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:20 smithi045 conmon[33655]: mgr.24542) 472 : cluster [DBG] pgmap v963: 25 pgs: 25 active+clean; 685 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 10 MiB/s rd, 84 KiB/s wr, 5 op/s 2021-07-31T01:58:21.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:20 smithi045 conmon[39718]: cluster 2021-07-31T01:58:19. 2021-07-31T01:58:21.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:20 smithi045 conmon[39718]: 725264+0000 mon.a (mon.0) 1454 : cluster [DBG] osdmap e783: 8 total, 8 up, 8 in 2021-07-31T01:58:21.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:20 smithi045 conmon[39718]: cluster 2021-07-31T01:58:21.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:20 smithi045 conmon[39718]: 2021-07-31T01:58:19.951547+0000 mgr.x 2021-07-31T01:58:21.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:20 smithi045 conmon[39718]: (mgr.24542) 472 : cluster [DBG] pgmap v963: 25 pgs: 25 active+clean; 685 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 10 MiB/s rd, 84 KiB/s wr, 5 op/s 2021-07-31T01:58:21.128 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 47 v 1856 (ObjNum 1031 snap 335 seq_num 1031) dirty exists 2021-07-31T01:58:21.128 INFO:tasks.rados.rados.0.smithi045.stdout:3806: done (5 left) 2021-07-31T01:58:21.128 INFO:tasks.rados.rados.0.smithi045.stdout:3809: done (4 left) 2021-07-31T01:58:21.128 INFO:tasks.rados.rados.0.smithi045.stdout:3810: done (3 left) 2021-07-31T01:58:21.129 INFO:tasks.rados.rados.0.smithi045.stdout:3812: done (2 left) 2021-07-31T01:58:21.129 INFO:tasks.rados.rados.0.smithi045.stdout:3813: done (1 left) 2021-07-31T01:58:21.129 INFO:tasks.rados.rados.0.smithi045.stdout:3814: done (0 left) 2021-07-31T01:58:21.129 INFO:tasks.rados.rados.0.smithi045.stdout:3815: snap_remove snap 351 2021-07-31T01:58:21.691 INFO:tasks.rados.rados.0.smithi045.stdout:3815: done (0 left) 2021-07-31T01:58:21.692 INFO:tasks.rados.rados.0.smithi045.stdout:3816: snap_remove snap 368 2021-07-31T01:58:22.699 INFO:tasks.rados.rados.0.smithi045.stdout:3816: done (0 left) 2021-07-31T01:58:22.700 INFO:tasks.rados.rados.0.smithi045.stdout:3817: read oid 22 snap 370 2021-07-31T01:58:22.700 INFO:tasks.rados.rados.0.smithi045.stdout:3817: expect (ObjNum 958 snap 293 seq_num 958) 2021-07-31T01:58:22.700 INFO:tasks.rados.rados.0.smithi045.stdout:3818: read oid 41 snap -1 2021-07-31T01:58:22.700 INFO:tasks.rados.rados.0.smithi045.stdout:3818: expect deleted 2021-07-31T01:58:22.701 INFO:tasks.rados.rados.0.smithi045.stdout:3819: snap_create 2021-07-31T01:58:22.707 INFO:tasks.rados.rados.0.smithi045.stdout:3817: expect (ObjNum 958 snap 293 seq_num 958) 2021-07-31T01:58:23.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:22 smithi154 conmon[34553]: cluster 2021-07-31T01:58:21.690470+0000 mon.a (mon.0) 1455 : cluster 2021-07-31T01:58:23.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:22 smithi154 conmon[34553]: [DBG] osdmap e784: 8 total, 8 up, 8 in 2021-07-31T01:58:23.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:22 smithi154 conmon[34553]: cluster 2021-07-31T01:58:21.952126+0000 mgr.x (mgr.24542) 473 : cluster [DBG] pgmap v965: 25 pgs: 25 active+clean; 685 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 9.6 MiB/s rd, 79 KiB/s wr, 5 op/s 2021-07-31T01:58:23.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:22 smithi045 conmon[39718]: cluster 2021-07-31T01:58:21.690470+0000 mon.a (mon.0) 1455 : cluster 2021-07-31T01:58:23.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:22 smithi045 conmon[39718]: [DBG] osdmap e784: 8 total, 8 up, 8 in 2021-07-31T01:58:23.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:22 smithi045 conmon[39718]: cluster 2021-07-31T01:58:21.952126+0000 mgr.x (mgr.24542) 473 : cluster [DBG] pgmap v965: 25 pgs: 25 active+clean; 685 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 9.6 MiB/s rd, 79 KiB/s wr, 5 op/s 2021-07-31T01:58:23.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:22 smithi045 conmon[33655]: cluster 2021-07-31T01:58:21.690470+0000 mon.a (mon.0) 1455 : cluster [DBG] osdmap e784: 8 total, 8 up, 8 in 2021-07-31T01:58:23.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:22 smithi045 conmon[33655]: cluster 2021-07-31T01:58:21.952126+0000 mgr.x (mgr.24542) 473 : cluster [DBG] pgmap v965: 25 pgs: 25 active+clean; 685 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 9.6 MiB/s rd, 79 KiB/s wr, 5 op/s 2021-07-31T01:58:23.706 INFO:tasks.rados.rados.0.smithi045.stdout:3817: done (2 left) 2021-07-31T01:58:23.706 INFO:tasks.rados.rados.0.smithi045.stdout:3818: done (1 left) 2021-07-31T01:58:23.707 INFO:tasks.rados.rados.0.smithi045.stdout:3819: done (0 left) 2021-07-31T01:58:23.708 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 35 current snap is 374 2021-07-31T01:58:23.708 INFO:tasks.rados.rados.0.smithi045.stdout:3820: seq_num 1163 ranges {1007616=286720} 2021-07-31T01:58:23.713 INFO:tasks.rados.rados.0.smithi045.stdout:3820: writing smithi04569269-35 from 1007616 to 1294336 tid 1 2021-07-31T01:58:23.713 INFO:tasks.rados.rados.0.smithi045.stdout:3821: delete oid 41 current snap is 374 2021-07-31T01:58:23.715 INFO:tasks.rados.rados.0.smithi045.stdout:3821: done (1 left) 2021-07-31T01:58:23.716 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 14 current snap is 374 2021-07-31T01:58:23.716 INFO:tasks.rados.rados.0.smithi045.stdout:3822: seq_num 1165 ranges {0=319488} 2021-07-31T01:58:23.717 INFO:tasks.rados.rados.0.smithi045.stdout:3822: writing smithi04569269-14 from 0 to 319488 tid 1 2021-07-31T01:58:23.718 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 17 current snap is 374 2021-07-31T01:58:23.718 INFO:tasks.rados.rados.0.smithi045.stdout:3823: seq_num 1166 ranges {5373952=335872} 2021-07-31T01:58:23.749 INFO:tasks.rados.rados.0.smithi045.stdout:3823: writing smithi04569269-17 from 5373952 to 5709824 tid 1 2021-07-31T01:58:23.750 INFO:tasks.rados.rados.0.smithi045.stdout:3824: delete oid 23 current snap is 374 2021-07-31T01:58:23.750 INFO:tasks.rados.rados.0.smithi045.stdout:3820: finishing write tid 1 to smithi04569269-35 2021-07-31T01:58:23.750 INFO:tasks.rados.rados.0.smithi045.stdout:3820: finishing write tid 2 to smithi04569269-35 2021-07-31T01:58:23.751 INFO:tasks.rados.rados.0.smithi045.stdout:3820: finishing write tid 3 to smithi04569269-35 2021-07-31T01:58:23.751 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 35 v 2859 (ObjNum 1163 snap 374 seq_num 1163) dirty exists 2021-07-31T01:58:23.751 INFO:tasks.rados.rados.0.smithi045.stdout:3820: left oid 35 (ObjNum 1163 snap 374 seq_num 1163) 2021-07-31T01:58:23.751 INFO:tasks.rados.rados.0.smithi045.stdout:3822: finishing write tid 1 to smithi04569269-14 2021-07-31T01:58:23.752 INFO:tasks.rados.rados.0.smithi045.stdout:3822: finishing write tid 2 to smithi04569269-14 2021-07-31T01:58:23.752 INFO:tasks.rados.rados.0.smithi045.stdout:3822: finishing write tid 3 to smithi04569269-14 2021-07-31T01:58:23.752 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 14 v 2096 (ObjNum 1165 snap 374 seq_num 1165) dirty exists 2021-07-31T01:58:23.753 INFO:tasks.rados.rados.0.smithi045.stdout:3822: left oid 14 (ObjNum 1165 snap 374 seq_num 1165) 2021-07-31T01:58:23.754 INFO:tasks.rados.rados.0.smithi045.stdout:3820: done (3 left) 2021-07-31T01:58:23.754 INFO:tasks.rados.rados.0.smithi045.stdout:3822: done (2 left) 2021-07-31T01:58:23.754 INFO:tasks.rados.rados.0.smithi045.stdout:3824: done (1 left) 2021-07-31T01:58:23.755 INFO:tasks.rados.rados.0.smithi045.stdout:3825: read oid 42 snap -1 2021-07-31T01:58:23.755 INFO:tasks.rados.rados.0.smithi045.stdout:3825: expect (ObjNum 1137 snap 361 seq_num 1137) 2021-07-31T01:58:23.755 INFO:tasks.rados.rados.0.smithi045.stdout:3826: snap_remove snap 372 2021-07-31T01:58:24.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:23 smithi154 conmon[34553]: cluster 2021-07-31T01:58:22.699093+0000 mon.a (mon.0) 1456 : cluster [DBG] osdmap e785: 8 total, 8 up, 8 in 2021-07-31T01:58:24.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:23 smithi045 conmon[33655]: cluster 2021-07-31T01:58:22.699093+0000 mon.a (mon.0) 2021-07-31T01:58:24.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:23 smithi045 conmon[33655]: 1456 : cluster [DBG] osdmap e785: 8 total, 8 up, 8 in 2021-07-31T01:58:24.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:23 smithi045 conmon[39718]: cluster 2021-07-31T01:58:22.699093+0000 mon.a (mon.0) 1456 : cluster [DBG] 2021-07-31T01:58:24.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:23 smithi045 conmon[39718]: osdmap e785: 8 total, 8 up, 8 in 2021-07-31T01:58:24.027 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:58:23 smithi045 conmon[61210]: level=error ts=2021-07-31T01:58:23.528Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:58:24.028 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:58:23 smithi045 conmon[61210]: level=error ts=2021-07-31T01:58:23.528Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:58:24.028 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:58:23 smithi045 conmon[61210]: level=error ts=2021-07-31T01:58:23.528Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T01:58:24.712 INFO:tasks.rados.rados.0.smithi045.stdout:3826: done (2 left) 2021-07-31T01:58:24.712 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 13 current snap is 374 2021-07-31T01:58:24.713 INFO:tasks.rados.rados.0.smithi045.stdout:3823: finishing write tid 1 to smithi04569269-17 2021-07-31T01:58:24.713 INFO:tasks.rados.rados.0.smithi045.stdout:3827: seq_num 1168 ranges {0=368640} 2021-07-31T01:58:24.714 INFO:tasks.rados.rados.0.smithi045.stdout:3827: writing smithi04569269-13 from 0 to 368640 tid 1 2021-07-31T01:58:24.715 INFO:tasks.rados.rados.0.smithi045.stdout:3828: delete oid 1 current snap is 374 2021-07-31T01:58:24.715 INFO:tasks.rados.rados.0.smithi045.stdout:3823: finishing write tid 2 to smithi04569269-17 2021-07-31T01:58:24.715 INFO:tasks.rados.rados.0.smithi045.stdout:3823: finishing write tid 3 to smithi04569269-17 2021-07-31T01:58:24.715 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 17 v 2381 (ObjNum 1166 snap 374 seq_num 1166) dirty exists 2021-07-31T01:58:24.715 INFO:tasks.rados.rados.0.smithi045.stdout:3823: left oid 17 (ObjNum 1166 snap 374 seq_num 1166) 2021-07-31T01:58:24.716 INFO:tasks.rados.rados.0.smithi045.stdout:3825: expect (ObjNum 1137 snap 361 seq_num 1137) 2021-07-31T01:58:24.803 INFO:tasks.rados.rados.0.smithi045.stdout:3827: finishing write tid 1 to smithi04569269-13 2021-07-31T01:58:24.804 INFO:tasks.rados.rados.0.smithi045.stdout:3827: finishing write tid 2 to smithi04569269-13 2021-07-31T01:58:24.804 INFO:tasks.rados.rados.0.smithi045.stdout:3827: finishing write tid 3 to smithi04569269-13 2021-07-31T01:58:24.804 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 13 v 2371 (ObjNum 1168 snap 374 seq_num 1168) dirty exists 2021-07-31T01:58:24.804 INFO:tasks.rados.rados.0.smithi045.stdout:3827: left oid 13 (ObjNum 1168 snap 374 seq_num 1168) 2021-07-31T01:58:24.804 INFO:tasks.rados.rados.0.smithi045.stdout:3823: done (3 left) 2021-07-31T01:58:24.805 INFO:tasks.rados.rados.0.smithi045.stdout:3825: done (2 left) 2021-07-31T01:58:24.805 INFO:tasks.rados.rados.0.smithi045.stdout:3827: done (1 left) 2021-07-31T01:58:24.805 INFO:tasks.rados.rados.0.smithi045.stdout:3828: done (0 left) 2021-07-31T01:58:24.805 INFO:tasks.rados.rados.0.smithi045.stdout:3829: read oid 12 snap -1 2021-07-31T01:58:24.806 INFO:tasks.rados.rados.0.smithi045.stdout:3829: expect (ObjNum 0 snap 0 seq_num 0) 2021-07-31T01:58:24.806 INFO:tasks.rados.rados.0.smithi045.stdout:3830: read oid 5 snap -1 2021-07-31T01:58:24.806 INFO:tasks.rados.rados.0.smithi045.stdout:3830: expect (ObjNum 1136 snap 360 seq_num 1136) 2021-07-31T01:58:24.806 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 27 current snap is 374 2021-07-31T01:58:24.806 INFO:tasks.rados.rados.0.smithi045.stdout:3831: seq_num 1170 ranges {0=401408} 2021-07-31T01:58:24.807 INFO:tasks.rados.rados.0.smithi045.stdout:3831: writing smithi04569269-27 from 0 to 401408 tid 1 2021-07-31T01:58:24.808 INFO:tasks.rados.rados.0.smithi045.stdout:3832: read oid 15 snap 325 2021-07-31T01:58:24.808 INFO:tasks.rados.rados.0.smithi045.stdout:3832: expect deleted 2021-07-31T01:58:24.808 INFO:tasks.rados.rados.0.smithi045.stdout:3833: snap_create 2021-07-31T01:58:24.808 INFO:tasks.rados.rados.0.smithi045.stdout:3829: expect (ObjNum 0 snap 0 seq_num 0) 2021-07-31T01:58:24.811 INFO:tasks.rados.rados.0.smithi045.stdout:3831: finishing write tid 1 to smithi04569269-27 2021-07-31T01:58:24.812 INFO:tasks.rados.rados.0.smithi045.stdout:3831: finishing write tid 2 to smithi04569269-27 2021-07-31T01:58:24.814 INFO:tasks.rados.rados.0.smithi045.stdout:3831: finishing write tid 3 to smithi04569269-27 2021-07-31T01:58:24.814 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 27 v 2098 (ObjNum 1170 snap 374 seq_num 1170) dirty exists 2021-07-31T01:58:24.814 INFO:tasks.rados.rados.0.smithi045.stdout:3831: left oid 27 (ObjNum 1170 snap 374 seq_num 1170) 2021-07-31T01:58:24.825 INFO:tasks.rados.rados.0.smithi045.stdout:3830: expect (ObjNum 1136 snap 360 seq_num 1136) 2021-07-31T01:58:25.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:24 smithi154 conmon[34553]: cluster 2021-07-31T01:58:23.705492+0000 mon.a (mon.0) 1457 : cluster [DBG] osdmap e786: 8 total, 8 up, 8 in 2021-07-31T01:58:25.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:24 smithi154 conmon[34553]: cluster 2021-07-31T01:58:23.952623+0000 mgr.x (mgr.24542) 474 : cluster [DBG] pgmap v968: 25 pgs: 25 active+clean; 688 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 14 MiB/s rd, 759 KiB/s wr, 21 op/s 2021-07-31T01:58:25.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:24 smithi045 conmon[33655]: cluster 2021-07-31T01:58:23.705492+0000 mon.a (mon.0) 1457 2021-07-31T01:58:25.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:24 smithi045 conmon[33655]: : cluster [DBG] osdmap e786: 8 total, 8 up, 8 in 2021-07-31T01:58:25.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:24 smithi045 conmon[33655]: cluster 2021-07-31T01:58:23.952623+0000 mgr.x (mgr.24542) 474 : cluster [DBG] 2021-07-31T01:58:25.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:24 smithi045 conmon[33655]: pgmap v968: 25 pgs: 25 active+clean; 688 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 14 MiB/s rd, 759 KiB/s wr, 21 op/s 2021-07-31T01:58:25.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:24 smithi045 conmon[39718]: cluster 2021-07-31T01:58:23.705492+0000 mon.a ( 2021-07-31T01:58:25.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:24 smithi045 conmon[39718]: mon.0) 1457 : cluster [DBG] osdmap e786: 8 total, 8 up, 8 in 2021-07-31T01:58:25.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:24 smithi045 conmon[39718]: cluster 2021-07-31T01: 2021-07-31T01:58:25.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:24 smithi045 conmon[39718]: 58:23.952623+0000 mgr.x (mgr.24542) 474 : cluster [DBG] pgmap v968: 25 pgs: 25 active+clean; 688 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 14 MiB/s rd, 759 KiB/s wr, 21 op/s 2021-07-31T01:58:25.718 INFO:tasks.rados.rados.0.smithi045.stdout:3829: done (4 left) 2021-07-31T01:58:25.718 INFO:tasks.rados.rados.0.smithi045.stdout:3830: done (3 left) 2021-07-31T01:58:25.718 INFO:tasks.rados.rados.0.smithi045.stdout:3831: done (2 left) 2021-07-31T01:58:25.719 INFO:tasks.rados.rados.0.smithi045.stdout:3832: done (1 left) 2021-07-31T01:58:25.719 INFO:tasks.rados.rados.0.smithi045.stdout:3833: done (0 left) 2021-07-31T01:58:25.719 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 15 current snap is 375 2021-07-31T01:58:25.719 INFO:tasks.rados.rados.0.smithi045.stdout:3834: seq_num 1171 ranges {3088384=417792} 2021-07-31T01:58:25.737 INFO:tasks.rados.rados.0.smithi045.stdout:3834: writing smithi04569269-15 from 3088384 to 3506176 tid 1 2021-07-31T01:58:25.737 INFO:tasks.rados.rados.0.smithi045.stdout:3835: read oid 5 snap 373 2021-07-31T01:58:25.738 INFO:tasks.rados.rados.0.smithi045.stdout:3835: expect (ObjNum 1136 snap 360 seq_num 1136) 2021-07-31T01:58:25.738 INFO:tasks.rados.rados.0.smithi045.stdout:3836: snap_create 2021-07-31T01:58:25.744 INFO:tasks.rados.rados.0.smithi045.stdout:3834: finishing write tid 1 to smithi04569269-15 2021-07-31T01:58:25.744 INFO:tasks.rados.rados.0.smithi045.stdout:3834: finishing write tid 2 to smithi04569269-15 2021-07-31T01:58:25.745 INFO:tasks.rados.rados.0.smithi045.stdout:3834: finishing write tid 3 to smithi04569269-15 2021-07-31T01:58:25.746 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 15 v 2539 (ObjNum 1171 snap 375 seq_num 1171) dirty exists 2021-07-31T01:58:25.746 INFO:tasks.rados.rados.0.smithi045.stdout:3834: left oid 15 (ObjNum 1171 snap 375 seq_num 1171) 2021-07-31T01:58:25.758 INFO:tasks.rados.rados.0.smithi045.stdout:3835: expect (ObjNum 1136 snap 360 seq_num 1136) 2021-07-31T01:58:26.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:25 smithi154 conmon[34553]: cluster 2021-07-31T01:58:24.711590+0000 mon.a (mon.0) 1458 : cluster [DBG] osdmap e787: 8 total, 8 up, 8 in 2021-07-31T01:58:26.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:25 smithi045 conmon[33655]: cluster 2021-07-31T01:58:24.711590+0000 mon.a (mon.0) 1458 : cluster [DBG] osdmap e787: 8 total, 8 up, 8 in 2021-07-31T01:58:26.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:25 smithi045 conmon[39718]: cluster 2021-07-31T01:58:24.711590+0000 mon.a (mon.0) 1458 : cluster [DBG] osdmap e787: 8 total, 8 up, 8 in 2021-07-31T01:58:26.693 INFO:tasks.rados.rados.0.smithi045.stdout:3834: done (2 left) 2021-07-31T01:58:26.694 INFO:tasks.rados.rados.0.smithi045.stdout:3835: done (1 left) 2021-07-31T01:58:26.694 INFO:tasks.rados.rados.0.smithi045.stdout:3836: done (0 left) 2021-07-31T01:58:26.694 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 24 current snap is 376 2021-07-31T01:58:26.694 INFO:tasks.rados.rados.0.smithi045.stdout:3837: seq_num 1172 ranges {245760=434176} 2021-07-31T01:58:26.697 INFO:tasks.rados.rados.0.smithi045.stdout:3837: writing smithi04569269-24 from 245760 to 679936 tid 1 2021-07-31T01:58:26.698 INFO:tasks.rados.rados.0.smithi045.stdout:3838: rollback oid 9 current snap is 376 2021-07-31T01:58:26.698 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 9 to 375 2021-07-31T01:58:26.698 INFO:tasks.rados.rados.0.smithi045.stdout:3839: snap_remove snap 343 2021-07-31T01:58:27.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:26 smithi154 conmon[34553]: cluster 2021-07-31T01:58:25.717323+0000 mon.a (mon.0) 1459 : cluster [DBG] 2021-07-31T01:58:27.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:26 smithi154 conmon[34553]: osdmap e788: 8 total, 8 up, 8 in 2021-07-31T01:58:27.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:26 smithi154 conmon[34553]: cluster 2021-07-31T01:58:25.953033+0000 mgr.x (mgr.24542) 475 : cluster [DBG] pgmap v971: 25 pgs: 25 active+clean; 688 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 14 MiB/s rd, 802 KiB/s wr, 23 op/s 2021-07-31T01:58:27.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:26 smithi154 conmon[34553]: cluster 2021-07-31T01:58:26.692585+0000 mon.a (mon.0) 1460 : cluster [DBG] osdmap e789: 8 total, 8 up, 8 in 2021-07-31T01:58:27.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:26 smithi045 conmon[33655]: cluster 2021-07-31T01:58:25.717323+0000 mon.a (mon.0) 1459 : cluster [DBG] osdmap e788: 8 total, 8 up, 8 in 2021-07-31T01:58:27.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:26 smithi045 conmon[33655]: cluster 2021-07-31T01:58:25.953033+0000 mgr.x (mgr.24542) 475 : cluster [DBG] pgmap v971: 25 pgs: 25 active+clean; 688 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 14 MiB/s rd, 802 KiB/s wr, 23 op/s 2021-07-31T01:58:27.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:26 smithi045 conmon[33655]: cluster 2021-07-31T01:58:26.692585+0000 mon.a ( 2021-07-31T01:58:27.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:26 smithi045 conmon[33655]: mon.0) 1460 : cluster [DBG] osdmap e789: 8 total, 8 up, 8 in 2021-07-31T01:58:27.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:26 smithi045 conmon[39718]: cluster 2021-07-31T01:58: 2021-07-31T01:58:27.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:26 smithi045 conmon[39718]: 25.717323+0000 mon.a (mon.0) 1459 : cluster [DBG] osdmap e788: 8 total, 8 up, 8 in 2021-07-31T01:58:27.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:26 smithi045 conmon[39718]: cluster 2021-07 2021-07-31T01:58:27.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:26 smithi045 conmon[39718]: -31T01:58:25.953033 2021-07-31T01:58:27.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:26 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 475 : cluster [DBG] 2021-07-31T01:58:27.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:26 smithi045 conmon[39718]: pgmap v971: 25 pgs: 25 active+clean; 688 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 14 MiB/s rd, 802 KiB/s wr, 23 op/s 2021-07-31T01:58:27.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:26 smithi045 conmon[39718]: cluster 2021-07-31T01:58:27.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:26 smithi045 conmon[39718]: 2021-07-31T01 2021-07-31T01:58:27.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:26 smithi045 conmon[39718]: :58:26.692585+0000 2021-07-31T01:58:27.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:26 smithi045 conmon[39718]: mon.a (mon.0) 1460 : cluster 2021-07-31T01:58:27.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:26 smithi045 conmon[39718]: [DBG] osdmap e789: 8 total, 8 up, 8 in 2021-07-31T01:58:27.703 INFO:tasks.rados.rados.0.smithi045.stdout:3839: done (2 left) 2021-07-31T01:58:27.703 INFO:tasks.rados.rados.0.smithi045.stdout:3840: delete oid 22 current snap is 376 2021-07-31T01:58:27.704 INFO:tasks.rados.rados.0.smithi045.stdout:3838: finishing rollback tid 0 to smithi04569269-9 2021-07-31T01:58:27.704 INFO:tasks.rados.rados.0.smithi045.stdout:3837: finishing write tid 1 to smithi04569269-24 2021-07-31T01:58:27.704 INFO:tasks.rados.rados.0.smithi045.stdout:3838: finishing rollback tid 1 to smithi04569269-9 2021-07-31T01:58:27.704 INFO:tasks.rados.rados.0.smithi045.stdout:3837: finishing write tid 2 to smithi04569269-24 2021-07-31T01:58:27.704 INFO:tasks.rados.rados.0.smithi045.stdout:3838: finishing rollback tid 2 to smithi04569269-9 2021-07-31T01:58:27.705 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 9 v 2379 (ObjNum 990 snap 307 seq_num 990) dirty exists 2021-07-31T01:58:27.705 INFO:tasks.rados.rados.0.smithi045.stdout:3837: finishing write tid 3 to smithi04569269-24 2021-07-31T01:58:27.705 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 24 v 1675 (ObjNum 1172 snap 376 seq_num 1172) dirty exists 2021-07-31T01:58:27.705 INFO:tasks.rados.rados.0.smithi045.stdout:3837: left oid 24 (ObjNum 1172 snap 376 seq_num 1172) 2021-07-31T01:58:27.710 INFO:tasks.rados.rados.0.smithi045.stdout:3837: done (2 left) 2021-07-31T01:58:27.710 INFO:tasks.rados.rados.0.smithi045.stdout:3838: done (1 left) 2021-07-31T01:58:27.710 INFO:tasks.rados.rados.0.smithi045.stdout:3840: done (0 left) 2021-07-31T01:58:27.710 INFO:tasks.rados.rados.0.smithi045.stdout:3841: read oid 12 snap -1 2021-07-31T01:58:27.711 INFO:tasks.rados.rados.0.smithi045.stdout:3841: expect (ObjNum 0 snap 0 seq_num 0) 2021-07-31T01:58:27.711 INFO:tasks.rados.rados.0.smithi045.stdout:3842: rmattr oid 17 current snap is 376 2021-07-31T01:58:27.711 INFO:tasks.rados.rados.0.smithi045.stdout:3842: done (1 left) 2021-07-31T01:58:27.711 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 39 current snap is 376 2021-07-31T01:58:27.712 INFO:tasks.rados.rados.0.smithi045.stdout:3843: seq_num 1174 ranges {1941504=466944} 2021-07-31T01:58:27.724 INFO:tasks.rados.rados.0.smithi045.stdout:3843: writing smithi04569269-39 from 1941504 to 2408448 tid 1 2021-07-31T01:58:27.724 INFO:tasks.rados.rados.0.smithi045.stdout:3844: copy_from oid 11 from oid 27 current snap is 376 2021-07-31T01:58:27.724 INFO:tasks.rados.rados.0.smithi045.stdout:3841: expect (ObjNum 0 snap 0 seq_num 0) 2021-07-31T01:58:27.726 INFO:tasks.rados.rados.0.smithi045.stdout:3841: done (2 left) 2021-07-31T01:58:27.727 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 22 current snap is 376 2021-07-31T01:58:27.727 INFO:tasks.rados.rados.0.smithi045.stdout:3845: seq_num 1175 ranges {0=483328} 2021-07-31T01:58:27.730 INFO:tasks.rados.rados.0.smithi045.stdout:3845: writing smithi04569269-22 from 0 to 483328 tid 1 2021-07-31T01:58:27.730 INFO:tasks.rados.rados.0.smithi045.stdout:3846: read oid 21 snap -1 2021-07-31T01:58:27.730 INFO:tasks.rados.rados.0.smithi045.stdout:3846: expect (ObjNum 1108 snap 351 seq_num 1108) 2021-07-31T01:58:27.730 INFO:tasks.rados.rados.0.smithi045.stdout:3847: read oid 46 snap -1 2021-07-31T01:58:27.731 INFO:tasks.rados.rados.0.smithi045.stdout:3847: expect (ObjNum 1159 snap 372 seq_num 1159) 2021-07-31T01:58:27.731 INFO:tasks.rados.rados.0.smithi045.stdout:3848: copy_from oid 43 from oid 31 current snap is 376 2021-07-31T01:58:27.731 INFO:tasks.rados.rados.0.smithi045.stdout:3849: snap_remove snap 360 2021-07-31T01:58:28.708 INFO:tasks.rados.rados.0.smithi045.stdout:3849: done (6 left) 2021-07-31T01:58:28.709 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 32 current snap is 376 2021-07-31T01:58:28.709 INFO:tasks.rados.rados.0.smithi045.stdout:3850: seq_num 1176 ranges {11329536=499712} 2021-07-31T01:58:28.774 INFO:tasks.rados.rados.0.smithi045.stdout:3850: writing smithi04569269-32 from 11329536 to 11829248 tid 1 2021-07-31T01:58:28.775 INFO:tasks.rados.rados.0.smithi045.stdout:3851: rmattr oid 1 current snap is 376 2021-07-31T01:58:28.775 INFO:tasks.rados.rados.0.smithi045.stdout:3851: done (7 left) 2021-07-31T01:58:28.775 INFO:tasks.rados.rados.0.smithi045.stdout:3852: copy_from oid 15 from oid 16 current snap is 376 2021-07-31T01:58:28.776 INFO:tasks.rados.rados.0.smithi045.stdout:3845: finishing write tid 1 to smithi04569269-22 2021-07-31T01:58:28.776 INFO:tasks.rados.rados.0.smithi045.stdout:3845: finishing write tid 2 to smithi04569269-22 2021-07-31T01:58:28.776 INFO:tasks.rados.rados.0.smithi045.stdout:3845: finishing write tid 3 to smithi04569269-22 2021-07-31T01:58:28.776 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 22 v 1862 (ObjNum 1175 snap 376 seq_num 1175) dirty exists 2021-07-31T01:58:28.776 INFO:tasks.rados.rados.0.smithi045.stdout:3845: left oid 22 (ObjNum 1175 snap 376 seq_num 1175) 2021-07-31T01:58:28.777 INFO:tasks.rados.rados.0.smithi045.stdout:3843: finishing write tid 1 to smithi04569269-39 2021-07-31T01:58:28.777 INFO:tasks.rados.rados.0.smithi045.stdout:3843: finishing write tid 2 to smithi04569269-39 2021-07-31T01:58:28.777 INFO:tasks.rados.rados.0.smithi045.stdout:3843: finishing write tid 3 to smithi04569269-39 2021-07-31T01:58:28.777 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 39 v 2064 (ObjNum 1174 snap 376 seq_num 1174) dirty exists 2021-07-31T01:58:28.778 INFO:tasks.rados.rados.0.smithi045.stdout:3843: left oid 39 (ObjNum 1174 snap 376 seq_num 1174) 2021-07-31T01:58:28.778 INFO:tasks.rados.rados.0.smithi045.stdout:3844: finishing copy_from to smithi04569269-11 2021-07-31T01:58:28.778 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 11 v 2098 (ObjNum 1170 snap 374 seq_num 1170) dirty exists 2021-07-31T01:58:28.779 INFO:tasks.rados.rados.0.smithi045.stdout:3844: finishing copy_from racing read to smithi04569269-11 2021-07-31T01:58:28.779 INFO:tasks.rados.rados.0.smithi045.stdout:3846: expect (ObjNum 1108 snap 351 seq_num 1108) 2021-07-31T01:58:28.920 INFO:tasks.rados.rados.0.smithi045.stdout:3847: expect (ObjNum 1159 snap 372 seq_num 1159) 2021-07-31T01:58:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:28 smithi154 conmon[34553]: cluster 2021-07-31T01:58:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:28 smithi154 conmon[34553]: 2021-07-31T01:58:27.698415+0000 mon.a (mon.0) 1461 : cluster [DBG] osdmap e790: 8 total, 8 up, 8 in 2021-07-31T01:58:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:28 smithi154 conmon[34553]: cluster 2021-07-31T01:58:29.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:28 smithi154 conmon[34553]: 2021 2021-07-31T01:58:29.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:28 smithi154 conmon[34553]: -07-31T01:58:27.953522+0000 2021-07-31T01:58:29.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:28 smithi154 conmon[34553]: mgr.x (mgr.24542) 476 : cluster [DBG] pgmap v974: 25 pgs: 25 active+clean; 688 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2021-07-31T01:58:29.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:28 smithi154 conmon[34553]: audit 2021-07-31T01:58:29.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:28 smithi154 conmon[34553]: 2021-07-31 2021-07-31T01:58:29.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:28 smithi154 conmon[34553]: T01:58:27.976961+0000 mon.b (mon.2) 2021-07-31T01:58:29.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:28 smithi154 conmon[34553]: 117 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:58:29.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:28 smithi154 conmon[34553]: audit 2021-07-31 2021-07-31T01:58:29.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:28 smithi154 conmon[34553]: T01:58:27.977636+0000 mon.a (mon.0) 1462 : audit [INF] 2021-07-31T01:58:29.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:28 smithi154 conmon[34553]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:58:29.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:28 smithi154 conmon[34553]: audit 2021-07-31T01:58 2021-07-31T01:58:29.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:28 smithi154 conmon[34553]: :27.993636+0000 mon.b (mon.2) 118 : 2021-07-31T01:58:29.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:28 smithi154 conmon[34553]: audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T01:58:29.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:28 smithi154 conmon[34553]: audit 2021-07-31T01:58:27. 2021-07-31T01:58:29.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:28 smithi154 conmon[34553]: 994212+0000 mon.a (mon.0) 2021-07-31T01:58:29.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:28 smithi154 conmon[34553]: 1463 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T01:58:29.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:28 smithi045 conmon[39718]: cluster 2021-07-31T01:58:27.698415+0000 mon.a ( 2021-07-31T01:58:29.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:28 smithi045 conmon[39718]: mon.0) 1461 : cluster [DBG] osdmap e790: 8 total, 8 up, 8 in 2021-07-31T01:58:29.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:28 smithi045 conmon[39718]: cluster 2021-07-31T01:58:27.953522+0000 mgr.x (mgr.24542) 476 2021-07-31T01:58:29.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:28 smithi045 conmon[39718]: : cluster [DBG] pgmap v974: 25 pgs: 25 active+clean; 688 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2021-07-31T01:58:29.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:28 smithi045 conmon[39718]: audit 2021-07-31T01:58: 2021-07-31T01:58:29.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:28 smithi045 conmon[39718]: 27.976961+0000 mon.b (mon.2) 117 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:58:29.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:28 smithi045 conmon[39718]: audit 2021-07-31T01:58:27.977636+0000 mon.a (mon. 2021-07-31T01:58:29.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:28 smithi045 conmon[39718]: 0) 1462 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:58:29.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:28 smithi045 conmon[39718]: audit 2021-07-31T01:58:29.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:28 smithi045 conmon[39718]: 2021-07-31T01:58:27.993636+0000 mon.b (mon.2) 118 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T01:58:29.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:28 smithi045 conmon[39718]: audit 2021-07-31T01:58:27.994212+0000 mon.a (mon 2021-07-31T01:58:29.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:28 smithi045 conmon[39718]: .0) 1463 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T01:58:29.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:28 smithi045 conmon[33655]: cluster 2021-07-31T01:58:27.698415+0000 mon.a (mon.0) 2021-07-31T01:58:29.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:28 smithi045 conmon[33655]: 1461 : cluster [DBG] osdmap e790: 8 total, 8 up, 8 in 2021-07-31T01:58:29.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:28 smithi045 conmon[33655]: cluster 2021-07-31T01:58:27.953522+0000 mgr.x (mgr.24542) 476 : cluster [DBG] pgmap v974: 25 pgs: 25 active+clean; 688 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2021-07-31T01:58:29.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:28 smithi045 conmon[33655]: audit 2021-07 2021-07-31T01:58:29.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:28 smithi045 conmon[33655]: -31T01:58:27.976961 2021-07-31T01:58:29.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:28 smithi045 conmon[33655]: +0000 mon.b (mon.2) 117 : 2021-07-31T01:58:29.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:28 smithi045 conmon[33655]: audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:58:29.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:28 smithi045 conmon[33655]: audit 2021-07-31T01:58:29.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:28 smithi045 conmon[33655]: 2021-07-31T01 2021-07-31T01:58:29.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:28 smithi045 conmon[33655]: :58:27.977636+0000 mon.a (mon.0) 2021-07-31T01:58:29.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:28 smithi045 conmon[33655]: 1462 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:58:29.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:28 smithi045 conmon[33655]: audit 2021-07- 2021-07-31T01:58:29.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:28 smithi045 conmon[33655]: 31T01:58:27.993636+0000 mon.b (mon.2) 118 : 2021-07-31T01:58:29.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:28 smithi045 conmon[33655]: audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T01:58:29.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:28 smithi045 conmon[33655]: audit 2021-07-31T01:58 2021-07-31T01:58:29.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:28 smithi045 conmon[33655]: :27.994212+0000 mon.a (mon.0) 1463 : audit [INF] 2021-07-31T01:58:29.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:28 smithi045 conmon[33655]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T01:58:29.108 INFO:tasks.rados.rados.0.smithi045.stdout:3848: finishing copy_from to smithi04569269-43 2021-07-31T01:58:29.109 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 43 v 2805 (ObjNum 1006 snap 322 seq_num 1006) dirty exists 2021-07-31T01:58:29.109 INFO:tasks.rados.rados.0.smithi045.stdout:3848: finishing copy_from racing read to smithi04569269-43 2021-07-31T01:58:29.109 INFO:tasks.rados.rados.0.smithi045.stdout:3843: done (7 left) 2021-07-31T01:58:29.109 INFO:tasks.rados.rados.0.smithi045.stdout:3844: done (6 left) 2021-07-31T01:58:29.110 INFO:tasks.rados.rados.0.smithi045.stdout:3845: done (5 left) 2021-07-31T01:58:29.110 INFO:tasks.rados.rados.0.smithi045.stdout:3846: done (4 left) 2021-07-31T01:58:29.110 INFO:tasks.rados.rados.0.smithi045.stdout:3847: done (3 left) 2021-07-31T01:58:29.110 INFO:tasks.rados.rados.0.smithi045.stdout:3848: done (2 left) 2021-07-31T01:58:29.111 INFO:tasks.rados.rados.0.smithi045.stdout:3853: snap_create 2021-07-31T01:58:29.111 INFO:tasks.rados.rados.0.smithi045.stdout:3850: finishing write tid 1 to smithi04569269-32 2021-07-31T01:58:29.111 INFO:tasks.rados.rados.0.smithi045.stdout:3850: finishing write tid 2 to smithi04569269-32 2021-07-31T01:58:29.111 INFO:tasks.rados.rados.0.smithi045.stdout:3850: finishing write tid 3 to smithi04569269-32 2021-07-31T01:58:29.111 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 32 v 1887 (ObjNum 1176 snap 376 seq_num 1176) dirty exists 2021-07-31T01:58:29.112 INFO:tasks.rados.rados.0.smithi045.stdout:3850: left oid 32 (ObjNum 1176 snap 376 seq_num 1176) 2021-07-31T01:58:29.112 INFO:tasks.rados.rados.0.smithi045.stdout:3852: finishing copy_from to smithi04569269-15 2021-07-31T01:58:29.112 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 15 v 1859 (ObjNum 944 snap 286 seq_num 944) dirty exists 2021-07-31T01:58:29.112 INFO:tasks.rados.rados.0.smithi045.stdout:3852: finishing copy_from racing read to smithi04569269-15 2021-07-31T01:58:29.704 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: debug 2021-07-31T01:58:29.299+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T01:58:29.705 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:58:29.705 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:58:29.705 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: yield 2021-07-31T01:58:29.706 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:58:29.706 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:58:29.706 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:58:29.706 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T01:58:29.707 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:58:29.707 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T01:58:29.707 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:58:29.707 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T01:58:29.708 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:58:29.708 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: task.result() 2021-07-31T01:58:29.708 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T01:58:29.708 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: debug 2021-07-31T01:58:29.311+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T01:58:29.709 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:58:29.709 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:58:29.709 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: yield 2021-07-31T01:58:29.709 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:58:29.709 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:58:29.710 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:58:29.710 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T01:58:29.710 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:58:29.711 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T01:58:29.711 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:58:29.711 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T01:58:29.711 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:58:29.712 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: task.result() 2021-07-31T01:58:29.712 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T01:58:29.712 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: debug 2021-07-31T01:58:29.336+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T01:58:29.713 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:58:29.713 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:58:29.713 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: yield 2021-07-31T01:58:29.713 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:58:29.713 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:58:29.714 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:58:29.714 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T01:58:29.714 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:58:29.715 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T01:58:29.715 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:58:29.715 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T01:58:29.715 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:58:29.716 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: task.result() 2021-07-31T01:58:29.716 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T01:58:29.716 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: debug 2021-07-31T01:58:29.337+0000 7fcb9295f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:58:29.716 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:58:29.717 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:58:29.717 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: yield 2021-07-31T01:58:29.717 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:58:29.717 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:58:29.718 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:58:29.718 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T01:58:29.718 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:58:29.718 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T01:58:29.719 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:58:29.719 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T01:58:29.719 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:58:29.719 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: task.result() 2021-07-31T01:58:29.720 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T01:58:29.720 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: 2021-07-31T01:58:29.720 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T01:58:29.720 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: 2021-07-31T01:58:29.721 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:58:29.721 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:58:29.721 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: return f(*arg) 2021-07-31T01:58:29.721 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:58:29.722 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:58:29.722 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:58:29.722 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:58:29.722 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:58:29.723 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:58:29.723 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:58:29.723 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T01:58:29.723 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:58:29.724 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: raise self._exception 2021-07-31T01:58:29.724 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:58:29.724 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:58:29.724 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:58:29.725 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T01:58:29.726 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:58:29.726 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: raise 2021-07-31T01:58:29.727 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:58:29.727 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T01:58:29.727 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:58:29.727 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:58:29.728 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T01:58:29.728 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: debug 2021-07-31T01:58:29.345+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T01:58:29.728 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:58:29.728 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:58:29.729 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: yield 2021-07-31T01:58:29.729 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:58:29.729 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:58:29.729 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:58:29.729 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T01:58:29.730 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:58:29.730 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T01:58:29.730 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:58:29.731 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T01:58:29.731 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:58:29.731 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: task.result() 2021-07-31T01:58:29.731 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T01:58:29.731 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: debug 2021-07-31T01:58:29.346+0000 7fcb93160700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:58:29.732 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:58:29.732 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:58:29.732 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: yield 2021-07-31T01:58:29.732 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:58:29.733 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:58:29.733 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:58:29.733 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T01:58:29.733 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:58:29.734 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T01:58:29.735 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:58:29.735 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T01:58:29.735 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:58:29.736 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: task.result() 2021-07-31T01:58:29.736 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T01:58:29.736 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: 2021-07-31T01:58:29.736 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T01:58:29.737 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: 2021-07-31T01:58:29.737 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:58:29.737 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:58:29.737 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: return f(*arg) 2021-07-31T01:58:29.737 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:58:29.738 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:58:29.738 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:58:29.738 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:58:29.738 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:58:29.739 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:58:29.739 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:58:29.739 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T01:58:29.739 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:58:29.739 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: raise self._exception 2021-07-31T01:58:29.740 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:58:29.740 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:58:29.740 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:58:29.740 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T01:58:29.741 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:58:29.741 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: raise 2021-07-31T01:58:29.741 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:58:29.741 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T01:58:29.741 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:58:29.742 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:58:29.742 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T01:58:29.742 INFO:tasks.rados.rados.0.smithi045.stdout:3850: done (2 left) 2021-07-31T01:58:29.742 INFO:tasks.rados.rados.0.smithi045.stdout:3852: done (1 left) 2021-07-31T01:58:29.743 INFO:tasks.rados.rados.0.smithi045.stdout:3853: done (0 left) 2021-07-31T01:58:29.743 INFO:tasks.rados.rados.0.smithi045.stdout:3854: rollback oid 50 current snap is 377 2021-07-31T01:58:29.743 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 50 to 331 2021-07-31T01:58:29.743 INFO:tasks.rados.rados.0.smithi045.stdout:3855: rollback oid 8 current snap is 377 2021-07-31T01:58:29.744 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 8 to 350 2021-07-31T01:58:29.744 INFO:tasks.rados.rados.0.smithi045.stdout:3856: snap_remove snap 367 2021-07-31T01:58:30.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[34553]: cluster 2021-07-31T01:58:28.704909+0000 mon.a (mon.0) 2021-07-31T01:58:30.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[34553]: 1464 : cluster [DBG] osdmap e791: 8 total, 8 up, 8 in 2021-07-31T01:58:30.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[34553]: audit 2021-07-31T01:58:29.256159+0000 mon.b (mon.2) 119 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:58:30.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[34553]: 2021-07-31T01:58:30.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[34553]: audit 2021-07-31T01:58:29.257666+0000 2021-07-31T01:58:30.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[34553]: mon.b (mon.2) 120 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:58:30.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[34553]: audit 2021- 2021-07-31T01:58:30.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[34553]: 07-31T01:58:29.258759+0000 2021-07-31T01:58:30.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[34553]: mon.b (mon.2) 121 : audit [INF] 2021-07-31T01:58:30.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:29 smithi154 conmon[34553]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:58:30.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:29 smithi045 conmon[33655]: cluster 2021-07-31T01:58:28.704909+0000 mon.a 2021-07-31T01:58:30.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:29 smithi045 conmon[33655]: (mon.0) 1464 : cluster [DBG] osdmap e791: 8 total, 8 up, 8 in 2021-07-31T01:58:30.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:29 smithi045 conmon[33655]: audit 2021-07-31T01:58:29.256159+0000 mon.b (mon.2) 119 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:58:30.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:29 smithi045 conmon[33655]: audit 2021- 2021-07-31T01:58:30.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:29 smithi045 conmon[33655]: 07-31T01:58:29.257666+0000 mon.b (mon.2) 120 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:58:30.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:29 smithi045 conmon[33655]: audit 2021-07-31T01:58:29.258759 2021-07-31T01:58:30.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:29 smithi045 conmon[33655]: +0000 mon.b (mon.2) 121 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:58:30.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:29 smithi045 conmon[39718]: cluster 2021-07-31T01:58:28.704909+0000 2021-07-31T01:58:30.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:29 smithi045 conmon[39718]: mon.a (mon.0) 1464 : cluster [DBG] osdmap e791: 8 total, 8 up, 8 in 2021-07-31T01:58:30.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:29 smithi045 conmon[39718]: audit 2021- 2021-07-31T01:58:30.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:29 smithi045 conmon[39718]: 07-31T01:58:29.256159+0000 mon.b (mon.2 2021-07-31T01:58:30.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:29 smithi045 conmon[39718]: ) 119 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:58:30.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:29 smithi045 conmon[39718]: audit 2021-07-31T01:58:29.257666+0000 mon.b (mon 2021-07-31T01:58:30.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:29 smithi045 conmon[39718]: .2) 120 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:58:30.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:29 smithi045 conmon[39718]: audit 2021-07 2021-07-31T01:58:30.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:29 smithi045 conmon[39718]: -31T01:58:29.258759+0000 2021-07-31T01:58:30.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:29 smithi045 conmon[39718]: mon.b (mon.2) 121 : audit [INF] 2021-07-31T01:58:30.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:29 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:58:30.719 INFO:tasks.rados.rados.0.smithi045.stdout:3856: done (2 left) 2021-07-31T01:58:30.719 INFO:tasks.rados.rados.0.smithi045.stdout:3857: snap_remove snap 338 2021-07-31T01:58:31.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: cephadm 2021-07-31T01:58:29.259493+0000 mgr.x (mgr.24542) 477 : cephadm 2021-07-31T01:58:31.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T01:58:31.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: cephadm 2021-07-31T01:58:29.300900+0000 mgr.x (mgr.24542) 478 : cephadm [ERR] 2021-07-31T01:58:31.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:58:31.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:58:31.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: yield 2021-07-31T01:58:31.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:58:31.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:58:31.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:58:31.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T01:58:31.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:58:31.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T01:58:31.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:58:31.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T01:58:31.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:58:31.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: task.result() 2021-07-31T01:58:31.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T01:58:31.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: cephadm 2021-07-31T01:58:31.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: 2021-07-31T01:58:29.301499+0000 mgr.x (mgr.24542) 479 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T01:58:31.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: cephadm 2021-07-31T01:58:29.312850+0000 mgr.x (mgr.24542) 480 : cephadm [ERR] 2021-07-31T01:58:31.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:58:31.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:58:31.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: yield 2021-07-31T01:58:31.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:58:31.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:58:31.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:58:31.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T01:58:31.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:58:31.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T01:58:31.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:58:31.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T01:58:31.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:58:31.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: task.result() 2021-07-31T01:58:31.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T01:58:31.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: cephadm 2021-07-31T01:58:29.313507+0000 mgr.x (mgr.24542) 481 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T01:58:31.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: cephadm 2021-07-31T01:58:29.337744+0000 mgr.x (mgr.24542) 482 : cephadm [ERR] 2021-07-31T01:58:31.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:58:31.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:58:31.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: yield 2021-07-31T01:58:31.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:58:31.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:58:31.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:58:31.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T01:58:31.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:58:31.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T01:58:31.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:58:31.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T01:58:31.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:58:31.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: task.result() 2021-07-31T01:58:31.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T01:58:31.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: cephadm 2021-07-31T01:58:29.338186+0000 mgr.x (mgr.24542) 483 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:58:31.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:58:31.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:58:31.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: yield 2021-07-31T01:58:31.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:58:31.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:58:31.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:58:31.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T01:58:31.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:58:31.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T01:58:31.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:58:31.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T01:58:31.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:58:31.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: task.result() 2021-07-31T01:58:31.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T01:58:31.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: 2021-07-31T01:58:31.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T01:58:31.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: 2021-07-31T01:58:31.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:58:31.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:58:31.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: return f(*arg) 2021-07-31T01:58:31.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:58:31.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:58:31.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:58:31.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:58:31.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:58:31.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:58:31.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:58:31.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T01:58:31.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:58:31.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: raise self._exception 2021-07-31T01:58:31.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:58:31.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:58:31.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:58:31.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T01:58:31.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:58:31.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: raise 2021-07-31T01:58:31.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:58:31.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T01:58:31.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:58:31.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:58:31.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T01:58:31.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: cephadm 2021-07-31T01:58:31.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: 2021-07-31T01:58:29.346772+0000 mgr.x (mgr.24542) 484 : cephadm [ERR] 2021-07-31T01:58:31.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:58:31.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:58:31.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: yield 2021-07-31T01:58:31.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:58:31.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:58:31.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:58:31.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T01:58:31.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:58:31.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T01:58:31.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:58:31.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T01:58:31.035 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:58:31.035 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: task.result() 2021-07-31T01:58:31.036 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T01:58:31.036 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: cephadm 2021-07-31T01:58:29.347185+0000 mgr.x (mgr.24542) 485 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:58:31.037 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:58:31.037 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:58:31.038 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: yield 2021-07-31T01:58:31.038 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:58:31.038 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:58:31.039 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:58:31.039 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T01:58:31.040 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:58:31.040 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T01:58:31.040 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:58:31.041 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T01:58:31.041 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:58:31.041 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: task.result() 2021-07-31T01:58:31.042 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T01:58:31.042 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: 2021-07-31T01:58:31.042 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T01:58:31.042 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: 2021-07-31T01:58:31.043 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:58:31.043 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:58:31.043 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: return f(*arg) 2021-07-31T01:58:31.043 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:58:31.044 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:58:31.044 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:58:31.044 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:58:31.045 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:58:31.045 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:58:31.045 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:58:31.045 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T01:58:31.046 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:58:31.046 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: raise self._exception 2021-07-31T01:58:31.046 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:58:31.047 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:58:31.047 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:58:31.047 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T01:58:31.047 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:58:31.048 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: raise 2021-07-31T01:58:31.048 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:58:31.048 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T01:58:31.048 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:58:31.049 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:58:31.049 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T01:58:31.049 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: cluster 2021-07-31T01:58:29.710074+0000 mon.a (mon.0) 1465 : cluster [DBG] osdmap e792: 8 total, 8 up, 8 in 2021-07-31T01:58:31.050 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:30 smithi154 conmon[34553]: cluster 2021-07-31T01:58:29.954039+0000 mgr.x (mgr.24542) 486 : cluster [DBG] pgmap v977: 25 pgs: 25 active+clean; 688 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 12 MiB/s rd, 1.6 MiB/s wr, 13 op/s 2021-07-31T01:58:31.051 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: cephadm 2021-07-31T01:58:29.259493+0000 mgr.x (mgr.24542) 477 : cephadm 2021-07-31T01:58:31.051 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T01:58:31.051 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: cephadm 2021-07-31T01:58:29.300900+0000 mgr.x (mgr.24542) 478 : cephadm [ERR] 2021-07-31T01:58:31.051 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:58:31.052 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:58:31.052 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: yield 2021-07-31T01:58:31.052 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:58:31.052 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:58:31.053 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:58:31.053 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T01:58:31.053 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:58:31.053 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T01:58:31.054 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:58:31.054 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T01:58:31.054 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:58:31.054 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: task.result() 2021-07-31T01:58:31.055 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T01:58:31.055 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: cephadm 2021-07-31T01:58:29.301499+0000 mgr.x (mgr.24542) 479 2021-07-31T01:58:31.055 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T01:58:31.055 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: cephadm 2021-07-31T01:58:29.312850+0000 mgr.x (mgr.24542) 480 : cephadm [ERR] 2021-07-31T01:58:31.056 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:58:31.056 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:58:31.057 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: yield 2021-07-31T01:58:31.057 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:58:31.057 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:58:31.058 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:58:31.058 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T01:58:31.059 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:58:31.059 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T01:58:31.059 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:58:31.060 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T01:58:31.060 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:58:31.060 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: task.result() 2021-07-31T01:58:31.060 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T01:58:31.061 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: cephadm 2021-07 2021-07-31T01:58:31.061 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: -31T01:58:29.313507+0000 mgr.x (mgr.24542) 481 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T01:58:31.061 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: cephadm 2021-07-31T01:58:29.337744+0000 mgr.x 2021-07-31T01:58:31.061 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: (mgr.24542) 482 : cephadm [ERR] 2021-07-31T01:58:31.061 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:58:31.062 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: cephadm 2021-07-31T01:58:29.259493+0000 mgr.x (mgr. 2021-07-31T01:58:31.062 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: 24542) 477 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T01:58:31.063 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: cephadm 2021 2021-07-31T01:58:31.063 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: -07-31T01:58:29.300900+0000 mgr.x (mgr.24542) 478 : cephadm [ERR] 2021-07-31T01:58:31.063 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:58:31.064 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:58:31.064 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: yield 2021-07-31T01:58:31.065 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:58:31.065 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:58:31.065 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:58:31.066 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T01:58:31.066 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:58:31.067 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T01:58:31.067 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:58:31.067 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T01:58:31.068 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:58:31.068 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: task.result() 2021-07-31T01:58:31.068 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T01:58:31.069 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: cephadm 2021-07-31T01:58:29.301499+0000 mgr.x (mgr.24542) 479 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T01:58:31.069 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: cephadm 2021-07-31 2021-07-31T01:58:31.069 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: T01:58:29.312850+0000 mgr.x (mgr.24542 2021-07-31T01:58:31.069 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: ) 480 : cephadm [ERR] 2021-07-31T01:58:31.070 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:58:31.070 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:58:31.070 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: yield 2021-07-31T01:58:31.071 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:58:31.071 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:58:31.071 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:58:31.071 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T01:58:31.072 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:58:31.072 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T01:58:31.072 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:58:31.072 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T01:58:31.073 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:58:31.073 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: task.result() 2021-07-31T01:58:31.073 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T01:58:31.073 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: cephadm 2021-07-31T01:58:29.313507+0000 mgr.x (mgr.24542) 481 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T01:58:31.073 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: cephadm 2021-07-31T01:58:29.337744 2021-07-31T01:58:31.074 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 482 : cephadm [ERR] 2021-07-31T01:58:31.074 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: 2021-07-31T01:58:31.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:58:31.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: yield 2021-07-31T01:58:31.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:58:31.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:58:31.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:58:31.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T01:58:31.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:58:31.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T01:58:31.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:58:31.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T01:58:31.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:58:31.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: task.result() 2021-07-31T01:58:31.078 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T01:58:31.078 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: cephadm 2021-07-31T01:58:29.338186+0000 mgr.x (mgr.24542) 483 : cephadm 2021-07-31T01:58:31.078 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:58:31.078 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:58:31.079 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:58:31.079 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: yield 2021-07-31T01:58:31.080 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:58:31.080 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:58:31.080 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:58:31.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T01:58:31.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:58:31.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T01:58:31.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:58:31.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T01:58:31.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:58:31.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: task.result() 2021-07-31T01:58:31.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T01:58:31.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: 2021-07-31T01:58:31.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T01:58:31.084 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: 2021-07-31T01:58:31.084 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:58:31.084 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:58:31.085 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: return f(*arg) 2021-07-31T01:58:31.085 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:58:31.085 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:58:31.085 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:58:31.085 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:58:31.086 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:58:31.086 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:58:31.087 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:58:31.087 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:58:31.087 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: yield 2021-07-31T01:58:31.088 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:58:31.088 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:58:31.089 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:58:31.089 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T01:58:31.089 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:58:31.090 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T01:58:31.090 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:58:31.091 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T01:58:31.091 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:58:31.092 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: task.result() 2021-07-31T01:58:31.092 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T01:58:31.093 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: cephadm 2021-07-31T01:58:29.338186+0000 mgr.x (mgr.24542) 483 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:58:31.093 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:58:31.093 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:58:31.094 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: yield 2021-07-31T01:58:31.094 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:58:31.095 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:58:31.095 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:58:31.096 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T01:58:31.096 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:58:31.096 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T01:58:31.096 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:58:31.097 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T01:58:31.097 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:58:31.097 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: task.result() 2021-07-31T01:58:31.098 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T01:58:31.098 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: 2021-07-31T01:58:31.099 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T01:58:31.099 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: 2021-07-31T01:58:31.099 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:58:31.100 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:58:31.100 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: return f(*arg) 2021-07-31T01:58:31.100 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:58:31.101 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:58:31.101 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:58:31.101 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:58:31.101 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:58:31.101 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:58:31.102 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:58:31.102 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T01:58:31.103 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:58:31.103 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: raise self._exception 2021-07-31T01:58:31.103 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:58:31.103 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:58:31.104 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:58:31.104 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T01:58:31.104 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:58:31.104 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: raise 2021-07-31T01:58:31.105 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:58:31.105 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T01:58:31.105 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:58:31.105 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:58:31.105 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T01:58:31.106 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: cephadm 2021-07-31T01:58:29.346772+0000 mgr.x (mgr.24542) 484 : cephadm [ERR] 2021-07-31T01:58:31.106 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:58:31.106 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:58:31.106 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: yield 2021-07-31T01:58:31.106 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:58:31.107 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:58:31.107 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:58:31.107 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T01:58:31.107 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:58:31.108 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T01:58:31.108 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:58:31.108 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T01:58:31.108 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:58:31.108 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: task.result() 2021-07-31T01:58:31.109 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T01:58:31.109 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: cephadm 2021-07-31T01:58:29.347185+0000 mgr.x (mgr.24542) 485 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:58:31.109 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:58:31.109 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:58:31.109 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: yield 2021-07-31T01:58:31.110 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:58:31.110 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:58:31.110 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:58:31.110 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T01:58:31.111 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:58:31.111 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:58:31.112 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T01:58:31.112 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:58:31.112 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: raise self._exception 2021-07-31T01:58:31.112 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:58:31.112 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:58:31.113 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:58:31.113 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T01:58:31.113 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:58:31.113 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: raise 2021-07-31T01:58:31.114 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:58:31.114 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T01:58:31.114 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:58:31.114 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:58:31.114 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T01:58:31.115 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: cephadm 2021-07 2021-07-31T01:58:31.115 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: -31T01:58:29.346772+0000 mgr.x (mgr.24542) 484 : cephadm [ERR] 2021-07-31T01:58:31.115 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:58:31.116 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:58:31.116 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: yield 2021-07-31T01:58:31.116 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:58:31.116 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:58:31.117 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:58:31.117 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T01:58:31.117 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:58:31.117 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T01:58:31.118 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:58:31.118 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T01:58:31.118 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:58:31.118 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: task.result() 2021-07-31T01:58:31.118 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T01:58:31.119 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: cephadm 2021-07-31T01:58:29.347185+0000 mgr.x (mgr.24542) 485 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:58:31.119 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:58:31.119 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:58:31.119 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: yield 2021-07-31T01:58:31.120 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:58:31.120 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:58:31.120 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:58:31.120 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T01:58:31.121 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T01:58:31.121 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:58:31.122 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T01:58:31.122 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:58:31.122 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: task.result() 2021-07-31T01:58:31.122 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T01:58:31.123 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: 2021-07-31T01:58:31.123 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T01:58:31.123 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: 2021-07-31T01:58:31.123 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:58:31.123 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:58:31.124 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: return f(*arg) 2021-07-31T01:58:31.124 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:58:31.124 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:58:31.124 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:58:31.125 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:58:31.125 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:58:31.125 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:58:31.125 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:58:31.125 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T01:58:31.126 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:58:31.126 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: raise self._exception 2021-07-31T01:58:31.126 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:58:31.126 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:58:31.126 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:58:31.127 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T01:58:31.127 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:58:31.127 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: raise 2021-07-31T01:58:31.127 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:58:31.128 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T01:58:31.128 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:58:31.128 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:58:31.128 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T01:58:31.129 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: cluster 2021-07-31T01:58:29.710074+0000 mon.a (mon.0) 1465 : cluster [DBG] osdmap e792: 8 total, 8 up, 8 in 2021-07-31T01:58:31.129 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[33655]: cluster 2021-07-31T01:58:29.954039+0000 mgr.x (mgr.24542) 486 : cluster [DBG] pgmap v977: 25 pgs: 25 active+clean; 688 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 12 MiB/s rd, 1.6 MiB/s wr, 13 op/s 2021-07-31T01:58:31.129 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:58:31.129 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T01:58:31.130 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:58:31.130 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T01:58:31.130 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:58:31.130 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: task.result() 2021-07-31T01:58:31.131 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T01:58:31.131 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: 2021-07-31T01:58:31.131 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T01:58:31.131 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: 2021-07-31T01:58:31.131 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:58:31.132 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:58:31.132 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: return f(*arg) 2021-07-31T01:58:31.132 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:58:31.132 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:58:31.133 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:58:31.133 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:58:31.133 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:58:31.133 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:58:31.133 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:58:31.134 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T01:58:31.134 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:58:31.134 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: raise self._exception 2021-07-31T01:58:31.134 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:58:31.134 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:58:31.135 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:58:31.135 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T01:58:31.135 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:58:31.135 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: raise 2021-07-31T01:58:31.136 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:58:31.136 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T01:58:31.136 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:58:31.136 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:58:31.136 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T01:58:31.137 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: cluster 2021-07-31T01:58:29.710074+0000 mon.a (mon.0) 1465 : cluster [DBG] osdmap e792: 8 total, 8 up, 8 in 2021-07-31T01:58:31.137 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:30 smithi045 conmon[39718]: cluster 2021-07-31T01:58:29.954039+0000 mgr.x (mgr.24542) 486 : cluster [DBG] pgmap v977: 25 pgs: 25 active+clean; 688 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 12 MiB/s rd, 1.6 MiB/s wr, 13 op/s 2021-07-31T01:58:31.693 INFO:tasks.rados.rados.0.smithi045.stdout:3857: done (2 left) 2021-07-31T01:58:31.694 INFO:tasks.rados.rados.0.smithi045.stdout:3858: snap_create 2021-07-31T01:58:31.694 INFO:tasks.rados.rados.0.smithi045.stdout:3855: finishing rollback tid 0 to smithi04569269-8 2021-07-31T01:58:31.694 INFO:tasks.rados.rados.0.smithi045.stdout:3855: finishing rollback tid 1 to smithi04569269-8 2021-07-31T01:58:31.694 INFO:tasks.rados.rados.0.smithi045.stdout:3855: finishing rollback tid 2 to smithi04569269-8 2021-07-31T01:58:31.695 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 8 v 2889 (ObjNum 994 snap 309 seq_num 994) dirty exists 2021-07-31T01:58:31.695 INFO:tasks.rados.rados.0.smithi045.stdout:3854: finishing rollback tid 0 to smithi04569269-50 2021-07-31T01:58:31.695 INFO:tasks.rados.rados.0.smithi045.stdout:3854: finishing rollback tid 1 to smithi04569269-50 2021-07-31T01:58:31.696 INFO:tasks.rados.rados.0.smithi045.stdout:3854: finishing rollback tid 2 to smithi04569269-50 2021-07-31T01:58:31.696 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 50 v 2384 (ObjNum 1008 snap 322 seq_num 1008) dirty exists 2021-07-31T01:58:32.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:31 smithi154 conmon[34553]: cluster 2021-07-31T01:58:30.718169+0000 mon.a (mon.0 2021-07-31T01:58:32.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:31 smithi154 conmon[34553]: ) 1466 : cluster [DBG] osdmap e793: 8 total, 8 up, 8 in 2021-07-31T01:58:32.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:31 smithi154 conmon[34553]: cluster 2021-07-31T01:58:31.692951+0000 mon.a (mon.0) 1467 : cluster [DBG] osdmap e794: 8 total, 8 up, 8 in 2021-07-31T01:58:32.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:31 smithi045 conmon[39718]: cluster 2021-07-31T01:58:30.718169+0000 2021-07-31T01:58:32.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:31 smithi045 conmon[39718]: mon.a (mon.0) 1466 : cluster [DBG] osdmap e793: 8 total, 8 up, 8 in 2021-07-31T01:58:32.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:31 smithi045 conmon[39718]: cluster 2021-07-31T01:58: 2021-07-31T01:58:32.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:31 smithi045 conmon[39718]: 31.692951+0000 mon.a (mon.0) 1467 : cluster [DBG] osdmap e794: 8 total, 8 up, 8 in 2021-07-31T01:58:32.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:31 smithi045 conmon[33655]: cluster 2021-07-31T01:58:30.718169 2021-07-31T01:58:32.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:31 smithi045 conmon[33655]: +0000 mon.a (mon.0) 1466 : cluster [DBG] osdmap e793: 8 total, 8 up, 8 in 2021-07-31T01:58:32.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:31 smithi045 conmon[33655]: cluster 2021-07-31T01:58:31.692951+0000 mon.a (mon.0) 1467 : cluster [DBG] osdmap e794: 8 total, 8 up, 8 in 2021-07-31T01:58:32.700 INFO:tasks.rados.rados.0.smithi045.stdout:3854: done (2 left) 2021-07-31T01:58:32.700 INFO:tasks.rados.rados.0.smithi045.stdout:3855: done (1 left) 2021-07-31T01:58:32.701 INFO:tasks.rados.rados.0.smithi045.stdout:3858: done (0 left) 2021-07-31T01:58:32.701 INFO:tasks.rados.rados.0.smithi045.stdout:3859: rmattr oid 26 current snap is 378 2021-07-31T01:58:32.701 INFO:tasks.rados.rados.0.smithi045.stdout:3859: done (0 left) 2021-07-31T01:58:32.701 INFO:tasks.rados.rados.0.smithi045.stdout:3860: snap_remove snap 354 2021-07-31T01:58:33.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:32 smithi154 conmon[34553]: cluster 2021-07-31T01:58:31.954614+0000 mgr.x (mgr.24542) 487 : cluster [DBG] 2021-07-31T01:58:33.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:32 smithi154 conmon[34553]: pgmap v980: 25 pgs: 25 active+clean; 688 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 12 MiB/s rd, 1.6 MiB/s wr, 13 op/s 2021-07-31T01:58:33.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:32 smithi154 conmon[34553]: cluster 2021-07-31T01:58:32.699501+0000 mon.a (mon.0) 1468 : cluster [DBG] osdmap e795: 8 total, 8 up, 8 in 2021-07-31T01:58:33.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:32 smithi045 conmon[33655]: cluster 2021-07-31T01:58:31.954614+0000 mgr.x (mgr.24542) 487 : cluster [DBG] 2021-07-31T01:58:33.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:32 smithi045 conmon[33655]: pgmap v980: 25 pgs: 25 active+clean; 688 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 12 MiB/s rd, 1.6 MiB/s wr, 13 op/s 2021-07-31T01:58:33.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:32 smithi045 conmon[33655]: cluster 2021-07-31T01:58:32.699501+0000 mon.a (mon.0) 1468 : cluster [DBG] osdmap e795: 8 total, 8 up, 8 in 2021-07-31T01:58:33.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:32 smithi045 conmon[39718]: cluster 2021-07-31T01:58:31.954614+0000 mgr.x (mgr. 2021-07-31T01:58:33.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:32 smithi045 conmon[39718]: 24542) 487 : cluster [DBG] pgmap v980: 25 pgs: 25 active+clean; 688 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 12 MiB/s rd, 1.6 MiB/s wr, 13 op/s 2021-07-31T01:58:33.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:32 smithi045 conmon[39718]: cluster 2021-07-31T01:58:32.699501+0000 mon.a (mon.0) 1468 2021-07-31T01:58:33.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:32 smithi045 conmon[39718]: : cluster [DBG] osdmap e795: 8 total, 8 up, 8 in 2021-07-31T01:58:33.701 INFO:tasks.rados.rados.0.smithi045.stdout:3860: done (0 left) 2021-07-31T01:58:33.702 INFO:tasks.rados.rados.0.smithi045.stdout:3861: snap_create 2021-07-31T01:58:34.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:58:33 smithi045 conmon[61210]: level=error ts=2021-07-31T01:58:33.528Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:58:34.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:58:33 smithi045 conmon[61210]: level=error ts=2021-07-31T01:58:33.528Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:58:34.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:58:33 smithi045 conmon[61210]: level=error ts=2021-07-31T01:58:33.528Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T01:58:34.708 INFO:tasks.rados.rados.0.smithi045.stdout:3861: done (0 left) 2021-07-31T01:58:34.708 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 50 current snap is 379 2021-07-31T01:58:34.708 INFO:tasks.rados.rados.0.smithi045.stdout:3862: seq_num 1177 ranges {4448256=516096} 2021-07-31T01:58:34.735 INFO:tasks.rados.rados.0.smithi045.stdout:3862: writing smithi04569269-50 from 4448256 to 4964352 tid 1 2021-07-31T01:58:34.735 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 7 current snap is 379 2021-07-31T01:58:34.735 INFO:tasks.rados.rados.0.smithi045.stdout:3863: seq_num 1178 ranges {0=532480} 2021-07-31T01:58:34.738 INFO:tasks.rados.rados.0.smithi045.stdout:3863: writing smithi04569269-7 from 0 to 532480 tid 1 2021-07-31T01:58:34.738 INFO:tasks.rados.rados.0.smithi045.stdout:3864: rollback oid 43 current snap is 379 2021-07-31T01:58:34.739 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 43 to 371 2021-07-31T01:58:34.739 INFO:tasks.rados.rados.0.smithi045.stdout:3865: delete oid 23 current snap is 379 2021-07-31T01:58:34.743 INFO:tasks.rados.rados.0.smithi045.stdout:3862: finishing write tid 1 to smithi04569269-50 2021-07-31T01:58:34.746 INFO:tasks.rados.rados.0.smithi045.stdout:3862: finishing write tid 2 to smithi04569269-50 2021-07-31T01:58:34.746 INFO:tasks.rados.rados.0.smithi045.stdout:3864: finishing rollback tid 0 to smithi04569269-43 2021-07-31T01:58:34.746 INFO:tasks.rados.rados.0.smithi045.stdout:3865: done (3 left) 2021-07-31T01:58:34.747 INFO:tasks.rados.rados.0.smithi045.stdout:3866: read oid 1 snap 371 2021-07-31T01:58:34.747 INFO:tasks.rados.rados.0.smithi045.stdout:3866: expect (ObjNum 1116 snap 351 seq_num 1116) 2021-07-31T01:58:34.747 INFO:tasks.rados.rados.0.smithi045.stdout:3867: read oid 23 snap -1 2021-07-31T01:58:34.747 INFO:tasks.rados.rados.0.smithi045.stdout:3867: expect deleted 2021-07-31T01:58:34.747 INFO:tasks.rados.rados.0.smithi045.stdout:3868: snap_create 2021-07-31T01:58:34.748 INFO:tasks.rados.rados.0.smithi045.stdout:3863: finishing write tid 1 to smithi04569269-7 2021-07-31T01:58:34.748 INFO:tasks.rados.rados.0.smithi045.stdout:3863: finishing write tid 2 to smithi04569269-7 2021-07-31T01:58:34.749 INFO:tasks.rados.rados.0.smithi045.stdout:3862: finishing write tid 3 to smithi04569269-50 2021-07-31T01:58:34.749 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 50 v 2386 (ObjNum 1177 snap 379 seq_num 1177) dirty exists 2021-07-31T01:58:34.749 INFO:tasks.rados.rados.0.smithi045.stdout:3862: left oid 50 (ObjNum 1177 snap 379 seq_num 1177) 2021-07-31T01:58:34.749 INFO:tasks.rados.rados.0.smithi045.stdout:3864: finishing rollback tid 1 to smithi04569269-43 2021-07-31T01:58:34.750 INFO:tasks.rados.rados.0.smithi045.stdout:3864: finishing rollback tid 2 to smithi04569269-43 2021-07-31T01:58:34.750 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 43 v 2808 (ObjNum 1113 snap 351 seq_num 1113) dirty exists 2021-07-31T01:58:34.750 INFO:tasks.rados.rados.0.smithi045.stdout:3863: finishing write tid 3 to smithi04569269-7 2021-07-31T01:58:34.750 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 7 v 2166 (ObjNum 1178 snap 379 seq_num 1178) dirty exists 2021-07-31T01:58:34.751 INFO:tasks.rados.rados.0.smithi045.stdout:3863: left oid 7 (ObjNum 1178 snap 379 seq_num 1178) 2021-07-31T01:58:34.770 INFO:tasks.rados.rados.0.smithi045.stdout:3866: expect (ObjNum 1116 snap 351 seq_num 1116) 2021-07-31T01:58:35.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:34 smithi154 conmon[34553]: cluster 2021-07-31T01:58:33.701112+0000 mon.a (mon.0) 1469 : cluster [DBG] osdmap e796: 8 total, 8 up, 8 in 2021-07-31T01:58:35.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:34 smithi154 conmon[34553]: cluster 2021-07-31T01:58:33.955302+0000 mgr.x (mgr.24542) 488 : cluster [DBG] pgmap v983: 25 pgs: 25 active+clean; 695 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2021-07-31T01:58:35.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:34 smithi045 conmon[39718]: cluster 2021-07-31T01:58:33. 2021-07-31T01:58:35.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:34 smithi045 conmon[39718]: 701112+0000 mon.a (mon.0) 1469 : cluster [DBG] osdmap e796: 8 total, 8 up, 8 in 2021-07-31T01:58:35.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:34 smithi045 conmon[39718]: cluster 2021-07-31T01:58:33.955302 2021-07-31T01:58:35.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:34 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 488 : cluster [DBG] pgmap v983: 25 pgs: 25 active+clean; 695 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2021-07-31T01:58:35.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:34 smithi045 conmon[39718]: 2021-07-31T01:58:35.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:34 smithi045 conmon[33655]: cluster 2021-07-31T01:58:33.701112+0000 mon.a (mon.0) 1469 : 2021-07-31T01:58:35.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:34 smithi045 conmon[33655]: cluster [DBG] osdmap e796: 8 total, 8 up, 8 in 2021-07-31T01:58:35.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:34 smithi045 conmon[33655]: cluster 2021-07-31T01:58:33.955302+0000 mgr.x (mgr.24542) 488 : cluster [DBG] pgmap v983: 25 pgs: 25 active+clean; 695 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2021-07-31T01:58:35.713 INFO:tasks.rados.rados.0.smithi045.stdout:3862: done (5 left) 2021-07-31T01:58:35.713 INFO:tasks.rados.rados.0.smithi045.stdout:3863: done (4 left) 2021-07-31T01:58:35.713 INFO:tasks.rados.rados.0.smithi045.stdout:3864: done (3 left) 2021-07-31T01:58:35.713 INFO:tasks.rados.rados.0.smithi045.stdout:3866: done (2 left) 2021-07-31T01:58:35.714 INFO:tasks.rados.rados.0.smithi045.stdout:3867: done (1 left) 2021-07-31T01:58:35.714 INFO:tasks.rados.rados.0.smithi045.stdout:3868: done (0 left) 2021-07-31T01:58:35.714 INFO:tasks.rados.rados.0.smithi045.stdout:3869: snap_create 2021-07-31T01:58:36.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:35 smithi154 conmon[34553]: cluster 2021-07-31T01:58:34.706890+0000 mon.a (mon.0) 1470 : cluster [DBG] osdmap e797: 8 total, 8 up, 8 in 2021-07-31T01:58:36.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:35 smithi045 conmon[39718]: cluster 2021-07-31T01:58:34.706890+0000 mon.a 2021-07-31T01:58:36.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:35 smithi045 conmon[39718]: (mon.0) 1470 : cluster [DBG] osdmap e797: 8 total, 8 up, 8 in 2021-07-31T01:58:36.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:35 smithi045 conmon[33655]: cluster 2021-07-31T01:58:34.706890 2021-07-31T01:58:36.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:35 smithi045 conmon[33655]: +0000 mon.a (mon.0) 1470 : cluster [DBG] osdmap e797: 8 total, 8 up, 8 in 2021-07-31T01:58:36.695 INFO:tasks.rados.rados.0.smithi045.stdout:3869: done (0 left) 2021-07-31T01:58:36.695 INFO:tasks.rados.rados.0.smithi045.stdout:3870: snap_create 2021-07-31T01:58:37.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:36 smithi154 conmon[34553]: cluster 2021-07-31T01:58:35.712266+0000 mon.a (mon.0) 1471 : cluster [DBG] osdmap e798: 8 total, 8 up, 8 in 2021-07-31T01:58:37.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:36 smithi154 conmon[34553]: cluster 2021-07-31T01:58:35.955757+0000 mgr.x (mgr.24542) 489 : cluster [DBG] pgmap v986: 25 pgs: 25 active+clean; 695 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2021-07-31T01:58:37.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:36 smithi154 conmon[34553]: cluster 2021-07-31T01:58:36.694205+0000 mon.a (mon.0) 1472 : cluster [DBG] osdmap e799: 8 total, 8 up, 8 in 2021-07-31T01:58:37.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:36 smithi045 conmon[39718]: cluster 2021-07-31T01:58:35.712266+0000 mon.a (mon.0) 1471 : cluster 2021-07-31T01:58:37.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:36 smithi045 conmon[39718]: [DBG] osdmap e798: 8 total, 8 up, 8 in 2021-07-31T01:58:37.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:36 smithi045 conmon[39718]: cluster 2021-07-31T01:58:35.955757+0000 mgr.x (mgr.24542) 489 : cluster [DBG] pgmap v986: 25 pgs: 25 active+clean; 695 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2021-07-31T01:58:37.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:36 smithi045 conmon[39718]: cluster 2021-07-31T01:58:36.694205+0000 mon.a (mon 2021-07-31T01:58:37.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:36 smithi045 conmon[39718]: .0) 1472 : cluster [DBG] osdmap e799: 8 total, 8 up, 8 in 2021-07-31T01:58:37.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:36 smithi045 conmon[33655]: cluster 2021-07-31T01:58:35.712266+0000 mon.a (mon.0) 1471 : cluster [DBG] osdmap e798: 8 total, 8 up, 8 in 2021-07-31T01:58:37.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:36 smithi045 conmon[33655]: cluster 2021-07-31T01:58:35.955757+0000 mgr.x (mgr.24542) 489 : cluster [DBG] pgmap v986: 25 pgs: 25 active+clean; 695 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2021-07-31T01:58:37.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:36 smithi045 conmon[33655]: cluster 2021-07-31T01:58:36.694205+0000 mon.a ( 2021-07-31T01:58:37.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:36 smithi045 conmon[33655]: mon.0) 1472 : cluster [DBG] osdmap e799: 8 total, 8 up, 8 in 2021-07-31T01:58:37.700 INFO:tasks.rados.rados.0.smithi045.stdout:3870: done (0 left) 2021-07-31T01:58:37.701 INFO:tasks.rados.rados.0.smithi045.stdout:3871: delete oid 12 current snap is 382 2021-07-31T01:58:37.705 INFO:tasks.rados.rados.0.smithi045.stdout:3871: done (0 left) 2021-07-31T01:58:37.705 INFO:tasks.rados.rados.0.smithi045.stdout:3872: rmattr oid 39 current snap is 382 2021-07-31T01:58:37.705 INFO:tasks.rados.rados.0.smithi045.stdout:3873: rollback oid 42 current snap is 382 2021-07-31T01:58:37.706 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 42 to 350 2021-07-31T01:58:37.706 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 38 current snap is 382 2021-07-31T01:58:37.706 INFO:tasks.rados.rados.0.smithi045.stdout:3874: seq_num 1181 ranges {573440=581632} 2021-07-31T01:58:37.712 INFO:tasks.rados.rados.0.smithi045.stdout:3874: writing smithi04569269-38 from 573440 to 1155072 tid 1 2021-07-31T01:58:37.712 INFO:tasks.rados.rados.0.smithi045.stdout:3875: snap_create 2021-07-31T01:58:37.712 INFO:tasks.rados.rados.0.smithi045.stdout:3873: finishing rollback tid 0 to smithi04569269-42 2021-07-31T01:58:37.712 INFO:tasks.rados.rados.0.smithi045.stdout:3873: finishing rollback tid 1 to smithi04569269-42 2021-07-31T01:58:37.713 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 42 v 2388 (ObjNum 1617123420 snap 1515541362 seq_num 1634753126) dirty dne 2021-07-31T01:58:37.713 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 39 v 2099 (ObjNum 1174 snap 376 seq_num 1174) dirty exists 2021-07-31T01:58:37.717 INFO:tasks.rados.rados.0.smithi045.stdout:3874: finishing write tid 1 to smithi04569269-38 2021-07-31T01:58:37.718 INFO:tasks.rados.rados.0.smithi045.stdout:3874: finishing write tid 2 to smithi04569269-38 2021-07-31T01:58:37.719 INFO:tasks.rados.rados.0.smithi045.stdout:3874: finishing write tid 3 to smithi04569269-38 2021-07-31T01:58:37.719 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 38 v 2101 (ObjNum 1181 snap 382 seq_num 1181) dirty exists 2021-07-31T01:58:37.719 INFO:tasks.rados.rados.0.smithi045.stdout:3874: left oid 38 (ObjNum 1181 snap 382 seq_num 1181) 2021-07-31T01:58:38.707 INFO:tasks.rados.rados.0.smithi045.stdout:3872: done (3 left) 2021-07-31T01:58:38.707 INFO:tasks.rados.rados.0.smithi045.stdout:3873: done (2 left) 2021-07-31T01:58:38.707 INFO:tasks.rados.rados.0.smithi045.stdout:3874: done (1 left) 2021-07-31T01:58:38.707 INFO:tasks.rados.rados.0.smithi045.stdout:3875: done (0 left) 2021-07-31T01:58:38.708 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 43 current snap is 383 2021-07-31T01:58:38.708 INFO:tasks.rados.rados.0.smithi045.stdout:3876: seq_num 1182 ranges {6111232=606208} 2021-07-31T01:58:38.743 INFO:tasks.rados.rados.0.smithi045.stdout:3876: writing smithi04569269-43 from 6111232 to 6717440 tid 1 2021-07-31T01:58:38.743 INFO:tasks.rados.rados.0.smithi045.stdout:3877: delete oid 39 current snap is 383 2021-07-31T01:58:38.749 INFO:tasks.rados.rados.0.smithi045.stdout:3877: done (1 left) 2021-07-31T01:58:38.749 INFO:tasks.rados.rados.0.smithi045.stdout:3878: snap_remove snap 377 2021-07-31T01:58:39.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:38 smithi154 conmon[34553]: cluster 2021-07-31T01:58:37.700210+0000 mon.a (mon.0) 1473 : cluster [DBG] osdmap e800: 8 total, 8 up, 8 in 2021-07-31T01:58:39.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:38 smithi154 conmon[34553]: cluster 2021-07-31T01:58:37.956288+0000 mgr.x (mgr.24542) 490 : cluster [DBG] pgmap v989: 25 pgs: 25 active+clean; 695 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T01:58:39.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:38 smithi045 conmon[39718]: cluster 2021-07-31T01:58:37.700210+0000 mon.a (mon 2021-07-31T01:58:39.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:38 smithi045 conmon[39718]: .0) 1473 : cluster [DBG] osdmap e800: 8 total, 8 up, 8 in 2021-07-31T01:58:39.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:38 smithi045 conmon[39718]: cluster 2021-07-31T01:58:37.956288+0000 mgr.x (mgr.24542) 490 : 2021-07-31T01:58:39.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:38 smithi045 conmon[39718]: cluster [DBG] pgmap v989: 25 pgs: 25 active+clean; 695 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T01:58:39.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:38 smithi045 conmon[33655]: cluster 2021-07-31T01:58:37.700210+0000 mon.a (mon.0) 2021-07-31T01:58:39.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:38 smithi045 conmon[33655]: 1473 : cluster [DBG] osdmap e800: 8 total, 8 up, 8 in 2021-07-31T01:58:39.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:38 smithi045 conmon[33655]: cluster 2021-07-31T01:58:37.956288+0000 mgr.x (mgr.24542) 490 : cluster [DBG] pgmap v989: 25 pgs: 25 active+clean; 695 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T01:58:39.712 INFO:tasks.rados.rados.0.smithi045.stdout:3878: done (1 left) 2021-07-31T01:58:39.712 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 16 current snap is 383 2021-07-31T01:58:39.712 INFO:tasks.rados.rados.0.smithi045.stdout:3879: seq_num 1184 ranges {1417216=638976} 2021-07-31T01:58:39.724 INFO:tasks.rados.rados.0.smithi045.stdout:3879: writing smithi04569269-16 from 1417216 to 2056192 tid 1 2021-07-31T01:58:39.724 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 2 current snap is 383 2021-07-31T01:58:39.724 INFO:tasks.rados.rados.0.smithi045.stdout:3880: seq_num 1185 ranges {10002432=655360} 2021-07-31T01:58:39.783 INFO:tasks.rados.rados.0.smithi045.stdout:3880: writing smithi04569269-2 from 10002432 to 10657792 tid 1 2021-07-31T01:58:39.783 INFO:tasks.rados.rados.0.smithi045.stdout:3881: delete oid 29 current snap is 383 2021-07-31T01:58:39.784 INFO:tasks.rados.rados.0.smithi045.stdout:3876: finishing write tid 1 to smithi04569269-43 2021-07-31T01:58:39.784 INFO:tasks.rados.rados.0.smithi045.stdout:3876: finishing write tid 2 to smithi04569269-43 2021-07-31T01:58:39.784 INFO:tasks.rados.rados.0.smithi045.stdout:3876: finishing write tid 3 to smithi04569269-43 2021-07-31T01:58:39.784 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 43 v 2810 (ObjNum 1182 snap 383 seq_num 1182) dirty exists 2021-07-31T01:58:39.784 INFO:tasks.rados.rados.0.smithi045.stdout:3876: left oid 43 (ObjNum 1182 snap 383 seq_num 1182) 2021-07-31T01:58:39.785 INFO:tasks.rados.rados.0.smithi045.stdout:3879: finishing write tid 1 to smithi04569269-16 2021-07-31T01:58:39.785 INFO:tasks.rados.rados.0.smithi045.stdout:3879: finishing write tid 2 to smithi04569269-16 2021-07-31T01:58:39.785 INFO:tasks.rados.rados.0.smithi045.stdout:3879: finishing write tid 3 to smithi04569269-16 2021-07-31T01:58:39.785 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 16 v 1864 (ObjNum 1184 snap 383 seq_num 1184) dirty exists 2021-07-31T01:58:39.786 INFO:tasks.rados.rados.0.smithi045.stdout:3879: left oid 16 (ObjNum 1184 snap 383 seq_num 1184) 2021-07-31T01:58:39.787 INFO:tasks.rados.rados.0.smithi045.stdout:3876: done (3 left) 2021-07-31T01:58:39.788 INFO:tasks.rados.rados.0.smithi045.stdout:3879: done (2 left) 2021-07-31T01:58:39.788 INFO:tasks.rados.rados.0.smithi045.stdout:3881: done (1 left) 2021-07-31T01:58:39.788 INFO:tasks.rados.rados.0.smithi045.stdout:3882: snap_create 2021-07-31T01:58:39.788 INFO:tasks.rados.rados.0.smithi045.stdout:3880: finishing write tid 1 to smithi04569269-2 2021-07-31T01:58:39.788 INFO:tasks.rados.rados.0.smithi045.stdout:3880: finishing write tid 2 to smithi04569269-2 2021-07-31T01:58:39.789 INFO:tasks.rados.rados.0.smithi045.stdout:3880: finishing write tid 3 to smithi04569269-2 2021-07-31T01:58:39.789 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 2 v 2381 (ObjNum 1185 snap 383 seq_num 1185) dirty exists 2021-07-31T01:58:39.789 INFO:tasks.rados.rados.0.smithi045.stdout:3880: left oid 2 (ObjNum 1185 snap 383 seq_num 1185) 2021-07-31T01:58:40.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:39 smithi154 conmon[34553]: cluster 2021-07-31T01:58:38.705868+0000 mon.a (mon.0) 1474 : cluster [DBG] osdmap e801: 8 total, 8 up, 8 in 2021-07-31T01:58:40.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:39 smithi154 conmon[34553]: 2021-07-31T01:58:40.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:39 smithi045 conmon[39718]: cluster 2021-07-31T01:58:38.705868+0000 mon.a (mon.0) 1474 : cluster [DBG] osdmap e801: 8 total, 8 up, 8 in 2021-07-31T01:58:40.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:39 smithi045 conmon[33655]: cluster 2021-07-31T01:58:38.705868+0000 mon.a (mon.0 2021-07-31T01:58:40.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:39 smithi045 conmon[33655]: ) 1474 : cluster [DBG] osdmap e801: 8 total, 8 up, 8 in 2021-07-31T01:58:40.718 INFO:tasks.rados.rados.0.smithi045.stdout:3880: done (1 left) 2021-07-31T01:58:40.719 INFO:tasks.rados.rados.0.smithi045.stdout:3882: done (0 left) 2021-07-31T01:58:40.719 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 42 current snap is 384 2021-07-31T01:58:40.719 INFO:tasks.rados.rados.0.smithi045.stdout:3883: seq_num 1187 ranges {0=688128} 2021-07-31T01:58:40.722 INFO:tasks.rados.rados.0.smithi045.stdout:3883: writing smithi04569269-42 from 0 to 688128 tid 1 2021-07-31T01:58:40.722 INFO:tasks.rados.rados.0.smithi045.stdout:3884: rmattr oid 22 current snap is 384 2021-07-31T01:58:40.723 INFO:tasks.rados.rados.0.smithi045.stdout:3884: done (1 left) 2021-07-31T01:58:40.723 INFO:tasks.rados.rados.0.smithi045.stdout:3885: read oid 49 snap 303 2021-07-31T01:58:40.723 INFO:tasks.rados.rados.0.smithi045.stdout:3885: expect (ObjNum 980 snap 303 seq_num 980) 2021-07-31T01:58:40.723 INFO:tasks.rados.rados.0.smithi045.stdout:3886: delete oid 11 current snap is 384 2021-07-31T01:58:40.729 INFO:tasks.rados.rados.0.smithi045.stdout:3886: done (2 left) 2021-07-31T01:58:40.729 INFO:tasks.rados.rados.0.smithi045.stdout:3887: snap_remove snap 366 2021-07-31T01:58:41.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:40 smithi154 conmon[34553]: cluster 2021-07-31T01:58:39.711191+0000 mon.a (mon.0) 1475 2021-07-31T01:58:41.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:40 smithi154 conmon[34553]: : cluster [DBG] osdmap e802: 8 total, 8 up, 8 in 2021-07-31T01:58:41.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:40 smithi154 conmon[34553]: cluster 2021-07-31T01:58:39.956746+0000 mgr.x (mgr.24542) 491 : cluster [DBG] pgmap v992: 25 pgs: 25 active+clean; 703 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.8 MiB/s rd, 546 KiB/s wr, 8 op/s 2021-07-31T01:58:41.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:40 smithi045 conmon[39718]: cluster 2021-07-31T01:58:39.711191+0000 mon.a (mon.0) 1475 : cluster 2021-07-31T01:58:41.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:40 smithi045 conmon[39718]: [DBG] osdmap e802: 8 total, 8 up, 8 in 2021-07-31T01:58:41.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:40 smithi045 conmon[39718]: cluster 2021-07-31T01:58:39.956746+0000 mgr.x (mgr.24542) 491 : cluster [DBG] pgmap v992: 25 pgs: 25 active+clean; 703 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.8 MiB/s rd, 546 KiB/s wr, 8 op/s 2021-07-31T01:58:41.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:40 smithi045 conmon[33655]: cluster 2021-07-31T01:58:39.711191+0000 2021-07-31T01:58:41.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:40 smithi045 conmon[33655]: mon.a (mon.0) 1475 : cluster [DBG] osdmap e802: 8 total, 8 up, 8 in 2021-07-31T01:58:41.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:40 smithi045 conmon[33655]: cluster 2021-07-31T01:58:39.956746+0000 mgr.x (mgr.24542) 491 : cluster [DBG] pgmap v992: 25 pgs: 25 active+clean; 703 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.8 MiB/s rd, 546 KiB/s wr, 8 op/s 2021-07-31T01:58:41.724 INFO:tasks.rados.rados.0.smithi045.stdout:3887: done (2 left) 2021-07-31T01:58:41.725 INFO:tasks.rados.rados.0.smithi045.stdout:3888: rollback oid 23 current snap is 384 2021-07-31T01:58:41.725 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 23 to 373 2021-07-31T01:58:41.725 INFO:tasks.rados.rados.0.smithi045.stdout:3889: snap_remove snap 316 2021-07-31T01:58:42.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:41 smithi154 conmon[34553]: cluster 2021-07-31T01:58:40.717474+0000 mon.a (mon.0) 1476 : cluster [DBG] osdmap e803: 8 total, 8 up, 8 in 2021-07-31T01:58:42.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:41 smithi045 conmon[39718]: cluster 2021-07-31T01:58:40.717474+0000 mon.a (mon.0) 2021-07-31T01:58:42.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:41 smithi045 conmon[39718]: 1476 : cluster [DBG] osdmap e803: 8 total, 8 up, 8 in 2021-07-31T01:58:42.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:41 smithi045 conmon[33655]: cluster 2021-07-31T01:58:40.717474+0000 mon.a (mon.0 2021-07-31T01:58:42.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:41 smithi045 conmon[33655]: ) 1476 : cluster [DBG] osdmap e803: 8 total, 8 up, 8 in 2021-07-31T01:58:42.731 INFO:tasks.rados.rados.0.smithi045.stdout:3889: done (3 left) 2021-07-31T01:58:42.732 INFO:tasks.rados.rados.0.smithi045.stdout:3890: delete oid 12 current snap is 384 2021-07-31T01:58:42.732 INFO:tasks.rados.rados.0.smithi045.stdout:3883: finishing write tid 1 to smithi04569269-42 2021-07-31T01:58:42.732 INFO:tasks.rados.rados.0.smithi045.stdout:3883: finishing write tid 2 to smithi04569269-42 2021-07-31T01:58:42.732 INFO:tasks.rados.rados.0.smithi045.stdout:3883: finishing write tid 3 to smithi04569269-42 2021-07-31T01:58:42.733 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 42 v 2390 (ObjNum 1187 snap 384 seq_num 1187) dirty exists 2021-07-31T01:58:42.733 INFO:tasks.rados.rados.0.smithi045.stdout:3883: left oid 42 (ObjNum 1187 snap 384 seq_num 1187) 2021-07-31T01:58:42.733 INFO:tasks.rados.rados.0.smithi045.stdout:3885: expect (ObjNum 980 snap 303 seq_num 980) 2021-07-31T01:58:43.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:42 smithi154 conmon[34553]: cluster 2021-07-31T01:58:41.722720+0000 mon.a (mon.0) 1477 : cluster [DBG] osdmap e804: 8 total, 8 up, 8 in 2021-07-31T01:58:43.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:42 smithi154 conmon[34553]: cluster 2021-07-31T01:58:41.957182+0000 mgr.x (mgr.24542) 492 : cluster [DBG] pgmap v995: 25 pgs: 25 active+clean; 703 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.8 MiB/s rd, 546 KiB/s wr, 8 op/s 2021-07-31T01:58:43.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:42 smithi045 conmon[39718]: cluster 2021-07-31T01:58:41.722720+0000 2021-07-31T01:58:43.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:42 smithi045 conmon[39718]: mon.a (mon.0) 1477 : cluster [DBG] osdmap e804: 8 total, 8 up, 8 in 2021-07-31T01:58:43.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:42 smithi045 conmon[39718]: cluster 2021-07-31T01:58: 2021-07-31T01:58:43.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:42 smithi045 conmon[39718]: 41.957182+0000 mgr.x (mgr.24542) 492 : cluster [DBG] pgmap v995: 25 pgs: 25 active+clean; 703 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.8 MiB/s rd, 546 KiB/s wr, 8 op/s 2021-07-31T01:58:43.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:42 smithi045 conmon[33655]: cluster 2021-07-31T01:58:41.722720+0000 mon.a (mon.0) 1477 2021-07-31T01:58:43.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:42 smithi045 conmon[33655]: : cluster [DBG] osdmap e804: 8 total, 8 up, 8 in 2021-07-31T01:58:43.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:42 smithi045 conmon[33655]: cluster 2021-07-31T01:58:41.957182+0000 mgr.x (mgr.24542) 492 : 2021-07-31T01:58:43.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:42 smithi045 conmon[33655]: cluster [DBG] pgmap v995: 25 pgs: 25 active+clean; 703 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.8 MiB/s rd, 546 KiB/s wr, 8 op/s 2021-07-31T01:58:44.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:43 smithi154 conmon[34553]: cluster 2021-07-31T01:58:42.728895+0000 mon.a (mon.0) 1478 : cluster [DBG] osdmap e805: 8 total, 8 up, 8 in 2021-07-31T01:58:44.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:43 smithi045 conmon[33655]: cluster 2021-07-31T01:58:42.728895+0000 mon.a ( 2021-07-31T01:58:44.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:43 smithi045 conmon[33655]: mon.0) 1478 : cluster [DBG] osdmap e805: 8 total, 8 up, 8 in 2021-07-31T01:58:44.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:43 smithi045 conmon[39718]: cluster 2021-07-31T01:58:42.728895 2021-07-31T01:58:44.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:43 smithi045 conmon[39718]: +0000 mon.a (mon.0) 1478 : cluster [DBG] osdmap e805: 8 total, 8 up, 8 in 2021-07-31T01:58:44.027 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:58:43 smithi045 conmon[61210]: level=error ts=2021-07-31T01:58:43.529Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:58:44.028 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:58:43 smithi045 conmon[61210]: level=error ts=2021-07-31T01:58:43.529Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:58:44.028 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:58:43 smithi045 conmon[61210]: level=error ts=2021-07-31T01:58:43.529Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T01:58:44.647 INFO:tasks.rados.rados.0.smithi045.stdout:3888: finishing rollback tid 1 to smithi04569269-23 2021-07-31T01:58:44.648 INFO:tasks.rados.rados.0.smithi045.stdout:3888: finishing rollback tid 2 to smithi04569269-23 2021-07-31T01:58:44.648 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 23 v 2105 (ObjNum 1049 snap 342 seq_num 1049) dirty exists 2021-07-31T01:58:44.648 INFO:tasks.rados.rados.0.smithi045.stdout:3883: done (3 left) 2021-07-31T01:58:44.648 INFO:tasks.rados.rados.0.smithi045.stdout:3885: done (2 left) 2021-07-31T01:58:44.649 INFO:tasks.rados.rados.0.smithi045.stdout:3888: done (1 left) 2021-07-31T01:58:44.649 INFO:tasks.rados.rados.0.smithi045.stdout:3890: done (0 left) 2021-07-31T01:58:44.649 INFO:tasks.rados.rados.0.smithi045.stdout:3891: delete oid 18 current snap is 384 2021-07-31T01:58:44.650 INFO:tasks.rados.rados.0.smithi045.stdout:3891: done (0 left) 2021-07-31T01:58:44.650 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 50 current snap is 384 2021-07-31T01:58:44.651 INFO:tasks.rados.rados.0.smithi045.stdout:3892: seq_num 1191 ranges {4964352=753664} 2021-07-31T01:58:44.682 INFO:tasks.rados.rados.0.smithi045.stdout:3892: writing smithi04569269-50 from 4964352 to 5718016 tid 1 2021-07-31T01:58:44.682 INFO:tasks.rados.rados.0.smithi045.stdout:3893: copy_from oid 35 from oid 17 current snap is 384 2021-07-31T01:58:44.683 INFO:tasks.rados.rados.0.smithi045.stdout:3894: copy_from oid 41 from oid 15 current snap is 384 2021-07-31T01:58:44.683 INFO:tasks.rados.rados.0.smithi045.stdout:3895: rollback oid 48 current snap is 384 2021-07-31T01:58:44.683 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 48 to 303 2021-07-31T01:58:44.684 INFO:tasks.rados.rados.0.smithi045.stdout:3896: snap_remove snap 380 2021-07-31T01:58:44.736 INFO:tasks.rados.rados.0.smithi045.stdout:3895: finishing rollback tid 0 to smithi04569269-48 2021-07-31T01:58:44.736 INFO:tasks.rados.rados.0.smithi045.stdout:3895: finishing rollback tid 1 to smithi04569269-48 2021-07-31T01:58:44.736 INFO:tasks.rados.rados.0.smithi045.stdout:3895: finishing rollback tid 2 to smithi04569269-48 2021-07-31T01:58:44.737 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 48 v 1906 (ObjNum 977 snap 299 seq_num 977) dirty exists 2021-07-31T01:58:44.737 INFO:tasks.rados.rados.0.smithi045.stdout:3892: finishing write tid 1 to smithi04569269-50 2021-07-31T01:58:44.737 INFO:tasks.rados.rados.0.smithi045.stdout:3892: finishing write tid 2 to smithi04569269-50 2021-07-31T01:58:44.737 INFO:tasks.rados.rados.0.smithi045.stdout:3894: finishing copy_from to smithi04569269-41 2021-07-31T01:58:44.738 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 41 v 1859 (ObjNum 944 snap 286 seq_num 944) dirty exists 2021-07-31T01:58:44.738 INFO:tasks.rados.rados.0.smithi045.stdout:3894: finishing copy_from racing read to smithi04569269-41 2021-07-31T01:58:44.738 INFO:tasks.rados.rados.0.smithi045.stdout:3892: finishing write tid 3 to smithi04569269-50 2021-07-31T01:58:44.738 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 50 v 2392 (ObjNum 1191 snap 384 seq_num 1191) dirty exists 2021-07-31T01:58:44.739 INFO:tasks.rados.rados.0.smithi045.stdout:3892: left oid 50 (ObjNum 1191 snap 384 seq_num 1191) 2021-07-31T01:58:44.739 INFO:tasks.rados.rados.0.smithi045.stdout:3893: finishing copy_from to smithi04569269-35 2021-07-31T01:58:44.739 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 35 v 2381 (ObjNum 1166 snap 374 seq_num 1166) dirty exists 2021-07-31T01:58:44.739 INFO:tasks.rados.rados.0.smithi045.stdout:3893: finishing copy_from racing read to smithi04569269-35 2021-07-31T01:58:44.739 INFO:tasks.rados.rados.0.smithi045.stdout:3892: done (4 left) 2021-07-31T01:58:44.740 INFO:tasks.rados.rados.0.smithi045.stdout:3893: done (3 left) 2021-07-31T01:58:44.740 INFO:tasks.rados.rados.0.smithi045.stdout:3894: done (2 left) 2021-07-31T01:58:44.740 INFO:tasks.rados.rados.0.smithi045.stdout:3895: done (1 left) 2021-07-31T01:58:44.740 INFO:tasks.rados.rados.0.smithi045.stdout:3896: done (0 left) 2021-07-31T01:58:44.741 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 3 current snap is 384 2021-07-31T01:58:44.741 INFO:tasks.rados.rados.0.smithi045.stdout:3897: seq_num 1192 ranges {9854976=770048} 2021-07-31T01:58:44.796 INFO:tasks.rados.rados.0.smithi045.stdout:3897: writing smithi04569269-3 from 9854976 to 10625024 tid 1 2021-07-31T01:58:44.796 INFO:tasks.rados.rados.0.smithi045.stdout:3898: rollback oid 49 current snap is 384 2021-07-31T01:58:44.797 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 49 to 374 2021-07-31T01:58:44.797 INFO:tasks.rados.rados.0.smithi045.stdout:3899: read oid 44 snap -1 2021-07-31T01:58:44.798 INFO:tasks.rados.rados.0.smithi045.stdout:3899: expect (ObjNum 955 snap 289 seq_num 955) 2021-07-31T01:58:44.798 INFO:tasks.rados.rados.0.smithi045.stdout:3900: copy_from oid 45 from oid 16 current snap is 384 2021-07-31T01:58:44.798 INFO:tasks.rados.rados.0.smithi045.stdout:3901: copy_from oid 5 from oid 30 current snap is 384 2021-07-31T01:58:44.799 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 35 current snap is 384 2021-07-31T01:58:44.799 INFO:tasks.rados.rados.0.smithi045.stdout:3902: seq_num 1193 ranges {5709824=786432} 2021-07-31T01:58:44.832 INFO:tasks.rados.rados.0.smithi045.stdout:3902: writing smithi04569269-35 from 5709824 to 6496256 tid 1 2021-07-31T01:58:44.832 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 2 current snap is 384 2021-07-31T01:58:44.833 INFO:tasks.rados.rados.0.smithi045.stdout:3903: seq_num 1194 ranges {10657792=802816} 2021-07-31T01:58:44.894 INFO:tasks.rados.rados.0.smithi045.stdout:3903: writing smithi04569269-2 from 10657792 to 11460608 tid 1 2021-07-31T01:58:44.894 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 26 current snap is 384 2021-07-31T01:58:44.894 INFO:tasks.rados.rados.0.smithi045.stdout:3904: seq_num 1195 ranges {4620288=417792,5038080=401408} 2021-07-31T01:58:44.921 INFO:tasks.rados.rados.0.smithi045.stdout:3904: writing smithi04569269-26 from 4620288 to 5038080 tid 1 2021-07-31T01:58:44.924 INFO:tasks.rados.rados.0.smithi045.stdout:3904: writing smithi04569269-26 from 5038080 to 5439488 tid 2 2021-07-31T01:58:44.924 INFO:tasks.rados.rados.0.smithi045.stdout:3905: delete oid 9 current snap is 384 2021-07-31T01:58:44.924 INFO:tasks.rados.rados.0.smithi045.stdout:3901: finishing copy_from to smithi04569269-5 2021-07-31T01:58:44.924 INFO:tasks.rados.rados.0.smithi045.stdout:3901: got expected ENOENT (src dne) 2021-07-31T01:58:44.925 INFO:tasks.rados.rados.0.smithi045.stdout:3901: finishing copy_from racing read to smithi04569269-5 2021-07-31T01:58:44.925 INFO:tasks.rados.rados.0.smithi045.stdout:3897: finishing write tid 1 to smithi04569269-3 2021-07-31T01:58:44.925 INFO:tasks.rados.rados.0.smithi045.stdout:3897: finishing write tid 2 to smithi04569269-3 2021-07-31T01:58:44.925 INFO:tasks.rados.rados.0.smithi045.stdout:3898: finishing rollback tid 0 to smithi04569269-49 2021-07-31T01:58:44.926 INFO:tasks.rados.rados.0.smithi045.stdout:3898: finishing rollback tid 1 to smithi04569269-49 2021-07-31T01:58:44.926 INFO:tasks.rados.rados.0.smithi045.stdout:3897: finishing write tid 3 to smithi04569269-3 2021-07-31T01:58:44.926 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 3 v 2950 (ObjNum 1192 snap 384 seq_num 1192) dirty exists 2021-07-31T01:58:44.926 INFO:tasks.rados.rados.0.smithi045.stdout:3897: left oid 3 (ObjNum 1192 snap 384 seq_num 1192) 2021-07-31T01:58:44.927 INFO:tasks.rados.rados.0.smithi045.stdout:3898: finishing rollback tid 2 to smithi04569269-49 2021-07-31T01:58:44.927 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 49 v 2954 (ObjNum 1090 snap 349 seq_num 1090) dirty exists 2021-07-31T01:58:44.927 INFO:tasks.rados.rados.0.smithi045.stdout:3899: expect (ObjNum 955 snap 289 seq_num 955) 2021-07-31T01:58:45.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:44 smithi154 conmon[34553]: cluster 2021-07-31T01:58:43.957914+0000 mgr.x (mgr.24542 2021-07-31T01:58:45.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:44 smithi154 conmon[34553]: ) 493 : cluster [DBG] pgmap v997: 25 pgs: 25 active+clean; 702 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 11 MiB/s rd, 456 KiB/s wr, 3 op/s 2021-07-31T01:58:45.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:44 smithi045 conmon[33655]: cluster 2021-07-31T01:58:43.957914+0000 mgr.x (mgr.24542 2021-07-31T01:58:45.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:44 smithi045 conmon[33655]: ) 493 : cluster [DBG] pgmap v997: 25 pgs: 25 active+clean; 702 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 11 MiB/s rd, 456 KiB/s wr, 3 op/s 2021-07-31T01:58:45.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:44 smithi045 conmon[39718]: cluster 2021-07-31T01:58:43.957914 2021-07-31T01:58:45.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:44 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 493 : cluster [DBG] pgmap v997: 25 pgs: 25 active+clean; 702 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 11 MiB/s rd, 456 KiB/s wr, 3 op/s 2021-07-31T01:58:45.043 INFO:tasks.rados.rados.0.smithi045.stdout:3900: finishing copy_from to smithi04569269-45 2021-07-31T01:58:45.043 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 45 v 1864 (ObjNum 1184 snap 383 seq_num 1184) dirty exists 2021-07-31T01:58:45.043 INFO:tasks.rados.rados.0.smithi045.stdout:3900: finishing copy_from racing read to smithi04569269-45 2021-07-31T01:58:45.043 INFO:tasks.rados.rados.0.smithi045.stdout:3902: finishing write tid 1 to smithi04569269-35 2021-07-31T01:58:45.044 INFO:tasks.rados.rados.0.smithi045.stdout:3902: finishing write tid 2 to smithi04569269-35 2021-07-31T01:58:45.044 INFO:tasks.rados.rados.0.smithi045.stdout:3902: finishing write tid 3 to smithi04569269-35 2021-07-31T01:58:45.044 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 35 v 2956 (ObjNum 1193 snap 384 seq_num 1193) dirty exists 2021-07-31T01:58:45.044 INFO:tasks.rados.rados.0.smithi045.stdout:3902: left oid 35 (ObjNum 1193 snap 384 seq_num 1193) 2021-07-31T01:58:45.045 INFO:tasks.rados.rados.0.smithi045.stdout:3903: finishing write tid 1 to smithi04569269-2 2021-07-31T01:58:45.045 INFO:tasks.rados.rados.0.smithi045.stdout:3903: finishing write tid 2 to smithi04569269-2 2021-07-31T01:58:45.045 INFO:tasks.rados.rados.0.smithi045.stdout:3903: finishing write tid 3 to smithi04569269-2 2021-07-31T01:58:45.045 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 2 v 2383 (ObjNum 1194 snap 384 seq_num 1194) dirty exists 2021-07-31T01:58:45.045 INFO:tasks.rados.rados.0.smithi045.stdout:3903: left oid 2 (ObjNum 1194 snap 384 seq_num 1194) 2021-07-31T01:58:45.046 INFO:tasks.rados.rados.0.smithi045.stdout:3897: done (8 left) 2021-07-31T01:58:45.046 INFO:tasks.rados.rados.0.smithi045.stdout:3898: done (7 left) 2021-07-31T01:58:45.046 INFO:tasks.rados.rados.0.smithi045.stdout:3899: done (6 left) 2021-07-31T01:58:45.046 INFO:tasks.rados.rados.0.smithi045.stdout:3900: done (5 left) 2021-07-31T01:58:45.047 INFO:tasks.rados.rados.0.smithi045.stdout:3901: done (4 left) 2021-07-31T01:58:45.047 INFO:tasks.rados.rados.0.smithi045.stdout:3902: done (3 left) 2021-07-31T01:58:45.047 INFO:tasks.rados.rados.0.smithi045.stdout:3903: done (2 left) 2021-07-31T01:58:45.047 INFO:tasks.rados.rados.0.smithi045.stdout:3905: done (1 left) 2021-07-31T01:58:45.048 INFO:tasks.rados.rados.0.smithi045.stdout:3906: snap_remove snap 331 2021-07-31T01:58:45.740 INFO:tasks.rados.rados.0.smithi045.stdout:3906: done (1 left) 2021-07-31T01:58:45.741 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 3 current snap is 384 2021-07-31T01:58:45.741 INFO:tasks.rados.rados.0.smithi045.stdout:3904: finishing write tid 1 to smithi04569269-26 2021-07-31T01:58:45.741 INFO:tasks.rados.rados.0.smithi045.stdout:3904: finishing write tid 2 to smithi04569269-26 2021-07-31T01:58:45.742 INFO:tasks.rados.rados.0.smithi045.stdout:3904: finishing write tid 3 to smithi04569269-26 2021-07-31T01:58:45.742 INFO:tasks.rados.rados.0.smithi045.stdout:3904: finishing write tid 4 to smithi04569269-26 2021-07-31T01:58:45.742 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 26 v 1890 (ObjNum 1195 snap 384 seq_num 1195) dirty exists 2021-07-31T01:58:45.742 INFO:tasks.rados.rados.0.smithi045.stdout:3904: left oid 26 (ObjNum 1195 snap 384 seq_num 1195) 2021-07-31T01:58:45.742 INFO:tasks.rados.rados.0.smithi045.stdout:3907: seq_num 1197 ranges {10625024=450560,11075584=401408} 2021-07-31T01:58:45.777 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T01:58:45.802 INFO:tasks.rados.rados.0.smithi045.stdout:3907: writing smithi04569269-3 from 10625024 to 11075584 tid 1 2021-07-31T01:58:45.805 INFO:tasks.rados.rados.0.smithi045.stdout:3907: writing smithi04569269-3 from 11075584 to 11476992 tid 2 2021-07-31T01:58:45.805 INFO:tasks.rados.rados.0.smithi045.stdout:3904: done (1 left) 2021-07-31T01:58:45.805 INFO:tasks.rados.rados.0.smithi045.stdout:3908: read oid 39 snap -1 2021-07-31T01:58:45.806 INFO:tasks.rados.rados.0.smithi045.stdout:3908: expect deleted 2021-07-31T01:58:45.806 INFO:tasks.rados.rados.0.smithi045.stdout:3909: read oid 8 snap 363 2021-07-31T01:58:45.806 INFO:tasks.rados.rados.0.smithi045.stdout:3909: expect deleted 2021-07-31T01:58:45.806 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 4 current snap is 384 2021-07-31T01:58:45.806 INFO:tasks.rados.rados.0.smithi045.stdout:3910: seq_num 1198 ranges {3334144=466944,3801088=401408} 2021-07-31T01:58:45.825 INFO:tasks.rados.rados.0.smithi045.stdout:3910: writing smithi04569269-4 from 3334144 to 3801088 tid 1 2021-07-31T01:58:45.828 INFO:tasks.rados.rados.0.smithi045.stdout:3910: writing smithi04569269-4 from 3801088 to 4202496 tid 2 2021-07-31T01:58:45.828 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 6 current snap is 384 2021-07-31T01:58:45.828 INFO:tasks.rados.rados.0.smithi045.stdout:3911: seq_num 1199 ranges {0=483328,483328=401408} 2021-07-31T01:58:45.830 INFO:tasks.rados.rados.0.smithi045.stdout:3911: writing smithi04569269-6 from 0 to 483328 tid 1 2021-07-31T01:58:45.833 INFO:tasks.rados.rados.0.smithi045.stdout:3911: writing smithi04569269-6 from 483328 to 884736 tid 2 2021-07-31T01:58:45.834 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 35 current snap is 384 2021-07-31T01:58:45.834 INFO:tasks.rados.rados.0.smithi045.stdout:3907: finishing write tid 1 to smithi04569269-3 2021-07-31T01:58:45.834 INFO:tasks.rados.rados.0.smithi045.stdout:3907: finishing write tid 2 to smithi04569269-3 2021-07-31T01:58:45.834 INFO:tasks.rados.rados.0.smithi045.stdout:3912: seq_num 1200 ranges {6496256=499712,6995968=401408} 2021-07-31T01:58:45.871 INFO:tasks.rados.rados.0.smithi045.stdout:3912: writing smithi04569269-35 from 6496256 to 6995968 tid 1 2021-07-31T01:58:45.874 INFO:tasks.rados.rados.0.smithi045.stdout:3912: writing smithi04569269-35 from 6995968 to 7397376 tid 2 2021-07-31T01:58:45.874 INFO:tasks.rados.rados.0.smithi045.stdout:3908: done (5 left) 2021-07-31T01:58:45.875 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 33 current snap is 384 2021-07-31T01:58:45.875 INFO:tasks.rados.rados.0.smithi045.stdout:3907: finishing write tid 3 to smithi04569269-3 2021-07-31T01:58:45.875 INFO:tasks.rados.rados.0.smithi045.stdout:3907: finishing write tid 4 to smithi04569269-3 2021-07-31T01:58:45.875 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 3 v 2971 (ObjNum 1197 snap 384 seq_num 1197) dirty exists 2021-07-31T01:58:45.876 INFO:tasks.rados.rados.0.smithi045.stdout:3907: left oid 3 (ObjNum 1197 snap 384 seq_num 1197) 2021-07-31T01:58:45.876 INFO:tasks.rados.rados.0.smithi045.stdout:3913: seq_num 1201 ranges {1515520=524288,2039808=401408} 2021-07-31T01:58:45.885 INFO:tasks.rados.rados.0.smithi045.stdout:3913: writing smithi04569269-33 from 1515520 to 2039808 tid 1 2021-07-31T01:58:45.889 INFO:tasks.rados.rados.0.smithi045.stdout:3913: writing smithi04569269-33 from 2039808 to 2441216 tid 2 2021-07-31T01:58:45.889 INFO:tasks.rados.rados.0.smithi045.stdout:3907: done (5 left) 2021-07-31T01:58:45.890 INFO:tasks.rados.rados.0.smithi045.stdout:3909: done (4 left) 2021-07-31T01:58:45.890 INFO:tasks.rados.rados.0.smithi045.stdout:3914: read oid 15 snap 376 2021-07-31T01:58:45.890 INFO:tasks.rados.rados.0.smithi045.stdout:3914: expect (ObjNum 944 snap 286 seq_num 944) 2021-07-31T01:58:45.890 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 41 current snap is 384 2021-07-31T01:58:45.891 INFO:tasks.rados.rados.0.smithi045.stdout:3910: finishing write tid 1 to smithi04569269-4 2021-07-31T01:58:45.891 INFO:tasks.rados.rados.0.smithi045.stdout:3910: finishing write tid 2 to smithi04569269-4 2021-07-31T01:58:45.891 INFO:tasks.rados.rados.0.smithi045.stdout:3910: finishing write tid 3 to smithi04569269-4 2021-07-31T01:58:45.891 INFO:tasks.rados.rados.0.smithi045.stdout:3910: finishing write tid 4 to smithi04569269-4 2021-07-31T01:58:45.891 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 4 v 2543 (ObjNum 1198 snap 384 seq_num 1198) dirty exists 2021-07-31T01:58:45.892 INFO:tasks.rados.rados.0.smithi045.stdout:3910: left oid 4 (ObjNum 1198 snap 384 seq_num 1198) 2021-07-31T01:58:45.892 INFO:tasks.rados.rados.0.smithi045.stdout:3915: seq_num 1202 ranges {1417216=540672,1957888=401408} 2021-07-31T01:58:45.901 INFO:tasks.rados.rados.0.smithi045.stdout:3915: writing smithi04569269-41 from 1417216 to 1957888 tid 1 2021-07-31T01:58:45.905 INFO:tasks.rados.rados.0.smithi045.stdout:3915: writing smithi04569269-41 from 1957888 to 2359296 tid 2 2021-07-31T01:58:45.905 INFO:tasks.rados.rados.0.smithi045.stdout:3910: done (5 left) 2021-07-31T01:58:45.905 INFO:tasks.rados.rados.0.smithi045.stdout:3916: snap_create 2021-07-31T01:58:45.906 INFO:tasks.rados.rados.0.smithi045.stdout:3911: finishing write tid 1 to smithi04569269-6 2021-07-31T01:58:45.906 INFO:tasks.rados.rados.0.smithi045.stdout:3911: finishing write tid 2 to smithi04569269-6 2021-07-31T01:58:45.906 INFO:tasks.rados.rados.0.smithi045.stdout:3911: finishing write tid 3 to smithi04569269-6 2021-07-31T01:58:45.906 INFO:tasks.rados.rados.0.smithi045.stdout:3911: finishing write tid 4 to smithi04569269-6 2021-07-31T01:58:45.907 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 6 v 2375 (ObjNum 1199 snap 384 seq_num 1199) dirty exists 2021-07-31T01:58:45.907 INFO:tasks.rados.rados.0.smithi045.stdout:3911: left oid 6 (ObjNum 1199 snap 384 seq_num 1199) 2021-07-31T01:58:45.907 INFO:tasks.rados.rados.0.smithi045.stdout:3912: finishing write tid 1 to smithi04569269-35 2021-07-31T01:58:45.907 INFO:tasks.rados.rados.0.smithi045.stdout:3912: finishing write tid 2 to smithi04569269-35 2021-07-31T01:58:45.907 INFO:tasks.rados.rados.0.smithi045.stdout:3912: finishing write tid 3 to smithi04569269-35 2021-07-31T01:58:45.908 INFO:tasks.rados.rados.0.smithi045.stdout:3912: finishing write tid 4 to smithi04569269-35 2021-07-31T01:58:45.908 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 35 v 2974 (ObjNum 1200 snap 384 seq_num 1200) dirty exists 2021-07-31T01:58:45.908 INFO:tasks.rados.rados.0.smithi045.stdout:3912: left oid 35 (ObjNum 1200 snap 384 seq_num 1200) 2021-07-31T01:58:45.909 INFO:tasks.rados.rados.0.smithi045.stdout:3913: finishing write tid 1 to smithi04569269-33 2021-07-31T01:58:45.909 INFO:tasks.rados.rados.0.smithi045.stdout:3913: finishing write tid 2 to smithi04569269-33 2021-07-31T01:58:45.909 INFO:tasks.rados.rados.0.smithi045.stdout:3913: finishing write tid 3 to smithi04569269-33 2021-07-31T01:58:45.909 INFO:tasks.rados.rados.0.smithi045.stdout:3914: expect (ObjNum 944 snap 286 seq_num 944) 2021-07-31T01:58:45.947 INFO:tasks.rados.rados.0.smithi045.stdout:3913: finishing write tid 4 to smithi04569269-33 2021-07-31T01:58:45.948 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 33 v 2108 (ObjNum 1201 snap 384 seq_num 1201) dirty exists 2021-07-31T01:58:45.948 INFO:tasks.rados.rados.0.smithi045.stdout:3913: left oid 33 (ObjNum 1201 snap 384 seq_num 1201) 2021-07-31T01:58:45.948 INFO:tasks.rados.rados.0.smithi045.stdout:3915: finishing write tid 1 to smithi04569269-41 2021-07-31T01:58:45.948 INFO:tasks.rados.rados.0.smithi045.stdout:3915: finishing write tid 2 to smithi04569269-41 2021-07-31T01:58:45.949 INFO:tasks.rados.rados.0.smithi045.stdout:3915: finishing write tid 3 to smithi04569269-41 2021-07-31T01:58:45.949 INFO:tasks.rados.rados.0.smithi045.stdout:3915: finishing write tid 4 to smithi04569269-41 2021-07-31T01:58:45.949 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 41 v 2169 (ObjNum 1202 snap 384 seq_num 1202) dirty exists 2021-07-31T01:58:45.949 INFO:tasks.rados.rados.0.smithi045.stdout:3915: left oid 41 (ObjNum 1202 snap 384 seq_num 1202) 2021-07-31T01:58:46.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:45 smithi154 conmon[34553]: cluster 2021-07-31T01:58:44.733428+0000 mon.a (mon.0) 1479 : cluster [DBG] osdmap e806: 8 total, 8 up, 8 in 2021-07-31T01:58:46.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:45 smithi045 conmon[33655]: cluster 2021-07-31T01:58:44.733428+0000 mon.a 2021-07-31T01:58:46.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:45 smithi045 conmon[33655]: (mon.0) 1479 : cluster [DBG] osdmap e806: 8 total, 8 up, 8 in 2021-07-31T01:58:46.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:45 smithi045 conmon[39718]: cluster 2021-07-31T01:58:44.733428+0000 2021-07-31T01:58:46.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:45 smithi045 conmon[39718]: mon.a (mon.0) 1479 : cluster [DBG] osdmap e806: 8 total, 8 up, 8 in 2021-07-31T01:58:46.072 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T01:58:46.073 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 13m ago 14m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T01:58:46.073 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 11m ago 14m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T01:58:46.073 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 11m ago 17m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T01:58:46.074 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 13m ago 19m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T01:58:46.074 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 13m ago 19m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T01:58:46.074 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 11m ago 17m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T01:58:46.074 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 13m ago 18m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T01:58:46.074 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 13m ago 14m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T01:58:46.075 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 11m ago 14m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T01:58:46.075 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 13m ago 17m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T01:58:46.075 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 13m ago 17m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T01:58:46.075 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 13m ago 16m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T01:58:46.076 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 13m ago 16m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T01:58:46.076 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 11m ago 16m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T01:58:46.076 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 11m ago 15m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T01:58:46.076 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 11m ago 15m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T01:58:46.076 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 11m ago 15m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T01:58:46.077 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 11m ago 14m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T01:58:46.406 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T01:58:46.406 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T01:58:46.406 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T01:58:46.406 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T01:58:46.407 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T01:58:46.407 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T01:58:46.407 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T01:58:46.407 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T01:58:46.408 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T01:58:46.408 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T01:58:46.408 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T01:58:46.408 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T01:58:46.409 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T01:58:46.409 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T01:58:46.409 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T01:58:46.409 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T01:58:46.410 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T01:58:46.697 INFO:tasks.rados.rados.0.smithi045.stdout:3911: done (5 left) 2021-07-31T01:58:46.697 INFO:tasks.rados.rados.0.smithi045.stdout:3912: done (4 left) 2021-07-31T01:58:46.697 INFO:tasks.rados.rados.0.smithi045.stdout:3913: done (3 left) 2021-07-31T01:58:46.698 INFO:tasks.rados.rados.0.smithi045.stdout:3914: done (2 left) 2021-07-31T01:58:46.698 INFO:tasks.rados.rados.0.smithi045.stdout:3915: done (1 left) 2021-07-31T01:58:46.698 INFO:tasks.rados.rados.0.smithi045.stdout:3916: done (0 left) 2021-07-31T01:58:46.698 INFO:tasks.rados.rados.0.smithi045.stdout:3917: read oid 1 snap 373 2021-07-31T01:58:46.699 INFO:tasks.rados.rados.0.smithi045.stdout:3917: expect (ObjNum 1116 snap 351 seq_num 1116) 2021-07-31T01:58:46.699 INFO:tasks.rados.rados.0.smithi045.stdout:3918: snap_remove snap 370 2021-07-31T01:58:47.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:46 smithi154 conmon[34553]: cluster 2021-07-31T01:58:45.739318+0000 mon.a (mon.0) 1480 : cluster [DBG] osdmap e807: 8 total, 8 up, 8 in 2021-07-31T01:58:47.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:46 smithi154 conmon[34553]: audit 2021-07-31T01:58:45.759783+0000 mgr.x (mgr.24542) 494 : audit [DBG] from='client.15108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:58:47.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:46 smithi154 conmon[34553]: cluster 2021-07-31T01:58:45.958299+0000 mgr.x (mgr.24542) 495 : cluster [DBG] pgmap v1000: 25 pgs: 25 active+clean; 702 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 11 MiB/s rd, 457 KiB/s wr, 3 op/s 2021-07-31T01:58:47.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:46 smithi154 conmon[34553]: audit 2021-07-31T01:58:46.405739+0000 mon.c (mon.1) 35 : audit [DBG] from='client.? 172.21.15.45:0/3119728355' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T01:58:47.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:46 smithi154 conmon[34553]: 2021-07-31T01:58:47.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:46 smithi154 conmon[34553]: cluster 2021 2021-07-31T01:58:47.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:46 smithi154 conmon[34553]: -07-31T01:58:46.695964+0000 mon.a (mon.0 2021-07-31T01:58:47.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:46 smithi154 conmon[34553]: ) 1481 : cluster [DBG] osdmap e808: 8 total, 8 up, 8 in 2021-07-31T01:58:47.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:46 smithi045 conmon[39718]: cluster 2021-07-31T01:58:45.739318 2021-07-31T01:58:47.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:46 smithi045 conmon[39718]: +0000 mon.a (mon.0) 1480 : cluster [DBG] osdmap e807: 8 total, 8 up, 8 in 2021-07-31T01:58:47.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:46 smithi045 conmon[39718]: audit 2021-07-31T01:58:47.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:46 smithi045 conmon[39718]: 2021-07-31T01:58:45.759783 2021-07-31T01:58:47.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:46 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 494 : audit 2021-07-31T01:58:47.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:46 smithi045 conmon[39718]: [DBG] from='client.15108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:58:47.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:46 smithi045 conmon[39718]: cluster 2021-07-31T01:58:47.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:46 smithi045 conmon[39718]: 2021-07 2021-07-31T01:58:47.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:46 smithi045 conmon[39718]: -31T01:58:45.958299+0000 2021-07-31T01:58:47.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:46 smithi045 conmon[39718]: mgr.x (mgr.24542) 495 : cluster 2021-07-31T01:58:47.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:46 smithi045 conmon[39718]: [DBG] pgmap v1000: 25 pgs: 25 active+clean; 702 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 11 MiB/s rd, 457 KiB/s wr, 3 op/s 2021-07-31T01:58:47.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:46 smithi045 conmon[39718]: audit 2021-07-31T01:58:47.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:46 smithi045 conmon[39718]: 2021-07 2021-07-31T01:58:47.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:46 smithi045 conmon[39718]: -31T01:58:46.405739 2021-07-31T01:58:47.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:46 smithi045 conmon[39718]: +0000 mon.c (mon.1) 35 : audit 2021-07-31T01:58:47.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:46 smithi045 conmon[39718]: [DBG] from='client.? 172.21.15.45:0/3119728355' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T01:58:47.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:46 smithi045 conmon[39718]: cluster 2021-07-31T01:58:47.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:46 smithi045 conmon[39718]: 2021-07-31T01: 2021-07-31T01:58:47.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:46 smithi045 conmon[39718]: 58:46.695964+0000 mon.a ( 2021-07-31T01:58:47.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:46 smithi045 conmon[39718]: mon.0) 1481 : cluster [DBG] osdmap e808: 8 total, 8 up, 8 in 2021-07-31T01:58:47.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:46 smithi045 conmon[39718]: 2021-07-31T01:58:47.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:46 smithi045 conmon[33655]: cluster 2021-07-31T01:58:45.739318+0000 mon.a (mon. 2021-07-31T01:58:47.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:46 smithi045 conmon[33655]: 0) 1480 : cluster [DBG] osdmap e807: 8 total, 8 up, 8 in 2021-07-31T01:58:47.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:46 smithi045 conmon[33655]: audit 2021-07-31T01:58:45.759783+0000 mgr.x (mgr.24542) 494 : audit [DBG] from='client.15108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:58:47.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:46 smithi045 conmon[33655]: cluster 2021-07-31T01: 2021-07-31T01:58:47.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:46 smithi045 conmon[33655]: 58:45.958299+0000 mgr.x (mgr.24542) 495 : cluster 2021-07-31T01:58:47.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:46 smithi045 conmon[33655]: [DBG] pgmap v1000: 25 pgs: 25 active+clean; 702 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 11 MiB/s rd, 457 KiB/s wr, 3 op/s 2021-07-31T01:58:47.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:46 smithi045 conmon[33655]: audit 2021-07-31T01:58: 2021-07-31T01:58:47.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:46 smithi045 conmon[33655]: 46.405739+0000 mon.c (mon.1) 35 : audit [DBG] 2021-07-31T01:58:47.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:46 smithi045 conmon[33655]: from='client.? 172.21.15.45:0/3119728355' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T01:58:47.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:46 smithi045 conmon[33655]: cluster 2021-07-31T01:58:47.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:46 smithi045 conmon[33655]: 2021-07-31T01:58:46.695964+0000 mon.a (mon. 2021-07-31T01:58:47.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:46 smithi045 conmon[33655]: 0) 1481 : cluster [DBG] osdmap e808: 8 total, 8 up, 8 in 2021-07-31T01:58:47.703 INFO:tasks.rados.rados.0.smithi045.stdout:3918: done (1 left) 2021-07-31T01:58:47.704 INFO:tasks.rados.rados.0.smithi045.stdout:3919: delete oid 28 current snap is 385 2021-07-31T01:58:47.704 INFO:tasks.rados.rados.0.smithi045.stdout:3917: expect (ObjNum 1116 snap 351 seq_num 1116) 2021-07-31T01:58:48.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:47 smithi154 conmon[34553]: audit 2021-07-31T01:58:46.065336+0000 mgr.x (mgr.24542) 496 : audit 2021-07-31T01:58:48.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:47 smithi154 conmon[34553]: [DBG] from='client.15114 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:58:48.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:47 smithi154 conmon[34553]: cluster 2021-07-31T01:58:47.702746+0000 mon.a (mon.0) 1482 : cluster [DBG] osdmap e809: 8 total, 8 up, 8 in 2021-07-31T01:58:48.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:47 smithi045 conmon[33655]: audit 2021-07-31T01:58:46.065336+0000 mgr.x ( 2021-07-31T01:58:48.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:47 smithi045 conmon[33655]: mgr.24542) 496 : audit [DBG] from='client.15114 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:58:48.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:47 smithi045 conmon[33655]: cluster 2021-07-31T01:58:47.702746+0000 mon.a (mon. 2021-07-31T01:58:48.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:47 smithi045 conmon[33655]: 0) 1482 : cluster [DBG] osdmap e809: 8 total, 8 up, 8 in 2021-07-31T01:58:48.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:47 smithi045 conmon[39718]: audit 2021-07-31T01 2021-07-31T01:58:48.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:47 smithi045 conmon[39718]: :58:46.065336+0000 mgr.x (mgr.24542) 496 : audit [DBG] from='client.15114 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:58:48.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:47 smithi045 conmon[39718]: cluster 2021-07-31T01:58: 2021-07-31T01:58:48.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:47 smithi045 conmon[39718]: 47.702746+0000 mon.a (mon.0) 1482 : cluster [DBG] osdmap e809: 8 total, 8 up, 8 in 2021-07-31T01:58:48.193 INFO:tasks.rados.rados.0.smithi045.stdout:3917: done (1 left) 2021-07-31T01:58:48.194 INFO:tasks.rados.rados.0.smithi045.stdout:3919: done (0 left) 2021-07-31T01:58:48.194 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 44 current snap is 385 2021-07-31T01:58:48.194 INFO:tasks.rados.rados.0.smithi045.stdout:3920: seq_num 1204 ranges {3391488=573440,3964928=401408} 2021-07-31T01:58:48.215 INFO:tasks.rados.rados.0.smithi045.stdout:3920: writing smithi04569269-44 from 3391488 to 3964928 tid 1 2021-07-31T01:58:48.217 INFO:tasks.rados.rados.0.smithi045.stdout:3920: writing smithi04569269-44 from 3964928 to 4366336 tid 2 2021-07-31T01:58:48.217 INFO:tasks.rados.rados.0.smithi045.stdout:3921: snap_remove snap 376 2021-07-31T01:58:48.751 INFO:tasks.rados.rados.0.smithi045.stdout:3921: done (1 left) 2021-07-31T01:58:48.751 INFO:tasks.rados.rados.0.smithi045.stdout:3922: delete oid 50 current snap is 385 2021-07-31T01:58:48.751 INFO:tasks.rados.rados.0.smithi045.stdout:3920: finishing write tid 1 to smithi04569269-44 2021-07-31T01:58:48.752 INFO:tasks.rados.rados.0.smithi045.stdout:3920: finishing write tid 2 to smithi04569269-44 2021-07-31T01:58:48.752 INFO:tasks.rados.rados.0.smithi045.stdout:3920: finishing write tid 3 to smithi04569269-44 2021-07-31T01:58:48.752 INFO:tasks.rados.rados.0.smithi045.stdout:3920: finishing write tid 4 to smithi04569269-44 2021-07-31T01:58:48.752 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 44 v 2813 (ObjNum 1204 snap 385 seq_num 1204) dirty exists 2021-07-31T01:58:48.752 INFO:tasks.rados.rados.0.smithi045.stdout:3920: left oid 44 (ObjNum 1204 snap 385 seq_num 1204) 2021-07-31T01:58:48.765 INFO:tasks.rados.rados.0.smithi045.stdout:3920: done (1 left) 2021-07-31T01:58:48.765 INFO:tasks.rados.rados.0.smithi045.stdout:3922: done (0 left) 2021-07-31T01:58:48.765 INFO:tasks.rados.rados.0.smithi045.stdout:3923: copy_from oid 29 from oid 12 current snap is 385 2021-07-31T01:58:48.766 INFO:tasks.rados.rados.0.smithi045.stdout:3924: copy_from oid 46 from oid 11 current snap is 385 2021-07-31T01:58:48.766 INFO:tasks.rados.rados.0.smithi045.stdout:3925: read oid 17 snap 384 2021-07-31T01:58:48.766 INFO:tasks.rados.rados.0.smithi045.stdout:3925: expect (ObjNum 1166 snap 374 seq_num 1166) 2021-07-31T01:58:48.766 INFO:tasks.rados.rados.0.smithi045.stdout:3926: snap_create 2021-07-31T01:58:48.767 INFO:tasks.rados.rados.0.smithi045.stdout:3924: finishing copy_from to smithi04569269-46 2021-07-31T01:58:48.767 INFO:tasks.rados.rados.0.smithi045.stdout:3924: got expected ENOENT (src dne) 2021-07-31T01:58:48.767 INFO:tasks.rados.rados.0.smithi045.stdout:3924: finishing copy_from racing read to smithi04569269-46 2021-07-31T01:58:48.769 INFO:tasks.rados.rados.0.smithi045.stdout:3923: finishing copy_from to smithi04569269-29 2021-07-31T01:58:48.769 INFO:tasks.rados.rados.0.smithi045.stdout:3923: got expected ENOENT (src dne) 2021-07-31T01:58:48.770 INFO:tasks.rados.rados.0.smithi045.stdout:3923: finishing copy_from racing read to smithi04569269-29 2021-07-31T01:58:48.793 INFO:tasks.rados.rados.0.smithi045.stdout:3925: expect (ObjNum 1166 snap 374 seq_num 1166) 2021-07-31T01:58:49.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:48 smithi154 conmon[34553]: cluster 2021-07-31T01:58:47.958822+0000 mgr.x (mgr.24542) 497 : cluster [DBG] pgmap v1003: 25 pgs: 25 active+clean; 702 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T01:58:49.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:48 smithi154 conmon[34553]: 2021-07-31T01:58:49.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:48 smithi045 conmon[39718]: cluster 2021-07-31T01:58:47.958822+0000 mgr.x 2021-07-31T01:58:49.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:48 smithi045 conmon[39718]: (mgr.24542) 497 : cluster [DBG] pgmap v1003: 25 pgs: 25 active+clean; 702 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T01:58:49.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:48 smithi045 conmon[33655]: cluster 2021-07-31T01:58:47.958822+0000 mgr.x (mgr.24542 2021-07-31T01:58:49.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:48 smithi045 conmon[33655]: ) 497 : cluster [DBG] pgmap v1003: 25 pgs: 25 active+clean; 702 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T01:58:49.756 INFO:tasks.rados.rados.0.smithi045.stdout:3923: done (3 left) 2021-07-31T01:58:49.756 INFO:tasks.rados.rados.0.smithi045.stdout:3924: done (2 left) 2021-07-31T01:58:49.756 INFO:tasks.rados.rados.0.smithi045.stdout:3925: done (1 left) 2021-07-31T01:58:49.757 INFO:tasks.rados.rados.0.smithi045.stdout:3926: done (0 left) 2021-07-31T01:58:49.757 INFO:tasks.rados.rados.0.smithi045.stdout:3927: snap_remove snap 325 2021-07-31T01:58:50.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:49 smithi154 conmon[34553]: cluster 2021-07-31T01:58:48.750261+0000 mon.a (mon.0) 1483 : 2021-07-31T01:58:50.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:49 smithi154 conmon[34553]: cluster [DBG] osdmap e810: 8 total, 8 up, 8 in 2021-07-31T01:58:50.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:49 smithi045 conmon[33655]: cluster 2021-07-31T01:58:48.750261+0000 mon.a (mon.0) 1483 : cluster 2021-07-31T01:58:50.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:49 smithi045 conmon[33655]: [DBG] osdmap e810: 8 total, 8 up, 8 in 2021-07-31T01:58:50.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:49 smithi045 conmon[39718]: cluster 2021-07-31T01:58:48.750261+0000 2021-07-31T01:58:50.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:49 smithi045 conmon[39718]: mon.a (mon.0) 1483 : cluster [DBG] osdmap e810: 8 total, 8 up, 8 in 2021-07-31T01:58:50.763 INFO:tasks.rados.rados.0.smithi045.stdout:3927: done (0 left) 2021-07-31T01:58:50.764 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 28 current snap is 386 2021-07-31T01:58:50.764 INFO:tasks.rados.rados.0.smithi045.stdout:3928: seq_num 1206 ranges {0=606208,606208=401408} 2021-07-31T01:58:50.767 INFO:tasks.rados.rados.0.smithi045.stdout:3928: writing smithi04569269-28 from 0 to 606208 tid 1 2021-07-31T01:58:50.770 INFO:tasks.rados.rados.0.smithi045.stdout:3928: writing smithi04569269-28 from 606208 to 1007616 tid 2 2021-07-31T01:58:50.770 INFO:tasks.rados.rados.0.smithi045.stdout:3929: rollback oid 22 current snap is 386 2021-07-31T01:58:50.771 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 22 to 348 2021-07-31T01:58:50.771 INFO:tasks.rados.rados.0.smithi045.stdout:3930: read oid 30 snap -1 2021-07-31T01:58:50.771 INFO:tasks.rados.rados.0.smithi045.stdout:3930: expect deleted 2021-07-31T01:58:50.771 INFO:tasks.rados.rados.0.smithi045.stdout:3931: read oid 49 snap -1 2021-07-31T01:58:50.772 INFO:tasks.rados.rados.0.smithi045.stdout:3931: expect (ObjNum 1090 snap 349 seq_num 1090) 2021-07-31T01:58:50.772 INFO:tasks.rados.rados.0.smithi045.stdout:3932: rmattr oid 39 current snap is 386 2021-07-31T01:58:50.772 INFO:tasks.rados.rados.0.smithi045.stdout:3932: done (4 left) 2021-07-31T01:58:50.772 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 15 current snap is 386 2021-07-31T01:58:50.773 INFO:tasks.rados.rados.0.smithi045.stdout:3933: seq_num 1207 ranges {1417216=622592,2039808=401408} 2021-07-31T01:58:50.782 INFO:tasks.rados.rados.0.smithi045.stdout:3933: writing smithi04569269-15 from 1417216 to 2039808 tid 1 2021-07-31T01:58:50.786 INFO:tasks.rados.rados.0.smithi045.stdout:3933: writing smithi04569269-15 from 2039808 to 2441216 tid 2 2021-07-31T01:58:50.786 INFO:tasks.rados.rados.0.smithi045.stdout:3934: read oid 27 snap 373 2021-07-31T01:58:50.786 INFO:tasks.rados.rados.0.smithi045.stdout:3934: expect (ObjNum -1103377216 snap 21873 seq_num 3189089280) 2021-07-31T01:58:50.787 INFO:tasks.rados.rados.0.smithi045.stdout:3935: read oid 39 snap -1 2021-07-31T01:58:50.787 INFO:tasks.rados.rados.0.smithi045.stdout:3935: expect deleted 2021-07-31T01:58:50.787 INFO:tasks.rados.rados.0.smithi045.stdout:3936: delete oid 13 current snap is 386 2021-07-31T01:58:50.787 INFO:tasks.rados.rados.0.smithi045.stdout:3929: finishing rollback tid 0 to smithi04569269-22 2021-07-31T01:58:50.788 INFO:tasks.rados.rados.0.smithi045.stdout:3929: finishing rollback tid 1 to smithi04569269-22 2021-07-31T01:58:50.788 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 22 v 1866 (ObjNum -1105218192 snap 21873 seq_num 2885683408) dirty dne 2021-07-31T01:58:50.788 INFO:tasks.rados.rados.0.smithi045.stdout:3928: finishing write tid 1 to smithi04569269-28 2021-07-31T01:58:50.788 INFO:tasks.rados.rados.0.smithi045.stdout:3928: finishing write tid 2 to smithi04569269-28 2021-07-31T01:58:50.789 INFO:tasks.rados.rados.0.smithi045.stdout:3928: finishing write tid 3 to smithi04569269-28 2021-07-31T01:58:50.789 INFO:tasks.rados.rados.0.smithi045.stdout:3928: finishing write tid 4 to smithi04569269-28 2021-07-31T01:58:50.790 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 28 v 2397 (ObjNum 1206 snap 386 seq_num 1206) dirty exists 2021-07-31T01:58:50.790 INFO:tasks.rados.rados.0.smithi045.stdout:3928: left oid 28 (ObjNum 1206 snap 386 seq_num 1206) 2021-07-31T01:58:50.790 INFO:tasks.rados.rados.0.smithi045.stdout:3934: expect (ObjNum -1103377216 snap 21873 seq_num 3189089280) 2021-07-31T01:58:50.793 INFO:tasks.rados.rados.0.smithi045.stdout:3933: finishing write tid 1 to smithi04569269-15 2021-07-31T01:58:50.793 INFO:tasks.rados.rados.0.smithi045.stdout:3933: finishing write tid 2 to smithi04569269-15 2021-07-31T01:58:50.796 INFO:tasks.rados.rados.0.smithi045.stdout:3928: done (7 left) 2021-07-31T01:58:50.796 INFO:tasks.rados.rados.0.smithi045.stdout:3929: done (6 left) 2021-07-31T01:58:50.797 INFO:tasks.rados.rados.0.smithi045.stdout:3930: done (5 left) 2021-07-31T01:58:50.797 INFO:tasks.rados.rados.0.smithi045.stdout:3934: done (4 left) 2021-07-31T01:58:50.797 INFO:tasks.rados.rados.0.smithi045.stdout:3935: done (3 left) 2021-07-31T01:58:50.797 INFO:tasks.rados.rados.0.smithi045.stdout:3936: done (2 left) 2021-07-31T01:58:50.798 INFO:tasks.rados.rados.0.smithi045.stdout:3937: copy_from oid 42 from oid 38 current snap is 386 2021-07-31T01:58:50.798 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 40 current snap is 386 2021-07-31T01:58:50.798 INFO:tasks.rados.rados.0.smithi045.stdout:3938: seq_num 1209 ranges {0=655360,655360=401408} 2021-07-31T01:58:50.800 INFO:tasks.rados.rados.0.smithi045.stdout:3938: writing smithi04569269-40 from 0 to 655360 tid 1 2021-07-31T01:58:50.803 INFO:tasks.rados.rados.0.smithi045.stdout:3938: writing smithi04569269-40 from 655360 to 1056768 tid 2 2021-07-31T01:58:50.803 INFO:tasks.rados.rados.0.smithi045.stdout:3939: snap_create 2021-07-31T01:58:50.804 INFO:tasks.rados.rados.0.smithi045.stdout:3933: finishing write tid 3 to smithi04569269-15 2021-07-31T01:58:50.804 INFO:tasks.rados.rados.0.smithi045.stdout:3931: expect (ObjNum 1090 snap 349 seq_num 1090) 2021-07-31T01:58:50.891 INFO:tasks.rados.rados.0.smithi045.stdout:3933: finishing write tid 4 to smithi04569269-15 2021-07-31T01:58:50.891 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 15 v 2546 (ObjNum 1207 snap 386 seq_num 1207) dirty exists 2021-07-31T01:58:50.891 INFO:tasks.rados.rados.0.smithi045.stdout:3933: left oid 15 (ObjNum 1207 snap 386 seq_num 1207) 2021-07-31T01:58:50.892 INFO:tasks.rados.rados.0.smithi045.stdout:3937: finishing copy_from to smithi04569269-42 2021-07-31T01:58:50.892 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 42 v 2101 (ObjNum 1181 snap 382 seq_num 1181) dirty exists 2021-07-31T01:58:50.892 INFO:tasks.rados.rados.0.smithi045.stdout:3938: finishing write tid 1 to smithi04569269-40 2021-07-31T01:58:50.892 INFO:tasks.rados.rados.0.smithi045.stdout:3937: finishing copy_from racing read to smithi04569269-42 2021-07-31T01:58:50.893 INFO:tasks.rados.rados.0.smithi045.stdout:3938: finishing write tid 2 to smithi04569269-40 2021-07-31T01:58:50.893 INFO:tasks.rados.rados.0.smithi045.stdout:3938: finishing write tid 3 to smithi04569269-40 2021-07-31T01:58:50.893 INFO:tasks.rados.rados.0.smithi045.stdout:3938: finishing write tid 4 to smithi04569269-40 2021-07-31T01:58:50.893 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 40 v 2379 (ObjNum 1209 snap 386 seq_num 1209) dirty exists 2021-07-31T01:58:50.894 INFO:tasks.rados.rados.0.smithi045.stdout:3938: left oid 40 (ObjNum 1209 snap 386 seq_num 1209) 2021-07-31T01:58:51.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:50 smithi045 conmon[39718]: cluster 2021-07-31T01:58:49.755586+0000 mon.a ( 2021-07-31T01:58:51.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:50 smithi045 conmon[39718]: mon.0) 1484 : cluster [DBG] osdmap e811: 8 total, 8 up, 8 in 2021-07-31T01:58:51.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:50 smithi045 conmon[39718]: cluster 2021-07-31T01:58:49. 2021-07-31T01:58:51.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:50 smithi045 conmon[39718]: 959335+0000 mgr.x (mgr.24542) 498 : cluster [DBG] pgmap v1006: 25 pgs: 25 active+clean; 723 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 13 MiB/s rd, 4.6 MiB/s wr, 19 op/s 2021-07-31T01:58:51.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:50 smithi045 conmon[33655]: cluster 2021-07-31T01:58:49.755586+0000 mon.a (mon.0) 1484 : cluster [DBG] osdmap e811: 8 total, 8 up, 8 in 2021-07-31T01:58:51.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:50 smithi045 conmon[33655]: cluster 2021-07-31T01:58:49.959335+0000 mgr.x (mgr.24542) 498 : cluster [DBG] pgmap v1006: 25 pgs: 25 active+clean; 723 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 13 MiB/s rd, 4.6 MiB/s wr, 19 op/s 2021-07-31T01:58:51.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:50 smithi154 conmon[34553]: cluster 2021-07-31T01:58:49.755586+0000 mon.a (mon.0) 1484 : cluster [DBG] 2021-07-31T01:58:51.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:50 smithi154 conmon[34553]: osdmap e811: 8 total, 8 up, 8 in 2021-07-31T01:58:51.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:50 smithi154 conmon[34553]: cluster 2021-07-31T01:58:49.959335+0000 mgr.x (mgr.24542) 498 : cluster [DBG] pgmap v1006: 25 pgs: 25 active+clean; 723 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 13 MiB/s rd, 4.6 MiB/s wr, 19 op/s 2021-07-31T01:58:51.700 INFO:tasks.rados.rados.0.smithi045.stdout:3931: done (4 left) 2021-07-31T01:58:51.700 INFO:tasks.rados.rados.0.smithi045.stdout:3933: done (3 left) 2021-07-31T01:58:51.700 INFO:tasks.rados.rados.0.smithi045.stdout:3937: done (2 left) 2021-07-31T01:58:51.700 INFO:tasks.rados.rados.0.smithi045.stdout:3938: done (1 left) 2021-07-31T01:58:51.701 INFO:tasks.rados.rados.0.smithi045.stdout:3939: done (0 left) 2021-07-31T01:58:51.701 INFO:tasks.rados.rados.0.smithi045.stdout:3940: rmattr oid 42 current snap is 387 2021-07-31T01:58:51.701 INFO:tasks.rados.rados.0.smithi045.stdout:3940: done (0 left) 2021-07-31T01:58:51.701 INFO:tasks.rados.rados.0.smithi045.stdout:3941: read oid 24 snap -1 2021-07-31T01:58:51.701 INFO:tasks.rados.rados.0.smithi045.stdout:3941: expect (ObjNum 1172 snap 376 seq_num 1172) 2021-07-31T01:58:51.702 INFO:tasks.rados.rados.0.smithi045.stdout:3942: snap_remove snap 356 2021-07-31T01:58:52.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:51 smithi045 conmon[33655]: cluster 2021-07-31T01:58:50.762184 2021-07-31T01:58:52.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:51 smithi045 conmon[33655]: +0000 mon.a (mon.0) 1485 : cluster [DBG] osdmap e812: 8 total, 8 up, 8 in 2021-07-31T01:58:52.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:51 smithi045 conmon[33655]: cluster 2021-07-31T01:58:51.698781+0000 mon.a (mon.0) 1486 2021-07-31T01:58:52.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:51 smithi045 conmon[33655]: : cluster [DBG] osdmap e813: 8 total, 8 up, 8 in 2021-07-31T01:58:52.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:51 smithi045 conmon[39718]: cluster 2021-07-31T01:58:50.762184+0000 2021-07-31T01:58:52.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:51 smithi045 conmon[39718]: mon.a (mon.0) 1485 : cluster [DBG] osdmap e812: 8 total, 8 up, 8 in 2021-07-31T01:58:52.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:51 smithi045 conmon[39718]: cluster 2021 2021-07-31T01:58:52.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:51 smithi045 conmon[39718]: -07-31T01:58:51.698781+0000 mon.a (mon.0) 1486 : cluster [DBG] osdmap e813: 8 total, 8 up, 8 in 2021-07-31T01:58:52.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:51 smithi154 conmon[34553]: cluster 2021-07-31T01:58:50.762184+0000 mon.a (mon.0) 1485 : cluster [DBG] osdmap e812: 8 total, 8 up, 8 in 2021-07-31T01:58:52.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:51 smithi154 conmon[34553]: cluster 2021-07-31T01:58:51.698781+0000 mon.a (mon.0) 1486 : cluster [DBG] osdmap e813: 8 total, 8 up, 8 in 2021-07-31T01:58:52.707 INFO:tasks.rados.rados.0.smithi045.stdout:3942: done (1 left) 2021-07-31T01:58:52.707 INFO:tasks.rados.rados.0.smithi045.stdout:3943: read oid 25 snap -1 2021-07-31T01:58:52.708 INFO:tasks.rados.rados.0.smithi045.stdout:3943: expect (ObjNum 1157 snap 372 seq_num 1157) 2021-07-31T01:58:52.708 INFO:tasks.rados.rados.0.smithi045.stdout:3944: snap_remove snap 375 2021-07-31T01:58:53.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:52 smithi045 conmon[39718]: cluster 2021-07-31T01:58:51.959722+0000 mgr.x (mgr. 2021-07-31T01:58:53.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:52 smithi045 conmon[39718]: 24542) 499 : cluster [DBG] pgmap v1009: 25 pgs: 25 active+clean; 723 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 13 MiB/s rd, 4.6 MiB/s wr, 19 op/s 2021-07-31T01:58:53.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:52 smithi045 conmon[39718]: cluster 2021-07-31T01:58:52.705086+0000 mon.a (mon.0) 1487 : cluster 2021-07-31T01:58:53.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:52 smithi045 conmon[39718]: [DBG] osdmap e814: 8 total, 8 up, 8 in 2021-07-31T01:58:53.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:52 smithi045 conmon[33655]: cluster 2021-07-31T01:58:51 2021-07-31T01:58:53.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:52 smithi045 conmon[33655]: .959722+0000 mgr.x (mgr.24542) 499 : cluster [DBG] pgmap v1009: 25 pgs: 25 active+clean; 723 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 13 MiB/s rd, 4.6 MiB/s wr, 19 op/s 2021-07-31T01:58:53.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:52 smithi045 conmon[33655]: cluster 2021-07 2021-07-31T01:58:53.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:52 smithi045 conmon[33655]: -31T01:58:52.705086+0000 mon.a (mon 2021-07-31T01:58:53.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:52 smithi045 conmon[33655]: .0) 1487 : cluster [DBG] 2021-07-31T01:58:53.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:52 smithi045 conmon[33655]: osdmap e814: 8 total, 8 up, 8 in 2021-07-31T01:58:53.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:52 smithi154 conmon[34553]: cluster 2021-07-31T01:58:51.959722+0000 mgr.x (mgr.24542) 499 2021-07-31T01:58:53.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:52 smithi154 conmon[34553]: : cluster [DBG] pgmap v1009: 25 pgs: 25 active+clean; 723 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 13 MiB/s rd, 4.6 MiB/s wr, 19 op/s 2021-07-31T01:58:53.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:52 smithi154 conmon[34553]: cluster 2021-07-31T01:58:52.705086+0000 mon.a (mon.0) 1487 : cluster [DBG] osdmap e814: 8 total, 8 up, 8 in 2021-07-31T01:58:53.708 INFO:tasks.rados.rados.0.smithi045.stdout:3944: done (2 left) 2021-07-31T01:58:53.709 INFO:tasks.rados.rados.0.smithi045.stdout:3945: rmattr oid 43 current snap is 387 2021-07-31T01:58:53.709 INFO:tasks.rados.rados.0.smithi045.stdout:3945: done (2 left) 2021-07-31T01:58:53.709 INFO:tasks.rados.rados.0.smithi045.stdout:3946: read oid 19 snap -1 2021-07-31T01:58:53.709 INFO:tasks.rados.rados.0.smithi045.stdout:3946: expect deleted 2021-07-31T01:58:53.710 INFO:tasks.rados.rados.0.smithi045.stdout:3947: copy_from oid 31 from oid 37 current snap is 387 2021-07-31T01:58:53.710 INFO:tasks.rados.rados.0.smithi045.stdout:3948: read oid 3 snap 371 2021-07-31T01:58:53.710 INFO:tasks.rados.rados.0.smithi045.stdout:3948: expect (ObjNum 881 snap 272 seq_num 881) 2021-07-31T01:58:53.710 INFO:tasks.rados.rados.0.smithi045.stdout:3949: snap_remove snap 384 2021-07-31T01:58:54.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:58:53 smithi045 conmon[61210]: level=error ts=2021-07-31T01:58:53.529Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:58:54.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:58:53 smithi045 conmon[61210]: level=error ts=2021-07-31T01:58:53.529Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:58:54.027 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:58:53 smithi045 conmon[61210]: level=error ts=2021-07-31T01:58:53.529Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T01:58:54.714 INFO:tasks.rados.rados.0.smithi045.stdout:3949: done (5 left) 2021-07-31T01:58:54.714 INFO:tasks.rados.rados.0.smithi045.stdout:3950: delete oid 30 current snap is 387 2021-07-31T01:58:54.717 INFO:tasks.rados.rados.0.smithi045.stdout:3941: expect (ObjNum 1172 snap 376 seq_num 1172) 2021-07-31T01:58:54.739 INFO:tasks.rados.rados.0.smithi045.stdout:3943: expect (ObjNum 1157 snap 372 seq_num 1157) 2021-07-31T01:58:54.744 INFO:tasks.rados.rados.0.smithi045.stdout:3948: expect (ObjNum 881 snap 272 seq_num 881) 2021-07-31T01:58:55.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:54 smithi154 conmon[34553]: cluster 2021-07-31T01:58:53.706766+0000 mon.a (mon. 2021-07-31T01:58:55.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:54 smithi154 conmon[34553]: 0) 1488 : cluster [DBG] osdmap e815: 8 total, 8 up, 8 in 2021-07-31T01:58:55.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:54 smithi154 conmon[34553]: cluster 2021-07-31T01:58:53.960465+0000 mgr.x (mgr.24542) 500 : cluster [DBG] pgmap v1012: 25 pgs: 25 active+clean; 712 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 9.3 MiB/s rd, 2.1 MiB/s wr, 9 op/s 2021-07-31T01:58:55.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:54 smithi045 conmon[39718]: cluster 2021-07-31T01:58:53.706766+0000 mon.a (mon. 2021-07-31T01:58:55.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:54 smithi045 conmon[39718]: 0) 1488 : cluster [DBG] osdmap e815: 8 total, 8 up, 8 in 2021-07-31T01:58:55.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:54 smithi045 conmon[39718]: cluster 2021-07-31T01:58:53.960465+0000 mgr.x (mgr.24542) 500 : cluster 2021-07-31T01:58:55.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:54 smithi045 conmon[39718]: [DBG] pgmap v1012: 25 pgs: 25 active+clean; 712 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 9.3 MiB/s rd, 2.1 MiB/s wr, 9 op/s 2021-07-31T01:58:55.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:54 smithi045 conmon[33655]: cluster 2021-07-31T01:58:53.706766+0000 mon.a ( 2021-07-31T01:58:55.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:54 smithi045 conmon[33655]: mon.0) 1488 : cluster [DBG] osdmap e815: 8 total, 8 up, 8 in 2021-07-31T01:58:55.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:54 smithi045 conmon[33655]: cluster 2021-07-31T01:58:53.960465+0000 2021-07-31T01:58:55.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:54 smithi045 conmon[33655]: mgr.x (mgr.24542) 500 : cluster [DBG] pgmap v1012: 25 pgs: 25 active+clean; 712 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 9.3 MiB/s rd, 2.1 MiB/s wr, 9 op/s 2021-07-31T01:58:55.767 INFO:tasks.rados.rados.0.smithi045.stdout:3947: finishing copy_from to smithi04569269-31 2021-07-31T01:58:55.767 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 31 v 1899 (ObjNum 1045 snap 340 seq_num 1045) dirty exists 2021-07-31T01:58:55.768 INFO:tasks.rados.rados.0.smithi045.stdout:3947: finishing copy_from racing read to smithi04569269-31 2021-07-31T01:58:55.768 INFO:tasks.rados.rados.0.smithi045.stdout:3941: done (5 left) 2021-07-31T01:58:55.768 INFO:tasks.rados.rados.0.smithi045.stdout:3943: done (4 left) 2021-07-31T01:58:55.768 INFO:tasks.rados.rados.0.smithi045.stdout:3946: done (3 left) 2021-07-31T01:58:55.769 INFO:tasks.rados.rados.0.smithi045.stdout:3947: done (2 left) 2021-07-31T01:58:55.769 INFO:tasks.rados.rados.0.smithi045.stdout:3948: done (1 left) 2021-07-31T01:58:55.769 INFO:tasks.rados.rados.0.smithi045.stdout:3950: done (0 left) 2021-07-31T01:58:55.769 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 33 current snap is 387 2021-07-31T01:58:55.770 INFO:tasks.rados.rados.0.smithi045.stdout:3951: seq_num 1211 ranges {2441216=688128,3129344=401408} 2021-07-31T01:58:55.783 INFO:tasks.rados.rados.0.smithi045.stdout:3951: writing smithi04569269-33 from 2441216 to 3129344 tid 1 2021-07-31T01:58:55.786 INFO:tasks.rados.rados.0.smithi045.stdout:3951: writing smithi04569269-33 from 3129344 to 3530752 tid 2 2021-07-31T01:58:55.786 INFO:tasks.rados.rados.0.smithi045.stdout:3952: read oid 39 snap -1 2021-07-31T01:58:55.786 INFO:tasks.rados.rados.0.smithi045.stdout:3952: expect deleted 2021-07-31T01:58:55.787 INFO:tasks.rados.rados.0.smithi045.stdout:3953: read oid 42 snap -1 2021-07-31T01:58:55.787 INFO:tasks.rados.rados.0.smithi045.stdout:3953: expect (ObjNum 1181 snap 382 seq_num 1181) 2021-07-31T01:58:55.787 INFO:tasks.rados.rados.0.smithi045.stdout:3954: copy_from oid 28 from oid 9 current snap is 387 2021-07-31T01:58:55.787 INFO:tasks.rados.rados.0.smithi045.stdout:3955: read oid 6 snap -1 2021-07-31T01:58:55.788 INFO:tasks.rados.rados.0.smithi045.stdout:3955: expect (ObjNum 1199 snap 384 seq_num 1199) 2021-07-31T01:58:55.788 INFO:tasks.rados.rados.0.smithi045.stdout:3956: snap_create 2021-07-31T01:58:55.797 INFO:tasks.rados.rados.0.smithi045.stdout:3951: finishing write tid 1 to smithi04569269-33 2021-07-31T01:58:55.797 INFO:tasks.rados.rados.0.smithi045.stdout:3951: finishing write tid 2 to smithi04569269-33 2021-07-31T01:58:55.797 INFO:tasks.rados.rados.0.smithi045.stdout:3951: finishing write tid 3 to smithi04569269-33 2021-07-31T01:58:55.797 INFO:tasks.rados.rados.0.smithi045.stdout:3955: expect (ObjNum 1199 snap 384 seq_num 1199) 2021-07-31T01:58:55.818 INFO:tasks.rados.rados.0.smithi045.stdout:3953: expect (ObjNum 1181 snap 382 seq_num 1181) 2021-07-31T01:58:55.857 INFO:tasks.rados.rados.0.smithi045.stdout:3954: finishing copy_from to smithi04569269-28 2021-07-31T01:58:55.857 INFO:tasks.rados.rados.0.smithi045.stdout:3954: got expected ENOENT (src dne) 2021-07-31T01:58:55.857 INFO:tasks.rados.rados.0.smithi045.stdout:3954: finishing copy_from racing read to smithi04569269-28 2021-07-31T01:58:55.858 INFO:tasks.rados.rados.0.smithi045.stdout:3951: finishing write tid 4 to smithi04569269-33 2021-07-31T01:58:55.858 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 33 v 2111 (ObjNum 1211 snap 387 seq_num 1211) dirty exists 2021-07-31T01:58:55.858 INFO:tasks.rados.rados.0.smithi045.stdout:3951: left oid 33 (ObjNum 1211 snap 387 seq_num 1211) 2021-07-31T01:58:56.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:55 smithi154 conmon[34553]: cluster 2021-07-31T01:58:54.713457+0000 mon.a (mon.0) 1489 : cluster [DBG] osdmap e816: 8 total, 8 up, 8 in 2021-07-31T01:58:56.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:55 smithi045 conmon[39718]: cluster 2021-07-31T01:58:54.713457 2021-07-31T01:58:56.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:55 smithi045 conmon[39718]: +0000 mon.a (mon.0) 1489 : cluster [DBG] osdmap e816: 8 total, 8 up, 8 in 2021-07-31T01:58:56.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:55 smithi045 conmon[33655]: cluster 2021-07-31T01:58:54.713457+0000 2021-07-31T01:58:56.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:55 smithi045 conmon[33655]: mon.a (mon.0) 1489 : cluster [DBG] osdmap e816: 8 total, 8 up, 8 in 2021-07-31T01:58:56.701 INFO:tasks.rados.rados.0.smithi045.stdout:3951: done (5 left) 2021-07-31T01:58:56.701 INFO:tasks.rados.rados.0.smithi045.stdout:3952: done (4 left) 2021-07-31T01:58:56.701 INFO:tasks.rados.rados.0.smithi045.stdout:3953: done (3 left) 2021-07-31T01:58:56.702 INFO:tasks.rados.rados.0.smithi045.stdout:3954: done (2 left) 2021-07-31T01:58:56.702 INFO:tasks.rados.rados.0.smithi045.stdout:3955: done (1 left) 2021-07-31T01:58:56.702 INFO:tasks.rados.rados.0.smithi045.stdout:3956: done (0 left) 2021-07-31T01:58:56.703 INFO:tasks.rados.rados.0.smithi045.stdout:3957: setattr oid 23 current snap is 388 2021-07-31T01:58:56.703 INFO:tasks.rados.rados.0.smithi045.stdout:3958: setattr oid 43 current snap is 388 2021-07-31T01:58:56.705 INFO:tasks.rados.rados.0.smithi045.stdout:3959: snap_remove snap 379 2021-07-31T01:58:57.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:56 smithi154 conmon[34553]: cluster 2021-07-31T01:58:55.960863+0000 mgr.x (mgr.24542) 501 : cluster [DBG] pgmap v1014: 25 pgs: 25 active+clean; 712 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 8.8 MiB/s rd, 2.0 MiB/s wr, 9 op/s 2021-07-31T01:58:57.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:56 smithi154 conmon[34553]: 2021-07-31T01:58:57.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:56 smithi154 conmon[34553]: cluster 2021-07-31T01:58:56.700268+0000 mon.a (mon.0) 1490 : cluster [DBG] osdmap e817: 8 total, 8 up, 8 in 2021-07-31T01:58:57.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:56 smithi045 conmon[39718]: cluster 2021-07-31T01:58: 2021-07-31T01:58:57.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:56 smithi045 conmon[39718]: 55.960863+0000 mgr.x (mgr.24542) 501 : cluster [DBG] pgmap v1014: 25 pgs: 25 active+clean; 712 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 8.8 MiB/s rd, 2.0 MiB/s wr, 9 op/s 2021-07-31T01:58:57.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:56 smithi045 conmon[39718]: cluster 2021-07-31T01:58 2021-07-31T01:58:57.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:56 smithi045 conmon[39718]: :56.700268+0000 mon.a (mon.0) 1490 : 2021-07-31T01:58:57.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:56 smithi045 conmon[39718]: cluster [DBG] osdmap e817: 8 total, 8 up, 8 in 2021-07-31T01:58:57.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:56 smithi045 conmon[33655]: cluster 2021-07-31T01:58:55.960863+0000 mgr.x (mgr.24542) 2021-07-31T01:58:57.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:56 smithi045 conmon[33655]: 501 : cluster [DBG] pgmap v1014: 25 pgs: 25 active+clean; 712 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 8.8 MiB/s rd, 2.0 MiB/s wr, 9 op/s 2021-07-31T01:58:57.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:56 smithi045 conmon[33655]: cluster 2021-07-31T01:58:56.700268+0000 2021-07-31T01:58:57.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:56 smithi045 conmon[33655]: mon.a (mon.0) 1490 : cluster [DBG] osdmap e817: 8 total, 8 up, 8 in 2021-07-31T01:58:57.710 INFO:tasks.rados.rados.0.smithi045.stdout:3959: done (2 left) 2021-07-31T01:58:57.710 INFO:tasks.rados.rados.0.smithi045.stdout:3960: setattr oid 2 current snap is 388 2021-07-31T01:58:57.710 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 23 v 2112 (ObjNum 1049 snap 342 seq_num 1049) dirty exists 2021-07-31T01:58:57.711 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 43 v 2814 (ObjNum 1182 snap 383 seq_num 1182) dirty exists 2021-07-31T01:58:57.712 INFO:tasks.rados.rados.0.smithi045.stdout:3957: done (2 left) 2021-07-31T01:58:57.713 INFO:tasks.rados.rados.0.smithi045.stdout:3958: done (1 left) 2021-07-31T01:58:57.713 INFO:tasks.rados.rados.0.smithi045.stdout:3961: setattr oid 16 current snap is 388 2021-07-31T01:58:57.715 INFO:tasks.rados.rados.0.smithi045.stdout:3962: read oid 12 snap -1 2021-07-31T01:58:57.716 INFO:tasks.rados.rados.0.smithi045.stdout:3962: expect deleted 2021-07-31T01:58:57.716 INFO:tasks.rados.rados.0.smithi045.stdout:3963: read oid 9 snap -1 2021-07-31T01:58:57.716 INFO:tasks.rados.rados.0.smithi045.stdout:3963: expect deleted 2021-07-31T01:58:57.716 INFO:tasks.rados.rados.0.smithi045.stdout:3964: setattr oid 17 current snap is 388 2021-07-31T01:58:57.718 INFO:tasks.rados.rados.0.smithi045.stdout:3965: read oid 15 snap -1 2021-07-31T01:58:57.718 INFO:tasks.rados.rados.0.smithi045.stdout:3965: expect (ObjNum 1207 snap 386 seq_num 1207) 2021-07-31T01:58:57.718 INFO:tasks.rados.rados.0.smithi045.stdout:3966: snap_remove snap 385 2021-07-31T01:58:58.715 INFO:tasks.rados.rados.0.smithi045.stdout:3966: done (6 left) 2021-07-31T01:58:58.715 INFO:tasks.rados.rados.0.smithi045.stdout:3967: rmattr oid 35 current snap is 388 2021-07-31T01:58:58.716 INFO:tasks.rados.rados.0.smithi045.stdout:3967: done (6 left) 2021-07-31T01:58:58.716 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 29 current snap is 388 2021-07-31T01:58:58.717 INFO:tasks.rados.rados.0.smithi045.stdout:3968: seq_num 1212 ranges {0=704512,704512=401408} 2021-07-31T01:58:58.718 INFO:tasks.rados.rados.0.smithi045.stdout:3968: writing smithi04569269-29 from 0 to 704512 tid 1 2021-07-31T01:58:58.722 INFO:tasks.rados.rados.0.smithi045.stdout:3968: writing smithi04569269-29 from 704512 to 1105920 tid 2 2021-07-31T01:58:58.723 INFO:tasks.rados.rados.0.smithi045.stdout:3969: read oid 32 snap -1 2021-07-31T01:58:58.723 INFO:tasks.rados.rados.0.smithi045.stdout:3969: expect (ObjNum 1176 snap 376 seq_num 1176) 2021-07-31T01:58:58.724 INFO:tasks.rados.rados.0.smithi045.stdout:3970: snap_create 2021-07-31T01:58:58.724 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 2 v 2385 (ObjNum 1194 snap 384 seq_num 1194) dirty exists 2021-07-31T01:58:58.724 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 16 v 1867 (ObjNum 1184 snap 383 seq_num 1184) dirty exists 2021-07-31T01:58:58.725 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 17 v 2398 (ObjNum 1166 snap 374 seq_num 1166) dirty exists 2021-07-31T01:58:58.725 INFO:tasks.rados.rados.0.smithi045.stdout:3965: expect (ObjNum 1207 snap 386 seq_num 1207) 2021-07-31T01:58:58.825 INFO:tasks.rados.rados.0.smithi045.stdout:3968: finishing write tid 1 to smithi04569269-29 2021-07-31T01:58:58.825 INFO:tasks.rados.rados.0.smithi045.stdout:3968: finishing write tid 2 to smithi04569269-29 2021-07-31T01:58:58.825 INFO:tasks.rados.rados.0.smithi045.stdout:3968: finishing write tid 3 to smithi04569269-29 2021-07-31T01:58:58.825 INFO:tasks.rados.rados.0.smithi045.stdout:3968: finishing write tid 4 to smithi04569269-29 2021-07-31T01:58:58.826 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 29 v 1434 (ObjNum 1212 snap 388 seq_num 1212) dirty exists 2021-07-31T01:58:58.826 INFO:tasks.rados.rados.0.smithi045.stdout:3968: left oid 29 (ObjNum 1212 snap 388 seq_num 1212) 2021-07-31T01:58:58.826 INFO:tasks.rados.rados.0.smithi045.stdout:3969: expect (ObjNum 1176 snap 376 seq_num 1176) 2021-07-31T01:58:59.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:58 smithi154 conmon[34553]: cluster 2021-07-31T01:58:57.709310+0000 mon.a (mon.0) 1491 : cluster [DBG] 2021-07-31T01:58:59.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:58 smithi154 conmon[34553]: osdmap e818: 8 total, 8 up, 8 in 2021-07-31T01:58:59.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:58 smithi154 conmon[34553]: cluster 2021-07-31T01:58:57.961456+0000 mgr.x (mgr.24542) 502 : cluster [DBG] pgmap v1017: 25 pgs: 25 active+clean; 712 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 8.8 MiB/s rd, 2.0 MiB/s wr, 9 op/s 2021-07-31T01:58:59.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:58 smithi045 conmon[39718]: cluster 2021-07-31T01:58:57.709310+0000 mon.a (mon. 2021-07-31T01:58:59.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:58 smithi045 conmon[39718]: 0) 1491 : cluster [DBG] osdmap e818: 8 total, 8 up, 8 in 2021-07-31T01:58:59.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:58 smithi045 conmon[39718]: cluster 2021-07-31T01:58:57.961456+0000 mgr.x 2021-07-31T01:58:59.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:58 smithi045 conmon[39718]: (mgr.24542) 502 : cluster [DBG] pgmap v1017: 25 pgs: 25 active+clean; 712 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 8.8 MiB/s rd, 2.0 MiB/s wr, 9 op/s 2021-07-31T01:58:59.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:58 smithi045 conmon[33655]: cluster 2021-07-31T01:58:57.709310+0000 mon.a (mon.0) 1491 2021-07-31T01:58:59.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:58 smithi045 conmon[33655]: : cluster [DBG] osdmap e818: 8 total, 8 up, 8 in 2021-07-31T01:58:59.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:58 smithi045 conmon[33655]: cluster 2021-07-31T01:58:57.961456+0000 mgr.x (mgr.24542) 502 : cluster [DBG] pgmap v1017: 25 pgs: 25 active+clean; 712 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 8.8 MiB/s rd, 2.0 MiB/s wr, 9 op/s 2021-07-31T01:59:00.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:59 smithi154 conmon[34553]: cluster 2021-07-31T01:58:58.713677+0000 mon.a (mon.0) 1492 : cluster [DBG] 2021-07-31T01:59:00.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:58:59 smithi154 conmon[34553]: osdmap e819: 8 total, 8 up, 8 in 2021-07-31T01:59:00.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:59 smithi045 conmon[39718]: cluster 2021-07-31T01:58:58.713677+0000 mon.a ( 2021-07-31T01:59:00.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:58:59 smithi045 conmon[39718]: mon.0) 1492 : cluster [DBG] osdmap e819: 8 total, 8 up, 8 in 2021-07-31T01:59:00.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:58:59 smithi045 conmon[33655]: cluster 2021-07-31T01:58:58.713677+0000 mon.a (mon.0) 1492 : cluster [DBG] osdmap e819: 8 total, 8 up, 8 in 2021-07-31T01:59:00.300 INFO:tasks.rados.rados.0.smithi045.stdout:3960: done (8 left) 2021-07-31T01:59:00.300 INFO:tasks.rados.rados.0.smithi045.stdout:3961: done (7 left) 2021-07-31T01:59:00.300 INFO:tasks.rados.rados.0.smithi045.stdout:3962: done (6 left) 2021-07-31T01:59:00.301 INFO:tasks.rados.rados.0.smithi045.stdout:3963: done (5 left) 2021-07-31T01:59:00.301 INFO:tasks.rados.rados.0.smithi045.stdout:3964: done (4 left) 2021-07-31T01:59:00.301 INFO:tasks.rados.rados.0.smithi045.stdout:3965: done (3 left) 2021-07-31T01:59:00.301 INFO:tasks.rados.rados.0.smithi045.stdout:3968: done (2 left) 2021-07-31T01:59:00.301 INFO:tasks.rados.rados.0.smithi045.stdout:3969: done (1 left) 2021-07-31T01:59:00.302 INFO:tasks.rados.rados.0.smithi045.stdout:3970: done (0 left) 2021-07-31T01:59:00.302 INFO:tasks.rados.rados.0.smithi045.stdout:3971: read oid 26 snap -1 2021-07-31T01:59:00.302 INFO:tasks.rados.rados.0.smithi045.stdout:3971: expect (ObjNum 1195 snap 384 seq_num 1195) 2021-07-31T01:59:00.302 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 39 current snap is 389 2021-07-31T01:59:00.303 INFO:tasks.rados.rados.0.smithi045.stdout:3972: seq_num 1213 ranges {0=720896,720896=401408} 2021-07-31T01:59:00.303 INFO:tasks.rados.rados.0.smithi045.stdout:3972: writing smithi04569269-39 from 0 to 720896 tid 1 2021-07-31T01:59:00.305 INFO:tasks.rados.rados.0.smithi045.stdout:3972: writing smithi04569269-39 from 720896 to 1122304 tid 2 2021-07-31T01:59:00.306 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 18 current snap is 389 2021-07-31T01:59:00.306 INFO:tasks.rados.rados.0.smithi045.stdout:3973: seq_num 1214 ranges {0=737280,737280=401408} 2021-07-31T01:59:00.310 INFO:tasks.rados.rados.0.smithi045.stdout:3973: writing smithi04569269-18 from 0 to 737280 tid 1 2021-07-31T01:59:00.312 INFO:tasks.rados.rados.0.smithi045.stdout:3973: writing smithi04569269-18 from 737280 to 1138688 tid 2 2021-07-31T01:59:00.312 INFO:tasks.rados.rados.0.smithi045.stdout:3974: copy_from oid 42 from oid 7 current snap is 389 2021-07-31T01:59:00.313 INFO:tasks.rados.rados.0.smithi045.stdout:3975: snap_create 2021-07-31T01:59:00.313 INFO:tasks.rados.rados.0.smithi045.stdout:3972: finishing write tid 1 to smithi04569269-39 2021-07-31T01:59:00.315 INFO:tasks.rados.rados.0.smithi045.stdout:3972: finishing write tid 2 to smithi04569269-39 2021-07-31T01:59:00.316 INFO:tasks.rados.rados.0.smithi045.stdout:3972: finishing write tid 3 to smithi04569269-39 2021-07-31T01:59:00.317 INFO:tasks.rados.rados.0.smithi045.stdout:3973: finishing write tid 1 to smithi04569269-18 2021-07-31T01:59:00.317 INFO:tasks.rados.rados.0.smithi045.stdout:3973: finishing write tid 2 to smithi04569269-18 2021-07-31T01:59:00.318 INFO:tasks.rados.rados.0.smithi045.stdout:3973: finishing write tid 3 to smithi04569269-18 2021-07-31T01:59:00.318 INFO:tasks.rados.rados.0.smithi045.stdout:3972: finishing write tid 4 to smithi04569269-39 2021-07-31T01:59:00.319 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 39 v 2115 (ObjNum 1213 snap 389 seq_num 1213) dirty exists 2021-07-31T01:59:00.319 INFO:tasks.rados.rados.0.smithi045.stdout:3972: left oid 39 (ObjNum 1213 snap 389 seq_num 1213) 2021-07-31T01:59:00.320 INFO:tasks.rados.rados.0.smithi045.stdout:3973: finishing write tid 4 to smithi04569269-18 2021-07-31T01:59:00.320 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 18 v 2549 (ObjNum 1214 snap 389 seq_num 1214) dirty exists 2021-07-31T01:59:00.320 INFO:tasks.rados.rados.0.smithi045.stdout:3973: left oid 18 (ObjNum 1214 snap 389 seq_num 1214) 2021-07-31T01:59:00.324 INFO:tasks.rados.rados.0.smithi045.stdout:3971: expect (ObjNum 1195 snap 384 seq_num 1195) 2021-07-31T01:59:00.637 INFO:tasks.rados.rados.0.smithi045.stdout:3974: finishing copy_from to smithi04569269-42 2021-07-31T01:59:00.637 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 42 v 2166 (ObjNum 1178 snap 379 seq_num 1178) dirty exists 2021-07-31T01:59:00.638 INFO:tasks.rados.rados.0.smithi045.stdout:3974: finishing copy_from racing read to smithi04569269-42 2021-07-31T01:59:00.724 INFO:tasks.rados.rados.0.smithi045.stdout:3971: done (4 left) 2021-07-31T01:59:00.725 INFO:tasks.rados.rados.0.smithi045.stdout:3972: done (3 left) 2021-07-31T01:59:00.725 INFO:tasks.rados.rados.0.smithi045.stdout:3973: done (2 left) 2021-07-31T01:59:00.725 INFO:tasks.rados.rados.0.smithi045.stdout:3974: done (1 left) 2021-07-31T01:59:00.725 INFO:tasks.rados.rados.0.smithi045.stdout:3975: done (0 left) 2021-07-31T01:59:00.725 INFO:tasks.rados.rados.0.smithi045.stdout:3976: rollback oid 41 current snap is 390 2021-07-31T01:59:00.726 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 41 to 295 2021-07-31T01:59:00.726 INFO:tasks.rados.rados.0.smithi045.stdout:3977: rollback oid 49 current snap is 390 2021-07-31T01:59:00.726 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 49 to 357 2021-07-31T01:59:00.726 INFO:tasks.rados.rados.0.smithi045.stdout:3978: rmattr oid 32 current snap is 390 2021-07-31T01:59:00.727 INFO:tasks.rados.rados.0.smithi045.stdout:3979: setattr oid 8 current snap is 390 2021-07-31T01:59:00.728 INFO:tasks.rados.rados.0.smithi045.stdout:3980: read oid 42 snap -1 2021-07-31T01:59:00.729 INFO:tasks.rados.rados.0.smithi045.stdout:3980: expect (ObjNum 1178 snap 379 seq_num 1178) 2021-07-31T01:59:00.729 INFO:tasks.rados.rados.0.smithi045.stdout:3981: snap_remove snap 378 2021-07-31T01:59:01.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:00 smithi154 conmon[34553]: cluster 2021-07-31T01:58:59.718034+0000 mon.a (mon.0) 1493 : cluster [DBG] 2021-07-31T01:59:01.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:00 smithi154 conmon[34553]: osdmap e820: 8 total, 8 up, 8 in 2021-07-31T01:59:01.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:00 smithi154 conmon[34553]: cluster 2021-07-31T01:58:59.962044+0000 mgr.x (mgr.24542) 503 : cluster [DBG] pgmap v1020: 25 pgs: 25 active+clean; 713 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 12 MiB/s rd, 536 KiB/s wr, 31 op/s 2021-07-31T01:59:01.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:00 smithi045 conmon[39718]: cluster 2021-07-31T01:58:59.718034 2021-07-31T01:59:01.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:00 smithi045 conmon[39718]: +0000 mon.a (mon.0) 1493 : cluster [DBG] osdmap e820: 8 total, 8 up, 8 in 2021-07-31T01:59:01.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:00 smithi045 conmon[39718]: cluster 2021-07-31T01:59:01.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:00 smithi045 conmon[39718]: 2021-07-31T01:58: 2021-07-31T01:59:01.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:00 smithi045 conmon[39718]: 59.962044+0000 mgr.x (mgr 2021-07-31T01:59:01.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:00 smithi045 conmon[39718]: .24542) 503 : cluster [DBG] 2021-07-31T01:59:01.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:00 smithi045 conmon[39718]: pgmap v1020: 25 pgs: 25 active+clean; 713 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 12 MiB/s rd, 536 KiB/s wr, 31 op/s 2021-07-31T01:59:01.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:00 smithi045 conmon[33655]: cluster 2021-07-31T01:58:59.718034+0000 mon.a ( 2021-07-31T01:59:01.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:00 smithi045 conmon[33655]: mon.0) 1493 : cluster [DBG] osdmap e820: 8 total, 8 up, 8 in 2021-07-31T01:59:01.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:00 smithi045 conmon[33655]: cluster 2021-07-31T01:58:59.962044+0000 mgr.x (mgr.24542) 503 : cluster 2021-07-31T01:59:01.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:00 smithi045 conmon[33655]: [DBG] pgmap v1020: 25 pgs: 25 active+clean; 713 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 12 MiB/s rd, 536 KiB/s wr, 31 op/s 2021-07-31T01:59:01.703 INFO:tasks.rados.rados.0.smithi045.stdout:3981: done (5 left) 2021-07-31T01:59:01.704 INFO:tasks.rados.rados.0.smithi045.stdout:3982: snap_create 2021-07-31T01:59:01.704 INFO:tasks.rados.rados.0.smithi045.stdout:3977: finishing rollback tid 0 to smithi04569269-49 2021-07-31T01:59:01.704 INFO:tasks.rados.rados.0.smithi045.stdout:3976: finishing rollback tid 0 to smithi04569269-41 2021-07-31T01:59:01.704 INFO:tasks.rados.rados.0.smithi045.stdout:3977: finishing rollback tid 1 to smithi04569269-49 2021-07-31T01:59:01.705 INFO:tasks.rados.rados.0.smithi045.stdout:3980: expect (ObjNum 1178 snap 379 seq_num 1178) 2021-07-31T01:59:01.724 INFO:tasks.rados.rados.0.smithi045.stdout:3977: finishing rollback tid 2 to smithi04569269-49 2021-07-31T01:59:01.724 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 49 v 3040 (ObjNum 1090 snap 349 seq_num 1090) dirty exists 2021-07-31T01:59:01.724 INFO:tasks.rados.rados.0.smithi045.stdout:3976: finishing rollback tid 1 to smithi04569269-41 2021-07-31T01:59:01.724 INFO:tasks.rados.rados.0.smithi045.stdout:3976: finishing rollback tid 2 to smithi04569269-41 2021-07-31T01:59:01.725 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 41 v 2172 (ObjNum 948 snap 286 seq_num 948) dirty exists 2021-07-31T01:59:01.725 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 8 v 3042 (ObjNum 994 snap 309 seq_num 994) dirty exists 2021-07-31T01:59:01.725 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 32 v 1891 (ObjNum 1176 snap 376 seq_num 1176) dirty exists 2021-07-31T01:59:02.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:01 smithi154 conmon[34553]: cluster 2021-07-31T01:59:00. 2021-07-31T01:59:02.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:01 smithi154 conmon[34553]: 723348+0000 mon.a (mon.0) 1494 : cluster [DBG] osdmap e821: 8 total, 8 up, 8 in 2021-07-31T01:59:02.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:01 smithi154 conmon[34553]: cluster 2021-07-31T01:59 2021-07-31T01:59:02.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:01 smithi154 conmon[34553]: :01.702331+0000 mon.a (mon.0) 1495 : cluster [DBG] 2021-07-31T01:59:02.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:01 smithi154 conmon[34553]: osdmap e822: 8 total, 8 up, 8 in 2021-07-31T01:59:02.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:01 smithi045 conmon[33655]: cluster 2021-07-31T01:59:00.723348+0000 mon.a (mon.0) 1494 : cluster [DBG] 2021-07-31T01:59:02.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:01 smithi045 conmon[33655]: osdmap e821: 8 total, 8 up, 8 in 2021-07-31T01:59:02.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:01 smithi045 conmon[33655]: cluster 2021-07-31T01:59:01.702331+0000 mon.a (mon.0) 1495 : cluster [DBG] osdmap e822: 8 total, 8 up, 8 in 2021-07-31T01:59:02.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:01 smithi045 conmon[39718]: cluster 2021-07-31T01:59:00.723348+0000 2021-07-31T01:59:02.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:01 smithi045 conmon[39718]: mon.a (mon.0) 1494 : cluster [DBG] osdmap e821: 8 total, 8 up, 8 in 2021-07-31T01:59:02.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:01 smithi045 conmon[39718]: cluster 2021-07-31T01:59:02.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:01 smithi045 conmon[39718]: 2021-07-31 2021-07-31T01:59:02.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:01 smithi045 conmon[39718]: T01:59:01.702331+0000 mon.a (mon.0) 1495 : 2021-07-31T01:59:02.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:01 smithi045 conmon[39718]: cluster [DBG] osdmap e822: 8 total, 8 up, 8 in 2021-07-31T01:59:02.718 INFO:tasks.rados.rados.0.smithi045.stdout:3976: done (5 left) 2021-07-31T01:59:02.718 INFO:tasks.rados.rados.0.smithi045.stdout:3977: done (4 left) 2021-07-31T01:59:02.718 INFO:tasks.rados.rados.0.smithi045.stdout:3978: done (3 left) 2021-07-31T01:59:02.719 INFO:tasks.rados.rados.0.smithi045.stdout:3979: done (2 left) 2021-07-31T01:59:02.719 INFO:tasks.rados.rados.0.smithi045.stdout:3980: done (1 left) 2021-07-31T01:59:02.719 INFO:tasks.rados.rados.0.smithi045.stdout:3982: done (0 left) 2021-07-31T01:59:02.719 INFO:tasks.rados.rados.0.smithi045.stdout:3983: snap_remove snap 390 2021-07-31T01:59:03.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:02 smithi154 conmon[34553]: cluster 2021-07-31T01:59:01.962473+0000 mgr.x (mgr.24542) 504 : cluster [DBG] pgmap v1023: 25 pgs: 25 active+clean; 713 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 12 MiB/s rd, 536 KiB/s wr, 31 op/s 2021-07-31T01:59:03.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:02 smithi154 conmon[34553]: cluster 2021-07-31T01:59:02.717564+0000 mon.a (mon.0) 1496 : cluster [DBG] osdmap e823: 8 total, 8 up, 8 in 2021-07-31T01:59:03.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:02 smithi045 conmon[33655]: cluster 2021-07-31T01:59:01.962473+0000 mgr.x (mgr.24542) 504 : cluster [DBG] pgmap v1023: 25 pgs: 25 active+clean; 713 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 12 MiB/s rd, 536 KiB/s wr, 31 op/s 2021-07-31T01:59:03.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:02 smithi045 conmon[33655]: cluster 2021-07-31T01:59:02.717564+0000 mon.a (mon.0) 1496 : cluster [DBG] osdmap e823: 8 total, 8 up, 8 in 2021-07-31T01:59:03.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:02 smithi045 conmon[39718]: cluster 2021-07-31T01:59:01.962473+0000 mgr.x 2021-07-31T01:59:03.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:02 smithi045 conmon[39718]: (mgr.24542) 504 : cluster [DBG] pgmap v1023: 25 pgs: 25 active+clean; 713 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 12 MiB/s rd, 536 KiB/s wr, 31 op/s 2021-07-31T01:59:03.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:02 smithi045 conmon[39718]: cluster 2021-07-31T01:59:02. 2021-07-31T01:59:03.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:02 smithi045 conmon[39718]: 717564+0000 mon.a (mon.0) 1496 : cluster [DBG] osdmap e823: 8 total, 8 up, 8 in 2021-07-31T01:59:03.720 INFO:tasks.rados.rados.0.smithi045.stdout:3983: done (0 left) 2021-07-31T01:59:03.720 INFO:tasks.rados.rados.0.smithi045.stdout:3984: snap_remove snap 340 2021-07-31T01:59:04.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:59:03 smithi045 conmon[61210]: level=error ts=2021-07-31T01:59:03.529Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:59:04.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:59:03 smithi045 conmon[61210]: level=error ts=2021-07-31T01:59:03.529Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:59:04.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:59:03 smithi045 conmon[61210]: level=error ts=2021-07-31T01:59:03.529Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T01:59:04.727 INFO:tasks.rados.rados.0.smithi045.stdout:3984: done (0 left) 2021-07-31T01:59:04.727 INFO:tasks.rados.rados.0.smithi045.stdout:3985: read oid 24 snap 373 2021-07-31T01:59:04.727 INFO:tasks.rados.rados.0.smithi045.stdout:3985: expect (ObjNum 1161 snap 373 seq_num 1161) 2021-07-31T01:59:04.728 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 20 current snap is 391 2021-07-31T01:59:04.728 INFO:tasks.rados.rados.0.smithi045.stdout:3986: seq_num 1215 ranges {1417216=753664,2170880=401408} 2021-07-31T01:59:04.739 INFO:tasks.rados.rados.0.smithi045.stdout:3986: writing smithi04569269-20 from 1417216 to 2170880 tid 1 2021-07-31T01:59:04.743 INFO:tasks.rados.rados.0.smithi045.stdout:3986: writing smithi04569269-20 from 2170880 to 2572288 tid 2 2021-07-31T01:59:04.743 INFO:tasks.rados.rados.0.smithi045.stdout:3987: rollback oid 31 current snap is 391 2021-07-31T01:59:04.743 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 31 to 352 2021-07-31T01:59:04.743 INFO:tasks.rados.rados.0.smithi045.stdout:3988: rmattr oid 1 current snap is 391 2021-07-31T01:59:04.744 INFO:tasks.rados.rados.0.smithi045.stdout:3988: done (3 left) 2021-07-31T01:59:04.744 INFO:tasks.rados.rados.0.smithi045.stdout:3989: read oid 39 snap -1 2021-07-31T01:59:04.744 INFO:tasks.rados.rados.0.smithi045.stdout:3989: expect (ObjNum 1213 snap 389 seq_num 1213) 2021-07-31T01:59:04.744 INFO:tasks.rados.rados.0.smithi045.stdout:3990: rollback oid 15 current snap is 391 2021-07-31T01:59:04.745 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 15 to 303 2021-07-31T01:59:04.745 INFO:tasks.rados.rados.0.smithi045.stdout:3985: expect (ObjNum 1161 snap 373 seq_num 1161) 2021-07-31T01:59:04.753 INFO:tasks.rados.rados.0.smithi045.stdout:3985: done (4 left) 2021-07-31T01:59:04.753 INFO:tasks.rados.rados.0.smithi045.stdout:3991: read oid 29 snap 373 2021-07-31T01:59:04.753 INFO:tasks.rados.rados.0.smithi045.stdout:3991: expect (ObjNum 1150 snap 365 seq_num 1150) 2021-07-31T01:59:04.753 INFO:tasks.rados.rados.0.smithi045.stdout:append oid 32 current snap is 391 2021-07-31T01:59:04.754 INFO:tasks.rados.rados.0.smithi045.stdout:3992: seq_num 1216 ranges {11829248=770048,12599296=401408} 2021-07-31T01:59:04.822 INFO:tasks.rados.rados.0.smithi045.stdout:3992: writing smithi04569269-32 from 11829248 to 12599296 tid 1 2021-07-31T01:59:04.825 INFO:tasks.rados.rados.0.smithi045.stdout:3992: writing smithi04569269-32 from 12599296 to 13000704 tid 2 2021-07-31T01:59:04.826 INFO:tasks.rados.rados.0.smithi045.stdout:3993: read oid 13 snap -1 2021-07-31T01:59:04.826 INFO:tasks.rados.rados.0.smithi045.stdout:3993: expect deleted 2021-07-31T01:59:04.826 INFO:tasks.rados.rados.0.smithi045.stdout:3994: rmattr oid 37 current snap is 391 2021-07-31T01:59:04.826 INFO:tasks.rados.rados.0.smithi045.stdout:3994: done (7 left) 2021-07-31T01:59:04.826 INFO:tasks.rados.rados.0.smithi045.stdout:3995: read oid 25 snap -1 2021-07-31T01:59:04.827 INFO:tasks.rados.rados.0.smithi045.stdout:3995: expect (ObjNum 1157 snap 372 seq_num 1157) 2021-07-31T01:59:04.827 INFO:tasks.rados.rados.0.smithi045.stdout:3996: delete oid 10 current snap is 391 2021-07-31T01:59:04.827 INFO:tasks.rados.rados.0.smithi045.stdout:3990: finishing rollback tid 0 to smithi04569269-15 2021-07-31T01:59:04.827 INFO:tasks.rados.rados.0.smithi045.stdout:3989: expect (ObjNum 1213 snap 389 seq_num 1213) 2021-07-31T01:59:04.855 INFO:tasks.rados.rados.0.smithi045.stdout:3986: finishing write tid 1 to smithi04569269-20 2021-07-31T01:59:04.856 INFO:tasks.rados.rados.0.smithi045.stdout:3990: finishing rollback tid 1 to smithi04569269-15 2021-07-31T01:59:04.856 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 15 v 2551 (ObjNum 288 snap 0 seq_num 33) dirty dne 2021-07-31T01:59:04.856 INFO:tasks.rados.rados.0.smithi045.stdout:3986: finishing write tid 2 to smithi04569269-20 2021-07-31T01:59:04.856 INFO:tasks.rados.rados.0.smithi045.stdout:3986: finishing write tid 3 to smithi04569269-20 2021-07-31T01:59:04.857 INFO:tasks.rados.rados.0.smithi045.stdout:3987: finishing rollback tid 0 to smithi04569269-31 2021-07-31T01:59:04.857 INFO:tasks.rados.rados.0.smithi045.stdout:3987: finishing rollback tid 1 to smithi04569269-31 2021-07-31T01:59:04.857 INFO:tasks.rados.rados.0.smithi045.stdout:3989: done (8 left) 2021-07-31T01:59:04.857 INFO:tasks.rados.rados.0.smithi045.stdout:3990: done (7 left) 2021-07-31T01:59:04.858 INFO:tasks.rados.rados.0.smithi045.stdout:3996: done (6 left) 2021-07-31T01:59:04.858 INFO:tasks.rados.rados.0.smithi045.stdout:3997: rmattr oid 16 current snap is 391 2021-07-31T01:59:04.858 INFO:tasks.rados.rados.0.smithi045.stdout:3987: finishing rollback tid 2 to smithi04569269-31 2021-07-31T01:59:04.858 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 31 v 3064 (ObjNum 1006 snap 322 seq_num 1006) dirty exists 2021-07-31T01:59:04.859 INFO:tasks.rados.rados.0.smithi045.stdout:3986: finishing write tid 4 to smithi04569269-20 2021-07-31T01:59:04.859 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 20 v 3060 (ObjNum 1215 snap 391 seq_num 1215) dirty exists 2021-07-31T01:59:04.859 INFO:tasks.rados.rados.0.smithi045.stdout:3986: left oid 20 (ObjNum 1215 snap 391 seq_num 1215) 2021-07-31T01:59:04.859 INFO:tasks.rados.rados.0.smithi045.stdout:3991: expect (ObjNum 1150 snap 365 seq_num 1150) 2021-07-31T01:59:04.905 INFO:tasks.rados.rados.0.smithi045.stdout:3995: expect (ObjNum 1157 snap 372 seq_num 1157) 2021-07-31T01:59:04.910 INFO:tasks.rados.rados.0.smithi045.stdout:3992: finishing write tid 1 to smithi04569269-32 2021-07-31T01:59:04.910 INFO:tasks.rados.rados.0.smithi045.stdout:3992: finishing write tid 2 to smithi04569269-32 2021-07-31T01:59:04.910 INFO:tasks.rados.rados.0.smithi045.stdout:3992: finishing write tid 3 to smithi04569269-32 2021-07-31T01:59:04.911 INFO:tasks.rados.rados.0.smithi045.stdout:3992: finishing write tid 4 to smithi04569269-32 2021-07-31T01:59:04.911 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 32 v 1894 (ObjNum 1216 snap 391 seq_num 1216) dirty exists 2021-07-31T01:59:04.911 INFO:tasks.rados.rados.0.smithi045.stdout:3992: left oid 32 (ObjNum 1216 snap 391 seq_num 1216) 2021-07-31T01:59:04.911 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 16 v 1868 (ObjNum 1184 snap 383 seq_num 1184) dirty exists 2021-07-31T01:59:04.912 INFO:tasks.rados.rados.0.smithi045.stdout:3986: done (6 left) 2021-07-31T01:59:04.912 INFO:tasks.rados.rados.0.smithi045.stdout:3987: done (5 left) 2021-07-31T01:59:04.912 INFO:tasks.rados.rados.0.smithi045.stdout:3991: done (4 left) 2021-07-31T01:59:04.912 INFO:tasks.rados.rados.0.smithi045.stdout:3992: done (3 left) 2021-07-31T01:59:04.912 INFO:tasks.rados.rados.0.smithi045.stdout:3993: done (2 left) 2021-07-31T01:59:04.913 INFO:tasks.rados.rados.0.smithi045.stdout:3995: done (1 left) 2021-07-31T01:59:04.913 INFO:tasks.rados.rados.0.smithi045.stdout:3997: done (0 left) 2021-07-31T01:59:04.913 INFO:tasks.rados.rados.0.smithi045.stdout:3998: rollback oid 25 current snap is 391 2021-07-31T01:59:04.914 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 25 to 352 2021-07-31T01:59:04.914 INFO:tasks.rados.rados.0.smithi045.stdout:3999: rollback oid 9 current snap is 391 2021-07-31T01:59:04.914 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 9 to 363 2021-07-31T01:59:04.914 INFO:tasks.rados.rados.0.smithi045.stdout: waiting on 2 2021-07-31T01:59:04.916 INFO:tasks.rados.rados.0.smithi045.stdout:3999: finishing rollback tid 1 to smithi04569269-9 2021-07-31T01:59:04.916 INFO:tasks.rados.rados.0.smithi045.stdout:3999: finishing rollback tid 2 to smithi04569269-9 2021-07-31T01:59:04.916 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 9 v 2387 (ObjNum 1045 snap 340 seq_num 1045) dirty exists 2021-07-31T01:59:04.916 INFO:tasks.rados.rados.0.smithi045.stdout:3999: done (1 left) 2021-07-31T01:59:04.917 INFO:tasks.rados.rados.0.smithi045.stdout: waiting on 1 2021-07-31T01:59:04.917 INFO:tasks.rados.rados.0.smithi045.stdout:3998: finishing rollback tid 0 to smithi04569269-25 2021-07-31T01:59:04.917 INFO:tasks.rados.rados.0.smithi045.stdout:3998: finishing rollback tid 1 to smithi04569269-25 2021-07-31T01:59:04.917 INFO:tasks.rados.rados.0.smithi045.stdout:3998: finishing rollback tid 2 to smithi04569269-25 2021-07-31T01:59:04.917 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 25 v 1859 (ObjNum 837 snap 263 seq_num 837) dirty exists 2021-07-31T01:59:04.918 INFO:tasks.rados.rados.0.smithi045.stdout:3998: done (0 left) 2021-07-31T01:59:04.918 INFO:tasks.rados.rados.0.smithi045.stderr:0 errors. 2021-07-31T01:59:04.918 INFO:tasks.rados.rados.0.smithi045.stderr: 2021-07-31T01:59:04.921 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell --fsid f52d2c14-f19f-11eb-8c24-001a4aab830c -- ceph osd dump --format=json 2021-07-31T01:59:04.996 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:04 smithi045 conmon[39718]: cluster 2021-07-31T01:59:03.719584+0000 mon.a (mon. 2021-07-31T01:59:04.996 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:04 smithi045 conmon[39718]: 0) 1497 : cluster [DBG] osdmap e824: 8 total, 8 up, 8 in 2021-07-31T01:59:04.996 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:04 smithi045 conmon[39718]: cluster 2021-07-31T01:59:03.962951+0000 2021-07-31T01:59:04.997 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:04 smithi045 conmon[39718]: mgr.x (mgr.24542) 505 : cluster [DBG] pgmap v1026: 25 pgs: 25 active+clean; 715 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 4.3 MiB/s rd, 682 KiB/s wr, 10 op/s 2021-07-31T01:59:04.997 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:04 smithi045 conmon[33655]: cluster 2021-07-31T01:59:03.719584+0000 2021-07-31T01:59:04.997 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:04 smithi045 conmon[33655]: mon.a (mon.0) 1497 : cluster [DBG] osdmap e824: 8 total, 8 up, 8 in 2021-07-31T01:59:04.998 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:04 smithi045 conmon[33655]: cluster 2021-07-31T01:59:03 2021-07-31T01:59:04.998 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:04 smithi045 conmon[33655]: .962951+0000 mgr.x (mgr.24542) 505 : cluster [DBG] pgmap v1026: 25 pgs: 25 active+clean; 715 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 4.3 MiB/s rd, 682 KiB/s wr, 10 op/s 2021-07-31T01:59:05.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:04 smithi154 conmon[34553]: cluster 2021-07-31T01:59:03.719584+0000 mon.a (mon.0) 1497 : cluster 2021-07-31T01:59:05.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:04 smithi154 conmon[34553]: [DBG] osdmap e824: 8 total, 8 up, 8 in 2021-07-31T01:59:05.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:04 smithi154 conmon[34553]: cluster 2021-07-31T01:59:03.962951+0000 mgr.x (mgr.24542) 505 : cluster [DBG] pgmap v1026: 25 pgs: 25 active+clean; 715 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 4.3 MiB/s rd, 682 KiB/s wr, 10 op/s 2021-07-31T01:59:05.421 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f52d2c14-f19f-11eb-8c24-001a4aab830c/mon.a/config 2021-07-31T01:59:05.862 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:05 smithi045 conmon[33655]: cluster 2021-07-31T01:59:04.724809+0000 2021-07-31T01:59:05.863 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:05 smithi045 conmon[33655]: mon.a (mon.0) 1498 : cluster [DBG] osdmap e825: 8 total, 8 up, 8 in 2021-07-31T01:59:05.863 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:05 smithi045 conmon[39718]: cluster 2021-07-31T01:59:04 2021-07-31T01:59:05.863 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:05 smithi045 conmon[39718]: .724809+0000 mon.a (mon.0) 1498 : cluster [DBG] osdmap e825: 8 total, 8 up, 8 in 2021-07-31T01:59:06.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:05 smithi154 conmon[34553]: cluster 2021-07-31T01:59:04.724809+0000 mon.a (mon.0) 1498 : cluster [DBG] osdmap e825: 8 total, 8 up, 8 in 2021-07-31T01:59:06.892 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:06 smithi045 conmon[33655]: cluster 2021-07-31T01:59:05.963508+0000 mgr.x (mgr.24542) 506 : cluster [DBG] pgmap v1028: 25 pgs: 25 active+clean; 715 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 4.0 MiB/s rd, 640 KiB/s wr, 10 op/s 2021-07-31T01:59:06.892 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:06 smithi045 conmon[33655]: cluster 2021-07-31T01:59:06.703972+0000 mon.a (mon.0) 1499 : cluster [DBG] osdmap e826: 8 total, 8 up, 8 in 2021-07-31T01:59:06.893 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:06 smithi045 conmon[39718]: cluster 2021-07-31T01:59:05.963508+0000 mgr.x (mgr.24542) 506 : cluster [DBG] pgmap v1028: 25 pgs: 25 active+clean; 715 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 4.0 MiB/s rd, 640 KiB/s wr, 10 op/s 2021-07-31T01:59:06.893 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:06 smithi045 conmon[39718]: cluster 2021-07-31T01:59:06.703972+0000 mon.a (mon.0) 1499 : cluster [DBG] osdmap e826: 8 total, 8 up, 8 in 2021-07-31T01:59:07.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:06 smithi154 conmon[34553]: cluster 2021-07-31T01:59:05.963508+0000 mgr.x (mgr.24542) 506 : cluster [DBG] pgmap v1028: 25 pgs: 25 active+clean; 715 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 4.0 MiB/s rd, 640 KiB/s wr, 10 op/s 2021-07-31T01:59:07.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:06 smithi154 conmon[34553]: 2021-07-31T01:59:07.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:06 smithi154 conmon[34553]: cluster 2021-07-31T01:59:06.703972+0000 mon.a (mon.0) 1499 : cluster [DBG] osdmap e826: 8 total, 8 up, 8 in 2021-07-31T01:59:07.251 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:59:07.251 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":826,"fsid":"f52d2c14-f19f-11eb-8c24-001a4aab830c","created":"2021-07-31T01:38:50.680976+0000","modified":"2021-07-31T01:59:06.698815+0000","last_up_change":"2021-07-31T01:43:46.985098+0000","last_in_change":"2021-07-31T01:43:30.970323+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":19,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":".mgr","create_time":"2021-07-31T01:42:07.901706+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"151","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr":{},"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-07-31T01:44:10.685434+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2021-07-31T01:45:51.225389+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"825","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":760,"snap_epoch":825,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"teuthologyprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"20724b9a-515c-4878-9e4f-301e21a5a985","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":2841572524},{"type":"v1","addr":"172.21.15.45:6803","nonce":2841572524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":2841572524},{"type":"v1","addr":"172.21.15.45:6805","nonce":2841572524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":2841572524},{"type":"v1","addr":"172.21.15.45:6809","nonce":2841572524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":2841572524},{"type":"v1","addr":"172.21.15.45:6807","nonce":2841572524}]},"public_addr":"172.21.15.45:6803/2841572524","cluster_addr":"172.21.15.45:6805/2841572524","heartbeat_back_addr":"172.21.15.45:6809/2841572524","heartbeat_front_addr":"172.21.15.45:6807/2841572524","state":["exists","up"]},{"osd":1,"uuid":"f47f025a-6f29-417a-9bd1-9af9f91f38a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1747309179},{"type":"v1","addr":"172.21.15.45:6811","nonce":1747309179}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1747309179},{"type":"v1","addr":"172.21.15.45:6813","nonce":1747309179}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":1747309179},{"type":"v1","addr":"172.21.15.45:6817","nonce":1747309179}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1747309179},{"type":"v1","addr":"172.21.15.45:6815","nonce":1747309179}]},"public_addr":"172.21.15.45:6811/1747309179","cluster_addr":"172.21.15.45:6813/1747309179","heartbeat_back_addr":"172.21.15.45:6817/1747309179","heartbeat_front_addr":"172.21.15.45:6815/1747309179","state":["exists","up"]},{"osd":2,"uuid":"bfbe641e-b9d7-4005-8fb5-bc7d5af76388","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2518799988},{"type":"v1","addr":"172.21.15.45:6819","nonce":2518799988}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2518799988},{"type":"v1","addr":"172.21.15.45:6821","nonce":2518799988}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2518799988},{"type":"v1","addr":"172.21.15.45:6825","nonce":2518799988}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2518799988},{"type":"v1","addr":"172.21.15.45:6823","nonce":2518799988}]},"public_addr":"172.21.15.45:6819/2518799988","cluster_addr":"172.21.15.45:6821/2518799988","heartbeat_back_addr":"172.21.15.45:6825/2518799988","heartbeat_front_addr":"172.21.15.45:6823/2518799988","state":["exists","up"]},{"osd":3,"uuid":"37b826a3-b611-4001-b9a7-1966032e56d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3024698157},{"type":"v1","addr":"172.21.15.45:6827","nonce":3024698157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3024698157},{"type":"v1","addr":"172.21.15.45:6829","nonce":3024698157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3024698157},{"type":"v1","addr":"172.21.15.45:6833","nonce":3024698157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3024698157},{"type":"v1","addr":"172.21.15.45:6831","nonce":3024698157}]},"public_addr":"172.21.15.45:6827/3024698157","cluster_addr":"172.21.15.45:6829/3024698157","heartbeat_back_addr":"172.21.15.45:6833/3024698157","heartbeat_front_addr":"172.21.15.45:6831/3024698157","state":["exists","up"]},{"osd":4,"uuid":"69881d9d-3e56-4483-b76b-7dba14cfaac3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":4277144246},{"type":"v1","addr":"172.21.15.154:6801","nonce":4277144246}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":4277144246},{"type":"v1","addr":"172.21.15.154:6803","nonce":4277144246}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":4277144246},{"type":"v1","addr":"172.21.15.154:6807","nonce":4277144246}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":4277144246},{"type":"v1","addr":"172.21.15.154:6805","nonce":4277144246}]},"public_addr":"172.21.15.154:6801/4277144246","cluster_addr":"172.21.15.154:6803/4277144246","heartbeat_back_addr":"172.21.15.154:6807/4277144246","heartbeat_front_addr":"172.21.15.154:6805/4277144246","state":["exists","up"]},{"osd":5,"uuid":"7dcad412-f42a-478b-886a-8f4bd82a04ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":579203063},{"type":"v1","addr":"172.21.15.154:6809","nonce":579203063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":579203063},{"type":"v1","addr":"172.21.15.154:6811","nonce":579203063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":579203063},{"type":"v1","addr":"172.21.15.154:6815","nonce":579203063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":579203063},{"type":"v1","addr":"172.21.15.154:6813","nonce":579203063}]},"public_addr":"172.21.15.154:6809/579203063","cluster_addr":"172.21.15.154:6811/579203063","heartbeat_back_addr":"172.21.15.154:6815/579203063","heartbeat_front_addr":"172.21.15.154:6813/579203063","state":["exists","up"]},{"osd":6,"uuid":"94258018-67f7-49d7-83c3-85070ff8c2f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":3963397470},{"type":"v1","addr":"172.21.15.154:6817","nonce":3963397470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":3963397470},{"type":"v1","addr":"172.21.15.154:6819","nonce":3963397470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":3963397470},{"type":"v1","addr":"172.21.15.154:6823","nonce":3963397470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":3963397470},{"type":"v1","addr":"172.21.15.154:6821","nonce":3963397470}]},"public_addr":"172.21.15.154:6817/3963397470","cluster_addr":"172.21.15.154:6819/3963397470","heartbeat_back_addr":"172.21.15.154:6823/3963397470","heartbeat_front_addr":"172.21.15.154:6821/3963397470","state":["exists","up"]},{"osd":7,"uuid":"da82715a-b8b7-4786-bbc2-ee0cc3ac9b18","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1263188610},{"type":"v1","addr":"172.21.15.154:6825","nonce":1263188610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1263188610},{"type":"v1","addr":"172.21.15.154:6827","nonce":1263188610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1263188610},{"type":"v1","addr":"172.21.15.154:6831","nonce":1263188610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1263188610},{"type":"v1","addr":"172.21.15.154:6829","nonce":1263188610}]},"public_addr":"172.21.15.154:6825/1263188610","cluster_addr":"172.21.15.154:6827/1263188610","heartbeat_back_addr":"172.21.15.154:6831/1263188610","heartbeat_front_addr":"172.21.15.154:6829/1263188610","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-07-31T01:41:26.799531+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-07-31T01:41:46.224506+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-07-31T01:42:06.246092+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-07-31T01:42:26.612987+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-07-31T01:42:45.424194+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-07-31T01:43:06.036564+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-07-31T01:43:25.839643+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-07-31T01:43:45.387900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:6800/1006624185":"2021-08-01T01:47:26.793400+0000","172.21.15.45:0/4027209229":"2021-08-01T01:47:26.793400+0000","172.21.15.45:0/3355281269":"2021-08-01T01:39:39.238202+0000","172.21.15.45:6801/2610418727":"2021-08-01T01:39:39.238202+0000","172.21.15.45:6800/2610418727":"2021-08-01T01:39:39.238202+0000","172.21.15.45:6800/487754334":"2021-08-01T01:39:15.200151+0000","172.21.15.45:6801/1006624185":"2021-08-01T01:47:26.793400+0000","172.21.15.45:0/536736800":"2021-08-01T01:39:39.238202+0000","172.21.15.45:0/4040877953":"2021-08-01T01:39:15.200151+0000","172.21.15.45:0/2869241044":"2021-08-01T01:47:26.793400+0000","172.21.15.45:0/2377876452":"2021-08-01T01:39:15.200151+0000","172.21.15.45:6801/487754334":"2021-08-01T01:39:15.200151+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"teuthologyprofile":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":641,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":732,"length":1},{"begin":758,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-07-31T01:59:07.992 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:07 smithi045 conmon[33655]: audit 2021-07-31T01:59:07.249908+0000 2021-07-31T01:59:07.993 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:07 smithi045 conmon[33655]: mon.a (mon.0) 1500 : audit [DBG] from='client.? 172.21.15.45:0/1208459269' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-07-31T01:59:07.993 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:07 smithi045 conmon[39718]: audit 2021-07-31T01:59:07.249908+0000 mon.a ( 2021-07-31T01:59:07.994 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:07 smithi045 conmon[39718]: mon.0) 1500 : audit [DBG] from='client.? 172.21.15.45:0/1208459269' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-07-31T01:59:08.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:07 smithi154 conmon[34553]: audit 2021-07-31T01:59:07.249908+0000 mon.a (mon.0) 1500 : audit [DBG] from='client.? 172.21.15.45:0/1208459269' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-07-31T01:59:08.046 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell --fsid f52d2c14-f19f-11eb-8c24-001a4aab830c -- ceph pg dump --format=json 2021-07-31T01:59:08.537 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f52d2c14-f19f-11eb-8c24-001a4aab830c/mon.a/config 2021-07-31T01:59:08.998 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:08 smithi045 conmon[39718]: cluster 2021-07-31T01:59:07.964007+0000 mgr.x (mgr.24542) 507 : cluster 2021-07-31T01:59:08.999 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:08 smithi045 conmon[39718]: [DBG] pgmap v1030: 25 pgs: 25 active+clean; 715 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 3.3 MiB/s rd, 520 KiB/s wr, 8 op/s 2021-07-31T01:59:08.999 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:08 smithi045 conmon[33655]: cluster 2021-07-31T01:59:07 2021-07-31T01:59:08.999 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:08 smithi045 conmon[33655]: .964007+0000 mgr.x (mgr.24542) 507 : cluster [DBG] pgmap v1030: 25 pgs: 25 active+clean; 715 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 3.3 MiB/s rd, 520 KiB/s wr, 8 op/s 2021-07-31T01:59:09.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:08 smithi154 conmon[34553]: cluster 2021-07-31T01:59:07.964007+0000 mgr.x (mgr.24542) 507 : cluster [DBG] pgmap v1030: 25 pgs: 25 active+clean; 715 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 3.3 MiB/s rd, 520 KiB/s wr, 8 op/s 2021-07-31T01:59:10.251 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-07-31T01:59:10.252 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2021-07-31T01:59:11.046 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":1031,"stamp":"2021-07-31T01:59:09.964223+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":760807968,"num_objects":341,"num_object_clones":289,"num_object_copies":1023,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":341,"num_whiteouts":10,"num_read":4002,"num_read_kb":6237425,"num_write":8603,"num_write_kb":1941704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18247,"ondisk_log_size":18247,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":75,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":3339096,"kb_used_data":1016152,"kb_used_omap":0,"kb_used_meta":2322880,"kb_avail":746622120,"statfs":{"total":767960285184,"available":764541050880,"internally_reserved":0,"allocated":1040539648,"data_stored":2027376904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2378629120},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":13131776,"num_objects":9,"num_object_clones":9,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":-1,"num_read":30,"num_read_kb":25402,"num_write":67,"num_write_kb":5000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.244750"},"pg_stats":[{"pgid":"3.f","version":"825'767","reported_seq":3174,"reported_epoch":826,"state":"active+clean","last_fresh":"2021-07-31T01:59:06.826631+0000","last_change":"2021-07-31T01:59:04.736615+0000","last_active":"2021-07-31T01:59:06.826631+0000","last_peered":"2021-07-31T01:59:06.826631+0000","last_clean":"2021-07-31T01:59:06.826631+0000","last_became_active":"2021-07-31T01:45:52.737194+0000","last_became_peered":"2021-07-31T01:45:52.737194+0000","last_unstale":"2021-07-31T01:59:06.826631+0000","last_undegraded":"2021-07-31T01:59:06.826631+0000","last_fullsized":"2021-07-31T01:59:06.826631+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-07-31T01:45:51.724604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-07-31T01:45:51.724604+0000","last_clean_scrub_stamp":"2021-07-31T01:45:51.724604+0000","log_size":767,"ondisk_log_size":767,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44163072,"num_objects":14,"num_object_clones":12,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":181,"num_read_kb":341724,"num_write":365,"num_write_kb":96128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"281","length":"1"}]},{"pgid":"3.e","version":"825'1681","reported_seq":3997,"reported_epoch":826,"state":"active+clean","last_fresh":"2021-07-31T01:59:06.826528+0000","last_change":"2021-07-31T01:59:04.755784+0000","last_active":"2021-07-31T01:59:06.826528+0000","last_peered":"2021-07-31T01:59:06.826528+0000","last_clean":"2021-07-31T01:59:06.826528+0000","last_became_active":"2021-07-31T01:45:52.738470+0000","last_became_peered":"2021-07-31T01:45:52.738470+0000","last_unstale":"2021-07-31T01:59:06.826528+0000","last_undegraded":"2021-07-31T01:59:06.826528+0000","last_fullsized":"2021-07-31T01:59:06.826528+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-07-31T01:45:51.724604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-07-31T01:45:51.724604+0000","last_clean_scrub_stamp":"2021-07-31T01:45:51.724604+0000","log_size":1681,"ondisk_log_size":1681,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":66338816,"num_objects":25,"num_object_clones":21,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":2,"num_read":314,"num_read_kb":520124,"num_write":681,"num_write_kb":160168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"281","length":"1"}]},{"pgid":"3.d","version":"825'2236","reported_seq":4580,"reported_epoch":826,"state":"active+clean","last_fresh":"2021-07-31T01:59:06.826550+0000","last_change":"2021-07-31T01:59:04.762457+0000","last_active":"2021-07-31T01:59:06.826550+0000","last_peered":"2021-07-31T01:59:06.826550+0000","last_clean":"2021-07-31T01:59:06.826550+0000","last_became_active":"2021-07-31T01:45:52.741010+0000","last_became_peered":"2021-07-31T01:45:52.741010+0000","last_unstale":"2021-07-31T01:59:06.826550+0000","last_undegraded":"2021-07-31T01:59:06.826550+0000","last_fullsized":"2021-07-31T01:59:06.826550+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-07-31T01:45:51.724604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-07-31T01:45:51.724604+0000","last_clean_scrub_stamp":"2021-07-31T01:45:51.724604+0000","log_size":2236,"ondisk_log_size":2236,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":107429888,"num_objects":43,"num_object_clones":37,"num_object_copies":129,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":2,"num_read":507,"num_read_kb":835508,"num_write":973,"num_write_kb":234560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"281","length":"1"}]},{"pgid":"3.c","version":"825'1905","reported_seq":4242,"reported_epoch":826,"state":"active+clean","last_fresh":"2021-07-31T01:59:06.826864+0000","last_change":"2021-07-31T01:59:04.757496+0000","last_active":"2021-07-31T01:59:06.826864+0000","last_peered":"2021-07-31T01:59:06.826864+0000","last_clean":"2021-07-31T01:59:06.826864+0000","last_became_active":"2021-07-31T01:45:52.741022+0000","last_became_peered":"2021-07-31T01:45:52.741022+0000","last_unstale":"2021-07-31T01:59:06.826864+0000","last_undegraded":"2021-07-31T01:59:06.826864+0000","last_fullsized":"2021-07-31T01:59:06.826864+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-07-31T01:45:51.724604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-07-31T01:45:51.724604+0000","last_clean_scrub_stamp":"2021-07-31T01:45:51.724604+0000","log_size":1905,"ondisk_log_size":1905,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":56467456,"num_objects":37,"num_object_clones":31,"num_object_copies":111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":1,"num_read":425,"num_read_kb":615958,"num_write":847,"num_write_kb":186512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"281","length":"1"}]},{"pgid":"3.b","version":"825'3074","reported_seq":5320,"reported_epoch":826,"state":"active+clean","last_fresh":"2021-07-31T01:59:06.826771+0000","last_change":"2021-07-31T01:59:04.775408+0000","last_active":"2021-07-31T01:59:06.826771+0000","last_peered":"2021-07-31T01:59:06.826771+0000","last_clean":"2021-07-31T01:59:06.826771+0000","last_became_active":"2021-07-31T01:45:52.741057+0000","last_became_peered":"2021-07-31T01:45:52.741057+0000","last_unstale":"2021-07-31T01:59:06.826771+0000","last_undegraded":"2021-07-31T01:59:06.826771+0000","last_fullsized":"2021-07-31T01:59:06.826771+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-07-31T01:45:51.724604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-07-31T01:45:51.724604+0000","last_clean_scrub_stamp":"2021-07-31T01:45:51.724604+0000","log_size":3074,"ondisk_log_size":3074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":134266880,"num_objects":62,"num_object_clones":54,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":608,"num_read_kb":1057693,"num_write":1453,"num_write_kb":342712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"281","length":"1"}]},{"pgid":"3.a","version":"825'1690","reported_seq":3982,"reported_epoch":826,"state":"active+clean","last_fresh":"2021-07-31T01:59:06.827384+0000","last_change":"2021-07-31T01:59:04.756738+0000","last_active":"2021-07-31T01:59:06.827384+0000","last_peered":"2021-07-31T01:59:06.827384+0000","last_clean":"2021-07-31T01:59:06.827384+0000","last_became_active":"2021-07-31T01:45:52.736070+0000","last_became_peered":"2021-07-31T01:45:52.736070+0000","last_unstale":"2021-07-31T01:59:06.827384+0000","last_undegraded":"2021-07-31T01:59:06.827384+0000","last_fullsized":"2021-07-31T01:59:06.827384+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-07-31T01:45:51.724604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-07-31T01:45:51.724604+0000","last_clean_scrub_stamp":"2021-07-31T01:45:51.724604+0000","log_size":1690,"ondisk_log_size":1690,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":68231168,"num_objects":31,"num_object_clones":27,"num_object_copies":93,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":288,"num_read_kb":398951,"num_write":866,"num_write_kb":159320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"281","length":"1"}]},{"pgid":"3.9","version":"825'699","reported_seq":3101,"reported_epoch":826,"state":"active+clean","last_fresh":"2021-07-31T01:59:06.704071+0000","last_change":"2021-07-31T01:59:04.737341+0000","last_active":"2021-07-31T01:59:06.704071+0000","last_peered":"2021-07-31T01:59:06.704071+0000","last_clean":"2021-07-31T01:59:06.704071+0000","last_became_active":"2021-07-31T01:45:52.737316+0000","last_became_peered":"2021-07-31T01:45:52.737316+0000","last_unstale":"2021-07-31T01:59:06.704071+0000","last_undegraded":"2021-07-31T01:59:06.704071+0000","last_fullsized":"2021-07-31T01:59:06.704071+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-07-31T01:45:51.724604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-07-31T01:45:51.724604+0000","last_clean_scrub_stamp":"2021-07-31T01:45:51.724604+0000","log_size":699,"ondisk_log_size":699,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37756928,"num_objects":11,"num_object_clones":9,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":153,"num_read_kb":250111,"num_write":401,"num_write_kb":86240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"281","length":"1"}]},{"pgid":"3.8","version":"825'733","reported_seq":3104,"reported_epoch":826,"state":"active+clean","last_fresh":"2021-07-31T01:59:06.826716+0000","last_change":"2021-07-31T01:59:04.729880+0000","last_active":"2021-07-31T01:59:06.826716+0000","last_peered":"2021-07-31T01:59:06.826716+0000","last_clean":"2021-07-31T01:59:06.826716+0000","last_became_active":"2021-07-31T01:45:52.736600+0000","last_became_peered":"2021-07-31T01:45:52.736600+0000","last_unstale":"2021-07-31T01:59:06.826716+0000","last_undegraded":"2021-07-31T01:59:06.826716+0000","last_fullsized":"2021-07-31T01:59:06.826716+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-07-31T01:45:51.724604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-07-31T01:45:51.724604+0000","last_clean_scrub_stamp":"2021-07-31T01:45:51.724604+0000","log_size":733,"ondisk_log_size":733,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15089664,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":150,"num_read_kb":125209,"num_write":291,"num_write_kb":58544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"281","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":766,"reported_epoch":825,"state":"active+clean","last_fresh":"2021-07-31T01:59:04.725848+0000","last_change":"2021-07-31T01:44:12.036948+0000","last_active":"2021-07-31T01:59:04.725848+0000","last_peered":"2021-07-31T01:59:04.725848+0000","last_clean":"2021-07-31T01:59:04.725848+0000","last_became_active":"2021-07-31T01:44:12.036181+0000","last_became_peered":"2021-07-31T01:44:12.036181+0000","last_unstale":"2021-07-31T01:59:04.725848+0000","last_undegraded":"2021-07-31T01:59:04.725848+0000","last_fullsized":"2021-07-31T01:59:04.725848+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-07-31T01:44:10.735599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-07-31T01:44:10.735599+0000","last_clean_scrub_stamp":"2021-07-31T01:44:10.735599+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"825'378","reported_seq":2806,"reported_epoch":826,"state":"active+clean","last_fresh":"2021-07-31T01:59:06.826988+0000","last_change":"2021-07-31T01:59:04.738860+0000","last_active":"2021-07-31T01:59:06.826988+0000","last_peered":"2021-07-31T01:59:06.826988+0000","last_clean":"2021-07-31T01:59:06.826988+0000","last_became_active":"2021-07-31T01:45:52.741890+0000","last_became_peered":"2021-07-31T01:45:52.741890+0000","last_unstale":"2021-07-31T01:59:06.826988+0000","last_undegraded":"2021-07-31T01:59:06.826988+0000","last_fullsized":"2021-07-31T01:59:06.826988+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-07-31T01:45:51.724604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-07-31T01:45:51.724604+0000","last_clean_scrub_stamp":"2021-07-31T01:45:51.724604+0000","log_size":378,"ondisk_log_size":378,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18194432,"num_objects":8,"num_object_clones":7,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":109,"num_read_kb":302792,"num_write":132,"num_write_kb":39344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"281","length":"1"}]},{"pgid":"2.2","version":"0'0","reported_seq":758,"reported_epoch":825,"state":"active+clean","last_fresh":"2021-07-31T01:59:04.727611+0000","last_change":"2021-07-31T01:44:11.748203+0000","last_active":"2021-07-31T01:59:04.727611+0000","last_peered":"2021-07-31T01:59:04.727611+0000","last_clean":"2021-07-31T01:59:04.727611+0000","last_became_active":"2021-07-31T01:44:11.747897+0000","last_became_peered":"2021-07-31T01:44:11.747897+0000","last_unstale":"2021-07-31T01:59:04.727611+0000","last_undegraded":"2021-07-31T01:59:04.727611+0000","last_fullsized":"2021-07-31T01:59:04.727611+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-07-31T01:44:10.735599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-07-31T01:44:10.735599+0000","last_clean_scrub_stamp":"2021-07-31T01:44:10.735599+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"825'379","reported_seq":2798,"reported_epoch":826,"state":"active+clean","last_fresh":"2021-07-31T01:59:06.703987+0000","last_change":"2021-07-31T01:59:04.744434+0000","last_active":"2021-07-31T01:59:06.703987+0000","last_peered":"2021-07-31T01:59:06.703987+0000","last_clean":"2021-07-31T01:59:06.703987+0000","last_became_active":"2021-07-31T01:45:52.737428+0000","last_became_peered":"2021-07-31T01:45:52.737428+0000","last_unstale":"2021-07-31T01:59:06.703987+0000","last_undegraded":"2021-07-31T01:59:06.703987+0000","last_fullsized":"2021-07-31T01:59:06.703987+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-07-31T01:45:51.724604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-07-31T01:45:51.724604+0000","last_clean_scrub_stamp":"2021-07-31T01:45:51.724604+0000","log_size":379,"ondisk_log_size":379,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4415488,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":78,"num_read_kb":99392,"num_write":236,"num_write_kb":48192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"281","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":759,"reported_epoch":825,"state":"active+clean","last_fresh":"2021-07-31T01:59:04.729258+0000","last_change":"2021-07-31T01:44:12.036886+0000","last_active":"2021-07-31T01:59:04.729258+0000","last_peered":"2021-07-31T01:59:04.729258+0000","last_clean":"2021-07-31T01:59:04.729258+0000","last_became_active":"2021-07-31T01:44:12.036017+0000","last_became_peered":"2021-07-31T01:44:12.036017+0000","last_unstale":"2021-07-31T01:59:04.729258+0000","last_undegraded":"2021-07-31T01:59:04.729258+0000","last_fullsized":"2021-07-31T01:59:04.729258+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-07-31T01:44:10.735599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-07-31T01:44:10.735599+0000","last_clean_scrub_stamp":"2021-07-31T01:44:10.735599+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"825'1457","reported_seq":3849,"reported_epoch":826,"state":"active+clean","last_fresh":"2021-07-31T01:59:06.826000+0000","last_change":"2021-07-31T01:59:04.738979+0000","last_active":"2021-07-31T01:59:06.826000+0000","last_peered":"2021-07-31T01:59:06.826000+0000","last_clean":"2021-07-31T01:59:06.826000+0000","last_became_active":"2021-07-31T01:45:52.737767+0000","last_became_peered":"2021-07-31T01:45:52.737767+0000","last_unstale":"2021-07-31T01:59:06.826000+0000","last_undegraded":"2021-07-31T01:59:06.826000+0000","last_fullsized":"2021-07-31T01:59:06.826000+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-07-31T01:45:51.724604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-07-31T01:45:51.724604+0000","last_clean_scrub_stamp":"2021-07-31T01:45:51.724604+0000","log_size":1457,"ondisk_log_size":1457,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":67903488,"num_objects":25,"num_object_clones":21,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":1,"num_read":346,"num_read_kb":641167,"num_write":721,"num_write_kb":167880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"281","length":"1"}]},{"pgid":"2.3","version":"0'0","reported_seq":758,"reported_epoch":825,"state":"active+clean","last_fresh":"2021-07-31T01:59:04.727587+0000","last_change":"2021-07-31T01:44:11.747832+0000","last_active":"2021-07-31T01:59:04.727587+0000","last_peered":"2021-07-31T01:59:04.727587+0000","last_clean":"2021-07-31T01:59:04.727587+0000","last_became_active":"2021-07-31T01:44:11.747418+0000","last_became_peered":"2021-07-31T01:44:11.747418+0000","last_unstale":"2021-07-31T01:59:04.727587+0000","last_undegraded":"2021-07-31T01:59:04.727587+0000","last_fullsized":"2021-07-31T01:59:04.727587+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-07-31T01:44:10.735599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-07-31T01:44:10.735599+0000","last_clean_scrub_stamp":"2021-07-31T01:44:10.735599+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"157'117","reported_seq":920,"reported_epoch":825,"state":"active+clean","last_fresh":"2021-07-31T01:59:04.730620+0000","last_change":"2021-07-31T01:43:48.139877+0000","last_active":"2021-07-31T01:59:04.730620+0000","last_peered":"2021-07-31T01:59:04.730620+0000","last_clean":"2021-07-31T01:59:04.730620+0000","last_became_active":"2021-07-31T01:43:48.139559+0000","last_became_peered":"2021-07-31T01:43:48.139559+0000","last_unstale":"2021-07-31T01:59:04.730620+0000","last_undegraded":"2021-07-31T01:59:04.730620+0000","last_fullsized":"2021-07-31T01:59:04.730620+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-07-31T01:42:08.446854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-07-31T01:42:08.446854+0000","last_clean_scrub_stamp":"2021-07-31T01:42:08.446854+0000","log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"825'646","reported_seq":3049,"reported_epoch":826,"state":"active+clean","last_fresh":"2021-07-31T01:59:06.827091+0000","last_change":"2021-07-31T01:59:04.732155+0000","last_active":"2021-07-31T01:59:06.827091+0000","last_peered":"2021-07-31T01:59:06.827091+0000","last_clean":"2021-07-31T01:59:06.827091+0000","last_became_active":"2021-07-31T01:45:52.736399+0000","last_became_peered":"2021-07-31T01:45:52.736399+0000","last_unstale":"2021-07-31T01:59:06.827091+0000","last_undegraded":"2021-07-31T01:59:06.827091+0000","last_fullsized":"2021-07-31T01:59:06.827091+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-07-31T01:45:51.724604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-07-31T01:45:51.724604+0000","last_clean_scrub_stamp":"2021-07-31T01:45:51.724604+0000","log_size":646,"ondisk_log_size":646,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":17661952,"num_objects":11,"num_object_clones":9,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":143,"num_read_kb":180080,"num_write":320,"num_write_kb":54816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"281","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":767,"reported_epoch":825,"state":"active+clean","last_fresh":"2021-07-31T01:59:04.730455+0000","last_change":"2021-07-31T01:44:12.036812+0000","last_active":"2021-07-31T01:59:04.730455+0000","last_peered":"2021-07-31T01:59:04.730455+0000","last_clean":"2021-07-31T01:59:04.730455+0000","last_became_active":"2021-07-31T01:44:12.036370+0000","last_became_peered":"2021-07-31T01:44:12.036370+0000","last_unstale":"2021-07-31T01:59:04.730455+0000","last_undegraded":"2021-07-31T01:59:04.730455+0000","last_fullsized":"2021-07-31T01:59:04.730455+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-07-31T01:44:10.735599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-07-31T01:44:10.735599+0000","last_clean_scrub_stamp":"2021-07-31T01:44:10.735599+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"825'794","reported_seq":3220,"reported_epoch":826,"state":"active+clean","last_fresh":"2021-07-31T01:59:06.703065+0000","last_change":"2021-07-31T01:59:04.743627+0000","last_active":"2021-07-31T01:59:06.703065+0000","last_peered":"2021-07-31T01:59:06.703065+0000","last_clean":"2021-07-31T01:59:06.703065+0000","last_became_active":"2021-07-31T01:45:52.740938+0000","last_became_peered":"2021-07-31T01:45:52.740938+0000","last_unstale":"2021-07-31T01:59:06.703065+0000","last_undegraded":"2021-07-31T01:59:06.703065+0000","last_fullsized":"2021-07-31T01:59:06.703065+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-07-31T01:45:51.724604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-07-31T01:45:51.724604+0000","last_clean_scrub_stamp":"2021-07-31T01:45:51.724604+0000","log_size":794,"ondisk_log_size":794,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16416768,"num_objects":17,"num_object_clones":14,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":2,"num_read":220,"num_read_kb":196863,"num_write":393,"num_write_kb":76104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"281","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":767,"reported_epoch":825,"state":"active+clean","last_fresh":"2021-07-31T01:59:04.729860+0000","last_change":"2021-07-31T01:44:12.036816+0000","last_active":"2021-07-31T01:59:04.729860+0000","last_peered":"2021-07-31T01:59:04.729860+0000","last_clean":"2021-07-31T01:59:04.729860+0000","last_became_active":"2021-07-31T01:44:12.036440+0000","last_became_peered":"2021-07-31T01:44:12.036440+0000","last_unstale":"2021-07-31T01:59:04.729860+0000","last_undegraded":"2021-07-31T01:59:04.729860+0000","last_fullsized":"2021-07-31T01:59:04.729860+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-07-31T01:44:10.735599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-07-31T01:44:10.735599+0000","last_clean_scrub_stamp":"2021-07-31T01:44:10.735599+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"825'963","reported_seq":3394,"reported_epoch":826,"state":"active+clean","last_fresh":"2021-07-31T01:59:06.826017+0000","last_change":"2021-07-31T01:59:04.735403+0000","last_active":"2021-07-31T01:59:06.826017+0000","last_peered":"2021-07-31T01:59:06.826017+0000","last_clean":"2021-07-31T01:59:06.826017+0000","last_became_active":"2021-07-31T01:45:52.738779+0000","last_became_peered":"2021-07-31T01:45:52.738779+0000","last_unstale":"2021-07-31T01:59:06.826017+0000","last_undegraded":"2021-07-31T01:59:06.826017+0000","last_fullsized":"2021-07-31T01:59:06.826017+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-07-31T01:45:51.724604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-07-31T01:45:51.724604+0000","last_clean_scrub_stamp":"2021-07-31T01:45:51.724604+0000","log_size":963,"ondisk_log_size":963,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":39854080,"num_objects":23,"num_object_clones":20,"num_object_copies":69,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":1,"num_read":239,"num_read_kb":490123,"num_write":463,"num_write_kb":111856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"281","length":"1"}]},{"pgid":"3.7","version":"0'0","reported_seq":2365,"reported_epoch":826,"state":"active+clean","last_fresh":"2021-07-31T01:59:06.826910+0000","last_change":"2021-07-31T01:59:04.733515+0000","last_active":"2021-07-31T01:59:06.826910+0000","last_peered":"2021-07-31T01:59:06.826910+0000","last_clean":"2021-07-31T01:59:06.826910+0000","last_became_active":"2021-07-31T01:45:52.740971+0000","last_became_peered":"2021-07-31T01:45:52.740971+0000","last_unstale":"2021-07-31T01:59:06.826910+0000","last_undegraded":"2021-07-31T01:59:06.826910+0000","last_fullsized":"2021-07-31T01:59:06.826910+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-07-31T01:45:51.724604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-07-31T01:45:51.724604+0000","last_clean_scrub_stamp":"2021-07-31T01:45:51.724604+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"281","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":766,"reported_epoch":825,"state":"active+clean","last_fresh":"2021-07-31T01:59:04.725858+0000","last_change":"2021-07-31T01:44:11.748087+0000","last_active":"2021-07-31T01:59:04.725858+0000","last_peered":"2021-07-31T01:59:04.725858+0000","last_clean":"2021-07-31T01:59:04.725858+0000","last_became_active":"2021-07-31T01:44:11.747789+0000","last_became_peered":"2021-07-31T01:44:11.747789+0000","last_unstale":"2021-07-31T01:59:04.725858+0000","last_undegraded":"2021-07-31T01:59:04.725858+0000","last_fullsized":"2021-07-31T01:59:04.725858+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-07-31T01:44:10.735599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-07-31T01:44:10.735599+0000","last_clean_scrub_stamp":"2021-07-31T01:44:10.735599+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":761,"reported_epoch":825,"state":"active+clean","last_fresh":"2021-07-31T01:59:04.726319+0000","last_change":"2021-07-31T01:44:11.747859+0000","last_active":"2021-07-31T01:59:04.726319+0000","last_peered":"2021-07-31T01:59:04.726319+0000","last_clean":"2021-07-31T01:59:04.726319+0000","last_became_active":"2021-07-31T01:44:11.747575+0000","last_became_peered":"2021-07-31T01:44:11.747575+0000","last_unstale":"2021-07-31T01:59:04.726319+0000","last_undegraded":"2021-07-31T01:59:04.726319+0000","last_fullsized":"2021-07-31T01:59:04.726319+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-07-31T01:44:10.735599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-07-31T01:44:10.735599+0000","last_clean_scrub_stamp":"2021-07-31T01:44:10.735599+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"825'728","reported_seq":3104,"reported_epoch":826,"state":"active+clean","last_fresh":"2021-07-31T01:59:06.702782+0000","last_change":"2021-07-31T01:59:04.747299+0000","last_active":"2021-07-31T01:59:06.702782+0000","last_peered":"2021-07-31T01:59:06.702782+0000","last_clean":"2021-07-31T01:59:06.702782+0000","last_became_active":"2021-07-31T01:45:52.739416+0000","last_became_peered":"2021-07-31T01:45:52.739416+0000","last_unstale":"2021-07-31T01:59:06.702782+0000","last_undegraded":"2021-07-31T01:59:06.702782+0000","last_fullsized":"2021-07-31T01:59:06.702782+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-07-31T01:45:51.724604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-07-31T01:45:51.724604+0000","last_clean_scrub_stamp":"2021-07-31T01:45:51.724604+0000","log_size":728,"ondisk_log_size":728,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":66027520,"num_objects":17,"num_object_clones":15,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":125,"num_read_kb":181630,"num_write":302,"num_write_kb":116912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"281","length":"1"}]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":760217600,"num_objects":339,"num_object_clones":289,"num_object_copies":1017,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":339,"num_whiteouts":10,"num_read":3886,"num_read_kb":6237325,"num_write":8444,"num_write_kb":1939288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":984821760,"data_stored":1994612736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18130,"ondisk_log_size":18130,"up":48,"acting":48,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":48,"seq":206158430396,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":382612,"kb_used_data":116108,"kb_used_omap":0,"kb_used_meta":266496,"kb_avail":93362540,"statfs":{"total":95995035648,"available":95603240960,"internally_reserved":0,"allocated":118894592,"data_stored":215297909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":272891904},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 31 01:58:14 2021","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.59399999999999997,"15min":0.69399999999999995},"min":{"1min":0.42299999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.79000000000000004,"5min":0.873,"15min":22.087},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.59399999999999997,"15min":0.59899999999999998},"min":{"1min":0.30499999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.80400000000000005,"5min":0.92900000000000005,"15min":2.101},"last":0.35599999999999998}]},{"osd":1,"last update":"Sat Jul 31 01:58:14 2021","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.61899999999999999,"15min":0.60899999999999999},"min":{"1min":0.46200000000000002,"5min":0.27600000000000002,"15min":0.25900000000000001},"max":{"1min":0.77400000000000002,"5min":0.96999999999999997,"15min":2.0579999999999998},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.61499999999999999,"15min":0.60099999999999998},"min":{"1min":0.35399999999999998,"5min":0.29099999999999998,"15min":0.27800000000000002},"max":{"1min":0.81299999999999994,"5min":1.159,"15min":2.427},"last":0.46999999999999997}]},{"osd":2,"last update":"Sat Jul 31 01:58:14 2021","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.61899999999999999,"15min":0.60099999999999998},"min":{"1min":0.39400000000000002,"5min":0.31,"15min":0.31},"max":{"1min":0.79600000000000004,"5min":0.93200000000000005,"15min":1.0920000000000001},"last":0.37},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.626,"15min":0.628},"min":{"1min":0.38900000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.78900000000000003,"5min":0.90200000000000002,"15min":2.5059999999999998},"last":0.495}]},{"osd":3,"last update":"Sat Jul 31 01:58:14 2021","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.626,"15min":0.60899999999999999},"min":{"1min":0.44900000000000001,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.85699999999999998,"5min":0.88100000000000001,"15min":1.0740000000000001},"last":0.434},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.66400000000000003,"15min":0.65100000000000002},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.38500000000000001},"max":{"1min":0.83999999999999997,"5min":0.95999999999999996,"15min":2.4089999999999998},"last":0.60299999999999998}]},{"osd":4,"last update":"Sat Jul 31 01:58:14 2021","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.54800000000000004,"15min":0.51300000000000001},"min":{"1min":0.315,"5min":0.315,"15min":0.28299999999999997},"max":{"1min":0.82599999999999996,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.628},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.57499999999999996,"15min":0.55000000000000004},"min":{"1min":0.34999999999999998,"5min":0.34899999999999998,"15min":0.317},"max":{"1min":0.72399999999999998,"5min":0.85499999999999998,"15min":1.042},"last":0.51700000000000002}]},{"osd":5,"last update":"Sat Jul 31 01:58:14 2021","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.57399999999999995,"15min":0.55900000000000005},"min":{"1min":0.38300000000000001,"5min":0.36199999999999999,"15min":0.313},"max":{"1min":0.80500000000000005,"5min":0.874,"15min":2.9049999999999998},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.61599999999999999,"15min":0.59399999999999997},"min":{"1min":0.373,"5min":0.36099999999999999,"15min":0.30599999999999999},"max":{"1min":0.83399999999999996,"5min":1.1839999999999999,"15min":1.3300000000000001},"last":0.53400000000000003}]},{"osd":6,"last update":"Sat Jul 31 01:58:14 2021","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.63800000000000001,"15min":0.61499999999999999},"min":{"1min":0.42199999999999999,"5min":0.41399999999999998,"15min":0.35499999999999998},"max":{"1min":0.75700000000000001,"5min":1.2070000000000001,"15min":1.3879999999999999},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.66200000000000003,"15min":0.63700000000000001},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.36699999999999999},"max":{"1min":0.85499999999999998,"5min":0.91400000000000003,"15min":1.1359999999999999},"last":0.55100000000000005}]}]},{"osd":6,"up_from":42,"seq":180388626624,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":413448,"kb_used_data":121792,"kb_used_omap":0,"kb_used_meta":291648,"kb_avail":93331704,"statfs":{"total":95995035648,"available":95571664896,"internally_reserved":0,"allocated":124715008,"data_stored":239554421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":298647552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 31 01:58:58 2021","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.57799999999999996,"15min":0.58699999999999997},"min":{"1min":0.317,"5min":0.28799999999999998,"15min":0.28299999999999997},"max":{"1min":0.84399999999999997,"5min":0.86299999999999999,"15min":2.532},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.58999999999999997,"15min":0.59199999999999997},"min":{"1min":0.40899999999999997,"5min":0.32200000000000001,"15min":0.29399999999999998},"max":{"1min":0.85299999999999998,"5min":1.1839999999999999,"15min":2.3690000000000002},"last":0.51100000000000001}]},{"osd":1,"last update":"Sat Jul 31 01:58:58 2021","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.60899999999999999,"15min":0.60599999999999998},"min":{"1min":0.377,"5min":0.35599999999999998,"15min":0.32000000000000001},"max":{"1min":0.873,"5min":1.2629999999999999,"15min":1.4630000000000001},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.623,"5min":0.623,"15min":0.60599999999999998},"min":{"1min":0.441,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.752,"5min":0.92400000000000004,"15min":0.95899999999999996},"last":0.62}]},{"osd":2,"last update":"Sat Jul 31 01:58:58 2021","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.58499999999999996,"15min":0.58799999999999997},"min":{"1min":0.41099999999999998,"5min":0.318,"15min":0.28699999999999998},"max":{"1min":0.80800000000000005,"5min":0.92500000000000004,"15min":1.3140000000000001},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.61899999999999999,"15min":0.61099999999999999},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.749,"5min":1.232,"15min":1.73},"last":0.59899999999999998}]},{"osd":3,"last update":"Sat Jul 31 01:58:58 2021","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.627,"15min":0.628},"min":{"1min":0.40600000000000003,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.90400000000000003,"5min":1.0669999999999999,"15min":5.0549999999999997},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.65200000000000002,"15min":0.63500000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.84899999999999998,"5min":0.95299999999999996,"15min":1.111},"last":0.66300000000000003}]},{"osd":4,"last update":"Sat Jul 31 01:58:58 2021","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.52800000000000002,"15min":0.51000000000000001},"min":{"1min":0.36499999999999999,"5min":0.33200000000000002,"15min":0.253},"max":{"1min":0.67800000000000005,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.57199999999999995,"15min":0.55100000000000005},"min":{"1min":0.36799999999999999,"5min":0.34899999999999998,"15min":0.28499999999999998},"max":{"1min":0.82399999999999995,"5min":0.86499999999999999,"15min":1.012},"last":0.54100000000000004}]},{"osd":5,"last update":"Sat Jul 31 01:58:58 2021","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.57199999999999995,"15min":0.54500000000000004},"min":{"1min":0.42099999999999999,"5min":0.39300000000000002,"15min":0.30099999999999999},"max":{"1min":0.72399999999999998,"5min":1.268,"15min":1.268},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.59999999999999998,"15min":0.57799999999999996},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.309},"max":{"1min":0.82899999999999996,"5min":0.84899999999999998,"15min":1.044},"last":0.64000000000000001}]},{"osd":7,"last update":"Sat Jul 31 01:58:12 2021","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.64100000000000001,"15min":0.61499999999999999},"min":{"1min":0.442,"5min":0.41199999999999998,"15min":0.36399999999999999},"max":{"1min":1.4510000000000001,"5min":1.4510000000000001,"15min":1.4510000000000001},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.64100000000000001,"15min":0.61199999999999999},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.32100000000000001},"max":{"1min":0.98599999999999999,"5min":0.98699999999999999,"15min":1.6910000000000001},"last":0.69699999999999995}]}]},{"osd":1,"up_from":13,"seq":55834575060,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":478612,"kb_used_data":139788,"kb_used_omap":0,"kb_used_meta":338816,"kb_avail":93266540,"statfs":{"total":95995035648,"available":95504936960,"internally_reserved":0,"allocated":143142912,"data_stored":291384661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":346947584},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 31 01:58:15 2021","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.39000000000000001,"15min":0.40999999999999998},"min":{"1min":0.22600000000000001,"5min":0.22,"15min":0.20300000000000001},"max":{"1min":0.63800000000000001,"5min":0.85399999999999998,"15min":4.4569999999999999},"last":0.313},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.45000000000000001,"15min":0.439},"min":{"1min":0.29999999999999999,"5min":0.245,"15min":0.245},"max":{"1min":0.66000000000000003,"5min":1.145,"15min":1.145},"last":0.497}]},{"osd":2,"last update":"Sat Jul 31 01:58:36 2021","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.47199999999999998,"15min":0.45400000000000001},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.22800000000000001},"max":{"1min":0.69699999999999995,"5min":3.6509999999999998,"15min":4.5970000000000004},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.433,"5min":0.46999999999999997,"15min":0.47599999999999998},"min":{"1min":0.24299999999999999,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.61899999999999999,"5min":1.1890000000000001,"15min":4.6680000000000001},"last":0.38700000000000001}]},{"osd":3,"last update":"Sat Jul 31 01:58:11 2021","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48099999999999998,"15min":0.47499999999999998},"min":{"1min":0.27800000000000002,"5min":0.26800000000000002,"15min":0.22700000000000001},"max":{"1min":0.71899999999999997,"5min":1.2869999999999999,"15min":1.2869999999999999},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.47999999999999998,"15min":0.46500000000000002},"min":{"1min":0.309,"5min":0.247,"15min":0.245},"max":{"1min":0.77500000000000002,"5min":3.4409999999999998,"15min":3.4409999999999998},"last":0.40100000000000002}]},{"osd":4,"last update":"Sat Jul 31 01:58:15 2021","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.69899999999999995,"15min":0.67700000000000005},"min":{"1min":0.42899999999999999,"5min":0.38900000000000001,"15min":0.34300000000000003},"max":{"1min":0.90500000000000003,"5min":3.5950000000000002,"15min":4.6859999999999999},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.70899999999999996,"15min":0.67000000000000004},"min":{"1min":0.53000000000000003,"5min":0.36599999999999999,"15min":0.35299999999999998},"max":{"1min":0.97699999999999998,"5min":4.5949999999999998,"15min":4.5949999999999998},"last":0.51400000000000001}]},{"osd":5,"last update":"Sat Jul 31 01:58:36 2021","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.73099999999999998,"15min":0.71399999999999997},"min":{"1min":0.57099999999999995,"5min":0.44400000000000001,"15min":0.41199999999999998},"max":{"1min":1.032,"5min":3.4969999999999999,"15min":4.5309999999999997},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.71199999999999997,"15min":0.69699999999999995},"min":{"1min":0.51300000000000001,"5min":0.44,"15min":0.39600000000000002},"max":{"1min":1.016,"5min":1.3240000000000001,"15min":1.458},"last":0.53100000000000003}]},{"osd":6,"last update":"Sat Jul 31 01:58:55 2021","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.67800000000000005,"15min":0.67300000000000004},"min":{"1min":0.52300000000000002,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.05,"5min":1.4990000000000001,"15min":3.036},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.75,"15min":0.73399999999999999},"min":{"1min":0.55400000000000005,"5min":0.46700000000000003,"15min":0.39500000000000002},"max":{"1min":0.91300000000000003,"5min":3.6240000000000001,"15min":4.6360000000000001},"last":0.61699999999999999}]},{"osd":7,"last update":"Sat Jul 31 01:58:11 2021","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.76800000000000002,"15min":0.75800000000000001},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.44600000000000001},"max":{"1min":1.073,"5min":3.5550000000000002,"15min":8.5920000000000005},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.70599999999999996,"15min":0.69799999999999995},"min":{"1min":0.45200000000000001,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.96499999999999997,"5min":1.9730000000000001,"15min":2.9249999999999998},"last":0.68700000000000006}]}]},{"osd":0,"up_from":8,"seq":34359738583,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":335272,"kb_used_data":112288,"kb_used_omap":0,"kb_used_meta":222976,"kb_avail":93409880,"statfs":{"total":95995035648,"available":95651717120,"internally_reserved":0,"allocated":114982912,"data_stored":267280245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":228327424},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jul 31 01:58:20 2021","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.39300000000000002,"15min":0.40600000000000003},"min":{"1min":0.29099999999999998,"5min":0.191,"15min":0.191},"max":{"1min":0.57599999999999996,"5min":0.78400000000000003,"15min":2.339},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.42599999999999999,"15min":0.437},"min":{"1min":0.26800000000000002,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.68100000000000005,"5min":0.68400000000000005,"15min":2.7749999999999999},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Jul 31 01:58:41 2021","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.39400000000000002,"15min":0.40500000000000003},"min":{"1min":0.28399999999999997,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.53800000000000003,"5min":0.59899999999999998,"15min":2.1659999999999999},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.45800000000000002,"15min":0.47599999999999998},"min":{"1min":0.28100000000000003,"5min":0.249,"15min":0.23499999999999999},"max":{"1min":0.65100000000000002,"5min":0.68200000000000005,"15min":3.3580000000000001},"last":0.45600000000000002}]},{"osd":3,"last update":"Sat Jul 31 01:58:53 2021","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.48499999999999999,"15min":0.49299999999999999},"min":{"1min":0.317,"5min":0.29099999999999998,"15min":0.27800000000000002},"max":{"1min":0.73299999999999998,"5min":0.76100000000000001,"15min":3.246},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.46200000000000002,"15min":0.46999999999999997},"min":{"1min":0.32000000000000001,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.66900000000000004,"5min":1.032,"15min":2.903},"last":0.48499999999999999}]},{"osd":4,"last update":"Sat Jul 31 01:58:20 2021","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66400000000000003,"15min":0.65400000000000003},"min":{"1min":0.499,"5min":0.35499999999999998,"15min":0.33200000000000002},"max":{"1min":0.89600000000000002,"5min":1.1319999999999999,"15min":3.5350000000000001},"last":1.194},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.66200000000000003,"15min":0.66800000000000004},"min":{"1min":0.5,"5min":0.40799999999999997,"15min":0.34599999999999997},"max":{"1min":0.83199999999999996,"5min":1.0820000000000001,"15min":3.2970000000000002},"last":0.59599999999999997}]},{"osd":5,"last update":"Sat Jul 31 01:58:41 2021","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.68700000000000006,"15min":0.69199999999999995},"min":{"1min":0.49199999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1,"5min":1.0089999999999999,"15min":3.3980000000000001},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.74099999999999999,"15min":0.73599999999999999},"min":{"1min":0.56799999999999995,"5min":0.432,"15min":0.40300000000000002},"max":{"1min":1.0549999999999999,"5min":1.0549999999999999,"15min":2.8570000000000002},"last":0.71499999999999997}]},{"osd":6,"last update":"Sat Jul 31 01:58:49 2021","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.70899999999999996,"15min":0.71099999999999997},"min":{"1min":0.49099999999999999,"5min":0.41899999999999998,"15min":0.41099999999999998},"max":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":3.1800000000000002},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.745,"5min":0.73799999999999999,"15min":0.74399999999999999},"min":{"1min":0.52500000000000002,"5min":0.48999999999999999,"15min":0.39400000000000002},"max":{"1min":1.0089999999999999,"5min":1.0309999999999999,"15min":3.028},"last":0.78300000000000003}]},{"osd":7,"last update":"Sat Jul 31 01:58:20 2021","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72499999999999998},"min":{"1min":0.53700000000000003,"5min":0.47299999999999998,"15min":0.44600000000000001},"max":{"1min":0.97399999999999998,"5min":0.97899999999999998,"15min":3.1190000000000002},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.70599999999999996,"15min":0.69999999999999996},"min":{"1min":0.54300000000000004,"5min":0.378,"15min":0.378},"max":{"1min":0.91600000000000004,"5min":1.1020000000000001,"15min":2.968},"last":1.304}]}]},{"osd":2,"up_from":18,"seq":77309411536,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":286816,"kb_used_data":80920,"kb_used_omap":0,"kb_used_meta":205888,"kb_avail":93458336,"statfs":{"total":95995035648,"available":95701336064,"internally_reserved":0,"allocated":82862080,"data_stored":150789461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":210829312},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 31 01:58:45 2021","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.40899999999999997,"15min":0.39900000000000002},"min":{"1min":0.26000000000000001,"5min":0.23200000000000001,"15min":0.20599999999999999},"max":{"1min":0.60599999999999998,"5min":1.2390000000000001,"15min":1.9810000000000001},"last":0.501},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.45200000000000001,"15min":0.436},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":1.0089999999999999,"5min":2.6379999999999999,"15min":2.6379999999999999},"last":0.56399999999999995}]},{"osd":1,"last update":"Sat Jul 31 01:58:45 2021","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52000000000000002,"15min":0.45100000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":2.3460000000000001,"5min":6.8399999999999999,"15min":6.8399999999999999},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.496,"5min":0.47299999999999998,"15min":0.46500000000000002},"min":{"1min":0.251,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":1.169,"5min":2.0219999999999998,"15min":2.0219999999999998},"last":0.39500000000000002}]},{"osd":3,"last update":"Sat Jul 31 01:59:05 2021","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.53700000000000003,"15min":0.496},"min":{"1min":0.32500000000000001,"5min":0.255,"15min":0.255},"max":{"1min":0.85799999999999998,"5min":3.6840000000000002,"15min":3.6840000000000002},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.66900000000000004,"15min":0.55400000000000005},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.69699999999999995,"5min":8.4160000000000004,"15min":8.4160000000000004},"last":0.46700000000000003}]},{"osd":4,"last update":"Sat Jul 31 01:58:10 2021","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.746,"15min":0.68899999999999995},"min":{"1min":0.52600000000000002,"5min":0.375,"15min":0.35099999999999998},"max":{"1min":0.93999999999999995,"5min":6.6539999999999999,"15min":6.6539999999999999},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.89300000000000002,"5min":0.71999999999999997,"15min":0.67900000000000005},"min":{"1min":0.40500000000000003,"5min":0.36699999999999999,"15min":0.33600000000000002},"max":{"1min":3.7309999999999999,"5min":3.7309999999999999,"15min":3.7309999999999999},"last":0.65700000000000003}]},{"osd":5,"last update":"Sat Jul 31 01:58:45 2021","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.70899999999999996,"15min":0.68200000000000005},"min":{"1min":0.42199999999999999,"5min":0.42099999999999999,"15min":0.39800000000000002},"max":{"1min":1.53,"5min":3.786,"15min":3.786},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.68500000000000005,"15min":0.68300000000000005},"min":{"1min":0.46100000000000002,"5min":0.45100000000000001,"15min":0.41299999999999998},"max":{"1min":0.97399999999999998,"5min":1.167,"15min":1.224},"last":0.69099999999999995}]},{"osd":6,"last update":"Sat Jul 31 01:59:05 2021","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.68799999999999994,"15min":0.68200000000000005},"min":{"1min":0.47199999999999998,"5min":0.46200000000000002,"15min":0.37},"max":{"1min":0.86399999999999999,"5min":1.2330000000000001,"15min":2.8239999999999998},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.748,"15min":0.72599999999999998},"min":{"1min":0.48599999999999999,"5min":0.47999999999999998,"15min":0.40500000000000003},"max":{"1min":0.86199999999999999,"5min":3.819,"15min":3.819},"last":0.745}]},{"osd":7,"last update":"Sat Jul 31 01:58:10 2021","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.70999999999999996,"15min":0.71199999999999997},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.47099999999999997},"max":{"1min":0.98199999999999998,"5min":1.264,"15min":1.298},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.69899999999999995,"15min":0.70099999999999996},"min":{"1min":0.46700000000000003,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.046,"5min":1.1060000000000001,"15min":2.8479999999999999},"last":0.78700000000000003}]}]},{"osd":3,"up_from":25,"seq":107374182604,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":563248,"kb_used_data":168872,"kb_used_omap":0,"kb_used_meta":394368,"kb_avail":93181904,"statfs":{"total":95995035648,"available":95418269696,"internally_reserved":0,"allocated":172924928,"data_stored":309468501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":403832832},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 31 01:58:54 2021","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.39600000000000002,"15min":0.41099999999999998},"min":{"1min":0.23999999999999999,"5min":0.182,"15min":0.182},"max":{"1min":0.61299999999999999,"5min":0.90100000000000002,"15min":1.8879999999999999},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.64400000000000002,"15min":0.504},"min":{"1min":0.24099999999999999,"5min":0.22,"15min":0.19800000000000001},"max":{"1min":0.69999999999999996,"5min":21.100999999999999,"15min":21.100999999999999},"last":0.28599999999999998}]},{"osd":1,"last update":"Sat Jul 31 01:58:54 2021","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.46500000000000002,"15min":0.44900000000000001},"min":{"1min":0.318,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.70999999999999996,"5min":1.3149999999999999,"15min":1.3149999999999999},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.44,"15min":0.44600000000000001},"min":{"1min":0.30499999999999999,"5min":0.246,"15min":0.22700000000000001},"max":{"1min":0.60599999999999998,"5min":1.0149999999999999,"15min":3.2280000000000002},"last":0.32900000000000001}]},{"osd":2,"last update":"Sat Jul 31 01:58:54 2021","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.46700000000000003,"15min":0.48099999999999998},"min":{"1min":0.30499999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.67600000000000005,"5min":1.514,"15min":3.1909999999999998},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.48099999999999998,"15min":0.47799999999999998},"min":{"1min":0.30499999999999999,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.64100000000000001,"5min":1.1879999999999999,"15min":1.786},"last":0.30399999999999999}]},{"osd":4,"last update":"Sat Jul 31 01:58:27 2021","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.66400000000000003,"15min":0.65400000000000003},"min":{"1min":0.46100000000000002,"5min":0.379,"15min":0.36699999999999999},"max":{"1min":1.5700000000000001,"5min":1.5700000000000001,"15min":1.5700000000000001},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.68000000000000005,"15min":0.68500000000000005},"min":{"1min":0.499,"5min":0.40799999999999997,"15min":0.40300000000000002},"max":{"1min":0.91800000000000004,"5min":1.111,"15min":1.6739999999999999},"last":0.435}]},{"osd":5,"last update":"Sat Jul 31 01:58:40 2021","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.70199999999999996,"15min":0.69899999999999995},"min":{"1min":0.51400000000000001,"5min":0.42199999999999999,"15min":0.41099999999999998},"max":{"1min":0.95899999999999996,"5min":1.3420000000000001,"15min":1.645},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.68400000000000005,"15min":0.68500000000000005},"min":{"1min":0.49299999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.95799999999999996,"5min":1.6539999999999999,"15min":1.6539999999999999},"last":0.44800000000000001}]},{"osd":6,"last update":"Sat Jul 31 01:58:54 2021","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.67400000000000004,"15min":0.69199999999999995},"min":{"1min":0.45900000000000002,"5min":0.41999999999999998,"15min":0.40300000000000002},"max":{"1min":0.95299999999999996,"5min":1.0229999999999999,"15min":3.0529999999999999},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.72599999999999998,"15min":0.72799999999999998},"min":{"1min":0.41499999999999998,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.90600000000000003,"5min":1.266,"15min":1.587},"last":0.57899999999999996}]},{"osd":7,"last update":"Sat Jul 31 01:58:27 2021","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.75800000000000001,"15min":0.745},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.45400000000000001},"max":{"1min":3.1160000000000001,"5min":3.1160000000000001,"15min":3.1160000000000001},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.72799999999999998,"15min":0.71899999999999997},"min":{"1min":0.52700000000000002,"5min":0.436,"15min":0.436},"max":{"1min":1.6180000000000001,"5min":1.6180000000000001,"15min":1.6180000000000001},"last":0.51400000000000001}]}]},{"osd":4,"up_from":30,"seq":128849019079,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":565156,"kb_used_data":182044,"kb_used_omap":0,"kb_used_meta":383104,"kb_avail":93179996,"statfs":{"total":95995035648,"available":95416315904,"internally_reserved":0,"allocated":186413056,"data_stored":386760021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":392298496},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 31 01:58:24 2021","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59999999999999998,"15min":0.58299999999999996},"min":{"1min":0.44800000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.82799999999999996,"5min":3.7149999999999999,"15min":3.7149999999999999},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.63200000000000001,"15min":0.61699999999999999},"min":{"1min":0.43099999999999999,"5min":0.371,"15min":0.33800000000000002},"max":{"1min":0.754,"5min":2.387,"15min":2.387},"last":0.71999999999999997}]},{"osd":1,"last update":"Sat Jul 31 01:58:24 2021","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.627,"15min":0.61199999999999999},"min":{"1min":0.40400000000000003,"5min":0.40000000000000002,"15min":0.34200000000000003},"max":{"1min":0.876,"5min":2.6720000000000002,"15min":2.6720000000000002},"last":0.622},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.57999999999999996,"15min":0.59299999999999997},"min":{"1min":0.48499999999999999,"5min":0.36599999999999999,"15min":0.29599999999999999},"max":{"1min":0.70299999999999996,"5min":0.86499999999999999,"15min":1.569},"last":0.60799999999999998}]},{"osd":2,"last update":"Sat Jul 31 01:58:24 2021","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.622,"15min":0.61199999999999999},"min":{"1min":0.52500000000000002,"5min":0.39900000000000002,"15min":0.34200000000000003},"max":{"1min":0.77500000000000002,"5min":1.4790000000000001,"15min":1.4790000000000001},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.65700000000000003,"15min":0.62},"min":{"1min":0.56699999999999995,"5min":0.38900000000000001,"15min":0.33100000000000002},"max":{"1min":0.85999999999999999,"5min":2.464,"15min":2.464},"last":0.59199999999999997}]},{"osd":3,"last update":"Sat Jul 31 01:58:24 2021","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.64600000000000002,"15min":0.63},"min":{"1min":0.439,"5min":0.42399999999999999,"15min":0.35099999999999998},"max":{"1min":0.80300000000000005,"5min":2.52,"15min":2.52},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.65800000000000003,"15min":0.63800000000000001},"min":{"1min":0.54600000000000004,"5min":0.39900000000000002,"15min":0.373},"max":{"1min":0.79700000000000004,"5min":2.7229999999999999,"15min":2.7229999999999999},"last":0.57899999999999996}]},{"osd":5,"last update":"Sat Jul 31 01:58:30 2021","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.55300000000000005,"15min":0.51500000000000001},"min":{"1min":0.35799999999999998,"5min":0.33100000000000002,"15min":0.311},"max":{"1min":0.77100000000000002,"5min":2.3119999999999998,"15min":2.3119999999999998},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.57999999999999996,"15min":0.55200000000000005},"min":{"1min":0.40400000000000003,"5min":0.312,"15min":0.312},"max":{"1min":0.76100000000000001,"5min":2.6440000000000001,"15min":2.6440000000000001},"last":0.502}]},{"osd":6,"last update":"Sat Jul 31 01:58:52 2021","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55300000000000005,"15min":0.53900000000000003},"min":{"1min":0.377,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.75800000000000001,"5min":1.8480000000000001,"15min":1.8480000000000001},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.67000000000000004,"15min":0.64500000000000002},"min":{"1min":0.45400000000000001,"5min":0.40799999999999997,"15min":0.38500000000000001},"max":{"1min":0.82899999999999996,"5min":2.3500000000000001,"15min":2.3500000000000001},"last":0.74299999999999999}]},{"osd":7,"last update":"Sat Jul 31 01:58:24 2021","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.63600000000000001,"15min":0.60199999999999998},"min":{"1min":0.436,"5min":0.38800000000000001,"15min":0.371},"max":{"1min":0.82799999999999996,"5min":2.4420000000000002,"15min":2.4420000000000002},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.65900000000000003,"15min":0.621},"min":{"1min":0.38100000000000001,"5min":0.377,"15min":0.36899999999999999},"max":{"1min":0.79400000000000004,"5min":2.7559999999999998,"15min":2.7559999999999998},"last":0.67100000000000004}]}]},{"osd":5,"up_from":36,"seq":154618822851,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":313932,"kb_used_data":94340,"kb_used_omap":0,"kb_used_meta":219584,"kb_avail":93431220,"statfs":{"total":95995035648,"available":95673569280,"internally_reserved":0,"allocated":96604160,"data_stored":166841685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":224854016},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 31 01:58:37 2021","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.53900000000000003,"15min":0.54200000000000004},"min":{"1min":0.39000000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.83299999999999996},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.629,"5min":0.58699999999999997,"15min":0.57899999999999996},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.623}]},{"osd":1,"last update":"Sat Jul 31 01:58:37 2021","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.61399999999999999,"15min":0.60299999999999998},"min":{"1min":0.40899999999999997,"5min":0.316,"15min":0.316},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":1.1890000000000001},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.56399999999999995,"15min":0.56000000000000005},"min":{"1min":0.38200000000000001,"5min":0.33500000000000002,"15min":0.29099999999999998},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":1.1739999999999999},"last":0.63500000000000001}]},{"osd":2,"last update":"Sat Jul 31 01:58:37 2021","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.57199999999999995,"15min":0.57299999999999995},"min":{"1min":0.43099999999999999,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":1.0109999999999999},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.61199999999999999,"15min":0.61099999999999999},"min":{"1min":0.45100000000000001,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.022,"5min":1.022,"15min":1.1180000000000001},"last":0.70599999999999996}]},{"osd":3,"last update":"Sat Jul 31 01:58:37 2021","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.59699999999999998,"15min":0.59099999999999997},"min":{"1min":0.46700000000000003,"5min":0.41499999999999998,"15min":0.317},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.628,"15min":0.61699999999999999},"min":{"1min":0.51300000000000001,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.93500000000000005,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.59299999999999997}]},{"osd":4,"last update":"Sat Jul 31 01:58:37 2021","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.498,"15min":0.48199999999999998},"min":{"1min":0.29399999999999998,"5min":0.27700000000000002,"15min":0.27500000000000002},"max":{"1min":0.66700000000000004,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54400000000000004,"15min":0.52400000000000002},"min":{"1min":0.35699999999999998,"5min":0.30499999999999999,"15min":0.30299999999999999},"max":{"1min":0.79800000000000004,"5min":0.93999999999999995,"15min":1.3240000000000001},"last":0.69699999999999995}]},{"osd":6,"last update":"Sat Jul 31 01:58:58 2021","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.55100000000000005,"15min":0.53900000000000003},"min":{"1min":0.36199999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.626,"15min":0.60799999999999998},"min":{"1min":0.45200000000000001,"5min":0.36799999999999999,"15min":0.34000000000000002},"max":{"1min":0.84899999999999998,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.60899999999999999}]},{"osd":7,"last update":"Sat Jul 31 01:58:04 2021","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.61099999999999999,"15min":0.57799999999999996},"min":{"1min":0.41099999999999998,"5min":0.34399999999999997,"15min":0.33300000000000002},"max":{"1min":0.79800000000000004,"5min":3.0350000000000001,"15min":3.0350000000000001},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.625,"5min":0.624,"15min":0.61399999999999999},"min":{"1min":0.41099999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":1.161},"last":0.65600000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":107646976,"data_stored":262815744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":136400896,"data_stored":287510528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":76120064,"data_stored":146915328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":166182912,"data_stored":305594368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":179671040,"data_stored":382885888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":89862144,"data_stored":162967552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":117379072,"data_stored":235089920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":111558656,"data_stored":210833408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-07-31T01:59:11.050 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_0 2021-07-31T01:59:11.051 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell --fsid f52d2c14-f19f-11eb-8c24-001a4aab830c -- ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2021-07-31T01:59:11.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:11 smithi045 conmon[33655]: cluster 2021-07-31T01:59:09.964523+0000 mgr.x (mgr.24542) 508 : cluster [DBG] pgmap v1031: 25 pgs: 25 active+clean; 726 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 4.0 MiB/s rd, 801 KiB/s wr, 15 op/s 2021-07-31T01:59:11.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:11 smithi045 conmon[39718]: cluster 2021-07-31T01:59:09.964523+0000 mgr.x (mgr.24542) 508 : cluster [DBG] pgmap v1031: 25 pgs: 25 active+clean; 726 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 4.0 MiB/s rd, 801 KiB/s wr, 15 op/s 2021-07-31T01:59:11.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:11 smithi154 conmon[34553]: cluster 2021-07-31T01:59:09.964523+0000 mgr.x (mgr.24542) 508 : cluster [DBG] pgmap v1031: 25 pgs: 25 active+clean; 726 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 4.0 MiB/s rd, 801 KiB/s wr, 15 op/s 2021-07-31T01:59:11.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:11 smithi154 conmon[34553]: 2021-07-31T01:59:11.504 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f52d2c14-f19f-11eb-8c24-001a4aab830c/mon.a/config 2021-07-31T01:59:12.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:12 smithi045 conmon[33655]: audit 2021-07-31T01:59:10.245009+0000 2021-07-31T01:59:12.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:12 smithi045 conmon[33655]: mgr.x (mgr.24542) 509 : audit [DBG] from='client.25039 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-07-31T01:59:12.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:12 smithi045 conmon[33655]: cluster 2021-07-31T01:59:11.704691+0000 mon.a (mon.0) 1501 : cluster [DBG] osdmap e827: 8 total, 8 up, 8 in 2021-07-31T01:59:12.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:12 smithi045 conmon[39718]: audit 2021-07-31T01:59:10.245009+0000 2021-07-31T01:59:12.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:12 smithi045 conmon[39718]: mgr.x (mgr.24542) 509 : audit [DBG] from='client.25039 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-07-31T01:59:12.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:12 smithi045 conmon[39718]: cluster 2021-07-31T01:59:11.704691 2021-07-31T01:59:12.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:12 smithi045 conmon[39718]: +0000 mon.a (mon.0) 1501 : cluster [DBG] 2021-07-31T01:59:12.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:12 smithi045 conmon[39718]: osdmap e827: 8 total, 8 up, 8 in 2021-07-31T01:59:12.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:12 smithi154 conmon[34553]: audit 2021-07-31T01:59:10.245009+0000 mgr.x (mgr.24542) 509 : audit [DBG] from='client.25039 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-07-31T01:59:12.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:12 smithi154 conmon[34553]: cluster 2021-07-31T01:59:11.704691+0000 mon.a (mon.0) 1501 : cluster [DBG] osdmap e827: 8 total, 8 up, 8 in 2021-07-31T01:59:13.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:13 smithi045 conmon[33655]: cluster 2021-07-31T01:59:11 2021-07-31T01:59:13.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:13 smithi045 conmon[33655]: .965183+0000 mgr.x (mgr.24542) 510 : cluster [DBG] pgmap v1033: 25 pgs: 25 active+clean; 726 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.1 MiB/s rd, 314 KiB/s wr, 7 op/s 2021-07-31T01:59:13.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:13 smithi045 conmon[39718]: cluster 2021-07-31T01:59:11.965183 2021-07-31T01:59:13.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:13 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 510 : cluster [DBG] pgmap v1033: 25 pgs: 25 active+clean; 726 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.1 MiB/s rd, 314 KiB/s wr, 7 op/s 2021-07-31T01:59:13.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:13 smithi154 conmon[34553]: cluster 2021-07-31T01:59:11.965183+0000 mgr.x (mgr.24542) 510 : cluster [DBG] pgmap v1033: 25 pgs: 25 active+clean; 726 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.1 MiB/s rd, 314 KiB/s wr, 7 op/s 2021-07-31T01:59:14.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:59:13 smithi045 conmon[61210]: level=error ts=2021-07-31T01:59:13.529Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:59:14.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:59:13 smithi045 conmon[61210]: level=error ts=2021-07-31T01:59:13.529Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:59:14.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:59:13 smithi045 conmon[61210]: level=error ts=2021-07-31T01:59:13.530Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T01:59:14.039 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'unique_pool_0' removed 2021-07-31T01:59:14.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:14 smithi045 conmon[33655]: audit 2021-07-31T01:59:13.370186+0000 mon.a (mon.0) 1502 : audit [INF] from='client.? 172.21.15.45:0/2133041191' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2021-07-31T01:59:14.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:14 smithi045 conmon[39718]: audit 2021-07-31T01:59:13.370186+0000 mon.a (mon.0) 1502 : 2021-07-31T01:59:14.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:14 smithi045 conmon[39718]: audit [INF] from='client.? 172.21.15.45:0/2133041191' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2021-07-31T01:59:14.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:14 smithi154 conmon[34553]: audit 2021-07-31T01:59:13.370186+0000 mon.a (mon.0) 1502 : audit [INF] 2021-07-31T01:59:14.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:14 smithi154 conmon[34553]: from='client.? 172.21.15.45:0/2133041191' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2021-07-31T01:59:14.833 INFO:teuthology.task.full_sequential:In full_sequential, running task print... 2021-07-31T01:59:14.833 INFO:teuthology.task.print:**** done end ec-rados-default.yaml 2021-07-31T01:59:14.834 INFO:teuthology.task.full_sequential:In full_sequential, running task print... 2021-07-31T01:59:14.834 INFO:teuthology.task.print:**** done start rados_api.yaml 2021-07-31T01:59:14.834 INFO:teuthology.task.full_sequential:In full_sequential, running task workunit... 2021-07-31T01:59:14.840 INFO:tasks.workunit:Pulling workunits from ref pacific 2021-07-31T01:59:14.841 INFO:tasks.workunit:Making a separate scratch dir for every client... 2021-07-31T01:59:14.842 DEBUG:teuthology.orchestra.run.smithi045:> stat -- /home/ubuntu/cephtest/mnt.0 2021-07-31T01:59:14.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-07-31T01:59:14.861 INFO:teuthology.orchestra.run.smithi045.stderr:stat: cannot stat '/home/ubuntu/cephtest/mnt.0': No such file or directory 2021-07-31T01:59:14.861 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -- /home/ubuntu/cephtest/mnt.0 2021-07-31T01:59:14.917 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2021-07-31T01:59:14.918 DEBUG:teuthology.orchestra.run.smithi045:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2021-07-31T01:59:14.974 INFO:tasks.workunit:timeout=3h 2021-07-31T01:59:14.975 INFO:tasks.workunit:cleanup=True 2021-07-31T01:59:14.975 DEBUG:teuthology.orchestra.run.smithi045:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone --depth 1 --branch pacific git://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 2021-07-31T01:59:15.033 INFO:tasks.workunit.client.0.smithi045.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2021-07-31T01:59:15.058 INFO:tasks.workunit.client.0.smithi045.stderr:warning: Could not find remote branch pacific to clone. 2021-07-31T01:59:15.059 INFO:tasks.workunit.client.0.smithi045.stderr:fatal: Remote branch pacific not found in upstream origin 2021-07-31T01:59:15.059 DEBUG:teuthology.orchestra.run:got remote process result: 128 2021-07-31T01:59:15.060 INFO:tasks.workunit:failed to check out 'pacific' from git://git.ceph.com/ceph-ci.git; will also try in git://git.ceph.com/ceph.git 2021-07-31T01:59:15.060 DEBUG:teuthology.orchestra.run.smithi045:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone --depth 1 --branch pacific git://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 2021-07-31T01:59:15.117 INFO:tasks.workunit.client.0.smithi045.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2021-07-31T01:59:15.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:15 smithi154 conmon[34553]: cluster 2021-07-31T01:59:13.965671+0000 mgr.x (mgr.24542) 511 : cluster 2021-07-31T01:59:15.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:15 smithi154 conmon[34553]: [DBG] pgmap v1034: 25 pgs: 25 active+clean; 726 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 987 KiB/s rd, 284 KiB/s wr, 6 op/s 2021-07-31T01:59:15.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:15 smithi154 conmon[34553]: audit 2021-07-31T01:59:14.031813+0000 mon.a (mon.0) 1503 : audit [INF] from='client.? 172.21.15.45:0/2133041191' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2021-07-31T01:59:15.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:15 smithi154 conmon[34553]: cluster 2021 2021-07-31T01:59:15.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:15 smithi154 conmon[34553]: -07-31T01:59:14.031918+0000 mon.a (mon.0) 1504 : cluster [DBG] osdmap e828: 8 total, 8 up, 8 in 2021-07-31T01:59:15.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:15 smithi045 conmon[39718]: cluster 2021-07-31T01:59:13.965671+0000 mgr.x ( 2021-07-31T01:59:15.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:15 smithi045 conmon[39718]: mgr.24542) 511 : cluster [DBG] pgmap v1034: 25 pgs: 25 active+clean; 726 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 987 KiB/s rd, 284 KiB/s wr, 6 op/s 2021-07-31T01:59:15.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:15 smithi045 conmon[39718]: audit 2021-07-31T01:59:14.031813+0000 mon.a (mon.0) 1503 : audit [INF] from='client.? 172.21.15.45:0/2133041191' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2021-07-31T01:59:15.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:15 smithi045 conmon[39718]: cluster 2021-07-31T01:59:14.031918+0000 mon.a (mon.0) 1504 : cluster [DBG] osdmap e828: 8 total, 8 up, 8 in 2021-07-31T01:59:15.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:15 smithi045 conmon[33655]: cluster 2021-07-31T01:59:13.965671+0000 mgr.x (mgr. 2021-07-31T01:59:15.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:15 smithi045 conmon[33655]: 24542) 511 : cluster [DBG] pgmap v1034: 25 pgs: 25 active+clean; 726 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 987 KiB/s rd, 284 KiB/s wr, 6 op/s 2021-07-31T01:59:15.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:15 smithi045 conmon[33655]: audit 2021-07-31T01:59:14.031813+0000 mon.a (mon.0) 1503 : audit [INF] from='client.? 172.21.15.45:0/2133041191' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2021-07-31T01:59:15.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:15 smithi045 conmon[33655]: cluster 2021-07-31T01:59:14.031918+0000 mon.a (mon.0) 1504 : cluster [DBG] osdmap e828: 8 total, 8 up, 8 in 2021-07-31T01:59:16.724 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T01:59:17.006 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T01:59:17.006 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 14m ago 14m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T01:59:17.007 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 11m ago 14m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T01:59:17.007 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 11m ago 18m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T01:59:17.008 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 14m ago 20m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T01:59:17.008 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 14m ago 20m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T01:59:17.008 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 11m ago 18m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T01:59:17.009 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 14m ago 18m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T01:59:17.009 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 14m ago 15m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T01:59:17.010 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 11m ago 15m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T01:59:17.010 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 14m ago 17m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T01:59:17.011 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 14m ago 17m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T01:59:17.011 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 14m ago 17m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T01:59:17.012 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 14m ago 16m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T01:59:17.012 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 11m ago 16m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T01:59:17.012 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 11m ago 16m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T01:59:17.013 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 11m ago 15m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T01:59:17.013 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 11m ago 15m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T01:59:17.014 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 11m ago 15m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T01:59:17.351 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T01:59:17.351 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T01:59:17.351 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T01:59:17.352 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T01:59:17.352 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T01:59:17.352 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T01:59:17.352 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T01:59:17.353 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T01:59:17.353 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T01:59:17.353 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T01:59:17.353 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T01:59:17.354 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T01:59:17.354 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T01:59:17.354 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T01:59:17.354 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T01:59:17.354 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T01:59:17.355 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T01:59:17.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:17 smithi154 conmon[34553]: cluster 2021-07-31T01:59:15.966258+0000 mgr.x (mgr.24542) 512 : cluster [DBG] pgmap v1036: 9 pgs: 9 active+clean; 577 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T01:59:17.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:17 smithi045 conmon[33655]: cluster 2021-07-31T01:59:15.966258+0000 mgr.x 2021-07-31T01:59:17.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:17 smithi045 conmon[33655]: (mgr.24542) 512 : cluster [DBG] pgmap v1036: 9 pgs: 9 active+clean; 577 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T01:59:17.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:17 smithi045 conmon[39718]: cluster 2021-07-31T01:59:15.966258+0000 mgr.x 2021-07-31T01:59:17.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:17 smithi045 conmon[39718]: (mgr.24542) 512 : cluster [DBG] pgmap v1036: 9 pgs: 9 active+clean; 577 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T01:59:18.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:18 smithi154 conmon[34553]: audit 2021-07-31T01:59:16.711759+0000 mgr.x (mgr.24542) 513 : audit 2021-07-31T01:59:18.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:18 smithi154 conmon[34553]: [DBG] from='client.25051 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:59:18.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:18 smithi154 conmon[34553]: audit 2021-07-31T01:59:16.997693+0000 mgr.x (mgr.24542) 514 : audit [DBG] from='client.15150 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:59:18.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:18 smithi154 conmon[34553]: audit 2021-07-31T01:59:17.350869+0000 mon.c (mon.1) 36 : audit 2021-07-31T01:59:18.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:18 smithi154 conmon[34553]: [DBG] from='client.? 172.21.15.45:0/4002444917' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T01:59:18.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:18 smithi045 conmon[33655]: audit 2021-07-31T01:59:16.711759+0000 mgr.x (mgr.24542 2021-07-31T01:59:18.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:18 smithi045 conmon[33655]: ) 513 : audit [DBG] from='client.25051 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:59:18.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:18 smithi045 conmon[33655]: audit 2021-07-31T01:59:16.997693+0000 mgr.x (mgr.24542) 514 : audit [DBG] from='client.15150 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:59:18.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:18 smithi045 conmon[33655]: audit 2021-07-31T01:59:17.350869+0000 mon.c (mon.1) 36 2021-07-31T01:59:18.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:18 smithi045 conmon[33655]: : audit [DBG] from='client.? 172.21.15.45:0/4002444917' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T01:59:18.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:18 smithi045 conmon[39718]: audit 2021-07-31T01:59:16.711759+0000 2021-07-31T01:59:18.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:18 smithi045 conmon[39718]: mgr.x (mgr.24542) 513 : audit [DBG] from='client.25051 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:59:18.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:18 smithi045 conmon[39718]: audit 2021-07-31T01:59:18.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:18 smithi045 conmon[39718]: 2021-07-31T01:59:16.997693+0000 mgr.x (mgr.24542) 514 : 2021-07-31T01:59:18.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:18 smithi045 conmon[39718]: audit [DBG] from='client.15150 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:59:18.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:18 smithi045 conmon[39718]: audit 2021-07-31T01:59:18.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:18 smithi045 conmon[39718]: 2021-07-31T01:59 2021-07-31T01:59:18.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:18 smithi045 conmon[39718]: :17.350869+0000 mon.c (mon.1) 36 2021-07-31T01:59:18.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:18 smithi045 conmon[39718]: : audit [DBG] from='client.? 172.21.15.45:0/4002444917' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T01:59:19.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:19 smithi154 conmon[34553]: cluster 2021-07-31T01:59:17.966644+0000 mgr.x (mgr.24542) 515 : cluster [DBG] 2021-07-31T01:59:19.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:19 smithi154 conmon[34553]: pgmap v1037: 9 pgs: 9 active+clean; 577 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T01:59:19.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:19 smithi045 conmon[33655]: cluster 2021-07-31T01:59:17.966644+0000 mgr.x (mgr.24542 2021-07-31T01:59:19.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:19 smithi045 conmon[33655]: ) 515 : cluster [DBG] pgmap v1037: 9 pgs: 9 active+clean; 577 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T01:59:19.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:19 smithi045 conmon[39718]: cluster 2021-07-31T01:59:17.966644+0000 2021-07-31T01:59:19.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:19 smithi045 conmon[39718]: mgr.x (mgr.24542) 515 : cluster [DBG] pgmap v1037: 9 pgs: 9 active+clean; 577 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T01:59:20.227 DEBUG:teuthology.orchestra.run.smithi045:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2021-07-31T01:59:20.293 INFO:tasks.workunit.client.0.smithi045.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2021-07-31T01:59:20.295 INFO:tasks.workunit.client.0.smithi045.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2021-07-31T01:59:20.295 INFO:tasks.workunit.client.0.smithi045.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2021-07-31T01:59:20.989 INFO:tasks.workunit.client.0.smithi045.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2021-07-31T01:59:21.062 INFO:tasks.workunit.client.0.smithi045.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2021-07-31T01:59:21.102 INFO:tasks.workunit.client.0.smithi045.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2021-07-31T01:59:21.103 INFO:tasks.workunit.client.0.smithi045.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2021-07-31T01:59:21.104 INFO:tasks.workunit.client.0.smithi045.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2021-07-31T01:59:21.142 INFO:tasks.workunit.client.0.smithi045.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2021-07-31T01:59:21.145 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2021-07-31T01:59:21.145 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2021-07-31T01:59:21.201 INFO:tasks.workunit:Running workunits matching cls on client.0... 2021-07-31T01:59:21.203 INFO:tasks.workunit:Running workunit cls/test_cls_2pc_queue.sh... 2021-07-31T01:59:21.203 DEBUG:teuthology.orchestra.run.smithi045:workunit test cls/test_cls_2pc_queue.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=pacific TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_2pc_queue.sh 2021-07-31T01:59:21.271 INFO:tasks.workunit.client.0.smithi045.stdout:Running main() from gmock_main.cc 2021-07-31T01:59:21.271 INFO:tasks.workunit.client.0.smithi045.stdout:[==========] Running 22 tests from 1 test suite. 2021-07-31T01:59:21.271 INFO:tasks.workunit.client.0.smithi045.stdout:[----------] Global test environment set-up. 2021-07-31T01:59:21.272 INFO:tasks.workunit.client.0.smithi045.stdout:[----------] 22 tests from TestCls2PCQueue 2021-07-31T01:59:21.272 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestCls2PCQueue.GetCapacity 2021-07-31T01:59:21.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:21 smithi154 conmon[34553]: cluster 2021-07-31T01:59:19.967056+0000 mgr.x (mgr.24542) 516 : 2021-07-31T01:59:21.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:21 smithi154 conmon[34553]: cluster [DBG] pgmap v1038: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:59:21.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:21 smithi045 conmon[39718]: cluster 2021-07-31T01:59:19.967056+0000 mgr.x (mgr.24542) 516 : cluster [DBG] pgmap v1038: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:59:21.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:21 smithi045 conmon[39718]: 2021-07-31T01:59:21.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:21 smithi045 conmon[33655]: cluster 2021-07-31T01:59:19.967056+0000 mgr.x (mgr.24542) 516 2021-07-31T01:59:21.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:21 smithi045 conmon[33655]: : cluster [DBG] pgmap v1038: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:59:23.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:23 smithi154 conmon[34553]: cluster 2021-07-31T01:59:23.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:23 smithi154 conmon[34553]: 2021-07-31T01:59:23.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:23 smithi154 conmon[34553]: 2021 2021-07-31T01:59:23.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:23 smithi154 conmon[34553]: - 2021-07-31T01:59:23.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:23 smithi154 conmon[34553]: 07 2021-07-31T01:59:23.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:23 smithi154 conmon[34553]: - 2021-07-31T01:59:23.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:23 smithi154 conmon[34553]: 31 2021-07-31T01:59:23.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:23 smithi154 conmon[34553]: T 2021-07-31T01:59:23.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:23 smithi154 conmon[34553]: 01 2021-07-31T01:59:23.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:23 smithi154 conmon[34553]: : 2021-07-31T01:59:23.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:23 smithi154 conmon[34553]: 59:21.967596+0000 2021-07-31T01:59:23.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:23 smithi154 conmon[34553]: mgr.x (mgr.24542) 517 : cluster [DBG] pgmap v1039: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:59:23.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:23 smithi154 conmon[34553]: cluster 2021-07-31T01:59:22.047729+0000 mon.a (mon.0) 1505 : cluster [DBG] osdmap e829: 8 total, 8 up, 8 in 2021-07-31T01:59:23.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:23 smithi154 conmon[34553]: audit 2021-07-31T01:59:22.049667+0000 mon.a (mon.0) 2021-07-31T01:59:23.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:23 smithi154 conmon[34553]: 1506 : audit [INF] from='client.? 172.21.15.45:0/1444462574' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T01:59:23.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:23 smithi045 conmon[33655]: cluster 2021-07-31T01:59:21.967596 2021-07-31T01:59:23.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:23 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 517 : cluster [DBG] pgmap v1039: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:59:23.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:23 smithi045 conmon[33655]: cluster 2021-07-31T01:59:22.047729+0000 mon.a (mon.0) 1505 : cluster [DBG] osdmap e829: 8 total, 8 up, 8 in 2021-07-31T01:59:23.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:23 smithi045 conmon[33655]: audit 2021-07-31T01:59:22.049667+0000 mon.a (mon.0) 1506 : audit [INF] from='client.? 172.21.15.45:0/1444462574' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T01:59:23.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:23 smithi045 conmon[39718]: cluster 2021-07-31T01:59:23.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:23 smithi045 conmon[39718]: 2021-07-31T01:59:23.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:23 smithi045 conmon[39718]: 2021 2021-07-31T01:59:23.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:23 smithi045 conmon[39718]: - 2021-07-31T01:59:23.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:23 smithi045 conmon[39718]: 07 2021-07-31T01:59:23.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:23 smithi045 conmon[39718]: - 2021-07-31T01:59:23.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:23 smithi045 conmon[39718]: 31 2021-07-31T01:59:23.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:23 smithi045 conmon[39718]: T 2021-07-31T01:59:23.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:23 smithi045 conmon[39718]: 01 2021-07-31T01:59:23.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:23 smithi045 conmon[39718]: : 2021-07-31T01:59:23.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:23 smithi045 conmon[39718]: 59 2021-07-31T01:59:23.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:23 smithi045 conmon[39718]: : 2021-07-31T01:59:23.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:23 smithi045 conmon[39718]: 21 2021-07-31T01:59:23.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:23 smithi045 conmon[39718]: . 2021-07-31T01:59:23.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:23 smithi045 conmon[39718]: 967596 2021-07-31T01:59:23.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:23 smithi045 conmon[39718]: +0000 2021-07-31T01:59:23.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:23 smithi045 conmon[39718]: 2021-07-31T01:59:23.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:23 smithi045 conmon[39718]: mgr.x 2021-07-31T01:59:23.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:23 smithi045 conmon[39718]: ( 2021-07-31T01:59:23.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:23 smithi045 conmon[39718]: mgr 2021-07-31T01:59:23.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:23 smithi045 conmon[39718]: . 2021-07-31T01:59:23.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:23 smithi045 conmon[39718]: 24542 2021-07-31T01:59:23.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:23 smithi045 conmon[39718]: ) 2021-07-31T01:59:23.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:23 smithi045 conmon[39718]: 517 2021-07-31T01:59:23.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:23 smithi045 conmon[39718]: : 2021-07-31T01:59:23.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:23 smithi045 conmon[39718]: cluster 2021-07-31T01:59:23.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:23 smithi045 conmon[39718]: 2021-07-31T01:59:23.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:23 smithi045 conmon[39718]: [DBG] 2021-07-31T01:59:23.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:23 smithi045 conmon[39718]: 2021-07-31T01:59:23.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:23 smithi045 conmon[39718]: pgmap v1039: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:59:23.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:23 smithi045 conmon[39718]: 2021-07-31T01:59:23.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:23 smithi045 conmon[39718]: cluster 2021-07-31T01:59:23.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:23 smithi045 conmon[39718]: 2021-07-31T01:59:23.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:23 smithi045 conmon[39718]: 2021 2021-07-31T01:59:23.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:23 smithi045 conmon[39718]: - 2021-07-31T01:59:23.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:23 smithi045 conmon[39718]: 07 2021-07-31T01:59:23.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:23 smithi045 conmon[39718]: - 2021-07-31T01:59:23.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:23 smithi045 conmon[39718]: 31 2021-07-31T01:59:23.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:23 smithi045 conmon[39718]: T 2021-07-31T01:59:23.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:23 smithi045 conmon[39718]: 01 2021-07-31T01:59:23.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:23 smithi045 conmon[39718]: : 2021-07-31T01:59:23.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:23 smithi045 conmon[39718]: 59 2021-07-31T01:59:23.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:23 smithi045 conmon[39718]: : 2021-07-31T01:59:23.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:23 smithi045 conmon[39718]: 22 2021-07-31T01:59:23.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:23 smithi045 conmon[39718]: . 2021-07-31T01:59:23.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:23 smithi045 conmon[39718]: 047729 2021-07-31T01:59:23.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:23 smithi045 conmon[39718]: +0000 2021-07-31T01:59:23.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:23 smithi045 conmon[39718]: 2021-07-31T01:59:23.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:23 smithi045 conmon[39718]: mon.a 2021-07-31T01:59:23.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:23 smithi045 conmon[39718]: ( 2021-07-31T01:59:23.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:23 smithi045 conmon[39718]: mon 2021-07-31T01:59:23.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:23 smithi045 conmon[39718]: . 2021-07-31T01:59:23.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:23 smithi045 conmon[39718]: 0 2021-07-31T01:59:23.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:23 smithi045 conmon[39718]: ) 2021-07-31T01:59:23.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:23 smithi045 conmon[39718]: 1505 2021-07-31T01:59:23.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:23 smithi045 conmon[39718]: : 2021-07-31T01:59:23.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:23 smithi045 conmon[39718]: cluster 2021-07-31T01:59:23.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:23 smithi045 conmon[39718]: 2021-07-31T01:59:23.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:23 smithi045 conmon[39718]: [DBG] 2021-07-31T01:59:23.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:23 smithi045 conmon[39718]: 2021-07-31T01:59:23.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:23 smithi045 conmon[39718]: osdmap e829: 8 total, 8 up, 8 in 2021-07-31T01:59:23.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:23 smithi045 conmon[39718]: 2021-07-31T01:59:23.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:23 smithi045 conmon[39718]: audit 2021-07-31T01:59:23.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:23 smithi045 conmon[39718]: 2021-07-31T01:59:23.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:23 smithi045 conmon[39718]: 2021 2021-07-31T01:59:23.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:23 smithi045 conmon[39718]: - 2021-07-31T01:59:23.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:23 smithi045 conmon[39718]: 07 2021-07-31T01:59:23.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:23 smithi045 conmon[39718]: - 2021-07-31T01:59:23.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:23 smithi045 conmon[39718]: 31 2021-07-31T01:59:23.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:23 smithi045 conmon[39718]: T 2021-07-31T01:59:23.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:23 smithi045 conmon[39718]: 01 2021-07-31T01:59:23.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:23 smithi045 conmon[39718]: : 2021-07-31T01:59:23.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:23 smithi045 conmon[39718]: 59 2021-07-31T01:59:23.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:23 smithi045 conmon[39718]: :22.049667+0000 mon.a (mon.0) 1506 : audit [INF] from='client.? 172.21.15.45:0/1444462574' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T01:59:24.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:59:23 smithi045 conmon[61210]: level=error ts=2021-07-31T01:59:23.530Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:59:24.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:59:23 smithi045 conmon[61210]: level=error ts=2021-07-31T01:59:23.530Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:59:24.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:59:23 smithi045 conmon[61210]: level=error ts=2021-07-31T01:59:23.530Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T01:59:24.068 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestCls2PCQueue.GetCapacity (2797 ms) 2021-07-31T01:59:24.069 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestCls2PCQueue.AsyncGetCapacity 2021-07-31T01:59:24.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:24 smithi154 conmon[34553]: audit 2021-07-31T01:59:23.050188+0000 mon.a (mon.0) 1507 : audit [INF] from='client.? 172.21.15.45:0/1444462574' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-1","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T01:59:24.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:24 smithi154 conmon[34553]: cluster 2021-07-31T01:59:23.050443+0000 mon.a (mon.0) 1508 : cluster [DBG] osdmap e830: 8 total, 8 up, 8 in 2021-07-31T01:59:24.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:24 smithi154 conmon[34553]: cluster 2021-07-31T01:59:23.968088+0000 mgr.x (mgr.24542) 518 : cluster [DBG] pgmap v1042: 41 pgs: 41 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 127 B/s rd, 127 B/s wr, 0 op/s 2021-07-31T01:59:24.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:24 smithi045 conmon[33655]: audit 2021-07-31T01:59:23.050188+0000 mon.a (mon.0) 1507 : audit [INF] 2021-07-31T01:59:24.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:24 smithi045 conmon[33655]: from='client.? 172.21.15.45:0/1444462574' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-1","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T01:59:24.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:24 smithi045 conmon[33655]: cluster 2021-07-31T01:59:23.050443+0000 mon.a (mon.0) 1508 : cluster [DBG] osdmap e830: 8 total, 8 up, 8 in 2021-07-31T01:59:24.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:24 smithi045 conmon[33655]: cluster 2021-07-31T01:59:23. 2021-07-31T01:59:24.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:24 smithi045 conmon[33655]: 968088+0000 mgr.x (mgr.24542) 518 : cluster [DBG] pgmap v1042: 41 pgs: 41 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 127 B/s rd, 127 B/s wr, 0 op/s 2021-07-31T01:59:24.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:24 smithi045 conmon[39718]: audit 2021-07-31T01:59:23.050188+0000 mon.a (mon.0 2021-07-31T01:59:24.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:24 smithi045 conmon[39718]: ) 1507 : audit [INF] from='client.? 172.21.15.45:0/1444462574' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-1","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T01:59:24.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:24 smithi045 conmon[39718]: cluster 2021-07-31T01:59:23.050443+0000 mon.a 2021-07-31T01:59:24.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:24 smithi045 conmon[39718]: (mon.0) 1508 : cluster [DBG] osdmap e830: 8 total, 8 up, 8 in 2021-07-31T01:59:24.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:24 smithi045 conmon[39718]: cluster 2021-07-31T01:59:24.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:24 smithi045 conmon[39718]: 2021-07-31T01:59:23.968088+0000 mgr.x 2021-07-31T01:59:24.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:24 smithi045 conmon[39718]: (mgr.24542) 518 : cluster [DBG] pgmap v1042: 41 pgs: 41 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 127 B/s rd, 127 B/s wr, 0 op/s 2021-07-31T01:59:24.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:24 smithi045 conmon[39718]: 2021-07-31T01:59:25.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:25 smithi154 conmon[34553]: cluster 2021-07-31T01:59:24.066481+0000 mon.a (mon.0) 1509 : cluster [DBG] osdmap e831: 8 total, 8 up, 8 in 2021-07-31T01:59:25.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:25 smithi045 conmon[39718]: cluster 2021-07-31T01:59:24.066481+0000 2021-07-31T01:59:25.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:25 smithi045 conmon[39718]: mon.a (mon.0) 1509 : cluster [DBG] osdmap e831: 8 total, 8 up, 8 in 2021-07-31T01:59:25.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:25 smithi045 conmon[33655]: cluster 2021-07-31T01:59:24. 2021-07-31T01:59:25.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:25 smithi045 conmon[33655]: 066481+0000 mon.a (mon.0) 1509 : cluster [DBG] osdmap e831: 8 total, 8 up, 8 in 2021-07-31T01:59:26.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:26 smithi154 conmon[34553]: cluster 2021-07-31T01:59:25.071777+0000 mon.a (mon.0) 1510 : 2021-07-31T01:59:26.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:26 smithi154 conmon[34553]: cluster [DBG] osdmap e832: 8 total, 8 up, 8 in 2021-07-31T01:59:26.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:26 smithi154 conmon[34553]: audit 2021-07-31T01:59:25.072578+0000 mon.a (mon.0) 1511 : audit [INF] from='client.? 172.21.15.45:0/958852783' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T01:59:26.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:26 smithi154 conmon[34553]: cluster 2021-07-31T01:59:25.968616+0000 mgr.x (mgr.24542) 519 2021-07-31T01:59:26.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:26 smithi154 conmon[34553]: : cluster [DBG] pgmap v1045: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:59:26.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:26 smithi045 conmon[39718]: cluster 2021-07-31T01:59:25.071777+0000 mon.a (mon. 2021-07-31T01:59:26.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:26 smithi045 conmon[39718]: 0) 1510 : cluster [DBG] osdmap e832: 8 total, 8 up, 8 in 2021-07-31T01:59:26.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:26 smithi045 conmon[39718]: audit 2021-07-31T01:59:25. 2021-07-31T01:59:26.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:26 smithi045 conmon[39718]: 072578+0000 mon.a (mon.0) 1511 : audit [INF] from='client.? 172.21.15.45:0/958852783' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T01:59:26.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:26 smithi045 conmon[39718]: cluster 2021-07- 2021-07-31T01:59:26.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:26 smithi045 conmon[39718]: 31T01:59:25.968616+0000 mgr.x (mgr. 2021-07-31T01:59:26.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:26 smithi045 conmon[39718]: 24542) 519 : cluster [DBG] pgmap v1045: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:59:26.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:26 smithi045 conmon[33655]: cluster 2021-07-31T01:59:25.071777+0000 mon.a (mon.0) 1510 2021-07-31T01:59:26.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:26 smithi045 conmon[33655]: : cluster [DBG] osdmap e832: 8 total, 8 up, 8 in 2021-07-31T01:59:26.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:26 smithi045 conmon[33655]: audit 2021-07-31T01:59:25.072578+0000 mon.a (mon.0) 1511 : audit [INF] from='client.? 172.21.15.45:0/958852783' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T01:59:26.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:26 smithi045 conmon[33655]: cluster 2021-07-31T01:59:25.968616+0000 mgr.x (mgr. 2021-07-31T01:59:26.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:26 smithi045 conmon[33655]: 24542) 519 : cluster [DBG] pgmap v1045: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:59:26.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:26 smithi045 conmon[33655]: 2021-07-31T01:59:27.087 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestCls2PCQueue.AsyncGetCapacity (3019 ms) 2021-07-31T01:59:27.088 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestCls2PCQueue.Reserve 2021-07-31T01:59:27.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:27 smithi154 conmon[34553]: audit 2021-07-31T01:59:26.080141+0000 mon.a (mon.0) 1512 : audit [INF] 2021-07-31T01:59:27.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:27 smithi154 conmon[34553]: from='client.? 172.21.15.45:0/958852783' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-2","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T01:59:27.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:27 smithi154 conmon[34553]: cluster 2021-07-31T01:59:26.080429+0000 mon.a (mon.0) 1513 : cluster [DBG] osdmap e833: 8 total, 8 up, 8 in 2021-07-31T01:59:27.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:27 smithi045 conmon[33655]: audit 2021-07-31T01:59:26.080141+0000 mon.a (mon.0 2021-07-31T01:59:27.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:27 smithi045 conmon[33655]: ) 1512 : audit [INF] from='client.? 172.21.15.45:0/958852783' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-2","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T01:59:27.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:27 smithi045 conmon[33655]: cluster 2021-07-31T01:59:26.080429+0000 mon.a 2021-07-31T01:59:27.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:27 smithi045 conmon[33655]: (mon.0) 1513 : cluster [DBG] osdmap e833: 8 total, 8 up, 8 in 2021-07-31T01:59:27.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:27 smithi045 conmon[39718]: audit 2021-07-31T01:59:26 2021-07-31T01:59:27.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:27 smithi045 conmon[39718]: .080141+0000 mon.a (mon.0) 1512 : audit [INF] from='client.? 172.21.15.45:0/958852783' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-2","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T01:59:27.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:27 smithi045 conmon[39718]: cluster 2021-07-31T01:59:27.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:27 smithi045 conmon[39718]: 2021-07-31T01 2021-07-31T01:59:27.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:27 smithi045 conmon[39718]: :59:26.080429+0000 mon.a 2021-07-31T01:59:27.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:27 smithi045 conmon[39718]: (mon.0) 1513 : cluster 2021-07-31T01:59:27.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:27 smithi045 conmon[39718]: [DBG] osdmap e833: 8 total, 8 up, 8 in 2021-07-31T01:59:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:28 smithi154 conmon[34553]: cluster 2021-07-31T01:59:27.085562+0000 mon.a (mon 2021-07-31T01:59:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:28 smithi154 conmon[34553]: .0) 1514 : cluster [DBG] osdmap e834: 8 total, 8 up, 8 in 2021-07-31T01:59:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:28 smithi154 conmon[34553]: cluster 2021-07-31T01:59:27.969055+0000 mgr.x (mgr.24542) 520 : cluster [DBG] pgmap v1048: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:59:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:28 smithi154 conmon[34553]: audit 2021-07-31T01:59:27.979998+0000 mon.b (mon 2021-07-31T01:59:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:28 smithi154 conmon[34553]: .2) 122 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:59:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:28 smithi154 conmon[34553]: audit 2021-07-31T01:59:27.980552+0000 mon.a 2021-07-31T01:59:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:28 smithi154 conmon[34553]: (mon.0) 1515 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:59:28.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:28 smithi154 conmon[34553]: audit 2021-07-31T01:59:27.997713 2021-07-31T01:59:28.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:28 smithi154 conmon[34553]: +0000 mon.b (mon.2) 123 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T01:59:28.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:28 smithi154 conmon[34553]: 2021-07-31T01:59:28.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:28 smithi154 conmon[34553]: audit 2021-07- 2021-07-31T01:59:28.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:28 smithi154 conmon[34553]: 31T01:59:27.998345+0000 mon.a (mon.0) 1516 : audit [INF] 2021-07-31T01:59:28.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:28 smithi154 conmon[34553]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T01:59:28.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:28 smithi045 conmon[39718]: cluster 2021-07-31T01:59:27.085562+0000 mon.a (mon.0) 2021-07-31T01:59:28.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:28 smithi045 conmon[39718]: 1514 : cluster [DBG] osdmap e834: 8 total, 8 up, 8 in 2021-07-31T01:59:28.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:28 smithi045 conmon[39718]: cluster 2021-07-31T01:59:27.969055+0000 mgr.x (mgr.24542) 520 : cluster 2021-07-31T01:59:28.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:28 smithi045 conmon[39718]: [DBG] pgmap v1048: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:59:28.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:28 smithi045 conmon[39718]: audit 2021-07-31T01:59:27.979998+0000 2021-07-31T01:59:28.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:28 smithi045 conmon[39718]: mon.b (mon.2) 122 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:59:28.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:28 smithi045 conmon[39718]: audit 2021-07-31T01:59:28.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:28 smithi045 conmon[39718]: 2021-07-31 2021-07-31T01:59:28.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:28 smithi045 conmon[39718]: T01:59:27.980552+0000 mon.a 2021-07-31T01:59:28.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:28 smithi045 conmon[39718]: (mon.0) 1515 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:59:28.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:28 smithi045 conmon[39718]: 2021-07-31T01:59:28.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:28 smithi045 conmon[39718]: audit 2021-07-31T01:59:28.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:28 smithi045 conmon[39718]: 2021-07-31T01:59:27.997713+0000 2021-07-31T01:59:28.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:28 smithi045 conmon[39718]: mon.b (mon.2) 123 : audit 2021-07-31T01:59:28.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:28 smithi045 conmon[39718]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T01:59:28.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:28 smithi045 conmon[39718]: audit 2021-07-31T01:59: 2021-07-31T01:59:28.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:28 smithi045 conmon[39718]: 27.998345+0000 mon.a (mon.0) 2021-07-31T01:59:28.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:28 smithi045 conmon[39718]: 1516 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T01:59:28.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:28 smithi045 conmon[33655]: cluster 2021-07-31T01:59:27.085562+0000 mon.a (mon.0) 1514 : cluster 2021-07-31T01:59:28.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:28 smithi045 conmon[33655]: [DBG] osdmap e834: 8 total, 8 up, 8 in 2021-07-31T01:59:28.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:28 smithi045 conmon[33655]: cluster 2021-07-31T01:59:27.969055+0000 mgr.x (mgr.24542) 520 : cluster [DBG] pgmap v1048: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:59:28.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:28 smithi045 conmon[33655]: audit 2021-07-31T01:59:27.979998+0000 mon.b (mon.2 2021-07-31T01:59:28.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:28 smithi045 conmon[33655]: ) 122 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:59:28.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:28 smithi045 conmon[33655]: audit 2021-07-31T01:59: 2021-07-31T01:59:28.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:28 smithi045 conmon[33655]: 27.980552+0000 mon.a (mon.0) 1515 : audit [INF] 2021-07-31T01:59:28.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:28 smithi045 conmon[33655]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T01:59:28.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:28 smithi045 conmon[33655]: audit 2021-07-31T01:59:28.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:28 smithi045 conmon[33655]: 2021-07-31T01:59:27.997713+0000 2021-07-31T01:59:28.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:28 smithi045 conmon[33655]: mon.b (mon.2) 123 : audit 2021-07-31T01:59:28.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:28 smithi045 conmon[33655]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T01:59:28.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:28 smithi045 conmon[33655]: audit 2021-07-31T01:59:28.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:28 smithi045 conmon[33655]: 2021-07-31T01:59 2021-07-31T01:59:28.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:28 smithi045 conmon[33655]: :27.998345+0000 mon.a (mon. 2021-07-31T01:59:28.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:28 smithi045 conmon[33655]: 0) 1516 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T01:59:29.375 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[34553]: cluster 2021-07-31T01:59:29.375 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[34553]: 2021-07-31T01:59:29.375 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[34553]: 2021 2021-07-31T01:59:29.375 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[34553]: - 2021-07-31T01:59:29.376 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[34553]: 07 2021-07-31T01:59:29.376 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[34553]: - 2021-07-31T01:59:29.376 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[34553]: 31 2021-07-31T01:59:29.376 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[34553]: T01:59:28.092060+0000 mon.a (mon.0) 2021-07-31T01:59:29.377 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[34553]: 1517 : cluster [DBG] osdmap e835: 8 total, 8 up, 8 in 2021-07-31T01:59:29.377 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[34553]: audit 2021-07-31T01:59:29.377 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[34553]: 2021-07-31T01: 2021-07-31T01:59:29.377 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[34553]: 59:28.094369+0000 mon.c (mon.1) 2021-07-31T01:59:29.378 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[34553]: 37 : audit [INF] from='client.? 172.21.15.45:0/1485984041' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T01:59:29.378 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[34553]: audit 2021-07-31T01:59:29.378 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[34553]: 2021-07-31T01 2021-07-31T01:59:29.378 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[34553]: :59:28.095308+0000 mon.a (mon.0) 1518 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T01:59:29.379 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:29 smithi045 conmon[39718]: cluster 2021-07-31T01:59:29.379 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:29 smithi045 conmon[39718]: 2021-07-31T01:59:29.380 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:29 smithi045 conmon[39718]: 2021 2021-07-31T01:59:29.380 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:29 smithi045 conmon[39718]: - 2021-07-31T01:59:29.380 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:29 smithi045 conmon[39718]: 07 2021-07-31T01:59:29.381 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:29 smithi045 conmon[39718]: -31T01:59:28.092060+0000 mon.a 2021-07-31T01:59:29.381 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:29 smithi045 conmon[39718]: (mon.0) 1517 : cluster [DBG] osdmap e835: 8 total, 8 up, 8 in 2021-07-31T01:59:29.381 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:29 smithi045 conmon[39718]: audit 2021-07-31T01:59:29.381 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:29 smithi045 conmon[39718]: 2021-07-31T01:59:28.094369+0000 mon.c (mon.1) 37 : audit 2021-07-31T01:59:29.381 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:29 smithi045 conmon[39718]: [INF] from='client.? 172.21.15.45:0/1485984041' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T01:59:29.382 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:29 smithi045 conmon[39718]: audit 2021-07-31T01: 2021-07-31T01:59:29.382 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:29 smithi045 conmon[39718]: 59:28.095308+0000 mon.a (mon.0) 2021-07-31T01:59:29.382 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:29 smithi045 conmon[39718]: 1518 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T01:59:29.383 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:29 smithi045 conmon[33655]: cluster 2021-07-31T01:59:28.092060+0000 mon.a (mon. 2021-07-31T01:59:29.383 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:29 smithi045 conmon[33655]: 0) 1517 : cluster [DBG] osdmap e835: 8 total, 8 up, 8 in 2021-07-31T01:59:29.383 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:29 smithi045 conmon[33655]: audit 2021-07-31T01:59:28.094369+0000 mon.c (mon.1) 37 : audit [INF] from='client.? 172.21.15.45:0/1485984041' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T01:59:29.383 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:29 smithi045 conmon[33655]: audit 2021-07-31T01:59:28.095308+0000 mon.a (mon.0) 1518 : audit 2021-07-31T01:59:29.383 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:29 smithi045 conmon[33655]: [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T01:59:29.753 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: debug 2021-07-31T01:59:29.393+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T01:59:29.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:59:29.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:59:29.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: yield 2021-07-31T01:59:29.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:59:29.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:59:29.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:59:29.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T01:59:29.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:59:29.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T01:59:29.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:59:29.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T01:59:29.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:59:29.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: task.result() 2021-07-31T01:59:29.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T01:59:29.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: debug 2021-07-31T01:59:29.394+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T01:59:29.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:59:29.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:59:29.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: yield 2021-07-31T01:59:29.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:59:29.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:59:29.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:59:29.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T01:59:29.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:59:29.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T01:59:29.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:59:29.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T01:59:29.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:59:29.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: task.result() 2021-07-31T01:59:29.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T01:59:29.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: debug 2021-07-31T01:59:29.437+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T01:59:29.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:59:29.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:59:29.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: yield 2021-07-31T01:59:29.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:59:29.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:59:29.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:59:29.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T01:59:29.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:59:29.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T01:59:29.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:59:29.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T01:59:29.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:59:29.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: task.result() 2021-07-31T01:59:29.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T01:59:29.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: debug 2021-07-31T01:59:29.438+0000 7fcb9215e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:59:29.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:59:29.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:59:29.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: yield 2021-07-31T01:59:29.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:59:29.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:59:29.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:59:29.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T01:59:29.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:59:29.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T01:59:29.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:59:29.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T01:59:29.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:59:29.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: task.result() 2021-07-31T01:59:29.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T01:59:29.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: 2021-07-31T01:59:29.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T01:59:29.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: 2021-07-31T01:59:29.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:59:29.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:59:29.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: return f(*arg) 2021-07-31T01:59:29.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:59:29.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:59:29.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:59:29.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:59:29.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:59:29.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:59:29.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:59:29.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T01:59:29.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:59:29.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: raise self._exception 2021-07-31T01:59:29.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:59:29.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:59:29.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:59:29.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T01:59:29.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:59:29.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: raise 2021-07-31T01:59:29.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:59:29.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T01:59:29.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:59:29.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:59:29.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T01:59:29.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: debug 2021-07-31T01:59:29.439+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T01:59:29.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:59:29.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:59:29.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: yield 2021-07-31T01:59:29.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:59:29.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:59:29.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:59:29.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T01:59:29.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:59:29.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T01:59:29.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:59:29.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T01:59:29.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:59:29.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: task.result() 2021-07-31T01:59:29.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T01:59:29.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: debug 2021-07-31T01:59:29.439+0000 7fcb9195d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:59:29.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:59:29.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:59:29.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: yield 2021-07-31T01:59:29.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:59:29.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:59:29.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:59:29.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T01:59:29.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:59:29.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T01:59:29.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:59:29.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T01:59:29.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:59:29.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: task.result() 2021-07-31T01:59:29.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T01:59:29.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: 2021-07-31T01:59:29.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T01:59:29.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: 2021-07-31T01:59:29.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T01:59:29.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:59:29.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: return f(*arg) 2021-07-31T01:59:29.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:59:29.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:59:29.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:59:29.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:59:29.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:59:29.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:59:29.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:59:29.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T01:59:29.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:59:29.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: raise self._exception 2021-07-31T01:59:29.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:59:29.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:59:29.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:59:29.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T01:59:29.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:59:29.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: raise 2021-07-31T01:59:29.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:59:29.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T01:59:29.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:59:29.786 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:59:29.786 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 01:59:29 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T01:59:30.112 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestCls2PCQueue.Reserve (3025 ms) 2021-07-31T01:59:30.113 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestCls2PCQueue.AsyncReserve 2021-07-31T01:59:30.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: audit 2021-07-31T01:59:29.093382+0000 mon.a (mon.0) 1519 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-3","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T01:59:30.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: 2021-07-31T01:59:30.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: cluster 2021-07-31T01:59:29.093631+0000 mon.a (mon.0) 1520 : cluster [DBG] osdmap e836: 8 total, 8 up, 8 in 2021-07-31T01:59:30.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: audit 2021-07-31T01:59:30.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: 2021-07-31T01:59:29.347957+0000 mon.b (mon.2) 124 : audit [DBG] 2021-07-31T01:59:30.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:59:30.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: audit 2021-07-31 2021-07-31T01:59:30.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: T01:59:29.349507+0000 mon.b (mon.2) 125 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:59:30.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: audit 2021-07-31T01:59:29.350378+0000 mon.b (mon. 2021-07-31T01:59:30.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: 2) 126 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:59:30.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: cephadm 2021-07-31T01:59:29.351122+0000 mgr.x (mgr.24542 2021-07-31T01:59:30.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: ) 521 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T01:59:30.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: cephadm 2021-07- 2021-07-31T01:59:30.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: 31T01:59:29.394739+0000 mgr.x (mgr.24542) 2021-07-31T01:59:30.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: 522 : cephadm [ERR] 2021-07-31T01:59:30.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:59:30.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:59:30.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: yield 2021-07-31T01:59:30.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:59:30.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:59:30.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:59:30.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T01:59:30.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:59:30.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T01:59:30.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:59:30.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T01:59:30.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:59:30.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: task.result() 2021-07-31T01:59:30.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T01:59:30.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: cephadm 2021-07-31T01:59:29.395943+0000 mgr.x (mgr.24542) 523 : cephadm [ERR] 2021-07-31T01:59:30.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:59:30.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:59:30.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: yield 2021-07-31T01:59:30.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:59:30.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:59:30.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:59:30.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T01:59:30.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:59:30.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T01:59:30.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:59:30.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T01:59:30.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:59:30.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: task.result() 2021-07-31T01:59:30.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T01:59:30.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: cephadm 2021-07-31T01:59:29.396474+0000 mgr.x (mgr.24542 2021-07-31T01:59:30.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: ) 524 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T01:59:30.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: cephadm 2021-07-31T01:59:29.396783+0000 mgr.x (mgr.24542) 525 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T01:59:30.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: cephadm 2021-07-31T01:59 2021-07-31T01:59:30.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: :29.438405+0000 mgr.x (mgr.24542) 526 : cephadm [ERR] 2021-07-31T01:59:30.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:59:30.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:59:30.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: yield 2021-07-31T01:59:30.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:59:30.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:59:30.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:59:30.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T01:59:30.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:59:30.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T01:59:30.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:59:30.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T01:59:30.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:59:30.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: task.result() 2021-07-31T01:59:30.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T01:59:30.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: cephadm 2021-07-31T01:59:29.439661+0000 mgr.x (mgr.24542) 527 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:59:30.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:59:30.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:59:30.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: yield 2021-07-31T01:59:30.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:59:30.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:59:30.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:59:30.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T01:59:30.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:59:30.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T01:59:30.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:59:30.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T01:59:30.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:59:30.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: task.result() 2021-07-31T01:59:30.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T01:59:30.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: 2021-07-31T01:59:30.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T01:59:30.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: 2021-07-31T01:59:30.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:59:30.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:59:30.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: return f(*arg) 2021-07-31T01:59:30.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:59:30.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:59:30.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:59:30.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:59:30.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:59:30.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:59:30.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:59:30.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T01:59:30.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:59:30.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: raise self._exception 2021-07-31T01:59:30.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:59:30.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:59:30.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:59:30.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T01:59:30.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:59:30.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: raise 2021-07-31T01:59:30.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:59:30.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T01:59:30.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:59:30.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:59:30.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T01:59:30.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: cephadm 2021-07-31T01:59:29.440391+0000 mgr.x (mgr.24542) 528 : cephadm [ERR] 2021-07-31T01:59:30.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:59:30.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:59:30.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: yield 2021-07-31T01:59:30.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:59:30.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:59:30.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:59:30.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T01:59:30.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:59:30.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T01:59:30.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:59:30.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T01:59:30.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:59:30.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: task.result() 2021-07-31T01:59:30.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T01:59:30.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: cephadm 2021-07-31T01:59:29.440778+0000 mgr.x (mgr.24542) 529 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:59:30.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:59:30.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:59:30.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: yield 2021-07-31T01:59:30.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:59:30.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:59:30.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:59:30.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T01:59:30.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:59:30.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T01:59:30.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:59:30.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T01:59:30.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:59:30.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: task.result() 2021-07-31T01:59:30.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T01:59:30.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: 2021-07-31T01:59:30.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T01:59:30.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: 2021-07-31T01:59:30.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T01:59:30.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:59:30.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: return f(*arg) 2021-07-31T01:59:30.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:59:30.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:59:30.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:59:30.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:59:30.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:59:30.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:59:30.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:59:30.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T01:59:30.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:59:30.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: raise self._exception 2021-07-31T01:59:30.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:59:30.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:59:30.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:59:30.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T01:59:30.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:59:30.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: raise 2021-07-31T01:59:30.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:59:30.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T01:59:30.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:59:30.541 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:59:30.541 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T01:59:30.541 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:30 smithi154 conmon[34553]: cluster 2021-07-31T01:59:29.969606+0000 mgr.x (mgr.24542) 530 : cluster [DBG] pgmap v1051: 41 pgs: 32 creating+peering, 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:59:30.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: audit 2021-07-31T01:59:30.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: 2021-07-31T01:59:30.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: 2021-07-31T01:59:29.093382+0000 2021-07-31T01:59:30.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: mon.a (mon.0) 1519 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-3","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T01:59:30.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: cluster 2021 2021-07-31T01:59:30.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: -07-31T01:59:29.093631+0000 2021-07-31T01:59:30.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: mon.a (mon.0) 1520 : cluster [DBG] osdmap e836: 8 total, 8 up, 8 in 2021-07-31T01:59:30.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: audit 2021-07-31T01:59:30.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: 2021-07-31T01:59:30.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: 2021 2021-07-31T01:59:30.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: - 2021-07-31T01:59:30.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: 07 2021-07-31T01:59:30.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: - 2021-07-31T01:59:30.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: 31 2021-07-31T01:59:30.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: T 2021-07-31T01:59:30.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: 01 2021-07-31T01:59:30.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: : 2021-07-31T01:59:30.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: 59 2021-07-31T01:59:30.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: : 2021-07-31T01:59:30.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: 29 2021-07-31T01:59:30.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: . 2021-07-31T01:59:30.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: 347957 2021-07-31T01:59:30.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: +0000 2021-07-31T01:59:30.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: 2021-07-31T01:59:30.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: mon.b 2021-07-31T01:59:30.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: ( 2021-07-31T01:59:30.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: mon 2021-07-31T01:59:30.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: . 2021-07-31T01:59:30.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: 2 2021-07-31T01:59:30.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: ) 2021-07-31T01:59:30.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: 124 2021-07-31T01:59:30.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: : 2021-07-31T01:59:30.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: audit 2021-07-31T01:59:30.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: 2021-07-31T01:59:30.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: [DBG] 2021-07-31T01:59:30.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: 2021-07-31T01:59:30.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:59:30.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: 2021-07-31T01:59:30.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: audit 2021-07-31T01:59:30.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: 2021-07-31T01:59:30.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: 2021 2021-07-31T01:59:30.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: - 2021-07-31T01:59:30.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: 07 2021-07-31T01:59:30.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: - 2021-07-31T01:59:30.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: 31 2021-07-31T01:59:30.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: T 2021-07-31T01:59:30.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: 01 2021-07-31T01:59:30.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: : 2021-07-31T01:59:30.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: 59 2021-07-31T01:59:30.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: : 2021-07-31T01:59:30.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: 29 2021-07-31T01:59:30.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: . 2021-07-31T01:59:30.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: 349507 2021-07-31T01:59:30.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: +0000 2021-07-31T01:59:30.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: 2021-07-31T01:59:30.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: mon.b 2021-07-31T01:59:30.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: ( 2021-07-31T01:59:30.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: mon 2021-07-31T01:59:30.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: . 2021-07-31T01:59:30.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: 2 2021-07-31T01:59:30.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: ) 2021-07-31T01:59:30.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: 125 2021-07-31T01:59:30.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: : 2021-07-31T01:59:30.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: audit 2021-07-31T01:59:30.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: 2021-07-31T01:59:30.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: [DBG] 2021-07-31T01:59:30.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: 2021-07-31T01:59:30.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:59:30.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: 2021-07-31T01:59:30.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: audit 2021-07-31T01:59:30.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: 2021-07-31T01:59:30.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: 2021 2021-07-31T01:59:30.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: - 2021-07-31T01:59:30.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: 07 2021-07-31T01:59:30.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: - 2021-07-31T01:59:30.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: 31 2021-07-31T01:59:30.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: T 2021-07-31T01:59:30.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: 01 2021-07-31T01:59:30.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: : 2021-07-31T01:59:30.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: 59:29.350378+0000 mon.b (mon.2) 2021-07-31T01:59:30.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: 126 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:59:30.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: cephadm 2021-07 2021-07-31T01:59:30.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: -31T01:59:29.351122+0000 mgr.x (mgr.24542) 521 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T01:59:30.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: cephadm 2021-07-31T01:59:30.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: 2021-07-31T01:59:29.394739+0000 mgr.x ( 2021-07-31T01:59:30.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: mgr.24542) 522 : cephadm [ERR] 2021-07-31T01:59:30.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:59:30.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:59:30.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: yield 2021-07-31T01:59:30.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:59:30.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:59:30.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:59:30.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T01:59:30.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:59:30.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T01:59:30.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:59:30.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T01:59:30.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:59:30.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: task.result() 2021-07-31T01:59:30.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T01:59:30.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: cephadm 2021-07-31T01:59:29.395943+0000 mgr.x (mgr.24542) 523 : cephadm [ERR] 2021-07-31T01:59:30.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:59:30.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:59:30.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: yield 2021-07-31T01:59:30.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:59:30.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:59:30.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:59:30.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T01:59:30.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:59:30.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T01:59:30.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:59:30.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T01:59:30.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:59:30.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: task.result() 2021-07-31T01:59:30.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T01:59:30.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: cephadm 2021-07-31T01:59:29.396474+0000 2021-07-31T01:59:30.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: mgr.x (mgr.24542) 524 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T01:59:30.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: cephadm 2021-07-31T01:59:29.396783+0000 mgr.x (mgr.24542) 525 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T01:59:30.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: cephadm 2021-07- 2021-07-31T01:59:30.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: 31T01:59:29.438405+0000 mgr.x (mgr.24542) 526 : cephadm [ERR] 2021-07-31T01:59:30.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:59:30.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:59:30.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: yield 2021-07-31T01:59:30.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:59:30.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:59:30.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:59:30.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T01:59:30.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:59:30.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T01:59:30.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:59:30.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T01:59:30.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:59:30.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: task.result() 2021-07-31T01:59:30.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T01:59:30.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: cephadm 2021-07-31T01:59:29.439661+0000 mgr.x (mgr.24542) 527 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:59:30.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:59:30.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:59:30.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: yield 2021-07-31T01:59:30.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:59:30.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:59:30.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:59:30.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T01:59:30.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:59:30.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T01:59:30.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:59:30.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T01:59:30.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:59:30.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: task.result() 2021-07-31T01:59:30.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T01:59:30.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: 2021-07-31T01:59:30.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T01:59:30.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: 2021-07-31T01:59:30.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:59:30.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:59:30.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: return f(*arg) 2021-07-31T01:59:30.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:59:30.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:59:30.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:59:30.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:59:30.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:59:30.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:59:30.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:59:30.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T01:59:30.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:59:30.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: raise self._exception 2021-07-31T01:59:30.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:59:30.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:59:30.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:59:30.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T01:59:30.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:59:30.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: raise 2021-07-31T01:59:30.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:59:30.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T01:59:30.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:59:30.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:59:30.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T01:59:30.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: cephadm 2021-07-31T01:59:29. 2021-07-31T01:59:30.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: 440391+0000 mgr.x (mgr.24542) 528 : cephadm [ERR] 2021-07-31T01:59:30.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:59:30.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:59:30.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: yield 2021-07-31T01:59:30.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:59:30.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:59:30.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:59:30.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T01:59:30.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:59:30.585 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T01:59:30.585 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:59:30.585 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T01:59:30.585 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:59:30.586 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: task.result() 2021-07-31T01:59:30.586 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T01:59:30.586 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: cephadm 2021-07-31T01:59:29.440778+0000 mgr.x (mgr.24542) 529 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:59:30.586 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:59:30.586 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:59:30.587 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: yield 2021-07-31T01:59:30.587 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:59:30.587 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:59:30.587 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:59:30.587 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T01:59:30.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:59:30.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T01:59:30.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:59:30.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T01:59:30.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:59:30.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: task.result() 2021-07-31T01:59:30.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T01:59:30.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: 2021-07-31T01:59:30.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T01:59:30.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: 2021-07-31T01:59:30.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T01:59:30.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:59:30.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: return f(*arg) 2021-07-31T01:59:30.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:59:30.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:59:30.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:59:30.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:59:30.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:59:30.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:59:30.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:59:30.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T01:59:30.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:59:30.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: raise self._exception 2021-07-31T01:59:30.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:59:30.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:59:30.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:59:30.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T01:59:30.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:59:30.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: raise 2021-07-31T01:59:30.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:59:30.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T01:59:30.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:59:30.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:59:30.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T01:59:30.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[39718]: cluster 2021-07-31T01:59:29.969606+0000 mgr.x (mgr.24542) 530 : cluster [DBG] pgmap v1051: 41 pgs: 32 creating+peering, 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:59:30.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: audit 2021-07-31T01:59:29.093382 2021-07-31T01:59:30.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: +0000 mon.a (mon.0) 1519 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-3","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T01:59:30.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: cluster 2021-07-31T01:59:29.093631+0000 mon.a (mon.0) 1520 : cluster [DBG] osdmap e836: 8 total, 8 up, 8 in 2021-07-31T01:59:30.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: audit 2021-07-31T01:59:29.347957+0000 mon.b (mon.2) 124 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T01:59:30.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: audit 2021-07-31T01 2021-07-31T01:59:30.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: :59:29.349507+0000 mon.b (mon.2) 125 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T01:59:30.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: audit 2021-07-31T01:59:30.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: 2021-07-31T01:59:29.350378+0000 mon.b (mon. 2021-07-31T01:59:30.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: 2) 126 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T01:59:30.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: cephadm 2021-07-31T 2021-07-31T01:59:30.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: 01:59:29.351122+0000 mgr.x (mgr.24542) 521 : 2021-07-31T01:59:30.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T01:59:30.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: cephadm 2021-07 2021-07-31T01:59:30.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: -31T01:59:29.394739+0000 mgr.x (mgr. 2021-07-31T01:59:30.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: 24542) 522 : cephadm [ERR] 2021-07-31T01:59:30.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:59:30.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:59:30.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: yield 2021-07-31T01:59:30.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:59:30.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:59:30.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:59:30.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T01:59:30.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:59:30.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T01:59:30.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:59:30.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T01:59:30.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:59:30.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: task.result() 2021-07-31T01:59:30.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T01:59:30.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: cephadm 2021-07-31T01:59:29.395943+0000 mgr.x (mgr.24542) 523 : cephadm [ERR] 2021-07-31T01:59:30.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:59:30.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:59:30.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: yield 2021-07-31T01:59:30.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:59:30.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:59:30.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:59:30.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T01:59:30.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:59:30.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T01:59:30.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:59:30.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T01:59:30.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:59:30.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: task.result() 2021-07-31T01:59:30.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T01:59:30.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: cephadm 2021-07-31T01:59:30.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: 2021-07-31T01:59:29.396474+0000 mgr.x (mgr.24542) 524 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T01:59:30.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: cephadm 2021-07-31T01:59:29.396783+0000 mgr.x (mgr.24542) 525 2021-07-31T01:59:30.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T01:59:30.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: cephadm 2021-07-31 2021-07-31T01:59:30.608 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: T01:59:29.438405+0000 mgr.x (mgr.24542 2021-07-31T01:59:30.608 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: ) 526 : cephadm [ERR] 2021-07-31T01:59:30.608 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:59:30.608 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:59:30.609 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: yield 2021-07-31T01:59:30.609 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:59:30.609 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:59:30.609 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:59:30.609 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T01:59:30.610 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:59:30.610 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T01:59:30.610 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:59:30.610 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T01:59:30.611 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:59:30.611 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: task.result() 2021-07-31T01:59:30.611 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T01:59:30.611 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: cephadm 2021-07-31T01:59:29.439661+0000 mgr.x (mgr.24542) 2021-07-31T01:59:30.611 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: 527 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:59:30.612 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:59:30.612 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:59:30.612 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: yield 2021-07-31T01:59:30.612 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:59:30.613 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:59:30.613 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:59:30.613 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T01:59:30.613 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:59:30.613 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T01:59:30.614 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:59:30.614 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T01:59:30.614 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:59:30.614 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: task.result() 2021-07-31T01:59:30.614 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T01:59:30.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: 2021-07-31T01:59:30.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T01:59:30.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: 2021-07-31T01:59:30.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:59:30.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:59:30.616 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: return f(*arg) 2021-07-31T01:59:30.616 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:59:30.616 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:59:30.616 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:59:30.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:59:30.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:59:30.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:59:30.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:59:30.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T01:59:30.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:59:30.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: raise self._exception 2021-07-31T01:59:30.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:59:30.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:59:30.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:59:30.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T01:59:30.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:59:30.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: raise 2021-07-31T01:59:30.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:59:30.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T01:59:30.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:59:30.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:59:30.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T01:59:30.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: cephadm 2021-07-31T01:59:29.440391+0000 mgr.x (mgr.24542) 528 : cephadm [ERR] 2021-07-31T01:59:30.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:59:30.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:59:30.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: yield 2021-07-31T01:59:30.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:59:30.622 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:59:30.622 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:59:30.622 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T01:59:30.622 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:59:30.622 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T01:59:30.623 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:59:30.623 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T01:59:30.623 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:59:30.623 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: task.result() 2021-07-31T01:59:30.623 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T01:59:30.624 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: cephadm 2021-07-31T01:59:29.440778+0000 mgr.x (mgr.24542) 529 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T01:59:30.624 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:59:30.624 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T01:59:30.624 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: yield 2021-07-31T01:59:30.625 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T01:59:30.625 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T01:59:30.625 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T01:59:30.625 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T01:59:30.625 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T01:59:30.626 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T01:59:30.626 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T01:59:30.626 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T01:59:30.626 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T01:59:30.626 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: task.result() 2021-07-31T01:59:30.627 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T01:59:30.627 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: 2021-07-31T01:59:30.627 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T01:59:30.627 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: 2021-07-31T01:59:30.627 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T01:59:30.628 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T01:59:30.628 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: return f(*arg) 2021-07-31T01:59:30.628 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T01:59:30.628 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T01:59:30.629 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T01:59:30.629 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T01:59:30.629 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T01:59:30.629 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T01:59:30.629 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T01:59:30.630 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T01:59:30.630 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T01:59:30.630 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: raise self._exception 2021-07-31T01:59:30.630 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T01:59:30.631 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T01:59:30.631 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T01:59:30.631 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T01:59:30.631 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T01:59:30.631 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: raise 2021-07-31T01:59:30.632 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T01:59:30.632 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T01:59:30.632 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T01:59:30.632 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T01:59:30.633 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T01:59:30.633 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: cluster 2021-07-31T01:59:29.969606+0000 mgr.x (mgr.24542) 2021-07-31T01:59:30.633 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:30 smithi045 conmon[33655]: 530 : cluster [DBG] pgmap v1051: 41 pgs: 32 creating+peering, 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:59:31.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:31 smithi154 conmon[34553]: cluster 2021-07-31T01:59:30.108194+0000 mon.a (mon.0) 1521 : cluster [DBG] osdmap e837: 8 total, 8 up, 8 in 2021-07-31T01:59:31.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:31 smithi045 conmon[33655]: cluster 2021-07-31T01:59:30.108194+0000 mon.a 2021-07-31T01:59:31.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:31 smithi045 conmon[33655]: (mon.0) 1521 : cluster [DBG] osdmap e837: 8 total, 8 up, 8 in 2021-07-31T01:59:31.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:31 smithi045 conmon[39718]: cluster 2021-07-31T01:59:30.108194 2021-07-31T01:59:31.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:31 smithi045 conmon[39718]: +0000 mon.a (mon.0) 1521 : cluster [DBG] osdmap e837: 8 total, 8 up, 8 in 2021-07-31T01:59:32.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:32 smithi154 conmon[34553]: cluster 2021-07-31T01:59:31.112362+0000 mon.a (mon.0) 1522 : cluster [DBG] osdmap e838: 8 total, 8 up, 8 in 2021-07-31T01:59:32.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:32 smithi154 conmon[34553]: 2021-07-31T01:59:32.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:32 smithi154 conmon[34553]: audit 2021-07-31T01:59:31.116032+0000 mon.a (mon.0) 1523 : audit [INF] from='client.? 172.21.15.45:0/696419908' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T01:59:32.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:32 smithi154 conmon[34553]: cluster 2021-07-31T01:59:31.970062+0000 2021-07-31T01:59:32.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:32 smithi154 conmon[34553]: mgr.x (mgr.24542) 531 : cluster [DBG] pgmap v1054: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:59:32.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:32 smithi045 conmon[39718]: cluster 2021-07-31T01:59:31.112362+0000 mon.a (mon.0) 1522 : cluster [DBG] osdmap e838: 8 total, 8 up, 8 in 2021-07-31T01:59:32.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:32 smithi045 conmon[39718]: audit 2021-07-31T01:59:31.116032+0000 mon.a (mon.0) 1523 : audit [INF] from='client.? 172.21.15.45:0/696419908' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T01:59:32.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:32 smithi045 conmon[39718]: cluster 2021-07 2021-07-31T01:59:32.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:32 smithi045 conmon[39718]: -31T01:59:31.970062+0000 mgr.x (mgr.24542) 531 : 2021-07-31T01:59:32.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:32 smithi045 conmon[39718]: cluster [DBG] pgmap v1054: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:59:32.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:32 smithi045 conmon[33655]: cluster 2021-07-31T01:59 2021-07-31T01:59:32.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:32 smithi045 conmon[33655]: :31.112362+0000 mon.a (mon.0) 1522 : cluster [DBG] osdmap e838: 8 total, 8 up, 8 in 2021-07-31T01:59:32.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:32 smithi045 conmon[33655]: audit 2021-07-31T01:59:31.116032+0000 mon.a (mon.0) 1523 : audit [INF] 2021-07-31T01:59:32.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:32 smithi045 conmon[33655]: from='client.? 172.21.15.45:0/696419908' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T01:59:32.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:32 smithi045 conmon[33655]: cluster 2021-07 2021-07-31T01:59:32.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:32 smithi045 conmon[33655]: -31T01:59:31.970062+0000 mgr.x (mgr.24542) 531 2021-07-31T01:59:32.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:32 smithi045 conmon[33655]: : cluster [DBG] pgmap v1054: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:59:33.123 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestCls2PCQueue.AsyncReserve (3011 ms) 2021-07-31T01:59:33.123 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestCls2PCQueue.Commit 2021-07-31T01:59:33.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:33 smithi154 conmon[34553]: audit 2021-07-31T01:59:33.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:33 smithi154 conmon[34553]: 2021-07-31T01:59:32 2021-07-31T01:59:33.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:33 smithi154 conmon[34553]: .118833+0000 mon.a (mon.0) 1524 : audit [INF] from='client.? 172.21.15.45:0/696419908' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-4","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T01:59:33.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:33 smithi154 conmon[34553]: cluster 2021-07-31T01:59:33.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:33 smithi154 conmon[34553]: 2021-07-31T 2021-07-31T01:59:33.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:33 smithi154 conmon[34553]: 01:59:32.119114+0000 mon.a ( 2021-07-31T01:59:33.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:33 smithi154 conmon[34553]: mon.0) 1525 : cluster [DBG] 2021-07-31T01:59:33.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:33 smithi154 conmon[34553]: osdmap e839: 8 total, 8 up, 8 in 2021-07-31T01:59:33.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:33 smithi045 conmon[39718]: audit 2021-07-31T01:59:32.118833+0000 mon.a (mon.0) 1524 2021-07-31T01:59:33.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:33 smithi045 conmon[39718]: : audit [INF] from='client.? 172.21.15.45:0/696419908' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-4","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T01:59:33.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:33 smithi045 conmon[39718]: cluster 2021-07-31T01:59:32.119114+0000 mon.a (mon.0) 1525 : cluster [DBG] osdmap e839: 8 total, 8 up, 8 in 2021-07-31T01:59:33.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:33 smithi045 conmon[33655]: audit 2021-07-31T01:59:32.118833+0000 mon.a (mon.0) 1524 : 2021-07-31T01:59:33.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:33 smithi045 conmon[33655]: audit [INF] from='client.? 172.21.15.45:0/696419908' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-4","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T01:59:33.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:33 smithi045 conmon[33655]: cluster 2021-07-31T01:59:32.119114+0000 mon.a (mon.0) 1525 : cluster [DBG] osdmap e839: 8 total, 8 up, 8 in 2021-07-31T01:59:34.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:59:33 smithi045 conmon[61210]: level=error ts=2021-07-31T01:59:33.530Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:59:34.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:59:33 smithi045 conmon[61210]: level=error ts=2021-07-31T01:59:33.530Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:59:34.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:59:33 smithi045 conmon[61210]: level=error ts=2021-07-31T01:59:33.530Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T01:59:34.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:34 smithi154 conmon[34553]: cluster 2021-07-31T01:59:33.120425+0000 mon.a (mon.0) 1526 : cluster [DBG] osdmap e840: 8 total, 8 up, 8 in 2021-07-31T01:59:34.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:34 smithi154 conmon[34553]: cluster 2021-07-31T01:59:33.970492+0000 mgr.x (mgr.24542) 532 : cluster [DBG] pgmap v1057: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:59:34.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:34 smithi045 conmon[39718]: cluster 2021-07-31T01:59:33.120425+0000 mon.a (mon.0) 1526 2021-07-31T01:59:34.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:34 smithi045 conmon[39718]: : cluster [DBG] osdmap e840: 8 total, 8 up, 8 in 2021-07-31T01:59:34.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:34 smithi045 conmon[39718]: cluster 2021-07-31T01:59:33.970492+0000 mgr.x (mgr.24542) 532 : cluster [DBG] pgmap v1057: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:59:34.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:34 smithi045 conmon[33655]: cluster 2021-07-31T01:59:33.120425+0000 mon.a (mon. 2021-07-31T01:59:34.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:34 smithi045 conmon[33655]: 0) 1526 : cluster [DBG] osdmap e840: 8 total, 8 up, 8 in 2021-07-31T01:59:34.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:34 smithi045 conmon[33655]: cluster 2021-07-31T01:59:33.970492+0000 mgr.x (mgr.24542) 532 : cluster [DBG] pgmap v1057: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:59:35.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:35 smithi154 conmon[34553]: cluster 2021-07-31T01:59:35.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:35 smithi154 conmon[34553]: 2021-07-31T01:59:35.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:35 smithi154 conmon[34553]: 2021 2021-07-31T01:59:35.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:35 smithi154 conmon[34553]: - 2021-07-31T01:59:35.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:35 smithi154 conmon[34553]: 07 2021-07-31T01:59:35.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:35 smithi154 conmon[34553]: - 2021-07-31T01:59:35.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:35 smithi154 conmon[34553]: 31 2021-07-31T01:59:35.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:35 smithi154 conmon[34553]: T 2021-07-31T01:59:35.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:35 smithi154 conmon[34553]: 01 2021-07-31T01:59:35.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:35 smithi154 conmon[34553]: : 2021-07-31T01:59:35.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:35 smithi154 conmon[34553]: 59 2021-07-31T01:59:35.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:35 smithi154 conmon[34553]: : 2021-07-31T01:59:35.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:35 smithi154 conmon[34553]: 34 2021-07-31T01:59:35.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:35 smithi154 conmon[34553]: . 2021-07-31T01:59:35.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:35 smithi154 conmon[34553]: 131769 2021-07-31T01:59:35.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:35 smithi154 conmon[34553]: +0000 2021-07-31T01:59:35.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:35 smithi154 conmon[34553]: 2021-07-31T01:59:35.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:35 smithi154 conmon[34553]: mon.a 2021-07-31T01:59:35.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:35 smithi154 conmon[34553]: ( 2021-07-31T01:59:35.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:35 smithi154 conmon[34553]: mon.0) 1527 : cluster [DBG] osdmap e841: 8 total, 8 up, 8 in 2021-07-31T01:59:35.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:35 smithi154 conmon[34553]: audit 2021-07- 2021-07-31T01:59:35.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:35 smithi154 conmon[34553]: 31T01:59:34.133736+0000 mon.a (mon.0) 1528 : audit 2021-07-31T01:59:35.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:35 smithi154 conmon[34553]: [INF] from='client.? 172.21.15.45:0/714972328' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T01:59:35.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:35 smithi045 conmon[33655]: cluster 2021-07-31T01:59:34.131769+0000 mon.a (mon.0) 1527 : cluster [DBG] osdmap e841: 8 total, 8 up, 8 in 2021-07-31T01:59:35.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:35 smithi045 conmon[33655]: audit 2021-07-31T01:59:34.133736+0000 mon.a (mon.0) 1528 : audit [INF] from='client.? 172.21.15.45:0/714972328' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T01:59:35.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:35 smithi045 conmon[39718]: cluster 2021-07-31T01:59:35.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:35 smithi045 conmon[39718]: 2021-07-31T01:59:35.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:35 smithi045 conmon[39718]: 2021-07-31T01:59:34.131769+0000 2021-07-31T01:59:35.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:35 smithi045 conmon[39718]: mon.a (mon.0) 1527 : cluster [DBG] osdmap e841: 8 total, 8 up, 8 in 2021-07-31T01:59:35.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:35 smithi045 conmon[39718]: audit 2021-07-31T01:59:34.133736+0000 mon.a (mon.0) 1528 : audit [INF] from='client.? 172.21.15.45:0/714972328' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T01:59:36.149 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestCls2PCQueue.Commit (3026 ms) 2021-07-31T01:59:36.149 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestCls2PCQueue.Abort 2021-07-31T01:59:36.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:36 smithi154 conmon[34553]: audit 2021-07-31T01:59:35.133177+0000 mon.a (mon. 2021-07-31T01:59:36.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:36 smithi154 conmon[34553]: 0) 1529 : audit [INF] from='client.? 172.21.15.45:0/714972328' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-5","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T01:59:36.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:36 smithi154 conmon[34553]: cluster 2021-07-31T01:59:35.133480+0000 mon.a (mon.0) 1530 : cluster [DBG] osdmap e842: 8 total, 8 up, 8 in 2021-07-31T01:59:36.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:36 smithi154 conmon[34553]: cluster 2021-07-31T01:59:35.970864+0000 mgr.x (mgr 2021-07-31T01:59:36.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:36 smithi154 conmon[34553]: .24542) 533 : cluster [DBG] pgmap v1060: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:59:36.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:36 smithi154 conmon[34553]: 2021-07-31T01:59:36.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:36 smithi045 conmon[39718]: audit 2021-07-31T01:59:35.133177+0000 mon.a (mon 2021-07-31T01:59:36.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:36 smithi045 conmon[39718]: .0) 1529 : audit [INF] from='client.? 172.21.15.45:0/714972328' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-5","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T01:59:36.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:36 smithi045 conmon[39718]: cluster 2021-07 2021-07-31T01:59:36.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:36 smithi045 conmon[39718]: -31T01:59:35.133480+0000 mon.a (mon 2021-07-31T01:59:36.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:36 smithi045 conmon[39718]: .0) 1530 : cluster [DBG] osdmap e842: 8 total, 8 up, 8 in 2021-07-31T01:59:36.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:36 smithi045 conmon[39718]: cluster 2021-07-31T01:59:35.970864+0000 mgr.x (mgr.24542) 533 : cluster [DBG] pgmap v1060: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:59:36.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:36 smithi045 conmon[33655]: audit 2021-07-31T01:59:35.133177+0000 mon.a (mon.0) 1529 2021-07-31T01:59:36.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:36 smithi045 conmon[33655]: : audit [INF] from='client.? 172.21.15.45:0/714972328' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-5","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T01:59:36.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:36 smithi045 conmon[33655]: cluster 2021-07-31T01:59:35.133480+0000 mon.a (mon.0) 1530 : cluster [DBG] osdmap e842: 8 total, 8 up, 8 in 2021-07-31T01:59:36.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:36 smithi045 conmon[33655]: 2021-07-31T01:59:36.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:36 smithi045 conmon[33655]: cluster 2021- 2021-07-31T01:59:36.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:36 smithi045 conmon[33655]: 07-31T01:59:35.970864+0000 mgr.x (mgr.24542) 533 : 2021-07-31T01:59:36.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:36 smithi045 conmon[33655]: cluster [DBG] pgmap v1060: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:59:37.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:37 smithi154 conmon[34553]: cluster 2021-07-31T01:59:36.147240+0000 mon.a (mon.0) 1531 : cluster [DBG] osdmap e843: 8 total, 8 up, 8 in 2021-07-31T01:59:37.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:37 smithi045 conmon[33655]: cluster 2021-07-31T01:59:36.147240+0000 mon.a (mon.0) 1531 : 2021-07-31T01:59:37.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:37 smithi045 conmon[33655]: cluster [DBG] osdmap e843: 8 total, 8 up, 8 in 2021-07-31T01:59:37.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:37 smithi045 conmon[39718]: cluster 2021-07-31T01:59:36.147240 2021-07-31T01:59:37.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:37 smithi045 conmon[39718]: +0000 mon.a (mon.0) 1531 : cluster [DBG] osdmap e843: 8 total, 8 up, 8 in 2021-07-31T01:59:38.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:38 smithi154 conmon[34553]: cluster 2021-07-31T01:59:37.152507+0000 mon.a (mon.0) 1532 : cluster [DBG] osdmap e844: 8 total, 8 up, 8 in 2021-07-31T01:59:38.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:38 smithi154 conmon[34553]: audit 2021-07-31T01:59 2021-07-31T01:59:38.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:38 smithi154 conmon[34553]: :37.153990+0000 mon.c (mon.1) 38 : audit [INF] from='client.? 172.21.15.45:0/1178734428' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T01:59:38.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:38 smithi154 conmon[34553]: audit 2021-07-31T01:59:37.155378+0000 mon.a (mon.0) 1533 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T01:59:38.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:38 smithi154 conmon[34553]: cluster 2021-07-31T01:59:37.971328+0000 mgr.x (mgr.24542) 534 : cluster [DBG] pgmap v1063: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:59:38.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:38 smithi045 conmon[33655]: cluster 2021-07-31T01:59:37.152507+0000 mon.a (mon.0) 1532 : 2021-07-31T01:59:38.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:38 smithi045 conmon[33655]: cluster [DBG] osdmap e844: 8 total, 8 up, 8 in 2021-07-31T01:59:38.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:38 smithi045 conmon[33655]: audit 2021-07-31T01:59:37.153990+0000 mon.c (mon.1) 38 : audit [INF] from='client.? 172.21.15.45:0/1178734428' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T01:59:38.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:38 smithi045 conmon[33655]: audit 2021-07-31T01:59:37.155378+0000 mon.a (mon.0) 1533 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T01:59:38.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:38 smithi045 conmon[33655]: 2021-07-31T01:59:38.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:38 smithi045 conmon[33655]: cluster 2021-07-31T01:59:37.971328+0000 mgr.x (mgr.24542) 534 : cluster [DBG] 2021-07-31T01:59:38.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:38 smithi045 conmon[33655]: pgmap v1063: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:59:38.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:38 smithi045 conmon[39718]: cluster 2021-07-31T01:59: 2021-07-31T01:59:38.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:38 smithi045 conmon[39718]: 37.152507+0000 mon.a (mon.0) 1532 : cluster [DBG] osdmap e844: 8 total, 8 up, 8 in 2021-07-31T01:59:38.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:38 smithi045 conmon[39718]: audit 2021-07-31T01:59:38.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:38 smithi045 conmon[39718]: 2021-07-31T01: 2021-07-31T01:59:38.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:38 smithi045 conmon[39718]: 59:37.153990+0000 mon.c 2021-07-31T01:59:38.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:38 smithi045 conmon[39718]: (mon.1) 38 : audit 2021-07-31T01:59:38.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:38 smithi045 conmon[39718]: [INF] from='client.? 172.21.15.45:0/1178734428' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T01:59:38.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:38 smithi045 conmon[39718]: audit 2021-07-31T01:59:38.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:38 smithi045 conmon[39718]: 2021-07-31T01: 2021-07-31T01:59:38.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:38 smithi045 conmon[39718]: 59:37.155378+0000 mon.a 2021-07-31T01:59:38.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:38 smithi045 conmon[39718]: (mon.0) 1533 : audit 2021-07-31T01:59:38.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:38 smithi045 conmon[39718]: [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T01:59:38.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:38 smithi045 conmon[39718]: cluster 2021-07-31T01:59:38.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:38 smithi045 conmon[39718]: 2021-07-31T01:59:37.971328+0000 mgr.x (mgr.24542) 534 : cluster 2021-07-31T01:59:38.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:38 smithi045 conmon[39718]: [DBG] pgmap v1063: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:59:39.171 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestCls2PCQueue.Abort (3022 ms) 2021-07-31T01:59:39.171 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestCls2PCQueue.ReserveError 2021-07-31T01:59:39.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:39 smithi154 conmon[34553]: audit 2021-07-31T01:59:38.158675+0000 mon.a (mon 2021-07-31T01:59:39.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:39 smithi154 conmon[34553]: .0) 1534 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-6","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T01:59:39.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:39 smithi154 conmon[34553]: cluster 2021-07-31T01:59:38.158910+0000 mon.a (mon.0) 1535 : cluster [DBG] osdmap e845: 8 total, 8 up, 8 in 2021-07-31T01:59:39.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:39 smithi045 conmon[33655]: audit 2021-07-31T01:59:38.158675+0000 mon.a (mon.0) 1534 : 2021-07-31T01:59:39.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:39 smithi045 conmon[33655]: audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-6","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T01:59:39.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:39 smithi045 conmon[33655]: cluster 2021-07-31T01:59:38.158910+0000 mon.a (mon.0) 1535 : cluster [DBG] osdmap e845: 8 total, 8 up, 8 in 2021-07-31T01:59:39.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:39 smithi045 conmon[39718]: audit 2021-07-31T01:59:38. 2021-07-31T01:59:39.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:39 smithi045 conmon[39718]: 158675+0000 mon.a (mon.0) 1534 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-6","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T01:59:39.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:39 smithi045 conmon[39718]: cluster 2021-07-31T01:59:39.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:39 smithi045 conmon[39718]: 2021-07-31T01:59:38.158910 2021-07-31T01:59:39.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:39 smithi045 conmon[39718]: +0000 mon.a (mon.0) 1535 : cluster [DBG] 2021-07-31T01:59:39.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:39 smithi045 conmon[39718]: osdmap e845: 8 total, 8 up, 8 in 2021-07-31T01:59:40.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:40 smithi154 conmon[34553]: cluster 2021-07-31T01:59:40.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:40 smithi154 conmon[34553]: 2021-07-31T01:59:39.165927+0000 mon.a (mon.0) 1536 : cluster [DBG] 2021-07-31T01:59:40.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:40 smithi154 conmon[34553]: osdmap e846: 8 total, 8 up, 8 in 2021-07-31T01:59:40.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:40 smithi154 conmon[34553]: cluster 2021-07- 2021-07-31T01:59:40.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:40 smithi154 conmon[34553]: 31T01:59:39.971726+0000 mgr.x (mgr.24542) 535 : cluster [DBG] pgmap v1066: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:59:40.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:40 smithi045 conmon[39718]: cluster 2021-07-31T01:59:39.165927+0000 mon.a (mon 2021-07-31T01:59:40.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:40 smithi045 conmon[39718]: .0) 1536 : cluster [DBG] osdmap e846: 8 total, 8 up, 8 in 2021-07-31T01:59:40.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:40 smithi045 conmon[39718]: cluster 2021-07-31T01:59:39.971726+0000 mgr.x (mgr.24542) 535 : cluster [DBG] pgmap v1066: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:59:40.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:40 smithi045 conmon[33655]: cluster 2021-07-31T01:59:39.165927+0000 mon.a (mon.0) 1536 : cluster 2021-07-31T01:59:40.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:40 smithi045 conmon[33655]: [DBG] osdmap e846: 8 total, 8 up, 8 in 2021-07-31T01:59:40.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:40 smithi045 conmon[33655]: cluster 2021-07-31T01:59:39.971726+0000 mgr.x (mgr.24542) 535 : cluster [DBG] pgmap v1066: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:59:41.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:41 smithi154 conmon[34553]: cluster 2021-07-31T01:59:41.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:41 smithi154 conmon[34553]: 2021-07-31T01:59:41.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:41 smithi154 conmon[34553]: 2021 2021-07-31T01:59:41.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:41 smithi154 conmon[34553]: - 2021-07-31T01:59:41.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:41 smithi154 conmon[34553]: 07 2021-07-31T01:59:41.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:41 smithi154 conmon[34553]: -31T01:59:40.171763 2021-07-31T01:59:41.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:41 smithi154 conmon[34553]: +0000 mon.a (mon.0) 1537 : cluster [DBG] osdmap e847: 8 total, 8 up, 8 in 2021-07-31T01:59:41.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:41 smithi154 conmon[34553]: audit 2021-07-31T01:59:40.175678+0000 2021-07-31T01:59:41.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:41 smithi154 conmon[34553]: mon.c (mon.1) 39 : audit [INF] from='client.? 172.21.15.45:0/2077473978' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T01:59:41.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:41 smithi154 conmon[34553]: audit 2021-07-31T01:59:41.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:41 smithi154 conmon[34553]: 2021-07- 2021-07-31T01:59:41.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:41 smithi154 conmon[34553]: 31T01:59:40.176445+0000 mon.a (mon. 2021-07-31T01:59:41.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:41 smithi154 conmon[34553]: 0) 1538 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T01:59:41.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:41 smithi045 conmon[33655]: cluster 2021-07-31T01:59:40.171763+0000 mon.a (mon.0) 1537 : cluster 2021-07-31T01:59:41.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:41 smithi045 conmon[33655]: [DBG] osdmap e847: 8 total, 8 up, 8 in 2021-07-31T01:59:41.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:41 smithi045 conmon[33655]: audit 2021-07-31T01:59:40.175678+0000 mon.c (mon.1) 39 : audit [INF] from='client.? 172.21.15.45:0/2077473978' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T01:59:41.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:41 smithi045 conmon[33655]: audit 2021-07-31T01:59:40. 2021-07-31T01:59:41.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:41 smithi045 conmon[33655]: 176445+0000 mon.a (mon.0) 1538 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T01:59:41.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:41 smithi045 conmon[39718]: cluster 2021-07-31T01:59:41.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:41 smithi045 conmon[39718]: 2021-07-31T01:59:41.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:41 smithi045 conmon[39718]: 2021 2021-07-31T01:59:41.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:41 smithi045 conmon[39718]: - 2021-07-31T01:59:41.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:41 smithi045 conmon[39718]: 07 2021-07-31T01:59:41.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:41 smithi045 conmon[39718]: - 2021-07-31T01:59:41.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:41 smithi045 conmon[39718]: 31 2021-07-31T01:59:41.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:41 smithi045 conmon[39718]: T01:59:40.171763+0000 mon.a (mon.0) 1537 : 2021-07-31T01:59:41.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:41 smithi045 conmon[39718]: cluster [DBG] osdmap e847: 8 total, 8 up, 8 in 2021-07-31T01:59:41.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:41 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T01:59:41.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:41 smithi045 conmon[39718]: T01:59:40.175678+0000 mon.c ( 2021-07-31T01:59:41.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:41 smithi045 conmon[39718]: mon.1) 39 : audit [INF] from='client.? 172.21.15.45:0/2077473978' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T01:59:41.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:41 smithi045 conmon[39718]: 2021-07-31T01:59:41.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:41 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T01:59:41.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:41 smithi045 conmon[39718]: 31T01:59:40.176445+0000 mon.a 2021-07-31T01:59:41.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:41 smithi045 conmon[39718]: (mon.0) 1538 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T01:59:41.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:41 smithi045 conmon[39718]: 2021-07-31T01:59:42.177 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestCls2PCQueue.ReserveError (3006 ms) 2021-07-31T01:59:42.178 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestCls2PCQueue.CommitError 2021-07-31T01:59:42.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:42 smithi154 conmon[34553]: audit 2021-07-31T01:59:41.173357+0000 2021-07-31T01:59:42.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:42 smithi154 conmon[34553]: mon.a (mon.0) 1539 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-7","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T01:59:42.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:42 smithi154 conmon[34553]: cluster 2021-07-31T01:59:41.173669+0000 mon.a ( 2021-07-31T01:59:42.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:42 smithi154 conmon[34553]: mon.0) 1540 : cluster [DBG] osdmap e848: 8 total, 8 up, 8 in 2021-07-31T01:59:42.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:42 smithi154 conmon[34553]: cluster 2021-07-31T01:59:42.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:42 smithi154 conmon[34553]: 2021-07-31T01: 2021-07-31T01:59:42.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:42 smithi154 conmon[34553]: 59:41.972180+0000 mgr.x ( 2021-07-31T01:59:42.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:42 smithi154 conmon[34553]: mgr.24542) 536 : cluster [DBG] 2021-07-31T01:59:42.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:42 smithi154 conmon[34553]: pgmap v1069: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:59:42.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:42 smithi154 conmon[34553]: cluster 2021 2021-07-31T01:59:42.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:42 smithi154 conmon[34553]: -07-31T01:59:42.174336+0000 mon.a (mon.0) 1541 : 2021-07-31T01:59:42.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:42 smithi154 conmon[34553]: cluster [DBG] osdmap e849: 8 total, 8 up, 8 in 2021-07-31T01:59:42.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:42 smithi045 conmon[39718]: audit 2021-07-31T01:59:41.173357+0000 mon.a (mon 2021-07-31T01:59:42.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:42 smithi045 conmon[39718]: .0) 1539 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-7","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T01:59:42.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:42 smithi045 conmon[39718]: cluster 2021 2021-07-31T01:59:42.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:42 smithi045 conmon[39718]: -07-31T01:59:41.173669+0000 mon.a (mon.0) 1540 : cluster [DBG] osdmap e848: 8 total, 8 up, 8 in 2021-07-31T01:59:42.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:42 smithi045 conmon[39718]: 2021-07-31T01:59:42.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:42 smithi045 conmon[39718]: cluster 2021-07-31T01:59:42.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:42 smithi045 conmon[39718]: 2021-07-31T 2021-07-31T01:59:42.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:42 smithi045 conmon[39718]: 01:59:41.972180+0000 mgr.x (mgr. 2021-07-31T01:59:42.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:42 smithi045 conmon[39718]: 24542) 536 : cluster [DBG] pgmap v1069: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:59:42.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:42 smithi045 conmon[39718]: cluster 2021-07-31T01:59:42.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:42 smithi045 conmon[39718]: 2021-07-31T01:59:42.174336+0000 mon.a (mon. 2021-07-31T01:59:42.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:42 smithi045 conmon[39718]: 0) 1541 : cluster [DBG] osdmap e849: 8 total, 8 up, 8 in 2021-07-31T01:59:42.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:42 smithi045 conmon[33655]: audit 2021-07-31T01:59:41.173357+0000 mon.a (mon.0 2021-07-31T01:59:42.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:42 smithi045 conmon[33655]: ) 1539 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-7","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T01:59:42.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:42 smithi045 conmon[33655]: cluster 2021-07-31T01:59:41.173669+0000 mon.a (mon.0) 1540 : cluster [DBG] osdmap e848: 8 total, 8 up, 8 in 2021-07-31T01:59:42.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:42 smithi045 conmon[33655]: cluster 2021-07-31T01:59:41.972180+0000 mgr.x (mgr.24542) 536 : cluster [DBG] 2021-07-31T01:59:42.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:42 smithi045 conmon[33655]: pgmap v1069: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:59:42.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:42 smithi045 conmon[33655]: cluster 2021-07-31T01: 2021-07-31T01:59:42.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:42 smithi045 conmon[33655]: 59:42.174336+0000 mon.a (mon.0) 1541 : cluster [DBG] osdmap e849: 8 total, 8 up, 8 in 2021-07-31T01:59:44.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:59:43 smithi045 conmon[61210]: level=error ts=2021-07-31T01:59:43.530Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:59:44.027 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:59:43 smithi045 conmon[61210]: level=error ts=2021-07-31T01:59:43.530Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:59:44.027 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:59:43 smithi045 conmon[61210]: level=error ts=2021-07-31T01:59:43.530Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T01:59:44.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:44 smithi154 conmon[34553]: cluster 2021-07-31T01:59:43.189419+0000 mon.a (mon.0) 2021-07-31T01:59:44.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:44 smithi154 conmon[34553]: 1542 : cluster [DBG] osdmap e850: 8 total, 8 up, 8 in 2021-07-31T01:59:44.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:44 smithi154 conmon[34553]: audit 2021-07-31T01:59:43.191434+0000 mon.c (mon.1) 40 : audit [INF] from='client.? 172.21.15.45:0/3025285611' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T01:59:44.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:44 smithi154 conmon[34553]: audit 2021-07-31T01:59:43.192816 2021-07-31T01:59:44.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:44 smithi154 conmon[34553]: +0000 mon.a (mon.0) 1543 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T01:59:44.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:44 smithi154 conmon[34553]: cluster 2021-07 2021-07-31T01:59:44.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:44 smithi154 conmon[34553]: -31T01:59:43.972749+0000 mgr.x (mgr.24542) 537 : cluster 2021-07-31T01:59:44.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:44 smithi154 conmon[34553]: [DBG] pgmap v1072: 41 pgs: 32 creating+peering, 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:59:44.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:44 smithi045 conmon[39718]: cluster 2021-07-31T01:59:43.189419+0000 2021-07-31T01:59:44.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:44 smithi045 conmon[39718]: mon.a (mon.0) 1542 : cluster [DBG] osdmap e850: 8 total, 8 up, 8 in 2021-07-31T01:59:44.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:44 smithi045 conmon[39718]: audit 2021-07-31T01 2021-07-31T01:59:44.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:44 smithi045 conmon[39718]: :59:43.191434+0000 mon.c (mon.1) 2021-07-31T01:59:44.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:44 smithi045 conmon[39718]: 40 : audit [INF] from='client.? 172.21.15.45:0/3025285611' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T01:59:44.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:44 smithi045 conmon[39718]: audit 2021-07-31T01:59:44.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:44 smithi045 conmon[39718]: 2021-07-31T01:59:43. 2021-07-31T01:59:44.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:44 smithi045 conmon[39718]: 192816+0000 mon.a (mon.0) 1543 : 2021-07-31T01:59:44.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:44 smithi045 conmon[39718]: audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T01:59:44.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:44 smithi045 conmon[39718]: cluster 2021-07-31T 2021-07-31T01:59:44.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:44 smithi045 conmon[39718]: 01:59:43.972749+0000 mgr.x (mgr. 2021-07-31T01:59:44.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:44 smithi045 conmon[39718]: 24542) 537 : cluster [DBG] pgmap v1072: 41 pgs: 32 creating+peering, 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:59:44.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:44 smithi045 conmon[33655]: cluster 2021-07-31T01:59:43.189419+0000 mon.a (mon.0) 1542 2021-07-31T01:59:44.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:44 smithi045 conmon[33655]: : cluster [DBG] osdmap e850: 8 total, 8 up, 8 in 2021-07-31T01:59:44.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:44 smithi045 conmon[33655]: audit 2021-07-31T01:59:43.191434+0000 mon.c (mon.1) 40 : audit [INF] from='client.? 172.21.15.45:0/3025285611' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T01:59:44.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:44 smithi045 conmon[33655]: audit 2021-07-31T01:59:43.192816 2021-07-31T01:59:44.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:44 smithi045 conmon[33655]: +0000 mon.a (mon.0) 1543 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T01:59:44.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:44 smithi045 conmon[33655]: cluster 2021 2021-07-31T01:59:44.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:44 smithi045 conmon[33655]: -07-31T01:59:43.972749+0000 2021-07-31T01:59:44.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:44 smithi045 conmon[33655]: mgr.x (mgr.24542) 537 : cluster [DBG] 2021-07-31T01:59:44.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:44 smithi045 conmon[33655]: pgmap v1072: 41 pgs: 32 creating+peering, 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:59:45.199 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestCls2PCQueue.CommitError (3022 ms) 2021-07-31T01:59:45.200 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestCls2PCQueue.AbortError 2021-07-31T01:59:45.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:45 smithi154 conmon[34553]: audit 2021-07-31T01:59:45.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:45 smithi154 conmon[34553]: 2021- 2021-07-31T01:59:45.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:45 smithi154 conmon[34553]: 07-31T01:59:44.194925+0000 mon.a (mon.0) 1544 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-8","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T01:59:45.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:45 smithi154 conmon[34553]: cluster 2021-07-31T01: 2021-07-31T01:59:45.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:45 smithi154 conmon[34553]: 59:44.195294+0000 mon.a (mon.0) 1545 : cluster [DBG] osdmap e851: 8 total, 8 up, 8 in 2021-07-31T01:59:45.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:45 smithi045 conmon[33655]: audit 2021-07-31T01:59:45.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:45 smithi045 conmon[33655]: 2021-07-31T01:59: 2021-07-31T01:59:45.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:45 smithi045 conmon[33655]: 44.194925+0000 mon.a (mon.0) 1544 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-8","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T01:59:45.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:45 smithi045 conmon[33655]: cluster 2021-07-31T01:59: 2021-07-31T01:59:45.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:45 smithi045 conmon[33655]: 44.195294+0000 mon.a (mon.0) 1545 : 2021-07-31T01:59:45.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:45 smithi045 conmon[33655]: cluster [DBG] osdmap e851: 8 total, 8 up, 8 in 2021-07-31T01:59:45.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:45 smithi045 conmon[39718]: audit 2021-07-31T01:59:44.194925+0000 mon.a ( 2021-07-31T01:59:45.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:45 smithi045 conmon[39718]: mon.0) 1544 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-8","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T01:59:45.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:45 smithi045 conmon[39718]: cluster 2021-07-31T01 2021-07-31T01:59:45.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:45 smithi045 conmon[39718]: :59:44.195294+0000 mon.a (mon.0) 1545 : cluster [DBG] 2021-07-31T01:59:45.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:45 smithi045 conmon[39718]: osdmap e851: 8 total, 8 up, 8 in 2021-07-31T01:59:46.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:46 smithi154 conmon[34553]: cluster 2021-07-31T01:59:45.196831+0000 mon.a (mon.0) 1546 : cluster 2021-07-31T01:59:46.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:46 smithi154 conmon[34553]: [DBG] osdmap e852: 8 total, 8 up, 8 in 2021-07-31T01:59:46.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:46 smithi154 conmon[34553]: cluster 2021-07-31T01:59:45.973121+0000 mgr.x (mgr.24542) 538 : cluster [DBG] pgmap v1075: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:59:46.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:46 smithi045 conmon[33655]: cluster 2021-07-31T01:59:45.196831+0000 mon.a (mon 2021-07-31T01:59:46.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:46 smithi045 conmon[33655]: .0) 1546 : cluster [DBG] osdmap e852: 8 total, 8 up, 8 in 2021-07-31T01:59:46.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:46 smithi045 conmon[33655]: cluster 2021-07-31T01:59:45.973121+0000 mgr.x (mgr.24542) 538 : cluster [DBG] pgmap v1075: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:59:46.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:46 smithi045 conmon[39718]: cluster 2021-07-31T01: 2021-07-31T01:59:46.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:46 smithi045 conmon[39718]: 59:45.196831+0000 mon.a (mon.0) 1546 : cluster [DBG] osdmap e852: 8 total, 8 up, 8 in 2021-07-31T01:59:46.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:46 smithi045 conmon[39718]: cluster 2021 2021-07-31T01:59:46.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:46 smithi045 conmon[39718]: -07-31T01:59:45. 2021-07-31T01:59:46.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:46 smithi045 conmon[39718]: 973121+0000 mgr.x (mgr.24542) 538 : 2021-07-31T01:59:46.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:46 smithi045 conmon[39718]: cluster [DBG] pgmap v1075: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:59:47.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:47 smithi154 conmon[34553]: cluster 2021-07-31T01:59:46.208005+0000 mon.a (mon.0) 1547 : cluster [DBG] osdmap e853: 8 total, 8 up, 8 in 2021-07-31T01:59:47.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:47 smithi154 conmon[34553]: audit 2021-07-31T01:59:46.209656+0000 mon.c (mon.1) 41 : audit [INF] from='client.? 172.21.15.45:0/3106930294' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T01:59:47.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:47 smithi154 conmon[34553]: audit 2021-07-31T01:59:46.210128+0000 mon.a (mon.0) 1548 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T01:59:47.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:47 smithi045 conmon[39718]: cluster 2021-07-31T01:59:46.208005+0000 mon.a (mon.0) 2021-07-31T01:59:47.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:47 smithi045 conmon[39718]: 1547 : cluster [DBG] osdmap e853: 8 total, 8 up, 8 in 2021-07-31T01:59:47.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:47 smithi045 conmon[39718]: audit 2021-07-31T01:59:46.209656+0000 mon.c (mon.1) 41 2021-07-31T01:59:47.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:47 smithi045 conmon[39718]: : audit [INF] from='client.? 172.21.15.45:0/3106930294' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T01:59:47.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:47 smithi045 conmon[39718]: audit 2021-07-31T01:59:46.210128+0000 mon.a (mon.0) 1548 2021-07-31T01:59:47.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:47 smithi045 conmon[39718]: : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T01:59:47.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:47 smithi045 conmon[33655]: cluster 2021-07-31T01:59:46.208005+0000 mon.a (mon.0) 1547 : cluster [DBG] osdmap e853: 8 total, 8 up, 8 in 2021-07-31T01:59:47.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:47 smithi045 conmon[33655]: audit 2021-07-31T01:59:47.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:47 smithi045 conmon[33655]: 2021-07-31T01:59:46.209656+0000 mon.c (mon.1) 41 : audit [INF] from='client.? 172.21.15.45:0/3106930294' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T01:59:47.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:47 smithi045 conmon[33655]: audit 2021-07-31T01:59:46.210128+0000 mon.a (mon.0) 1548 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T01:59:47.659 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T01:59:47.937 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T01:59:47.937 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 14m ago 15m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T01:59:47.937 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 12m ago 15m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T01:59:47.938 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 12m ago 18m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T01:59:47.938 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 14m ago 20m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T01:59:47.938 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 14m ago 20m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T01:59:47.938 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 12m ago 18m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T01:59:47.939 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 14m ago 19m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T01:59:47.939 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 14m ago 15m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T01:59:47.939 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 12m ago 15m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T01:59:47.939 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 14m ago 18m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T01:59:47.940 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 14m ago 18m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T01:59:47.940 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 14m ago 17m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T01:59:47.940 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 14m ago 17m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T01:59:47.940 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 12m ago 17m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T01:59:47.940 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 12m ago 16m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T01:59:47.941 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 12m ago 16m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T01:59:47.941 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 12m ago 16m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T01:59:47.941 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 12m ago 15m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T01:59:48.223 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestCls2PCQueue.AbortError (3024 ms) 2021-07-31T01:59:48.223 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestCls2PCQueue.MultiReserve 2021-07-31T01:59:48.278 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T01:59:48.278 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T01:59:48.279 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T01:59:48.279 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T01:59:48.279 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T01:59:48.279 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T01:59:48.280 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T01:59:48.280 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T01:59:48.280 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T01:59:48.280 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T01:59:48.280 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T01:59:48.281 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T01:59:48.281 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T01:59:48.281 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T01:59:48.281 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T01:59:48.281 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T01:59:48.282 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T01:59:48.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:48 smithi154 conmon[34553]: audit 2021-07-31T01:59:47.215279+0000 mon.a (mon.0) 1549 : audit 2021-07-31T01:59:48.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:48 smithi154 conmon[34553]: [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-9","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T01:59:48.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:48 smithi154 conmon[34553]: cluster 2021-07-31T01:59:47.215495+0000 mon.a (mon.0) 1550 : cluster [DBG] osdmap e854: 8 total, 8 up, 8 in 2021-07-31T01:59:48.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:48 smithi154 conmon[34553]: audit 2021- 2021-07-31T01:59:48.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:48 smithi154 conmon[34553]: 07-31T01:59:47.646697+0000 mgr.x (mgr.24542) 539 : audit [DBG] from='client.15216 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:59:48.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:48 smithi154 conmon[34553]: audit 2021-07-31T01:59: 2021-07-31T01:59:48.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:48 smithi154 conmon[34553]: 47.929083+0000 mgr.x (mgr.24542) 540 : audit [DBG] from='client.15222 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:59:48.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:48 smithi154 conmon[34553]: cluster 2021-07-31T01:59:48.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:48 smithi154 conmon[34553]: 2021-07-31T01:59:47.973458+0000 mgr.x 2021-07-31T01:59:48.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:48 smithi154 conmon[34553]: (mgr.24542) 541 : cluster [DBG] pgmap v1078: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:59:48.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:48 smithi045 conmon[33655]: audit 2021-07-31T01:59:47.215279+0000 mon.a ( 2021-07-31T01:59:48.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:48 smithi045 conmon[33655]: mon.0) 1549 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-9","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T01:59:48.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:48 smithi045 conmon[33655]: cluster 2021-07-31T01:59:47.215495+0000 mon.a (mon.0) 1550 : cluster [DBG] osdmap e854: 8 total, 8 up, 8 in 2021-07-31T01:59:48.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:48 smithi045 conmon[33655]: audit 2021-07 2021-07-31T01:59:48.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:48 smithi045 conmon[33655]: -31T01:59:47.646697+0000 mgr.x ( 2021-07-31T01:59:48.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:48 smithi045 conmon[33655]: mgr.24542) 539 : audit [DBG] from='client.15216 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:59:48.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:48 smithi045 conmon[33655]: audit 2021-07-31T01:59:48.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:48 smithi045 conmon[33655]: 2021-07-31T01:59: 2021-07-31T01:59:48.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:48 smithi045 conmon[33655]: 47.929083+0000 mgr.x (mgr.24542) 540 2021-07-31T01:59:48.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:48 smithi045 conmon[33655]: : audit [DBG] from='client.15222 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:59:48.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:48 smithi045 conmon[33655]: cluster 2021 2021-07-31T01:59:48.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:48 smithi045 conmon[33655]: -07-31T01:59:47.973458+0000 2021-07-31T01:59:48.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:48 smithi045 conmon[33655]: mgr.x (mgr.24542) 541 : cluster [DBG] pgmap v1078: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:59:48.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:48 smithi045 conmon[39718]: audit 2021-07-31T01:59 2021-07-31T01:59:48.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:48 smithi045 conmon[39718]: :47.215279+0000 mon.a (mon.0) 1549 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-9","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T01:59:48.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:48 smithi045 conmon[39718]: cluster 2021-07-31T01:59 2021-07-31T01:59:48.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:48 smithi045 conmon[39718]: :47.215495+0000 mon.a (mon.0) 2021-07-31T01:59:48.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:48 smithi045 conmon[39718]: 1550 : cluster [DBG] osdmap e854: 8 total, 8 up, 8 in 2021-07-31T01:59:48.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:48 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T01:59:48.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:48 smithi045 conmon[39718]: 31T01:59:47.646697+0000 2021-07-31T01:59:48.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:48 smithi045 conmon[39718]: mgr.x (mgr.24542) 539 : audit [DBG] 2021-07-31T01:59:48.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:48 smithi045 conmon[39718]: from='client.15216 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:59:48.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:48 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T01:59:48.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:48 smithi045 conmon[39718]: T01:59:47.929083+0000 mgr.x ( 2021-07-31T01:59:48.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:48 smithi045 conmon[39718]: mgr.24542) 540 : audit [DBG] 2021-07-31T01:59:48.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:48 smithi045 conmon[39718]: from='client.15222 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T01:59:48.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:48 smithi045 conmon[39718]: cluster 2021-07-31T01:59:48.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:48 smithi045 conmon[39718]: 2021-07-31T01:59:47 2021-07-31T01:59:48.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:48 smithi045 conmon[39718]: .973458+0000 mgr.x (mgr.24542) 541 : 2021-07-31T01:59:48.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:48 smithi045 conmon[39718]: cluster [DBG] pgmap v1078: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:59:49.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:49 smithi154 conmon[34553]: cluster 2021-07-31T01:59:48.220708+0000 mon.a (mon.0) 1551 : cluster [DBG] osdmap e855: 8 total, 8 up, 8 in 2021-07-31T01:59:49.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:49 smithi154 conmon[34553]: audit 2021-07-31T01:59:48.278115+0000 mon.c (mon.1) 42 : audit [DBG] from='client.? 172.21.15.45:0/1195944470' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T01:59:49.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:49 smithi045 conmon[33655]: cluster 2021-07-31T01:59:48.220708+0000 mon.a (mon.0) 1551 2021-07-31T01:59:49.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:49 smithi045 conmon[33655]: : cluster [DBG] osdmap e855: 8 total, 8 up, 8 in 2021-07-31T01:59:49.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:49 smithi045 conmon[33655]: audit 2021-07-31T01:59:48.278115+0000 mon.c (mon.1) 42 : audit [DBG] 2021-07-31T01:59:49.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:49 smithi045 conmon[33655]: from='client.? 172.21.15.45:0/1195944470' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T01:59:49.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:49 smithi045 conmon[39718]: cluster 2021-07-31T01:59:48.220708 2021-07-31T01:59:49.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:49 smithi045 conmon[39718]: +0000 mon.a (mon.0) 1551 : cluster [DBG] osdmap e855: 8 total, 8 up, 8 in 2021-07-31T01:59:49.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:49 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T01:59:49.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:49 smithi045 conmon[39718]: 31T01:59:48.278115+0000 mon.c (mon.1 2021-07-31T01:59:49.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:49 smithi045 conmon[39718]: ) 42 : audit [DBG] from='client.? 172.21.15.45:0/1195944470' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T01:59:50.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:50 smithi154 conmon[34553]: cluster 2021-07-31T01:59:49.226775+0000 mon.a (mon.0) 1552 : cluster [DBG] osdmap e856: 8 total, 8 up, 8 in 2021-07-31T01:59:50.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:50 smithi154 conmon[34553]: 2021-07-31T01:59:50.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:50 smithi154 conmon[34553]: audit 2021-07-31T01:59:49.228014+0000 mon.a (mon.0) 1553 : audit [INF] from='client.? 172.21.15.45:0/1597262857' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T01:59:50.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:50 smithi154 conmon[34553]: cluster 2021-07-31T01:59:49.973932+0000 mgr.x (mgr.24542) 542 : cluster [DBG] pgmap v1081: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:59:50.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:50 smithi045 conmon[39718]: cluster 2021-07-31T01:59:49.226775+0000 mon.a (mon.0) 2021-07-31T01:59:50.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:50 smithi045 conmon[39718]: 1552 : cluster [DBG] osdmap e856: 8 total, 8 up, 8 in 2021-07-31T01:59:50.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:50 smithi045 conmon[39718]: audit 2021-07-31T01:59:49.228014+0000 mon.a (mon.0) 1553 : audit [INF] from='client.? 172.21.15.45:0/1597262857' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T01:59:50.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:50 smithi045 conmon[39718]: cluster 2021-07-31T01:59:49.973932+0000 mgr.x (mgr.24542) 542 : cluster [DBG] pgmap v1081: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:59:50.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:50 smithi045 conmon[33655]: cluster 2021-07-31T01:59:49.226775+0000 mon.a (mon.0 2021-07-31T01:59:50.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:50 smithi045 conmon[33655]: ) 1552 : cluster [DBG] osdmap e856: 8 total, 8 up, 8 in 2021-07-31T01:59:50.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:50 smithi045 conmon[33655]: audit 2021-07-31T01:59:49.228014+0000 mon.a (mon.0) 1553 : audit [INF] from='client.? 172.21.15.45:0/1597262857' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T01:59:50.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:50 smithi045 conmon[33655]: cluster 2021-07-31T01:59:49 2021-07-31T01:59:50.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:50 smithi045 conmon[33655]: .973932+0000 mgr.x (mgr.24542) 542 : cluster [DBG] pgmap v1081: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:59:50.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:50 smithi045 conmon[33655]: 2021-07-31T01:59:51.241 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestCls2PCQueue.MultiReserve (3018 ms) 2021-07-31T01:59:51.241 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestCls2PCQueue.MultiCommit 2021-07-31T01:59:51.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:51 smithi154 conmon[34553]: audit 2021-07-31T01:59:50.233747+0000 mon.a (mon.0) 1554 : audit 2021-07-31T01:59:51.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:51 smithi154 conmon[34553]: [INF] from='client.? 172.21.15.45:0/1597262857' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-10","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T01:59:51.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:51 smithi154 conmon[34553]: cluster 2021-07-31T01:59:50.233964+0000 mon.a (mon.0) 1555 : cluster [DBG] osdmap e857: 8 total, 8 up, 8 in 2021-07-31T01:59:51.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:51 smithi045 conmon[33655]: audit 2021-07-31T01:59:50.233747+0000 mon.a (mon.0) 1554 : audit [INF] from='client.? 172.21.15.45:0/1597262857' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-10","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T01:59:51.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:51 smithi045 conmon[33655]: cluster 2021-07-31T01:59:50.233964+0000 mon.a (mon.0) 1555 : cluster [DBG] osdmap e857: 8 total, 8 up, 8 in 2021-07-31T01:59:51.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:51 smithi045 conmon[39718]: audit 2021-07-31T01:59:50.233747+0000 mon.a (mon.0 2021-07-31T01:59:51.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:51 smithi045 conmon[39718]: ) 1554 : audit [INF] from='client.? 172.21.15.45:0/1597262857' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-10","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T01:59:51.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:51 smithi045 conmon[39718]: cluster 2021-07-31T01:59:50 2021-07-31T01:59:51.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:51 smithi045 conmon[39718]: .233964+0000 mon.a (mon.0) 1555 : cluster [DBG] osdmap e857: 8 total, 8 up, 8 in 2021-07-31T01:59:51.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:51 smithi045 conmon[39718]: 2021-07-31T01:59:52.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:52 smithi154 conmon[34553]: cluster 2021-07-31T01:59:51.239434+0000 mon.a (mon.0) 1556 : cluster [DBG] osdmap e858: 8 total, 8 up, 8 in 2021-07-31T01:59:52.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:52 smithi154 conmon[34553]: cluster 2021-07-31T01:59:51.974315+0000 mgr.x (mgr.24542) 543 : cluster [DBG] pgmap v1084: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:59:52.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:52 smithi045 conmon[39718]: cluster 2021-07-31T01:59:51.239434+0000 mon.a 2021-07-31T01:59:52.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:52 smithi045 conmon[39718]: (mon.0) 1556 : cluster [DBG] osdmap e858: 8 total, 8 up, 8 in 2021-07-31T01:59:52.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:52 smithi045 conmon[39718]: cluster 2021-07-31T01 2021-07-31T01:59:52.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:52 smithi045 conmon[39718]: :59:51.974315+0000 mgr.x (mgr.24542) 543 : cluster [DBG] pgmap v1084: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:59:52.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:52 smithi045 conmon[33655]: cluster 2021-07-31T01:59 2021-07-31T01:59:52.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:52 smithi045 conmon[33655]: :51.239434+0000 mon.a (mon.0) 1556 : cluster [DBG] osdmap e858: 8 total, 8 up, 8 in 2021-07-31T01:59:52.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:52 smithi045 conmon[33655]: cluster 2021-07-31T01:59:52.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:52 smithi045 conmon[33655]: 2021-07-31T01:59:51.974315+0000 mgr.x (mgr.24542) 543 2021-07-31T01:59:52.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:52 smithi045 conmon[33655]: : cluster [DBG] pgmap v1084: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:59:53.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:53 smithi154 conmon[34553]: cluster 2021-07-31T01:59:52.244739+0000 mon.a (mon.0) 1557 : cluster [DBG] osdmap e859: 8 total, 8 up, 8 in 2021-07-31T01:59:53.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:53 smithi154 conmon[34553]: audit 2021-07-31T01:59:52 2021-07-31T01:59:53.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:53 smithi154 conmon[34553]: .245589+0000 mon.a (mon.0) 1558 : audit [INF] from='client.? 172.21.15.45:0/4002744968' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T01:59:53.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:53 smithi045 conmon[33655]: cluster 2021-07-31T01:59:52.244739+0000 mon.a (mon.0) 1557 : cluster [DBG] osdmap e859: 8 total, 8 up, 8 in 2021-07-31T01:59:53.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:53 smithi045 conmon[33655]: audit 2021-07-31T01:59:52.245589+0000 mon.a (mon.0) 1558 : audit [INF] from='client.? 172.21.15.45:0/4002744968' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T01:59:53.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:53 smithi045 conmon[39718]: cluster 2021-07-31T01:59:52.244739+0000 mon.a 2021-07-31T01:59:53.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:53 smithi045 conmon[39718]: (mon.0) 1557 : cluster [DBG] osdmap e859: 8 total, 8 up, 8 in 2021-07-31T01:59:53.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:53 smithi045 conmon[39718]: audit 2021-07-31T01: 2021-07-31T01:59:53.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:53 smithi045 conmon[39718]: 59:52.245589+0000 mon.a (mon.0) 1558 : audit 2021-07-31T01:59:53.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:53 smithi045 conmon[39718]: [INF] from='client.? 172.21.15.45:0/4002744968' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T01:59:54.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:59:53 smithi045 conmon[61210]: level=error ts=2021-07-31T01:59:53.530Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:59:54.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:59:53 smithi045 conmon[61210]: level=error ts=2021-07-31T01:59:53.530Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T01:59:54.027 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 01:59:53 smithi045 conmon[61210]: level=error ts=2021-07-31T01:59:53.531Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T01:59:54.261 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestCls2PCQueue.MultiCommit (3019 ms) 2021-07-31T01:59:54.261 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestCls2PCQueue.MultiAbort 2021-07-31T01:59:54.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:54 smithi045 conmon[39718]: audit 2021-07-31T01:59:53.251389+0000 mon.a (mon.0) 1559 : audit 2021-07-31T01:59:54.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:54 smithi045 conmon[39718]: [INF] from='client.? 172.21.15.45:0/4002744968' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-11","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T01:59:54.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:54 smithi045 conmon[39718]: cluster 2021-07-31T01:59:53.251688+0000 mon.a (mon.0) 1560 : cluster [DBG] osdmap e860: 8 total, 8 up, 8 in 2021-07-31T01:59:54.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:54 smithi045 conmon[39718]: cluster 2021 2021-07-31T01:59:54.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:54 smithi045 conmon[39718]: -07-31T01:59:53.974881+0000 mgr.x (mgr.24542) 544 : cluster [DBG] pgmap v1087: 41 pgs: 41 active+clean; 661 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 55 KiB/s rd, 688 KiB/s wr, 743 op/s 2021-07-31T01:59:54.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:54 smithi045 conmon[33655]: audit 2021-07-31T01:59:53.251389+0000 mon.a (mon.0) 1559 2021-07-31T01:59:54.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:54 smithi045 conmon[33655]: : audit [INF] from='client.? 172.21.15.45:0/4002744968' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-11","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T01:59:54.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:54 smithi045 conmon[33655]: cluster 2021-07-31T01:59:53.251688+0000 mon.a (mon.0) 1560 : cluster [DBG] osdmap e860: 8 total, 8 up, 8 in 2021-07-31T01:59:54.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:54 smithi045 conmon[33655]: cluster 2021-07-31T01:59:54.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:54 smithi045 conmon[33655]: 2021-07-31T01 2021-07-31T01:59:54.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:54 smithi045 conmon[33655]: :59:53.974881+0000 mgr.x (mgr 2021-07-31T01:59:54.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:54 smithi045 conmon[33655]: .24542) 544 : cluster [DBG] pgmap v1087: 41 pgs: 41 active+clean; 661 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 55 KiB/s rd, 688 KiB/s wr, 743 op/s 2021-07-31T01:59:54.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:54 smithi045 conmon[33655]: 2021-07-31T01:59:54.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:54 smithi154 conmon[34553]: audit 2021-07-31T01:59:53.251389+0000 mon.a (mon.0) 2021-07-31T01:59:54.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:54 smithi154 conmon[34553]: 1559 : audit [INF] from='client.? 172.21.15.45:0/4002744968' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-11","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T01:59:54.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:54 smithi154 conmon[34553]: cluster 2021-07-31T01:59:53.251688+0000 mon.a (mon.0) 1560 : cluster [DBG] osdmap e860: 8 total, 8 up, 8 in 2021-07-31T01:59:54.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:54 smithi154 conmon[34553]: cluster 2021-07-31T01:59:53.974881+0000 mgr.x (mgr.24542) 544 : 2021-07-31T01:59:54.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:54 smithi154 conmon[34553]: cluster [DBG] pgmap v1087: 41 pgs: 41 active+clean; 661 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 55 KiB/s rd, 688 KiB/s wr, 743 op/s 2021-07-31T01:59:55.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:55 smithi045 conmon[39718]: cluster 2021-07-31T01:59:54 2021-07-31T01:59:55.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:55 smithi045 conmon[39718]: .258467+0000 mon.a (mon.0) 1561 : cluster [DBG] osdmap e861: 8 total, 8 up, 8 in 2021-07-31T01:59:55.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:55 smithi045 conmon[33655]: cluster 2021-07-31T01:59:54.258467+0000 mon.a (mon. 2021-07-31T01:59:55.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:55 smithi045 conmon[33655]: 0) 1561 : cluster [DBG] osdmap e861: 8 total, 8 up, 8 in 2021-07-31T01:59:55.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:55 smithi154 conmon[34553]: cluster 2021-07-31T01:59:54.258467+0000 mon.a (mon.0) 1561 : cluster [DBG] osdmap e861: 8 total, 8 up, 8 in 2021-07-31T01:59:55.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:55 smithi154 conmon[34553]: 2021-07-31T01:59:56.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:56 smithi045 conmon[33655]: cluster 2021-07-31T01:59:55.263799+0000 mon.a (mon.0) 2021-07-31T01:59:56.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:56 smithi045 conmon[33655]: 1562 : cluster [DBG] osdmap e862: 8 total, 8 up, 8 in 2021-07-31T01:59:56.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:56 smithi045 conmon[33655]: audit 2021-07-31T01:59:55.266687+0000 mon.b (mon.2) 127 : audit [INF] from='client.? 172.21.15.45:0/1996937561' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T01:59:56.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:56 smithi045 conmon[33655]: audit 2021-07-31 2021-07-31T01:59:56.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:56 smithi045 conmon[33655]: T01:59:55.267449+0000 mon.a (mon.0) 1563 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T01:59:56.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:56 smithi045 conmon[33655]: 2021-07-31T01:59:56.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:56 smithi045 conmon[33655]: cluster 2021-07 2021-07-31T01:59:56.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:56 smithi045 conmon[33655]: -31T01:59:55.975380+0000 2021-07-31T01:59:56.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:56 smithi045 conmon[33655]: mgr.x (mgr.24542) 545 : cluster [DBG] 2021-07-31T01:59:56.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:56 smithi045 conmon[33655]: pgmap v1090: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:59:56.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:56 smithi045 conmon[39718]: cluster 2021-07-31T01:59:55.263799+0000 mon.a ( 2021-07-31T01:59:56.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:56 smithi045 conmon[39718]: mon.0) 1562 : cluster [DBG] osdmap e862: 8 total, 8 up, 8 in 2021-07-31T01:59:56.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:56 smithi045 conmon[39718]: audit 2021-07-31T01:59:55.266687+0000 mon.b 2021-07-31T01:59:56.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:56 smithi045 conmon[39718]: (mon.2) 127 : audit [INF] from='client.? 172.21.15.45:0/1996937561' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T01:59:56.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:56 smithi045 conmon[39718]: audit 2021-07-31T01:59:55.267449+0000 2021-07-31T01:59:56.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:56 smithi045 conmon[39718]: mon.a (mon.0) 1563 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T01:59:56.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:56 smithi045 conmon[39718]: cluster 2021-07-31T01:59:55.975380+0000 2021-07-31T01:59:56.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:56 smithi045 conmon[39718]: mgr.x (mgr.24542) 545 : cluster [DBG] pgmap v1090: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:59:56.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:56 smithi154 conmon[34553]: cluster 2021-07-31T01:59:55.263799+0000 mon.a (mon.0) 1562 : cluster 2021-07-31T01:59:56.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:56 smithi154 conmon[34553]: [DBG] osdmap e862: 8 total, 8 up, 8 in 2021-07-31T01:59:56.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:56 smithi154 conmon[34553]: audit 2021-07-31T01:59:55.266687+0000 mon.b (mon.2) 127 : audit [INF] from='client.? 172.21.15.45:0/1996937561' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T01:59:56.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:56 smithi154 conmon[34553]: audit 2021-07-31T01: 2021-07-31T01:59:56.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:56 smithi154 conmon[34553]: 59:55.267449+0000 mon.a (mon.0) 1563 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T01:59:56.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:56 smithi154 conmon[34553]: cluster 2021-07-31T01:59:56.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:56 smithi154 conmon[34553]: 2021-07-31T01:59:55.975380+0000 mgr.x (mgr.24542) 545 : cluster 2021-07-31T01:59:56.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:56 smithi154 conmon[34553]: [DBG] pgmap v1090: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:59:57.274 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestCls2PCQueue.MultiAbort (3013 ms) 2021-07-31T01:59:57.274 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestCls2PCQueue.ReserveCommit 2021-07-31T01:59:57.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:57 smithi154 conmon[34553]: audit 2021-07-31T01:59:56.270880+0000 mon.a (mon. 2021-07-31T01:59:57.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:57 smithi154 conmon[34553]: 0) 1564 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-12","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T01:59:57.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:57 smithi154 conmon[34553]: cluster 2021-07-31T01:59:56.271086+0000 mon.a (mon.0) 1565 : cluster [DBG] osdmap e863: 8 total, 8 up, 8 in 2021-07-31T01:59:57.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:57 smithi045 conmon[39718]: audit 2021-07-31T01:59:56.270880 2021-07-31T01:59:57.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:57 smithi045 conmon[39718]: +0000 mon.a (mon.0) 1564 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-12","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T01:59:57.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:57 smithi045 conmon[39718]: cluster 2021-07-31T01:59:56.271086+0000 mon.a (mon.0) 1565 : cluster [DBG] osdmap e863: 8 total, 8 up, 8 in 2021-07-31T01:59:57.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:57 smithi045 conmon[33655]: audit 2021-07-31T01:59:56.270880+0000 mon.a (mon.0) 1564 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-12","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T01:59:57.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:57 smithi045 conmon[33655]: cluster 2021-07-31T01:59:56.271086+0000 mon.a (mon.0) 1565 : cluster [DBG] osdmap e863: 8 total, 8 up, 8 in 2021-07-31T01:59:58.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:58 smithi154 conmon[34553]: cluster 2021-07-31T01:59:57.271458+0000 mon.a (mon.0 2021-07-31T01:59:58.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:58 smithi154 conmon[34553]: ) 1566 : cluster [DBG] osdmap e864: 8 total, 8 up, 8 in 2021-07-31T01:59:58.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:58 smithi154 conmon[34553]: cluster 2021-07-31T01:59:57.975760+0000 mgr.x (mgr.24542) 546 : cluster [DBG] pgmap v1093: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:59:58.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:58 smithi045 conmon[39718]: cluster 2021-07-31T01:59:57.271458+0000 2021-07-31T01:59:58.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:58 smithi045 conmon[39718]: mon.a (mon.0) 1566 : cluster [DBG] osdmap e864: 8 total, 8 up, 8 in 2021-07-31T01:59:58.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:58 smithi045 conmon[39718]: cluster 2021-07-31T 2021-07-31T01:59:58.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:58 smithi045 conmon[39718]: 01:59:57.975760+0000 mgr.x (mgr.24542) 546 : cluster [DBG] pgmap v1093: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:59:58.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:58 smithi045 conmon[39718]: 2021-07-31T01:59:58.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:58 smithi045 conmon[33655]: cluster 2021-07-31T01:59:57.271458+0000 mon.a 2021-07-31T01:59:58.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:58 smithi045 conmon[33655]: (mon.0) 1566 : cluster [DBG] osdmap e864: 8 total, 8 up, 8 in 2021-07-31T01:59:58.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:58 smithi045 conmon[33655]: cluster 2021-07-31T01:59:57.975760+0000 mgr.x (mgr.24542) 2021-07-31T01:59:58.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:58 smithi045 conmon[33655]: 546 : cluster [DBG] pgmap v1093: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2021-07-31T01:59:59.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:59 smithi154 conmon[34553]: cluster 2021-07-31T01:59:58.288742+0000 mon.a (mon.0) 1567 : cluster [DBG] osdmap e865: 8 total, 8 up, 8 in 2021-07-31T01:59:59.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 01:59:59 smithi154 conmon[34553]: audit 2021-07-31T01:59:58.289506+0000 mon.a (mon.0) 1568 : audit [INF] from='client.? 172.21.15.45:0/2290598225' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T01:59:59.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:59 smithi045 conmon[39718]: cluster 2021-07-31T01:59:58.288742+0000 2021-07-31T01:59:59.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:59 smithi045 conmon[39718]: mon.a (mon.0) 1567 : cluster [DBG] osdmap e865: 8 total, 8 up, 8 in 2021-07-31T01:59:59.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:59 smithi045 conmon[39718]: audit 2021 2021-07-31T01:59:59.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:59 smithi045 conmon[39718]: -07-31T01:59 2021-07-31T01:59:59.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:59 smithi045 conmon[39718]: :58.289506+0000 mon.a ( 2021-07-31T01:59:59.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:59 smithi045 conmon[39718]: mon.0) 1568 : audit [INF] 2021-07-31T01:59:59.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 01:59:59 smithi045 conmon[39718]: from='client.? 172.21.15.45:0/2290598225' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T01:59:59.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:59 smithi045 conmon[33655]: cluster 2021-07-31T01:59:58.288742+0000 2021-07-31T01:59:59.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:59 smithi045 conmon[33655]: mon.a (mon.0) 1567 : cluster [DBG] osdmap e865: 8 total, 8 up, 8 in 2021-07-31T01:59:59.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:59 smithi045 conmon[33655]: audit 2021-07-31 2021-07-31T01:59:59.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:59 smithi045 conmon[33655]: T01:59:58.289506+0000 mon.a (mon.0 2021-07-31T01:59:59.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 01:59:59 smithi045 conmon[33655]: ) 1568 : audit [INF] from='client.? 172.21.15.45:0/2290598225' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:00:00.304 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestCls2PCQueue.ReserveCommit (3029 ms) 2021-07-31T02:00:00.304 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestCls2PCQueue.ReserveAbort 2021-07-31T02:00:00.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:00 smithi154 conmon[34553]: audit 2021-07-31T01:59:59.294359+0000 mon.a (mon.0) 2021-07-31T02:00:00.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:00 smithi154 conmon[34553]: 1569 : audit [INF] from='client.? 172.21.15.45:0/2290598225' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-13","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:00:00.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:00 smithi154 conmon[34553]: cluster 2021-07-31T01:59:59.294626+0000 mon.a (mon.0) 1570 : cluster [DBG] osdmap e866: 8 total, 8 up, 8 in 2021-07-31T02:00:00.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:00 smithi154 conmon[34553]: cluster 2021-07-31T02:00:00.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:00 smithi154 conmon[34553]: 2021-07-31T01:59:59.976131+0000 mgr.x (mgr.24542) 547 : cluster [DBG] pgmap v1096: 41 pgs: 32 creating+peering, 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2021-07-31T02:00:00.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:00 smithi154 conmon[34553]: cluster 2021-07-31T02 2021-07-31T02:00:00.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:00 smithi154 conmon[34553]: :00:00.000137+0000 mon.a (mon.0) 2021-07-31T02:00:00.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:00 smithi154 conmon[34553]: 1571 : cluster [INF] overall HEALTH_OK 2021-07-31T02:00:00.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:00 smithi045 conmon[33655]: audit 2021-07-31T01:59:59.294359+0000 mon.a (mon.0) 2021-07-31T02:00:00.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:00 smithi045 conmon[33655]: 1569 : audit [INF] from='client.? 172.21.15.45:0/2290598225' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-13","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:00:00.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:00 smithi045 conmon[33655]: cluster 2021-07-31T01:59:59.294626+0000 mon.a (mon.0) 1570 : cluster [DBG] osdmap e866: 8 total, 8 up, 8 in 2021-07-31T02:00:00.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:00 smithi045 conmon[33655]: cluster 2021-07-31T02:00:00.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:00 smithi045 conmon[33655]: 2021-07-31T01:59:59.976131+0000 mgr.x (mgr.24542) 547 : cluster [DBG] pgmap v1096: 41 pgs: 32 creating+peering, 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2021-07-31T02:00:00.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:00 smithi045 conmon[33655]: cluster 2021-07-31T02:00:00.000137+0000 mon.a ( 2021-07-31T02:00:00.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:00 smithi045 conmon[33655]: mon.0) 1571 : cluster [INF] overall HEALTH_OK 2021-07-31T02:00:00.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:00 smithi045 conmon[39718]: audit 2021-07-31T01:59:59 2021-07-31T02:00:00.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:00 smithi045 conmon[39718]: .294359+0000 mon.a (mon.0) 1569 : audit [INF] from='client.? 172.21.15.45:0/2290598225' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-13","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:00:00.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:00 smithi045 conmon[39718]: cluster 2021 2021-07-31T02:00:00.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:00 smithi045 conmon[39718]: -07-31T01:59:59.294626+0000 mon.a (mon 2021-07-31T02:00:00.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:00 smithi045 conmon[39718]: .0) 1570 : cluster [DBG] osdmap e866: 8 total, 8 up, 8 in 2021-07-31T02:00:00.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:00 smithi045 conmon[39718]: cluster 2021-07- 2021-07-31T02:00:00.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:00 smithi045 conmon[39718]: 31T01:59:59.976131+0000 mgr.x (mgr.24542) 2021-07-31T02:00:00.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:00 smithi045 conmon[39718]: 547 : cluster [DBG] pgmap v1096: 41 pgs: 32 creating+peering, 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2021-07-31T02:00:00.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:00 smithi045 conmon[39718]: cluster 2021-07-31T02:00: 2021-07-31T02:00:00.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:00 smithi045 conmon[39718]: 00.000137+0000 mon.a (mon.0) 1571 : cluster [INF] overall HEALTH_OK 2021-07-31T02:00:01.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:01 smithi154 conmon[34553]: cluster 2021-07-31T02:00:00.299923+0000 mon.a (mon.0) 1572 : cluster [DBG] osdmap e867: 8 total, 8 up, 8 in 2021-07-31T02:00:01.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:01 smithi045 conmon[39718]: cluster 2021-07-31T02:00:00.299923+0000 2021-07-31T02:00:01.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:01 smithi045 conmon[39718]: mon.a (mon.0) 1572 : cluster [DBG] osdmap e867: 8 total, 8 up, 8 in 2021-07-31T02:00:01.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:01 smithi045 conmon[33655]: cluster 2021-07-31T02:00:00.299923+0000 2021-07-31T02:00:01.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:01 smithi045 conmon[33655]: mon.a (mon.0) 1572 : cluster [DBG] osdmap e867: 8 total, 8 up, 8 in 2021-07-31T02:00:02.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:02 smithi154 conmon[34553]: cluster 2021-07-31T02:00:02.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:02 smithi154 conmon[34553]: 2021- 2021-07-31T02:00:02.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:02 smithi154 conmon[34553]: 07-31T02:00:01.305318+0000 mon.a (mon.0) 1573 : cluster [DBG] osdmap e868: 8 total, 8 up, 8 in 2021-07-31T02:00:02.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:02 smithi154 conmon[34553]: audit 2021-07-31T02 2021-07-31T02:00:02.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:02 smithi154 conmon[34553]: :00:01.306096+0000 mon.a (mon.0) 1574 : audit [INF] from='client.? 172.21.15.45:0/304979126' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:00:02.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:02 smithi154 conmon[34553]: 2021-07-31T02:00:02.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:02 smithi154 conmon[34553]: cluster 2021-07-31T02:00:01.976575 2021-07-31T02:00:02.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:02 smithi154 conmon[34553]: +0000 mgr.x (mgr.24542) 548 : cluster [DBG] 2021-07-31T02:00:02.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:02 smithi154 conmon[34553]: pgmap v1099: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2021-07-31T02:00:02.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:02 smithi045 conmon[39718]: cluster 2021-07-31T02:00:02.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:02 smithi045 conmon[39718]: 2021-07-31T02:00:02.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:02 smithi045 conmon[39718]: 2021-07-31T02:00:01.305318 2021-07-31T02:00:02.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:02 smithi045 conmon[39718]: +0000 mon.a (mon.0) 1573 : cluster [DBG] osdmap e868: 8 total, 8 up, 8 in 2021-07-31T02:00:02.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:02 smithi045 conmon[39718]: audit 2021-07-31T02:00:02.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:02 smithi045 conmon[39718]: 2021-07-31T02:00:01.306096+0000 mon.a (mon.0) 1574 : audit [INF] 2021-07-31T02:00:02.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:02 smithi045 conmon[39718]: from='client.? 172.21.15.45:0/304979126' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:00:02.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:02 smithi045 conmon[39718]: cluster 2021-07-31T02:00:02.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:02 smithi045 conmon[39718]: 2021-07-31T02:00:01.976575+0000 mgr.x (mgr.24542) 2021-07-31T02:00:02.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:02 smithi045 conmon[39718]: 548 : cluster [DBG] pgmap v1099: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2021-07-31T02:00:02.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:02 smithi045 conmon[33655]: cluster 2021-07-31T02:00:02.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:02 smithi045 conmon[33655]: 2021-07-31T02:00:02.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:02 smithi045 conmon[33655]: 2021 2021-07-31T02:00:02.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:02 smithi045 conmon[33655]: - 2021-07-31T02:00:02.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:02 smithi045 conmon[33655]: 07 2021-07-31T02:00:02.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:02 smithi045 conmon[33655]: - 2021-07-31T02:00:02.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:02 smithi045 conmon[33655]: 31 2021-07-31T02:00:02.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:02 smithi045 conmon[33655]: T 2021-07-31T02:00:02.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:02 smithi045 conmon[33655]: 02 2021-07-31T02:00:02.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:02 smithi045 conmon[33655]: : 2021-07-31T02:00:02.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:02 smithi045 conmon[33655]: 00 2021-07-31T02:00:02.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:02 smithi045 conmon[33655]: : 2021-07-31T02:00:02.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:02 smithi045 conmon[33655]: 01 2021-07-31T02:00:02.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:02 smithi045 conmon[33655]: . 2021-07-31T02:00:02.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:02 smithi045 conmon[33655]: 305318 2021-07-31T02:00:02.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:02 smithi045 conmon[33655]: +0000 2021-07-31T02:00:02.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:02 smithi045 conmon[33655]: 2021-07-31T02:00:02.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:02 smithi045 conmon[33655]: mon.a 2021-07-31T02:00:02.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:02 smithi045 conmon[33655]: ( 2021-07-31T02:00:02.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:02 smithi045 conmon[33655]: mon 2021-07-31T02:00:02.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:02 smithi045 conmon[33655]: . 2021-07-31T02:00:02.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:02 smithi045 conmon[33655]: 0 2021-07-31T02:00:02.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:02 smithi045 conmon[33655]: ) 2021-07-31T02:00:02.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:02 smithi045 conmon[33655]: 1573 2021-07-31T02:00:02.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:02 smithi045 conmon[33655]: : 2021-07-31T02:00:02.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:02 smithi045 conmon[33655]: cluster 2021-07-31T02:00:02.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:02 smithi045 conmon[33655]: 2021-07-31T02:00:02.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:02 smithi045 conmon[33655]: [DBG] 2021-07-31T02:00:02.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:02 smithi045 conmon[33655]: 2021-07-31T02:00:02.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:02 smithi045 conmon[33655]: osdmap e868: 8 total, 8 up, 8 in 2021-07-31T02:00:02.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:02 smithi045 conmon[33655]: 2021-07-31T02:00:02.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:02 smithi045 conmon[33655]: audit 2021-07-31T02:00:01.306096+0000 mon.a (mon.0) 1574 : audit 2021-07-31T02:00:02.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:02 smithi045 conmon[33655]: [INF] from='client.? 172.21.15.45:0/304979126' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:00:02.788 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:02 smithi045 conmon[33655]: cluster 2021-07-31T02:00:01.976575 2021-07-31T02:00:02.788 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:02 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 548 : cluster [DBG] pgmap v1099: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2021-07-31T02:00:03.325 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestCls2PCQueue.ReserveAbort (3021 ms) 2021-07-31T02:00:03.325 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestCls2PCQueue.ManualCleanup 2021-07-31T02:00:03.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:03 smithi154 conmon[34553]: audit 2021-07-31T02:00:02.308600+0000 mon.a (mon.0) 1575 : 2021-07-31T02:00:03.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:03 smithi154 conmon[34553]: audit [INF] from='client.? 172.21.15.45:0/304979126' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-14","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:00:03.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:03 smithi154 conmon[34553]: cluster 2021-07-31T02:00:02.308913+0000 mon.a (mon.0) 1576 : cluster [DBG] osdmap e869: 8 total, 8 up, 8 in 2021-07-31T02:00:03.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:00:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:00:03.531Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:00:03.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:00:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:00:03.531Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:00:03.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:00:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:00:03.531Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:00:03.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:03 smithi045 conmon[33655]: audit 2021-07-31T02:00:02 2021-07-31T02:00:03.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:03 smithi045 conmon[33655]: .308600+0000 mon.a (mon.0) 1575 : audit [INF] from='client.? 172.21.15.45:0/304979126' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-14","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:00:03.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:03 smithi045 conmon[33655]: cluster 2021-07-31T02:00:03.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:03 smithi045 conmon[33655]: 2021-07-31T02:00:02.308913 2021-07-31T02:00:03.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:03 smithi045 conmon[33655]: +0000 mon.a (mon.0 2021-07-31T02:00:03.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:03 smithi045 conmon[33655]: ) 1576 : cluster [DBG] 2021-07-31T02:00:03.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:03 smithi045 conmon[33655]: osdmap e869: 8 total, 8 up, 8 in 2021-07-31T02:00:03.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:03 smithi045 conmon[39718]: audit 2021-07-31T02:00: 2021-07-31T02:00:03.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:03 smithi045 conmon[39718]: 02.308600+0000 mon.a (mon.0) 1575 : audit [INF] from='client.? 172.21.15.45:0/304979126' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-14","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:00:03.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:03 smithi045 conmon[39718]: cluster 2021-07 2021-07-31T02:00:03.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:03 smithi045 conmon[39718]: -31T02:00:02.308913+0000 mon.a (mon.0 2021-07-31T02:00:03.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:03 smithi045 conmon[39718]: ) 1576 : cluster [DBG] osdmap e869: 8 total, 8 up, 8 in 2021-07-31T02:00:03.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:03 smithi045 conmon[39718]: 2021-07-31T02:00:04.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:04 smithi154 conmon[34553]: cluster 2021-07-31T02:00:03.323586+0000 mon.a (mon.0) 1577 : cluster 2021-07-31T02:00:04.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:04 smithi154 conmon[34553]: [DBG] osdmap e870: 8 total, 8 up, 8 in 2021-07-31T02:00:04.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:04 smithi154 conmon[34553]: cluster 2021-07-31T02:00:03.977089+0000 mgr.x (mgr.24542) 549 : cluster [DBG] pgmap v1102: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2021-07-31T02:00:04.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:04 smithi045 conmon[33655]: cluster 2021-07-31T02:00:03.323586+0000 mon.a 2021-07-31T02:00:04.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:04 smithi045 conmon[33655]: (mon.0) 1577 : cluster [DBG] osdmap e870: 8 total, 8 up, 8 in 2021-07-31T02:00:04.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:04 smithi045 conmon[33655]: cluster 2021-07-31T02:00:03.977089+0000 mgr.x (mgr.24542) 549 : cluster [DBG] pgmap v1102: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2021-07-31T02:00:04.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:04 smithi045 conmon[39718]: cluster 2021-07-31T02:00: 2021-07-31T02:00:04.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:04 smithi045 conmon[39718]: 03.323586+0000 mon.a (mon.0) 1577 : cluster [DBG] osdmap e870: 8 total, 8 up, 8 in 2021-07-31T02:00:04.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:04 smithi045 conmon[39718]: cluster 2021-07 2021-07-31T02:00:04.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:04 smithi045 conmon[39718]: -31T02:00:03.977089+0000 mgr.x (mgr.24542) 549 : 2021-07-31T02:00:04.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:04 smithi045 conmon[39718]: cluster [DBG] pgmap v1102: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2021-07-31T02:00:05.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:05 smithi154 conmon[34553]: cluster 2021-07-31T02:00:05.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:05 smithi154 conmon[34553]: 2021-07-31T02:00:05.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:05 smithi154 conmon[34553]: 2021-07-31T02:00 2021-07-31T02:00:05.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:05 smithi154 conmon[34553]: :04.329032+0000 mon.a (mon.0) 1578 : cluster [DBG] osdmap e871: 8 total, 8 up, 8 in 2021-07-31T02:00:05.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:05 smithi154 conmon[34553]: audit 2021-07-31T02:00:05.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:05 smithi154 conmon[34553]: 2021-07-31T02: 2021-07-31T02:00:05.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:05 smithi154 conmon[34553]: 00:04.330662+0000 mon.a (mon.0) 1579 2021-07-31T02:00:05.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:05 smithi154 conmon[34553]: : audit [INF] from='client.? 172.21.15.45:0/2536642502' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:00:05.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[39718]: cluster 2021-07-31T02:00:05.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[39718]: 2021-07-31T02:00:05.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[39718]: 2021 2021-07-31T02:00:05.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[39718]: - 2021-07-31T02:00:05.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[39718]: 07 2021-07-31T02:00:05.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[39718]: - 2021-07-31T02:00:05.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[39718]: 31 2021-07-31T02:00:05.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[39718]: T 2021-07-31T02:00:05.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[39718]: 02 2021-07-31T02:00:05.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[39718]: : 2021-07-31T02:00:05.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[39718]: 00 2021-07-31T02:00:05.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[39718]: : 2021-07-31T02:00:05.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[39718]: 04 2021-07-31T02:00:05.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[39718]: . 2021-07-31T02:00:05.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[39718]: 329032 2021-07-31T02:00:05.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[39718]: +0000 2021-07-31T02:00:05.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[39718]: 2021-07-31T02:00:05.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[39718]: mon.a 2021-07-31T02:00:05.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[39718]: ( 2021-07-31T02:00:05.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[39718]: mon 2021-07-31T02:00:05.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[39718]: . 2021-07-31T02:00:05.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[39718]: 0 2021-07-31T02:00:05.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[39718]: ) 2021-07-31T02:00:05.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[39718]: 1578 2021-07-31T02:00:05.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[39718]: : 2021-07-31T02:00:05.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[39718]: cluster 2021-07-31T02:00:05.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[39718]: 2021-07-31T02:00:05.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[39718]: [DBG] 2021-07-31T02:00:05.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[39718]: 2021-07-31T02:00:05.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[39718]: osdmap e871: 8 total, 8 up, 8 in 2021-07-31T02:00:05.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[39718]: 2021-07-31T02:00:05.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[39718]: audit 2021-07-31T02:00:05.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[39718]: 2021-07-31T02:00:05.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[39718]: 2021 2021-07-31T02:00:05.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[39718]: - 2021-07-31T02:00:05.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[39718]: 07-31T02:00:04.330662+0000 mon.a ( 2021-07-31T02:00:05.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[39718]: mon.0) 1579 : audit [INF] from='client.? 172.21.15.45:0/2536642502' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:00:05.788 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[33655]: cluster 2021-07-31T02:00:05.788 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[33655]: 2021-07-31T02:00:05.788 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[33655]: 2021 2021-07-31T02:00:05.788 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[33655]: - 2021-07-31T02:00:05.789 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[33655]: 07 2021-07-31T02:00:05.789 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[33655]: - 2021-07-31T02:00:05.789 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[33655]: 31 2021-07-31T02:00:05.789 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[33655]: T 2021-07-31T02:00:05.790 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[33655]: 02 2021-07-31T02:00:05.790 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[33655]: : 2021-07-31T02:00:05.790 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[33655]: 00 2021-07-31T02:00:05.790 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[33655]: : 2021-07-31T02:00:05.791 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[33655]: 04 2021-07-31T02:00:05.791 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[33655]: . 2021-07-31T02:00:05.791 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[33655]: 329032 2021-07-31T02:00:05.791 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[33655]: +0000 2021-07-31T02:00:05.792 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[33655]: 2021-07-31T02:00:05.792 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[33655]: mon.a 2021-07-31T02:00:05.792 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[33655]: ( 2021-07-31T02:00:05.792 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[33655]: mon 2021-07-31T02:00:05.792 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[33655]: . 2021-07-31T02:00:05.793 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[33655]: 0 2021-07-31T02:00:05.793 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[33655]: ) 2021-07-31T02:00:05.793 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[33655]: 1578 2021-07-31T02:00:05.793 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[33655]: : 2021-07-31T02:00:05.794 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[33655]: cluster 2021-07-31T02:00:05.794 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[33655]: 2021-07-31T02:00:05.795 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[33655]: [DBG] 2021-07-31T02:00:05.795 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[33655]: 2021-07-31T02:00:05.795 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[33655]: osdmap e871: 8 total, 8 up, 8 in 2021-07-31T02:00:05.795 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[33655]: 2021-07-31T02:00:05.796 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[33655]: audit 2021-07-31T02:00:05.796 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[33655]: 2021-07-31T02:00:05.796 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[33655]: 2021 2021-07-31T02:00:05.797 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[33655]: - 2021-07-31T02:00:05.797 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[33655]: 07 2021-07-31T02:00:05.797 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[33655]: - 2021-07-31T02:00:05.798 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[33655]: 31 2021-07-31T02:00:05.798 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[33655]: T 2021-07-31T02:00:05.798 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[33655]: 02 2021-07-31T02:00:05.798 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[33655]: : 2021-07-31T02:00:05.799 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[33655]: 00 2021-07-31T02:00:05.799 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[33655]: : 2021-07-31T02:00:05.799 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[33655]: 04 2021-07-31T02:00:05.799 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[33655]: . 2021-07-31T02:00:05.800 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[33655]: 330662 2021-07-31T02:00:05.800 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[33655]: +0000 2021-07-31T02:00:05.800 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[33655]: 2021-07-31T02:00:05.800 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[33655]: mon.a 2021-07-31T02:00:05.800 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[33655]: ( 2021-07-31T02:00:05.801 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[33655]: mon 2021-07-31T02:00:05.801 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[33655]: . 2021-07-31T02:00:05.801 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[33655]: 0 2021-07-31T02:00:05.801 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[33655]: ) 2021-07-31T02:00:05.802 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[33655]: 1579 2021-07-31T02:00:05.802 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[33655]: : 2021-07-31T02:00:05.802 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[33655]: audit 2021-07-31T02:00:05.802 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[33655]: 2021-07-31T02:00:05.803 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[33655]: [INF] 2021-07-31T02:00:05.803 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[33655]: 2021-07-31T02:00:05.803 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[33655]: from='client.? 172.21.15.45:0/2536642502' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:00:05.803 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:05 smithi045 conmon[33655]: 2021-07-31T02:00:06.350 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestCls2PCQueue.ManualCleanup (3023 ms) 2021-07-31T02:00:06.351 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestCls2PCQueue.Cleanup 2021-07-31T02:00:06.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:06 smithi154 conmon[34553]: audit 2021-07-31T02:00:05.331565+0000 mon.a (mon 2021-07-31T02:00:06.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:06 smithi154 conmon[34553]: .0) 1580 : audit [INF] from='client.? 172.21.15.45:0/2536642502' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-15","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:00:06.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:06 smithi154 conmon[34553]: cluster 2021-07-31T02:00:05.331844+0000 mon.a (mon.0) 1581 : cluster [DBG] osdmap e872: 8 total, 8 up, 8 in 2021-07-31T02:00:06.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:06 smithi154 conmon[34553]: cluster 2021-07-31T02:00:06.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:06 smithi154 conmon[34553]: 2021-07-31T02:00:05.977440+0000 mgr.x (mgr.24542) 550 : cluster [DBG] pgmap v1105: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2021-07-31T02:00:06.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:06 smithi045 conmon[39718]: audit 2021-07-31T02:00:05.331565+0000 mon.a 2021-07-31T02:00:06.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:06 smithi045 conmon[39718]: (mon.0) 1580 : audit [INF] from='client.? 172.21.15.45:0/2536642502' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-15","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:00:06.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:06 smithi045 conmon[39718]: cluster 2021-07-31T02:00:05 2021-07-31T02:00:06.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:06 smithi045 conmon[39718]: .331844+0000 mon.a (mon.0) 1581 : cluster [DBG] osdmap e872: 8 total, 8 up, 8 in 2021-07-31T02:00:06.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:06 smithi045 conmon[39718]: cluster 2021-07-31T02:00:06.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:06 smithi045 conmon[39718]: 2021-07-31T 2021-07-31T02:00:06.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:06 smithi045 conmon[39718]: 02:00:05.977440+0000 mgr.x (mgr.24542 2021-07-31T02:00:06.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:06 smithi045 conmon[39718]: ) 550 : cluster [DBG] pgmap v1105: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2021-07-31T02:00:06.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:06 smithi045 conmon[33655]: audit 2021-07-31T02:00:05.331565+0000 2021-07-31T02:00:06.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:06 smithi045 conmon[33655]: mon.a (mon.0) 1580 : audit [INF] from='client.? 172.21.15.45:0/2536642502' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-15","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:00:06.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:06 smithi045 conmon[33655]: cluster 2021-07-31T02:00:05.331844+0000 mon.a (mon.0) 1581 2021-07-31T02:00:06.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:06 smithi045 conmon[33655]: : cluster [DBG] osdmap e872: 8 total, 8 up, 8 in 2021-07-31T02:00:06.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:06 smithi045 conmon[33655]: cluster 2021-07-31T02:00:05.977440+0000 2021-07-31T02:00:06.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:06 smithi045 conmon[33655]: mgr.x (mgr.24542) 550 : cluster [DBG] pgmap v1105: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2021-07-31T02:00:07.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:07 smithi154 conmon[34553]: cluster 2021-07-31T02:00:06.346141+0000 mon.a (mon.0) 1582 : cluster [DBG] osdmap e873: 8 total, 8 up, 8 in 2021-07-31T02:00:07.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:07 smithi045 conmon[39718]: cluster 2021-07-31T02:00:06.346141+0000 mon.a 2021-07-31T02:00:07.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:07 smithi045 conmon[39718]: (mon.0) 1582 : cluster [DBG] osdmap e873: 8 total, 8 up, 8 in 2021-07-31T02:00:07.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:07 smithi045 conmon[33655]: cluster 2021-07-31T02:00:06 2021-07-31T02:00:07.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:07 smithi045 conmon[33655]: .346141+0000 mon.a (mon.0) 1582 : cluster [DBG] osdmap e873: 8 total, 8 up, 8 in 2021-07-31T02:00:08.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:08 smithi154 conmon[34553]: cluster 2021-07-31T02:00:08.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:08 smithi154 conmon[34553]: 2021-07-31T02:00:07.351388+0000 mon.a (mon.0) 1583 : cluster [DBG] osdmap e874: 8 total, 8 up, 8 in 2021-07-31T02:00:08.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:08 smithi154 conmon[34553]: audit 2021-07-31T02:00:07.354029+0000 mon.c (mon.1) 43 : audit [INF] from='client.? 172.21.15.45:0/1259056224' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:00:08.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:08 smithi154 conmon[34553]: audit 2021-07-31T02:00:07.355881+0000 mon.a ( 2021-07-31T02:00:08.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:08 smithi154 conmon[34553]: mon.0) 1584 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:00:08.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:08 smithi154 conmon[34553]: cluster 2021-07 2021-07-31T02:00:08.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:08 smithi154 conmon[34553]: -31T02:00:07.977994+0000 mgr.x (mgr. 2021-07-31T02:00:08.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:08 smithi154 conmon[34553]: 24542) 551 : cluster [DBG] pgmap v1108: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2021-07-31T02:00:08.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:08 smithi045 conmon[39718]: cluster 2021-07-31T02:00:07.351388+0000 mon.a (mon. 2021-07-31T02:00:08.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:08 smithi045 conmon[39718]: 0) 1583 : cluster [DBG] osdmap e874: 8 total, 8 up, 8 in 2021-07-31T02:00:08.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:08 smithi045 conmon[39718]: audit 2021-07-31T02:00:07.354029+0000 mon.c (mon.1) 43 2021-07-31T02:00:08.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:08 smithi045 conmon[39718]: : audit [INF] from='client.? 172.21.15.45:0/1259056224' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:00:08.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:08 smithi045 conmon[39718]: audit 2021-07-31T02:00:07.355881+0000 mon.a (mon 2021-07-31T02:00:08.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:08 smithi045 conmon[39718]: .0) 1584 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:00:08.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:08 smithi045 conmon[39718]: cluster 2021-07-31T02:00:08.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:08 smithi045 conmon[39718]: 2021-07-31T02:00:07.977994+0000 mgr.x ( 2021-07-31T02:00:08.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:08 smithi045 conmon[39718]: mgr.24542) 551 : cluster [DBG] pgmap v1108: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2021-07-31T02:00:08.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:08 smithi045 conmon[33655]: cluster 2021-07-31T02:00:07.351388+0000 mon.a (mon.0) 1583 2021-07-31T02:00:08.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:08 smithi045 conmon[33655]: : cluster [DBG] osdmap e874: 8 total, 8 up, 8 in 2021-07-31T02:00:08.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:08 smithi045 conmon[33655]: audit 2021-07-31T02:00:07.354029+0000 mon.c (mon.1) 43 2021-07-31T02:00:08.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:08 smithi045 conmon[33655]: : audit [INF] from='client.? 172.21.15.45:0/1259056224' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:00:08.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:08 smithi045 conmon[33655]: audit 2021-07-31T02:00:07.355881+0000 mon.a 2021-07-31T02:00:08.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:08 smithi045 conmon[33655]: (mon.0) 1584 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:00:08.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:08 smithi045 conmon[33655]: cluster 2021-07-31T02:00:07 2021-07-31T02:00:08.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:08 smithi045 conmon[33655]: .977994+0000 mgr.x (mgr.24542) 551 : cluster [DBG] pgmap v1108: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2021-07-31T02:00:09.362 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestCls2PCQueue.Cleanup (3014 ms) 2021-07-31T02:00:09.363 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestCls2PCQueue.MultiProducer 2021-07-31T02:00:09.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:09 smithi154 conmon[34553]: audit 2021-07-31T02:00:08.359073+0000 mon.a 2021-07-31T02:00:09.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:09 smithi154 conmon[34553]: (mon.0) 1585 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-16","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:00:09.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:09 smithi154 conmon[34553]: cluster 2021-07-31T02:00:08.359278+0000 mon.a (mon.0) 1586 : cluster [DBG] osdmap e875: 8 total, 8 up, 8 in 2021-07-31T02:00:09.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:09 smithi045 conmon[39718]: audit 2021-07-31T02:00:08.359073+0000 mon.a 2021-07-31T02:00:09.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:09 smithi045 conmon[39718]: (mon.0) 1585 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-16","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:00:09.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:09 smithi045 conmon[39718]: cluster 2021-07-31T02:00:08 2021-07-31T02:00:09.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:09 smithi045 conmon[39718]: .359278+0000 mon.a (mon.0) 1586 : cluster [DBG] osdmap e875: 8 total, 8 up, 8 in 2021-07-31T02:00:09.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:09 smithi045 conmon[33655]: audit 2021-07-31T02:00:09.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:09 smithi045 conmon[33655]: 2021-07-31T02:00:09.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:09 smithi045 conmon[33655]: 2021-07-31T02:00:08.359073+0000 2021-07-31T02:00:09.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:09 smithi045 conmon[33655]: mon.a (mon.0) 1585 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-16","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:00:09.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:09 smithi045 conmon[33655]: cluster 2021-07-31T02:00:09.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:09 smithi045 conmon[33655]: 2021-07-31T02:00:08.359278+0000 mon.a (mon.0) 1586 : cluster [DBG] osdmap e875: 8 total, 8 up, 8 in 2021-07-31T02:00:10.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:10 smithi154 conmon[34553]: cluster 2021-07-31T02:00:09.359477+0000 mon.a (mon.0) 1587 : cluster 2021-07-31T02:00:10.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:10 smithi154 conmon[34553]: [DBG] osdmap e876: 8 total, 8 up, 8 in 2021-07-31T02:00:10.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:10 smithi154 conmon[34553]: cluster 2021-07-31T02:00:09.978581+0000 mgr.x (mgr.24542) 552 : cluster [DBG] pgmap v1111: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2021-07-31T02:00:10.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:10 smithi045 conmon[39718]: cluster 2021-07-31T02:00:09.359477+0000 mon.a (mon.0) 2021-07-31T02:00:10.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:10 smithi045 conmon[39718]: 1587 : cluster [DBG] osdmap e876: 8 total, 8 up, 8 in 2021-07-31T02:00:10.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:10 smithi045 conmon[39718]: cluster 2021-07-31T02:00:09.978581+0000 mgr.x (mgr.24542) 552 : cluster [DBG] pgmap v1111: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2021-07-31T02:00:10.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:10 smithi045 conmon[33655]: cluster 2021-07-31T02:00:09 2021-07-31T02:00:10.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:10 smithi045 conmon[33655]: .359477+0000 mon.a (mon.0) 1587 : cluster [DBG] osdmap e876: 8 total, 8 up, 8 in 2021-07-31T02:00:10.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:10 smithi045 conmon[33655]: cluster 2021-07-31T02:00:09.978581+0000 mgr.x (mgr.24542) 552 : cluster [DBG] pgmap v1111: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2021-07-31T02:00:11.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:11 smithi154 conmon[34553]: cluster 2021-07-31T02:00:11.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:11 smithi154 conmon[34553]: 2021-07-31T02:00:11.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:11 smithi154 conmon[34553]: 2021 2021-07-31T02:00:11.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:11 smithi154 conmon[34553]: - 2021-07-31T02:00:11.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:11 smithi154 conmon[34553]: 07 2021-07-31T02:00:11.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:11 smithi154 conmon[34553]: - 2021-07-31T02:00:11.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:11 smithi154 conmon[34553]: 31 2021-07-31T02:00:11.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:11 smithi154 conmon[34553]: T 2021-07-31T02:00:11.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:11 smithi154 conmon[34553]: 02 2021-07-31T02:00:11.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:11 smithi154 conmon[34553]: : 2021-07-31T02:00:11.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:11 smithi154 conmon[34553]: 00 2021-07-31T02:00:11.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:11 smithi154 conmon[34553]: : 2021-07-31T02:00:11.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:11 smithi154 conmon[34553]: 10 2021-07-31T02:00:11.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:11 smithi154 conmon[34553]: . 2021-07-31T02:00:11.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:11 smithi154 conmon[34553]: 370732 2021-07-31T02:00:11.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:11 smithi154 conmon[34553]: +0000 2021-07-31T02:00:11.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:11 smithi154 conmon[34553]: 2021-07-31T02:00:11.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:11 smithi154 conmon[34553]: mon.a 2021-07-31T02:00:11.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:11 smithi154 conmon[34553]: ( 2021-07-31T02:00:11.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:11 smithi154 conmon[34553]: mon 2021-07-31T02:00:11.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:11 smithi154 conmon[34553]: .0) 1588 : cluster [DBG] osdmap e877: 8 total, 8 up, 8 in 2021-07-31T02:00:11.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:11 smithi154 conmon[34553]: audit 2021-07-31T02:00:11.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:11 smithi154 conmon[34553]: 2021-07-31T02 2021-07-31T02:00:11.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:11 smithi154 conmon[34553]: :00:10.372106+0000 mon.a (mon.0) 1589 : audit 2021-07-31T02:00:11.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:11 smithi154 conmon[34553]: [INF] from='client.? 172.21.15.45:0/2453478425' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:00:11.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:11 smithi045 conmon[33655]: cluster 2021-07-31T02:00:11.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:11 smithi045 conmon[33655]: 2021-07-31T02:00:11.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:11 smithi045 conmon[33655]: 2021 2021-07-31T02:00:11.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:11 smithi045 conmon[33655]: - 2021-07-31T02:00:11.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:11 smithi045 conmon[33655]: 07 2021-07-31T02:00:11.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:11 smithi045 conmon[33655]: - 2021-07-31T02:00:11.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:11 smithi045 conmon[33655]: 31 2021-07-31T02:00:11.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:11 smithi045 conmon[33655]: T 2021-07-31T02:00:11.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:11 smithi045 conmon[33655]: 02 2021-07-31T02:00:11.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:11 smithi045 conmon[33655]: : 2021-07-31T02:00:11.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:11 smithi045 conmon[33655]: 00 2021-07-31T02:00:11.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:11 smithi045 conmon[33655]: : 2021-07-31T02:00:11.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:11 smithi045 conmon[33655]: 10 2021-07-31T02:00:11.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:11 smithi045 conmon[33655]: . 2021-07-31T02:00:11.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:11 smithi045 conmon[33655]: 370732 2021-07-31T02:00:11.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:11 smithi045 conmon[33655]: +0000 2021-07-31T02:00:11.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:11 smithi045 conmon[33655]: 2021-07-31T02:00:11.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:11 smithi045 conmon[33655]: mon.a 2021-07-31T02:00:11.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:11 smithi045 conmon[33655]: ( 2021-07-31T02:00:11.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:11 smithi045 conmon[33655]: mon 2021-07-31T02:00:11.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:11 smithi045 conmon[33655]: . 2021-07-31T02:00:11.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:11 smithi045 conmon[33655]: 0) 1588 : cluster [DBG] osdmap e877: 8 total, 8 up, 8 in 2021-07-31T02:00:11.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:11 smithi045 conmon[33655]: 2021-07-31T02:00:11.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:11 smithi045 conmon[33655]: audit 2021-07-31T02:00:10.372106+0000 2021-07-31T02:00:11.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:11 smithi045 conmon[33655]: mon.a (mon.0) 1589 : audit [INF] from='client.? 172.21.15.45:0/2453478425' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:00:11.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:11 smithi045 conmon[33655]: 2021-07-31T02:00:11.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:11 smithi045 conmon[39718]: cluster 2021-07-31T02:00:11.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:11 smithi045 conmon[39718]: 2021-07-31T02:00:11.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:11 smithi045 conmon[39718]: 2021 2021-07-31T02:00:11.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:11 smithi045 conmon[39718]: - 2021-07-31T02:00:11.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:11 smithi045 conmon[39718]: 07 2021-07-31T02:00:11.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:11 smithi045 conmon[39718]: - 2021-07-31T02:00:11.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:11 smithi045 conmon[39718]: 31T02:00:10.370732+0000 mon.a ( 2021-07-31T02:00:11.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:11 smithi045 conmon[39718]: mon.0) 1588 : cluster [DBG] osdmap e877: 8 total, 8 up, 8 in 2021-07-31T02:00:11.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:11 smithi045 conmon[39718]: audit 2021-07-31T02:00:11.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:11 smithi045 conmon[39718]: 2021-07-31T02:00:11.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:11 smithi045 conmon[39718]: 2021 2021-07-31T02:00:11.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:11 smithi045 conmon[39718]: - 2021-07-31T02:00:11.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:11 smithi045 conmon[39718]: 07 2021-07-31T02:00:11.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:11 smithi045 conmon[39718]: - 2021-07-31T02:00:11.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:11 smithi045 conmon[39718]: 31 2021-07-31T02:00:11.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:11 smithi045 conmon[39718]: T 2021-07-31T02:00:11.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:11 smithi045 conmon[39718]: 02 2021-07-31T02:00:11.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:11 smithi045 conmon[39718]: : 2021-07-31T02:00:11.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:11 smithi045 conmon[39718]: 00 2021-07-31T02:00:11.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:11 smithi045 conmon[39718]: : 2021-07-31T02:00:11.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:11 smithi045 conmon[39718]: 10 2021-07-31T02:00:11.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:11 smithi045 conmon[39718]: . 2021-07-31T02:00:11.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:11 smithi045 conmon[39718]: 372106 2021-07-31T02:00:11.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:11 smithi045 conmon[39718]: +0000 2021-07-31T02:00:11.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:11 smithi045 conmon[39718]: 2021-07-31T02:00:11.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:11 smithi045 conmon[39718]: mon.a 2021-07-31T02:00:11.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:11 smithi045 conmon[39718]: ( 2021-07-31T02:00:11.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:11 smithi045 conmon[39718]: mon 2021-07-31T02:00:11.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:11 smithi045 conmon[39718]: .0) 1589 : audit [INF] from='client.? 172.21.15.45:0/2453478425' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:00:12.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:12 smithi154 conmon[34553]: audit 2021-07-31T02:00:11.372445+0000 mon.a (mon 2021-07-31T02:00:12.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:12 smithi154 conmon[34553]: .0) 1590 : audit [INF] from='client.? 172.21.15.45:0/2453478425' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-17","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:00:12.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:12 smithi154 conmon[34553]: cluster 2021-07-31T02:00:11.372657+0000 mon.a (mon.0) 1591 : cluster [DBG] osdmap e878: 8 total, 8 up, 8 in 2021-07-31T02:00:12.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:12 smithi154 conmon[34553]: cluster 2021-07-31T02:00:11.978974+0000 mgr.x (mgr.24542) 553 : cluster 2021-07-31T02:00:12.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:12 smithi154 conmon[34553]: [DBG] pgmap v1114: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2021-07-31T02:00:12.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:12 smithi045 conmon[33655]: audit 2021-07-31T02:00:11.372445+0000 mon.a (mon.0) 1590 2021-07-31T02:00:12.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:12 smithi045 conmon[33655]: : audit [INF] from='client.? 172.21.15.45:0/2453478425' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-17","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:00:12.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:12 smithi045 conmon[33655]: cluster 2021-07-31T02:00:11.372657+0000 mon.a (mon.0) 1591 : cluster [DBG] osdmap e878: 8 total, 8 up, 8 in 2021-07-31T02:00:12.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:12 smithi045 conmon[33655]: cluster 2021-07-31T02:00:12.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:12 smithi045 conmon[33655]: 2021-07-31T02:00:11.978974+0000 mgr.x (mgr.24542) 553 : cluster [DBG] pgmap v1114: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2021-07-31T02:00:12.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:12 smithi045 conmon[39718]: audit 2021-07-31T02:00:11.372445+0000 2021-07-31T02:00:12.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:12 smithi045 conmon[39718]: mon.a (mon.0) 1590 : audit [INF] from='client.? 172.21.15.45:0/2453478425' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-17","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:00:12.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:12 smithi045 conmon[39718]: cluster 2021-07-31T 2021-07-31T02:00:12.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:12 smithi045 conmon[39718]: 02:00:11.372657+0000 mon.a (mon.0) 1591 : cluster [DBG] osdmap e878: 8 total, 8 up, 8 in 2021-07-31T02:00:12.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:12 smithi045 conmon[39718]: cluster 2021-07-31T02:00:12.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:12 smithi045 conmon[39718]: 2021-07-31T02:00:11.978974 2021-07-31T02:00:12.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:12 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 553 : cluster [DBG] 2021-07-31T02:00:12.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:12 smithi045 conmon[39718]: pgmap v1114: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2021-07-31T02:00:13.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:13 smithi154 conmon[34553]: cluster 2021-07-31T02:00:12.380798+0000 mon.a (mon.0 2021-07-31T02:00:13.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:13 smithi154 conmon[34553]: ) 1592 : cluster [DBG] osdmap e879: 8 total, 8 up, 8 in 2021-07-31T02:00:13.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:13 smithi045 conmon[39718]: cluster 2021-07-31T02:00:12.380798+0000 mon.a 2021-07-31T02:00:13.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:13 smithi045 conmon[39718]: (mon.0) 1592 : cluster [DBG] osdmap e879: 8 total, 8 up, 8 in 2021-07-31T02:00:13.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:00:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:00:13.531Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:00:13.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:00:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:00:13.531Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:00:13.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:00:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:00:13.531Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:00:13.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:13 smithi045 conmon[33655]: cluster 2021-07-31T02:00:12.380798+0000 mon.a ( 2021-07-31T02:00:13.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:13 smithi045 conmon[33655]: mon.0) 1592 : cluster [DBG] osdmap e879: 8 total, 8 up, 8 in 2021-07-31T02:00:14.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:14 smithi154 conmon[34553]: cluster 2021-07-31T02:00:13.979441+0000 mgr.x ( 2021-07-31T02:00:14.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:14 smithi154 conmon[34553]: mgr.24542) 554 : cluster [DBG] pgmap v1116: 41 pgs: 41 active+clean; 1.0 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 424 KiB/s rd, 4.3 MiB/s wr, 4.87k op/s 2021-07-31T02:00:14.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:14 smithi045 conmon[39718]: cluster 2021-07-31T02:00:13.979441+0000 2021-07-31T02:00:14.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:14 smithi045 conmon[39718]: mgr.x (mgr.24542) 554 : cluster [DBG] pgmap v1116: 41 pgs: 41 active+clean; 1.0 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 424 KiB/s rd, 4.3 MiB/s wr, 4.87k op/s 2021-07-31T02:00:14.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:14 smithi045 conmon[33655]: cluster 2021-07-31T02:00:13.979441+0000 mgr.x (mgr 2021-07-31T02:00:14.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:14 smithi045 conmon[33655]: .24542) 554 : cluster [DBG] pgmap v1116: 41 pgs: 41 active+clean; 1.0 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 424 KiB/s rd, 4.3 MiB/s wr, 4.87k op/s 2021-07-31T02:00:17.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:17 smithi154 conmon[34553]: cluster 2021-07-31T02:00:15.979746+0000 mgr.x (mgr.24542) 555 : cluster [DBG] pgmap v1117: 41 pgs: 41 active+clean; 1.0 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 327 KiB/s rd, 3.3 MiB/s wr, 3.75k op/s 2021-07-31T02:00:17.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:17 smithi045 conmon[33655]: cluster 2021-07-31T02:00:15.979746+0000 mgr.x (mgr.24542 2021-07-31T02:00:17.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:17 smithi045 conmon[33655]: ) 555 : cluster [DBG] pgmap v1117: 41 pgs: 41 active+clean; 1.0 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 327 KiB/s rd, 3.3 MiB/s wr, 3.75k op/s 2021-07-31T02:00:17.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:17 smithi045 conmon[39718]: cluster 2021-07-31T02:00 2021-07-31T02:00:17.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:17 smithi045 conmon[39718]: :15.979746+0000 mgr.x (mgr.24542) 555 : cluster [DBG] pgmap v1117: 41 pgs: 41 active+clean; 1.0 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 327 KiB/s rd, 3.3 MiB/s wr, 3.75k op/s 2021-07-31T02:00:18.591 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:00:18.912 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:00:18.913 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 15m ago 15m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:00:18.913 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 12m ago 15m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:00:18.913 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 12m ago 19m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:00:18.913 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 15m ago 21m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:00:18.914 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 15m ago 21m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:00:18.914 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 12m ago 19m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:00:18.915 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 15m ago 19m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:00:18.915 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 15m ago 16m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:00:18.915 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 12m ago 16m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:00:18.915 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 15m ago 18m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:00:18.915 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 15m ago 18m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:00:18.916 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 15m ago 18m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:00:18.916 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 15m ago 17m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:00:18.916 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 12m ago 17m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:00:18.916 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 12m ago 17m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:00:18.917 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 12m ago 16m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:00:18.917 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 12m ago 16m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:00:18.918 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 12m ago 16m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:00:19.271 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:00:19.272 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:00:19.272 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:00:19.272 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:00:19.272 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:00:19.273 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:00:19.273 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:00:19.273 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:00:19.273 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:00:19.274 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:00:19.274 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:00:19.274 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:00:19.274 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:00:19.274 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:00:19.275 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:00:19.275 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:00:19.275 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:00:19.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:19 smithi154 conmon[34553]: cluster 2021-07-31T02:00:17.980103+0000 mgr.x (mgr. 2021-07-31T02:00:19.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:19 smithi154 conmon[34553]: 24542) 556 : cluster [DBG] pgmap v1118: 41 pgs: 41 active+clean; 1.0 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 258 KiB/s rd, 2.6 MiB/s wr, 2.96k op/s 2021-07-31T02:00:19.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:19 smithi045 conmon[39718]: cluster 2021-07-31T02:00:17 2021-07-31T02:00:19.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:19 smithi045 conmon[39718]: .980103+0000 mgr.x (mgr.24542) 556 : cluster [DBG] pgmap v1118: 41 pgs: 41 active+clean; 1.0 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 258 KiB/s rd, 2.6 MiB/s wr, 2.96k op/s 2021-07-31T02:00:19.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:19 smithi045 conmon[33655]: cluster 2021-07-31T02:00:17.980103+0000 mgr.x ( 2021-07-31T02:00:19.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:19 smithi045 conmon[33655]: mgr.24542) 556 : cluster [DBG] pgmap v1118: 41 pgs: 41 active+clean; 1.0 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 258 KiB/s rd, 2.6 MiB/s wr, 2.96k op/s 2021-07-31T02:00:20.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:20 smithi154 conmon[34553]: audit 2021-07-31T02:00:18.577820+0000 mgr.x (mgr.24542) 2021-07-31T02:00:20.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:20 smithi154 conmon[34553]: 557 : audit [DBG] from='client.15282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:00:20.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:20 smithi154 conmon[34553]: audit 2021-07-31T02:00:18.903683+0000 mgr.x (mgr.24542) 558 : audit [DBG] from='client.25195 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:00:20.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:20 smithi154 conmon[34553]: audit 2021-07-31T02:00:19.271401+0000 2021-07-31T02:00:20.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:20 smithi154 conmon[34553]: mon.a (mon.0) 1593 : audit [DBG] from='client.? 172.21.15.45:0/1127248111' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:00:20.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:20 smithi045 conmon[39718]: audit 2021-07-31T02:00:18.577820 2021-07-31T02:00:20.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:20 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 557 : audit [DBG] from='client.15282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:00:20.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:20 smithi045 conmon[39718]: audit 2021-07-31T02:00:20.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:20 smithi045 conmon[39718]: 2021-07-31T02:00:18.903683+0000 mgr.x (mgr 2021-07-31T02:00:20.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:20 smithi045 conmon[39718]: .24542) 558 : audit [DBG] from='client.25195 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:00:20.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:20 smithi045 conmon[39718]: audit 2021 2021-07-31T02:00:20.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:20 smithi045 conmon[39718]: -07-31T02:00:19. 2021-07-31T02:00:20.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:20 smithi045 conmon[39718]: 271401+0000 mon.a (mon.0) 2021-07-31T02:00:20.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:20 smithi045 conmon[39718]: 1593 : audit [DBG] from='client.? 172.21.15.45:0/1127248111' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:00:20.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:20 smithi045 conmon[33655]: audit 2021-07-31T02:00:18.577820 2021-07-31T02:00:20.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:20 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 557 : audit [DBG] from='client.15282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:00:20.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:20 smithi045 conmon[33655]: audit 2021-07-31T02:00:18.903683+0000 mgr.x (mgr.24542) 558 : audit [DBG] from='client.25195 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:00:20.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:20 smithi045 conmon[33655]: 2021-07-31T02:00:20.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:20 smithi045 conmon[33655]: audit 2021-07-31T02:00 2021-07-31T02:00:20.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:20 smithi045 conmon[33655]: :19.271401+0000 mon.a (mon.0 2021-07-31T02:00:20.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:20 smithi045 conmon[33655]: ) 1593 : audit [DBG] 2021-07-31T02:00:20.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:20 smithi045 conmon[33655]: from='client.? 172.21.15.45:0/1127248111' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:00:21.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:21 smithi154 conmon[34553]: cluster 2021-07-31T02:00:19.980516+0000 mgr.x 2021-07-31T02:00:21.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:21 smithi154 conmon[34553]: (mgr.24542) 559 : cluster [DBG] pgmap v1119: 41 pgs: 41 active+clean; 1.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 587 KiB/s rd, 6.0 MiB/s wr, 6.75k op/s 2021-07-31T02:00:21.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:21 smithi045 conmon[39718]: cluster 2021-07-31T02:00:19.980516+0000 mgr.x (mgr 2021-07-31T02:00:21.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:21 smithi045 conmon[39718]: .24542) 559 : cluster [DBG] pgmap v1119: 41 pgs: 41 active+clean; 1.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 587 KiB/s rd, 6.0 MiB/s wr, 6.75k op/s 2021-07-31T02:00:21.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:21 smithi045 conmon[33655]: cluster 2021-07-31T02:00:19.980516+0000 mgr.x (mgr.24542) 559 : cluster [DBG] pgmap v1119: 41 pgs: 41 active+clean; 1.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 587 KiB/s rd, 6.0 MiB/s wr, 6.75k op/s 2021-07-31T02:00:23.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:22 smithi154 conmon[34553]: cluster 2021-07-31T02:00:21.980873+0000 mgr.x (mgr.24542 2021-07-31T02:00:23.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:22 smithi154 conmon[34553]: ) 560 : cluster [DBG] pgmap v1120: 41 pgs: 41 active+clean; 1.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 506 KiB/s rd, 5.2 MiB/s wr, 5.81k op/s 2021-07-31T02:00:23.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:22 smithi045 conmon[39718]: cluster 2021-07-31T02:00:21.980873 2021-07-31T02:00:23.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:22 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 560 : cluster [DBG] pgmap v1120: 41 pgs: 41 active+clean; 1.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 506 KiB/s rd, 5.2 MiB/s wr, 5.81k op/s 2021-07-31T02:00:23.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:22 smithi045 conmon[33655]: cluster 2021-07-31T02:00:21.980873+0000 mgr.x ( 2021-07-31T02:00:23.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:22 smithi045 conmon[33655]: mgr.24542) 560 : cluster [DBG] pgmap v1120: 41 pgs: 41 active+clean; 1.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 506 KiB/s rd, 5.2 MiB/s wr, 5.81k op/s 2021-07-31T02:00:24.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:00:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:00:23.531Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:00:24.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:00:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:00:23.531Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:00:24.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:00:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:00:23.531Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:00:25.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:25 smithi154 conmon[34553]: cluster 2021-07-31T02:00:23.981363+0000 mgr.x (mgr. 2021-07-31T02:00:25.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:25 smithi154 conmon[34553]: 24542) 561 : cluster [DBG] pgmap v1121: 41 pgs: 41 active+clean; 2.3 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 687 KiB/s rd, 6.4 MiB/s wr, 7.24k op/s 2021-07-31T02:00:25.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:25 smithi045 conmon[39718]: cluster 2021-07-31T02:00:23. 2021-07-31T02:00:25.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:25 smithi045 conmon[39718]: 981363+0000 mgr.x (mgr.24542) 561 : cluster [DBG] pgmap v1121: 41 pgs: 41 active+clean; 2.3 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 687 KiB/s rd, 6.4 MiB/s wr, 7.24k op/s 2021-07-31T02:00:25.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:25 smithi045 conmon[33655]: cluster 2021-07-31T02:00:23.981363+0000 mgr.x (mgr.24542) 2021-07-31T02:00:25.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:25 smithi045 conmon[33655]: 561 : cluster [DBG] pgmap v1121: 41 pgs: 41 active+clean; 2.3 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 687 KiB/s rd, 6.4 MiB/s wr, 7.24k op/s 2021-07-31T02:00:27.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:27 smithi154 conmon[34553]: cluster 2021-07-31T02:00:25.981736+0000 mgr.x (mgr.24542 2021-07-31T02:00:27.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:27 smithi154 conmon[34553]: ) 562 : cluster [DBG] pgmap v1122: 41 pgs: 41 active+clean; 2.3 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 500 KiB/s rd, 4.5 MiB/s wr, 5.12k op/s 2021-07-31T02:00:27.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:27 smithi045 conmon[39718]: cluster 2021-07-31T02:00: 2021-07-31T02:00:27.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:27 smithi045 conmon[39718]: 25.981736+0000 mgr.x (mgr.24542) 562 : cluster [DBG] pgmap v1122: 41 pgs: 41 active+clean; 2.3 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 500 KiB/s rd, 4.5 MiB/s wr, 5.12k op/s 2021-07-31T02:00:27.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:27 smithi045 conmon[33655]: cluster 2021-07-31T02:00:25.981736+0000 mgr.x ( 2021-07-31T02:00:27.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:27 smithi045 conmon[33655]: mgr.24542) 562 : cluster [DBG] pgmap v1122: 41 pgs: 41 active+clean; 2.3 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 500 KiB/s rd, 4.5 MiB/s wr, 5.12k op/s 2021-07-31T02:00:28.047 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestCls2PCQueue.MultiProducer (18684 ms) 2021-07-31T02:00:28.047 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestCls2PCQueue.AsyncConsumer 2021-07-31T02:00:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:28 smithi154 conmon[34553]: audit 2021-07-31T02:00:27.844336+0000 mon.b ( 2021-07-31T02:00:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:28 smithi154 conmon[34553]: mon.2) 128 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.2", "id": [7, 0]}]: dispatch 2021-07-31T02:00:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:28 smithi154 conmon[34553]: audit 2021-07-31T02:00:27.844527+0000 mon.b (mon.2) 129 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.8", "id": [7, 5]}]: dispatch 2021-07-31T02:00:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:28 smithi154 conmon[34553]: 2021-07-31T02:00:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:28 smithi154 conmon[34553]: audit 2021 2021-07-31T02:00:28.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:28 smithi154 conmon[34553]: -07-31T02:00:27.844696+0000 mon.b (mon.2 2021-07-31T02:00:28.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:28 smithi154 conmon[34553]: ) 130 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.c", "id": [7, 4]}]: dispatch 2021-07-31T02:00:28.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:28 smithi154 conmon[34553]: audit 2021-07-31T02:00:28.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:28 smithi154 conmon[34553]: 2021-07-31T02:00 2021-07-31T02:00:28.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:28 smithi154 conmon[34553]: :27.844859+0000 mon.b (mon.2) 2021-07-31T02:00:28.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:28 smithi154 conmon[34553]: 131 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.13", "id": [7, 4]}]: dispatch 2021-07-31T02:00:28.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:28 smithi154 conmon[34553]: audit 2021-07-31T02:00:28.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:28 smithi154 conmon[34553]: 2021-07-31T02:00:27.844962 2021-07-31T02:00:28.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:28 smithi154 conmon[34553]: +0000 mon.a (mon.0) 1594 : audit 2021-07-31T02:00:28.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:28 smithi154 conmon[34553]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.2", "id": [7, 0]}]: dispatch 2021-07-31T02:00:28.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:28 smithi154 conmon[34553]: audit 2021-07-31T02: 2021-07-31T02:00:28.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:28 smithi154 conmon[34553]: 00:27.845382+0000 mon.a (mon.0) 1595 2021-07-31T02:00:28.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:28 smithi154 conmon[34553]: : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.8", "id": [7, 5]}]: dispatch 2021-07-31T02:00:28.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:28 smithi154 conmon[34553]: audit 2021-07-31T02:00 2021-07-31T02:00:28.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:28 smithi154 conmon[34553]: :27.845561+0000 mon.a (mon.0) 1596 : audit 2021-07-31T02:00:28.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:28 smithi154 conmon[34553]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.c", "id": [7, 4]}]: dispatch 2021-07-31T02:00:28.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:28 smithi154 conmon[34553]: audit 2021-07-31T02:00:28.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:28 smithi154 conmon[34553]: 2021-07-31 2021-07-31T02:00:28.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:28 smithi154 conmon[34553]: T02:00:27.845736+0000 mon.a (mon.0) 2021-07-31T02:00:28.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:28 smithi154 conmon[34553]: 1597 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.13", "id": [7, 4]}]: dispatch 2021-07-31T02:00:28.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:28 smithi154 conmon[34553]: audit 2021-07 2021-07-31T02:00:28.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:28 smithi154 conmon[34553]: -31T02:00:27.983190+0000 mon.b (mon.2) 2021-07-31T02:00:28.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:28 smithi154 conmon[34553]: 132 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:00:28.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:28 smithi154 conmon[34553]: audit 2021- 2021-07-31T02:00:28.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:28 smithi154 conmon[34553]: 07-31T02:00:27.983528+0000 mon.a (mon 2021-07-31T02:00:28.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:28 smithi154 conmon[34553]: .0) 1598 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:00:28.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:28 smithi154 conmon[34553]: audit 2021-07-31 2021-07-31T02:00:28.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:28 smithi154 conmon[34553]: T02:00:28.000926+0000 mon.b (mon.2) 133 : 2021-07-31T02:00:28.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:28 smithi154 conmon[34553]: audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:00:28.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:28 smithi154 conmon[34553]: audit 2021-07-31T02:00:28.001458+0000 2021-07-31T02:00:28.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:28 smithi154 conmon[34553]: mon.a (mon.0) 1599 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:00:28.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:28 smithi045 conmon[33655]: audit 2021-07-31T02:00:27.844336+0000 mon.b (mon.2 2021-07-31T02:00:28.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:28 smithi045 conmon[33655]: ) 128 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.2", "id": [7, 0]}]: dispatch 2021-07-31T02:00:28.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:28 smithi045 conmon[33655]: audit 2021-07-31T02:00:27.844527+0000 mon.b (mon.2) 129 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.8", "id": [7, 5]}]: dispatch 2021-07-31T02:00:28.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:28 smithi045 conmon[33655]: audit 2021-07-31T02:00:27.844696+0000 mon.b 2021-07-31T02:00:28.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:28 smithi045 conmon[33655]: (mon.2) 130 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.c", "id": [7, 4]}]: dispatch 2021-07-31T02:00:28.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:28 smithi045 conmon[33655]: 2021-07-31T02:00:28.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:28 smithi045 conmon[33655]: audit 2021-07-31T02:00:28.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:28 smithi045 conmon[33655]: 2021-07-31T02:00:27.844859+0000 mon.b (mon. 2021-07-31T02:00:28.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:28 smithi045 conmon[33655]: 2) 131 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.13", "id": [7, 4]}]: dispatch 2021-07-31T02:00:28.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:28 smithi045 conmon[33655]: audit 2021-07-31T02:00: 2021-07-31T02:00:28.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:28 smithi045 conmon[33655]: 27.844962+0000 mon.a (mon.0) 1594 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.2", "id": [7, 0]}]: dispatch 2021-07-31T02:00:28.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:28 smithi045 conmon[33655]: audit 2021-07-31T02: 2021-07-31T02:00:28.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:28 smithi045 conmon[33655]: 00:27.845382+0000 mon.a (mon.0) 1595 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.8", "id": [7, 5]}]: dispatch 2021-07-31T02:00:28.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:28 smithi045 conmon[33655]: audit 2021-07-31T02:00 2021-07-31T02:00:28.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:28 smithi045 conmon[33655]: :27.845561+0000 mon.a (mon.0) 1596 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.c", "id": [7, 4]}]: dispatch 2021-07-31T02:00:28.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:28 smithi045 conmon[33655]: audit 2021-07-31T02:00 2021-07-31T02:00:28.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:28 smithi045 conmon[33655]: :27.845736+0000 mon.a (mon.0) 1597 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.13", "id": [7, 4]}]: dispatch 2021-07-31T02:00:28.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:28 smithi045 conmon[33655]: 2021-07-31T02:00:28.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:28 smithi045 conmon[33655]: audit 2021-07-31T02 2021-07-31T02:00:28.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:28 smithi045 conmon[33655]: :00:27.983190+0000 mon.b (mon.2) 132 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:00:28.535 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:28 smithi045 conmon[33655]: audit 2021-07-31T02:00:27 2021-07-31T02:00:28.535 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:28 smithi045 conmon[33655]: .983528+0000 mon.a (mon.0) 1598 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:00:28.535 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:28 smithi045 conmon[33655]: 2021-07-31T02:00:28.535 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:28 smithi045 conmon[33655]: audit 2021-07-31T02 2021-07-31T02:00:28.535 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:28 smithi045 conmon[33655]: :00:28.000926+0000 mon.b (mon.2) 133 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:00:28.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:28 smithi045 conmon[33655]: audit 2021- 2021-07-31T02:00:28.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:28 smithi045 conmon[33655]: 07-31T02:00:28.001458+0000 2021-07-31T02:00:28.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:28 smithi045 conmon[33655]: mon.a (mon.0) 1599 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:00:28.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:28 smithi045 conmon[39718]: audit 2021-07-31T02:00:27.844336+0000 mon.b (mon. 2021-07-31T02:00:28.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:28 smithi045 conmon[39718]: 2) 128 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.2", "id": [7, 0]}]: dispatch 2021-07-31T02:00:28.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:28 smithi045 conmon[39718]: audit 2021-07-31T02:00:27.844527+0000 mon.b (mon.2) 129 : audit 2021-07-31T02:00:28.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:28 smithi045 conmon[39718]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.8", "id": [7, 5]}]: dispatch 2021-07-31T02:00:28.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:28 smithi045 conmon[39718]: audit 2021-07-31T02:00:27.844696+0000 2021-07-31T02:00:28.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:28 smithi045 conmon[39718]: mon.b (mon.2) 130 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.c", "id": [7, 4]}]: dispatch 2021-07-31T02:00:28.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:28 smithi045 conmon[39718]: audit 2021-07-31T02:00:28.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:28 smithi045 conmon[39718]: 2021-07-31T02:00:27.844859+0000 mon.b (mon.2 2021-07-31T02:00:28.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:28 smithi045 conmon[39718]: ) 131 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.13", "id": [7, 4]}]: dispatch 2021-07-31T02:00:28.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:28 smithi045 conmon[39718]: audit 2021-07-31T02:00:27.844962+0000 mon.a 2021-07-31T02:00:28.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:28 smithi045 conmon[39718]: (mon.0) 1594 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.2", "id": [7, 0]}]: dispatch 2021-07-31T02:00:28.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:28 smithi045 conmon[39718]: audit 2021-07-31T02:00:27.845382+0000 mon.a (mon.0) 1595 : audit 2021-07-31T02:00:28.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:28 smithi045 conmon[39718]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.8", "id": [7, 5]}]: dispatch 2021-07-31T02:00:28.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:28 smithi045 conmon[39718]: audit 2021-07-31T02:00:28.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:28 smithi045 conmon[39718]: 2021-07-31T02:00:27.845561+0000 mon.a (mon 2021-07-31T02:00:28.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:28 smithi045 conmon[39718]: .0) 1596 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.c", "id": [7, 4]}]: dispatch 2021-07-31T02:00:28.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:28 smithi045 conmon[39718]: audit 2021 2021-07-31T02:00:28.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:28 smithi045 conmon[39718]: -07-31T02:00:27.845736+0000 2021-07-31T02:00:28.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:28 smithi045 conmon[39718]: mon.a (mon.0) 1597 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.13", "id": [7, 4]}]: dispatch 2021-07-31T02:00:28.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:28 smithi045 conmon[39718]: 2021-07-31T02:00:28.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:28 smithi045 conmon[39718]: audit 2021-07-31T02:00:28.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:28 smithi045 conmon[39718]: 2021-07-31T02:00:27.983190+0000 2021-07-31T02:00:28.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:28 smithi045 conmon[39718]: mon.b (mon.2) 132 : audit [INF] 2021-07-31T02:00:28.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:28 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:00:28.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:28 smithi045 conmon[39718]: audit 2021-07-31T02:00:27. 2021-07-31T02:00:28.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:28 smithi045 conmon[39718]: 983528+0000 mon.a (mon.0) 1598 : audit 2021-07-31T02:00:28.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:28 smithi045 conmon[39718]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:00:28.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:28 smithi045 conmon[39718]: audit 2021-07 2021-07-31T02:00:28.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:28 smithi045 conmon[39718]: -31T02:00:28.000926+0000 mon.b 2021-07-31T02:00:28.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:28 smithi045 conmon[39718]: (mon.2) 133 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:00:28.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:28 smithi045 conmon[39718]: audit 2021- 2021-07-31T02:00:28.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:28 smithi045 conmon[39718]: 07-31T02:00:28.001458+0000 2021-07-31T02:00:28.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:28 smithi045 conmon[39718]: mon.a (mon.0) 1599 : audit 2021-07-31T02:00:28.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:28 smithi045 conmon[39718]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:00:29.469 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[34553]: cluster 2021-07-31T02:00:27.982057+0000 mgr.x (mgr.24542) 563 : cluster [DBG] pgmap v1123: 41 pgs: 41 active+clean; 2.3 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 500 KiB/s rd, 4.5 MiB/s wr, 5.12k op/s 2021-07-31T02:00:29.469 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[34553]: audit 2021-07-31T02:00:28.044849+0000 mon.a (mon.0) 1600 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.2", "id": [7, 0]}]': finished 2021-07-31T02:00:29.469 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[34553]: audit 2021-07-31T02:00:28.044947+0000 mon.a (mon.0) 1601 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.8", "id": [7, 5]}]': finished 2021-07-31T02:00:29.470 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[34553]: audit 2021-07-31T02:00:28.045059+0000 mon.a (mon 2021-07-31T02:00:29.470 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[34553]: .0) 1602 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.c", "id": [7, 4]}]': finished 2021-07-31T02:00:29.470 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[34553]: audit 2021-07 2021-07-31T02:00:29.470 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[34553]: -31T02:00:28.045139+0000 mon.a 2021-07-31T02:00:29.471 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[34553]: (mon.0) 1603 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.13", "id": [7, 4]}]': finished 2021-07-31T02:00:29.471 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[34553]: cluster 2021-07-31T02:00: 2021-07-31T02:00:29.471 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[34553]: 28.045217+0000 mon.a (mon.0) 1604 : cluster [DBG] 2021-07-31T02:00:29.471 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[34553]: osdmap e880: 8 total, 8 up, 8 in 2021-07-31T02:00:29.472 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:29 smithi045 conmon[33655]: cluster 2021-07-31T02:00:27.982057+0000 mgr.x (mgr. 2021-07-31T02:00:29.472 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:29 smithi045 conmon[33655]: 24542) 563 : cluster [DBG] pgmap v1123: 41 pgs: 41 active+clean; 2.3 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 500 KiB/s rd, 4.5 MiB/s wr, 5.12k op/s 2021-07-31T02:00:29.472 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:29 smithi045 conmon[33655]: audit 2021-07-31T02:00:28.044849+0000 mon.a (mon.0) 1600 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.2", "id": [7, 0]}]': finished 2021-07-31T02:00:29.473 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:29 smithi045 conmon[33655]: audit 2021-07-31T02:00:28.044947+0000 mon.a (mon.0) 1601 2021-07-31T02:00:29.473 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:29 smithi045 conmon[33655]: : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.8", "id": [7, 5]}]': finished 2021-07-31T02:00:29.473 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:29 smithi045 conmon[33655]: audit 2021-07-31T02:00:29.473 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:29 smithi045 conmon[33655]: 2021-07-31T02:00:28.045059+0000 mon.a (mon.0) 1602 2021-07-31T02:00:29.474 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:29 smithi045 conmon[33655]: : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.c", "id": [7, 4]}]': finished 2021-07-31T02:00:29.474 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:29 smithi045 conmon[33655]: audit 2021-07-31T 2021-07-31T02:00:29.474 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:29 smithi045 conmon[33655]: 02:00:28.045139+0000 mon.a (mon.0) 1603 : audit [INF] 2021-07-31T02:00:29.474 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:29 smithi045 conmon[33655]: from='mgr.24542 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.13", "id": [7, 4]}]': finished 2021-07-31T02:00:29.474 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:29 smithi045 conmon[33655]: cluster 2021-07-31T02:00:28.045217+0000 mon.a ( 2021-07-31T02:00:29.475 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:29 smithi045 conmon[33655]: mon.0) 1604 : cluster [DBG] osdmap e880: 8 total, 8 up, 8 in 2021-07-31T02:00:29.475 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:29 smithi045 conmon[39718]: cluster 2021-07-31T02:00:27.982057+0000 mgr.x (mgr.24542) 563 : cluster [DBG] 2021-07-31T02:00:29.475 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:29 smithi045 conmon[39718]: pgmap v1123: 41 pgs: 41 active+clean; 2.3 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 500 KiB/s rd, 4.5 MiB/s wr, 5.12k op/s 2021-07-31T02:00:29.476 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:29 smithi045 conmon[39718]: audit 2021-07-31T02:00:28.044849+0000 2021-07-31T02:00:29.476 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:29 smithi045 conmon[39718]: mon.a (mon.0) 1600 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.2", "id": [7, 0]}]': finished 2021-07-31T02:00:29.476 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:29 smithi045 conmon[39718]: audit 2021-07-31T02:00:28.044947 2021-07-31T02:00:29.476 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:29 smithi045 conmon[39718]: +0000 mon.a (mon.0) 1601 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.8", "id": [7, 5]}]': finished 2021-07-31T02:00:29.476 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:29 smithi045 conmon[39718]: audit 2021-07-31T02:00:29.477 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:29 smithi045 conmon[39718]: 2021-07-31T02:00:28. 2021-07-31T02:00:29.477 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:29 smithi045 conmon[39718]: 045059+0000 mon.a (mon.0) 1602 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.c", "id": [7, 4]}]': finished 2021-07-31T02:00:29.477 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:29 smithi045 conmon[39718]: audit 2021-07 2021-07-31T02:00:29.477 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:29 smithi045 conmon[39718]: -31T02:00:28.045139+0000 mon.a (mon 2021-07-31T02:00:29.478 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:29 smithi045 conmon[39718]: .0) 1603 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.13", "id": [7, 4]}]': finished 2021-07-31T02:00:29.478 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:29 smithi045 conmon[39718]: cluster 2021- 2021-07-31T02:00:29.478 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:29 smithi045 conmon[39718]: 07-31T02:00:28.045217+0000 mon.a ( 2021-07-31T02:00:29.478 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:29 smithi045 conmon[39718]: mon.0) 1604 : cluster [DBG] osdmap e880: 8 total, 8 up, 8 in 2021-07-31T02:00:29.753 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: debug 2021-07-31T02:00:29.486+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:00:29.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:00:29.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:00:29.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: yield 2021-07-31T02:00:29.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:00:29.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:00:29.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:00:29.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:00:29.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:00:29.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:00:29.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:00:29.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:00:29.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:00:29.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: task.result() 2021-07-31T02:00:29.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:00:29.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: debug 2021-07-31T02:00:29.488+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:00:29.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:00:29.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:00:29.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: yield 2021-07-31T02:00:29.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:00:29.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:00:29.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:00:29.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:00:29.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:00:29.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:00:29.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:00:29.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:00:29.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:00:29.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: task.result() 2021-07-31T02:00:29.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:00:29.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: 2021-07-31T02:00:29.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: debug 2021-07-31T02:00:29.524+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:00:29.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:00:29.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:00:29.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: yield 2021-07-31T02:00:29.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:00:29.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:00:29.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:00:29.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:00:29.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:00:29.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:00:29.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:00:29.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:00:29.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:00:29.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: task.result() 2021-07-31T02:00:29.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:00:29.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: debug 2021-07-31T02:00:29.525+0000 7fcb9115c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:00:29.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:00:29.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:00:29.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: yield 2021-07-31T02:00:29.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:00:29.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:00:29.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:00:29.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:00:29.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:00:29.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:00:29.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:00:29.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:00:29.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:00:29.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: task.result() 2021-07-31T02:00:29.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:00:29.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: 2021-07-31T02:00:29.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:00:29.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: 2021-07-31T02:00:29.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:00:29.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:00:29.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:00:29.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:00:29.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:00:29.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:00:29.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:00:29.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:00:29.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:00:29.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:00:29.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:00:29.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:00:29.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:00:29.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:00:29.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:00:29.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:00:29.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:00:29.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:00:29.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: raise 2021-07-31T02:00:29.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:00:29.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:00:29.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:00:29.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:00:29.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:00:29.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: debug 2021-07-31T02:00:29.532+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:00:29.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:00:29.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:00:29.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: yield 2021-07-31T02:00:29.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:00:29.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:00:29.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:00:29.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:00:29.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:00:29.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:00:29.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:00:29.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:00:29.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:00:29.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: task.result() 2021-07-31T02:00:29.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:00:29.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: debug 2021-07-31T02:00:29.532+0000 7fcb9095b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:00:29.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:00:29.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:00:29.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: yield 2021-07-31T02:00:29.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:00:29.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:00:29.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:00:29.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:00:29.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:00:29.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:00:29.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:00:29.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:00:29.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:00:29.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: task.result() 2021-07-31T02:00:29.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:00:29.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: 2021-07-31T02:00:29.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:00:29.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: 2021-07-31T02:00:29.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:00:29.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:00:29.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:00:29.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:00:29.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:00:29.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:00:29.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:00:29.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:00:29.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:00:29.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:00:29.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:00:29.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:00:29.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:00:29.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:00:29.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:00:29.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:00:29.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:00:29.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:00:29.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: raise 2021-07-31T02:00:29.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:00:29.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:00:29.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:00:29.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:00:29.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:00:29 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:00:30.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:30 smithi154 conmon[34553]: cluster 2021-07-31T02:00:29.050407+0000 mon.a (mon.0) 2021-07-31T02:00:30.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:30 smithi154 conmon[34553]: 1605 : cluster [DBG] osdmap e881: 8 total, 8 up, 8 in 2021-07-31T02:00:30.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:30 smithi154 conmon[34553]: audit 2021-07-31T02:00:29.051992+0000 mon.c (mon.1) 44 : audit [INF] from='client.? 172.21.15.45:0/702521033' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:00:30.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:30 smithi154 conmon[34553]: audit 2021-07-31T02:00:30.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:30 smithi154 conmon[34553]: 2021-07-31T02:00:29.052572+0000 mon.a (mon.0) 1606 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:00:30.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:30 smithi154 conmon[34553]: audit 2021-07-31T02:00:29.441711+0000 mon.b (mon.2) 2021-07-31T02:00:30.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:30 smithi154 conmon[34553]: 134 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:00:30.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:30 smithi154 conmon[34553]: audit 2021-07- 2021-07-31T02:00:30.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:30 smithi154 conmon[34553]: 31T02:00:29.443318+0000 mon.b (mon.2) 135 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:00:30.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:30 smithi154 conmon[34553]: audit 2021-07-31T02:00:30.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:30 smithi154 conmon[34553]: 2021-07-31T02:00:29.444318+0000 mon.b (mon.2) 136 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:00:30.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:30 smithi045 conmon[39718]: cluster 2021-07-31T02:00:29.050407+0000 2021-07-31T02:00:30.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:30 smithi045 conmon[39718]: mon.a (mon.0) 1605 : cluster [DBG] osdmap e881: 8 total, 8 up, 8 in 2021-07-31T02:00:30.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:30 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T02:00:30.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:30 smithi045 conmon[39718]: T02:00:29.051992+0000 mon.c (mon.1) 44 : audit [INF] 2021-07-31T02:00:30.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:30 smithi045 conmon[39718]: from='client.? 172.21.15.45:0/702521033' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:00:30.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:30 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T02:00:30.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:30 smithi045 conmon[39718]: 31T02:00:29.052572+0000 mon.a (mon 2021-07-31T02:00:30.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:30 smithi045 conmon[39718]: .0) 1606 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:00:30.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:30 smithi045 conmon[39718]: audit 2021-07 2021-07-31T02:00:30.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:30 smithi045 conmon[39718]: -31T02:00:29.441711+0000 mon.b (mon.2) 134 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:00:30.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:30 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T02:00:30.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:30 smithi045 conmon[39718]: T02:00:29.443318+0000 mon.b (mon.2) 135 : audit [DBG] 2021-07-31T02:00:30.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:30 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:00:30.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:30 smithi045 conmon[39718]: audit 2021- 2021-07-31T02:00:30.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:30 smithi045 conmon[39718]: 07-31T02:00:29.444318+0000 mon.b (mon.2) 136 2021-07-31T02:00:30.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:30 smithi045 conmon[39718]: : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:00:30.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:30 smithi045 conmon[33655]: cluster 2021-07-31T02:00:29.050407+0000 mon.a (mon.0) 1605 : cluster [DBG] osdmap e881: 8 total, 8 up, 8 in 2021-07-31T02:00:30.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:30 smithi045 conmon[33655]: audit 2021-07-31T02:00:29.051992+0000 mon.c (mon.1) 44 : audit [INF] from='client.? 172.21.15.45:0/702521033' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:00:30.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:30 smithi045 conmon[33655]: audit 2021-07-31T02:00:29.052572+0000 mon.a (mon.0) 1606 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:00:30.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:30 smithi045 conmon[33655]: audit 2021-07-31T02:00:29.441711+0000 mon.b (mon.2) 134 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:00:30.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:30 smithi045 conmon[33655]: audit 2021-07-31T02:00:29.443318+0000 mon.b (mon.2) 135 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:00:30.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:30 smithi045 conmon[33655]: audit 2021-07-31T02:00:29.444318+0000 mon.b (mon.2) 136 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:00:31.061 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestCls2PCQueue.AsyncConsumer (3014 ms) 2021-07-31T02:00:31.062 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestCls2PCQueue.MultiProducerConsumer 2021-07-31T02:00:31.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:00:31.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: 2021-07-31T02:00:31.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: 2021 2021-07-31T02:00:31.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: - 2021-07-31T02:00:31.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: 07 2021-07-31T02:00:31.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: - 2021-07-31T02:00:31.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: 31 2021-07-31T02:00:31.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: T 2021-07-31T02:00:31.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: 02 2021-07-31T02:00:31.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: : 2021-07-31T02:00:31.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: 00 2021-07-31T02:00:31.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: : 2021-07-31T02:00:31.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: 29 2021-07-31T02:00:31.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: . 2021-07-31T02:00:31.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: 445168 2021-07-31T02:00:31.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: +0000 2021-07-31T02:00:31.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: mgr.x (mgr.24542) 564 : cephadm [WRN] 2021-07-31T02:00:31.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:00:31.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: cephadm 2021-07- 2021-07-31T02:00:31.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: 31T02:00:29.487880+0000 mgr.x (mgr.24542) 565 : cephadm [ERR] 2021-07-31T02:00:31.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:00:31.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:00:31.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: yield 2021-07-31T02:00:31.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:00:31.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:00:31.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:00:31.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:00:31.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:00:31.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:00:31.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:00:31.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:00:31.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:00:31.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: task.result() 2021-07-31T02:00:31.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:00:31.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:00:29.488704+0000 mgr.x (mgr.24542) 566 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:00:31.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:00:29.489905+0000 mgr.x (mgr.24542) 567 : cephadm 2021-07-31T02:00:31.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: [ERR] 2021-07-31T02:00:31.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:00:31.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:00:31.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: yield 2021-07-31T02:00:31.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:00:31.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:00:31.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:00:31.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:00:31.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:00:31.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:00:31.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:00:31.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:00:31.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:00:31.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: task.result() 2021-07-31T02:00:31.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:00:31.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:00:29.499735+0000 mgr.x (mgr.24542) 568 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:00:31.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:00:29.525748+0000 mgr.x (mgr.24542) 569 : cephadm [ERR] 2021-07-31T02:00:31.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:00:31.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:00:31.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: yield 2021-07-31T02:00:31.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:00:31.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:00:31.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:00:31.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:00:31.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:00:31.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:00:31.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:00:31.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:00:31.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:00:31.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: task.result() 2021-07-31T02:00:31.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:00:31.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:00:29.526151+0000 mgr.x (mgr.24542) 570 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:00:31.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:00:31.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:00:31.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: yield 2021-07-31T02:00:31.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:00:31.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:00:31.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:00:31.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:00:31.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:00:31.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:00:31.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:00:31.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:00:31.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:00:31.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: task.result() 2021-07-31T02:00:31.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:00:31.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: 2021-07-31T02:00:31.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:00:31.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: 2021-07-31T02:00:31.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:00:31.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:00:31.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:00:31.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:00:31.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:00:31.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:00:31.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:00:31.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:00:31.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:00:31.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:00:31.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:00:31.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:00:31.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:00:31.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:00:31.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:00:31.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:00:31.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:00:31.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:00:31.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: raise 2021-07-31T02:00:31.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:00:31.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:00:31.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:00:31.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:00:31.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:00:31.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:00:29.533062+0000 mgr.x ( 2021-07-31T02:00:31.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: mgr.24542) 571 : cephadm [ERR] 2021-07-31T02:00:31.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:00:31.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:00:31.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: yield 2021-07-31T02:00:31.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:00:31.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:00:31.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:00:31.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:00:31.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:00:31.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:00:31.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:00:31.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:00:31.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:00:31.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: task.result() 2021-07-31T02:00:31.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:00:31.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:00:29.533612+0000 mgr.x (mgr.24542) 572 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:00:31.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:00:31.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:00:31.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: yield 2021-07-31T02:00:31.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:00:29.445168+0000 mgr.x (mgr.24542) 564 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:00:31.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: 2021-07-31T02:00:31.535 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:00:29.487880+0000 mgr.x (mgr.24542) 565 : cephadm [ERR] 2021-07-31T02:00:31.535 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:00:31.535 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:00:31.535 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: yield 2021-07-31T02:00:31.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:00:31.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:00:31.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:00:31.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:00:31.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:00:31.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:00:31.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:00:31.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:00:31.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:00:31.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: task.result() 2021-07-31T02:00:31.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:00:31.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:00:29.488704+0000 mgr.x (mgr.24542) 566 : cephadm [INF] 2021-07-31T02:00:31.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:00:31.539 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:00:29.489905+0000 mgr.x (mgr.24542) 567 : cephadm [ERR] 2021-07-31T02:00:31.539 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:00:31.539 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:00:31.539 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: yield 2021-07-31T02:00:31.539 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:00:31.540 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:00:31.540 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:00:31.540 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:00:31.540 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:00:31.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:00:31.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:00:31.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:00:31.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:00:31.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: task.result() 2021-07-31T02:00:31.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:00:31.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:00:29.499735+0000 mgr.x ( 2021-07-31T02:00:31.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: mgr.24542) 568 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:00:31.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:00:29.525748+0000 mgr.x (mgr.24542) 569 : cephadm [ERR] 2021-07-31T02:00:31.543 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:00:31.543 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:00:31.543 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:00:31.544 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:00:31.544 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:00:31.544 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:00:31.544 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:00:31.545 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:00:31.545 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:00:31.545 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:00:31.545 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:00:31.546 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: task.result() 2021-07-31T02:00:31.546 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:00:31.546 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: 2021-07-31T02:00:31.546 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:00:31.547 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: 2021-07-31T02:00:31.547 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:00:31.547 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:00:31.547 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:00:31.547 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:00:31.548 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:00:31.548 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:00:31.548 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:00:31.548 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:00:31.549 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:00:31.549 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:00:31.549 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:00:31.549 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:00:31.550 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:00:31.550 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:00:31.550 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:00:31.550 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:00:31.551 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:00:31.551 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:00:31.551 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: raise 2021-07-31T02:00:31.551 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:00:31.552 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:00:31.552 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:00:31.552 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:00:31.552 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:00:31.553 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: cluster 2021-07-31T02:00:29.982431+0000 mgr.x (mgr.24542) 573 : cluster [DBG] pgmap v1126: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2021-07-31T02:00:31.553 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: audit 2021-07-31T02:00:31.553 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: 2021-07-31T02:00:30.058072+0000 mon.a (mon.0) 1607 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-18","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:00:31.553 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:31 smithi154 conmon[34553]: cluster 2021-07-31T02:00:30.058370+0000 mon.a (mon.0) 1608 : cluster [DBG] osdmap e882: 8 total, 8 up, 8 in 2021-07-31T02:00:31.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:00:29.445168+0000 mgr.x (mgr. 2021-07-31T02:00:31.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: 24542) 564 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:00:31.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: cephadm 2021-07-31T 2021-07-31T02:00:31.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: 02:00:29.487880+0000 mgr.x (mgr.24542) 565 : cephadm 2021-07-31T02:00:31.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: [ERR] 2021-07-31T02:00:31.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:00:31.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:00:31.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: yield 2021-07-31T02:00:31.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:00:31.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:00:31.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:00:31.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:00:31.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:00:31.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:00:31.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:00:31.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:00:31.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:00:31.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: task.result() 2021-07-31T02:00:31.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:00:31.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:00:29.488704+0000 mgr.x (mgr.24542) 566 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:00:31.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: cephadm 2021-07 2021-07-31T02:00:31.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: -31T02:00:29 2021-07-31T02:00:31.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: .489905+0000 mgr.x (mgr. 2021-07-31T02:00:31.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: 24542) 567 : cephadm [ERR] 2021-07-31T02:00:31.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:00:31.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:00:31.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: yield 2021-07-31T02:00:31.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:00:31.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:00:31.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:00:31.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:00:31.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:00:31.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:00:31.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:00:31.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:00:31.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:00:31.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: task.result() 2021-07-31T02:00:31.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:00:31.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: 2021-07-31T02:00:31.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:00:29.499735+0000 mgr.x (mgr.24542) 568 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:00:31.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: cephadm 2021-07-31T02: 2021-07-31T02:00:31.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: 00:29.525748+0000 mgr.x (mgr. 2021-07-31T02:00:31.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: yield 2021-07-31T02:00:31.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:00:31.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:00:31.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:00:31.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:00:31.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:00:31.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:00:31.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:00:31.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:00:31.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:00:31.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: task.result() 2021-07-31T02:00:31.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:00:31.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:00:29.526151+0000 mgr.x (mgr.24542) 570 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:00:31.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:00:31.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:00:31.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: yield 2021-07-31T02:00:31.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:00:31.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:00:31.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:00:31.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:00:31.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:00:31.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:00:31.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:00:31.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:00:31.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:00:31.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: task.result() 2021-07-31T02:00:31.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:00:31.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: 2021-07-31T02:00:31.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:00:31.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: 2021-07-31T02:00:31.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:00:31.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:00:31.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:00:31.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:00:31.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:00:31.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:00:31.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:00:31.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:00:31.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:00:31.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:00:31.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: 24542) 569 : cephadm [ERR] 2021-07-31T02:00:31.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:00:31.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:00:31.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: yield 2021-07-31T02:00:31.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:00:31.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:00:31.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:00:31.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:00:31.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:00:31.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:00:31.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:00:31.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:00:31.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:00:31.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: task.result() 2021-07-31T02:00:31.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:00:31.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:00:29.526151+0000 mgr.x (mgr.24542) 570 : cephadm [ERR] 2021-07-31T02:00:31.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:00:31.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:00:31.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:00:31.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: yield 2021-07-31T02:00:31.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:00:31.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:00:31.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:00:31.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:00:31.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:00:31.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:00:31.585 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:00:31.585 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:00:31.585 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:00:31.585 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: task.result() 2021-07-31T02:00:31.586 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:00:31.586 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: 2021-07-31T02:00:31.586 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:00:31.586 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: 2021-07-31T02:00:31.587 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:00:31.587 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:00:31.587 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:00:31.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:00:31.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:00:31.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:00:31.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:00:31.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:00:31.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:00:31.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:00:31.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:00:31.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:00:31.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:00:31.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:00:31.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:00:31.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:00:31.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:00:31.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:00:31.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: raise 2021-07-31T02:00:31.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:00:31.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:00:31.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:00:31.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:00:31.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:00:31.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:00:29.533062+0000 mgr.x (mgr.24542) 571 : cephadm [ERR] 2021-07-31T02:00:31.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:00:31.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:00:31.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: yield 2021-07-31T02:00:31.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:00:31.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:00:31.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:00:31.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:00:31.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:00:31.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:00:31.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:00:31.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:00:31.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:00:31.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: task.result() 2021-07-31T02:00:31.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:00:31.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:00:29.533612+0000 mgr.x (mgr.24542) 572 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:00:31.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:00:31.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:00:31.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: yield 2021-07-31T02:00:31.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:00:31.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:00:31.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:00:31.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:00:31.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:00:31.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:00:31.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:00:31.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:00:31.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:00:31.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:00:31.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:00:31.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: raise 2021-07-31T02:00:31.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:00:31.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:00:31.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:00:31.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:00:31.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:00:31.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:00:29.533062+0000 mgr.x (mgr. 2021-07-31T02:00:31.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: 24542) 571 : cephadm [ERR] 2021-07-31T02:00:31.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:00:31.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:00:31.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: yield 2021-07-31T02:00:31.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:00:31.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:00:31.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:00:31.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:00:31.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:00:31.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:00:31.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:00:31.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:00:31.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:00:31.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: task.result() 2021-07-31T02:00:31.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:00:31.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:00:29.533612+0000 mgr.x (mgr.24542) 572 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:00:31.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:00:31.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:00:31.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: yield 2021-07-31T02:00:31.608 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:00:31.608 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:00:31.608 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:00:31.608 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:00:31.609 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:00:31.609 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:00:31.610 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:00:31.610 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:00:31.610 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:00:31.610 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:00:31.611 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:00:31.611 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:00:31.611 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: task.result() 2021-07-31T02:00:31.611 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:00:31.612 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: 2021-07-31T02:00:31.612 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:00:31.612 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: 2021-07-31T02:00:31.612 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:00:31.613 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:00:31.613 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:00:31.613 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:00:31.613 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:00:31.614 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:00:31.614 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:00:31.614 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:00:31.614 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:00:31.615 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:00:31.615 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:00:31.615 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:00:31.615 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:00:31.616 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:00:31.616 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:00:31.616 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:00:31.616 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:00:31.617 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:00:31.617 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: raise 2021-07-31T02:00:31.617 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:00:31.617 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:00:31.618 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:00:31.618 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:00:31.618 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:00:31.618 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: cluster 2021-07-31T02:00:29.982431 2021-07-31T02:00:31.619 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 573 : cluster [DBG] pgmap v1126: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2021-07-31T02:00:31.619 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: audit 2021-07-31T02:00:30.058072+0000 mon.a (mon.0) 1607 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-18","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:00:31.619 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[39718]: cluster 2021-07-31T02:00:30.058370+0000 mon.a (mon.0) 1608 : cluster [DBG] osdmap e882: 8 total, 8 up, 8 in 2021-07-31T02:00:31.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:00:31.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:00:31.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:00:31.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: task.result() 2021-07-31T02:00:31.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:00:31.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: 2021-07-31T02:00:31.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:00:31.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: 2021-07-31T02:00:31.622 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:00:31.622 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:00:31.622 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:00:31.622 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:00:31.623 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:00:31.623 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:00:31.623 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:00:31.623 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:00:31.624 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:00:31.624 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:00:31.624 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:00:31.624 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:00:31.625 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:00:31.625 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:00:31.625 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:00:31.625 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:00:31.625 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:00:31.626 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:00:31.626 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: raise 2021-07-31T02:00:31.626 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:00:31.626 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:00:31.627 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:00:31.627 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:00:31.627 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:00:31.627 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: cluster 2021-07-31T02:00:29.982431+0000 mgr.x (mgr.24542) 573 : cluster [DBG] pgmap v1126: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2021-07-31T02:00:31.628 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: audit 2021-07-31T02:00:30.058072+0000 mon.a (mon.0) 1607 : 2021-07-31T02:00:31.628 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-18","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:00:31.628 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:31 smithi045 conmon[33655]: cluster 2021-07-31T02:00:30.058370+0000 mon.a (mon.0) 1608 : cluster [DBG] osdmap e882: 8 total, 8 up, 8 in 2021-07-31T02:00:32.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:32 smithi154 conmon[34553]: cluster 2021-07-31T02:00:32.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:32 smithi154 conmon[34553]: 2021-07-31T02:00:31 2021-07-31T02:00:32.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:32 smithi154 conmon[34553]: .058921+0000 mon.a (mon.0) 1609 : cluster [DBG] osdmap e883: 8 total, 8 up, 8 in 2021-07-31T02:00:32.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:32 smithi154 conmon[34553]: cluster 2021-07-31T02: 2021-07-31T02:00:32.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:32 smithi154 conmon[34553]: 00:31.982899+0000 mgr.x (mgr.24542) 574 2021-07-31T02:00:32.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:32 smithi154 conmon[34553]: : cluster [DBG] pgmap v1129: 9 pgs: 9 active+clean; 577 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2021-07-31T02:00:32.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:32 smithi045 conmon[39718]: cluster 2021-07-31T02:00:31 2021-07-31T02:00:32.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:32 smithi045 conmon[39718]: .058921+0000 mon.a (mon.0) 1609 : cluster [DBG] osdmap e883: 8 total, 8 up, 8 in 2021-07-31T02:00:32.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:32 smithi045 conmon[39718]: cluster 2021-07-31 2021-07-31T02:00:32.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:32 smithi045 conmon[39718]: T02:00:31.982899+0000 2021-07-31T02:00:32.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:32 smithi045 conmon[39718]: mgr.x (mgr.24542) 574 : cluster 2021-07-31T02:00:32.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:32 smithi045 conmon[39718]: [DBG] pgmap v1129: 9 pgs: 9 active+clean; 577 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2021-07-31T02:00:32.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:32 smithi045 conmon[33655]: cluster 2021-07-31T02:00:31.058921+0000 mon.a (mon.0) 1609 : cluster [DBG] osdmap e883: 8 total, 8 up, 8 in 2021-07-31T02:00:32.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:32 smithi045 conmon[33655]: cluster 2021-07-31T02:00:31.982899+0000 mgr.x (mgr.24542) 574 : cluster [DBG] pgmap v1129: 9 pgs: 9 active+clean; 577 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2021-07-31T02:00:33.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:33 smithi154 conmon[34553]: cluster 2021-07-31T02:00:33.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:33 smithi154 conmon[34553]: 2021-07-31T02:00:33.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:33 smithi154 conmon[34553]: 2021 2021-07-31T02:00:33.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:33 smithi154 conmon[34553]: - 2021-07-31T02:00:33.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:33 smithi154 conmon[34553]: 07 2021-07-31T02:00:33.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:33 smithi154 conmon[34553]: - 2021-07-31T02:00:33.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:33 smithi154 conmon[34553]: 31 2021-07-31T02:00:33.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:33 smithi154 conmon[34553]: T 2021-07-31T02:00:33.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:33 smithi154 conmon[34553]: 02 2021-07-31T02:00:33.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:33 smithi154 conmon[34553]: : 2021-07-31T02:00:33.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:33 smithi154 conmon[34553]: 00:32.074966+0000 mon.a (mon.0) 1610 : cluster 2021-07-31T02:00:33.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:33 smithi154 conmon[34553]: [DBG] osdmap e884: 8 total, 8 up, 8 in 2021-07-31T02:00:33.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:33 smithi154 conmon[34553]: audit 2021-07-31T02: 2021-07-31T02:00:33.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:33 smithi154 conmon[34553]: 00:32.078430+0000 mon.b (mon.2) 137 : audit [INF] from='client.? 172.21.15.45:0/1437211126' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:00:33.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:33 smithi154 conmon[34553]: audit 2021-07-31T02:00:33.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:33 smithi154 conmon[34553]: 2021-07-31T02 2021-07-31T02:00:33.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:33 smithi154 conmon[34553]: :00:32.082882+0000 mon.a 2021-07-31T02:00:33.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:33 smithi154 conmon[34553]: (mon.0) 1611 : audit 2021-07-31T02:00:33.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:33 smithi154 conmon[34553]: [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:00:33.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:33 smithi045 conmon[33655]: cluster 2021-07-31T02:00:32.074966+0000 mon.a (mon.0) 1610 : cluster [DBG] osdmap e884: 8 total, 8 up, 8 in 2021-07-31T02:00:33.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:33 smithi045 conmon[33655]: audit 2021-07-31T02:00:32.078430+0000 mon.b (mon.2) 137 : audit [INF] from='client.? 172.21.15.45:0/1437211126' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:00:33.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:33 smithi045 conmon[33655]: audit 2021-07-31T02:00:32.082882+0000 mon.a (mon.0) 1611 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:00:33.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:33 smithi045 conmon[39718]: cluster 2021-07-31T02:00:33.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:33 smithi045 conmon[39718]: 2021-07-31T 2021-07-31T02:00:33.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:33 smithi045 conmon[39718]: 02:00:32.074966+0000 mon.a (mon.0) 1610 : cluster [DBG] osdmap e884: 8 total, 8 up, 8 in 2021-07-31T02:00:33.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:33 smithi045 conmon[39718]: audit 2021- 2021-07-31T02:00:33.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:33 smithi045 conmon[39718]: 07-31T02:00:32.078430+0000 mon.b (mon.2) 137 : audit 2021-07-31T02:00:33.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:33 smithi045 conmon[39718]: [INF] from='client.? 172.21.15.45:0/1437211126' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:00:33.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:33 smithi045 conmon[39718]: audit 2021-07-31T02 2021-07-31T02:00:33.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:33 smithi045 conmon[39718]: :00:32.082882+0000 mon.a (mon.0) 1611 2021-07-31T02:00:33.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:33 smithi045 conmon[39718]: : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:00:34.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:00:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:00:33.531Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:00:34.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:00:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:00:33.531Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:00:34.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:00:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:00:33.531Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:00:34.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:34 smithi154 conmon[34553]: audit 2021-07-31T02:00:33.083217+0000 mon.a ( 2021-07-31T02:00:34.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:34 smithi154 conmon[34553]: mon.0) 1612 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-19","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:00:34.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:34 smithi154 conmon[34553]: cluster 2021-07-31T02:00:33.083508+0000 mon.a (mon.0) 1613 : cluster [DBG] osdmap e885: 8 total, 8 up, 8 in 2021-07-31T02:00:34.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:34 smithi154 conmon[34553]: cluster 2021-07-31 2021-07-31T02:00:34.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:34 smithi154 conmon[34553]: T02:00:33.983299+0000 mgr.x (mgr.24542) 575 : cluster [DBG] pgmap v1132: 41 pgs: 41 active+clean; 707 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 215 KiB/s rd, 1.2 MiB/s wr, 1.43k op/s 2021-07-31T02:00:34.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:34 smithi045 conmon[33655]: audit 2021-07-31T02:00:33.083217+0000 mon.a (mon. 2021-07-31T02:00:34.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:34 smithi045 conmon[33655]: 0) 1612 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-19","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:00:34.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:34 smithi045 conmon[33655]: cluster 2021-07-31T02:00:33.083508+0000 mon.a (mon.0) 1613 : cluster [DBG] osdmap e885: 8 total, 8 up, 8 in 2021-07-31T02:00:34.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:34 smithi045 conmon[33655]: cluster 2021-07-31T02: 2021-07-31T02:00:34.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:34 smithi045 conmon[33655]: 00:33.983299+0000 mgr.x (mgr.24542) 575 : cluster [DBG] pgmap v1132: 41 pgs: 41 active+clean; 707 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 215 KiB/s rd, 1.2 MiB/s wr, 1.43k op/s 2021-07-31T02:00:34.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:34 smithi045 conmon[39718]: audit 2021-07-31T02:00:34.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:34 smithi045 conmon[39718]: 2021-07 2021-07-31T02:00:34.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:34 smithi045 conmon[39718]: -31T02:00:33.083217+0000 mon.a (mon.0) 1612 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-19","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:00:34.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:34 smithi045 conmon[39718]: 2021-07-31T02:00:34.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:34 smithi045 conmon[39718]: cluster 2021 2021-07-31T02:00:34.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:34 smithi045 conmon[39718]: -07-31T02:00:33.083508+0000 2021-07-31T02:00:34.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:34 smithi045 conmon[39718]: mon.a (mon.0) 1613 : cluster [DBG] osdmap e885: 8 total, 8 up, 8 in 2021-07-31T02:00:34.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:34 smithi045 conmon[39718]: cluster 2021-07-31T02:00:33.983299+0000 mgr.x (mgr 2021-07-31T02:00:34.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:34 smithi045 conmon[39718]: .24542) 575 : cluster [DBG] pgmap v1132: 41 pgs: 41 active+clean; 707 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 215 KiB/s rd, 1.2 MiB/s wr, 1.43k op/s 2021-07-31T02:00:35.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:35 smithi154 conmon[34553]: cluster 2021-07-31T02:00:34.095341+0000 mon.a (mon. 2021-07-31T02:00:35.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:35 smithi154 conmon[34553]: 0) 1614 : cluster [DBG] osdmap e886: 8 total, 8 up, 8 in 2021-07-31T02:00:35.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:35 smithi045 conmon[33655]: cluster 2021-07-31T02:00:34.095341+0000 mon.a ( 2021-07-31T02:00:35.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:35 smithi045 conmon[33655]: mon.0) 1614 : cluster [DBG] osdmap e886: 8 total, 8 up, 8 in 2021-07-31T02:00:35.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:35 smithi045 conmon[39718]: cluster 2021-07-31T02:00:34. 2021-07-31T02:00:35.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:35 smithi045 conmon[39718]: 095341+0000 mon.a (mon.0) 1614 : cluster [DBG] osdmap e886: 8 total, 8 up, 8 in 2021-07-31T02:00:36.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:36 smithi154 conmon[34553]: cluster 2021-07-31T02:00:35.983614+0000 mgr.x (mgr.24542) 576 : 2021-07-31T02:00:36.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:36 smithi154 conmon[34553]: cluster [DBG] pgmap v1134: 41 pgs: 41 active+clean; 707 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 175 KiB/s rd, 980 KiB/s wr, 1.16k op/s 2021-07-31T02:00:36.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:36 smithi045 conmon[39718]: cluster 2021-07-31T02:00:35.983614 2021-07-31T02:00:36.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:36 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 576 : cluster [DBG] pgmap v1134: 41 pgs: 41 active+clean; 707 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 175 KiB/s rd, 980 KiB/s wr, 1.16k op/s 2021-07-31T02:00:36.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:36 smithi045 conmon[33655]: cluster 2021-07-31T02:00:35.983614+0000 2021-07-31T02:00:36.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:36 smithi045 conmon[33655]: mgr.x (mgr.24542) 576 : cluster [DBG] pgmap v1134: 41 pgs: 41 active+clean; 707 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 175 KiB/s rd, 980 KiB/s wr, 1.16k op/s 2021-07-31T02:00:39.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:39 smithi154 conmon[34553]: cluster 2021-07-31T02:00:37.983941+0000 mgr.x (mgr.24542) 577 : cluster 2021-07-31T02:00:39.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:39 smithi154 conmon[34553]: [DBG] pgmap v1135: 41 pgs: 41 active+clean; 707 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 144 KiB/s rd, 805 KiB/s wr, 950 op/s 2021-07-31T02:00:39.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:39 smithi045 conmon[33655]: cluster 2021-07-31T02:00:37.983941+0000 mgr.x (mgr.24542) 577 : cluster 2021-07-31T02:00:39.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:39 smithi045 conmon[33655]: [DBG] pgmap v1135: 41 pgs: 41 active+clean; 707 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 144 KiB/s rd, 805 KiB/s wr, 950 op/s 2021-07-31T02:00:39.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:39 smithi045 conmon[39718]: cluster 2021-07-31T02 2021-07-31T02:00:39.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:39 smithi045 conmon[39718]: :00:37.983941+0000 mgr.x (mgr.24542) 577 : cluster [DBG] pgmap v1135: 41 pgs: 41 active+clean; 707 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 144 KiB/s rd, 805 KiB/s wr, 950 op/s 2021-07-31T02:00:41.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:41 smithi154 conmon[34553]: cluster 2021-07-31T02:00:39.984357 2021-07-31T02:00:41.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:41 smithi154 conmon[34553]: +0000 mgr.x (mgr.24542) 578 : cluster [DBG] pgmap v1136: 41 pgs: 41 active+clean; 1.3 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 748 KiB/s rd, 3.8 MiB/s wr, 4.63k op/s 2021-07-31T02:00:41.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:41 smithi045 conmon[39718]: cluster 2021-07-31T02:00:39.984357 2021-07-31T02:00:41.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:41 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 578 : cluster [DBG] pgmap v1136: 41 pgs: 41 active+clean; 1.3 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 748 KiB/s rd, 3.8 MiB/s wr, 4.63k op/s 2021-07-31T02:00:41.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:41 smithi045 conmon[33655]: cluster 2021-07-31T02:00:39.984357 2021-07-31T02:00:41.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:41 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 578 : cluster [DBG] pgmap v1136: 41 pgs: 41 active+clean; 1.3 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 748 KiB/s rd, 3.8 MiB/s wr, 4.63k op/s 2021-07-31T02:00:43.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:42 smithi154 conmon[34553]: cluster 2021-07-31T02:00:41.984649+0000 mgr.x (mgr 2021-07-31T02:00:43.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:42 smithi154 conmon[34553]: .24542) 579 : cluster [DBG] pgmap v1137: 41 pgs: 41 active+clean; 1.3 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 664 KiB/s rd, 3.4 MiB/s wr, 4.11k op/s 2021-07-31T02:00:43.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:42 smithi045 conmon[33655]: cluster 2021-07-31T02:00:41.984649+0000 mgr.x (mgr 2021-07-31T02:00:43.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:42 smithi045 conmon[33655]: .24542) 579 : cluster [DBG] pgmap v1137: 41 pgs: 41 active+clean; 1.3 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 664 KiB/s rd, 3.4 MiB/s wr, 4.11k op/s 2021-07-31T02:00:43.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:42 smithi045 conmon[39718]: cluster 2021-07-31T02:00:41.984649+0000 2021-07-31T02:00:43.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:42 smithi045 conmon[39718]: mgr.x (mgr.24542) 579 : cluster [DBG] pgmap v1137: 41 pgs: 41 active+clean; 1.3 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 664 KiB/s rd, 3.4 MiB/s wr, 4.11k op/s 2021-07-31T02:00:44.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:00:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:00:43.531Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:00:44.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:00:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:00:43.531Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:00:44.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:00:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:00:43.531Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:00:45.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:45 smithi154 conmon[34553]: cluster 2021-07-31T02:00:43.985074+0000 mgr.x (mgr.24542) 2021-07-31T02:00:45.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:45 smithi154 conmon[34553]: 580 : cluster [DBG] pgmap v1138: 41 pgs: 41 active+clean; 1.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 905 KiB/s rd, 4.5 MiB/s wr, 5.54k op/s 2021-07-31T02:00:45.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:45 smithi045 conmon[33655]: cluster 2021-07-31T02:00:43.985074+0000 mgr.x 2021-07-31T02:00:45.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:45 smithi045 conmon[33655]: (mgr.24542) 580 : cluster [DBG] pgmap v1138: 41 pgs: 41 active+clean; 1.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 905 KiB/s rd, 4.5 MiB/s wr, 5.54k op/s 2021-07-31T02:00:45.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:45 smithi045 conmon[39718]: cluster 2021-07-31T02:00:43.985074+0000 mgr.x 2021-07-31T02:00:45.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:45 smithi045 conmon[39718]: (mgr.24542) 580 : cluster [DBG] pgmap v1138: 41 pgs: 41 active+clean; 1.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 905 KiB/s rd, 4.5 MiB/s wr, 5.54k op/s 2021-07-31T02:00:47.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:47 smithi154 conmon[34553]: cluster 2021-07-31T02:00:45.985423+0000 mgr.x (mgr.24542 2021-07-31T02:00:47.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:47 smithi154 conmon[34553]: ) 581 : cluster [DBG] pgmap v1139: 41 pgs: 41 active+clean; 1.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 761 KiB/s rd, 3.8 MiB/s wr, 4.66k op/s 2021-07-31T02:00:47.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:47 smithi045 conmon[39718]: cluster 2021-07-31T02 2021-07-31T02:00:47.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:47 smithi045 conmon[39718]: :00:45.985423+0000 mgr.x (mgr.24542) 581 : cluster [DBG] pgmap v1139: 41 pgs: 41 active+clean; 1.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 761 KiB/s rd, 3.8 MiB/s wr, 4.66k op/s 2021-07-31T02:00:47.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:47 smithi045 conmon[33655]: cluster 2021-07-31T02:00:45.985423+0000 mgr.x (mgr.24542 2021-07-31T02:00:47.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:47 smithi045 conmon[33655]: ) 581 : cluster [DBG] pgmap v1139: 41 pgs: 41 active+clean; 1.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 761 KiB/s rd, 3.8 MiB/s wr, 4.66k op/s 2021-07-31T02:00:49.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:49 smithi154 conmon[34553]: cluster 2021-07-31T02:00:47.985736+0000 mgr.x (mgr.24542) 582 : cluster 2021-07-31T02:00:49.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:49 smithi154 conmon[34553]: [DBG] pgmap v1140: 41 pgs: 41 active+clean; 1.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 754 KiB/s rd, 3.8 MiB/s wr, 4.62k op/s 2021-07-31T02:00:49.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:49 smithi045 conmon[33655]: cluster 2021-07-31T02:00:47.985736+0000 mgr.x (mgr.24542) 582 : cluster [DBG] 2021-07-31T02:00:49.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:49 smithi045 conmon[33655]: pgmap v1140: 41 pgs: 41 active+clean; 1.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 754 KiB/s rd, 3.8 MiB/s wr, 4.62k op/s 2021-07-31T02:00:49.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:49 smithi045 conmon[39718]: cluster 2021-07-31T02:00:47.985736+0000 2021-07-31T02:00:49.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:49 smithi045 conmon[39718]: mgr.x (mgr.24542) 582 : cluster [DBG] pgmap v1140: 41 pgs: 41 active+clean; 1.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 754 KiB/s rd, 3.8 MiB/s wr, 4.62k op/s 2021-07-31T02:00:49.601 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:00:49.944 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:00:49.945 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 15m ago 16m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:00:49.945 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 13m ago 16m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:00:49.945 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 13m ago 19m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:00:49.946 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 15m ago 21m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:00:49.946 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 15m ago 21m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:00:49.946 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 13m ago 20m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:00:49.946 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 15m ago 20m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:00:49.946 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 15m ago 16m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:00:49.947 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 13m ago 16m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:00:49.947 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 15m ago 19m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:00:49.947 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 15m ago 19m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:00:49.947 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 15m ago 18m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:00:49.948 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 15m ago 18m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:00:49.948 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 13m ago 18m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:00:49.948 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 13m ago 17m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:00:49.948 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 13m ago 17m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:00:49.949 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 13m ago 17m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:00:49.949 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 13m ago 16m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:00:50.350 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:00:50.351 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:00:50.351 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:00:50.351 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:00:50.352 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:00:50.352 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:00:50.352 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:00:50.353 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:00:50.353 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:00:50.353 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:00:50.353 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:00:50.353 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:00:50.354 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:00:50.354 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:00:50.354 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:00:50.354 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:00:50.355 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:00:51.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:51 smithi154 conmon[34553]: audit 2021-07-31T02:00:49.588021+0000 mgr.x (mgr 2021-07-31T02:00:51.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:51 smithi154 conmon[34553]: .24542) 583 : audit [DBG] from='client.25219 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:00:51.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:51 smithi154 conmon[34553]: audit 2021-07-31T02:00:49.934938+0000 mgr.x (mgr.24542) 584 : audit [DBG] from='client.25225 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:00:51.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:51 smithi154 conmon[34553]: cluster 2021- 2021-07-31T02:00:51.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:51 smithi154 conmon[34553]: 07-31T02:00:49. 2021-07-31T02:00:51.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:51 smithi154 conmon[34553]: 986154+0000 mgr.x (mgr.24542) 2021-07-31T02:00:51.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:51 smithi154 conmon[34553]: 585 : cluster [DBG] pgmap v1141: 41 pgs: 41 active+clean; 1.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.0 MiB/s rd, 4.7 MiB/s wr, 5.87k op/s 2021-07-31T02:00:51.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:51 smithi154 conmon[34553]: audit 2021-07- 2021-07-31T02:00:51.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:51 smithi154 conmon[34553]: 31T02:00:50.350356+0000 2021-07-31T02:00:51.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:51 smithi154 conmon[34553]: mon.a (mon.0) 1615 : audit 2021-07-31T02:00:51.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:51 smithi154 conmon[34553]: [DBG] from='client.? 172.21.15.45:0/2246239068' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:00:51.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:51 smithi045 conmon[33655]: audit 2021-07-31T02:00:49.588021+0000 mgr.x (mgr 2021-07-31T02:00:51.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:51 smithi045 conmon[33655]: .24542) 583 : audit [DBG] from='client.25219 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:00:51.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:51 smithi045 conmon[33655]: audit 2021-07-31T02:00:49.934938+0000 mgr.x (mgr.24542) 584 : audit 2021-07-31T02:00:51.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:51 smithi045 conmon[33655]: [DBG] from='client.25225 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:00:51.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:51 smithi045 conmon[33655]: cluster 2021-07-31T02:00:51.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:51 smithi045 conmon[33655]: 2021-07-31T02:00:49.986154+0000 mgr.x (mgr.24542) 2021-07-31T02:00:51.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:51 smithi045 conmon[33655]: 585 : cluster [DBG] pgmap v1141: 41 pgs: 41 active+clean; 1.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.0 MiB/s rd, 4.7 MiB/s wr, 5.87k op/s 2021-07-31T02:00:51.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:51 smithi045 conmon[33655]: audit 2021 2021-07-31T02:00:51.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:51 smithi045 conmon[33655]: -07-31T02:00:50.350356+0000 mon.a ( 2021-07-31T02:00:51.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:51 smithi045 conmon[33655]: mon.0) 1615 : audit [DBG] from='client.? 172.21.15.45:0/2246239068' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:00:51.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:51 smithi045 conmon[39718]: audit 2021-07-31T02: 2021-07-31T02:00:51.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:51 smithi045 conmon[39718]: 00:49.588021+0000 mgr.x (mgr.24542) 583 : audit [DBG] from='client.25219 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:00:51.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:51 smithi045 conmon[39718]: audit 2021- 2021-07-31T02:00:51.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:51 smithi045 conmon[39718]: 07-31T02:00:49.934938+0000 mgr.x ( 2021-07-31T02:00:51.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:51 smithi045 conmon[39718]: mgr.24542) 584 : audit [DBG] 2021-07-31T02:00:51.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:51 smithi045 conmon[39718]: from='client.25225 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:00:51.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:51 smithi045 conmon[39718]: cluster 2021-07-31T02:00:51.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:51 smithi045 conmon[39718]: 2021-07-31T 2021-07-31T02:00:51.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:51 smithi045 conmon[39718]: 02:00:49.986154+0000 mgr.x 2021-07-31T02:00:51.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:51 smithi045 conmon[39718]: (mgr.24542) 585 : cluster 2021-07-31T02:00:51.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:51 smithi045 conmon[39718]: [DBG] pgmap v1141: 41 pgs: 41 active+clean; 1.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.0 MiB/s rd, 4.7 MiB/s wr, 5.87k op/s 2021-07-31T02:00:51.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:51 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T02:00:51.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:51 smithi045 conmon[39718]: 31T02:00:50.350356+0000 mon.a 2021-07-31T02:00:51.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:51 smithi045 conmon[39718]: (mon.0) 1615 : 2021-07-31T02:00:51.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:51 smithi045 conmon[39718]: audit [DBG] from='client.? 172.21.15.45:0/2246239068' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:00:53.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:52 smithi154 conmon[34553]: cluster 2021-07-31T02:00:51.986515+0000 mgr.x (mgr.24542) 586 : 2021-07-31T02:00:53.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:52 smithi154 conmon[34553]: cluster [DBG] pgmap v1142: 41 pgs: 41 active+clean; 1.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 645 KiB/s rd, 2.6 MiB/s wr, 3.29k op/s 2021-07-31T02:00:53.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:52 smithi045 conmon[39718]: cluster 2021-07-31T02:00:51.986515 2021-07-31T02:00:53.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:52 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 586 : cluster [DBG] pgmap v1142: 41 pgs: 41 active+clean; 1.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 645 KiB/s rd, 2.6 MiB/s wr, 3.29k op/s 2021-07-31T02:00:53.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:52 smithi045 conmon[33655]: cluster 2021-07-31T02:00:51.986515+0000 mgr.x ( 2021-07-31T02:00:53.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:52 smithi045 conmon[33655]: mgr.24542) 586 : cluster [DBG] pgmap v1142: 41 pgs: 41 active+clean; 1.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 645 KiB/s rd, 2.6 MiB/s wr, 3.29k op/s 2021-07-31T02:00:54.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:00:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:00:53.532Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:00:54.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:00:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:00:53.532Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:00:54.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:00:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:00:53.532Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:00:55.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:55 smithi154 conmon[34553]: cluster 2021-07-31T02:00:53.986951+0000 mgr.x (mgr.24542) 587 : cluster 2021-07-31T02:00:55.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:55 smithi154 conmon[34553]: [DBG] pgmap v1143: 41 pgs: 41 active+clean; 1.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 885 KiB/s rd, 3.2 MiB/s wr, 4.21k op/s 2021-07-31T02:00:55.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:55 smithi045 conmon[39718]: cluster 2021-07-31T02:00:53.986951+0000 mgr.x (mgr.24542) 587 : cluster [DBG] pgmap v1143: 41 pgs: 41 active+clean; 1.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 885 KiB/s rd, 3.2 MiB/s wr, 4.21k op/s 2021-07-31T02:00:55.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:55 smithi045 conmon[33655]: cluster 2021-07-31T02:00:53.986951+0000 mgr.x (mgr.24542 2021-07-31T02:00:55.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:55 smithi045 conmon[33655]: ) 587 : cluster [DBG] pgmap v1143: 41 pgs: 41 active+clean; 1.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 885 KiB/s rd, 3.2 MiB/s wr, 4.21k op/s 2021-07-31T02:00:57.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:57 smithi154 conmon[34553]: cluster 2021-07-31T02:00:55.987251+0000 mgr.x (mgr.24542) 588 2021-07-31T02:00:57.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:57 smithi154 conmon[34553]: : cluster [DBG] pgmap v1144: 41 pgs: 41 active+clean; 1.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 551 KiB/s rd, 1.6 MiB/s wr, 2.17k op/s 2021-07-31T02:00:57.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:57 smithi045 conmon[33655]: cluster 2021-07-31T02:00:55.987251+0000 2021-07-31T02:00:57.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:57 smithi045 conmon[33655]: mgr.x (mgr.24542) 588 : cluster [DBG] pgmap v1144: 41 pgs: 41 active+clean; 1.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 551 KiB/s rd, 1.6 MiB/s wr, 2.17k op/s 2021-07-31T02:00:57.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:57 smithi045 conmon[39718]: cluster 2021-07-31T02:00:55.987251+0000 2021-07-31T02:00:57.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:57 smithi045 conmon[39718]: mgr.x (mgr.24542) 588 : cluster [DBG] pgmap v1144: 41 pgs: 41 active+clean; 1.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 551 KiB/s rd, 1.6 MiB/s wr, 2.17k op/s 2021-07-31T02:00:59.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:59 smithi154 conmon[34553]: cluster 2021-07-31T02:00:57.987603 2021-07-31T02:00:59.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:00:59 smithi154 conmon[34553]: +0000 mgr.x (mgr.24542) 589 : cluster [DBG] pgmap v1145: 41 pgs: 41 active+clean; 1.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 551 KiB/s rd, 1.6 MiB/s wr, 2.17k op/s 2021-07-31T02:00:59.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:59 smithi045 conmon[33655]: cluster 2021-07-31T02:00: 2021-07-31T02:00:59.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:00:59 smithi045 conmon[33655]: 57.987603+0000 mgr.x (mgr.24542) 589 : cluster [DBG] pgmap v1145: 41 pgs: 41 active+clean; 1.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 551 KiB/s rd, 1.6 MiB/s wr, 2.17k op/s 2021-07-31T02:00:59.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:59 smithi045 conmon[39718]: cluster 2021-07-31T02:00 2021-07-31T02:00:59.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:00:59 smithi045 conmon[39718]: :57.987603+0000 mgr.x (mgr.24542) 589 : cluster [DBG] pgmap v1145: 41 pgs: 41 active+clean; 1.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 551 KiB/s rd, 1.6 MiB/s wr, 2.17k op/s 2021-07-31T02:01:01.055 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestCls2PCQueue.MultiProducerConsumer (29994 ms) 2021-07-31T02:01:01.055 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestCls2PCQueue.ReserveSpillover 2021-07-31T02:01:01.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:01 smithi154 conmon[34553]: cluster 2021-07-31T02:00:59.988102+0000 mgr.x (mgr.24542) 590 : cluster 2021-07-31T02:01:01.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:01 smithi154 conmon[34553]: [DBG] pgmap v1146: 41 pgs: 41 active+clean; 1.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 796 KiB/s rd, 2.1 MiB/s wr, 3.00k op/s 2021-07-31T02:01:01.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:01 smithi045 conmon[39718]: cluster 2021-07-31T02:00:59.988102 2021-07-31T02:01:01.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:01 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 590 : cluster [DBG] pgmap v1146: 41 pgs: 41 active+clean; 1.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 796 KiB/s rd, 2.1 MiB/s wr, 3.00k op/s 2021-07-31T02:01:01.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:01 smithi045 conmon[33655]: cluster 2021-07-31T02:00:59.988102+0000 mgr.x 2021-07-31T02:01:01.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:01 smithi045 conmon[33655]: (mgr.24542) 590 : cluster [DBG] pgmap v1146: 41 pgs: 41 active+clean; 1.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 796 KiB/s rd, 2.1 MiB/s wr, 3.00k op/s 2021-07-31T02:01:02.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:02 smithi154 conmon[34553]: cluster 2021-07-31T02:01:01.053313+0000 mon.a (mon.0) 1616 : cluster [DBG] osdmap e887: 8 total, 8 up, 8 in 2021-07-31T02:01:02.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:02 smithi154 conmon[34553]: 2021-07-31T02:01:02.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:02 smithi045 conmon[39718]: cluster 2021-07-31T02:01:01.053313+0000 mon.a (mon.0) 2021-07-31T02:01:02.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:02 smithi045 conmon[39718]: 1616 : cluster [DBG] osdmap e887: 8 total, 8 up, 8 in 2021-07-31T02:01:02.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:02 smithi045 conmon[33655]: cluster 2021-07-31T02:01 2021-07-31T02:01:02.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:02 smithi045 conmon[33655]: :01.053313+0000 mon.a (mon.0) 1616 : cluster [DBG] osdmap e887: 8 total, 8 up, 8 in 2021-07-31T02:01:03.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:03 smithi154 conmon[34553]: cluster 2021-07-31T02:01:01.988600+0000 mgr.x (mgr.24542) 591 : cluster 2021-07-31T02:01:03.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:03 smithi154 conmon[34553]: [DBG] pgmap v1148: 9 pgs: 9 active+clean; 577 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail 2021-07-31T02:01:03.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:03 smithi154 conmon[34553]: cluster 2021-07-31T02:01:02.060174+0000 mon.a (mon.0) 1617 : cluster [DBG] osdmap e888: 8 total, 8 up, 8 in 2021-07-31T02:01:03.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:03 smithi154 conmon[34553]: audit 2021-07- 2021-07-31T02:01:03.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:03 smithi154 conmon[34553]: 31T02:01:02.063318+0000 mon.a (mon.0) 1618 : audit [INF] from='client.? 172.21.15.45:0/1428527449' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:01:03.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:03 smithi045 conmon[33655]: cluster 2021-07-31T02:01:01.988600+0000 mgr.x 2021-07-31T02:01:03.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:03 smithi045 conmon[33655]: (mgr.24542) 591 : cluster [DBG] pgmap v1148: 9 pgs: 9 active+clean; 577 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail 2021-07-31T02:01:03.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:03 smithi045 conmon[33655]: cluster 2021-07-31T02:01:02.060174+0000 mon.a (mon.0) 1617 : cluster [DBG] osdmap e888: 8 total, 8 up, 8 in 2021-07-31T02:01:03.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:03 smithi045 conmon[33655]: audit 2021-07-31T02:01:02.063318+0000 mon.a (mon.0) 1618 : audit [INF] from='client.? 172.21.15.45:0/1428527449' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:01:03.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:03 smithi045 conmon[39718]: cluster 2021-07-31T02:01:03.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:03 smithi045 conmon[39718]: 2021-07-31T02:01:03.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:03 smithi045 conmon[39718]: 2021 2021-07-31T02:01:03.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:03 smithi045 conmon[39718]: - 2021-07-31T02:01:03.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:03 smithi045 conmon[39718]: 07 2021-07-31T02:01:03.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:03 smithi045 conmon[39718]: - 2021-07-31T02:01:03.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:03 smithi045 conmon[39718]: 31 2021-07-31T02:01:03.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:03 smithi045 conmon[39718]: T 2021-07-31T02:01:03.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:03 smithi045 conmon[39718]: 02 2021-07-31T02:01:03.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:03 smithi045 conmon[39718]: : 2021-07-31T02:01:03.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:03 smithi045 conmon[39718]: 01 2021-07-31T02:01:03.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:03 smithi045 conmon[39718]: : 2021-07-31T02:01:03.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:03 smithi045 conmon[39718]: 01 2021-07-31T02:01:03.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:03 smithi045 conmon[39718]: . 2021-07-31T02:01:03.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:03 smithi045 conmon[39718]: 988600 2021-07-31T02:01:03.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:03 smithi045 conmon[39718]: +0000 2021-07-31T02:01:03.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:03 smithi045 conmon[39718]: 2021-07-31T02:01:03.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:03 smithi045 conmon[39718]: mgr.x 2021-07-31T02:01:03.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:03 smithi045 conmon[39718]: ( 2021-07-31T02:01:03.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:03 smithi045 conmon[39718]: mgr 2021-07-31T02:01:03.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:03 smithi045 conmon[39718]: . 2021-07-31T02:01:03.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:03 smithi045 conmon[39718]: 24542 2021-07-31T02:01:03.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:03 smithi045 conmon[39718]: ) 2021-07-31T02:01:03.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:03 smithi045 conmon[39718]: 591 2021-07-31T02:01:03.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:03 smithi045 conmon[39718]: : 2021-07-31T02:01:03.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:03 smithi045 conmon[39718]: cluster 2021-07-31T02:01:03.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:03 smithi045 conmon[39718]: 2021-07-31T02:01:03.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:03 smithi045 conmon[39718]: [DBG] 2021-07-31T02:01:03.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:03 smithi045 conmon[39718]: 2021-07-31T02:01:03.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:03 smithi045 conmon[39718]: pgmap v1148: 9 pgs: 9 active+clean; 577 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail 2021-07-31T02:01:03.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:03 smithi045 conmon[39718]: 2021-07-31T02:01:03.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:03 smithi045 conmon[39718]: cluster 2021-07-31T02:01:03.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:03 smithi045 conmon[39718]: 2021-07-31T02:01:02 2021-07-31T02:01:03.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:03 smithi045 conmon[39718]: .060174+0000 mon.a (mon.0) 1617 : cluster [DBG] osdmap e888: 8 total, 8 up, 8 in 2021-07-31T02:01:03.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:03 smithi045 conmon[39718]: audit 2021-07-31T02:01:02.063318 2021-07-31T02:01:03.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:03 smithi045 conmon[39718]: +0000 mon.a (mon.0) 1618 : audit [INF] from='client.? 172.21.15.45:0/1428527449' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:01:03.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:03 smithi045 conmon[39718]: 2021-07-31T02:01:04.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:01:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:01:03.532Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:01:04.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:01:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:01:03.532Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:01:04.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:01:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:01:03.532Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:01:04.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:04 smithi154 conmon[34553]: audit 2021-07-31T02:01:03.063077+0000 mon.a ( 2021-07-31T02:01:04.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:04 smithi154 conmon[34553]: mon.0) 1619 : audit [INF] from='client.? 172.21.15.45:0/1428527449' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-20","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:01:04.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:04 smithi154 conmon[34553]: cluster 2021-07-31T02:01:03.063277+0000 mon.a (mon.0) 1620 : cluster [DBG] osdmap e889: 8 total, 8 up, 8 in 2021-07-31T02:01:04.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:04 smithi154 conmon[34553]: cluster 2021-07-31T02:01:03.988986+0000 2021-07-31T02:01:04.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:04 smithi154 conmon[34553]: mgr.x (mgr.24542) 592 : cluster [DBG] pgmap v1151: 41 pgs: 41 active+clean; 588 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail 2021-07-31T02:01:04.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:04 smithi045 conmon[39718]: audit 2021-07-31T02:01:03 2021-07-31T02:01:04.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:04 smithi045 conmon[39718]: .063077+0000 mon.a (mon.0) 1619 : audit [INF] from='client.? 172.21.15.45:0/1428527449' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-20","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:01:04.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:04 smithi045 conmon[39718]: cluster 2021-07-31T02:01: 2021-07-31T02:01:04.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:04 smithi045 conmon[39718]: 03.063277+0000 mon.a (mon.0) 1620 : 2021-07-31T02:01:04.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:04 smithi045 conmon[39718]: cluster [DBG] osdmap e889: 8 total, 8 up, 8 in 2021-07-31T02:01:04.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:04 smithi045 conmon[39718]: cluster 2021-07-31T02:01:04.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:04 smithi045 conmon[39718]: 2021-07-31T02:01:03.988986 2021-07-31T02:01:04.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:04 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 592 : cluster [DBG] 2021-07-31T02:01:04.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:04 smithi045 conmon[39718]: pgmap v1151: 41 pgs: 41 active+clean; 588 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail 2021-07-31T02:01:04.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:04 smithi045 conmon[33655]: audit 2021-07-31T02:01:03.063077+0000 mon.a (mon.0) 2021-07-31T02:01:04.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:04 smithi045 conmon[33655]: 1619 : audit [INF] from='client.? 172.21.15.45:0/1428527449' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-20","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:01:04.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:04 smithi045 conmon[33655]: cluster 2021-07-31T02:01:03.063277+0000 mon.a (mon.0) 1620 : cluster 2021-07-31T02:01:04.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:04 smithi045 conmon[33655]: [DBG] osdmap e889: 8 total, 8 up, 8 in 2021-07-31T02:01:04.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:04 smithi045 conmon[33655]: cluster 2021-07-31T02:01:04.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:04 smithi045 conmon[33655]: 2021-07-31T02 2021-07-31T02:01:04.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:04 smithi045 conmon[33655]: :01:03.988986+0000 mgr.x (mgr. 2021-07-31T02:01:04.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:04 smithi045 conmon[33655]: 24542) 592 : cluster [DBG] pgmap v1151: 41 pgs: 41 active+clean; 588 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail 2021-07-31T02:01:05.083 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestCls2PCQueue.ReserveSpillover (4028 ms) 2021-07-31T02:01:05.083 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestCls2PCQueue.CommitSpillover 2021-07-31T02:01:05.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:05 smithi154 conmon[34553]: cluster 2021-07-31T02:01:04.076000+0000 mon.a (mon.0) 1621 2021-07-31T02:01:05.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:05 smithi154 conmon[34553]: : cluster [DBG] osdmap e890: 8 total, 8 up, 8 in 2021-07-31T02:01:05.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:05 smithi045 conmon[33655]: cluster 2021-07-31T02:01:04.076000+0000 mon.a (mon.0) 1621 : cluster [DBG] osdmap e890: 8 total, 8 up, 8 in 2021-07-31T02:01:05.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:05 smithi045 conmon[39718]: cluster 2021-07-31T02:01:04.076000+0000 mon.a 2021-07-31T02:01:05.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:05 smithi045 conmon[39718]: (mon.0) 1621 : cluster [DBG] osdmap e890: 8 total, 8 up, 8 in 2021-07-31T02:01:06.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:06 smithi154 conmon[34553]: cluster 2021-07-31T02:01:05.080805+0000 mon.a (mon.0) 1622 2021-07-31T02:01:06.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:06 smithi154 conmon[34553]: : cluster [DBG] osdmap e891: 8 total, 8 up, 8 in 2021-07-31T02:01:06.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:06 smithi154 conmon[34553]: cluster 2021-07-31T02:01:05.989361+0000 mgr.x (mgr.24542) 593 : cluster [DBG] pgmap v1154: 9 pgs: 9 active+clean; 577 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail 2021-07-31T02:01:06.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:06 smithi045 conmon[39718]: cluster 2021-07-31T02:01:05.080805+0000 2021-07-31T02:01:06.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:06 smithi045 conmon[39718]: mon.a (mon.0) 1622 : cluster [DBG] osdmap e891: 8 total, 8 up, 8 in 2021-07-31T02:01:06.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:06 smithi045 conmon[39718]: cluster 2021-07-31T02: 2021-07-31T02:01:06.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:06 smithi045 conmon[39718]: 01:05.989361+0000 mgr.x (mgr.24542) 593 : cluster [DBG] pgmap v1154: 9 pgs: 9 active+clean; 577 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail 2021-07-31T02:01:06.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:06 smithi045 conmon[33655]: cluster 2021-07-31T02:01:05.080805+0000 mon.a (mon.0 2021-07-31T02:01:06.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:06 smithi045 conmon[33655]: ) 1622 : cluster [DBG] osdmap e891: 8 total, 8 up, 8 in 2021-07-31T02:01:06.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:06 smithi045 conmon[33655]: cluster 2021-07-31T02:01:05.989361+0000 mgr.x (mgr.24542) 593 : cluster [DBG] pgmap v1154: 9 pgs: 9 active+clean; 577 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail 2021-07-31T02:01:07.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:07 smithi154 conmon[34553]: cluster 2021-07-31T02:01:07.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:07 smithi154 conmon[34553]: 2021-07-31T02:01:07.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:07 smithi154 conmon[34553]: 2021 2021-07-31T02:01:07.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:07 smithi154 conmon[34553]: -07-31T02: 2021-07-31T02:01:07.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:07 smithi154 conmon[34553]: 01:06.087390+0000 mon.a (mon.0) 1623 : cluster [DBG] osdmap e892: 8 total, 8 up, 8 in 2021-07-31T02:01:07.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:07 smithi154 conmon[34553]: audit 2021-07-31T02:01:07.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:07 smithi154 conmon[34553]: 2021-07-31T02:01:06.089469+0000 mon.c ( 2021-07-31T02:01:07.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:07 smithi154 conmon[34553]: mon.1) 45 : audit [INF] from='client.? 172.21.15.45:0/3039461971' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:01:07.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:07 smithi154 conmon[34553]: audit 2021- 2021-07-31T02:01:07.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:07 smithi154 conmon[34553]: 07-31T02:01:06.090608+0000 mon.a (mon.0 2021-07-31T02:01:07.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:07 smithi154 conmon[34553]: ) 1624 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:01:07.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:07 smithi045 conmon[33655]: cluster 2021-07-31T02:01:07.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:07 smithi045 conmon[33655]: 2021-07-31T02:01:07.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:07 smithi045 conmon[33655]: 2021 2021-07-31T02:01:07.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:07 smithi045 conmon[33655]: - 2021-07-31T02:01:07.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:07 smithi045 conmon[33655]: 07-31T02:01:06 2021-07-31T02:01:07.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:07 smithi045 conmon[33655]: .087390+0000 mon.a (mon.0) 1623 : cluster [DBG] osdmap e892: 8 total, 8 up, 8 in 2021-07-31T02:01:07.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:07 smithi045 conmon[33655]: audit 2021-07-31T02:01:07.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:07 smithi045 conmon[33655]: 2021-07-31T02:01:07.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:07 smithi045 conmon[33655]: 2021-07-31T02:01 2021-07-31T02:01:07.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:07 smithi045 conmon[33655]: :06.089469+0000 mon.c (mon.1) 45 : audit [INF] from='client.? 172.21.15.45:0/3039461971' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:01:07.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:07 smithi045 conmon[33655]: audit 2021-07-31T 2021-07-31T02:01:07.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:07 smithi045 conmon[33655]: 02:01:06.090608+0000 mon.a (mon.0) 1624 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:01:07.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:07 smithi045 conmon[33655]: 2021-07-31T02:01:07.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:07 smithi045 conmon[39718]: cluster 2021-07-31T02:01:07.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:07 smithi045 conmon[39718]: 2021-07-31T02:01:07.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:07 smithi045 conmon[39718]: 2021 2021-07-31T02:01:07.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:07 smithi045 conmon[39718]: - 2021-07-31T02:01:07.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:07 smithi045 conmon[39718]: 07 2021-07-31T02:01:07.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:07 smithi045 conmon[39718]: - 2021-07-31T02:01:07.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:07 smithi045 conmon[39718]: 31T02:01:06.087390+0000 mon.a (mon.0 2021-07-31T02:01:07.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:07 smithi045 conmon[39718]: ) 1623 : cluster [DBG] osdmap e892: 8 total, 8 up, 8 in 2021-07-31T02:01:07.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:07 smithi045 conmon[39718]: audit 2021-07-31T02:01:07.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:07 smithi045 conmon[39718]: 2021-07-31T 2021-07-31T02:01:07.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:07 smithi045 conmon[39718]: 02:01:06.089469+0000 mon.c (mon 2021-07-31T02:01:07.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:07 smithi045 conmon[39718]: .1) 45 : audit [INF] 2021-07-31T02:01:07.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:07 smithi045 conmon[39718]: from='client.? 172.21.15.45:0/3039461971' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:01:07.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:07 smithi045 conmon[39718]: audit 2021-07 2021-07-31T02:01:07.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:07 smithi045 conmon[39718]: -31T02:01:06.090608+0000 mon.a (mon.0 2021-07-31T02:01:07.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:07 smithi045 conmon[39718]: ) 1624 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:01:08.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:08 smithi154 conmon[34553]: audit 2021-07-31T02:01:08.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:08 smithi154 conmon[34553]: 2021-07-31T02:01:08.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:08 smithi154 conmon[34553]: 2021 2021-07-31T02:01:08.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:08 smithi154 conmon[34553]: - 2021-07-31T02:01:08.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:08 smithi154 conmon[34553]: 07 2021-07-31T02:01:08.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:08 smithi154 conmon[34553]: - 2021-07-31T02:01:08.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:08 smithi154 conmon[34553]: 31T02:01:07.089387+0000 mon.a 2021-07-31T02:01:08.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:08 smithi154 conmon[34553]: (mon.0) 1625 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-21","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:01:08.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:08 smithi154 conmon[34553]: cluster 2021-07-31T02:01:08.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:08 smithi154 conmon[34553]: 2021-07-31T02:01: 2021-07-31T02:01:08.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:08 smithi154 conmon[34553]: 07.089682+0000 mon.a (mon.0) 2021-07-31T02:01:08.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:08 smithi154 conmon[34553]: 1626 : cluster [DBG] osdmap e893: 8 total, 8 up, 8 in 2021-07-31T02:01:08.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:08 smithi154 conmon[34553]: cluster 2021-07-31T02:01:08.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:08 smithi154 conmon[34553]: 2021-07- 2021-07-31T02:01:08.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:08 smithi154 conmon[34553]: 31T02:01:07.989718+0000 mgr.x (mgr 2021-07-31T02:01:08.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:08 smithi154 conmon[34553]: .24542) 594 : cluster [DBG] pgmap v1157: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail 2021-07-31T02:01:08.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:08 smithi045 conmon[39718]: audit 2021-07-31T02:01:07.089387+0000 mon.a (mon.0) 1625 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-21","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:01:08.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:08 smithi045 conmon[39718]: cluster 2021-07-31T02:01:07.089682+0000 mon.a (mon.0) 1626 : cluster [DBG] osdmap e893: 8 total, 8 up, 8 in 2021-07-31T02:01:08.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:08 smithi045 conmon[39718]: cluster 2021-07-31T02:01:07.989718+0000 mgr.x (mgr.24542) 594 : cluster [DBG] pgmap v1157: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail 2021-07-31T02:01:08.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:08 smithi045 conmon[33655]: audit 2021-07-31T02:01:07.089387+0000 2021-07-31T02:01:08.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:08 smithi045 conmon[33655]: mon.a (mon.0) 1625 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-21","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:01:08.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:08 smithi045 conmon[33655]: cluster 2021-07-31T02:01:07.089682+0000 mon.a (mon.0) 1626 : cluster 2021-07-31T02:01:08.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:08 smithi045 conmon[33655]: [DBG] osdmap e893: 8 total, 8 up, 8 in 2021-07-31T02:01:08.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:08 smithi045 conmon[33655]: cluster 2021-07-31T02:01:08.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:08 smithi045 conmon[33655]: 2021-07-31T02:01:07.989718+0000 mgr.x ( 2021-07-31T02:01:08.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:08 smithi045 conmon[33655]: mgr.24542) 594 : cluster [DBG] pgmap v1157: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail 2021-07-31T02:01:09.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:09 smithi154 conmon[34553]: cluster 2021-07-31T02:01:08.103129+0000 mon.a (mon.0) 1627 : cluster [DBG] osdmap e894: 8 total, 8 up, 8 in 2021-07-31T02:01:09.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:09 smithi154 conmon[34553]: 2021-07-31T02:01:09.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:09 smithi045 conmon[33655]: cluster 2021-07-31T02:01:08 2021-07-31T02:01:09.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:09 smithi045 conmon[33655]: .103129+0000 mon.a (mon.0) 1627 : cluster [DBG] osdmap e894: 8 total, 8 up, 8 in 2021-07-31T02:01:09.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:09 smithi045 conmon[39718]: cluster 2021-07-31T02:01 2021-07-31T02:01:09.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:09 smithi045 conmon[39718]: :08.103129+0000 mon.a (mon.0) 1627 : cluster [DBG] osdmap e894: 8 total, 8 up, 8 in 2021-07-31T02:01:10.108 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestCls2PCQueue.CommitSpillover (5025 ms) 2021-07-31T02:01:10.109 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestCls2PCQueue.AbortSpillover 2021-07-31T02:01:10.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:10 smithi154 conmon[34553]: cluster 2021-07-31T02:01:09.990099+0000 mgr.x (mgr.24542 2021-07-31T02:01:10.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:10 smithi154 conmon[34553]: ) 595 : cluster [DBG] pgmap v1159: 41 pgs: 41 active+clean; 615 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 3.8 MiB/s rd, 4.0 MiB/s wr, 1.03k op/s 2021-07-31T02:01:10.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:10 smithi045 conmon[39718]: cluster 2021-07-31T02:01:09 2021-07-31T02:01:10.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:10 smithi045 conmon[39718]: .990099+0000 mgr.x (mgr.24542) 595 : cluster [DBG] pgmap v1159: 41 pgs: 41 active+clean; 615 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 3.8 MiB/s rd, 4.0 MiB/s wr, 1.03k op/s 2021-07-31T02:01:10.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:10 smithi045 conmon[33655]: cluster 2021-07-31T02:01:10.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:10 smithi045 conmon[33655]: 2021-07-31T02:01:10.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:10 smithi045 conmon[33655]: 2021 2021-07-31T02:01:10.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:10 smithi045 conmon[33655]: - 2021-07-31T02:01:10.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:10 smithi045 conmon[33655]: 07 2021-07-31T02:01:10.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:10 smithi045 conmon[33655]: - 2021-07-31T02:01:10.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:10 smithi045 conmon[33655]: 31 2021-07-31T02:01:10.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:10 smithi045 conmon[33655]: T 2021-07-31T02:01:10.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:10 smithi045 conmon[33655]: 02 2021-07-31T02:01:10.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:10 smithi045 conmon[33655]: : 2021-07-31T02:01:10.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:10 smithi045 conmon[33655]: 01 2021-07-31T02:01:10.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:10 smithi045 conmon[33655]: : 2021-07-31T02:01:10.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:10 smithi045 conmon[33655]: 09 2021-07-31T02:01:10.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:10 smithi045 conmon[33655]: . 2021-07-31T02:01:10.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:10 smithi045 conmon[33655]: 990099 2021-07-31T02:01:10.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:10 smithi045 conmon[33655]: +0000 2021-07-31T02:01:10.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:10 smithi045 conmon[33655]: 2021-07-31T02:01:10.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:10 smithi045 conmon[33655]: mgr.x 2021-07-31T02:01:10.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:10 smithi045 conmon[33655]: ( 2021-07-31T02:01:10.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:10 smithi045 conmon[33655]: mgr 2021-07-31T02:01:10.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:10 smithi045 conmon[33655]: . 2021-07-31T02:01:10.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:10 smithi045 conmon[33655]: 24542 2021-07-31T02:01:10.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:10 smithi045 conmon[33655]: ) 2021-07-31T02:01:10.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:10 smithi045 conmon[33655]: 595 2021-07-31T02:01:10.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:10 smithi045 conmon[33655]: : 2021-07-31T02:01:10.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:10 smithi045 conmon[33655]: cluster 2021-07-31T02:01:10.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:10 smithi045 conmon[33655]: 2021-07-31T02:01:10.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:10 smithi045 conmon[33655]: [DBG] 2021-07-31T02:01:10.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:10 smithi045 conmon[33655]: 2021-07-31T02:01:10.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:10 smithi045 conmon[33655]: pgmap v1159: 41 pgs: 41 active+clean; 615 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 3.8 MiB/s rd, 4.0 MiB/s wr, 1.03k op/s 2021-07-31T02:01:10.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:10 smithi045 conmon[33655]: 2021-07-31T02:01:11.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:11 smithi154 conmon[34553]: cluster 2021-07-31T02:01:10.106368+0000 mon.a (mon.0) 1628 : cluster [DBG] osdmap e895: 8 total, 8 up, 8 in 2021-07-31T02:01:11.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:11 smithi045 conmon[33655]: cluster 2021-07-31T02:01: 2021-07-31T02:01:11.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:11 smithi045 conmon[33655]: 10.106368+0000 mon.a (mon.0) 1628 : cluster [DBG] osdmap e895: 8 total, 8 up, 8 in 2021-07-31T02:01:11.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:11 smithi045 conmon[39718]: cluster 2021-07-31T02:01:10.106368+0000 mon.a ( 2021-07-31T02:01:11.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:11 smithi045 conmon[39718]: mon.0) 1628 : cluster [DBG] osdmap e895: 8 total, 8 up, 8 in 2021-07-31T02:01:12.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:12 smithi154 conmon[34553]: cluster 2021-07-31T02:01:11.118921+0000 mon.a (mon.0) 1629 : cluster [DBG] osdmap e896: 8 total, 8 up, 8 in 2021-07-31T02:01:12.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:12 smithi154 conmon[34553]: audit 2021-07-31T02:01:11.121810+0000 mon.c (mon.1) 46 : audit [INF] from='client.? 172.21.15.45:0/3248325521' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:01:12.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:12 smithi154 conmon[34553]: audit 2021-07-31T02:01:11.126463+0000 mon.a (mon.0) 1630 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:01:12.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:12 smithi154 conmon[34553]: cluster 2021-07-31T02:01:11.990584+0000 mgr.x ( 2021-07-31T02:01:12.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:12 smithi154 conmon[34553]: mgr.24542) 596 : cluster [DBG] pgmap v1162: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail 2021-07-31T02:01:12.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:12 smithi045 conmon[33655]: cluster 2021-07-31T02:01:11.118921+0000 mon.a (mon.0) 1629 : 2021-07-31T02:01:12.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:12 smithi045 conmon[33655]: cluster [DBG] osdmap e896: 8 total, 8 up, 8 in 2021-07-31T02:01:12.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:12 smithi045 conmon[33655]: audit 2021-07-31T02:01:11.121810+0000 mon.c (mon.1) 46 : audit [INF] from='client.? 172.21.15.45:0/3248325521' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:01:12.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:12 smithi045 conmon[33655]: audit 2021-07-31T02:01:11.126463+0000 mon.a (mon.0) 1630 : 2021-07-31T02:01:12.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:12 smithi045 conmon[33655]: audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:01:12.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:12 smithi045 conmon[33655]: cluster 2021-07-31T02:01:11.990584+0000 mgr.x (mgr. 2021-07-31T02:01:12.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:12 smithi045 conmon[33655]: 24542) 596 : cluster [DBG] pgmap v1162: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail 2021-07-31T02:01:12.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:12 smithi045 conmon[39718]: cluster 2021-07-31T02:01:11.118921+0000 mon.a (mon.0) 1629 : cluster [DBG] osdmap e896: 8 total, 8 up, 8 in 2021-07-31T02:01:12.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:12 smithi045 conmon[39718]: audit 2021-07-31T02:01:11.121810 2021-07-31T02:01:12.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:12 smithi045 conmon[39718]: +0000 mon.c (mon.1) 46 : audit [INF] from='client.? 172.21.15.45:0/3248325521' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:01:12.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:12 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T02:01:12.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:12 smithi045 conmon[39718]: 02:01:11.126463+0000 mon.a (mon.0) 2021-07-31T02:01:12.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:12 smithi045 conmon[39718]: 1630 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:01:12.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:12 smithi045 conmon[39718]: cluster 2021-07-31T02:01 2021-07-31T02:01:12.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:12 smithi045 conmon[39718]: :11.990584+0000 mgr.x (mgr.24542) 596 : cluster [DBG] pgmap v1162: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail 2021-07-31T02:01:12.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:12 smithi045 conmon[39718]: 2021-07-31T02:01:13.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:13 smithi154 conmon[34553]: audit 2021-07-31T02:01:12.125618+0000 mon.a ( 2021-07-31T02:01:13.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:13 smithi154 conmon[34553]: mon.0) 1631 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-22","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:01:13.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:13 smithi154 conmon[34553]: cluster 2021-07-31T02:01:12.125889+0000 mon.a (mon.0) 1632 : cluster [DBG] osdmap e897: 8 total, 8 up, 8 in 2021-07-31T02:01:13.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:13 smithi045 conmon[33655]: audit 2021-07-31T02:01:12.125618+0000 mon.a 2021-07-31T02:01:13.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:13 smithi045 conmon[33655]: (mon.0) 1631 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-22","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:01:13.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:13 smithi045 conmon[33655]: cluster 2021-07-31T02:01:12.125889+0000 mon.a (mon.0) 1632 : cluster [DBG] osdmap e897: 8 total, 8 up, 8 in 2021-07-31T02:01:13.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:13 smithi045 conmon[39718]: audit 2021-07-31T02:01:12.125618 2021-07-31T02:01:13.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:13 smithi045 conmon[39718]: +0000 mon.a (mon.0) 1631 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-72714-22","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:01:13.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:13 smithi045 conmon[39718]: cluster 2021-07-31T02: 2021-07-31T02:01:13.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:13 smithi045 conmon[39718]: 01:12.125889+0000 mon.a ( 2021-07-31T02:01:13.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:13 smithi045 conmon[39718]: mon.0) 1632 : cluster 2021-07-31T02:01:13.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:13 smithi045 conmon[39718]: [DBG] osdmap e897: 8 total, 8 up, 8 in 2021-07-31T02:01:14.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:01:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:01:13.532Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:01:14.212 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:01:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:01:13.532Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:01:14.213 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:01:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:01:13.532Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:01:14.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:14 smithi154 conmon[34553]: cluster 2021-07-31T02:01:13.125849+0000 mon.a (mon.0) 2021-07-31T02:01:14.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:14 smithi154 conmon[34553]: 1633 : cluster [DBG] osdmap e898: 8 total, 8 up, 8 in 2021-07-31T02:01:14.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:14 smithi154 conmon[34553]: cluster 2021-07-31T02:01:13.990990+0000 mgr.x (mgr.24542) 597 : cluster [DBG] pgmap v1165: 41 pgs: 41 active+clean; 600 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 170 KiB/s rd, 20 op/s 2021-07-31T02:01:14.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:14 smithi045 conmon[39718]: cluster 2021-07-31T02:01:13. 2021-07-31T02:01:14.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:14 smithi045 conmon[39718]: 125849+0000 mon.a (mon.0) 1633 : cluster [DBG] osdmap e898: 8 total, 8 up, 8 in 2021-07-31T02:01:14.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:14 smithi045 conmon[39718]: cluster 2021-07-31T02:01:14.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:14 smithi045 conmon[39718]: 2021-07-31T02:01:13.990990 2021-07-31T02:01:14.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:14 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 597 : cluster 2021-07-31T02:01:14.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:14 smithi045 conmon[39718]: [DBG] pgmap v1165: 41 pgs: 41 active+clean; 600 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 170 KiB/s rd, 20 op/s 2021-07-31T02:01:14.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:14 smithi045 conmon[33655]: cluster 2021-07-31T02:01:13 2021-07-31T02:01:14.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:14 smithi045 conmon[33655]: .125849+0000 mon.a (mon.0) 1633 : cluster [DBG] osdmap e898: 8 total, 8 up, 8 in 2021-07-31T02:01:14.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:14 smithi045 conmon[33655]: cluster 2021-07-31T02:01:13.990990+0000 mgr.x 2021-07-31T02:01:14.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:14 smithi045 conmon[33655]: (mgr.24542) 597 : cluster [DBG] pgmap v1165: 41 pgs: 41 active+clean; 600 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 170 KiB/s rd, 20 op/s 2021-07-31T02:01:15.146 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestCls2PCQueue.AbortSpillover (5038 ms) 2021-07-31T02:01:15.146 INFO:tasks.workunit.client.0.smithi045.stdout:[----------] 22 tests from TestCls2PCQueue (113875 ms total) 2021-07-31T02:01:15.146 INFO:tasks.workunit.client.0.smithi045.stdout: 2021-07-31T02:01:15.147 INFO:tasks.workunit.client.0.smithi045.stdout:[----------] Global test environment tear-down 2021-07-31T02:01:15.147 INFO:tasks.workunit.client.0.smithi045.stdout:[==========] 22 tests from 1 test suite ran. (113875 ms total) 2021-07-31T02:01:15.147 INFO:tasks.workunit.client.0.smithi045.stdout:[ PASSED ] 22 tests. 2021-07-31T02:01:15.149 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-07-31T02:01:15.149 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2021-07-31T02:01:15.187 INFO:tasks.workunit:Running workunit cls/test_cls_cmpomap.sh... 2021-07-31T02:01:15.188 DEBUG:teuthology.orchestra.run.smithi045:workunit test cls/test_cls_cmpomap.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=pacific TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_cmpomap.sh 2021-07-31T02:01:15.256 INFO:tasks.workunit.client.0.smithi045.stdout:Running main() from gmock_main.cc 2021-07-31T02:01:15.256 INFO:tasks.workunit.client.0.smithi045.stdout:[==========] Running 29 tests from 1 test suite. 2021-07-31T02:01:15.257 INFO:tasks.workunit.client.0.smithi045.stdout:[----------] Global test environment set-up. 2021-07-31T02:01:16.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:16 smithi154 conmon[34553]: cluster 2021-07-31T02:01 2021-07-31T02:01:16.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:16 smithi154 conmon[34553]: :15.144647+0000 mon.a (mon.0) 1634 : cluster [DBG] osdmap e899: 8 total, 8 up, 8 in 2021-07-31T02:01:16.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:16 smithi154 conmon[34553]: cluster 2021-07-31T02:01:15.991417+0000 2021-07-31T02:01:16.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:16 smithi154 conmon[34553]: mgr.x (mgr.24542) 598 : cluster [DBG] pgmap v1167: 9 pgs: 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:01:16.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:16 smithi045 conmon[33655]: cluster 2021-07-31T02:01:15.144647+0000 mon.a (mon.0) 1634 2021-07-31T02:01:16.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:16 smithi045 conmon[33655]: : cluster [DBG] osdmap e899: 8 total, 8 up, 8 in 2021-07-31T02:01:16.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:16 smithi045 conmon[33655]: cluster 2021-07-31T02:01:15.991417+0000 mgr.x (mgr.24542) 598 : cluster [DBG] pgmap v1167: 9 pgs: 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:01:16.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:16 smithi045 conmon[39718]: cluster 2021-07-31T02:01:15.144647 2021-07-31T02:01:16.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:16 smithi045 conmon[39718]: +0000 mon.a (mon.0) 1634 : cluster [DBG] osdmap e899: 8 total, 8 up, 8 in 2021-07-31T02:01:16.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:16 smithi045 conmon[39718]: cluster 2021- 2021-07-31T02:01:16.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:16 smithi045 conmon[39718]: 07-31T02:01:15.991417+0000 mgr.x (mgr.24542) 598 2021-07-31T02:01:16.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:16 smithi045 conmon[39718]: : cluster [DBG] pgmap v1167: 9 pgs: 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:01:17.155 INFO:tasks.workunit.client.0.smithi045.stdout:[----------] 29 tests from CmpOmap 2021-07-31T02:01:17.155 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] CmpOmap.cmp_vals_noexist_str 2021-07-31T02:01:17.176 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] CmpOmap.cmp_vals_noexist_str (20 ms) 2021-07-31T02:01:17.176 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] CmpOmap.cmp_vals_noexist_str_default 2021-07-31T02:01:17.180 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] CmpOmap.cmp_vals_noexist_str_default (5 ms) 2021-07-31T02:01:17.180 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] CmpOmap.cmp_vals_noexist_u64 2021-07-31T02:01:17.185 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] CmpOmap.cmp_vals_noexist_u64 (5 ms) 2021-07-31T02:01:17.186 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] CmpOmap.cmp_vals_noexist_u64_default 2021-07-31T02:01:17.190 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] CmpOmap.cmp_vals_noexist_u64_default (4 ms) 2021-07-31T02:01:17.191 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] CmpOmap.cmp_vals_str 2021-07-31T02:01:17.196 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] CmpOmap.cmp_vals_str (8 ms) 2021-07-31T02:01:17.197 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] CmpOmap.cmp_vals_u64 2021-07-31T02:01:17.207 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] CmpOmap.cmp_vals_u64 (10 ms) 2021-07-31T02:01:17.207 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] CmpOmap.cmp_vals_u64_invalid_input 2021-07-31T02:01:17.210 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] CmpOmap.cmp_vals_u64_invalid_input (4 ms) 2021-07-31T02:01:17.211 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] CmpOmap.cmp_vals_u64_invalid_default 2021-07-31T02:01:17.214 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] CmpOmap.cmp_vals_u64_invalid_default (3 ms) 2021-07-31T02:01:17.215 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] CmpOmap.cmp_vals_u64_invalid_value 2021-07-31T02:01:17.218 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] CmpOmap.cmp_vals_u64_invalid_value (5 ms) 2021-07-31T02:01:17.219 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] CmpOmap.cmp_vals_at_max_keys 2021-07-31T02:01:17.219 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] CmpOmap.cmp_vals_at_max_keys (0 ms) 2021-07-31T02:01:17.220 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] CmpOmap.cmp_vals_over_max_keys 2021-07-31T02:01:17.220 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] CmpOmap.cmp_vals_over_max_keys (0 ms) 2021-07-31T02:01:17.220 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] CmpOmap.cmp_set_vals_noexist_str 2021-07-31T02:01:17.227 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] CmpOmap.cmp_set_vals_noexist_str (7 ms) 2021-07-31T02:01:17.227 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] CmpOmap.cmp_set_vals_noexist_str_default 2021-07-31T02:01:17.234 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] CmpOmap.cmp_set_vals_noexist_str_default (7 ms) 2021-07-31T02:01:17.234 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] CmpOmap.cmp_set_vals_noexist_u64 2021-07-31T02:01:17.240 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] CmpOmap.cmp_set_vals_noexist_u64 (6 ms) 2021-07-31T02:01:17.240 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] CmpOmap.cmp_set_vals_noexist_u64_default 2021-07-31T02:01:17.248 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] CmpOmap.cmp_set_vals_noexist_u64_default (8 ms) 2021-07-31T02:01:17.248 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] CmpOmap.cmp_set_vals_str 2021-07-31T02:01:17.256 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] CmpOmap.cmp_set_vals_str (8 ms) 2021-07-31T02:01:17.256 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] CmpOmap.cmp_set_vals_u64 2021-07-31T02:01:17.264 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] CmpOmap.cmp_set_vals_u64 (8 ms) 2021-07-31T02:01:17.264 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] CmpOmap.cmp_set_vals_u64_einval 2021-07-31T02:01:17.266 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] CmpOmap.cmp_set_vals_u64_einval (2 ms) 2021-07-31T02:01:17.266 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] CmpOmap.cmp_set_vals_u64_eio 2021-07-31T02:01:17.268 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] CmpOmap.cmp_set_vals_u64_eio (3 ms) 2021-07-31T02:01:17.269 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] CmpOmap.cmp_set_vals_at_max_keys 2021-07-31T02:01:17.269 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] CmpOmap.cmp_set_vals_at_max_keys (1 ms) 2021-07-31T02:01:17.269 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] CmpOmap.cmp_set_vals_over_max_keys 2021-07-31T02:01:17.270 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] CmpOmap.cmp_set_vals_over_max_keys (0 ms) 2021-07-31T02:01:17.270 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] CmpOmap.cmp_rm_keys_noexist_str 2021-07-31T02:01:17.276 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] CmpOmap.cmp_rm_keys_noexist_str (7 ms) 2021-07-31T02:01:17.277 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] CmpOmap.cmp_rm_keys_noexist_u64 2021-07-31T02:01:17.282 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] CmpOmap.cmp_rm_keys_noexist_u64 (6 ms) 2021-07-31T02:01:17.283 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] CmpOmap.cmp_rm_keys_str 2021-07-31T02:01:17.290 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] CmpOmap.cmp_rm_keys_str (7 ms) 2021-07-31T02:01:17.290 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] CmpOmap.cmp_rm_keys_u64 2021-07-31T02:01:17.297 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] CmpOmap.cmp_rm_keys_u64 (7 ms) 2021-07-31T02:01:17.297 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] CmpOmap.cmp_rm_keys_u64_einval 2021-07-31T02:01:17.299 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] CmpOmap.cmp_rm_keys_u64_einval (2 ms) 2021-07-31T02:01:17.299 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] CmpOmap.cmp_rm_keys_u64_eio 2021-07-31T02:01:17.301 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] CmpOmap.cmp_rm_keys_u64_eio (2 ms) 2021-07-31T02:01:17.301 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] CmpOmap.cmp_rm_keys_at_max_keys 2021-07-31T02:01:17.302 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] CmpOmap.cmp_rm_keys_at_max_keys (1 ms) 2021-07-31T02:01:17.302 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] CmpOmap.cmp_rm_keys_over_max_keys 2021-07-31T02:01:17.303 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] CmpOmap.cmp_rm_keys_over_max_keys (1 ms) 2021-07-31T02:01:17.303 INFO:tasks.workunit.client.0.smithi045.stdout:[----------] 29 tests from CmpOmap (148 ms total) 2021-07-31T02:01:17.303 INFO:tasks.workunit.client.0.smithi045.stdout: 2021-07-31T02:01:17.303 INFO:tasks.workunit.client.0.smithi045.stdout:[----------] Global test environment tear-down 2021-07-31T02:01:17.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:17 smithi154 conmon[34553]: cluster 2021-07-31T02:01:17.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:17 smithi154 conmon[34553]: 2021-07-31T02:01:16.151797 2021-07-31T02:01:17.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:17 smithi154 conmon[34553]: +0000 mon.a (mon.0) 1635 : cluster [DBG] osdmap e900: 8 total, 8 up, 8 in 2021-07-31T02:01:17.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:17 smithi154 conmon[34553]: audit 2021-07-31T02:01:16.152758+0000 mon.c (mon.1 2021-07-31T02:01:17.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:17 smithi154 conmon[34553]: ) 47 : audit [INF] from='client.? 172.21.15.45:0/3518236973' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73479-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:01:17.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:17 smithi154 conmon[34553]: audit 2021-07-31T02:01:17.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:17 smithi154 conmon[34553]: 2021-07-31T02:01:17.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:17 smithi154 conmon[34553]: 2021 2021-07-31T02:01:17.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:17 smithi154 conmon[34553]: - 2021-07-31T02:01:17.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:17 smithi154 conmon[34553]: 07 2021-07-31T02:01:17.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:17 smithi154 conmon[34553]: - 2021-07-31T02:01:17.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:17 smithi154 conmon[34553]: 31 2021-07-31T02:01:17.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:17 smithi154 conmon[34553]: T 2021-07-31T02:01:17.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:17 smithi154 conmon[34553]: 02 2021-07-31T02:01:17.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:17 smithi154 conmon[34553]: : 2021-07-31T02:01:17.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:17 smithi154 conmon[34553]: 01 2021-07-31T02:01:17.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:17 smithi154 conmon[34553]: : 2021-07-31T02:01:17.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:17 smithi154 conmon[34553]: 16 2021-07-31T02:01:17.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:17 smithi154 conmon[34553]: . 2021-07-31T02:01:17.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:17 smithi154 conmon[34553]: 153620 2021-07-31T02:01:17.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:17 smithi154 conmon[34553]: +0000 2021-07-31T02:01:17.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:17 smithi154 conmon[34553]: 2021-07-31T02:01:17.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:17 smithi154 conmon[34553]: mon.a 2021-07-31T02:01:17.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:17 smithi154 conmon[34553]: ( 2021-07-31T02:01:17.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:17 smithi154 conmon[34553]: mon 2021-07-31T02:01:17.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:17 smithi154 conmon[34553]: . 2021-07-31T02:01:17.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:17 smithi154 conmon[34553]: 0 2021-07-31T02:01:17.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:17 smithi154 conmon[34553]: ) 2021-07-31T02:01:17.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:17 smithi154 conmon[34553]: 1636 2021-07-31T02:01:17.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:17 smithi154 conmon[34553]: : 2021-07-31T02:01:17.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:17 smithi154 conmon[34553]: audit 2021-07-31T02:01:17.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:17 smithi154 conmon[34553]: 2021-07-31T02:01:17.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:17 smithi154 conmon[34553]: [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73479-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:01:17.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:17 smithi045 conmon[33655]: cluster 2021-07-31T02:01:16.151797+0000 mon.a (mon.0) 1635 : cluster [DBG] osdmap e900: 8 total, 8 up, 8 in 2021-07-31T02:01:17.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:17 smithi045 conmon[33655]: audit 2021-07-31T02:01:16.152758+0000 mon.c (mon.1) 47 : audit [INF] from='client.? 172.21.15.45:0/3518236973' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73479-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:01:17.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:17 smithi045 conmon[33655]: audit 2021-07-31T02:01:16.153620+0000 mon.a (mon.0) 1636 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73479-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:01:17.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:17 smithi045 conmon[39718]: cluster 2021-07-31T02:01:17.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:17 smithi045 conmon[39718]: 2021-07-31T02:01:17.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:17 smithi045 conmon[39718]: 2021 2021-07-31T02:01:17.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:17 smithi045 conmon[39718]: - 2021-07-31T02:01:17.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:17 smithi045 conmon[39718]: 07 2021-07-31T02:01:17.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:17 smithi045 conmon[39718]: - 2021-07-31T02:01:17.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:17 smithi045 conmon[39718]: 31 2021-07-31T02:01:17.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:17 smithi045 conmon[39718]: T 2021-07-31T02:01:17.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:17 smithi045 conmon[39718]: 02 2021-07-31T02:01:17.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:17 smithi045 conmon[39718]: : 2021-07-31T02:01:17.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:17 smithi045 conmon[39718]: 01 2021-07-31T02:01:17.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:17 smithi045 conmon[39718]: : 2021-07-31T02:01:17.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:17 smithi045 conmon[39718]: 16 2021-07-31T02:01:17.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:17 smithi045 conmon[39718]: . 2021-07-31T02:01:17.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:17 smithi045 conmon[39718]: 151797 2021-07-31T02:01:17.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:17 smithi045 conmon[39718]: +0000 2021-07-31T02:01:17.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:17 smithi045 conmon[39718]: mon.a (mon.0) 1635 : cluster [DBG] osdmap e900: 8 total, 8 up, 8 in 2021-07-31T02:01:17.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:17 smithi045 conmon[39718]: audit 2021-07-31T02:01:16.152758+0000 mon.c (mon.1 2021-07-31T02:01:17.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:17 smithi045 conmon[39718]: ) 47 : audit [INF] from='client.? 172.21.15.45:0/3518236973' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73479-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:01:17.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:17 smithi045 conmon[39718]: audit 2021-07-31T02:01: 2021-07-31T02:01:17.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:17 smithi045 conmon[39718]: 16.153620+0000 mon.a (mon.0) 1636 : 2021-07-31T02:01:17.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:17 smithi045 conmon[39718]: audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73479-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:01:18.172 INFO:tasks.workunit.client.0.smithi045.stdout:[==========] 29 tests from 1 test suite ran. (2916 ms total) 2021-07-31T02:01:18.172 INFO:tasks.workunit.client.0.smithi045.stdout:[ PASSED ] 29 tests. 2021-07-31T02:01:18.174 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-07-31T02:01:18.175 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2021-07-31T02:01:18.245 INFO:tasks.workunit:Running workunit cls/test_cls_hello.sh... 2021-07-31T02:01:18.246 DEBUG:teuthology.orchestra.run.smithi045:workunit test cls/test_cls_hello.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=pacific TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_hello.sh 2021-07-31T02:01:18.317 INFO:tasks.workunit.client.0.smithi045.stdout:Running main() from gmock_main.cc 2021-07-31T02:01:18.317 INFO:tasks.workunit.client.0.smithi045.stdout:[==========] Running 6 tests from 1 test suite. 2021-07-31T02:01:18.318 INFO:tasks.workunit.client.0.smithi045.stdout:[----------] Global test environment set-up. 2021-07-31T02:01:18.318 INFO:tasks.workunit.client.0.smithi045.stdout:[----------] 6 tests from ClsHello 2021-07-31T02:01:18.318 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] ClsHello.SayHello 2021-07-31T02:01:18.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:18 smithi154 conmon[34553]: audit 2021-07-31T02:01:17.153357+0000 mon.a (mon.0) 1637 : audit [INF] 2021-07-31T02:01:18.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:18 smithi154 conmon[34553]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73479-1","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:01:18.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:18 smithi154 conmon[34553]: cluster 2021-07-31T02:01:17.153673+0000 mon.a (mon.0) 1638 : cluster [DBG] osdmap e901: 8 total, 8 up, 8 in 2021-07-31T02:01:18.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:18 smithi154 conmon[34553]: cluster 2021-07-31T02:01:18.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:18 smithi154 conmon[34553]: 2021-07-31T02:01:17.992009+0000 mgr.x (mgr.24542) 599 : cluster [DBG] pgmap v1170: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:01:18.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:18 smithi045 conmon[33655]: audit 2021-07-31T02:01:17.153357+0000 mon.a (mon.0 2021-07-31T02:01:18.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:18 smithi045 conmon[33655]: ) 1637 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73479-1","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:01:18.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:18 smithi045 conmon[33655]: cluster 2021-07-31T02:01:17.153673+0000 mon.a (mon.0) 1638 : cluster [DBG] osdmap e901: 8 total, 8 up, 8 in 2021-07-31T02:01:18.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:18 smithi045 conmon[33655]: cluster 2021-07-31T02:01:17.992009+0000 mgr.x (mgr 2021-07-31T02:01:18.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:18 smithi045 conmon[33655]: .24542) 599 : cluster [DBG] pgmap v1170: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:01:18.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:18 smithi045 conmon[39718]: audit 2021-07-31T02:01:17.153357+0000 mon.a 2021-07-31T02:01:18.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:18 smithi045 conmon[39718]: (mon.0) 1637 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73479-1","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:01:18.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:18 smithi045 conmon[39718]: cluster 2021-07-31T02:01:17.153673+0000 mon.a (mon 2021-07-31T02:01:18.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:18 smithi045 conmon[39718]: .0) 1638 : cluster [DBG] osdmap e901: 8 total, 8 up, 8 in 2021-07-31T02:01:18.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:18 smithi045 conmon[39718]: cluster 2021-07-31T 2021-07-31T02:01:18.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:18 smithi045 conmon[39718]: 02:01:17.992009+0000 mgr.x (mgr. 2021-07-31T02:01:18.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:18 smithi045 conmon[39718]: 24542) 599 : cluster [DBG] pgmap v1170: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:01:18.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:18 smithi045 conmon[39718]: 2021-07-31T02:01:19.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:19 smithi154 conmon[34553]: cluster 2021-07-31T02:01:19.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:19 smithi154 conmon[34553]: 2021-07-31T02:01:19.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:19 smithi154 conmon[34553]: 2021 2021-07-31T02:01:19.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:19 smithi154 conmon[34553]: - 2021-07-31T02:01:19.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:19 smithi154 conmon[34553]: 07 2021-07-31T02:01:19.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:19 smithi154 conmon[34553]: - 2021-07-31T02:01:19.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:19 smithi154 conmon[34553]: 31 2021-07-31T02:01:19.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:19 smithi154 conmon[34553]: T 2021-07-31T02:01:19.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:19 smithi154 conmon[34553]: 02 2021-07-31T02:01:19.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:19 smithi154 conmon[34553]: : 2021-07-31T02:01:19.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:19 smithi154 conmon[34553]: 01 2021-07-31T02:01:19.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:19 smithi154 conmon[34553]: : 2021-07-31T02:01:19.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:19 smithi154 conmon[34553]: 18 2021-07-31T02:01:19.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:19 smithi154 conmon[34553]: . 2021-07-31T02:01:19.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:19 smithi154 conmon[34553]: 168896 2021-07-31T02:01:19.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:19 smithi154 conmon[34553]: +0000 2021-07-31T02:01:19.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:19 smithi154 conmon[34553]: 2021-07-31T02:01:19.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:19 smithi154 conmon[34553]: mon.a 2021-07-31T02:01:19.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:19 smithi154 conmon[34553]: ( 2021-07-31T02:01:19.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:19 smithi154 conmon[34553]: mon 2021-07-31T02:01:19.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:19 smithi154 conmon[34553]: . 2021-07-31T02:01:19.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:19 smithi154 conmon[34553]: 0 2021-07-31T02:01:19.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:19 smithi154 conmon[34553]: ) 2021-07-31T02:01:19.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:19 smithi154 conmon[34553]: 1639 2021-07-31T02:01:19.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:19 smithi154 conmon[34553]: : 2021-07-31T02:01:19.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:19 smithi154 conmon[34553]: cluster 2021-07-31T02:01:19.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:19 smithi154 conmon[34553]: 2021-07-31T02:01:19.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:19 smithi154 conmon[34553]: [DBG] 2021-07-31T02:01:19.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:19 smithi154 conmon[34553]: 2021-07-31T02:01:19.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:19 smithi154 conmon[34553]: osdmap e902: 8 total, 8 up, 8 in 2021-07-31T02:01:19.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:19 smithi154 conmon[34553]: 2021-07-31T02:01:19.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:19 smithi045 conmon[33655]: cluster 2021-07-31T02:01:18.168896 2021-07-31T02:01:19.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:19 smithi045 conmon[33655]: +0000 mon.a (mon.0) 1639 : cluster [DBG] osdmap e902: 8 total, 8 up, 8 in 2021-07-31T02:01:19.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:19 smithi045 conmon[39718]: cluster 2021-07-31T02:01:19.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:19 smithi045 conmon[39718]: 2021-07-31T02:01: 2021-07-31T02:01:19.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:19 smithi045 conmon[39718]: 18.168896+0000 mon.a (mon.0) 1639 : cluster [DBG] osdmap e902: 8 total, 8 up, 8 in 2021-07-31T02:01:20.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:20 smithi154 conmon[34553]: cluster 2021-07-31T02:01:19.182387+0000 mon.a (mon.0) 1640 : 2021-07-31T02:01:20.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:20 smithi154 conmon[34553]: cluster [DBG] osdmap e903: 8 total, 8 up, 8 in 2021-07-31T02:01:20.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:20 smithi154 conmon[34553]: audit 2021-07-31T02:01:19.183347+0000 mon.c (mon.1) 48 : audit [INF] from='client.? 172.21.15.45:0/2259770638' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73532-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:01:20.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:20 smithi154 conmon[34553]: audit 2021-07-31T02:01:19.183880+0000 mon.a (mon 2021-07-31T02:01:20.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:20 smithi154 conmon[34553]: .0) 1641 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73532-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:01:20.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:20 smithi154 conmon[34553]: cluster 2021-07-31T02:01:19.992483 2021-07-31T02:01:20.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:20 smithi154 conmon[34553]: +0000 mgr.x (mgr.24542) 600 : cluster [DBG] pgmap v1173: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:01:20.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:20 smithi045 conmon[33655]: cluster 2021-07-31T02:01:19.182387+0000 mon.a (mon.0) 2021-07-31T02:01:20.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:20 smithi045 conmon[33655]: 1640 : cluster [DBG] osdmap e903: 8 total, 8 up, 8 in 2021-07-31T02:01:20.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:20 smithi045 conmon[33655]: audit 2021-07-31T02:01:19.183347+0000 mon.c (mon.1) 48 : audit [INF] from='client.? 172.21.15.45:0/2259770638' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73532-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:01:20.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:20 smithi045 conmon[33655]: audit 2021-07-31T 2021-07-31T02:01:20.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:20 smithi045 conmon[33655]: 02:01:19.183880+0000 mon.a (mon.0) 1641 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73532-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:01:20.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:20 smithi045 conmon[33655]: cluster 2021-07-31T02:01:20.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:20 smithi045 conmon[33655]: 2021-07-31 2021-07-31T02:01:20.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:20 smithi045 conmon[33655]: T02:01:19.992483 2021-07-31T02:01:20.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:20 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 2021-07-31T02:01:20.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:20 smithi045 conmon[33655]: 600 : cluster [DBG] pgmap v1173: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:01:20.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:20 smithi045 conmon[33655]: 2021-07-31T02:01:20.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:20 smithi045 conmon[39718]: cluster 2021-07-31T02:01:19 2021-07-31T02:01:20.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:20 smithi045 conmon[39718]: .182387+0000 mon.a (mon.0) 1640 : cluster [DBG] osdmap e903: 8 total, 8 up, 8 in 2021-07-31T02:01:20.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:20 smithi045 conmon[39718]: audit 2021-07-31T02:01:20.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:20 smithi045 conmon[39718]: 2021-07-31T02: 2021-07-31T02:01:20.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:20 smithi045 conmon[39718]: 01:19.183347+0000 mon.c 2021-07-31T02:01:20.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:20 smithi045 conmon[39718]: (mon.1) 48 : audit 2021-07-31T02:01:20.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:20 smithi045 conmon[39718]: [INF] from='client.? 172.21.15.45:0/2259770638' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73532-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:01:20.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:20 smithi045 conmon[39718]: audit 2021-07-31T02:01:20.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:20 smithi045 conmon[39718]: 2021-07-31T02:01 2021-07-31T02:01:20.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:20 smithi045 conmon[39718]: :19.183880+0000 mon.a ( 2021-07-31T02:01:20.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:20 smithi045 conmon[39718]: mon.0) 1641 : audit 2021-07-31T02:01:20.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:20 smithi045 conmon[39718]: [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73532-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:01:20.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:20 smithi045 conmon[39718]: cluster 2021-07-31T02:01:19.992483 2021-07-31T02:01:20.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:20 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 600 : cluster [DBG] pgmap v1173: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:01:20.667 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:01:20.947 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:01:20.948 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 16m ago 17m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:01:20.948 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 13m ago 16m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:01:20.948 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 13m ago 20m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:01:20.948 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 16m ago 22m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:01:20.948 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 16m ago 22m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:01:20.949 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 13m ago 20m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:01:20.949 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 16m ago 20m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:01:20.949 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 16m ago 17m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:01:20.949 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 13m ago 17m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:01:20.950 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 16m ago 19m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:01:20.950 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 16m ago 19m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:01:20.950 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 16m ago 19m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:01:20.950 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 16m ago 18m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:01:20.951 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 13m ago 18m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:01:20.951 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 13m ago 18m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:01:20.951 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 13m ago 17m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:01:20.951 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 13m ago 17m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:01:20.952 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 13m ago 17m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:01:21.191 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] ClsHello.SayHello (2874 ms) 2021-07-31T02:01:21.191 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] ClsHello.RecordHello 2021-07-31T02:01:21.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:21 smithi154 conmon[34553]: audit 2021-07-31T02:01:20.188698+0000 mon.a (mon.0) 1642 : audit 2021-07-31T02:01:21.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:21 smithi154 conmon[34553]: [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73532-1","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:01:21.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:21 smithi154 conmon[34553]: cluster 2021-07-31T02:01:20.189097+0000 mon.a (mon.0) 1643 : cluster [DBG] osdmap e904: 8 total, 8 up, 8 in 2021-07-31T02:01:21.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:21 smithi154 conmon[34553]: audit 2021-07-31T02:01:20.655166+0000 mgr.x (mgr.24542) 601 : audit [DBG] from='client.15360 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:01:21.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:21 smithi154 conmon[34553]: audit 2021-07-31T02:01:20.936847+0000 mgr.x (mgr.24542) 602 : audit [DBG] from='client.25273 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:01:21.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:21 smithi045 conmon[33655]: audit 2021-07-31T02:01:20.188698+0000 mon.a (mon.0) 1642 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73532-1","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:01:21.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:21 smithi045 conmon[33655]: cluster 2021-07-31T02:01:20.189097+0000 mon.a (mon.0) 1643 : cluster [DBG] osdmap e904: 8 total, 8 up, 8 in 2021-07-31T02:01:21.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:21 smithi045 conmon[33655]: audit 2021-07-31T02:01 2021-07-31T02:01:21.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:21 smithi045 conmon[33655]: :20.655166+0000 mgr.x (mgr.24542) 601 : audit [DBG] from='client.15360 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:01:21.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:21 smithi045 conmon[33655]: audit 2021-07-31T02:01:20.936847+0000 mgr.x (mgr.24542) 602 : audit [DBG] from='client.25273 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:01:21.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:21 smithi045 conmon[39718]: audit 2021-07-31T02:01:21.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:21 smithi045 conmon[39718]: 2021-07-31T02 2021-07-31T02:01:21.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:21 smithi045 conmon[39718]: :01:20.188698+0000 mon.a (mon.0) 1642 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73532-1","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:01:21.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:21 smithi045 conmon[39718]: cluster 2021 2021-07-31T02:01:21.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:21 smithi045 conmon[39718]: -07-31T02:01:20.189097+0000 2021-07-31T02:01:21.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:21 smithi045 conmon[39718]: mon.a (mon.0) 1643 : cluster [DBG] osdmap e904: 8 total, 8 up, 8 in 2021-07-31T02:01:21.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:21 smithi045 conmon[39718]: 2021-07-31T02:01:21.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:21 smithi045 conmon[39718]: audit 2021-07-31T02:01:20.655166+0000 2021-07-31T02:01:21.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:21 smithi045 conmon[39718]: mgr.x (mgr.24542) 601 : audit [DBG] from='client.15360 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:01:21.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:21 smithi045 conmon[39718]: audit 2021-07-31T02:01:20.936847+0000 mgr.x (mgr 2021-07-31T02:01:21.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:21 smithi045 conmon[39718]: .24542) 602 : audit [DBG] from='client.25273 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:01:21.654 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:01:21.654 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:01:21.654 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:01:21.655 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:01:21.655 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:01:21.655 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:01:21.655 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:01:21.655 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:01:21.656 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:01:21.656 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:01:21.656 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:01:21.656 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:01:21.657 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:01:21.657 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:01:21.657 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:01:21.657 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:01:21.657 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:01:22.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:22 smithi154 conmon[34553]: cluster 2021-07-31T02:01:21.188677+0000 mon.a (mon.0) 1644 : cluster [DBG] 2021-07-31T02:01:22.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:22 smithi154 conmon[34553]: osdmap e905: 8 total, 8 up, 8 in 2021-07-31T02:01:22.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:22 smithi154 conmon[34553]: audit 2021-07-31T02:01:21.653517+0000 mon.a (mon.0) 1645 : audit [DBG] from='client.? 172.21.15.45:0/2353257647' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:01:22.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:22 smithi154 conmon[34553]: cluster 2021-07-31T02:01:21.993003+0000 mgr.x 2021-07-31T02:01:22.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:22 smithi154 conmon[34553]: (mgr.24542) 603 : cluster [DBG] pgmap v1176: 9 pgs: 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:01:22.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:22 smithi045 conmon[39718]: cluster 2021-07-31T02:01:21.188677+0000 mon.a (mon. 2021-07-31T02:01:22.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:22 smithi045 conmon[39718]: 0) 1644 : cluster [DBG] osdmap e905: 8 total, 8 up, 8 in 2021-07-31T02:01:22.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:22 smithi045 conmon[39718]: audit 2021-07-31T02: 2021-07-31T02:01:22.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:22 smithi045 conmon[39718]: 01:21.653517+0000 mon.a (mon.0) 1645 : audit [DBG] from='client.? 172.21.15.45:0/2353257647' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:01:22.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:22 smithi045 conmon[39718]: cluster 2021-07-31T02:01:22.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:22 smithi045 conmon[39718]: 2021-07-31T02:01:21.993003+0000 mgr.x 2021-07-31T02:01:22.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:22 smithi045 conmon[39718]: (mgr.24542) 603 : cluster [DBG] pgmap v1176: 9 pgs: 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:01:22.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:22 smithi045 conmon[33655]: cluster 2021-07-31T02:01:21.188677+0000 mon.a (mon.0) 1644 2021-07-31T02:01:22.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:22 smithi045 conmon[33655]: : cluster [DBG] osdmap e905: 8 total, 8 up, 8 in 2021-07-31T02:01:22.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:22 smithi045 conmon[33655]: audit 2021-07-31T02:01 2021-07-31T02:01:22.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:22 smithi045 conmon[33655]: :21.653517+0000 mon.a (mon.0) 1645 : audit 2021-07-31T02:01:22.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:22 smithi045 conmon[33655]: [DBG] from='client.? 172.21.15.45:0/2353257647' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:01:22.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:22 smithi045 conmon[33655]: cluster 2021-07 2021-07-31T02:01:22.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:22 smithi045 conmon[33655]: -31T02:01:21.993003+0000 mgr.x (mgr.24542) 603 : cluster 2021-07-31T02:01:22.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:22 smithi045 conmon[33655]: [DBG] pgmap v1176: 9 pgs: 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:01:23.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:23 smithi154 conmon[34553]: cluster 2021-07-31T02:01:23.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:23 smithi154 conmon[34553]: 2021-07-31T02:01:23.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:23 smithi154 conmon[34553]: 2021-07-31T02:01:22 2021-07-31T02:01:23.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:23 smithi154 conmon[34553]: .200646+0000 mon.a (mon.0) 1646 : cluster [DBG] osdmap e906: 8 total, 8 up, 8 in 2021-07-31T02:01:23.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:23 smithi154 conmon[34553]: audit 2021-07-31T02:01:23.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:23 smithi154 conmon[34553]: 2021-07-31T 2021-07-31T02:01:23.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:23 smithi154 conmon[34553]: 02:01:22.201817+0000 mon.c (mon.1 2021-07-31T02:01:23.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:23 smithi154 conmon[34553]: ) 49 : audit [INF] from='client.? 172.21.15.45:0/3280736645' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73532-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:01:23.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:23 smithi154 conmon[34553]: audit 2021-07-31T 2021-07-31T02:01:23.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:23 smithi154 conmon[34553]: 02:01:22.202811+0000 mon.a (mon.0) 1647 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73532-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:01:23.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:23 smithi045 conmon[39718]: cluster 2021-07-31T02:01:23.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:23 smithi045 conmon[39718]: 2021-07-31T02:01:23.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:23 smithi045 conmon[39718]: 2021 2021-07-31T02:01:23.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:23 smithi045 conmon[39718]: - 2021-07-31T02:01:23.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:23 smithi045 conmon[39718]: 07 2021-07-31T02:01:23.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:23 smithi045 conmon[39718]: - 2021-07-31T02:01:23.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:23 smithi045 conmon[39718]: 31 2021-07-31T02:01:23.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:23 smithi045 conmon[39718]: T 2021-07-31T02:01:23.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:23 smithi045 conmon[39718]: 02:01:22.200646 2021-07-31T02:01:23.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:23 smithi045 conmon[39718]: +0000 mon.a (mon.0) 1646 : cluster [DBG] osdmap e906: 8 total, 8 up, 8 in 2021-07-31T02:01:23.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:23 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T02:01:23.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:23 smithi045 conmon[39718]: 31T02:01:22.201817+0000 mon.c (mon.1) 49 : 2021-07-31T02:01:23.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:23 smithi045 conmon[39718]: audit [INF] from='client.? 172.21.15.45:0/3280736645' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73532-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:01:23.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:23 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T02:01:23.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:23 smithi045 conmon[39718]: T02:01:22.202811+0000 mon.a (mon.0 2021-07-31T02:01:23.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:23 smithi045 conmon[39718]: ) 1647 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73532-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:01:23.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:23 smithi045 conmon[33655]: cluster 2021-07-31T02:01:23.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:23 smithi045 conmon[33655]: 2021-07-31T02:01:23.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:23 smithi045 conmon[33655]: 2021 2021-07-31T02:01:23.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:23 smithi045 conmon[33655]: - 2021-07-31T02:01:23.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:23 smithi045 conmon[33655]: 07 2021-07-31T02:01:23.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:23 smithi045 conmon[33655]: - 2021-07-31T02:01:23.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:23 smithi045 conmon[33655]: 31 2021-07-31T02:01:23.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:23 smithi045 conmon[33655]: T 2021-07-31T02:01:23.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:23 smithi045 conmon[33655]: 02:01:22.200646+0000 mon.a (mon.0) 2021-07-31T02:01:23.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:23 smithi045 conmon[33655]: 1646 : cluster [DBG] osdmap e906: 8 total, 8 up, 8 in 2021-07-31T02:01:23.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:23 smithi045 conmon[33655]: audit 2021-07-31T02:01:23.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:23 smithi045 conmon[33655]: 2021-07-31T02:01:22.201817+0000 mon.c (mon.1) 49 : audit 2021-07-31T02:01:23.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:23 smithi045 conmon[33655]: [INF] from='client.? 172.21.15.45:0/3280736645' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73532-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:01:23.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:23 smithi045 conmon[33655]: audit 2021-07-31T02:01:22. 2021-07-31T02:01:23.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:23 smithi045 conmon[33655]: 202811+0000 mon.a (mon.0) 1647 : audit 2021-07-31T02:01:23.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:23 smithi045 conmon[33655]: [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73532-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:01:24.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:01:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:01:23.532Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:01:24.027 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:01:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:01:23.532Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:01:24.027 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:01:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:01:23.532Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:01:24.217 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] ClsHello.RecordHello (3026 ms) 2021-07-31T02:01:24.217 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] ClsHello.WriteReturnData 2021-07-31T02:01:24.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:24 smithi154 conmon[34553]: audit 2021-07-31T02:01:23.202459+0000 mon.a (mon.0) 1648 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73532-2","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:01:24.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:24 smithi154 conmon[34553]: cluster 2021-07-31T02:01:23.202798+0000 mon.a (mon.0) 1649 : cluster [DBG] osdmap e907: 8 total, 8 up, 8 in 2021-07-31T02:01:24.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:24 smithi154 conmon[34553]: cluster 2021-07-31T02:01:23.993471+0000 mgr.x (mgr.24542) 604 : cluster [DBG] pgmap v1179: 41 pgs: 41 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 511 B/s rd, 511 B/s wr, 1 op/s 2021-07-31T02:01:24.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:24 smithi045 conmon[33655]: audit 2021-07-31T02:01:23.202459+0000 mon.a (mon.0) 1648 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73532-2","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:01:24.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:24 smithi045 conmon[33655]: 2021-07-31T02:01:24.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:24 smithi045 conmon[33655]: cluster 2021-07-31T02:01:23.202798+0000 mon.a (mon.0) 1649 : cluster [DBG] osdmap e907: 8 total, 8 up, 8 in 2021-07-31T02:01:24.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:24 smithi045 conmon[33655]: cluster 2021-07 2021-07-31T02:01:24.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:24 smithi045 conmon[33655]: -31T02:01:23.993471+0000 mgr.x (mgr.24542) 604 : cluster [DBG] pgmap v1179: 41 pgs: 41 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 511 B/s rd, 511 B/s wr, 1 op/s 2021-07-31T02:01:24.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:24 smithi045 conmon[39718]: audit 2021-07-31T02:01:23.202459+0000 mon.a (mon. 2021-07-31T02:01:24.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:24 smithi045 conmon[39718]: 0) 1648 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73532-2","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:01:24.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:24 smithi045 conmon[39718]: cluster 2021-07-31T02:01:23.202798+0000 mon.a (mon.0) 1649 : 2021-07-31T02:01:24.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:24 smithi045 conmon[39718]: cluster [DBG] osdmap e907: 8 total, 8 up, 8 in 2021-07-31T02:01:24.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:24 smithi045 conmon[39718]: cluster 2021 2021-07-31T02:01:24.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:24 smithi045 conmon[39718]: -07-31T02:01:23.993471+0000 mgr.x (mgr 2021-07-31T02:01:24.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:24 smithi045 conmon[39718]: .24542) 604 : cluster [DBG] pgmap v1179: 41 pgs: 41 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 511 B/s rd, 511 B/s wr, 1 op/s 2021-07-31T02:01:25.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:25 smithi154 conmon[34553]: cluster 2021-07-31T02:01:24.213452+0000 mon.a (mon.0) 1650 : cluster 2021-07-31T02:01:25.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:25 smithi154 conmon[34553]: [DBG] osdmap e908: 8 total, 8 up, 8 in 2021-07-31T02:01:25.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:25 smithi045 conmon[39718]: cluster 2021-07-31T02:01:25.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:25 smithi045 conmon[39718]: 2021-07-31T02:01:25.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:25 smithi045 conmon[39718]: 2021 2021-07-31T02:01:25.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:25 smithi045 conmon[39718]: - 2021-07-31T02:01:25.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:25 smithi045 conmon[39718]: 07 2021-07-31T02:01:25.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:25 smithi045 conmon[39718]: - 2021-07-31T02:01:25.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:25 smithi045 conmon[39718]: 31 2021-07-31T02:01:25.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:25 smithi045 conmon[39718]: T 2021-07-31T02:01:25.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:25 smithi045 conmon[39718]: 02 2021-07-31T02:01:25.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:25 smithi045 conmon[39718]: : 2021-07-31T02:01:25.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:25 smithi045 conmon[39718]: 01 2021-07-31T02:01:25.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:25 smithi045 conmon[39718]: : 2021-07-31T02:01:25.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:25 smithi045 conmon[39718]: 24 2021-07-31T02:01:25.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:25 smithi045 conmon[39718]: . 2021-07-31T02:01:25.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:25 smithi045 conmon[39718]: 213452 2021-07-31T02:01:25.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:25 smithi045 conmon[39718]: +0000 2021-07-31T02:01:25.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:25 smithi045 conmon[39718]: 2021-07-31T02:01:25.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:25 smithi045 conmon[39718]: mon.a 2021-07-31T02:01:25.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:25 smithi045 conmon[39718]: ( 2021-07-31T02:01:25.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:25 smithi045 conmon[39718]: mon 2021-07-31T02:01:25.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:25 smithi045 conmon[39718]: . 2021-07-31T02:01:25.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:25 smithi045 conmon[39718]: 0 2021-07-31T02:01:25.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:25 smithi045 conmon[39718]: ) 1650 : cluster [DBG] osdmap e908: 8 total, 8 up, 8 in 2021-07-31T02:01:25.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:25 smithi045 conmon[33655]: cluster 2021-07-31T02:01:25.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:25 smithi045 conmon[33655]: 2021-07-31T02:01:25.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:25 smithi045 conmon[33655]: 2021 2021-07-31T02:01:25.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:25 smithi045 conmon[33655]: - 2021-07-31T02:01:25.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:25 smithi045 conmon[33655]: 07 2021-07-31T02:01:25.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:25 smithi045 conmon[33655]: - 2021-07-31T02:01:25.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:25 smithi045 conmon[33655]: 31 2021-07-31T02:01:25.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:25 smithi045 conmon[33655]: T 2021-07-31T02:01:25.535 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:25 smithi045 conmon[33655]: 02 2021-07-31T02:01:25.535 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:25 smithi045 conmon[33655]: : 2021-07-31T02:01:25.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:25 smithi045 conmon[33655]: 01 2021-07-31T02:01:25.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:25 smithi045 conmon[33655]: : 2021-07-31T02:01:25.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:25 smithi045 conmon[33655]: 24 2021-07-31T02:01:25.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:25 smithi045 conmon[33655]: . 2021-07-31T02:01:25.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:25 smithi045 conmon[33655]: 213452 2021-07-31T02:01:25.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:25 smithi045 conmon[33655]: +0000 2021-07-31T02:01:25.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:25 smithi045 conmon[33655]: 2021-07-31T02:01:25.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:25 smithi045 conmon[33655]: mon.a 2021-07-31T02:01:25.539 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:25 smithi045 conmon[33655]: ( 2021-07-31T02:01:25.539 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:25 smithi045 conmon[33655]: mon 2021-07-31T02:01:25.540 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:25 smithi045 conmon[33655]: . 2021-07-31T02:01:25.540 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:25 smithi045 conmon[33655]: 0 2021-07-31T02:01:25.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:25 smithi045 conmon[33655]: ) 2021-07-31T02:01:25.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:25 smithi045 conmon[33655]: 1650 2021-07-31T02:01:25.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:25 smithi045 conmon[33655]: : 2021-07-31T02:01:25.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:25 smithi045 conmon[33655]: cluster 2021-07-31T02:01:25.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:25 smithi045 conmon[33655]: 2021-07-31T02:01:25.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:25 smithi045 conmon[33655]: [DBG] 2021-07-31T02:01:25.543 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:25 smithi045 conmon[33655]: 2021-07-31T02:01:25.543 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:25 smithi045 conmon[33655]: osdmap e908: 8 total, 8 up, 8 in 2021-07-31T02:01:25.544 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:25 smithi045 conmon[33655]: 2021-07-31T02:01:26.224 INFO:tasks.workunit.client.0.smithi045.stdout:require_osd_release = pacific 2021-07-31T02:01:26.232 INFO:tasks.workunit.client.0.smithi045.stdout:0 2021-07-31T02:01:26.240 INFO:tasks.workunit.client.0.smithi045.stdout:1 2021-07-31T02:01:26.241 INFO:tasks.workunit.client.0.smithi045.stdout:2 2021-07-31T02:01:26.242 INFO:tasks.workunit.client.0.smithi045.stdout:3 2021-07-31T02:01:26.243 INFO:tasks.workunit.client.0.smithi045.stdout:4 2021-07-31T02:01:26.244 INFO:tasks.workunit.client.0.smithi045.stdout:5 2021-07-31T02:01:26.245 INFO:tasks.workunit.client.0.smithi045.stdout:6 2021-07-31T02:01:26.246 INFO:tasks.workunit.client.0.smithi045.stdout:7 2021-07-31T02:01:26.247 INFO:tasks.workunit.client.0.smithi045.stdout:8 2021-07-31T02:01:26.248 INFO:tasks.workunit.client.0.smithi045.stdout:9 2021-07-31T02:01:26.249 INFO:tasks.workunit.client.0.smithi045.stdout:10 2021-07-31T02:01:26.250 INFO:tasks.workunit.client.0.smithi045.stdout:11 2021-07-31T02:01:26.251 INFO:tasks.workunit.client.0.smithi045.stdout:12 2021-07-31T02:01:26.252 INFO:tasks.workunit.client.0.smithi045.stdout:13 2021-07-31T02:01:26.254 INFO:tasks.workunit.client.0.smithi045.stdout:14 2021-07-31T02:01:26.255 INFO:tasks.workunit.client.0.smithi045.stdout:15 2021-07-31T02:01:26.256 INFO:tasks.workunit.client.0.smithi045.stdout:16 2021-07-31T02:01:26.257 INFO:tasks.workunit.client.0.smithi045.stdout:17 2021-07-31T02:01:26.258 INFO:tasks.workunit.client.0.smithi045.stdout:18 2021-07-31T02:01:26.259 INFO:tasks.workunit.client.0.smithi045.stdout:19 2021-07-31T02:01:26.260 INFO:tasks.workunit.client.0.smithi045.stdout:20 2021-07-31T02:01:26.262 INFO:tasks.workunit.client.0.smithi045.stdout:21 2021-07-31T02:01:26.263 INFO:tasks.workunit.client.0.smithi045.stdout:22 2021-07-31T02:01:26.264 INFO:tasks.workunit.client.0.smithi045.stdout:23 2021-07-31T02:01:26.265 INFO:tasks.workunit.client.0.smithi045.stdout:24 2021-07-31T02:01:26.266 INFO:tasks.workunit.client.0.smithi045.stdout:25 2021-07-31T02:01:26.267 INFO:tasks.workunit.client.0.smithi045.stdout:26 2021-07-31T02:01:26.268 INFO:tasks.workunit.client.0.smithi045.stdout:27 2021-07-31T02:01:26.269 INFO:tasks.workunit.client.0.smithi045.stdout:28 2021-07-31T02:01:26.271 INFO:tasks.workunit.client.0.smithi045.stdout:29 2021-07-31T02:01:26.272 INFO:tasks.workunit.client.0.smithi045.stdout:30 2021-07-31T02:01:26.273 INFO:tasks.workunit.client.0.smithi045.stdout:31 2021-07-31T02:01:26.274 INFO:tasks.workunit.client.0.smithi045.stdout:32 2021-07-31T02:01:26.275 INFO:tasks.workunit.client.0.smithi045.stdout:33 2021-07-31T02:01:26.276 INFO:tasks.workunit.client.0.smithi045.stdout:34 2021-07-31T02:01:26.277 INFO:tasks.workunit.client.0.smithi045.stdout:35 2021-07-31T02:01:26.278 INFO:tasks.workunit.client.0.smithi045.stdout:36 2021-07-31T02:01:26.279 INFO:tasks.workunit.client.0.smithi045.stdout:37 2021-07-31T02:01:26.280 INFO:tasks.workunit.client.0.smithi045.stdout:38 2021-07-31T02:01:26.282 INFO:tasks.workunit.client.0.smithi045.stdout:39 2021-07-31T02:01:26.283 INFO:tasks.workunit.client.0.smithi045.stdout:40 2021-07-31T02:01:26.284 INFO:tasks.workunit.client.0.smithi045.stdout:41 2021-07-31T02:01:26.285 INFO:tasks.workunit.client.0.smithi045.stdout:42 2021-07-31T02:01:26.287 INFO:tasks.workunit.client.0.smithi045.stdout:43 2021-07-31T02:01:26.287 INFO:tasks.workunit.client.0.smithi045.stdout:44 2021-07-31T02:01:26.289 INFO:tasks.workunit.client.0.smithi045.stdout:45 2021-07-31T02:01:26.290 INFO:tasks.workunit.client.0.smithi045.stdout:46 2021-07-31T02:01:26.291 INFO:tasks.workunit.client.0.smithi045.stdout:47 2021-07-31T02:01:26.292 INFO:tasks.workunit.client.0.smithi045.stdout:48 2021-07-31T02:01:26.293 INFO:tasks.workunit.client.0.smithi045.stdout:49 2021-07-31T02:01:26.294 INFO:tasks.workunit.client.0.smithi045.stdout:50 2021-07-31T02:01:26.295 INFO:tasks.workunit.client.0.smithi045.stdout:51 2021-07-31T02:01:26.296 INFO:tasks.workunit.client.0.smithi045.stdout:52 2021-07-31T02:01:26.297 INFO:tasks.workunit.client.0.smithi045.stdout:53 2021-07-31T02:01:26.298 INFO:tasks.workunit.client.0.smithi045.stdout:54 2021-07-31T02:01:26.299 INFO:tasks.workunit.client.0.smithi045.stdout:55 2021-07-31T02:01:26.300 INFO:tasks.workunit.client.0.smithi045.stdout:56 2021-07-31T02:01:26.301 INFO:tasks.workunit.client.0.smithi045.stdout:57 2021-07-31T02:01:26.302 INFO:tasks.workunit.client.0.smithi045.stdout:58 2021-07-31T02:01:26.304 INFO:tasks.workunit.client.0.smithi045.stdout:59 2021-07-31T02:01:26.305 INFO:tasks.workunit.client.0.smithi045.stdout:60 2021-07-31T02:01:26.306 INFO:tasks.workunit.client.0.smithi045.stdout:61 2021-07-31T02:01:26.307 INFO:tasks.workunit.client.0.smithi045.stdout:62 2021-07-31T02:01:26.308 INFO:tasks.workunit.client.0.smithi045.stdout:63 2021-07-31T02:01:26.309 INFO:tasks.workunit.client.0.smithi045.stdout:64 2021-07-31T02:01:26.310 INFO:tasks.workunit.client.0.smithi045.stdout:65 2021-07-31T02:01:26.311 INFO:tasks.workunit.client.0.smithi045.stdout:66 2021-07-31T02:01:26.312 INFO:tasks.workunit.client.0.smithi045.stdout:67 2021-07-31T02:01:26.313 INFO:tasks.workunit.client.0.smithi045.stdout:68 2021-07-31T02:01:26.314 INFO:tasks.workunit.client.0.smithi045.stdout:69 2021-07-31T02:01:26.315 INFO:tasks.workunit.client.0.smithi045.stdout:70 2021-07-31T02:01:26.317 INFO:tasks.workunit.client.0.smithi045.stdout:71 2021-07-31T02:01:26.317 INFO:tasks.workunit.client.0.smithi045.stdout:72 2021-07-31T02:01:26.319 INFO:tasks.workunit.client.0.smithi045.stdout:73 2021-07-31T02:01:26.320 INFO:tasks.workunit.client.0.smithi045.stdout:74 2021-07-31T02:01:26.320 INFO:tasks.workunit.client.0.smithi045.stdout:75 2021-07-31T02:01:26.321 INFO:tasks.workunit.client.0.smithi045.stdout:76 2021-07-31T02:01:26.322 INFO:tasks.workunit.client.0.smithi045.stdout:77 2021-07-31T02:01:26.323 INFO:tasks.workunit.client.0.smithi045.stdout:78 2021-07-31T02:01:26.324 INFO:tasks.workunit.client.0.smithi045.stdout:79 2021-07-31T02:01:26.325 INFO:tasks.workunit.client.0.smithi045.stdout:80 2021-07-31T02:01:26.326 INFO:tasks.workunit.client.0.smithi045.stdout:81 2021-07-31T02:01:26.327 INFO:tasks.workunit.client.0.smithi045.stdout:82 2021-07-31T02:01:26.328 INFO:tasks.workunit.client.0.smithi045.stdout:83 2021-07-31T02:01:26.329 INFO:tasks.workunit.client.0.smithi045.stdout:84 2021-07-31T02:01:26.330 INFO:tasks.workunit.client.0.smithi045.stdout:85 2021-07-31T02:01:26.331 INFO:tasks.workunit.client.0.smithi045.stdout:86 2021-07-31T02:01:26.332 INFO:tasks.workunit.client.0.smithi045.stdout:87 2021-07-31T02:01:26.333 INFO:tasks.workunit.client.0.smithi045.stdout:88 2021-07-31T02:01:26.333 INFO:tasks.workunit.client.0.smithi045.stdout:89 2021-07-31T02:01:26.334 INFO:tasks.workunit.client.0.smithi045.stdout:90 2021-07-31T02:01:26.335 INFO:tasks.workunit.client.0.smithi045.stdout:91 2021-07-31T02:01:26.336 INFO:tasks.workunit.client.0.smithi045.stdout:92 2021-07-31T02:01:26.337 INFO:tasks.workunit.client.0.smithi045.stdout:93 2021-07-31T02:01:26.338 INFO:tasks.workunit.client.0.smithi045.stdout:94 2021-07-31T02:01:26.339 INFO:tasks.workunit.client.0.smithi045.stdout:95 2021-07-31T02:01:26.340 INFO:tasks.workunit.client.0.smithi045.stdout:96 2021-07-31T02:01:26.341 INFO:tasks.workunit.client.0.smithi045.stdout:97 2021-07-31T02:01:26.342 INFO:tasks.workunit.client.0.smithi045.stdout:98 2021-07-31T02:01:26.343 INFO:tasks.workunit.client.0.smithi045.stdout:99 2021-07-31T02:01:26.343 INFO:tasks.workunit.client.0.smithi045.stdout:100 2021-07-31T02:01:26.345 INFO:tasks.workunit.client.0.smithi045.stdout:101 2021-07-31T02:01:26.345 INFO:tasks.workunit.client.0.smithi045.stdout:102 2021-07-31T02:01:26.346 INFO:tasks.workunit.client.0.smithi045.stdout:103 2021-07-31T02:01:26.347 INFO:tasks.workunit.client.0.smithi045.stdout:104 2021-07-31T02:01:26.348 INFO:tasks.workunit.client.0.smithi045.stdout:105 2021-07-31T02:01:26.349 INFO:tasks.workunit.client.0.smithi045.stdout:106 2021-07-31T02:01:26.350 INFO:tasks.workunit.client.0.smithi045.stdout:107 2021-07-31T02:01:26.351 INFO:tasks.workunit.client.0.smithi045.stdout:108 2021-07-31T02:01:26.352 INFO:tasks.workunit.client.0.smithi045.stdout:109 2021-07-31T02:01:26.353 INFO:tasks.workunit.client.0.smithi045.stdout:110 2021-07-31T02:01:26.354 INFO:tasks.workunit.client.0.smithi045.stdout:111 2021-07-31T02:01:26.355 INFO:tasks.workunit.client.0.smithi045.stdout:112 2021-07-31T02:01:26.355 INFO:tasks.workunit.client.0.smithi045.stdout:113 2021-07-31T02:01:26.356 INFO:tasks.workunit.client.0.smithi045.stdout:114 2021-07-31T02:01:26.357 INFO:tasks.workunit.client.0.smithi045.stdout:115 2021-07-31T02:01:26.358 INFO:tasks.workunit.client.0.smithi045.stdout:116 2021-07-31T02:01:26.359 INFO:tasks.workunit.client.0.smithi045.stdout:117 2021-07-31T02:01:26.360 INFO:tasks.workunit.client.0.smithi045.stdout:118 2021-07-31T02:01:26.361 INFO:tasks.workunit.client.0.smithi045.stdout:119 2021-07-31T02:01:26.362 INFO:tasks.workunit.client.0.smithi045.stdout:120 2021-07-31T02:01:26.363 INFO:tasks.workunit.client.0.smithi045.stdout:121 2021-07-31T02:01:26.364 INFO:tasks.workunit.client.0.smithi045.stdout:122 2021-07-31T02:01:26.365 INFO:tasks.workunit.client.0.smithi045.stdout:123 2021-07-31T02:01:26.366 INFO:tasks.workunit.client.0.smithi045.stdout:124 2021-07-31T02:01:26.366 INFO:tasks.workunit.client.0.smithi045.stdout:125 2021-07-31T02:01:26.367 INFO:tasks.workunit.client.0.smithi045.stdout:126 2021-07-31T02:01:26.368 INFO:tasks.workunit.client.0.smithi045.stdout:127 2021-07-31T02:01:26.369 INFO:tasks.workunit.client.0.smithi045.stdout:128 2021-07-31T02:01:26.370 INFO:tasks.workunit.client.0.smithi045.stdout:129 2021-07-31T02:01:26.371 INFO:tasks.workunit.client.0.smithi045.stdout:130 2021-07-31T02:01:26.372 INFO:tasks.workunit.client.0.smithi045.stdout:131 2021-07-31T02:01:26.373 INFO:tasks.workunit.client.0.smithi045.stdout:132 2021-07-31T02:01:26.374 INFO:tasks.workunit.client.0.smithi045.stdout:133 2021-07-31T02:01:26.375 INFO:tasks.workunit.client.0.smithi045.stdout:134 2021-07-31T02:01:26.376 INFO:tasks.workunit.client.0.smithi045.stdout:135 2021-07-31T02:01:26.376 INFO:tasks.workunit.client.0.smithi045.stdout:136 2021-07-31T02:01:26.377 INFO:tasks.workunit.client.0.smithi045.stdout:137 2021-07-31T02:01:26.378 INFO:tasks.workunit.client.0.smithi045.stdout:138 2021-07-31T02:01:26.379 INFO:tasks.workunit.client.0.smithi045.stdout:139 2021-07-31T02:01:26.380 INFO:tasks.workunit.client.0.smithi045.stdout:140 2021-07-31T02:01:26.381 INFO:tasks.workunit.client.0.smithi045.stdout:141 2021-07-31T02:01:26.382 INFO:tasks.workunit.client.0.smithi045.stdout:142 2021-07-31T02:01:26.383 INFO:tasks.workunit.client.0.smithi045.stdout:143 2021-07-31T02:01:26.384 INFO:tasks.workunit.client.0.smithi045.stdout:144 2021-07-31T02:01:26.385 INFO:tasks.workunit.client.0.smithi045.stdout:145 2021-07-31T02:01:26.386 INFO:tasks.workunit.client.0.smithi045.stdout:146 2021-07-31T02:01:26.387 INFO:tasks.workunit.client.0.smithi045.stdout:147 2021-07-31T02:01:26.387 INFO:tasks.workunit.client.0.smithi045.stdout:148 2021-07-31T02:01:26.388 INFO:tasks.workunit.client.0.smithi045.stdout:149 2021-07-31T02:01:26.389 INFO:tasks.workunit.client.0.smithi045.stdout:150 2021-07-31T02:01:26.390 INFO:tasks.workunit.client.0.smithi045.stdout:151 2021-07-31T02:01:26.391 INFO:tasks.workunit.client.0.smithi045.stdout:152 2021-07-31T02:01:26.392 INFO:tasks.workunit.client.0.smithi045.stdout:153 2021-07-31T02:01:26.393 INFO:tasks.workunit.client.0.smithi045.stdout:154 2021-07-31T02:01:26.394 INFO:tasks.workunit.client.0.smithi045.stdout:155 2021-07-31T02:01:26.395 INFO:tasks.workunit.client.0.smithi045.stdout:156 2021-07-31T02:01:26.397 INFO:tasks.workunit.client.0.smithi045.stdout:157 2021-07-31T02:01:26.398 INFO:tasks.workunit.client.0.smithi045.stdout:158 2021-07-31T02:01:26.398 INFO:tasks.workunit.client.0.smithi045.stdout:159 2021-07-31T02:01:26.399 INFO:tasks.workunit.client.0.smithi045.stdout:160 2021-07-31T02:01:26.400 INFO:tasks.workunit.client.0.smithi045.stdout:161 2021-07-31T02:01:26.401 INFO:tasks.workunit.client.0.smithi045.stdout:162 2021-07-31T02:01:26.402 INFO:tasks.workunit.client.0.smithi045.stdout:163 2021-07-31T02:01:26.403 INFO:tasks.workunit.client.0.smithi045.stdout:164 2021-07-31T02:01:26.404 INFO:tasks.workunit.client.0.smithi045.stdout:165 2021-07-31T02:01:26.405 INFO:tasks.workunit.client.0.smithi045.stdout:166 2021-07-31T02:01:26.406 INFO:tasks.workunit.client.0.smithi045.stdout:167 2021-07-31T02:01:26.407 INFO:tasks.workunit.client.0.smithi045.stdout:168 2021-07-31T02:01:26.408 INFO:tasks.workunit.client.0.smithi045.stdout:169 2021-07-31T02:01:26.409 INFO:tasks.workunit.client.0.smithi045.stdout:170 2021-07-31T02:01:26.410 INFO:tasks.workunit.client.0.smithi045.stdout:171 2021-07-31T02:01:26.411 INFO:tasks.workunit.client.0.smithi045.stdout:172 2021-07-31T02:01:26.412 INFO:tasks.workunit.client.0.smithi045.stdout:173 2021-07-31T02:01:26.413 INFO:tasks.workunit.client.0.smithi045.stdout:174 2021-07-31T02:01:26.414 INFO:tasks.workunit.client.0.smithi045.stdout:175 2021-07-31T02:01:26.414 INFO:tasks.workunit.client.0.smithi045.stdout:176 2021-07-31T02:01:26.415 INFO:tasks.workunit.client.0.smithi045.stdout:177 2021-07-31T02:01:26.416 INFO:tasks.workunit.client.0.smithi045.stdout:178 2021-07-31T02:01:26.418 INFO:tasks.workunit.client.0.smithi045.stdout:179 2021-07-31T02:01:26.418 INFO:tasks.workunit.client.0.smithi045.stdout:180 2021-07-31T02:01:26.419 INFO:tasks.workunit.client.0.smithi045.stdout:181 2021-07-31T02:01:26.420 INFO:tasks.workunit.client.0.smithi045.stdout:182 2021-07-31T02:01:26.421 INFO:tasks.workunit.client.0.smithi045.stdout:183 2021-07-31T02:01:26.422 INFO:tasks.workunit.client.0.smithi045.stdout:184 2021-07-31T02:01:26.423 INFO:tasks.workunit.client.0.smithi045.stdout:185 2021-07-31T02:01:26.424 INFO:tasks.workunit.client.0.smithi045.stdout:186 2021-07-31T02:01:26.425 INFO:tasks.workunit.client.0.smithi045.stdout:187 2021-07-31T02:01:26.426 INFO:tasks.workunit.client.0.smithi045.stdout:188 2021-07-31T02:01:26.426 INFO:tasks.workunit.client.0.smithi045.stdout:189 2021-07-31T02:01:26.427 INFO:tasks.workunit.client.0.smithi045.stdout:190 2021-07-31T02:01:26.429 INFO:tasks.workunit.client.0.smithi045.stdout:191 2021-07-31T02:01:26.429 INFO:tasks.workunit.client.0.smithi045.stdout:192 2021-07-31T02:01:26.430 INFO:tasks.workunit.client.0.smithi045.stdout:193 2021-07-31T02:01:26.431 INFO:tasks.workunit.client.0.smithi045.stdout:194 2021-07-31T02:01:26.432 INFO:tasks.workunit.client.0.smithi045.stdout:195 2021-07-31T02:01:26.433 INFO:tasks.workunit.client.0.smithi045.stdout:196 2021-07-31T02:01:26.434 INFO:tasks.workunit.client.0.smithi045.stdout:197 2021-07-31T02:01:26.435 INFO:tasks.workunit.client.0.smithi045.stdout:198 2021-07-31T02:01:26.435 INFO:tasks.workunit.client.0.smithi045.stdout:199 2021-07-31T02:01:26.436 INFO:tasks.workunit.client.0.smithi045.stdout:200 2021-07-31T02:01:26.437 INFO:tasks.workunit.client.0.smithi045.stdout:201 2021-07-31T02:01:26.438 INFO:tasks.workunit.client.0.smithi045.stdout:202 2021-07-31T02:01:26.439 INFO:tasks.workunit.client.0.smithi045.stdout:203 2021-07-31T02:01:26.440 INFO:tasks.workunit.client.0.smithi045.stdout:204 2021-07-31T02:01:26.441 INFO:tasks.workunit.client.0.smithi045.stdout:205 2021-07-31T02:01:26.442 INFO:tasks.workunit.client.0.smithi045.stdout:206 2021-07-31T02:01:26.443 INFO:tasks.workunit.client.0.smithi045.stdout:207 2021-07-31T02:01:26.444 INFO:tasks.workunit.client.0.smithi045.stdout:208 2021-07-31T02:01:26.445 INFO:tasks.workunit.client.0.smithi045.stdout:209 2021-07-31T02:01:26.446 INFO:tasks.workunit.client.0.smithi045.stdout:210 2021-07-31T02:01:26.446 INFO:tasks.workunit.client.0.smithi045.stdout:211 2021-07-31T02:01:26.448 INFO:tasks.workunit.client.0.smithi045.stdout:212 2021-07-31T02:01:26.448 INFO:tasks.workunit.client.0.smithi045.stdout:213 2021-07-31T02:01:26.449 INFO:tasks.workunit.client.0.smithi045.stdout:214 2021-07-31T02:01:26.450 INFO:tasks.workunit.client.0.smithi045.stdout:215 2021-07-31T02:01:26.451 INFO:tasks.workunit.client.0.smithi045.stdout:216 2021-07-31T02:01:26.452 INFO:tasks.workunit.client.0.smithi045.stdout:217 2021-07-31T02:01:26.453 INFO:tasks.workunit.client.0.smithi045.stdout:218 2021-07-31T02:01:26.454 INFO:tasks.workunit.client.0.smithi045.stdout:219 2021-07-31T02:01:26.455 INFO:tasks.workunit.client.0.smithi045.stdout:220 2021-07-31T02:01:26.456 INFO:tasks.workunit.client.0.smithi045.stdout:221 2021-07-31T02:01:26.457 INFO:tasks.workunit.client.0.smithi045.stdout:222 2021-07-31T02:01:26.458 INFO:tasks.workunit.client.0.smithi045.stdout:223 2021-07-31T02:01:26.459 INFO:tasks.workunit.client.0.smithi045.stdout:224 2021-07-31T02:01:26.460 INFO:tasks.workunit.client.0.smithi045.stdout:225 2021-07-31T02:01:26.460 INFO:tasks.workunit.client.0.smithi045.stdout:226 2021-07-31T02:01:26.461 INFO:tasks.workunit.client.0.smithi045.stdout:227 2021-07-31T02:01:26.463 INFO:tasks.workunit.client.0.smithi045.stdout:228 2021-07-31T02:01:26.464 INFO:tasks.workunit.client.0.smithi045.stdout:229 2021-07-31T02:01:26.464 INFO:tasks.workunit.client.0.smithi045.stdout:230 2021-07-31T02:01:26.465 INFO:tasks.workunit.client.0.smithi045.stdout:231 2021-07-31T02:01:26.466 INFO:tasks.workunit.client.0.smithi045.stdout:232 2021-07-31T02:01:26.467 INFO:tasks.workunit.client.0.smithi045.stdout:233 2021-07-31T02:01:26.468 INFO:tasks.workunit.client.0.smithi045.stdout:234 2021-07-31T02:01:26.469 INFO:tasks.workunit.client.0.smithi045.stdout:235 2021-07-31T02:01:26.470 INFO:tasks.workunit.client.0.smithi045.stdout:236 2021-07-31T02:01:26.471 INFO:tasks.workunit.client.0.smithi045.stdout:237 2021-07-31T02:01:26.472 INFO:tasks.workunit.client.0.smithi045.stdout:238 2021-07-31T02:01:26.472 INFO:tasks.workunit.client.0.smithi045.stdout:239 2021-07-31T02:01:26.473 INFO:tasks.workunit.client.0.smithi045.stdout:240 2021-07-31T02:01:26.474 INFO:tasks.workunit.client.0.smithi045.stdout:241 2021-07-31T02:01:26.475 INFO:tasks.workunit.client.0.smithi045.stdout:242 2021-07-31T02:01:26.476 INFO:tasks.workunit.client.0.smithi045.stdout:243 2021-07-31T02:01:26.477 INFO:tasks.workunit.client.0.smithi045.stdout:244 2021-07-31T02:01:26.478 INFO:tasks.workunit.client.0.smithi045.stdout:245 2021-07-31T02:01:26.479 INFO:tasks.workunit.client.0.smithi045.stdout:246 2021-07-31T02:01:26.480 INFO:tasks.workunit.client.0.smithi045.stdout:247 2021-07-31T02:01:26.481 INFO:tasks.workunit.client.0.smithi045.stdout:248 2021-07-31T02:01:26.481 INFO:tasks.workunit.client.0.smithi045.stdout:249 2021-07-31T02:01:26.482 INFO:tasks.workunit.client.0.smithi045.stdout:250 2021-07-31T02:01:26.483 INFO:tasks.workunit.client.0.smithi045.stdout:251 2021-07-31T02:01:26.484 INFO:tasks.workunit.client.0.smithi045.stdout:252 2021-07-31T02:01:26.485 INFO:tasks.workunit.client.0.smithi045.stdout:253 2021-07-31T02:01:26.486 INFO:tasks.workunit.client.0.smithi045.stdout:254 2021-07-31T02:01:26.487 INFO:tasks.workunit.client.0.smithi045.stdout:255 2021-07-31T02:01:26.488 INFO:tasks.workunit.client.0.smithi045.stdout:256 2021-07-31T02:01:26.489 INFO:tasks.workunit.client.0.smithi045.stdout:257 2021-07-31T02:01:26.489 INFO:tasks.workunit.client.0.smithi045.stdout:258 2021-07-31T02:01:26.490 INFO:tasks.workunit.client.0.smithi045.stdout:259 2021-07-31T02:01:26.491 INFO:tasks.workunit.client.0.smithi045.stdout:260 2021-07-31T02:01:26.492 INFO:tasks.workunit.client.0.smithi045.stdout:261 2021-07-31T02:01:26.493 INFO:tasks.workunit.client.0.smithi045.stdout:262 2021-07-31T02:01:26.494 INFO:tasks.workunit.client.0.smithi045.stdout:263 2021-07-31T02:01:26.495 INFO:tasks.workunit.client.0.smithi045.stdout:264 2021-07-31T02:01:26.496 INFO:tasks.workunit.client.0.smithi045.stdout:265 2021-07-31T02:01:26.497 INFO:tasks.workunit.client.0.smithi045.stdout:266 2021-07-31T02:01:26.498 INFO:tasks.workunit.client.0.smithi045.stdout:267 2021-07-31T02:01:26.499 INFO:tasks.workunit.client.0.smithi045.stdout:268 2021-07-31T02:01:26.500 INFO:tasks.workunit.client.0.smithi045.stdout:269 2021-07-31T02:01:26.501 INFO:tasks.workunit.client.0.smithi045.stdout:270 2021-07-31T02:01:26.501 INFO:tasks.workunit.client.0.smithi045.stdout:271 2021-07-31T02:01:26.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:26 smithi154 conmon[34553]: cluster 2021-07-31T02:01:25.216216+0000 mon.a (mon.0) 1651 : cluster [DBG] 2021-07-31T02:01:26.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:26 smithi154 conmon[34553]: osdmap e909: 8 total, 8 up, 8 in 2021-07-31T02:01:26.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:26 smithi154 conmon[34553]: audit 2021-07-31T02:01:25.220827+0000 mon.a (mon.0) 1652 : audit [INF] from='client.? 172.21.15.45:0/2013046777' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73532-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:01:26.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:26 smithi154 conmon[34553]: cluster 2021-07-31T02:01:25.993903+0000 mgr.x (mgr.24542) 605 : cluster [DBG] pgmap v1182: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:01:26.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:26 smithi154 conmon[34553]: 2021-07-31T02:01:26.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:26 smithi154 conmon[34553]: audit 2021-07-31T02:01:26.218420+0000 mon.a (mon.0) 2021-07-31T02:01:26.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:26 smithi154 conmon[34553]: 1653 : audit [INF] from='client.? 172.21.15.45:0/2013046777' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73532-3","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:01:26.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:26 smithi154 conmon[34553]: cluster 2021-07-31T02:01:26 2021-07-31T02:01:26.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:26 smithi154 conmon[34553]: .218627+0000 mon.a (mon.0) 1654 : cluster 2021-07-31T02:01:26.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:26 smithi154 conmon[34553]: [DBG] osdmap e910: 8 total, 8 up, 8 in 2021-07-31T02:01:26.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:26 smithi154 conmon[34553]: audit 2021-07-31T02 2021-07-31T02:01:26.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:26 smithi154 conmon[34553]: :01:26.221450+0000 mon.a (mon.0 2021-07-31T02:01:26.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:26 smithi154 conmon[34553]: ) 1655 : audit [DBG] from='client.? 172.21.15.45:0/2013046777' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2021-07-31T02:01:26.506 INFO:tasks.workunit.client.0.smithi045.stdout:272 2021-07-31T02:01:26.507 INFO:tasks.workunit.client.0.smithi045.stdout:273 2021-07-31T02:01:26.507 INFO:tasks.workunit.client.0.smithi045.stdout:274 2021-07-31T02:01:26.507 INFO:tasks.workunit.client.0.smithi045.stdout:275 2021-07-31T02:01:26.507 INFO:tasks.workunit.client.0.smithi045.stdout:276 2021-07-31T02:01:26.508 INFO:tasks.workunit.client.0.smithi045.stdout:277 2021-07-31T02:01:26.508 INFO:tasks.workunit.client.0.smithi045.stdout:278 2021-07-31T02:01:26.509 INFO:tasks.workunit.client.0.smithi045.stdout:279 2021-07-31T02:01:26.510 INFO:tasks.workunit.client.0.smithi045.stdout:280 2021-07-31T02:01:26.511 INFO:tasks.workunit.client.0.smithi045.stdout:281 2021-07-31T02:01:26.512 INFO:tasks.workunit.client.0.smithi045.stdout:282 2021-07-31T02:01:26.512 INFO:tasks.workunit.client.0.smithi045.stdout:283 2021-07-31T02:01:26.513 INFO:tasks.workunit.client.0.smithi045.stdout:284 2021-07-31T02:01:26.514 INFO:tasks.workunit.client.0.smithi045.stdout:285 2021-07-31T02:01:26.515 INFO:tasks.workunit.client.0.smithi045.stdout:286 2021-07-31T02:01:26.516 INFO:tasks.workunit.client.0.smithi045.stdout:287 2021-07-31T02:01:26.517 INFO:tasks.workunit.client.0.smithi045.stdout:288 2021-07-31T02:01:26.518 INFO:tasks.workunit.client.0.smithi045.stdout:289 2021-07-31T02:01:26.519 INFO:tasks.workunit.client.0.smithi045.stdout:290 2021-07-31T02:01:26.520 INFO:tasks.workunit.client.0.smithi045.stdout:291 2021-07-31T02:01:26.521 INFO:tasks.workunit.client.0.smithi045.stdout:292 2021-07-31T02:01:26.522 INFO:tasks.workunit.client.0.smithi045.stdout:293 2021-07-31T02:01:26.523 INFO:tasks.workunit.client.0.smithi045.stdout:294 2021-07-31T02:01:26.524 INFO:tasks.workunit.client.0.smithi045.stdout:295 2021-07-31T02:01:26.525 INFO:tasks.workunit.client.0.smithi045.stdout:296 2021-07-31T02:01:26.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:26 smithi045 conmon[33655]: cluster 2021-07-31T02:01:25.216216 2021-07-31T02:01:26.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:26 smithi045 conmon[33655]: +0000 mon.a (mon.0) 1651 : cluster [DBG] osdmap e909: 8 total, 8 up, 8 in 2021-07-31T02:01:26.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:26 smithi045 conmon[33655]: audit 2021-07-31T02:01:25.220827+0000 mon.a (mon.0) 1652 : audit [INF] from='client.? 172.21.15.45:0/2013046777' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73532-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:01:26.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:26 smithi045 conmon[33655]: cluster 2021-07-31T02:01:25.993903+0000 mgr.x (mgr.24542) 605 : cluster [DBG] pgmap v1182: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:01:26.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:26 smithi045 conmon[33655]: audit 2021-07-31T02:01:26.218420+0000 mon.a (mon.0) 1653 : audit [INF] from='client.? 172.21.15.45:0/2013046777' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73532-3","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:01:26.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:26 smithi045 conmon[33655]: cluster 2021-07-31T02:01:26.218627+0000 mon.a (mon.0) 1654 : cluster [DBG] osdmap e910: 8 total, 8 up, 8 in 2021-07-31T02:01:26.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:26 smithi045 conmon[33655]: audit 2021-07-31T02:01:26.221450+0000 mon.a (mon.0) 1655 : audit [DBG] from='client.? 172.21.15.45:0/2013046777' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2021-07-31T02:01:26.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:26 smithi045 conmon[39718]: cluster 2021-07-31T02:01:26.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:26 smithi045 conmon[39718]: 2021-07-31T02:01:26.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:26 smithi045 conmon[39718]: 2021-07-31T02:01:25.216216 2021-07-31T02:01:26.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:26 smithi045 conmon[39718]: +0000 mon.a (mon.0) 1651 : cluster [DBG] osdmap e909: 8 total, 8 up, 8 in 2021-07-31T02:01:26.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:26 smithi045 conmon[39718]: audit 2021-07-31T02:01:26.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:26 smithi045 conmon[39718]: 2021-07-31T 2021-07-31T02:01:26.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:26 smithi045 conmon[39718]: 02:01:25.220827+0000 mon.a (mon.0) 1652 2021-07-31T02:01:26.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:26 smithi045 conmon[39718]: : audit [INF] from='client.? 172.21.15.45:0/2013046777' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73532-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:01:26.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:26 smithi045 conmon[39718]: cluster 2021- 2021-07-31T02:01:26.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:26 smithi045 conmon[39718]: 07-31T02:01:25.993903+0000 mgr.x 2021-07-31T02:01:26.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:26 smithi045 conmon[39718]: (mgr.24542) 605 : cluster [DBG] 2021-07-31T02:01:26.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:26 smithi045 conmon[39718]: pgmap v1182: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:01:26.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:26 smithi045 conmon[39718]: audit 2021-07-31T02:01:26.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:26 smithi045 conmon[39718]: 2021-07-31T02: 2021-07-31T02:01:26.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:26 smithi045 conmon[39718]: 01:26.218420+0000 mon.a (mon. 2021-07-31T02:01:26.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:26 smithi045 conmon[39718]: 0) 1653 : audit [INF] from='client.? 172.21.15.45:0/2013046777' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73532-3","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:01:26.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:26 smithi045 conmon[39718]: cluster 2021-07-31T02:01:26.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:26 smithi045 conmon[39718]: 2021-07-31T02:01:26. 2021-07-31T02:01:26.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:26 smithi045 conmon[39718]: 218627+0000 mon.a (mon.0) 1654 : cluster 2021-07-31T02:01:26.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:26 smithi045 conmon[39718]: [DBG] osdmap e910: 8 total, 8 up, 8 in 2021-07-31T02:01:26.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:26 smithi045 conmon[39718]: audit 2021-07-31T02:01:26.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:26 smithi045 conmon[39718]: 2021-07-31T02:01 2021-07-31T02:01:26.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:26 smithi045 conmon[39718]: :26.221450+0000 mon.a (mon.0 2021-07-31T02:01:26.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:26 smithi045 conmon[39718]: ) 1655 : audit [DBG] from='client.? 172.21.15.45:0/2013046777' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2021-07-31T02:01:26.536 INFO:tasks.workunit.client.0.smithi045.stdout:297 2021-07-31T02:01:26.536 INFO:tasks.workunit.client.0.smithi045.stdout:298 2021-07-31T02:01:26.536 INFO:tasks.workunit.client.0.smithi045.stdout:299 2021-07-31T02:01:26.536 INFO:tasks.workunit.client.0.smithi045.stdout:300 2021-07-31T02:01:26.537 INFO:tasks.workunit.client.0.smithi045.stdout:301 2021-07-31T02:01:26.537 INFO:tasks.workunit.client.0.smithi045.stdout:302 2021-07-31T02:01:26.537 INFO:tasks.workunit.client.0.smithi045.stdout:303 2021-07-31T02:01:26.537 INFO:tasks.workunit.client.0.smithi045.stdout:304 2021-07-31T02:01:26.537 INFO:tasks.workunit.client.0.smithi045.stdout:305 2021-07-31T02:01:26.538 INFO:tasks.workunit.client.0.smithi045.stdout:306 2021-07-31T02:01:26.538 INFO:tasks.workunit.client.0.smithi045.stdout:307 2021-07-31T02:01:26.539 INFO:tasks.workunit.client.0.smithi045.stdout:308 2021-07-31T02:01:26.539 INFO:tasks.workunit.client.0.smithi045.stdout:309 2021-07-31T02:01:26.540 INFO:tasks.workunit.client.0.smithi045.stdout:310 2021-07-31T02:01:26.540 INFO:tasks.workunit.client.0.smithi045.stdout:311 2021-07-31T02:01:26.540 INFO:tasks.workunit.client.0.smithi045.stdout:312 2021-07-31T02:01:26.541 INFO:tasks.workunit.client.0.smithi045.stdout:313 2021-07-31T02:01:26.542 INFO:tasks.workunit.client.0.smithi045.stdout:314 2021-07-31T02:01:26.543 INFO:tasks.workunit.client.0.smithi045.stdout:315 2021-07-31T02:01:26.544 INFO:tasks.workunit.client.0.smithi045.stdout:316 2021-07-31T02:01:26.545 INFO:tasks.workunit.client.0.smithi045.stdout:317 2021-07-31T02:01:26.546 INFO:tasks.workunit.client.0.smithi045.stdout:318 2021-07-31T02:01:26.547 INFO:tasks.workunit.client.0.smithi045.stdout:319 2021-07-31T02:01:26.548 INFO:tasks.workunit.client.0.smithi045.stdout:320 2021-07-31T02:01:26.548 INFO:tasks.workunit.client.0.smithi045.stdout:321 2021-07-31T02:01:26.549 INFO:tasks.workunit.client.0.smithi045.stdout:322 2021-07-31T02:01:26.550 INFO:tasks.workunit.client.0.smithi045.stdout:323 2021-07-31T02:01:26.551 INFO:tasks.workunit.client.0.smithi045.stdout:324 2021-07-31T02:01:26.552 INFO:tasks.workunit.client.0.smithi045.stdout:325 2021-07-31T02:01:26.553 INFO:tasks.workunit.client.0.smithi045.stdout:326 2021-07-31T02:01:26.554 INFO:tasks.workunit.client.0.smithi045.stdout:327 2021-07-31T02:01:26.555 INFO:tasks.workunit.client.0.smithi045.stdout:328 2021-07-31T02:01:26.556 INFO:tasks.workunit.client.0.smithi045.stdout:329 2021-07-31T02:01:26.557 INFO:tasks.workunit.client.0.smithi045.stdout:330 2021-07-31T02:01:26.558 INFO:tasks.workunit.client.0.smithi045.stdout:331 2021-07-31T02:01:26.559 INFO:tasks.workunit.client.0.smithi045.stdout:332 2021-07-31T02:01:26.560 INFO:tasks.workunit.client.0.smithi045.stdout:333 2021-07-31T02:01:26.561 INFO:tasks.workunit.client.0.smithi045.stdout:334 2021-07-31T02:01:26.562 INFO:tasks.workunit.client.0.smithi045.stdout:335 2021-07-31T02:01:26.563 INFO:tasks.workunit.client.0.smithi045.stdout:336 2021-07-31T02:01:26.564 INFO:tasks.workunit.client.0.smithi045.stdout:337 2021-07-31T02:01:26.565 INFO:tasks.workunit.client.0.smithi045.stdout:338 2021-07-31T02:01:26.566 INFO:tasks.workunit.client.0.smithi045.stdout:339 2021-07-31T02:01:26.566 INFO:tasks.workunit.client.0.smithi045.stdout:340 2021-07-31T02:01:26.567 INFO:tasks.workunit.client.0.smithi045.stdout:341 2021-07-31T02:01:26.568 INFO:tasks.workunit.client.0.smithi045.stdout:342 2021-07-31T02:01:26.569 INFO:tasks.workunit.client.0.smithi045.stdout:343 2021-07-31T02:01:26.570 INFO:tasks.workunit.client.0.smithi045.stdout:344 2021-07-31T02:01:26.571 INFO:tasks.workunit.client.0.smithi045.stdout:345 2021-07-31T02:01:26.572 INFO:tasks.workunit.client.0.smithi045.stdout:346 2021-07-31T02:01:26.573 INFO:tasks.workunit.client.0.smithi045.stdout:347 2021-07-31T02:01:26.574 INFO:tasks.workunit.client.0.smithi045.stdout:348 2021-07-31T02:01:26.575 INFO:tasks.workunit.client.0.smithi045.stdout:349 2021-07-31T02:01:26.576 INFO:tasks.workunit.client.0.smithi045.stdout:350 2021-07-31T02:01:26.577 INFO:tasks.workunit.client.0.smithi045.stdout:351 2021-07-31T02:01:26.578 INFO:tasks.workunit.client.0.smithi045.stdout:352 2021-07-31T02:01:26.579 INFO:tasks.workunit.client.0.smithi045.stdout:353 2021-07-31T02:01:26.580 INFO:tasks.workunit.client.0.smithi045.stdout:354 2021-07-31T02:01:26.581 INFO:tasks.workunit.client.0.smithi045.stdout:355 2021-07-31T02:01:26.581 INFO:tasks.workunit.client.0.smithi045.stdout:356 2021-07-31T02:01:26.583 INFO:tasks.workunit.client.0.smithi045.stdout:357 2021-07-31T02:01:26.584 INFO:tasks.workunit.client.0.smithi045.stdout:358 2021-07-31T02:01:26.585 INFO:tasks.workunit.client.0.smithi045.stdout:359 2021-07-31T02:01:26.585 INFO:tasks.workunit.client.0.smithi045.stdout:360 2021-07-31T02:01:26.586 INFO:tasks.workunit.client.0.smithi045.stdout:361 2021-07-31T02:01:26.587 INFO:tasks.workunit.client.0.smithi045.stdout:362 2021-07-31T02:01:26.588 INFO:tasks.workunit.client.0.smithi045.stdout:363 2021-07-31T02:01:26.589 INFO:tasks.workunit.client.0.smithi045.stdout:364 2021-07-31T02:01:26.590 INFO:tasks.workunit.client.0.smithi045.stdout:365 2021-07-31T02:01:26.591 INFO:tasks.workunit.client.0.smithi045.stdout:366 2021-07-31T02:01:26.592 INFO:tasks.workunit.client.0.smithi045.stdout:367 2021-07-31T02:01:26.593 INFO:tasks.workunit.client.0.smithi045.stdout:368 2021-07-31T02:01:26.594 INFO:tasks.workunit.client.0.smithi045.stdout:369 2021-07-31T02:01:26.595 INFO:tasks.workunit.client.0.smithi045.stdout:370 2021-07-31T02:01:26.596 INFO:tasks.workunit.client.0.smithi045.stdout:371 2021-07-31T02:01:26.597 INFO:tasks.workunit.client.0.smithi045.stdout:372 2021-07-31T02:01:26.598 INFO:tasks.workunit.client.0.smithi045.stdout:373 2021-07-31T02:01:26.599 INFO:tasks.workunit.client.0.smithi045.stdout:374 2021-07-31T02:01:26.600 INFO:tasks.workunit.client.0.smithi045.stdout:375 2021-07-31T02:01:26.601 INFO:tasks.workunit.client.0.smithi045.stdout:376 2021-07-31T02:01:26.602 INFO:tasks.workunit.client.0.smithi045.stdout:377 2021-07-31T02:01:26.603 INFO:tasks.workunit.client.0.smithi045.stdout:378 2021-07-31T02:01:26.603 INFO:tasks.workunit.client.0.smithi045.stdout:379 2021-07-31T02:01:26.604 INFO:tasks.workunit.client.0.smithi045.stdout:380 2021-07-31T02:01:26.605 INFO:tasks.workunit.client.0.smithi045.stdout:381 2021-07-31T02:01:26.606 INFO:tasks.workunit.client.0.smithi045.stdout:382 2021-07-31T02:01:26.607 INFO:tasks.workunit.client.0.smithi045.stdout:383 2021-07-31T02:01:26.608 INFO:tasks.workunit.client.0.smithi045.stdout:384 2021-07-31T02:01:26.609 INFO:tasks.workunit.client.0.smithi045.stdout:385 2021-07-31T02:01:26.610 INFO:tasks.workunit.client.0.smithi045.stdout:386 2021-07-31T02:01:26.611 INFO:tasks.workunit.client.0.smithi045.stdout:387 2021-07-31T02:01:26.612 INFO:tasks.workunit.client.0.smithi045.stdout:388 2021-07-31T02:01:26.612 INFO:tasks.workunit.client.0.smithi045.stdout:389 2021-07-31T02:01:26.613 INFO:tasks.workunit.client.0.smithi045.stdout:390 2021-07-31T02:01:26.614 INFO:tasks.workunit.client.0.smithi045.stdout:391 2021-07-31T02:01:26.616 INFO:tasks.workunit.client.0.smithi045.stdout:392 2021-07-31T02:01:26.617 INFO:tasks.workunit.client.0.smithi045.stdout:393 2021-07-31T02:01:26.617 INFO:tasks.workunit.client.0.smithi045.stdout:394 2021-07-31T02:01:26.618 INFO:tasks.workunit.client.0.smithi045.stdout:395 2021-07-31T02:01:26.619 INFO:tasks.workunit.client.0.smithi045.stdout:396 2021-07-31T02:01:26.620 INFO:tasks.workunit.client.0.smithi045.stdout:397 2021-07-31T02:01:26.621 INFO:tasks.workunit.client.0.smithi045.stdout:398 2021-07-31T02:01:26.622 INFO:tasks.workunit.client.0.smithi045.stdout:399 2021-07-31T02:01:26.623 INFO:tasks.workunit.client.0.smithi045.stdout:400 2021-07-31T02:01:26.624 INFO:tasks.workunit.client.0.smithi045.stdout:401 2021-07-31T02:01:26.625 INFO:tasks.workunit.client.0.smithi045.stdout:402 2021-07-31T02:01:26.626 INFO:tasks.workunit.client.0.smithi045.stdout:403 2021-07-31T02:01:26.627 INFO:tasks.workunit.client.0.smithi045.stdout:404 2021-07-31T02:01:26.628 INFO:tasks.workunit.client.0.smithi045.stdout:405 2021-07-31T02:01:26.628 INFO:tasks.workunit.client.0.smithi045.stdout:406 2021-07-31T02:01:26.629 INFO:tasks.workunit.client.0.smithi045.stdout:407 2021-07-31T02:01:26.630 INFO:tasks.workunit.client.0.smithi045.stdout:408 2021-07-31T02:01:26.631 INFO:tasks.workunit.client.0.smithi045.stdout:409 2021-07-31T02:01:26.632 INFO:tasks.workunit.client.0.smithi045.stdout:410 2021-07-31T02:01:26.633 INFO:tasks.workunit.client.0.smithi045.stdout:411 2021-07-31T02:01:26.642 INFO:tasks.workunit.client.0.smithi045.stdout:412 2021-07-31T02:01:26.643 INFO:tasks.workunit.client.0.smithi045.stdout:413 2021-07-31T02:01:26.643 INFO:tasks.workunit.client.0.smithi045.stdout:414 2021-07-31T02:01:26.644 INFO:tasks.workunit.client.0.smithi045.stdout:415 2021-07-31T02:01:26.644 INFO:tasks.workunit.client.0.smithi045.stdout:416 2021-07-31T02:01:26.644 INFO:tasks.workunit.client.0.smithi045.stdout:417 2021-07-31T02:01:26.645 INFO:tasks.workunit.client.0.smithi045.stdout:418 2021-07-31T02:01:26.645 INFO:tasks.workunit.client.0.smithi045.stdout:419 2021-07-31T02:01:26.646 INFO:tasks.workunit.client.0.smithi045.stdout:420 2021-07-31T02:01:26.648 INFO:tasks.workunit.client.0.smithi045.stdout:421 2021-07-31T02:01:26.648 INFO:tasks.workunit.client.0.smithi045.stdout:422 2021-07-31T02:01:26.648 INFO:tasks.workunit.client.0.smithi045.stdout:423 2021-07-31T02:01:26.649 INFO:tasks.workunit.client.0.smithi045.stdout:424 2021-07-31T02:01:26.649 INFO:tasks.workunit.client.0.smithi045.stdout:425 2021-07-31T02:01:26.650 INFO:tasks.workunit.client.0.smithi045.stdout:426 2021-07-31T02:01:26.651 INFO:tasks.workunit.client.0.smithi045.stdout:427 2021-07-31T02:01:26.651 INFO:tasks.workunit.client.0.smithi045.stdout:428 2021-07-31T02:01:26.652 INFO:tasks.workunit.client.0.smithi045.stdout:429 2021-07-31T02:01:26.653 INFO:tasks.workunit.client.0.smithi045.stdout:430 2021-07-31T02:01:26.653 INFO:tasks.workunit.client.0.smithi045.stdout:431 2021-07-31T02:01:26.654 INFO:tasks.workunit.client.0.smithi045.stdout:432 2021-07-31T02:01:26.655 INFO:tasks.workunit.client.0.smithi045.stdout:433 2021-07-31T02:01:26.656 INFO:tasks.workunit.client.0.smithi045.stdout:434 2021-07-31T02:01:26.656 INFO:tasks.workunit.client.0.smithi045.stdout:435 2021-07-31T02:01:26.657 INFO:tasks.workunit.client.0.smithi045.stdout:436 2021-07-31T02:01:26.658 INFO:tasks.workunit.client.0.smithi045.stdout:437 2021-07-31T02:01:26.659 INFO:tasks.workunit.client.0.smithi045.stdout:438 2021-07-31T02:01:26.660 INFO:tasks.workunit.client.0.smithi045.stdout:439 2021-07-31T02:01:26.661 INFO:tasks.workunit.client.0.smithi045.stdout:440 2021-07-31T02:01:26.662 INFO:tasks.workunit.client.0.smithi045.stdout:441 2021-07-31T02:01:26.663 INFO:tasks.workunit.client.0.smithi045.stdout:442 2021-07-31T02:01:26.664 INFO:tasks.workunit.client.0.smithi045.stdout:443 2021-07-31T02:01:26.665 INFO:tasks.workunit.client.0.smithi045.stdout:444 2021-07-31T02:01:26.666 INFO:tasks.workunit.client.0.smithi045.stdout:445 2021-07-31T02:01:26.667 INFO:tasks.workunit.client.0.smithi045.stdout:446 2021-07-31T02:01:26.668 INFO:tasks.workunit.client.0.smithi045.stdout:447 2021-07-31T02:01:26.669 INFO:tasks.workunit.client.0.smithi045.stdout:448 2021-07-31T02:01:26.670 INFO:tasks.workunit.client.0.smithi045.stdout:449 2021-07-31T02:01:26.671 INFO:tasks.workunit.client.0.smithi045.stdout:450 2021-07-31T02:01:26.672 INFO:tasks.workunit.client.0.smithi045.stdout:451 2021-07-31T02:01:26.673 INFO:tasks.workunit.client.0.smithi045.stdout:452 2021-07-31T02:01:26.674 INFO:tasks.workunit.client.0.smithi045.stdout:453 2021-07-31T02:01:26.675 INFO:tasks.workunit.client.0.smithi045.stdout:454 2021-07-31T02:01:26.676 INFO:tasks.workunit.client.0.smithi045.stdout:455 2021-07-31T02:01:26.677 INFO:tasks.workunit.client.0.smithi045.stdout:456 2021-07-31T02:01:26.678 INFO:tasks.workunit.client.0.smithi045.stdout:457 2021-07-31T02:01:26.679 INFO:tasks.workunit.client.0.smithi045.stdout:458 2021-07-31T02:01:26.680 INFO:tasks.workunit.client.0.smithi045.stdout:459 2021-07-31T02:01:26.681 INFO:tasks.workunit.client.0.smithi045.stdout:460 2021-07-31T02:01:26.682 INFO:tasks.workunit.client.0.smithi045.stdout:461 2021-07-31T02:01:26.683 INFO:tasks.workunit.client.0.smithi045.stdout:462 2021-07-31T02:01:26.684 INFO:tasks.workunit.client.0.smithi045.stdout:463 2021-07-31T02:01:26.685 INFO:tasks.workunit.client.0.smithi045.stdout:464 2021-07-31T02:01:26.686 INFO:tasks.workunit.client.0.smithi045.stdout:465 2021-07-31T02:01:26.687 INFO:tasks.workunit.client.0.smithi045.stdout:466 2021-07-31T02:01:26.688 INFO:tasks.workunit.client.0.smithi045.stdout:467 2021-07-31T02:01:26.689 INFO:tasks.workunit.client.0.smithi045.stdout:468 2021-07-31T02:01:26.690 INFO:tasks.workunit.client.0.smithi045.stdout:469 2021-07-31T02:01:26.691 INFO:tasks.workunit.client.0.smithi045.stdout:470 2021-07-31T02:01:26.692 INFO:tasks.workunit.client.0.smithi045.stdout:471 2021-07-31T02:01:26.693 INFO:tasks.workunit.client.0.smithi045.stdout:472 2021-07-31T02:01:26.694 INFO:tasks.workunit.client.0.smithi045.stdout:473 2021-07-31T02:01:26.695 INFO:tasks.workunit.client.0.smithi045.stdout:474 2021-07-31T02:01:26.696 INFO:tasks.workunit.client.0.smithi045.stdout:475 2021-07-31T02:01:26.697 INFO:tasks.workunit.client.0.smithi045.stdout:476 2021-07-31T02:01:26.698 INFO:tasks.workunit.client.0.smithi045.stdout:477 2021-07-31T02:01:26.699 INFO:tasks.workunit.client.0.smithi045.stdout:478 2021-07-31T02:01:26.700 INFO:tasks.workunit.client.0.smithi045.stdout:479 2021-07-31T02:01:26.701 INFO:tasks.workunit.client.0.smithi045.stdout:480 2021-07-31T02:01:26.703 INFO:tasks.workunit.client.0.smithi045.stdout:481 2021-07-31T02:01:26.704 INFO:tasks.workunit.client.0.smithi045.stdout:482 2021-07-31T02:01:26.705 INFO:tasks.workunit.client.0.smithi045.stdout:483 2021-07-31T02:01:26.706 INFO:tasks.workunit.client.0.smithi045.stdout:484 2021-07-31T02:01:26.707 INFO:tasks.workunit.client.0.smithi045.stdout:485 2021-07-31T02:01:26.708 INFO:tasks.workunit.client.0.smithi045.stdout:486 2021-07-31T02:01:26.708 INFO:tasks.workunit.client.0.smithi045.stdout:487 2021-07-31T02:01:26.709 INFO:tasks.workunit.client.0.smithi045.stdout:488 2021-07-31T02:01:26.710 INFO:tasks.workunit.client.0.smithi045.stdout:489 2021-07-31T02:01:26.711 INFO:tasks.workunit.client.0.smithi045.stdout:490 2021-07-31T02:01:26.712 INFO:tasks.workunit.client.0.smithi045.stdout:491 2021-07-31T02:01:26.713 INFO:tasks.workunit.client.0.smithi045.stdout:492 2021-07-31T02:01:26.714 INFO:tasks.workunit.client.0.smithi045.stdout:493 2021-07-31T02:01:26.715 INFO:tasks.workunit.client.0.smithi045.stdout:494 2021-07-31T02:01:26.716 INFO:tasks.workunit.client.0.smithi045.stdout:495 2021-07-31T02:01:26.717 INFO:tasks.workunit.client.0.smithi045.stdout:496 2021-07-31T02:01:26.718 INFO:tasks.workunit.client.0.smithi045.stdout:497 2021-07-31T02:01:26.719 INFO:tasks.workunit.client.0.smithi045.stdout:498 2021-07-31T02:01:26.721 INFO:tasks.workunit.client.0.smithi045.stdout:499 2021-07-31T02:01:26.722 INFO:tasks.workunit.client.0.smithi045.stdout:500 2021-07-31T02:01:26.723 INFO:tasks.workunit.client.0.smithi045.stdout:501 2021-07-31T02:01:26.723 INFO:tasks.workunit.client.0.smithi045.stdout:502 2021-07-31T02:01:26.725 INFO:tasks.workunit.client.0.smithi045.stdout:503 2021-07-31T02:01:26.726 INFO:tasks.workunit.client.0.smithi045.stdout:504 2021-07-31T02:01:26.727 INFO:tasks.workunit.client.0.smithi045.stdout:505 2021-07-31T02:01:26.728 INFO:tasks.workunit.client.0.smithi045.stdout:506 2021-07-31T02:01:26.729 INFO:tasks.workunit.client.0.smithi045.stdout:507 2021-07-31T02:01:26.730 INFO:tasks.workunit.client.0.smithi045.stdout:508 2021-07-31T02:01:26.731 INFO:tasks.workunit.client.0.smithi045.stdout:509 2021-07-31T02:01:26.732 INFO:tasks.workunit.client.0.smithi045.stdout:510 2021-07-31T02:01:26.733 INFO:tasks.workunit.client.0.smithi045.stdout:511 2021-07-31T02:01:26.734 INFO:tasks.workunit.client.0.smithi045.stdout:512 2021-07-31T02:01:26.735 INFO:tasks.workunit.client.0.smithi045.stdout:513 2021-07-31T02:01:26.736 INFO:tasks.workunit.client.0.smithi045.stdout:514 2021-07-31T02:01:26.737 INFO:tasks.workunit.client.0.smithi045.stdout:515 2021-07-31T02:01:26.737 INFO:tasks.workunit.client.0.smithi045.stdout:516 2021-07-31T02:01:26.739 INFO:tasks.workunit.client.0.smithi045.stdout:517 2021-07-31T02:01:26.740 INFO:tasks.workunit.client.0.smithi045.stdout:518 2021-07-31T02:01:26.741 INFO:tasks.workunit.client.0.smithi045.stdout:519 2021-07-31T02:01:26.742 INFO:tasks.workunit.client.0.smithi045.stdout:520 2021-07-31T02:01:26.743 INFO:tasks.workunit.client.0.smithi045.stdout:521 2021-07-31T02:01:26.744 INFO:tasks.workunit.client.0.smithi045.stdout:522 2021-07-31T02:01:26.745 INFO:tasks.workunit.client.0.smithi045.stdout:523 2021-07-31T02:01:26.746 INFO:tasks.workunit.client.0.smithi045.stdout:524 2021-07-31T02:01:26.747 INFO:tasks.workunit.client.0.smithi045.stdout:525 2021-07-31T02:01:26.748 INFO:tasks.workunit.client.0.smithi045.stdout:526 2021-07-31T02:01:26.749 INFO:tasks.workunit.client.0.smithi045.stdout:527 2021-07-31T02:01:26.750 INFO:tasks.workunit.client.0.smithi045.stdout:528 2021-07-31T02:01:26.751 INFO:tasks.workunit.client.0.smithi045.stdout:529 2021-07-31T02:01:26.752 INFO:tasks.workunit.client.0.smithi045.stdout:530 2021-07-31T02:01:26.753 INFO:tasks.workunit.client.0.smithi045.stdout:531 2021-07-31T02:01:26.754 INFO:tasks.workunit.client.0.smithi045.stdout:532 2021-07-31T02:01:26.755 INFO:tasks.workunit.client.0.smithi045.stdout:533 2021-07-31T02:01:26.756 INFO:tasks.workunit.client.0.smithi045.stdout:534 2021-07-31T02:01:26.757 INFO:tasks.workunit.client.0.smithi045.stdout:535 2021-07-31T02:01:26.758 INFO:tasks.workunit.client.0.smithi045.stdout:536 2021-07-31T02:01:26.759 INFO:tasks.workunit.client.0.smithi045.stdout:537 2021-07-31T02:01:26.759 INFO:tasks.workunit.client.0.smithi045.stdout:538 2021-07-31T02:01:26.760 INFO:tasks.workunit.client.0.smithi045.stdout:539 2021-07-31T02:01:26.762 INFO:tasks.workunit.client.0.smithi045.stdout:540 2021-07-31T02:01:26.762 INFO:tasks.workunit.client.0.smithi045.stdout:541 2021-07-31T02:01:26.764 INFO:tasks.workunit.client.0.smithi045.stdout:542 2021-07-31T02:01:26.765 INFO:tasks.workunit.client.0.smithi045.stdout:543 2021-07-31T02:01:26.766 INFO:tasks.workunit.client.0.smithi045.stdout:544 2021-07-31T02:01:26.767 INFO:tasks.workunit.client.0.smithi045.stdout:545 2021-07-31T02:01:26.768 INFO:tasks.workunit.client.0.smithi045.stdout:546 2021-07-31T02:01:26.769 INFO:tasks.workunit.client.0.smithi045.stdout:547 2021-07-31T02:01:26.770 INFO:tasks.workunit.client.0.smithi045.stdout:548 2021-07-31T02:01:26.771 INFO:tasks.workunit.client.0.smithi045.stdout:549 2021-07-31T02:01:26.772 INFO:tasks.workunit.client.0.smithi045.stdout:550 2021-07-31T02:01:26.773 INFO:tasks.workunit.client.0.smithi045.stdout:551 2021-07-31T02:01:26.774 INFO:tasks.workunit.client.0.smithi045.stdout:552 2021-07-31T02:01:26.775 INFO:tasks.workunit.client.0.smithi045.stdout:553 2021-07-31T02:01:26.776 INFO:tasks.workunit.client.0.smithi045.stdout:554 2021-07-31T02:01:26.777 INFO:tasks.workunit.client.0.smithi045.stdout:555 2021-07-31T02:01:26.778 INFO:tasks.workunit.client.0.smithi045.stdout:556 2021-07-31T02:01:26.779 INFO:tasks.workunit.client.0.smithi045.stdout:557 2021-07-31T02:01:26.780 INFO:tasks.workunit.client.0.smithi045.stdout:558 2021-07-31T02:01:26.781 INFO:tasks.workunit.client.0.smithi045.stdout:559 2021-07-31T02:01:26.782 INFO:tasks.workunit.client.0.smithi045.stdout:560 2021-07-31T02:01:26.784 INFO:tasks.workunit.client.0.smithi045.stdout:561 2021-07-31T02:01:26.785 INFO:tasks.workunit.client.0.smithi045.stdout:562 2021-07-31T02:01:26.786 INFO:tasks.workunit.client.0.smithi045.stdout:563 2021-07-31T02:01:26.787 INFO:tasks.workunit.client.0.smithi045.stdout:564 2021-07-31T02:01:26.788 INFO:tasks.workunit.client.0.smithi045.stdout:565 2021-07-31T02:01:26.789 INFO:tasks.workunit.client.0.smithi045.stdout:566 2021-07-31T02:01:26.790 INFO:tasks.workunit.client.0.smithi045.stdout:567 2021-07-31T02:01:26.791 INFO:tasks.workunit.client.0.smithi045.stdout:568 2021-07-31T02:01:26.793 INFO:tasks.workunit.client.0.smithi045.stdout:569 2021-07-31T02:01:26.793 INFO:tasks.workunit.client.0.smithi045.stdout:570 2021-07-31T02:01:26.794 INFO:tasks.workunit.client.0.smithi045.stdout:571 2021-07-31T02:01:26.795 INFO:tasks.workunit.client.0.smithi045.stdout:572 2021-07-31T02:01:26.796 INFO:tasks.workunit.client.0.smithi045.stdout:573 2021-07-31T02:01:26.796 INFO:tasks.workunit.client.0.smithi045.stdout:574 2021-07-31T02:01:26.798 INFO:tasks.workunit.client.0.smithi045.stdout:575 2021-07-31T02:01:26.799 INFO:tasks.workunit.client.0.smithi045.stdout:576 2021-07-31T02:01:26.799 INFO:tasks.workunit.client.0.smithi045.stdout:577 2021-07-31T02:01:26.801 INFO:tasks.workunit.client.0.smithi045.stdout:578 2021-07-31T02:01:26.802 INFO:tasks.workunit.client.0.smithi045.stdout:579 2021-07-31T02:01:26.803 INFO:tasks.workunit.client.0.smithi045.stdout:580 2021-07-31T02:01:26.804 INFO:tasks.workunit.client.0.smithi045.stdout:581 2021-07-31T02:01:26.805 INFO:tasks.workunit.client.0.smithi045.stdout:582 2021-07-31T02:01:26.806 INFO:tasks.workunit.client.0.smithi045.stdout:583 2021-07-31T02:01:26.807 INFO:tasks.workunit.client.0.smithi045.stdout:584 2021-07-31T02:01:26.808 INFO:tasks.workunit.client.0.smithi045.stdout:585 2021-07-31T02:01:26.809 INFO:tasks.workunit.client.0.smithi045.stdout:586 2021-07-31T02:01:26.810 INFO:tasks.workunit.client.0.smithi045.stdout:587 2021-07-31T02:01:26.811 INFO:tasks.workunit.client.0.smithi045.stdout:588 2021-07-31T02:01:26.812 INFO:tasks.workunit.client.0.smithi045.stdout:589 2021-07-31T02:01:26.813 INFO:tasks.workunit.client.0.smithi045.stdout:590 2021-07-31T02:01:26.814 INFO:tasks.workunit.client.0.smithi045.stdout:591 2021-07-31T02:01:26.815 INFO:tasks.workunit.client.0.smithi045.stdout:592 2021-07-31T02:01:26.816 INFO:tasks.workunit.client.0.smithi045.stdout:593 2021-07-31T02:01:26.817 INFO:tasks.workunit.client.0.smithi045.stdout:594 2021-07-31T02:01:26.818 INFO:tasks.workunit.client.0.smithi045.stdout:595 2021-07-31T02:01:26.819 INFO:tasks.workunit.client.0.smithi045.stdout:596 2021-07-31T02:01:26.820 INFO:tasks.workunit.client.0.smithi045.stdout:597 2021-07-31T02:01:26.821 INFO:tasks.workunit.client.0.smithi045.stdout:598 2021-07-31T02:01:26.822 INFO:tasks.workunit.client.0.smithi045.stdout:599 2021-07-31T02:01:26.823 INFO:tasks.workunit.client.0.smithi045.stdout:600 2021-07-31T02:01:26.824 INFO:tasks.workunit.client.0.smithi045.stdout:601 2021-07-31T02:01:26.825 INFO:tasks.workunit.client.0.smithi045.stdout:602 2021-07-31T02:01:26.826 INFO:tasks.workunit.client.0.smithi045.stdout:603 2021-07-31T02:01:26.827 INFO:tasks.workunit.client.0.smithi045.stdout:604 2021-07-31T02:01:26.828 INFO:tasks.workunit.client.0.smithi045.stdout:605 2021-07-31T02:01:26.829 INFO:tasks.workunit.client.0.smithi045.stdout:606 2021-07-31T02:01:26.830 INFO:tasks.workunit.client.0.smithi045.stdout:607 2021-07-31T02:01:26.831 INFO:tasks.workunit.client.0.smithi045.stdout:608 2021-07-31T02:01:26.833 INFO:tasks.workunit.client.0.smithi045.stdout:609 2021-07-31T02:01:26.834 INFO:tasks.workunit.client.0.smithi045.stdout:610 2021-07-31T02:01:26.835 INFO:tasks.workunit.client.0.smithi045.stdout:611 2021-07-31T02:01:26.836 INFO:tasks.workunit.client.0.smithi045.stdout:612 2021-07-31T02:01:26.837 INFO:tasks.workunit.client.0.smithi045.stdout:613 2021-07-31T02:01:26.838 INFO:tasks.workunit.client.0.smithi045.stdout:614 2021-07-31T02:01:26.839 INFO:tasks.workunit.client.0.smithi045.stdout:615 2021-07-31T02:01:26.841 INFO:tasks.workunit.client.0.smithi045.stdout:616 2021-07-31T02:01:26.841 INFO:tasks.workunit.client.0.smithi045.stdout:617 2021-07-31T02:01:26.842 INFO:tasks.workunit.client.0.smithi045.stdout:618 2021-07-31T02:01:26.844 INFO:tasks.workunit.client.0.smithi045.stdout:619 2021-07-31T02:01:26.844 INFO:tasks.workunit.client.0.smithi045.stdout:620 2021-07-31T02:01:26.846 INFO:tasks.workunit.client.0.smithi045.stdout:621 2021-07-31T02:01:26.847 INFO:tasks.workunit.client.0.smithi045.stdout:622 2021-07-31T02:01:26.848 INFO:tasks.workunit.client.0.smithi045.stdout:623 2021-07-31T02:01:26.849 INFO:tasks.workunit.client.0.smithi045.stdout:624 2021-07-31T02:01:26.850 INFO:tasks.workunit.client.0.smithi045.stdout:625 2021-07-31T02:01:26.851 INFO:tasks.workunit.client.0.smithi045.stdout:626 2021-07-31T02:01:26.852 INFO:tasks.workunit.client.0.smithi045.stdout:627 2021-07-31T02:01:26.853 INFO:tasks.workunit.client.0.smithi045.stdout:628 2021-07-31T02:01:26.854 INFO:tasks.workunit.client.0.smithi045.stdout:629 2021-07-31T02:01:26.855 INFO:tasks.workunit.client.0.smithi045.stdout:630 2021-07-31T02:01:26.856 INFO:tasks.workunit.client.0.smithi045.stdout:631 2021-07-31T02:01:26.858 INFO:tasks.workunit.client.0.smithi045.stdout:632 2021-07-31T02:01:26.859 INFO:tasks.workunit.client.0.smithi045.stdout:633 2021-07-31T02:01:26.860 INFO:tasks.workunit.client.0.smithi045.stdout:634 2021-07-31T02:01:26.861 INFO:tasks.workunit.client.0.smithi045.stdout:635 2021-07-31T02:01:26.862 INFO:tasks.workunit.client.0.smithi045.stdout:636 2021-07-31T02:01:26.863 INFO:tasks.workunit.client.0.smithi045.stdout:637 2021-07-31T02:01:26.864 INFO:tasks.workunit.client.0.smithi045.stdout:638 2021-07-31T02:01:26.865 INFO:tasks.workunit.client.0.smithi045.stdout:639 2021-07-31T02:01:26.866 INFO:tasks.workunit.client.0.smithi045.stdout:640 2021-07-31T02:01:26.867 INFO:tasks.workunit.client.0.smithi045.stdout:641 2021-07-31T02:01:26.868 INFO:tasks.workunit.client.0.smithi045.stdout:642 2021-07-31T02:01:26.869 INFO:tasks.workunit.client.0.smithi045.stdout:643 2021-07-31T02:01:26.870 INFO:tasks.workunit.client.0.smithi045.stdout:644 2021-07-31T02:01:26.871 INFO:tasks.workunit.client.0.smithi045.stdout:645 2021-07-31T02:01:26.872 INFO:tasks.workunit.client.0.smithi045.stdout:646 2021-07-31T02:01:26.873 INFO:tasks.workunit.client.0.smithi045.stdout:647 2021-07-31T02:01:26.874 INFO:tasks.workunit.client.0.smithi045.stdout:648 2021-07-31T02:01:26.875 INFO:tasks.workunit.client.0.smithi045.stdout:649 2021-07-31T02:01:26.876 INFO:tasks.workunit.client.0.smithi045.stdout:650 2021-07-31T02:01:26.877 INFO:tasks.workunit.client.0.smithi045.stdout:651 2021-07-31T02:01:26.878 INFO:tasks.workunit.client.0.smithi045.stdout:652 2021-07-31T02:01:26.880 INFO:tasks.workunit.client.0.smithi045.stdout:653 2021-07-31T02:01:26.881 INFO:tasks.workunit.client.0.smithi045.stdout:654 2021-07-31T02:01:26.882 INFO:tasks.workunit.client.0.smithi045.stdout:655 2021-07-31T02:01:26.884 INFO:tasks.workunit.client.0.smithi045.stdout:656 2021-07-31T02:01:26.885 INFO:tasks.workunit.client.0.smithi045.stdout:657 2021-07-31T02:01:26.886 INFO:tasks.workunit.client.0.smithi045.stdout:658 2021-07-31T02:01:26.887 INFO:tasks.workunit.client.0.smithi045.stdout:659 2021-07-31T02:01:26.889 INFO:tasks.workunit.client.0.smithi045.stdout:660 2021-07-31T02:01:26.890 INFO:tasks.workunit.client.0.smithi045.stdout:661 2021-07-31T02:01:26.891 INFO:tasks.workunit.client.0.smithi045.stdout:662 2021-07-31T02:01:26.892 INFO:tasks.workunit.client.0.smithi045.stdout:663 2021-07-31T02:01:26.893 INFO:tasks.workunit.client.0.smithi045.stdout:664 2021-07-31T02:01:26.894 INFO:tasks.workunit.client.0.smithi045.stdout:665 2021-07-31T02:01:26.896 INFO:tasks.workunit.client.0.smithi045.stdout:666 2021-07-31T02:01:26.897 INFO:tasks.workunit.client.0.smithi045.stdout:667 2021-07-31T02:01:26.898 INFO:tasks.workunit.client.0.smithi045.stdout:668 2021-07-31T02:01:26.900 INFO:tasks.workunit.client.0.smithi045.stdout:669 2021-07-31T02:01:26.901 INFO:tasks.workunit.client.0.smithi045.stdout:670 2021-07-31T02:01:26.902 INFO:tasks.workunit.client.0.smithi045.stdout:671 2021-07-31T02:01:26.903 INFO:tasks.workunit.client.0.smithi045.stdout:672 2021-07-31T02:01:26.905 INFO:tasks.workunit.client.0.smithi045.stdout:673 2021-07-31T02:01:26.906 INFO:tasks.workunit.client.0.smithi045.stdout:674 2021-07-31T02:01:26.907 INFO:tasks.workunit.client.0.smithi045.stdout:675 2021-07-31T02:01:26.908 INFO:tasks.workunit.client.0.smithi045.stdout:676 2021-07-31T02:01:26.909 INFO:tasks.workunit.client.0.smithi045.stdout:677 2021-07-31T02:01:26.910 INFO:tasks.workunit.client.0.smithi045.stdout:678 2021-07-31T02:01:26.911 INFO:tasks.workunit.client.0.smithi045.stdout:679 2021-07-31T02:01:26.912 INFO:tasks.workunit.client.0.smithi045.stdout:680 2021-07-31T02:01:26.913 INFO:tasks.workunit.client.0.smithi045.stdout:681 2021-07-31T02:01:26.914 INFO:tasks.workunit.client.0.smithi045.stdout:682 2021-07-31T02:01:26.915 INFO:tasks.workunit.client.0.smithi045.stdout:683 2021-07-31T02:01:26.917 INFO:tasks.workunit.client.0.smithi045.stdout:684 2021-07-31T02:01:26.918 INFO:tasks.workunit.client.0.smithi045.stdout:685 2021-07-31T02:01:26.919 INFO:tasks.workunit.client.0.smithi045.stdout:686 2021-07-31T02:01:26.920 INFO:tasks.workunit.client.0.smithi045.stdout:687 2021-07-31T02:01:26.921 INFO:tasks.workunit.client.0.smithi045.stdout:688 2021-07-31T02:01:26.922 INFO:tasks.workunit.client.0.smithi045.stdout:689 2021-07-31T02:01:26.923 INFO:tasks.workunit.client.0.smithi045.stdout:690 2021-07-31T02:01:26.924 INFO:tasks.workunit.client.0.smithi045.stdout:691 2021-07-31T02:01:26.925 INFO:tasks.workunit.client.0.smithi045.stdout:692 2021-07-31T02:01:26.926 INFO:tasks.workunit.client.0.smithi045.stdout:693 2021-07-31T02:01:26.927 INFO:tasks.workunit.client.0.smithi045.stdout:694 2021-07-31T02:01:26.928 INFO:tasks.workunit.client.0.smithi045.stdout:695 2021-07-31T02:01:26.930 INFO:tasks.workunit.client.0.smithi045.stdout:696 2021-07-31T02:01:26.931 INFO:tasks.workunit.client.0.smithi045.stdout:697 2021-07-31T02:01:26.932 INFO:tasks.workunit.client.0.smithi045.stdout:698 2021-07-31T02:01:26.933 INFO:tasks.workunit.client.0.smithi045.stdout:699 2021-07-31T02:01:26.934 INFO:tasks.workunit.client.0.smithi045.stdout:700 2021-07-31T02:01:26.936 INFO:tasks.workunit.client.0.smithi045.stdout:701 2021-07-31T02:01:26.937 INFO:tasks.workunit.client.0.smithi045.stdout:702 2021-07-31T02:01:26.938 INFO:tasks.workunit.client.0.smithi045.stdout:703 2021-07-31T02:01:26.939 INFO:tasks.workunit.client.0.smithi045.stdout:704 2021-07-31T02:01:26.940 INFO:tasks.workunit.client.0.smithi045.stdout:705 2021-07-31T02:01:26.941 INFO:tasks.workunit.client.0.smithi045.stdout:706 2021-07-31T02:01:26.942 INFO:tasks.workunit.client.0.smithi045.stdout:707 2021-07-31T02:01:26.943 INFO:tasks.workunit.client.0.smithi045.stdout:708 2021-07-31T02:01:26.944 INFO:tasks.workunit.client.0.smithi045.stdout:709 2021-07-31T02:01:26.945 INFO:tasks.workunit.client.0.smithi045.stdout:710 2021-07-31T02:01:26.946 INFO:tasks.workunit.client.0.smithi045.stdout:711 2021-07-31T02:01:26.947 INFO:tasks.workunit.client.0.smithi045.stdout:712 2021-07-31T02:01:26.948 INFO:tasks.workunit.client.0.smithi045.stdout:713 2021-07-31T02:01:26.949 INFO:tasks.workunit.client.0.smithi045.stdout:714 2021-07-31T02:01:26.950 INFO:tasks.workunit.client.0.smithi045.stdout:715 2021-07-31T02:01:26.951 INFO:tasks.workunit.client.0.smithi045.stdout:716 2021-07-31T02:01:26.952 INFO:tasks.workunit.client.0.smithi045.stdout:717 2021-07-31T02:01:26.953 INFO:tasks.workunit.client.0.smithi045.stdout:718 2021-07-31T02:01:26.954 INFO:tasks.workunit.client.0.smithi045.stdout:719 2021-07-31T02:01:26.956 INFO:tasks.workunit.client.0.smithi045.stdout:720 2021-07-31T02:01:26.957 INFO:tasks.workunit.client.0.smithi045.stdout:721 2021-07-31T02:01:26.958 INFO:tasks.workunit.client.0.smithi045.stdout:722 2021-07-31T02:01:26.959 INFO:tasks.workunit.client.0.smithi045.stdout:723 2021-07-31T02:01:26.960 INFO:tasks.workunit.client.0.smithi045.stdout:724 2021-07-31T02:01:26.961 INFO:tasks.workunit.client.0.smithi045.stdout:725 2021-07-31T02:01:26.962 INFO:tasks.workunit.client.0.smithi045.stdout:726 2021-07-31T02:01:26.963 INFO:tasks.workunit.client.0.smithi045.stdout:727 2021-07-31T02:01:26.965 INFO:tasks.workunit.client.0.smithi045.stdout:728 2021-07-31T02:01:26.966 INFO:tasks.workunit.client.0.smithi045.stdout:729 2021-07-31T02:01:26.967 INFO:tasks.workunit.client.0.smithi045.stdout:730 2021-07-31T02:01:26.968 INFO:tasks.workunit.client.0.smithi045.stdout:731 2021-07-31T02:01:26.969 INFO:tasks.workunit.client.0.smithi045.stdout:732 2021-07-31T02:01:26.970 INFO:tasks.workunit.client.0.smithi045.stdout:733 2021-07-31T02:01:26.971 INFO:tasks.workunit.client.0.smithi045.stdout:734 2021-07-31T02:01:26.972 INFO:tasks.workunit.client.0.smithi045.stdout:735 2021-07-31T02:01:26.973 INFO:tasks.workunit.client.0.smithi045.stdout:736 2021-07-31T02:01:26.974 INFO:tasks.workunit.client.0.smithi045.stdout:737 2021-07-31T02:01:26.975 INFO:tasks.workunit.client.0.smithi045.stdout:738 2021-07-31T02:01:26.976 INFO:tasks.workunit.client.0.smithi045.stdout:739 2021-07-31T02:01:26.977 INFO:tasks.workunit.client.0.smithi045.stdout:740 2021-07-31T02:01:26.978 INFO:tasks.workunit.client.0.smithi045.stdout:741 2021-07-31T02:01:26.979 INFO:tasks.workunit.client.0.smithi045.stdout:742 2021-07-31T02:01:26.980 INFO:tasks.workunit.client.0.smithi045.stdout:743 2021-07-31T02:01:26.981 INFO:tasks.workunit.client.0.smithi045.stdout:744 2021-07-31T02:01:26.983 INFO:tasks.workunit.client.0.smithi045.stdout:745 2021-07-31T02:01:26.984 INFO:tasks.workunit.client.0.smithi045.stdout:746 2021-07-31T02:01:26.985 INFO:tasks.workunit.client.0.smithi045.stdout:747 2021-07-31T02:01:26.986 INFO:tasks.workunit.client.0.smithi045.stdout:748 2021-07-31T02:01:26.992 INFO:tasks.workunit.client.0.smithi045.stdout:749 2021-07-31T02:01:26.993 INFO:tasks.workunit.client.0.smithi045.stdout:750 2021-07-31T02:01:26.993 INFO:tasks.workunit.client.0.smithi045.stdout:751 2021-07-31T02:01:26.993 INFO:tasks.workunit.client.0.smithi045.stdout:752 2021-07-31T02:01:26.993 INFO:tasks.workunit.client.0.smithi045.stdout:753 2021-07-31T02:01:26.994 INFO:tasks.workunit.client.0.smithi045.stdout:754 2021-07-31T02:01:26.995 INFO:tasks.workunit.client.0.smithi045.stdout:755 2021-07-31T02:01:26.996 INFO:tasks.workunit.client.0.smithi045.stdout:756 2021-07-31T02:01:26.998 INFO:tasks.workunit.client.0.smithi045.stdout:757 2021-07-31T02:01:26.998 INFO:tasks.workunit.client.0.smithi045.stdout:758 2021-07-31T02:01:26.999 INFO:tasks.workunit.client.0.smithi045.stdout:759 2021-07-31T02:01:27.000 INFO:tasks.workunit.client.0.smithi045.stdout:760 2021-07-31T02:01:27.001 INFO:tasks.workunit.client.0.smithi045.stdout:761 2021-07-31T02:01:27.003 INFO:tasks.workunit.client.0.smithi045.stdout:762 2021-07-31T02:01:27.003 INFO:tasks.workunit.client.0.smithi045.stdout:763 2021-07-31T02:01:27.005 INFO:tasks.workunit.client.0.smithi045.stdout:764 2021-07-31T02:01:27.006 INFO:tasks.workunit.client.0.smithi045.stdout:765 2021-07-31T02:01:27.007 INFO:tasks.workunit.client.0.smithi045.stdout:766 2021-07-31T02:01:27.008 INFO:tasks.workunit.client.0.smithi045.stdout:767 2021-07-31T02:01:27.009 INFO:tasks.workunit.client.0.smithi045.stdout:768 2021-07-31T02:01:27.010 INFO:tasks.workunit.client.0.smithi045.stdout:769 2021-07-31T02:01:27.011 INFO:tasks.workunit.client.0.smithi045.stdout:770 2021-07-31T02:01:27.012 INFO:tasks.workunit.client.0.smithi045.stdout:771 2021-07-31T02:01:27.014 INFO:tasks.workunit.client.0.smithi045.stdout:772 2021-07-31T02:01:27.015 INFO:tasks.workunit.client.0.smithi045.stdout:773 2021-07-31T02:01:27.016 INFO:tasks.workunit.client.0.smithi045.stdout:774 2021-07-31T02:01:27.017 INFO:tasks.workunit.client.0.smithi045.stdout:775 2021-07-31T02:01:27.018 INFO:tasks.workunit.client.0.smithi045.stdout:776 2021-07-31T02:01:27.019 INFO:tasks.workunit.client.0.smithi045.stdout:777 2021-07-31T02:01:27.020 INFO:tasks.workunit.client.0.smithi045.stdout:778 2021-07-31T02:01:27.021 INFO:tasks.workunit.client.0.smithi045.stdout:779 2021-07-31T02:01:27.022 INFO:tasks.workunit.client.0.smithi045.stdout:780 2021-07-31T02:01:27.023 INFO:tasks.workunit.client.0.smithi045.stdout:781 2021-07-31T02:01:27.025 INFO:tasks.workunit.client.0.smithi045.stdout:782 2021-07-31T02:01:27.026 INFO:tasks.workunit.client.0.smithi045.stdout:783 2021-07-31T02:01:27.027 INFO:tasks.workunit.client.0.smithi045.stdout:784 2021-07-31T02:01:27.028 INFO:tasks.workunit.client.0.smithi045.stdout:785 2021-07-31T02:01:27.029 INFO:tasks.workunit.client.0.smithi045.stdout:786 2021-07-31T02:01:27.030 INFO:tasks.workunit.client.0.smithi045.stdout:787 2021-07-31T02:01:27.031 INFO:tasks.workunit.client.0.smithi045.stdout:788 2021-07-31T02:01:27.032 INFO:tasks.workunit.client.0.smithi045.stdout:789 2021-07-31T02:01:27.033 INFO:tasks.workunit.client.0.smithi045.stdout:790 2021-07-31T02:01:27.035 INFO:tasks.workunit.client.0.smithi045.stdout:791 2021-07-31T02:01:27.036 INFO:tasks.workunit.client.0.smithi045.stdout:792 2021-07-31T02:01:27.037 INFO:tasks.workunit.client.0.smithi045.stdout:793 2021-07-31T02:01:27.038 INFO:tasks.workunit.client.0.smithi045.stdout:794 2021-07-31T02:01:27.039 INFO:tasks.workunit.client.0.smithi045.stdout:795 2021-07-31T02:01:27.040 INFO:tasks.workunit.client.0.smithi045.stdout:796 2021-07-31T02:01:27.041 INFO:tasks.workunit.client.0.smithi045.stdout:797 2021-07-31T02:01:27.042 INFO:tasks.workunit.client.0.smithi045.stdout:798 2021-07-31T02:01:27.043 INFO:tasks.workunit.client.0.smithi045.stdout:799 2021-07-31T02:01:27.044 INFO:tasks.workunit.client.0.smithi045.stdout:800 2021-07-31T02:01:27.046 INFO:tasks.workunit.client.0.smithi045.stdout:801 2021-07-31T02:01:27.047 INFO:tasks.workunit.client.0.smithi045.stdout:802 2021-07-31T02:01:27.048 INFO:tasks.workunit.client.0.smithi045.stdout:803 2021-07-31T02:01:27.049 INFO:tasks.workunit.client.0.smithi045.stdout:804 2021-07-31T02:01:27.050 INFO:tasks.workunit.client.0.smithi045.stdout:805 2021-07-31T02:01:27.051 INFO:tasks.workunit.client.0.smithi045.stdout:806 2021-07-31T02:01:27.052 INFO:tasks.workunit.client.0.smithi045.stdout:807 2021-07-31T02:01:27.053 INFO:tasks.workunit.client.0.smithi045.stdout:808 2021-07-31T02:01:27.054 INFO:tasks.workunit.client.0.smithi045.stdout:809 2021-07-31T02:01:27.055 INFO:tasks.workunit.client.0.smithi045.stdout:810 2021-07-31T02:01:27.056 INFO:tasks.workunit.client.0.smithi045.stdout:811 2021-07-31T02:01:27.058 INFO:tasks.workunit.client.0.smithi045.stdout:812 2021-07-31T02:01:27.059 INFO:tasks.workunit.client.0.smithi045.stdout:813 2021-07-31T02:01:27.060 INFO:tasks.workunit.client.0.smithi045.stdout:814 2021-07-31T02:01:27.062 INFO:tasks.workunit.client.0.smithi045.stdout:815 2021-07-31T02:01:27.063 INFO:tasks.workunit.client.0.smithi045.stdout:816 2021-07-31T02:01:27.064 INFO:tasks.workunit.client.0.smithi045.stdout:817 2021-07-31T02:01:27.065 INFO:tasks.workunit.client.0.smithi045.stdout:818 2021-07-31T02:01:27.066 INFO:tasks.workunit.client.0.smithi045.stdout:819 2021-07-31T02:01:27.068 INFO:tasks.workunit.client.0.smithi045.stdout:820 2021-07-31T02:01:27.069 INFO:tasks.workunit.client.0.smithi045.stdout:821 2021-07-31T02:01:27.070 INFO:tasks.workunit.client.0.smithi045.stdout:822 2021-07-31T02:01:27.071 INFO:tasks.workunit.client.0.smithi045.stdout:823 2021-07-31T02:01:27.072 INFO:tasks.workunit.client.0.smithi045.stdout:824 2021-07-31T02:01:27.073 INFO:tasks.workunit.client.0.smithi045.stdout:825 2021-07-31T02:01:27.074 INFO:tasks.workunit.client.0.smithi045.stdout:826 2021-07-31T02:01:27.075 INFO:tasks.workunit.client.0.smithi045.stdout:827 2021-07-31T02:01:27.076 INFO:tasks.workunit.client.0.smithi045.stdout:828 2021-07-31T02:01:27.078 INFO:tasks.workunit.client.0.smithi045.stdout:829 2021-07-31T02:01:27.079 INFO:tasks.workunit.client.0.smithi045.stdout:830 2021-07-31T02:01:27.080 INFO:tasks.workunit.client.0.smithi045.stdout:831 2021-07-31T02:01:27.081 INFO:tasks.workunit.client.0.smithi045.stdout:832 2021-07-31T02:01:27.082 INFO:tasks.workunit.client.0.smithi045.stdout:833 2021-07-31T02:01:27.083 INFO:tasks.workunit.client.0.smithi045.stdout:834 2021-07-31T02:01:27.084 INFO:tasks.workunit.client.0.smithi045.stdout:835 2021-07-31T02:01:27.086 INFO:tasks.workunit.client.0.smithi045.stdout:836 2021-07-31T02:01:27.087 INFO:tasks.workunit.client.0.smithi045.stdout:837 2021-07-31T02:01:27.088 INFO:tasks.workunit.client.0.smithi045.stdout:838 2021-07-31T02:01:27.090 INFO:tasks.workunit.client.0.smithi045.stdout:839 2021-07-31T02:01:27.091 INFO:tasks.workunit.client.0.smithi045.stdout:840 2021-07-31T02:01:27.092 INFO:tasks.workunit.client.0.smithi045.stdout:841 2021-07-31T02:01:27.093 INFO:tasks.workunit.client.0.smithi045.stdout:842 2021-07-31T02:01:27.094 INFO:tasks.workunit.client.0.smithi045.stdout:843 2021-07-31T02:01:27.096 INFO:tasks.workunit.client.0.smithi045.stdout:844 2021-07-31T02:01:27.097 INFO:tasks.workunit.client.0.smithi045.stdout:845 2021-07-31T02:01:27.099 INFO:tasks.workunit.client.0.smithi045.stdout:846 2021-07-31T02:01:27.100 INFO:tasks.workunit.client.0.smithi045.stdout:847 2021-07-31T02:01:27.101 INFO:tasks.workunit.client.0.smithi045.stdout:848 2021-07-31T02:01:27.102 INFO:tasks.workunit.client.0.smithi045.stdout:849 2021-07-31T02:01:27.104 INFO:tasks.workunit.client.0.smithi045.stdout:850 2021-07-31T02:01:27.105 INFO:tasks.workunit.client.0.smithi045.stdout:851 2021-07-31T02:01:27.106 INFO:tasks.workunit.client.0.smithi045.stdout:852 2021-07-31T02:01:27.107 INFO:tasks.workunit.client.0.smithi045.stdout:853 2021-07-31T02:01:27.109 INFO:tasks.workunit.client.0.smithi045.stdout:854 2021-07-31T02:01:27.110 INFO:tasks.workunit.client.0.smithi045.stdout:855 2021-07-31T02:01:27.111 INFO:tasks.workunit.client.0.smithi045.stdout:856 2021-07-31T02:01:27.112 INFO:tasks.workunit.client.0.smithi045.stdout:857 2021-07-31T02:01:27.113 INFO:tasks.workunit.client.0.smithi045.stdout:858 2021-07-31T02:01:27.114 INFO:tasks.workunit.client.0.smithi045.stdout:859 2021-07-31T02:01:27.115 INFO:tasks.workunit.client.0.smithi045.stdout:860 2021-07-31T02:01:27.116 INFO:tasks.workunit.client.0.smithi045.stdout:861 2021-07-31T02:01:27.118 INFO:tasks.workunit.client.0.smithi045.stdout:862 2021-07-31T02:01:27.119 INFO:tasks.workunit.client.0.smithi045.stdout:863 2021-07-31T02:01:27.120 INFO:tasks.workunit.client.0.smithi045.stdout:864 2021-07-31T02:01:27.121 INFO:tasks.workunit.client.0.smithi045.stdout:865 2021-07-31T02:01:27.122 INFO:tasks.workunit.client.0.smithi045.stdout:866 2021-07-31T02:01:27.124 INFO:tasks.workunit.client.0.smithi045.stdout:867 2021-07-31T02:01:27.125 INFO:tasks.workunit.client.0.smithi045.stdout:868 2021-07-31T02:01:27.126 INFO:tasks.workunit.client.0.smithi045.stdout:869 2021-07-31T02:01:27.128 INFO:tasks.workunit.client.0.smithi045.stdout:870 2021-07-31T02:01:27.129 INFO:tasks.workunit.client.0.smithi045.stdout:871 2021-07-31T02:01:27.130 INFO:tasks.workunit.client.0.smithi045.stdout:872 2021-07-31T02:01:27.131 INFO:tasks.workunit.client.0.smithi045.stdout:873 2021-07-31T02:01:27.132 INFO:tasks.workunit.client.0.smithi045.stdout:874 2021-07-31T02:01:27.134 INFO:tasks.workunit.client.0.smithi045.stdout:875 2021-07-31T02:01:27.135 INFO:tasks.workunit.client.0.smithi045.stdout:876 2021-07-31T02:01:27.136 INFO:tasks.workunit.client.0.smithi045.stdout:877 2021-07-31T02:01:27.137 INFO:tasks.workunit.client.0.smithi045.stdout:878 2021-07-31T02:01:27.138 INFO:tasks.workunit.client.0.smithi045.stdout:879 2021-07-31T02:01:27.139 INFO:tasks.workunit.client.0.smithi045.stdout:880 2021-07-31T02:01:27.141 INFO:tasks.workunit.client.0.smithi045.stdout:881 2021-07-31T02:01:27.142 INFO:tasks.workunit.client.0.smithi045.stdout:882 2021-07-31T02:01:27.143 INFO:tasks.workunit.client.0.smithi045.stdout:883 2021-07-31T02:01:27.144 INFO:tasks.workunit.client.0.smithi045.stdout:884 2021-07-31T02:01:27.146 INFO:tasks.workunit.client.0.smithi045.stdout:885 2021-07-31T02:01:27.147 INFO:tasks.workunit.client.0.smithi045.stdout:886 2021-07-31T02:01:27.148 INFO:tasks.workunit.client.0.smithi045.stdout:887 2021-07-31T02:01:27.149 INFO:tasks.workunit.client.0.smithi045.stdout:888 2021-07-31T02:01:27.151 INFO:tasks.workunit.client.0.smithi045.stdout:889 2021-07-31T02:01:27.152 INFO:tasks.workunit.client.0.smithi045.stdout:890 2021-07-31T02:01:27.153 INFO:tasks.workunit.client.0.smithi045.stdout:891 2021-07-31T02:01:27.154 INFO:tasks.workunit.client.0.smithi045.stdout:892 2021-07-31T02:01:27.155 INFO:tasks.workunit.client.0.smithi045.stdout:893 2021-07-31T02:01:27.157 INFO:tasks.workunit.client.0.smithi045.stdout:894 2021-07-31T02:01:27.158 INFO:tasks.workunit.client.0.smithi045.stdout:895 2021-07-31T02:01:27.159 INFO:tasks.workunit.client.0.smithi045.stdout:896 2021-07-31T02:01:27.161 INFO:tasks.workunit.client.0.smithi045.stdout:897 2021-07-31T02:01:27.162 INFO:tasks.workunit.client.0.smithi045.stdout:898 2021-07-31T02:01:27.163 INFO:tasks.workunit.client.0.smithi045.stdout:899 2021-07-31T02:01:27.164 INFO:tasks.workunit.client.0.smithi045.stdout:900 2021-07-31T02:01:27.165 INFO:tasks.workunit.client.0.smithi045.stdout:901 2021-07-31T02:01:27.166 INFO:tasks.workunit.client.0.smithi045.stdout:902 2021-07-31T02:01:27.167 INFO:tasks.workunit.client.0.smithi045.stdout:903 2021-07-31T02:01:27.168 INFO:tasks.workunit.client.0.smithi045.stdout:904 2021-07-31T02:01:27.169 INFO:tasks.workunit.client.0.smithi045.stdout:905 2021-07-31T02:01:27.171 INFO:tasks.workunit.client.0.smithi045.stdout:906 2021-07-31T02:01:27.173 INFO:tasks.workunit.client.0.smithi045.stdout:907 2021-07-31T02:01:27.173 INFO:tasks.workunit.client.0.smithi045.stdout:908 2021-07-31T02:01:27.174 INFO:tasks.workunit.client.0.smithi045.stdout:909 2021-07-31T02:01:27.175 INFO:tasks.workunit.client.0.smithi045.stdout:910 2021-07-31T02:01:27.176 INFO:tasks.workunit.client.0.smithi045.stdout:911 2021-07-31T02:01:27.178 INFO:tasks.workunit.client.0.smithi045.stdout:912 2021-07-31T02:01:27.179 INFO:tasks.workunit.client.0.smithi045.stdout:913 2021-07-31T02:01:27.180 INFO:tasks.workunit.client.0.smithi045.stdout:914 2021-07-31T02:01:27.181 INFO:tasks.workunit.client.0.smithi045.stdout:915 2021-07-31T02:01:27.182 INFO:tasks.workunit.client.0.smithi045.stdout:916 2021-07-31T02:01:27.183 INFO:tasks.workunit.client.0.smithi045.stdout:917 2021-07-31T02:01:27.185 INFO:tasks.workunit.client.0.smithi045.stdout:918 2021-07-31T02:01:27.186 INFO:tasks.workunit.client.0.smithi045.stdout:919 2021-07-31T02:01:27.187 INFO:tasks.workunit.client.0.smithi045.stdout:920 2021-07-31T02:01:27.188 INFO:tasks.workunit.client.0.smithi045.stdout:921 2021-07-31T02:01:27.189 INFO:tasks.workunit.client.0.smithi045.stdout:922 2021-07-31T02:01:27.190 INFO:tasks.workunit.client.0.smithi045.stdout:923 2021-07-31T02:01:27.191 INFO:tasks.workunit.client.0.smithi045.stdout:924 2021-07-31T02:01:27.193 INFO:tasks.workunit.client.0.smithi045.stdout:925 2021-07-31T02:01:27.194 INFO:tasks.workunit.client.0.smithi045.stdout:926 2021-07-31T02:01:27.195 INFO:tasks.workunit.client.0.smithi045.stdout:927 2021-07-31T02:01:27.196 INFO:tasks.workunit.client.0.smithi045.stdout:928 2021-07-31T02:01:27.198 INFO:tasks.workunit.client.0.smithi045.stdout:929 2021-07-31T02:01:27.198 INFO:tasks.workunit.client.0.smithi045.stdout:930 2021-07-31T02:01:27.200 INFO:tasks.workunit.client.0.smithi045.stdout:931 2021-07-31T02:01:27.201 INFO:tasks.workunit.client.0.smithi045.stdout:932 2021-07-31T02:01:27.202 INFO:tasks.workunit.client.0.smithi045.stdout:933 2021-07-31T02:01:27.203 INFO:tasks.workunit.client.0.smithi045.stdout:934 2021-07-31T02:01:27.205 INFO:tasks.workunit.client.0.smithi045.stdout:935 2021-07-31T02:01:27.206 INFO:tasks.workunit.client.0.smithi045.stdout:936 2021-07-31T02:01:27.207 INFO:tasks.workunit.client.0.smithi045.stdout:937 2021-07-31T02:01:27.209 INFO:tasks.workunit.client.0.smithi045.stdout:938 2021-07-31T02:01:27.209 INFO:tasks.workunit.client.0.smithi045.stdout:939 2021-07-31T02:01:27.211 INFO:tasks.workunit.client.0.smithi045.stdout:940 2021-07-31T02:01:27.212 INFO:tasks.workunit.client.0.smithi045.stdout:941 2021-07-31T02:01:27.213 INFO:tasks.workunit.client.0.smithi045.stdout:942 2021-07-31T02:01:27.214 INFO:tasks.workunit.client.0.smithi045.stdout:943 2021-07-31T02:01:27.215 INFO:tasks.workunit.client.0.smithi045.stdout:944 2021-07-31T02:01:27.217 INFO:tasks.workunit.client.0.smithi045.stdout:945 2021-07-31T02:01:27.218 INFO:tasks.workunit.client.0.smithi045.stdout:946 2021-07-31T02:01:27.220 INFO:tasks.workunit.client.0.smithi045.stdout:947 2021-07-31T02:01:27.221 INFO:tasks.workunit.client.0.smithi045.stdout:948 2021-07-31T02:01:27.222 INFO:tasks.workunit.client.0.smithi045.stdout:949 2021-07-31T02:01:27.223 INFO:tasks.workunit.client.0.smithi045.stdout:950 2021-07-31T02:01:27.224 INFO:tasks.workunit.client.0.smithi045.stdout:951 2021-07-31T02:01:27.225 INFO:tasks.workunit.client.0.smithi045.stdout:952 2021-07-31T02:01:27.227 INFO:tasks.workunit.client.0.smithi045.stdout:953 2021-07-31T02:01:27.228 INFO:tasks.workunit.client.0.smithi045.stdout:954 2021-07-31T02:01:27.229 INFO:tasks.workunit.client.0.smithi045.stdout:955 2021-07-31T02:01:27.230 INFO:tasks.workunit.client.0.smithi045.stdout:956 2021-07-31T02:01:27.231 INFO:tasks.workunit.client.0.smithi045.stdout:957 2021-07-31T02:01:27.233 INFO:tasks.workunit.client.0.smithi045.stdout:958 2021-07-31T02:01:27.234 INFO:tasks.workunit.client.0.smithi045.stdout:959 2021-07-31T02:01:27.235 INFO:tasks.workunit.client.0.smithi045.stdout:960 2021-07-31T02:01:27.236 INFO:tasks.workunit.client.0.smithi045.stdout:961 2021-07-31T02:01:27.237 INFO:tasks.workunit.client.0.smithi045.stdout:962 2021-07-31T02:01:27.238 INFO:tasks.workunit.client.0.smithi045.stdout:963 2021-07-31T02:01:27.239 INFO:tasks.workunit.client.0.smithi045.stdout:964 2021-07-31T02:01:27.241 INFO:tasks.workunit.client.0.smithi045.stdout:965 2021-07-31T02:01:27.242 INFO:tasks.workunit.client.0.smithi045.stdout:966 2021-07-31T02:01:27.243 INFO:tasks.workunit.client.0.smithi045.stdout:967 2021-07-31T02:01:27.244 INFO:tasks.workunit.client.0.smithi045.stdout:968 2021-07-31T02:01:27.246 INFO:tasks.workunit.client.0.smithi045.stdout:969 2021-07-31T02:01:27.247 INFO:tasks.workunit.client.0.smithi045.stdout:970 2021-07-31T02:01:27.248 INFO:tasks.workunit.client.0.smithi045.stdout:971 2021-07-31T02:01:27.249 INFO:tasks.workunit.client.0.smithi045.stdout:972 2021-07-31T02:01:27.251 INFO:tasks.workunit.client.0.smithi045.stdout:973 2021-07-31T02:01:27.252 INFO:tasks.workunit.client.0.smithi045.stdout:974 2021-07-31T02:01:27.253 INFO:tasks.workunit.client.0.smithi045.stdout:975 2021-07-31T02:01:27.254 INFO:tasks.workunit.client.0.smithi045.stdout:976 2021-07-31T02:01:27.255 INFO:tasks.workunit.client.0.smithi045.stdout:977 2021-07-31T02:01:27.257 INFO:tasks.workunit.client.0.smithi045.stdout:978 2021-07-31T02:01:27.258 INFO:tasks.workunit.client.0.smithi045.stdout:979 2021-07-31T02:01:27.259 INFO:tasks.workunit.client.0.smithi045.stdout:980 2021-07-31T02:01:27.260 INFO:tasks.workunit.client.0.smithi045.stdout:981 2021-07-31T02:01:27.261 INFO:tasks.workunit.client.0.smithi045.stdout:982 2021-07-31T02:01:27.262 INFO:tasks.workunit.client.0.smithi045.stdout:983 2021-07-31T02:01:27.264 INFO:tasks.workunit.client.0.smithi045.stdout:984 2021-07-31T02:01:27.265 INFO:tasks.workunit.client.0.smithi045.stdout:985 2021-07-31T02:01:27.266 INFO:tasks.workunit.client.0.smithi045.stdout:986 2021-07-31T02:01:27.267 INFO:tasks.workunit.client.0.smithi045.stdout:987 2021-07-31T02:01:27.269 INFO:tasks.workunit.client.0.smithi045.stdout:988 2021-07-31T02:01:27.270 INFO:tasks.workunit.client.0.smithi045.stdout:989 2021-07-31T02:01:27.271 INFO:tasks.workunit.client.0.smithi045.stdout:990 2021-07-31T02:01:27.272 INFO:tasks.workunit.client.0.smithi045.stdout:991 2021-07-31T02:01:27.274 INFO:tasks.workunit.client.0.smithi045.stdout:992 2021-07-31T02:01:27.275 INFO:tasks.workunit.client.0.smithi045.stdout:993 2021-07-31T02:01:27.276 INFO:tasks.workunit.client.0.smithi045.stdout:994 2021-07-31T02:01:27.277 INFO:tasks.workunit.client.0.smithi045.stdout:995 2021-07-31T02:01:27.278 INFO:tasks.workunit.client.0.smithi045.stdout:996 2021-07-31T02:01:27.280 INFO:tasks.workunit.client.0.smithi045.stdout:997 2021-07-31T02:01:27.281 INFO:tasks.workunit.client.0.smithi045.stdout:998 2021-07-31T02:01:27.282 INFO:tasks.workunit.client.0.smithi045.stdout:999 2021-07-31T02:01:28.227 INFO:tasks.workunit.client.0.smithi045.stdout:00000000 79 6f 75 20 6d 69 67 68 74 20 73 65 65 20 74 68 |you might see th| 2021-07-31T02:01:28.228 INFO:tasks.workunit.client.0.smithi045.stdout:00000010 69 73 |is| 2021-07-31T02:01:28.228 INFO:tasks.workunit.client.0.smithi045.stdout:00000012 2021-07-31T02:01:28.228 INFO:tasks.workunit.client.0.smithi045.stdout:00000000 79 6f 75 20 6d 69 67 68 74 20 73 65 65 20 74 68 |you might see th| 2021-07-31T02:01:28.228 INFO:tasks.workunit.client.0.smithi045.stdout:00000010 69 73 |is| 2021-07-31T02:01:28.229 INFO:tasks.workunit.client.0.smithi045.stdout:00000012 2021-07-31T02:01:28.229 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] ClsHello.WriteReturnData (4010 ms) 2021-07-31T02:01:28.229 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] ClsHello.Loud 2021-07-31T02:01:28.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:28 smithi154 conmon[34553]: cluster 2021-07-31T02:01:27.218299+0000 mon.a (mon.0) 1656 : cluster [DBG] osdmap e911: 8 total, 8 up, 8 in 2021-07-31T02:01:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:28 smithi154 conmon[34553]: audit 2021-07-31T02:01:27.985770+0000 mon.b (mon.2) 138 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:01:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:28 smithi154 conmon[34553]: audit 2021-07-31T02:01:27.986448+0000 mon.a (mon.0) 1657 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:01:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:28 smithi154 conmon[34553]: cluster 2021-07-31T02:01 2021-07-31T02:01:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:28 smithi154 conmon[34553]: :27.994274+0000 mgr.x (mgr.24542) 606 : cluster [DBG] pgmap v1185: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:01:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:28 smithi154 conmon[34553]: audit 2021-07-31T02:01:28.003758+0000 mon.b (mon. 2021-07-31T02:01:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:28 smithi154 conmon[34553]: 2) 139 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:01:28.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:28 smithi154 conmon[34553]: audit 2021-07-31T02:01:28 2021-07-31T02:01:28.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:28 smithi154 conmon[34553]: .004237+0000 mon.a (mon.0) 1658 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:01:28.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:28 smithi045 conmon[39718]: cluster 2021-07-31T02:01:27.218299+0000 mon.a (mon 2021-07-31T02:01:28.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:28 smithi045 conmon[39718]: .0) 1656 : cluster [DBG] osdmap e911: 8 total, 8 up, 8 in 2021-07-31T02:01:28.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:28 smithi045 conmon[39718]: audit 2021-07-31T02:01:27.985770+0000 mon.b (mon.2 2021-07-31T02:01:28.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:28 smithi045 conmon[39718]: ) 138 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:01:28.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:28 smithi045 conmon[39718]: audit 2021-07-31T02:01 2021-07-31T02:01:28.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:28 smithi045 conmon[39718]: :27.986448+0000 mon.a (mon.0) 2021-07-31T02:01:28.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:28 smithi045 conmon[39718]: 1657 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:01:28.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:28 smithi045 conmon[39718]: cluster 2021-07-31T02: 2021-07-31T02:01:28.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:28 smithi045 conmon[39718]: 01:27.994274+0000 mgr.x (mgr.24542) 606 : cluster 2021-07-31T02:01:28.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:28 smithi045 conmon[39718]: [DBG] pgmap v1185: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:01:28.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:28 smithi045 conmon[39718]: audit 2021-07-31T02:01:28.003758+0000 2021-07-31T02:01:28.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:28 smithi045 conmon[39718]: mon.b (mon.2) 139 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:01:28.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:28 smithi045 conmon[39718]: audit 2021-07-31T02:01:28.004237 2021-07-31T02:01:28.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:28 smithi045 conmon[39718]: +0000 mon.a (mon.0) 1658 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:01:28.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:28 smithi045 conmon[33655]: cluster 2021-07-31T02:01:27.218299+0000 mon.a (mon.0) 2021-07-31T02:01:28.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:28 smithi045 conmon[33655]: 1656 : cluster [DBG] osdmap e911: 8 total, 8 up, 8 in 2021-07-31T02:01:28.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:28 smithi045 conmon[33655]: audit 2021-07-31T02:01:27.985770+0000 mon.b (mon.2) 138 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:01:28.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:28 smithi045 conmon[33655]: audit 2021-07-31T02:01:27.986448+0000 mon.a (mon 2021-07-31T02:01:28.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:28 smithi045 conmon[33655]: .0) 1657 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:01:28.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:28 smithi045 conmon[33655]: cluster 2021-07-31T02:01: 2021-07-31T02:01:28.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:28 smithi045 conmon[33655]: 27.994274+0000 mgr.x (mgr.24542) 606 : cluster 2021-07-31T02:01:28.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:28 smithi045 conmon[33655]: [DBG] pgmap v1185: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:01:28.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:28 smithi045 conmon[33655]: audit 2021-07-31 2021-07-31T02:01:28.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:28 smithi045 conmon[33655]: T02:01:28.003758+0000 mon.b (mon.2 2021-07-31T02:01:28.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:28 smithi045 conmon[33655]: ) 139 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:01:28.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:28 smithi045 conmon[33655]: audit 2021 2021-07-31T02:01:28.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:28 smithi045 conmon[33655]: -07-31T02:01:28.004237+0000 2021-07-31T02:01:28.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:28 smithi045 conmon[33655]: mon.a (mon.0) 1658 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:01:29.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[34553]: cluster 2021-07-31T02:01:28.225083+0000 mon.a (mon.0) 1659 : cluster [DBG] osdmap e912: 8 total, 8 up, 8 in 2021-07-31T02:01:29.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:29 smithi045 conmon[33655]: cluster 2021-07-31T02:01:28.225083+0000 2021-07-31T02:01:29.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:29 smithi045 conmon[33655]: mon.a (mon.0) 1659 : cluster [DBG] osdmap e912: 8 total, 8 up, 8 in 2021-07-31T02:01:29.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:29 smithi045 conmon[39718]: cluster 2021-07-31T02:01:28.225083+0000 mon.a (mon.0) 1659 : cluster [DBG] 2021-07-31T02:01:29.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:29 smithi045 conmon[39718]: osdmap e912: 8 total, 8 up, 8 in 2021-07-31T02:01:30.003 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: debug 2021-07-31T02:01:29.580+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:01:30.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:01:30.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:01:30.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: yield 2021-07-31T02:01:30.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:01:30.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:01:30.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:01:30.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:01:30.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:01:30.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:01:30.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:01:30.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:01:30.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:01:30.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: task.result() 2021-07-31T02:01:30.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:01:30.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: debug 2021-07-31T02:01:29.592+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:01:30.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:01:30.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:01:30.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: yield 2021-07-31T02:01:30.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:01:30.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:01:30.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:01:30.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:01:30.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:01:30.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:01:30.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:01:30.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:01:30.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:01:30.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: task.result() 2021-07-31T02:01:30.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:01:30.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: debug 2021-07-31T02:01:29.617+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:01:30.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:01:30.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:01:30.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: yield 2021-07-31T02:01:30.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:01:30.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:01:30.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:01:30.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:01:30.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:01:30.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:01:30.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:01:30.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:01:30.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:01:30.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: task.result() 2021-07-31T02:01:30.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:01:30.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: debug 2021-07-31T02:01:29.618+0000 7fcb97969700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:01:30.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:01:30.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:01:30.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: yield 2021-07-31T02:01:30.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:01:30.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:01:30.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:01:30.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:01:30.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:01:30.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:01:30.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:01:30.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:01:30.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:01:30.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: task.result() 2021-07-31T02:01:30.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:01:30.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: 2021-07-31T02:01:30.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:01:30.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: 2021-07-31T02:01:30.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:01:30.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:01:30.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:01:30.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:01:30.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:01:30.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:01:30.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:01:30.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:01:30.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:01:30.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:01:30.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:01:30.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:01:30.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:01:30.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:01:30.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:01:30.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:01:30.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:01:30.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:01:30.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: raise 2021-07-31T02:01:30.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:01:30.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:01:30.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:01:30.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:01:30.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:01:30.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: debug 2021-07-31T02:01:29.626+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:01:30.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:01:30.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:01:30.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: yield 2021-07-31T02:01:30.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:01:30.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:01:30.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:01:30.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:01:30.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:01:30.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:01:30.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:01:30.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:01:30.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:01:30.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: task.result() 2021-07-31T02:01:30.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:01:30.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: debug 2021-07-31T02:01:29.627+0000 7fcb9816a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:01:30.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:01:30.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:01:30.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: yield 2021-07-31T02:01:30.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:01:30.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:01:30.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:01:30.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:01:30.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:01:30.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:01:30.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:01:30.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:01:30.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:01:30.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: task.result() 2021-07-31T02:01:30.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:01:30.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: 2021-07-31T02:01:30.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:01:30.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: 2021-07-31T02:01:30.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:01:30.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:01:30.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:01:30.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:01:30.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:01:30.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:01:30.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:01:30.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:01:30.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:01:30.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:01:30.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:01:30.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:01:30.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:01:30.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:01:30.035 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:01:30.035 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:01:30.035 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:01:30.035 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:01:30.036 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: raise 2021-07-31T02:01:30.036 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:01:30.036 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:01:30.036 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:01:30.036 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:01:30.037 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:01:29 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:01:30.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: cluster 2021-07-31T02:01:29.230373+0000 mon.a (mon.0) 1660 : cluster [DBG] osdmap e913: 8 total, 8 up, 8 in 2021-07-31T02:01:30.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: 2021-07-31T02:01:30.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: audit 2021-07-31T02:01:29.233032+0000 mon.c (mon.1) 50 : audit [INF] from='client.? 172.21.15.45:0/306473575' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73532-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:01:30.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: audit 2021-07-31T02:01:29.233808+0000 mon.a (mon.0) 1661 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73532-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:01:30.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: 2021-07-31T02:01:30.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: audit 2021-07-31T02:01:29.534521+0000 mon.b (mon 2021-07-31T02:01:30.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: .2) 140 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:01:30.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: audit 2021-07-31T02:01:29.536098+0000 mon.b (mon 2021-07-31T02:01:30.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: .2) 141 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:01:30.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: audit 2021-07-31T 2021-07-31T02:01:30.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: 02:01:29.537027+0000 mon.b (mon.2) 142 2021-07-31T02:01:30.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:01:30.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: cephadm 2021-07-31 2021-07-31T02:01:30.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: T02:01:29.537822+0000 mgr.x (mgr. 2021-07-31T02:01:30.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: 24542) 607 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:01:30.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:01:30.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: 2021-07-31T02:01:30.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: 2021 2021-07-31T02:01:30.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: - 2021-07-31T02:01:30.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: 07 2021-07-31T02:01:30.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: - 2021-07-31T02:01:30.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: 31 2021-07-31T02:01:30.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: T 2021-07-31T02:01:30.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: 02 2021-07-31T02:01:30.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: : 2021-07-31T02:01:30.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: 01 2021-07-31T02:01:30.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: : 2021-07-31T02:01:30.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: 29.581633+0000 mgr.x (mgr.24542) 608 2021-07-31T02:01:30.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: : cephadm [ERR] 2021-07-31T02:01:30.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:01:30.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:01:30.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: yield 2021-07-31T02:01:30.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:01:30.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:01:30.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:01:30.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:01:30.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:01:30.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:01:30.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:01:30.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:01:30.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:01:30.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:01:30.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:01:30.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:01:29.582151+0000 mgr.x (mgr.24542) 609 : cephadm [INF] 2021-07-31T02:01:30.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:01:30.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:01:30.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: 2021-07-31T02:01:29.593288 2021-07-31T02:01:30.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: +0000 mgr.x (mgr.24542) 610 : cephadm [ERR] 2021-07-31T02:01:30.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: 2021-07-31T02:01:30.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:01:30.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:01:30.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: yield 2021-07-31T02:01:30.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:01:30.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:01:30.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:01:30.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:01:30.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:01:30.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:01:30.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:01:30.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:01:30.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:01:30.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:01:30.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:01:30.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:01:29.594263+0000 mgr.x (mgr.24542) 611 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:01:30.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:01:29.618721+0000 2021-07-31T02:01:30.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: mgr.x (mgr.24542) 612 : cephadm [ERR] 2021-07-31T02:01:30.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:01:30.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:01:30.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: yield 2021-07-31T02:01:30.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:01:30.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:01:30.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:01:30.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:01:30.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:01:30.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:01:30.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:01:30.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:01:30.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:01:30.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:01:30.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:01:30.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:01:29.619155+0000 mgr.x (mgr.24542) 613 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:01:30.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:01:30.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:01:30.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: yield 2021-07-31T02:01:30.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:01:30.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:01:30.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:01:30.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:01:30.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:01:30.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:01:30.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:01:30.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:01:30.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:01:30.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:01:30.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:01:30.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: 2021-07-31T02:01:30.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:01:30.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: 2021-07-31T02:01:30.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:01:30.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:01:30.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:01:30.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:01:30.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:01:30.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:01:30.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:01:30.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:01:30.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:01:30.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:01:30.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:01:30.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:01:30.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:01:30.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:01:30.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:01:30.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:01:30.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:01:30.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:01:30.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: raise 2021-07-31T02:01:30.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:01:30.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:01:30.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:01:30.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:01:30.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:01:30.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:01:29.627828+0000 mgr.x (mgr.24542) 614 : cephadm [ERR] 2021-07-31T02:01:30.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:01:30.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: cluster 2021-07-31T02:01:29.230373+0000 mon.a (mon.0) 1660 : cluster [DBG] osdmap e913: 8 total, 8 up, 8 in 2021-07-31T02:01:30.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: 2021-07-31T02:01:30.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: audit 2021-07-31T02:01:29.233032+0000 mon.c (mon.1) 50 : audit [INF] from='client.? 172.21.15.45:0/306473575' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73532-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:01:30.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: audit 2021-07-31 2021-07-31T02:01:30.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: T02:01:29.233808+0000 mon.a (mon.0) 1661 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73532-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:01:30.535 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: audit 2021-07-31T02:01:30.535 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: 2021-07-31T02:01:29.534521+0000 mon.b (mon 2021-07-31T02:01:30.535 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: .2) 140 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:01:30.535 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: audit 2021-07-31T02:01 2021-07-31T02:01:30.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: :29.536098+0000 mon.b (mon.2 2021-07-31T02:01:30.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: ) 141 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:01:30.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: 2021-07-31T02:01:30.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: audit 2021 2021-07-31T02:01:30.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: -07-31T02:01: 2021-07-31T02:01:30.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: 29.537027+0000 mon.b (mon. 2021-07-31T02:01:30.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: 2) 142 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:01:30.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:01 2021-07-31T02:01:30.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: :29.537822+0000 mgr.x (mgr.24542) 607 : cephadm 2021-07-31T02:01:30.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:01:30.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:01:30.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: 2021-07-31 2021-07-31T02:01:30.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: T02:01:29.581633+0000 mgr.x (mgr 2021-07-31T02:01:30.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: .24542) 608 : cephadm [ERR] 2021-07-31T02:01:30.539 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:01:30.539 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:01:30.539 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: yield 2021-07-31T02:01:30.539 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:01:30.540 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:01:30.540 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:01:30.540 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:01:30.540 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:01:30.540 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:01:30.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:01:30.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:01:30.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:01:30.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:01:30.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:01:30.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:01:29.582151+0000 mgr.x (mgr.24542) 609 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:01:30.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: cluster 2021-07-31T02:01:29.230373 2021-07-31T02:01:30.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: +0000 mon.a (mon.0) 1660 : cluster [DBG] osdmap e913: 8 total, 8 up, 8 in 2021-07-31T02:01:30.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: audit 2021 2021-07-31T02:01:30.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: -07-31T02:01:29.233032+0000 mon.c (mon.1) 2021-07-31T02:01:30.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: 50 : audit [INF] from='client.? 172.21.15.45:0/306473575' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73532-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:01:30.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: audit 2021 2021-07-31T02:01:30.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: -07-31T02:01:29.233808+0000 mon.a (mon.0 2021-07-31T02:01:30.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: ) 1661 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73532-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:01:30.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T02:01:30.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: 31T02:01:29.534521+0000 mon.b (mon. 2021-07-31T02:01:30.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: 2) 140 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:01:30.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: audit 2021 2021-07-31T02:01:30.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: -07-31T02:01:29. 2021-07-31T02:01:30.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: 536098+0000 mon.b (mon.2 2021-07-31T02:01:30.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: ) 141 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:01:30.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: 2021-07-31T02:01:30.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: audit 2021- 2021-07-31T02:01:30.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: 07-31T02:01:29.537027+0000 mon.b (mon 2021-07-31T02:01:30.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: .2) 142 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:01:30.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: cephadm 2021-07-31 2021-07-31T02:01:30.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: T02:01:29.537822+0000 mgr.x (mgr.24542) 607 : 2021-07-31T02:01:30.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:01:30.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:01:30.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: 2021-07-31T02:01:29.581633 2021-07-31T02:01:30.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 608 : cephadm [ERR] 2021-07-31T02:01:30.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: 2021-07-31T02:01:30.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:01:30.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:01:30.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: yield 2021-07-31T02:01:30.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:01:30.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:01:30.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:01:30.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:01:30.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:01:30.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:01:30.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:01:30.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:01:30.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:01:30.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:01:30.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:01:30.551 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:01:30.551 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: yield 2021-07-31T02:01:30.552 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:01:30.552 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:01:30.552 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:01:30.552 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:01:30.553 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:01:30.553 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:01:30.553 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:01:30.553 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:01:30.553 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:01:30.554 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:01:30.554 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:01:30.554 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:01:29.628208+0000 mgr.x (mgr.24542) 615 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:01:30.554 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:01:30.554 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:01:30.555 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: yield 2021-07-31T02:01:30.555 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:01:30.555 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:01:30.555 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:01:30.556 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:01:30.556 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:01:30.556 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:01:30.556 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:01:30.556 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:01:30.557 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:01:30.557 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:01:30.557 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:01:30.557 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: 2021-07-31T02:01:30.557 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:01:30.558 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: 2021-07-31T02:01:30.558 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:01:30.558 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:01:30.558 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:01:30.559 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:01:30.559 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:01:30.559 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:01:30.559 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:01:30.559 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:01:30.560 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:01:30.560 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:01:30.560 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:01:30.560 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:01:30.560 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:01:30.561 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:01:30.561 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:01:30.561 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:01:30.561 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:01:30.562 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:01:30.562 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: raise 2021-07-31T02:01:30.562 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:01:30.562 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:01:30.562 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:01:30.563 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:01:30.563 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:01:30.563 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:30 smithi154 conmon[34553]: cluster 2021-07-31T02:01:29.994638+0000 mgr.x (mgr.24542) 616 : cluster [DBG] pgmap v1188: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:01:30.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: cephadm 2021-07- 2021-07-31T02:01:30.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: 31T02:01:29.593288+0000 mgr.x 2021-07-31T02:01:30.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: (mgr.24542) 610 : cephadm 2021-07-31T02:01:30.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: [ERR] 2021-07-31T02:01:30.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:01:30.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:01:30.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: yield 2021-07-31T02:01:30.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:01:30.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:01:30.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:01:30.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:01:30.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:01:30.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:01:30.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:01:30.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:01:30.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:01:30.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:01:30.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:01:30.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:01:29.594263+0000 mgr.x (mgr.24542) 611 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:01:30.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:01:30.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: 2021-07-31T02:01:29.618721+0000 2021-07-31T02:01:30.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: mgr.x (mgr.24542) 612 : cephadm 2021-07-31T02:01:30.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: [ERR] 2021-07-31T02:01:30.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:01:30.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:01:30.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: yield 2021-07-31T02:01:30.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:01:30.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:01:30.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:01:30.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:01:30.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:01:30.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:01:30.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:01:30.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:01:30.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:01:30.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:01:30.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:01:30.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:01:29.619155+0000 mgr.x (mgr.24542) 613 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:01:30.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:01:30.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:01:30.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: yield 2021-07-31T02:01:30.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:01:30.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:01:29.582151+0000 mgr.x (mgr.24542) 609 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:01:30.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:01:30.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: 2021-07-31T02:01:29.593288+0000 2021-07-31T02:01:30.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: mgr.x (mgr.24542) 610 : cephadm [ERR] 2021-07-31T02:01:30.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:01:30.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:01:30.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: yield 2021-07-31T02:01:30.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:01:30.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:01:30.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:01:30.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:01:30.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:01:30.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:01:30.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:01:30.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:01:30.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:01:30.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:01:30.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:01:30.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:01:29.594263+0000 mgr.x (mgr.24542) 611 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:01:30.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: cephadm 2021- 2021-07-31T02:01:30.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: 07-31T02:01:29.618721+0000 mgr.x ( 2021-07-31T02:01:30.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: mgr.24542) 612 : cephadm [ERR] 2021-07-31T02:01:30.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:01:30.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:01:30.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: yield 2021-07-31T02:01:30.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:01:30.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:01:30.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:01:30.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:01:30.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:01:30.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:01:30.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:01:30.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:01:30.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:01:30.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:01:30.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:01:30.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:01:29.619155+0000 mgr.x (mgr.24542) 613 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:01:30.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:01:30.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:01:30.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: yield 2021-07-31T02:01:30.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:01:30.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:01:30.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:01:30.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:01:30.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:01:30.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:01:30.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:01:30.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:01:30.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:01:30.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:01:30.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:01:30.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: 2021-07-31T02:01:30.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:01:30.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: 2021-07-31T02:01:30.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:01:30.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:01:30.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:01:30.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:01:30.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:01:30.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:01:30.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:01:30.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:01:30.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:01:30.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:01:30.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:01:30.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:01:30.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:01:30.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:01:30.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:01:30.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:01:30.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:01:30.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:01:30.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: raise 2021-07-31T02:01:30.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:01:30.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:01:30.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:01:30.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:01:30.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:01:30.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:01:29.627828+0000 mgr.x (mgr.24542) 614 : cephadm [ERR] 2021-07-31T02:01:30.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:01:30.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:01:30.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: yield 2021-07-31T02:01:30.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:01:30.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:01:30.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:01:30.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:01:30.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:01:30.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:01:30.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:01:30.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:01:30.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:01:30.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:01:30.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:01:30.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: 2021-07-31T02:01:30.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:01:30.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: 2021-07-31T02:01:30.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:01:30.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:01:30.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:01:30.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:01:30.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:01:30.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:01:30.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:01:30.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:01:30.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:01:30.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:01:30.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:01:30.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:01:30.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:01:30.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:01:30.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:01:30.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:01:30.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:01:30.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:01:30.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: raise 2021-07-31T02:01:30.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:01:30.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:01:30.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:01:30.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:01:30.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:01:30.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:01:29.627828+0000 mgr.x (mgr.24542) 614 : cephadm [ERR] 2021-07-31T02:01:30.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:01:30.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:01:30.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: yield 2021-07-31T02:01:30.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:01:30.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:01:30.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:01:30.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:01:30.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:01:30.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:01:30.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:01:30.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:01:30.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:01:30.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:01:30.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:01:29.628208+0000 mgr.x (mgr.24542) 615 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:01:30.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:01:30.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:01:30.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: yield 2021-07-31T02:01:30.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:01:30.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:01:30.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:01:30.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:01:30.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:01:30.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:01:30.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:01:30.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:01:30.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:01:30.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:01:30.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:01:30.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: 2021-07-31T02:01:30.608 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:01:30.608 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: 2021-07-31T02:01:30.608 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:01:30.608 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:01:30.609 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:01:30.609 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:01:30.609 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:01:30.609 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:01:30.609 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:01:30.610 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:01:30.610 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:01:30.610 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:01:30.610 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:01:30.611 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:01:30.611 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:01:30.611 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:01:30.611 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:01:30.611 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:01:30.612 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:01:30.612 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:01:30.612 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: raise 2021-07-31T02:01:30.612 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:01:30.612 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:01:30.613 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:01:30.613 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:01:30.613 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:01:30.613 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: cluster 2021-07-31T02:01:29.994638+0000 mgr.x (mgr.24542) 2021-07-31T02:01:30.614 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[33655]: 616 : cluster [DBG] pgmap v1188: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:01:30.614 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:01:30.614 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:01:30.615 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:01:30.615 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:01:30.615 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:01:30.615 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:01:30.616 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:01:30.616 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:01:30.616 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:01:30.616 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:01:30.616 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:01:30.617 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:01:29.628208+0000 mgr.x (mgr.24542) 615 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:01:30.617 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:01:30.617 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:01:30.617 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: yield 2021-07-31T02:01:30.618 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:01:30.618 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:01:30.618 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:01:30.618 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:01:30.618 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:01:30.619 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:01:30.619 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:01:30.619 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:01:30.619 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:01:30.620 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:01:30.620 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:01:30.620 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: 2021-07-31T02:01:30.620 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:01:30.620 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: 2021-07-31T02:01:30.621 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:01:30.621 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:01:30.621 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:01:30.621 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:01:30.622 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:01:30.622 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:01:30.622 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:01:30.622 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:01:30.622 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:01:30.623 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:01:30.623 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:01:30.623 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:01:30.623 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:01:30.623 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:01:30.624 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:01:30.624 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:01:30.624 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:01:30.624 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:01:30.625 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: raise 2021-07-31T02:01:30.625 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:01:30.625 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:01:30.625 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:01:30.625 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:01:30.626 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:01:30.626 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:30 smithi045 conmon[39718]: cluster 2021-07-31T02:01:29.994638+0000 mgr.x (mgr.24542) 616 : cluster [DBG] pgmap v1188: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:01:31.242 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] ClsHello.Loud (3015 ms) 2021-07-31T02:01:31.243 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] ClsHello.BadMethods 2021-07-31T02:01:31.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:31 smithi154 conmon[34553]: audit 2021-07-31T02:01:30.238134+0000 mon.a (mon.0) 1662 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73532-4","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:01:31.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:31 smithi154 conmon[34553]: cluster 2021-07-31T02:01:30.238360+0000 mon.a (mon.0) 1663 : cluster [DBG] osdmap e914: 8 total, 8 up, 8 in 2021-07-31T02:01:31.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:31 smithi045 conmon[33655]: audit 2021-07-31T02:01:30.238134+0000 mon.a (mon.0) 1662 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73532-4","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:01:31.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:31 smithi045 conmon[33655]: cluster 2021-07-31T02:01:30.238360+0000 mon.a (mon.0) 1663 : cluster [DBG] osdmap e914: 8 total, 8 up, 8 in 2021-07-31T02:01:31.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:31 smithi045 conmon[39718]: audit 2021-07-31T02:01:31.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:31 smithi045 conmon[39718]: 2021-07-31T02:01:31.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:31 smithi045 conmon[39718]: 2021 2021-07-31T02:01:31.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:31 smithi045 conmon[39718]: - 2021-07-31T02:01:31.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:31 smithi045 conmon[39718]: 07 2021-07-31T02:01:31.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:31 smithi045 conmon[39718]: - 2021-07-31T02:01:31.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:31 smithi045 conmon[39718]: 31 2021-07-31T02:01:31.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:31 smithi045 conmon[39718]: T 2021-07-31T02:01:31.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:31 smithi045 conmon[39718]: 02 2021-07-31T02:01:31.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:31 smithi045 conmon[39718]: : 2021-07-31T02:01:31.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:31 smithi045 conmon[39718]: 01 2021-07-31T02:01:31.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:31 smithi045 conmon[39718]: : 2021-07-31T02:01:31.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:31 smithi045 conmon[39718]: 30 2021-07-31T02:01:31.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:31 smithi045 conmon[39718]: . 2021-07-31T02:01:31.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:31 smithi045 conmon[39718]: 238134 2021-07-31T02:01:31.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:31 smithi045 conmon[39718]: +0000 2021-07-31T02:01:31.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:31 smithi045 conmon[39718]: 2021-07-31T02:01:31.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:31 smithi045 conmon[39718]: mon.a 2021-07-31T02:01:31.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:31 smithi045 conmon[39718]: (mon.0) 1662 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73532-4","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:01:31.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:31 smithi045 conmon[39718]: cluster 2021-07-31 2021-07-31T02:01:31.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:31 smithi045 conmon[39718]: T02:01:30.238360+0000 mon.a (mon.0 2021-07-31T02:01:31.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:31 smithi045 conmon[39718]: ) 1663 : cluster [DBG] osdmap e914: 8 total, 8 up, 8 in 2021-07-31T02:01:32.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:32 smithi154 conmon[34553]: cluster 2021-07-31T02:01:31.239351+0000 mon.a (mon.0) 1664 : cluster [DBG] osdmap e915: 8 total, 8 up, 8 in 2021-07-31T02:01:32.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:32 smithi154 conmon[34553]: cluster 2021-07-31T02:01:31.995102+0000 mgr.x (mgr.24542) 617 : cluster [DBG] pgmap v1191: 9 pgs: 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:01:32.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:32 smithi045 conmon[33655]: cluster 2021-07-31T02:01:31.239351+0000 mon.a (mon.0 2021-07-31T02:01:32.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:32 smithi045 conmon[33655]: ) 1664 : cluster [DBG] osdmap e915: 8 total, 8 up, 8 in 2021-07-31T02:01:32.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:32 smithi045 conmon[33655]: cluster 2021-07-31T02:01:31.995102+0000 mgr.x (mgr.24542) 617 : cluster [DBG] pgmap v1191: 9 pgs: 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:01:32.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:32 smithi045 conmon[39718]: cluster 2021-07-31T02:01:31.239351+0000 mon.a 2021-07-31T02:01:32.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:32 smithi045 conmon[39718]: (mon.0) 1664 : cluster [DBG] osdmap e915: 8 total, 8 up, 8 in 2021-07-31T02:01:32.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:32 smithi045 conmon[39718]: cluster 2021-07-31T02:01:31. 2021-07-31T02:01:32.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:32 smithi045 conmon[39718]: 995102+0000 mgr.x (mgr.24542) 617 : cluster [DBG] pgmap v1191: 9 pgs: 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:01:33.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:33 smithi045 conmon[33655]: cluster 2021-07-31T02:01:32.253390+0000 2021-07-31T02:01:33.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:33 smithi045 conmon[33655]: mon.a (mon.0) 1665 : cluster [DBG] osdmap e916: 8 total, 8 up, 8 in 2021-07-31T02:01:33.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:33 smithi045 conmon[33655]: audit 2021-07-31T02:01:32.254461 2021-07-31T02:01:33.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:33 smithi045 conmon[33655]: +0000 mon.c (mon.1) 51 : audit [INF] from='client.? 172.21.15.45:0/699377303' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73532-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:01:33.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:33 smithi045 conmon[33655]: audit 2021-07 2021-07-31T02:01:33.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:33 smithi045 conmon[33655]: -31T02:01:32.255061 2021-07-31T02:01:33.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:33 smithi045 conmon[33655]: +0000 mon.a (mon.0) 1666 2021-07-31T02:01:33.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:33 smithi045 conmon[33655]: : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73532-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:01:33.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:33 smithi045 conmon[39718]: cluster 2021-07-31T02:01:32.253390+0000 mon.a ( 2021-07-31T02:01:33.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:33 smithi045 conmon[39718]: mon.0) 1665 : cluster [DBG] osdmap e916: 8 total, 8 up, 8 in 2021-07-31T02:01:33.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:33 smithi045 conmon[39718]: audit 2021-07-31T02:01:32.254461+0000 mon.c (mon.1) 51 : 2021-07-31T02:01:33.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:33 smithi045 conmon[39718]: audit [INF] from='client.? 172.21.15.45:0/699377303' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73532-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:01:33.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:33 smithi045 conmon[39718]: audit 2021-07-31T02:01:32 2021-07-31T02:01:33.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:33 smithi045 conmon[39718]: .255061+0000 mon.a (mon.0) 1666 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73532-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:01:33.753 INFO:journalctl@ceph.osd.6.smithi154.stdout:Jul 31 02:01:33 smithi154 conmon[45576]: debug 2021-07-31T02:01:33.273+0000 7f47c50ad700 -1 osd.6 pg_epoch: 917 pg[31.2( v 917'1 (0'0,917'1] local-lis/les=916/917 n=1 ec=916/916 lis/c=916/916 les/c/f=917/917/0 sis=916) [6,4,5] r=0 lpr=916 crt=917'1 lcod 0'0 mlcod 0'0 active+clean] method hello.bad_reader tried to read object but is not marked RD 2021-07-31T02:01:33.753 INFO:journalctl@ceph.osd.6.smithi154.stdout:Jul 31 02:01:33 smithi154 conmon[45576]: debug 2021-07-31T02:01:33.274+0000 7f47c50ad700 -1 osd.6 pg_epoch: 917 pg[31.2( v 917'2 (0'0,917'2] local-lis/les=916/917 n=1 ec=916/916 lis/c=916/916 les/c/f=917/917/0 sis=916) [6,4,5] r=0 lpr=916 crt=917'1 lcod 917'1 mlcod 917'1 active+clean] method hello.bad_writer tried to update object but is not marked WR 2021-07-31T02:01:33.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:33 smithi154 conmon[34553]: cluster 2021-07-31T02:01:32.253390+0000 mon.a (mon.0) 1665 : cluster [DBG] osdmap e916: 8 total, 8 up, 8 in 2021-07-31T02:01:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:33 smithi154 conmon[34553]: audit 2021-07-31T02:01:32.254461+0000 mon.c (mon.1) 51 : audit [INF] from='client.? 172.21.15.45:0/699377303' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73532-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:01:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:33 smithi154 conmon[34553]: audit 2021-07-31T02: 2021-07-31T02:01:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:33 smithi154 conmon[34553]: 01:32.255061+0000 mon.a (mon.0) 1666 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73532-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:01:34.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:01:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:01:33.533Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:01:34.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:01:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:01:33.533Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:01:34.027 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:01:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:01:33.533Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:01:34.268 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] ClsHello.BadMethods (3026 ms) 2021-07-31T02:01:34.269 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] ClsHello.Filter 2021-07-31T02:01:34.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:34 smithi045 conmon[33655]: audit 2021-07-31T02:01:33.259930+0000 mon.a (mon.0) 1667 2021-07-31T02:01:34.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:34 smithi045 conmon[33655]: : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73532-5","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:01:34.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:34 smithi045 conmon[33655]: cluster 2021-07-31T02:01:33.260212+0000 mon.a (mon.0) 1668 : cluster [DBG] osdmap e917: 8 total, 8 up, 8 in 2021-07-31T02:01:34.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:34 smithi045 conmon[33655]: cluster 2021-07-31T02:01:33.995758+0000 mgr.x (mgr.24542) 618 : cluster [DBG] pgmap v1194: 41 pgs: 41 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2021-07-31T02:01:34.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:34 smithi045 conmon[39718]: audit 2021-07-31T02:01:33 2021-07-31T02:01:34.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:34 smithi045 conmon[39718]: .259930+0000 mon.a (mon.0) 1667 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73532-5","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:01:34.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:34 smithi045 conmon[39718]: cluster 2021-07-31T02:01:34.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:34 smithi045 conmon[39718]: 2021-07-31T02:01 2021-07-31T02:01:34.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:34 smithi045 conmon[39718]: :33.260212+0000 mon.a (mon.0) 1668 : 2021-07-31T02:01:34.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:34 smithi045 conmon[39718]: cluster [DBG] osdmap e917: 8 total, 8 up, 8 in 2021-07-31T02:01:34.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:34 smithi045 conmon[39718]: cluster 2021-07-31T02:01:34.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:34 smithi045 conmon[39718]: 2021-07-31T02:01:33.995758 2021-07-31T02:01:34.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:34 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 618 : cluster 2021-07-31T02:01:34.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:34 smithi045 conmon[39718]: [DBG] pgmap v1194: 41 pgs: 41 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2021-07-31T02:01:34.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:34 smithi154 conmon[34553]: audit 2021-07-31T02:01:33.259930+0000 mon.a (mon.0) 1667 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73532-5","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:01:34.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:34 smithi154 conmon[34553]: cluster 2021-07-31T02:01:34.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:34 smithi154 conmon[34553]: 2021-07-31T02:01:33.260212+0000 mon.a (mon.0) 1668 : cluster [DBG] osdmap e917: 8 total, 8 up, 8 in 2021-07-31T02:01:34.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:34 smithi154 conmon[34553]: cluster 2021-07-31T02:01:33.995758+0000 mgr.x (mgr.24542) 618 : cluster [DBG] pgmap v1194: 41 pgs: 41 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2021-07-31T02:01:35.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:35 smithi045 conmon[33655]: cluster 2021-07-31T02:01:34 2021-07-31T02:01:35.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:35 smithi045 conmon[33655]: .265324+0000 mon.a (mon.0) 1669 : cluster [DBG] osdmap e918: 8 total, 8 up, 8 in 2021-07-31T02:01:35.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:35 smithi045 conmon[39718]: cluster 2021-07-31T02:01:34. 2021-07-31T02:01:35.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:35 smithi045 conmon[39718]: 265324+0000 mon.a (mon.0) 1669 : cluster [DBG] osdmap e918: 8 total, 8 up, 8 in 2021-07-31T02:01:35.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:35 smithi154 conmon[34553]: cluster 2021-07-31T02:01:34.265324+0000 mon.a (mon.0) 1669 : 2021-07-31T02:01:35.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:35 smithi154 conmon[34553]: cluster [DBG] osdmap e918: 8 total, 8 up, 8 in 2021-07-31T02:01:36.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:36 smithi045 conmon[39718]: cluster 2021-07-31T02:01:36.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:36 smithi045 conmon[39718]: 2021-07-31T02:01:35.270903 2021-07-31T02:01:36.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:36 smithi045 conmon[39718]: +0000 mon.a (mon.0) 1670 : cluster [DBG] osdmap e919: 8 total, 8 up, 8 in 2021-07-31T02:01:36.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:36 smithi045 conmon[39718]: audit 2021-07-31T02:01:36.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:36 smithi045 conmon[39718]: 2021-07-31T02:01:35.272336+0000 mon.c (mon.1) 52 : audit 2021-07-31T02:01:36.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:36 smithi045 conmon[39718]: [INF] from='client.? 172.21.15.45:0/2751178836' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73532-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:01:36.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:36 smithi045 conmon[39718]: audit 2021-07-31T02:01:36.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:36 smithi045 conmon[39718]: 2021-07-31T02:01:35.273360+0000 mon.a (mon.0) 2021-07-31T02:01:36.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:36 smithi045 conmon[39718]: 1671 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73532-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:01:36.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:36 smithi045 conmon[39718]: cluster 2021-07-31T02:01:35.996159+0000 mgr.x 2021-07-31T02:01:36.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:36 smithi045 conmon[39718]: (mgr.24542) 619 : cluster [DBG] pgmap v1197: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:01:36.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:36 smithi045 conmon[39718]: 2021-07-31T02:01:36.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:36 smithi045 conmon[33655]: cluster 2021-07-31T02:01:35.270903+0000 mon.a (mon.0) 1670 2021-07-31T02:01:36.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:36 smithi045 conmon[33655]: : cluster [DBG] osdmap e919: 8 total, 8 up, 8 in 2021-07-31T02:01:36.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:36 smithi045 conmon[33655]: audit 2021-07-31T02:01:35.272336+0000 mon.c (mon.1) 52 : audit [INF] from='client.? 172.21.15.45:0/2751178836' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73532-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:01:36.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:36 smithi045 conmon[33655]: audit 2021-07-31T02:01:36.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:36 smithi045 conmon[33655]: 2021-07-31T02:01:35.273360+0000 2021-07-31T02:01:36.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:36 smithi045 conmon[33655]: mon.a (mon.0) 1671 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73532-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:01:36.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:36 smithi045 conmon[33655]: cluster 2021-07-31T02:01:36.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:36 smithi045 conmon[33655]: 2021-07-31T02:01:35.996159+0000 mgr.x (mgr. 2021-07-31T02:01:36.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:36 smithi045 conmon[33655]: 24542) 619 : cluster [DBG] pgmap v1197: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:01:36.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:36 smithi154 conmon[34553]: cluster 2021-07-31T02:01:35.270903+0000 mon.a (mon.0) 1670 : cluster [DBG] osdmap e919: 8 total, 8 up, 8 in 2021-07-31T02:01:36.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:36 smithi154 conmon[34553]: audit 2021-07-31T02:01:35.272336+0000 mon.c (mon.1) 52 : audit [INF] from='client.? 172.21.15.45:0/2751178836' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73532-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:01:36.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:36 smithi154 conmon[34553]: audit 2021-07-31T02:01:35.273360+0000 mon.a (mon.0) 1671 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73532-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:01:36.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:36 smithi154 conmon[34553]: cluster 2021-07-31T02:01:35.996159+0000 mgr.x (mgr.24542) 619 : cluster [DBG] pgmap v1197: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:01:36.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:36 smithi154 conmon[34553]: 2021-07-31T02:01:37.282 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] ClsHello.Filter (3014 ms) 2021-07-31T02:01:37.282 INFO:tasks.workunit.client.0.smithi045.stdout:[----------] 6 tests from ClsHello (18965 ms total) 2021-07-31T02:01:37.282 INFO:tasks.workunit.client.0.smithi045.stdout: 2021-07-31T02:01:37.283 INFO:tasks.workunit.client.0.smithi045.stdout:[----------] Global test environment tear-down 2021-07-31T02:01:37.283 INFO:tasks.workunit.client.0.smithi045.stdout:[==========] 6 tests from 1 test suite ran. (18965 ms total) 2021-07-31T02:01:37.283 INFO:tasks.workunit.client.0.smithi045.stdout:[ PASSED ] 6 tests. 2021-07-31T02:01:37.285 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-07-31T02:01:37.285 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2021-07-31T02:01:37.356 INFO:tasks.workunit:Running workunit cls/test_cls_journal.sh... 2021-07-31T02:01:37.357 DEBUG:teuthology.orchestra.run.smithi045:workunit test cls/test_cls_journal.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=pacific TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_journal.sh 2021-07-31T02:01:37.427 INFO:tasks.workunit.client.0.smithi045.stdout:Running main() from gmock_main.cc 2021-07-31T02:01:37.427 INFO:tasks.workunit.client.0.smithi045.stdout:[==========] Running 27 tests from 1 test suite. 2021-07-31T02:01:37.428 INFO:tasks.workunit.client.0.smithi045.stdout:[----------] Global test environment set-up. 2021-07-31T02:01:37.428 INFO:tasks.workunit.client.0.smithi045.stdout:[----------] 27 tests from TestClsJournal 2021-07-31T02:01:37.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:37 smithi154 conmon[34553]: audit 2021-07-31T02:01:36.277995+0000 mon.a (mon. 2021-07-31T02:01:37.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:37 smithi154 conmon[34553]: 0) 1672 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73532-6","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:01:37.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:37 smithi154 conmon[34553]: cluster 2021-07-31T02:01:36.278314+0000 mon.a (mon.0) 1673 : cluster [DBG] osdmap e920: 8 total, 8 up, 8 in 2021-07-31T02:01:37.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:37 smithi045 conmon[39718]: audit 2021-07-31T02:01:36.277995+0000 mon.a (mon.0 2021-07-31T02:01:37.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:37 smithi045 conmon[39718]: ) 1672 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73532-6","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:01:37.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:37 smithi045 conmon[39718]: cluster 2021-07-31T02:01:36.278314+0000 mon.a (mon.0) 1673 : cluster [DBG] osdmap e920: 8 total, 8 up, 8 in 2021-07-31T02:01:37.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:37 smithi045 conmon[33655]: audit 2021-07-31T02:01:36.277995+0000 mon.a (mon.0) 1672 : 2021-07-31T02:01:37.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:37 smithi045 conmon[33655]: audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73532-6","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:01:37.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:37 smithi045 conmon[33655]: cluster 2021-07-31T02:01:36.278314+0000 mon.a (mon.0) 1673 : cluster 2021-07-31T02:01:37.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:37 smithi045 conmon[33655]: [DBG] osdmap e920: 8 total, 8 up, 8 in 2021-07-31T02:01:38.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:38 smithi154 conmon[34553]: cluster 2021-07-31T02:01:37.278872+0000 mon.a (mon.0) 1674 : cluster [DBG] 2021-07-31T02:01:38.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:38 smithi154 conmon[34553]: osdmap e921: 8 total, 8 up, 8 in 2021-07-31T02:01:38.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:38 smithi154 conmon[34553]: cluster 2021-07-31T02:01:37.996636+0000 mgr.x (mgr.24542) 620 : cluster [DBG] pgmap v1200: 9 pgs: 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:01:38.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:38 smithi045 conmon[33655]: cluster 2021-07-31T02:01: 2021-07-31T02:01:38.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:38 smithi045 conmon[33655]: 37.278872+0000 mon.a (mon.0) 1674 : cluster [DBG] osdmap e921: 8 total, 8 up, 8 in 2021-07-31T02:01:38.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:38 smithi045 conmon[33655]: cluster 2021-07-31T02:01:37.996636+0000 mgr.x (mgr.24542) 620 : cluster [DBG] pgmap v1200: 9 pgs: 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:01:38.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:38 smithi045 conmon[39718]: cluster 2021-07-31T02:01:37.278872 2021-07-31T02:01:38.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:38 smithi045 conmon[39718]: +0000 mon.a (mon.0) 1674 : cluster [DBG] osdmap e921: 8 total, 8 up, 8 in 2021-07-31T02:01:38.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:38 smithi045 conmon[39718]: cluster 2021-07-31T02:01:37 2021-07-31T02:01:38.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:38 smithi045 conmon[39718]: .996636+0000 mgr.x (mgr.24542) 620 : cluster [DBG] pgmap v1200: 9 pgs: 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:01:39.307 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsJournal.Create 2021-07-31T02:01:39.315 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsJournal.Create (8 ms) 2021-07-31T02:01:39.315 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsJournal.MinimumSet 2021-07-31T02:01:39.320 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsJournal.MinimumSet (5 ms) 2021-07-31T02:01:39.320 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsJournal.MinimumSetStale 2021-07-31T02:01:39.327 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsJournal.MinimumSetStale (7 ms) 2021-07-31T02:01:39.327 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsJournal.MinimumSetOrderConstraint 2021-07-31T02:01:39.330 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsJournal.MinimumSetOrderConstraint (3 ms) 2021-07-31T02:01:39.330 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsJournal.ActiveSet 2021-07-31T02:01:39.333 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsJournal.ActiveSet (4 ms) 2021-07-31T02:01:39.334 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsJournal.ActiveSetStale 2021-07-31T02:01:39.337 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsJournal.ActiveSetStale (3 ms) 2021-07-31T02:01:39.337 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsJournal.CreateDuplicate 2021-07-31T02:01:39.339 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsJournal.CreateDuplicate (2 ms) 2021-07-31T02:01:39.339 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsJournal.GetClient 2021-07-31T02:01:39.342 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsJournal.GetClient (3 ms) 2021-07-31T02:01:39.342 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsJournal.ClientRegister 2021-07-31T02:01:39.345 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsJournal.ClientRegister (3 ms) 2021-07-31T02:01:39.345 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsJournal.ClientRegisterDuplicate 2021-07-31T02:01:39.349 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsJournal.ClientRegisterDuplicate (5 ms) 2021-07-31T02:01:39.349 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsJournal.ClientUpdateData 2021-07-31T02:01:39.354 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsJournal.ClientUpdateData (4 ms) 2021-07-31T02:01:39.354 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsJournal.ClientUpdateState 2021-07-31T02:01:39.359 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsJournal.ClientUpdateState (5 ms) 2021-07-31T02:01:39.359 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsJournal.ClientUnregister 2021-07-31T02:01:39.362 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsJournal.ClientUnregister (4 ms) 2021-07-31T02:01:39.363 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsJournal.ClientUnregisterDNE 2021-07-31T02:01:39.367 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsJournal.ClientUnregisterDNE (4 ms) 2021-07-31T02:01:39.367 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsJournal.ClientUnregisterPruneTags 2021-07-31T02:01:39.471 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsJournal.ClientUnregisterPruneTags (104 ms) 2021-07-31T02:01:39.471 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsJournal.ClientCommit 2021-07-31T02:01:39.474 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsJournal.ClientCommit (3 ms) 2021-07-31T02:01:39.474 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsJournal.ClientCommitInvalid 2021-07-31T02:01:39.477 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsJournal.ClientCommitInvalid (4 ms) 2021-07-31T02:01:39.478 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsJournal.ClientCommitDNE 2021-07-31T02:01:39.478 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsJournal.ClientCommitDNE (1 ms) 2021-07-31T02:01:39.479 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsJournal.ClientList 2021-07-31T02:01:39.521 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsJournal.ClientList (42 ms) 2021-07-31T02:01:39.521 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsJournal.GetNextTagTid 2021-07-31T02:01:39.525 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsJournal.GetNextTagTid (4 ms) 2021-07-31T02:01:39.525 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsJournal.TagCreate 2021-07-31T02:01:39.534 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsJournal.TagCreate (9 ms) 2021-07-31T02:01:39.534 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsJournal.TagCreatePrunesTags 2021-07-31T02:01:39.541 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsJournal.TagCreatePrunesTags (7 ms) 2021-07-31T02:01:39.541 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsJournal.TagList 2021-07-31T02:01:39.641 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsJournal.TagList (100 ms) 2021-07-31T02:01:39.643 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsJournal.GuardAppend 2021-07-31T02:01:39.644 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsJournal.GuardAppend (3 ms) 2021-07-31T02:01:39.644 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsJournal.GuardAppendDNE 2021-07-31T02:01:39.645 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsJournal.GuardAppendDNE (1 ms) 2021-07-31T02:01:39.645 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsJournal.GuardAppendOverflow 2021-07-31T02:01:39.647 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsJournal.GuardAppendOverflow (2 ms) 2021-07-31T02:01:39.647 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsJournal.Append 2021-07-31T02:01:39.655 INFO:tasks.workunit.client.0.smithi045.stdout:padding is not enabled 2021-07-31T02:01:39.655 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsJournal.Append (8 ms) 2021-07-31T02:01:39.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:39 smithi154 conmon[34553]: cluster 2021-07-31T02:01:38.290395+0000 mon.a (mon. 2021-07-31T02:01:39.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:39 smithi154 conmon[34553]: 0) 1675 : cluster [DBG] osdmap e922: 8 total, 8 up, 8 in 2021-07-31T02:01:39.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:39 smithi154 conmon[34553]: audit 2021-07-31T02:01:38.291711+0000 mon.c (mon.1) 53 : audit [INF] from='client.? 172.21.15.45:0/957291315' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73746-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:01:39.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:39 smithi154 conmon[34553]: audit 2021 2021-07-31T02:01:39.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:39 smithi154 conmon[34553]: -07-31T02:01:38 2021-07-31T02:01:39.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:39 smithi154 conmon[34553]: .292715+0000 mon.a (mon.0 2021-07-31T02:01:39.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:39 smithi154 conmon[34553]: ) 1676 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73746-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:01:39.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:39 smithi045 conmon[33655]: cluster 2021-07-31 2021-07-31T02:01:39.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:39 smithi045 conmon[33655]: T02:01:38.290395+0000 mon.a (mon.0) 1675 : cluster [DBG] osdmap e922: 8 total, 8 up, 8 in 2021-07-31T02:01:39.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:39 smithi045 conmon[33655]: audit 2021-07-31T02:01:38.291711+0000 mon.c (mon.1) 53 : audit [INF] from='client.? 172.21.15.45:0/957291315' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73746-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:01:39.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:39 smithi045 conmon[33655]: audit 2021-07-31T02:01:38.292715+0000 mon.a (mon.0) 1676 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73746-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:01:39.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:39 smithi045 conmon[39718]: cluster 2021-07-31T02:01:39.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:39 smithi045 conmon[39718]: 2021-07-31T02:01:39.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:39 smithi045 conmon[39718]: 2021 2021-07-31T02:01:39.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:39 smithi045 conmon[39718]: - 2021-07-31T02:01:39.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:39 smithi045 conmon[39718]: 07-31T02:01:38.290395+0000 2021-07-31T02:01:39.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:39 smithi045 conmon[39718]: mon.a (mon.0) 1675 : cluster [DBG] osdmap e922: 8 total, 8 up, 8 in 2021-07-31T02:01:39.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:39 smithi045 conmon[39718]: audit 2021-07-31T02:01:39.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:39 smithi045 conmon[39718]: 2021-07-31T02:01:39.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:39 smithi045 conmon[39718]: 2021 2021-07-31T02:01:39.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:39 smithi045 conmon[39718]: - 2021-07-31T02:01:39.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:39 smithi045 conmon[39718]: 07 2021-07-31T02:01:39.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:39 smithi045 conmon[39718]: - 2021-07-31T02:01:39.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:39 smithi045 conmon[39718]: 31 2021-07-31T02:01:39.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:39 smithi045 conmon[39718]: T 2021-07-31T02:01:39.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:39 smithi045 conmon[39718]: 02 2021-07-31T02:01:39.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:39 smithi045 conmon[39718]: : 2021-07-31T02:01:39.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:39 smithi045 conmon[39718]: 01 2021-07-31T02:01:39.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:39 smithi045 conmon[39718]: : 2021-07-31T02:01:39.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:39 smithi045 conmon[39718]: 38.291711+0000 mon.c (mon.1) 53 2021-07-31T02:01:39.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:39 smithi045 conmon[39718]: : audit [INF] from='client.? 172.21.15.45:0/957291315' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73746-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:01:39.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:39 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T02:01:39.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:39 smithi045 conmon[39718]: 31T02:01:38.292715+0000 mon.a (mon.0) 1676 : audit [INF] 2021-07-31T02:01:39.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:39 smithi045 conmon[39718]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73746-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:01:39.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:39 smithi045 conmon[39718]: 2021-07-31T02:01:40.309 INFO:tasks.workunit.client.0.smithi045.stdout:[----------] 27 tests from TestClsJournal (348 ms total) 2021-07-31T02:01:40.309 INFO:tasks.workunit.client.0.smithi045.stdout: 2021-07-31T02:01:40.309 INFO:tasks.workunit.client.0.smithi045.stdout:[----------] Global test environment tear-down 2021-07-31T02:01:40.310 INFO:tasks.workunit.client.0.smithi045.stdout:[==========] 27 tests from 1 test suite ran. (2882 ms total) 2021-07-31T02:01:40.310 INFO:tasks.workunit.client.0.smithi045.stdout:[ PASSED ] 27 tests. 2021-07-31T02:01:40.311 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-07-31T02:01:40.311 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2021-07-31T02:01:40.381 INFO:tasks.workunit:Running workunit cls/test_cls_lock.sh... 2021-07-31T02:01:40.382 DEBUG:teuthology.orchestra.run.smithi045:workunit test cls/test_cls_lock.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=pacific TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_lock.sh 2021-07-31T02:01:40.453 INFO:tasks.workunit.client.0.smithi045.stdout:Running main() from gmock_main.cc 2021-07-31T02:01:40.454 INFO:tasks.workunit.client.0.smithi045.stdout:[==========] Running 11 tests from 1 test suite. 2021-07-31T02:01:40.454 INFO:tasks.workunit.client.0.smithi045.stdout:[----------] Global test environment set-up. 2021-07-31T02:01:40.454 INFO:tasks.workunit.client.0.smithi045.stdout:[----------] 11 tests from ClsLock 2021-07-31T02:01:40.454 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] ClsLock.TestMultiLocking 2021-07-31T02:01:40.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:40 smithi154 conmon[34553]: audit 2021-07-31T02:01:39.292320+0000 mon.a (mon.0) 1677 : audit [INF] 2021-07-31T02:01:40.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:40 smithi154 conmon[34553]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73746-1","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:01:40.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:40 smithi154 conmon[34553]: cluster 2021-07-31T02:01:39.292636+0000 mon.a (mon.0) 1678 : cluster [DBG] osdmap e923: 8 total, 8 up, 8 in 2021-07-31T02:01:40.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:40 smithi154 conmon[34553]: cluster 2021-07-31T02:01:39.997183 2021-07-31T02:01:40.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:40 smithi154 conmon[34553]: +0000 mgr.x (mgr.24542) 621 : cluster [DBG] pgmap v1203: 41 pgs: 32 creating+peering, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:01:40.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:40 smithi045 conmon[33655]: audit 2021-07-31T02:01:39.292320+0000 mon.a (mon.0) 1677 2021-07-31T02:01:40.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:40 smithi045 conmon[33655]: : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73746-1","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:01:40.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:40 smithi045 conmon[33655]: cluster 2021-07-31T02:01:39.292636+0000 mon.a (mon.0) 1678 : cluster [DBG] osdmap e923: 8 total, 8 up, 8 in 2021-07-31T02:01:40.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:40 smithi045 conmon[33655]: cluster 2021-07-31T02:01:39.997183 2021-07-31T02:01:40.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:40 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 621 : cluster [DBG] pgmap v1203: 41 pgs: 32 creating+peering, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:01:40.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:40 smithi045 conmon[39718]: audit 2021-07-31T02:01:39.292320+0000 mon.a (mon. 2021-07-31T02:01:40.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:40 smithi045 conmon[39718]: 0) 1677 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73746-1","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:01:40.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:40 smithi045 conmon[39718]: cluster 2021-07-31T02:01:39 2021-07-31T02:01:40.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:40 smithi045 conmon[39718]: .292636+0000 mon.a (mon.0) 1678 : cluster [DBG] osdmap e923: 8 total, 8 up, 8 in 2021-07-31T02:01:40.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:40 smithi045 conmon[39718]: cluster 2021 2021-07-31T02:01:40.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:40 smithi045 conmon[39718]: -07-31T02:01:39.997183+0000 mgr.x ( 2021-07-31T02:01:40.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:40 smithi045 conmon[39718]: mgr.24542) 621 : cluster [DBG] pgmap v1203: 41 pgs: 32 creating+peering, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:01:41.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:41 smithi154 conmon[34553]: cluster 2021-07-31T02:01:40.305820+0000 mon.a (mon.0) 1679 : cluster [DBG] osdmap e924: 8 total, 8 up, 8 in 2021-07-31T02:01:41.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:41 smithi045 conmon[33655]: cluster 2021-07-31T02 2021-07-31T02:01:41.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:41 smithi045 conmon[33655]: :01:40.305820+0000 mon.a (mon.0) 1679 : cluster [DBG] osdmap e924: 8 total, 8 up, 8 in 2021-07-31T02:01:41.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:41 smithi045 conmon[39718]: cluster 2021-07-31T02:01:40.305820+0000 mon.a 2021-07-31T02:01:41.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:41 smithi045 conmon[39718]: (mon.0) 1679 : cluster [DBG] osdmap e924: 8 total, 8 up, 8 in 2021-07-31T02:01:42.353 INFO:tasks.workunit.client.0.smithi045.stdout:lock: mylock 2021-07-31T02:01:42.353 INFO:tasks.workunit.client.0.smithi045.stdout: lock_type: exclusive 2021-07-31T02:01:42.354 INFO:tasks.workunit.client.0.smithi045.stdout: tag: 2021-07-31T02:01:42.354 INFO:tasks.workunit.client.0.smithi045.stdout: lockers: 2021-07-31T02:01:42.354 INFO:tasks.workunit.client.0.smithi045.stdout: client.25321 expiration=2021-07-31T02:03:42.349940+0000 addr=v1:172.21.15.45:0/2470353504 cookie= 2021-07-31T02:01:42.357 INFO:tasks.workunit.client.0.smithi045.stdout:lock: mylock 2021-07-31T02:01:42.357 INFO:tasks.workunit.client.0.smithi045.stdout: lock_type: shared 2021-07-31T02:01:42.357 INFO:tasks.workunit.client.0.smithi045.stdout: tag: 2021-07-31T02:01:42.358 INFO:tasks.workunit.client.0.smithi045.stdout: lockers: 2021-07-31T02:01:42.358 INFO:tasks.workunit.client.0.smithi045.stdout: client.25321 expiration=2021-07-31T02:03:42.356152+0000 addr=v1:172.21.15.45:0/2470353504 cookie= 2021-07-31T02:01:42.358 INFO:tasks.workunit.client.0.smithi045.stdout: client.25327 expiration=0.000000 addr=v1:172.21.15.45:0/243005555 cookie= 2021-07-31T02:01:42.359 INFO:tasks.workunit.client.0.smithi045.stdout:lock: mylock 2021-07-31T02:01:42.359 INFO:tasks.workunit.client.0.smithi045.stdout: lock_type: shared 2021-07-31T02:01:42.359 INFO:tasks.workunit.client.0.smithi045.stdout: tag: 2021-07-31T02:01:42.359 INFO:tasks.workunit.client.0.smithi045.stdout: lockers: 2021-07-31T02:01:42.360 INFO:tasks.workunit.client.0.smithi045.stdout: client.25327 expiration=0.000000 addr=v1:172.21.15.45:0/243005555 cookie= 2021-07-31T02:01:42.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:42 smithi154 conmon[34553]: cluster 2021-07-31T02:01:42.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:42 smithi154 conmon[34553]: 2021-07-31T02:01:42.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:42 smithi154 conmon[34553]: 2021 2021-07-31T02:01:42.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:42 smithi154 conmon[34553]: - 2021-07-31T02:01:42.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:42 smithi154 conmon[34553]: 07-31T02:01:41 2021-07-31T02:01:42.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:42 smithi154 conmon[34553]: .310984+0000 mon.a (mon.0) 1680 : cluster [DBG] osdmap e925: 8 total, 8 up, 8 in 2021-07-31T02:01:42.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:42 smithi154 conmon[34553]: audit 2021-07 2021-07-31T02:01:42.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:42 smithi154 conmon[34553]: -31T02:01:41.312434+0000 mon.c (mon.1) 54 : audit 2021-07-31T02:01:42.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:42 smithi154 conmon[34553]: [INF] from='client.? 172.21.15.45:0/2470353504' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:01:42.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:42 smithi154 conmon[34553]: audit 2021-07-31T 2021-07-31T02:01:42.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:42 smithi154 conmon[34553]: 02:01:41.312889+0000 mon.a (mon.0 2021-07-31T02:01:42.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:42 smithi154 conmon[34553]: ) 1681 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:01:42.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:42 smithi154 conmon[34553]: cluster 2021-07- 2021-07-31T02:01:42.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:42 smithi154 conmon[34553]: 31T02:01:41.997672+0000 mgr.x (mgr 2021-07-31T02:01:42.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:42 smithi154 conmon[34553]: .24542) 622 : cluster [DBG] pgmap v1206: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:01:42.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:42 smithi045 conmon[33655]: cluster 2021-07-31T02:01:41.310984+0000 mon.a (mon.0) 1680 : cluster [DBG] osdmap e925: 8 total, 8 up, 8 in 2021-07-31T02:01:42.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:42 smithi045 conmon[33655]: audit 2021-07-31T02:01:41.312434+0000 mon.c (mon.1) 54 : audit [INF] from='client.? 172.21.15.45:0/2470353504' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:01:42.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:42 smithi045 conmon[33655]: audit 2021-07-31T02:01:41.312889+0000 mon.a (mon.0) 1681 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:01:42.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:42 smithi045 conmon[33655]: cluster 2021-07-31T02:01:41.997672+0000 mgr.x (mgr.24542) 622 : cluster [DBG] pgmap v1206: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:01:42.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:42 smithi045 conmon[39718]: cluster 2021-07-31T02:01:42.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:42 smithi045 conmon[39718]: 2021- 2021-07-31T02:01:42.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:42 smithi045 conmon[39718]: 07-31T02:01:41.310984+0000 mon.a (mon.0) 1680 : cluster [DBG] osdmap e925: 8 total, 8 up, 8 in 2021-07-31T02:01:42.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:42 smithi045 conmon[39718]: audit 2021-07-31T02:01:42.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:42 smithi045 conmon[39718]: 2021- 2021-07-31T02:01:42.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:42 smithi045 conmon[39718]: 07-31T02:01:41.312434+0000 mon.c (mon.1) 54 : audit [INF] from='client.? 172.21.15.45:0/2470353504' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:01:42.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:42 smithi045 conmon[39718]: audit 2021-07-31T02:01:41.312889+0000 mon.a (mon.0) 1681 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:01:42.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:42 smithi045 conmon[39718]: cluster 2021-07-31T02:01:41.997672+0000 mgr.x (mgr.24542) 622 : cluster [DBG] pgmap v1206: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:01:43.331 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] ClsLock.TestMultiLocking (2878 ms) 2021-07-31T02:01:43.331 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] ClsLock.TestMeta 2021-07-31T02:01:43.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:43 smithi154 conmon[34553]: audit 2021-07-31T02:01:42.313797+0000 mon.a (mon.0) 1682 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-1","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:01:43.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:43 smithi154 conmon[34553]: cluster 2021-07-31T02 2021-07-31T02:01:43.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:43 smithi154 conmon[34553]: :01:42.314113+0000 mon.a (mon.0) 1683 : cluster [DBG] osdmap e926: 8 total, 8 up, 8 in 2021-07-31T02:01:43.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:43 smithi045 conmon[33655]: audit 2021-07-31T02:01:42.313797+0000 mon.a (mon.0) 1682 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-1","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:01:43.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:43 smithi045 conmon[33655]: cluster 2021-07-31T02:01:43.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:43 smithi045 conmon[33655]: 2021-07-31T02:01:42.314113+0000 mon.a (mon.0) 1683 : cluster [DBG] osdmap e926: 8 total, 8 up, 8 in 2021-07-31T02:01:43.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:01:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:01:43.533Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:01:43.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:01:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:01:43.533Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:01:43.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:01:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:01:43.533Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:01:43.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:43 smithi045 conmon[39718]: audit 2021-07-31T02:01:42.313797+0000 mon.a (mon. 2021-07-31T02:01:43.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:43 smithi045 conmon[39718]: 0) 1682 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-1","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:01:43.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:43 smithi045 conmon[39718]: cluster 2021-07-31T02:01:42.314113+0000 2021-07-31T02:01:43.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:43 smithi045 conmon[39718]: mon.a (mon.0) 1683 : cluster [DBG] osdmap e926: 8 total, 8 up, 8 in 2021-07-31T02:01:44.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:44 smithi154 conmon[34553]: cluster 2021-07-31T02:01:43.327447+0000 mon.a (mon.0) 1684 2021-07-31T02:01:44.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:44 smithi154 conmon[34553]: : cluster [DBG] osdmap e927: 8 total, 8 up, 8 in 2021-07-31T02:01:44.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:44 smithi154 conmon[34553]: cluster 2021-07-31T02:01:43.998124+0000 mgr.x (mgr.24542) 623 : cluster [DBG] pgmap v1209: 9 pgs: 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:01:44.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:44 smithi045 conmon[33655]: cluster 2021-07-31T02:01:43. 2021-07-31T02:01:44.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:44 smithi045 conmon[33655]: 327447+0000 mon.a (mon.0) 1684 : cluster [DBG] osdmap e927: 8 total, 8 up, 8 in 2021-07-31T02:01:44.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:44 smithi045 conmon[33655]: cluster 2021-07-31T02:01:43.998124+0000 mgr.x 2021-07-31T02:01:44.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:44 smithi045 conmon[33655]: (mgr.24542) 623 : cluster [DBG] pgmap v1209: 9 pgs: 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:01:44.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:44 smithi045 conmon[39718]: cluster 2021-07-31T02:01:43.327447+0000 2021-07-31T02:01:44.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:44 smithi045 conmon[39718]: mon.a (mon.0) 1684 : cluster [DBG] osdmap e927: 8 total, 8 up, 8 in 2021-07-31T02:01:44.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:44 smithi045 conmon[39718]: cluster 2021-07-31T02:01:44.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:44 smithi045 conmon[39718]: 2021-07-31T02:01:43.998124+0000 mgr.x (mgr.24542) 2021-07-31T02:01:44.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:44 smithi045 conmon[39718]: 623 : cluster [DBG] pgmap v1209: 9 pgs: 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:01:45.369 INFO:tasks.workunit.client.0.smithi045.stdout:lock: mylock 2021-07-31T02:01:45.370 INFO:tasks.workunit.client.0.smithi045.stdout: lock_type: shared 2021-07-31T02:01:45.371 INFO:tasks.workunit.client.0.smithi045.stdout: tag: 2021-07-31T02:01:45.371 INFO:tasks.workunit.client.0.smithi045.stdout: lockers: 2021-07-31T02:01:45.371 INFO:tasks.workunit.client.0.smithi045.stdout: client.25339 expiration=0.000000 addr=v1:172.21.15.45:0/372326297 cookie= 2021-07-31T02:01:45.371 INFO:tasks.workunit.client.0.smithi045.stdout:lock: mylock 2021-07-31T02:01:45.372 INFO:tasks.workunit.client.0.smithi045.stdout: lock_type: exclusive 2021-07-31T02:01:45.372 INFO:tasks.workunit.client.0.smithi045.stdout: tag: new_tag 2021-07-31T02:01:45.373 INFO:tasks.workunit.client.0.smithi045.stdout: lockers: 2021-07-31T02:01:45.373 INFO:tasks.workunit.client.0.smithi045.stdout: client.25333 expiration=0.000000 addr=v1:172.21.15.45:0/1076340444 cookie= 2021-07-31T02:01:45.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:45 smithi154 conmon[34553]: cluster 2021-07-31T02:01:45.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:45 smithi154 conmon[34553]: 2021-07-31T02:01:45.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:45 smithi154 conmon[34553]: 2021 2021-07-31T02:01:45.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:45 smithi154 conmon[34553]: - 2021-07-31T02:01:45.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:45 smithi154 conmon[34553]: 07 2021-07-31T02:01:45.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:45 smithi154 conmon[34553]: - 2021-07-31T02:01:45.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:45 smithi154 conmon[34553]: 31 2021-07-31T02:01:45.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:45 smithi154 conmon[34553]: T 2021-07-31T02:01:45.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:45 smithi154 conmon[34553]: 02 2021-07-31T02:01:45.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:45 smithi154 conmon[34553]: : 2021-07-31T02:01:45.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:45 smithi154 conmon[34553]: 01 2021-07-31T02:01:45.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:45 smithi154 conmon[34553]: : 2021-07-31T02:01:45.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:45 smithi154 conmon[34553]: 44 2021-07-31T02:01:45.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:45 smithi154 conmon[34553]: . 2021-07-31T02:01:45.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:45 smithi154 conmon[34553]: 332956 2021-07-31T02:01:45.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:45 smithi154 conmon[34553]: +0000 2021-07-31T02:01:45.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:45 smithi154 conmon[34553]: 2021-07-31T02:01:45.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:45 smithi154 conmon[34553]: mon.a 2021-07-31T02:01:45.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:45 smithi154 conmon[34553]: ( 2021-07-31T02:01:45.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:45 smithi154 conmon[34553]: mon 2021-07-31T02:01:45.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:45 smithi154 conmon[34553]: . 2021-07-31T02:01:45.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:45 smithi154 conmon[34553]: 0 2021-07-31T02:01:45.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:45 smithi154 conmon[34553]: ) 2021-07-31T02:01:45.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:45 smithi154 conmon[34553]: 1685 2021-07-31T02:01:45.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:45 smithi154 conmon[34553]: : 2021-07-31T02:01:45.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:45 smithi154 conmon[34553]: cluster 2021-07-31T02:01:45.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:45 smithi154 conmon[34553]: 2021-07-31T02:01:45.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:45 smithi154 conmon[34553]: [DBG] 2021-07-31T02:01:45.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:45 smithi154 conmon[34553]: 2021-07-31T02:01:45.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:45 smithi154 conmon[34553]: osdmap e928: 8 total, 8 up, 8 in 2021-07-31T02:01:45.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:45 smithi154 conmon[34553]: 2021-07-31T02:01:45.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:45 smithi154 conmon[34553]: audit 2021-07-31T02:01:45.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:45 smithi154 conmon[34553]: 2021-07-31T02:01:44. 2021-07-31T02:01:45.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:45 smithi154 conmon[34553]: 335075+0000 mon.c (mon.1) 55 : audit [INF] from='client.? 172.21.15.45:0/1076340444' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:01:45.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:45 smithi154 conmon[34553]: audit 2021-07-31T02:01:44. 2021-07-31T02:01:45.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:45 smithi154 conmon[34553]: 335711+0000 mon.a (mon.0) 1686 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:01:45.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:45 smithi045 conmon[33655]: cluster 2021-07-31T02:01:44.332956+0000 2021-07-31T02:01:45.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:45 smithi045 conmon[33655]: mon.a (mon.0) 1685 : cluster [DBG] osdmap e928: 8 total, 8 up, 8 in 2021-07-31T02:01:45.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:45 smithi045 conmon[33655]: audit 2021-07-31T02:01:44.335075+0000 mon.c (mon.1) 55 : audit [INF] from='client.? 172.21.15.45:0/1076340444' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:01:45.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:45 smithi045 conmon[33655]: audit 2021-07-31T02:01:44.335711+0000 mon.a (mon.0) 1686 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:01:45.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:45 smithi045 conmon[39718]: cluster 2021-07-31T02:01:45.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:45 smithi045 conmon[39718]: 2021-07-31T02:01:45.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:45 smithi045 conmon[39718]: 2021 2021-07-31T02:01:45.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:45 smithi045 conmon[39718]: - 2021-07-31T02:01:45.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:45 smithi045 conmon[39718]: 07 2021-07-31T02:01:45.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:45 smithi045 conmon[39718]: - 2021-07-31T02:01:45.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:45 smithi045 conmon[39718]: 31 2021-07-31T02:01:45.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:45 smithi045 conmon[39718]: T 2021-07-31T02:01:45.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:45 smithi045 conmon[39718]: 02 2021-07-31T02:01:45.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:45 smithi045 conmon[39718]: : 2021-07-31T02:01:45.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:45 smithi045 conmon[39718]: 01 2021-07-31T02:01:45.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:45 smithi045 conmon[39718]: : 2021-07-31T02:01:45.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:45 smithi045 conmon[39718]: 44 2021-07-31T02:01:45.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:45 smithi045 conmon[39718]: . 2021-07-31T02:01:45.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:45 smithi045 conmon[39718]: 332956 2021-07-31T02:01:45.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:45 smithi045 conmon[39718]: +0000 2021-07-31T02:01:45.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:45 smithi045 conmon[39718]: 2021-07-31T02:01:45.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:45 smithi045 conmon[39718]: mon.a 2021-07-31T02:01:45.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:45 smithi045 conmon[39718]: ( 2021-07-31T02:01:45.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:45 smithi045 conmon[39718]: mon 2021-07-31T02:01:45.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:45 smithi045 conmon[39718]: . 2021-07-31T02:01:45.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:45 smithi045 conmon[39718]: 0 2021-07-31T02:01:45.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:45 smithi045 conmon[39718]: ) 2021-07-31T02:01:45.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:45 smithi045 conmon[39718]: 1685 2021-07-31T02:01:45.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:45 smithi045 conmon[39718]: : 2021-07-31T02:01:45.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:45 smithi045 conmon[39718]: cluster 2021-07-31T02:01:45.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:45 smithi045 conmon[39718]: 2021-07-31T02:01:45.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:45 smithi045 conmon[39718]: [DBG] 2021-07-31T02:01:45.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:45 smithi045 conmon[39718]: 2021-07-31T02:01:45.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:45 smithi045 conmon[39718]: osdmap e928: 8 total, 8 up, 8 in 2021-07-31T02:01:45.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:45 smithi045 conmon[39718]: 2021-07-31T02:01:45.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:45 smithi045 conmon[39718]: audit 2021-07-31T02:01:45.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:45 smithi045 conmon[39718]: 2021-07-31T02:01:45.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:45 smithi045 conmon[39718]: 2021 2021-07-31T02:01:45.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:45 smithi045 conmon[39718]: - 2021-07-31T02:01:45.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:45 smithi045 conmon[39718]: 07 2021-07-31T02:01:45.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:45 smithi045 conmon[39718]: - 2021-07-31T02:01:45.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:45 smithi045 conmon[39718]: 31 2021-07-31T02:01:45.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:45 smithi045 conmon[39718]: T 2021-07-31T02:01:45.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:45 smithi045 conmon[39718]: 02 2021-07-31T02:01:45.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:45 smithi045 conmon[39718]: : 2021-07-31T02:01:45.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:45 smithi045 conmon[39718]: 01 2021-07-31T02:01:45.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:45 smithi045 conmon[39718]: : 2021-07-31T02:01:45.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:45 smithi045 conmon[39718]: 44 2021-07-31T02:01:45.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:45 smithi045 conmon[39718]: .335075+0000 mon.c (mon.1) 55 : audit 2021-07-31T02:01:45.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:45 smithi045 conmon[39718]: [INF] from='client.? 172.21.15.45:0/1076340444' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:01:45.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:45 smithi045 conmon[39718]: audit 2021-07 2021-07-31T02:01:45.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:45 smithi045 conmon[39718]: -31T02:01:44.335711+0000 mon.a (mon.0) 1686 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:01:45.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:45 smithi045 conmon[39718]: 2021-07-31T02:01:46.353 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] ClsLock.TestMeta (3022 ms) 2021-07-31T02:01:46.354 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] ClsLock.TestCookie 2021-07-31T02:01:46.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:46 smithi154 conmon[34553]: audit 2021-07-31T02:01:45.334997+0000 mon.a (mon.0) 1687 : audit [INF] 2021-07-31T02:01:46.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:46 smithi154 conmon[34553]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-2","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:01:46.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:46 smithi154 conmon[34553]: cluster 2021-07-31T02:01:45.335305+0000 mon.a (mon.0) 1688 : cluster [DBG] osdmap e929: 8 total, 8 up, 8 in 2021-07-31T02:01:46.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:46 smithi154 conmon[34553]: cluster 2021-07-31T02:01:45.998636+0000 mgr.x (mgr.24542) 624 : 2021-07-31T02:01:46.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:46 smithi154 conmon[34553]: cluster [DBG] pgmap v1212: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:01:46.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:46 smithi045 conmon[33655]: audit 2021-07-31T02:01:45.334997+0000 mon.a (mon.0) 1687 : audit 2021-07-31T02:01:46.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:46 smithi045 conmon[33655]: [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-2","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:01:46.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:46 smithi045 conmon[33655]: cluster 2021-07-31T02:01:45.335305+0000 mon.a (mon.0) 1688 : cluster [DBG] osdmap e929: 8 total, 8 up, 8 in 2021-07-31T02:01:46.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:46 smithi045 conmon[33655]: cluster 2021-07-31T02:01:45.998636+0000 mgr.x 2021-07-31T02:01:46.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:46 smithi045 conmon[33655]: (mgr.24542) 624 : cluster [DBG] pgmap v1212: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:01:46.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:46 smithi045 conmon[39718]: audit 2021-07-31T02:01:45.334997+0000 mon.a (mon 2021-07-31T02:01:46.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:46 smithi045 conmon[39718]: .0) 1687 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-2","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:01:46.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:46 smithi045 conmon[39718]: cluster 2021-07-31T02:01 2021-07-31T02:01:46.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:46 smithi045 conmon[39718]: :45.335305+0000 mon.a (mon.0) 1688 : cluster [DBG] osdmap e929: 8 total, 8 up, 8 in 2021-07-31T02:01:46.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:46 smithi045 conmon[39718]: cluster 2021-07-31T02:01:46.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:46 smithi045 conmon[39718]: 2021-07-31T 2021-07-31T02:01:46.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:46 smithi045 conmon[39718]: 02:01:45.998636+0000 mgr.x (mgr.24542) 624 : cluster [DBG] pgmap v1212: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:01:46.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:46 smithi045 conmon[39718]: 2021-07-31T02:01:47.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:47 smithi154 conmon[34553]: cluster 2021-07-31T02:01:46.349274+0000 mon.a (mon.0) 1689 : cluster 2021-07-31T02:01:47.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:47 smithi154 conmon[34553]: [DBG] osdmap e930: 8 total, 8 up, 8 in 2021-07-31T02:01:47.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:47 smithi045 conmon[39718]: cluster 2021-07-31T02:01:46.349274+0000 mon.a (mon.0) 1689 : cluster [DBG] osdmap e930: 8 total, 8 up, 8 in 2021-07-31T02:01:47.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:47 smithi045 conmon[33655]: cluster 2021-07-31T02:01:46.349274+0000 mon.a (mon.0) 1689 : 2021-07-31T02:01:47.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:47 smithi045 conmon[33655]: cluster [DBG] osdmap e930: 8 total, 8 up, 8 in 2021-07-31T02:01:48.583 INFO:tasks.workunit.client.0.smithi045.stdout:lock: mylock 2021-07-31T02:01:48.583 INFO:tasks.workunit.client.0.smithi045.stdout: lock_type: exclusive 2021-07-31T02:01:48.583 INFO:tasks.workunit.client.0.smithi045.stdout: tag: 2021-07-31T02:01:48.584 INFO:tasks.workunit.client.0.smithi045.stdout: lockers: 2021-07-31T02:01:48.585 INFO:tasks.workunit.client.0.smithi045.stdout:lock: mylock 2021-07-31T02:01:48.585 INFO:tasks.workunit.client.0.smithi045.stdout: lock_type: shared 2021-07-31T02:01:48.585 INFO:tasks.workunit.client.0.smithi045.stdout: tag: 2021-07-31T02:01:48.586 INFO:tasks.workunit.client.0.smithi045.stdout: lockers: 2021-07-31T02:01:48.586 INFO:tasks.workunit.client.0.smithi045.stdout: client.25345 expiration=0.000000 addr=v1:172.21.15.45:0/2092169417 cookie= 2021-07-31T02:01:48.586 INFO:tasks.workunit.client.0.smithi045.stdout: client.25345 expiration=0.000000 addr=v1:172.21.15.45:0/2092169417 cookie=new cookie 2021-07-31T02:01:48.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:48 smithi154 conmon[34553]: cluster 2021-07-31T02:01:47.354994+0000 mon.a (mon.0) 1690 : cluster [DBG] osdmap e931: 8 total, 8 up, 8 in 2021-07-31T02:01:48.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:48 smithi154 conmon[34553]: 2021-07-31T02:01:48.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:48 smithi154 conmon[34553]: audit 2021-07-31T02:01:47.355997+0000 mon.c (mon.1) 56 : audit [INF] from='client.? 172.21.15.45:0/2092169417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:01:48.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:48 smithi154 conmon[34553]: audit 2021-07-31T02:01:47.358507+0000 mon.a (mon.0) 1691 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:01:48.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:48 smithi154 conmon[34553]: cluster 2021-07-31T02:01:47.999097+0000 mgr.x (mgr.24542) 625 : cluster [DBG] pgmap v1215: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:01:48.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:48 smithi045 conmon[39718]: cluster 2021-07-31T02:01:47.354994+0000 mon.a 2021-07-31T02:01:48.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:48 smithi045 conmon[39718]: (mon.0) 1690 : cluster [DBG] osdmap e931: 8 total, 8 up, 8 in 2021-07-31T02:01:48.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:48 smithi045 conmon[39718]: audit 2021-07-31T02:01:47 2021-07-31T02:01:48.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:48 smithi045 conmon[39718]: .355997+0000 mon.c (mon.1) 56 : audit [INF] from='client.? 172.21.15.45:0/2092169417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:01:48.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:48 smithi045 conmon[39718]: audit 2021- 2021-07-31T02:01:48.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:48 smithi045 conmon[39718]: 07-31T02:01:47.358507+0000 mon.a (mon. 2021-07-31T02:01:48.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:48 smithi045 conmon[39718]: 0) 1691 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:01:48.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:48 smithi045 conmon[39718]: cluster 2021-07-31T02:01:47.999097+0000 mgr.x ( 2021-07-31T02:01:48.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:48 smithi045 conmon[39718]: mgr.24542) 625 : cluster [DBG] pgmap v1215: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:01:48.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:48 smithi045 conmon[33655]: cluster 2021-07-31T02:01:47.354994+0000 mon.a (mon.0) 1690 : cluster [DBG] 2021-07-31T02:01:48.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:48 smithi045 conmon[33655]: osdmap e931: 8 total, 8 up, 8 in 2021-07-31T02:01:48.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:48 smithi045 conmon[33655]: audit 2021-07-31T02:01:47.355997+0000 mon.c (mon.1) 56 : audit [INF] from='client.? 172.21.15.45:0/2092169417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:01:48.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:48 smithi045 conmon[33655]: audit 2021-07-31T02:01:47.358507+0000 mon.a (mon.0) 1691 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:01:48.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:48 smithi045 conmon[33655]: 2021-07-31T02:01:48.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:48 smithi045 conmon[33655]: cluster 2021-07-31T02:01:47.999097+0000 mgr.x (mgr 2021-07-31T02:01:48.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:48 smithi045 conmon[33655]: .24542) 625 : cluster [DBG] pgmap v1215: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:01:49.365 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] ClsLock.TestCookie (3012 ms) 2021-07-31T02:01:49.366 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] ClsLock.TestMultipleLocks 2021-07-31T02:01:49.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:49 smithi154 conmon[34553]: audit 2021-07-31T02:01:48.361460+0000 mon.a (mon.0) 1692 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-3","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:01:49.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:49 smithi154 conmon[34553]: cluster 2021-07-31T02:01:48.361683+0000 mon.a (mon.0) 1693 : cluster [DBG] osdmap e932: 8 total, 8 up, 8 in 2021-07-31T02:01:49.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:49 smithi045 conmon[39718]: audit 2021-07-31T02:01:48.361460+0000 mon.a (mon.0) 1692 : 2021-07-31T02:01:49.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:49 smithi045 conmon[39718]: audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-3","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:01:49.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:49 smithi045 conmon[39718]: cluster 2021-07-31T02:01:48.361683+0000 mon.a (mon.0) 1693 : cluster [DBG] osdmap e932: 8 total, 8 up, 8 in 2021-07-31T02:01:49.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:49 smithi045 conmon[33655]: audit 2021-07-31T02:01:48.361460+0000 mon.a (mon.0 2021-07-31T02:01:49.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:49 smithi045 conmon[33655]: ) 1692 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-3","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:01:49.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:49 smithi045 conmon[33655]: cluster 2021-07-31T02:01:48.361683+0000 mon.a (mon.0) 1693 : cluster [DBG] osdmap e932: 8 total, 8 up, 8 in 2021-07-31T02:01:50.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:50 smithi154 conmon[34553]: cluster 2021-07-31T02:01:49.363172+0000 mon.a (mon.0) 1694 : cluster 2021-07-31T02:01:50.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:50 smithi154 conmon[34553]: [DBG] osdmap e933: 8 total, 8 up, 8 in 2021-07-31T02:01:50.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:50 smithi154 conmon[34553]: cluster 2021-07-31T02:01:49.999465+0000 mgr.x (mgr.24542) 626 : cluster [DBG] pgmap v1218: 9 pgs: 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:01:50.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:50 smithi045 conmon[33655]: cluster 2021-07-31T02:01:49.363172+0000 mon.a (mon.0) 1694 : cluster 2021-07-31T02:01:50.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:50 smithi045 conmon[33655]: [DBG] osdmap e933: 8 total, 8 up, 8 in 2021-07-31T02:01:50.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:50 smithi045 conmon[33655]: cluster 2021-07-31T02:01:49.999465+0000 mgr.x (mgr.24542) 626 : cluster [DBG] pgmap v1218: 9 pgs: 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:01:50.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:50 smithi045 conmon[39718]: cluster 2021-07-31T02:01:49.363172+0000 mon.a (mon.0) 1694 : 2021-07-31T02:01:50.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:50 smithi045 conmon[39718]: cluster [DBG] osdmap e933: 8 total, 8 up, 8 in 2021-07-31T02:01:50.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:50 smithi045 conmon[39718]: cluster 2021-07-31T02:01:49.999465+0000 mgr.x (mgr.24542) 626 : cluster [DBG] pgmap v1218: 9 pgs: 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:01:51.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:51 smithi154 conmon[34553]: cluster 2021-07-31T02:01:51.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:51 smithi154 conmon[34553]: 2021-07-31T02:01:50.374697+0000 2021-07-31T02:01:51.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:51 smithi154 conmon[34553]: mon.a (mon.0) 1695 : cluster [DBG] osdmap e934: 8 total, 8 up, 8 in 2021-07-31T02:01:51.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:51 smithi154 conmon[34553]: audit 2021-07-31T02:01:51.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:51 smithi154 conmon[34553]: 2021-07-31T02:01:50.377438+0000 mon.a (mon.0) 1696 : audit [INF] from='client.? 172.21.15.45:0/2451151900' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:01:51.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:51 smithi045 conmon[33655]: cluster 2021-07-31T02:01:51.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:51 smithi045 conmon[33655]: 2021-07-31T 2021-07-31T02:01:51.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:51 smithi045 conmon[33655]: 02:01:50.374697+0000 mon.a (mon.0) 1695 : cluster [DBG] osdmap e934: 8 total, 8 up, 8 in 2021-07-31T02:01:51.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:51 smithi045 conmon[33655]: audit 2021-07-31 2021-07-31T02:01:51.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:51 smithi045 conmon[33655]: T02:01:50.377438+0000 mon.a (mon.0) 1696 : audit [INF] from='client.? 172.21.15.45:0/2451151900' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:01:51.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:51 smithi045 conmon[33655]: 2021-07-31T02:01:51.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:51 smithi045 conmon[39718]: cluster 2021-07-31T02:01:51.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:51 smithi045 conmon[39718]: 2021-07-31T02:01:51.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:51 smithi045 conmon[39718]: 2021-07-31T02:01:50.374697+0000 mon.a 2021-07-31T02:01:51.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:51 smithi045 conmon[39718]: (mon.0) 1695 : cluster [DBG] osdmap e934: 8 total, 8 up, 8 in 2021-07-31T02:01:51.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:51 smithi045 conmon[39718]: audit 2021-07-31T02:01:51.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:51 smithi045 conmon[39718]: 2021-07-31T02:01:51.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:51 smithi045 conmon[39718]: 2021 2021-07-31T02:01:51.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:51 smithi045 conmon[39718]: -07-31T02:01:50. 2021-07-31T02:01:51.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:51 smithi045 conmon[39718]: 377438+0000 mon.a (mon.0) 1696 : audit [INF] from='client.? 172.21.15.45:0/2451151900' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:01:51.969 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:01:52.245 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:01:52.245 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 16m ago 17m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:01:52.245 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 14m ago 17m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:01:52.245 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 14m ago 20m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:01:52.246 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 16m ago 22m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:01:52.246 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 16m ago 23m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:01:52.246 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 14m ago 21m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:01:52.246 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 16m ago 21m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:01:52.247 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 16m ago 17m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:01:52.247 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 14m ago 17m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:01:52.247 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 16m ago 20m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:01:52.247 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 16m ago 20m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:01:52.248 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 16m ago 19m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:01:52.248 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 16m ago 19m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:01:52.248 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 14m ago 19m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:01:52.248 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 14m ago 18m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:01:52.248 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 14m ago 18m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:01:52.249 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 14m ago 18m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:01:52.249 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 14m ago 17m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:01:52.387 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] ClsLock.TestMultipleLocks (3021 ms) 2021-07-31T02:01:52.388 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] ClsLock.TestLockDuration 2021-07-31T02:01:52.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:52 smithi154 conmon[34553]: audit 2021-07-31T02:01:51.377607+0000 mon.a (mon.0) 1697 : audit [INF] from='client.? 172.21.15.45:0/2451151900' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-4","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:01:52.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:52 smithi154 conmon[34553]: 2021-07-31T02:01:52.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:52 smithi154 conmon[34553]: cluster 2021-07-31T02:01:51.377904+0000 mon.a (mon.0) 1698 : cluster [DBG] osdmap e935: 8 total, 8 up, 8 in 2021-07-31T02:01:52.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:52 smithi154 conmon[34553]: audit 2021-07-31T02:01:51.957077+0000 mgr.x (mgr.24542) 627 : audit [DBG] from='client.25357 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:01:52.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:52 smithi154 conmon[34553]: cluster 2021-07-31T02:01:51.999970+0000 mgr.x (mgr.24542) 628 : cluster [DBG] pgmap v1221: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:01:52.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:52 smithi045 conmon[39718]: audit 2021-07-31T02:01:51.377607+0000 mon.a (mon.0) 1697 : audit [INF] from='client.? 172.21.15.45:0/2451151900' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-4","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:01:52.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:52 smithi045 conmon[39718]: cluster 2021-07-31T02:01:51.377904+0000 mon.a (mon.0) 1698 : cluster [DBG] osdmap e935: 8 total, 8 up, 8 in 2021-07-31T02:01:52.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:52 smithi045 conmon[39718]: audit 2021-07-31T02:01:51.957077+0000 mgr.x (mgr.24542) 627 : audit [DBG] from='client.25357 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:01:52.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:52 smithi045 conmon[39718]: cluster 2021-07-31T02:01:51.999970+0000 mgr.x (mgr.24542) 628 : cluster [DBG] pgmap v1221: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:01:52.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:52 smithi045 conmon[33655]: audit 2021-07-31T02:01:52.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:52 smithi045 conmon[33655]: 2021-07 2021-07-31T02:01:52.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:52 smithi045 conmon[33655]: -31T02:01:51.377607+0000 mon.a (mon.0) 1697 : audit [INF] from='client.? 172.21.15.45:0/2451151900' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-4","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:01:52.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:52 smithi045 conmon[33655]: cluster 2021-07-31T02:01:52.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:52 smithi045 conmon[33655]: 2021-07-31T02:01:51.377904+0000 mon.a (mon.0) 2021-07-31T02:01:52.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:52 smithi045 conmon[33655]: 1698 : cluster [DBG] osdmap e935: 8 total, 8 up, 8 in 2021-07-31T02:01:52.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:52 smithi045 conmon[33655]: audit 2021-07-31T02:01:52.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:52 smithi045 conmon[33655]: 2021-07 2021-07-31T02:01:52.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:52 smithi045 conmon[33655]: -31T02:01:51.957077+0000 2021-07-31T02:01:52.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:52 smithi045 conmon[33655]: mgr.x (mgr.24542) 627 : audit [DBG] from='client.25357 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:01:52.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:52 smithi045 conmon[33655]: 2021-07-31T02:01:52.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:52 smithi045 conmon[33655]: cluster 2021-07-31T02:01:52.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:52 smithi045 conmon[33655]: 2021-07-31T02:01:51.999970+0000 mgr.x ( 2021-07-31T02:01:52.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:52 smithi045 conmon[33655]: mgr.24542) 628 : cluster [DBG] pgmap v1221: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:01:52.846 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:01:52.847 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:01:52.847 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:01:52.848 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:01:52.848 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:01:52.848 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:01:52.848 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:01:52.849 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:01:52.849 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:01:52.850 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:01:52.850 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:01:52.850 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:01:52.850 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:01:52.851 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:01:52.851 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:01:52.851 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:01:52.851 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:01:53.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:53 smithi154 conmon[34553]: audit 2021-07-31T02:01:52.236535+0000 mgr.x (mgr.24542) 629 : audit [DBG] from='client.25363 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:01:53.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:53 smithi154 conmon[34553]: cluster 2021-07-31T02:01:52.386025+0000 mon.a (mon.0) 1699 : cluster [DBG] osdmap e936: 8 total, 8 up, 8 in 2021-07-31T02:01:53.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:53 smithi154 conmon[34553]: audit 2021-07-31T02:01:52.846295+0000 mon.a (mon.0) 1700 : audit [DBG] from='client.? 172.21.15.45:0/3714266777' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:01:53.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:01:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:01:53.533Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:01:53.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:01:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:01:53.533Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:01:53.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:01:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:01:53.533Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:01:53.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:53 smithi045 conmon[33655]: audit 2021-07-31T02:01:52.236535+0000 mgr.x (mgr. 2021-07-31T02:01:53.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:53 smithi045 conmon[33655]: 24542) 629 : audit [DBG] from='client.25363 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:01:53.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:53 smithi045 conmon[33655]: cluster 2021-07-31T02:01:52.386025+0000 mon.a (mon.0) 1699 : cluster [DBG] osdmap e936: 8 total, 8 up, 8 in 2021-07-31T02:01:53.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:53 smithi045 conmon[33655]: audit 2021-07-31T02:01:52.846295+0000 2021-07-31T02:01:53.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:53 smithi045 conmon[33655]: mon.a (mon.0) 1700 : audit [DBG] from='client.? 172.21.15.45:0/3714266777' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:01:53.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:53 smithi045 conmon[39718]: audit 2021-07-31T02:01:52.236535+0000 2021-07-31T02:01:53.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:53 smithi045 conmon[39718]: mgr.x (mgr.24542) 629 : audit [DBG] from='client.25363 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:01:53.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:53 smithi045 conmon[39718]: cluster 2021-07-31T02:01:53.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:53 smithi045 conmon[39718]: 2021-07-31T02:01:52.386025+0000 mon.a (mon.0) 1699 : cluster [DBG] 2021-07-31T02:01:53.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:53 smithi045 conmon[39718]: osdmap e936: 8 total, 8 up, 8 in 2021-07-31T02:01:53.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:53 smithi045 conmon[39718]: audit 2021-07-31T02:01:52 2021-07-31T02:01:53.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:53 smithi045 conmon[39718]: .846295+0000 mon.a (mon.0) 1700 : audit [DBG] 2021-07-31T02:01:53.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:53 smithi045 conmon[39718]: from='client.? 172.21.15.45:0/3714266777' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:01:54.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:54 smithi154 conmon[34553]: cluster 2021-07-31T02:01:53.397407+0000 mon.a (mon. 2021-07-31T02:01:54.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:54 smithi154 conmon[34553]: 0) 1701 : cluster [DBG] osdmap e937: 8 total, 8 up, 8 in 2021-07-31T02:01:54.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:54 smithi154 conmon[34553]: audit 2021-07-31T02:01:53.398557+0000 mon.c (mon.1) 57 : audit [INF] from='client.? 172.21.15.45:0/124206977' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:01:54.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:54 smithi154 conmon[34553]: audit 2021-07-31T02:01:53.399126+0000 mon.a (mon.0) 1702 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:01:54.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:54 smithi154 conmon[34553]: cluster 2021-07-31T02:01:54.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:54 smithi154 conmon[34553]: 2021-07-31T02:01:54.000443+0000 mgr.x ( 2021-07-31T02:01:54.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:54 smithi154 conmon[34553]: mgr.24542) 630 : cluster [DBG] pgmap v1224: 41 pgs: 32 creating+peering, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:01:54.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:54 smithi045 conmon[39718]: cluster 2021-07-31T02:01:54.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:54 smithi045 conmon[39718]: 2021-07-31T02:01:54.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:54 smithi045 conmon[39718]: 2021 2021-07-31T02:01:54.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:54 smithi045 conmon[39718]: -07-31T02:01:53.397407+0000 2021-07-31T02:01:54.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:54 smithi045 conmon[39718]: mon.a (mon.0) 1701 : cluster [DBG] osdmap e937: 8 total, 8 up, 8 in 2021-07-31T02:01:54.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:54 smithi045 conmon[39718]: audit 2021-07-31T02 2021-07-31T02:01:54.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:54 smithi045 conmon[39718]: :01:53.398557+0000 mon.c (mon.1) 2021-07-31T02:01:54.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:54 smithi045 conmon[39718]: 57 : audit [INF] from='client.? 172.21.15.45:0/124206977' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:01:54.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:54 smithi045 conmon[39718]: audit 2021 2021-07-31T02:01:54.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:54 smithi045 conmon[39718]: -07-31T02:01:53.399126+0000 2021-07-31T02:01:54.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:54 smithi045 conmon[39718]: mon.a (mon.0) 1702 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:01:54.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:54 smithi045 conmon[39718]: 2021-07-31T02:01:54.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:54 smithi045 conmon[39718]: cluster 2021-07-31T02:01:54.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:54 smithi045 conmon[39718]: 2021-07-31T02:01:54.000443+0000 mgr.x ( 2021-07-31T02:01:54.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:54 smithi045 conmon[39718]: mgr.24542) 630 : cluster [DBG] pgmap v1224: 41 pgs: 32 creating+peering, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:01:54.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:54 smithi045 conmon[33655]: cluster 2021-07-31T02:01:53.397407+0000 mon.a (mon.0) 1701 : cluster 2021-07-31T02:01:54.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:54 smithi045 conmon[33655]: [DBG] osdmap e937: 8 total, 8 up, 8 in 2021-07-31T02:01:54.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:54 smithi045 conmon[33655]: audit 2021-07-31T02:01:53.398557+0000 mon.c (mon.1) 57 : audit [INF] from='client.? 172.21.15.45:0/124206977' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:01:54.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:54 smithi045 conmon[33655]: audit 2021-07-31 2021-07-31T02:01:54.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:54 smithi045 conmon[33655]: T02:01:53.399126+0000 2021-07-31T02:01:54.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:54 smithi045 conmon[33655]: mon.a (mon.0) 1702 2021-07-31T02:01:54.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:54 smithi045 conmon[33655]: : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:01:54.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:54 smithi045 conmon[33655]: cluster 2021-07-31T02:01:54.000443 2021-07-31T02:01:54.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:54 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 630 : cluster [DBG] pgmap v1224: 41 pgs: 32 creating+peering, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:01:55.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:55 smithi154 conmon[34553]: audit 2021-07-31T02:01:54.403703+0000 mon.a (mon.0) 1703 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-5","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:01:55.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:55 smithi154 conmon[34553]: 2021-07-31T02:01:55.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:55 smithi154 conmon[34553]: cluster 2021-07-31T02:01:54.404016+0000 mon.a (mon.0) 1704 : cluster [DBG] osdmap e938: 8 total, 8 up, 8 in 2021-07-31T02:01:55.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:55 smithi045 conmon[33655]: audit 2021-07-31T02:01:54.403703+0000 mon.a (mon.0) 1703 : audit 2021-07-31T02:01:55.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:55 smithi045 conmon[33655]: [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-5","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:01:55.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:55 smithi045 conmon[33655]: cluster 2021-07-31T02:01:54.404016+0000 mon.a (mon.0) 1704 : cluster [DBG] osdmap e938: 8 total, 8 up, 8 in 2021-07-31T02:01:55.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:55 smithi045 conmon[39718]: audit 2021-07-31T02:01:54.403703+0000 mon.a (mon 2021-07-31T02:01:55.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:55 smithi045 conmon[39718]: .0) 1703 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-5","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:01:55.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:55 smithi045 conmon[39718]: cluster 2021-07-31T02:01:54.404016+0000 2021-07-31T02:01:55.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:55 smithi045 conmon[39718]: mon.a (mon.0) 1704 : cluster [DBG] osdmap e938: 8 total, 8 up, 8 in 2021-07-31T02:01:56.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:56 smithi154 conmon[34553]: cluster 2021-07-31T02:01:55.404807+0000 mon.a (mon.0) 1705 : cluster [DBG] osdmap e939: 8 total, 8 up, 8 in 2021-07-31T02:01:56.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:56 smithi154 conmon[34553]: cluster 2021-07-31T02:01:56.001058+0000 mgr.x (mgr.24542) 631 : cluster [DBG] pgmap v1227: 41 pgs: 32 creating+peering, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:01:56.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:56 smithi045 conmon[33655]: cluster 2021-07-31T02:01:55.404807+0000 mon.a (mon.0) 1705 : cluster [DBG] 2021-07-31T02:01:56.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:56 smithi045 conmon[33655]: osdmap e939: 8 total, 8 up, 8 in 2021-07-31T02:01:56.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:56 smithi045 conmon[33655]: cluster 2021-07-31T02:01:56.001058+0000 mgr.x (mgr.24542) 631 : cluster [DBG] pgmap v1227: 41 pgs: 32 creating+peering, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:01:56.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:56 smithi045 conmon[39718]: cluster 2021-07-31T02:01:55.404807+0000 mon.a (mon.0 2021-07-31T02:01:56.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:56 smithi045 conmon[39718]: ) 1705 : cluster [DBG] osdmap e939: 8 total, 8 up, 8 in 2021-07-31T02:01:56.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:56 smithi045 conmon[39718]: cluster 2021-07-31T02:01:56.001058+0000 mgr.x (mgr.24542) 631 : cluster [DBG] pgmap v1227: 41 pgs: 32 creating+peering, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:01:59.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:01:59 smithi154 conmon[34553]: cluster 2021-07-31T02:01:58.001445+0000 mgr.x (mgr.24542) 632 : cluster [DBG] pgmap v1228: 41 pgs: 32 creating+peering, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:01:59.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:59 smithi045 conmon[33655]: cluster 2021-07-31T02:01:58.001445+0000 mgr.x (mgr. 2021-07-31T02:01:59.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:01:59 smithi045 conmon[33655]: 24542) 632 : cluster [DBG] pgmap v1228: 41 pgs: 32 creating+peering, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:01:59.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:59 smithi045 conmon[39718]: cluster 2021-07-31T02:01:58.001445+0000 mgr.x 2021-07-31T02:01:59.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:01:59 smithi045 conmon[39718]: (mgr.24542) 632 : cluster [DBG] pgmap v1228: 41 pgs: 32 creating+peering, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:02:00.067 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] ClsLock.TestLockDuration (7680 ms) 2021-07-31T02:02:00.067 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] ClsLock.TestAssertLocked 2021-07-31T02:02:01.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:01 smithi154 conmon[34553]: cluster 2021-07-31T02:02:00.002047+0000 mgr.x (mgr.24542) 633 : cluster [DBG] pgmap v1229: 41 pgs: 41 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 0 op/s 2021-07-31T02:02:01.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:01 smithi154 conmon[34553]: 2021-07-31T02:02:01.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:01 smithi154 conmon[34553]: cluster 2021-07-31T02:02:00.064476+0000 mon.a (mon.0) 1706 : cluster [DBG] osdmap e940: 8 total, 8 up, 8 in 2021-07-31T02:02:01.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:01 smithi045 conmon[33655]: cluster 2021-07-31T02:02:00.002047 2021-07-31T02:02:01.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:01 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 633 : cluster [DBG] pgmap v1229: 41 pgs: 41 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 0 op/s 2021-07-31T02:02:01.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:01 smithi045 conmon[33655]: cluster 2021-07-31T02:02: 2021-07-31T02:02:01.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:01 smithi045 conmon[33655]: 00.064476+0000 mon.a (mon.0) 1706 : cluster [DBG] 2021-07-31T02:02:01.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:01 smithi045 conmon[33655]: osdmap e940: 8 total, 8 up, 8 in 2021-07-31T02:02:01.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:01 smithi045 conmon[39718]: cluster 2021-07-31T02:02:00.002047+0000 mgr.x (mgr.24542) 633 : cluster 2021-07-31T02:02:01.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:01 smithi045 conmon[39718]: [DBG] pgmap v1229: 41 pgs: 41 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 0 op/s 2021-07-31T02:02:01.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:01 smithi045 conmon[39718]: cluster 2021-07-31T02:02:00. 2021-07-31T02:02:01.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:01 smithi045 conmon[39718]: 064476+0000 mon.a (mon.0) 1706 : cluster [DBG] osdmap e940: 8 total, 8 up, 8 in 2021-07-31T02:02:02.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:02 smithi154 conmon[34553]: cluster 2021-07-31T02:02:02.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:02 smithi154 conmon[34553]: 2021-07-31T02:02:02.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:02 smithi154 conmon[34553]: 2021 2021-07-31T02:02:02.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:02 smithi154 conmon[34553]: - 2021-07-31T02:02:02.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:02 smithi154 conmon[34553]: 07 2021-07-31T02:02:02.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:02 smithi154 conmon[34553]: - 2021-07-31T02:02:02.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:02 smithi154 conmon[34553]: 31 2021-07-31T02:02:02.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:02 smithi154 conmon[34553]: T 2021-07-31T02:02:02.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:02 smithi154 conmon[34553]: 02 2021-07-31T02:02:02.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:02 smithi154 conmon[34553]: : 2021-07-31T02:02:02.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:02 smithi154 conmon[34553]: 02 2021-07-31T02:02:02.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:02 smithi154 conmon[34553]: : 2021-07-31T02:02:02.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:02 smithi154 conmon[34553]: 01 2021-07-31T02:02:02.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:02 smithi154 conmon[34553]: . 2021-07-31T02:02:02.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:02 smithi154 conmon[34553]: 070403 2021-07-31T02:02:02.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:02 smithi154 conmon[34553]: +0000 2021-07-31T02:02:02.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:02 smithi154 conmon[34553]: 2021-07-31T02:02:02.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:02 smithi154 conmon[34553]: mon.a (mon.0) 1707 : 2021-07-31T02:02:02.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:02 smithi154 conmon[34553]: cluster [DBG] osdmap e941: 8 total, 8 up, 8 in 2021-07-31T02:02:02.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:02 smithi154 conmon[34553]: audit 2021-07-31T02: 2021-07-31T02:02:02.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:02 smithi154 conmon[34553]: 02:01.071725+0000 mon.c (mon.1) 58 : audit [INF] 2021-07-31T02:02:02.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:02 smithi154 conmon[34553]: from='client.? 172.21.15.45:0/3542401327' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:02:02.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:02 smithi154 conmon[34553]: audit 2021-07-31T02:02:02.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:02 smithi154 conmon[34553]: 2021-07-31 2021-07-31T02:02:02.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:02 smithi154 conmon[34553]: T02:02:01.072294+0000 mon.a (mon.0) 2021-07-31T02:02:02.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:02 smithi154 conmon[34553]: 1708 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:02:02.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:02 smithi045 conmon[39718]: cluster 2021-07-31T02:02:01.070403+0000 mon.a (mon 2021-07-31T02:02:02.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:02 smithi045 conmon[39718]: .0) 1707 : cluster [DBG] osdmap e941: 8 total, 8 up, 8 in 2021-07-31T02:02:02.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:02 smithi045 conmon[39718]: audit 2021-07-31T02:02:01.071725+0000 mon.c ( 2021-07-31T02:02:02.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:02 smithi045 conmon[39718]: mon.1) 58 : audit [INF] from='client.? 172.21.15.45:0/3542401327' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:02:02.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:02 smithi045 conmon[39718]: audit 2021-07-31T02:02:02.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:02 smithi045 conmon[39718]: 2021-07-31T02:02:02.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:02 smithi045 conmon[39718]: 2021 2021-07-31T02:02:02.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:02 smithi045 conmon[39718]: - 2021-07-31T02:02:02.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:02 smithi045 conmon[39718]: 07 2021-07-31T02:02:02.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:02 smithi045 conmon[39718]: - 2021-07-31T02:02:02.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:02 smithi045 conmon[39718]: 31 2021-07-31T02:02:02.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:02 smithi045 conmon[39718]: T 2021-07-31T02:02:02.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:02 smithi045 conmon[39718]: 02:02:01.072294+0000 mon.a ( 2021-07-31T02:02:02.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:02 smithi045 conmon[39718]: mon.0) 1708 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:02:02.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:02 smithi045 conmon[33655]: cluster 2021-07-31T02:02:02.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:02 smithi045 conmon[33655]: 2021-07-31T02:02:02.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:02 smithi045 conmon[33655]: 2021 2021-07-31T02:02:02.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:02 smithi045 conmon[33655]: - 2021-07-31T02:02:02.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:02 smithi045 conmon[33655]: 07-31T02:02:01.070403+0000 mon.a 2021-07-31T02:02:02.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:02 smithi045 conmon[33655]: (mon.0) 1707 : cluster [DBG] osdmap e941: 8 total, 8 up, 8 in 2021-07-31T02:02:02.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:02 smithi045 conmon[33655]: audit 2021-07-31T02:02:02.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:02 smithi045 conmon[33655]: 2021-07-31T02:02:02.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:02 smithi045 conmon[33655]: 2021 2021-07-31T02:02:02.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:02 smithi045 conmon[33655]: - 2021-07-31T02:02:02.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:02 smithi045 conmon[33655]: 07 2021-07-31T02:02:02.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:02 smithi045 conmon[33655]: - 2021-07-31T02:02:02.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:02 smithi045 conmon[33655]: 31 2021-07-31T02:02:02.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:02 smithi045 conmon[33655]: T 2021-07-31T02:02:02.535 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:02 smithi045 conmon[33655]: 02:02:01.071725+0000 mon.c (mon.1) 58 : 2021-07-31T02:02:02.535 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:02 smithi045 conmon[33655]: audit [INF] from='client.? 172.21.15.45:0/3542401327' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:02:02.535 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:02 smithi045 conmon[33655]: audit 2021-07-31T02: 2021-07-31T02:02:02.535 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:02 smithi045 conmon[33655]: 02:01.072294+0000 mon.a (mon.0) 1708 2021-07-31T02:02:02.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:02 smithi045 conmon[33655]: : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:02:03.088 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] ClsLock.TestAssertLocked (3021 ms) 2021-07-31T02:02:03.089 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] ClsLock.TestSetCookie 2021-07-31T02:02:03.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:03 smithi154 conmon[34553]: cluster 2021-07-31T02:02:02.002471+0000 mgr.x (mgr.24542) 634 : cluster [DBG] pgmap v1232: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:02:03.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:03 smithi154 conmon[34553]: 2021-07-31T02:02:03.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:03 smithi154 conmon[34553]: audit 2021-07-31T02:02:02.073039+0000 mon.a (mon.0) 1709 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-6","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:02:03.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:03 smithi154 conmon[34553]: cluster 2021-07-31T02:02:02.073341+0000 mon.a (mon.0) 1710 : cluster [DBG] osdmap e942: 8 total, 8 up, 8 in 2021-07-31T02:02:03.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:03 smithi154 conmon[34553]: 2021-07-31T02:02:03.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:03 smithi045 conmon[33655]: cluster 2021-07-31T02:02:02.002471+0000 mgr.x (mgr.24542) 634 : cluster 2021-07-31T02:02:03.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:03 smithi045 conmon[33655]: [DBG] pgmap v1232: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:02:03.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:03 smithi045 conmon[33655]: audit 2021-07-31T02:02:02.073039+0000 mon.a (mon.0) 1709 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-6","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:02:03.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:03 smithi045 conmon[33655]: cluster 2021-07-31T02:02:02. 2021-07-31T02:02:03.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:03 smithi045 conmon[33655]: 073341+0000 mon.a (mon.0) 1710 : cluster [DBG] osdmap e942: 8 total, 8 up, 8 in 2021-07-31T02:02:03.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:03 smithi045 conmon[39718]: cluster 2021-07-31T02:02:02.002471 2021-07-31T02:02:03.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:03 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 634 : cluster [DBG] pgmap v1232: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:02:03.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:03 smithi045 conmon[39718]: audit 2021-07-31T02: 2021-07-31T02:02:03.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:03 smithi045 conmon[39718]: 02:02.073039+0000 mon.a 2021-07-31T02:02:03.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:03 smithi045 conmon[39718]: (mon.0) 1709 : audit 2021-07-31T02:02:03.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:03 smithi045 conmon[39718]: [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-6","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:02:03.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:03 smithi045 conmon[39718]: cluster 2021 2021-07-31T02:02:03.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:03 smithi045 conmon[39718]: -07-31T02:02:02.073341 2021-07-31T02:02:03.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:03 smithi045 conmon[39718]: +0000 mon.a (mon.0 2021-07-31T02:02:03.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:03 smithi045 conmon[39718]: ) 1710 : cluster [DBG] osdmap e942: 8 total, 8 up, 8 in 2021-07-31T02:02:03.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:03 smithi045 conmon[39718]: 2021-07-31T02:02:04.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:02:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:02:03.533Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:02:04.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:02:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:02:03.533Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:02:04.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:02:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:02:03.533Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:02:04.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:04 smithi154 conmon[34553]: cluster 2021-07-31T02:02:03.086501+0000 mon.a (mon.0) 1711 : cluster [DBG] 2021-07-31T02:02:04.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:04 smithi154 conmon[34553]: osdmap e943: 8 total, 8 up, 8 in 2021-07-31T02:02:04.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:04 smithi154 conmon[34553]: cluster 2021-07-31T02:02:04.003028+0000 mgr.x (mgr.24542) 635 : cluster [DBG] pgmap v1235: 9 pgs: 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:02:04.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:04 smithi045 conmon[39718]: cluster 2021-07-31T02:02:03.086501+0000 mon.a ( 2021-07-31T02:02:04.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:04 smithi045 conmon[39718]: mon.0) 1711 : cluster [DBG] osdmap e943: 8 total, 8 up, 8 in 2021-07-31T02:02:04.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:04 smithi045 conmon[39718]: cluster 2021-07- 2021-07-31T02:02:04.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:04 smithi045 conmon[39718]: 31T02:02:04.003028+0000 mgr.x (mgr.24542) 635 : cluster [DBG] pgmap v1235: 9 pgs: 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:02:04.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:04 smithi045 conmon[39718]: 2021-07-31T02:02:04.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:04 smithi045 conmon[33655]: cluster 2021-07-31T02:02:03.086501+0000 mon.a (mon.0) 1711 2021-07-31T02:02:04.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:04 smithi045 conmon[33655]: : cluster [DBG] osdmap e943: 8 total, 8 up, 8 in 2021-07-31T02:02:04.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:04 smithi045 conmon[33655]: cluster 2021-07-31T02:02:04.003028+0000 mgr.x (mgr.24542) 635 : cluster [DBG] pgmap v1235: 9 pgs: 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:02:05.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:05 smithi154 conmon[34553]: cluster 2021-07-31T02:02:04.093920+0000 mon.a (mon 2021-07-31T02:02:05.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:05 smithi154 conmon[34553]: .0) 1712 : cluster [DBG] osdmap e944: 8 total, 8 up, 8 in 2021-07-31T02:02:05.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:05 smithi154 conmon[34553]: audit 2021-07-31T02:02:04.096588+0000 mon.a (mon.0) 1713 : audit [INF] from='client.? 172.21.15.45:0/3659570974' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:02:05.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:05 smithi045 conmon[33655]: cluster 2021-07-31T02:02:04.093920+0000 2021-07-31T02:02:05.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:05 smithi045 conmon[33655]: mon.a (mon.0) 1712 : cluster [DBG] osdmap e944: 8 total, 8 up, 8 in 2021-07-31T02:02:05.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:05 smithi045 conmon[33655]: audit 2021-07-31T02:02:04.096588+0000 mon.a (mon.0) 1713 : audit [INF] from='client.? 172.21.15.45:0/3659570974' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:02:05.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:05 smithi045 conmon[39718]: cluster 2021-07-31T02:02:05.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:05 smithi045 conmon[39718]: 2021-07-31T02:02:05.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:05 smithi045 conmon[39718]: 2021 2021-07-31T02:02:05.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:05 smithi045 conmon[39718]: - 2021-07-31T02:02:05.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:05 smithi045 conmon[39718]: 07 2021-07-31T02:02:05.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:05 smithi045 conmon[39718]: - 2021-07-31T02:02:05.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:05 smithi045 conmon[39718]: 31 2021-07-31T02:02:05.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:05 smithi045 conmon[39718]: T 2021-07-31T02:02:05.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:05 smithi045 conmon[39718]: 02 2021-07-31T02:02:05.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:05 smithi045 conmon[39718]: : 2021-07-31T02:02:05.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:05 smithi045 conmon[39718]: 02 2021-07-31T02:02:05.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:05 smithi045 conmon[39718]: : 2021-07-31T02:02:05.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:05 smithi045 conmon[39718]: 04 2021-07-31T02:02:05.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:05 smithi045 conmon[39718]: . 2021-07-31T02:02:05.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:05 smithi045 conmon[39718]: 093920 2021-07-31T02:02:05.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:05 smithi045 conmon[39718]: +0000 2021-07-31T02:02:05.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:05 smithi045 conmon[39718]: 2021-07-31T02:02:05.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:05 smithi045 conmon[39718]: mon.a 2021-07-31T02:02:05.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:05 smithi045 conmon[39718]: ( 2021-07-31T02:02:05.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:05 smithi045 conmon[39718]: mon 2021-07-31T02:02:05.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:05 smithi045 conmon[39718]: . 2021-07-31T02:02:05.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:05 smithi045 conmon[39718]: 0 2021-07-31T02:02:05.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:05 smithi045 conmon[39718]: ) 2021-07-31T02:02:05.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:05 smithi045 conmon[39718]: 1712 2021-07-31T02:02:05.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:05 smithi045 conmon[39718]: : 2021-07-31T02:02:05.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:05 smithi045 conmon[39718]: cluster 2021-07-31T02:02:05.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:05 smithi045 conmon[39718]: 2021-07-31T02:02:05.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:05 smithi045 conmon[39718]: [DBG] 2021-07-31T02:02:05.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:05 smithi045 conmon[39718]: 2021-07-31T02:02:05.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:05 smithi045 conmon[39718]: osdmap e944: 8 total, 8 up, 8 in 2021-07-31T02:02:05.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:05 smithi045 conmon[39718]: 2021-07-31T02:02:05.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:05 smithi045 conmon[39718]: audit 2021-07-31T02:02:05.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:05 smithi045 conmon[39718]: 2021-07-31T02:02:04.096588+0000 mon.a (mon. 2021-07-31T02:02:05.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:05 smithi045 conmon[39718]: 0) 1713 : audit [INF] from='client.? 172.21.15.45:0/3659570974' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:02:06.114 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] ClsLock.TestSetCookie (3026 ms) 2021-07-31T02:02:06.115 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] ClsLock.TestRenew 2021-07-31T02:02:06.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:06 smithi154 conmon[34553]: audit 2021-07-31T02:02:05.095099+0000 mon.a (mon.0) 1714 : audit [INF] 2021-07-31T02:02:06.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:06 smithi154 conmon[34553]: from='client.? 172.21.15.45:0/3659570974' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-7","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:02:06.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:06 smithi154 conmon[34553]: cluster 2021-07-31T02:02:05.095399+0000 mon.a (mon.0) 1715 : cluster [DBG] osdmap e945: 8 total, 8 up, 8 in 2021-07-31T02:02:06.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:06 smithi154 conmon[34553]: cluster 2021-07-31T02:02:06.003481+0000 2021-07-31T02:02:06.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:06 smithi154 conmon[34553]: mgr.x (mgr.24542) 636 : cluster [DBG] pgmap v1238: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:02:06.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:06 smithi045 conmon[39718]: audit 2021-07-31T02:02:05.095099+0000 mon.a ( 2021-07-31T02:02:06.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:06 smithi045 conmon[39718]: mon.0) 1714 : audit [INF] from='client.? 172.21.15.45:0/3659570974' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-7","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:02:06.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:06 smithi045 conmon[39718]: cluster 2021-07-31T02:02:05.095399+0000 mon.a (mon. 2021-07-31T02:02:06.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:06 smithi045 conmon[39718]: 0) 1715 : cluster [DBG] osdmap e945: 8 total, 8 up, 8 in 2021-07-31T02:02:06.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:06 smithi045 conmon[39718]: cluster 2021-07-31T02:02:06.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:06 smithi045 conmon[39718]: 2021-07-31T02:02:06.003481+0000 mgr.x 2021-07-31T02:02:06.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:06 smithi045 conmon[39718]: (mgr.24542) 636 : cluster [DBG] pgmap v1238: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:02:06.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:06 smithi045 conmon[39718]: 2021-07-31T02:02:06.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:06 smithi045 conmon[33655]: audit 2021-07-31T02:02:05.095099+0000 mon.a (mon.0) 1714 : 2021-07-31T02:02:06.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:06 smithi045 conmon[33655]: audit [INF] from='client.? 172.21.15.45:0/3659570974' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-7","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:02:06.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:06 smithi045 conmon[33655]: cluster 2021-07-31T02:02:05.095399+0000 2021-07-31T02:02:06.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:06 smithi045 conmon[33655]: mon.a (mon.0) 1715 : cluster [DBG] osdmap e945: 8 total, 8 up, 8 in 2021-07-31T02:02:06.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:06 smithi045 conmon[33655]: cluster 2021-07- 2021-07-31T02:02:06.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:06 smithi045 conmon[33655]: 31T02:02:06.003481+0000 mgr.x 2021-07-31T02:02:06.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:06 smithi045 conmon[33655]: (mgr.24542) 636 : cluster [DBG] pgmap v1238: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:02:06.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:06 smithi045 conmon[33655]: 2021-07-31T02:02:07.112 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:07 smithi045 conmon[33655]: cluster 2021-07-31T02:02:06.112530 2021-07-31T02:02:07.113 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:07 smithi045 conmon[33655]: +0000 mon.a (mon.0) 1716 : cluster [DBG] osdmap e946: 8 total, 8 up, 8 in 2021-07-31T02:02:07.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:07 smithi154 conmon[34553]: cluster 2021-07-31T02:02:06.112530+0000 mon.a (mon.0) 1716 : cluster [DBG] osdmap e946: 8 total, 8 up, 8 in 2021-07-31T02:02:07.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:07 smithi154 conmon[34553]: 2021-07-31T02:02:07.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:07 smithi045 conmon[39718]: cluster 2021-07-31T02:02:06.112530+0000 mon.a (mon. 2021-07-31T02:02:07.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:07 smithi045 conmon[39718]: 0) 1716 : cluster [DBG] osdmap e946: 8 total, 8 up, 8 in 2021-07-31T02:02:08.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:08 smithi154 conmon[34553]: cluster 2021-07-31T02:02:07.117555+0000 mon.a (mon.0) 2021-07-31T02:02:08.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:08 smithi154 conmon[34553]: 1717 : cluster [DBG] osdmap e947: 8 total, 8 up, 8 in 2021-07-31T02:02:08.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:08 smithi154 conmon[34553]: audit 2021-07-31T02:02:07.119014+0000 mon.c (mon.1) 59 : audit [INF] from='client.? 172.21.15.45:0/1409858082' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:02:08.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:08 smithi154 conmon[34553]: audit 2021-07-31T 2021-07-31T02:02:08.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:08 smithi154 conmon[34553]: 02:02:07.119488+0000 mon.a (mon.0) 1718 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:02:08.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:08 smithi154 conmon[34553]: cluster 2021-07-31T02:02:08.003942+0000 2021-07-31T02:02:08.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:08 smithi154 conmon[34553]: mgr.x (mgr.24542) 637 : cluster [DBG] pgmap v1241: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:02:08.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:08 smithi045 conmon[33655]: cluster 2021-07-31T02:02:07.117555+0000 mon.a (mon.0) 1717 : cluster [DBG] 2021-07-31T02:02:08.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:08 smithi045 conmon[33655]: osdmap e947: 8 total, 8 up, 8 in 2021-07-31T02:02:08.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:08 smithi045 conmon[33655]: audit 2021-07-31T02:02:07.119014+0000 mon.c (mon.1) 59 : audit [INF] from='client.? 172.21.15.45:0/1409858082' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:02:08.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:08 smithi045 conmon[33655]: audit 2021-07-31T02 2021-07-31T02:02:08.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:08 smithi045 conmon[33655]: :02:07.119488+0000 mon.a (mon.0) 1718 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:02:08.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:08 smithi045 conmon[33655]: cluster 2021-07-31T02:02: 2021-07-31T02:02:08.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:08 smithi045 conmon[33655]: 08.003942+0000 mgr.x (mgr.24542) 637 : cluster [DBG] 2021-07-31T02:02:08.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:08 smithi045 conmon[33655]: pgmap v1241: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:02:08.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:08 smithi045 conmon[39718]: cluster 2021-07-31T02:02:07. 2021-07-31T02:02:08.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:08 smithi045 conmon[39718]: 117555+0000 mon.a (mon.0) 1717 : cluster [DBG] osdmap e947: 8 total, 8 up, 8 in 2021-07-31T02:02:08.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:08 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T02:02:08.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:08 smithi045 conmon[39718]: 31T02:02:07.119014 2021-07-31T02:02:08.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:08 smithi045 conmon[39718]: +0000 mon.c (mon.1) 2021-07-31T02:02:08.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:08 smithi045 conmon[39718]: 59 : audit [INF] from='client.? 172.21.15.45:0/1409858082' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:02:08.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:08 smithi045 conmon[39718]: audit 2021-07-31T02:02:08.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:08 smithi045 conmon[39718]: 2021-07-31T02:02:07. 2021-07-31T02:02:08.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:08 smithi045 conmon[39718]: 119488+0000 mon.a (mon.0 2021-07-31T02:02:08.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:08 smithi045 conmon[39718]: ) 1718 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:02:08.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:08 smithi045 conmon[39718]: 2021-07-31T02:02:08.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:08 smithi045 conmon[39718]: cluster 2021 2021-07-31T02:02:08.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:08 smithi045 conmon[39718]: -07-31T02:02:08.003942+0000 mgr.x (mgr 2021-07-31T02:02:08.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:08 smithi045 conmon[39718]: .24542) 637 : cluster [DBG] pgmap v1241: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:02:09.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:09 smithi154 conmon[34553]: audit 2021-07-31T02:02:08.123784+0000 mon.a (mon.0) 1719 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-8","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:02:09.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:09 smithi154 conmon[34553]: 2021-07-31T02:02:09.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:09 smithi154 conmon[34553]: cluster 2021-07-31T02:02:08.124119+0000 mon.a (mon.0) 1720 : cluster [DBG] osdmap e948: 8 total, 8 up, 8 in 2021-07-31T02:02:09.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:09 smithi045 conmon[39718]: audit 2021-07-31T02:02:08.123784+0000 mon.a 2021-07-31T02:02:09.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:09 smithi045 conmon[39718]: (mon.0) 1719 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-8","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:02:09.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:09 smithi045 conmon[39718]: cluster 2021-07-31T02:02:08.124119+0000 2021-07-31T02:02:09.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:09 smithi045 conmon[39718]: mon.a (mon.0) 1720 : cluster [DBG] osdmap e948: 8 total, 8 up, 8 in 2021-07-31T02:02:09.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:09 smithi045 conmon[33655]: audit 2021-07-31T02:02:08.123784+0000 mon.a (mon.0) 1719 : audit 2021-07-31T02:02:09.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:09 smithi045 conmon[33655]: [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-8","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:02:09.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:09 smithi045 conmon[33655]: cluster 2021-07-31T02:02:08.124119+0000 mon.a (mon.0) 1720 : cluster [DBG] osdmap e948: 8 total, 8 up, 8 in 2021-07-31T02:02:10.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:10 smithi154 conmon[34553]: cluster 2021-07-31T02:02:09.128950+0000 mon.a (mon.0) 1721 : cluster [DBG] osdmap e949: 8 total, 8 up, 8 in 2021-07-31T02:02:10.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:10 smithi154 conmon[34553]: 2021-07-31T02:02:10.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:10 smithi154 conmon[34553]: cluster 2021-07-31T02:02:10.004474+0000 mgr.x (mgr.24542) 638 : cluster [DBG] pgmap v1244: 41 pgs: 41 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 0 op/s 2021-07-31T02:02:10.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:10 smithi045 conmon[39718]: cluster 2021-07-31T02:02:09.128950+0000 mon.a (mon.0) 1721 : cluster [DBG] 2021-07-31T02:02:10.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:10 smithi045 conmon[39718]: osdmap e949: 8 total, 8 up, 8 in 2021-07-31T02:02:10.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:10 smithi045 conmon[39718]: cluster 2021-07-31T02:02:10.004474+0000 mgr.x (mgr.24542) 638 : cluster [DBG] pgmap v1244: 41 pgs: 41 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 0 op/s 2021-07-31T02:02:10.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:10 smithi045 conmon[39718]: 2021-07-31T02:02:10.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:10 smithi045 conmon[33655]: cluster 2021-07-31T02:02:09.128950+0000 2021-07-31T02:02:10.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:10 smithi045 conmon[33655]: mon.a (mon.0) 1721 : cluster [DBG] osdmap e949: 8 total, 8 up, 8 in 2021-07-31T02:02:10.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:10 smithi045 conmon[33655]: cluster 2021-07-31T02:02:10.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:10 smithi045 conmon[33655]: 2021-07-31T02:02: 2021-07-31T02:02:10.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:10 smithi045 conmon[33655]: 10.004474+0000 mgr.x (mgr.24542) 638 : cluster [DBG] pgmap v1244: 41 pgs: 41 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 0 op/s 2021-07-31T02:02:10.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:10 smithi045 conmon[33655]: 2021-07-31T02:02:13.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:12 smithi154 conmon[34553]: cluster 2021-07-31T02:02:12.004967+0000 mgr.x (mgr.24542) 639 : cluster [DBG] pgmap v1245: 41 pgs: 41 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 0 op/s 2021-07-31T02:02:13.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:12 smithi045 conmon[39718]: cluster 2021-07-31T02:02:12.004967+0000 mgr.x (mgr.24542) 2021-07-31T02:02:13.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:12 smithi045 conmon[39718]: 639 : cluster [DBG] pgmap v1245: 41 pgs: 41 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 0 op/s 2021-07-31T02:02:13.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:12 smithi045 conmon[33655]: cluster 2021-07-31T02:02:12.004967+0000 mgr.x (mgr.24542) 639 : cluster 2021-07-31T02:02:13.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:12 smithi045 conmon[33655]: [DBG] pgmap v1245: 41 pgs: 41 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 0 op/s 2021-07-31T02:02:14.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:02:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:02:13.534Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:02:14.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:02:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:02:13.534Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:02:14.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:02:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:02:13.534Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:02:15.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:15 smithi154 conmon[34553]: cluster 2021-07-31T02:02:14.005708+0000 mgr.x (mgr.24542) 640 : cluster [DBG] pgmap v1246: 41 pgs: 41 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 148 B/s rd, 0 B/s wr, 0 op/s 2021-07-31T02:02:15.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:15 smithi045 conmon[33655]: cluster 2021-07-31T02:02:14.005708 2021-07-31T02:02:15.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:15 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 640 : cluster [DBG] pgmap v1246: 41 pgs: 41 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 148 B/s rd, 0 B/s wr, 0 op/s 2021-07-31T02:02:15.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:15 smithi045 conmon[39718]: cluster 2021-07-31T02:02:14. 2021-07-31T02:02:15.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:15 smithi045 conmon[39718]: 005708+0000 mgr.x (mgr.24542) 640 : cluster [DBG] pgmap v1246: 41 pgs: 41 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 148 B/s rd, 0 B/s wr, 0 op/s 2021-07-31T02:02:17.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:17 smithi154 conmon[34553]: cluster 2021-07-31T02:02:16.006132+0000 mgr.x (mgr.24542) 641 : cluster [DBG] pgmap v1247: 41 pgs: 41 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 127 B/s rd, 0 B/s wr, 0 op/s 2021-07-31T02:02:17.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:17 smithi045 conmon[33655]: cluster 2021-07-31T02:02:16.006132+0000 mgr.x (mgr.24542) 641 : cluster [DBG] pgmap v1247: 41 pgs: 41 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 127 B/s rd, 0 B/s wr, 0 op/s 2021-07-31T02:02:17.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:17 smithi045 conmon[39718]: cluster 2021-07-31T02:02:16.006132+0000 mgr.x (mgr. 2021-07-31T02:02:17.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:17 smithi045 conmon[39718]: 24542) 641 : cluster [DBG] pgmap v1247: 41 pgs: 41 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 127 B/s rd, 0 B/s wr, 0 op/s 2021-07-31T02:02:19.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:19 smithi154 conmon[34553]: cluster 2021-07-31T02:02:18.006558+0000 mgr.x (mgr.24542) 642 : cluster [DBG] pgmap v1248: 41 pgs: 41 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 103 B/s rd, 0 B/s wr, 0 op/s 2021-07-31T02:02:19.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:19 smithi045 conmon[33655]: cluster 2021-07-31T02:02:19.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:19 smithi045 conmon[33655]: 2021-07-31T02:02:18.006558+0000 mgr.x (mgr.24542) 642 : cluster [DBG] pgmap v1248: 41 pgs: 41 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 103 B/s rd, 0 B/s wr, 0 op/s 2021-07-31T02:02:19.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:19 smithi045 conmon[39718]: cluster 2021-07-31T02:02:18.006558+0000 mgr.x ( 2021-07-31T02:02:19.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:19 smithi045 conmon[39718]: mgr.24542) 642 : cluster [DBG] pgmap v1248: 41 pgs: 41 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 103 B/s rd, 0 B/s wr, 0 op/s 2021-07-31T02:02:21.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:21 smithi154 conmon[34553]: cluster 2021-07-31T02:02:20.007110+0000 mgr.x (mgr.24542) 643 : cluster [DBG] pgmap v1249: 41 pgs: 41 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 282 B/s rd, 0 B/s wr, 0 op/s 2021-07-31T02:02:21.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:21 smithi045 conmon[33655]: cluster 2021-07-31T02:02:20.007110 2021-07-31T02:02:21.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:21 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 643 : cluster [DBG] pgmap v1249: 41 pgs: 41 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 282 B/s rd, 0 B/s wr, 0 op/s 2021-07-31T02:02:21.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:21 smithi045 conmon[39718]: cluster 2021-07-31T02:02:20.007110+0000 mgr.x ( 2021-07-31T02:02:21.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:21 smithi045 conmon[39718]: mgr.24542) 643 : cluster [DBG] pgmap v1249: 41 pgs: 41 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 282 B/s rd, 0 B/s wr, 0 op/s 2021-07-31T02:02:23.160 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:02:23.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:22 smithi154 conmon[34553]: cluster 2021-07-31T02:02:22.007585+0000 mgr.x (mgr.24542) 644 : cluster [DBG] pgmap v1250: 41 pgs: 41 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 255 B/s rd, 0 B/s wr, 0 op/s 2021-07-31T02:02:23.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:22 smithi045 conmon[33655]: cluster 2021-07-31T02:02:22.007585+0000 mgr.x (mgr.24542) 644 : cluster [DBG] pgmap v1250: 41 pgs: 41 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 255 B/s rd, 0 B/s wr, 0 op/s 2021-07-31T02:02:23.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:22 smithi045 conmon[39718]: cluster 2021-07-31T02:02:22.007585+0000 mgr.x ( 2021-07-31T02:02:23.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:22 smithi045 conmon[39718]: mgr.24542) 644 : cluster [DBG] pgmap v1250: 41 pgs: 41 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 255 B/s rd, 0 B/s wr, 0 op/s 2021-07-31T02:02:23.442 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:02:23.442 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 17m ago 18m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:02:23.442 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 14m ago 17m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:02:23.443 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 14m ago 21m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:02:23.443 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 17m ago 23m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:02:23.443 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 17m ago 23m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:02:23.443 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 14m ago 21m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:02:23.444 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 17m ago 21m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:02:23.444 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 17m ago 18m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:02:23.444 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 14m ago 18m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:02:23.444 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 17m ago 21m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:02:23.444 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 17m ago 20m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:02:23.445 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 17m ago 20m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:02:23.445 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 17m ago 20m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:02:23.445 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 14m ago 19m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:02:23.445 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 14m ago 19m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:02:23.446 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 14m ago 19m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:02:23.446 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 14m ago 18m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:02:23.446 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 14m ago 18m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:02:23.772 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:02:23.772 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:02:23.772 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:02:23.773 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:02:23.773 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:02:23.773 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:02:23.773 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:02:23.773 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:02:23.774 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:02:23.774 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:02:23.775 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:02:23.775 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:02:23.775 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:02:23.775 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:02:23.776 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:02:23.776 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:02:23.776 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:02:23.851 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:02:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:02:23.534Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:02:23.851 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:02:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:02:23.534Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:02:23.851 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:02:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:02:23.534Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:02:24.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:23 smithi154 conmon[34553]: audit 2021-07-31T02:02:23.771729+0000 mon.a (mon.0) 1722 : audit [DBG] from='client.? 172.21.15.45:0/1864139507' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:02:24.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:23 smithi045 conmon[39718]: audit 2021-07-31T02:02:23.771729 2021-07-31T02:02:24.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:23 smithi045 conmon[39718]: +0000 mon.a (mon.0) 1722 : audit [DBG] from='client.? 172.21.15.45:0/1864139507' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:02:24.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:23 smithi045 conmon[33655]: audit 2021-07-31T02:02:23.771729 2021-07-31T02:02:24.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:23 smithi045 conmon[33655]: +0000 mon.a (mon.0) 1722 : audit [DBG] from='client.? 172.21.15.45:0/1864139507' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:02:25.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:24 smithi154 conmon[34553]: audit 2021-07-31T02:02:23.148225+0000 mgr.x (mgr.24542) 645 : audit [DBG] from='client.25399 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:02:25.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:24 smithi154 conmon[34553]: audit 2021-07-31T02:02:23.431658+0000 mgr.x (mgr.24542) 646 : audit [DBG] from='client.15498 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:02:25.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:24 smithi154 conmon[34553]: cluster 2021-07-31T02:02:24.008227+0000 mgr.x (mgr.24542) 647 : cluster [DBG] pgmap v1251: 41 pgs: 41 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 341 B/s rd, 0 B/s wr, 0 op/s 2021-07-31T02:02:25.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:24 smithi045 conmon[33655]: audit 2021-07-31T02:02:23.148225+0000 2021-07-31T02:02:25.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:24 smithi045 conmon[33655]: mgr.x (mgr.24542) 645 : audit [DBG] from='client.25399 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:02:25.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:24 smithi045 conmon[33655]: audit 2021-07-31 2021-07-31T02:02:25.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:24 smithi045 conmon[33655]: T02:02:23.431658+0000 mgr.x (mgr.24542) 646 : audit [DBG] from='client.15498 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:02:25.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:24 smithi045 conmon[33655]: cluster 2021-07-31T 2021-07-31T02:02:25.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:24 smithi045 conmon[33655]: 02:02:24.008227+0000 mgr.x (mgr.24542) 647 : cluster [DBG] pgmap v1251: 41 pgs: 41 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 341 B/s rd, 0 B/s wr, 0 op/s 2021-07-31T02:02:25.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:24 smithi045 conmon[39718]: audit 2021-07-31T02:02:23.148225+0000 mgr.x (mgr.24542) 645 : 2021-07-31T02:02:25.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:24 smithi045 conmon[39718]: audit [DBG] from='client.25399 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:02:25.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:24 smithi045 conmon[39718]: audit 2021-07-31T02:02:23.431658+0000 mgr.x (mgr.24542) 646 : audit [DBG] from='client.15498 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:02:25.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:24 smithi045 conmon[39718]: cluster 2021-07-31T02:02:24.008227+0000 mgr.x (mgr.24542) 647 : 2021-07-31T02:02:25.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:24 smithi045 conmon[39718]: cluster [DBG] pgmap v1251: 41 pgs: 41 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 341 B/s rd, 0 B/s wr, 0 op/s 2021-07-31T02:02:27.068 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] ClsLock.TestRenew (20954 ms) 2021-07-31T02:02:27.069 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] ClsLock.TestExclusiveEphemeralBasic 2021-07-31T02:02:27.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:27 smithi154 conmon[34553]: cluster 2021-07-31T02:02:26.008680+0000 mgr.x (mgr.24542) 648 : cluster [DBG] pgmap v1252: 41 pgs: 41 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 255 B/s rd, 0 B/s wr, 0 op/s 2021-07-31T02:02:27.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:27 smithi045 conmon[39718]: cluster 2021-07-31T02:02:26.008680+0000 mgr.x (mgr. 2021-07-31T02:02:27.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:27 smithi045 conmon[39718]: 24542) 648 : cluster [DBG] pgmap v1252: 41 pgs: 41 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 255 B/s rd, 0 B/s wr, 0 op/s 2021-07-31T02:02:27.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:27 smithi045 conmon[33655]: cluster 2021-07-31T02:02:27.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:27 smithi045 conmon[33655]: 2021-07-31T02: 2021-07-31T02:02:27.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:27 smithi045 conmon[33655]: 02:26.008680+0000 mgr.x (mgr.24542) 648 : cluster [DBG] pgmap v1252: 41 pgs: 41 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 255 B/s rd, 0 B/s wr, 0 op/s 2021-07-31T02:02:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:28 smithi154 conmon[34553]: cluster 2021-07-31T02:02:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:28 smithi154 conmon[34553]: 2021-07-31T02:02:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:28 smithi154 conmon[34553]: 2021 2021-07-31T02:02:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:28 smithi154 conmon[34553]: - 2021-07-31T02:02:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:28 smithi154 conmon[34553]: 07-31T02:02:27.065387+0000 mon.a (mon.0) 1723 : cluster [DBG] osdmap e950: 8 total, 8 up, 8 in 2021-07-31T02:02:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:28 smithi154 conmon[34553]: audit 2021-07-31T02:02:27.989169+0000 mon.b (mon.2) 143 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:02:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:28 smithi154 conmon[34553]: audit 2021-07-31T02:02:27.989997+0000 mon.a (mon.0) 1724 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:02:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:28 smithi154 conmon[34553]: audit 2021-07-31T02:02:28.006941+0000 mon.b (mon.2) 144 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:02:28.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:28 smithi154 conmon[34553]: audit 2021-07-31T02:02:28.007438+0000 mon.a (mon.0) 1725 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:02:28.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:28 smithi045 conmon[33655]: cluster 2021-07-31T02:02:27.065387+0000 2021-07-31T02:02:28.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:28 smithi045 conmon[33655]: mon.a (mon.0) 1723 : cluster [DBG] osdmap e950: 8 total, 8 up, 8 in 2021-07-31T02:02:28.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:28 smithi045 conmon[33655]: audit 2021-07-31T02:02:27.989169+0000 mon.b (mon.2) 143 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:02:28.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:28 smithi045 conmon[33655]: audit 2021- 2021-07-31T02:02:28.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:28 smithi045 conmon[33655]: 07-31T02:02:27 2021-07-31T02:02:28.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:28 smithi045 conmon[33655]: .989997+0000 mon.a (mon.0 2021-07-31T02:02:28.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:28 smithi045 conmon[33655]: ) 1724 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:02:28.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:28 smithi045 conmon[33655]: audit 2021- 2021-07-31T02:02:28.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:28 smithi045 conmon[33655]: 07-31T02:02:28. 2021-07-31T02:02:28.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:28 smithi045 conmon[33655]: 006941+0000 mon.b (mon.2) 144 : 2021-07-31T02:02:28.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:28 smithi045 conmon[33655]: audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:02:28.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:28 smithi045 conmon[33655]: audit 2021-07-31T02:02:28.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:28 smithi045 conmon[33655]: 2021-07-31T02: 2021-07-31T02:02:28.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:28 smithi045 conmon[33655]: 02:28.007438+0000 mon.a ( 2021-07-31T02:02:28.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:28 smithi045 conmon[33655]: mon.0) 1725 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:02:28.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:28 smithi045 conmon[33655]: 2021-07-31T02:02:28.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:28 smithi045 conmon[39718]: cluster 2021-07-31T02:02:28.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:28 smithi045 conmon[39718]: 2021 2021-07-31T02:02:28.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:28 smithi045 conmon[39718]: -07-31T02:02:27.065387+0000 mon.a (mon.0) 1723 : cluster [DBG] osdmap e950: 8 total, 8 up, 8 in 2021-07-31T02:02:28.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:28 smithi045 conmon[39718]: audit 2021-07-31T02:02:28.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:28 smithi045 conmon[39718]: 2021-07-31T02:02:28.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:28 smithi045 conmon[39718]: 2021 2021-07-31T02:02:28.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:28 smithi045 conmon[39718]: - 2021-07-31T02:02:28.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:28 smithi045 conmon[39718]: 07 2021-07-31T02:02:28.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:28 smithi045 conmon[39718]: - 2021-07-31T02:02:28.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:28 smithi045 conmon[39718]: 31 2021-07-31T02:02:28.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:28 smithi045 conmon[39718]: T 2021-07-31T02:02:28.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:28 smithi045 conmon[39718]: 02 2021-07-31T02:02:28.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:28 smithi045 conmon[39718]: : 2021-07-31T02:02:28.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:28 smithi045 conmon[39718]: 02 2021-07-31T02:02:28.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:28 smithi045 conmon[39718]: : 2021-07-31T02:02:28.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:28 smithi045 conmon[39718]: 27 2021-07-31T02:02:28.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:28 smithi045 conmon[39718]: .989169+0000 mon.b ( 2021-07-31T02:02:28.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:28 smithi045 conmon[39718]: mon.2) 143 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:02:28.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:28 smithi045 conmon[39718]: audit 2021 2021-07-31T02:02:28.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:28 smithi045 conmon[39718]: -07-31T02:02:27.989997+0000 mon.a ( 2021-07-31T02:02:28.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:28 smithi045 conmon[39718]: mon.0) 1724 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:02:28.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:28 smithi045 conmon[39718]: audit 2021- 2021-07-31T02:02:28.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:28 smithi045 conmon[39718]: 07-31T02:02:28.006941+0000 mon.b (mon.2) 144 2021-07-31T02:02:28.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:28 smithi045 conmon[39718]: : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:02:28.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:28 smithi045 conmon[39718]: audit 2021-07-31T02:02:28.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:28 smithi045 conmon[39718]: 2021-07-31T02:02:28.007438+0000 2021-07-31T02:02:28.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:28 smithi045 conmon[39718]: mon.a (mon.0) 1725 : audit [INF] 2021-07-31T02:02:28.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:28 smithi045 conmon[39718]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:02:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[34553]: cluster 2021-07-31T02:02:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[34553]: 2021-07-31T02:02:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[34553]: 2021 2021-07-31T02:02:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[34553]: -07-31T02:02:28.009009+0000 mgr.x 2021-07-31T02:02:29.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[34553]: (mgr.24542) 649 : cluster [DBG] pgmap v1254: 9 pgs: 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:02:29.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[34553]: cluster 2021 2021-07-31T02:02:29.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[34553]: -07-31T02:02:28.077378+0000 mon.a 2021-07-31T02:02:29.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[34553]: (mon.0) 1726 : cluster [DBG] osdmap e951: 8 total, 8 up, 8 in 2021-07-31T02:02:29.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[34553]: audit 2021-07 2021-07-31T02:02:29.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[34553]: -31T02:02:28.079099+0000 mon.c (mon.1) 2021-07-31T02:02:29.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[34553]: 60 : audit [INF] from='client.? 172.21.15.45:0/1752184575' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:02:29.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[34553]: audit 2021-07- 2021-07-31T02:02:29.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[34553]: 31T02:02:28.079869+0000 mon.a (mon.0) 1727 : audit 2021-07-31T02:02:29.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[34553]: [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:02:29.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:29 smithi045 conmon[33655]: cluster 2021-07-31T02:02:28.009009+0000 mgr.x (mgr. 2021-07-31T02:02:29.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:29 smithi045 conmon[33655]: 24542) 649 : cluster [DBG] pgmap v1254: 9 pgs: 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:02:29.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:29 smithi045 conmon[33655]: cluster 2021-07-31T02:02:28.077378+0000 mon.a (mon.0) 1726 : cluster [DBG] osdmap e951: 8 total, 8 up, 8 in 2021-07-31T02:02:29.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:29 smithi045 conmon[33655]: audit 2021-07-31T02:02:28.079099+0000 mon.c (mon.1) 60 : audit 2021-07-31T02:02:29.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:29 smithi045 conmon[33655]: [INF] from='client.? 172.21.15.45:0/1752184575' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:02:29.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:29 smithi045 conmon[33655]: audit 2021-07-31T02:02 2021-07-31T02:02:29.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:29 smithi045 conmon[33655]: :28.079869+0000 mon.a (mon.0) 1727 : audit 2021-07-31T02:02:29.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:29 smithi045 conmon[33655]: [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:02:29.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:29 smithi045 conmon[39718]: cluster 2021-07-31T02:02:29.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:29 smithi045 conmon[39718]: 2021-07-31T02:02:29.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:29 smithi045 conmon[39718]: 2021 2021-07-31T02:02:29.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:29 smithi045 conmon[39718]: -07-31T02:02:28.009009+0000 2021-07-31T02:02:29.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:29 smithi045 conmon[39718]: mgr.x (mgr.24542) 649 : cluster [DBG] pgmap v1254: 9 pgs: 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:02:29.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:29 smithi045 conmon[39718]: cluster 2021-07-31T02:02:28 2021-07-31T02:02:29.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:29 smithi045 conmon[39718]: .077378+0000 mon.a (mon.0) 1726 : cluster 2021-07-31T02:02:29.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:29 smithi045 conmon[39718]: [DBG] osdmap e951: 8 total, 8 up, 8 in 2021-07-31T02:02:29.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:29 smithi045 conmon[39718]: audit 2021-07 2021-07-31T02:02:29.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:29 smithi045 conmon[39718]: -31T02:02:28.079099+0000 mon.c 2021-07-31T02:02:29.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:29 smithi045 conmon[39718]: (mon.1) 60 : audit [INF] from='client.? 172.21.15.45:0/1752184575' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:02:29.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:29 smithi045 conmon[39718]: audit 2021- 2021-07-31T02:02:29.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:29 smithi045 conmon[39718]: 07-31T02:02:28.079869+0000 mon.a (mon 2021-07-31T02:02:29.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:29 smithi045 conmon[39718]: .0) 1727 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:02:30.003 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: debug 2021-07-31T02:02:29.674+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:02:30.003 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:02:30.003 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:02:30.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: yield 2021-07-31T02:02:30.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:02:30.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:02:30.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:02:30.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:02:30.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:02:30.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:02:30.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:02:30.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:02:30.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:02:30.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: task.result() 2021-07-31T02:02:30.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:02:30.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: debug 2021-07-31T02:02:29.685+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:02:30.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:02:30.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:02:30.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: yield 2021-07-31T02:02:30.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:02:30.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:02:30.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:02:30.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:02:30.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:02:30.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:02:30.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:02:30.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:02:30.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:02:30.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: task.result() 2021-07-31T02:02:30.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:02:30.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: debug 2021-07-31T02:02:29.711+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:02:30.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:02:30.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:02:30.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: yield 2021-07-31T02:02:30.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:02:30.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:02:30.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:02:30.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:02:30.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:02:30.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:02:30.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:02:30.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:02:30.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:02:30.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: task.result() 2021-07-31T02:02:30.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:02:30.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: debug 2021-07-31T02:02:29.712+0000 7fcb93961700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:02:30.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:02:30.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:02:30.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: yield 2021-07-31T02:02:30.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:02:30.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:02:30.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:02:30.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:02:30.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:02:30.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:02:30.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:02:30.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:02:30.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:02:30.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: task.result() 2021-07-31T02:02:30.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:02:30.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: 2021-07-31T02:02:30.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:02:30.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: 2021-07-31T02:02:30.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:02:30.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:02:30.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:02:30.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:02:30.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:02:30.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:02:30.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:02:30.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:02:30.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:02:30.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:02:30.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:02:30.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:02:30.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:02:30.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:02:30.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:02:30.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:02:30.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:02:30.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:02:30.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: raise 2021-07-31T02:02:30.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:02:30.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:02:30.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:02:30.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:02:30.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:02:30.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: debug 2021-07-31T02:02:29.720+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:02:30.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:02:30.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:02:30.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: yield 2021-07-31T02:02:30.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:02:30.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:02:30.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:02:30.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:02:30.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:02:30.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:02:30.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:02:30.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:02:30.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:02:30.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: task.result() 2021-07-31T02:02:30.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:02:30.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: debug 2021-07-31T02:02:29.721+0000 7fcb94162700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:02:30.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:02:30.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:02:30.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: yield 2021-07-31T02:02:30.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:02:30.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:02:30.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:02:30.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:02:30.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:02:30.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:02:30.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:02:30.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:02:30.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:02:30.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: task.result() 2021-07-31T02:02:30.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:02:30.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: 2021-07-31T02:02:30.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:02:30.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: 2021-07-31T02:02:30.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:02:30.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:02:30.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:02:30.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:02:30.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:02:30.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:02:30.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:02:30.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:02:30.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:02:30.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:02:30.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:02:30.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:02:30.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:02:30.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:02:30.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:02:30.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:02:30.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:02:30.035 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:02:30.035 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: raise 2021-07-31T02:02:30.035 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:02:30.035 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:02:30.036 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:02:30.036 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:02:30.036 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:02:29 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:02:30.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: audit 2021-07-31T02:02:29.079612+0000 mon.a (mon.0) 1728 : 2021-07-31T02:02:30.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-9","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:02:30.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: cluster 2021-07-31T02:02:29.079820+0000 mon.a (mon.0) 1729 : cluster [DBG] osdmap e952: 8 total, 8 up, 8 in 2021-07-31T02:02:30.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: audit 2021-07-31T02:02:29.628897+0000 mon.b (mon.2 2021-07-31T02:02:30.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: ) 145 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:02:30.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: audit 2021-07-31T02:02:30.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: 2021-07-31T02:02:29. 2021-07-31T02:02:30.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: 630323+0000 mon.b (mon.2) 146 : audit 2021-07-31T02:02:30.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:02:30.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: audit 2021-07-31T02:02:30.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: 2021-07-31T02:02:29.631183+0000 2021-07-31T02:02:30.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: mon.b (mon.2) 147 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:02:30.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: 2021-07-31T02:02:30.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: cephadm 2021 2021-07-31T02:02:30.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: -07-31T02:02:29.631948+0000 2021-07-31T02:02:30.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: mgr.x (mgr.24542) 650 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:02:30.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: cephadm 2021-07-31 2021-07-31T02:02:30.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: T02:02:29.675246+0000 mgr.x (mgr.24542) 651 : 2021-07-31T02:02:30.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: cephadm [ERR] 2021-07-31T02:02:30.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:02:30.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:02:30.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: yield 2021-07-31T02:02:30.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:02:30.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:02:30.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:02:30.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:02:30.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:02:30.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:02:30.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:02:30.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:02:30.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:02:30.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:02:30.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:02:30.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:02:29.675721+0000 mgr.x (mgr.24542) 652 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:02:30.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:02:29.686758+0000 mgr.x (mgr.24542) 653 : cephadm [ERR] 2021-07-31T02:02:30.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: 2021-07-31T02:02:30.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:02:30.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:02:30.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: yield 2021-07-31T02:02:30.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:02:30.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:02:30.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:02:30.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:02:30.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:02:30.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:02:30.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:02:30.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:02:30.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:02:30.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:02:30.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:02:30.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:02 2021-07-31T02:02:30.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: :29.687700+0000 mgr.x (mgr.24542) 654 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:02:30.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:02:29.712726+0000 mgr.x (mgr.24542) 655 : cephadm [ERR] 2021-07-31T02:02:30.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:02:30.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:02:30.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: yield 2021-07-31T02:02:30.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:02:30.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:02:30.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:02:30.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:02:30.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:02:30.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:02:30.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:02:30.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:02:30.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:02:30.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:02:30.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:02:30.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:02:29.713169+0000 mgr.x (mgr.24542) 656 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:02:30.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:02:30.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:02:30.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: yield 2021-07-31T02:02:30.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:02:30.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:02:30.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:02:30.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:02:30.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:02:30.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:02:30.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:02:30.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:02:30.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:02:30.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:02:30.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:02:30.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: 2021-07-31T02:02:30.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:02:30.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: 2021-07-31T02:02:30.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:02:30.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:02:30.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:02:30.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:02:30.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:02:30.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:02:30.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:02:30.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:02:30.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:02:30.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:02:30.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:02:30.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:02:30.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:02:30.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:02:30.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:02:30.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:02:30.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:02:30.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:02:30.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: raise 2021-07-31T02:02:30.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:02:30.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:02:30.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:02:30.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:02:30.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:02:30.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:02:29.721620+0000 mgr.x (mgr.24542) 657 : cephadm [ERR] 2021-07-31T02:02:30.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:02:30.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:02:30.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: yield 2021-07-31T02:02:30.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:02:30.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:02:30.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:02:30.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:02:30.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:02:30.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:02:30.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: audit 2021-07-31T02:02:29.079612+0000 mon.a (mon. 2021-07-31T02:02:30.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: 0) 1728 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-9","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:02:30.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: cluster 2021-07-31T02:02:29.079820+0000 mon.a (mon. 2021-07-31T02:02:30.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: 0) 1729 : cluster [DBG] osdmap e952: 8 total, 8 up, 8 in 2021-07-31T02:02:30.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: audit 2021-07-31T02:02:29. 2021-07-31T02:02:30.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: 628897+0000 mon.b (mon.2) 145 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:02:30.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: audit 2021-07-31T02:02:30.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: 2021-07-31T02:02:29.630323+0000 mon.b (mon.2) 146 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:02:30.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: 2021-07-31T02:02:30.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: audit 2021-07-31T02: 2021-07-31T02:02:30.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: 02:29.631183+0000 mon.b (mon.2) 147 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:02:30.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: cephadm 2021-07-31T02 2021-07-31T02:02:30.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: :02:29.631948+0000 mgr.x (mgr.24542) 650 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:02:30.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: cephadm 2021-07-31T02: 2021-07-31T02:02:30.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: 02:29.675246+0000 mgr.x (mgr.24542) 651 : cephadm [ERR] 2021-07-31T02:02:30.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: 2021-07-31T02:02:30.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:02:30.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:02:30.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: yield 2021-07-31T02:02:30.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:02:30.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:02:30.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:02:30.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:02:30.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:02:30.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:02:30.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:02:30.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:02:30.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:02:30.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:02:30.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:02:30.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:02:30.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: 2021-07-31T02:02:29.675721+0000 mgr.x (mgr.24542) 652 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:02:30.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:02:29.686758+0000 mgr.x (mgr.24542) 653 : cephadm [ERR] 2021-07-31T02:02:30.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:02:30.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:02:30.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: yield 2021-07-31T02:02:30.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:02:30.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: audit 2021-07-31T02:02:29.079612+0000 mon.a (mon.0) 1728 2021-07-31T02:02:30.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-9","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:02:30.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: cluster 2021-07-31T02:02:29.079820+0000 mon.a (mon.0) 1729 : cluster [DBG] osdmap e952: 8 total, 8 up, 8 in 2021-07-31T02:02:30.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: audit 2021-07-31T02:02:29.628897+0000 mon.b (mon.2) 145 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:02:30.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: audit 2021-07-31T02:02:29.630323+0000 mon.b (mon.2) 146 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:02:30.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: audit 2021-07-31T02:02:29.631183+0000 mon.b (mon.2) 147 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:02:30.543 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:02:29.631948+0000 mgr.x (mgr.24542 2021-07-31T02:02:30.543 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: ) 650 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:02:30.543 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:02:29.675246+0000 mgr.x (mgr.24542) 2021-07-31T02:02:30.543 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: 651 : cephadm [ERR] 2021-07-31T02:02:30.544 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:02:30.544 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:02:30.544 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: yield 2021-07-31T02:02:30.544 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:02:30.544 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:02:30.545 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:02:30.545 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:02:30.545 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:02:30.545 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:02:30.545 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:02:30.546 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:02:30.546 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:02:30.546 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:02:30.546 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:02:30.547 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:02:30.547 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: 2021-07-31T02:02:29.675721+0000 mgr.x (mgr.24542) 652 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:02:30.547 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:02:29.686758+0000 mgr.x (mgr.24542) 653 : cephadm [ERR] 2021-07-31T02:02:30.547 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:02:30.547 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:02:30.548 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: yield 2021-07-31T02:02:30.548 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:02:30.548 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:02:30.548 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:02:30.549 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:02:30.549 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:02:30.549 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:02:30.549 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:02:30.549 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:02:30.550 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:02:29.722059+0000 mgr.x (mgr.24542) 658 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:02:30.550 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:02:30.550 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:02:30.550 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: yield 2021-07-31T02:02:30.551 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:02:30.551 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:02:30.551 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:02:30.551 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:02:30.551 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:02:30.552 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:02:30.552 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:02:30.552 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:02:30.552 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:02:30.552 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:02:30.553 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:02:30.553 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: 2021-07-31T02:02:30.553 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:02:30.553 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: 2021-07-31T02:02:30.554 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:02:30.554 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:02:30.554 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:02:30.554 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:02:30.555 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:02:30.555 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:02:30.555 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:02:30.555 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:02:30.555 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:02:30.556 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:02:30.556 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:02:30.556 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:02:30.556 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:02:30.556 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:02:30.557 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:02:30.557 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:02:30.557 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:02:30.557 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:02:30.558 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: raise 2021-07-31T02:02:30.558 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:02:30.558 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:02:30.558 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:02:30.559 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:02:30.559 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:02:30.559 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:30 smithi154 conmon[34553]: cluster 2021-07-31T02:02:30.009438+0000 mgr.x (mgr.24542) 659 : cluster [DBG] pgmap v1257: 41 pgs: 32 creating+peering, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:02:30.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:02:30.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:02:30.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:02:30.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:02:30.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:02:30.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:02:30.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:02:30.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:02:30.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:02:30.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:02:30.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:02:29.687700+0000 mgr.x (mgr.24542) 654 : 2021-07-31T02:02:30.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:02:30.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:02:29.712726+0000 mgr.x (mgr.24542) 655 : cephadm [ERR] 2021-07-31T02:02:30.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:02:30.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:02:30.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: yield 2021-07-31T02:02:30.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:02:30.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:02:30.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:02:30.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:02:30.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:02:30.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:02:30.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:02:30.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:02:30.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:02:30.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:02:30.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:02:30.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:02:29.713169+0000 mgr.x (mgr.24542) 656 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:02:30.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:02:30.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:02:30.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: yield 2021-07-31T02:02:30.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:02:30.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:02:30.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:02:30.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:02:30.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:02:30.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:02:30.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:02:30.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:02:30.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:02:30.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:02:30.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:02:30.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: 2021-07-31T02:02:30.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:02:30.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: 2021-07-31T02:02:30.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:02:30.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:02:30.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:02:30.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:02:30.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:02:30.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:02:30.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:02:30.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:02:30.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:02:30.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:02:30.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:02:30.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:02:30.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:02:30.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:02:30.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:02:30.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:02:30.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:02:30.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:02:30.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: raise 2021-07-31T02:02:30.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:02:30.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:02:30.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:02:30.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:02:30.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:02:30.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:02:29.721620+0000 mgr.x (mgr.24542) 657 2021-07-31T02:02:30.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: : cephadm [ERR] 2021-07-31T02:02:30.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:02:30.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:02:30.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: yield 2021-07-31T02:02:30.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:02:30.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:02:30.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:02:30.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:02:30.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:02:30.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:02:30.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:02:30.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:02:30.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:02:30.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:02:30.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:02:30.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:02:30.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:02:30.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:02:30.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:02:30.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:02:30.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: 2021-07-31T02:02:29.687700+0000 mgr.x (mgr.24542) 654 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:02:30.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:02:29.712726+0000 mgr.x (mgr.24542) 655 : cephadm [ERR] 2021-07-31T02:02:30.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:02:30.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:02:30.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: yield 2021-07-31T02:02:30.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:02:30.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:02:30.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:02:30.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:02:30.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:02:30.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:02:30.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:02:30.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:02:30.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:02:30.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:02:30.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:02:30.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:02:29.713169+0000 mgr.x (mgr.24542) 656 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:02:30.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:02:30.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:02:30.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: yield 2021-07-31T02:02:30.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:02:30.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:02:30.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:02:30.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:02:30.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:02:30.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:02:30.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:02:30.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:02:30.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:02:30.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:02:30.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:02:30.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: 2021-07-31T02:02:30.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:02:30.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:02:30.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:02:30.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:02:30.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:02:29.722059+0000 mgr.x (mgr.24542) 658 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:02:30.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:02:30.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:02:30.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: yield 2021-07-31T02:02:30.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:02:30.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:02:30.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:02:30.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:02:30.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:02:30.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:02:30.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:02:30.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:02:30.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:02:30.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:02:30.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:02:30.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: 2021-07-31T02:02:30.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:02:30.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: 2021-07-31T02:02:30.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:02:30.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:02:30.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:02:30.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:02:30.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:02:30.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:02:30.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:02:30.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:02:30.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:02:30.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:02:30.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:02:30.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:02:30.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:02:30.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:02:30.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:02:30.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:02:30.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:02:30.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:02:30.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: raise 2021-07-31T02:02:30.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:02:30.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:02:30.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:02:30.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:02:30.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:02:30.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[39718]: cluster 2021-07-31T02:02:30.009438+0000 mgr.x (mgr.24542) 659 : cluster [DBG] pgmap v1257: 41 pgs: 32 creating+peering, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:02:30.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:02:30.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: 2021-07-31T02:02:30.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:02:30.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:02:30.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:02:30.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:02:30.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:02:30.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:02:30.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:02:30.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:02:30.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:02:30.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:02:30.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:02:30.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:02:30.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:02:30.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:02:30.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:02:30.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:02:30.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:02:30.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:02:30.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: raise 2021-07-31T02:02:30.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:02:30.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:02:30.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:02:30.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:02:30.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:02:30.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: cephadm 2021 2021-07-31T02:02:30.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: -07-31T02:02:29.721620+0000 mgr.x (mgr.24542) 657 : cephadm [ERR] 2021-07-31T02:02:30.608 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:02:30.608 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:02:30.608 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: yield 2021-07-31T02:02:30.608 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:02:30.608 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:02:30.609 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:02:30.609 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:02:30.609 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:02:30.609 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:02:30.610 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:02:30.610 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:02:30.610 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:02:30.610 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:02:30.610 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:02:30.611 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:02:29.722059+0000 mgr.x (mgr.24542) 658 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:02:30.611 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:02:30.611 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:02:30.611 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: yield 2021-07-31T02:02:30.612 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:02:30.612 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:02:30.612 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:02:30.612 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:02:30.612 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:02:30.613 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:02:30.613 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:02:30.613 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:02:30.613 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:02:30.613 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:02:30.614 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:02:30.614 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: 2021-07-31T02:02:30.614 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:02:30.614 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: 2021-07-31T02:02:30.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:02:30.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:02:30.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:02:30.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:02:30.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:02:30.616 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:02:30.616 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:02:30.616 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:02:30.616 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:02:30.616 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:02:30.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:02:30.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:02:30.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:02:30.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:02:30.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:02:30.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:02:30.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:02:30.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:02:30.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: raise 2021-07-31T02:02:30.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:02:30.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:02:30.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:02:30.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:02:30.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:02:30.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:30 smithi045 conmon[33655]: cluster 2021-07-31T02:02:30.009438+0000 mgr.x (mgr.24542) 659 : cluster [DBG] pgmap v1257: 41 pgs: 32 creating+peering, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:02:31.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:31 smithi154 conmon[34553]: cluster 2021-07-31T02:02:30.098526+0000 mon.a (mon.0) 1730 : cluster [DBG] osdmap e953: 8 total, 8 up, 8 in 2021-07-31T02:02:31.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:31 smithi154 conmon[34553]: 2021-07-31T02:02:31.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:31 smithi045 conmon[39718]: cluster 2021-07-31T02:02:30.098526+0000 mon.a (mon.0) 1730 : 2021-07-31T02:02:31.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:31 smithi045 conmon[39718]: cluster [DBG] osdmap e953: 8 total, 8 up, 8 in 2021-07-31T02:02:31.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:31 smithi045 conmon[33655]: cluster 2021-07-31T02:02:30.098526 2021-07-31T02:02:31.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:31 smithi045 conmon[33655]: +0000 mon.a (mon.0) 1730 : cluster [DBG] osdmap e953: 8 total, 8 up, 8 in 2021-07-31T02:02:33.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:32 smithi154 conmon[34553]: cluster 2021-07-31T02:02:32.009891+0000 mgr.x (mgr.24542) 660 : 2021-07-31T02:02:33.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:32 smithi154 conmon[34553]: cluster [DBG] pgmap v1259: 41 pgs: 32 creating+peering, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:02:33.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:32 smithi045 conmon[33655]: cluster 2021-07-31T02:02:32.009891+0000 mgr.x (mgr.24542) 660 : 2021-07-31T02:02:33.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:32 smithi045 conmon[33655]: cluster [DBG] pgmap v1259: 41 pgs: 32 creating+peering, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:02:33.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:32 smithi045 conmon[39718]: cluster 2021-07-31T02:02:32.009891+0000 mgr.x (mgr.24542) 660 : cluster 2021-07-31T02:02:33.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:32 smithi045 conmon[39718]: [DBG] pgmap v1259: 41 pgs: 32 creating+peering, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:02:33.858 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] ClsLock.TestExclusiveEphemeralBasic (6789 ms) 2021-07-31T02:02:33.858 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] ClsLock.TestExclusiveEphemeralStealEphemeral 2021-07-31T02:02:34.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:02:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:02:33.534Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:02:34.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:02:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:02:33.534Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:02:34.027 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:02:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:02:33.534Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:02:35.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:34 smithi154 conmon[34553]: cluster 2021-07-31T02:02:33.855421+0000 mon.a (mon.0) 1731 2021-07-31T02:02:35.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:34 smithi154 conmon[34553]: : cluster [DBG] osdmap e954: 8 total, 8 up, 8 in 2021-07-31T02:02:35.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:34 smithi154 conmon[34553]: cluster 2021-07-31T02:02:34.010405+0000 mgr.x (mgr.24542) 661 : cluster [DBG] pgmap v1261: 9 pgs: 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:02:35.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:34 smithi045 conmon[39718]: cluster 2021-07-31T02:02:33.855421+0000 mon.a ( 2021-07-31T02:02:35.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:34 smithi045 conmon[39718]: mon.0) 1731 : cluster [DBG] osdmap e954: 8 total, 8 up, 8 in 2021-07-31T02:02:35.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:34 smithi045 conmon[39718]: cluster 2021-07-31T02:02:34.010405+0000 2021-07-31T02:02:35.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:34 smithi045 conmon[39718]: mgr.x (mgr.24542) 661 : cluster [DBG] pgmap v1261: 9 pgs: 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:02:35.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:34 smithi045 conmon[33655]: cluster 2021-07-31T02:02:33.855421+0000 mon.a (mon. 2021-07-31T02:02:35.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:34 smithi045 conmon[33655]: 0) 1731 : cluster [DBG] osdmap e954: 8 total, 8 up, 8 in 2021-07-31T02:02:35.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:34 smithi045 conmon[33655]: cluster 2021-07-31T02:02:34.010405+0000 mgr.x (mgr.24542) 661 : cluster [DBG] pgmap v1261: 9 pgs: 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:02:36.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:35 smithi154 conmon[34553]: cluster 2021-07-31T02:02:34.860881+0000 mon.a (mon.0 2021-07-31T02:02:36.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:35 smithi154 conmon[34553]: ) 1732 : cluster [DBG] osdmap e955: 8 total, 8 up, 8 in 2021-07-31T02:02:36.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:35 smithi154 conmon[34553]: audit 2021-07-31T02:02:34.861967+0000 mon.a (mon.0) 1733 : audit [INF] from='client.? 172.21.15.45:0/1127161941' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:02:36.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:35 smithi045 conmon[39718]: cluster 2021-07-31T02:02:34.860881+0000 mon.a (mon.0) 2021-07-31T02:02:36.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:35 smithi045 conmon[39718]: 1732 : cluster [DBG] osdmap e955: 8 total, 8 up, 8 in 2021-07-31T02:02:36.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:35 smithi045 conmon[39718]: audit 2021-07-31T02:02:34.861967+0000 mon.a (mon.0) 1733 : audit [INF] from='client.? 172.21.15.45:0/1127161941' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:02:36.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:35 smithi045 conmon[33655]: cluster 2021-07-31T02:02:34.860881+0000 mon.a ( 2021-07-31T02:02:36.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:35 smithi045 conmon[33655]: mon.0) 1732 : cluster [DBG] osdmap e955: 8 total, 8 up, 8 in 2021-07-31T02:02:36.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:35 smithi045 conmon[33655]: audit 2021-07-31T02:02:34.861967+0000 mon.a (mon.0) 1733 : audit [INF] from='client.? 172.21.15.45:0/1127161941' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:02:37.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:36 smithi154 conmon[34553]: audit 2021-07-31T02:02:35.863179+0000 mon.a (mon.0) 1734 : audit 2021-07-31T02:02:37.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:36 smithi154 conmon[34553]: [INF] from='client.? 172.21.15.45:0/1127161941' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-10","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:02:37.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:36 smithi154 conmon[34553]: cluster 2021-07-31T02:02:35.863394+0000 mon.a (mon.0) 1735 : cluster [DBG] osdmap e956: 8 total, 8 up, 8 in 2021-07-31T02:02:37.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:36 smithi154 conmon[34553]: cluster 2021-07-31T02:02:36.010811+0000 mgr.x (mgr.24542) 662 : cluster [DBG] pgmap v1264: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:02:37.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:36 smithi045 conmon[39718]: audit 2021-07-31T02:02:35.863179+0000 2021-07-31T02:02:37.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:36 smithi045 conmon[39718]: mon.a (mon.0) 1734 : audit [INF] from='client.? 172.21.15.45:0/1127161941' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-10","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:02:37.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:36 smithi045 conmon[39718]: cluster 2021-07-31T02:02:35.863394 2021-07-31T02:02:37.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:36 smithi045 conmon[39718]: +0000 mon.a (mon.0) 1735 : cluster [DBG] osdmap e956: 8 total, 8 up, 8 in 2021-07-31T02:02:37.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:36 smithi045 conmon[39718]: cluster 2021-07-31 2021-07-31T02:02:37.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:36 smithi045 conmon[39718]: T02:02:36.010811+0000 mgr.x ( 2021-07-31T02:02:37.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:36 smithi045 conmon[39718]: mgr.24542) 662 : cluster [DBG] pgmap v1264: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:02:37.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:36 smithi045 conmon[33655]: audit 2021-07-31T02:02:35.863179+0000 mon.a (mon.0) 1734 : audit [INF] from='client.? 172.21.15.45:0/1127161941' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-10","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:02:37.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:36 smithi045 conmon[33655]: cluster 2021-07-31T02:02:35.863394+0000 mon.a (mon.0) 1735 : cluster [DBG] osdmap e956: 8 total, 8 up, 8 in 2021-07-31T02:02:37.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:36 smithi045 conmon[33655]: cluster 2021-07-31T02:02:36.010811+0000 mgr.x (mgr.24542) 2021-07-31T02:02:37.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:36 smithi045 conmon[33655]: 662 : cluster [DBG] pgmap v1264: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:02:38.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:37 smithi154 conmon[34553]: cluster 2021-07-31T02:02:36.878681+0000 mon.a (mon.0) 1736 : cluster [DBG] osdmap e957: 8 total, 8 up, 8 in 2021-07-31T02:02:38.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:37 smithi045 conmon[33655]: cluster 2021-07-31T02:02:36.878681+0000 mon.a (mon.0) 1736 2021-07-31T02:02:38.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:37 smithi045 conmon[33655]: : cluster [DBG] osdmap e957: 8 total, 8 up, 8 in 2021-07-31T02:02:38.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:37 smithi045 conmon[39718]: cluster 2021-07-31T02:02:36.878681+0000 mon.a (mon.0) 1736 : cluster [DBG] osdmap e957: 8 total, 8 up, 8 in 2021-07-31T02:02:39.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:38 smithi154 conmon[34553]: cluster 2021-07-31T02:02:38.011406+0000 mgr.x (mgr.24542) 663 : cluster 2021-07-31T02:02:39.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:38 smithi154 conmon[34553]: [DBG] pgmap v1266: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:02:39.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:38 smithi045 conmon[33655]: cluster 2021-07-31T02:02:38.011406+0000 mgr.x (mgr.24542) 663 : cluster 2021-07-31T02:02:39.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:38 smithi045 conmon[33655]: [DBG] pgmap v1266: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:02:39.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:38 smithi045 conmon[39718]: cluster 2021-07-31T02:02:38.011406 2021-07-31T02:02:39.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:38 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 663 : cluster [DBG] pgmap v1266: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:02:41.078 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] ClsLock.TestExclusiveEphemeralStealEphemeral (7220 ms) 2021-07-31T02:02:41.078 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] ClsLock.TestExclusiveEphemeralStealExclusive 2021-07-31T02:02:41.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:41 smithi154 conmon[34553]: cluster 2021-07-31T02:02:40.012111+0000 mgr.x (mgr.24542) 664 : cluster [DBG] pgmap v1267: 41 pgs: 41 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 0 op/s 2021-07-31T02:02:41.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:41 smithi045 conmon[39718]: cluster 2021-07-31T02:02:40. 2021-07-31T02:02:41.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:41 smithi045 conmon[39718]: 012111+0000 mgr.x (mgr.24542) 664 : cluster [DBG] pgmap v1267: 41 pgs: 41 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 0 op/s 2021-07-31T02:02:41.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:41 smithi045 conmon[33655]: cluster 2021-07-31T02:02:40.012111+0000 mgr.x (mgr.24542) 664 : cluster [DBG] 2021-07-31T02:02:41.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:41 smithi045 conmon[33655]: pgmap v1267: 41 pgs: 41 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 0 op/s 2021-07-31T02:02:42.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:42 smithi154 conmon[34553]: cluster 2021-07-31T02:02:41.075820+0000 mon.a (mon.0) 1737 : cluster [DBG] osdmap e958: 8 total, 8 up, 8 in 2021-07-31T02:02:42.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:42 smithi045 conmon[39718]: cluster 2021-07-31T02:02:41.075820+0000 mon.a 2021-07-31T02:02:42.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:42 smithi045 conmon[39718]: (mon.0) 1737 : cluster [DBG] osdmap e958: 8 total, 8 up, 8 in 2021-07-31T02:02:42.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:42 smithi045 conmon[33655]: cluster 2021-07-31T02:02: 2021-07-31T02:02:42.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:42 smithi045 conmon[33655]: 41.075820+0000 mon.a (mon.0) 1737 : cluster [DBG] osdmap e958: 8 total, 8 up, 8 in 2021-07-31T02:02:43.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:43 smithi154 conmon[34553]: cluster 2021-07-31T02:02:43.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:43 smithi154 conmon[34553]: 2021-07-31T02:02:43.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:43 smithi154 conmon[34553]: 2021-07-31T02:02:42.012557+0000 mgr.x 2021-07-31T02:02:43.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:43 smithi154 conmon[34553]: (mgr.24542) 665 : cluster [DBG] pgmap v1269: 9 pgs: 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:02:43.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:43 smithi154 conmon[34553]: cluster 2021-07-31T02:02 2021-07-31T02:02:43.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:43 smithi154 conmon[34553]: :42.080724+0000 mon.a (mon.0) 1738 : cluster [DBG] 2021-07-31T02:02:43.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:43 smithi154 conmon[34553]: osdmap e959: 8 total, 8 up, 8 in 2021-07-31T02:02:43.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:43 smithi154 conmon[34553]: audit 2021-07-31T02:02: 2021-07-31T02:02:43.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:43 smithi154 conmon[34553]: 42.082234+0000 mon.c (mon.1) 61 : audit [INF] 2021-07-31T02:02:43.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:43 smithi154 conmon[34553]: from='client.? 172.21.15.45:0/2197804790' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:02:43.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:43 smithi154 conmon[34553]: audit 2021-07-31T02:02:42. 2021-07-31T02:02:43.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:43 smithi154 conmon[34553]: 082796+0000 mon.a (mon.0) 1739 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:02:43.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:43 smithi154 conmon[34553]: 2021-07-31T02:02:43.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:43 smithi045 conmon[33655]: cluster 2021-07-31T02:02:42.012557+0000 mgr.x (mgr.24542 2021-07-31T02:02:43.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:43 smithi045 conmon[33655]: ) 665 : cluster [DBG] pgmap v1269: 9 pgs: 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:02:43.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:43 smithi045 conmon[33655]: cluster 2021-07-31T02:02:42.080724+0000 mon.a (mon.0) 1738 : cluster [DBG] osdmap e959: 8 total, 8 up, 8 in 2021-07-31T02:02:43.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:43 smithi045 conmon[33655]: audit 2021-07-31T02:02:42.082234+0000 mon.c (mon.1) 61 : audit [INF] from='client.? 172.21.15.45:0/2197804790' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:02:43.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:43 smithi045 conmon[33655]: audit 2021-07-31T02:02:42.082796+0000 mon.a (mon.0) 1739 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:02:43.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:43 smithi045 conmon[39718]: cluster 2021-07-31T02:02:43.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:43 smithi045 conmon[39718]: 2021-07-31T02:02:43.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:43 smithi045 conmon[39718]: 2021-07-31T02:02:42.012557+0000 mgr.x (mgr 2021-07-31T02:02:43.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:43 smithi045 conmon[39718]: .24542) 665 : cluster [DBG] pgmap v1269: 9 pgs: 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:02:43.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:43 smithi045 conmon[39718]: cluster 2021-07-31T02:02:43.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:43 smithi045 conmon[39718]: 2021-07-31T02:02:42.080724+0000 mon.a (mon.0) 1738 : cluster [DBG] 2021-07-31T02:02:43.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:43 smithi045 conmon[39718]: osdmap e959: 8 total, 8 up, 8 in 2021-07-31T02:02:43.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:43 smithi045 conmon[39718]: audit 2021-07-31T02:02:42.082234 2021-07-31T02:02:43.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:43 smithi045 conmon[39718]: +0000 mon.c (mon.1) 61 : audit [INF] from='client.? 172.21.15.45:0/2197804790' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:02:43.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:43 smithi045 conmon[39718]: 2021-07-31T02:02:43.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:43 smithi045 conmon[39718]: audit 2021 2021-07-31T02:02:43.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:43 smithi045 conmon[39718]: -07-31T02:02:42.082796+0000 mon.a (mon.0) 1739 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:02:44.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:02:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:02:43.534Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:02:44.027 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:02:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:02:43.534Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:02:44.027 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:02:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:02:43.534Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:02:44.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:44 smithi154 conmon[34553]: audit 2021-07-31T02:02:43.082726+0000 mon.a (mon.0) 1740 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-11","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:02:44.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:44 smithi154 conmon[34553]: cluster 2021-07-31T02:02:43.083039+0000 mon.a (mon.0) 1741 : cluster [DBG] osdmap e960: 8 total, 8 up, 8 in 2021-07-31T02:02:44.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:44 smithi154 conmon[34553]: cluster 2021-07-31T02:02:44.013112+0000 mgr.x (mgr.24542) 666 : cluster [DBG] pgmap v1272: 41 pgs: 41 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 0 op/s 2021-07-31T02:02:44.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:44 smithi154 conmon[34553]: cluster 2021-07-31T02:02:44.084414+0000 mon.a (mon 2021-07-31T02:02:44.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:44 smithi154 conmon[34553]: .0) 1742 : cluster [DBG] osdmap e961: 8 total, 8 up, 8 in 2021-07-31T02:02:44.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:44 smithi045 conmon[33655]: audit 2021-07-31T02:02:43.082726+0000 mon.a (mon.0) 2021-07-31T02:02:44.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:44 smithi045 conmon[33655]: 1740 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-11","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:02:44.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:44 smithi045 conmon[33655]: cluster 2021-07-31T02:02:43.083039+0000 mon.a (mon.0) 1741 : cluster [DBG] osdmap e960: 8 total, 8 up, 8 in 2021-07-31T02:02:44.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:44 smithi045 conmon[33655]: cluster 2021-07-31T02:02:44.013112+0000 mgr.x (mgr 2021-07-31T02:02:44.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:44 smithi045 conmon[33655]: .24542) 666 : cluster [DBG] pgmap v1272: 41 pgs: 41 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 0 op/s 2021-07-31T02:02:44.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:44 smithi045 conmon[33655]: cluster 2021-07-31T02:02:44.084414+0000 2021-07-31T02:02:44.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:44 smithi045 conmon[33655]: mon.a (mon.0) 1742 : cluster [DBG] osdmap e961: 8 total, 8 up, 8 in 2021-07-31T02:02:44.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:44 smithi045 conmon[39718]: audit 2021-07-31T02:02:43.082726+0000 mon.a ( 2021-07-31T02:02:44.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:44 smithi045 conmon[39718]: mon.0) 1740 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-73799-11","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:02:44.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:44 smithi045 conmon[39718]: cluster 2021-07-31T02:02:43.083039+0000 mon.a (mon.0) 1741 2021-07-31T02:02:44.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:44 smithi045 conmon[39718]: : cluster [DBG] osdmap e960: 8 total, 8 up, 8 in 2021-07-31T02:02:44.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:44 smithi045 conmon[39718]: cluster 2021-07-31T02:02:44. 2021-07-31T02:02:44.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:44 smithi045 conmon[39718]: 013112+0000 mgr.x (mgr.24542) 666 : cluster [DBG] pgmap v1272: 41 pgs: 41 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 0 op/s 2021-07-31T02:02:44.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:44 smithi045 conmon[39718]: cluster 2021-07-31T02:02:44.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:44 smithi045 conmon[39718]: 2021-07-31T02:02:44.084414+0000 mon.a (mon.0) 1742 : cluster [DBG] osdmap e961: 8 total, 8 up, 8 in 2021-07-31T02:02:47.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:47 smithi154 conmon[34553]: cluster 2021-07-31T02:02:46.013678+0000 mgr.x (mgr.24542) 667 : cluster [DBG] 2021-07-31T02:02:47.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:47 smithi154 conmon[34553]: pgmap v1274: 41 pgs: 41 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 0 op/s 2021-07-31T02:02:47.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:47 smithi045 conmon[39718]: cluster 2021-07-31T02:02:46.013678+0000 mgr.x (mgr.24542) 2021-07-31T02:02:47.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:47 smithi045 conmon[39718]: 667 : cluster [DBG] pgmap v1274: 41 pgs: 41 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 0 op/s 2021-07-31T02:02:47.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:47 smithi045 conmon[33655]: cluster 2021-07-31T02:02:46.013678+0000 mgr.x (mgr.24542) 2021-07-31T02:02:47.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:47 smithi045 conmon[33655]: 667 : cluster [DBG] pgmap v1274: 41 pgs: 41 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 0 op/s 2021-07-31T02:02:49.078 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] ClsLock.TestExclusiveEphemeralStealExclusive (8000 ms) 2021-07-31T02:02:49.079 INFO:tasks.workunit.client.0.smithi045.stdout:[----------] 11 tests from ClsLock (68625 ms total) 2021-07-31T02:02:49.079 INFO:tasks.workunit.client.0.smithi045.stdout: 2021-07-31T02:02:49.079 INFO:tasks.workunit.client.0.smithi045.stdout:[----------] Global test environment tear-down 2021-07-31T02:02:49.079 INFO:tasks.workunit.client.0.smithi045.stdout:[==========] 11 tests from 1 test suite ran. (68625 ms total) 2021-07-31T02:02:49.080 INFO:tasks.workunit.client.0.smithi045.stdout:[ PASSED ] 11 tests. 2021-07-31T02:02:49.081 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-07-31T02:02:49.082 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2021-07-31T02:02:49.152 INFO:tasks.workunit:Running workunit cls/test_cls_log.sh... 2021-07-31T02:02:49.153 DEBUG:teuthology.orchestra.run.smithi045:workunit test cls/test_cls_log.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=pacific TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_log.sh 2021-07-31T02:02:49.224 INFO:tasks.workunit.client.0.smithi045.stdout:Running main() from gmock_main.cc 2021-07-31T02:02:49.225 INFO:tasks.workunit.client.0.smithi045.stdout:[==========] Running 4 tests from 1 test suite. 2021-07-31T02:02:49.225 INFO:tasks.workunit.client.0.smithi045.stdout:[----------] Global test environment set-up. 2021-07-31T02:02:49.225 INFO:tasks.workunit.client.0.smithi045.stdout:[----------] 4 tests from cls_log 2021-07-31T02:02:49.225 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] cls_log.test_log_add_same_time 2021-07-31T02:02:49.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:49 smithi154 conmon[34553]: cluster 2021-07-31T02:02:48.014189+0000 mgr.x (mgr.24542) 668 : cluster [DBG] pgmap v1275: 41 pgs: 41 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 0 op/s 2021-07-31T02:02:49.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:49 smithi045 conmon[39718]: cluster 2021-07-31T02:02:48.014189+0000 mgr.x (mgr.24542) 668 : 2021-07-31T02:02:49.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:49 smithi045 conmon[39718]: cluster [DBG] pgmap v1275: 41 pgs: 41 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 0 op/s 2021-07-31T02:02:49.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:49 smithi045 conmon[33655]: cluster 2021-07-31T02:02:49.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:49 smithi045 conmon[33655]: 2021-07-31T02:02:48.014189 2021-07-31T02:02:49.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:49 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 668 : cluster [DBG] pgmap v1275: 41 pgs: 41 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 0 op/s 2021-07-31T02:02:50.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:50 smithi154 conmon[34553]: cluster 2021-07-31T02:02:49.074981+0000 mon.a (mon.0) 1743 : cluster 2021-07-31T02:02:50.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:50 smithi154 conmon[34553]: [DBG] osdmap e962: 8 total, 8 up, 8 in 2021-07-31T02:02:50.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:50 smithi045 conmon[33655]: cluster 2021-07-31T02:02:49.074981+0000 mon.a 2021-07-31T02:02:50.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:50 smithi045 conmon[33655]: (mon.0) 1743 : cluster [DBG] osdmap e962: 8 total, 8 up, 8 in 2021-07-31T02:02:50.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:50 smithi045 conmon[39718]: cluster 2021-07-31T02:02:49.074981+0000 mon.a (mon. 2021-07-31T02:02:50.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:50 smithi045 conmon[39718]: 0) 1743 : cluster [DBG] osdmap e962: 8 total, 8 up, 8 in 2021-07-31T02:02:51.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:51 smithi154 conmon[34553]: cluster 2021-07-31T02:02:50.014714+0000 mgr.x ( 2021-07-31T02:02:51.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:51 smithi154 conmon[34553]: mgr.24542) 669 : cluster [DBG] pgmap v1277: 9 pgs: 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:02:51.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:51 smithi154 conmon[34553]: cluster 2021-07-31T02:02:50.086018+0000 mon.a (mon.0) 1744 : cluster [DBG] osdmap e963: 8 total, 8 up, 8 in 2021-07-31T02:02:51.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:51 smithi154 conmon[34553]: 2021-07-31T02:02:51.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:51 smithi154 conmon[34553]: audit 2021-07-31T 2021-07-31T02:02:51.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:51 smithi154 conmon[34553]: 02:02:50.087274+0000 mon.a (mon.0) 1745 : audit [INF] from='client.? 172.21.15.45:0/3001847515' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74208-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:02:51.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:51 smithi045 conmon[39718]: cluster 2021-07-31T02:02:50.014714+0000 mgr.x (mgr.24542) 2021-07-31T02:02:51.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:51 smithi045 conmon[39718]: 669 : cluster [DBG] pgmap v1277: 9 pgs: 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:02:51.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:51 smithi045 conmon[39718]: cluster 2021-07-31T02:02:50.086018+0000 mon.a (mon.0) 1744 : cluster [DBG] osdmap e963: 8 total, 8 up, 8 in 2021-07-31T02:02:51.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:51 smithi045 conmon[39718]: 2021-07-31T02:02:51.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:51 smithi045 conmon[39718]: audit 2021-07-31T02:02:50.087274+0000 mon.a (mon.0) 2021-07-31T02:02:51.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:51 smithi045 conmon[39718]: 1745 : audit [INF] from='client.? 172.21.15.45:0/3001847515' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74208-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:02:51.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:51 smithi045 conmon[33655]: cluster 2021-07-31T02:02:50.014714+0000 mgr.x (mgr.24542) 669 : cluster [DBG] 2021-07-31T02:02:51.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:51 smithi045 conmon[33655]: pgmap v1277: 9 pgs: 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:02:51.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:51 smithi045 conmon[33655]: cluster 2021-07-31T02:02:50.086018+0000 mon.a (mon.0) 1744 : cluster [DBG] osdmap e963: 8 total, 8 up, 8 in 2021-07-31T02:02:51.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:51 smithi045 conmon[33655]: audit 2021-07-31T02:02:50.087274+0000 2021-07-31T02:02:51.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:51 smithi045 conmon[33655]: mon.a (mon.0) 1745 : audit [INF] from='client.? 172.21.15.45:0/3001847515' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74208-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:02:51.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:51 smithi045 conmon[33655]: 2021-07-31T02:02:52.099 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] cls_log.test_log_add_same_time (2875 ms) 2021-07-31T02:02:52.099 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] cls_log.test_log_add_different_time 2021-07-31T02:02:52.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:52 smithi154 conmon[34553]: audit 2021-07-31T02:02:51.094222+0000 mon.a (mon.0) 1746 2021-07-31T02:02:52.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:52 smithi154 conmon[34553]: : audit [INF] from='client.? 172.21.15.45:0/3001847515' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74208-1","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:02:52.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:52 smithi154 conmon[34553]: cluster 2021-07-31T02:02:51.094518+0000 mon.a (mon.0) 1747 : cluster [DBG] osdmap e964: 8 total, 8 up, 8 in 2021-07-31T02:02:52.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:52 smithi154 conmon[34553]: cluster 2021-07-31T02:02:52.015254+0000 mgr.x (mgr.24542) 2021-07-31T02:02:52.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:52 smithi154 conmon[34553]: 670 : cluster [DBG] pgmap v1280: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:02:52.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:52 smithi045 conmon[33655]: audit 2021-07-31T02:02:51.094222+0000 mon.a (mon.0) 1746 : audit [INF] from='client.? 172.21.15.45:0/3001847515' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74208-1","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:02:52.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:52 smithi045 conmon[33655]: cluster 2021-07-31T02:02:51.094518+0000 mon.a (mon.0) 1747 : cluster [DBG] osdmap e964: 8 total, 8 up, 8 in 2021-07-31T02:02:52.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:52 smithi045 conmon[33655]: cluster 2021-07-31T02:02:52.015254+0000 mgr.x (mgr.24542) 670 : cluster [DBG] pgmap v1280: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:02:52.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:52 smithi045 conmon[39718]: audit 2021-07-31T02:02:52.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:52 smithi045 conmon[39718]: 2021-07-31T02:02:52.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:52 smithi045 conmon[39718]: 2021 2021-07-31T02:02:52.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:52 smithi045 conmon[39718]: - 2021-07-31T02:02:52.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:52 smithi045 conmon[39718]: 07 2021-07-31T02:02:52.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:52 smithi045 conmon[39718]: - 2021-07-31T02:02:52.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:52 smithi045 conmon[39718]: 31 2021-07-31T02:02:52.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:52 smithi045 conmon[39718]: T 2021-07-31T02:02:52.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:52 smithi045 conmon[39718]: 02 2021-07-31T02:02:52.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:52 smithi045 conmon[39718]: : 2021-07-31T02:02:52.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:52 smithi045 conmon[39718]: 02:51.094222+0000 mon.a (mon.0) 2021-07-31T02:02:52.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:52 smithi045 conmon[39718]: 1746 : audit [INF] from='client.? 172.21.15.45:0/3001847515' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74208-1","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:02:52.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:52 smithi045 conmon[39718]: cluster 2021-07 2021-07-31T02:02:52.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:52 smithi045 conmon[39718]: -31T02:02:51. 2021-07-31T02:02:52.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:52 smithi045 conmon[39718]: 094518+0000 mon.a (mon.0 2021-07-31T02:02:52.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:52 smithi045 conmon[39718]: ) 1747 : cluster [DBG] osdmap e964: 8 total, 8 up, 8 in 2021-07-31T02:02:52.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:52 smithi045 conmon[39718]: cluster 2021-07 2021-07-31T02:02:52.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:52 smithi045 conmon[39718]: -31T02:02:52.015254+0000 mgr.x (mgr.24542) 670 2021-07-31T02:02:52.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:52 smithi045 conmon[39718]: : cluster [DBG] pgmap v1280: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:02:53.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:53 smithi154 conmon[34553]: cluster 2021-07-31T02:02:52.096021+0000 mon.a (mon.0) 1748 : cluster [DBG] 2021-07-31T02:02:53.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:53 smithi154 conmon[34553]: osdmap e965: 8 total, 8 up, 8 in 2021-07-31T02:02:53.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:53 smithi045 conmon[39718]: cluster 2021-07-31T02:02:52.096021+0000 mon.a (mon.0) 1748 : cluster [DBG] osdmap e965: 8 total, 8 up, 8 in 2021-07-31T02:02:53.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:53 smithi045 conmon[33655]: cluster 2021-07-31T02:02:53.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:53 smithi045 conmon[33655]: 2021-07-31T02:02:53.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:53 smithi045 conmon[33655]: 2021 2021-07-31T02:02:53.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:53 smithi045 conmon[33655]: - 2021-07-31T02:02:53.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:53 smithi045 conmon[33655]: 07-31T02:02:52.096021+0000 2021-07-31T02:02:53.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:53 smithi045 conmon[33655]: mon.a (mon.0) 1748 : cluster [DBG] osdmap e965: 8 total, 8 up, 8 in 2021-07-31T02:02:54.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:02:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:02:53.534Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:02:54.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:02:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:02:53.535Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:02:54.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:02:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:02:53.535Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:02:54.084 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:02:54.374 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:02:54.375 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 17m ago 18m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:02:54.375 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 15m ago 18m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:02:54.375 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 15m ago 21m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:02:54.375 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 17m ago 23m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:02:54.376 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 17m ago 24m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:02:54.376 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 15m ago 22m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:02:54.376 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 17m ago 22m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:02:54.376 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 17m ago 18m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:02:54.376 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 15m ago 18m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:02:54.377 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 17m ago 21m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:02:54.377 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 17m ago 21m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:02:54.377 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 17m ago 20m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:02:54.377 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 17m ago 20m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:02:54.378 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 15m ago 20m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:02:54.378 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 15m ago 19m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:02:54.378 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 15m ago 19m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:02:54.378 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 15m ago 19m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:02:54.378 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 15m ago 18m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:02:54.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:54 smithi154 conmon[34553]: cluster 2021-07-31T02:02:54.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:54 smithi154 conmon[34553]: 2021-07-31T02:02:54.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:54 smithi154 conmon[34553]: 2021 2021-07-31T02:02:54.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:54 smithi154 conmon[34553]: - 2021-07-31T02:02:54.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:54 smithi154 conmon[34553]: 07-31T02:02:53.102491+0000 mon.a (mon 2021-07-31T02:02:54.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:54 smithi154 conmon[34553]: .0) 1749 : cluster [DBG] osdmap e966: 8 total, 8 up, 8 in 2021-07-31T02:02:54.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:54 smithi154 conmon[34553]: audit 2021- 2021-07-31T02:02:54.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:54 smithi154 conmon[34553]: 07-31T02:02:53. 2021-07-31T02:02:54.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:54 smithi154 conmon[34553]: 105244+0000 mon.c (mon.1) 62 : audit 2021-07-31T02:02:54.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:54 smithi154 conmon[34553]: [INF] from='client.? 172.21.15.45:0/4202301108' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74208-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:02:54.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:54 smithi154 conmon[34553]: audit 2021-07-31 2021-07-31T02:02:54.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:54 smithi154 conmon[34553]: T02:02:53.110499+0000 mon.a (mon.0 2021-07-31T02:02:54.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:54 smithi154 conmon[34553]: ) 1750 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74208-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:02:54.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:54 smithi154 conmon[34553]: cluster 2021-07-31T02:02:54.015747+0000 mgr.x (mgr.24542) 671 : cluster [DBG] pgmap v1283: 41 pgs: 32 creating+peering, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:02:54.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:54 smithi154 conmon[34553]: audit 2021-07-31T02:02:54.071804+0000 2021-07-31T02:02:54.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:54 smithi154 conmon[34553]: mgr.x (mgr.24542) 672 : audit [DBG] from='client.25447 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:02:54.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:54 smithi045 conmon[39718]: cluster 2021-07-31T02:02:54.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:54 smithi045 conmon[39718]: 2021-07-31T02:02:54.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:54 smithi045 conmon[39718]: 2021 2021-07-31T02:02:54.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:54 smithi045 conmon[39718]: - 2021-07-31T02:02:54.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:54 smithi045 conmon[39718]: 07-31T02:02:53.102491 2021-07-31T02:02:54.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:54 smithi045 conmon[39718]: +0000 mon.a (mon.0) 1749 : cluster [DBG] osdmap e966: 8 total, 8 up, 8 in 2021-07-31T02:02:54.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:54 smithi045 conmon[39718]: audit 2021-07-31T02 2021-07-31T02:02:54.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:54 smithi045 conmon[39718]: :02:53.105244+0000 mon.c (mon.1) 62 2021-07-31T02:02:54.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:54 smithi045 conmon[39718]: : audit [INF] from='client.? 172.21.15.45:0/4202301108' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74208-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:02:54.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:54 smithi045 conmon[39718]: audit 2021-07-31T02: 2021-07-31T02:02:54.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:54 smithi045 conmon[39718]: 02:53.110499+0000 mon.a (mon.0) 2021-07-31T02:02:54.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:54 smithi045 conmon[39718]: 1750 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74208-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:02:54.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:54 smithi045 conmon[39718]: cluster 2021-07-31 2021-07-31T02:02:54.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:54 smithi045 conmon[39718]: T02:02:54.015747+0000 mgr.x (mgr. 2021-07-31T02:02:54.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:54 smithi045 conmon[39718]: 24542) 671 : cluster [DBG] pgmap v1283: 41 pgs: 32 creating+peering, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:02:54.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:54 smithi045 conmon[39718]: audit 2021-07-31T02: 2021-07-31T02:02:54.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:54 smithi045 conmon[39718]: 02:54.071804+0000 mgr.x (mgr.24542 2021-07-31T02:02:54.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:54 smithi045 conmon[39718]: ) 672 : audit [DBG] from='client.25447 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:02:54.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:54 smithi045 conmon[33655]: cluster 2021-07-31T02:02:53.102491+0000 mon.a (mon. 2021-07-31T02:02:54.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:54 smithi045 conmon[33655]: 0) 1749 : cluster [DBG] osdmap e966: 8 total, 8 up, 8 in 2021-07-31T02:02:54.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:54 smithi045 conmon[33655]: audit 2021-07-31T02:02:53.105244+0000 mon.c (mon.1) 62 : audit [INF] from='client.? 172.21.15.45:0/4202301108' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74208-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:02:54.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:54 smithi045 conmon[33655]: audit 2021-07-31T02:02:54.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:54 smithi045 conmon[33655]: 2021-07-31T02:02:53.110499+0000 mon.a (mon.0) 1750 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74208-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:02:54.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:54 smithi045 conmon[33655]: cluster 2021-07-31T02:02:54.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:54 smithi045 conmon[33655]: 2021-07-31T02:02:54. 2021-07-31T02:02:54.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:54 smithi045 conmon[33655]: 015747+0000 mgr.x (mgr.24542) 671 2021-07-31T02:02:54.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:54 smithi045 conmon[33655]: : cluster [DBG] pgmap v1283: 41 pgs: 32 creating+peering, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:02:54.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:54 smithi045 conmon[33655]: audit 2021-07-31T 2021-07-31T02:02:54.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:54 smithi045 conmon[33655]: 02:02:54.071804+0000 mgr.x ( 2021-07-31T02:02:54.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:54 smithi045 conmon[33655]: mgr.24542) 672 : audit [DBG] from='client.25447 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:02:54.713 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:02:54.713 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:02:54.713 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:02:54.713 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:02:54.714 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:02:54.714 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:02:54.714 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:02:54.714 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:02:54.715 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:02:54.715 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:02:54.715 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:02:54.715 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:02:54.715 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:02:54.716 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:02:54.716 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:02:54.717 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:02:54.717 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:02:55.119 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] cls_log.test_log_add_different_time (3020 ms) 2021-07-31T02:02:55.119 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] cls_log.trim_by_time 2021-07-31T02:02:55.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:55 smithi154 conmon[34553]: audit 2021-07-31T02:02:55.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:55 smithi154 conmon[34553]: 2021-07-31T02:02:54 2021-07-31T02:02:55.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:55 smithi154 conmon[34553]: .114692+0000 mon.a (mon.0) 1751 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74208-2","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:02:55.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:55 smithi154 conmon[34553]: cluster 2021-07-31T02:02:54.114918+0000 2021-07-31T02:02:55.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:55 smithi154 conmon[34553]: mon.a (mon.0) 1752 : cluster [DBG] osdmap e967: 8 total, 8 up, 8 in 2021-07-31T02:02:55.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:55 smithi154 conmon[34553]: audit 2021-07-31T02:02:55.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:55 smithi154 conmon[34553]: 2021-07-31T02:02:54.366108+0000 mgr.x ( 2021-07-31T02:02:55.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:55 smithi154 conmon[34553]: mgr.24542) 673 : audit [DBG] from='client.15546 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:02:55.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:55 smithi154 conmon[34553]: audit 2021- 2021-07-31T02:02:55.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:55 smithi154 conmon[34553]: 07-31T02:02:54.712626+0000 mon.a ( 2021-07-31T02:02:55.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:55 smithi154 conmon[34553]: mon.0) 1753 : audit [DBG] from='client.? 172.21.15.45:0/811668736' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:02:55.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:55 smithi045 conmon[33655]: audit 2021-07-31T02:02:54.114692+0000 mon.a 2021-07-31T02:02:55.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:55 smithi045 conmon[33655]: (mon.0) 1751 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74208-2","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:02:55.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:55 smithi045 conmon[33655]: cluster 2021-07-31T02:02:54.114918+0000 mon.a (mon.0) 1752 : cluster [DBG] osdmap e967: 8 total, 8 up, 8 in 2021-07-31T02:02:55.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:55 smithi045 conmon[33655]: audit 2021-07-31T02:02:54.366108+0000 mgr.x (mgr.24542) 673 : audit [DBG] from='client.15546 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:02:55.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:55 smithi045 conmon[33655]: audit 2021-07-31T02:02:54.712626+0000 mon.a (mon.0) 1753 : audit [DBG] from='client.? 172.21.15.45:0/811668736' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:02:55.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:55 smithi045 conmon[39718]: audit 2021-07-31T02:02:55.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:55 smithi045 conmon[39718]: 2021-07-31T02:02:55.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:55 smithi045 conmon[39718]: 2021-07-31T02:02:54 2021-07-31T02:02:55.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:55 smithi045 conmon[39718]: .114692+0000 mon.a (mon.0) 1751 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74208-2","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:02:55.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:55 smithi045 conmon[39718]: cluster 2021-07- 2021-07-31T02:02:55.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:55 smithi045 conmon[39718]: 31T02:02:54.114918+0000 2021-07-31T02:02:55.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:55 smithi045 conmon[39718]: mon.a (mon.0) 1752 : cluster [DBG] osdmap e967: 8 total, 8 up, 8 in 2021-07-31T02:02:55.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:55 smithi045 conmon[39718]: 2021-07-31T02:02:55.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:55 smithi045 conmon[39718]: audit 2021-07-31T02 2021-07-31T02:02:55.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:55 smithi045 conmon[39718]: :02:54.366108+0000 mgr.x (mgr. 2021-07-31T02:02:55.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:55 smithi045 conmon[39718]: 24542) 673 : audit [DBG] from='client.15546 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:02:55.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:55 smithi045 conmon[39718]: audit 2021-07 2021-07-31T02:02:55.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:55 smithi045 conmon[39718]: -31T02:02:54.712626+0000 mon.a (mon 2021-07-31T02:02:55.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:55 smithi045 conmon[39718]: .0) 1753 : audit [DBG] from='client.? 172.21.15.45:0/811668736' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:02:56.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:56 smithi154 conmon[34553]: cluster 2021-07-31T02:02:55.116209+0000 mon.a (mon.0) 1754 : cluster [DBG] 2021-07-31T02:02:56.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:56 smithi154 conmon[34553]: osdmap e968: 8 total, 8 up, 8 in 2021-07-31T02:02:56.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:56 smithi154 conmon[34553]: cluster 2021-07-31T02:02:56.016137+0000 mgr.x (mgr.24542) 674 : cluster [DBG] pgmap v1286: 9 pgs: 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:02:56.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:56 smithi045 conmon[33655]: cluster 2021-07-31T02:02:55.116209+0000 mon.a 2021-07-31T02:02:56.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:56 smithi045 conmon[33655]: (mon.0) 1754 : cluster [DBG] osdmap e968: 8 total, 8 up, 8 in 2021-07-31T02:02:56.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:56 smithi045 conmon[33655]: cluster 2021-07-31T02:02:56.016137+0000 mgr.x (mgr.24542) 674 : cluster [DBG] pgmap v1286: 9 pgs: 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:02:56.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:56 smithi045 conmon[39718]: cluster 2021-07-31T02:02:55. 2021-07-31T02:02:56.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:56 smithi045 conmon[39718]: 116209+0000 mon.a (mon.0) 1754 : cluster [DBG] osdmap e968: 8 total, 8 up, 8 in 2021-07-31T02:02:56.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:56 smithi045 conmon[39718]: cluster 2021-07-31T02:02:56.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:56 smithi045 conmon[39718]: 2021-07-31T02:02:56.016137+0000 mgr.x (mgr.24542) 2021-07-31T02:02:56.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:56 smithi045 conmon[39718]: 674 : cluster [DBG] pgmap v1286: 9 pgs: 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:02:57.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:57 smithi154 conmon[34553]: cluster 2021-07-31T02:02:56.127521+0000 mon.a (mon.0) 1755 : cluster [DBG] osdmap e969: 8 total, 8 up, 8 in 2021-07-31T02:02:57.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:57 smithi154 conmon[34553]: audit 2021-07-31T02:02:56.128583+0000 mon.a (mon.0) 1756 : audit [INF] from='client.? 172.21.15.45:0/497089184' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74208-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:02:57.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:57 smithi045 conmon[39718]: cluster 2021-07-31T02:02:56.127521+0000 mon.a (mon.0) 1755 2021-07-31T02:02:57.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:57 smithi045 conmon[39718]: : cluster [DBG] osdmap e969: 8 total, 8 up, 8 in 2021-07-31T02:02:57.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:57 smithi045 conmon[39718]: audit 2021-07-31T02:02:56.128583+0000 mon.a (mon.0) 1756 : 2021-07-31T02:02:57.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:57 smithi045 conmon[39718]: audit [INF] from='client.? 172.21.15.45:0/497089184' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74208-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:02:57.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:57 smithi045 conmon[33655]: cluster 2021-07-31T02:02:57.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:57 smithi045 conmon[33655]: 2021-07-31T02:02:57.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:57 smithi045 conmon[33655]: 2021 2021-07-31T02:02:57.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:57 smithi045 conmon[33655]: - 2021-07-31T02:02:57.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:57 smithi045 conmon[33655]: 07 2021-07-31T02:02:57.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:57 smithi045 conmon[33655]: - 2021-07-31T02:02:57.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:57 smithi045 conmon[33655]: 31 2021-07-31T02:02:57.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:57 smithi045 conmon[33655]: T 2021-07-31T02:02:57.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:57 smithi045 conmon[33655]: 02 2021-07-31T02:02:57.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:57 smithi045 conmon[33655]: : 2021-07-31T02:02:57.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:57 smithi045 conmon[33655]: 02 2021-07-31T02:02:57.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:57 smithi045 conmon[33655]: : 2021-07-31T02:02:57.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:57 smithi045 conmon[33655]: 56 2021-07-31T02:02:57.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:57 smithi045 conmon[33655]: . 2021-07-31T02:02:57.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:57 smithi045 conmon[33655]: 127521 2021-07-31T02:02:57.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:57 smithi045 conmon[33655]: +0000 2021-07-31T02:02:57.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:57 smithi045 conmon[33655]: 2021-07-31T02:02:57.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:57 smithi045 conmon[33655]: mon.a 2021-07-31T02:02:57.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:57 smithi045 conmon[33655]: ( 2021-07-31T02:02:57.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:57 smithi045 conmon[33655]: mon 2021-07-31T02:02:57.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:57 smithi045 conmon[33655]: . 2021-07-31T02:02:57.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:57 smithi045 conmon[33655]: 0 2021-07-31T02:02:57.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:57 smithi045 conmon[33655]: ) 2021-07-31T02:02:57.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:57 smithi045 conmon[33655]: 1755 2021-07-31T02:02:57.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:57 smithi045 conmon[33655]: : 2021-07-31T02:02:57.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:57 smithi045 conmon[33655]: cluster 2021-07-31T02:02:57.535 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:57 smithi045 conmon[33655]: 2021-07-31T02:02:57.535 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:57 smithi045 conmon[33655]: [DBG] 2021-07-31T02:02:57.535 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:57 smithi045 conmon[33655]: 2021-07-31T02:02:57.535 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:57 smithi045 conmon[33655]: osdmap e969: 8 total, 8 up, 8 in 2021-07-31T02:02:57.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:57 smithi045 conmon[33655]: 2021-07-31T02:02:57.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:57 smithi045 conmon[33655]: audit 2021-07-31T02:02:57.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:57 smithi045 conmon[33655]: 2021-07-31T02:02:57.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:57 smithi045 conmon[33655]: 2021 2021-07-31T02:02:57.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:57 smithi045 conmon[33655]: - 2021-07-31T02:02:57.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:57 smithi045 conmon[33655]: 07 2021-07-31T02:02:57.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:57 smithi045 conmon[33655]: - 2021-07-31T02:02:57.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:57 smithi045 conmon[33655]: 31 2021-07-31T02:02:57.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:57 smithi045 conmon[33655]: T 2021-07-31T02:02:57.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:57 smithi045 conmon[33655]: 02 2021-07-31T02:02:57.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:57 smithi045 conmon[33655]: : 2021-07-31T02:02:57.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:57 smithi045 conmon[33655]: 02 2021-07-31T02:02:57.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:57 smithi045 conmon[33655]: : 2021-07-31T02:02:57.539 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:57 smithi045 conmon[33655]: 56 2021-07-31T02:02:57.539 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:57 smithi045 conmon[33655]: . 2021-07-31T02:02:57.539 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:57 smithi045 conmon[33655]: 128583 2021-07-31T02:02:57.539 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:57 smithi045 conmon[33655]: +0000 2021-07-31T02:02:57.539 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:57 smithi045 conmon[33655]: 2021-07-31T02:02:57.540 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:57 smithi045 conmon[33655]: mon.a 2021-07-31T02:02:57.540 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:57 smithi045 conmon[33655]: ( 2021-07-31T02:02:57.540 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:57 smithi045 conmon[33655]: mon 2021-07-31T02:02:57.540 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:57 smithi045 conmon[33655]: . 2021-07-31T02:02:57.540 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:57 smithi045 conmon[33655]: 0 2021-07-31T02:02:57.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:57 smithi045 conmon[33655]: ) 2021-07-31T02:02:57.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:57 smithi045 conmon[33655]: 1756 : audit [INF] from='client.? 172.21.15.45:0/497089184' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74208-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:02:58.145 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] cls_log.trim_by_time (3026 ms) 2021-07-31T02:02:58.145 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] cls_log.trim_by_marker 2021-07-31T02:02:58.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:58 smithi154 conmon[34553]: audit 2021-07-31T02:02:57.129152+0000 mon.a (mon.0) 1757 : audit [INF] from='client.? 172.21.15.45:0/497089184' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74208-3","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:02:58.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:58 smithi154 conmon[34553]: 2021-07-31T02:02:58.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:58 smithi154 conmon[34553]: cluster 2021-07-31T02:02:57.129460+0000 mon.a (mon.0) 1758 : cluster [DBG] osdmap e970: 8 total, 8 up, 8 in 2021-07-31T02:02:58.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:58 smithi154 conmon[34553]: cluster 2021-07-31T02:02:58.016548+0000 2021-07-31T02:02:58.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:58 smithi154 conmon[34553]: mgr.x (mgr.24542) 675 : cluster [DBG] pgmap v1289: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:02:58.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:58 smithi045 conmon[39718]: audit 2021-07-31T02:02:57.129152+0000 mon.a (mon 2021-07-31T02:02:58.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:58 smithi045 conmon[39718]: .0) 1757 : audit [INF] from='client.? 172.21.15.45:0/497089184' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74208-3","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:02:58.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:58 smithi045 conmon[39718]: cluster 2021-07-31T02:02:57.129460+0000 mon.a (mon.0) 2021-07-31T02:02:58.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:58 smithi045 conmon[39718]: 1758 : cluster [DBG] osdmap e970: 8 total, 8 up, 8 in 2021-07-31T02:02:58.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:58 smithi045 conmon[39718]: cluster 2021 2021-07-31T02:02:58.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:58 smithi045 conmon[39718]: -07-31T02:02:58.016548+0000 mgr.x ( 2021-07-31T02:02:58.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:58 smithi045 conmon[39718]: mgr.24542) 675 : cluster [DBG] pgmap v1289: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:02:58.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:58 smithi045 conmon[33655]: audit 2021-07-31T02:02:57.129152+0000 mon.a (mon.0) 1757 2021-07-31T02:02:58.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:58 smithi045 conmon[33655]: : audit [INF] from='client.? 172.21.15.45:0/497089184' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74208-3","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:02:58.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:58 smithi045 conmon[33655]: cluster 2021-07-31T02:02:57.129460+0000 mon.a (mon.0) 1758 : cluster [DBG] osdmap e970: 8 total, 8 up, 8 in 2021-07-31T02:02:58.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:58 smithi045 conmon[33655]: cluster 2021-07-31T02:02 2021-07-31T02:02:58.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:58 smithi045 conmon[33655]: :58.016548+0000 mgr.x (mgr.24542) 675 : cluster [DBG] pgmap v1289: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:02:59.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:59 smithi154 conmon[34553]: cluster 2021-07-31T02:02:58.141886+0000 mon.a (mon.0) 1759 : 2021-07-31T02:02:59.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:02:59 smithi154 conmon[34553]: cluster [DBG] osdmap e971: 8 total, 8 up, 8 in 2021-07-31T02:02:59.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:59 smithi045 conmon[39718]: cluster 2021-07-31T02:02:58.141886+0000 mon.a (mon. 2021-07-31T02:02:59.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:02:59 smithi045 conmon[39718]: 0) 1759 : cluster [DBG] osdmap e971: 8 total, 8 up, 8 in 2021-07-31T02:02:59.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:59 smithi045 conmon[33655]: cluster 2021-07-31T02:02:58.141886 2021-07-31T02:02:59.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:02:59 smithi045 conmon[33655]: +0000 mon.a (mon.0) 1759 : cluster [DBG] osdmap e971: 8 total, 8 up, 8 in 2021-07-31T02:03:00.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:00 smithi154 conmon[34553]: cluster 2021-07-31T02:02:59.144308+0000 mon.a (mon.0) 1760 : cluster [DBG] 2021-07-31T02:03:00.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:00 smithi154 conmon[34553]: osdmap e972: 8 total, 8 up, 8 in 2021-07-31T02:03:00.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:00 smithi154 conmon[34553]: audit 2021-07-31T02:02:59.146851+0000 mon.a (mon.0) 1761 : audit [INF] from='client.? 172.21.15.45:0/3037969894' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74208-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:03:00.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:00 smithi154 conmon[34553]: cluster 2021-07-31T02:03:00.017030+0000 mgr.x (mgr.24542) 676 : cluster [DBG] pgmap v1292: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:03:00.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:00 smithi154 conmon[34553]: audit 2021-07-31T02:03:00.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:00 smithi154 conmon[34553]: 2021-07-31T02:03:00.145186+0000 mon.a (mon.0) 1762 : audit [INF] from='client.? 172.21.15.45:0/3037969894' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74208-4","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:03:00.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:00 smithi154 conmon[34553]: cluster 2021-07-31T02:03:00.145446+0000 2021-07-31T02:03:00.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:00 smithi154 conmon[34553]: mon.a (mon.0) 1763 : cluster [DBG] osdmap e973: 8 total, 8 up, 8 in 2021-07-31T02:03:00.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:00 smithi045 conmon[33655]: cluster 2021-07-31T02:02:59. 2021-07-31T02:03:00.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:00 smithi045 conmon[33655]: 144308+0000 mon.a (mon.0) 1760 : cluster [DBG] osdmap e972: 8 total, 8 up, 8 in 2021-07-31T02:03:00.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:00 smithi045 conmon[33655]: audit 2021-07-31T02:03:00.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:00 smithi045 conmon[33655]: 2021-07-31T02:02:59.146851+0000 mon.a (mon.0) 1761 : audit [INF] from='client.? 172.21.15.45:0/3037969894' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74208-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:03:00.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:00 smithi045 conmon[33655]: cluster 2021-07-31T02:03:00.017030+0000 2021-07-31T02:03:00.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:00 smithi045 conmon[33655]: mgr.x (mgr.24542) 676 : cluster [DBG] pgmap v1292: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:03:00.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:00 smithi045 conmon[33655]: audit 2021-07-31T 2021-07-31T02:03:00.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:00 smithi045 conmon[33655]: 02:03:00.145186+0000 mon.a (mon.0) 2021-07-31T02:03:00.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:00 smithi045 conmon[33655]: 1762 : audit [INF] from='client.? 172.21.15.45:0/3037969894' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74208-4","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:03:00.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:00 smithi045 conmon[33655]: cluster 2021-07-31T02:03:00.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:00 smithi045 conmon[33655]: 2021-07-31T 2021-07-31T02:03:00.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:00 smithi045 conmon[33655]: 02:03:00.145446+0000 2021-07-31T02:03:00.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:00 smithi045 conmon[33655]: mon.a (mon.0) 1763 : cluster [DBG] 2021-07-31T02:03:00.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:00 smithi045 conmon[33655]: osdmap e973: 8 total, 8 up, 8 in 2021-07-31T02:03:00.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:00 smithi045 conmon[39718]: cluster 2021-07-31T02:02: 2021-07-31T02:03:00.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:00 smithi045 conmon[39718]: 59.144308+0000 mon.a (mon.0) 1760 : cluster [DBG] osdmap e972: 8 total, 8 up, 8 in 2021-07-31T02:03:00.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:00 smithi045 conmon[39718]: audit 2021- 2021-07-31T02:03:00.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:00 smithi045 conmon[39718]: 07-31T02:02:59.146851+0000 mon.a (mon. 2021-07-31T02:03:00.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:00 smithi045 conmon[39718]: 0) 1761 : audit [INF] 2021-07-31T02:03:00.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:00 smithi045 conmon[39718]: from='client.? 172.21.15.45:0/3037969894' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74208-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:03:00.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:00 smithi045 conmon[39718]: cluster 2021-07 2021-07-31T02:03:00.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:00 smithi045 conmon[39718]: -31T02:03:00.017030+0000 mgr.x (mgr. 2021-07-31T02:03:00.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:00 smithi045 conmon[39718]: 24542) 676 : cluster [DBG] pgmap v1292: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:03:00.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:00 smithi045 conmon[39718]: 2021-07-31T02:03:00.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:00 smithi045 conmon[39718]: audit 2021-07-31T02:03:00.145186+0000 mon.a (mon 2021-07-31T02:03:00.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:00 smithi045 conmon[39718]: .0) 1762 : audit [INF] 2021-07-31T02:03:00.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:00 smithi045 conmon[39718]: from='client.? 172.21.15.45:0/3037969894' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74208-4","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:03:00.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:00 smithi045 conmon[39718]: cluster 2021-07-31T02:03:00.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:00 smithi045 conmon[39718]: 2021-07 2021-07-31T02:03:00.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:00 smithi045 conmon[39718]: -31T02:03:00. 2021-07-31T02:03:00.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:00 smithi045 conmon[39718]: 145446+0000 mon.a (mon.0 2021-07-31T02:03:00.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:00 smithi045 conmon[39718]: ) 1763 : cluster [DBG] osdmap e973: 8 total, 8 up, 8 in 2021-07-31T02:03:00.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:00 smithi045 conmon[39718]: 2021-07-31T02:03:01.159 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] cls_log.trim_by_marker (3014 ms) 2021-07-31T02:03:01.160 INFO:tasks.workunit.client.0.smithi045.stdout:[----------] 4 tests from cls_log (11935 ms total) 2021-07-31T02:03:01.160 INFO:tasks.workunit.client.0.smithi045.stdout: 2021-07-31T02:03:01.160 INFO:tasks.workunit.client.0.smithi045.stdout:[----------] Global test environment tear-down 2021-07-31T02:03:01.160 INFO:tasks.workunit.client.0.smithi045.stdout:[==========] 4 tests from 1 test suite ran. (11935 ms total) 2021-07-31T02:03:01.160 INFO:tasks.workunit.client.0.smithi045.stdout:[ PASSED ] 4 tests. 2021-07-31T02:03:01.162 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-07-31T02:03:01.162 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2021-07-31T02:03:01.233 INFO:tasks.workunit:Running workunit cls/test_cls_numops.sh... 2021-07-31T02:03:01.234 DEBUG:teuthology.orchestra.run.smithi045:workunit test cls/test_cls_numops.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=pacific TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_numops.sh 2021-07-31T02:03:01.306 INFO:tasks.workunit.client.0.smithi045.stdout:Running main() from gmock_main.cc 2021-07-31T02:03:01.306 INFO:tasks.workunit.client.0.smithi045.stdout:[==========] Running 4 tests from 1 test suite. 2021-07-31T02:03:01.306 INFO:tasks.workunit.client.0.smithi045.stdout:[----------] Global test environment set-up. 2021-07-31T02:03:01.306 INFO:tasks.workunit.client.0.smithi045.stdout:[----------] 4 tests from ClsNumOps 2021-07-31T02:03:01.307 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] ClsNumOps.Add 2021-07-31T02:03:02.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:02 smithi154 conmon[34553]: cluster 2021-07-31T02:03:01.157288+0000 mon.a (mon.0) 1764 : cluster [DBG] osdmap e974: 8 total, 8 up, 8 in 2021-07-31T02:03:02.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:02 smithi154 conmon[34553]: cluster 2021-07-31T02:03:02.017535+0000 mgr.x (mgr.24542) 677 : cluster [DBG] pgmap v1295: 9 pgs: 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:03:02.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:02 smithi045 conmon[39718]: cluster 2021-07-31T02:03:01. 2021-07-31T02:03:02.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:02 smithi045 conmon[39718]: 157288+0000 mon.a (mon.0) 1764 : cluster [DBG] osdmap e974: 8 total, 8 up, 8 in 2021-07-31T02:03:02.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:02 smithi045 conmon[39718]: cluster 2021-07-31T02:03:02.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:02 smithi045 conmon[39718]: 2021-07- 2021-07-31T02:03:02.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:02 smithi045 conmon[39718]: 31T02:03:02.017535 2021-07-31T02:03:02.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:02 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 677 2021-07-31T02:03:02.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:02 smithi045 conmon[39718]: : cluster [DBG] pgmap v1295: 9 pgs: 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:03:02.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:02 smithi045 conmon[33655]: cluster 2021-07-31T02:03:01 2021-07-31T02:03:02.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:02 smithi045 conmon[33655]: .157288+0000 mon.a (mon.0) 1764 : cluster [DBG] osdmap e974: 8 total, 8 up, 8 in 2021-07-31T02:03:02.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:02 smithi045 conmon[33655]: cluster 2021-07-31T02:03 2021-07-31T02:03:02.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:02 smithi045 conmon[33655]: :02.017535+0000 mgr.x (mgr.24542) 677 : cluster 2021-07-31T02:03:02.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:02 smithi045 conmon[33655]: [DBG] pgmap v1295: 9 pgs: 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:03:03.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:03 smithi154 conmon[34553]: cluster 2021-07-31T02:03:03.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:03 smithi154 conmon[34553]: 2021-07-31T02:03:03.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:03 smithi154 conmon[34553]: 2021 2021-07-31T02:03:03.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:03 smithi154 conmon[34553]: - 2021-07-31T02:03:03.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:03 smithi154 conmon[34553]: 07 2021-07-31T02:03:03.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:03 smithi154 conmon[34553]: - 2021-07-31T02:03:03.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:03 smithi154 conmon[34553]: 31 2021-07-31T02:03:03.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:03 smithi154 conmon[34553]: T 2021-07-31T02:03:03.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:03 smithi154 conmon[34553]: 02 2021-07-31T02:03:03.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:03 smithi154 conmon[34553]: : 2021-07-31T02:03:03.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:03 smithi154 conmon[34553]: 03:02.162496+0000 mon.a (mon.0) 1765 : cluster [DBG] osdmap e975: 8 total, 8 up, 8 in 2021-07-31T02:03:03.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:03 smithi154 conmon[34553]: 2021-07-31T02:03:03.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:03 smithi154 conmon[34553]: audit 2021-07-31T02:03:02 2021-07-31T02:03:03.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:03 smithi154 conmon[34553]: .163186+0000 mon.a (mon.0) 1766 : audit 2021-07-31T02:03:03.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:03 smithi154 conmon[34553]: [INF] from='client.? 172.21.15.45:0/1624933575' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74384-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:03:03.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:03 smithi045 conmon[33655]: cluster 2021-07-31T02:03:03.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:03 smithi045 conmon[33655]: 2021-07-31T02:03:03.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:03 smithi045 conmon[33655]: 2021 2021-07-31T02:03:03.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:03 smithi045 conmon[33655]: - 2021-07-31T02:03:03.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:03 smithi045 conmon[33655]: 07 2021-07-31T02:03:03.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:03 smithi045 conmon[33655]: - 2021-07-31T02:03:03.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:03 smithi045 conmon[33655]: 31 2021-07-31T02:03:03.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:03 smithi045 conmon[33655]: T 2021-07-31T02:03:03.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:03 smithi045 conmon[33655]: 02:03:02.162496+0000 mon.a (mon.0 2021-07-31T02:03:03.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:03 smithi045 conmon[33655]: ) 1765 : cluster [DBG] osdmap e975: 8 total, 8 up, 8 in 2021-07-31T02:03:03.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:03 smithi045 conmon[33655]: audit 2021-07-31T02:03:03.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:03 smithi045 conmon[33655]: 2021-07-31T02:03:02.163186+0000 mon.a (mon.0) 1766 : audit [INF] from='client.? 172.21.15.45:0/1624933575' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74384-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:03:03.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:03 smithi045 conmon[39718]: cluster 2021-07-31T02:03:03.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:03 smithi045 conmon[39718]: 2021-07-31T02:03:03.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:03 smithi045 conmon[39718]: 2021 2021-07-31T02:03:03.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:03 smithi045 conmon[39718]: - 2021-07-31T02:03:03.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:03 smithi045 conmon[39718]: 07 2021-07-31T02:03:03.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:03 smithi045 conmon[39718]: - 2021-07-31T02:03:03.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:03 smithi045 conmon[39718]: 31 2021-07-31T02:03:03.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:03 smithi045 conmon[39718]: T 2021-07-31T02:03:03.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:03 smithi045 conmon[39718]: 02 2021-07-31T02:03:03.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:03 smithi045 conmon[39718]: : 2021-07-31T02:03:03.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:03 smithi045 conmon[39718]: 03 2021-07-31T02:03:03.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:03 smithi045 conmon[39718]: : 2021-07-31T02:03:03.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:03 smithi045 conmon[39718]: 02 2021-07-31T02:03:03.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:03 smithi045 conmon[39718]: . 2021-07-31T02:03:03.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:03 smithi045 conmon[39718]: 162496 2021-07-31T02:03:03.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:03 smithi045 conmon[39718]: +0000 2021-07-31T02:03:03.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:03 smithi045 conmon[39718]: 2021-07-31T02:03:03.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:03 smithi045 conmon[39718]: mon.a 2021-07-31T02:03:03.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:03 smithi045 conmon[39718]: ( 2021-07-31T02:03:03.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:03 smithi045 conmon[39718]: mon 2021-07-31T02:03:03.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:03 smithi045 conmon[39718]: . 2021-07-31T02:03:03.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:03 smithi045 conmon[39718]: 0 2021-07-31T02:03:03.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:03 smithi045 conmon[39718]: ) 2021-07-31T02:03:03.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:03 smithi045 conmon[39718]: 1765 2021-07-31T02:03:03.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:03 smithi045 conmon[39718]: : 2021-07-31T02:03:03.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:03 smithi045 conmon[39718]: cluster 2021-07-31T02:03:03.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:03 smithi045 conmon[39718]: 2021-07-31T02:03:03.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:03 smithi045 conmon[39718]: [DBG] 2021-07-31T02:03:03.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:03 smithi045 conmon[39718]: 2021-07-31T02:03:03.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:03 smithi045 conmon[39718]: osdmap e975: 8 total, 8 up, 8 in 2021-07-31T02:03:03.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:03 smithi045 conmon[39718]: 2021-07-31T02:03:03.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:03 smithi045 conmon[39718]: audit 2021-07-31T02:03:03.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:03 smithi045 conmon[39718]: 2021-07-31T02:03:03.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:03 smithi045 conmon[39718]: 2021 2021-07-31T02:03:03.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:03 smithi045 conmon[39718]: - 2021-07-31T02:03:03.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:03 smithi045 conmon[39718]: 07 2021-07-31T02:03:03.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:03 smithi045 conmon[39718]: - 2021-07-31T02:03:03.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:03 smithi045 conmon[39718]: 31 2021-07-31T02:03:03.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:03 smithi045 conmon[39718]: T 2021-07-31T02:03:03.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:03 smithi045 conmon[39718]: 02 2021-07-31T02:03:03.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:03 smithi045 conmon[39718]: : 2021-07-31T02:03:03.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:03 smithi045 conmon[39718]: 03 2021-07-31T02:03:03.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:03 smithi045 conmon[39718]: : 2021-07-31T02:03:03.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:03 smithi045 conmon[39718]: 02 2021-07-31T02:03:03.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:03 smithi045 conmon[39718]: . 2021-07-31T02:03:03.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:03 smithi045 conmon[39718]: 163186 2021-07-31T02:03:03.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:03 smithi045 conmon[39718]: +0000 2021-07-31T02:03:03.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:03 smithi045 conmon[39718]: 2021-07-31T02:03:03.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:03 smithi045 conmon[39718]: mon.a 2021-07-31T02:03:03.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:03 smithi045 conmon[39718]: ( 2021-07-31T02:03:03.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:03 smithi045 conmon[39718]: mon 2021-07-31T02:03:03.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:03 smithi045 conmon[39718]: . 2021-07-31T02:03:03.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:03 smithi045 conmon[39718]: 0 2021-07-31T02:03:03.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:03 smithi045 conmon[39718]: ) 2021-07-31T02:03:03.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:03 smithi045 conmon[39718]: 1766 2021-07-31T02:03:03.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:03 smithi045 conmon[39718]: : 2021-07-31T02:03:03.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:03 smithi045 conmon[39718]: audit 2021-07-31T02:03:03.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:03 smithi045 conmon[39718]: 2021-07-31T02:03:03.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:03 smithi045 conmon[39718]: [INF] 2021-07-31T02:03:03.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:03 smithi045 conmon[39718]: 2021-07-31T02:03:03.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:03 smithi045 conmon[39718]: from='client.? 172.21.15.45:0/1624933575' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74384-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:03:03.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:03 smithi045 conmon[39718]: 2021-07-31T02:03:04.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:03:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:03:03.535Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:03:04.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:03:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:03:03.535Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:03:04.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:03:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:03:03.535Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:03:04.183 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] ClsNumOps.Add (2877 ms) 2021-07-31T02:03:04.183 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] ClsNumOps.Sub 2021-07-31T02:03:04.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:04 smithi154 conmon[34553]: audit 2021-07-31T02:03:03.165474+0000 mon.a (mon.0) 1767 : audit [INF] from='client.? 172.21.15.45:0/1624933575' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74384-1","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:03:04.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:04 smithi154 conmon[34553]: cluster 2021-07-31T02:03:04.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:04 smithi154 conmon[34553]: 2021-07-31T02:03:03.165753+0000 mon.a (mon.0) 1768 : cluster [DBG] osdmap e976: 8 total, 8 up, 8 in 2021-07-31T02:03:04.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:04 smithi154 conmon[34553]: cluster 2021-07-31T02:03:04.018117+0000 mgr.x (mgr.24542) 678 : cluster [DBG] pgmap v1298: 41 pgs: 41 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2021-07-31T02:03:04.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:04 smithi045 conmon[39718]: audit 2021-07-31T02:03:03.165474 2021-07-31T02:03:04.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:04 smithi045 conmon[39718]: +0000 mon.a (mon.0) 1767 : audit [INF] from='client.? 172.21.15.45:0/1624933575' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74384-1","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:03:04.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:04 smithi045 conmon[39718]: cluster 2021 2021-07-31T02:03:04.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:04 smithi045 conmon[39718]: -07-31T02:03:03.165753+0000 mon.a 2021-07-31T02:03:04.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:04 smithi045 conmon[39718]: (mon.0) 1768 : cluster [DBG] osdmap e976: 8 total, 8 up, 8 in 2021-07-31T02:03:04.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:04 smithi045 conmon[39718]: cluster 2021- 2021-07-31T02:03:04.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:04 smithi045 conmon[39718]: 07-31T02:03:04.018117+0000 mgr.x (mgr. 2021-07-31T02:03:04.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:04 smithi045 conmon[39718]: 24542) 678 : cluster [DBG] pgmap v1298: 41 pgs: 41 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2021-07-31T02:03:04.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:04 smithi045 conmon[33655]: audit 2021-07-31T02:03:03.165474+0000 2021-07-31T02:03:04.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:04 smithi045 conmon[33655]: mon.a (mon.0) 1767 : audit [INF] from='client.? 172.21.15.45:0/1624933575' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74384-1","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:03:04.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:04 smithi045 conmon[33655]: cluster 2021- 2021-07-31T02:03:04.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:04 smithi045 conmon[33655]: 07-31T02:03:03.165753+0000 mon.a (mon.0) 1768 : 2021-07-31T02:03:04.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:04 smithi045 conmon[33655]: cluster [DBG] osdmap e976: 8 total, 8 up, 8 in 2021-07-31T02:03:04.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:04 smithi045 conmon[33655]: cluster 2021-07-31T02:03:04.018117 2021-07-31T02:03:04.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:04 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 678 : cluster [DBG] pgmap v1298: 41 pgs: 41 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2021-07-31T02:03:05.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:05 smithi154 conmon[34553]: cluster 2021-07-31T02:03:04.180430+0000 mon.a (mon.0) 1769 : cluster [DBG] osdmap e977: 8 total, 8 up, 8 in 2021-07-31T02:03:05.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:05 smithi154 conmon[34553]: 2021-07-31T02:03:05.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:05 smithi045 conmon[39718]: cluster 2021-07-31T02:03:04.180430+0000 2021-07-31T02:03:05.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:05 smithi045 conmon[39718]: mon.a (mon.0) 1769 : cluster [DBG] osdmap e977: 8 total, 8 up, 8 in 2021-07-31T02:03:05.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:05 smithi045 conmon[33655]: cluster 2021-07-31T02:03: 2021-07-31T02:03:05.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:05 smithi045 conmon[33655]: 04.180430+0000 mon.a (mon.0) 1769 : cluster [DBG] osdmap e977: 8 total, 8 up, 8 in 2021-07-31T02:03:06.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:06 smithi154 conmon[34553]: cluster 2021-07-31T02:03:05.185080+0000 mon.a (mon.0) 1770 : 2021-07-31T02:03:06.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:06 smithi154 conmon[34553]: cluster [DBG] osdmap e978: 8 total, 8 up, 8 in 2021-07-31T02:03:06.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:06 smithi154 conmon[34553]: audit 2021-07-31T02:03:05.186512+0000 mon.c (mon.1) 63 : audit [INF] from='client.? 172.21.15.45:0/22857664' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74384-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:03:06.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:06 smithi154 conmon[34553]: audit 2021-07-31T02:03:05.186981+0000 mon.a (mon.0) 1771 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74384-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:03:06.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:06 smithi154 conmon[34553]: cluster 2021-07-31T02:03:06.018521+0000 mgr.x (mgr.24542) 679 : cluster [DBG] pgmap v1301: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:03:06.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:06 smithi045 conmon[39718]: cluster 2021-07-31T02:03 2021-07-31T02:03:06.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:06 smithi045 conmon[39718]: :05.185080+0000 mon.a (mon.0) 1770 : cluster [DBG] osdmap e978: 8 total, 8 up, 8 in 2021-07-31T02:03:06.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:06 smithi045 conmon[39718]: audit 2021-07-31T02:03:05.186512+0000 mon.c ( 2021-07-31T02:03:06.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:06 smithi045 conmon[39718]: mon.1) 63 : audit [INF] from='client.? 172.21.15.45:0/22857664' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74384-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:03:06.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:06 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T02:03:06.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:06 smithi045 conmon[39718]: 31T02:03:05.186981+0000 mon.a ( 2021-07-31T02:03:06.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:06 smithi045 conmon[39718]: mon.0) 1771 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74384-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:03:06.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:06 smithi045 conmon[39718]: cluster 2021-07-31 2021-07-31T02:03:06.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:06 smithi045 conmon[39718]: T02:03:06.018521+0000 mgr.x (mgr.24542) 679 : 2021-07-31T02:03:06.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:06 smithi045 conmon[39718]: cluster [DBG] pgmap v1301: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:03:06.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:06 smithi045 conmon[33655]: cluster 2021-07-31T02:03:05.185080+0000 mon.a (mon.0) 1770 : 2021-07-31T02:03:06.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:06 smithi045 conmon[33655]: cluster [DBG] osdmap e978: 8 total, 8 up, 8 in 2021-07-31T02:03:06.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:06 smithi045 conmon[33655]: audit 2021-07-31T02:03:05.186512+0000 mon.c (mon.1) 63 : audit [INF] from='client.? 172.21.15.45:0/22857664' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74384-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:03:06.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:06 smithi045 conmon[33655]: audit 2021-07-31T02:03:06.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:06 smithi045 conmon[33655]: 2021-07-31T02:03:05.186981+0000 mon.a (mon.0) 1771 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74384-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:03:06.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:06 smithi045 conmon[33655]: 2021-07-31T02:03:06.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:06 smithi045 conmon[33655]: cluster 2021-07-31T02:03:06.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:06 smithi045 conmon[33655]: 2021-07-31T02:03:06.018521+0000 mgr.x ( 2021-07-31T02:03:06.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:06 smithi045 conmon[33655]: mgr.24542) 679 : cluster [DBG] pgmap v1301: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:03:07.199 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] ClsNumOps.Sub (3016 ms) 2021-07-31T02:03:07.200 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] ClsNumOps.Mul 2021-07-31T02:03:07.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:07 smithi154 conmon[34553]: audit 2021-07-31T02:03:06.192345+0000 mon.a (mon.0) 1772 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74384-2","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:03:07.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:07 smithi154 conmon[34553]: 2021-07-31T02:03:07.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:07 smithi154 conmon[34553]: cluster 2021-07-31T02:03:06.192951+0000 mon.a (mon.0) 1773 : cluster [DBG] osdmap e979: 8 total, 8 up, 8 in 2021-07-31T02:03:07.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:07 smithi045 conmon[33655]: audit 2021-07-31T02:03:06.192345+0000 mon.a (mon.0) 1772 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74384-2","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:03:07.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:07 smithi045 conmon[33655]: cluster 2021-07-31T02:03:06.192951+0000 mon.a (mon.0) 1773 : cluster [DBG] osdmap e979: 8 total, 8 up, 8 in 2021-07-31T02:03:07.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:07 smithi045 conmon[39718]: audit 2021-07-31T02:03:06.192345+0000 mon.a (mon.0) 1772 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74384-2","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:03:07.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:07 smithi045 conmon[39718]: cluster 2021-07-31T02:03:06.192951+0000 mon.a (mon.0) 1773 : cluster [DBG] osdmap e979: 8 total, 8 up, 8 in 2021-07-31T02:03:07.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:07 smithi045 conmon[39718]: 2021-07-31T02:03:08.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:08 smithi154 conmon[34553]: cluster 2021-07-31T02:03:07.197211+0000 mon.a (mon.0) 1774 : cluster 2021-07-31T02:03:08.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:08 smithi154 conmon[34553]: [DBG] osdmap e980: 8 total, 8 up, 8 in 2021-07-31T02:03:08.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:08 smithi154 conmon[34553]: cluster 2021-07-31T02:03:08.018901+0000 mgr.x (mgr.24542) 680 : cluster [DBG] pgmap v1304: 9 pgs: 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:03:08.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:08 smithi045 conmon[33655]: cluster 2021-07-31T02:03:07.197211+0000 mon.a ( 2021-07-31T02:03:08.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:08 smithi045 conmon[33655]: mon.0) 1774 : cluster [DBG] osdmap e980: 8 total, 8 up, 8 in 2021-07-31T02:03:08.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:08 smithi045 conmon[33655]: cluster 2021-07-31T02:03:08.018901+0000 mgr.x (mgr.24542) 680 : cluster [DBG] pgmap v1304: 9 pgs: 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:03:08.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:08 smithi045 conmon[39718]: cluster 2021-07-31T02:03:07.197211+0000 2021-07-31T02:03:08.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:08 smithi045 conmon[39718]: mon.a (mon.0) 1774 : cluster [DBG] osdmap e980: 8 total, 8 up, 8 in 2021-07-31T02:03:08.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:08 smithi045 conmon[39718]: cluster 2021-07- 2021-07-31T02:03:08.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:08 smithi045 conmon[39718]: 31T02:03:08.018901+0000 mgr.x (mgr.24542) 680 : 2021-07-31T02:03:08.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:08 smithi045 conmon[39718]: cluster [DBG] pgmap v1304: 9 pgs: 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:03:09.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:09 smithi154 conmon[34553]: cluster 2021-07-31T02:03:09.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:09 smithi154 conmon[34553]: 2021-07-31T02:03:09.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:09 smithi154 conmon[34553]: 2021 2021-07-31T02:03:09.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:09 smithi154 conmon[34553]: - 2021-07-31T02:03:09.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:09 smithi154 conmon[34553]: 07 2021-07-31T02:03:09.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:09 smithi154 conmon[34553]: - 2021-07-31T02:03:09.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:09 smithi154 conmon[34553]: 31 2021-07-31T02:03:09.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:09 smithi154 conmon[34553]: T 2021-07-31T02:03:09.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:09 smithi154 conmon[34553]: 02 2021-07-31T02:03:09.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:09 smithi154 conmon[34553]: :03:08.203771 2021-07-31T02:03:09.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:09 smithi154 conmon[34553]: +0000 mon.a (mon.0) 1775 : cluster [DBG] osdmap e981: 8 total, 8 up, 8 in 2021-07-31T02:03:09.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:09 smithi154 conmon[34553]: audit 2021-07-31T02:03:09.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:09 smithi154 conmon[34553]: 2021-07 2021-07-31T02:03:09.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:09 smithi154 conmon[34553]: -31T02:03:08.205937+0000 mon.c (mon.1) 64 : 2021-07-31T02:03:09.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:09 smithi154 conmon[34553]: audit [INF] from='client.? 172.21.15.45:0/1426640961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74384-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:03:09.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:09 smithi154 conmon[34553]: audit 2021 2021-07-31T02:03:09.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:09 smithi154 conmon[34553]: -07-31T02:03:08.208370+0000 mon.a 2021-07-31T02:03:09.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:09 smithi154 conmon[34553]: (mon.0) 1776 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74384-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:03:09.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:09 smithi045 conmon[33655]: cluster 2021-07-31T02:03:08.203771+0000 mon.a (mon.0 2021-07-31T02:03:09.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:09 smithi045 conmon[33655]: ) 1775 : cluster [DBG] osdmap e981: 8 total, 8 up, 8 in 2021-07-31T02:03:09.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:09 smithi045 conmon[33655]: audit 2021-07-31T02:03:08.205937+0000 mon.c (mon.1) 64 : audit [INF] from='client.? 172.21.15.45:0/1426640961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74384-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:03:09.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:09 smithi045 conmon[33655]: audit 2021-07-31T02:03:08.208370+0000 mon.a (mon.0) 1776 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74384-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:03:09.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:09 smithi045 conmon[39718]: cluster 2021-07-31T02:03:08.203771+0000 mon.a ( 2021-07-31T02:03:09.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:09 smithi045 conmon[39718]: mon.0) 1775 : cluster [DBG] osdmap e981: 8 total, 8 up, 8 in 2021-07-31T02:03:09.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:09 smithi045 conmon[39718]: audit 2021-07 2021-07-31T02:03:09.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:09 smithi045 conmon[39718]: -31T02:03:08.205937+0000 mon.c (mon.1) 64 : 2021-07-31T02:03:09.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:09 smithi045 conmon[39718]: audit [INF] from='client.? 172.21.15.45:0/1426640961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74384-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:03:09.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:09 smithi045 conmon[39718]: audit 2021- 2021-07-31T02:03:09.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:09 smithi045 conmon[39718]: 07-31T02:03:08.208370+0000 mon.a (mon 2021-07-31T02:03:09.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:09 smithi045 conmon[39718]: .0) 1776 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74384-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:03:10.210 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] ClsNumOps.Mul (3011 ms) 2021-07-31T02:03:10.211 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] ClsNumOps.Div 2021-07-31T02:03:10.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:10 smithi154 conmon[34553]: audit 2021-07-31T02:03:09.205598+0000 mon.a (mon 2021-07-31T02:03:10.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:10 smithi154 conmon[34553]: .0) 1777 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74384-3","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:03:10.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:10 smithi154 conmon[34553]: cluster 2021-07-31T02:03:09.205891+0000 mon.a (mon.0) 1778 : cluster [DBG] osdmap e982: 8 total, 8 up, 8 in 2021-07-31T02:03:10.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:10 smithi154 conmon[34553]: cluster 2021-07-31 2021-07-31T02:03:10.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:10 smithi154 conmon[34553]: T02:03:10.019491+0000 2021-07-31T02:03:10.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:10 smithi154 conmon[34553]: mgr.x (mgr.24542) 681 : cluster 2021-07-31T02:03:10.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:10 smithi154 conmon[34553]: [DBG] pgmap v1307: 41 pgs: 32 creating+peering, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:03:10.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:10 smithi154 conmon[34553]: cluster 2021 2021-07-31T02:03:10.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:10 smithi154 conmon[34553]: -07-31T02:03:10.207337 2021-07-31T02:03:10.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:10 smithi154 conmon[34553]: +0000 mon.a (mon.0) 1779 : 2021-07-31T02:03:10.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:10 smithi154 conmon[34553]: cluster [DBG] osdmap e983: 8 total, 8 up, 8 in 2021-07-31T02:03:10.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:10 smithi045 conmon[39718]: audit 2021-07-31T02:03:09.205598+0000 mon.a (mon.0) 1777 2021-07-31T02:03:10.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:10 smithi045 conmon[39718]: : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74384-3","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:03:10.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:10 smithi045 conmon[39718]: cluster 2021-07-31T02:03:09.205891+0000 mon.a (mon.0) 1778 : cluster 2021-07-31T02:03:10.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:10 smithi045 conmon[39718]: [DBG] osdmap e982: 8 total, 8 up, 8 in 2021-07-31T02:03:10.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:10 smithi045 conmon[39718]: cluster 2021-07-31T02 2021-07-31T02:03:10.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:10 smithi045 conmon[39718]: :03:10.019491+0000 mgr.x (mgr.24542) 681 : cluster [DBG] pgmap v1307: 41 pgs: 32 creating+peering, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:03:10.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:10 smithi045 conmon[39718]: cluster 2021-07- 2021-07-31T02:03:10.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:10 smithi045 conmon[39718]: 31T02:03:10.207337+0000 mon.a (mon.0 2021-07-31T02:03:10.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:10 smithi045 conmon[39718]: ) 1779 : cluster [DBG] osdmap e983: 8 total, 8 up, 8 in 2021-07-31T02:03:10.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:10 smithi045 conmon[33655]: audit 2021-07-31T 2021-07-31T02:03:10.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:10 smithi045 conmon[33655]: 02:03:09.205598+0000 mon.a (mon.0) 1777 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74384-3","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:03:10.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:10 smithi045 conmon[33655]: cluster 2021 2021-07-31T02:03:10.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:10 smithi045 conmon[33655]: -07-31T02:03:09.205891+0000 2021-07-31T02:03:10.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:10 smithi045 conmon[33655]: mon.a (mon.0) 1778 : cluster [DBG] 2021-07-31T02:03:10.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:10 smithi045 conmon[33655]: osdmap e982: 8 total, 8 up, 8 in 2021-07-31T02:03:10.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:10 smithi045 conmon[33655]: cluster 2021-07-31T02 2021-07-31T02:03:10.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:10 smithi045 conmon[33655]: :03:10.019491+0000 mgr.x (mgr.24542) 681 : cluster 2021-07-31T02:03:10.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:10 smithi045 conmon[33655]: [DBG] pgmap v1307: 41 pgs: 32 creating+peering, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:03:10.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:10 smithi045 conmon[33655]: cluster 2021-07-31T02:03:10.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:10 smithi045 conmon[33655]: 2021-07-31T02:03:10.207337+0000 mon.a (mon.0) 1779 : cluster [DBG] 2021-07-31T02:03:10.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:10 smithi045 conmon[33655]: osdmap e983: 8 total, 8 up, 8 in 2021-07-31T02:03:12.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:12 smithi154 conmon[34553]: cluster 2021-07-31T02:03:11.222980+0000 mon.a (mon.0) 1780 : cluster [DBG] osdmap e984: 8 total, 8 up, 8 in 2021-07-31T02:03:12.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:12 smithi154 conmon[34553]: 2021-07-31T02:03:12.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:12 smithi154 conmon[34553]: audit 2021-07-31T02:03:11.224946+0000 mon.c (mon.1) 65 : audit [INF] from='client.? 172.21.15.45:0/508460009' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74384-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:03:12.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:12 smithi154 conmon[34553]: audit 2021-07-31T02:03:11 2021-07-31T02:03:12.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:12 smithi154 conmon[34553]: .226491+0000 mon.a (mon.0) 1781 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74384-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:03:12.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:12 smithi154 conmon[34553]: cluster 2021-07-31T02: 2021-07-31T02:03:12.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:12 smithi154 conmon[34553]: 03:12.020024+0000 mgr.x (mgr.24542) 2021-07-31T02:03:12.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:12 smithi154 conmon[34553]: 682 : cluster [DBG] pgmap v1310: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:03:12.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:12 smithi045 conmon[33655]: cluster 2021-07-31T02:03:11 2021-07-31T02:03:12.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:12 smithi045 conmon[33655]: .222980+0000 mon.a (mon.0) 1780 : cluster [DBG] osdmap e984: 8 total, 8 up, 8 in 2021-07-31T02:03:12.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:12 smithi045 conmon[33655]: audit 2021-07-31T02:03:11.224946+0000 mon.c 2021-07-31T02:03:12.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:12 smithi045 conmon[33655]: (mon.1) 65 : audit [INF] from='client.? 172.21.15.45:0/508460009' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74384-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:03:12.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:12 smithi045 conmon[33655]: audit 2021-07-31T02:03:12.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:12 smithi045 conmon[33655]: 2021-07-31T02:03:11.226491+0000 mon.a (mon.0 2021-07-31T02:03:12.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:12 smithi045 conmon[33655]: ) 1781 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74384-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:03:12.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:12 smithi045 conmon[33655]: cluster 2021-07-31T 2021-07-31T02:03:12.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:12 smithi045 conmon[33655]: 02:03:12.020024+0000 mgr.x (mgr.24542) 682 : cluster [DBG] pgmap v1310: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:03:12.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:12 smithi045 conmon[39718]: cluster 2021-07-31T02:03:11.222980+0000 mon.a (mon.0 2021-07-31T02:03:12.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:12 smithi045 conmon[39718]: ) 1780 : cluster [DBG] osdmap e984: 8 total, 8 up, 8 in 2021-07-31T02:03:12.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:12 smithi045 conmon[39718]: audit 2021-07-31T02:03:11.224946+0000 mon.c (mon.1) 65 : audit [INF] from='client.? 172.21.15.45:0/508460009' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74384-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:03:12.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:12 smithi045 conmon[39718]: audit 2021-07-31T02:03:11.226491+0000 mon.a (mon.0) 1781 : audit 2021-07-31T02:03:12.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:12 smithi045 conmon[39718]: [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74384-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:03:12.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:12 smithi045 conmon[39718]: cluster 2021-07-31T02:03:12.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:12 smithi045 conmon[39718]: 2021-07-31T02:03:12.020024+0000 mgr.x (mgr.24542) 682 : cluster [DBG] pgmap v1310: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:03:12.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:12 smithi045 conmon[39718]: 2021-07-31T02:03:13.233 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] ClsNumOps.Div (3023 ms) 2021-07-31T02:03:13.234 INFO:tasks.workunit.client.0.smithi045.stdout:[----------] 4 tests from ClsNumOps (11927 ms total) 2021-07-31T02:03:13.234 INFO:tasks.workunit.client.0.smithi045.stdout: 2021-07-31T02:03:13.234 INFO:tasks.workunit.client.0.smithi045.stdout:[----------] Global test environment tear-down 2021-07-31T02:03:13.235 INFO:tasks.workunit.client.0.smithi045.stdout:[==========] 4 tests from 1 test suite ran. (11927 ms total) 2021-07-31T02:03:13.235 INFO:tasks.workunit.client.0.smithi045.stdout:[ PASSED ] 4 tests. 2021-07-31T02:03:13.236 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-07-31T02:03:13.237 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2021-07-31T02:03:13.265 INFO:tasks.workunit:Running workunit cls/test_cls_rbd.sh... 2021-07-31T02:03:13.266 DEBUG:teuthology.orchestra.run.smithi045:workunit test cls/test_cls_rbd.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=pacific TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh 2021-07-31T02:03:13.336 INFO:tasks.workunit.client.0.smithi045.stdout:Running main() from gmock_main.cc 2021-07-31T02:03:13.337 INFO:tasks.workunit.client.0.smithi045.stdout:[==========] Running 69 tests from 1 test suite. 2021-07-31T02:03:13.337 INFO:tasks.workunit.client.0.smithi045.stdout:[----------] Global test environment set-up. 2021-07-31T02:03:13.337 INFO:tasks.workunit.client.0.smithi045.stdout:[----------] 69 tests from TestClsRbd 2021-07-31T02:03:13.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:13 smithi154 conmon[34553]: audit 2021-07-31T02:03:12.228767+0000 mon.a (mon.0) 1782 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74384-4","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:03:13.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:13 smithi154 conmon[34553]: 2021-07-31T02:03:13.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:13 smithi154 conmon[34553]: cluster 2021-07-31T02:03:12.229046+0000 mon.a (mon.0) 1783 : cluster [DBG] osdmap e985: 8 total, 8 up, 8 in 2021-07-31T02:03:13.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:13 smithi045 conmon[33655]: audit 2021-07-31T02:03:12.228767+0000 mon.a (mon.0) 1782 : audit 2021-07-31T02:03:13.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:13 smithi045 conmon[33655]: [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74384-4","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:03:13.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:13 smithi045 conmon[33655]: cluster 2021-07-31T02:03:12.229046+0000 mon.a (mon.0) 1783 : cluster [DBG] osdmap e985: 8 total, 8 up, 8 in 2021-07-31T02:03:13.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:13 smithi045 conmon[39718]: audit 2021-07-31T02:03:13.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:13 smithi045 conmon[39718]: 2021- 2021-07-31T02:03:13.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:13 smithi045 conmon[39718]: 07-31T02:03:12.228767+0000 mon.a (mon.0) 1782 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74384-4","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:03:13.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:13 smithi045 conmon[39718]: cluster 2021 2021-07-31T02:03:13.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:13 smithi045 conmon[39718]: -07-31T02:03:12.229046+0000 mon.a (mon. 2021-07-31T02:03:13.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:13 smithi045 conmon[39718]: 0) 1783 : cluster [DBG] osdmap e985: 8 total, 8 up, 8 in 2021-07-31T02:03:14.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:03:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:03:13.535Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:03:14.027 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:03:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:03:13.535Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:03:14.027 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:03:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:03:13.535Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:03:14.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:14 smithi154 conmon[34553]: cluster 2021-07-31T02:03:13.229845+0000 mon.a (mon.0) 1784 : cluster [DBG] osdmap e986: 8 total, 8 up, 8 in 2021-07-31T02:03:14.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:14 smithi154 conmon[34553]: cluster 2021-07-31T02:03:14.020456+0000 mgr.x (mgr.24542) 683 : cluster [DBG] pgmap v1313: 9 pgs: 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:03:14.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:14 smithi045 conmon[33655]: cluster 2021-07-31T02:03:13.229845+0000 mon.a (mon.0) 1784 2021-07-31T02:03:14.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:14 smithi045 conmon[33655]: : cluster [DBG] osdmap e986: 8 total, 8 up, 8 in 2021-07-31T02:03:14.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:14 smithi045 conmon[33655]: cluster 2021-07-31T02:03:14.020456+0000 mgr.x (mgr.24542) 683 : cluster [DBG] pgmap v1313: 9 pgs: 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:03:14.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:14 smithi045 conmon[39718]: cluster 2021-07-31T02:03:13. 2021-07-31T02:03:14.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:14 smithi045 conmon[39718]: 229845+0000 mon.a (mon.0) 1784 : cluster [DBG] osdmap e986: 8 total, 8 up, 8 in 2021-07-31T02:03:14.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:14 smithi045 conmon[39718]: cluster 2021-07 2021-07-31T02:03:14.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:14 smithi045 conmon[39718]: -31T02:03:14.020456+0000 mgr.x (mgr.24542) 683 : cluster 2021-07-31T02:03:14.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:14 smithi045 conmon[39718]: [DBG] pgmap v1313: 9 pgs: 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:03:15.260 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsRbd.get_all_features 2021-07-31T02:03:15.265 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsRbd.get_all_features (5 ms) 2021-07-31T02:03:15.265 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsRbd.copyup 2021-07-31T02:03:15.448 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsRbd.copyup (183 ms) 2021-07-31T02:03:15.449 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsRbd.sparse_copyup 2021-07-31T02:03:15.464 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsRbd.sparse_copyup (15 ms) 2021-07-31T02:03:15.464 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsRbd.get_and_set_id 2021-07-31T02:03:15.472 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsRbd.get_and_set_id (8 ms) 2021-07-31T02:03:15.472 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsRbd.add_remove_child 2021-07-31T02:03:15.484 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsRbd.add_remove_child (13 ms) 2021-07-31T02:03:15.485 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsRbd.directory_methods 2021-07-31T02:03:15.512 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsRbd.directory_methods (27 ms) 2021-07-31T02:03:15.512 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsRbd.create 2021-07-31T02:03:15.525 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsRbd.create (13 ms) 2021-07-31T02:03:15.525 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsRbd.get_features 2021-07-31T02:03:15.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:15 smithi045 conmon[33655]: cluster 2021-07-31T02:03:14.241051+0000 mon.a (mon.0) 1785 : cluster [DBG] osdmap e987: 8 total, 8 up, 8 in 2021-07-31T02:03:15.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:15 smithi045 conmon[33655]: audit 2021-07-31T02:03:14.242460+0000 mon.c (mon.1) 66 : audit [INF] from='client.? 172.21.15.45:0/1688263379' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74488-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:03:15.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:15 smithi045 conmon[33655]: audit 2021-07-31T02:03:14.242943+0000 mon.a (mon.0) 1786 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74488-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:03:15.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:15 smithi045 conmon[39718]: cluster 2021-07-31T02:03:15.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:15 smithi045 conmon[39718]: 2021-07-31T02:03:15.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:15 smithi045 conmon[39718]: 2021 2021-07-31T02:03:15.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:15 smithi045 conmon[39718]: - 2021-07-31T02:03:15.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:15 smithi045 conmon[39718]: 07 2021-07-31T02:03:15.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:15 smithi045 conmon[39718]: - 2021-07-31T02:03:15.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:15 smithi045 conmon[39718]: 31 2021-07-31T02:03:15.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:15 smithi045 conmon[39718]: T 2021-07-31T02:03:15.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:15 smithi045 conmon[39718]: 02 2021-07-31T02:03:15.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:15 smithi045 conmon[39718]: : 2021-07-31T02:03:15.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:15 smithi045 conmon[39718]: 03 2021-07-31T02:03:15.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:15 smithi045 conmon[39718]: : 2021-07-31T02:03:15.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:15 smithi045 conmon[39718]: 14 2021-07-31T02:03:15.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:15 smithi045 conmon[39718]: . 2021-07-31T02:03:15.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:15 smithi045 conmon[39718]: 241051 2021-07-31T02:03:15.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:15 smithi045 conmon[39718]: +0000 2021-07-31T02:03:15.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:15 smithi045 conmon[39718]: 2021-07-31T02:03:15.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:15 smithi045 conmon[39718]: mon.a 2021-07-31T02:03:15.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:15 smithi045 conmon[39718]: ( 2021-07-31T02:03:15.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:15 smithi045 conmon[39718]: mon 2021-07-31T02:03:15.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:15 smithi045 conmon[39718]: . 2021-07-31T02:03:15.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:15 smithi045 conmon[39718]: 0 2021-07-31T02:03:15.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:15 smithi045 conmon[39718]: ) 2021-07-31T02:03:15.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:15 smithi045 conmon[39718]: 1785 2021-07-31T02:03:15.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:15 smithi045 conmon[39718]: : 2021-07-31T02:03:15.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:15 smithi045 conmon[39718]: cluster 2021-07-31T02:03:15.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:15 smithi045 conmon[39718]: 2021-07-31T02:03:15.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:15 smithi045 conmon[39718]: [DBG] 2021-07-31T02:03:15.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:15 smithi045 conmon[39718]: 2021-07-31T02:03:15.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:15 smithi045 conmon[39718]: osdmap e987: 8 total, 8 up, 8 in 2021-07-31T02:03:15.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:15 smithi045 conmon[39718]: 2021-07-31T02:03:15.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:15 smithi045 conmon[39718]: audit 2021-07-31T02:03:15.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:15 smithi045 conmon[39718]: 2021-07-31T02:03:15.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:15 smithi045 conmon[39718]: 2021 2021-07-31T02:03:15.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:15 smithi045 conmon[39718]: - 2021-07-31T02:03:15.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:15 smithi045 conmon[39718]: 07 2021-07-31T02:03:15.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:15 smithi045 conmon[39718]: - 2021-07-31T02:03:15.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:15 smithi045 conmon[39718]: 31 2021-07-31T02:03:15.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:15 smithi045 conmon[39718]: T 2021-07-31T02:03:15.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:15 smithi045 conmon[39718]: 02 2021-07-31T02:03:15.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:15 smithi045 conmon[39718]: : 2021-07-31T02:03:15.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:15 smithi045 conmon[39718]: 03 2021-07-31T02:03:15.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:15 smithi045 conmon[39718]: : 2021-07-31T02:03:15.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:15 smithi045 conmon[39718]: 14 2021-07-31T02:03:15.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:15 smithi045 conmon[39718]: . 2021-07-31T02:03:15.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:15 smithi045 conmon[39718]: 242460 2021-07-31T02:03:15.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:15 smithi045 conmon[39718]: +0000 2021-07-31T02:03:15.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:15 smithi045 conmon[39718]: 2021-07-31T02:03:15.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:15 smithi045 conmon[39718]: mon.c 2021-07-31T02:03:15.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:15 smithi045 conmon[39718]: ( 2021-07-31T02:03:15.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:15 smithi045 conmon[39718]: mon 2021-07-31T02:03:15.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:15 smithi045 conmon[39718]: . 2021-07-31T02:03:15.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:15 smithi045 conmon[39718]: 1 2021-07-31T02:03:15.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:15 smithi045 conmon[39718]: ) 2021-07-31T02:03:15.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:15 smithi045 conmon[39718]: 66 2021-07-31T02:03:15.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:15 smithi045 conmon[39718]: : 2021-07-31T02:03:15.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:15 smithi045 conmon[39718]: audit 2021-07-31T02:03:15.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:15 smithi045 conmon[39718]: 2021-07-31T02:03:15.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:15 smithi045 conmon[39718]: [INF] 2021-07-31T02:03:15.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:15 smithi045 conmon[39718]: 2021-07-31T02:03:15.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:15 smithi045 conmon[39718]: from='client.? 172.21.15.45:0/1688263379' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74488-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:03:15.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:15 smithi045 conmon[39718]: 2021-07-31T02:03:15.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:15 smithi045 conmon[39718]: audit 2021-07-31T02:03:15.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:15 smithi045 conmon[39718]: 2021-07-31T02:03:15.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:15 smithi045 conmon[39718]: 2021 2021-07-31T02:03:15.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:15 smithi045 conmon[39718]: - 2021-07-31T02:03:15.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:15 smithi045 conmon[39718]: 07 2021-07-31T02:03:15.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:15 smithi045 conmon[39718]: - 2021-07-31T02:03:15.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:15 smithi045 conmon[39718]: 31 2021-07-31T02:03:15.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:15 smithi045 conmon[39718]: T 2021-07-31T02:03:15.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:15 smithi045 conmon[39718]: 02 2021-07-31T02:03:15.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:15 smithi045 conmon[39718]: : 2021-07-31T02:03:15.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:15 smithi045 conmon[39718]: 03 2021-07-31T02:03:15.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:15 smithi045 conmon[39718]: : 2021-07-31T02:03:15.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:15 smithi045 conmon[39718]: 14 2021-07-31T02:03:15.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:15 smithi045 conmon[39718]: . 2021-07-31T02:03:15.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:15 smithi045 conmon[39718]: 242943 2021-07-31T02:03:15.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:15 smithi045 conmon[39718]: +0000 2021-07-31T02:03:15.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:15 smithi045 conmon[39718]: 2021-07-31T02:03:15.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:15 smithi045 conmon[39718]: mon.a (mon.0) 1786 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74488-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:03:15.545 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsRbd.get_features (2 ms) 2021-07-31T02:03:15.546 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsRbd.get_object_prefix 2021-07-31T02:03:15.546 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsRbd.get_object_prefix (2 ms) 2021-07-31T02:03:15.546 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsRbd.get_create_timestamp 2021-07-31T02:03:15.547 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsRbd.get_create_timestamp (1 ms) 2021-07-31T02:03:15.547 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsRbd.get_access_timestamp 2021-07-31T02:03:15.547 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsRbd.get_access_timestamp (2 ms) 2021-07-31T02:03:15.547 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsRbd.get_modify_timestamp 2021-07-31T02:03:15.548 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsRbd.get_modify_timestamp (2 ms) 2021-07-31T02:03:15.548 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsRbd.get_data_pool 2021-07-31T02:03:15.548 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsRbd.get_data_pool (3 ms) 2021-07-31T02:03:15.548 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsRbd.get_size 2021-07-31T02:03:15.548 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsRbd.get_size (4 ms) 2021-07-31T02:03:15.549 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsRbd.set_size 2021-07-31T02:03:15.549 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsRbd.set_size (5 ms) 2021-07-31T02:03:15.549 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsRbd.protection_status 2021-07-31T02:03:15.564 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsRbd.protection_status (19 ms) 2021-07-31T02:03:15.565 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsRbd.snapshot_limits 2021-07-31T02:03:15.572 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsRbd.snapshot_limits (7 ms) 2021-07-31T02:03:15.572 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsRbd.parents_v1 2021-07-31T02:03:15.622 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsRbd.parents_v1 (51 ms) 2021-07-31T02:03:15.623 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsRbd.parents_v2 2021-07-31T02:03:15.653 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsRbd.parents_v2 (30 ms) 2021-07-31T02:03:15.653 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsRbd.snapshots 2021-07-31T02:03:15.676 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsRbd.snapshots (23 ms) 2021-07-31T02:03:15.676 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsRbd.snapid_race 2021-07-31T02:03:15.681 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsRbd.snapid_race (5 ms) 2021-07-31T02:03:15.681 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsRbd.stripingv2 2021-07-31T02:03:15.690 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsRbd.stripingv2 (10 ms) 2021-07-31T02:03:15.691 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsRbd.object_map_save 2021-07-31T02:03:15.694 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsRbd.object_map_save (4 ms) 2021-07-31T02:03:15.695 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsRbd.object_map_resize 2021-07-31T02:03:15.699 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsRbd.object_map_resize (4 ms) 2021-07-31T02:03:15.699 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsRbd.object_map_update 2021-07-31T02:03:15.708 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsRbd.object_map_update (9 ms) 2021-07-31T02:03:15.709 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsRbd.object_map_load_enoent 2021-07-31T02:03:15.709 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsRbd.object_map_load_enoent (0 ms) 2021-07-31T02:03:15.709 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsRbd.object_map_snap_add 2021-07-31T02:03:15.715 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsRbd.object_map_snap_add (8 ms) 2021-07-31T02:03:15.716 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsRbd.object_map_snap_remove 2021-07-31T02:03:15.724 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsRbd.object_map_snap_remove (8 ms) 2021-07-31T02:03:15.725 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsRbd.flags 2021-07-31T02:03:15.729 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsRbd.flags (5 ms) 2021-07-31T02:03:15.729 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsRbd.metadata 2021-07-31T02:03:15.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:15 smithi154 conmon[34553]: cluster 2021-07-31T02:03:14.241051+0000 mon.a (mon.0 2021-07-31T02:03:15.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:15 smithi154 conmon[34553]: ) 1785 : cluster [DBG] osdmap e987: 8 total, 8 up, 8 in 2021-07-31T02:03:15.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:15 smithi154 conmon[34553]: audit 2021-07-31T02:03:14.242460+0000 mon.c (mon.1) 66 : audit [INF] from='client.? 172.21.15.45:0/1688263379' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74488-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:03:15.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:15 smithi154 conmon[34553]: audit 2021-07-31T02:03: 2021-07-31T02:03:15.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:15 smithi154 conmon[34553]: 14.242943+0000 mon.a (mon.0) 1786 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74488-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:03:15.757 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsRbd.metadata (28 ms) 2021-07-31T02:03:15.757 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsRbd.set_features 2021-07-31T02:03:15.764 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsRbd.set_features (7 ms) 2021-07-31T02:03:15.764 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsRbd.mirror 2021-07-31T02:03:15.801 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsRbd.mirror (37 ms) 2021-07-31T02:03:15.801 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsRbd.mirror_image 2021-07-31T02:03:15.816 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsRbd.mirror_image (15 ms) 2021-07-31T02:03:15.816 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsRbd.mirror_image_status 2021-07-31T02:03:16.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:16 smithi045 conmon[33655]: audit 2021-07-31T02:03:15.244049 2021-07-31T02:03:16.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:16 smithi045 conmon[33655]: +0000 mon.a (mon.0) 1787 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74488-1","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:03:16.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:16 smithi045 conmon[33655]: cluster 2021-07-31T02:03:15.244411+0000 mon.a (mon.0) 1788 : cluster [DBG] osdmap e988: 8 total, 8 up, 8 in 2021-07-31T02:03:16.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:16 smithi045 conmon[33655]: cluster 2021-07-31T02:03:16.020810+0000 mgr.x (mgr.24542) 684 : cluster [DBG] pgmap v1316: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:03:16.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:16 smithi045 conmon[33655]: cluster 2021-07-31T02:03:16.243828+0000 mon.a (mon.0) 1789 : cluster [DBG] osdmap e989: 8 total, 8 up, 8 in 2021-07-31T02:03:16.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:16 smithi045 conmon[39718]: audit 2021-07-31T02:03:15.244049+0000 mon.a (mon.0) 1787 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74488-1","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:03:16.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:16 smithi045 conmon[39718]: cluster 2021-07 2021-07-31T02:03:16.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:16 smithi045 conmon[39718]: -31T02:03:15.244411+0000 mon.a (mon.0) 1788 : cluster [DBG] osdmap e988: 8 total, 8 up, 8 in 2021-07-31T02:03:16.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:16 smithi045 conmon[39718]: cluster 2021- 2021-07-31T02:03:16.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:16 smithi045 conmon[39718]: 07-31T02:03:16.020810+0000 mgr.x (mgr.24542) 2021-07-31T02:03:16.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:16 smithi045 conmon[39718]: 684 : cluster [DBG] pgmap v1316: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:03:16.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:16 smithi045 conmon[39718]: cluster 2021-07-31T02 2021-07-31T02:03:16.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:16 smithi045 conmon[39718]: :03:16.243828+0000 mon.a (mon.0 2021-07-31T02:03:16.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:16 smithi045 conmon[39718]: ) 1789 : cluster [DBG] osdmap e989: 8 total, 8 up, 8 in 2021-07-31T02:03:16.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:16 smithi045 conmon[39718]: 2021-07-31T02:03:16.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:16 smithi154 conmon[34553]: audit 2021-07-31T02:03:15.244049+0000 mon.a (mon.0 2021-07-31T02:03:16.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:16 smithi154 conmon[34553]: ) 1787 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74488-1","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:03:16.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:16 smithi154 conmon[34553]: cluster 2021-07-31T02:03:15.244411+0000 mon.a (mon.0) 1788 : cluster [DBG] osdmap e988: 8 total, 8 up, 8 in 2021-07-31T02:03:16.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:16 smithi154 conmon[34553]: cluster 2021-07-31T02:03:16. 2021-07-31T02:03:16.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:16 smithi154 conmon[34553]: 020810+0000 mgr.x (mgr.24542) 684 : cluster 2021-07-31T02:03:16.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:16 smithi154 conmon[34553]: [DBG] pgmap v1316: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:03:16.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:16 smithi154 conmon[34553]: cluster 2021-07-31T02:03:16.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:16 smithi154 conmon[34553]: 2021-07-31T02:03 2021-07-31T02:03:16.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:16 smithi154 conmon[34553]: :16.243828+0000 mon.a (mon.0) 2021-07-31T02:03:16.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:16 smithi154 conmon[34553]: 1789 : cluster [DBG] osdmap e989: 8 total, 8 up, 8 in 2021-07-31T02:03:18.374 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsRbd.mirror_image_status (2558 ms) 2021-07-31T02:03:18.374 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsRbd.mirror_image_map 2021-07-31T02:03:18.456 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsRbd.mirror_image_map (82 ms) 2021-07-31T02:03:18.456 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsRbd.mirror_instances 2021-07-31T02:03:18.464 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsRbd.mirror_instances (8 ms) 2021-07-31T02:03:18.465 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsRbd.mirror_snapshot 2021-07-31T02:03:18.475 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsRbd.mirror_snapshot (11 ms) 2021-07-31T02:03:18.475 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsRbd.group_dir_list 2021-07-31T02:03:18.477 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsRbd.group_dir_list (3 ms) 2021-07-31T02:03:18.478 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsRbd.group_dir_add 2021-07-31T02:03:18.480 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsRbd.group_dir_add (2 ms) 2021-07-31T02:03:18.480 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsRbd.dir_add_already_existing 2021-07-31T02:03:18.482 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsRbd.dir_add_already_existing (3 ms) 2021-07-31T02:03:18.483 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsRbd.group_dir_rename 2021-07-31T02:03:18.489 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsRbd.group_dir_rename (6 ms) 2021-07-31T02:03:18.489 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsRbd.group_dir_remove 2021-07-31T02:03:18.492 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsRbd.group_dir_remove (3 ms) 2021-07-31T02:03:18.492 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsRbd.group_dir_remove_missing 2021-07-31T02:03:18.496 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsRbd.group_dir_remove_missing (4 ms) 2021-07-31T02:03:18.496 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsRbd.group_image_add 2021-07-31T02:03:18.499 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsRbd.group_image_add (3 ms) 2021-07-31T02:03:18.499 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsRbd.group_image_remove 2021-07-31T02:03:18.502 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsRbd.group_image_remove (3 ms) 2021-07-31T02:03:18.502 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsRbd.group_image_list 2021-07-31T02:03:18.505 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsRbd.group_image_list (3 ms) 2021-07-31T02:03:18.505 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsRbd.group_image_clean 2021-07-31T02:03:18.510 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsRbd.group_image_clean (6 ms) 2021-07-31T02:03:18.511 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsRbd.image_group_add 2021-07-31T02:03:18.513 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsRbd.image_group_add (2 ms) 2021-07-31T02:03:18.513 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsRbd.image_group_remove 2021-07-31T02:03:18.515 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsRbd.image_group_remove (3 ms) 2021-07-31T02:03:18.516 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsRbd.image_group_get 2021-07-31T02:03:18.518 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsRbd.image_group_get (2 ms) 2021-07-31T02:03:18.518 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsRbd.group_snap_set_empty_name 2021-07-31T02:03:18.520 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsRbd.group_snap_set_empty_name (2 ms) 2021-07-31T02:03:18.520 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsRbd.group_snap_set_empty_id 2021-07-31T02:03:18.522 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsRbd.group_snap_set_empty_id (2 ms) 2021-07-31T02:03:18.523 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsRbd.group_snap_set_duplicate_id 2021-07-31T02:03:18.525 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsRbd.group_snap_set_duplicate_id (4 ms) 2021-07-31T02:03:18.526 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsRbd.group_snap_set_duplicate_name 2021-07-31T02:03:18.528 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsRbd.group_snap_set_duplicate_name (2 ms) 2021-07-31T02:03:18.528 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsRbd.group_snap_set 2021-07-31T02:03:18.530 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsRbd.group_snap_set (3 ms) 2021-07-31T02:03:18.531 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsRbd.group_snap_list 2021-07-31T02:03:18.533 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsRbd.group_snap_list (3 ms) 2021-07-31T02:03:18.534 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsRbd.group_snap_list_max_return 2021-07-31T02:03:18.550 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsRbd.group_snap_list_max_return (17 ms) 2021-07-31T02:03:18.551 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsRbd.group_snap_remove 2021-07-31T02:03:18.555 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsRbd.group_snap_remove (4 ms) 2021-07-31T02:03:18.555 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsRbd.group_snap_get_by_id 2021-07-31T02:03:18.557 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsRbd.group_snap_get_by_id (3 ms) 2021-07-31T02:03:18.557 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsRbd.trash_methods 2021-07-31T02:03:18.565 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsRbd.trash_methods (7 ms) 2021-07-31T02:03:18.566 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsRbd.op_features 2021-07-31T02:03:18.572 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsRbd.op_features (7 ms) 2021-07-31T02:03:18.572 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsRbd.clone_parent 2021-07-31T02:03:18.593 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsRbd.clone_parent (21 ms) 2021-07-31T02:03:18.594 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsRbd.clone_parent_ns 2021-07-31T02:03:18.604 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsRbd.clone_parent_ns (11 ms) 2021-07-31T02:03:18.604 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsRbd.clone_child 2021-07-31T02:03:18.615 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsRbd.clone_child (11 ms) 2021-07-31T02:03:18.615 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsRbd.namespace_methods 2021-07-31T02:03:18.622 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsRbd.namespace_methods (7 ms) 2021-07-31T02:03:18.622 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsRbd.migration 2021-07-31T02:03:18.631 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsRbd.migration (9 ms) 2021-07-31T02:03:18.631 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsRbd.migration_v1 2021-07-31T02:03:18.644 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsRbd.migration_v1 (13 ms) 2021-07-31T02:03:18.644 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsRbd.assert_snapc_seq 2021-07-31T02:03:19.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:19 smithi154 conmon[34553]: cluster 2021-07-31T02:03:19.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:19 smithi154 conmon[34553]: 2021-07-31T02:03:19.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:19 smithi154 conmon[34553]: 2021-07-31T02: 2021-07-31T02:03:19.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:19 smithi154 conmon[34553]: 03:18.021149+0000 mgr.x (mgr.24542) 685 : cluster [DBG] pgmap v1318: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:03:19.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:19 smithi045 conmon[33655]: cluster 2021-07-31T02:03:18.021149+0000 mgr.x (mgr.24542) 685 : 2021-07-31T02:03:19.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:19 smithi045 conmon[33655]: cluster [DBG] pgmap v1318: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:03:19.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:19 smithi045 conmon[39718]: cluster 2021-07-31T02:03:18.021149+0000 mgr.x (mgr 2021-07-31T02:03:19.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:19 smithi045 conmon[39718]: .24542) 685 : cluster [DBG] pgmap v1318: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:03:20.093 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsRbd.assert_snapc_seq (1449 ms) 2021-07-31T02:03:20.093 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestClsRbd.sparsify 2021-07-31T02:03:20.133 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestClsRbd.sparsify (40 ms) 2021-07-31T02:03:20.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:20 smithi154 conmon[34553]: cluster 2021-07-31T02:03:19.081080+0000 mon.a (mon.0) 1790 : cluster [DBG] osdmap e990: 8 total, 8 up, 8 in 2021-07-31T02:03:20.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:20 smithi154 conmon[34553]: 2021-07-31T02:03:20.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:20 smithi045 conmon[33655]: cluster 2021-07-31T02:03:19.081080+0000 2021-07-31T02:03:20.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:20 smithi045 conmon[33655]: mon.a (mon.0) 1790 : cluster [DBG] osdmap e990: 8 total, 8 up, 8 in 2021-07-31T02:03:20.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:20 smithi045 conmon[39718]: cluster 2021-07-31T02:03:19.081080+0000 mon.a ( 2021-07-31T02:03:20.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:20 smithi045 conmon[39718]: mon.0) 1790 : cluster [DBG] osdmap e990: 8 total, 8 up, 8 in 2021-07-31T02:03:21.100 INFO:tasks.workunit.client.0.smithi045.stdout:[----------] 69 tests from TestClsRbd (4873 ms total) 2021-07-31T02:03:21.100 INFO:tasks.workunit.client.0.smithi045.stdout: 2021-07-31T02:03:21.101 INFO:tasks.workunit.client.0.smithi045.stdout:[----------] Global test environment tear-down 2021-07-31T02:03:21.101 INFO:tasks.workunit.client.0.smithi045.stdout:[==========] 69 tests from 1 test suite ran. (7762 ms total) 2021-07-31T02:03:21.101 INFO:tasks.workunit.client.0.smithi045.stdout:[ PASSED ] 69 tests. 2021-07-31T02:03:21.102 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-07-31T02:03:21.103 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2021-07-31T02:03:21.174 INFO:tasks.workunit:Running workunit cls/test_cls_refcount.sh... 2021-07-31T02:03:21.175 DEBUG:teuthology.orchestra.run.smithi045:workunit test cls/test_cls_refcount.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=pacific TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_refcount.sh 2021-07-31T02:03:21.248 INFO:tasks.workunit.client.0.smithi045.stdout:Running main() from gmock_main.cc 2021-07-31T02:03:21.248 INFO:tasks.workunit.client.0.smithi045.stdout:[==========] Running 5 tests from 1 test suite. 2021-07-31T02:03:21.248 INFO:tasks.workunit.client.0.smithi045.stdout:[----------] Global test environment set-up. 2021-07-31T02:03:21.248 INFO:tasks.workunit.client.0.smithi045.stdout:[----------] 5 tests from cls_rgw 2021-07-31T02:03:21.249 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] cls_rgw.test_implicit 2021-07-31T02:03:21.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:21 smithi154 conmon[34553]: cluster 2021-07-31T02:03:20.021616+0000 mgr.x (mgr.24542) 686 : cluster [DBG] pgmap v1320: 41 pgs: 41 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 2.7 MiB/s rd, 1.4 MiB/s wr, 2.08k op/s 2021-07-31T02:03:21.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:21 smithi154 conmon[34553]: 2021-07-31T02:03:21.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:21 smithi154 conmon[34553]: cluster 2021-07-31T02:03:20.091895+0000 mon.a (mon.0) 1791 : cluster [DBG] osdmap e991: 8 total, 8 up, 8 in 2021-07-31T02:03:21.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:21 smithi045 conmon[33655]: cluster 2021-07-31T02:03:20.021616+0000 mgr.x (mgr.24542) 686 : cluster 2021-07-31T02:03:21.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:21 smithi045 conmon[33655]: [DBG] pgmap v1320: 41 pgs: 41 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 2.7 MiB/s rd, 1.4 MiB/s wr, 2.08k op/s 2021-07-31T02:03:21.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:21 smithi045 conmon[33655]: cluster 2021-07-31T02:03:20.091895+0000 mon.a (mon.0) 1791 : cluster [DBG] 2021-07-31T02:03:21.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:21 smithi045 conmon[33655]: osdmap e991: 8 total, 8 up, 8 in 2021-07-31T02:03:21.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:21 smithi045 conmon[39718]: cluster 2021-07-31T02:03:20.021616+0000 2021-07-31T02:03:21.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:21 smithi045 conmon[39718]: mgr.x (mgr.24542) 686 : cluster [DBG] pgmap v1320: 41 pgs: 41 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 2.7 MiB/s rd, 1.4 MiB/s wr, 2.08k op/s 2021-07-31T02:03:21.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:21 smithi045 conmon[39718]: cluster 2021-07-31T02:03:21.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:21 smithi045 conmon[39718]: 2021-07-31T02:03:20.091895+0000 mon.a (mon. 2021-07-31T02:03:21.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:21 smithi045 conmon[39718]: 0) 1791 : cluster [DBG] osdmap e991: 8 total, 8 up, 8 in 2021-07-31T02:03:22.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:22 smithi154 conmon[34553]: cluster 2021-07-31T02:03:21.097098+0000 mon.a (mon.0) 1792 : cluster [DBG] osdmap e992: 8 total, 8 up, 8 in 2021-07-31T02:03:22.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:22 smithi154 conmon[34553]: cluster 2021-07-31T02:03:22.022177+0000 mgr.x (mgr.24542) 687 : cluster [DBG] pgmap v1323: 9 pgs: 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:03:22.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:22 smithi045 conmon[33655]: cluster 2021-07-31T02:03:21.097098+0000 mon.a (mon 2021-07-31T02:03:22.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:22 smithi045 conmon[33655]: .0) 1792 : cluster [DBG] osdmap e992: 8 total, 8 up, 8 in 2021-07-31T02:03:22.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:22 smithi045 conmon[33655]: cluster 2021-07-31T02:03:22.022177+0000 mgr.x (mgr.24542) 687 : cluster [DBG] pgmap v1323: 9 pgs: 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:03:22.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:22 smithi045 conmon[39718]: cluster 2021-07-31T02:03:21.097098+0000 mon.a 2021-07-31T02:03:22.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:22 smithi045 conmon[39718]: (mon.0) 1792 : cluster [DBG] osdmap e992: 8 total, 8 up, 8 in 2021-07-31T02:03:22.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:22 smithi045 conmon[39718]: cluster 2021-07 2021-07-31T02:03:22.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:22 smithi045 conmon[39718]: -31T02:03:22.022177+0000 mgr.x (mgr.24542) 687 : cluster [DBG] pgmap v1323: 9 pgs: 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:03:23.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:23 smithi154 conmon[34553]: cluster 2021-07-31T02:03:22.102053+0000 mon.a (mon.0) 1793 : cluster [DBG] osdmap e993: 8 total, 8 up, 8 in 2021-07-31T02:03:23.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:23 smithi154 conmon[34553]: audit 2021-07-31T02:03:22.105019+0000 mon.c (mon.1) 2021-07-31T02:03:23.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:23 smithi154 conmon[34553]: 67 : audit [INF] from='client.? 172.21.15.45:0/293728892' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74549-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:03:23.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:23 smithi154 conmon[34553]: audit 2021-07-31T02:03:22.105539+0000 mon.a (mon.0) 1794 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74549-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:03:23.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:23 smithi045 conmon[33655]: cluster 2021-07-31T02:03:22.102053+0000 mon.a (mon.0) 2021-07-31T02:03:23.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:23 smithi045 conmon[33655]: 1793 : cluster [DBG] osdmap e993: 8 total, 8 up, 8 in 2021-07-31T02:03:23.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:23 smithi045 conmon[33655]: audit 2021-07-31T02:03:22.105019+0000 mon.c (mon.1) 67 : audit [INF] from='client.? 172.21.15.45:0/293728892' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74549-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:03:23.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:23 smithi045 conmon[33655]: audit 2021-07-31T02:03: 2021-07-31T02:03:23.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:23 smithi045 conmon[33655]: 22.105539+0000 mon.a (mon.0) 1794 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74549-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:03:23.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:23 smithi045 conmon[39718]: cluster 2021-07-31T02:03:22 2021-07-31T02:03:23.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:23 smithi045 conmon[39718]: .102053+0000 mon.a (mon.0) 1793 : cluster [DBG] osdmap e993: 8 total, 8 up, 8 in 2021-07-31T02:03:23.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:23 smithi045 conmon[39718]: audit 2021-07-31T02:03 2021-07-31T02:03:23.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:23 smithi045 conmon[39718]: :22.105019+0000 mon.c (mon.1) 67 2021-07-31T02:03:23.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:23 smithi045 conmon[39718]: : audit [INF] from='client.? 172.21.15.45:0/293728892' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74549-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:03:23.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:23 smithi045 conmon[39718]: audit 2021- 2021-07-31T02:03:23.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:23 smithi045 conmon[39718]: 07-31T02:03:22.105539 2021-07-31T02:03:23.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:23 smithi045 conmon[39718]: +0000 mon.a (mon.0) 1794 : audit 2021-07-31T02:03:23.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:23 smithi045 conmon[39718]: [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74549-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:03:24.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:03:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:03:23.535Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:03:24.027 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:03:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:03:23.535Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:03:24.027 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:03:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:03:23.535Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:03:24.118 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] cls_rgw.test_implicit (2870 ms) 2021-07-31T02:03:24.118 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] cls_rgw.test_implicit_idempotent 2021-07-31T02:03:24.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:24 smithi154 conmon[34553]: audit 2021-07-31T02:03:23.109502+0000 mon.a (mon.0) 1795 : audit 2021-07-31T02:03:24.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:24 smithi154 conmon[34553]: [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74549-1","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:03:24.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:24 smithi154 conmon[34553]: cluster 2021-07-31T02:03:23.109694+0000 mon.a (mon.0) 1796 : cluster [DBG] osdmap e994: 8 total, 8 up, 8 in 2021-07-31T02:03:24.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:24 smithi154 conmon[34553]: cluster 2021-07-31T02:03:24.022667 2021-07-31T02:03:24.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:24 smithi154 conmon[34553]: +0000 mgr.x (mgr.24542) 688 : cluster [DBG] pgmap v1326: 41 pgs: 41 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:03:24.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:24 smithi045 conmon[33655]: audit 2021-07-31T02:03:23.109502+0000 mon.a (mon.0) 1795 : audit 2021-07-31T02:03:24.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:24 smithi045 conmon[33655]: [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74549-1","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:03:24.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:24 smithi045 conmon[33655]: cluster 2021-07-31T02:03:23.109694+0000 mon.a (mon.0) 1796 : cluster [DBG] osdmap e994: 8 total, 8 up, 8 in 2021-07-31T02:03:24.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:24 smithi045 conmon[33655]: cluster 2021-07-31 2021-07-31T02:03:24.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:24 smithi045 conmon[33655]: T02:03:24.022667+0000 mgr.x (mgr.24542) 688 : cluster [DBG] pgmap v1326: 41 pgs: 41 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:03:24.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:24 smithi045 conmon[39718]: audit 2021-07-31T02:03:23.109502+0000 2021-07-31T02:03:24.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:24 smithi045 conmon[39718]: mon.a (mon.0) 1795 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74549-1","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:03:24.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:24 smithi045 conmon[39718]: cluster 2021- 2021-07-31T02:03:24.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:24 smithi045 conmon[39718]: 07-31T02:03:23.109694+0000 mon.a (mon 2021-07-31T02:03:24.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:24 smithi045 conmon[39718]: .0) 1796 : cluster [DBG] osdmap e994: 8 total, 8 up, 8 in 2021-07-31T02:03:24.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:24 smithi045 conmon[39718]: cluster 2021-07 2021-07-31T02:03:24.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:24 smithi045 conmon[39718]: -31T02:03:24.022667+0000 2021-07-31T02:03:24.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:24 smithi045 conmon[39718]: mgr.x (mgr.24542) 688 : cluster [DBG] 2021-07-31T02:03:24.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:24 smithi045 conmon[39718]: pgmap v1326: 41 pgs: 41 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:03:25.015 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:03:25.299 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:03:25.300 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 18m ago 19m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:03:25.300 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 15m ago 18m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:03:25.300 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 15m ago 22m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:03:25.300 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 18m ago 24m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:03:25.301 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 18m ago 24m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:03:25.301 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 15m ago 22m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:03:25.301 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 18m ago 22m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:03:25.301 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 18m ago 19m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:03:25.302 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 15m ago 19m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:03:25.302 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 18m ago 22m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:03:25.302 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 18m ago 21m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:03:25.302 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 18m ago 21m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:03:25.303 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 18m ago 21m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:03:25.303 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 15m ago 20m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:03:25.303 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 15m ago 20m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:03:25.303 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 15m ago 20m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:03:25.303 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 15m ago 19m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:03:25.304 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 15m ago 19m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:03:25.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:25 smithi154 conmon[34553]: cluster 2021-07-31T02:03:24.115732+0000 mon.a (mon.0) 1797 : cluster [DBG] osdmap e995: 8 total, 8 up, 8 in 2021-07-31T02:03:25.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:25 smithi154 conmon[34553]: audit 2021-07-31T02:03:25.003443+0000 mgr.x (mgr.24542) 689 : audit [DBG] from='client.15612 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:03:25.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:25 smithi045 conmon[33655]: cluster 2021-07-31T02:03:24.115732+0000 mon.a 2021-07-31T02:03:25.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:25 smithi045 conmon[33655]: (mon.0) 1797 : cluster [DBG] osdmap e995: 8 total, 8 up, 8 in 2021-07-31T02:03:25.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:25 smithi045 conmon[33655]: audit 2021- 2021-07-31T02:03:25.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:25 smithi045 conmon[33655]: 07-31T02:03:25.003443+0000 2021-07-31T02:03:25.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:25 smithi045 conmon[33655]: mgr.x (mgr.24542) 689 : 2021-07-31T02:03:25.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:25 smithi045 conmon[33655]: audit [DBG] from='client.15612 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:03:25.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:25 smithi045 conmon[39718]: cluster 2021-07-31T02:03 2021-07-31T02:03:25.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:25 smithi045 conmon[39718]: :24.115732+0000 mon.a (mon.0) 1797 : cluster [DBG] osdmap e995: 8 total, 8 up, 8 in 2021-07-31T02:03:25.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:25 smithi045 conmon[39718]: audit 2021- 2021-07-31T02:03:25.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:25 smithi045 conmon[39718]: 07-31T02:03:25 2021-07-31T02:03:25.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:25 smithi045 conmon[39718]: .003443+0000 mgr.x (mgr. 2021-07-31T02:03:25.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:25 smithi045 conmon[39718]: 24542) 689 : audit [DBG] 2021-07-31T02:03:25.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:25 smithi045 conmon[39718]: from='client.15612 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:03:25.636 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:03:25.636 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:03:25.637 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:03:25.637 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:03:25.637 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:03:25.637 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:03:25.637 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:03:25.638 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:03:25.638 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:03:25.638 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:03:25.638 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:03:25.640 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:03:25.640 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:03:25.640 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:03:25.641 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:03:25.641 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:03:25.642 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:03:26.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:26 smithi154 conmon[34553]: cluster 2021-07-31T02:03:25.122048+0000 mon.a (mon.0) 1798 : 2021-07-31T02:03:26.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:26 smithi154 conmon[34553]: cluster [DBG] osdmap e996: 8 total, 8 up, 8 in 2021-07-31T02:03:26.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:26 smithi154 conmon[34553]: audit 2021-07-31T02:03:25.122823+0000 mon.a (mon.0) 1799 : audit [INF] from='client.? 172.21.15.45:0/65362390' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74549-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:03:26.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:26 smithi154 conmon[34553]: audit 2021-07- 2021-07-31T02:03:26.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:26 smithi154 conmon[34553]: 31T02:03:25.291191+0000 mgr.x (mgr.24542) 690 : audit [DBG] from='client.15618 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:03:26.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:26 smithi154 conmon[34553]: audit 2021-07-31T02:03:25. 2021-07-31T02:03:26.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:26 smithi154 conmon[34553]: 636067+0000 mon.a (mon.0) 1800 : audit [DBG] from='client.? 172.21.15.45:0/4178657252' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:03:26.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:26 smithi154 conmon[34553]: cluster 2021 2021-07-31T02:03:26.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:26 smithi154 conmon[34553]: -07-31T02:03:26.023140+0000 mgr.x ( 2021-07-31T02:03:26.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:26 smithi154 conmon[34553]: mgr.24542) 691 : cluster [DBG] pgmap v1329: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:03:26.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:26 smithi045 conmon[33655]: cluster 2021-07-31T02:03:25.122048+0000 mon.a (mon.0 2021-07-31T02:03:26.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:26 smithi045 conmon[33655]: ) 1798 : cluster [DBG] osdmap e996: 8 total, 8 up, 8 in 2021-07-31T02:03:26.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:26 smithi045 conmon[33655]: audit 2021-07-31T02:03:25.122823+0000 mon.a (mon.0) 1799 : audit [INF] from='client.? 172.21.15.45:0/65362390' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74549-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:03:26.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:26 smithi045 conmon[33655]: audit 2021-07-31T02:03:26.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:26 smithi045 conmon[33655]: 2021-07-31T02:03:25.291191+0000 mgr.x (mgr. 2021-07-31T02:03:26.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:26 smithi045 conmon[33655]: 24542) 690 : audit [DBG] from='client.15618 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:03:26.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:26 smithi045 conmon[33655]: audit 2021- 2021-07-31T02:03:26.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:26 smithi045 conmon[33655]: 07-31T02:03:25.636067+0000 2021-07-31T02:03:26.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:26 smithi045 conmon[33655]: mon.a (mon.0) 1800 : 2021-07-31T02:03:26.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:26 smithi045 conmon[33655]: audit [DBG] from='client.? 172.21.15.45:0/4178657252' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:03:26.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:26 smithi045 conmon[33655]: cluster 2021-07 2021-07-31T02:03:26.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:26 smithi045 conmon[33655]: -31T02:03:26.023140+0000 mgr.x (mgr. 2021-07-31T02:03:26.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:26 smithi045 conmon[33655]: 24542) 691 : cluster [DBG] pgmap v1329: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:03:26.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:26 smithi045 conmon[39718]: cluster 2021-07-31T02:03:25.122048+0000 2021-07-31T02:03:26.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:26 smithi045 conmon[39718]: mon.a (mon.0) 1798 : cluster [DBG] osdmap e996: 8 total, 8 up, 8 in 2021-07-31T02:03:26.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:26 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T02:03:26.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:26 smithi045 conmon[39718]: 02:03:25.122823+0000 mon.a (mon.0) 1799 : audit [INF] from='client.? 172.21.15.45:0/65362390' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74549-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:03:26.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:26 smithi045 conmon[39718]: audit 2021-07-31T02:03:25.291191 2021-07-31T02:03:26.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:26 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 690 : audit [DBG] 2021-07-31T02:03:26.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:26 smithi045 conmon[39718]: from='client.15618 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:03:26.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:26 smithi045 conmon[39718]: audit 2021-07-31T02:03:26.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:26 smithi045 conmon[39718]: 2021-07-31T02:03:25.636067+0000 2021-07-31T02:03:26.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:26 smithi045 conmon[39718]: mon.a (mon.0) 1800 : audit [DBG] 2021-07-31T02:03:26.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:26 smithi045 conmon[39718]: from='client.? 172.21.15.45:0/4178657252' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:03:26.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:26 smithi045 conmon[39718]: cluster 2021-07-31T02:03:26.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:26 smithi045 conmon[39718]: 2021-07-31T02:03:26.023140+0000 2021-07-31T02:03:26.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:26 smithi045 conmon[39718]: mgr.x (mgr.24542) 691 : cluster [DBG] 2021-07-31T02:03:26.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:26 smithi045 conmon[39718]: pgmap v1329: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:03:27.132 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] cls_rgw.test_implicit_idempotent (3014 ms) 2021-07-31T02:03:27.133 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] cls_rgw.test_put_snap 2021-07-31T02:03:27.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:27 smithi154 conmon[34553]: audit 2021-07-31T02:03:27.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:27 smithi154 conmon[34553]: 2021-07-31T02:03:27.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:27 smithi154 conmon[34553]: 2021 2021-07-31T02:03:27.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:27 smithi154 conmon[34553]: - 2021-07-31T02:03:27.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:27 smithi154 conmon[34553]: 07 2021-07-31T02:03:27.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:27 smithi154 conmon[34553]: - 2021-07-31T02:03:27.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:27 smithi154 conmon[34553]: 31T02:03:26. 2021-07-31T02:03:27.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:27 smithi154 conmon[34553]: 128794+0000 mon.a (mon.0) 1801 : audit [INF] from='client.? 172.21.15.45:0/65362390' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74549-2","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:03:27.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:27 smithi154 conmon[34553]: cluster 2021-07-31T02:03:27.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:27 smithi154 conmon[34553]: 2021-07-31T02:03:26.129025+0000 2021-07-31T02:03:27.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:27 smithi154 conmon[34553]: mon.a (mon.0) 1802 : cluster [DBG] osdmap e997: 8 total, 8 up, 8 in 2021-07-31T02:03:27.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:27 smithi045 conmon[33655]: audit 2021-07-31T02:03:26.128794+0000 mon.a (mon.0) 1801 : 2021-07-31T02:03:27.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:27 smithi045 conmon[33655]: audit [INF] from='client.? 172.21.15.45:0/65362390' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74549-2","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:03:27.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:27 smithi045 conmon[33655]: cluster 2021-07-31T02:03:26.129025+0000 mon.a (mon.0) 1802 : cluster [DBG] osdmap e997: 8 total, 8 up, 8 in 2021-07-31T02:03:27.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:27 smithi045 conmon[39718]: audit 2021-07-31T02:03:26.128794+0000 mon.a ( 2021-07-31T02:03:27.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:27 smithi045 conmon[39718]: mon.0) 1801 : audit [INF] from='client.? 172.21.15.45:0/65362390' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74549-2","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:03:27.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:27 smithi045 conmon[39718]: cluster 2021-07- 2021-07-31T02:03:27.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:27 smithi045 conmon[39718]: 31T02:03:26.129025+0000 mon.a (mon.0) 1802 : cluster 2021-07-31T02:03:27.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:27 smithi045 conmon[39718]: [DBG] osdmap e997: 8 total, 8 up, 8 in 2021-07-31T02:03:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:28 smithi154 conmon[34553]: cluster 2021-07-31T02:03:27.129615+0000 mon.a (mon 2021-07-31T02:03:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:28 smithi154 conmon[34553]: .0) 1803 : cluster [DBG] osdmap e998: 8 total, 8 up, 8 in 2021-07-31T02:03:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:28 smithi154 conmon[34553]: audit 2021-07-31T02:03:27.994863+0000 mon.b (mon.2) 148 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:03:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:28 smithi154 conmon[34553]: audit 2021-07-31T02:03:27.995530+0000 2021-07-31T02:03:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:28 smithi154 conmon[34553]: mon.a (mon.0) 1804 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:03:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:28 smithi154 conmon[34553]: audit 2021-07-31T02:03 2021-07-31T02:03:28.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:28 smithi154 conmon[34553]: :28.012402+0000 mon.b (mon.2) 149 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:03:28.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:28 smithi154 conmon[34553]: audit 2021 2021-07-31T02:03:28.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:28 smithi154 conmon[34553]: -07-31T02:03:28.013103+0000 2021-07-31T02:03:28.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:28 smithi154 conmon[34553]: mon.a (mon.0) 1805 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:03:28.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:28 smithi154 conmon[34553]: 2021-07-31T02:03:28.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:28 smithi154 conmon[34553]: cluster 2021-07-31T02:03:28.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:28 smithi154 conmon[34553]: 2021-07-31T02:03:28.023503+0000 mgr.x (mgr. 2021-07-31T02:03:28.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:28 smithi154 conmon[34553]: 24542) 692 : cluster [DBG] pgmap v1332: 9 pgs: 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:03:28.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:28 smithi045 conmon[33655]: cluster 2021-07-31T02:03:27.129615+0000 2021-07-31T02:03:28.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:28 smithi045 conmon[33655]: mon.a (mon.0) 1803 : cluster [DBG] osdmap e998: 8 total, 8 up, 8 in 2021-07-31T02:03:28.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:28 smithi045 conmon[33655]: audit 2021-07-31T02:03:27.994863+0000 mon.b (mon.2) 148 2021-07-31T02:03:28.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:28 smithi045 conmon[33655]: : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:03:28.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:28 smithi045 conmon[33655]: audit 2021-07-31T02:03:27.995530+0000 2021-07-31T02:03:28.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:28 smithi045 conmon[33655]: mon.a (mon.0) 1804 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:03:28.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:28 smithi045 conmon[33655]: audit 2021-07-31T02 2021-07-31T02:03:28.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:28 smithi045 conmon[33655]: :03:28.012402+0000 mon.b (mon.2) 149 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:03:28.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:28 smithi045 conmon[33655]: 2021-07-31T02:03:28.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:28 smithi045 conmon[33655]: audit 2021-07-31 2021-07-31T02:03:28.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:28 smithi045 conmon[33655]: T02:03:28.013103+0000 mon.a (mon.0 2021-07-31T02:03:28.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:28 smithi045 conmon[33655]: ) 1805 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:03:28.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:28 smithi045 conmon[33655]: cluster 2021-07- 2021-07-31T02:03:28.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:28 smithi045 conmon[33655]: 31T02:03:28.023503+0000 mgr.x (mgr.24542) 2021-07-31T02:03:28.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:28 smithi045 conmon[33655]: 692 : cluster [DBG] pgmap v1332: 9 pgs: 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:03:28.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:28 smithi045 conmon[39718]: cluster 2021-07-31T02:03:27.129615+0000 mon.a (mon 2021-07-31T02:03:28.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:28 smithi045 conmon[39718]: .0) 1803 : cluster [DBG] osdmap e998: 8 total, 8 up, 8 in 2021-07-31T02:03:28.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:28 smithi045 conmon[39718]: audit 2021-07-31T02:03:27. 2021-07-31T02:03:28.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:28 smithi045 conmon[39718]: 994863+0000 mon.b (mon.2) 148 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:03:28.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:28 smithi045 conmon[39718]: audit 2021-07-31T02 2021-07-31T02:03:28.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:28 smithi045 conmon[39718]: :03:27.995530+0000 mon.a (mon.0) 2021-07-31T02:03:28.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:28 smithi045 conmon[39718]: 1804 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:03:28.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:28 smithi045 conmon[39718]: audit 2021-07-31T02 2021-07-31T02:03:28.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:28 smithi045 conmon[39718]: :03:28.012402+0000 mon.b (mon.2) 2021-07-31T02:03:28.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:28 smithi045 conmon[39718]: 149 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:03:28.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:28 smithi045 conmon[39718]: audit 2021-07-31T02:03:28.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:28 smithi045 conmon[39718]: 2021-07- 2021-07-31T02:03:28.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:28 smithi045 conmon[39718]: 31T02:03:28.013103+0000 mon.a (mon 2021-07-31T02:03:28.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:28 smithi045 conmon[39718]: .0) 1805 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:03:28.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:28 smithi045 conmon[39718]: cluster 2021-07-31 2021-07-31T02:03:28.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:28 smithi045 conmon[39718]: T02:03:28.023503+0000 mgr.x (mgr.24542) 692 : cluster 2021-07-31T02:03:28.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:28 smithi045 conmon[39718]: [DBG] pgmap v1332: 9 pgs: 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:03:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[34553]: cluster 2021-07-31T02:03:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[34553]: 2021-07-31T02:03:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[34553]: 2021 2021-07-31T02:03:29.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[34553]: -07-31T02: 2021-07-31T02:03:29.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[34553]: 03:28.143766+0000 mon.a (mon.0) 1806 : cluster [DBG] osdmap e999: 8 total, 8 up, 8 in 2021-07-31T02:03:29.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[34553]: audit 2021-07-31T02:03:29.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[34553]: 2021-07-31T02:03:28.144666 2021-07-31T02:03:29.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[34553]: +0000 mon.c (mon.1) 68 : audit [INF] 2021-07-31T02:03:29.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[34553]: from='client.? 172.21.15.45:0/1417384092' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74549-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:03:29.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[34553]: audit 2021-07-31T02:03:29.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[34553]: 2021-07-31T02:03:28.145256+0000 mon.a (mon.0) 1807 : 2021-07-31T02:03:29.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[34553]: audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74549-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:03:29.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:29 smithi045 conmon[39718]: cluster 2021-07-31T02:03:29.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:29 smithi045 conmon[39718]: 2021-07-31T02:03:29.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:29 smithi045 conmon[39718]: 2021 2021-07-31T02:03:29.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:29 smithi045 conmon[39718]: - 2021-07-31T02:03:29.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:29 smithi045 conmon[39718]: 07 2021-07-31T02:03:29.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:29 smithi045 conmon[39718]: - 2021-07-31T02:03:29.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:29 smithi045 conmon[39718]: 31 2021-07-31T02:03:29.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:29 smithi045 conmon[39718]: T 2021-07-31T02:03:29.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:29 smithi045 conmon[39718]: 02 2021-07-31T02:03:29.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:29 smithi045 conmon[39718]: : 2021-07-31T02:03:29.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:29 smithi045 conmon[39718]: 03 2021-07-31T02:03:29.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:29 smithi045 conmon[39718]: : 2021-07-31T02:03:29.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:29 smithi045 conmon[39718]: 28 2021-07-31T02:03:29.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:29 smithi045 conmon[39718]: . 2021-07-31T02:03:29.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:29 smithi045 conmon[39718]: 143766 2021-07-31T02:03:29.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:29 smithi045 conmon[39718]: +0000 2021-07-31T02:03:29.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:29 smithi045 conmon[39718]: 2021-07-31T02:03:29.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:29 smithi045 conmon[39718]: mon.a 2021-07-31T02:03:29.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:29 smithi045 conmon[39718]: ( 2021-07-31T02:03:29.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:29 smithi045 conmon[39718]: mon 2021-07-31T02:03:29.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:29 smithi045 conmon[39718]: . 2021-07-31T02:03:29.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:29 smithi045 conmon[39718]: 0 2021-07-31T02:03:29.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:29 smithi045 conmon[39718]: ) 2021-07-31T02:03:29.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:29 smithi045 conmon[39718]: 1806 2021-07-31T02:03:29.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:29 smithi045 conmon[39718]: : 2021-07-31T02:03:29.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:29 smithi045 conmon[39718]: cluster 2021-07-31T02:03:29.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:29 smithi045 conmon[39718]: 2021-07-31T02:03:29.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:29 smithi045 conmon[39718]: [DBG] 2021-07-31T02:03:29.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:29 smithi045 conmon[39718]: 2021-07-31T02:03:29.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:29 smithi045 conmon[39718]: osdmap e999: 8 total, 8 up, 8 in 2021-07-31T02:03:29.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:29 smithi045 conmon[39718]: 2021-07-31T02:03:29.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:29 smithi045 conmon[39718]: audit 2021-07-31T02:03:29.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:29 smithi045 conmon[39718]: 2021-07-31T02:03:29.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:29 smithi045 conmon[39718]: 2021 2021-07-31T02:03:29.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:29 smithi045 conmon[39718]: - 2021-07-31T02:03:29.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:29 smithi045 conmon[39718]: 07 2021-07-31T02:03:29.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:29 smithi045 conmon[39718]: - 2021-07-31T02:03:29.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:29 smithi045 conmon[39718]: 31 2021-07-31T02:03:29.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:29 smithi045 conmon[39718]: T 2021-07-31T02:03:29.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:29 smithi045 conmon[39718]: 02 2021-07-31T02:03:29.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:29 smithi045 conmon[39718]: : 2021-07-31T02:03:29.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:29 smithi045 conmon[39718]: 03 2021-07-31T02:03:29.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:29 smithi045 conmon[39718]: : 2021-07-31T02:03:29.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:29 smithi045 conmon[39718]: 28 2021-07-31T02:03:29.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:29 smithi045 conmon[39718]: . 2021-07-31T02:03:29.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:29 smithi045 conmon[39718]: 144666 2021-07-31T02:03:29.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:29 smithi045 conmon[39718]: +0000 2021-07-31T02:03:29.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:29 smithi045 conmon[39718]: 2021-07-31T02:03:29.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:29 smithi045 conmon[39718]: mon.c 2021-07-31T02:03:29.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:29 smithi045 conmon[39718]: ( 2021-07-31T02:03:29.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:29 smithi045 conmon[39718]: mon 2021-07-31T02:03:29.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:29 smithi045 conmon[39718]: . 2021-07-31T02:03:29.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:29 smithi045 conmon[39718]: 1 2021-07-31T02:03:29.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:29 smithi045 conmon[39718]: ) 2021-07-31T02:03:29.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:29 smithi045 conmon[39718]: 68 2021-07-31T02:03:29.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:29 smithi045 conmon[39718]: : 2021-07-31T02:03:29.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:29 smithi045 conmon[39718]: audit 2021-07-31T02:03:29.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:29 smithi045 conmon[39718]: 2021-07-31T02:03:29.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:29 smithi045 conmon[39718]: [INF] 2021-07-31T02:03:29.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:29 smithi045 conmon[39718]: 2021-07-31T02:03:29.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:29 smithi045 conmon[39718]: from='client.? 172.21.15.45:0/1417384092' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74549-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:03:29.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:29 smithi045 conmon[39718]: 2021-07-31T02:03:29.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:29 smithi045 conmon[39718]: audit 2021-07-31T02:03:29.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:29 smithi045 conmon[39718]: 2021-07-31T02:03:29.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:29 smithi045 conmon[39718]: 2021-07-31T02:03:28.145256+0000 2021-07-31T02:03:29.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:29 smithi045 conmon[39718]: mon.a (mon.0) 1807 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74549-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:03:29.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:29 smithi045 conmon[33655]: cluster 2021-07-31T02:03:29.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:29 smithi045 conmon[33655]: 2021-07-31T02:03:29.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:29 smithi045 conmon[33655]: 2021 2021-07-31T02:03:29.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:29 smithi045 conmon[33655]: - 2021-07-31T02:03:29.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:29 smithi045 conmon[33655]: 07-31T02:03:28.143766+0000 mon.a (mon 2021-07-31T02:03:29.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:29 smithi045 conmon[33655]: .0) 1806 : cluster [DBG] osdmap e999: 8 total, 8 up, 8 in 2021-07-31T02:03:29.543 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:29 smithi045 conmon[33655]: audit 2021-07-31T02:03:29.543 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:29 smithi045 conmon[33655]: 2021-07-31T02:03:29.543 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:29 smithi045 conmon[33655]: 2021-07-31T02: 2021-07-31T02:03:29.543 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:29 smithi045 conmon[33655]: 03:28.144666 2021-07-31T02:03:29.544 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:29 smithi045 conmon[33655]: +0000 mon.c (mon.1) 68 : audit [INF] from='client.? 172.21.15.45:0/1417384092' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74549-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:03:29.544 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:29 smithi045 conmon[33655]: audit 2021-07 2021-07-31T02:03:29.544 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:29 smithi045 conmon[33655]: -31T02:03:28.145256+0000 mon.a (mon.0) 1807 : 2021-07-31T02:03:29.544 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:29 smithi045 conmon[33655]: audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74549-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:03:30.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: debug 2021-07-31T02:03:29.770+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:03:30.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:03:30.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:03:30.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: yield 2021-07-31T02:03:30.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:03:30.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:03:30.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:03:30.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:03:30.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:03:30.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:03:30.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:03:30.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:03:30.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:03:30.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: task.result() 2021-07-31T02:03:30.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:03:30.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: debug 2021-07-31T02:03:29.781+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:03:30.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:03:30.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:03:30.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: yield 2021-07-31T02:03:30.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:03:30.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:03:30.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:03:30.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:03:30.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:03:30.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:03:30.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:03:30.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:03:30.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:03:30.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: task.result() 2021-07-31T02:03:30.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:03:30.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: debug 2021-07-31T02:03:29.807+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:03:30.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:03:30.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:03:30.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: yield 2021-07-31T02:03:30.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:03:30.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:03:30.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:03:30.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:03:30.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:03:30.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:03:30.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:03:30.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:03:30.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:03:30.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: task.result() 2021-07-31T02:03:30.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:03:30.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: debug 2021-07-31T02:03:29.807+0000 7fcb9295f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:03:30.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:03:30.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:03:30.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: yield 2021-07-31T02:03:30.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:03:30.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:03:30.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:03:30.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:03:30.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:03:30.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:03:30.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:03:30.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:03:30.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:03:30.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: task.result() 2021-07-31T02:03:30.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:03:30.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: 2021-07-31T02:03:30.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:03:30.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: 2021-07-31T02:03:30.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:03:30.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:03:30.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:03:30.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:03:30.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:03:30.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:03:30.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:03:30.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:03:30.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:03:30.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:03:30.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:03:30.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:03:30.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:03:30.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:03:30.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:03:30.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:03:30.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:03:30.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:03:30.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: raise 2021-07-31T02:03:30.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:03:30.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:03:30.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:03:30.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:03:30.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:03:30.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: debug 2021-07-31T02:03:29.816+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:03:30.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:03:30.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:03:30.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: yield 2021-07-31T02:03:30.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:03:30.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:03:30.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:03:30.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:03:30.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:03:30.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:03:30.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:03:30.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:03:30.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:03:30.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: task.result() 2021-07-31T02:03:30.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:03:30.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: debug 2021-07-31T02:03:29.816+0000 7fcb93160700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:03:30.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:03:30.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:03:30.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: yield 2021-07-31T02:03:30.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:03:30.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:03:30.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:03:30.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:03:30.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:03:30.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:03:30.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:03:30.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:03:30.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:03:30.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: task.result() 2021-07-31T02:03:30.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:03:30.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: 2021-07-31T02:03:30.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:03:30.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: 2021-07-31T02:03:30.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:03:30.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:03:30.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:03:30.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:03:30.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:03:30.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:03:30.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:03:30.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:03:30.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:03:30.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:03:30.035 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:03:30.035 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:03:30.035 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:03:30.035 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:03:30.035 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:03:30.036 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:03:30.036 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:03:30.036 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:03:30.036 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: raise 2021-07-31T02:03:30.037 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:03:30.037 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:03:30.037 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:03:30.037 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:03:30.037 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:03:29 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:03:30.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: audit 2021-07-31T02:03:29.145665+0000 mon.a (mon.0) 2021-07-31T02:03:30.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: 1808 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74549-3","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:03:30.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: cluster 2021-07-31T02:03:29.146003+0000 mon.a (mon.0) 1809 : cluster [DBG] osdmap e1000: 8 total, 8 up, 8 in 2021-07-31T02:03:30.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: audit 2021-07-31T02:03:29.723021+0000 mon.b (mon.2) 150 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:03:30.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: audit 2021-07-31T02:03:29.724511+0000 mon.b (mon.2) 151 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:03:30.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: audit 2021-07-31T02: 2021-07-31T02:03:30.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: 03:29.725416+0000 mon.b (mon.2) 152 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:03:30.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:03:30.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: 2021-07-31T02 2021-07-31T02:03:30.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: :03:29.726163+0000 mgr.x (mgr.24542) 693 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:03:30.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:03:30.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: 2021-07-31T02:03:29.770954 2021-07-31T02:03:30.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: +0000 mgr.x (mgr.24542) 694 : 2021-07-31T02:03:30.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: cephadm [ERR] 2021-07-31T02:03:30.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:03:30.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:03:30.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: yield 2021-07-31T02:03:30.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:03:30.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:03:30.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:03:30.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:03:30.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:03:30.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:03:30.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:03:30.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:03:30.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:03:30.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:03:30.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:03:30.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:03:29.771555+0000 mgr.x (mgr.24542) 695 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:03:30.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:03 2021-07-31T02:03:30.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: :29.782623+0000 mgr.x (mgr.24542) 696 : cephadm [ERR] 2021-07-31T02:03:30.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:03:30.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:03:30.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: yield 2021-07-31T02:03:30.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:03:30.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:03:30.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:03:30.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:03:30.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:03:30.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:03:30.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:03:30.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:03:30.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:03:30.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:03:30.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:03:30.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:03:29.783580+0000 mgr.x (mgr.24542) 697 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:03:30.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:03:29.808240+0000 mgr.x (mgr.24542) 698 : cephadm [ERR] 2021-07-31T02:03:30.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:03:30.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:03:30.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: yield 2021-07-31T02:03:30.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:03:30.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:03:30.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:03:30.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:03:30.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:03:30.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:03:30.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:03:30.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:03:30.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:03:30.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:03:30.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:03:30.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:03:29.808877+0000 mgr.x (mgr.24542) 699 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:03:30.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:03:30.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:03:30.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: yield 2021-07-31T02:03:30.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:03:30.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:03:30.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:03:30.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:03:30.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:03:30.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:03:30.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:03:30.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:03:30.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:03:30.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:03:30.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:03:30.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: 2021-07-31T02:03:30.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:03:30.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: 2021-07-31T02:03:30.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:03:30.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:03:30.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:03:30.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:03:30.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:03:30.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:03:30.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:03:30.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:03:30.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:03:30.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:03:30.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:03:30.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:03:30.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:03:30.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:03:30.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:03:30.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:03:30.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:03:30.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:03:30.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: raise 2021-07-31T02:03:30.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:03:30.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:03:30.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:03:30.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:03:30.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:03:30.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:03:29.817370+0000 mgr.x (mgr.24542) 700 : cephadm [ERR] 2021-07-31T02:03:30.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:03:30.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:03:30.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: yield 2021-07-31T02:03:30.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:03:30.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:03:30.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:03:30.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:03:30.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:03:30.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:03:30.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:03:30.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:03:30.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:03:30.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:03:30.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:03:30.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:03:29.817727+0000 mgr.x (mgr.24542) 701 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:03:30.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:03:30.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:03:30.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: yield 2021-07-31T02:03:30.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:03:30.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:03:30.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:03:30.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:03:30.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:03:30.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:03:30.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:03:30.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:03:30.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:03:30.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:03:30.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:03:30.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: 2021-07-31T02:03:30.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:03:30.541 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: 2021-07-31T02:03:30.541 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:03:30.541 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:03:30.541 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:03:30.541 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:03:30.542 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:03:30.542 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:03:30.543 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:03:30.543 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:03:30.543 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:03:30.544 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:03:30.544 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:03:30.544 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:03:30.545 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:03:30.545 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:03:30.545 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:03:30.546 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:03:30.546 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:03:30.546 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:03:30.547 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: raise 2021-07-31T02:03:30.547 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:03:30.548 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:03:30.548 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:03:30.549 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:03:30.549 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:03:30.549 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: cluster 2021-07-31T02:03:30.023961+0000 mgr.x (mgr.24542) 702 : cluster [DBG] 2021-07-31T02:03:30.550 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:30 smithi154 conmon[34553]: pgmap v1335: 41 pgs: 32 creating+peering, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:03:30.551 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: audit 2021-07-31T02:03:29.145665+0000 mon.a (mon.0) 1808 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74549-3","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:03:30.551 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: cluster 2021-07-31T02:03:29.146003+0000 mon.a (mon.0) 1809 : cluster [DBG] osdmap e1000: 8 total, 8 up, 8 in 2021-07-31T02:03:30.552 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: audit 2021-07-31T02:03:29.723021+0000 mon.b (mon.2) 150 : audit 2021-07-31T02:03:30.552 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:03:30.553 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: audit 2021-07-31T 2021-07-31T02:03:30.553 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: 02:03:29.724511+0000 mon.b (mon. 2021-07-31T02:03:30.553 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: 2) 151 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:03:30.553 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: audit 2021-07-31T02:03:30.554 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: 2021-07-31T02:03:29.725416+0000 2021-07-31T02:03:30.554 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: mon.b (mon.2) 152 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:03:30.554 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:03:30.554 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: 2021-07-31T02:03:29.726163+0000 2021-07-31T02:03:30.555 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: mgr.x (mgr.24542) 693 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:03:30.555 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: cephadm 2021-07-31T02 2021-07-31T02:03:30.555 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: :03:29.770954+0000 mgr.x (mgr.24542) 694 : cephadm [ERR] 2021-07-31T02:03:30.555 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:03:30.555 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:03:30.556 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: yield 2021-07-31T02:03:30.556 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:03:30.556 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:03:30.556 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:03:30.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:03:30.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:03:30.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:03:30.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:03:30.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:03:30.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:03:30.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:03:30.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:03:30.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:03:29.771555+0000 mgr.x (mgr.24542) 695 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:03:30.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:03:29.782623+0000 mgr.x (mgr.24542) 696 : cephadm [ERR] 2021-07-31T02:03:30.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:03:30.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:03:30.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: yield 2021-07-31T02:03:30.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:03:30.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: audit 2021-07-31T02:03:29.145665+0000 mon.a (mon. 2021-07-31T02:03:30.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: 0) 1808 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74549-3","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:03:30.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: cluster 2021-07-31T02:03:29.146003 2021-07-31T02:03:30.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: +0000 mon.a (mon.0) 1809 : cluster [DBG] osdmap e1000: 8 total, 8 up, 8 in 2021-07-31T02:03:30.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T02:03:30.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: T02:03:29.723021+0000 mon.b (mon.2) 150 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:03:30.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: audit 2021-07-31T02:03:29.724511+0000 mon.b ( 2021-07-31T02:03:30.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: mon.2) 151 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:03:30.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: audit 2021-07-31T02:03:29.725416+0000 mon.b (mon. 2021-07-31T02:03:30.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: 2) 152 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:03:30.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: cephadm 2021-07 2021-07-31T02:03:30.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: -31T02:03:29.726163+0000 mgr.x (mgr. 2021-07-31T02:03:30.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: 24542) 693 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:03:30.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: cephadm 2021-07-31T02 2021-07-31T02:03:30.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: :03:29.770954+0000 mgr.x (mgr.24542) 694 : 2021-07-31T02:03:30.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: cephadm [ERR] 2021-07-31T02:03:30.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:03:30.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:03:30.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: yield 2021-07-31T02:03:30.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:03:30.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:03:30.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:03:30.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:03:30.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:03:30.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:03:30.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:03:30.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:03:30.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:03:30.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:03:30.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:03:30.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:03:29.771555+0000 mgr.x (mgr.24542) 695 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:03:30.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:03:30.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: 2021-07-31T02:03:29.782623+0000 mgr.x (mgr.24542 2021-07-31T02:03:30.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: ) 696 : cephadm [ERR] 2021-07-31T02:03:30.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:03:30.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:03:30.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: yield 2021-07-31T02:03:30.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:03:30.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:03:30.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:03:30.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:03:30.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:03:30.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:03:30.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:03:30.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:03:30.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:03:30.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:03:30.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:03:30.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:03:29.783580+0000 mgr.x (mgr.24542) 697 : cephadm [INF] 2021-07-31T02:03:30.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:03:30.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:03:29.808240+0000 mgr.x (mgr.24542) 698 : cephadm [ERR] 2021-07-31T02:03:30.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:03:30.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:03:30.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: yield 2021-07-31T02:03:30.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:03:30.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:03:30.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:03:30.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:03:30.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:03:30.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:03:30.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:03:30.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:03:30.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:03:30.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:03:30.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:03:30.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:03:29.808877+0000 mgr.x (mgr.24542) 699 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:03:30.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:03:30.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:03:30.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: yield 2021-07-31T02:03:30.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:03:30.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:03:30.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:03:30.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:03:30.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:03:30.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:03:30.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:03:30.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:03:30.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:03:30.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:03:30.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:03:30.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:03:30.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:03:30.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:03:30.585 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:03:30.585 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:03:30.585 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:03:30.586 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:03:30.586 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:03:30.587 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:03:29.783580+0000 mgr.x (mgr.24542) 697 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:03:30.587 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:03:29.808240+0000 mgr.x (mgr.24542) 698 : cephadm [ERR] 2021-07-31T02:03:30.587 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:03:30.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:03:30.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: yield 2021-07-31T02:03:30.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:03:30.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:03:30.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:03:30.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:03:30.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:03:30.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:03:30.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:03:30.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:03:30.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:03:30.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:03:30.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:03:30.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:03:29.808877+0000 mgr.x (mgr.24542) 699 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:03:30.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:03:30.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:03:30.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: yield 2021-07-31T02:03:30.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:03:30.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:03:30.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:03:30.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:03:30.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:03:30.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:03:30.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:03:30.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:03:30.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:03:30.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:03:30.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: 2021-07-31T02:03:30.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:03:30.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: 2021-07-31T02:03:30.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:03:30.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:03:30.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:03:30.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:03:30.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:03:30.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:03:30.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:03:30.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:03:30.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:03:30.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:03:30.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:03:30.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:03:30.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:03:30.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:03:30.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:03:30.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:03:30.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:03:30.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:03:30.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: raise 2021-07-31T02:03:30.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:03:30.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:03:30.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:03:30.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:03:30.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:03:30.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:03:29.817370+0000 mgr.x (mgr.24542) 700 : cephadm [ERR] 2021-07-31T02:03:30.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:03:30.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:03:30.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: yield 2021-07-31T02:03:30.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:03:30.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:03:30.608 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:03:30.608 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:03:30.608 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:03:30.608 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:03:30.609 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:03:30.609 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:03:30.609 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:03:30.610 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:03:30.610 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:03:30.610 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: 2021-07-31T02:03:30.610 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:03:30.611 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: 2021-07-31T02:03:30.611 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:03:30.611 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:03:30.611 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:03:30.612 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:03:30.612 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:03:30.612 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:03:30.612 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:03:30.612 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:03:30.613 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:03:30.613 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:03:30.613 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:03:30.613 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:03:30.614 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:03:30.614 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:03:30.614 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:03:30.614 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:03:30.615 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:03:30.615 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:03:30.615 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: raise 2021-07-31T02:03:30.615 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:03:30.616 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:03:30.616 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:03:30.616 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:03:30.617 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:03:30.617 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:03:29.817370+0000 mgr.x (mgr.24542) 700 : cephadm [ERR] 2021-07-31T02:03:30.618 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:03:30.618 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:03:30.618 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: yield 2021-07-31T02:03:30.619 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:03:30.619 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:03:30.619 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:03:30.620 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:03:30.620 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:03:30.621 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:03:30.621 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:03:30.622 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:03:30.623 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:03:30.623 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:03:30.624 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:03:29.817727+0000 mgr.x (mgr.24542) 701 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:03:30.624 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:03:30.624 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:03:30.625 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: yield 2021-07-31T02:03:30.625 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:03:30.626 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:03:30.626 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:03:30.627 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:03:30.627 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:03:30.627 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:03:30.628 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:03:30.628 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:03:30.628 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:03:30.629 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:03:30.629 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:03:30.630 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: 2021-07-31T02:03:30.630 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:03:30.630 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: 2021-07-31T02:03:30.631 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:03:30.631 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:03:30.632 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:03:30.632 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:03:30.632 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:03:30.633 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:03:30.633 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:03:30.633 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:03:30.634 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:03:30.634 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:03:30.635 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:03:30.635 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:03:30.635 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:03:30.636 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:03:30.636 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:03:30.636 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:03:30.637 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:03:30.637 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:03:30.637 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: raise 2021-07-31T02:03:30.638 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:03:30.638 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:03:30.639 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:03:30.639 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:03:30.639 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:03:30.640 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[33655]: cluster 2021-07-31T02:03:30.023961+0000 mgr.x (mgr.24542) 702 : cluster [DBG] pgmap v1335: 41 pgs: 32 creating+peering, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:03:30.641 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:03:30.641 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:03:30.641 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:03:30.642 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:03:30.642 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:03 2021-07-31T02:03:30.642 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: :29.817727+0000 mgr.x (mgr.24542) 701 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:03:30.643 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:03:30.643 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:03:30.643 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: yield 2021-07-31T02:03:30.644 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:03:30.644 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:03:30.644 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:03:30.644 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:03:30.645 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:03:30.645 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:03:30.645 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:03:30.645 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:03:30.645 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:03:30.646 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:03:30.646 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:03:30.646 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: 2021-07-31T02:03:30.646 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:03:30.647 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: 2021-07-31T02:03:30.647 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:03:30.647 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:03:30.647 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:03:30.648 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:03:30.648 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:03:30.648 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:03:30.648 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:03:30.648 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:03:30.649 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:03:30.649 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:03:30.649 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:03:30.649 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:03:30.649 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:03:30.650 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:03:30.650 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:03:30.650 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:03:30.650 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:03:30.651 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:03:30.651 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: raise 2021-07-31T02:03:30.651 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:03:30.651 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:03:30.651 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:03:30.652 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:03:30.652 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:03:30.652 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:30 smithi045 conmon[39718]: cluster 2021-07-31T02:03:30.023961+0000 mgr.x (mgr.24542) 702 : cluster [DBG] pgmap v1335: 41 pgs: 32 creating+peering, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:03:31.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:31 smithi154 conmon[34553]: cluster 2021-07-31T02:03:30.160319+0000 mon.a (mon.0) 1810 : cluster [DBG] osdmap e1001: 8 total, 8 up, 8 in 2021-07-31T02:03:31.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:31 smithi045 conmon[33655]: cluster 2021-07-31T02:03:30.160319 2021-07-31T02:03:31.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:31 smithi045 conmon[33655]: +0000 mon.a (mon.0) 1810 : cluster [DBG] osdmap e1001: 8 total, 8 up, 8 in 2021-07-31T02:03:31.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:31 smithi045 conmon[39718]: cluster 2021-07-31T02:03:30.160319+0000 mon.a 2021-07-31T02:03:31.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:31 smithi045 conmon[39718]: (mon.0) 1810 : cluster [DBG] osdmap e1001: 8 total, 8 up, 8 in 2021-07-31T02:03:33.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:32 smithi154 conmon[34553]: cluster 2021-07-31T02:03:32.024424+0000 mgr.x (mgr. 2021-07-31T02:03:33.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:32 smithi154 conmon[34553]: 24542) 703 : cluster [DBG] pgmap v1337: 41 pgs: 32 creating+peering, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:03:33.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:32 smithi045 conmon[39718]: cluster 2021-07-31T02:03:32.024424+0000 mgr.x (mgr 2021-07-31T02:03:33.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:32 smithi045 conmon[39718]: .24542) 703 : cluster [DBG] pgmap v1337: 41 pgs: 32 creating+peering, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:03:33.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:32 smithi045 conmon[33655]: cluster 2021-07-31T02:03:32.024424+0000 mgr.x (mgr.24542) 703 : 2021-07-31T02:03:33.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:32 smithi045 conmon[33655]: cluster [DBG] pgmap v1337: 41 pgs: 32 creating+peering, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:03:33.873 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:03:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:03:33.536Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:03:33.874 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:03:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:03:33.536Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:03:33.874 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:03:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:03:33.536Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:03:34.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:33 smithi154 conmon[34553]: cluster 2021-07-31T02:03:32.868549+0000 mon.a (mon.0) 1811 : cluster [DBG] 2021-07-31T02:03:34.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:33 smithi154 conmon[34553]: osdmap e1002: 8 total, 8 up, 8 in 2021-07-31T02:03:34.254 INFO:journalctl@ceph.osd.5.smithi154.stdout:Jul 31 02:03:33 smithi154 conmon[42050]: debug 2021-07-31T02:03:34.254 INFO:journalctl@ceph.osd.5.smithi154.stdout:Jul 31 02:03:33 smithi154 conmon[42050]: 2021-07-31T02:03:33.994+0000 7fc9b005a700 -1 osd.5 pg_epoch: 1003 pg[56.6( v 1002'4 (0'0,1002'4] local-lis/les=999/1000 n=2 ec=999/999 lis/c=999/999 les/c/f=1000/1000/0 sis=999) [5,7,3] r=0 lpr=999 crt=1001'3 lcod 1001'3 mlcod 1001'3 active+clean+snaptrim trimq=[1~1]] removing snap head 2021-07-31T02:03:34.254 INFO:journalctl@ceph.osd.5.smithi154.stdout:Jul 31 02:03:33 smithi154 conmon[42050]: 2021-07-31T02:03:34.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:33 smithi045 conmon[39718]: cluster 2021-07-31T02:03:32. 2021-07-31T02:03:34.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:33 smithi045 conmon[39718]: 868549+0000 mon.a (mon.0) 1811 : cluster [DBG] osdmap e1002: 8 total, 8 up, 8 in 2021-07-31T02:03:34.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:33 smithi045 conmon[33655]: cluster 2021-07-31T02:03:34.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:33 smithi045 conmon[33655]: 2021-07-31T02:03:34.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:33 smithi045 conmon[33655]: 2021 2021-07-31T02:03:34.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:33 smithi045 conmon[33655]: - 2021-07-31T02:03:34.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:33 smithi045 conmon[33655]: 07 2021-07-31T02:03:34.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:33 smithi045 conmon[33655]: - 2021-07-31T02:03:34.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:33 smithi045 conmon[33655]: 31 2021-07-31T02:03:34.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:33 smithi045 conmon[33655]: T 2021-07-31T02:03:34.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:33 smithi045 conmon[33655]: 02 2021-07-31T02:03:34.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:33 smithi045 conmon[33655]: :03:32.868549+0000 mon.a (mon. 2021-07-31T02:03:34.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:33 smithi045 conmon[33655]: 0) 1811 : cluster [DBG] osdmap e1002: 8 total, 8 up, 8 in 2021-07-31T02:03:35.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:34 smithi154 conmon[34553]: cluster 2021-07-31T02:03:33.870856+0000 mon.a (mon 2021-07-31T02:03:35.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:34 smithi154 conmon[34553]: .0) 1812 : cluster [DBG] osdmap e1003: 8 total, 8 up, 8 in 2021-07-31T02:03:35.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:34 smithi154 conmon[34553]: cluster 2021-07-31T02:03:34.025163+0000 mgr.x (mgr.24542) 704 : cluster [DBG] pgmap v1340: 41 pgs: 41 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 209 B/s wr, 0 op/s 2021-07-31T02:03:35.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:34 smithi045 conmon[33655]: cluster 2021-07-31T02:03:33.870856+0000 mon.a (mon.0) 1812 : 2021-07-31T02:03:35.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:34 smithi045 conmon[33655]: cluster [DBG] osdmap e1003: 8 total, 8 up, 8 in 2021-07-31T02:03:35.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:34 smithi045 conmon[33655]: cluster 2021-07-31T02:03:34.025163+0000 mgr.x (mgr.24542) 704 : cluster [DBG] pgmap v1340: 41 pgs: 41 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 209 B/s wr, 0 op/s 2021-07-31T02:03:35.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:34 smithi045 conmon[39718]: cluster 2021-07-31T02:03:33.870856+0000 2021-07-31T02:03:35.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:34 smithi045 conmon[39718]: mon.a (mon.0) 1812 : cluster [DBG] osdmap e1003: 8 total, 8 up, 8 in 2021-07-31T02:03:35.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:34 smithi045 conmon[39718]: cluster 2021-07-31T02:03 2021-07-31T02:03:35.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:34 smithi045 conmon[39718]: :34.025163+0000 mgr.x (mgr.24542) 704 : cluster [DBG] pgmap v1340: 41 pgs: 41 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 209 B/s wr, 0 op/s 2021-07-31T02:03:35.889 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] cls_rgw.test_put_snap (8757 ms) 2021-07-31T02:03:35.890 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] cls_rgw.test_explicit 2021-07-31T02:03:36.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:35 smithi154 conmon[34553]: cluster 2021-07-31T02:03:34.881326+0000 mon.a (mon.0) 1813 : cluster [DBG] osdmap e1004: 8 total, 8 up, 8 in 2021-07-31T02:03:36.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:35 smithi154 conmon[34553]: 2021-07-31T02:03:36.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:35 smithi045 conmon[33655]: cluster 2021-07-31T02:03:34.881326+0000 2021-07-31T02:03:36.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:35 smithi045 conmon[33655]: mon.a (mon.0) 1813 : cluster [DBG] osdmap e1004: 8 total, 8 up, 8 in 2021-07-31T02:03:36.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:35 smithi045 conmon[39718]: cluster 2021-07-31T02:03:34.881326+0000 mon.a ( 2021-07-31T02:03:36.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:35 smithi045 conmon[39718]: mon.0) 1813 : cluster [DBG] osdmap e1004: 8 total, 8 up, 8 in 2021-07-31T02:03:37.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:36 smithi154 conmon[34553]: cluster 2021-07-31T02:03:35.887255+0000 mon.a (mon.0) 1814 : cluster [DBG] osdmap e1005: 8 total, 8 up, 8 in 2021-07-31T02:03:37.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:36 smithi154 conmon[34553]: 2021-07-31T02:03:37.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:36 smithi154 conmon[34553]: cluster 2021-07-31T02:03:36.025530+0000 mgr.x (mgr.24542) 705 : cluster [DBG] pgmap v1343: 9 pgs: 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:03:37.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:36 smithi045 conmon[39718]: cluster 2021-07-31T02:03:35.887255+0000 mon.a (mon.0 2021-07-31T02:03:37.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:36 smithi045 conmon[39718]: ) 1814 : cluster [DBG] osdmap e1005: 8 total, 8 up, 8 in 2021-07-31T02:03:37.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:36 smithi045 conmon[39718]: cluster 2021-07-31T02:03:36.025530+0000 mgr.x (mgr 2021-07-31T02:03:37.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:36 smithi045 conmon[39718]: .24542) 705 : cluster [DBG] pgmap v1343: 9 pgs: 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:03:37.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:36 smithi045 conmon[33655]: cluster 2021-07-31T02:03:35.887255+0000 mon.a (mon 2021-07-31T02:03:37.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:36 smithi045 conmon[33655]: .0) 1814 : cluster [DBG] osdmap e1005: 8 total, 8 up, 8 in 2021-07-31T02:03:37.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:36 smithi045 conmon[33655]: cluster 2021-07-31T02:03:36.025530+0000 mgr.x (mgr.24542) 705 : cluster [DBG] 2021-07-31T02:03:37.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:36 smithi045 conmon[33655]: pgmap v1343: 9 pgs: 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:03:38.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:37 smithi154 conmon[34553]: cluster 2021-07-31T02:03:36.891870+0000 mon.a (mon.0) 1815 : cluster [DBG] 2021-07-31T02:03:38.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:37 smithi154 conmon[34553]: osdmap e1006: 8 total, 8 up, 8 in 2021-07-31T02:03:38.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:37 smithi154 conmon[34553]: audit 2021-07-31T02:03:36.893425+0000 mon.c (mon.1) 69 : audit [INF] from='client.? 172.21.15.45:0/3955079603' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74549-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:03:38.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:37 smithi154 conmon[34553]: audit 2021-07-31T02:03: 2021-07-31T02:03:38.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:37 smithi154 conmon[34553]: 36.894189+0000 mon.a (mon.0) 1816 : audit [INF] 2021-07-31T02:03:38.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:37 smithi154 conmon[34553]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74549-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:03:38.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:37 smithi045 conmon[33655]: cluster 2021-07-31T02:03:36.891870+0000 mon.a (mon.0) 1815 : cluster [DBG] 2021-07-31T02:03:38.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:37 smithi045 conmon[33655]: osdmap e1006: 8 total, 8 up, 8 in 2021-07-31T02:03:38.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:37 smithi045 conmon[33655]: audit 2021-07-31T02:03:36.893425+0000 mon.c (mon.1) 69 : audit [INF] from='client.? 172.21.15.45:0/3955079603' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74549-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:03:38.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:37 smithi045 conmon[33655]: audit 2021-07-31T02:03:38.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:37 smithi045 conmon[33655]: 2021-07-31T02:03:36.894189+0000 mon.a (mon.0) 1816 : 2021-07-31T02:03:38.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:37 smithi045 conmon[33655]: audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74549-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:03:38.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:37 smithi045 conmon[39718]: cluster 2021-07-31T02:03:36.891870+0000 mon.a 2021-07-31T02:03:38.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:37 smithi045 conmon[39718]: (mon.0) 1815 : cluster [DBG] osdmap e1006: 8 total, 8 up, 8 in 2021-07-31T02:03:38.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:37 smithi045 conmon[39718]: audit 2021-07 2021-07-31T02:03:38.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:37 smithi045 conmon[39718]: -31T02:03:36.893425+0000 mon.c (mon.1) 2021-07-31T02:03:38.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:37 smithi045 conmon[39718]: 69 : audit [INF] from='client.? 172.21.15.45:0/3955079603' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74549-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:03:38.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:37 smithi045 conmon[39718]: audit 2021- 2021-07-31T02:03:38.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:37 smithi045 conmon[39718]: 07-31T02:03:36.894189+0000 2021-07-31T02:03:38.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:37 smithi045 conmon[39718]: mon.a (mon.0) 1816 : audit [INF] 2021-07-31T02:03:38.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:37 smithi045 conmon[39718]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74549-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:03:38.908 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] cls_rgw.test_explicit (3019 ms) 2021-07-31T02:03:38.909 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] cls_rgw.set 2021-07-31T02:03:39.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:38 smithi154 conmon[34553]: audit 2021-07-31T02:03:37.897937+0000 mon.a (mon.0) 1817 : audit 2021-07-31T02:03:39.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:38 smithi154 conmon[34553]: [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74549-4","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:03:39.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:38 smithi154 conmon[34553]: cluster 2021-07-31T02:03:37.898256+0000 mon.a (mon.0) 1818 : cluster [DBG] osdmap e1007: 8 total, 8 up, 8 in 2021-07-31T02:03:39.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:38 smithi154 conmon[34553]: cluster 2021-07-31T02:03:39.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:38 smithi154 conmon[34553]: 2021-07-31T02:03:38.026052+0000 mgr.x (mgr.24542) 706 : cluster [DBG] pgmap v1346: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:03:39.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:38 smithi045 conmon[33655]: audit 2021-07-31T02:03:37.897937+0000 mon.a (mon.0) 1817 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74549-4","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:03:39.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:38 smithi045 conmon[33655]: 2021-07-31T02:03:39.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:38 smithi045 conmon[33655]: cluster 2021-07-31T02:03:37.898256+0000 mon.a (mon.0) 1818 : cluster [DBG] osdmap e1007: 8 total, 8 up, 8 in 2021-07-31T02:03:39.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:38 smithi045 conmon[33655]: cluster 2021-07-31 2021-07-31T02:03:39.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:38 smithi045 conmon[33655]: T02:03:38.026052+0000 mgr.x (mgr.24542) 706 : cluster [DBG] pgmap v1346: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:03:39.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:38 smithi045 conmon[39718]: audit 2021-07-31T02:03:37.897937+0000 mon.a (mon.0) 2021-07-31T02:03:39.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:38 smithi045 conmon[39718]: 1817 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74549-4","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:03:39.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:38 smithi045 conmon[39718]: cluster 2021-07-31T02:03:37.898256+0000 mon.a (mon.0) 1818 : cluster [DBG] 2021-07-31T02:03:39.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:38 smithi045 conmon[39718]: osdmap e1007: 8 total, 8 up, 8 in 2021-07-31T02:03:39.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:38 smithi045 conmon[39718]: cluster 2021-07-31T02:03:39.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:38 smithi045 conmon[39718]: 2021-07-31T02:03:38.026052 2021-07-31T02:03:39.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:38 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 706 : cluster [DBG] 2021-07-31T02:03:39.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:38 smithi045 conmon[39718]: pgmap v1346: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:03:40.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:39 smithi154 conmon[34553]: cluster 2021-07-31T02:03:38.904358+0000 mon.a (mon.0) 1819 : cluster [DBG] osdmap e1008: 8 total, 8 up, 8 in 2021-07-31T02:03:40.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:39 smithi045 conmon[39718]: cluster 2021-07-31T02:03:38.904358+0000 mon.a 2021-07-31T02:03:40.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:39 smithi045 conmon[39718]: (mon.0) 1819 : cluster [DBG] osdmap e1008: 8 total, 8 up, 8 in 2021-07-31T02:03:40.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:39 smithi045 conmon[33655]: cluster 2021-07-31T02:03:38.904358+0000 mon.a (mon. 2021-07-31T02:03:40.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:39 smithi045 conmon[33655]: 0) 1819 : cluster [DBG] osdmap e1008: 8 total, 8 up, 8 in 2021-07-31T02:03:41.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:40 smithi154 conmon[34553]: cluster 2021-07-31T02:03:39.909562+0000 mon.a (mon.0) 1820 : cluster 2021-07-31T02:03:41.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:40 smithi154 conmon[34553]: [DBG] osdmap e1009: 8 total, 8 up, 8 in 2021-07-31T02:03:41.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:40 smithi154 conmon[34553]: audit 2021-07-31T02:03:39.911997+0000 mon.a (mon.0) 1821 : audit [INF] from='client.? 172.21.15.45:0/3727777619' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74549-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:03:41.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:40 smithi154 conmon[34553]: cluster 2021-07-31 2021-07-31T02:03:41.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:40 smithi154 conmon[34553]: T02:03:40.026506+0000 mgr.x (mgr.24542) 2021-07-31T02:03:41.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:40 smithi154 conmon[34553]: 707 : cluster [DBG] pgmap v1349: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:03:41.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:40 smithi045 conmon[33655]: cluster 2021-07-31T02:03:39.909562+0000 mon.a (mon.0) 2021-07-31T02:03:41.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:40 smithi045 conmon[33655]: 1820 : cluster [DBG] osdmap e1009: 8 total, 8 up, 8 in 2021-07-31T02:03:41.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:40 smithi045 conmon[33655]: audit 2021-07-31T02:03:39.911997+0000 mon.a (mon.0) 1821 : audit [INF] from='client.? 172.21.15.45:0/3727777619' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74549-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:03:41.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:40 smithi045 conmon[33655]: cluster 2021-07-31T02:03:40.026506+0000 mgr.x (mgr.24542) 707 : cluster [DBG] 2021-07-31T02:03:41.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:40 smithi045 conmon[33655]: pgmap v1349: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:03:41.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:40 smithi045 conmon[39718]: cluster 2021-07-31T02:03:41.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:40 smithi045 conmon[39718]: 2021-07-31T02:03 2021-07-31T02:03:41.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:40 smithi045 conmon[39718]: :39.909562+0000 mon.a (mon.0) 1820 : cluster [DBG] osdmap e1009: 8 total, 8 up, 8 in 2021-07-31T02:03:41.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:40 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T02:03:41.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:40 smithi045 conmon[39718]: 31T02:03:39.911997+0000 mon.a (mon.0) 1821 : audit [INF] 2021-07-31T02:03:41.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:40 smithi045 conmon[39718]: from='client.? 172.21.15.45:0/3727777619' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74549-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:03:41.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:40 smithi045 conmon[39718]: cluster 2021-07-31T02:03:41.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:40 smithi045 conmon[39718]: 2021-07 2021-07-31T02:03:41.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:40 smithi045 conmon[39718]: -31T02:03:40 2021-07-31T02:03:41.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:40 smithi045 conmon[39718]: .026506+0000 mgr.x (mgr.24542 2021-07-31T02:03:41.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:40 smithi045 conmon[39718]: ) 707 : cluster [DBG] pgmap v1349: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:03:41.924 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] cls_rgw.set (3016 ms) 2021-07-31T02:03:41.924 INFO:tasks.workunit.client.0.smithi045.stdout:[----------] 5 tests from cls_rgw (20676 ms total) 2021-07-31T02:03:41.924 INFO:tasks.workunit.client.0.smithi045.stdout: 2021-07-31T02:03:41.925 INFO:tasks.workunit.client.0.smithi045.stdout:[----------] Global test environment tear-down 2021-07-31T02:03:41.925 INFO:tasks.workunit.client.0.smithi045.stdout:[==========] 5 tests from 1 test suite ran. (20676 ms total) 2021-07-31T02:03:41.925 INFO:tasks.workunit.client.0.smithi045.stdout:[ PASSED ] 5 tests. 2021-07-31T02:03:41.926 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-07-31T02:03:41.927 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2021-07-31T02:03:41.957 INFO:tasks.workunit:Running workunit cls/test_cls_rgw.sh... 2021-07-31T02:03:41.957 DEBUG:teuthology.orchestra.run.smithi045:workunit test cls/test_cls_rgw.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=pacific TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rgw.sh 2021-07-31T02:03:42.030 INFO:tasks.workunit.client.0.smithi045.stdout:Running main() from gmock_main.cc 2021-07-31T02:03:42.031 INFO:tasks.workunit.client.0.smithi045.stdout:[==========] Running 14 tests from 1 test suite. 2021-07-31T02:03:42.031 INFO:tasks.workunit.client.0.smithi045.stdout:[----------] Global test environment set-up. 2021-07-31T02:03:42.031 INFO:tasks.workunit.client.0.smithi045.stdout:[----------] 14 tests from cls_rgw 2021-07-31T02:03:42.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:41 smithi154 conmon[34553]: audit 2021-07-31T02:03:40.916443+0000 mon.a (mon.0) 1822 : audit 2021-07-31T02:03:42.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:41 smithi154 conmon[34553]: [INF] from='client.? 172.21.15.45:0/3727777619' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74549-5","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:03:42.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:41 smithi154 conmon[34553]: cluster 2021-07-31T02:03:40.916731+0000 mon.a (mon.0) 1823 : cluster [DBG] osdmap e1010: 8 total, 8 up, 8 in 2021-07-31T02:03:42.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:41 smithi045 conmon[33655]: audit 2021-07-31T02:03:40.916443+0000 mon.a (mon.0) 1822 : audit 2021-07-31T02:03:42.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:41 smithi045 conmon[33655]: [INF] from='client.? 172.21.15.45:0/3727777619' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74549-5","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:03:42.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:41 smithi045 conmon[33655]: cluster 2021-07-31T02:03:40.916731+0000 mon.a (mon.0) 1823 : cluster [DBG] osdmap e1010: 8 total, 8 up, 8 in 2021-07-31T02:03:42.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:41 smithi045 conmon[39718]: audit 2021-07-31T02:03:40.916443+0000 2021-07-31T02:03:42.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:41 smithi045 conmon[39718]: mon.a (mon.0) 1822 : audit [INF] from='client.? 172.21.15.45:0/3727777619' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74549-5","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:03:42.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:41 smithi045 conmon[39718]: cluster 2021-07-31T 2021-07-31T02:03:42.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:41 smithi045 conmon[39718]: 02:03:40.916731+0000 mon.a (mon.0) 2021-07-31T02:03:42.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:41 smithi045 conmon[39718]: 1823 : cluster [DBG] osdmap e1010: 8 total, 8 up, 8 in 2021-07-31T02:03:43.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:42 smithi154 conmon[34553]: cluster 2021-07-31T02:03:41.921647+0000 mon.a (mon.0) 1824 : cluster [DBG] 2021-07-31T02:03:43.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:42 smithi154 conmon[34553]: osdmap e1011: 8 total, 8 up, 8 in 2021-07-31T02:03:43.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:42 smithi154 conmon[34553]: cluster 2021-07-31T02:03:42.026878+0000 mgr.x (mgr.24542) 708 : cluster [DBG] pgmap v1352: 9 pgs: 9 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:03:43.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:42 smithi045 conmon[33655]: cluster 2021-07-31T02:03:41.921647+0000 mon.a (mon.0 2021-07-31T02:03:43.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:42 smithi045 conmon[33655]: ) 1824 : cluster [DBG] osdmap e1011: 8 total, 8 up, 8 in 2021-07-31T02:03:43.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:42 smithi045 conmon[33655]: cluster 2021-07-31T02:03:42.026878+0000 mgr.x (mgr.24542) 708 : cluster [DBG] pgmap v1352: 9 pgs: 9 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:03:43.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:42 smithi045 conmon[39718]: cluster 2021-07-31T02:03:41.921647+0000 mon.a (mon.0 2021-07-31T02:03:43.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:42 smithi045 conmon[39718]: ) 1824 : cluster [DBG] osdmap e1011: 8 total, 8 up, 8 in 2021-07-31T02:03:43.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:42 smithi045 conmon[39718]: cluster 2021-07-31T02:03:42.026878+0000 mgr.x (mgr.24542) 708 : cluster [DBG] pgmap v1352: 9 pgs: 9 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:03:43.926 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:03:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:03:43.536Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:03:43.927 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:03:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:03:43.536Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:03:43.927 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:03:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:03:43.536Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:03:43.934 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] cls_rgw.index_basic 2021-07-31T02:03:43.981 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] cls_rgw.index_basic (47 ms) 2021-07-31T02:03:43.981 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] cls_rgw.index_multiple_obj_writers 2021-07-31T02:03:44.009 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] cls_rgw.index_multiple_obj_writers (28 ms) 2021-07-31T02:03:44.009 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] cls_rgw.index_remove_object 2021-07-31T02:03:44.047 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] cls_rgw.index_remove_object (38 ms) 2021-07-31T02:03:44.048 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] cls_rgw.index_suggest 2021-07-31T02:03:44.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:43 smithi154 conmon[34553]: cluster 2021-07-31T02:03:42.926505+0000 mon.a (mon.0) 1825 : cluster [DBG] osdmap e1012: 8 total, 8 up, 8 in 2021-07-31T02:03:44.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:43 smithi154 conmon[34553]: audit 2021-07-31T02:03:42.928040+0000 mon.c (mon.1) 70 : audit [INF] from='client.? 172.21.15.45:0/4283038488' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74746-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:03:44.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:43 smithi154 conmon[34553]: audit 2021-07-31T02:03:42.928445+0000 mon.a (mon.0) 1826 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74746-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:03:44.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:43 smithi045 conmon[39718]: cluster 2021-07-31T02:03:42.926505+0000 mon.a 2021-07-31T02:03:44.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:43 smithi045 conmon[39718]: (mon.0) 1825 : cluster [DBG] osdmap e1012: 8 total, 8 up, 8 in 2021-07-31T02:03:44.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:43 smithi045 conmon[39718]: audit 2021-07-31T02 2021-07-31T02:03:44.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:43 smithi045 conmon[39718]: :03:42.928040+0000 mon.c (mon.1) 70 : audit [INF] 2021-07-31T02:03:44.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:43 smithi045 conmon[39718]: from='client.? 172.21.15.45:0/4283038488' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74746-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:03:44.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:43 smithi045 conmon[39718]: audit 2021-07-31T02:03: 2021-07-31T02:03:44.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:43 smithi045 conmon[39718]: 42.928445+0000 mon.a (mon.0) 2021-07-31T02:03:44.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:43 smithi045 conmon[39718]: 1826 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74746-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:03:44.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:43 smithi045 conmon[33655]: cluster 2021-07-31T02:03:42.926505+0000 mon.a (mon 2021-07-31T02:03:44.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:43 smithi045 conmon[33655]: .0) 1825 : cluster [DBG] osdmap e1012: 8 total, 8 up, 8 in 2021-07-31T02:03:44.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:43 smithi045 conmon[33655]: audit 2021-07-31T02:03:42.928040+0000 mon.c (mon.1) 70 : audit 2021-07-31T02:03:44.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:43 smithi045 conmon[33655]: [INF] from='client.? 172.21.15.45:0/4283038488' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74746-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:03:44.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:43 smithi045 conmon[33655]: audit 2021-07-31T02:03:42 2021-07-31T02:03:44.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:43 smithi045 conmon[33655]: .928445+0000 mon.a (mon.0) 1826 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74746-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:03:44.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:43 smithi045 conmon[33655]: 2021-07-31T02:03:45.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:44 smithi154 conmon[34553]: audit 2021-07-31T02:03:43.933108+0000 2021-07-31T02:03:45.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:44 smithi154 conmon[34553]: mon.a (mon.0) 1827 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74746-1","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:03:45.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:44 smithi154 conmon[34553]: cluster 2021-07-31T02:03:43.933345+0000 mon.a (mon.0) 1828 : cluster [DBG] osdmap e1013: 8 total, 8 up, 8 in 2021-07-31T02:03:45.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:44 smithi154 conmon[34553]: cluster 2021-07-31T02:03:44.027227+0000 mgr.x (mgr.24542) 709 : cluster [DBG] pgmap v1355: 41 pgs: 32 creating+peering, 9 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:03:45.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:44 smithi154 conmon[34553]: 2021-07-31T02:03:45.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:44 smithi045 conmon[39718]: audit 2021-07-31T02:03:43.933108+0000 mon.a (mon.0) 1827 2021-07-31T02:03:45.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:44 smithi045 conmon[39718]: : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74746-1","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:03:45.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:44 smithi045 conmon[39718]: cluster 2021-07-31T02:03:43.933345+0000 mon.a (mon.0 2021-07-31T02:03:45.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:44 smithi045 conmon[39718]: ) 1828 : cluster [DBG] osdmap e1013: 8 total, 8 up, 8 in 2021-07-31T02:03:45.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:44 smithi045 conmon[39718]: cluster 2021-07-31T02:03:45.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:44 smithi045 conmon[39718]: 2021-07-31T02:03:44.027227+0000 mgr.x (mgr.24542) 2021-07-31T02:03:45.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:44 smithi045 conmon[39718]: 709 : cluster [DBG] pgmap v1355: 41 pgs: 32 creating+peering, 9 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:03:45.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:44 smithi045 conmon[33655]: audit 2021-07-31T02:03:43.933108+0000 mon.a (mon.0) 1827 : 2021-07-31T02:03:45.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:44 smithi045 conmon[33655]: audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74746-1","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:03:45.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:44 smithi045 conmon[33655]: cluster 2021-07-31T02:03:43.933345+0000 mon.a (mon.0) 1828 : cluster [DBG] osdmap e1013: 8 total, 8 up, 8 in 2021-07-31T02:03:45.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:44 smithi045 conmon[33655]: cluster 2021-07-31T02: 2021-07-31T02:03:45.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:44 smithi045 conmon[33655]: 03:44.027227+0000 mgr.x ( 2021-07-31T02:03:45.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:44 smithi045 conmon[33655]: mgr.24542) 709 : cluster [DBG] 2021-07-31T02:03:45.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:44 smithi045 conmon[33655]: pgmap v1355: 41 pgs: 32 creating+peering, 9 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:03:45.519 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] cls_rgw.index_suggest (1472 ms) 2021-07-31T02:03:45.520 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] cls_rgw.index_list 2021-07-31T02:03:45.543 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] cls_rgw.index_list (24 ms) 2021-07-31T02:03:45.543 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] cls_rgw.index_list_delimited 2021-07-31T02:03:46.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:45 smithi154 conmon[34553]: cluster 2021-07-31T02:03:44.938004+0000 mon.a (mon. 2021-07-31T02:03:46.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:45 smithi154 conmon[34553]: 0) 1829 : cluster [DBG] osdmap e1014: 8 total, 8 up, 8 in 2021-07-31T02:03:46.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:45 smithi045 conmon[39718]: cluster 2021-07-31T02:03:44.938004 2021-07-31T02:03:46.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:45 smithi045 conmon[39718]: +0000 mon.a (mon.0) 1829 : cluster [DBG] osdmap e1014: 8 total, 8 up, 8 in 2021-07-31T02:03:46.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:45 smithi045 conmon[33655]: cluster 2021-07-31T02:03:44.938004+0000 mon.a (mon.0) 1829 : cluster [DBG] osdmap e1014: 8 total, 8 up, 8 in 2021-07-31T02:03:47.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:46 smithi154 conmon[34553]: cluster 2021-07-31T02:03:46.027529+0000 mgr.x (mgr. 2021-07-31T02:03:47.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:46 smithi154 conmon[34553]: 24542) 710 : cluster [DBG] pgmap v1357: 41 pgs: 32 creating+peering, 9 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:03:47.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:46 smithi045 conmon[39718]: cluster 2021-07-31T02: 2021-07-31T02:03:47.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:46 smithi045 conmon[39718]: 03:46.027529+0000 mgr.x (mgr.24542) 710 : cluster [DBG] pgmap v1357: 41 pgs: 32 creating+peering, 9 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:03:47.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:46 smithi045 conmon[33655]: cluster 2021-07-31T02:03:46.027529+0000 mgr.x ( 2021-07-31T02:03:47.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:46 smithi045 conmon[33655]: mgr.24542) 710 : cluster [DBG] pgmap v1357: 41 pgs: 32 creating+peering, 9 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:03:49.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:49 smithi154 conmon[34553]: cluster 2021-07-31T02:03:48.027885+0000 mgr.x (mgr.24542) 711 : cluster [DBG] pgmap v1358: 41 pgs: 32 creating+peering, 9 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:03:49.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:49 smithi045 conmon[33655]: cluster 2021-07-31T02:03:48.027885+0000 mgr.x (mgr.24542) 711 : cluster [DBG] pgmap v1358: 41 pgs: 32 creating+peering, 9 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:03:49.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:49 smithi045 conmon[39718]: cluster 2021-07-31T02:03:48.027885+0000 2021-07-31T02:03:49.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:49 smithi045 conmon[39718]: mgr.x (mgr.24542) 711 : cluster [DBG] pgmap v1358: 41 pgs: 32 creating+peering, 9 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:03:51.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:51 smithi154 conmon[34553]: cluster 2021-07-31T02:03:50.028324+0000 mgr.x (mgr.24542) 712 : cluster [DBG] 2021-07-31T02:03:51.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:51 smithi154 conmon[34553]: pgmap v1359: 41 pgs: 41 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 744 KiB/s rd, 479 KiB/s wr, 1.45k op/s 2021-07-31T02:03:51.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:51 smithi045 conmon[39718]: cluster 2021-07-31T02:03:50.028324+0000 mgr.x (mgr.24542) 712 : cluster [DBG] pgmap v1359: 41 pgs: 41 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 744 KiB/s rd, 479 KiB/s wr, 1.45k op/s 2021-07-31T02:03:51.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:51 smithi045 conmon[33655]: cluster 2021-07-31T02:03: 2021-07-31T02:03:51.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:51 smithi045 conmon[33655]: 50.028324+0000 mgr.x (mgr.24542) 712 : cluster [DBG] pgmap v1359: 41 pgs: 41 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 744 KiB/s rd, 479 KiB/s wr, 1.45k op/s 2021-07-31T02:03:53.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:52 smithi154 conmon[34553]: cluster 2021-07-31T02:03:52.028663+0000 mgr.x (mgr.24542) 713 : 2021-07-31T02:03:53.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:52 smithi154 conmon[34553]: cluster [DBG] pgmap v1360: 41 pgs: 41 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 653 KiB/s rd, 420 KiB/s wr, 1.27k op/s 2021-07-31T02:03:53.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:52 smithi045 conmon[33655]: cluster 2021-07-31T02:03:52. 2021-07-31T02:03:53.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:52 smithi045 conmon[33655]: 028663+0000 mgr.x (mgr.24542) 713 : cluster [DBG] pgmap v1360: 41 pgs: 41 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 653 KiB/s rd, 420 KiB/s wr, 1.27k op/s 2021-07-31T02:03:53.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:52 smithi045 conmon[39718]: cluster 2021-07-31T02:03:52.028663 2021-07-31T02:03:53.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:52 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 713 : cluster [DBG] pgmap v1360: 41 pgs: 41 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 653 KiB/s rd, 420 KiB/s wr, 1.27k op/s 2021-07-31T02:03:54.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:03:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:03:53.536Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:03:54.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:03:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:03:53.536Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:03:54.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:03:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:03:53.536Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:03:55.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:55 smithi154 conmon[34553]: cluster 2021-07-31T02: 2021-07-31T02:03:55.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:55 smithi154 conmon[34553]: 03:54.029083+0000 mgr.x (mgr.24542) 714 : cluster [DBG] pgmap v1361: 41 pgs: 41 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 957 KiB/s rd, 626 KiB/s wr, 1.88k op/s 2021-07-31T02:03:55.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:55 smithi045 conmon[39718]: cluster 2021-07-31T02: 2021-07-31T02:03:55.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:55 smithi045 conmon[39718]: 03:54.029083+0000 mgr.x (mgr.24542) 714 : cluster [DBG] pgmap v1361: 41 pgs: 41 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 957 KiB/s rd, 626 KiB/s wr, 1.88k op/s 2021-07-31T02:03:55.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:55 smithi045 conmon[33655]: cluster 2021-07-31T02:03:54.029083 2021-07-31T02:03:55.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:55 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 714 : cluster [DBG] pgmap v1361: 41 pgs: 41 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 957 KiB/s rd, 626 KiB/s wr, 1.88k op/s 2021-07-31T02:03:55.964 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:03:56.285 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:03:56.286 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 18m ago 19m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:03:56.286 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 16m ago 19m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:03:56.286 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 16m ago 22m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:03:56.286 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 18m ago 24m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:03:56.287 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 18m ago 25m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:03:56.287 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 16m ago 23m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:03:56.287 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 18m ago 23m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:03:56.287 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 18m ago 19m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:03:56.287 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 16m ago 19m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:03:56.288 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 18m ago 22m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:03:56.288 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 18m ago 22m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:03:56.288 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 18m ago 21m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:03:56.288 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 18m ago 21m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:03:56.289 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 16m ago 21m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:03:56.289 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 16m ago 20m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:03:56.289 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 16m ago 20m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:03:56.289 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 16m ago 20m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:03:56.289 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 16m ago 19m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:03:56.662 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:03:56.662 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:03:56.662 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:03:56.663 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:03:56.663 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:03:56.663 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:03:56.663 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:03:56.664 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:03:56.664 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:03:56.664 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:03:56.664 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:03:56.665 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:03:56.665 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:03:56.665 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:03:56.666 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:03:56.666 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:03:56.666 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:03:57.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:57 smithi154 conmon[34553]: audit 2021-07-31T02:03:55.951047+0000 mgr.x 2021-07-31T02:03:57.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:57 smithi154 conmon[34553]: (mgr.24542) 715 : audit [DBG] from='client.25561 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:03:57.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:57 smithi154 conmon[34553]: cluster 2021-07-31T02:03:56.029392+0000 mgr.x (mgr.24542) 716 : cluster [DBG] pgmap v1362: 41 pgs: 41 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 863 KiB/s rd, 565 KiB/s wr, 1.70k op/s 2021-07-31T02:03:57.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:57 smithi154 conmon[34553]: audit 2021-07-31T02:03:56 2021-07-31T02:03:57.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:57 smithi154 conmon[34553]: .661784+0000 mon.c (mon.1) 71 : audit 2021-07-31T02:03:57.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:57 smithi154 conmon[34553]: [DBG] from='client.? 172.21.15.45:0/1931592980' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:03:57.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:57 smithi045 conmon[33655]: audit 2021-07-31T02:03:55 2021-07-31T02:03:57.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:57 smithi045 conmon[33655]: .951047+0000 mgr.x (mgr.24542) 715 : audit [DBG] from='client.25561 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:03:57.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:57 smithi045 conmon[33655]: cluster 2021-07- 2021-07-31T02:03:57.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:57 smithi045 conmon[33655]: 31T02:03:56.029392+0000 mgr.x (mgr.24542) 716 : cluster [DBG] pgmap v1362: 41 pgs: 41 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 863 KiB/s rd, 565 KiB/s wr, 1.70k op/s 2021-07-31T02:03:57.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:57 smithi045 conmon[33655]: audit 2021- 2021-07-31T02:03:57.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:57 smithi045 conmon[33655]: 07-31T02:03:56.661784+0000 2021-07-31T02:03:57.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:57 smithi045 conmon[33655]: mon.c (mon.1) 71 : audit [DBG] from='client.? 172.21.15.45:0/1931592980' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:03:57.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:57 smithi045 conmon[33655]: 2021-07-31T02:03:57.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:57 smithi045 conmon[39718]: audit 2021-07-31T02:03:55.951047 2021-07-31T02:03:57.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:57 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 715 : audit [DBG] from='client.25561 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:03:57.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:57 smithi045 conmon[39718]: cluster 2021-07-31 2021-07-31T02:03:57.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:57 smithi045 conmon[39718]: T02:03:56.029392+0000 mgr.x (mgr.24542) 716 : cluster [DBG] pgmap v1362: 41 pgs: 41 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 863 KiB/s rd, 565 KiB/s wr, 1.70k op/s 2021-07-31T02:03:57.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:57 smithi045 conmon[39718]: 2021-07-31T02:03:57.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:57 smithi045 conmon[39718]: audit 2021-07-31T02:03:57.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:57 smithi045 conmon[39718]: 2021-07-31T02:03:56.661784+0000 2021-07-31T02:03:57.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:57 smithi045 conmon[39718]: mon.c (mon.1) 71 : audit [DBG] from='client.? 172.21.15.45:0/1931592980' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:03:58.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:58 smithi154 conmon[34553]: audit 2021-07-31T02:03:56.275487+0000 mgr.x (mgr.24542) 717 : audit [DBG] from='client.25567 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:03:58.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:58 smithi045 conmon[33655]: audit 2021-07-31T02: 2021-07-31T02:03:58.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:58 smithi045 conmon[33655]: 03:56.275487+0000 mgr.x (mgr.24542) 717 : audit [DBG] from='client.25567 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:03:58.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:58 smithi045 conmon[39718]: audit 2021-07-31T02: 2021-07-31T02:03:58.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:58 smithi045 conmon[39718]: 03:56.275487+0000 mgr.x (mgr.24542) 717 : audit [DBG] from='client.25567 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:03:59.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:59 smithi154 conmon[34553]: cluster 2021-07-31T02:03:58.029720+0000 mgr.x (mgr.24542) 2021-07-31T02:03:59.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:03:59 smithi154 conmon[34553]: 718 : cluster [DBG] pgmap v1363: 41 pgs: 41 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 798 KiB/s rd, 522 KiB/s wr, 1.57k op/s 2021-07-31T02:03:59.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:59 smithi045 conmon[33655]: cluster 2021-07-31T02:03:58.029720+0000 mgr.x (mgr. 2021-07-31T02:03:59.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:03:59 smithi045 conmon[33655]: 24542) 718 : cluster [DBG] pgmap v1363: 41 pgs: 41 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 798 KiB/s rd, 522 KiB/s wr, 1.57k op/s 2021-07-31T02:03:59.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:59 smithi045 conmon[39718]: cluster 2021-07-31T02:03:58.029720 2021-07-31T02:03:59.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:03:59 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 718 : cluster [DBG] pgmap v1363: 41 pgs: 41 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 798 KiB/s rd, 522 KiB/s wr, 1.57k op/s 2021-07-31T02:04:00.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:00 smithi154 conmon[34553]: cluster 2021-07-31T02:04:00.030145+0000 mgr.x (mgr.24542) 719 2021-07-31T02:04:00.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:00 smithi154 conmon[34553]: : cluster [DBG] pgmap v1364: 41 pgs: 41 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 1.0 MiB/s rd, 696 KiB/s wr, 2.09k op/s 2021-07-31T02:04:00.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:00 smithi045 conmon[33655]: cluster 2021-07-31T02:04:00.030145+0000 mgr.x (mgr.24542) 719 : cluster [DBG] pgmap v1364: 41 pgs: 41 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 1.0 MiB/s rd, 696 KiB/s wr, 2.09k op/s 2021-07-31T02:04:00.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:00 smithi045 conmon[33655]: 2021-07-31T02:04:00.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:00 smithi045 conmon[39718]: cluster 2021-07-31T02:04:00. 2021-07-31T02:04:00.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:00 smithi045 conmon[39718]: 030145+0000 mgr.x (mgr.24542) 719 : cluster [DBG] pgmap v1364: 41 pgs: 41 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 1.0 MiB/s rd, 696 KiB/s wr, 2.09k op/s 2021-07-31T02:04:03.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:02 smithi154 conmon[34553]: cluster 2021-07-31T02:04:02.030520+0000 mgr.x (mgr.24542) 720 : cluster 2021-07-31T02:04:03.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:02 smithi154 conmon[34553]: [DBG] pgmap v1365: 41 pgs: 41 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 618 KiB/s rd, 412 KiB/s wr, 1.24k op/s 2021-07-31T02:04:03.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:02 smithi045 conmon[33655]: cluster 2021-07-31T02:04:02.030520+0000 2021-07-31T02:04:03.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:02 smithi045 conmon[33655]: mgr.x (mgr.24542) 720 : cluster [DBG] pgmap v1365: 41 pgs: 41 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 618 KiB/s rd, 412 KiB/s wr, 1.24k op/s 2021-07-31T02:04:03.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:02 smithi045 conmon[39718]: cluster 2021-07-31T02:04:02.030520 2021-07-31T02:04:03.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:02 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 720 : cluster [DBG] pgmap v1365: 41 pgs: 41 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 618 KiB/s rd, 412 KiB/s wr, 1.24k op/s 2021-07-31T02:04:04.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:04:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:04:03.536Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:04:04.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:04:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:04:03.536Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:04:04.027 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:04:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:04:03.536Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:04:05.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:05 smithi154 conmon[34553]: cluster 2021-07-31T02:04:04.030969+0000 mgr.x (mgr. 2021-07-31T02:04:05.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:05 smithi154 conmon[34553]: 24542) 721 : cluster [DBG] pgmap v1366: 41 pgs: 41 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 823 KiB/s rd, 549 KiB/s wr, 1.65k op/s 2021-07-31T02:04:05.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:05 smithi045 conmon[33655]: cluster 2021-07-31T02:04:04.030969+0000 mgr.x ( 2021-07-31T02:04:05.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:05 smithi045 conmon[33655]: mgr.24542) 721 : cluster [DBG] pgmap v1366: 41 pgs: 41 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 823 KiB/s rd, 549 KiB/s wr, 1.65k op/s 2021-07-31T02:04:05.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:05 smithi045 conmon[39718]: cluster 2021-07-31T02:04:04.030969+0000 mgr.x (mgr 2021-07-31T02:04:05.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:05 smithi045 conmon[39718]: .24542) 721 : cluster [DBG] pgmap v1366: 41 pgs: 41 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 823 KiB/s rd, 549 KiB/s wr, 1.65k op/s 2021-07-31T02:04:07.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:07 smithi154 conmon[34553]: cluster 2021-07-31T02:04:06.031266 2021-07-31T02:04:07.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:07 smithi154 conmon[34553]: +0000 mgr.x (mgr.24542) 722 : cluster [DBG] pgmap v1367: 41 pgs: 41 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 466 KiB/s rd, 310 KiB/s wr, 931 op/s 2021-07-31T02:04:07.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:07 smithi045 conmon[39718]: cluster 2021-07-31T02:04:06. 2021-07-31T02:04:07.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:07 smithi045 conmon[39718]: 031266+0000 mgr.x (mgr.24542) 722 : cluster [DBG] pgmap v1367: 41 pgs: 41 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 466 KiB/s rd, 310 KiB/s wr, 931 op/s 2021-07-31T02:04:07.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:07 smithi045 conmon[33655]: cluster 2021-07-31T02:04:06.031266+0000 mgr.x (mgr.24542 2021-07-31T02:04:07.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:07 smithi045 conmon[33655]: ) 722 : cluster [DBG] pgmap v1367: 41 pgs: 41 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 466 KiB/s rd, 310 KiB/s wr, 931 op/s 2021-07-31T02:04:09.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:09 smithi154 conmon[34553]: cluster 2021-07-31T02:04:08.031655+0000 mgr.x (mgr.24542) 723 : cluster [DBG] pgmap v1368: 41 pgs: 41 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 466 KiB/s rd, 310 KiB/s wr, 931 op/s 2021-07-31T02:04:09.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:09 smithi045 conmon[39718]: cluster 2021-07-31T02:04:08. 2021-07-31T02:04:09.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:09 smithi045 conmon[39718]: 031655+0000 mgr.x (mgr.24542) 723 : cluster [DBG] pgmap v1368: 41 pgs: 41 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 466 KiB/s rd, 310 KiB/s wr, 931 op/s 2021-07-31T02:04:09.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:09 smithi045 conmon[33655]: cluster 2021-07-31T02:04:08.031655 2021-07-31T02:04:09.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:09 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 723 : cluster [DBG] pgmap v1368: 41 pgs: 41 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 466 KiB/s rd, 310 KiB/s wr, 931 op/s 2021-07-31T02:04:10.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:10 smithi154 conmon[34553]: cluster 2021-07-31T02:04:10.032090+0000 mgr.x (mgr.24542) 724 2021-07-31T02:04:10.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:10 smithi154 conmon[34553]: : cluster [DBG] pgmap v1369: 41 pgs: 41 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 642 KiB/s rd, 428 KiB/s wr, 1.28k op/s 2021-07-31T02:04:10.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:10 smithi045 conmon[39718]: cluster 2021-07-31T02:04:10.032090 2021-07-31T02:04:10.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:10 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 724 : cluster [DBG] pgmap v1369: 41 pgs: 41 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 642 KiB/s rd, 428 KiB/s wr, 1.28k op/s 2021-07-31T02:04:10.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:10 smithi045 conmon[33655]: cluster 2021-07-31T02:04:10.032090+0000 2021-07-31T02:04:10.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:10 smithi045 conmon[33655]: mgr.x (mgr.24542) 724 : cluster [DBG] pgmap v1369: 41 pgs: 41 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 642 KiB/s rd, 428 KiB/s wr, 1.28k op/s 2021-07-31T02:04:13.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:12 smithi154 conmon[34553]: cluster 2021-07-31T02:04:12.032434+0000 mgr.x (mgr.24542) 725 : cluster 2021-07-31T02:04:13.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:12 smithi154 conmon[34553]: [DBG] pgmap v1370: 41 pgs: 41 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 382 KiB/s rd, 254 KiB/s wr, 763 op/s 2021-07-31T02:04:13.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:12 smithi045 conmon[39718]: cluster 2021-07-31T02:04:12.032434+0000 mgr.x (mgr 2021-07-31T02:04:13.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:12 smithi045 conmon[39718]: .24542) 725 : cluster [DBG] pgmap v1370: 41 pgs: 41 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 382 KiB/s rd, 254 KiB/s wr, 763 op/s 2021-07-31T02:04:13.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:12 smithi045 conmon[33655]: cluster 2021-07-31T02: 2021-07-31T02:04:13.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:12 smithi045 conmon[33655]: 04:12.032434+0000 mgr.x (mgr.24542) 725 : cluster [DBG] pgmap v1370: 41 pgs: 41 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 382 KiB/s rd, 254 KiB/s wr, 763 op/s 2021-07-31T02:04:14.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:04:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:04:13.537Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:04:14.027 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:04:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:04:13.537Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:04:14.027 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:04:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:04:13.537Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:04:15.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:15 smithi154 conmon[34553]: cluster 2021-07-31T02:04:14 2021-07-31T02:04:15.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:15 smithi154 conmon[34553]: .032840+0000 mgr.x (mgr.24542) 726 : cluster [DBG] pgmap v1371: 41 pgs: 41 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 556 KiB/s rd, 371 KiB/s wr, 1.11k op/s 2021-07-31T02:04:15.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:15 smithi045 conmon[33655]: cluster 2021-07-31T02:04:14.032840+0000 2021-07-31T02:04:15.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:15 smithi045 conmon[33655]: mgr.x (mgr.24542) 726 : cluster [DBG] pgmap v1371: 41 pgs: 41 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 556 KiB/s rd, 371 KiB/s wr, 1.11k op/s 2021-07-31T02:04:15.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:15 smithi045 conmon[39718]: cluster 2021-07-31T02:04:14. 2021-07-31T02:04:15.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:15 smithi045 conmon[39718]: 032840+0000 mgr.x (mgr.24542) 726 : cluster [DBG] pgmap v1371: 41 pgs: 41 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 556 KiB/s rd, 371 KiB/s wr, 1.11k op/s 2021-07-31T02:04:17.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:17 smithi154 conmon[34553]: cluster 2021-07-31T02:04:16.033120 2021-07-31T02:04:17.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:17 smithi154 conmon[34553]: +0000 mgr.x (mgr.24542) 727 : cluster [DBG] pgmap v1372: 41 pgs: 41 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 351 KiB/s rd, 234 KiB/s wr, 702 op/s 2021-07-31T02:04:17.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:17 smithi045 conmon[39718]: cluster 2021-07-31T02:04:16.033120 2021-07-31T02:04:17.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:17 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 727 : cluster [DBG] pgmap v1372: 41 pgs: 41 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 351 KiB/s rd, 234 KiB/s wr, 702 op/s 2021-07-31T02:04:17.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:17 smithi045 conmon[33655]: cluster 2021-07-31T02:04:16.033120+0000 mgr.x 2021-07-31T02:04:17.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:17 smithi045 conmon[33655]: (mgr.24542) 727 : cluster [DBG] pgmap v1372: 41 pgs: 41 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 351 KiB/s rd, 234 KiB/s wr, 702 op/s 2021-07-31T02:04:19.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:19 smithi154 conmon[34553]: cluster 2021-07-31T02:04:18.033438+0000 mgr.x (mgr 2021-07-31T02:04:19.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:19 smithi154 conmon[34553]: .24542) 728 : cluster [DBG] pgmap v1373: 41 pgs: 41 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 351 KiB/s rd, 234 KiB/s wr, 702 op/s 2021-07-31T02:04:19.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:19 smithi045 conmon[33655]: cluster 2021-07-31T02:04:18.033438+0000 mgr.x (mgr 2021-07-31T02:04:19.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:19 smithi045 conmon[33655]: .24542) 728 : cluster [DBG] pgmap v1373: 41 pgs: 41 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 351 KiB/s rd, 234 KiB/s wr, 702 op/s 2021-07-31T02:04:19.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:19 smithi045 conmon[39718]: cluster 2021-07-31T02:04:18.033438+0000 mgr.x (mgr 2021-07-31T02:04:19.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:19 smithi045 conmon[39718]: .24542) 728 : cluster [DBG] pgmap v1373: 41 pgs: 41 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 351 KiB/s rd, 234 KiB/s wr, 702 op/s 2021-07-31T02:04:20.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:20 smithi154 conmon[34553]: cluster 2021-07-31T02:04:20.033888+0000 mgr.x (mgr. 2021-07-31T02:04:20.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:20 smithi154 conmon[34553]: 24542) 729 : cluster [DBG] pgmap v1374: 41 pgs: 41 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 526 KiB/s rd, 351 KiB/s wr, 1.05k op/s 2021-07-31T02:04:20.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:20 smithi045 conmon[39718]: cluster 2021-07-31T02:04:20.033888+0000 2021-07-31T02:04:20.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:20 smithi045 conmon[39718]: mgr.x (mgr.24542) 729 : cluster [DBG] pgmap v1374: 41 pgs: 41 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 526 KiB/s rd, 351 KiB/s wr, 1.05k op/s 2021-07-31T02:04:20.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:20 smithi045 conmon[33655]: cluster 2021-07-31T02:04:20.033888+0000 mgr.x ( 2021-07-31T02:04:20.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:20 smithi045 conmon[33655]: mgr.24542) 729 : cluster [DBG] pgmap v1374: 41 pgs: 41 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 526 KiB/s rd, 351 KiB/s wr, 1.05k op/s 2021-07-31T02:04:23.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:22 smithi154 conmon[34553]: cluster 2021-07-31T02:04:22.034208+0000 mgr.x (mgr.24542) 730 : cluster 2021-07-31T02:04:23.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:22 smithi154 conmon[34553]: [DBG] pgmap v1375: 41 pgs: 41 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 349 KiB/s rd, 233 KiB/s wr, 698 op/s 2021-07-31T02:04:23.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:22 smithi045 conmon[39718]: cluster 2021-07-31T02:04:22.034208+0000 mgr.x 2021-07-31T02:04:23.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:22 smithi045 conmon[39718]: (mgr.24542) 730 : cluster [DBG] pgmap v1375: 41 pgs: 41 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 349 KiB/s rd, 233 KiB/s wr, 698 op/s 2021-07-31T02:04:23.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:22 smithi045 conmon[33655]: cluster 2021-07-31T02:04:22.034208+0000 mgr.x (mgr 2021-07-31T02:04:23.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:22 smithi045 conmon[33655]: .24542) 730 : cluster [DBG] pgmap v1375: 41 pgs: 41 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 349 KiB/s rd, 233 KiB/s wr, 698 op/s 2021-07-31T02:04:24.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:04:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:04:23.537Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:04:24.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:04:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:04:23.537Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:04:24.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:04:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:04:23.537Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:04:25.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:25 smithi154 conmon[34553]: cluster 2021-07-31T02:04:24.034637 2021-07-31T02:04:25.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:25 smithi154 conmon[34553]: +0000 mgr.x (mgr.24542) 731 : cluster [DBG] pgmap v1376: 41 pgs: 41 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 522 KiB/s rd, 348 KiB/s wr, 1.04k op/s 2021-07-31T02:04:25.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:25 smithi045 conmon[33655]: cluster 2021-07-31T02:04:24.034637+0000 2021-07-31T02:04:25.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:25 smithi045 conmon[33655]: mgr.x (mgr.24542) 731 : cluster [DBG] pgmap v1376: 41 pgs: 41 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 522 KiB/s rd, 348 KiB/s wr, 1.04k op/s 2021-07-31T02:04:25.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:25 smithi045 conmon[39718]: cluster 2021-07-31T02:04:24.034637+0000 2021-07-31T02:04:25.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:25 smithi045 conmon[39718]: mgr.x (mgr.24542) 731 : cluster [DBG] pgmap v1376: 41 pgs: 41 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 522 KiB/s rd, 348 KiB/s wr, 1.04k op/s 2021-07-31T02:04:26.976 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:04:27.268 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:04:27.268 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 19m ago 20m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:04:27.269 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 17m ago 19m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:04:27.269 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 17m ago 23m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:04:27.269 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 19m ago 25m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:04:27.269 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 19m ago 25m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:04:27.269 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 17m ago 23m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:04:27.270 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 19m ago 23m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:04:27.270 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 19m ago 20m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:04:27.270 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 17m ago 20m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:04:27.270 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 19m ago 23m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:04:27.271 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 19m ago 22m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:04:27.271 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 19m ago 22m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:04:27.271 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 19m ago 22m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:04:27.271 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 17m ago 21m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:04:27.271 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 17m ago 21m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:04:27.272 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 17m ago 21m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:04:27.272 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 17m ago 20m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:04:27.272 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 17m ago 20m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:04:27.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:27 smithi154 conmon[34553]: cluster 2021-07-31T02:04:26.035004+0000 mgr.x (mgr.24542) 732 : 2021-07-31T02:04:27.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:27 smithi154 conmon[34553]: cluster [DBG] pgmap v1377: 41 pgs: 41 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 348 KiB/s rd, 232 KiB/s wr, 695 op/s 2021-07-31T02:04:27.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:27 smithi045 conmon[33655]: cluster 2021-07-31T02:04:26.035004+0000 mgr.x ( 2021-07-31T02:04:27.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:27 smithi045 conmon[33655]: mgr.24542) 732 : cluster [DBG] pgmap v1377: 41 pgs: 41 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 348 KiB/s rd, 232 KiB/s wr, 695 op/s 2021-07-31T02:04:27.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:27 smithi045 conmon[39718]: cluster 2021-07-31T02 2021-07-31T02:04:27.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:27 smithi045 conmon[39718]: :04:26.035004+0000 mgr.x (mgr.24542) 732 : cluster [DBG] pgmap v1377: 41 pgs: 41 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 348 KiB/s rd, 232 KiB/s wr, 695 op/s 2021-07-31T02:04:27.658 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:04:27.658 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:04:27.658 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:04:27.658 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:04:27.659 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:04:27.659 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:04:27.659 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:04:27.659 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:04:27.660 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:04:27.660 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:04:27.660 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:04:27.660 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:04:27.661 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:04:27.661 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:04:27.662 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:04:27.662 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:04:27.663 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:04:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:28 smithi154 conmon[34553]: audit 2021-07-31T02:04:26.963670+0000 mgr.x (mgr.24542) 733 2021-07-31T02:04:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:28 smithi154 conmon[34553]: : audit [DBG] from='client.15672 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:04:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:28 smithi154 conmon[34553]: audit 2021-07-31T02:04:27.657544+0000 mon.c (mon.1) 72 : audit [DBG] from='client.? 172.21.15.45:0/948849365' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:04:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:28 smithi154 conmon[34553]: audit 2021-07-31T02:04:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:28 smithi154 conmon[34553]: 2021-07-31T02:04:27.998113+0000 mon.b (mon.2) 153 : audit 2021-07-31T02:04:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:28 smithi154 conmon[34553]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:04:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:28 smithi154 conmon[34553]: audit 2021-07 2021-07-31T02:04:28.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:28 smithi154 conmon[34553]: -31T02:04:27.998529 2021-07-31T02:04:28.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:28 smithi154 conmon[34553]: +0000 mon.a (mon.0) 1830 : audit 2021-07-31T02:04:28.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:28 smithi154 conmon[34553]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:04:28.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:28 smithi154 conmon[34553]: audit 2021-07- 2021-07-31T02:04:28.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:28 smithi154 conmon[34553]: 31T02:04:28.016075+0000 2021-07-31T02:04:28.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:28 smithi154 conmon[34553]: mon.b (mon.2) 154 : audit [INF] 2021-07-31T02:04:28.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:28 smithi154 conmon[34553]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:04:28.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:28 smithi154 conmon[34553]: audit 2021-07- 2021-07-31T02:04:28.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:28 smithi154 conmon[34553]: 31T02:04:28.016447 2021-07-31T02:04:28.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:28 smithi154 conmon[34553]: +0000 mon.a (mon.0) 1831 : 2021-07-31T02:04:28.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:28 smithi154 conmon[34553]: audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:04:28.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:28 smithi045 conmon[33655]: audit 2021-07-31T02:04:26.963670+0000 mgr.x (mgr 2021-07-31T02:04:28.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:28 smithi045 conmon[33655]: .24542) 733 : audit [DBG] from='client.15672 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:04:28.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:28 smithi045 conmon[33655]: audit 2021-07-31T02:04:27.657544+0000 mon.c (mon.1) 72 : audit [DBG] from='client.? 172.21.15.45:0/948849365' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:04:28.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:28 smithi045 conmon[33655]: audit 2021-07-31T02:04:27.998113+0000 2021-07-31T02:04:28.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:28 smithi045 conmon[33655]: mon.b (mon.2) 153 : audit [INF] 2021-07-31T02:04:28.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:28 smithi045 conmon[33655]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:04:28.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:28 smithi045 conmon[33655]: audit 2021-07-31T02: 2021-07-31T02:04:28.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:28 smithi045 conmon[33655]: 04:27.998529+0000 mon.a (mon.0) 1830 2021-07-31T02:04:28.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:28 smithi045 conmon[33655]: : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:04:28.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:28 smithi045 conmon[33655]: audit 2021-07-31T02:04:28.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:28 smithi045 conmon[33655]: 2021-07-31T02: 2021-07-31T02:04:28.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:28 smithi045 conmon[33655]: 04:28.016075+0000 mon.b (mon. 2021-07-31T02:04:28.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:28 smithi045 conmon[33655]: 2) 154 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:04:28.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:28 smithi045 conmon[33655]: audit 2021- 2021-07-31T02:04:28.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:28 smithi045 conmon[33655]: 07-31T02:04:28 2021-07-31T02:04:28.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:28 smithi045 conmon[33655]: .016447+0000 mon.a (mon. 2021-07-31T02:04:28.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:28 smithi045 conmon[33655]: 0) 1831 : audit [INF] 2021-07-31T02:04:28.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:28 smithi045 conmon[33655]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:04:28.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:28 smithi045 conmon[39718]: audit 2021-07-31T02: 2021-07-31T02:04:28.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:28 smithi045 conmon[39718]: 04:26.963670+0000 mgr.x (mgr.24542) 733 : audit [DBG] from='client.15672 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:04:28.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:28 smithi045 conmon[39718]: audit 2021 2021-07-31T02:04:28.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:28 smithi045 conmon[39718]: -07-31T02:04:27 2021-07-31T02:04:28.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:28 smithi045 conmon[39718]: .657544+0000 mon.c (mon 2021-07-31T02:04:28.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:28 smithi045 conmon[39718]: .1) 72 : audit [DBG] 2021-07-31T02:04:28.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:28 smithi045 conmon[39718]: from='client.? 172.21.15.45:0/948849365' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:04:28.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:28 smithi045 conmon[39718]: audit 2021 2021-07-31T02:04:28.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:28 smithi045 conmon[39718]: -07-31T02:04: 2021-07-31T02:04:28.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:28 smithi045 conmon[39718]: 27.998113+0000 mon.b (mon 2021-07-31T02:04:28.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:28 smithi045 conmon[39718]: .2) 153 : audit [INF] 2021-07-31T02:04:28.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:28 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:04:28.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:28 smithi045 conmon[39718]: audit 2021-07-31T02:04:28.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:28 smithi045 conmon[39718]: 2021-07- 2021-07-31T02:04:28.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:28 smithi045 conmon[39718]: 31T02:04:27. 2021-07-31T02:04:28.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:28 smithi045 conmon[39718]: 998529+0000 mon.a (mon.0 2021-07-31T02:04:28.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:28 smithi045 conmon[39718]: ) 1830 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:04:28.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:28 smithi045 conmon[39718]: 2021-07-31T02:04:28.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:28 smithi045 conmon[39718]: audit 2021-07-31T02:04:28.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:28 smithi045 conmon[39718]: 2021-07- 2021-07-31T02:04:28.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:28 smithi045 conmon[39718]: 31T02:04:28 2021-07-31T02:04:28.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:28 smithi045 conmon[39718]: .016075+0000 mon.b (mon. 2021-07-31T02:04:28.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:28 smithi045 conmon[39718]: 2) 154 : audit [INF] 2021-07-31T02:04:28.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:28 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:04:28.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:28 smithi045 conmon[39718]: audit 2021-07-31T02:04:28.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:28 smithi045 conmon[39718]: 2021-07- 2021-07-31T02:04:28.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:28 smithi045 conmon[39718]: 31T02:04:28 2021-07-31T02:04:28.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:28 smithi045 conmon[39718]: .016447+0000 mon.a (mon. 2021-07-31T02:04:28.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:28 smithi045 conmon[39718]: 0) 1831 : audit [INF] 2021-07-31T02:04:28.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:28 smithi045 conmon[39718]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:04:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[34553]: audit 2021-07-31T02:04:27.256336+0000 mgr.x (mgr.24542) 2021-07-31T02:04:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[34553]: 734 : audit [DBG] from='client.15678 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:04:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[34553]: cluster 2021-07-31T02:04:28.035316+0000 mgr.x (mgr.24542) 735 : cluster [DBG] pgmap v1378: 41 pgs: 41 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 348 KiB/s rd, 232 KiB/s wr, 695 op/s 2021-07-31T02:04:29.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:29 smithi045 conmon[33655]: audit 2021-07-31T02:04:27.256336+0000 mgr.x ( 2021-07-31T02:04:29.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:29 smithi045 conmon[33655]: mgr.24542) 734 : audit [DBG] from='client.15678 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:04:29.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:29 smithi045 conmon[33655]: cluster 2021-07-31T02:04:28.035316+0000 mgr.x (mgr.24542) 735 : cluster [DBG] pgmap v1378: 41 pgs: 41 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 348 KiB/s rd, 232 KiB/s wr, 695 op/s 2021-07-31T02:04:29.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:29 smithi045 conmon[39718]: audit 2021-07-31T02:04:27.256336 2021-07-31T02:04:29.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:29 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 734 : audit [DBG] from='client.15678 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:04:29.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:29 smithi045 conmon[39718]: cluster 2021-07 2021-07-31T02:04:29.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:29 smithi045 conmon[39718]: -31T02:04:28.035316+0000 mgr.x (mgr.24542) 735 : 2021-07-31T02:04:29.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:29 smithi045 conmon[39718]: cluster [DBG] pgmap v1378: 41 pgs: 41 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 348 KiB/s rd, 232 KiB/s wr, 695 op/s 2021-07-31T02:04:30.102 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: debug 2021-07-31T02:04:29.867+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:04:30.102 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:04:30.103 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:04:30.103 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: yield 2021-07-31T02:04:30.103 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:04:30.103 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:04:30.104 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:04:30.104 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:04:30.104 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:04:30.104 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:04:30.105 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:04:30.105 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:04:30.105 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:04:30.106 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: task.result() 2021-07-31T02:04:30.106 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:04:30.106 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: debug 2021-07-31T02:04:29.879+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:04:30.106 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:04:30.107 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:04:30.107 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: yield 2021-07-31T02:04:30.107 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:04:30.107 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:04:30.108 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:04:30.109 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:04:30.109 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:04:30.109 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:04:30.110 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:04:30.110 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:04:30.110 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:04:30.110 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: task.result() 2021-07-31T02:04:30.110 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:04:30.111 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: debug 2021-07-31T02:04:29.906+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:04:30.111 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:04:30.111 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:04:30.112 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: yield 2021-07-31T02:04:30.112 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:04:30.112 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:04:30.112 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:04:30.112 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:04:30.113 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:04:30.113 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:04:30.113 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:04:30.114 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:04:30.114 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:04:30.114 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: task.result() 2021-07-31T02:04:30.114 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:04:30.115 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: debug 2021-07-31T02:04:29.906+0000 7fcb9195d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:04:30.115 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:04:30.116 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:04:30.116 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: yield 2021-07-31T02:04:30.116 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:04:30.116 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:04:30.116 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:04:30.117 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:04:30.117 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:04:30.117 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:04:30.117 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:04:30.118 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:04:30.118 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:04:30.119 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: task.result() 2021-07-31T02:04:30.119 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:04:30.119 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: 2021-07-31T02:04:30.119 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:04:30.119 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: 2021-07-31T02:04:30.120 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:04:30.120 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:04:30.120 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:04:30.120 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:04:30.120 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:04:30.121 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:04:30.121 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:04:30.121 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:04:30.121 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:04:30.122 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:04:30.122 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:04:30.122 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:04:30.122 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:04:30.123 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:04:30.123 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:04:30.123 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:04:30.124 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:04:30.124 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:04:30.125 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: raise 2021-07-31T02:04:30.125 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:04:30.125 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:04:30.125 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:04:30.126 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:04:30.126 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:04:30.127 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: debug 2021-07-31T02:04:29.914+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:04:30.127 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:04:30.128 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:04:30.128 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: yield 2021-07-31T02:04:30.129 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:04:30.129 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:04:30.129 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:04:30.129 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:04:30.129 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:04:30.130 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:04:30.130 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:04:30.130 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:04:30.130 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:04:30.131 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: task.result() 2021-07-31T02:04:30.131 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:04:30.131 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: debug 2021-07-31T02:04:29.915+0000 7fcb9215e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:04:30.131 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:04:30.131 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:04:30.132 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: yield 2021-07-31T02:04:30.132 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:04:30.132 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:04:30.132 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:04:30.133 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:04:30.133 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:04:30.133 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:04:30.133 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:04:30.134 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:04:30.134 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:04:30.134 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: task.result() 2021-07-31T02:04:30.134 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:04:30.135 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: 2021-07-31T02:04:30.135 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:04:30.135 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: 2021-07-31T02:04:30.135 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:04:30.135 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:04:30.136 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:04:30.136 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:04:30.136 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:04:30.136 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:04:30.137 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:04:30.137 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:04:30.137 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:04:30.138 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:04:30.138 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:04:30.138 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:04:30.138 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:04:30.138 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:04:30.139 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:04:30.139 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:04:30.139 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:04:30.139 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:04:30.140 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: raise 2021-07-31T02:04:30.140 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:04:30.140 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:04:30.140 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:04:30.140 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:04:30.141 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:04:29 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:04:30.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: audit 2021-07-31T02:04:29.818428+0000 mon.b (mon. 2021-07-31T02:04:30.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: 2) 155 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:04:30.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: audit 2021-07-31T02:04:29.819755+0000 mon.b (mon.2) 156 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:04:30.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: audit 2021-07-31T02:04:30.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: 2021-07-31T02:04:29.820665+0000 mon.b (mon.2) 157 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:04:30.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: cephadm 2021- 2021-07-31T02:04:30.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: 07-31T02:04:29.821340 2021-07-31T02:04:30.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: +0000 mgr.x (mgr.24542) 736 : cephadm [WRN] 2021-07-31T02:04:30.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:04:30.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: cephadm 2021-07-31T 2021-07-31T02:04:30.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: 02:04:29.868643+0000 mgr.x (mgr. 2021-07-31T02:04:30.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: 24542) 737 : cephadm [ERR] 2021-07-31T02:04:30.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:04:30.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:04:30.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: yield 2021-07-31T02:04:30.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:04:30.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:04:30.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:04:30.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:04:30.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:04:30.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:04:30.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:04:30.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:04:30.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:04:30.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:04:30.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:04:30.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:04:29.869083+0000 mgr.x (mgr.24542) 738 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:04:30.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:04:29.880673+0000 mgr.x (mgr.24542) 739 2021-07-31T02:04:30.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: : cephadm [ERR] 2021-07-31T02:04:30.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:04:30.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:04:30.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: yield 2021-07-31T02:04:30.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:04:30.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:04:30.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:04:30.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:04:30.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:04:30.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:04:30.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:04:30.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:04:30.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:04:30.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:04:30.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:04:30.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: cephadm 2021- 2021-07-31T02:04:30.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: 07-31T02:04:29.881429+0000 mgr.x (mgr.24542) 740 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:04:30.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:04:29.907403+0000 mgr.x (mgr.24542) 741 : cephadm [ERR] 2021-07-31T02:04:30.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:04:30.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:04:30.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: yield 2021-07-31T02:04:30.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:04:30.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:04:30.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:04:30.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:04:30.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:04:30.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:04:30.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:04:30.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:04:30.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:04:30.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:04:30.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:04:30.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:04:29.907815+0000 mgr.x (mgr.24542) 742 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:04:30.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:04:30.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:04:30.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: yield 2021-07-31T02:04:30.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:04:30.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:04:30.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:04:30.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:04:30.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:04:30.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:04:30.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:04:30.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:04:30.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:04:30.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:04:30.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:04:30.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: 2021-07-31T02:04:30.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:04:30.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: 2021-07-31T02:04:30.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:04:30.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:04:30.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:04:30.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:04:30.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:04:30.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:04:30.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:04:30.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:04:30.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:04:30.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:04:30.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:04:30.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:04:30.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:04:30.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:04:30.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:04:30.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:04:30.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:04:30.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:04:30.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: raise 2021-07-31T02:04:30.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:04:30.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:04:30.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:04:30.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:04:30.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:04:30.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:04:29.915855+0000 mgr.x (mgr.24542) 743 : cephadm [ERR] 2021-07-31T02:04:30.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:04:30.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:04:30.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: yield 2021-07-31T02:04:30.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:04:30.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:04:30.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:04:30.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:04:30.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:04:30.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:04:30.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:04:30.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:04:30.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:04:30.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:04:30.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:04:30.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:04:29.916395+0000 mgr.x (mgr.24542) 744 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:04:30.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:04:30.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:04:30.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: yield 2021-07-31T02:04:30.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:04:30.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:04:30.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:04:30.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:04:30.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:04:30.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:04:30.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:04:30.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:04:30.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:04:30.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:04:30.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:04:30.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: 2021-07-31T02:04:30.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:04:30.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: 2021-07-31T02:04:30.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:04:30.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:04:30.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:04:30.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:04:30.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:04:30.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:04:30.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:04:30.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:04:30.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:04:30.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:04:30.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:04:30.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:04:30.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:04:30.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:04:30.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:04:30.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:04:30.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:04:30.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:04:30.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: raise 2021-07-31T02:04:30.541 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:04:30.541 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:04:30.541 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:04:30.541 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:04:30.541 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:04:30.542 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:30 smithi154 conmon[34553]: cluster 2021-07-31T02:04:30.035746+0000 mgr.x (mgr.24542) 745 : cluster [DBG] pgmap v1379: 41 pgs: 41 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 517 KiB/s rd, 345 KiB/s wr, 1.03k op/s 2021-07-31T02:04:30.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: audit 2021-07-31T02:04:29.818428+0000 2021-07-31T02:04:30.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: mon.b (mon.2) 155 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:04:30.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: audit 2021-07-31T02:04:29.819755+0000 mon.b (mon.2) 156 : audit [DBG] 2021-07-31T02:04:30.543 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:04:30.543 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: audit 2021-07-31T02:04:29.820665+0000 mon.b ( 2021-07-31T02:04:30.543 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: mon.2) 157 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:04:30.543 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:04:30.544 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: 2021-07-31T02:04:29.821340+0000 mgr.x (mgr.24542) 736 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:04:30.544 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:04:29.868643+0000 mgr.x (mgr.24542) 737 : cephadm [ERR] 2021-07-31T02:04:30.544 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: 2021-07-31T02:04:30.544 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:04:30.545 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:04:30.545 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: yield 2021-07-31T02:04:30.545 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:04:30.545 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:04:30.546 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:04:30.546 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:04:30.546 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:04:30.546 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:04:30.547 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:04:30.547 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:04:30.547 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:04:30.547 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:04:30.547 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:04:30.548 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: cephadm 2021- 2021-07-31T02:04:30.548 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: 07-31T02:04:29.869083+0000 mgr.x (mgr.24542) 738 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:04:30.548 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: cephadm 2021-07 2021-07-31T02:04:30.548 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: -31T02:04:29.880673+0000 mgr.x (mgr 2021-07-31T02:04:30.548 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: .24542) 739 : cephadm [ERR] 2021-07-31T02:04:30.549 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:04:30.549 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:04:30.549 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: yield 2021-07-31T02:04:30.550 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:04:30.550 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:04:30.550 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:04:30.550 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:04:30.551 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:04:30.551 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:04:30.551 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:04:30.551 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:04:30.551 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:04:30.552 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:04:30.552 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:04:30.552 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:04:29.881429+0000 mgr.x (mgr.24542) 740 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:04:30.552 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:04:30.553 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: 2021-07-31T02 2021-07-31T02:04:30.553 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: :04:29.907403+0000 mgr.x 2021-07-31T02:04:30.553 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: (mgr.24542) 741 : cephadm [ERR] 2021-07-31T02:04:30.553 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: 2021-07-31T02:04:30.554 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:04:30.554 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:04:30.554 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: yield 2021-07-31T02:04:30.554 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:04:30.554 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:04:30.555 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:04:30.555 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:04:30.555 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:04:30.555 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:04:30.556 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:04:30.556 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:04:30.556 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:04:30.556 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:04:30.556 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:04:30.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:04:29.907815+0000 mgr.x (mgr.24542) 742 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:04:30.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:04:30.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:04:30.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: yield 2021-07-31T02:04:30.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:04:30.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:04:30.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:04:30.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:04:30.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:04:30.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:04:30.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:04:30.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:04:30.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:04:30.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:04:30.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:04:30.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: 2021-07-31T02:04:30.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:04:30.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: 2021-07-31T02:04:30.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: audit 2021-07-31T02 2021-07-31T02:04:30.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: :04:29.818428+0000 mon.b (mon.2) 155 : audit [DBG] 2021-07-31T02:04:30.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:04:30.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: audit 2021- 2021-07-31T02:04:30.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: 07-31T02:04:29.819755+0000 mon.b (mon 2021-07-31T02:04:30.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: .2) 156 : audit [DBG] 2021-07-31T02:04:30.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:04:30.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: audit 2021-07 2021-07-31T02:04:30.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: -31T02:04:29.820665+0000 mon.b 2021-07-31T02:04:30.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: (mon.2) 157 : audit 2021-07-31T02:04:30.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:04:30.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: cephadm 2021- 2021-07-31T02:04:30.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: 07-31T02:04:29.821340+0000 2021-07-31T02:04:30.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: mgr.x (mgr.24542) 736 2021-07-31T02:04:30.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:04:30.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: cephadm 2021- 2021-07-31T02:04:30.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: 07-31T02:04:29.868643+0000 2021-07-31T02:04:30.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: mgr.x (mgr.24542) 737 : 2021-07-31T02:04:30.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: cephadm [ERR] 2021-07-31T02:04:30.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:04:30.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:04:30.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: yield 2021-07-31T02:04:30.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:04:30.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:04:30.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:04:30.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:04:30.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:04:30.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:04:30.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:04:30.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:04:30.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:04:30.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:04:30.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:04:30.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:04:29.869083+0000 mgr.x (mgr.24542) 738 : cephadm 2021-07-31T02:04:30.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:04:30.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: cephadm 2021-07 2021-07-31T02:04:30.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: -31T02:04:29.880673+0000 mgr.x 2021-07-31T02:04:30.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: (mgr.24542) 739 : cephadm 2021-07-31T02:04:30.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: [ERR] 2021-07-31T02:04:30.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:04:30.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:04:30.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: yield 2021-07-31T02:04:30.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:04:30.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:04:30.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:04:30.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:04:30.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:04:30.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:04:30.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:04:30.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:04:30.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:04:30.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:04:30.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:04:30.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:04:30.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:04:30.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:04:30.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:04:30.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:04:30.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:04:30.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:04:30.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:04:30.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:04:30.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: raise 2021-07-31T02:04:30.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:04:30.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:04:30.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:04:30.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:04:30.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:04:30.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:04:29.915855+0000 mgr.x (mgr.24542) 743 : cephadm [ERR] 2021-07-31T02:04:30.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:04:30.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:04:30.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: yield 2021-07-31T02:04:30.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:04:30.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:04:30.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:04:30.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:04:30.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:04:30.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:04:30.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:04:30.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:04:30.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:04:30.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:04:30.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:04:30.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:04:29.916395+0000 mgr.x (mgr.24542) 744 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:04:30.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:04:30.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:04:30.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: yield 2021-07-31T02:04:30.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:04:30.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:04:30.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:04:30.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:04:30.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:04:30.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:04:30.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:04:30.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:04:30.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:04:30.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:04:30.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:04:30.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: 2021-07-31T02:04:30.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:04:30.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: 2021-07-31T02:04:30.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:04:30.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:04:30.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:04:30.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:04:30.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:04:30.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:04:30.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:04:30.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:04:30.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:04:30.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:04:30.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:04:30.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:04:30.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:04:30.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:04:30.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:04:30.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:04:30.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:04:30.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:04:30.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: raise 2021-07-31T02:04:30.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:04:30.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:04:30.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:04:30.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:04:30.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:04:30.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:04:30.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:04:30.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:04:30.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:04:30.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:04:30.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:04:30.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:04:30.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:04:30.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:04:29.881429+0000 mgr.x (mgr.24542) 740 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:04:30.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: cephadm 2021-07-31T 2021-07-31T02:04:30.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: 02:04:29.907403+0000 mgr.x (mgr.24542) 741 : 2021-07-31T02:04:30.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: cephadm [ERR] 2021-07-31T02:04:30.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:04:30.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:04:30.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: yield 2021-07-31T02:04:30.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:04:30.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:04:30.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:04:30.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:04:30.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:04:30.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:04:30.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:04:30.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:04:30.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:04:30.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:04:30.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:04:30.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:04:29.907815+0000 mgr.x (mgr.24542) 742 : cephadm 2021-07-31T02:04:30.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:04:30.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:04:30.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:04:30.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: yield 2021-07-31T02:04:30.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:04:30.603 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:04:30.603 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:04:30.603 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:04:30.603 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:04:30.603 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:04:30.604 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:04:30.604 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:04:30.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:04:30.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: cluster 2021-07-31T02:04:30. 2021-07-31T02:04:30.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[33655]: 035746+0000 mgr.x (mgr.24542) 745 : cluster [DBG] pgmap v1379: 41 pgs: 41 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 517 KiB/s rd, 345 KiB/s wr, 1.03k op/s 2021-07-31T02:04:30.606 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:04:30.606 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:04:30.606 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:04:30.606 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: 2021-07-31T02:04:30.607 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:04:30.607 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: 2021-07-31T02:04:30.607 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:04:30.607 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:04:30.608 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:04:30.608 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:04:30.608 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:04:30.608 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:04:30.608 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:04:30.609 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:04:30.609 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:04:30.609 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:04:30.609 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:04:30.610 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:04:30.610 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:04:30.610 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:04:30.610 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:04:30.610 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:04:30.611 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:04:30.611 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:04:30.611 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: raise 2021-07-31T02:04:30.611 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:04:30.612 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:04:30.612 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:04:30.612 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:04:30.612 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:04:30.612 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:04:29.915855+0000 mgr.x (mgr.24542) 743 : cephadm [ERR] 2021-07-31T02:04:30.613 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:04:30.613 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:04:30.613 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: yield 2021-07-31T02:04:30.613 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:04:30.613 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:04:30.614 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:04:30.614 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:04:30.614 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:04:30.614 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:04:30.615 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:04:30.615 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:04:30.615 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:04:30.615 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:04:30.615 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:04:30.616 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:04:29.916395+0000 mgr.x (mgr.24542) 744 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:04:30.616 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:04:30.616 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:04:30.616 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: yield 2021-07-31T02:04:30.617 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:04:30.617 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:04:30.617 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:04:30.617 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:04:30.617 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:04:30.618 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:04:30.618 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:04:30.618 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:04:30.618 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:04:30.618 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:04:30.619 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:04:30.619 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: 2021-07-31T02:04:30.619 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:04:30.619 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: 2021-07-31T02:04:30.620 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:04:30.620 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:04:30.620 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:04:30.620 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:04:30.620 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:04:30.621 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:04:30.621 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:04:30.621 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:04:30.621 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:04:30.622 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:04:30.622 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:04:30.622 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:04:30.622 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:04:30.622 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:04:30.623 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:04:30.623 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:04:30.623 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:04:30.623 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:04:30.623 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: raise 2021-07-31T02:04:30.624 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:04:30.624 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:04:30.624 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:04:30.624 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:04:30.625 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:04:30.625 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:30 smithi045 conmon[39718]: cluster 2021-07-31T02:04:30.035746+0000 mgr.x (mgr.24542) 745 : cluster [DBG] pgmap v1379: 41 pgs: 41 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 517 KiB/s rd, 345 KiB/s wr, 1.03k op/s 2021-07-31T02:04:33.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:32 smithi154 conmon[34553]: cluster 2021-07-31T02:04:32.036067+0000 mgr.x (mgr. 2021-07-31T02:04:33.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:32 smithi154 conmon[34553]: 24542) 746 : cluster [DBG] pgmap v1380: 41 pgs: 41 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 343 KiB/s rd, 228 KiB/s wr, 685 op/s 2021-07-31T02:04:33.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:32 smithi045 conmon[33655]: cluster 2021-07-31T02:04:32 2021-07-31T02:04:33.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:32 smithi045 conmon[33655]: .036067+0000 mgr.x (mgr.24542) 746 : cluster [DBG] pgmap v1380: 41 pgs: 41 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 343 KiB/s rd, 228 KiB/s wr, 685 op/s 2021-07-31T02:04:33.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:32 smithi045 conmon[39718]: cluster 2021-07-31T02:04:32.036067 2021-07-31T02:04:33.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:32 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 746 : cluster [DBG] pgmap v1380: 41 pgs: 41 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 343 KiB/s rd, 228 KiB/s wr, 685 op/s 2021-07-31T02:04:34.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:04:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:04:33.537Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:04:34.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:04:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:04:33.537Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:04:34.027 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:04:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:04:33.537Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:04:35.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:35 smithi154 conmon[34553]: cluster 2021-07-31T02:04:34.036510+0000 mgr.x (mgr.24542) 2021-07-31T02:04:35.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:35 smithi154 conmon[34553]: 747 : cluster [DBG] pgmap v1381: 41 pgs: 41 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 511 KiB/s rd, 341 KiB/s wr, 1.02k op/s 2021-07-31T02:04:35.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:35 smithi045 conmon[33655]: cluster 2021-07-31T02:04:34.036510+0000 mgr.x ( 2021-07-31T02:04:35.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:35 smithi045 conmon[33655]: mgr.24542) 747 : cluster [DBG] pgmap v1381: 41 pgs: 41 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 511 KiB/s rd, 341 KiB/s wr, 1.02k op/s 2021-07-31T02:04:35.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:35 smithi045 conmon[39718]: cluster 2021-07-31T02:04:34.036510+0000 mgr.x 2021-07-31T02:04:35.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:35 smithi045 conmon[39718]: (mgr.24542) 747 : cluster [DBG] pgmap v1381: 41 pgs: 41 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 511 KiB/s rd, 341 KiB/s wr, 1.02k op/s 2021-07-31T02:04:37.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:37 smithi154 conmon[34553]: cluster 2021-07-31T02:04:36.036888+0000 mgr.x (mgr.24542 2021-07-31T02:04:37.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:37 smithi154 conmon[34553]: ) 748 : cluster [DBG] pgmap v1382: 41 pgs: 41 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 338 KiB/s rd, 226 KiB/s wr, 676 op/s 2021-07-31T02:04:37.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:37 smithi045 conmon[33655]: cluster 2021-07-31T02:04:36.036888+0000 mgr.x (mgr.24542) 2021-07-31T02:04:37.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:37 smithi045 conmon[33655]: 748 : cluster [DBG] pgmap v1382: 41 pgs: 41 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 338 KiB/s rd, 226 KiB/s wr, 676 op/s 2021-07-31T02:04:37.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:37 smithi045 conmon[39718]: cluster 2021-07-31T02:04:36. 2021-07-31T02:04:37.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:37 smithi045 conmon[39718]: 036888+0000 mgr.x (mgr.24542) 748 : cluster [DBG] pgmap v1382: 41 pgs: 41 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 338 KiB/s rd, 226 KiB/s wr, 676 op/s 2021-07-31T02:04:38.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:38 smithi154 conmon[34553]: cluster 2021-07-31T02:04:38.037246+0000 mgr.x (mgr.24542 2021-07-31T02:04:38.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:38 smithi154 conmon[34553]: ) 749 : cluster [DBG] pgmap v1383: 41 pgs: 41 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 338 KiB/s rd, 226 KiB/s wr, 676 op/s 2021-07-31T02:04:38.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:38 smithi045 conmon[39718]: cluster 2021-07-31T02:04:38.037246 2021-07-31T02:04:38.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:38 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 749 : cluster [DBG] pgmap v1383: 41 pgs: 41 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 338 KiB/s rd, 226 KiB/s wr, 676 op/s 2021-07-31T02:04:38.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:38 smithi045 conmon[33655]: cluster 2021-07-31T02:04:38.037246 2021-07-31T02:04:38.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:38 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 749 : cluster [DBG] pgmap v1383: 41 pgs: 41 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 338 KiB/s rd, 226 KiB/s wr, 676 op/s 2021-07-31T02:04:41.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:41 smithi154 conmon[34553]: cluster 2021-07-31T02:04:40.037705+0000 mgr.x (mgr. 2021-07-31T02:04:41.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:41 smithi154 conmon[34553]: 24542) 750 : cluster [DBG] pgmap v1384: 41 pgs: 41 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 509 KiB/s rd, 339 KiB/s wr, 1.02k op/s 2021-07-31T02:04:41.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:41 smithi045 conmon[33655]: cluster 2021-07-31T02:04:40.037705+0000 mgr.x (mgr.24542) 750 : 2021-07-31T02:04:41.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:41 smithi045 conmon[33655]: cluster [DBG] pgmap v1384: 41 pgs: 41 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 509 KiB/s rd, 339 KiB/s wr, 1.02k op/s 2021-07-31T02:04:41.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:41 smithi045 conmon[39718]: cluster 2021-07-31T02:04:40. 2021-07-31T02:04:41.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:41 smithi045 conmon[39718]: 037705+0000 mgr.x (mgr.24542) 750 : cluster [DBG] pgmap v1384: 41 pgs: 41 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 509 KiB/s rd, 339 KiB/s wr, 1.02k op/s 2021-07-31T02:04:43.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:42 smithi154 conmon[34553]: cluster 2021-07-31T02:04:42.038074+0000 mgr.x (mgr. 2021-07-31T02:04:43.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:42 smithi154 conmon[34553]: 24542) 751 : cluster [DBG] pgmap v1385: 41 pgs: 41 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 339 KiB/s rd, 226 KiB/s wr, 678 op/s 2021-07-31T02:04:43.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:42 smithi045 conmon[39718]: cluster 2021-07-31T02: 2021-07-31T02:04:43.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:42 smithi045 conmon[39718]: 04:42.038074+0000 mgr.x (mgr.24542) 751 : cluster [DBG] pgmap v1385: 41 pgs: 41 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 339 KiB/s rd, 226 KiB/s wr, 678 op/s 2021-07-31T02:04:43.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:42 smithi045 conmon[33655]: cluster 2021-07-31T02:04:42.038074+0000 mgr.x (mgr. 2021-07-31T02:04:43.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:42 smithi045 conmon[33655]: 24542) 751 : cluster [DBG] pgmap v1385: 41 pgs: 41 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 339 KiB/s rd, 226 KiB/s wr, 678 op/s 2021-07-31T02:04:43.383 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] cls_rgw.index_list_delimited (57840 ms) 2021-07-31T02:04:43.383 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] cls_rgw.bi_list 2021-07-31T02:04:43.453 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] cls_rgw.bi_list (69 ms) 2021-07-31T02:04:43.454 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] cls_rgw.gc_set 2021-07-31T02:04:43.489 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] cls_rgw.gc_set (37 ms) 2021-07-31T02:04:43.490 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] cls_rgw.gc_list 2021-07-31T02:04:43.530 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] cls_rgw.gc_list (41 ms) 2021-07-31T02:04:43.531 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] cls_rgw.gc_defer 2021-07-31T02:04:44.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:04:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:04:43.537Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:04:44.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:04:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:04:43.537Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:04:44.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:04:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:04:43.537Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:04:45.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:45 smithi154 conmon[34553]: cluster 2021-07-31T02:04:44.038632+0000 mgr.x (mgr.24542) 752 : cluster 2021-07-31T02:04:45.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:45 smithi154 conmon[34553]: [DBG] pgmap v1386: 41 pgs: 41 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 623 KiB/s rd, 338 KiB/s wr, 1.01k op/s 2021-07-31T02:04:45.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:45 smithi154 conmon[34553]: cluster 2021-07-31T02:04:44.042228+0000 mon.a (mon.0) 1832 : cluster [DBG] osdmap e1015: 8 total, 8 up, 8 in 2021-07-31T02:04:45.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:45 smithi154 conmon[34553]: audit 2021-07- 2021-07-31T02:04:45.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:45 smithi154 conmon[34553]: 31T02:04:44.043631+0000 mon.a (mon.0) 1833 : audit [INF] from='client.? 172.21.15.45:0/1589524186' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74746-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:04:45.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:45 smithi045 conmon[33655]: cluster 2021-07-31T02:04: 2021-07-31T02:04:45.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:45 smithi045 conmon[33655]: 44.038632+0000 mgr.x (mgr.24542) 752 : cluster [DBG] pgmap v1386: 41 pgs: 41 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 623 KiB/s rd, 338 KiB/s wr, 1.01k op/s 2021-07-31T02:04:45.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:45 smithi045 conmon[33655]: cluster 2021-07-31 2021-07-31T02:04:45.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:45 smithi045 conmon[33655]: T02:04:44.042228+0000 mon.a (mon.0) 1832 : cluster 2021-07-31T02:04:45.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:45 smithi045 conmon[33655]: [DBG] osdmap e1015: 8 total, 8 up, 8 in 2021-07-31T02:04:45.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:45 smithi045 conmon[33655]: audit 2021- 2021-07-31T02:04:45.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:45 smithi045 conmon[33655]: 07-31T02:04:44.043631+0000 2021-07-31T02:04:45.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:45 smithi045 conmon[33655]: mon.a (mon.0) 1833 2021-07-31T02:04:45.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:45 smithi045 conmon[33655]: : audit [INF] from='client.? 172.21.15.45:0/1589524186' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74746-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:04:45.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:45 smithi045 conmon[39718]: cluster 2021-07-31T02:04:45.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:45 smithi045 conmon[39718]: 2021 2021-07-31T02:04:45.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:45 smithi045 conmon[39718]: -07-31T02:04:44.038632+0000 mgr.x (mgr.24542) 752 : cluster [DBG] pgmap v1386: 41 pgs: 41 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 623 KiB/s rd, 338 KiB/s wr, 1.01k op/s 2021-07-31T02:04:45.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:45 smithi045 conmon[39718]: cluster 2021-07- 2021-07-31T02:04:45.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:45 smithi045 conmon[39718]: 31T02:04:44.042228+0000 mon.a (mon.0) 1832 : cluster [DBG] 2021-07-31T02:04:45.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:45 smithi045 conmon[39718]: osdmap e1015: 8 total, 8 up, 8 in 2021-07-31T02:04:45.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:45 smithi045 conmon[39718]: audit 2021-07-31T02:04:45.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:45 smithi045 conmon[39718]: 2021-07-31T02:04:44.043631+0000 2021-07-31T02:04:45.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:45 smithi045 conmon[39718]: mon.a (mon.0) 1833 : audit [INF] from='client.? 172.21.15.45:0/1589524186' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74746-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:04:45.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:45 smithi045 conmon[39718]: 2021-07-31T02:04:46.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:46 smithi154 conmon[34553]: audit 2021-07-31T02:04:45.048964+0000 mon.a (mon.0) 1834 : audit [INF] from='client.? 172.21.15.45:0/1589524186' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74746-2","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:04:46.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:46 smithi154 conmon[34553]: cluster 2021-07-31T02:04:46.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:46 smithi154 conmon[34553]: 2021-07-31T02:04:45.049272+0000 mon.a (mon.0) 1835 : cluster [DBG] osdmap e1016: 8 total, 8 up, 8 in 2021-07-31T02:04:46.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:46 smithi045 conmon[33655]: audit 2021-07-31T02:04:45.048964+0000 mon.a (mon.0 2021-07-31T02:04:46.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:46 smithi045 conmon[33655]: ) 1834 : audit [INF] from='client.? 172.21.15.45:0/1589524186' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74746-2","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:04:46.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:46 smithi045 conmon[33655]: cluster 2021-07-31T02:04:45.049272+0000 2021-07-31T02:04:46.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:46 smithi045 conmon[33655]: mon.a (mon.0) 1835 : cluster [DBG] osdmap e1016: 8 total, 8 up, 8 in 2021-07-31T02:04:46.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:46 smithi045 conmon[39718]: audit 2021-07-31T02:04:45.048964+0000 mon.a (mon.0) 1834 : 2021-07-31T02:04:46.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:46 smithi045 conmon[39718]: audit [INF] from='client.? 172.21.15.45:0/1589524186' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74746-2","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:04:46.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:46 smithi045 conmon[39718]: cluster 2021-07-31T02:04:45.049272+0000 mon.a 2021-07-31T02:04:46.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:46 smithi045 conmon[39718]: (mon.0) 1835 : cluster [DBG] osdmap e1016: 8 total, 8 up, 8 in 2021-07-31T02:04:47.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:47 smithi154 conmon[34553]: cluster 2021-07-31T02:04:46.039098+0000 mgr.x (mgr.24542) 753 : cluster [DBG] pgmap v1389: 73 pgs: 32 unknown, 41 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 681 KiB/s rd, 339 KiB/s wr, 1.01k op/s 2021-07-31T02:04:47.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:47 smithi154 conmon[34553]: cluster 2021-07-31T02:04:46.060499+0000 mon.a (mon.0) 1836 : cluster [DBG] osdmap e1017: 8 total, 8 up, 8 in 2021-07-31T02:04:47.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:47 smithi045 conmon[39718]: cluster 2021-07-31T02:04:46.039098+0000 mgr.x (mgr.24542 2021-07-31T02:04:47.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:47 smithi045 conmon[39718]: ) 753 : cluster [DBG] pgmap v1389: 73 pgs: 32 unknown, 41 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 681 KiB/s rd, 339 KiB/s wr, 1.01k op/s 2021-07-31T02:04:47.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:47 smithi045 conmon[39718]: cluster 2021-07-31T02:04:46.060499+0000 mon.a ( 2021-07-31T02:04:47.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:47 smithi045 conmon[39718]: mon.0) 1836 : cluster [DBG] osdmap e1017: 8 total, 8 up, 8 in 2021-07-31T02:04:47.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:47 smithi045 conmon[33655]: cluster 2021-07-31T02:04:46.039098+0000 mgr.x (mgr.24542) 753 : 2021-07-31T02:04:47.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:47 smithi045 conmon[33655]: cluster [DBG] pgmap v1389: 73 pgs: 32 unknown, 41 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 681 KiB/s rd, 339 KiB/s wr, 1.01k op/s 2021-07-31T02:04:47.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:47 smithi045 conmon[33655]: cluster 2021-07-31T02:04:46.060499+0000 mon.a (mon.0) 1836 : cluster [DBG] osdmap e1017: 8 total, 8 up, 8 in 2021-07-31T02:04:49.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:49 smithi154 conmon[34553]: cluster 2021-07-31T02:04:48.039695+0000 mgr.x (mgr.24542) 754 : cluster [DBG] pgmap v1391: 73 pgs: 32 unknown, 41 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 567 KiB/s rd, 224 KiB/s wr, 660 op/s 2021-07-31T02:04:49.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:49 smithi045 conmon[39718]: cluster 2021-07-31T02:04:48.039695+0000 mgr.x (mgr.24542) 754 2021-07-31T02:04:49.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:49 smithi045 conmon[39718]: : cluster [DBG] pgmap v1391: 73 pgs: 32 unknown, 41 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 567 KiB/s rd, 224 KiB/s wr, 660 op/s 2021-07-31T02:04:49.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:49 smithi045 conmon[33655]: cluster 2021-07-31T02:04:48.039695+0000 mgr.x ( 2021-07-31T02:04:49.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:49 smithi045 conmon[33655]: mgr.24542) 754 : cluster [DBG] pgmap v1391: 73 pgs: 32 unknown, 41 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 567 KiB/s rd, 224 KiB/s wr, 660 op/s 2021-07-31T02:04:51.073 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] cls_rgw.gc_defer (7542 ms) 2021-07-31T02:04:51.073 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] cls_rgw.usage_basic 2021-07-31T02:04:51.148 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] cls_rgw.usage_basic (75 ms) 2021-07-31T02:04:51.148 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] cls_rgw.usage_clear_no_obj 2021-07-31T02:04:51.152 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] cls_rgw.usage_clear_no_obj (4 ms) 2021-07-31T02:04:51.152 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] cls_rgw.usage_clear 2021-07-31T02:04:51.251 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] cls_rgw.usage_clear (99 ms) 2021-07-31T02:04:51.251 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] cls_rgw.bi_log_trim 2021-07-31T02:04:51.308 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] cls_rgw.bi_log_trim (57 ms) 2021-07-31T02:04:51.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:51 smithi154 conmon[34553]: cluster 2021-07-31T02:04:50.040471+0000 mgr.x (mgr.24542) 2021-07-31T02:04:51.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:51 smithi154 conmon[34553]: 755 : cluster [DBG] pgmap v1392: 73 pgs: 73 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 682 B/s wr, 1 op/s 2021-07-31T02:04:51.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:51 smithi045 conmon[39718]: cluster 2021-07-31T02:04:51.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:51 smithi045 conmon[39718]: 2021-07-31T02:04:50. 2021-07-31T02:04:51.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:51 smithi045 conmon[39718]: 040471+0000 mgr.x (mgr.24542) 755 : cluster [DBG] pgmap v1392: 73 pgs: 73 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 682 B/s wr, 1 op/s 2021-07-31T02:04:51.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:51 smithi045 conmon[33655]: cluster 2021-07-31T02:04:51.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:51 smithi045 conmon[33655]: 2021-07-31T02:04:51.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:51 smithi045 conmon[33655]: 2021-07-31T02:04 2021-07-31T02:04:51.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:51 smithi045 conmon[33655]: :50.040471+0000 mgr.x (mgr.24542) 755 : cluster [DBG] pgmap v1392: 73 pgs: 73 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 682 B/s wr, 1 op/s 2021-07-31T02:04:52.080 INFO:tasks.workunit.client.0.smithi045.stdout:[----------] 14 tests from cls_rgw (67374 ms total) 2021-07-31T02:04:52.080 INFO:tasks.workunit.client.0.smithi045.stdout: 2021-07-31T02:04:52.081 INFO:tasks.workunit.client.0.smithi045.stdout:[----------] Global test environment tear-down 2021-07-31T02:04:52.081 INFO:tasks.workunit.client.0.smithi045.stdout:[==========] 14 tests from 1 test suite ran. (70050 ms total) 2021-07-31T02:04:52.081 INFO:tasks.workunit.client.0.smithi045.stdout:[ PASSED ] 14 tests. 2021-07-31T02:04:52.083 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-07-31T02:04:52.083 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2021-07-31T02:04:52.164 INFO:tasks.workunit:Running workunit cls/test_cls_sdk.sh... 2021-07-31T02:04:52.164 DEBUG:teuthology.orchestra.run.smithi045:workunit test cls/test_cls_sdk.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=pacific TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_sdk.sh 2021-07-31T02:04:52.235 INFO:tasks.workunit.client.0.smithi045.stdout:Running main() from gmock_main.cc 2021-07-31T02:04:52.235 INFO:tasks.workunit.client.0.smithi045.stdout:[==========] Running 2 tests from 1 test suite. 2021-07-31T02:04:52.236 INFO:tasks.workunit.client.0.smithi045.stdout:[----------] Global test environment set-up. 2021-07-31T02:04:52.236 INFO:tasks.workunit.client.0.smithi045.stdout:[----------] 2 tests from ClsSDK 2021-07-31T02:04:52.236 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] ClsSDK.TestSDKCoverageWrite 2021-07-31T02:04:52.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:52 smithi154 conmon[34553]: cluster 2021-07-31T02:04:51.070049+0000 mon.a (mon.0) 1837 : cluster [DBG] osdmap e1018: 8 total, 8 up, 8 in 2021-07-31T02:04:52.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:52 smithi045 conmon[39718]: cluster 2021-07-31T02:04:52.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:52 smithi045 conmon[39718]: 2021-07-31T02:04:52.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:52 smithi045 conmon[39718]: 2021 2021-07-31T02:04:52.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:52 smithi045 conmon[39718]: - 2021-07-31T02:04:52.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:52 smithi045 conmon[39718]: 07 2021-07-31T02:04:52.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:52 smithi045 conmon[39718]: - 2021-07-31T02:04:52.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:52 smithi045 conmon[39718]: 31 2021-07-31T02:04:52.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:52 smithi045 conmon[39718]: T 2021-07-31T02:04:52.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:52 smithi045 conmon[39718]: 02 2021-07-31T02:04:52.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:52 smithi045 conmon[39718]: : 2021-07-31T02:04:52.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:52 smithi045 conmon[39718]: 04 2021-07-31T02:04:52.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:52 smithi045 conmon[39718]: : 2021-07-31T02:04:52.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:52 smithi045 conmon[39718]: 51 2021-07-31T02:04:52.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:52 smithi045 conmon[39718]: . 2021-07-31T02:04:52.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:52 smithi045 conmon[39718]: 070049 2021-07-31T02:04:52.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:52 smithi045 conmon[39718]: +0000 2021-07-31T02:04:52.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:52 smithi045 conmon[39718]: mon.a (mon.0) 1837 : cluster [DBG] osdmap e1018: 8 total, 8 up, 8 in 2021-07-31T02:04:52.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:52 smithi045 conmon[33655]: cluster 2021-07-31T02:04:52.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:52 smithi045 conmon[33655]: 2021-07-31T02:04:52.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:52 smithi045 conmon[33655]: 2021 2021-07-31T02:04:52.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:52 smithi045 conmon[33655]: - 2021-07-31T02:04:52.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:52 smithi045 conmon[33655]: 07 2021-07-31T02:04:52.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:52 smithi045 conmon[33655]: - 2021-07-31T02:04:52.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:52 smithi045 conmon[33655]: 31 2021-07-31T02:04:52.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:52 smithi045 conmon[33655]: T 2021-07-31T02:04:52.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:52 smithi045 conmon[33655]: 02 2021-07-31T02:04:52.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:52 smithi045 conmon[33655]: : 2021-07-31T02:04:52.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:52 smithi045 conmon[33655]: 04 2021-07-31T02:04:52.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:52 smithi045 conmon[33655]: : 2021-07-31T02:04:52.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:52 smithi045 conmon[33655]: 51 2021-07-31T02:04:52.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:52 smithi045 conmon[33655]: . 2021-07-31T02:04:52.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:52 smithi045 conmon[33655]: 070049 2021-07-31T02:04:52.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:52 smithi045 conmon[33655]: +0000 2021-07-31T02:04:52.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:52 smithi045 conmon[33655]: 2021-07-31T02:04:52.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:52 smithi045 conmon[33655]: mon.a 2021-07-31T02:04:52.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:52 smithi045 conmon[33655]: ( 2021-07-31T02:04:52.535 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:52 smithi045 conmon[33655]: mon 2021-07-31T02:04:52.535 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:52 smithi045 conmon[33655]: . 2021-07-31T02:04:52.535 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:52 smithi045 conmon[33655]: 0 2021-07-31T02:04:52.535 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:52 smithi045 conmon[33655]: ) 2021-07-31T02:04:52.535 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:52 smithi045 conmon[33655]: 1837 2021-07-31T02:04:52.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:52 smithi045 conmon[33655]: : 2021-07-31T02:04:52.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:52 smithi045 conmon[33655]: cluster 2021-07-31T02:04:52.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:52 smithi045 conmon[33655]: 2021-07-31T02:04:52.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:52 smithi045 conmon[33655]: [DBG] 2021-07-31T02:04:52.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:52 smithi045 conmon[33655]: 2021-07-31T02:04:52.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:52 smithi045 conmon[33655]: osdmap e1018: 8 total, 8 up, 8 in 2021-07-31T02:04:52.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:52 smithi045 conmon[33655]: 2021-07-31T02:04:53.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:53 smithi154 conmon[34553]: cluster 2021-07-31T02:04:52.040845+0000 mgr.x (mgr.24542) 756 : cluster [DBG] pgmap v1394: 41 pgs: 41 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:04:53.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:53 smithi154 conmon[34553]: 2021-07-31T02:04:53.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:53 smithi154 conmon[34553]: cluster 2021-07-31T02:04:52.077220+0000 mon.a (mon.0) 1838 : cluster [DBG] osdmap e1019: 8 total, 8 up, 8 in 2021-07-31T02:04:53.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:53 smithi045 conmon[33655]: cluster 2021-07-31T02:04:52.040845+0000 mgr.x ( 2021-07-31T02:04:53.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:53 smithi045 conmon[33655]: mgr.24542) 756 : cluster [DBG] pgmap v1394: 41 pgs: 41 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:04:53.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:53 smithi045 conmon[33655]: cluster 2021-07-31T02:04:52.077220+0000 mon.a (mon.0) 1838 : cluster [DBG] 2021-07-31T02:04:53.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:53 smithi045 conmon[33655]: osdmap e1019: 8 total, 8 up, 8 in 2021-07-31T02:04:53.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:53 smithi045 conmon[39718]: cluster 2021-07-31T02:04:52.040845+0000 2021-07-31T02:04:53.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:53 smithi045 conmon[39718]: mgr.x (mgr.24542) 756 : cluster [DBG] pgmap v1394: 41 pgs: 41 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:04:53.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:53 smithi045 conmon[39718]: cluster 2021 2021-07-31T02:04:53.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:53 smithi045 conmon[39718]: -07-31T02:04:52.077220+0000 mon.a (mon.0) 1838 : cluster [DBG] osdmap e1019: 8 total, 8 up, 8 in 2021-07-31T02:04:54.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:04:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:04:53.537Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:04:54.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:04:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:04:53.537Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:04:54.027 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:04:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:04:53.537Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:04:54.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:54 smithi154 conmon[34553]: cluster 2021-07-31T02:04:53.088182+0000 mon.a (mon.0) 1839 : cluster [DBG] osdmap e1020: 8 total, 8 up, 8 in 2021-07-31T02:04:54.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:54 smithi154 conmon[34553]: audit 2021-07-31T02:04:53.089887+0000 mon.c (mon.1) 73 : audit [INF] from='client.? 172.21.15.45:0/1014413964' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74972-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:04:54.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:54 smithi154 conmon[34553]: audit 2021-07-31T02:04:53.091006+0000 mon.a (mon.0) 1840 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74972-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:04:54.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:54 smithi045 conmon[39718]: cluster 2021-07-31T02:04:53.088182+0000 mon.a (mon.0) 1839 : 2021-07-31T02:04:54.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:54 smithi045 conmon[39718]: cluster [DBG] osdmap e1020: 8 total, 8 up, 8 in 2021-07-31T02:04:54.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:54 smithi045 conmon[39718]: audit 2021-07-31T02:04:53.089887+0000 mon.c (mon.1) 73 : audit [INF] from='client.? 172.21.15.45:0/1014413964' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74972-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:04:54.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:54 smithi045 conmon[39718]: 2021-07-31T02:04:54.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:54 smithi045 conmon[39718]: audit 2021-07-31T02:04:53.091006 2021-07-31T02:04:54.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:54 smithi045 conmon[39718]: +0000 mon.a (mon.0) 1840 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74972-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:04:54.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:54 smithi045 conmon[33655]: cluster 2021-07-31T02:04:53.088182 2021-07-31T02:04:54.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:54 smithi045 conmon[33655]: +0000 mon.a (mon.0) 1839 : cluster [DBG] osdmap e1020: 8 total, 8 up, 8 in 2021-07-31T02:04:54.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:54 smithi045 conmon[33655]: audit 2021-07-31T02:04:53.089887+0000 mon.c (mon.1) 73 : audit [INF] 2021-07-31T02:04:54.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:54 smithi045 conmon[33655]: from='client.? 172.21.15.45:0/1014413964' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74972-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:04:54.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:54 smithi045 conmon[33655]: audit 2021-07 2021-07-31T02:04:54.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:54 smithi045 conmon[33655]: -31T02:04:53.091006+0000 mon.a (mon.0) 2021-07-31T02:04:54.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:54 smithi045 conmon[33655]: 1840 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74972-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:04:55.101 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] ClsSDK.TestSDKCoverageWrite (2865 ms) 2021-07-31T02:04:55.101 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] ClsSDK.TestSDKCoverageReplay 2021-07-31T02:04:55.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:55 smithi154 conmon[34553]: cluster 2021-07-31T02:04:54.041361+0000 mgr.x (mgr.24542) 757 : cluster [DBG] pgmap v1397: 41 pgs: 32 creating+peering, 9 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:04:55.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:55 smithi154 conmon[34553]: audit 2021-07-31T02:04:54.094392+0000 mon.a (mon.0) 1841 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74972-1","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:04:55.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:55 smithi154 conmon[34553]: cluster 2021-07-31T02:04:54.094653+0000 mon.a (mon.0) 1842 : cluster [DBG] osdmap e1021: 8 total, 8 up, 8 in 2021-07-31T02:04:55.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:55 smithi045 conmon[33655]: cluster 2021-07-31T02:04:54.041361+0000 mgr.x (mgr.24542) 2021-07-31T02:04:55.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:55 smithi045 conmon[33655]: 757 : cluster [DBG] pgmap v1397: 41 pgs: 32 creating+peering, 9 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:04:55.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:55 smithi045 conmon[33655]: audit 2021-07-31T02:04:54.094392+0000 mon.a (mon.0) 1841 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74972-1","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:04:55.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:55 smithi045 conmon[33655]: cluster 2021-07-31T02:04:54.094653+0000 mon.a (mon.0) 1842 : cluster [DBG] osdmap e1021: 8 total, 8 up, 8 in 2021-07-31T02:04:55.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:55 smithi045 conmon[39718]: cluster 2021-07-31T02:04:55.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:55 smithi045 conmon[39718]: 2021- 2021-07-31T02:04:55.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:55 smithi045 conmon[39718]: 07-31T02:04:54.041361+0000 mgr.x (mgr.24542) 757 : cluster [DBG] pgmap v1397: 41 pgs: 32 creating+peering, 9 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:04:55.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:55 smithi045 conmon[39718]: audit 2021 2021-07-31T02:04:55.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:55 smithi045 conmon[39718]: -07-31T02:04:54.094392+0000 mon.a (mon.0) 1841 : audit 2021-07-31T02:04:55.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:55 smithi045 conmon[39718]: [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74972-1","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:04:55.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:55 smithi045 conmon[39718]: cluster 2021- 2021-07-31T02:04:55.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:55 smithi045 conmon[39718]: 07-31T02:04:54.094653+0000 mon.a (mon 2021-07-31T02:04:55.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:55 smithi045 conmon[39718]: .0) 1842 : cluster [DBG] osdmap e1021: 8 total, 8 up, 8 in 2021-07-31T02:04:56.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:56 smithi154 conmon[34553]: cluster 2021-07-31T02:04:55.096201+0000 mon.a (mon 2021-07-31T02:04:56.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:56 smithi154 conmon[34553]: .0) 1843 : cluster [DBG] osdmap e1022: 8 total, 8 up, 8 in 2021-07-31T02:04:56.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:56 smithi154 conmon[34553]: cluster 2021-07-31T02:04:56.041835+0000 mgr.x (mgr.24542) 758 : cluster [DBG] pgmap v1400: 9 pgs: 9 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:04:56.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:56 smithi045 conmon[33655]: cluster 2021-07-31T02:04:55.096201+0000 mon.a (mon.0) 2021-07-31T02:04:56.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:56 smithi045 conmon[33655]: 1843 : cluster [DBG] osdmap e1022: 8 total, 8 up, 8 in 2021-07-31T02:04:56.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:56 smithi045 conmon[33655]: cluster 2021-07-31T02:04:56.041835+0000 mgr.x (mgr.24542) 758 : cluster [DBG] pgmap v1400: 9 pgs: 9 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:04:56.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:56 smithi045 conmon[39718]: cluster 2021-07-31T02:04:55.096201+0000 mon.a 2021-07-31T02:04:56.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:56 smithi045 conmon[39718]: (mon.0) 1843 : cluster [DBG] osdmap e1022: 8 total, 8 up, 8 in 2021-07-31T02:04:56.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:56 smithi045 conmon[39718]: cluster 2021-07-31T02:04:56.041835+0000 mgr.x (mgr.24542) 758 : 2021-07-31T02:04:56.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:56 smithi045 conmon[39718]: cluster [DBG] pgmap v1400: 9 pgs: 9 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:04:57.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:57 smithi154 conmon[34553]: cluster 2021-07-31T02:04:56.108397+0000 mon.a (mon.0) 1844 : cluster [DBG] osdmap e1023: 8 total, 8 up, 8 in 2021-07-31T02:04:57.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:57 smithi154 conmon[34553]: 2021-07-31T02:04:57.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:57 smithi154 conmon[34553]: audit 2021-07-31T02:04:56.109786+0000 mon.c (mon.1) 74 : audit [INF] from='client.? 172.21.15.45:0/3958966897' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74972-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:04:57.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:57 smithi154 conmon[34553]: audit 2021-07-31T02:04:56.110644+0000 mon.a ( 2021-07-31T02:04:57.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:57 smithi154 conmon[34553]: mon.0) 1845 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74972-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:04:57.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:57 smithi045 conmon[39718]: cluster 2021-07-31T02:04:56.108397+0000 mon.a (mon.0 2021-07-31T02:04:57.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:57 smithi045 conmon[39718]: ) 1844 : cluster [DBG] osdmap e1023: 8 total, 8 up, 8 in 2021-07-31T02:04:57.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:57 smithi045 conmon[39718]: audit 2021-07-31T02:04:56.109786+0000 mon.c 2021-07-31T02:04:57.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:57 smithi045 conmon[39718]: (mon.1) 74 : audit [INF] from='client.? 172.21.15.45:0/3958966897' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74972-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:04:57.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:57 smithi045 conmon[39718]: audit 2021-07-31T02:04:56.110644+0000 mon.a (mon.0) 1845 2021-07-31T02:04:57.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:57 smithi045 conmon[39718]: : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74972-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:04:57.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:57 smithi045 conmon[33655]: cluster 2021-07-31T02:04:56. 2021-07-31T02:04:57.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:57 smithi045 conmon[33655]: 108397+0000 mon.a (mon.0) 1844 : cluster [DBG] osdmap e1023: 8 total, 8 up, 8 in 2021-07-31T02:04:57.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:57 smithi045 conmon[33655]: audit 2021-07-31T02:04: 2021-07-31T02:04:57.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:57 smithi045 conmon[33655]: 56.109786+0000 mon.c (mon.1) 74 : audit [INF] from='client.? 172.21.15.45:0/3958966897' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74972-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:04:57.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:57 smithi045 conmon[33655]: audit 2021-07-31T02:04:56. 2021-07-31T02:04:57.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:57 smithi045 conmon[33655]: 110644+0000 mon.a (mon.0) 1845 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74972-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:04:57.971 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:04:58.124 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] ClsSDK.TestSDKCoverageReplay (3024 ms) 2021-07-31T02:04:58.124 INFO:tasks.workunit.client.0.smithi045.stdout:[----------] 2 tests from ClsSDK (5889 ms total) 2021-07-31T02:04:58.125 INFO:tasks.workunit.client.0.smithi045.stdout: 2021-07-31T02:04:58.125 INFO:tasks.workunit.client.0.smithi045.stdout:[----------] Global test environment tear-down 2021-07-31T02:04:58.125 INFO:tasks.workunit.client.0.smithi045.stdout:[==========] 2 tests from 1 test suite ran. (5889 ms total) 2021-07-31T02:04:58.125 INFO:tasks.workunit.client.0.smithi045.stdout:[ PASSED ] 2 tests. 2021-07-31T02:04:58.126 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-07-31T02:04:58.127 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2021-07-31T02:04:58.192 INFO:tasks.workunit:Stopping ['cls'] on client.0... 2021-07-31T02:04:58.192 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2021-07-31T02:04:58.256 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:04:58.257 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 19m ago 20m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:04:58.257 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 17m ago 20m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:04:58.257 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 17m ago 23m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:04:58.257 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 19m ago 25m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:04:58.258 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 19m ago 26m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:04:58.258 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 17m ago 24m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:04:58.259 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 19m ago 24m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:04:58.259 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 19m ago 20m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:04:58.259 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 17m ago 20m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:04:58.259 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 19m ago 23m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:04:58.260 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 19m ago 23m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:04:58.260 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 19m ago 22m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:04:58.260 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 19m ago 22m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:04:58.261 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 17m ago 22m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:04:58.261 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 17m ago 21m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:04:58.261 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 17m ago 21m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:04:58.261 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 17m ago 21m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:04:58.262 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 17m ago 20m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:04:58.425 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:58 smithi045 conmon[39718]: audit 2021-07-31T02:04:57.116294+0000 2021-07-31T02:04:58.426 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:58 smithi045 conmon[39718]: mon.a (mon.0) 1846 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74972-2","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:04:58.426 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:58 smithi045 conmon[39718]: cluster 2021 2021-07-31T02:04:58.427 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:58 smithi045 conmon[39718]: -07-31T02:04:57.116590+0000 mon.a (mon.0) 1847 : cluster 2021-07-31T02:04:58.427 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:58 smithi045 conmon[39718]: [DBG] osdmap e1024: 8 total, 8 up, 8 in 2021-07-31T02:04:58.427 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:58 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T02:04:58.428 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:58 smithi045 conmon[39718]: T02:04:57.958764+0000 mgr.x (mgr 2021-07-31T02:04:58.428 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:58 smithi045 conmon[39718]: .24542) 759 : audit [DBG] from='client.25615 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:04:58.428 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:58 smithi045 conmon[39718]: cluster 2021-07-31T 2021-07-31T02:04:58.428 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:58 smithi045 conmon[39718]: 02:04:58.042216+0000 mgr.x (mgr.24542) 760 2021-07-31T02:04:58.429 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:58 smithi045 conmon[39718]: : cluster [DBG] pgmap v1403: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:04:58.429 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:58 smithi045 conmon[33655]: audit 2021-07-31T02:04:57 2021-07-31T02:04:58.429 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:58 smithi045 conmon[33655]: .116294+0000 mon.a (mon.0) 1846 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74972-2","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:04:58.430 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:58 smithi045 conmon[33655]: cluster 2021-07-31T02: 2021-07-31T02:04:58.430 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:58 smithi045 conmon[33655]: 04:57.116590+0000 mon.a (mon.0) 1847 : cluster [DBG] osdmap e1024: 8 total, 8 up, 8 in 2021-07-31T02:04:58.430 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:58 smithi045 conmon[33655]: audit 2021-07 2021-07-31T02:04:58.431 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:58 smithi045 conmon[33655]: -31T02:04:57.958764+0000 mgr.x (mgr. 2021-07-31T02:04:58.431 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:58 smithi045 conmon[33655]: 24542) 759 : audit [DBG] from='client.25615 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:04:58.431 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:58 smithi045 conmon[33655]: 2021-07-31T02:04:58.431 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:58 smithi045 conmon[33655]: cluster 2021-07-31T02:04:58.431 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:58 smithi045 conmon[33655]: 2021-07 2021-07-31T02:04:58.432 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:58 smithi045 conmon[33655]: -31T02:04:58. 2021-07-31T02:04:58.432 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:58 smithi045 conmon[33655]: 042216+0000 mgr.x (mgr. 2021-07-31T02:04:58.432 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:58 smithi045 conmon[33655]: 24542) 760 : cluster [DBG] 2021-07-31T02:04:58.433 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:58 smithi045 conmon[33655]: pgmap v1403: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:04:58.434 DEBUG:teuthology.parallel:result is None 2021-07-31T02:04:58.435 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2021-07-31T02:04:58.498 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2021-07-31T02:04:58.499 DEBUG:teuthology.orchestra.run.smithi045:> rmdir -- /home/ubuntu/cephtest/mnt.0 2021-07-31T02:04:58.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:58 smithi154 conmon[34553]: audit 2021-07-31T02:04:57.116294+0000 mon.a (mon.0) 1846 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi045-74972-2","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:04:58.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:58 smithi154 conmon[34553]: cluster 2021-07-31T02:04:57.116590+0000 mon.a (mon.0) 1847 : cluster [DBG] osdmap e1024: 8 total, 8 up, 8 in 2021-07-31T02:04:58.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:58 smithi154 conmon[34553]: audit 2021-07-31T02:04:57.958764+0000 mgr.x ( 2021-07-31T02:04:58.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:58 smithi154 conmon[34553]: mgr.24542) 759 : audit [DBG] from='client.25615 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:04:58.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:58 smithi154 conmon[34553]: cluster 2021-07-31T02: 2021-07-31T02:04:58.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:58 smithi154 conmon[34553]: 04:58.042216+0000 mgr.x (mgr.24542) 760 : cluster [DBG] pgmap v1403: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:04:58.553 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2021-07-31T02:04:58.553 INFO:teuthology.task.full_sequential:In full_sequential, running task print... 2021-07-31T02:04:58.554 INFO:teuthology.task.print:**** done end rados_api.yaml 2021-07-31T02:04:58.554 INFO:teuthology.task.full_sequential:In full_sequential, running task print... 2021-07-31T02:04:58.554 INFO:teuthology.task.print:**** done start rados_loadgenbig.yaml 2021-07-31T02:04:58.555 INFO:teuthology.task.full_sequential:In full_sequential, running task workunit... 2021-07-31T02:04:58.556 INFO:tasks.workunit:Pulling workunits from ref pacific 2021-07-31T02:04:58.556 INFO:tasks.workunit:Making a separate scratch dir for every client... 2021-07-31T02:04:58.557 DEBUG:teuthology.orchestra.run.smithi045:> stat -- /home/ubuntu/cephtest/mnt.0 2021-07-31T02:04:58.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-07-31T02:04:58.608 INFO:teuthology.orchestra.run.smithi045.stderr:stat: cannot stat '/home/ubuntu/cephtest/mnt.0': No such file or directory 2021-07-31T02:04:58.608 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -- /home/ubuntu/cephtest/mnt.0 2021-07-31T02:04:58.609 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:04:58.609 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:04:58.610 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:04:58.610 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:04:58.610 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:04:58.610 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:04:58.611 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:04:58.611 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:04:58.611 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:04:58.611 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:04:58.611 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:04:58.612 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:04:58.612 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:04:58.612 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:04:58.612 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:04:58.613 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:04:58.613 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:04:58.663 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2021-07-31T02:04:58.664 DEBUG:teuthology.orchestra.run.smithi045:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2021-07-31T02:04:58.717 INFO:tasks.workunit:timeout=3h 2021-07-31T02:04:58.718 INFO:tasks.workunit:cleanup=True 2021-07-31T02:04:58.718 DEBUG:teuthology.orchestra.run.smithi045:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone --depth 1 --branch pacific git://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 2021-07-31T02:04:58.773 INFO:tasks.workunit.client.0.smithi045.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2021-07-31T02:04:58.797 INFO:tasks.workunit.client.0.smithi045.stderr:warning: Could not find remote branch pacific to clone. 2021-07-31T02:04:58.798 INFO:tasks.workunit.client.0.smithi045.stderr:fatal: Remote branch pacific not found in upstream origin 2021-07-31T02:04:58.798 DEBUG:teuthology.orchestra.run:got remote process result: 128 2021-07-31T02:04:58.799 INFO:tasks.workunit:failed to check out 'pacific' from git://git.ceph.com/ceph-ci.git; will also try in git://git.ceph.com/ceph.git 2021-07-31T02:04:58.800 DEBUG:teuthology.orchestra.run.smithi045:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone --depth 1 --branch pacific git://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 2021-07-31T02:04:58.855 INFO:tasks.workunit.client.0.smithi045.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2021-07-31T02:04:59.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:59 smithi154 conmon[34553]: cluster 2021-07-31T02:04:58.122418+0000 mon.a ( 2021-07-31T02:04:59.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:59 smithi154 conmon[34553]: mon.0) 1848 : cluster [DBG] osdmap e1025: 8 total, 8 up, 8 in 2021-07-31T02:04:59.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:59 smithi154 conmon[34553]: audit 2021-07-31T02:04:58.248109+0000 mgr.x (mgr.24542) 761 : audit [DBG] from='client.25621 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:04:59.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:59 smithi154 conmon[34553]: audit 2021-07-31T02:04:58.607962+0000 mon.c (mon. 2021-07-31T02:04:59.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:04:59 smithi154 conmon[34553]: 1) 75 : audit [DBG] from='client.? 172.21.15.45:0/1329126588' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:04:59.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:59 smithi045 conmon[33655]: cluster 2021-07-31T02:04:58 2021-07-31T02:04:59.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:59 smithi045 conmon[33655]: .122418+0000 mon.a (mon.0) 1848 : cluster [DBG] osdmap e1025: 8 total, 8 up, 8 in 2021-07-31T02:04:59.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:59 smithi045 conmon[33655]: audit 2021-07-31 2021-07-31T02:04:59.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:59 smithi045 conmon[33655]: T02:04:58.248109+0000 mgr.x (mgr.24542) 761 : audit 2021-07-31T02:04:59.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:59 smithi045 conmon[33655]: [DBG] from='client.25621 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:04:59.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:59 smithi045 conmon[33655]: audit 2021- 2021-07-31T02:04:59.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:59 smithi045 conmon[33655]: 07-31T02:04:58.607962+0000 2021-07-31T02:04:59.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:59 smithi045 conmon[33655]: mon.c (mon.1) 75 : 2021-07-31T02:04:59.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:04:59 smithi045 conmon[33655]: audit [DBG] from='client.? 172.21.15.45:0/1329126588' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:04:59.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:59 smithi045 conmon[39718]: cluster 2021-07-31T02:04:58.122418 2021-07-31T02:04:59.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:59 smithi045 conmon[39718]: +0000 mon.a (mon.0) 1848 : cluster [DBG] osdmap e1025: 8 total, 8 up, 8 in 2021-07-31T02:04:59.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:59 smithi045 conmon[39718]: audit 2021-07-31T02:04:59.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:59 smithi045 conmon[39718]: 2021-07-31T02:04:58.248109+0000 mgr.x (mgr.24542) 761 2021-07-31T02:04:59.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:59 smithi045 conmon[39718]: : audit [DBG] from='client.25621 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:04:59.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:59 smithi045 conmon[39718]: audit 2021 2021-07-31T02:04:59.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:59 smithi045 conmon[39718]: -07-31T02:04:58.607962+0000 2021-07-31T02:04:59.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:59 smithi045 conmon[39718]: mon.c (mon.1) 75 : audit [DBG] 2021-07-31T02:04:59.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:04:59 smithi045 conmon[39718]: from='client.? 172.21.15.45:0/1329126588' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:05:00.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:00 smithi154 conmon[34553]: cluster 2021-07-31T02:05:00.042803+0000 mgr.x (mgr.24542) 2021-07-31T02:05:00.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:00 smithi154 conmon[34553]: 762 : cluster [DBG] pgmap v1405: 9 pgs: 9 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:05:00.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:00 smithi045 conmon[33655]: cluster 2021-07-31T02:05:00. 2021-07-31T02:05:00.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:00 smithi045 conmon[33655]: 042803+0000 mgr.x (mgr.24542) 762 : cluster [DBG] pgmap v1405: 9 pgs: 9 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:05:00.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:00 smithi045 conmon[39718]: cluster 2021-07-31T02:05:00 2021-07-31T02:05:00.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:00 smithi045 conmon[39718]: .042803+0000 mgr.x (mgr.24542) 762 : cluster [DBG] pgmap v1405: 9 pgs: 9 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:05:03.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:02 smithi154 conmon[34553]: cluster 2021-07-31T02:05:02.043189+0000 mgr.x (mgr.24542) 763 : cluster [DBG] pgmap v1406: 9 pgs: 9 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:05:03.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:02 smithi045 conmon[33655]: cluster 2021-07-31T02:05:02.043189+0000 mgr.x ( 2021-07-31T02:05:03.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:02 smithi045 conmon[33655]: mgr.24542) 763 : cluster [DBG] pgmap v1406: 9 pgs: 9 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:05:03.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:02 smithi045 conmon[39718]: cluster 2021-07-31T02:05:02.043189 2021-07-31T02:05:03.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:02 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 763 : cluster [DBG] pgmap v1406: 9 pgs: 9 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:05:03.963 DEBUG:teuthology.orchestra.run.smithi045:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2021-07-31T02:05:04.018 INFO:tasks.workunit.client.0.smithi045.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2021-07-31T02:05:04.020 INFO:tasks.workunit.client.0.smithi045.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2021-07-31T02:05:04.020 INFO:tasks.workunit.client.0.smithi045.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2021-07-31T02:05:04.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:05:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:05:03.538Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:05:04.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:05:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:05:03.538Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:05:04.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:05:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:05:03.538Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:05:04.067 INFO:tasks.workunit.client.0.smithi045.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2021-07-31T02:05:04.115 INFO:tasks.workunit.client.0.smithi045.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2021-07-31T02:05:04.152 INFO:tasks.workunit.client.0.smithi045.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2021-07-31T02:05:04.154 INFO:tasks.workunit.client.0.smithi045.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2021-07-31T02:05:04.154 INFO:tasks.workunit.client.0.smithi045.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2021-07-31T02:05:04.191 INFO:tasks.workunit.client.0.smithi045.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2021-07-31T02:05:04.194 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2021-07-31T02:05:04.194 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2021-07-31T02:05:04.249 INFO:tasks.workunit:Running workunits matching rados/load-gen-big.sh on client.0... 2021-07-31T02:05:04.251 INFO:tasks.workunit:Running workunit rados/load-gen-big.sh... 2021-07-31T02:05:04.251 DEBUG:teuthology.orchestra.run.smithi045:workunit test rados/load-gen-big.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=pacific TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/load-gen-big.sh 2021-07-31T02:05:04.345 INFO:tasks.workunit.client.0.smithi045.stdout:run length 1200 seconds 2021-07-31T02:05:04.346 INFO:tasks.workunit.client.0.smithi045.stdout:preparing 10240 objects 2021-07-31T02:05:05.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:05 smithi154 conmon[34553]: cluster 2021-07-31T02:05:04.043617+0000 2021-07-31T02:05:05.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:05 smithi154 conmon[34553]: mgr.x (mgr.24542) 764 : cluster [DBG] pgmap v1407: 9 pgs: 9 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:05:05.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:05 smithi045 conmon[33655]: cluster 2021-07-31T02:05:05.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:05 smithi045 conmon[33655]: 2021-07-31T02:05:05.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:05 smithi045 conmon[33655]: 2021 2021-07-31T02:05:05.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:05 smithi045 conmon[33655]: - 2021-07-31T02:05:05.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:05 smithi045 conmon[33655]: 07 2021-07-31T02:05:05.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:05 smithi045 conmon[33655]: - 2021-07-31T02:05:05.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:05 smithi045 conmon[33655]: 31 2021-07-31T02:05:05.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:05 smithi045 conmon[33655]: T 2021-07-31T02:05:05.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:05 smithi045 conmon[33655]: 02 2021-07-31T02:05:05.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:05 smithi045 conmon[33655]: : 2021-07-31T02:05:05.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:05 smithi045 conmon[33655]: 05 2021-07-31T02:05:05.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:05 smithi045 conmon[33655]: : 2021-07-31T02:05:05.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:05 smithi045 conmon[33655]: 04 2021-07-31T02:05:05.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:05 smithi045 conmon[33655]: . 2021-07-31T02:05:05.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:05 smithi045 conmon[33655]: 043617 2021-07-31T02:05:05.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:05 smithi045 conmon[33655]: +0000 2021-07-31T02:05:05.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:05 smithi045 conmon[33655]: 2021-07-31T02:05:05.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:05 smithi045 conmon[33655]: mgr.x 2021-07-31T02:05:05.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:05 smithi045 conmon[33655]: ( 2021-07-31T02:05:05.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:05 smithi045 conmon[33655]: mgr 2021-07-31T02:05:05.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:05 smithi045 conmon[33655]: . 2021-07-31T02:05:05.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:05 smithi045 conmon[33655]: 24542 2021-07-31T02:05:05.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:05 smithi045 conmon[33655]: ) 2021-07-31T02:05:05.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:05 smithi045 conmon[33655]: 764 2021-07-31T02:05:05.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:05 smithi045 conmon[33655]: : 2021-07-31T02:05:05.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:05 smithi045 conmon[33655]: cluster 2021-07-31T02:05:05.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:05 smithi045 conmon[33655]: 2021-07-31T02:05:05.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:05 smithi045 conmon[33655]: [DBG] 2021-07-31T02:05:05.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:05 smithi045 conmon[33655]: 2021-07-31T02:05:05.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:05 smithi045 conmon[33655]: pgmap v1407: 9 pgs: 9 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:05:05.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:05 smithi045 conmon[33655]: 2021-07-31T02:05:05.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:05 smithi045 conmon[39718]: cluster 2021-07-31T02:05:05.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:05 smithi045 conmon[39718]: 2021-07-31T02:05:05.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:05 smithi045 conmon[39718]: 2021 2021-07-31T02:05:05.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:05 smithi045 conmon[39718]: - 2021-07-31T02:05:05.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:05 smithi045 conmon[39718]: 07 2021-07-31T02:05:05.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:05 smithi045 conmon[39718]: - 2021-07-31T02:05:05.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:05 smithi045 conmon[39718]: 31 2021-07-31T02:05:05.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:05 smithi045 conmon[39718]: T 2021-07-31T02:05:05.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:05 smithi045 conmon[39718]: 02 2021-07-31T02:05:05.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:05 smithi045 conmon[39718]: : 2021-07-31T02:05:05.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:05 smithi045 conmon[39718]: 05 2021-07-31T02:05:05.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:05 smithi045 conmon[39718]: : 2021-07-31T02:05:05.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:05 smithi045 conmon[39718]: 04 2021-07-31T02:05:05.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:05 smithi045 conmon[39718]: . 2021-07-31T02:05:05.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:05 smithi045 conmon[39718]: 043617 2021-07-31T02:05:05.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:05 smithi045 conmon[39718]: +0000 2021-07-31T02:05:05.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:05 smithi045 conmon[39718]: 2021-07-31T02:05:05.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:05 smithi045 conmon[39718]: mgr.x 2021-07-31T02:05:05.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:05 smithi045 conmon[39718]: ( 2021-07-31T02:05:05.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:05 smithi045 conmon[39718]: mgr 2021-07-31T02:05:05.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:05 smithi045 conmon[39718]: . 2021-07-31T02:05:05.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:05 smithi045 conmon[39718]: 24542 2021-07-31T02:05:05.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:05 smithi045 conmon[39718]: ) 2021-07-31T02:05:05.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:05 smithi045 conmon[39718]: 764 2021-07-31T02:05:05.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:05 smithi045 conmon[39718]: : 2021-07-31T02:05:05.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:05 smithi045 conmon[39718]: cluster 2021-07-31T02:05:05.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:05 smithi045 conmon[39718]: 2021-07-31T02:05:05.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:05 smithi045 conmon[39718]: [DBG] 2021-07-31T02:05:05.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:05 smithi045 conmon[39718]: 2021-07-31T02:05:05.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:05 smithi045 conmon[39718]: pgmap v1407: 9 pgs: 9 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:05:05.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:05 smithi045 conmon[39718]: 2021-07-31T02:05:06.193 INFO:tasks.workunit.client.0.smithi045.stdout:load-gen will run 1200 seconds 2021-07-31T02:05:06.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:06 smithi154 conmon[34553]: cluster 2021-07-31T02:05:06.044012+0000 mgr.x ( 2021-07-31T02:05:06.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:06 smithi154 conmon[34553]: mgr.24542) 765 : cluster [DBG] pgmap v1408: 9 pgs: 9 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:05:06.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:06 smithi045 conmon[33655]: cluster 2021-07-31T02:05:06.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:06 smithi045 conmon[33655]: 2021-07-31T02:05:06.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:06 smithi045 conmon[33655]: 2021 2021-07-31T02:05:06.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:06 smithi045 conmon[33655]: - 2021-07-31T02:05:06.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:06 smithi045 conmon[33655]: 07-31T02: 2021-07-31T02:05:06.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:06 smithi045 conmon[33655]: 05:06.044012+0000 mgr.x (mgr.24542) 765 : cluster [DBG] pgmap v1408: 9 pgs: 9 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:05:06.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:06 smithi045 conmon[39718]: cluster 2021-07-31T02:05:06.044012+0000 mgr.x (mgr.24542) 765 : cluster [DBG] pgmap v1408: 9 pgs: 9 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:05:07.200 INFO:tasks.workunit.client.0.smithi045.stdout: 1: throughput=0MB/sec pending data=0 2021-07-31T02:05:07.201 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-OTVXK_cb9TJIFWt off=16743009 len=615039 2021-07-31T02:05:07.201 INFO:tasks.workunit.client.0.smithi045.stdout:op 0 completed, throughput=0.583MB/sec 2021-07-31T02:05:07.201 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ZLWQ7g8sFCZhpCE off=6395216 len=1375554 2021-07-31T02:05:07.206 INFO:tasks.workunit.client.0.smithi045.stdout:op 1 completed, throughput=1.87MB/sec 2021-07-31T02:05:07.207 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-b44-1OF7ADWUV28 off=19675869 len=1381670 2021-07-31T02:05:07.211 INFO:tasks.workunit.client.0.smithi045.stdout:op 2 completed, throughput=3.16MB/sec 2021-07-31T02:05:07.211 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-88BTvXPlA4VEEYW off=11822187 len=202723 2021-07-31T02:05:07.212 INFO:tasks.workunit.client.0.smithi045.stdout:op 3 completed, throughput=3.35MB/sec 2021-07-31T02:05:07.212 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-s7b1Om-MYrcMJ7c off=11090554 len=2006087 2021-07-31T02:05:07.224 INFO:tasks.workunit.client.0.smithi045.stdout:op 4 completed, throughput=5.16MB/sec 2021-07-31T02:05:08.225 INFO:tasks.workunit.client.0.smithi045.stdout: 2: throughput=2.62MB/sec pending data=0 2021-07-31T02:05:08.225 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ZE8uW0grAnLbZmG off=16248454 len=805676 2021-07-31T02:05:08.226 INFO:tasks.workunit.client.0.smithi045.stdout:op 5 completed, throughput=3MB/sec 2021-07-31T02:05:08.226 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-QljyQIOlwjHkWNL off=6530230 len=461020 2021-07-31T02:05:08.231 INFO:tasks.workunit.client.0.smithi045.stdout:op 6 completed, throughput=3.2MB/sec 2021-07-31T02:05:08.231 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-VdxOX0fGtPHbAWn off=10694004 len=176023 2021-07-31T02:05:08.232 INFO:tasks.workunit.client.0.smithi045.stdout:op 7 completed, throughput=3.29MB/sec 2021-07-31T02:05:08.232 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Ds3tA1g-fUSxv2T off=3538894 len=2020461 2021-07-31T02:05:08.244 INFO:tasks.workunit.client.0.smithi045.stdout:op 8 completed, throughput=4.21MB/sec 2021-07-31T02:05:08.245 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-6TxDUpiGOQlWBai off=5752841 len=2016632 2021-07-31T02:05:08.249 INFO:tasks.workunit.client.0.smithi045.stdout:op 9 completed, throughput=5.13MB/sec 2021-07-31T02:05:09.249 INFO:tasks.workunit.client.0.smithi045.stdout: 3: throughput=3.45MB/sec pending data=0 2021-07-31T02:05:09.250 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj--Lw5efBB6KSazLb off=7867401 len=186618 2021-07-31T02:05:09.253 INFO:tasks.workunit.client.0.smithi045.stdout:op 10 completed, throughput=3.51MB/sec 2021-07-31T02:05:09.253 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-XZrNj_1Esm1w59u off=4821219 len=1668267 2021-07-31T02:05:09.264 INFO:tasks.workunit.client.0.smithi045.stdout:op 11 completed, throughput=4.01MB/sec 2021-07-31T02:05:09.264 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Ibm4gBJaGNikh-a off=441188 len=94010 2021-07-31T02:05:09.265 INFO:tasks.workunit.client.0.smithi045.stdout:op 12 completed, throughput=4.04MB/sec 2021-07-31T02:05:09.265 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-SMlsaKO2D5f9IiZ off=5957377 len=1228173 2021-07-31T02:05:09.271 INFO:tasks.workunit.client.0.smithi045.stdout:op 13 completed, throughput=4.41MB/sec 2021-07-31T02:05:09.271 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-PTRYDA8sBf_TGfl off=17445788 len=370340 2021-07-31T02:05:09.275 INFO:tasks.workunit.client.0.smithi045.stdout:op 14 completed, throughput=4.52MB/sec 2021-07-31T02:05:09.275 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-gIiCQOKKZBgjc26 off=5446602 len=1210283 2021-07-31T02:05:09.282 INFO:tasks.workunit.client.0.smithi045.stdout:op 15 completed, throughput=4.88MB/sec 2021-07-31T02:05:09.283 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-IT994w-zzTq5TSD off=283896 len=1067859 2021-07-31T02:05:09.287 INFO:tasks.workunit.client.0.smithi045.stdout:op 16 completed, throughput=5.21MB/sec 2021-07-31T02:05:09.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:09 smithi154 conmon[34553]: cluster 2021-07-31T02:05:08.044379+0000 mgr.x (mgr.24542) 766 : cluster [DBG] pgmap v1409: 9 pgs: 9 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:05:09.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:09 smithi154 conmon[34553]: 2021-07-31T02:05:09.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:09 smithi045 conmon[39718]: cluster 2021-07-31T02:05:08.044379+0000 mgr.x 2021-07-31T02:05:09.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:09 smithi045 conmon[39718]: (mgr.24542) 766 : cluster [DBG] pgmap v1409: 9 pgs: 9 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:05:09.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:09 smithi045 conmon[33655]: cluster 2021-07-31T02:05:08.044379+0000 mgr.x (mgr.24542) 766 2021-07-31T02:05:09.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:09 smithi045 conmon[33655]: : cluster [DBG] pgmap v1409: 9 pgs: 9 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:05:10.287 INFO:tasks.workunit.client.0.smithi045.stdout: 4: throughput=3.93MB/sec pending data=0 2021-07-31T02:05:10.288 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-nQmqik6rSb-8yGB off=13687837 len=1936922 2021-07-31T02:05:10.289 INFO:tasks.workunit.client.0.smithi045.stdout:op 17 completed, throughput=4.38MB/sec 2021-07-31T02:05:10.290 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-esulQnO1wUazAxE off=3487114 len=271723 2021-07-31T02:05:10.293 INFO:tasks.workunit.client.0.smithi045.stdout:op 18 completed, throughput=4.44MB/sec 2021-07-31T02:05:10.293 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-nhHkTt3iy9KmeP9 off=8918049 len=1674525 2021-07-31T02:05:10.301 INFO:tasks.workunit.client.0.smithi045.stdout:op 19 completed, throughput=4.82MB/sec 2021-07-31T02:05:10.302 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-NGHnigaIxY2C929 off=11783731 len=1940413 2021-07-31T02:05:10.311 INFO:tasks.workunit.client.0.smithi045.stdout:op 20 completed, throughput=5.26MB/sec 2021-07-31T02:05:10.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:10 smithi154 conmon[34553]: cluster 2021-07-31T02:05:10.044839+0000 mgr.x (mgr.24542) 767 : cluster 2021-07-31T02:05:10.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:10 smithi154 conmon[34553]: [DBG] pgmap v1410: 9 pgs: 9 active+clean; 127 GiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 488 KiB/s rd, 1.2 MiB/s wr, 859 op/s 2021-07-31T02:05:10.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:10 smithi045 conmon[33655]: cluster 2021-07-31T02:05:10 2021-07-31T02:05:10.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:10 smithi045 conmon[33655]: .044839+0000 mgr.x (mgr.24542) 767 : cluster [DBG] pgmap v1410: 9 pgs: 9 active+clean; 127 GiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 488 KiB/s rd, 1.2 MiB/s wr, 859 op/s 2021-07-31T02:05:10.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:10 smithi045 conmon[39718]: cluster 2021-07-31T02:05:10.044839+0000 mgr.x ( 2021-07-31T02:05:10.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:10 smithi045 conmon[39718]: mgr.24542) 767 : cluster [DBG] pgmap v1410: 9 pgs: 9 active+clean; 127 GiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 488 KiB/s rd, 1.2 MiB/s wr, 859 op/s 2021-07-31T02:05:11.311 INFO:tasks.workunit.client.0.smithi045.stdout: 5: throughput=4.23MB/sec pending data=0 2021-07-31T02:05:11.312 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Pxn2NS5Fae3kPbt off=928466 len=1459192 2021-07-31T02:05:11.313 INFO:tasks.workunit.client.0.smithi045.stdout:op 21 completed, throughput=4.5MB/sec 2021-07-31T02:05:11.313 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ilII2E07xCdpyBi off=3496078 len=941299 2021-07-31T02:05:11.316 INFO:tasks.workunit.client.0.smithi045.stdout:op 22 completed, throughput=4.67MB/sec 2021-07-31T02:05:11.316 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-HtNNVENjgABP48Q off=7931019 len=2065540 2021-07-31T02:05:11.327 INFO:tasks.workunit.client.0.smithi045.stdout:op 23 completed, throughput=5.05MB/sec 2021-07-31T02:05:12.328 INFO:tasks.workunit.client.0.smithi045.stdout: 6: throughput=4.22MB/sec pending data=0 2021-07-31T02:05:12.328 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-QHTxRuEp4Vg_uAh off=414031 len=1519598 2021-07-31T02:05:12.330 INFO:tasks.workunit.client.0.smithi045.stdout:op 24 completed, throughput=4.46MB/sec 2021-07-31T02:05:12.330 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-hATmpHv74V9zpA2 off=245383 len=947099 2021-07-31T02:05:12.331 INFO:tasks.workunit.client.0.smithi045.stdout:op 25 completed, throughput=4.61MB/sec 2021-07-31T02:05:12.331 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj--cpeJAQotKs8aYH off=1812165 len=895225 2021-07-31T02:05:12.336 INFO:tasks.workunit.client.0.smithi045.stdout:op 26 completed, throughput=4.74MB/sec 2021-07-31T02:05:12.336 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-FuuMR3ZiZ6jgC2W off=10864412 len=540880 2021-07-31T02:05:12.340 INFO:tasks.workunit.client.0.smithi045.stdout:op 27 completed, throughput=4.82MB/sec 2021-07-31T02:05:12.340 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-x0SGiRhSgekBeo3 off=6170613 len=1707902 2021-07-31T02:05:12.350 INFO:tasks.workunit.client.0.smithi045.stdout:op 28 completed, throughput=5.08MB/sec 2021-07-31T02:05:13.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:12 smithi154 conmon[34553]: cluster 2021-07-31T02:05:12.045407+0000 mgr.x (mgr.24542) 768 : cluster [DBG] pgmap v1411: 9 pgs: 9 active+clean; 127 GiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 485 KiB/s rd, 1.2 MiB/s wr, 853 op/s 2021-07-31T02:05:13.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:12 smithi045 conmon[33655]: cluster 2021-07-31T02:05:12.045407+0000 mgr.x (mgr.24542) 768 : cluster 2021-07-31T02:05:13.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:12 smithi045 conmon[33655]: [DBG] pgmap v1411: 9 pgs: 9 active+clean; 127 GiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 485 KiB/s rd, 1.2 MiB/s wr, 853 op/s 2021-07-31T02:05:13.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:12 smithi045 conmon[39718]: cluster 2021-07-31T02:05:12.045407+0000 mgr.x (mgr.24542) 768 : cluster 2021-07-31T02:05:13.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:12 smithi045 conmon[39718]: [DBG] pgmap v1411: 9 pgs: 9 active+clean; 127 GiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 485 KiB/s rd, 1.2 MiB/s wr, 853 op/s 2021-07-31T02:05:13.350 INFO:tasks.workunit.client.0.smithi045.stdout: 7: throughput=4.37MB/sec pending data=0 2021-07-31T02:05:13.350 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-VQqtBX05ZWB81kG off=1557256 len=1990626 2021-07-31T02:05:13.359 INFO:tasks.workunit.client.0.smithi045.stdout:op 29 completed, throughput=4.63MB/sec 2021-07-31T02:05:13.359 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-eIGqqhF9eT-OsLt off=19726352 len=1780844 2021-07-31T02:05:13.360 INFO:tasks.workunit.client.0.smithi045.stdout:op 30 completed, throughput=4.87MB/sec 2021-07-31T02:05:13.360 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-GuiAlH5fgIXjG1V off=2959862 len=985062 2021-07-31T02:05:13.366 INFO:tasks.workunit.client.0.smithi045.stdout:op 31 completed, throughput=4.99MB/sec 2021-07-31T02:05:13.366 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-cpYWXlOWYOLVzlB off=7201626 len=679482 2021-07-31T02:05:13.370 INFO:tasks.workunit.client.0.smithi045.stdout:op 32 completed, throughput=5.08MB/sec 2021-07-31T02:05:14.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:05:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:05:13.538Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:05:14.027 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:05:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:05:13.538Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:05:14.027 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:05:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:05:13.538Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:05:14.370 INFO:tasks.workunit.client.0.smithi045.stdout: 8: throughput=4.46MB/sec pending data=0 2021-07-31T02:05:14.371 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-cnhZUIfvWJo7PAE off=104136 len=845909 2021-07-31T02:05:14.375 INFO:tasks.workunit.client.0.smithi045.stdout:op 33 completed, throughput=4.55MB/sec 2021-07-31T02:05:14.376 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-yhoiudgJiJO2VcY off=3676579 len=1961077 2021-07-31T02:05:14.386 INFO:tasks.workunit.client.0.smithi045.stdout:op 34 completed, throughput=4.78MB/sec 2021-07-31T02:05:14.386 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-fMFBwp5B6lZR0X9 off=5745990 len=109873 2021-07-31T02:05:14.387 INFO:tasks.workunit.client.0.smithi045.stdout:op 35 completed, throughput=4.79MB/sec 2021-07-31T02:05:14.387 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-KPO0tLG_gAnWhy- off=10747892 len=1332089 2021-07-31T02:05:14.389 INFO:tasks.workunit.client.0.smithi045.stdout:op 36 completed, throughput=4.94MB/sec 2021-07-31T02:05:14.389 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-l18BJbeEoZK_Use off=324432 len=967210 2021-07-31T02:05:14.390 INFO:tasks.workunit.client.0.smithi045.stdout:op 37 completed, throughput=5.05MB/sec 2021-07-31T02:05:15.391 INFO:tasks.workunit.client.0.smithi045.stdout: 9: throughput=4.5MB/sec pending data=0 2021-07-31T02:05:15.391 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-e4h65FN5M1L52dP off=3003267 len=680605 2021-07-31T02:05:15.397 INFO:tasks.workunit.client.0.smithi045.stdout:op 38 completed, throughput=4.57MB/sec 2021-07-31T02:05:15.397 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-opal_2WFGch0ekW off=13853173 len=1516952 2021-07-31T02:05:15.399 INFO:tasks.workunit.client.0.smithi045.stdout:op 39 completed, throughput=4.73MB/sec 2021-07-31T02:05:15.399 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-p51IwWfz9DYdT2t off=2388604 len=445469 2021-07-31T02:05:15.400 INFO:tasks.workunit.client.0.smithi045.stdout:op 40 completed, throughput=4.77MB/sec 2021-07-31T02:05:15.400 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-gqpQMw5eOHEE3X8 off=1034793 len=1826296 2021-07-31T02:05:15.402 INFO:tasks.workunit.client.0.smithi045.stdout:op 41 completed, throughput=4.96MB/sec 2021-07-31T02:05:15.403 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-PYQo-SxRmLLk02S off=355566 len=408683 2021-07-31T02:05:15.406 INFO:tasks.workunit.client.0.smithi045.stdout:op 42 completed, throughput=5MB/sec 2021-07-31T02:05:15.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:15 smithi154 conmon[34553]: cluster 2021-07-31T02:05:14.045903+0000 mgr.x (mgr.24542) 769 : cluster [DBG] pgmap v1412: 9 pgs: 9 active+clean; 127 GiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 1.2 MiB/s rd, 2.7 MiB/s wr, 855 op/s 2021-07-31T02:05:15.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:15 smithi045 conmon[39718]: cluster 2021-07-31T02:05:14.045903+0000 mgr.x (mgr.24542) 769 : 2021-07-31T02:05:15.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:15 smithi045 conmon[39718]: cluster [DBG] pgmap v1412: 9 pgs: 9 active+clean; 127 GiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 1.2 MiB/s rd, 2.7 MiB/s wr, 855 op/s 2021-07-31T02:05:15.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:15 smithi045 conmon[33655]: cluster 2021-07-31T02:05:14.045903+0000 mgr.x (mgr. 2021-07-31T02:05:15.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:15 smithi045 conmon[33655]: 24542) 769 : cluster [DBG] pgmap v1412: 9 pgs: 9 active+clean; 127 GiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 1.2 MiB/s rd, 2.7 MiB/s wr, 855 op/s 2021-07-31T02:05:16.407 INFO:tasks.workunit.client.0.smithi045.stdout: 10: throughput=4.51MB/sec pending data=0 2021-07-31T02:05:16.407 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-1yXmHfPXrdEa0yX off=6154425 len=1654759 2021-07-31T02:05:16.416 INFO:tasks.workunit.client.0.smithi045.stdout:op 43 completed, throughput=4.66MB/sec 2021-07-31T02:05:16.417 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-7xHIHqet0b2eF1z off=23430651 len=505118 2021-07-31T02:05:16.419 INFO:tasks.workunit.client.0.smithi045.stdout:op 44 completed, throughput=4.71MB/sec 2021-07-31T02:05:16.420 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-eJD5LTJDweURF46 off=15345761 len=1717337 2021-07-31T02:05:16.427 INFO:tasks.workunit.client.0.smithi045.stdout:op 45 completed, throughput=4.86MB/sec 2021-07-31T02:05:16.428 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-WMqlErxSUMOpC-B off=2328119 len=652131 2021-07-31T02:05:16.428 INFO:tasks.workunit.client.0.smithi045.stdout:op 46 completed, throughput=4.92MB/sec 2021-07-31T02:05:16.429 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-o8k5LHpmJwXbbx6 off=2971784 len=1146020 2021-07-31T02:05:16.429 INFO:tasks.workunit.client.0.smithi045.stdout:op 47 completed, throughput=5.03MB/sec 2021-07-31T02:05:16.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:16 smithi154 conmon[34553]: cluster 2021-07-31T02:05:16.046295+0000 mgr.x (mgr.24542) 770 : cluster [DBG] 2021-07-31T02:05:16.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:16 smithi154 conmon[34553]: pgmap v1413: 9 pgs: 9 active+clean; 127 GiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 1.2 MiB/s rd, 2.7 MiB/s wr, 855 op/s 2021-07-31T02:05:16.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:16 smithi045 conmon[39718]: cluster 2021-07-31T02:05:16.046295+0000 mgr.x ( 2021-07-31T02:05:16.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:16 smithi045 conmon[39718]: mgr.24542) 770 : cluster [DBG] pgmap v1413: 9 pgs: 9 active+clean; 127 GiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 1.2 MiB/s rd, 2.7 MiB/s wr, 855 op/s 2021-07-31T02:05:16.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:16 smithi045 conmon[33655]: cluster 2021-07-31T02:05:16.046295+0000 mgr.x (mgr. 2021-07-31T02:05:16.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:16 smithi045 conmon[33655]: 24542) 770 : cluster [DBG] pgmap v1413: 9 pgs: 9 active+clean; 127 GiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 1.2 MiB/s rd, 2.7 MiB/s wr, 855 op/s 2021-07-31T02:05:17.429 INFO:tasks.workunit.client.0.smithi045.stdout: 11: throughput=4.58MB/sec pending data=0 2021-07-31T02:05:17.430 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Tl4-HNSsq4CmbeY off=272555 len=857384 2021-07-31T02:05:17.431 INFO:tasks.workunit.client.0.smithi045.stdout:op 48 completed, throughput=4.65MB/sec 2021-07-31T02:05:17.432 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-7vxKQAVSRtPRkO0 off=2015297 len=1328497 2021-07-31T02:05:17.437 INFO:tasks.workunit.client.0.smithi045.stdout:op 49 completed, throughput=4.76MB/sec 2021-07-31T02:05:17.437 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-h_zsptbLZnYuWCz off=144502 len=386467 2021-07-31T02:05:17.438 INFO:tasks.workunit.client.0.smithi045.stdout:op 50 completed, throughput=4.8MB/sec 2021-07-31T02:05:17.439 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-r-Hq__JLsIuHx_Q off=2530058 len=196818 2021-07-31T02:05:17.441 INFO:tasks.workunit.client.0.smithi045.stdout:op 51 completed, throughput=4.81MB/sec 2021-07-31T02:05:17.441 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-4bgWnX42oUWvtyl off=9863622 len=1326617 2021-07-31T02:05:17.448 INFO:tasks.workunit.client.0.smithi045.stdout:op 52 completed, throughput=4.92MB/sec 2021-07-31T02:05:17.448 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-hpEvZkq1d5iKLWd off=930595 len=1882295 2021-07-31T02:05:17.454 INFO:tasks.workunit.client.0.smithi045.stdout:op 53 completed, throughput=5.08MB/sec 2021-07-31T02:05:18.454 INFO:tasks.workunit.client.0.smithi045.stdout: 12: throughput=4.66MB/sec pending data=0 2021-07-31T02:05:18.455 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-z4k0v7ovyDPTQX3 off=10320126 len=2056615 2021-07-31T02:05:18.457 INFO:tasks.workunit.client.0.smithi045.stdout:op 54 completed, throughput=4.82MB/sec 2021-07-31T02:05:18.457 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-lO7xLuGK2qAjJY4 off=35105 len=112202 2021-07-31T02:05:18.458 INFO:tasks.workunit.client.0.smithi045.stdout:op 55 completed, throughput=4.83MB/sec 2021-07-31T02:05:18.458 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-FQ9nsBZg3YvXEFt off=6371964 len=1144693 2021-07-31T02:05:18.459 INFO:tasks.workunit.client.0.smithi045.stdout:op 56 completed, throughput=4.92MB/sec 2021-07-31T02:05:18.459 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-8ZqdZrGn0i_LtVo off=3056216 len=2024845 2021-07-31T02:05:18.465 INFO:tasks.workunit.client.0.smithi045.stdout:op 57 completed, throughput=5.08MB/sec 2021-07-31T02:05:19.466 INFO:tasks.workunit.client.0.smithi045.stdout: 13: throughput=4.69MB/sec pending data=0 2021-07-31T02:05:19.466 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-FCjIEwj2VLQVGyl off=744295 len=311584 2021-07-31T02:05:19.467 INFO:tasks.workunit.client.0.smithi045.stdout:op 58 completed, throughput=4.71MB/sec 2021-07-31T02:05:19.467 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Lu581urqq3I1yeq off=1601664 len=1829662 2021-07-31T02:05:19.475 INFO:tasks.workunit.client.0.smithi045.stdout:op 59 completed, throughput=4.84MB/sec 2021-07-31T02:05:19.475 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ycVj3lOHgPKXZuh off=8168799 len=1305660 2021-07-31T02:05:19.482 INFO:tasks.workunit.client.0.smithi045.stdout:op 60 completed, throughput=4.93MB/sec 2021-07-31T02:05:19.482 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-344JD2CQ8-PIKwN off=16939437 len=268920 2021-07-31T02:05:19.483 INFO:tasks.workunit.client.0.smithi045.stdout:op 61 completed, throughput=4.95MB/sec 2021-07-31T02:05:19.483 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-uJtx-1zKAal7ji6 off=4095276 len=160886 2021-07-31T02:05:19.485 INFO:tasks.workunit.client.0.smithi045.stdout:op 62 completed, throughput=4.96MB/sec 2021-07-31T02:05:19.485 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-AauFYHcB9yb0z3J off=5828917 len=1261693 2021-07-31T02:05:19.487 INFO:tasks.workunit.client.0.smithi045.stdout:op 63 completed, throughput=5.05MB/sec 2021-07-31T02:05:19.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:19 smithi154 conmon[34553]: cluster 2021-07-31T02:05:18.046839+0000 mgr.x (mgr.24542) 771 : cluster [DBG] pgmap v1414: 9 pgs: 9 active+clean; 127 GiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 1.2 MiB/s rd, 2.7 MiB/s wr, 855 op/s 2021-07-31T02:05:19.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:19 smithi045 conmon[39718]: cluster 2021-07-31T02:05:18.046839+0000 mgr.x (mgr.24542 2021-07-31T02:05:19.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:19 smithi045 conmon[39718]: ) 771 : cluster [DBG] pgmap v1414: 9 pgs: 9 active+clean; 127 GiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 1.2 MiB/s rd, 2.7 MiB/s wr, 855 op/s 2021-07-31T02:05:19.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:19 smithi045 conmon[33655]: cluster 2021-07-31T02:05:18.046839+0000 mgr.x (mgr.24542) 771 2021-07-31T02:05:19.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:19 smithi045 conmon[33655]: : cluster [DBG] pgmap v1414: 9 pgs: 9 active+clean; 127 GiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 1.2 MiB/s rd, 2.7 MiB/s wr, 855 op/s 2021-07-31T02:05:20.488 INFO:tasks.workunit.client.0.smithi045.stdout: 14: throughput=4.7MB/sec pending data=0 2021-07-31T02:05:20.488 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Cw8RTzwbzd3CKkB off=655863 len=2096205 2021-07-31T02:05:20.500 INFO:tasks.workunit.client.0.smithi045.stdout:op 64 completed, throughput=4.84MB/sec 2021-07-31T02:05:20.500 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-PDcORlt_KJzyw6a off=11058616 len=339985 2021-07-31T02:05:20.501 INFO:tasks.workunit.client.0.smithi045.stdout:op 65 completed, throughput=4.86MB/sec 2021-07-31T02:05:20.501 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-rYMKugMPEHbSu-f off=7521556 len=234106 2021-07-31T02:05:20.502 INFO:tasks.workunit.client.0.smithi045.stdout:op 66 completed, throughput=4.87MB/sec 2021-07-31T02:05:20.502 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-aQpKvP81NC6hlyK off=4339750 len=915202 2021-07-31T02:05:20.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:20 smithi154 conmon[34553]: cluster 2021-07-31T02:05:20.047342+0000 mgr.x (mgr.24542) 772 : cluster [DBG] pgmap v1415: 9 pgs: 9 active+clean; 127 GiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 2.5 MiB/s rd, 3.6 MiB/s wr, 857 op/s 2021-07-31T02:05:20.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:20 smithi154 conmon[34553]: 2021-07-31T02:05:20.507 INFO:tasks.workunit.client.0.smithi045.stdout:op 67 completed, throughput=4.93MB/sec 2021-07-31T02:05:20.507 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-iflzABRyKWUaeVR off=416965 len=1630828 2021-07-31T02:05:20.511 INFO:tasks.workunit.client.0.smithi045.stdout:op 68 completed, throughput=5.04MB/sec 2021-07-31T02:05:20.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:20 smithi045 conmon[33655]: cluster 2021-07-31T02:05:20.047342+0000 2021-07-31T02:05:20.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:20 smithi045 conmon[33655]: mgr.x (mgr.24542) 772 : cluster [DBG] pgmap v1415: 9 pgs: 9 active+clean; 127 GiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 2.5 MiB/s rd, 3.6 MiB/s wr, 857 op/s 2021-07-31T02:05:20.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:20 smithi045 conmon[39718]: cluster 2021-07-31T02:05:20.047342+0000 mgr.x 2021-07-31T02:05:20.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:20 smithi045 conmon[39718]: (mgr.24542) 772 : cluster [DBG] pgmap v1415: 9 pgs: 9 active+clean; 127 GiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 2.5 MiB/s rd, 3.6 MiB/s wr, 857 op/s 2021-07-31T02:05:21.511 INFO:tasks.workunit.client.0.smithi045.stdout: 15: throughput=4.71MB/sec pending data=0 2021-07-31T02:05:21.511 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-G5N7loPZk4C89Y7 off=695852 len=2027434 2021-07-31T02:05:21.519 INFO:tasks.workunit.client.0.smithi045.stdout:op 69 completed, throughput=4.83MB/sec 2021-07-31T02:05:21.520 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-srvN0I5MF1mBkBb off=8615410 len=1107413 2021-07-31T02:05:21.522 INFO:tasks.workunit.client.0.smithi045.stdout:op 70 completed, throughput=4.9MB/sec 2021-07-31T02:05:21.522 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-XTgvvlVNRHSVhyU off=458739 len=880467 2021-07-31T02:05:21.527 INFO:tasks.workunit.client.0.smithi045.stdout:op 71 completed, throughput=4.96MB/sec 2021-07-31T02:05:21.527 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-J4uRqKaYfrgOLl7 off=2664600 len=453373 2021-07-31T02:05:21.528 INFO:tasks.workunit.client.0.smithi045.stdout:op 72 completed, throughput=4.98MB/sec 2021-07-31T02:05:21.528 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Mli3Jc0dXcMvuvv off=769113 len=600343 2021-07-31T02:05:21.529 INFO:tasks.workunit.client.0.smithi045.stdout:op 73 completed, throughput=5.02MB/sec 2021-07-31T02:05:22.530 INFO:tasks.workunit.client.0.smithi045.stdout: 16: throughput=4.71MB/sec pending data=0 2021-07-31T02:05:22.530 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-POW-eGj5jRrg-F2 off=9178153 len=1784701 2021-07-31T02:05:22.532 INFO:tasks.workunit.client.0.smithi045.stdout:op 74 completed, throughput=4.82MB/sec 2021-07-31T02:05:22.532 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-hTkLpyql--YukR5 off=0 len=1064698 2021-07-31T02:05:22.538 INFO:tasks.workunit.client.0.smithi045.stdout:op 75 completed, throughput=4.88MB/sec 2021-07-31T02:05:22.539 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-XnnIGNjU-GbKQRi off=15765072 len=2049729 2021-07-31T02:05:22.551 INFO:tasks.workunit.client.0.smithi045.stdout:op 76 completed, throughput=4.99MB/sec 2021-07-31T02:05:22.551 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-nGO04_mpajDRKk7 off=7941060 len=1254714 2021-07-31T02:05:22.553 INFO:tasks.workunit.client.0.smithi045.stdout:op 77 completed, throughput=5.06MB/sec 2021-07-31T02:05:23.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:22 smithi154 conmon[34553]: cluster 2021-07-31T02:05:22.047764+0000 mgr.x (mgr.24542) 773 : cluster [DBG] pgmap v1416: 9 pgs: 9 active+clean; 127 GiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.3 MiB/s wr, 3 op/s 2021-07-31T02:05:23.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:22 smithi045 conmon[33655]: cluster 2021-07-31T02:05:22.047764+0000 mgr.x (mgr.24542) 773 : cluster [DBG] 2021-07-31T02:05:23.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:22 smithi045 conmon[33655]: pgmap v1416: 9 pgs: 9 active+clean; 127 GiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.3 MiB/s wr, 3 op/s 2021-07-31T02:05:23.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:22 smithi045 conmon[39718]: cluster 2021-07-31T02:05:22.047764+0000 mgr.x (mgr 2021-07-31T02:05:23.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:22 smithi045 conmon[39718]: .24542) 773 : cluster [DBG] pgmap v1416: 9 pgs: 9 active+clean; 127 GiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.3 MiB/s wr, 3 op/s 2021-07-31T02:05:23.554 INFO:tasks.workunit.client.0.smithi045.stdout: 17: throughput=4.77MB/sec pending data=0 2021-07-31T02:05:23.554 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-fcZeBDcByzELhaF off=2475699 len=2031332 2021-07-31T02:05:23.566 INFO:tasks.workunit.client.0.smithi045.stdout:op 78 completed, throughput=4.88MB/sec 2021-07-31T02:05:23.566 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-uBsRIpX040hgNdL off=691391 len=1536323 2021-07-31T02:05:23.567 INFO:tasks.workunit.client.0.smithi045.stdout:op 79 completed, throughput=4.96MB/sec 2021-07-31T02:05:23.568 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-t__nBtxIrdv3yOr off=2209124 len=831561 2021-07-31T02:05:23.571 INFO:tasks.workunit.client.0.smithi045.stdout:op 80 completed, throughput=5.01MB/sec 2021-07-31T02:05:24.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:05:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:05:23.538Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:05:24.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:05:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:05:23.538Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:05:24.027 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:05:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:05:23.538Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:05:24.570 INFO:tasks.workunit.client.0.smithi045.stdout: 18: throughput=4.74MB/sec pending data=0 2021-07-31T02:05:24.570 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-fhY46r5j-cr-1L4 off=3123666 len=1751575 2021-07-31T02:05:24.573 INFO:tasks.workunit.client.0.smithi045.stdout:op 81 completed, throughput=4.83MB/sec 2021-07-31T02:05:24.573 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Utb60ZhKy21Hf8i off=2841396 len=1264708 2021-07-31T02:05:24.579 INFO:tasks.workunit.client.0.smithi045.stdout:op 82 completed, throughput=4.89MB/sec 2021-07-31T02:05:24.580 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-9pkFMh9r4UsDbcX off=15590310 len=1060633 2021-07-31T02:05:24.584 INFO:tasks.workunit.client.0.smithi045.stdout:op 83 completed, throughput=4.94MB/sec 2021-07-31T02:05:24.585 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-m0jkpnIZaAi5zY_ off=2804741 len=329208 2021-07-31T02:05:24.588 INFO:tasks.workunit.client.0.smithi045.stdout:op 84 completed, throughput=4.96MB/sec 2021-07-31T02:05:24.588 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-W_RGLRcQ-8SKzLc off=1569828 len=836303 2021-07-31T02:05:24.589 INFO:tasks.workunit.client.0.smithi045.stdout:op 85 completed, throughput=5MB/sec 2021-07-31T02:05:25.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:25 smithi154 conmon[34553]: cluster 2021-07-31T02:05:24.048222+0000 mgr.x (mgr.24542) 774 : cluster [DBG] pgmap v1417: 9 pgs: 9 active+clean; 127 GiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 2.9 MiB/s rd, 3.5 MiB/s wr, 5 op/s 2021-07-31T02:05:25.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:25 smithi154 conmon[34553]: 2021-07-31T02:05:25.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:25 smithi045 conmon[33655]: cluster 2021-07-31T02:05:24.048222+0000 mgr.x (mgr.24542) 774 2021-07-31T02:05:25.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:25 smithi045 conmon[33655]: : cluster [DBG] pgmap v1417: 9 pgs: 9 active+clean; 127 GiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 2.9 MiB/s rd, 3.5 MiB/s wr, 5 op/s 2021-07-31T02:05:25.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:25 smithi045 conmon[39718]: cluster 2021-07-31T02:05:24 2021-07-31T02:05:25.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:25 smithi045 conmon[39718]: .048222+0000 mgr.x (mgr.24542) 774 : cluster [DBG] pgmap v1417: 9 pgs: 9 active+clean; 127 GiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 2.9 MiB/s rd, 3.5 MiB/s wr, 5 op/s 2021-07-31T02:05:25.589 INFO:tasks.workunit.client.0.smithi045.stdout: 19: throughput=4.75MB/sec pending data=0 2021-07-31T02:05:25.589 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-3XRVTENl-eGF8wj off=4194808 len=243967 2021-07-31T02:05:25.591 INFO:tasks.workunit.client.0.smithi045.stdout:op 86 completed, throughput=4.76MB/sec 2021-07-31T02:05:25.591 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-t04dh74wX1p64F4 off=8102168 len=633824 2021-07-31T02:05:25.593 INFO:tasks.workunit.client.0.smithi045.stdout:op 87 completed, throughput=4.79MB/sec 2021-07-31T02:05:25.593 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-YrftZTKmwWoSEhm off=568950 len=1732289 2021-07-31T02:05:25.596 INFO:tasks.workunit.client.0.smithi045.stdout:op 88 completed, throughput=4.87MB/sec 2021-07-31T02:05:25.596 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-RguJYZ4s2hb3ROE off=9747213 len=848945 2021-07-31T02:05:25.601 INFO:tasks.workunit.client.0.smithi045.stdout:op 89 completed, throughput=4.91MB/sec 2021-07-31T02:05:25.601 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-csP3bTcx86hYOQW off=562596 len=626846 2021-07-31T02:05:25.605 INFO:tasks.workunit.client.0.smithi045.stdout:op 90 completed, throughput=4.94MB/sec 2021-07-31T02:05:25.606 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-drFNCrsMOOsD0z0 off=143565 len=1723095 2021-07-31T02:05:25.612 INFO:tasks.workunit.client.0.smithi045.stdout:op 91 completed, throughput=5.03MB/sec 2021-07-31T02:05:26.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:26 smithi154 conmon[34553]: cluster 2021-07-31T02:05:26.048610+0000 mgr.x (mgr.24542) 775 : cluster [DBG] pgmap v1418: 9 pgs: 9 active+clean; 127 GiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 3 op/s 2021-07-31T02:05:26.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:26 smithi045 conmon[39718]: cluster 2021-07-31T02:05:26.048610+0000 mgr.x ( 2021-07-31T02:05:26.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:26 smithi045 conmon[39718]: mgr.24542) 775 : cluster [DBG] pgmap v1418: 9 pgs: 9 active+clean; 127 GiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 3 op/s 2021-07-31T02:05:26.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:26 smithi045 conmon[33655]: cluster 2021-07-31T02:05:26.048610+0000 mgr.x (mgr.24542) 775 : cluster 2021-07-31T02:05:26.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:26 smithi045 conmon[33655]: [DBG] pgmap v1418: 9 pgs: 9 active+clean; 127 GiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 3 op/s 2021-07-31T02:05:26.612 INFO:tasks.workunit.client.0.smithi045.stdout: 20: throughput=4.78MB/sec pending data=0 2021-07-31T02:05:26.613 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-u8S-BGpR1XW_INZ off=3596659 len=347257 2021-07-31T02:05:26.616 INFO:tasks.workunit.client.0.smithi045.stdout:op 92 completed, throughput=4.79MB/sec 2021-07-31T02:05:26.617 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-XmHHAQqFJFYjWcu off=10973066 len=1343837 2021-07-31T02:05:26.619 INFO:tasks.workunit.client.0.smithi045.stdout:op 93 completed, throughput=4.86MB/sec 2021-07-31T02:05:26.619 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-oAoXbBUZndaVYB1 off=5579001 len=2012992 2021-07-31T02:05:26.629 INFO:tasks.workunit.client.0.smithi045.stdout:op 94 completed, throughput=4.95MB/sec 2021-07-31T02:05:26.630 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-aDUkonTXmCDeBwE off=4798711 len=1937322 2021-07-31T02:05:26.635 INFO:tasks.workunit.client.0.smithi045.stdout:op 95 completed, throughput=5.04MB/sec 2021-07-31T02:05:27.643 INFO:tasks.workunit.client.0.smithi045.stdout: 21: throughput=4.8MB/sec pending data=0 2021-07-31T02:05:27.644 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-hF5mQD_keLcCXv4 off=4754101 len=326406 2021-07-31T02:05:27.644 INFO:tasks.workunit.client.0.smithi045.stdout:op 96 completed, throughput=4.82MB/sec 2021-07-31T02:05:27.644 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Cfmj-ael7gYlpzK off=4228265 len=924285 2021-07-31T02:05:27.644 INFO:tasks.workunit.client.0.smithi045.stdout:op 97 completed, throughput=4.86MB/sec 2021-07-31T02:05:27.644 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-EcYBx9BUpVM-9tE off=1464509 len=1593209 2021-07-31T02:05:27.646 INFO:tasks.workunit.client.0.smithi045.stdout:op 98 completed, throughput=4.93MB/sec 2021-07-31T02:05:27.646 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-dT6iqAeAprOs6ml off=1314904 len=1950080 2021-07-31T02:05:27.654 INFO:tasks.workunit.client.0.smithi045.stdout:op 99 completed, throughput=5.01MB/sec 2021-07-31T02:05:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:28 smithi154 conmon[34553]: audit 2021-07-31T02:05:28.000858+0000 mon.b (mon.2) 158 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:05:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:28 smithi154 conmon[34553]: 2021-07-31T02:05:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:28 smithi154 conmon[34553]: audit 2021-07-31T02:05:28.001435+0000 mon.a (mon.0) 1849 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:05:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:28 smithi154 conmon[34553]: audit 2021-07-31T02:05:28.019320+0000 mon.b (mon.2) 159 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:05:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:28 smithi154 conmon[34553]: audit 2021-07-31T02:05:28.020038+0000 mon.a (mon.0) 1850 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:05:28.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:28 smithi045 conmon[39718]: audit 2021-07-31T02:05:28.000858+0000 mon.b ( 2021-07-31T02:05:28.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:28 smithi045 conmon[39718]: mon.2) 158 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:05:28.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:28 smithi045 conmon[39718]: audit 2021-07-31T02:05:28.001435+0000 2021-07-31T02:05:28.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:28 smithi045 conmon[39718]: mon.a (mon.0) 1849 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:05:28.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:28 smithi045 conmon[39718]: audit 2021-07 2021-07-31T02:05:28.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:28 smithi045 conmon[39718]: -31T02:05:28.019320+0000 2021-07-31T02:05:28.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:28 smithi045 conmon[39718]: mon.b (mon.2) 159 : audit [INF] 2021-07-31T02:05:28.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:28 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:05:28.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:28 smithi045 conmon[39718]: audit 2021-07-31T02:05:28.020038+0000 mon.a (mon.0) 1850 : audit [INF] 2021-07-31T02:05:28.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:28 smithi045 conmon[39718]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:05:28.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:28 smithi045 conmon[33655]: audit 2021-07-31T02:05:28.000858+0000 mon.b (mon.2) 158 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:05:28.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:28 smithi045 conmon[33655]: audit 2021-07-31T02:05:28.001435+0000 mon.a (mon.0) 1849 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:05:28.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:28 smithi045 conmon[33655]: audit 2021-07-31T02:05:28.019320+0000 mon.b (mon. 2021-07-31T02:05:28.535 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:28 smithi045 conmon[33655]: 2) 159 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:05:28.535 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:28 smithi045 conmon[33655]: audit 2021-07 2021-07-31T02:05:28.535 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:28 smithi045 conmon[33655]: -31T02:05:28. 2021-07-31T02:05:28.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:28 smithi045 conmon[33655]: 020038+0000 mon.a (mon.0) 2021-07-31T02:05:28.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:28 smithi045 conmon[33655]: 1850 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:05:28.654 INFO:tasks.workunit.client.0.smithi045.stdout: 22: throughput=4.79MB/sec pending data=0 2021-07-31T02:05:28.655 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-izH_iPEyh7cJDjL off=2931852 len=1009628 2021-07-31T02:05:28.657 INFO:tasks.workunit.client.0.smithi045.stdout:op 100 completed, throughput=4.83MB/sec 2021-07-31T02:05:28.657 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj--3eVZYodROdj9hW off=4129569 len=1147012 2021-07-31T02:05:28.659 INFO:tasks.workunit.client.0.smithi045.stdout:op 101 completed, throughput=4.88MB/sec 2021-07-31T02:05:28.659 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj--r1gcFRMuq1VSRO off=8115448 len=1376328 2021-07-31T02:05:28.667 INFO:tasks.workunit.client.0.smithi045.stdout:op 102 completed, throughput=4.94MB/sec 2021-07-31T02:05:28.667 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-gaofJgxNRIOI8mg off=78867 len=1061641 2021-07-31T02:05:28.672 INFO:tasks.workunit.client.0.smithi045.stdout:op 103 completed, throughput=4.98MB/sec 2021-07-31T02:05:28.673 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-d3p00c3cWU0835J off=3562161 len=1934536 2021-07-31T02:05:28.681 INFO:tasks.workunit.client.0.smithi045.stdout:op 104 completed, throughput=5.06MB/sec 2021-07-31T02:05:28.925 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:05:29.204 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:05:29.204 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 20m ago 21m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:05:29.205 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 18m ago 20m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:05:29.205 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 18m ago 24m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:05:29.205 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 20m ago 26m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:05:29.205 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 20m ago 26m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:05:29.205 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 18m ago 24m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:05:29.206 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 20m ago 24m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:05:29.206 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 20m ago 21m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:05:29.206 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 18m ago 21m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:05:29.206 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 20m ago 24m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:05:29.207 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 20m ago 23m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:05:29.207 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 20m ago 23m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:05:29.207 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 20m ago 23m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:05:29.207 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 18m ago 22m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:05:29.207 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 18m ago 22m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:05:29.208 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 18m ago 22m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:05:29.208 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 18m ago 21m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:05:29.208 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 18m ago 21m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:05:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:29 smithi154 conmon[34553]: cluster 2021-07-31T02:05:28.049215+0000 mgr.x (mgr.24542) 776 : cluster [DBG] pgmap v1419: 9 pgs: 9 active+clean; 127 GiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 3 op/s 2021-07-31T02:05:29.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:29 smithi045 conmon[39718]: cluster 2021-07-31T02:05:28.049215+0000 2021-07-31T02:05:29.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:29 smithi045 conmon[39718]: mgr.x (mgr.24542) 776 : cluster [DBG] pgmap v1419: 9 pgs: 9 active+clean; 127 GiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 3 op/s 2021-07-31T02:05:29.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:29 smithi045 conmon[33655]: cluster 2021-07-31T02:05:28.049215 2021-07-31T02:05:29.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:29 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 776 : cluster [DBG] pgmap v1419: 9 pgs: 9 active+clean; 127 GiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 3 op/s 2021-07-31T02:05:29.539 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:05:29.539 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:05:29.539 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:05:29.540 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:05:29.540 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:05:29.541 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:05:29.541 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:05:29.541 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:05:29.541 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:05:29.542 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:05:29.542 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:05:29.542 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:05:29.542 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:05:29.543 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:05:29.544 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:05:29.545 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:05:29.547 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:05:29.682 INFO:tasks.workunit.client.0.smithi045.stdout: 23: throughput=4.84MB/sec pending data=0 2021-07-31T02:05:29.682 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Zrex0rKZc-cUaor off=1583301 len=1372232 2021-07-31T02:05:29.692 INFO:tasks.workunit.client.0.smithi045.stdout:op 105 completed, throughput=4.9MB/sec 2021-07-31T02:05:29.692 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-_Amx5NKeWSL9j1L off=2736398 len=1752939 2021-07-31T02:05:29.694 INFO:tasks.workunit.client.0.smithi045.stdout:op 106 completed, throughput=4.97MB/sec 2021-07-31T02:05:29.695 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-NDU_h2mNTmnER-S off=2224200 len=1171002 2021-07-31T02:05:29.700 INFO:tasks.workunit.client.0.smithi045.stdout:op 107 completed, throughput=5.01MB/sec 2021-07-31T02:05:30.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[34553]: audit 2021-07-31T02:05:28.912849+0000 mgr.x (mgr.24542) 777 : audit [DBG] 2021-07-31T02:05:30.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[34553]: from='client.25642 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:05:30.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[34553]: audit 2021-07-31T02:05:29.538440+0000 mon.c (mon.1) 76 : audit [DBG] from='client.? 172.21.15.45:0/2776038075' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:05:30.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[34553]: audit 2021-07-31T02:05:30.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[34553]: 2021-07-31T02:05:29.917274+0000 mon.b (mon.2) 160 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:05:30.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[34553]: audit 2021-07-31T02:05 2021-07-31T02:05:30.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[34553]: :29.918679+0000 mon.b (mon.2) 161 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:05:30.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[34553]: audit 2021-07- 2021-07-31T02:05:30.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[34553]: 31T02:05:29.919547+0000 mon.b (mon.2) 162 : 2021-07-31T02:05:30.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[34553]: audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:05:30.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:29 smithi154 conmon[57492]: debug 2021-07-31T02:05:29.963+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:05:30.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:05:30.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:05:30.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:29 smithi154 conmon[57492]: yield 2021-07-31T02:05:30.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:05:30.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:05:30.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:05:30.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:05:30.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:05:30.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:05:30.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:05:30.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:05:30.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:05:30.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:29 smithi154 conmon[57492]: task.result() 2021-07-31T02:05:30.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:05:30.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:29 smithi154 conmon[57492]: debug 2021-07-31T02:05:29.965+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:05:30.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:05:30.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:05:30.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:29 smithi154 conmon[57492]: yield 2021-07-31T02:05:30.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:05:30.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:05:30.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:05:30.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:05:30.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:05:30.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:05:30.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:05:30.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:05:30.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:05:30.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:29 smithi154 conmon[57492]: task.result() 2021-07-31T02:05:30.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:05:30.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: debug 2021-07-31T02:05:30.000+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:05:30.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:05:30.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:05:30.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: yield 2021-07-31T02:05:30.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:05:30.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:05:30.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:05:30.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:05:30.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:05:30.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:05:30.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:05:30.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:05:30.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:05:30.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:05:30.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:05:30.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: debug 2021-07-31T02:05:30.001+0000 7fcb9115c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:05:30.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:05:30.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:05:30.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: yield 2021-07-31T02:05:30.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:05:30.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:05:30.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:05:30.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:05:30.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:05:30.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:05:30.284 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:05:30.284 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:05:30.284 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:05:30.284 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:05:30.285 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:05:30.285 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: 2021-07-31T02:05:30.285 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:05:30.285 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: 2021-07-31T02:05:30.286 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:05:30.286 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:05:30.286 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:05:30.286 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:05:30.286 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:05:30.287 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:05:30.287 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:05:30.287 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:05:30.287 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:05:30.288 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:05:30.288 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:05:30.288 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:05:30.288 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:05:30.288 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:05:30.289 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:05:30.289 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:05:30.289 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:05:30.289 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:05:30.290 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: raise 2021-07-31T02:05:30.290 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:05:30.290 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:05:30.290 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:05:30.290 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:05:30.291 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:05:30.291 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: debug 2021-07-31T02:05:30.008+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:05:30.291 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:05:30.292 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:05:30.292 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: yield 2021-07-31T02:05:30.292 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:05:30.292 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:05:30.292 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:05:30.293 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:05:30.293 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:05:30.293 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:05:30.293 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:05:30.293 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:05:30.294 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:05:30.294 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:05:30.294 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:05:30.294 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: debug 2021-07-31T02:05:30.008+0000 7fcb9095b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:05:30.295 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:05:30.295 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:05:30.295 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: yield 2021-07-31T02:05:30.295 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:05:30.295 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:05:30.296 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:05:30.296 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:05:30.296 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:05:30.296 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:05:30.297 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:05:30.297 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:05:30.297 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:05:30.297 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:05:30.297 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:05:30.298 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: 2021-07-31T02:05:30.298 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:05:30.298 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: 2021-07-31T02:05:30.298 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:05:30.299 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:05:30.299 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:05:30.299 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:05:30.299 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:05:30.300 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:05:30.300 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:05:30.300 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:05:30.300 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:05:30.301 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:05:30.301 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:05:30.301 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:05:30.301 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:05:30.301 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:05:30.302 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:05:30.302 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:05:30.302 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:05:30.302 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:05:30.302 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: raise 2021-07-31T02:05:30.303 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:05:30.303 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:05:30.303 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:05:30.303 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:05:30.304 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:05:30 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:05:30.304 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:30 smithi045 conmon[39718]: audit 2021-07-31T02:05:30.305 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:30 smithi045 conmon[39718]: 2021-07-31T02:05:28 2021-07-31T02:05:30.305 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:30 smithi045 conmon[39718]: .912849+0000 mgr.x (mgr.24542) 777 : audit [DBG] from='client.25642 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:05:30.305 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:30 smithi045 conmon[39718]: audit 2021- 2021-07-31T02:05:30.305 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:30 smithi045 conmon[39718]: 07-31T02:05:29.538440+0000 mon.c ( 2021-07-31T02:05:30.306 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:30 smithi045 conmon[39718]: mon.1) 76 : audit [DBG] from='client.? 172.21.15.45:0/2776038075' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:05:30.306 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:30 smithi045 conmon[39718]: audit 2021-07-31T02:05:30.306 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:30 smithi045 conmon[39718]: 2021-07-31T02:05:29. 2021-07-31T02:05:30.306 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:30 smithi045 conmon[39718]: 917274+0000 mon.b (mon.2) 160 : audit [DBG] 2021-07-31T02:05:30.307 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:30 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:05:30.307 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:30 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T02:05:30.307 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:30 smithi045 conmon[39718]: T02:05:29.918679+0000 mon.b (mon. 2021-07-31T02:05:30.307 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:30 smithi045 conmon[39718]: 2) 161 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:05:30.307 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:30 smithi045 conmon[39718]: audit 2021-07-31T02 2021-07-31T02:05:30.308 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:30 smithi045 conmon[39718]: :05:29.919547+0000 mon.b (mon.2) 162 : audit 2021-07-31T02:05:30.308 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:30 smithi045 conmon[39718]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:05:30.308 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:30 smithi045 conmon[33655]: audit 2021-07-31T02:05:30.309 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:30 smithi045 conmon[33655]: 2021-07-31T02:05:28.912849+0000 mgr.x (mgr.24542) 777 : audit [DBG] from='client.25642 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:05:30.309 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:30 smithi045 conmon[33655]: audit 2021 2021-07-31T02:05:30.309 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:30 smithi045 conmon[33655]: -07-31T02:05:29.538440+0000 mon.c (mon 2021-07-31T02:05:30.309 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:30 smithi045 conmon[33655]: .1) 76 : audit [DBG] 2021-07-31T02:05:30.309 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:30 smithi045 conmon[33655]: from='client.? 172.21.15.45:0/2776038075' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:05:30.310 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:30 smithi045 conmon[33655]: audit 2021-07-31 2021-07-31T02:05:30.310 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:30 smithi045 conmon[33655]: T02:05:29.917274+0000 mon.b (mon.2) 160 : audit [DBG] 2021-07-31T02:05:30.310 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:30 smithi045 conmon[33655]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:05:30.310 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:30 smithi045 conmon[33655]: audit 2021-07-31T02:05:29 2021-07-31T02:05:30.311 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:30 smithi045 conmon[33655]: .918679+0000 mon.b (mon.2) 161 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:05:30.311 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:30 smithi045 conmon[33655]: audit 2021-07-31T 2021-07-31T02:05:30.311 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:30 smithi045 conmon[33655]: 02:05:29.919547+0000 mon.b (mon.2) 162 : audit [INF] 2021-07-31T02:05:30.311 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:30 smithi045 conmon[33655]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:05:30.700 INFO:tasks.workunit.client.0.smithi045.stdout: 24: throughput=4.81MB/sec pending data=0 2021-07-31T02:05:30.700 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-IoV2cMIbLBqUCPH off=8688087 len=269237 2021-07-31T02:05:30.704 INFO:tasks.workunit.client.0.smithi045.stdout:op 108 completed, throughput=4.82MB/sec 2021-07-31T02:05:30.704 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-UU9D5shgaZT8sXB off=9055705 len=2094859 2021-07-31T02:05:30.707 INFO:tasks.workunit.client.0.smithi045.stdout:op 109 completed, throughput=4.9MB/sec 2021-07-31T02:05:30.710 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-OJccfo6H3Ot3Gdf off=1411992 len=934757 2021-07-31T02:05:30.713 INFO:tasks.workunit.client.0.smithi045.stdout:op 110 completed, throughput=4.94MB/sec 2021-07-31T02:05:30.713 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Ukep62J6bnmPS7C off=5664791 len=1711961 2021-07-31T02:05:30.722 INFO:tasks.workunit.client.0.smithi045.stdout:op 111 completed, throughput=5MB/sec 2021-07-31T02:05:31.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: audit 2021-07-31T02:05:29.195913+0000 mgr.x (mgr.24542) 778 : audit [DBG] from='client.15741 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:05:31.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:05:29.920312+0000 mgr.x (mgr.24542) 779 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:05:31.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:05:29.964244+0000 mgr.x (mgr.24542) 780 : cephadm [ERR] 2021-07-31T02:05:31.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: 2021-07-31T02:05:31.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:05:31.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:05:31.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: yield 2021-07-31T02:05:31.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:05:31.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:05:31.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:05:31.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:05:31.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:05:31.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:05:31.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:05:31.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:05:31.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:05:31.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: task.result() 2021-07-31T02:05:31.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:05:31.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:05:29.965053+0000 mgr.x (mgr. 2021-07-31T02:05:31.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: 24542) 781 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:05:31.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:05:29.966303+0000 mgr.x (mgr.24542) 782 : cephadm [ERR] 2021-07-31T02:05:31.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:05:31.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:05:31.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: yield 2021-07-31T02:05:31.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:05:31.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:05:31.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:05:31.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:05:31.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:05:31.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:05:31.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:05:31.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:05:31.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:05:31.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: task.result() 2021-07-31T02:05:31.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:05:31.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:05:29.976074+0000 mgr.x (mgr.24542) 783 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:05:31.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: cephadm 2021 2021-07-31T02:05:31.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: -07-31T02:05:30.001538+0000 mgr.x (mgr.24542) 784 : cephadm [ERR] 2021-07-31T02:05:31.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:05:31.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:05:31.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: yield 2021-07-31T02:05:31.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:05:31.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:05:31.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:05:31.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:05:31.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:05:31.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:05:31.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:05:31.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:05:31.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:05:31.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: task.result() 2021-07-31T02:05:31.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:05:31.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:05:30.001990+0000 mgr.x (mgr.24542) 785 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:05:31.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:05:31.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:05:31.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: yield 2021-07-31T02:05:31.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:05:31.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:05:31.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:05:31.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:05:31.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:05:31.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:05:31.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:05:31.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:05:31.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:05:31.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: task.result() 2021-07-31T02:05:31.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:05:31.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: 2021-07-31T02:05:31.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:05:31.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: 2021-07-31T02:05:31.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:05:31.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:05:31.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:05:31.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:05:31.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:05:31.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:05:31.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:05:31.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:05:31.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:05:31.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:05:31.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:05:31.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:05:31.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:05:31.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:05:31.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:05:31.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:05:31.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:05:31.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:05:31.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: raise 2021-07-31T02:05:31.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:05:31.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:05:31.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:05:31.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:05:31.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:05:31.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:05:30.009047+0000 mgr.x (mgr.24542) 786 : cephadm [ERR] 2021-07-31T02:05:31.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:05:31.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:05:31.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: yield 2021-07-31T02:05:31.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:05:31.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:05:31.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:05:31.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:05:31.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:05:31.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:05:31.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:05:31.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:05:31.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:05:31.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: task.result() 2021-07-31T02:05:31.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:05:31.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: cephadm 2021-07-31 2021-07-31T02:05:31.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: T02:05:30.009459+0000 mgr.x (mgr.24542) 787 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:05:31.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:05:31.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:05:31.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: yield 2021-07-31T02:05:31.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:05:31.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:05:31.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:05:31.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:05:31.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:05:31.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:05:31.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:05:31.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:05:31.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:05:31.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: task.result() 2021-07-31T02:05:31.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:05:31.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: 2021-07-31T02:05:31.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:05:31.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: 2021-07-31T02:05:31.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:05:31.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:05:31.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:05:31.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:05:31.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:05:31.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:05:31.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:05:31.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:05:31.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:05:31.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:05:31.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:05:31.541 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:05:31.541 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:05:31.541 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:05:31.541 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:05:31.542 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:05:31.542 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:05:31.542 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:05:31.542 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: raise 2021-07-31T02:05:31.542 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:05:31.543 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:05:31.543 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:05:31.543 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:05:31.543 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:05:31.544 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:31 smithi154 conmon[34553]: cluster 2021-07-31T02:05:30.049616+0000 mgr.x (mgr.24542) 788 : cluster [DBG] pgmap v1420: 9 pgs: 9 active+clean; 127 GiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 3.2 MiB/s rd, 3.3 MiB/s wr, 5 op/s 2021-07-31T02:05:31.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: audit 2021-07-31T02:05:29.195913+0000 mgr.x 2021-07-31T02:05:31.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: (mgr.24542) 778 : audit [DBG] from='client.15741 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:05:31.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: cephadm 2021-07-31T02: 2021-07-31T02:05:31.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: 05:29.920312+0000 mgr.x (mgr.24542) 779 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:05:31.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: cephadm 2021 2021-07-31T02:05:31.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: -07-31T02:05:29.964244+0000 2021-07-31T02:05:31.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: mgr.x (mgr.24542) 780 : cephadm [ERR] 2021-07-31T02:05:31.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:05:31.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:05:31.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: yield 2021-07-31T02:05:31.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:05:31.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:05:31.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:05:31.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:05:31.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:05:31.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:05:31.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:05:31.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:05:31.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:05:31.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: task.result() 2021-07-31T02:05:31.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:05:31.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:05:29.965053+0000 mgr.x (mgr.24542) 781 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:05:31.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: cephadm 2021-07-31T02: 2021-07-31T02:05:31.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: 05:29.966303+0000 mgr.x (mgr.24542) 782 : cephadm [ERR] 2021-07-31T02:05:31.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:05:31.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:05:31.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: yield 2021-07-31T02:05:31.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:05:31.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:05:31.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:05:31.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:05:31.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:05:31.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:05:31.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:05:31.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:05:31.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:05:31.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: task.result() 2021-07-31T02:05:31.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:05:31.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:05:29.976074+0000 mgr.x (mgr.24542) 783 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:05:31.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:05:30.001538+0000 mgr.x (mgr 2021-07-31T02:05:31.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: .24542) 784 : cephadm [ERR] 2021-07-31T02:05:31.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:05:31.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:05:31.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: yield 2021-07-31T02:05:31.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:05:31.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:05:31.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:05:31.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:05:31.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:05:31.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:05:31.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:05:31.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:05:31.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:05:31.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: task.result() 2021-07-31T02:05:31.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:05:31.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:05:30.001990+0000 mgr.x (mgr.24542) 785 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:05:31.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:05:31.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:05:31.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: yield 2021-07-31T02:05:31.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:05:31.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:05:31.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:05:31.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:05:31.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:05:31.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:05:31.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:05:31.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:05:31.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:05:31.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: task.result() 2021-07-31T02:05:31.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:05:31.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: 2021-07-31T02:05:31.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:05:31.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: 2021-07-31T02:05:31.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:05:31.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:05:31.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:05:31.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:05:31.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:05:31.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:05:31.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:05:31.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: audit 2021-07-31T02:05:29.195913+0000 mgr.x (mgr.24542) 778 : audit [DBG] from='client.15741 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:05:31.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:05:29.920312+0000 mgr.x (mgr.24542) 779 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:05:31.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:05:29.964244+0000 mgr.x ( 2021-07-31T02:05:31.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: mgr.24542) 780 : cephadm [ERR] 2021-07-31T02:05:31.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:05:31.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:05:31.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: yield 2021-07-31T02:05:31.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:05:31.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:05:31.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:05:31.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:05:31.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:05:31.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:05:31.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:05:31.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:05:31.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:05:31.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: task.result() 2021-07-31T02:05:31.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:05:31.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:05:29.965053+0000 mgr.x (mgr.24542) 781 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:05:31.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:05:31.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: 2021-07- 2021-07-31T02:05:31.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: 31T02:05:29. 2021-07-31T02:05:31.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: 966303+0000 mgr.x (mgr.24542 2021-07-31T02:05:31.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: ) 782 : cephadm [ERR] 2021-07-31T02:05:31.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:05:31.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:05:31.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: yield 2021-07-31T02:05:31.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:05:31.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:05:31.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:05:31.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:05:31.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:05:31.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:05:31.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:05:31.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:05:31.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:05:31.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: task.result() 2021-07-31T02:05:31.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:05:31.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: 2021-07-31T02:05:31.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: cephadm 2021-07-31T02: 2021-07-31T02:05:31.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:05:31.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:05:31.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:05:31.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:05:31.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:05:31.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:05:31.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:05:31.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:05:31.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:05:31.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:05:31.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:05:31.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: raise 2021-07-31T02:05:31.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:05:31.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:05:31.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:05:31.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:05:31.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:05:31.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:05: 2021-07-31T02:05:31.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: 30.009047+0000 mgr.x (mgr.24542) 786 : cephadm [ERR] 2021-07-31T02:05:31.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:05:31.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:05:31.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: yield 2021-07-31T02:05:31.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:05:31.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:05:31.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:05:31.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:05:31.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:05:31.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:05:31.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:05:31.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:05:31.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:05:31.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: task.result() 2021-07-31T02:05:31.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:05:31.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:05:30.009459+0000 mgr.x (mgr.24542) 787 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:05:31.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:05:31.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:05:31.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: yield 2021-07-31T02:05:31.585 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:05:31.585 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:05:31.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: 05:29.976074+0000 mgr.x (mgr 2021-07-31T02:05:31.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: .24542) 783 : cephadm [INF] 2021-07-31T02:05:31.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:05:31.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: cephadm 2021- 2021-07-31T02:05:31.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: 07-31T02:05:30.001538 2021-07-31T02:05:31.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 2021-07-31T02:05:31.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: 784 : cephadm [ERR] 2021-07-31T02:05:31.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:05:31.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:05:31.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: yield 2021-07-31T02:05:31.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:05:31.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:05:31.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:05:31.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:05:31.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:05:31.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:05:31.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:05:31.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:05:31.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:05:31.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: task.result() 2021-07-31T02:05:31.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:05:31.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: cephadm 2021 2021-07-31T02:05:31.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: -07-31T02:05: 2021-07-31T02:05:31.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: 30.001990+0000 mgr.x ( 2021-07-31T02:05:31.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: mgr.24542) 785 : cephadm [ERR] 2021-07-31T02:05:31.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:05:31.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:05:31.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:05:31.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: yield 2021-07-31T02:05:31.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:05:31.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:05:31.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:05:31.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:05:31.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:05:31.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:05:31.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:05:31.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:05:31.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:05:31.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: task.result() 2021-07-31T02:05:31.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:05:31.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: 2021-07-31T02:05:31.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:05:31.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: 2021-07-31T02:05:31.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:05:31.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:05:31.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:05:31.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:05:31.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:05:31.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:05:31.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:05:31.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:05:31.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:05:31.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:05:31.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: task.result() 2021-07-31T02:05:31.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:05:31.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: 2021-07-31T02:05:31.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:05:31.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: 2021-07-31T02:05:31.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:05:31.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:05:31.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:05:31.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:05:31.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:05:31.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:05:31.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:05:31.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:05:31.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:05:31.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:05:31.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:05:31.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:05:31.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:05:31.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:05:31.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:05:31.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:05:31.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:05:31.603 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:05:31.603 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: raise 2021-07-31T02:05:31.603 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:05:31.603 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:05:31.603 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:05:31.604 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:05:31.604 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:05:31.604 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[39718]: cluster 2021-07-31T02:05:30.049616+0000 mgr.x (mgr.24542) 788 : cluster [DBG] pgmap v1420: 9 pgs: 9 active+clean; 127 GiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 3.2 MiB/s rd, 3.3 MiB/s wr, 5 op/s 2021-07-31T02:05:31.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:05:31.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:05:31.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:05:31.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:05:31.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:05:31.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:05:31.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:05:31.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:05:31.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:05:31.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:05:31.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:05:31.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:05:31.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:05:31.608 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:05:31.608 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:05:31.608 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: raise 2021-07-31T02:05:31.608 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:05:31.609 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:05:31.609 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:05:31.609 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:05:31.609 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:05:31.609 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:05:30.009047+0000 mgr.x (mgr.24542) 786 : cephadm [ERR] 2021-07-31T02:05:31.610 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:05:31.610 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:05:31.610 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: yield 2021-07-31T02:05:31.610 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:05:31.610 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:05:31.611 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:05:31.611 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:05:31.611 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:05:31.611 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:05:31.612 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:05:31.612 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:05:31.612 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:05:31.612 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: task.result() 2021-07-31T02:05:31.612 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:05:31.613 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:05:30.009459+0000 mgr.x (mgr.24542) 787 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:05:31.613 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:05:31.613 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:05:31.613 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: yield 2021-07-31T02:05:31.614 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:05:31.614 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:05:31.614 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:05:31.614 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:05:31.614 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:05:31.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:05:31.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:05:31.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:05:31.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:05:31.616 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: task.result() 2021-07-31T02:05:31.616 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:05:31.616 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: 2021-07-31T02:05:31.616 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:05:31.616 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: 2021-07-31T02:05:31.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:05:31.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:05:31.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:05:31.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:05:31.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:05:31.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:05:31.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:05:31.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:05:31.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:05:31.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:05:31.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:05:31.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:05:31.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:05:31.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:05:31.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:05:31.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:05:31.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:05:31.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:05:31.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: raise 2021-07-31T02:05:31.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:05:31.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:05:31.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:05:31.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:05:31.622 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:05:31.622 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:31 smithi045 conmon[33655]: cluster 2021-07-31T02:05:30.049616+0000 mgr.x (mgr.24542) 788 : cluster [DBG] pgmap v1420: 9 pgs: 9 active+clean; 127 GiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 3.2 MiB/s rd, 3.3 MiB/s wr, 5 op/s 2021-07-31T02:05:31.722 INFO:tasks.workunit.client.0.smithi045.stdout: 25: throughput=4.8MB/sec pending data=0 2021-07-31T02:05:31.722 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-MEaVUjU-0LMQe4V off=18394493 len=1254154 2021-07-31T02:05:31.725 INFO:tasks.workunit.client.0.smithi045.stdout:op 112 completed, throughput=4.85MB/sec 2021-07-31T02:05:31.725 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-xdNpNCcqH7ymv4T off=437059 len=1661831 2021-07-31T02:05:31.734 INFO:tasks.workunit.client.0.smithi045.stdout:op 113 completed, throughput=4.91MB/sec 2021-07-31T02:05:31.735 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-3oaEaKjJHjP8JQq off=14020552 len=201507 2021-07-31T02:05:31.736 INFO:tasks.workunit.client.0.smithi045.stdout:op 114 completed, throughput=4.92MB/sec 2021-07-31T02:05:31.736 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-gU3MLaDjZw3LP33 off=10986417 len=663906 2021-07-31T02:05:31.739 INFO:tasks.workunit.client.0.smithi045.stdout:op 115 completed, throughput=4.94MB/sec 2021-07-31T02:05:31.739 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-rwK8ez5npq5T5a1 off=1082474 len=994293 2021-07-31T02:05:31.745 INFO:tasks.workunit.client.0.smithi045.stdout:op 116 completed, throughput=4.98MB/sec 2021-07-31T02:05:31.745 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-q3zZzNDJBmBjSM7 off=1229158 len=563518 2021-07-31T02:05:31.745 INFO:tasks.workunit.client.0.smithi045.stdout:op 117 completed, throughput=5MB/sec 2021-07-31T02:05:31.746 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-QgxWNVpRStzoDVv off=12987776 len=439818 2021-07-31T02:05:31.748 INFO:tasks.workunit.client.0.smithi045.stdout:op 118 completed, throughput=5.02MB/sec 2021-07-31T02:05:32.749 INFO:tasks.workunit.client.0.smithi045.stdout: 26: throughput=4.83MB/sec pending data=0 2021-07-31T02:05:32.749 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-VCAW-MJX9SAngCr off=4487127 len=1470936 2021-07-31T02:05:32.750 INFO:tasks.workunit.client.0.smithi045.stdout:op 119 completed, throughput=4.88MB/sec 2021-07-31T02:05:32.751 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-8wVb-yl9H8ieYjF off=1285090 len=737488 2021-07-31T02:05:32.751 INFO:tasks.workunit.client.0.smithi045.stdout:op 120 completed, throughput=4.9MB/sec 2021-07-31T02:05:32.752 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-yEws7U_P9Tl2nuP off=4686233 len=32258 2021-07-31T02:05:32.752 INFO:tasks.workunit.client.0.smithi045.stdout:op 121 completed, throughput=4.91MB/sec 2021-07-31T02:05:32.753 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-JlEe_SRgfreIE1t off=5358573 len=661003 2021-07-31T02:05:32.757 INFO:tasks.workunit.client.0.smithi045.stdout:op 122 completed, throughput=4.93MB/sec 2021-07-31T02:05:32.758 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-HGXznX6oPexwcl7 off=10222977 len=797784 2021-07-31T02:05:32.762 INFO:tasks.workunit.client.0.smithi045.stdout:op 123 completed, throughput=4.96MB/sec 2021-07-31T02:05:32.762 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-gkiNuvL9vhoyr8Q off=8344204 len=1776200 2021-07-31T02:05:32.773 INFO:tasks.workunit.client.0.smithi045.stdout:op 124 completed, throughput=5.02MB/sec 2021-07-31T02:05:33.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:32 smithi154 conmon[34553]: cluster 2021-07-31T02:05:32.050196+0000 mgr.x (mgr.24542) 789 : cluster [DBG] pgmap v1421: 9 pgs: 9 active+clean; 127 GiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.4 MiB/s wr, 3 op/s 2021-07-31T02:05:33.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:32 smithi045 conmon[39718]: cluster 2021-07-31T02:05:32. 2021-07-31T02:05:33.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:32 smithi045 conmon[39718]: 050196+0000 mgr.x (mgr.24542) 789 : cluster [DBG] pgmap v1421: 9 pgs: 9 active+clean; 127 GiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.4 MiB/s wr, 3 op/s 2021-07-31T02:05:33.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:32 smithi045 conmon[33655]: cluster 2021-07-31T02:05:32.050196+0000 mgr.x (mgr.24542) 789 : 2021-07-31T02:05:33.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:32 smithi045 conmon[33655]: cluster [DBG] pgmap v1421: 9 pgs: 9 active+clean; 127 GiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.4 MiB/s wr, 3 op/s 2021-07-31T02:05:33.779 INFO:tasks.workunit.client.0.smithi045.stdout: 27: throughput=4.84MB/sec pending data=0 2021-07-31T02:05:33.779 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-jtxTg2m8qpt59FX off=0 len=1211551 2021-07-31T02:05:33.780 INFO:tasks.workunit.client.0.smithi045.stdout:op 125 completed, throughput=4.88MB/sec 2021-07-31T02:05:33.781 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-GFBefaY4GtBrUQT off=769087 len=1139046 2021-07-31T02:05:33.787 INFO:tasks.workunit.client.0.smithi045.stdout:op 126 completed, throughput=4.92MB/sec 2021-07-31T02:05:33.787 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-P9UFllgbnGTL6_6 off=8762187 len=391155 2021-07-31T02:05:33.790 INFO:tasks.workunit.client.0.smithi045.stdout:op 127 completed, throughput=4.93MB/sec 2021-07-31T02:05:33.790 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-5Vi8lAXys3ZJOCU off=2747445 len=631627 2021-07-31T02:05:33.791 INFO:tasks.workunit.client.0.smithi045.stdout:op 128 completed, throughput=4.95MB/sec 2021-07-31T02:05:33.791 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-nLn5CZM0DzYAvyR off=12129619 len=441939 2021-07-31T02:05:33.795 INFO:tasks.workunit.client.0.smithi045.stdout:op 129 completed, throughput=4.96MB/sec 2021-07-31T02:05:33.796 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-lFHWl_ySVUxwiAu off=852669 len=503142 2021-07-31T02:05:33.798 INFO:tasks.workunit.client.0.smithi045.stdout:op 130 completed, throughput=4.98MB/sec 2021-07-31T02:05:33.798 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-AP_1sSd-UpCzH4N off=708204 len=1008365 2021-07-31T02:05:33.800 INFO:tasks.workunit.client.0.smithi045.stdout:op 131 completed, throughput=5.02MB/sec 2021-07-31T02:05:34.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:05:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:05:33.539Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:05:34.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:05:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:05:33.539Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:05:34.027 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:05:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:05:33.539Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:05:34.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:34 smithi154 conmon[34553]: cluster 2021-07-31T02:05:34.050636+0000 mgr.x (mgr.24542) 790 : cluster [DBG] pgmap v1422: 9 pgs: 9 active+clean; 127 GiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 2.5 MiB/s rd, 3.8 MiB/s wr, 6 op/s 2021-07-31T02:05:34.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:34 smithi045 conmon[33655]: cluster 2021-07-31T02:05:34.050636+0000 mgr.x (mgr.24542) 790 : cluster [DBG] pgmap v1422: 9 pgs: 9 active+clean; 127 GiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 2.5 MiB/s rd, 3.8 MiB/s wr, 6 op/s 2021-07-31T02:05:34.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:34 smithi045 conmon[39718]: cluster 2021-07-31T02:05:34.050636 2021-07-31T02:05:34.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:34 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 790 : cluster [DBG] pgmap v1422: 9 pgs: 9 active+clean; 127 GiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 2.5 MiB/s rd, 3.8 MiB/s wr, 6 op/s 2021-07-31T02:05:34.799 INFO:tasks.workunit.client.0.smithi045.stdout: 28: throughput=4.84MB/sec pending data=0 2021-07-31T02:05:34.800 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-CQbwCrmrXC8C_pi off=17370441 len=1245062 2021-07-31T02:05:34.803 INFO:tasks.workunit.client.0.smithi045.stdout:op 132 completed, throughput=4.88MB/sec 2021-07-31T02:05:34.804 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ThIl5N15djr9hJ8 off=14907732 len=1787323 2021-07-31T02:05:34.806 INFO:tasks.workunit.client.0.smithi045.stdout:op 133 completed, throughput=4.94MB/sec 2021-07-31T02:05:34.807 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-c0hwCPywk4JpD6T off=3568261 len=2019488 2021-07-31T02:05:34.818 INFO:tasks.workunit.client.0.smithi045.stdout:op 134 completed, throughput=5.01MB/sec 2021-07-31T02:05:35.818 INFO:tasks.workunit.client.0.smithi045.stdout: 29: throughput=4.84MB/sec pending data=0 2021-07-31T02:05:35.818 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-LcNugv35I7ghClc off=18139512 len=20080 2021-07-31T02:05:35.819 INFO:tasks.workunit.client.0.smithi045.stdout:op 135 completed, throughput=4.84MB/sec 2021-07-31T02:05:35.819 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-7ofvl0iuvwKENtG off=251723 len=1291609 2021-07-31T02:05:35.821 INFO:tasks.workunit.client.0.smithi045.stdout:op 136 completed, throughput=4.88MB/sec 2021-07-31T02:05:35.822 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Y-0ezNu0E2WyL7S off=192351 len=1886305 2021-07-31T02:05:35.829 INFO:tasks.workunit.client.0.smithi045.stdout:op 137 completed, throughput=4.94MB/sec 2021-07-31T02:05:35.830 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-INL5vVFpIBlxwYk off=2120309 len=1439881 2021-07-31T02:05:35.836 INFO:tasks.workunit.client.0.smithi045.stdout:op 138 completed, throughput=4.98MB/sec 2021-07-31T02:05:35.836 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-FbZ5yAuIoN5BuTh off=1869920 len=1407810 2021-07-31T02:05:35.838 INFO:tasks.workunit.client.0.smithi045.stdout:op 139 completed, throughput=5.03MB/sec 2021-07-31T02:05:36.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:36 smithi154 conmon[34553]: cluster 2021-07-31T02:05:36.051149+0000 mgr.x (mgr.24542) 791 : cluster [DBG] 2021-07-31T02:05:36.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:36 smithi154 conmon[34553]: pgmap v1423: 9 pgs: 9 active+clean; 127 GiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.6 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2021-07-31T02:05:36.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:36 smithi045 conmon[33655]: cluster 2021-07-31T02:05:36.051149+0000 mgr.x (mgr.24542) 791 : cluster 2021-07-31T02:05:36.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:36 smithi045 conmon[33655]: [DBG] pgmap v1423: 9 pgs: 9 active+clean; 127 GiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.6 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2021-07-31T02:05:36.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:36 smithi045 conmon[39718]: cluster 2021-07-31T02:05:36. 2021-07-31T02:05:36.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:36 smithi045 conmon[39718]: 051149+0000 mgr.x (mgr.24542) 791 : cluster [DBG] pgmap v1423: 9 pgs: 9 active+clean; 127 GiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.6 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2021-07-31T02:05:36.838 INFO:tasks.workunit.client.0.smithi045.stdout: 30: throughput=4.86MB/sec pending data=0 2021-07-31T02:05:36.838 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-MNrOXmKHJN4yfus off=4019335 len=314124 2021-07-31T02:05:36.840 INFO:tasks.workunit.client.0.smithi045.stdout:op 140 completed, throughput=4.87MB/sec 2021-07-31T02:05:36.840 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-gI4PteDt3e0MHEN off=3250841 len=224427 2021-07-31T02:05:36.841 INFO:tasks.workunit.client.0.smithi045.stdout:op 141 completed, throughput=4.88MB/sec 2021-07-31T02:05:36.841 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Ztwrfcc1QqZ7P2G off=21703528 len=2020050 2021-07-31T02:05:36.852 INFO:tasks.workunit.client.0.smithi045.stdout:op 142 completed, throughput=4.94MB/sec 2021-07-31T02:05:36.852 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-oXQQVl09QpnBpIl off=22566353 len=1455935 2021-07-31T02:05:36.860 INFO:tasks.workunit.client.0.smithi045.stdout:op 143 completed, throughput=4.98MB/sec 2021-07-31T02:05:36.861 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-rNurojN3hQa9-Gb off=1469708 len=605714 2021-07-31T02:05:36.861 INFO:tasks.workunit.client.0.smithi045.stdout:op 144 completed, throughput=5MB/sec 2021-07-31T02:05:37.862 INFO:tasks.workunit.client.0.smithi045.stdout: 31: throughput=4.85MB/sec pending data=0 2021-07-31T02:05:37.863 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-2x6FNHz5YXbK2MI off=8795173 len=75866 2021-07-31T02:05:37.863 INFO:tasks.workunit.client.0.smithi045.stdout:op 145 completed, throughput=4.85MB/sec 2021-07-31T02:05:37.864 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-anJMRKAgXFzGwiW off=0 len=1399189 2021-07-31T02:05:37.866 INFO:tasks.workunit.client.0.smithi045.stdout:op 146 completed, throughput=4.89MB/sec 2021-07-31T02:05:37.867 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-IF-IRrkbslpSdXk off=76504 len=1484641 2021-07-31T02:05:37.875 INFO:tasks.workunit.client.0.smithi045.stdout:op 147 completed, throughput=4.93MB/sec 2021-07-31T02:05:37.875 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-xXKjQLrR8dBjBYN off=753561 len=192077 2021-07-31T02:05:37.878 INFO:tasks.workunit.client.0.smithi045.stdout:op 148 completed, throughput=4.94MB/sec 2021-07-31T02:05:37.878 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ZwAOLjmcirFQGKg off=6020728 len=255525 2021-07-31T02:05:37.880 INFO:tasks.workunit.client.0.smithi045.stdout:op 149 completed, throughput=4.95MB/sec 2021-07-31T02:05:37.880 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-SsQMwSoxZnXrzxC off=604569 len=1845752 2021-07-31T02:05:37.882 INFO:tasks.workunit.client.0.smithi045.stdout:op 150 completed, throughput=5MB/sec 2021-07-31T02:05:38.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:38 smithi154 conmon[34553]: cluster 2021-07-31T02:05:38.051495+0000 mgr.x (mgr.24542) 792 : cluster 2021-07-31T02:05:38.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:38 smithi154 conmon[34553]: [DBG] pgmap v1424: 9 pgs: 9 active+clean; 127 GiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.6 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2021-07-31T02:05:38.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:38 smithi045 conmon[39718]: cluster 2021-07-31T02:05:38.051495+0000 mgr.x (mgr.24542) 2021-07-31T02:05:38.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:38 smithi045 conmon[39718]: 792 : cluster [DBG] pgmap v1424: 9 pgs: 9 active+clean; 127 GiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.6 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2021-07-31T02:05:38.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:38 smithi045 conmon[33655]: cluster 2021-07-31T02:05 2021-07-31T02:05:38.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:38 smithi045 conmon[33655]: :38.051495+0000 mgr.x (mgr.24542) 792 : cluster [DBG] pgmap v1424: 9 pgs: 9 active+clean; 127 GiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.6 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2021-07-31T02:05:38.883 INFO:tasks.workunit.client.0.smithi045.stdout: 32: throughput=4.85MB/sec pending data=0 2021-07-31T02:05:38.883 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-6I6OiHtv-h7i8re off=12571439 len=598791 2021-07-31T02:05:38.887 INFO:tasks.workunit.client.0.smithi045.stdout:op 151 completed, throughput=4.86MB/sec 2021-07-31T02:05:38.887 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-KLQjuhXy9FK6b11 off=5962832 len=1175166 2021-07-31T02:05:38.893 INFO:tasks.workunit.client.0.smithi045.stdout:op 152 completed, throughput=4.9MB/sec 2021-07-31T02:05:38.894 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Tr-cpMGaVGWh-ls off=10050090 len=1017776 2021-07-31T02:05:38.896 INFO:tasks.workunit.client.0.smithi045.stdout:op 153 completed, throughput=4.93MB/sec 2021-07-31T02:05:38.896 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-urpqVxhE6M5-6Nf off=8637935 len=1552239 2021-07-31T02:05:38.903 INFO:tasks.workunit.client.0.smithi045.stdout:op 154 completed, throughput=4.97MB/sec 2021-07-31T02:05:38.904 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-reu_RDYI-xO6MJA off=1690456 len=829210 2021-07-31T02:05:38.908 INFO:tasks.workunit.client.0.smithi045.stdout:op 155 completed, throughput=4.99MB/sec 2021-07-31T02:05:38.909 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-f3J399bJQbJ2PgJ off=7461287 len=1725429 2021-07-31T02:05:38.912 INFO:tasks.workunit.client.0.smithi045.stdout:op 156 completed, throughput=5.04MB/sec 2021-07-31T02:05:39.913 INFO:tasks.workunit.client.0.smithi045.stdout: 33: throughput=4.89MB/sec pending data=0 2021-07-31T02:05:39.913 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-3YQjIUVP-OBxot9 off=5832020 len=793225 2021-07-31T02:05:39.914 INFO:tasks.workunit.client.0.smithi045.stdout:op 157 completed, throughput=4.92MB/sec 2021-07-31T02:05:39.914 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-JapL04FUHpsybZd off=8545207 len=759040 2021-07-31T02:05:39.919 INFO:tasks.workunit.client.0.smithi045.stdout:op 158 completed, throughput=4.94MB/sec 2021-07-31T02:05:39.920 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-8r9j0R5CX1TqIzB off=2725458 len=1673962 2021-07-31T02:05:39.928 INFO:tasks.workunit.client.0.smithi045.stdout:op 159 completed, throughput=4.98MB/sec 2021-07-31T02:05:39.929 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-iHHG86sFR2uwLiC off=6132024 len=1600118 2021-07-31T02:05:39.936 INFO:tasks.workunit.client.0.smithi045.stdout:op 160 completed, throughput=5.03MB/sec 2021-07-31T02:05:40.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:40 smithi154 conmon[34553]: cluster 2021-07-31T02:05:40.052118+0000 mgr.x (mgr.24542) 793 2021-07-31T02:05:40.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:40 smithi154 conmon[34553]: : cluster [DBG] pgmap v1425: 9 pgs: 9 active+clean; 127 GiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 2.4 MiB/s rd, 3.7 MiB/s wr, 5 op/s 2021-07-31T02:05:40.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:40 smithi045 conmon[33655]: cluster 2021-07-31T02:05:40.052118 2021-07-31T02:05:40.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:40 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 793 : cluster [DBG] pgmap v1425: 9 pgs: 9 active+clean; 127 GiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 2.4 MiB/s rd, 3.7 MiB/s wr, 5 op/s 2021-07-31T02:05:40.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:40 smithi045 conmon[39718]: cluster 2021-07-31T02:05:40.052118+0000 mgr.x (mgr.24542) 793 : cluster [DBG] 2021-07-31T02:05:40.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:40 smithi045 conmon[39718]: pgmap v1425: 9 pgs: 9 active+clean; 127 GiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 2.4 MiB/s rd, 3.7 MiB/s wr, 5 op/s 2021-07-31T02:05:40.936 INFO:tasks.workunit.client.0.smithi045.stdout: 34: throughput=4.88MB/sec pending data=0 2021-07-31T02:05:40.936 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-l6pYtDl-tZuZhwk off=10951715 len=936225 2021-07-31T02:05:40.937 INFO:tasks.workunit.client.0.smithi045.stdout:op 161 completed, throughput=4.91MB/sec 2021-07-31T02:05:40.937 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-sEUgXEAtgogu_Qt off=16071692 len=423013 2021-07-31T02:05:40.942 INFO:tasks.workunit.client.0.smithi045.stdout:op 162 completed, throughput=4.92MB/sec 2021-07-31T02:05:40.942 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-H3Ad39OUzUpw-1i off=11530696 len=440091 2021-07-31T02:05:40.943 INFO:tasks.workunit.client.0.smithi045.stdout:op 163 completed, throughput=4.93MB/sec 2021-07-31T02:05:40.943 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-MeXvW5sy7KTH8-y off=11514443 len=613531 2021-07-31T02:05:40.944 INFO:tasks.workunit.client.0.smithi045.stdout:op 164 completed, throughput=4.95MB/sec 2021-07-31T02:05:40.945 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-XMeQRahL3nMrk-6 off=7785909 len=1958719 2021-07-31T02:05:40.948 INFO:tasks.workunit.client.0.smithi045.stdout:op 165 completed, throughput=5MB/sec 2021-07-31T02:05:41.949 INFO:tasks.workunit.client.0.smithi045.stdout: 35: throughput=4.86MB/sec pending data=0 2021-07-31T02:05:41.949 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-SVrSZ7hlh3ClH7M off=183531 len=1258800 2021-07-31T02:05:41.956 INFO:tasks.workunit.client.0.smithi045.stdout:op 166 completed, throughput=4.89MB/sec 2021-07-31T02:05:41.956 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-GBHJI6PMIgDsUeM off=6355633 len=30630 2021-07-31T02:05:41.957 INFO:tasks.workunit.client.0.smithi045.stdout:op 167 completed, throughput=4.89MB/sec 2021-07-31T02:05:41.957 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-yt9OeUwmeASRhcz off=3062911 len=1884114 2021-07-31T02:05:41.967 INFO:tasks.workunit.client.0.smithi045.stdout:op 168 completed, throughput=4.94MB/sec 2021-07-31T02:05:41.967 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-zyMPlwATaJCIYUa off=9801296 len=116870 2021-07-31T02:05:41.968 INFO:tasks.workunit.client.0.smithi045.stdout:op 169 completed, throughput=4.95MB/sec 2021-07-31T02:05:41.968 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-gBCZZmTEiY0Cnyi off=619893 len=1918875 2021-07-31T02:05:41.977 INFO:tasks.workunit.client.0.smithi045.stdout:op 170 completed, throughput=5MB/sec 2021-07-31T02:05:41.978 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-CcvQblH8ePgs6b7 off=9305631 len=2004590 2021-07-31T02:05:41.981 INFO:tasks.workunit.client.0.smithi045.stdout:op 171 completed, throughput=5.05MB/sec 2021-07-31T02:05:42.981 INFO:tasks.workunit.client.0.smithi045.stdout: 36: throughput=4.91MB/sec pending data=0 2021-07-31T02:05:42.982 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ySnNqmRTAy8Qsfq off=1859695 len=1931649 2021-07-31T02:05:42.983 INFO:tasks.workunit.client.0.smithi045.stdout:op 172 completed, throughput=4.96MB/sec 2021-07-31T02:05:42.984 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-TKteTsP_2wUPFfr off=8762063 len=733348 2021-07-31T02:05:42.989 INFO:tasks.workunit.client.0.smithi045.stdout:op 173 completed, throughput=4.98MB/sec 2021-07-31T02:05:42.989 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-coyBE3gEcEjBPtL off=20997040 len=315333 2021-07-31T02:05:42.992 INFO:tasks.workunit.client.0.smithi045.stdout:op 174 completed, throughput=4.99MB/sec 2021-07-31T02:05:42.992 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-XoLX6dKSWRQG6Zu off=2157476 len=709290 2021-07-31T02:05:42.993 INFO:tasks.workunit.client.0.smithi045.stdout:op 175 completed, throughput=5.01MB/sec 2021-07-31T02:05:43.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:42 smithi154 conmon[34553]: cluster 2021-07-31T02:05:42.052522+0000 mgr.x (mgr.24542) 794 : cluster [DBG] pgmap v1426: 9 pgs: 9 active+clean; 127 GiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.5 MiB/s rd, 2.4 MiB/s wr, 3 op/s 2021-07-31T02:05:43.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:42 smithi045 conmon[39718]: cluster 2021-07-31T02:05:42.052522+0000 mgr.x (mgr.24542) 794 : cluster 2021-07-31T02:05:43.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:42 smithi045 conmon[39718]: [DBG] pgmap v1426: 9 pgs: 9 active+clean; 127 GiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.5 MiB/s rd, 2.4 MiB/s wr, 3 op/s 2021-07-31T02:05:43.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:42 smithi045 conmon[33655]: cluster 2021-07-31T02:05:42.052522+0000 mgr.x (mgr.24542) 794 : 2021-07-31T02:05:43.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:42 smithi045 conmon[33655]: cluster [DBG] pgmap v1426: 9 pgs: 9 active+clean; 127 GiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.5 MiB/s rd, 2.4 MiB/s wr, 3 op/s 2021-07-31T02:05:43.994 INFO:tasks.workunit.client.0.smithi045.stdout: 37: throughput=4.87MB/sec pending data=0 2021-07-31T02:05:43.994 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-zHfvXERZOdlrCBC off=2437560 len=1966188 2021-07-31T02:05:44.007 INFO:tasks.workunit.client.0.smithi045.stdout:op 176 completed, throughput=4.92MB/sec 2021-07-31T02:05:44.007 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-gr9Bd19eNwcPdOe off=760915 len=362267 2021-07-31T02:05:44.010 INFO:tasks.workunit.client.0.smithi045.stdout:op 177 completed, throughput=4.93MB/sec 2021-07-31T02:05:44.010 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-zUJ08OPkKnZ5snV off=1046085 len=1162548 2021-07-31T02:05:44.016 INFO:tasks.workunit.client.0.smithi045.stdout:op 178 completed, throughput=4.96MB/sec 2021-07-31T02:05:44.016 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-3_cIK8fygFHOD5r off=1845195 len=907059 2021-07-31T02:05:44.018 INFO:tasks.workunit.client.0.smithi045.stdout:op 179 completed, throughput=4.98MB/sec 2021-07-31T02:05:44.019 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-7wKXdFWhMjBy9fn off=960146 len=2094218 2021-07-31T02:05:44.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:05:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:05:43.539Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:05:44.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:05:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:05:43.539Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:05:44.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:05:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:05:43.539Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:05:44.029 INFO:tasks.workunit.client.0.smithi045.stdout:op 180 completed, throughput=5.03MB/sec 2021-07-31T02:05:44.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:44 smithi154 conmon[34553]: cluster 2021-07-31T02:05:44.053036+0000 mgr.x (mgr.24542) 795 : cluster [DBG] pgmap v1427: 9 pgs: 9 active+clean; 127 GiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 2.5 MiB/s rd, 3.4 MiB/s wr, 5 op/s 2021-07-31T02:05:44.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:44 smithi045 conmon[39718]: cluster 2021-07-31T02:05:44.053036+0000 mgr.x (mgr.24542) 795 : cluster [DBG] 2021-07-31T02:05:44.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:44 smithi045 conmon[39718]: pgmap v1427: 9 pgs: 9 active+clean; 127 GiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 2.5 MiB/s rd, 3.4 MiB/s wr, 5 op/s 2021-07-31T02:05:44.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:44 smithi045 conmon[33655]: cluster 2021-07-31T02:05:44.053036+0000 mgr.x (mgr.24542) 795 : cluster [DBG] pgmap v1427: 9 pgs: 9 active+clean; 127 GiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 2.5 MiB/s rd, 3.4 MiB/s wr, 5 op/s 2021-07-31T02:05:45.030 INFO:tasks.workunit.client.0.smithi045.stdout: 38: throughput=4.9MB/sec pending data=0 2021-07-31T02:05:45.030 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-1LGsdjHeImDTqBW off=2359227 len=1425514 2021-07-31T02:05:45.038 INFO:tasks.workunit.client.0.smithi045.stdout:op 181 completed, throughput=4.94MB/sec 2021-07-31T02:05:45.038 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-tUalvqURPP-qKsb off=8696679 len=1858542 2021-07-31T02:05:45.044 INFO:tasks.workunit.client.0.smithi045.stdout:op 182 completed, throughput=4.98MB/sec 2021-07-31T02:05:45.045 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-CmiS_6svk9jElVg off=7443776 len=1144462 2021-07-31T02:05:45.046 INFO:tasks.workunit.client.0.smithi045.stdout:op 183 completed, throughput=5.01MB/sec 2021-07-31T02:05:46.046 INFO:tasks.workunit.client.0.smithi045.stdout: 39: throughput=4.88MB/sec pending data=0 2021-07-31T02:05:46.046 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-0fp96ZiJoo9Jekm off=1050421 len=1871343 2021-07-31T02:05:46.049 INFO:tasks.workunit.client.0.smithi045.stdout:op 184 completed, throughput=4.93MB/sec 2021-07-31T02:05:46.050 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-nls-6uHGq7Fp_bo off=3802338 len=1661862 2021-07-31T02:05:46.058 INFO:tasks.workunit.client.0.smithi045.stdout:op 185 completed, throughput=4.97MB/sec 2021-07-31T02:05:46.058 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-I0s09yVbsOwhgp3 off=15591571 len=1553237 2021-07-31T02:05:46.064 INFO:tasks.workunit.client.0.smithi045.stdout:op 186 completed, throughput=5MB/sec 2021-07-31T02:05:46.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:46 smithi154 conmon[34553]: cluster 2021-07-31T02:05:46.053412+0000 mgr.x (mgr.24542) 796 : cluster [DBG] pgmap v1428: 9 pgs: 9 active+clean; 127 GiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.0 MiB/s wr, 3 op/s 2021-07-31T02:05:46.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:46 smithi045 conmon[33655]: cluster 2021-07-31T02:05:46.053412+0000 mgr.x (mgr.24542) 796 : cluster [DBG] pgmap v1428: 9 pgs: 9 active+clean; 127 GiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.0 MiB/s wr, 3 op/s 2021-07-31T02:05:46.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:46 smithi045 conmon[39718]: cluster 2021-07-31T02:05:46.053412+0000 mgr.x ( 2021-07-31T02:05:46.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:46 smithi045 conmon[39718]: mgr.24542) 796 : cluster [DBG] pgmap v1428: 9 pgs: 9 active+clean; 127 GiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.0 MiB/s wr, 3 op/s 2021-07-31T02:05:47.065 INFO:tasks.workunit.client.0.smithi045.stdout: 40: throughput=4.88MB/sec pending data=0 2021-07-31T02:05:47.065 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-AsT7dxYKs8JaOyL off=1969317 len=1760973 2021-07-31T02:05:47.074 INFO:tasks.workunit.client.0.smithi045.stdout:op 187 completed, throughput=4.92MB/sec 2021-07-31T02:05:47.075 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-5o5IvOj9E6fYjTm off=4430576 len=25524 2021-07-31T02:05:47.075 INFO:tasks.workunit.client.0.smithi045.stdout:op 188 completed, throughput=4.92MB/sec 2021-07-31T02:05:47.075 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-I8CIWJqDA1x75Af off=1992857 len=1480580 2021-07-31T02:05:47.078 INFO:tasks.workunit.client.0.smithi045.stdout:op 189 completed, throughput=4.96MB/sec 2021-07-31T02:05:47.079 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-9mASJSI6nISBwn6 off=18314201 len=991606 2021-07-31T02:05:47.084 INFO:tasks.workunit.client.0.smithi045.stdout:op 190 completed, throughput=4.98MB/sec 2021-07-31T02:05:47.084 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-vdYcov9Bu0kr02o off=1107312 len=1529317 2021-07-31T02:05:47.084 INFO:tasks.workunit.client.0.smithi045.stdout:op 191 completed, throughput=5.01MB/sec 2021-07-31T02:05:48.083 INFO:tasks.workunit.client.0.smithi045.stdout: 41: throughput=4.89MB/sec pending data=0 2021-07-31T02:05:48.084 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-jw3hzc_5ijVwzy4 off=8905257 len=384405 2021-07-31T02:05:48.086 INFO:tasks.workunit.client.0.smithi045.stdout:op 192 completed, throughput=4.9MB/sec 2021-07-31T02:05:48.086 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-5YmezBOLXZfGclL off=8653020 len=429580 2021-07-31T02:05:48.089 INFO:tasks.workunit.client.0.smithi045.stdout:op 193 completed, throughput=4.91MB/sec 2021-07-31T02:05:48.090 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-KC_-eutS8MogMIX off=15691072 len=797709 2021-07-31T02:05:48.106 INFO:tasks.workunit.client.0.smithi045.stdout:op 194 completed, throughput=4.93MB/sec 2021-07-31T02:05:48.107 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-AksugRhbKh_rXTE off=1833659 len=182843 2021-07-31T02:05:48.107 INFO:tasks.workunit.client.0.smithi045.stdout:op 195 completed, throughput=4.93MB/sec 2021-07-31T02:05:48.108 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-kLLvAcqJpVN3RxI off=9417071 len=1137363 2021-07-31T02:05:48.108 INFO:tasks.workunit.client.0.smithi045.stdout:op 196 completed, throughput=4.96MB/sec 2021-07-31T02:05:48.108 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-O9ImbqorBtjn0gj off=3830072 len=971375 2021-07-31T02:05:48.109 INFO:tasks.workunit.client.0.smithi045.stdout:op 197 completed, throughput=4.98MB/sec 2021-07-31T02:05:48.109 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-zUKrd9eZFOAacz4 off=5268992 len=383381 2021-07-31T02:05:48.110 INFO:tasks.workunit.client.0.smithi045.stdout:op 198 completed, throughput=4.99MB/sec 2021-07-31T02:05:48.110 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ewmGP3_voToyKyY off=5252717 len=1042917 2021-07-31T02:05:48.111 INFO:tasks.workunit.client.0.smithi045.stdout:op 199 completed, throughput=5.01MB/sec 2021-07-31T02:05:48.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:48 smithi154 conmon[34553]: cluster 2021-07-31T02:05:48.053996+0000 mgr.x (mgr.24542) 797 : cluster [DBG] pgmap v1429: 9 pgs: 9 active+clean; 127 GiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.0 MiB/s wr, 3 op/s 2021-07-31T02:05:48.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:48 smithi045 conmon[39718]: cluster 2021-07-31T02:05:48.053996+0000 mgr.x 2021-07-31T02:05:48.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:48 smithi045 conmon[39718]: (mgr.24542) 797 : cluster [DBG] pgmap v1429: 9 pgs: 9 active+clean; 127 GiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.0 MiB/s wr, 3 op/s 2021-07-31T02:05:48.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:48 smithi045 conmon[33655]: cluster 2021-07-31T02:05:48.053996+0000 mgr.x ( 2021-07-31T02:05:48.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:48 smithi045 conmon[33655]: mgr.24542) 797 : cluster [DBG] pgmap v1429: 9 pgs: 9 active+clean; 127 GiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.0 MiB/s wr, 3 op/s 2021-07-31T02:05:49.108 INFO:tasks.workunit.client.0.smithi045.stdout: 42: throughput=4.9MB/sec pending data=0 2021-07-31T02:05:49.108 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-UeHfjQEp6cZV-Es off=7557260 len=1751633 2021-07-31T02:05:49.111 INFO:tasks.workunit.client.0.smithi045.stdout:op 200 completed, throughput=4.93MB/sec 2021-07-31T02:05:49.112 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-q6ZbbqatJsqclTp off=2927681 len=990376 2021-07-31T02:05:49.114 INFO:tasks.workunit.client.0.smithi045.stdout:op 201 completed, throughput=4.96MB/sec 2021-07-31T02:05:49.114 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Mx3GAimzRBcR84L off=18567359 len=2019922 2021-07-31T02:05:49.124 INFO:tasks.workunit.client.0.smithi045.stdout:op 202 completed, throughput=5MB/sec 2021-07-31T02:05:49.125 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-jj8j_PKfnw-fQoF off=12425 len=884831 2021-07-31T02:05:49.129 INFO:tasks.workunit.client.0.smithi045.stdout:op 203 completed, throughput=5.02MB/sec 2021-07-31T02:05:50.129 INFO:tasks.workunit.client.0.smithi045.stdout: 43: throughput=4.9MB/sec pending data=0 2021-07-31T02:05:50.130 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-PTurmwpbhYgY9qe off=865476 len=1634508 2021-07-31T02:05:50.136 INFO:tasks.workunit.client.0.smithi045.stdout:op 204 completed, throughput=4.94MB/sec 2021-07-31T02:05:50.137 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-7T4PTYIaPq7IS7Z off=7130706 len=450188 2021-07-31T02:05:50.140 INFO:tasks.workunit.client.0.smithi045.stdout:op 205 completed, throughput=4.95MB/sec 2021-07-31T02:05:50.140 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-1xBca2f6GWprRpb off=11428437 len=1155811 2021-07-31T02:05:50.142 INFO:tasks.workunit.client.0.smithi045.stdout:op 206 completed, throughput=4.97MB/sec 2021-07-31T02:05:50.143 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-4H62YZXJeTdqhjy off=8381454 len=1228676 2021-07-31T02:05:50.148 INFO:tasks.workunit.client.0.smithi045.stdout:op 207 completed, throughput=5MB/sec 2021-07-31T02:05:50.148 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-NMe_FrbLPqG2yhW off=1087303 len=1107882 2021-07-31T02:05:50.151 INFO:tasks.workunit.client.0.smithi045.stdout:op 208 completed, throughput=5.02MB/sec 2021-07-31T02:05:50.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:50 smithi154 conmon[34553]: cluster 2021-07-31T02:05:50.054614+0000 mgr.x (mgr.24542) 798 : cluster [DBG] pgmap v1430: 9 pgs: 9 active+clean; 127 GiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 2.7 MiB/s rd, 3.3 MiB/s wr, 5 op/s 2021-07-31T02:05:50.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:50 smithi045 conmon[39718]: cluster 2021-07-31T02:05:50.054614+0000 2021-07-31T02:05:50.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:50 smithi045 conmon[39718]: mgr.x (mgr.24542) 798 : cluster [DBG] pgmap v1430: 9 pgs: 9 active+clean; 127 GiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 2.7 MiB/s rd, 3.3 MiB/s wr, 5 op/s 2021-07-31T02:05:50.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:50 smithi045 conmon[33655]: cluster 2021-07-31T02:05:50.054614 2021-07-31T02:05:50.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:50 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 798 : cluster [DBG] pgmap v1430: 9 pgs: 9 active+clean; 127 GiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 2.7 MiB/s rd, 3.3 MiB/s wr, 5 op/s 2021-07-31T02:05:51.152 INFO:tasks.workunit.client.0.smithi045.stdout: 44: throughput=4.91MB/sec pending data=0 2021-07-31T02:05:51.152 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-V8unBdA4JBKJhVj off=13776626 len=1960161 2021-07-31T02:05:51.156 INFO:tasks.workunit.client.0.smithi045.stdout:op 209 completed, throughput=4.95MB/sec 2021-07-31T02:05:51.156 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Dv45_m6MLjwFJnt off=5580104 len=2003621 2021-07-31T02:05:51.160 INFO:tasks.workunit.client.0.smithi045.stdout:op 210 completed, throughput=4.99MB/sec 2021-07-31T02:05:51.160 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-pvUCbZX_Lx7hycE off=9160464 len=588585 2021-07-31T02:05:51.164 INFO:tasks.workunit.client.0.smithi045.stdout:op 211 completed, throughput=5.01MB/sec 2021-07-31T02:05:52.165 INFO:tasks.workunit.client.0.smithi045.stdout: 45: throughput=4.9MB/sec pending data=0 2021-07-31T02:05:52.165 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-mA5Cllv814Ewq5x off=6985434 len=442151 2021-07-31T02:05:52.166 INFO:tasks.workunit.client.0.smithi045.stdout:op 212 completed, throughput=4.91MB/sec 2021-07-31T02:05:52.166 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-W-2btogXK6LMo66 off=4924649 len=696984 2021-07-31T02:05:52.171 INFO:tasks.workunit.client.0.smithi045.stdout:op 213 completed, throughput=4.92MB/sec 2021-07-31T02:05:52.171 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-aLV-iKHDlKVyJhf off=23276957 len=1767628 2021-07-31T02:05:52.174 INFO:tasks.workunit.client.0.smithi045.stdout:op 214 completed, throughput=4.96MB/sec 2021-07-31T02:05:52.174 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-NzRTQLlxDhFF9j8 off=5071319 len=1323898 2021-07-31T02:05:52.177 INFO:tasks.workunit.client.0.smithi045.stdout:op 215 completed, throughput=4.98MB/sec 2021-07-31T02:05:52.177 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-71-k2eKlll7AdF_ off=5663062 len=1823364 2021-07-31T02:05:52.187 INFO:tasks.workunit.client.0.smithi045.stdout:op 216 completed, throughput=5.02MB/sec 2021-07-31T02:05:53.188 INFO:tasks.workunit.client.0.smithi045.stdout: 46: throughput=4.91MB/sec pending data=0 2021-07-31T02:05:53.188 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-obaaTd8RSaqnzD1 off=3800195 len=559695 2021-07-31T02:05:53.192 INFO:tasks.workunit.client.0.smithi045.stdout:op 217 completed, throughput=4.92MB/sec 2021-07-31T02:05:53.192 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ZjaVv8izkhzdEVh off=8049563 len=305600 2021-07-31T02:05:53.195 INFO:tasks.workunit.client.0.smithi045.stdout:op 218 completed, throughput=4.93MB/sec 2021-07-31T02:05:53.195 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-SvEXE3X2jyr_svC off=6521528 len=1059377 2021-07-31T02:05:53.197 INFO:tasks.workunit.client.0.smithi045.stdout:op 219 completed, throughput=4.95MB/sec 2021-07-31T02:05:53.198 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-7rOT6Q5pYEFCQS0 off=4685162 len=718400 2021-07-31T02:05:53.199 INFO:tasks.workunit.client.0.smithi045.stdout:op 220 completed, throughput=4.97MB/sec 2021-07-31T02:05:53.200 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-gYTwLTmDklJRPOH off=4249319 len=636371 2021-07-31T02:05:53.201 INFO:tasks.workunit.client.0.smithi045.stdout:op 221 completed, throughput=4.98MB/sec 2021-07-31T02:05:53.201 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-HbwKiUZVfytkjCe off=5068688 len=1948192 2021-07-31T02:05:53.209 INFO:tasks.workunit.client.0.smithi045.stdout:op 222 completed, throughput=5.02MB/sec 2021-07-31T02:05:53.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:52 smithi154 conmon[34553]: cluster 2021-07-31T02:05:52.055024+0000 mgr.x (mgr.24542) 799 : cluster [DBG] pgmap v1431: 9 pgs: 9 active+clean; 127 GiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 3 op/s 2021-07-31T02:05:53.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:52 smithi045 conmon[39718]: cluster 2021-07-31T02:05:52. 2021-07-31T02:05:53.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:52 smithi045 conmon[39718]: 055024+0000 mgr.x (mgr.24542) 799 : cluster [DBG] pgmap v1431: 9 pgs: 9 active+clean; 127 GiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 3 op/s 2021-07-31T02:05:53.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:52 smithi045 conmon[33655]: cluster 2021-07-31T02:05:52.055024+0000 mgr.x ( 2021-07-31T02:05:53.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:52 smithi045 conmon[33655]: mgr.24542) 799 : cluster [DBG] pgmap v1431: 9 pgs: 9 active+clean; 127 GiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 3 op/s 2021-07-31T02:05:54.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:05:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:05:53.539Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:05:54.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:05:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:05:53.539Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:05:54.027 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:05:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:05:53.539Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:05:54.210 INFO:tasks.workunit.client.0.smithi045.stdout: 47: throughput=4.91MB/sec pending data=0 2021-07-31T02:05:54.210 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-kKyf6kWa4f1svmT off=744834 len=1967295 2021-07-31T02:05:54.221 INFO:tasks.workunit.client.0.smithi045.stdout:op 223 completed, throughput=4.95MB/sec 2021-07-31T02:05:54.222 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-r228jZLa3wlS_Mp off=2021029 len=45537 2021-07-31T02:05:54.223 INFO:tasks.workunit.client.0.smithi045.stdout:op 224 completed, throughput=4.95MB/sec 2021-07-31T02:05:54.223 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-kjjDFLtHrVx5xcc off=6644169 len=151816 2021-07-31T02:05:54.225 INFO:tasks.workunit.client.0.smithi045.stdout:op 225 completed, throughput=4.95MB/sec 2021-07-31T02:05:54.226 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-WAWikBp8vyCTksD off=11680589 len=655793 2021-07-31T02:05:54.230 INFO:tasks.workunit.client.0.smithi045.stdout:op 226 completed, throughput=4.97MB/sec 2021-07-31T02:05:54.230 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-1eKV2r2rXuocl42 off=3623692 len=1027239 2021-07-31T02:05:54.237 INFO:tasks.workunit.client.0.smithi045.stdout:op 227 completed, throughput=4.99MB/sec 2021-07-31T02:05:54.237 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-0z5zJHX90ZF_zyR off=12348093 len=427496 2021-07-31T02:05:54.240 INFO:tasks.workunit.client.0.smithi045.stdout:op 228 completed, throughput=4.99MB/sec 2021-07-31T02:05:54.240 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-XYYI2rvzVaxpfwA off=3679182 len=201937 2021-07-31T02:05:54.243 INFO:tasks.workunit.client.0.smithi045.stdout:op 229 completed, throughput=5MB/sec 2021-07-31T02:05:54.244 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-WqGuOf_Wt7YcQQ_ off=12391361 len=1510283 2021-07-31T02:05:54.247 INFO:tasks.workunit.client.0.smithi045.stdout:op 230 completed, throughput=5.03MB/sec 2021-07-31T02:05:54.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:54 smithi154 conmon[34553]: cluster 2021-07-31T02:05:54.055617+0000 mgr.x (mgr.24542) 800 : cluster [DBG] pgmap v1432: 9 pgs: 9 active+clean; 127 GiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 3.0 MiB/s rd, 3.3 MiB/s wr, 5 op/s 2021-07-31T02:05:54.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:54 smithi154 conmon[34553]: 2021-07-31T02:05:54.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:54 smithi045 conmon[39718]: cluster 2021-07-31T02:05:54.055617+0000 mgr.x (mgr 2021-07-31T02:05:54.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:54 smithi045 conmon[39718]: .24542) 800 : cluster [DBG] pgmap v1432: 9 pgs: 9 active+clean; 127 GiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 3.0 MiB/s rd, 3.3 MiB/s wr, 5 op/s 2021-07-31T02:05:54.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:54 smithi045 conmon[33655]: cluster 2021-07-31T02:05:54.055617+0000 mgr.x (mgr.24542) 800 : cluster [DBG] pgmap v1432: 9 pgs: 9 active+clean; 127 GiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 3.0 MiB/s rd, 3.3 MiB/s wr, 5 op/s 2021-07-31T02:05:55.247 INFO:tasks.workunit.client.0.smithi045.stdout: 48: throughput=4.93MB/sec pending data=0 2021-07-31T02:05:55.247 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-XvHGXOoCRh0Orpc off=4703481 len=1709429 2021-07-31T02:05:55.249 INFO:tasks.workunit.client.0.smithi045.stdout:op 231 completed, throughput=4.96MB/sec 2021-07-31T02:05:55.249 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-bthAZXHrcQXUuBr off=6281670 len=349582 2021-07-31T02:05:55.249 INFO:tasks.workunit.client.0.smithi045.stdout:op 232 completed, throughput=4.96MB/sec 2021-07-31T02:05:55.250 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-EZRHGnZPUbpPdDq off=3725958 len=68516 2021-07-31T02:05:55.250 INFO:tasks.workunit.client.0.smithi045.stdout:op 233 completed, throughput=4.97MB/sec 2021-07-31T02:05:55.251 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-6SKJ9d9CbwSFS_o off=2104317 len=145914 2021-07-31T02:05:55.251 INFO:tasks.workunit.client.0.smithi045.stdout:op 234 completed, throughput=4.97MB/sec 2021-07-31T02:05:55.251 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-UTNh45giVZI3VBR off=10105438 len=1842868 2021-07-31T02:05:55.261 INFO:tasks.workunit.client.0.smithi045.stdout:op 235 completed, throughput=5MB/sec 2021-07-31T02:05:56.262 INFO:tasks.workunit.client.0.smithi045.stdout: 49: throughput=4.9MB/sec pending data=0 2021-07-31T02:05:56.262 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-i5Ssl5TegCUDQWB off=5495854 len=1141542 2021-07-31T02:05:56.269 INFO:tasks.workunit.client.0.smithi045.stdout:op 236 completed, throughput=4.92MB/sec 2021-07-31T02:05:56.269 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-R2cpbEdL12emhVP off=2957247 len=1191675 2021-07-31T02:05:56.275 INFO:tasks.workunit.client.0.smithi045.stdout:op 237 completed, throughput=4.95MB/sec 2021-07-31T02:05:56.275 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Rx1t1nEk_UE38XU off=6954737 len=1917064 2021-07-31T02:05:56.277 INFO:tasks.workunit.client.0.smithi045.stdout:op 238 completed, throughput=4.98MB/sec 2021-07-31T02:05:56.277 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-X5eO_iwrlVUujuN off=47575 len=1857198 2021-07-31T02:05:56.285 INFO:tasks.workunit.client.0.smithi045.stdout:op 239 completed, throughput=5.02MB/sec 2021-07-31T02:05:56.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:56 smithi154 conmon[34553]: cluster 2021-07-31T02:05:56.056022+0000 mgr.x (mgr.24542) 801 : cluster [DBG] pgmap v1433: 9 pgs: 9 active+clean; 127 GiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.2 MiB/s wr, 3 op/s 2021-07-31T02:05:56.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:56 smithi045 conmon[39718]: cluster 2021-07-31T02:05 2021-07-31T02:05:56.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:56 smithi045 conmon[39718]: :56.056022+0000 mgr.x (mgr.24542) 801 : cluster [DBG] pgmap v1433: 9 pgs: 9 active+clean; 127 GiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.2 MiB/s wr, 3 op/s 2021-07-31T02:05:56.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:56 smithi045 conmon[33655]: cluster 2021-07-31T02:05:56.056022+0000 mgr.x (mgr.24542) 2021-07-31T02:05:56.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:56 smithi045 conmon[33655]: 801 : cluster [DBG] pgmap v1433: 9 pgs: 9 active+clean; 127 GiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.2 MiB/s wr, 3 op/s 2021-07-31T02:05:57.286 INFO:tasks.workunit.client.0.smithi045.stdout: 50: throughput=4.92MB/sec pending data=0 2021-07-31T02:05:57.286 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-78Px3gdzusAfn8V off=15381127 len=488147 2021-07-31T02:05:57.290 INFO:tasks.workunit.client.0.smithi045.stdout:op 240 completed, throughput=4.93MB/sec 2021-07-31T02:05:57.290 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-kYg9s11bbSsv4Dq off=3654749 len=842950 2021-07-31T02:05:57.295 INFO:tasks.workunit.client.0.smithi045.stdout:op 241 completed, throughput=4.94MB/sec 2021-07-31T02:05:57.295 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-P2tbMElE4MPO_Pq off=581689 len=1539446 2021-07-31T02:05:57.301 INFO:tasks.workunit.client.0.smithi045.stdout:op 242 completed, throughput=4.97MB/sec 2021-07-31T02:05:57.302 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-6VRSAkjtIZvXzcC off=3530105 len=2064009 2021-07-31T02:05:57.305 INFO:tasks.workunit.client.0.smithi045.stdout:op 243 completed, throughput=5.01MB/sec 2021-07-31T02:05:58.306 INFO:tasks.workunit.client.0.smithi045.stdout: 51: throughput=4.91MB/sec pending data=0 2021-07-31T02:05:58.306 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-26r3pJcnC_Mq670 off=1354379 len=1850467 2021-07-31T02:05:58.317 INFO:tasks.workunit.client.0.smithi045.stdout:op 244 completed, throughput=4.95MB/sec 2021-07-31T02:05:58.317 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-LlL9cqfkdyg8JFH off=15399890 len=134144 2021-07-31T02:05:58.320 INFO:tasks.workunit.client.0.smithi045.stdout:op 245 completed, throughput=4.95MB/sec 2021-07-31T02:05:58.320 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-uNmhLzAjixdFgt2 off=1427190 len=1082999 2021-07-31T02:05:58.325 INFO:tasks.workunit.client.0.smithi045.stdout:op 246 completed, throughput=4.97MB/sec 2021-07-31T02:05:58.326 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ZjOfjxUICp9XJ7z off=597016 len=867385 2021-07-31T02:05:58.330 INFO:tasks.workunit.client.0.smithi045.stdout:op 247 completed, throughput=4.98MB/sec 2021-07-31T02:05:58.330 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-0pG8Rcnmw7X27pD off=6535661 len=1723395 2021-07-31T02:05:58.333 INFO:tasks.workunit.client.0.smithi045.stdout:op 248 completed, throughput=5.01MB/sec 2021-07-31T02:05:58.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:05:58 smithi154 conmon[34553]: cluster 2021-07-31T02:05:58.056567+0000 mgr.x (mgr.24542) 802 : cluster [DBG] pgmap v1434: 9 pgs: 9 active+clean; 127 GiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.2 MiB/s wr, 3 op/s 2021-07-31T02:05:58.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:58 smithi045 conmon[33655]: cluster 2021-07-31T02:05:58.056567+0000 mgr.x ( 2021-07-31T02:05:58.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:05:58 smithi045 conmon[33655]: mgr.24542) 802 : cluster [DBG] pgmap v1434: 9 pgs: 9 active+clean; 127 GiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.2 MiB/s wr, 3 op/s 2021-07-31T02:05:58.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:58 smithi045 conmon[39718]: cluster 2021-07-31T02:05:58.056567+0000 mgr.x 2021-07-31T02:05:58.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:05:58 smithi045 conmon[39718]: (mgr.24542) 802 : cluster [DBG] pgmap v1434: 9 pgs: 9 active+clean; 127 GiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.2 MiB/s wr, 3 op/s 2021-07-31T02:05:59.333 INFO:tasks.workunit.client.0.smithi045.stdout: 52: throughput=4.92MB/sec pending data=0 2021-07-31T02:05:59.333 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-d6lNhKb6pwBZW9z off=177746 len=701531 2021-07-31T02:05:59.335 INFO:tasks.workunit.client.0.smithi045.stdout:op 249 completed, throughput=4.93MB/sec 2021-07-31T02:05:59.335 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-wRaYAAsXRMPg_-n off=8606969 len=662189 2021-07-31T02:05:59.340 INFO:tasks.workunit.client.0.smithi045.stdout:op 250 completed, throughput=4.94MB/sec 2021-07-31T02:05:59.341 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-FYhFPLVZZQwo4Q8 off=9309788 len=905734 2021-07-31T02:05:59.342 INFO:tasks.workunit.client.0.smithi045.stdout:op 251 completed, throughput=4.96MB/sec 2021-07-31T02:05:59.342 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Jc_dbmpTrQtw6bc off=1345812 len=36097 2021-07-31T02:05:59.345 INFO:tasks.workunit.client.0.smithi045.stdout:op 252 completed, throughput=4.96MB/sec 2021-07-31T02:05:59.345 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-VTQz_7xMBmc5RI1 off=4293148 len=1880625 2021-07-31T02:05:59.347 INFO:tasks.workunit.client.0.smithi045.stdout:op 253 completed, throughput=4.99MB/sec 2021-07-31T02:05:59.348 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-KAimJ9bD6zrEw34 off=115307 len=1406036 2021-07-31T02:05:59.349 INFO:tasks.workunit.client.0.smithi045.stdout:op 254 completed, throughput=5.02MB/sec 2021-07-31T02:05:59.851 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:06:00.131 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:06:00.131 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 20m ago 21m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:06:00.131 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 18m ago 21m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:06:00.132 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 18m ago 24m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:06:00.132 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 20m ago 27m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:06:00.132 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 20m ago 27m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:06:00.132 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 18m ago 25m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:06:00.133 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 20m ago 25m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:06:00.133 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 20m ago 21m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:06:00.133 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 18m ago 21m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:06:00.133 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 20m ago 24m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:06:00.133 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 20m ago 24m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:06:00.134 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 20m ago 23m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:06:00.134 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 20m ago 23m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:06:00.134 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 18m ago 23m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:06:00.134 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 18m ago 22m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:06:00.134 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 18m ago 22m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:06:00.135 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 18m ago 22m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:06:00.135 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 18m ago 21m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:06:00.348 INFO:tasks.workunit.client.0.smithi045.stdout: 53: throughput=4.93MB/sec pending data=0 2021-07-31T02:06:00.349 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-mESGi0iGeddSKBv off=14029454 len=1583432 2021-07-31T02:06:00.349 INFO:tasks.workunit.client.0.smithi045.stdout:op 255 completed, throughput=4.95MB/sec 2021-07-31T02:06:00.350 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-_VhsLc1eG2XOUPH off=5406770 len=1436428 2021-07-31T02:06:00.352 INFO:tasks.workunit.client.0.smithi045.stdout:op 256 completed, throughput=4.98MB/sec 2021-07-31T02:06:00.353 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-7RN8GK_U9_R6VAn off=4179530 len=290286 2021-07-31T02:06:00.353 INFO:tasks.workunit.client.0.smithi045.stdout:op 257 completed, throughput=4.98MB/sec 2021-07-31T02:06:00.354 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-hQPeTBA4kLM1WAi off=12788238 len=1707220 2021-07-31T02:06:00.361 INFO:tasks.workunit.client.0.smithi045.stdout:op 258 completed, throughput=5.01MB/sec 2021-07-31T02:06:00.472 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:06:00.472 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:06:00.473 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:06:00.473 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:06:00.473 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:06:00.473 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:06:00.473 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:06:00.474 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:06:00.474 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:06:00.474 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:06:00.475 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:06:00.475 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:06:00.475 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:06:00.475 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:06:00.475 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:06:00.476 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:06:00.476 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:06:00.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:00 smithi154 conmon[34553]: audit 2021-07-31T02:05:59.838882+0000 mgr.x (mgr.24542) 803 : audit [DBG] from='client.15753 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:06:00.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:00 smithi154 conmon[34553]: cluster 2021-07-31T02:06:00. 2021-07-31T02:06:00.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:00 smithi154 conmon[34553]: 057069+0000 mgr.x (mgr.24542) 804 : cluster [DBG] pgmap v1435: 9 pgs: 9 active+clean; 127 GiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 2.8 MiB/s rd, 3.6 MiB/s wr, 6 op/s 2021-07-31T02:06:00.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:00 smithi045 conmon[39718]: audit 2021-07-31T02: 2021-07-31T02:06:00.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:00 smithi045 conmon[39718]: 05:59.838882+0000 mgr.x (mgr.24542) 803 : audit [DBG] from='client.15753 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:06:00.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:00 smithi045 conmon[39718]: cluster 2021 2021-07-31T02:06:00.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:00 smithi045 conmon[39718]: -07-31T02:06:00. 2021-07-31T02:06:00.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:00 smithi045 conmon[39718]: 057069+0000 mgr.x (mgr.24542 2021-07-31T02:06:00.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:00 smithi045 conmon[39718]: ) 804 : cluster [DBG] pgmap v1435: 9 pgs: 9 active+clean; 127 GiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 2.8 MiB/s rd, 3.6 MiB/s wr, 6 op/s 2021-07-31T02:06:00.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:00 smithi045 conmon[33655]: audit 2021-07-31T02:05:59.838882+0000 mgr.x (mgr.24542) 803 : audit 2021-07-31T02:06:00.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:00 smithi045 conmon[33655]: [DBG] from='client.15753 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:06:00.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:00 smithi045 conmon[33655]: cluster 2021-07-31T02:06:00.057069+0000 mgr.x (mgr.24542) 804 : cluster [DBG] pgmap v1435: 9 pgs: 9 active+clean; 127 GiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 2.8 MiB/s rd, 3.6 MiB/s wr, 6 op/s 2021-07-31T02:06:01.361 INFO:tasks.workunit.client.0.smithi045.stdout: 54: throughput=4.92MB/sec pending data=0 2021-07-31T02:06:01.362 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-A8Uj5rJFHWoU9xU off=4842785 len=323625 2021-07-31T02:06:01.365 INFO:tasks.workunit.client.0.smithi045.stdout:op 259 completed, throughput=4.93MB/sec 2021-07-31T02:06:01.365 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-XmrvbJ933-ImEJO off=136641 len=1138239 2021-07-31T02:06:01.377 INFO:tasks.workunit.client.0.smithi045.stdout:op 260 completed, throughput=4.95MB/sec 2021-07-31T02:06:01.377 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-c7LJZgvo80SZVKJ off=2427649 len=729276 2021-07-31T02:06:01.382 INFO:tasks.workunit.client.0.smithi045.stdout:op 261 completed, throughput=4.96MB/sec 2021-07-31T02:06:01.382 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-UVrwu22dj7o061j off=10805283 len=1862900 2021-07-31T02:06:01.390 INFO:tasks.workunit.client.0.smithi045.stdout:op 262 completed, throughput=4.99MB/sec 2021-07-31T02:06:01.390 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-i1J8N_HUvmnvS2d off=17784484 len=380565 2021-07-31T02:06:01.391 INFO:tasks.workunit.client.0.smithi045.stdout:op 263 completed, throughput=5MB/sec 2021-07-31T02:06:01.391 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-XjmQmirruPBM3IK off=4084214 len=1622088 2021-07-31T02:06:01.394 INFO:tasks.workunit.client.0.smithi045.stdout:op 264 completed, throughput=5.02MB/sec 2021-07-31T02:06:01.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:01 smithi154 conmon[34553]: audit 2021-07-31T02:06:00.120241+0000 mgr.x (mgr.24542) 805 : audit [DBG] from='client.15759 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:06:01.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:01 smithi154 conmon[34553]: audit 2021-07-31T02:06:00.471879+0000 2021-07-31T02:06:01.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:01 smithi154 conmon[34553]: mon.a (mon.0) 1851 : audit [DBG] from='client.? 172.21.15.45:0/1032781988' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:06:01.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:01 smithi045 conmon[33655]: audit 2021-07-31T02:06:00.120241+0000 mgr.x (mgr.24542) 805 : audit [DBG] from='client.15759 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:06:01.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:01 smithi045 conmon[33655]: audit 2021-07-31T02:06:00.471879+0000 mon.a (mon.0) 1851 : audit [DBG] from='client.? 172.21.15.45:0/1032781988' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:06:01.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:01 smithi045 conmon[39718]: audit 2021-07-31T02:06:00 2021-07-31T02:06:01.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:01 smithi045 conmon[39718]: .120241+0000 mgr.x (mgr.24542) 805 : audit [DBG] from='client.15759 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:06:01.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:01 smithi045 conmon[39718]: audit 2021-07-31T02:06:00 2021-07-31T02:06:01.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:01 smithi045 conmon[39718]: .471879+0000 mon.a (mon.0) 1851 : audit 2021-07-31T02:06:01.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:01 smithi045 conmon[39718]: [DBG] from='client.? 172.21.15.45:0/1032781988' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:06:02.395 INFO:tasks.workunit.client.0.smithi045.stdout: 55: throughput=4.93MB/sec pending data=0 2021-07-31T02:06:02.396 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj--mqqbg3twZjo4z_ off=292288 len=1655439 2021-07-31T02:06:02.404 INFO:tasks.workunit.client.0.smithi045.stdout:op 265 completed, throughput=4.96MB/sec 2021-07-31T02:06:02.404 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-K2LSEHa3qJHnVXA off=19604400 len=355370 2021-07-31T02:06:02.408 INFO:tasks.workunit.client.0.smithi045.stdout:op 266 completed, throughput=4.97MB/sec 2021-07-31T02:06:02.408 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-hWz8emgWx0F8zTj off=4744666 len=84831 2021-07-31T02:06:02.409 INFO:tasks.workunit.client.0.smithi045.stdout:op 267 completed, throughput=4.97MB/sec 2021-07-31T02:06:02.409 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-eq9Hk8NHIC2u23Y off=238241 len=822319 2021-07-31T02:06:02.409 INFO:tasks.workunit.client.0.smithi045.stdout:op 268 completed, throughput=4.98MB/sec 2021-07-31T02:06:02.410 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-bQs-U-_D-OmJc4T off=1748823 len=361892 2021-07-31T02:06:02.413 INFO:tasks.workunit.client.0.smithi045.stdout:op 269 completed, throughput=4.99MB/sec 2021-07-31T02:06:02.413 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-CQVxMzGijLJ80pm off=17378707 len=1342978 2021-07-31T02:06:02.415 INFO:tasks.workunit.client.0.smithi045.stdout:op 270 completed, throughput=5.01MB/sec 2021-07-31T02:06:03.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:02 smithi154 conmon[34553]: cluster 2021-07-31T02:06:02.057478+0000 mgr.x (mgr.24542) 806 : cluster [DBG] pgmap v1436: 9 pgs: 9 active+clean; 127 GiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2021-07-31T02:06:03.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:02 smithi045 conmon[33655]: cluster 2021-07-31T02:06:02.057478+0000 mgr.x (mgr.24542) 806 : cluster [DBG] pgmap v1436: 9 pgs: 9 active+clean; 127 GiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2021-07-31T02:06:03.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:02 smithi045 conmon[39718]: cluster 2021-07-31T02:06:02.057478+0000 mgr.x 2021-07-31T02:06:03.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:02 smithi045 conmon[39718]: (mgr.24542) 806 : cluster [DBG] pgmap v1436: 9 pgs: 9 active+clean; 127 GiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2021-07-31T02:06:03.416 INFO:tasks.workunit.client.0.smithi045.stdout: 56: throughput=4.92MB/sec pending data=0 2021-07-31T02:06:03.417 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-KUYhaX8KiCq-geI off=2505314 len=1822265 2021-07-31T02:06:03.425 INFO:tasks.workunit.client.0.smithi045.stdout:op 271 completed, throughput=4.95MB/sec 2021-07-31T02:06:03.425 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-VvPXuptRnauYts7 off=7897177 len=621259 2021-07-31T02:06:03.429 INFO:tasks.workunit.client.0.smithi045.stdout:op 272 completed, throughput=4.96MB/sec 2021-07-31T02:06:03.429 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-mucv3x_QWmfGdXP off=5685115 len=1170398 2021-07-31T02:06:03.431 INFO:tasks.workunit.client.0.smithi045.stdout:op 273 completed, throughput=4.98MB/sec 2021-07-31T02:06:03.432 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-mZREm7EyapCT95B off=7593917 len=1383367 2021-07-31T02:06:03.438 INFO:tasks.workunit.client.0.smithi045.stdout:op 274 completed, throughput=5.01MB/sec 2021-07-31T02:06:04.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:06:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:06:03.540Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:06:04.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:06:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:06:03.540Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:06:04.027 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:06:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:06:03.540Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:06:04.439 INFO:tasks.workunit.client.0.smithi045.stdout: 57: throughput=4.92MB/sec pending data=0 2021-07-31T02:06:04.439 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-fn1UFioeX4eKULC off=2580139 len=1789739 2021-07-31T02:06:04.450 INFO:tasks.workunit.client.0.smithi045.stdout:op 275 completed, throughput=4.95MB/sec 2021-07-31T02:06:04.450 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-_cn3koLs2YDTFV_ off=14168361 len=1145479 2021-07-31T02:06:04.459 INFO:tasks.workunit.client.0.smithi045.stdout:op 276 completed, throughput=4.97MB/sec 2021-07-31T02:06:04.459 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-u0Em0IvcgNzzcJh off=3833582 len=1536962 2021-07-31T02:06:04.465 INFO:tasks.workunit.client.0.smithi045.stdout:op 277 completed, throughput=4.99MB/sec 2021-07-31T02:06:04.465 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-RTu8B-GdPoiYeGb off=10356739 len=25103 2021-07-31T02:06:04.465 INFO:tasks.workunit.client.0.smithi045.stdout:op 278 completed, throughput=4.99MB/sec 2021-07-31T02:06:04.465 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-qhy1oUnj2D8LKmc off=6007889 len=935150 2021-07-31T02:06:04.467 INFO:tasks.workunit.client.0.smithi045.stdout:op 279 completed, throughput=5.01MB/sec 2021-07-31T02:06:04.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:04 smithi154 conmon[34553]: cluster 2021-07-31T02:06:04.057910+0000 mgr.x (mgr.24542) 807 : cluster [DBG] pgmap v1437: 9 pgs: 9 active+clean; 127 GiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 3.0 MiB/s rd, 3.3 MiB/s wr, 6 op/s 2021-07-31T02:06:04.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:04 smithi045 conmon[39718]: cluster 2021-07-31T02:06:04.057910+0000 mgr.x (mgr.24542 2021-07-31T02:06:04.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:04 smithi045 conmon[39718]: ) 807 : cluster [DBG] pgmap v1437: 9 pgs: 9 active+clean; 127 GiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 3.0 MiB/s rd, 3.3 MiB/s wr, 6 op/s 2021-07-31T02:06:04.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:04 smithi045 conmon[33655]: cluster 2021-07-31T02:06:04.057910+0000 mgr.x (mgr.24542) 807 : cluster [DBG] pgmap v1437: 9 pgs: 9 active+clean; 127 GiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 3.0 MiB/s rd, 3.3 MiB/s wr, 6 op/s 2021-07-31T02:06:05.467 INFO:tasks.workunit.client.0.smithi045.stdout: 58: throughput=4.92MB/sec pending data=0 2021-07-31T02:06:05.467 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-RTqn9L3MbpTkwxV off=10958422 len=1532162 2021-07-31T02:06:05.474 INFO:tasks.workunit.client.0.smithi045.stdout:op 280 completed, throughput=4.95MB/sec 2021-07-31T02:06:05.474 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-_BIO6QW1iDEF4LU off=6323213 len=22908 2021-07-31T02:06:05.476 INFO:tasks.workunit.client.0.smithi045.stdout:op 281 completed, throughput=4.95MB/sec 2021-07-31T02:06:05.476 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-5o5IvOj9E6fYjTm off=4521652 len=860542 2021-07-31T02:06:05.481 INFO:tasks.workunit.client.0.smithi045.stdout:op 282 completed, throughput=4.96MB/sec 2021-07-31T02:06:05.481 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj--rLEIM5Sj1v4OoJ off=17949911 len=2036278 2021-07-31T02:06:05.491 INFO:tasks.workunit.client.0.smithi045.stdout:op 283 completed, throughput=4.99MB/sec 2021-07-31T02:06:05.492 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-fyM0xIR4hURn67H off=13154933 len=1966748 2021-07-31T02:06:05.501 INFO:tasks.workunit.client.0.smithi045.stdout:op 284 completed, throughput=5.02MB/sec 2021-07-31T02:06:06.501 INFO:tasks.workunit.client.0.smithi045.stdout: 59: throughput=4.94MB/sec pending data=0 2021-07-31T02:06:06.501 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-behrfp440k6WDLd off=12411122 len=327260 2021-07-31T02:06:06.502 INFO:tasks.workunit.client.0.smithi045.stdout:op 285 completed, throughput=4.94MB/sec 2021-07-31T02:06:06.502 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-OPHPCDIhPF4X5q_ off=818491 len=289326 2021-07-31T02:06:06.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:06 smithi154 conmon[34553]: cluster 2021-07-31T02:06:06.058431+0000 mgr.x (mgr.24542) 808 : cluster [DBG] pgmap v1438: 9 pgs: 9 active+clean; 127 GiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:06:06.505 INFO:tasks.workunit.client.0.smithi045.stdout:op 286 completed, throughput=4.95MB/sec 2021-07-31T02:06:06.506 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-WEzpc9usmI3a1T2 off=1868353 len=772540 2021-07-31T02:06:06.510 INFO:tasks.workunit.client.0.smithi045.stdout:op 287 completed, throughput=4.96MB/sec 2021-07-31T02:06:06.510 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-jxYle9AGrEJ8c6R off=604266 len=1759156 2021-07-31T02:06:06.520 INFO:tasks.workunit.client.0.smithi045.stdout:op 288 completed, throughput=4.99MB/sec 2021-07-31T02:06:06.521 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ilwfIcyVe6SasvV off=21161458 len=1192531 2021-07-31T02:06:06.522 INFO:tasks.workunit.client.0.smithi045.stdout:op 289 completed, throughput=5.01MB/sec 2021-07-31T02:06:06.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:06 smithi045 conmon[33655]: cluster 2021-07-31T02:06:06.058431+0000 mgr.x (mgr.24542) 808 : cluster 2021-07-31T02:06:06.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:06 smithi045 conmon[33655]: [DBG] pgmap v1438: 9 pgs: 9 active+clean; 127 GiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:06:06.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:06 smithi045 conmon[39718]: cluster 2021-07-31T02:06:06.058431+0000 mgr.x (mgr. 2021-07-31T02:06:06.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:06 smithi045 conmon[39718]: 24542) 808 : cluster [DBG] pgmap v1438: 9 pgs: 9 active+clean; 127 GiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:06:07.522 INFO:tasks.workunit.client.0.smithi045.stdout: 60: throughput=4.92MB/sec pending data=0 2021-07-31T02:06:07.522 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-oBwimPpXeCoErHH off=17886933 len=906032 2021-07-31T02:06:07.527 INFO:tasks.workunit.client.0.smithi045.stdout:op 290 completed, throughput=4.94MB/sec 2021-07-31T02:06:07.528 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-wDvfYPrXB92HFU6 off=662473 len=1725105 2021-07-31T02:06:07.531 INFO:tasks.workunit.client.0.smithi045.stdout:op 291 completed, throughput=4.96MB/sec 2021-07-31T02:06:07.531 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-a2pySLU18jNAuk3 off=1858605 len=810728 2021-07-31T02:06:07.536 INFO:tasks.workunit.client.0.smithi045.stdout:op 292 completed, throughput=4.98MB/sec 2021-07-31T02:06:07.536 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-th6Dz3o_kVGFfYV off=2637570 len=46012 2021-07-31T02:06:07.539 INFO:tasks.workunit.client.0.smithi045.stdout:op 293 completed, throughput=4.98MB/sec 2021-07-31T02:06:07.539 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Mcdg033EvUY7cYK off=10395732 len=2064057 2021-07-31T02:06:07.550 INFO:tasks.workunit.client.0.smithi045.stdout:op 294 completed, throughput=5.01MB/sec 2021-07-31T02:06:08.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:08 smithi154 conmon[34553]: cluster 2021-07-31T02:06:08.059002+0000 mgr.x (mgr.24542) 809 : cluster [DBG] pgmap v1439: 9 pgs: 9 active+clean; 127 GiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:06:08.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:08 smithi045 conmon[33655]: cluster 2021-07-31T02:06:08.059002+0000 mgr.x (mgr.24542) 809 : cluster [DBG] 2021-07-31T02:06:08.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:08 smithi045 conmon[33655]: pgmap v1439: 9 pgs: 9 active+clean; 127 GiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:06:08.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:08 smithi045 conmon[39718]: cluster 2021-07-31T02:06:08.059002+0000 mgr.x ( 2021-07-31T02:06:08.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:08 smithi045 conmon[39718]: mgr.24542) 809 : cluster [DBG] pgmap v1439: 9 pgs: 9 active+clean; 127 GiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:06:08.550 INFO:tasks.workunit.client.0.smithi045.stdout: 61: throughput=4.93MB/sec pending data=0 2021-07-31T02:06:08.551 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-m49oBbT1NmepZz4 off=1854535 len=196811 2021-07-31T02:06:08.551 INFO:tasks.workunit.client.0.smithi045.stdout:op 295 completed, throughput=4.93MB/sec 2021-07-31T02:06:08.552 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-bYOel9m2N0axzxJ off=14394384 len=1571722 2021-07-31T02:06:08.555 INFO:tasks.workunit.client.0.smithi045.stdout:op 296 completed, throughput=4.95MB/sec 2021-07-31T02:06:08.555 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-TqsX1h5BOkFGlqx off=390727 len=856531 2021-07-31T02:06:08.559 INFO:tasks.workunit.client.0.smithi045.stdout:op 297 completed, throughput=4.97MB/sec 2021-07-31T02:06:08.559 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Bg0uMvUqiytl8sM off=3905664 len=2043180 2021-07-31T02:06:08.568 INFO:tasks.workunit.client.0.smithi045.stdout:op 298 completed, throughput=5MB/sec 2021-07-31T02:06:08.568 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ft6OCPPQmw13hFU off=13666031 len=82571 2021-07-31T02:06:08.569 INFO:tasks.workunit.client.0.smithi045.stdout:op 299 completed, throughput=5MB/sec 2021-07-31T02:06:08.569 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-YDHGjNZaLfonsTt off=7302829 len=1079729 2021-07-31T02:06:08.573 INFO:tasks.workunit.client.0.smithi045.stdout:op 300 completed, throughput=5.01MB/sec 2021-07-31T02:06:09.574 INFO:tasks.workunit.client.0.smithi045.stdout: 62: throughput=4.94MB/sec pending data=0 2021-07-31T02:06:09.575 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-7_0OtZ6bRXc47wE off=5482765 len=229405 2021-07-31T02:06:09.577 INFO:tasks.workunit.client.0.smithi045.stdout:op 301 completed, throughput=4.94MB/sec 2021-07-31T02:06:09.577 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-zj4721td-NM6MB7 off=2847433 len=70306 2021-07-31T02:06:09.580 INFO:tasks.workunit.client.0.smithi045.stdout:op 302 completed, throughput=4.94MB/sec 2021-07-31T02:06:09.580 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-avJBI_kxhsgyPK_ off=2835398 len=978824 2021-07-31T02:06:09.585 INFO:tasks.workunit.client.0.smithi045.stdout:op 303 completed, throughput=4.95MB/sec 2021-07-31T02:06:09.586 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Br1_QoG71QCKc6m off=19554031 len=1349354 2021-07-31T02:06:09.587 INFO:tasks.workunit.client.0.smithi045.stdout:op 304 completed, throughput=4.97MB/sec 2021-07-31T02:06:09.587 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-e0V0Bqd9IxdIfEB off=621618 len=1908651 2021-07-31T02:06:09.589 INFO:tasks.workunit.client.0.smithi045.stdout:op 305 completed, throughput=5MB/sec 2021-07-31T02:06:10.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:10 smithi154 conmon[34553]: cluster 2021-07-31T02:06:10.059458+0000 mgr.x (mgr.24542) 810 : cluster [DBG] pgmap v1440: 9 pgs: 9 active+clean; 127 GiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 2.3 MiB/s rd, 4.1 MiB/s wr, 6 op/s 2021-07-31T02:06:10.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:10 smithi045 conmon[39718]: cluster 2021-07-31T02:06:10.059458+0000 2021-07-31T02:06:10.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:10 smithi045 conmon[39718]: mgr.x (mgr.24542) 810 : cluster [DBG] pgmap v1440: 9 pgs: 9 active+clean; 127 GiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 2.3 MiB/s rd, 4.1 MiB/s wr, 6 op/s 2021-07-31T02:06:10.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:10 smithi045 conmon[33655]: cluster 2021-07-31T02:06:10.059458+0000 mgr.x (mgr.24542) 2021-07-31T02:06:10.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:10 smithi045 conmon[33655]: 810 : cluster [DBG] pgmap v1440: 9 pgs: 9 active+clean; 127 GiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 2.3 MiB/s rd, 4.1 MiB/s wr, 6 op/s 2021-07-31T02:06:10.590 INFO:tasks.workunit.client.0.smithi045.stdout: 63: throughput=4.92MB/sec pending data=0 2021-07-31T02:06:10.591 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-UQdDWnAc088fvF- off=2664633 len=49787 2021-07-31T02:06:10.591 INFO:tasks.workunit.client.0.smithi045.stdout:op 306 completed, throughput=4.93MB/sec 2021-07-31T02:06:10.592 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Tr-cpMGaVGWh-ls off=14489128 len=1533511 2021-07-31T02:06:10.598 INFO:tasks.workunit.client.0.smithi045.stdout:op 307 completed, throughput=4.95MB/sec 2021-07-31T02:06:10.598 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-YkTJ19KOhik4pZJ off=2880799 len=176314 2021-07-31T02:06:10.600 INFO:tasks.workunit.client.0.smithi045.stdout:op 308 completed, throughput=4.95MB/sec 2021-07-31T02:06:10.600 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-hPBAG7wUeeIBTzB off=3177069 len=1064094 2021-07-31T02:06:10.603 INFO:tasks.workunit.client.0.smithi045.stdout:op 309 completed, throughput=4.97MB/sec 2021-07-31T02:06:10.604 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-XRjQc6mCrhBeWlN off=9673467 len=1008070 2021-07-31T02:06:10.605 INFO:tasks.workunit.client.0.smithi045.stdout:op 310 completed, throughput=4.98MB/sec 2021-07-31T02:06:10.605 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-eH8UP65fR4TZEJb off=195382 len=972940 2021-07-31T02:06:10.609 INFO:tasks.workunit.client.0.smithi045.stdout:op 311 completed, throughput=4.99MB/sec 2021-07-31T02:06:10.610 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-xdp-LSZc4OXkpeV off=9543317 len=358222 2021-07-31T02:06:10.611 INFO:tasks.workunit.client.0.smithi045.stdout:op 312 completed, throughput=5MB/sec 2021-07-31T02:06:10.611 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Iw7iSn3jWRGq6_y off=1083405 len=1619294 2021-07-31T02:06:10.616 INFO:tasks.workunit.client.0.smithi045.stdout:op 313 completed, throughput=5.02MB/sec 2021-07-31T02:06:11.617 INFO:tasks.workunit.client.0.smithi045.stdout: 64: throughput=4.95MB/sec pending data=0 2021-07-31T02:06:11.617 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-m0jkpnIZaAi5zY_ off=964685 len=1696519 2021-07-31T02:06:11.620 INFO:tasks.workunit.client.0.smithi045.stdout:op 314 completed, throughput=4.97MB/sec 2021-07-31T02:06:11.620 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-3hlWtMk5Rqhuuf9 off=16086219 len=1757486 2021-07-31T02:06:11.624 INFO:tasks.workunit.client.0.smithi045.stdout:op 315 completed, throughput=5MB/sec 2021-07-31T02:06:11.624 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-NUXgw8bxXrzB6iH off=109311 len=1768012 2021-07-31T02:06:11.626 INFO:tasks.workunit.client.0.smithi045.stdout:op 316 completed, throughput=5.02MB/sec 2021-07-31T02:06:12.626 INFO:tasks.workunit.client.0.smithi045.stdout: 65: throughput=4.95MB/sec pending data=0 2021-07-31T02:06:12.626 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-L4wduYBpfSLLwJp off=1457117 len=1606807 2021-07-31T02:06:12.630 INFO:tasks.workunit.client.0.smithi045.stdout:op 317 completed, throughput=4.97MB/sec 2021-07-31T02:06:12.630 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-5fEixHyL9-JaSf9 off=6339214 len=1732999 2021-07-31T02:06:12.638 INFO:tasks.workunit.client.0.smithi045.stdout:op 318 completed, throughput=4.99MB/sec 2021-07-31T02:06:12.638 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-bJwf3AtjzdEtzM0 off=9272289 len=117403 2021-07-31T02:06:12.641 INFO:tasks.workunit.client.0.smithi045.stdout:op 319 completed, throughput=4.99MB/sec 2021-07-31T02:06:12.641 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-5i0GWwnvt_kbRGg off=10592436 len=1668879 2021-07-31T02:06:12.643 INFO:tasks.workunit.client.0.smithi045.stdout:op 320 completed, throughput=5.02MB/sec 2021-07-31T02:06:13.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:12 smithi154 conmon[34553]: cluster 2021-07-31T02:06:12.059974+0000 mgr.x (mgr.24542) 811 : cluster [DBG] pgmap v1441: 9 pgs: 9 active+clean; 127 GiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.6 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2021-07-31T02:06:13.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:12 smithi045 conmon[39718]: cluster 2021-07-31T02:06:12.059974+0000 mgr.x 2021-07-31T02:06:13.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:12 smithi045 conmon[39718]: (mgr.24542) 811 : cluster [DBG] pgmap v1441: 9 pgs: 9 active+clean; 127 GiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.6 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2021-07-31T02:06:13.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:12 smithi045 conmon[33655]: cluster 2021-07-31T02:06:12.059974+0000 mgr.x (mgr.24542) 811 : cluster [DBG] pgmap v1441: 9 pgs: 9 active+clean; 127 GiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.6 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2021-07-31T02:06:13.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:12 smithi045 conmon[33655]: 2021-07-31T02:06:13.643 INFO:tasks.workunit.client.0.smithi045.stdout: 66: throughput=4.94MB/sec pending data=0 2021-07-31T02:06:13.644 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj--8PAeCECnYqTTJI off=4447306 len=1212646 2021-07-31T02:06:13.645 INFO:tasks.workunit.client.0.smithi045.stdout:op 321 completed, throughput=4.96MB/sec 2021-07-31T02:06:13.645 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-g-VDDGfneU0CfVR off=0 len=1369478 2021-07-31T02:06:13.648 INFO:tasks.workunit.client.0.smithi045.stdout:op 322 completed, throughput=4.98MB/sec 2021-07-31T02:06:13.649 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Ecn8H0y1bed3LTG off=4833097 len=213056 2021-07-31T02:06:13.652 INFO:tasks.workunit.client.0.smithi045.stdout:op 323 completed, throughput=4.98MB/sec 2021-07-31T02:06:13.653 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-aCF9F9mdD3Y2pzb off=2290360 len=1849150 2021-07-31T02:06:13.655 INFO:tasks.workunit.client.0.smithi045.stdout:op 324 completed, throughput=5.01MB/sec 2021-07-31T02:06:14.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:06:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:06:13.540Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:06:14.027 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:06:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:06:13.540Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:06:14.027 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:06:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:06:13.540Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:06:14.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:14 smithi154 conmon[34553]: cluster 2021-07-31T02:06:14.060456+0000 mgr.x (mgr.24542) 812 : cluster [DBG] pgmap v1442: 9 pgs: 9 active+clean; 127 GiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 3.0 MiB/s rd, 3.4 MiB/s wr, 6 op/s 2021-07-31T02:06:14.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:14 smithi045 conmon[33655]: cluster 2021-07-31T02:06:14.060456+0000 mgr.x (mgr.24542) 812 : cluster [DBG] pgmap v1442: 9 pgs: 9 active+clean; 127 GiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 3.0 MiB/s rd, 3.4 MiB/s wr, 6 op/s 2021-07-31T02:06:14.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:14 smithi045 conmon[39718]: cluster 2021-07-31T02:06:14.060456+0000 mgr.x (mgr. 2021-07-31T02:06:14.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:14 smithi045 conmon[39718]: 24542) 812 : cluster [DBG] pgmap v1442: 9 pgs: 9 active+clean; 127 GiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 3.0 MiB/s rd, 3.4 MiB/s wr, 6 op/s 2021-07-31T02:06:14.655 INFO:tasks.workunit.client.0.smithi045.stdout: 67: throughput=4.94MB/sec pending data=0 2021-07-31T02:06:14.656 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-sivrVUCAYRXjqEW off=0 len=1216384 2021-07-31T02:06:14.661 INFO:tasks.workunit.client.0.smithi045.stdout:op 325 completed, throughput=4.95MB/sec 2021-07-31T02:06:14.662 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-DAUGZQTtSyIUOL_ off=16222664 len=1177657 2021-07-31T02:06:14.663 INFO:tasks.workunit.client.0.smithi045.stdout:op 326 completed, throughput=4.97MB/sec 2021-07-31T02:06:14.664 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-DfVHK-EybuMWamu off=8094500 len=681555 2021-07-31T02:06:14.665 INFO:tasks.workunit.client.0.smithi045.stdout:op 327 completed, throughput=4.98MB/sec 2021-07-31T02:06:14.665 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Jw7M0eBy18CCw2Q off=0 len=1203542 2021-07-31T02:06:14.671 INFO:tasks.workunit.client.0.smithi045.stdout:op 328 completed, throughput=4.99MB/sec 2021-07-31T02:06:14.672 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-FTy9mq7QMyI2eqk off=11088963 len=1511656 2021-07-31T02:06:14.678 INFO:tasks.workunit.client.0.smithi045.stdout:op 329 completed, throughput=5.01MB/sec 2021-07-31T02:06:15.679 INFO:tasks.workunit.client.0.smithi045.stdout: 68: throughput=4.94MB/sec pending data=0 2021-07-31T02:06:15.679 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-tUqH4F9vP0nvGhM off=1390824 len=978601 2021-07-31T02:06:15.686 INFO:tasks.workunit.client.0.smithi045.stdout:op 330 completed, throughput=4.96MB/sec 2021-07-31T02:06:15.686 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-yt9OeUwmeASRhcz off=5126593 len=658579 2021-07-31T02:06:15.687 INFO:tasks.workunit.client.0.smithi045.stdout:op 331 completed, throughput=4.96MB/sec 2021-07-31T02:06:15.687 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Crqym90x4paAITS off=2739543 len=1383850 2021-07-31T02:06:15.690 INFO:tasks.workunit.client.0.smithi045.stdout:op 332 completed, throughput=4.98MB/sec 2021-07-31T02:06:15.690 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-skP3LgUSHqurg1T off=5801392 len=1993601 2021-07-31T02:06:15.699 INFO:tasks.workunit.client.0.smithi045.stdout:op 333 completed, throughput=5.01MB/sec 2021-07-31T02:06:16.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:16 smithi154 conmon[34553]: cluster 2021-07-31T02:06:16.060973+0000 mgr.x (mgr.24542) 813 : cluster [DBG] pgmap v1443: 9 pgs: 9 active+clean; 127 GiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:06:16.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:16 smithi045 conmon[33655]: cluster 2021-07-31T02:06:16.060973+0000 mgr.x (mgr.24542) 813 2021-07-31T02:06:16.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:16 smithi045 conmon[33655]: : cluster [DBG] pgmap v1443: 9 pgs: 9 active+clean; 127 GiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:06:16.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:16 smithi045 conmon[39718]: cluster 2021-07-31T02:06:16.060973+0000 mgr.x (mgr 2021-07-31T02:06:16.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:16 smithi045 conmon[39718]: .24542) 813 : cluster [DBG] pgmap v1443: 9 pgs: 9 active+clean; 127 GiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:06:16.699 INFO:tasks.workunit.client.0.smithi045.stdout: 69: throughput=4.94MB/sec pending data=0 2021-07-31T02:06:16.700 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ptX82A0IIATDAQD off=6205558 len=777138 2021-07-31T02:06:16.702 INFO:tasks.workunit.client.0.smithi045.stdout:op 334 completed, throughput=4.95MB/sec 2021-07-31T02:06:16.702 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-_K1K4CQg8YeIDrU off=4532809 len=648278 2021-07-31T02:06:16.703 INFO:tasks.workunit.client.0.smithi045.stdout:op 335 completed, throughput=4.96MB/sec 2021-07-31T02:06:16.703 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-rNurojN3hQa9-Gb off=855378 len=1976275 2021-07-31T02:06:16.711 INFO:tasks.workunit.client.0.smithi045.stdout:op 336 completed, throughput=4.98MB/sec 2021-07-31T02:06:16.711 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-0sUeajHUWcy5RG3 off=10264213 len=2020277 2021-07-31T02:06:16.721 INFO:tasks.workunit.client.0.smithi045.stdout:op 337 completed, throughput=5.01MB/sec 2021-07-31T02:06:17.721 INFO:tasks.workunit.client.0.smithi045.stdout: 70: throughput=4.94MB/sec pending data=0 2021-07-31T02:06:17.722 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-D7wHdXTR-CRv7Iw off=1301574 len=278414 2021-07-31T02:06:17.725 INFO:tasks.workunit.client.0.smithi045.stdout:op 338 completed, throughput=4.94MB/sec 2021-07-31T02:06:17.725 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-j741i2ThI8C1pyo off=7203594 len=1474350 2021-07-31T02:06:17.732 INFO:tasks.workunit.client.0.smithi045.stdout:op 339 completed, throughput=4.96MB/sec 2021-07-31T02:06:17.732 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-62Mj2K4USb9msA8 off=5800039 len=2054173 2021-07-31T02:06:17.735 INFO:tasks.workunit.client.0.smithi045.stdout:op 340 completed, throughput=4.99MB/sec 2021-07-31T02:06:17.736 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-uGNsQySQ2uyYcK8 off=8426568 len=912766 2021-07-31T02:06:17.740 INFO:tasks.workunit.client.0.smithi045.stdout:op 341 completed, throughput=5MB/sec 2021-07-31T02:06:18.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:18 smithi154 conmon[34553]: cluster 2021-07-31T02:06:18.061382+0000 mgr.x (mgr.24542) 814 : cluster [DBG] pgmap v1444: 9 pgs: 9 active+clean; 127 GiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:06:18.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:18 smithi154 conmon[34553]: 2021-07-31T02:06:18.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:18 smithi045 conmon[39718]: cluster 2021-07-31T02:06:18.061382+0000 mgr.x ( 2021-07-31T02:06:18.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:18 smithi045 conmon[39718]: mgr.24542) 814 : cluster [DBG] pgmap v1444: 9 pgs: 9 active+clean; 127 GiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:06:18.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:18 smithi045 conmon[33655]: cluster 2021-07-31T02:06:18.061382+0000 mgr.x (mgr.24542 2021-07-31T02:06:18.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:18 smithi045 conmon[33655]: ) 814 : cluster [DBG] pgmap v1444: 9 pgs: 9 active+clean; 127 GiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:06:18.742 INFO:tasks.workunit.client.0.smithi045.stdout: 71: throughput=4.93MB/sec pending data=0 2021-07-31T02:06:18.742 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-cV-Yq-KmUZBXuVY off=9112459 len=1925284 2021-07-31T02:06:18.743 INFO:tasks.workunit.client.0.smithi045.stdout:op 342 completed, throughput=4.96MB/sec 2021-07-31T02:06:18.744 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-1PJlAG1RicgqUub off=1498235 len=859827 2021-07-31T02:06:18.748 INFO:tasks.workunit.client.0.smithi045.stdout:op 343 completed, throughput=4.97MB/sec 2021-07-31T02:06:18.749 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-zhB42iU4Td73PFL off=11434487 len=60369 2021-07-31T02:06:18.750 INFO:tasks.workunit.client.0.smithi045.stdout:op 344 completed, throughput=4.97MB/sec 2021-07-31T02:06:18.751 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-nq1zRYVjltyNUqo off=19055435 len=437364 2021-07-31T02:06:18.752 INFO:tasks.workunit.client.0.smithi045.stdout:op 345 completed, throughput=4.98MB/sec 2021-07-31T02:06:18.752 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ZhXET56qli040lx off=5699603 len=1207411 2021-07-31T02:06:18.753 INFO:tasks.workunit.client.0.smithi045.stdout:op 346 completed, throughput=4.99MB/sec 2021-07-31T02:06:18.753 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-_uniGlZIF0C6thd off=1922434 len=670276 2021-07-31T02:06:18.758 INFO:tasks.workunit.client.0.smithi045.stdout:op 347 completed, throughput=5MB/sec 2021-07-31T02:06:18.758 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ukPpQXHXWmviqRu off=1451628 len=472341 2021-07-31T02:06:18.759 INFO:tasks.workunit.client.0.smithi045.stdout:op 348 completed, throughput=5.01MB/sec 2021-07-31T02:06:19.759 INFO:tasks.workunit.client.0.smithi045.stdout: 72: throughput=4.94MB/sec pending data=0 2021-07-31T02:06:19.760 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ZHLMXcyYMO42gYs off=198155 len=716880 2021-07-31T02:06:19.765 INFO:tasks.workunit.client.0.smithi045.stdout:op 349 completed, throughput=4.95MB/sec 2021-07-31T02:06:19.765 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-HgGSJABPpaj-jWn off=7559752 len=1518149 2021-07-31T02:06:19.776 INFO:tasks.workunit.client.0.smithi045.stdout:op 350 completed, throughput=4.97MB/sec 2021-07-31T02:06:19.776 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-3S1euuRyMFhSMrc off=2376110 len=1489948 2021-07-31T02:06:19.778 INFO:tasks.workunit.client.0.smithi045.stdout:op 351 completed, throughput=4.98MB/sec 2021-07-31T02:06:19.778 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Rrk-WprM2ZG4CsQ off=9270957 len=216717 2021-07-31T02:06:19.780 INFO:tasks.workunit.client.0.smithi045.stdout:op 352 completed, throughput=4.99MB/sec 2021-07-31T02:06:19.780 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-6WK_3fIkqC7mde_ off=615565 len=1826296 2021-07-31T02:06:19.788 INFO:tasks.workunit.client.0.smithi045.stdout:op 353 completed, throughput=5.01MB/sec 2021-07-31T02:06:20.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:20 smithi154 conmon[34553]: cluster 2021-07-31T02:06:20.061983+0000 mgr.x (mgr.24542) 815 : cluster [DBG] pgmap v1445: 9 pgs: 9 active+clean; 127 GiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 2.8 MiB/s rd, 3.6 MiB/s wr, 6 op/s 2021-07-31T02:06:20.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:20 smithi045 conmon[39718]: cluster 2021-07-31T02:06:20.061983+0000 mgr.x (mgr.24542) 815 : cluster [DBG] pgmap v1445: 9 pgs: 9 active+clean; 127 GiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 2.8 MiB/s rd, 3.6 MiB/s wr, 6 op/s 2021-07-31T02:06:20.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:20 smithi045 conmon[33655]: cluster 2021-07-31T02:06:20.061983+0000 mgr.x (mgr.24542) 815 : 2021-07-31T02:06:20.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:20 smithi045 conmon[33655]: cluster [DBG] pgmap v1445: 9 pgs: 9 active+clean; 127 GiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 2.8 MiB/s rd, 3.6 MiB/s wr, 6 op/s 2021-07-31T02:06:20.789 INFO:tasks.workunit.client.0.smithi045.stdout: 73: throughput=4.94MB/sec pending data=0 2021-07-31T02:06:20.789 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ncaKYKfTSNZl1Ku off=1142112 len=141540 2021-07-31T02:06:20.790 INFO:tasks.workunit.client.0.smithi045.stdout:op 354 completed, throughput=4.95MB/sec 2021-07-31T02:06:20.790 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-XW4qlWuhyzpb8i9 off=12148705 len=2030621 2021-07-31T02:06:20.800 INFO:tasks.workunit.client.0.smithi045.stdout:op 355 completed, throughput=4.97MB/sec 2021-07-31T02:06:20.801 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-AyaktyklxW3UjyK off=4596117 len=1359981 2021-07-31T02:06:20.807 INFO:tasks.workunit.client.0.smithi045.stdout:op 356 completed, throughput=4.99MB/sec 2021-07-31T02:06:20.808 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-AvaSvg0Js2DUYGA off=2309471 len=1052171 2021-07-31T02:06:20.813 INFO:tasks.workunit.client.0.smithi045.stdout:op 357 completed, throughput=5MB/sec 2021-07-31T02:06:21.814 INFO:tasks.workunit.client.0.smithi045.stdout: 74: throughput=4.93MB/sec pending data=0 2021-07-31T02:06:21.814 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-n2P8DAvPlyvAZrI off=11507086 len=1257028 2021-07-31T02:06:21.820 INFO:tasks.workunit.client.0.smithi045.stdout:op 358 completed, throughput=4.95MB/sec 2021-07-31T02:06:21.820 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-CV16Yy0uS5FVh0v off=3801937 len=1930617 2021-07-31T02:06:21.822 INFO:tasks.workunit.client.0.smithi045.stdout:op 359 completed, throughput=4.97MB/sec 2021-07-31T02:06:21.823 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-KyrJRoo3xj88m-1 off=5219173 len=1037637 2021-07-31T02:06:21.823 INFO:tasks.workunit.client.0.smithi045.stdout:op 360 completed, throughput=4.99MB/sec 2021-07-31T02:06:21.824 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-6bg0Es5zIO1Vexe off=3143605 len=306255 2021-07-31T02:06:21.826 INFO:tasks.workunit.client.0.smithi045.stdout:op 361 completed, throughput=4.99MB/sec 2021-07-31T02:06:21.827 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-3RHYrT5KHNJnP1F off=9186749 len=1856663 2021-07-31T02:06:21.837 INFO:tasks.workunit.client.0.smithi045.stdout:op 362 completed, throughput=5.01MB/sec 2021-07-31T02:06:22.837 INFO:tasks.workunit.client.0.smithi045.stdout: 75: throughput=4.95MB/sec pending data=0 2021-07-31T02:06:22.838 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ofxtdf7txDQTeKj off=11755470 len=830148 2021-07-31T02:06:22.839 INFO:tasks.workunit.client.0.smithi045.stdout:op 363 completed, throughput=4.96MB/sec 2021-07-31T02:06:22.840 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-NO-Tz9cS7B_w_rc off=8648195 len=102139 2021-07-31T02:06:22.842 INFO:tasks.workunit.client.0.smithi045.stdout:op 364 completed, throughput=4.96MB/sec 2021-07-31T02:06:22.842 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-OVW3VkzmQDPXBjx off=2346625 len=1260875 2021-07-31T02:06:22.845 INFO:tasks.workunit.client.0.smithi045.stdout:op 365 completed, throughput=4.97MB/sec 2021-07-31T02:06:22.845 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-RkXRSjBPXgDVXvj off=2784620 len=460134 2021-07-31T02:06:22.846 INFO:tasks.workunit.client.0.smithi045.stdout:op 366 completed, throughput=4.98MB/sec 2021-07-31T02:06:22.846 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-UtxxbiSspaV7toL off=6834545 len=1468835 2021-07-31T02:06:22.849 INFO:tasks.workunit.client.0.smithi045.stdout:op 367 completed, throughput=5MB/sec 2021-07-31T02:06:22.849 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-k-KcgXJkGB7jovB off=9455242 len=117040 2021-07-31T02:06:22.850 INFO:tasks.workunit.client.0.smithi045.stdout:op 368 completed, throughput=5MB/sec 2021-07-31T02:06:22.850 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-NNSEdjVqjBeNvNv off=5900795 len=1102276 2021-07-31T02:06:22.852 INFO:tasks.workunit.client.0.smithi045.stdout:op 369 completed, throughput=5.01MB/sec 2021-07-31T02:06:23.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:22 smithi154 conmon[34553]: cluster 2021-07-31T02:06:22.062549+0000 mgr.x (mgr.24542) 816 : cluster [DBG] 2021-07-31T02:06:23.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:22 smithi154 conmon[34553]: pgmap v1446: 9 pgs: 9 active+clean; 127 GiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 2.3 MiB/s rd, 1.9 MiB/s wr, 3 op/s 2021-07-31T02:06:23.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:22 smithi045 conmon[33655]: cluster 2021-07-31T02:06:22.062549+0000 mgr.x (mgr.24542) 816 : cluster [DBG] pgmap v1446: 9 pgs: 9 active+clean; 127 GiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 2.3 MiB/s rd, 1.9 MiB/s wr, 3 op/s 2021-07-31T02:06:23.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:22 smithi045 conmon[39718]: cluster 2021-07-31T02:06:22.062549+0000 mgr.x (mgr.24542) 816 2021-07-31T02:06:23.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:22 smithi045 conmon[39718]: : cluster [DBG] pgmap v1446: 9 pgs: 9 active+clean; 127 GiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 2.3 MiB/s rd, 1.9 MiB/s wr, 3 op/s 2021-07-31T02:06:23.852 INFO:tasks.workunit.client.0.smithi045.stdout: 76: throughput=4.95MB/sec pending data=0 2021-07-31T02:06:23.853 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-PJUY82XUshQ16pI off=4527533 len=837715 2021-07-31T02:06:23.859 INFO:tasks.workunit.client.0.smithi045.stdout:op 370 completed, throughput=4.96MB/sec 2021-07-31T02:06:23.859 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-M7yqmJicVnLnYR5 off=7983784 len=1061321 2021-07-31T02:06:23.860 INFO:tasks.workunit.client.0.smithi045.stdout:op 371 completed, throughput=4.97MB/sec 2021-07-31T02:06:23.860 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-QlXeXnzljQSXLqR off=5245922 len=448517 2021-07-31T02:06:23.861 INFO:tasks.workunit.client.0.smithi045.stdout:op 372 completed, throughput=4.98MB/sec 2021-07-31T02:06:23.861 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-CO0ZiV66VKftx52 off=7219515 len=1328745 2021-07-31T02:06:23.868 INFO:tasks.workunit.client.0.smithi045.stdout:op 373 completed, throughput=4.99MB/sec 2021-07-31T02:06:23.869 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-FwRBYRxLLIgoCgd off=12441616 len=933597 2021-07-31T02:06:23.874 INFO:tasks.workunit.client.0.smithi045.stdout:op 374 completed, throughput=5MB/sec 2021-07-31T02:06:24.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:06:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:06:23.540Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:06:24.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:06:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:06:23.540Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:06:24.027 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:06:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:06:23.540Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:06:24.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:24 smithi154 conmon[34553]: cluster 2021-07-31T02:06:24.063077+0000 mgr.x (mgr.24542) 817 : cluster [DBG] pgmap v1447: 9 pgs: 9 active+clean; 127 GiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 3.2 MiB/s rd, 3.0 MiB/s wr, 6 op/s 2021-07-31T02:06:24.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:24 smithi154 conmon[34553]: 2021-07-31T02:06:24.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:24 smithi045 conmon[39718]: cluster 2021-07-31T02:06:24.063077+0000 mgr.x (mgr.24542) 817 : cluster [DBG] pgmap v1447: 9 pgs: 9 active+clean; 127 GiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 3.2 MiB/s rd, 3.0 MiB/s wr, 6 op/s 2021-07-31T02:06:24.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:24 smithi045 conmon[33655]: cluster 2021-07-31T02:06:24.063077+0000 mgr.x (mgr.24542) 817 : cluster 2021-07-31T02:06:24.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:24 smithi045 conmon[33655]: [DBG] pgmap v1447: 9 pgs: 9 active+clean; 127 GiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 3.2 MiB/s rd, 3.0 MiB/s wr, 6 op/s 2021-07-31T02:06:24.875 INFO:tasks.workunit.client.0.smithi045.stdout: 77: throughput=4.94MB/sec pending data=0 2021-07-31T02:06:24.876 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-W-RREfjSYPj4Fql off=2731682 len=1159117 2021-07-31T02:06:24.881 INFO:tasks.workunit.client.0.smithi045.stdout:op 375 completed, throughput=4.95MB/sec 2021-07-31T02:06:24.881 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-vINJRhHAkz6HcsZ off=12891027 len=2062455 2021-07-31T02:06:24.888 INFO:tasks.workunit.client.0.smithi045.stdout:op 376 completed, throughput=4.98MB/sec 2021-07-31T02:06:24.888 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-cK_M9AOMKW6vxkC off=6906295 len=133864 2021-07-31T02:06:24.891 INFO:tasks.workunit.client.0.smithi045.stdout:op 377 completed, throughput=4.98MB/sec 2021-07-31T02:06:24.891 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-gW-b-xAij0JJ0v9 off=5730321 len=1301414 2021-07-31T02:06:24.892 INFO:tasks.workunit.client.0.smithi045.stdout:op 378 completed, throughput=5MB/sec 2021-07-31T02:06:24.893 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-VXRwqubrG8s63bQ off=1328820 len=1151415 2021-07-31T02:06:24.899 INFO:tasks.workunit.client.0.smithi045.stdout:op 379 completed, throughput=5.01MB/sec 2021-07-31T02:06:25.899 INFO:tasks.workunit.client.0.smithi045.stdout: 78: throughput=4.95MB/sec pending data=0 2021-07-31T02:06:25.899 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-sXgoF-2q8rL-Ghj off=10008855 len=1837745 2021-07-31T02:06:25.910 INFO:tasks.workunit.client.0.smithi045.stdout:op 380 completed, throughput=4.97MB/sec 2021-07-31T02:06:25.910 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-kZllLPugi_PDzeS off=13913598 len=1583443 2021-07-31T02:06:25.913 INFO:tasks.workunit.client.0.smithi045.stdout:op 381 completed, throughput=4.99MB/sec 2021-07-31T02:06:25.913 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-XfqGUFGuWZB4T6O off=4702685 len=1973352 2021-07-31T02:06:25.923 INFO:tasks.workunit.client.0.smithi045.stdout:op 382 completed, throughput=5.01MB/sec 2021-07-31T02:06:26.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:26 smithi154 conmon[34553]: cluster 2021-07-31T02:06:26.063619+0000 mgr.x (mgr.24542) 818 : cluster [DBG] pgmap v1448: 9 pgs: 9 active+clean; 127 GiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2021-07-31T02:06:26.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:26 smithi045 conmon[39718]: cluster 2021-07-31T02:06:26.063619+0000 mgr.x (mgr.24542) 818 : cluster [DBG] 2021-07-31T02:06:26.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:26 smithi045 conmon[39718]: pgmap v1448: 9 pgs: 9 active+clean; 127 GiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2021-07-31T02:06:26.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:26 smithi045 conmon[33655]: cluster 2021-07-31T02:06:26.063619+0000 mgr.x (mgr.24542) 818 : cluster 2021-07-31T02:06:26.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:26 smithi045 conmon[33655]: [DBG] pgmap v1448: 9 pgs: 9 active+clean; 127 GiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2021-07-31T02:06:26.924 INFO:tasks.workunit.client.0.smithi045.stdout: 79: throughput=4.95MB/sec pending data=0 2021-07-31T02:06:26.924 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-8JvLVyK0Qh-O154 off=4710979 len=1243234 2021-07-31T02:06:26.932 INFO:tasks.workunit.client.0.smithi045.stdout:op 383 completed, throughput=4.96MB/sec 2021-07-31T02:06:26.932 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-D2pRl_5xQSj9GDw off=7769156 len=1234278 2021-07-31T02:06:26.934 INFO:tasks.workunit.client.0.smithi045.stdout:op 384 completed, throughput=4.98MB/sec 2021-07-31T02:06:26.934 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Rz65RqqnijPXNX- off=12075757 len=73831 2021-07-31T02:06:26.937 INFO:tasks.workunit.client.0.smithi045.stdout:op 385 completed, throughput=4.98MB/sec 2021-07-31T02:06:26.937 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-XjmQmirruPBM3IK off=2998524 len=1481195 2021-07-31T02:06:26.944 INFO:tasks.workunit.client.0.smithi045.stdout:op 386 completed, throughput=4.99MB/sec 2021-07-31T02:06:26.944 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-XLTN-VSmZuomLHM off=4148252 len=1643401 2021-07-31T02:06:26.951 INFO:tasks.workunit.client.0.smithi045.stdout:op 387 completed, throughput=5.01MB/sec 2021-07-31T02:06:27.951 INFO:tasks.workunit.client.0.smithi045.stdout: 80: throughput=4.95MB/sec pending data=0 2021-07-31T02:06:27.952 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-MXcWRgfuwtECJYl off=8611684 len=1738054 2021-07-31T02:06:27.954 INFO:tasks.workunit.client.0.smithi045.stdout:op 388 completed, throughput=4.97MB/sec 2021-07-31T02:06:27.954 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-V2pNX_-4K4yEV-O off=5875113 len=2032325 2021-07-31T02:06:27.958 INFO:tasks.workunit.client.0.smithi045.stdout:op 389 completed, throughput=4.99MB/sec 2021-07-31T02:06:27.958 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-3YPyxr5TLG5yeAb off=6269007 len=33661 2021-07-31T02:06:27.959 INFO:tasks.workunit.client.0.smithi045.stdout:op 390 completed, throughput=5MB/sec 2021-07-31T02:06:27.959 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-uDYmJPk3k82pEap off=2887328 len=1904997 2021-07-31T02:06:27.962 INFO:tasks.workunit.client.0.smithi045.stdout:op 391 completed, throughput=5.02MB/sec 2021-07-31T02:06:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:28 smithi154 conmon[34553]: audit 2021-07-31T02:06:28.004262+0000 mon.b (mon.2) 163 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:06:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:28 smithi154 conmon[34553]: audit 2021-07-31T02:06:28.004948+0000 mon.a (mon.0) 1852 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:06:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:28 smithi154 conmon[34553]: audit 2021-07-31T02:06:28. 2021-07-31T02:06:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:28 smithi154 conmon[34553]: 022827+0000 mon.b (mon.2) 164 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:06:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:28 smithi154 conmon[34553]: audit 2021-07- 2021-07-31T02:06:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:28 smithi154 conmon[34553]: 31T02:06:28.023312+0000 mon.a (mon.0 2021-07-31T02:06:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:28 smithi154 conmon[34553]: ) 1853 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:06:28.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:28 smithi045 conmon[39718]: audit 2021-07-31T02:06:28.004262+0000 mon.b (mon. 2021-07-31T02:06:28.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:28 smithi045 conmon[39718]: 2) 163 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:06:28.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:28 smithi045 conmon[39718]: audit 2021-07-31T02:06:28.004948+0000 2021-07-31T02:06:28.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:28 smithi045 conmon[39718]: mon.a (mon.0) 1852 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:06:28.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:28 smithi045 conmon[39718]: audit 2021-07 2021-07-31T02:06:28.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:28 smithi045 conmon[39718]: -31T02:06:28.022827+0000 mon.b (mon.2) 164 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:06:28.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:28 smithi045 conmon[39718]: audit 2021-07-31T02:06:28 2021-07-31T02:06:28.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:28 smithi045 conmon[39718]: .023312+0000 mon.a (mon.0) 1853 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:06:28.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:28 smithi045 conmon[33655]: audit 2021-07-31T02:06:28.004262+0000 mon.b (mon.2) 163 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:06:28.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:28 smithi045 conmon[33655]: audit 2021-07-31T02:06:28.004948+0000 mon.a (mon.0) 1852 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:06:28.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:28 smithi045 conmon[33655]: audit 2021-07-31 2021-07-31T02:06:28.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:28 smithi045 conmon[33655]: T02:06:28.022827+0000 mon.b (mon.2) 164 2021-07-31T02:06:28.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:28 smithi045 conmon[33655]: : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:06:28.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:28 smithi045 conmon[33655]: audit 2021- 2021-07-31T02:06:28.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:28 smithi045 conmon[33655]: 07-31T02:06:28.023312+0000 2021-07-31T02:06:28.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:28 smithi045 conmon[33655]: mon.a (mon.0) 1853 : 2021-07-31T02:06:28.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:28 smithi045 conmon[33655]: audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:06:28.965 INFO:tasks.workunit.client.0.smithi045.stdout: 81: throughput=4.96MB/sec pending data=0 2021-07-31T02:06:28.966 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-1RBON71utCF-ZiF off=20227162 len=893166 2021-07-31T02:06:28.966 INFO:tasks.workunit.client.0.smithi045.stdout:op 392 completed, throughput=4.97MB/sec 2021-07-31T02:06:28.966 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-MEaVUjU-0LMQe4V off=127913 len=872976 2021-07-31T02:06:28.967 INFO:tasks.workunit.client.0.smithi045.stdout:op 393 completed, throughput=4.98MB/sec 2021-07-31T02:06:28.967 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-l779tUfi7_ZKGjb off=11631714 len=1690517 2021-07-31T02:06:28.969 INFO:tasks.workunit.client.0.smithi045.stdout:op 394 completed, throughput=5MB/sec 2021-07-31T02:06:28.970 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-b517QtjShB-cCyJ off=3376369 len=244018 2021-07-31T02:06:28.973 INFO:tasks.workunit.client.0.smithi045.stdout:op 395 completed, throughput=5MB/sec 2021-07-31T02:06:28.973 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-QYjusKGejy9Mw8M off=10336538 len=2049708 2021-07-31T02:06:28.985 INFO:tasks.workunit.client.0.smithi045.stdout:op 396 completed, throughput=5.02MB/sec 2021-07-31T02:06:29.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:29 smithi154 conmon[34553]: cluster 2021-07-31T02:06:28.064060+0000 mgr.x (mgr.24542) 819 : cluster [DBG] pgmap v1449: 9 pgs: 9 active+clean; 127 GiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2021-07-31T02:06:29.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:29 smithi045 conmon[33655]: cluster 2021-07-31T02:06:28.064060+0000 mgr.x (mgr.24542) 819 : cluster [DBG] 2021-07-31T02:06:29.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:29 smithi045 conmon[33655]: pgmap v1449: 9 pgs: 9 active+clean; 127 GiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2021-07-31T02:06:29.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:29 smithi045 conmon[39718]: cluster 2021-07-31T02:06:28.064060+0000 mgr.x (mgr.24542 2021-07-31T02:06:29.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:29 smithi045 conmon[39718]: ) 819 : cluster [DBG] pgmap v1449: 9 pgs: 9 active+clean; 127 GiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2021-07-31T02:06:29.985 INFO:tasks.workunit.client.0.smithi045.stdout: 82: throughput=4.96MB/sec pending data=0 2021-07-31T02:06:29.985 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-a4afjFS2YFMQEOq off=11238208 len=1469726 2021-07-31T02:06:29.993 INFO:tasks.workunit.client.0.smithi045.stdout:op 397 completed, throughput=4.98MB/sec 2021-07-31T02:06:29.993 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ds0iyjxZyUvceqd off=4103518 len=1204679 2021-07-31T02:06:29.995 INFO:tasks.workunit.client.0.smithi045.stdout:op 398 completed, throughput=4.99MB/sec 2021-07-31T02:06:29.995 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-U2iXd8v___-DQnR off=2956104 len=1740335 2021-07-31T02:06:30.003 INFO:tasks.workunit.client.0.smithi045.stdout:op 399 completed, throughput=5.01MB/sec 2021-07-31T02:06:30.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[34553]: audit 2021-07-31T02:06:30.010290+0000 mon.b (mon.2) 165 : audit 2021-07-31T02:06:30.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[34553]: [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:06:30.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[34553]: audit 2021-07-31T02:06:30.011745+0000 mon.b (mon 2021-07-31T02:06:30.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[34553]: .2) 166 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:06:30.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[34553]: audit 2021-07 2021-07-31T02:06:30.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[34553]: -31T02:06:30.012810 2021-07-31T02:06:30.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[34553]: +0000 mon.b (mon.2) 167 : audit [INF] 2021-07-31T02:06:30.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[34553]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:06:30.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: debug 2021-07-31T02:06:30.054+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:06:30.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:06:30.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:06:30.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: yield 2021-07-31T02:06:30.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:06:30.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:06:30.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:06:30.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:06:30.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:06:30.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:06:30.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:06:30.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:06:30.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:06:30.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:06:30.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:06:30.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: debug 2021-07-31T02:06:30.067+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:06:30.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:06:30.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:06:30.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: yield 2021-07-31T02:06:30.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:06:30.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:06:30.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:06:30.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:06:30.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:06:30.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:06:30.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:06:30.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:06:30.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:06:30.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:06:30.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:06:30.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: debug 2021-07-31T02:06:30.091+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:06:30.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:06:30.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:06:30.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: yield 2021-07-31T02:06:30.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:06:30.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:06:30.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:06:30.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:06:30.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:06:30.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:06:30.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:06:30.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:06:30.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:06:30.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:06:30.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:06:30.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: debug 2021-07-31T02:06:30.092+0000 7fcb97969700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:06:30.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:06:30.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:06:30.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: yield 2021-07-31T02:06:30.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:06:30.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:06:30.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:06:30.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:06:30.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:06:30.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:06:30.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:06:30.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:06:30.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:06:30.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:06:30.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:06:30.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: 2021-07-31T02:06:30.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:06:30.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: 2021-07-31T02:06:30.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:06:30.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:06:30.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:06:30.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:06:30.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:06:30.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:06:30.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:06:30.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:06:30.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:06:30.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:06:30.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:06:30.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:06:30.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:06:30.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:06:30.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:06:30.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:06:30.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:06:30.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:06:30.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: raise 2021-07-31T02:06:30.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:06:30.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:06:30.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:06:30.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:06:30.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:06:30.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: debug 2021-07-31T02:06:30.100+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:06:30.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:06:30.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:06:30.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: yield 2021-07-31T02:06:30.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:06:30.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:06:30.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:06:30.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:06:30.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:06:30.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:06:30.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:06:30.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:06:30.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:06:30.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:06:30.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:06:30.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: debug 2021-07-31T02:06:30.101+0000 7fcb9816a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:06:30.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:06:30.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:06:30.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: yield 2021-07-31T02:06:30.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:06:30.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:06:30.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:06:30.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:06:30.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:06:30.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:06:30.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:06:30.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:06:30.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:06:30.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:06:30.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:06:30.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: 2021-07-31T02:06:30.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:06:30.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: 2021-07-31T02:06:30.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:06:30.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:06:30.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:06:30.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:06:30.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:06:30.535 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:06:30.535 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:06:30.535 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:06:30.535 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:06:30.535 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:06:30.536 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:06:30.536 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:06:30.536 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:06:30.536 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:06:30.537 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:06:30.537 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:06:30.537 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:06:30.537 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:06:30.537 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: raise 2021-07-31T02:06:30.538 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:06:30.538 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:06:30.538 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:06:30.538 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:06:30.538 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:06:30 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:06:30.539 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:30 smithi045 conmon[33655]: audit 2021-07-31T02:06:30.010290+0000 2021-07-31T02:06:30.539 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:30 smithi045 conmon[33655]: mon.b (mon.2) 165 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:06:30.540 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:30 smithi045 conmon[33655]: audit 2021 2021-07-31T02:06:30.540 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:30 smithi045 conmon[33655]: -07-31T02:06:30.011745+0000 mon.b (mon.2) 166 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:06:30.540 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:30 smithi045 conmon[33655]: audit 2021-07-31T02:06:30.540 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:30 smithi045 conmon[33655]: 2021-07-31T02:06:30.012810 2021-07-31T02:06:30.540 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:30 smithi045 conmon[33655]: +0000 mon.b (mon.2) 167 : audit 2021-07-31T02:06:30.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:30 smithi045 conmon[33655]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:06:30.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:30 smithi045 conmon[39718]: audit 2021-07-31T02:06:30.010290+0000 mon.b (mon. 2021-07-31T02:06:30.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:30 smithi045 conmon[39718]: 2) 165 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:06:30.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:30 smithi045 conmon[39718]: audit 2021-07-31T02:06:30.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:30 smithi045 conmon[39718]: 2021-07-31T02:06:30.011745+0000 mon.b (mon.2) 166 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:06:30.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:30 smithi045 conmon[39718]: 2021-07-31T02:06:30.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:30 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T02:06:30.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:30 smithi045 conmon[39718]: 31T02:06:30.012810+0000 mon.b ( 2021-07-31T02:06:30.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:30 smithi045 conmon[39718]: mon.2) 167 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:06:30.784 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:06:31.003 INFO:tasks.workunit.client.0.smithi045.stdout: 83: throughput=4.95MB/sec pending data=0 2021-07-31T02:06:31.003 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-1t7ox4eUoKJYlz7 off=1320353 len=1116385 2021-07-31T02:06:31.007 INFO:tasks.workunit.client.0.smithi045.stdout:op 400 completed, throughput=4.96MB/sec 2021-07-31T02:06:31.008 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Rj3U9C8Z01H_Xcr off=21774561 len=300599 2021-07-31T02:06:31.008 INFO:tasks.workunit.client.0.smithi045.stdout:op 401 completed, throughput=4.97MB/sec 2021-07-31T02:06:31.009 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-uXgXzTPfumXhf-1 off=11098522 len=1294532 2021-07-31T02:06:31.011 INFO:tasks.workunit.client.0.smithi045.stdout:op 402 completed, throughput=4.98MB/sec 2021-07-31T02:06:31.011 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-FPEsCjBN9PU6epc off=7912485 len=176395 2021-07-31T02:06:31.014 INFO:tasks.workunit.client.0.smithi045.stdout:op 403 completed, throughput=4.98MB/sec 2021-07-31T02:06:31.014 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-BOZMxj_SJH07uuQ off=6947550 len=1075908 2021-07-31T02:06:31.020 INFO:tasks.workunit.client.0.smithi045.stdout:op 404 completed, throughput=5MB/sec 2021-07-31T02:06:31.021 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-H7GCay7mk51gZn9 off=8926598 len=903222 2021-07-31T02:06:31.025 INFO:tasks.workunit.client.0.smithi045.stdout:op 405 completed, throughput=5.01MB/sec 2021-07-31T02:06:31.068 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:06:31.069 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 21m ago 22m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:06:31.069 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 19m ago 21m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:06:31.069 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 19m ago 25m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:06:31.069 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 21m ago 27m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:06:31.070 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 21m ago 27m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:06:31.070 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 19m ago 25m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:06:31.070 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 21m ago 25m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:06:31.070 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 21m ago 22m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:06:31.071 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 19m ago 22m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:06:31.071 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 21m ago 25m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:06:31.071 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 21m ago 24m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:06:31.071 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 21m ago 24m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:06:31.072 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 21m ago 24m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:06:31.072 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 19m ago 23m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:06:31.072 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 19m ago 23m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:06:31.072 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 19m ago 23m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:06:31.072 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 19m ago 22m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:06:31.073 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 19m ago 22m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:06:31.407 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:06:31.448 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:06:31.449 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:06:31.449 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:06:31.449 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:06:31.450 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:06:31.450 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:06:31.450 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:06:31.450 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:06:31.451 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:06:31.451 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:06:31.451 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:06:31.451 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:06:31.451 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:06:31.452 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:06:31.452 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:06:31.452 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:06:31.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:06:30.013561+0000 mgr.x (mgr.24542) 820 : cephadm [WRN] 2021-07-31T02:06:31.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:06:31.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:06:30.056406+0000 mgr.x (mgr.24542) 821 : cephadm [ERR] 2021-07-31T02:06:31.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:06:31.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:06:31.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: yield 2021-07-31T02:06:31.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:06:31.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:06:31.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:06:31.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:06:31.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:06:31.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:06:31.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:06:31.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:06:31.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:06:31.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: task.result() 2021-07-31T02:06:31.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:06:31.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:06:30.056821 2021-07-31T02:06:31.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: +0000 mgr.x (mgr.24542) 822 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:06:31.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: cluster 2021-07-31T02:06:30.064428+0000 mgr.x (mgr.24542) 823 : cluster [DBG] pgmap v1450: 9 pgs: 9 active+clean; 127 GiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 2.6 MiB/s rd, 3.4 MiB/s wr, 5 op/s 2021-07-31T02:06:31.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:06:30.069009+0000 mgr.x (mgr.24542) 824 : cephadm [ERR] 2021-07-31T02:06:31.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:06:31.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:06:31.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: yield 2021-07-31T02:06:31.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:06:31.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:06:31.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:06:31.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:06:31.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:06:31.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:06:31.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:06:31.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:06:31.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:06:31.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: task.result() 2021-07-31T02:06:31.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:06:31.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:06:30.069810+0000 mgr.x (mgr.24542) 825 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:06:31.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:06:30.093812+0000 mgr.x (mgr.24542) 826 : cephadm [ERR] 2021-07-31T02:06:31.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:06:31.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:06:31.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: yield 2021-07-31T02:06:31.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:06:31.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:06:31.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:06:31.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:06:31.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:06:31.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:06:31.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:06:31.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:06:31.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:06:31.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: task.result() 2021-07-31T02:06:31.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:06:31.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:06:30.094412 2021-07-31T02:06:31.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: +0000 mgr.x (mgr.24542) 827 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:06:31.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:06:31.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:06:31.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: yield 2021-07-31T02:06:31.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:06:31.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:06:31.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:06:31.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:06:31.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:06:31.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:06:31.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:06:31.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:06:31.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:06:31.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: task.result() 2021-07-31T02:06:31.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:06:31.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: 2021-07-31T02:06:31.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:06:31.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: 2021-07-31T02:06:31.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:06:31.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:06:31.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:06:31.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:06:31.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:06:31.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:06:31.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:06:31.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:06:31.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:06:31.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:06:31.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:06:31.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:06:31.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:06:31.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:06:31.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:06:31.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:06:31.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:06:31.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:06:31.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: raise 2021-07-31T02:06:31.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:06:31.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:06:31.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:06:31.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:06:31.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:06:31.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:06:30.102809+0000 mgr.x (mgr.24542) 828 : cephadm [ERR] 2021-07-31T02:06:31.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:06:31.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:06:31.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: yield 2021-07-31T02:06:31.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:06:31.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:06:31.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:06:31.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:06:31.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:06:31.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:06:31.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:06:31.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:06:31.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:06:31.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: task.result() 2021-07-31T02:06:31.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:06:31.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:06:30.103165+0000 mgr.x (mgr.24542) 829 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:06:31.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:06:31.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:06:31.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: yield 2021-07-31T02:06:31.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:06:31.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:06:31.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:06:31.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:06:31.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:06:31.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:06:31.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:06:31.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:06:31.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:06:31.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: task.result() 2021-07-31T02:06:31.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:06:31.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: 2021-07-31T02:06:31.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:06:31.541 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: 2021-07-31T02:06:31.541 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:06:31.541 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:06:31.541 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:06:31.542 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:06:31.542 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:06:31.542 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:06:31.543 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:06:31.543 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:06:31.543 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:06:31.543 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:06:31.543 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:06:31.544 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:06:31.544 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:06:31.544 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:06:31.544 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:06:31.544 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:06:31.545 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:06:31.545 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:06:31.545 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: raise 2021-07-31T02:06:31.545 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:06:31.546 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:06:31.546 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:06:31.546 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:06:31.546 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:06:31.546 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:31 smithi154 conmon[34553]: 2021-07-31T02:06:31.547 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:06:30.013561 2021-07-31T02:06:31.547 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 820 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:06:31.548 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: cephadm 2021 2021-07-31T02:06:31.548 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: -07-31T02:06:30.056406+0000 mgr.x (mgr.24542) 821 : cephadm [ERR] 2021-07-31T02:06:31.548 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:06:31.548 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:06:31.549 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: yield 2021-07-31T02:06:31.549 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:06:31.549 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:06:31.549 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:06:31.549 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:06:31.550 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:06:31.550 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:06:31.550 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:06:31.550 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:06:31.551 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:06:31.551 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: task.result() 2021-07-31T02:06:31.551 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:06:31.551 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:06:30.056821+0000 mgr.x (mgr.24542) 822 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:06:31.551 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: cluster 2021-07-31T02:06:31.552 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: 2021-07-31T02:06 2021-07-31T02:06:31.552 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: :30.064428+0000 2021-07-31T02:06:31.552 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: mgr.x (mgr.24542) 823 : cluster 2021-07-31T02:06:31.553 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: [DBG] pgmap v1450: 9 pgs: 9 active+clean; 127 GiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 2.6 MiB/s rd, 3.4 MiB/s wr, 5 op/s 2021-07-31T02:06:31.553 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:06:31.553 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: 2021-07-31T02:06:30 2021-07-31T02:06:31.553 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: .069009+0000 mgr.x (mgr.24542 2021-07-31T02:06:31.553 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: ) 824 : cephadm [ERR] 2021-07-31T02:06:31.554 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:06:31.554 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:06:31.554 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: yield 2021-07-31T02:06:31.554 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:06:31.555 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:06:31.555 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:06:31.555 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:06:31.555 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:06:31.555 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:06:31.556 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:06:31.556 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:06:31.556 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:06:31.556 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: task.result() 2021-07-31T02:06:31.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:06:30. 2021-07-31T02:06:31.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: 013561+0000 mgr.x (mgr.24542) 820 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:06:31.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:06:31.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: 2021-07-31T02:06:30.056406+0000 mgr.x (mgr.24542) 2021-07-31T02:06:31.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: 821 : cephadm [ERR] 2021-07-31T02:06:31.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: 2021-07-31T02:06:31.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:06:31.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:06:31.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: yield 2021-07-31T02:06:31.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:06:31.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:06:31.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:06:31.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:06:31.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:06:31.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:06:31.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:06:31.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:06:31.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:06:31.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: task.result() 2021-07-31T02:06:31.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:06:31.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:06:30.056821+0000 mgr.x (mgr.24542) 822 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:06:31.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: cluster 2021-07-31T02:06:30.064428+0000 mgr.x (mgr.24542) 823 : cluster [DBG] pgmap v1450: 9 pgs: 9 active+clean; 127 GiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 2.6 MiB/s rd, 3.4 MiB/s wr, 5 op/s 2021-07-31T02:06:31.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:06:30.069009+0000 2021-07-31T02:06:31.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: mgr.x (mgr.24542) 824 : cephadm [ERR] 2021-07-31T02:06:31.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:06:31.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:06:31.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: yield 2021-07-31T02:06:31.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:06:31.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:06:31.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:06:31.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:06:31.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:06:31.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:06:31.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:06:31.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:06:31.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:06:31.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: task.result() 2021-07-31T02:06:31.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:06:31.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: cephadm 2021-07-31T02: 2021-07-31T02:06:31.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:06:31.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:06:30.069810+0000 mgr.x (mgr.24542) 825 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:06:31.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: cephadm 2021 2021-07-31T02:06:31.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: -07-31T02:06:30.093812+0000 2021-07-31T02:06:31.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: mgr.x (mgr.24542) 826 : cephadm 2021-07-31T02:06:31.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: [ERR] 2021-07-31T02:06:31.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:06:31.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:06:31.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: yield 2021-07-31T02:06:31.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:06:31.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:06:31.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:06:31.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:06:31.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:06:31.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:06:31.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:06:31.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:06:31.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:06:31.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: task.result() 2021-07-31T02:06:31.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:06:31.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:06:30.094412+0000 mgr.x (mgr.24542) 827 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:06:31.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:06:31.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:06:31.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: yield 2021-07-31T02:06:31.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:06:31.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:06:31.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:06:31.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:06:31.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:06:31.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:06:31.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:06:31.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:06:31.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:06:31.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: task.result() 2021-07-31T02:06:31.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:06:31.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: 2021-07-31T02:06:31.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:06:31.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: 2021-07-31T02:06:31.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:06:31.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:06:31.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:06:31.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:06:31.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:06:31.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: 06:30.069810+0000 mgr.x (mgr.24542) 825 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:06:31.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:06:30.093812+0000 mgr.x (mgr.24542) 826 : cephadm [ERR] 2021-07-31T02:06:31.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:06:31.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:06:31.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: yield 2021-07-31T02:06:31.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:06:31.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:06:31.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:06:31.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:06:31.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:06:31.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:06:31.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:06:31.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:06:31.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:06:31.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: task.result() 2021-07-31T02:06:31.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:06:31.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:06:30.094412+0000 mgr.x (mgr.24542) 827 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:06:31.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:06:31.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:06:31.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: yield 2021-07-31T02:06:31.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:06:31.585 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:06:31.585 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:06:31.585 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:06:31.585 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:06:31.585 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:06:31.586 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:06:31.586 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:06:31.586 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:06:31.587 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: task.result() 2021-07-31T02:06:31.587 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:06:31.587 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: 2021-07-31T02:06:31.587 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:06:31.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: 2021-07-31T02:06:31.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:06:31.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:06:31.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:06:31.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:06:31.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:06:31.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:06:31.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:06:31.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:06:31.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:06:31.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:06:31.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:06:31.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:06:31.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:06:31.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:06:31.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:06:31.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:06:31.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:06:31.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:06:31.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:06:31.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:06:31.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: raise 2021-07-31T02:06:31.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:06:31.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:06:31.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:06:31.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:06:31.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:06:31.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:06:30.102809+0000 mgr.x (mgr.24542) 828 : cephadm [ERR] 2021-07-31T02:06:31.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:06:31.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:06:31.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: yield 2021-07-31T02:06:31.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:06:31.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:06:31.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:06:31.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:06:31.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:06:31.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:06:31.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:06:31.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:06:31.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:06:31.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: task.result() 2021-07-31T02:06:31.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:06:31.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:06:30.103165+0000 mgr.x (mgr.24542) 829 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:06:31.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:06:31.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:06:31.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: yield 2021-07-31T02:06:31.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:06:31.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:06:31.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:06:31.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:06:31.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:06:31.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:06:31.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:06:31.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:06:31.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:06:31.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:06:31.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:06:31.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:06:31.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: raise 2021-07-31T02:06:31.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:06:31.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:06:31.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:06:31.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:06:31.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:06:31.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: cephadm 2021- 2021-07-31T02:06:31.603 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: 07-31T02:06:30.102809+0000 mgr.x (mgr.24542) 828 : cephadm [ERR] 2021-07-31T02:06:31.603 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:06:31.603 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:06:31.603 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: yield 2021-07-31T02:06:31.604 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:06:31.604 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:06:31.604 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:06:31.604 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:06:31.604 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:06:31.605 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:06:31.605 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:06:31.605 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:06:31.605 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:06:31.605 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: task.result() 2021-07-31T02:06:31.606 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:06:31.606 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:06:30.103165+0000 mgr.x (mgr.24542) 829 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:06:31.606 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:06:31.606 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:06:31.607 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: yield 2021-07-31T02:06:31.607 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:06:31.607 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:06:31.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:06:31.608 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:06:31.608 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:06:31.608 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:06:31.608 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:06:31.609 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:06:31.609 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:06:31.609 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:06:31.609 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: task.result() 2021-07-31T02:06:31.610 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:06:31.610 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: 2021-07-31T02:06:31.610 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:06:31.610 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: 2021-07-31T02:06:31.610 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:06:31.611 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:06:31.611 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:06:31.611 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:06:31.611 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:06:31.612 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:06:31.612 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:06:31.612 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:06:31.612 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:06:31.612 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:06:31.613 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:06:31.613 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:06:31.613 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:06:31.613 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:06:31.614 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:06:31.614 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:06:31.614 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:06:31.614 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:06:31.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: raise 2021-07-31T02:06:31.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:06:31.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:06:31.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:06:31.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:06:31.616 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:06:31.616 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:06:31.616 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:06:31.616 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:06:31.617 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:06:31.617 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:06:31.617 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:06:31.617 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:06:31.617 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: task.result() 2021-07-31T02:06:31.618 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:06:31.618 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: 2021-07-31T02:06:31.618 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:06:31.618 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: 2021-07-31T02:06:31.619 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:06:31.619 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:06:31.619 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:06:31.619 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:06:31.620 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:06:31.620 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:06:31.620 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:06:31.620 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:06:31.620 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:06:31.621 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:06:31.621 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:06:31.621 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:06:31.621 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:06:31.622 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:06:31.622 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:06:31.622 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:06:31.622 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:06:31.622 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:06:31.623 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: raise 2021-07-31T02:06:31.623 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:06:31.623 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:06:31.623 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:06:31.623 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:06:31.624 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:31 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:06:32.026 INFO:tasks.workunit.client.0.smithi045.stdout: 84: throughput=4.95MB/sec pending data=0 2021-07-31T02:06:32.026 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-5qMVj5d6M3ZX2md off=4993549 len=1371576 2021-07-31T02:06:32.029 INFO:tasks.workunit.client.0.smithi045.stdout:op 406 completed, throughput=4.96MB/sec 2021-07-31T02:06:32.029 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-fAQ37dxHXTZT9U8 off=2164156 len=1391327 2021-07-31T02:06:32.031 INFO:tasks.workunit.client.0.smithi045.stdout:op 407 completed, throughput=4.98MB/sec 2021-07-31T02:06:32.032 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-lP_x5-1Af2xFvxO off=2712233 len=1097022 2021-07-31T02:06:32.033 INFO:tasks.workunit.client.0.smithi045.stdout:op 408 completed, throughput=4.99MB/sec 2021-07-31T02:06:32.033 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-6aw5-YOxfazXJdx off=15744324 len=1100530 2021-07-31T02:06:32.035 INFO:tasks.workunit.client.0.smithi045.stdout:op 409 completed, throughput=5MB/sec 2021-07-31T02:06:32.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:32 smithi154 conmon[34553]: audit 2021-07-31T02:06:30.772052+0000 mgr.x (mgr.24542) 830 : audit [DBG] from='client.25678 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:06:32.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:32 smithi154 conmon[34553]: audit 2021-07-31T02:06:31.057522+0000 mgr.x (mgr.24542) 831 : audit [DBG] from='client.25684 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:06:32.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:32 smithi154 conmon[34553]: audit 2021-07-31T02:06:31.406677+0000 mon.a (mon.0) 1854 2021-07-31T02:06:32.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:32 smithi154 conmon[34553]: : audit [DBG] from='client.? 172.21.15.45:0/1598186643' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:06:32.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:32 smithi045 conmon[39718]: audit 2021-07-31T02:06:30.772052+0000 2021-07-31T02:06:32.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:32 smithi045 conmon[39718]: mgr.x (mgr.24542) 830 : audit [DBG] from='client.25678 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:06:32.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:32 smithi045 conmon[39718]: audit 2021-07-31T02:06:31. 2021-07-31T02:06:32.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:32 smithi045 conmon[39718]: 057522+0000 mgr.x (mgr.24542) 831 : audit [DBG] from='client.25684 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:06:32.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:32 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T02:06:32.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:32 smithi045 conmon[39718]: T02:06:31.406677+0000 mon.a (mon.0 2021-07-31T02:06:32.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:32 smithi045 conmon[39718]: ) 1854 : audit [DBG] from='client.? 172.21.15.45:0/1598186643' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:06:32.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:32 smithi045 conmon[33655]: audit 2021-07-31T02:06:30.772052+0000 mgr.x (mgr.24542 2021-07-31T02:06:32.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:32 smithi045 conmon[33655]: ) 830 : audit [DBG] from='client.25678 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:06:32.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:32 smithi045 conmon[33655]: audit 2021-07-31T02:06:31.057522+0000 mgr.x (mgr.24542 2021-07-31T02:06:32.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:32 smithi045 conmon[33655]: ) 831 : audit [DBG] from='client.25684 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:06:32.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:32 smithi045 conmon[33655]: audit 2021- 2021-07-31T02:06:32.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:32 smithi045 conmon[33655]: 07-31T02:06:31.406677 2021-07-31T02:06:32.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:32 smithi045 conmon[33655]: +0000 mon.a (mon.0) 1854 : audit [DBG] from='client.? 172.21.15.45:0/1598186643' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:06:32.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:32 smithi045 conmon[33655]: 2021-07-31T02:06:33.036 INFO:tasks.workunit.client.0.smithi045.stdout: 85: throughput=4.94MB/sec pending data=0 2021-07-31T02:06:33.036 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-h2d8xVdfLG6hYwE off=14528416 len=700012 2021-07-31T02:06:33.037 INFO:tasks.workunit.client.0.smithi045.stdout:op 410 completed, throughput=4.95MB/sec 2021-07-31T02:06:33.037 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-peG_kQdfvZNd0xm off=5153263 len=612761 2021-07-31T02:06:33.042 INFO:tasks.workunit.client.0.smithi045.stdout:op 411 completed, throughput=4.96MB/sec 2021-07-31T02:06:33.042 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-RCgwizbieefxOgK off=9142688 len=340532 2021-07-31T02:06:33.045 INFO:tasks.workunit.client.0.smithi045.stdout:op 412 completed, throughput=4.96MB/sec 2021-07-31T02:06:33.045 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-fp4va5uACUz655y off=7216595 len=764853 2021-07-31T02:06:33.049 INFO:tasks.workunit.client.0.smithi045.stdout:op 413 completed, throughput=4.97MB/sec 2021-07-31T02:06:33.049 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-DeiGVDBJ_NlNDbp off=4827500 len=1427896 2021-07-31T02:06:33.056 INFO:tasks.workunit.client.0.smithi045.stdout:op 414 completed, throughput=4.98MB/sec 2021-07-31T02:06:33.056 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-sDLuCZVn7TMCI7r off=2571425 len=1160114 2021-07-31T02:06:33.058 INFO:tasks.workunit.client.0.smithi045.stdout:op 415 completed, throughput=5MB/sec 2021-07-31T02:06:33.059 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-nls-6uHGq7Fp_bo off=5924594 len=65965 2021-07-31T02:06:33.059 INFO:tasks.workunit.client.0.smithi045.stdout:op 416 completed, throughput=5MB/sec 2021-07-31T02:06:33.059 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-eZ677Lgnvv5S_pG off=5078839 len=948063 2021-07-31T02:06:33.061 INFO:tasks.workunit.client.0.smithi045.stdout:op 417 completed, throughput=5.01MB/sec 2021-07-31T02:06:33.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:33 smithi154 conmon[34553]: cluster 2021-07-31T02:06:32.065007+0000 mgr.x (mgr.24542) 832 : cluster [DBG] pgmap v1451: 9 pgs: 9 active+clean; 127 GiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.2 MiB/s wr, 3 op/s 2021-07-31T02:06:33.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:33 smithi045 conmon[33655]: cluster 2021-07-31T02:06:32.065007+0000 mgr.x (mgr.24542) 832 : cluster [DBG] 2021-07-31T02:06:33.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:33 smithi045 conmon[33655]: pgmap v1451: 9 pgs: 9 active+clean; 127 GiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.2 MiB/s wr, 3 op/s 2021-07-31T02:06:33.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:33 smithi045 conmon[39718]: cluster 2021-07-31T02:06:32.065007+0000 mgr.x ( 2021-07-31T02:06:33.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:33 smithi045 conmon[39718]: mgr.24542) 832 : cluster [DBG] pgmap v1451: 9 pgs: 9 active+clean; 127 GiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.2 MiB/s wr, 3 op/s 2021-07-31T02:06:34.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:06:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:06:33.540Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:06:34.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:06:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:06:33.540Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:06:34.027 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:06:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:06:33.540Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:06:34.063 INFO:tasks.workunit.client.0.smithi045.stdout: 86: throughput=4.95MB/sec pending data=0 2021-07-31T02:06:34.063 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-IMLOggbWTWdNKxo off=1562295 len=1157484 2021-07-31T02:06:34.068 INFO:tasks.workunit.client.0.smithi045.stdout:op 418 completed, throughput=4.96MB/sec 2021-07-31T02:06:34.068 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-SbTD1-bUYMr3aRS off=3059361 len=1455090 2021-07-31T02:06:34.079 INFO:tasks.workunit.client.0.smithi045.stdout:op 419 completed, throughput=4.98MB/sec 2021-07-31T02:06:34.079 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-9qHtFgoQFdufDjs off=2435765 len=1275457 2021-07-31T02:06:34.083 INFO:tasks.workunit.client.0.smithi045.stdout:op 420 completed, throughput=4.99MB/sec 2021-07-31T02:06:34.083 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-oRzTjALFsJyGbwC off=2299276 len=1037030 2021-07-31T02:06:34.089 INFO:tasks.workunit.client.0.smithi045.stdout:op 421 completed, throughput=5MB/sec 2021-07-31T02:06:35.090 INFO:tasks.workunit.client.0.smithi045.stdout: 87: throughput=4.95MB/sec pending data=0 2021-07-31T02:06:35.090 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-hPBAG7wUeeIBTzB off=2200190 len=1396853 2021-07-31T02:06:35.096 INFO:tasks.workunit.client.0.smithi045.stdout:op 422 completed, throughput=4.96MB/sec 2021-07-31T02:06:35.096 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Q6FbF1MGFMGTCge off=7876285 len=1685390 2021-07-31T02:06:35.101 INFO:tasks.workunit.client.0.smithi045.stdout:op 423 completed, throughput=4.98MB/sec 2021-07-31T02:06:35.101 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-0dDwerH5eIc7YlR off=249860 len=1088980 2021-07-31T02:06:35.107 INFO:tasks.workunit.client.0.smithi045.stdout:op 424 completed, throughput=4.99MB/sec 2021-07-31T02:06:35.107 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-pqCxyeIca4ujWbJ off=6794519 len=340243 2021-07-31T02:06:35.110 INFO:tasks.workunit.client.0.smithi045.stdout:op 425 completed, throughput=4.99MB/sec 2021-07-31T02:06:35.111 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-d67Y0DzhEl_aJo0 off=1352727 len=1373115 2021-07-31T02:06:35.113 INFO:tasks.workunit.client.0.smithi045.stdout:op 426 completed, throughput=5.01MB/sec 2021-07-31T02:06:35.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:35 smithi154 conmon[34553]: cluster 2021-07-31T02:06:34.065423+0000 mgr.x (mgr.24542) 833 : cluster [DBG] pgmap v1452: 9 pgs: 9 active+clean; 127 GiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 2.8 MiB/s rd, 3.2 MiB/s wr, 5 op/s 2021-07-31T02:06:35.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:35 smithi045 conmon[39718]: cluster 2021-07-31T02:06:34.065423+0000 mgr.x (mgr.24542) 833 : cluster 2021-07-31T02:06:35.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:35 smithi045 conmon[39718]: [DBG] pgmap v1452: 9 pgs: 9 active+clean; 127 GiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 2.8 MiB/s rd, 3.2 MiB/s wr, 5 op/s 2021-07-31T02:06:35.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:35 smithi045 conmon[33655]: cluster 2021-07-31T02:06:34. 2021-07-31T02:06:35.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:35 smithi045 conmon[33655]: 065423+0000 mgr.x (mgr.24542) 833 : cluster [DBG] pgmap v1452: 9 pgs: 9 active+clean; 127 GiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 2.8 MiB/s rd, 3.2 MiB/s wr, 5 op/s 2021-07-31T02:06:36.113 INFO:tasks.workunit.client.0.smithi045.stdout: 88: throughput=4.95MB/sec pending data=0 2021-07-31T02:06:36.114 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ZitL5bwfrtU46pc off=3244771 len=1928561 2021-07-31T02:06:36.117 INFO:tasks.workunit.client.0.smithi045.stdout:op 427 completed, throughput=4.97MB/sec 2021-07-31T02:06:36.118 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-g4KouB0_jC_8scQ off=5180280 len=53630 2021-07-31T02:06:36.120 INFO:tasks.workunit.client.0.smithi045.stdout:op 428 completed, throughput=4.97MB/sec 2021-07-31T02:06:36.121 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-yWZye-BPNUGZzAC off=22321488 len=1707113 2021-07-31T02:06:36.129 INFO:tasks.workunit.client.0.smithi045.stdout:op 429 completed, throughput=4.99MB/sec 2021-07-31T02:06:36.130 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-05NVdhRNh7taAdy off=4182079 len=1548073 2021-07-31T02:06:36.139 INFO:tasks.workunit.client.0.smithi045.stdout:op 430 completed, throughput=5.01MB/sec 2021-07-31T02:06:37.131 INFO:tasks.workunit.client.0.smithi045.stdout: 89: throughput=4.95MB/sec pending data=0 2021-07-31T02:06:37.132 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-F8Y_ynRgPTitZ6X off=13456043 len=1972756 2021-07-31T02:06:37.135 INFO:tasks.workunit.client.0.smithi045.stdout:op 431 completed, throughput=4.97MB/sec 2021-07-31T02:06:37.136 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-xcH5rgh022FOHMM off=22337296 len=1095445 2021-07-31T02:06:37.137 INFO:tasks.workunit.client.0.smithi045.stdout:op 432 completed, throughput=4.98MB/sec 2021-07-31T02:06:37.137 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-nnFYxafxxiVcATa off=1991569 len=1433126 2021-07-31T02:06:37.140 INFO:tasks.workunit.client.0.smithi045.stdout:op 433 completed, throughput=5MB/sec 2021-07-31T02:06:37.141 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-6HQwfgEyiVqBOK0 off=7659990 len=1048788 2021-07-31T02:06:37.141 INFO:tasks.workunit.client.0.smithi045.stdout:op 434 completed, throughput=5.01MB/sec 2021-07-31T02:06:37.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:37 smithi154 conmon[34553]: cluster 2021-07-31T02:06:36.065808+0000 mgr.x (mgr.24542) 834 : cluster [DBG] pgmap v1453: 9 pgs: 9 active+clean; 127 GiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.1 MiB/s wr, 3 op/s 2021-07-31T02:06:37.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:37 smithi154 conmon[34553]: 2021-07-31T02:06:37.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:37 smithi045 conmon[39718]: cluster 2021-07-31T02:06:36.065808+0000 mgr.x (mgr.24542) 2021-07-31T02:06:37.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:37 smithi045 conmon[39718]: 834 : cluster [DBG] pgmap v1453: 9 pgs: 9 active+clean; 127 GiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.1 MiB/s wr, 3 op/s 2021-07-31T02:06:37.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:37 smithi045 conmon[33655]: cluster 2021-07-31T02:06:36.065808+0000 mgr.x (mgr.24542) 834 : cluster [DBG] pgmap v1453: 9 pgs: 9 active+clean; 127 GiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.1 MiB/s wr, 3 op/s 2021-07-31T02:06:38.142 INFO:tasks.workunit.client.0.smithi045.stdout: 90: throughput=4.96MB/sec pending data=0 2021-07-31T02:06:38.142 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ZTBpcu2UXnxVMNT off=4788904 len=644212 2021-07-31T02:06:38.144 INFO:tasks.workunit.client.0.smithi045.stdout:op 435 completed, throughput=4.96MB/sec 2021-07-31T02:06:38.144 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-N2Tdm1vN3QG8pbB off=2522226 len=1583384 2021-07-31T02:06:38.146 INFO:tasks.workunit.client.0.smithi045.stdout:op 436 completed, throughput=4.98MB/sec 2021-07-31T02:06:38.146 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-NFZs4th8ptah842 off=6232119 len=1716597 2021-07-31T02:06:38.149 INFO:tasks.workunit.client.0.smithi045.stdout:op 437 completed, throughput=5MB/sec 2021-07-31T02:06:38.149 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-08yrXYxLL6e2_ci off=11878932 len=61871 2021-07-31T02:06:38.149 INFO:tasks.workunit.client.0.smithi045.stdout:op 438 completed, throughput=5MB/sec 2021-07-31T02:06:38.150 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Nxv_aOHcd1QHBnL off=7492120 len=1119783 2021-07-31T02:06:38.156 INFO:tasks.workunit.client.0.smithi045.stdout:op 439 completed, throughput=5.01MB/sec 2021-07-31T02:06:39.155 INFO:tasks.workunit.client.0.smithi045.stdout: 91: throughput=4.95MB/sec pending data=0 2021-07-31T02:06:39.155 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-qbDTaPk1Mpufejn off=16359624 len=1848451 2021-07-31T02:06:39.157 INFO:tasks.workunit.client.0.smithi045.stdout:op 440 completed, throughput=4.97MB/sec 2021-07-31T02:06:39.158 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-O7JreyMhukAcZYo off=6227745 len=458888 2021-07-31T02:06:39.162 INFO:tasks.workunit.client.0.smithi045.stdout:op 441 completed, throughput=4.98MB/sec 2021-07-31T02:06:39.162 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-lVwHUwlx758RTn4 off=97697 len=1863863 2021-07-31T02:06:39.165 INFO:tasks.workunit.client.0.smithi045.stdout:op 442 completed, throughput=5MB/sec 2021-07-31T02:06:39.165 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-RXWtrSgcZ6k8TfS off=17058373 len=647983 2021-07-31T02:06:39.167 INFO:tasks.workunit.client.0.smithi045.stdout:op 443 completed, throughput=5MB/sec 2021-07-31T02:06:39.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:39 smithi154 conmon[34553]: cluster 2021-07-31T02:06:38.066139+0000 mgr.x (mgr.24542) 835 : cluster [DBG] pgmap v1454: 9 pgs: 9 active+clean; 127 GiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.1 MiB/s wr, 3 op/s 2021-07-31T02:06:39.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:39 smithi154 conmon[34553]: 2021-07-31T02:06:39.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:39 smithi045 conmon[33655]: cluster 2021-07-31T02:06:39.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:39 smithi045 conmon[33655]: 2021-07-31T02:06:38.066139+0000 mgr.x (mgr.24542) 835 : cluster [DBG] pgmap v1454: 9 pgs: 9 active+clean; 127 GiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.1 MiB/s wr, 3 op/s 2021-07-31T02:06:39.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:39 smithi045 conmon[39718]: cluster 2021-07-31T02:06:38.066139+0000 mgr.x (mgr.24542) 835 : cluster [DBG] 2021-07-31T02:06:39.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:39 smithi045 conmon[39718]: pgmap v1454: 9 pgs: 9 active+clean; 127 GiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.1 MiB/s wr, 3 op/s 2021-07-31T02:06:40.167 INFO:tasks.workunit.client.0.smithi045.stdout: 92: throughput=4.95MB/sec pending data=0 2021-07-31T02:06:40.167 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-aWLdD51FLTNvhxE off=6233024 len=677534 2021-07-31T02:06:40.173 INFO:tasks.workunit.client.0.smithi045.stdout:op 444 completed, throughput=4.96MB/sec 2021-07-31T02:06:40.174 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-YRZ5vHe2MuiRgwr off=5384101 len=299913 2021-07-31T02:06:40.176 INFO:tasks.workunit.client.0.smithi045.stdout:op 445 completed, throughput=4.96MB/sec 2021-07-31T02:06:40.176 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-0cYoQvbIlAxJflJ off=9596488 len=998591 2021-07-31T02:06:40.183 INFO:tasks.workunit.client.0.smithi045.stdout:op 446 completed, throughput=4.97MB/sec 2021-07-31T02:06:40.184 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-68ktNM5RdtJ3IwK off=5282512 len=1407963 2021-07-31T02:06:40.185 INFO:tasks.workunit.client.0.smithi045.stdout:op 447 completed, throughput=4.98MB/sec 2021-07-31T02:06:40.186 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-nGX5zOYhLWz7Ja1 off=2126610 len=1736313 2021-07-31T02:06:40.188 INFO:tasks.workunit.client.0.smithi045.stdout:op 448 completed, throughput=5MB/sec 2021-07-31T02:06:41.188 INFO:tasks.workunit.client.0.smithi045.stdout: 93: throughput=4.95MB/sec pending data=0 2021-07-31T02:06:41.188 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-gad9dNqh4eGwAzn off=4597097 len=793648 2021-07-31T02:06:41.190 INFO:tasks.workunit.client.0.smithi045.stdout:op 449 completed, throughput=4.96MB/sec 2021-07-31T02:06:41.190 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-kvv4ykVDOokt1G_ off=6102000 len=1250495 2021-07-31T02:06:41.197 INFO:tasks.workunit.client.0.smithi045.stdout:op 450 completed, throughput=4.97MB/sec 2021-07-31T02:06:41.197 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-14YJhboSjCLmjMw off=5874694 len=544233 2021-07-31T02:06:41.202 INFO:tasks.workunit.client.0.smithi045.stdout:op 451 completed, throughput=4.97MB/sec 2021-07-31T02:06:41.202 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ggS3fLYGYc40Di5 off=2248082 len=925076 2021-07-31T02:06:41.207 INFO:tasks.workunit.client.0.smithi045.stdout:op 452 completed, throughput=4.98MB/sec 2021-07-31T02:06:41.207 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-j4X3m9Cs5EY-1Cy off=2786202 len=1823318 2021-07-31T02:06:41.210 INFO:tasks.workunit.client.0.smithi045.stdout:op 453 completed, throughput=5MB/sec 2021-07-31T02:06:41.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:41 smithi154 conmon[34553]: cluster 2021-07-31T02:06:40.066565+0000 mgr.x (mgr.24542) 836 : cluster [DBG] pgmap v1455: 9 pgs: 9 active+clean; 127 GiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.9 MiB/s wr, 5 op/s 2021-07-31T02:06:41.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:41 smithi045 conmon[33655]: cluster 2021-07-31T02:06:40.066565+0000 mgr.x (mgr.24542) 836 : cluster [DBG] 2021-07-31T02:06:41.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:41 smithi045 conmon[33655]: pgmap v1455: 9 pgs: 9 active+clean; 127 GiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.9 MiB/s wr, 5 op/s 2021-07-31T02:06:41.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:41 smithi045 conmon[39718]: cluster 2021-07-31T02:06:40.066565+0000 mgr.x (mgr.24542) 2021-07-31T02:06:41.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:41 smithi045 conmon[39718]: 836 : cluster [DBG] pgmap v1455: 9 pgs: 9 active+clean; 127 GiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.9 MiB/s wr, 5 op/s 2021-07-31T02:06:42.211 INFO:tasks.workunit.client.0.smithi045.stdout: 94: throughput=4.95MB/sec pending data=0 2021-07-31T02:06:42.211 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-o1VUZIME7h8WR33 off=3784805 len=1927065 2021-07-31T02:06:42.215 INFO:tasks.workunit.client.0.smithi045.stdout:op 454 completed, throughput=4.97MB/sec 2021-07-31T02:06:42.215 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-HP3VwN8UVDEN9I9 off=616245 len=1700884 2021-07-31T02:06:42.218 INFO:tasks.workunit.client.0.smithi045.stdout:op 455 completed, throughput=4.98MB/sec 2021-07-31T02:06:42.218 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-2KvsNHSHK17lHqQ off=7294448 len=334895 2021-07-31T02:06:42.222 INFO:tasks.workunit.client.0.smithi045.stdout:op 456 completed, throughput=4.99MB/sec 2021-07-31T02:06:42.223 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-h31WbXNk9TR_OwJ off=10747901 len=1868030 2021-07-31T02:06:42.226 INFO:tasks.workunit.client.0.smithi045.stdout:op 457 completed, throughput=5.01MB/sec 2021-07-31T02:06:43.226 INFO:tasks.workunit.client.0.smithi045.stdout: 95: throughput=4.95MB/sec pending data=0 2021-07-31T02:06:43.227 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ZMph6kkMhJVAfqS off=6387602 len=1543692 2021-07-31T02:06:43.229 INFO:tasks.workunit.client.0.smithi045.stdout:op 458 completed, throughput=4.97MB/sec 2021-07-31T02:06:43.230 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ZbZwyO4Nim4SqRg off=4854759 len=391128 2021-07-31T02:06:43.230 INFO:tasks.workunit.client.0.smithi045.stdout:op 459 completed, throughput=4.97MB/sec 2021-07-31T02:06:43.231 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-8GUnAsPF9pgqYwI off=6288113 len=981832 2021-07-31T02:06:43.232 INFO:tasks.workunit.client.0.smithi045.stdout:op 460 completed, throughput=4.98MB/sec 2021-07-31T02:06:43.232 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-BzmoUU2mlyBJE4X off=14738463 len=393296 2021-07-31T02:06:43.233 INFO:tasks.workunit.client.0.smithi045.stdout:op 461 completed, throughput=4.99MB/sec 2021-07-31T02:06:43.233 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-MVOJ96Az7NPkAjX off=405209 len=179606 2021-07-31T02:06:43.237 INFO:tasks.workunit.client.0.smithi045.stdout:op 462 completed, throughput=4.99MB/sec 2021-07-31T02:06:43.237 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-VYvVp2RsZaie0ZU off=1293196 len=4901 2021-07-31T02:06:43.239 INFO:tasks.workunit.client.0.smithi045.stdout:op 463 completed, throughput=4.99MB/sec 2021-07-31T02:06:43.240 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ka70wPKgkvZBDLs off=5525781 len=120465 2021-07-31T02:06:43.242 INFO:tasks.workunit.client.0.smithi045.stdout:op 464 completed, throughput=4.99MB/sec 2021-07-31T02:06:43.242 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-LLI1nemDFbtVDX7 off=128606 len=1743807 2021-07-31T02:06:43.245 INFO:tasks.workunit.client.0.smithi045.stdout:op 465 completed, throughput=5.01MB/sec 2021-07-31T02:06:43.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:42 smithi154 conmon[34553]: cluster 2021-07-31T02:06:42.067002+0000 mgr.x (mgr.24542) 837 : cluster [DBG] pgmap v1456: 9 pgs: 9 active+clean; 127 GiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 2.4 MiB/s rd, 1.8 MiB/s wr, 3 op/s 2021-07-31T02:06:43.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:42 smithi045 conmon[39718]: cluster 2021-07-31T02:06:42.067002+0000 mgr.x (mgr.24542) 837 : cluster [DBG] 2021-07-31T02:06:43.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:42 smithi045 conmon[39718]: pgmap v1456: 9 pgs: 9 active+clean; 127 GiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 2.4 MiB/s rd, 1.8 MiB/s wr, 3 op/s 2021-07-31T02:06:43.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:42 smithi045 conmon[33655]: cluster 2021-07-31T02:06:42.067002+0000 mgr.x (mgr.24542) 837 : cluster [DBG] pgmap v1456: 9 pgs: 9 active+clean; 127 GiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 2.4 MiB/s rd, 1.8 MiB/s wr, 3 op/s 2021-07-31T02:06:44.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:06:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:06:43.541Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:06:44.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:06:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:06:43.541Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:06:44.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:06:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:06:43.541Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:06:44.245 INFO:tasks.workunit.client.0.smithi045.stdout: 96: throughput=4.95MB/sec pending data=0 2021-07-31T02:06:44.246 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-uzNjzq8fM6sqsJb off=4070747 len=746239 2021-07-31T02:06:44.246 INFO:tasks.workunit.client.0.smithi045.stdout:op 466 completed, throughput=4.96MB/sec 2021-07-31T02:06:44.247 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-SEuSQaYUBzmNcJu off=1352177 len=671092 2021-07-31T02:06:44.251 INFO:tasks.workunit.client.0.smithi045.stdout:op 467 completed, throughput=4.97MB/sec 2021-07-31T02:06:44.251 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-GCKHDXrK8ZSv-du off=11489218 len=923580 2021-07-31T02:06:44.258 INFO:tasks.workunit.client.0.smithi045.stdout:op 468 completed, throughput=4.98MB/sec 2021-07-31T02:06:44.258 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-6ZdfWkmWK05V24j off=109560 len=1457067 2021-07-31T02:06:44.261 INFO:tasks.workunit.client.0.smithi045.stdout:op 469 completed, throughput=4.99MB/sec 2021-07-31T02:06:44.261 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-0hrkI9oweZ8MPfT off=5653265 len=1854652 2021-07-31T02:06:44.271 INFO:tasks.workunit.client.0.smithi045.stdout:op 470 completed, throughput=5.01MB/sec 2021-07-31T02:06:44.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:44 smithi154 conmon[34553]: cluster 2021-07-31T02:06:44.067580+0000 mgr.x (mgr.24542) 838 : cluster [DBG] pgmap v1457: 9 pgs: 9 active+clean; 127 GiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 4.0 MiB/s rd, 2.3 MiB/s wr, 6 op/s 2021-07-31T02:06:44.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:44 smithi045 conmon[39718]: cluster 2021-07-31T02:06:44.067580+0000 mgr.x (mgr.24542) 838 : cluster [DBG] 2021-07-31T02:06:44.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:44 smithi045 conmon[39718]: pgmap v1457: 9 pgs: 9 active+clean; 127 GiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 4.0 MiB/s rd, 2.3 MiB/s wr, 6 op/s 2021-07-31T02:06:44.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:44 smithi045 conmon[33655]: cluster 2021-07-31T02:06:44.067580+0000 mgr.x (mgr.24542) 838 2021-07-31T02:06:44.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:44 smithi045 conmon[33655]: : cluster [DBG] pgmap v1457: 9 pgs: 9 active+clean; 127 GiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 4.0 MiB/s rd, 2.3 MiB/s wr, 6 op/s 2021-07-31T02:06:45.272 INFO:tasks.workunit.client.0.smithi045.stdout: 97: throughput=4.96MB/sec pending data=0 2021-07-31T02:06:45.272 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Ug4IlHTRCkzsjOW off=0 len=1233461 2021-07-31T02:06:45.277 INFO:tasks.workunit.client.0.smithi045.stdout:op 471 completed, throughput=4.97MB/sec 2021-07-31T02:06:45.277 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-sgWUDav7E1ua__t off=4653283 len=670297 2021-07-31T02:06:45.283 INFO:tasks.workunit.client.0.smithi045.stdout:op 472 completed, throughput=4.98MB/sec 2021-07-31T02:06:45.283 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-HgXLb0L4vv6pAyQ off=5278592 len=285297 2021-07-31T02:06:45.284 INFO:tasks.workunit.client.0.smithi045.stdout:op 473 completed, throughput=4.98MB/sec 2021-07-31T02:06:45.284 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-OlVyuKUPOcULTMZ off=2675998 len=347059 2021-07-31T02:06:45.287 INFO:tasks.workunit.client.0.smithi045.stdout:op 474 completed, throughput=4.98MB/sec 2021-07-31T02:06:45.287 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-bTInmVkgQ2NgGv9 off=7730192 len=480469 2021-07-31T02:06:45.288 INFO:tasks.workunit.client.0.smithi045.stdout:op 475 completed, throughput=4.99MB/sec 2021-07-31T02:06:45.288 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-3RurpQlC7J4yxq9 off=1487725 len=418916 2021-07-31T02:06:45.291 INFO:tasks.workunit.client.0.smithi045.stdout:op 476 completed, throughput=4.99MB/sec 2021-07-31T02:06:45.292 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-7xoMVvkKm4KGMzH off=6735311 len=930244 2021-07-31T02:06:45.297 INFO:tasks.workunit.client.0.smithi045.stdout:op 477 completed, throughput=5MB/sec 2021-07-31T02:06:45.297 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-sa6Wdwji7265iJj off=3418381 len=1101761 2021-07-31T02:06:45.302 INFO:tasks.workunit.client.0.smithi045.stdout:op 478 completed, throughput=5.01MB/sec 2021-07-31T02:06:46.302 INFO:tasks.workunit.client.0.smithi045.stdout: 98: throughput=4.96MB/sec pending data=0 2021-07-31T02:06:46.302 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-4EFkbjJMsoxiWzO off=2093660 len=2081111 2021-07-31T02:06:46.304 INFO:tasks.workunit.client.0.smithi045.stdout:op 479 completed, throughput=4.98MB/sec 2021-07-31T02:06:46.305 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ilnJxBzeCa7t0T7 off=8186296 len=1485560 2021-07-31T02:06:46.313 INFO:tasks.workunit.client.0.smithi045.stdout:op 480 completed, throughput=4.99MB/sec 2021-07-31T02:06:46.313 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Yb_Hx2yq4b6iMqX off=1504792 len=1575411 2021-07-31T02:06:46.316 INFO:tasks.workunit.client.0.smithi045.stdout:op 481 completed, throughput=5.01MB/sec 2021-07-31T02:06:46.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:46 smithi154 conmon[34553]: cluster 2021-07-31T02:06:46.067980+0000 mgr.x (mgr.24542) 839 : cluster [DBG] pgmap v1458: 9 pgs: 9 active+clean; 127 GiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 2.9 MiB/s rd, 1.3 MiB/s wr, 3 op/s 2021-07-31T02:06:46.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:46 smithi045 conmon[39718]: cluster 2021-07-31T02:06:46.067980+0000 mgr.x (mgr.24542) 839 : cluster [DBG] 2021-07-31T02:06:46.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:46 smithi045 conmon[39718]: pgmap v1458: 9 pgs: 9 active+clean; 127 GiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 2.9 MiB/s rd, 1.3 MiB/s wr, 3 op/s 2021-07-31T02:06:46.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:46 smithi045 conmon[33655]: cluster 2021-07-31T02:06:46.067980+0000 mgr.x (mgr.24542) 839 2021-07-31T02:06:46.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:46 smithi045 conmon[33655]: : cluster [DBG] pgmap v1458: 9 pgs: 9 active+clean; 127 GiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 2.9 MiB/s rd, 1.3 MiB/s wr, 3 op/s 2021-07-31T02:06:47.316 INFO:tasks.workunit.client.0.smithi045.stdout: 99: throughput=4.96MB/sec pending data=0 2021-07-31T02:06:47.316 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-nuj1fJDiCa7goVo off=7339391 len=293835 2021-07-31T02:06:47.318 INFO:tasks.workunit.client.0.smithi045.stdout:op 482 completed, throughput=4.96MB/sec 2021-07-31T02:06:47.318 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-kNIcxlvhVoMVdRW off=4960200 len=1287583 2021-07-31T02:06:47.327 INFO:tasks.workunit.client.0.smithi045.stdout:op 483 completed, throughput=4.97MB/sec 2021-07-31T02:06:47.327 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-lytt7eYBADYZO9r off=4605870 len=52972 2021-07-31T02:06:47.328 INFO:tasks.workunit.client.0.smithi045.stdout:op 484 completed, throughput=4.97MB/sec 2021-07-31T02:06:47.329 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-7CfRSM6XzQ-9tSh off=5919446 len=320119 2021-07-31T02:06:47.329 INFO:tasks.workunit.client.0.smithi045.stdout:op 485 completed, throughput=4.98MB/sec 2021-07-31T02:06:47.329 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-exXgX1UJ8YuOLJb off=6337400 len=1094001 2021-07-31T02:06:47.330 INFO:tasks.workunit.client.0.smithi045.stdout:op 486 completed, throughput=4.99MB/sec 2021-07-31T02:06:47.330 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-nd7-4l5l8pvK2E9 off=1157565 len=213154 2021-07-31T02:06:47.333 INFO:tasks.workunit.client.0.smithi045.stdout:op 487 completed, throughput=4.99MB/sec 2021-07-31T02:06:47.333 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-YZV5976H38O5wSc off=1516471 len=2070117 2021-07-31T02:06:47.337 INFO:tasks.workunit.client.0.smithi045.stdout:op 488 completed, throughput=5.01MB/sec 2021-07-31T02:06:48.337 INFO:tasks.workunit.client.0.smithi045.stdout: 100: throughput=4.96MB/sec pending data=0 2021-07-31T02:06:48.338 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-7abcL89bryy_Wsx off=1651764 len=859965 2021-07-31T02:06:48.343 INFO:tasks.workunit.client.0.smithi045.stdout:op 489 completed, throughput=4.97MB/sec 2021-07-31T02:06:48.343 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-L33Rz6KhEJcWb-z off=1440334 len=2003677 2021-07-31T02:06:48.355 INFO:tasks.workunit.client.0.smithi045.stdout:op 490 completed, throughput=4.98MB/sec 2021-07-31T02:06:48.355 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-798lWbByYDT-NcV off=16299147 len=1674013 2021-07-31T02:06:48.358 INFO:tasks.workunit.client.0.smithi045.stdout:op 491 completed, throughput=5MB/sec 2021-07-31T02:06:48.358 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-7QomePAvHxGiZod off=10941095 len=95593 2021-07-31T02:06:48.361 INFO:tasks.workunit.client.0.smithi045.stdout:op 492 completed, throughput=5MB/sec 2021-07-31T02:06:48.361 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-HvI3u8RLBar0OKy off=16448313 len=1745050 2021-07-31T02:06:48.364 INFO:tasks.workunit.client.0.smithi045.stdout:op 493 completed, throughput=5.02MB/sec 2021-07-31T02:06:48.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:48 smithi154 conmon[34553]: cluster 2021-07-31T02:06:48.068525+0000 mgr.x (mgr.24542) 840 : cluster [DBG] pgmap v1459: 9 pgs: 9 active+clean; 127 GiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 2.9 MiB/s rd, 1.3 MiB/s wr, 3 op/s 2021-07-31T02:06:48.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:48 smithi045 conmon[39718]: cluster 2021-07-31T02:06:48.068525+0000 mgr.x (mgr.24542) 840 : cluster 2021-07-31T02:06:48.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:48 smithi045 conmon[39718]: [DBG] pgmap v1459: 9 pgs: 9 active+clean; 127 GiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 2.9 MiB/s rd, 1.3 MiB/s wr, 3 op/s 2021-07-31T02:06:48.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:48 smithi045 conmon[33655]: cluster 2021-07-31T02:06:48.068525+0000 mgr.x (mgr 2021-07-31T02:06:48.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:48 smithi045 conmon[33655]: .24542) 840 : cluster [DBG] pgmap v1459: 9 pgs: 9 active+clean; 127 GiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 2.9 MiB/s rd, 1.3 MiB/s wr, 3 op/s 2021-07-31T02:06:49.364 INFO:tasks.workunit.client.0.smithi045.stdout: 101: throughput=4.97MB/sec pending data=0 2021-07-31T02:06:49.365 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-lt7EPIiODoij3mx off=9758373 len=1329528 2021-07-31T02:06:49.367 INFO:tasks.workunit.client.0.smithi045.stdout:op 494 completed, throughput=4.98MB/sec 2021-07-31T02:06:49.367 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ndgHXF8fsWSoPtr off=378351 len=787936 2021-07-31T02:06:49.372 INFO:tasks.workunit.client.0.smithi045.stdout:op 495 completed, throughput=4.99MB/sec 2021-07-31T02:06:49.373 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-xd0J4e1dv3VGVDX off=12712258 len=1938283 2021-07-31T02:06:49.380 INFO:tasks.workunit.client.0.smithi045.stdout:op 496 completed, throughput=5MB/sec 2021-07-31T02:06:50.380 INFO:tasks.workunit.client.0.smithi045.stdout: 102: throughput=4.96MB/sec pending data=0 2021-07-31T02:06:50.381 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Hh1gfx_NAhzNUWA off=11111231 len=451557 2021-07-31T02:06:50.385 INFO:tasks.workunit.client.0.smithi045.stdout:op 497 completed, throughput=4.96MB/sec 2021-07-31T02:06:50.385 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-PCQWa7EPKJvkXK8 off=15557230 len=656827 2021-07-31T02:06:50.386 INFO:tasks.workunit.client.0.smithi045.stdout:op 498 completed, throughput=4.97MB/sec 2021-07-31T02:06:50.386 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-V6_ya9c57R5rOhT off=6404889 len=314007 2021-07-31T02:06:50.387 INFO:tasks.workunit.client.0.smithi045.stdout:op 499 completed, throughput=4.97MB/sec 2021-07-31T02:06:50.387 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-06HjYjIOmLtwaD3 off=795759 len=1640517 2021-07-31T02:06:50.390 INFO:tasks.workunit.client.0.smithi045.stdout:op 500 completed, throughput=4.98MB/sec 2021-07-31T02:06:50.391 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-eFfzydc3BJr1U2i off=10707925 len=1946940 2021-07-31T02:06:50.400 INFO:tasks.workunit.client.0.smithi045.stdout:op 501 completed, throughput=5MB/sec 2021-07-31T02:06:50.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:50 smithi154 conmon[34553]: cluster 2021-07-31T02:06:50.068971+0000 mgr.x (mgr.24542) 841 : cluster [DBG] pgmap v1460: 9 pgs: 9 active+clean; 127 GiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 4.0 MiB/s rd, 2.4 MiB/s wr, 6 op/s 2021-07-31T02:06:50.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:50 smithi045 conmon[33655]: cluster 2021-07-31T02:06:50.068971+0000 mgr.x (mgr.24542) 841 2021-07-31T02:06:50.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:50 smithi045 conmon[33655]: : cluster [DBG] pgmap v1460: 9 pgs: 9 active+clean; 127 GiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 4.0 MiB/s rd, 2.4 MiB/s wr, 6 op/s 2021-07-31T02:06:50.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:50 smithi045 conmon[39718]: cluster 2021-07-31T02:06:50.068971+0000 mgr.x (mgr.24542) 841 : cluster [DBG] 2021-07-31T02:06:50.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:50 smithi045 conmon[39718]: pgmap v1460: 9 pgs: 9 active+clean; 127 GiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 4.0 MiB/s rd, 2.4 MiB/s wr, 6 op/s 2021-07-31T02:06:51.403 INFO:tasks.workunit.client.0.smithi045.stdout: 103: throughput=4.95MB/sec pending data=0 2021-07-31T02:06:51.403 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-gDpxK10vpMIY5fJ off=4013192 len=247779 2021-07-31T02:06:51.404 INFO:tasks.workunit.client.0.smithi045.stdout:op 502 completed, throughput=4.96MB/sec 2021-07-31T02:06:51.404 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-XnnIGNjU-GbKQRi off=7462950 len=1554809 2021-07-31T02:06:51.404 INFO:tasks.workunit.client.0.smithi045.stdout:op 503 completed, throughput=4.97MB/sec 2021-07-31T02:06:51.405 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-HEd-2YYQKVuWWBx off=6812578 len=1122013 2021-07-31T02:06:51.411 INFO:tasks.workunit.client.0.smithi045.stdout:op 504 completed, throughput=4.98MB/sec 2021-07-31T02:06:51.412 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-zEImey_SM8FuILU off=3767370 len=974741 2021-07-31T02:06:51.416 INFO:tasks.workunit.client.0.smithi045.stdout:op 505 completed, throughput=4.99MB/sec 2021-07-31T02:06:51.416 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-uOw2TeDVjIcFFxG off=17231083 len=1685123 2021-07-31T02:06:51.418 INFO:tasks.workunit.client.0.smithi045.stdout:op 506 completed, throughput=5MB/sec 2021-07-31T02:06:52.418 INFO:tasks.workunit.client.0.smithi045.stdout: 104: throughput=4.96MB/sec pending data=0 2021-07-31T02:06:52.418 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-IzQnCtOiUtowLcs off=7709121 len=1170194 2021-07-31T02:06:52.425 INFO:tasks.workunit.client.0.smithi045.stdout:op 507 completed, throughput=4.97MB/sec 2021-07-31T02:06:52.425 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-PI_gLTGMS8cwdyg off=6483642 len=2019856 2021-07-31T02:06:52.428 INFO:tasks.workunit.client.0.smithi045.stdout:op 508 completed, throughput=4.98MB/sec 2021-07-31T02:06:52.429 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-NlWnqm7trv54l4A off=3147562 len=601134 2021-07-31T02:06:52.433 INFO:tasks.workunit.client.0.smithi045.stdout:op 509 completed, throughput=4.99MB/sec 2021-07-31T02:06:52.433 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-KgMO3sJqezBUojZ off=3512491 len=967086 2021-07-31T02:06:52.434 INFO:tasks.workunit.client.0.smithi045.stdout:op 510 completed, throughput=5MB/sec 2021-07-31T02:06:52.434 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-WBOtnuTtIrIRxc9 off=15857587 len=313331 2021-07-31T02:06:52.439 INFO:tasks.workunit.client.0.smithi045.stdout:op 511 completed, throughput=5MB/sec 2021-07-31T02:06:53.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:52 smithi154 conmon[34553]: cluster 2021-07-31T02:06:52.069497+0000 mgr.x (mgr.24542) 842 : 2021-07-31T02:06:53.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:52 smithi154 conmon[34553]: cluster [DBG] pgmap v1461: 9 pgs: 9 active+clean; 127 GiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 2.7 MiB/s rd, 1.6 MiB/s wr, 4 op/s 2021-07-31T02:06:53.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:52 smithi045 conmon[33655]: cluster 2021-07-31T02:06:52. 2021-07-31T02:06:53.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:52 smithi045 conmon[33655]: 069497+0000 mgr.x (mgr.24542) 842 : cluster [DBG] pgmap v1461: 9 pgs: 9 active+clean; 127 GiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 2.7 MiB/s rd, 1.6 MiB/s wr, 4 op/s 2021-07-31T02:06:53.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:52 smithi045 conmon[39718]: cluster 2021-07-31T02:06:52.069497+0000 mgr.x (mgr.24542) 2021-07-31T02:06:53.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:52 smithi045 conmon[39718]: 842 : cluster [DBG] pgmap v1461: 9 pgs: 9 active+clean; 127 GiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 2.7 MiB/s rd, 1.6 MiB/s wr, 4 op/s 2021-07-31T02:06:53.438 INFO:tasks.workunit.client.0.smithi045.stdout: 105: throughput=4.95MB/sec pending data=0 2021-07-31T02:06:53.438 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-GsPoW0CSc5k6y6I off=6793910 len=1710027 2021-07-31T02:06:53.440 INFO:tasks.workunit.client.0.smithi045.stdout:op 512 completed, throughput=4.97MB/sec 2021-07-31T02:06:53.440 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Y-qhJP50Kdh-WqA off=409682 len=1899550 2021-07-31T02:06:53.444 INFO:tasks.workunit.client.0.smithi045.stdout:op 513 completed, throughput=4.99MB/sec 2021-07-31T02:06:53.444 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-XuNmc4FuSAl1V8g off=9097252 len=862197 2021-07-31T02:06:53.449 INFO:tasks.workunit.client.0.smithi045.stdout:op 514 completed, throughput=4.99MB/sec 2021-07-31T02:06:53.450 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-eLhco4hrzcnsY2G off=2483676 len=1110824 2021-07-31T02:06:53.451 INFO:tasks.workunit.client.0.smithi045.stdout:op 515 completed, throughput=5MB/sec 2021-07-31T02:06:54.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:06:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:06:53.541Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:06:54.027 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:06:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:06:53.541Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:06:54.027 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:06:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:06:53.541Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:06:54.451 INFO:tasks.workunit.client.0.smithi045.stdout: 106: throughput=4.96MB/sec pending data=0 2021-07-31T02:06:54.452 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-IF-IRrkbslpSdXk off=4801563 len=1162945 2021-07-31T02:06:54.458 INFO:tasks.workunit.client.0.smithi045.stdout:op 516 completed, throughput=4.97MB/sec 2021-07-31T02:06:54.458 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-kR2tU1G_jve2rsx off=3695474 len=1790415 2021-07-31T02:06:54.461 INFO:tasks.workunit.client.0.smithi045.stdout:op 517 completed, throughput=4.98MB/sec 2021-07-31T02:06:54.462 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Ott343r-GM9Wkm3 off=14039997 len=1913210 2021-07-31T02:06:54.465 INFO:tasks.workunit.client.0.smithi045.stdout:op 518 completed, throughput=5MB/sec 2021-07-31T02:06:54.465 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-jDBoQULDcyiYVjE off=7031030 len=2011880 2021-07-31T02:06:54.469 INFO:tasks.workunit.client.0.smithi045.stdout:op 519 completed, throughput=5.02MB/sec 2021-07-31T02:06:54.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:54 smithi154 conmon[34553]: cluster 2021-07-31T02:06:54.070124+0000 mgr.x (mgr.24542) 843 : cluster [DBG] 2021-07-31T02:06:54.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:54 smithi154 conmon[34553]: pgmap v1462: 9 pgs: 9 active+clean; 127 GiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 3.9 MiB/s rd, 2.4 MiB/s wr, 6 op/s 2021-07-31T02:06:54.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:54 smithi045 conmon[33655]: cluster 2021-07-31T02:06:54.070124+0000 mgr.x (mgr.24542) 843 : cluster [DBG] pgmap v1462: 9 pgs: 9 active+clean; 127 GiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 3.9 MiB/s rd, 2.4 MiB/s wr, 6 op/s 2021-07-31T02:06:54.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:54 smithi045 conmon[39718]: cluster 2021-07-31T02:06:54.070124+0000 mgr.x (mgr. 2021-07-31T02:06:54.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:54 smithi045 conmon[39718]: 24542) 843 : cluster [DBG] pgmap v1462: 9 pgs: 9 active+clean; 127 GiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 3.9 MiB/s rd, 2.4 MiB/s wr, 6 op/s 2021-07-31T02:06:55.469 INFO:tasks.workunit.client.0.smithi045.stdout: 107: throughput=4.97MB/sec pending data=0 2021-07-31T02:06:55.469 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-rIX-rkqm10uOrJQ off=2595443 len=1588798 2021-07-31T02:06:55.471 INFO:tasks.workunit.client.0.smithi045.stdout:op 520 completed, throughput=4.98MB/sec 2021-07-31T02:06:55.472 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ql-BjSVNDXkk-1K off=1728073 len=1751674 2021-07-31T02:06:55.481 INFO:tasks.workunit.client.0.smithi045.stdout:op 521 completed, throughput=5MB/sec 2021-07-31T02:06:55.482 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-_MGZeMu_9twxvWy off=2740211 len=1288195 2021-07-31T02:06:55.489 INFO:tasks.workunit.client.0.smithi045.stdout:op 522 completed, throughput=5.01MB/sec 2021-07-31T02:06:56.489 INFO:tasks.workunit.client.0.smithi045.stdout: 108: throughput=4.96MB/sec pending data=0 2021-07-31T02:06:56.490 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-WJxDBboH7Km5iYi off=9618157 len=320384 2021-07-31T02:06:56.493 INFO:tasks.workunit.client.0.smithi045.stdout:op 523 completed, throughput=4.97MB/sec 2021-07-31T02:06:56.493 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-yKiUYw2pelba-i4 off=15702692 len=664215 2021-07-31T02:06:56.494 INFO:tasks.workunit.client.0.smithi045.stdout:op 524 completed, throughput=4.97MB/sec 2021-07-31T02:06:56.495 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-L1g7PoEZ3eRKz_8 off=1679268 len=224014 2021-07-31T02:06:56.498 INFO:tasks.workunit.client.0.smithi045.stdout:op 525 completed, throughput=4.97MB/sec 2021-07-31T02:06:56.498 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-DGJX9QXk-g360Sh off=1098120 len=872586 2021-07-31T02:06:56.499 INFO:tasks.workunit.client.0.smithi045.stdout:op 526 completed, throughput=4.98MB/sec 2021-07-31T02:06:56.500 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-S45uy5OZhk-qx8T off=1063227 len=1253831 2021-07-31T02:06:56.501 INFO:tasks.workunit.client.0.smithi045.stdout:op 527 completed, throughput=4.99MB/sec 2021-07-31T02:06:56.502 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-q1IZ_aD0WumL3Dg off=6140069 len=1432058 2021-07-31T02:06:56.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:56 smithi154 conmon[34553]: cluster 2021-07-31T02:06:56.070482+0000 mgr.x (mgr.24542) 844 : cluster [DBG] pgmap v1463: 9 pgs: 9 active+clean; 127 GiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 2.3 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2021-07-31T02:06:56.504 INFO:tasks.workunit.client.0.smithi045.stdout:op 528 completed, throughput=5.01MB/sec 2021-07-31T02:06:56.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:56 smithi045 conmon[33655]: cluster 2021-07-31T02:06:56.070482+0000 mgr.x (mgr.24542) 844 : cluster [DBG] pgmap v1463: 9 pgs: 9 active+clean; 127 GiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 2.3 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2021-07-31T02:06:56.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:56 smithi045 conmon[39718]: cluster 2021-07-31T02:06:56.070482+0000 mgr.x (mgr. 2021-07-31T02:06:56.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:56 smithi045 conmon[39718]: 24542) 844 : cluster [DBG] pgmap v1463: 9 pgs: 9 active+clean; 127 GiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 2.3 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2021-07-31T02:06:57.503 INFO:tasks.workunit.client.0.smithi045.stdout: 109: throughput=4.96MB/sec pending data=0 2021-07-31T02:06:57.503 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-m_fh0epu7QWFL4C off=6734517 len=805506 2021-07-31T02:06:57.505 INFO:tasks.workunit.client.0.smithi045.stdout:op 529 completed, throughput=4.97MB/sec 2021-07-31T02:06:57.506 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-0yWBxy9ABci7Hkc off=3777662 len=1480002 2021-07-31T02:06:57.514 INFO:tasks.workunit.client.0.smithi045.stdout:op 530 completed, throughput=4.98MB/sec 2021-07-31T02:06:57.514 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-0xCnyIdmNzYZj_q off=2755312 len=2304 2021-07-31T02:06:57.515 INFO:tasks.workunit.client.0.smithi045.stdout:op 531 completed, throughput=4.98MB/sec 2021-07-31T02:06:57.515 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-UFF75lcVxs9R_Wz off=2693008 len=51288 2021-07-31T02:06:57.517 INFO:tasks.workunit.client.0.smithi045.stdout:op 532 completed, throughput=4.98MB/sec 2021-07-31T02:06:57.517 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-68ktNM5RdtJ3IwK off=2914453 len=237513 2021-07-31T02:06:57.520 INFO:tasks.workunit.client.0.smithi045.stdout:op 533 completed, throughput=4.98MB/sec 2021-07-31T02:06:57.520 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-wF-srTnJifucTT7 off=8409521 len=274107 2021-07-31T02:06:57.521 INFO:tasks.workunit.client.0.smithi045.stdout:op 534 completed, throughput=4.98MB/sec 2021-07-31T02:06:57.521 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-GO2vh1MHmtqo4pG off=3387718 len=824490 2021-07-31T02:06:57.523 INFO:tasks.workunit.client.0.smithi045.stdout:op 535 completed, throughput=4.99MB/sec 2021-07-31T02:06:57.523 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-O7-KomE4ZEZbo8V off=1704712 len=1668327 2021-07-31T02:06:57.525 INFO:tasks.workunit.client.0.smithi045.stdout:op 536 completed, throughput=5MB/sec 2021-07-31T02:06:58.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:06:58 smithi154 conmon[34553]: cluster 2021-07-31T02:06:58.070837+0000 mgr.x (mgr.24542) 845 : cluster [DBG] pgmap v1464: 9 pgs: 9 active+clean; 127 GiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 2.3 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2021-07-31T02:06:58.525 INFO:tasks.workunit.client.0.smithi045.stdout: 110: throughput=4.96MB/sec pending data=0 2021-07-31T02:06:58.526 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-mM8xo79X3RekzUq off=2022835 len=534004 2021-07-31T02:06:58.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:06:58 smithi045 conmon[39718]: cluster 2021-07-31T02:06:58.070837+0000 mgr.x (mgr.24542) 845 : cluster [DBG] pgmap v1464: 9 pgs: 9 active+clean; 127 GiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 2.3 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2021-07-31T02:06:58.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:06:58 smithi045 conmon[33655]: cluster 2021-07-31T02:06:58.070837+0000 mgr.x (mgr.24542) 845 : cluster [DBG] pgmap v1464: 9 pgs: 9 active+clean; 127 GiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 2.3 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2021-07-31T02:06:58.529 INFO:tasks.workunit.client.0.smithi045.stdout:op 537 completed, throughput=4.96MB/sec 2021-07-31T02:06:58.529 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-YF3o8oFy14HJbSY off=8352640 len=1428789 2021-07-31T02:06:58.536 INFO:tasks.workunit.client.0.smithi045.stdout:op 538 completed, throughput=4.98MB/sec 2021-07-31T02:06:58.537 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Gov0pVKZvKM391V off=2639205 len=33826 2021-07-31T02:06:58.538 INFO:tasks.workunit.client.0.smithi045.stdout:op 539 completed, throughput=4.98MB/sec 2021-07-31T02:06:58.538 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-n87tWTGKjCaWvOO off=6969230 len=362305 2021-07-31T02:06:58.541 INFO:tasks.workunit.client.0.smithi045.stdout:op 540 completed, throughput=4.98MB/sec 2021-07-31T02:06:58.542 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-j5suao-vFSnzFcg off=5913765 len=1600464 2021-07-31T02:06:58.554 INFO:tasks.workunit.client.0.smithi045.stdout:op 541 completed, throughput=4.99MB/sec 2021-07-31T02:06:58.554 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-HSJljMvE4T48XdB off=11036992 len=918840 2021-07-31T02:06:58.559 INFO:tasks.workunit.client.0.smithi045.stdout:op 542 completed, throughput=5MB/sec 2021-07-31T02:06:59.559 INFO:tasks.workunit.client.0.smithi045.stdout: 111: throughput=4.96MB/sec pending data=0 2021-07-31T02:06:59.559 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-g-nGzHrLkliEqEZ off=23461037 len=112693 2021-07-31T02:06:59.562 INFO:tasks.workunit.client.0.smithi045.stdout:op 543 completed, throughput=4.96MB/sec 2021-07-31T02:06:59.562 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-CILWH9Bg9kcTDEm off=17559518 len=1349490 2021-07-31T02:06:59.564 INFO:tasks.workunit.client.0.smithi045.stdout:op 544 completed, throughput=4.97MB/sec 2021-07-31T02:06:59.564 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-s7WAPqHnI_Xn2kf off=190755 len=1454070 2021-07-31T02:06:59.572 INFO:tasks.workunit.client.0.smithi045.stdout:op 545 completed, throughput=4.98MB/sec 2021-07-31T02:06:59.573 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-aWdun6YqXpRTSkN off=1688697 len=624447 2021-07-31T02:06:59.573 INFO:tasks.workunit.client.0.smithi045.stdout:op 546 completed, throughput=4.99MB/sec 2021-07-31T02:06:59.573 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-QdX9B-Lt-UDsmKS off=1891424 len=1383955 2021-07-31T02:06:59.579 INFO:tasks.workunit.client.0.smithi045.stdout:op 547 completed, throughput=5MB/sec 2021-07-31T02:06:59.579 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-_2b8hRVKvmEdMih off=8819649 len=973754 2021-07-31T02:06:59.580 INFO:tasks.workunit.client.0.smithi045.stdout:op 548 completed, throughput=5MB/sec 2021-07-31T02:07:00.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:00 smithi154 conmon[34553]: cluster 2021-07-31T02:07:00.071293+0000 mgr.x (mgr.24542) 846 : cluster [DBG] pgmap v1465: 9 pgs: 9 active+clean; 127 GiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 3.5 MiB/s rd, 2.9 MiB/s wr, 6 op/s 2021-07-31T02:07:00.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:00 smithi045 conmon[33655]: cluster 2021-07-31T02:07:00.071293+0000 mgr.x (mgr.24542) 846 : cluster [DBG] pgmap v1465: 9 pgs: 9 active+clean; 127 GiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 3.5 MiB/s rd, 2.9 MiB/s wr, 6 op/s 2021-07-31T02:07:00.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:00 smithi045 conmon[39718]: cluster 2021-07-31T02:07:00.071293+0000 mgr.x (mgr.24542) 846 : cluster [DBG] pgmap v1465: 9 pgs: 9 active+clean; 127 GiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 3.5 MiB/s rd, 2.9 MiB/s wr, 6 op/s 2021-07-31T02:07:00.582 INFO:tasks.workunit.client.0.smithi045.stdout: 112: throughput=4.96MB/sec pending data=0 2021-07-31T02:07:00.582 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-EymFuTsHg4ungLR off=4228148 len=1216442 2021-07-31T02:07:00.583 INFO:tasks.workunit.client.0.smithi045.stdout:op 549 completed, throughput=4.97MB/sec 2021-07-31T02:07:00.584 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-vbrC7OSF56KiFGA off=579552 len=195557 2021-07-31T02:07:00.585 INFO:tasks.workunit.client.0.smithi045.stdout:op 550 completed, throughput=4.97MB/sec 2021-07-31T02:07:00.585 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-TEczTXzaP3qGr8B off=21792852 len=1204122 2021-07-31T02:07:00.586 INFO:tasks.workunit.client.0.smithi045.stdout:op 551 completed, throughput=4.98MB/sec 2021-07-31T02:07:00.587 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-EI3j92un-CanWcn off=10012837 len=405520 2021-07-31T02:07:00.587 INFO:tasks.workunit.client.0.smithi045.stdout:op 552 completed, throughput=4.99MB/sec 2021-07-31T02:07:00.588 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Pa_H_d1Jho_-Qdw off=13376 len=26604 2021-07-31T02:07:00.588 INFO:tasks.workunit.client.0.smithi045.stdout:op 553 completed, throughput=4.99MB/sec 2021-07-31T02:07:00.589 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-4aS0T0PPktSY7Tr off=10561821 len=1966 2021-07-31T02:07:00.589 INFO:tasks.workunit.client.0.smithi045.stdout:op 554 completed, throughput=4.99MB/sec 2021-07-31T02:07:00.589 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ZKd3tdyyCM8qsJE off=1151901 len=743476 2021-07-31T02:07:00.594 INFO:tasks.workunit.client.0.smithi045.stdout:op 555 completed, throughput=4.99MB/sec 2021-07-31T02:07:00.594 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-K7WARNxKnwbqbHX off=18720969 len=597686 2021-07-31T02:07:00.595 INFO:tasks.workunit.client.0.smithi045.stdout:op 556 completed, throughput=5MB/sec 2021-07-31T02:07:00.596 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-PSmfkLZtG9l1KBS off=847362 len=669363 2021-07-31T02:07:00.599 INFO:tasks.workunit.client.0.smithi045.stdout:op 557 completed, throughput=5MB/sec 2021-07-31T02:07:01.600 INFO:tasks.workunit.client.0.smithi045.stdout: 113: throughput=4.96MB/sec pending data=0 2021-07-31T02:07:01.600 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-HjKV_2NXTTTBFtP off=9638495 len=206605 2021-07-31T02:07:01.601 INFO:tasks.workunit.client.0.smithi045.stdout:op 558 completed, throughput=4.96MB/sec 2021-07-31T02:07:01.601 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-QPRncLuNjugh2yh off=3802814 len=119611 2021-07-31T02:07:01.603 INFO:tasks.workunit.client.0.smithi045.stdout:op 559 completed, throughput=4.96MB/sec 2021-07-31T02:07:01.604 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-7Ul156uEZrakqei off=10926550 len=1003317 2021-07-31T02:07:01.604 INFO:tasks.workunit.client.0.smithi045.stdout:op 560 completed, throughput=4.97MB/sec 2021-07-31T02:07:01.605 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-0wCck9QEy3oa_pA off=1112810 len=1563886 2021-07-31T02:07:01.608 INFO:tasks.workunit.client.0.smithi045.stdout:op 561 completed, throughput=4.98MB/sec 2021-07-31T02:07:01.609 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-YV4VA6HL85fYUio off=2032682 len=1799747 2021-07-31T02:07:01.615 INFO:tasks.workunit.client.0.smithi045.stdout:op 562 completed, throughput=5MB/sec 2021-07-31T02:07:01.616 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-8rWSCM_jZuuQlbc off=11550505 len=1553334 2021-07-31T02:07:01.616 INFO:tasks.workunit.client.0.smithi045.stdout:op 563 completed, throughput=5.01MB/sec 2021-07-31T02:07:01.724 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:07:02.001 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:07:02.001 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 21m ago 22m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:07:02.001 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 19m ago 22m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:07:02.002 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 19m ago 25m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:07:02.002 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 21m ago 28m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:07:02.002 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 21m ago 28m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:07:02.002 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 19m ago 26m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:07:02.003 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 21m ago 26m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:07:02.003 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 21m ago 22m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:07:02.004 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 19m ago 22m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:07:02.004 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 21m ago 25m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:07:02.004 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 21m ago 25m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:07:02.004 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 21m ago 24m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:07:02.004 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 21m ago 24m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:07:02.005 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 19m ago 24m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:07:02.005 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 19m ago 24m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:07:02.005 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 19m ago 23m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:07:02.005 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 19m ago 23m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:07:02.006 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 19m ago 23m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:07:02.336 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:07:02.336 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:07:02.336 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:07:02.336 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:07:02.337 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:07:02.337 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:07:02.337 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:07:02.337 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:07:02.338 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:07:02.338 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:07:02.338 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:07:02.338 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:07:02.339 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:07:02.339 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:07:02.339 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:07:02.339 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:07:02.339 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:07:02.615 INFO:tasks.workunit.client.0.smithi045.stdout: 114: throughput=4.97MB/sec pending data=0 2021-07-31T02:07:02.616 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-E4m8OX9fuWb5E5O off=21514683 len=1936323 2021-07-31T02:07:02.626 INFO:tasks.workunit.client.0.smithi045.stdout:op 564 completed, throughput=4.98MB/sec 2021-07-31T02:07:02.627 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-OUyTc-L21xcNMip off=15073054 len=1978246 2021-07-31T02:07:02.637 INFO:tasks.workunit.client.0.smithi045.stdout:op 565 completed, throughput=5MB/sec 2021-07-31T02:07:02.637 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ZwIwLC798Qbcjd7 off=2036123 len=30623 2021-07-31T02:07:02.639 INFO:tasks.workunit.client.0.smithi045.stdout:op 566 completed, throughput=5MB/sec 2021-07-31T02:07:02.639 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-NzsP6RcATzGNEIz off=13218127 len=458659 2021-07-31T02:07:02.641 INFO:tasks.workunit.client.0.smithi045.stdout:op 567 completed, throughput=5MB/sec 2021-07-31T02:07:03.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:02 smithi154 conmon[34553]: audit 2021-07-31T02:07:01.712352+0000 mgr.x (mgr.24542) 847 : audit [DBG] from='client.15789 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:07:03.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:02 smithi154 conmon[34553]: 2021-07-31T02:07:03.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:02 smithi154 conmon[34553]: audit 2021-07-31T02:07:01.992377+0000 mgr.x (mgr.24542) 848 : audit [DBG] from='client.25702 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:07:03.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:02 smithi154 conmon[34553]: cluster 2021-07-31T02:07:02.071664+0000 mgr.x 2021-07-31T02:07:03.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:02 smithi154 conmon[34553]: (mgr.24542) 849 : cluster [DBG] pgmap v1466: 9 pgs: 9 active+clean; 127 GiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 2.4 MiB/s rd, 1.7 MiB/s wr, 4 op/s 2021-07-31T02:07:03.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:02 smithi154 conmon[34553]: audit 2021-07-31T02:07:03.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:02 smithi154 conmon[34553]: 2021-07-31T02:07:02.335639+0000 mon.a (mon.0) 1855 : audit [DBG] from='client.? 172.21.15.45:0/1236022246' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:07:03.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:02 smithi045 conmon[33655]: audit 2021-07-31T02:07:01.712352+0000 mgr.x (mgr.24542 2021-07-31T02:07:03.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:02 smithi045 conmon[33655]: ) 847 : audit [DBG] from='client.15789 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:07:03.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:02 smithi045 conmon[33655]: audit 2021-07-31T02:07:01.992377+0000 mgr.x (mgr.24542) 848 : audit [DBG] from='client.25702 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:07:03.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:02 smithi045 conmon[33655]: cluster 2021-07-31T02:07:02 2021-07-31T02:07:03.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:02 smithi045 conmon[33655]: .071664+0000 mgr.x (mgr.24542 2021-07-31T02:07:03.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:02 smithi045 conmon[33655]: ) 849 : cluster [DBG] pgmap v1466: 9 pgs: 9 active+clean; 127 GiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 2.4 MiB/s rd, 1.7 MiB/s wr, 4 op/s 2021-07-31T02:07:03.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:02 smithi045 conmon[33655]: audit 2021 2021-07-31T02:07:03.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:02 smithi045 conmon[33655]: -07-31T02:07:02.335639 2021-07-31T02:07:03.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:02 smithi045 conmon[33655]: +0000 mon.a (mon.0) 1855 2021-07-31T02:07:03.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:02 smithi045 conmon[33655]: : audit [DBG] from='client.? 172.21.15.45:0/1236022246' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:07:03.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:02 smithi045 conmon[39718]: audit 2021-07-31T02:07:01.712352+0000 mgr.x (mgr.24542) 2021-07-31T02:07:03.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:02 smithi045 conmon[39718]: 847 : audit [DBG] from='client.15789 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:07:03.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:02 smithi045 conmon[39718]: audit 2021-07-31T02:07:01.992377+0000 mgr.x ( 2021-07-31T02:07:03.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:02 smithi045 conmon[39718]: mgr.24542) 848 : audit [DBG] from='client.25702 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:07:03.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:02 smithi045 conmon[39718]: cluster 2021-07-31T02:07: 2021-07-31T02:07:03.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:02 smithi045 conmon[39718]: 02.071664+0000 mgr.x (mgr.24542) 849 : cluster 2021-07-31T02:07:03.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:02 smithi045 conmon[39718]: [DBG] pgmap v1466: 9 pgs: 9 active+clean; 127 GiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 2.4 MiB/s rd, 1.7 MiB/s wr, 4 op/s 2021-07-31T02:07:03.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:02 smithi045 conmon[39718]: audit 2021 2021-07-31T02:07:03.282 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:02 smithi045 conmon[39718]: -07-31T02:07:02.335639+0000 mon.a ( 2021-07-31T02:07:03.282 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:02 smithi045 conmon[39718]: mon.0) 1855 : audit [DBG] from='client.? 172.21.15.45:0/1236022246' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:07:03.642 INFO:tasks.workunit.client.0.smithi045.stdout: 115: throughput=4.96MB/sec pending data=0 2021-07-31T02:07:03.642 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-YwmHSw3m9h7QGBX off=10808531 len=660420 2021-07-31T02:07:03.644 INFO:tasks.workunit.client.0.smithi045.stdout:op 568 completed, throughput=4.96MB/sec 2021-07-31T02:07:03.644 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-LrcrG6zLh-5eMvV off=19946557 len=337399 2021-07-31T02:07:03.645 INFO:tasks.workunit.client.0.smithi045.stdout:op 569 completed, throughput=4.97MB/sec 2021-07-31T02:07:03.645 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-C6J1yZYcZe2N6N3 off=2296735 len=811507 2021-07-31T02:07:03.647 INFO:tasks.workunit.client.0.smithi045.stdout:op 570 completed, throughput=4.97MB/sec 2021-07-31T02:07:03.648 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-6zSkLFVRMB5jbfu off=1437647 len=1350155 2021-07-31T02:07:03.653 INFO:tasks.workunit.client.0.smithi045.stdout:op 571 completed, throughput=4.98MB/sec 2021-07-31T02:07:03.654 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-noHmvD6qT9p4_l1 off=2645484 len=889588 2021-07-31T02:07:03.655 INFO:tasks.workunit.client.0.smithi045.stdout:op 572 completed, throughput=4.99MB/sec 2021-07-31T02:07:03.655 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Z3pr8wqpwYKsUyW off=17806492 len=21332 2021-07-31T02:07:03.657 INFO:tasks.workunit.client.0.smithi045.stdout:op 573 completed, throughput=4.99MB/sec 2021-07-31T02:07:03.657 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-YBfgOxTi49HnStM off=14236151 len=1040552 2021-07-31T02:07:03.663 INFO:tasks.workunit.client.0.smithi045.stdout:op 574 completed, throughput=5MB/sec 2021-07-31T02:07:03.668 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-z1V_mMMNAdWQ-sA off=8056870 len=488220 2021-07-31T02:07:03.669 INFO:tasks.workunit.client.0.smithi045.stdout:op 575 completed, throughput=5MB/sec 2021-07-31T02:07:04.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:07:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:07:03.541Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:07:04.027 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:07:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:07:03.541Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:07:04.027 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:07:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:07:03.541Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:07:04.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:04 smithi154 conmon[34553]: cluster 2021-07-31T02:07:04.072098+0000 mgr.x (mgr.24542) 850 : cluster [DBG] pgmap v1467: 9 pgs: 9 active+clean; 127 GiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 3.5 MiB/s rd, 2.8 MiB/s wr, 6 op/s 2021-07-31T02:07:04.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:04 smithi154 conmon[34553]: 2021-07-31T02:07:04.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:04 smithi045 conmon[33655]: cluster 2021-07-31T02:07:04.072098+0000 mgr.x (mgr.24542) 850 2021-07-31T02:07:04.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:04 smithi045 conmon[33655]: : cluster [DBG] pgmap v1467: 9 pgs: 9 active+clean; 127 GiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 3.5 MiB/s rd, 2.8 MiB/s wr, 6 op/s 2021-07-31T02:07:04.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:04 smithi045 conmon[39718]: cluster 2021-07-31T02:07:04.072098 2021-07-31T02:07:04.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:04 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 850 : cluster [DBG] pgmap v1467: 9 pgs: 9 active+clean; 127 GiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 3.5 MiB/s rd, 2.8 MiB/s wr, 6 op/s 2021-07-31T02:07:04.664 INFO:tasks.workunit.client.0.smithi045.stdout: 116: throughput=4.96MB/sec pending data=0 2021-07-31T02:07:04.664 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-IzoDw2jUVWrThqi off=17900601 len=1823534 2021-07-31T02:07:04.667 INFO:tasks.workunit.client.0.smithi045.stdout:op 576 completed, throughput=4.98MB/sec 2021-07-31T02:07:04.668 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-TJZYYBdt-q7KrNi off=10477318 len=1617997 2021-07-31T02:07:04.675 INFO:tasks.workunit.client.0.smithi045.stdout:op 577 completed, throughput=4.99MB/sec 2021-07-31T02:07:04.676 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-i1S_NMVu419NBN2 off=8220692 len=1635306 2021-07-31T02:07:04.677 INFO:tasks.workunit.client.0.smithi045.stdout:op 578 completed, throughput=5MB/sec 2021-07-31T02:07:05.682 INFO:tasks.workunit.client.0.smithi045.stdout: 117: throughput=4.96MB/sec pending data=0 2021-07-31T02:07:05.683 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-DI7KXt8W0DQtK0m off=1933839 len=1360259 2021-07-31T02:07:05.683 INFO:tasks.workunit.client.0.smithi045.stdout:op 579 completed, throughput=4.97MB/sec 2021-07-31T02:07:05.683 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ifGna9o8QOzz81B off=4787417 len=1421036 2021-07-31T02:07:05.687 INFO:tasks.workunit.client.0.smithi045.stdout:op 580 completed, throughput=4.98MB/sec 2021-07-31T02:07:05.688 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-nIPeBt0OOz8Xcg_ off=8829250 len=204354 2021-07-31T02:07:05.688 INFO:tasks.workunit.client.0.smithi045.stdout:op 581 completed, throughput=4.98MB/sec 2021-07-31T02:07:05.689 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ol1hYC5wp5lNUu- off=9364440 len=18084 2021-07-31T02:07:05.689 INFO:tasks.workunit.client.0.smithi045.stdout:op 582 completed, throughput=4.98MB/sec 2021-07-31T02:07:05.690 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-6dirAJd3C4BTXaB off=831119 len=1275998 2021-07-31T02:07:05.695 INFO:tasks.workunit.client.0.smithi045.stdout:op 583 completed, throughput=4.99MB/sec 2021-07-31T02:07:05.695 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-o9srov6v_B_7iZw off=558660 len=1196654 2021-07-31T02:07:05.697 INFO:tasks.workunit.client.0.smithi045.stdout:op 584 completed, throughput=5MB/sec 2021-07-31T02:07:06.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:06 smithi154 conmon[34553]: cluster 2021-07-31T02:07:06.072395+0000 mgr.x (mgr.24542) 851 : cluster [DBG] pgmap v1468: 9 pgs: 9 active+clean; 127 GiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 2.3 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2021-07-31T02:07:06.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:06 smithi154 conmon[34553]: 2021-07-31T02:07:06.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:06 smithi045 conmon[33655]: cluster 2021-07-31T02:07:06.072395+0000 mgr.x (mgr.24542) 851 : cluster [DBG] 2021-07-31T02:07:06.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:06 smithi045 conmon[33655]: pgmap v1468: 9 pgs: 9 active+clean; 127 GiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 2.3 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2021-07-31T02:07:06.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:06 smithi045 conmon[39718]: cluster 2021-07-31T02:07:06.072395+0000 mgr.x (mgr. 2021-07-31T02:07:06.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:06 smithi045 conmon[39718]: 24542) 851 : cluster [DBG] pgmap v1468: 9 pgs: 9 active+clean; 127 GiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 2.3 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2021-07-31T02:07:06.697 INFO:tasks.workunit.client.0.smithi045.stdout: 118: throughput=4.96MB/sec pending data=0 2021-07-31T02:07:06.698 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-hCZ0mjQLo8y-Cba off=927421 len=145656 2021-07-31T02:07:06.699 INFO:tasks.workunit.client.0.smithi045.stdout:op 585 completed, throughput=4.96MB/sec 2021-07-31T02:07:06.699 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-kkrJOcDmDlp1yWH off=6132767 len=1563184 2021-07-31T02:07:06.707 INFO:tasks.workunit.client.0.smithi045.stdout:op 586 completed, throughput=4.97MB/sec 2021-07-31T02:07:06.707 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-7Ds8yBLHxYbsnYL off=11993206 len=2014139 2021-07-31T02:07:06.710 INFO:tasks.workunit.client.0.smithi045.stdout:op 587 completed, throughput=4.99MB/sec 2021-07-31T02:07:06.711 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ipOFAOeLdN4OWNy off=9115407 len=979617 2021-07-31T02:07:06.712 INFO:tasks.workunit.client.0.smithi045.stdout:op 588 completed, throughput=5MB/sec 2021-07-31T02:07:06.713 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-GZRcyGO-RyUil0x off=14968263 len=354126 2021-07-31T02:07:06.716 INFO:tasks.workunit.client.0.smithi045.stdout:op 589 completed, throughput=5MB/sec 2021-07-31T02:07:07.717 INFO:tasks.workunit.client.0.smithi045.stdout: 119: throughput=4.96MB/sec pending data=0 2021-07-31T02:07:07.717 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-iAyJ1ESL4Bc32Xo off=18816012 len=1859954 2021-07-31T02:07:07.721 INFO:tasks.workunit.client.0.smithi045.stdout:op 590 completed, throughput=4.97MB/sec 2021-07-31T02:07:07.721 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-N5I4S3bIb5-UFzZ off=1850257 len=887362 2021-07-31T02:07:07.723 INFO:tasks.workunit.client.0.smithi045.stdout:op 591 completed, throughput=4.98MB/sec 2021-07-31T02:07:07.723 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-BTGksIBn3jqP3zX off=12717564 len=1650082 2021-07-31T02:07:07.731 INFO:tasks.workunit.client.0.smithi045.stdout:op 592 completed, throughput=4.99MB/sec 2021-07-31T02:07:07.731 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ueaq5YRAizwpQii off=4420290 len=1544066 2021-07-31T02:07:07.734 INFO:tasks.workunit.client.0.smithi045.stdout:op 593 completed, throughput=5.01MB/sec 2021-07-31T02:07:08.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:08 smithi154 conmon[34553]: cluster 2021-07-31T02:07:08.072803+0000 mgr.x (mgr.24542) 852 : cluster 2021-07-31T02:07:08.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:08 smithi154 conmon[34553]: [DBG] pgmap v1469: 9 pgs: 9 active+clean; 127 GiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 2.3 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2021-07-31T02:07:08.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:08 smithi045 conmon[39718]: cluster 2021-07-31T02:07:08.072803+0000 mgr.x (mgr 2021-07-31T02:07:08.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:08 smithi045 conmon[39718]: .24542) 852 : cluster [DBG] pgmap v1469: 9 pgs: 9 active+clean; 127 GiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 2.3 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2021-07-31T02:07:08.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:08 smithi045 conmon[33655]: cluster 2021-07-31T02:07:08.072803+0000 mgr.x (mgr.24542 2021-07-31T02:07:08.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:08 smithi045 conmon[33655]: ) 852 : cluster [DBG] pgmap v1469: 9 pgs: 9 active+clean; 127 GiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 2.3 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2021-07-31T02:07:08.734 INFO:tasks.workunit.client.0.smithi045.stdout: 120: throughput=4.96MB/sec pending data=0 2021-07-31T02:07:08.734 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-T1fCcz8lWmOOZf0 off=0 len=1571539 2021-07-31T02:07:08.736 INFO:tasks.workunit.client.0.smithi045.stdout:op 594 completed, throughput=4.98MB/sec 2021-07-31T02:07:08.736 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-q6jla3xzo4U0AXa off=3764104 len=1902427 2021-07-31T02:07:08.740 INFO:tasks.workunit.client.0.smithi045.stdout:op 595 completed, throughput=4.99MB/sec 2021-07-31T02:07:08.741 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-4kO2WD9Tih3T-oG off=5244821 len=382036 2021-07-31T02:07:08.744 INFO:tasks.workunit.client.0.smithi045.stdout:op 596 completed, throughput=4.99MB/sec 2021-07-31T02:07:08.744 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-WYFtiXzPj4297px off=13704419 len=253158 2021-07-31T02:07:08.745 INFO:tasks.workunit.client.0.smithi045.stdout:op 597 completed, throughput=5MB/sec 2021-07-31T02:07:08.745 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-GB2UbU2-Dsqzqub off=9615041 len=563709 2021-07-31T02:07:08.749 INFO:tasks.workunit.client.0.smithi045.stdout:op 598 completed, throughput=5MB/sec 2021-07-31T02:07:09.750 INFO:tasks.workunit.client.0.smithi045.stdout: 121: throughput=4.96MB/sec pending data=0 2021-07-31T02:07:09.750 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-CFI3l7JXS-ytQc3 off=3010508 len=401955 2021-07-31T02:07:09.753 INFO:tasks.workunit.client.0.smithi045.stdout:op 599 completed, throughput=4.96MB/sec 2021-07-31T02:07:09.753 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-qrqIwpre-rMS7b8 off=20194570 len=1685141 2021-07-31T02:07:09.762 INFO:tasks.workunit.client.0.smithi045.stdout:op 600 completed, throughput=4.98MB/sec 2021-07-31T02:07:09.762 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-91nyivwrxr7iLN2 off=15008764 len=609808 2021-07-31T02:07:09.763 INFO:tasks.workunit.client.0.smithi045.stdout:op 601 completed, throughput=4.98MB/sec 2021-07-31T02:07:09.763 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-9qW8ptxooT-IVlh off=11768891 len=1466032 2021-07-31T02:07:09.764 INFO:tasks.workunit.client.0.smithi045.stdout:op 602 completed, throughput=4.99MB/sec 2021-07-31T02:07:09.764 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-yfrQPuUJF_ymf2B off=13966103 len=1353318 2021-07-31T02:07:09.766 INFO:tasks.workunit.client.0.smithi045.stdout:op 603 completed, throughput=5MB/sec 2021-07-31T02:07:10.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:10 smithi154 conmon[34553]: cluster 2021-07-31T02:07:10.073281+0000 mgr.x (mgr.24542) 853 : cluster [DBG] pgmap v1470: 9 pgs: 9 active+clean; 127 GiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 3.7 MiB/s rd, 2.6 MiB/s wr, 6 op/s 2021-07-31T02:07:10.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:10 smithi045 conmon[39718]: cluster 2021-07-31T02:07:10.073281+0000 mgr.x ( 2021-07-31T02:07:10.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:10 smithi045 conmon[39718]: mgr.24542) 853 : cluster [DBG] pgmap v1470: 9 pgs: 9 active+clean; 127 GiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 3.7 MiB/s rd, 2.6 MiB/s wr, 6 op/s 2021-07-31T02:07:10.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:10 smithi045 conmon[33655]: cluster 2021-07-31T02:07:10.073281+0000 mgr.x (mgr.24542) 853 : 2021-07-31T02:07:10.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:10 smithi045 conmon[33655]: cluster [DBG] pgmap v1470: 9 pgs: 9 active+clean; 127 GiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 3.7 MiB/s rd, 2.6 MiB/s wr, 6 op/s 2021-07-31T02:07:10.766 INFO:tasks.workunit.client.0.smithi045.stdout: 122: throughput=4.96MB/sec pending data=0 2021-07-31T02:07:10.767 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-U9q8RMt2zUnZi2n off=3068415 len=1920658 2021-07-31T02:07:10.770 INFO:tasks.workunit.client.0.smithi045.stdout:op 604 completed, throughput=4.98MB/sec 2021-07-31T02:07:10.770 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-o8PxBdaErLwOAKJ off=15770633 len=1523660 2021-07-31T02:07:10.778 INFO:tasks.workunit.client.0.smithi045.stdout:op 605 completed, throughput=4.99MB/sec 2021-07-31T02:07:10.778 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Q5FaNt2_ZgXaySp off=9293035 len=2085016 2021-07-31T02:07:10.786 INFO:tasks.workunit.client.0.smithi045.stdout:op 606 completed, throughput=5MB/sec 2021-07-31T02:07:11.786 INFO:tasks.workunit.client.0.smithi045.stdout: 123: throughput=4.96MB/sec pending data=0 2021-07-31T02:07:11.787 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-zfOujXhbxYQhuWT off=1745437 len=468331 2021-07-31T02:07:11.791 INFO:tasks.workunit.client.0.smithi045.stdout:op 607 completed, throughput=4.97MB/sec 2021-07-31T02:07:11.791 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-O7LCa_yxjrJZun9 off=24452617 len=307975 2021-07-31T02:07:11.793 INFO:tasks.workunit.client.0.smithi045.stdout:op 608 completed, throughput=4.97MB/sec 2021-07-31T02:07:11.793 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-RpWoy3fuYe3J2Rv off=6572555 len=537370 2021-07-31T02:07:11.796 INFO:tasks.workunit.client.0.smithi045.stdout:op 609 completed, throughput=4.97MB/sec 2021-07-31T02:07:11.796 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Aq5lK39JJi32dcY off=6945570 len=1516494 2021-07-31T02:07:11.804 INFO:tasks.workunit.client.0.smithi045.stdout:op 610 completed, throughput=4.98MB/sec 2021-07-31T02:07:11.804 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-GWhuUAd90lYEJDm off=5494865 len=1218290 2021-07-31T02:07:11.811 INFO:tasks.workunit.client.0.smithi045.stdout:op 611 completed, throughput=4.99MB/sec 2021-07-31T02:07:11.811 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-sqkv0wPZ09jrgBq off=6205297 len=1200595 2021-07-31T02:07:11.816 INFO:tasks.workunit.client.0.smithi045.stdout:op 612 completed, throughput=5MB/sec 2021-07-31T02:07:12.816 INFO:tasks.workunit.client.0.smithi045.stdout: 124: throughput=4.96MB/sec pending data=0 2021-07-31T02:07:12.816 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-z6V_Li1d0Zfvh-e off=4039208 len=2083224 2021-07-31T02:07:12.821 INFO:tasks.workunit.client.0.smithi045.stdout:op 613 completed, throughput=4.98MB/sec 2021-07-31T02:07:12.821 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-EP5KeMJYYluwXho off=14436153 len=1828500 2021-07-31T02:07:12.822 INFO:tasks.workunit.client.0.smithi045.stdout:op 614 completed, throughput=4.99MB/sec 2021-07-31T02:07:12.822 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ypZyEcn38QcBmoi off=12417605 len=1693103 2021-07-31T02:07:12.825 INFO:tasks.workunit.client.0.smithi045.stdout:op 615 completed, throughput=5MB/sec 2021-07-31T02:07:13.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:12 smithi154 conmon[34553]: cluster 2021-07-31T02:07:12.073826+0000 mgr.x (mgr.24542) 854 : cluster [DBG] pgmap v1471: 9 pgs: 9 active+clean; 127 GiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.7 MiB/s wr, 4 op/s 2021-07-31T02:07:13.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:12 smithi045 conmon[39718]: cluster 2021-07-31T02:07:12.073826+0000 mgr.x ( 2021-07-31T02:07:13.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:12 smithi045 conmon[39718]: mgr.24542) 854 : cluster [DBG] pgmap v1471: 9 pgs: 9 active+clean; 127 GiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.7 MiB/s wr, 4 op/s 2021-07-31T02:07:13.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:12 smithi045 conmon[33655]: cluster 2021-07-31T02:07:12.073826+0000 mgr.x (mgr.24542) 854 : cluster [DBG] pgmap v1471: 9 pgs: 9 active+clean; 127 GiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.7 MiB/s wr, 4 op/s 2021-07-31T02:07:13.826 INFO:tasks.workunit.client.0.smithi045.stdout: 125: throughput=4.97MB/sec pending data=0 2021-07-31T02:07:13.826 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-E_7XHoLTNo06ytC off=21963962 len=1749785 2021-07-31T02:07:13.839 INFO:tasks.workunit.client.0.smithi045.stdout:op 616 completed, throughput=4.98MB/sec 2021-07-31T02:07:13.841 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-eqc0mSn7783V1em off=2362172 len=115839 2021-07-31T02:07:13.841 INFO:tasks.workunit.client.0.smithi045.stdout:op 617 completed, throughput=4.98MB/sec 2021-07-31T02:07:13.841 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-g2VBeL2XMEfk_ZP off=82005 len=1579225 2021-07-31T02:07:13.842 INFO:tasks.workunit.client.0.smithi045.stdout:op 618 completed, throughput=4.99MB/sec 2021-07-31T02:07:13.842 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj--9aC6VGRFCmhkk6 off=471556 len=1506733 2021-07-31T02:07:13.842 INFO:tasks.workunit.client.0.smithi045.stdout:op 619 completed, throughput=5MB/sec 2021-07-31T02:07:14.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:07:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:07:13.541Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:07:14.027 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:07:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:07:13.541Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:07:14.027 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:07:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:07:13.541Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:07:14.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:14 smithi154 conmon[34553]: cluster 2021-07-31T02:07:14.074214+0000 mgr.x (mgr.24542) 855 : cluster [DBG] pgmap v1472: 9 pgs: 9 active+clean; 127 GiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 3.5 MiB/s rd, 2.8 MiB/s wr, 6 op/s 2021-07-31T02:07:14.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:14 smithi045 conmon[39718]: cluster 2021-07-31T02:07:14.074214+0000 mgr.x (mgr.24542) 855 : cluster [DBG] pgmap v1472: 9 pgs: 9 active+clean; 127 GiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 3.5 MiB/s rd, 2.8 MiB/s wr, 6 op/s 2021-07-31T02:07:14.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:14 smithi045 conmon[33655]: cluster 2021 2021-07-31T02:07:14.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:14 smithi045 conmon[33655]: -07-31T02:07:14.074214+0000 mgr.x (mgr.24542) 855 : cluster [DBG] pgmap v1472: 9 pgs: 9 active+clean; 127 GiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 3.5 MiB/s rd, 2.8 MiB/s wr, 6 op/s 2021-07-31T02:07:14.841 INFO:tasks.workunit.client.0.smithi045.stdout: 126: throughput=4.96MB/sec pending data=0 2021-07-31T02:07:14.842 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-hkEZblRbsCtH9eG off=1221697 len=78768 2021-07-31T02:07:14.844 INFO:tasks.workunit.client.0.smithi045.stdout:op 620 completed, throughput=4.96MB/sec 2021-07-31T02:07:14.844 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-gZ517vR5L5B0mLC off=932441 len=1855987 2021-07-31T02:07:14.853 INFO:tasks.workunit.client.0.smithi045.stdout:op 621 completed, throughput=4.98MB/sec 2021-07-31T02:07:14.853 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-nq1zRYVjltyNUqo off=16316924 len=1548500 2021-07-31T02:07:14.856 INFO:tasks.workunit.client.0.smithi045.stdout:op 622 completed, throughput=4.99MB/sec 2021-07-31T02:07:14.856 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-YKAYjkVOFeeahxb off=2273545 len=1224634 2021-07-31T02:07:14.858 INFO:tasks.workunit.client.0.smithi045.stdout:op 623 completed, throughput=5MB/sec 2021-07-31T02:07:14.859 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ZW0JXhKbMp8Ykfd off=11748404 len=1233192 2021-07-31T02:07:14.865 INFO:tasks.workunit.client.0.smithi045.stdout:op 624 completed, throughput=5.01MB/sec 2021-07-31T02:07:15.865 INFO:tasks.workunit.client.0.smithi045.stdout: 127: throughput=4.97MB/sec pending data=0 2021-07-31T02:07:15.866 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-HvNm_xb2CjbfbjF off=6363147 len=2059534 2021-07-31T02:07:15.877 INFO:tasks.workunit.client.0.smithi045.stdout:op 625 completed, throughput=4.98MB/sec 2021-07-31T02:07:15.877 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-VXwlXWJY7XybnL4 off=1622100 len=89310 2021-07-31T02:07:15.878 INFO:tasks.workunit.client.0.smithi045.stdout:op 626 completed, throughput=4.98MB/sec 2021-07-31T02:07:15.878 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Tohq6IQ19DrUmb1 off=15336860 len=532665 2021-07-31T02:07:15.878 INFO:tasks.workunit.client.0.smithi045.stdout:op 627 completed, throughput=4.99MB/sec 2021-07-31T02:07:15.879 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-vI_cpA05tFbc_fX off=2879395 len=491814 2021-07-31T02:07:15.879 INFO:tasks.workunit.client.0.smithi045.stdout:op 628 completed, throughput=4.99MB/sec 2021-07-31T02:07:15.880 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-UIm4_exNHAy5Pi2 off=7786342 len=1975466 2021-07-31T02:07:15.884 INFO:tasks.workunit.client.0.smithi045.stdout:op 629 completed, throughput=5MB/sec 2021-07-31T02:07:16.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:16 smithi154 conmon[34553]: cluster 2021-07-31T02:07:16.074619+0000 mgr.x (mgr.24542) 856 : cluster [DBG] pgmap v1473: 9 pgs: 9 active+clean; 127 GiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 2.4 MiB/s rd, 1.8 MiB/s wr, 3 op/s 2021-07-31T02:07:16.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:16 smithi154 conmon[34553]: 2021-07-31T02:07:16.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:16 smithi045 conmon[39718]: cluster 2021-07-31T02:07:16.074619+0000 mgr.x (mgr 2021-07-31T02:07:16.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:16 smithi045 conmon[39718]: .24542) 856 : cluster [DBG] pgmap v1473: 9 pgs: 9 active+clean; 127 GiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 2.4 MiB/s rd, 1.8 MiB/s wr, 3 op/s 2021-07-31T02:07:16.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:16 smithi045 conmon[33655]: cluster 2021-07-31T02:07:16.074619+0000 mgr.x (mgr.24542) 2021-07-31T02:07:16.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:16 smithi045 conmon[33655]: 856 : cluster [DBG] pgmap v1473: 9 pgs: 9 active+clean; 127 GiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 2.4 MiB/s rd, 1.8 MiB/s wr, 3 op/s 2021-07-31T02:07:16.884 INFO:tasks.workunit.client.0.smithi045.stdout: 128: throughput=4.97MB/sec pending data=0 2021-07-31T02:07:16.884 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-5GF_ku87erd24kX off=6561017 len=15207 2021-07-31T02:07:16.885 INFO:tasks.workunit.client.0.smithi045.stdout:op 630 completed, throughput=4.97MB/sec 2021-07-31T02:07:16.886 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ZjOfjxUICp9XJ7z off=2338387 len=640734 2021-07-31T02:07:16.889 INFO:tasks.workunit.client.0.smithi045.stdout:op 631 completed, throughput=4.97MB/sec 2021-07-31T02:07:16.890 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-KRfHIqOYSbLQw4_ off=15047051 len=224863 2021-07-31T02:07:16.893 INFO:tasks.workunit.client.0.smithi045.stdout:op 632 completed, throughput=4.97MB/sec 2021-07-31T02:07:16.894 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-uwmXcdm_cQHicdS off=821149 len=1405324 2021-07-31T02:07:16.896 INFO:tasks.workunit.client.0.smithi045.stdout:op 633 completed, throughput=4.98MB/sec 2021-07-31T02:07:16.897 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-UbVQ7X6411seNY8 off=11725329 len=944685 2021-07-31T02:07:16.898 INFO:tasks.workunit.client.0.smithi045.stdout:op 634 completed, throughput=4.99MB/sec 2021-07-31T02:07:16.898 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-h4uXx4-HzSwnU4f off=17617471 len=629763 2021-07-31T02:07:16.904 INFO:tasks.workunit.client.0.smithi045.stdout:op 635 completed, throughput=4.99MB/sec 2021-07-31T02:07:16.904 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-CGrOe9vYRnczhdB off=1406019 len=815385 2021-07-31T02:07:16.908 INFO:tasks.workunit.client.0.smithi045.stdout:op 636 completed, throughput=5MB/sec 2021-07-31T02:07:16.909 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-L5VDmMU_b5XZuOY off=2638657 len=965239 2021-07-31T02:07:16.910 INFO:tasks.workunit.client.0.smithi045.stdout:op 637 completed, throughput=5.01MB/sec 2021-07-31T02:07:17.911 INFO:tasks.workunit.client.0.smithi045.stdout: 129: throughput=4.97MB/sec pending data=0 2021-07-31T02:07:17.911 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-pA2yJHmYg2Rv_H0 off=6437163 len=1302655 2021-07-31T02:07:17.918 INFO:tasks.workunit.client.0.smithi045.stdout:op 638 completed, throughput=4.98MB/sec 2021-07-31T02:07:17.918 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-gqp0PjJMuRfleHV off=15338080 len=1988120 2021-07-31T02:07:17.919 INFO:tasks.workunit.client.0.smithi045.stdout:op 639 completed, throughput=4.99MB/sec 2021-07-31T02:07:17.920 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-zfIX2OXJUiGiT5f off=2899355 len=1482043 2021-07-31T02:07:17.927 INFO:tasks.workunit.client.0.smithi045.stdout:op 640 completed, throughput=5MB/sec 2021-07-31T02:07:18.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:18 smithi154 conmon[34553]: cluster 2021-07-31T02:07:18.075075+0000 mgr.x (mgr.24542) 857 : cluster [DBG] pgmap v1474: 9 pgs: 9 active+clean; 127 GiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 2.4 MiB/s rd, 1.8 MiB/s wr, 3 op/s 2021-07-31T02:07:18.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:18 smithi154 conmon[34553]: 2021-07-31T02:07:18.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:18 smithi045 conmon[39718]: cluster 2021-07-31T02:07:18.075075+0000 mgr.x (mgr 2021-07-31T02:07:18.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:18 smithi045 conmon[39718]: .24542) 857 : cluster [DBG] pgmap v1474: 9 pgs: 9 active+clean; 127 GiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 2.4 MiB/s rd, 1.8 MiB/s wr, 3 op/s 2021-07-31T02:07:18.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:18 smithi045 conmon[33655]: cluster 2021-07-31T02:07:18.075075+0000 mgr.x (mgr.24542) 857 2021-07-31T02:07:18.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:18 smithi045 conmon[33655]: : cluster [DBG] pgmap v1474: 9 pgs: 9 active+clean; 127 GiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 2.4 MiB/s rd, 1.8 MiB/s wr, 3 op/s 2021-07-31T02:07:18.927 INFO:tasks.workunit.client.0.smithi045.stdout: 130: throughput=4.96MB/sec pending data=0 2021-07-31T02:07:18.927 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-MjUU8wXuxqhih3p off=12819802 len=1421822 2021-07-31T02:07:18.934 INFO:tasks.workunit.client.0.smithi045.stdout:op 641 completed, throughput=4.97MB/sec 2021-07-31T02:07:18.935 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-QwGTq0miJ-G5WvK off=6544416 len=1000172 2021-07-31T02:07:18.936 INFO:tasks.workunit.client.0.smithi045.stdout:op 642 completed, throughput=4.98MB/sec 2021-07-31T02:07:18.936 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-8xh2gXjSSR05GlT off=6956551 len=572457 2021-07-31T02:07:18.938 INFO:tasks.workunit.client.0.smithi045.stdout:op 643 completed, throughput=4.99MB/sec 2021-07-31T02:07:18.938 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Jxou8PjORsBIqhk off=8276725 len=562696 2021-07-31T02:07:18.939 INFO:tasks.workunit.client.0.smithi045.stdout:op 644 completed, throughput=4.99MB/sec 2021-07-31T02:07:18.940 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-jm-_N2JOW9bgKmF off=12849979 len=1708725 2021-07-31T02:07:18.947 INFO:tasks.workunit.client.0.smithi045.stdout:op 645 completed, throughput=5MB/sec 2021-07-31T02:07:19.947 INFO:tasks.workunit.client.0.smithi045.stdout: 131: throughput=4.96MB/sec pending data=0 2021-07-31T02:07:19.948 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-V-gwh4YvIQzu1sf off=16611839 len=1621759 2021-07-31T02:07:19.950 INFO:tasks.workunit.client.0.smithi045.stdout:op 646 completed, throughput=4.98MB/sec 2021-07-31T02:07:19.951 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-PYBPMGkrWmkCVDL off=6667265 len=1810097 2021-07-31T02:07:19.952 INFO:tasks.workunit.client.0.smithi045.stdout:op 647 completed, throughput=4.99MB/sec 2021-07-31T02:07:19.953 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-EuO_2rWVmquDvYA off=6890231 len=1632537 2021-07-31T02:07:19.961 INFO:tasks.workunit.client.0.smithi045.stdout:op 648 completed, throughput=5MB/sec 2021-07-31T02:07:19.961 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-VvdBgRVshWd-ntf off=1338613 len=292894 2021-07-31T02:07:19.964 INFO:tasks.workunit.client.0.smithi045.stdout:op 649 completed, throughput=5MB/sec 2021-07-31T02:07:20.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:20 smithi154 conmon[34553]: cluster 2021-07-31T02:07:20.075600+0000 mgr.x (mgr.24542) 858 : cluster [DBG] 2021-07-31T02:07:20.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:20 smithi154 conmon[34553]: pgmap v1475: 9 pgs: 9 active+clean; 127 GiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 3.3 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2021-07-31T02:07:20.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:20 smithi045 conmon[39718]: cluster 2021-07-31T02:07:20.075600 2021-07-31T02:07:20.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:20 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 858 : cluster [DBG] pgmap v1475: 9 pgs: 9 active+clean; 127 GiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 3.3 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2021-07-31T02:07:20.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:20 smithi045 conmon[33655]: cluster 2021-07-31T02:07:20.075600+0000 mgr.x (mgr.24542) 858 : cluster [DBG] pgmap v1475: 9 pgs: 9 active+clean; 127 GiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 3.3 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2021-07-31T02:07:20.964 INFO:tasks.workunit.client.0.smithi045.stdout: 132: throughput=4.96MB/sec pending data=0 2021-07-31T02:07:20.965 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-P65KBRdEgT6DZxr off=10280590 len=18689 2021-07-31T02:07:20.966 INFO:tasks.workunit.client.0.smithi045.stdout:op 650 completed, throughput=4.96MB/sec 2021-07-31T02:07:20.967 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-i6-7i3s39Fd2SuL off=14445428 len=862468 2021-07-31T02:07:20.968 INFO:tasks.workunit.client.0.smithi045.stdout:op 651 completed, throughput=4.97MB/sec 2021-07-31T02:07:20.968 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-reAwVBQlmonpf2f off=965970 len=646417 2021-07-31T02:07:20.970 INFO:tasks.workunit.client.0.smithi045.stdout:op 652 completed, throughput=4.98MB/sec 2021-07-31T02:07:20.971 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-RMj-1d5jp8kgjIG off=2952159 len=818487 2021-07-31T02:07:20.971 INFO:tasks.workunit.client.0.smithi045.stdout:op 653 completed, throughput=4.98MB/sec 2021-07-31T02:07:20.971 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Qh_82q_GZ-uNM0V off=4046178 len=1556315 2021-07-31T02:07:20.979 INFO:tasks.workunit.client.0.smithi045.stdout:op 654 completed, throughput=4.99MB/sec 2021-07-31T02:07:20.979 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-RpVG6bPwSBhDe5- off=13077639 len=507215 2021-07-31T02:07:20.980 INFO:tasks.workunit.client.0.smithi045.stdout:op 655 completed, throughput=5MB/sec 2021-07-31T02:07:20.980 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-6N_yhGlYfQ06bZs off=8719534 len=1271946 2021-07-31T02:07:20.981 INFO:tasks.workunit.client.0.smithi045.stdout:op 656 completed, throughput=5MB/sec 2021-07-31T02:07:21.982 INFO:tasks.workunit.client.0.smithi045.stdout: 133: throughput=4.97MB/sec pending data=0 2021-07-31T02:07:21.982 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-IfaembPM5a4nOcT off=8362799 len=212981 2021-07-31T02:07:21.983 INFO:tasks.workunit.client.0.smithi045.stdout:op 657 completed, throughput=4.97MB/sec 2021-07-31T02:07:21.983 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-nbEmenR1ff8lCZ5 off=4299906 len=1470811 2021-07-31T02:07:21.986 INFO:tasks.workunit.client.0.smithi045.stdout:op 658 completed, throughput=4.98MB/sec 2021-07-31T02:07:21.986 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-LJiSGnVW9BZVcKT off=2850406 len=937627 2021-07-31T02:07:21.987 INFO:tasks.workunit.client.0.smithi045.stdout:op 659 completed, throughput=4.99MB/sec 2021-07-31T02:07:21.987 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-gZ517vR5L5B0mLC off=16301530 len=612677 2021-07-31T02:07:21.991 INFO:tasks.workunit.client.0.smithi045.stdout:op 660 completed, throughput=4.99MB/sec 2021-07-31T02:07:21.992 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ztExfRw6I5LbVVP off=4233858 len=745762 2021-07-31T02:07:21.993 INFO:tasks.workunit.client.0.smithi045.stdout:op 661 completed, throughput=4.99MB/sec 2021-07-31T02:07:21.993 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-nWOVJZSFQEJlrKB off=4526461 len=1974627 2021-07-31T02:07:22.004 INFO:tasks.workunit.client.0.smithi045.stdout:op 662 completed, throughput=5.01MB/sec 2021-07-31T02:07:23.005 INFO:tasks.workunit.client.0.smithi045.stdout: 134: throughput=4.97MB/sec pending data=0 2021-07-31T02:07:23.005 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-L5VDmMU_b5XZuOY off=2256459 len=141070 2021-07-31T02:07:23.006 INFO:tasks.workunit.client.0.smithi045.stdout:op 663 completed, throughput=4.97MB/sec 2021-07-31T02:07:23.007 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-mrtnxBGkFNwd84H off=7117171 len=1080706 2021-07-31T02:07:23.011 INFO:tasks.workunit.client.0.smithi045.stdout:op 664 completed, throughput=4.98MB/sec 2021-07-31T02:07:23.011 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-yF3XEpoOz-C91Er off=21395979 len=1044276 2021-07-31T02:07:23.014 INFO:tasks.workunit.client.0.smithi045.stdout:op 665 completed, throughput=4.99MB/sec 2021-07-31T02:07:23.014 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-FLwjeyXfWv6E-dv off=744364 len=1149181 2021-07-31T02:07:23.015 INFO:tasks.workunit.client.0.smithi045.stdout:op 666 completed, throughput=4.99MB/sec 2021-07-31T02:07:23.015 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-NSsgiAOAloDhBH6 off=10923495 len=740415 2021-07-31T02:07:23.020 INFO:tasks.workunit.client.0.smithi045.stdout:op 667 completed, throughput=5MB/sec 2021-07-31T02:07:23.020 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-NNfnEYWzETxnpAg off=3437228 len=569612 2021-07-31T02:07:23.023 INFO:tasks.workunit.client.0.smithi045.stdout:op 668 completed, throughput=5MB/sec 2021-07-31T02:07:23.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:22 smithi154 conmon[34553]: cluster 2021-07-31T02:07:22.076106+0000 mgr.x (mgr.24542 2021-07-31T02:07:23.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:22 smithi154 conmon[34553]: ) 859 : cluster [DBG] pgmap v1476: 9 pgs: 9 active+clean; 127 GiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 1.9 MiB/s rd, 1.9 MiB/s wr, 3 op/s 2021-07-31T02:07:23.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:22 smithi045 conmon[39718]: cluster 2021-07-31T02: 2021-07-31T02:07:23.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:22 smithi045 conmon[39718]: 07:22.076106+0000 mgr.x (mgr.24542) 859 : cluster [DBG] pgmap v1476: 9 pgs: 9 active+clean; 127 GiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 1.9 MiB/s rd, 1.9 MiB/s wr, 3 op/s 2021-07-31T02:07:23.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:22 smithi045 conmon[33655]: cluster 2021-07-31T02:07:22.076106+0000 mgr.x (mgr.24542) 859 : cluster 2021-07-31T02:07:23.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:22 smithi045 conmon[33655]: [DBG] pgmap v1476: 9 pgs: 9 active+clean; 127 GiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 1.9 MiB/s rd, 1.9 MiB/s wr, 3 op/s 2021-07-31T02:07:24.025 INFO:tasks.workunit.client.0.smithi045.stdout: 135: throughput=4.97MB/sec pending data=0 2021-07-31T02:07:24.026 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-IIqg7uZ9Ypo6d0W off=15133792 len=274370 2021-07-31T02:07:24.026 INFO:tasks.workunit.client.0.smithi045.stdout:op 669 completed, throughput=4.97MB/sec 2021-07-31T02:07:24.026 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-m0oy3ZTrArURMNS off=8335047 len=1598937 2021-07-31T02:07:24.027 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:07:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:07:23.542Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:07:24.027 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:07:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:07:23.542Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:07:24.027 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:07:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:07:23.542Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:07:24.028 INFO:tasks.workunit.client.0.smithi045.stdout:op 670 completed, throughput=4.98MB/sec 2021-07-31T02:07:24.028 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-kq77mB74nB7TM6J off=8814548 len=340504 2021-07-31T02:07:24.032 INFO:tasks.workunit.client.0.smithi045.stdout:op 671 completed, throughput=4.98MB/sec 2021-07-31T02:07:24.032 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-oubRgnZ0h5lCBhs off=1871632 len=726496 2021-07-31T02:07:24.037 INFO:tasks.workunit.client.0.smithi045.stdout:op 672 completed, throughput=4.99MB/sec 2021-07-31T02:07:24.037 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-MS1ob_LhDjYm8kD off=674245 len=1882316 2021-07-31T02:07:24.040 INFO:tasks.workunit.client.0.smithi045.stdout:op 673 completed, throughput=5MB/sec 2021-07-31T02:07:24.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:24 smithi154 conmon[34553]: cluster 2021-07-31T02:07:24.076604+0000 mgr.x (mgr.24542) 860 : cluster [DBG] pgmap v1477: 9 pgs: 9 active+clean; 127 GiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2021-07-31T02:07:24.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:24 smithi045 conmon[39718]: cluster 2021-07-31T02:07:24.076604+0000 mgr.x (mgr.24542) 860 : cluster [DBG] pgmap v1477: 9 pgs: 9 active+clean; 127 GiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2021-07-31T02:07:24.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:24 smithi045 conmon[33655]: cluster 2021-07-31T02:07:24.076604+0000 mgr.x (mgr.24542) 860 2021-07-31T02:07:24.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:24 smithi045 conmon[33655]: : cluster [DBG] pgmap v1477: 9 pgs: 9 active+clean; 127 GiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2021-07-31T02:07:25.041 INFO:tasks.workunit.client.0.smithi045.stdout: 136: throughput=4.96MB/sec pending data=0 2021-07-31T02:07:25.042 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-GCKHDXrK8ZSv-du off=11467308 len=1511660 2021-07-31T02:07:25.050 INFO:tasks.workunit.client.0.smithi045.stdout:op 674 completed, throughput=4.97MB/sec 2021-07-31T02:07:25.050 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ksu22G93J80kj_x off=4375208 len=1562169 2021-07-31T02:07:25.056 INFO:tasks.workunit.client.0.smithi045.stdout:op 675 completed, throughput=4.98MB/sec 2021-07-31T02:07:25.056 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-6AddjSu_RB8aFYC off=8189036 len=156422 2021-07-31T02:07:25.057 INFO:tasks.workunit.client.0.smithi045.stdout:op 676 completed, throughput=4.99MB/sec 2021-07-31T02:07:25.058 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-7bBLCoqRNFuYml8 off=9359258 len=233703 2021-07-31T02:07:25.058 INFO:tasks.workunit.client.0.smithi045.stdout:op 677 completed, throughput=4.99MB/sec 2021-07-31T02:07:25.059 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-PQGHtA02LRCW7SO off=5720072 len=1262599 2021-07-31T02:07:25.062 INFO:tasks.workunit.client.0.smithi045.stdout:op 678 completed, throughput=5MB/sec 2021-07-31T02:07:25.063 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-b_pwJIqGn-X9FqK off=6030977 len=1380986 2021-07-31T02:07:25.066 INFO:tasks.workunit.client.0.smithi045.stdout:op 679 completed, throughput=5.01MB/sec 2021-07-31T02:07:26.065 INFO:tasks.workunit.client.0.smithi045.stdout: 137: throughput=4.97MB/sec pending data=0 2021-07-31T02:07:26.066 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-cespfu2rxo0A5fX off=1210135 len=681947 2021-07-31T02:07:26.067 INFO:tasks.workunit.client.0.smithi045.stdout:op 680 completed, throughput=4.97MB/sec 2021-07-31T02:07:26.067 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-CbTCEm7W3ZGNCVA off=339976 len=1019344 2021-07-31T02:07:26.069 INFO:tasks.workunit.client.0.smithi045.stdout:op 681 completed, throughput=4.98MB/sec 2021-07-31T02:07:26.069 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-we7QMuVsPHcQX6K off=14427408 len=1867551 2021-07-31T02:07:26.078 INFO:tasks.workunit.client.0.smithi045.stdout:op 682 completed, throughput=4.99MB/sec 2021-07-31T02:07:26.078 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-TaxpQv1u5br5kTi off=415746 len=1852754 2021-07-31T02:07:26.087 INFO:tasks.workunit.client.0.smithi045.stdout:op 683 completed, throughput=5.01MB/sec 2021-07-31T02:07:26.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:26 smithi154 conmon[34553]: cluster 2021-07-31T02:07:26.077018+0000 mgr.x (mgr.24542) 861 : cluster [DBG] pgmap v1478: 9 pgs: 9 active+clean; 127 GiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 2.1 MiB/s rd, 1.7 MiB/s wr, 4 op/s 2021-07-31T02:07:26.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:26 smithi045 conmon[39718]: cluster 2021-07-31T02:07:26.077018+0000 mgr.x (mgr.24542) 861 : cluster [DBG] pgmap v1478: 9 pgs: 9 active+clean; 127 GiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 2.1 MiB/s rd, 1.7 MiB/s wr, 4 op/s 2021-07-31T02:07:26.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:26 smithi045 conmon[33655]: cluster 2021-07-31T02:07:26.077018+0000 mgr.x (mgr.24542) 861 2021-07-31T02:07:26.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:26 smithi045 conmon[33655]: : cluster [DBG] pgmap v1478: 9 pgs: 9 active+clean; 127 GiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 2.1 MiB/s rd, 1.7 MiB/s wr, 4 op/s 2021-07-31T02:07:27.088 INFO:tasks.workunit.client.0.smithi045.stdout: 138: throughput=4.97MB/sec pending data=0 2021-07-31T02:07:27.088 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-I4fxLk43AB83aJV off=7079787 len=1947482 2021-07-31T02:07:27.098 INFO:tasks.workunit.client.0.smithi045.stdout:op 684 completed, throughput=4.98MB/sec 2021-07-31T02:07:27.098 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-EcYBx9BUpVM-9tE off=8993681 len=1893486 2021-07-31T02:07:27.107 INFO:tasks.workunit.client.0.smithi045.stdout:op 685 completed, throughput=5MB/sec 2021-07-31T02:07:27.108 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ibZvexC1l4gFjmb off=17579183 len=1901025 2021-07-31T02:07:27.110 INFO:tasks.workunit.client.0.smithi045.stdout:op 686 completed, throughput=5.01MB/sec 2021-07-31T02:07:28.111 INFO:tasks.workunit.client.0.smithi045.stdout: 139: throughput=4.97MB/sec pending data=0 2021-07-31T02:07:28.111 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-P4qbznHPqkQ0XaT off=6916041 len=438668 2021-07-31T02:07:28.114 INFO:tasks.workunit.client.0.smithi045.stdout:op 687 completed, throughput=4.98MB/sec 2021-07-31T02:07:28.115 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-_0X2vkb6uB0k0tH off=13107819 len=121519 2021-07-31T02:07:28.115 INFO:tasks.workunit.client.0.smithi045.stdout:op 688 completed, throughput=4.98MB/sec 2021-07-31T02:07:28.115 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-sgTcM1A2o4k9Rha off=19668877 len=701007 2021-07-31T02:07:28.116 INFO:tasks.workunit.client.0.smithi045.stdout:op 689 completed, throughput=4.98MB/sec 2021-07-31T02:07:28.116 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-rI-0tOCByytXnbV off=5652407 len=864305 2021-07-31T02:07:28.118 INFO:tasks.workunit.client.0.smithi045.stdout:op 690 completed, throughput=4.99MB/sec 2021-07-31T02:07:28.118 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-esulQnO1wUazAxE off=5697563 len=1144459 2021-07-31T02:07:28.121 INFO:tasks.workunit.client.0.smithi045.stdout:op 691 completed, throughput=4.99MB/sec 2021-07-31T02:07:28.122 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-HqbWHqJbpkgr-eZ off=9138122 len=760353 2021-07-31T02:07:28.122 INFO:tasks.workunit.client.0.smithi045.stdout:op 692 completed, throughput=5MB/sec 2021-07-31T02:07:28.123 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-wQ2injUmKCMYH7T off=2037887 len=866932 2021-07-31T02:07:28.126 INFO:tasks.workunit.client.0.smithi045.stdout:op 693 completed, throughput=5.01MB/sec 2021-07-31T02:07:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:28 smithi154 conmon[34553]: audit 2021-07-31T02:07:28.007529+0000 mon.b (mon.2) 168 : audit [INF] 2021-07-31T02:07:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:28 smithi154 conmon[34553]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:07:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:28 smithi154 conmon[34553]: audit 2021-07-31T02:07:28.008294+0000 mon.a (mon.0) 1856 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:07:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:28 smithi154 conmon[34553]: audit 2021-07-31T 2021-07-31T02:07:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:28 smithi154 conmon[34553]: 02:07:28.026007+0000 mon.b (mon.2) 169 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:07:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:28 smithi154 conmon[34553]: audit 2021-07 2021-07-31T02:07:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:28 smithi154 conmon[34553]: -31T02:07:28.026625+0000 2021-07-31T02:07:28.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:28 smithi154 conmon[34553]: mon.a (mon.0) 1857 : audit 2021-07-31T02:07:28.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:28 smithi154 conmon[34553]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:07:28.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:28 smithi045 conmon[39718]: audit 2021-07-31T02:07:28. 2021-07-31T02:07:28.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:28 smithi045 conmon[39718]: 007529+0000 mon.b (mon.2) 168 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:07:28.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:28 smithi045 conmon[39718]: audit 2021-07-31T02:07:28.008294+0000 2021-07-31T02:07:28.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:28 smithi045 conmon[39718]: mon.a (mon.0) 1856 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:07:28.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:28 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T02:07:28.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:28 smithi045 conmon[39718]: T02:07:28.026007+0000 mon.b (mon.2) 169 : audit 2021-07-31T02:07:28.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:28 smithi045 conmon[39718]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:07:28.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:28 smithi045 conmon[39718]: audit 2021-07-31T02:07:28.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:28 smithi045 conmon[39718]: 2021-07-31T02:07:28 2021-07-31T02:07:28.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:28 smithi045 conmon[39718]: .026625+0000 mon.a (mon.0) 2021-07-31T02:07:28.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:28 smithi045 conmon[39718]: 1857 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:07:28.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:28 smithi045 conmon[33655]: audit 2021-07-31T02:07:28.007529 2021-07-31T02:07:28.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:28 smithi045 conmon[33655]: +0000 mon.b (mon.2) 168 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:07:28.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:28 smithi045 conmon[33655]: audit 2021-07-31T02:07:28.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:28 smithi045 conmon[33655]: 2021-07-31T02:07:28.008294+0000 mon.a 2021-07-31T02:07:28.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:28 smithi045 conmon[33655]: (mon.0) 1856 : audit 2021-07-31T02:07:28.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:28 smithi045 conmon[33655]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:07:28.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:28 smithi045 conmon[33655]: audit 2021-07-31T02:07:28.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:28 smithi045 conmon[33655]: 2021-07-31T02:07:28 2021-07-31T02:07:28.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:28 smithi045 conmon[33655]: . 2021-07-31T02:07:28.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:28 smithi045 conmon[33655]: 026007+0000 mon.b 2021-07-31T02:07:28.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:28 smithi045 conmon[33655]: (mon.2) 169 : audit [INF] 2021-07-31T02:07:28.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:28 smithi045 conmon[33655]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:07:28.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:28 smithi045 conmon[33655]: audit 2021-07-31T02:07:28.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:28 smithi045 conmon[33655]: 2021-07-31T02:07: 2021-07-31T02:07:28.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:28 smithi045 conmon[33655]: 28.026625+0000 mon.a ( 2021-07-31T02:07:28.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:28 smithi045 conmon[33655]: mon.0) 1857 : audit 2021-07-31T02:07:28.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:28 smithi045 conmon[33655]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:07:29.127 INFO:tasks.workunit.client.0.smithi045.stdout: 140: throughput=4.97MB/sec pending data=0 2021-07-31T02:07:29.127 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-69CsBvN8yoRHOkZ off=18875094 len=1428112 2021-07-31T02:07:29.134 INFO:tasks.workunit.client.0.smithi045.stdout:op 694 completed, throughput=4.98MB/sec 2021-07-31T02:07:29.135 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ePb60XVRBd_RVKk off=5631246 len=1478061 2021-07-31T02:07:29.141 INFO:tasks.workunit.client.0.smithi045.stdout:op 695 completed, throughput=4.99MB/sec 2021-07-31T02:07:29.141 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-PMvbUf_c1aJAhL8 off=4135731 len=1114074 2021-07-31T02:07:29.144 INFO:tasks.workunit.client.0.smithi045.stdout:op 696 completed, throughput=5MB/sec 2021-07-31T02:07:29.144 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-xtOmgYeJjCfKJVW off=4298928 len=1414673 2021-07-31T02:07:29.151 INFO:tasks.workunit.client.0.smithi045.stdout:op 697 completed, throughput=5.01MB/sec 2021-07-31T02:07:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:29 smithi154 conmon[34553]: cluster 2021-07-31T02:07:28.077478+0000 mgr.x (mgr.24542) 862 : cluster [DBG] pgmap v1479: 9 pgs: 9 active+clean; 127 GiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 2.1 MiB/s rd, 1.7 MiB/s wr, 4 op/s 2021-07-31T02:07:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:29 smithi154 conmon[34553]: 2021-07-31T02:07:29.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:29 smithi045 conmon[33655]: cluster 2021-07-31T02:07:28.077478+0000 mgr.x (mgr.24542) 862 : cluster [DBG] pgmap v1479: 9 pgs: 9 active+clean; 127 GiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 2.1 MiB/s rd, 1.7 MiB/s wr, 4 op/s 2021-07-31T02:07:29.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:29 smithi045 conmon[39718]: cluster 2021-07-31T02:07:28. 2021-07-31T02:07:29.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:29 smithi045 conmon[39718]: 077478+0000 mgr.x (mgr.24542) 862 : cluster [DBG] pgmap v1479: 9 pgs: 9 active+clean; 127 GiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 2.1 MiB/s rd, 1.7 MiB/s wr, 4 op/s 2021-07-31T02:07:30.151 INFO:tasks.workunit.client.0.smithi045.stdout: 141: throughput=4.97MB/sec pending data=0 2021-07-31T02:07:30.151 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-CO8GIgKZ8C3FJCx off=731931 len=1886579 2021-07-31T02:07:30.154 INFO:tasks.workunit.client.0.smithi045.stdout:op 698 completed, throughput=4.98MB/sec 2021-07-31T02:07:30.155 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-p4Rmc9ROIU6Usn_ off=8537943 len=909219 2021-07-31T02:07:30.156 INFO:tasks.workunit.client.0.smithi045.stdout:op 699 completed, throughput=4.99MB/sec 2021-07-31T02:07:30.157 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-71uX8-5Q9o0gvY6 off=488303 len=882371 2021-07-31T02:07:30.161 INFO:tasks.workunit.client.0.smithi045.stdout:op 700 completed, throughput=4.99MB/sec 2021-07-31T02:07:30.162 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-NLj1_Xbtg7fuuD4 off=95348 len=1496372 2021-07-31T02:07:30.163 INFO:tasks.workunit.client.0.smithi045.stdout:op 701 completed, throughput=5MB/sec 2021-07-31T02:07:30.201 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[34553]: cluster 2021-07-31T02:07:30.078072+0000 mgr.x (mgr.24542) 863 : 2021-07-31T02:07:30.202 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[34553]: cluster [DBG] pgmap v1480: 9 pgs: 9 active+clean; 127 GiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.9 MiB/s wr, 6 op/s 2021-07-31T02:07:30.202 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[34553]: audit 2021-07-31T02:07:30.103935+0000 mon.b (mon.2) 170 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:07:30.202 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[34553]: audit 2021-07-31T02:07:30. 2021-07-31T02:07:30.202 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[34553]: 105316+0000 mon.b (mon.2) 171 : 2021-07-31T02:07:30.203 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[34553]: audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:07:30.203 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[34553]: audit 2021-07 2021-07-31T02:07:30.203 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[34553]: -31T02:07:30.106381+0000 2021-07-31T02:07:30.203 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[34553]: mon.b (mon.2) 172 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:07:30.204 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[34553]: 2021-07-31T02:07:30.204 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[34553]: cephadm 2021-07 2021-07-31T02:07:30.204 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[34553]: -31T02:07:30. 2021-07-31T02:07:30.204 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[34553]: 107110+0000 mgr.x (mgr.24542) 2021-07-31T02:07:30.205 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[34553]: 864 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:07:30.205 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: debug 2021-07-31T02:07:30.150+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:07:30.206 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:07:30.206 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:07:30.206 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: yield 2021-07-31T02:07:30.206 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:07:30.206 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:07:30.207 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:07:30.207 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:07:30.207 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:07:30.207 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:07:30.208 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:07:30.208 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:07:30.208 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:07:30.208 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:07:30.208 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:07:30.209 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: debug 2021-07-31T02:07:30.163+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:07:30.209 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:07:30.209 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:07:30.209 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: yield 2021-07-31T02:07:30.210 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:07:30.210 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:07:30.210 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:07:30.210 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:07:30.210 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:07:30.211 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:07:30.211 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:07:30.212 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:07:30.212 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:07:30.212 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:07:30.212 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:07:30.213 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: debug 2021-07-31T02:07:30.189+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:07:30.213 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:07:30.213 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:07:30.213 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: yield 2021-07-31T02:07:30.213 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:07:30.214 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:07:30.214 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:07:30.214 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:07:30.214 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:07:30.215 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:07:30.215 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:07:30.215 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:07:30.215 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:07:30.215 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:07:30.216 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:07:30.216 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: debug 2021-07-31T02:07:30.190+0000 7fcb93961700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:07:30.216 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:07:30.216 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:07:30.217 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: yield 2021-07-31T02:07:30.217 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:07:30.217 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:07:30.217 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:07:30.217 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:07:30.218 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:07:30.218 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:07:30.218 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:07:30.218 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:07:30.219 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:07:30.219 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:07:30.219 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:07:30.219 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: 2021-07-31T02:07:30.220 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:07:30.220 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: 2021-07-31T02:07:30.220 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:07:30.220 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:07:30.220 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:07:30.221 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:07:30.221 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:07:30.221 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:07:30.221 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:07:30.222 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:07:30.222 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:07:30.222 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:07:30.222 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:07:30.222 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:07:30.223 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:07:30.223 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:07:30.223 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:07:30.223 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:07:30.224 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:07:30.224 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:07:30.224 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: raise 2021-07-31T02:07:30.224 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:07:30.224 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:07:30.225 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:07:30.225 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:07:30.225 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:07:30.225 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: debug 2021-07-31T02:07:30.198+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:07:30.226 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:07:30.226 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:07:30.226 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: yield 2021-07-31T02:07:30.226 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:07:30.226 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:07:30.227 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:07:30.227 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:07:30.227 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:07:30.227 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:07:30.228 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:07:30.228 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:07:30.228 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:07:30.228 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:07:30.228 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:07:30.229 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: debug 2021-07-31T02:07:30.199+0000 7fcb94162700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:07:30.229 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:07:30.229 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:07:30.229 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: yield 2021-07-31T02:07:30.503 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:07:30.503 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:07:30.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:07:30.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:07:30.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:07:30.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:07:30.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:07:30.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:07:30.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:07:30.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:07:30.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:07:30.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: 2021-07-31T02:07:30.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:07:30.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: 2021-07-31T02:07:30.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:07:30.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:07:30.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:07:30.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:07:30.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:07:30.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:07:30.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:07:30.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:07:30.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:07:30.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:07:30.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:07:30.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:07:30.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:07:30.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:07:30.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:07:30.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:07:30.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:07:30.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:07:30.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: raise 2021-07-31T02:07:30.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:07:30.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:07:30.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:07:30.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:07:30.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:07:30 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:07:30.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:30 smithi045 conmon[39718]: cluster 2021-07-31T02:07:30.078072+0000 mgr.x ( 2021-07-31T02:07:30.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:30 smithi045 conmon[39718]: mgr.24542) 863 : cluster [DBG] pgmap v1480: 9 pgs: 9 active+clean; 127 GiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.9 MiB/s wr, 6 op/s 2021-07-31T02:07:30.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:30 smithi045 conmon[39718]: audit 2021-07-31T02: 2021-07-31T02:07:30.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:30 smithi045 conmon[39718]: 07:30.103935+0000 mon.b (mon.2) 170 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:07:30.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:30 smithi045 conmon[39718]: audit 2021-07-31T02:07:30.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:30 smithi045 conmon[39718]: 2021-07-31T 2021-07-31T02:07:30.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:30 smithi045 conmon[39718]: 02:07:30.105316+0000 mon.b (mon.2 2021-07-31T02:07:30.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:30 smithi045 conmon[39718]: ) 171 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:07:30.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:30 smithi045 conmon[39718]: audit 2021-07 2021-07-31T02:07:30.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:30 smithi045 conmon[39718]: -31T02:07:30.106381+0000 mon.b (mon.2) 172 : audit 2021-07-31T02:07:30.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:30 smithi045 conmon[39718]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:07:30.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:07:30.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:30 smithi045 conmon[39718]: 2021-07-31T02:07:30.107110+0000 2021-07-31T02:07:30.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:30 smithi045 conmon[39718]: mgr.x (mgr.24542) 864 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:07:30.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:30 smithi045 conmon[39718]: 2021-07-31T02:07:30.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:30 smithi045 conmon[33655]: cluster 2021-07-31T02:07:30.078072+0000 mgr.x (mgr.24542) 863 : cluster [DBG] 2021-07-31T02:07:30.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:30 smithi045 conmon[33655]: pgmap v1480: 9 pgs: 9 active+clean; 127 GiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.9 MiB/s wr, 6 op/s 2021-07-31T02:07:30.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:30 smithi045 conmon[33655]: audit 2021-07-31T02:07:30.103935+0000 mon.b (mon.2) 170 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:07:30.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:30 smithi045 conmon[33655]: audit 2021-07-31 2021-07-31T02:07:30.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:30 smithi045 conmon[33655]: T02:07:30.105316+0000 mon.b (mon.2) 171 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:07:30.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:30 smithi045 conmon[33655]: audit 2021-07- 2021-07-31T02:07:30.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:30 smithi045 conmon[33655]: 31T02:07:30.106381+0000 mon.b (mon.2) 172 2021-07-31T02:07:30.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:30 smithi045 conmon[33655]: : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:07:30.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:07:30.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:30 smithi045 conmon[33655]: 2021-07-31T02:07:30.107110 2021-07-31T02:07:30.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:30 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 864 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:07:30.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:30 smithi045 conmon[33655]: 2021-07-31T02:07:31.164 INFO:tasks.workunit.client.0.smithi045.stdout: 142: throughput=4.97MB/sec pending data=0 2021-07-31T02:07:31.165 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-7WgQgfzBzRb-UO6 off=14249624 len=1391519 2021-07-31T02:07:31.172 INFO:tasks.workunit.client.0.smithi045.stdout:op 702 completed, throughput=4.98MB/sec 2021-07-31T02:07:31.172 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-g7pLxAr_4MI8TPu off=14169094 len=1437917 2021-07-31T02:07:31.179 INFO:tasks.workunit.client.0.smithi045.stdout:op 703 completed, throughput=4.99MB/sec 2021-07-31T02:07:31.179 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-8Ozj9iiyhS_1NaP off=4750304 len=1042204 2021-07-31T02:07:31.185 INFO:tasks.workunit.client.0.smithi045.stdout:op 704 completed, throughput=4.99MB/sec 2021-07-31T02:07:31.185 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-5fERY9I3KUdyKjw off=1453462 len=266820 2021-07-31T02:07:31.186 INFO:tasks.workunit.client.0.smithi045.stdout:op 705 completed, throughput=5MB/sec 2021-07-31T02:07:31.187 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-HSJljMvE4T48XdB off=13207527 len=592807 2021-07-31T02:07:31.187 INFO:tasks.workunit.client.0.smithi045.stdout:op 706 completed, throughput=5MB/sec 2021-07-31T02:07:31.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:07:30.151747+0000 mgr.x (mgr. 2021-07-31T02:07:31.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: 24542) 865 : cephadm [ERR] 2021-07-31T02:07:31.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:07:31.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:07:31.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: yield 2021-07-31T02:07:31.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:07:31.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:07:31.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:07:31.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:07:31.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:07:31.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:07:31.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:07:31.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:07:31.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:07:31.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: task.result() 2021-07-31T02:07:31.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:07:31.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:07:30.152304+0000 mgr.x (mgr.24542) 866 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:07:31.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:07:30.164833+0000 mgr.x (mgr.24542) 867 : cephadm [ERR] 2021-07-31T02:07:31.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:07:31.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:07:31.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: yield 2021-07-31T02:07:31.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:07:31.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:07:31.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:07:31.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:07:31.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:07:31.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:07:31.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:07:31.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:07:31.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:07:31.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: task.result() 2021-07-31T02:07:31.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:07:31.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:07:30.166080+0000 mgr.x (mgr.24542) 868 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:07:31.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:07:30.190833+0000 mgr.x (mgr.24542) 869 : cephadm [ERR] 2021-07-31T02:07:31.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:07:31.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:07:31.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: yield 2021-07-31T02:07:31.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:07:31.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:07:31.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:07:31.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:07:31.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:07:31.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:07:31.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:07:31.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:07:31.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:07:31.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: task.result() 2021-07-31T02:07:31.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:07:31.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:07:30.191266+0000 mgr.x (mgr.24542) 870 : 2021-07-31T02:07:31.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:07:31.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:07:31.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:07:31.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: yield 2021-07-31T02:07:31.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:07:31.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:07:31.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:07:31.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:07:31.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:07:31.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:07:31.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:07:31.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:07:31.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:07:31.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: task.result() 2021-07-31T02:07:31.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:07:31.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: 2021-07-31T02:07:31.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:07:31.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: 2021-07-31T02:07:31.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:07:31.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:07:31.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:07:31.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:07:31.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:07:31.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:07:31.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:07:31.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:07:31.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:07:31.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:07:31.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:07:31.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:07:31.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:07:31.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:07:31.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:07:31.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:07:31.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:07:31.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:07:31.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: raise 2021-07-31T02:07:31.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:07:31.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:07:31.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:07:31.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:07:31.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:07:31.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:07:30.199901+0000 mgr.x (mgr.24542) 871 : cephadm [ERR] 2021-07-31T02:07:31.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:07:31.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:07:31.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: yield 2021-07-31T02:07:31.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:07:31.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:07:31.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:07:31.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:07:31.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:07:31.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:07:31.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:07:31.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:07:31.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:07:31.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: task.result() 2021-07-31T02:07:31.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:07:31.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: cephadm 2021-07-31T02: 2021-07-31T02:07:31.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: 07:30.200287+0000 mgr.x (mgr.24542) 872 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:07:31.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:07:31.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:07:31.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: yield 2021-07-31T02:07:31.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:07:31.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:07:31.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:07:31.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:07:31.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:07:31.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:07:31.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:07:31.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:07:31.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:07:31.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: task.result() 2021-07-31T02:07:31.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:07:31.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: 2021-07-31T02:07:31.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:07:31.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: 2021-07-31T02:07:31.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:07:31.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:07:31.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:07:31.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:07:31.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:07:31.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:07:31.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:07:31.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:07:31.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:07:31.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:07:31.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:07:31.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:07:31.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:07:31.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:07:31.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:07:31.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:07:31.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:07:31.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:07:31.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: raise 2021-07-31T02:07:31.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:07:31.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:07:31.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:07:31.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:07:31.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:31 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:07:31.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:07:30.151747+0000 mgr.x ( 2021-07-31T02:07:31.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: mgr.24542) 865 : cephadm [ERR] 2021-07-31T02:07:31.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:07:31.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:07:31.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: yield 2021-07-31T02:07:31.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:07:31.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:07:31.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:07:31.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:07:31.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:07:31.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:07:31.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:07:31.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:07:31.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:07:31.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: task.result() 2021-07-31T02:07:31.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:07:31.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:07:30 2021-07-31T02:07:31.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: .152304+0000 mgr.x (mgr.24542) 866 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:07:31.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:07:30.164833 2021-07-31T02:07:31.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 867 : cephadm [ERR] 2021-07-31T02:07:31.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: 2021-07-31T02:07:31.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:07:31.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:07:31.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: yield 2021-07-31T02:07:31.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:07:31.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:07:31.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:07:31.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:07:31.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:07:31.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:07:31.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:07:31.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:07:31.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:07:31.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: task.result() 2021-07-31T02:07:31.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:07:31.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:07:30.166080+0000 mgr.x (mgr.24542) 868 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:07:31.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:07:31.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: 2021-07- 2021-07-31T02:07:31.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: 31T02:07:30. 2021-07-31T02:07:31.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: 190833+0000 mgr.x (mgr.24542 2021-07-31T02:07:31.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: ) 869 : cephadm [ERR] 2021-07-31T02:07:31.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:07:31.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:07:31.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: yield 2021-07-31T02:07:31.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:07:31.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:07:31.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:07:31.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:07:31.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:07:31.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:07:31.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:07:31.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:07:31.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:07:31.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: task.result() 2021-07-31T02:07:31.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:07:31.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:07:30.191266+0000 mgr.x (mgr.24542) 870 : cephadm 2021-07-31T02:07:31.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:07:31.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:07:31.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:07:31.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: yield 2021-07-31T02:07:31.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:07:31.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:07:31.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:07:31.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:07:31.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:07:31.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:07:31.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:07:31.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:07:31.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:07:31.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: task.result() 2021-07-31T02:07:31.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:07:31.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: 2021-07-31T02:07:31.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:07:31.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: 2021-07-31T02:07:31.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:07:31.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:07:31.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:07:31.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:07:31.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:07:31.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:07:31.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:07:31.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:07:31.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:07:31.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:07:31.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:07:31.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:07:30.151747+0000 mgr.x (mgr 2021-07-31T02:07:31.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: .24542) 865 : cephadm [ERR] 2021-07-31T02:07:31.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:07:31.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:07:31.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: yield 2021-07-31T02:07:31.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:07:31.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:07:31.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:07:31.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:07:31.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:07:31.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:07:31.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:07:31.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:07:31.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:07:31.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: task.result() 2021-07-31T02:07:31.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:07:31.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:07:30.152304+0000 mgr.x (mgr.24542) 866 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:07:31.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:07:30.164833+0000 mgr.x (mgr.24542) 867 : cephadm [ERR] 2021-07-31T02:07:31.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:07:31.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:07:31.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: yield 2021-07-31T02:07:31.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:07:31.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:07:31.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:07:31.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:07:31.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:07:31.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:07:31.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:07:31.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:07:31.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:07:31.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: task.result() 2021-07-31T02:07:31.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:07:31.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:07:30.166080+0000 mgr.x (mgr.24542) 868 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:07:31.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:07:30.190833+0000 mgr.x (mgr.24542) 869 : cephadm [ERR] 2021-07-31T02:07:31.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:07:31.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:07:31.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: yield 2021-07-31T02:07:31.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:07:31.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:07:31.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:07:31.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:07:31.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:07:31.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:07:31.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:07:31.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:07:31.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: raise 2021-07-31T02:07:31.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:07:31.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:07:31.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:07:31.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:07:31.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:07:31.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:07:30.199901+0000 mgr.x (mgr.24542) 871 : cephadm [ERR] 2021-07-31T02:07:31.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:07:31.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:07:31.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: yield 2021-07-31T02:07:31.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:07:31.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:07:31.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:07:31.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:07:31.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:07:31.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:07:31.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:07:31.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:07:31.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:07:31.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: task.result() 2021-07-31T02:07:31.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:07:31.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:07:30.200287+0000 mgr.x (mgr.24542) 872 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:07:31.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:07:31.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:07:31.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: yield 2021-07-31T02:07:31.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:07:31.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:07:31.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:07:31.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:07:31.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:07:31.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:07:31.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:07:31.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:07:31.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:07:31.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:07:31.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:07:31.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:07:31.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:07:31.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:07:31.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: task.result() 2021-07-31T02:07:31.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:07:31.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: cephadm 2021-07-31T02 2021-07-31T02:07:31.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: :07:30.191266+0000 mgr.x (mgr.24542) 870 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:07:31.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:07:31.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:07:31.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: yield 2021-07-31T02:07:31.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:07:31.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:07:31.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:07:31.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:07:31.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:07:31.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:07:31.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:07:31.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:07:31.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:07:31.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: task.result() 2021-07-31T02:07:31.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:07:31.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: 2021-07-31T02:07:31.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:07:31.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: 2021-07-31T02:07:31.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:07:31.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:07:31.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:07:31.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:07:31.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:07:31.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:07:31.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:07:31.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:07:31.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:07:31.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:07:31.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:07:31.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:07:31.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:07:31.586 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:07:31.586 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:07:31.586 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:07:31.587 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: task.result() 2021-07-31T02:07:31.587 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:07:31.587 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: 2021-07-31T02:07:31.587 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:07:31.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: 2021-07-31T02:07:31.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:07:31.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:07:31.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:07:31.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:07:31.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:07:31.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:07:31.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:07:31.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:07:31.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:07:31.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:07:31.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:07:31.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:07:31.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:07:31.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:07:31.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:07:31.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:07:31.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:07:31.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:07:31.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: raise 2021-07-31T02:07:31.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:07:31.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:07:31.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:07:31.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:07:31.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:07:31.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:07:31.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:07:31.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:07:31.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:07:31.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:07:31.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: raise 2021-07-31T02:07:31.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:07:31.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:07:31.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:07:31.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:07:31.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:07:31.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:07:30.199901+0000 mgr.x (mgr.24542) 871 : cephadm [ERR] 2021-07-31T02:07:31.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:07:31.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:07:31.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: yield 2021-07-31T02:07:31.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:07:31.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:07:31.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:07:31.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:07:31.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:07:31.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:07:31.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:07:31.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:07:31.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:07:31.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: task.result() 2021-07-31T02:07:31.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:07:31.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:07:30.200287+0000 mgr.x (mgr.24542) 872 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:07:31.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:07:31.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:07:31.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: yield 2021-07-31T02:07:31.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:07:31.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:07:31.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:07:31.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:07:31.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:07:31.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:07:31.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:07:31.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:07:31.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:07:31.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: task.result() 2021-07-31T02:07:31.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:07:31.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: 2021-07-31T02:07:31.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:07:31.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: 2021-07-31T02:07:31.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:07:31.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:07:31.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:07:31.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:07:31.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:07:31.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:07:31.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:07:31.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:07:31.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:07:31.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:07:31.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:07:31.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:07:31.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:07:31.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:07:31.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:07:31.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:07:31.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:07:31.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:07:31.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: raise 2021-07-31T02:07:31.608 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:07:31.608 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:07:31.608 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:07:31.608 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:07:31.608 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:31 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:07:32.187 INFO:tasks.workunit.client.0.smithi045.stdout: 143: throughput=4.97MB/sec pending data=0 2021-07-31T02:07:32.188 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-TMlLOAmPK3LWtQ_ off=7892049 len=909237 2021-07-31T02:07:32.189 INFO:tasks.workunit.client.0.smithi045.stdout:op 707 completed, throughput=4.97MB/sec 2021-07-31T02:07:32.190 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-rsdLMQ3G7Qc8hYs off=773995 len=1760233 2021-07-31T02:07:32.193 INFO:tasks.workunit.client.0.smithi045.stdout:op 708 completed, throughput=4.98MB/sec 2021-07-31T02:07:32.193 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-P5K93bBzZffOhEf off=8012922 len=698117 2021-07-31T02:07:32.198 INFO:tasks.workunit.client.0.smithi045.stdout:op 709 completed, throughput=4.99MB/sec 2021-07-31T02:07:32.198 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-TQGezdugAkg3y68 off=3083625 len=747941 2021-07-31T02:07:32.199 INFO:tasks.workunit.client.0.smithi045.stdout:op 710 completed, throughput=4.99MB/sec 2021-07-31T02:07:32.200 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-eHbpOx-7yru3sET off=1160038 len=322833 2021-07-31T02:07:32.203 INFO:tasks.workunit.client.0.smithi045.stdout:op 711 completed, throughput=4.99MB/sec 2021-07-31T02:07:32.203 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-T9JPpmX2jiOu4kq off=185212 len=1021963 2021-07-31T02:07:32.209 INFO:tasks.workunit.client.0.smithi045.stdout:op 712 completed, throughput=5MB/sec 2021-07-31T02:07:32.648 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:07:32.927 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:07:32.927 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 22m ago 23m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:07:32.927 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 20m ago 22m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:07:32.928 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 20m ago 26m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:07:32.928 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 22m ago 28m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:07:32.928 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 22m ago 28m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:07:32.928 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 20m ago 26m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:07:32.929 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 22m ago 26m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:07:32.929 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 22m ago 23m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:07:32.929 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 20m ago 23m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:07:32.929 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 22m ago 26m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:07:32.929 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 22m ago 25m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:07:32.930 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 22m ago 25m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:07:32.930 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 22m ago 25m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:07:32.930 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 20m ago 24m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:07:32.930 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 20m ago 24m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:07:32.931 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 20m ago 24m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:07:32.931 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 20m ago 23m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:07:32.931 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 20m ago 23m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:07:33.210 INFO:tasks.workunit.client.0.smithi045.stdout: 144: throughput=4.97MB/sec pending data=0 2021-07-31T02:07:33.210 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Vl-RyQ037cWCJMB off=1286432 len=1372887 2021-07-31T02:07:33.211 INFO:tasks.workunit.client.0.smithi045.stdout:op 713 completed, throughput=4.98MB/sec 2021-07-31T02:07:33.211 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-HjjPGkMAUoIwybr off=10564596 len=279639 2021-07-31T02:07:33.212 INFO:tasks.workunit.client.0.smithi045.stdout:op 714 completed, throughput=4.98MB/sec 2021-07-31T02:07:33.212 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Qczx2Fc-3yfVkZt off=1541644 len=2032963 2021-07-31T02:07:33.219 INFO:tasks.workunit.client.0.smithi045.stdout:op 715 completed, throughput=4.99MB/sec 2021-07-31T02:07:33.219 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Uaf4pxezrL-itAZ off=10838527 len=1913595 2021-07-31T02:07:33.229 INFO:tasks.workunit.client.0.smithi045.stdout:op 716 completed, throughput=5MB/sec 2021-07-31T02:07:33.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:32 smithi154 conmon[34553]: cluster 2021-07-31T02:07:32.078588+0000 mgr.x (mgr. 2021-07-31T02:07:33.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:32 smithi154 conmon[34553]: 24542) 873 : cluster [DBG] pgmap v1481: 9 pgs: 9 active+clean; 127 GiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2021-07-31T02:07:33.264 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:07:33.264 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:07:33.265 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:07:33.265 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:07:33.265 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:07:33.265 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:07:33.265 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:07:33.266 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:07:33.266 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:07:33.266 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:07:33.267 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:07:33.267 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:07:33.267 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:07:33.267 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:07:33.268 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:07:33.268 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:07:33.268 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:07:33.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:32 smithi045 conmon[33655]: cluster 2021-07-31T02:07:32.078588 2021-07-31T02:07:33.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:32 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 873 : cluster [DBG] pgmap v1481: 9 pgs: 9 active+clean; 127 GiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2021-07-31T02:07:33.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:32 smithi045 conmon[39718]: cluster 2021-07-31T02:07 2021-07-31T02:07:33.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:32 smithi045 conmon[39718]: :32.078588+0000 mgr.x (mgr.24542) 873 : cluster [DBG] pgmap v1481: 9 pgs: 9 active+clean; 127 GiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2021-07-31T02:07:33.934 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:07:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:07:33.542Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:07:33.934 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:07:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:07:33.542Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:07:33.934 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:07:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:07:33.542Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:07:34.230 INFO:tasks.workunit.client.0.smithi045.stdout: 145: throughput=4.97MB/sec pending data=0 2021-07-31T02:07:34.230 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-se7AgWyWhJED85Y off=4255060 len=1532114 2021-07-31T02:07:34.232 INFO:tasks.workunit.client.0.smithi045.stdout:op 717 completed, throughput=4.98MB/sec 2021-07-31T02:07:34.233 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-3nhbMErZGFPTf8k off=2267189 len=1805104 2021-07-31T02:07:34.241 INFO:tasks.workunit.client.0.smithi045.stdout:op 718 completed, throughput=4.99MB/sec 2021-07-31T02:07:34.241 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-3dwwkXU4O3Dfr6n off=1309808 len=746337 2021-07-31T02:07:34.246 INFO:tasks.workunit.client.0.smithi045.stdout:op 719 completed, throughput=4.99MB/sec 2021-07-31T02:07:34.246 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-sANfuXY9mE05mfb off=404200 len=1864961 2021-07-31T02:07:34.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:33 smithi154 conmon[34553]: audit 2021-07-31T02:07:32.635309+0000 mgr.x (mgr.24542) 2021-07-31T02:07:34.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:33 smithi154 conmon[34553]: 874 : audit [DBG] from='client.15807 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:07:34.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:33 smithi154 conmon[34553]: audit 2021-07-31T02:07:32.918960+0000 mgr.x (mgr.24542) 875 : audit [DBG] from='client.15813 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:07:34.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:33 smithi154 conmon[34553]: audit 2021-07-31T02:07:33.264395+0000 mon.a (mon.0) 1858 2021-07-31T02:07:34.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:33 smithi154 conmon[34553]: : audit [DBG] from='client.? 172.21.15.45:0/3024285785' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:07:34.257 INFO:tasks.workunit.client.0.smithi045.stdout:op 720 completed, throughput=5.01MB/sec 2021-07-31T02:07:34.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:33 smithi045 conmon[39718]: audit 2021-07-31T02:07:32.635309+0000 mgr.x (mgr.24542) 874 : audit 2021-07-31T02:07:34.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:33 smithi045 conmon[39718]: [DBG] from='client.15807 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:07:34.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:33 smithi045 conmon[39718]: audit 2021-07-31T02:07:32.918960+0000 mgr.x (mgr.24542) 875 : audit [DBG] from='client.15813 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:07:34.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:33 smithi045 conmon[39718]: audit 2021-07-31T02:07:33.264395+0000 mon.a (mon.0) 1858 : audit [DBG] from='client.? 172.21.15.45:0/3024285785' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:07:34.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:33 smithi045 conmon[39718]: 2021-07-31T02:07:34.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:33 smithi045 conmon[33655]: audit 2021-07-31T02:07:32.635309 2021-07-31T02:07:34.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:33 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 874 : audit [DBG] from='client.15807 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:07:34.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:33 smithi045 conmon[33655]: audit 2021-07-31T02:07:34.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:33 smithi045 conmon[33655]: 2021-07-31T02:07:32.918960+0000 2021-07-31T02:07:34.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:33 smithi045 conmon[33655]: mgr.x (mgr.24542) 875 2021-07-31T02:07:34.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:33 smithi045 conmon[33655]: : audit [DBG] from='client.15813 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:07:34.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:33 smithi045 conmon[33655]: audit 2021-07- 2021-07-31T02:07:34.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:33 smithi045 conmon[33655]: 31T02:07:33.264395+0000 mon.a (mon.0) 1858 : audit [DBG] from='client.? 172.21.15.45:0/3024285785' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:07:34.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:33 smithi045 conmon[33655]: 2021-07-31T02:07:35.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:34 smithi154 conmon[34553]: cluster 2021-07-31T02:07:34.079063+0000 mgr.x (mgr.24542) 876 : cluster [DBG] 2021-07-31T02:07:35.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:34 smithi154 conmon[34553]: pgmap v1482: 9 pgs: 9 active+clean; 127 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 3.1 MiB/s rd, 3.3 MiB/s wr, 6 op/s 2021-07-31T02:07:35.257 INFO:tasks.workunit.client.0.smithi045.stdout: 146: throughput=4.97MB/sec pending data=0 2021-07-31T02:07:35.258 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-cGQwUTWAoZUxKJ3 off=4044347 len=1110900 2021-07-31T02:07:35.265 INFO:tasks.workunit.client.0.smithi045.stdout:op 721 completed, throughput=4.98MB/sec 2021-07-31T02:07:35.265 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-FVnKXZgTZ9mVjiP off=530829 len=1388698 2021-07-31T02:07:35.271 INFO:tasks.workunit.client.0.smithi045.stdout:op 722 completed, throughput=4.99MB/sec 2021-07-31T02:07:35.271 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-QnmVTgpBWMBFDqZ off=1935739 len=732920 2021-07-31T02:07:35.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:34 smithi045 conmon[39718]: cluster 2021-07-31T02:07:34.079063+0000 mgr.x 2021-07-31T02:07:35.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:34 smithi045 conmon[39718]: (mgr.24542) 876 : cluster [DBG] pgmap v1482: 9 pgs: 9 active+clean; 127 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 3.1 MiB/s rd, 3.3 MiB/s wr, 6 op/s 2021-07-31T02:07:35.277 INFO:tasks.workunit.client.0.smithi045.stdout:op 723 completed, throughput=4.99MB/sec 2021-07-31T02:07:35.277 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-S34JDsA1DbreG8a off=6730753 len=206373 2021-07-31T02:07:35.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:34 smithi045 conmon[33655]: cluster 2021-07-31T02:07:34.079063+0000 mgr.x (mgr.24542) 876 : 2021-07-31T02:07:35.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:34 smithi045 conmon[33655]: cluster [DBG] pgmap v1482: 9 pgs: 9 active+clean; 127 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 3.1 MiB/s rd, 3.3 MiB/s wr, 6 op/s 2021-07-31T02:07:35.279 INFO:tasks.workunit.client.0.smithi045.stdout:op 724 completed, throughput=4.99MB/sec 2021-07-31T02:07:35.279 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-1553Mm6ASgmQjyb off=2996086 len=1995559 2021-07-31T02:07:35.281 INFO:tasks.workunit.client.0.smithi045.stdout:op 725 completed, throughput=5.01MB/sec 2021-07-31T02:07:36.281 INFO:tasks.workunit.client.0.smithi045.stdout: 147: throughput=4.97MB/sec pending data=0 2021-07-31T02:07:36.282 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj--mmTxqMnK49-ilb off=3349193 len=1667738 2021-07-31T02:07:36.292 INFO:tasks.workunit.client.0.smithi045.stdout:op 726 completed, throughput=4.98MB/sec 2021-07-31T02:07:36.292 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-O5RpS7nKiucb4_h off=2617141 len=881701 2021-07-31T02:07:36.294 INFO:tasks.workunit.client.0.smithi045.stdout:op 727 completed, throughput=4.99MB/sec 2021-07-31T02:07:36.294 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-J4teSPwCewftwc- off=9424077 len=888301 2021-07-31T02:07:36.296 INFO:tasks.workunit.client.0.smithi045.stdout:op 728 completed, throughput=4.99MB/sec 2021-07-31T02:07:36.296 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Raz9LHDjJS2NNea off=10666798 len=1358328 2021-07-31T02:07:36.298 INFO:tasks.workunit.client.0.smithi045.stdout:op 729 completed, throughput=5MB/sec 2021-07-31T02:07:36.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:36 smithi154 conmon[34553]: cluster 2021-07-31T02:07:36.079576+0000 mgr.x (mgr.24542) 877 : cluster [DBG] pgmap v1483: 9 pgs: 9 active+clean; 127 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 3 op/s 2021-07-31T02:07:36.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:36 smithi154 conmon[34553]: 2021-07-31T02:07:36.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:36 smithi045 conmon[39718]: cluster 2021-07-31T02:07:36.079576+0000 mgr.x (mgr.24542) 877 : cluster [DBG] pgmap v1483: 9 pgs: 9 active+clean; 127 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 3 op/s 2021-07-31T02:07:36.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:36 smithi045 conmon[33655]: cluster 2021-07-31T02:07:36.079576+0000 mgr.x 2021-07-31T02:07:36.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:36 smithi045 conmon[33655]: (mgr.24542) 877 : cluster [DBG] pgmap v1483: 9 pgs: 9 active+clean; 127 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 3 op/s 2021-07-31T02:07:37.299 INFO:tasks.workunit.client.0.smithi045.stdout: 148: throughput=4.97MB/sec pending data=0 2021-07-31T02:07:37.299 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-eFfzydc3BJr1U2i off=9961997 len=500110 2021-07-31T02:07:37.300 INFO:tasks.workunit.client.0.smithi045.stdout:op 730 completed, throughput=4.97MB/sec 2021-07-31T02:07:37.301 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-wYik7cwJ3Gydf46 off=1494995 len=321714 2021-07-31T02:07:37.304 INFO:tasks.workunit.client.0.smithi045.stdout:op 731 completed, throughput=4.98MB/sec 2021-07-31T02:07:37.304 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj--ujFMx_81huQkbX off=3110261 len=175993 2021-07-31T02:07:37.305 INFO:tasks.workunit.client.0.smithi045.stdout:op 732 completed, throughput=4.98MB/sec 2021-07-31T02:07:37.305 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-6-g-x8X8jufFm9a off=879031 len=228154 2021-07-31T02:07:37.305 INFO:tasks.workunit.client.0.smithi045.stdout:op 733 completed, throughput=4.98MB/sec 2021-07-31T02:07:37.306 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-530ZToBMkZfTxMc off=3425540 len=2045640 2021-07-31T02:07:37.309 INFO:tasks.workunit.client.0.smithi045.stdout:op 734 completed, throughput=4.99MB/sec 2021-07-31T02:07:37.309 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-0b7UUKydmv9NeU4 off=153025 len=2091689 2021-07-31T02:07:37.311 INFO:tasks.workunit.client.0.smithi045.stdout:op 735 completed, throughput=5MB/sec 2021-07-31T02:07:38.312 INFO:tasks.workunit.client.0.smithi045.stdout: 149: throughput=4.97MB/sec pending data=0 2021-07-31T02:07:38.312 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-AzacIh4FpaxTG09 off=10470109 len=210610 2021-07-31T02:07:38.313 INFO:tasks.workunit.client.0.smithi045.stdout:op 736 completed, throughput=4.97MB/sec 2021-07-31T02:07:38.313 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-teHmSa6z8c3AdFk off=8167674 len=427476 2021-07-31T02:07:38.314 INFO:tasks.workunit.client.0.smithi045.stdout:op 737 completed, throughput=4.97MB/sec 2021-07-31T02:07:38.314 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-QX3tOgY-9wMhRV5 off=7558474 len=1662619 2021-07-31T02:07:38.317 INFO:tasks.workunit.client.0.smithi045.stdout:op 738 completed, throughput=4.98MB/sec 2021-07-31T02:07:38.317 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-9HygaF8I6RCB78G off=7736914 len=1549260 2021-07-31T02:07:38.321 INFO:tasks.workunit.client.0.smithi045.stdout:op 739 completed, throughput=4.99MB/sec 2021-07-31T02:07:38.321 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-a466fw4PfYwfIA1 off=1606192 len=890457 2021-07-31T02:07:38.327 INFO:tasks.workunit.client.0.smithi045.stdout:op 740 completed, throughput=5MB/sec 2021-07-31T02:07:38.328 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Y_dsBGuxUGH1mbN off=19943709 len=1680912 2021-07-31T02:07:38.329 INFO:tasks.workunit.client.0.smithi045.stdout:op 741 completed, throughput=5.01MB/sec 2021-07-31T02:07:38.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:38 smithi154 conmon[34553]: cluster 2021-07-31T02:07:38.080037+0000 mgr.x (mgr.24542) 878 : cluster [DBG] 2021-07-31T02:07:38.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:38 smithi154 conmon[34553]: pgmap v1484: 9 pgs: 9 active+clean; 127 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 3 op/s 2021-07-31T02:07:38.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:38 smithi045 conmon[33655]: cluster 2021-07-31T02:07:38.080037+0000 mgr.x (mgr.24542) 878 : 2021-07-31T02:07:38.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:38 smithi045 conmon[33655]: cluster [DBG] pgmap v1484: 9 pgs: 9 active+clean; 127 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 3 op/s 2021-07-31T02:07:38.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:38 smithi045 conmon[39718]: cluster 2021-07-31T02:07:38.080037+0000 2021-07-31T02:07:38.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:38 smithi045 conmon[39718]: mgr.x (mgr.24542) 878 : cluster [DBG] pgmap v1484: 9 pgs: 9 active+clean; 127 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 3 op/s 2021-07-31T02:07:39.330 INFO:tasks.workunit.client.0.smithi045.stdout: 150: throughput=4.98MB/sec pending data=0 2021-07-31T02:07:39.330 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-j_UYGuzFF1nI0Fu off=1201352 len=523230 2021-07-31T02:07:39.334 INFO:tasks.workunit.client.0.smithi045.stdout:op 742 completed, throughput=4.98MB/sec 2021-07-31T02:07:39.334 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-H7MlgJJLeuQlMXZ off=3436100 len=748734 2021-07-31T02:07:39.336 INFO:tasks.workunit.client.0.smithi045.stdout:op 743 completed, throughput=4.99MB/sec 2021-07-31T02:07:39.336 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-LXUjr_n0-ILUcBS off=16151213 len=1123763 2021-07-31T02:07:39.343 INFO:tasks.workunit.client.0.smithi045.stdout:op 744 completed, throughput=4.99MB/sec 2021-07-31T02:07:39.343 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-PCQWa7EPKJvkXK8 off=14200399 len=1006198 2021-07-31T02:07:39.348 INFO:tasks.workunit.client.0.smithi045.stdout:op 745 completed, throughput=5MB/sec 2021-07-31T02:07:39.348 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-FopaZLw_yVvUX74 off=62836 len=765292 2021-07-31T02:07:39.349 INFO:tasks.workunit.client.0.smithi045.stdout:op 746 completed, throughput=5MB/sec 2021-07-31T02:07:40.348 INFO:tasks.workunit.client.0.smithi045.stdout: 151: throughput=4.97MB/sec pending data=0 2021-07-31T02:07:40.349 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-nUmBTG9wqE6NT8N off=10515950 len=1144817 2021-07-31T02:07:40.356 INFO:tasks.workunit.client.0.smithi045.stdout:op 747 completed, throughput=4.98MB/sec 2021-07-31T02:07:40.356 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-YIB0_xGPhpDf0Oj off=6805524 len=1932656 2021-07-31T02:07:40.363 INFO:tasks.workunit.client.0.smithi045.stdout:op 748 completed, throughput=4.99MB/sec 2021-07-31T02:07:40.363 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-EmyerpA3vuFTJu8 off=15946321 len=909510 2021-07-31T02:07:40.368 INFO:tasks.workunit.client.0.smithi045.stdout:op 749 completed, throughput=4.99MB/sec 2021-07-31T02:07:40.368 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-IbYPcOrkwsI-RkL off=12959954 len=1913501 2021-07-31T02:07:40.378 INFO:tasks.workunit.client.0.smithi045.stdout:op 750 completed, throughput=5.01MB/sec 2021-07-31T02:07:40.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:40 smithi154 conmon[34553]: cluster 2021-07-31T02:07:40.080683+0000 mgr.x (mgr.24542) 879 : cluster 2021-07-31T02:07:40.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:40 smithi154 conmon[34553]: [DBG] pgmap v1485: 9 pgs: 9 active+clean; 127 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 3.3 MiB/s rd, 3.2 MiB/s wr, 6 op/s 2021-07-31T02:07:40.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:40 smithi045 conmon[39718]: cluster 2021-07-31T02:07:40.080683+0000 mgr.x (mgr.24542) 879 : cluster 2021-07-31T02:07:40.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:40 smithi045 conmon[39718]: [DBG] pgmap v1485: 9 pgs: 9 active+clean; 127 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 3.3 MiB/s rd, 3.2 MiB/s wr, 6 op/s 2021-07-31T02:07:40.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:40 smithi045 conmon[33655]: cluster 2021-07-31T02:07:40.080683+0000 mgr.x (mgr.24542) 879 : cluster [DBG] pgmap v1485: 9 pgs: 9 active+clean; 127 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 3.3 MiB/s rd, 3.2 MiB/s wr, 6 op/s 2021-07-31T02:07:41.380 INFO:tasks.workunit.client.0.smithi045.stdout: 152: throughput=4.97MB/sec pending data=0 2021-07-31T02:07:41.381 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-qMkvyf4AWHAsz5T off=2106113 len=1596662 2021-07-31T02:07:41.387 INFO:tasks.workunit.client.0.smithi045.stdout:op 751 completed, throughput=4.98MB/sec 2021-07-31T02:07:41.387 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-_LYl3UCP1t9VZTO off=17976523 len=1974007 2021-07-31T02:07:41.389 INFO:tasks.workunit.client.0.smithi045.stdout:op 752 completed, throughput=5MB/sec 2021-07-31T02:07:41.389 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-4LgNtFgvDttT66V off=2744905 len=2037578 2021-07-31T02:07:41.393 INFO:tasks.workunit.client.0.smithi045.stdout:op 753 completed, throughput=5.01MB/sec 2021-07-31T02:07:42.393 INFO:tasks.workunit.client.0.smithi045.stdout: 153: throughput=4.98MB/sec pending data=0 2021-07-31T02:07:42.394 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-6TxDUpiGOQlWBai off=965814 len=1624901 2021-07-31T02:07:42.397 INFO:tasks.workunit.client.0.smithi045.stdout:op 754 completed, throughput=4.99MB/sec 2021-07-31T02:07:42.397 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-wgrEg6h8Yl-KVdq off=4762454 len=1118477 2021-07-31T02:07:42.398 INFO:tasks.workunit.client.0.smithi045.stdout:op 755 completed, throughput=4.99MB/sec 2021-07-31T02:07:42.399 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-gH6cuFVZkSKit23 off=7286764 len=59084 2021-07-31T02:07:42.401 INFO:tasks.workunit.client.0.smithi045.stdout:op 756 completed, throughput=4.99MB/sec 2021-07-31T02:07:42.402 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-QmA8NSDdPuSk5Va off=898062 len=828190 2021-07-31T02:07:42.403 INFO:tasks.workunit.client.0.smithi045.stdout:op 757 completed, throughput=5MB/sec 2021-07-31T02:07:42.404 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-UCkYoTnk-Fgb6-4 off=17672848 len=552156 2021-07-31T02:07:42.407 INFO:tasks.workunit.client.0.smithi045.stdout:op 758 completed, throughput=5MB/sec 2021-07-31T02:07:43.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:42 smithi154 conmon[34553]: cluster 2021-07-31T02:07:42.081174+0000 mgr.x (mgr.24542) 880 : cluster [DBG] pgmap v1486: 9 pgs: 9 active+clean; 127 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.1 MiB/s wr, 3 op/s 2021-07-31T02:07:43.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:42 smithi045 conmon[33655]: cluster 2021-07-31T02:07:42.081174+0000 mgr.x (mgr.24542) 880 : cluster [DBG] pgmap v1486: 9 pgs: 9 active+clean; 127 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.1 MiB/s wr, 3 op/s 2021-07-31T02:07:43.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:42 smithi045 conmon[39718]: cluster 2021-07-31T02:07:42.081174+0000 mgr.x 2021-07-31T02:07:43.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:42 smithi045 conmon[39718]: (mgr.24542) 880 : cluster [DBG] pgmap v1486: 9 pgs: 9 active+clean; 127 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.1 MiB/s wr, 3 op/s 2021-07-31T02:07:43.408 INFO:tasks.workunit.client.0.smithi045.stdout: 154: throughput=4.97MB/sec pending data=0 2021-07-31T02:07:43.408 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-8izG7Xy5XEdKzgc off=7186099 len=81453 2021-07-31T02:07:43.409 INFO:tasks.workunit.client.0.smithi045.stdout:op 759 completed, throughput=4.97MB/sec 2021-07-31T02:07:43.409 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-bGd90N-Yd9WkarA off=1073561 len=234739 2021-07-31T02:07:43.412 INFO:tasks.workunit.client.0.smithi045.stdout:op 760 completed, throughput=4.97MB/sec 2021-07-31T02:07:43.412 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-KEwZ7Twq-6aaivO off=6073745 len=1787917 2021-07-31T02:07:43.415 INFO:tasks.workunit.client.0.smithi045.stdout:op 761 completed, throughput=4.98MB/sec 2021-07-31T02:07:43.415 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-CnZN4KuB94DE7ZP off=9005217 len=2009065 2021-07-31T02:07:43.419 INFO:tasks.workunit.client.0.smithi045.stdout:op 762 completed, throughput=4.99MB/sec 2021-07-31T02:07:43.419 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Fa_6DUfSax93QRR off=0 len=1845731 2021-07-31T02:07:43.422 INFO:tasks.workunit.client.0.smithi045.stdout:op 763 completed, throughput=5MB/sec 2021-07-31T02:07:44.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:07:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:07:43.542Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:07:44.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:07:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:07:43.542Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:07:44.027 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:07:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:07:43.542Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:07:44.423 INFO:tasks.workunit.client.0.smithi045.stdout: 155: throughput=4.97MB/sec pending data=0 2021-07-31T02:07:44.423 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-WOLOl9bb_YWMGwS off=2451783 len=1512871 2021-07-31T02:07:44.426 INFO:tasks.workunit.client.0.smithi045.stdout:op 764 completed, throughput=4.98MB/sec 2021-07-31T02:07:44.427 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-70VooUW9a_oCSxU off=7866514 len=275644 2021-07-31T02:07:44.429 INFO:tasks.workunit.client.0.smithi045.stdout:op 765 completed, throughput=4.98MB/sec 2021-07-31T02:07:44.430 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-k3-bhOQPuvdp5Rr off=1507932 len=1944102 2021-07-31T02:07:44.432 INFO:tasks.workunit.client.0.smithi045.stdout:op 766 completed, throughput=5MB/sec 2021-07-31T02:07:44.433 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-XGggFLT_oXlpzRF off=9274812 len=1020134 2021-07-31T02:07:44.438 INFO:tasks.workunit.client.0.smithi045.stdout:op 767 completed, throughput=5MB/sec 2021-07-31T02:07:44.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:44 smithi154 conmon[34553]: cluster 2021-07-31T02:07:44.081613+0000 mgr.x (mgr.24542) 881 : cluster [DBG] pgmap v1487: 9 pgs: 9 active+clean; 127 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 3.5 MiB/s rd, 2.9 MiB/s wr, 5 op/s 2021-07-31T02:07:44.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:44 smithi045 conmon[39718]: cluster 2021-07-31T02:07: 2021-07-31T02:07:44.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:44 smithi045 conmon[39718]: 44.081613+0000 mgr.x (mgr.24542) 881 : cluster [DBG] pgmap v1487: 9 pgs: 9 active+clean; 127 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 3.5 MiB/s rd, 2.9 MiB/s wr, 5 op/s 2021-07-31T02:07:44.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:44 smithi045 conmon[33655]: cluster 2021-07-31T02:07:44.081613+0000 mgr.x (mgr.24542) 881 : cluster [DBG] pgmap v1487: 9 pgs: 9 active+clean; 127 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 3.5 MiB/s rd, 2.9 MiB/s wr, 5 op/s 2021-07-31T02:07:45.438 INFO:tasks.workunit.client.0.smithi045.stdout: 156: throughput=4.97MB/sec pending data=0 2021-07-31T02:07:45.439 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-5xG5mDflhrJ9DS0 off=1668278 len=575765 2021-07-31T02:07:45.443 INFO:tasks.workunit.client.0.smithi045.stdout:op 768 completed, throughput=4.97MB/sec 2021-07-31T02:07:45.443 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-wRJMINvPQf2eB0p off=17757440 len=774365 2021-07-31T02:07:45.448 INFO:tasks.workunit.client.0.smithi045.stdout:op 769 completed, throughput=4.98MB/sec 2021-07-31T02:07:45.448 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-1BgbgVNT-dtML4M off=7483963 len=554329 2021-07-31T02:07:45.449 INFO:tasks.workunit.client.0.smithi045.stdout:op 770 completed, throughput=4.98MB/sec 2021-07-31T02:07:45.450 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-01c1q1JKLkp4G6R off=3342134 len=288406 2021-07-31T02:07:45.452 INFO:tasks.workunit.client.0.smithi045.stdout:op 771 completed, throughput=4.98MB/sec 2021-07-31T02:07:45.452 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-0HiydPRkRaj0Ebi off=745782 len=307409 2021-07-31T02:07:45.453 INFO:tasks.workunit.client.0.smithi045.stdout:op 772 completed, throughput=4.98MB/sec 2021-07-31T02:07:45.453 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-CbRSWTyxqb5KRTi off=8566567 len=934980 2021-07-31T02:07:45.458 INFO:tasks.workunit.client.0.smithi045.stdout:op 773 completed, throughput=4.99MB/sec 2021-07-31T02:07:45.458 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-lt7EPIiODoij3mx off=10178368 len=909533 2021-07-31T02:07:45.459 INFO:tasks.workunit.client.0.smithi045.stdout:op 774 completed, throughput=5MB/sec 2021-07-31T02:07:45.459 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-E6jWwH7GkStywrU off=8165632 len=828122 2021-07-31T02:07:45.461 INFO:tasks.workunit.client.0.smithi045.stdout:op 775 completed, throughput=5MB/sec 2021-07-31T02:07:45.462 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Jv8Q2Iq0CT8c2F6 off=15824716 len=856721 2021-07-31T02:07:45.463 INFO:tasks.workunit.client.0.smithi045.stdout:op 776 completed, throughput=5.01MB/sec 2021-07-31T02:07:46.464 INFO:tasks.workunit.client.0.smithi045.stdout: 157: throughput=4.97MB/sec pending data=0 2021-07-31T02:07:46.464 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-7YHhj9PhuSDbYCx off=33253 len=1348631 2021-07-31T02:07:46.471 INFO:tasks.workunit.client.0.smithi045.stdout:op 777 completed, throughput=4.98MB/sec 2021-07-31T02:07:46.471 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-RmVTngt5Xapef8x off=8217870 len=569608 2021-07-31T02:07:46.475 INFO:tasks.workunit.client.0.smithi045.stdout:op 778 completed, throughput=4.98MB/sec 2021-07-31T02:07:46.475 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-GiJqIbFdgcVz2c5 off=1629135 len=212084 2021-07-31T02:07:46.478 INFO:tasks.workunit.client.0.smithi045.stdout:op 779 completed, throughput=4.99MB/sec 2021-07-31T02:07:46.478 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-3Ztim0fajIHxZB8 off=3354183 len=456547 2021-07-31T02:07:46.479 INFO:tasks.workunit.client.0.smithi045.stdout:op 780 completed, throughput=4.99MB/sec 2021-07-31T02:07:46.479 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-O3a59R1-le6Ra39 off=2346448 len=1931123 2021-07-31T02:07:46.481 INFO:tasks.workunit.client.0.smithi045.stdout:op 781 completed, throughput=5MB/sec 2021-07-31T02:07:46.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:46 smithi154 conmon[34553]: cluster 2021-07-31T02:07:46.082099+0000 mgr.x (mgr.24542) 882 : cluster [DBG] pgmap v1488: 9 pgs: 9 active+clean; 127 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 2.6 MiB/s rd, 1.7 MiB/s wr, 3 op/s 2021-07-31T02:07:46.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:46 smithi154 conmon[34553]: 2021-07-31T02:07:46.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:46 smithi045 conmon[39718]: cluster 2021-07-31T02:07:46.082099+0000 mgr.x (mgr 2021-07-31T02:07:46.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:46 smithi045 conmon[39718]: .24542) 882 : cluster [DBG] pgmap v1488: 9 pgs: 9 active+clean; 127 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 2.6 MiB/s rd, 1.7 MiB/s wr, 3 op/s 2021-07-31T02:07:46.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:46 smithi045 conmon[33655]: cluster 2021-07-31T02:07:46.082099+0000 mgr.x (mgr.24542) 882 : cluster [DBG] pgmap v1488: 9 pgs: 9 active+clean; 127 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 2.6 MiB/s rd, 1.7 MiB/s wr, 3 op/s 2021-07-31T02:07:47.481 INFO:tasks.workunit.client.0.smithi045.stdout: 158: throughput=4.97MB/sec pending data=0 2021-07-31T02:07:47.482 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-AV70aJx0P0ESzJ_ off=8366567 len=437247 2021-07-31T02:07:47.485 INFO:tasks.workunit.client.0.smithi045.stdout:op 782 completed, throughput=4.97MB/sec 2021-07-31T02:07:47.485 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-yIDikZ3VKL-cb2E off=12595682 len=321026 2021-07-31T02:07:47.489 INFO:tasks.workunit.client.0.smithi045.stdout:op 783 completed, throughput=4.97MB/sec 2021-07-31T02:07:47.489 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-twfKpoKIQM-YhWK off=7849875 len=119483 2021-07-31T02:07:47.491 INFO:tasks.workunit.client.0.smithi045.stdout:op 784 completed, throughput=4.97MB/sec 2021-07-31T02:07:47.491 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-hNgwzoYWVVrSIbw off=3147040 len=1585412 2021-07-31T02:07:47.499 INFO:tasks.workunit.client.0.smithi045.stdout:op 785 completed, throughput=4.98MB/sec 2021-07-31T02:07:47.500 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Cidr-LMyKP6ZEwA off=6420024 len=897979 2021-07-31T02:07:47.501 INFO:tasks.workunit.client.0.smithi045.stdout:op 786 completed, throughput=4.99MB/sec 2021-07-31T02:07:47.501 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-KEZ3LN6EVpkuAQg off=9486403 len=310396 2021-07-31T02:07:47.505 INFO:tasks.workunit.client.0.smithi045.stdout:op 787 completed, throughput=4.99MB/sec 2021-07-31T02:07:47.505 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-esulQnO1wUazAxE off=1591923 len=985301 2021-07-31T02:07:47.507 INFO:tasks.workunit.client.0.smithi045.stdout:op 788 completed, throughput=5MB/sec 2021-07-31T02:07:47.507 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-x7lb2tyyb9WxkqH off=3628661 len=248855 2021-07-31T02:07:47.508 INFO:tasks.workunit.client.0.smithi045.stdout:op 789 completed, throughput=5MB/sec 2021-07-31T02:07:47.508 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-eAzak_Rxde5_uGH off=17413396 len=579104 2021-07-31T02:07:47.512 INFO:tasks.workunit.client.0.smithi045.stdout:op 790 completed, throughput=5MB/sec 2021-07-31T02:07:48.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:48 smithi154 conmon[34553]: cluster 2021-07-31T02:07:48.082651+0000 mgr.x (mgr.24542) 883 : cluster [DBG] pgmap v1489: 9 pgs: 9 active+clean; 127 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 2.6 MiB/s rd, 1.7 MiB/s wr, 3 op/s 2021-07-31T02:07:48.514 INFO:tasks.workunit.client.0.smithi045.stdout: 159: throughput=4.97MB/sec pending data=0 2021-07-31T02:07:48.514 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-N0OzNL2-0znPmlL off=8690274 len=1698726 2021-07-31T02:07:48.515 INFO:tasks.workunit.client.0.smithi045.stdout:op 791 completed, throughput=4.98MB/sec 2021-07-31T02:07:48.515 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-GQob3-2vqq09KCL off=8562919 len=1193734 2021-07-31T02:07:48.522 INFO:tasks.workunit.client.0.smithi045.stdout:op 792 completed, throughput=4.99MB/sec 2021-07-31T02:07:48.522 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-UGS5iC7GOR3i4yj off=10288699 len=355965 2021-07-31T02:07:48.526 INFO:tasks.workunit.client.0.smithi045.stdout:op 793 completed, throughput=4.99MB/sec 2021-07-31T02:07:48.527 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-lgSt7f-aMtUL1Wk off=6845312 len=826763 2021-07-31T02:07:48.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:48 smithi045 conmon[33655]: cluster 2021-07-31T02:07:48.082651+0000 mgr.x (mgr.24542) 883 : 2021-07-31T02:07:48.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:48 smithi045 conmon[33655]: cluster [DBG] pgmap v1489: 9 pgs: 9 active+clean; 127 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 2.6 MiB/s rd, 1.7 MiB/s wr, 3 op/s 2021-07-31T02:07:48.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:48 smithi045 conmon[39718]: cluster 2021-07-31T02:07:48.082651+0000 mgr.x ( 2021-07-31T02:07:48.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:48 smithi045 conmon[39718]: mgr.24542) 883 : cluster [DBG] pgmap v1489: 9 pgs: 9 active+clean; 127 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 2.6 MiB/s rd, 1.7 MiB/s wr, 3 op/s 2021-07-31T02:07:48.530 INFO:tasks.workunit.client.0.smithi045.stdout:op 794 completed, throughput=4.99MB/sec 2021-07-31T02:07:48.531 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-qZkT2s2kMjAVg8h off=14034616 len=2000321 2021-07-31T02:07:48.540 INFO:tasks.workunit.client.0.smithi045.stdout:op 795 completed, throughput=5MB/sec 2021-07-31T02:07:49.540 INFO:tasks.workunit.client.0.smithi045.stdout: 160: throughput=4.97MB/sec pending data=0 2021-07-31T02:07:49.541 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ny8JH0022yKP0Mx off=3661870 len=895211 2021-07-31T02:07:49.542 INFO:tasks.workunit.client.0.smithi045.stdout:op 796 completed, throughput=4.98MB/sec 2021-07-31T02:07:49.543 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-7GYb4OivI5Fhh12 off=6020812 len=1519500 2021-07-31T02:07:49.544 INFO:tasks.workunit.client.0.smithi045.stdout:op 797 completed, throughput=4.99MB/sec 2021-07-31T02:07:49.545 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-TqTIJ1H4o2BUqSQ off=4734639 len=372144 2021-07-31T02:07:49.548 INFO:tasks.workunit.client.0.smithi045.stdout:op 798 completed, throughput=4.99MB/sec 2021-07-31T02:07:49.548 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-6YuKRLfSQWlgP4B off=3947051 len=1862079 2021-07-31T02:07:49.557 INFO:tasks.workunit.client.0.smithi045.stdout:op 799 completed, throughput=5MB/sec 2021-07-31T02:07:50.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:50 smithi154 conmon[34553]: cluster 2021-07-31T02:07:50.083272+0000 mgr.x (mgr.24542) 884 : cluster [DBG] pgmap v1490: 9 pgs: 9 active+clean; 127 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 3.6 MiB/s rd, 2.8 MiB/s wr, 6 op/s 2021-07-31T02:07:50.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:50 smithi154 conmon[34553]: 2021-07-31T02:07:50.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:50 smithi045 conmon[39718]: cluster 2021-07-31T02:07:50.083272+0000 mgr.x (mgr.24542) 884 2021-07-31T02:07:50.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:50 smithi045 conmon[39718]: : cluster [DBG] pgmap v1490: 9 pgs: 9 active+clean; 127 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 3.6 MiB/s rd, 2.8 MiB/s wr, 6 op/s 2021-07-31T02:07:50.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:50 smithi045 conmon[33655]: cluster 2021-07-31T02:07:50.083272+0000 mgr.x (mgr. 2021-07-31T02:07:50.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:50 smithi045 conmon[33655]: 24542) 884 : cluster [DBG] pgmap v1490: 9 pgs: 9 active+clean; 127 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 3.6 MiB/s rd, 2.8 MiB/s wr, 6 op/s 2021-07-31T02:07:50.558 INFO:tasks.workunit.client.0.smithi045.stdout: 161: throughput=4.97MB/sec pending data=0 2021-07-31T02:07:50.558 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Gg8w6iw0csnrsul off=16607979 len=455301 2021-07-31T02:07:50.559 INFO:tasks.workunit.client.0.smithi045.stdout:op 800 completed, throughput=4.97MB/sec 2021-07-31T02:07:50.559 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-5BQqin0HNmRzu1N off=2688048 len=1674111 2021-07-31T02:07:50.567 INFO:tasks.workunit.client.0.smithi045.stdout:op 801 completed, throughput=4.98MB/sec 2021-07-31T02:07:50.567 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-f45tQEAXgOP7FZw off=13379351 len=772101 2021-07-31T02:07:50.568 INFO:tasks.workunit.client.0.smithi045.stdout:op 802 completed, throughput=4.99MB/sec 2021-07-31T02:07:50.569 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-TnXkjbphsmoJfZM off=2678168 len=1806982 2021-07-31T02:07:50.578 INFO:tasks.workunit.client.0.smithi045.stdout:op 803 completed, throughput=5MB/sec 2021-07-31T02:07:50.578 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-pd5DFONjbFWpQJc off=7220939 len=2023747 2021-07-31T02:07:50.586 INFO:tasks.workunit.client.0.smithi045.stdout:op 804 completed, throughput=5.01MB/sec 2021-07-31T02:07:51.586 INFO:tasks.workunit.client.0.smithi045.stdout: 162: throughput=4.98MB/sec pending data=0 2021-07-31T02:07:51.587 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-d3t-lH20Xe7XgQD off=5231502 len=774714 2021-07-31T02:07:51.588 INFO:tasks.workunit.client.0.smithi045.stdout:op 805 completed, throughput=4.98MB/sec 2021-07-31T02:07:51.588 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-vnXReMDshDRF532 off=2812835 len=987068 2021-07-31T02:07:51.595 INFO:tasks.workunit.client.0.smithi045.stdout:op 806 completed, throughput=4.99MB/sec 2021-07-31T02:07:51.595 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-S1bOZv9SizZ7wBp off=5590235 len=1413551 2021-07-31T02:07:51.600 INFO:tasks.workunit.client.0.smithi045.stdout:op 807 completed, throughput=5MB/sec 2021-07-31T02:07:51.601 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-IP5vu2p84LHEsHB off=2202240 len=1697809 2021-07-31T02:07:51.608 INFO:tasks.workunit.client.0.smithi045.stdout:op 808 completed, throughput=5.01MB/sec 2021-07-31T02:07:52.608 INFO:tasks.workunit.client.0.smithi045.stdout: 163: throughput=4.98MB/sec pending data=0 2021-07-31T02:07:52.609 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-UTNh45giVZI3VBR off=3983670 len=198964 2021-07-31T02:07:52.612 INFO:tasks.workunit.client.0.smithi045.stdout:op 809 completed, throughput=4.98MB/sec 2021-07-31T02:07:52.612 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Z53qZoeW5k9IJAe off=10283878 len=1911744 2021-07-31T02:07:52.624 INFO:tasks.workunit.client.0.smithi045.stdout:op 810 completed, throughput=4.99MB/sec 2021-07-31T02:07:52.625 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-_vGNpZ5QgRbyyyG off=4844606 len=709848 2021-07-31T02:07:52.625 INFO:tasks.workunit.client.0.smithi045.stdout:op 811 completed, throughput=4.99MB/sec 2021-07-31T02:07:52.626 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-0k0qPLQEmZtdlJW off=3108652 len=46473 2021-07-31T02:07:52.626 INFO:tasks.workunit.client.0.smithi045.stdout:op 812 completed, throughput=4.99MB/sec 2021-07-31T02:07:52.626 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-aa_jmupEM4PMPnl off=7653804 len=1417302 2021-07-31T02:07:52.633 INFO:tasks.workunit.client.0.smithi045.stdout:op 813 completed, throughput=5MB/sec 2021-07-31T02:07:52.635 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Rlet7D4V_uDPmsG off=2579220 len=226390 2021-07-31T02:07:52.636 INFO:tasks.workunit.client.0.smithi045.stdout:op 814 completed, throughput=5MB/sec 2021-07-31T02:07:53.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:52 smithi154 conmon[34553]: cluster 2021-07-31T02:07:52.083714+0000 mgr.x (mgr.24542) 885 : cluster [DBG] pgmap v1491: 9 pgs: 9 active+clean; 127 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 2.2 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2021-07-31T02:07:53.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:52 smithi045 conmon[33655]: cluster 2021-07-31T02:07:52.083714+0000 mgr.x (mgr.24542) 885 : cluster [DBG] pgmap v1491: 9 pgs: 9 active+clean; 127 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 2.2 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2021-07-31T02:07:53.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:52 smithi045 conmon[39718]: cluster 2021-07-31T02:07:52.083714+0000 mgr.x (mgr. 2021-07-31T02:07:53.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:52 smithi045 conmon[39718]: 24542) 885 : cluster [DBG] pgmap v1491: 9 pgs: 9 active+clean; 127 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 2.2 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2021-07-31T02:07:53.634 INFO:tasks.workunit.client.0.smithi045.stdout: 164: throughput=4.97MB/sec pending data=0 2021-07-31T02:07:53.634 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-p25MlgT3AGP7FFQ off=4257218 len=630470 2021-07-31T02:07:53.638 INFO:tasks.workunit.client.0.smithi045.stdout:op 815 completed, throughput=4.97MB/sec 2021-07-31T02:07:53.638 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-cZGtrt9WKg4q0Mn off=3425222 len=1166655 2021-07-31T02:07:53.641 INFO:tasks.workunit.client.0.smithi045.stdout:op 816 completed, throughput=4.98MB/sec 2021-07-31T02:07:53.641 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-du7Fokj-bb3AaY5 off=0 len=1176406 2021-07-31T02:07:53.643 INFO:tasks.workunit.client.0.smithi045.stdout:op 817 completed, throughput=4.99MB/sec 2021-07-31T02:07:53.643 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-b_xlEwTGdgp8fWS off=1841213 len=411930 2021-07-31T02:07:53.644 INFO:tasks.workunit.client.0.smithi045.stdout:op 818 completed, throughput=4.99MB/sec 2021-07-31T02:07:53.644 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-8LNdegSBNuwMdJF off=21464287 len=89738 2021-07-31T02:07:53.647 INFO:tasks.workunit.client.0.smithi045.stdout:op 819 completed, throughput=4.99MB/sec 2021-07-31T02:07:53.647 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-PzhmJL0IbJHLeXE off=6139249 len=423322 2021-07-31T02:07:53.651 INFO:tasks.workunit.client.0.smithi045.stdout:op 820 completed, throughput=4.99MB/sec 2021-07-31T02:07:53.651 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-B4XM5TUoP8uVrkg off=5504823 len=627748 2021-07-31T02:07:53.654 INFO:tasks.workunit.client.0.smithi045.stdout:op 821 completed, throughput=5MB/sec 2021-07-31T02:07:53.655 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Ly-ROr9t-XXqS9M off=22012284 len=2019064 2021-07-31T02:07:53.657 INFO:tasks.workunit.client.0.smithi045.stdout:op 822 completed, throughput=5.01MB/sec 2021-07-31T02:07:54.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:07:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:07:53.543Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:07:54.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:07:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:07:53.543Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:07:54.027 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:07:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:07:53.543Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:07:54.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:54 smithi154 conmon[34553]: cluster 2021-07-31T02:07:54.084296+0000 mgr.x (mgr.24542) 886 : cluster [DBG] pgmap v1492: 9 pgs: 9 active+clean; 127 GiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 3.0 MiB/s rd, 3.3 MiB/s wr, 6 op/s 2021-07-31T02:07:54.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:54 smithi045 conmon[33655]: cluster 2021-07-31T02:07:54.084296+0000 mgr.x (mgr. 2021-07-31T02:07:54.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:54 smithi045 conmon[33655]: 24542) 886 : cluster [DBG] pgmap v1492: 9 pgs: 9 active+clean; 127 GiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 3.0 MiB/s rd, 3.3 MiB/s wr, 6 op/s 2021-07-31T02:07:54.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:54 smithi045 conmon[39718]: cluster 2021-07-31T02:07:54.084296+0000 mgr.x (mgr.24542) 886 : cluster [DBG] pgmap v1492: 9 pgs: 9 active+clean; 127 GiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 3.0 MiB/s rd, 3.3 MiB/s wr, 6 op/s 2021-07-31T02:07:54.657 INFO:tasks.workunit.client.0.smithi045.stdout: 165: throughput=4.98MB/sec pending data=0 2021-07-31T02:07:54.658 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-RqRIy09u5pTWbBq off=4910876 len=1073843 2021-07-31T02:07:54.664 INFO:tasks.workunit.client.0.smithi045.stdout:op 823 completed, throughput=4.98MB/sec 2021-07-31T02:07:54.664 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-csP3bTcx86hYOQW off=140320 len=652007 2021-07-31T02:07:54.666 INFO:tasks.workunit.client.0.smithi045.stdout:op 824 completed, throughput=4.99MB/sec 2021-07-31T02:07:54.667 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-VehwZ3xXjmZUjeh off=8480744 len=1161425 2021-07-31T02:07:54.674 INFO:tasks.workunit.client.0.smithi045.stdout:op 825 completed, throughput=4.99MB/sec 2021-07-31T02:07:54.674 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-mf6flpTafxvorF2 off=5011062 len=1105147 2021-07-31T02:07:54.679 INFO:tasks.workunit.client.0.smithi045.stdout:op 826 completed, throughput=5MB/sec 2021-07-31T02:07:54.679 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-h35_lKd4LI818ry off=13310824 len=1070946 2021-07-31T02:07:54.680 INFO:tasks.workunit.client.0.smithi045.stdout:op 827 completed, throughput=5.01MB/sec 2021-07-31T02:07:55.680 INFO:tasks.workunit.client.0.smithi045.stdout: 166: throughput=4.98MB/sec pending data=0 2021-07-31T02:07:55.681 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-smR2g0MMQbHjgCl off=316915 len=1744568 2021-07-31T02:07:55.682 INFO:tasks.workunit.client.0.smithi045.stdout:op 828 completed, throughput=4.99MB/sec 2021-07-31T02:07:55.683 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-cHOTssXsMLNEYPU off=5728825 len=1949902 2021-07-31T02:07:55.684 INFO:tasks.workunit.client.0.smithi045.stdout:op 829 completed, throughput=5MB/sec 2021-07-31T02:07:55.685 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-SFS5IW2tW_G9HRd off=1792958 len=690982 2021-07-31T02:07:55.686 INFO:tasks.workunit.client.0.smithi045.stdout:op 830 completed, throughput=5MB/sec 2021-07-31T02:07:56.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:56 smithi154 conmon[34553]: cluster 2021-07-31T02:07:56.084846+0000 mgr.x (mgr.24542) 887 : cluster [DBG] pgmap v1493: 9 pgs: 9 active+clean; 127 GiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2021-07-31T02:07:56.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:56 smithi154 conmon[34553]: 2021-07-31T02:07:56.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:56 smithi045 conmon[33655]: cluster 2021-07-31T02:07:56.084846+0000 mgr.x (mgr.24542) 887 2021-07-31T02:07:56.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:56 smithi045 conmon[33655]: : cluster [DBG] pgmap v1493: 9 pgs: 9 active+clean; 127 GiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2021-07-31T02:07:56.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:56 smithi045 conmon[39718]: cluster 2021-07-31T02:07:56.084846+0000 mgr.x ( 2021-07-31T02:07:56.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:56 smithi045 conmon[39718]: mgr.24542) 887 : cluster [DBG] pgmap v1493: 9 pgs: 9 active+clean; 127 GiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2021-07-31T02:07:56.687 INFO:tasks.workunit.client.0.smithi045.stdout: 167: throughput=4.97MB/sec pending data=0 2021-07-31T02:07:56.687 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-B_h6oK1qe7BSpyj off=1431455 len=351792 2021-07-31T02:07:56.691 INFO:tasks.workunit.client.0.smithi045.stdout:op 831 completed, throughput=4.97MB/sec 2021-07-31T02:07:56.691 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-NVSdX0Z3I1YhBAV off=6920473 len=210010 2021-07-31T02:07:56.692 INFO:tasks.workunit.client.0.smithi045.stdout:op 832 completed, throughput=4.97MB/sec 2021-07-31T02:07:56.692 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-of97sWxHrab1Xav off=16122078 len=1603892 2021-07-31T02:07:56.700 INFO:tasks.workunit.client.0.smithi045.stdout:op 833 completed, throughput=4.98MB/sec 2021-07-31T02:07:56.700 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-x09dDi2bUZ-BTT0 off=4330780 len=402737 2021-07-31T02:07:56.701 INFO:tasks.workunit.client.0.smithi045.stdout:op 834 completed, throughput=4.99MB/sec 2021-07-31T02:07:56.701 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj--yPNkJ0ChjdVIWs off=14353184 len=681392 2021-07-31T02:07:56.702 INFO:tasks.workunit.client.0.smithi045.stdout:op 835 completed, throughput=4.99MB/sec 2021-07-31T02:07:56.702 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ZE8uW0grAnLbZmG off=23221185 len=1511341 2021-07-31T02:07:56.710 INFO:tasks.workunit.client.0.smithi045.stdout:op 836 completed, throughput=5MB/sec 2021-07-31T02:07:56.711 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-YKUQT1_NnigttiD off=1166119 len=30276 2021-07-31T02:07:56.712 INFO:tasks.workunit.client.0.smithi045.stdout:op 837 completed, throughput=5MB/sec 2021-07-31T02:07:56.713 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ENzYzdSVQOHEPOX off=9914074 len=1580428 2021-07-31T02:07:56.719 INFO:tasks.workunit.client.0.smithi045.stdout:op 838 completed, throughput=5.01MB/sec 2021-07-31T02:07:57.720 INFO:tasks.workunit.client.0.smithi045.stdout: 168: throughput=4.98MB/sec pending data=0 2021-07-31T02:07:57.720 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-NNjI4Cua4QleFMs off=2663177 len=1022407 2021-07-31T02:07:57.721 INFO:tasks.workunit.client.0.smithi045.stdout:op 839 completed, throughput=4.98MB/sec 2021-07-31T02:07:57.721 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Oeh8MzZB6hUTWfY off=755481 len=751251 2021-07-31T02:07:57.726 INFO:tasks.workunit.client.0.smithi045.stdout:op 840 completed, throughput=4.99MB/sec 2021-07-31T02:07:57.727 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-HnD_unClsLag52q off=6227181 len=1008672 2021-07-31T02:07:57.727 INFO:tasks.workunit.client.0.smithi045.stdout:op 841 completed, throughput=4.99MB/sec 2021-07-31T02:07:57.728 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-OVW3VkzmQDPXBjx off=1443646 len=1876494 2021-07-31T02:07:57.736 INFO:tasks.workunit.client.0.smithi045.stdout:op 842 completed, throughput=5MB/sec 2021-07-31T02:07:58.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:07:58 smithi154 conmon[34553]: cluster 2021-07-31T02:07:58.085405+0000 mgr.x (mgr.24542) 888 : cluster [DBG] pgmap v1494: 9 pgs: 9 active+clean; 127 GiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2021-07-31T02:07:58.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:58 smithi045 conmon[33655]: cluster 2021-07-31T02:07:58 2021-07-31T02:07:58.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:07:58 smithi045 conmon[33655]: .085405+0000 mgr.x (mgr.24542) 888 : cluster [DBG] pgmap v1494: 9 pgs: 9 active+clean; 127 GiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2021-07-31T02:07:58.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:58 smithi045 conmon[39718]: cluster 2021-07-31T02:07:58.085405+0000 mgr.x (mgr 2021-07-31T02:07:58.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:07:58 smithi045 conmon[39718]: .24542) 888 : cluster [DBG] pgmap v1494: 9 pgs: 9 active+clean; 127 GiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2021-07-31T02:07:58.737 INFO:tasks.workunit.client.0.smithi045.stdout: 169: throughput=4.97MB/sec pending data=0 2021-07-31T02:07:58.737 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-EvIehX-O1ew71jF off=738307 len=1092985 2021-07-31T02:07:58.744 INFO:tasks.workunit.client.0.smithi045.stdout:op 843 completed, throughput=4.98MB/sec 2021-07-31T02:07:58.745 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ZXRPM1qfWq2x1tK off=3706858 len=1026866 2021-07-31T02:07:58.745 INFO:tasks.workunit.client.0.smithi045.stdout:op 844 completed, throughput=4.98MB/sec 2021-07-31T02:07:58.746 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-WLEWKDMo51WQ1kD off=10842036 len=1759888 2021-07-31T02:07:58.747 INFO:tasks.workunit.client.0.smithi045.stdout:op 845 completed, throughput=4.99MB/sec 2021-07-31T02:07:58.747 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-7v3KyJF-23liV9C off=325796 len=1608423 2021-07-31T02:07:58.755 INFO:tasks.workunit.client.0.smithi045.stdout:op 846 completed, throughput=5MB/sec 2021-07-31T02:07:59.755 INFO:tasks.workunit.client.0.smithi045.stdout: 170: throughput=4.97MB/sec pending data=0 2021-07-31T02:07:59.755 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-bYYQVXeXlcqg-d8 off=14560676 len=1884916 2021-07-31T02:07:59.758 INFO:tasks.workunit.client.0.smithi045.stdout:op 847 completed, throughput=4.98MB/sec 2021-07-31T02:07:59.759 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-cGO9xGQX3rLmUkm off=16355179 len=1711082 2021-07-31T02:07:59.761 INFO:tasks.workunit.client.0.smithi045.stdout:op 848 completed, throughput=4.99MB/sec 2021-07-31T02:07:59.761 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-jeqQI2hizFqrPZe off=17267373 len=7486 2021-07-31T02:07:59.763 INFO:tasks.workunit.client.0.smithi045.stdout:op 849 completed, throughput=4.99MB/sec 2021-07-31T02:07:59.764 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-cHb5kdIz50evCSU off=733790 len=266004 2021-07-31T02:07:59.765 INFO:tasks.workunit.client.0.smithi045.stdout:op 850 completed, throughput=5MB/sec 2021-07-31T02:07:59.765 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-_ZkiWSOljmcCIdm off=797073 len=1217004 2021-07-31T02:07:59.767 INFO:tasks.workunit.client.0.smithi045.stdout:op 851 completed, throughput=5MB/sec 2021-07-31T02:08:00.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:00 smithi154 conmon[34553]: cluster 2021-07-31T02:08:00.085860+0000 mgr.x (mgr.24542) 889 : cluster [DBG] pgmap v1495: 9 pgs: 9 active+clean; 127 GiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 2.8 MiB/s rd, 3.5 MiB/s wr, 6 op/s 2021-07-31T02:08:00.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:00 smithi154 conmon[34553]: 2021-07-31T02:08:00.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:00 smithi045 conmon[33655]: cluster 2021-07-31T02:08: 2021-07-31T02:08:00.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:00 smithi045 conmon[33655]: 00.085860+0000 mgr.x (mgr.24542) 889 : cluster [DBG] pgmap v1495: 9 pgs: 9 active+clean; 127 GiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 2.8 MiB/s rd, 3.5 MiB/s wr, 6 op/s 2021-07-31T02:08:00.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:00 smithi045 conmon[39718]: cluster 2021-07-31T02:08:00.085860+0000 mgr.x ( 2021-07-31T02:08:00.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:00 smithi045 conmon[39718]: mgr.24542) 889 : cluster [DBG] pgmap v1495: 9 pgs: 9 active+clean; 127 GiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 2.8 MiB/s rd, 3.5 MiB/s wr, 6 op/s 2021-07-31T02:08:00.767 INFO:tasks.workunit.client.0.smithi045.stdout: 171: throughput=4.97MB/sec pending data=0 2021-07-31T02:08:00.767 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-wHbRhrLwoL-a6vL off=2057942 len=811096 2021-07-31T02:08:00.772 INFO:tasks.workunit.client.0.smithi045.stdout:op 852 completed, throughput=4.98MB/sec 2021-07-31T02:08:00.772 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-4DdPykWD2kCHfku off=5631005 len=1310739 2021-07-31T02:08:00.779 INFO:tasks.workunit.client.0.smithi045.stdout:op 853 completed, throughput=4.98MB/sec 2021-07-31T02:08:00.779 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-GuiAlH5fgIXjG1V off=1271313 len=1761289 2021-07-31T02:08:00.788 INFO:tasks.workunit.client.0.smithi045.stdout:op 854 completed, throughput=4.99MB/sec 2021-07-31T02:08:00.788 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-RMj-1d5jp8kgjIG off=4330493 len=454527 2021-07-31T02:08:00.791 INFO:tasks.workunit.client.0.smithi045.stdout:op 855 completed, throughput=5MB/sec 2021-07-31T02:08:00.791 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-maTblfFXZsomL_f off=1516 len=1318310 2021-07-31T02:08:00.797 INFO:tasks.workunit.client.0.smithi045.stdout:op 856 completed, throughput=5MB/sec 2021-07-31T02:08:01.798 INFO:tasks.workunit.client.0.smithi045.stdout: 172: throughput=4.97MB/sec pending data=0 2021-07-31T02:08:01.799 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-M0avjlK-7uJk1QF off=7042800 len=1770121 2021-07-31T02:08:01.801 INFO:tasks.workunit.client.0.smithi045.stdout:op 857 completed, throughput=4.98MB/sec 2021-07-31T02:08:01.801 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-QsTCu3IDfuZs3Ba off=16627960 len=1992212 2021-07-31T02:08:01.814 INFO:tasks.workunit.client.0.smithi045.stdout:op 858 completed, throughput=4.99MB/sec 2021-07-31T02:08:01.815 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-PHzmXfny4cRKUvP off=18370883 len=763291 2021-07-31T02:08:01.815 INFO:tasks.workunit.client.0.smithi045.stdout:op 859 completed, throughput=5MB/sec 2021-07-31T02:08:01.816 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-dzymKn9ocR72nBT off=464954 len=1991987 2021-07-31T02:08:01.824 INFO:tasks.workunit.client.0.smithi045.stdout:op 860 completed, throughput=5.01MB/sec 2021-07-31T02:08:02.824 INFO:tasks.workunit.client.0.smithi045.stdout: 173: throughput=4.98MB/sec pending data=0 2021-07-31T02:08:02.825 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Xtamv6RDe6E9JKC off=9907768 len=1460244 2021-07-31T02:08:02.833 INFO:tasks.workunit.client.0.smithi045.stdout:op 861 completed, throughput=4.99MB/sec 2021-07-31T02:08:02.833 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-gT65OXmMI3BRDqr off=92188 len=1062016 2021-07-31T02:08:02.834 INFO:tasks.workunit.client.0.smithi045.stdout:op 862 completed, throughput=4.99MB/sec 2021-07-31T02:08:02.834 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-LPXFbm1TwlkVhr7 off=17299424 len=965681 2021-07-31T02:08:02.840 INFO:tasks.workunit.client.0.smithi045.stdout:op 863 completed, throughput=5MB/sec 2021-07-31T02:08:02.840 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-GBMGbkJ0JLkysLO off=8982594 len=380282 2021-07-31T02:08:02.843 INFO:tasks.workunit.client.0.smithi045.stdout:op 864 completed, throughput=5MB/sec 2021-07-31T02:08:03.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:02 smithi154 conmon[34553]: cluster 2021-07-31T02:08:02.086364+0000 mgr.x (mgr.24542) 890 : cluster [DBG] pgmap v1496: 9 pgs: 9 active+clean; 127 GiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2021-07-31T02:08:03.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:02 smithi045 conmon[33655]: cluster 2021-07-31T02:08:02.086364+0000 mgr.x (mgr.24542) 890 : cluster [DBG] pgmap v1496: 9 pgs: 9 active+clean; 127 GiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2021-07-31T02:08:03.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:02 smithi045 conmon[39718]: cluster 2021-07-31T02:08:02.086364+0000 2021-07-31T02:08:03.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:02 smithi045 conmon[39718]: mgr.x (mgr.24542) 890 : cluster [DBG] pgmap v1496: 9 pgs: 9 active+clean; 127 GiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2021-07-31T02:08:03.576 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:08:03.843 INFO:tasks.workunit.client.0.smithi045.stdout: 174: throughput=4.97MB/sec pending data=0 2021-07-31T02:08:03.843 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-n1axjKl9fBQf7gK off=4027618 len=2055165 2021-07-31T02:08:03.845 INFO:tasks.workunit.client.0.smithi045.stdout:op 865 completed, throughput=4.98MB/sec 2021-07-31T02:08:03.845 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Stqc6EYogFr_Pk8 off=3923875 len=190685 2021-07-31T02:08:03.846 INFO:tasks.workunit.client.0.smithi045.stdout:op 866 completed, throughput=4.98MB/sec 2021-07-31T02:08:03.846 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-oSnEWN8tqdCYqJu off=9165441 len=323877 2021-07-31T02:08:03.847 INFO:tasks.workunit.client.0.smithi045.stdout:op 867 completed, throughput=4.99MB/sec 2021-07-31T02:08:03.847 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-UvRTMALg5jt51PF off=4744567 len=1887238 2021-07-31T02:08:03.852 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:08:03.852 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 22m ago 23m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:08:03.852 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 20m ago 23m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:08:03.852 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 20m ago 26m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:08:03.853 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 22m ago 29m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:08:03.853 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 22m ago 29m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:08:03.853 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 20m ago 27m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:08:03.853 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 22m ago 27m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:08:03.854 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 22m ago 23m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:08:03.854 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 20m ago 23m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:08:03.854 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 22m ago 26m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:08:03.854 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 22m ago 26m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:08:03.855 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 22m ago 26m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:08:03.855 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 22m ago 25m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:08:03.855 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 20m ago 25m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:08:03.855 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 20m ago 25m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:08:03.855 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 20m ago 24m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:08:03.856 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 20m ago 24m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:08:03.856 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 20m ago 24m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:08:03.857 INFO:tasks.workunit.client.0.smithi045.stdout:op 868 completed, throughput=5MB/sec 2021-07-31T02:08:03.857 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj--MwFyvUE5mdSPlN off=1459011 len=1740823 2021-07-31T02:08:03.859 INFO:tasks.workunit.client.0.smithi045.stdout:op 869 completed, throughput=5.01MB/sec 2021-07-31T02:08:04.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:08:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:08:03.543Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:08:04.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:08:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:08:03.543Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:08:04.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:08:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:08:03.543Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:08:04.191 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:08:04.191 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:08:04.192 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:08:04.192 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:08:04.192 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:08:04.192 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:08:04.193 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:08:04.193 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:08:04.193 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:08:04.194 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:08:04.194 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:08:04.194 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:08:04.194 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:08:04.195 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:08:04.195 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:08:04.195 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:08:04.195 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:08:04.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:04 smithi154 conmon[34553]: audit 2021-07-31T02:08:03.564694+0000 mgr.x (mgr.24542) 891 : audit [DBG] from='client.25732 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:08:04.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:04 smithi154 conmon[34553]: audit 2021-07- 2021-07-31T02:08:04.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:04 smithi154 conmon[34553]: 31T02:08:03.842423+0000 mgr.x (mgr.24542) 892 : audit [DBG] from='client.15831 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:08:04.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:04 smithi154 conmon[34553]: cluster 2021-07-31T02:08:04.086869+0000 mgr.x (mgr.24542) 893 : cluster [DBG] pgmap v1497: 9 pgs: 9 active+clean; 127 GiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 2.8 MiB/s rd, 3.4 MiB/s wr, 6 op/s 2021-07-31T02:08:04.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:04 smithi045 conmon[39718]: audit 2021-07-31T02:08:03 2021-07-31T02:08:04.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:04 smithi045 conmon[39718]: .564694+0000 mgr.x (mgr.24542) 891 : audit [DBG] from='client.25732 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:08:04.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:04 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T02:08:04.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:04 smithi045 conmon[39718]: 02:08:03.842423+0000 mgr.x (mgr. 2021-07-31T02:08:04.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:04 smithi045 conmon[39718]: 24542) 892 : audit [DBG] from='client.15831 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:08:04.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:04 smithi045 conmon[39718]: cluster 2021-07-31T02:08:04.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:04 smithi045 conmon[39718]: 2021-07-31T02:08:04.086869 2021-07-31T02:08:04.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:04 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 893 : cluster [DBG] 2021-07-31T02:08:04.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:04 smithi045 conmon[39718]: pgmap v1497: 9 pgs: 9 active+clean; 127 GiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 2.8 MiB/s rd, 3.4 MiB/s wr, 6 op/s 2021-07-31T02:08:04.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:04 smithi045 conmon[33655]: audit 2021-07-31T02:08:03.564694+0000 mgr.x (mgr.24542 2021-07-31T02:08:04.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:04 smithi045 conmon[33655]: ) 891 : audit [DBG] from='client.25732 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:08:04.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:04 smithi045 conmon[33655]: audit 2021-07-31T02:08:03.842423+0000 mgr.x (mgr.24542) 892 : audit [DBG] from='client.15831 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:08:04.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:04 smithi045 conmon[33655]: 2021-07-31T02:08:04.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:04 smithi045 conmon[33655]: cluster 2021-07-31T02 2021-07-31T02:08:04.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:04 smithi045 conmon[33655]: :08:04.086869+0000 mgr.x (mgr 2021-07-31T02:08:04.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:04 smithi045 conmon[33655]: .24542) 893 : cluster [DBG] pgmap v1497: 9 pgs: 9 active+clean; 127 GiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 2.8 MiB/s rd, 3.4 MiB/s wr, 6 op/s 2021-07-31T02:08:04.859 INFO:tasks.workunit.client.0.smithi045.stdout: 175: throughput=4.98MB/sec pending data=0 2021-07-31T02:08:04.860 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-52gBoj5t15yseAR off=24280422 len=1235830 2021-07-31T02:08:04.866 INFO:tasks.workunit.client.0.smithi045.stdout:op 870 completed, throughput=4.98MB/sec 2021-07-31T02:08:04.866 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-sLtMf7SSYVIMZfo off=14103151 len=717217 2021-07-31T02:08:04.867 INFO:tasks.workunit.client.0.smithi045.stdout:op 871 completed, throughput=4.99MB/sec 2021-07-31T02:08:04.868 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-g0KcH_bjZkvz7Qk off=14815410 len=189141 2021-07-31T02:08:04.871 INFO:tasks.workunit.client.0.smithi045.stdout:op 872 completed, throughput=4.99MB/sec 2021-07-31T02:08:04.871 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-l7UZa-TZPQ_Eu37 off=2527432 len=246620 2021-07-31T02:08:04.874 INFO:tasks.workunit.client.0.smithi045.stdout:op 873 completed, throughput=4.99MB/sec 2021-07-31T02:08:04.874 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-jTmjfRn1j9usrZx off=17504222 len=749786 2021-07-31T02:08:04.878 INFO:tasks.workunit.client.0.smithi045.stdout:op 874 completed, throughput=4.99MB/sec 2021-07-31T02:08:04.878 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-H4SjosD7m9vP9ao off=979498 len=216880 2021-07-31T02:08:04.879 INFO:tasks.workunit.client.0.smithi045.stdout:op 875 completed, throughput=5MB/sec 2021-07-31T02:08:04.880 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-zLCwcM3azzzuksn off=1366686 len=1106136 2021-07-31T02:08:04.885 INFO:tasks.workunit.client.0.smithi045.stdout:op 876 completed, throughput=5MB/sec 2021-07-31T02:08:05.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:05 smithi154 conmon[34553]: audit 2021-07-31T02:08:04.191208+0000 mon.c (mon.1) 77 : audit [DBG] from='client.? 172.21.15.45:0/1366697039' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:08:05.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:05 smithi045 conmon[33655]: audit 2021-07-31T02:08:04.191208+0000 mon.c (mon.1) 77 : audit [DBG] from='client.? 172.21.15.45:0/1366697039' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:08:05.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:05 smithi045 conmon[39718]: audit 2021-07-31T02:08:04.191208+0000 2021-07-31T02:08:05.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:05 smithi045 conmon[39718]: mon.c (mon.1) 77 : audit [DBG] from='client.? 172.21.15.45:0/1366697039' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:08:05.885 INFO:tasks.workunit.client.0.smithi045.stdout: 176: throughput=4.97MB/sec pending data=0 2021-07-31T02:08:05.885 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-fWu6OVPc2rIfbkP off=10913452 len=573589 2021-07-31T02:08:05.886 INFO:tasks.workunit.client.0.smithi045.stdout:op 877 completed, throughput=4.98MB/sec 2021-07-31T02:08:05.887 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-oupZ8hEyNI6iZOO off=833429 len=1663226 2021-07-31T02:08:05.888 INFO:tasks.workunit.client.0.smithi045.stdout:op 878 completed, throughput=4.98MB/sec 2021-07-31T02:08:05.889 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ZihpqmZq5i_OZGA off=9671889 len=675029 2021-07-31T02:08:05.893 INFO:tasks.workunit.client.0.smithi045.stdout:op 879 completed, throughput=4.99MB/sec 2021-07-31T02:08:05.894 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-144Jq1UIB80iiJM off=7349893 len=1220051 2021-07-31T02:08:05.899 INFO:tasks.workunit.client.0.smithi045.stdout:op 880 completed, throughput=4.99MB/sec 2021-07-31T02:08:05.900 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-yc0mFR4-STRCZKr off=3126896 len=1311474 2021-07-31T02:08:05.907 INFO:tasks.workunit.client.0.smithi045.stdout:op 881 completed, throughput=5MB/sec 2021-07-31T02:08:06.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:06 smithi154 conmon[34553]: cluster 2021-07-31T02:08:06.087422+0000 mgr.x (mgr.24542) 894 : cluster [DBG] pgmap v1498: 9 pgs: 9 active+clean; 127 GiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.1 MiB/s wr, 3 op/s 2021-07-31T02:08:06.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:06 smithi045 conmon[33655]: cluster 2021-07-31T02:08:06.087422+0000 mgr.x (mgr.24542) 894 : cluster 2021-07-31T02:08:06.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:06 smithi045 conmon[33655]: [DBG] pgmap v1498: 9 pgs: 9 active+clean; 127 GiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.1 MiB/s wr, 3 op/s 2021-07-31T02:08:06.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:06 smithi045 conmon[39718]: cluster 2021-07-31T02:08:06.087422+0000 mgr.x 2021-07-31T02:08:06.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:06 smithi045 conmon[39718]: (mgr.24542) 894 : cluster [DBG] pgmap v1498: 9 pgs: 9 active+clean; 127 GiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.1 MiB/s wr, 3 op/s 2021-07-31T02:08:06.907 INFO:tasks.workunit.client.0.smithi045.stdout: 177: throughput=4.97MB/sec pending data=0 2021-07-31T02:08:06.908 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-hboWcFVVLDbM3wG off=9947177 len=2022376 2021-07-31T02:08:06.919 INFO:tasks.workunit.client.0.smithi045.stdout:op 882 completed, throughput=4.98MB/sec 2021-07-31T02:08:06.920 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-BITVg8S21xIHV-B off=2662907 len=1116466 2021-07-31T02:08:06.925 INFO:tasks.workunit.client.0.smithi045.stdout:op 883 completed, throughput=4.99MB/sec 2021-07-31T02:08:06.925 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-kRn71Fzi81qmnDR off=2635674 len=473825 2021-07-31T02:08:06.926 INFO:tasks.workunit.client.0.smithi045.stdout:op 884 completed, throughput=4.99MB/sec 2021-07-31T02:08:06.926 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-EUzebIosRa1XtQI off=10068275 len=152741 2021-07-31T02:08:06.929 INFO:tasks.workunit.client.0.smithi045.stdout:op 885 completed, throughput=4.99MB/sec 2021-07-31T02:08:06.929 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-CgoMlzc_FbEjVP7 off=13249427 len=207367 2021-07-31T02:08:06.929 INFO:tasks.workunit.client.0.smithi045.stdout:op 886 completed, throughput=4.99MB/sec 2021-07-31T02:08:06.930 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-KOx2y_6sBb6ynSv off=4782009 len=71207 2021-07-31T02:08:06.932 INFO:tasks.workunit.client.0.smithi045.stdout:op 887 completed, throughput=4.99MB/sec 2021-07-31T02:08:06.932 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-wF-srTnJifucTT7 off=829655 len=8018 2021-07-31T02:08:06.934 INFO:tasks.workunit.client.0.smithi045.stdout:op 888 completed, throughput=4.99MB/sec 2021-07-31T02:08:06.934 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-7fEtLycdyUT9GHJ off=15677496 len=357973 2021-07-31T02:08:06.937 INFO:tasks.workunit.client.0.smithi045.stdout:op 889 completed, throughput=5MB/sec 2021-07-31T02:08:06.937 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-UdQXXqh3zrcJB8j off=5348532 len=88477 2021-07-31T02:08:06.937 INFO:tasks.workunit.client.0.smithi045.stdout:op 890 completed, throughput=5MB/sec 2021-07-31T02:08:06.938 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-qtKYFUeHRkZoE5_ off=20822360 len=1351493 2021-07-31T02:08:06.945 INFO:tasks.workunit.client.0.smithi045.stdout:op 891 completed, throughput=5MB/sec 2021-07-31T02:08:07.945 INFO:tasks.workunit.client.0.smithi045.stdout: 178: throughput=4.98MB/sec pending data=0 2021-07-31T02:08:07.945 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ldGSuWju5o8vsW- off=8387921 len=15698 2021-07-31T02:08:07.947 INFO:tasks.workunit.client.0.smithi045.stdout:op 892 completed, throughput=4.98MB/sec 2021-07-31T02:08:07.948 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-YGakDjpjoKNl2Bx off=0 len=1098803 2021-07-31T02:08:07.949 INFO:tasks.workunit.client.0.smithi045.stdout:op 893 completed, throughput=4.98MB/sec 2021-07-31T02:08:07.949 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ym-6Gj3VhsbpnoA off=8366492 len=241448 2021-07-31T02:08:07.953 INFO:tasks.workunit.client.0.smithi045.stdout:op 894 completed, throughput=4.98MB/sec 2021-07-31T02:08:07.953 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-2y0-x-jjYCdlptE off=6309429 len=1219677 2021-07-31T02:08:07.959 INFO:tasks.workunit.client.0.smithi045.stdout:op 895 completed, throughput=4.99MB/sec 2021-07-31T02:08:07.959 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-o8t3S0uC1yhEOJh off=187305 len=1153861 2021-07-31T02:08:07.961 INFO:tasks.workunit.client.0.smithi045.stdout:op 896 completed, throughput=5MB/sec 2021-07-31T02:08:07.962 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-W2yFz3NmcN3kw7I off=33331 len=1504231 2021-07-31T02:08:07.963 INFO:tasks.workunit.client.0.smithi045.stdout:op 897 completed, throughput=5MB/sec 2021-07-31T02:08:08.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:08 smithi154 conmon[34553]: cluster 2021-07-31T02:08:08.087874+0000 mgr.x (mgr.24542) 895 : cluster [DBG] pgmap v1499: 9 pgs: 9 active+clean; 127 GiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.1 MiB/s wr, 3 op/s 2021-07-31T02:08:08.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:08 smithi045 conmon[33655]: cluster 2021-07-31T02:08:08.087874+0000 mgr.x (mgr 2021-07-31T02:08:08.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:08 smithi045 conmon[33655]: .24542) 895 : cluster [DBG] pgmap v1499: 9 pgs: 9 active+clean; 127 GiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.1 MiB/s wr, 3 op/s 2021-07-31T02:08:08.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:08 smithi045 conmon[39718]: cluster 2021-07-31T02:08:08.087874+0000 mgr.x ( 2021-07-31T02:08:08.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:08 smithi045 conmon[39718]: mgr.24542) 895 : cluster [DBG] pgmap v1499: 9 pgs: 9 active+clean; 127 GiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.1 MiB/s wr, 3 op/s 2021-07-31T02:08:08.963 INFO:tasks.workunit.client.0.smithi045.stdout: 179: throughput=4.98MB/sec pending data=0 2021-07-31T02:08:08.963 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-B-Ju4FXF-uIvj3n off=5586711 len=1361206 2021-07-31T02:08:08.966 INFO:tasks.workunit.client.0.smithi045.stdout:op 898 completed, throughput=4.98MB/sec 2021-07-31T02:08:08.966 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-PUuYZYYTIOOl5Fj off=15052613 len=575082 2021-07-31T02:08:08.970 INFO:tasks.workunit.client.0.smithi045.stdout:op 899 completed, throughput=4.99MB/sec 2021-07-31T02:08:08.971 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-zQxiov2JIfNAmNy off=3298576 len=259582 2021-07-31T02:08:08.973 INFO:tasks.workunit.client.0.smithi045.stdout:op 900 completed, throughput=4.99MB/sec 2021-07-31T02:08:08.973 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-hW002TzO8Wj8KLa off=5075211 len=1805233 2021-07-31T02:08:08.982 INFO:tasks.workunit.client.0.smithi045.stdout:op 901 completed, throughput=5MB/sec 2021-07-31T02:08:08.982 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-JR7Ox1avPE-Scum off=16660519 len=93341 2021-07-31T02:08:08.983 INFO:tasks.workunit.client.0.smithi045.stdout:op 902 completed, throughput=5MB/sec 2021-07-31T02:08:08.983 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj--uxn4rZnieSUxa6 off=1671159 len=721655 2021-07-31T02:08:08.984 INFO:tasks.workunit.client.0.smithi045.stdout:op 903 completed, throughput=5MB/sec 2021-07-31T02:08:09.985 INFO:tasks.workunit.client.0.smithi045.stdout: 180: throughput=4.97MB/sec pending data=0 2021-07-31T02:08:09.986 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-W2vnWK9OVqavYjU off=17239697 len=1365936 2021-07-31T02:08:09.993 INFO:tasks.workunit.client.0.smithi045.stdout:op 904 completed, throughput=4.98MB/sec 2021-07-31T02:08:09.994 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-w7YlkJGJrn0CSLF off=5482117 len=881118 2021-07-31T02:08:09.997 INFO:tasks.workunit.client.0.smithi045.stdout:op 905 completed, throughput=4.98MB/sec 2021-07-31T02:08:09.998 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-6lUMbtTrancL6DD off=9718908 len=300356 2021-07-31T02:08:10.001 INFO:tasks.workunit.client.0.smithi045.stdout:op 906 completed, throughput=4.99MB/sec 2021-07-31T02:08:10.001 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-y4mKgct3uhOAFAd off=999073 len=1773508 2021-07-31T02:08:10.004 INFO:tasks.workunit.client.0.smithi045.stdout:op 907 completed, throughput=4.99MB/sec 2021-07-31T02:08:10.004 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-8vKoxKvOBgEK5gO off=2819905 len=895202 2021-07-31T02:08:10.010 INFO:tasks.workunit.client.0.smithi045.stdout:op 908 completed, throughput=5MB/sec 2021-07-31T02:08:10.010 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Y3pX6OXITb4Ju4O off=64993 len=696290 2021-07-31T02:08:10.012 INFO:tasks.workunit.client.0.smithi045.stdout:op 909 completed, throughput=5MB/sec 2021-07-31T02:08:10.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:10 smithi154 conmon[34553]: cluster 2021-07-31T02:08:10.088471+0000 mgr.x (mgr.24542) 896 : cluster [DBG] pgmap v1500: 9 pgs: 9 active+clean; 127 GiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 2.6 MiB/s rd, 3.3 MiB/s wr, 6 op/s 2021-07-31T02:08:10.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:10 smithi045 conmon[39718]: cluster 2021-07-31T02:08:10.088471+0000 mgr.x 2021-07-31T02:08:10.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:10 smithi045 conmon[39718]: (mgr.24542) 896 : cluster [DBG] pgmap v1500: 9 pgs: 9 active+clean; 127 GiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 2.6 MiB/s rd, 3.3 MiB/s wr, 6 op/s 2021-07-31T02:08:10.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:10 smithi045 conmon[33655]: cluster 2021-07-31T02:08:10.088471 2021-07-31T02:08:10.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:10 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 896 : cluster [DBG] pgmap v1500: 9 pgs: 9 active+clean; 127 GiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 2.6 MiB/s rd, 3.3 MiB/s wr, 6 op/s 2021-07-31T02:08:11.012 INFO:tasks.workunit.client.0.smithi045.stdout: 181: throughput=4.98MB/sec pending data=0 2021-07-31T02:08:11.012 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ThU0-nQU1cQS-FT off=273086 len=154906 2021-07-31T02:08:11.013 INFO:tasks.workunit.client.0.smithi045.stdout:op 910 completed, throughput=4.98MB/sec 2021-07-31T02:08:11.013 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-aR0YV4CR3Ifae9q off=177383 len=1384070 2021-07-31T02:08:11.020 INFO:tasks.workunit.client.0.smithi045.stdout:op 911 completed, throughput=4.98MB/sec 2021-07-31T02:08:11.021 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-9Le9pD3SOFr5eo3 off=393849 len=933368 2021-07-31T02:08:11.023 INFO:tasks.workunit.client.0.smithi045.stdout:op 912 completed, throughput=4.99MB/sec 2021-07-31T02:08:11.023 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-c83q3iJaKTPWv-r off=8286434 len=1242653 2021-07-31T02:08:11.025 INFO:tasks.workunit.client.0.smithi045.stdout:op 913 completed, throughput=4.99MB/sec 2021-07-31T02:08:11.025 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-f1mYcrPSpI5eg5r off=7725378 len=1360660 2021-07-31T02:08:11.028 INFO:tasks.workunit.client.0.smithi045.stdout:op 914 completed, throughput=5MB/sec 2021-07-31T02:08:12.028 INFO:tasks.workunit.client.0.smithi045.stdout: 182: throughput=4.97MB/sec pending data=0 2021-07-31T02:08:12.028 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-c6e17IRRkcBSd8s off=8928105 len=1827489 2021-07-31T02:08:12.039 INFO:tasks.workunit.client.0.smithi045.stdout:op 915 completed, throughput=4.98MB/sec 2021-07-31T02:08:12.039 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Uyq6_h1LGFbj1QU off=20035696 len=984150 2021-07-31T02:08:12.044 INFO:tasks.workunit.client.0.smithi045.stdout:op 916 completed, throughput=4.99MB/sec 2021-07-31T02:08:12.045 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-aLV-iKHDlKVyJhf off=10803277 len=371794 2021-07-31T02:08:12.048 INFO:tasks.workunit.client.0.smithi045.stdout:op 917 completed, throughput=4.99MB/sec 2021-07-31T02:08:12.048 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-2qxJQQCgY5W7dLy off=14607175 len=2043350 2021-07-31T02:08:12.052 INFO:tasks.workunit.client.0.smithi045.stdout:op 918 completed, throughput=5MB/sec 2021-07-31T02:08:13.052 INFO:tasks.workunit.client.0.smithi045.stdout: 183: throughput=4.97MB/sec pending data=0 2021-07-31T02:08:13.052 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-iWuNV5LdZnzSAeV off=11829058 len=1648172 2021-07-31T02:08:13.054 INFO:tasks.workunit.client.0.smithi045.stdout:op 919 completed, throughput=4.98MB/sec 2021-07-31T02:08:13.054 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-za8WxnBTPjhlUxJ off=907395 len=1363125 2021-07-31T02:08:13.064 INFO:tasks.workunit.client.0.smithi045.stdout:op 920 completed, throughput=4.99MB/sec 2021-07-31T02:08:13.065 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-cOSU6BM6ZcqAlEv off=5708787 len=1924141 2021-07-31T02:08:13.075 INFO:tasks.workunit.client.0.smithi045.stdout:op 921 completed, throughput=5MB/sec 2021-07-31T02:08:13.075 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-th9JgZUwoZo7eWY off=16442564 len=1736313 2021-07-31T02:08:13.079 INFO:tasks.workunit.client.0.smithi045.stdout:op 922 completed, throughput=5.01MB/sec 2021-07-31T02:08:13.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:12 smithi154 conmon[34553]: cluster 2021-07-31T02:08:12.088835+0000 mgr.x (mgr.24542) 897 : cluster [DBG] pgmap v1501: 9 pgs: 9 active+clean; 127 GiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 1.6 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2021-07-31T02:08:13.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:12 smithi045 conmon[39718]: cluster 2021-07-31T02:08:12.088835+0000 mgr.x (mgr. 2021-07-31T02:08:13.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:12 smithi045 conmon[39718]: 24542) 897 : cluster [DBG] pgmap v1501: 9 pgs: 9 active+clean; 127 GiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 1.6 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2021-07-31T02:08:13.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:12 smithi045 conmon[33655]: cluster 2021-07-31T02:08:12.088835+0000 mgr.x (mgr.24542 2021-07-31T02:08:13.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:12 smithi045 conmon[33655]: ) 897 : cluster [DBG] pgmap v1501: 9 pgs: 9 active+clean; 127 GiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 1.6 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2021-07-31T02:08:14.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:08:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:08:13.543Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:08:14.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:08:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:08:13.543Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:08:14.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:08:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:08:13.543Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:08:14.079 INFO:tasks.workunit.client.0.smithi045.stdout: 184: throughput=4.98MB/sec pending data=0 2021-07-31T02:08:14.080 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-OC-y-US79rsTfRS off=1331753 len=1940964 2021-07-31T02:08:14.085 INFO:tasks.workunit.client.0.smithi045.stdout:op 923 completed, throughput=4.99MB/sec 2021-07-31T02:08:14.085 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Xz3ByaET0A-yXoY off=18788851 len=113543 2021-07-31T02:08:14.085 INFO:tasks.workunit.client.0.smithi045.stdout:op 924 completed, throughput=4.99MB/sec 2021-07-31T02:08:14.086 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-2V9JDTPxlxJWq7V off=6844648 len=1838315 2021-07-31T02:08:14.087 INFO:tasks.workunit.client.0.smithi045.stdout:op 925 completed, throughput=5MB/sec 2021-07-31T02:08:14.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:14 smithi154 conmon[34553]: cluster 2021-07-31T02:08:14.089421+0000 mgr.x (mgr.24542) 898 : cluster [DBG] pgmap v1502: 9 pgs: 9 active+clean; 127 GiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 2.7 MiB/s rd, 3.3 MiB/s wr, 6 op/s 2021-07-31T02:08:14.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:14 smithi154 conmon[34553]: 2021-07-31T02:08:14.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:14 smithi045 conmon[39718]: cluster 2021-07-31T02:08:14 2021-07-31T02:08:14.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:14 smithi045 conmon[39718]: .089421+0000 mgr.x (mgr.24542) 898 : cluster [DBG] pgmap v1502: 9 pgs: 9 active+clean; 127 GiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 2.7 MiB/s rd, 3.3 MiB/s wr, 6 op/s 2021-07-31T02:08:14.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:14 smithi045 conmon[33655]: cluster 2021-07-31T02:08:14.089421+0000 mgr.x 2021-07-31T02:08:14.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:14 smithi045 conmon[33655]: (mgr.24542) 898 : cluster [DBG] pgmap v1502: 9 pgs: 9 active+clean; 127 GiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 2.7 MiB/s rd, 3.3 MiB/s wr, 6 op/s 2021-07-31T02:08:15.087 INFO:tasks.workunit.client.0.smithi045.stdout: 185: throughput=4.97MB/sec pending data=0 2021-07-31T02:08:15.087 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-77dXheKz-BUkCHr off=21275357 len=1727449 2021-07-31T02:08:15.094 INFO:tasks.workunit.client.0.smithi045.stdout:op 926 completed, throughput=4.98MB/sec 2021-07-31T02:08:15.094 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-yG5gShIDM0FkpOm off=9101057 len=1775715 2021-07-31T02:08:15.097 INFO:tasks.workunit.client.0.smithi045.stdout:op 927 completed, throughput=4.99MB/sec 2021-07-31T02:08:15.098 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-iVT0m7QU-Ukwx5u off=22358783 len=685182 2021-07-31T02:08:15.099 INFO:tasks.workunit.client.0.smithi045.stdout:op 928 completed, throughput=4.99MB/sec 2021-07-31T02:08:15.100 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-eJ-5sglcHTPK2qQ off=3329889 len=2026463 2021-07-31T02:08:15.103 INFO:tasks.workunit.client.0.smithi045.stdout:op 929 completed, throughput=5MB/sec 2021-07-31T02:08:16.103 INFO:tasks.workunit.client.0.smithi045.stdout: 186: throughput=4.98MB/sec pending data=0 2021-07-31T02:08:16.104 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-QCf_FKtSDSxIiaa off=961576 len=421293 2021-07-31T02:08:16.105 INFO:tasks.workunit.client.0.smithi045.stdout:op 930 completed, throughput=4.98MB/sec 2021-07-31T02:08:16.105 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-HNAny_QRw5NQMww off=13183654 len=1718668 2021-07-31T02:08:16.108 INFO:tasks.workunit.client.0.smithi045.stdout:op 931 completed, throughput=4.99MB/sec 2021-07-31T02:08:16.108 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-RguJt3CTkcaQOP1 off=9893479 len=1633066 2021-07-31T02:08:16.110 INFO:tasks.workunit.client.0.smithi045.stdout:op 932 completed, throughput=5MB/sec 2021-07-31T02:08:16.110 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-GPhCWp9LZWwtDEG off=11574598 len=1527533 2021-07-31T02:08:16.113 INFO:tasks.workunit.client.0.smithi045.stdout:op 933 completed, throughput=5MB/sec 2021-07-31T02:08:16.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:16 smithi154 conmon[34553]: cluster 2021-07-31T02:08:16.089937+0000 mgr.x (mgr.24542) 899 : cluster [DBG] pgmap v1503: 9 pgs: 9 active+clean; 127 GiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2021-07-31T02:08:16.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:16 smithi045 conmon[39718]: cluster 2021-07-31T02:08:16.089937 2021-07-31T02:08:16.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:16 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 899 : cluster [DBG] pgmap v1503: 9 pgs: 9 active+clean; 127 GiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2021-07-31T02:08:16.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:16 smithi045 conmon[33655]: cluster 2021-07-31T02:08:16.089937+0000 mgr.x (mgr.24542) 899 : cluster [DBG] pgmap v1503: 9 pgs: 9 active+clean; 127 GiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2021-07-31T02:08:17.113 INFO:tasks.workunit.client.0.smithi045.stdout: 187: throughput=4.98MB/sec pending data=0 2021-07-31T02:08:17.114 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-wuH9UCdLiE8oZQz off=9483878 len=319282 2021-07-31T02:08:17.115 INFO:tasks.workunit.client.0.smithi045.stdout:op 934 completed, throughput=4.98MB/sec 2021-07-31T02:08:17.115 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-lcSmzX8iW33oZmi off=11930136 len=46284 2021-07-31T02:08:17.116 INFO:tasks.workunit.client.0.smithi045.stdout:op 935 completed, throughput=4.98MB/sec 2021-07-31T02:08:17.116 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-VXChwy-OdXabMHK off=6459133 len=699353 2021-07-31T02:08:17.117 INFO:tasks.workunit.client.0.smithi045.stdout:op 936 completed, throughput=4.98MB/sec 2021-07-31T02:08:17.117 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-dm8ZT5H-reFLcaO off=9911917 len=367200 2021-07-31T02:08:17.120 INFO:tasks.workunit.client.0.smithi045.stdout:op 937 completed, throughput=4.99MB/sec 2021-07-31T02:08:17.120 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-_wBtAf_D6m0xhuC off=331198 len=1404410 2021-07-31T02:08:17.123 INFO:tasks.workunit.client.0.smithi045.stdout:op 938 completed, throughput=4.99MB/sec 2021-07-31T02:08:17.123 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Jmc1SOr4B5E-7bK off=722241 len=1359937 2021-07-31T02:08:17.129 INFO:tasks.workunit.client.0.smithi045.stdout:op 939 completed, throughput=5MB/sec 2021-07-31T02:08:17.130 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-PqjPS-PV4_8OiSa off=5087653 len=1735354 2021-07-31T02:08:17.131 INFO:tasks.workunit.client.0.smithi045.stdout:op 940 completed, throughput=5.01MB/sec 2021-07-31T02:08:18.132 INFO:tasks.workunit.client.0.smithi045.stdout: 188: throughput=4.98MB/sec pending data=0 2021-07-31T02:08:18.132 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-i5mQWD0N0vM_LT9 off=1852911 len=1923249 2021-07-31T02:08:18.135 INFO:tasks.workunit.client.0.smithi045.stdout:op 941 completed, throughput=4.99MB/sec 2021-07-31T02:08:18.136 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-KogbaYxG13W_-lz off=4004177 len=852037 2021-07-31T02:08:18.140 INFO:tasks.workunit.client.0.smithi045.stdout:op 942 completed, throughput=5MB/sec 2021-07-31T02:08:18.141 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-v597zHqo4gze0_r off=2239531 len=568403 2021-07-31T02:08:18.145 INFO:tasks.workunit.client.0.smithi045.stdout:op 943 completed, throughput=5MB/sec 2021-07-31T02:08:18.146 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-KCntu5HM6FueaMe off=5721106 len=1362435 2021-07-31T02:08:18.147 INFO:tasks.workunit.client.0.smithi045.stdout:op 944 completed, throughput=5MB/sec 2021-07-31T02:08:18.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:18 smithi154 conmon[34553]: cluster 2021-07-31T02:08:18.090553+0000 mgr.x (mgr.24542) 900 : cluster [DBG] pgmap v1504: 9 pgs: 9 active+clean; 127 GiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2021-07-31T02:08:18.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:18 smithi045 conmon[39718]: cluster 2021-07-31T02:08:18.090553+0000 2021-07-31T02:08:18.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:18 smithi045 conmon[39718]: mgr.x (mgr.24542) 900 : cluster [DBG] pgmap v1504: 9 pgs: 9 active+clean; 127 GiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2021-07-31T02:08:18.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:18 smithi045 conmon[33655]: cluster 2021-07-31T02:08:18.090553+0000 mgr.x (mgr.24542) 900 : 2021-07-31T02:08:18.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:18 smithi045 conmon[33655]: cluster [DBG] pgmap v1504: 9 pgs: 9 active+clean; 127 GiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2021-07-31T02:08:19.147 INFO:tasks.workunit.client.0.smithi045.stdout: 189: throughput=4.98MB/sec pending data=0 2021-07-31T02:08:19.147 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-g-nGzHrLkliEqEZ off=8766274 len=1479068 2021-07-31T02:08:19.150 INFO:tasks.workunit.client.0.smithi045.stdout:op 945 completed, throughput=4.99MB/sec 2021-07-31T02:08:19.150 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-i17QSSuzLJPzcfF off=16632651 len=1499481 2021-07-31T02:08:19.157 INFO:tasks.workunit.client.0.smithi045.stdout:op 946 completed, throughput=4.99MB/sec 2021-07-31T02:08:19.158 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-3IfKJp9TS84gWmB off=3225424 len=1468386 2021-07-31T02:08:19.165 INFO:tasks.workunit.client.0.smithi045.stdout:op 947 completed, throughput=5MB/sec 2021-07-31T02:08:20.165 INFO:tasks.workunit.client.0.smithi045.stdout: 190: throughput=4.97MB/sec pending data=0 2021-07-31T02:08:20.166 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-C-mMYfsJvCRPHgU off=22313034 len=831918 2021-07-31T02:08:20.170 INFO:tasks.workunit.client.0.smithi045.stdout:op 948 completed, throughput=4.98MB/sec 2021-07-31T02:08:20.170 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-3CQijZDsNvYkhAY off=6555583 len=848810 2021-07-31T02:08:20.175 INFO:tasks.workunit.client.0.smithi045.stdout:op 949 completed, throughput=4.98MB/sec 2021-07-31T02:08:20.175 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-GXbsOW4-3pvo8PI off=7596795 len=668562 2021-07-31T02:08:20.179 INFO:tasks.workunit.client.0.smithi045.stdout:op 950 completed, throughput=4.99MB/sec 2021-07-31T02:08:20.180 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-1E3e7JP55Q_HdC8 off=5330373 len=1772957 2021-07-31T02:08:20.188 INFO:tasks.workunit.client.0.smithi045.stdout:op 951 completed, throughput=4.99MB/sec 2021-07-31T02:08:20.189 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-TUhls6WYM5SG-Y9 off=4823334 len=289165 2021-07-31T02:08:20.189 INFO:tasks.workunit.client.0.smithi045.stdout:op 952 completed, throughput=5MB/sec 2021-07-31T02:08:20.189 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Y-qhJP50Kdh-WqA off=1325052 len=245535 2021-07-31T02:08:20.191 INFO:tasks.workunit.client.0.smithi045.stdout:op 953 completed, throughput=5MB/sec 2021-07-31T02:08:20.192 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-DyWSM2Q5qvuhJFz off=4376784 len=1245227 2021-07-31T02:08:20.194 INFO:tasks.workunit.client.0.smithi045.stdout:op 954 completed, throughput=5MB/sec 2021-07-31T02:08:20.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:20 smithi154 conmon[34553]: cluster 2021-07-31T02:08:20.091058+0000 mgr.x (mgr.24542) 901 : cluster [DBG] pgmap v1505: 9 pgs: 9 active+clean; 127 GiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 3.4 MiB/s rd, 2.7 MiB/s wr, 6 op/s 2021-07-31T02:08:20.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:20 smithi045 conmon[39718]: cluster 2021-07-31T02:08:20.091058+0000 mgr.x (mgr.24542 2021-07-31T02:08:20.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:20 smithi045 conmon[39718]: ) 901 : cluster [DBG] pgmap v1505: 9 pgs: 9 active+clean; 127 GiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 3.4 MiB/s rd, 2.7 MiB/s wr, 6 op/s 2021-07-31T02:08:20.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:20 smithi045 conmon[33655]: cluster 2021-07-31T02:08:20.091058+0000 mgr.x (mgr.24542) 901 : cluster [DBG] pgmap v1505: 9 pgs: 9 active+clean; 127 GiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 3.4 MiB/s rd, 2.7 MiB/s wr, 6 op/s 2021-07-31T02:08:21.194 INFO:tasks.workunit.client.0.smithi045.stdout: 191: throughput=4.98MB/sec pending data=0 2021-07-31T02:08:21.194 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-tQt3I9d34gj1-ij off=13061382 len=1789591 2021-07-31T02:08:21.196 INFO:tasks.workunit.client.0.smithi045.stdout:op 955 completed, throughput=4.99MB/sec 2021-07-31T02:08:21.197 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-quvqep5K2ONN4yJ off=3321474 len=1652377 2021-07-31T02:08:21.200 INFO:tasks.workunit.client.0.smithi045.stdout:op 956 completed, throughput=4.99MB/sec 2021-07-31T02:08:21.200 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-6V9pdz0O4z5EL40 off=5997297 len=452894 2021-07-31T02:08:21.200 INFO:tasks.workunit.client.0.smithi045.stdout:op 957 completed, throughput=5MB/sec 2021-07-31T02:08:21.201 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-sIBenQXi4Rxuqo_ off=3530312 len=1292084 2021-07-31T02:08:21.202 INFO:tasks.workunit.client.0.smithi045.stdout:op 958 completed, throughput=5MB/sec 2021-07-31T02:08:22.203 INFO:tasks.workunit.client.0.smithi045.stdout: 192: throughput=4.98MB/sec pending data=0 2021-07-31T02:08:22.203 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-tQt3I9d34gj1-ij off=13552383 len=1298590 2021-07-31T02:08:22.205 INFO:tasks.workunit.client.0.smithi045.stdout:op 959 completed, throughput=4.98MB/sec 2021-07-31T02:08:22.206 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-YNsoof9r-Jp7kxj off=4401089 len=679967 2021-07-31T02:08:22.210 INFO:tasks.workunit.client.0.smithi045.stdout:op 960 completed, throughput=4.99MB/sec 2021-07-31T02:08:22.210 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-WwWQpgkmpuhgoLv off=659767 len=1685354 2021-07-31T02:08:22.212 INFO:tasks.workunit.client.0.smithi045.stdout:op 961 completed, throughput=4.99MB/sec 2021-07-31T02:08:22.213 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-QYg-JKpcTkC_yrm off=18888879 len=467568 2021-07-31T02:08:22.217 INFO:tasks.workunit.client.0.smithi045.stdout:op 962 completed, throughput=5MB/sec 2021-07-31T02:08:22.218 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-PoOA2mRnauABaD- off=1080059 len=2004129 2021-07-31T02:08:22.228 INFO:tasks.workunit.client.0.smithi045.stdout:op 963 completed, throughput=5.01MB/sec 2021-07-31T02:08:23.228 INFO:tasks.workunit.client.0.smithi045.stdout: 193: throughput=4.98MB/sec pending data=0 2021-07-31T02:08:23.229 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-mtD20Ustw6sxJWQ off=9260588 len=1398218 2021-07-31T02:08:23.243 INFO:tasks.workunit.client.0.smithi045.stdout:op 964 completed, throughput=4.99MB/sec 2021-07-31T02:08:23.243 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-bZHBe4L2KEKxzum off=14555373 len=6733 2021-07-31T02:08:23.244 INFO:tasks.workunit.client.0.smithi045.stdout:op 965 completed, throughput=4.99MB/sec 2021-07-31T02:08:23.244 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-d3p00c3cWU0835J off=573261 len=184584 2021-07-31T02:08:23.245 INFO:tasks.workunit.client.0.smithi045.stdout:op 966 completed, throughput=4.99MB/sec 2021-07-31T02:08:23.245 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-8Dpzzy66OxoivyY off=1000654 len=169234 2021-07-31T02:08:23.245 INFO:tasks.workunit.client.0.smithi045.stdout:op 967 completed, throughput=4.99MB/sec 2021-07-31T02:08:23.245 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-kJ4lFomhvry0Nd1 off=5345377 len=863394 2021-07-31T02:08:23.250 INFO:tasks.workunit.client.0.smithi045.stdout:op 968 completed, throughput=4.99MB/sec 2021-07-31T02:08:23.250 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-D5tayx7G_a2PSGH off=2187652 len=1540311 2021-07-31T02:08:23.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:22 smithi154 conmon[34553]: cluster 2021-07-31T02:08:22.091481+0000 mgr.x (mgr.24542) 902 2021-07-31T02:08:23.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:22 smithi154 conmon[34553]: : cluster [DBG] pgmap v1506: 9 pgs: 9 active+clean; 127 GiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 2.8 MiB/s rd, 1.5 MiB/s wr, 3 op/s 2021-07-31T02:08:23.259 INFO:tasks.workunit.client.0.smithi045.stdout:op 969 completed, throughput=5MB/sec 2021-07-31T02:08:23.259 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-IXu-TRjiWAB7o7g off=14155674 len=823561 2021-07-31T02:08:23.260 INFO:tasks.workunit.client.0.smithi045.stdout:op 970 completed, throughput=5MB/sec 2021-07-31T02:08:23.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:22 smithi045 conmon[33655]: cluster 2021-07-31T02:08:22.091481+0000 2021-07-31T02:08:23.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:22 smithi045 conmon[33655]: mgr.x (mgr.24542) 902 : cluster [DBG] pgmap v1506: 9 pgs: 9 active+clean; 127 GiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 2.8 MiB/s rd, 1.5 MiB/s wr, 3 op/s 2021-07-31T02:08:23.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:22 smithi045 conmon[39718]: cluster 2021-07-31T02:08:22. 2021-07-31T02:08:23.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:22 smithi045 conmon[39718]: 091481+0000 mgr.x (mgr.24542) 902 : cluster [DBG] pgmap v1506: 9 pgs: 9 active+clean; 127 GiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 2.8 MiB/s rd, 1.5 MiB/s wr, 3 op/s 2021-07-31T02:08:24.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:08:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:08:23.544Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:08:24.027 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:08:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:08:23.544Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:08:24.027 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:08:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:08:23.544Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:08:24.261 INFO:tasks.workunit.client.0.smithi045.stdout: 194: throughput=4.98MB/sec pending data=0 2021-07-31T02:08:24.261 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-e6X1lpvVp7fh_hJ off=7801712 len=1955856 2021-07-31T02:08:24.271 INFO:tasks.workunit.client.0.smithi045.stdout:op 971 completed, throughput=4.99MB/sec 2021-07-31T02:08:24.271 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-knBxVf_Y1c0GG2z off=504013 len=1137138 2021-07-31T02:08:24.277 INFO:tasks.workunit.client.0.smithi045.stdout:op 972 completed, throughput=4.99MB/sec 2021-07-31T02:08:24.278 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ETg0z_Z12hK-iFO off=255490 len=739340 2021-07-31T02:08:24.283 INFO:tasks.workunit.client.0.smithi045.stdout:op 973 completed, throughput=5MB/sec 2021-07-31T02:08:24.283 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-kc-Ba5FZI_aRsmj off=3361883 len=1960868 2021-07-31T02:08:24.290 INFO:tasks.workunit.client.0.smithi045.stdout:op 974 completed, throughput=5.01MB/sec 2021-07-31T02:08:24.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:24 smithi154 conmon[34553]: cluster 2021-07-31T02:08:24.091972+0000 mgr.x (mgr.24542) 903 : cluster [DBG] pgmap v1507: 9 pgs: 9 active+clean; 127 GiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 3.7 MiB/s rd, 2.6 MiB/s wr, 6 op/s 2021-07-31T02:08:24.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:24 smithi045 conmon[39718]: cluster 2021-07-31T02:08:24.091972+0000 mgr.x (mgr 2021-07-31T02:08:24.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:24 smithi045 conmon[39718]: .24542) 903 : cluster [DBG] pgmap v1507: 9 pgs: 9 active+clean; 127 GiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 3.7 MiB/s rd, 2.6 MiB/s wr, 6 op/s 2021-07-31T02:08:24.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:24 smithi045 conmon[33655]: cluster 2021-07-31T02:08: 2021-07-31T02:08:24.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:24 smithi045 conmon[33655]: 24.091972+0000 mgr.x (mgr.24542) 903 : cluster [DBG] pgmap v1507: 9 pgs: 9 active+clean; 127 GiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 3.7 MiB/s rd, 2.6 MiB/s wr, 6 op/s 2021-07-31T02:08:25.290 INFO:tasks.workunit.client.0.smithi045.stdout: 195: throughput=4.98MB/sec pending data=0 2021-07-31T02:08:25.291 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-4Jf2vXBqQ5t9pLO off=5935177 len=1586333 2021-07-31T02:08:25.293 INFO:tasks.workunit.client.0.smithi045.stdout:op 975 completed, throughput=4.99MB/sec 2021-07-31T02:08:25.294 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-MQL8Y_c7C-_R9Xe off=17532625 len=1795586 2021-07-31T02:08:25.303 INFO:tasks.workunit.client.0.smithi045.stdout:op 976 completed, throughput=5MB/sec 2021-07-31T02:08:25.304 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-R-NmqMvN4VRVk4N off=5385967 len=48984 2021-07-31T02:08:25.306 INFO:tasks.workunit.client.0.smithi045.stdout:op 977 completed, throughput=5MB/sec 2021-07-31T02:08:25.307 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-wbI8_0lLN_RWjd0 off=7974400 len=736805 2021-07-31T02:08:25.308 INFO:tasks.workunit.client.0.smithi045.stdout:op 978 completed, throughput=5MB/sec 2021-07-31T02:08:25.308 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-WooGvr_t-SAp8Zl off=62615 len=632901 2021-07-31T02:08:25.317 INFO:tasks.workunit.client.0.smithi045.stdout:op 979 completed, throughput=5MB/sec 2021-07-31T02:08:26.309 INFO:tasks.workunit.client.0.smithi045.stdout: 196: throughput=4.98MB/sec pending data=0 2021-07-31T02:08:26.310 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-QLPlH4aX9dbauL2 off=8562878 len=1771931 2021-07-31T02:08:26.321 INFO:tasks.workunit.client.0.smithi045.stdout:op 980 completed, throughput=4.99MB/sec 2021-07-31T02:08:26.321 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ibwg098LFTcVmMx off=16793281 len=548047 2021-07-31T02:08:26.323 INFO:tasks.workunit.client.0.smithi045.stdout:op 981 completed, throughput=4.99MB/sec 2021-07-31T02:08:26.323 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-0InLzCFplYFJEhX off=1005849 len=1350529 2021-07-31T02:08:26.325 INFO:tasks.workunit.client.0.smithi045.stdout:op 982 completed, throughput=5MB/sec 2021-07-31T02:08:26.325 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-etXERW2iUXnQ1yT off=10662154 len=1143851 2021-07-31T02:08:26.330 INFO:tasks.workunit.client.0.smithi045.stdout:op 983 completed, throughput=5MB/sec 2021-07-31T02:08:26.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:26 smithi154 conmon[34553]: cluster 2021-07-31T02:08:26.092373+0000 mgr.x (mgr.24542) 904 : cluster [DBG] pgmap v1508: 9 pgs: 9 active+clean; 127 GiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 2.7 MiB/s rd, 1.5 MiB/s wr, 3 op/s 2021-07-31T02:08:26.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:26 smithi045 conmon[39718]: cluster 2021-07-31T02:08:26.092373+0000 mgr.x (mgr 2021-07-31T02:08:26.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:26 smithi045 conmon[39718]: .24542) 904 : cluster [DBG] pgmap v1508: 9 pgs: 9 active+clean; 127 GiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 2.7 MiB/s rd, 1.5 MiB/s wr, 3 op/s 2021-07-31T02:08:26.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:26 smithi045 conmon[33655]: cluster 2021-07-31T02:08:26.092373+0000 mgr.x (mgr.24542) 904 : cluster 2021-07-31T02:08:26.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:26 smithi045 conmon[33655]: [DBG] pgmap v1508: 9 pgs: 9 active+clean; 127 GiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 2.7 MiB/s rd, 1.5 MiB/s wr, 3 op/s 2021-07-31T02:08:27.331 INFO:tasks.workunit.client.0.smithi045.stdout: 197: throughput=4.98MB/sec pending data=0 2021-07-31T02:08:27.331 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ZCQnCdE6p2ykBdQ off=2993844 len=634952 2021-07-31T02:08:27.333 INFO:tasks.workunit.client.0.smithi045.stdout:op 984 completed, throughput=4.98MB/sec 2021-07-31T02:08:27.333 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-noHmvD6qT9p4_l1 off=15104049 len=540662 2021-07-31T02:08:27.334 INFO:tasks.workunit.client.0.smithi045.stdout:op 985 completed, throughput=4.98MB/sec 2021-07-31T02:08:27.334 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-o0yNZ9ZPtxAUnK0 off=22245243 len=1911495 2021-07-31T02:08:27.343 INFO:tasks.workunit.client.0.smithi045.stdout:op 986 completed, throughput=4.99MB/sec 2021-07-31T02:08:27.343 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-vwtteh_odr1IxfR off=6316869 len=700202 2021-07-31T02:08:27.345 INFO:tasks.workunit.client.0.smithi045.stdout:op 987 completed, throughput=4.99MB/sec 2021-07-31T02:08:27.345 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-jrb-q-MX4Amsppc off=10118024 len=1552959 2021-07-31T02:08:27.352 INFO:tasks.workunit.client.0.smithi045.stdout:op 988 completed, throughput=5MB/sec 2021-07-31T02:08:28.352 INFO:tasks.workunit.client.0.smithi045.stdout: 198: throughput=4.98MB/sec pending data=0 2021-07-31T02:08:28.353 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-3yEr6toSzHR7iKw off=7889986 len=1897445 2021-07-31T02:08:28.364 INFO:tasks.workunit.client.0.smithi045.stdout:op 989 completed, throughput=4.98MB/sec 2021-07-31T02:08:28.364 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-vDwDZ66CViZsMkO off=15460338 len=916855 2021-07-31T02:08:28.369 INFO:tasks.workunit.client.0.smithi045.stdout:op 990 completed, throughput=4.99MB/sec 2021-07-31T02:08:28.369 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-9JKDFvSJh1IK0b1 off=761273 len=247148 2021-07-31T02:08:28.370 INFO:tasks.workunit.client.0.smithi045.stdout:op 991 completed, throughput=4.99MB/sec 2021-07-31T02:08:28.371 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-W5Qp8JqogBnLD_1 off=12671482 len=1925804 2021-07-31T02:08:28.378 INFO:tasks.workunit.client.0.smithi045.stdout:op 992 completed, throughput=5MB/sec 2021-07-31T02:08:28.378 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-pBN5p5dX4xgcgs7 off=4808405 len=397222 2021-07-31T02:08:28.378 INFO:tasks.workunit.client.0.smithi045.stdout:op 993 completed, throughput=5MB/sec 2021-07-31T02:08:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:28 smithi154 conmon[34553]: audit 2021-07-31T02:08:28.010512+0000 mon.b (mon.2) 173 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:08:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:28 smithi154 conmon[34553]: 2021-07-31T02:08:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:28 smithi154 conmon[34553]: audit 2021-07-31T02:08:28.011265+0000 mon.a (mon.0) 1859 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:08:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:28 smithi154 conmon[34553]: audit 2021-07-31T02:08:28. 2021-07-31T02:08:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:28 smithi154 conmon[34553]: 029191+0000 mon.b (mon.2) 174 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:08:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:28 smithi154 conmon[34553]: audit 2021-07 2021-07-31T02:08:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:28 smithi154 conmon[34553]: -31T02:08:28.029915+0000 mon.a (mon.0) 1860 : 2021-07-31T02:08:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:28 smithi154 conmon[34553]: audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:08:28.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:28 smithi045 conmon[39718]: audit 2021-07-31T02:08:28.010512+0000 mon.b (mon.2) 173 2021-07-31T02:08:28.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:28 smithi045 conmon[39718]: : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:08:28.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:28 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T02:08:28.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:28 smithi045 conmon[39718]: T02:08:28.011265+0000 mon.a (mon.0) 1859 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:08:28.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:28 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T02:08:28.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:28 smithi045 conmon[39718]: 31T02:08:28.029191+0000 mon.b (mon.2) 174 2021-07-31T02:08:28.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:28 smithi045 conmon[39718]: : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:08:28.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:28 smithi045 conmon[39718]: audit 2021-07-31T02:08:28 2021-07-31T02:08:28.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:28 smithi045 conmon[39718]: .029915+0000 mon.a (mon.0) 1860 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:08:28.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:28 smithi045 conmon[33655]: audit 2021-07-31T02:08:28.010512+0000 mon.b (mon.2) 2021-07-31T02:08:28.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:28 smithi045 conmon[33655]: 173 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:08:28.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:28 smithi045 conmon[33655]: audit 2021-07-31T02:08:28.011265+0000 mon.a (mon.0) 1859 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:08:28.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:28 smithi045 conmon[33655]: audit 2021-07-31T02:08:28.029191+0000 mon.b (mon.2) 174 : audit 2021-07-31T02:08:28.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:28 smithi045 conmon[33655]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:08:28.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:28 smithi045 conmon[33655]: audit 2021 2021-07-31T02:08:28.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:28 smithi045 conmon[33655]: -07-31T02:08:28.029915+0000 mon.a (mon.0) 1860 : 2021-07-31T02:08:28.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:28 smithi045 conmon[33655]: audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:08:29.379 INFO:tasks.workunit.client.0.smithi045.stdout: 199: throughput=4.98MB/sec pending data=0 2021-07-31T02:08:29.379 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-pcxwl_i_Lt1-eP7 off=75377 len=1749295 2021-07-31T02:08:29.387 INFO:tasks.workunit.client.0.smithi045.stdout:op 994 completed, throughput=4.98MB/sec 2021-07-31T02:08:29.387 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-mwqrCyDSnEW3N5h off=3904866 len=438208 2021-07-31T02:08:29.388 INFO:tasks.workunit.client.0.smithi045.stdout:op 995 completed, throughput=4.99MB/sec 2021-07-31T02:08:29.389 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-SkdGA8AP2s4SS3V off=5775486 len=2044855 2021-07-31T02:08:29.399 INFO:tasks.workunit.client.0.smithi045.stdout:op 996 completed, throughput=5MB/sec 2021-07-31T02:08:29.399 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-UgkkT0nw5xSrmJi off=4751200 len=61710 2021-07-31T02:08:29.400 INFO:tasks.workunit.client.0.smithi045.stdout:op 997 completed, throughput=5MB/sec 2021-07-31T02:08:29.401 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-O6FquxKR9tcSOII off=5983062 len=467179 2021-07-31T02:08:29.404 INFO:tasks.workunit.client.0.smithi045.stdout:op 998 completed, throughput=5MB/sec 2021-07-31T02:08:29.404 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-NYToXIW7MR0ECDD off=693196 len=1829427 2021-07-31T02:08:29.410 INFO:tasks.workunit.client.0.smithi045.stdout:op 999 completed, throughput=5.01MB/sec 2021-07-31T02:08:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:29 smithi154 conmon[34553]: cluster 2021-07-31T02:08:28.092803+0000 mgr.x (mgr.24542) 905 : cluster [DBG] pgmap v1509: 9 pgs: 9 active+clean; 127 GiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 2.7 MiB/s rd, 1.5 MiB/s wr, 3 op/s 2021-07-31T02:08:29.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:29 smithi045 conmon[39718]: cluster 2021-07-31T02:08:28.092803+0000 mgr.x (mgr. 2021-07-31T02:08:29.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:29 smithi045 conmon[39718]: 24542) 905 : cluster [DBG] pgmap v1509: 9 pgs: 9 active+clean; 127 GiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 2.7 MiB/s rd, 1.5 MiB/s wr, 3 op/s 2021-07-31T02:08:29.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:29 smithi045 conmon[33655]: cluster 2021-07-31T02:08:28.092803+0000 mgr.x (mgr 2021-07-31T02:08:29.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:29 smithi045 conmon[33655]: .24542) 905 : cluster [DBG] pgmap v1509: 9 pgs: 9 active+clean; 127 GiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 2.7 MiB/s rd, 1.5 MiB/s wr, 3 op/s 2021-07-31T02:08:30.410 INFO:tasks.workunit.client.0.smithi045.stdout: 200: throughput=4.98MB/sec pending data=0 2021-07-31T02:08:30.410 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-JWm4tl7W0YWV1N3 off=18502282 len=1972007 2021-07-31T02:08:30.419 INFO:tasks.workunit.client.0.smithi045.stdout:op 1000 completed, throughput=4.99MB/sec 2021-07-31T02:08:30.419 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-5Bj46l09VDdWX6Q off=1952267 len=927903 2021-07-31T02:08:30.420 INFO:tasks.workunit.client.0.smithi045.stdout:op 1001 completed, throughput=4.99MB/sec 2021-07-31T02:08:30.420 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-j35_bxrHqw-AZGX off=22583 len=1360361 2021-07-31T02:08:30.421 INFO:tasks.workunit.client.0.smithi045.stdout:op 1002 completed, throughput=5MB/sec 2021-07-31T02:08:30.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[34553]: cluster 2021-07-31T02:08:30.093306+0000 mgr.x (mgr.24542) 906 : cluster [DBG] pgmap v1510: 9 pgs: 9 active+clean; 127 GiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 3.2 MiB/s rd, 3.1 MiB/s wr, 5 op/s 2021-07-31T02:08:30.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: debug 2021-07-31T02:08:30.245+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:08:30.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:08:30.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:08:30.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: yield 2021-07-31T02:08:30.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:08:30.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:08:30.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:08:30.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:08:30.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:08:30.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:08:30.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:08:30.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:08:30.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:08:30.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:08:30.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:08:30.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: debug 2021-07-31T02:08:30.247+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:08:30.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:08:30.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:08:30.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: yield 2021-07-31T02:08:30.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:08:30.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:08:30.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:08:30.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:08:30.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:08:30.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:08:30.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:08:30.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:08:30.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:08:30.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:08:30.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:08:30.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: debug 2021-07-31T02:08:30.282+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:08:30.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:08:30.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:08:30.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: yield 2021-07-31T02:08:30.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:08:30.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:08:30.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:08:30.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:08:30.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:08:30.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:08:30.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:08:30.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:08:30.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:08:30.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:08:30.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:08:30.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: debug 2021-07-31T02:08:30.282+0000 7fcb9295f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:08:30.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:08:30.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:08:30.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: yield 2021-07-31T02:08:30.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:08:30.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:08:30.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:08:30.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:08:30.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:08:30.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:08:30.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:08:30.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:08:30.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:08:30.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:08:30.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:08:30.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: 2021-07-31T02:08:30.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:08:30.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: 2021-07-31T02:08:30.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:08:30.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:08:30.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:08:30.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:08:30.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:08:30.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:08:30.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:08:30.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:08:30.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:08:30.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:08:30.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:08:30.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:08:30.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:08:30.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:08:30.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:08:30.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:08:30.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:08:30.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:08:30.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: raise 2021-07-31T02:08:30.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:08:30.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:08:30.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:08:30.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:08:30.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:08:30.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: debug 2021-07-31T02:08:30.290+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:08:30.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:08:30.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:08:30.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: yield 2021-07-31T02:08:30.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:08:30.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:08:30.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:08:30.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:08:30.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:08:30.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:08:30.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:08:30.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:08:30.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:08:30.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:08:30.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:08:30.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: debug 2021-07-31T02:08:30.290+0000 7fcb93160700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:08:30.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:08:30.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:08:30.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: yield 2021-07-31T02:08:30.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:08:30.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:08:30.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:08:30.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:08:30.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:08:30.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:08:30.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:08:30.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:08:30.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:08:30.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:08:30.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:08:30.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: 2021-07-31T02:08:30.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:08:30.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: 2021-07-31T02:08:30.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:08:30.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:08:30.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:08:30.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:08:30.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:08:30.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:08:30.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:08:30.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:08:30.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:08:30.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:08:30.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:08:30.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:08:30.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:08:30.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:08:30.535 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:08:30.535 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:08:30.535 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:08:30.535 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:08:30.535 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: raise 2021-07-31T02:08:30.536 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:08:30.536 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:08:30.536 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:08:30.536 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:08:30.537 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:08:30 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:08:30.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:30 smithi045 conmon[39718]: cluster 2021-07-31T02:08:30.093306+0000 mgr.x (mgr 2021-07-31T02:08:30.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:30 smithi045 conmon[39718]: .24542) 906 : cluster [DBG] pgmap v1510: 9 pgs: 9 active+clean; 127 GiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 3.2 MiB/s rd, 3.1 MiB/s wr, 5 op/s 2021-07-31T02:08:30.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:30 smithi045 conmon[33655]: cluster 2021-07-31T02:08:30.093306+0000 2021-07-31T02:08:30.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:30 smithi045 conmon[33655]: mgr.x (mgr.24542) 906 : cluster [DBG] pgmap v1510: 9 pgs: 9 active+clean; 127 GiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 3.2 MiB/s rd, 3.1 MiB/s wr, 5 op/s 2021-07-31T02:08:31.422 INFO:tasks.workunit.client.0.smithi045.stdout: 201: throughput=4.98MB/sec pending data=0 2021-07-31T02:08:31.422 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-fQ4fsLKakiVHdNG off=9716810 len=849142 2021-07-31T02:08:31.423 INFO:tasks.workunit.client.0.smithi045.stdout:op 1003 completed, throughput=4.98MB/sec 2021-07-31T02:08:31.423 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-c7LJZgvo80SZVKJ off=3776463 len=273235 2021-07-31T02:08:31.424 INFO:tasks.workunit.client.0.smithi045.stdout:op 1004 completed, throughput=4.98MB/sec 2021-07-31T02:08:31.424 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-g4JhYDQtkuWnIf9 off=19148354 len=458929 2021-07-31T02:08:31.425 INFO:tasks.workunit.client.0.smithi045.stdout:op 1005 completed, throughput=4.98MB/sec 2021-07-31T02:08:31.426 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-i4XkE4ccFKb290o off=10457768 len=1406128 2021-07-31T02:08:31.433 INFO:tasks.workunit.client.0.smithi045.stdout:op 1006 completed, throughput=4.99MB/sec 2021-07-31T02:08:31.433 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-2V787ekf4tetliE off=2217591 len=705689 2021-07-31T02:08:31.437 INFO:tasks.workunit.client.0.smithi045.stdout:op 1007 completed, throughput=4.99MB/sec 2021-07-31T02:08:31.437 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-8ecVUkybZmbvw8U off=9185293 len=1413459 2021-07-31T02:08:31.444 INFO:tasks.workunit.client.0.smithi045.stdout:op 1008 completed, throughput=5MB/sec 2021-07-31T02:08:31.445 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-k_rHJ26RLSinwAQ off=13660560 len=1587686 2021-07-31T02:08:31.452 INFO:tasks.workunit.client.0.smithi045.stdout:op 1009 completed, throughput=5.01MB/sec 2021-07-31T02:08:31.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: audit 2021-07-31T02:08:30.201111+0000 mon.b (mon.2) 175 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:08:31.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: audit 2021-07-31T02:08:30.202440+0000 mon.b (mon.2) 176 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:08:31.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: audit 2021-07-31T02:08:30.203210+0000 mon.b (mon.2) 177 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:08:31.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:08:30.203924+0000 mgr.x (mgr.24542) 907 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:08:31.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: cephadm 2021-07-31T02: 2021-07-31T02:08:31.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: 08:30.246679+0000 mgr.x (mgr.24542) 908 : cephadm [ERR] 2021-07-31T02:08:31.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:08:31.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:08:31.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: yield 2021-07-31T02:08:31.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:08:31.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:08:31.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:08:31.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:08:31.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:08:31.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:08:31.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:08:31.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:08:31.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:08:31.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: task.result() 2021-07-31T02:08:31.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:08:31.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:08:30.247424+0000 mgr.x (mgr.24542) 909 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:08:31.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:08:30.248566+0000 mgr.x (mgr.24542) 910 : cephadm [ERR] 2021-07-31T02:08:31.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:08:31.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:08:31.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: yield 2021-07-31T02:08:31.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:08:31.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:08:31.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:08:31.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:08:31.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:08:31.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:08:31.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:08:31.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:08:31.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:08:31.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: task.result() 2021-07-31T02:08:31.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:08:31.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:08:30.258379+0000 mgr.x (mgr.24542) 911 : 2021-07-31T02:08:31.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:08:31.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:08:30.283379+0000 mgr.x (mgr.24542) 912 : cephadm [ERR] 2021-07-31T02:08:31.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:08:31.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:08:31.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: yield 2021-07-31T02:08:31.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:08:31.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:08:31.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:08:31.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:08:31.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:08:31.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:08:31.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:08:31.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:08:31.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:08:31.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: task.result() 2021-07-31T02:08:31.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:08:31.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:08:30.283799+0000 mgr.x (mgr.24542) 913 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:08:31.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:08:31.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:08:31.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: yield 2021-07-31T02:08:31.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:08:31.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:08:31.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:08:31.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:08:31.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:08:31.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:08:31.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:08:31.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:08:31.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:08:31.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: task.result() 2021-07-31T02:08:31.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:08:31.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: 2021-07-31T02:08:31.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:08:31.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: 2021-07-31T02:08:31.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:08:31.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:08:31.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:08:31.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:08:31.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:08:31.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:08:31.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:08:31.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:08:31.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:08:31.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:08:31.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:08:31.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:08:31.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:08:31.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:08:31.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:08:31.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:08:31.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:08:31.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:08:31.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: raise 2021-07-31T02:08:31.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:08:31.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:08:31.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:08:31.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:08:31.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:08:31.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:08:30.291227 2021-07-31T02:08:31.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: +0000 mgr.x (mgr.24542) 914 : cephadm [ERR] 2021-07-31T02:08:31.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:08:31.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:08:31.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: yield 2021-07-31T02:08:31.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:08:31.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:08:31.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:08:31.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:08:31.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:08:31.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:08:31.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:08:31.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:08:31.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:08:31.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: task.result() 2021-07-31T02:08:31.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:08:31.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:08:30.291609+0000 mgr.x (mgr.24542) 915 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:08:31.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:08:31.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:08:31.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: yield 2021-07-31T02:08:31.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: audit 2021-07-31T02:08:30.201111+0000 mon.b (mon.2) 175 2021-07-31T02:08:31.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:08:31.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: audit 2021-07-31T02:08:30.202440+0000 mon.b (mon.2) 176 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:08:31.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: audit 2021-07-31T02: 2021-07-31T02:08:31.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: 08:30.203210+0000 mon.b ( 2021-07-31T02:08:31.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: mon.2) 177 : audit [INF] 2021-07-31T02:08:31.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:08:31.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: cephadm 2021 2021-07-31T02:08:31.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: -07-31T02:08:30.203924 2021-07-31T02:08:31.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 907 : 2021-07-31T02:08:31.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:08:31.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: cephadm 2021-07 2021-07-31T02:08:31.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: -31T02:08:30.246679+0000 mgr.x 2021-07-31T02:08:31.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: (mgr.24542) 908 : cephadm [ERR] 2021-07-31T02:08:31.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: 2021-07-31T02:08:31.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:08:31.535 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:08:31.535 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: yield 2021-07-31T02:08:31.535 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:08:31.535 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:08:31.535 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:08:31.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:08:31.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:08:31.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:08:31.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:08:31.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:08:31.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:08:31.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: task.result() 2021-07-31T02:08:31.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:08:31.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:08:30.247424+0000 mgr.x (mgr.24542) 909 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:08:31.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:08:30. 2021-07-31T02:08:31.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: 248566+0000 mgr.x (mgr.24542) 910 : cephadm [ERR] 2021-07-31T02:08:31.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:08:31.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:08:31.539 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: yield 2021-07-31T02:08:31.539 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:08:31.539 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:08:31.539 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:08:31.539 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:08:31.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:08:31.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:08:31.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:08:31.541 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:08:31.541 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:08:31.541 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:08:31.541 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:08:31.542 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:08:31.542 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:08:31.542 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: task.result() 2021-07-31T02:08:31.542 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:08:31.543 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: 2021-07-31T02:08:31.543 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:08:31.543 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: 2021-07-31T02:08:31.543 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:08:31.544 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:08:31.544 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:08:31.544 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:08:31.544 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:08:31.544 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:08:31.545 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:08:31.545 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:08:31.545 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:08:31.545 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:08:31.546 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:08:31.546 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:08:31.546 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:08:31.546 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:08:31.546 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:08:31.547 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:08:31.547 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:08:31.547 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:08:31.547 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: raise 2021-07-31T02:08:31.548 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:08:31.548 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:08:31.548 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:08:31.548 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:08:31.548 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:31 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:08:31.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: audit 2021-07-31T02:08: 2021-07-31T02:08:31.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: 30.201111+0000 mon.b (mon.2) 175 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:08:31.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: audit 2021-07-31T02:08:31.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: 2021-07-31T02:08:30.202440+0000 2021-07-31T02:08:31.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: mon.b (mon.2) 176 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:08:31.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T02:08:31.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: 31T02:08:30.203210+0000 mon.b (mon.2) 2021-07-31T02:08:31.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: 177 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:08:31.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: cephadm 2021-07 2021-07-31T02:08:31.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: -31T02:08:30.203924+0000 mgr.x (mgr.24542) 2021-07-31T02:08:31.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: 907 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:08:31.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: cephadm 2021-07- 2021-07-31T02:08:31.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: 31T02:08:30.246679+0000 mgr.x (mgr.24542 2021-07-31T02:08:31.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: ) 908 : cephadm [ERR] 2021-07-31T02:08:31.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:08:31.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:08:31.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: yield 2021-07-31T02:08:31.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:08:31.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:08:31.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:08:31.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:08:31.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:08:31.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:08:31.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:08:31.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:08:31.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:08:31.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: task.result() 2021-07-31T02:08:31.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:08:31.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: 2021-07-31T02:08:31.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:08:30.247424+0000 mgr.x (mgr.24542) 909 2021-07-31T02:08:31.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:08:31.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: cephadm 2021-07- 2021-07-31T02:08:31.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: 31T02:08:30.248566+0000 mgr.x (mgr.24542) 2021-07-31T02:08:31.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: 910 : cephadm [ERR] 2021-07-31T02:08:31.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:08:31.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:08:31.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: yield 2021-07-31T02:08:31.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:08:31.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:08:31.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:08:31.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:08:31.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:08:31.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:08:31.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:08:31.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:08:31.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:08:31.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: task.result() 2021-07-31T02:08:31.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:08:31.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: 2021-07-31T02:08:31.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:08:30.258379+0000 mgr.x (mgr.24542) 911 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:08:31.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: cephadm 2021- 2021-07-31T02:08:31.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: 07-31T02:08:30.283379+0000 mgr.x (mgr.24542 2021-07-31T02:08:31.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: ) 912 : cephadm [ERR] 2021-07-31T02:08:31.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:08:31.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:08:31.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: yield 2021-07-31T02:08:31.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:08:31.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:08:31.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:08:31.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:08:31.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:08:31.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:08:31.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:08:31.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:08:31.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:08:31.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: task.result() 2021-07-31T02:08:31.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:08:31.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:08:30.283799+0000 mgr.x (mgr.24542) 913 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:08:31.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:08:31.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:08:31.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: yield 2021-07-31T02:08:31.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:08:31.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:08:31.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:08:31.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:08:31.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:08:31.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:08:31.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:08:31.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:08:31.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:08:31.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: task.result() 2021-07-31T02:08:31.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:08:31.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: 2021-07-31T02:08:31.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:08:31.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:08:31.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:08:31.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:08:31.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:08:31.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: task.result() 2021-07-31T02:08:31.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:08:31.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:08:30.258379+0000 mgr.x (mgr.24542) 911 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:08:31.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: cephadm 2021-07 2021-07-31T02:08:31.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: -31T02:08:30.283379+0000 mgr.x (mgr.24542 2021-07-31T02:08:31.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: ) 912 : cephadm [ERR] 2021-07-31T02:08:31.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:08:31.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:08:31.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: yield 2021-07-31T02:08:31.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:08:31.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:08:31.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:08:31.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:08:31.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:08:31.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:08:31.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:08:31.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:08:31.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:08:31.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: task.result() 2021-07-31T02:08:31.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:08:31.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: 2021-07-31T02:08:31.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:08:30.283799+0000 mgr.x ( 2021-07-31T02:08:31.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: mgr.24542) 913 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:08:31.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:08:31.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:08:31.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: yield 2021-07-31T02:08:31.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:08:31.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:08:31.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:08:31.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:08:31.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:08:31.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:08:31.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:08:31.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:08:31.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:08:31.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: task.result() 2021-07-31T02:08:31.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:08:31.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: 2021-07-31T02:08:31.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:08:31.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:08:31.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:08:31.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:08:31.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:08:31.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:08:31.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:08:31.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:08:31.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:08:31.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:08:31.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:08:31.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:08:31.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:08:31.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:08:31.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:08:31.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:08:31.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:08:31.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:08:31.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: raise 2021-07-31T02:08:31.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:08:31.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:08:31.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:08:31.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:08:31.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:08:31.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:08:30.291227+0000 mgr.x (mgr.24542) 914 : cephadm [ERR] 2021-07-31T02:08:31.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:08:31.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:08:31.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: yield 2021-07-31T02:08:31.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:08:31.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:08:31.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:08:31.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:08:31.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:08:31.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:08:31.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:08:31.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:08:31.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:08:31.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: task.result() 2021-07-31T02:08:31.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:08:31.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:08:31.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: 2021-07-31T02:08:31.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:08:31.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: 2021-07-31T02:08:31.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:08:31.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:08:31.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:08:31.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:08:31.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:08:31.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:08:31.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:08:31.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:08:31.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:08:31.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:08:31.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:08:31.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:08:31.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:08:31.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:08:31.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:08:31.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:08:31.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:08:31.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:08:31.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: raise 2021-07-31T02:08:31.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:08:31.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:08:31.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:08:31.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:08:31.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:08:31.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:08:30.291227+0000 mgr.x (mgr.24542) 914 : cephadm [ERR] 2021-07-31T02:08:31.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:08:31.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:08:31.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: yield 2021-07-31T02:08:31.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:08:31.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:08:31.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:08:31.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:08:31.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:08:31.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:08:31.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:08:31.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:08:31.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:08:31.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:08:30.291609+0000 mgr.x (mgr.24542) 915 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:08:31.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:08:31.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:08:31.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: yield 2021-07-31T02:08:31.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:08:31.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:08:31.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:08:31.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:08:31.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:08:31.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:08:31.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:08:31.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:08:31.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:08:31.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: task.result() 2021-07-31T02:08:31.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:08:31.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: 2021-07-31T02:08:31.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:08:31.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: 2021-07-31T02:08:31.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:08:31.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:08:31.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:08:31.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:08:31.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:08:31.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:08:31.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:08:31.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:08:31.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:08:31.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:08:31.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:08:31.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:08:31.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:08:31.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:08:31.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:08:31.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:08:31.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:08:31.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:08:31.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: raise 2021-07-31T02:08:31.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:08:31.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:08:31.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:08:31.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:08:31.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:08:31.607 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: task.result() 2021-07-31T02:08:31.607 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:08:31.608 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:08:30.291609+0000 mgr.x 2021-07-31T02:08:31.608 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: (mgr.24542) 915 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:08:31.608 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:08:31.608 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:08:31.609 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: yield 2021-07-31T02:08:31.609 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:08:31.609 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:08:31.609 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:08:31.609 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:08:31.610 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:08:31.610 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:08:31.610 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:08:31.610 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:08:31.611 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:08:31.611 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: task.result() 2021-07-31T02:08:31.611 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:08:31.611 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: 2021-07-31T02:08:31.612 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:08:31.612 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: 2021-07-31T02:08:31.612 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:08:31.612 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:08:31.612 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:08:31.613 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:08:31.613 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:08:31.613 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:08:31.613 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:08:31.614 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:08:31.614 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:08:31.614 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:08:31.614 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:08:31.614 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:08:31.615 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:08:31.615 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:08:31.615 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:08:31.615 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:08:31.616 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:08:31.616 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:08:31.616 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: raise 2021-07-31T02:08:31.616 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:08:31.617 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:08:31.617 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:08:31.617 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:08:31.617 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:31 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:08:32.453 INFO:tasks.workunit.client.0.smithi045.stdout: 202: throughput=4.98MB/sec pending data=0 2021-07-31T02:08:32.453 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-xDm66Ssee6Zx2rO off=2507089 len=1206908 2021-07-31T02:08:32.461 INFO:tasks.workunit.client.0.smithi045.stdout:op 1010 completed, throughput=4.99MB/sec 2021-07-31T02:08:32.462 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-xXKjQLrR8dBjBYN off=3549727 len=704234 2021-07-31T02:08:32.462 INFO:tasks.workunit.client.0.smithi045.stdout:op 1011 completed, throughput=4.99MB/sec 2021-07-31T02:08:32.463 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-WGlNjZ21vIVF6Gp off=19415407 len=676319 2021-07-31T02:08:32.467 INFO:tasks.workunit.client.0.smithi045.stdout:op 1012 completed, throughput=4.99MB/sec 2021-07-31T02:08:32.467 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-0VCCewCcLv_IDKR off=17068204 len=1898988 2021-07-31T02:08:32.477 INFO:tasks.workunit.client.0.smithi045.stdout:op 1013 completed, throughput=5MB/sec 2021-07-31T02:08:33.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:32 smithi154 conmon[34553]: cluster 2021-07-31T02:08:32.093720+0000 mgr.x (mgr.24542) 916 : cluster [DBG] pgmap v1511: 9 pgs: 9 active+clean; 127 GiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.5 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2021-07-31T02:08:33.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:32 smithi045 conmon[39718]: cluster 2021-07-31T02:08:32.093720+0000 mgr.x ( 2021-07-31T02:08:33.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:32 smithi045 conmon[39718]: mgr.24542) 916 : cluster [DBG] pgmap v1511: 9 pgs: 9 active+clean; 127 GiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.5 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2021-07-31T02:08:33.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:32 smithi045 conmon[33655]: cluster 2021-07-31T02:08:32.093720+0000 mgr.x (mgr.24542) 916 2021-07-31T02:08:33.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:32 smithi045 conmon[33655]: : cluster [DBG] pgmap v1511: 9 pgs: 9 active+clean; 127 GiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.5 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2021-07-31T02:08:33.478 INFO:tasks.workunit.client.0.smithi045.stdout: 203: throughput=4.98MB/sec pending data=0 2021-07-31T02:08:33.478 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-XPnVLUTUyV_NdEA off=552745 len=1663078 2021-07-31T02:08:33.480 INFO:tasks.workunit.client.0.smithi045.stdout:op 1014 completed, throughput=4.99MB/sec 2021-07-31T02:08:33.480 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-j9myoAgZv_jGFQ6 off=7608017 len=703577 2021-07-31T02:08:33.482 INFO:tasks.workunit.client.0.smithi045.stdout:op 1015 completed, throughput=4.99MB/sec 2021-07-31T02:08:33.482 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-qUR-tiKUYBYCTPk off=3623961 len=591913 2021-07-31T02:08:33.483 INFO:tasks.workunit.client.0.smithi045.stdout:op 1016 completed, throughput=4.99MB/sec 2021-07-31T02:08:33.483 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-nXJjCfSZN8dbQ7o off=3056108 len=523692 2021-07-31T02:08:33.487 INFO:tasks.workunit.client.0.smithi045.stdout:op 1017 completed, throughput=4.99MB/sec 2021-07-31T02:08:33.488 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-rQNTC9mM2nUPrE- off=22134736 len=1580218 2021-07-31T02:08:33.494 INFO:tasks.workunit.client.0.smithi045.stdout:op 1018 completed, throughput=5MB/sec 2021-07-31T02:08:34.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:08:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:08:33.544Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:08:34.027 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:08:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:08:33.544Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:08:34.027 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:08:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:08:33.544Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:08:34.494 INFO:tasks.workunit.client.0.smithi045.stdout: 204: throughput=4.98MB/sec pending data=0 2021-07-31T02:08:34.494 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ZeCn5gS-tngfawL off=440404 len=2096320 2021-07-31T02:08:34.501 INFO:tasks.workunit.client.0.smithi045.stdout:op 1019 completed, throughput=4.99MB/sec 2021-07-31T02:08:34.502 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-PeRrFuaz1MFGOfu off=1190803 len=218894 2021-07-31T02:08:34.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:34 smithi154 conmon[34553]: cluster 2021-07-31T02:08:34.094214+0000 mgr.x (mgr.24542) 917 : cluster [DBG] pgmap v1512: 9 pgs: 9 active+clean; 127 GiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 2.1 MiB/s rd, 4.2 MiB/s wr, 6 op/s 2021-07-31T02:08:34.505 INFO:tasks.workunit.client.0.smithi045.stdout:op 1020 completed, throughput=4.99MB/sec 2021-07-31T02:08:34.505 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-88X14lSnuUg4FJ2 off=19172735 len=133787 2021-07-31T02:08:34.507 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:08:34.507 INFO:tasks.workunit.client.0.smithi045.stdout:op 1021 completed, throughput=4.99MB/sec 2021-07-31T02:08:34.508 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-w9zICT-MsXvZN4K off=9622519 len=128846 2021-07-31T02:08:34.509 INFO:tasks.workunit.client.0.smithi045.stdout:op 1022 completed, throughput=4.99MB/sec 2021-07-31T02:08:34.510 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj--p6vhMWKE9ZDVaG off=1231991 len=1363229 2021-07-31T02:08:34.515 INFO:tasks.workunit.client.0.smithi045.stdout:op 1023 completed, throughput=5MB/sec 2021-07-31T02:08:34.516 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-9RM0E7N8FgaqBV0 off=1210813 len=879524 2021-07-31T02:08:34.521 INFO:tasks.workunit.client.0.smithi045.stdout:op 1024 completed, throughput=5MB/sec 2021-07-31T02:08:34.521 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-3BEtQU8JUMgBuhO off=3325941 len=301142 2021-07-31T02:08:34.522 INFO:tasks.workunit.client.0.smithi045.stdout:op 1025 completed, throughput=5MB/sec 2021-07-31T02:08:34.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:34 smithi045 conmon[33655]: cluster 2021-07-31T02:08:34.094214+0000 mgr.x (mgr.24542) 917 : cluster 2021-07-31T02:08:34.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:34 smithi045 conmon[33655]: [DBG] pgmap v1512: 9 pgs: 9 active+clean; 127 GiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 2.1 MiB/s rd, 4.2 MiB/s wr, 6 op/s 2021-07-31T02:08:34.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:34 smithi045 conmon[39718]: cluster 2021-07-31T02:08:34.094214+0000 mgr.x 2021-07-31T02:08:34.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:34 smithi045 conmon[39718]: (mgr.24542) 917 : cluster [DBG] pgmap v1512: 9 pgs: 9 active+clean; 127 GiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 2.1 MiB/s rd, 4.2 MiB/s wr, 6 op/s 2021-07-31T02:08:34.785 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:08:34.786 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 23m ago 24m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:08:34.786 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 21m ago 23m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:08:34.786 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 21m ago 27m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:08:34.787 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 23m ago 29m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:08:34.788 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 23m ago 29m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:08:34.789 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 21m ago 27m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:08:34.790 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 23m ago 27m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:08:34.791 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 23m ago 24m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:08:34.791 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 21m ago 24m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:08:34.791 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 23m ago 27m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:08:34.791 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 23m ago 26m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:08:34.792 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 23m ago 26m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:08:34.792 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 23m ago 26m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:08:34.792 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 21m ago 25m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:08:34.792 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 21m ago 25m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:08:34.792 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 21m ago 25m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:08:34.793 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 21m ago 24m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:08:34.793 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 21m ago 24m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:08:35.122 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:08:35.122 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:08:35.123 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:08:35.123 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:08:35.123 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:08:35.123 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:08:35.123 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:08:35.124 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:08:35.124 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:08:35.124 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:08:35.124 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:08:35.125 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:08:35.125 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:08:35.125 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:08:35.125 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:08:35.126 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:08:35.126 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:08:35.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:35 smithi154 conmon[34553]: audit 2021-07-31T02:08:34.491466+0000 mgr.x (mgr.24542) 918 : audit [DBG] from='client.15843 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:08:35.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:35 smithi154 conmon[34553]: audit 2021-07-31T02:08:34.777067+0000 mgr.x (mgr.24542) 919 : audit [DBG] from='client.15849 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:08:35.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:35 smithi154 conmon[34553]: audit 2021-07-31T02:08:35.121941+0000 mon.a (mon.0) 1861 : audit [DBG] from='client.? 172.21.15.45:0/1288080043' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:08:35.522 INFO:tasks.workunit.client.0.smithi045.stdout: 205: throughput=4.98MB/sec pending data=0 2021-07-31T02:08:35.522 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-P6niQ39jPJBkK4T off=12621 len=1053313 2021-07-31T02:08:35.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:35 smithi045 conmon[33655]: audit 2021-07-31T02:08:34.491466+0000 mgr.x ( 2021-07-31T02:08:35.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:35 smithi045 conmon[33655]: mgr.24542) 918 : audit [DBG] from='client.15843 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:08:35.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:35 smithi045 conmon[33655]: audit 2021-07-31T02:08:34.777067+0000 mgr.x (mgr.24542) 919 : audit [DBG] from='client.15849 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:08:35.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:35 smithi045 conmon[33655]: audit 2021-07-31T02:08:35 2021-07-31T02:08:35.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:35 smithi045 conmon[33655]: .121941+0000 mon.a (mon.0) 1861 : 2021-07-31T02:08:35.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:35 smithi045 conmon[33655]: audit [DBG] from='client.? 172.21.15.45:0/1288080043' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:08:35.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:35 smithi045 conmon[39718]: audit 2021-07-31T02 2021-07-31T02:08:35.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:35 smithi045 conmon[39718]: :08:34.491466+0000 mgr.x (mgr.24542) 918 : audit [DBG] from='client.15843 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:08:35.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:35 smithi045 conmon[39718]: audit 2021 2021-07-31T02:08:35.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:35 smithi045 conmon[39718]: -07-31T02:08 2021-07-31T02:08:35.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:35 smithi045 conmon[39718]: :34.777067+0000 mgr.x (mgr.24542 2021-07-31T02:08:35.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:35 smithi045 conmon[39718]: ) 919 : audit [DBG] from='client.15849 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:08:35.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:35 smithi045 conmon[39718]: audit 2021-07-31T02:08:35.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:35 smithi045 conmon[39718]: 2021-07-31T 2021-07-31T02:08:35.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:35 smithi045 conmon[39718]: 02:08:35.121941+0000 mon.a 2021-07-31T02:08:35.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:35 smithi045 conmon[39718]: (mon.0) 1861 : audit [DBG] 2021-07-31T02:08:35.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:35 smithi045 conmon[39718]: from='client.? 172.21.15.45:0/1288080043' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:08:35.530 INFO:tasks.workunit.client.0.smithi045.stdout:op 1026 completed, throughput=4.98MB/sec 2021-07-31T02:08:35.531 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-1aOprUMhSHPZ4mA off=1911807 len=1933604 2021-07-31T02:08:35.531 INFO:tasks.workunit.client.0.smithi045.stdout:op 1027 completed, throughput=4.99MB/sec 2021-07-31T02:08:35.531 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Bg0uMvUqiytl8sM off=10467510 len=1032279 2021-07-31T02:08:35.532 INFO:tasks.workunit.client.0.smithi045.stdout:op 1028 completed, throughput=4.99MB/sec 2021-07-31T02:08:35.532 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-eZ677Lgnvv5S_pG off=4610730 len=1343602 2021-07-31T02:08:35.538 INFO:tasks.workunit.client.0.smithi045.stdout:op 1029 completed, throughput=5MB/sec 2021-07-31T02:08:36.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:36 smithi154 conmon[34553]: cluster 2021-07-31T02:08:36.094815+0000 mgr.x (mgr.24542) 920 : cluster [DBG] pgmap v1513: 9 pgs: 9 active+clean; 127 GiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.2 MiB/s rd, 3.0 MiB/s wr, 3 op/s 2021-07-31T02:08:36.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:36 smithi045 conmon[39718]: cluster 2021-07-31T02:08:36.094815+0000 mgr.x 2021-07-31T02:08:36.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:36 smithi045 conmon[39718]: (mgr.24542) 920 : cluster [DBG] pgmap v1513: 9 pgs: 9 active+clean; 127 GiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.2 MiB/s rd, 3.0 MiB/s wr, 3 op/s 2021-07-31T02:08:36.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:36 smithi045 conmon[33655]: cluster 2021-07-31T02:08:36.094815+0000 mgr.x 2021-07-31T02:08:36.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:36 smithi045 conmon[33655]: (mgr.24542) 920 : cluster [DBG] pgmap v1513: 9 pgs: 9 active+clean; 127 GiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.2 MiB/s rd, 3.0 MiB/s wr, 3 op/s 2021-07-31T02:08:36.538 INFO:tasks.workunit.client.0.smithi045.stdout: 206: throughput=4.98MB/sec pending data=0 2021-07-31T02:08:36.538 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-eAzak_Rxde5_uGH off=16540510 len=1488110 2021-07-31T02:08:36.541 INFO:tasks.workunit.client.0.smithi045.stdout:op 1030 completed, throughput=4.98MB/sec 2021-07-31T02:08:36.541 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-4rQt6c4ksDyOHms off=1540414 len=121478 2021-07-31T02:08:36.544 INFO:tasks.workunit.client.0.smithi045.stdout:op 1031 completed, throughput=4.98MB/sec 2021-07-31T02:08:36.544 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-rC4z62U8n2z549p off=17054343 len=88209 2021-07-31T02:08:36.545 INFO:tasks.workunit.client.0.smithi045.stdout:op 1032 completed, throughput=4.98MB/sec 2021-07-31T02:08:36.545 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-CEGlKlXIH7rwPh6 off=6884716 len=1097862 2021-07-31T02:08:36.551 INFO:tasks.workunit.client.0.smithi045.stdout:op 1033 completed, throughput=4.99MB/sec 2021-07-31T02:08:36.551 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-a3zXFgHEEkTEABC off=15231253 len=1385315 2021-07-31T02:08:36.557 INFO:tasks.workunit.client.0.smithi045.stdout:op 1034 completed, throughput=5MB/sec 2021-07-31T02:08:36.557 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-71uX8-5Q9o0gvY6 off=438744 len=652899 2021-07-31T02:08:36.558 INFO:tasks.workunit.client.0.smithi045.stdout:op 1035 completed, throughput=5MB/sec 2021-07-31T02:08:36.558 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-HoQYT0Z9hIVt16V off=8558561 len=198767 2021-07-31T02:08:36.561 INFO:tasks.workunit.client.0.smithi045.stdout:op 1036 completed, throughput=5MB/sec 2021-07-31T02:08:36.561 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-KTFQGEVdIOen2GS off=6524107 len=881806 2021-07-31T02:08:36.563 INFO:tasks.workunit.client.0.smithi045.stdout:op 1037 completed, throughput=5MB/sec 2021-07-31T02:08:37.564 INFO:tasks.workunit.client.0.smithi045.stdout: 207: throughput=4.98MB/sec pending data=0 2021-07-31T02:08:37.564 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-oAWseaTpHXryReu off=4285497 len=2011480 2021-07-31T02:08:37.572 INFO:tasks.workunit.client.0.smithi045.stdout:op 1038 completed, throughput=4.99MB/sec 2021-07-31T02:08:37.573 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-_cn3koLs2YDTFV_ off=9536123 len=330140 2021-07-31T02:08:37.574 INFO:tasks.workunit.client.0.smithi045.stdout:op 1039 completed, throughput=4.99MB/sec 2021-07-31T02:08:37.574 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-sOc_qWtMgjj7m_s off=6801600 len=536495 2021-07-31T02:08:37.578 INFO:tasks.workunit.client.0.smithi045.stdout:op 1040 completed, throughput=4.99MB/sec 2021-07-31T02:08:37.578 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-5ih8l5a7eAuAYbb off=17313425 len=49024 2021-07-31T02:08:37.581 INFO:tasks.workunit.client.0.smithi045.stdout:op 1041 completed, throughput=4.99MB/sec 2021-07-31T02:08:37.581 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ofk1vHwhfebvIJN off=5281141 len=726768 2021-07-31T02:08:37.582 INFO:tasks.workunit.client.0.smithi045.stdout:op 1042 completed, throughput=5MB/sec 2021-07-31T02:08:37.582 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-yHg5zISRD1FHQ0c off=689054 len=1015105 2021-07-31T02:08:37.583 INFO:tasks.workunit.client.0.smithi045.stdout:op 1043 completed, throughput=5MB/sec 2021-07-31T02:08:37.583 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-J6K3BjSPdYDpIfj off=1566312 len=866272 2021-07-31T02:08:37.585 INFO:tasks.workunit.client.0.smithi045.stdout:op 1044 completed, throughput=5MB/sec 2021-07-31T02:08:38.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:38 smithi154 conmon[34553]: cluster 2021-07-31T02:08:38.095313+0000 mgr.x (mgr.24542) 921 : cluster [DBG] 2021-07-31T02:08:38.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:38 smithi154 conmon[34553]: pgmap v1514: 9 pgs: 9 active+clean; 127 GiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.2 MiB/s rd, 3.0 MiB/s wr, 3 op/s 2021-07-31T02:08:38.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:38 smithi045 conmon[39718]: cluster 2021-07-31T02:08:38.095313+0000 mgr.x (mgr.24542) 921 : cluster [DBG] pgmap v1514: 9 pgs: 9 active+clean; 127 GiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.2 MiB/s rd, 3.0 MiB/s wr, 3 op/s 2021-07-31T02:08:38.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:38 smithi045 conmon[33655]: cluster 2021-07-31T02:08:38.095313+0000 mgr.x (mgr.24542) 921 : cluster 2021-07-31T02:08:38.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:38 smithi045 conmon[33655]: [DBG] pgmap v1514: 9 pgs: 9 active+clean; 127 GiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.2 MiB/s rd, 3.0 MiB/s wr, 3 op/s 2021-07-31T02:08:38.586 INFO:tasks.workunit.client.0.smithi045.stdout: 208: throughput=4.98MB/sec pending data=0 2021-07-31T02:08:38.586 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ruMiaCFsvN1gSq_ off=12172397 len=1061131 2021-07-31T02:08:38.592 INFO:tasks.workunit.client.0.smithi045.stdout:op 1045 completed, throughput=4.98MB/sec 2021-07-31T02:08:38.592 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-alLkx2LzWBFCLhW off=5966146 len=571988 2021-07-31T02:08:38.595 INFO:tasks.workunit.client.0.smithi045.stdout:op 1046 completed, throughput=4.99MB/sec 2021-07-31T02:08:38.596 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-2uVib0SRNaXQSS6 off=10121187 len=1529379 2021-07-31T02:08:38.603 INFO:tasks.workunit.client.0.smithi045.stdout:op 1047 completed, throughput=4.99MB/sec 2021-07-31T02:08:38.603 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-LrcrG6zLh-5eMvV off=23408719 len=692660 2021-07-31T02:08:38.604 INFO:tasks.workunit.client.0.smithi045.stdout:op 1048 completed, throughput=5MB/sec 2021-07-31T02:08:38.605 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-b0uKqDUGVSr8mEs off=14091493 len=280621 2021-07-31T02:08:38.608 INFO:tasks.workunit.client.0.smithi045.stdout:op 1049 completed, throughput=5MB/sec 2021-07-31T02:08:38.608 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-2M7oBM4AkKWSb5z off=3639714 len=30426 2021-07-31T02:08:38.611 INFO:tasks.workunit.client.0.smithi045.stdout:op 1050 completed, throughput=5MB/sec 2021-07-31T02:08:38.611 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-lbqPLd3YodLcvSG off=5210158 len=1403215 2021-07-31T02:08:38.617 INFO:tasks.workunit.client.0.smithi045.stdout:op 1051 completed, throughput=5MB/sec 2021-07-31T02:08:39.617 INFO:tasks.workunit.client.0.smithi045.stdout: 209: throughput=4.98MB/sec pending data=0 2021-07-31T02:08:39.617 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj--J_0rv1lf7N0t0N off=9212314 len=80929 2021-07-31T02:08:39.620 INFO:tasks.workunit.client.0.smithi045.stdout:op 1052 completed, throughput=4.98MB/sec 2021-07-31T02:08:39.620 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-vus7rTduTTXjLoz off=3872862 len=2055400 2021-07-31T02:08:39.624 INFO:tasks.workunit.client.0.smithi045.stdout:op 1053 completed, throughput=4.99MB/sec 2021-07-31T02:08:39.624 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-CW14914wwHU5Q4I off=6851490 len=1080341 2021-07-31T02:08:39.630 INFO:tasks.workunit.client.0.smithi045.stdout:op 1054 completed, throughput=5MB/sec 2021-07-31T02:08:39.630 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-jrb-q-MX4Amsppc off=4542190 len=368276 2021-07-31T02:08:39.632 INFO:tasks.workunit.client.0.smithi045.stdout:op 1055 completed, throughput=5MB/sec 2021-07-31T02:08:39.633 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-lxTsQ6qwBeRx_nt off=1062123 len=1188182 2021-07-31T02:08:39.634 INFO:tasks.workunit.client.0.smithi045.stdout:op 1056 completed, throughput=5MB/sec 2021-07-31T02:08:40.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:40 smithi154 conmon[34553]: cluster 2021-07-31T02:08:40.095822+0000 mgr.x (mgr.24542) 922 : cluster [DBG] pgmap v1515: 9 pgs: 9 active+clean; 127 GiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 2.0 MiB/s rd, 4.4 MiB/s wr, 6 op/s 2021-07-31T02:08:40.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:40 smithi045 conmon[39718]: cluster 2021-07-31T02:08:40.095822+0000 mgr.x (mgr.24542) 922 : cluster [DBG] 2021-07-31T02:08:40.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:40 smithi045 conmon[39718]: pgmap v1515: 9 pgs: 9 active+clean; 127 GiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 2.0 MiB/s rd, 4.4 MiB/s wr, 6 op/s 2021-07-31T02:08:40.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:40 smithi045 conmon[33655]: cluster 2021-07-31T02:08:40.095822+0000 mgr.x (mgr.24542) 922 2021-07-31T02:08:40.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:40 smithi045 conmon[33655]: : cluster [DBG] pgmap v1515: 9 pgs: 9 active+clean; 127 GiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 2.0 MiB/s rd, 4.4 MiB/s wr, 6 op/s 2021-07-31T02:08:40.634 INFO:tasks.workunit.client.0.smithi045.stdout: 210: throughput=4.98MB/sec pending data=0 2021-07-31T02:08:40.635 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-7lgvvcm5Wth0282 off=9077813 len=506622 2021-07-31T02:08:40.636 INFO:tasks.workunit.client.0.smithi045.stdout:op 1057 completed, throughput=4.98MB/sec 2021-07-31T02:08:40.636 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-q4T8yIwfoDLuyV9 off=8095238 len=1688838 2021-07-31T02:08:40.638 INFO:tasks.workunit.client.0.smithi045.stdout:op 1058 completed, throughput=4.99MB/sec 2021-07-31T02:08:40.639 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-WaudyVAfHxn4HuY off=1760772 len=1671751 2021-07-31T02:08:40.641 INFO:tasks.workunit.client.0.smithi045.stdout:op 1059 completed, throughput=5MB/sec 2021-07-31T02:08:40.642 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Y40bEpieWJKBoR3 off=7612743 len=1897240 2021-07-31T02:08:40.652 INFO:tasks.workunit.client.0.smithi045.stdout:op 1060 completed, throughput=5MB/sec 2021-07-31T02:08:41.652 INFO:tasks.workunit.client.0.smithi045.stdout: 211: throughput=4.98MB/sec pending data=0 2021-07-31T02:08:41.652 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-idby92tRNrJnCPA off=1416093 len=1334292 2021-07-31T02:08:41.655 INFO:tasks.workunit.client.0.smithi045.stdout:op 1061 completed, throughput=4.99MB/sec 2021-07-31T02:08:41.655 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-pHDnwZHo1PwwfRl off=4073854 len=581328 2021-07-31T02:08:41.656 INFO:tasks.workunit.client.0.smithi045.stdout:op 1062 completed, throughput=4.99MB/sec 2021-07-31T02:08:41.656 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-HjKV_2NXTTTBFtP off=7100376 len=1650024 2021-07-31T02:08:41.658 INFO:tasks.workunit.client.0.smithi045.stdout:op 1063 completed, throughput=5MB/sec 2021-07-31T02:08:41.658 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-eNjo41nON66byyZ off=3317168 len=1967063 2021-07-31T02:08:41.660 INFO:tasks.workunit.client.0.smithi045.stdout:op 1064 completed, throughput=5MB/sec 2021-07-31T02:08:42.660 INFO:tasks.workunit.client.0.smithi045.stdout: 212: throughput=4.98MB/sec pending data=0 2021-07-31T02:08:42.661 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-2JvEp9ZkDSAc53R off=8809080 len=215114 2021-07-31T02:08:42.662 INFO:tasks.workunit.client.0.smithi045.stdout:op 1065 completed, throughput=4.98MB/sec 2021-07-31T02:08:42.662 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-AQtq7nyNxkSD4hT off=819333 len=830438 2021-07-31T02:08:42.667 INFO:tasks.workunit.client.0.smithi045.stdout:op 1066 completed, throughput=4.99MB/sec 2021-07-31T02:08:42.667 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-hxX7c7UO7MGKNk2 off=6785532 len=2051473 2021-07-31T02:08:42.670 INFO:tasks.workunit.client.0.smithi045.stdout:op 1067 completed, throughput=5MB/sec 2021-07-31T02:08:42.671 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-TYtGkDrLw5TtEMu off=4226107 len=575300 2021-07-31T02:08:42.672 INFO:tasks.workunit.client.0.smithi045.stdout:op 1068 completed, throughput=5MB/sec 2021-07-31T02:08:42.672 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-njkWucNjrKC5xmc off=3449637 len=201456 2021-07-31T02:08:42.676 INFO:tasks.workunit.client.0.smithi045.stdout:op 1069 completed, throughput=5MB/sec 2021-07-31T02:08:42.676 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ZQq0z1g5yRwmdON off=6978573 len=1444766 2021-07-31T02:08:42.682 INFO:tasks.workunit.client.0.smithi045.stdout:op 1070 completed, throughput=5MB/sec 2021-07-31T02:08:43.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:42 smithi154 conmon[34553]: cluster 2021-07-31T02:08:42.096305+0000 mgr.x (mgr.24542) 923 : cluster [DBG] pgmap v1516: 9 pgs: 9 active+clean; 127 GiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 1.4 MiB/s rd, 2.9 MiB/s wr, 4 op/s 2021-07-31T02:08:43.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:42 smithi045 conmon[39718]: cluster 2021-07-31T02:08:42.096305+0000 mgr.x (mgr.24542) 923 : cluster 2021-07-31T02:08:43.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:42 smithi045 conmon[39718]: [DBG] pgmap v1516: 9 pgs: 9 active+clean; 127 GiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 1.4 MiB/s rd, 2.9 MiB/s wr, 4 op/s 2021-07-31T02:08:43.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:42 smithi045 conmon[33655]: cluster 2021-07-31T02:08:42.096305+0000 mgr.x (mgr.24542) 923 : cluster [DBG] 2021-07-31T02:08:43.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:42 smithi045 conmon[33655]: pgmap v1516: 9 pgs: 9 active+clean; 127 GiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 1.4 MiB/s rd, 2.9 MiB/s wr, 4 op/s 2021-07-31T02:08:43.682 INFO:tasks.workunit.client.0.smithi045.stdout: 213: throughput=4.98MB/sec pending data=0 2021-07-31T02:08:43.682 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-HpL6qvYQuOLy2bG off=3322215 len=1073457 2021-07-31T02:08:43.684 INFO:tasks.workunit.client.0.smithi045.stdout:op 1071 completed, throughput=4.99MB/sec 2021-07-31T02:08:43.685 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-OTPTKlrWMjazS2l off=8273134 len=470909 2021-07-31T02:08:43.688 INFO:tasks.workunit.client.0.smithi045.stdout:op 1072 completed, throughput=4.99MB/sec 2021-07-31T02:08:43.689 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-z9ob_xqMJC6YTBI off=5447839 len=1585655 2021-07-31T02:08:43.695 INFO:tasks.workunit.client.0.smithi045.stdout:op 1073 completed, throughput=5MB/sec 2021-07-31T02:08:43.695 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-riLaJG7vVLIvkLU off=5106662 len=369208 2021-07-31T02:08:43.696 INFO:tasks.workunit.client.0.smithi045.stdout:op 1074 completed, throughput=5MB/sec 2021-07-31T02:08:43.697 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-bX0gtZ1E5C26qP5 off=1460517 len=731865 2021-07-31T02:08:43.701 INFO:tasks.workunit.client.0.smithi045.stdout:op 1075 completed, throughput=5MB/sec 2021-07-31T02:08:43.701 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-4NCDynZNJTGzmlZ off=6256560 len=184781 2021-07-31T02:08:43.702 INFO:tasks.workunit.client.0.smithi045.stdout:op 1076 completed, throughput=5MB/sec 2021-07-31T02:08:44.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:08:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:08:43.544Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:08:44.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:08:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:08:43.544Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:08:44.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:08:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:08:43.544Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:08:44.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:44 smithi154 conmon[34553]: cluster 2021-07-31T02:08:44.096841+0000 mgr.x (mgr.24542) 924 : cluster [DBG] pgmap v1517: 9 pgs: 9 active+clean; 127 GiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 2.8 MiB/s rd, 3.6 MiB/s wr, 6 op/s 2021-07-31T02:08:44.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:44 smithi045 conmon[33655]: cluster 2021-07-31T02:08:44.096841+0000 mgr.x (mgr.24542) 924 : cluster [DBG] 2021-07-31T02:08:44.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:44 smithi045 conmon[33655]: pgmap v1517: 9 pgs: 9 active+clean; 127 GiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 2.8 MiB/s rd, 3.6 MiB/s wr, 6 op/s 2021-07-31T02:08:44.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:44 smithi045 conmon[39718]: cluster 2021-07-31T02:08:44.096841+0000 mgr.x (mgr.24542) 924 : cluster [DBG] 2021-07-31T02:08:44.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:44 smithi045 conmon[39718]: pgmap v1517: 9 pgs: 9 active+clean; 127 GiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 2.8 MiB/s rd, 3.6 MiB/s wr, 6 op/s 2021-07-31T02:08:44.702 INFO:tasks.workunit.client.0.smithi045.stdout: 214: throughput=4.98MB/sec pending data=0 2021-07-31T02:08:44.702 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-BQHiJgWT6u9Qf1O off=11213456 len=916085 2021-07-31T02:08:44.707 INFO:tasks.workunit.client.0.smithi045.stdout:op 1077 completed, throughput=4.98MB/sec 2021-07-31T02:08:44.708 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Jrqps1mUaGURVeT off=3246095 len=1420829 2021-07-31T02:08:44.714 INFO:tasks.workunit.client.0.smithi045.stdout:op 1078 completed, throughput=4.99MB/sec 2021-07-31T02:08:44.714 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-vQ-YOdggX2KnotB off=19153769 len=1134249 2021-07-31T02:08:44.716 INFO:tasks.workunit.client.0.smithi045.stdout:op 1079 completed, throughput=4.99MB/sec 2021-07-31T02:08:44.716 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-iYPI9LIbPXcCQSy off=856605 len=1641494 2021-07-31T02:08:44.719 INFO:tasks.workunit.client.0.smithi045.stdout:op 1080 completed, throughput=5MB/sec 2021-07-31T02:08:44.720 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-j1rmNezSU7G3ZRX off=12472578 len=1470991 2021-07-31T02:08:44.722 INFO:tasks.workunit.client.0.smithi045.stdout:op 1081 completed, throughput=5.01MB/sec 2021-07-31T02:08:45.722 INFO:tasks.workunit.client.0.smithi045.stdout: 215: throughput=4.98MB/sec pending data=0 2021-07-31T02:08:45.722 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-GbcY-EpI09ri07i off=1977926 len=1132379 2021-07-31T02:08:45.728 INFO:tasks.workunit.client.0.smithi045.stdout:op 1082 completed, throughput=4.99MB/sec 2021-07-31T02:08:45.728 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-MbOzC8BO8klGl_a off=683003 len=1712755 2021-07-31T02:08:45.731 INFO:tasks.workunit.client.0.smithi045.stdout:op 1083 completed, throughput=5MB/sec 2021-07-31T02:08:45.731 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-TaxpQv1u5br5kTi off=1363058 len=766224 2021-07-31T02:08:45.736 INFO:tasks.workunit.client.0.smithi045.stdout:op 1084 completed, throughput=5MB/sec 2021-07-31T02:08:45.737 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-z3EAEpYycYi63DA off=670538 len=566823 2021-07-31T02:08:45.740 INFO:tasks.workunit.client.0.smithi045.stdout:op 1085 completed, throughput=5MB/sec 2021-07-31T02:08:46.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:46 smithi154 conmon[34553]: cluster 2021-07-31T02:08:46.097256+0000 mgr.x (mgr.24542) 925 : cluster [DBG] pgmap v1518: 9 pgs: 9 active+clean; 127 GiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2021-07-31T02:08:46.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:46 smithi045 conmon[33655]: cluster 2021-07-31T02:08:46.097256+0000 mgr.x (mgr.24542) 925 : cluster [DBG] pgmap v1518: 9 pgs: 9 active+clean; 127 GiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2021-07-31T02:08:46.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:46 smithi045 conmon[39718]: cluster 2021-07-31T02:08:46.097256+0000 mgr.x (mgr.24542) 925 : cluster [DBG] pgmap v1518: 9 pgs: 9 active+clean; 127 GiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2021-07-31T02:08:46.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:46 smithi045 conmon[39718]: 2021-07-31T02:08:46.740 INFO:tasks.workunit.client.0.smithi045.stdout: 216: throughput=4.98MB/sec pending data=0 2021-07-31T02:08:46.741 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-esqEBgaTlYC5FL5 off=4447596 len=335785 2021-07-31T02:08:46.741 INFO:tasks.workunit.client.0.smithi045.stdout:op 1086 completed, throughput=4.98MB/sec 2021-07-31T02:08:46.742 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-I8CIWJqDA1x75Af off=1516045 len=1743849 2021-07-31T02:08:46.745 INFO:tasks.workunit.client.0.smithi045.stdout:op 1087 completed, throughput=4.99MB/sec 2021-07-31T02:08:46.745 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ifo3042bwz5ZGr9 off=3972285 len=1659050 2021-07-31T02:08:46.753 INFO:tasks.workunit.client.0.smithi045.stdout:op 1088 completed, throughput=4.99MB/sec 2021-07-31T02:08:46.753 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ujS-1FxJayK5sJL off=265021 len=1895661 2021-07-31T02:08:46.757 INFO:tasks.workunit.client.0.smithi045.stdout:op 1089 completed, throughput=5MB/sec 2021-07-31T02:08:47.757 INFO:tasks.workunit.client.0.smithi045.stdout: 217: throughput=4.98MB/sec pending data=0 2021-07-31T02:08:47.758 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-4PLRoznGxCyNIG9 off=11310912 len=1955135 2021-07-31T02:08:47.759 INFO:tasks.workunit.client.0.smithi045.stdout:op 1090 completed, throughput=4.99MB/sec 2021-07-31T02:08:47.760 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-uUo2D-drHKw-Hfc off=9207737 len=1406578 2021-07-31T02:08:47.767 INFO:tasks.workunit.client.0.smithi045.stdout:op 1091 completed, throughput=4.99MB/sec 2021-07-31T02:08:47.767 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-GZRcyGO-RyUil0x off=11112562 len=880789 2021-07-31T02:08:47.773 INFO:tasks.workunit.client.0.smithi045.stdout:op 1092 completed, throughput=5MB/sec 2021-07-31T02:08:47.773 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Qchp3YTziCZTbu3 off=2088694 len=675302 2021-07-31T02:08:47.777 INFO:tasks.workunit.client.0.smithi045.stdout:op 1093 completed, throughput=5MB/sec 2021-07-31T02:08:48.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:48 smithi154 conmon[34553]: cluster 2021-07-31T02:08:48.097933+0000 mgr.x (mgr.24542) 926 : cluster [DBG] pgmap v1519: 9 pgs: 9 active+clean; 127 GiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2021-07-31T02:08:48.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:48 smithi045 conmon[33655]: cluster 2021-07-31T02:08:48.097933+0000 mgr.x (mgr.24542) 926 : cluster [DBG] pgmap v1519: 9 pgs: 9 active+clean; 127 GiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2021-07-31T02:08:48.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:48 smithi045 conmon[39718]: cluster 2021-07-31T02:08:48.097933+0000 mgr.x (mgr.24542) 926 : cluster [DBG] pgmap v1519: 9 pgs: 9 active+clean; 127 GiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2021-07-31T02:08:48.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:48 smithi045 conmon[39718]: 2021-07-31T02:08:48.777 INFO:tasks.workunit.client.0.smithi045.stdout: 218: throughput=4.98MB/sec pending data=0 2021-07-31T02:08:48.778 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-0X9ihBllZ7FEDJk off=8426924 len=1439061 2021-07-31T02:08:48.780 INFO:tasks.workunit.client.0.smithi045.stdout:op 1094 completed, throughput=4.98MB/sec 2021-07-31T02:08:48.780 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-BVumI-GK1lG5wwU off=671508 len=1683691 2021-07-31T02:08:48.783 INFO:tasks.workunit.client.0.smithi045.stdout:op 1095 completed, throughput=4.99MB/sec 2021-07-31T02:08:48.784 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-hhhs6R0geONWsAk off=5768890 len=1211534 2021-07-31T02:08:48.785 INFO:tasks.workunit.client.0.smithi045.stdout:op 1096 completed, throughput=5MB/sec 2021-07-31T02:08:48.786 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-uBsRIpX040hgNdL off=2236664 len=212854 2021-07-31T02:08:48.787 INFO:tasks.workunit.client.0.smithi045.stdout:op 1097 completed, throughput=5MB/sec 2021-07-31T02:08:48.788 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-BwoZjEKoy5RUfL- off=6883371 len=1489671 2021-07-31T02:08:48.788 INFO:tasks.workunit.client.0.smithi045.stdout:op 1098 completed, throughput=5MB/sec 2021-07-31T02:08:49.789 INFO:tasks.workunit.client.0.smithi045.stdout: 219: throughput=4.98MB/sec pending data=0 2021-07-31T02:08:49.789 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-SD31mwz-UKcpqj9 off=7696262 len=1613824 2021-07-31T02:08:49.792 INFO:tasks.workunit.client.0.smithi045.stdout:op 1099 completed, throughput=4.99MB/sec 2021-07-31T02:08:49.792 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-n2OSoKtUKLHYnlu off=0 len=1109302 2021-07-31T02:08:49.798 INFO:tasks.workunit.client.0.smithi045.stdout:op 1100 completed, throughput=4.99MB/sec 2021-07-31T02:08:49.799 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-04bN1F6_dz-j1Mh off=2722429 len=786776 2021-07-31T02:08:49.800 INFO:tasks.workunit.client.0.smithi045.stdout:op 1101 completed, throughput=5MB/sec 2021-07-31T02:08:49.800 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-E8jvNpOd2LjaH7m off=1134267 len=367900 2021-07-31T02:08:49.804 INFO:tasks.workunit.client.0.smithi045.stdout:op 1102 completed, throughput=5MB/sec 2021-07-31T02:08:49.804 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-f1KM9p7T7O011Y8 off=4780788 len=1534202 2021-07-31T02:08:49.805 INFO:tasks.workunit.client.0.smithi045.stdout:op 1103 completed, throughput=5MB/sec 2021-07-31T02:08:50.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:50 smithi154 conmon[34553]: cluster 2021-07-31T02:08:50.098519+0000 mgr.x (mgr.24542) 927 : cluster [DBG] pgmap v1520: 9 pgs: 9 active+clean; 127 GiB data, 4.7 GiB used, 710 GiB / 715 GiB avail; 3.6 MiB/s rd, 2.8 MiB/s wr, 6 op/s 2021-07-31T02:08:50.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:50 smithi045 conmon[33655]: cluster 2021-07-31T02:08:50.098519+0000 mgr.x (mgr.24542) 927 : cluster [DBG] pgmap v1520: 9 pgs: 9 active+clean; 127 GiB data, 4.7 GiB used, 710 GiB / 715 GiB avail; 3.6 MiB/s rd, 2.8 MiB/s wr, 6 op/s 2021-07-31T02:08:50.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:50 smithi045 conmon[33655]: 2021-07-31T02:08:50.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:50 smithi045 conmon[39718]: cluster 2021-07-31T02:08:50 2021-07-31T02:08:50.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:50 smithi045 conmon[39718]: .098519+0000 mgr.x (mgr.24542) 927 : cluster [DBG] pgmap v1520: 9 pgs: 9 active+clean; 127 GiB data, 4.7 GiB used, 710 GiB / 715 GiB avail; 3.6 MiB/s rd, 2.8 MiB/s wr, 6 op/s 2021-07-31T02:08:50.806 INFO:tasks.workunit.client.0.smithi045.stdout: 220: throughput=4.98MB/sec pending data=0 2021-07-31T02:08:50.806 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-mEM3vbd1GnOnao0 off=12423384 len=17181 2021-07-31T02:08:50.809 INFO:tasks.workunit.client.0.smithi045.stdout:op 1104 completed, throughput=4.98MB/sec 2021-07-31T02:08:50.809 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-HpL9E8mszzdQyMI off=1424211 len=849528 2021-07-31T02:08:50.810 INFO:tasks.workunit.client.0.smithi045.stdout:op 1105 completed, throughput=4.99MB/sec 2021-07-31T02:08:50.810 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-IYtgfnebUavc4tS off=10001529 len=31663 2021-07-31T02:08:50.812 INFO:tasks.workunit.client.0.smithi045.stdout:op 1106 completed, throughput=4.99MB/sec 2021-07-31T02:08:50.813 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-8SMip5mRoIRA76m off=6479770 len=889755 2021-07-31T02:08:50.814 INFO:tasks.workunit.client.0.smithi045.stdout:op 1107 completed, throughput=4.99MB/sec 2021-07-31T02:08:50.814 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-3A-b36bHM2Q4I6m off=7055796 len=206839 2021-07-31T02:08:50.817 INFO:tasks.workunit.client.0.smithi045.stdout:op 1108 completed, throughput=4.99MB/sec 2021-07-31T02:08:50.817 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-x1ID5GPlsUWXp5I off=2965682 len=1789847 2021-07-31T02:08:50.824 INFO:tasks.workunit.client.0.smithi045.stdout:op 1109 completed, throughput=5MB/sec 2021-07-31T02:08:50.825 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-KhTcIO-t1DNQhov off=4643934 len=1411201 2021-07-31T02:08:50.826 INFO:tasks.workunit.client.0.smithi045.stdout:op 1110 completed, throughput=5MB/sec 2021-07-31T02:08:51.827 INFO:tasks.workunit.client.0.smithi045.stdout: 221: throughput=4.98MB/sec pending data=0 2021-07-31T02:08:51.827 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-vPHgRjvHqxzT-uo off=4208304 len=132585 2021-07-31T02:08:51.828 INFO:tasks.workunit.client.0.smithi045.stdout:op 1111 completed, throughput=4.98MB/sec 2021-07-31T02:08:51.828 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-UzKX6HETvb2zsx_ off=7347082 len=244140 2021-07-31T02:08:51.831 INFO:tasks.workunit.client.0.smithi045.stdout:op 1112 completed, throughput=4.98MB/sec 2021-07-31T02:08:51.832 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ElrXV-uHpnvNxZQ off=12967189 len=162598 2021-07-31T02:08:51.832 INFO:tasks.workunit.client.0.smithi045.stdout:op 1113 completed, throughput=4.98MB/sec 2021-07-31T02:08:51.833 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-OclGGPxCSOopwPL off=2348531 len=1945890 2021-07-31T02:08:51.843 INFO:tasks.workunit.client.0.smithi045.stdout:op 1114 completed, throughput=4.99MB/sec 2021-07-31T02:08:51.843 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-EvIehX-O1ew71jF off=1074305 len=1119574 2021-07-31T02:08:51.850 INFO:tasks.workunit.client.0.smithi045.stdout:op 1115 completed, throughput=5MB/sec 2021-07-31T02:08:51.850 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-4DdPykWD2kCHfku off=422403 len=1200772 2021-07-31T02:08:51.852 INFO:tasks.workunit.client.0.smithi045.stdout:op 1116 completed, throughput=5MB/sec 2021-07-31T02:08:52.852 INFO:tasks.workunit.client.0.smithi045.stdout: 222: throughput=4.98MB/sec pending data=0 2021-07-31T02:08:52.853 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-sIBenQXi4Rxuqo_ off=457544 len=627798 2021-07-31T02:08:52.857 INFO:tasks.workunit.client.0.smithi045.stdout:op 1117 completed, throughput=4.98MB/sec 2021-07-31T02:08:52.857 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-f5LwPHQNEoo5YIY off=517123 len=1952063 2021-07-31T02:08:52.860 INFO:tasks.workunit.client.0.smithi045.stdout:op 1118 completed, throughput=4.99MB/sec 2021-07-31T02:08:52.861 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-blVoQjOnA5517zv off=4001381 len=337223 2021-07-31T02:08:52.862 INFO:tasks.workunit.client.0.smithi045.stdout:op 1119 completed, throughput=4.99MB/sec 2021-07-31T02:08:52.862 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-NNSEdjVqjBeNvNv off=2927186 len=300283 2021-07-31T02:08:52.863 INFO:tasks.workunit.client.0.smithi045.stdout:op 1120 completed, throughput=4.99MB/sec 2021-07-31T02:08:52.863 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-o2_W3xznTlEmqM_ off=7446046 len=1664996 2021-07-31T02:08:52.874 INFO:tasks.workunit.client.0.smithi045.stdout:op 1121 completed, throughput=5MB/sec 2021-07-31T02:08:52.875 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-euf2jV3ze8QryK_ off=10832125 len=96047 2021-07-31T02:08:52.875 INFO:tasks.workunit.client.0.smithi045.stdout:op 1122 completed, throughput=5MB/sec 2021-07-31T02:08:52.875 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-wejcTO4rp4VAuZB off=54950 len=1581594 2021-07-31T02:08:52.882 INFO:tasks.workunit.client.0.smithi045.stdout:op 1123 completed, throughput=5.01MB/sec 2021-07-31T02:08:53.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:52 smithi154 conmon[34553]: cluster 2021-07-31T02:08:52.099116+0000 mgr.x (mgr.24542) 928 : cluster [DBG] pgmap v1521: 9 pgs: 9 active+clean; 127 GiB data, 4.7 GiB used, 710 GiB / 715 GiB avail; 2.8 MiB/s rd, 1.4 MiB/s wr, 3 op/s 2021-07-31T02:08:53.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:52 smithi154 conmon[34553]: 2021-07-31T02:08:53.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:52 smithi045 conmon[33655]: cluster 2021-07-31T02:08:52.099116+0000 mgr.x (mgr.24542) 928 : cluster 2021-07-31T02:08:53.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:52 smithi045 conmon[33655]: [DBG] pgmap v1521: 9 pgs: 9 active+clean; 127 GiB data, 4.7 GiB used, 710 GiB / 715 GiB avail; 2.8 MiB/s rd, 1.4 MiB/s wr, 3 op/s 2021-07-31T02:08:53.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:52 smithi045 conmon[39718]: cluster 2021-07-31T02:08:52.099116+0000 mgr.x (mgr.24542) 2021-07-31T02:08:53.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:52 smithi045 conmon[39718]: 928 : cluster [DBG] pgmap v1521: 9 pgs: 9 active+clean; 127 GiB data, 4.7 GiB used, 710 GiB / 715 GiB avail; 2.8 MiB/s rd, 1.4 MiB/s wr, 3 op/s 2021-07-31T02:08:53.882 INFO:tasks.workunit.client.0.smithi045.stdout: 223: throughput=4.98MB/sec pending data=0 2021-07-31T02:08:53.883 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-J4Tyh_U7W0_GLRA off=3520925 len=547862 2021-07-31T02:08:53.883 INFO:tasks.workunit.client.0.smithi045.stdout:op 1124 completed, throughput=4.99MB/sec 2021-07-31T02:08:53.884 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Asjx1tqfj-CJcPR off=336503 len=1736450 2021-07-31T02:08:53.895 INFO:tasks.workunit.client.0.smithi045.stdout:op 1125 completed, throughput=4.99MB/sec 2021-07-31T02:08:53.895 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-e8Oywb1vB3lpn9W off=14524492 len=629018 2021-07-31T02:08:53.898 INFO:tasks.workunit.client.0.smithi045.stdout:op 1126 completed, throughput=5MB/sec 2021-07-31T02:08:53.899 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ptRY6ilBzMDbz6o off=6363407 len=14432 2021-07-31T02:08:53.901 INFO:tasks.workunit.client.0.smithi045.stdout:op 1127 completed, throughput=5MB/sec 2021-07-31T02:08:53.901 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-xLd3tIVNUge0Ru0 off=2798857 len=1726379 2021-07-31T02:08:53.907 INFO:tasks.workunit.client.0.smithi045.stdout:op 1128 completed, throughput=5MB/sec 2021-07-31T02:08:54.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:08:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:08:53.545Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:08:54.027 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:08:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:08:53.545Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:08:54.027 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:08:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:08:53.545Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:08:54.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:54 smithi154 conmon[34553]: cluster 2021-07-31T02:08:54.099570+0000 mgr.x (mgr.24542) 929 : cluster [DBG] pgmap v1522: 9 pgs: 9 active+clean; 127 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 3.7 MiB/s rd, 2.6 MiB/s wr, 6 op/s 2021-07-31T02:08:54.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:54 smithi045 conmon[33655]: cluster 2021 2021-07-31T02:08:54.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:54 smithi045 conmon[33655]: -07-31T02:08:54.099570+0000 mgr.x (mgr.24542) 929 : cluster [DBG] pgmap v1522: 9 pgs: 9 active+clean; 127 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 3.7 MiB/s rd, 2.6 MiB/s wr, 6 op/s 2021-07-31T02:08:54.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:54 smithi045 conmon[39718]: cluster 2021-07-31T02:08:54.099570+0000 mgr.x 2021-07-31T02:08:54.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:54 smithi045 conmon[39718]: (mgr.24542) 929 : cluster [DBG] pgmap v1522: 9 pgs: 9 active+clean; 127 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 3.7 MiB/s rd, 2.6 MiB/s wr, 6 op/s 2021-07-31T02:08:54.908 INFO:tasks.workunit.client.0.smithi045.stdout: 224: throughput=4.98MB/sec pending data=0 2021-07-31T02:08:55.082 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-SfU6LzmdVNGqPzV off=2316837 len=4655 2021-07-31T02:08:55.083 INFO:tasks.workunit.client.0.smithi045.stdout:op 1129 completed, throughput=4.98MB/sec 2021-07-31T02:08:55.083 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-M9yoiMPyi7Y7gln off=912682 len=1023348 2021-07-31T02:08:55.085 INFO:tasks.workunit.client.0.smithi045.stdout:op 1130 completed, throughput=4.98MB/sec 2021-07-31T02:08:55.085 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-V1wm-BefsiR8Kk- off=1391503 len=1518730 2021-07-31T02:08:55.085 INFO:tasks.workunit.client.0.smithi045.stdout:op 1131 completed, throughput=4.99MB/sec 2021-07-31T02:08:55.086 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Q5h6YgaXC-_BDKN off=9565700 len=687737 2021-07-31T02:08:55.086 INFO:tasks.workunit.client.0.smithi045.stdout:op 1132 completed, throughput=4.99MB/sec 2021-07-31T02:08:55.086 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-VRP6QTrNaaqsIwx off=19636833 len=1192891 2021-07-31T02:08:55.086 INFO:tasks.workunit.client.0.smithi045.stdout:op 1133 completed, throughput=5MB/sec 2021-07-31T02:08:55.086 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-fQAU24v8gqjEBA3 off=211457 len=1874715 2021-07-31T02:08:55.087 INFO:tasks.workunit.client.0.smithi045.stdout:op 1134 completed, throughput=5.01MB/sec 2021-07-31T02:08:55.921 INFO:tasks.workunit.client.0.smithi045.stdout: 225: throughput=4.98MB/sec pending data=0 2021-07-31T02:08:55.921 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-g2ppNHb_0WP-4Fj off=715597 len=1099954 2021-07-31T02:08:55.923 INFO:tasks.workunit.client.0.smithi045.stdout:op 1135 completed, throughput=4.99MB/sec 2021-07-31T02:08:55.923 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-C8z6DEicSAmqxW2 off=2792484 len=1555428 2021-07-31T02:08:55.926 INFO:tasks.workunit.client.0.smithi045.stdout:op 1136 completed, throughput=5MB/sec 2021-07-31T02:08:55.926 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-0AcfqQgv6O-5N5D off=3660852 len=213371 2021-07-31T02:08:55.929 INFO:tasks.workunit.client.0.smithi045.stdout:op 1137 completed, throughput=5MB/sec 2021-07-31T02:08:55.930 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-t0LekeKL1S0eSt1 off=11997150 len=1542722 2021-07-31T02:08:55.937 INFO:tasks.workunit.client.0.smithi045.stdout:op 1138 completed, throughput=5MB/sec 2021-07-31T02:08:56.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:56 smithi154 conmon[34553]: cluster 2021-07-31T02:08:56.100145+0000 mgr.x (mgr.24542) 930 : cluster [DBG] pgmap v1523: 9 pgs: 9 active+clean; 127 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 2.4 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2021-07-31T02:08:56.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:56 smithi045 conmon[33655]: cluster 2021-07-31T02:08:56.100145+0000 mgr.x (mgr.24542) 930 : cluster [DBG] 2021-07-31T02:08:56.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:56 smithi045 conmon[33655]: pgmap v1523: 9 pgs: 9 active+clean; 127 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 2.4 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2021-07-31T02:08:56.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:56 smithi045 conmon[39718]: cluster 2021-07-31T02:08:56.100145+0000 2021-07-31T02:08:56.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:56 smithi045 conmon[39718]: mgr.x (mgr.24542) 930 : cluster [DBG] pgmap v1523: 9 pgs: 9 active+clean; 127 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 2.4 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2021-07-31T02:08:56.937 INFO:tasks.workunit.client.0.smithi045.stdout: 226: throughput=4.98MB/sec pending data=0 2021-07-31T02:08:56.937 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-1QrWZd9IaDwjdMk off=6311035 len=1521033 2021-07-31T02:08:56.939 INFO:tasks.workunit.client.0.smithi045.stdout:op 1139 completed, throughput=4.99MB/sec 2021-07-31T02:08:56.939 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-woJoF1UB7_NG8qU off=5145198 len=696980 2021-07-31T02:08:56.944 INFO:tasks.workunit.client.0.smithi045.stdout:op 1140 completed, throughput=4.99MB/sec 2021-07-31T02:08:56.944 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-KC7FtRNGHuzHYdV off=1262222 len=1796110 2021-07-31T02:08:56.947 INFO:tasks.workunit.client.0.smithi045.stdout:op 1141 completed, throughput=5MB/sec 2021-07-31T02:08:56.948 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-8J8hb31ae47mNET off=9809099 len=834734 2021-07-31T02:08:56.952 INFO:tasks.workunit.client.0.smithi045.stdout:op 1142 completed, throughput=5MB/sec 2021-07-31T02:08:57.952 INFO:tasks.workunit.client.0.smithi045.stdout: 227: throughput=4.98MB/sec pending data=0 2021-07-31T02:08:57.953 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-rRNO8mEF_jlZVha off=8555597 len=1610256 2021-07-31T02:08:57.962 INFO:tasks.workunit.client.0.smithi045.stdout:op 1143 completed, throughput=4.99MB/sec 2021-07-31T02:08:57.963 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-knOGqV9xHpDbxD8 off=4626664 len=1915908 2021-07-31T02:08:57.966 INFO:tasks.workunit.client.0.smithi045.stdout:op 1144 completed, throughput=4.99MB/sec 2021-07-31T02:08:57.966 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-54nRRXHjWjUWlUw off=1918287 len=1094594 2021-07-31T02:08:57.968 INFO:tasks.workunit.client.0.smithi045.stdout:op 1145 completed, throughput=5MB/sec 2021-07-31T02:08:57.969 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-HBIMpg7mbjBymVh off=2045770 len=1881391 2021-07-31T02:08:57.976 INFO:tasks.workunit.client.0.smithi045.stdout:op 1146 completed, throughput=5.01MB/sec 2021-07-31T02:08:58.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:08:58 smithi154 conmon[34553]: cluster 2021-07-31T02:08:58.100608+0000 mgr.x (mgr.24542) 931 : cluster [DBG] pgmap v1524: 9 pgs: 9 active+clean; 127 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 2.4 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2021-07-31T02:08:58.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:58 smithi045 conmon[33655]: cluster 2021-07-31T02:08:58.100608+0000 mgr.x (mgr.24542) 931 : 2021-07-31T02:08:58.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:08:58 smithi045 conmon[33655]: cluster [DBG] pgmap v1524: 9 pgs: 9 active+clean; 127 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 2.4 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2021-07-31T02:08:58.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:58 smithi045 conmon[39718]: cluster 2021-07-31T02:08:58.100608+0000 mgr.x ( 2021-07-31T02:08:58.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:08:58 smithi045 conmon[39718]: mgr.24542) 931 : cluster [DBG] pgmap v1524: 9 pgs: 9 active+clean; 127 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 2.4 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2021-07-31T02:08:58.976 INFO:tasks.workunit.client.0.smithi045.stdout: 228: throughput=4.98MB/sec pending data=0 2021-07-31T02:08:58.976 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-CKPsZelnKbmjeyY off=3350833 len=324616 2021-07-31T02:08:58.977 INFO:tasks.workunit.client.0.smithi045.stdout:op 1147 completed, throughput=4.99MB/sec 2021-07-31T02:08:58.978 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-OoZ-a_cUCe9yPjV off=8322105 len=147311 2021-07-31T02:08:58.981 INFO:tasks.workunit.client.0.smithi045.stdout:op 1148 completed, throughput=4.99MB/sec 2021-07-31T02:08:58.981 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-57QagppET9QYLAH off=15725302 len=1790928 2021-07-31T02:08:58.984 INFO:tasks.workunit.client.0.smithi045.stdout:op 1149 completed, throughput=4.99MB/sec 2021-07-31T02:08:58.984 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ns6qz_VldJ_yh4p off=12866096 len=1612786 2021-07-31T02:08:58.987 INFO:tasks.workunit.client.0.smithi045.stdout:op 1150 completed, throughput=5MB/sec 2021-07-31T02:08:58.987 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-huUekd3WNJx1rn7 off=15079848 len=579099 2021-07-31T02:08:58.989 INFO:tasks.workunit.client.0.smithi045.stdout:op 1151 completed, throughput=5MB/sec 2021-07-31T02:08:59.989 INFO:tasks.workunit.client.0.smithi045.stdout: 229: throughput=4.98MB/sec pending data=0 2021-07-31T02:08:59.990 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-qLv2tNrnfaI3Fkj off=14026537 len=55667 2021-07-31T02:08:59.991 INFO:tasks.workunit.client.0.smithi045.stdout:op 1152 completed, throughput=4.98MB/sec 2021-07-31T02:08:59.991 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-nqTC42YNGB1L459 off=1580639 len=1667792 2021-07-31T02:08:59.993 INFO:tasks.workunit.client.0.smithi045.stdout:op 1153 completed, throughput=4.99MB/sec 2021-07-31T02:08:59.994 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-xMi7vVAqal5gSId off=2825133 len=499209 2021-07-31T02:08:59.998 INFO:tasks.workunit.client.0.smithi045.stdout:op 1154 completed, throughput=4.99MB/sec 2021-07-31T02:08:59.999 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-cklxGBfUSSXsvcI off=2002776 len=75541 2021-07-31T02:09:00.000 INFO:tasks.workunit.client.0.smithi045.stdout:op 1155 completed, throughput=4.99MB/sec 2021-07-31T02:09:00.000 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-SvEXE3X2jyr_svC off=6368250 len=406284 2021-07-31T02:09:00.003 INFO:tasks.workunit.client.0.smithi045.stdout:op 1156 completed, throughput=4.99MB/sec 2021-07-31T02:09:00.003 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-DNaiLtZJpKwwVM4 off=4569438 len=2970 2021-07-31T02:09:00.004 INFO:tasks.workunit.client.0.smithi045.stdout:op 1157 completed, throughput=4.99MB/sec 2021-07-31T02:09:00.005 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-HYJ5jb60lCUQXp5 off=9953615 len=881141 2021-07-31T02:09:00.008 INFO:tasks.workunit.client.0.smithi045.stdout:op 1158 completed, throughput=4.99MB/sec 2021-07-31T02:09:00.009 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-deqo38dpTYcIKXg off=7382795 len=2069644 2021-07-31T02:09:00.012 INFO:tasks.workunit.client.0.smithi045.stdout:op 1159 completed, throughput=5MB/sec 2021-07-31T02:09:00.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:00 smithi154 conmon[34553]: cluster 2021-07-31T02:09:00.101097+0000 mgr.x (mgr.24542) 932 : cluster [DBG] 2021-07-31T02:09:00.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:00 smithi154 conmon[34553]: pgmap v1525: 9 pgs: 9 active+clean; 127 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 3.6 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2021-07-31T02:09:00.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:00 smithi045 conmon[39718]: cluster 2021-07-31T02:09:00.101097+0000 mgr.x (mgr 2021-07-31T02:09:00.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:00 smithi045 conmon[39718]: .24542) 932 : cluster [DBG] pgmap v1525: 9 pgs: 9 active+clean; 127 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 3.6 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2021-07-31T02:09:00.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:00 smithi045 conmon[33655]: cluster 2021-07-31T02:09: 2021-07-31T02:09:00.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:00 smithi045 conmon[33655]: 00.101097+0000 mgr.x (mgr.24542) 932 : cluster [DBG] pgmap v1525: 9 pgs: 9 active+clean; 127 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 3.6 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2021-07-31T02:09:01.012 INFO:tasks.workunit.client.0.smithi045.stdout: 230: throughput=4.98MB/sec pending data=0 2021-07-31T02:09:01.012 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-pkkS3rX6qnHm84T off=15227127 len=488506 2021-07-31T02:09:01.013 INFO:tasks.workunit.client.0.smithi045.stdout:op 1160 completed, throughput=4.98MB/sec 2021-07-31T02:09:01.013 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-CNohrBvk2HfUrXf off=1080278 len=1755954 2021-07-31T02:09:01.015 INFO:tasks.workunit.client.0.smithi045.stdout:op 1161 completed, throughput=4.99MB/sec 2021-07-31T02:09:01.015 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-D81RM1ZSfKrHmT9 off=8735593 len=515168 2021-07-31T02:09:01.019 INFO:tasks.workunit.client.0.smithi045.stdout:op 1162 completed, throughput=4.99MB/sec 2021-07-31T02:09:01.019 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-c83q3iJaKTPWv-r off=4713962 len=592022 2021-07-31T02:09:01.021 INFO:tasks.workunit.client.0.smithi045.stdout:op 1163 completed, throughput=5MB/sec 2021-07-31T02:09:01.021 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ZG8ltvG5SiZvXpG off=3263189 len=467151 2021-07-31T02:09:01.022 INFO:tasks.workunit.client.0.smithi045.stdout:op 1164 completed, throughput=5MB/sec 2021-07-31T02:09:01.022 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-nJqqiUNK9JemKRx off=7677752 len=1850927 2021-07-31T02:09:01.034 INFO:tasks.workunit.client.0.smithi045.stdout:op 1165 completed, throughput=5MB/sec 2021-07-31T02:09:02.034 INFO:tasks.workunit.client.0.smithi045.stdout: 231: throughput=4.98MB/sec pending data=0 2021-07-31T02:09:02.034 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-52wyaGKlmrdvqm8 off=6662594 len=360698 2021-07-31T02:09:02.035 INFO:tasks.workunit.client.0.smithi045.stdout:op 1166 completed, throughput=4.98MB/sec 2021-07-31T02:09:02.035 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-v_-dLxOYTPdNI6X off=3608418 len=303145 2021-07-31T02:09:02.039 INFO:tasks.workunit.client.0.smithi045.stdout:op 1167 completed, throughput=4.99MB/sec 2021-07-31T02:09:02.039 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-MxbFoTyshC58sj_ off=2023045 len=865771 2021-07-31T02:09:02.044 INFO:tasks.workunit.client.0.smithi045.stdout:op 1168 completed, throughput=4.99MB/sec 2021-07-31T02:09:02.045 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-HLGHuXPZC9uN0Ur off=14164216 len=1854308 2021-07-31T02:09:02.056 INFO:tasks.workunit.client.0.smithi045.stdout:op 1169 completed, throughput=5MB/sec 2021-07-31T02:09:02.056 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Jnp8Cuo49SYtg_I off=4807557 len=40860 2021-07-31T02:09:02.058 INFO:tasks.workunit.client.0.smithi045.stdout:op 1170 completed, throughput=5MB/sec 2021-07-31T02:09:02.058 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-nI5MNAmhK4HKNO2 off=10327603 len=38546 2021-07-31T02:09:02.059 INFO:tasks.workunit.client.0.smithi045.stdout:op 1171 completed, throughput=5MB/sec 2021-07-31T02:09:02.059 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-m9jy6wgCgnoCYTt off=2029046 len=844038 2021-07-31T02:09:02.063 INFO:tasks.workunit.client.0.smithi045.stdout:op 1172 completed, throughput=5MB/sec 2021-07-31T02:09:02.064 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-F-quC22BYrIPVfx off=14736942 len=119662 2021-07-31T02:09:02.064 INFO:tasks.workunit.client.0.smithi045.stdout:op 1173 completed, throughput=5MB/sec 2021-07-31T02:09:03.065 INFO:tasks.workunit.client.0.smithi045.stdout: 232: throughput=4.98MB/sec pending data=0 2021-07-31T02:09:03.066 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-7B-NJDNPMAWzWmo off=2103249 len=432684 2021-07-31T02:09:03.068 INFO:tasks.workunit.client.0.smithi045.stdout:op 1174 completed, throughput=4.98MB/sec 2021-07-31T02:09:03.068 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-98tTjKX26vY3ly8 off=1390721 len=798563 2021-07-31T02:09:03.073 INFO:tasks.workunit.client.0.smithi045.stdout:op 1175 completed, throughput=4.98MB/sec 2021-07-31T02:09:03.073 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-fpzr6QL9vZdnl6X off=1020438 len=434921 2021-07-31T02:09:03.074 INFO:tasks.workunit.client.0.smithi045.stdout:op 1176 completed, throughput=4.99MB/sec 2021-07-31T02:09:03.074 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-dJ_nHT0UuQXvwZY off=4166883 len=1701098 2021-07-31T02:09:03.082 INFO:tasks.workunit.client.0.smithi045.stdout:op 1177 completed, throughput=4.99MB/sec 2021-07-31T02:09:03.083 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-XzyPOEC7M1JqT9v off=2971768 len=2041550 2021-07-31T02:09:03.092 INFO:tasks.workunit.client.0.smithi045.stdout:op 1178 completed, throughput=5MB/sec 2021-07-31T02:09:03.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:02 smithi154 conmon[34553]: cluster 2021-07-31T02:09:02.101519+0000 mgr.x (mgr.24542) 2021-07-31T02:09:03.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:02 smithi154 conmon[34553]: 933 : cluster [DBG] pgmap v1526: 9 pgs: 9 active+clean; 127 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 2.2 MiB/s rd, 1.7 MiB/s wr, 3 op/s 2021-07-31T02:09:03.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:02 smithi045 conmon[33655]: cluster 2021-07-31T02:09:02.101519+0000 mgr.x (mgr.24542) 933 : cluster [DBG] pgmap v1526: 9 pgs: 9 active+clean; 127 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 2.2 MiB/s rd, 1.7 MiB/s wr, 3 op/s 2021-07-31T02:09:03.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:02 smithi045 conmon[39718]: cluster 2021-07-31T02:09:02.101519+0000 mgr.x (mgr. 2021-07-31T02:09:03.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:02 smithi045 conmon[39718]: 24542) 933 : cluster [DBG] pgmap v1526: 9 pgs: 9 active+clean; 127 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 2.2 MiB/s rd, 1.7 MiB/s wr, 3 op/s 2021-07-31T02:09:04.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:09:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:09:03.545Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:09:04.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:09:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:09:03.545Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:09:04.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:09:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:09:03.545Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:09:04.092 INFO:tasks.workunit.client.0.smithi045.stdout: 233: throughput=4.98MB/sec pending data=0 2021-07-31T02:09:04.092 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ulhGKBstSSlYPtE off=13712820 len=1019909 2021-07-31T02:09:04.098 INFO:tasks.workunit.client.0.smithi045.stdout:op 1179 completed, throughput=4.98MB/sec 2021-07-31T02:09:04.098 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-7aiO_TSwYpncme9 off=15133911 len=1841835 2021-07-31T02:09:04.106 INFO:tasks.workunit.client.0.smithi045.stdout:op 1180 completed, throughput=4.99MB/sec 2021-07-31T02:09:04.106 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-nMYci_KvsRo9C4C off=22593082 len=41691 2021-07-31T02:09:04.107 INFO:tasks.workunit.client.0.smithi045.stdout:op 1181 completed, throughput=4.99MB/sec 2021-07-31T02:09:04.108 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-fxZCRa4N1kxCeJ8 off=989048 len=822832 2021-07-31T02:09:04.112 INFO:tasks.workunit.client.0.smithi045.stdout:op 1182 completed, throughput=4.99MB/sec 2021-07-31T02:09:04.112 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-bGd90N-Yd9WkarA off=3516173 len=467214 2021-07-31T02:09:04.113 INFO:tasks.workunit.client.0.smithi045.stdout:op 1183 completed, throughput=5MB/sec 2021-07-31T02:09:04.113 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-JXeqYJJ2vYXAO-w off=3381113 len=2029405 2021-07-31T02:09:04.117 INFO:tasks.workunit.client.0.smithi045.stdout:op 1184 completed, throughput=5MB/sec 2021-07-31T02:09:04.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:04 smithi154 conmon[34553]: cluster 2021-07-31T02:09:04.101998+0000 mgr.x (mgr.24542) 934 : cluster [DBG] 2021-07-31T02:09:04.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:04 smithi154 conmon[34553]: pgmap v1527: 9 pgs: 9 active+clean; 127 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.8 MiB/s wr, 6 op/s 2021-07-31T02:09:04.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:04 smithi045 conmon[33655]: cluster 2021-07-31T02:09:04.101998+0000 mgr.x (mgr.24542) 934 : cluster [DBG] pgmap v1527: 9 pgs: 9 active+clean; 127 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.8 MiB/s wr, 6 op/s 2021-07-31T02:09:04.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:04 smithi045 conmon[33655]: 2021-07-31T02:09:04.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:04 smithi045 conmon[39718]: cluster 2021-07-31T02:09:04.101998+0000 mgr.x (mgr. 2021-07-31T02:09:04.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:04 smithi045 conmon[39718]: 24542) 934 : cluster [DBG] pgmap v1527: 9 pgs: 9 active+clean; 127 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.8 MiB/s wr, 6 op/s 2021-07-31T02:09:05.117 INFO:tasks.workunit.client.0.smithi045.stdout: 234: throughput=4.98MB/sec pending data=0 2021-07-31T02:09:05.117 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-6s6ub4AXa6w0PnJ off=21388843 len=1074348 2021-07-31T02:09:05.120 INFO:tasks.workunit.client.0.smithi045.stdout:op 1185 completed, throughput=4.99MB/sec 2021-07-31T02:09:05.120 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-mRAwGbMU4Eq4dzZ off=13400398 len=1547762 2021-07-31T02:09:05.129 INFO:tasks.workunit.client.0.smithi045.stdout:op 1186 completed, throughput=4.99MB/sec 2021-07-31T02:09:05.130 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-WexEIXv-vKCBlD2 off=2952792 len=1848155 2021-07-31T02:09:05.140 INFO:tasks.workunit.client.0.smithi045.stdout:op 1187 completed, throughput=5MB/sec 2021-07-31T02:09:05.432 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:09:05.711 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:09:05.711 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 23m ago 24m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:09:05.711 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 21m ago 24m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:09:05.711 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 21m ago 27m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:09:05.712 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 23m ago 30m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:09:05.712 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 23m ago 30m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:09:05.713 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 21m ago 28m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:09:05.713 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 23m ago 28m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:09:05.713 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 23m ago 24m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:09:05.714 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 21m ago 24m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:09:05.714 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 23m ago 27m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:09:05.714 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 23m ago 27m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:09:05.714 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 23m ago 27m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:09:05.715 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 23m ago 26m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:09:05.715 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 21m ago 26m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:09:05.715 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 21m ago 26m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:09:05.715 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 21m ago 25m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:09:05.716 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 21m ago 25m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:09:05.716 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 21m ago 25m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:09:06.045 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:09:06.045 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:09:06.045 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:09:06.046 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:09:06.046 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:09:06.046 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:09:06.046 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:09:06.047 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:09:06.047 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:09:06.047 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:09:06.047 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:09:06.048 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:09:06.048 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:09:06.048 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:09:06.048 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:09:06.048 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:09:06.049 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:09:06.140 INFO:tasks.workunit.client.0.smithi045.stdout: 235: throughput=4.98MB/sec pending data=0 2021-07-31T02:09:06.141 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-1eKV2r2rXuocl42 off=3799915 len=844690 2021-07-31T02:09:06.145 INFO:tasks.workunit.client.0.smithi045.stdout:op 1188 completed, throughput=4.98MB/sec 2021-07-31T02:09:06.146 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-WUBMRAAuYWk2eac off=12880859 len=73101 2021-07-31T02:09:06.148 INFO:tasks.workunit.client.0.smithi045.stdout:op 1189 completed, throughput=4.98MB/sec 2021-07-31T02:09:06.148 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-UvRTMALg5jt51PF off=205879 len=1470792 2021-07-31T02:09:06.154 INFO:tasks.workunit.client.0.smithi045.stdout:op 1190 completed, throughput=4.99MB/sec 2021-07-31T02:09:06.155 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-JWh1UJB7HD__KOy off=5898433 len=1381495 2021-07-31T02:09:06.159 INFO:tasks.workunit.client.0.smithi045.stdout:op 1191 completed, throughput=4.99MB/sec 2021-07-31T02:09:06.160 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-b0zjj4ccgRgu8o4 off=3784125 len=129858 2021-07-31T02:09:06.160 INFO:tasks.workunit.client.0.smithi045.stdout:op 1192 completed, throughput=4.99MB/sec 2021-07-31T02:09:06.161 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-kjjDFLtHrVx5xcc off=1954255 len=677590 2021-07-31T02:09:06.165 INFO:tasks.workunit.client.0.smithi045.stdout:op 1193 completed, throughput=5MB/sec 2021-07-31T02:09:06.165 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-fRM_PR8xj0pImq5 off=12358809 len=517191 2021-07-31T02:09:06.166 INFO:tasks.workunit.client.0.smithi045.stdout:op 1194 completed, throughput=5MB/sec 2021-07-31T02:09:06.166 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ZY8yZ4GJnmfYGt_ off=17141701 len=1699853 2021-07-31T02:09:06.167 INFO:tasks.workunit.client.0.smithi045.stdout:op 1195 completed, throughput=5.01MB/sec 2021-07-31T02:09:06.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:06 smithi154 conmon[34553]: audit 2021-07-31T02:09:06.044901+0000 mon.c (mon.1) 78 : 2021-07-31T02:09:06.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:06 smithi154 conmon[34553]: audit [DBG] from='client.? 172.21.15.45:0/3281104989' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:09:06.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:06 smithi045 conmon[33655]: audit 2021-07-31T02:09:06.044901+0000 mon.c ( 2021-07-31T02:09:06.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:06 smithi045 conmon[33655]: mon.1) 78 : audit [DBG] from='client.? 172.21.15.45:0/3281104989' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:09:06.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:06 smithi045 conmon[39718]: audit 2021-07-31T02:09: 2021-07-31T02:09:06.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:06 smithi045 conmon[39718]: 06.044901+0000 mon.c (mon.1) 78 : audit [DBG] from='client.? 172.21.15.45:0/3281104989' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:09:07.168 INFO:tasks.workunit.client.0.smithi045.stdout: 236: throughput=4.99MB/sec pending data=0 2021-07-31T02:09:07.168 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-5TdRYtgBHTB_F_N off=6077476 len=593309 2021-07-31T02:09:07.169 INFO:tasks.workunit.client.0.smithi045.stdout:op 1196 completed, throughput=4.99MB/sec 2021-07-31T02:09:07.169 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-kg6F1wLQS4r4S6P off=5369958 len=977089 2021-07-31T02:09:07.172 INFO:tasks.workunit.client.0.smithi045.stdout:op 1197 completed, throughput=4.99MB/sec 2021-07-31T02:09:07.172 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-luXLSbY968LQcN0 off=4660351 len=1890119 2021-07-31T02:09:07.180 INFO:tasks.workunit.client.0.smithi045.stdout:op 1198 completed, throughput=5MB/sec 2021-07-31T02:09:07.181 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-kP17mOgw5BOFT4T off=1955778 len=434930 2021-07-31T02:09:07.184 INFO:tasks.workunit.client.0.smithi045.stdout:op 1199 completed, throughput=5MB/sec 2021-07-31T02:09:07.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:07 smithi154 conmon[34553]: audit 2021-07-31T02:09:05.419837+0000 mgr.x (mgr.24542) 935 : audit 2021-07-31T02:09:07.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:07 smithi154 conmon[34553]: [DBG] from='client.15861 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:09:07.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:07 smithi154 conmon[34553]: audit 2021-07-31T02:09:05.702465+0000 mgr.x (mgr.24542) 936 : audit [DBG] from='client.25774 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:09:07.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:07 smithi154 conmon[34553]: cluster 2021-07-31T02:09:07.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:07 smithi154 conmon[34553]: 2021-07-31T02:09:06.102325+0000 mgr.x (mgr.24542) 937 : cluster [DBG] pgmap v1528: 9 pgs: 9 active+clean; 127 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 2.2 MiB/s rd, 1.6 MiB/s wr, 4 op/s 2021-07-31T02:09:07.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:07 smithi045 conmon[33655]: audit 2021-07-31T02:09:05.419837+0000 mgr.x (mgr.24542) 935 2021-07-31T02:09:07.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:07 smithi045 conmon[33655]: : audit [DBG] from='client.15861 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:09:07.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:07 smithi045 conmon[33655]: audit 2021-07-31T02:09:05.702465+0000 mgr.x (mgr.24542) 936 : audit [DBG] from='client.25774 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:09:07.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:07 smithi045 conmon[33655]: cluster 2021-07-31T02: 2021-07-31T02:09:07.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:07 smithi045 conmon[33655]: 09:06.102325+0000 mgr.x (mgr.24542) 937 : cluster [DBG] pgmap v1528: 9 pgs: 9 active+clean; 127 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 2.2 MiB/s rd, 1.6 MiB/s wr, 4 op/s 2021-07-31T02:09:07.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:07 smithi045 conmon[39718]: audit 2021-07-31T02:09:05.419837+0000 mgr.x (mgr.24542) 935 : audit [DBG] from='client.15861 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:09:07.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:07 smithi045 conmon[39718]: audit 2021-07-31T02:09:05.702465+0000 mgr.x (mgr.24542) 2021-07-31T02:09:07.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:07 smithi045 conmon[39718]: 936 : audit [DBG] from='client.25774 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:09:07.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:07 smithi045 conmon[39718]: cluster 2021-07-31T02: 2021-07-31T02:09:07.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:07 smithi045 conmon[39718]: 09:06.102325+0000 mgr.x (mgr.24542) 2021-07-31T02:09:07.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:07 smithi045 conmon[39718]: 937 : cluster [DBG] pgmap v1528: 9 pgs: 9 active+clean; 127 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 2.2 MiB/s rd, 1.6 MiB/s wr, 4 op/s 2021-07-31T02:09:08.185 INFO:tasks.workunit.client.0.smithi045.stdout: 237: throughput=4.98MB/sec pending data=0 2021-07-31T02:09:08.186 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-FyXNqxoGF97028h off=5749149 len=1627210 2021-07-31T02:09:08.193 INFO:tasks.workunit.client.0.smithi045.stdout:op 1200 completed, throughput=4.99MB/sec 2021-07-31T02:09:08.193 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-S5amb7c9-Bmz9ds off=988335 len=1799838 2021-07-31T02:09:08.195 INFO:tasks.workunit.client.0.smithi045.stdout:op 1201 completed, throughput=4.99MB/sec 2021-07-31T02:09:08.195 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-9V7Teu8NlXih2mM off=1606058 len=296109 2021-07-31T02:09:08.198 INFO:tasks.workunit.client.0.smithi045.stdout:op 1202 completed, throughput=4.99MB/sec 2021-07-31T02:09:08.199 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ga1D2Z43X68btdZ off=3611517 len=884871 2021-07-31T02:09:08.203 INFO:tasks.workunit.client.0.smithi045.stdout:op 1203 completed, throughput=5MB/sec 2021-07-31T02:09:08.203 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-T_DYn3CGHPElA19 off=1458023 len=1621529 2021-07-31T02:09:08.206 INFO:tasks.workunit.client.0.smithi045.stdout:op 1204 completed, throughput=5MB/sec 2021-07-31T02:09:08.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:08 smithi154 conmon[34553]: cluster 2021-07-31T02:09:08.102770+0000 mgr.x (mgr.24542) 938 : 2021-07-31T02:09:08.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:08 smithi154 conmon[34553]: cluster [DBG] pgmap v1529: 9 pgs: 9 active+clean; 127 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 2.2 MiB/s rd, 1.6 MiB/s wr, 4 op/s 2021-07-31T02:09:08.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:08 smithi045 conmon[33655]: cluster 2021-07-31T02:09:08.102770+0000 2021-07-31T02:09:08.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:08 smithi045 conmon[33655]: mgr.x (mgr.24542) 938 : cluster [DBG] pgmap v1529: 9 pgs: 9 active+clean; 127 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 2.2 MiB/s rd, 1.6 MiB/s wr, 4 op/s 2021-07-31T02:09:08.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:08 smithi045 conmon[39718]: cluster 2021-07-31T02:09:08.102770+0000 mgr.x (mgr.24542 2021-07-31T02:09:08.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:08 smithi045 conmon[39718]: ) 938 : cluster [DBG] pgmap v1529: 9 pgs: 9 active+clean; 127 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 2.2 MiB/s rd, 1.6 MiB/s wr, 4 op/s 2021-07-31T02:09:09.206 INFO:tasks.workunit.client.0.smithi045.stdout: 238: throughput=4.98MB/sec pending data=0 2021-07-31T02:09:09.206 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-wo99qCkjgGuAUoT off=6377543 len=1111267 2021-07-31T02:09:09.213 INFO:tasks.workunit.client.0.smithi045.stdout:op 1205 completed, throughput=4.99MB/sec 2021-07-31T02:09:09.213 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-002meGc_aSyV_Y9 off=14113890 len=1956748 2021-07-31T02:09:09.217 INFO:tasks.workunit.client.0.smithi045.stdout:op 1206 completed, throughput=4.99MB/sec 2021-07-31T02:09:09.217 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-L6atXDZ2E-TWDqD off=11890241 len=1612727 2021-07-31T02:09:09.219 INFO:tasks.workunit.client.0.smithi045.stdout:op 1207 completed, throughput=5MB/sec 2021-07-31T02:09:10.220 INFO:tasks.workunit.client.0.smithi045.stdout: 239: throughput=4.98MB/sec pending data=0 2021-07-31T02:09:10.220 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-S4LW1jJhkZurigh off=1668648 len=271567 2021-07-31T02:09:10.223 INFO:tasks.workunit.client.0.smithi045.stdout:op 1208 completed, throughput=4.98MB/sec 2021-07-31T02:09:10.223 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-v9kEt_J6J-qRfZb off=2003901 len=1505640 2021-07-31T02:09:10.231 INFO:tasks.workunit.client.0.smithi045.stdout:op 1209 completed, throughput=4.99MB/sec 2021-07-31T02:09:10.232 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-RM4qM4thUoHXEFh off=5262510 len=200401 2021-07-31T02:09:10.232 INFO:tasks.workunit.client.0.smithi045.stdout:op 1210 completed, throughput=4.99MB/sec 2021-07-31T02:09:10.232 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-2mSO9Va2hc559e3 off=12801953 len=1248759 2021-07-31T02:09:10.238 INFO:tasks.workunit.client.0.smithi045.stdout:op 1211 completed, throughput=4.99MB/sec 2021-07-31T02:09:10.239 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-aE8T3PhJ0cEcpLb off=13994390 len=138103 2021-07-31T02:09:10.239 INFO:tasks.workunit.client.0.smithi045.stdout:op 1212 completed, throughput=4.99MB/sec 2021-07-31T02:09:10.240 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-CLEwxtQ3ktNzODM off=974158 len=132680 2021-07-31T02:09:10.241 INFO:tasks.workunit.client.0.smithi045.stdout:op 1213 completed, throughput=4.99MB/sec 2021-07-31T02:09:10.241 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-79w4z835cDja7bv off=2825478 len=1018793 2021-07-31T02:09:10.244 INFO:tasks.workunit.client.0.smithi045.stdout:op 1214 completed, throughput=5MB/sec 2021-07-31T02:09:10.245 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-odx4_Ag-eFJyTNX off=4295311 len=1633408 2021-07-31T02:09:10.251 INFO:tasks.workunit.client.0.smithi045.stdout:op 1215 completed, throughput=5MB/sec 2021-07-31T02:09:10.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:10 smithi154 conmon[34553]: cluster 2021-07-31T02:09:10.103193+0000 mgr.x (mgr.24542) 939 : cluster [DBG] pgmap v1530: 9 pgs: 9 active+clean; 127 GiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 3.1 MiB/s rd, 3.0 MiB/s wr, 6 op/s 2021-07-31T02:09:10.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:10 smithi045 conmon[39718]: cluster 2021-07-31T02:09:10.103193+0000 mgr.x (mgr.24542 2021-07-31T02:09:10.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:10 smithi045 conmon[39718]: ) 939 : cluster [DBG] pgmap v1530: 9 pgs: 9 active+clean; 127 GiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 3.1 MiB/s rd, 3.0 MiB/s wr, 6 op/s 2021-07-31T02:09:10.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:10 smithi045 conmon[33655]: cluster 2021-07-31T02:09:10.103193+0000 mgr.x (mgr.24542) 939 : cluster 2021-07-31T02:09:10.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:10 smithi045 conmon[33655]: [DBG] pgmap v1530: 9 pgs: 9 active+clean; 127 GiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 3.1 MiB/s rd, 3.0 MiB/s wr, 6 op/s 2021-07-31T02:09:11.252 INFO:tasks.workunit.client.0.smithi045.stdout: 240: throughput=4.98MB/sec pending data=0 2021-07-31T02:09:11.253 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-5gEEreSAIVCCA8Z off=2170127 len=1081126 2021-07-31T02:09:11.254 INFO:tasks.workunit.client.0.smithi045.stdout:op 1216 completed, throughput=4.99MB/sec 2021-07-31T02:09:11.254 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ZW0JXhKbMp8Ykfd off=4175171 len=1522984 2021-07-31T02:09:11.261 INFO:tasks.workunit.client.0.smithi045.stdout:op 1217 completed, throughput=4.99MB/sec 2021-07-31T02:09:11.262 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ELW_VmI_iJ8b6Ij off=2696205 len=528860 2021-07-31T02:09:11.263 INFO:tasks.workunit.client.0.smithi045.stdout:op 1218 completed, throughput=5MB/sec 2021-07-31T02:09:11.264 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-9Je3z0f96YeKj97 off=22663069 len=2013821 2021-07-31T02:09:11.264 INFO:tasks.workunit.client.0.smithi045.stdout:op 1219 completed, throughput=5MB/sec 2021-07-31T02:09:12.265 INFO:tasks.workunit.client.0.smithi045.stdout: 241: throughput=4.98MB/sec pending data=0 2021-07-31T02:09:12.265 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-2M7oBM4AkKWSb5z off=3933802 len=1767850 2021-07-31T02:09:12.274 INFO:tasks.workunit.client.0.smithi045.stdout:op 1220 completed, throughput=4.99MB/sec 2021-07-31T02:09:12.275 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-x3PwHSVwMyNA5eq off=13680839 len=582565 2021-07-31T02:09:12.279 INFO:tasks.workunit.client.0.smithi045.stdout:op 1221 completed, throughput=4.99MB/sec 2021-07-31T02:09:12.279 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-u4t7iKL06qvUxRb off=4388901 len=1339926 2021-07-31T02:09:12.285 INFO:tasks.workunit.client.0.smithi045.stdout:op 1222 completed, throughput=5MB/sec 2021-07-31T02:09:12.286 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-01QF2PB6p_GIBn2 off=18882136 len=1358470 2021-07-31T02:09:12.292 INFO:tasks.workunit.client.0.smithi045.stdout:op 1223 completed, throughput=5MB/sec 2021-07-31T02:09:13.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:12 smithi154 conmon[34553]: cluster 2021-07-31T02:09:12.103723+0000 mgr.x (mgr.24542) 940 : cluster 2021-07-31T02:09:13.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:12 smithi154 conmon[34553]: [DBG] pgmap v1531: 9 pgs: 9 active+clean; 127 GiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:09:13.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:12 smithi045 conmon[39718]: cluster 2021-07-31T02:09:12.103723+0000 mgr.x 2021-07-31T02:09:13.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:12 smithi045 conmon[39718]: (mgr.24542) 940 : cluster [DBG] pgmap v1531: 9 pgs: 9 active+clean; 127 GiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:09:13.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:12 smithi045 conmon[33655]: cluster 2021-07-31T02:09:12.103723+0000 mgr.x (mgr.24542) 940 : 2021-07-31T02:09:13.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:12 smithi045 conmon[33655]: cluster [DBG] pgmap v1531: 9 pgs: 9 active+clean; 127 GiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:09:13.292 INFO:tasks.workunit.client.0.smithi045.stdout: 242: throughput=4.98MB/sec pending data=0 2021-07-31T02:09:13.292 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj--PV7L_TWi0FP92p off=0 len=1202693 2021-07-31T02:09:13.299 INFO:tasks.workunit.client.0.smithi045.stdout:op 1224 completed, throughput=4.99MB/sec 2021-07-31T02:09:13.299 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ykPUB7DLNYA651r off=5703951 len=1197306 2021-07-31T02:09:13.305 INFO:tasks.workunit.client.0.smithi045.stdout:op 1225 completed, throughput=4.99MB/sec 2021-07-31T02:09:13.306 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-KAG41_Wld_vL9RF off=7423766 len=857590 2021-07-31T02:09:13.307 INFO:tasks.workunit.client.0.smithi045.stdout:op 1226 completed, throughput=4.99MB/sec 2021-07-31T02:09:13.308 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-zQjf--bIipdJKY3 off=2894936 len=565172 2021-07-31T02:09:13.309 INFO:tasks.workunit.client.0.smithi045.stdout:op 1227 completed, throughput=5MB/sec 2021-07-31T02:09:13.309 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-oWYBgpTECQ8jCIp off=15601837 len=61653 2021-07-31T02:09:13.310 INFO:tasks.workunit.client.0.smithi045.stdout:op 1228 completed, throughput=5MB/sec 2021-07-31T02:09:13.310 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-mdBO1tnRGxdkRNJ off=2677178 len=808233 2021-07-31T02:09:13.315 INFO:tasks.workunit.client.0.smithi045.stdout:op 1229 completed, throughput=5MB/sec 2021-07-31T02:09:13.315 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-46-UcOrBKTI55yM off=4118414 len=949902 2021-07-31T02:09:13.321 INFO:tasks.workunit.client.0.smithi045.stdout:op 1230 completed, throughput=5MB/sec 2021-07-31T02:09:14.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:09:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:09:13.545Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:09:14.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:09:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:09:13.545Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:09:14.027 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:09:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:09:13.545Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:09:14.322 INFO:tasks.workunit.client.0.smithi045.stdout: 243: throughput=4.98MB/sec pending data=0 2021-07-31T02:09:14.322 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-3_cIK8fygFHOD5r off=3917659 len=1649387 2021-07-31T02:09:14.333 INFO:tasks.workunit.client.0.smithi045.stdout:op 1231 completed, throughput=4.99MB/sec 2021-07-31T02:09:14.333 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-KlFHdHZHS1_iuiP off=8494038 len=552475 2021-07-31T02:09:14.334 INFO:tasks.workunit.client.0.smithi045.stdout:op 1232 completed, throughput=4.99MB/sec 2021-07-31T02:09:14.335 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-sTfQjUy1Myq7PgE off=23137747 len=1336398 2021-07-31T02:09:14.341 INFO:tasks.workunit.client.0.smithi045.stdout:op 1233 completed, throughput=5MB/sec 2021-07-31T02:09:14.341 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-xoLJsHHSmklxRXN off=2936914 len=1514280 2021-07-31T02:09:14.344 INFO:tasks.workunit.client.0.smithi045.stdout:op 1234 completed, throughput=5MB/sec 2021-07-31T02:09:14.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:14 smithi154 conmon[34553]: cluster 2021-07-31T02:09:14.104293+0000 mgr.x (mgr.24542) 941 : cluster [DBG] pgmap v1532: 9 pgs: 9 active+clean; 127 GiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 2.6 MiB/s rd, 3.8 MiB/s wr, 6 op/s 2021-07-31T02:09:14.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:14 smithi045 conmon[39718]: cluster 2021-07-31T02:09:14.104293+0000 mgr.x (mgr.24542) 2021-07-31T02:09:14.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:14 smithi045 conmon[39718]: 941 : cluster [DBG] pgmap v1532: 9 pgs: 9 active+clean; 127 GiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 2.6 MiB/s rd, 3.8 MiB/s wr, 6 op/s 2021-07-31T02:09:14.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:14 smithi045 conmon[33655]: cluster 2021-07-31T02:09:14.104293+0000 mgr.x 2021-07-31T02:09:14.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:14 smithi045 conmon[33655]: (mgr.24542) 941 : cluster [DBG] pgmap v1532: 9 pgs: 9 active+clean; 127 GiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 2.6 MiB/s rd, 3.8 MiB/s wr, 6 op/s 2021-07-31T02:09:15.344 INFO:tasks.workunit.client.0.smithi045.stdout: 244: throughput=4.98MB/sec pending data=0 2021-07-31T02:09:15.345 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-bUyY5gulL8AuIBD off=9473875 len=126494 2021-07-31T02:09:15.347 INFO:tasks.workunit.client.0.smithi045.stdout:op 1235 completed, throughput=4.98MB/sec 2021-07-31T02:09:15.348 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-y_MaXaOYDMUZOUr off=842255 len=804034 2021-07-31T02:09:15.352 INFO:tasks.workunit.client.0.smithi045.stdout:op 1236 completed, throughput=4.99MB/sec 2021-07-31T02:09:15.353 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Cvrskrr8Coj_37L off=8124176 len=1070937 2021-07-31T02:09:15.359 INFO:tasks.workunit.client.0.smithi045.stdout:op 1237 completed, throughput=4.99MB/sec 2021-07-31T02:09:15.360 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-10-BWDmRRfGFrxW off=6928625 len=95160 2021-07-31T02:09:15.360 INFO:tasks.workunit.client.0.smithi045.stdout:op 1238 completed, throughput=4.99MB/sec 2021-07-31T02:09:15.360 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-bF1XEO3bcVF9WZV off=1833785 len=985775 2021-07-31T02:09:15.361 INFO:tasks.workunit.client.0.smithi045.stdout:op 1239 completed, throughput=4.99MB/sec 2021-07-31T02:09:15.361 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-u5LTPUk9Bhqy_aq off=1728899 len=245479 2021-07-31T02:09:15.364 INFO:tasks.workunit.client.0.smithi045.stdout:op 1240 completed, throughput=4.99MB/sec 2021-07-31T02:09:15.364 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-bqJkQ-Cof9zaWnJ off=9609584 len=1426574 2021-07-31T02:09:15.367 INFO:tasks.workunit.client.0.smithi045.stdout:op 1241 completed, throughput=5MB/sec 2021-07-31T02:09:15.367 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-beulyNHa5a9ZRQz off=4816861 len=1756963 2021-07-31T02:09:15.377 INFO:tasks.workunit.client.0.smithi045.stdout:op 1242 completed, throughput=5.01MB/sec 2021-07-31T02:09:16.377 INFO:tasks.workunit.client.0.smithi045.stdout: 245: throughput=4.99MB/sec pending data=0 2021-07-31T02:09:16.378 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-jJ4j-s5nSxL_nkl off=12917463 len=1732657 2021-07-31T02:09:16.386 INFO:tasks.workunit.client.0.smithi045.stdout:op 1243 completed, throughput=4.99MB/sec 2021-07-31T02:09:16.386 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-OTvAxkwdKGiVi4Y off=2799255 len=834875 2021-07-31T02:09:16.388 INFO:tasks.workunit.client.0.smithi045.stdout:op 1244 completed, throughput=5MB/sec 2021-07-31T02:09:16.388 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-MSwuGQi66O0RX99 off=2719271 len=416532 2021-07-31T02:09:16.392 INFO:tasks.workunit.client.0.smithi045.stdout:op 1245 completed, throughput=5MB/sec 2021-07-31T02:09:16.392 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-bldbcNio5oNA6k4 off=2753936 len=566240 2021-07-31T02:09:16.395 INFO:tasks.workunit.client.0.smithi045.stdout:op 1246 completed, throughput=5MB/sec 2021-07-31T02:09:16.396 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-btG6jhOj-d51ku_ off=11983700 len=333882 2021-07-31T02:09:16.397 INFO:tasks.workunit.client.0.smithi045.stdout:op 1247 completed, throughput=5MB/sec 2021-07-31T02:09:16.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:16 smithi154 conmon[34553]: cluster 2021-07-31T02:09:16.104858+0000 mgr.x (mgr.24542) 942 : cluster [DBG] pgmap v1533: 9 pgs: 9 active+clean; 127 GiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.6 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2021-07-31T02:09:16.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:16 smithi045 conmon[39718]: cluster 2021-07-31T02:09:16.104858+0000 2021-07-31T02:09:16.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:16 smithi045 conmon[39718]: mgr.x (mgr.24542) 942 : cluster [DBG] pgmap v1533: 9 pgs: 9 active+clean; 127 GiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.6 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2021-07-31T02:09:16.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:16 smithi045 conmon[33655]: cluster 2021-07-31T02:09:16.104858+0000 mgr.x (mgr.24542) 942 : cluster [DBG] 2021-07-31T02:09:16.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:16 smithi045 conmon[33655]: pgmap v1533: 9 pgs: 9 active+clean; 127 GiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.6 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2021-07-31T02:09:17.397 INFO:tasks.workunit.client.0.smithi045.stdout: 246: throughput=4.98MB/sec pending data=0 2021-07-31T02:09:17.398 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-gyIOj1M0zYumg_v off=6277520 len=1479214 2021-07-31T02:09:17.405 INFO:tasks.workunit.client.0.smithi045.stdout:op 1248 completed, throughput=4.99MB/sec 2021-07-31T02:09:17.405 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-7gbNGt0Zrrsb_LM off=5385754 len=1870071 2021-07-31T02:09:17.414 INFO:tasks.workunit.client.0.smithi045.stdout:op 1249 completed, throughput=4.99MB/sec 2021-07-31T02:09:17.414 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-MaB6eqxSD4m1TLb off=19639010 len=1375317 2021-07-31T02:09:17.421 INFO:tasks.workunit.client.0.smithi045.stdout:op 1250 completed, throughput=5MB/sec 2021-07-31T02:09:17.421 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-O7-KomE4ZEZbo8V off=4397131 len=1505887 2021-07-31T02:09:17.423 INFO:tasks.workunit.client.0.smithi045.stdout:op 1251 completed, throughput=5MB/sec 2021-07-31T02:09:18.423 INFO:tasks.workunit.client.0.smithi045.stdout: 247: throughput=4.98MB/sec pending data=0 2021-07-31T02:09:18.424 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-hgmplc9oGyswsnk off=2982193 len=958995 2021-07-31T02:09:18.425 INFO:tasks.workunit.client.0.smithi045.stdout:op 1252 completed, throughput=4.99MB/sec 2021-07-31T02:09:18.425 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-0HHGBX52VyWnqey off=1235911 len=1436070 2021-07-31T02:09:18.426 INFO:tasks.workunit.client.0.smithi045.stdout:op 1253 completed, throughput=4.99MB/sec 2021-07-31T02:09:18.427 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-xmsHePgQJDgYHYs off=9899786 len=1996697 2021-07-31T02:09:18.440 INFO:tasks.workunit.client.0.smithi045.stdout:op 1254 completed, throughput=5MB/sec 2021-07-31T02:09:18.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:18 smithi154 conmon[34553]: cluster 2021-07-31T02:09:18.105464+0000 mgr.x (mgr.24542) 943 : cluster [DBG] pgmap v1534: 9 pgs: 9 active+clean; 127 GiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.6 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2021-07-31T02:09:18.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:18 smithi045 conmon[39718]: cluster 2021-07-31T02:09:18.105464+0000 mgr.x (mgr.24542) 943 2021-07-31T02:09:18.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:18 smithi045 conmon[39718]: : cluster [DBG] pgmap v1534: 9 pgs: 9 active+clean; 127 GiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.6 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2021-07-31T02:09:18.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:18 smithi045 conmon[33655]: cluster 2021-07-31T02:09: 2021-07-31T02:09:18.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:18 smithi045 conmon[33655]: 18.105464+0000 mgr.x (mgr.24542) 943 : cluster [DBG] pgmap v1534: 9 pgs: 9 active+clean; 127 GiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.6 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2021-07-31T02:09:19.440 INFO:tasks.workunit.client.0.smithi045.stdout: 248: throughput=4.98MB/sec pending data=0 2021-07-31T02:09:19.440 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-KwPx9Aa08xBAhK7 off=3056979 len=1693044 2021-07-31T02:09:19.448 INFO:tasks.workunit.client.0.smithi045.stdout:op 1255 completed, throughput=4.99MB/sec 2021-07-31T02:09:19.448 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-A8HvoxLmGkYPTMg off=1807988 len=1896814 2021-07-31T02:09:19.455 INFO:tasks.workunit.client.0.smithi045.stdout:op 1256 completed, throughput=4.99MB/sec 2021-07-31T02:09:19.456 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-OTEJh6MHgwx9WTj off=7686781 len=1938653 2021-07-31T02:09:19.463 INFO:tasks.workunit.client.0.smithi045.stdout:op 1257 completed, throughput=5MB/sec 2021-07-31T02:09:20.464 INFO:tasks.workunit.client.0.smithi045.stdout: 249: throughput=4.98MB/sec pending data=0 2021-07-31T02:09:20.464 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-m_HCSe04NXPX7Ry off=1400115 len=1904426 2021-07-31T02:09:20.467 INFO:tasks.workunit.client.0.smithi045.stdout:op 1258 completed, throughput=4.99MB/sec 2021-07-31T02:09:20.467 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-EAl1iRl2lrA4EA3 off=2959814 len=306622 2021-07-31T02:09:20.472 INFO:tasks.workunit.client.0.smithi045.stdout:op 1259 completed, throughput=4.99MB/sec 2021-07-31T02:09:20.472 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-XBSguK1hgyxjIij off=14545501 len=984504 2021-07-31T02:09:20.474 INFO:tasks.workunit.client.0.smithi045.stdout:op 1260 completed, throughput=4.99MB/sec 2021-07-31T02:09:20.474 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-o864C5EWS1Q3z8t off=13029189 len=742201 2021-07-31T02:09:20.475 INFO:tasks.workunit.client.0.smithi045.stdout:op 1261 completed, throughput=5MB/sec 2021-07-31T02:09:20.475 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Wuz4zGEmtjqbKFY off=8467861 len=1217578 2021-07-31T02:09:20.481 INFO:tasks.workunit.client.0.smithi045.stdout:op 1262 completed, throughput=5MB/sec 2021-07-31T02:09:20.481 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-nLn5CZM0DzYAvyR off=10188846 len=951648 2021-07-31T02:09:20.482 INFO:tasks.workunit.client.0.smithi045.stdout:op 1263 completed, throughput=5MB/sec 2021-07-31T02:09:20.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:20 smithi154 conmon[34553]: cluster 2021-07-31T02:09:20.106040+0000 mgr.x (mgr.24542) 944 : cluster [DBG] pgmap v1535: 9 pgs: 9 active+clean; 127 GiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 2.4 MiB/s rd, 4.0 MiB/s wr, 6 op/s 2021-07-31T02:09:20.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:20 smithi154 conmon[34553]: 2021-07-31T02:09:20.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:20 smithi045 conmon[39718]: cluster 2021-07-31T02:09:20.106040+0000 mgr.x (mgr 2021-07-31T02:09:20.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:20 smithi045 conmon[39718]: .24542) 944 : cluster [DBG] pgmap v1535: 9 pgs: 9 active+clean; 127 GiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 2.4 MiB/s rd, 4.0 MiB/s wr, 6 op/s 2021-07-31T02:09:20.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:20 smithi045 conmon[33655]: cluster 2021-07-31T02:09:20.106040+0000 mgr.x (mgr.24542) 944 : cluster [DBG] 2021-07-31T02:09:20.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:20 smithi045 conmon[33655]: pgmap v1535: 9 pgs: 9 active+clean; 127 GiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 2.4 MiB/s rd, 4.0 MiB/s wr, 6 op/s 2021-07-31T02:09:21.483 INFO:tasks.workunit.client.0.smithi045.stdout: 250: throughput=4.98MB/sec pending data=0 2021-07-31T02:09:21.483 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-xnPtV7_edaTx4Zk off=9367440 len=684667 2021-07-31T02:09:21.487 INFO:tasks.workunit.client.0.smithi045.stdout:op 1264 completed, throughput=4.99MB/sec 2021-07-31T02:09:21.488 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ZQpvpKT5_IzGhZ_ off=2367121 len=64402 2021-07-31T02:09:21.490 INFO:tasks.workunit.client.0.smithi045.stdout:op 1265 completed, throughput=4.99MB/sec 2021-07-31T02:09:21.490 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-q94zkWFtZYoWZfb off=6175497 len=466648 2021-07-31T02:09:21.492 INFO:tasks.workunit.client.0.smithi045.stdout:op 1266 completed, throughput=4.99MB/sec 2021-07-31T02:09:21.492 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-gV1p9-nAOR2WJn4 off=19392424 len=831278 2021-07-31T02:09:21.493 INFO:tasks.workunit.client.0.smithi045.stdout:op 1267 completed, throughput=4.99MB/sec 2021-07-31T02:09:21.493 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Aajtwv0RsTHHQSE off=0 len=1420087 2021-07-31T02:09:21.500 INFO:tasks.workunit.client.0.smithi045.stdout:op 1268 completed, throughput=5MB/sec 2021-07-31T02:09:21.500 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-zO2yUV29EtkSTOS off=3241458 len=1514898 2021-07-31T02:09:21.502 INFO:tasks.workunit.client.0.smithi045.stdout:op 1269 completed, throughput=5MB/sec 2021-07-31T02:09:22.503 INFO:tasks.workunit.client.0.smithi045.stdout: 251: throughput=4.98MB/sec pending data=0 2021-07-31T02:09:22.503 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-z9ob_xqMJC6YTBI off=4718947 len=998064 2021-07-31T02:09:22.505 INFO:tasks.workunit.client.0.smithi045.stdout:op 1270 completed, throughput=4.99MB/sec 2021-07-31T02:09:22.506 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-AHdKZ6XqLvlBUT2 off=738794 len=791000 2021-07-31T02:09:22.511 INFO:tasks.workunit.client.0.smithi045.stdout:op 1271 completed, throughput=4.99MB/sec 2021-07-31T02:09:22.511 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-5RAF5N_cfKHGAdq off=11093463 len=581570 2021-07-31T02:09:22.515 INFO:tasks.workunit.client.0.smithi045.stdout:op 1272 completed, throughput=4.99MB/sec 2021-07-31T02:09:22.515 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-939AZ2c4MuSOMTc off=6581101 len=972845 2021-07-31T02:09:22.517 INFO:tasks.workunit.client.0.smithi045.stdout:op 1273 completed, throughput=4.99MB/sec 2021-07-31T02:09:22.517 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-RAxmY3c6ghtl0r_ off=9297802 len=123994 2021-07-31T02:09:22.520 INFO:tasks.workunit.client.0.smithi045.stdout:op 1274 completed, throughput=5MB/sec 2021-07-31T02:09:22.520 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-5zKdH4gbWzViw4W off=1108428 len=241241 2021-07-31T02:09:22.523 INFO:tasks.workunit.client.0.smithi045.stdout:op 1275 completed, throughput=5MB/sec 2021-07-31T02:09:22.523 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-jytK-yF8jeTb3kv off=1796436 len=101236 2021-07-31T02:09:22.525 INFO:tasks.workunit.client.0.smithi045.stdout:op 1276 completed, throughput=5MB/sec 2021-07-31T02:09:22.525 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-KiX8G1kHe_tIqNR off=10631783 len=1030904 2021-07-31T02:09:22.527 INFO:tasks.workunit.client.0.smithi045.stdout:op 1277 completed, throughput=5MB/sec 2021-07-31T02:09:23.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:22 smithi154 conmon[34553]: cluster 2021-07-31T02:09:22.106409+0000 mgr.x (mgr.24542) 945 : cluster [DBG] 2021-07-31T02:09:23.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:22 smithi154 conmon[34553]: pgmap v1536: 9 pgs: 9 active+clean; 127 GiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 1.5 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2021-07-31T02:09:23.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:22 smithi045 conmon[39718]: cluster 2021-07-31T02:09:22.106409+0000 mgr.x 2021-07-31T02:09:23.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:22 smithi045 conmon[39718]: (mgr.24542) 945 : cluster [DBG] pgmap v1536: 9 pgs: 9 active+clean; 127 GiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 1.5 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2021-07-31T02:09:23.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:22 smithi045 conmon[33655]: cluster 2021-07-31T02:09:22 2021-07-31T02:09:23.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:22 smithi045 conmon[33655]: .106409+0000 mgr.x (mgr.24542) 945 : cluster [DBG] pgmap v1536: 9 pgs: 9 active+clean; 127 GiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 1.5 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2021-07-31T02:09:23.527 INFO:tasks.workunit.client.0.smithi045.stdout: 252: throughput=4.98MB/sec pending data=0 2021-07-31T02:09:23.528 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-N0OzNL2-0znPmlL off=827579 len=609452 2021-07-31T02:09:23.531 INFO:tasks.workunit.client.0.smithi045.stdout:op 1278 completed, throughput=4.98MB/sec 2021-07-31T02:09:23.532 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Xg5stzv3dvYqMDS off=4600342 len=434036 2021-07-31T02:09:23.532 INFO:tasks.workunit.client.0.smithi045.stdout:op 1279 completed, throughput=4.98MB/sec 2021-07-31T02:09:23.532 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-POW-eGj5jRrg-F2 off=3191782 len=1518612 2021-07-31T02:09:23.534 INFO:tasks.workunit.client.0.smithi045.stdout:op 1280 completed, throughput=4.99MB/sec 2021-07-31T02:09:23.535 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Qchp3YTziCZTbu3 off=1697745 len=1013428 2021-07-31T02:09:23.539 INFO:tasks.workunit.client.0.smithi045.stdout:op 1281 completed, throughput=4.99MB/sec 2021-07-31T02:09:23.540 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-gV1p9-nAOR2WJn4 off=11676489 len=469879 2021-07-31T02:09:23.541 INFO:tasks.workunit.client.0.smithi045.stdout:op 1282 completed, throughput=5MB/sec 2021-07-31T02:09:23.541 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-wuH9UCdLiE8oZQz off=13618093 len=2072203 2021-07-31T02:09:23.553 INFO:tasks.workunit.client.0.smithi045.stdout:op 1283 completed, throughput=5MB/sec 2021-07-31T02:09:24.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:09:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:09:23.545Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:09:24.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:09:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:09:23.545Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:09:24.027 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:09:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:09:23.546Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:09:24.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:24 smithi154 conmon[34553]: cluster 2021-07-31T02:09:24.106862+0000 mgr.x (mgr.24542) 946 : cluster [DBG] pgmap v1537: 9 pgs: 9 active+clean; 127 GiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 2.5 MiB/s rd, 3.9 MiB/s wr, 6 op/s 2021-07-31T02:09:24.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:24 smithi045 conmon[39718]: cluster 2021-07-31T02:09:24.106862+0000 mgr.x ( 2021-07-31T02:09:24.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:24 smithi045 conmon[39718]: mgr.24542) 946 : cluster [DBG] pgmap v1537: 9 pgs: 9 active+clean; 127 GiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 2.5 MiB/s rd, 3.9 MiB/s wr, 6 op/s 2021-07-31T02:09:24.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:24 smithi045 conmon[33655]: cluster 2021-07-31T02:09:24 2021-07-31T02:09:24.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:24 smithi045 conmon[33655]: .106862+0000 mgr.x (mgr.24542) 946 : cluster [DBG] pgmap v1537: 9 pgs: 9 active+clean; 127 GiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 2.5 MiB/s rd, 3.9 MiB/s wr, 6 op/s 2021-07-31T02:09:24.553 INFO:tasks.workunit.client.0.smithi045.stdout: 253: throughput=4.98MB/sec pending data=0 2021-07-31T02:09:24.553 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-aif1Cj21eAeihSo off=2582923 len=618240 2021-07-31T02:09:24.557 INFO:tasks.workunit.client.0.smithi045.stdout:op 1284 completed, throughput=4.99MB/sec 2021-07-31T02:09:24.558 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-t-SIhRL_1Xp26R- off=3212351 len=943630 2021-07-31T02:09:24.563 INFO:tasks.workunit.client.0.smithi045.stdout:op 1285 completed, throughput=4.99MB/sec 2021-07-31T02:09:24.563 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-dNTA2uW5bR9ubsa off=9300798 len=710705 2021-07-31T02:09:24.568 INFO:tasks.workunit.client.0.smithi045.stdout:op 1286 completed, throughput=4.99MB/sec 2021-07-31T02:09:24.568 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-3HvjtlK-ms_CG5E off=4217791 len=1370788 2021-07-31T02:09:24.575 INFO:tasks.workunit.client.0.smithi045.stdout:op 1287 completed, throughput=5MB/sec 2021-07-31T02:09:24.575 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-rwgcYE6qkkSFv0c off=22096618 len=1499729 2021-07-31T02:09:24.576 INFO:tasks.workunit.client.0.smithi045.stdout:op 1288 completed, throughput=5MB/sec 2021-07-31T02:09:25.577 INFO:tasks.workunit.client.0.smithi045.stdout: 254: throughput=4.98MB/sec pending data=0 2021-07-31T02:09:25.577 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-JhiD6W8JPCtsJMq off=16813823 len=355019 2021-07-31T02:09:25.578 INFO:tasks.workunit.client.0.smithi045.stdout:op 1289 completed, throughput=4.98MB/sec 2021-07-31T02:09:25.579 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-7DsyUo94sP45PZF off=520447 len=340924 2021-07-31T02:09:25.580 INFO:tasks.workunit.client.0.smithi045.stdout:op 1290 completed, throughput=4.99MB/sec 2021-07-31T02:09:25.580 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-xvEUCKfPM3IXS73 off=16962922 len=2054468 2021-07-31T02:09:25.588 INFO:tasks.workunit.client.0.smithi045.stdout:op 1291 completed, throughput=4.99MB/sec 2021-07-31T02:09:25.588 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-nmvFdBNTHbctcrG off=16815714 len=1976254 2021-07-31T02:09:25.598 INFO:tasks.workunit.client.0.smithi045.stdout:op 1292 completed, throughput=5MB/sec 2021-07-31T02:09:25.598 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-NASXaOrxPTFwvNw off=6663989 len=1055102 2021-07-31T02:09:25.603 INFO:tasks.workunit.client.0.smithi045.stdout:op 1293 completed, throughput=5MB/sec 2021-07-31T02:09:26.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:26 smithi154 conmon[34553]: cluster 2021-07-31T02:09:26.107505+0000 mgr.x (mgr.24542) 947 : cluster [DBG] pgmap v1538: 9 pgs: 9 active+clean; 127 GiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2021-07-31T02:09:26.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:26 smithi154 conmon[34553]: 2021-07-31T02:09:26.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:26 smithi045 conmon[39718]: cluster 2021-07-31T02:09:26.107505+0000 mgr.x (mgr.24542 2021-07-31T02:09:26.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:26 smithi045 conmon[39718]: ) 947 : cluster [DBG] pgmap v1538: 9 pgs: 9 active+clean; 127 GiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2021-07-31T02:09:26.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:26 smithi045 conmon[33655]: cluster 2021-07-31T02:09:26.107505+0000 mgr.x (mgr.24542) 2021-07-31T02:09:26.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:26 smithi045 conmon[33655]: 947 : cluster [DBG] pgmap v1538: 9 pgs: 9 active+clean; 127 GiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2021-07-31T02:09:26.603 INFO:tasks.workunit.client.0.smithi045.stdout: 255: throughput=4.98MB/sec pending data=0 2021-07-31T02:09:26.604 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-w4f2q9PL5orzzMy off=13385570 len=851877 2021-07-31T02:09:26.605 INFO:tasks.workunit.client.0.smithi045.stdout:op 1294 completed, throughput=4.99MB/sec 2021-07-31T02:09:26.606 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-j5REFR_VyLRDXZv off=806410 len=354158 2021-07-31T02:09:26.609 INFO:tasks.workunit.client.0.smithi045.stdout:op 1295 completed, throughput=4.99MB/sec 2021-07-31T02:09:26.610 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-18u5-MO3U7WSFCz off=0 len=1342127 2021-07-31T02:09:26.612 INFO:tasks.workunit.client.0.smithi045.stdout:op 1296 completed, throughput=4.99MB/sec 2021-07-31T02:09:26.612 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-2zSrHpe8qyRGsIr off=6731151 len=1400091 2021-07-31T02:09:26.613 INFO:tasks.workunit.client.0.smithi045.stdout:op 1297 completed, throughput=5MB/sec 2021-07-31T02:09:26.613 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj--yPNkJ0ChjdVIWs off=23058865 len=242093 2021-07-31T02:09:26.616 INFO:tasks.workunit.client.0.smithi045.stdout:op 1298 completed, throughput=5MB/sec 2021-07-31T02:09:26.617 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-xYyQzatKnigNbKX off=13776461 len=993588 2021-07-31T02:09:26.622 INFO:tasks.workunit.client.0.smithi045.stdout:op 1299 completed, throughput=5MB/sec 2021-07-31T02:09:27.622 INFO:tasks.workunit.client.0.smithi045.stdout: 256: throughput=4.98MB/sec pending data=0 2021-07-31T02:09:27.623 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-BGzbiIsDy-IgvCj off=10074908 len=809076 2021-07-31T02:09:27.624 INFO:tasks.workunit.client.0.smithi045.stdout:op 1300 completed, throughput=4.99MB/sec 2021-07-31T02:09:27.624 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-kFtLvUhVTcLKfHN off=15816483 len=1890136 2021-07-31T02:09:27.636 INFO:tasks.workunit.client.0.smithi045.stdout:op 1301 completed, throughput=4.99MB/sec 2021-07-31T02:09:27.636 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Se8VyWw9Z7w0OtI off=1152458 len=1980778 2021-07-31T02:09:27.646 INFO:tasks.workunit.client.0.smithi045.stdout:op 1302 completed, throughput=5MB/sec 2021-07-31T02:09:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:28 smithi154 conmon[34553]: audit 2021-07-31T02:09:28.013618+0000 mon.b (mon.2) 178 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:09:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:28 smithi154 conmon[34553]: 2021-07-31T02:09:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:28 smithi154 conmon[34553]: audit 2021-07-31T02:09:28.014395+0000 mon.a (mon.0) 1862 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:09:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:28 smithi154 conmon[34553]: audit 2021-07-31T02:09:28.032753+0000 mon.b (mon.2) 179 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:09:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:28 smithi154 conmon[34553]: audit 2021-07-31T02:09:28.033472+0000 mon.a (mon.0) 1863 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:09:28.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:28 smithi045 conmon[39718]: audit 2021-07-31T02:09:28. 2021-07-31T02:09:28.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:28 smithi045 conmon[39718]: 013618+0000 mon.b (mon.2) 178 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:09:28.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:28 smithi045 conmon[39718]: audit 2021-07-31T02:09:28 2021-07-31T02:09:28.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:28 smithi045 conmon[39718]: .014395+0000 mon.a (mon.0) 1862 : audit [INF] 2021-07-31T02:09:28.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:28 smithi045 conmon[39718]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:09:28.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:28 smithi045 conmon[39718]: audit 2021-07-31T02: 2021-07-31T02:09:28.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:28 smithi045 conmon[39718]: 09:28.032753+0000 mon.b (mon.2) 2021-07-31T02:09:28.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:28 smithi045 conmon[39718]: 179 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:09:28.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:28 smithi045 conmon[39718]: audit 2021 2021-07-31T02:09:28.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:28 smithi045 conmon[39718]: -07-31T02:09:28.033472+0000 2021-07-31T02:09:28.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:28 smithi045 conmon[39718]: mon.a (mon.0) 1863 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:09:28.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:28 smithi045 conmon[39718]: 2021-07-31T02:09:28.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:28 smithi045 conmon[33655]: audit 2021-07-31T02:09:28.013618+0000 mon.b (mon.2) 178 : 2021-07-31T02:09:28.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:28 smithi045 conmon[33655]: audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:09:28.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:28 smithi045 conmon[33655]: audit 2021-07-31T02:09:28.014395+0000 mon.a (mon.0) 1862 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:09:28.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:28 smithi045 conmon[33655]: audit 2021-07-31T02:09 2021-07-31T02:09:28.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:28 smithi045 conmon[33655]: :28.032753+0000 mon.b (mon.2) 179 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:09:28.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:28 smithi045 conmon[33655]: audit 2021-07-31T02:09:28. 2021-07-31T02:09:28.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:28 smithi045 conmon[33655]: 033472+0000 mon.a (mon.0) 1863 : audit 2021-07-31T02:09:28.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:28 smithi045 conmon[33655]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:09:28.646 INFO:tasks.workunit.client.0.smithi045.stdout: 257: throughput=4.98MB/sec pending data=0 2021-07-31T02:09:28.647 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-39bJ2bcMUUOoUyI off=20085 len=1638992 2021-07-31T02:09:28.649 INFO:tasks.workunit.client.0.smithi045.stdout:op 1303 completed, throughput=4.99MB/sec 2021-07-31T02:09:28.650 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-hr5DrqfARxyujj8 off=10316283 len=222820 2021-07-31T02:09:28.652 INFO:tasks.workunit.client.0.smithi045.stdout:op 1304 completed, throughput=4.99MB/sec 2021-07-31T02:09:28.653 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-sr2Fg5HJ-6GvpyD off=284525 len=330970 2021-07-31T02:09:28.653 INFO:tasks.workunit.client.0.smithi045.stdout:op 1305 completed, throughput=4.99MB/sec 2021-07-31T02:09:28.654 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-QpsOQDetinCZWTN off=3308170 len=782573 2021-07-31T02:09:28.658 INFO:tasks.workunit.client.0.smithi045.stdout:op 1306 completed, throughput=4.99MB/sec 2021-07-31T02:09:28.658 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ci9iXV1cFvWk0T2 off=3019237 len=96706 2021-07-31T02:09:28.660 INFO:tasks.workunit.client.0.smithi045.stdout:op 1307 completed, throughput=4.99MB/sec 2021-07-31T02:09:28.660 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-qTbt4vAo_ak3ey2 off=1062941 len=1558051 2021-07-31T02:09:28.669 INFO:tasks.workunit.client.0.smithi045.stdout:op 1308 completed, throughput=5MB/sec 2021-07-31T02:09:28.669 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-cThRP-qRYQ8F8wY off=7776176 len=744813 2021-07-31T02:09:28.673 INFO:tasks.workunit.client.0.smithi045.stdout:op 1309 completed, throughput=5MB/sec 2021-07-31T02:09:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:29 smithi154 conmon[34553]: cluster 2021-07-31T02:09:28.107927+0000 mgr.x (mgr.24542) 948 : cluster [DBG] pgmap v1539: 9 pgs: 9 active+clean; 127 GiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2021-07-31T02:09:29.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:29 smithi045 conmon[39718]: cluster 2021-07-31T02:09: 2021-07-31T02:09:29.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:29 smithi045 conmon[39718]: 28.107927+0000 mgr.x (mgr.24542) 948 : cluster [DBG] pgmap v1539: 9 pgs: 9 active+clean; 127 GiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2021-07-31T02:09:29.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:29 smithi045 conmon[33655]: cluster 2021-07-31T02:09:28.107927+0000 mgr.x (mgr.24542) 948 : cluster [DBG] 2021-07-31T02:09:29.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:29 smithi045 conmon[33655]: pgmap v1539: 9 pgs: 9 active+clean; 127 GiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2021-07-31T02:09:29.673 INFO:tasks.workunit.client.0.smithi045.stdout: 258: throughput=4.98MB/sec pending data=0 2021-07-31T02:09:29.674 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-AqBoRuqDg47mhPl off=15760026 len=1650712 2021-07-31T02:09:29.677 INFO:tasks.workunit.client.0.smithi045.stdout:op 1310 completed, throughput=4.99MB/sec 2021-07-31T02:09:29.677 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Y7DjHqZ6v8KY1F2 off=3218899 len=1962734 2021-07-31T02:09:29.690 INFO:tasks.workunit.client.0.smithi045.stdout:op 1311 completed, throughput=4.99MB/sec 2021-07-31T02:09:29.691 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-kkrJOcDmDlp1yWH off=3328408 len=317493 2021-07-31T02:09:29.692 INFO:tasks.workunit.client.0.smithi045.stdout:op 1312 completed, throughput=5MB/sec 2021-07-31T02:09:29.693 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-tWXZnMxwNnaDNGD off=10843805 len=684493 2021-07-31T02:09:29.696 INFO:tasks.workunit.client.0.smithi045.stdout:op 1313 completed, throughput=5MB/sec 2021-07-31T02:09:29.697 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-XljygX5xR7VBQ_j off=8014281 len=685273 2021-07-31T02:09:29.697 INFO:tasks.workunit.client.0.smithi045.stdout:op 1314 completed, throughput=5MB/sec 2021-07-31T02:09:30.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[34553]: cluster 2021-07-31T02:09:30.108398+0000 mgr.x (mgr.24542 2021-07-31T02:09:30.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[34553]: ) 949 : cluster [DBG] pgmap v1540: 9 pgs: 9 active+clean; 127 GiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 2.4 MiB/s rd, 3.9 MiB/s wr, 6 op/s 2021-07-31T02:09:30.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: debug 2021-07-31T02:09:30.337+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:09:30.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:09:30.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:09:30.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: yield 2021-07-31T02:09:30.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:09:30.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:09:30.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:09:30.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:09:30.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:09:30.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:09:30.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:09:30.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:09:30.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:09:30.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:09:30.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:09:30.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: debug 2021-07-31T02:09:30.349+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:09:30.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:09:30.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:09:30.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: yield 2021-07-31T02:09:30.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:09:30.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:09:30.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:09:30.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:09:30.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:09:30.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:09:30.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:09:30.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:09:30.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:09:30.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:09:30.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:09:30.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: debug 2021-07-31T02:09:30.374+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:09:30.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:09:30.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:09:30.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: yield 2021-07-31T02:09:30.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:09:30.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:09:30.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:09:30.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:09:30.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:09:30.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:09:30.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:09:30.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:09:30.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:09:30.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:09:30.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:09:30.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: debug 2021-07-31T02:09:30.375+0000 7fcb9215e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:09:30.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:09:30.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:09:30.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: yield 2021-07-31T02:09:30.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:09:30.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:09:30.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:09:30.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:09:30.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:09:30.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:09:30.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:09:30.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:09:30.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:09:30.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:09:30.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:09:30.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: 2021-07-31T02:09:30.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:09:30.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: 2021-07-31T02:09:30.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:09:30.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:09:30.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:09:30.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:09:30.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:09:30.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:09:30.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:09:30.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:09:30.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:09:30.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:09:30.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:09:30.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:09:30.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:09:30.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:09:30.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:09:30.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:09:30.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:09:30.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:09:30.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: raise 2021-07-31T02:09:30.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:09:30.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:09:30.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:09:30.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:09:30.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:09:30.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: debug 2021-07-31T02:09:30.383+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:09:30.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:09:30.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:09:30.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: yield 2021-07-31T02:09:30.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:09:30.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:09:30.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:09:30.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:09:30.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:09:30.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:09:30.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:09:30.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:09:30.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:09:30.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:09:30.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:09:30.535 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: debug 2021-07-31T02:09:30.384+0000 7fcb9195d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:09:30.535 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:09:30.535 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:09:30.535 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: yield 2021-07-31T02:09:30.535 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:09:30.536 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:09:30.536 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:09:30.536 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:09:30.537 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:09:30.537 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:09:30.537 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:09:30.537 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:09:30.538 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:09:30.538 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:09:30.538 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:09:30.538 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: 2021-07-31T02:09:30.539 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:09:30.539 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: 2021-07-31T02:09:30.539 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:09:30.539 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:09:30.539 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:09:30.540 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:09:30.540 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:09:30.540 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:09:30.540 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:09:30.541 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:09:30.541 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:09:30.541 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:09:30.542 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:09:30.542 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:09:30.542 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:09:30.543 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:09:30.543 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:09:30.543 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:09:30.544 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:09:30.544 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:09:30.544 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: raise 2021-07-31T02:09:30.545 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:09:30.545 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:09:30.545 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:09:30.545 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:09:30.545 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:09:30 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:09:30.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:30 smithi045 conmon[39718]: cluster 2021-07-31T02:09: 2021-07-31T02:09:30.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:30 smithi045 conmon[39718]: 30.108398+0000 mgr.x (mgr.24542) 949 : cluster [DBG] pgmap v1540: 9 pgs: 9 active+clean; 127 GiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 2.4 MiB/s rd, 3.9 MiB/s wr, 6 op/s 2021-07-31T02:09:30.547 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:30 smithi045 conmon[33655]: cluster 2021-07-31T02:09:30.108398+0000 mgr.x (mgr.24542) 949 : cluster [DBG] 2021-07-31T02:09:30.547 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:30 smithi045 conmon[33655]: pgmap v1540: 9 pgs: 9 active+clean; 127 GiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 2.4 MiB/s rd, 3.9 MiB/s wr, 6 op/s 2021-07-31T02:09:30.698 INFO:tasks.workunit.client.0.smithi045.stdout: 259: throughput=4.98MB/sec pending data=0 2021-07-31T02:09:30.698 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Ek2_68ZHHA6hcOP off=8533556 len=331603 2021-07-31T02:09:30.699 INFO:tasks.workunit.client.0.smithi045.stdout:op 1315 completed, throughput=4.98MB/sec 2021-07-31T02:09:30.699 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-zRjM_izezVP8mXE off=1070782 len=217040 2021-07-31T02:09:30.702 INFO:tasks.workunit.client.0.smithi045.stdout:op 1316 completed, throughput=4.98MB/sec 2021-07-31T02:09:30.703 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-t2nfwWf-IGsLD-A off=86435 len=719374 2021-07-31T02:09:30.707 INFO:tasks.workunit.client.0.smithi045.stdout:op 1317 completed, throughput=4.99MB/sec 2021-07-31T02:09:30.707 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ivYzZQwPooDiuN1 off=6454701 len=810758 2021-07-31T02:09:30.712 INFO:tasks.workunit.client.0.smithi045.stdout:op 1318 completed, throughput=4.99MB/sec 2021-07-31T02:09:30.713 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ZuhwE9wp836OzOF off=1251491 len=1121774 2021-07-31T02:09:30.714 INFO:tasks.workunit.client.0.smithi045.stdout:op 1319 completed, throughput=4.99MB/sec 2021-07-31T02:09:30.715 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-q6vvCIDYwR3tINX off=1214315 len=179308 2021-07-31T02:09:30.717 INFO:tasks.workunit.client.0.smithi045.stdout:op 1320 completed, throughput=4.99MB/sec 2021-07-31T02:09:30.718 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-skbsYNcz-OPeqzC off=10540275 len=625233 2021-07-31T02:09:30.719 INFO:tasks.workunit.client.0.smithi045.stdout:op 1321 completed, throughput=5MB/sec 2021-07-31T02:09:30.720 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-l6YMy-bnoTqYDPr off=0 len=1204681 2021-07-31T02:09:30.722 INFO:tasks.workunit.client.0.smithi045.stdout:op 1322 completed, throughput=5MB/sec 2021-07-31T02:09:30.723 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-yhggmvvm_2JP5iH off=0 len=1873491 2021-07-31T02:09:30.729 INFO:tasks.workunit.client.0.smithi045.stdout:op 1323 completed, throughput=5.01MB/sec 2021-07-31T02:09:31.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: audit 2021-07-31T02:09:30.292441+0000 mon.b (mon.2) 180 : audit [DBG] 2021-07-31T02:09:31.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:09:31.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: audit 2021-07-31T02:09:30.293889+0000 mon.b (mon.2) 181 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:09:31.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: audit 2021-07-31T02:09:31.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: 2021-07-31T02:09:30.295008+0000 mon.b (mon.2) 182 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:09:31.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:09:30. 2021-07-31T02:09:31.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: 295805+0000 mgr.x (mgr.24542) 950 : cephadm [WRN] 2021-07-31T02:09:31.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:09:31.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:09: 2021-07-31T02:09:31.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: 30.338747+0000 mgr.x (mgr.24542) 951 : cephadm 2021-07-31T02:09:31.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: [ERR] 2021-07-31T02:09:31.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:09:31.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:09:31.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: yield 2021-07-31T02:09:31.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:09:31.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:09:31.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:09:31.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:09:31.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:09:31.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:09:31.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:09:31.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:09:31.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:09:31.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: task.result() 2021-07-31T02:09:31.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:09:31.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:09:30.339215+0000 mgr.x (mgr.24542) 952 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:09:31.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:09:30.350317+0000 mgr.x (mgr.24542) 953 2021-07-31T02:09:31.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: : cephadm [ERR] 2021-07-31T02:09:31.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:09:31.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:09:31.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: yield 2021-07-31T02:09:31.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:09:31.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:09:31.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:09:31.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:09:31.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:09:31.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:09:31.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:09:31.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:09:31.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:09:31.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: task.result() 2021-07-31T02:09:31.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:09:31.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:09:30.350734+0000 mgr.x (mgr.24542) 954 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:09:31.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:09:30.375779+0000 mgr.x (mgr.24542) 955 : cephadm [ERR] 2021-07-31T02:09:31.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:09:31.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:09:31.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: yield 2021-07-31T02:09:31.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:09:31.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:09:31.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:09:31.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:09:31.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:09:31.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:09:31.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:09:31.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:09:31.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:09:31.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: task.result() 2021-07-31T02:09:31.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:09:31.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:09:30.376234+0000 mgr.x (mgr.24542) 956 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:09:31.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:09:31.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:09:31.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: yield 2021-07-31T02:09:31.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:09:31.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:09:31.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:09:31.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:09:31.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:09:31.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:09:31.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:09:31.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:09:31.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:09:31.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: task.result() 2021-07-31T02:09:31.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:09:31.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: 2021-07-31T02:09:31.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:09:31.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: 2021-07-31T02:09:31.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:09:31.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:09:31.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:09:31.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:09:31.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:09:31.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:09:31.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:09:31.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:09:31.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:09:31.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:09:31.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:09:31.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:09:31.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:09:31.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:09:31.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:09:31.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:09:31.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:09:31.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:09:31.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: raise 2021-07-31T02:09:31.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:09:31.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:09:31.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:09:31.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:09:31.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:09:31.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:09:30.384870+0000 mgr.x ( 2021-07-31T02:09:31.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: mgr.24542) 957 : cephadm [ERR] 2021-07-31T02:09:31.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:09:31.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:09:31.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: yield 2021-07-31T02:09:31.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:09:31.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:09:31.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:09:31.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:09:31.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:09:31.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:09:31.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:09:31.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:09:31.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:09:31.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: task.result() 2021-07-31T02:09:31.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:09:31.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:09:30.385694+0000 mgr.x (mgr.24542) 958 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:09:31.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:09:31.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:09:31.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: yield 2021-07-31T02:09:31.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:09:31.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:09:31.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:09:31.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:09:31.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:09:31.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:09:31.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:09:31.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:09:31.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:09:31.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: task.result() 2021-07-31T02:09:31.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:09:31.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: 2021-07-31T02:09:31.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:09:31.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: 2021-07-31T02:09:31.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:09:31.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:09:31.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:09:31.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:09:31.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:09:31.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:09:31.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:09:31.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:09:31.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:09:31.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:09:31.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:09:31.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:09:31.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:09:31.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:09:31.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:09:31.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:09:31.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:09:31.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:09:31.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: raise 2021-07-31T02:09:31.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:09:31.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:09:31.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:09:31.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:09:31.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:31 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:09:31.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: audit 2021-07-31T02:09:30.292441+0000 mon.b ( 2021-07-31T02:09:31.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: mon.2) 180 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:09:31.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: audit 2021-07-31T02:09: 2021-07-31T02:09:31.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: 30.293889+0000 mon.b (mon.2) 181 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:09:31.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: audit 2021-07-31T02:09:30. 2021-07-31T02:09:31.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: 295008+0000 mon.b (mon.2) 182 2021-07-31T02:09:31.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:09:31.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: cephadm 2021-07 2021-07-31T02:09:31.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: -31T02:09:30.295805 2021-07-31T02:09:31.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 950 : cephadm 2021-07-31T02:09:31.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:09:31.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:09:31.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: 2021-07-31T02:09 2021-07-31T02:09:31.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: :30.338747+0000 mgr.x (mgr.24542) 951 : cephadm [ERR] 2021-07-31T02:09:31.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:09:31.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:09:31.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: yield 2021-07-31T02:09:31.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:09:31.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:09:31.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:09:31.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:09:31.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:09:31.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:09:31.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:09:31.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:09:31.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:09:31.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: task.result() 2021-07-31T02:09:31.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:09:31.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:09:30.339215+0000 mgr.x (mgr.24542) 952 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:09:31.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: cephadm 2021-07-31 2021-07-31T02:09:31.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: T02:09:30.350317 2021-07-31T02:09:31.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 953 : cephadm 2021-07-31T02:09:31.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: [ERR] 2021-07-31T02:09:31.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:09:31.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:09:31.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: yield 2021-07-31T02:09:31.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:09:31.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:09:31.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:09:31.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:09:31.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:09:31.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:09:31.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:09:31.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:09:31.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:09:31.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: task.result() 2021-07-31T02:09:31.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:09:31.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:09:30.350734+0000 mgr.x (mgr.24542) 954 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:09:31.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: cephadm 2021-07- 2021-07-31T02:09:31.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: 31T02:09:30.375779+0000 2021-07-31T02:09:31.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: mgr.x (mgr.24542) 955 : cephadm [ERR] 2021-07-31T02:09:31.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: 2021-07-31T02:09:31.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:09:31.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:09:31.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: yield 2021-07-31T02:09:31.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:09:31.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:09:31.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:09:31.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:09:31.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:09:31.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:09:31.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:09:31.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:09:31.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:09:31.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: task.result() 2021-07-31T02:09:31.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:09:31.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:09:30.376234+0000 mgr.x (mgr.24542) 956 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:09:31.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:09:31.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:09:31.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: yield 2021-07-31T02:09:31.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:09:31.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:09:31.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:09:31.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:09:31.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:09:31.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:09:31.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:09:31.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:09:31.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:09:31.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: task.result() 2021-07-31T02:09:31.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:09:31.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: 2021-07-31T02:09:31.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: audit 2021-07-31T02:09:30.292441+0000 2021-07-31T02:09:31.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: mon.b (mon.2) 180 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:09:31.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: audit 2021-07- 2021-07-31T02:09:31.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: 31T02:09:30.293889+0000 mon.b (mon.2) 181 : audit 2021-07-31T02:09:31.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:09:31.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: audit 2021-07-31T02:09 2021-07-31T02:09:31.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: :30.295008+0000 mon.b (mon.2) 182 : 2021-07-31T02:09:31.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:09:31.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: cephadm 2021- 2021-07-31T02:09:31.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: 07-31T02:09:30.295805+0000 mgr.x 2021-07-31T02:09:31.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: (mgr.24542) 950 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:09:31.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:09: 2021-07-31T02:09:31.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: 30.338747+0000 mgr.x (mgr.24542) 951 : cephadm [ERR] 2021-07-31T02:09:31.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: 2021-07-31T02:09:31.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:09:31.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:09:31.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: yield 2021-07-31T02:09:31.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:09:31.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:09:31.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:09:31.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:09:31.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:09:31.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:09:31.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:09:31.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:09:31.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:09:31.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: task.result() 2021-07-31T02:09:31.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:09:31.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: cephadm 2021-07-31T02 2021-07-31T02:09:31.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: :09:30.339215+0000 mgr.x (mgr.24542) 952 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:09:31.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:09:30.350317+0000 mgr.x (mgr.24542) 953 : cephadm [ERR] 2021-07-31T02:09:31.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:09:31.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:09:31.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: yield 2021-07-31T02:09:31.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:09:31.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:09:31.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:09:31.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:09:31.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:09:31.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:09:31.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: 2021-07-31T02:09:31.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:09:31.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:09:31.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:09:31.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:09:31.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:09:31.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:09:31.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:09:31.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:09:31.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:09:31.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:09:31.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:09:31.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:09:31.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:09:31.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:09:31.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:09:31.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:09:31.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:09:31.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:09:31.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: raise 2021-07-31T02:09:31.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:09:31.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:09:31.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:09:31.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:09:31.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:09:31.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: 2021-07-31T02:09:31.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:09:30.384870+0000 mgr.x (mgr.24542) 957 : cephadm [ERR] 2021-07-31T02:09:31.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:09:31.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:09:31.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: yield 2021-07-31T02:09:31.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:09:31.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:09:31.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:09:31.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:09:31.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:09:31.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:09:31.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:09:31.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:09:31.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:09:31.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: task.result() 2021-07-31T02:09:31.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:09:31.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:09:31.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:09:31.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:09:31.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: task.result() 2021-07-31T02:09:31.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:09:31.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:09:30.350734+0000 mgr.x (mgr.24542) 954 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:09:31.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:09:30.375779+0000 mgr.x (mgr.24542) 955 : cephadm [ERR] 2021-07-31T02:09:31.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:09:31.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:09:31.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: yield 2021-07-31T02:09:31.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:09:31.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:09:31.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:09:31.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:09:31.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:09:31.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:09:31.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:09:31.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:09:31.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:09:31.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: task.result() 2021-07-31T02:09:31.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:09:31.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:09:30.376234+0000 mgr.x (mgr.24542) 956 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:09:31.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:09:31.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:09:31.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: yield 2021-07-31T02:09:31.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:09:31.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:09:31.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:09:31.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:09:31.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:09:31.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:09:31.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:09:31.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:09:31.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:09:31.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: task.result() 2021-07-31T02:09:31.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:09:31.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: 2021-07-31T02:09:31.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:09:31.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: 2021-07-31T02:09:31.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:09:31.587 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:09:31.587 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:09:30.385694+0000 mgr.x (mgr.24542) 958 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:09:31.587 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:09:31.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:09:31.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: yield 2021-07-31T02:09:31.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:09:31.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:09:31.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:09:31.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:09:31.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:09:31.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:09:31.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:09:31.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:09:31.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:09:31.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: task.result() 2021-07-31T02:09:31.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:09:31.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: 2021-07-31T02:09:31.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:09:31.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: 2021-07-31T02:09:31.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:09:31.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:09:31.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:09:31.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:09:31.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:09:31.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:09:31.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:09:31.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:09:31.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:09:31.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:09:31.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:09:31.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:09:31.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:09:31.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:09:31.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:09:31.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:09:31.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:09:31.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:09:31.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: raise 2021-07-31T02:09:31.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:09:31.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:09:31.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:09:31.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:09:31.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:09:31.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:09:31.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:09:31.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:09:31.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:09:31.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:09:31.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:09:31.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:09:31.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:09:31.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:09:31.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:09:31.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:09:31.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:09:31.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:09:31.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:09:31.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:09:31.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:09:31.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:09:31.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: raise 2021-07-31T02:09:31.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:09:31.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:09:31.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:09:31.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:09:31.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:09:31.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:09:30.384870+0000 mgr.x (mgr.24542) 957 : cephadm 2021-07-31T02:09:31.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: [ERR] 2021-07-31T02:09:31.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:09:31.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:09:31.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: yield 2021-07-31T02:09:31.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:09:31.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:09:31.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:09:31.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:09:31.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:09:31.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:09:31.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:09:31.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:09:31.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:09:31.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: task.result() 2021-07-31T02:09:31.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:09:31.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:09:30.385694+0000 mgr.x (mgr.24542) 958 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:09:31.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:09:31.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:09:31.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: yield 2021-07-31T02:09:31.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:09:31.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:09:31.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:09:31.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:09:31.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:09:31.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:09:31.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:09:31.608 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:09:31.608 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:09:31.608 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: task.result() 2021-07-31T02:09:31.608 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:09:31.608 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: 2021-07-31T02:09:31.609 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:09:31.609 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: 2021-07-31T02:09:31.609 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:09:31.609 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:09:31.609 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:09:31.610 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:09:31.610 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:09:31.610 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:09:31.610 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:09:31.611 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:09:31.611 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:09:31.611 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:09:31.611 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:09:31.611 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:09:31.612 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:09:31.612 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:09:31.612 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:09:31.612 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:09:31.612 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:09:31.613 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:09:31.613 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: raise 2021-07-31T02:09:31.613 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:09:31.613 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:09:31.614 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:09:31.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:09:31.616 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:31 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:09:31.729 INFO:tasks.workunit.client.0.smithi045.stdout: 260: throughput=4.99MB/sec pending data=0 2021-07-31T02:09:31.729 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-1BJt0dS8GOvfugX off=1013862 len=1350028 2021-07-31T02:09:31.731 INFO:tasks.workunit.client.0.smithi045.stdout:op 1324 completed, throughput=4.99MB/sec 2021-07-31T02:09:31.731 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-VRonhqzGg78cl4N off=6255204 len=452907 2021-07-31T02:09:31.735 INFO:tasks.workunit.client.0.smithi045.stdout:op 1325 completed, throughput=4.99MB/sec 2021-07-31T02:09:31.735 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-gA3ByGWNfH_cCT- off=272140 len=1815748 2021-07-31T02:09:31.745 INFO:tasks.workunit.client.0.smithi045.stdout:op 1326 completed, throughput=5MB/sec 2021-07-31T02:09:32.742 INFO:tasks.workunit.client.0.smithi045.stdout: 261: throughput=4.98MB/sec pending data=0 2021-07-31T02:09:32.743 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-MMVqzaiHtTNocqc off=126834 len=2053637 2021-07-31T02:09:32.752 INFO:tasks.workunit.client.0.smithi045.stdout:op 1327 completed, throughput=4.99MB/sec 2021-07-31T02:09:32.752 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-iZqL6qNDqjwjUql off=21844918 len=1534914 2021-07-31T02:09:32.755 INFO:tasks.workunit.client.0.smithi045.stdout:op 1328 completed, throughput=4.99MB/sec 2021-07-31T02:09:32.755 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-9biLRsOfrPMZsuS off=4436857 len=1701435 2021-07-31T02:09:32.757 INFO:tasks.workunit.client.0.smithi045.stdout:op 1329 completed, throughput=5MB/sec 2021-07-31T02:09:32.757 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-PCQWa7EPKJvkXK8 off=9421604 len=185377 2021-07-31T02:09:32.758 INFO:tasks.workunit.client.0.smithi045.stdout:op 1330 completed, throughput=5MB/sec 2021-07-31T02:09:33.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:32 smithi154 conmon[34553]: cluster 2021-07-31T02:09:32.108975+0000 mgr.x (mgr.24542) 959 : cluster [DBG] pgmap v1541: 9 pgs: 9 active+clean; 127 GiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1.6 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2021-07-31T02:09:33.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:32 smithi045 conmon[39718]: cluster 2021-07-31T02:09:32.108975+0000 mgr.x ( 2021-07-31T02:09:33.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:32 smithi045 conmon[39718]: mgr.24542) 959 : cluster [DBG] pgmap v1541: 9 pgs: 9 active+clean; 127 GiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1.6 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2021-07-31T02:09:33.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:32 smithi045 conmon[33655]: cluster 2021-07-31T02:09:32.108975+0000 mgr.x (mgr.24542) 2021-07-31T02:09:33.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:32 smithi045 conmon[33655]: 959 : cluster [DBG] pgmap v1541: 9 pgs: 9 active+clean; 127 GiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1.6 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2021-07-31T02:09:33.757 INFO:tasks.workunit.client.0.smithi045.stdout: 262: throughput=4.98MB/sec pending data=0 2021-07-31T02:09:33.758 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-uzNjzq8fM6sqsJb off=11213224 len=1978878 2021-07-31T02:09:33.766 INFO:tasks.workunit.client.0.smithi045.stdout:op 1331 completed, throughput=4.99MB/sec 2021-07-31T02:09:33.766 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-VQqtBX05ZWB81kG off=2353894 len=1409912 2021-07-31T02:09:33.771 INFO:tasks.workunit.client.0.smithi045.stdout:op 1332 completed, throughput=4.99MB/sec 2021-07-31T02:09:33.772 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-3hlWtMk5Rqhuuf9 off=23028933 len=586724 2021-07-31T02:09:33.773 INFO:tasks.workunit.client.0.smithi045.stdout:op 1333 completed, throughput=5MB/sec 2021-07-31T02:09:33.773 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-j_nTj7UBuMwTPm8 off=7799949 len=48742 2021-07-31T02:09:33.774 INFO:tasks.workunit.client.0.smithi045.stdout:op 1334 completed, throughput=5MB/sec 2021-07-31T02:09:33.774 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-kGljQU4a5KoBuPn off=2822770 len=1833457 2021-07-31T02:09:33.777 INFO:tasks.workunit.client.0.smithi045.stdout:op 1335 completed, throughput=5MB/sec 2021-07-31T02:09:34.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:09:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:09:33.546Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:09:34.027 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:09:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:09:33.546Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:09:34.027 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:09:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:09:33.546Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:09:34.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:34 smithi154 conmon[34553]: cluster 2021-07-31T02:09:34.109414+0000 mgr.x (mgr.24542) 960 : cluster [DBG] pgmap v1542: 9 pgs: 9 active+clean; 127 GiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 2.7 MiB/s rd, 3.8 MiB/s wr, 6 op/s 2021-07-31T02:09:34.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:34 smithi045 conmon[33655]: cluster 2021-07-31T02:09:34.109414+0000 mgr.x (mgr.24542 2021-07-31T02:09:34.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:34 smithi045 conmon[33655]: ) 960 : cluster [DBG] pgmap v1542: 9 pgs: 9 active+clean; 127 GiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 2.7 MiB/s rd, 3.8 MiB/s wr, 6 op/s 2021-07-31T02:09:34.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:34 smithi045 conmon[39718]: cluster 2021-07-31T02:09:34.109414+0000 mgr.x (mgr 2021-07-31T02:09:34.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:34 smithi045 conmon[39718]: .24542) 960 : cluster [DBG] pgmap v1542: 9 pgs: 9 active+clean; 127 GiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 2.7 MiB/s rd, 3.8 MiB/s wr, 6 op/s 2021-07-31T02:09:34.777 INFO:tasks.workunit.client.0.smithi045.stdout: 263: throughput=4.98MB/sec pending data=0 2021-07-31T02:09:34.777 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-PjZbfV6KN3f57jY off=2069076 len=497559 2021-07-31T02:09:34.778 INFO:tasks.workunit.client.0.smithi045.stdout:op 1336 completed, throughput=4.99MB/sec 2021-07-31T02:09:34.778 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-bkKWlvFP_PFvbYd off=6528335 len=627608 2021-07-31T02:09:34.783 INFO:tasks.workunit.client.0.smithi045.stdout:op 1337 completed, throughput=4.99MB/sec 2021-07-31T02:09:34.783 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-arBiO6atXfVl3cT off=6303521 len=217557 2021-07-31T02:09:34.784 INFO:tasks.workunit.client.0.smithi045.stdout:op 1338 completed, throughput=4.99MB/sec 2021-07-31T02:09:34.784 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-kLZRSFyPCgZTpCU off=16388282 len=1358468 2021-07-31T02:09:34.785 INFO:tasks.workunit.client.0.smithi045.stdout:op 1339 completed, throughput=4.99MB/sec 2021-07-31T02:09:34.785 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-2cQIbM_J33tqS9C off=10635212 len=38152 2021-07-31T02:09:34.786 INFO:tasks.workunit.client.0.smithi045.stdout:op 1340 completed, throughput=4.99MB/sec 2021-07-31T02:09:34.786 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-zr9Ny6Ck69VD3F5 off=14510339 len=140723 2021-07-31T02:09:34.787 INFO:tasks.workunit.client.0.smithi045.stdout:op 1341 completed, throughput=4.99MB/sec 2021-07-31T02:09:34.787 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-E1HI4i2DRbxdxYy off=12706133 len=293721 2021-07-31T02:09:34.788 INFO:tasks.workunit.client.0.smithi045.stdout:op 1342 completed, throughput=4.99MB/sec 2021-07-31T02:09:34.788 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-g8W7cJwNg7PK998 off=10611171 len=663905 2021-07-31T02:09:34.789 INFO:tasks.workunit.client.0.smithi045.stdout:op 1343 completed, throughput=5MB/sec 2021-07-31T02:09:34.789 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-rIX-rkqm10uOrJQ off=89469 len=188943 2021-07-31T02:09:34.790 INFO:tasks.workunit.client.0.smithi045.stdout:op 1344 completed, throughput=5MB/sec 2021-07-31T02:09:34.790 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-xPHh1f4uqVd_Exz off=4998345 len=1363886 2021-07-31T02:09:34.792 INFO:tasks.workunit.client.0.smithi045.stdout:op 1345 completed, throughput=5MB/sec 2021-07-31T02:09:35.793 INFO:tasks.workunit.client.0.smithi045.stdout: 264: throughput=4.98MB/sec pending data=0 2021-07-31T02:09:35.793 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-oBwimPpXeCoErHH off=12230816 len=216220 2021-07-31T02:09:35.794 INFO:tasks.workunit.client.0.smithi045.stdout:op 1346 completed, throughput=4.98MB/sec 2021-07-31T02:09:35.794 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-EXs3fnfmoYZ9s0t off=16754455 len=1990965 2021-07-31T02:09:35.804 INFO:tasks.workunit.client.0.smithi045.stdout:op 1347 completed, throughput=4.99MB/sec 2021-07-31T02:09:35.804 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-UgSe5Tv07m7PmA6 off=1497225 len=1377627 2021-07-31T02:09:35.807 INFO:tasks.workunit.client.0.smithi045.stdout:op 1348 completed, throughput=5MB/sec 2021-07-31T02:09:35.807 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Pdx6bQQXfprJ-FK off=15477134 len=305225 2021-07-31T02:09:35.811 INFO:tasks.workunit.client.0.smithi045.stdout:op 1349 completed, throughput=5MB/sec 2021-07-31T02:09:35.811 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-4KT6bZJSojrj952 off=11869428 len=1443170 2021-07-31T02:09:35.818 INFO:tasks.workunit.client.0.smithi045.stdout:op 1350 completed, throughput=5MB/sec 2021-07-31T02:09:36.363 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:09:36.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:36 smithi154 conmon[34553]: cluster 2021-07-31T02:09:36.109886+0000 mgr.x (mgr.24542) 961 : cluster [DBG] pgmap v1543: 9 pgs: 9 active+clean; 127 GiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2021-07-31T02:09:36.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:36 smithi045 conmon[33655]: cluster 2021-07-31T02:09:36.109886 2021-07-31T02:09:36.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:36 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 961 : cluster [DBG] pgmap v1543: 9 pgs: 9 active+clean; 127 GiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2021-07-31T02:09:36.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:36 smithi045 conmon[39718]: cluster 2021-07-31T02:09:36.109886+0000 mgr.x ( 2021-07-31T02:09:36.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:36 smithi045 conmon[39718]: mgr.24542) 961 : cluster [DBG] pgmap v1543: 9 pgs: 9 active+clean; 127 GiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2021-07-31T02:09:36.641 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:09:36.642 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 24m ago 25m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:09:36.642 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 22m ago 24m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:09:36.643 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 22m ago 28m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:09:36.643 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 24m ago 30m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:09:36.643 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 24m ago 30m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:09:36.644 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 22m ago 28m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:09:36.644 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 24m ago 28m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:09:36.644 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 24m ago 25m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:09:36.644 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 22m ago 25m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:09:36.645 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 24m ago 28m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:09:36.645 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 24m ago 27m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:09:36.645 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 24m ago 27m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:09:36.645 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 24m ago 27m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:09:36.645 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 22m ago 26m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:09:36.646 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 22m ago 26m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:09:36.646 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 22m ago 26m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:09:36.646 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 22m ago 25m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:09:36.646 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 22m ago 25m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:09:36.818 INFO:tasks.workunit.client.0.smithi045.stdout: 265: throughput=4.98MB/sec pending data=0 2021-07-31T02:09:36.818 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Db7c6LJHwQNocTj off=4267714 len=899105 2021-07-31T02:09:36.819 INFO:tasks.workunit.client.0.smithi045.stdout:op 1351 completed, throughput=4.99MB/sec 2021-07-31T02:09:36.820 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-goZDI2me3pK1W-h off=4355149 len=690974 2021-07-31T02:09:36.824 INFO:tasks.workunit.client.0.smithi045.stdout:op 1352 completed, throughput=4.99MB/sec 2021-07-31T02:09:36.824 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-b-ZHcdEYDvWCxgy off=9273581 len=573107 2021-07-31T02:09:36.825 INFO:tasks.workunit.client.0.smithi045.stdout:op 1353 completed, throughput=4.99MB/sec 2021-07-31T02:09:36.825 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-DMl4w6S8AbTAOG0 off=982933 len=13015 2021-07-31T02:09:36.827 INFO:tasks.workunit.client.0.smithi045.stdout:op 1354 completed, throughput=4.99MB/sec 2021-07-31T02:09:36.827 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Qx_uF8FOuXIgAoR off=12928922 len=1328287 2021-07-31T02:09:36.833 INFO:tasks.workunit.client.0.smithi045.stdout:op 1355 completed, throughput=5MB/sec 2021-07-31T02:09:36.833 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-_NnIAXKsLFqVc69 off=16504109 len=1812150 2021-07-31T02:09:36.836 INFO:tasks.workunit.client.0.smithi045.stdout:op 1356 completed, throughput=5MB/sec 2021-07-31T02:09:36.980 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:09:36.980 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:09:36.981 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:09:36.981 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:09:36.981 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:09:36.982 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:09:36.982 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:09:36.982 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:09:36.982 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:09:36.983 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:09:36.983 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:09:36.983 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:09:36.984 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:09:36.984 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:09:36.984 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:09:36.985 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:09:36.985 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:09:37.837 INFO:tasks.workunit.client.0.smithi045.stdout: 266: throughput=4.98MB/sec pending data=0 2021-07-31T02:09:37.837 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-vl0_q5xjfalvxAs off=2434861 len=1194498 2021-07-31T02:09:37.840 INFO:tasks.workunit.client.0.smithi045.stdout:op 1357 completed, throughput=4.99MB/sec 2021-07-31T02:09:37.840 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ucBMYmdwfv8TZ-E off=7624664 len=1152646 2021-07-31T02:09:37.847 INFO:tasks.workunit.client.0.smithi045.stdout:op 1358 completed, throughput=4.99MB/sec 2021-07-31T02:09:37.847 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-6TpOrS5kN4U4pKE off=3496962 len=913426 2021-07-31T02:09:37.849 INFO:tasks.workunit.client.0.smithi045.stdout:op 1359 completed, throughput=5MB/sec 2021-07-31T02:09:37.849 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-5Bj46l09VDdWX6Q off=3240580 len=160282 2021-07-31T02:09:37.850 INFO:tasks.workunit.client.0.smithi045.stdout:op 1360 completed, throughput=5MB/sec 2021-07-31T02:09:37.850 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj--7zkYfgq05Nel47 off=9064210 len=816879 2021-07-31T02:09:37.855 INFO:tasks.workunit.client.0.smithi045.stdout:op 1361 completed, throughput=5MB/sec 2021-07-31T02:09:37.858 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-X0BoZHLh9zactg- off=7384304 len=1366578 2021-07-31T02:09:37.861 INFO:tasks.workunit.client.0.smithi045.stdout:op 1362 completed, throughput=5MB/sec 2021-07-31T02:09:38.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:37 smithi154 conmon[34553]: audit 2021-07-31T02:09:36.350297+0000 mgr.x (mgr.24542) 962 : audit [DBG] from='client.15879 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:09:38.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:37 smithi154 conmon[34553]: audit 2021-07-31T02:09:36.633583+0000 mgr.x (mgr.24542) 963 : audit [DBG] from='client.15885 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:09:38.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:37 smithi154 conmon[34553]: audit 2021-07-31T02:09:36.979844+0000 mon.c (mon.1) 79 : audit [DBG] from='client.? 172.21.15.45:0/84070454' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:09:38.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:37 smithi154 conmon[34553]: 2021-07-31T02:09:38.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:37 smithi045 conmon[33655]: audit 2021-07-31T02:09:36.350297+0000 mgr.x (mgr.24542) 962 2021-07-31T02:09:38.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:37 smithi045 conmon[33655]: : audit [DBG] from='client.15879 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:09:38.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:37 smithi045 conmon[33655]: audit 2021-07-31T02:09:36.633583+0000 mgr.x (mgr.24542) 963 : audit [DBG] from='client.15885 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:09:38.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:37 smithi045 conmon[33655]: audit 2021-07-31T 2021-07-31T02:09:38.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:37 smithi045 conmon[33655]: 02:09:36.979844+0000 mon.c (mon.1) 79 : audit [DBG] from='client.? 172.21.15.45:0/84070454' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:09:38.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:37 smithi045 conmon[39718]: audit 2021-07-31T02:09:36.350297+0000 mgr.x (mgr 2021-07-31T02:09:38.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:37 smithi045 conmon[39718]: .24542) 962 : audit [DBG] from='client.15879 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:09:38.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:37 smithi045 conmon[39718]: audit 2021-07-31T02:09:36.633583+0000 mgr.x (mgr.24542) 2021-07-31T02:09:38.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:37 smithi045 conmon[39718]: 963 : audit [DBG] from='client.15885 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:09:38.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:37 smithi045 conmon[39718]: audit 2021 2021-07-31T02:09:38.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:37 smithi045 conmon[39718]: -07-31T02:09:36.979844+0000 mon.c (mon 2021-07-31T02:09:38.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:37 smithi045 conmon[39718]: .1) 79 : audit [DBG] from='client.? 172.21.15.45:0/84070454' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:09:38.862 INFO:tasks.workunit.client.0.smithi045.stdout: 267: throughput=4.98MB/sec pending data=0 2021-07-31T02:09:38.862 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Ml0toHrKAZKd9Ni off=2184431 len=1629168 2021-07-31T02:09:38.865 INFO:tasks.workunit.client.0.smithi045.stdout:op 1363 completed, throughput=4.99MB/sec 2021-07-31T02:09:38.865 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-EoNGeSelxURWflt off=298577 len=971063 2021-07-31T02:09:38.866 INFO:tasks.workunit.client.0.smithi045.stdout:op 1364 completed, throughput=4.99MB/sec 2021-07-31T02:09:38.867 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-VJZF1OgHieG_Mgx off=766554 len=1748061 2021-07-31T02:09:38.875 INFO:tasks.workunit.client.0.smithi045.stdout:op 1365 completed, throughput=5MB/sec 2021-07-31T02:09:38.875 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-VON5yHOisV0714R off=1991249 len=1415778 2021-07-31T02:09:38.882 INFO:tasks.workunit.client.0.smithi045.stdout:op 1366 completed, throughput=5MB/sec 2021-07-31T02:09:39.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:38 smithi154 conmon[34553]: cluster 2021-07-31T02:09:38.110432+0000 mgr.x (mgr.24542) 964 : cluster 2021-07-31T02:09:39.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:38 smithi154 conmon[34553]: [DBG] pgmap v1544: 9 pgs: 9 active+clean; 127 GiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2021-07-31T02:09:39.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:38 smithi045 conmon[33655]: cluster 2021-07-31T02:09:38.110432+0000 mgr.x (mgr.24542 2021-07-31T02:09:39.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:38 smithi045 conmon[33655]: ) 964 : cluster [DBG] pgmap v1544: 9 pgs: 9 active+clean; 127 GiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2021-07-31T02:09:39.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:38 smithi045 conmon[39718]: cluster 2021-07-31T02:09:38.110432+0000 2021-07-31T02:09:39.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:38 smithi045 conmon[39718]: mgr.x (mgr.24542) 964 : cluster [DBG] pgmap v1544: 9 pgs: 9 active+clean; 127 GiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2021-07-31T02:09:39.881 INFO:tasks.workunit.client.0.smithi045.stdout: 268: throughput=4.99MB/sec pending data=0 2021-07-31T02:09:39.882 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Dkklr5eXD2IL8Yy off=2909845 len=659023 2021-07-31T02:09:39.883 INFO:tasks.workunit.client.0.smithi045.stdout:op 1367 completed, throughput=4.99MB/sec 2021-07-31T02:09:39.883 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-lEUmxkBV2ym8Jlk off=6515531 len=1440004 2021-07-31T02:09:39.887 INFO:tasks.workunit.client.0.smithi045.stdout:op 1368 completed, throughput=4.99MB/sec 2021-07-31T02:09:39.889 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-zD2bjMQBYqZ-qBI off=12772449 len=1204987 2021-07-31T02:09:39.893 INFO:tasks.workunit.client.0.smithi045.stdout:op 1369 completed, throughput=5MB/sec 2021-07-31T02:09:39.894 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-SYXP22-f3o8NkTx off=9209698 len=1414811 2021-07-31T02:09:39.900 INFO:tasks.workunit.client.0.smithi045.stdout:op 1370 completed, throughput=5MB/sec 2021-07-31T02:09:40.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:40 smithi154 conmon[34553]: cluster 2021-07-31T02:09:40.110961+0000 mgr.x (mgr.24542) 965 : cluster [DBG] pgmap v1545: 9 pgs: 9 active+clean; 127 GiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 2.9 MiB/s rd, 3.4 MiB/s wr, 6 op/s 2021-07-31T02:09:40.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:40 smithi154 conmon[34553]: 2021-07-31T02:09:40.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:40 smithi045 conmon[39718]: cluster 2021-07-31T02:09:40.110961+0000 mgr.x ( 2021-07-31T02:09:40.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:40 smithi045 conmon[39718]: mgr.24542) 965 : cluster [DBG] pgmap v1545: 9 pgs: 9 active+clean; 127 GiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 2.9 MiB/s rd, 3.4 MiB/s wr, 6 op/s 2021-07-31T02:09:40.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:40 smithi045 conmon[33655]: cluster 2021-07-31T02:09:40.110961+0000 mgr.x (mgr.24542) 965 2021-07-31T02:09:40.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:40 smithi045 conmon[33655]: : cluster [DBG] pgmap v1545: 9 pgs: 9 active+clean; 127 GiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 2.9 MiB/s rd, 3.4 MiB/s wr, 6 op/s 2021-07-31T02:09:40.900 INFO:tasks.workunit.client.0.smithi045.stdout: 269: throughput=4.98MB/sec pending data=0 2021-07-31T02:09:40.901 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-8gPNGz7-60JGBX- off=2080633 len=2028826 2021-07-31T02:09:40.904 INFO:tasks.workunit.client.0.smithi045.stdout:op 1371 completed, throughput=4.99MB/sec 2021-07-31T02:09:40.905 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Wq8qdZP51nzsWvQ off=5277869 len=1179625 2021-07-31T02:09:40.911 INFO:tasks.workunit.client.0.smithi045.stdout:op 1372 completed, throughput=5MB/sec 2021-07-31T02:09:40.911 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-G0nqrhR6YU9cSt0 off=741833 len=351932 2021-07-31T02:09:40.912 INFO:tasks.workunit.client.0.smithi045.stdout:op 1373 completed, throughput=5MB/sec 2021-07-31T02:09:40.912 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-JyBiRvb8EmY7hGY off=15297458 len=1296933 2021-07-31T02:09:40.913 INFO:tasks.workunit.client.0.smithi045.stdout:op 1374 completed, throughput=5MB/sec 2021-07-31T02:09:41.910 INFO:tasks.workunit.client.0.smithi045.stdout: 270: throughput=4.98MB/sec pending data=0 2021-07-31T02:09:41.911 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ksU_Uk5v2eyLAJ_ off=1213059 len=675709 2021-07-31T02:09:41.916 INFO:tasks.workunit.client.0.smithi045.stdout:op 1375 completed, throughput=4.99MB/sec 2021-07-31T02:09:41.916 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-344JD2CQ8-PIKwN off=1532798 len=1598449 2021-07-31T02:09:41.922 INFO:tasks.workunit.client.0.smithi045.stdout:op 1376 completed, throughput=4.99MB/sec 2021-07-31T02:09:41.923 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-9nN42m0dsRvD5FY off=4330191 len=179915 2021-07-31T02:09:41.923 INFO:tasks.workunit.client.0.smithi045.stdout:op 1377 completed, throughput=4.99MB/sec 2021-07-31T02:09:41.924 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-aqRdfNgi6Il95wC off=7778526 len=1233162 2021-07-31T02:09:41.926 INFO:tasks.workunit.client.0.smithi045.stdout:op 1378 completed, throughput=5MB/sec 2021-07-31T02:09:41.926 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-vT9ZvGHISHzIbKR off=6787390 len=1219024 2021-07-31T02:09:41.932 INFO:tasks.workunit.client.0.smithi045.stdout:op 1379 completed, throughput=5MB/sec 2021-07-31T02:09:41.933 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-m3qjk1YV2FInWHX off=6353014 len=370302 2021-07-31T02:09:41.933 INFO:tasks.workunit.client.0.smithi045.stdout:op 1380 completed, throughput=5MB/sec 2021-07-31T02:09:42.934 INFO:tasks.workunit.client.0.smithi045.stdout: 271: throughput=4.98MB/sec pending data=0 2021-07-31T02:09:42.934 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-oupZ8hEyNI6iZOO off=1065416 len=1489168 2021-07-31T02:09:42.935 INFO:tasks.workunit.client.0.smithi045.stdout:op 1381 completed, throughput=4.99MB/sec 2021-07-31T02:09:42.935 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Z8ftF6T3vs1zREJ off=3434784 len=119748 2021-07-31T02:09:42.939 INFO:tasks.workunit.client.0.smithi045.stdout:op 1382 completed, throughput=4.99MB/sec 2021-07-31T02:09:42.939 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-SNJAJU_E43Cf8fo off=10397371 len=1895236 2021-07-31T02:09:42.941 INFO:tasks.workunit.client.0.smithi045.stdout:op 1383 completed, throughput=4.99MB/sec 2021-07-31T02:09:42.942 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-z7j-rx2n5Z3o99z off=2899498 len=634798 2021-07-31T02:09:42.943 INFO:tasks.workunit.client.0.smithi045.stdout:op 1384 completed, throughput=5MB/sec 2021-07-31T02:09:42.943 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-qTbt4vAo_ak3ey2 off=860398 len=1373371 2021-07-31T02:09:42.952 INFO:tasks.workunit.client.0.smithi045.stdout:op 1385 completed, throughput=5MB/sec 2021-07-31T02:09:43.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:42 smithi154 conmon[34553]: cluster 2021-07-31T02:09:42.111495+0000 mgr.x (mgr.24542) 966 : cluster [DBG] pgmap v1546: 9 pgs: 9 active+clean; 127 GiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2021-07-31T02:09:43.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:42 smithi045 conmon[39718]: cluster 2021-07-31T02:09:42.111495 2021-07-31T02:09:43.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:42 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 966 : cluster [DBG] pgmap v1546: 9 pgs: 9 active+clean; 127 GiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2021-07-31T02:09:43.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:42 smithi045 conmon[33655]: cluster 2021-07-31T02:09:42.111495+0000 mgr.x (mgr.24542) 966 : 2021-07-31T02:09:43.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:42 smithi045 conmon[33655]: cluster [DBG] pgmap v1546: 9 pgs: 9 active+clean; 127 GiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2021-07-31T02:09:43.952 INFO:tasks.workunit.client.0.smithi045.stdout: 272: throughput=4.98MB/sec pending data=0 2021-07-31T02:09:43.952 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-8c_NOUoV6OYLiqU off=3481589 len=1380343 2021-07-31T02:09:43.958 INFO:tasks.workunit.client.0.smithi045.stdout:op 1386 completed, throughput=4.99MB/sec 2021-07-31T02:09:43.959 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-XdLrYvQBh1wFMfJ off=11242410 len=721342 2021-07-31T02:09:43.960 INFO:tasks.workunit.client.0.smithi045.stdout:op 1387 completed, throughput=4.99MB/sec 2021-07-31T02:09:43.960 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-UyqTsxaPoonsoS- off=11342064 len=1916440 2021-07-31T02:09:43.962 INFO:tasks.workunit.client.0.smithi045.stdout:op 1388 completed, throughput=5MB/sec 2021-07-31T02:09:43.963 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-GRbYUXfI_IaGDSX off=6107271 len=793820 2021-07-31T02:09:43.968 INFO:tasks.workunit.client.0.smithi045.stdout:op 1389 completed, throughput=5MB/sec 2021-07-31T02:09:43.969 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-xG9CP1l5c8Lmk44 off=15847873 len=1623630 2021-07-31T02:09:43.977 INFO:tasks.workunit.client.0.smithi045.stdout:op 1390 completed, throughput=5MB/sec 2021-07-31T02:09:44.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:09:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:09:43.546Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:09:44.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:09:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:09:43.546Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:09:44.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:09:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:09:43.546Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:09:44.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:44 smithi154 conmon[34553]: cluster 2021-07-31T02:09:44.112066+0000 mgr.x (mgr.24542) 967 : cluster [DBG] pgmap v1547: 9 pgs: 9 active+clean; 127 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.8 MiB/s wr, 6 op/s 2021-07-31T02:09:44.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:44 smithi045 conmon[33655]: cluster 2021-07-31T02:09:44.112066+0000 mgr.x (mgr.24542) 967 : 2021-07-31T02:09:44.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:44 smithi045 conmon[33655]: cluster [DBG] pgmap v1547: 9 pgs: 9 active+clean; 127 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.8 MiB/s wr, 6 op/s 2021-07-31T02:09:44.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:44 smithi045 conmon[39718]: cluster 2021-07-31T02:09:44.112066+0000 mgr.x (mgr.24542) 967 : cluster [DBG] 2021-07-31T02:09:44.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:44 smithi045 conmon[39718]: pgmap v1547: 9 pgs: 9 active+clean; 127 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.8 MiB/s wr, 6 op/s 2021-07-31T02:09:44.978 INFO:tasks.workunit.client.0.smithi045.stdout: 273: throughput=4.99MB/sec pending data=0 2021-07-31T02:09:44.978 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-a2pySLU18jNAuk3 off=9592030 len=538783 2021-07-31T02:09:44.979 INFO:tasks.workunit.client.0.smithi045.stdout:op 1391 completed, throughput=4.99MB/sec 2021-07-31T02:09:44.979 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-9eUT9y-QYVY6F05 off=10682260 len=699101 2021-07-31T02:09:44.980 INFO:tasks.workunit.client.0.smithi045.stdout:op 1392 completed, throughput=4.99MB/sec 2021-07-31T02:09:44.980 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-yT_bPIpDN3LRUOc off=3190103 len=2060259 2021-07-31T02:09:44.982 INFO:tasks.workunit.client.0.smithi045.stdout:op 1393 completed, throughput=5MB/sec 2021-07-31T02:09:44.982 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-vUrC8wcZ_yr6Eiw off=1452916 len=1874557 2021-07-31T02:09:44.998 INFO:tasks.workunit.client.0.smithi045.stdout:op 1394 completed, throughput=5MB/sec 2021-07-31T02:09:45.998 INFO:tasks.workunit.client.0.smithi045.stdout: 274: throughput=4.99MB/sec pending data=0 2021-07-31T02:09:45.998 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-8eVjIcHHuDlhZ9q off=977546 len=1478615 2021-07-31T02:09:45.999 INFO:tasks.workunit.client.0.smithi045.stdout:op 1395 completed, throughput=4.99MB/sec 2021-07-31T02:09:46.000 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-VY8zdbdIbBNKzjs off=572103 len=1358746 2021-07-31T02:09:46.007 INFO:tasks.workunit.client.0.smithi045.stdout:op 1396 completed, throughput=5MB/sec 2021-07-31T02:09:46.007 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Hv_i7Mu7KBGrl_6 off=4740701 len=1159630 2021-07-31T02:09:46.009 INFO:tasks.workunit.client.0.smithi045.stdout:op 1397 completed, throughput=5MB/sec 2021-07-31T02:09:46.009 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-7rOT6Q5pYEFCQS0 off=6653382 len=1803984 2021-07-31T02:09:46.020 INFO:tasks.workunit.client.0.smithi045.stdout:op 1398 completed, throughput=5.01MB/sec 2021-07-31T02:09:46.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:46 smithi154 conmon[34553]: cluster 2021-07-31T02:09:46.112560+0000 mgr.x (mgr.24542) 968 : cluster [DBG] pgmap v1548: 9 pgs: 9 active+clean; 127 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 2.2 MiB/s rd, 1.6 MiB/s wr, 4 op/s 2021-07-31T02:09:46.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:46 smithi045 conmon[33655]: cluster 2021-07-31T02:09:46.112560+0000 mgr.x (mgr.24542) 2021-07-31T02:09:46.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:46 smithi045 conmon[33655]: 968 : cluster [DBG] pgmap v1548: 9 pgs: 9 active+clean; 127 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 2.2 MiB/s rd, 1.6 MiB/s wr, 4 op/s 2021-07-31T02:09:46.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:46 smithi045 conmon[39718]: cluster 2021-07-31T02:09:46.112560+0000 mgr.x (mgr.24542) 968 : cluster [DBG] 2021-07-31T02:09:46.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:46 smithi045 conmon[39718]: pgmap v1548: 9 pgs: 9 active+clean; 127 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 2.2 MiB/s rd, 1.6 MiB/s wr, 4 op/s 2021-07-31T02:09:47.020 INFO:tasks.workunit.client.0.smithi045.stdout: 275: throughput=4.99MB/sec pending data=0 2021-07-31T02:09:47.021 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-JhZp-DPtzU5R1Zf off=3584250 len=1100676 2021-07-31T02:09:47.022 INFO:tasks.workunit.client.0.smithi045.stdout:op 1399 completed, throughput=4.99MB/sec 2021-07-31T02:09:47.022 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-XIkNgzn0erEf6o8 off=9590362 len=1663509 2021-07-31T02:09:47.025 INFO:tasks.workunit.client.0.smithi045.stdout:op 1400 completed, throughput=5MB/sec 2021-07-31T02:09:47.026 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-xaM_xU2F4qtHT-4 off=11260992 len=1985372 2021-07-31T02:09:47.036 INFO:tasks.workunit.client.0.smithi045.stdout:op 1401 completed, throughput=5MB/sec 2021-07-31T02:09:48.035 INFO:tasks.workunit.client.0.smithi045.stdout: 276: throughput=4.99MB/sec pending data=0 2021-07-31T02:09:48.036 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-OAF4jbxUncGio0t off=0 len=1712553 2021-07-31T02:09:48.044 INFO:tasks.workunit.client.0.smithi045.stdout:op 1402 completed, throughput=4.99MB/sec 2021-07-31T02:09:48.044 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-HKYrL4xhWqDrdfy off=1822715 len=296536 2021-07-31T02:09:48.047 INFO:tasks.workunit.client.0.smithi045.stdout:op 1403 completed, throughput=4.99MB/sec 2021-07-31T02:09:48.047 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj--rdkxMFINt7iYS_ off=1952089 len=152693 2021-07-31T02:09:48.048 INFO:tasks.workunit.client.0.smithi045.stdout:op 1404 completed, throughput=4.99MB/sec 2021-07-31T02:09:48.048 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-2m_AuBBVBqe_YXR off=3414282 len=276072 2021-07-31T02:09:48.049 INFO:tasks.workunit.client.0.smithi045.stdout:op 1405 completed, throughput=4.99MB/sec 2021-07-31T02:09:48.049 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ZMph6kkMhJVAfqS off=20179815 len=617781 2021-07-31T02:09:48.051 INFO:tasks.workunit.client.0.smithi045.stdout:op 1406 completed, throughput=5MB/sec 2021-07-31T02:09:48.051 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-17R80h1QyUm5CtH off=994563 len=394978 2021-07-31T02:09:48.052 INFO:tasks.workunit.client.0.smithi045.stdout:op 1407 completed, throughput=5MB/sec 2021-07-31T02:09:48.052 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-IT994w-zzTq5TSD off=680600 len=671155 2021-07-31T02:09:48.057 INFO:tasks.workunit.client.0.smithi045.stdout:op 1408 completed, throughput=5MB/sec 2021-07-31T02:09:48.057 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-_qWEP_xtB7_goLj off=7473129 len=1970207 2021-07-31T02:09:48.059 INFO:tasks.workunit.client.0.smithi045.stdout:op 1409 completed, throughput=5.01MB/sec 2021-07-31T02:09:48.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:48 smithi154 conmon[34553]: cluster 2021-07-31T02:09:48.112932+0000 mgr.x (mgr.24542) 969 : cluster 2021-07-31T02:09:48.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:48 smithi154 conmon[34553]: [DBG] pgmap v1549: 9 pgs: 9 active+clean; 127 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 2.2 MiB/s rd, 1.6 MiB/s wr, 4 op/s 2021-07-31T02:09:48.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:48 smithi045 conmon[33655]: cluster 2021-07-31T02:09:48.112932 2021-07-31T02:09:48.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:48 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 969 : cluster [DBG] pgmap v1549: 9 pgs: 9 active+clean; 127 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 2.2 MiB/s rd, 1.6 MiB/s wr, 4 op/s 2021-07-31T02:09:48.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:48 smithi045 conmon[39718]: cluster 2021-07-31T02:09:48.112932+0000 mgr.x (mgr.24542) 969 : cluster [DBG] pgmap v1549: 9 pgs: 9 active+clean; 127 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 2.2 MiB/s rd, 1.6 MiB/s wr, 4 op/s 2021-07-31T02:09:48.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:48 smithi045 conmon[39718]: 2021-07-31T02:09:49.060 INFO:tasks.workunit.client.0.smithi045.stdout: 277: throughput=4.99MB/sec pending data=0 2021-07-31T02:09:49.060 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-DHBZ1U-gAflILvc off=10417378 len=1122896 2021-07-31T02:09:49.063 INFO:tasks.workunit.client.0.smithi045.stdout:op 1410 completed, throughput=4.99MB/sec 2021-07-31T02:09:49.063 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-0GdeCPak4c1nShd off=2143167 len=767684 2021-07-31T02:09:49.069 INFO:tasks.workunit.client.0.smithi045.stdout:op 1411 completed, throughput=4.99MB/sec 2021-07-31T02:09:49.069 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-6dZa4xNX7zPgQex off=12134439 len=513064 2021-07-31T02:09:49.072 INFO:tasks.workunit.client.0.smithi045.stdout:op 1412 completed, throughput=5MB/sec 2021-07-31T02:09:49.072 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-yQe_hw6G1dy6OFy off=5626212 len=125397 2021-07-31T02:09:49.073 INFO:tasks.workunit.client.0.smithi045.stdout:op 1413 completed, throughput=5MB/sec 2021-07-31T02:09:49.073 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-s202OmuUknJh65O off=6187202 len=1248021 2021-07-31T02:09:49.079 INFO:tasks.workunit.client.0.smithi045.stdout:op 1414 completed, throughput=5MB/sec 2021-07-31T02:09:50.079 INFO:tasks.workunit.client.0.smithi045.stdout: 278: throughput=4.98MB/sec pending data=0 2021-07-31T02:09:50.080 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-45aE6UGzR5KYQuS off=15646428 len=588191 2021-07-31T02:09:50.081 INFO:tasks.workunit.client.0.smithi045.stdout:op 1415 completed, throughput=4.99MB/sec 2021-07-31T02:09:50.081 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-hL-XkJs2AzCsEck off=5549531 len=494390 2021-07-31T02:09:50.082 INFO:tasks.workunit.client.0.smithi045.stdout:op 1416 completed, throughput=4.99MB/sec 2021-07-31T02:09:50.083 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-uv-AUS495Bp9o2U off=7854085 len=1210951 2021-07-31T02:09:50.089 INFO:tasks.workunit.client.0.smithi045.stdout:op 1417 completed, throughput=4.99MB/sec 2021-07-31T02:09:50.090 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-k_rHJ26RLSinwAQ off=1204827 len=1042309 2021-07-31T02:09:50.091 INFO:tasks.workunit.client.0.smithi045.stdout:op 1418 completed, throughput=4.99MB/sec 2021-07-31T02:09:50.091 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-VvPXuptRnauYts7 off=7975857 len=113162 2021-07-31T02:09:50.092 INFO:tasks.workunit.client.0.smithi045.stdout:op 1419 completed, throughput=4.99MB/sec 2021-07-31T02:09:50.093 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ldwJi3O9Xv__eb1 off=4731243 len=869258 2021-07-31T02:09:50.096 INFO:tasks.workunit.client.0.smithi045.stdout:op 1420 completed, throughput=5MB/sec 2021-07-31T02:09:50.097 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-tGdzYg0344MOioV off=10898219 len=1973719 2021-07-31T02:09:50.100 INFO:tasks.workunit.client.0.smithi045.stdout:op 1421 completed, throughput=5MB/sec 2021-07-31T02:09:50.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:50 smithi154 conmon[34553]: cluster 2021-07-31T02:09:50.113437+0000 mgr.x (mgr.24542) 970 : cluster [DBG] pgmap v1550: 9 pgs: 9 active+clean; 127 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 3.3 MiB/s rd, 2.7 MiB/s wr, 6 op/s 2021-07-31T02:09:50.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:50 smithi045 conmon[33655]: cluster 2021-07-31T02:09:50.113437+0000 mgr.x (mgr.24542) 970 : cluster 2021-07-31T02:09:50.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:50 smithi045 conmon[33655]: [DBG] pgmap v1550: 9 pgs: 9 active+clean; 127 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 3.3 MiB/s rd, 2.7 MiB/s wr, 6 op/s 2021-07-31T02:09:50.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:50 smithi045 conmon[39718]: cluster 2021-07-31T02:09:50.113437+0000 mgr.x 2021-07-31T02:09:50.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:50 smithi045 conmon[39718]: (mgr.24542) 970 : cluster [DBG] pgmap v1550: 9 pgs: 9 active+clean; 127 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 3.3 MiB/s rd, 2.7 MiB/s wr, 6 op/s 2021-07-31T02:09:51.100 INFO:tasks.workunit.client.0.smithi045.stdout: 279: throughput=4.99MB/sec pending data=0 2021-07-31T02:09:51.100 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-sGaG7lbW2L5lJWn off=11755027 len=1518391 2021-07-31T02:09:51.108 INFO:tasks.workunit.client.0.smithi045.stdout:op 1422 completed, throughput=4.99MB/sec 2021-07-31T02:09:51.109 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-8grCejhkLmalBWF off=7902937 len=9379 2021-07-31T02:09:51.109 INFO:tasks.workunit.client.0.smithi045.stdout:op 1423 completed, throughput=4.99MB/sec 2021-07-31T02:09:51.110 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj--Vm-jndREWtFz70 off=15643083 len=386201 2021-07-31T02:09:51.110 INFO:tasks.workunit.client.0.smithi045.stdout:op 1424 completed, throughput=4.99MB/sec 2021-07-31T02:09:51.111 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Skr4JNzQSjP2CH_ off=8671048 len=1974342 2021-07-31T02:09:51.120 INFO:tasks.workunit.client.0.smithi045.stdout:op 1425 completed, throughput=5MB/sec 2021-07-31T02:09:51.120 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-VQHiQcV4PKq8R2f off=5024870 len=1820331 2021-07-31T02:09:51.127 INFO:tasks.workunit.client.0.smithi045.stdout:op 1426 completed, throughput=5.01MB/sec 2021-07-31T02:09:52.127 INFO:tasks.workunit.client.0.smithi045.stdout: 280: throughput=4.99MB/sec pending data=0 2021-07-31T02:09:52.128 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-sv_ROd5pkuzo-LY off=13803985 len=1188891 2021-07-31T02:09:52.129 INFO:tasks.workunit.client.0.smithi045.stdout:op 1427 completed, throughput=4.99MB/sec 2021-07-31T02:09:52.129 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-lytt7eYBADYZO9r off=4934849 len=610360 2021-07-31T02:09:52.135 INFO:tasks.workunit.client.0.smithi045.stdout:op 1428 completed, throughput=4.99MB/sec 2021-07-31T02:09:52.136 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-JXev7rbEkqXZxeZ off=2284134 len=1946214 2021-07-31T02:09:52.144 INFO:tasks.workunit.client.0.smithi045.stdout:op 1429 completed, throughput=5MB/sec 2021-07-31T02:09:52.145 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-MFcK1LOazb5C8JF off=9926242 len=896747 2021-07-31T02:09:52.146 INFO:tasks.workunit.client.0.smithi045.stdout:op 1430 completed, throughput=5MB/sec 2021-07-31T02:09:53.147 INFO:tasks.workunit.client.0.smithi045.stdout: 281: throughput=4.99MB/sec pending data=0 2021-07-31T02:09:53.147 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Dnz7p5MKvGuZbCs off=2725118 len=1436117 2021-07-31T02:09:53.148 INFO:tasks.workunit.client.0.smithi045.stdout:op 1431 completed, throughput=4.99MB/sec 2021-07-31T02:09:53.148 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-QokjmixBOO-bK-c off=6950468 len=461776 2021-07-31T02:09:53.150 INFO:tasks.workunit.client.0.smithi045.stdout:op 1432 completed, throughput=4.99MB/sec 2021-07-31T02:09:53.150 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-q_jNkCbFF4cl_Ak off=13230111 len=490222 2021-07-31T02:09:53.152 INFO:tasks.workunit.client.0.smithi045.stdout:op 1433 completed, throughput=4.99MB/sec 2021-07-31T02:09:53.152 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-T8SlBY9Ztxo9MvA off=2119720 len=1642131 2021-07-31T02:09:53.160 INFO:tasks.workunit.client.0.smithi045.stdout:op 1434 completed, throughput=5MB/sec 2021-07-31T02:09:53.160 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-XRuN2wDoA2BQAnN off=862880 len=286031 2021-07-31T02:09:53.164 INFO:tasks.workunit.client.0.smithi045.stdout:op 1435 completed, throughput=5MB/sec 2021-07-31T02:09:53.164 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-n9hhpUBf_hr0HSS off=9289077 len=1990715 2021-07-31T02:09:53.168 INFO:tasks.workunit.client.0.smithi045.stdout:op 1436 completed, throughput=5.01MB/sec 2021-07-31T02:09:53.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:52 smithi154 conmon[34553]: cluster 2021-07-31T02:09:52.113884+0000 mgr.x (mgr.24542) 971 : cluster 2021-07-31T02:09:53.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:52 smithi154 conmon[34553]: [DBG] pgmap v1551: 9 pgs: 9 active+clean; 127 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 2.2 MiB/s rd, 1.8 MiB/s wr, 3 op/s 2021-07-31T02:09:53.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:52 smithi045 conmon[33655]: cluster 2021-07-31T02:09:52.113884+0000 mgr.x (mgr 2021-07-31T02:09:53.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:52 smithi045 conmon[33655]: .24542) 971 : cluster [DBG] pgmap v1551: 9 pgs: 9 active+clean; 127 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 2.2 MiB/s rd, 1.8 MiB/s wr, 3 op/s 2021-07-31T02:09:53.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:52 smithi045 conmon[39718]: cluster 2021-07-31T02:09:52.113884+0000 mgr.x (mgr.24542 2021-07-31T02:09:53.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:52 smithi045 conmon[39718]: ) 971 : cluster [DBG] pgmap v1551: 9 pgs: 9 active+clean; 127 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 2.2 MiB/s rd, 1.8 MiB/s wr, 3 op/s 2021-07-31T02:09:54.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:09:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:09:53.546Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:09:54.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:09:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:09:53.546Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:09:54.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:09:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:09:53.547Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:09:54.168 INFO:tasks.workunit.client.0.smithi045.stdout: 282: throughput=4.99MB/sec pending data=0 2021-07-31T02:09:54.169 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-8LWgvJxWYg3kJNE off=16093884 len=498850 2021-07-31T02:09:54.170 INFO:tasks.workunit.client.0.smithi045.stdout:op 1437 completed, throughput=4.99MB/sec 2021-07-31T02:09:54.171 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-iBcIdqpmQkxmime off=1454967 len=156287 2021-07-31T02:09:54.172 INFO:tasks.workunit.client.0.smithi045.stdout:op 1438 completed, throughput=4.99MB/sec 2021-07-31T02:09:54.172 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-pkkS3rX6qnHm84T off=17398358 len=1984988 2021-07-31T02:09:54.174 INFO:tasks.workunit.client.0.smithi045.stdout:op 1439 completed, throughput=5MB/sec 2021-07-31T02:09:54.175 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-kQbJ6evDEY0xI8g off=2598609 len=324464 2021-07-31T02:09:54.179 INFO:tasks.workunit.client.0.smithi045.stdout:op 1440 completed, throughput=5MB/sec 2021-07-31T02:09:54.179 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj--emCwoYVkHIgaGL off=5436174 len=73583 2021-07-31T02:09:54.181 INFO:tasks.workunit.client.0.smithi045.stdout:op 1441 completed, throughput=5MB/sec 2021-07-31T02:09:54.181 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-HTtjy87SnGvfa-8 off=782176 len=930142 2021-07-31T02:09:54.185 INFO:tasks.workunit.client.0.smithi045.stdout:op 1442 completed, throughput=5MB/sec 2021-07-31T02:09:54.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:54 smithi154 conmon[34553]: cluster 2021-07-31T02:09:54.114273+0000 mgr.x (mgr.24542) 972 : cluster [DBG] pgmap v1552: 9 pgs: 9 active+clean; 127 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.9 MiB/s wr, 5 op/s 2021-07-31T02:09:54.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:54 smithi045 conmon[33655]: cluster 2021-07-31T02:09:54.114273+0000 mgr.x (mgr.24542) 2021-07-31T02:09:54.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:54 smithi045 conmon[33655]: 972 : cluster [DBG] pgmap v1552: 9 pgs: 9 active+clean; 127 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.9 MiB/s wr, 5 op/s 2021-07-31T02:09:54.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:54 smithi045 conmon[39718]: cluster 2021-07-31T02:09:54.114273+0000 mgr.x (mgr.24542) 972 : 2021-07-31T02:09:54.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:54 smithi045 conmon[39718]: cluster [DBG] pgmap v1552: 9 pgs: 9 active+clean; 127 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.9 MiB/s wr, 5 op/s 2021-07-31T02:09:55.185 INFO:tasks.workunit.client.0.smithi045.stdout: 283: throughput=4.98MB/sec pending data=0 2021-07-31T02:09:55.186 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Vg1ccG5SHsccYeq off=20736615 len=353886 2021-07-31T02:09:55.187 INFO:tasks.workunit.client.0.smithi045.stdout:op 1443 completed, throughput=4.99MB/sec 2021-07-31T02:09:55.188 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-biF-oMy_uzySKfO off=7156394 len=311183 2021-07-31T02:09:55.191 INFO:tasks.workunit.client.0.smithi045.stdout:op 1444 completed, throughput=4.99MB/sec 2021-07-31T02:09:55.191 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-JwLSvZdCjWwsZw9 off=9745871 len=4828 2021-07-31T02:09:55.193 INFO:tasks.workunit.client.0.smithi045.stdout:op 1445 completed, throughput=4.99MB/sec 2021-07-31T02:09:55.193 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-0_5dCYy0sHUjb9z off=16831643 len=1711461 2021-07-31T02:09:55.201 INFO:tasks.workunit.client.0.smithi045.stdout:op 1446 completed, throughput=4.99MB/sec 2021-07-31T02:09:55.201 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ThlDkvxHWYjRTJF off=1172357 len=882110 2021-07-31T02:09:55.202 INFO:tasks.workunit.client.0.smithi045.stdout:op 1447 completed, throughput=4.99MB/sec 2021-07-31T02:09:55.202 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-WaudyVAfHxn4HuY off=844070 len=441522 2021-07-31T02:09:55.206 INFO:tasks.workunit.client.0.smithi045.stdout:op 1448 completed, throughput=5MB/sec 2021-07-31T02:09:55.206 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-B7XGElWzh0gs44p off=21590330 len=288122 2021-07-31T02:09:55.209 INFO:tasks.workunit.client.0.smithi045.stdout:op 1449 completed, throughput=5MB/sec 2021-07-31T02:09:55.209 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-sMQhbeqMUfbMaJ0 off=2482277 len=824390 2021-07-31T02:09:55.213 INFO:tasks.workunit.client.0.smithi045.stdout:op 1450 completed, throughput=5MB/sec 2021-07-31T02:09:55.213 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-9cV8QExIqOX4AOW off=5170331 len=75110 2021-07-31T02:09:55.214 INFO:tasks.workunit.client.0.smithi045.stdout:op 1451 completed, throughput=5MB/sec 2021-07-31T02:09:55.214 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-YQS5Vo9wlhEazWW off=147398 len=474187 2021-07-31T02:09:55.215 INFO:tasks.workunit.client.0.smithi045.stdout:op 1452 completed, throughput=5MB/sec 2021-07-31T02:09:56.215 INFO:tasks.workunit.client.0.smithi045.stdout: 284: throughput=4.98MB/sec pending data=0 2021-07-31T02:09:56.215 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-VE8r6MXw9AoPqdM off=1356598 len=271903 2021-07-31T02:09:56.217 INFO:tasks.workunit.client.0.smithi045.stdout:op 1453 completed, throughput=4.98MB/sec 2021-07-31T02:09:56.217 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-N3gKLh6FUqcvAPb off=14274675 len=1660031 2021-07-31T02:09:56.226 INFO:tasks.workunit.client.0.smithi045.stdout:op 1454 completed, throughput=4.99MB/sec 2021-07-31T02:09:56.226 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-gfBPpBBj0mzY9tn off=3049631 len=138439 2021-07-31T02:09:56.227 INFO:tasks.workunit.client.0.smithi045.stdout:op 1455 completed, throughput=4.99MB/sec 2021-07-31T02:09:56.227 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-FTy9mq7QMyI2eqk off=16007664 len=1012696 2021-07-31T02:09:56.229 INFO:tasks.workunit.client.0.smithi045.stdout:op 1456 completed, throughput=4.99MB/sec 2021-07-31T02:09:56.229 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-6ID9Ci9XhJPRcPm off=5379426 len=1281028 2021-07-31T02:09:56.236 INFO:tasks.workunit.client.0.smithi045.stdout:op 1457 completed, throughput=5MB/sec 2021-07-31T02:09:56.236 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-5D6_wmi6K6iU_Us off=9490813 len=928955 2021-07-31T02:09:56.241 INFO:tasks.workunit.client.0.smithi045.stdout:op 1458 completed, throughput=5MB/sec 2021-07-31T02:09:56.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:56 smithi154 conmon[34553]: cluster 2021-07-31T02:09:56.114676+0000 mgr.x (mgr.24542) 973 : cluster [DBG] pgmap v1553: 9 pgs: 9 active+clean; 127 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2021-07-31T02:09:56.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:56 smithi045 conmon[33655]: cluster 2021-07-31T02:09:56 2021-07-31T02:09:56.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:56 smithi045 conmon[33655]: .114676+0000 mgr.x (mgr.24542) 973 : cluster [DBG] pgmap v1553: 9 pgs: 9 active+clean; 127 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2021-07-31T02:09:56.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:56 smithi045 conmon[39718]: cluster 2021-07-31T02:09:56.114676+0000 mgr.x (mgr.24542 2021-07-31T02:09:56.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:56 smithi045 conmon[39718]: ) 973 : cluster [DBG] pgmap v1553: 9 pgs: 9 active+clean; 127 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2021-07-31T02:09:57.241 INFO:tasks.workunit.client.0.smithi045.stdout: 285: throughput=4.98MB/sec pending data=0 2021-07-31T02:09:57.242 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-1RBON71utCF-ZiF off=15921445 len=521180 2021-07-31T02:09:57.243 INFO:tasks.workunit.client.0.smithi045.stdout:op 1459 completed, throughput=4.99MB/sec 2021-07-31T02:09:57.243 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-7gbNGt0Zrrsb_LM off=3747820 len=1634937 2021-07-31T02:09:57.250 INFO:tasks.workunit.client.0.smithi045.stdout:op 1460 completed, throughput=4.99MB/sec 2021-07-31T02:09:57.251 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-yijLJmXf_P0-iOE off=397944 len=487293 2021-07-31T02:09:57.254 INFO:tasks.workunit.client.0.smithi045.stdout:op 1461 completed, throughput=4.99MB/sec 2021-07-31T02:09:57.255 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-yuUapVIVAAIDPJL off=10775189 len=246945 2021-07-31T02:09:57.255 INFO:tasks.workunit.client.0.smithi045.stdout:op 1462 completed, throughput=4.99MB/sec 2021-07-31T02:09:57.256 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-BrtTj2NEQsPqmcQ off=2247392 len=100110 2021-07-31T02:09:57.256 INFO:tasks.workunit.client.0.smithi045.stdout:op 1463 completed, throughput=4.99MB/sec 2021-07-31T02:09:57.257 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-x6pS7f6e9Qmgkas off=8529389 len=1674985 2021-07-31T02:09:57.259 INFO:tasks.workunit.client.0.smithi045.stdout:op 1464 completed, throughput=5MB/sec 2021-07-31T02:09:57.259 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Yl9TupV_MPdOetL off=141660 len=100724 2021-07-31T02:09:57.260 INFO:tasks.workunit.client.0.smithi045.stdout:op 1465 completed, throughput=5MB/sec 2021-07-31T02:09:57.260 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-CeAPlVeMrBRBml9 off=3754274 len=1301142 2021-07-31T02:09:57.261 INFO:tasks.workunit.client.0.smithi045.stdout:op 1466 completed, throughput=5MB/sec 2021-07-31T02:09:58.261 INFO:tasks.workunit.client.0.smithi045.stdout: 286: throughput=4.99MB/sec pending data=0 2021-07-31T02:09:58.262 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-h07qxxQHPMypG5v off=2754649 len=624291 2021-07-31T02:09:58.263 INFO:tasks.workunit.client.0.smithi045.stdout:op 1467 completed, throughput=4.99MB/sec 2021-07-31T02:09:58.264 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-3d5AeRQxIHX_I4n off=12591482 len=47887 2021-07-31T02:09:58.264 INFO:tasks.workunit.client.0.smithi045.stdout:op 1468 completed, throughput=4.99MB/sec 2021-07-31T02:09:58.264 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-kE2pkLntcgYswq0 off=22699301 len=1526680 2021-07-31T02:09:58.272 INFO:tasks.workunit.client.0.smithi045.stdout:op 1469 completed, throughput=4.99MB/sec 2021-07-31T02:09:58.272 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-SMlsaKO2D5f9IiZ off=1067995 len=140789 2021-07-31T02:09:58.273 INFO:tasks.workunit.client.0.smithi045.stdout:op 1470 completed, throughput=4.99MB/sec 2021-07-31T02:09:58.273 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-vyQ-kqHeSorTEcu off=9613894 len=1707808 2021-07-31T02:09:58.280 INFO:tasks.workunit.client.0.smithi045.stdout:op 1471 completed, throughput=5MB/sec 2021-07-31T02:09:58.280 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-gK34GSS2Jp2Vg9t off=19255311 len=1071487 2021-07-31T02:09:58.282 INFO:tasks.workunit.client.0.smithi045.stdout:op 1472 completed, throughput=5MB/sec 2021-07-31T02:09:58.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:58 smithi154 conmon[34553]: cluster 2021-07-31T02:09:58.115176+0000 mgr.x (mgr.24542) 974 : cluster [DBG] pgmap v1554: 9 pgs: 9 active+clean; 127 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2021-07-31T02:09:58.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:09:58 smithi154 conmon[34553]: 2021-07-31T02:09:58.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:58 smithi045 conmon[33655]: cluster 2021-07-31T02:09:58.115176+0000 mgr.x (mgr.24542) 974 : cluster [DBG] 2021-07-31T02:09:58.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:09:58 smithi045 conmon[33655]: pgmap v1554: 9 pgs: 9 active+clean; 127 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2021-07-31T02:09:58.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:58 smithi045 conmon[39718]: cluster 2021-07-31T02:09:58.115176+0000 mgr.x (mgr.24542) 2021-07-31T02:09:58.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:09:58 smithi045 conmon[39718]: 974 : cluster [DBG] pgmap v1554: 9 pgs: 9 active+clean; 127 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2021-07-31T02:09:59.282 INFO:tasks.workunit.client.0.smithi045.stdout: 287: throughput=4.99MB/sec pending data=0 2021-07-31T02:09:59.283 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-hdk9VYEZ2im_ShY off=9093126 len=469866 2021-07-31T02:09:59.287 INFO:tasks.workunit.client.0.smithi045.stdout:op 1473 completed, throughput=4.99MB/sec 2021-07-31T02:09:59.288 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-QjO9cE60oukb8Q5 off=14234461 len=1270354 2021-07-31T02:09:59.292 INFO:tasks.workunit.client.0.smithi045.stdout:op 1474 completed, throughput=4.99MB/sec 2021-07-31T02:09:59.293 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-2-Hj5ZgydgR2J7N off=852824 len=558417 2021-07-31T02:09:59.294 INFO:tasks.workunit.client.0.smithi045.stdout:op 1475 completed, throughput=4.99MB/sec 2021-07-31T02:09:59.294 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Mj7rqVEvQ-gNFgt off=5446815 len=830079 2021-07-31T02:09:59.295 INFO:tasks.workunit.client.0.smithi045.stdout:op 1476 completed, throughput=5MB/sec 2021-07-31T02:09:59.295 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-9rFMzN1YFJy19lw off=539520 len=1157356 2021-07-31T02:09:59.296 INFO:tasks.workunit.client.0.smithi045.stdout:op 1477 completed, throughput=5MB/sec 2021-07-31T02:09:59.297 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-NYToXIW7MR0ECDD off=1645040 len=1934651 2021-07-31T02:09:59.303 INFO:tasks.workunit.client.0.smithi045.stdout:op 1478 completed, throughput=5.01MB/sec 2021-07-31T02:10:00.303 INFO:tasks.workunit.client.0.smithi045.stdout: 288: throughput=4.99MB/sec pending data=0 2021-07-31T02:10:00.303 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-RGHGO0YQBO4ocuV off=9018294 len=1071824 2021-07-31T02:10:00.304 INFO:tasks.workunit.client.0.smithi045.stdout:op 1479 completed, throughput=4.99MB/sec 2021-07-31T02:10:00.304 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-aZwtmoEoctan8Tk off=1731590 len=863432 2021-07-31T02:10:00.310 INFO:tasks.workunit.client.0.smithi045.stdout:op 1480 completed, throughput=4.99MB/sec 2021-07-31T02:10:00.310 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-YnzVUxSL2pALOrt off=12857611 len=2029004 2021-07-31T02:10:00.320 INFO:tasks.workunit.client.0.smithi045.stdout:op 1481 completed, throughput=5MB/sec 2021-07-31T02:10:00.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:00 smithi154 conmon[34553]: cluster 2021-07-31T02:10:00.000159+0000 mon.a (mon.0) 1864 : cluster [INF] 2021-07-31T02:10:00.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:00 smithi154 conmon[34553]: overall HEALTH_OK 2021-07-31T02:10:00.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:00 smithi045 conmon[39718]: cluster 2021-07-31T02:10:00.000159+0000 mon.a (mon.0) 1864 : cluster 2021-07-31T02:10:00.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:00 smithi045 conmon[39718]: [INF] overall HEALTH_OK 2021-07-31T02:10:00.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:00 smithi045 conmon[33655]: cluster 2021-07-31T02:10:00.000159+0000 mon.a (mon.0) 1864 : cluster [INF] overall HEALTH_OK 2021-07-31T02:10:01.320 INFO:tasks.workunit.client.0.smithi045.stdout: 289: throughput=4.98MB/sec pending data=0 2021-07-31T02:10:01.321 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-WuqHNrvuRCysxFx off=2987774 len=301159 2021-07-31T02:10:01.321 INFO:tasks.workunit.client.0.smithi045.stdout:op 1482 completed, throughput=4.98MB/sec 2021-07-31T02:10:01.321 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-JkjuPnTXLW531kV off=11375178 len=1327227 2021-07-31T02:10:01.324 INFO:tasks.workunit.client.0.smithi045.stdout:op 1483 completed, throughput=4.99MB/sec 2021-07-31T02:10:01.324 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-XC2tqTuZY-khSxD off=15991563 len=2086183 2021-07-31T02:10:01.328 INFO:tasks.workunit.client.0.smithi045.stdout:op 1484 completed, throughput=5MB/sec 2021-07-31T02:10:01.328 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-LpK9fR8U8IzQCMC off=702925 len=1347150 2021-07-31T02:10:01.331 INFO:tasks.workunit.client.0.smithi045.stdout:op 1485 completed, throughput=5MB/sec 2021-07-31T02:10:01.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:01 smithi154 conmon[34553]: cluster 2021-07-31T02:10:00.115610+0000 mgr.x (mgr.24542) 975 : cluster [DBG] pgmap v1555: 9 pgs: 9 active+clean; 127 GiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 3.1 MiB/s rd, 3.4 MiB/s wr, 7 op/s 2021-07-31T02:10:01.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:01 smithi045 conmon[39718]: cluster 2021-07-31T02:10:00.115610+0000 mgr.x ( 2021-07-31T02:10:01.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:01 smithi045 conmon[39718]: mgr.24542) 975 : cluster [DBG] pgmap v1555: 9 pgs: 9 active+clean; 127 GiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 3.1 MiB/s rd, 3.4 MiB/s wr, 7 op/s 2021-07-31T02:10:01.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:01 smithi045 conmon[33655]: cluster 2021-07-31T02:10:00.115610+0000 mgr.x (mgr.24542) 975 : cluster [DBG] pgmap v1555: 9 pgs: 9 active+clean; 127 GiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 3.1 MiB/s rd, 3.4 MiB/s wr, 7 op/s 2021-07-31T02:10:02.332 INFO:tasks.workunit.client.0.smithi045.stdout: 290: throughput=4.98MB/sec pending data=0 2021-07-31T02:10:02.332 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ueylAduZC5_gUtw off=0 len=1557720 2021-07-31T02:10:02.339 INFO:tasks.workunit.client.0.smithi045.stdout:op 1486 completed, throughput=4.99MB/sec 2021-07-31T02:10:02.340 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-qSCdkLfE451GMu2 off=8214836 len=1090202 2021-07-31T02:10:02.342 INFO:tasks.workunit.client.0.smithi045.stdout:op 1487 completed, throughput=4.99MB/sec 2021-07-31T02:10:02.342 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-U0ShOLlBPb4pMwh off=12365104 len=1097572 2021-07-31T02:10:02.347 INFO:tasks.workunit.client.0.smithi045.stdout:op 1488 completed, throughput=4.99MB/sec 2021-07-31T02:10:02.347 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-N2Tdm1vN3QG8pbB off=1354865 len=966958 2021-07-31T02:10:02.352 INFO:tasks.workunit.client.0.smithi045.stdout:op 1489 completed, throughput=5MB/sec 2021-07-31T02:10:02.353 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj--rLEIM5Sj1v4OoJ off=10094674 len=1424809 2021-07-31T02:10:02.355 INFO:tasks.workunit.client.0.smithi045.stdout:op 1490 completed, throughput=5MB/sec 2021-07-31T02:10:03.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:02 smithi154 conmon[34553]: cluster 2021-07-31T02:10:02.116051+0000 mgr.x (mgr.24542) 976 : cluster [DBG] pgmap v1556: 9 pgs: 9 active+clean; 127 GiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 5 op/s 2021-07-31T02:10:03.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:02 smithi154 conmon[34553]: 2021-07-31T02:10:03.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:02 smithi045 conmon[39718]: cluster 2021-07-31T02:10:02.116051+0000 2021-07-31T02:10:03.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:02 smithi045 conmon[39718]: mgr.x (mgr.24542) 976 : cluster [DBG] pgmap v1556: 9 pgs: 9 active+clean; 127 GiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 5 op/s 2021-07-31T02:10:03.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:02 smithi045 conmon[33655]: cluster 2021-07-31T02:10:02.116051+0000 2021-07-31T02:10:03.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:02 smithi045 conmon[33655]: mgr.x (mgr.24542) 976 : cluster [DBG] pgmap v1556: 9 pgs: 9 active+clean; 127 GiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 5 op/s 2021-07-31T02:10:03.356 INFO:tasks.workunit.client.0.smithi045.stdout: 291: throughput=4.99MB/sec pending data=0 2021-07-31T02:10:03.356 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-evx4ZVta2-eng65 off=6852417 len=1456238 2021-07-31T02:10:03.358 INFO:tasks.workunit.client.0.smithi045.stdout:op 1491 completed, throughput=4.99MB/sec 2021-07-31T02:10:03.359 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-xy1-obOxvZ7KBQ9 off=6258705 len=2030988 2021-07-31T02:10:03.367 INFO:tasks.workunit.client.0.smithi045.stdout:op 1492 completed, throughput=5MB/sec 2021-07-31T02:10:03.368 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-hjgZj8ntAmp9Num off=5100247 len=1098891 2021-07-31T02:10:03.369 INFO:tasks.workunit.client.0.smithi045.stdout:op 1493 completed, throughput=5MB/sec 2021-07-31T02:10:04.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:10:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:10:03.547Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:10:04.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:10:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:10:03.547Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:10:04.027 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:10:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:10:03.547Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:10:04.370 INFO:tasks.workunit.client.0.smithi045.stdout: 292: throughput=4.98MB/sec pending data=0 2021-07-31T02:10:04.370 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-XfeQVsNGtzOW5NP off=2192120 len=1463963 2021-07-31T02:10:04.378 INFO:tasks.workunit.client.0.smithi045.stdout:op 1494 completed, throughput=4.99MB/sec 2021-07-31T02:10:04.378 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-0czY1nkpghxy4PG off=6722170 len=836454 2021-07-31T02:10:04.380 INFO:tasks.workunit.client.0.smithi045.stdout:op 1495 completed, throughput=4.99MB/sec 2021-07-31T02:10:04.380 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-gFjRVRJmKUC26qj off=4523749 len=1302619 2021-07-31T02:10:04.383 INFO:tasks.workunit.client.0.smithi045.stdout:op 1496 completed, throughput=4.99MB/sec 2021-07-31T02:10:04.383 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-4GRmocuwQHZ3nE_ off=1701966 len=788509 2021-07-31T02:10:04.388 INFO:tasks.workunit.client.0.smithi045.stdout:op 1497 completed, throughput=5MB/sec 2021-07-31T02:10:04.388 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-RNRTwPLnVMxTBAH off=19012032 len=285874 2021-07-31T02:10:04.389 INFO:tasks.workunit.client.0.smithi045.stdout:op 1498 completed, throughput=5MB/sec 2021-07-31T02:10:04.389 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-gozz13JipLLaSO1 off=871842 len=915907 2021-07-31T02:10:04.394 INFO:tasks.workunit.client.0.smithi045.stdout:op 1499 completed, throughput=5MB/sec 2021-07-31T02:10:04.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:04 smithi154 conmon[34553]: cluster 2021-07-31T02:10:04.116454+0000 mgr.x (mgr.24542) 977 : cluster [DBG] pgmap v1557: 9 pgs: 9 active+clean; 127 GiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 3.0 MiB/s rd, 3.3 MiB/s wr, 6 op/s 2021-07-31T02:10:04.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:04 smithi154 conmon[34553]: 2021-07-31T02:10:04.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:04 smithi045 conmon[33655]: cluster 2021-07-31T02:10:04.116454+0000 mgr.x ( 2021-07-31T02:10:04.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:04 smithi045 conmon[33655]: mgr.24542) 977 : cluster [DBG] pgmap v1557: 9 pgs: 9 active+clean; 127 GiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 3.0 MiB/s rd, 3.3 MiB/s wr, 6 op/s 2021-07-31T02:10:04.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:04 smithi045 conmon[39718]: cluster 2021-07-31T02:10:04.116454+0000 mgr.x (mgr.24542) 977 : cluster [DBG] pgmap v1557: 9 pgs: 9 active+clean; 127 GiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 3.0 MiB/s rd, 3.3 MiB/s wr, 6 op/s 2021-07-31T02:10:05.394 INFO:tasks.workunit.client.0.smithi045.stdout: 293: throughput=4.98MB/sec pending data=0 2021-07-31T02:10:05.395 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-6LMiBX-LU0t2Kx0 off=14942382 len=1233664 2021-07-31T02:10:05.400 INFO:tasks.workunit.client.0.smithi045.stdout:op 1500 completed, throughput=4.99MB/sec 2021-07-31T02:10:05.400 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-CnqjKtG15bzVGn3 off=8677623 len=1383527 2021-07-31T02:10:05.403 INFO:tasks.workunit.client.0.smithi045.stdout:op 1501 completed, throughput=4.99MB/sec 2021-07-31T02:10:05.403 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-AddZykeo1IDrdNi off=6559643 len=595665 2021-07-31T02:10:05.410 INFO:tasks.workunit.client.0.smithi045.stdout:op 1502 completed, throughput=4.99MB/sec 2021-07-31T02:10:05.410 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-NMEZMWrhPdjAAWF off=1178884 len=970831 2021-07-31T02:10:05.415 INFO:tasks.workunit.client.0.smithi045.stdout:op 1503 completed, throughput=5MB/sec 2021-07-31T02:10:05.415 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-mtD20Ustw6sxJWQ off=515859 len=426396 2021-07-31T02:10:05.416 INFO:tasks.workunit.client.0.smithi045.stdout:op 1504 completed, throughput=5MB/sec 2021-07-31T02:10:05.416 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-fZVdS3tY_CPFBaC off=3618637 len=258598 2021-07-31T02:10:05.417 INFO:tasks.workunit.client.0.smithi045.stdout:op 1505 completed, throughput=5MB/sec 2021-07-31T02:10:05.417 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-J-ktnLpCivqvueR off=2407805 len=1441771 2021-07-31T02:10:05.425 INFO:tasks.workunit.client.0.smithi045.stdout:op 1506 completed, throughput=5MB/sec 2021-07-31T02:10:06.426 INFO:tasks.workunit.client.0.smithi045.stdout: 294: throughput=4.99MB/sec pending data=0 2021-07-31T02:10:06.427 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-IvDL5FUtu_4wqEs off=1468342 len=1169629 2021-07-31T02:10:06.433 INFO:tasks.workunit.client.0.smithi045.stdout:op 1507 completed, throughput=4.99MB/sec 2021-07-31T02:10:06.433 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-aHosV8cKSlk91Hq off=27110 len=895314 2021-07-31T02:10:06.438 INFO:tasks.workunit.client.0.smithi045.stdout:op 1508 completed, throughput=4.99MB/sec 2021-07-31T02:10:06.439 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-062xNccAVX2AuzS off=6178409 len=1730873 2021-07-31T02:10:06.446 INFO:tasks.workunit.client.0.smithi045.stdout:op 1509 completed, throughput=5MB/sec 2021-07-31T02:10:06.447 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-C7vzAY--GpRKWF2 off=4779226 len=279380 2021-07-31T02:10:06.450 INFO:tasks.workunit.client.0.smithi045.stdout:op 1510 completed, throughput=5MB/sec 2021-07-31T02:10:06.450 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-v9PKmdNJgr2HB6K off=2178260 len=202435 2021-07-31T02:10:06.451 INFO:tasks.workunit.client.0.smithi045.stdout:op 1511 completed, throughput=5MB/sec 2021-07-31T02:10:06.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:06 smithi154 conmon[34553]: cluster 2021-07-31T02:10:06.117021+0000 mgr.x (mgr.24542) 978 : cluster [DBG] pgmap v1558: 9 pgs: 9 active+clean; 127 GiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2021-07-31T02:10:06.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:06 smithi045 conmon[39718]: cluster 2021-07-31T02:10:06.117021+0000 2021-07-31T02:10:06.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:06 smithi045 conmon[39718]: mgr.x (mgr.24542) 978 : cluster [DBG] pgmap v1558: 9 pgs: 9 active+clean; 127 GiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2021-07-31T02:10:06.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:06 smithi045 conmon[33655]: cluster 2021-07-31T02:10:06.117021+0000 mgr.x (mgr.24542) 978 : cluster [DBG] pgmap v1558: 9 pgs: 9 active+clean; 127 GiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2021-07-31T02:10:07.293 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:10:07.450 INFO:tasks.workunit.client.0.smithi045.stdout: 295: throughput=4.98MB/sec pending data=0 2021-07-31T02:10:07.451 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-UolCfIm-DiexUQX off=2465790 len=224047 2021-07-31T02:10:07.453 INFO:tasks.workunit.client.0.smithi045.stdout:op 1512 completed, throughput=4.98MB/sec 2021-07-31T02:10:07.453 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-PVK7g2Fb71QBsva off=9871744 len=1993709 2021-07-31T02:10:07.455 INFO:tasks.workunit.client.0.smithi045.stdout:op 1513 completed, throughput=4.99MB/sec 2021-07-31T02:10:07.455 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-TxeXMAV6sK_5Z30 off=4606624 len=1388583 2021-07-31T02:10:07.461 INFO:tasks.workunit.client.0.smithi045.stdout:op 1514 completed, throughput=4.99MB/sec 2021-07-31T02:10:07.462 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-hEOAXqmA7DeX64h off=4721519 len=1729025 2021-07-31T02:10:07.468 INFO:tasks.workunit.client.0.smithi045.stdout:op 1515 completed, throughput=5MB/sec 2021-07-31T02:10:07.572 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:10:07.572 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 25m ago 25m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:10:07.573 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 22m ago 25m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:10:07.573 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 22m ago 29m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:10:07.574 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 25m ago 31m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:10:07.574 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 25m ago 31m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:10:07.575 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 22m ago 29m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:10:07.575 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 25m ago 29m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:10:07.576 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 25m ago 25m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:10:07.576 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 22m ago 25m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:10:07.576 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 25m ago 28m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:10:07.577 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 25m ago 28m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:10:07.577 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 25m ago 28m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:10:07.578 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 25m ago 27m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:10:07.578 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 22m ago 27m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:10:07.579 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 22m ago 27m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:10:07.579 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 22m ago 26m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:10:07.580 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 22m ago 26m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:10:07.580 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 22m ago 26m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:10:07.906 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:10:07.906 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:10:07.907 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:10:07.907 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:10:07.907 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:10:07.907 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:10:07.908 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:10:07.908 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:10:07.908 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:10:07.908 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:10:07.909 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:10:07.909 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:10:07.909 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:10:07.909 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:10:07.909 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:10:07.910 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:10:07.910 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:10:08.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:07 smithi154 conmon[34553]: audit 2021-07-31T02:10:07.906213+0000 mon.c (mon.1) 80 : audit [DBG] from='client.? 172.21.15.45:0/3442888763' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:10:08.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:07 smithi045 conmon[33655]: audit 2021-07-31T02:10:07.906213+0000 mon.c ( 2021-07-31T02:10:08.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:07 smithi045 conmon[33655]: mon.1) 80 : audit [DBG] from='client.? 172.21.15.45:0/3442888763' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:10:08.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:07 smithi045 conmon[39718]: audit 2021-07-31T02:10:07.906213+0000 2021-07-31T02:10:08.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:07 smithi045 conmon[39718]: mon.c (mon.1) 80 : audit [DBG] from='client.? 172.21.15.45:0/3442888763' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:10:08.468 INFO:tasks.workunit.client.0.smithi045.stdout: 296: throughput=4.98MB/sec pending data=0 2021-07-31T02:10:08.469 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-tFGskTwObBVosse off=15074691 len=1404365 2021-07-31T02:10:08.471 INFO:tasks.workunit.client.0.smithi045.stdout:op 1516 completed, throughput=4.99MB/sec 2021-07-31T02:10:08.471 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-hVQhRcVednEuDH1 off=18250160 len=1157321 2021-07-31T02:10:08.474 INFO:tasks.workunit.client.0.smithi045.stdout:op 1517 completed, throughput=4.99MB/sec 2021-07-31T02:10:08.474 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-btkyn5pIM6Kemk- off=10147432 len=928341 2021-07-31T02:10:08.475 INFO:tasks.workunit.client.0.smithi045.stdout:op 1518 completed, throughput=4.99MB/sec 2021-07-31T02:10:08.475 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-CuGgidyxwv7cQeo off=8421780 len=66607 2021-07-31T02:10:08.478 INFO:tasks.workunit.client.0.smithi045.stdout:op 1519 completed, throughput=4.99MB/sec 2021-07-31T02:10:08.478 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-_nj5XXAWk73HENF off=3970003 len=1944370 2021-07-31T02:10:08.481 INFO:tasks.workunit.client.0.smithi045.stdout:op 1520 completed, throughput=5MB/sec 2021-07-31T02:10:09.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:08 smithi154 conmon[34553]: audit 2021-07-31T02:10:07.280565+0000 mgr.x (mgr.24542) 979 : audit [DBG] from='client.25804 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:10:09.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:08 smithi154 conmon[34553]: audit 2021-07-31T02:10:07.563625+0000 mgr.x (mgr.24542) 980 : audit [DBG] from='client.25810 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:10:09.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:08 smithi154 conmon[34553]: cluster 2021-07-31T02:10:08.117582+0000 mgr.x (mgr 2021-07-31T02:10:09.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:08 smithi154 conmon[34553]: .24542) 981 : cluster [DBG] pgmap v1559: 9 pgs: 9 active+clean; 127 GiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2021-07-31T02:10:09.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:08 smithi045 conmon[33655]: audit 2021-07-31T02:10 2021-07-31T02:10:09.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:08 smithi045 conmon[33655]: :07.280565+0000 mgr.x (mgr.24542) 979 : audit [DBG] from='client.25804 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:10:09.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:08 smithi045 conmon[33655]: audit 2021 2021-07-31T02:10:09.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:08 smithi045 conmon[33655]: -07-31T02:10 2021-07-31T02:10:09.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:08 smithi045 conmon[33655]: :07.563625+0000 2021-07-31T02:10:09.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:08 smithi045 conmon[33655]: mgr.x (mgr.24542) 980 : audit [DBG] 2021-07-31T02:10:09.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:08 smithi045 conmon[33655]: from='client.25810 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:10:09.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:08 smithi045 conmon[33655]: cluster 2021-07-31T02:10:09.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:08 smithi045 conmon[33655]: 2021-07-31T02:10 2021-07-31T02:10:09.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:08 smithi045 conmon[33655]: :08.117582+0000 mgr.x ( 2021-07-31T02:10:09.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:08 smithi045 conmon[33655]: mgr.24542) 981 : cluster [DBG] 2021-07-31T02:10:09.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:08 smithi045 conmon[33655]: pgmap v1559: 9 pgs: 9 active+clean; 127 GiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2021-07-31T02:10:09.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:08 smithi045 conmon[39718]: audit 2021-07-31T02:10:07.280565+0000 mgr.x (mgr.24542 2021-07-31T02:10:09.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:08 smithi045 conmon[39718]: ) 979 : audit [DBG] from='client.25804 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:10:09.282 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:08 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T02:10:09.282 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:08 smithi045 conmon[39718]: 31T02:10:07.563625+0000 mgr.x (mgr.24542) 980 : 2021-07-31T02:10:09.282 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:08 smithi045 conmon[39718]: audit [DBG] from='client.25810 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:10:09.282 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:08 smithi045 conmon[39718]: cluster 2021-07-31 2021-07-31T02:10:09.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:08 smithi045 conmon[39718]: T02:10:08.117582+0000 mgr.x ( 2021-07-31T02:10:09.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:08 smithi045 conmon[39718]: mgr.24542) 981 : cluster [DBG] pgmap v1559: 9 pgs: 9 active+clean; 127 GiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2021-07-31T02:10:09.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:08 smithi045 conmon[39718]: 2021-07-31T02:10:09.482 INFO:tasks.workunit.client.0.smithi045.stdout: 297: throughput=4.98MB/sec pending data=0 2021-07-31T02:10:09.482 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-eFl7nrKurBeYPR0 off=4539305 len=1117257 2021-07-31T02:10:09.489 INFO:tasks.workunit.client.0.smithi045.stdout:op 1521 completed, throughput=4.99MB/sec 2021-07-31T02:10:09.489 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-rJR1O4xMog_avnD off=9694710 len=1176400 2021-07-31T02:10:09.491 INFO:tasks.workunit.client.0.smithi045.stdout:op 1522 completed, throughput=4.99MB/sec 2021-07-31T02:10:09.491 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-J4AP39jNBdeyz8n off=8846034 len=231546 2021-07-31T02:10:09.494 INFO:tasks.workunit.client.0.smithi045.stdout:op 1523 completed, throughput=4.99MB/sec 2021-07-31T02:10:09.495 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-S25vXJB67FGJnXM off=11175015 len=1916838 2021-07-31T02:10:09.497 INFO:tasks.workunit.client.0.smithi045.stdout:op 1524 completed, throughput=5MB/sec 2021-07-31T02:10:09.497 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-AuTjHoyiqQd727W off=15001590 len=692252 2021-07-31T02:10:09.501 INFO:tasks.workunit.client.0.smithi045.stdout:op 1525 completed, throughput=5MB/sec 2021-07-31T02:10:10.502 INFO:tasks.workunit.client.0.smithi045.stdout: 298: throughput=4.98MB/sec pending data=0 2021-07-31T02:10:10.503 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-rp0tkZihfWv5Mf_ off=2024368 len=51570 2021-07-31T02:10:10.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:10 smithi154 conmon[34553]: cluster 2021-07-31T02:10:10.118226+0000 mgr.x (mgr.24542) 982 : cluster [DBG] pgmap v1560: 9 pgs: 9 active+clean; 127 GiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 3.0 MiB/s rd, 3.3 MiB/s wr, 6 op/s 2021-07-31T02:10:10.505 INFO:tasks.workunit.client.0.smithi045.stdout:op 1526 completed, throughput=4.98MB/sec 2021-07-31T02:10:10.506 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-8BBLICB_A4WuR__ off=22665603 len=1733313 2021-07-31T02:10:10.510 INFO:tasks.workunit.client.0.smithi045.stdout:op 1527 completed, throughput=4.99MB/sec 2021-07-31T02:10:10.511 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-r477bAluKhKSQu1 off=1094763 len=630311 2021-07-31T02:10:10.516 INFO:tasks.workunit.client.0.smithi045.stdout:op 1528 completed, throughput=4.99MB/sec 2021-07-31T02:10:10.516 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-DaULm6W74aTTZjz off=10014490 len=14340 2021-07-31T02:10:10.516 INFO:tasks.workunit.client.0.smithi045.stdout:op 1529 completed, throughput=4.99MB/sec 2021-07-31T02:10:10.516 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-zZS4m6V5j4dqIjW off=11687790 len=1587308 2021-07-31T02:10:10.523 INFO:tasks.workunit.client.0.smithi045.stdout:op 1530 completed, throughput=5MB/sec 2021-07-31T02:10:10.523 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-LXI7ycrIlZKZ9qx off=4356235 len=1602981 2021-07-31T02:10:10.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:10 smithi045 conmon[39718]: cluster 2021-07-31T02:10:10.118226+0000 mgr.x (mgr.24542) 982 : 2021-07-31T02:10:10.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:10 smithi045 conmon[39718]: cluster [DBG] pgmap v1560: 9 pgs: 9 active+clean; 127 GiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 3.0 MiB/s rd, 3.3 MiB/s wr, 6 op/s 2021-07-31T02:10:10.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:10 smithi045 conmon[33655]: cluster 2021-07-31T02:10:10.118226+0000 mgr.x (mgr.24542) 982 : cluster 2021-07-31T02:10:10.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:10 smithi045 conmon[33655]: [DBG] pgmap v1560: 9 pgs: 9 active+clean; 127 GiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 3.0 MiB/s rd, 3.3 MiB/s wr, 6 op/s 2021-07-31T02:10:10.531 INFO:tasks.workunit.client.0.smithi045.stdout:op 1531 completed, throughput=5MB/sec 2021-07-31T02:10:11.531 INFO:tasks.workunit.client.0.smithi045.stdout: 299: throughput=4.98MB/sec pending data=0 2021-07-31T02:10:11.531 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-UiQW6BGblFhK_tH off=12215009 len=476748 2021-07-31T02:10:11.532 INFO:tasks.workunit.client.0.smithi045.stdout:op 1532 completed, throughput=4.99MB/sec 2021-07-31T02:10:11.532 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-_-n47baslBapu2h off=1146957 len=1327428 2021-07-31T02:10:11.539 INFO:tasks.workunit.client.0.smithi045.stdout:op 1533 completed, throughput=4.99MB/sec 2021-07-31T02:10:11.539 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-YQ4EdotCNIPTBak off=9316462 len=1411198 2021-07-31T02:10:11.547 INFO:tasks.workunit.client.0.smithi045.stdout:op 1534 completed, throughput=4.99MB/sec 2021-07-31T02:10:11.547 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-fBCunYw1I9ZdN-T off=5093219 len=564594 2021-07-31T02:10:11.551 INFO:tasks.workunit.client.0.smithi045.stdout:op 1535 completed, throughput=5MB/sec 2021-07-31T02:10:11.551 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-yaccHlHpk2JJNp3 off=2151506 len=1356921 2021-07-31T02:10:11.552 INFO:tasks.workunit.client.0.smithi045.stdout:op 1536 completed, throughput=5MB/sec 2021-07-31T02:10:12.553 INFO:tasks.workunit.client.0.smithi045.stdout: 300: throughput=4.98MB/sec pending data=0 2021-07-31T02:10:12.553 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-cHFntdH-XECQg0N off=2204641 len=78906 2021-07-31T02:10:12.556 INFO:tasks.workunit.client.0.smithi045.stdout:op 1537 completed, throughput=4.98MB/sec 2021-07-31T02:10:12.556 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-6gJFq9FX7sbmDIY off=5581196 len=1353314 2021-07-31T02:10:12.557 INFO:tasks.workunit.client.0.smithi045.stdout:op 1538 completed, throughput=4.99MB/sec 2021-07-31T02:10:12.558 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-0xAgNlp5A3AJQbF off=4758173 len=976238 2021-07-31T02:10:12.559 INFO:tasks.workunit.client.0.smithi045.stdout:op 1539 completed, throughput=4.99MB/sec 2021-07-31T02:10:12.559 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-y3UC2QEfjo7-V9H off=4523745 len=2027328 2021-07-31T02:10:12.563 INFO:tasks.workunit.client.0.smithi045.stdout:op 1540 completed, throughput=5MB/sec 2021-07-31T02:10:12.564 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj--N_yhUFa_iTFlf6 off=4458099 len=669146 2021-07-31T02:10:12.568 INFO:tasks.workunit.client.0.smithi045.stdout:op 1541 completed, throughput=5MB/sec 2021-07-31T02:10:12.568 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj--ZJwgMri5a7P9xj off=15185044 len=1841581 2021-07-31T02:10:12.579 INFO:tasks.workunit.client.0.smithi045.stdout:op 1542 completed, throughput=5.01MB/sec 2021-07-31T02:10:13.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:12 smithi154 conmon[34553]: cluster 2021-07-31T02:10:12.118637+0000 mgr.x (mgr.24542) 983 : cluster [DBG] pgmap v1561: 9 pgs: 9 active+clean; 127 GiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.2 MiB/s wr, 3 op/s 2021-07-31T02:10:13.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:12 smithi045 conmon[33655]: cluster 2021-07-31T02:10:12.118637+0000 mgr.x (mgr.24542) 983 2021-07-31T02:10:13.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:12 smithi045 conmon[33655]: : cluster [DBG] pgmap v1561: 9 pgs: 9 active+clean; 127 GiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.2 MiB/s wr, 3 op/s 2021-07-31T02:10:13.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:12 smithi045 conmon[39718]: cluster 2021-07-31T02:10:12.118637+0000 mgr.x (mgr 2021-07-31T02:10:13.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:12 smithi045 conmon[39718]: .24542) 983 : cluster [DBG] pgmap v1561: 9 pgs: 9 active+clean; 127 GiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.2 MiB/s wr, 3 op/s 2021-07-31T02:10:13.580 INFO:tasks.workunit.client.0.smithi045.stdout: 301: throughput=4.99MB/sec pending data=0 2021-07-31T02:10:13.580 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-sJJsijcRg-jp_En off=5585155 len=633565 2021-07-31T02:10:13.581 INFO:tasks.workunit.client.0.smithi045.stdout:op 1543 completed, throughput=4.99MB/sec 2021-07-31T02:10:13.582 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Z-L7tRFPeQinJBP off=13056601 len=267516 2021-07-31T02:10:13.582 INFO:tasks.workunit.client.0.smithi045.stdout:op 1544 completed, throughput=4.99MB/sec 2021-07-31T02:10:13.582 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-6sGMKYVUcXpRhFH off=6931500 len=91196 2021-07-31T02:10:13.585 INFO:tasks.workunit.client.0.smithi045.stdout:op 1545 completed, throughput=4.99MB/sec 2021-07-31T02:10:13.585 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-leQRC49bHlodOYN off=16671794 len=323946 2021-07-31T02:10:13.586 INFO:tasks.workunit.client.0.smithi045.stdout:op 1546 completed, throughput=4.99MB/sec 2021-07-31T02:10:13.586 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-pBgVsrk07prWpuJ off=3625791 len=1189932 2021-07-31T02:10:13.592 INFO:tasks.workunit.client.0.smithi045.stdout:op 1547 completed, throughput=5MB/sec 2021-07-31T02:10:13.592 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Gc322Y7xtETGmh4 off=2012514 len=1344411 2021-07-31T02:10:13.600 INFO:tasks.workunit.client.0.smithi045.stdout:op 1548 completed, throughput=5MB/sec 2021-07-31T02:10:14.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:10:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:10:13.547Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:10:14.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:10:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:10:13.547Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:10:14.027 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:10:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:10:13.547Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:10:14.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:14 smithi154 conmon[34553]: cluster 2021-07-31T02:10:14.119175+0000 mgr.x (mgr.24542) 984 : cluster [DBG] pgmap v1562: 9 pgs: 9 active+clean; 127 GiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 2.9 MiB/s rd, 3.4 MiB/s wr, 6 op/s 2021-07-31T02:10:14.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:14 smithi045 conmon[39718]: cluster 2021-07-31T02:10:14.119175+0000 mgr.x (mgr. 2021-07-31T02:10:14.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:14 smithi045 conmon[39718]: 24542) 984 : cluster [DBG] pgmap v1562: 9 pgs: 9 active+clean; 127 GiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 2.9 MiB/s rd, 3.4 MiB/s wr, 6 op/s 2021-07-31T02:10:14.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:14 smithi045 conmon[33655]: cluster 2021-07-31T02:10:14.119175+0000 mgr.x (mgr.24542) 984 : 2021-07-31T02:10:14.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:14 smithi045 conmon[33655]: cluster [DBG] pgmap v1562: 9 pgs: 9 active+clean; 127 GiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 2.9 MiB/s rd, 3.4 MiB/s wr, 6 op/s 2021-07-31T02:10:14.601 INFO:tasks.workunit.client.0.smithi045.stdout: 302: throughput=4.98MB/sec pending data=0 2021-07-31T02:10:14.601 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-LY4-bCSilRIRACl off=13968674 len=574732 2021-07-31T02:10:14.605 INFO:tasks.workunit.client.0.smithi045.stdout:op 1549 completed, throughput=4.99MB/sec 2021-07-31T02:10:14.606 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-g2VBeL2XMEfk_ZP off=1546214 len=2042730 2021-07-31T02:10:14.608 INFO:tasks.workunit.client.0.smithi045.stdout:op 1550 completed, throughput=4.99MB/sec 2021-07-31T02:10:14.608 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-N5v8CcDD5c2HwFW off=8044423 len=136841 2021-07-31T02:10:14.611 INFO:tasks.workunit.client.0.smithi045.stdout:op 1551 completed, throughput=4.99MB/sec 2021-07-31T02:10:14.611 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-DtUxmparsYsXsL4 off=13253330 len=771879 2021-07-31T02:10:14.615 INFO:tasks.workunit.client.0.smithi045.stdout:op 1552 completed, throughput=4.99MB/sec 2021-07-31T02:10:14.615 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-6IVUanuNKa4G-yT off=925566 len=726407 2021-07-31T02:10:14.617 INFO:tasks.workunit.client.0.smithi045.stdout:op 1553 completed, throughput=5MB/sec 2021-07-31T02:10:14.617 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-3bCNQfTl3XSTvYt off=947056 len=2090837 2021-07-31T02:10:14.626 INFO:tasks.workunit.client.0.smithi045.stdout:op 1554 completed, throughput=5MB/sec 2021-07-31T02:10:15.626 INFO:tasks.workunit.client.0.smithi045.stdout: 303: throughput=4.99MB/sec pending data=0 2021-07-31T02:10:15.627 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-jtpvgLoLXYqubos off=15980772 len=813422 2021-07-31T02:10:15.631 INFO:tasks.workunit.client.0.smithi045.stdout:op 1555 completed, throughput=4.99MB/sec 2021-07-31T02:10:15.632 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-w95TzPMp9haFCd2 off=3487177 len=715170 2021-07-31T02:10:15.636 INFO:tasks.workunit.client.0.smithi045.stdout:op 1556 completed, throughput=4.99MB/sec 2021-07-31T02:10:15.636 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-CwjO3iSDB2dsrfz off=7736630 len=1685900 2021-07-31T02:10:15.639 INFO:tasks.workunit.client.0.smithi045.stdout:op 1557 completed, throughput=5MB/sec 2021-07-31T02:10:15.640 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-3a3xcI3WlcIsnlg off=8310809 len=1499731 2021-07-31T02:10:15.641 INFO:tasks.workunit.client.0.smithi045.stdout:op 1558 completed, throughput=5MB/sec 2021-07-31T02:10:16.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:16 smithi154 conmon[34553]: cluster 2021-07-31T02:10:16.119678+0000 mgr.x (mgr.24542) 985 : cluster [DBG] 2021-07-31T02:10:16.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:16 smithi154 conmon[34553]: pgmap v1563: 9 pgs: 9 active+clean; 127 GiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2021-07-31T02:10:16.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:16 smithi045 conmon[39718]: cluster 2021-07-31T02:10:16.119678+0000 mgr.x ( 2021-07-31T02:10:16.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:16 smithi045 conmon[39718]: mgr.24542) 985 : cluster [DBG] pgmap v1563: 9 pgs: 9 active+clean; 127 GiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2021-07-31T02:10:16.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:16 smithi045 conmon[33655]: cluster 2021-07-31T02:10:16.119678+0000 mgr.x (mgr.24542) 985 : cluster 2021-07-31T02:10:16.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:16 smithi045 conmon[33655]: [DBG] pgmap v1563: 9 pgs: 9 active+clean; 127 GiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2021-07-31T02:10:16.641 INFO:tasks.workunit.client.0.smithi045.stdout: 304: throughput=4.99MB/sec pending data=0 2021-07-31T02:10:16.642 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Z95MN_tOSdj9_0t off=18166498 len=2028263 2021-07-31T02:10:16.652 INFO:tasks.workunit.client.0.smithi045.stdout:op 1559 completed, throughput=4.99MB/sec 2021-07-31T02:10:16.652 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-L1T9zZGhgNB9d73 off=2553730 len=1284809 2021-07-31T02:10:16.654 INFO:tasks.workunit.client.0.smithi045.stdout:op 1560 completed, throughput=5MB/sec 2021-07-31T02:10:16.654 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-zHfvXERZOdlrCBC off=2490311 len=1733352 2021-07-31T02:10:16.662 INFO:tasks.workunit.client.0.smithi045.stdout:op 1561 completed, throughput=5MB/sec 2021-07-31T02:10:17.662 INFO:tasks.workunit.client.0.smithi045.stdout: 305: throughput=4.98MB/sec pending data=0 2021-07-31T02:10:17.663 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-pA2yJHmYg2Rv_H0 off=11123569 len=706138 2021-07-31T02:10:17.663 INFO:tasks.workunit.client.0.smithi045.stdout:op 1562 completed, throughput=4.99MB/sec 2021-07-31T02:10:17.664 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-LLl5j5HyA-k7JIk off=2644688 len=116461 2021-07-31T02:10:17.664 INFO:tasks.workunit.client.0.smithi045.stdout:op 1563 completed, throughput=4.99MB/sec 2021-07-31T02:10:17.665 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-DfUTBIV-hZI0KjY off=17535472 len=751031 2021-07-31T02:10:17.669 INFO:tasks.workunit.client.0.smithi045.stdout:op 1564 completed, throughput=4.99MB/sec 2021-07-31T02:10:17.669 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-RcoNnBsPKoTFPrq off=1587012 len=154012 2021-07-31T02:10:17.672 INFO:tasks.workunit.client.0.smithi045.stdout:op 1565 completed, throughput=4.99MB/sec 2021-07-31T02:10:17.673 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-0y2GHz8m_MdiBkS off=14197444 len=469142 2021-07-31T02:10:17.673 INFO:tasks.workunit.client.0.smithi045.stdout:op 1566 completed, throughput=4.99MB/sec 2021-07-31T02:10:17.674 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-IsbWnXZRn8xkprD off=2724378 len=341234 2021-07-31T02:10:17.676 INFO:tasks.workunit.client.0.smithi045.stdout:op 1567 completed, throughput=4.99MB/sec 2021-07-31T02:10:17.677 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-B125fNl1cpYf17u off=11792251 len=375067 2021-07-31T02:10:17.680 INFO:tasks.workunit.client.0.smithi045.stdout:op 1568 completed, throughput=4.99MB/sec 2021-07-31T02:10:17.681 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-8bx__hrhnG0-wx5 off=6480923 len=1539641 2021-07-31T02:10:17.683 INFO:tasks.workunit.client.0.smithi045.stdout:op 1569 completed, throughput=5MB/sec 2021-07-31T02:10:17.683 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-VXRwqubrG8s63bQ off=2408919 len=381081 2021-07-31T02:10:17.684 INFO:tasks.workunit.client.0.smithi045.stdout:op 1570 completed, throughput=5MB/sec 2021-07-31T02:10:17.684 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-jUC5pBQIyaLucje off=1127791 len=518018 2021-07-31T02:10:17.688 INFO:tasks.workunit.client.0.smithi045.stdout:op 1571 completed, throughput=5MB/sec 2021-07-31T02:10:18.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:18 smithi154 conmon[34553]: cluster 2021-07-31T02:10:18.120254+0000 mgr.x (mgr.24542) 986 : cluster [DBG] pgmap v1564: 9 pgs: 9 active+clean; 127 GiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2021-07-31T02:10:18.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:18 smithi154 conmon[34553]: 2021-07-31T02:10:18.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:18 smithi045 conmon[39718]: cluster 2021-07-31T02:10:18.120254+0000 2021-07-31T02:10:18.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:18 smithi045 conmon[39718]: mgr.x (mgr.24542) 986 : cluster [DBG] pgmap v1564: 9 pgs: 9 active+clean; 127 GiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2021-07-31T02:10:18.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:18 smithi045 conmon[33655]: cluster 2021-07-31T02:10:18.120254 2021-07-31T02:10:18.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:18 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 986 : cluster [DBG] pgmap v1564: 9 pgs: 9 active+clean; 127 GiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2021-07-31T02:10:18.689 INFO:tasks.workunit.client.0.smithi045.stdout: 306: throughput=4.98MB/sec pending data=0 2021-07-31T02:10:18.689 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-qPAT3tVjNwifMvs off=3113718 len=832980 2021-07-31T02:10:18.690 INFO:tasks.workunit.client.0.smithi045.stdout:op 1572 completed, throughput=4.99MB/sec 2021-07-31T02:10:18.690 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj--xrLxyvjEyKkatP off=628949 len=731030 2021-07-31T02:10:18.695 INFO:tasks.workunit.client.0.smithi045.stdout:op 1573 completed, throughput=4.99MB/sec 2021-07-31T02:10:18.695 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ahz3miqaOVyKUvE off=6174942 len=1337730 2021-07-31T02:10:18.702 INFO:tasks.workunit.client.0.smithi045.stdout:op 1574 completed, throughput=4.99MB/sec 2021-07-31T02:10:18.702 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-xbNE-ycYzm2h3NQ off=7322116 len=1546102 2021-07-31T02:10:18.704 INFO:tasks.workunit.client.0.smithi045.stdout:op 1575 completed, throughput=5MB/sec 2021-07-31T02:10:18.704 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-VKspf-c7xufM9Mf off=19312039 len=672069 2021-07-31T02:10:18.706 INFO:tasks.workunit.client.0.smithi045.stdout:op 1576 completed, throughput=5MB/sec 2021-07-31T02:10:18.706 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-DwCWMjmwYh1fRcp off=18015021 len=1422493 2021-07-31T02:10:18.708 INFO:tasks.workunit.client.0.smithi045.stdout:op 1577 completed, throughput=5MB/sec 2021-07-31T02:10:19.708 INFO:tasks.workunit.client.0.smithi045.stdout: 307: throughput=4.99MB/sec pending data=0 2021-07-31T02:10:19.709 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-il0s9k9INctgZni off=4237779 len=463868 2021-07-31T02:10:19.713 INFO:tasks.workunit.client.0.smithi045.stdout:op 1578 completed, throughput=4.99MB/sec 2021-07-31T02:10:19.714 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-SYXP22-f3o8NkTx off=9326937 len=1463725 2021-07-31T02:10:19.720 INFO:tasks.workunit.client.0.smithi045.stdout:op 1579 completed, throughput=4.99MB/sec 2021-07-31T02:10:19.721 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-2uCs9Xk1To8fTIv off=465398 len=724694 2021-07-31T02:10:19.723 INFO:tasks.workunit.client.0.smithi045.stdout:op 1580 completed, throughput=5MB/sec 2021-07-31T02:10:19.723 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-oNtRnP1IPm6mue4 off=3488648 len=264483 2021-07-31T02:10:19.726 INFO:tasks.workunit.client.0.smithi045.stdout:op 1581 completed, throughput=5MB/sec 2021-07-31T02:10:19.726 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-mBW0fLIMgV-oGgG off=309159 len=1068240 2021-07-31T02:10:19.731 INFO:tasks.workunit.client.0.smithi045.stdout:op 1582 completed, throughput=5MB/sec 2021-07-31T02:10:19.732 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-uAUN-TeSzotdjYl off=31205 len=1831524 2021-07-31T02:10:19.742 INFO:tasks.workunit.client.0.smithi045.stdout:op 1583 completed, throughput=5.01MB/sec 2021-07-31T02:10:20.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:20 smithi154 conmon[34553]: cluster 2021-07-31T02:10:20.120696+0000 mgr.x (mgr.24542) 987 : cluster 2021-07-31T02:10:20.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:20 smithi154 conmon[34553]: [DBG] pgmap v1565: 9 pgs: 9 active+clean; 127 GiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 3.0 MiB/s rd, 3.5 MiB/s wr, 6 op/s 2021-07-31T02:10:20.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:20 smithi045 conmon[39718]: cluster 2021-07-31T02:10:20.120696+0000 mgr.x (mgr. 2021-07-31T02:10:20.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:20 smithi045 conmon[39718]: 24542) 987 : cluster [DBG] pgmap v1565: 9 pgs: 9 active+clean; 127 GiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 3.0 MiB/s rd, 3.5 MiB/s wr, 6 op/s 2021-07-31T02:10:20.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:20 smithi045 conmon[33655]: cluster 2021-07-31T02:10:20.120696+0000 mgr.x (mgr 2021-07-31T02:10:20.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:20 smithi045 conmon[33655]: .24542) 987 : cluster [DBG] pgmap v1565: 9 pgs: 9 active+clean; 127 GiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 3.0 MiB/s rd, 3.5 MiB/s wr, 6 op/s 2021-07-31T02:10:20.742 INFO:tasks.workunit.client.0.smithi045.stdout: 308: throughput=4.99MB/sec pending data=0 2021-07-31T02:10:20.742 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-i8zkis1GuEV20zY off=5496069 len=2086651 2021-07-31T02:10:20.753 INFO:tasks.workunit.client.0.smithi045.stdout:op 1584 completed, throughput=5MB/sec 2021-07-31T02:10:20.753 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-TuRyDdI_6g3agTm off=196649 len=120445 2021-07-31T02:10:20.756 INFO:tasks.workunit.client.0.smithi045.stdout:op 1585 completed, throughput=5MB/sec 2021-07-31T02:10:20.756 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-xvEUCKfPM3IXS73 off=16666811 len=583114 2021-07-31T02:10:20.756 INFO:tasks.workunit.client.0.smithi045.stdout:op 1586 completed, throughput=5MB/sec 2021-07-31T02:10:20.757 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-xJR6i6olMjmxXzo off=6160967 len=775845 2021-07-31T02:10:20.762 INFO:tasks.workunit.client.0.smithi045.stdout:op 1587 completed, throughput=5MB/sec 2021-07-31T02:10:20.762 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-9GDchWO_S1ZzEkS off=1156617 len=1195659 2021-07-31T02:10:20.763 INFO:tasks.workunit.client.0.smithi045.stdout:op 1588 completed, throughput=5MB/sec 2021-07-31T02:10:21.764 INFO:tasks.workunit.client.0.smithi045.stdout: 309: throughput=4.99MB/sec pending data=0 2021-07-31T02:10:21.764 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-QKmOX2UJUY6nFw5 off=9492308 len=159293 2021-07-31T02:10:21.767 INFO:tasks.workunit.client.0.smithi045.stdout:op 1589 completed, throughput=4.99MB/sec 2021-07-31T02:10:21.767 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-DUX2vlCrbXLxzl4 off=12167499 len=1268362 2021-07-31T02:10:21.770 INFO:tasks.workunit.client.0.smithi045.stdout:op 1590 completed, throughput=4.99MB/sec 2021-07-31T02:10:21.770 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-XQ6eUa_4DTDH6r7 off=12568103 len=818261 2021-07-31T02:10:21.772 INFO:tasks.workunit.client.0.smithi045.stdout:op 1591 completed, throughput=4.99MB/sec 2021-07-31T02:10:21.772 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-91nyivwrxr7iLN2 off=9436333 len=317784 2021-07-31T02:10:21.772 INFO:tasks.workunit.client.0.smithi045.stdout:op 1592 completed, throughput=5MB/sec 2021-07-31T02:10:21.773 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-_I2WC-PAyZqQ8-G off=1868741 len=2039683 2021-07-31T02:10:21.775 INFO:tasks.workunit.client.0.smithi045.stdout:op 1593 completed, throughput=5MB/sec 2021-07-31T02:10:22.776 INFO:tasks.workunit.client.0.smithi045.stdout: 310: throughput=4.99MB/sec pending data=0 2021-07-31T02:10:22.776 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-zQxiov2JIfNAmNy off=431510 len=719999 2021-07-31T02:10:22.777 INFO:tasks.workunit.client.0.smithi045.stdout:op 1594 completed, throughput=4.99MB/sec 2021-07-31T02:10:22.777 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-kkNBLm7KknbU6KC off=13910096 len=661802 2021-07-31T02:10:22.778 INFO:tasks.workunit.client.0.smithi045.stdout:op 1595 completed, throughput=4.99MB/sec 2021-07-31T02:10:22.779 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Y3pX6OXITb4Ju4O off=2721898 len=853432 2021-07-31T02:10:22.780 INFO:tasks.workunit.client.0.smithi045.stdout:op 1596 completed, throughput=4.99MB/sec 2021-07-31T02:10:22.781 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Jya3NR3_HgHM463 off=12285541 len=1088778 2021-07-31T02:10:22.787 INFO:tasks.workunit.client.0.smithi045.stdout:op 1597 completed, throughput=5MB/sec 2021-07-31T02:10:22.788 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-mjjk1xsanXSHb0Q off=4439912 len=1057180 2021-07-31T02:10:22.789 INFO:tasks.workunit.client.0.smithi045.stdout:op 1598 completed, throughput=5MB/sec 2021-07-31T02:10:22.789 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-QCf_FKtSDSxIiaa off=4838825 len=360961 2021-07-31T02:10:22.793 INFO:tasks.workunit.client.0.smithi045.stdout:op 1599 completed, throughput=5MB/sec 2021-07-31T02:10:22.794 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-7X3D1SekWRRYZGb off=1335486 len=207480 2021-07-31T02:10:22.795 INFO:tasks.workunit.client.0.smithi045.stdout:op 1600 completed, throughput=5MB/sec 2021-07-31T02:10:23.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:22 smithi154 conmon[34553]: cluster 2021-07-31T02:10:22.121204+0000 mgr.x (mgr.24542) 988 : cluster [DBG] pgmap v1566: 9 pgs: 9 active+clean; 127 GiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2021-07-31T02:10:23.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:22 smithi045 conmon[39718]: cluster 2021-07-31T02:10:22.121204+0000 mgr.x (mgr.24542) 988 : cluster [DBG] pgmap v1566: 9 pgs: 9 active+clean; 127 GiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2021-07-31T02:10:23.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:22 smithi045 conmon[33655]: cluster 2021-07-31T02:10:22.121204+0000 mgr.x (mgr.24542) 988 : cluster 2021-07-31T02:10:23.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:22 smithi045 conmon[33655]: [DBG] pgmap v1566: 9 pgs: 9 active+clean; 127 GiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2021-07-31T02:10:23.794 INFO:tasks.workunit.client.0.smithi045.stdout: 311: throughput=4.98MB/sec pending data=0 2021-07-31T02:10:23.795 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-t899wmZ5ktyJM_H off=904632 len=1982764 2021-07-31T02:10:23.798 INFO:tasks.workunit.client.0.smithi045.stdout:op 1601 completed, throughput=4.99MB/sec 2021-07-31T02:10:23.798 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ahz3miqaOVyKUvE off=2759044 len=345198 2021-07-31T02:10:23.802 INFO:tasks.workunit.client.0.smithi045.stdout:op 1602 completed, throughput=4.99MB/sec 2021-07-31T02:10:23.802 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-sQVP-1RcJwGs9mE off=4250132 len=496728 2021-07-31T02:10:23.804 INFO:tasks.workunit.client.0.smithi045.stdout:op 1603 completed, throughput=4.99MB/sec 2021-07-31T02:10:23.805 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-b4KzjJtpWr-JAu4 off=3232435 len=1904662 2021-07-31T02:10:23.806 INFO:tasks.workunit.client.0.smithi045.stdout:op 1604 completed, throughput=5MB/sec 2021-07-31T02:10:23.807 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-zD0c3MV_NgxmY32 off=1333389 len=1136420 2021-07-31T02:10:23.809 INFO:tasks.workunit.client.0.smithi045.stdout:op 1605 completed, throughput=5MB/sec 2021-07-31T02:10:24.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:10:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:10:23.547Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:10:24.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:10:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:10:23.547Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:10:24.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:10:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:10:23.547Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:10:24.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:24 smithi154 conmon[34553]: cluster 2021-07-31T02:10:24.121621+0000 mgr.x (mgr.24542) 989 : cluster [DBG] pgmap v1567: 9 pgs: 9 active+clean; 127 GiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 3.3 MiB/s rd, 3.1 MiB/s wr, 7 op/s 2021-07-31T02:10:24.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:24 smithi045 conmon[39718]: cluster 2021-07-31T02:10:24.121621+0000 mgr.x (mgr.24542) 989 2021-07-31T02:10:24.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:24 smithi045 conmon[39718]: : cluster [DBG] pgmap v1567: 9 pgs: 9 active+clean; 127 GiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 3.3 MiB/s rd, 3.1 MiB/s wr, 7 op/s 2021-07-31T02:10:24.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:24 smithi045 conmon[33655]: cluster 2021-07-31T02:10:24.121621+0000 mgr.x (mgr.24542) 989 : 2021-07-31T02:10:24.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:24 smithi045 conmon[33655]: cluster [DBG] pgmap v1567: 9 pgs: 9 active+clean; 127 GiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 3.3 MiB/s rd, 3.1 MiB/s wr, 7 op/s 2021-07-31T02:10:24.809 INFO:tasks.workunit.client.0.smithi045.stdout: 312: throughput=4.99MB/sec pending data=0 2021-07-31T02:10:24.810 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-1dhp5c7sVBfR-Aj off=7875501 len=997480 2021-07-31T02:10:24.812 INFO:tasks.workunit.client.0.smithi045.stdout:op 1606 completed, throughput=4.99MB/sec 2021-07-31T02:10:24.812 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-bmWlcpnQG34924g off=18714326 len=1019448 2021-07-31T02:10:24.814 INFO:tasks.workunit.client.0.smithi045.stdout:op 1607 completed, throughput=4.99MB/sec 2021-07-31T02:10:24.814 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-sp034GNgSzJj52o off=0 len=1375098 2021-07-31T02:10:24.816 INFO:tasks.workunit.client.0.smithi045.stdout:op 1608 completed, throughput=5MB/sec 2021-07-31T02:10:24.816 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-KOx2y_6sBb6ynSv off=205214 len=1705001 2021-07-31T02:10:24.818 INFO:tasks.workunit.client.0.smithi045.stdout:op 1609 completed, throughput=5MB/sec 2021-07-31T02:10:25.818 INFO:tasks.workunit.client.0.smithi045.stdout: 313: throughput=4.99MB/sec pending data=0 2021-07-31T02:10:25.819 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-DjdFZUwjhotrYgX off=12988313 len=947795 2021-07-31T02:10:25.819 INFO:tasks.workunit.client.0.smithi045.stdout:op 1610 completed, throughput=4.99MB/sec 2021-07-31T02:10:25.820 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-kVzH4t4eLfZ8W3u off=2541975 len=2000393 2021-07-31T02:10:25.828 INFO:tasks.workunit.client.0.smithi045.stdout:op 1611 completed, throughput=4.99MB/sec 2021-07-31T02:10:25.828 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-FT23sZ4pfeRwBKX off=18877653 len=124786 2021-07-31T02:10:25.831 INFO:tasks.workunit.client.0.smithi045.stdout:op 1612 completed, throughput=4.99MB/sec 2021-07-31T02:10:25.831 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-KObhKCfNgWbBEZv off=15103598 len=893764 2021-07-31T02:10:25.836 INFO:tasks.workunit.client.0.smithi045.stdout:op 1613 completed, throughput=5MB/sec 2021-07-31T02:10:25.836 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-yQuqjsl_PY50KMt off=4627475 len=2015238 2021-07-31T02:10:25.839 INFO:tasks.workunit.client.0.smithi045.stdout:op 1614 completed, throughput=5MB/sec 2021-07-31T02:10:26.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:26 smithi154 conmon[34553]: cluster 2021-07-31T02:10:26.122201+0000 mgr.x (mgr.24542) 990 : cluster [DBG] pgmap v1568: 9 pgs: 9 active+clean; 127 GiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.8 MiB/s wr, 4 op/s 2021-07-31T02:10:26.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:26 smithi045 conmon[39718]: cluster 2021-07-31T02:10:26.122201+0000 mgr.x (mgr. 2021-07-31T02:10:26.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:26 smithi045 conmon[39718]: 24542) 990 : cluster [DBG] pgmap v1568: 9 pgs: 9 active+clean; 127 GiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.8 MiB/s wr, 4 op/s 2021-07-31T02:10:26.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:26 smithi045 conmon[33655]: cluster 2021-07-31T02:10:26.122201+0000 mgr.x (mgr.24542) 990 : cluster 2021-07-31T02:10:26.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:26 smithi045 conmon[33655]: [DBG] pgmap v1568: 9 pgs: 9 active+clean; 127 GiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.8 MiB/s wr, 4 op/s 2021-07-31T02:10:26.840 INFO:tasks.workunit.client.0.smithi045.stdout: 314: throughput=4.99MB/sec pending data=0 2021-07-31T02:10:26.841 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-4jJTxjwjFIhAzj3 off=2190829 len=1249957 2021-07-31T02:10:26.843 INFO:tasks.workunit.client.0.smithi045.stdout:op 1615 completed, throughput=4.99MB/sec 2021-07-31T02:10:26.843 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-qwmTvwV5Ch7f5bq off=6016407 len=78199 2021-07-31T02:10:26.844 INFO:tasks.workunit.client.0.smithi045.stdout:op 1616 completed, throughput=4.99MB/sec 2021-07-31T02:10:26.844 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-eK-K4ezvey3uOtj off=8635290 len=568587 2021-07-31T02:10:26.845 INFO:tasks.workunit.client.0.smithi045.stdout:op 1617 completed, throughput=4.99MB/sec 2021-07-31T02:10:26.845 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Q41gEWwQQIXaqRn off=16671957 len=1113155 2021-07-31T02:10:26.846 INFO:tasks.workunit.client.0.smithi045.stdout:op 1618 completed, throughput=5MB/sec 2021-07-31T02:10:26.847 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-6JCz3CxqHdDsGFu off=14071835 len=1179636 2021-07-31T02:10:26.852 INFO:tasks.workunit.client.0.smithi045.stdout:op 1619 completed, throughput=5MB/sec 2021-07-31T02:10:26.853 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-QxAyLrv0UF_fJZ0 off=7817754 len=1317409 2021-07-31T02:10:26.859 INFO:tasks.workunit.client.0.smithi045.stdout:op 1620 completed, throughput=5MB/sec 2021-07-31T02:10:27.860 INFO:tasks.workunit.client.0.smithi045.stdout: 315: throughput=4.99MB/sec pending data=0 2021-07-31T02:10:27.860 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Tu08iqyr-BT4oOT off=6001611 len=837076 2021-07-31T02:10:27.865 INFO:tasks.workunit.client.0.smithi045.stdout:op 1621 completed, throughput=4.99MB/sec 2021-07-31T02:10:27.865 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-srvN0I5MF1mBkBb off=2571183 len=1077394 2021-07-31T02:10:27.867 INFO:tasks.workunit.client.0.smithi045.stdout:op 1622 completed, throughput=4.99MB/sec 2021-07-31T02:10:27.867 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-zGeXhz2UK05e884 off=8096258 len=2073166 2021-07-31T02:10:27.876 INFO:tasks.workunit.client.0.smithi045.stdout:op 1623 completed, throughput=5MB/sec 2021-07-31T02:10:27.876 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-hROke7ddFeOBzMz off=11946379 len=1642193 2021-07-31T02:10:27.884 INFO:tasks.workunit.client.0.smithi045.stdout:op 1624 completed, throughput=5MB/sec 2021-07-31T02:10:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:28 smithi154 conmon[34553]: audit 2021-07-31T02:10:28.016856+0000 mon.b (mon.2) 183 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:10:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:28 smithi154 conmon[34553]: 2021-07-31T02:10:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:28 smithi154 conmon[34553]: audit 2021-07-31T02:10:28.018085+0000 mon.a (mon.0) 1865 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:10:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:28 smithi154 conmon[34553]: audit 2021-07-31T02:10:28.036289+0000 mon.b (mon.2) 184 : audit 2021-07-31T02:10:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:28 smithi154 conmon[34553]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:10:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:28 smithi154 conmon[34553]: audit 2021-07-31T02:10:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:28 smithi154 conmon[34553]: 2021-07-31T02:10:28 2021-07-31T02:10:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:28 smithi154 conmon[34553]: .037240+0000 mon.a (mon.0) 2021-07-31T02:10:28.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:28 smithi154 conmon[34553]: 1866 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:10:28.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:28 smithi045 conmon[33655]: audit 2021-07-31T02:10:28.016856+0000 mon.b (mon.2) 183 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:10:28.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:28 smithi045 conmon[33655]: audit 2021-07-31T02:10:28.018085+0000 mon.a (mon.0) 1865 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:10:28.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:28 smithi045 conmon[33655]: audit 2021-07-31T02:10:28.036289+0000 mon.b (mon.2) 2021-07-31T02:10:28.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:28 smithi045 conmon[33655]: 184 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:10:28.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:28 smithi045 conmon[33655]: audit 2021-07-31 2021-07-31T02:10:28.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:28 smithi045 conmon[33655]: T02:10:28.037240+0000 mon.a (mon.0) 1866 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:10:28.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:28 smithi045 conmon[39718]: audit 2021-07-31T02:10:28.016856+0000 mon.b (mon.2) 2021-07-31T02:10:28.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:28 smithi045 conmon[39718]: 183 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:10:28.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:28 smithi045 conmon[39718]: audit 2021-07-31T02:10:28.018085+0000 mon.a (mon. 2021-07-31T02:10:28.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:28 smithi045 conmon[39718]: 0) 1865 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:10:28.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:28 smithi045 conmon[39718]: audit 2021-07-31T02:10:28.036289+0000 2021-07-31T02:10:28.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:28 smithi045 conmon[39718]: mon.b (mon.2) 184 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:10:28.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:28 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T02:10:28.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:28 smithi045 conmon[39718]: T02:10:28.037240+0000 mon.a (mon.0) 1866 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:10:28.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:28 smithi045 conmon[39718]: 2021-07-31T02:10:28.884 INFO:tasks.workunit.client.0.smithi045.stdout: 316: throughput=4.99MB/sec pending data=0 2021-07-31T02:10:28.885 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-w3i_e8q-u5fRwHG off=10201895 len=945076 2021-07-31T02:10:28.890 INFO:tasks.workunit.client.0.smithi045.stdout:op 1625 completed, throughput=4.99MB/sec 2021-07-31T02:10:28.891 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-jpL_W0zcluOU2u0 off=9022461 len=668547 2021-07-31T02:10:28.892 INFO:tasks.workunit.client.0.smithi045.stdout:op 1626 completed, throughput=4.99MB/sec 2021-07-31T02:10:28.892 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-eJ-5sglcHTPK2qQ off=3180652 len=259723 2021-07-31T02:10:28.893 INFO:tasks.workunit.client.0.smithi045.stdout:op 1627 completed, throughput=4.99MB/sec 2021-07-31T02:10:28.894 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-AYUXbDzmj7_ilUw off=13888837 len=709140 2021-07-31T02:10:28.897 INFO:tasks.workunit.client.0.smithi045.stdout:op 1628 completed, throughput=5MB/sec 2021-07-31T02:10:28.897 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-7uCcD3YHOR3HzsR off=10375970 len=1542394 2021-07-31T02:10:28.899 INFO:tasks.workunit.client.0.smithi045.stdout:op 1629 completed, throughput=5MB/sec 2021-07-31T02:10:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:29 smithi154 conmon[34553]: cluster 2021-07-31T02:10:28.122636+0000 mgr.x (mgr.24542) 991 : cluster [DBG] pgmap v1569: 9 pgs: 9 active+clean; 127 GiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.8 MiB/s wr, 4 op/s 2021-07-31T02:10:29.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:29 smithi045 conmon[39718]: cluster 2021-07-31T02:10:28.122636+0000 2021-07-31T02:10:29.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:29 smithi045 conmon[39718]: mgr.x (mgr.24542) 991 : cluster [DBG] pgmap v1569: 9 pgs: 9 active+clean; 127 GiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.8 MiB/s wr, 4 op/s 2021-07-31T02:10:29.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:29 smithi045 conmon[33655]: cluster 2021-07-31T02:10:28.122636+0000 mgr.x (mgr.24542) 991 : cluster [DBG] pgmap v1569: 9 pgs: 9 active+clean; 127 GiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.8 MiB/s wr, 4 op/s 2021-07-31T02:10:29.899 INFO:tasks.workunit.client.0.smithi045.stdout: 317: throughput=4.99MB/sec pending data=0 2021-07-31T02:10:29.899 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-g-1l3BzUuo3GtbQ off=9257786 len=1975704 2021-07-31T02:10:29.901 INFO:tasks.workunit.client.0.smithi045.stdout:op 1630 completed, throughput=4.99MB/sec 2021-07-31T02:10:29.901 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Y3y7rHSzJX-vBdQ off=17536123 len=1335121 2021-07-31T02:10:29.909 INFO:tasks.workunit.client.0.smithi045.stdout:op 1631 completed, throughput=4.99MB/sec 2021-07-31T02:10:29.909 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-8LOyay4jtdxEwAD off=55372 len=1179433 2021-07-31T02:10:29.915 INFO:tasks.workunit.client.0.smithi045.stdout:op 1632 completed, throughput=5MB/sec 2021-07-31T02:10:29.915 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ww8H-G_32pWN6BG off=6705403 len=435668 2021-07-31T02:10:29.918 INFO:tasks.workunit.client.0.smithi045.stdout:op 1633 completed, throughput=5MB/sec 2021-07-31T02:10:29.919 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-3qg1ET8SvUvJ_AS off=5536407 len=1712388 2021-07-31T02:10:29.927 INFO:tasks.workunit.client.0.smithi045.stdout:op 1634 completed, throughput=5MB/sec 2021-07-31T02:10:30.465 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[34553]: cluster 2021-07-31T02:10:30.123268+0000 mgr.x (mgr.24542) 992 : cluster [DBG] pgmap v1570: 9 pgs: 9 active+clean; 127 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 3.5 MiB/s rd, 2.8 MiB/s wr, 6 op/s 2021-07-31T02:10:30.466 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: debug 2021-07-31T02:10:30.432+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:10:30.466 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:10:30.466 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:10:30.467 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: yield 2021-07-31T02:10:30.467 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:10:30.467 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:10:30.467 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:10:30.468 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:10:30.468 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:10:30.468 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:10:30.468 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:10:30.468 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:10:30.469 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:10:30.469 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:10:30.470 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:10:30.470 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: debug 2021-07-31T02:10:30.444+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:10:30.470 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:10:30.471 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:10:30.471 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: yield 2021-07-31T02:10:30.471 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:10:30.471 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:10:30.471 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:10:30.472 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:10:30.472 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:10:30.472 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:10:30.472 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:10:30.472 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:10:30.473 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:10:30.473 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:10:30.473 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:10:30.475 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:30 smithi045 conmon[39718]: cluster 2021-07-31T02:10:30.123268+0000 mgr.x ( 2021-07-31T02:10:30.475 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:30 smithi045 conmon[39718]: mgr.24542) 992 : cluster [DBG] pgmap v1570: 9 pgs: 9 active+clean; 127 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 3.5 MiB/s rd, 2.8 MiB/s wr, 6 op/s 2021-07-31T02:10:30.476 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:30 smithi045 conmon[33655]: cluster 2021-07-31T02:10:30.123268+0000 mgr.x (mgr.24542) 992 : 2021-07-31T02:10:30.476 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:30 smithi045 conmon[33655]: cluster [DBG] pgmap v1570: 9 pgs: 9 active+clean; 127 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 3.5 MiB/s rd, 2.8 MiB/s wr, 6 op/s 2021-07-31T02:10:30.753 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: debug 2021-07-31T02:10:30.474+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:10:30.753 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:10:30.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:10:30.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: yield 2021-07-31T02:10:30.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:10:30.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:10:30.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:10:30.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:10:30.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:10:30.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:10:30.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:10:30.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:10:30.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:10:30.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:10:30.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:10:30.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: debug 2021-07-31T02:10:30.475+0000 7fcb9095b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:10:30.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:10:30.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:10:30.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: yield 2021-07-31T02:10:30.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:10:30.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:10:30.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:10:30.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:10:30.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:10:30.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:10:30.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:10:30.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:10:30.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:10:30.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:10:30.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:10:30.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: 2021-07-31T02:10:30.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:10:30.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: 2021-07-31T02:10:30.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:10:30.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:10:30.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:10:30.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:10:30.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:10:30.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:10:30.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:10:30.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:10:30.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:10:30.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:10:30.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:10:30.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:10:30.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:10:30.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:10:30.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:10:30.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:10:30.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:10:30.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:10:30.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: raise 2021-07-31T02:10:30.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:10:30.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:10:30.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:10:30.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:10:30.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:10:30.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: debug 2021-07-31T02:10:30.484+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:10:30.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:10:30.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:10:30.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: yield 2021-07-31T02:10:30.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:10:30.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:10:30.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:10:30.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:10:30.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:10:30.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:10:30.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:10:30.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:10:30.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:10:30.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:10:30.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:10:30.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: debug 2021-07-31T02:10:30.485+0000 7fcb9115c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:10:30.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:10:30.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:10:30.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: yield 2021-07-31T02:10:30.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:10:30.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:10:30.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:10:30.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:10:30.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:10:30.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:10:30.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:10:30.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:10:30.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:10:30.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:10:30.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:10:30.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: 2021-07-31T02:10:30.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:10:30.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: 2021-07-31T02:10:30.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:10:30.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:10:30.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:10:30.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:10:30.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:10:30.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:10:30.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:10:30.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:10:30.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:10:30.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:10:30.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:10:30.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:10:30.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:10:30.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:10:30.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:10:30.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:10:30.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:10:30.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:10:30.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: raise 2021-07-31T02:10:30.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:10:30.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:10:30.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:10:30.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:10:30.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:10:30 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:10:30.927 INFO:tasks.workunit.client.0.smithi045.stdout: 318: throughput=4.99MB/sec pending data=0 2021-07-31T02:10:30.927 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Z9iy9Uk3l0GBAde off=2520806 len=1292495 2021-07-31T02:10:30.933 INFO:tasks.workunit.client.0.smithi045.stdout:op 1635 completed, throughput=4.99MB/sec 2021-07-31T02:10:30.933 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-mZ1zOtqsQ1T5tto off=4814720 len=664137 2021-07-31T02:10:30.938 INFO:tasks.workunit.client.0.smithi045.stdout:op 1636 completed, throughput=4.99MB/sec 2021-07-31T02:10:30.938 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-HT6053OexfcbrGh off=5954606 len=1653132 2021-07-31T02:10:30.944 INFO:tasks.workunit.client.0.smithi045.stdout:op 1637 completed, throughput=5MB/sec 2021-07-31T02:10:30.945 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-hbduplg0YAi3KNC off=4561921 len=1149049 2021-07-31T02:10:30.946 INFO:tasks.workunit.client.0.smithi045.stdout:op 1638 completed, throughput=5MB/sec 2021-07-31T02:10:31.225 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: audit 2021-07-31T02:10:30.386621+0000 mon.b (mon.2) 2021-07-31T02:10:31.226 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: 185 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:10:31.226 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: audit 2021-07-31T02:10:30.388061+0000 mon.b (mon.2) 186 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:10:31.226 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: audit 2021-07-31T02:10:30.389103+0000 mon.b (mon.2) 187 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:10:31.226 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:10:30.389868+0000 mgr.x (mgr.24542) 993 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:10:31.227 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:10:30.433410+0000 mgr.x (mgr.24542) 994 : cephadm 2021-07-31T02:10:31.227 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: [ERR] 2021-07-31T02:10:31.227 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:10:31.227 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:10:31.227 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: yield 2021-07-31T02:10:31.228 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:10:31.228 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:10:31.228 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:10:31.228 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:10:31.229 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:10:31.229 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:10:31.231 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:10:31.231 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:10:31.232 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:10:31.232 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: task.result() 2021-07-31T02:10:31.233 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:10:31.233 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:10:30.433823+0000 mgr.x ( 2021-07-31T02:10:31.234 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: mgr.24542) 995 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:10:31.234 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:10:30.445052+0000 mgr.x (mgr.24542) 996 : cephadm [ERR] 2021-07-31T02:10:31.235 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:10:31.235 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:10:31.236 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: yield 2021-07-31T02:10:31.236 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:10:31.236 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:10:31.237 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:10:31.238 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:10:31.238 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:10:31.238 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:10:31.239 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:10:31.239 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:10:31.241 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: audit 2021-07-31T02:10:30.386621 2021-07-31T02:10:31.241 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: +0000 mon.b (mon.2) 185 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:10:31.241 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: audit 2021-07-31T02 2021-07-31T02:10:31.242 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: :10:30.388061+0000 mon.b 2021-07-31T02:10:31.242 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: (mon.2) 186 : audit 2021-07-31T02:10:31.242 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:10:31.242 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: audit 2021 2021-07-31T02:10:31.243 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: -07-31T02:10:30.389103 2021-07-31T02:10:31.243 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: +0000 mon.b (mon.2) 2021-07-31T02:10:31.243 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: 187 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:10:31.243 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: cephadm 2021 2021-07-31T02:10:31.243 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: -07-31T02:10:30. 2021-07-31T02:10:31.244 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: 389868+0000 mgr.x (mgr. 2021-07-31T02:10:31.244 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: 24542) 993 : cephadm [WRN] 2021-07-31T02:10:31.244 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:10:31.244 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: cephadm 2021 2021-07-31T02:10:31.245 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: -07-31T02:10:30. 2021-07-31T02:10:31.245 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: 433410+0000 mgr.x (mgr.24542 2021-07-31T02:10:31.245 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: ) 994 : cephadm [ERR] 2021-07-31T02:10:31.245 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:10:31.245 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:10:31.246 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: yield 2021-07-31T02:10:31.246 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:10:31.246 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:10:31.246 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:10:31.246 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:10:31.247 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:10:31.247 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:10:31.247 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:10:31.247 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:10:31.247 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:10:31.248 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: task.result() 2021-07-31T02:10:31.248 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:10:31.248 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:10:30.433823+0000 mgr.x (mgr.24542) 995 : cephadm [INF] 2021-07-31T02:10:31.248 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:10:31.249 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: cephadm 2021 2021-07-31T02:10:31.249 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: -07-31T02:10:30. 2021-07-31T02:10:31.249 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: 445052+0000 mgr.x (mgr.24542 2021-07-31T02:10:31.249 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: ) 996 : cephadm [ERR] 2021-07-31T02:10:31.249 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:10:31.250 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:10:31.250 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: yield 2021-07-31T02:10:31.250 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:10:31.250 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:10:31.251 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:10:31.251 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: task.result() 2021-07-31T02:10:31.251 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:10:31.251 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:10:30.445557+0000 mgr.x (mgr.24542 2021-07-31T02:10:31.252 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: ) 997 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:10:31.252 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:10:30.475661+0000 mgr.x (mgr.24542) 998 2021-07-31T02:10:31.252 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: : cephadm [ERR] 2021-07-31T02:10:31.252 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:10:31.252 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:10:31.253 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: yield 2021-07-31T02:10:31.253 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:10:31.253 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:10:31.253 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:10:31.254 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:10:31.254 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:10:31.254 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:10:31.254 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:10:31.254 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:10:31.255 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:10:31.255 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: task.result() 2021-07-31T02:10:31.255 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:10:31.255 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:10:30.476122+0000 mgr.x (mgr.24542) 999 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:10:31.255 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:10:31.256 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:10:31.256 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: yield 2021-07-31T02:10:31.256 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:10:31.256 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:10:31.256 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:10:31.257 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:10:31.257 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:10:31.257 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:10:31.257 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:10:31.258 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:10:31.258 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:10:31.258 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: task.result() 2021-07-31T02:10:31.258 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:10:31.258 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: 2021-07-31T02:10:31.259 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:10:31.259 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: 2021-07-31T02:10:31.259 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:10:31.259 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:10:31.259 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:10:31.260 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:10:31.261 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:10:31.261 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:10:31.261 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:10:31.261 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:10:31.261 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:10:31.262 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:10:31.262 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: task.result() 2021-07-31T02:10:31.262 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:10:31.262 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: 2021-07-31T02:10:31.263 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:10:30.445557 2021-07-31T02:10:31.263 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 997 : cephadm 2021-07-31T02:10:31.263 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:10:31.263 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: cephadm 2021 2021-07-31T02:10:31.264 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: -07-31T02:10:30 2021-07-31T02:10:31.264 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: .475661+0000 mgr.x (mgr. 2021-07-31T02:10:31.264 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: 24542) 998 : cephadm [ERR] 2021-07-31T02:10:31.264 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: 2021-07-31T02:10:31.265 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:10:31.265 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:10:31.265 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: yield 2021-07-31T02:10:31.265 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:10:31.265 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:10:31.266 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:10:31.266 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:10:31.266 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:10:31.266 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:10:31.266 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:10:31.267 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:10:31.267 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:10:31.267 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: task.result() 2021-07-31T02:10:31.267 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:10:31.267 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:10:30.476122+0000 mgr.x (mgr.24542) 999 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:10:31.268 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:10:31.268 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:10:31.268 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: yield 2021-07-31T02:10:31.268 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:10:31.269 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:10:31.269 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:10:31.269 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:10:31.269 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:10:31.269 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:10:31.270 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:10:31.270 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:10:31.270 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:10:31.270 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: task.result() 2021-07-31T02:10:31.270 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:10:31.271 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: 2021-07-31T02:10:31.271 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:10:31.271 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: 2021-07-31T02:10:31.271 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:10:31.271 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:10:31.272 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:10:31.272 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:10:31.272 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:10:31.272 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:10:31.273 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:10:31.273 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:10:31.273 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:10:31.273 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:10:31.273 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:10:31.274 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:10:31.274 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:10:31.274 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:10:31.274 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:10:31.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:10:31.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:10:31.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:10:31.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: raise 2021-07-31T02:10:31.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:10:31.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:10:31.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:10:31.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:10:31.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:10:31.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: 2021-07-31T02:10:31.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:10:30.485387+0000 mgr.x (mgr.24542) 1000 : cephadm [ERR] 2021-07-31T02:10:31.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:10:31.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:10:31.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: yield 2021-07-31T02:10:31.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:10:31.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:10:31.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:10:31.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:10:31.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:10:31.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:10:31.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:10:31.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:10:31.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:10:31.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:10:31.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:10:31.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:10:31.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:10:31.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:10:31.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:10:31.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:10:31.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:10:31.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:10:31.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:10:31.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:10:31.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: raise 2021-07-31T02:10:31.283 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:10:31.283 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:10:31.283 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:10:31.283 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:10:31.283 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:10:31.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:10:30.485387+0000 mgr.x (mgr.24542) 1000 : cephadm [ERR] 2021-07-31T02:10:31.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:10:31.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:10:31.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: yield 2021-07-31T02:10:31.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:10:31.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:10:31.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:10:31.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:10:31.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:10:31.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:10:31.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:10:31.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:10:31.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:10:31.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: task.result() 2021-07-31T02:10:31.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:10:31.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:10:30.486165+0000 mgr.x (mgr.24542) 1001 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:10:31.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:10:31.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:10:31.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:10:31.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:10:31.289 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:10:31.289 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:10:31.289 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: task.result() 2021-07-31T02:10:31.289 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:10:31.289 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:10:30.486165+0000 mgr.x (mgr.24542) 1001 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:10:31.290 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:10:31.290 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:10:31.290 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: yield 2021-07-31T02:10:31.291 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:10:31.291 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:10:31.291 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:10:31.291 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:10:31.291 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:10:31.292 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:10:31.292 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:10:31.292 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:10:31.292 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:10:31.292 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: task.result() 2021-07-31T02:10:31.293 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:10:31.293 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: 2021-07-31T02:10:31.293 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:10:31.293 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: 2021-07-31T02:10:31.293 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:10:31.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:10:31.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:10:31.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:10:31.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:10:31.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:10:31.295 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:10:31.295 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:10:31.295 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:10:31.295 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:10:31.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:10:31.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:10:31.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:10:31.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:10:31.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:10:31.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:10:31.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:10:31.297 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: yield 2021-07-31T02:10:31.298 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:10:31.298 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:10:31.298 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:10:31.298 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:10:31.299 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:10:31.299 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:10:31.299 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:10:31.299 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:10:31.299 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:10:31.300 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: task.result() 2021-07-31T02:10:31.300 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:10:31.300 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: 2021-07-31T02:10:31.300 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:10:31.300 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: 2021-07-31T02:10:31.301 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:10:31.301 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:10:31.301 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:10:31.301 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:10:31.302 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:10:31.302 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:10:31.302 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:10:31.302 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:10:31.302 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:10:31.303 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:10:31.303 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:10:31.303 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:10:31.303 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:10:31.303 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:10:31.304 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:10:31.304 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:10:31.304 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:10:31.304 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:10:31.305 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: raise 2021-07-31T02:10:31.305 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:10:31.305 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:10:31.305 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:10:31.305 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:10:31.306 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:10:31.306 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:10:31.306 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: raise 2021-07-31T02:10:31.306 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:10:31.307 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:10:31.307 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:10:31.307 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:10:31.307 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:31 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:10:31.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: audit 2021-07-31T02:10:30.386621+0000 mon.b (mon.2) 185 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:10:31.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: audit 2021-07-31T02:10:30.388061+0000 mon.b ( 2021-07-31T02:10:31.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: mon.2) 186 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:10:31.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: audit 2021-07-31T02:10:30.389103+0000 mon.b (mon.2) 187 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:10:31.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:10:30.389868+0000 mgr.x (mgr.24542) 993 : 2021-07-31T02:10:31.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:10:31.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: cephadm 2021-07- 2021-07-31T02:10:31.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: 31T02:10:30.433410+0000 2021-07-31T02:10:31.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: mgr.x (mgr.24542) 994 : cephadm 2021-07-31T02:10:31.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: [ERR] 2021-07-31T02:10:31.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:10:31.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:10:31.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: yield 2021-07-31T02:10:31.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:10:31.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:10:31.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:10:31.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:10:31.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:10:31.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:10:31.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:10:31.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:10:31.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:10:31.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: task.result() 2021-07-31T02:10:31.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:10:31.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:10:30.433823+0000 mgr.x (mgr.24542) 995 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:10:31.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:10:31.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: 2021-07-31T02:10: 2021-07-31T02:10:31.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: 30.445052+0000 mgr.x (mgr. 2021-07-31T02:10:31.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: 24542) 996 : cephadm [ERR] 2021-07-31T02:10:31.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:10:31.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:10:31.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: yield 2021-07-31T02:10:31.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:10:31.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:10:31.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:10:31.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:10:31.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:10:31.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:10:31.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:10:31.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:10:31.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:10:31.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: task.result() 2021-07-31T02:10:31.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:10:31.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: 2021-07-31T02:10:31.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:10:30.445557+0000 mgr.x (mgr.24542) 997 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:10:31.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: cephadm 2021- 2021-07-31T02:10:31.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: 07-31T02:10:30.475661+0000 2021-07-31T02:10:31.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: mgr.x (mgr.24542) 998 : cephadm 2021-07-31T02:10:31.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: [ERR] 2021-07-31T02:10:31.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:10:31.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:10:31.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: yield 2021-07-31T02:10:31.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:10:31.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:10:31.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:10:31.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:10:31.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:10:31.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:10:31.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:10:31.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:10:31.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:10:31.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: task.result() 2021-07-31T02:10:31.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:10:31.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:10:30.476122+0000 mgr.x (mgr.24542) 999 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:10:31.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:10:31.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:10:31.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: yield 2021-07-31T02:10:31.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:10:31.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:10:31.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:10:31.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:10:31.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:10:31.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:10:31.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:10:31.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:10:31.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:10:31.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: task.result() 2021-07-31T02:10:31.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:10:31.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: 2021-07-31T02:10:31.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:10:31.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: 2021-07-31T02:10:31.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:10:31.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:10:31.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:10:31.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:10:31.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:10:31.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:10:31.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:10:31.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:10:31.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:10:31.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:10:31.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:10:31.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:10:31.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:10:31.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:10:31.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:10:31.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:10:31.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:10:31.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:10:31.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: raise 2021-07-31T02:10:31.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:10:31.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:10:31.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:10:31.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:10:31.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:10:31.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:10:30.485387+0000 mgr.x (mgr.24542) 1000 : cephadm [ERR] 2021-07-31T02:10:31.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:10:31.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:10:31.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: yield 2021-07-31T02:10:31.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:10:31.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:10:31.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:10:31.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:10:31.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:10:31.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:10:31.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:10:31.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:10:31.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:10:31.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: task.result() 2021-07-31T02:10:31.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:10:31.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:10:30.486165+0000 mgr.x (mgr.24542) 1001 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:10:31.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:10:31.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:10:31.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: yield 2021-07-31T02:10:31.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:10:31.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:10:31.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:10:31.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:10:31.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:10:31.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:10:31.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:10:31.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:10:31.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:10:31.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: task.result() 2021-07-31T02:10:31.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:10:31.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: 2021-07-31T02:10:31.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:10:31.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: 2021-07-31T02:10:31.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:10:31.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:10:31.541 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:10:31.541 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:10:31.541 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:10:31.541 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:10:31.541 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:10:31.542 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:10:31.542 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:10:31.542 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:10:31.542 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:10:31.543 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:10:31.543 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:10:31.543 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:10:31.543 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:10:31.543 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:10:31.544 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:10:31.544 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:10:31.544 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: raise 2021-07-31T02:10:31.544 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:10:31.544 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:10:31.545 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:10:31.545 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:10:31.545 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:31 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:10:31.946 INFO:tasks.workunit.client.0.smithi045.stdout: 319: throughput=4.99MB/sec pending data=0 2021-07-31T02:10:31.946 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-D1w5D7P6icRR67P off=5950625 len=1764580 2021-07-31T02:10:31.949 INFO:tasks.workunit.client.0.smithi045.stdout:op 1639 completed, throughput=4.99MB/sec 2021-07-31T02:10:31.950 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-3RurpQlC7J4yxq9 off=7054793 len=91031 2021-07-31T02:10:31.950 INFO:tasks.workunit.client.0.smithi045.stdout:op 1640 completed, throughput=4.99MB/sec 2021-07-31T02:10:31.951 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-_Md2qHraN77ugSu off=3682723 len=739189 2021-07-31T02:10:31.951 INFO:tasks.workunit.client.0.smithi045.stdout:op 1641 completed, throughput=4.99MB/sec 2021-07-31T02:10:31.952 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-XeHjrL4bjURL5kW off=2070703 len=463599 2021-07-31T02:10:31.955 INFO:tasks.workunit.client.0.smithi045.stdout:op 1642 completed, throughput=5MB/sec 2021-07-31T02:10:31.956 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ypYJTeCFfessbhD off=5200290 len=1672755 2021-07-31T02:10:31.957 INFO:tasks.workunit.client.0.smithi045.stdout:op 1643 completed, throughput=5MB/sec 2021-07-31T02:10:32.958 INFO:tasks.workunit.client.0.smithi045.stdout: 320: throughput=4.99MB/sec pending data=0 2021-07-31T02:10:32.958 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-SEhZsSLa_g2rpfi off=1035652 len=329514 2021-07-31T02:10:32.959 INFO:tasks.workunit.client.0.smithi045.stdout:op 1644 completed, throughput=4.99MB/sec 2021-07-31T02:10:32.960 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-gYdZynkc-8HfspE off=6683089 len=1329390 2021-07-31T02:10:32.961 INFO:tasks.workunit.client.0.smithi045.stdout:op 1645 completed, throughput=4.99MB/sec 2021-07-31T02:10:32.962 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-gWHBnT7YpZkMZrM off=299406 len=553941 2021-07-31T02:10:32.963 INFO:tasks.workunit.client.0.smithi045.stdout:op 1646 completed, throughput=4.99MB/sec 2021-07-31T02:10:32.963 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-aE8T3PhJ0cEcpLb off=1578905 len=408558 2021-07-31T02:10:32.964 INFO:tasks.workunit.client.0.smithi045.stdout:op 1647 completed, throughput=4.99MB/sec 2021-07-31T02:10:32.965 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-4dZRmHAZo25ecLS off=3257414 len=3785 2021-07-31T02:10:32.965 INFO:tasks.workunit.client.0.smithi045.stdout:op 1648 completed, throughput=4.99MB/sec 2021-07-31T02:10:32.966 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-2n42mQihIVU7kig off=7456913 len=1954491 2021-07-31T02:10:32.968 INFO:tasks.workunit.client.0.smithi045.stdout:op 1649 completed, throughput=5MB/sec 2021-07-31T02:10:32.968 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-vnXReMDshDRF532 off=6887260 len=893057 2021-07-31T02:10:32.973 INFO:tasks.workunit.client.0.smithi045.stdout:op 1650 completed, throughput=5MB/sec 2021-07-31T02:10:33.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:32 smithi154 conmon[34553]: cluster 2021-07-31T02:10:32.123670+0000 mgr.x (mgr.24542) 1002 : cluster [DBG] pgmap v1571: 9 pgs: 9 active+clean; 127 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 2.4 MiB/s rd, 1.7 MiB/s wr, 4 op/s 2021-07-31T02:10:33.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:32 smithi154 conmon[34553]: 2021-07-31T02:10:33.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:32 smithi045 conmon[39718]: cluster 2021-07-31T02:10:32.123670+0000 mgr.x (mgr 2021-07-31T02:10:33.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:32 smithi045 conmon[39718]: .24542) 1002 : cluster [DBG] pgmap v1571: 9 pgs: 9 active+clean; 127 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 2.4 MiB/s rd, 1.7 MiB/s wr, 4 op/s 2021-07-31T02:10:33.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:32 smithi045 conmon[33655]: cluster 2021-07-31T02:10:32.123670+0000 2021-07-31T02:10:33.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:32 smithi045 conmon[33655]: mgr.x (mgr.24542) 1002 : cluster [DBG] pgmap v1571: 9 pgs: 9 active+clean; 127 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 2.4 MiB/s rd, 1.7 MiB/s wr, 4 op/s 2021-07-31T02:10:33.973 INFO:tasks.workunit.client.0.smithi045.stdout: 321: throughput=4.99MB/sec pending data=0 2021-07-31T02:10:33.974 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-fnnFfHdxU0TA4Gv off=244049 len=1499615 2021-07-31T02:10:33.976 INFO:tasks.workunit.client.0.smithi045.stdout:op 1651 completed, throughput=4.99MB/sec 2021-07-31T02:10:33.976 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-vbfi8vj8o_qcMpZ off=2722962 len=126044 2021-07-31T02:10:33.977 INFO:tasks.workunit.client.0.smithi045.stdout:op 1652 completed, throughput=4.99MB/sec 2021-07-31T02:10:33.977 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-BJ1-QsOLY9wLjH6 off=5893579 len=818313 2021-07-31T02:10:33.982 INFO:tasks.workunit.client.0.smithi045.stdout:op 1653 completed, throughput=4.99MB/sec 2021-07-31T02:10:33.983 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-aUgDX2ssD877crJ off=4338055 len=1672829 2021-07-31T02:10:33.984 INFO:tasks.workunit.client.0.smithi045.stdout:op 1654 completed, throughput=5MB/sec 2021-07-31T02:10:33.985 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-1-dxVp0c5gcgTZt off=3208959 len=1139605 2021-07-31T02:10:33.987 INFO:tasks.workunit.client.0.smithi045.stdout:op 1655 completed, throughput=5MB/sec 2021-07-31T02:10:34.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:10:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:10:33.548Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:10:34.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:10:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:10:33.548Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:10:34.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:10:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:10:33.548Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:10:34.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:34 smithi154 conmon[34553]: cluster 2021-07-31T02:10:34.124106+0000 mgr.x (mgr.24542) 1003 : cluster [DBG] pgmap v1572: 9 pgs: 9 active+clean; 127 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 3.4 MiB/s rd, 2.5 MiB/s wr, 6 op/s 2021-07-31T02:10:34.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:34 smithi154 conmon[34553]: 2021-07-31T02:10:34.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:34 smithi045 conmon[33655]: cluster 2021-07-31T02:10:34.124106+0000 mgr.x 2021-07-31T02:10:34.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:34 smithi045 conmon[33655]: (mgr.24542) 1003 : cluster [DBG] pgmap v1572: 9 pgs: 9 active+clean; 127 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 3.4 MiB/s rd, 2.5 MiB/s wr, 6 op/s 2021-07-31T02:10:34.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:34 smithi045 conmon[39718]: cluster 2021-07-31T02:10:34.124106+0000 mgr.x (mgr.24542) 1003 : 2021-07-31T02:10:34.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:34 smithi045 conmon[39718]: cluster [DBG] pgmap v1572: 9 pgs: 9 active+clean; 127 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 3.4 MiB/s rd, 2.5 MiB/s wr, 6 op/s 2021-07-31T02:10:34.987 INFO:tasks.workunit.client.0.smithi045.stdout: 322: throughput=4.99MB/sec pending data=0 2021-07-31T02:10:34.988 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-cyPMmf0VIo1ZovU off=13073605 len=1879280 2021-07-31T02:10:34.996 INFO:tasks.workunit.client.0.smithi045.stdout:op 1656 completed, throughput=4.99MB/sec 2021-07-31T02:10:34.997 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-OJ1-aI5-eMrZdqE off=1713565 len=724950 2021-07-31T02:10:35.001 INFO:tasks.workunit.client.0.smithi045.stdout:op 1657 completed, throughput=4.99MB/sec 2021-07-31T02:10:35.002 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ZvrrfmitjDeZKaz off=7662076 len=777716 2021-07-31T02:10:35.005 INFO:tasks.workunit.client.0.smithi045.stdout:op 1658 completed, throughput=5MB/sec 2021-07-31T02:10:35.006 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-slrzUkt69J4ItF- off=15413162 len=879084 2021-07-31T02:10:35.008 INFO:tasks.workunit.client.0.smithi045.stdout:op 1659 completed, throughput=5MB/sec 2021-07-31T02:10:35.008 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-MIRfCebh28P63Im off=1374497 len=1296120 2021-07-31T02:10:35.013 INFO:tasks.workunit.client.0.smithi045.stdout:op 1660 completed, throughput=5MB/sec 2021-07-31T02:10:36.014 INFO:tasks.workunit.client.0.smithi045.stdout: 323: throughput=4.99MB/sec pending data=0 2021-07-31T02:10:36.014 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-03faM6jp-4uJRGh off=6503043 len=508283 2021-07-31T02:10:36.015 INFO:tasks.workunit.client.0.smithi045.stdout:op 1661 completed, throughput=4.99MB/sec 2021-07-31T02:10:36.016 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-hQZZQ_QBSoM8fQ6 off=11905790 len=300209 2021-07-31T02:10:36.019 INFO:tasks.workunit.client.0.smithi045.stdout:op 1662 completed, throughput=4.99MB/sec 2021-07-31T02:10:36.019 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-2LBUHfYsq7I3m1d off=7405715 len=1925437 2021-07-31T02:10:36.028 INFO:tasks.workunit.client.0.smithi045.stdout:op 1663 completed, throughput=4.99MB/sec 2021-07-31T02:10:36.029 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-kH9FcDL6a0oA06H off=5907729 len=1352047 2021-07-31T02:10:36.030 INFO:tasks.workunit.client.0.smithi045.stdout:op 1664 completed, throughput=5MB/sec 2021-07-31T02:10:36.030 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-8V11C5n1GlK7c79 off=5925045 len=433803 2021-07-31T02:10:36.031 INFO:tasks.workunit.client.0.smithi045.stdout:op 1665 completed, throughput=5MB/sec 2021-07-31T02:10:36.031 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Y5Vw2co-chrumee off=4009987 len=533029 2021-07-31T02:10:36.034 INFO:tasks.workunit.client.0.smithi045.stdout:op 1666 completed, throughput=5MB/sec 2021-07-31T02:10:36.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:36 smithi154 conmon[34553]: cluster 2021-07-31T02:10:36.124609+0000 mgr.x (mgr.24542) 1004 : cluster [DBG] pgmap v1573: 9 pgs: 9 active+clean; 127 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 2.1 MiB/s rd, 1.7 MiB/s wr, 3 op/s 2021-07-31T02:10:36.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:36 smithi045 conmon[33655]: cluster 2021-07-31T02:10:36.124609+0000 mgr.x (mgr. 2021-07-31T02:10:36.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:36 smithi045 conmon[33655]: 24542) 1004 : cluster [DBG] pgmap v1573: 9 pgs: 9 active+clean; 127 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 2.1 MiB/s rd, 1.7 MiB/s wr, 3 op/s 2021-07-31T02:10:36.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:36 smithi045 conmon[39718]: cluster 2021-07-31T02:10:36.124609+0000 mgr.x 2021-07-31T02:10:36.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:36 smithi045 conmon[39718]: (mgr.24542) 1004 : cluster [DBG] pgmap v1573: 9 pgs: 9 active+clean; 127 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 2.1 MiB/s rd, 1.7 MiB/s wr, 3 op/s 2021-07-31T02:10:37.035 INFO:tasks.workunit.client.0.smithi045.stdout: 324: throughput=4.99MB/sec pending data=0 2021-07-31T02:10:37.035 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-DdHPICOyRQrJv0q off=2683196 len=77415 2021-07-31T02:10:37.037 INFO:tasks.workunit.client.0.smithi045.stdout:op 1667 completed, throughput=4.99MB/sec 2021-07-31T02:10:37.038 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-C-mMYfsJvCRPHgU off=13417950 len=1286563 2021-07-31T02:10:37.039 INFO:tasks.workunit.client.0.smithi045.stdout:op 1668 completed, throughput=4.99MB/sec 2021-07-31T02:10:37.039 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-AOUcMh_yLPthuUg off=5603399 len=1654642 2021-07-31T02:10:37.048 INFO:tasks.workunit.client.0.smithi045.stdout:op 1669 completed, throughput=4.99MB/sec 2021-07-31T02:10:37.049 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-75Jgfh_moBaku3V off=4061858 len=187850 2021-07-31T02:10:37.049 INFO:tasks.workunit.client.0.smithi045.stdout:op 1670 completed, throughput=4.99MB/sec 2021-07-31T02:10:37.049 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-1cC2-AR_ZpdcCg0 off=22230483 len=1363842 2021-07-31T02:10:37.058 INFO:tasks.workunit.client.0.smithi045.stdout:op 1671 completed, throughput=5MB/sec 2021-07-31T02:10:37.058 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-JIb0L-SDfImjQHv off=2759825 len=431115 2021-07-31T02:10:37.061 INFO:tasks.workunit.client.0.smithi045.stdout:op 1672 completed, throughput=5MB/sec 2021-07-31T02:10:37.062 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-I4EIaWf7X3lvjZO off=854008 len=90509 2021-07-31T02:10:37.062 INFO:tasks.workunit.client.0.smithi045.stdout:op 1673 completed, throughput=5MB/sec 2021-07-31T02:10:38.062 INFO:tasks.workunit.client.0.smithi045.stdout: 325: throughput=4.98MB/sec pending data=0 2021-07-31T02:10:38.063 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-WsckOWBbg_ZQC4u off=16389648 len=282144 2021-07-31T02:10:38.066 INFO:tasks.workunit.client.0.smithi045.stdout:op 1674 completed, throughput=4.99MB/sec 2021-07-31T02:10:38.066 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-QgI4gBsKmGuo_K4 off=9456724 len=1152815 2021-07-31T02:10:38.071 INFO:tasks.workunit.client.0.smithi045.stdout:op 1675 completed, throughput=4.99MB/sec 2021-07-31T02:10:38.071 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-nvwtvcO1FE3bSjk off=475323 len=690970 2021-07-31T02:10:38.073 INFO:tasks.workunit.client.0.smithi045.stdout:op 1676 completed, throughput=4.99MB/sec 2021-07-31T02:10:38.073 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-K7JAHGzmbpiTlVl off=669796 len=370258 2021-07-31T02:10:38.074 INFO:tasks.workunit.client.0.smithi045.stdout:op 1677 completed, throughput=4.99MB/sec 2021-07-31T02:10:38.074 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-d6lNhKb6pwBZW9z off=1156271 len=1454826 2021-07-31T02:10:38.076 INFO:tasks.workunit.client.0.smithi045.stdout:op 1678 completed, throughput=5MB/sec 2021-07-31T02:10:38.077 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-QH3i59TWn3fOhV5 off=561436 len=421286 2021-07-31T02:10:38.080 INFO:tasks.workunit.client.0.smithi045.stdout:op 1679 completed, throughput=5MB/sec 2021-07-31T02:10:38.080 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-GZflFLPPgITXUKa off=812307 len=880237 2021-07-31T02:10:38.081 INFO:tasks.workunit.client.0.smithi045.stdout:op 1680 completed, throughput=5MB/sec 2021-07-31T02:10:38.081 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-nXEXw69-qZS80WA off=17406968 len=5741 2021-07-31T02:10:38.081 INFO:tasks.workunit.client.0.smithi045.stdout:op 1681 completed, throughput=5MB/sec 2021-07-31T02:10:38.082 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-FoDXRLnsS-sKVSV off=1587247 len=2031896 2021-07-31T02:10:38.091 INFO:tasks.workunit.client.0.smithi045.stdout:op 1682 completed, throughput=5.01MB/sec 2021-07-31T02:10:38.222 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:10:38.499 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:10:38.499 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 25m ago 26m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:10:38.500 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 23m ago 26m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:10:38.500 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 23m ago 29m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:10:38.500 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 25m ago 31m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:10:38.500 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 25m ago 31m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:10:38.500 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 23m ago 29m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:10:38.501 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 25m ago 29m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:10:38.501 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 25m ago 26m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:10:38.501 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 23m ago 26m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:10:38.501 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 25m ago 29m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:10:38.502 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 25m ago 28m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:10:38.502 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 25m ago 28m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:10:38.502 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 25m ago 28m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:10:38.502 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 23m ago 27m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:10:38.502 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 23m ago 27m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:10:38.503 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 23m ago 27m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:10:38.503 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 23m ago 26m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:10:38.503 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 23m ago 26m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:10:38.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:38 smithi154 conmon[34553]: cluster 2021-07-31T02:10:38.125242+0000 mgr.x (mgr.24542) 1005 : cluster [DBG] pgmap v1574: 9 pgs: 9 active+clean; 127 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 2.1 MiB/s rd, 1.7 MiB/s wr, 3 op/s 2021-07-31T02:10:38.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:38 smithi045 conmon[33655]: cluster 2021-07-31T02:10:38. 2021-07-31T02:10:38.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:38 smithi045 conmon[33655]: 125242+0000 mgr.x (mgr.24542) 1005 : cluster [DBG] pgmap v1574: 9 pgs: 9 active+clean; 127 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 2.1 MiB/s rd, 1.7 MiB/s wr, 3 op/s 2021-07-31T02:10:38.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:38 smithi045 conmon[39718]: cluster 2021-07-31T02:10 2021-07-31T02:10:38.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:38 smithi045 conmon[39718]: :38.125242+0000 mgr.x (mgr.24542) 1005 : cluster [DBG] pgmap v1574: 9 pgs: 9 active+clean; 127 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 2.1 MiB/s rd, 1.7 MiB/s wr, 3 op/s 2021-07-31T02:10:38.837 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:10:38.837 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:10:38.837 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:10:38.837 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:10:38.838 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:10:38.838 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:10:38.838 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:10:38.838 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:10:38.839 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:10:38.839 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:10:38.839 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:10:38.839 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:10:38.840 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:10:38.840 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:10:38.840 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:10:38.840 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:10:38.840 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:10:39.091 INFO:tasks.workunit.client.0.smithi045.stdout: 326: throughput=4.99MB/sec pending data=0 2021-07-31T02:10:39.092 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-_wdOAF1Hk8loRAH off=670593 len=1272413 2021-07-31T02:10:39.099 INFO:tasks.workunit.client.0.smithi045.stdout:op 1683 completed, throughput=4.99MB/sec 2021-07-31T02:10:39.099 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-iac-f_KsNQls-1M off=8746217 len=903131 2021-07-31T02:10:39.104 INFO:tasks.workunit.client.0.smithi045.stdout:op 1684 completed, throughput=5MB/sec 2021-07-31T02:10:39.104 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-5Oyk-jp9jrkKvHT off=9569967 len=820877 2021-07-31T02:10:39.104 INFO:tasks.workunit.client.0.smithi045.stdout:op 1685 completed, throughput=5MB/sec 2021-07-31T02:10:39.105 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-s78ElQYhbaFV4Sr off=10218546 len=1428195 2021-07-31T02:10:39.110 INFO:tasks.workunit.client.0.smithi045.stdout:op 1686 completed, throughput=5MB/sec 2021-07-31T02:10:39.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:39 smithi154 conmon[34553]: audit 2021-07-31T02:10:38.209998+0000 mgr.x (mgr.24542) 1006 : audit [DBG] from='client.15915 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:10:39.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:39 smithi154 conmon[34553]: audit 2021-07-31T02:10:38.490840 2021-07-31T02:10:39.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:39 smithi154 conmon[34553]: +0000 mgr.x (mgr.24542) 1007 : audit [DBG] from='client.15921 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:10:39.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:39 smithi154 conmon[34553]: audit 2021-07-31T02:10:38.837132+0000 mon.a (mon.0) 1867 : audit [DBG] from='client.? 172.21.15.45:0/3073009326' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:10:39.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:39 smithi045 conmon[33655]: audit 2021-07-31T02:10:38.209998+0000 mgr.x (mgr.24542) 2021-07-31T02:10:39.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:39 smithi045 conmon[33655]: 1006 : audit [DBG] from='client.15915 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:10:39.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:39 smithi045 conmon[33655]: audit 2021-07-31T02:10:38.490840+0000 mgr.x (mgr.24542) 1007 : audit [DBG] from='client.15921 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:10:39.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:39 smithi045 conmon[33655]: audit 2021- 2021-07-31T02:10:39.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:39 smithi045 conmon[33655]: 07-31T02:10:38.837132+0000 mon.a (mon.0) 1867 : audit [DBG] from='client.? 172.21.15.45:0/3073009326' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:10:39.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:39 smithi045 conmon[39718]: audit 2021-07-31T02: 2021-07-31T02:10:39.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:39 smithi045 conmon[39718]: 10:38.209998+0000 mgr.x (mgr.24542) 1006 : audit [DBG] from='client.15915 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:10:39.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:39 smithi045 conmon[39718]: audit 2021 2021-07-31T02:10:39.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:39 smithi045 conmon[39718]: -07-31T02:10:38 2021-07-31T02:10:39.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:39 smithi045 conmon[39718]: .490840+0000 mgr.x (mgr. 2021-07-31T02:10:39.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:39 smithi045 conmon[39718]: 24542) 1007 : audit [DBG] 2021-07-31T02:10:39.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:39 smithi045 conmon[39718]: from='client.15921 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:10:39.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:39 smithi045 conmon[39718]: audit 2021 2021-07-31T02:10:39.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:39 smithi045 conmon[39718]: -07-31T02:10 2021-07-31T02:10:39.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:39 smithi045 conmon[39718]: :38.837132+0000 mon.a ( 2021-07-31T02:10:39.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:39 smithi045 conmon[39718]: mon.0) 1867 : audit 2021-07-31T02:10:39.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:39 smithi045 conmon[39718]: [DBG] from='client.? 172.21.15.45:0/3073009326' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:10:40.111 INFO:tasks.workunit.client.0.smithi045.stdout: 327: throughput=4.99MB/sec pending data=0 2021-07-31T02:10:40.112 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-K7WARNxKnwbqbHX off=472604 len=825812 2021-07-31T02:10:40.113 INFO:tasks.workunit.client.0.smithi045.stdout:op 1687 completed, throughput=4.99MB/sec 2021-07-31T02:10:40.113 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-pdEi6gWebN0k_MX off=5341567 len=526406 2021-07-31T02:10:40.117 INFO:tasks.workunit.client.0.smithi045.stdout:op 1688 completed, throughput=4.99MB/sec 2021-07-31T02:10:40.117 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj--ZgHAremJDf1XMV off=2690774 len=1340629 2021-07-31T02:10:40.119 INFO:tasks.workunit.client.0.smithi045.stdout:op 1689 completed, throughput=5MB/sec 2021-07-31T02:10:40.119 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-sw3reTsJLK4s7iS off=6726025 len=1351364 2021-07-31T02:10:40.125 INFO:tasks.workunit.client.0.smithi045.stdout:op 1690 completed, throughput=5MB/sec 2021-07-31T02:10:40.126 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-09oqWEd8d5m2sz_ off=3273048 len=1643555 2021-07-31T02:10:40.129 INFO:tasks.workunit.client.0.smithi045.stdout:op 1691 completed, throughput=5MB/sec 2021-07-31T02:10:40.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:40 smithi154 conmon[34553]: cluster 2021-07-31T02:10:40.125801+0000 mgr.x (mgr.24542) 1008 : cluster [DBG] pgmap v1575: 9 pgs: 9 active+clean; 127 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.9 MiB/s wr, 6 op/s 2021-07-31T02:10:40.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:40 smithi154 conmon[34553]: 2021-07-31T02:10:40.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:40 smithi045 conmon[33655]: cluster 2021-07-31T02:10:40.125801+0000 mgr.x (mgr.24542) 1008 : cluster [DBG] pgmap v1575: 9 pgs: 9 active+clean; 127 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.9 MiB/s wr, 6 op/s 2021-07-31T02:10:40.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:40 smithi045 conmon[39718]: cluster 2021-07-31T02:10:40.125801+0000 mgr.x 2021-07-31T02:10:40.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:40 smithi045 conmon[39718]: (mgr.24542) 1008 : cluster [DBG] pgmap v1575: 9 pgs: 9 active+clean; 127 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.9 MiB/s wr, 6 op/s 2021-07-31T02:10:41.130 INFO:tasks.workunit.client.0.smithi045.stdout: 328: throughput=4.99MB/sec pending data=0 2021-07-31T02:10:41.130 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-tPDd0kBILAVFfwk off=5854507 len=803793 2021-07-31T02:10:41.131 INFO:tasks.workunit.client.0.smithi045.stdout:op 1692 completed, throughput=4.99MB/sec 2021-07-31T02:10:41.131 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-9vUc7wuPnuySMxI off=1826924 len=1534691 2021-07-31T02:10:41.138 INFO:tasks.workunit.client.0.smithi045.stdout:op 1693 completed, throughput=5MB/sec 2021-07-31T02:10:41.140 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-pe0TgQZ4WWdpIeo off=5556462 len=1501032 2021-07-31T02:10:41.145 INFO:tasks.workunit.client.0.smithi045.stdout:op 1694 completed, throughput=5MB/sec 2021-07-31T02:10:41.145 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-IpyF5hEFjsPRiGx off=12564031 len=1588529 2021-07-31T02:10:41.148 INFO:tasks.workunit.client.0.smithi045.stdout:op 1695 completed, throughput=5MB/sec 2021-07-31T02:10:42.148 INFO:tasks.workunit.client.0.smithi045.stdout: 329: throughput=4.99MB/sec pending data=0 2021-07-31T02:10:42.149 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-VDB5HwhuDegF5QB off=4910893 len=2069378 2021-07-31T02:10:42.153 INFO:tasks.workunit.client.0.smithi045.stdout:op 1696 completed, throughput=4.99MB/sec 2021-07-31T02:10:42.153 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-DwdhZACx0z9wIEa off=7145776 len=1275221 2021-07-31T02:10:42.159 INFO:tasks.workunit.client.0.smithi045.stdout:op 1697 completed, throughput=5MB/sec 2021-07-31T02:10:42.159 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-hPvwkOKN9xdnI_b off=19960465 len=1416367 2021-07-31T02:10:42.167 INFO:tasks.workunit.client.0.smithi045.stdout:op 1698 completed, throughput=5MB/sec 2021-07-31T02:10:43.167 INFO:tasks.workunit.client.0.smithi045.stdout: 330: throughput=4.99MB/sec pending data=0 2021-07-31T02:10:43.167 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-cZGtrt9WKg4q0Mn off=1243795 len=1490532 2021-07-31T02:10:43.174 INFO:tasks.workunit.client.0.smithi045.stdout:op 1699 completed, throughput=4.99MB/sec 2021-07-31T02:10:43.174 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-i7PdET8PY7I5INq off=16764387 len=2071437 2021-07-31T02:10:43.184 INFO:tasks.workunit.client.0.smithi045.stdout:op 1700 completed, throughput=5MB/sec 2021-07-31T02:10:43.184 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-uqqz7y-j_f_3RS0 off=21991968 len=1031931 2021-07-31T02:10:43.186 INFO:tasks.workunit.client.0.smithi045.stdout:op 1701 completed, throughput=5MB/sec 2021-07-31T02:10:43.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:42 smithi154 conmon[34553]: cluster 2021-07-31T02:10:42.126176+0000 mgr.x (mgr.24542) 1009 : cluster [DBG] pgmap v1576: 9 pgs: 9 active+clean; 127 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2021-07-31T02:10:43.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:42 smithi154 conmon[34553]: 2021-07-31T02:10:43.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:42 smithi045 conmon[33655]: cluster 2021-07-31T02:10:42.126176+0000 2021-07-31T02:10:43.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:42 smithi045 conmon[33655]: mgr.x (mgr.24542) 1009 : cluster [DBG] pgmap v1576: 9 pgs: 9 active+clean; 127 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2021-07-31T02:10:43.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:42 smithi045 conmon[39718]: cluster 2021-07-31T02:10:42.126176+0000 mgr.x 2021-07-31T02:10:43.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:42 smithi045 conmon[39718]: (mgr.24542) 1009 : cluster [DBG] pgmap v1576: 9 pgs: 9 active+clean; 127 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2021-07-31T02:10:44.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:10:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:10:43.548Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:10:44.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:10:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:10:43.548Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:10:44.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:10:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:10:43.548Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:10:44.187 INFO:tasks.workunit.client.0.smithi045.stdout: 331: throughput=4.99MB/sec pending data=0 2021-07-31T02:10:44.187 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Kva2NgRjnd4wzXf off=6378516 len=874794 2021-07-31T02:10:44.192 INFO:tasks.workunit.client.0.smithi045.stdout:op 1702 completed, throughput=4.99MB/sec 2021-07-31T02:10:44.192 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-86GGN3C7BBLGT04 off=18002216 len=1599925 2021-07-31T02:10:44.201 INFO:tasks.workunit.client.0.smithi045.stdout:op 1703 completed, throughput=4.99MB/sec 2021-07-31T02:10:44.201 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-uLS3LKDh3HrrhJr off=15480422 len=2066705 2021-07-31T02:10:44.203 INFO:tasks.workunit.client.0.smithi045.stdout:op 1704 completed, throughput=5MB/sec 2021-07-31T02:10:44.203 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Rg_8M6GPmqmgG2J off=3397971 len=1375975 2021-07-31T02:10:44.210 INFO:tasks.workunit.client.0.smithi045.stdout:op 1705 completed, throughput=5MB/sec 2021-07-31T02:10:44.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:44 smithi154 conmon[34553]: cluster 2021-07-31T02:10:44.126755+0000 mgr.x (mgr.24542) 1010 : cluster [DBG] pgmap v1577: 9 pgs: 9 active+clean; 127 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 2.8 MiB/s rd, 3.2 MiB/s wr, 6 op/s 2021-07-31T02:10:44.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:44 smithi045 conmon[39718]: cluster 2021-07-31T02:10:44.126755+0000 mgr.x 2021-07-31T02:10:44.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:44 smithi045 conmon[39718]: (mgr.24542) 1010 : cluster [DBG] pgmap v1577: 9 pgs: 9 active+clean; 127 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 2.8 MiB/s rd, 3.2 MiB/s wr, 6 op/s 2021-07-31T02:10:44.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:44 smithi045 conmon[33655]: cluster 2021-07-31T02:10:44.126755+0000 2021-07-31T02:10:44.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:44 smithi045 conmon[33655]: mgr.x (mgr.24542) 1010 : cluster [DBG] pgmap v1577: 9 pgs: 9 active+clean; 127 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 2.8 MiB/s rd, 3.2 MiB/s wr, 6 op/s 2021-07-31T02:10:45.210 INFO:tasks.workunit.client.0.smithi045.stdout: 332: throughput=4.99MB/sec pending data=0 2021-07-31T02:10:45.211 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-1iHtBF0-MhXsPxU off=3867833 len=1915077 2021-07-31T02:10:45.219 INFO:tasks.workunit.client.0.smithi045.stdout:op 1706 completed, throughput=4.99MB/sec 2021-07-31T02:10:45.219 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-3yi8KqybK40WaDy off=9750445 len=1472588 2021-07-31T02:10:45.224 INFO:tasks.workunit.client.0.smithi045.stdout:op 1707 completed, throughput=5MB/sec 2021-07-31T02:10:45.225 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-EBERxDJafG8EdDl off=5584831 len=1129118 2021-07-31T02:10:45.232 INFO:tasks.workunit.client.0.smithi045.stdout:op 1708 completed, throughput=5MB/sec 2021-07-31T02:10:45.233 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj--IalQQMVTg_jU5v off=11647644 len=513455 2021-07-31T02:10:45.233 INFO:tasks.workunit.client.0.smithi045.stdout:op 1709 completed, throughput=5MB/sec 2021-07-31T02:10:46.234 INFO:tasks.workunit.client.0.smithi045.stdout: 333: throughput=4.99MB/sec pending data=0 2021-07-31T02:10:46.234 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-wQQIVBeoDVDEpFq off=2304651 len=858898 2021-07-31T02:10:46.236 INFO:tasks.workunit.client.0.smithi045.stdout:op 1710 completed, throughput=4.99MB/sec 2021-07-31T02:10:46.236 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-RpVG6bPwSBhDe5- off=7673909 len=394752 2021-07-31T02:10:46.240 INFO:tasks.workunit.client.0.smithi045.stdout:op 1711 completed, throughput=4.99MB/sec 2021-07-31T02:10:46.240 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ABYmoS1ViNGX0Bg off=1700576 len=179097 2021-07-31T02:10:46.241 INFO:tasks.workunit.client.0.smithi045.stdout:op 1712 completed, throughput=4.99MB/sec 2021-07-31T02:10:46.241 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-j3GPAwvIev55Npo off=1553497 len=1179825 2021-07-31T02:10:46.248 INFO:tasks.workunit.client.0.smithi045.stdout:op 1713 completed, throughput=4.99MB/sec 2021-07-31T02:10:46.248 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-qEEqAEmOyAeEfkk off=6376764 len=1943610 2021-07-31T02:10:46.251 INFO:tasks.workunit.client.0.smithi045.stdout:op 1714 completed, throughput=5MB/sec 2021-07-31T02:10:46.252 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-qjGB00ijLa8QEFo off=3256214 len=1534100 2021-07-31T02:10:46.253 INFO:tasks.workunit.client.0.smithi045.stdout:op 1715 completed, throughput=5MB/sec 2021-07-31T02:10:46.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:46 smithi154 conmon[34553]: cluster 2021-07-31T02:10:46.127299+0000 mgr.x (mgr.24542) 1011 : cluster [DBG] 2021-07-31T02:10:46.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:46 smithi154 conmon[34553]: pgmap v1578: 9 pgs: 9 active+clean; 127 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:10:46.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:46 smithi045 conmon[33655]: cluster 2021-07-31T02:10:46.127299+0000 2021-07-31T02:10:46.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:46 smithi045 conmon[33655]: mgr.x (mgr.24542) 1011 : cluster [DBG] pgmap v1578: 9 pgs: 9 active+clean; 127 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:10:46.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:46 smithi045 conmon[39718]: cluster 2021-07-31T02:10:46.127299+0000 2021-07-31T02:10:46.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:46 smithi045 conmon[39718]: mgr.x (mgr.24542) 1011 : cluster [DBG] pgmap v1578: 9 pgs: 9 active+clean; 127 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:10:47.254 INFO:tasks.workunit.client.0.smithi045.stdout: 334: throughput=4.99MB/sec pending data=0 2021-07-31T02:10:47.254 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-BXMuN1bUYR9vJ9z off=1496950 len=1422393 2021-07-31T02:10:47.256 INFO:tasks.workunit.client.0.smithi045.stdout:op 1716 completed, throughput=4.99MB/sec 2021-07-31T02:10:47.257 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-enzLWDhHg-8LRs9 off=7832173 len=880958 2021-07-31T02:10:47.259 INFO:tasks.workunit.client.0.smithi045.stdout:op 1717 completed, throughput=4.99MB/sec 2021-07-31T02:10:47.259 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj--ppWpItqGYW6J9x off=616611 len=1077899 2021-07-31T02:10:47.261 INFO:tasks.workunit.client.0.smithi045.stdout:op 1718 completed, throughput=5MB/sec 2021-07-31T02:10:47.261 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Q33yQgAtKSZZfkb off=806572 len=1550539 2021-07-31T02:10:47.263 INFO:tasks.workunit.client.0.smithi045.stdout:op 1719 completed, throughput=5MB/sec 2021-07-31T02:10:48.264 INFO:tasks.workunit.client.0.smithi045.stdout: 335: throughput=4.99MB/sec pending data=0 2021-07-31T02:10:48.264 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-jBdb4QbRvPXJETa off=8447270 len=1417368 2021-07-31T02:10:48.265 INFO:tasks.workunit.client.0.smithi045.stdout:op 1720 completed, throughput=4.99MB/sec 2021-07-31T02:10:48.265 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-3svhiHl7KlgWXvx off=3981794 len=1651519 2021-07-31T02:10:48.267 INFO:tasks.workunit.client.0.smithi045.stdout:op 1721 completed, throughput=5MB/sec 2021-07-31T02:10:48.267 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-MrA5sXPGk29vzz9 off=5182332 len=718690 2021-07-31T02:10:48.269 INFO:tasks.workunit.client.0.smithi045.stdout:op 1722 completed, throughput=5MB/sec 2021-07-31T02:10:48.269 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-1qNYlU3qqq5vALh off=3944323 len=1561047 2021-07-31T02:10:48.275 INFO:tasks.workunit.client.0.smithi045.stdout:op 1723 completed, throughput=5MB/sec 2021-07-31T02:10:48.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:48 smithi154 conmon[34553]: cluster 2021-07-31T02:10:48.127859+0000 mgr.x (mgr.24542) 1012 : cluster [DBG] 2021-07-31T02:10:48.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:48 smithi154 conmon[34553]: pgmap v1579: 9 pgs: 9 active+clean; 127 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:10:48.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:48 smithi045 conmon[39718]: cluster 2021-07-31T02:10:48.127859+0000 mgr.x (mgr.24542 2021-07-31T02:10:48.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:48 smithi045 conmon[39718]: ) 1012 : cluster [DBG] pgmap v1579: 9 pgs: 9 active+clean; 127 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:10:48.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:48 smithi045 conmon[33655]: cluster 2021-07-31T02:10:48.127859+0000 2021-07-31T02:10:48.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:48 smithi045 conmon[33655]: mgr.x (mgr.24542) 1012 : cluster [DBG] pgmap v1579: 9 pgs: 9 active+clean; 127 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:10:49.276 INFO:tasks.workunit.client.0.smithi045.stdout: 336: throughput=4.99MB/sec pending data=0 2021-07-31T02:10:49.276 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-exZrJ8ucv5HyAk_ off=552935 len=86387 2021-07-31T02:10:49.279 INFO:tasks.workunit.client.0.smithi045.stdout:op 1724 completed, throughput=4.99MB/sec 2021-07-31T02:10:49.279 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-N4iLjP5ggojSRfP off=5136450 len=1077352 2021-07-31T02:10:49.281 INFO:tasks.workunit.client.0.smithi045.stdout:op 1725 completed, throughput=4.99MB/sec 2021-07-31T02:10:49.281 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Aygq1oVcGB-bhrL off=200637 len=166606 2021-07-31T02:10:49.282 INFO:tasks.workunit.client.0.smithi045.stdout:op 1726 completed, throughput=4.99MB/sec 2021-07-31T02:10:49.282 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-uwmXcdm_cQHicdS off=373017 len=1960349 2021-07-31T02:10:49.286 INFO:tasks.workunit.client.0.smithi045.stdout:op 1727 completed, throughput=5MB/sec 2021-07-31T02:10:49.286 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-w8CB1nzZbh8mX-r off=426262 len=1064014 2021-07-31T02:10:49.288 INFO:tasks.workunit.client.0.smithi045.stdout:op 1728 completed, throughput=5MB/sec 2021-07-31T02:10:49.288 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-QORCCETYgtFoF9C off=7453328 len=812760 2021-07-31T02:10:49.289 INFO:tasks.workunit.client.0.smithi045.stdout:op 1729 completed, throughput=5MB/sec 2021-07-31T02:10:50.289 INFO:tasks.workunit.client.0.smithi045.stdout: 337: throughput=4.99MB/sec pending data=0 2021-07-31T02:10:50.290 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-lGUmt8Vcw3q-16- off=5680109 len=656303 2021-07-31T02:10:50.294 INFO:tasks.workunit.client.0.smithi045.stdout:op 1730 completed, throughput=4.99MB/sec 2021-07-31T02:10:50.295 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-UtpfLzFJLMiK91N off=15446424 len=100315 2021-07-31T02:10:50.297 INFO:tasks.workunit.client.0.smithi045.stdout:op 1731 completed, throughput=4.99MB/sec 2021-07-31T02:10:50.297 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-YXnAGe93Gi5at3i off=4849373 len=685489 2021-07-31T02:10:50.301 INFO:tasks.workunit.client.0.smithi045.stdout:op 1732 completed, throughput=4.99MB/sec 2021-07-31T02:10:50.301 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-N7zHEBkKBMJ_LAv off=10990499 len=1945274 2021-07-31T02:10:50.310 INFO:tasks.workunit.client.0.smithi045.stdout:op 1733 completed, throughput=5MB/sec 2021-07-31T02:10:50.310 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-H0bXTlNfX7cVUuZ off=13297182 len=295582 2021-07-31T02:10:50.311 INFO:tasks.workunit.client.0.smithi045.stdout:op 1734 completed, throughput=5MB/sec 2021-07-31T02:10:50.311 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-xxLOvDXRGNZnUed off=8313090 len=1653418 2021-07-31T02:10:50.318 INFO:tasks.workunit.client.0.smithi045.stdout:op 1735 completed, throughput=5MB/sec 2021-07-31T02:10:50.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:50 smithi154 conmon[34553]: cluster 2021-07-31T02:10:50.128729+0000 mgr.x (mgr.24542) 1013 : cluster [DBG] pgmap v1580: 9 pgs: 9 active+clean; 127 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 3.1 MiB/s rd, 3.3 MiB/s wr, 6 op/s 2021-07-31T02:10:50.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:50 smithi154 conmon[34553]: 2021-07-31T02:10:50.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:50 smithi045 conmon[33655]: cluster 2021-07-31T02:10:50.128729+0000 2021-07-31T02:10:50.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:50 smithi045 conmon[33655]: mgr.x (mgr.24542) 1013 : cluster [DBG] pgmap v1580: 9 pgs: 9 active+clean; 127 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 3.1 MiB/s rd, 3.3 MiB/s wr, 6 op/s 2021-07-31T02:10:50.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:50 smithi045 conmon[39718]: cluster 2021-07-31T02:10:50.128729+0000 mgr.x 2021-07-31T02:10:50.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:50 smithi045 conmon[39718]: (mgr.24542) 1013 : cluster [DBG] pgmap v1580: 9 pgs: 9 active+clean; 127 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 3.1 MiB/s rd, 3.3 MiB/s wr, 6 op/s 2021-07-31T02:10:51.318 INFO:tasks.workunit.client.0.smithi045.stdout: 338: throughput=4.99MB/sec pending data=0 2021-07-31T02:10:51.319 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ZaoGhqY8O1e0Z-O off=3959566 len=1641506 2021-07-31T02:10:51.322 INFO:tasks.workunit.client.0.smithi045.stdout:op 1736 completed, throughput=4.99MB/sec 2021-07-31T02:10:51.322 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-IQBvJWR-tXOXfyh off=607007 len=293443 2021-07-31T02:10:51.325 INFO:tasks.workunit.client.0.smithi045.stdout:op 1737 completed, throughput=4.99MB/sec 2021-07-31T02:10:51.325 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-_Ys3XRLIe8yDOaE off=5782124 len=42862 2021-07-31T02:10:51.327 INFO:tasks.workunit.client.0.smithi045.stdout:op 1738 completed, throughput=4.99MB/sec 2021-07-31T02:10:51.328 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-So79PsGYi6511Ef off=12297738 len=1625334 2021-07-31T02:10:51.335 INFO:tasks.workunit.client.0.smithi045.stdout:op 1739 completed, throughput=5MB/sec 2021-07-31T02:10:51.335 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-RWp8k6dADXPKWaA off=2038977 len=741478 2021-07-31T02:10:51.336 INFO:tasks.workunit.client.0.smithi045.stdout:op 1740 completed, throughput=5MB/sec 2021-07-31T02:10:51.336 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-yuRvTX6rOs7y2QV off=3518954 len=1960723 2021-07-31T02:10:51.339 INFO:tasks.workunit.client.0.smithi045.stdout:op 1741 completed, throughput=5MB/sec 2021-07-31T02:10:52.340 INFO:tasks.workunit.client.0.smithi045.stdout: 339: throughput=4.99MB/sec pending data=0 2021-07-31T02:10:52.340 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-58AV5KWWReLvucb off=2621120 len=1106089 2021-07-31T02:10:52.346 INFO:tasks.workunit.client.0.smithi045.stdout:op 1742 completed, throughput=4.99MB/sec 2021-07-31T02:10:52.346 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-KsI7kbeU-_5hNVT off=8058351 len=900355 2021-07-31T02:10:52.351 INFO:tasks.workunit.client.0.smithi045.stdout:op 1743 completed, throughput=4.99MB/sec 2021-07-31T02:10:52.352 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-pvEPTGuEdRZiYYM off=10810527 len=1268333 2021-07-31T02:10:52.353 INFO:tasks.workunit.client.0.smithi045.stdout:op 1744 completed, throughput=5MB/sec 2021-07-31T02:10:52.353 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-IwOiML-DhV0dkkG off=2748020 len=439175 2021-07-31T02:10:52.355 INFO:tasks.workunit.client.0.smithi045.stdout:op 1745 completed, throughput=5MB/sec 2021-07-31T02:10:52.355 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-v49kuv88Gv7ve7r off=21188949 len=1019958 2021-07-31T02:10:52.360 INFO:tasks.workunit.client.0.smithi045.stdout:op 1746 completed, throughput=5MB/sec 2021-07-31T02:10:53.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:52 smithi154 conmon[34553]: cluster 2021-07-31T02:10:52.129287+0000 mgr.x (mgr.24542) 1014 : cluster [DBG] pgmap v1581: 9 pgs: 9 active+clean; 127 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.1 MiB/s wr, 3 op/s 2021-07-31T02:10:53.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:52 smithi045 conmon[33655]: cluster 2021-07-31T02:10:52.129287+0000 mgr.x (mgr.24542) 1014 2021-07-31T02:10:53.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:52 smithi045 conmon[33655]: : cluster [DBG] pgmap v1581: 9 pgs: 9 active+clean; 127 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.1 MiB/s wr, 3 op/s 2021-07-31T02:10:53.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:52 smithi045 conmon[39718]: cluster 2021-07-31T02:10:52.129287+0000 mgr.x (mgr.24542) 2021-07-31T02:10:53.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:52 smithi045 conmon[39718]: 1014 : cluster [DBG] pgmap v1581: 9 pgs: 9 active+clean; 127 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.1 MiB/s wr, 3 op/s 2021-07-31T02:10:53.360 INFO:tasks.workunit.client.0.smithi045.stdout: 340: throughput=4.99MB/sec pending data=0 2021-07-31T02:10:53.361 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-b0zjj4ccgRgu8o4 off=5022049 len=1803783 2021-07-31T02:10:53.371 INFO:tasks.workunit.client.0.smithi045.stdout:op 1747 completed, throughput=4.99MB/sec 2021-07-31T02:10:53.371 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-88-82woaAPXscyo off=1350780 len=811842 2021-07-31T02:10:53.375 INFO:tasks.workunit.client.0.smithi045.stdout:op 1748 completed, throughput=4.99MB/sec 2021-07-31T02:10:53.375 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Bd_1Dfr4A03CoN_ off=12972478 len=1907349 2021-07-31T02:10:53.377 INFO:tasks.workunit.client.0.smithi045.stdout:op 1749 completed, throughput=5MB/sec 2021-07-31T02:10:54.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:10:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:10:53.548Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:10:54.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:10:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:10:53.548Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:10:54.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:10:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:10:53.548Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:10:54.378 INFO:tasks.workunit.client.0.smithi045.stdout: 341: throughput=4.99MB/sec pending data=0 2021-07-31T02:10:54.378 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-UktezRh55BMhrSM off=1447562 len=1128558 2021-07-31T02:10:54.385 INFO:tasks.workunit.client.0.smithi045.stdout:op 1750 completed, throughput=4.99MB/sec 2021-07-31T02:10:54.386 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj--MRXouWzCm6TIWO off=2927710 len=52207 2021-07-31T02:10:54.388 INFO:tasks.workunit.client.0.smithi045.stdout:op 1751 completed, throughput=4.99MB/sec 2021-07-31T02:10:54.388 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj--gH7UJfSGLlT0-s off=637550 len=361916 2021-07-31T02:10:54.391 INFO:tasks.workunit.client.0.smithi045.stdout:op 1752 completed, throughput=4.99MB/sec 2021-07-31T02:10:54.392 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-LhejbDfvNBzmEbZ off=1227239 len=1346037 2021-07-31T02:10:54.394 INFO:tasks.workunit.client.0.smithi045.stdout:op 1753 completed, throughput=4.99MB/sec 2021-07-31T02:10:54.394 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-5zyHe2rO2nL3dyU off=5116425 len=520041 2021-07-31T02:10:54.395 INFO:tasks.workunit.client.0.smithi045.stdout:op 1754 completed, throughput=4.99MB/sec 2021-07-31T02:10:54.395 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-peG_kQdfvZNd0xm off=5841064 len=341876 2021-07-31T02:10:54.396 INFO:tasks.workunit.client.0.smithi045.stdout:op 1755 completed, throughput=5MB/sec 2021-07-31T02:10:54.396 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-5PeGKwU-O1BtEIt off=3715137 len=1342425 2021-07-31T02:10:54.398 INFO:tasks.workunit.client.0.smithi045.stdout:op 1756 completed, throughput=5MB/sec 2021-07-31T02:10:54.399 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-e2_bR_U700pF_4Q off=2068026 len=2089794 2021-07-31T02:10:54.401 INFO:tasks.workunit.client.0.smithi045.stdout:op 1757 completed, throughput=5.01MB/sec 2021-07-31T02:10:54.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:54 smithi154 conmon[34553]: cluster 2021-07-31T02:10:54.129870+0000 mgr.x (mgr.24542) 1015 : cluster [DBG] pgmap v1582: 9 pgs: 9 active+clean; 127 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 3.1 MiB/s rd, 3.1 MiB/s wr, 5 op/s 2021-07-31T02:10:54.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:54 smithi045 conmon[39718]: cluster 2021-07-31T02:10:54.129870+0000 mgr.x (mgr.24542) 1015 : cluster [DBG] pgmap v1582: 9 pgs: 9 active+clean; 127 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 3.1 MiB/s rd, 3.1 MiB/s wr, 5 op/s 2021-07-31T02:10:54.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:54 smithi045 conmon[33655]: cluster 2021-07-31T02:10:54.129870+0000 mgr.x (mgr.24542) 1015 : cluster [DBG] pgmap v1582: 9 pgs: 9 active+clean; 127 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 3.1 MiB/s rd, 3.1 MiB/s wr, 5 op/s 2021-07-31T02:10:55.401 INFO:tasks.workunit.client.0.smithi045.stdout: 342: throughput=4.99MB/sec pending data=0 2021-07-31T02:10:55.401 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-gVxY5-xO_jZXOo9 off=1138920 len=763302 2021-07-31T02:10:55.406 INFO:tasks.workunit.client.0.smithi045.stdout:op 1758 completed, throughput=4.99MB/sec 2021-07-31T02:10:55.406 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-cbPSoj5e7pRx9N8 off=10793011 len=1413485 2021-07-31T02:10:55.412 INFO:tasks.workunit.client.0.smithi045.stdout:op 1759 completed, throughput=5MB/sec 2021-07-31T02:10:55.412 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-hE0PDbivhzj7yWU off=1554164 len=583735 2021-07-31T02:10:55.416 INFO:tasks.workunit.client.0.smithi045.stdout:op 1760 completed, throughput=5MB/sec 2021-07-31T02:10:55.417 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-qynU59K4tCOf-Mv off=8093537 len=1249508 2021-07-31T02:10:55.419 INFO:tasks.workunit.client.0.smithi045.stdout:op 1761 completed, throughput=5MB/sec 2021-07-31T02:10:56.419 INFO:tasks.workunit.client.0.smithi045.stdout: 343: throughput=4.99MB/sec pending data=0 2021-07-31T02:10:56.419 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-OiRY36F3EYWxKT8 off=1958040 len=806080 2021-07-31T02:10:56.421 INFO:tasks.workunit.client.0.smithi045.stdout:op 1762 completed, throughput=4.99MB/sec 2021-07-31T02:10:56.422 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-CuKcv01mSBvyvdy off=4699608 len=1207945 2021-07-31T02:10:56.429 INFO:tasks.workunit.client.0.smithi045.stdout:op 1763 completed, throughput=4.99MB/sec 2021-07-31T02:10:56.429 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-gyvBqJR8usnRSl2 off=9457495 len=1402986 2021-07-31T02:10:56.431 INFO:tasks.workunit.client.0.smithi045.stdout:op 1764 completed, throughput=5MB/sec 2021-07-31T02:10:56.432 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-A8qJR0oavdgoGge off=382779 len=554185 2021-07-31T02:10:56.433 INFO:tasks.workunit.client.0.smithi045.stdout:op 1765 completed, throughput=5MB/sec 2021-07-31T02:10:56.434 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-VY8zdbdIbBNKzjs off=1353638 len=1019982 2021-07-31T02:10:56.437 INFO:tasks.workunit.client.0.smithi045.stdout:op 1766 completed, throughput=5MB/sec 2021-07-31T02:10:56.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:56 smithi154 conmon[34553]: cluster 2021-07-31T02:10:56.130270+0000 mgr.x (mgr.24542) 1016 : cluster [DBG] pgmap v1583: 9 pgs: 9 active+clean; 127 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 2.3 MiB/s rd, 1.9 MiB/s wr, 3 op/s 2021-07-31T02:10:56.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:56 smithi045 conmon[39718]: cluster 2021-07-31T02:10:56.130270+0000 mgr.x 2021-07-31T02:10:56.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:56 smithi045 conmon[39718]: (mgr.24542) 1016 : cluster [DBG] pgmap v1583: 9 pgs: 9 active+clean; 127 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 2.3 MiB/s rd, 1.9 MiB/s wr, 3 op/s 2021-07-31T02:10:56.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:56 smithi045 conmon[33655]: cluster 2021-07-31T02:10:56.130270+0000 mgr.x (mgr.24542) 1016 : cluster [DBG] 2021-07-31T02:10:56.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:56 smithi045 conmon[33655]: pgmap v1583: 9 pgs: 9 active+clean; 127 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 2.3 MiB/s rd, 1.9 MiB/s wr, 3 op/s 2021-07-31T02:10:57.437 INFO:tasks.workunit.client.0.smithi045.stdout: 344: throughput=4.99MB/sec pending data=0 2021-07-31T02:10:57.438 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-t2nfwWf-IGsLD-A off=17626654 len=2032032 2021-07-31T02:10:57.440 INFO:tasks.workunit.client.0.smithi045.stdout:op 1767 completed, throughput=4.99MB/sec 2021-07-31T02:10:57.440 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-r06gjEhzQ7BEdNB off=18186360 len=928466 2021-07-31T02:10:57.447 INFO:tasks.workunit.client.0.smithi045.stdout:op 1768 completed, throughput=4.99MB/sec 2021-07-31T02:10:57.447 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-HxW_Zb1kG8dKG1s off=18738320 len=975031 2021-07-31T02:10:57.453 INFO:tasks.workunit.client.0.smithi045.stdout:op 1769 completed, throughput=5MB/sec 2021-07-31T02:10:57.453 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ibEjXrvOeA5-INc off=22642578 len=1047913 2021-07-31T02:10:57.454 INFO:tasks.workunit.client.0.smithi045.stdout:op 1770 completed, throughput=5MB/sec 2021-07-31T02:10:57.454 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-NhhdqNd8ZoInEbU off=4250406 len=1259910 2021-07-31T02:10:57.460 INFO:tasks.workunit.client.0.smithi045.stdout:op 1771 completed, throughput=5MB/sec 2021-07-31T02:10:58.461 INFO:tasks.workunit.client.0.smithi045.stdout: 345: throughput=4.99MB/sec pending data=0 2021-07-31T02:10:58.651 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Vtw5RDhukMg94G_ off=6671927 len=1690098 2021-07-31T02:10:58.652 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:10:58 smithi154 conmon[34553]: cluster 2021-07-31T02:10:58.130760+0000 mgr.x (mgr.24542) 1017 : cluster [DBG] pgmap v1584: 9 pgs: 9 active+clean; 127 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 2.3 MiB/s rd, 1.9 MiB/s wr, 3 op/s 2021-07-31T02:10:58.653 INFO:tasks.workunit.client.0.smithi045.stdout:op 1772 completed, throughput=4.99MB/sec 2021-07-31T02:10:58.653 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-AdtHcRpx3GHxhSZ off=5477465 len=899366 2021-07-31T02:10:58.653 INFO:tasks.workunit.client.0.smithi045.stdout:op 1773 completed, throughput=5MB/sec 2021-07-31T02:10:58.654 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-28sNeF7Gnl7pES5 off=2699472 len=1335175 2021-07-31T02:10:58.654 INFO:tasks.workunit.client.0.smithi045.stdout:op 1774 completed, throughput=5MB/sec 2021-07-31T02:10:58.654 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-4qkZkyOnr8Qyaxq off=2505626 len=721138 2021-07-31T02:10:58.654 INFO:tasks.workunit.client.0.smithi045.stdout:op 1775 completed, throughput=5MB/sec 2021-07-31T02:10:58.655 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:58 smithi045 conmon[39718]: cluster 2021-07-31T02:10: 2021-07-31T02:10:58.655 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:10:58 smithi045 conmon[39718]: 58.130760+0000 mgr.x (mgr.24542) 1017 : cluster [DBG] pgmap v1584: 9 pgs: 9 active+clean; 127 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 2.3 MiB/s rd, 1.9 MiB/s wr, 3 op/s 2021-07-31T02:10:58.655 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:58 smithi045 conmon[33655]: cluster 2021-07-31T02:10:58.130760+0000 mgr.x (mgr.24542) 1017 : cluster 2021-07-31T02:10:58.656 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:10:58 smithi045 conmon[33655]: [DBG] pgmap v1584: 9 pgs: 9 active+clean; 127 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 2.3 MiB/s rd, 1.9 MiB/s wr, 3 op/s 2021-07-31T02:10:59.478 INFO:tasks.workunit.client.0.smithi045.stdout: 346: throughput=4.99MB/sec pending data=0 2021-07-31T02:10:59.478 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-qCWTPNXHCmw61W8 off=3929675 len=1074690 2021-07-31T02:10:59.479 INFO:tasks.workunit.client.0.smithi045.stdout:op 1776 completed, throughput=4.99MB/sec 2021-07-31T02:10:59.479 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-jvRNTeHuFl7hByz off=11794518 len=759390 2021-07-31T02:10:59.481 INFO:tasks.workunit.client.0.smithi045.stdout:op 1777 completed, throughput=4.99MB/sec 2021-07-31T02:10:59.481 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-3hlWtMk5Rqhuuf9 off=11108691 len=1291289 2021-07-31T02:10:59.488 INFO:tasks.workunit.client.0.smithi045.stdout:op 1778 completed, throughput=5MB/sec 2021-07-31T02:10:59.489 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-7-SHpeugjZ0c2lv off=15110908 len=256781 2021-07-31T02:10:59.492 INFO:tasks.workunit.client.0.smithi045.stdout:op 1779 completed, throughput=5MB/sec 2021-07-31T02:10:59.492 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-YyBWRA8ezneJPSZ off=714273 len=65166 2021-07-31T02:10:59.493 INFO:tasks.workunit.client.0.smithi045.stdout:op 1780 completed, throughput=5MB/sec 2021-07-31T02:10:59.493 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ggJmMwvxDzFcjKU off=8513176 len=181754 2021-07-31T02:10:59.494 INFO:tasks.workunit.client.0.smithi045.stdout:op 1781 completed, throughput=5MB/sec 2021-07-31T02:10:59.494 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-AJg8Ck7NexIHGQI off=213787 len=1528566 2021-07-31T02:10:59.496 INFO:tasks.workunit.client.0.smithi045.stdout:op 1782 completed, throughput=5MB/sec 2021-07-31T02:11:00.496 INFO:tasks.workunit.client.0.smithi045.stdout: 347: throughput=4.99MB/sec pending data=0 2021-07-31T02:11:00.496 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-qYxuzuADUIl_y_V off=7796336 len=1099253 2021-07-31T02:11:00.497 INFO:tasks.workunit.client.0.smithi045.stdout:op 1783 completed, throughput=4.99MB/sec 2021-07-31T02:11:00.498 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-kiaWOyoSUD_PdoI off=698465 len=112988 2021-07-31T02:11:00.499 INFO:tasks.workunit.client.0.smithi045.stdout:op 1784 completed, throughput=4.99MB/sec 2021-07-31T02:11:00.499 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-iZqL6qNDqjwjUql off=13592298 len=1305653 2021-07-31T02:11:00.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:00 smithi154 conmon[34553]: cluster 2021-07-31T02:11:00.131178+0000 mgr.x (mgr.24542) 1018 : cluster [DBG] pgmap v1585: 9 pgs: 9 active+clean; 127 GiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 3.6 MiB/s rd, 2.8 MiB/s wr, 6 op/s 2021-07-31T02:11:00.505 INFO:tasks.workunit.client.0.smithi045.stdout:op 1785 completed, throughput=4.99MB/sec 2021-07-31T02:11:00.505 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Asjx1tqfj-CJcPR off=512860 len=811097 2021-07-31T02:11:00.509 INFO:tasks.workunit.client.0.smithi045.stdout:op 1786 completed, throughput=5MB/sec 2021-07-31T02:11:00.509 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ehPPxOPbliwyoEM off=3597029 len=872124 2021-07-31T02:11:00.513 INFO:tasks.workunit.client.0.smithi045.stdout:op 1787 completed, throughput=5MB/sec 2021-07-31T02:11:00.514 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-6aw5-YOxfazXJdx off=9570495 len=2002396 2021-07-31T02:11:00.517 INFO:tasks.workunit.client.0.smithi045.stdout:op 1788 completed, throughput=5MB/sec 2021-07-31T02:11:00.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:00 smithi045 conmon[33655]: cluster 2021-07-31T02:11:00.131178+0000 mgr.x (mgr.24542) 1018 : cluster [DBG] 2021-07-31T02:11:00.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:00 smithi045 conmon[33655]: pgmap v1585: 9 pgs: 9 active+clean; 127 GiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 3.6 MiB/s rd, 2.8 MiB/s wr, 6 op/s 2021-07-31T02:11:00.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:00 smithi045 conmon[39718]: cluster 2021-07-31T02:11:00 2021-07-31T02:11:00.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:00 smithi045 conmon[39718]: .131178+0000 mgr.x (mgr.24542) 1018 : cluster [DBG] pgmap v1585: 9 pgs: 9 active+clean; 127 GiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 3.6 MiB/s rd, 2.8 MiB/s wr, 6 op/s 2021-07-31T02:11:01.517 INFO:tasks.workunit.client.0.smithi045.stdout: 348: throughput=4.99MB/sec pending data=0 2021-07-31T02:11:01.518 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ZVIUkCjXjT3806t off=1760853 len=1736344 2021-07-31T02:11:01.523 INFO:tasks.workunit.client.0.smithi045.stdout:op 1789 completed, throughput=4.99MB/sec 2021-07-31T02:11:01.523 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-tPDd0kBILAVFfwk off=960199 len=568527 2021-07-31T02:11:01.525 INFO:tasks.workunit.client.0.smithi045.stdout:op 1790 completed, throughput=4.99MB/sec 2021-07-31T02:11:01.526 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-df0V8BwTBVlXOD_ off=13171037 len=96923 2021-07-31T02:11:01.526 INFO:tasks.workunit.client.0.smithi045.stdout:op 1791 completed, throughput=5MB/sec 2021-07-31T02:11:01.527 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-KCntu5HM6FueaMe off=4452207 len=1428301 2021-07-31T02:11:01.528 INFO:tasks.workunit.client.0.smithi045.stdout:op 1792 completed, throughput=5MB/sec 2021-07-31T02:11:01.528 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-0p4djvQaXoHhWtP off=184482 len=1468423 2021-07-31T02:11:01.531 INFO:tasks.workunit.client.0.smithi045.stdout:op 1793 completed, throughput=5MB/sec 2021-07-31T02:11:02.531 INFO:tasks.workunit.client.0.smithi045.stdout: 349: throughput=4.99MB/sec pending data=0 2021-07-31T02:11:02.532 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-g0OsLXCHtOarp06 off=14319956 len=1534035 2021-07-31T02:11:02.539 INFO:tasks.workunit.client.0.smithi045.stdout:op 1794 completed, throughput=4.99MB/sec 2021-07-31T02:11:02.539 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-vn6DrAnyTDW4KqM off=9981699 len=763605 2021-07-31T02:11:02.541 INFO:tasks.workunit.client.0.smithi045.stdout:op 1795 completed, throughput=4.99MB/sec 2021-07-31T02:11:02.541 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-eH8UP65fR4TZEJb off=0 len=1411116 2021-07-31T02:11:02.546 INFO:tasks.workunit.client.0.smithi045.stdout:op 1796 completed, throughput=5MB/sec 2021-07-31T02:11:02.547 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-QJa7OTUzcb7BMtI off=5770380 len=1627306 2021-07-31T02:11:02.550 INFO:tasks.workunit.client.0.smithi045.stdout:op 1797 completed, throughput=5MB/sec 2021-07-31T02:11:03.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:02 smithi154 conmon[34553]: cluster 2021-07-31T02:11:02.131591+0000 mgr.x (mgr.24542) 1019 : cluster [DBG] pgmap v1586: 9 pgs: 9 active+clean; 127 GiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 2.3 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2021-07-31T02:11:03.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:02 smithi045 conmon[33655]: cluster 2021-07-31T02:11:02.131591+0000 mgr.x (mgr 2021-07-31T02:11:03.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:02 smithi045 conmon[33655]: .24542) 1019 : cluster [DBG] pgmap v1586: 9 pgs: 9 active+clean; 127 GiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 2.3 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2021-07-31T02:11:03.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:02 smithi045 conmon[39718]: cluster 2021-07-31T02:11:02.131591+0000 mgr.x (mgr.24542) 1019 2021-07-31T02:11:03.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:02 smithi045 conmon[39718]: : cluster [DBG] pgmap v1586: 9 pgs: 9 active+clean; 127 GiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 2.3 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2021-07-31T02:11:03.550 INFO:tasks.workunit.client.0.smithi045.stdout: 350: throughput=4.99MB/sec pending data=0 2021-07-31T02:11:03.551 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-bOu5p-KhmfeyNAb off=0 len=1787168 2021-07-31T02:11:03.562 INFO:tasks.workunit.client.0.smithi045.stdout:op 1798 completed, throughput=4.99MB/sec 2021-07-31T02:11:03.562 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-y1_x4lCC0EVUBTg off=906473 len=526031 2021-07-31T02:11:03.569 INFO:tasks.workunit.client.0.smithi045.stdout:op 1799 completed, throughput=4.99MB/sec 2021-07-31T02:11:03.569 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-4Z8-t6PRyobQ-SH off=11521274 len=758641 2021-07-31T02:11:03.569 INFO:tasks.workunit.client.0.smithi045.stdout:op 1800 completed, throughput=5MB/sec 2021-07-31T02:11:03.569 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-UgpYju6Z7gMJb20 off=250368 len=1878321 2021-07-31T02:11:03.579 INFO:tasks.workunit.client.0.smithi045.stdout:op 1801 completed, throughput=5MB/sec 2021-07-31T02:11:04.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:11:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:11:03.549Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:11:04.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:11:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:11:03.549Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:11:04.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:11:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:11:03.549Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:11:04.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:04 smithi154 conmon[34553]: cluster 2021-07-31T02:11:04.132089+0000 mgr.x (mgr.24542) 1020 : cluster [DBG] pgmap v1587: 9 pgs: 9 active+clean; 127 GiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 3.6 MiB/s rd, 2.8 MiB/s wr, 6 op/s 2021-07-31T02:11:04.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:04 smithi045 conmon[33655]: cluster 2021-07-31T02:11:04.132089+0000 mgr.x (mgr.24542) 1020 : cluster [DBG] pgmap v1587: 9 pgs: 9 active+clean; 127 GiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 3.6 MiB/s rd, 2.8 MiB/s wr, 6 op/s 2021-07-31T02:11:04.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:04 smithi045 conmon[39718]: cluster 2021-07-31T02:11:04.132089+0000 mgr.x (mgr.24542) 1020 : cluster 2021-07-31T02:11:04.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:04 smithi045 conmon[39718]: [DBG] pgmap v1587: 9 pgs: 9 active+clean; 127 GiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 3.6 MiB/s rd, 2.8 MiB/s wr, 6 op/s 2021-07-31T02:11:04.579 INFO:tasks.workunit.client.0.smithi045.stdout: 351: throughput=4.99MB/sec pending data=0 2021-07-31T02:11:04.579 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-J6kHNJNwU9enKRu off=4111388 len=2014982 2021-07-31T02:11:04.581 INFO:tasks.workunit.client.0.smithi045.stdout:op 1802 completed, throughput=4.99MB/sec 2021-07-31T02:11:04.581 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-PMabaKNpvifzjAZ off=8300635 len=833374 2021-07-31T02:11:04.583 INFO:tasks.workunit.client.0.smithi045.stdout:op 1803 completed, throughput=5MB/sec 2021-07-31T02:11:04.584 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-vSvtL0n2NnD2Til off=2273922 len=362832 2021-07-31T02:11:04.585 INFO:tasks.workunit.client.0.smithi045.stdout:op 1804 completed, throughput=5MB/sec 2021-07-31T02:11:04.585 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-UM3yb81156y_dyG off=13258880 len=848842 2021-07-31T02:11:04.587 INFO:tasks.workunit.client.0.smithi045.stdout:op 1805 completed, throughput=5MB/sec 2021-07-31T02:11:04.587 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-NLj1_Xbtg7fuuD4 off=522051 len=21752 2021-07-31T02:11:04.589 INFO:tasks.workunit.client.0.smithi045.stdout:op 1806 completed, throughput=5MB/sec 2021-07-31T02:11:04.589 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-V6_ya9c57R5rOhT off=12657280 len=2080533 2021-07-31T02:11:04.591 INFO:tasks.workunit.client.0.smithi045.stdout:op 1807 completed, throughput=5MB/sec 2021-07-31T02:11:05.592 INFO:tasks.workunit.client.0.smithi045.stdout: 352: throughput=4.99MB/sec pending data=0 2021-07-31T02:11:05.593 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-pB3M2HxztPw3Ceg off=4865283 len=1507914 2021-07-31T02:11:05.599 INFO:tasks.workunit.client.0.smithi045.stdout:op 1808 completed, throughput=4.99MB/sec 2021-07-31T02:11:05.599 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-w5-sf1yWNy-qFrr off=2167759 len=380789 2021-07-31T02:11:05.600 INFO:tasks.workunit.client.0.smithi045.stdout:op 1809 completed, throughput=4.99MB/sec 2021-07-31T02:11:05.601 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-uY43j2zuzmWOUD9 off=1680671 len=319225 2021-07-31T02:11:05.603 INFO:tasks.workunit.client.0.smithi045.stdout:op 1810 completed, throughput=5MB/sec 2021-07-31T02:11:05.603 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-o8v0dZO29DpzNvh off=6506242 len=226907 2021-07-31T02:11:05.606 INFO:tasks.workunit.client.0.smithi045.stdout:op 1811 completed, throughput=5MB/sec 2021-07-31T02:11:05.607 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-IaBYIc7JXt03KLe off=1547912 len=1462864 2021-07-31T02:11:05.613 INFO:tasks.workunit.client.0.smithi045.stdout:op 1812 completed, throughput=5MB/sec 2021-07-31T02:11:05.613 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-fXaHkSEQJqNd3LE off=2144874 len=585001 2021-07-31T02:11:05.614 INFO:tasks.workunit.client.0.smithi045.stdout:op 1813 completed, throughput=5MB/sec 2021-07-31T02:11:06.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:06 smithi154 conmon[34553]: cluster 2021-07-31T02:11:06.132496+0000 mgr.x (mgr.24542) 1021 : cluster [DBG] pgmap v1588: 9 pgs: 9 active+clean; 127 GiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.8 MiB/s wr, 4 op/s 2021-07-31T02:11:06.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:06 smithi045 conmon[33655]: cluster 2021-07-31T02:11:06.132496+0000 mgr.x ( 2021-07-31T02:11:06.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:06 smithi045 conmon[33655]: mgr.24542) 1021 : cluster [DBG] pgmap v1588: 9 pgs: 9 active+clean; 127 GiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.8 MiB/s wr, 4 op/s 2021-07-31T02:11:06.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:06 smithi045 conmon[39718]: cluster 2021-07-31T02:11:06.132496+0000 mgr.x (mgr.24542) 2021-07-31T02:11:06.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:06 smithi045 conmon[39718]: 1021 : cluster [DBG] pgmap v1588: 9 pgs: 9 active+clean; 127 GiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.8 MiB/s wr, 4 op/s 2021-07-31T02:11:06.614 INFO:tasks.workunit.client.0.smithi045.stdout: 353: throughput=4.99MB/sec pending data=0 2021-07-31T02:11:06.615 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-WWjsGGoT70e32sX off=18334721 len=1913619 2021-07-31T02:11:06.625 INFO:tasks.workunit.client.0.smithi045.stdout:op 1814 completed, throughput=4.99MB/sec 2021-07-31T02:11:06.625 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-5GE4g19vq9-9jAZ off=2504440 len=1478256 2021-07-31T02:11:06.627 INFO:tasks.workunit.client.0.smithi045.stdout:op 1815 completed, throughput=5MB/sec 2021-07-31T02:11:06.627 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-8ecVUkybZmbvw8U off=3292725 len=192583 2021-07-31T02:11:06.630 INFO:tasks.workunit.client.0.smithi045.stdout:op 1816 completed, throughput=5MB/sec 2021-07-31T02:11:06.631 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ubvxNbRNCiMkqlo off=0 len=1419468 2021-07-31T02:11:06.632 INFO:tasks.workunit.client.0.smithi045.stdout:op 1817 completed, throughput=5MB/sec 2021-07-31T02:11:07.633 INFO:tasks.workunit.client.0.smithi045.stdout: 354: throughput=4.99MB/sec pending data=0 2021-07-31T02:11:07.633 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ex6MqtuywJ-Tcyr off=919226 len=568271 2021-07-31T02:11:07.637 INFO:tasks.workunit.client.0.smithi045.stdout:op 1818 completed, throughput=4.99MB/sec 2021-07-31T02:11:07.638 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-7bBLCoqRNFuYml8 off=10446943 len=1425722 2021-07-31T02:11:07.644 INFO:tasks.workunit.client.0.smithi045.stdout:op 1819 completed, throughput=4.99MB/sec 2021-07-31T02:11:07.644 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-kYg9s11bbSsv4Dq off=5711680 len=182898 2021-07-31T02:11:07.645 INFO:tasks.workunit.client.0.smithi045.stdout:op 1820 completed, throughput=4.99MB/sec 2021-07-31T02:11:07.645 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Vkmap_BPyZInjg_ off=18143900 len=1047763 2021-07-31T02:11:07.647 INFO:tasks.workunit.client.0.smithi045.stdout:op 1821 completed, throughput=5MB/sec 2021-07-31T02:11:07.647 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-KQCMlg4kku32csF off=10221602 len=166169 2021-07-31T02:11:07.649 INFO:tasks.workunit.client.0.smithi045.stdout:op 1822 completed, throughput=5MB/sec 2021-07-31T02:11:07.650 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-QqPQkiO8n8dzikK off=12432633 len=1281313 2021-07-31T02:11:07.651 INFO:tasks.workunit.client.0.smithi045.stdout:op 1823 completed, throughput=5MB/sec 2021-07-31T02:11:07.651 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-hcucSIBjLFGNR23 off=4153943 len=1659862 2021-07-31T02:11:07.655 INFO:tasks.workunit.client.0.smithi045.stdout:op 1824 completed, throughput=5MB/sec 2021-07-31T02:11:08.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:08 smithi154 conmon[34553]: cluster 2021-07-31T02:11:08.132937+0000 mgr.x (mgr.24542) 1022 : cluster [DBG] pgmap v1589: 9 pgs: 9 active+clean; 127 GiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.8 MiB/s wr, 4 op/s 2021-07-31T02:11:08.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:08 smithi045 conmon[33655]: cluster 2021-07-31T02:11:08.132937+0000 mgr.x (mgr.24542) 1022 : cluster [DBG] pgmap v1589: 9 pgs: 9 active+clean; 127 GiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.8 MiB/s wr, 4 op/s 2021-07-31T02:11:08.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:08 smithi045 conmon[39718]: cluster 2021-07-31T02:11:08.132937+0000 2021-07-31T02:11:08.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:08 smithi045 conmon[39718]: mgr.x (mgr.24542) 1022 : cluster [DBG] pgmap v1589: 9 pgs: 9 active+clean; 127 GiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.8 MiB/s wr, 4 op/s 2021-07-31T02:11:08.655 INFO:tasks.workunit.client.0.smithi045.stdout: 355: throughput=4.99MB/sec pending data=0 2021-07-31T02:11:08.655 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-1_RkxXgV-3LGoT0 off=4500119 len=666792 2021-07-31T02:11:08.656 INFO:tasks.workunit.client.0.smithi045.stdout:op 1825 completed, throughput=4.99MB/sec 2021-07-31T02:11:08.656 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-CbRSWTyxqb5KRTi off=7178700 len=1703096 2021-07-31T02:11:08.659 INFO:tasks.workunit.client.0.smithi045.stdout:op 1826 completed, throughput=5MB/sec 2021-07-31T02:11:08.660 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-oMOHFXQODAchGUq off=833666 len=854889 2021-07-31T02:11:08.665 INFO:tasks.workunit.client.0.smithi045.stdout:op 1827 completed, throughput=5MB/sec 2021-07-31T02:11:08.666 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-KSIkOSfbSdITAwz off=4740797 len=1651247 2021-07-31T02:11:08.673 INFO:tasks.workunit.client.0.smithi045.stdout:op 1828 completed, throughput=5MB/sec 2021-07-31T02:11:09.150 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:11:09.427 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:11:09.428 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 26m ago 26m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:11:09.428 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 23m ago 26m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:11:09.428 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 23m ago 30m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:11:09.429 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 26m ago 32m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:11:09.429 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 26m ago 32m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:11:09.429 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 23m ago 30m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:11:09.429 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 26m ago 30m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:11:09.430 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 26m ago 27m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:11:09.430 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 23m ago 26m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:11:09.431 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 26m ago 29m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:11:09.431 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 26m ago 29m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:11:09.431 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 26m ago 29m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:11:09.431 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 26m ago 28m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:11:09.432 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 23m ago 28m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:11:09.432 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 23m ago 28m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:11:09.432 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 23m ago 27m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:11:09.432 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 23m ago 27m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:11:09.432 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 23m ago 27m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:11:09.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:09 smithi154 conmon[34553]: audit 2021-07-31T02:11:09.138062+0000 mgr.x (mgr.24542) 1023 : audit [DBG] from='client.25840 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:11:09.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:09 smithi045 conmon[33655]: audit 2021-07-31T02:11:09.138062+0000 mgr.x (mgr.24542) 1023 : 2021-07-31T02:11:09.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:09 smithi045 conmon[33655]: audit [DBG] from='client.25840 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:11:09.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:09 smithi045 conmon[39718]: audit 2021-07-31T02:11:09.138062 2021-07-31T02:11:09.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:09 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 1023 : audit [DBG] from='client.25840 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:11:09.674 INFO:tasks.workunit.client.0.smithi045.stdout: 356: throughput=4.99MB/sec pending data=0 2021-07-31T02:11:09.674 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-lIWxsYX2LVXl9fx off=8265681 len=1584920 2021-07-31T02:11:09.677 INFO:tasks.workunit.client.0.smithi045.stdout:op 1829 completed, throughput=4.99MB/sec 2021-07-31T02:11:09.677 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-EVCu3g2FllivrxF off=2778671 len=1600343 2021-07-31T02:11:09.680 INFO:tasks.workunit.client.0.smithi045.stdout:op 1830 completed, throughput=5MB/sec 2021-07-31T02:11:09.680 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-krd6KespVtcKRss off=139307 len=414667 2021-07-31T02:11:09.680 INFO:tasks.workunit.client.0.smithi045.stdout:op 1831 completed, throughput=5MB/sec 2021-07-31T02:11:09.681 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-F_C6T5M-RZdNvub off=2769466 len=1814956 2021-07-31T02:11:09.690 INFO:tasks.workunit.client.0.smithi045.stdout:op 1832 completed, throughput=5MB/sec 2021-07-31T02:11:09.765 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:11:09.765 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:11:09.765 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:11:09.765 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:11:09.766 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:11:09.766 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:11:09.766 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:11:09.766 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:11:09.767 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:11:09.767 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:11:09.767 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:11:09.767 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:11:09.768 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:11:09.768 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:11:09.768 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:11:09.769 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:11:09.769 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:11:10.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:10 smithi154 conmon[34553]: audit 2021-07-31T02:11:09.419185+0000 mgr.x (mgr.24542) 1024 : audit [DBG] from='client.15939 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:11:10.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:10 smithi154 conmon[34553]: audit 2021-07-31T02:11:09.765000+0000 mon.c (mon.1) 81 : audit [DBG] from='client.? 172.21.15.45:0/3648415157' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:11:10.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:10 smithi154 conmon[34553]: cluster 2021-07-31T02:11:10.133358+0000 mgr.x (mgr.24542) 1025 : cluster [DBG] pgmap v1590: 9 pgs: 9 active+clean; 127 GiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 3.8 MiB/s rd, 2.6 MiB/s wr, 6 op/s 2021-07-31T02:11:10.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:10 smithi045 conmon[39718]: audit 2021-07-31T02:11 2021-07-31T02:11:10.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:10 smithi045 conmon[39718]: :09.419185+0000 mgr.x (mgr.24542) 1024 : audit [DBG] from='client.15939 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:11:10.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:10 smithi045 conmon[39718]: audit 2021 2021-07-31T02:11:10.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:10 smithi045 conmon[39718]: -07-31T02:11:09.765000+0000 mon.c 2021-07-31T02:11:10.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:10 smithi045 conmon[39718]: (mon.1) 81 : audit [DBG] 2021-07-31T02:11:10.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:10 smithi045 conmon[39718]: from='client.? 172.21.15.45:0/3648415157' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:11:10.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:10 smithi045 conmon[39718]: cluster 2021- 2021-07-31T02:11:10.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:10 smithi045 conmon[39718]: 07-31T02:11:10.133358+0000 mgr.x (mgr.24542) 1025 : 2021-07-31T02:11:10.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:10 smithi045 conmon[39718]: cluster [DBG] pgmap v1590: 9 pgs: 9 active+clean; 127 GiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 3.8 MiB/s rd, 2.6 MiB/s wr, 6 op/s 2021-07-31T02:11:10.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:10 smithi045 conmon[33655]: audit 2021-07-31T02:11:09.419185+0000 mgr.x (mgr.24542) 1024 : audit [DBG] 2021-07-31T02:11:10.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:10 smithi045 conmon[33655]: from='client.15939 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:11:10.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:10 smithi045 conmon[33655]: audit 2021-07-31T02:11:09.765000+0000 mon.c (mon.1) 81 : audit [DBG] from='client.? 172.21.15.45:0/3648415157' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:11:10.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:10 smithi045 conmon[33655]: cluster 2021-07-31 2021-07-31T02:11:10.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:10 smithi045 conmon[33655]: T02:11:10.133358+0000 mgr.x (mgr.24542) 1025 : cluster 2021-07-31T02:11:10.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:10 smithi045 conmon[33655]: [DBG] pgmap v1590: 9 pgs: 9 active+clean; 127 GiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 3.8 MiB/s rd, 2.6 MiB/s wr, 6 op/s 2021-07-31T02:11:10.690 INFO:tasks.workunit.client.0.smithi045.stdout: 357: throughput=4.99MB/sec pending data=0 2021-07-31T02:11:10.691 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-B5IBIPazzp63J54 off=16807086 len=1198731 2021-07-31T02:11:10.693 INFO:tasks.workunit.client.0.smithi045.stdout:op 1833 completed, throughput=4.99MB/sec 2021-07-31T02:11:10.693 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-7gbNGt0Zrrsb_LM off=3619385 len=961506 2021-07-31T02:11:10.695 INFO:tasks.workunit.client.0.smithi045.stdout:op 1834 completed, throughput=4.99MB/sec 2021-07-31T02:11:10.696 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-EZDhq9qWhX8Sltb off=2697872 len=127578 2021-07-31T02:11:10.698 INFO:tasks.workunit.client.0.smithi045.stdout:op 1835 completed, throughput=4.99MB/sec 2021-07-31T02:11:10.699 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-MoVr3ci5FK3S9T2 off=14223887 len=683429 2021-07-31T02:11:10.700 INFO:tasks.workunit.client.0.smithi045.stdout:op 1836 completed, throughput=5MB/sec 2021-07-31T02:11:10.700 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-dD6ZcyXIqKBP2fY off=15799325 len=995631 2021-07-31T02:11:10.706 INFO:tasks.workunit.client.0.smithi045.stdout:op 1837 completed, throughput=5MB/sec 2021-07-31T02:11:10.706 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-MSBYYJz2qMA_VCY off=3634303 len=649526 2021-07-31T02:11:10.710 INFO:tasks.workunit.client.0.smithi045.stdout:op 1838 completed, throughput=5MB/sec 2021-07-31T02:11:11.711 INFO:tasks.workunit.client.0.smithi045.stdout: 358: throughput=4.99MB/sec pending data=0 2021-07-31T02:11:11.711 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-cnVZS00z_a1r8u9 off=15745742 len=71118 2021-07-31T02:11:11.714 INFO:tasks.workunit.client.0.smithi045.stdout:op 1839 completed, throughput=4.99MB/sec 2021-07-31T02:11:11.714 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-unO9_hsTEVaRpGl off=9502168 len=1415611 2021-07-31T02:11:11.721 INFO:tasks.workunit.client.0.smithi045.stdout:op 1840 completed, throughput=4.99MB/sec 2021-07-31T02:11:11.721 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-0pG8Rcnmw7X27pD off=5585351 len=691843 2021-07-31T02:11:11.723 INFO:tasks.workunit.client.0.smithi045.stdout:op 1841 completed, throughput=4.99MB/sec 2021-07-31T02:11:11.723 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-I4nJkJdASX0cFAC off=9793051 len=2061215 2021-07-31T02:11:11.726 INFO:tasks.workunit.client.0.smithi045.stdout:op 1842 completed, throughput=5MB/sec 2021-07-31T02:11:11.727 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-7-TKJoWBqmZiSm3 off=4793777 len=1458091 2021-07-31T02:11:11.729 INFO:tasks.workunit.client.0.smithi045.stdout:op 1843 completed, throughput=5MB/sec 2021-07-31T02:11:12.730 INFO:tasks.workunit.client.0.smithi045.stdout: 359: throughput=4.99MB/sec pending data=0 2021-07-31T02:11:12.730 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Ud_thX_63dt5K0n off=9028590 len=2015756 2021-07-31T02:11:12.732 INFO:tasks.workunit.client.0.smithi045.stdout:op 1844 completed, throughput=4.99MB/sec 2021-07-31T02:11:12.732 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-h2C32J4IcdHyQ32 off=3788989 len=1027764 2021-07-31T02:11:12.739 INFO:tasks.workunit.client.0.smithi045.stdout:op 1845 completed, throughput=5MB/sec 2021-07-31T02:11:12.739 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-2XQ7YkZyFKggAVP off=12818110 len=401756 2021-07-31T02:11:12.740 INFO:tasks.workunit.client.0.smithi045.stdout:op 1846 completed, throughput=5MB/sec 2021-07-31T02:11:12.740 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-J4teSPwCewftwc- off=7422123 len=1031275 2021-07-31T02:11:12.745 INFO:tasks.workunit.client.0.smithi045.stdout:op 1847 completed, throughput=5MB/sec 2021-07-31T02:11:12.746 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-35uRXI9rE4okHtv off=1778444 len=1222863 2021-07-31T02:11:12.748 INFO:tasks.workunit.client.0.smithi045.stdout:op 1848 completed, throughput=5MB/sec 2021-07-31T02:11:13.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:12 smithi154 conmon[34553]: cluster 2021-07-31T02:11:12.133900+0000 mgr.x (mgr.24542) 1026 : cluster 2021-07-31T02:11:13.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:12 smithi154 conmon[34553]: [DBG] pgmap v1591: 9 pgs: 9 active+clean; 127 GiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.7 MiB/s wr, 4 op/s 2021-07-31T02:11:13.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:12 smithi045 conmon[33655]: cluster 2021-07-31T02:11:12.133900+0000 mgr.x (mgr.24542) 1026 : cluster [DBG] 2021-07-31T02:11:13.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:12 smithi045 conmon[33655]: pgmap v1591: 9 pgs: 9 active+clean; 127 GiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.7 MiB/s wr, 4 op/s 2021-07-31T02:11:13.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:12 smithi045 conmon[39718]: cluster 2021-07-31T02:11:12.133900+0000 mgr.x (mgr. 2021-07-31T02:11:13.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:12 smithi045 conmon[39718]: 24542) 1026 : cluster [DBG] pgmap v1591: 9 pgs: 9 active+clean; 127 GiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.7 MiB/s wr, 4 op/s 2021-07-31T02:11:13.748 INFO:tasks.workunit.client.0.smithi045.stdout: 360: throughput=4.99MB/sec pending data=0 2021-07-31T02:11:13.749 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Aaw2IqTv8aUxV8D off=20638782 len=1236192 2021-07-31T02:11:13.755 INFO:tasks.workunit.client.0.smithi045.stdout:op 1849 completed, throughput=4.99MB/sec 2021-07-31T02:11:13.755 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-h-JLz0rJ5zWaMfc off=4640778 len=1452352 2021-07-31T02:11:13.758 INFO:tasks.workunit.client.0.smithi045.stdout:op 1850 completed, throughput=5MB/sec 2021-07-31T02:11:13.758 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-wgMkYuz1KBcyCsG off=5583945 len=977702 2021-07-31T02:11:13.759 INFO:tasks.workunit.client.0.smithi045.stdout:op 1851 completed, throughput=5MB/sec 2021-07-31T02:11:13.760 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-r228jZLa3wlS_Mp off=991981 len=1738782 2021-07-31T02:11:13.767 INFO:tasks.workunit.client.0.smithi045.stdout:op 1852 completed, throughput=5MB/sec 2021-07-31T02:11:14.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:11:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:11:13.549Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:11:14.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:11:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:11:13.549Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:11:14.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:11:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:11:13.549Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:11:14.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:14 smithi154 conmon[34553]: cluster 2021-07-31T02:11:14.134423+0000 mgr.x (mgr.24542) 1027 : cluster [DBG] pgmap v1592: 9 pgs: 9 active+clean; 127 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 3.9 MiB/s rd, 2.5 MiB/s wr, 6 op/s 2021-07-31T02:11:14.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:14 smithi045 conmon[33655]: cluster 2021-07-31T02:11:14.134423+0000 mgr.x (mgr. 2021-07-31T02:11:14.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:14 smithi045 conmon[33655]: 24542) 1027 : cluster [DBG] pgmap v1592: 9 pgs: 9 active+clean; 127 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 3.9 MiB/s rd, 2.5 MiB/s wr, 6 op/s 2021-07-31T02:11:14.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:14 smithi045 conmon[39718]: cluster 2021-07-31T02:11:14 2021-07-31T02:11:14.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:14 smithi045 conmon[39718]: .134423+0000 mgr.x (mgr.24542) 1027 : cluster [DBG] pgmap v1592: 9 pgs: 9 active+clean; 127 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 3.9 MiB/s rd, 2.5 MiB/s wr, 6 op/s 2021-07-31T02:11:14.768 INFO:tasks.workunit.client.0.smithi045.stdout: 361: throughput=4.99MB/sec pending data=0 2021-07-31T02:11:14.768 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-BOz_f703UBaJhN1 off=624653 len=1961299 2021-07-31T02:11:14.770 INFO:tasks.workunit.client.0.smithi045.stdout:op 1853 completed, throughput=4.99MB/sec 2021-07-31T02:11:14.771 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-QSlXBRy11P1N3W5 off=1244406 len=1610177 2021-07-31T02:11:14.773 INFO:tasks.workunit.client.0.smithi045.stdout:op 1854 completed, throughput=5MB/sec 2021-07-31T02:11:14.774 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-T162e8Fgcsm716r off=783388 len=688819 2021-07-31T02:11:14.775 INFO:tasks.workunit.client.0.smithi045.stdout:op 1855 completed, throughput=5MB/sec 2021-07-31T02:11:14.775 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ii298wlQJ6T_3uP off=9171461 len=248306 2021-07-31T02:11:14.776 INFO:tasks.workunit.client.0.smithi045.stdout:op 1856 completed, throughput=5MB/sec 2021-07-31T02:11:15.777 INFO:tasks.workunit.client.0.smithi045.stdout: 362: throughput=4.99MB/sec pending data=0 2021-07-31T02:11:15.777 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-rXo3gDfx1V8TPXb off=5927560 len=1569784 2021-07-31T02:11:15.779 INFO:tasks.workunit.client.0.smithi045.stdout:op 1857 completed, throughput=4.99MB/sec 2021-07-31T02:11:15.779 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-1v8pWL-dluyjWYj off=683534 len=245470 2021-07-31T02:11:15.779 INFO:tasks.workunit.client.0.smithi045.stdout:op 1858 completed, throughput=4.99MB/sec 2021-07-31T02:11:15.780 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-uozWcmoHeIOIkJZ off=7801426 len=2357 2021-07-31T02:11:15.782 INFO:tasks.workunit.client.0.smithi045.stdout:op 1859 completed, throughput=4.99MB/sec 2021-07-31T02:11:15.782 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-zyfV2qi5Cz0OVRQ off=17403971 len=114532 2021-07-31T02:11:15.784 INFO:tasks.workunit.client.0.smithi045.stdout:op 1860 completed, throughput=4.99MB/sec 2021-07-31T02:11:15.785 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-OTPTKlrWMjazS2l off=6092693 len=634219 2021-07-31T02:11:15.789 INFO:tasks.workunit.client.0.smithi045.stdout:op 1861 completed, throughput=4.99MB/sec 2021-07-31T02:11:15.789 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-s_DPnLAw8AYvuFI off=2453978 len=1656984 2021-07-31T02:11:15.796 INFO:tasks.workunit.client.0.smithi045.stdout:op 1862 completed, throughput=5MB/sec 2021-07-31T02:11:15.797 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-NEtkQEltnRKLVK9 off=6565709 len=367610 2021-07-31T02:11:15.797 INFO:tasks.workunit.client.0.smithi045.stdout:op 1863 completed, throughput=5MB/sec 2021-07-31T02:11:15.798 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Mfv1nb24-b3mKKf off=292049 len=952472 2021-07-31T02:11:15.803 INFO:tasks.workunit.client.0.smithi045.stdout:op 1864 completed, throughput=5MB/sec 2021-07-31T02:11:16.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:16 smithi154 conmon[34553]: cluster 2021-07-31T02:11:16.135007+0000 mgr.x (mgr.24542) 1028 : cluster [DBG] pgmap v1593: 9 pgs: 9 active+clean; 127 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 2.6 MiB/s rd, 1.6 MiB/s wr, 4 op/s 2021-07-31T02:11:16.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:16 smithi045 conmon[39718]: cluster 2021-07-31T02:11:16 2021-07-31T02:11:16.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:16 smithi045 conmon[39718]: .135007+0000 mgr.x (mgr.24542) 1028 : cluster [DBG] pgmap v1593: 9 pgs: 9 active+clean; 127 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 2.6 MiB/s rd, 1.6 MiB/s wr, 4 op/s 2021-07-31T02:11:16.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:16 smithi045 conmon[33655]: cluster 2021-07-31T02:11:16.135007+0000 mgr.x (mgr.24542) 1028 : 2021-07-31T02:11:16.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:16 smithi045 conmon[33655]: cluster [DBG] pgmap v1593: 9 pgs: 9 active+clean; 127 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 2.6 MiB/s rd, 1.6 MiB/s wr, 4 op/s 2021-07-31T02:11:16.803 INFO:tasks.workunit.client.0.smithi045.stdout: 363: throughput=4.99MB/sec pending data=0 2021-07-31T02:11:16.803 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-uSsE53rGjot3e3a off=4125695 len=51343 2021-07-31T02:11:16.804 INFO:tasks.workunit.client.0.smithi045.stdout:op 1865 completed, throughput=4.99MB/sec 2021-07-31T02:11:16.804 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-O1vDZxHX4HFtNmA off=9675891 len=885562 2021-07-31T02:11:16.805 INFO:tasks.workunit.client.0.smithi045.stdout:op 1866 completed, throughput=4.99MB/sec 2021-07-31T02:11:16.806 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-bWQ8ivNGWnBTGmP off=11228120 len=79827 2021-07-31T02:11:16.808 INFO:tasks.workunit.client.0.smithi045.stdout:op 1867 completed, throughput=4.99MB/sec 2021-07-31T02:11:16.808 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-e31ua89nWChX-5G off=10352960 len=1540259 2021-07-31T02:11:16.810 INFO:tasks.workunit.client.0.smithi045.stdout:op 1868 completed, throughput=4.99MB/sec 2021-07-31T02:11:16.810 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ZtvXeLC67XU81Gj off=7110442 len=398485 2021-07-31T02:11:16.811 INFO:tasks.workunit.client.0.smithi045.stdout:op 1869 completed, throughput=4.99MB/sec 2021-07-31T02:11:16.811 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-oCYIG4_n8OfQL0d off=13147608 len=954568 2021-07-31T02:11:16.812 INFO:tasks.workunit.client.0.smithi045.stdout:op 1870 completed, throughput=5MB/sec 2021-07-31T02:11:16.812 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-bYsfMM3RkSinzGt off=4047769 len=1209788 2021-07-31T02:11:16.815 INFO:tasks.workunit.client.0.smithi045.stdout:op 1871 completed, throughput=5MB/sec 2021-07-31T02:11:17.815 INFO:tasks.workunit.client.0.smithi045.stdout: 364: throughput=4.99MB/sec pending data=0 2021-07-31T02:11:17.815 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-FCNoxEE1tRq_ZoA off=5316024 len=894486 2021-07-31T02:11:17.817 INFO:tasks.workunit.client.0.smithi045.stdout:op 1872 completed, throughput=4.99MB/sec 2021-07-31T02:11:17.818 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-t1PKyznLcIp5CwZ off=9427570 len=1739658 2021-07-31T02:11:17.820 INFO:tasks.workunit.client.0.smithi045.stdout:op 1873 completed, throughput=4.99MB/sec 2021-07-31T02:11:17.821 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-iUXFo2bQda8Yp_T off=12603226 len=264026 2021-07-31T02:11:17.821 INFO:tasks.workunit.client.0.smithi045.stdout:op 1874 completed, throughput=4.99MB/sec 2021-07-31T02:11:17.822 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-6oMo8wThX_yrJUz off=11460070 len=16936 2021-07-31T02:11:17.824 INFO:tasks.workunit.client.0.smithi045.stdout:op 1875 completed, throughput=4.99MB/sec 2021-07-31T02:11:17.824 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-592_LQCHCHjs4dq off=8733973 len=272212 2021-07-31T02:11:17.825 INFO:tasks.workunit.client.0.smithi045.stdout:op 1876 completed, throughput=4.99MB/sec 2021-07-31T02:11:17.825 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-0L6jv5X7zw0yMGG off=9589940 len=1845727 2021-07-31T02:11:17.827 INFO:tasks.workunit.client.0.smithi045.stdout:op 1877 completed, throughput=5MB/sec 2021-07-31T02:11:17.827 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ABYmoS1ViNGX0Bg off=11732396 len=331788 2021-07-31T02:11:17.828 INFO:tasks.workunit.client.0.smithi045.stdout:op 1878 completed, throughput=5MB/sec 2021-07-31T02:11:18.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:18 smithi154 conmon[34553]: cluster 2021-07-31T02:11:18.135577+0000 mgr.x (mgr.24542) 1029 : cluster [DBG] pgmap v1594: 9 pgs: 9 active+clean; 127 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 2.6 MiB/s rd, 1.6 MiB/s wr, 4 op/s 2021-07-31T02:11:18.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:18 smithi045 conmon[39718]: cluster 2021-07-31T02:11:18.135577+0000 mgr.x (mgr 2021-07-31T02:11:18.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:18 smithi045 conmon[39718]: .24542) 1029 : cluster [DBG] pgmap v1594: 9 pgs: 9 active+clean; 127 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 2.6 MiB/s rd, 1.6 MiB/s wr, 4 op/s 2021-07-31T02:11:18.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:18 smithi045 conmon[33655]: cluster 2021-07-31T02:11:18.135577+0000 mgr.x (mgr.24542) 1029 : cluster [DBG] pgmap v1594: 9 pgs: 9 active+clean; 127 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 2.6 MiB/s rd, 1.6 MiB/s wr, 4 op/s 2021-07-31T02:11:18.828 INFO:tasks.workunit.client.0.smithi045.stdout: 365: throughput=4.99MB/sec pending data=0 2021-07-31T02:11:18.829 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-_g1sZ6TnWDsC7X7 off=2121574 len=1879455 2021-07-31T02:11:18.833 INFO:tasks.workunit.client.0.smithi045.stdout:op 1879 completed, throughput=4.99MB/sec 2021-07-31T02:11:18.833 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-CzELxnDoQ7FjDpj off=13516282 len=883423 2021-07-31T02:11:18.838 INFO:tasks.workunit.client.0.smithi045.stdout:op 1880 completed, throughput=4.99MB/sec 2021-07-31T02:11:18.838 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-R_2tukePLtHJW48 off=87666 len=423899 2021-07-31T02:11:18.838 INFO:tasks.workunit.client.0.smithi045.stdout:op 1881 completed, throughput=4.99MB/sec 2021-07-31T02:11:18.839 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-798lWbByYDT-NcV off=18350615 len=798952 2021-07-31T02:11:18.840 INFO:tasks.workunit.client.0.smithi045.stdout:op 1882 completed, throughput=5MB/sec 2021-07-31T02:11:18.841 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-SYhskRfl8bUumVf off=283156 len=612565 2021-07-31T02:11:18.843 INFO:tasks.workunit.client.0.smithi045.stdout:op 1883 completed, throughput=5MB/sec 2021-07-31T02:11:18.843 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-rw0LB9Hr-lYvBIf off=417734 len=848433 2021-07-31T02:11:18.847 INFO:tasks.workunit.client.0.smithi045.stdout:op 1884 completed, throughput=5MB/sec 2021-07-31T02:11:19.848 INFO:tasks.workunit.client.0.smithi045.stdout: 366: throughput=4.99MB/sec pending data=0 2021-07-31T02:11:19.848 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-5djuiUzak6IBI-Q off=3795819 len=1638655 2021-07-31T02:11:19.849 INFO:tasks.workunit.client.0.smithi045.stdout:op 1885 completed, throughput=4.99MB/sec 2021-07-31T02:11:19.850 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-8okU43eeHkaFrb_ off=10032786 len=578926 2021-07-31T02:11:19.854 INFO:tasks.workunit.client.0.smithi045.stdout:op 1886 completed, throughput=4.99MB/sec 2021-07-31T02:11:19.854 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ffa2eFzNnZBFTyS off=3740051 len=469977 2021-07-31T02:11:19.860 INFO:tasks.workunit.client.0.smithi045.stdout:op 1887 completed, throughput=4.99MB/sec 2021-07-31T02:11:19.860 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-dFd8Vb4OhFD4j39 off=2779661 len=1143893 2021-07-31T02:11:19.861 INFO:tasks.workunit.client.0.smithi045.stdout:op 1888 completed, throughput=5MB/sec 2021-07-31T02:11:19.861 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-AVRkb_gxV7Qid_N off=9602151 len=1723158 2021-07-31T02:11:19.869 INFO:tasks.workunit.client.0.smithi045.stdout:op 1889 completed, throughput=5MB/sec 2021-07-31T02:11:20.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:20 smithi154 conmon[34553]: cluster 2021-07-31T02:11:20.136195+0000 mgr.x (mgr.24542) 1030 : cluster [DBG] pgmap v1595: 9 pgs: 9 active+clean; 127 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 4.3 MiB/s rd, 2.0 MiB/s wr, 6 op/s 2021-07-31T02:11:20.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:20 smithi045 conmon[33655]: cluster 2021-07-31T02:11:20.136195+0000 mgr.x (mgr.24542) 1030 : cluster [DBG] 2021-07-31T02:11:20.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:20 smithi045 conmon[33655]: pgmap v1595: 9 pgs: 9 active+clean; 127 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 4.3 MiB/s rd, 2.0 MiB/s wr, 6 op/s 2021-07-31T02:11:20.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:20 smithi045 conmon[39718]: cluster 2021-07-31T02:11:20.136195+0000 2021-07-31T02:11:20.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:20 smithi045 conmon[39718]: mgr.x (mgr.24542) 1030 : cluster [DBG] pgmap v1595: 9 pgs: 9 active+clean; 127 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 4.3 MiB/s rd, 2.0 MiB/s wr, 6 op/s 2021-07-31T02:11:20.869 INFO:tasks.workunit.client.0.smithi045.stdout: 367: throughput=4.99MB/sec pending data=0 2021-07-31T02:11:20.869 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-YWHF8OaiecrCk2B off=310133 len=130051 2021-07-31T02:11:20.873 INFO:tasks.workunit.client.0.smithi045.stdout:op 1890 completed, throughput=4.99MB/sec 2021-07-31T02:11:20.873 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ZYrwd-ODhfERACw off=5844470 len=1109976 2021-07-31T02:11:20.878 INFO:tasks.workunit.client.0.smithi045.stdout:op 1891 completed, throughput=4.99MB/sec 2021-07-31T02:11:20.878 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Jthwa4edvwUZipj off=18340919 len=315953 2021-07-31T02:11:20.881 INFO:tasks.workunit.client.0.smithi045.stdout:op 1892 completed, throughput=4.99MB/sec 2021-07-31T02:11:20.882 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-K93a-o5czImgH4g off=143941 len=1934592 2021-07-31T02:11:20.885 INFO:tasks.workunit.client.0.smithi045.stdout:op 1893 completed, throughput=5MB/sec 2021-07-31T02:11:20.885 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-HDxGKNGMY4WaAIE off=6556204 len=501071 2021-07-31T02:11:20.886 INFO:tasks.workunit.client.0.smithi045.stdout:op 1894 completed, throughput=5MB/sec 2021-07-31T02:11:20.887 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-QkD5nPXOH3Xs9RA off=1730273 len=919332 2021-07-31T02:11:20.888 INFO:tasks.workunit.client.0.smithi045.stdout:op 1895 completed, throughput=5MB/sec 2021-07-31T02:11:20.888 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-5dlyW5nlvDUA6WI off=16933642 len=10355 2021-07-31T02:11:20.890 INFO:tasks.workunit.client.0.smithi045.stdout:op 1896 completed, throughput=5MB/sec 2021-07-31T02:11:20.890 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-06HjYjIOmLtwaD3 off=1319567 len=2064320 2021-07-31T02:11:20.900 INFO:tasks.workunit.client.0.smithi045.stdout:op 1897 completed, throughput=5.01MB/sec 2021-07-31T02:11:21.901 INFO:tasks.workunit.client.0.smithi045.stdout: 368: throughput=4.99MB/sec pending data=0 2021-07-31T02:11:21.901 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-kGljQU4a5KoBuPn off=4115185 len=1072087 2021-07-31T02:11:21.903 INFO:tasks.workunit.client.0.smithi045.stdout:op 1898 completed, throughput=4.99MB/sec 2021-07-31T02:11:21.903 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-fgGFwLHtgk7WQDB off=4338232 len=483699 2021-07-31T02:11:21.907 INFO:tasks.workunit.client.0.smithi045.stdout:op 1899 completed, throughput=5MB/sec 2021-07-31T02:11:21.908 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-9rkQXmKYGR48AVC off=99812 len=204667 2021-07-31T02:11:21.910 INFO:tasks.workunit.client.0.smithi045.stdout:op 1900 completed, throughput=5MB/sec 2021-07-31T02:11:21.911 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Asjx1tqfj-CJcPR off=3478966 len=594047 2021-07-31T02:11:21.912 INFO:tasks.workunit.client.0.smithi045.stdout:op 1901 completed, throughput=5MB/sec 2021-07-31T02:11:21.912 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-iD01HAj6xh3ruMB off=14219454 len=776238 2021-07-31T02:11:21.916 INFO:tasks.workunit.client.0.smithi045.stdout:op 1902 completed, throughput=5MB/sec 2021-07-31T02:11:21.917 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ektveztNpP8a_Xl off=8623157 len=1423566 2021-07-31T02:11:21.923 INFO:tasks.workunit.client.0.smithi045.stdout:op 1903 completed, throughput=5MB/sec 2021-07-31T02:11:22.923 INFO:tasks.workunit.client.0.smithi045.stdout: 369: throughput=4.99MB/sec pending data=0 2021-07-31T02:11:22.924 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Yr4E_MEoR8MDMu_ off=3147787 len=162288 2021-07-31T02:11:22.925 INFO:tasks.workunit.client.0.smithi045.stdout:op 1904 completed, throughput=4.99MB/sec 2021-07-31T02:11:22.925 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Ao_RzBTlUjh27Mw off=1031881 len=1597803 2021-07-31T02:11:22.927 INFO:tasks.workunit.client.0.smithi045.stdout:op 1905 completed, throughput=4.99MB/sec 2021-07-31T02:11:22.927 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-o8v0dZO29DpzNvh off=16002330 len=1357077 2021-07-31T02:11:22.928 INFO:tasks.workunit.client.0.smithi045.stdout:op 1906 completed, throughput=5MB/sec 2021-07-31T02:11:22.929 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-IP5vu2p84LHEsHB off=3793018 len=528918 2021-07-31T02:11:22.933 INFO:tasks.workunit.client.0.smithi045.stdout:op 1907 completed, throughput=5MB/sec 2021-07-31T02:11:22.933 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-8c_NOUoV6OYLiqU off=16889123 len=22076 2021-07-31T02:11:22.934 INFO:tasks.workunit.client.0.smithi045.stdout:op 1908 completed, throughput=5MB/sec 2021-07-31T02:11:22.934 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-7HNItv6lYtRTEEi off=10980297 len=963550 2021-07-31T02:11:22.939 INFO:tasks.workunit.client.0.smithi045.stdout:op 1909 completed, throughput=5MB/sec 2021-07-31T02:11:23.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:22 smithi154 conmon[34553]: cluster 2021-07-31T02:11:22.136717+0000 mgr.x (mgr.24542) 1031 : cluster [DBG] pgmap v1596: 9 pgs: 9 active+clean; 127 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 3.0 MiB/s rd, 1.2 MiB/s wr, 4 op/s 2021-07-31T02:11:23.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:22 smithi154 conmon[34553]: 2021-07-31T02:11:23.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:22 smithi045 conmon[39718]: cluster 2021-07-31T02:11:22.136717+0000 mgr.x ( 2021-07-31T02:11:23.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:22 smithi045 conmon[39718]: mgr.24542) 1031 : cluster [DBG] pgmap v1596: 9 pgs: 9 active+clean; 127 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 3.0 MiB/s rd, 1.2 MiB/s wr, 4 op/s 2021-07-31T02:11:23.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:22 smithi045 conmon[33655]: cluster 2021-07-31T02:11:22.136717+0000 mgr.x 2021-07-31T02:11:23.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:22 smithi045 conmon[33655]: (mgr.24542) 1031 : cluster [DBG] pgmap v1596: 9 pgs: 9 active+clean; 127 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 3.0 MiB/s rd, 1.2 MiB/s wr, 4 op/s 2021-07-31T02:11:23.939 INFO:tasks.workunit.client.0.smithi045.stdout: 370: throughput=4.99MB/sec pending data=0 2021-07-31T02:11:23.940 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-R6NrF8m_sfhMwx9 off=7678788 len=1005433 2021-07-31T02:11:23.945 INFO:tasks.workunit.client.0.smithi045.stdout:op 1910 completed, throughput=4.99MB/sec 2021-07-31T02:11:23.946 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-mlWNoeRPkctXUYV off=32453 len=1628385 2021-07-31T02:11:23.955 INFO:tasks.workunit.client.0.smithi045.stdout:op 1911 completed, throughput=4.99MB/sec 2021-07-31T02:11:23.955 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-zTxjwS9PCPNPYok off=6038395 len=1210433 2021-07-31T02:11:23.958 INFO:tasks.workunit.client.0.smithi045.stdout:op 1912 completed, throughput=5MB/sec 2021-07-31T02:11:23.958 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-fCfEqnYEJ3mz2Hg off=7681442 len=1343246 2021-07-31T02:11:23.964 INFO:tasks.workunit.client.0.smithi045.stdout:op 1913 completed, throughput=5MB/sec 2021-07-31T02:11:24.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:11:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:11:23.549Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:11:24.065 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:11:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:11:23.549Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:11:24.065 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:11:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:11:23.549Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:11:24.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:24 smithi154 conmon[34553]: cluster 2021-07-31T02:11:24.137213+0000 mgr.x (mgr.24542) 1032 : cluster [DBG] pgmap v1597: 9 pgs: 9 active+clean; 127 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 3.9 MiB/s rd, 2.1 MiB/s wr, 6 op/s 2021-07-31T02:11:24.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:24 smithi045 conmon[39718]: cluster 2021-07-31T02:11:24.137213+0000 mgr.x (mgr. 2021-07-31T02:11:24.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:24 smithi045 conmon[39718]: 24542) 1032 : cluster [DBG] pgmap v1597: 9 pgs: 9 active+clean; 127 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 3.9 MiB/s rd, 2.1 MiB/s wr, 6 op/s 2021-07-31T02:11:24.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:24 smithi045 conmon[33655]: cluster 2021-07-31T02:11:24.137213+0000 mgr.x (mgr.24542) 2021-07-31T02:11:24.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:24 smithi045 conmon[33655]: 1032 : cluster [DBG] pgmap v1597: 9 pgs: 9 active+clean; 127 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 3.9 MiB/s rd, 2.1 MiB/s wr, 6 op/s 2021-07-31T02:11:24.964 INFO:tasks.workunit.client.0.smithi045.stdout: 371: throughput=4.99MB/sec pending data=0 2021-07-31T02:11:24.965 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-mPvKpQ78CmGTWfv off=17720761 len=156100 2021-07-31T02:11:24.965 INFO:tasks.workunit.client.0.smithi045.stdout:op 1914 completed, throughput=4.99MB/sec 2021-07-31T02:11:24.966 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-XuNmc4FuSAl1V8g off=4815950 len=925798 2021-07-31T02:11:24.971 INFO:tasks.workunit.client.0.smithi045.stdout:op 1915 completed, throughput=4.99MB/sec 2021-07-31T02:11:24.971 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ZDoCFW_0WYtvNFC off=1239916 len=5719 2021-07-31T02:11:24.973 INFO:tasks.workunit.client.0.smithi045.stdout:op 1916 completed, throughput=4.99MB/sec 2021-07-31T02:11:24.974 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-S-VE6ZoNpQ2f-g4 off=12293188 len=1736668 2021-07-31T02:11:24.982 INFO:tasks.workunit.client.0.smithi045.stdout:op 1917 completed, throughput=4.99MB/sec 2021-07-31T02:11:24.983 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-XhEXR9sIqgJiZTZ off=9847327 len=1770945 2021-07-31T02:11:24.989 INFO:tasks.workunit.client.0.smithi045.stdout:op 1918 completed, throughput=5MB/sec 2021-07-31T02:11:24.989 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-KsVaGeDjfGqVChl off=4256847 len=1292166 2021-07-31T02:11:24.991 INFO:tasks.workunit.client.0.smithi045.stdout:op 1919 completed, throughput=5MB/sec 2021-07-31T02:11:25.992 INFO:tasks.workunit.client.0.smithi045.stdout: 372: throughput=4.99MB/sec pending data=0 2021-07-31T02:11:25.993 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-4Y-4umrYiFljPuR off=9530101 len=1504686 2021-07-31T02:11:26.001 INFO:tasks.workunit.client.0.smithi045.stdout:op 1920 completed, throughput=4.99MB/sec 2021-07-31T02:11:26.001 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-SwAY_4e31BGE35L off=8778191 len=1372164 2021-07-31T02:11:26.002 INFO:tasks.workunit.client.0.smithi045.stdout:op 1921 completed, throughput=5MB/sec 2021-07-31T02:11:26.003 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-A-nVmB9nNR5ry22 off=991727 len=1746448 2021-07-31T02:11:26.005 INFO:tasks.workunit.client.0.smithi045.stdout:op 1922 completed, throughput=5MB/sec 2021-07-31T02:11:26.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:26 smithi154 conmon[34553]: cluster 2021-07-31T02:11:26.137789+0000 mgr.x (mgr.24542) 1033 : cluster [DBG] pgmap v1598: 9 pgs: 9 active+clean; 127 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.3 MiB/s wr, 4 op/s 2021-07-31T02:11:26.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:26 smithi045 conmon[39718]: cluster 2021-07-31T02:11:26.137789+0000 mgr.x (mgr.24542 2021-07-31T02:11:26.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:26 smithi045 conmon[39718]: ) 1033 : cluster [DBG] pgmap v1598: 9 pgs: 9 active+clean; 127 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.3 MiB/s wr, 4 op/s 2021-07-31T02:11:26.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:26 smithi045 conmon[33655]: cluster 2021-07-31T02:11:26.137789+0000 mgr.x (mgr.24542) 1033 : cluster [DBG] pgmap v1598: 9 pgs: 9 active+clean; 127 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.3 MiB/s wr, 4 op/s 2021-07-31T02:11:27.006 INFO:tasks.workunit.client.0.smithi045.stdout: 373: throughput=4.99MB/sec pending data=0 2021-07-31T02:11:27.006 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-dwCz9x2a483fg5b off=8466985 len=1299416 2021-07-31T02:11:27.009 INFO:tasks.workunit.client.0.smithi045.stdout:op 1923 completed, throughput=4.99MB/sec 2021-07-31T02:11:27.009 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-DGJX9QXk-g360Sh off=655327 len=1315379 2021-07-31T02:11:27.015 INFO:tasks.workunit.client.0.smithi045.stdout:op 1924 completed, throughput=4.99MB/sec 2021-07-31T02:11:27.015 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-gkiNuvL9vhoyr8Q off=4468601 len=1242361 2021-07-31T02:11:27.023 INFO:tasks.workunit.client.0.smithi045.stdout:op 1925 completed, throughput=5MB/sec 2021-07-31T02:11:27.023 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-T4vsbN5z-OmllLy off=1245477 len=1554609 2021-07-31T02:11:27.030 INFO:tasks.workunit.client.0.smithi045.stdout:op 1926 completed, throughput=5MB/sec 2021-07-31T02:11:28.030 INFO:tasks.workunit.client.0.smithi045.stdout: 374: throughput=4.99MB/sec pending data=0 2021-07-31T02:11:28.030 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-3YPyxr5TLG5yeAb off=10734510 len=350650 2021-07-31T02:11:28.031 INFO:tasks.workunit.client.0.smithi045.stdout:op 1927 completed, throughput=4.99MB/sec 2021-07-31T02:11:28.031 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-kV-js4JYYNYH0AT off=4636535 len=2042214 2021-07-31T02:11:28.043 INFO:tasks.workunit.client.0.smithi045.stdout:op 1928 completed, throughput=4.99MB/sec 2021-07-31T02:11:28.043 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Hj3JNzXThnk8a9J off=181808 len=1966537 2021-07-31T02:11:28.053 INFO:tasks.workunit.client.0.smithi045.stdout:op 1929 completed, throughput=5MB/sec 2021-07-31T02:11:28.053 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-9JESFRVqWcxlDM9 off=5877186 len=1918534 2021-07-31T02:11:28.061 INFO:tasks.workunit.client.0.smithi045.stdout:op 1930 completed, throughput=5MB/sec 2021-07-31T02:11:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:28 smithi154 conmon[34553]: audit 2021-07-31T02:11:28.020084+0000 mon.b (mon.2) 188 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:11:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:28 smithi154 conmon[34553]: audit 2021-07-31T02:11:28.021136+0000 mon.a (mon.0) 1868 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:11:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:28 smithi154 conmon[34553]: audit 2021-07-31T02:11:28.039622+0000 mon.b (mon.2) 2021-07-31T02:11:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:28 smithi154 conmon[34553]: 189 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:11:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:28 smithi154 conmon[34553]: audit 2021-07-31T02:11:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:28 smithi154 conmon[34553]: 2021-07-31T02:11:28. 2021-07-31T02:11:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:28 smithi154 conmon[34553]: 040704+0000 mon.a (mon.0) 1869 : audit 2021-07-31T02:11:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:28 smithi154 conmon[34553]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:11:28.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:28 smithi045 conmon[33655]: audit 2021-07-31T02:11:28.020084+0000 mon.b (mon.2) 2021-07-31T02:11:28.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:28 smithi045 conmon[33655]: 188 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:11:28.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:28 smithi045 conmon[33655]: audit 2021-07-31T02:11:28.021136+0000 mon.a (mon.0) 1868 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:11:28.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:28 smithi045 conmon[33655]: audit 2021-07-31T02:11:28.039622+0000 mon.b (mon 2021-07-31T02:11:28.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:28 smithi045 conmon[33655]: .2) 189 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:11:28.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:28 smithi045 conmon[33655]: audit 2021 2021-07-31T02:11:28.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:28 smithi045 conmon[33655]: -07-31T02:11:28.040704+0000 mon.a 2021-07-31T02:11:28.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:28 smithi045 conmon[33655]: (mon.0) 1869 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:11:28.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:28 smithi045 conmon[39718]: audit 2021-07-31T02:11:28.020084+0000 mon.b (mon. 2021-07-31T02:11:28.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:28 smithi045 conmon[39718]: 2) 188 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:11:28.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:28 smithi045 conmon[39718]: audit 2021-07-31T02:11: 2021-07-31T02:11:28.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:28 smithi045 conmon[39718]: 28.021136+0000 mon.a (mon.0) 1868 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:11:28.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:28 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T02:11:28.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:28 smithi045 conmon[39718]: T02:11:28.039622+0000 mon.b (mon.2) 2021-07-31T02:11:28.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:28 smithi045 conmon[39718]: 189 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:11:28.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:28 smithi045 conmon[39718]: audit 2021 2021-07-31T02:11:28.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:28 smithi045 conmon[39718]: -07-31T02:11:28.040704+0000 mon.a ( 2021-07-31T02:11:28.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:28 smithi045 conmon[39718]: mon.0) 1869 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:11:29.061 INFO:tasks.workunit.client.0.smithi045.stdout: 375: throughput=4.99MB/sec pending data=0 2021-07-31T02:11:29.061 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-t2nfwWf-IGsLD-A off=17679176 len=1979510 2021-07-31T02:11:29.063 INFO:tasks.workunit.client.0.smithi045.stdout:op 1931 completed, throughput=4.99MB/sec 2021-07-31T02:11:29.064 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-8V11C5n1GlK7c79 off=643834 len=1232218 2021-07-31T02:11:29.067 INFO:tasks.workunit.client.0.smithi045.stdout:op 1932 completed, throughput=5MB/sec 2021-07-31T02:11:29.067 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-I0od85IwvL5FwD1 off=132642 len=1502467 2021-07-31T02:11:29.070 INFO:tasks.workunit.client.0.smithi045.stdout:op 1933 completed, throughput=5MB/sec 2021-07-31T02:11:29.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:29 smithi154 conmon[34553]: cluster 2021-07-31T02:11:28.138350+0000 mgr.x (mgr.24542) 1034 : cluster [DBG] pgmap v1599: 9 pgs: 9 active+clean; 127 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.3 MiB/s wr, 4 op/s 2021-07-31T02:11:29.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:29 smithi045 conmon[39718]: cluster 2021-07-31T02:11:28.138350+0000 mgr.x (mgr 2021-07-31T02:11:29.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:29 smithi045 conmon[39718]: .24542) 1034 : cluster [DBG] pgmap v1599: 9 pgs: 9 active+clean; 127 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.3 MiB/s wr, 4 op/s 2021-07-31T02:11:29.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:29 smithi045 conmon[33655]: cluster 2021-07-31T02:11:28.138350+0000 mgr.x (mgr.24542) 1034 : cluster [DBG] pgmap v1599: 9 pgs: 9 active+clean; 127 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.3 MiB/s wr, 4 op/s 2021-07-31T02:11:30.071 INFO:tasks.workunit.client.0.smithi045.stdout: 376: throughput=4.99MB/sec pending data=0 2021-07-31T02:11:30.071 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-9qHtFgoQFdufDjs off=1495456 len=861040 2021-07-31T02:11:30.072 INFO:tasks.workunit.client.0.smithi045.stdout:op 1934 completed, throughput=4.99MB/sec 2021-07-31T02:11:30.072 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-rnW_hYSg8t5xqeD off=7270053 len=1935132 2021-07-31T02:11:30.082 INFO:tasks.workunit.client.0.smithi045.stdout:op 1935 completed, throughput=4.99MB/sec 2021-07-31T02:11:30.082 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-QhYOshkZzzHbqTh off=6483246 len=1784222 2021-07-31T02:11:30.084 INFO:tasks.workunit.client.0.smithi045.stdout:op 1936 completed, throughput=5MB/sec 2021-07-31T02:11:30.084 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-MLGjTEO57_4AjtF off=7056050 len=1457104 2021-07-31T02:11:30.086 INFO:tasks.workunit.client.0.smithi045.stdout:op 1937 completed, throughput=5MB/sec 2021-07-31T02:11:30.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[34553]: cluster 2021-07-31T02:11:30.138783+0000 mgr.x (mgr.24542) 1035 : cluster [DBG] pgmap v1600: 9 pgs: 9 active+clean; 127 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.9 MiB/s wr, 6 op/s 2021-07-31T02:11:30.515 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:30 smithi045 conmon[39718]: cluster 2021-07-31T02:11:30.138783+0000 mgr.x (mgr.24542) 1035 : cluster [DBG] 2021-07-31T02:11:30.515 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:30 smithi045 conmon[39718]: pgmap v1600: 9 pgs: 9 active+clean; 127 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.9 MiB/s wr, 6 op/s 2021-07-31T02:11:30.515 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:30 smithi045 conmon[33655]: cluster 2021-07-31T02:11:30.138783+0000 mgr.x (mgr.24542) 1035 : 2021-07-31T02:11:30.516 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:30 smithi045 conmon[33655]: cluster [DBG] pgmap v1600: 9 pgs: 9 active+clean; 127 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.9 MiB/s wr, 6 op/s 2021-07-31T02:11:31.003 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: debug 2021-07-31T02:11:30.532+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:11:31.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:11:31.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:11:31.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: yield 2021-07-31T02:11:31.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:11:31.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:11:31.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:11:31.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:11:31.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:11:31.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:11:31.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:11:31.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:11:31.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:11:31.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:11:31.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:11:31.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: debug 2021-07-31T02:11:30.543+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:11:31.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:11:31.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:11:31.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: yield 2021-07-31T02:11:31.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:11:31.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:11:31.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:11:31.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:11:31.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:11:31.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:11:31.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:11:31.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:11:31.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:11:31.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:11:31.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:11:31.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: debug 2021-07-31T02:11:30.568+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:11:31.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:11:31.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:11:31.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: yield 2021-07-31T02:11:31.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:11:31.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:11:31.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:11:31.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:11:31.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:11:31.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:11:31.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:11:31.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:11:31.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:11:31.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:11:31.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:11:31.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: debug 2021-07-31T02:11:30.569+0000 7fcb97969700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:11:31.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:11:31.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:11:31.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: yield 2021-07-31T02:11:31.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:11:31.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:11:31.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:11:31.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:11:31.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:11:31.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:11:31.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:11:31.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:11:31.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:11:31.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:11:31.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:11:31.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: 2021-07-31T02:11:31.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:11:31.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: 2021-07-31T02:11:31.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:11:31.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:11:31.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:11:31.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:11:31.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:11:31.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:11:31.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:11:31.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:11:31.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:11:31.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:11:31.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:11:31.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:11:31.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:11:31.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:11:31.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:11:31.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:11:31.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:11:31.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:11:31.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: raise 2021-07-31T02:11:31.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:11:31.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:11:31.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:11:31.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:11:31.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:11:31.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: debug 2021-07-31T02:11:30.577+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:11:31.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:11:31.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:11:31.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: yield 2021-07-31T02:11:31.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:11:31.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:11:31.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:11:31.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:11:31.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:11:31.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:11:31.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:11:31.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:11:31.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:11:31.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:11:31.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:11:31.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: debug 2021-07-31T02:11:30.578+0000 7fcb9816a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:11:31.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:11:31.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:11:31.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: yield 2021-07-31T02:11:31.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:11:31.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:11:31.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:11:31.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:11:31.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:11:31.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:11:31.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:11:31.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:11:31.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:11:31.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:11:31.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:11:31.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: 2021-07-31T02:11:31.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:11:31.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: 2021-07-31T02:11:31.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:11:31.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:11:31.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:11:31.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:11:31.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:11:31.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:11:31.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:11:31.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:11:31.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:11:31.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:11:31.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:11:31.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:11:31.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:11:31.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:11:31.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:11:31.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:11:31.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:11:31.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:11:31.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: raise 2021-07-31T02:11:31.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:11:31.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:11:31.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:11:31.035 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:11:31.035 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:11:30 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:11:31.085 INFO:tasks.workunit.client.0.smithi045.stdout: 377: throughput=4.99MB/sec pending data=0 2021-07-31T02:11:31.086 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-PVK7g2Fb71QBsva off=4407978 len=920061 2021-07-31T02:11:31.087 INFO:tasks.workunit.client.0.smithi045.stdout:op 1938 completed, throughput=4.99MB/sec 2021-07-31T02:11:31.087 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-yXkJJZ-pyhq383t off=5776641 len=1036592 2021-07-31T02:11:31.089 INFO:tasks.workunit.client.0.smithi045.stdout:op 1939 completed, throughput=4.99MB/sec 2021-07-31T02:11:31.090 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-90BGvqmzwUgY3o2 off=1174286 len=144777 2021-07-31T02:11:31.092 INFO:tasks.workunit.client.0.smithi045.stdout:op 1940 completed, throughput=4.99MB/sec 2021-07-31T02:11:31.092 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-M1Cw6CIQrAnKr1i off=6614398 len=1756724 2021-07-31T02:11:31.101 INFO:tasks.workunit.client.0.smithi045.stdout:op 1941 completed, throughput=5MB/sec 2021-07-31T02:11:31.102 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-oubRgnZ0h5lCBhs off=18819105 len=1586066 2021-07-31T02:11:31.109 INFO:tasks.workunit.client.0.smithi045.stdout:op 1942 completed, throughput=5MB/sec 2021-07-31T02:11:31.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: audit 2021-07-31T02:11:30.487169+0000 mon.b (mon.2) 190 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:11:31.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: audit 2021-07-31T02:11:30.488572+0000 mon.b (mon.2) 191 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:11:31.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: audit 2021-07-31T02:11:30.489534+0000 mon.b (mon.2) 192 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:11:31.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: cephadm 2021-07-31T 2021-07-31T02:11:31.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: 02:11:30.490265+0000 mgr.x (mgr.24542) 1036 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:11:31.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:11:30.533397+0000 mgr.x (mgr. 2021-07-31T02:11:31.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: 24542) 1037 : cephadm [ERR] 2021-07-31T02:11:31.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:11:31.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:11:31.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: yield 2021-07-31T02:11:31.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:11:31.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:11:31.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:11:31.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:11:31.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:11:31.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:11:31.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:11:31.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:11:31.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:11:31.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: task.result() 2021-07-31T02:11:31.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:11:31.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:11:30.533782+0000 mgr.x (mgr.24542) 1038 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:11:31.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:11:30.544625+0000 mgr.x (mgr.24542) 1039 : cephadm [ERR] 2021-07-31T02:11:31.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:11:31.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:11:31.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: yield 2021-07-31T02:11:31.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:11:31.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:11:31.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:11:31.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:11:31.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:11:31.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:11:31.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:11:31.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:11:31.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:11:31.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: task.result() 2021-07-31T02:11:31.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:11:31.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:11:30.545447+0000 mgr.x (mgr.24542) 1040 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:11:31.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:11:30.569714+0000 mgr.x (mgr.24542) 1041 : cephadm [ERR] 2021-07-31T02:11:31.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:11:31.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:11:31.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: yield 2021-07-31T02:11:31.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:11:31.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:11:31.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:11:31.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:11:31.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:11:31.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:11:31.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:11:31.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:11:31.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:11:31.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: task.result() 2021-07-31T02:11:31.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:11:31.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:11:30.570095+0000 mgr.x (mgr.24542) 1042 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:11:31.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:11:31.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:11:31.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: yield 2021-07-31T02:11:31.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:11:31.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:11:31.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:11:31.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:11:31.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:11:31.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:11:31.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:11:31.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:11:31.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:11:31.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: task.result() 2021-07-31T02:11:31.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:11:31.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: 2021-07-31T02:11:31.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:11:31.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: 2021-07-31T02:11:31.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:11:31.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:11:31.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:11:31.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:11:31.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:11:31.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:11:31.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:11:31.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:11:31.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:11:31.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:11:31.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:11:31.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:11:31.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:11:31.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:11:31.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:11:31.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:11:31.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:11:31.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:11:31.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: raise 2021-07-31T02:11:31.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:11:31.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:11:31.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:11:31.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:11:31.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:11:31.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:11:30.578549+0000 mgr.x (mgr.24542) 1043 : cephadm [ERR] 2021-07-31T02:11:31.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:11:31.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:11:31.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: yield 2021-07-31T02:11:31.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:11:31.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:11:31.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:11:31.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:11:31.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:11:31.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:11:31.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:11:31.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:11:31.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:11:31.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: task.result() 2021-07-31T02:11:31.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:11:31.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:11:30.579296+0000 mgr.x (mgr.24542) 1044 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:11:31.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:11:31.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:11:31.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: yield 2021-07-31T02:11:31.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:11:31.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:11:31.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:11:31.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:11:31.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:11:31.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:11:31.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:11:31.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:11:31.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:11:31.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: task.result() 2021-07-31T02:11:31.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:11:31.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: 2021-07-31T02:11:31.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:11:31.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: 2021-07-31T02:11:31.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:11:31.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:11:31.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:11:31.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:11:31.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:11:31.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:11:31.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:11:31.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:11:31.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:11:31.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:11:31.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:11:31.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:11:31.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:11:31.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:11:31.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:11:31.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:11:31.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:11:31.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:11:31.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: raise 2021-07-31T02:11:31.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:11:31.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:11:31.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:11:31.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:11:31.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:31 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:11:31.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: audit 2021-07-31T02:11:30.487169+0000 mon.b (mon.2) 190 : audit 2021-07-31T02:11:31.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:11:31.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: audit 2021-07-31T02:11:30.488572+0000 mon.b (mon.2) 191 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:11:31.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: audit 2021-07-31T02:11:30.489534+0000 mon.b (mon.2) 192 : audit 2021-07-31T02:11:31.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:11:31.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: cephadm 2021- 2021-07-31T02:11:31.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: 07-31T02:11:30.490265+0000 mgr.x (mgr.24542) 1036 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:11:31.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:11 2021-07-31T02:11:31.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: :30.533397+0000 mgr.x (mgr.24542) 1037 : cephadm [ERR] 2021-07-31T02:11:31.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:11:31.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:11:31.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: yield 2021-07-31T02:11:31.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:11:31.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:11:31.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:11:31.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:11:31.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:11:31.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:11:31.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:11:31.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:11:31.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:11:31.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: task.result() 2021-07-31T02:11:31.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:11:31.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: 2021-07-31T02:11:31.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:11:30.533782+0000 mgr.x (mgr.24542) 1038 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:11:31.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:11:30.544625+0000 mgr.x (mgr.24542) 1039 : cephadm [ERR] 2021-07-31T02:11:31.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:11:31.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:11:31.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: yield 2021-07-31T02:11:31.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:11:31.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:11:31.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:11:31.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:11:31.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:11:31.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:11:31.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:11:31.545 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: audit 2021-07-31T02:11:30.487169+0000 2021-07-31T02:11:31.545 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: mon.b (mon.2) 190 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:11:31.546 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: audit 2021-07-31T02:11:30.488572+0000 mon.b (mon.2) 191 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:11:31.546 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: audit 2021- 2021-07-31T02:11:31.546 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: 07-31T02:11:30.489534+0000 mon.b (mon.2) 192 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:11:31.546 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: cephadm 2021- 2021-07-31T02:11:31.546 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: 07-31T02:11:30.490265+0000 mgr.x (mgr.24542) 1036 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:11:31.547 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: cephadm 2021 2021-07-31T02:11:31.547 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: -07-31T02:11:30.533397+0000 mgr.x (mgr.24542) 1037 : cephadm [ERR] 2021-07-31T02:11:31.547 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:11:31.547 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:11:31.548 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: yield 2021-07-31T02:11:31.548 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:11:31.548 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:11:31.548 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:11:31.548 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:11:31.549 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:11:31.549 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:11:31.549 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:11:31.549 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:11:31.550 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:11:31.550 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: task.result() 2021-07-31T02:11:31.550 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:11:31.550 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:11:30.533782+0000 mgr.x (mgr.24542) 1038 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:11:31.551 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:11:30.544625+0000 mgr.x (mgr.24542) 1039 : cephadm [ERR] 2021-07-31T02:11:31.551 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:11:31.551 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:11:31.551 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: yield 2021-07-31T02:11:31.551 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:11:31.552 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:11:31.552 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:11:31.552 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:11:31.552 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:11:31.552 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:11:31.553 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:11:31.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:11:31.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:11:31.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: task.result() 2021-07-31T02:11:31.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:11:31.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:11:31.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: 2021-07-31T02:11:30.545447+0000 mgr.x (mgr.24542) 1040 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:11:31.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:11:30.569714+0000 mgr.x (mgr.24542) 1041 : cephadm [ERR] 2021-07-31T02:11:31.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:11:31.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:11:31.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: yield 2021-07-31T02:11:31.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:11:31.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:11:31.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:11:31.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:11:31.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:11:31.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:11:31.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:11:31.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:11:31.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:11:31.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: task.result() 2021-07-31T02:11:31.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:11:31.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:11:30.570095+0000 mgr.x (mgr.24542) 1042 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:11:31.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:11:31.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:11:31.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: yield 2021-07-31T02:11:31.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:11:31.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:11:31.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:11:31.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:11:31.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:11:31.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:11:31.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:11:31.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:11:31.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:11:31.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: task.result() 2021-07-31T02:11:31.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:11:31.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: 2021-07-31T02:11:31.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:11:31.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: 2021-07-31T02:11:31.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:11:31.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:11:31.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:11:31.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:11:31.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:11:31.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:11:31.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:11:31.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:11:31.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:11:31.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:11:31.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:11:31.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:11:31.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:11:31.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:11:31.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:11:31.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:11:31.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:11:31.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:11:31.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: raise 2021-07-31T02:11:31.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:11:31.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:11:31.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:11:31.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:11:31.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:11:31.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:11:30.578549+0000 mgr.x ( 2021-07-31T02:11:31.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: mgr.24542) 1043 : cephadm [ERR] 2021-07-31T02:11:31.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:11:31.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:11:31.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: yield 2021-07-31T02:11:31.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:11:31.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:11:31.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:11:31.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:11:31.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:11:31.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:11:31.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:11:31.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:11:31.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:11:31.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: task.result() 2021-07-31T02:11:31.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:11:31.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:11:30.579296+0000 mgr.x (mgr.24542) 1044 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:11:31.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:11:31.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:11:31.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:11:31.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: task.result() 2021-07-31T02:11:31.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:11:31.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:11:30.545447+0000 mgr.x (mgr.24542) 1040 : cephadm [INF] 2021-07-31T02:11:31.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:11:31.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:11:30.569714+0000 mgr.x (mgr.24542) 1041 : cephadm [ERR] 2021-07-31T02:11:31.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:11:31.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:11:31.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: yield 2021-07-31T02:11:31.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:11:31.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:11:31.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:11:31.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:11:31.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:11:31.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:11:31.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:11:31.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:11:31.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:11:31.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: task.result() 2021-07-31T02:11:31.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:11:31.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:11:30.570095+0000 mgr.x (mgr.24542) 1042 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:11:31.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:11:31.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:11:31.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: yield 2021-07-31T02:11:31.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:11:31.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:11:31.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:11:31.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:11:31.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:11:31.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:11:31.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:11:31.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:11:31.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:11:31.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: task.result() 2021-07-31T02:11:31.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:11:31.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: 2021-07-31T02:11:31.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:11:31.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: 2021-07-31T02:11:31.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:11:31.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:11:31.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:11:31.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:11:31.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: yield 2021-07-31T02:11:31.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:11:31.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:11:31.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:11:31.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:11:31.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:11:31.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:11:31.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:11:31.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:11:31.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:11:31.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: task.result() 2021-07-31T02:11:31.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:11:31.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: 2021-07-31T02:11:31.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:11:31.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: 2021-07-31T02:11:31.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:11:31.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:11:31.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:11:31.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:11:31.585 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:11:31.585 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:11:31.585 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:11:31.585 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:11:31.585 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:11:31.586 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:11:31.586 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:11:31.586 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:11:31.586 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:11:31.587 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:11:31.587 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:11:31.587 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:11:31.587 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:11:31.587 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:11:31.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: raise 2021-07-31T02:11:31.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:11:31.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:11:31.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:11:31.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:11:31.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:11:31.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:11:31.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:11:31.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:11:31.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:11:31.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:11:31.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:11:31.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:11:31.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:11:31.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:11:31.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:11:31.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:11:31.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:11:31.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:11:31.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:11:31.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:11:31.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: raise 2021-07-31T02:11:31.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:11:31.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:11:31.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:11:31.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:11:31.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:11:31.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: cephadm 2021-07-31T02 2021-07-31T02:11:31.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: :11:30.578549+0000 mgr.x (mgr.24542) 1043 : cephadm [ERR] 2021-07-31T02:11:31.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:11:31.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:11:31.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: yield 2021-07-31T02:11:31.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:11:31.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:11:31.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:11:31.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:11:31.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:11:31.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:11:31.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:11:31.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:11:31.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:11:31.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: task.result() 2021-07-31T02:11:31.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:11:31.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:11:30.579296+0000 mgr.x (mgr.24542) 1044 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:11:31.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:11:31.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:11:31.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: yield 2021-07-31T02:11:31.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:11:31.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:11:31.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:11:31.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:11:31.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:11:31.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:11:31.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:11:31.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:11:31.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:11:31.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: task.result() 2021-07-31T02:11:31.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:11:31.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: 2021-07-31T02:11:31.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:11:31.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: 2021-07-31T02:11:31.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:11:31.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:11:31.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:11:31.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:11:31.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:11:31.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:11:31.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:11:31.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:11:31.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:11:31.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:11:31.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:11:31.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:11:31.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:11:31.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:11:31.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:11:31.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:11:31.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:11:31.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:11:31.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: raise 2021-07-31T02:11:31.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:11:31.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:11:31.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:11:31.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:11:31.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:31 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:11:32.110 INFO:tasks.workunit.client.0.smithi045.stdout: 378: throughput=4.99MB/sec pending data=0 2021-07-31T02:11:32.110 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-_uniGlZIF0C6thd off=635380 len=944870 2021-07-31T02:11:32.111 INFO:tasks.workunit.client.0.smithi045.stdout:op 1943 completed, throughput=4.99MB/sec 2021-07-31T02:11:32.112 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ht2N1q_84de77H7 off=5721110 len=320625 2021-07-31T02:11:32.112 INFO:tasks.workunit.client.0.smithi045.stdout:op 1944 completed, throughput=4.99MB/sec 2021-07-31T02:11:32.113 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-L06khN7vmsZghCW off=20838946 len=484529 2021-07-31T02:11:32.116 INFO:tasks.workunit.client.0.smithi045.stdout:op 1945 completed, throughput=4.99MB/sec 2021-07-31T02:11:32.116 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-wDvfYPrXB92HFU6 off=2635009 len=1759472 2021-07-31T02:11:32.119 INFO:tasks.workunit.client.0.smithi045.stdout:op 1946 completed, throughput=5MB/sec 2021-07-31T02:11:32.119 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-27OtLgSRigaqwC7 off=2166586 len=564986 2021-07-31T02:11:32.120 INFO:tasks.workunit.client.0.smithi045.stdout:op 1947 completed, throughput=5MB/sec 2021-07-31T02:11:32.120 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-qXutj75xTdFZUxD off=6580236 len=436788 2021-07-31T02:11:32.124 INFO:tasks.workunit.client.0.smithi045.stdout:op 1948 completed, throughput=5MB/sec 2021-07-31T02:11:33.124 INFO:tasks.workunit.client.0.smithi045.stdout: 379: throughput=4.99MB/sec pending data=0 2021-07-31T02:11:33.125 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-0BBhixC4uPOb1hH off=3694340 len=2014992 2021-07-31T02:11:33.128 INFO:tasks.workunit.client.0.smithi045.stdout:op 1949 completed, throughput=4.99MB/sec 2021-07-31T02:11:33.129 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-3UTJW9hQD_8i2Ud off=7737776 len=526194 2021-07-31T02:11:33.130 INFO:tasks.workunit.client.0.smithi045.stdout:op 1950 completed, throughput=4.99MB/sec 2021-07-31T02:11:33.130 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-nGX5zOYhLWz7Ja1 off=12857 len=839910 2021-07-31T02:11:33.135 INFO:tasks.workunit.client.0.smithi045.stdout:op 1951 completed, throughput=5MB/sec 2021-07-31T02:11:33.135 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-vJ0X_yuV7Sr36UY off=4674936 len=1182784 2021-07-31T02:11:33.136 INFO:tasks.workunit.client.0.smithi045.stdout:op 1952 completed, throughput=5MB/sec 2021-07-31T02:11:33.137 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-d9Mz7ryY5-Ko9d6 off=17206137 len=1180152 2021-07-31T02:11:33.138 INFO:tasks.workunit.client.0.smithi045.stdout:op 1953 completed, throughput=5MB/sec 2021-07-31T02:11:33.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:32 smithi154 conmon[34553]: cluster 2021-07-31T02:11:32.139248+0000 mgr.x (mgr.24542) 1045 : cluster [DBG] pgmap v1601: 9 pgs: 9 active+clean; 127 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.5 MiB/s rd, 2.4 MiB/s wr, 3 op/s 2021-07-31T02:11:33.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:32 smithi045 conmon[33655]: cluster 2021-07-31T02:11:32.139248+0000 mgr.x (mgr.24542) 1045 : cluster [DBG] pgmap v1601: 9 pgs: 9 active+clean; 127 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.5 MiB/s rd, 2.4 MiB/s wr, 3 op/s 2021-07-31T02:11:33.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:32 smithi045 conmon[39718]: cluster 2021-07-31T02:11:32.139248+0000 mgr.x (mgr.24542) 1045 2021-07-31T02:11:33.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:32 smithi045 conmon[39718]: : cluster [DBG] pgmap v1601: 9 pgs: 9 active+clean; 127 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.5 MiB/s rd, 2.4 MiB/s wr, 3 op/s 2021-07-31T02:11:34.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:11:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:11:33.550Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:11:34.027 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:11:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:11:33.550Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:11:34.027 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:11:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:11:33.550Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:11:34.139 INFO:tasks.workunit.client.0.smithi045.stdout: 380: throughput=4.99MB/sec pending data=0 2021-07-31T02:11:34.139 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ny8JH0022yKP0Mx off=11383879 len=46769 2021-07-31T02:11:34.142 INFO:tasks.workunit.client.0.smithi045.stdout:op 1954 completed, throughput=4.99MB/sec 2021-07-31T02:11:34.142 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-y-OKoXcgkv4yZzr off=2561692 len=813172 2021-07-31T02:11:34.147 INFO:tasks.workunit.client.0.smithi045.stdout:op 1955 completed, throughput=4.99MB/sec 2021-07-31T02:11:34.147 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-rA9_iRKnBvUpk4R off=2047911 len=646646 2021-07-31T02:11:34.148 INFO:tasks.workunit.client.0.smithi045.stdout:op 1956 completed, throughput=4.99MB/sec 2021-07-31T02:11:34.149 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-k577Ih0VQ9CYfAy off=13222806 len=1669903 2021-07-31T02:11:34.150 INFO:tasks.workunit.client.0.smithi045.stdout:op 1957 completed, throughput=5MB/sec 2021-07-31T02:11:34.150 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-C8z6DEicSAmqxW2 off=6536327 len=1031785 2021-07-31T02:11:34.152 INFO:tasks.workunit.client.0.smithi045.stdout:op 1958 completed, throughput=5MB/sec 2021-07-31T02:11:34.153 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-E7RXN-G8etsWC_p off=2024991 len=1459200 2021-07-31T02:11:34.159 INFO:tasks.workunit.client.0.smithi045.stdout:op 1959 completed, throughput=5MB/sec 2021-07-31T02:11:34.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:34 smithi154 conmon[34553]: cluster 2021-07-31T02:11:34.139749+0000 mgr.x (mgr.24542) 1046 : cluster [DBG] pgmap v1602: 9 pgs: 9 active+clean; 127 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 3.0 MiB/s rd, 3.0 MiB/s wr, 5 op/s 2021-07-31T02:11:34.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:34 smithi045 conmon[39718]: cluster 2021-07-31T02:11:34.139749+0000 mgr.x ( 2021-07-31T02:11:34.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:34 smithi045 conmon[39718]: mgr.24542) 1046 : cluster [DBG] pgmap v1602: 9 pgs: 9 active+clean; 127 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 3.0 MiB/s rd, 3.0 MiB/s wr, 5 op/s 2021-07-31T02:11:34.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:34 smithi045 conmon[33655]: cluster 2021-07-31T02:11:34.139749+0000 mgr.x (mgr.24542) 2021-07-31T02:11:34.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:34 smithi045 conmon[33655]: 1046 : cluster [DBG] pgmap v1602: 9 pgs: 9 active+clean; 127 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 3.0 MiB/s rd, 3.0 MiB/s wr, 5 op/s 2021-07-31T02:11:35.159 INFO:tasks.workunit.client.0.smithi045.stdout: 381: throughput=4.99MB/sec pending data=0 2021-07-31T02:11:35.160 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-4IcSUHLKh9FkPxv off=163891 len=971182 2021-07-31T02:11:35.165 INFO:tasks.workunit.client.0.smithi045.stdout:op 1960 completed, throughput=4.99MB/sec 2021-07-31T02:11:35.165 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-xFf3fjFtggOYlUp off=12782597 len=1272725 2021-07-31T02:11:35.171 INFO:tasks.workunit.client.0.smithi045.stdout:op 1961 completed, throughput=5MB/sec 2021-07-31T02:11:35.172 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-_Bc1q9531rKVXJJ off=2964219 len=2060234 2021-07-31T02:11:35.175 INFO:tasks.workunit.client.0.smithi045.stdout:op 1962 completed, throughput=5MB/sec 2021-07-31T02:11:36.176 INFO:tasks.workunit.client.0.smithi045.stdout: 382: throughput=4.99MB/sec pending data=0 2021-07-31T02:11:36.177 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-_UqK7Ce6NBvEtNh off=2333087 len=805144 2021-07-31T02:11:36.178 INFO:tasks.workunit.client.0.smithi045.stdout:op 1963 completed, throughput=4.99MB/sec 2021-07-31T02:11:36.178 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-vlxbByxzlvQNOIU off=5503879 len=607246 2021-07-31T02:11:36.180 INFO:tasks.workunit.client.0.smithi045.stdout:op 1964 completed, throughput=4.99MB/sec 2021-07-31T02:11:36.180 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-dOvzKICEXbdjQD_ off=13001428 len=1519070 2021-07-31T02:11:36.189 INFO:tasks.workunit.client.0.smithi045.stdout:op 1965 completed, throughput=4.99MB/sec 2021-07-31T02:11:36.189 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-w3O8MHO4mqWF4jt off=7464427 len=1286113 2021-07-31T02:11:36.195 INFO:tasks.workunit.client.0.smithi045.stdout:op 1966 completed, throughput=5MB/sec 2021-07-31T02:11:36.195 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-_IlJbfWqmewJYOZ off=9178994 len=1721915 2021-07-31T02:11:36.198 INFO:tasks.workunit.client.0.smithi045.stdout:op 1967 completed, throughput=5MB/sec 2021-07-31T02:11:36.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:36 smithi154 conmon[34553]: cluster 2021-07-31T02:11:36.140133+0000 mgr.x (mgr.24542) 1047 : cluster [DBG] pgmap v1603: 9 pgs: 9 active+clean; 127 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.2 MiB/s wr, 3 op/s 2021-07-31T02:11:36.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:36 smithi045 conmon[33655]: cluster 2021-07-31T02:11:36.140133+0000 mgr.x (mgr.24542) 1047 : 2021-07-31T02:11:36.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:36 smithi045 conmon[33655]: cluster [DBG] pgmap v1603: 9 pgs: 9 active+clean; 127 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.2 MiB/s wr, 3 op/s 2021-07-31T02:11:36.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:36 smithi045 conmon[39718]: cluster 2021-07-31T02:11:36.140133 2021-07-31T02:11:36.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:36 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 1047 : cluster [DBG] pgmap v1603: 9 pgs: 9 active+clean; 127 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.2 MiB/s wr, 3 op/s 2021-07-31T02:11:37.198 INFO:tasks.workunit.client.0.smithi045.stdout: 383: throughput=4.99MB/sec pending data=0 2021-07-31T02:11:37.199 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-qarVkQMzjtTTmG5 off=6442497 len=772823 2021-07-31T02:11:37.200 INFO:tasks.workunit.client.0.smithi045.stdout:op 1968 completed, throughput=4.99MB/sec 2021-07-31T02:11:37.200 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-7_IBBnlWIDAoPyQ off=3484925 len=222675 2021-07-31T02:11:37.201 INFO:tasks.workunit.client.0.smithi045.stdout:op 1969 completed, throughput=4.99MB/sec 2021-07-31T02:11:37.201 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-UmnLV9VcrG-wyqc off=1114769 len=1114560 2021-07-31T02:11:37.207 INFO:tasks.workunit.client.0.smithi045.stdout:op 1970 completed, throughput=4.99MB/sec 2021-07-31T02:11:37.208 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Q5kh1bi-N9lHDzO off=376494 len=1884184 2021-07-31T02:11:37.209 INFO:tasks.workunit.client.0.smithi045.stdout:op 1971 completed, throughput=5MB/sec 2021-07-31T02:11:37.210 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-z8FLJCEYSAIRgg8 off=3238508 len=1926127 2021-07-31T02:11:37.212 INFO:tasks.workunit.client.0.smithi045.stdout:op 1972 completed, throughput=5MB/sec 2021-07-31T02:11:38.212 INFO:tasks.workunit.client.0.smithi045.stdout: 384: throughput=4.99MB/sec pending data=0 2021-07-31T02:11:38.213 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Ug4IlHTRCkzsjOW off=636634 len=469506 2021-07-31T02:11:38.216 INFO:tasks.workunit.client.0.smithi045.stdout:op 1973 completed, throughput=4.99MB/sec 2021-07-31T02:11:38.217 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-eUVCIt1OXE3JJJT off=275165 len=412305 2021-07-31T02:11:38.218 INFO:tasks.workunit.client.0.smithi045.stdout:op 1974 completed, throughput=4.99MB/sec 2021-07-31T02:11:38.218 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-z7dGQ4Nartn0-hD off=7675532 len=680808 2021-07-31T02:11:38.223 INFO:tasks.workunit.client.0.smithi045.stdout:op 1975 completed, throughput=4.99MB/sec 2021-07-31T02:11:38.223 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-9_9-OFHH0ySadfj off=2657085 len=461971 2021-07-31T02:11:38.226 INFO:tasks.workunit.client.0.smithi045.stdout:op 1976 completed, throughput=5MB/sec 2021-07-31T02:11:38.227 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-54EWg5qR5LNa8lZ off=2245295 len=24272 2021-07-31T02:11:38.227 INFO:tasks.workunit.client.0.smithi045.stdout:op 1977 completed, throughput=5MB/sec 2021-07-31T02:11:38.227 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Y2CgLmBuM9OQmVD off=2790375 len=1070772 2021-07-31T02:11:38.232 INFO:tasks.workunit.client.0.smithi045.stdout:op 1978 completed, throughput=5MB/sec 2021-07-31T02:11:38.232 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-eZJObBkxKmVexsj off=8818503 len=933074 2021-07-31T02:11:38.239 INFO:tasks.workunit.client.0.smithi045.stdout:op 1979 completed, throughput=5MB/sec 2021-07-31T02:11:38.240 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-8SMip5mRoIRA76m off=5952685 len=1501539 2021-07-31T02:11:38.246 INFO:tasks.workunit.client.0.smithi045.stdout:op 1980 completed, throughput=5MB/sec 2021-07-31T02:11:38.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:38 smithi154 conmon[34553]: cluster 2021-07-31T02:11:38.140590+0000 mgr.x (mgr.24542) 1048 : cluster [DBG] pgmap v1604: 9 pgs: 9 active+clean; 127 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.2 MiB/s wr, 3 op/s 2021-07-31T02:11:38.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:38 smithi154 conmon[34553]: 2021-07-31T02:11:38.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:38 smithi045 conmon[33655]: cluster 2021-07-31T02:11: 2021-07-31T02:11:38.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:38 smithi045 conmon[33655]: 38.140590+0000 mgr.x (mgr.24542) 1048 : cluster [DBG] pgmap v1604: 9 pgs: 9 active+clean; 127 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.2 MiB/s wr, 3 op/s 2021-07-31T02:11:38.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:38 smithi045 conmon[39718]: cluster 2021-07-31T02:11:38.140590+0000 mgr.x 2021-07-31T02:11:38.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:38 smithi045 conmon[39718]: (mgr.24542) 1048 : cluster [DBG] pgmap v1604: 9 pgs: 9 active+clean; 127 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.2 MiB/s wr, 3 op/s 2021-07-31T02:11:39.247 INFO:tasks.workunit.client.0.smithi045.stdout: 385: throughput=4.99MB/sec pending data=0 2021-07-31T02:11:39.247 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-v7SNkPZoEcBt_qD off=9192883 len=411901 2021-07-31T02:11:39.248 INFO:tasks.workunit.client.0.smithi045.stdout:op 1981 completed, throughput=4.99MB/sec 2021-07-31T02:11:39.248 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-hQ_L2dzQ7-UtWws off=1580929 len=436279 2021-07-31T02:11:39.252 INFO:tasks.workunit.client.0.smithi045.stdout:op 1982 completed, throughput=4.99MB/sec 2021-07-31T02:11:39.252 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-DitGDkJg-AA2e_c off=20400972 len=1449693 2021-07-31T02:11:39.255 INFO:tasks.workunit.client.0.smithi045.stdout:op 1983 completed, throughput=5MB/sec 2021-07-31T02:11:39.255 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-APrKPajdxkq-k8q off=5362846 len=2020611 2021-07-31T02:11:39.266 INFO:tasks.workunit.client.0.smithi045.stdout:op 1984 completed, throughput=5MB/sec 2021-07-31T02:11:40.076 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:11:40.267 INFO:tasks.workunit.client.0.smithi045.stdout: 386: throughput=4.99MB/sec pending data=0 2021-07-31T02:11:40.267 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-PNjJgDy6My9KYh6 off=6306741 len=408035 2021-07-31T02:11:40.270 INFO:tasks.workunit.client.0.smithi045.stdout:op 1985 completed, throughput=4.99MB/sec 2021-07-31T02:11:40.271 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-x4o5D9PFfOmzkno off=4403958 len=1635066 2021-07-31T02:11:40.273 INFO:tasks.workunit.client.0.smithi045.stdout:op 1986 completed, throughput=4.99MB/sec 2021-07-31T02:11:40.274 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-uCGGJxYyw7Ryjn3 off=19887465 len=1383180 2021-07-31T02:11:40.274 INFO:tasks.workunit.client.0.smithi045.stdout:op 1987 completed, throughput=5MB/sec 2021-07-31T02:11:40.275 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-3ZlmsrfMUXgGNZD off=72220 len=453600 2021-07-31T02:11:40.278 INFO:tasks.workunit.client.0.smithi045.stdout:op 1988 completed, throughput=5MB/sec 2021-07-31T02:11:40.278 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-jEsQU9Hy7r5ukcI off=3504623 len=1433129 2021-07-31T02:11:40.280 INFO:tasks.workunit.client.0.smithi045.stdout:op 1989 completed, throughput=5MB/sec 2021-07-31T02:11:40.364 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:11:40.365 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 26m ago 27m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:11:40.365 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 24m ago 27m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:11:40.365 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 24m ago 30m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:11:40.366 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 26m ago 32m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:11:40.366 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 26m ago 32m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:11:40.366 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 24m ago 30m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:11:40.366 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 26m ago 31m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:11:40.366 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 26m ago 27m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:11:40.367 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 24m ago 27m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:11:40.367 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 26m ago 30m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:11:40.367 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 26m ago 29m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:11:40.367 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 26m ago 29m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:11:40.368 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 26m ago 29m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:11:40.368 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 24m ago 28m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:11:40.368 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 24m ago 28m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:11:40.368 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 24m ago 28m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:11:40.368 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 24m ago 27m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:11:40.369 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 24m ago 27m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:11:40.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:40 smithi154 conmon[34553]: audit 2021-07-31T02:11:40.064621+0000 mgr.x (mgr.24542) 1049 : audit [DBG] from='client.15951 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:11:40.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:40 smithi154 conmon[34553]: cluster 2021-07-31T02:11:40.141258+0000 mgr.x (mgr.24542) 1050 : cluster [DBG] pgmap v1605: 9 pgs: 9 active+clean; 127 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 3.2 MiB/s rd, 3.2 MiB/s wr, 5 op/s 2021-07-31T02:11:40.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:40 smithi045 conmon[33655]: audit 2021-07-31T02:11:40.064621+0000 mgr.x (mgr.24542) 1049 : audit [DBG] from='client.15951 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:11:40.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:40 smithi045 conmon[33655]: cluster 2021-07-31T02:11:40.141258+0000 mgr.x (mgr.24542) 1050 : cluster [DBG] pgmap v1605: 9 pgs: 9 active+clean; 127 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 3.2 MiB/s rd, 3.2 MiB/s wr, 5 op/s 2021-07-31T02:11:40.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:40 smithi045 conmon[39718]: audit 2021-07-31T02:11:40.064621 2021-07-31T02:11:40.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:40 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 1049 : audit [DBG] from='client.15951 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:11:40.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:40 smithi045 conmon[39718]: cluster 2021-07-31T02:11:40.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:40 smithi045 conmon[39718]: 2021-07-31T02:11:40.141258+0000 mgr.x (mgr.24542) 1050 : 2021-07-31T02:11:40.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:40 smithi045 conmon[39718]: cluster [DBG] pgmap v1605: 9 pgs: 9 active+clean; 127 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 3.2 MiB/s rd, 3.2 MiB/s wr, 5 op/s 2021-07-31T02:11:40.699 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:11:40.700 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:11:40.700 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:11:40.700 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:11:40.701 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:11:40.701 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:11:40.701 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:11:40.701 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:11:40.702 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:11:40.702 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:11:40.702 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:11:40.702 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:11:40.703 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:11:40.703 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:11:40.703 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:11:40.703 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:11:40.703 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:11:41.280 INFO:tasks.workunit.client.0.smithi045.stdout: 387: throughput=4.99MB/sec pending data=0 2021-07-31T02:11:41.281 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-npSKXBGSA0hji6m off=12284555 len=230177 2021-07-31T02:11:41.282 INFO:tasks.workunit.client.0.smithi045.stdout:op 1990 completed, throughput=4.99MB/sec 2021-07-31T02:11:41.282 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-9Kx0YupPSAvs5lr off=17212487 len=1246678 2021-07-31T02:11:41.289 INFO:tasks.workunit.client.0.smithi045.stdout:op 1991 completed, throughput=4.99MB/sec 2021-07-31T02:11:41.289 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-cVT4jgVPoJpK48_ off=5713751 len=590523 2021-07-31T02:11:41.290 INFO:tasks.workunit.client.0.smithi045.stdout:op 1992 completed, throughput=4.99MB/sec 2021-07-31T02:11:41.290 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-7n8d4xPIkC3b7br off=8225509 len=405349 2021-07-31T02:11:41.291 INFO:tasks.workunit.client.0.smithi045.stdout:op 1993 completed, throughput=4.99MB/sec 2021-07-31T02:11:41.291 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-b0HVAvtr-o2IUoh off=14147962 len=76864 2021-07-31T02:11:41.294 INFO:tasks.workunit.client.0.smithi045.stdout:op 1994 completed, throughput=4.99MB/sec 2021-07-31T02:11:41.294 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ppjiShPNq7fdfFH off=1390126 len=453374 2021-07-31T02:11:41.297 INFO:tasks.workunit.client.0.smithi045.stdout:op 1995 completed, throughput=5MB/sec 2021-07-31T02:11:41.298 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-edNmQdJjyCtr6GH off=3555877 len=670006 2021-07-31T02:11:41.299 INFO:tasks.workunit.client.0.smithi045.stdout:op 1996 completed, throughput=5MB/sec 2021-07-31T02:11:41.299 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-M1nHfVn11b8SNWg off=565615 len=1585591 2021-07-31T02:11:41.306 INFO:tasks.workunit.client.0.smithi045.stdout:op 1997 completed, throughput=5MB/sec 2021-07-31T02:11:41.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:41 smithi154 conmon[34553]: audit 2021-07-31T02:11:40.355484+0000 mgr.x (mgr.24542) 1051 : audit [DBG] from='client.15957 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:11:41.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:41 smithi154 conmon[34553]: 2021-07-31T02:11:41.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:41 smithi154 conmon[34553]: audit 2021-07-31T02:11:40.699959+0000 mon.a (mon.0) 1870 : audit [DBG] from='client.? 172.21.15.45:0/3550356738' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:11:41.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:41 smithi045 conmon[33655]: audit 2021-07-31T02:11:40.355484+0000 2021-07-31T02:11:41.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:41 smithi045 conmon[33655]: mgr.x (mgr.24542) 1051 : audit [DBG] from='client.15957 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:11:41.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:41 smithi045 conmon[33655]: audit 2021-07-31T02:11:40.699959+0000 mon.a (mon.0) 1870 2021-07-31T02:11:41.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:41 smithi045 conmon[33655]: : audit [DBG] from='client.? 172.21.15.45:0/3550356738' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:11:41.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:41 smithi045 conmon[39718]: audit 2021-07-31T02:11: 2021-07-31T02:11:41.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:41 smithi045 conmon[39718]: 40.355484+0000 mgr.x (mgr.24542) 1051 : audit [DBG] from='client.15957 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:11:41.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:41 smithi045 conmon[39718]: audit 2021-07-31T02:11:41.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:41 smithi045 conmon[39718]: 2021-07-31T02 2021-07-31T02:11:41.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:41 smithi045 conmon[39718]: :11:40.699959+0000 mon.a 2021-07-31T02:11:41.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:41 smithi045 conmon[39718]: (mon.0) 1870 : audit 2021-07-31T02:11:41.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:41 smithi045 conmon[39718]: [DBG] from='client.? 172.21.15.45:0/3550356738' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:11:42.306 INFO:tasks.workunit.client.0.smithi045.stdout: 388: throughput=4.99MB/sec pending data=0 2021-07-31T02:11:42.307 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-KR3blaFiZYENR0q off=3839670 len=134070 2021-07-31T02:11:42.308 INFO:tasks.workunit.client.0.smithi045.stdout:op 1998 completed, throughput=4.99MB/sec 2021-07-31T02:11:42.308 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ub5Dtl_n_T1Op4R off=2358658 len=1110148 2021-07-31T02:11:42.313 INFO:tasks.workunit.client.0.smithi045.stdout:op 1999 completed, throughput=4.99MB/sec 2021-07-31T02:11:42.313 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-cfko8p_VnHsO_Gm off=11560971 len=745023 2021-07-31T02:11:42.315 INFO:tasks.workunit.client.0.smithi045.stdout:op 2000 completed, throughput=4.99MB/sec 2021-07-31T02:11:42.315 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-2Es7lbl0-PMdoOz off=1388473 len=1633347 2021-07-31T02:11:42.322 INFO:tasks.workunit.client.0.smithi045.stdout:op 2001 completed, throughput=5MB/sec 2021-07-31T02:11:42.322 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-7w80Fa_L2VsfvSc off=2012659 len=704912 2021-07-31T02:11:42.323 INFO:tasks.workunit.client.0.smithi045.stdout:op 2002 completed, throughput=5MB/sec 2021-07-31T02:11:42.324 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ovSQVZIxQENSCPe off=11887675 len=1681007 2021-07-31T02:11:42.330 INFO:tasks.workunit.client.0.smithi045.stdout:op 2003 completed, throughput=5MB/sec 2021-07-31T02:11:43.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:42 smithi154 conmon[34553]: cluster 2021-07-31T02:11:42.141625+0000 mgr.x (mgr.24542) 1052 : cluster [DBG] pgmap v1606: 9 pgs: 9 active+clean; 127 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 2.6 MiB/s rd, 1.7 MiB/s wr, 4 op/s 2021-07-31T02:11:43.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:42 smithi045 conmon[33655]: cluster 2021-07-31T02:11:42.141625+0000 mgr.x 2021-07-31T02:11:43.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:42 smithi045 conmon[33655]: (mgr.24542) 1052 : cluster [DBG] pgmap v1606: 9 pgs: 9 active+clean; 127 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 2.6 MiB/s rd, 1.7 MiB/s wr, 4 op/s 2021-07-31T02:11:43.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:42 smithi045 conmon[39718]: cluster 2021-07-31T02:11:42.141625+0000 mgr.x (mgr 2021-07-31T02:11:43.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:42 smithi045 conmon[39718]: .24542) 1052 : cluster [DBG] pgmap v1606: 9 pgs: 9 active+clean; 127 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 2.6 MiB/s rd, 1.7 MiB/s wr, 4 op/s 2021-07-31T02:11:43.331 INFO:tasks.workunit.client.0.smithi045.stdout: 389: throughput=4.99MB/sec pending data=0 2021-07-31T02:11:43.331 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-XzQ44ht2vTwZhU1 off=12981446 len=580102 2021-07-31T02:11:43.332 INFO:tasks.workunit.client.0.smithi045.stdout:op 2004 completed, throughput=4.99MB/sec 2021-07-31T02:11:43.332 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-j_JYs6ClVLf--J1 off=1668683 len=452631 2021-07-31T02:11:43.333 INFO:tasks.workunit.client.0.smithi045.stdout:op 2005 completed, throughput=4.99MB/sec 2021-07-31T02:11:43.333 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-aJg_wwcdFcdaYWI off=6270667 len=12301 2021-07-31T02:11:43.336 INFO:tasks.workunit.client.0.smithi045.stdout:op 2006 completed, throughput=4.99MB/sec 2021-07-31T02:11:43.336 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-88X14lSnuUg4FJ2 off=19067074 len=178576 2021-07-31T02:11:43.339 INFO:tasks.workunit.client.0.smithi045.stdout:op 2007 completed, throughput=4.99MB/sec 2021-07-31T02:11:43.339 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-dGgGMZJh_JuFkL7 off=16231234 len=1029656 2021-07-31T02:11:43.345 INFO:tasks.workunit.client.0.smithi045.stdout:op 2008 completed, throughput=4.99MB/sec 2021-07-31T02:11:43.345 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Kecz1wshhRLmI54 off=1534481 len=2042143 2021-07-31T02:11:43.353 INFO:tasks.workunit.client.0.smithi045.stdout:op 2009 completed, throughput=5MB/sec 2021-07-31T02:11:43.353 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-bWqJSWls2wmagdq off=155905 len=936881 2021-07-31T02:11:43.354 INFO:tasks.workunit.client.0.smithi045.stdout:op 2010 completed, throughput=5MB/sec 2021-07-31T02:11:44.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:11:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:11:43.550Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:11:44.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:11:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:11:43.550Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:11:44.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:11:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:11:43.550Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:11:44.354 INFO:tasks.workunit.client.0.smithi045.stdout: 390: throughput=4.99MB/sec pending data=0 2021-07-31T02:11:44.355 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-BpgQACqDtxt7Azf off=6765051 len=86967 2021-07-31T02:11:44.357 INFO:tasks.workunit.client.0.smithi045.stdout:op 2011 completed, throughput=4.99MB/sec 2021-07-31T02:11:44.357 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-fABjIaLoxNXmqtv off=8866492 len=1090554 2021-07-31T02:11:44.363 INFO:tasks.workunit.client.0.smithi045.stdout:op 2012 completed, throughput=4.99MB/sec 2021-07-31T02:11:44.363 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-rHj42S-ZMkbKHzS off=440852 len=1514110 2021-07-31T02:11:44.369 INFO:tasks.workunit.client.0.smithi045.stdout:op 2013 completed, throughput=5MB/sec 2021-07-31T02:11:44.369 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-BFq01F0wiWtEgU- off=1719904 len=75924 2021-07-31T02:11:44.369 INFO:tasks.workunit.client.0.smithi045.stdout:op 2014 completed, throughput=5MB/sec 2021-07-31T02:11:44.370 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-bDcBk3PwkFLLuRN off=15159447 len=1527670 2021-07-31T02:11:44.376 INFO:tasks.workunit.client.0.smithi045.stdout:op 2015 completed, throughput=5MB/sec 2021-07-31T02:11:44.376 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-9Drg4fW1PMJmA39 off=18858140 len=251303 2021-07-31T02:11:44.377 INFO:tasks.workunit.client.0.smithi045.stdout:op 2016 completed, throughput=5MB/sec 2021-07-31T02:11:44.377 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-766x7a38hRnonXF off=3782112 len=1002849 2021-07-31T02:11:44.382 INFO:tasks.workunit.client.0.smithi045.stdout:op 2017 completed, throughput=5MB/sec 2021-07-31T02:11:44.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:44 smithi154 conmon[34553]: cluster 2021-07-31T02:11:44.142171+0000 mgr.x (mgr.24542) 1053 : cluster [DBG] pgmap v1607: 9 pgs: 9 active+clean; 127 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 3.6 MiB/s rd, 2.8 MiB/s wr, 6 op/s 2021-07-31T02:11:44.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:44 smithi045 conmon[33655]: cluster 2021-07-31T02 2021-07-31T02:11:44.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:44 smithi045 conmon[33655]: :11:44.142171+0000 mgr.x (mgr.24542) 1053 : cluster [DBG] pgmap v1607: 9 pgs: 9 active+clean; 127 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 3.6 MiB/s rd, 2.8 MiB/s wr, 6 op/s 2021-07-31T02:11:44.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:44 smithi045 conmon[39718]: cluster 2021-07-31T02:11:44.142171+0000 mgr.x (mgr. 2021-07-31T02:11:44.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:44 smithi045 conmon[39718]: 24542) 1053 : cluster [DBG] pgmap v1607: 9 pgs: 9 active+clean; 127 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 3.6 MiB/s rd, 2.8 MiB/s wr, 6 op/s 2021-07-31T02:11:45.382 INFO:tasks.workunit.client.0.smithi045.stdout: 391: throughput=4.99MB/sec pending data=0 2021-07-31T02:11:45.382 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-8vm4a3zjB2X9sBr off=13941983 len=1626350 2021-07-31T02:11:45.384 INFO:tasks.workunit.client.0.smithi045.stdout:op 2018 completed, throughput=4.99MB/sec 2021-07-31T02:11:45.384 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-spUIOpiAxwQJE7s off=790287 len=971312 2021-07-31T02:11:45.387 INFO:tasks.workunit.client.0.smithi045.stdout:op 2019 completed, throughput=5MB/sec 2021-07-31T02:11:45.387 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-d1TagMSb_hBJ0pU off=3708133 len=1685103 2021-07-31T02:11:45.395 INFO:tasks.workunit.client.0.smithi045.stdout:op 2020 completed, throughput=5MB/sec 2021-07-31T02:11:45.395 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-gFjRVRJmKUC26qj off=9379824 len=387278 2021-07-31T02:11:45.398 INFO:tasks.workunit.client.0.smithi045.stdout:op 2021 completed, throughput=5MB/sec 2021-07-31T02:11:46.398 INFO:tasks.workunit.client.0.smithi045.stdout: 392: throughput=4.99MB/sec pending data=0 2021-07-31T02:11:46.399 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-AZ1y1UvvrAlJ6Er off=11442369 len=1006480 2021-07-31T02:11:46.400 INFO:tasks.workunit.client.0.smithi045.stdout:op 2022 completed, throughput=4.99MB/sec 2021-07-31T02:11:46.400 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-i2oiVef6H4enCIK off=12955416 len=954326 2021-07-31T02:11:46.407 INFO:tasks.workunit.client.0.smithi045.stdout:op 2023 completed, throughput=4.99MB/sec 2021-07-31T02:11:46.407 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Jthwa4edvwUZipj off=261857 len=692797 2021-07-31T02:11:46.408 INFO:tasks.workunit.client.0.smithi045.stdout:op 2024 completed, throughput=4.99MB/sec 2021-07-31T02:11:46.409 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-1KuF6K-ThWiv5Ap off=5092778 len=541603 2021-07-31T02:11:46.409 INFO:tasks.workunit.client.0.smithi045.stdout:op 2025 completed, throughput=5MB/sec 2021-07-31T02:11:46.410 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-9jCbLF5GrfRslj3 off=532783 len=1854737 2021-07-31T02:11:46.412 INFO:tasks.workunit.client.0.smithi045.stdout:op 2026 completed, throughput=5MB/sec 2021-07-31T02:11:46.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:46 smithi154 conmon[34553]: cluster 2021-07-31T02:11:46.142646+0000 mgr.x (mgr.24542) 1054 : cluster [DBG] 2021-07-31T02:11:46.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:46 smithi154 conmon[34553]: pgmap v1608: 9 pgs: 9 active+clean; 127 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2021-07-31T02:11:46.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:46 smithi045 conmon[33655]: cluster 2021-07-31T02:11:46.142646+0000 mgr.x (mgr.24542 2021-07-31T02:11:46.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:46 smithi045 conmon[33655]: ) 1054 : cluster [DBG] pgmap v1608: 9 pgs: 9 active+clean; 127 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2021-07-31T02:11:46.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:46 smithi045 conmon[39718]: cluster 2021-07-31T02:11:46.142646+0000 mgr.x (mgr 2021-07-31T02:11:46.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:46 smithi045 conmon[39718]: .24542) 1054 : cluster [DBG] pgmap v1608: 9 pgs: 9 active+clean; 127 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2021-07-31T02:11:47.412 INFO:tasks.workunit.client.0.smithi045.stdout: 393: throughput=4.99MB/sec pending data=0 2021-07-31T02:11:47.413 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ryGbD3QglsGAcqX off=5111774 len=844561 2021-07-31T02:11:47.417 INFO:tasks.workunit.client.0.smithi045.stdout:op 2027 completed, throughput=4.99MB/sec 2021-07-31T02:11:47.418 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Ie4e3AswqRVFOm2 off=5904370 len=1247378 2021-07-31T02:11:47.419 INFO:tasks.workunit.client.0.smithi045.stdout:op 2028 completed, throughput=4.99MB/sec 2021-07-31T02:11:47.419 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-xxLOvDXRGNZnUed off=12261069 len=969551 2021-07-31T02:11:47.424 INFO:tasks.workunit.client.0.smithi045.stdout:op 2029 completed, throughput=4.99MB/sec 2021-07-31T02:11:47.425 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-hj5c_H-3QlYT8AQ off=17976 len=1054903 2021-07-31T02:11:47.430 INFO:tasks.workunit.client.0.smithi045.stdout:op 2030 completed, throughput=5MB/sec 2021-07-31T02:11:47.430 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-_r_C211FRn071y5 off=7272080 len=1207705 2021-07-31T02:11:47.435 INFO:tasks.workunit.client.0.smithi045.stdout:op 2031 completed, throughput=5MB/sec 2021-07-31T02:11:47.436 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-8t6Wn6qWZPKRHN3 off=2804329 len=38435 2021-07-31T02:11:47.436 INFO:tasks.workunit.client.0.smithi045.stdout:op 2032 completed, throughput=5MB/sec 2021-07-31T02:11:48.436 INFO:tasks.workunit.client.0.smithi045.stdout: 394: throughput=4.99MB/sec pending data=0 2021-07-31T02:11:48.437 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-dm8ZT5H-reFLcaO off=12968280 len=1394150 2021-07-31T02:11:48.440 INFO:tasks.workunit.client.0.smithi045.stdout:op 2033 completed, throughput=4.99MB/sec 2021-07-31T02:11:48.440 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-pVVB3C2vX24HBJY off=1780927 len=803110 2021-07-31T02:11:48.446 INFO:tasks.workunit.client.0.smithi045.stdout:op 2034 completed, throughput=4.99MB/sec 2021-07-31T02:11:48.446 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ABYmoS1ViNGX0Bg off=5908621 len=1143546 2021-07-31T02:11:48.448 INFO:tasks.workunit.client.0.smithi045.stdout:op 2035 completed, throughput=5MB/sec 2021-07-31T02:11:48.448 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-_B1t4AwZYZCNsQO off=6273567 len=1044847 2021-07-31T02:11:48.450 INFO:tasks.workunit.client.0.smithi045.stdout:op 2036 completed, throughput=5MB/sec 2021-07-31T02:11:48.450 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-SiwwPL7Lf3fqVst off=10897809 len=131888 2021-07-31T02:11:48.451 INFO:tasks.workunit.client.0.smithi045.stdout:op 2037 completed, throughput=5MB/sec 2021-07-31T02:11:48.451 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-wC0lT7plUxVTDWN off=13501222 len=180814 2021-07-31T02:11:48.455 INFO:tasks.workunit.client.0.smithi045.stdout:op 2038 completed, throughput=5MB/sec 2021-07-31T02:11:48.455 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-eefeIOexFigW3gJ off=12420792 len=182888 2021-07-31T02:11:48.456 INFO:tasks.workunit.client.0.smithi045.stdout:op 2039 completed, throughput=5MB/sec 2021-07-31T02:11:48.456 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-YVjIRpbgE3IUAGx off=10143043 len=657286 2021-07-31T02:11:48.457 INFO:tasks.workunit.client.0.smithi045.stdout:op 2040 completed, throughput=5MB/sec 2021-07-31T02:11:48.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:48 smithi154 conmon[34553]: cluster 2021-07-31T02:11:48.143040+0000 mgr.x (mgr.24542) 1055 : cluster [DBG] pgmap v1609: 9 pgs: 9 active+clean; 127 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2021-07-31T02:11:48.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:48 smithi045 conmon[33655]: cluster 2021-07-31T02:11:48.143040+0000 mgr.x (mgr.24542) 1055 : cluster [DBG] pgmap v1609: 9 pgs: 9 active+clean; 127 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2021-07-31T02:11:48.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:48 smithi045 conmon[39718]: cluster 2021-07-31T02:11:48.143040+0000 mgr.x ( 2021-07-31T02:11:48.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:48 smithi045 conmon[39718]: mgr.24542) 1055 : cluster [DBG] pgmap v1609: 9 pgs: 9 active+clean; 127 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2021-07-31T02:11:49.457 INFO:tasks.workunit.client.0.smithi045.stdout: 395: throughput=4.99MB/sec pending data=0 2021-07-31T02:11:49.458 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-RkXRSjBPXgDVXvj off=1082085 len=265574 2021-07-31T02:11:49.458 INFO:tasks.workunit.client.0.smithi045.stdout:op 2041 completed, throughput=4.99MB/sec 2021-07-31T02:11:49.459 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Jn5I_S0H6NNOa-P off=16839174 len=582091 2021-07-31T02:11:49.463 INFO:tasks.workunit.client.0.smithi045.stdout:op 2042 completed, throughput=4.99MB/sec 2021-07-31T02:11:49.463 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-biHf1MFw3d8bYY2 off=22433529 len=1418124 2021-07-31T02:11:49.465 INFO:tasks.workunit.client.0.smithi045.stdout:op 2043 completed, throughput=4.99MB/sec 2021-07-31T02:11:49.465 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-RM4qM4thUoHXEFh off=5206836 len=550278 2021-07-31T02:11:49.468 INFO:tasks.workunit.client.0.smithi045.stdout:op 2044 completed, throughput=4.99MB/sec 2021-07-31T02:11:49.469 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-CirH5NuKu-yh0dq off=3758916 len=1247082 2021-07-31T02:11:49.475 INFO:tasks.workunit.client.0.smithi045.stdout:op 2045 completed, throughput=5MB/sec 2021-07-31T02:11:49.475 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-zUJ08OPkKnZ5snV off=893014 len=1315619 2021-07-31T02:11:49.477 INFO:tasks.workunit.client.0.smithi045.stdout:op 2046 completed, throughput=5MB/sec 2021-07-31T02:11:50.477 INFO:tasks.workunit.client.0.smithi045.stdout: 396: throughput=4.99MB/sec pending data=0 2021-07-31T02:11:50.478 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ruvYgNVU-2ohQBy off=809710 len=1576114 2021-07-31T02:11:50.485 INFO:tasks.workunit.client.0.smithi045.stdout:op 2047 completed, throughput=4.99MB/sec 2021-07-31T02:11:50.485 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-kYnetiMRsDxxyey off=14093781 len=791085 2021-07-31T02:11:50.487 INFO:tasks.workunit.client.0.smithi045.stdout:op 2048 completed, throughput=4.99MB/sec 2021-07-31T02:11:50.487 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-5_Qcy-Ai3pzy_LT off=1505060 len=686580 2021-07-31T02:11:50.489 INFO:tasks.workunit.client.0.smithi045.stdout:op 2049 completed, throughput=5MB/sec 2021-07-31T02:11:50.489 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-9B5zt0U1_PSW0jy off=13609425 len=535038 2021-07-31T02:11:50.493 INFO:tasks.workunit.client.0.smithi045.stdout:op 2050 completed, throughput=5MB/sec 2021-07-31T02:11:50.493 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-T8SlBY9Ztxo9MvA off=7299626 len=1225797 2021-07-31T02:11:50.495 INFO:tasks.workunit.client.0.smithi045.stdout:op 2051 completed, throughput=5MB/sec 2021-07-31T02:11:50.495 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-VvBmRCSwev7ezWu off=17866563 len=2018815 2021-07-31T02:11:50.497 INFO:tasks.workunit.client.0.smithi045.stdout:op 2052 completed, throughput=5MB/sec 2021-07-31T02:11:50.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:50 smithi154 conmon[34553]: cluster 2021-07-31T02:11:50.143500+0000 mgr.x (mgr.24542) 1056 : cluster [DBG] pgmap v1610: 9 pgs: 9 active+clean; 127 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 3.1 MiB/s rd, 3.3 MiB/s wr, 7 op/s 2021-07-31T02:11:50.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:50 smithi045 conmon[33655]: cluster 2021-07-31T02:11:50.143500 2021-07-31T02:11:50.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:50 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 1056 : cluster [DBG] pgmap v1610: 9 pgs: 9 active+clean; 127 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 3.1 MiB/s rd, 3.3 MiB/s wr, 7 op/s 2021-07-31T02:11:50.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:50 smithi045 conmon[39718]: cluster 2021-07-31T02:11:50.143500+0000 mgr.x (mgr. 2021-07-31T02:11:50.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:50 smithi045 conmon[39718]: 24542) 1056 : cluster [DBG] pgmap v1610: 9 pgs: 9 active+clean; 127 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 3.1 MiB/s rd, 3.3 MiB/s wr, 7 op/s 2021-07-31T02:11:51.498 INFO:tasks.workunit.client.0.smithi045.stdout: 397: throughput=4.99MB/sec pending data=0 2021-07-31T02:11:51.498 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-RKyXoBs0nQNjfpK off=5503182 len=1922284 2021-07-31T02:11:51.501 INFO:tasks.workunit.client.0.smithi045.stdout:op 2053 completed, throughput=5MB/sec 2021-07-31T02:11:51.501 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-8D9__EphrtguKYn off=10976750 len=2036126 2021-07-31T02:11:51.510 INFO:tasks.workunit.client.0.smithi045.stdout:op 2054 completed, throughput=5MB/sec 2021-07-31T02:11:52.511 INFO:tasks.workunit.client.0.smithi045.stdout: 398: throughput=4.99MB/sec pending data=0 2021-07-31T02:11:52.512 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-gHuUhaZGQ8WjZOl off=218460 len=1466427 2021-07-31T02:11:52.513 INFO:tasks.workunit.client.0.smithi045.stdout:op 2055 completed, throughput=4.99MB/sec 2021-07-31T02:11:52.513 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-bWQ8ivNGWnBTGmP off=4395608 len=2091464 2021-07-31T02:11:52.515 INFO:tasks.workunit.client.0.smithi045.stdout:op 2056 completed, throughput=5MB/sec 2021-07-31T02:11:52.515 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-rle-Iq54Yd7tQR- off=920453 len=598468 2021-07-31T02:11:52.520 INFO:tasks.workunit.client.0.smithi045.stdout:op 2057 completed, throughput=5MB/sec 2021-07-31T02:11:52.520 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-FOHxy2oItACmelK off=1112948 len=1668077 2021-07-31T02:11:52.523 INFO:tasks.workunit.client.0.smithi045.stdout:op 2058 completed, throughput=5MB/sec 2021-07-31T02:11:53.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:52 smithi154 conmon[34553]: cluster 2021-07-31T02:11:52.144022+0000 mgr.x (mgr.24542) 1057 : cluster [DBG] pgmap v1611: 9 pgs: 9 active+clean; 127 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2021-07-31T02:11:53.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:52 smithi045 conmon[33655]: cluster 2021-07-31T02:11:52.144022+0000 mgr.x (mgr.24542) 1057 2021-07-31T02:11:53.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:52 smithi045 conmon[33655]: : cluster [DBG] pgmap v1611: 9 pgs: 9 active+clean; 127 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2021-07-31T02:11:53.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:52 smithi045 conmon[39718]: cluster 2021-07-31T02:11:52.144022+0000 mgr.x (mgr 2021-07-31T02:11:53.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:52 smithi045 conmon[39718]: .24542) 1057 : cluster [DBG] pgmap v1611: 9 pgs: 9 active+clean; 127 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2021-07-31T02:11:53.523 INFO:tasks.workunit.client.0.smithi045.stdout: 399: throughput=4.99MB/sec pending data=0 2021-07-31T02:11:53.524 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-11DHI2Mn3JK8VKH off=3435239 len=127694 2021-07-31T02:11:53.524 INFO:tasks.workunit.client.0.smithi045.stdout:op 2059 completed, throughput=4.99MB/sec 2021-07-31T02:11:53.525 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-U1OIU0CgwoUMGM3 off=12438284 len=1381672 2021-07-31T02:11:53.531 INFO:tasks.workunit.client.0.smithi045.stdout:op 2060 completed, throughput=4.99MB/sec 2021-07-31T02:11:53.532 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-IUtt3bMUUZtkwzp off=6581265 len=1593860 2021-07-31T02:11:53.539 INFO:tasks.workunit.client.0.smithi045.stdout:op 2061 completed, throughput=5MB/sec 2021-07-31T02:11:53.539 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-hd5AGJ1Cr1pHXAq off=7867304 len=1220826 2021-07-31T02:11:53.546 INFO:tasks.workunit.client.0.smithi045.stdout:op 2062 completed, throughput=5MB/sec 2021-07-31T02:11:53.546 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-lYw5RTEQqNhdHX3 off=3290445 len=534923 2021-07-31T02:11:53.549 INFO:tasks.workunit.client.0.smithi045.stdout:op 2063 completed, throughput=5MB/sec 2021-07-31T02:11:54.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:11:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:11:53.550Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:11:54.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:11:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:11:53.550Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:11:54.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:11:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:11:53.550Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:11:54.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:54 smithi154 conmon[34553]: cluster 2021-07-31T02:11:54.144535+0000 mgr.x (mgr.24542) 1058 : cluster [DBG] pgmap v1612: 9 pgs: 9 active+clean; 127 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 3.1 MiB/s rd, 3.1 MiB/s wr, 6 op/s 2021-07-31T02:11:54.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:54 smithi045 conmon[33655]: cluster 2021-07-31T02:11:54.144535+0000 mgr.x ( 2021-07-31T02:11:54.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:54 smithi045 conmon[33655]: mgr.24542) 1058 : cluster [DBG] pgmap v1612: 9 pgs: 9 active+clean; 127 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 3.1 MiB/s rd, 3.1 MiB/s wr, 6 op/s 2021-07-31T02:11:54.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:54 smithi045 conmon[39718]: cluster 2021-07-31T02:11:54.144535+0000 mgr.x ( 2021-07-31T02:11:54.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:54 smithi045 conmon[39718]: mgr.24542) 1058 : cluster [DBG] pgmap v1612: 9 pgs: 9 active+clean; 127 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 3.1 MiB/s rd, 3.1 MiB/s wr, 6 op/s 2021-07-31T02:11:54.550 INFO:tasks.workunit.client.0.smithi045.stdout: 400: throughput=4.99MB/sec pending data=0 2021-07-31T02:11:54.550 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Jnp8Cuo49SYtg_I off=17108379 len=72012 2021-07-31T02:11:54.551 INFO:tasks.workunit.client.0.smithi045.stdout:op 2064 completed, throughput=4.99MB/sec 2021-07-31T02:11:54.551 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-6lUMbtTrancL6DD off=3165584 len=441653 2021-07-31T02:11:54.555 INFO:tasks.workunit.client.0.smithi045.stdout:op 2065 completed, throughput=4.99MB/sec 2021-07-31T02:11:54.555 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-WZC-Sl4ic5fQd87 off=2027181 len=1482785 2021-07-31T02:11:54.556 INFO:tasks.workunit.client.0.smithi045.stdout:op 2066 completed, throughput=4.99MB/sec 2021-07-31T02:11:54.557 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ooWg0D2X2WAO_cS off=10581646 len=480848 2021-07-31T02:11:54.560 INFO:tasks.workunit.client.0.smithi045.stdout:op 2067 completed, throughput=4.99MB/sec 2021-07-31T02:11:54.560 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-j5U2CD--6pHmzio off=420274 len=100350 2021-07-31T02:11:54.562 INFO:tasks.workunit.client.0.smithi045.stdout:op 2068 completed, throughput=4.99MB/sec 2021-07-31T02:11:54.563 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-8q9lO6IkzUVm-J0 off=3068513 len=1613054 2021-07-31T02:11:54.563 INFO:tasks.workunit.client.0.smithi045.stdout:op 2069 completed, throughput=5MB/sec 2021-07-31T02:11:54.564 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-E0U6Fkqckw7L8eL off=10153546 len=486843 2021-07-31T02:11:54.567 INFO:tasks.workunit.client.0.smithi045.stdout:op 2070 completed, throughput=5MB/sec 2021-07-31T02:11:54.567 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-WSvZXJ7bzA-KLyx off=5965915 len=792568 2021-07-31T02:11:54.568 INFO:tasks.workunit.client.0.smithi045.stdout:op 2071 completed, throughput=5MB/sec 2021-07-31T02:11:55.569 INFO:tasks.workunit.client.0.smithi045.stdout: 401: throughput=4.99MB/sec pending data=0 2021-07-31T02:11:55.569 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-NtnYozM__B5_5y3 off=1441957 len=229154 2021-07-31T02:11:55.572 INFO:tasks.workunit.client.0.smithi045.stdout:op 2072 completed, throughput=4.99MB/sec 2021-07-31T02:11:55.572 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-rDk8OQN2Skzcg5L off=3489949 len=491431 2021-07-31T02:11:55.575 INFO:tasks.workunit.client.0.smithi045.stdout:op 2073 completed, throughput=4.99MB/sec 2021-07-31T02:11:55.576 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-HppxL3WTWenBThE off=5164980 len=1589767 2021-07-31T02:11:55.583 INFO:tasks.workunit.client.0.smithi045.stdout:op 2074 completed, throughput=4.99MB/sec 2021-07-31T02:11:55.583 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-5vso8YdgtyaHxho off=13204132 len=1262768 2021-07-31T02:11:55.585 INFO:tasks.workunit.client.0.smithi045.stdout:op 2075 completed, throughput=5MB/sec 2021-07-31T02:11:55.586 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-KzmvKLylvMzvyZE off=6554748 len=2088498 2021-07-31T02:11:55.596 INFO:tasks.workunit.client.0.smithi045.stdout:op 2076 completed, throughput=5MB/sec 2021-07-31T02:11:56.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:56 smithi154 conmon[34553]: cluster 2021-07-31T02:11:56.144966+0000 mgr.x (mgr.24542) 1059 : cluster [DBG] pgmap v1613: 9 pgs: 9 active+clean; 127 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2021-07-31T02:11:56.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:56 smithi154 conmon[34553]: 2021-07-31T02:11:56.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:56 smithi045 conmon[39718]: cluster 2021-07-31T02:11:56.144966+0000 mgr.x 2021-07-31T02:11:56.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:56 smithi045 conmon[39718]: (mgr.24542) 1059 : cluster [DBG] pgmap v1613: 9 pgs: 9 active+clean; 127 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2021-07-31T02:11:56.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:56 smithi045 conmon[33655]: cluster 2021-07-31T02:11:56.144966+0000 mgr.x (mgr.24542) 1059 2021-07-31T02:11:56.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:56 smithi045 conmon[33655]: : cluster [DBG] pgmap v1613: 9 pgs: 9 active+clean; 127 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2021-07-31T02:11:56.596 INFO:tasks.workunit.client.0.smithi045.stdout: 402: throughput=4.99MB/sec pending data=0 2021-07-31T02:11:56.597 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-v8fpJbbwy1jkeAt off=8815127 len=1087709 2021-07-31T02:11:56.599 INFO:tasks.workunit.client.0.smithi045.stdout:op 2077 completed, throughput=4.99MB/sec 2021-07-31T02:11:56.599 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Aaw2IqTv8aUxV8D off=14149932 len=792071 2021-07-31T02:11:56.601 INFO:tasks.workunit.client.0.smithi045.stdout:op 2078 completed, throughput=4.99MB/sec 2021-07-31T02:11:56.601 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-0GEf7HDHn6S23IK off=4411146 len=1140553 2021-07-31T02:11:56.602 INFO:tasks.workunit.client.0.smithi045.stdout:op 2079 completed, throughput=5MB/sec 2021-07-31T02:11:56.603 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ehPPxOPbliwyoEM off=9201960 len=806898 2021-07-31T02:11:56.607 INFO:tasks.workunit.client.0.smithi045.stdout:op 2080 completed, throughput=5MB/sec 2021-07-31T02:11:56.607 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ABYmoS1ViNGX0Bg off=3772436 len=1547311 2021-07-31T02:11:56.610 INFO:tasks.workunit.client.0.smithi045.stdout:op 2081 completed, throughput=5MB/sec 2021-07-31T02:11:57.610 INFO:tasks.workunit.client.0.smithi045.stdout: 403: throughput=4.99MB/sec pending data=0 2021-07-31T02:11:57.611 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-GpUeJCY87ihsDC6 off=18885117 len=637611 2021-07-31T02:11:57.612 INFO:tasks.workunit.client.0.smithi045.stdout:op 2082 completed, throughput=4.99MB/sec 2021-07-31T02:11:57.613 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Ztwrfcc1QqZ7P2G off=17111531 len=31058 2021-07-31T02:11:57.613 INFO:tasks.workunit.client.0.smithi045.stdout:op 2083 completed, throughput=4.99MB/sec 2021-07-31T02:11:57.613 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-GBFO4DzcH2QTTJm off=20569907 len=1884426 2021-07-31T02:11:57.622 INFO:tasks.workunit.client.0.smithi045.stdout:op 2084 completed, throughput=5MB/sec 2021-07-31T02:11:57.622 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-k6kcQkayMXwrA71 off=4371998 len=1241104 2021-07-31T02:11:57.628 INFO:tasks.workunit.client.0.smithi045.stdout:op 2085 completed, throughput=5MB/sec 2021-07-31T02:11:57.629 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-cWaRiff9Yc_pSfC off=339979 len=69155 2021-07-31T02:11:57.631 INFO:tasks.workunit.client.0.smithi045.stdout:op 2086 completed, throughput=5MB/sec 2021-07-31T02:11:57.631 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-_dt--LMBNR89_RG off=7663549 len=1153519 2021-07-31T02:11:57.638 INFO:tasks.workunit.client.0.smithi045.stdout:op 2087 completed, throughput=5MB/sec 2021-07-31T02:11:58.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:58 smithi154 conmon[34553]: cluster 2021-07-31T02:11:58.145374+0000 mgr.x (mgr.24542) 1060 : cluster [DBG] pgmap v1614: 9 pgs: 9 active+clean; 127 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2021-07-31T02:11:58.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:11:58 smithi154 conmon[34553]: 2021-07-31T02:11:58.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:11:58 smithi045 conmon[33655]: cluster 2021-07-31T02:11:58.145374+0000 mgr.x (mgr.24542) 1060 : cluster [DBG] pgmap v1614: 9 pgs: 9 active+clean; 127 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2021-07-31T02:11:58.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:58 smithi045 conmon[39718]: cluster 2021-07-31T02:11:58.145374+0000 mgr.x ( 2021-07-31T02:11:58.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:11:58 smithi045 conmon[39718]: mgr.24542) 1060 : cluster [DBG] pgmap v1614: 9 pgs: 9 active+clean; 127 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2021-07-31T02:11:58.638 INFO:tasks.workunit.client.0.smithi045.stdout: 404: throughput=4.99MB/sec pending data=0 2021-07-31T02:11:58.638 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-N-zX3Zy52YIlcZq off=4485507 len=788358 2021-07-31T02:11:58.640 INFO:tasks.workunit.client.0.smithi045.stdout:op 2088 completed, throughput=4.99MB/sec 2021-07-31T02:11:58.640 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ULpxi5ixjN0JoEh off=8475962 len=1286600 2021-07-31T02:11:58.643 INFO:tasks.workunit.client.0.smithi045.stdout:op 2089 completed, throughput=4.99MB/sec 2021-07-31T02:11:58.643 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-6rxbA8Y-NthF5TT off=2814459 len=91076 2021-07-31T02:11:58.644 INFO:tasks.workunit.client.0.smithi045.stdout:op 2090 completed, throughput=4.99MB/sec 2021-07-31T02:11:58.645 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-husI1smTB7sfdqr off=2741085 len=1786167 2021-07-31T02:11:58.647 INFO:tasks.workunit.client.0.smithi045.stdout:op 2091 completed, throughput=5MB/sec 2021-07-31T02:11:58.647 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-IrW4FLbtjUgh92e off=2213141 len=1051560 2021-07-31T02:11:58.653 INFO:tasks.workunit.client.0.smithi045.stdout:op 2092 completed, throughput=5MB/sec 2021-07-31T02:11:59.653 INFO:tasks.workunit.client.0.smithi045.stdout: 405: throughput=4.99MB/sec pending data=0 2021-07-31T02:11:59.653 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-EUVwGMUjOEkG1gR off=990336 len=935642 2021-07-31T02:11:59.654 INFO:tasks.workunit.client.0.smithi045.stdout:op 2093 completed, throughput=4.99MB/sec 2021-07-31T02:11:59.655 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-biF-oMy_uzySKfO off=7876808 len=997349 2021-07-31T02:11:59.661 INFO:tasks.workunit.client.0.smithi045.stdout:op 2094 completed, throughput=4.99MB/sec 2021-07-31T02:11:59.661 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-K5Yo8AKPIQTN_EV off=5660944 len=1823326 2021-07-31T02:11:59.664 INFO:tasks.workunit.client.0.smithi045.stdout:op 2095 completed, throughput=5MB/sec 2021-07-31T02:11:59.665 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-a9uZc5YgFmJJ4C4 off=11734643 len=1010053 2021-07-31T02:11:59.666 INFO:tasks.workunit.client.0.smithi045.stdout:op 2096 completed, throughput=5MB/sec 2021-07-31T02:11:59.667 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-AcUqu09ilfo0-In off=9469419 len=1589394 2021-07-31T02:11:59.668 INFO:tasks.workunit.client.0.smithi045.stdout:op 2097 completed, throughput=5MB/sec 2021-07-31T02:12:00.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:00 smithi154 conmon[34553]: cluster 2021-07-31T02:12:00.145901+0000 mgr.x (mgr.24542) 1061 : cluster [DBG] 2021-07-31T02:12:00.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:00 smithi154 conmon[34553]: pgmap v1615: 9 pgs: 9 active+clean; 127 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 3.4 MiB/s rd, 3.0 MiB/s wr, 6 op/s 2021-07-31T02:12:00.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:00 smithi045 conmon[39718]: cluster 2021-07-31T02:12:00.145901+0000 mgr.x (mgr.24542) 1061 : cluster [DBG] pgmap v1615: 9 pgs: 9 active+clean; 127 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 3.4 MiB/s rd, 3.0 MiB/s wr, 6 op/s 2021-07-31T02:12:00.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:00 smithi045 conmon[33655]: cluster 2021-07-31T02:12:00.145901+0000 mgr.x (mgr.24542) 1061 : cluster [DBG] pgmap v1615: 9 pgs: 9 active+clean; 127 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 3.4 MiB/s rd, 3.0 MiB/s wr, 6 op/s 2021-07-31T02:12:00.668 INFO:tasks.workunit.client.0.smithi045.stdout: 406: throughput=4.99MB/sec pending data=0 2021-07-31T02:12:00.669 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-wH3Y1s688MiiFL3 off=792583 len=590736 2021-07-31T02:12:00.673 INFO:tasks.workunit.client.0.smithi045.stdout:op 2098 completed, throughput=4.99MB/sec 2021-07-31T02:12:00.673 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-6oDdizSuTOXa6L9 off=442236 len=1166091 2021-07-31T02:12:00.679 INFO:tasks.workunit.client.0.smithi045.stdout:op 2099 completed, throughput=4.99MB/sec 2021-07-31T02:12:00.680 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-iVadgO5Ju2X0BRY off=6737098 len=516160 2021-07-31T02:12:00.681 INFO:tasks.workunit.client.0.smithi045.stdout:op 2100 completed, throughput=5MB/sec 2021-07-31T02:12:00.681 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-m25SL3ehfxrbB_s off=9038876 len=639480 2021-07-31T02:12:00.682 INFO:tasks.workunit.client.0.smithi045.stdout:op 2101 completed, throughput=5MB/sec 2021-07-31T02:12:00.682 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-fJbofk_o6LtqbVU off=8286954 len=704566 2021-07-31T02:12:00.686 INFO:tasks.workunit.client.0.smithi045.stdout:op 2102 completed, throughput=5MB/sec 2021-07-31T02:12:00.687 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-78PD3hZnOq3jtGY off=9625916 len=456388 2021-07-31T02:12:00.691 INFO:tasks.workunit.client.0.smithi045.stdout:op 2103 completed, throughput=5MB/sec 2021-07-31T02:12:00.691 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-vVfdRK2A8iQthOr off=624510 len=1046931 2021-07-31T02:12:00.693 INFO:tasks.workunit.client.0.smithi045.stdout:op 2104 completed, throughput=5MB/sec 2021-07-31T02:12:01.694 INFO:tasks.workunit.client.0.smithi045.stdout: 407: throughput=4.99MB/sec pending data=0 2021-07-31T02:12:01.694 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-a0HkERDIALuVnjG off=22615047 len=946967 2021-07-31T02:12:01.696 INFO:tasks.workunit.client.0.smithi045.stdout:op 2105 completed, throughput=4.99MB/sec 2021-07-31T02:12:01.696 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-PGl_p8EAAbuMXbG off=4810358 len=829972 2021-07-31T02:12:01.698 INFO:tasks.workunit.client.0.smithi045.stdout:op 2106 completed, throughput=4.99MB/sec 2021-07-31T02:12:01.698 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-JmJwNfdWmbujpGD off=16678130 len=18489 2021-07-31T02:12:01.699 INFO:tasks.workunit.client.0.smithi045.stdout:op 2107 completed, throughput=4.99MB/sec 2021-07-31T02:12:01.699 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-7DISFaDA_zwHd3x off=273587 len=1653970 2021-07-31T02:12:01.700 INFO:tasks.workunit.client.0.smithi045.stdout:op 2108 completed, throughput=5MB/sec 2021-07-31T02:12:01.701 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-cyj-FlO1UMuvREl off=5032235 len=368461 2021-07-31T02:12:01.701 INFO:tasks.workunit.client.0.smithi045.stdout:op 2109 completed, throughput=5MB/sec 2021-07-31T02:12:01.702 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-t9V2zl4l54pdHp2 off=14971379 len=712474 2021-07-31T02:12:01.703 INFO:tasks.workunit.client.0.smithi045.stdout:op 2110 completed, throughput=5MB/sec 2021-07-31T02:12:02.703 INFO:tasks.workunit.client.0.smithi045.stdout: 408: throughput=4.99MB/sec pending data=0 2021-07-31T02:12:02.703 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-OixEOjW3KpHFXJx off=1641955 len=974322 2021-07-31T02:12:02.705 INFO:tasks.workunit.client.0.smithi045.stdout:op 2111 completed, throughput=4.99MB/sec 2021-07-31T02:12:02.706 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ZOhlhkZyTIEWGgX off=7654082 len=1676664 2021-07-31T02:12:02.713 INFO:tasks.workunit.client.0.smithi045.stdout:op 2112 completed, throughput=4.99MB/sec 2021-07-31T02:12:02.713 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-GfoermrejKEu3Cr off=8211336 len=1375073 2021-07-31T02:12:02.716 INFO:tasks.workunit.client.0.smithi045.stdout:op 2113 completed, throughput=5MB/sec 2021-07-31T02:12:02.716 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Ui6NXTDvodHNsv5 off=5652657 len=1108868 2021-07-31T02:12:02.721 INFO:tasks.workunit.client.0.smithi045.stdout:op 2114 completed, throughput=5MB/sec 2021-07-31T02:12:02.722 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-VVELVm9K7Pp8fJR off=3491635 len=2071188 2021-07-31T02:12:02.732 INFO:tasks.workunit.client.0.smithi045.stdout:op 2115 completed, throughput=5MB/sec 2021-07-31T02:12:03.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:02 smithi154 conmon[34553]: cluster 2021-07-31T02:12:02.146331+0000 mgr.x (mgr.24542) 1062 : cluster [DBG] pgmap v1616: 9 pgs: 9 active+clean; 127 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 2.3 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2021-07-31T02:12:03.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:02 smithi045 conmon[39718]: cluster 2021-07-31T02:12:02.146331+0000 mgr.x (mgr.24542) 1062 : cluster 2021-07-31T02:12:03.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:02 smithi045 conmon[39718]: [DBG] pgmap v1616: 9 pgs: 9 active+clean; 127 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 2.3 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2021-07-31T02:12:03.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:02 smithi045 conmon[33655]: cluster 2021-07-31T02:12:02 2021-07-31T02:12:03.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:02 smithi045 conmon[33655]: .146331+0000 mgr.x (mgr.24542) 1062 : cluster [DBG] pgmap v1616: 9 pgs: 9 active+clean; 127 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 2.3 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2021-07-31T02:12:03.732 INFO:tasks.workunit.client.0.smithi045.stdout: 409: throughput=4.99MB/sec pending data=0 2021-07-31T02:12:03.733 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-_8eCIfXfKZzp0HC off=1408930 len=2005972 2021-07-31T02:12:03.736 INFO:tasks.workunit.client.0.smithi045.stdout:op 2116 completed, throughput=5MB/sec 2021-07-31T02:12:03.736 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-78bxYNHFp1bUMrF off=15474853 len=1764008 2021-07-31T02:12:03.738 INFO:tasks.workunit.client.0.smithi045.stdout:op 2117 completed, throughput=5MB/sec 2021-07-31T02:12:04.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:12:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:12:03.550Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:12:04.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:12:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:12:03.550Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:12:04.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:12:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:12:03.551Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:12:04.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:04 smithi154 conmon[34553]: cluster 2021-07-31T02:12:04.146731+0000 mgr.x (mgr.24542) 1063 : cluster [DBG] pgmap v1617: 9 pgs: 9 active+clean; 127 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 3.8 MiB/s rd, 2.6 MiB/s wr, 6 op/s 2021-07-31T02:12:04.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:04 smithi154 conmon[34553]: 2021-07-31T02:12:04.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:04 smithi045 conmon[33655]: cluster 2021-07-31T02:12:04.146731+0000 mgr.x (mgr.24542) 1063 2021-07-31T02:12:04.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:04 smithi045 conmon[33655]: : cluster [DBG] pgmap v1617: 9 pgs: 9 active+clean; 127 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 3.8 MiB/s rd, 2.6 MiB/s wr, 6 op/s 2021-07-31T02:12:04.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:04 smithi045 conmon[39718]: cluster 2021-07-31T02:12:04.146731+0000 mgr.x (mgr.24542) 1063 : cluster 2021-07-31T02:12:04.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:04 smithi045 conmon[39718]: [DBG] pgmap v1617: 9 pgs: 9 active+clean; 127 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 3.8 MiB/s rd, 2.6 MiB/s wr, 6 op/s 2021-07-31T02:12:04.738 INFO:tasks.workunit.client.0.smithi045.stdout: 410: throughput=4.99MB/sec pending data=0 2021-07-31T02:12:04.739 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-_Vq2P7tpbszGmYN off=3724903 len=988866 2021-07-31T02:12:04.744 INFO:tasks.workunit.client.0.smithi045.stdout:op 2118 completed, throughput=4.99MB/sec 2021-07-31T02:12:04.745 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-qr9UCc_oKhwYiJs off=4327846 len=541024 2021-07-31T02:12:04.746 INFO:tasks.workunit.client.0.smithi045.stdout:op 2119 completed, throughput=4.99MB/sec 2021-07-31T02:12:04.746 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-lD3LqJRYnXg3-9J off=11952465 len=1016061 2021-07-31T02:12:04.753 INFO:tasks.workunit.client.0.smithi045.stdout:op 2120 completed, throughput=4.99MB/sec 2021-07-31T02:12:04.753 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-PpQxgff2nQ6yGE6 off=3986919 len=51181 2021-07-31T02:12:04.755 INFO:tasks.workunit.client.0.smithi045.stdout:op 2121 completed, throughput=4.99MB/sec 2021-07-31T02:12:04.755 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-AIX-VF-VCVyPLHX off=2045457 len=929636 2021-07-31T02:12:04.761 INFO:tasks.workunit.client.0.smithi045.stdout:op 2122 completed, throughput=5MB/sec 2021-07-31T02:12:04.761 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-js4kWNNxDUlJ2Pg off=2311432 len=975380 2021-07-31T02:12:04.765 INFO:tasks.workunit.client.0.smithi045.stdout:op 2123 completed, throughput=5MB/sec 2021-07-31T02:12:04.766 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-0hduCQwqZPlueea off=483674 len=1148749 2021-07-31T02:12:04.766 INFO:tasks.workunit.client.0.smithi045.stdout:op 2124 completed, throughput=5MB/sec 2021-07-31T02:12:05.767 INFO:tasks.workunit.client.0.smithi045.stdout: 411: throughput=4.99MB/sec pending data=0 2021-07-31T02:12:05.767 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-jZYIG2ZbJ68K-yu off=10979311 len=1796184 2021-07-31T02:12:05.770 INFO:tasks.workunit.client.0.smithi045.stdout:op 2125 completed, throughput=4.99MB/sec 2021-07-31T02:12:05.770 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-sUmhEMfzSQ2mGC1 off=3727607 len=1950722 2021-07-31T02:12:05.780 INFO:tasks.workunit.client.0.smithi045.stdout:op 2126 completed, throughput=5MB/sec 2021-07-31T02:12:05.781 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-hlGPA52sHCFKVtL off=5694807 len=963370 2021-07-31T02:12:05.785 INFO:tasks.workunit.client.0.smithi045.stdout:op 2127 completed, throughput=5MB/sec 2021-07-31T02:12:06.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:06 smithi154 conmon[34553]: cluster 2021-07-31T02:12:06.147151+0000 mgr.x (mgr.24542) 1064 : cluster [DBG] pgmap v1618: 9 pgs: 9 active+clean; 127 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 2.6 MiB/s rd, 1.7 MiB/s wr, 4 op/s 2021-07-31T02:12:06.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:06 smithi045 conmon[39718]: cluster 2021-07-31T02:12:06.147151+0000 mgr.x (mgr.24542) 1064 2021-07-31T02:12:06.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:06 smithi045 conmon[39718]: : cluster [DBG] pgmap v1618: 9 pgs: 9 active+clean; 127 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 2.6 MiB/s rd, 1.7 MiB/s wr, 4 op/s 2021-07-31T02:12:06.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:06 smithi045 conmon[33655]: cluster 2021-07-31T02:12:06.147151+0000 mgr.x (mgr.24542) 1064 : 2021-07-31T02:12:06.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:06 smithi045 conmon[33655]: cluster [DBG] pgmap v1618: 9 pgs: 9 active+clean; 127 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 2.6 MiB/s rd, 1.7 MiB/s wr, 4 op/s 2021-07-31T02:12:06.786 INFO:tasks.workunit.client.0.smithi045.stdout: 412: throughput=4.99MB/sec pending data=0 2021-07-31T02:12:06.786 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-T8bBHH5qQxe11Ux off=5001007 len=201573 2021-07-31T02:12:06.789 INFO:tasks.workunit.client.0.smithi045.stdout:op 2128 completed, throughput=4.99MB/sec 2021-07-31T02:12:06.789 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-q_kSlH7C5y5e49c off=5775680 len=1240751 2021-07-31T02:12:06.791 INFO:tasks.workunit.client.0.smithi045.stdout:op 2129 completed, throughput=4.99MB/sec 2021-07-31T02:12:06.792 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-JsQe02G4HagvRel off=9395614 len=974348 2021-07-31T02:12:06.797 INFO:tasks.workunit.client.0.smithi045.stdout:op 2130 completed, throughput=4.99MB/sec 2021-07-31T02:12:06.797 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-jbLNsn8MRsdqWo9 off=4347550 len=763481 2021-07-31T02:12:06.798 INFO:tasks.workunit.client.0.smithi045.stdout:op 2131 completed, throughput=5MB/sec 2021-07-31T02:12:06.799 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-9LAztMJ-yvGb2Wp off=5418869 len=2070899 2021-07-31T02:12:06.809 INFO:tasks.workunit.client.0.smithi045.stdout:op 2132 completed, throughput=5MB/sec 2021-07-31T02:12:06.809 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-I7111xEwQAYnMG0 off=18906543 len=2016168 2021-07-31T02:12:06.811 INFO:tasks.workunit.client.0.smithi045.stdout:op 2133 completed, throughput=5MB/sec 2021-07-31T02:12:07.812 INFO:tasks.workunit.client.0.smithi045.stdout: 413: throughput=4.99MB/sec pending data=0 2021-07-31T02:12:07.812 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-qWCwzu5-9NWiKMy off=2776624 len=560605 2021-07-31T02:12:07.816 INFO:tasks.workunit.client.0.smithi045.stdout:op 2134 completed, throughput=4.99MB/sec 2021-07-31T02:12:07.817 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-NA9J56ZicyHnz9U off=4884010 len=1877379 2021-07-31T02:12:07.819 INFO:tasks.workunit.client.0.smithi045.stdout:op 2135 completed, throughput=5MB/sec 2021-07-31T02:12:07.820 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-hJ1XdtJ_25NeHAE off=715583 len=1163856 2021-07-31T02:12:07.822 INFO:tasks.workunit.client.0.smithi045.stdout:op 2136 completed, throughput=5MB/sec 2021-07-31T02:12:08.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:08 smithi154 conmon[34553]: cluster 2021-07-31T02:12:08.147664+0000 mgr.x (mgr.24542) 1065 : cluster [DBG] pgmap v1619: 9 pgs: 9 active+clean; 127 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 2.6 MiB/s rd, 1.7 MiB/s wr, 4 op/s 2021-07-31T02:12:08.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:08 smithi045 conmon[33655]: cluster 2021-07-31T02:12:08.147664+0000 mgr.x (mgr.24542) 2021-07-31T02:12:08.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:08 smithi045 conmon[33655]: 1065 : cluster [DBG] pgmap v1619: 9 pgs: 9 active+clean; 127 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 2.6 MiB/s rd, 1.7 MiB/s wr, 4 op/s 2021-07-31T02:12:08.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:08 smithi045 conmon[39718]: cluster 2021-07-31T02:12:08.147664+0000 mgr.x (mgr.24542) 1065 : cluster [DBG] pgmap v1619: 9 pgs: 9 active+clean; 127 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 2.6 MiB/s rd, 1.7 MiB/s wr, 4 op/s 2021-07-31T02:12:08.822 INFO:tasks.workunit.client.0.smithi045.stdout: 414: throughput=4.99MB/sec pending data=0 2021-07-31T02:12:08.823 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-PntbPEBu9GMHdEm off=18951586 len=1951657 2021-07-31T02:12:08.825 INFO:tasks.workunit.client.0.smithi045.stdout:op 2137 completed, throughput=4.99MB/sec 2021-07-31T02:12:08.825 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-o8JTAgGg5vevwjs off=6869022 len=1760568 2021-07-31T02:12:08.828 INFO:tasks.workunit.client.0.smithi045.stdout:op 2138 completed, throughput=5MB/sec 2021-07-31T02:12:08.829 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-K0PmXOZWCP7_nfj off=198855 len=1399797 2021-07-31T02:12:08.835 INFO:tasks.workunit.client.0.smithi045.stdout:op 2139 completed, throughput=5MB/sec 2021-07-31T02:12:09.835 INFO:tasks.workunit.client.0.smithi045.stdout: 415: throughput=4.99MB/sec pending data=0 2021-07-31T02:12:09.836 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-VuXPd9x8T-gheYB off=1758972 len=827079 2021-07-31T02:12:09.840 INFO:tasks.workunit.client.0.smithi045.stdout:op 2140 completed, throughput=4.99MB/sec 2021-07-31T02:12:09.840 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Bz1NqGLXEE-FUWd off=12295594 len=933180 2021-07-31T02:12:09.846 INFO:tasks.workunit.client.0.smithi045.stdout:op 2141 completed, throughput=4.99MB/sec 2021-07-31T02:12:09.846 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-p7mEadYFcZQpzNa off=3061142 len=60982 2021-07-31T02:12:09.848 INFO:tasks.workunit.client.0.smithi045.stdout:op 2142 completed, throughput=4.99MB/sec 2021-07-31T02:12:09.848 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-srz6t2mNoB9qsu8 off=1143501 len=821293 2021-07-31T02:12:09.853 INFO:tasks.workunit.client.0.smithi045.stdout:op 2143 completed, throughput=4.99MB/sec 2021-07-31T02:12:09.853 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-rkxiqRTfsUCanIf off=9171264 len=490997 2021-07-31T02:12:09.854 INFO:tasks.workunit.client.0.smithi045.stdout:op 2144 completed, throughput=5MB/sec 2021-07-31T02:12:09.854 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-EmN0uzpKbzkWMja off=5494266 len=1213429 2021-07-31T02:12:09.855 INFO:tasks.workunit.client.0.smithi045.stdout:op 2145 completed, throughput=5MB/sec 2021-07-31T02:12:09.856 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-6TpOrS5kN4U4pKE off=2471479 len=1635637 2021-07-31T02:12:09.858 INFO:tasks.workunit.client.0.smithi045.stdout:op 2146 completed, throughput=5MB/sec 2021-07-31T02:12:10.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:10 smithi154 conmon[34553]: cluster 2021-07-31T02:12:10.148176+0000 mgr.x (mgr.24542) 1066 : cluster [DBG] pgmap v1620: 9 pgs: 9 active+clean; 127 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 3.7 MiB/s rd, 2.6 MiB/s wr, 6 op/s 2021-07-31T02:12:10.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:10 smithi045 conmon[33655]: cluster 2021-07-31T02:12:10.148176+0000 mgr.x (mgr 2021-07-31T02:12:10.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:10 smithi045 conmon[33655]: .24542) 1066 : cluster [DBG] pgmap v1620: 9 pgs: 9 active+clean; 127 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 3.7 MiB/s rd, 2.6 MiB/s wr, 6 op/s 2021-07-31T02:12:10.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:10 smithi045 conmon[39718]: cluster 2021-07-31T02:12:10.148176+0000 mgr.x (mgr.24542 2021-07-31T02:12:10.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:10 smithi045 conmon[39718]: ) 1066 : cluster [DBG] pgmap v1620: 9 pgs: 9 active+clean; 127 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 3.7 MiB/s rd, 2.6 MiB/s wr, 6 op/s 2021-07-31T02:12:10.859 INFO:tasks.workunit.client.0.smithi045.stdout: 416: throughput=4.99MB/sec pending data=0 2021-07-31T02:12:10.859 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj--sJJOMkxRLWL0os off=12638936 len=1325784 2021-07-31T02:12:10.860 INFO:tasks.workunit.client.0.smithi045.stdout:op 2147 completed, throughput=4.99MB/sec 2021-07-31T02:12:10.860 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-9EB_kN-McskoScR off=4562413 len=105497 2021-07-31T02:12:10.863 INFO:tasks.workunit.client.0.smithi045.stdout:op 2148 completed, throughput=4.99MB/sec 2021-07-31T02:12:10.863 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-z8VjQhHJ18ggNhr off=11460369 len=1790183 2021-07-31T02:12:10.866 INFO:tasks.workunit.client.0.smithi045.stdout:op 2149 completed, throughput=5MB/sec 2021-07-31T02:12:10.866 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-a9EyAxFMSd-7rCG off=3602846 len=1933142 2021-07-31T02:12:10.876 INFO:tasks.workunit.client.0.smithi045.stdout:op 2150 completed, throughput=5MB/sec 2021-07-31T02:12:11.012 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:12:11.292 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:12:11.293 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 27m ago 27m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:12:11.293 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 24m ago 27m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:12:11.293 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 24m ago 31m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:12:11.293 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 27m ago 33m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:12:11.294 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 27m ago 33m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:12:11.294 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 24m ago 31m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:12:11.294 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 27m ago 31m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:12:11.294 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 27m ago 28m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:12:11.295 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 24m ago 28m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:12:11.295 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 27m ago 30m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:12:11.295 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 27m ago 30m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:12:11.295 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 27m ago 30m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:12:11.295 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 27m ago 29m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:12:11.296 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 24m ago 29m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:12:11.296 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 24m ago 29m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:12:11.296 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 24m ago 28m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:12:11.296 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 24m ago 28m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:12:11.296 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 24m ago 28m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:12:11.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:11 smithi154 conmon[34553]: audit 2021-07-31T02:12:11.000744+0000 mgr.x (mgr.24542) 1067 : audit [DBG] from='client.15969 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:12:11.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:11 smithi045 conmon[39718]: audit 2021-07-31T02:12:11.000744+0000 mgr.x (mgr 2021-07-31T02:12:11.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:11 smithi045 conmon[39718]: .24542) 1067 : audit [DBG] from='client.15969 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:12:11.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:11 smithi045 conmon[33655]: audit 2021-07-31T02:12:11.000744+0000 mgr.x (mgr.24542) 1067 : 2021-07-31T02:12:11.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:11 smithi045 conmon[33655]: audit [DBG] from='client.15969 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:12:11.627 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:12:11.627 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:12:11.628 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:12:11.628 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:12:11.628 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:12:11.628 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:12:11.629 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:12:11.629 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:12:11.629 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:12:11.629 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:12:11.630 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:12:11.630 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:12:11.630 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:12:11.630 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:12:11.630 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:12:11.631 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:12:11.631 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:12:11.876 INFO:tasks.workunit.client.0.smithi045.stdout: 417: throughput=4.99MB/sec pending data=0 2021-07-31T02:12:11.877 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-DGPR5uVVCHnguMh off=2532993 len=514793 2021-07-31T02:12:11.877 INFO:tasks.workunit.client.0.smithi045.stdout:op 2151 completed, throughput=4.99MB/sec 2021-07-31T02:12:11.878 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-LLJ9D1ZasFYEaOY off=13607634 len=1230935 2021-07-31T02:12:11.884 INFO:tasks.workunit.client.0.smithi045.stdout:op 2152 completed, throughput=4.99MB/sec 2021-07-31T02:12:11.885 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-mouBOotPgSkXu02 off=17496369 len=2082415 2021-07-31T02:12:11.888 INFO:tasks.workunit.client.0.smithi045.stdout:op 2153 completed, throughput=5MB/sec 2021-07-31T02:12:11.888 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-FndNK_CEafn7tNY off=2837874 len=764442 2021-07-31T02:12:11.890 INFO:tasks.workunit.client.0.smithi045.stdout:op 2154 completed, throughput=5MB/sec 2021-07-31T02:12:11.890 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-BDi3ybexySxyeIQ off=432767 len=1931848 2021-07-31T02:12:11.893 INFO:tasks.workunit.client.0.smithi045.stdout:op 2155 completed, throughput=5MB/sec 2021-07-31T02:12:12.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:12 smithi154 conmon[34553]: audit 2021-07-31T02:12:11.284555+0000 mgr.x (mgr.24542) 1068 : audit [DBG] from='client.15975 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:12:12.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:12 smithi154 conmon[34553]: audit 2021-07-31T02:12:11.627547+0000 mon.a (mon.0) 1871 : audit [DBG] from='client.? 172.21.15.45:0/518101918' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:12:12.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:12 smithi154 conmon[34553]: cluster 2021-07-31T02:12:12.148599+0000 mgr.x (mgr.24542) 1069 : cluster [DBG] pgmap v1621: 9 pgs: 9 active+clean; 127 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 2.6 MiB/s rd, 1.7 MiB/s wr, 3 op/s 2021-07-31T02:12:12.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:12 smithi045 conmon[39718]: audit 2021-07-31T02:12:11.284555+0000 mgr.x (mgr.24542) 1068 2021-07-31T02:12:12.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:12 smithi045 conmon[39718]: : audit [DBG] from='client.15975 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:12:12.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:12 smithi045 conmon[39718]: audit 2021-07-31T02:12:11.627547+0000 mon.a (mon.0) 1871 : audit [DBG] from='client.? 172.21.15.45:0/518101918' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:12:12.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:12 smithi045 conmon[39718]: cluster 2021-07-31T02:12:12.148599+0000 mgr.x (mgr.24542) 1069 : cluster [DBG] pgmap v1621: 9 pgs: 9 active+clean; 127 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 2.6 MiB/s rd, 1.7 MiB/s wr, 3 op/s 2021-07-31T02:12:12.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:12 smithi045 conmon[33655]: audit 2021-07-31T02:12:11.284555+0000 mgr.x (mgr.24542) 1068 2021-07-31T02:12:12.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:12 smithi045 conmon[33655]: : audit [DBG] from='client.15975 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:12:12.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:12 smithi045 conmon[33655]: audit 2021-07-31T02:12:11.627547+0000 mon.a (mon.0) 1871 : audit [DBG] from='client.? 172.21.15.45:0/518101918' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:12:12.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:12 smithi045 conmon[33655]: cluster 2021-07 2021-07-31T02:12:12.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:12 smithi045 conmon[33655]: -31T02:12:12.148599+0000 mgr.x (mgr.24542) 1069 : cluster [DBG] pgmap v1621: 9 pgs: 9 active+clean; 127 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 2.6 MiB/s rd, 1.7 MiB/s wr, 3 op/s 2021-07-31T02:12:12.893 INFO:tasks.workunit.client.0.smithi045.stdout: 418: throughput=4.99MB/sec pending data=0 2021-07-31T02:12:12.893 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-GIU4ci4lj3o4sSe off=8128424 len=1845697 2021-07-31T02:12:12.905 INFO:tasks.workunit.client.0.smithi045.stdout:op 2156 completed, throughput=5MB/sec 2021-07-31T02:12:12.906 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-bWRBGDjstOLoEkU off=998644 len=654539 2021-07-31T02:12:12.907 INFO:tasks.workunit.client.0.smithi045.stdout:op 2157 completed, throughput=5MB/sec 2021-07-31T02:12:12.907 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-6V9pdz0O4z5EL40 off=9322910 len=1637443 2021-07-31T02:12:12.908 INFO:tasks.workunit.client.0.smithi045.stdout:op 2158 completed, throughput=5MB/sec 2021-07-31T02:12:13.909 INFO:tasks.workunit.client.0.smithi045.stdout: 419: throughput=4.99MB/sec pending data=0 2021-07-31T02:12:13.909 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-9wMb7cimJ4NZ5JG off=5090904 len=2014116 2021-07-31T02:12:13.911 INFO:tasks.workunit.client.0.smithi045.stdout:op 2159 completed, throughput=4.99MB/sec 2021-07-31T02:12:13.912 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-WsckOWBbg_ZQC4u off=11744037 len=251914 2021-07-31T02:12:13.913 INFO:tasks.workunit.client.0.smithi045.stdout:op 2160 completed, throughput=4.99MB/sec 2021-07-31T02:12:13.913 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-YgnSEdsQzVv9Jcl off=4054696 len=1938246 2021-07-31T02:12:13.916 INFO:tasks.workunit.client.0.smithi045.stdout:op 2161 completed, throughput=5MB/sec 2021-07-31T02:12:13.917 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-SFhmBvfyeK8FP8x off=2127467 len=683331 2021-07-31T02:12:13.921 INFO:tasks.workunit.client.0.smithi045.stdout:op 2162 completed, throughput=5MB/sec 2021-07-31T02:12:14.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:12:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:12:13.551Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:12:14.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:12:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:12:13.551Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:12:14.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:12:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:12:13.551Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:12:14.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:14 smithi154 conmon[34553]: cluster 2021-07-31T02:12:14.149109+0000 mgr.x (mgr.24542) 1070 : cluster [DBG] 2021-07-31T02:12:14.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:14 smithi154 conmon[34553]: pgmap v1622: 9 pgs: 9 active+clean; 127 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 4.0 MiB/s rd, 2.3 MiB/s wr, 5 op/s 2021-07-31T02:12:14.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:14 smithi045 conmon[39718]: cluster 2021-07-31T02:12:14.149109+0000 mgr.x (mgr.24542) 1070 : 2021-07-31T02:12:14.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:14 smithi045 conmon[39718]: cluster [DBG] pgmap v1622: 9 pgs: 9 active+clean; 127 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 4.0 MiB/s rd, 2.3 MiB/s wr, 5 op/s 2021-07-31T02:12:14.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:14 smithi045 conmon[33655]: cluster 2021-07-31T02:12:14.149109+0000 mgr.x (mgr.24542) 1070 : 2021-07-31T02:12:14.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:14 smithi045 conmon[33655]: cluster [DBG] pgmap v1622: 9 pgs: 9 active+clean; 127 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 4.0 MiB/s rd, 2.3 MiB/s wr, 5 op/s 2021-07-31T02:12:14.921 INFO:tasks.workunit.client.0.smithi045.stdout: 420: throughput=4.99MB/sec pending data=0 2021-07-31T02:12:14.922 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-T3xgKZX_S44a3kw off=883652 len=1265036 2021-07-31T02:12:14.927 INFO:tasks.workunit.client.0.smithi045.stdout:op 2163 completed, throughput=4.99MB/sec 2021-07-31T02:12:14.927 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-RyPiVDgtFuRLYrd off=185357 len=1397328 2021-07-31T02:12:14.934 INFO:tasks.workunit.client.0.smithi045.stdout:op 2164 completed, throughput=4.99MB/sec 2021-07-31T02:12:14.935 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-kevv5qVmwbTIWj- off=328850 len=1586087 2021-07-31T02:12:14.941 INFO:tasks.workunit.client.0.smithi045.stdout:op 2165 completed, throughput=5MB/sec 2021-07-31T02:12:14.941 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-MvbhGOA3zb0700Y off=5934102 len=1044836 2021-07-31T02:12:14.942 INFO:tasks.workunit.client.0.smithi045.stdout:op 2166 completed, throughput=5MB/sec 2021-07-31T02:12:15.942 INFO:tasks.workunit.client.0.smithi045.stdout: 421: throughput=4.99MB/sec pending data=0 2021-07-31T02:12:15.943 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-wugExeRyPNZ4QSX off=19408812 len=516108 2021-07-31T02:12:15.943 INFO:tasks.workunit.client.0.smithi045.stdout:op 2167 completed, throughput=4.99MB/sec 2021-07-31T02:12:15.944 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Lbrytz60YN820-e off=18171674 len=260277 2021-07-31T02:12:15.944 INFO:tasks.workunit.client.0.smithi045.stdout:op 2168 completed, throughput=4.99MB/sec 2021-07-31T02:12:15.945 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-MLQc7VD8VsZsGIS off=174152 len=332938 2021-07-31T02:12:15.948 INFO:tasks.workunit.client.0.smithi045.stdout:op 2169 completed, throughput=4.99MB/sec 2021-07-31T02:12:15.948 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-2hGBwtz7u2Rkq0z off=2973237 len=1018144 2021-07-31T02:12:15.949 INFO:tasks.workunit.client.0.smithi045.stdout:op 2170 completed, throughput=4.99MB/sec 2021-07-31T02:12:15.949 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-r9yczMXvEei9OXV off=5139639 len=1701541 2021-07-31T02:12:15.956 INFO:tasks.workunit.client.0.smithi045.stdout:op 2171 completed, throughput=5MB/sec 2021-07-31T02:12:15.957 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj--fss0bH8kwPuXBP off=4089178 len=794137 2021-07-31T02:12:15.958 INFO:tasks.workunit.client.0.smithi045.stdout:op 2172 completed, throughput=5MB/sec 2021-07-31T02:12:15.958 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-YPUJdr2i1kE-R0E off=18715298 len=546426 2021-07-31T02:12:15.959 INFO:tasks.workunit.client.0.smithi045.stdout:op 2173 completed, throughput=5MB/sec 2021-07-31T02:12:15.960 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-q94zkWFtZYoWZfb off=8156909 len=630008 2021-07-31T02:12:15.963 INFO:tasks.workunit.client.0.smithi045.stdout:op 2174 completed, throughput=5MB/sec 2021-07-31T02:12:16.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:16 smithi154 conmon[34553]: cluster 2021-07-31T02:12:16.149513+0000 mgr.x (mgr.24542) 1071 : cluster [DBG] 2021-07-31T02:12:16.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:16 smithi154 conmon[34553]: pgmap v1623: 9 pgs: 9 active+clean; 127 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 2.6 MiB/s rd, 1.6 MiB/s wr, 3 op/s 2021-07-31T02:12:16.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:16 smithi045 conmon[39718]: cluster 2021-07-31T02:12:16.149513+0000 mgr.x (mgr.24542) 1071 : cluster [DBG] 2021-07-31T02:12:16.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:16 smithi045 conmon[39718]: pgmap v1623: 9 pgs: 9 active+clean; 127 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 2.6 MiB/s rd, 1.6 MiB/s wr, 3 op/s 2021-07-31T02:12:16.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:16 smithi045 conmon[33655]: cluster 2021-07-31T02:12:16.149513+0000 mgr.x (mgr.24542) 2021-07-31T02:12:16.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:16 smithi045 conmon[33655]: 1071 : cluster [DBG] pgmap v1623: 9 pgs: 9 active+clean; 127 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 2.6 MiB/s rd, 1.6 MiB/s wr, 3 op/s 2021-07-31T02:12:16.963 INFO:tasks.workunit.client.0.smithi045.stdout: 422: throughput=4.99MB/sec pending data=0 2021-07-31T02:12:16.963 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-_qZ-EUeZ6q3Cb8p off=2224746 len=60185 2021-07-31T02:12:16.964 INFO:tasks.workunit.client.0.smithi045.stdout:op 2175 completed, throughput=4.99MB/sec 2021-07-31T02:12:16.964 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-iac-f_KsNQls-1M off=1446502 len=56688 2021-07-31T02:12:16.967 INFO:tasks.workunit.client.0.smithi045.stdout:op 2176 completed, throughput=4.99MB/sec 2021-07-31T02:12:16.967 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-6VRSAkjtIZvXzcC off=11424511 len=672032 2021-07-31T02:12:16.968 INFO:tasks.workunit.client.0.smithi045.stdout:op 2177 completed, throughput=4.99MB/sec 2021-07-31T02:12:16.969 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-2Z6TFRriSmGvQUj off=1367140 len=892829 2021-07-31T02:12:16.971 INFO:tasks.workunit.client.0.smithi045.stdout:op 2178 completed, throughput=4.99MB/sec 2021-07-31T02:12:16.971 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-AzacIh4FpaxTG09 off=7579723 len=1289921 2021-07-31T02:12:16.973 INFO:tasks.workunit.client.0.smithi045.stdout:op 2179 completed, throughput=5MB/sec 2021-07-31T02:12:16.973 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-eFW3DTwveJ6luDk off=764369 len=369886 2021-07-31T02:12:16.977 INFO:tasks.workunit.client.0.smithi045.stdout:op 2180 completed, throughput=5MB/sec 2021-07-31T02:12:16.977 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-TQGezdugAkg3y68 off=1230196 len=1898944 2021-07-31T02:12:16.985 INFO:tasks.workunit.client.0.smithi045.stdout:op 2181 completed, throughput=5MB/sec 2021-07-31T02:12:17.986 INFO:tasks.workunit.client.0.smithi045.stdout: 423: throughput=4.99MB/sec pending data=0 2021-07-31T02:12:17.986 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ZT3pCfH3pG3mmtY off=23374749 len=532196 2021-07-31T02:12:17.990 INFO:tasks.workunit.client.0.smithi045.stdout:op 2182 completed, throughput=4.99MB/sec 2021-07-31T02:12:17.990 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-m36VxIUkNpQRRkd off=5938264 len=1607884 2021-07-31T02:12:17.998 INFO:tasks.workunit.client.0.smithi045.stdout:op 2183 completed, throughput=4.99MB/sec 2021-07-31T02:12:17.999 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-AhwM1o79TgczaGq off=5483435 len=1732880 2021-07-31T02:12:18.000 INFO:tasks.workunit.client.0.smithi045.stdout:op 2184 completed, throughput=5MB/sec 2021-07-31T02:12:18.000 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-XCqPTzz41ez6Ieg off=1778122 len=117331 2021-07-31T02:12:18.003 INFO:tasks.workunit.client.0.smithi045.stdout:op 2185 completed, throughput=5MB/sec 2021-07-31T02:12:18.003 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-vfqfrrEHmZvzhbi off=15519317 len=1201476 2021-07-31T02:12:18.009 INFO:tasks.workunit.client.0.smithi045.stdout:op 2186 completed, throughput=5MB/sec 2021-07-31T02:12:18.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:18 smithi154 conmon[34553]: cluster 2021-07-31T02:12:18.150004+0000 mgr.x (mgr.24542) 2021-07-31T02:12:18.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:18 smithi154 conmon[34553]: 1072 : cluster [DBG] pgmap v1624: 9 pgs: 9 active+clean; 127 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 2.6 MiB/s rd, 1.6 MiB/s wr, 3 op/s 2021-07-31T02:12:18.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:18 smithi045 conmon[33655]: cluster 2021-07-31T02:12:18.150004+0000 2021-07-31T02:12:18.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:18 smithi045 conmon[33655]: mgr.x (mgr.24542) 1072 : cluster [DBG] pgmap v1624: 9 pgs: 9 active+clean; 127 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 2.6 MiB/s rd, 1.6 MiB/s wr, 3 op/s 2021-07-31T02:12:18.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:18 smithi045 conmon[39718]: cluster 2021-07-31T02:12:18.150004+0000 mgr.x (mgr.24542) 2021-07-31T02:12:18.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:18 smithi045 conmon[39718]: 1072 : cluster [DBG] pgmap v1624: 9 pgs: 9 active+clean; 127 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 2.6 MiB/s rd, 1.6 MiB/s wr, 3 op/s 2021-07-31T02:12:19.009 INFO:tasks.workunit.client.0.smithi045.stdout: 424: throughput=4.99MB/sec pending data=0 2021-07-31T02:12:19.009 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-6s4bi5tSKUBQ2-M off=4279596 len=1627187 2021-07-31T02:12:19.011 INFO:tasks.workunit.client.0.smithi045.stdout:op 2187 completed, throughput=4.99MB/sec 2021-07-31T02:12:19.011 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-FVxtS5ICOIMi59n off=13906206 len=66362 2021-07-31T02:12:19.014 INFO:tasks.workunit.client.0.smithi045.stdout:op 2188 completed, throughput=4.99MB/sec 2021-07-31T02:12:19.015 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-cLmpNH-YinbzdOz off=2582093 len=1039072 2021-07-31T02:12:19.020 INFO:tasks.workunit.client.0.smithi045.stdout:op 2189 completed, throughput=4.99MB/sec 2021-07-31T02:12:19.020 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-xTAYTN-yjlzarKx off=798167 len=572515 2021-07-31T02:12:19.022 INFO:tasks.workunit.client.0.smithi045.stdout:op 2190 completed, throughput=5MB/sec 2021-07-31T02:12:19.022 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Czo08kHMF6NGFxp off=2789800 len=179691 2021-07-31T02:12:19.022 INFO:tasks.workunit.client.0.smithi045.stdout:op 2191 completed, throughput=5MB/sec 2021-07-31T02:12:19.023 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-VyGS2bYAJfe8h9W off=15815624 len=188656 2021-07-31T02:12:19.023 INFO:tasks.workunit.client.0.smithi045.stdout:op 2192 completed, throughput=5MB/sec 2021-07-31T02:12:19.024 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-zaDxzYEUI_0pz7J off=9145272 len=49154 2021-07-31T02:12:19.024 INFO:tasks.workunit.client.0.smithi045.stdout:op 2193 completed, throughput=5MB/sec 2021-07-31T02:12:19.025 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-6jBMyZULwjql9iw off=3160921 len=1267686 2021-07-31T02:12:19.027 INFO:tasks.workunit.client.0.smithi045.stdout:op 2194 completed, throughput=5MB/sec 2021-07-31T02:12:19.027 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-7Gf9Ci1zyo44_Ok off=1513324 len=665521 2021-07-31T02:12:19.031 INFO:tasks.workunit.client.0.smithi045.stdout:op 2195 completed, throughput=5MB/sec 2021-07-31T02:12:20.032 INFO:tasks.workunit.client.0.smithi045.stdout: 425: throughput=4.99MB/sec pending data=0 2021-07-31T02:12:20.032 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-O0bTzVs8HAlXAam off=4470511 len=669822 2021-07-31T02:12:20.034 INFO:tasks.workunit.client.0.smithi045.stdout:op 2196 completed, throughput=4.99MB/sec 2021-07-31T02:12:20.034 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ZeCn5gS-tngfawL off=1593213 len=1728158 2021-07-31T02:12:20.042 INFO:tasks.workunit.client.0.smithi045.stdout:op 2197 completed, throughput=4.99MB/sec 2021-07-31T02:12:20.042 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-oR_minXstoGg_X6 off=6320008 len=1263481 2021-07-31T02:12:20.049 INFO:tasks.workunit.client.0.smithi045.stdout:op 2198 completed, throughput=5MB/sec 2021-07-31T02:12:20.049 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-NeiqrRixhky0Sli off=2498425 len=1606661 2021-07-31T02:12:20.056 INFO:tasks.workunit.client.0.smithi045.stdout:op 2199 completed, throughput=5MB/sec 2021-07-31T02:12:20.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:20 smithi154 conmon[34553]: cluster 2021-07-31T02:12:20.150450+0000 mgr.x (mgr.24542) 1073 : cluster 2021-07-31T02:12:20.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:20 smithi154 conmon[34553]: [DBG] pgmap v1625: 9 pgs: 9 active+clean; 127 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 3.3 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2021-07-31T02:12:20.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:20 smithi045 conmon[39718]: cluster 2021-07-31T02:12:20.150450+0000 mgr.x (mgr.24542) 2021-07-31T02:12:20.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:20 smithi045 conmon[39718]: 1073 : cluster [DBG] pgmap v1625: 9 pgs: 9 active+clean; 127 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 3.3 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2021-07-31T02:12:20.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:20 smithi045 conmon[33655]: cluster 2021-07-31T02:12:20.150450+0000 mgr.x (mgr.24542) 1073 : cluster 2021-07-31T02:12:20.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:20 smithi045 conmon[33655]: [DBG] pgmap v1625: 9 pgs: 9 active+clean; 127 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 3.3 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2021-07-31T02:12:21.057 INFO:tasks.workunit.client.0.smithi045.stdout: 426: throughput=4.99MB/sec pending data=0 2021-07-31T02:12:21.057 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-uAT95zbNWE_Ux0U off=4027861 len=1923682 2021-07-31T02:12:21.059 INFO:tasks.workunit.client.0.smithi045.stdout:op 2200 completed, throughput=4.99MB/sec 2021-07-31T02:12:21.059 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-tu69hWTYnwjXIHr off=7855020 len=1850487 2021-07-31T02:12:21.068 INFO:tasks.workunit.client.0.smithi045.stdout:op 2201 completed, throughput=5MB/sec 2021-07-31T02:12:21.068 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-9AQt9wTLkrJ5iPB off=641787 len=375910 2021-07-31T02:12:21.069 INFO:tasks.workunit.client.0.smithi045.stdout:op 2202 completed, throughput=5MB/sec 2021-07-31T02:12:21.069 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-zdWQI3m3PzEDoXH off=4365086 len=702603 2021-07-31T02:12:21.071 INFO:tasks.workunit.client.0.smithi045.stdout:op 2203 completed, throughput=5MB/sec 2021-07-31T02:12:21.071 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-mtD20Ustw6sxJWQ off=2230276 len=1014611 2021-07-31T02:12:21.077 INFO:tasks.workunit.client.0.smithi045.stdout:op 2204 completed, throughput=5MB/sec 2021-07-31T02:12:22.078 INFO:tasks.workunit.client.0.smithi045.stdout: 427: throughput=4.99MB/sec pending data=0 2021-07-31T02:12:22.079 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-SrBumC4zogdfjgg off=4384785 len=350933 2021-07-31T02:12:22.080 INFO:tasks.workunit.client.0.smithi045.stdout:op 2205 completed, throughput=4.99MB/sec 2021-07-31T02:12:22.080 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ft6OCPPQmw13hFU off=11321803 len=1200450 2021-07-31T02:12:22.088 INFO:tasks.workunit.client.0.smithi045.stdout:op 2206 completed, throughput=4.99MB/sec 2021-07-31T02:12:22.089 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-UrCJ1S4_NCNHCm7 off=1335378 len=1590958 2021-07-31T02:12:22.095 INFO:tasks.workunit.client.0.smithi045.stdout:op 2207 completed, throughput=5MB/sec 2021-07-31T02:12:22.095 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-sZTAOTSj_Zbj6Gl off=2305840 len=646539 2021-07-31T02:12:22.099 INFO:tasks.workunit.client.0.smithi045.stdout:op 2208 completed, throughput=5MB/sec 2021-07-31T02:12:22.100 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-H7MlgJJLeuQlMXZ off=10267410 len=529075 2021-07-31T02:12:22.101 INFO:tasks.workunit.client.0.smithi045.stdout:op 2209 completed, throughput=5MB/sec 2021-07-31T02:12:22.101 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-DCkBYbCHDjJtK0A off=3909571 len=1636072 2021-07-31T02:12:22.102 INFO:tasks.workunit.client.0.smithi045.stdout:op 2210 completed, throughput=5MB/sec 2021-07-31T02:12:23.103 INFO:tasks.workunit.client.0.smithi045.stdout: 428: throughput=4.99MB/sec pending data=0 2021-07-31T02:12:23.103 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-V1wm-BefsiR8Kk- off=1324164 len=410321 2021-07-31T02:12:23.107 INFO:tasks.workunit.client.0.smithi045.stdout:op 2211 completed, throughput=4.99MB/sec 2021-07-31T02:12:23.107 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-B3Iuug7l74PmaLl off=13929675 len=1029411 2021-07-31T02:12:23.109 INFO:tasks.workunit.client.0.smithi045.stdout:op 2212 completed, throughput=4.99MB/sec 2021-07-31T02:12:23.109 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-hjugnlPtAkHBtlE off=4389318 len=679343 2021-07-31T02:12:23.111 INFO:tasks.workunit.client.0.smithi045.stdout:op 2213 completed, throughput=5MB/sec 2021-07-31T02:12:23.111 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-cOt9YInsPdpKHkS off=9845107 len=298174 2021-07-31T02:12:23.114 INFO:tasks.workunit.client.0.smithi045.stdout:op 2214 completed, throughput=5MB/sec 2021-07-31T02:12:23.114 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-opn1BnGhjNOtxnx off=3047805 len=973694 2021-07-31T02:12:23.119 INFO:tasks.workunit.client.0.smithi045.stdout:op 2215 completed, throughput=5MB/sec 2021-07-31T02:12:23.119 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ztExfRw6I5LbVVP off=1547526 len=165379 2021-07-31T02:12:23.120 INFO:tasks.workunit.client.0.smithi045.stdout:op 2216 completed, throughput=5MB/sec 2021-07-31T02:12:23.120 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-vn6DrAnyTDW4KqM off=4972272 len=1592353 2021-07-31T02:12:23.123 INFO:tasks.workunit.client.0.smithi045.stdout:op 2217 completed, throughput=5MB/sec 2021-07-31T02:12:23.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:22 smithi154 conmon[34553]: cluster 2021-07-31T02:12:22.150852+0000 mgr.x (mgr.24542) 1074 2021-07-31T02:12:23.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:22 smithi154 conmon[34553]: : cluster [DBG] pgmap v1626: 9 pgs: 9 active+clean; 127 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 2.2 MiB/s rd, 1.7 MiB/s wr, 3 op/s 2021-07-31T02:12:23.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:22 smithi045 conmon[39718]: cluster 2021-07-31T02:12:22.150852+0000 mgr.x 2021-07-31T02:12:23.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:22 smithi045 conmon[39718]: (mgr.24542) 1074 : cluster [DBG] pgmap v1626: 9 pgs: 9 active+clean; 127 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 2.2 MiB/s rd, 1.7 MiB/s wr, 3 op/s 2021-07-31T02:12:23.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:22 smithi045 conmon[33655]: cluster 2021-07-31T02: 2021-07-31T02:12:23.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:22 smithi045 conmon[33655]: 12:22.150852+0000 mgr.x (mgr.24542) 1074 : cluster [DBG] pgmap v1626: 9 pgs: 9 active+clean; 127 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 2.2 MiB/s rd, 1.7 MiB/s wr, 3 op/s 2021-07-31T02:12:24.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:12:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:12:23.551Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:12:24.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:12:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:12:23.551Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:12:24.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:12:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:12:23.551Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:12:24.123 INFO:tasks.workunit.client.0.smithi045.stdout: 429: throughput=4.99MB/sec pending data=0 2021-07-31T02:12:24.123 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-dkWPhNDJ91y7N9y off=2572500 len=979497 2021-07-31T02:12:24.125 INFO:tasks.workunit.client.0.smithi045.stdout:op 2218 completed, throughput=4.99MB/sec 2021-07-31T02:12:24.126 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-iup9RKNrKeKa1Jz off=7354910 len=1572270 2021-07-31T02:12:24.128 INFO:tasks.workunit.client.0.smithi045.stdout:op 2219 completed, throughput=5MB/sec 2021-07-31T02:12:24.129 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-OnAwrOlOwl9pyDR off=1951172 len=2064812 2021-07-31T02:12:24.131 INFO:tasks.workunit.client.0.smithi045.stdout:op 2220 completed, throughput=5MB/sec 2021-07-31T02:12:24.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:24 smithi154 conmon[34553]: cluster 2021-07-31T02:12:24.151408+0000 mgr.x (mgr.24542) 1075 : cluster [DBG] 2021-07-31T02:12:24.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:24 smithi154 conmon[34553]: pgmap v1627: 9 pgs: 9 active+clean; 127 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.8 MiB/s wr, 6 op/s 2021-07-31T02:12:24.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:24 smithi045 conmon[39718]: cluster 2021-07-31T02:12:24.151408+0000 mgr.x (mgr. 2021-07-31T02:12:24.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:24 smithi045 conmon[39718]: 24542) 1075 : cluster [DBG] pgmap v1627: 9 pgs: 9 active+clean; 127 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.8 MiB/s wr, 6 op/s 2021-07-31T02:12:24.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:24 smithi045 conmon[33655]: cluster 2021-07-31T02:12:24.151408 2021-07-31T02:12:24.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:24 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 1075 : cluster [DBG] pgmap v1627: 9 pgs: 9 active+clean; 127 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.8 MiB/s wr, 6 op/s 2021-07-31T02:12:25.131 INFO:tasks.workunit.client.0.smithi045.stdout: 430: throughput=4.99MB/sec pending data=0 2021-07-31T02:12:25.132 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-YYUVz5IyTX6Oub7 off=5173488 len=1733148 2021-07-31T02:12:25.134 INFO:tasks.workunit.client.0.smithi045.stdout:op 2221 completed, throughput=4.99MB/sec 2021-07-31T02:12:25.134 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-WBjt5m69sI0Nvhl off=2224507 len=95718 2021-07-31T02:12:25.136 INFO:tasks.workunit.client.0.smithi045.stdout:op 2222 completed, throughput=4.99MB/sec 2021-07-31T02:12:25.136 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-4zT9oHMzBw7buTX off=14382465 len=248207 2021-07-31T02:12:25.142 INFO:tasks.workunit.client.0.smithi045.stdout:op 2223 completed, throughput=4.99MB/sec 2021-07-31T02:12:25.142 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-CJL6rRdrm4lswnM off=559159 len=1112583 2021-07-31T02:12:25.145 INFO:tasks.workunit.client.0.smithi045.stdout:op 2224 completed, throughput=5MB/sec 2021-07-31T02:12:25.146 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-SFhmBvfyeK8FP8x off=8139776 len=62985 2021-07-31T02:12:25.148 INFO:tasks.workunit.client.0.smithi045.stdout:op 2225 completed, throughput=5MB/sec 2021-07-31T02:12:25.149 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-UMpdZIuZ3OQnXD9 off=3484570 len=1812957 2021-07-31T02:12:25.156 INFO:tasks.workunit.client.0.smithi045.stdout:op 2226 completed, throughput=5MB/sec 2021-07-31T02:12:26.156 INFO:tasks.workunit.client.0.smithi045.stdout: 431: throughput=4.99MB/sec pending data=0 2021-07-31T02:12:26.157 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ENfEMnanxb2EZA7 off=1843263 len=502646 2021-07-31T02:12:26.160 INFO:tasks.workunit.client.0.smithi045.stdout:op 2227 completed, throughput=4.99MB/sec 2021-07-31T02:12:26.161 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-YA7Pffw_7sCe53T off=7729655 len=1373456 2021-07-31T02:12:26.163 INFO:tasks.workunit.client.0.smithi045.stdout:op 2228 completed, throughput=4.99MB/sec 2021-07-31T02:12:26.163 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-B9KvvDK_1HYHL85 off=15724425 len=2025522 2021-07-31T02:12:26.174 INFO:tasks.workunit.client.0.smithi045.stdout:op 2229 completed, throughput=5MB/sec 2021-07-31T02:12:26.174 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-C8pFHELGrXDkDs7 off=3532382 len=1071214 2021-07-31T02:12:26.180 INFO:tasks.workunit.client.0.smithi045.stdout:op 2230 completed, throughput=5MB/sec 2021-07-31T02:12:26.180 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-106-iq117YxZjCk off=7990871 len=247915 2021-07-31T02:12:26.182 INFO:tasks.workunit.client.0.smithi045.stdout:op 2231 completed, throughput=5MB/sec 2021-07-31T02:12:26.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:26 smithi154 conmon[34553]: cluster 2021-07-31T02:12:26.151787+0000 mgr.x (mgr.24542) 1076 : cluster 2021-07-31T02:12:26.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:26 smithi154 conmon[34553]: [DBG] pgmap v1628: 9 pgs: 9 active+clean; 127 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2021-07-31T02:12:26.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:26 smithi045 conmon[39718]: cluster 2021-07-31T02:12:26.151787+0000 mgr.x (mgr.24542) 1076 : cluster [DBG] pgmap v1628: 9 pgs: 9 active+clean; 127 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2021-07-31T02:12:26.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:26 smithi045 conmon[33655]: cluster 2021-07-31T02:12:26.151787+0000 mgr.x (mgr.24542) 2021-07-31T02:12:26.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:26 smithi045 conmon[33655]: 1076 : cluster [DBG] pgmap v1628: 9 pgs: 9 active+clean; 127 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2021-07-31T02:12:27.182 INFO:tasks.workunit.client.0.smithi045.stdout: 432: throughput=4.99MB/sec pending data=0 2021-07-31T02:12:27.183 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ixGuvkgsO98mcbo off=9426862 len=1177011 2021-07-31T02:12:27.189 INFO:tasks.workunit.client.0.smithi045.stdout:op 2232 completed, throughput=4.99MB/sec 2021-07-31T02:12:27.189 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-zh7CJNEy460vxOs off=11263439 len=850985 2021-07-31T02:12:27.191 INFO:tasks.workunit.client.0.smithi045.stdout:op 2233 completed, throughput=4.99MB/sec 2021-07-31T02:12:27.191 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-gHgrcqScWWvjmbG off=3847404 len=998897 2021-07-31T02:12:27.196 INFO:tasks.workunit.client.0.smithi045.stdout:op 2234 completed, throughput=5MB/sec 2021-07-31T02:12:27.197 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-uZrQnkz4JJ4jVFo off=7441555 len=833459 2021-07-31T02:12:27.198 INFO:tasks.workunit.client.0.smithi045.stdout:op 2235 completed, throughput=5MB/sec 2021-07-31T02:12:27.199 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-r76tlRDVp-xrwO9 off=3051689 len=2058676 2021-07-31T02:12:27.208 INFO:tasks.workunit.client.0.smithi045.stdout:op 2236 completed, throughput=5MB/sec 2021-07-31T02:12:28.208 INFO:tasks.workunit.client.0.smithi045.stdout: 433: throughput=4.99MB/sec pending data=0 2021-07-31T02:12:28.209 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-QyYEDN6aQgYW6Ly off=4048972 len=2063438 2021-07-31T02:12:28.212 INFO:tasks.workunit.client.0.smithi045.stdout:op 2237 completed, throughput=4.99MB/sec 2021-07-31T02:12:28.213 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-92XN7f9HJ3xLGlD off=1909774 len=1999611 2021-07-31T02:12:28.216 INFO:tasks.workunit.client.0.smithi045.stdout:op 2238 completed, throughput=5MB/sec 2021-07-31T02:12:28.216 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-UGMhCwHJVWmC8nG off=1024274 len=1741566 2021-07-31T02:12:28.224 INFO:tasks.workunit.client.0.smithi045.stdout:op 2239 completed, throughput=5MB/sec 2021-07-31T02:12:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:28 smithi154 conmon[34553]: audit 2021-07-31T02:12:28.023031+0000 mon.b (mon.2) 193 : audit 2021-07-31T02:12:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:28 smithi154 conmon[34553]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:12:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:28 smithi154 conmon[34553]: audit 2021-07-31T02:12:28.024043+0000 mon.a (mon.0) 1872 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:12:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:28 smithi154 conmon[34553]: audit 2021-07-31T02:12:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:28 smithi154 conmon[34553]: 2021-07-31T02:12:28.043417+0000 mon.b (mon.2) 194 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:12:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:28 smithi154 conmon[34553]: audit 2021-07-31T02: 2021-07-31T02:12:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:28 smithi154 conmon[34553]: 12:28.044314+0000 mon.a (mon.0) 1873 : audit 2021-07-31T02:12:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:28 smithi154 conmon[34553]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:12:28.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:28 smithi045 conmon[39718]: audit 2021-07-31T02:12:28.023031 2021-07-31T02:12:28.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:28 smithi045 conmon[39718]: +0000 mon.b (mon.2) 193 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:12:28.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:28 smithi045 conmon[39718]: audit 2021-07 2021-07-31T02:12:28.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:28 smithi045 conmon[39718]: -31T02:12:28.024043+0000 mon.a (mon.0) 2021-07-31T02:12:28.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:28 smithi045 conmon[39718]: 1872 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:12:28.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:28 smithi045 conmon[39718]: audit 2021- 2021-07-31T02:12:28.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:28 smithi045 conmon[39718]: 07-31T02:12:28.043417+0000 2021-07-31T02:12:28.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:28 smithi045 conmon[39718]: mon.b (mon.2) 194 : audit [INF] 2021-07-31T02:12:28.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:28 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:12:28.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:28 smithi045 conmon[39718]: audit 2021-07-31T02:12:28.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:28 smithi045 conmon[39718]: 2021-07-31T02:12 2021-07-31T02:12:28.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:28 smithi045 conmon[39718]: :28.044314+0000 mon.a (mon 2021-07-31T02:12:28.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:28 smithi045 conmon[39718]: .0) 1873 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:12:28.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:28 smithi045 conmon[33655]: audit 2021-07-31T02:12:28.023031+0000 mon.b (mon.2) 193 : audit [INF] 2021-07-31T02:12:28.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:28 smithi045 conmon[33655]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:12:28.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:28 smithi045 conmon[33655]: audit 2021-07-31T02:12:28.024043+0000 mon.a (mon.0) 1872 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:12:28.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:28 smithi045 conmon[33655]: audit 2021-07-31 2021-07-31T02:12:28.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:28 smithi045 conmon[33655]: T02:12:28.043417+0000 mon.b (mon 2021-07-31T02:12:28.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:28 smithi045 conmon[33655]: .2) 194 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:12:28.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:28 smithi045 conmon[33655]: audit 2021-07 2021-07-31T02:12:28.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:28 smithi045 conmon[33655]: -31T02:12:28.044314+0000 mon.a (mon.0) 1873 : audit [INF] 2021-07-31T02:12:28.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:28 smithi045 conmon[33655]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:12:29.224 INFO:tasks.workunit.client.0.smithi045.stdout: 434: throughput=4.99MB/sec pending data=0 2021-07-31T02:12:29.225 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-OJccfo6H3Ot3Gdf off=2750969 len=1704156 2021-07-31T02:12:29.227 INFO:tasks.workunit.client.0.smithi045.stdout:op 2240 completed, throughput=4.99MB/sec 2021-07-31T02:12:29.227 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-CjrBK4Psa9z6ez6 off=1670753 len=163345 2021-07-31T02:12:29.230 INFO:tasks.workunit.client.0.smithi045.stdout:op 2241 completed, throughput=5MB/sec 2021-07-31T02:12:29.230 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-3bIFe5YNnD4hCFi off=23317102 len=494887 2021-07-31T02:12:29.234 INFO:tasks.workunit.client.0.smithi045.stdout:op 2242 completed, throughput=5MB/sec 2021-07-31T02:12:29.234 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-25CKU_OUPiBll-H off=5272113 len=41489 2021-07-31T02:12:29.235 INFO:tasks.workunit.client.0.smithi045.stdout:op 2243 completed, throughput=5MB/sec 2021-07-31T02:12:29.235 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ukPUPFPKOA1tvz5 off=18352871 len=1668309 2021-07-31T02:12:29.242 INFO:tasks.workunit.client.0.smithi045.stdout:op 2244 completed, throughput=5MB/sec 2021-07-31T02:12:29.242 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-b6J447j0ZhC0iiW off=7805801 len=778432 2021-07-31T02:12:29.247 INFO:tasks.workunit.client.0.smithi045.stdout:op 2245 completed, throughput=5MB/sec 2021-07-31T02:12:29.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:29 smithi154 conmon[34553]: cluster 2021-07-31T02:12:28.152183+0000 mgr.x (mgr.24542) 1077 : cluster 2021-07-31T02:12:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:29 smithi154 conmon[34553]: [DBG] pgmap v1629: 9 pgs: 9 active+clean; 127 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2021-07-31T02:12:29.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:29 smithi045 conmon[39718]: cluster 2021-07-31T02:12:28.152183+0000 mgr.x (mgr 2021-07-31T02:12:29.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:29 smithi045 conmon[39718]: .24542) 1077 : cluster [DBG] pgmap v1629: 9 pgs: 9 active+clean; 127 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2021-07-31T02:12:29.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:29 smithi045 conmon[33655]: cluster 2021-07-31T02:12:28.152183+0000 2021-07-31T02:12:29.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:29 smithi045 conmon[33655]: mgr.x (mgr.24542) 1077 : cluster [DBG] pgmap v1629: 9 pgs: 9 active+clean; 127 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2021-07-31T02:12:30.247 INFO:tasks.workunit.client.0.smithi045.stdout: 435: throughput=4.99MB/sec pending data=0 2021-07-31T02:12:30.247 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-FCbiwZSAyXSl6AP off=6438283 len=2007690 2021-07-31T02:12:30.255 INFO:tasks.workunit.client.0.smithi045.stdout:op 2246 completed, throughput=4.99MB/sec 2021-07-31T02:12:30.256 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-zJlLHVuTam6TNcS off=2850888 len=1226277 2021-07-31T02:12:30.256 INFO:tasks.workunit.client.0.smithi045.stdout:op 2247 completed, throughput=5MB/sec 2021-07-31T02:12:30.257 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-_yU3b2WFLalHyHu off=7555446 len=1703613 2021-07-31T02:12:30.264 INFO:tasks.workunit.client.0.smithi045.stdout:op 2248 completed, throughput=5MB/sec 2021-07-31T02:12:30.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[34553]: cluster 2021-07-31T02:12:30.152748+0000 mgr.x (mgr.24542) 1078 : cluster 2021-07-31T02:12:30.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[34553]: [DBG] pgmap v1630: 9 pgs: 9 active+clean; 127 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 2.8 MiB/s rd, 3.3 MiB/s wr, 6 op/s 2021-07-31T02:12:30.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:30 smithi045 conmon[33655]: cluster 2021-07-31T02:12:30 2021-07-31T02:12:30.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:30 smithi045 conmon[33655]: .152748+0000 mgr.x (mgr.24542) 1078 : cluster [DBG] pgmap v1630: 9 pgs: 9 active+clean; 127 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 2.8 MiB/s rd, 3.3 MiB/s wr, 6 op/s 2021-07-31T02:12:30.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:30 smithi045 conmon[39718]: cluster 2021-07-31T02:12: 2021-07-31T02:12:30.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:30 smithi045 conmon[39718]: 30.152748+0000 mgr.x (mgr.24542) 1078 : cluster [DBG] pgmap v1630: 9 pgs: 9 active+clean; 127 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 2.8 MiB/s rd, 3.3 MiB/s wr, 6 op/s 2021-07-31T02:12:31.003 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: debug 2021-07-31T02:12:30.625+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:12:31.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:12:31.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:12:31.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: yield 2021-07-31T02:12:31.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:12:31.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:12:31.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:12:31.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:12:31.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:12:31.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:12:31.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:12:31.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:12:31.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:12:31.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:12:31.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:12:31.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: debug 2021-07-31T02:12:30.637+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:12:31.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:12:31.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:12:31.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: yield 2021-07-31T02:12:31.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:12:31.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:12:31.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:12:31.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:12:31.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:12:31.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:12:31.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:12:31.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:12:31.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:12:31.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:12:31.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:12:31.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: debug 2021-07-31T02:12:30.662+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:12:31.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:12:31.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:12:31.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: yield 2021-07-31T02:12:31.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:12:31.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:12:31.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:12:31.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:12:31.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:12:31.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:12:31.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:12:31.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:12:31.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:12:31.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:12:31.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:12:31.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: debug 2021-07-31T02:12:30.663+0000 7fcb94162700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:12:31.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:12:31.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:12:31.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: yield 2021-07-31T02:12:31.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:12:31.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:12:31.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:12:31.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:12:31.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:12:31.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:12:31.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:12:31.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:12:31.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:12:31.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:12:31.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:12:31.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: 2021-07-31T02:12:31.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:12:31.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: 2021-07-31T02:12:31.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:12:31.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:12:31.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:12:31.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:12:31.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:12:31.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:12:31.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:12:31.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:12:31.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:12:31.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:12:31.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:12:31.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:12:31.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:12:31.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:12:31.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:12:31.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:12:31.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:12:31.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:12:31.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: raise 2021-07-31T02:12:31.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:12:31.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:12:31.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:12:31.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:12:31.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:12:31.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: debug 2021-07-31T02:12:30.671+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:12:31.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:12:31.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:12:31.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: yield 2021-07-31T02:12:31.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:12:31.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:12:31.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:12:31.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:12:31.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:12:31.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:12:31.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:12:31.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:12:31.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:12:31.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:12:31.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:12:31.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: debug 2021-07-31T02:12:30.672+0000 7fcb93961700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:12:31.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:12:31.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:12:31.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: yield 2021-07-31T02:12:31.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:12:31.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:12:31.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:12:31.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:12:31.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:12:31.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:12:31.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:12:31.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:12:31.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:12:31.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:12:31.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:12:31.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: 2021-07-31T02:12:31.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:12:31.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: 2021-07-31T02:12:31.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:12:31.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:12:31.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:12:31.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:12:31.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:12:31.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:12:31.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:12:31.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:12:31.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:12:31.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:12:31.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:12:31.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:12:31.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:12:31.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:12:31.035 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:12:31.035 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:12:31.035 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:12:31.036 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:12:31.036 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: raise 2021-07-31T02:12:31.037 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:12:31.037 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:12:31.037 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:12:31.037 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:12:31.037 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:12:30 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:12:31.265 INFO:tasks.workunit.client.0.smithi045.stdout: 436: throughput=4.99MB/sec pending data=0 2021-07-31T02:12:31.265 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-xYyQzatKnigNbKX off=13355198 len=1618601 2021-07-31T02:12:31.268 INFO:tasks.workunit.client.0.smithi045.stdout:op 2249 completed, throughput=4.99MB/sec 2021-07-31T02:12:31.269 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-OwRvjgfHIo8j48v off=10155877 len=1874796 2021-07-31T02:12:31.277 INFO:tasks.workunit.client.0.smithi045.stdout:op 2250 completed, throughput=5MB/sec 2021-07-31T02:12:31.278 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-QWYmXhsFNee0STQ off=0 len=1245776 2021-07-31T02:12:31.279 INFO:tasks.workunit.client.0.smithi045.stdout:op 2251 completed, throughput=5MB/sec 2021-07-31T02:12:31.279 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-CAk6WLKNJvNc312 off=5790358 len=355928 2021-07-31T02:12:31.280 INFO:tasks.workunit.client.0.smithi045.stdout:op 2252 completed, throughput=5MB/sec 2021-07-31T02:12:31.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: audit 2021-07-31T02:12:30.580127+0000 mon.b (mon.2) 195 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:12:31.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: audit 2021-07-31T02:12:30.581667+0000 mon.b (mon.2) 196 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:12:31.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: audit 2021-07-31T02:12 2021-07-31T02:12:31.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: :30.582629+0000 mon.b (mon.2) 197 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:12:31.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: cephadm 2021-07-31 2021-07-31T02:12:31.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: T02:12:30.583498+0000 mgr.x (mgr 2021-07-31T02:12:31.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: .24542) 1079 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:12:31.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: cephadm 2021-07-31T 2021-07-31T02:12:31.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: 02:12:30.626598+0000 mgr.x (mgr.24542) 1080 : cephadm 2021-07-31T02:12:31.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: [ERR] 2021-07-31T02:12:31.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:12:31.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:12:31.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: yield 2021-07-31T02:12:31.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:12:31.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:12:31.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:12:31.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:12:31.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:12:31.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:12:31.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:12:31.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:12:31.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:12:31.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: task.result() 2021-07-31T02:12:31.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:12:31.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:12:30.627025+0000 mgr.x (mgr.24542) 1081 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:12:31.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:12:30.638336+0000 mgr.x (mgr.24542) 1082 : cephadm [ERR] 2021-07-31T02:12:31.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:12:31.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:12:31.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: yield 2021-07-31T02:12:31.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:12:31.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:12:31.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:12:31.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:12:31.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:12:31.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:12:31.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:12:31.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:12:31.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:12:31.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: task.result() 2021-07-31T02:12:31.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:12:31.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: 2021-07-31T02:12:31.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:12:30.639152+0000 mgr.x (mgr.24542) 1083 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:12:31.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:12:30.663674+0000 mgr.x (mgr.24542) 1084 : cephadm [ERR] 2021-07-31T02:12:31.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:12:31.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:12:31.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: yield 2021-07-31T02:12:31.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:12:31.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:12:31.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:12:31.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:12:31.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:12:31.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:12:31.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:12:31.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:12:31.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:12:31.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: task.result() 2021-07-31T02:12:31.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:12:31.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:12:30.664106+0000 mgr.x (mgr.24542) 1085 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:12:31.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:12:31.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:12:31.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: yield 2021-07-31T02:12:31.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:12:31.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:12:31.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:12:31.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:12:31.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:12:31.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:12:31.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:12:31.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:12:31.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:12:31.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: task.result() 2021-07-31T02:12:31.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:12:31.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: 2021-07-31T02:12:31.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:12:31.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: 2021-07-31T02:12:31.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:12:31.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:12:31.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:12:31.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:12:31.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:12:31.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:12:31.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:12:31.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:12:31.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:12:31.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:12:31.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:12:31.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:12:31.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:12:31.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:12:31.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:12:31.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:12:31.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:12:31.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:12:31.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: raise 2021-07-31T02:12:31.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:12:31.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:12:31.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:12:31.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:12:31.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:12:31.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:12:30.672587+0000 2021-07-31T02:12:31.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: mgr.x (mgr.24542) 1086 : cephadm [ERR] 2021-07-31T02:12:31.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:12:31.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:12:31.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: yield 2021-07-31T02:12:31.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:12:31.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:12:31.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:12:31.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:12:31.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:12:31.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:12:31.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:12:31.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:12:31.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:12:31.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: task.result() 2021-07-31T02:12:31.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:12:31.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:12:30.673651+0000 mgr.x (mgr.24542) 1087 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:12:31.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:12:31.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:12:31.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: yield 2021-07-31T02:12:31.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:12:31.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:12:31.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:12:31.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:12:31.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:12:31.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:12:31.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:12:31.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:12:31.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:12:31.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: task.result() 2021-07-31T02:12:31.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:12:31.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: 2021-07-31T02:12:31.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:12:31.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: 2021-07-31T02:12:31.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:12:31.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:12:31.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:12:31.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:12:31.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:12:31.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:12:31.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:12:31.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:12:31.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:12:31.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:12:31.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:12:31.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:12:31.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:12:31.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:12:31.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:12:31.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:12:31.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:12:31.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:12:31.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: raise 2021-07-31T02:12:31.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:12:31.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:12:31.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:12:31.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:12:31.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:31 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:12:31.539 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: audit 2021-07-31T02:12:30. 2021-07-31T02:12:31.539 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: 580127+0000 mon.b (mon.2) 195 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:12:31.540 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: audit 2021-07-31T02:12:31.540 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: 2021-07 2021-07-31T02:12:31.540 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: -31T02:12:30.581667 2021-07-31T02:12:31.540 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: +0000 mon.b (mon.2) 196 : audit [DBG] 2021-07-31T02:12:31.540 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:12:31.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: audit 2021- 2021-07-31T02:12:31.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: 07-31T02:12:30.582629+0000 mon.b (mon 2021-07-31T02:12:31.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: .2) 197 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:12:31.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:12: 2021-07-31T02:12:31.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: 30.583498+0000 mgr.x (mgr.24542) 1079 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:12:31.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:12:30.626598+0000 mgr.x (mgr.24542) 1080 : cephadm 2021-07-31T02:12:31.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: [ERR] 2021-07-31T02:12:31.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:12:31.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:12:31.543 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: yield 2021-07-31T02:12:31.543 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:12:31.543 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:12:31.543 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:12:31.544 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:12:31.544 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:12:31.544 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:12:31.544 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:12:31.544 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:12:31.545 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:12:31.545 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: task.result() 2021-07-31T02:12:31.545 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:12:31.545 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: cephadm 2021-07- 2021-07-31T02:12:31.546 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: 31T02:12:30.627025+0000 mgr.x (mgr.24542) 1081 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:12:31.546 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:12:30.638336+0000 mgr.x (mgr.24542) 1082 : cephadm [ERR] 2021-07-31T02:12:31.546 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:12:31.546 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:12:31.546 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: yield 2021-07-31T02:12:31.547 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:12:31.547 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:12:31.547 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:12:31.547 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:12:31.547 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:12:31.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: audit 2021-07-31T02:12:30.580127+0000 2021-07-31T02:12:31.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: mon.b (mon.2) 195 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:12:31.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: audit 2021-07-31T02:12:31.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: 2021-07-31 2021-07-31T02:12:31.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: T02:12:30.581667+0000 2021-07-31T02:12:31.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: mon.b (mon.2) 196 : audit [DBG] 2021-07-31T02:12:31.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:12:31.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: audit 2021 2021-07-31T02:12:31.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: -07-31T02:12: 2021-07-31T02:12:31.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: 30.582629+0000 mon.b (mon. 2021-07-31T02:12:31.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: 2) 197 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:12:31.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: cephadm 2021-07 2021-07-31T02:12:31.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: -31T02:12:30.583498+0000 2021-07-31T02:12:31.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: mgr.x (mgr.24542) 1079 : cephadm [WRN] 2021-07-31T02:12:31.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:12:31.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:12:31.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: 2021-07-31 2021-07-31T02:12:31.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: T02:12:30.626598+0000 2021-07-31T02:12:31.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: mgr.x (mgr.24542) 1080 : cephadm 2021-07-31T02:12:31.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: [ERR] 2021-07-31T02:12:31.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:12:31.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:12:31.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: yield 2021-07-31T02:12:31.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:12:31.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:12:31.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:12:31.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:12:31.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:12:31.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:12:31.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:12:31.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:12:31.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:12:31.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: task.result() 2021-07-31T02:12:31.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:12:31.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:12:30.627025+0000 mgr.x (mgr.24542) 1081 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:12:31.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:12:31.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: 2021-07-31T 2021-07-31T02:12:31.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: 02:12:30.638336+0000 mgr.x 2021-07-31T02:12:31.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: (mgr.24542) 1082 : cephadm [ERR] 2021-07-31T02:12:31.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: 2021-07-31T02:12:31.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:12:31.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:12:31.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: yield 2021-07-31T02:12:31.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:12:31.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:12:31.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:12:31.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:12:31.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:12:31.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: task.result() 2021-07-31T02:12:31.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:12:31.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: cephadm 2021-07-31T02 2021-07-31T02:12:31.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: :12:30.639152+0000 mgr.x (mgr.24542) 1083 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:12:31.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:12:30.663674+0000 mgr.x (mgr.24542) 1084 : cephadm [ERR] 2021-07-31T02:12:31.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:12:31.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:12:31.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: yield 2021-07-31T02:12:31.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:12:31.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:12:31.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:12:31.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:12:31.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:12:31.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:12:31.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:12:31.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:12:31.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:12:31.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: task.result() 2021-07-31T02:12:31.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:12:31.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:12:30.664106+0000 mgr.x (mgr.24542) 1085 : 2021-07-31T02:12:31.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:12:31.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:12:31.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:12:31.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: yield 2021-07-31T02:12:31.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:12:31.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:12:31.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:12:31.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:12:31.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:12:31.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:12:31.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:12:31.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:12:31.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:12:31.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: task.result() 2021-07-31T02:12:31.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:12:31.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: 2021-07-31T02:12:31.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:12:31.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: 2021-07-31T02:12:31.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:12:31.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:12:31.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:12:31.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:12:31.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:12:31.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:12:31.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:12:31.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:12:31.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: task.result() 2021-07-31T02:12:31.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:12:31.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:12:30.639152+0000 mgr.x (mgr.24542) 1083 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:12:31.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:12:31.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: 2021-07-31T02: 2021-07-31T02:12:31.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: 12:30.663674+0000 mgr.x ( 2021-07-31T02:12:31.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: mgr.24542) 1084 : cephadm [ERR] 2021-07-31T02:12:31.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:12:31.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:12:31.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: yield 2021-07-31T02:12:31.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:12:31.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:12:31.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:12:31.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:12:31.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:12:31.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:12:31.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:12:31.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:12:31.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:12:31.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: task.result() 2021-07-31T02:12:31.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:12:31.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: 2021-07-31T02:12:31.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:12:30.664106+0000 mgr.x (mgr.24542) 1085 2021-07-31T02:12:31.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:12:31.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:12:31.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:12:31.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: yield 2021-07-31T02:12:31.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:12:31.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:12:31.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:12:31.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:12:31.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:12:31.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:12:31.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:12:31.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:12:31.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:12:31.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:12:31.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:12:31.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:12:31.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:12:31.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:12:31.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:12:31.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:12:31.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:12:31.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:12:31.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:12:31.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:12:31.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:12:31.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:12:31.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:12:31.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:12:31.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:12:31.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: raise 2021-07-31T02:12:31.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:12:31.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:12:31.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:12:31.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:12:31.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:12:31.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:12:30.672587+0000 mgr.x (mgr.24542) 1086 : cephadm [ERR] 2021-07-31T02:12:31.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:12:31.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:12:31.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: yield 2021-07-31T02:12:31.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:12:31.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:12:31.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:12:31.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:12:31.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:12:31.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:12:31.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:12:31.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:12:31.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:12:31.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: task.result() 2021-07-31T02:12:31.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:12:31.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:12:30.673651+0000 mgr.x (mgr.24542) 1087 : 2021-07-31T02:12:31.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:12:31.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:12:31.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: task.result() 2021-07-31T02:12:31.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:12:31.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: 2021-07-31T02:12:31.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:12:31.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: 2021-07-31T02:12:31.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:12:31.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:12:31.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:12:31.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:12:31.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:12:31.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:12:31.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:12:31.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:12:31.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:12:31.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:12:31.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:12:31.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:12:31.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:12:31.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:12:31.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:12:31.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:12:31.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:12:31.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:12:31.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: raise 2021-07-31T02:12:31.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:12:31.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:12:31.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:12:31.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:12:31.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:12:31.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:12:30.672587+0000 mgr.x (mgr.24542) 1086 : cephadm [ERR] 2021-07-31T02:12:31.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:12:31.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:12:31.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: yield 2021-07-31T02:12:31.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:12:31.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:12:31.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:12:31.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:12:31.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:12:31.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:12:31.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:12:31.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:12:31.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:12:31.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: yield 2021-07-31T02:12:31.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:12:31.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:12:31.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:12:31.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:12:31.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:12:31.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:12:31.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:12:31.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:12:31.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:12:31.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: task.result() 2021-07-31T02:12:31.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:12:31.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: 2021-07-31T02:12:31.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:12:31.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: 2021-07-31T02:12:31.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:12:31.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:12:31.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:12:31.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:12:31.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:12:31.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:12:31.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:12:31.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:12:31.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:12:31.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:12:31.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:12:31.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:12:31.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:12:31.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:12:31.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:12:31.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:12:31.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:12:31.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:12:31.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: raise 2021-07-31T02:12:31.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:12:31.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:12:31.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:12:31.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:12:31.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:12:31.607 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:12:31.607 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:12:31.607 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:12:31.607 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: task.result() 2021-07-31T02:12:31.608 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:12:31.608 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:12:30.673651+0000 mgr.x (mgr.24542) 1087 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:12:31.608 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:12:31.608 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:12:31.609 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: yield 2021-07-31T02:12:31.609 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:12:31.609 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:12:31.609 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:12:31.609 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:12:31.610 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:12:31.610 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:12:31.610 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:12:31.610 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:12:31.611 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:12:31.611 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: task.result() 2021-07-31T02:12:31.611 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:12:31.611 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: 2021-07-31T02:12:31.611 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:12:31.612 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: 2021-07-31T02:12:31.612 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:12:31.612 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:12:31.612 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:12:31.613 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:12:31.613 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:12:31.613 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:12:31.613 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:12:31.613 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:12:31.614 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:12:31.614 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:12:31.614 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:12:31.614 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:12:31.615 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:12:31.615 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:12:31.615 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:12:31.615 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:12:31.616 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:12:31.616 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:12:31.616 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: raise 2021-07-31T02:12:31.616 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:12:31.617 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:12:31.617 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:12:31.617 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:12:31.617 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:31 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:12:32.281 INFO:tasks.workunit.client.0.smithi045.stdout: 437: throughput=4.99MB/sec pending data=0 2021-07-31T02:12:32.281 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-sDnYnNFIQ4XMv57 off=9925154 len=796553 2021-07-31T02:12:32.286 INFO:tasks.workunit.client.0.smithi045.stdout:op 2253 completed, throughput=4.99MB/sec 2021-07-31T02:12:32.286 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-DybewbmPBy34ovl off=19111079 len=1125629 2021-07-31T02:12:32.289 INFO:tasks.workunit.client.0.smithi045.stdout:op 2254 completed, throughput=4.99MB/sec 2021-07-31T02:12:32.290 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj--cHkaSFZYqGVPRP off=1800030 len=1910314 2021-07-31T02:12:32.299 INFO:tasks.workunit.client.0.smithi045.stdout:op 2255 completed, throughput=5MB/sec 2021-07-31T02:12:32.299 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-fRM_PR8xj0pImq5 off=11256687 len=299570 2021-07-31T02:12:32.300 INFO:tasks.workunit.client.0.smithi045.stdout:op 2256 completed, throughput=5MB/sec 2021-07-31T02:12:32.300 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-KEwZ7Twq-6aaivO off=6807433 len=1782060 2021-07-31T02:12:32.303 INFO:tasks.workunit.client.0.smithi045.stdout:op 2257 completed, throughput=5MB/sec 2021-07-31T02:12:33.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:32 smithi154 conmon[34553]: cluster 2021-07-31T02:12:32.153218+0000 mgr.x (mgr.24542) 1088 : cluster [DBG] pgmap v1631: 9 pgs: 9 active+clean; 127 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2021-07-31T02:12:33.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:32 smithi154 conmon[34553]: 2021-07-31T02:12:33.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:32 smithi045 conmon[33655]: cluster 2021-07-31T02:12:32.153218 2021-07-31T02:12:33.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:32 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 1088 : cluster [DBG] pgmap v1631: 9 pgs: 9 active+clean; 127 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2021-07-31T02:12:33.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:32 smithi045 conmon[39718]: cluster 2021-07-31T02:12:32.153218+0000 mgr.x ( 2021-07-31T02:12:33.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:32 smithi045 conmon[39718]: mgr.24542) 1088 : cluster [DBG] pgmap v1631: 9 pgs: 9 active+clean; 127 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2021-07-31T02:12:33.304 INFO:tasks.workunit.client.0.smithi045.stdout: 438: throughput=4.99MB/sec pending data=0 2021-07-31T02:12:33.304 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-wLQFNPlPltmTJFX off=4855188 len=1031520 2021-07-31T02:12:33.310 INFO:tasks.workunit.client.0.smithi045.stdout:op 2258 completed, throughput=4.99MB/sec 2021-07-31T02:12:33.310 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-7uvyIzPd2fXy3rR off=9451255 len=1460432 2021-07-31T02:12:33.313 INFO:tasks.workunit.client.0.smithi045.stdout:op 2259 completed, throughput=5MB/sec 2021-07-31T02:12:33.313 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ugg2xMpau_8nN3H off=6783575 len=275976 2021-07-31T02:12:33.314 INFO:tasks.workunit.client.0.smithi045.stdout:op 2260 completed, throughput=5MB/sec 2021-07-31T02:12:33.314 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-2vH640eV1MMwjhD off=11832628 len=1595630 2021-07-31T02:12:33.320 INFO:tasks.workunit.client.0.smithi045.stdout:op 2261 completed, throughput=5MB/sec 2021-07-31T02:12:33.320 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-PA6vp8ZA0gXFClw off=721845 len=941858 2021-07-31T02:12:33.325 INFO:tasks.workunit.client.0.smithi045.stdout:op 2262 completed, throughput=5MB/sec 2021-07-31T02:12:34.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:12:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:12:33.552Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:12:34.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:12:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:12:33.552Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:12:34.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:12:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:12:33.552Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:12:34.326 INFO:tasks.workunit.client.0.smithi045.stdout: 439: throughput=4.99MB/sec pending data=0 2021-07-31T02:12:34.326 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-yT_bPIpDN3LRUOc off=24046839 len=257440 2021-07-31T02:12:34.326 INFO:tasks.workunit.client.0.smithi045.stdout:op 2263 completed, throughput=4.99MB/sec 2021-07-31T02:12:34.327 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-lFHWl_ySVUxwiAu off=5637690 len=580812 2021-07-31T02:12:34.328 INFO:tasks.workunit.client.0.smithi045.stdout:op 2264 completed, throughput=4.99MB/sec 2021-07-31T02:12:34.328 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-5kwJU9ct5dw9G8z off=1764520 len=560291 2021-07-31T02:12:34.332 INFO:tasks.workunit.client.0.smithi045.stdout:op 2265 completed, throughput=4.99MB/sec 2021-07-31T02:12:34.333 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-fz17lPLFRD1yTRR off=4030279 len=596894 2021-07-31T02:12:34.337 INFO:tasks.workunit.client.0.smithi045.stdout:op 2266 completed, throughput=4.99MB/sec 2021-07-31T02:12:34.337 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-TFjXlRF0G18ZqzY off=6204598 len=94604 2021-07-31T02:12:34.337 INFO:tasks.workunit.client.0.smithi045.stdout:op 2267 completed, throughput=4.99MB/sec 2021-07-31T02:12:34.338 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-peG_kQdfvZNd0xm off=1698289 len=150158 2021-07-31T02:12:34.338 INFO:tasks.workunit.client.0.smithi045.stdout:op 2268 completed, throughput=4.99MB/sec 2021-07-31T02:12:34.339 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-AnRZv_jr_lwc0jt off=6672818 len=247027 2021-07-31T02:12:34.341 INFO:tasks.workunit.client.0.smithi045.stdout:op 2269 completed, throughput=5MB/sec 2021-07-31T02:12:34.341 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-hXflMp-avCwKJiH off=4008607 len=829449 2021-07-31T02:12:34.347 INFO:tasks.workunit.client.0.smithi045.stdout:op 2270 completed, throughput=5MB/sec 2021-07-31T02:12:34.347 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ykoemC5KzHgFT-o off=4907342 len=1934820 2021-07-31T02:12:34.349 INFO:tasks.workunit.client.0.smithi045.stdout:op 2271 completed, throughput=5MB/sec 2021-07-31T02:12:34.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:34 smithi154 conmon[34553]: cluster 2021-07-31T02:12:34.153822+0000 mgr.x (mgr.24542) 1089 : cluster [DBG] 2021-07-31T02:12:34.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:34 smithi154 conmon[34553]: pgmap v1632: 9 pgs: 9 active+clean; 127 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 3.0 MiB/s rd, 3.4 MiB/s wr, 6 op/s 2021-07-31T02:12:34.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:34 smithi045 conmon[39718]: cluster 2021-07-31T02:12:34.153822+0000 mgr.x (mgr.24542) 1089 : cluster [DBG] pgmap v1632: 9 pgs: 9 active+clean; 127 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 3.0 MiB/s rd, 3.4 MiB/s wr, 6 op/s 2021-07-31T02:12:34.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:34 smithi045 conmon[39718]: 2021-07-31T02:12:34.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:34 smithi045 conmon[33655]: cluster 2021-07-31T02:12:34.153822+0000 mgr.x (mgr.24542) 1089 2021-07-31T02:12:34.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:34 smithi045 conmon[33655]: : cluster [DBG] pgmap v1632: 9 pgs: 9 active+clean; 127 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 3.0 MiB/s rd, 3.4 MiB/s wr, 6 op/s 2021-07-31T02:12:35.349 INFO:tasks.workunit.client.0.smithi045.stdout: 440: throughput=4.99MB/sec pending data=0 2021-07-31T02:12:35.350 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-YDv9l7ksFHIs4BC off=1403000 len=435778 2021-07-31T02:12:35.353 INFO:tasks.workunit.client.0.smithi045.stdout:op 2272 completed, throughput=4.99MB/sec 2021-07-31T02:12:35.353 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-E8RUqi30zBYEFS1 off=14361473 len=396917 2021-07-31T02:12:35.357 INFO:tasks.workunit.client.0.smithi045.stdout:op 2273 completed, throughput=4.99MB/sec 2021-07-31T02:12:35.357 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-msb4qtmjNCG7QG- off=3592368 len=499673 2021-07-31T02:12:35.358 INFO:tasks.workunit.client.0.smithi045.stdout:op 2274 completed, throughput=4.99MB/sec 2021-07-31T02:12:35.358 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-BDXYS7AQV3U4d52 off=0 len=1209057 2021-07-31T02:12:35.361 INFO:tasks.workunit.client.0.smithi045.stdout:op 2275 completed, throughput=4.99MB/sec 2021-07-31T02:12:35.361 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-gTDM3NnGuwHT4y- off=4748697 len=956751 2021-07-31T02:12:35.362 INFO:tasks.workunit.client.0.smithi045.stdout:op 2276 completed, throughput=5MB/sec 2021-07-31T02:12:35.362 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-OFC0LUI32tMdg5K off=644441 len=1447348 2021-07-31T02:12:35.364 INFO:tasks.workunit.client.0.smithi045.stdout:op 2277 completed, throughput=5MB/sec 2021-07-31T02:12:36.365 INFO:tasks.workunit.client.0.smithi045.stdout: 441: throughput=4.99MB/sec pending data=0 2021-07-31T02:12:36.365 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-BmCJeD91zY0FdiJ off=10191238 len=1475354 2021-07-31T02:12:36.371 INFO:tasks.workunit.client.0.smithi045.stdout:op 2278 completed, throughput=4.99MB/sec 2021-07-31T02:12:36.372 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-LnOT--8tuwZad0E off=1860789 len=1289836 2021-07-31T02:12:36.375 INFO:tasks.workunit.client.0.smithi045.stdout:op 2279 completed, throughput=4.99MB/sec 2021-07-31T02:12:36.375 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-0hduCQwqZPlueea off=2851054 len=992463 2021-07-31T02:12:36.375 INFO:tasks.workunit.client.0.smithi045.stdout:op 2280 completed, throughput=5MB/sec 2021-07-31T02:12:36.376 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-gomcfebI8BibztY off=3971090 len=1775979 2021-07-31T02:12:36.377 INFO:tasks.workunit.client.0.smithi045.stdout:op 2281 completed, throughput=5MB/sec 2021-07-31T02:12:36.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:36 smithi154 conmon[34553]: cluster 2021-07-31T02:12:36.154156+0000 mgr.x (mgr.24542) 1090 : cluster [DBG] pgmap v1633: 9 pgs: 9 active+clean; 127 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.2 MiB/s wr, 3 op/s 2021-07-31T02:12:36.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:36 smithi045 conmon[39718]: cluster 2021-07-31T02:12:36.154156+0000 mgr.x (mgr.24542) 1090 : cluster 2021-07-31T02:12:36.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:36 smithi045 conmon[39718]: [DBG] pgmap v1633: 9 pgs: 9 active+clean; 127 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.2 MiB/s wr, 3 op/s 2021-07-31T02:12:36.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:36 smithi045 conmon[33655]: cluster 2021-07-31T02:12:36.154156 2021-07-31T02:12:36.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:36 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 1090 : cluster [DBG] pgmap v1633: 9 pgs: 9 active+clean; 127 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.2 MiB/s wr, 3 op/s 2021-07-31T02:12:37.378 INFO:tasks.workunit.client.0.smithi045.stdout: 442: throughput=4.99MB/sec pending data=0 2021-07-31T02:12:37.378 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-pd3doCQCK-ndp1w off=3252125 len=1404673 2021-07-31T02:12:37.385 INFO:tasks.workunit.client.0.smithi045.stdout:op 2282 completed, throughput=4.99MB/sec 2021-07-31T02:12:37.386 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-bOu5p-KhmfeyNAb off=437312 len=15404 2021-07-31T02:12:37.388 INFO:tasks.workunit.client.0.smithi045.stdout:op 2283 completed, throughput=4.99MB/sec 2021-07-31T02:12:37.388 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-56rdzlhlKWdXd1X off=3930031 len=863361 2021-07-31T02:12:37.393 INFO:tasks.workunit.client.0.smithi045.stdout:op 2284 completed, throughput=4.99MB/sec 2021-07-31T02:12:37.394 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ziiI2rAxl0Szo9k off=3728475 len=608568 2021-07-31T02:12:37.395 INFO:tasks.workunit.client.0.smithi045.stdout:op 2285 completed, throughput=5MB/sec 2021-07-31T02:12:37.395 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-fPKCp08ZZUwf08T off=9562923 len=552544 2021-07-31T02:12:37.397 INFO:tasks.workunit.client.0.smithi045.stdout:op 2286 completed, throughput=5MB/sec 2021-07-31T02:12:37.397 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-IpfE5yz-RjlJImS off=1314107 len=1571867 2021-07-31T02:12:37.399 INFO:tasks.workunit.client.0.smithi045.stdout:op 2287 completed, throughput=5MB/sec 2021-07-31T02:12:37.399 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-qMkvyf4AWHAsz5T off=2775668 len=927107 2021-07-31T02:12:37.404 INFO:tasks.workunit.client.0.smithi045.stdout:op 2288 completed, throughput=5MB/sec 2021-07-31T02:12:38.405 INFO:tasks.workunit.client.0.smithi045.stdout: 443: throughput=4.99MB/sec pending data=0 2021-07-31T02:12:38.405 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-WSVB5fCF2gbOEiL off=10922263 len=491985 2021-07-31T02:12:38.406 INFO:tasks.workunit.client.0.smithi045.stdout:op 2289 completed, throughput=4.99MB/sec 2021-07-31T02:12:38.406 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-S4wzaKTkPiNp0n2 off=12163891 len=1581280 2021-07-31T02:12:38.413 INFO:tasks.workunit.client.0.smithi045.stdout:op 2290 completed, throughput=4.99MB/sec 2021-07-31T02:12:38.413 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-fGGmzklWvrvxBVq off=5888081 len=785350 2021-07-31T02:12:38.415 INFO:tasks.workunit.client.0.smithi045.stdout:op 2291 completed, throughput=5MB/sec 2021-07-31T02:12:38.415 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-If7vMeF2hisCmpq off=9002771 len=1395355 2021-07-31T02:12:38.417 INFO:tasks.workunit.client.0.smithi045.stdout:op 2292 completed, throughput=5MB/sec 2021-07-31T02:12:38.417 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-IMaiiH4Mei_TGH3 off=1885929 len=220896 2021-07-31T02:12:38.421 INFO:tasks.workunit.client.0.smithi045.stdout:op 2293 completed, throughput=5MB/sec 2021-07-31T02:12:38.421 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-s202OmuUknJh65O off=4893612 len=1809382 2021-07-31T02:12:38.430 INFO:tasks.workunit.client.0.smithi045.stdout:op 2294 completed, throughput=5MB/sec 2021-07-31T02:12:38.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:38 smithi154 conmon[34553]: cluster 2021-07-31T02:12:38.154554+0000 mgr.x (mgr.24542) 1091 : cluster [DBG] pgmap v1634: 9 pgs: 9 active+clean; 127 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.2 MiB/s wr, 3 op/s 2021-07-31T02:12:38.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:38 smithi045 conmon[33655]: cluster 2021-07-31T02:12:38.154554+0000 mgr.x (mgr.24542) 1091 : cluster [DBG] pgmap v1634: 9 pgs: 9 active+clean; 127 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.2 MiB/s wr, 3 op/s 2021-07-31T02:12:38.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:38 smithi045 conmon[33655]: 2021-07-31T02:12:38.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:38 smithi045 conmon[39718]: cluster 2021-07-31T02:12:38.154554+0000 mgr.x (mgr.24542) 1091 : cluster 2021-07-31T02:12:38.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:38 smithi045 conmon[39718]: [DBG] pgmap v1634: 9 pgs: 9 active+clean; 127 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.2 MiB/s wr, 3 op/s 2021-07-31T02:12:39.430 INFO:tasks.workunit.client.0.smithi045.stdout: 444: throughput=4.99MB/sec pending data=0 2021-07-31T02:12:39.431 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-m0kbI_a0_KOoFNC off=10238627 len=2066796 2021-07-31T02:12:39.434 INFO:tasks.workunit.client.0.smithi045.stdout:op 2295 completed, throughput=5MB/sec 2021-07-31T02:12:39.434 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-gnytgr3xm87wp4N off=1667082 len=1114460 2021-07-31T02:12:39.441 INFO:tasks.workunit.client.0.smithi045.stdout:op 2296 completed, throughput=5MB/sec 2021-07-31T02:12:39.441 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-yVwnttVgbGbwzX0 off=441158 len=533083 2021-07-31T02:12:39.442 INFO:tasks.workunit.client.0.smithi045.stdout:op 2297 completed, throughput=5MB/sec 2021-07-31T02:12:40.442 INFO:tasks.workunit.client.0.smithi045.stdout: 445: throughput=4.99MB/sec pending data=0 2021-07-31T02:12:40.442 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-hlV0p41u-hOBMAL off=7723808 len=310894 2021-07-31T02:12:40.445 INFO:tasks.workunit.client.0.smithi045.stdout:op 2298 completed, throughput=4.99MB/sec 2021-07-31T02:12:40.446 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-MzXmxbfu3Dku_E2 off=7538306 len=1906852 2021-07-31T02:12:40.457 INFO:tasks.workunit.client.0.smithi045.stdout:op 2299 completed, throughput=4.99MB/sec 2021-07-31T02:12:40.457 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj--FgOCfILbFKqxNV off=3276727 len=30375 2021-07-31T02:12:40.458 INFO:tasks.workunit.client.0.smithi045.stdout:op 2300 completed, throughput=4.99MB/sec 2021-07-31T02:12:40.458 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-D2pRl_5xQSj9GDw off=9814524 len=1994697 2021-07-31T02:12:40.461 INFO:tasks.workunit.client.0.smithi045.stdout:op 2301 completed, throughput=5MB/sec 2021-07-31T02:12:40.462 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-FPEsCjBN9PU6epc off=14895105 len=296358 2021-07-31T02:12:40.462 INFO:tasks.workunit.client.0.smithi045.stdout:op 2302 completed, throughput=5MB/sec 2021-07-31T02:12:40.463 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-dlqlJTSCC_obRgi off=7388424 len=1066971 2021-07-31T02:12:40.468 INFO:tasks.workunit.client.0.smithi045.stdout:op 2303 completed, throughput=5MB/sec 2021-07-31T02:12:40.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:40 smithi154 conmon[34553]: cluster 2021-07-31T02:12:40.155076+0000 mgr.x (mgr.24542) 1092 : cluster [DBG] pgmap v1635: 9 pgs: 9 active+clean; 127 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 3.3 MiB/s rd, 3.1 MiB/s wr, 6 op/s 2021-07-31T02:12:40.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:40 smithi045 conmon[33655]: cluster 2021-07-31T02:12:40.155076+0000 2021-07-31T02:12:40.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:40 smithi045 conmon[33655]: mgr.x (mgr.24542) 1092 : cluster [DBG] pgmap v1635: 9 pgs: 9 active+clean; 127 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 3.3 MiB/s rd, 3.1 MiB/s wr, 6 op/s 2021-07-31T02:12:40.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:40 smithi045 conmon[39718]: cluster 2021-07-31T02:12:40.155076+0000 mgr.x (mgr.24542) 1092 : cluster 2021-07-31T02:12:40.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:40 smithi045 conmon[39718]: [DBG] pgmap v1635: 9 pgs: 9 active+clean; 127 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 3.3 MiB/s rd, 3.1 MiB/s wr, 6 op/s 2021-07-31T02:12:41.468 INFO:tasks.workunit.client.0.smithi045.stdout: 446: throughput=4.99MB/sec pending data=0 2021-07-31T02:12:41.469 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-e4nChf2Zj8RPUW3 off=951920 len=855890 2021-07-31T02:12:41.474 INFO:tasks.workunit.client.0.smithi045.stdout:op 2304 completed, throughput=4.99MB/sec 2021-07-31T02:12:41.474 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-_PgBOGpn6H-gKHG off=12159740 len=376928 2021-07-31T02:12:41.477 INFO:tasks.workunit.client.0.smithi045.stdout:op 2305 completed, throughput=4.99MB/sec 2021-07-31T02:12:41.478 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-b0smQUAY8hrYCSw off=6884423 len=197730 2021-07-31T02:12:41.479 INFO:tasks.workunit.client.0.smithi045.stdout:op 2306 completed, throughput=4.99MB/sec 2021-07-31T02:12:41.479 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-QIuzvWVEemQ6Glb off=2702681 len=1946342 2021-07-31T02:12:41.482 INFO:tasks.workunit.client.0.smithi045.stdout:op 2307 completed, throughput=5MB/sec 2021-07-31T02:12:41.483 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-HkqxcAd8uokLe0Y off=11436705 len=332559 2021-07-31T02:12:41.483 INFO:tasks.workunit.client.0.smithi045.stdout:op 2308 completed, throughput=5MB/sec 2021-07-31T02:12:41.484 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-0aBK3LfHa40vNiB off=12808059 len=1831005 2021-07-31T02:12:41.491 INFO:tasks.workunit.client.0.smithi045.stdout:op 2309 completed, throughput=5MB/sec 2021-07-31T02:12:41.940 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:12:42.216 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:12:42.216 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 27m ago 28m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:12:42.217 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 25m ago 28m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:12:42.217 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 25m ago 31m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:12:42.217 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 27m ago 33m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:12:42.217 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 27m ago 33m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:12:42.217 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 25m ago 31m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:12:42.218 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 27m ago 32m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:12:42.218 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 27m ago 28m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:12:42.218 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 25m ago 28m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:12:42.218 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 27m ago 31m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:12:42.219 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 27m ago 31m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:12:42.219 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 27m ago 30m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:12:42.219 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 27m ago 30m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:12:42.219 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 25m ago 30m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:12:42.220 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 25m ago 29m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:12:42.220 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 25m ago 29m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:12:42.220 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 25m ago 29m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:12:42.220 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 25m ago 28m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:12:42.491 INFO:tasks.workunit.client.0.smithi045.stdout: 447: throughput=4.99MB/sec pending data=0 2021-07-31T02:12:42.491 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-FaKdzxnFqBgPJJx off=7162802 len=1176749 2021-07-31T02:12:42.497 INFO:tasks.workunit.client.0.smithi045.stdout:op 2310 completed, throughput=4.99MB/sec 2021-07-31T02:12:42.497 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-B8UbM4lYT0Y8_fQ off=2450202 len=246774 2021-07-31T02:12:42.498 INFO:tasks.workunit.client.0.smithi045.stdout:op 2311 completed, throughput=4.99MB/sec 2021-07-31T02:12:42.498 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-w9GaKh1ARGsqZ8d off=1730605 len=2038275 2021-07-31T02:12:42.500 INFO:tasks.workunit.client.0.smithi045.stdout:op 2312 completed, throughput=5MB/sec 2021-07-31T02:12:42.500 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-EfZagKjx_pf3Lwy off=11997501 len=1321493 2021-07-31T02:12:42.505 INFO:tasks.workunit.client.0.smithi045.stdout:op 2313 completed, throughput=5MB/sec 2021-07-31T02:12:42.506 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-sINioyK7aps0ZKw off=9738385 len=1657447 2021-07-31T02:12:42.515 INFO:tasks.workunit.client.0.smithi045.stdout:op 2314 completed, throughput=5MB/sec 2021-07-31T02:12:42.558 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:12:42.558 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:12:42.558 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:12:42.559 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:12:42.559 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:12:42.559 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:12:42.559 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:12:42.559 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:12:42.560 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:12:42.560 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:12:42.560 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:12:42.560 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:12:42.561 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:12:42.561 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:12:42.561 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:12:42.561 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:12:42.561 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:12:43.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:43 smithi045 conmon[33655]: audit 2021-07-31T02:12:41.928017+0000 mgr.x (mgr.24542 2021-07-31T02:12:43.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:43 smithi045 conmon[33655]: ) 1093 : audit [DBG] from='client.25894 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:12:43.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:43 smithi045 conmon[33655]: cluster 2021-07-31T02:12:42.155454+0000 mgr.x (mgr.24542) 1094 : cluster [DBG] pgmap v1636: 9 pgs: 9 active+clean; 127 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2021-07-31T02:12:43.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:43 smithi045 conmon[33655]: audit 2021- 2021-07-31T02:12:43.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:43 smithi045 conmon[33655]: 07-31T02:12:42. 2021-07-31T02:12:43.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:43 smithi045 conmon[33655]: 558085+0000 mon.a (mon.0) 2021-07-31T02:12:43.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:43 smithi045 conmon[33655]: 1874 : audit [DBG] from='client.? 172.21.15.45:0/565657697' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:12:43.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:43 smithi045 conmon[39718]: audit 2021-07-31T02:12:41.928017+0000 mgr.x (mgr.24542) 1093 : audit 2021-07-31T02:12:43.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:43 smithi045 conmon[39718]: [DBG] from='client.25894 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:12:43.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:43 smithi045 conmon[39718]: cluster 2021-07-31T02:12:42.155454+0000 mgr.x ( 2021-07-31T02:12:43.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:43 smithi045 conmon[39718]: mgr.24542) 1094 : cluster [DBG] pgmap v1636: 9 pgs: 9 active+clean; 127 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2021-07-31T02:12:43.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:43 smithi045 conmon[39718]: audit 2021-07-31T02:12:43.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:43 smithi045 conmon[39718]: 2021-07-31T02:12:42. 2021-07-31T02:12:43.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:43 smithi045 conmon[39718]: 558085+0000 mon.a (mon.0 2021-07-31T02:12:43.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:43 smithi045 conmon[39718]: ) 1874 : audit [DBG] 2021-07-31T02:12:43.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:43 smithi045 conmon[39718]: from='client.? 172.21.15.45:0/565657697' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:12:43.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:43 smithi154 conmon[34553]: audit 2021-07-31T02:12:41.928017+0000 mgr.x (mgr.24542) 1093 : audit [DBG] from='client.25894 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:12:43.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:43 smithi154 conmon[34553]: cluster 2021-07-31T02:12:42.155454 2021-07-31T02:12:43.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:43 smithi154 conmon[34553]: +0000 mgr.x (mgr.24542) 1094 : cluster [DBG] pgmap v1636: 9 pgs: 9 active+clean; 127 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2021-07-31T02:12:43.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:43 smithi154 conmon[34553]: audit 2021-07-31T02:12:42.558085+0000 mon.a (mon.0) 1874 : audit [DBG] from='client.? 172.21.15.45:0/565657697' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:12:43.516 INFO:tasks.workunit.client.0.smithi045.stdout: 448: throughput=4.99MB/sec pending data=0 2021-07-31T02:12:43.516 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-1gI74SKP-nrTIpd off=4983763 len=1939426 2021-07-31T02:12:43.524 INFO:tasks.workunit.client.0.smithi045.stdout:op 2315 completed, throughput=5MB/sec 2021-07-31T02:12:43.524 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-AKTRX6L2b8jF5pE off=12998077 len=854850 2021-07-31T02:12:43.529 INFO:tasks.workunit.client.0.smithi045.stdout:op 2316 completed, throughput=5MB/sec 2021-07-31T02:12:43.530 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-uHVNuNcQI1XY19I off=13817101 len=1387022 2021-07-31T02:12:43.531 INFO:tasks.workunit.client.0.smithi045.stdout:op 2317 completed, throughput=5MB/sec 2021-07-31T02:12:43.999 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:12:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:12:43.552Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:12:44.000 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:12:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:12:43.552Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:12:44.000 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:12:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:12:43.552Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:12:44.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:44 smithi154 conmon[34553]: audit 2021-07-31T02:12:42.208735+0000 mgr.x (mgr.24542) 1095 : audit [DBG] from='client.25900 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:12:44.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:44 smithi045 conmon[39718]: audit 2021-07-31T02:12:42.208735+0000 mgr.x (mgr.24542) 1095 : audit 2021-07-31T02:12:44.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:44 smithi045 conmon[39718]: [DBG] from='client.25900 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:12:44.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:44 smithi045 conmon[33655]: audit 2021-07-31T02:12:42.208735+0000 2021-07-31T02:12:44.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:44 smithi045 conmon[33655]: mgr.x (mgr.24542) 1095 : audit [DBG] from='client.25900 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:12:44.531 INFO:tasks.workunit.client.0.smithi045.stdout: 449: throughput=4.99MB/sec pending data=0 2021-07-31T02:12:44.532 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-8uK687LXSjA7mpT off=6979513 len=128005 2021-07-31T02:12:44.532 INFO:tasks.workunit.client.0.smithi045.stdout:op 2318 completed, throughput=4.99MB/sec 2021-07-31T02:12:44.532 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-vmq_jTSWUqXbRO0 off=13886598 len=1713558 2021-07-31T02:12:44.534 INFO:tasks.workunit.client.0.smithi045.stdout:op 2319 completed, throughput=4.99MB/sec 2021-07-31T02:12:44.534 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-z38nsdwZfT6Zwzd off=21606646 len=1437331 2021-07-31T02:12:44.541 INFO:tasks.workunit.client.0.smithi045.stdout:op 2320 completed, throughput=5MB/sec 2021-07-31T02:12:44.541 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-DNKz6CHc9prdkb1 off=7988383 len=605141 2021-07-31T02:12:44.546 INFO:tasks.workunit.client.0.smithi045.stdout:op 2321 completed, throughput=5MB/sec 2021-07-31T02:12:44.547 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj--tRlPKiESfXsg_W off=1568792 len=671727 2021-07-31T02:12:44.550 INFO:tasks.workunit.client.0.smithi045.stdout:op 2322 completed, throughput=5MB/sec 2021-07-31T02:12:44.550 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-A56qERtc66L5V-F off=1242219 len=1127328 2021-07-31T02:12:44.556 INFO:tasks.workunit.client.0.smithi045.stdout:op 2323 completed, throughput=5MB/sec 2021-07-31T02:12:45.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:45 smithi045 conmon[33655]: cluster 2021-07-31T02:12:44.155865+0000 mgr.x 2021-07-31T02:12:45.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:45 smithi045 conmon[33655]: (mgr.24542) 1096 : cluster [DBG] pgmap v1637: 9 pgs: 9 active+clean; 127 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 3.1 MiB/s rd, 3.2 MiB/s wr, 6 op/s 2021-07-31T02:12:45.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:45 smithi045 conmon[39718]: cluster 2021-07-31T02:12:44.155865+0000 mgr.x (mgr.24542) 1096 : 2021-07-31T02:12:45.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:45 smithi045 conmon[39718]: cluster [DBG] pgmap v1637: 9 pgs: 9 active+clean; 127 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 3.1 MiB/s rd, 3.2 MiB/s wr, 6 op/s 2021-07-31T02:12:45.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:45 smithi154 conmon[34553]: cluster 2021-07-31T02:12:44.155865+0000 mgr.x (mgr.24542) 1096 : cluster [DBG] pgmap v1637: 9 pgs: 9 active+clean; 127 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 3.1 MiB/s rd, 3.2 MiB/s wr, 6 op/s 2021-07-31T02:12:45.557 INFO:tasks.workunit.client.0.smithi045.stdout: 450: throughput=4.99MB/sec pending data=0 2021-07-31T02:12:45.557 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-0sUeajHUWcy5RG3 off=2147243 len=419175 2021-07-31T02:12:45.558 INFO:tasks.workunit.client.0.smithi045.stdout:op 2324 completed, throughput=4.99MB/sec 2021-07-31T02:12:45.558 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-vPJqcVCBIeRHJe3 off=828637 len=472279 2021-07-31T02:12:45.562 INFO:tasks.workunit.client.0.smithi045.stdout:op 2325 completed, throughput=4.99MB/sec 2021-07-31T02:12:45.563 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-2R-FBxaEWFFLLii off=5988282 len=1715390 2021-07-31T02:12:45.571 INFO:tasks.workunit.client.0.smithi045.stdout:op 2326 completed, throughput=5MB/sec 2021-07-31T02:12:45.571 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj--V1XeHQtPJi-f-6 off=7233272 len=378180 2021-07-31T02:12:45.574 INFO:tasks.workunit.client.0.smithi045.stdout:op 2327 completed, throughput=5MB/sec 2021-07-31T02:12:45.575 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-W7ECCPnMNBFRhYA off=4280718 len=740981 2021-07-31T02:12:45.579 INFO:tasks.workunit.client.0.smithi045.stdout:op 2328 completed, throughput=5MB/sec 2021-07-31T02:12:45.579 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-J_qGi_XI7yCcPDu off=17871486 len=67251 2021-07-31T02:12:45.580 INFO:tasks.workunit.client.0.smithi045.stdout:op 2329 completed, throughput=5MB/sec 2021-07-31T02:12:45.580 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-TuRyDdI_6g3agTm off=794669 len=826341 2021-07-31T02:12:45.581 INFO:tasks.workunit.client.0.smithi045.stdout:op 2330 completed, throughput=5MB/sec 2021-07-31T02:12:45.582 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-UfNa_rdxhEtZBKA off=3843024 len=551910 2021-07-31T02:12:45.585 INFO:tasks.workunit.client.0.smithi045.stdout:op 2331 completed, throughput=5MB/sec 2021-07-31T02:12:46.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:46 smithi154 conmon[34553]: cluster 2021-07-31T02:12:46.156313+0000 mgr.x (mgr.24542) 1097 : cluster [DBG] pgmap v1638: 9 pgs: 9 active+clean; 127 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2021-07-31T02:12:46.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:46 smithi045 conmon[39718]: cluster 2021-07-31T02:12:46.156313+0000 2021-07-31T02:12:46.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:46 smithi045 conmon[39718]: mgr.x (mgr.24542) 1097 : cluster [DBG] pgmap v1638: 9 pgs: 9 active+clean; 127 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2021-07-31T02:12:46.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:46 smithi045 conmon[33655]: cluster 2021-07-31T02:12:46.156313+0000 mgr.x (mgr.24542) 1097 : cluster [DBG] pgmap v1638: 9 pgs: 9 active+clean; 127 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2021-07-31T02:12:46.585 INFO:tasks.workunit.client.0.smithi045.stdout: 451: throughput=4.99MB/sec pending data=0 2021-07-31T02:12:46.586 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-uUo2D-drHKw-Hfc off=2944366 len=1534567 2021-07-31T02:12:46.588 INFO:tasks.workunit.client.0.smithi045.stdout:op 2332 completed, throughput=4.99MB/sec 2021-07-31T02:12:46.589 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-i6-7i3s39Fd2SuL off=1187500 len=1969038 2021-07-31T02:12:46.592 INFO:tasks.workunit.client.0.smithi045.stdout:op 2333 completed, throughput=5MB/sec 2021-07-31T02:12:46.592 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ayMMO8pYMwopXty off=2103420 len=996948 2021-07-31T02:12:46.594 INFO:tasks.workunit.client.0.smithi045.stdout:op 2334 completed, throughput=5MB/sec 2021-07-31T02:12:46.595 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-kYwGHDnu_93hdGF off=5940447 len=1161752 2021-07-31T02:12:46.597 INFO:tasks.workunit.client.0.smithi045.stdout:op 2335 completed, throughput=5MB/sec 2021-07-31T02:12:47.597 INFO:tasks.workunit.client.0.smithi045.stdout: 452: throughput=4.99MB/sec pending data=0 2021-07-31T02:12:47.598 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-5Igw2PZsf6eNmZg off=3626138 len=892849 2021-07-31T02:12:47.599 INFO:tasks.workunit.client.0.smithi045.stdout:op 2336 completed, throughput=4.99MB/sec 2021-07-31T02:12:47.599 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-3Z1qf5hAip8SNVK off=1178300 len=279620 2021-07-31T02:12:47.600 INFO:tasks.workunit.client.0.smithi045.stdout:op 2337 completed, throughput=4.99MB/sec 2021-07-31T02:12:47.600 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Jlwqe4rpaWVvlb- off=13939208 len=1071454 2021-07-31T02:12:47.601 INFO:tasks.workunit.client.0.smithi045.stdout:op 2338 completed, throughput=5MB/sec 2021-07-31T02:12:47.601 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ZGiCtV2AuJtcbs6 off=844349 len=1412045 2021-07-31T02:12:47.610 INFO:tasks.workunit.client.0.smithi045.stdout:op 2339 completed, throughput=5MB/sec 2021-07-31T02:12:47.610 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-exXgX1UJ8YuOLJb off=8146978 len=792673 2021-07-31T02:12:47.611 INFO:tasks.workunit.client.0.smithi045.stdout:op 2340 completed, throughput=5MB/sec 2021-07-31T02:12:48.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:48 smithi154 conmon[34553]: cluster 2021-07-31T02:12:48.156746+0000 mgr.x (mgr.24542) 1098 : cluster [DBG] pgmap v1639: 9 pgs: 9 active+clean; 127 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2021-07-31T02:12:48.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:48 smithi045 conmon[39718]: cluster 2021-07-31T02:12: 2021-07-31T02:12:48.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:48 smithi045 conmon[39718]: 48.156746+0000 mgr.x (mgr.24542) 1098 : cluster [DBG] pgmap v1639: 9 pgs: 9 active+clean; 127 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2021-07-31T02:12:48.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:48 smithi045 conmon[33655]: cluster 2021-07-31T02:12:48.156746+0000 mgr.x (mgr.24542) 1098 : cluster [DBG] pgmap v1639: 9 pgs: 9 active+clean; 127 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2021-07-31T02:12:48.612 INFO:tasks.workunit.client.0.smithi045.stdout: 453: throughput=4.99MB/sec pending data=0 2021-07-31T02:12:48.612 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-hJ1EaVjzmpQa_oL off=1228202 len=58570 2021-07-31T02:12:48.615 INFO:tasks.workunit.client.0.smithi045.stdout:op 2341 completed, throughput=4.99MB/sec 2021-07-31T02:12:48.615 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-3lQNORn8YTV1xHx off=644134 len=1644993 2021-07-31T02:12:48.622 INFO:tasks.workunit.client.0.smithi045.stdout:op 2342 completed, throughput=4.99MB/sec 2021-07-31T02:12:48.623 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-5SyOXylr4mkmX2N off=103473 len=1504233 2021-07-31T02:12:48.624 INFO:tasks.workunit.client.0.smithi045.stdout:op 2343 completed, throughput=5MB/sec 2021-07-31T02:12:48.625 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-nhPnuilQIZoJPZx off=5125050 len=679237 2021-07-31T02:12:48.629 INFO:tasks.workunit.client.0.smithi045.stdout:op 2344 completed, throughput=5MB/sec 2021-07-31T02:12:48.629 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-K7JAHGzmbpiTlVl off=1213374 len=363282 2021-07-31T02:12:48.630 INFO:tasks.workunit.client.0.smithi045.stdout:op 2345 completed, throughput=5MB/sec 2021-07-31T02:12:48.630 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-gIAfasB0AYcx8iv off=140647 len=561280 2021-07-31T02:12:48.631 INFO:tasks.workunit.client.0.smithi045.stdout:op 2346 completed, throughput=5MB/sec 2021-07-31T02:12:48.631 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-u1wsS2m6G0vSlD0 off=21059525 len=1048392 2021-07-31T02:12:48.638 INFO:tasks.workunit.client.0.smithi045.stdout:op 2347 completed, throughput=5MB/sec 2021-07-31T02:12:49.638 INFO:tasks.workunit.client.0.smithi045.stdout: 454: throughput=4.99MB/sec pending data=0 2021-07-31T02:12:49.639 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-zAkAgptHEwdNo-c off=4676850 len=544122 2021-07-31T02:12:49.643 INFO:tasks.workunit.client.0.smithi045.stdout:op 2348 completed, throughput=4.99MB/sec 2021-07-31T02:12:49.643 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-XYYI2rvzVaxpfwA off=12310244 len=1602747 2021-07-31T02:12:49.646 INFO:tasks.workunit.client.0.smithi045.stdout:op 2349 completed, throughput=4.99MB/sec 2021-07-31T02:12:49.646 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-R6NrF8m_sfhMwx9 off=12996644 len=1126893 2021-07-31T02:12:49.648 INFO:tasks.workunit.client.0.smithi045.stdout:op 2350 completed, throughput=5MB/sec 2021-07-31T02:12:49.648 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-kY3Cbo76Xva1C1y off=3321003 len=2067384 2021-07-31T02:12:49.661 INFO:tasks.workunit.client.0.smithi045.stdout:op 2351 completed, throughput=5MB/sec 2021-07-31T02:12:50.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:50 smithi154 conmon[34553]: cluster 2021-07-31T02:12:50.157245+0000 mgr.x (mgr.24542) 1099 : cluster [DBG] pgmap v1640: 9 pgs: 9 active+clean; 127 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 3.3 MiB/s rd, 3.0 MiB/s wr, 7 op/s 2021-07-31T02:12:50.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:50 smithi045 conmon[39718]: cluster 2021-07-31T02:12:50.157245+0000 2021-07-31T02:12:50.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:50 smithi045 conmon[39718]: mgr.x (mgr.24542) 1099 : cluster [DBG] pgmap v1640: 9 pgs: 9 active+clean; 127 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 3.3 MiB/s rd, 3.0 MiB/s wr, 7 op/s 2021-07-31T02:12:50.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:50 smithi045 conmon[33655]: cluster 2021-07-31T02:12:50.157245+0000 mgr.x (mgr.24542) 1099 : cluster 2021-07-31T02:12:50.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:50 smithi045 conmon[33655]: [DBG] pgmap v1640: 9 pgs: 9 active+clean; 127 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 3.3 MiB/s rd, 3.0 MiB/s wr, 7 op/s 2021-07-31T02:12:50.663 INFO:tasks.workunit.client.0.smithi045.stdout: 455: throughput=4.99MB/sec pending data=0 2021-07-31T02:12:50.664 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-H-UoUDmMPaoQ56F off=5323929 len=1323134 2021-07-31T02:12:50.664 INFO:tasks.workunit.client.0.smithi045.stdout:op 2352 completed, throughput=4.99MB/sec 2021-07-31T02:12:50.665 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-73QfN5DjRyH8Rmk off=7378849 len=1780277 2021-07-31T02:12:50.666 INFO:tasks.workunit.client.0.smithi045.stdout:op 2353 completed, throughput=5MB/sec 2021-07-31T02:12:50.667 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-W2shfq9tloYx5A_ off=2110829 len=1697906 2021-07-31T02:12:50.670 INFO:tasks.workunit.client.0.smithi045.stdout:op 2354 completed, throughput=5MB/sec 2021-07-31T02:12:50.670 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-8j1T-kyS-OL0lOG off=4068220 len=843582 2021-07-31T02:12:50.671 INFO:tasks.workunit.client.0.smithi045.stdout:op 2355 completed, throughput=5MB/sec 2021-07-31T02:12:51.672 INFO:tasks.workunit.client.0.smithi045.stdout: 456: throughput=4.99MB/sec pending data=0 2021-07-31T02:12:51.672 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-OewACFv1ZTWhC7u off=16191336 len=745781 2021-07-31T02:12:51.674 INFO:tasks.workunit.client.0.smithi045.stdout:op 2356 completed, throughput=4.99MB/sec 2021-07-31T02:12:51.674 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-GdThmyeBEcCV02X off=2279731 len=1546703 2021-07-31T02:12:51.684 INFO:tasks.workunit.client.0.smithi045.stdout:op 2357 completed, throughput=5MB/sec 2021-07-31T02:12:51.685 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-F8jwBdYVefnT3yf off=1456689 len=1237836 2021-07-31T02:12:51.685 INFO:tasks.workunit.client.0.smithi045.stdout:op 2358 completed, throughput=5MB/sec 2021-07-31T02:12:51.685 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-MOkk9LOnYoLm4ov off=1254780 len=28984 2021-07-31T02:12:51.686 INFO:tasks.workunit.client.0.smithi045.stdout:op 2359 completed, throughput=5MB/sec 2021-07-31T02:12:51.686 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Dem06xNWr0rFSoK off=0 len=1362938 2021-07-31T02:12:51.694 INFO:tasks.workunit.client.0.smithi045.stdout:op 2360 completed, throughput=5MB/sec 2021-07-31T02:12:52.694 INFO:tasks.workunit.client.0.smithi045.stdout: 457: throughput=4.99MB/sec pending data=0 2021-07-31T02:12:52.694 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-RFY5lZv68_rAKUR off=1525486 len=857525 2021-07-31T02:12:52.699 INFO:tasks.workunit.client.0.smithi045.stdout:op 2361 completed, throughput=4.99MB/sec 2021-07-31T02:12:52.700 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-MzXmxbfu3Dku_E2 off=343384 len=1920869 2021-07-31T02:12:52.711 INFO:tasks.workunit.client.0.smithi045.stdout:op 2362 completed, throughput=5MB/sec 2021-07-31T02:12:52.712 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-W-W7l-FfCMwE_5o off=3753733 len=2092338 2021-07-31T02:12:52.715 INFO:tasks.workunit.client.0.smithi045.stdout:op 2363 completed, throughput=5MB/sec 2021-07-31T02:12:52.715 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Rhha6zcLX21pEzn off=14637319 len=147380 2021-07-31T02:12:52.718 INFO:tasks.workunit.client.0.smithi045.stdout:op 2364 completed, throughput=5MB/sec 2021-07-31T02:12:53.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:53 smithi154 conmon[34553]: cluster 2021-07-31T02:12:52.157854+0000 2021-07-31T02:12:53.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:53 smithi154 conmon[34553]: mgr.x (mgr.24542) 1100 : cluster [DBG] pgmap v1641: 9 pgs: 9 active+clean; 127 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2021-07-31T02:12:53.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:53 smithi045 conmon[39718]: cluster 2021-07-31T02:12:52.157854+0000 mgr.x (mgr. 2021-07-31T02:12:53.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:53 smithi045 conmon[39718]: 24542) 1100 : cluster [DBG] pgmap v1641: 9 pgs: 9 active+clean; 127 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2021-07-31T02:12:53.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:52 smithi045 conmon[33655]: cluster 2021-07-31T02:12:52.157854+0000 mgr.x (mgr.24542) 1100 : cluster 2021-07-31T02:12:53.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:52 smithi045 conmon[33655]: [DBG] pgmap v1641: 9 pgs: 9 active+clean; 127 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2021-07-31T02:12:53.718 INFO:tasks.workunit.client.0.smithi045.stdout: 458: throughput=4.99MB/sec pending data=0 2021-07-31T02:12:53.718 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-62fxJ1T7LMYwU-8 off=3078531 len=1647775 2021-07-31T02:12:53.720 INFO:tasks.workunit.client.0.smithi045.stdout:op 2365 completed, throughput=4.99MB/sec 2021-07-31T02:12:53.720 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-K-Xr66VDnAPud3x off=5020187 len=1405816 2021-07-31T02:12:53.728 INFO:tasks.workunit.client.0.smithi045.stdout:op 2366 completed, throughput=5MB/sec 2021-07-31T02:12:53.728 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-cgXjT2n2lC3VZZt off=8552607 len=1973719 2021-07-31T02:12:53.737 INFO:tasks.workunit.client.0.smithi045.stdout:op 2367 completed, throughput=5MB/sec 2021-07-31T02:12:53.737 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-0NYLwnGrCRTgrGs off=3423163 len=1690845 2021-07-31T02:12:53.740 INFO:tasks.workunit.client.0.smithi045.stdout:op 2368 completed, throughput=5MB/sec 2021-07-31T02:12:54.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:12:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:12:53.552Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:12:54.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:12:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:12:53.552Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:12:54.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:12:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:12:53.552Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:12:54.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:54 smithi154 conmon[34553]: cluster 2021-07-31T02:12:54.158358+0000 mgr.x (mgr.24542) 1101 : cluster [DBG] pgmap v1642: 9 pgs: 9 active+clean; 127 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 3.3 MiB/s rd, 3.1 MiB/s wr, 6 op/s 2021-07-31T02:12:54.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:54 smithi045 conmon[39718]: cluster 2021-07-31T02:12:54.158358+0000 mgr.x (mgr. 2021-07-31T02:12:54.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:54 smithi045 conmon[39718]: 24542) 1101 : cluster [DBG] pgmap v1642: 9 pgs: 9 active+clean; 127 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 3.3 MiB/s rd, 3.1 MiB/s wr, 6 op/s 2021-07-31T02:12:54.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:54 smithi045 conmon[33655]: cluster 2021-07-31T02:12:54.158358+0000 mgr.x (mgr.24542) 1101 : cluster [DBG] 2021-07-31T02:12:54.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:54 smithi045 conmon[33655]: pgmap v1642: 9 pgs: 9 active+clean; 127 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 3.3 MiB/s rd, 3.1 MiB/s wr, 6 op/s 2021-07-31T02:12:54.740 INFO:tasks.workunit.client.0.smithi045.stdout: 459: throughput=4.99MB/sec pending data=0 2021-07-31T02:12:54.741 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-w95TzPMp9haFCd2 off=2854163 len=750226 2021-07-31T02:12:54.746 INFO:tasks.workunit.client.0.smithi045.stdout:op 2369 completed, throughput=4.99MB/sec 2021-07-31T02:12:54.746 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-wuPmnNa3RH2R6wq off=3236634 len=132005 2021-07-31T02:12:54.749 INFO:tasks.workunit.client.0.smithi045.stdout:op 2370 completed, throughput=4.99MB/sec 2021-07-31T02:12:54.749 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-KzmvKLylvMzvyZE off=13771037 len=905788 2021-07-31T02:12:54.754 INFO:tasks.workunit.client.0.smithi045.stdout:op 2371 completed, throughput=5MB/sec 2021-07-31T02:12:54.755 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ZVnZ1rJ4UufOcXQ off=21146350 len=1770624 2021-07-31T02:12:54.758 INFO:tasks.workunit.client.0.smithi045.stdout:op 2372 completed, throughput=5MB/sec 2021-07-31T02:12:54.758 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-fgz52z61ZAqkicr off=5344600 len=2009031 2021-07-31T02:12:54.762 INFO:tasks.workunit.client.0.smithi045.stdout:op 2373 completed, throughput=5MB/sec 2021-07-31T02:12:55.762 INFO:tasks.workunit.client.0.smithi045.stdout: 460: throughput=4.99MB/sec pending data=0 2021-07-31T02:12:55.762 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-XtRJ2SCpa_0rnj2 off=2658080 len=25129 2021-07-31T02:12:55.765 INFO:tasks.workunit.client.0.smithi045.stdout:op 2374 completed, throughput=4.99MB/sec 2021-07-31T02:12:55.765 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-w8B3ojABztIQfSE off=1197642 len=65624 2021-07-31T02:12:55.766 INFO:tasks.workunit.client.0.smithi045.stdout:op 2375 completed, throughput=4.99MB/sec 2021-07-31T02:12:55.766 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-wGG6Mm27Lk1vkrJ off=2365884 len=243232 2021-07-31T02:12:55.767 INFO:tasks.workunit.client.0.smithi045.stdout:op 2376 completed, throughput=4.99MB/sec 2021-07-31T02:12:55.767 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-vhniBh4lpASmCO6 off=18770398 len=763517 2021-07-31T02:12:55.768 INFO:tasks.workunit.client.0.smithi045.stdout:op 2377 completed, throughput=4.99MB/sec 2021-07-31T02:12:55.769 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-wgrEg6h8Yl-KVdq off=5830366 len=1084551 2021-07-31T02:12:55.770 INFO:tasks.workunit.client.0.smithi045.stdout:op 2378 completed, throughput=5MB/sec 2021-07-31T02:12:55.771 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-0wqyuuHKukjfAiv off=5369751 len=1674143 2021-07-31T02:12:55.772 INFO:tasks.workunit.client.0.smithi045.stdout:op 2379 completed, throughput=5MB/sec 2021-07-31T02:12:56.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:56 smithi154 conmon[34553]: cluster 2021-07-31T02:12:56.158799+0000 mgr.x (mgr.24542) 1102 : cluster [DBG] pgmap v1643: 9 pgs: 9 active+clean; 127 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 2.4 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2021-07-31T02:12:56.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:56 smithi045 conmon[33655]: cluster 2021-07-31T02:12:56.158799+0000 mgr.x (mgr.24542) 1102 : cluster 2021-07-31T02:12:56.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:56 smithi045 conmon[33655]: [DBG] pgmap v1643: 9 pgs: 9 active+clean; 127 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 2.4 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2021-07-31T02:12:56.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:56 smithi045 conmon[39718]: cluster 2021-07-31T02:12:56.158799+0000 mgr.x (mgr.24542) 2021-07-31T02:12:56.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:56 smithi045 conmon[39718]: 1102 : cluster [DBG] pgmap v1643: 9 pgs: 9 active+clean; 127 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 2.4 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2021-07-31T02:12:56.773 INFO:tasks.workunit.client.0.smithi045.stdout: 461: throughput=4.99MB/sec pending data=0 2021-07-31T02:12:56.773 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-HfuMDIiRExZqVYz off=136541 len=606582 2021-07-31T02:12:56.778 INFO:tasks.workunit.client.0.smithi045.stdout:op 2380 completed, throughput=4.99MB/sec 2021-07-31T02:12:56.778 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-zut-Xgb7mAypJJy off=3678554 len=1379754 2021-07-31T02:12:56.780 INFO:tasks.workunit.client.0.smithi045.stdout:op 2381 completed, throughput=4.99MB/sec 2021-07-31T02:12:56.780 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-a5Fk4OVYygXI0rN off=10875295 len=1621230 2021-07-31T02:12:56.787 INFO:tasks.workunit.client.0.smithi045.stdout:op 2382 completed, throughput=5MB/sec 2021-07-31T02:12:56.788 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-pu9EtZL41P9GrB_ off=4236444 len=353209 2021-07-31T02:12:56.788 INFO:tasks.workunit.client.0.smithi045.stdout:op 2383 completed, throughput=5MB/sec 2021-07-31T02:12:56.789 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-KF47JS-VeJK30eq off=976548 len=1731372 2021-07-31T02:12:56.798 INFO:tasks.workunit.client.0.smithi045.stdout:op 2384 completed, throughput=5MB/sec 2021-07-31T02:12:57.799 INFO:tasks.workunit.client.0.smithi045.stdout: 462: throughput=4.99MB/sec pending data=0 2021-07-31T02:12:57.799 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-X02JJSBhPlM5Uw4 off=4533053 len=183136 2021-07-31T02:12:57.801 INFO:tasks.workunit.client.0.smithi045.stdout:op 2385 completed, throughput=4.99MB/sec 2021-07-31T02:12:57.802 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ElrXV-uHpnvNxZQ off=1242957 len=953219 2021-07-31T02:12:57.807 INFO:tasks.workunit.client.0.smithi045.stdout:op 2386 completed, throughput=4.99MB/sec 2021-07-31T02:12:57.807 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-BjLANYqg6buF4Ax off=11646461 len=120530 2021-07-31T02:12:57.809 INFO:tasks.workunit.client.0.smithi045.stdout:op 2387 completed, throughput=4.99MB/sec 2021-07-31T02:12:57.810 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj--Ds4HgXl2EN9QRo off=11492578 len=1052898 2021-07-31T02:12:57.815 INFO:tasks.workunit.client.0.smithi045.stdout:op 2388 completed, throughput=4.99MB/sec 2021-07-31T02:12:57.815 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-CGETB-uWc7Qj9w3 off=8114906 len=499386 2021-07-31T02:12:57.818 INFO:tasks.workunit.client.0.smithi045.stdout:op 2389 completed, throughput=5MB/sec 2021-07-31T02:12:57.818 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-mOH4IMk0DUTgg9V off=6233913 len=1667103 2021-07-31T02:12:57.821 INFO:tasks.workunit.client.0.smithi045.stdout:op 2390 completed, throughput=5MB/sec 2021-07-31T02:12:57.821 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-0Ekrsvp9RbBRvoI off=12383769 len=1666799 2021-07-31T02:12:57.823 INFO:tasks.workunit.client.0.smithi045.stdout:op 2391 completed, throughput=5MB/sec 2021-07-31T02:12:58.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:12:58 smithi154 conmon[34553]: cluster 2021-07-31T02:12:58.159220+0000 mgr.x (mgr.24542) 1103 : cluster [DBG] pgmap v1644: 9 pgs: 9 active+clean; 127 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 2.4 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2021-07-31T02:12:58.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:58 smithi045 conmon[39718]: cluster 2021-07-31T02:12:58 2021-07-31T02:12:58.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:12:58 smithi045 conmon[39718]: .159220+0000 mgr.x (mgr.24542) 1103 : cluster [DBG] pgmap v1644: 9 pgs: 9 active+clean; 127 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 2.4 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2021-07-31T02:12:58.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:58 smithi045 conmon[33655]: cluster 2021-07-31T02:12:58.159220+0000 mgr.x ( 2021-07-31T02:12:58.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:12:58 smithi045 conmon[33655]: mgr.24542) 1103 : cluster [DBG] pgmap v1644: 9 pgs: 9 active+clean; 127 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 2.4 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2021-07-31T02:12:58.824 INFO:tasks.workunit.client.0.smithi045.stdout: 463: throughput=4.99MB/sec pending data=0 2021-07-31T02:12:58.825 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-e08hQQg50CECh1r off=1512027 len=1547310 2021-07-31T02:12:58.831 INFO:tasks.workunit.client.0.smithi045.stdout:op 2392 completed, throughput=4.99MB/sec 2021-07-31T02:12:58.831 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-MYkpkJhoBgasXsI off=7225896 len=726417 2021-07-31T02:12:58.832 INFO:tasks.workunit.client.0.smithi045.stdout:op 2393 completed, throughput=5MB/sec 2021-07-31T02:12:58.833 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-etjmhQsllpTYvc_ off=871183 len=1569251 2021-07-31T02:12:58.834 INFO:tasks.workunit.client.0.smithi045.stdout:op 2394 completed, throughput=5MB/sec 2021-07-31T02:12:58.834 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-6KiyueChoiyMZ8K off=6846208 len=247211 2021-07-31T02:12:58.837 INFO:tasks.workunit.client.0.smithi045.stdout:op 2395 completed, throughput=5MB/sec 2021-07-31T02:12:59.838 INFO:tasks.workunit.client.0.smithi045.stdout: 464: throughput=4.99MB/sec pending data=0 2021-07-31T02:12:59.838 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj--psAEIktIQXYKHC off=141574 len=1505235 2021-07-31T02:12:59.846 INFO:tasks.workunit.client.0.smithi045.stdout:op 2396 completed, throughput=4.99MB/sec 2021-07-31T02:12:59.846 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-stIU74RkyD8up2y off=8332464 len=897864 2021-07-31T02:12:59.848 INFO:tasks.workunit.client.0.smithi045.stdout:op 2397 completed, throughput=4.99MB/sec 2021-07-31T02:12:59.848 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-YHHsI724ebdcWnW off=5465309 len=621847 2021-07-31T02:12:59.849 INFO:tasks.workunit.client.0.smithi045.stdout:op 2398 completed, throughput=5MB/sec 2021-07-31T02:12:59.850 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-BdrCjivzz0cTNTB off=0 len=1370720 2021-07-31T02:12:59.852 INFO:tasks.workunit.client.0.smithi045.stdout:op 2399 completed, throughput=5MB/sec 2021-07-31T02:12:59.853 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-d3JZMv5vRlzeBOZ off=6481651 len=1601473 2021-07-31T02:12:59.860 INFO:tasks.workunit.client.0.smithi045.stdout:op 2400 completed, throughput=5MB/sec 2021-07-31T02:13:00.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:00 smithi154 conmon[34553]: cluster 2021-07-31T02:13:00.159862+0000 mgr.x (mgr.24542) 1104 : cluster [DBG] pgmap v1645: 9 pgs: 9 active+clean; 127 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 3.6 MiB/s rd, 2.8 MiB/s wr, 6 op/s 2021-07-31T02:13:00.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:00 smithi045 conmon[33655]: cluster 2021-07-31T02:13:00. 2021-07-31T02:13:00.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:00 smithi045 conmon[33655]: 159862+0000 mgr.x (mgr.24542) 1104 : cluster [DBG] pgmap v1645: 9 pgs: 9 active+clean; 127 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 3.6 MiB/s rd, 2.8 MiB/s wr, 6 op/s 2021-07-31T02:13:00.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:00 smithi045 conmon[39718]: cluster 2021-07-31T02:13:00.159862+0000 mgr.x (mgr 2021-07-31T02:13:00.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:00 smithi045 conmon[39718]: .24542) 1104 : cluster [DBG] pgmap v1645: 9 pgs: 9 active+clean; 127 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 3.6 MiB/s rd, 2.8 MiB/s wr, 6 op/s 2021-07-31T02:13:00.860 INFO:tasks.workunit.client.0.smithi045.stdout: 465: throughput=4.99MB/sec pending data=0 2021-07-31T02:13:00.861 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-dScZ19rnZA2G1Lf off=13332419 len=1516435 2021-07-31T02:13:00.862 INFO:tasks.workunit.client.0.smithi045.stdout:op 2401 completed, throughput=4.99MB/sec 2021-07-31T02:13:00.862 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-2KGw0RcT5G_63Pf off=1605693 len=692431 2021-07-31T02:13:00.863 INFO:tasks.workunit.client.0.smithi045.stdout:op 2402 completed, throughput=5MB/sec 2021-07-31T02:13:00.863 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-YyZHeD2aakPfOjk off=9977104 len=1280062 2021-07-31T02:13:00.869 INFO:tasks.workunit.client.0.smithi045.stdout:op 2403 completed, throughput=5MB/sec 2021-07-31T02:13:00.869 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-lF1mk7AQaXZ5JSw off=14039451 len=380923 2021-07-31T02:13:00.872 INFO:tasks.workunit.client.0.smithi045.stdout:op 2404 completed, throughput=5MB/sec 2021-07-31T02:13:00.872 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-3j-PuLsmygG9afz off=6694192 len=1049989 2021-07-31T02:13:00.874 INFO:tasks.workunit.client.0.smithi045.stdout:op 2405 completed, throughput=5MB/sec 2021-07-31T02:13:01.874 INFO:tasks.workunit.client.0.smithi045.stdout: 466: throughput=4.99MB/sec pending data=0 2021-07-31T02:13:01.875 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-JgnaeSGFdl-e6Bl off=8689801 len=1736993 2021-07-31T02:13:01.883 INFO:tasks.workunit.client.0.smithi045.stdout:op 2406 completed, throughput=4.99MB/sec 2021-07-31T02:13:01.883 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-arjrQlVVUF2Nwzg off=5812196 len=1714432 2021-07-31T02:13:01.886 INFO:tasks.workunit.client.0.smithi045.stdout:op 2407 completed, throughput=5MB/sec 2021-07-31T02:13:01.886 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-G0od48Uc9oLXZ0A off=716978 len=1423877 2021-07-31T02:13:01.889 INFO:tasks.workunit.client.0.smithi045.stdout:op 2408 completed, throughput=5MB/sec 2021-07-31T02:13:01.889 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-gAKkMrspPHKp4E- off=1681423 len=1834791 2021-07-31T02:13:01.899 INFO:tasks.workunit.client.0.smithi045.stdout:op 2409 completed, throughput=5MB/sec 2021-07-31T02:13:02.899 INFO:tasks.workunit.client.0.smithi045.stdout: 467: throughput=4.99MB/sec pending data=0 2021-07-31T02:13:02.899 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-06AsdFUYY60nNbm off=351217 len=495274 2021-07-31T02:13:02.900 INFO:tasks.workunit.client.0.smithi045.stdout:op 2410 completed, throughput=4.99MB/sec 2021-07-31T02:13:02.900 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-DzPezMr1bPOc92t off=13778355 len=1525175 2021-07-31T02:13:02.907 INFO:tasks.workunit.client.0.smithi045.stdout:op 2411 completed, throughput=5MB/sec 2021-07-31T02:13:02.907 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-xSTNxB0-XmvKsjP off=4944172 len=322855 2021-07-31T02:13:02.908 INFO:tasks.workunit.client.0.smithi045.stdout:op 2412 completed, throughput=5MB/sec 2021-07-31T02:13:02.908 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-VThbvV6ApCbIfYp off=173864 len=1093961 2021-07-31T02:13:02.910 INFO:tasks.workunit.client.0.smithi045.stdout:op 2413 completed, throughput=5MB/sec 2021-07-31T02:13:02.911 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-9B-Qb7v5eYmzR6w off=8067093 len=667691 2021-07-31T02:13:02.912 INFO:tasks.workunit.client.0.smithi045.stdout:op 2414 completed, throughput=5MB/sec 2021-07-31T02:13:03.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:03 smithi045 conmon[39718]: cluster 2021-07-31T02:13:02.160330+0000 mgr.x (mgr.24542) 1105 : cluster [DBG] pgmap v1646: 9 pgs: 9 active+clean; 127 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 2.4 MiB/s rd, 1.8 MiB/s wr, 3 op/s 2021-07-31T02:13:03.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:03 smithi045 conmon[33655]: cluster 2021-07-31T02:13:02.160330+0000 mgr.x (mgr.24542) 1105 : cluster [DBG] 2021-07-31T02:13:03.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:03 smithi045 conmon[33655]: pgmap v1646: 9 pgs: 9 active+clean; 127 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 2.4 MiB/s rd, 1.8 MiB/s wr, 3 op/s 2021-07-31T02:13:03.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:03 smithi154 conmon[34553]: cluster 2021-07-31T02:13:02.160330+0000 mgr.x (mgr.24542) 1105 : cluster [DBG] 2021-07-31T02:13:03.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:03 smithi154 conmon[34553]: pgmap v1646: 9 pgs: 9 active+clean; 127 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 2.4 MiB/s rd, 1.8 MiB/s wr, 3 op/s 2021-07-31T02:13:03.912 INFO:tasks.workunit.client.0.smithi045.stdout: 468: throughput=4.99MB/sec pending data=0 2021-07-31T02:13:03.913 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-3CQijZDsNvYkhAY off=16915367 len=889055 2021-07-31T02:13:03.914 INFO:tasks.workunit.client.0.smithi045.stdout:op 2415 completed, throughput=4.99MB/sec 2021-07-31T02:13:03.914 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-88I7BJ2n8hsKknO off=12529360 len=1392638 2021-07-31T02:13:03.923 INFO:tasks.workunit.client.0.smithi045.stdout:op 2416 completed, throughput=4.99MB/sec 2021-07-31T02:13:03.923 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ul5TCBJYMwvUP7t off=12693165 len=1007050 2021-07-31T02:13:03.926 INFO:tasks.workunit.client.0.smithi045.stdout:op 2417 completed, throughput=5MB/sec 2021-07-31T02:13:03.926 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-LvGnLU-IZthiJNL off=6026114 len=1394424 2021-07-31T02:13:03.932 INFO:tasks.workunit.client.0.smithi045.stdout:op 2418 completed, throughput=5MB/sec 2021-07-31T02:13:03.932 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-1PJwKSXXtxmRJi6 off=5806055 len=921002 2021-07-31T02:13:03.935 INFO:tasks.workunit.client.0.smithi045.stdout:op 2419 completed, throughput=5MB/sec 2021-07-31T02:13:04.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:13:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:13:03.553Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:13:04.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:13:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:13:03.553Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:13:04.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:13:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:13:03.553Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:13:04.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:04 smithi154 conmon[34553]: cluster 2021-07-31T02:13:04.160880+0000 mgr.x (mgr.24542) 1106 : cluster [DBG] pgmap v1647: 9 pgs: 9 active+clean; 127 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 3.5 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2021-07-31T02:13:04.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:04 smithi154 conmon[34553]: 2021-07-31T02:13:04.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:04 smithi045 conmon[39718]: cluster 2021-07-31T02:13:04.160880+0000 mgr.x (mgr.24542) 1106 : cluster [DBG] pgmap v1647: 9 pgs: 9 active+clean; 127 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 3.5 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2021-07-31T02:13:04.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:04 smithi045 conmon[33655]: cluster 2021-07-31T02:13:04.160880+0000 mgr.x (mgr.24542) 1106 : cluster [DBG] pgmap v1647: 9 pgs: 9 active+clean; 127 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 3.5 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2021-07-31T02:13:04.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:04 smithi045 conmon[33655]: 2021-07-31T02:13:04.935 INFO:tasks.workunit.client.0.smithi045.stdout: 469: throughput=4.99MB/sec pending data=0 2021-07-31T02:13:04.935 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-VDB5HwhuDegF5QB off=7005199 len=555203 2021-07-31T02:13:04.939 INFO:tasks.workunit.client.0.smithi045.stdout:op 2420 completed, throughput=4.99MB/sec 2021-07-31T02:13:04.939 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-v5vsFhWf4y6iYFt off=3019330 len=2029159 2021-07-31T02:13:04.949 INFO:tasks.workunit.client.0.smithi045.stdout:op 2421 completed, throughput=5MB/sec 2021-07-31T02:13:04.949 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-2I4oV9nF9RhQJbg off=2846457 len=1267550 2021-07-31T02:13:04.951 INFO:tasks.workunit.client.0.smithi045.stdout:op 2422 completed, throughput=5MB/sec 2021-07-31T02:13:04.952 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-yBUdC4XVfZuRm1s off=1383424 len=1612436 2021-07-31T02:13:04.958 INFO:tasks.workunit.client.0.smithi045.stdout:op 2423 completed, throughput=5MB/sec 2021-07-31T02:13:05.959 INFO:tasks.workunit.client.0.smithi045.stdout: 470: throughput=4.99MB/sec pending data=0 2021-07-31T02:13:05.959 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-lThc90cNDZkgpBs off=1996878 len=1138148 2021-07-31T02:13:05.959 INFO:tasks.workunit.client.0.smithi045.stdout:op 2424 completed, throughput=4.99MB/sec 2021-07-31T02:13:05.960 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-gojq97kEEKgqQzW off=1176060 len=369916 2021-07-31T02:13:05.963 INFO:tasks.workunit.client.0.smithi045.stdout:op 2425 completed, throughput=4.99MB/sec 2021-07-31T02:13:05.963 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-9_9-OFHH0ySadfj off=11602773 len=878535 2021-07-31T02:13:05.964 INFO:tasks.workunit.client.0.smithi045.stdout:op 2426 completed, throughput=5MB/sec 2021-07-31T02:13:05.964 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-OKpjzsNK3Sy2DQw off=6668761 len=1350691 2021-07-31T02:13:05.965 INFO:tasks.workunit.client.0.smithi045.stdout:op 2427 completed, throughput=5MB/sec 2021-07-31T02:13:05.965 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-bbDAQgpAKtn1AFS off=12903966 len=111912 2021-07-31T02:13:05.967 INFO:tasks.workunit.client.0.smithi045.stdout:op 2428 completed, throughput=5MB/sec 2021-07-31T02:13:05.967 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-yK9NECZFLIuL3Sc off=97469 len=724375 2021-07-31T02:13:05.971 INFO:tasks.workunit.client.0.smithi045.stdout:op 2429 completed, throughput=5MB/sec 2021-07-31T02:13:05.972 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-5q6wVADctPF4-Ku off=10202279 len=1022062 2021-07-31T02:13:05.976 INFO:tasks.workunit.client.0.smithi045.stdout:op 2430 completed, throughput=5MB/sec 2021-07-31T02:13:06.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:06 smithi154 conmon[34553]: cluster 2021-07-31T02:13:06.161337+0000 mgr.x (mgr.24542) 1107 : cluster [DBG] pgmap v1648: 9 pgs: 9 active+clean; 127 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 2.3 MiB/s rd, 1.7 MiB/s wr, 4 op/s 2021-07-31T02:13:06.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:06 smithi154 conmon[34553]: 2021-07-31T02:13:06.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:06 smithi045 conmon[39718]: cluster 2021-07-31T02:13:06.161337+0000 mgr.x (mgr.24542) 2021-07-31T02:13:06.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:06 smithi045 conmon[39718]: 1107 : cluster [DBG] pgmap v1648: 9 pgs: 9 active+clean; 127 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 2.3 MiB/s rd, 1.7 MiB/s wr, 4 op/s 2021-07-31T02:13:06.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:06 smithi045 conmon[33655]: cluster 2021-07-31T02:13:06.161337+0000 mgr.x (mgr.24542) 1107 : cluster [DBG] 2021-07-31T02:13:06.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:06 smithi045 conmon[33655]: pgmap v1648: 9 pgs: 9 active+clean; 127 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 2.3 MiB/s rd, 1.7 MiB/s wr, 4 op/s 2021-07-31T02:13:06.977 INFO:tasks.workunit.client.0.smithi045.stdout: 471: throughput=4.99MB/sec pending data=0 2021-07-31T02:13:06.977 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-3fKMjDMTuAbCiH5 off=1600783 len=2077106 2021-07-31T02:13:06.987 INFO:tasks.workunit.client.0.smithi045.stdout:op 2431 completed, throughput=5MB/sec 2021-07-31T02:13:06.988 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-X--RYU5IHj1rXAp off=7378942 len=1937171 2021-07-31T02:13:06.997 INFO:tasks.workunit.client.0.smithi045.stdout:op 2432 completed, throughput=5MB/sec 2021-07-31T02:13:06.997 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-uQvg0LRgMkXOoTi off=2921393 len=1583233 2021-07-31T02:13:07.000 INFO:tasks.workunit.client.0.smithi045.stdout:op 2433 completed, throughput=5MB/sec 2021-07-31T02:13:08.000 INFO:tasks.workunit.client.0.smithi045.stdout: 472: throughput=4.99MB/sec pending data=0 2021-07-31T02:13:08.001 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-nf-LcntC7TRqO7N off=502622 len=42953 2021-07-31T02:13:08.001 INFO:tasks.workunit.client.0.smithi045.stdout:op 2434 completed, throughput=4.99MB/sec 2021-07-31T02:13:08.002 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-2o86vmBv3XMo_gP off=1287164 len=1177381 2021-07-31T02:13:08.003 INFO:tasks.workunit.client.0.smithi045.stdout:op 2435 completed, throughput=4.99MB/sec 2021-07-31T02:13:08.003 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-tFGskTwObBVosse off=15440541 len=1314027 2021-07-31T02:13:08.006 INFO:tasks.workunit.client.0.smithi045.stdout:op 2436 completed, throughput=5MB/sec 2021-07-31T02:13:08.006 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-YgWx9Z_eFB_jAkO off=1386309 len=421483 2021-07-31T02:13:08.007 INFO:tasks.workunit.client.0.smithi045.stdout:op 2437 completed, throughput=5MB/sec 2021-07-31T02:13:08.007 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-rOb9wzen_B8s7C0 off=1407168 len=1248330 2021-07-31T02:13:08.010 INFO:tasks.workunit.client.0.smithi045.stdout:op 2438 completed, throughput=5MB/sec 2021-07-31T02:13:08.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:08 smithi154 conmon[34553]: cluster 2021-07-31T02:13:08.161742+0000 mgr.x (mgr.24542) 1108 : cluster [DBG] pgmap v1649: 9 pgs: 9 active+clean; 127 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 2.3 MiB/s rd, 1.7 MiB/s wr, 4 op/s 2021-07-31T02:13:08.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:08 smithi045 conmon[39718]: cluster 2021-07-31T02:13:08. 2021-07-31T02:13:08.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:08 smithi045 conmon[39718]: 161742+0000 mgr.x (mgr.24542) 1108 : cluster [DBG] pgmap v1649: 9 pgs: 9 active+clean; 127 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 2.3 MiB/s rd, 1.7 MiB/s wr, 4 op/s 2021-07-31T02:13:08.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:08 smithi045 conmon[33655]: cluster 2021-07-31T02:13:08.161742+0000 mgr.x (mgr.24542) 1108 : cluster [DBG] pgmap v1649: 9 pgs: 9 active+clean; 127 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 2.3 MiB/s rd, 1.7 MiB/s wr, 4 op/s 2021-07-31T02:13:09.011 INFO:tasks.workunit.client.0.smithi045.stdout: 473: throughput=4.99MB/sec pending data=0 2021-07-31T02:13:09.011 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-F9kOd5Jhtsg0J6D off=5050165 len=835706 2021-07-31T02:13:09.012 INFO:tasks.workunit.client.0.smithi045.stdout:op 2439 completed, throughput=4.99MB/sec 2021-07-31T02:13:09.013 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-jaA--1lUd8saQF5 off=6598663 len=1931346 2021-07-31T02:13:09.021 INFO:tasks.workunit.client.0.smithi045.stdout:op 2440 completed, throughput=5MB/sec 2021-07-31T02:13:09.021 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-XT04ipFEOAObExl off=3369849 len=1514488 2021-07-31T02:13:09.023 INFO:tasks.workunit.client.0.smithi045.stdout:op 2441 completed, throughput=5MB/sec 2021-07-31T02:13:09.023 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj--icy5GugFOmmIV1 off=2621398 len=1034452 2021-07-31T02:13:09.029 INFO:tasks.workunit.client.0.smithi045.stdout:op 2442 completed, throughput=5MB/sec 2021-07-31T02:13:10.030 INFO:tasks.workunit.client.0.smithi045.stdout: 474: throughput=4.99MB/sec pending data=0 2021-07-31T02:13:10.030 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-aR8R1qFgRP7kOt7 off=1546178 len=210920 2021-07-31T02:13:10.031 INFO:tasks.workunit.client.0.smithi045.stdout:op 2443 completed, throughput=4.99MB/sec 2021-07-31T02:13:10.031 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Lul3mWCLKhl9AC6 off=11620330 len=1015719 2021-07-31T02:13:10.033 INFO:tasks.workunit.client.0.smithi045.stdout:op 2444 completed, throughput=4.99MB/sec 2021-07-31T02:13:10.033 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-EvCpxoZpc9eg0c4 off=16018662 len=754669 2021-07-31T02:13:10.038 INFO:tasks.workunit.client.0.smithi045.stdout:op 2445 completed, throughput=4.99MB/sec 2021-07-31T02:13:10.038 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-behrfp440k6WDLd off=21682866 len=275526 2021-07-31T02:13:10.041 INFO:tasks.workunit.client.0.smithi045.stdout:op 2446 completed, throughput=4.99MB/sec 2021-07-31T02:13:10.041 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-_wdOAF1Hk8loRAH off=1399156 len=1794385 2021-07-31T02:13:10.053 INFO:tasks.workunit.client.0.smithi045.stdout:op 2447 completed, throughput=5MB/sec 2021-07-31T02:13:10.053 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-odRqEdXsv4-zwbg off=19140501 len=474176 2021-07-31T02:13:10.054 INFO:tasks.workunit.client.0.smithi045.stdout:op 2448 completed, throughput=5MB/sec 2021-07-31T02:13:10.054 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-30KuCoSNnpooD1T off=1167601 len=1333914 2021-07-31T02:13:10.054 INFO:tasks.workunit.client.0.smithi045.stdout:op 2449 completed, throughput=5MB/sec 2021-07-31T02:13:10.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:10 smithi154 conmon[34553]: cluster 2021-07-31T02:13:10.162225+0000 mgr.x (mgr.24542) 1109 : cluster [DBG] pgmap v1650: 9 pgs: 9 active+clean; 127 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2021-07-31T02:13:10.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:10 smithi045 conmon[39718]: cluster 2021-07-31T02:13:10.162225 2021-07-31T02:13:10.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:10 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 1109 : cluster [DBG] pgmap v1650: 9 pgs: 9 active+clean; 127 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2021-07-31T02:13:10.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:10 smithi045 conmon[33655]: cluster 2021-07-31T02:13:10.162225+0000 mgr.x (mgr.24542) 1109 : 2021-07-31T02:13:10.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:10 smithi045 conmon[33655]: cluster [DBG] pgmap v1650: 9 pgs: 9 active+clean; 127 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2021-07-31T02:13:11.054 INFO:tasks.workunit.client.0.smithi045.stdout: 475: throughput=4.99MB/sec pending data=0 2021-07-31T02:13:11.056 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-S5CHSJ7x67gO67y off=16052523 len=221846 2021-07-31T02:13:11.058 INFO:tasks.workunit.client.0.smithi045.stdout:op 2450 completed, throughput=4.99MB/sec 2021-07-31T02:13:11.058 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-RTu8B-GdPoiYeGb off=2306500 len=1082630 2021-07-31T02:13:11.059 INFO:tasks.workunit.client.0.smithi045.stdout:op 2451 completed, throughput=4.99MB/sec 2021-07-31T02:13:11.059 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-0IsyIpna0Ke436a off=2074750 len=1477560 2021-07-31T02:13:11.062 INFO:tasks.workunit.client.0.smithi045.stdout:op 2452 completed, throughput=5MB/sec 2021-07-31T02:13:11.062 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-3WT2PITjjClg-qZ off=7326869 len=328440 2021-07-31T02:13:11.067 INFO:tasks.workunit.client.0.smithi045.stdout:op 2453 completed, throughput=5MB/sec 2021-07-31T02:13:11.067 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-14YJhboSjCLmjMw off=13478182 len=1615025 2021-07-31T02:13:11.075 INFO:tasks.workunit.client.0.smithi045.stdout:op 2454 completed, throughput=5MB/sec 2021-07-31T02:13:11.075 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-1A_FDR79JUuxLdN off=0 len=1588002 2021-07-31T02:13:11.082 INFO:tasks.workunit.client.0.smithi045.stdout:op 2455 completed, throughput=5MB/sec 2021-07-31T02:13:12.082 INFO:tasks.workunit.client.0.smithi045.stdout: 476: throughput=4.99MB/sec pending data=0 2021-07-31T02:13:12.082 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-3Ztim0fajIHxZB8 off=423730 len=896070 2021-07-31T02:13:12.088 INFO:tasks.workunit.client.0.smithi045.stdout:op 2456 completed, throughput=4.99MB/sec 2021-07-31T02:13:12.088 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-fn1UFioeX4eKULC off=16378912 len=341642 2021-07-31T02:13:12.089 INFO:tasks.workunit.client.0.smithi045.stdout:op 2457 completed, throughput=5MB/sec 2021-07-31T02:13:12.089 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-kkNBLm7KknbU6KC off=14428937 len=1733528 2021-07-31T02:13:12.096 INFO:tasks.workunit.client.0.smithi045.stdout:op 2458 completed, throughput=5MB/sec 2021-07-31T02:13:12.096 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-A6bwIcMEuD46FiN off=12814974 len=1067973 2021-07-31T02:13:12.098 INFO:tasks.workunit.client.0.smithi045.stdout:op 2459 completed, throughput=5MB/sec 2021-07-31T02:13:12.870 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:13:13.099 INFO:tasks.workunit.client.0.smithi045.stdout: 477: throughput=4.99MB/sec pending data=0 2021-07-31T02:13:13.099 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-wBLJ9V9eitjYQJC off=708298 len=1125294 2021-07-31T02:13:13.101 INFO:tasks.workunit.client.0.smithi045.stdout:op 2460 completed, throughput=4.99MB/sec 2021-07-31T02:13:13.101 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-8grCejhkLmalBWF off=5797677 len=101309 2021-07-31T02:13:13.102 INFO:tasks.workunit.client.0.smithi045.stdout:op 2461 completed, throughput=4.99MB/sec 2021-07-31T02:13:13.102 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-G0od48Uc9oLXZ0A off=5351391 len=447641 2021-07-31T02:13:13.106 INFO:tasks.workunit.client.0.smithi045.stdout:op 2462 completed, throughput=4.99MB/sec 2021-07-31T02:13:13.106 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-SoB1yQNE9t47K8D off=3232406 len=1427339 2021-07-31T02:13:13.112 INFO:tasks.workunit.client.0.smithi045.stdout:op 2463 completed, throughput=5MB/sec 2021-07-31T02:13:13.112 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-l2X9WPPy-FA29Zd off=23977592 len=220002 2021-07-31T02:13:13.113 INFO:tasks.workunit.client.0.smithi045.stdout:op 2464 completed, throughput=5MB/sec 2021-07-31T02:13:13.113 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Imfe0JtG2PiMKvn off=14680200 len=1414755 2021-07-31T02:13:13.120 INFO:tasks.workunit.client.0.smithi045.stdout:op 2465 completed, throughput=5MB/sec 2021-07-31T02:13:13.120 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-vr9kmrXemzwUj91 off=8644654 len=1321175 2021-07-31T02:13:13.121 INFO:tasks.workunit.client.0.smithi045.stdout:op 2466 completed, throughput=5MB/sec 2021-07-31T02:13:13.154 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:13:13.154 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 28m ago 28m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:13:13.154 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 25m ago 28m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:13:13.155 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 25m ago 32m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:13:13.155 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 28m ago 34m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:13:13.155 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 28m ago 34m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:13:13.155 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 25m ago 32m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:13:13.156 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 28m ago 32m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:13:13.156 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 28m ago 29m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:13:13.156 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 25m ago 29m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:13:13.156 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 28m ago 31m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:13:13.156 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 28m ago 31m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:13:13.157 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 28m ago 31m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:13:13.157 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 28m ago 30m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:13:13.157 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 25m ago 30m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:13:13.157 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 25m ago 30m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:13:13.158 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 25m ago 29m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:13:13.158 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 25m ago 29m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:13:13.158 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 25m ago 29m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:13:13.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:13 smithi045 conmon[39718]: cluster 2021-07-31T02:13 2021-07-31T02:13:13.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:13 smithi045 conmon[39718]: :12.162657+0000 mgr.x (mgr.24542) 1110 : cluster [DBG] pgmap v1651: 9 pgs: 9 active+clean; 127 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 2.0 MiB/s rd, 1.8 MiB/s wr, 3 op/s 2021-07-31T02:13:13.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:13 smithi045 conmon[33655]: cluster 2021-07-31T02:13:12.162657+0000 mgr.x (mgr. 2021-07-31T02:13:13.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:13 smithi045 conmon[33655]: 24542) 1110 : cluster [DBG] pgmap v1651: 9 pgs: 9 active+clean; 127 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 2.0 MiB/s rd, 1.8 MiB/s wr, 3 op/s 2021-07-31T02:13:13.488 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:13:13.488 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:13:13.488 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:13:13.488 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:13:13.489 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:13:13.489 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:13:13.489 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:13:13.489 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:13:13.490 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:13:13.490 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:13:13.490 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:13:13.490 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:13:13.491 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:13:13.491 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:13:13.491 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:13:13.491 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:13:13.491 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:13:13.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:13 smithi154 conmon[34553]: cluster 2021-07-31T02:13:12.162657+0000 mgr.x (mgr.24542) 1110 : cluster 2021-07-31T02:13:13.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:13 smithi154 conmon[34553]: [DBG] pgmap v1651: 9 pgs: 9 active+clean; 127 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 2.0 MiB/s rd, 1.8 MiB/s wr, 3 op/s 2021-07-31T02:13:14.022 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:13:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:13:13.553Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:13:14.022 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:13:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:13:13.553Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:13:14.023 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:13:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:13:13.553Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:13:14.122 INFO:tasks.workunit.client.0.smithi045.stdout: 478: throughput=4.99MB/sec pending data=0 2021-07-31T02:13:14.122 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-UluQt4Y16tU7KU9 off=7644270 len=647560 2021-07-31T02:13:14.126 INFO:tasks.workunit.client.0.smithi045.stdout:op 2467 completed, throughput=4.99MB/sec 2021-07-31T02:13:14.126 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-0ZpdrF6GPQLxbLv off=21513665 len=922747 2021-07-31T02:13:14.131 INFO:tasks.workunit.client.0.smithi045.stdout:op 2468 completed, throughput=4.99MB/sec 2021-07-31T02:13:14.131 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-hEOAXqmA7DeX64h off=7080400 len=1434476 2021-07-31T02:13:14.138 INFO:tasks.workunit.client.0.smithi045.stdout:op 2469 completed, throughput=5MB/sec 2021-07-31T02:13:14.138 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-EPYwVgZhFG4MnXP off=2404014 len=1318712 2021-07-31T02:13:14.144 INFO:tasks.workunit.client.0.smithi045.stdout:op 2470 completed, throughput=5MB/sec 2021-07-31T02:13:14.144 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-cJDkURvisfe4FbB off=0 len=1492813 2021-07-31T02:13:14.150 INFO:tasks.workunit.client.0.smithi045.stdout:op 2471 completed, throughput=5MB/sec 2021-07-31T02:13:14.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:14 smithi045 conmon[33655]: audit 2021-07-31T02:13:12.858361+0000 mgr.x (mgr.24542) 2021-07-31T02:13:14.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:14 smithi045 conmon[33655]: 1111 : audit [DBG] from='client.25912 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:13:14.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:14 smithi045 conmon[33655]: audit 2021-07-31T02:13:13.146529+0000 mgr.x (mgr.24542) 1112 : audit [DBG] from='client.25918 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:13:14.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:14 smithi045 conmon[33655]: audit 2021-07-31T02:13: 2021-07-31T02:13:14.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:14 smithi045 conmon[33655]: 13.487984+0000 mon.c (mon. 2021-07-31T02:13:14.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:14 smithi045 conmon[33655]: 1) 82 : audit [DBG] 2021-07-31T02:13:14.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:14 smithi045 conmon[33655]: from='client.? 172.21.15.45:0/3583521998' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:13:14.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:14 smithi045 conmon[39718]: audit 2021-07-31T02:13 2021-07-31T02:13:14.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:14 smithi045 conmon[39718]: :12.858361+0000 mgr.x (mgr.24542) 1111 : audit [DBG] from='client.25912 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:13:14.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:14 smithi045 conmon[39718]: audit 2021- 2021-07-31T02:13:14.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:14 smithi045 conmon[39718]: 07-31T02:13:13.146529+0000 mgr.x (mgr.24542) 1112 2021-07-31T02:13:14.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:14 smithi045 conmon[39718]: : audit [DBG] from='client.25918 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:13:14.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:14 smithi045 conmon[39718]: audit 2021-07-31T02:13:14.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:14 smithi045 conmon[39718]: 2021-07-31T02:13 2021-07-31T02:13:14.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:14 smithi045 conmon[39718]: :13.487984+0000 mon.c ( 2021-07-31T02:13:14.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:14 smithi045 conmon[39718]: mon.1) 82 : audit 2021-07-31T02:13:14.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:14 smithi045 conmon[39718]: [DBG] from='client.? 172.21.15.45:0/3583521998' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:13:14.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:14 smithi154 conmon[34553]: audit 2021-07-31T02:13:12.858361+0000 mgr.x (mgr.24542) 1111 : audit [DBG] from='client.25912 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:13:14.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:14 smithi154 conmon[34553]: audit 2021-07-31T02:13:13.146529+0000 mgr.x (mgr.24542) 1112 : audit [DBG] from='client.25918 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:13:14.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:14 smithi154 conmon[34553]: audit 2021-07-31T02:13:13.487984+0000 mon.c (mon.1) 82 : audit [DBG] from='client.? 172.21.15.45:0/3583521998' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:13:15.150 INFO:tasks.workunit.client.0.smithi045.stdout: 479: throughput=4.99MB/sec pending data=0 2021-07-31T02:13:15.152 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-XQkXK-M734yhgEX off=6452038 len=1968316 2021-07-31T02:13:15.158 INFO:tasks.workunit.client.0.smithi045.stdout:op 2472 completed, throughput=5MB/sec 2021-07-31T02:13:15.159 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ZjtzSB-UdrbgJfd off=11067437 len=1226654 2021-07-31T02:13:15.164 INFO:tasks.workunit.client.0.smithi045.stdout:op 2473 completed, throughput=5MB/sec 2021-07-31T02:13:15.165 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-cGQwUTWAoZUxKJ3 off=2853485 len=1518024 2021-07-31T02:13:15.169 INFO:tasks.workunit.client.0.smithi045.stdout:op 2474 completed, throughput=5MB/sec 2021-07-31T02:13:15.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:15 smithi154 conmon[34553]: cluster 2021-07-31T02:13:14.163214+0000 mgr.x (mgr.24542) 1113 : cluster [DBG] pgmap v1652: 9 pgs: 9 active+clean; 127 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 3.0 MiB/s rd, 3.0 MiB/s wr, 5 op/s 2021-07-31T02:13:15.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:15 smithi045 conmon[33655]: cluster 2021-07-31T02:13:14.163214+0000 mgr.x (mgr.24542) 1113 : cluster [DBG] pgmap v1652: 9 pgs: 9 active+clean; 127 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 3.0 MiB/s rd, 3.0 MiB/s wr, 5 op/s 2021-07-31T02:13:15.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:15 smithi045 conmon[39718]: cluster 2021-07-31T02:13:14.163214+0000 mgr.x (mgr.24542) 1113 2021-07-31T02:13:15.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:15 smithi045 conmon[39718]: : cluster [DBG] pgmap v1652: 9 pgs: 9 active+clean; 127 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 3.0 MiB/s rd, 3.0 MiB/s wr, 5 op/s 2021-07-31T02:13:16.169 INFO:tasks.workunit.client.0.smithi045.stdout: 480: throughput=4.99MB/sec pending data=0 2021-07-31T02:13:16.169 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-fZVdS3tY_CPFBaC off=6384092 len=465007 2021-07-31T02:13:16.173 INFO:tasks.workunit.client.0.smithi045.stdout:op 2475 completed, throughput=4.99MB/sec 2021-07-31T02:13:16.174 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-HKYrL4xhWqDrdfy off=873479 len=76532 2021-07-31T02:13:16.176 INFO:tasks.workunit.client.0.smithi045.stdout:op 2476 completed, throughput=4.99MB/sec 2021-07-31T02:13:16.176 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-gyIOj1M0zYumg_v off=7991508 len=270502 2021-07-31T02:13:16.179 INFO:tasks.workunit.client.0.smithi045.stdout:op 2477 completed, throughput=4.99MB/sec 2021-07-31T02:13:16.180 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-BhjN5t1NUA8f3w7 off=3533793 len=1221131 2021-07-31T02:13:16.181 INFO:tasks.workunit.client.0.smithi045.stdout:op 2478 completed, throughput=4.99MB/sec 2021-07-31T02:13:16.181 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-fD1h1HXeOXxE2xQ off=13662193 len=2034762 2021-07-31T02:13:16.192 INFO:tasks.workunit.client.0.smithi045.stdout:op 2479 completed, throughput=5MB/sec 2021-07-31T02:13:16.192 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-KWtXqnBVfiOTXDg off=14916352 len=1829236 2021-07-31T02:13:16.195 INFO:tasks.workunit.client.0.smithi045.stdout:op 2480 completed, throughput=5MB/sec 2021-07-31T02:13:16.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:16 smithi154 conmon[34553]: cluster 2021-07-31T02:13:16.163603+0000 mgr.x (mgr.24542) 1114 : cluster [DBG] pgmap v1653: 9 pgs: 9 active+clean; 127 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2021-07-31T02:13:16.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:16 smithi045 conmon[33655]: cluster 2021-07-31T02:13:16.163603+0000 mgr.x (mgr.24542) 1114 : cluster [DBG] 2021-07-31T02:13:16.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:16 smithi045 conmon[33655]: pgmap v1653: 9 pgs: 9 active+clean; 127 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2021-07-31T02:13:16.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:16 smithi045 conmon[39718]: cluster 2021-07-31T02:13:16.163603+0000 mgr.x (mgr.24542 2021-07-31T02:13:16.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:16 smithi045 conmon[39718]: ) 1114 : cluster [DBG] pgmap v1653: 9 pgs: 9 active+clean; 127 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2021-07-31T02:13:17.196 INFO:tasks.workunit.client.0.smithi045.stdout: 481: throughput=4.99MB/sec pending data=0 2021-07-31T02:13:17.197 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-q5wjkhckv-ynCTU off=1699844 len=1622031 2021-07-31T02:13:17.197 INFO:tasks.workunit.client.0.smithi045.stdout:op 2481 completed, throughput=5MB/sec 2021-07-31T02:13:17.197 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-OXVmHlhf0yjV00n off=16562732 len=420623 2021-07-31T02:13:17.199 INFO:tasks.workunit.client.0.smithi045.stdout:op 2482 completed, throughput=5MB/sec 2021-07-31T02:13:17.199 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-UOf1lFI3ROJVjGW off=5596640 len=2025602 2021-07-31T02:13:17.209 INFO:tasks.workunit.client.0.smithi045.stdout:op 2483 completed, throughput=5MB/sec 2021-07-31T02:13:17.210 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ITm2I58Qvc0ifOm off=7692679 len=1787806 2021-07-31T02:13:17.220 INFO:tasks.workunit.client.0.smithi045.stdout:op 2484 completed, throughput=5MB/sec 2021-07-31T02:13:18.220 INFO:tasks.workunit.client.0.smithi045.stdout: 482: throughput=4.99MB/sec pending data=0 2021-07-31T02:13:18.220 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-nBBu-KNaRXhOLpI off=10827062 len=1017158 2021-07-31T02:13:18.226 INFO:tasks.workunit.client.0.smithi045.stdout:op 2485 completed, throughput=5MB/sec 2021-07-31T02:13:18.226 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-FCNoxEE1tRq_ZoA off=695371 len=1363361 2021-07-31T02:13:18.229 INFO:tasks.workunit.client.0.smithi045.stdout:op 2486 completed, throughput=5MB/sec 2021-07-31T02:13:18.229 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-bs41xwfLFN-3N2i off=5992678 len=257905 2021-07-31T02:13:18.230 INFO:tasks.workunit.client.0.smithi045.stdout:op 2487 completed, throughput=5MB/sec 2021-07-31T02:13:18.230 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-d-x_oaoDzU2CrBP off=3765121 len=1775575 2021-07-31T02:13:18.240 INFO:tasks.workunit.client.0.smithi045.stdout:op 2488 completed, throughput=5MB/sec 2021-07-31T02:13:18.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:18 smithi154 conmon[34553]: cluster 2021-07-31T02:13:18.163963+0000 mgr.x (mgr.24542) 1115 : cluster [DBG] pgmap v1654: 9 pgs: 9 active+clean; 127 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2021-07-31T02:13:18.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:18 smithi045 conmon[33655]: cluster 2021-07-31T02:13:18.163963+0000 mgr.x 2021-07-31T02:13:18.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:18 smithi045 conmon[33655]: (mgr.24542) 1115 : cluster [DBG] pgmap v1654: 9 pgs: 9 active+clean; 127 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2021-07-31T02:13:18.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:18 smithi045 conmon[39718]: cluster 2021-07-31T02:13:18.163963+0000 mgr.x 2021-07-31T02:13:18.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:18 smithi045 conmon[39718]: (mgr.24542) 1115 : cluster [DBG] pgmap v1654: 9 pgs: 9 active+clean; 127 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2021-07-31T02:13:19.241 INFO:tasks.workunit.client.0.smithi045.stdout: 483: throughput=4.99MB/sec pending data=0 2021-07-31T02:13:19.241 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-yTEb9TbvkdEyHB5 off=4559252 len=545153 2021-07-31T02:13:19.245 INFO:tasks.workunit.client.0.smithi045.stdout:op 2489 completed, throughput=4.99MB/sec 2021-07-31T02:13:19.246 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-G8kAbr4fjxpMiQL off=7774382 len=176169 2021-07-31T02:13:19.246 INFO:tasks.workunit.client.0.smithi045.stdout:op 2490 completed, throughput=4.99MB/sec 2021-07-31T02:13:19.246 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Q5kh1bi-N9lHDzO off=10284340 len=1607528 2021-07-31T02:13:19.248 INFO:tasks.workunit.client.0.smithi045.stdout:op 2491 completed, throughput=5MB/sec 2021-07-31T02:13:19.248 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Yc15346Y9tTHkjh off=7034134 len=682978 2021-07-31T02:13:19.249 INFO:tasks.workunit.client.0.smithi045.stdout:op 2492 completed, throughput=5MB/sec 2021-07-31T02:13:19.250 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ZzQuZnrUvzbPMKU off=8278781 len=553645 2021-07-31T02:13:19.251 INFO:tasks.workunit.client.0.smithi045.stdout:op 2493 completed, throughput=5MB/sec 2021-07-31T02:13:19.251 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-4mQfGRlZKW57Sbv off=10863273 len=1068624 2021-07-31T02:13:19.254 INFO:tasks.workunit.client.0.smithi045.stdout:op 2494 completed, throughput=5MB/sec 2021-07-31T02:13:20.254 INFO:tasks.workunit.client.0.smithi045.stdout: 484: throughput=4.99MB/sec pending data=0 2021-07-31T02:13:20.255 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-lpJe7PQ2QmMxtti off=5575119 len=1384885 2021-07-31T02:13:20.261 INFO:tasks.workunit.client.0.smithi045.stdout:op 2495 completed, throughput=4.99MB/sec 2021-07-31T02:13:20.261 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-aVWaYf1vevY1UN3 off=19213976 len=1344872 2021-07-31T02:13:20.268 INFO:tasks.workunit.client.0.smithi045.stdout:op 2496 completed, throughput=5MB/sec 2021-07-31T02:13:20.269 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-6klbm0-h99DiLEZ off=1278027 len=1552661 2021-07-31T02:13:20.275 INFO:tasks.workunit.client.0.smithi045.stdout:op 2497 completed, throughput=5MB/sec 2021-07-31T02:13:20.275 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-sLtMf7SSYVIMZfo off=13369630 len=1970488 2021-07-31T02:13:20.286 INFO:tasks.workunit.client.0.smithi045.stdout:op 2498 completed, throughput=5MB/sec 2021-07-31T02:13:20.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:20 smithi154 conmon[34553]: cluster 2021-07-31T02:13:20.164416+0000 mgr.x (mgr.24542) 1116 : cluster [DBG] pgmap v1655: 9 pgs: 9 active+clean; 127 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 2.5 MiB/s rd, 3.6 MiB/s wr, 5 op/s 2021-07-31T02:13:20.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:20 smithi045 conmon[33655]: cluster 2021-07-31T02:13:20.164416+0000 mgr.x (mgr 2021-07-31T02:13:20.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:20 smithi045 conmon[33655]: .24542) 1116 : cluster [DBG] pgmap v1655: 9 pgs: 9 active+clean; 127 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 2.5 MiB/s rd, 3.6 MiB/s wr, 5 op/s 2021-07-31T02:13:20.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:20 smithi045 conmon[39718]: cluster 2021-07-31T02:13:20.164416+0000 2021-07-31T02:13:20.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:20 smithi045 conmon[39718]: mgr.x (mgr.24542) 1116 : cluster [DBG] pgmap v1655: 9 pgs: 9 active+clean; 127 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 2.5 MiB/s rd, 3.6 MiB/s wr, 5 op/s 2021-07-31T02:13:21.286 INFO:tasks.workunit.client.0.smithi045.stdout: 485: throughput=4.99MB/sec pending data=0 2021-07-31T02:13:21.287 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-JBGG-5FMgD3KX8R off=11430678 len=274525 2021-07-31T02:13:21.290 INFO:tasks.workunit.client.0.smithi045.stdout:op 2499 completed, throughput=4.99MB/sec 2021-07-31T02:13:21.290 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-7R3IjT34KFBMM8N off=13035688 len=1745993 2021-07-31T02:13:21.298 INFO:tasks.workunit.client.0.smithi045.stdout:op 2500 completed, throughput=5MB/sec 2021-07-31T02:13:21.299 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-0b_9-hViKhSpdjl off=13790793 len=439194 2021-07-31T02:13:21.302 INFO:tasks.workunit.client.0.smithi045.stdout:op 2501 completed, throughput=5MB/sec 2021-07-31T02:13:21.302 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-miydMw8N7oQ3yLT off=11915614 len=1526907 2021-07-31T02:13:21.304 INFO:tasks.workunit.client.0.smithi045.stdout:op 2502 completed, throughput=5MB/sec 2021-07-31T02:13:21.304 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-uqSo02htyd4Gw3f off=2478362 len=1932493 2021-07-31T02:13:21.315 INFO:tasks.workunit.client.0.smithi045.stdout:op 2503 completed, throughput=5MB/sec 2021-07-31T02:13:22.316 INFO:tasks.workunit.client.0.smithi045.stdout: 486: throughput=4.99MB/sec pending data=0 2021-07-31T02:13:22.316 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-VO75wD47tXNx65v off=21552276 len=1855841 2021-07-31T02:13:22.319 INFO:tasks.workunit.client.0.smithi045.stdout:op 2504 completed, throughput=5MB/sec 2021-07-31T02:13:22.320 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-W4XG1uSM2oXsFMh off=7011612 len=1409241 2021-07-31T02:13:22.326 INFO:tasks.workunit.client.0.smithi045.stdout:op 2505 completed, throughput=5MB/sec 2021-07-31T02:13:22.326 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-xd3WP-UUAr6OVWE off=0 len=1115345 2021-07-31T02:13:22.328 INFO:tasks.workunit.client.0.smithi045.stdout:op 2506 completed, throughput=5MB/sec 2021-07-31T02:13:23.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:23 smithi045 conmon[33655]: cluster 2021-07-31T02:13:22.164853+0000 mgr.x (mgr.24542) 1117 : cluster [DBG] pgmap v1656: 9 pgs: 9 active+clean; 127 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1.6 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2021-07-31T02:13:23.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:23 smithi045 conmon[39718]: cluster 2021-07-31T02:13:22.164853+0000 mgr.x (mgr 2021-07-31T02:13:23.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:23 smithi045 conmon[39718]: .24542) 1117 : cluster [DBG] pgmap v1656: 9 pgs: 9 active+clean; 127 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1.6 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2021-07-31T02:13:23.329 INFO:tasks.workunit.client.0.smithi045.stdout: 487: throughput=4.99MB/sec pending data=0 2021-07-31T02:13:23.329 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-G8kAbr4fjxpMiQL off=6273107 len=1253116 2021-07-31T02:13:23.331 INFO:tasks.workunit.client.0.smithi045.stdout:op 2507 completed, throughput=4.99MB/sec 2021-07-31T02:13:23.331 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Jdziy_x-FGyn3Hk off=7256666 len=43172 2021-07-31T02:13:23.334 INFO:tasks.workunit.client.0.smithi045.stdout:op 2508 completed, throughput=4.99MB/sec 2021-07-31T02:13:23.334 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Rxp9dncaCVJss6c off=10091681 len=1299553 2021-07-31T02:13:23.336 INFO:tasks.workunit.client.0.smithi045.stdout:op 2509 completed, throughput=5MB/sec 2021-07-31T02:13:23.337 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-A--oa_Zvg-maSkb off=10973724 len=251689 2021-07-31T02:13:23.337 INFO:tasks.workunit.client.0.smithi045.stdout:op 2510 completed, throughput=5MB/sec 2021-07-31T02:13:23.337 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-0LKKDPl6jmIeqYS off=4611593 len=432316 2021-07-31T02:13:23.341 INFO:tasks.workunit.client.0.smithi045.stdout:op 2511 completed, throughput=5MB/sec 2021-07-31T02:13:23.341 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Fm6P3nYw_uu1x6c off=2959091 len=1189430 2021-07-31T02:13:23.347 INFO:tasks.workunit.client.0.smithi045.stdout:op 2512 completed, throughput=5MB/sec 2021-07-31T02:13:23.347 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-2LvuM_N8mPEwdIc off=695066 len=1257672 2021-07-31T02:13:23.350 INFO:tasks.workunit.client.0.smithi045.stdout:op 2513 completed, throughput=5MB/sec 2021-07-31T02:13:23.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:23 smithi154 conmon[34553]: cluster 2021-07-31T02:13:22.164853+0000 mgr.x (mgr.24542) 1117 : cluster 2021-07-31T02:13:23.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:23 smithi154 conmon[34553]: [DBG] pgmap v1656: 9 pgs: 9 active+clean; 127 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1.6 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2021-07-31T02:13:24.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:13:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:13:23.553Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:13:24.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:13:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:13:23.553Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:13:24.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:13:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:13:23.553Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:13:24.350 INFO:tasks.workunit.client.0.smithi045.stdout: 488: throughput=4.99MB/sec pending data=0 2021-07-31T02:13:24.350 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-HxyovAuNi5sM7zY off=1450325 len=109772 2021-07-31T02:13:24.353 INFO:tasks.workunit.client.0.smithi045.stdout:op 2514 completed, throughput=4.99MB/sec 2021-07-31T02:13:24.353 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-nI5MNAmhK4HKNO2 off=5594863 len=810385 2021-07-31T02:13:24.358 INFO:tasks.workunit.client.0.smithi045.stdout:op 2515 completed, throughput=4.99MB/sec 2021-07-31T02:13:24.358 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-4XA3LY8rNWhhP_Q off=5225576 len=1222223 2021-07-31T02:13:24.365 INFO:tasks.workunit.client.0.smithi045.stdout:op 2516 completed, throughput=5MB/sec 2021-07-31T02:13:24.366 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-UNWDkC0v0Co9r0r off=2715488 len=816848 2021-07-31T02:13:24.371 INFO:tasks.workunit.client.0.smithi045.stdout:op 2517 completed, throughput=5MB/sec 2021-07-31T02:13:24.371 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-SKaDGFZBKCX1Ev4 off=4786147 len=247159 2021-07-31T02:13:24.374 INFO:tasks.workunit.client.0.smithi045.stdout:op 2518 completed, throughput=5MB/sec 2021-07-31T02:13:24.374 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-wShg9NI9PYmcB8m off=5524137 len=1304369 2021-07-31T02:13:24.375 INFO:tasks.workunit.client.0.smithi045.stdout:op 2519 completed, throughput=5MB/sec 2021-07-31T02:13:24.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:24 smithi154 conmon[34553]: cluster 2021-07-31T02:13:24.165474+0000 mgr.x (mgr.24542) 1118 : cluster [DBG] pgmap v1657: 9 pgs: 9 active+clean; 127 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 2.6 MiB/s rd, 3.8 MiB/s wr, 6 op/s 2021-07-31T02:13:24.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:24 smithi045 conmon[39718]: cluster 2021-07-31T02:13:24.165474+0000 mgr.x (mgr 2021-07-31T02:13:24.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:24 smithi045 conmon[39718]: .24542) 1118 : cluster [DBG] pgmap v1657: 9 pgs: 9 active+clean; 127 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 2.6 MiB/s rd, 3.8 MiB/s wr, 6 op/s 2021-07-31T02:13:24.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:24 smithi045 conmon[33655]: cluster 2021-07-31T02:13:24. 2021-07-31T02:13:24.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:24 smithi045 conmon[33655]: 165474+0000 mgr.x (mgr.24542) 1118 : cluster [DBG] pgmap v1657: 9 pgs: 9 active+clean; 127 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 2.6 MiB/s rd, 3.8 MiB/s wr, 6 op/s 2021-07-31T02:13:25.375 INFO:tasks.workunit.client.0.smithi045.stdout: 489: throughput=4.99MB/sec pending data=0 2021-07-31T02:13:25.376 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-0d450R293HLLM_t off=4468352 len=709332 2021-07-31T02:13:25.380 INFO:tasks.workunit.client.0.smithi045.stdout:op 2520 completed, throughput=4.99MB/sec 2021-07-31T02:13:25.380 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-SEuSQaYUBzmNcJu off=7889590 len=1965956 2021-07-31T02:13:25.389 INFO:tasks.workunit.client.0.smithi045.stdout:op 2521 completed, throughput=5MB/sec 2021-07-31T02:13:25.389 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-XR55tcA5Ba0UPqf off=14648052 len=513939 2021-07-31T02:13:25.390 INFO:tasks.workunit.client.0.smithi045.stdout:op 2522 completed, throughput=5MB/sec 2021-07-31T02:13:25.390 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-iovKpFiXcbyisY1 off=881747 len=15037 2021-07-31T02:13:25.391 INFO:tasks.workunit.client.0.smithi045.stdout:op 2523 completed, throughput=5MB/sec 2021-07-31T02:13:25.392 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ijgxT3Z9dMxBG5E off=8410106 len=979964 2021-07-31T02:13:25.392 INFO:tasks.workunit.client.0.smithi045.stdout:op 2524 completed, throughput=5MB/sec 2021-07-31T02:13:25.392 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-E8jvNpOd2LjaH7m off=3713312 len=356608 2021-07-31T02:13:25.395 INFO:tasks.workunit.client.0.smithi045.stdout:op 2525 completed, throughput=5MB/sec 2021-07-31T02:13:25.395 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-XnnIGNjU-GbKQRi off=11831153 len=393646 2021-07-31T02:13:25.396 INFO:tasks.workunit.client.0.smithi045.stdout:op 2526 completed, throughput=5MB/sec 2021-07-31T02:13:25.396 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-E_S_7Oqc3Ml9ogu off=8734893 len=1367900 2021-07-31T02:13:25.402 INFO:tasks.workunit.client.0.smithi045.stdout:op 2527 completed, throughput=5MB/sec 2021-07-31T02:13:26.402 INFO:tasks.workunit.client.0.smithi045.stdout: 490: throughput=4.99MB/sec pending data=0 2021-07-31T02:13:26.403 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Z3pr8wqpwYKsUyW off=7125171 len=1469954 2021-07-31T02:13:26.410 INFO:tasks.workunit.client.0.smithi045.stdout:op 2528 completed, throughput=4.99MB/sec 2021-07-31T02:13:26.411 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Ed8_qa4bQRyQWgw off=4102004 len=554323 2021-07-31T02:13:26.411 INFO:tasks.workunit.client.0.smithi045.stdout:op 2529 completed, throughput=5MB/sec 2021-07-31T02:13:26.412 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-kEb_MSIlJPCmlYe off=22916362 len=406675 2021-07-31T02:13:26.412 INFO:tasks.workunit.client.0.smithi045.stdout:op 2530 completed, throughput=5MB/sec 2021-07-31T02:13:26.413 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ThU0-nQU1cQS-FT off=0 len=1122758 2021-07-31T02:13:26.418 INFO:tasks.workunit.client.0.smithi045.stdout:op 2531 completed, throughput=5MB/sec 2021-07-31T02:13:26.419 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Ao3dGPK1PQ0I_L9 off=7903217 len=1354568 2021-07-31T02:13:26.424 INFO:tasks.workunit.client.0.smithi045.stdout:op 2532 completed, throughput=5MB/sec 2021-07-31T02:13:26.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:26 smithi154 conmon[34553]: cluster 2021-07-31T02:13:26.166052+0000 mgr.x (mgr.24542) 1119 : cluster [DBG] pgmap v1658: 9 pgs: 9 active+clean; 127 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.6 MiB/s wr, 3 op/s 2021-07-31T02:13:26.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:26 smithi045 conmon[39718]: cluster 2021-07-31T02:13:26.166052+0000 mgr.x (mgr. 2021-07-31T02:13:26.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:26 smithi045 conmon[39718]: 24542) 1119 : cluster [DBG] pgmap v1658: 9 pgs: 9 active+clean; 127 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.6 MiB/s wr, 3 op/s 2021-07-31T02:13:26.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:26 smithi045 conmon[33655]: cluster 2021- 2021-07-31T02:13:26.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:26 smithi045 conmon[33655]: 07-31T02:13:26.166052+0000 mgr.x (mgr.24542) 1119 : cluster [DBG] pgmap v1658: 9 pgs: 9 active+clean; 127 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.6 MiB/s wr, 3 op/s 2021-07-31T02:13:27.424 INFO:tasks.workunit.client.0.smithi045.stdout: 491: throughput=4.99MB/sec pending data=0 2021-07-31T02:13:27.424 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-GpUeJCY87ihsDC6 off=430509 len=1376708 2021-07-31T02:13:27.427 INFO:tasks.workunit.client.0.smithi045.stdout:op 2533 completed, throughput=4.99MB/sec 2021-07-31T02:13:27.427 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Xxv3V1igGcsRunV off=701826 len=118022 2021-07-31T02:13:27.431 INFO:tasks.workunit.client.0.smithi045.stdout:op 2534 completed, throughput=4.99MB/sec 2021-07-31T02:13:27.431 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-OTPTKlrWMjazS2l off=1329360 len=618692 2021-07-31T02:13:27.432 INFO:tasks.workunit.client.0.smithi045.stdout:op 2535 completed, throughput=4.99MB/sec 2021-07-31T02:13:27.432 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-8JvLVyK0Qh-O154 off=1003470 len=1661584 2021-07-31T02:13:27.435 INFO:tasks.workunit.client.0.smithi045.stdout:op 2536 completed, throughput=5MB/sec 2021-07-31T02:13:27.436 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ugKF1432057lGif off=12308679 len=1176272 2021-07-31T02:13:27.437 INFO:tasks.workunit.client.0.smithi045.stdout:op 2537 completed, throughput=5MB/sec 2021-07-31T02:13:28.438 INFO:tasks.workunit.client.0.smithi045.stdout: 492: throughput=4.99MB/sec pending data=0 2021-07-31T02:13:28.438 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-NaP6aWRKVzLUJzM off=9379557 len=756981 2021-07-31T02:13:28.439 INFO:tasks.workunit.client.0.smithi045.stdout:op 2538 completed, throughput=4.99MB/sec 2021-07-31T02:13:28.440 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-rE7GUzvKrtOsgTs off=10360546 len=245712 2021-07-31T02:13:28.440 INFO:tasks.workunit.client.0.smithi045.stdout:op 2539 completed, throughput=4.99MB/sec 2021-07-31T02:13:28.441 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Dem06xNWr0rFSoK off=255131 len=1007760 2021-07-31T02:13:28.448 INFO:tasks.workunit.client.0.smithi045.stdout:op 2540 completed, throughput=4.99MB/sec 2021-07-31T02:13:28.448 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-dWBVOXF_yu_p96d off=1353442 len=1183184 2021-07-31T02:13:28.453 INFO:tasks.workunit.client.0.smithi045.stdout:op 2541 completed, throughput=5MB/sec 2021-07-31T02:13:28.453 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-KYA5mwH6QvCyESB off=8071249 len=417462 2021-07-31T02:13:28.457 INFO:tasks.workunit.client.0.smithi045.stdout:op 2542 completed, throughput=5MB/sec 2021-07-31T02:13:28.457 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-_8LB-Bc-G-sxv3R off=9009064 len=1907319 2021-07-31T02:13:28.467 INFO:tasks.workunit.client.0.smithi045.stdout:op 2543 completed, throughput=5MB/sec 2021-07-31T02:13:28.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:28 smithi154 conmon[34553]: audit 2021-07-31T02:13:28.025981+0000 mon.b (mon.2) 198 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:13:28.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:28 smithi154 conmon[34553]: audit 2021-07-31T02:13:28.026657+0000 mon.a (mon.0) 1875 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:13:28.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:28 smithi154 conmon[34553]: audit 2021-07-31T02:13:28.046542+0000 mon.b (mon.2) 199 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:13:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:28 smithi154 conmon[34553]: audit 2021-07-31T02:13:28.047069+0000 mon.a (mon.0) 1876 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:13:28.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:28 smithi045 conmon[33655]: audit 2021-07-31T02:13:28.025981+0000 mon.b (mon.2) 2021-07-31T02:13:28.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:28 smithi045 conmon[33655]: 198 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:13:28.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:28 smithi045 conmon[33655]: audit 2021-07-31T02:13:28.026657+0000 mon.a (mon.0) 1875 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:13:28.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:28 smithi045 conmon[33655]: audit 2021-07-31T02:13:28.046542+0000 mon.b (mon.2) 199 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:13:28.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:28 smithi045 conmon[33655]: audit 2021-07-31T02:13:28.047069+0000 mon.a (mon. 2021-07-31T02:13:28.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:28 smithi045 conmon[33655]: 0) 1876 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:13:28.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:28 smithi045 conmon[39718]: audit 2021-07-31T02:13:28.025981+0000 mon.b (mon.2) 198 2021-07-31T02:13:28.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:28 smithi045 conmon[39718]: : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:13:28.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:28 smithi045 conmon[39718]: audit 2021-07-31T02:13:28.026657+0000 mon.a (mon.0) 1875 : audit [INF] 2021-07-31T02:13:28.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:28 smithi045 conmon[39718]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:13:28.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:28 smithi045 conmon[39718]: audit 2021-07-31T02:13:28.046542+0000 mon.b (mon. 2021-07-31T02:13:28.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:28 smithi045 conmon[39718]: 2) 199 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:13:28.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:28 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T02:13:28.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:28 smithi045 conmon[39718]: 02:13:28.047069+0000 mon.a (mon.0) 1876 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:13:29.468 INFO:tasks.workunit.client.0.smithi045.stdout: 493: throughput=4.99MB/sec pending data=0 2021-07-31T02:13:29.468 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Xex9FAbgfm0YMXm off=5005771 len=2053297 2021-07-31T02:13:29.470 INFO:tasks.workunit.client.0.smithi045.stdout:op 2544 completed, throughput=4.99MB/sec 2021-07-31T02:13:29.470 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-SMlsaKO2D5f9IiZ off=7611414 len=1841480 2021-07-31T02:13:29.479 INFO:tasks.workunit.client.0.smithi045.stdout:op 2545 completed, throughput=5MB/sec 2021-07-31T02:13:29.479 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-J4teSPwCewftwc- off=7209608 len=188644 2021-07-31T02:13:29.480 INFO:tasks.workunit.client.0.smithi045.stdout:op 2546 completed, throughput=5MB/sec 2021-07-31T02:13:29.480 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Y9x_lppST3XhFqc off=4829288 len=1773991 2021-07-31T02:13:29.481 INFO:tasks.workunit.client.0.smithi045.stdout:op 2547 completed, throughput=5MB/sec 2021-07-31T02:13:29.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:29 smithi154 conmon[34553]: cluster 2021-07-31T02:13:28.166508+0000 mgr.x (mgr.24542) 1120 : cluster [DBG] pgmap v1659: 9 pgs: 9 active+clean; 127 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.6 MiB/s wr, 3 op/s 2021-07-31T02:13:29.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:29 smithi154 conmon[34553]: 2021-07-31T02:13:29.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:29 smithi045 conmon[39718]: cluster 2021-07-31T02:13:28.166508+0000 mgr.x ( 2021-07-31T02:13:29.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:29 smithi045 conmon[39718]: mgr.24542) 1120 : cluster [DBG] pgmap v1659: 9 pgs: 9 active+clean; 127 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.6 MiB/s wr, 3 op/s 2021-07-31T02:13:29.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:29 smithi045 conmon[33655]: cluster 2021-07-31T02:13:28.166508+0000 2021-07-31T02:13:29.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:29 smithi045 conmon[33655]: mgr.x (mgr.24542) 1120 : cluster [DBG] pgmap v1659: 9 pgs: 9 active+clean; 127 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.6 MiB/s wr, 3 op/s 2021-07-31T02:13:30.482 INFO:tasks.workunit.client.0.smithi045.stdout: 494: throughput=4.99MB/sec pending data=0 2021-07-31T02:13:30.482 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-tTchVObJd-vdOUJ off=8203018 len=401150 2021-07-31T02:13:30.483 INFO:tasks.workunit.client.0.smithi045.stdout:op 2548 completed, throughput=4.99MB/sec 2021-07-31T02:13:30.484 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-aDxPi_BjP0PeBIM off=2040295 len=1599354 2021-07-31T02:13:30.491 INFO:tasks.workunit.client.0.smithi045.stdout:op 2549 completed, throughput=5MB/sec 2021-07-31T02:13:30.491 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-JY4WsIpyfhSi5Ip off=1173622 len=687814 2021-07-31T02:13:30.495 INFO:tasks.workunit.client.0.smithi045.stdout:op 2550 completed, throughput=5MB/sec 2021-07-31T02:13:30.495 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-qPAT3tVjNwifMvs off=3485198 len=1083542 2021-07-31T02:13:30.496 INFO:tasks.workunit.client.0.smithi045.stdout:op 2551 completed, throughput=5MB/sec 2021-07-31T02:13:30.496 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-T4RiHK1xkEYeVdP off=1143955 len=1202147 2021-07-31T02:13:30.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[34553]: cluster 2021-07-31T02:13:30.167109+0000 mgr.x (mgr.24542) 1121 : cluster [DBG] pgmap v1660: 9 pgs: 9 active+clean; 127 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 2.5 MiB/s rd, 3.9 MiB/s wr, 6 op/s 2021-07-31T02:13:30.502 INFO:tasks.workunit.client.0.smithi045.stdout:op 2552 completed, throughput=5MB/sec 2021-07-31T02:13:30.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:30 smithi045 conmon[33655]: cluster 2021-07-31T02:13:30.167109+0000 mgr.x (mgr 2021-07-31T02:13:30.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:30 smithi045 conmon[33655]: .24542) 1121 : cluster [DBG] pgmap v1660: 9 pgs: 9 active+clean; 127 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 2.5 MiB/s rd, 3.9 MiB/s wr, 6 op/s 2021-07-31T02:13:30.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:30 smithi045 conmon[39718]: cluster 2021-07-31T02:13:30.167109 2021-07-31T02:13:30.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:30 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 1121 : cluster [DBG] pgmap v1660: 9 pgs: 9 active+clean; 127 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 2.5 MiB/s rd, 3.9 MiB/s wr, 6 op/s 2021-07-31T02:13:31.003 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: debug 2021-07-31T02:13:30.719+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:13:31.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:13:31.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:13:31.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: yield 2021-07-31T02:13:31.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:13:31.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:13:31.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:13:31.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:13:31.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:13:31.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:13:31.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:13:31.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:13:31.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:13:31.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:13:31.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:13:31.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: debug 2021-07-31T02:13:30.731+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:13:31.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:13:31.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:13:31.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: yield 2021-07-31T02:13:31.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:13:31.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:13:31.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:13:31.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:13:31.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:13:31.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:13:31.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:13:31.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:13:31.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:13:31.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:13:31.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:13:31.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: debug 2021-07-31T02:13:30.756+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:13:31.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:13:31.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:13:31.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: yield 2021-07-31T02:13:31.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:13:31.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:13:31.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:13:31.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:13:31.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:13:31.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:13:31.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:13:31.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:13:31.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:13:31.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:13:31.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:13:31.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: debug 2021-07-31T02:13:30.757+0000 7fcb9295f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:13:31.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:13:31.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:13:31.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: yield 2021-07-31T02:13:31.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:13:31.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:13:31.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:13:31.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:13:31.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:13:31.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:13:31.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:13:31.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:13:31.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:13:31.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:13:31.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:13:31.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: 2021-07-31T02:13:31.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:13:31.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: 2021-07-31T02:13:31.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:13:31.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:13:31.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:13:31.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:13:31.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:13:31.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:13:31.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:13:31.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:13:31.035 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:13:31.035 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:13:31.035 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:13:31.036 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:13:31.036 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:13:31.037 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:13:31.037 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:13:31.038 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:13:31.038 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:13:31.039 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:13:31.039 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: raise 2021-07-31T02:13:31.039 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:13:31.040 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:13:31.040 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:13:31.041 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:13:31.041 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:13:31.042 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: debug 2021-07-31T02:13:30.765+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:13:31.042 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:13:31.042 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:13:31.043 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: yield 2021-07-31T02:13:31.043 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:13:31.044 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:13:31.044 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:13:31.045 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:13:31.045 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:13:31.045 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:13:31.046 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:13:31.046 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:13:31.047 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:13:31.047 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:13:31.048 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:13:31.048 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: debug 2021-07-31T02:13:30.766+0000 7fcb93160700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:13:31.049 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:13:31.049 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:13:31.049 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: yield 2021-07-31T02:13:31.050 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:13:31.050 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:13:31.051 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:13:31.051 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:13:31.052 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:13:31.052 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:13:31.052 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:13:31.053 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:13:31.053 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:13:31.054 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:13:31.054 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:13:31.055 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: 2021-07-31T02:13:31.055 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:13:31.055 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: 2021-07-31T02:13:31.056 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:13:31.056 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:13:31.057 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:13:31.057 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:13:31.058 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:13:31.058 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:13:31.059 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:13:31.059 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:13:31.059 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:13:31.060 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:13:31.060 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:13:31.061 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:13:31.061 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:13:31.062 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:13:31.062 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:13:31.063 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:13:31.063 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:13:31.063 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:13:31.064 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: raise 2021-07-31T02:13:31.064 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:13:31.065 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:13:31.065 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:13:31.065 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:13:31.066 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:13:30 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:13:31.502 INFO:tasks.workunit.client.0.smithi045.stdout: 495: throughput=4.99MB/sec pending data=0 2021-07-31T02:13:31.502 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-x9YKLrON4JEZQD3 off=5211557 len=1629091 2021-07-31T02:13:31.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: audit 2021-07-31T02:13:30.674683+0000 mon.b (mon.2) 200 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:13:31.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: audit 2021-07-31T02:13:30.676398+0000 mon.b (mon.2) 201 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:13:31.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: audit 2021-07-31T02:13:30.677263+0000 mon.b 2021-07-31T02:13:31.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: (mon.2) 202 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:13:31.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:13:30.678005+0000 mgr.x (mgr.24542) 1122 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:13:31.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: cephadm 2021-07 2021-07-31T02:13:31.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: -31T02:13:30.720539+0000 mgr.x 2021-07-31T02:13:31.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: (mgr.24542) 1123 : cephadm [ERR] 2021-07-31T02:13:31.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:13:31.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:13:31.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: yield 2021-07-31T02:13:31.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:13:31.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:13:31.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:13:31.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:13:31.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:13:31.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:13:31.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:13:31.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:13:31.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:13:31.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: task.result() 2021-07-31T02:13:31.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:13:31.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:13:30.721159+0000 mgr.x (mgr.24542) 1124 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:13:31.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:13:30.732246+0000 mgr.x ( 2021-07-31T02:13:31.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: mgr.24542) 1125 : cephadm [ERR] 2021-07-31T02:13:31.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:13:31.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:13:31.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: yield 2021-07-31T02:13:31.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:13:31.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:13:31.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:13:31.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:13:31.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:13:31.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:13:31.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:13:31.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:13:31.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:13:31.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: task.result() 2021-07-31T02:13:31.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:13:31.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:13:30.733111+0000 mgr.x (mgr.24542) 1126 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:13:31.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:13:30.757744+0000 mgr.x (mgr.24542) 1127 : cephadm [ERR] 2021-07-31T02:13:31.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: 2021-07-31T02:13:31.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:13:31.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:13:31.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: yield 2021-07-31T02:13:31.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:13:31.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:13:31.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:13:31.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:13:31.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:13:31.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:13:31.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:13:31.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:13:31.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:13:31.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: task.result() 2021-07-31T02:13:31.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:13:31.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:13:30.758568+0000 mgr.x (mgr.24542) 1128 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:13:31.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:13:31.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:13:31.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: yield 2021-07-31T02:13:31.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:13:31.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:13:31.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:13:31.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:13:31.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:13:31.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:13:31.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:13:31.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:13:31.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:13:31.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: task.result() 2021-07-31T02:13:31.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:13:31.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: 2021-07-31T02:13:31.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:13:31.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: 2021-07-31T02:13:31.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:13:31.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:13:31.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:13:31.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:13:31.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:13:31.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:13:31.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:13:31.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:13:31.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:13:31.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:13:31.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:13:31.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:13:31.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:13:31.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:13:31.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:13:31.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:13:31.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:13:31.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:13:31.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: raise 2021-07-31T02:13:31.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:13:31.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:13:31.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:13:31.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:13:31.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:13:31.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:13:30.766702+0000 mgr.x (mgr.24542) 1129 : cephadm 2021-07-31T02:13:31.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: [ERR] 2021-07-31T02:13:31.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:13:31.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:13:31.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: yield 2021-07-31T02:13:31.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:13:31.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:13:31.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:13:31.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:13:31.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:13:31.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:13:31.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:13:31.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:13:31.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:13:31.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: task.result() 2021-07-31T02:13:31.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:13:31.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:13:30.767277+0000 mgr.x (mgr.24542) 1130 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:13:31.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:13:31.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:13:31.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: yield 2021-07-31T02:13:31.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:13:31.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:13:31.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:13:31.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:13:31.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:13:31.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:13:31.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:13:31.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:13:31.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:13:31.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: task.result() 2021-07-31T02:13:31.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:13:31.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: 2021-07-31T02:13:31.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:13:31.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: 2021-07-31T02:13:31.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:13:31.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:13:31.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:13:31.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:13:31.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:13:31.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:13:31.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:13:31.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:13:31.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:13:31.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:13:31.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:13:31.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:13:31.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:13:31.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:13:31.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:13:31.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:13:31.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:13:31.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:13:31.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: raise 2021-07-31T02:13:31.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:13:31.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:13:31.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:13:31.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:13:31.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:31 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:13:31.541 INFO:tasks.workunit.client.0.smithi045.stdout:op 2553 completed, throughput=4.99MB/sec 2021-07-31T02:13:31.541 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ppGP93eBQo7XzB1 off=6663974 len=533123 2021-07-31T02:13:31.541 INFO:tasks.workunit.client.0.smithi045.stdout:op 2554 completed, throughput=4.99MB/sec 2021-07-31T02:13:31.542 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Qge01bP-hbHY3Pn off=8821964 len=182378 2021-07-31T02:13:31.542 INFO:tasks.workunit.client.0.smithi045.stdout:op 2555 completed, throughput=5MB/sec 2021-07-31T02:13:31.542 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Je7UlzrIdJ4ZX4T off=4517091 len=40230 2021-07-31T02:13:31.542 INFO:tasks.workunit.client.0.smithi045.stdout:op 2556 completed, throughput=5MB/sec 2021-07-31T02:13:31.543 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-TjMkotPJ1oaVzlo off=1572772 len=1688082 2021-07-31T02:13:31.543 INFO:tasks.workunit.client.0.smithi045.stdout:op 2557 completed, throughput=5MB/sec 2021-07-31T02:13:31.543 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-GeVi2lOmcaorRbG off=7534245 len=721714 2021-07-31T02:13:31.543 INFO:tasks.workunit.client.0.smithi045.stdout:op 2558 completed, throughput=5MB/sec 2021-07-31T02:13:31.543 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-vrg1L2IICKPle1f off=0 len=1579066 2021-07-31T02:13:31.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: audit 2021-07-31T02:13:30.674683+0000 mon.b (mon 2021-07-31T02:13:31.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: .2) 200 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:13:31.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: audit 2021-07-31T02: 2021-07-31T02:13:31.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: 13:30.676398+0000 mon.b (mon.2) 201 : audit [DBG] 2021-07-31T02:13:31.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:13:31.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: audit 2021-07-31T02:13 2021-07-31T02:13:31.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: :30.677263+0000 mon.b (mon.2 2021-07-31T02:13:31.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: ) 202 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:13:31.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: cephadm 2021 2021-07-31T02:13:31.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: -07-31T02:13:30.678005 2021-07-31T02:13:31.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 1122 : cephadm 2021-07-31T02:13:31.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:13:31.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: cephadm 2021-07- 2021-07-31T02:13:31.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: 31T02:13:30.720539+0000 2021-07-31T02:13:31.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: mgr.x (mgr.24542) 1123 : cephadm 2021-07-31T02:13:31.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: [ERR] 2021-07-31T02:13:31.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:13:31.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:13:31.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: yield 2021-07-31T02:13:31.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:13:31.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:13:31.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:13:31.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:13:31.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:13:31.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:13:31.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:13:31.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:13:31.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:13:31.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: task.result() 2021-07-31T02:13:31.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:13:31.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:13:30.721159+0000 mgr.x (mgr.24542) 1124 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:13:31.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: cephadm 2021-07- 2021-07-31T02:13:31.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: 31T02:13:30.732246+0000 mgr.x (mgr. 2021-07-31T02:13:31.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: 24542) 1125 : cephadm [ERR] 2021-07-31T02:13:31.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:13:31.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:13:31.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: yield 2021-07-31T02:13:31.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:13:31.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:13:31.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:13:31.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:13:31.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:13:31.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:13:31.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:13:31.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:13:31.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:13:31.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: task.result() 2021-07-31T02:13:31.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:13:31.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:13:30.733111+0000 mgr.x (mgr.24542) 1126 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:13:31.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:13: 2021-07-31T02:13:31.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: 30.757744+0000 mgr.x (mgr.24542) 1127 : cephadm 2021-07-31T02:13:31.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: [ERR] 2021-07-31T02:13:31.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:13:31.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:13:31.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: yield 2021-07-31T02:13:31.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:13:31.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:13:31.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:13:31.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:13:31.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:13:31.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:13:31.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:13:31.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:13:31.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:13:31.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: task.result() 2021-07-31T02:13:31.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:13:31.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:13:30.758568+0000 mgr.x (mgr.24542) 1128 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:13:31.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:13:31.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:13:31.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: yield 2021-07-31T02:13:31.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:13:31.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:13:31.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:13:31.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:13:31.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:13:31.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:13:31.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:13:31.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:13:31.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:13:31.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: task.result() 2021-07-31T02:13:31.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:13:31.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: 2021-07-31T02:13:31.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: audit 2021-07-31T02:13:30.674683+0000 2021-07-31T02:13:31.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: mon.b (mon.2) 200 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:13:31.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: audit 2021-07-31T02:13:30. 2021-07-31T02:13:31.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: 676398+0000 mon.b (mon.2) 201 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:13:31.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: audit 2021-07-31T02:13:30.677263 2021-07-31T02:13:31.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: +0000 mon.b (mon.2) 202 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:13:31.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: cephadm 2021-07 2021-07-31T02:13:31.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: -31T02:13:30.678005+0000 mgr.x 2021-07-31T02:13:31.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: (mgr.24542) 1122 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:13:31.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: cephadm 2021-07-31 2021-07-31T02:13:31.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: T02:13:30.720539+0000 2021-07-31T02:13:31.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: mgr.x (mgr.24542) 1123 2021-07-31T02:13:31.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: : cephadm [ERR] 2021-07-31T02:13:31.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:13:31.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:13:31.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: yield 2021-07-31T02:13:31.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:13:31.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:13:31.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:13:31.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:13:31.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:13:31.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:13:31.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:13:31.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:13:31.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:13:31.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: task.result() 2021-07-31T02:13:31.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:13:31.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:13:30.721159+0000 mgr.x (mgr.24542) 1124 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:13:31.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:13: 2021-07-31T02:13:31.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: 30.732246+0000 mgr.x (mgr.24542) 2021-07-31T02:13:31.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: 1125 : cephadm [ERR] 2021-07-31T02:13:31.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:13:31.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:13:31.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: yield 2021-07-31T02:13:31.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:13:31.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:13:31.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:13:31.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:13:31.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:13:31.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:13:31.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: 2021-07-31T02:13:31.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:13:31.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:13:31.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:13:31.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:13:31.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:13:31.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:13:31.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:13:31.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:13:31.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:13:31.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:13:31.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:13:31.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:13:31.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:13:31.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:13:31.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:13:31.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:13:31.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:13:31.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:13:31.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: raise 2021-07-31T02:13:31.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:13:31.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:13:31.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:13:31.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:13:31.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:13:31.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:13:30.766702+0000 mgr.x (mgr.24542) 1129 : cephadm [ERR] 2021-07-31T02:13:31.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:13:31.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:13:31.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: yield 2021-07-31T02:13:31.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:13:31.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:13:31.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:13:31.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:13:31.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:13:31.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:13:31.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:13:31.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:13:31.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:13:31.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: task.result() 2021-07-31T02:13:31.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:13:31.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:13:31.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:13:31.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:13:31.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: task.result() 2021-07-31T02:13:31.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:13:31.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:13:30.733111+0000 mgr.x (mgr.24542) 1126 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:13:31.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: cephadm 2021-07-31T 2021-07-31T02:13:31.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: 02:13:30.757744+0000 mgr.x (mgr.24542 2021-07-31T02:13:31.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: ) 1127 : cephadm [ERR] 2021-07-31T02:13:31.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:13:31.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:13:31.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: yield 2021-07-31T02:13:31.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:13:31.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:13:31.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:13:31.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:13:31.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:13:31.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:13:31.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:13:31.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:13:31.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:13:31.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: task.result() 2021-07-31T02:13:31.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:13:31.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:13:30.758568+0000 mgr.x (mgr.24542) 1128 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:13:31.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:13:31.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:13:31.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: yield 2021-07-31T02:13:31.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:13:31.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:13:31.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:13:31.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:13:31.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:13:31.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:13:31.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:13:31.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:13:31.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:13:31.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: task.result() 2021-07-31T02:13:31.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:13:31.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: 2021-07-31T02:13:31.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:13:31.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: 2021-07-31T02:13:31.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:13:31.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:13:30.767277+0000 mgr.x (mgr.24542) 1130 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:13:31.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:13:31.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:13:31.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: yield 2021-07-31T02:13:31.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:13:31.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:13:31.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:13:31.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:13:31.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:13:31.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:13:31.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:13:31.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:13:31.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:13:31.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: task.result() 2021-07-31T02:13:31.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:13:31.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: 2021-07-31T02:13:31.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:13:31.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: 2021-07-31T02:13:31.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:13:31.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:13:31.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:13:31.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:13:31.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:13:31.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:13:31.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:13:31.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:13:31.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:13:31.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:13:31.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:13:31.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:13:31.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:13:31.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:13:31.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:13:31.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:13:31.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:13:31.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:13:31.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: raise 2021-07-31T02:13:31.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:13:31.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:13:31.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:13:31.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:13:31.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:13:31.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:13:31.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:13:31.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:13:31.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:13:31.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:13:31.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:13:31.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:13:31.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:13:31.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:13:31.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:13:31.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:13:31.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:13:31.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:13:31.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:13:31.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:13:31.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:13:31.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:13:31.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:13:31.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: raise 2021-07-31T02:13:31.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:13:31.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:13:31.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:13:31.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:13:31.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:13:31.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:13:31.608 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: 2021-07-31T02:13:30.766702+0000 mgr.x (mgr.24542) 1129 : cephadm [ERR] 2021-07-31T02:13:31.608 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:13:31.608 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:13:31.608 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: yield 2021-07-31T02:13:31.608 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:13:31.609 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:13:31.609 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:13:31.609 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:13:31.609 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:13:31.609 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:13:31.610 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:13:31.610 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:13:31.610 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:13:31.610 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: task.result() 2021-07-31T02:13:31.611 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:13:31.611 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:13:30.767277+0000 mgr.x (mgr.24542) 1130 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:13:31.611 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:13:31.611 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:13:31.611 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: yield 2021-07-31T02:13:31.612 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:13:31.612 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:13:31.612 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:13:31.612 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:13:31.613 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:13:31.613 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:13:31.613 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:13:31.613 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:13:31.613 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:13:31.614 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: task.result() 2021-07-31T02:13:31.614 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:13:31.614 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: 2021-07-31T02:13:31.614 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:13:31.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: 2021-07-31T02:13:31.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:13:31.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:13:31.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:13:31.616 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:13:31.616 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:13:31.616 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:13:31.616 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:13:31.616 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:13:31.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:13:31.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:13:31.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:13:31.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:13:31.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:13:31.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:13:31.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:13:31.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:13:31.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:13:31.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:13:31.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: raise 2021-07-31T02:13:31.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:13:31.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:13:31.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:13:31.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:13:31.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:31 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:13:31.620 INFO:tasks.workunit.client.0.smithi045.stdout:op 2559 completed, throughput=5MB/sec 2021-07-31T02:13:32.528 INFO:tasks.workunit.client.0.smithi045.stdout: 496: throughput=4.99MB/sec pending data=0 2021-07-31T02:13:32.529 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-QmA8NSDdPuSk5Va off=8815786 len=2068825 2021-07-31T02:13:32.532 INFO:tasks.workunit.client.0.smithi045.stdout:op 2560 completed, throughput=5MB/sec 2021-07-31T02:13:32.532 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-2n36X5hmdr77O_t off=217615 len=1688216 2021-07-31T02:13:32.540 INFO:tasks.workunit.client.0.smithi045.stdout:op 2561 completed, throughput=5MB/sec 2021-07-31T02:13:32.540 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-2AnUizlV5Fq4tcl off=12942449 len=2004670 2021-07-31T02:13:32.542 INFO:tasks.workunit.client.0.smithi045.stdout:op 2562 completed, throughput=5MB/sec 2021-07-31T02:13:33.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:33 smithi154 conmon[34553]: cluster 2021-07-31T02:13:32.167621+0000 mgr.x (mgr.24542) 1131 : cluster [DBG] pgmap v1661: 9 pgs: 9 active+clean; 127 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:13:33.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:33 smithi045 conmon[39718]: cluster 2021-07-31T02:13:32.167621+0000 mgr.x (mgr 2021-07-31T02:13:33.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:33 smithi045 conmon[39718]: .24542) 1131 : cluster [DBG] pgmap v1661: 9 pgs: 9 active+clean; 127 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:13:33.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:33 smithi045 conmon[33655]: cluster 2021-07-31T02:13:32.167621+0000 mgr.x (mgr.24542) 1131 : cluster 2021-07-31T02:13:33.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:33 smithi045 conmon[33655]: [DBG] pgmap v1661: 9 pgs: 9 active+clean; 127 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:13:33.542 INFO:tasks.workunit.client.0.smithi045.stdout: 497: throughput=4.99MB/sec pending data=0 2021-07-31T02:13:33.543 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-sYBb75ilXOFz_7D off=18211160 len=388698 2021-07-31T02:13:33.547 INFO:tasks.workunit.client.0.smithi045.stdout:op 2563 completed, throughput=4.99MB/sec 2021-07-31T02:13:33.547 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-gk2tG9_1JzwWPAD off=20044602 len=1933442 2021-07-31T02:13:33.550 INFO:tasks.workunit.client.0.smithi045.stdout:op 2564 completed, throughput=5MB/sec 2021-07-31T02:13:33.550 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Br1_QoG71QCKc6m off=15117882 len=948793 2021-07-31T02:13:33.555 INFO:tasks.workunit.client.0.smithi045.stdout:op 2565 completed, throughput=5MB/sec 2021-07-31T02:13:33.556 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-U5O19p5vJb_iFEr off=6380706 len=1521876 2021-07-31T02:13:33.558 INFO:tasks.workunit.client.0.smithi045.stdout:op 2566 completed, throughput=5MB/sec 2021-07-31T02:13:34.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:13:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:13:33.553Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:13:34.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:13:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:13:33.553Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:13:34.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:13:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:13:33.553Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:13:34.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:34 smithi154 conmon[34553]: cluster 2021-07-31T02:13:34.168114+0000 mgr.x (mgr.24542) 1132 : cluster [DBG] pgmap v1662: 9 pgs: 9 active+clean; 127 GiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 3.1 MiB/s rd, 3.3 MiB/s wr, 6 op/s 2021-07-31T02:13:34.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:34 smithi154 conmon[34553]: 2021-07-31T02:13:34.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:34 smithi045 conmon[33655]: cluster 2021-07-31T02:13:34.168114+0000 mgr.x (mgr.24542) 1132 : 2021-07-31T02:13:34.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:34 smithi045 conmon[33655]: cluster [DBG] pgmap v1662: 9 pgs: 9 active+clean; 127 GiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 3.1 MiB/s rd, 3.3 MiB/s wr, 6 op/s 2021-07-31T02:13:34.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:34 smithi045 conmon[39718]: cluster 2021-07-31T02:13:34.168114+0000 mgr.x ( 2021-07-31T02:13:34.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:34 smithi045 conmon[39718]: mgr.24542) 1132 : cluster [DBG] pgmap v1662: 9 pgs: 9 active+clean; 127 GiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 3.1 MiB/s rd, 3.3 MiB/s wr, 6 op/s 2021-07-31T02:13:34.558 INFO:tasks.workunit.client.0.smithi045.stdout: 498: throughput=4.99MB/sec pending data=0 2021-07-31T02:13:34.559 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-m-7LLhtNbDzMp32 off=3316356 len=1667143 2021-07-31T02:13:34.562 INFO:tasks.workunit.client.0.smithi045.stdout:op 2567 completed, throughput=5MB/sec 2021-07-31T02:13:34.563 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-BPla3t4M_3bOFYQ off=2732455 len=46060 2021-07-31T02:13:34.563 INFO:tasks.workunit.client.0.smithi045.stdout:op 2568 completed, throughput=5MB/sec 2021-07-31T02:13:34.564 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-YDJabmycn9lWBEm off=2119294 len=1433099 2021-07-31T02:13:34.569 INFO:tasks.workunit.client.0.smithi045.stdout:op 2569 completed, throughput=5MB/sec 2021-07-31T02:13:34.569 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-7-2FprKW7EzAVoy off=11599054 len=1598434 2021-07-31T02:13:34.578 INFO:tasks.workunit.client.0.smithi045.stdout:op 2570 completed, throughput=5MB/sec 2021-07-31T02:13:35.578 INFO:tasks.workunit.client.0.smithi045.stdout: 499: throughput=4.99MB/sec pending data=0 2021-07-31T02:13:35.579 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-c7Wbwhco_HwKRIo off=16946340 len=1631002 2021-07-31T02:13:35.588 INFO:tasks.workunit.client.0.smithi045.stdout:op 2571 completed, throughput=4.99MB/sec 2021-07-31T02:13:35.588 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-sHTQ2VeFAJrvF0s off=3833363 len=2096376 2021-07-31T02:13:35.592 INFO:tasks.workunit.client.0.smithi045.stdout:op 2572 completed, throughput=5MB/sec 2021-07-31T02:13:35.592 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-PtuOIImT2uPRBLH off=6681664 len=11541 2021-07-31T02:13:35.594 INFO:tasks.workunit.client.0.smithi045.stdout:op 2573 completed, throughput=5MB/sec 2021-07-31T02:13:35.595 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-yYg-BVW_jkHEmhy off=17169444 len=477921 2021-07-31T02:13:35.599 INFO:tasks.workunit.client.0.smithi045.stdout:op 2574 completed, throughput=5MB/sec 2021-07-31T02:13:35.599 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-9pKFxUaXNHa8mqd off=6057948 len=1730067 2021-07-31T02:13:35.605 INFO:tasks.workunit.client.0.smithi045.stdout:op 2575 completed, throughput=5MB/sec 2021-07-31T02:13:36.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:36 smithi154 conmon[34553]: cluster 2021-07-31T02:13:36.168506+0000 mgr.x (mgr.24542) 1133 : cluster [DBG] pgmap v1663: 9 pgs: 9 active+clean; 127 GiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2021-07-31T02:13:36.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:36 smithi045 conmon[39718]: cluster 2021-07-31T02:13:36.168506+0000 mgr.x (mgr 2021-07-31T02:13:36.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:36 smithi045 conmon[39718]: .24542) 1133 : cluster [DBG] pgmap v1663: 9 pgs: 9 active+clean; 127 GiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2021-07-31T02:13:36.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:36 smithi045 conmon[33655]: cluster 2021-07-31T02:13:36. 2021-07-31T02:13:36.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:36 smithi045 conmon[33655]: 168506+0000 mgr.x (mgr.24542) 1133 : cluster [DBG] pgmap v1663: 9 pgs: 9 active+clean; 127 GiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2021-07-31T02:13:36.605 INFO:tasks.workunit.client.0.smithi045.stdout: 500: throughput=4.99MB/sec pending data=0 2021-07-31T02:13:36.606 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-fzcjaWXCqaaprmQ off=10459921 len=1105656 2021-07-31T02:13:36.608 INFO:tasks.workunit.client.0.smithi045.stdout:op 2576 completed, throughput=4.99MB/sec 2021-07-31T02:13:36.608 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-OBggPb1XR4N1keG off=5920258 len=747247 2021-07-31T02:13:36.613 INFO:tasks.workunit.client.0.smithi045.stdout:op 2577 completed, throughput=5MB/sec 2021-07-31T02:13:36.614 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-46p1NP229JO0Fm8 off=14942343 len=1847866 2021-07-31T02:13:36.617 INFO:tasks.workunit.client.0.smithi045.stdout:op 2578 completed, throughput=5MB/sec 2021-07-31T02:13:36.618 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-2hvDZq1Qo7a07ia off=1496081 len=775161 2021-07-31T02:13:36.622 INFO:tasks.workunit.client.0.smithi045.stdout:op 2579 completed, throughput=5MB/sec 2021-07-31T02:13:37.622 INFO:tasks.workunit.client.0.smithi045.stdout: 501: throughput=4.99MB/sec pending data=0 2021-07-31T02:13:37.623 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-HCCGN3-IiQ1K-XG off=4956377 len=2091414 2021-07-31T02:13:37.632 INFO:tasks.workunit.client.0.smithi045.stdout:op 2580 completed, throughput=4.99MB/sec 2021-07-31T02:13:37.632 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-lXDYcSVG9I1ZQIf off=79430 len=1667319 2021-07-31T02:13:37.635 INFO:tasks.workunit.client.0.smithi045.stdout:op 2581 completed, throughput=5MB/sec 2021-07-31T02:13:37.635 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Qq7V1fpaUGJP98- off=6743390 len=821485 2021-07-31T02:13:37.640 INFO:tasks.workunit.client.0.smithi045.stdout:op 2582 completed, throughput=5MB/sec 2021-07-31T02:13:37.640 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-BiDHBfYCbDj_9oG off=10013507 len=1021271 2021-07-31T02:13:37.645 INFO:tasks.workunit.client.0.smithi045.stdout:op 2583 completed, throughput=5MB/sec 2021-07-31T02:13:38.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:38 smithi154 conmon[34553]: cluster 2021-07-31T02:13:38.168951+0000 mgr.x (mgr.24542) 1134 : cluster [DBG] pgmap v1664: 9 pgs: 9 active+clean; 127 GiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2021-07-31T02:13:38.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:38 smithi045 conmon[33655]: cluster 2021-07-31T02:13:38.168951+0000 2021-07-31T02:13:38.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:38 smithi045 conmon[33655]: mgr.x (mgr.24542) 1134 : cluster [DBG] pgmap v1664: 9 pgs: 9 active+clean; 127 GiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2021-07-31T02:13:38.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:38 smithi045 conmon[39718]: cluster 2021-07-31T02:13:38.168951+0000 mgr.x (mgr 2021-07-31T02:13:38.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:38 smithi045 conmon[39718]: .24542) 1134 : cluster [DBG] pgmap v1664: 9 pgs: 9 active+clean; 127 GiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2021-07-31T02:13:38.645 INFO:tasks.workunit.client.0.smithi045.stdout: 502: throughput=4.99MB/sec pending data=0 2021-07-31T02:13:38.646 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-R-Jn9o9gO3-unJv off=20922391 len=1018838 2021-07-31T02:13:38.648 INFO:tasks.workunit.client.0.smithi045.stdout:op 2584 completed, throughput=4.99MB/sec 2021-07-31T02:13:38.648 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-7GiCrKcLIyY1ENG off=5287076 len=1113 2021-07-31T02:13:38.650 INFO:tasks.workunit.client.0.smithi045.stdout:op 2585 completed, throughput=4.99MB/sec 2021-07-31T02:13:38.651 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-xyQnrt2-FC-YxBU off=17288783 len=578967 2021-07-31T02:13:38.654 INFO:tasks.workunit.client.0.smithi045.stdout:op 2586 completed, throughput=4.99MB/sec 2021-07-31T02:13:38.655 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-CGETB-uWc7Qj9w3 off=1493883 len=1660973 2021-07-31T02:13:38.658 INFO:tasks.workunit.client.0.smithi045.stdout:op 2587 completed, throughput=5MB/sec 2021-07-31T02:13:38.658 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-dIdykDpxd2KY4xZ off=12159609 len=1673239 2021-07-31T02:13:38.665 INFO:tasks.workunit.client.0.smithi045.stdout:op 2588 completed, throughput=5MB/sec 2021-07-31T02:13:39.666 INFO:tasks.workunit.client.0.smithi045.stdout: 503: throughput=4.99MB/sec pending data=0 2021-07-31T02:13:39.666 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-FzpqTqiqKfMo5sR off=1758886 len=2013319 2021-07-31T02:13:39.668 INFO:tasks.workunit.client.0.smithi045.stdout:op 2589 completed, throughput=4.99MB/sec 2021-07-31T02:13:39.669 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-5dMyR7SPVILPcDc off=3038200 len=1713982 2021-07-31T02:13:39.670 INFO:tasks.workunit.client.0.smithi045.stdout:op 2590 completed, throughput=5MB/sec 2021-07-31T02:13:39.670 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-BK0EtxhLU1eXXs1 off=5357077 len=1951594 2021-07-31T02:13:39.671 INFO:tasks.workunit.client.0.smithi045.stdout:op 2591 completed, throughput=5MB/sec 2021-07-31T02:13:40.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:40 smithi154 conmon[34553]: cluster 2021-07-31T02:13:40.169498+0000 mgr.x (mgr.24542) 1135 : cluster [DBG] pgmap v1665: 9 pgs: 9 active+clean; 127 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 3.0 MiB/s rd, 3.3 MiB/s wr, 6 op/s 2021-07-31T02:13:40.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:40 smithi045 conmon[33655]: cluster 2021-07-31T02:13:40.169498+0000 mgr.x (mgr.24542) 1135 : cluster [DBG] pgmap v1665: 9 pgs: 9 active+clean; 127 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 3.0 MiB/s rd, 3.3 MiB/s wr, 6 op/s 2021-07-31T02:13:40.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:40 smithi045 conmon[39718]: cluster 2021-07-31T02:13:40.169498+0000 mgr.x (mgr 2021-07-31T02:13:40.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:40 smithi045 conmon[39718]: .24542) 1135 : cluster [DBG] pgmap v1665: 9 pgs: 9 active+clean; 127 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 3.0 MiB/s rd, 3.3 MiB/s wr, 6 op/s 2021-07-31T02:13:40.672 INFO:tasks.workunit.client.0.smithi045.stdout: 504: throughput=4.99MB/sec pending data=0 2021-07-31T02:13:40.672 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-w4EfWZe_NLAtvKO off=6119958 len=137703 2021-07-31T02:13:40.674 INFO:tasks.workunit.client.0.smithi045.stdout:op 2592 completed, throughput=4.99MB/sec 2021-07-31T02:13:40.675 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-CcvQblH8ePgs6b7 off=9081390 len=86097 2021-07-31T02:13:40.675 INFO:tasks.workunit.client.0.smithi045.stdout:op 2593 completed, throughput=4.99MB/sec 2021-07-31T02:13:40.676 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-2o86vmBv3XMo_gP off=735998 len=1728547 2021-07-31T02:13:40.677 INFO:tasks.workunit.client.0.smithi045.stdout:op 2594 completed, throughput=5MB/sec 2021-07-31T02:13:40.677 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-UIm4_exNHAy5Pi2 off=8222016 len=1575805 2021-07-31T02:13:40.680 INFO:tasks.workunit.client.0.smithi045.stdout:op 2595 completed, throughput=5MB/sec 2021-07-31T02:13:40.680 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-G0nqrhR6YU9cSt0 off=253931 len=824697 2021-07-31T02:13:40.685 INFO:tasks.workunit.client.0.smithi045.stdout:op 2596 completed, throughput=5MB/sec 2021-07-31T02:13:40.686 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-G0nqrhR6YU9cSt0 off=1529635 len=372678 2021-07-31T02:13:40.688 INFO:tasks.workunit.client.0.smithi045.stdout:op 2597 completed, throughput=5MB/sec 2021-07-31T02:13:41.689 INFO:tasks.workunit.client.0.smithi045.stdout: 505: throughput=4.99MB/sec pending data=0 2021-07-31T02:13:41.689 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-t4_KYXN6wRiMZdG off=8831501 len=75947 2021-07-31T02:13:41.692 INFO:tasks.workunit.client.0.smithi045.stdout:op 2598 completed, throughput=4.99MB/sec 2021-07-31T02:13:41.692 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-GRuKf1hpDjgHrkR off=10085435 len=1998293 2021-07-31T02:13:41.694 INFO:tasks.workunit.client.0.smithi045.stdout:op 2599 completed, throughput=4.99MB/sec 2021-07-31T02:13:41.694 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-lHe-PQVmQWCW85l off=17085608 len=214743 2021-07-31T02:13:41.695 INFO:tasks.workunit.client.0.smithi045.stdout:op 2600 completed, throughput=4.99MB/sec 2021-07-31T02:13:41.695 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-QMBtm_LuFX95rMI off=1763639 len=877892 2021-07-31T02:13:41.697 INFO:tasks.workunit.client.0.smithi045.stdout:op 2601 completed, throughput=5MB/sec 2021-07-31T02:13:41.697 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-RQYMOIeGdR2u3sR off=6697341 len=608781 2021-07-31T02:13:41.702 INFO:tasks.workunit.client.0.smithi045.stdout:op 2602 completed, throughput=5MB/sec 2021-07-31T02:13:41.702 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-2QfcaYaJ5cXk5Xk off=6320171 len=285686 2021-07-31T02:13:41.705 INFO:tasks.workunit.client.0.smithi045.stdout:op 2603 completed, throughput=5MB/sec 2021-07-31T02:13:41.705 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-nrjZ60iroJy9Ltq off=1371579 len=2096650 2021-07-31T02:13:41.709 INFO:tasks.workunit.client.0.smithi045.stdout:op 2604 completed, throughput=5MB/sec 2021-07-31T02:13:42.709 INFO:tasks.workunit.client.0.smithi045.stdout: 506: throughput=4.99MB/sec pending data=0 2021-07-31T02:13:42.710 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-oam-ZrkCuXsW_qs off=3070798 len=945134 2021-07-31T02:13:42.715 INFO:tasks.workunit.client.0.smithi045.stdout:op 2605 completed, throughput=4.99MB/sec 2021-07-31T02:13:42.715 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-_vWMMpM-DxJ--0r off=5777214 len=1493696 2021-07-31T02:13:42.717 INFO:tasks.workunit.client.0.smithi045.stdout:op 2606 completed, throughput=5MB/sec 2021-07-31T02:13:42.717 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-4iP4QsqFa_dnpDa off=4278658 len=337369 2021-07-31T02:13:42.721 INFO:tasks.workunit.client.0.smithi045.stdout:op 2607 completed, throughput=5MB/sec 2021-07-31T02:13:42.721 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-mGvI6qOdUMfoXGo off=1281564 len=1724223 2021-07-31T02:13:42.723 INFO:tasks.workunit.client.0.smithi045.stdout:op 2608 completed, throughput=5MB/sec 2021-07-31T02:13:43.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:43 smithi154 conmon[34553]: cluster 2021-07-31T02:13:42.170111+0000 mgr.x (mgr.24542) 1136 : cluster [DBG] 2021-07-31T02:13:43.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:43 smithi154 conmon[34553]: pgmap v1666: 9 pgs: 9 active+clean; 127 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 3 op/s 2021-07-31T02:13:43.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:43 smithi045 conmon[33655]: cluster 2021-07-31T02:13:42.170111+0000 mgr.x (mgr.24542) 1136 : 2021-07-31T02:13:43.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:43 smithi045 conmon[33655]: cluster [DBG] pgmap v1666: 9 pgs: 9 active+clean; 127 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 3 op/s 2021-07-31T02:13:43.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:43 smithi045 conmon[39718]: cluster 2021-07-31T02:13:42.170111+0000 mgr.x (mgr.24542 2021-07-31T02:13:43.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:43 smithi045 conmon[39718]: ) 1136 : cluster [DBG] pgmap v1666: 9 pgs: 9 active+clean; 127 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 3 op/s 2021-07-31T02:13:43.723 INFO:tasks.workunit.client.0.smithi045.stdout: 507: throughput=4.99MB/sec pending data=0 2021-07-31T02:13:43.723 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-skP3LgUSHqurg1T off=6969209 len=755546 2021-07-31T02:13:43.725 INFO:tasks.workunit.client.0.smithi045.stdout:op 2609 completed, throughput=4.99MB/sec 2021-07-31T02:13:43.725 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Qs-XFjZPiHYZK25 off=16100480 len=430091 2021-07-31T02:13:43.728 INFO:tasks.workunit.client.0.smithi045.stdout:op 2610 completed, throughput=4.99MB/sec 2021-07-31T02:13:43.728 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-WmCEZOpFptVnsXA off=5569265 len=633203 2021-07-31T02:13:43.729 INFO:tasks.workunit.client.0.smithi045.stdout:op 2611 completed, throughput=4.99MB/sec 2021-07-31T02:13:43.729 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-p0xLWUeTtp2NdiC off=4500177 len=829085 2021-07-31T02:13:43.732 INFO:tasks.workunit.client.0.smithi045.stdout:op 2612 completed, throughput=5MB/sec 2021-07-31T02:13:43.732 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ZVhWSxwrO-4Jb05 off=209926 len=1036313 2021-07-31T02:13:43.734 INFO:tasks.workunit.client.0.smithi045.stdout:op 2613 completed, throughput=5MB/sec 2021-07-31T02:13:43.735 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Icg2lrG3vqJuDzR off=2727786 len=1569129 2021-07-31T02:13:43.741 INFO:tasks.workunit.client.0.smithi045.stdout:op 2614 completed, throughput=5MB/sec 2021-07-31T02:13:43.741 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-D5tayx7G_a2PSGH off=10279314 len=1024054 2021-07-31T02:13:43.748 INFO:tasks.workunit.client.0.smithi045.stdout:op 2615 completed, throughput=5MB/sec 2021-07-31T02:13:43.806 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:13:44.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:13:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:13:43.554Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:13:44.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:13:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:13:43.554Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:13:44.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:13:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:13:43.554Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:13:44.085 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:13:44.085 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 28m ago 29m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:13:44.085 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 26m ago 29m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:13:44.085 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 26m ago 32m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:13:44.086 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 28m ago 34m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:13:44.086 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 28m ago 34m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:13:44.086 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 26m ago 32m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:13:44.086 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 28m ago 33m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:13:44.087 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 28m ago 29m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:13:44.087 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 26m ago 29m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:13:44.087 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 28m ago 32m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:13:44.087 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 28m ago 32m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:13:44.087 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 28m ago 31m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:13:44.088 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 28m ago 31m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:13:44.088 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 26m ago 31m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:13:44.088 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 26m ago 30m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:13:44.088 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 26m ago 30m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:13:44.088 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 26m ago 30m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:13:44.089 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 26m ago 29m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:13:44.422 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:13:44.423 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:13:44.423 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:13:44.423 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:13:44.424 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:13:44.424 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:13:44.424 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:13:44.424 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:13:44.424 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:13:44.425 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:13:44.425 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:13:44.425 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:13:44.425 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:13:44.426 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:13:44.426 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:13:44.426 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:13:44.426 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:13:44.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:44 smithi154 conmon[34553]: audit 2021-07-31T02:13:43.794514+0000 mgr.x (mgr.24542) 1137 : audit [DBG] from='client.25930 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:13:44.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:44 smithi154 conmon[34553]: audit 2021-07-31T02:13:44. 2021-07-31T02:13:44.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:44 smithi154 conmon[34553]: 077101+0000 mgr.x (mgr.24542) 1138 : audit [DBG] from='client.25936 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:13:44.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:44 smithi154 conmon[34553]: cluster 2021-07-31T02:13:44.170656+0000 mgr.x (mgr.24542) 1139 : cluster [DBG] pgmap v1667: 9 pgs: 9 active+clean; 127 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 3.8 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2021-07-31T02:13:44.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:44 smithi045 conmon[33655]: audit 2021-07-31T02:13:43. 2021-07-31T02:13:44.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:44 smithi045 conmon[33655]: 794514+0000 mgr.x (mgr.24542) 1137 : audit [DBG] from='client.25930 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:13:44.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:44 smithi045 conmon[33655]: audit 2021-07- 2021-07-31T02:13:44.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:44 smithi045 conmon[33655]: 31T02:13:44.077101+0000 mgr.x (mgr.24542) 1138 : 2021-07-31T02:13:44.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:44 smithi045 conmon[33655]: audit [DBG] from='client.25936 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:13:44.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:44 smithi045 conmon[33655]: cluster 2021-07 2021-07-31T02:13:44.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:44 smithi045 conmon[33655]: -31T02:13:44.170656+0000 2021-07-31T02:13:44.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:44 smithi045 conmon[33655]: mgr.x (mgr.24542) 1139 : cluster [DBG] 2021-07-31T02:13:44.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:44 smithi045 conmon[33655]: pgmap v1667: 9 pgs: 9 active+clean; 127 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 3.8 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2021-07-31T02:13:44.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:44 smithi045 conmon[39718]: audit 2021-07-31T02:13:43.794514+0000 2021-07-31T02:13:44.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:44 smithi045 conmon[39718]: mgr.x (mgr.24542) 1137 : audit [DBG] from='client.25930 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:13:44.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:44 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T02:13:44.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:44 smithi045 conmon[39718]: T02:13:44.077101+0000 mgr.x (mgr.24542) 1138 : audit [DBG] from='client.25936 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:13:44.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:44 smithi045 conmon[39718]: 2021-07-31T02:13:44.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:44 smithi045 conmon[39718]: cluster 2021-07-31T02: 2021-07-31T02:13:44.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:44 smithi045 conmon[39718]: 13:44.170656+0000 mgr.x (mgr.24542 2021-07-31T02:13:44.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:44 smithi045 conmon[39718]: ) 1139 : cluster [DBG] pgmap v1667: 9 pgs: 9 active+clean; 127 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 3.8 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2021-07-31T02:13:44.749 INFO:tasks.workunit.client.0.smithi045.stdout: 508: throughput=4.99MB/sec pending data=0 2021-07-31T02:13:44.749 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ti6DX9BNIgU7Y-N off=701544 len=1535155 2021-07-31T02:13:44.754 INFO:tasks.workunit.client.0.smithi045.stdout:op 2616 completed, throughput=4.99MB/sec 2021-07-31T02:13:44.755 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-di_45glY8bfmAQG off=6191812 len=720320 2021-07-31T02:13:44.756 INFO:tasks.workunit.client.0.smithi045.stdout:op 2617 completed, throughput=5MB/sec 2021-07-31T02:13:44.756 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-MUnZumriKSLdKlU off=3916803 len=751269 2021-07-31T02:13:44.758 INFO:tasks.workunit.client.0.smithi045.stdout:op 2618 completed, throughput=5MB/sec 2021-07-31T02:13:44.759 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-T3uWFWh3W7Ldsyt off=6617657 len=41561 2021-07-31T02:13:44.760 INFO:tasks.workunit.client.0.smithi045.stdout:op 2619 completed, throughput=5MB/sec 2021-07-31T02:13:44.761 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-cVipZBayLSXOgBK off=6428396 len=694522 2021-07-31T02:13:44.762 INFO:tasks.workunit.client.0.smithi045.stdout:op 2620 completed, throughput=5MB/sec 2021-07-31T02:13:44.762 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Vdgfwt0GsQtXOaw off=18108780 len=887197 2021-07-31T02:13:44.768 INFO:tasks.workunit.client.0.smithi045.stdout:op 2621 completed, throughput=5MB/sec 2021-07-31T02:13:45.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:45 smithi154 conmon[34553]: audit 2021-07-31T02:13:44.422722+0000 mon.a (mon.0) 1877 : audit [DBG] from='client.? 172.21.15.45:0/3435963833' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:13:45.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:45 smithi045 conmon[33655]: audit 2021-07-31T02:13:44.422722+0000 mon.a (mon.0) 1877 2021-07-31T02:13:45.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:45 smithi045 conmon[33655]: : audit [DBG] from='client.? 172.21.15.45:0/3435963833' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:13:45.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:45 smithi045 conmon[39718]: audit 2021-07-31T02:13:44.422722+0000 mon.a 2021-07-31T02:13:45.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:45 smithi045 conmon[39718]: (mon.0) 1877 : audit [DBG] from='client.? 172.21.15.45:0/3435963833' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:13:45.767 INFO:tasks.workunit.client.0.smithi045.stdout: 509: throughput=4.99MB/sec pending data=0 2021-07-31T02:13:45.768 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-X-0Cf5vCI0zs6C6 off=12060145 len=26071 2021-07-31T02:13:45.770 INFO:tasks.workunit.client.0.smithi045.stdout:op 2622 completed, throughput=4.99MB/sec 2021-07-31T02:13:45.770 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-2k_T8x4JHkyT6qI off=5239645 len=1734123 2021-07-31T02:13:45.771 INFO:tasks.workunit.client.0.smithi045.stdout:op 2623 completed, throughput=4.99MB/sec 2021-07-31T02:13:45.772 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-meYqCa50pB3nqEJ off=8502173 len=708464 2021-07-31T02:13:45.777 INFO:tasks.workunit.client.0.smithi045.stdout:op 2624 completed, throughput=5MB/sec 2021-07-31T02:13:45.777 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-27OtLgSRigaqwC7 off=1075215 len=1972756 2021-07-31T02:13:45.780 INFO:tasks.workunit.client.0.smithi045.stdout:op 2625 completed, throughput=5MB/sec 2021-07-31T02:13:45.781 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-SrBumC4zogdfjgg off=10427321 len=1926803 2021-07-31T02:13:45.791 INFO:tasks.workunit.client.0.smithi045.stdout:op 2626 completed, throughput=5MB/sec 2021-07-31T02:13:46.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:46 smithi154 conmon[34553]: cluster 2021-07-31T02:13:46.171161+0000 mgr.x (mgr.24542) 1140 : cluster [DBG] pgmap v1668: 9 pgs: 9 active+clean; 127 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.8 MiB/s wr, 4 op/s 2021-07-31T02:13:46.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:46 smithi045 conmon[33655]: cluster 2021-07-31T02:13:46.171161+0000 mgr.x (mgr.24542) 1140 2021-07-31T02:13:46.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:46 smithi045 conmon[33655]: : cluster [DBG] pgmap v1668: 9 pgs: 9 active+clean; 127 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.8 MiB/s wr, 4 op/s 2021-07-31T02:13:46.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:46 smithi045 conmon[39718]: cluster 2021-07-31T02:13:46.171161+0000 mgr.x (mgr 2021-07-31T02:13:46.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:46 smithi045 conmon[39718]: .24542) 1140 : cluster [DBG] pgmap v1668: 9 pgs: 9 active+clean; 127 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.8 MiB/s wr, 4 op/s 2021-07-31T02:13:46.791 INFO:tasks.workunit.client.0.smithi045.stdout: 510: throughput=4.99MB/sec pending data=0 2021-07-31T02:13:46.791 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-auAVIF8yvV__N0z off=1557805 len=321252 2021-07-31T02:13:46.795 INFO:tasks.workunit.client.0.smithi045.stdout:op 2627 completed, throughput=4.99MB/sec 2021-07-31T02:13:46.795 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-N6te10k3VtxE98X off=6877072 len=93200 2021-07-31T02:13:46.798 INFO:tasks.workunit.client.0.smithi045.stdout:op 2628 completed, throughput=4.99MB/sec 2021-07-31T02:13:46.799 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-v2CNTs2ncD2O_DL off=11594763 len=1644314 2021-07-31T02:13:46.806 INFO:tasks.workunit.client.0.smithi045.stdout:op 2629 completed, throughput=5MB/sec 2021-07-31T02:13:46.806 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-KxVpoT73kf0LXZD off=661993 len=667096 2021-07-31T02:13:46.810 INFO:tasks.workunit.client.0.smithi045.stdout:op 2630 completed, throughput=5MB/sec 2021-07-31T02:13:46.811 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-DCNEaaqtVtnOac- off=3962507 len=1805659 2021-07-31T02:13:46.813 INFO:tasks.workunit.client.0.smithi045.stdout:op 2631 completed, throughput=5MB/sec 2021-07-31T02:13:47.814 INFO:tasks.workunit.client.0.smithi045.stdout: 511: throughput=4.99MB/sec pending data=0 2021-07-31T02:13:47.814 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-54PZmK-2oHredxP off=4742097 len=455742 2021-07-31T02:13:47.818 INFO:tasks.workunit.client.0.smithi045.stdout:op 2632 completed, throughput=4.99MB/sec 2021-07-31T02:13:47.818 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-aGKZ5huif2sVCKd off=692138 len=1755549 2021-07-31T02:13:47.826 INFO:tasks.workunit.client.0.smithi045.stdout:op 2633 completed, throughput=5MB/sec 2021-07-31T02:13:47.827 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-uuxAXEF2hQ9OIeR off=5153918 len=468119 2021-07-31T02:13:47.830 INFO:tasks.workunit.client.0.smithi045.stdout:op 2634 completed, throughput=5MB/sec 2021-07-31T02:13:47.830 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-7Lf6dWTWef4HBys off=14211194 len=1402992 2021-07-31T02:13:47.831 INFO:tasks.workunit.client.0.smithi045.stdout:op 2635 completed, throughput=5MB/sec 2021-07-31T02:13:47.831 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ETYU3xndRpbeDJ1 off=18773251 len=857351 2021-07-31T02:13:47.833 INFO:tasks.workunit.client.0.smithi045.stdout:op 2636 completed, throughput=5MB/sec 2021-07-31T02:13:47.833 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-u5gyWKny5iFn5pe off=2405258 len=695786 2021-07-31T02:13:47.838 INFO:tasks.workunit.client.0.smithi045.stdout:op 2637 completed, throughput=5MB/sec 2021-07-31T02:13:48.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:48 smithi154 conmon[34553]: cluster 2021-07-31T02:13:48.171679+0000 mgr.x (mgr.24542) 1141 : cluster [DBG] pgmap v1669: 9 pgs: 9 active+clean; 127 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.8 MiB/s wr, 4 op/s 2021-07-31T02:13:48.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:48 smithi045 conmon[33655]: cluster 2021-07-31T02:13:48.171679+0000 mgr.x (mgr.24542 2021-07-31T02:13:48.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:48 smithi045 conmon[33655]: ) 1141 : cluster [DBG] pgmap v1669: 9 pgs: 9 active+clean; 127 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.8 MiB/s wr, 4 op/s 2021-07-31T02:13:48.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:48 smithi045 conmon[39718]: cluster 2021-07-31T02:13:48.171679+0000 mgr.x (mgr. 2021-07-31T02:13:48.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:48 smithi045 conmon[39718]: 24542) 1141 : cluster [DBG] pgmap v1669: 9 pgs: 9 active+clean; 127 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.8 MiB/s wr, 4 op/s 2021-07-31T02:13:48.838 INFO:tasks.workunit.client.0.smithi045.stdout: 512: throughput=4.99MB/sec pending data=0 2021-07-31T02:13:48.839 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-i73JiRkVZIxv1bU off=5709449 len=967484 2021-07-31T02:13:48.846 INFO:tasks.workunit.client.0.smithi045.stdout:op 2638 completed, throughput=4.99MB/sec 2021-07-31T02:13:48.847 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-HOhJ0Z-M6ZVtht3 off=1818216 len=1895325 2021-07-31T02:13:48.850 INFO:tasks.workunit.client.0.smithi045.stdout:op 2639 completed, throughput=5MB/sec 2021-07-31T02:13:48.850 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-U_YKFJY5ypfrIbI off=18295238 len=1558628 2021-07-31T02:13:48.853 INFO:tasks.workunit.client.0.smithi045.stdout:op 2640 completed, throughput=5MB/sec 2021-07-31T02:13:48.853 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-L5DHPZcFQBu0nAH off=7882013 len=926802 2021-07-31T02:13:48.859 INFO:tasks.workunit.client.0.smithi045.stdout:op 2641 completed, throughput=5MB/sec 2021-07-31T02:13:49.859 INFO:tasks.workunit.client.0.smithi045.stdout: 513: throughput=4.99MB/sec pending data=0 2021-07-31T02:13:49.859 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-gojq97kEEKgqQzW off=2409471 len=619773 2021-07-31T02:13:49.863 INFO:tasks.workunit.client.0.smithi045.stdout:op 2642 completed, throughput=4.99MB/sec 2021-07-31T02:13:49.863 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ia26eqiBSBPtcqM off=523680 len=1540512 2021-07-31T02:13:49.865 INFO:tasks.workunit.client.0.smithi045.stdout:op 2643 completed, throughput=5MB/sec 2021-07-31T02:13:49.865 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-n9XIVoyAhyCVhYL off=5595651 len=1202183 2021-07-31T02:13:49.866 INFO:tasks.workunit.client.0.smithi045.stdout:op 2644 completed, throughput=5MB/sec 2021-07-31T02:13:49.867 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-rqNINyQLbbbX-21 off=4363372 len=1759149 2021-07-31T02:13:49.875 INFO:tasks.workunit.client.0.smithi045.stdout:op 2645 completed, throughput=5MB/sec 2021-07-31T02:13:50.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:50 smithi154 conmon[34553]: cluster 2021-07-31T02:13:50.172158+0000 mgr.x (mgr.24542) 1142 : cluster [DBG] pgmap v1670: 9 pgs: 9 active+clean; 127 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 3.5 MiB/s rd, 2.8 MiB/s wr, 6 op/s 2021-07-31T02:13:50.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:50 smithi045 conmon[33655]: cluster 2021-07-31T02:13:50.172158+0000 mgr.x (mgr.24542) 1142 : cluster [DBG] pgmap v1670: 9 pgs: 9 active+clean; 127 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 3.5 MiB/s rd, 2.8 MiB/s wr, 6 op/s 2021-07-31T02:13:50.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:50 smithi045 conmon[39718]: cluster 2021-07-31T02:13:50.172158+0000 mgr.x (mgr. 2021-07-31T02:13:50.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:50 smithi045 conmon[39718]: 24542) 1142 : cluster [DBG] pgmap v1670: 9 pgs: 9 active+clean; 127 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 3.5 MiB/s rd, 2.8 MiB/s wr, 6 op/s 2021-07-31T02:13:50.875 INFO:tasks.workunit.client.0.smithi045.stdout: 514: throughput=4.99MB/sec pending data=0 2021-07-31T02:13:50.875 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-DsRtPGodGLBgA8Z off=756344 len=529495 2021-07-31T02:13:50.877 INFO:tasks.workunit.client.0.smithi045.stdout:op 2646 completed, throughput=4.99MB/sec 2021-07-31T02:13:50.877 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-tjHrTuc46JyCHS7 off=3310284 len=1837832 2021-07-31T02:13:50.887 INFO:tasks.workunit.client.0.smithi045.stdout:op 2647 completed, throughput=5MB/sec 2021-07-31T02:13:50.887 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-0opMQ202DDfqJ4N off=9093735 len=396395 2021-07-31T02:13:50.891 INFO:tasks.workunit.client.0.smithi045.stdout:op 2648 completed, throughput=5MB/sec 2021-07-31T02:13:50.891 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-v9Q-GJi5T4VYG0Z off=18792118 len=1430099 2021-07-31T02:13:50.899 INFO:tasks.workunit.client.0.smithi045.stdout:op 2649 completed, throughput=5MB/sec 2021-07-31T02:13:50.900 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-F4kzAI3-nwXxzAW off=9521964 len=161202 2021-07-31T02:13:50.902 INFO:tasks.workunit.client.0.smithi045.stdout:op 2650 completed, throughput=5MB/sec 2021-07-31T02:13:50.902 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-xTAYTN-yjlzarKx off=13851510 len=1420612 2021-07-31T02:13:50.908 INFO:tasks.workunit.client.0.smithi045.stdout:op 2651 completed, throughput=5MB/sec 2021-07-31T02:13:51.908 INFO:tasks.workunit.client.0.smithi045.stdout: 515: throughput=4.99MB/sec pending data=0 2021-07-31T02:13:51.909 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-HZnR6CkMR3RcKY8 off=4880813 len=491519 2021-07-31T02:13:51.910 INFO:tasks.workunit.client.0.smithi045.stdout:op 2652 completed, throughput=4.99MB/sec 2021-07-31T02:13:51.910 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-KRVf5Ci3UMuYfyb off=2199099 len=1455089 2021-07-31T02:13:51.917 INFO:tasks.workunit.client.0.smithi045.stdout:op 2653 completed, throughput=5MB/sec 2021-07-31T02:13:51.918 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-U4QRETZpBRIADCW off=1368804 len=1394193 2021-07-31T02:13:51.924 INFO:tasks.workunit.client.0.smithi045.stdout:op 2654 completed, throughput=5MB/sec 2021-07-31T02:13:51.925 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-QH3i59TWn3fOhV5 off=581267 len=493783 2021-07-31T02:13:51.928 INFO:tasks.workunit.client.0.smithi045.stdout:op 2655 completed, throughput=5MB/sec 2021-07-31T02:13:51.928 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-voELXgXc7t7nZB- off=6822788 len=1368638 2021-07-31T02:13:51.930 INFO:tasks.workunit.client.0.smithi045.stdout:op 2656 completed, throughput=5MB/sec 2021-07-31T02:13:52.931 INFO:tasks.workunit.client.0.smithi045.stdout: 516: throughput=4.99MB/sec pending data=0 2021-07-31T02:13:52.931 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-hdDDV6xc3X6xKPS off=3805477 len=954125 2021-07-31T02:13:52.932 INFO:tasks.workunit.client.0.smithi045.stdout:op 2657 completed, throughput=4.99MB/sec 2021-07-31T02:13:52.933 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-j_UYGuzFF1nI0Fu off=8802209 len=718940 2021-07-31T02:13:52.937 INFO:tasks.workunit.client.0.smithi045.stdout:op 2658 completed, throughput=4.99MB/sec 2021-07-31T02:13:52.937 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ScZSiykI4ANQIlF off=4083696 len=154273 2021-07-31T02:13:52.938 INFO:tasks.workunit.client.0.smithi045.stdout:op 2659 completed, throughput=4.99MB/sec 2021-07-31T02:13:52.938 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-C-mMYfsJvCRPHgU off=3338067 len=977683 2021-07-31T02:13:52.939 INFO:tasks.workunit.client.0.smithi045.stdout:op 2660 completed, throughput=5MB/sec 2021-07-31T02:13:52.939 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-6WyNSE-onQJ_oOi off=15930231 len=1909624 2021-07-31T02:13:52.946 INFO:tasks.workunit.client.0.smithi045.stdout:op 2661 completed, throughput=5MB/sec 2021-07-31T02:13:53.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:53 smithi154 conmon[34553]: cluster 2021-07-31T02:13:52.172678+0000 mgr.x (mgr.24542) 1143 : cluster [DBG] pgmap v1671: 9 pgs: 9 active+clean; 127 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 2.6 MiB/s rd, 1.6 MiB/s wr, 4 op/s 2021-07-31T02:13:53.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:53 smithi045 conmon[39718]: cluster 2021-07-31T02:13:52.172678+0000 mgr.x ( 2021-07-31T02:13:53.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:53 smithi045 conmon[39718]: mgr.24542) 1143 : cluster [DBG] pgmap v1671: 9 pgs: 9 active+clean; 127 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 2.6 MiB/s rd, 1.6 MiB/s wr, 4 op/s 2021-07-31T02:13:53.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:53 smithi045 conmon[33655]: cluster 2021-07-31T02:13:52.172678+0000 mgr.x 2021-07-31T02:13:53.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:53 smithi045 conmon[33655]: (mgr.24542) 1143 : cluster [DBG] pgmap v1671: 9 pgs: 9 active+clean; 127 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 2.6 MiB/s rd, 1.6 MiB/s wr, 4 op/s 2021-07-31T02:13:53.947 INFO:tasks.workunit.client.0.smithi045.stdout: 517: throughput=4.99MB/sec pending data=0 2021-07-31T02:13:53.947 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-HfuMDIiRExZqVYz off=1570638 len=314941 2021-07-31T02:13:53.948 INFO:tasks.workunit.client.0.smithi045.stdout:op 2662 completed, throughput=4.99MB/sec 2021-07-31T02:13:53.948 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-WYFtiXzPj4297px off=13976231 len=1360319 2021-07-31T02:13:53.951 INFO:tasks.workunit.client.0.smithi045.stdout:op 2663 completed, throughput=4.99MB/sec 2021-07-31T02:13:53.951 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-I6FWTve_smA9iUx off=14076748 len=1290932 2021-07-31T02:13:53.953 INFO:tasks.workunit.client.0.smithi045.stdout:op 2664 completed, throughput=5MB/sec 2021-07-31T02:13:53.953 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-bEiVQoOeot--dVg off=13273893 len=1698734 2021-07-31T02:13:53.956 INFO:tasks.workunit.client.0.smithi045.stdout:op 2665 completed, throughput=5MB/sec 2021-07-31T02:13:53.956 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-sUoEYvK6Fb3NA1c off=9839778 len=842898 2021-07-31T02:13:53.961 INFO:tasks.workunit.client.0.smithi045.stdout:op 2666 completed, throughput=5MB/sec 2021-07-31T02:13:54.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:13:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:13:53.554Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:13:54.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:13:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:13:53.554Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:13:54.027 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:13:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:13:53.554Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:13:54.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:54 smithi154 conmon[34553]: cluster 2021-07-31T02:13:54.173212+0000 mgr.x (mgr.24542) 1144 : cluster [DBG] pgmap v1672: 9 pgs: 9 active+clean; 127 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 3.3 MiB/s rd, 2.7 MiB/s wr, 6 op/s 2021-07-31T02:13:54.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:54 smithi045 conmon[33655]: cluster 2021-07-31T02:13:54.173212+0000 mgr.x (mgr 2021-07-31T02:13:54.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:54 smithi045 conmon[33655]: .24542) 1144 : cluster [DBG] pgmap v1672: 9 pgs: 9 active+clean; 127 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 3.3 MiB/s rd, 2.7 MiB/s wr, 6 op/s 2021-07-31T02:13:54.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:54 smithi045 conmon[39718]: cluster 2021-07-31T02:13:54.173212+0000 mgr.x ( 2021-07-31T02:13:54.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:54 smithi045 conmon[39718]: mgr.24542) 1144 : cluster [DBG] pgmap v1672: 9 pgs: 9 active+clean; 127 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 3.3 MiB/s rd, 2.7 MiB/s wr, 6 op/s 2021-07-31T02:13:54.961 INFO:tasks.workunit.client.0.smithi045.stdout: 518: throughput=4.99MB/sec pending data=0 2021-07-31T02:13:54.962 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ppjiShPNq7fdfFH off=1289692 len=720647 2021-07-31T02:13:54.966 INFO:tasks.workunit.client.0.smithi045.stdout:op 2667 completed, throughput=4.99MB/sec 2021-07-31T02:13:54.966 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-SbTD1-bUYMr3aRS off=13813828 len=1327396 2021-07-31T02:13:54.968 INFO:tasks.workunit.client.0.smithi045.stdout:op 2668 completed, throughput=4.99MB/sec 2021-07-31T02:13:54.969 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-a1abglOcj3yugs0 off=921209 len=1424130 2021-07-31T02:13:54.976 INFO:tasks.workunit.client.0.smithi045.stdout:op 2669 completed, throughput=5MB/sec 2021-07-31T02:13:54.976 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-VaOBumP4ZbjSnXM off=12543706 len=1192141 2021-07-31T02:13:54.977 INFO:tasks.workunit.client.0.smithi045.stdout:op 2670 completed, throughput=5MB/sec 2021-07-31T02:13:54.978 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-s78ElQYhbaFV4Sr off=7787455 len=1584773 2021-07-31T02:13:54.984 INFO:tasks.workunit.client.0.smithi045.stdout:op 2671 completed, throughput=5MB/sec 2021-07-31T02:13:55.985 INFO:tasks.workunit.client.0.smithi045.stdout: 519: throughput=4.99MB/sec pending data=0 2021-07-31T02:13:55.985 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-FGoJ6st2V46gIKN off=13277797 len=548297 2021-07-31T02:13:55.987 INFO:tasks.workunit.client.0.smithi045.stdout:op 2672 completed, throughput=4.99MB/sec 2021-07-31T02:13:55.987 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-eRvhhBzbbHFmJON off=146724 len=1003246 2021-07-31T02:13:55.992 INFO:tasks.workunit.client.0.smithi045.stdout:op 2673 completed, throughput=5MB/sec 2021-07-31T02:13:55.993 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-dT6iqAeAprOs6ml off=646028 len=2063246 2021-07-31T02:13:55.995 INFO:tasks.workunit.client.0.smithi045.stdout:op 2674 completed, throughput=5MB/sec 2021-07-31T02:13:55.995 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-9j0fw7NXEKXQAqU off=1262967 len=376976 2021-07-31T02:13:55.999 INFO:tasks.workunit.client.0.smithi045.stdout:op 2675 completed, throughput=5MB/sec 2021-07-31T02:13:55.999 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-zQGa3lqyTKEUtEr off=2324065 len=74069 2021-07-31T02:13:56.000 INFO:tasks.workunit.client.0.smithi045.stdout:op 2676 completed, throughput=5MB/sec 2021-07-31T02:13:56.000 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-vI_cpA05tFbc_fX off=7105904 len=204503 2021-07-31T02:13:56.000 INFO:tasks.workunit.client.0.smithi045.stdout:op 2677 completed, throughput=5MB/sec 2021-07-31T02:13:56.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:56 smithi154 conmon[34553]: cluster 2021-07-31T02:13:56.173639+0000 mgr.x (mgr.24542) 1145 : cluster [DBG] pgmap v1673: 9 pgs: 9 active+clean; 127 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.1 MiB/s wr, 3 op/s 2021-07-31T02:13:56.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:56 smithi045 conmon[39718]: cluster 2021-07-31T02:13:56.173639+0000 mgr.x (mgr.24542 2021-07-31T02:13:56.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:56 smithi045 conmon[39718]: ) 1145 : cluster [DBG] pgmap v1673: 9 pgs: 9 active+clean; 127 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.1 MiB/s wr, 3 op/s 2021-07-31T02:13:56.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:56 smithi045 conmon[33655]: cluster 2021-07-31T02:13:56.173639+0000 mgr.x (mgr.24542) 1145 : cluster 2021-07-31T02:13:56.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:56 smithi045 conmon[33655]: [DBG] pgmap v1673: 9 pgs: 9 active+clean; 127 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.1 MiB/s wr, 3 op/s 2021-07-31T02:13:57.000 INFO:tasks.workunit.client.0.smithi045.stdout: 520: throughput=4.99MB/sec pending data=0 2021-07-31T02:13:57.001 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-1dhluxOwXaO6Uoe off=13604538 len=1785693 2021-07-31T02:13:57.009 INFO:tasks.workunit.client.0.smithi045.stdout:op 2678 completed, throughput=4.99MB/sec 2021-07-31T02:13:57.009 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-icBtYEDLS9mBCbd off=8257362 len=1937070 2021-07-31T02:13:57.012 INFO:tasks.workunit.client.0.smithi045.stdout:op 2679 completed, throughput=5MB/sec 2021-07-31T02:13:57.012 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-WCCtM2rXFaq3Y_l off=14250660 len=539266 2021-07-31T02:13:57.016 INFO:tasks.workunit.client.0.smithi045.stdout:op 2680 completed, throughput=5MB/sec 2021-07-31T02:13:57.016 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Y3PoIHMSHRbuIGk off=1700085 len=1336420 2021-07-31T02:13:57.018 INFO:tasks.workunit.client.0.smithi045.stdout:op 2681 completed, throughput=5MB/sec 2021-07-31T02:13:58.018 INFO:tasks.workunit.client.0.smithi045.stdout: 521: throughput=4.99MB/sec pending data=0 2021-07-31T02:13:58.019 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-yANiYClSSOZaHbo off=1858290 len=1212422 2021-07-31T02:13:58.020 INFO:tasks.workunit.client.0.smithi045.stdout:op 2682 completed, throughput=4.99MB/sec 2021-07-31T02:13:58.020 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-xSubuR0KLJNlQTY off=20342290 len=627138 2021-07-31T02:13:58.025 INFO:tasks.workunit.client.0.smithi045.stdout:op 2683 completed, throughput=4.99MB/sec 2021-07-31T02:13:58.025 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-kpikYLYWExb-IQv off=8319566 len=485601 2021-07-31T02:13:58.029 INFO:tasks.workunit.client.0.smithi045.stdout:op 2684 completed, throughput=5MB/sec 2021-07-31T02:13:58.030 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-tyEHXqiXzhDOJ5T off=13877685 len=312794 2021-07-31T02:13:58.032 INFO:tasks.workunit.client.0.smithi045.stdout:op 2685 completed, throughput=5MB/sec 2021-07-31T02:13:58.033 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ykPUB7DLNYA651r off=10470417 len=256127 2021-07-31T02:13:58.036 INFO:tasks.workunit.client.0.smithi045.stdout:op 2686 completed, throughput=5MB/sec 2021-07-31T02:13:58.036 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-OzM7OmZ1GukIsVi off=17889228 len=430885 2021-07-31T02:13:58.039 INFO:tasks.workunit.client.0.smithi045.stdout:op 2687 completed, throughput=5MB/sec 2021-07-31T02:13:58.040 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-OS3Dzv9iprMuNWa off=1121755 len=241924 2021-07-31T02:13:58.040 INFO:tasks.workunit.client.0.smithi045.stdout:op 2688 completed, throughput=5MB/sec 2021-07-31T02:13:58.041 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-FaZzyczWtHctF_Q off=970052 len=1926706 2021-07-31T02:13:58.043 INFO:tasks.workunit.client.0.smithi045.stdout:op 2689 completed, throughput=5MB/sec 2021-07-31T02:13:58.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:13:58 smithi154 conmon[34553]: cluster 2021-07-31T02:13:58.174076+0000 mgr.x (mgr.24542) 1146 : cluster [DBG] pgmap v1674: 9 pgs: 9 active+clean; 127 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.1 MiB/s wr, 3 op/s 2021-07-31T02:13:58.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:58 smithi045 conmon[39718]: cluster 2021-07-31T02:13:58.174076+0000 mgr.x (mgr. 2021-07-31T02:13:58.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:13:58 smithi045 conmon[39718]: 24542) 1146 : cluster [DBG] pgmap v1674: 9 pgs: 9 active+clean; 127 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.1 MiB/s wr, 3 op/s 2021-07-31T02:13:58.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:58 smithi045 conmon[33655]: cluster 2021-07-31T02:13:58.174076+0000 mgr.x (mgr.24542 2021-07-31T02:13:58.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:13:58 smithi045 conmon[33655]: ) 1146 : cluster [DBG] pgmap v1674: 9 pgs: 9 active+clean; 127 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.1 MiB/s wr, 3 op/s 2021-07-31T02:13:59.044 INFO:tasks.workunit.client.0.smithi045.stdout: 522: throughput=4.99MB/sec pending data=0 2021-07-31T02:13:59.044 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-3dwwkXU4O3Dfr6n off=4749183 len=915441 2021-07-31T02:13:59.049 INFO:tasks.workunit.client.0.smithi045.stdout:op 2690 completed, throughput=4.99MB/sec 2021-07-31T02:13:59.049 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-rp0tkZihfWv5Mf_ off=20888124 len=2076452 2021-07-31T02:13:59.053 INFO:tasks.workunit.client.0.smithi045.stdout:op 2691 completed, throughput=5MB/sec 2021-07-31T02:13:59.054 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-KR3blaFiZYENR0q off=13338624 len=672321 2021-07-31T02:13:59.058 INFO:tasks.workunit.client.0.smithi045.stdout:op 2692 completed, throughput=5MB/sec 2021-07-31T02:13:59.058 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-D_gKjpWBTV2VLFH off=6848057 len=205822 2021-07-31T02:13:59.059 INFO:tasks.workunit.client.0.smithi045.stdout:op 2693 completed, throughput=5MB/sec 2021-07-31T02:13:59.059 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-CcHkewqpRHf8QUH off=5788363 len=422760 2021-07-31T02:13:59.060 INFO:tasks.workunit.client.0.smithi045.stdout:op 2694 completed, throughput=5MB/sec 2021-07-31T02:13:59.060 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-HVREHOLcTxl_-7w off=9577605 len=957142 2021-07-31T02:13:59.065 INFO:tasks.workunit.client.0.smithi045.stdout:op 2695 completed, throughput=5MB/sec 2021-07-31T02:14:00.065 INFO:tasks.workunit.client.0.smithi045.stdout: 523: throughput=4.99MB/sec pending data=0 2021-07-31T02:14:00.065 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-f-I0QkJIHtiuizx off=6635688 len=18937 2021-07-31T02:14:00.068 INFO:tasks.workunit.client.0.smithi045.stdout:op 2696 completed, throughput=4.99MB/sec 2021-07-31T02:14:00.068 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-odwa6d1_84s-ob8 off=14222979 len=1754505 2021-07-31T02:14:00.077 INFO:tasks.workunit.client.0.smithi045.stdout:op 2697 completed, throughput=4.99MB/sec 2021-07-31T02:14:00.077 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-_NaEcu3I2k4_viH off=7411108 len=993256 2021-07-31T02:14:00.078 INFO:tasks.workunit.client.0.smithi045.stdout:op 2698 completed, throughput=5MB/sec 2021-07-31T02:14:00.078 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-MGEwYNeASCDfg28 off=1889028 len=1045933 2021-07-31T02:14:00.080 INFO:tasks.workunit.client.0.smithi045.stdout:op 2699 completed, throughput=5MB/sec 2021-07-31T02:14:00.081 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-3x9WESnvoJIuzPA off=6646440 len=1865081 2021-07-31T02:14:00.083 INFO:tasks.workunit.client.0.smithi045.stdout:op 2700 completed, throughput=5MB/sec 2021-07-31T02:14:00.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:00 smithi154 conmon[34553]: cluster 2021-07-31T02:14:00.174568+0000 mgr.x (mgr.24542) 1147 : cluster [DBG] 2021-07-31T02:14:00.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:00 smithi154 conmon[34553]: pgmap v1675: 9 pgs: 9 active+clean; 127 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.9 MiB/s wr, 6 op/s 2021-07-31T02:14:00.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:00 smithi045 conmon[33655]: cluster 2021-07-31T02:14:00.174568+0000 mgr.x (mgr.24542) 1147 : cluster [DBG] pgmap v1675: 9 pgs: 9 active+clean; 127 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.9 MiB/s wr, 6 op/s 2021-07-31T02:14:00.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:00 smithi045 conmon[33655]: 2021-07-31T02:14:00.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:00 smithi045 conmon[39718]: cluster 2021-07-31T02:14:00.174568+0000 mgr.x ( 2021-07-31T02:14:00.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:00 smithi045 conmon[39718]: mgr.24542) 1147 : cluster [DBG] pgmap v1675: 9 pgs: 9 active+clean; 127 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.9 MiB/s wr, 6 op/s 2021-07-31T02:14:01.083 INFO:tasks.workunit.client.0.smithi045.stdout: 524: throughput=4.99MB/sec pending data=0 2021-07-31T02:14:01.084 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-w-gsTrF3txKub3L off=315826 len=1749752 2021-07-31T02:14:01.085 INFO:tasks.workunit.client.0.smithi045.stdout:op 2701 completed, throughput=4.99MB/sec 2021-07-31T02:14:01.086 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-d1ptEIqRzFE8LYZ off=7216407 len=1854287 2021-07-31T02:14:01.094 INFO:tasks.workunit.client.0.smithi045.stdout:op 2702 completed, throughput=5MB/sec 2021-07-31T02:14:01.094 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Rlet7D4V_uDPmsG off=11874234 len=1832722 2021-07-31T02:14:01.102 INFO:tasks.workunit.client.0.smithi045.stdout:op 2703 completed, throughput=5MB/sec 2021-07-31T02:14:02.102 INFO:tasks.workunit.client.0.smithi045.stdout: 525: throughput=4.99MB/sec pending data=0 2021-07-31T02:14:02.103 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-RgvD9vIaGkDKYOc off=9363786 len=1462532 2021-07-31T02:14:02.111 INFO:tasks.workunit.client.0.smithi045.stdout:op 2704 completed, throughput=4.99MB/sec 2021-07-31T02:14:02.111 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-zyRVCOJg2xR3bgb off=1085543 len=1292364 2021-07-31T02:14:02.114 INFO:tasks.workunit.client.0.smithi045.stdout:op 2705 completed, throughput=5MB/sec 2021-07-31T02:14:02.114 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-nhcM-RauDcmdOzN off=8202676 len=778201 2021-07-31T02:14:02.115 INFO:tasks.workunit.client.0.smithi045.stdout:op 2706 completed, throughput=5MB/sec 2021-07-31T02:14:02.115 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-BRJO8cntZivL3fJ off=8811121 len=1919930 2021-07-31T02:14:02.125 INFO:tasks.workunit.client.0.smithi045.stdout:op 2707 completed, throughput=5MB/sec 2021-07-31T02:14:03.126 INFO:tasks.workunit.client.0.smithi045.stdout: 526: throughput=4.99MB/sec pending data=0 2021-07-31T02:14:03.127 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-1rcs6sS29u6PV0L off=1938813 len=2050228 2021-07-31T02:14:03.130 INFO:tasks.workunit.client.0.smithi045.stdout:op 2708 completed, throughput=5MB/sec 2021-07-31T02:14:03.131 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-dGi2cwenskgpsy4 off=2394455 len=1487432 2021-07-31T02:14:03.132 INFO:tasks.workunit.client.0.smithi045.stdout:op 2709 completed, throughput=5MB/sec 2021-07-31T02:14:03.133 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-qka3IgmO-HxqAKo off=3300821 len=1552709 2021-07-31T02:14:03.139 INFO:tasks.workunit.client.0.smithi045.stdout:op 2710 completed, throughput=5MB/sec 2021-07-31T02:14:03.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:03 smithi154 conmon[34553]: cluster 2021-07-31T02:14:02.175111+0000 mgr.x (mgr.24542) 1148 2021-07-31T02:14:03.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:03 smithi154 conmon[34553]: : cluster [DBG] pgmap v1676: 9 pgs: 9 active+clean; 127 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1.9 MiB/s rd, 1.9 MiB/s wr, 3 op/s 2021-07-31T02:14:03.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:03 smithi045 conmon[33655]: cluster 2021-07-31T02:14:02 2021-07-31T02:14:03.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:03 smithi045 conmon[33655]: .175111+0000 mgr.x (mgr.24542) 1148 : cluster [DBG] pgmap v1676: 9 pgs: 9 active+clean; 127 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1.9 MiB/s rd, 1.9 MiB/s wr, 3 op/s 2021-07-31T02:14:03.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:03 smithi045 conmon[39718]: cluster 2021-07-31T02:14:02.175111+0000 mgr.x 2021-07-31T02:14:03.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:03 smithi045 conmon[39718]: (mgr.24542) 1148 : cluster [DBG] pgmap v1676: 9 pgs: 9 active+clean; 127 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1.9 MiB/s rd, 1.9 MiB/s wr, 3 op/s 2021-07-31T02:14:04.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:14:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:14:03.554Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:14:04.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:14:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:14:03.554Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:14:04.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:14:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:14:03.554Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:14:04.139 INFO:tasks.workunit.client.0.smithi045.stdout: 527: throughput=4.99MB/sec pending data=0 2021-07-31T02:14:04.140 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ZPDhc1srauLuX-m off=2710616 len=2044307 2021-07-31T02:14:04.143 INFO:tasks.workunit.client.0.smithi045.stdout:op 2711 completed, throughput=5MB/sec 2021-07-31T02:14:04.144 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-lbXO179QIbeGHju off=1496605 len=453087 2021-07-31T02:14:04.147 INFO:tasks.workunit.client.0.smithi045.stdout:op 2712 completed, throughput=5MB/sec 2021-07-31T02:14:04.147 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-9V7Teu8NlXih2mM off=309178 len=1592989 2021-07-31T02:14:04.150 INFO:tasks.workunit.client.0.smithi045.stdout:op 2713 completed, throughput=5MB/sec 2021-07-31T02:14:04.150 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-CjrBK4Psa9z6ez6 off=739997 len=52855 2021-07-31T02:14:04.151 INFO:tasks.workunit.client.0.smithi045.stdout:op 2714 completed, throughput=5MB/sec 2021-07-31T02:14:04.151 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-0aBK3LfHa40vNiB off=12451781 len=794218 2021-07-31T02:14:04.156 INFO:tasks.workunit.client.0.smithi045.stdout:op 2715 completed, throughput=5MB/sec 2021-07-31T02:14:04.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:04 smithi154 conmon[34553]: cluster 2021-07-31T02:14:04.175591+0000 mgr.x (mgr.24542) 1149 : cluster [DBG] pgmap v1677: 9 pgs: 9 active+clean; 127 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.9 MiB/s wr, 5 op/s 2021-07-31T02:14:04.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:04 smithi154 conmon[34553]: 2021-07-31T02:14:04.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:04 smithi045 conmon[39718]: cluster 2021-07-31T02:14:04.175591+0000 mgr.x ( 2021-07-31T02:14:04.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:04 smithi045 conmon[39718]: mgr.24542) 1149 : cluster [DBG] pgmap v1677: 9 pgs: 9 active+clean; 127 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.9 MiB/s wr, 5 op/s 2021-07-31T02:14:04.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:04 smithi045 conmon[33655]: cluster 2021-07-31T02:14:04.175591+0000 mgr.x (mgr.24542) 1149 : cluster 2021-07-31T02:14:04.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:04 smithi045 conmon[33655]: [DBG] pgmap v1677: 9 pgs: 9 active+clean; 127 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.9 MiB/s wr, 5 op/s 2021-07-31T02:14:05.156 INFO:tasks.workunit.client.0.smithi045.stdout: 528: throughput=4.99MB/sec pending data=0 2021-07-31T02:14:05.157 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-2ENs6pO-Kk2FeXX off=4905291 len=538374 2021-07-31T02:14:05.158 INFO:tasks.workunit.client.0.smithi045.stdout:op 2716 completed, throughput=4.99MB/sec 2021-07-31T02:14:05.158 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-7Gf9Ci1zyo44_Ok off=1946118 len=1681146 2021-07-31T02:14:05.168 INFO:tasks.workunit.client.0.smithi045.stdout:op 2717 completed, throughput=4.99MB/sec 2021-07-31T02:14:05.169 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-0YdwYL0hIs9yi76 off=5584702 len=887682 2021-07-31T02:14:05.171 INFO:tasks.workunit.client.0.smithi045.stdout:op 2718 completed, throughput=5MB/sec 2021-07-31T02:14:05.171 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj--Nu29J3N7cPye2y off=2621955 len=13310 2021-07-31T02:14:05.171 INFO:tasks.workunit.client.0.smithi045.stdout:op 2719 completed, throughput=5MB/sec 2021-07-31T02:14:05.171 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-qa4HHsyeGAdQZpz off=18447811 len=1902383 2021-07-31T02:14:05.182 INFO:tasks.workunit.client.0.smithi045.stdout:op 2720 completed, throughput=5MB/sec 2021-07-31T02:14:05.182 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-dGNUKtInK85bDRC off=2960757 len=1717259 2021-07-31T02:14:05.189 INFO:tasks.workunit.client.0.smithi045.stdout:op 2721 completed, throughput=5MB/sec 2021-07-31T02:14:06.190 INFO:tasks.workunit.client.0.smithi045.stdout: 529: throughput=4.99MB/sec pending data=0 2021-07-31T02:14:06.190 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-u3LTIEDW-FxojTj off=7083134 len=1115468 2021-07-31T02:14:06.191 INFO:tasks.workunit.client.0.smithi045.stdout:op 2722 completed, throughput=5MB/sec 2021-07-31T02:14:06.192 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-06AsdFUYY60nNbm off=13171728 len=1768735 2021-07-31T02:14:06.193 INFO:tasks.workunit.client.0.smithi045.stdout:op 2723 completed, throughput=5MB/sec 2021-07-31T02:14:06.194 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-IrtKQblzTxA4ST4 off=10046757 len=1684958 2021-07-31T02:14:06.202 INFO:tasks.workunit.client.0.smithi045.stdout:op 2724 completed, throughput=5MB/sec 2021-07-31T02:14:06.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:06 smithi154 conmon[34553]: cluster 2021-07-31T02:14:06.176165+0000 mgr.x (mgr.24542) 1150 : cluster [DBG] 2021-07-31T02:14:06.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:06 smithi154 conmon[34553]: pgmap v1678: 9 pgs: 9 active+clean; 127 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 2.4 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2021-07-31T02:14:06.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:06 smithi045 conmon[33655]: cluster 2021-07-31T02:14:06.176165+0000 mgr.x (mgr 2021-07-31T02:14:06.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:06 smithi045 conmon[33655]: .24542) 1150 : cluster [DBG] pgmap v1678: 9 pgs: 9 active+clean; 127 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 2.4 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2021-07-31T02:14:06.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:06 smithi045 conmon[39718]: cluster 2021-07-31T02:14:06 2021-07-31T02:14:06.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:06 smithi045 conmon[39718]: .176165+0000 mgr.x (mgr.24542) 1150 : cluster [DBG] pgmap v1678: 9 pgs: 9 active+clean; 127 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 2.4 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2021-07-31T02:14:07.202 INFO:tasks.workunit.client.0.smithi045.stdout: 530: throughput=4.99MB/sec pending data=0 2021-07-31T02:14:07.203 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-1V6B6NdG9qOXGBr off=11916571 len=1392136 2021-07-31T02:14:07.205 INFO:tasks.workunit.client.0.smithi045.stdout:op 2725 completed, throughput=4.99MB/sec 2021-07-31T02:14:07.205 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-LioSqs1XrhYIr-V off=318258 len=1102570 2021-07-31T02:14:07.207 INFO:tasks.workunit.client.0.smithi045.stdout:op 2726 completed, throughput=5MB/sec 2021-07-31T02:14:07.208 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-0BBhixC4uPOb1hH off=5041054 len=1439779 2021-07-31T02:14:07.217 INFO:tasks.workunit.client.0.smithi045.stdout:op 2727 completed, throughput=5MB/sec 2021-07-31T02:14:07.217 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-WshGeyjZxOgrYEu off=4292112 len=1867108 2021-07-31T02:14:07.227 INFO:tasks.workunit.client.0.smithi045.stdout:op 2728 completed, throughput=5MB/sec 2021-07-31T02:14:08.227 INFO:tasks.workunit.client.0.smithi045.stdout: 531: throughput=4.99MB/sec pending data=0 2021-07-31T02:14:08.227 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-tN3oVXTbK6uue5_ off=9278254 len=1632971 2021-07-31T02:14:08.236 INFO:tasks.workunit.client.0.smithi045.stdout:op 2729 completed, throughput=5MB/sec 2021-07-31T02:14:08.236 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-FznJ6BH9jIs0m1r off=4973269 len=1158625 2021-07-31T02:14:08.242 INFO:tasks.workunit.client.0.smithi045.stdout:op 2730 completed, throughput=5MB/sec 2021-07-31T02:14:08.242 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-W6-uOV2H1QyP3XQ off=22248359 len=967681 2021-07-31T02:14:08.246 INFO:tasks.workunit.client.0.smithi045.stdout:op 2731 completed, throughput=5MB/sec 2021-07-31T02:14:08.246 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-XqnVRSN9qkUR9kn off=0 len=1173339 2021-07-31T02:14:08.251 INFO:tasks.workunit.client.0.smithi045.stdout:op 2732 completed, throughput=5MB/sec 2021-07-31T02:14:08.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:08 smithi154 conmon[34553]: cluster 2021-07-31T02:14:08.176638+0000 mgr.x (mgr.24542) 1151 : 2021-07-31T02:14:08.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:08 smithi154 conmon[34553]: cluster [DBG] pgmap v1679: 9 pgs: 9 active+clean; 127 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 2.4 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2021-07-31T02:14:08.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:08 smithi045 conmon[39718]: cluster 2021-07-31T02:14:08.176638+0000 2021-07-31T02:14:08.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:08 smithi045 conmon[39718]: mgr.x (mgr.24542) 1151 : cluster [DBG] pgmap v1679: 9 pgs: 9 active+clean; 127 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 2.4 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2021-07-31T02:14:08.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:08 smithi045 conmon[33655]: cluster 2021-07-31T02:14:08.176638+0000 2021-07-31T02:14:08.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:08 smithi045 conmon[33655]: mgr.x (mgr.24542) 1151 : cluster [DBG] pgmap v1679: 9 pgs: 9 active+clean; 127 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 2.4 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2021-07-31T02:14:09.252 INFO:tasks.workunit.client.0.smithi045.stdout: 532: throughput=4.99MB/sec pending data=0 2021-07-31T02:14:09.252 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-mBW0fLIMgV-oGgG off=71070 len=1306329 2021-07-31T02:14:09.257 INFO:tasks.workunit.client.0.smithi045.stdout:op 2733 completed, throughput=4.99MB/sec 2021-07-31T02:14:09.258 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ivYzZQwPooDiuN1 off=3883590 len=408703 2021-07-31T02:14:09.259 INFO:tasks.workunit.client.0.smithi045.stdout:op 2734 completed, throughput=5MB/sec 2021-07-31T02:14:09.259 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-8BZApo4xCnLWIBn off=5461115 len=1821704 2021-07-31T02:14:09.270 INFO:tasks.workunit.client.0.smithi045.stdout:op 2735 completed, throughput=5MB/sec 2021-07-31T02:14:09.270 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-MH1HGfXCRf6m0IQ off=4678394 len=85445 2021-07-31T02:14:09.271 INFO:tasks.workunit.client.0.smithi045.stdout:op 2736 completed, throughput=5MB/sec 2021-07-31T02:14:09.271 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-bP8uQR7W3QjQmUf off=5320309 len=2036368 2021-07-31T02:14:09.280 INFO:tasks.workunit.client.0.smithi045.stdout:op 2737 completed, throughput=5MB/sec 2021-07-31T02:14:10.280 INFO:tasks.workunit.client.0.smithi045.stdout: 533: throughput=4.99MB/sec pending data=0 2021-07-31T02:14:10.280 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-DxPcpFTy7Egspiv off=15697631 len=719585 2021-07-31T02:14:10.281 INFO:tasks.workunit.client.0.smithi045.stdout:op 2738 completed, throughput=4.99MB/sec 2021-07-31T02:14:10.281 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-pjt6wA5qEy4bIJb off=2079333 len=447506 2021-07-31T02:14:10.283 INFO:tasks.workunit.client.0.smithi045.stdout:op 2739 completed, throughput=4.99MB/sec 2021-07-31T02:14:10.283 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-S-C5AGbn2VoT6b6 off=7066410 len=1970470 2021-07-31T02:14:10.293 INFO:tasks.workunit.client.0.smithi045.stdout:op 2740 completed, throughput=5MB/sec 2021-07-31T02:14:10.293 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-8z2zEeJ6_STTO5E off=1226922 len=1878456 2021-07-31T02:14:10.297 INFO:tasks.workunit.client.0.smithi045.stdout:op 2741 completed, throughput=5MB/sec 2021-07-31T02:14:10.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:10 smithi154 conmon[34553]: cluster 2021-07-31T02:14:10.177215+0000 mgr.x (mgr.24542) 1152 : cluster [DBG] pgmap v1680: 9 pgs: 9 active+clean; 127 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 3.3 MiB/s rd, 3.2 MiB/s wr, 5 op/s 2021-07-31T02:14:10.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:10 smithi045 conmon[39718]: cluster 2021-07-31T02:14:10.177215+0000 mgr.x (mgr.24542) 1152 : cluster [DBG] pgmap v1680: 9 pgs: 9 active+clean; 127 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 3.3 MiB/s rd, 3.2 MiB/s wr, 5 op/s 2021-07-31T02:14:10.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:10 smithi045 conmon[33655]: cluster 2021-07-31T02:14:10.177215+0000 mgr.x (mgr.24542) 1152 : cluster [DBG] pgmap v1680: 9 pgs: 9 active+clean; 127 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 3.3 MiB/s rd, 3.2 MiB/s wr, 5 op/s 2021-07-31T02:14:11.295 INFO:tasks.workunit.client.0.smithi045.stdout: 534: throughput=4.99MB/sec pending data=0 2021-07-31T02:14:11.296 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-jfDFCRV-XZzR77i off=4061017 len=1713968 2021-07-31T02:14:11.299 INFO:tasks.workunit.client.0.smithi045.stdout:op 2742 completed, throughput=5MB/sec 2021-07-31T02:14:11.299 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-13K4ttIeeNhKPE3 off=601645 len=1813809 2021-07-31T02:14:11.302 INFO:tasks.workunit.client.0.smithi045.stdout:op 2743 completed, throughput=5MB/sec 2021-07-31T02:14:11.302 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-PKQRRJWOWQG-Ecj off=7162192 len=560699 2021-07-31T02:14:11.307 INFO:tasks.workunit.client.0.smithi045.stdout:op 2744 completed, throughput=5MB/sec 2021-07-31T02:14:11.308 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-M585E0fCt2VFqqg off=22111561 len=445226 2021-07-31T02:14:11.310 INFO:tasks.workunit.client.0.smithi045.stdout:op 2745 completed, throughput=5MB/sec 2021-07-31T02:14:11.311 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-4jVbgslFS3slk51 off=2142008 len=1646367 2021-07-31T02:14:11.312 INFO:tasks.workunit.client.0.smithi045.stdout:op 2746 completed, throughput=5MB/sec 2021-07-31T02:14:12.313 INFO:tasks.workunit.client.0.smithi045.stdout: 535: throughput=4.99MB/sec pending data=0 2021-07-31T02:14:12.313 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-g64RATF7wXFhj2u off=7615587 len=883059 2021-07-31T02:14:12.314 INFO:tasks.workunit.client.0.smithi045.stdout:op 2747 completed, throughput=5MB/sec 2021-07-31T02:14:12.314 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-mk2E21SmGBU7T7O off=2231166 len=1976532 2021-07-31T02:14:12.325 INFO:tasks.workunit.client.0.smithi045.stdout:op 2748 completed, throughput=5MB/sec 2021-07-31T02:14:12.326 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-v5savTkIu4d9GlV off=2058515 len=1882450 2021-07-31T02:14:12.336 INFO:tasks.workunit.client.0.smithi045.stdout:op 2749 completed, throughput=5MB/sec 2021-07-31T02:14:13.337 INFO:tasks.workunit.client.0.smithi045.stdout: 536: throughput=4.99MB/sec pending data=0 2021-07-31T02:14:13.337 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-O0s_M-_iEM4lMIx off=2541161 len=499261 2021-07-31T02:14:13.342 INFO:tasks.workunit.client.0.smithi045.stdout:op 2750 completed, throughput=4.99MB/sec 2021-07-31T02:14:13.342 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-rE7GUzvKrtOsgTs off=9681008 len=925250 2021-07-31T02:14:13.343 INFO:tasks.workunit.client.0.smithi045.stdout:op 2751 completed, throughput=5MB/sec 2021-07-31T02:14:13.343 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-yAPRBlWxW-uYuEC off=3383640 len=1764700 2021-07-31T02:14:13.353 INFO:tasks.workunit.client.0.smithi045.stdout:op 2752 completed, throughput=5MB/sec 2021-07-31T02:14:13.353 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-GBHJI6PMIgDsUeM off=16855474 len=113912 2021-07-31T02:14:13.353 INFO:tasks.workunit.client.0.smithi045.stdout:op 2753 completed, throughput=5MB/sec 2021-07-31T02:14:13.353 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-r5NXg2PGtDxHWZN off=1275482 len=1868882 2021-07-31T02:14:13.355 INFO:tasks.workunit.client.0.smithi045.stdout:op 2754 completed, throughput=5MB/sec 2021-07-31T02:14:13.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:13 smithi154 conmon[34553]: cluster 2021-07-31T02:14:12.177573+0000 mgr.x (mgr.24542) 1153 : cluster [DBG] pgmap v1681: 9 pgs: 9 active+clean; 127 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 3 op/s 2021-07-31T02:14:13.547 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:13 smithi154 conmon[34553]: 2021-07-31T02:14:13.548 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:13 smithi045 conmon[33655]: cluster 2021-07-31T02:14:12.177573+0000 mgr.x (mgr.24542) 1153 : cluster [DBG] 2021-07-31T02:14:13.548 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:13 smithi045 conmon[33655]: pgmap v1681: 9 pgs: 9 active+clean; 127 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 3 op/s 2021-07-31T02:14:13.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:13 smithi045 conmon[39718]: cluster 2021-07-31T02:14:12.177573 2021-07-31T02:14:13.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:13 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 1153 : cluster [DBG] pgmap v1681: 9 pgs: 9 active+clean; 127 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 3 op/s 2021-07-31T02:14:14.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:14:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:14:13.554Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:14:14.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:14:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:14:13.554Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:14:14.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:14:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:14:13.554Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:14:14.355 INFO:tasks.workunit.client.0.smithi045.stdout: 537: throughput=4.99MB/sec pending data=0 2021-07-31T02:14:14.355 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-hQZZQ_QBSoM8fQ6 off=8724261 len=138472 2021-07-31T02:14:14.358 INFO:tasks.workunit.client.0.smithi045.stdout:op 2755 completed, throughput=4.99MB/sec 2021-07-31T02:14:14.359 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-JVhDB6nSkWxuKz9 off=600250 len=624927 2021-07-31T02:14:14.363 INFO:tasks.workunit.client.0.smithi045.stdout:op 2756 completed, throughput=4.99MB/sec 2021-07-31T02:14:14.363 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-5o-_UBK-1ZP8OXC off=5412714 len=1253229 2021-07-31T02:14:14.371 INFO:tasks.workunit.client.0.smithi045.stdout:op 2757 completed, throughput=5MB/sec 2021-07-31T02:14:14.371 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-J8bSCV8IzvbrU6n off=1206787 len=1697648 2021-07-31T02:14:14.374 INFO:tasks.workunit.client.0.smithi045.stdout:op 2758 completed, throughput=5MB/sec 2021-07-31T02:14:14.374 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-eFW3DTwveJ6luDk off=1967484 len=1087141 2021-07-31T02:14:14.380 INFO:tasks.workunit.client.0.smithi045.stdout:op 2759 completed, throughput=5MB/sec 2021-07-31T02:14:14.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:14 smithi154 conmon[34553]: cluster 2021-07-31T02:14:14.178115+0000 mgr.x (mgr.24542) 1154 : cluster [DBG] 2021-07-31T02:14:14.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:14 smithi154 conmon[34553]: pgmap v1682: 9 pgs: 9 active+clean; 127 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 2.9 MiB/s rd, 3.5 MiB/s wr, 5 op/s 2021-07-31T02:14:14.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:14 smithi045 conmon[39718]: cluster 2021-07-31T02:14:14.178115+0000 mgr.x (mgr.24542) 1154 : cluster [DBG] pgmap v1682: 9 pgs: 9 active+clean; 127 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 2.9 MiB/s rd, 3.5 MiB/s wr, 5 op/s 2021-07-31T02:14:14.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:14 smithi045 conmon[33655]: cluster 2021-07-31T02:14:14.178115+0000 mgr.x (mgr.24542) 1154 : cluster 2021-07-31T02:14:14.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:14 smithi045 conmon[33655]: [DBG] pgmap v1682: 9 pgs: 9 active+clean; 127 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 2.9 MiB/s rd, 3.5 MiB/s wr, 5 op/s 2021-07-31T02:14:14.738 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:14:15.015 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:14:15.015 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 29m ago 29m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:14:15.015 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 26m ago 29m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:14:15.016 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 26m ago 33m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:14:15.016 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 29m ago 35m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:14:15.016 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 29m ago 35m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:14:15.016 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 26m ago 33m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:14:15.016 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 29m ago 33m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:14:15.017 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 29m ago 30m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:14:15.017 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 26m ago 30m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:14:15.017 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 29m ago 32m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:14:15.017 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 29m ago 32m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:14:15.018 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 29m ago 32m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:14:15.018 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 29m ago 31m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:14:15.018 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 26m ago 31m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:14:15.018 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 26m ago 31m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:14:15.018 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 26m ago 30m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:14:15.019 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 26m ago 30m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:14:15.019 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 26m ago 30m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:14:15.351 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:14:15.351 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:14:15.352 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:14:15.352 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:14:15.352 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:14:15.352 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:14:15.353 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:14:15.353 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:14:15.353 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:14:15.353 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:14:15.353 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:14:15.354 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:14:15.354 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:14:15.354 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:14:15.354 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:14:15.355 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:14:15.355 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:14:15.380 INFO:tasks.workunit.client.0.smithi045.stdout: 538: throughput=4.99MB/sec pending data=0 2021-07-31T02:14:15.381 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-JqbrGqrDJ_KT0uf off=8739877 len=1186015 2021-07-31T02:14:15.381 INFO:tasks.workunit.client.0.smithi045.stdout:op 2760 completed, throughput=4.99MB/sec 2021-07-31T02:14:15.381 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-6Yax0oDYCBGWj6R off=7052976 len=1608606 2021-07-31T02:14:15.390 INFO:tasks.workunit.client.0.smithi045.stdout:op 2761 completed, throughput=5MB/sec 2021-07-31T02:14:15.390 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-dogRzCUSQSQydsU off=2929072 len=35977 2021-07-31T02:14:15.391 INFO:tasks.workunit.client.0.smithi045.stdout:op 2762 completed, throughput=5MB/sec 2021-07-31T02:14:15.392 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-_G1kgr5LI50S3rT off=1132550 len=595872 2021-07-31T02:14:15.393 INFO:tasks.workunit.client.0.smithi045.stdout:op 2763 completed, throughput=5MB/sec 2021-07-31T02:14:15.394 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-97hmL6AK10f9gQA off=1039050 len=1588838 2021-07-31T02:14:15.400 INFO:tasks.workunit.client.0.smithi045.stdout:op 2764 completed, throughput=5MB/sec 2021-07-31T02:14:15.400 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-8RXhhVUvw1sH758 off=2669793 len=1644005 2021-07-31T02:14:15.402 INFO:tasks.workunit.client.0.smithi045.stdout:op 2765 completed, throughput=5MB/sec 2021-07-31T02:14:15.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:15 smithi154 conmon[34553]: audit 2021-07-31T02:14:14.726458+0000 mgr.x (mgr.24542) 1155 : audit [DBG] from='client.16041 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:14:15.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:15 smithi154 conmon[34553]: audit 2021-07-31T02:14:15.007408+0000 mgr.x (mgr.24542) 1156 : audit [DBG] from='client.25954 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:14:15.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:15 smithi045 conmon[33655]: audit 2021-07-31T02:14:14.726458+0000 mgr.x ( 2021-07-31T02:14:15.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:15 smithi045 conmon[33655]: mgr.24542) 1155 : audit [DBG] from='client.16041 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:14:15.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:15 smithi045 conmon[33655]: audit 2021-07-31T02:14:15.007408+0000 mgr.x (mgr.24542) 1156 : audit 2021-07-31T02:14:15.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:15 smithi045 conmon[33655]: [DBG] from='client.25954 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:14:15.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:15 smithi045 conmon[39718]: audit 2021-07-31T02:14:14.726458+0000 2021-07-31T02:14:15.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:15 smithi045 conmon[39718]: mgr.x (mgr.24542) 1155 : audit [DBG] from='client.16041 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:14:15.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:15 smithi045 conmon[39718]: audit 2021-07-31T02:14:15 2021-07-31T02:14:15.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:15 smithi045 conmon[39718]: .007408+0000 mgr.x (mgr.24542) 1156 : audit [DBG] from='client.25954 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:14:16.402 INFO:tasks.workunit.client.0.smithi045.stdout: 539: throughput=4.99MB/sec pending data=0 2021-07-31T02:14:16.403 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-FSvxA5rhkgtY6Pf off=928819 len=968831 2021-07-31T02:14:16.407 INFO:tasks.workunit.client.0.smithi045.stdout:op 2766 completed, throughput=5MB/sec 2021-07-31T02:14:16.408 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-DOfOn_8SbjYGZoF off=3592169 len=1721987 2021-07-31T02:14:16.411 INFO:tasks.workunit.client.0.smithi045.stdout:op 2767 completed, throughput=5MB/sec 2021-07-31T02:14:16.411 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-QYjusKGejy9Mw8M off=7845080 len=1411698 2021-07-31T02:14:16.414 INFO:tasks.workunit.client.0.smithi045.stdout:op 2768 completed, throughput=5MB/sec 2021-07-31T02:14:16.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:16 smithi154 conmon[34553]: audit 2021-07-31T02:14:15.351629+0000 mon.a (mon.0) 1878 : audit [DBG] from='client.? 172.21.15.45:0/2770883819' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:14:16.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:16 smithi154 conmon[34553]: cluster 2021-07-31T02:14:16.178625+0000 mgr.x 2021-07-31T02:14:16.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:16 smithi154 conmon[34553]: (mgr.24542) 1157 : cluster [DBG] pgmap v1683: 9 pgs: 9 active+clean; 127 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.4 MiB/s wr, 3 op/s 2021-07-31T02:14:16.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:16 smithi045 conmon[33655]: audit 2021-07-31T02:14:15.351629+0000 mon.a (mon.0) 1878 : audit [DBG] from='client.? 172.21.15.45:0/2770883819' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:14:16.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:16 smithi045 conmon[33655]: cluster 2021-07-31T02:14:16.178625+0000 mgr.x (mgr.24542) 1157 : cluster [DBG] pgmap v1683: 9 pgs: 9 active+clean; 127 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.4 MiB/s wr, 3 op/s 2021-07-31T02:14:16.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:16 smithi045 conmon[39718]: audit 2021-07-31T02:14:15.351629+0000 mon.a (mon. 2021-07-31T02:14:16.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:16 smithi045 conmon[39718]: 0) 1878 : audit [DBG] from='client.? 172.21.15.45:0/2770883819' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:14:16.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:16 smithi045 conmon[39718]: cluster 2021-07-31T02:14:16.178625+0000 mgr.x (mgr.24542) 1157 2021-07-31T02:14:16.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:16 smithi045 conmon[39718]: : cluster [DBG] pgmap v1683: 9 pgs: 9 active+clean; 127 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.4 MiB/s wr, 3 op/s 2021-07-31T02:14:17.414 INFO:tasks.workunit.client.0.smithi045.stdout: 540: throughput=4.99MB/sec pending data=0 2021-07-31T02:14:17.414 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-_RlyCm7YkMh4LH0 off=766677 len=2040041 2021-07-31T02:14:17.418 INFO:tasks.workunit.client.0.smithi045.stdout:op 2769 completed, throughput=5MB/sec 2021-07-31T02:14:17.418 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-yOvSOt7tOinaTrP off=10438533 len=2022804 2021-07-31T02:14:17.431 INFO:tasks.workunit.client.0.smithi045.stdout:op 2770 completed, throughput=5MB/sec 2021-07-31T02:14:17.431 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-r_E6pOuHtv8Z6nt off=2878717 len=1142785 2021-07-31T02:14:17.436 INFO:tasks.workunit.client.0.smithi045.stdout:op 2771 completed, throughput=5MB/sec 2021-07-31T02:14:18.437 INFO:tasks.workunit.client.0.smithi045.stdout: 541: throughput=4.99MB/sec pending data=0 2021-07-31T02:14:18.437 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-yDWgMLQbWguYSYl off=9742664 len=1286552 2021-07-31T02:14:18.444 INFO:tasks.workunit.client.0.smithi045.stdout:op 2772 completed, throughput=4.99MB/sec 2021-07-31T02:14:18.444 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj--umxl9ajXkj6xJv off=6865316 len=1318139 2021-07-31T02:14:18.450 INFO:tasks.workunit.client.0.smithi045.stdout:op 2773 completed, throughput=5MB/sec 2021-07-31T02:14:18.450 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ebAd-h8rWlehn2q off=16576840 len=876427 2021-07-31T02:14:18.452 INFO:tasks.workunit.client.0.smithi045.stdout:op 2774 completed, throughput=5MB/sec 2021-07-31T02:14:18.452 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-KPO0tLG_gAnWhy- off=19563689 len=748919 2021-07-31T02:14:18.456 INFO:tasks.workunit.client.0.smithi045.stdout:op 2775 completed, throughput=5MB/sec 2021-07-31T02:14:18.457 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-sJaRX2NbkZSeoJB off=5194418 len=923299 2021-07-31T02:14:18.461 INFO:tasks.workunit.client.0.smithi045.stdout:op 2776 completed, throughput=5MB/sec 2021-07-31T02:14:18.461 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-e8Oywb1vB3lpn9W off=3575758 len=1057550 2021-07-31T02:14:18.462 INFO:tasks.workunit.client.0.smithi045.stdout:op 2777 completed, throughput=5MB/sec 2021-07-31T02:14:18.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:18 smithi154 conmon[34553]: cluster 2021-07-31T02:14:18.179196+0000 mgr.x (mgr.24542) 1158 : cluster [DBG] pgmap v1684: 9 pgs: 9 active+clean; 127 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.4 MiB/s wr, 3 op/s 2021-07-31T02:14:18.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:18 smithi045 conmon[33655]: cluster 2021-07-31T02:14:18.179196+0000 mgr.x (mgr.24542) 1158 : cluster [DBG] 2021-07-31T02:14:18.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:18 smithi045 conmon[33655]: pgmap v1684: 9 pgs: 9 active+clean; 127 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.4 MiB/s wr, 3 op/s 2021-07-31T02:14:18.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:18 smithi045 conmon[39718]: cluster 2021-07-31T02:14:18.179196+0000 mgr.x (mgr.24542 2021-07-31T02:14:18.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:18 smithi045 conmon[39718]: ) 1158 : cluster [DBG] pgmap v1684: 9 pgs: 9 active+clean; 127 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.4 MiB/s wr, 3 op/s 2021-07-31T02:14:19.463 INFO:tasks.workunit.client.0.smithi045.stdout: 542: throughput=4.99MB/sec pending data=0 2021-07-31T02:14:19.463 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-5eWeShYaStqbsk3 off=1176850 len=1590600 2021-07-31T02:14:19.471 INFO:tasks.workunit.client.0.smithi045.stdout:op 2778 completed, throughput=5MB/sec 2021-07-31T02:14:19.471 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-iXRulBWzMJKUC1p off=5486720 len=1747764 2021-07-31T02:14:19.473 INFO:tasks.workunit.client.0.smithi045.stdout:op 2779 completed, throughput=5MB/sec 2021-07-31T02:14:19.473 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-GNtppadkXOQvN_5 off=0 len=1208026 2021-07-31T02:14:19.479 INFO:tasks.workunit.client.0.smithi045.stdout:op 2780 completed, throughput=5MB/sec 2021-07-31T02:14:20.479 INFO:tasks.workunit.client.0.smithi045.stdout: 543: throughput=4.99MB/sec pending data=0 2021-07-31T02:14:20.479 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-6MgfWj9ssyQTJ3X off=8391889 len=1587518 2021-07-31T02:14:20.487 INFO:tasks.workunit.client.0.smithi045.stdout:op 2781 completed, throughput=4.99MB/sec 2021-07-31T02:14:20.487 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-LNSh7ITMZNy4MiK off=2660823 len=901942 2021-07-31T02:14:20.492 INFO:tasks.workunit.client.0.smithi045.stdout:op 2782 completed, throughput=5MB/sec 2021-07-31T02:14:20.492 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-OWgimuILaRdeBte off=18410049 len=1294324 2021-07-31T02:14:20.500 INFO:tasks.workunit.client.0.smithi045.stdout:op 2783 completed, throughput=5MB/sec 2021-07-31T02:14:20.500 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-N6gdfmxcbh0g1pG off=165310 len=860252 2021-07-31T02:14:20.502 INFO:tasks.workunit.client.0.smithi045.stdout:op 2784 completed, throughput=5MB/sec 2021-07-31T02:14:20.502 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-MQL8Y_c7C-_R9Xe off=13798438 len=1830365 2021-07-31T02:14:20.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:20 smithi154 conmon[34553]: cluster 2021-07-31T02:14:20.179708+0000 mgr.x (mgr.24542) 1159 : cluster [DBG] pgmap v1685: 9 pgs: 9 active+clean; 127 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 2.8 MiB/s rd, 3.6 MiB/s wr, 5 op/s 2021-07-31T02:14:20.503 INFO:tasks.workunit.client.0.smithi045.stdout:op 2785 completed, throughput=5MB/sec 2021-07-31T02:14:20.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:20 smithi045 conmon[33655]: cluster 2021-07-31T02:14:20.179708+0000 mgr.x (mgr.24542) 1159 : cluster [DBG] pgmap v1685: 9 pgs: 9 active+clean; 127 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 2.8 MiB/s rd, 3.6 MiB/s wr, 5 op/s 2021-07-31T02:14:20.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:20 smithi045 conmon[39718]: cluster 2021-07-31T02:14:20.179708+0000 mgr.x (mgr.24542) 2021-07-31T02:14:20.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:20 smithi045 conmon[39718]: 1159 : cluster [DBG] pgmap v1685: 9 pgs: 9 active+clean; 127 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 2.8 MiB/s rd, 3.6 MiB/s wr, 5 op/s 2021-07-31T02:14:21.503 INFO:tasks.workunit.client.0.smithi045.stdout: 544: throughput=4.99MB/sec pending data=0 2021-07-31T02:14:21.504 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-7_IBBnlWIDAoPyQ off=3218091 len=2007254 2021-07-31T02:14:21.512 INFO:tasks.workunit.client.0.smithi045.stdout:op 2786 completed, throughput=5MB/sec 2021-07-31T02:14:21.512 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-PzhmJL0IbJHLeXE off=5049535 len=1636660 2021-07-31T02:14:21.521 INFO:tasks.workunit.client.0.smithi045.stdout:op 2787 completed, throughput=5MB/sec 2021-07-31T02:14:21.522 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-KOFGaZXQPZ9WW3v off=3222356 len=246120 2021-07-31T02:14:21.522 INFO:tasks.workunit.client.0.smithi045.stdout:op 2788 completed, throughput=5MB/sec 2021-07-31T02:14:21.522 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-MRNSOetbYg7OOHH off=500624 len=1991385 2021-07-31T02:14:21.531 INFO:tasks.workunit.client.0.smithi045.stdout:op 2789 completed, throughput=5MB/sec 2021-07-31T02:14:22.532 INFO:tasks.workunit.client.0.smithi045.stdout: 545: throughput=4.99MB/sec pending data=0 2021-07-31T02:14:22.532 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-dcLhomXqvFhx-eF off=2579978 len=1311264 2021-07-31T02:14:22.540 INFO:tasks.workunit.client.0.smithi045.stdout:op 2790 completed, throughput=5MB/sec 2021-07-31T02:14:22.540 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Mkcb6sJxihI50J_ off=4286084 len=61068 2021-07-31T02:14:22.541 INFO:tasks.workunit.client.0.smithi045.stdout:op 2791 completed, throughput=5MB/sec 2021-07-31T02:14:22.541 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-m49oBbT1NmepZz4 off=8402885 len=428810 2021-07-31T02:14:22.544 INFO:tasks.workunit.client.0.smithi045.stdout:op 2792 completed, throughput=5MB/sec 2021-07-31T02:14:22.545 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-sxGEyzgouwlkAfR off=22486849 len=868968 2021-07-31T02:14:22.545 INFO:tasks.workunit.client.0.smithi045.stdout:op 2793 completed, throughput=5MB/sec 2021-07-31T02:14:22.546 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-YcwjsPiOD4fekr2 off=674898 len=551807 2021-07-31T02:14:22.550 INFO:tasks.workunit.client.0.smithi045.stdout:op 2794 completed, throughput=5MB/sec 2021-07-31T02:14:22.551 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-L13LQJWDQUpGSSX off=14613724 len=1403900 2021-07-31T02:14:22.552 INFO:tasks.workunit.client.0.smithi045.stdout:op 2795 completed, throughput=5MB/sec 2021-07-31T02:14:23.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:23 smithi154 conmon[34553]: cluster 2021-07-31T02:14:22.180072+0000 mgr.x (mgr.24542) 1160 : cluster [DBG] pgmap v1686: 9 pgs: 9 active+clean; 127 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.3 MiB/s wr, 3 op/s 2021-07-31T02:14:23.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:23 smithi154 conmon[34553]: 2021-07-31T02:14:23.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:23 smithi045 conmon[33655]: cluster 2021-07-31T02:14:22.180072+0000 mgr.x (mgr.24542) 1160 : cluster [DBG] pgmap v1686: 9 pgs: 9 active+clean; 127 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.3 MiB/s wr, 3 op/s 2021-07-31T02:14:23.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:23 smithi045 conmon[39718]: cluster 2021-07-31T02:14:22 2021-07-31T02:14:23.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:23 smithi045 conmon[39718]: .180072+0000 mgr.x (mgr.24542) 1160 : cluster [DBG] pgmap v1686: 9 pgs: 9 active+clean; 127 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.3 MiB/s wr, 3 op/s 2021-07-31T02:14:23.552 INFO:tasks.workunit.client.0.smithi045.stdout: 546: throughput=4.99MB/sec pending data=0 2021-07-31T02:14:23.553 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-G8kAbr4fjxpMiQL off=4232691 len=1543330 2021-07-31T02:14:23.560 INFO:tasks.workunit.client.0.smithi045.stdout:op 2796 completed, throughput=5MB/sec 2021-07-31T02:14:23.560 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-qs6ft1T0-a4z2S1 off=2999151 len=1424184 2021-07-31T02:14:23.567 INFO:tasks.workunit.client.0.smithi045.stdout:op 2797 completed, throughput=5MB/sec 2021-07-31T02:14:23.568 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-NJKRBqx9VvZOHjI off=3946609 len=1422679 2021-07-31T02:14:23.573 INFO:tasks.workunit.client.0.smithi045.stdout:op 2798 completed, throughput=5MB/sec 2021-07-31T02:14:24.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:14:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:14:23.555Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:14:24.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:14:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:14:23.555Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:14:24.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:14:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:14:23.555Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:14:24.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:24 smithi154 conmon[34553]: cluster 2021-07-31T02:14:24.180687+0000 mgr.x (mgr.24542) 1161 : cluster [DBG] 2021-07-31T02:14:24.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:24 smithi154 conmon[34553]: pgmap v1687: 9 pgs: 9 active+clean; 127 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 2.5 MiB/s rd, 3.8 MiB/s wr, 5 op/s 2021-07-31T02:14:24.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:24 smithi045 conmon[39718]: cluster 2021-07-31T02:14:24.180687+0000 mgr.x ( 2021-07-31T02:14:24.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:24 smithi045 conmon[39718]: mgr.24542) 1161 : cluster [DBG] pgmap v1687: 9 pgs: 9 active+clean; 127 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 2.5 MiB/s rd, 3.8 MiB/s wr, 5 op/s 2021-07-31T02:14:24.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:24 smithi045 conmon[33655]: cluster 2021-07-31T02:14:24.180687+0000 mgr.x (mgr. 2021-07-31T02:14:24.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:24 smithi045 conmon[33655]: 24542) 1161 : cluster [DBG] pgmap v1687: 9 pgs: 9 active+clean; 127 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 2.5 MiB/s rd, 3.8 MiB/s wr, 5 op/s 2021-07-31T02:14:24.574 INFO:tasks.workunit.client.0.smithi045.stdout: 547: throughput=4.99MB/sec pending data=0 2021-07-31T02:14:24.574 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-2vkgMIbkphZBiK8 off=1560341 len=386785 2021-07-31T02:14:24.575 INFO:tasks.workunit.client.0.smithi045.stdout:op 2799 completed, throughput=4.99MB/sec 2021-07-31T02:14:24.575 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-BWsV1KcCkS_KhRJ off=4412751 len=502908 2021-07-31T02:14:24.579 INFO:tasks.workunit.client.0.smithi045.stdout:op 2800 completed, throughput=4.99MB/sec 2021-07-31T02:14:24.579 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-K38moGzQRlgBuup off=10409763 len=2075298 2021-07-31T02:14:24.588 INFO:tasks.workunit.client.0.smithi045.stdout:op 2801 completed, throughput=5MB/sec 2021-07-31T02:14:24.588 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-w1O_RZs5SIZznUc off=4269277 len=1687712 2021-07-31T02:14:24.591 INFO:tasks.workunit.client.0.smithi045.stdout:op 2802 completed, throughput=5MB/sec 2021-07-31T02:14:24.591 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-fRpKIjKS9_33BBy off=1835283 len=500788 2021-07-31T02:14:24.595 INFO:tasks.workunit.client.0.smithi045.stdout:op 2803 completed, throughput=5MB/sec 2021-07-31T02:14:24.595 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-aoRal76aItlLaVs off=4058833 len=1020506 2021-07-31T02:14:24.596 INFO:tasks.workunit.client.0.smithi045.stdout:op 2804 completed, throughput=5MB/sec 2021-07-31T02:14:25.597 INFO:tasks.workunit.client.0.smithi045.stdout: 548: throughput=4.99MB/sec pending data=0 2021-07-31T02:14:25.597 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-x9EIRjJzD-HEhDT off=5875493 len=53883 2021-07-31T02:14:25.598 INFO:tasks.workunit.client.0.smithi045.stdout:op 2805 completed, throughput=4.99MB/sec 2021-07-31T02:14:25.598 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-QmUJJJYyGT0qyXQ off=227329 len=903051 2021-07-31T02:14:25.599 INFO:tasks.workunit.client.0.smithi045.stdout:op 2806 completed, throughput=4.99MB/sec 2021-07-31T02:14:25.600 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-gs5Z6ZhcCakqjHO off=819120 len=1577363 2021-07-31T02:14:25.602 INFO:tasks.workunit.client.0.smithi045.stdout:op 2807 completed, throughput=5MB/sec 2021-07-31T02:14:25.602 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-T1fOW0dD5w3bMUn off=5667830 len=1363542 2021-07-31T02:14:25.610 INFO:tasks.workunit.client.0.smithi045.stdout:op 2808 completed, throughput=5MB/sec 2021-07-31T02:14:25.611 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-DUkIeIKq_5h9J-n off=598467 len=1716491 2021-07-31T02:14:25.620 INFO:tasks.workunit.client.0.smithi045.stdout:op 2809 completed, throughput=5MB/sec 2021-07-31T02:14:26.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:26 smithi154 conmon[34553]: cluster 2021-07-31T02:14:26.181196+0000 mgr.x (mgr.24542) 1162 : cluster 2021-07-31T02:14:26.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:26 smithi154 conmon[34553]: [DBG] pgmap v1688: 9 pgs: 9 active+clean; 127 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 1.5 MiB/s rd, 2.7 MiB/s wr, 3 op/s 2021-07-31T02:14:26.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:26 smithi045 conmon[33655]: cluster 2021-07-31T02:14:26.181196+0000 mgr.x (mgr.24542) 1162 2021-07-31T02:14:26.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:26 smithi045 conmon[33655]: : cluster [DBG] pgmap v1688: 9 pgs: 9 active+clean; 127 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 1.5 MiB/s rd, 2.7 MiB/s wr, 3 op/s 2021-07-31T02:14:26.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:26 smithi045 conmon[39718]: cluster 2021-07-31T02:14:26.181196+0000 mgr.x 2021-07-31T02:14:26.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:26 smithi045 conmon[39718]: (mgr.24542) 1162 : cluster [DBG] pgmap v1688: 9 pgs: 9 active+clean; 127 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 1.5 MiB/s rd, 2.7 MiB/s wr, 3 op/s 2021-07-31T02:14:26.620 INFO:tasks.workunit.client.0.smithi045.stdout: 549: throughput=4.99MB/sec pending data=0 2021-07-31T02:14:26.620 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-alEdEWKwzPtBac9 off=6195243 len=678191 2021-07-31T02:14:26.625 INFO:tasks.workunit.client.0.smithi045.stdout:op 2810 completed, throughput=4.99MB/sec 2021-07-31T02:14:26.625 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-fNtGArrH1qICo1h off=5040157 len=1175686 2021-07-31T02:14:26.627 INFO:tasks.workunit.client.0.smithi045.stdout:op 2811 completed, throughput=5MB/sec 2021-07-31T02:14:26.627 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-vwtteh_odr1IxfR off=12135711 len=74603 2021-07-31T02:14:26.628 INFO:tasks.workunit.client.0.smithi045.stdout:op 2812 completed, throughput=5MB/sec 2021-07-31T02:14:26.628 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-yX7TXhfP5qb75my off=6342695 len=1082679 2021-07-31T02:14:26.634 INFO:tasks.workunit.client.0.smithi045.stdout:op 2813 completed, throughput=5MB/sec 2021-07-31T02:14:26.634 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-IF-IRrkbslpSdXk off=231489 len=216646 2021-07-31T02:14:26.635 INFO:tasks.workunit.client.0.smithi045.stdout:op 2814 completed, throughput=5MB/sec 2021-07-31T02:14:26.635 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-E14WJkE9j0cJysc off=2148682 len=895251 2021-07-31T02:14:26.636 INFO:tasks.workunit.client.0.smithi045.stdout:op 2815 completed, throughput=5MB/sec 2021-07-31T02:14:26.636 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-qd76fWKuCudRjpI off=5705980 len=1714787 2021-07-31T02:14:26.645 INFO:tasks.workunit.client.0.smithi045.stdout:op 2816 completed, throughput=5MB/sec 2021-07-31T02:14:27.646 INFO:tasks.workunit.client.0.smithi045.stdout: 550: throughput=4.99MB/sec pending data=0 2021-07-31T02:14:27.646 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-46p1NP229JO0Fm8 off=9127311 len=1232031 2021-07-31T02:14:27.648 INFO:tasks.workunit.client.0.smithi045.stdout:op 2817 completed, throughput=5MB/sec 2021-07-31T02:14:27.648 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-7zB_1yUrHjhyXMZ off=1732412 len=860065 2021-07-31T02:14:27.650 INFO:tasks.workunit.client.0.smithi045.stdout:op 2818 completed, throughput=5MB/sec 2021-07-31T02:14:27.650 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-81Pj-mhxRDJ9c8t off=360551 len=244990 2021-07-31T02:14:27.651 INFO:tasks.workunit.client.0.smithi045.stdout:op 2819 completed, throughput=5MB/sec 2021-07-31T02:14:27.651 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-HH1X5P-CmHlsOUW off=5300590 len=302594 2021-07-31T02:14:27.652 INFO:tasks.workunit.client.0.smithi045.stdout:op 2820 completed, throughput=5MB/sec 2021-07-31T02:14:27.652 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-nXJjCfSZN8dbQ7o off=1986641 len=410359 2021-07-31T02:14:27.653 INFO:tasks.workunit.client.0.smithi045.stdout:op 2821 completed, throughput=5MB/sec 2021-07-31T02:14:27.653 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-q6vvCIDYwR3tINX off=0 len=1506357 2021-07-31T02:14:27.659 INFO:tasks.workunit.client.0.smithi045.stdout:op 2822 completed, throughput=5MB/sec 2021-07-31T02:14:28.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:28 smithi154 conmon[34553]: audit 2021-07-31T02:14:28.029036+0000 mon.b (mon.2) 203 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:14:28.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:28 smithi154 conmon[34553]: audit 2021-07-31T02:14:28.029579+0000 mon.a (mon 2021-07-31T02:14:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:28 smithi154 conmon[34553]: .0) 1879 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:14:28.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:28 smithi045 conmon[39718]: audit 2021-07-31T02:14:28.029036+0000 mon.b (mon 2021-07-31T02:14:28.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:28 smithi045 conmon[39718]: .2) 203 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:14:28.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:28 smithi045 conmon[39718]: audit 2021-07-31T02:14:28.029579+0000 mon.a (mon.0) 1879 : 2021-07-31T02:14:28.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:28 smithi045 conmon[39718]: audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:14:28.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:28 smithi045 conmon[33655]: audit 2021-07-31T02:14:28.029036+0000 mon.b (mon.2) 203 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:14:28.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:28 smithi045 conmon[33655]: audit 2021-07-31T02:14:28.029579+0000 mon.a (mon.0) 1879 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:14:28.660 INFO:tasks.workunit.client.0.smithi045.stdout: 551: throughput=4.99MB/sec pending data=0 2021-07-31T02:14:28.660 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-znXlVP1DZIZGjyG off=1821501 len=960330 2021-07-31T02:14:28.665 INFO:tasks.workunit.client.0.smithi045.stdout:op 2823 completed, throughput=4.99MB/sec 2021-07-31T02:14:28.666 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-G5N7loPZk4C89Y7 off=3902960 len=605368 2021-07-31T02:14:28.669 INFO:tasks.workunit.client.0.smithi045.stdout:op 2824 completed, throughput=5MB/sec 2021-07-31T02:14:28.670 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-jKZcIi4C7gspZ_g off=7706355 len=1630112 2021-07-31T02:14:28.673 INFO:tasks.workunit.client.0.smithi045.stdout:op 2825 completed, throughput=5MB/sec 2021-07-31T02:14:28.673 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj--emCwoYVkHIgaGL off=9579829 len=1175051 2021-07-31T02:14:28.679 INFO:tasks.workunit.client.0.smithi045.stdout:op 2826 completed, throughput=5MB/sec 2021-07-31T02:14:28.679 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-FdYjvGQExBQTd_4 off=2461741 len=1366999 2021-07-31T02:14:28.686 INFO:tasks.workunit.client.0.smithi045.stdout:op 2827 completed, throughput=5MB/sec 2021-07-31T02:14:29.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:29 smithi154 conmon[34553]: audit 2021-07-31T02:14:28.050008+0000 mon.b (mon.2) 204 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:14:29.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:29 smithi154 conmon[34553]: audit 2021-07-31T02:14:28.050602+0000 mon.a (mon.0) 1880 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:14:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:29 smithi154 conmon[34553]: cluster 2021-07-31T02:14:28.181627+0000 mgr.x (mgr.24542) 1163 : cluster [DBG] pgmap v1689: 9 pgs: 9 active+clean; 127 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 1.5 MiB/s rd, 2.7 MiB/s wr, 3 op/s 2021-07-31T02:14:29.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:29 smithi045 conmon[39718]: audit 2021-07-31T02:14:28.050008+0000 mon.b (mon.2) 204 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:14:29.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:29 smithi045 conmon[39718]: 2021-07-31T02:14:29.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:29 smithi045 conmon[39718]: audit 2021-07-31T02:14:28.050602 2021-07-31T02:14:29.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:29 smithi045 conmon[39718]: +0000 mon.a (mon.0) 1880 : audit [INF] 2021-07-31T02:14:29.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:29 smithi045 conmon[39718]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:14:29.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:29 smithi045 conmon[39718]: cluster 2021-07-31 2021-07-31T02:14:29.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:29 smithi045 conmon[39718]: T02:14:28.181627+0000 2021-07-31T02:14:29.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:29 smithi045 conmon[39718]: mgr.x (mgr.24542) 1163 : cluster 2021-07-31T02:14:29.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:29 smithi045 conmon[39718]: [DBG] pgmap v1689: 9 pgs: 9 active+clean; 127 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 1.5 MiB/s rd, 2.7 MiB/s wr, 3 op/s 2021-07-31T02:14:29.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:29 smithi045 conmon[33655]: audit 2021-07-31T02:14:28.050008+0000 mon.b (mon.2) 204 : 2021-07-31T02:14:29.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:29 smithi045 conmon[33655]: audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:14:29.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:29 smithi045 conmon[33655]: audit 2021-07-31T02:14:28.050602+0000 mon.a (mon.0) 1880 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:14:29.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:29 smithi045 conmon[33655]: cluster 2021-07-31T02:14:28.181627+0000 mgr.x (mgr. 2021-07-31T02:14:29.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:29 smithi045 conmon[33655]: 24542) 1163 : cluster [DBG] pgmap v1689: 9 pgs: 9 active+clean; 127 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 1.5 MiB/s rd, 2.7 MiB/s wr, 3 op/s 2021-07-31T02:14:29.686 INFO:tasks.workunit.client.0.smithi045.stdout: 552: throughput=4.99MB/sec pending data=0 2021-07-31T02:14:29.686 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-GzWEkMXy0MfN7IA off=11617172 len=695378 2021-07-31T02:14:29.690 INFO:tasks.workunit.client.0.smithi045.stdout:op 2828 completed, throughput=4.99MB/sec 2021-07-31T02:14:29.691 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-trKQ7fGpo1rMzpC off=5892936 len=978588 2021-07-31T02:14:29.696 INFO:tasks.workunit.client.0.smithi045.stdout:op 2829 completed, throughput=5MB/sec 2021-07-31T02:14:29.696 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-RjfAyNCb_nBxG-v off=11134140 len=830612 2021-07-31T02:14:29.698 INFO:tasks.workunit.client.0.smithi045.stdout:op 2830 completed, throughput=5MB/sec 2021-07-31T02:14:29.698 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-OrjoM2ucoGoIIMI off=6502449 len=937288 2021-07-31T02:14:29.700 INFO:tasks.workunit.client.0.smithi045.stdout:op 2831 completed, throughput=5MB/sec 2021-07-31T02:14:29.700 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-rkxiqRTfsUCanIf off=7352284 len=1158418 2021-07-31T02:14:29.701 INFO:tasks.workunit.client.0.smithi045.stdout:op 2832 completed, throughput=5MB/sec 2021-07-31T02:14:30.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[34553]: cluster 2021-07-31T02:14:30.182083+0000 mgr.x (mgr.24542) 1164 : cluster [DBG] pgmap v1690: 9 pgs: 9 active+clean; 127 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 2.5 MiB/s rd, 3.9 MiB/s wr, 6 op/s 2021-07-31T02:14:30.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:30 smithi045 conmon[33655]: cluster 2021-07-31T02:14:30.182083+0000 mgr.x (mgr.24542) 1164 : cluster 2021-07-31T02:14:30.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:30 smithi045 conmon[33655]: [DBG] pgmap v1690: 9 pgs: 9 active+clean; 127 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 2.5 MiB/s rd, 3.9 MiB/s wr, 6 op/s 2021-07-31T02:14:30.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:30 smithi045 conmon[39718]: cluster 2021-07-31T02:14:30.182083 2021-07-31T02:14:30.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:30 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 1164 : cluster [DBG] pgmap v1690: 9 pgs: 9 active+clean; 127 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 2.5 MiB/s rd, 3.9 MiB/s wr, 6 op/s 2021-07-31T02:14:30.701 INFO:tasks.workunit.client.0.smithi045.stdout: 553: throughput=4.99MB/sec pending data=0 2021-07-31T02:14:30.701 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-e2oDdzBYmp08N9b off=1390070 len=53583 2021-07-31T02:14:30.702 INFO:tasks.workunit.client.0.smithi045.stdout:op 2833 completed, throughput=4.99MB/sec 2021-07-31T02:14:30.702 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-oWYhe3UNnbB6O7K off=5994940 len=1444895 2021-07-31T02:14:30.705 INFO:tasks.workunit.client.0.smithi045.stdout:op 2834 completed, throughput=4.99MB/sec 2021-07-31T02:14:30.705 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-YwQYZWrgl2nOLGZ off=6095055 len=14717 2021-07-31T02:14:30.705 INFO:tasks.workunit.client.0.smithi045.stdout:op 2835 completed, throughput=4.99MB/sec 2021-07-31T02:14:30.706 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-2m_AuBBVBqe_YXR off=3609921 len=1402699 2021-07-31T02:14:30.712 INFO:tasks.workunit.client.0.smithi045.stdout:op 2836 completed, throughput=5MB/sec 2021-07-31T02:14:30.713 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-mouBOotPgSkXu02 off=15512495 len=1055478 2021-07-31T02:14:30.720 INFO:tasks.workunit.client.0.smithi045.stdout:op 2837 completed, throughput=5MB/sec 2021-07-31T02:14:30.721 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-UkZ6-g6Jh3fm9ac off=4584740 len=948395 2021-07-31T02:14:30.725 INFO:tasks.workunit.client.0.smithi045.stdout:op 2838 completed, throughput=5MB/sec 2021-07-31T02:14:31.248 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: debug 2021-07-31T02:14:30.813+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:14:31.248 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:14:31.249 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:14:31.249 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: yield 2021-07-31T02:14:31.249 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:14:31.249 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:14:31.250 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:14:31.250 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:14:31.250 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:14:31.250 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:14:31.250 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:14:31.251 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:14:31.251 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:14:31.251 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:14:31.251 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:14:31.252 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: debug 2021-07-31T02:14:30.825+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:14:31.252 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:14:31.252 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:14:31.252 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: yield 2021-07-31T02:14:31.252 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:14:31.253 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:14:31.253 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:14:31.253 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:14:31.253 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:14:31.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:14:31.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:14:31.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:14:31.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:14:31.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:14:31.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:14:31.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: debug 2021-07-31T02:14:30.850+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:14:31.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:14:31.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:14:31.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: yield 2021-07-31T02:14:31.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:14:31.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:14:31.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:14:31.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:14:31.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:14:31.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:14:31.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:14:31.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:14:31.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:14:31.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:14:31.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:14:31.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: debug 2021-07-31T02:14:30.851+0000 7fcb9195d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:14:31.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:14:31.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:14:31.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: yield 2021-07-31T02:14:31.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:14:31.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:14:31.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:14:31.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:14:31.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:14:31.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:14:31.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:14:31.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:14:31.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:14:31.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:14:31.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:14:31.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: 2021-07-31T02:14:31.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:14:31.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: 2021-07-31T02:14:31.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:14:31.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:14:31.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:14:31.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:14:31.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:14:31.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:14:31.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:14:31.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:14:31.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:14:31.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:14:31.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:14:31.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:14:31.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:14:31.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:14:31.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:14:31.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:14:31.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:14:31.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:14:31.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: raise 2021-07-31T02:14:31.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:14:31.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:14:31.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:14:31.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:14:31.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:14:31.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: debug 2021-07-31T02:14:30.859+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:14:31.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:14:31.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:14:31.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: yield 2021-07-31T02:14:31.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:14:31.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:14:31.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:14:31.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:14:31.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:14:31.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:14:31.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:14:31.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:14:31.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:14:31.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:14:31.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:14:31.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: debug 2021-07-31T02:14:30.859+0000 7fcb9215e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:14:31.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:14:31.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:14:31.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: yield 2021-07-31T02:14:31.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:14:31.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:14:31.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:14:31.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:14:31.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:14:31.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:14:31.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:14:31.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:14:31.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:14:31.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:14:31.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:14:31.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: 2021-07-31T02:14:31.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:14:31.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: 2021-07-31T02:14:31.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:14:31.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:14:31.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:14:31.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:14:31.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:14:31.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:14:31.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:14:31.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:14:31.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:14:31.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:14:31.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:14:31.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:14:31.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:14:31.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:14:31.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:14:31.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:14:31.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:14:31.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:14:31.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: raise 2021-07-31T02:14:31.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:14:31.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:14:31.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:14:31.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:14:31.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:14:30 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:14:31.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: audit 2021-07-31T02:14:30.768293+0000 mon.b (mon.2) 205 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:14:31.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: audit 2021-07-31T02:14:30.769780+0000 mon.b (mon.2) 206 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:14:31.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: audit 2021-07-31T 2021-07-31T02:14:31.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: 02:14:30.770936+0000 mon.b (mon.2) 207 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:14:31.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: cephadm 2021- 2021-07-31T02:14:31.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: 07-31T02:14:30.771648+0000 2021-07-31T02:14:31.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: mgr.x (mgr.24542) 1165 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:14:31.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: 2021-07-31T02:14:31.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: cephadm 2021-07-31T02: 2021-07-31T02:14:31.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: 14:30.814688+0000 mgr.x (mgr. 2021-07-31T02:14:31.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: 24542) 1166 : cephadm [ERR] 2021-07-31T02:14:31.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:14:31.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:14:31.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: yield 2021-07-31T02:14:31.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:14:31.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:14:31.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:14:31.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:14:31.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:14:31.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:14:31.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:14:31.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:14:31.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:14:31.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: task.result() 2021-07-31T02:14:31.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:14:31.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:14:30.815126+0000 mgr.x (mgr.24542) 1167 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:14:31.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:14:30.826151+0000 2021-07-31T02:14:31.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: mgr.x (mgr.24542) 1168 : cephadm [ERR] 2021-07-31T02:14:31.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:14:31.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:14:31.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: yield 2021-07-31T02:14:31.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:14:31.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:14:31.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:14:31.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:14:31.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:14:31.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:14:31.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:14:31.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:14:31.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:14:31.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: task.result() 2021-07-31T02:14:31.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:14:31.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:14:30.827117+0000 mgr.x (mgr.24542) 1169 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:14:31.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: cephadm 2021-07-31T02 2021-07-31T02:14:31.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: :14:30.851651+0000 mgr.x (mgr.24542) 1170 2021-07-31T02:14:31.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: : cephadm [ERR] 2021-07-31T02:14:31.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:14:31.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:14:31.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: yield 2021-07-31T02:14:31.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:14:31.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:14:31.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:14:31.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:14:31.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:14:31.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:14:31.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:14:31.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:14:31.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:14:31.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: task.result() 2021-07-31T02:14:31.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:14:31.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:14:30.852306+0000 mgr.x (mgr.24542) 1171 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:14:31.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:14:31.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:14:31.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: yield 2021-07-31T02:14:31.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:14:31.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:14:31.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:14:31.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:14:31.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:14:31.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:14:31.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:14:31.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:14:31.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:14:31.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: task.result() 2021-07-31T02:14:31.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:14:31.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: 2021-07-31T02:14:31.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:14:31.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: 2021-07-31T02:14:31.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:14:31.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:14:31.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:14:31.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:14:31.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:14:31.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:14:31.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:14:31.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:14:31.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:14:31.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:14:31.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:14:31.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:14:31.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:14:31.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:14:31.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:14:31.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:14:31.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:14:31.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:14:31.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: raise 2021-07-31T02:14:31.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:14:31.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:14:31.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:14:31.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:14:31.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:14:31.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:14:30.860492+0000 mgr.x (mgr.24542) 1172 : cephadm [ERR] 2021-07-31T02:14:31.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:14:31.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:14:31.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: yield 2021-07-31T02:14:31.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:14:31.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:14:31.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:14:31.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:14:31.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:14:31.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:14:31.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:14:31.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:14:31.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:14:31.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: task.result() 2021-07-31T02:14:31.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:14:31.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:14:30.860890+0000 mgr.x (mgr.24542) 1173 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:14:31.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:14:31.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:14:31.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: yield 2021-07-31T02:14:31.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:14:31.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:14:31.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:14:31.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:14:31.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:14:31.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:14:31.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:14:31.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:14:31.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:14:31.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: task.result() 2021-07-31T02:14:31.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:14:31.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: 2021-07-31T02:14:31.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:14:31.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: 2021-07-31T02:14:31.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:14:31.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:14:31.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:14:31.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:14:31.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:14:31.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:14:31.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:14:31.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:14:31.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:14:31.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:14:31.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:14:31.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:14:31.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:14:31.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:14:31.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:14:31.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:14:31.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:14:31.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:14:31.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: raise 2021-07-31T02:14:31.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:14:31.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:14:31.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:14:31.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:14:31.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:31 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:14:31.540 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: audit 2021-07-31T02:14:30.768293 2021-07-31T02:14:31.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: +0000 mon.b (mon.2) 205 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:14:31.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: audit 2021- 2021-07-31T02:14:31.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: 07-31T02:14:30.769780+0000 mon.b (mon.2) 206 : audit [DBG] 2021-07-31T02:14:31.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:14:31.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: audit 2021-07-31T02:14:31.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: 2021 2021-07-31T02:14:31.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: -07-31T02:14:30.770936 2021-07-31T02:14:31.543 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: +0000 mon.b (mon.2) 207 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:14:31.543 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:14:30.771648 2021-07-31T02:14:31.543 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 1165 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:14:31.543 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: cephadm 2021-07-31 2021-07-31T02:14:31.543 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: T02:14:30.814688+0000 mgr.x (mgr.24542) 1166 : 2021-07-31T02:14:31.544 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: cephadm [ERR] 2021-07-31T02:14:31.544 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:14:31.544 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:14:31.544 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: yield 2021-07-31T02:14:31.545 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:14:31.545 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:14:31.545 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:14:31.545 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:14:31.545 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:14:31.546 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:14:31.546 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:14:31.546 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:14:31.546 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:14:31.547 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: task.result() 2021-07-31T02:14:31.547 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:14:31.547 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:14:30.815126+0000 mgr.x (mgr.24542) 1167 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:14:31.547 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:14:30.826151+0000 mgr.x 2021-07-31T02:14:31.547 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: (mgr.24542) 1168 : cephadm [ERR] 2021-07-31T02:14:31.548 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:14:31.548 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:14:31.548 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: yield 2021-07-31T02:14:31.548 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:14:31.549 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:14:31.549 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:14:31.549 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:14:31.549 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:14:31.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: audit 2021-07-31T02:14:30.768293+0000 mon.b ( 2021-07-31T02:14:31.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: mon.2) 205 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:14:31.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T02:14:31.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: 02:14:30.769780+0000 mon.b (mon.2) 206 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:14:31.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: audit 2021- 2021-07-31T02:14:31.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: 07-31T02:14:30.770936 2021-07-31T02:14:31.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: +0000 mon.b (mon.2) 207 : audit 2021-07-31T02:14:31.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:14:31.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: cephadm 2021 2021-07-31T02:14:31.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: -07-31T02:14: 2021-07-31T02:14:31.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: 30.771648+0000 mgr.x (mgr.24542 2021-07-31T02:14:31.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: ) 1165 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:14:31.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:14:31.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: 2021-07-31T 2021-07-31T02:14:31.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: 02:14:30.814688+0000 2021-07-31T02:14:31.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: mgr.x (mgr.24542) 1166 : cephadm [ERR] 2021-07-31T02:14:31.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: 2021-07-31T02:14:31.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:14:31.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:14:31.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: yield 2021-07-31T02:14:31.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:14:31.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:14:31.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:14:31.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:14:31.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:14:31.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:14:31.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:14:31.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:14:31.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:14:31.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: task.result() 2021-07-31T02:14:31.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:14:31.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:14:30.815126+0000 mgr.x (mgr.24542) 1167 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:14:31.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:14:31.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: 2021-07-31T 2021-07-31T02:14:31.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: 02:14:30.826151+0000 2021-07-31T02:14:31.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: mgr.x (mgr.24542) 1168 : cephadm [ERR] 2021-07-31T02:14:31.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:14:31.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:14:31.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: yield 2021-07-31T02:14:31.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:14:31.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:14:31.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:14:31.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:14:31.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:14:31.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:14:31.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:14:31.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: task.result() 2021-07-31T02:14:31.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:14:31.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:14:30.827117+0000 mgr.x (mgr.24542) 1169 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:14:31.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:14:30.851651+0000 mgr.x (mgr.24542) 1170 : cephadm [ERR] 2021-07-31T02:14:31.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:14:31.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:14:31.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: yield 2021-07-31T02:14:31.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:14:31.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:14:31.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:14:31.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:14:31.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:14:31.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:14:31.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:14:31.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:14:31.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:14:31.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: task.result() 2021-07-31T02:14:31.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:14:31.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:14 2021-07-31T02:14:31.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: :30.852306+0000 mgr.x (mgr.24542) 1171 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:14:31.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:14:31.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:14:31.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: yield 2021-07-31T02:14:31.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:14:31.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:14:31.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:14:31.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:14:31.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:14:31.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:14:31.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:14:31.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:14:31.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:14:31.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: task.result() 2021-07-31T02:14:31.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:14:31.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: 2021-07-31T02:14:31.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:14:31.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: 2021-07-31T02:14:31.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:14:31.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:14:31.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:14:31.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:14:31.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:14:31.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:14:31.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: task.result() 2021-07-31T02:14:31.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:14:31.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:14:30.827117+0000 mgr.x (mgr.24542) 1169 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:14:31.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: cephadm 2021- 2021-07-31T02:14:31.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: 07-31T02:14:30.851651 2021-07-31T02:14:31.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 2021-07-31T02:14:31.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: 1170 : cephadm [ERR] 2021-07-31T02:14:31.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:14:31.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:14:31.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: yield 2021-07-31T02:14:31.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:14:31.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:14:31.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:14:31.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:14:31.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:14:31.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:14:31.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:14:31.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:14:31.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:14:31.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: task.result() 2021-07-31T02:14:31.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:14:31.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:14:30.852306 2021-07-31T02:14:31.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 1171 : cephadm 2021-07-31T02:14:31.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:14:31.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:14:31.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:14:31.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: yield 2021-07-31T02:14:31.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:14:31.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:14:31.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:14:31.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:14:31.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:14:31.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:14:31.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:14:31.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:14:31.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:14:31.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:14:31.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:14:31.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:14:31.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:14:31.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:14:31.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:14:31.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:14:31.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:14:31.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:14:31.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:14:31.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:14:31.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:14:31.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:14:31.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:14:31.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:14:31.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:14:31.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:14:31.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:14:31.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: raise 2021-07-31T02:14:31.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:14:31.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:14:31.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:14:31.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:14:31.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:14:31.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:14:30.860492+0000 mgr.x (mgr.24542) 1172 : cephadm [ERR] 2021-07-31T02:14:31.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:14:31.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:14:31.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: yield 2021-07-31T02:14:31.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:14:31.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:14:31.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:14:31.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:14:31.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:14:31.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:14:31.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:14:31.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:14:31.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:14:31.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: task.result() 2021-07-31T02:14:31.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:14:31.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:14:30.860890+0000 mgr.x (mgr.24542) 1173 : 2021-07-31T02:14:31.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: task.result() 2021-07-31T02:14:31.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:14:31.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: 2021-07-31T02:14:31.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:14:31.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: 2021-07-31T02:14:31.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:14:31.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:14:31.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:14:31.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:14:31.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:14:31.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:14:31.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:14:31.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:14:31.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:14:31.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:14:31.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:14:31.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:14:31.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:14:31.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:14:31.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:14:31.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:14:31.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:14:31.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:14:31.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: raise 2021-07-31T02:14:31.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:14:31.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:14:31.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:14:31.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:14:31.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:14:31.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:14:30.860492+0000 mgr.x (mgr.24542) 1172 : cephadm [ERR] 2021-07-31T02:14:31.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:14:31.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:14:31.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: yield 2021-07-31T02:14:31.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:14:31.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:14:31.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:14:31.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:14:31.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:14:31.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:14:31.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:14:31.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:14:31.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:14:31.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:14:31.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:14:31.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: yield 2021-07-31T02:14:31.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:14:31.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:14:31.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:14:31.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:14:31.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:14:31.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:14:31.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:14:31.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:14:31.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:14:31.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: task.result() 2021-07-31T02:14:31.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:14:31.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: 2021-07-31T02:14:31.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:14:31.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: 2021-07-31T02:14:31.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:14:31.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:14:31.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:14:31.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:14:31.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:14:31.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:14:31.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:14:31.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:14:31.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:14:31.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:14:31.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:14:31.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:14:31.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:14:31.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:14:31.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:14:31.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:14:31.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:14:31.608 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:14:31.608 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: raise 2021-07-31T02:14:31.608 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:14:31.609 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:14:31.609 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:14:31.609 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:14:31.609 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:14:31.610 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:14:31.610 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: task.result() 2021-07-31T02:14:31.611 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:14:31.611 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:14:30.860890+0000 mgr.x (mgr.24542) 1173 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:14:31.611 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:14:31.611 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:14:31.612 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: yield 2021-07-31T02:14:31.612 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:14:31.612 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:14:31.612 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:14:31.612 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:14:31.613 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:14:31.613 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:14:31.613 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:14:31.613 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:14:31.614 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:14:31.614 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: task.result() 2021-07-31T02:14:31.614 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:14:31.614 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: 2021-07-31T02:14:31.614 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:14:31.615 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: 2021-07-31T02:14:31.615 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:14:31.615 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:14:31.615 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:14:31.616 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:14:31.616 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:14:31.616 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:14:31.616 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:14:31.616 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:14:31.617 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:14:31.617 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:14:31.617 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:14:31.617 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:14:31.618 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:14:31.618 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:14:31.618 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:14:31.618 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:14:31.618 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:14:31.619 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:14:31.619 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: raise 2021-07-31T02:14:31.619 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:14:31.619 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:14:31.619 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:14:31.620 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:14:31.620 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:31 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:14:31.725 INFO:tasks.workunit.client.0.smithi045.stdout: 554: throughput=4.99MB/sec pending data=0 2021-07-31T02:14:31.726 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-duGJgktSruP4pL3 off=13314916 len=929709 2021-07-31T02:14:31.727 INFO:tasks.workunit.client.0.smithi045.stdout:op 2839 completed, throughput=4.99MB/sec 2021-07-31T02:14:31.727 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-hpeh4IveRCOwcAp off=17535149 len=2045751 2021-07-31T02:14:31.735 INFO:tasks.workunit.client.0.smithi045.stdout:op 2840 completed, throughput=5MB/sec 2021-07-31T02:14:31.735 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ZVp5aPQH78sHqK- off=5495890 len=1274434 2021-07-31T02:14:31.741 INFO:tasks.workunit.client.0.smithi045.stdout:op 2841 completed, throughput=5MB/sec 2021-07-31T02:14:31.742 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Xj0r6OAvFJKeK-i off=14810673 len=2000176 2021-07-31T02:14:31.745 INFO:tasks.workunit.client.0.smithi045.stdout:op 2842 completed, throughput=5MB/sec 2021-07-31T02:14:32.745 INFO:tasks.workunit.client.0.smithi045.stdout: 555: throughput=4.99MB/sec pending data=0 2021-07-31T02:14:32.746 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-fml6HSsWmBvKuYR off=8290793 len=571476 2021-07-31T02:14:32.749 INFO:tasks.workunit.client.0.smithi045.stdout:op 2843 completed, throughput=4.99MB/sec 2021-07-31T02:14:32.750 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-BEXFG8d2toofyiY off=20869033 len=751169 2021-07-31T02:14:32.754 INFO:tasks.workunit.client.0.smithi045.stdout:op 2844 completed, throughput=4.99MB/sec 2021-07-31T02:14:32.754 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-PyH2Dty4TBYica7 off=2846714 len=360039 2021-07-31T02:14:32.755 INFO:tasks.workunit.client.0.smithi045.stdout:op 2845 completed, throughput=5MB/sec 2021-07-31T02:14:32.755 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-QUp1spqzRMLXZnb off=12976871 len=1098188 2021-07-31T02:14:32.757 INFO:tasks.workunit.client.0.smithi045.stdout:op 2846 completed, throughput=5MB/sec 2021-07-31T02:14:32.757 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ETg0z_Z12hK-iFO off=4517788 len=1622675 2021-07-31T02:14:32.764 INFO:tasks.workunit.client.0.smithi045.stdout:op 2847 completed, throughput=5MB/sec 2021-07-31T02:14:33.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:33 smithi154 conmon[34553]: cluster 2021-07-31T02:14:32.182517+0000 mgr.x (mgr.24542) 1174 : cluster [DBG] pgmap v1691: 9 pgs: 9 active+clean; 127 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 1.6 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2021-07-31T02:14:33.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:33 smithi045 conmon[33655]: cluster 2021-07-31T02:14:32.182517+0000 mgr.x (mgr.24542) 2021-07-31T02:14:33.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:33 smithi045 conmon[33655]: 1174 : cluster [DBG] pgmap v1691: 9 pgs: 9 active+clean; 127 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 1.6 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2021-07-31T02:14:33.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:33 smithi045 conmon[39718]: cluster 2021-07-31T02:14:32.182517+0000 mgr.x (mgr.24542) 2021-07-31T02:14:33.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:33 smithi045 conmon[39718]: 1174 : cluster [DBG] pgmap v1691: 9 pgs: 9 active+clean; 127 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 1.6 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2021-07-31T02:14:33.765 INFO:tasks.workunit.client.0.smithi045.stdout: 556: throughput=4.99MB/sec pending data=0 2021-07-31T02:14:33.765 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-CO8GIgKZ8C3FJCx off=1319977 len=1960763 2021-07-31T02:14:33.776 INFO:tasks.workunit.client.0.smithi045.stdout:op 2848 completed, throughput=4.99MB/sec 2021-07-31T02:14:33.777 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj--umxl9ajXkj6xJv off=2481846 len=1698039 2021-07-31T02:14:33.783 INFO:tasks.workunit.client.0.smithi045.stdout:op 2849 completed, throughput=5MB/sec 2021-07-31T02:14:33.784 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-L4K4DyprlU1U46c off=7665068 len=1216693 2021-07-31T02:14:33.784 INFO:tasks.workunit.client.0.smithi045.stdout:op 2850 completed, throughput=5MB/sec 2021-07-31T02:14:33.785 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-X_rsH1ZAkRCdw5x off=14829612 len=1283580 2021-07-31T02:14:33.786 INFO:tasks.workunit.client.0.smithi045.stdout:op 2851 completed, throughput=5MB/sec 2021-07-31T02:14:34.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:14:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:14:33.555Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:14:34.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:14:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:14:33.555Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:14:34.027 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:14:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:14:33.555Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:14:34.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:34 smithi154 conmon[34553]: cluster 2021-07-31T02:14:34.182958+0000 mgr.x (mgr.24542) 1175 : cluster [DBG] pgmap v1692: 9 pgs: 9 active+clean; 127 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 2.5 MiB/s rd, 3.9 MiB/s wr, 6 op/s 2021-07-31T02:14:34.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:34 smithi045 conmon[33655]: cluster 2021-07-31T02:14:34.182958+0000 mgr.x (mgr.24542) 1175 : cluster 2021-07-31T02:14:34.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:34 smithi045 conmon[33655]: [DBG] pgmap v1692: 9 pgs: 9 active+clean; 127 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 2.5 MiB/s rd, 3.9 MiB/s wr, 6 op/s 2021-07-31T02:14:34.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:34 smithi045 conmon[39718]: cluster 2021-07-31T02:14:34. 2021-07-31T02:14:34.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:34 smithi045 conmon[39718]: 182958+0000 mgr.x (mgr.24542) 1175 : cluster [DBG] pgmap v1692: 9 pgs: 9 active+clean; 127 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 2.5 MiB/s rd, 3.9 MiB/s wr, 6 op/s 2021-07-31T02:14:34.786 INFO:tasks.workunit.client.0.smithi045.stdout: 557: throughput=4.99MB/sec pending data=0 2021-07-31T02:14:34.787 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-fBCunYw1I9ZdN-T off=1128742 len=60808 2021-07-31T02:14:34.787 INFO:tasks.workunit.client.0.smithi045.stdout:op 2852 completed, throughput=4.99MB/sec 2021-07-31T02:14:34.788 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-bHydncoON19FU3d off=6698322 len=330660 2021-07-31T02:14:34.788 INFO:tasks.workunit.client.0.smithi045.stdout:op 2853 completed, throughput=4.99MB/sec 2021-07-31T02:14:34.788 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-MTNlXHEI6WoVDsD off=3828005 len=950865 2021-07-31T02:14:34.794 INFO:tasks.workunit.client.0.smithi045.stdout:op 2854 completed, throughput=4.99MB/sec 2021-07-31T02:14:34.794 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Gx9-4lTxFJUuHwX off=8213898 len=1950657 2021-07-31T02:14:34.797 INFO:tasks.workunit.client.0.smithi045.stdout:op 2855 completed, throughput=5MB/sec 2021-07-31T02:14:34.798 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-qV4Xb9Z6hyEyYYF off=19912656 len=2048002 2021-07-31T02:14:34.807 INFO:tasks.workunit.client.0.smithi045.stdout:op 2856 completed, throughput=5MB/sec 2021-07-31T02:14:35.807 INFO:tasks.workunit.client.0.smithi045.stdout: 558: throughput=4.99MB/sec pending data=0 2021-07-31T02:14:35.808 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj--2Uid3rN8YSuOOJ off=5995725 len=183751 2021-07-31T02:14:35.811 INFO:tasks.workunit.client.0.smithi045.stdout:op 2857 completed, throughput=4.99MB/sec 2021-07-31T02:14:35.811 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-iHgizVGTCYK7FQ0 off=4699290 len=743728 2021-07-31T02:14:35.812 INFO:tasks.workunit.client.0.smithi045.stdout:op 2858 completed, throughput=4.99MB/sec 2021-07-31T02:14:35.812 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-2zSrHpe8qyRGsIr off=7913895 len=1425430 2021-07-31T02:14:35.819 INFO:tasks.workunit.client.0.smithi045.stdout:op 2859 completed, throughput=5MB/sec 2021-07-31T02:14:35.819 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-R2de_FbTYVD3peU off=2499709 len=2047406 2021-07-31T02:14:35.821 INFO:tasks.workunit.client.0.smithi045.stdout:op 2860 completed, throughput=5MB/sec 2021-07-31T02:14:35.821 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-zNQi0uIR-9L1d8- off=5799771 len=98762 2021-07-31T02:14:35.822 INFO:tasks.workunit.client.0.smithi045.stdout:op 2861 completed, throughput=5MB/sec 2021-07-31T02:14:36.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:36 smithi154 conmon[34553]: cluster 2021-07-31T02:14:36.183386+0000 mgr.x (mgr.24542) 1176 : cluster [DBG] pgmap v1693: 9 pgs: 9 active+clean; 127 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:14:36.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:36 smithi045 conmon[33655]: cluster 2021-07-31T02:14 2021-07-31T02:14:36.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:36 smithi045 conmon[33655]: :36.183386+0000 mgr.x (mgr.24542) 1176 : cluster [DBG] pgmap v1693: 9 pgs: 9 active+clean; 127 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:14:36.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:36 smithi045 conmon[39718]: cluster 2021-07-31T02:14:36.183386+0000 mgr.x (mgr.24542) 1176 : cluster [DBG] pgmap v1693: 9 pgs: 9 active+clean; 127 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:14:36.822 INFO:tasks.workunit.client.0.smithi045.stdout: 559: throughput=4.99MB/sec pending data=0 2021-07-31T02:14:36.823 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-RF1cBxkgSD4wz_R off=13451277 len=695123 2021-07-31T02:14:36.828 INFO:tasks.workunit.client.0.smithi045.stdout:op 2862 completed, throughput=4.99MB/sec 2021-07-31T02:14:36.828 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-9BmS7WIJGZL2B7k off=1323870 len=1942357 2021-07-31T02:14:36.839 INFO:tasks.workunit.client.0.smithi045.stdout:op 2863 completed, throughput=5MB/sec 2021-07-31T02:14:36.840 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-NBCHev5j9t1tFjE off=7504773 len=2051575 2021-07-31T02:14:36.841 INFO:tasks.workunit.client.0.smithi045.stdout:op 2864 completed, throughput=5MB/sec 2021-07-31T02:14:36.842 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-TdUuMGJD9wMFjYK off=9856274 len=1143602 2021-07-31T02:14:36.847 INFO:tasks.workunit.client.0.smithi045.stdout:op 2865 completed, throughput=5MB/sec 2021-07-31T02:14:37.847 INFO:tasks.workunit.client.0.smithi045.stdout: 560: throughput=4.99MB/sec pending data=0 2021-07-31T02:14:37.847 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-CONKLo5A35HBKyQ off=5442063 len=1877116 2021-07-31T02:14:37.857 INFO:tasks.workunit.client.0.smithi045.stdout:op 2866 completed, throughput=5MB/sec 2021-07-31T02:14:37.857 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-sR_kJdJy9lE23_O off=2066583 len=1660107 2021-07-31T02:14:37.859 INFO:tasks.workunit.client.0.smithi045.stdout:op 2867 completed, throughput=5MB/sec 2021-07-31T02:14:37.859 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-QTB--hGjEDIRWVK off=1783249 len=618331 2021-07-31T02:14:37.864 INFO:tasks.workunit.client.0.smithi045.stdout:op 2868 completed, throughput=5MB/sec 2021-07-31T02:14:37.864 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-uFzawn7I3e1BmfR off=5446187 len=783669 2021-07-31T02:14:37.865 INFO:tasks.workunit.client.0.smithi045.stdout:op 2869 completed, throughput=5MB/sec 2021-07-31T02:14:38.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:38 smithi154 conmon[34553]: cluster 2021-07-31T02:14:38.183828+0000 mgr.x (mgr.24542) 1177 : cluster [DBG] pgmap v1694: 9 pgs: 9 active+clean; 127 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:14:38.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:38 smithi045 conmon[33655]: cluster 2021-07-31T02:14:38.183828+0000 mgr.x (mgr.24542) 1177 : cluster [DBG] pgmap v1694: 9 pgs: 9 active+clean; 127 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:14:38.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:38 smithi045 conmon[39718]: cluster 2021-07-31T02:14:38 2021-07-31T02:14:38.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:38 smithi045 conmon[39718]: .183828+0000 mgr.x (mgr.24542) 1177 : cluster [DBG] pgmap v1694: 9 pgs: 9 active+clean; 127 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:14:38.865 INFO:tasks.workunit.client.0.smithi045.stdout: 561: throughput=4.99MB/sec pending data=0 2021-07-31T02:14:38.865 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Zf5SoqWhJVEjvPp off=2407317 len=357497 2021-07-31T02:14:38.869 INFO:tasks.workunit.client.0.smithi045.stdout:op 2870 completed, throughput=4.99MB/sec 2021-07-31T02:14:38.869 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-di_45glY8bfmAQG off=3038345 len=817735 2021-07-31T02:14:38.874 INFO:tasks.workunit.client.0.smithi045.stdout:op 2871 completed, throughput=4.99MB/sec 2021-07-31T02:14:38.874 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-dtxK7fLA-WQ331s off=603233 len=133297 2021-07-31T02:14:38.875 INFO:tasks.workunit.client.0.smithi045.stdout:op 2872 completed, throughput=4.99MB/sec 2021-07-31T02:14:38.875 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-453MzuGpzX8wP-1 off=8992470 len=1896504 2021-07-31T02:14:38.886 INFO:tasks.workunit.client.0.smithi045.stdout:op 2873 completed, throughput=5MB/sec 2021-07-31T02:14:38.887 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-HlIt81PUlCmrj9X off=8247430 len=367703 2021-07-31T02:14:38.887 INFO:tasks.workunit.client.0.smithi045.stdout:op 2874 completed, throughput=5MB/sec 2021-07-31T02:14:38.887 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-9QPki4sw9B3rTDI off=563119 len=1938589 2021-07-31T02:14:38.894 INFO:tasks.workunit.client.0.smithi045.stdout:op 2875 completed, throughput=5MB/sec 2021-07-31T02:14:39.895 INFO:tasks.workunit.client.0.smithi045.stdout: 562: throughput=4.99MB/sec pending data=0 2021-07-31T02:14:39.895 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-zbSry9ryMeAptqb off=15503971 len=747164 2021-07-31T02:14:39.899 INFO:tasks.workunit.client.0.smithi045.stdout:op 2876 completed, throughput=4.99MB/sec 2021-07-31T02:14:39.899 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-L9BPgJEjpV76bTg off=8233905 len=1143553 2021-07-31T02:14:39.901 INFO:tasks.workunit.client.0.smithi045.stdout:op 2877 completed, throughput=4.99MB/sec 2021-07-31T02:14:39.901 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-WxPKQTwu_0zWbjb off=571871 len=1694880 2021-07-31T02:14:39.905 INFO:tasks.workunit.client.0.smithi045.stdout:op 2878 completed, throughput=5MB/sec 2021-07-31T02:14:39.906 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-9vUc7wuPnuySMxI off=13477969 len=404417 2021-07-31T02:14:39.908 INFO:tasks.workunit.client.0.smithi045.stdout:op 2879 completed, throughput=5MB/sec 2021-07-31T02:14:39.908 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-iHgizVGTCYK7FQ0 off=1070792 len=1050541 2021-07-31T02:14:39.914 INFO:tasks.workunit.client.0.smithi045.stdout:op 2880 completed, throughput=5MB/sec 2021-07-31T02:14:39.914 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-7YHhj9PhuSDbYCx off=8023383 len=900603 2021-07-31T02:14:39.916 INFO:tasks.workunit.client.0.smithi045.stdout:op 2881 completed, throughput=5MB/sec 2021-07-31T02:14:40.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:40 smithi154 conmon[34553]: cluster 2021-07-31T02:14:40.184305+0000 mgr.x (mgr.24542) 1178 : cluster [DBG] pgmap v1695: 9 pgs: 9 active+clean; 127 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 2.7 MiB/s rd, 3.7 MiB/s wr, 6 op/s 2021-07-31T02:14:40.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:40 smithi045 conmon[39718]: cluster 2021-07-31T02:14:40.184305+0000 mgr.x ( 2021-07-31T02:14:40.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:40 smithi045 conmon[39718]: mgr.24542) 1178 : cluster [DBG] pgmap v1695: 9 pgs: 9 active+clean; 127 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 2.7 MiB/s rd, 3.7 MiB/s wr, 6 op/s 2021-07-31T02:14:40.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:40 smithi045 conmon[33655]: cluster 2021-07-31T02:14:40.184305+0000 mgr.x (mgr.24542) 1178 : 2021-07-31T02:14:40.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:40 smithi045 conmon[33655]: cluster [DBG] pgmap v1695: 9 pgs: 9 active+clean; 127 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 2.7 MiB/s rd, 3.7 MiB/s wr, 6 op/s 2021-07-31T02:14:40.916 INFO:tasks.workunit.client.0.smithi045.stdout: 563: throughput=4.99MB/sec pending data=0 2021-07-31T02:14:40.917 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-t0I8xwbYlaNCa64 off=15744365 len=680170 2021-07-31T02:14:40.921 INFO:tasks.workunit.client.0.smithi045.stdout:op 2882 completed, throughput=4.99MB/sec 2021-07-31T02:14:40.921 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-vXBjYeWZoeDcJov off=1387872 len=728004 2021-07-31T02:14:40.923 INFO:tasks.workunit.client.0.smithi045.stdout:op 2883 completed, throughput=4.99MB/sec 2021-07-31T02:14:40.924 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-jPgwOhBy6hiijPB off=4368395 len=747940 2021-07-31T02:14:40.928 INFO:tasks.workunit.client.0.smithi045.stdout:op 2884 completed, throughput=5MB/sec 2021-07-31T02:14:40.928 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Zyo6H0Su935Cp4v off=1475673 len=1166213 2021-07-31T02:14:40.930 INFO:tasks.workunit.client.0.smithi045.stdout:op 2885 completed, throughput=5MB/sec 2021-07-31T02:14:40.930 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ue84fEQqPNjLsYp off=948775 len=196819 2021-07-31T02:14:40.933 INFO:tasks.workunit.client.0.smithi045.stdout:op 2886 completed, throughput=5MB/sec 2021-07-31T02:14:40.933 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-dZ9ORMWsZy0Gw3l off=13729276 len=703525 2021-07-31T02:14:40.935 INFO:tasks.workunit.client.0.smithi045.stdout:op 2887 completed, throughput=5MB/sec 2021-07-31T02:14:40.935 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-0SP0jXCz-FDAEWY off=13014255 len=1095825 2021-07-31T02:14:40.939 INFO:tasks.workunit.client.0.smithi045.stdout:op 2888 completed, throughput=5MB/sec 2021-07-31T02:14:41.940 INFO:tasks.workunit.client.0.smithi045.stdout: 564: throughput=4.99MB/sec pending data=0 2021-07-31T02:14:41.940 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-jBqJx3u-u5NjWXp off=20120985 len=1209250 2021-07-31T02:14:41.947 INFO:tasks.workunit.client.0.smithi045.stdout:op 2889 completed, throughput=4.99MB/sec 2021-07-31T02:14:41.947 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-1ZftCovgTih22YL off=2829543 len=614889 2021-07-31T02:14:41.948 INFO:tasks.workunit.client.0.smithi045.stdout:op 2890 completed, throughput=5MB/sec 2021-07-31T02:14:41.948 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-1HjjS9eWtH6wxix off=10300869 len=1525779 2021-07-31T02:14:41.952 INFO:tasks.workunit.client.0.smithi045.stdout:op 2891 completed, throughput=5MB/sec 2021-07-31T02:14:41.953 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-5uPLLYi9HVTJYYc off=1480485 len=2028561 2021-07-31T02:14:41.963 INFO:tasks.workunit.client.0.smithi045.stdout:op 2892 completed, throughput=5MB/sec 2021-07-31T02:14:42.963 INFO:tasks.workunit.client.0.smithi045.stdout: 565: throughput=4.99MB/sec pending data=0 2021-07-31T02:14:42.964 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-mIIL3tHmqW6W3s9 off=0 len=1828370 2021-07-31T02:14:42.966 INFO:tasks.workunit.client.0.smithi045.stdout:op 2893 completed, throughput=5MB/sec 2021-07-31T02:14:42.966 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Ajbg_PmU4AYxkdF off=12767738 len=1567458 2021-07-31T02:14:42.974 INFO:tasks.workunit.client.0.smithi045.stdout:op 2894 completed, throughput=5MB/sec 2021-07-31T02:14:42.975 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-VseS4W58yoSyWT1 off=5681095 len=723541 2021-07-31T02:14:42.979 INFO:tasks.workunit.client.0.smithi045.stdout:op 2895 completed, throughput=5MB/sec 2021-07-31T02:14:42.979 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-hhhs6R0geONWsAk off=13855796 len=532164 2021-07-31T02:14:42.980 INFO:tasks.workunit.client.0.smithi045.stdout:op 2896 completed, throughput=5MB/sec 2021-07-31T02:14:43.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:43 smithi154 conmon[34553]: cluster 2021-07-31T02:14:42.184682+0000 mgr.x (mgr.24542) 1179 : cluster [DBG] pgmap v1696: 9 pgs: 9 active+clean; 127 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.5 MiB/s wr, 3 op/s 2021-07-31T02:14:43.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:43 smithi045 conmon[39718]: cluster 2021-07-31T02:14:42.184682+0000 2021-07-31T02:14:43.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:43 smithi045 conmon[39718]: mgr.x (mgr.24542) 1179 : cluster [DBG] pgmap v1696: 9 pgs: 9 active+clean; 127 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.5 MiB/s wr, 3 op/s 2021-07-31T02:14:43.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:43 smithi045 conmon[33655]: cluster 2021-07-31T02:14:42.184682+0000 mgr.x (mgr.24542) 2021-07-31T02:14:43.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:43 smithi045 conmon[33655]: 1179 : cluster [DBG] pgmap v1696: 9 pgs: 9 active+clean; 127 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.5 MiB/s wr, 3 op/s 2021-07-31T02:14:43.981 INFO:tasks.workunit.client.0.smithi045.stdout: 566: throughput=4.99MB/sec pending data=0 2021-07-31T02:14:43.981 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-j4X3m9Cs5EY-1Cy off=5885965 len=1677734 2021-07-31T02:14:43.991 INFO:tasks.workunit.client.0.smithi045.stdout:op 2897 completed, throughput=4.99MB/sec 2021-07-31T02:14:43.991 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-IvDL5FUtu_4wqEs off=1713294 len=573571 2021-07-31T02:14:43.992 INFO:tasks.workunit.client.0.smithi045.stdout:op 2898 completed, throughput=5MB/sec 2021-07-31T02:14:43.992 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-8DgWsR6SIboutvL off=2733946 len=947756 2021-07-31T02:14:43.994 INFO:tasks.workunit.client.0.smithi045.stdout:op 2899 completed, throughput=5MB/sec 2021-07-31T02:14:43.994 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-WJ-JD6XKc5Cm67u off=997926 len=1042432 2021-07-31T02:14:43.996 INFO:tasks.workunit.client.0.smithi045.stdout:op 2900 completed, throughput=5MB/sec 2021-07-31T02:14:43.996 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-gIAfasB0AYcx8iv off=3528864 len=1668723 2021-07-31T02:14:43.999 INFO:tasks.workunit.client.0.smithi045.stdout:op 2901 completed, throughput=5MB/sec 2021-07-31T02:14:44.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:14:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:14:43.555Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:14:44.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:14:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:14:43.555Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:14:44.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:14:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:14:43.555Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:14:44.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:44 smithi154 conmon[34553]: cluster 2021-07-31T02:14:44.185314+0000 mgr.x (mgr.24542) 1180 : cluster [DBG] pgmap v1697: 9 pgs: 9 active+clean; 127 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 2.6 MiB/s rd, 3.3 MiB/s wr, 5 op/s 2021-07-31T02:14:44.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:44 smithi045 conmon[39718]: cluster 2021-07-31T02:14:44.185314+0000 mgr.x (mgr.24542 2021-07-31T02:14:44.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:44 smithi045 conmon[39718]: ) 1180 : cluster [DBG] pgmap v1697: 9 pgs: 9 active+clean; 127 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 2.6 MiB/s rd, 3.3 MiB/s wr, 5 op/s 2021-07-31T02:14:44.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:44 smithi045 conmon[33655]: cluster 2021-07-31T02:14:44.185314+0000 mgr.x (mgr.24542) 1180 : cluster [DBG] pgmap v1697: 9 pgs: 9 active+clean; 127 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 2.6 MiB/s rd, 3.3 MiB/s wr, 5 op/s 2021-07-31T02:14:44.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:44 smithi045 conmon[33655]: 2021-07-31T02:14:44.999 INFO:tasks.workunit.client.0.smithi045.stdout: 567: throughput=4.99MB/sec pending data=0 2021-07-31T02:14:45.000 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-YmOrI8lGJNiIly_ off=7387587 len=1332732 2021-07-31T02:14:45.002 INFO:tasks.workunit.client.0.smithi045.stdout:op 2902 completed, throughput=4.99MB/sec 2021-07-31T02:14:45.003 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-mdxXWtwe2voZwMT off=5995350 len=1439036 2021-07-31T02:14:45.005 INFO:tasks.workunit.client.0.smithi045.stdout:op 2903 completed, throughput=5MB/sec 2021-07-31T02:14:45.006 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-3wbhLlDiEdST1Nj off=17191756 len=1161568 2021-07-31T02:14:45.012 INFO:tasks.workunit.client.0.smithi045.stdout:op 2904 completed, throughput=5MB/sec 2021-07-31T02:14:45.012 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-e08hQQg50CECh1r off=4405642 len=481264 2021-07-31T02:14:45.013 INFO:tasks.workunit.client.0.smithi045.stdout:op 2905 completed, throughput=5MB/sec 2021-07-31T02:14:45.013 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-lThc90cNDZkgpBs off=11688693 len=1219589 2021-07-31T02:14:45.014 INFO:tasks.workunit.client.0.smithi045.stdout:op 2906 completed, throughput=5MB/sec 2021-07-31T02:14:45.665 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:14:45.942 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:14:45.942 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 29m ago 30m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:14:45.943 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 27m ago 30m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:14:45.943 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 27m ago 33m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:14:45.943 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 29m ago 35m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:14:45.943 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 29m ago 35m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:14:45.943 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 27m ago 33m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:14:45.944 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 29m ago 34m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:14:45.944 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 29m ago 30m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:14:45.944 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 27m ago 30m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:14:45.945 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 29m ago 33m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:14:45.945 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 29m ago 33m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:14:45.945 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 29m ago 32m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:14:45.945 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 29m ago 32m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:14:45.946 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 27m ago 32m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:14:45.946 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 27m ago 31m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:14:45.946 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 27m ago 31m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:14:45.946 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 27m ago 31m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:14:45.946 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 27m ago 30m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:14:46.015 INFO:tasks.workunit.client.0.smithi045.stdout: 568: throughput=4.99MB/sec pending data=0 2021-07-31T02:14:46.015 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-W6ou6xzEbaFsKtA off=3568917 len=1609400 2021-07-31T02:14:46.021 INFO:tasks.workunit.client.0.smithi045.stdout:op 2907 completed, throughput=5MB/sec 2021-07-31T02:14:46.021 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-xyAzBEyIpT_Vw6g off=12407390 len=183887 2021-07-31T02:14:46.022 INFO:tasks.workunit.client.0.smithi045.stdout:op 2908 completed, throughput=5MB/sec 2021-07-31T02:14:46.022 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-rhCPO7PS9BCN_DA off=1034502 len=1875078 2021-07-31T02:14:46.032 INFO:tasks.workunit.client.0.smithi045.stdout:op 2909 completed, throughput=5MB/sec 2021-07-31T02:14:46.032 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-dYWZolVwaK6Y5He off=14372067 len=622031 2021-07-31T02:14:46.035 INFO:tasks.workunit.client.0.smithi045.stdout:op 2910 completed, throughput=5MB/sec 2021-07-31T02:14:46.035 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Ds3tA1g-fUSxv2T off=16767597 len=1910116 2021-07-31T02:14:46.036 INFO:tasks.workunit.client.0.smithi045.stdout:op 2911 completed, throughput=5MB/sec 2021-07-31T02:14:46.282 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:14:46.282 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:14:46.283 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:14:46.283 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:14:46.283 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:14:46.283 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:14:46.283 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:14:46.284 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:14:46.284 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:14:46.284 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:14:46.284 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:14:46.285 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:14:46.285 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:14:46.285 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:14:46.285 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:14:46.286 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:14:46.286 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:14:46.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:46 smithi154 conmon[34553]: audit 2021-07-31T02:14:45.653829+0000 mgr.x (mgr.24542) 1181 : audit [DBG] from='client.16059 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:14:46.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:46 smithi154 conmon[34553]: audit 2021-07-31T02:14:45.934656+0000 mgr.x (mgr.24542) 1182 : audit [DBG] from='client.16065 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:14:46.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:46 smithi154 conmon[34553]: cluster 2021-07-31T02:14:46.185696+0000 mgr.x (mgr.24542) 1183 : cluster [DBG] pgmap v1698: 9 pgs: 9 active+clean; 127 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.1 MiB/s wr, 3 op/s 2021-07-31T02:14:46.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:46 smithi045 conmon[33655]: audit 2021-07-31T02:14:45.653829+0000 mgr.x (mgr. 2021-07-31T02:14:46.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:46 smithi045 conmon[33655]: 24542) 1181 : audit [DBG] from='client.16059 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:14:46.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:46 smithi045 conmon[33655]: audit 2021-07-31T02:14:45.934656+0000 mgr.x (mgr.24542) 1182 : audit [DBG] from='client.16065 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:14:46.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:46 smithi045 conmon[33655]: cluster 2021-07-31T02:14:46.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:46 smithi045 conmon[33655]: 2021-07-31T02:14:46.185696+0000 mgr.x (mgr.24542) 1183 : cluster [DBG] pgmap v1698: 9 pgs: 9 active+clean; 127 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.1 MiB/s wr, 3 op/s 2021-07-31T02:14:46.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:46 smithi045 conmon[39718]: audit 2021-07-31T02:14:45.653829 2021-07-31T02:14:46.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:46 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 1181 : audit [DBG] from='client.16059 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:14:46.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:46 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T02:14:46.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:46 smithi045 conmon[39718]: 02:14:45.934656+0000 mgr.x (mgr. 2021-07-31T02:14:46.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:46 smithi045 conmon[39718]: 24542) 1182 : audit [DBG] from='client.16065 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:14:46.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:46 smithi045 conmon[39718]: cluster 2021-07-31T02:14:46.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:46 smithi045 conmon[39718]: 2021-07-31T02:14:46.185696 2021-07-31T02:14:46.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:46 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 1183 : cluster 2021-07-31T02:14:46.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:46 smithi045 conmon[39718]: [DBG] pgmap v1698: 9 pgs: 9 active+clean; 127 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.1 MiB/s wr, 3 op/s 2021-07-31T02:14:47.037 INFO:tasks.workunit.client.0.smithi045.stdout: 569: throughput=4.99MB/sec pending data=0 2021-07-31T02:14:47.037 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-noHmvD6qT9p4_l1 off=8452930 len=351206 2021-07-31T02:14:47.038 INFO:tasks.workunit.client.0.smithi045.stdout:op 2912 completed, throughput=4.99MB/sec 2021-07-31T02:14:47.038 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-w7HynEimkwYVgl5 off=4888997 len=375915 2021-07-31T02:14:47.042 INFO:tasks.workunit.client.0.smithi045.stdout:op 2913 completed, throughput=5MB/sec 2021-07-31T02:14:47.042 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-7RN8GK_U9_R6VAn off=3967299 len=451873 2021-07-31T02:14:47.043 INFO:tasks.workunit.client.0.smithi045.stdout:op 2914 completed, throughput=5MB/sec 2021-07-31T02:14:47.044 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-MjTA3p_klDjmfYn off=654778 len=1903204 2021-07-31T02:14:47.053 INFO:tasks.workunit.client.0.smithi045.stdout:op 2915 completed, throughput=5MB/sec 2021-07-31T02:14:47.053 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-3d5AeRQxIHX_I4n off=9827097 len=1735251 2021-07-31T02:14:47.055 INFO:tasks.workunit.client.0.smithi045.stdout:op 2916 completed, throughput=5MB/sec 2021-07-31T02:14:47.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:47 smithi154 conmon[34553]: audit 2021-07-31T02:14:46.282390+0000 mon.c (mon.1) 83 : audit [DBG] from='client.? 172.21.15.45:0/1796759632' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:14:47.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:47 smithi045 conmon[33655]: audit 2021-07-31T02:14:46.282390+0000 mon.c (mon.1) 83 : 2021-07-31T02:14:47.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:47 smithi045 conmon[33655]: audit [DBG] from='client.? 172.21.15.45:0/1796759632' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:14:47.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:47 smithi045 conmon[39718]: audit 2021-07-31T02:14:46.282390+0000 mon.c (mon.1) 83 : audit 2021-07-31T02:14:47.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:47 smithi045 conmon[39718]: [DBG] from='client.? 172.21.15.45:0/1796759632' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:14:48.056 INFO:tasks.workunit.client.0.smithi045.stdout: 570: throughput=4.99MB/sec pending data=0 2021-07-31T02:14:48.057 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-tGmnbQKDNyAo3x0 off=4363169 len=831704 2021-07-31T02:14:48.058 INFO:tasks.workunit.client.0.smithi045.stdout:op 2917 completed, throughput=4.99MB/sec 2021-07-31T02:14:48.058 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-DeOP3WTjDmcELKn off=2584959 len=1705891 2021-07-31T02:14:48.066 INFO:tasks.workunit.client.0.smithi045.stdout:op 2918 completed, throughput=5MB/sec 2021-07-31T02:14:48.067 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-zskR3AzTvDtwrVf off=10596592 len=79102 2021-07-31T02:14:48.069 INFO:tasks.workunit.client.0.smithi045.stdout:op 2919 completed, throughput=5MB/sec 2021-07-31T02:14:48.069 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-xG9CP1l5c8Lmk44 off=5933263 len=1456481 2021-07-31T02:14:48.076 INFO:tasks.workunit.client.0.smithi045.stdout:op 2920 completed, throughput=5MB/sec 2021-07-31T02:14:48.076 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-DOEKNgbrSGo5mFB off=4193830 len=602176 2021-07-31T02:14:48.076 INFO:tasks.workunit.client.0.smithi045.stdout:op 2921 completed, throughput=5MB/sec 2021-07-31T02:14:48.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:48 smithi045 conmon[33655]: cluster 2021-07-31T02:14:48.186106+0000 mgr.x (mgr.24542) 1184 : cluster [DBG] 2021-07-31T02:14:48.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:48 smithi045 conmon[33655]: pgmap v1699: 9 pgs: 9 active+clean; 127 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.1 MiB/s wr, 3 op/s 2021-07-31T02:14:48.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:48 smithi045 conmon[39718]: cluster 2021-07-31T02:14:48.186106+0000 mgr.x (mgr.24542) 1184 : 2021-07-31T02:14:48.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:48 smithi045 conmon[39718]: cluster [DBG] pgmap v1699: 9 pgs: 9 active+clean; 127 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.1 MiB/s wr, 3 op/s 2021-07-31T02:14:48.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:48 smithi154 conmon[34553]: cluster 2021-07-31T02:14:48.186106+0000 mgr.x (mgr.24542) 1184 : cluster [DBG] pgmap v1699: 9 pgs: 9 active+clean; 127 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.1 MiB/s wr, 3 op/s 2021-07-31T02:14:48.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:48 smithi154 conmon[34553]: 2021-07-31T02:14:49.076 INFO:tasks.workunit.client.0.smithi045.stdout: 571: throughput=4.99MB/sec pending data=0 2021-07-31T02:14:49.076 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-r5NXg2PGtDxHWZN off=354823 len=1340285 2021-07-31T02:14:49.079 INFO:tasks.workunit.client.0.smithi045.stdout:op 2922 completed, throughput=4.99MB/sec 2021-07-31T02:14:49.079 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Q_iBfw6YswjUcTO off=3975459 len=1040511 2021-07-31T02:14:49.084 INFO:tasks.workunit.client.0.smithi045.stdout:op 2923 completed, throughput=5MB/sec 2021-07-31T02:14:49.085 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-EXs3fnfmoYZ9s0t off=17999369 len=289992 2021-07-31T02:14:49.088 INFO:tasks.workunit.client.0.smithi045.stdout:op 2924 completed, throughput=5MB/sec 2021-07-31T02:14:49.088 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-gHGaqNFaxdKKncF off=7656629 len=1242522 2021-07-31T02:14:49.096 INFO:tasks.workunit.client.0.smithi045.stdout:op 2925 completed, throughput=5MB/sec 2021-07-31T02:14:49.096 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-qtKYFUeHRkZoE5_ off=12232567 len=4586 2021-07-31T02:14:49.097 INFO:tasks.workunit.client.0.smithi045.stdout:op 2926 completed, throughput=5MB/sec 2021-07-31T02:14:49.098 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-wQQIVBeoDVDEpFq off=1966042 len=1110823 2021-07-31T02:14:49.099 INFO:tasks.workunit.client.0.smithi045.stdout:op 2927 completed, throughput=5MB/sec 2021-07-31T02:14:50.102 INFO:tasks.workunit.client.0.smithi045.stdout: 572: throughput=4.99MB/sec pending data=0 2021-07-31T02:14:50.102 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-M90zkSVbhRArPZQ off=3445373 len=1326574 2021-07-31T02:14:50.102 INFO:tasks.workunit.client.0.smithi045.stdout:op 2928 completed, throughput=4.99MB/sec 2021-07-31T02:14:50.103 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Ev0twc2-GlWMIbs off=1925605 len=1270734 2021-07-31T02:14:50.103 INFO:tasks.workunit.client.0.smithi045.stdout:op 2929 completed, throughput=5MB/sec 2021-07-31T02:14:50.104 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-z0bIvCxAm9csUpS off=19556973 len=104191 2021-07-31T02:14:50.105 INFO:tasks.workunit.client.0.smithi045.stdout:op 2930 completed, throughput=5MB/sec 2021-07-31T02:14:50.105 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Yl9TupV_MPdOetL off=2780735 len=836096 2021-07-31T02:14:50.105 INFO:tasks.workunit.client.0.smithi045.stdout:op 2931 completed, throughput=5MB/sec 2021-07-31T02:14:50.106 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-BpgZf7MK57Cr8TO off=12258729 len=1517788 2021-07-31T02:14:50.108 INFO:tasks.workunit.client.0.smithi045.stdout:op 2932 completed, throughput=5MB/sec 2021-07-31T02:14:50.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:50 smithi154 conmon[34553]: cluster 2021-07-31T02:14:50.186658+0000 mgr.x (mgr.24542) 1185 : cluster [DBG] pgmap v1700: 9 pgs: 9 active+clean; 127 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 2.8 MiB/s rd, 3.1 MiB/s wr, 5 op/s 2021-07-31T02:14:50.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:50 smithi045 conmon[33655]: cluster 2021-07-31T02:14:50.186658 2021-07-31T02:14:50.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:50 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 1185 : cluster [DBG] pgmap v1700: 9 pgs: 9 active+clean; 127 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 2.8 MiB/s rd, 3.1 MiB/s wr, 5 op/s 2021-07-31T02:14:50.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:50 smithi045 conmon[39718]: cluster 2021-07-31T02:14:50.186658+0000 mgr.x (mgr.24542) 1185 : cluster [DBG] 2021-07-31T02:14:50.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:50 smithi045 conmon[39718]: pgmap v1700: 9 pgs: 9 active+clean; 127 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 2.8 MiB/s rd, 3.1 MiB/s wr, 5 op/s 2021-07-31T02:14:51.108 INFO:tasks.workunit.client.0.smithi045.stdout: 573: throughput=4.99MB/sec pending data=0 2021-07-31T02:14:51.109 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-F-tpLtzdg8VxEab off=14412499 len=1035683 2021-07-31T02:14:51.115 INFO:tasks.workunit.client.0.smithi045.stdout:op 2933 completed, throughput=4.99MB/sec 2021-07-31T02:14:51.116 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-XBGz3XiItddm3ly off=7073029 len=1246131 2021-07-31T02:14:51.122 INFO:tasks.workunit.client.0.smithi045.stdout:op 2934 completed, throughput=5MB/sec 2021-07-31T02:14:51.122 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-TZxrIY00U3E4Aca off=8692647 len=1897885 2021-07-31T02:14:51.124 INFO:tasks.workunit.client.0.smithi045.stdout:op 2935 completed, throughput=5MB/sec 2021-07-31T02:14:51.124 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Cx6k3HvEvG9OSt3 off=2657922 len=228030 2021-07-31T02:14:51.127 INFO:tasks.workunit.client.0.smithi045.stdout:op 2936 completed, throughput=5MB/sec 2021-07-31T02:14:51.127 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-JXev7rbEkqXZxeZ off=487556 len=997633 2021-07-31T02:14:51.129 INFO:tasks.workunit.client.0.smithi045.stdout:op 2937 completed, throughput=5MB/sec 2021-07-31T02:14:52.129 INFO:tasks.workunit.client.0.smithi045.stdout: 574: throughput=4.99MB/sec pending data=0 2021-07-31T02:14:52.130 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-rPPw28F5iOCBxQ4 off=400299 len=1984135 2021-07-31T02:14:52.133 INFO:tasks.workunit.client.0.smithi045.stdout:op 2938 completed, throughput=4.99MB/sec 2021-07-31T02:14:52.133 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-7RN8GK_U9_R6VAn off=47249 len=352275 2021-07-31T02:14:52.134 INFO:tasks.workunit.client.0.smithi045.stdout:op 2939 completed, throughput=5MB/sec 2021-07-31T02:14:52.135 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-RL54q2H1GoVGwg3 off=22096227 len=721730 2021-07-31T02:14:52.136 INFO:tasks.workunit.client.0.smithi045.stdout:op 2940 completed, throughput=5MB/sec 2021-07-31T02:14:52.136 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-A0SsmxHGZr7bggg off=5773887 len=1250960 2021-07-31T02:14:52.137 INFO:tasks.workunit.client.0.smithi045.stdout:op 2941 completed, throughput=5MB/sec 2021-07-31T02:14:52.137 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-3k_dbKkUAM1qtI3 off=4948249 len=71378 2021-07-31T02:14:52.141 INFO:tasks.workunit.client.0.smithi045.stdout:op 2942 completed, throughput=5MB/sec 2021-07-31T02:14:52.141 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-b517QtjShB-cCyJ off=469102 len=103224 2021-07-31T02:14:52.142 INFO:tasks.workunit.client.0.smithi045.stdout:op 2943 completed, throughput=5MB/sec 2021-07-31T02:14:52.142 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-gn6OJHJp0cVDaWD off=2648261 len=352825 2021-07-31T02:14:52.142 INFO:tasks.workunit.client.0.smithi045.stdout:op 2944 completed, throughput=5MB/sec 2021-07-31T02:14:52.143 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj--ATwS6Ysw8DsHVT off=1161407 len=914178 2021-07-31T02:14:52.144 INFO:tasks.workunit.client.0.smithi045.stdout:op 2945 completed, throughput=5MB/sec 2021-07-31T02:14:53.144 INFO:tasks.workunit.client.0.smithi045.stdout: 575: throughput=4.99MB/sec pending data=0 2021-07-31T02:14:53.145 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-8oQ9-TDcKQDV0jS off=2894078 len=1025225 2021-07-31T02:14:53.150 INFO:tasks.workunit.client.0.smithi045.stdout:op 2946 completed, throughput=4.99MB/sec 2021-07-31T02:14:53.150 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Gl5o468D-kkoU66 off=4842865 len=1008083 2021-07-31T02:14:53.151 INFO:tasks.workunit.client.0.smithi045.stdout:op 2947 completed, throughput=5MB/sec 2021-07-31T02:14:53.151 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-fFLujL3eDYoZbS1 off=5096298 len=1621223 2021-07-31T02:14:53.159 INFO:tasks.workunit.client.0.smithi045.stdout:op 2948 completed, throughput=5MB/sec 2021-07-31T02:14:53.160 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-AsT7dxYKs8JaOyL off=513558 len=12271 2021-07-31T02:14:53.160 INFO:tasks.workunit.client.0.smithi045.stdout:op 2949 completed, throughput=5MB/sec 2021-07-31T02:14:53.161 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-7KyfgQK1CMhTDTk off=17941967 len=1385422 2021-07-31T02:14:53.167 INFO:tasks.workunit.client.0.smithi045.stdout:op 2950 completed, throughput=5MB/sec 2021-07-31T02:14:53.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:53 smithi154 conmon[34553]: cluster 2021-07-31T02:14:52.187166+0000 mgr.x (mgr.24542) 1186 : cluster [DBG] pgmap v1701: 9 pgs: 9 active+clean; 127 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 2.0 MiB/s rd, 1.8 MiB/s wr, 3 op/s 2021-07-31T02:14:53.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:53 smithi045 conmon[33655]: cluster 2021-07-31T02:14:52.187166+0000 mgr.x (mgr.24542) 1186 : cluster 2021-07-31T02:14:53.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:53 smithi045 conmon[33655]: [DBG] pgmap v1701: 9 pgs: 9 active+clean; 127 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 2.0 MiB/s rd, 1.8 MiB/s wr, 3 op/s 2021-07-31T02:14:53.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:53 smithi045 conmon[39718]: cluster 2021-07-31T02:14:52.187166+0000 mgr.x ( 2021-07-31T02:14:53.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:53 smithi045 conmon[39718]: mgr.24542) 1186 : cluster [DBG] pgmap v1701: 9 pgs: 9 active+clean; 127 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 2.0 MiB/s rd, 1.8 MiB/s wr, 3 op/s 2021-07-31T02:14:54.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:14:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:14:53.556Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:14:54.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:14:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:14:53.556Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:14:54.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:14:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:14:53.556Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:14:54.167 INFO:tasks.workunit.client.0.smithi045.stdout: 576: throughput=4.99MB/sec pending data=0 2021-07-31T02:14:54.167 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-RvyKH5gIYU0oaH2 off=910031 len=353772 2021-07-31T02:14:54.171 INFO:tasks.workunit.client.0.smithi045.stdout:op 2951 completed, throughput=4.99MB/sec 2021-07-31T02:14:54.171 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-O5jhxUVpwrVl_wu off=14544038 len=1818508 2021-07-31T02:14:54.174 INFO:tasks.workunit.client.0.smithi045.stdout:op 2952 completed, throughput=5MB/sec 2021-07-31T02:14:54.174 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-vyqFPqkY_i-Ra0o off=7226463 len=483727 2021-07-31T02:14:54.176 INFO:tasks.workunit.client.0.smithi045.stdout:op 2953 completed, throughput=5MB/sec 2021-07-31T02:14:54.176 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Cafw-w_DoAJlbyy off=10630147 len=860957 2021-07-31T02:14:54.180 INFO:tasks.workunit.client.0.smithi045.stdout:op 2954 completed, throughput=5MB/sec 2021-07-31T02:14:54.181 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-eI16AvOy5RciGMQ off=405982 len=687709 2021-07-31T02:14:54.182 INFO:tasks.workunit.client.0.smithi045.stdout:op 2955 completed, throughput=5MB/sec 2021-07-31T02:14:54.182 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-svr2bNyneTLYi_w off=1662085 len=97985 2021-07-31T02:14:54.185 INFO:tasks.workunit.client.0.smithi045.stdout:op 2956 completed, throughput=5MB/sec 2021-07-31T02:14:54.186 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-dwB5RrB-4K16Tps off=8810615 len=2022161 2021-07-31T02:14:54.188 INFO:tasks.workunit.client.0.smithi045.stdout:op 2957 completed, throughput=5MB/sec 2021-07-31T02:14:54.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:54 smithi154 conmon[34553]: cluster 2021-07-31T02:14:54.187592+0000 mgr.x (mgr.24542) 1187 : cluster [DBG] 2021-07-31T02:14:54.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:54 smithi154 conmon[34553]: pgmap v1702: 9 pgs: 9 active+clean; 127 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 3.4 MiB/s rd, 2.6 MiB/s wr, 6 op/s 2021-07-31T02:14:54.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:54 smithi045 conmon[33655]: cluster 2021-07-31T02:14:54.187592+0000 mgr.x (mgr.24542) 1187 : cluster [DBG] pgmap v1702: 9 pgs: 9 active+clean; 127 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 3.4 MiB/s rd, 2.6 MiB/s wr, 6 op/s 2021-07-31T02:14:54.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:54 smithi045 conmon[39718]: cluster 2021-07-31T02:14:54.187592+0000 2021-07-31T02:14:54.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:54 smithi045 conmon[39718]: mgr.x (mgr.24542) 1187 : cluster [DBG] pgmap v1702: 9 pgs: 9 active+clean; 127 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 3.4 MiB/s rd, 2.6 MiB/s wr, 6 op/s 2021-07-31T02:14:55.188 INFO:tasks.workunit.client.0.smithi045.stdout: 577: throughput=4.99MB/sec pending data=0 2021-07-31T02:14:55.189 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-WK9cIADhuDCeoeA off=10626199 len=962872 2021-07-31T02:14:55.194 INFO:tasks.workunit.client.0.smithi045.stdout:op 2958 completed, throughput=4.99MB/sec 2021-07-31T02:14:55.194 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-W6ou6xzEbaFsKtA off=707981 len=207050 2021-07-31T02:14:55.197 INFO:tasks.workunit.client.0.smithi045.stdout:op 2959 completed, throughput=5MB/sec 2021-07-31T02:14:55.197 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-T3a0SQHSj1oJx-k off=2911617 len=1933893 2021-07-31T02:14:55.207 INFO:tasks.workunit.client.0.smithi045.stdout:op 2960 completed, throughput=5MB/sec 2021-07-31T02:14:55.207 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-F9-d2kip3ZXN6UO off=6444254 len=272411 2021-07-31T02:14:55.210 INFO:tasks.workunit.client.0.smithi045.stdout:op 2961 completed, throughput=5MB/sec 2021-07-31T02:14:55.210 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-EYVWUW9jDbWMg60 off=51144 len=1649348 2021-07-31T02:14:55.219 INFO:tasks.workunit.client.0.smithi045.stdout:op 2962 completed, throughput=5MB/sec 2021-07-31T02:14:56.220 INFO:tasks.workunit.client.0.smithi045.stdout: 578: throughput=4.99MB/sec pending data=0 2021-07-31T02:14:56.220 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-HLPbFVKWolgPIAH off=991492 len=495534 2021-07-31T02:14:56.224 INFO:tasks.workunit.client.0.smithi045.stdout:op 2963 completed, throughput=4.99MB/sec 2021-07-31T02:14:56.224 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-qQtHCqD0QNpiavk off=18113829 len=1569350 2021-07-31T02:14:56.227 INFO:tasks.workunit.client.0.smithi045.stdout:op 2964 completed, throughput=5MB/sec 2021-07-31T02:14:56.228 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Ag6hMku528q-yTl off=2759467 len=1748937 2021-07-31T02:14:56.230 INFO:tasks.workunit.client.0.smithi045.stdout:op 2965 completed, throughput=5MB/sec 2021-07-31T02:14:56.230 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-2o86vmBv3XMo_gP off=756580 len=1707965 2021-07-31T02:14:56.239 INFO:tasks.workunit.client.0.smithi045.stdout:op 2966 completed, throughput=5MB/sec 2021-07-31T02:14:56.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:56 smithi154 conmon[34553]: cluster 2021-07-31T02:14:56.188127+0000 mgr.x (mgr.24542) 1188 : cluster 2021-07-31T02:14:56.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:56 smithi154 conmon[34553]: [DBG] pgmap v1703: 9 pgs: 9 active+clean; 127 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.7 MiB/s wr, 4 op/s 2021-07-31T02:14:56.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:56 smithi045 conmon[39718]: cluster 2021-07-31T02:14:56.188127+0000 mgr.x (mgr.24542) 1188 : cluster [DBG] pgmap v1703: 9 pgs: 9 active+clean; 127 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.7 MiB/s wr, 4 op/s 2021-07-31T02:14:56.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:56 smithi045 conmon[33655]: cluster 2021-07-31T02:14:56.188127+0000 mgr.x (mgr.24542) 1188 : cluster 2021-07-31T02:14:56.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:56 smithi045 conmon[33655]: [DBG] pgmap v1703: 9 pgs: 9 active+clean; 127 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.7 MiB/s wr, 4 op/s 2021-07-31T02:14:57.239 INFO:tasks.workunit.client.0.smithi045.stdout: 579: throughput=4.99MB/sec pending data=0 2021-07-31T02:14:57.240 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-LxfEHBTN_mR8V_0 off=81356 len=1386819 2021-07-31T02:14:57.241 INFO:tasks.workunit.client.0.smithi045.stdout:op 2967 completed, throughput=5MB/sec 2021-07-31T02:14:57.241 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-poe1bYrRhMlUlqX off=20306147 len=1165080 2021-07-31T02:14:57.244 INFO:tasks.workunit.client.0.smithi045.stdout:op 2968 completed, throughput=5MB/sec 2021-07-31T02:14:57.244 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-X-0Cf5vCI0zs6C6 off=2601756 len=308622 2021-07-31T02:14:57.245 INFO:tasks.workunit.client.0.smithi045.stdout:op 2969 completed, throughput=5MB/sec 2021-07-31T02:14:57.245 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-od0o9FDS94RNcht off=4848528 len=870080 2021-07-31T02:14:57.246 INFO:tasks.workunit.client.0.smithi045.stdout:op 2970 completed, throughput=5MB/sec 2021-07-31T02:14:57.246 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-6ip8AmOSXYAu0xi off=4384831 len=892230 2021-07-31T02:14:57.251 INFO:tasks.workunit.client.0.smithi045.stdout:op 2971 completed, throughput=5MB/sec 2021-07-31T02:14:58.251 INFO:tasks.workunit.client.0.smithi045.stdout: 580: throughput=4.99MB/sec pending data=0 2021-07-31T02:14:58.252 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-vK8L4caqxdcQr-H off=333871 len=19119 2021-07-31T02:14:58.254 INFO:tasks.workunit.client.0.smithi045.stdout:op 2972 completed, throughput=4.99MB/sec 2021-07-31T02:14:58.254 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-hCdEaZ4SAI7MDIL off=2014633 len=1584226 2021-07-31T02:14:58.256 INFO:tasks.workunit.client.0.smithi045.stdout:op 2973 completed, throughput=4.99MB/sec 2021-07-31T02:14:58.256 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-R_3nkRBvLuoUm_B off=2875768 len=365398 2021-07-31T02:14:58.259 INFO:tasks.workunit.client.0.smithi045.stdout:op 2974 completed, throughput=5MB/sec 2021-07-31T02:14:58.259 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-hDrCAaOePUX_gN0 off=9629430 len=1693369 2021-07-31T02:14:58.261 INFO:tasks.workunit.client.0.smithi045.stdout:op 2975 completed, throughput=5MB/sec 2021-07-31T02:14:58.261 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-3L8SNA3pgoU-FBs off=13899460 len=446690 2021-07-31T02:14:58.265 INFO:tasks.workunit.client.0.smithi045.stdout:op 2976 completed, throughput=5MB/sec 2021-07-31T02:14:58.266 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-lA3OODMbiahqJMd off=716267 len=936131 2021-07-31T02:14:58.271 INFO:tasks.workunit.client.0.smithi045.stdout:op 2977 completed, throughput=5MB/sec 2021-07-31T02:14:58.271 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-U1OIU0CgwoUMGM3 off=4998127 len=1539208 2021-07-31T02:14:58.272 INFO:tasks.workunit.client.0.smithi045.stdout:op 2978 completed, throughput=5MB/sec 2021-07-31T02:14:58.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:58 smithi154 conmon[34553]: cluster 2021-07-31T02:14:58.188676+0000 mgr.x (mgr.24542 2021-07-31T02:14:58.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:14:58 smithi154 conmon[34553]: ) 1189 : cluster [DBG] pgmap v1704: 9 pgs: 9 active+clean; 127 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.7 MiB/s wr, 4 op/s 2021-07-31T02:14:58.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:14:58 smithi045 conmon[33655]: cluster 2021-07-31T02:14:58.188676+0000 mgr.x (mgr.24542) 1189 : cluster [DBG] pgmap v1704: 9 pgs: 9 active+clean; 127 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.7 MiB/s wr, 4 op/s 2021-07-31T02:14:58.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:14:58 smithi045 conmon[39718]: cluster 2021-07-31T02:14:58.188676+0000 mgr.x (mgr.24542) 1189 : cluster [DBG] pgmap v1704: 9 pgs: 9 active+clean; 127 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.7 MiB/s wr, 4 op/s 2021-07-31T02:14:59.272 INFO:tasks.workunit.client.0.smithi045.stdout: 581: throughput=4.99MB/sec pending data=0 2021-07-31T02:14:59.272 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-V56V2sRa7fE26TC off=1723138 len=2079809 2021-07-31T02:14:59.276 INFO:tasks.workunit.client.0.smithi045.stdout:op 2979 completed, throughput=5MB/sec 2021-07-31T02:14:59.276 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-gn73TuICF5Uqwq0 off=8325058 len=1529611 2021-07-31T02:14:59.279 INFO:tasks.workunit.client.0.smithi045.stdout:op 2980 completed, throughput=5MB/sec 2021-07-31T02:14:59.279 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-7aTFiDihJ_EKbzh off=1136801 len=1940483 2021-07-31T02:14:59.282 INFO:tasks.workunit.client.0.smithi045.stdout:op 2981 completed, throughput=5MB/sec 2021-07-31T02:15:00.282 INFO:tasks.workunit.client.0.smithi045.stdout: 582: throughput=4.99MB/sec pending data=0 2021-07-31T02:15:00.283 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-hDqT4L0wECiQgHP off=2977456 len=2004614 2021-07-31T02:15:00.292 INFO:tasks.workunit.client.0.smithi045.stdout:op 2982 completed, throughput=5MB/sec 2021-07-31T02:15:00.293 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-dIdykDpxd2KY4xZ off=23782157 len=1689393 2021-07-31T02:15:00.296 INFO:tasks.workunit.client.0.smithi045.stdout:op 2983 completed, throughput=5MB/sec 2021-07-31T02:15:00.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:00 smithi045 conmon[33655]: cluster 2021-07-31T02:15:00.189174+0000 mgr.x (mgr.24542) 1190 : cluster 2021-07-31T02:15:00.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:00 smithi045 conmon[33655]: [DBG] pgmap v1705: 9 pgs: 9 active+clean; 127 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 3.9 MiB/s rd, 2.6 MiB/s wr, 6 op/s 2021-07-31T02:15:00.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:00 smithi045 conmon[39718]: cluster 2021-07-31T02:15:00 2021-07-31T02:15:00.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:00 smithi045 conmon[39718]: .189174+0000 mgr.x (mgr.24542) 1190 : cluster [DBG] pgmap v1705: 9 pgs: 9 active+clean; 127 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 3.9 MiB/s rd, 2.6 MiB/s wr, 6 op/s 2021-07-31T02:15:00.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:00 smithi154 conmon[34553]: cluster 2021-07-31T02:15:00.189174+0000 mgr.x (mgr.24542) 1190 : cluster [DBG] pgmap v1705: 9 pgs: 9 active+clean; 127 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 3.9 MiB/s rd, 2.6 MiB/s wr, 6 op/s 2021-07-31T02:15:01.296 INFO:tasks.workunit.client.0.smithi045.stdout: 583: throughput=4.99MB/sec pending data=0 2021-07-31T02:15:01.297 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Tymc5B839J9pLkf off=1712700 len=1960336 2021-07-31T02:15:01.304 INFO:tasks.workunit.client.0.smithi045.stdout:op 2984 completed, throughput=4.99MB/sec 2021-07-31T02:15:01.304 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-zQxiov2JIfNAmNy off=4974043 len=1059671 2021-07-31T02:15:01.306 INFO:tasks.workunit.client.0.smithi045.stdout:op 2985 completed, throughput=5MB/sec 2021-07-31T02:15:01.306 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-20oyDr0j61jGL03 off=20643229 len=1395286 2021-07-31T02:15:01.312 INFO:tasks.workunit.client.0.smithi045.stdout:op 2986 completed, throughput=5MB/sec 2021-07-31T02:15:01.313 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-SD31mwz-UKcpqj9 off=8096159 len=1481436 2021-07-31T02:15:01.318 INFO:tasks.workunit.client.0.smithi045.stdout:op 2987 completed, throughput=5MB/sec 2021-07-31T02:15:02.319 INFO:tasks.workunit.client.0.smithi045.stdout: 584: throughput=4.99MB/sec pending data=0 2021-07-31T02:15:02.319 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-uKeOmpowiPbKzlN off=4149094 len=863689 2021-07-31T02:15:02.320 INFO:tasks.workunit.client.0.smithi045.stdout:op 2988 completed, throughput=4.99MB/sec 2021-07-31T02:15:02.321 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ggJmMwvxDzFcjKU off=19530771 len=357259 2021-07-31T02:15:02.324 INFO:tasks.workunit.client.0.smithi045.stdout:op 2989 completed, throughput=4.99MB/sec 2021-07-31T02:15:02.325 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj--H6LsbaDe1EylGt off=8938903 len=1277053 2021-07-31T02:15:02.327 INFO:tasks.workunit.client.0.smithi045.stdout:op 2990 completed, throughput=5MB/sec 2021-07-31T02:15:02.327 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-KOFGaZXQPZ9WW3v off=1349005 len=504643 2021-07-31T02:15:02.331 INFO:tasks.workunit.client.0.smithi045.stdout:op 2991 completed, throughput=5MB/sec 2021-07-31T02:15:02.331 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ZLMlgdalxeVS6Ip off=3387845 len=1187665 2021-07-31T02:15:02.338 INFO:tasks.workunit.client.0.smithi045.stdout:op 2992 completed, throughput=5MB/sec 2021-07-31T02:15:02.338 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-uZ1_68LhfW68RNq off=9544614 len=467555 2021-07-31T02:15:02.342 INFO:tasks.workunit.client.0.smithi045.stdout:op 2993 completed, throughput=5MB/sec 2021-07-31T02:15:02.342 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ZtBnvncePtyennA off=848420 len=1183393 2021-07-31T02:15:02.343 INFO:tasks.workunit.client.0.smithi045.stdout:op 2994 completed, throughput=5MB/sec 2021-07-31T02:15:03.344 INFO:tasks.workunit.client.0.smithi045.stdout: 585: throughput=4.99MB/sec pending data=0 2021-07-31T02:15:03.344 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-qQbd6rwtEF8kHT0 off=10434479 len=1003322 2021-07-31T02:15:03.346 INFO:tasks.workunit.client.0.smithi045.stdout:op 2995 completed, throughput=5MB/sec 2021-07-31T02:15:03.347 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-YXnAGe93Gi5at3i off=2592076 len=494801 2021-07-31T02:15:03.347 INFO:tasks.workunit.client.0.smithi045.stdout:op 2996 completed, throughput=5MB/sec 2021-07-31T02:15:03.348 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-a9NwgNadGnBZRXI off=5892278 len=498733 2021-07-31T02:15:03.349 INFO:tasks.workunit.client.0.smithi045.stdout:op 2997 completed, throughput=5MB/sec 2021-07-31T02:15:03.349 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-2K_vgNCQXEdpv3F off=1382128 len=157918 2021-07-31T02:15:03.351 INFO:tasks.workunit.client.0.smithi045.stdout:op 2998 completed, throughput=5MB/sec 2021-07-31T02:15:03.352 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-3Sc_3bmwL1p2t2l off=10101374 len=1327540 2021-07-31T02:15:03.359 INFO:tasks.workunit.client.0.smithi045.stdout:op 2999 completed, throughput=5MB/sec 2021-07-31T02:15:03.359 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-qQygdfPgDD651pM off=15102983 len=774241 2021-07-31T02:15:03.364 INFO:tasks.workunit.client.0.smithi045.stdout:op 3000 completed, throughput=5MB/sec 2021-07-31T02:15:03.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:03 smithi154 conmon[34553]: cluster 2021-07-31T02:15:02.189624+0000 mgr.x (mgr.24542) 1191 : cluster [DBG] pgmap v1706: 9 pgs: 9 active+clean; 127 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 2.7 MiB/s rd, 1.6 MiB/s wr, 4 op/s 2021-07-31T02:15:03.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:03 smithi045 conmon[33655]: cluster 2021-07-31T02:15: 2021-07-31T02:15:03.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:03 smithi045 conmon[33655]: 02.189624+0000 mgr.x (mgr.24542) 1191 : cluster [DBG] pgmap v1706: 9 pgs: 9 active+clean; 127 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 2.7 MiB/s rd, 1.6 MiB/s wr, 4 op/s 2021-07-31T02:15:03.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:03 smithi045 conmon[39718]: cluster 2021-07-31T02:15:02.189624+0000 mgr.x 2021-07-31T02:15:03.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:03 smithi045 conmon[39718]: (mgr.24542) 1191 : cluster [DBG] pgmap v1706: 9 pgs: 9 active+clean; 127 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 2.7 MiB/s rd, 1.6 MiB/s wr, 4 op/s 2021-07-31T02:15:04.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:15:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:15:03.556Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:15:04.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:15:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:15:03.556Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:15:04.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:15:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:15:03.556Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:15:04.364 INFO:tasks.workunit.client.0.smithi045.stdout: 586: throughput=4.99MB/sec pending data=0 2021-07-31T02:15:04.365 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-XBSguK1hgyxjIij off=10405224 len=1652290 2021-07-31T02:15:04.368 INFO:tasks.workunit.client.0.smithi045.stdout:op 3001 completed, throughput=4.99MB/sec 2021-07-31T02:15:04.368 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-XMZC87IX9J0nhgA off=9203532 len=531637 2021-07-31T02:15:04.369 INFO:tasks.workunit.client.0.smithi045.stdout:op 3002 completed, throughput=5MB/sec 2021-07-31T02:15:04.370 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-fn1UFioeX4eKULC off=2448245 len=1456454 2021-07-31T02:15:04.379 INFO:tasks.workunit.client.0.smithi045.stdout:op 3003 completed, throughput=5MB/sec 2021-07-31T02:15:04.379 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Gw0Sr14i7YZ5MnM off=12135734 len=366339 2021-07-31T02:15:04.380 INFO:tasks.workunit.client.0.smithi045.stdout:op 3004 completed, throughput=5MB/sec 2021-07-31T02:15:04.380 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-AVXstKbXRT5q30v off=0 len=1854092 2021-07-31T02:15:04.383 INFO:tasks.workunit.client.0.smithi045.stdout:op 3005 completed, throughput=5MB/sec 2021-07-31T02:15:04.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:04 smithi154 conmon[34553]: cluster 2021-07-31T02:15:04.190148+0000 mgr.x (mgr.24542) 1192 : cluster [DBG] pgmap v1707: 9 pgs: 9 active+clean; 127 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 3.8 MiB/s rd, 2.5 MiB/s wr, 6 op/s 2021-07-31T02:15:04.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:04 smithi045 conmon[39718]: cluster 2021-07-31T02:15:04.190148+0000 mgr.x (mgr 2021-07-31T02:15:04.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:04 smithi045 conmon[39718]: .24542) 1192 : cluster [DBG] pgmap v1707: 9 pgs: 9 active+clean; 127 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 3.8 MiB/s rd, 2.5 MiB/s wr, 6 op/s 2021-07-31T02:15:04.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:04 smithi045 conmon[33655]: cluster 2021-07-31T02:15:04.190148+0000 mgr.x (mgr.24542) 2021-07-31T02:15:04.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:04 smithi045 conmon[33655]: 1192 : cluster [DBG] pgmap v1707: 9 pgs: 9 active+clean; 127 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 3.8 MiB/s rd, 2.5 MiB/s wr, 6 op/s 2021-07-31T02:15:05.384 INFO:tasks.workunit.client.0.smithi045.stdout: 587: throughput=4.99MB/sec pending data=0 2021-07-31T02:15:05.384 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-d1ptEIqRzFE8LYZ off=15924771 len=901045 2021-07-31T02:15:05.389 INFO:tasks.workunit.client.0.smithi045.stdout:op 3006 completed, throughput=4.99MB/sec 2021-07-31T02:15:05.389 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-y9F9KdoZFZjHGhQ off=20081945 len=181449 2021-07-31T02:15:05.390 INFO:tasks.workunit.client.0.smithi045.stdout:op 3007 completed, throughput=4.99MB/sec 2021-07-31T02:15:05.390 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-kynQz8aK99P2-mT off=6202167 len=546087 2021-07-31T02:15:05.394 INFO:tasks.workunit.client.0.smithi045.stdout:op 3008 completed, throughput=5MB/sec 2021-07-31T02:15:05.394 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-9niByNKVmOm4jad off=5415921 len=1564569 2021-07-31T02:15:05.397 INFO:tasks.workunit.client.0.smithi045.stdout:op 3009 completed, throughput=5MB/sec 2021-07-31T02:15:05.397 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-SDisC-CCUFAi1Fx off=18952579 len=267173 2021-07-31T02:15:05.400 INFO:tasks.workunit.client.0.smithi045.stdout:op 3010 completed, throughput=5MB/sec 2021-07-31T02:15:05.401 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-tR_L5l2A7l0_S8X off=1134089 len=1259306 2021-07-31T02:15:05.408 INFO:tasks.workunit.client.0.smithi045.stdout:op 3011 completed, throughput=5MB/sec 2021-07-31T02:15:05.408 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-TyvFW_XzxGsS5WG off=292732 len=1214374 2021-07-31T02:15:05.412 INFO:tasks.workunit.client.0.smithi045.stdout:op 3012 completed, throughput=5MB/sec 2021-07-31T02:15:06.412 INFO:tasks.workunit.client.0.smithi045.stdout: 588: throughput=4.99MB/sec pending data=0 2021-07-31T02:15:06.413 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-3iglE3PN_cH6IcG off=6933161 len=1645087 2021-07-31T02:15:06.421 INFO:tasks.workunit.client.0.smithi045.stdout:op 3013 completed, throughput=5MB/sec 2021-07-31T02:15:06.422 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-_ivv-kLi0zhtwQn off=534727 len=842599 2021-07-31T02:15:06.427 INFO:tasks.workunit.client.0.smithi045.stdout:op 3014 completed, throughput=5MB/sec 2021-07-31T02:15:06.427 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-kyBzeBknBC_QU_T off=1972884 len=1509510 2021-07-31T02:15:06.429 INFO:tasks.workunit.client.0.smithi045.stdout:op 3015 completed, throughput=5MB/sec 2021-07-31T02:15:06.430 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-P_93ADnVC5L4evc off=19202373 len=1239795 2021-07-31T02:15:06.431 INFO:tasks.workunit.client.0.smithi045.stdout:op 3016 completed, throughput=5MB/sec 2021-07-31T02:15:06.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:06 smithi154 conmon[34553]: cluster 2021-07-31T02:15:06.190682+0000 mgr.x (mgr.24542) 1193 : cluster [DBG] pgmap v1708: 9 pgs: 9 active+clean; 127 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 2.4 MiB/s rd, 1.8 MiB/s wr, 4 op/s 2021-07-31T02:15:06.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:06 smithi154 conmon[34553]: 2021-07-31T02:15:06.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:06 smithi045 conmon[39718]: cluster 2021-07-31T02:15:06.190682+0000 2021-07-31T02:15:06.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:06 smithi045 conmon[39718]: mgr.x (mgr.24542) 1193 : cluster [DBG] pgmap v1708: 9 pgs: 9 active+clean; 127 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 2.4 MiB/s rd, 1.8 MiB/s wr, 4 op/s 2021-07-31T02:15:06.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:06 smithi045 conmon[33655]: cluster 2021-07-31T02:15:06.190682+0000 mgr.x (mgr.24542) 1193 : cluster [DBG] 2021-07-31T02:15:06.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:06 smithi045 conmon[33655]: pgmap v1708: 9 pgs: 9 active+clean; 127 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 2.4 MiB/s rd, 1.8 MiB/s wr, 4 op/s 2021-07-31T02:15:07.432 INFO:tasks.workunit.client.0.smithi045.stdout: 589: throughput=4.99MB/sec pending data=0 2021-07-31T02:15:07.432 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-wDvfYPrXB92HFU6 off=819970 len=727006 2021-07-31T02:15:07.438 INFO:tasks.workunit.client.0.smithi045.stdout:op 3017 completed, throughput=4.99MB/sec 2021-07-31T02:15:07.438 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Px0UHcP4jRFcixI off=1674355 len=1145752 2021-07-31T02:15:07.440 INFO:tasks.workunit.client.0.smithi045.stdout:op 3018 completed, throughput=5MB/sec 2021-07-31T02:15:07.440 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-wvYoyqcNG1rknmY off=11231763 len=142329 2021-07-31T02:15:07.443 INFO:tasks.workunit.client.0.smithi045.stdout:op 3019 completed, throughput=5MB/sec 2021-07-31T02:15:07.443 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Uaf4pxezrL-itAZ off=8661863 len=179780 2021-07-31T02:15:07.444 INFO:tasks.workunit.client.0.smithi045.stdout:op 3020 completed, throughput=5MB/sec 2021-07-31T02:15:07.445 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-gnEm5GHucdNH_I6 off=5638358 len=1255991 2021-07-31T02:15:07.451 INFO:tasks.workunit.client.0.smithi045.stdout:op 3021 completed, throughput=5MB/sec 2021-07-31T02:15:07.451 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-lP25gbLtFz-gk3O off=11463749 len=933813 2021-07-31T02:15:07.454 INFO:tasks.workunit.client.0.smithi045.stdout:op 3022 completed, throughput=5MB/sec 2021-07-31T02:15:08.453 INFO:tasks.workunit.client.0.smithi045.stdout: 590: throughput=4.99MB/sec pending data=0 2021-07-31T02:15:08.453 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-BHsBtfG2eR-D6XL off=888907 len=913830 2021-07-31T02:15:08.455 INFO:tasks.workunit.client.0.smithi045.stdout:op 3023 completed, throughput=4.99MB/sec 2021-07-31T02:15:08.455 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-WX9AMjf1CQcOs-A off=1018772 len=258365 2021-07-31T02:15:08.456 INFO:tasks.workunit.client.0.smithi045.stdout:op 3024 completed, throughput=4.99MB/sec 2021-07-31T02:15:08.456 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-9LryLpHRoGXBdmw off=11611065 len=873956 2021-07-31T02:15:08.462 INFO:tasks.workunit.client.0.smithi045.stdout:op 3025 completed, throughput=4.99MB/sec 2021-07-31T02:15:08.462 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-_Q-NOj5i4nu8oe2 off=18040362 len=542403 2021-07-31T02:15:08.463 INFO:tasks.workunit.client.0.smithi045.stdout:op 3026 completed, throughput=5MB/sec 2021-07-31T02:15:08.463 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-kGont0arTT5mKYG off=3582928 len=686285 2021-07-31T02:15:08.464 INFO:tasks.workunit.client.0.smithi045.stdout:op 3027 completed, throughput=5MB/sec 2021-07-31T02:15:08.464 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-RqMMnU0dMHTf9SF off=5475009 len=42629 2021-07-31T02:15:08.465 INFO:tasks.workunit.client.0.smithi045.stdout:op 3028 completed, throughput=5MB/sec 2021-07-31T02:15:08.465 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ibnm7w8CKuSDiN6 off=3880059 len=1408309 2021-07-31T02:15:08.467 INFO:tasks.workunit.client.0.smithi045.stdout:op 3029 completed, throughput=5MB/sec 2021-07-31T02:15:08.468 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-tmg3oT4UBFrQI0b off=1551314 len=1038195 2021-07-31T02:15:08.469 INFO:tasks.workunit.client.0.smithi045.stdout:op 3030 completed, throughput=5MB/sec 2021-07-31T02:15:08.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:08 smithi154 conmon[34553]: cluster 2021-07-31T02:15:08.191116+0000 mgr.x (mgr.24542) 1194 : cluster [DBG] pgmap v1709: 9 pgs: 9 active+clean; 127 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 2.4 MiB/s rd, 1.8 MiB/s wr, 4 op/s 2021-07-31T02:15:08.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:08 smithi045 conmon[39718]: cluster 2021-07-31T02:15:08.191116+0000 2021-07-31T02:15:08.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:08 smithi045 conmon[39718]: mgr.x (mgr.24542) 1194 : cluster [DBG] pgmap v1709: 9 pgs: 9 active+clean; 127 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 2.4 MiB/s rd, 1.8 MiB/s wr, 4 op/s 2021-07-31T02:15:08.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:08 smithi045 conmon[33655]: cluster 2021-07-31T02:15:08.191116+0000 2021-07-31T02:15:08.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:08 smithi045 conmon[33655]: mgr.x (mgr.24542) 1194 : cluster [DBG] pgmap v1709: 9 pgs: 9 active+clean; 127 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 2.4 MiB/s rd, 1.8 MiB/s wr, 4 op/s 2021-07-31T02:15:09.470 INFO:tasks.workunit.client.0.smithi045.stdout: 591: throughput=4.99MB/sec pending data=0 2021-07-31T02:15:09.470 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-zs3CzS2LiRKrju8 off=20531148 len=182698 2021-07-31T02:15:09.473 INFO:tasks.workunit.client.0.smithi045.stdout:op 3031 completed, throughput=4.99MB/sec 2021-07-31T02:15:09.473 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-fIRDgyky-KUgEzO off=8969453 len=847478 2021-07-31T02:15:09.475 INFO:tasks.workunit.client.0.smithi045.stdout:op 3032 completed, throughput=4.99MB/sec 2021-07-31T02:15:09.475 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Ji7lyFomuV3n8hG off=7914041 len=545972 2021-07-31T02:15:09.481 INFO:tasks.workunit.client.0.smithi045.stdout:op 3033 completed, throughput=4.99MB/sec 2021-07-31T02:15:09.481 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-xSubuR0KLJNlQTY off=5083026 len=574156 2021-07-31T02:15:09.481 INFO:tasks.workunit.client.0.smithi045.stdout:op 3034 completed, throughput=5MB/sec 2021-07-31T02:15:09.481 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj--9lCR0zRH3wo5Bx off=17164728 len=962405 2021-07-31T02:15:09.485 INFO:tasks.workunit.client.0.smithi045.stdout:op 3035 completed, throughput=5MB/sec 2021-07-31T02:15:09.485 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Nb-nzI5WCdZlk0F off=9746678 len=797659 2021-07-31T02:15:09.490 INFO:tasks.workunit.client.0.smithi045.stdout:op 3036 completed, throughput=5MB/sec 2021-07-31T02:15:09.491 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-U6PX6QAf6q9xJve off=15822221 len=1606405 2021-07-31T02:15:09.492 INFO:tasks.workunit.client.0.smithi045.stdout:op 3037 completed, throughput=5MB/sec 2021-07-31T02:15:10.493 INFO:tasks.workunit.client.0.smithi045.stdout: 592: throughput=4.99MB/sec pending data=0 2021-07-31T02:15:10.493 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-3A-b36bHM2Q4I6m off=2291508 len=1541931 2021-07-31T02:15:10.496 INFO:tasks.workunit.client.0.smithi045.stdout:op 3038 completed, throughput=5MB/sec 2021-07-31T02:15:10.496 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Ru0xJHLkWSM2LDs off=9510884 len=986988 2021-07-31T02:15:10.498 INFO:tasks.workunit.client.0.smithi045.stdout:op 3039 completed, throughput=5MB/sec 2021-07-31T02:15:10.498 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-eVWhaai5femj_MK off=16456568 len=1031804 2021-07-31T02:15:10.500 INFO:tasks.workunit.client.0.smithi045.stdout:op 3040 completed, throughput=5MB/sec 2021-07-31T02:15:10.500 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-uy9oIRigLiaMOsC off=3982608 len=1238561 2021-07-31T02:15:10.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:10 smithi154 conmon[34553]: cluster 2021-07-31T02:15:10.191740+0000 mgr.x (mgr.24542) 1195 : cluster [DBG] pgmap v1710: 9 pgs: 9 active+clean; 127 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 3.7 MiB/s rd, 2.7 MiB/s wr, 6 op/s 2021-07-31T02:15:10.504 INFO:tasks.workunit.client.0.smithi045.stdout:op 3041 completed, throughput=5MB/sec 2021-07-31T02:15:10.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:10 smithi045 conmon[39718]: cluster 2021-07-31T02:15:10 2021-07-31T02:15:10.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:10 smithi045 conmon[39718]: .191740+0000 mgr.x (mgr.24542) 1195 : cluster [DBG] pgmap v1710: 9 pgs: 9 active+clean; 127 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 3.7 MiB/s rd, 2.7 MiB/s wr, 6 op/s 2021-07-31T02:15:10.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:10 smithi045 conmon[33655]: cluster 2021-07-31T02:15:10.191740+0000 mgr.x (mgr.24542) 1195 : cluster [DBG] 2021-07-31T02:15:10.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:10 smithi045 conmon[33655]: pgmap v1710: 9 pgs: 9 active+clean; 127 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 3.7 MiB/s rd, 2.7 MiB/s wr, 6 op/s 2021-07-31T02:15:11.503 INFO:tasks.workunit.client.0.smithi045.stdout: 593: throughput=4.99MB/sec pending data=0 2021-07-31T02:15:11.504 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-JfXGjd_SckgS_xq off=12130489 len=1660663 2021-07-31T02:15:11.506 INFO:tasks.workunit.client.0.smithi045.stdout:op 3042 completed, throughput=4.99MB/sec 2021-07-31T02:15:11.507 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-EVJkepkgSif_T1K off=445723 len=342445 2021-07-31T02:15:11.511 INFO:tasks.workunit.client.0.smithi045.stdout:op 3043 completed, throughput=5MB/sec 2021-07-31T02:15:11.511 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-dzColOP6EkMC9wA off=9119376 len=1377034 2021-07-31T02:15:11.518 INFO:tasks.workunit.client.0.smithi045.stdout:op 3044 completed, throughput=5MB/sec 2021-07-31T02:15:11.518 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-cHb5kdIz50evCSU off=305147 len=208936 2021-07-31T02:15:11.519 INFO:tasks.workunit.client.0.smithi045.stdout:op 3045 completed, throughput=5MB/sec 2021-07-31T02:15:11.519 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-l9ZZP0kqnJiVtVh off=6614857 len=311696 2021-07-31T02:15:11.522 INFO:tasks.workunit.client.0.smithi045.stdout:op 3046 completed, throughput=5MB/sec 2021-07-31T02:15:11.523 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj--wHon379yiToPxW off=2936848 len=992284 2021-07-31T02:15:11.524 INFO:tasks.workunit.client.0.smithi045.stdout:op 3047 completed, throughput=5MB/sec 2021-07-31T02:15:11.524 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-6IVUanuNKa4G-yT off=302755 len=751831 2021-07-31T02:15:11.530 INFO:tasks.workunit.client.0.smithi045.stdout:op 3048 completed, throughput=5MB/sec 2021-07-31T02:15:12.530 INFO:tasks.workunit.client.0.smithi045.stdout: 594: throughput=4.99MB/sec pending data=0 2021-07-31T02:15:12.530 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-RMbiocUY0u4xT8Y off=8281044 len=531501 2021-07-31T02:15:12.535 INFO:tasks.workunit.client.0.smithi045.stdout:op 3049 completed, throughput=4.99MB/sec 2021-07-31T02:15:12.535 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-7OHiZ_iP90gZKDW off=7347715 len=1296045 2021-07-31T02:15:12.541 INFO:tasks.workunit.client.0.smithi045.stdout:op 3050 completed, throughput=5MB/sec 2021-07-31T02:15:12.541 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-K9OMDhw-eyC1wEl off=2867001 len=1061874 2021-07-31T02:15:12.547 INFO:tasks.workunit.client.0.smithi045.stdout:op 3051 completed, throughput=5MB/sec 2021-07-31T02:15:12.547 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-XG8GtzZuTBVSy1J off=12707310 len=868841 2021-07-31T02:15:12.549 INFO:tasks.workunit.client.0.smithi045.stdout:op 3052 completed, throughput=5MB/sec 2021-07-31T02:15:12.550 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Z-ueRoYJcT7aS8w off=11845199 len=840752 2021-07-31T02:15:12.555 INFO:tasks.workunit.client.0.smithi045.stdout:op 3053 completed, throughput=5MB/sec 2021-07-31T02:15:12.555 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-3dvNOFNSmrYt3CF off=13213231 len=1425360 2021-07-31T02:15:12.557 INFO:tasks.workunit.client.0.smithi045.stdout:op 3054 completed, throughput=5MB/sec 2021-07-31T02:15:13.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:13 smithi154 conmon[34553]: cluster 2021-07-31T02:15:12.192276+0000 mgr.x (mgr.24542) 1196 : cluster [DBG] pgmap v1711: 9 pgs: 9 active+clean; 127 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 2.4 MiB/s rd, 1.8 MiB/s wr, 4 op/s 2021-07-31T02:15:13.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:13 smithi045 conmon[39718]: cluster 2021-07-31T02:15: 2021-07-31T02:15:13.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:13 smithi045 conmon[39718]: 12.192276+0000 mgr.x (mgr.24542) 1196 : cluster [DBG] pgmap v1711: 9 pgs: 9 active+clean; 127 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 2.4 MiB/s rd, 1.8 MiB/s wr, 4 op/s 2021-07-31T02:15:13.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:13 smithi045 conmon[33655]: cluster 2021-07-31T02:15:12.192276+0000 mgr.x (mgr.24542) 1196 : cluster [DBG] pgmap v1711: 9 pgs: 9 active+clean; 127 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 2.4 MiB/s rd, 1.8 MiB/s wr, 4 op/s 2021-07-31T02:15:13.559 INFO:tasks.workunit.client.0.smithi045.stdout: 595: throughput=4.99MB/sec pending data=0 2021-07-31T02:15:13.560 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj--B9anRLKpUSVRn6 off=4071887 len=2088026 2021-07-31T02:15:13.562 INFO:tasks.workunit.client.0.smithi045.stdout:op 3055 completed, throughput=5MB/sec 2021-07-31T02:15:13.562 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ZHLMXcyYMO42gYs off=9716652 len=768006 2021-07-31T02:15:13.566 INFO:tasks.workunit.client.0.smithi045.stdout:op 3056 completed, throughput=5MB/sec 2021-07-31T02:15:13.567 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-gzN3mBsi5wYAVu4 off=8198657 len=616257 2021-07-31T02:15:13.568 INFO:tasks.workunit.client.0.smithi045.stdout:op 3057 completed, throughput=5MB/sec 2021-07-31T02:15:13.568 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-we8qwDq6mqS2EMf off=2643469 len=1507363 2021-07-31T02:15:13.577 INFO:tasks.workunit.client.0.smithi045.stdout:op 3058 completed, throughput=5MB/sec 2021-07-31T02:15:14.027 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:15:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:15:13.556Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:15:14.028 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:15:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:15:13.556Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:15:14.028 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:15:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:15:13.556Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:15:14.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:14 smithi045 conmon[39718]: cluster 2021-07-31T02:15:14.192797+0000 mgr.x ( 2021-07-31T02:15:14.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:14 smithi045 conmon[39718]: mgr.24542) 1197 : cluster [DBG] pgmap v1712: 9 pgs: 9 active+clean; 127 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 3.6 MiB/s rd, 2.7 MiB/s wr, 6 op/s 2021-07-31T02:15:14.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:14 smithi045 conmon[33655]: cluster 2021-07-31T02:15:14.192797+0000 mgr.x (mgr.24542) 1197 : 2021-07-31T02:15:14.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:14 smithi045 conmon[33655]: cluster [DBG] pgmap v1712: 9 pgs: 9 active+clean; 127 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 3.6 MiB/s rd, 2.7 MiB/s wr, 6 op/s 2021-07-31T02:15:14.577 INFO:tasks.workunit.client.0.smithi045.stdout: 596: throughput=4.99MB/sec pending data=0 2021-07-31T02:15:14.577 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-WCCtM2rXFaq3Y_l off=7341443 len=1175227 2021-07-31T02:15:14.579 INFO:tasks.workunit.client.0.smithi045.stdout:op 3059 completed, throughput=4.99MB/sec 2021-07-31T02:15:14.580 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-rOb9wzen_B8s7C0 off=683779 len=1051044 2021-07-31T02:15:14.586 INFO:tasks.workunit.client.0.smithi045.stdout:op 3060 completed, throughput=5MB/sec 2021-07-31T02:15:14.586 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-v1-GWMH2leiW1uO off=1754686 len=1124707 2021-07-31T02:15:14.592 INFO:tasks.workunit.client.0.smithi045.stdout:op 3061 completed, throughput=5MB/sec 2021-07-31T02:15:14.592 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-n2OSoKtUKLHYnlu off=0 len=1109302 2021-07-31T02:15:14.596 INFO:tasks.workunit.client.0.smithi045.stdout:op 3062 completed, throughput=5MB/sec 2021-07-31T02:15:14.597 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-i1J8N_HUvmnvS2d off=4070527 len=1020235 2021-07-31T02:15:14.601 INFO:tasks.workunit.client.0.smithi045.stdout:op 3063 completed, throughput=5MB/sec 2021-07-31T02:15:14.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:14 smithi154 conmon[34553]: cluster 2021-07-31T02:15:14.192797+0000 mgr.x (mgr.24542) 1197 : cluster [DBG] pgmap v1712: 9 pgs: 9 active+clean; 127 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 3.6 MiB/s rd, 2.7 MiB/s wr, 6 op/s 2021-07-31T02:15:15.602 INFO:tasks.workunit.client.0.smithi045.stdout: 597: throughput=4.99MB/sec pending data=0 2021-07-31T02:15:15.602 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-TUaXo4N2Tz_P92T off=1788336 len=89736 2021-07-31T02:15:15.603 INFO:tasks.workunit.client.0.smithi045.stdout:op 3064 completed, throughput=4.99MB/sec 2021-07-31T02:15:15.603 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Ygcl2FsJ7_EP3en off=1003798 len=563162 2021-07-31T02:15:15.607 INFO:tasks.workunit.client.0.smithi045.stdout:op 3065 completed, throughput=4.99MB/sec 2021-07-31T02:15:15.607 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-rE7GUzvKrtOsgTs off=8504227 len=936837 2021-07-31T02:15:15.612 INFO:tasks.workunit.client.0.smithi045.stdout:op 3066 completed, throughput=5MB/sec 2021-07-31T02:15:15.612 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-9B5zt0U1_PSW0jy off=936783 len=2045026 2021-07-31T02:15:15.625 INFO:tasks.workunit.client.0.smithi045.stdout:op 3067 completed, throughput=5MB/sec 2021-07-31T02:15:15.626 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-lmt4J6X5R7gp8Nw off=327146 len=1337103 2021-07-31T02:15:15.626 INFO:tasks.workunit.client.0.smithi045.stdout:op 3068 completed, throughput=5MB/sec 2021-07-31T02:15:16.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:16 smithi045 conmon[33655]: cluster 2021-07-31T02:15 2021-07-31T02:15:16.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:16 smithi045 conmon[33655]: :16.193339+0000 mgr.x (mgr.24542) 1198 : cluster [DBG] pgmap v1713: 9 pgs: 9 active+clean; 127 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.8 MiB/s wr, 4 op/s 2021-07-31T02:15:16.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:16 smithi045 conmon[39718]: cluster 2021-07-31T02:15:16.193339+0000 mgr.x (mgr 2021-07-31T02:15:16.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:16 smithi045 conmon[39718]: .24542) 1198 : cluster [DBG] pgmap v1713: 9 pgs: 9 active+clean; 127 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.8 MiB/s wr, 4 op/s 2021-07-31T02:15:16.594 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:15:16.624 INFO:tasks.workunit.client.0.smithi045.stdout: 598: throughput=4.99MB/sec pending data=0 2021-07-31T02:15:16.625 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-8z2zEeJ6_STTO5E off=7891510 len=1159068 2021-07-31T02:15:16.625 INFO:tasks.workunit.client.0.smithi045.stdout:op 3069 completed, throughput=4.99MB/sec 2021-07-31T02:15:16.626 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-76eg5kgPjxKniY6 off=2858636 len=1882180 2021-07-31T02:15:16.633 INFO:tasks.workunit.client.0.smithi045.stdout:op 3070 completed, throughput=5MB/sec 2021-07-31T02:15:16.633 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-NdwzCnpQJFsG531 off=19642430 len=90359 2021-07-31T02:15:16.634 INFO:tasks.workunit.client.0.smithi045.stdout:op 3071 completed, throughput=5MB/sec 2021-07-31T02:15:16.635 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-j-braWVE2TFcoP7 off=5945430 len=2079139 2021-07-31T02:15:16.636 INFO:tasks.workunit.client.0.smithi045.stdout:op 3072 completed, throughput=5MB/sec 2021-07-31T02:15:16.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:16 smithi154 conmon[34553]: cluster 2021-07-31T02:15:16.193339+0000 mgr.x (mgr.24542) 1198 : cluster [DBG] pgmap v1713: 9 pgs: 9 active+clean; 127 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.8 MiB/s wr, 4 op/s 2021-07-31T02:15:16.873 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:15:16.873 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 30m ago 30m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:15:16.873 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 27m ago 30m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:15:16.874 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 27m ago 34m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:15:16.874 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 30m ago 36m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:15:16.874 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 30m ago 36m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:15:16.874 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 27m ago 34m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:15:16.875 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 30m ago 34m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:15:16.875 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 30m ago 31m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:15:16.875 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 27m ago 31m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:15:16.875 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 30m ago 33m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:15:16.875 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 30m ago 33m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:15:16.876 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 30m ago 33m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:15:16.876 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 30m ago 32m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:15:16.876 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 27m ago 32m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:15:16.876 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 27m ago 32m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:15:16.877 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 27m ago 31m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:15:16.877 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 27m ago 31m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:15:16.877 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 27m ago 31m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:15:17.208 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:15:17.209 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:15:17.209 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:15:17.209 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:15:17.210 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:15:17.210 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:15:17.210 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:15:17.210 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:15:17.211 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:15:17.211 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:15:17.211 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:15:17.211 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:15:17.211 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:15:17.212 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:15:17.212 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:15:17.212 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:15:17.212 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:15:17.637 INFO:tasks.workunit.client.0.smithi045.stdout: 599: throughput=4.99MB/sec pending data=0 2021-07-31T02:15:17.637 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-PA6vp8ZA0gXFClw off=2798541 len=1059162 2021-07-31T02:15:17.642 INFO:tasks.workunit.client.0.smithi045.stdout:op 3073 completed, throughput=4.99MB/sec 2021-07-31T02:15:17.643 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-i3mfFWSraVP7WZO off=2378214 len=1699118 2021-07-31T02:15:17.649 INFO:tasks.workunit.client.0.smithi045.stdout:op 3074 completed, throughput=5MB/sec 2021-07-31T02:15:17.650 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-KuRAYuVyXDeR3Za off=13588068 len=436973 2021-07-31T02:15:17.651 INFO:tasks.workunit.client.0.smithi045.stdout:op 3075 completed, throughput=5MB/sec 2021-07-31T02:15:17.652 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-SlrZolOF58yMhIT off=3228114 len=370139 2021-07-31T02:15:17.654 INFO:tasks.workunit.client.0.smithi045.stdout:op 3076 completed, throughput=5MB/sec 2021-07-31T02:15:17.654 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-3qXkF3ghetnhOf7 off=7863716 len=2074648 2021-07-31T02:15:17.664 INFO:tasks.workunit.client.0.smithi045.stdout:op 3077 completed, throughput=5MB/sec 2021-07-31T02:15:18.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:18 smithi154 conmon[34553]: audit 2021-07-31T02:15:16.583169+0000 mgr.x (mgr.24542) 1199 : audit [DBG] from='client.25984 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:15:18.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:18 smithi154 conmon[34553]: audit 2021-07-31T02:15:16.865240+0000 mgr.x (mgr.24542) 1200 : audit [DBG] from='client.16083 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:15:18.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:18 smithi154 conmon[34553]: audit 2021-07-31T02:15:17.209103+0000 mon.a (mon.0) 1881 : audit [DBG] from='client.? 172.21.15.45:0/1186372122' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:15:18.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:18 smithi045 conmon[33655]: audit 2021-07-31T02:15:16.583169+0000 2021-07-31T02:15:18.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:18 smithi045 conmon[33655]: mgr.x (mgr.24542) 1199 : audit [DBG] from='client.25984 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:15:18.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:18 smithi045 conmon[33655]: audit 2021-07-31T02:15:16.865240+0000 mgr.x (mgr.24542) 1200 : audit [DBG] from='client.16083 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:15:18.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:18 smithi045 conmon[33655]: audit 2021-07-31T02 2021-07-31T02:15:18.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:18 smithi045 conmon[33655]: :15:17.209103+0000 mon.a (mon.0) 1881 : audit 2021-07-31T02:15:18.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:18 smithi045 conmon[33655]: [DBG] from='client.? 172.21.15.45:0/1186372122' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:15:18.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:18 smithi045 conmon[39718]: audit 2021-07-31T02:15:16.583169+0000 mgr.x ( 2021-07-31T02:15:18.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:18 smithi045 conmon[39718]: mgr.24542) 1199 : audit [DBG] from='client.25984 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:15:18.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:18 smithi045 conmon[39718]: audit 2021-07-31T02:15:16.865240+0000 mgr.x (mgr.24542) 1200 : audit 2021-07-31T02:15:18.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:18 smithi045 conmon[39718]: [DBG] from='client.16083 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:15:18.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:18 smithi045 conmon[39718]: audit 2021-07-31T02:15:17.209103+0000 2021-07-31T02:15:18.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:18 smithi045 conmon[39718]: mon.a (mon.0) 1881 : audit [DBG] from='client.? 172.21.15.45:0/1186372122' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:15:18.664 INFO:tasks.workunit.client.0.smithi045.stdout: 600: throughput=4.99MB/sec pending data=0 2021-07-31T02:15:18.665 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-RqRIy09u5pTWbBq off=2123448 len=1714150 2021-07-31T02:15:18.674 INFO:tasks.workunit.client.0.smithi045.stdout:op 3078 completed, throughput=5MB/sec 2021-07-31T02:15:18.674 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-JkjuPnTXLW531kV off=3775012 len=1313587 2021-07-31T02:15:18.677 INFO:tasks.workunit.client.0.smithi045.stdout:op 3079 completed, throughput=5MB/sec 2021-07-31T02:15:18.677 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-3HvjtlK-ms_CG5E off=6816529 len=556155 2021-07-31T02:15:18.681 INFO:tasks.workunit.client.0.smithi045.stdout:op 3080 completed, throughput=5MB/sec 2021-07-31T02:15:18.681 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Z9iy9Uk3l0GBAde off=8522896 len=308402 2021-07-31T02:15:18.682 INFO:tasks.workunit.client.0.smithi045.stdout:op 3081 completed, throughput=5MB/sec 2021-07-31T02:15:18.682 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-V_agBvFwL0V1on_ off=4003938 len=818809 2021-07-31T02:15:18.684 INFO:tasks.workunit.client.0.smithi045.stdout:op 3082 completed, throughput=5MB/sec 2021-07-31T02:15:18.684 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-H2-DPNw3MKp_8cx off=5319764 len=1666629 2021-07-31T02:15:18.692 INFO:tasks.workunit.client.0.smithi045.stdout:op 3083 completed, throughput=5MB/sec 2021-07-31T02:15:19.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:19 smithi154 conmon[34553]: cluster 2021-07-31T02:15:18.193893+0000 mgr.x (mgr.24542) 1201 : cluster [DBG] pgmap v1714: 9 pgs: 9 active+clean; 127 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.8 MiB/s wr, 4 op/s 2021-07-31T02:15:19.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:19 smithi045 conmon[33655]: cluster 2021-07-31T02:15:18.193893+0000 mgr.x (mgr.24542) 1201 : cluster [DBG] 2021-07-31T02:15:19.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:19 smithi045 conmon[33655]: pgmap v1714: 9 pgs: 9 active+clean; 127 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.8 MiB/s wr, 4 op/s 2021-07-31T02:15:19.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:19 smithi045 conmon[39718]: cluster 2021-07-31T02:15:18.193893 2021-07-31T02:15:19.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:19 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 1201 : cluster [DBG] pgmap v1714: 9 pgs: 9 active+clean; 127 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.8 MiB/s wr, 4 op/s 2021-07-31T02:15:19.692 INFO:tasks.workunit.client.0.smithi045.stdout: 601: throughput=4.99MB/sec pending data=0 2021-07-31T02:15:19.692 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-f-KFuA8L1tAG1R7 off=3280086 len=1242601 2021-07-31T02:15:19.693 INFO:tasks.workunit.client.0.smithi045.stdout:op 3084 completed, throughput=5MB/sec 2021-07-31T02:15:19.694 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-FYtIbTH42MoUGGA off=892461 len=621586 2021-07-31T02:15:19.694 INFO:tasks.workunit.client.0.smithi045.stdout:op 3085 completed, throughput=5MB/sec 2021-07-31T02:15:19.695 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-5o-_UBK-1ZP8OXC off=10277488 len=2063464 2021-07-31T02:15:19.698 INFO:tasks.workunit.client.0.smithi045.stdout:op 3086 completed, throughput=5MB/sec 2021-07-31T02:15:20.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:20 smithi045 conmon[33655]: cluster 2021-07-31T02:15:20.194422+0000 mgr.x (mgr.24542) 1202 : 2021-07-31T02:15:20.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:20 smithi045 conmon[33655]: cluster [DBG] pgmap v1715: 9 pgs: 9 active+clean; 127 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 3.3 MiB/s rd, 3.2 MiB/s wr, 6 op/s 2021-07-31T02:15:20.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:20 smithi045 conmon[39718]: cluster 2021-07-31T02:15:20.194422+0000 mgr.x (mgr.24542) 1202 2021-07-31T02:15:20.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:20 smithi045 conmon[39718]: : cluster [DBG] pgmap v1715: 9 pgs: 9 active+clean; 127 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 3.3 MiB/s rd, 3.2 MiB/s wr, 6 op/s 2021-07-31T02:15:20.699 INFO:tasks.workunit.client.0.smithi045.stdout: 602: throughput=4.99MB/sec pending data=0 2021-07-31T02:15:20.699 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ZE85RyL4SUW6dZB off=5220361 len=1249799 2021-07-31T02:15:20.701 INFO:tasks.workunit.client.0.smithi045.stdout:op 3087 completed, throughput=4.99MB/sec 2021-07-31T02:15:20.701 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Efy4kGZv0miRBp4 off=2634538 len=515481 2021-07-31T02:15:20.703 INFO:tasks.workunit.client.0.smithi045.stdout:op 3088 completed, throughput=4.99MB/sec 2021-07-31T02:15:20.703 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-9HygaF8I6RCB78G off=326160 len=492156 2021-07-31T02:15:20.707 INFO:tasks.workunit.client.0.smithi045.stdout:op 3089 completed, throughput=5MB/sec 2021-07-31T02:15:20.707 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-LHMqu9GVIl2mdz6 off=21718282 len=600071 2021-07-31T02:15:20.711 INFO:tasks.workunit.client.0.smithi045.stdout:op 3090 completed, throughput=5MB/sec 2021-07-31T02:15:20.712 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ipOFAOeLdN4OWNy off=9151608 len=1629215 2021-07-31T02:15:20.714 INFO:tasks.workunit.client.0.smithi045.stdout:op 3091 completed, throughput=5MB/sec 2021-07-31T02:15:20.714 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-iXRulBWzMJKUC1p off=6030174 len=64137 2021-07-31T02:15:20.717 INFO:tasks.workunit.client.0.smithi045.stdout:op 3092 completed, throughput=5MB/sec 2021-07-31T02:15:20.717 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Sx24TesjXcqpOed off=2215459 len=167867 2021-07-31T02:15:20.717 INFO:tasks.workunit.client.0.smithi045.stdout:op 3093 completed, throughput=5MB/sec 2021-07-31T02:15:20.718 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-uozWcmoHeIOIkJZ off=6755190 len=775098 2021-07-31T02:15:20.719 INFO:tasks.workunit.client.0.smithi045.stdout:op 3094 completed, throughput=5MB/sec 2021-07-31T02:15:20.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:20 smithi154 conmon[34553]: cluster 2021-07-31T02:15:20.194422+0000 mgr.x (mgr.24542) 1202 : cluster [DBG] pgmap v1715: 9 pgs: 9 active+clean; 127 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 3.3 MiB/s rd, 3.2 MiB/s wr, 6 op/s 2021-07-31T02:15:21.719 INFO:tasks.workunit.client.0.smithi045.stdout: 603: throughput=4.99MB/sec pending data=0 2021-07-31T02:15:21.720 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-wEiVCHpURr2PzG0 off=5888360 len=1126472 2021-07-31T02:15:21.726 INFO:tasks.workunit.client.0.smithi045.stdout:op 3095 completed, throughput=4.99MB/sec 2021-07-31T02:15:21.726 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-sOc_qWtMgjj7m_s off=6274705 len=2089110 2021-07-31T02:15:21.736 INFO:tasks.workunit.client.0.smithi045.stdout:op 3096 completed, throughput=5MB/sec 2021-07-31T02:15:21.736 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-lzjgC0oRJLYfir2 off=4673085 len=848803 2021-07-31T02:15:21.741 INFO:tasks.workunit.client.0.smithi045.stdout:op 3097 completed, throughput=5MB/sec 2021-07-31T02:15:21.742 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-cpB313K_GpcnY7k off=23833798 len=691417 2021-07-31T02:15:21.747 INFO:tasks.workunit.client.0.smithi045.stdout:op 3098 completed, throughput=5MB/sec 2021-07-31T02:15:21.747 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-V4PNVzc92TtQ_V4 off=500715 len=1146622 2021-07-31T02:15:21.752 INFO:tasks.workunit.client.0.smithi045.stdout:op 3099 completed, throughput=5MB/sec 2021-07-31T02:15:22.753 INFO:tasks.workunit.client.0.smithi045.stdout: 604: throughput=4.99MB/sec pending data=0 2021-07-31T02:15:22.753 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-kd3_H6wVTIzZHWr off=0 len=1262547 2021-07-31T02:15:22.760 INFO:tasks.workunit.client.0.smithi045.stdout:op 3100 completed, throughput=4.99MB/sec 2021-07-31T02:15:22.760 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-RxqdKWFVXKgcwgi off=16380401 len=1203371 2021-07-31T02:15:22.761 INFO:tasks.workunit.client.0.smithi045.stdout:op 3101 completed, throughput=5MB/sec 2021-07-31T02:15:22.762 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-mnWsiMVqz9afmYL off=21838388 len=2084189 2021-07-31T02:15:22.769 INFO:tasks.workunit.client.0.smithi045.stdout:op 3102 completed, throughput=5MB/sec 2021-07-31T02:15:23.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:23 smithi154 conmon[34553]: cluster 2021-07-31T02:15:22.194858+0000 mgr.x (mgr.24542) 1203 : cluster [DBG] pgmap v1716: 9 pgs: 9 active+clean; 127 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2021-07-31T02:15:23.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:23 smithi045 conmon[33655]: cluster 2021-07-31T02:15:22.194858+0000 mgr.x (mgr.24542) 1203 : cluster 2021-07-31T02:15:23.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:23 smithi045 conmon[33655]: [DBG] pgmap v1716: 9 pgs: 9 active+clean; 127 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2021-07-31T02:15:23.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:23 smithi045 conmon[39718]: cluster 2021-07-31T02:15:22.194858+0000 2021-07-31T02:15:23.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:23 smithi045 conmon[39718]: mgr.x (mgr.24542) 1203 : cluster [DBG] pgmap v1716: 9 pgs: 9 active+clean; 127 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2021-07-31T02:15:23.770 INFO:tasks.workunit.client.0.smithi045.stdout: 605: throughput=4.99MB/sec pending data=0 2021-07-31T02:15:23.770 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-TZ0iuPqKWUVBxgK off=2883565 len=248727 2021-07-31T02:15:23.773 INFO:tasks.workunit.client.0.smithi045.stdout:op 3103 completed, throughput=4.99MB/sec 2021-07-31T02:15:23.773 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-SRzLytsqmpMCoH4 off=2886242 len=1944721 2021-07-31T02:15:23.777 INFO:tasks.workunit.client.0.smithi045.stdout:op 3104 completed, throughput=5MB/sec 2021-07-31T02:15:23.777 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-m0kbI_a0_KOoFNC off=6252526 len=176252 2021-07-31T02:15:23.780 INFO:tasks.workunit.client.0.smithi045.stdout:op 3105 completed, throughput=5MB/sec 2021-07-31T02:15:23.780 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-aE8T3PhJ0cEcpLb off=13515472 len=11326 2021-07-31T02:15:23.783 INFO:tasks.workunit.client.0.smithi045.stdout:op 3106 completed, throughput=5MB/sec 2021-07-31T02:15:23.783 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Z-GZCJm1s5rbgPJ off=965642 len=497760 2021-07-31T02:15:23.787 INFO:tasks.workunit.client.0.smithi045.stdout:op 3107 completed, throughput=5MB/sec 2021-07-31T02:15:23.787 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-tHpiq8Lbk696EDF off=7619333 len=713079 2021-07-31T02:15:23.792 INFO:tasks.workunit.client.0.smithi045.stdout:op 3108 completed, throughput=5MB/sec 2021-07-31T02:15:23.792 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-KTTH0FB57C5D0p2 off=8523775 len=708218 2021-07-31T02:15:23.797 INFO:tasks.workunit.client.0.smithi045.stdout:op 3109 completed, throughput=5MB/sec 2021-07-31T02:15:23.797 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-VlvWO0ioXujibSL off=7061585 len=687720 2021-07-31T02:15:23.798 INFO:tasks.workunit.client.0.smithi045.stdout:op 3110 completed, throughput=5MB/sec 2021-07-31T02:15:23.799 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-LymoCatHC_3B6lH off=10192282 len=1048499 2021-07-31T02:15:23.806 INFO:tasks.workunit.client.0.smithi045.stdout:op 3111 completed, throughput=5MB/sec 2021-07-31T02:15:24.029 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:15:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:15:23.556Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:15:24.029 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:15:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:15:23.557Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:15:24.029 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:15:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:15:23.557Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:15:24.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:24 smithi045 conmon[39718]: cluster 2021-07-31T02:15:24.195295+0000 mgr.x 2021-07-31T02:15:24.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:24 smithi045 conmon[39718]: (mgr.24542) 1204 : cluster [DBG] pgmap v1717: 9 pgs: 9 active+clean; 127 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 3.0 MiB/s rd, 3.4 MiB/s wr, 6 op/s 2021-07-31T02:15:24.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:24 smithi045 conmon[33655]: cluster 2021-07-31T02:15:24.195295+0000 mgr.x (mgr.24542) 1204 : cluster [DBG] pgmap v1717: 9 pgs: 9 active+clean; 127 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 3.0 MiB/s rd, 3.4 MiB/s wr, 6 op/s 2021-07-31T02:15:24.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:24 smithi154 conmon[34553]: cluster 2021-07-31T02:15:24.195295+0000 mgr.x (mgr.24542) 1204 : cluster [DBG] 2021-07-31T02:15:24.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:24 smithi154 conmon[34553]: pgmap v1717: 9 pgs: 9 active+clean; 127 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 3.0 MiB/s rd, 3.4 MiB/s wr, 6 op/s 2021-07-31T02:15:24.806 INFO:tasks.workunit.client.0.smithi045.stdout: 606: throughput=4.99MB/sec pending data=0 2021-07-31T02:15:24.807 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-YvMn0IvfUtDKP-2 off=2352276 len=1554448 2021-07-31T02:15:24.815 INFO:tasks.workunit.client.0.smithi045.stdout:op 3112 completed, throughput=5MB/sec 2021-07-31T02:15:24.815 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-rsdLMQ3G7Qc8hYs off=504349 len=560056 2021-07-31T02:15:24.816 INFO:tasks.workunit.client.0.smithi045.stdout:op 3113 completed, throughput=5MB/sec 2021-07-31T02:15:24.817 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ClcdrfcqXLpMF24 off=2996267 len=130889 2021-07-31T02:15:24.819 INFO:tasks.workunit.client.0.smithi045.stdout:op 3114 completed, throughput=5MB/sec 2021-07-31T02:15:24.819 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-PZjW-unHOOnFcRW off=8703689 len=1898395 2021-07-31T02:15:24.830 INFO:tasks.workunit.client.0.smithi045.stdout:op 3115 completed, throughput=5MB/sec 2021-07-31T02:15:24.830 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-92XN7f9HJ3xLGlD off=9729664 len=860050 2021-07-31T02:15:24.832 INFO:tasks.workunit.client.0.smithi045.stdout:op 3116 completed, throughput=5MB/sec 2021-07-31T02:15:25.832 INFO:tasks.workunit.client.0.smithi045.stdout: 607: throughput=4.99MB/sec pending data=0 2021-07-31T02:15:25.833 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-fpzr6QL9vZdnl6X off=1290897 len=1680133 2021-07-31T02:15:25.840 INFO:tasks.workunit.client.0.smithi045.stdout:op 3117 completed, throughput=4.99MB/sec 2021-07-31T02:15:25.840 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ACdt9dNreo30LUZ off=9911205 len=551860 2021-07-31T02:15:25.843 INFO:tasks.workunit.client.0.smithi045.stdout:op 3118 completed, throughput=5MB/sec 2021-07-31T02:15:25.843 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ul5TCBJYMwvUP7t off=13599629 len=322770 2021-07-31T02:15:25.844 INFO:tasks.workunit.client.0.smithi045.stdout:op 3119 completed, throughput=5MB/sec 2021-07-31T02:15:25.844 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-UgpYju6Z7gMJb20 off=2884284 len=700961 2021-07-31T02:15:25.849 INFO:tasks.workunit.client.0.smithi045.stdout:op 3120 completed, throughput=5MB/sec 2021-07-31T02:15:25.849 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-TnXkjbphsmoJfZM off=5663867 len=2031253 2021-07-31T02:15:25.858 INFO:tasks.workunit.client.0.smithi045.stdout:op 3121 completed, throughput=5MB/sec 2021-07-31T02:15:26.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:26 smithi045 conmon[33655]: cluster 2021-07-31T02:15:26.195794+0000 mgr.x 2021-07-31T02:15:26.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:26 smithi045 conmon[33655]: (mgr.24542) 1205 : cluster [DBG] pgmap v1718: 9 pgs: 9 active+clean; 127 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2021-07-31T02:15:26.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:26 smithi045 conmon[39718]: cluster 2021-07-31T02:15:26.195794+0000 mgr.x (mgr. 2021-07-31T02:15:26.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:26 smithi045 conmon[39718]: 24542) 1205 : cluster [DBG] pgmap v1718: 9 pgs: 9 active+clean; 127 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2021-07-31T02:15:26.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:26 smithi154 conmon[34553]: cluster 2021-07-31T02:15:26.195794+0000 mgr.x (mgr.24542) 1205 : cluster [DBG] pgmap v1718: 9 pgs: 9 active+clean; 127 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2021-07-31T02:15:26.858 INFO:tasks.workunit.client.0.smithi045.stdout: 608: throughput=4.99MB/sec pending data=0 2021-07-31T02:15:26.859 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-zmtdKtDngcBmuxf off=3345530 len=898218 2021-07-31T02:15:26.860 INFO:tasks.workunit.client.0.smithi045.stdout:op 3122 completed, throughput=4.99MB/sec 2021-07-31T02:15:26.860 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-RXWtrSgcZ6k8TfS off=12655494 len=1123736 2021-07-31T02:15:26.862 INFO:tasks.workunit.client.0.smithi045.stdout:op 3123 completed, throughput=5MB/sec 2021-07-31T02:15:26.863 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Y0h1aMQM5Bvqzzc off=7273185 len=1774172 2021-07-31T02:15:26.866 INFO:tasks.workunit.client.0.smithi045.stdout:op 3124 completed, throughput=5MB/sec 2021-07-31T02:15:26.866 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-zE61b9igQA5geD1 off=18468655 len=1025436 2021-07-31T02:15:26.873 INFO:tasks.workunit.client.0.smithi045.stdout:op 3125 completed, throughput=5MB/sec 2021-07-31T02:15:26.873 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-UOf1lFI3ROJVjGW off=6295983 len=1892719 2021-07-31T02:15:26.875 INFO:tasks.workunit.client.0.smithi045.stdout:op 3126 completed, throughput=5MB/sec 2021-07-31T02:15:27.876 INFO:tasks.workunit.client.0.smithi045.stdout: 609: throughput=4.99MB/sec pending data=0 2021-07-31T02:15:27.876 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Y2oo4nPEcIkTLeg off=64308 len=1755661 2021-07-31T02:15:27.883 INFO:tasks.workunit.client.0.smithi045.stdout:op 3127 completed, throughput=5MB/sec 2021-07-31T02:15:27.883 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-gGlL2q1j4KdC1TW off=18411139 len=1344232 2021-07-31T02:15:27.885 INFO:tasks.workunit.client.0.smithi045.stdout:op 3128 completed, throughput=5MB/sec 2021-07-31T02:15:27.886 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-nrjZ60iroJy9Ltq off=1864076 len=188383 2021-07-31T02:15:27.889 INFO:tasks.workunit.client.0.smithi045.stdout:op 3129 completed, throughput=5MB/sec 2021-07-31T02:15:27.889 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-3BEtQU8JUMgBuhO off=1031040 len=561653 2021-07-31T02:15:27.890 INFO:tasks.workunit.client.0.smithi045.stdout:op 3130 completed, throughput=5MB/sec 2021-07-31T02:15:28.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:28 smithi154 conmon[34553]: audit 2021-07-31T02:15:28.031988+0000 mon.b (mon.2) 208 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:15:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:28 smithi154 conmon[34553]: audit 2021-07-31T02:15:28.032566+0000 mon.a (mon.0) 1882 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:15:28.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:28 smithi045 conmon[39718]: audit 2021-07-31T02:15:28.031988+0000 mon.b (mon.2) 208 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:15:28.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:28 smithi045 conmon[39718]: audit 2021-07-31T02:15:28.032566+0000 mon.a (mon.0) 1882 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:15:28.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:28 smithi045 conmon[33655]: audit 2021-07-31T02:15:28.031988+0000 mon.b (mon.2) 208 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:15:28.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:28 smithi045 conmon[33655]: 2021-07-31T02:15:28.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:28 smithi045 conmon[33655]: audit 2021-07-31T02:15:28.032566+0000 mon.a (mon.0) 1882 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:15:28.890 INFO:tasks.workunit.client.0.smithi045.stdout: 610: throughput=4.99MB/sec pending data=0 2021-07-31T02:15:28.891 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-odwa6d1_84s-ob8 off=9199230 len=1073805 2021-07-31T02:15:28.896 INFO:tasks.workunit.client.0.smithi045.stdout:op 3131 completed, throughput=4.99MB/sec 2021-07-31T02:15:28.897 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-q7MppOYu_67-AxX off=711932 len=37716 2021-07-31T02:15:28.897 INFO:tasks.workunit.client.0.smithi045.stdout:op 3132 completed, throughput=4.99MB/sec 2021-07-31T02:15:28.897 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-oeeiJw3EpnmAddZ off=3709511 len=81580 2021-07-31T02:15:28.898 INFO:tasks.workunit.client.0.smithi045.stdout:op 3133 completed, throughput=4.99MB/sec 2021-07-31T02:15:28.898 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Tymc5B839J9pLkf off=634464 len=552684 2021-07-31T02:15:28.901 INFO:tasks.workunit.client.0.smithi045.stdout:op 3134 completed, throughput=4.99MB/sec 2021-07-31T02:15:28.902 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-oAWseaTpHXryReu off=8499616 len=536747 2021-07-31T02:15:28.905 INFO:tasks.workunit.client.0.smithi045.stdout:op 3135 completed, throughput=5MB/sec 2021-07-31T02:15:28.906 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-JVpCFkQJvtBMRn0 off=7969538 len=1024195 2021-07-31T02:15:28.908 INFO:tasks.workunit.client.0.smithi045.stdout:op 3136 completed, throughput=5MB/sec 2021-07-31T02:15:28.908 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-YaJaiIcXcwfO3-F off=3671408 len=1666103 2021-07-31T02:15:28.917 INFO:tasks.workunit.client.0.smithi045.stdout:op 3137 completed, throughput=5MB/sec 2021-07-31T02:15:28.917 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-rDkC-GlJkRocyCr off=287240 len=2005009 2021-07-31T02:15:28.921 INFO:tasks.workunit.client.0.smithi045.stdout:op 3138 completed, throughput=5MB/sec 2021-07-31T02:15:29.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:29 smithi154 conmon[34553]: audit 2021-07-31T02:15:28.053398+0000 mon.b (mon.2) 209 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:15:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:29 smithi154 conmon[34553]: audit 2021-07-31T02:15:28.053845+0000 mon.a (mon.0) 1883 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:15:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:29 smithi154 conmon[34553]: cluster 2021-07-31T02:15:28.196162+0000 mgr.x (mgr.24542) 1206 : cluster [DBG] pgmap v1719: 9 pgs: 9 active+clean; 127 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2021-07-31T02:15:29.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:29 smithi045 conmon[33655]: audit 2021-07-31T02:15:28.053398+0000 mon.b (mon. 2021-07-31T02:15:29.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:29 smithi045 conmon[33655]: 2) 209 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:15:29.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:29 smithi045 conmon[33655]: audit 2021-07-31T02:15:28.053845+0000 mon.a (mon.0) 1883 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:15:29.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:29 smithi045 conmon[33655]: cluster 2021-07-31T02:15:29.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:29 smithi045 conmon[33655]: 2021-07-31T02:15:28.196162+0000 mgr.x (mgr.24542) 1206 : cluster [DBG] pgmap v1719: 9 pgs: 9 active+clean; 127 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2021-07-31T02:15:29.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:29 smithi045 conmon[39718]: audit 2021-07-31T02:15:28.053398+0000 mon.b 2021-07-31T02:15:29.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:29 smithi045 conmon[39718]: (mon.2) 209 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:15:29.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:29 smithi045 conmon[39718]: audit 2021 2021-07-31T02:15:29.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:29 smithi045 conmon[39718]: -07-31T02:15:28.053845+0000 mon.a (mon. 2021-07-31T02:15:29.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:29 smithi045 conmon[39718]: 0) 1883 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:15:29.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:29 smithi045 conmon[39718]: cluster 2021-07 2021-07-31T02:15:29.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:29 smithi045 conmon[39718]: -31T02:15:28.196162+0000 2021-07-31T02:15:29.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:29 smithi045 conmon[39718]: mgr.x (mgr.24542) 1206 : cluster [DBG] pgmap v1719: 9 pgs: 9 active+clean; 127 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2021-07-31T02:15:29.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:29 smithi045 conmon[39718]: 2021-07-31T02:15:29.921 INFO:tasks.workunit.client.0.smithi045.stdout: 611: throughput=4.99MB/sec pending data=0 2021-07-31T02:15:29.921 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-3ZkEklhlhKEmuTx off=30719 len=1894234 2021-07-31T02:15:29.930 INFO:tasks.workunit.client.0.smithi045.stdout:op 3139 completed, throughput=5MB/sec 2021-07-31T02:15:29.931 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-d6g2HFoeIj2q0hw off=18091620 len=1995962 2021-07-31T02:15:29.934 INFO:tasks.workunit.client.0.smithi045.stdout:op 3140 completed, throughput=5MB/sec 2021-07-31T02:15:30.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:30 smithi045 conmon[33655]: cluster 2021-07-31T02:15:30.196690 2021-07-31T02:15:30.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:30 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 1207 : cluster [DBG] pgmap v1720: 9 pgs: 9 active+clean; 127 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 2.7 MiB/s rd, 3.7 MiB/s wr, 6 op/s 2021-07-31T02:15:30.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:30 smithi045 conmon[39718]: cluster 2021-07-31T02:15:30.196690+0000 mgr.x 2021-07-31T02:15:30.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:30 smithi045 conmon[39718]: (mgr.24542) 1207 : cluster [DBG] pgmap v1720: 9 pgs: 9 active+clean; 127 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 2.7 MiB/s rd, 3.7 MiB/s wr, 6 op/s 2021-07-31T02:15:30.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[34553]: cluster 2021-07-31T02:15:30.196690+0000 mgr.x (mgr.24542) 1207 : cluster [DBG] pgmap v1720: 9 pgs: 9 active+clean; 127 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 2.7 MiB/s rd, 3.7 MiB/s wr, 6 op/s 2021-07-31T02:15:30.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[34553]: 2021-07-31T02:15:30.934 INFO:tasks.workunit.client.0.smithi045.stdout: 612: throughput=4.99MB/sec pending data=0 2021-07-31T02:15:30.934 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-tgQfDKp2iVa5rhk off=4968231 len=989418 2021-07-31T02:15:30.935 INFO:tasks.workunit.client.0.smithi045.stdout:op 3141 completed, throughput=4.99MB/sec 2021-07-31T02:15:30.935 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Oh57DBDZT2rehB- off=3004965 len=1148766 2021-07-31T02:15:30.941 INFO:tasks.workunit.client.0.smithi045.stdout:op 3142 completed, throughput=5MB/sec 2021-07-31T02:15:30.941 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-OThyPzBoRn2K-Av off=959054 len=1276770 2021-07-31T02:15:30.944 INFO:tasks.workunit.client.0.smithi045.stdout:op 3143 completed, throughput=5MB/sec 2021-07-31T02:15:30.945 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-oBtydBJDalTi9EP off=710006 len=1093248 2021-07-31T02:15:30.950 INFO:tasks.workunit.client.0.smithi045.stdout:op 3144 completed, throughput=5MB/sec 2021-07-31T02:15:30.951 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-QUp1spqzRMLXZnb off=11461152 len=1340869 2021-07-31T02:15:30.953 INFO:tasks.workunit.client.0.smithi045.stdout:op 3145 completed, throughput=5MB/sec 2021-07-31T02:15:31.253 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: debug 2021-07-31T02:15:30.906+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:15:31.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:15:31.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:15:31.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: yield 2021-07-31T02:15:31.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:15:31.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:15:31.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:15:31.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:15:31.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:15:31.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:15:31.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:15:31.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:15:31.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:15:31.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:15:31.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:15:31.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: debug 2021-07-31T02:15:30.918+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:15:31.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:15:31.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:15:31.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: yield 2021-07-31T02:15:31.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:15:31.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:15:31.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:15:31.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:15:31.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:15:31.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:15:31.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:15:31.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:15:31.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:15:31.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:15:31.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:15:31.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: debug 2021-07-31T02:15:30.944+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:15:31.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:15:31.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:15:31.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: yield 2021-07-31T02:15:31.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:15:31.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:15:31.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:15:31.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:15:31.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:15:31.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:15:31.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:15:31.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:15:31.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:15:31.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:15:31.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:15:31.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: debug 2021-07-31T02:15:30.944+0000 7fcb9095b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:15:31.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:15:31.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:15:31.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: yield 2021-07-31T02:15:31.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:15:31.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:15:31.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:15:31.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:15:31.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:15:31.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:15:31.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:15:31.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:15:31.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:15:31.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:15:31.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:15:31.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: 2021-07-31T02:15:31.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:15:31.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: 2021-07-31T02:15:31.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:15:31.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:15:31.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:15:31.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:15:31.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:15:31.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:15:31.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:15:31.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:15:31.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:15:31.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:15:31.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:15:31.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:15:31.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:15:31.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:15:31.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:15:31.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:15:31.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:15:31.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:15:31.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: raise 2021-07-31T02:15:31.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:15:31.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:15:31.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:15:31.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:15:31.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:15:31.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: debug 2021-07-31T02:15:30.953+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:15:31.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:15:31.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:15:31.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: yield 2021-07-31T02:15:31.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:15:31.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:15:31.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:15:31.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:15:31.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:15:31.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:15:31.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:15:31.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:15:31.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:15:31.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:15:31.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:15:31.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: debug 2021-07-31T02:15:30.954+0000 7fcb9115c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:15:31.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:15:31.284 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:15:31.284 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: yield 2021-07-31T02:15:31.284 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:15:31.284 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:15:31.285 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:15:31.285 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:15:31.285 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:15:31.285 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:15:31.286 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:15:31.286 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:15:31.286 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:15:31.287 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:15:31.287 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:15:31.287 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: 2021-07-31T02:15:31.287 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:15:31.288 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: 2021-07-31T02:15:31.288 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:15:31.288 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:15:31.288 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:15:31.288 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:15:31.289 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:15:31.289 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:15:31.289 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:15:31.290 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:15:31.290 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:15:31.291 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:15:31.291 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:15:31.291 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:15:31.291 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:15:31.291 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:15:31.292 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:15:31.292 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:15:31.292 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:15:31.292 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:15:31.293 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: raise 2021-07-31T02:15:31.293 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:15:31.293 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:15:31.293 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:15:31.294 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:15:31.294 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:15:30 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:15:31.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: audit 2021-07-31T02:15: 2021-07-31T02:15:31.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: 30.861672+0000 mon.b (mon.2) 210 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:15:31.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T02:15:31.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: T02:15:30.863077+0000 mon.b (mon.2) 211 : audit [DBG] 2021-07-31T02:15:31.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:15:31.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: audit 2021 2021-07-31T02:15:31.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: -07-31T02:15:30.863947+0000 mon.b 2021-07-31T02:15:31.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: (mon.2) 212 : audit [INF] 2021-07-31T02:15:31.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:15:31.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:15:31.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: 2021-07-31T02:15:30.864667+0000 2021-07-31T02:15:31.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: mgr.x (mgr.24542) 1208 : cephadm [WRN] 2021-07-31T02:15:31.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:15:31.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:15:31.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: 2021-07-31T02:15:30.907642+0000 2021-07-31T02:15:31.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: mgr.x (mgr.24542) 1209 : cephadm [ERR] 2021-07-31T02:15:31.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: 2021-07-31T02:15:31.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:15:31.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:15:31.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: yield 2021-07-31T02:15:31.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:15:31.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:15:31.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:15:31.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:15:31.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:15:31.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:15:31.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:15:31.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:15:31.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:15:31.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: task.result() 2021-07-31T02:15:31.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:15:31.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:15:30.908146+0000 mgr.x (mgr.24542) 1210 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:15:31.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:15:30 2021-07-31T02:15:31.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: .919186+0000 mgr.x (mgr.24542) 1211 : cephadm 2021-07-31T02:15:31.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: [ERR] 2021-07-31T02:15:31.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:15:31.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:15:31.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: yield 2021-07-31T02:15:31.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:15:31.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:15:31.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:15:31.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: audit 2021-07-31T02:15:30.861672+0000 mon.b (mon.2) 210 2021-07-31T02:15:31.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:15:31.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: audit 2021-07-31T02:15:30.863077+0000 mon.b (mon.2) 211 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:15:31.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: audit 2021-07-31T02:15:31.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: 2021-07-31T02:15 2021-07-31T02:15:31.539 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: :30.863947+0000 mon.b (mon. 2021-07-31T02:15:31.539 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: 2) 212 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:15:31.539 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:15:30.864667+0000 mgr.x (mgr.24542) 1208 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:15:31.540 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:15:30.907642+0000 mgr.x (mgr. 2021-07-31T02:15:31.540 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: 24542) 1209 : cephadm [ERR] 2021-07-31T02:15:31.540 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:15:31.540 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:15:31.540 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: yield 2021-07-31T02:15:31.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:15:31.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:15:31.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:15:31.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:15:31.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:15:31.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:15:31.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:15:31.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:15:31.543 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:15:31.543 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: task.result() 2021-07-31T02:15:31.543 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:15:31.543 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:15:30.908146+0000 mgr.x (mgr.24542) 1210 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:15:31.544 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: cephadm 2021- 2021-07-31T02:15:31.544 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: 07-31T02:15:30.919186+0000 2021-07-31T02:15:31.544 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: mgr.x (mgr.24542) 1211 : cephadm 2021-07-31T02:15:31.544 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: [ERR] 2021-07-31T02:15:31.544 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:15:31.545 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:15:31.545 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: yield 2021-07-31T02:15:31.545 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:15:31.545 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:15:31.546 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:15:31.546 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:15:31.546 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:15:31.546 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:15:31.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:15:31.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:15:31.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:15:31.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:15:31.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:15:31.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:15:31.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: task.result() 2021-07-31T02:15:31.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:15:31.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:15:30.920085+0000 mgr.x (mgr.24542) 1212 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:15:31.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:15:30. 2021-07-31T02:15:31.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: 945226+0000 mgr.x (mgr.24542) 1213 : cephadm 2021-07-31T02:15:31.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: [ERR] 2021-07-31T02:15:31.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:15:31.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:15:31.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: yield 2021-07-31T02:15:31.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:15:31.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:15:31.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:15:31.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:15:31.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:15:31.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:15:31.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:15:31.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:15:31.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:15:31.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: task.result() 2021-07-31T02:15:31.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:15:31.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:15:30.945854+0000 mgr.x (mgr.24542) 1214 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:15:31.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:15:31.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:15:31.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: yield 2021-07-31T02:15:31.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:15:31.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:15:31.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:15:31.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:15:31.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:15:31.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:15:31.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:15:31.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:15:31.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:15:31.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: task.result() 2021-07-31T02:15:31.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:15:31.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:15:31.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:15:31.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:15:31.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: task.result() 2021-07-31T02:15:31.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:15:31.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:15:30.920085+0000 mgr.x (mgr.24542) 1212 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:15:31.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:15:31.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: 2021-07-31T02:15:30.945226+0000 mgr.x ( 2021-07-31T02:15:31.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: mgr.24542) 1213 : cephadm [ERR] 2021-07-31T02:15:31.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:15:31.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:15:31.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: yield 2021-07-31T02:15:31.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:15:31.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:15:31.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:15:31.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:15:31.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:15:31.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:15:31.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:15:31.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:15:31.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:15:31.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: task.result() 2021-07-31T02:15:31.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:15:31.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: 2021-07-31T02:15:31.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:15:30.945854+0000 mgr.x (mgr.24542) 1214 2021-07-31T02:15:31.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:15:31.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:15:31.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:15:31.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: yield 2021-07-31T02:15:31.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:15:31.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:15:31.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:15:31.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:15:31.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:15:31.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:15:31.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:15:31.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:15:31.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:15:31.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: task.result() 2021-07-31T02:15:31.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:15:31.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: 2021-07-31T02:15:31.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:15:31.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: 2021-07-31T02:15:31.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: 2021-07-31T02:15:31.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:15:31.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: 2021-07-31T02:15:31.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:15:31.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:15:31.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:15:31.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:15:31.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:15:31.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:15:31.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:15:31.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:15:31.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:15:31.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:15:31.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:15:31.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:15:31.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:15:31.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:15:31.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:15:31.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:15:31.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:15:31.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:15:31.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: raise 2021-07-31T02:15:31.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:15:31.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:15:31.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:15:31.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:15:31.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:15:31.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: 2021-07-31T02:15:31.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:15:30.954711+0000 mgr.x (mgr.24542) 1215 : cephadm [ERR] 2021-07-31T02:15:31.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:15:31.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:15:31.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: yield 2021-07-31T02:15:31.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:15:31.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:15:31.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:15:31.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:15:31.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:15:31.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:15:31.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:15:31.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:15:31.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:15:31.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:15:31.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:15:31.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:15:31.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:15:31.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:15:31.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:15:31.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:15:31.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:15:31.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:15:31.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:15:31.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:15:31.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:15:31.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:15:31.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:15:31.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:15:31.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:15:31.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:15:31.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:15:31.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: raise 2021-07-31T02:15:31.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:15:31.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:15:31.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:15:31.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:15:31.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:15:31.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:15:30.954711+0000 mgr.x (mgr.24542) 1215 : cephadm [ERR] 2021-07-31T02:15:31.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:15:31.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:15:31.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: yield 2021-07-31T02:15:31.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:15:31.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:15:31.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:15:31.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:15:31.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:15:31.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:15:31.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:15:31.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:15:31.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:15:31.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: task.result() 2021-07-31T02:15:31.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:15:31.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:15:30.955318+0000 mgr.x (mgr 2021-07-31T02:15:31.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: .24542) 1216 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:15:31.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:15:31.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:15:31.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: yield 2021-07-31T02:15:31.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:15:31.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:15:31.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:15:31.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:15:31.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:15:31.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:15:31.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:15:31.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:15:31.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:15:31.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: task.result() 2021-07-31T02:15:31.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:15:31.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: 2021-07-31T02:15:31.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:15:31.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: 2021-07-31T02:15:31.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:15:31.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:15:31.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:15:31.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:15:31.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:15:31.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:15:31.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:15:31.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:15:31.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:15:31.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:15:31.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:15:31.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:15:31.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:15:31.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:15:31.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:15:31.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:15:31.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:15:31.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:15:31.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: raise 2021-07-31T02:15:31.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:15:31.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:15:31.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:15:31.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:15:31.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:15:31.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: task.result() 2021-07-31T02:15:31.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:15:31.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:15:30.955318+0000 mgr.x (mgr.24542) 1216 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:15:31.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:15:31.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:15:31.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: yield 2021-07-31T02:15:31.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:15:31.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:15:31.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:15:31.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:15:31.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:15:31.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:15:31.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:15:31.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:15:31.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:15:31.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: task.result() 2021-07-31T02:15:31.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:15:31.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: 2021-07-31T02:15:31.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:15:31.603 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: 2021-07-31T02:15:31.603 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:15:31.603 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:15:31.603 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:15:31.603 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:15:31.604 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:15:31.604 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:15:31.604 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:15:31.604 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:15:31.605 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:15:31.605 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:15:31.605 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:15:31.605 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:15:31.605 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:15:31.606 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:15:31.606 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:15:31.606 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:15:31.606 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:15:31.607 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:15:31.607 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: raise 2021-07-31T02:15:31.607 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:15:31.607 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:15:31.607 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:15:31.608 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:15:31.608 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:31 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:15:31.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: audit 2021-07-31T02:15:30.861672+0000 mon.b (mon.2) 210 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:15:31.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: audit 2021-07-31T02:15:30.863077+0000 mon.b (mon.2) 211 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:15:31.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: audit 2021-07-31T02:15:31.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: 2021-07-31T02:15:30.863947+0000 mon.b (mon.2) 212 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:15:31.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: cephadm 2021-07-31 2021-07-31T02:15:31.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: T02:15:30.864667+0000 mgr.x (mgr.24542) 1208 : cephadm [WRN] 2021-07-31T02:15:31.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:15:31.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:15:30.907642+0000 mgr.x (mgr.24542) 1209 : cephadm [ERR] 2021-07-31T02:15:31.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:15:31.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:15:31.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: yield 2021-07-31T02:15:31.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:15:31.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:15:31.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:15:31.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:15:31.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:15:31.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:15:31.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:15:31.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:15:31.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:15:31.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: task.result() 2021-07-31T02:15:31.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:15:31.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:15:30.908146+0000 mgr.x (mgr.24542) 1210 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:15:31.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:15:30.919186+0000 mgr.x (mgr.24542) 1211 : cephadm [ERR] 2021-07-31T02:15:31.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:15:31.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:15:31.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: yield 2021-07-31T02:15:31.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:15:31.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:15:31.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:15:31.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:15:31.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:15:31.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:15:31.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:15:31.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:15:31.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:15:31.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: task.result() 2021-07-31T02:15:31.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:15:31.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:15:30 2021-07-31T02:15:31.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: .920085+0000 mgr.x (mgr.24542) 1212 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:15:31.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:15:30.945226+0000 mgr.x (mgr.24542) 1213 : cephadm [ERR] 2021-07-31T02:15:31.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:15:31.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:15:31.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: yield 2021-07-31T02:15:31.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:15:31.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:15:31.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:15:31.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:15:31.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:15:31.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:15:31.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:15:31.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:15:31.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:15:31.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: task.result() 2021-07-31T02:15:31.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:15:31.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:15:30.945854+0000 mgr.x (mgr.24542) 1214 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:15:31.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:15:31.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:15:31.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: yield 2021-07-31T02:15:31.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:15:31.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:15:31.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:15:31.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:15:31.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:15:31.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:15:31.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:15:31.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:15:31.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:15:31.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: task.result() 2021-07-31T02:15:31.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:15:31.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: 2021-07-31T02:15:31.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:15:31.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: 2021-07-31T02:15:31.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:15:31.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:15:31.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:15:31.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:15:31.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:15:31.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:15:31.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:15:31.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:15:31.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:15:31.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:15:31.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:15:31.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:15:31.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:15:31.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:15:31.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:15:31.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:15:31.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:15:31.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:15:31.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: raise 2021-07-31T02:15:31.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:15:31.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:15:31.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:15:31.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:15:31.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:15:31.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:15:30.954711+0000 mgr.x (mgr.24542) 1215 : cephadm [ERR] 2021-07-31T02:15:31.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:15:31.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:15:31.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: yield 2021-07-31T02:15:31.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:15:31.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:15:31.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:15:31.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:15:31.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:15:31.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:15:31.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:15:31.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:15:31.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:15:31.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: task.result() 2021-07-31T02:15:31.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:15:31.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:15:30.955318+0000 mgr.x (mgr.24542) 1216 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:15:31.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:15:31.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:15:31.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: yield 2021-07-31T02:15:31.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:15:31.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:15:31.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:15:31.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:15:31.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:15:31.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:15:31.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:15:31.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:15:31.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:15:31.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: task.result() 2021-07-31T02:15:31.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:15:31.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: 2021-07-31T02:15:31.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:15:31.782 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: 2021-07-31T02:15:31.782 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:15:31.782 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:15:31.782 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:15:31.783 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:15:31.783 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:15:31.783 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:15:31.783 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:15:31.783 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:15:31.784 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:15:31.784 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:15:31.784 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:15:31.784 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:15:31.784 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:15:31.785 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:15:31.785 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:15:31.785 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:15:31.785 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:15:31.786 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:15:31.786 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: raise 2021-07-31T02:15:31.786 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:15:31.786 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:15:31.787 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:15:31.787 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:15:31.787 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:31 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:15:31.953 INFO:tasks.workunit.client.0.smithi045.stdout: 613: throughput=4.99MB/sec pending data=0 2021-07-31T02:15:31.953 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-w75qiHUItf0Mpaw off=1348202 len=1129002 2021-07-31T02:15:31.959 INFO:tasks.workunit.client.0.smithi045.stdout:op 3146 completed, throughput=4.99MB/sec 2021-07-31T02:15:31.960 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-VctPA5CU15_qpTi off=11775309 len=1575558 2021-07-31T02:15:31.961 INFO:tasks.workunit.client.0.smithi045.stdout:op 3147 completed, throughput=5MB/sec 2021-07-31T02:15:31.962 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-OWgimuILaRdeBte off=18953528 len=835641 2021-07-31T02:15:31.964 INFO:tasks.workunit.client.0.smithi045.stdout:op 3148 completed, throughput=5MB/sec 2021-07-31T02:15:31.964 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-lnKuHMAI_aYVzD3 off=754905 len=2024190 2021-07-31T02:15:31.973 INFO:tasks.workunit.client.0.smithi045.stdout:op 3149 completed, throughput=5MB/sec 2021-07-31T02:15:32.973 INFO:tasks.workunit.client.0.smithi045.stdout: 614: throughput=4.99MB/sec pending data=0 2021-07-31T02:15:32.974 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-W4XG1uSM2oXsFMh off=2846998 len=2086858 2021-07-31T02:15:32.982 INFO:tasks.workunit.client.0.smithi045.stdout:op 3150 completed, throughput=5MB/sec 2021-07-31T02:15:32.982 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-bYK1xc6srLr248c off=5472704 len=1540861 2021-07-31T02:15:32.989 INFO:tasks.workunit.client.0.smithi045.stdout:op 3151 completed, throughput=5MB/sec 2021-07-31T02:15:32.989 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-V6_ya9c57R5rOhT off=16655834 len=1279230 2021-07-31T02:15:32.996 INFO:tasks.workunit.client.0.smithi045.stdout:op 3152 completed, throughput=5MB/sec 2021-07-31T02:15:33.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:33 smithi154 conmon[34553]: cluster 2021-07-31T02:15:32.197221+0000 mgr.x (mgr.24542) 1217 : cluster [DBG] pgmap v1721: 9 pgs: 9 active+clean; 127 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2021-07-31T02:15:33.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:33 smithi154 conmon[34553]: 2021-07-31T02:15:33.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:33 smithi045 conmon[33655]: cluster 2021-07-31T02:15:32.197221+0000 mgr.x (mgr.24542) 1217 : cluster 2021-07-31T02:15:33.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:33 smithi045 conmon[33655]: [DBG] pgmap v1721: 9 pgs: 9 active+clean; 127 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2021-07-31T02:15:33.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:33 smithi045 conmon[39718]: cluster 2021-07-31T02:15: 2021-07-31T02:15:33.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:33 smithi045 conmon[39718]: 32.197221+0000 mgr.x (mgr.24542) 1217 : cluster [DBG] pgmap v1721: 9 pgs: 9 active+clean; 127 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2021-07-31T02:15:33.996 INFO:tasks.workunit.client.0.smithi045.stdout: 615: throughput=4.99MB/sec pending data=0 2021-07-31T02:15:33.997 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-gDua79tfYXMc2kf off=15821559 len=1027041 2021-07-31T02:15:33.999 INFO:tasks.workunit.client.0.smithi045.stdout:op 3153 completed, throughput=4.99MB/sec 2021-07-31T02:15:33.999 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-RvyKH5gIYU0oaH2 off=735855 len=527948 2021-07-31T02:15:34.000 INFO:tasks.workunit.client.0.smithi045.stdout:op 3154 completed, throughput=5MB/sec 2021-07-31T02:15:34.000 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-cGoQWShKiTWcKld off=6741920 len=1243728 2021-07-31T02:15:34.007 INFO:tasks.workunit.client.0.smithi045.stdout:op 3155 completed, throughput=5MB/sec 2021-07-31T02:15:34.007 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-HefZVh47PbmVY7w off=644318 len=1938016 2021-07-31T02:15:34.015 INFO:tasks.workunit.client.0.smithi045.stdout:op 3156 completed, throughput=5MB/sec 2021-07-31T02:15:34.016 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-BgE9HJTs48HaQq_ off=13299927 len=1289306 2021-07-31T02:15:34.017 INFO:tasks.workunit.client.0.smithi045.stdout:op 3157 completed, throughput=5MB/sec 2021-07-31T02:15:34.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:15:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:15:33.557Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:15:34.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:15:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:15:33.557Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:15:34.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:15:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:15:33.557Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:15:34.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:34 smithi045 conmon[39718]: cluster 2021-07-31T02:15:34.197879+0000 mgr.x 2021-07-31T02:15:34.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:34 smithi045 conmon[39718]: (mgr.24542) 1218 : cluster [DBG] pgmap v1722: 9 pgs: 9 active+clean; 127 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 2.6 MiB/s rd, 3.3 MiB/s wr, 5 op/s 2021-07-31T02:15:34.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:34 smithi045 conmon[33655]: cluster 2021-07-31T02:15:34.197879+0000 mgr.x (mgr 2021-07-31T02:15:34.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:34 smithi045 conmon[33655]: .24542) 1218 : cluster [DBG] pgmap v1722: 9 pgs: 9 active+clean; 127 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 2.6 MiB/s rd, 3.3 MiB/s wr, 5 op/s 2021-07-31T02:15:34.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:34 smithi154 conmon[34553]: cluster 2021-07-31T02:15:34.197879+0000 mgr.x (mgr.24542) 1218 : cluster [DBG] pgmap v1722: 9 pgs: 9 active+clean; 127 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 2.6 MiB/s rd, 3.3 MiB/s wr, 5 op/s 2021-07-31T02:15:34.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:34 smithi154 conmon[34553]: 2021-07-31T02:15:35.017 INFO:tasks.workunit.client.0.smithi045.stdout: 616: throughput=4.99MB/sec pending data=0 2021-07-31T02:15:35.018 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-mcGRAJbW_7ifZ2v off=11940675 len=1185287 2021-07-31T02:15:35.020 INFO:tasks.workunit.client.0.smithi045.stdout:op 3158 completed, throughput=5MB/sec 2021-07-31T02:15:35.021 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-A-nVmB9nNR5ry22 off=571081 len=855395 2021-07-31T02:15:35.026 INFO:tasks.workunit.client.0.smithi045.stdout:op 3159 completed, throughput=5MB/sec 2021-07-31T02:15:35.027 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-gTDM3NnGuwHT4y- off=6954372 len=1021908 2021-07-31T02:15:35.028 INFO:tasks.workunit.client.0.smithi045.stdout:op 3160 completed, throughput=5MB/sec 2021-07-31T02:15:35.028 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ojM3n2kqEGT99nf off=4057620 len=92920 2021-07-31T02:15:35.029 INFO:tasks.workunit.client.0.smithi045.stdout:op 3161 completed, throughput=5MB/sec 2021-07-31T02:15:35.030 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-V5eVYOxVkRCLrA6 off=18185828 len=524074 2021-07-31T02:15:35.031 INFO:tasks.workunit.client.0.smithi045.stdout:op 3162 completed, throughput=5MB/sec 2021-07-31T02:15:35.031 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-RMbiocUY0u4xT8Y off=10244726 len=1012158 2021-07-31T02:15:35.032 INFO:tasks.workunit.client.0.smithi045.stdout:op 3163 completed, throughput=5MB/sec 2021-07-31T02:15:36.032 INFO:tasks.workunit.client.0.smithi045.stdout: 617: throughput=4.99MB/sec pending data=0 2021-07-31T02:15:36.033 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-yPRnGUOIxxWd9yY off=11330954 len=1481742 2021-07-31T02:15:36.040 INFO:tasks.workunit.client.0.smithi045.stdout:op 3164 completed, throughput=5MB/sec 2021-07-31T02:15:36.040 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-So2pbzlMYTcsdKp off=6598921 len=1471332 2021-07-31T02:15:36.047 INFO:tasks.workunit.client.0.smithi045.stdout:op 3165 completed, throughput=5MB/sec 2021-07-31T02:15:36.048 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-pcRyDpx6TzDcgpk off=483785 len=491691 2021-07-31T02:15:36.051 INFO:tasks.workunit.client.0.smithi045.stdout:op 3166 completed, throughput=5MB/sec 2021-07-31T02:15:36.051 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-28wbep0YWF62M3c off=912008 len=22260 2021-07-31T02:15:36.054 INFO:tasks.workunit.client.0.smithi045.stdout:op 3167 completed, throughput=5MB/sec 2021-07-31T02:15:36.055 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-f2hxmiLFd3hxXCi off=1160584 len=1586342 2021-07-31T02:15:36.063 INFO:tasks.workunit.client.0.smithi045.stdout:op 3168 completed, throughput=5MB/sec 2021-07-31T02:15:36.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:36 smithi045 conmon[39718]: cluster 2021-07-31T02:15:36.198351+0000 mgr.x ( 2021-07-31T02:15:36.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:36 smithi045 conmon[39718]: mgr.24542) 1219 : cluster [DBG] pgmap v1723: 9 pgs: 9 active+clean; 127 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 1.6 MiB/s rd, 2.2 MiB/s wr, 3 op/s 2021-07-31T02:15:36.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:36 smithi045 conmon[33655]: cluster 2021-07-31T02:15:36.198351+0000 mgr.x (mgr.24542) 1219 : cluster [DBG] pgmap v1723: 9 pgs: 9 active+clean; 127 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 1.6 MiB/s rd, 2.2 MiB/s wr, 3 op/s 2021-07-31T02:15:36.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:36 smithi154 conmon[34553]: cluster 2021-07-31T02:15:36.198351+0000 mgr.x (mgr.24542) 1219 : cluster [DBG] pgmap v1723: 9 pgs: 9 active+clean; 127 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 1.6 MiB/s rd, 2.2 MiB/s wr, 3 op/s 2021-07-31T02:15:36.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:36 smithi154 conmon[34553]: 2021-07-31T02:15:37.059 INFO:tasks.workunit.client.0.smithi045.stdout: 618: throughput=4.99MB/sec pending data=0 2021-07-31T02:15:37.060 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-OTVXK_cb9TJIFWt off=15255194 len=2082549 2021-07-31T02:15:37.062 INFO:tasks.workunit.client.0.smithi045.stdout:op 3169 completed, throughput=5MB/sec 2021-07-31T02:15:37.062 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-BRJO8cntZivL3fJ off=2133133 len=1902912 2021-07-31T02:15:37.065 INFO:tasks.workunit.client.0.smithi045.stdout:op 3170 completed, throughput=5MB/sec 2021-07-31T02:15:37.066 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-noHmvD6qT9p4_l1 off=11817284 len=481871 2021-07-31T02:15:37.070 INFO:tasks.workunit.client.0.smithi045.stdout:op 3171 completed, throughput=5MB/sec 2021-07-31T02:15:37.071 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-7xqlvf1EOGRIR_V off=4136402 len=1130841 2021-07-31T02:15:37.072 INFO:tasks.workunit.client.0.smithi045.stdout:op 3172 completed, throughput=5MB/sec 2021-07-31T02:15:38.072 INFO:tasks.workunit.client.0.smithi045.stdout: 619: throughput=4.99MB/sec pending data=0 2021-07-31T02:15:38.073 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-XbEdl1X8MpDyLXB off=5248869 len=197461 2021-07-31T02:15:38.073 INFO:tasks.workunit.client.0.smithi045.stdout:op 3173 completed, throughput=4.99MB/sec 2021-07-31T02:15:38.074 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-b_lBYavSv754Uwt off=13265734 len=466907 2021-07-31T02:15:38.074 INFO:tasks.workunit.client.0.smithi045.stdout:op 3174 completed, throughput=4.99MB/sec 2021-07-31T02:15:38.075 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-W2oQvAUTGNfGq2c off=12505140 len=1777711 2021-07-31T02:15:38.084 INFO:tasks.workunit.client.0.smithi045.stdout:op 3175 completed, throughput=5MB/sec 2021-07-31T02:15:38.084 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-4oaH9JrQjW-qZMy off=15158345 len=5096 2021-07-31T02:15:38.086 INFO:tasks.workunit.client.0.smithi045.stdout:op 3176 completed, throughput=5MB/sec 2021-07-31T02:15:38.086 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-cH--yxRfCIIPcue off=236215 len=310478 2021-07-31T02:15:38.088 INFO:tasks.workunit.client.0.smithi045.stdout:op 3177 completed, throughput=5MB/sec 2021-07-31T02:15:38.089 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-63McbXMswcmwKGX off=20870602 len=1695754 2021-07-31T02:15:38.089 INFO:tasks.workunit.client.0.smithi045.stdout:op 3178 completed, throughput=5MB/sec 2021-07-31T02:15:38.090 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-owW7eK5EnaBLlAv off=4155348 len=173617 2021-07-31T02:15:38.092 INFO:tasks.workunit.client.0.smithi045.stdout:op 3179 completed, throughput=5MB/sec 2021-07-31T02:15:38.093 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-lVAcFkQzMwUexyB off=2433241 len=1515777 2021-07-31T02:15:38.100 INFO:tasks.workunit.client.0.smithi045.stdout:op 3180 completed, throughput=5MB/sec 2021-07-31T02:15:38.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:38 smithi045 conmon[33655]: cluster 2021-07-31T02:15:38.198856+0000 mgr.x (mgr.24542) 1220 : cluster 2021-07-31T02:15:38.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:38 smithi045 conmon[33655]: [DBG] pgmap v1724: 9 pgs: 9 active+clean; 127 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 1.6 MiB/s rd, 2.2 MiB/s wr, 3 op/s 2021-07-31T02:15:38.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:38 smithi045 conmon[39718]: cluster 2021-07-31T02:15:38.198856+0000 2021-07-31T02:15:38.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:38 smithi045 conmon[39718]: mgr.x (mgr.24542) 1220 : cluster [DBG] pgmap v1724: 9 pgs: 9 active+clean; 127 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 1.6 MiB/s rd, 2.2 MiB/s wr, 3 op/s 2021-07-31T02:15:38.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:38 smithi154 conmon[34553]: cluster 2021-07-31T02:15:38.198856+0000 mgr.x (mgr.24542) 1220 : cluster [DBG] pgmap v1724: 9 pgs: 9 active+clean; 127 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 1.6 MiB/s rd, 2.2 MiB/s wr, 3 op/s 2021-07-31T02:15:39.101 INFO:tasks.workunit.client.0.smithi045.stdout: 620: throughput=4.99MB/sec pending data=0 2021-07-31T02:15:39.101 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-lUzPJTUuj1Zmyrd off=4535893 len=661999 2021-07-31T02:15:39.103 INFO:tasks.workunit.client.0.smithi045.stdout:op 3181 completed, throughput=4.99MB/sec 2021-07-31T02:15:39.103 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-FlliPVpAEHE81Zs off=5234925 len=493938 2021-07-31T02:15:39.104 INFO:tasks.workunit.client.0.smithi045.stdout:op 3182 completed, throughput=5MB/sec 2021-07-31T02:15:39.104 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-EDZocJC3pRABo2D off=3077845 len=382362 2021-07-31T02:15:39.108 INFO:tasks.workunit.client.0.smithi045.stdout:op 3183 completed, throughput=5MB/sec 2021-07-31T02:15:39.108 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-euG4kd344qZsdBV off=3224857 len=1079503 2021-07-31T02:15:39.117 INFO:tasks.workunit.client.0.smithi045.stdout:op 3184 completed, throughput=5MB/sec 2021-07-31T02:15:39.117 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-UIqzvyqJMYRB1Ls off=15535480 len=350377 2021-07-31T02:15:39.120 INFO:tasks.workunit.client.0.smithi045.stdout:op 3185 completed, throughput=5MB/sec 2021-07-31T02:15:39.120 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-wiZuqcsWfG7pwyb off=5562999 len=991847 2021-07-31T02:15:39.122 INFO:tasks.workunit.client.0.smithi045.stdout:op 3186 completed, throughput=5MB/sec 2021-07-31T02:15:39.122 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-mtD20Ustw6sxJWQ off=18081025 len=609041 2021-07-31T02:15:39.127 INFO:tasks.workunit.client.0.smithi045.stdout:op 3187 completed, throughput=5MB/sec 2021-07-31T02:15:40.127 INFO:tasks.workunit.client.0.smithi045.stdout: 621: throughput=4.99MB/sec pending data=0 2021-07-31T02:15:40.127 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-GRp-j6hKSIfd2rB off=0 len=1059024 2021-07-31T02:15:40.133 INFO:tasks.workunit.client.0.smithi045.stdout:op 3188 completed, throughput=4.99MB/sec 2021-07-31T02:15:40.133 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-g40SQJUgQO8o-uT off=4726248 len=1883020 2021-07-31T02:15:40.144 INFO:tasks.workunit.client.0.smithi045.stdout:op 3189 completed, throughput=5MB/sec 2021-07-31T02:15:40.145 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-2NyTC9I54fAyFoj off=1445028 len=1132660 2021-07-31T02:15:40.150 INFO:tasks.workunit.client.0.smithi045.stdout:op 3190 completed, throughput=5MB/sec 2021-07-31T02:15:40.151 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-wW03wooWI9yMPOX off=11857940 len=2049473 2021-07-31T02:15:40.158 INFO:tasks.workunit.client.0.smithi045.stdout:op 3191 completed, throughput=5MB/sec 2021-07-31T02:15:40.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:40 smithi045 conmon[39718]: cluster 2021-07-31T02:15:40.199331+0000 mgr.x ( 2021-07-31T02:15:40.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:40 smithi045 conmon[39718]: mgr.24542) 1221 : cluster [DBG] pgmap v1725: 9 pgs: 9 active+clean; 127 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 2.8 MiB/s rd, 3.3 MiB/s wr, 5 op/s 2021-07-31T02:15:40.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:40 smithi045 conmon[33655]: cluster 2021-07-31T02:15:40.199331+0000 mgr.x (mgr.24542) 1221 : cluster [DBG] pgmap v1725: 9 pgs: 9 active+clean; 127 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 2.8 MiB/s rd, 3.3 MiB/s wr, 5 op/s 2021-07-31T02:15:40.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:40 smithi154 conmon[34553]: cluster 2021-07-31T02:15:40.199331+0000 mgr.x (mgr.24542) 1221 : cluster [DBG] pgmap v1725: 9 pgs: 9 active+clean; 127 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 2.8 MiB/s rd, 3.3 MiB/s wr, 5 op/s 2021-07-31T02:15:41.158 INFO:tasks.workunit.client.0.smithi045.stdout: 622: throughput=4.99MB/sec pending data=0 2021-07-31T02:15:41.158 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-mZRsRy1IMRq50sE off=9333072 len=1277728 2021-07-31T02:15:41.165 INFO:tasks.workunit.client.0.smithi045.stdout:op 3192 completed, throughput=5MB/sec 2021-07-31T02:15:41.165 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-DJ_Sd8Vhd7DV_6z off=2213106 len=583797 2021-07-31T02:15:41.166 INFO:tasks.workunit.client.0.smithi045.stdout:op 3193 completed, throughput=5MB/sec 2021-07-31T02:15:41.166 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-QWFDl0oJiySXr1W off=13622 len=707519 2021-07-31T02:15:41.167 INFO:tasks.workunit.client.0.smithi045.stdout:op 3194 completed, throughput=5MB/sec 2021-07-31T02:15:41.168 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ApBjkbmY2J5mWgK off=13231571 len=1433922 2021-07-31T02:15:41.175 INFO:tasks.workunit.client.0.smithi045.stdout:op 3195 completed, throughput=5MB/sec 2021-07-31T02:15:41.175 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-tHB930KYX4D1tGE off=2112079 len=1997869 2021-07-31T02:15:41.185 INFO:tasks.workunit.client.0.smithi045.stdout:op 3196 completed, throughput=5MB/sec 2021-07-31T02:15:42.185 INFO:tasks.workunit.client.0.smithi045.stdout: 623: throughput=4.99MB/sec pending data=0 2021-07-31T02:15:42.186 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-JH8dbAh5jOPYAOf off=16368 len=1490754 2021-07-31T02:15:42.188 INFO:tasks.workunit.client.0.smithi045.stdout:op 3197 completed, throughput=5MB/sec 2021-07-31T02:15:42.188 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-by8kCB2UUXXF1J6 off=3647641 len=1936014 2021-07-31T02:15:42.199 INFO:tasks.workunit.client.0.smithi045.stdout:op 3198 completed, throughput=5MB/sec 2021-07-31T02:15:42.200 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-_ntCJm_AUl74ZLr off=411367 len=809843 2021-07-31T02:15:42.204 INFO:tasks.workunit.client.0.smithi045.stdout:op 3199 completed, throughput=5MB/sec 2021-07-31T02:15:43.204 INFO:tasks.workunit.client.0.smithi045.stdout: 624: throughput=4.99MB/sec pending data=0 2021-07-31T02:15:43.205 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-cs17fQ7bVQArZ9c off=15771646 len=1060638 2021-07-31T02:15:43.208 INFO:tasks.workunit.client.0.smithi045.stdout:op 3200 completed, throughput=4.99MB/sec 2021-07-31T02:15:43.208 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-nvRS0dHpIARld8p off=4020491 len=855860 2021-07-31T02:15:43.215 INFO:tasks.workunit.client.0.smithi045.stdout:op 3201 completed, throughput=5MB/sec 2021-07-31T02:15:43.215 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-_B1t4AwZYZCNsQO off=14076488 len=23849 2021-07-31T02:15:43.216 INFO:tasks.workunit.client.0.smithi045.stdout:op 3202 completed, throughput=5MB/sec 2021-07-31T02:15:43.216 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-FCNoxEE1tRq_ZoA off=4656582 len=588398 2021-07-31T02:15:43.219 INFO:tasks.workunit.client.0.smithi045.stdout:op 3203 completed, throughput=5MB/sec 2021-07-31T02:15:43.219 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-uJxv87NELZyapOe off=7777672 len=531257 2021-07-31T02:15:43.223 INFO:tasks.workunit.client.0.smithi045.stdout:op 3204 completed, throughput=5MB/sec 2021-07-31T02:15:43.223 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-MjTA3p_klDjmfYn off=613450 len=1944532 2021-07-31T02:15:43.227 INFO:tasks.workunit.client.0.smithi045.stdout:op 3205 completed, throughput=5MB/sec 2021-07-31T02:15:43.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:43 smithi154 conmon[34553]: cluster 2021-07-31T02:15:42.199712+0000 mgr.x (mgr.24542) 1222 : cluster 2021-07-31T02:15:43.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:43 smithi154 conmon[34553]: [DBG] pgmap v1726: 9 pgs: 9 active+clean; 127 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.0 MiB/s wr, 3 op/s 2021-07-31T02:15:43.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:43 smithi045 conmon[39718]: cluster 2021-07-31T02:15:42.199712+0000 mgr.x ( 2021-07-31T02:15:43.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:43 smithi045 conmon[39718]: mgr.24542) 1222 : cluster [DBG] pgmap v1726: 9 pgs: 9 active+clean; 127 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.0 MiB/s wr, 3 op/s 2021-07-31T02:15:43.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:43 smithi045 conmon[33655]: cluster 2021-07-31T02:15:42.199712+0000 mgr.x (mgr.24542) 1222 : cluster [DBG] pgmap v1726: 9 pgs: 9 active+clean; 127 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.0 MiB/s wr, 3 op/s 2021-07-31T02:15:44.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:15:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:15:43.557Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:15:44.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:15:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:15:43.557Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:15:44.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:15:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:15:43.557Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:15:44.228 INFO:tasks.workunit.client.0.smithi045.stdout: 625: throughput=4.99MB/sec pending data=0 2021-07-31T02:15:44.228 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-g8W7cJwNg7PK998 off=17805778 len=703250 2021-07-31T02:15:44.233 INFO:tasks.workunit.client.0.smithi045.stdout:op 3206 completed, throughput=4.99MB/sec 2021-07-31T02:15:44.234 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-z0GX8T2ePFLKwyK off=2820504 len=1813904 2021-07-31T02:15:44.240 INFO:tasks.workunit.client.0.smithi045.stdout:op 3207 completed, throughput=5MB/sec 2021-07-31T02:15:44.241 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-OUyTc-L21xcNMip off=3523881 len=386240 2021-07-31T02:15:44.242 INFO:tasks.workunit.client.0.smithi045.stdout:op 3208 completed, throughput=5MB/sec 2021-07-31T02:15:44.242 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-JQ0M5INchPHo3G4 off=0 len=1965729 2021-07-31T02:15:44.243 INFO:tasks.workunit.client.0.smithi045.stdout:op 3209 completed, throughput=5MB/sec 2021-07-31T02:15:44.244 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-hsxotLXNMhIMctd off=0 len=1372894 2021-07-31T02:15:44.249 INFO:tasks.workunit.client.0.smithi045.stdout:op 3210 completed, throughput=5MB/sec 2021-07-31T02:15:44.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:44 smithi045 conmon[33655]: cluster 2021-07-31T02:15: 2021-07-31T02:15:44.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:44 smithi045 conmon[33655]: 44.200128+0000 mgr.x (mgr.24542) 1223 : cluster [DBG] pgmap v1727: 9 pgs: 9 active+clean; 127 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 2.4 MiB/s rd, 3.5 MiB/s wr, 5 op/s 2021-07-31T02:15:44.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:44 smithi045 conmon[39718]: cluster 2021-07-31T02:15:44.200128+0000 2021-07-31T02:15:44.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:44 smithi045 conmon[39718]: mgr.x (mgr.24542) 1223 : cluster [DBG] pgmap v1727: 9 pgs: 9 active+clean; 127 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 2.4 MiB/s rd, 3.5 MiB/s wr, 5 op/s 2021-07-31T02:15:44.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:44 smithi154 conmon[34553]: cluster 2021-07-31T02:15:44.200128+0000 mgr.x (mgr.24542) 1223 : cluster [DBG] pgmap v1727: 9 pgs: 9 active+clean; 127 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 2.4 MiB/s rd, 3.5 MiB/s wr, 5 op/s 2021-07-31T02:15:44.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:44 smithi154 conmon[34553]: 2021-07-31T02:15:45.250 INFO:tasks.workunit.client.0.smithi045.stdout: 626: throughput=4.99MB/sec pending data=0 2021-07-31T02:15:45.250 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-0HCXYdRAS6S8iTr off=6809079 len=1961613 2021-07-31T02:15:45.260 INFO:tasks.workunit.client.0.smithi045.stdout:op 3211 completed, throughput=5MB/sec 2021-07-31T02:15:45.260 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-sLtMf7SSYVIMZfo off=13808271 len=263147 2021-07-31T02:15:45.263 INFO:tasks.workunit.client.0.smithi045.stdout:op 3212 completed, throughput=5MB/sec 2021-07-31T02:15:45.264 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-biz1e438j4lICun off=599143 len=739859 2021-07-31T02:15:45.265 INFO:tasks.workunit.client.0.smithi045.stdout:op 3213 completed, throughput=5MB/sec 2021-07-31T02:15:45.265 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-JH8dbAh5jOPYAOf off=4231290 len=256257 2021-07-31T02:15:45.266 INFO:tasks.workunit.client.0.smithi045.stdout:op 3214 completed, throughput=5MB/sec 2021-07-31T02:15:45.267 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-b_pwJIqGn-X9FqK off=13875488 len=1334616 2021-07-31T02:15:45.268 INFO:tasks.workunit.client.0.smithi045.stdout:op 3215 completed, throughput=5MB/sec 2021-07-31T02:15:46.269 INFO:tasks.workunit.client.0.smithi045.stdout: 627: throughput=4.99MB/sec pending data=0 2021-07-31T02:15:46.269 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-YnzVUxSL2pALOrt off=11331787 len=1103013 2021-07-31T02:15:46.271 INFO:tasks.workunit.client.0.smithi045.stdout:op 3216 completed, throughput=4.99MB/sec 2021-07-31T02:15:46.272 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-o7seDbIiD9TC930 off=532986 len=447486 2021-07-31T02:15:46.272 INFO:tasks.workunit.client.0.smithi045.stdout:op 3217 completed, throughput=5MB/sec 2021-07-31T02:15:46.273 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-e2oDdzBYmp08N9b off=2797690 len=731033 2021-07-31T02:15:46.276 INFO:tasks.workunit.client.0.smithi045.stdout:op 3218 completed, throughput=5MB/sec 2021-07-31T02:15:46.277 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-np0Wo9s6wfdJyTf off=251217 len=1377208 2021-07-31T02:15:46.282 INFO:tasks.workunit.client.0.smithi045.stdout:op 3219 completed, throughput=5MB/sec 2021-07-31T02:15:46.283 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-d9acoecj9mASPu_ off=1140076 len=2026202 2021-07-31T02:15:46.286 INFO:tasks.workunit.client.0.smithi045.stdout:op 3220 completed, throughput=5MB/sec 2021-07-31T02:15:46.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:46 smithi045 conmon[39718]: cluster 2021-07-31T02:15:46.200622+0000 mgr.x (mgr 2021-07-31T02:15:46.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:46 smithi045 conmon[39718]: .24542) 1224 : cluster [DBG] pgmap v1728: 9 pgs: 9 active+clean; 127 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2021-07-31T02:15:46.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:46 smithi045 conmon[33655]: cluster 2021-07-31T02:15:46.200622+0000 mgr.x (mgr.24542) 1224 2021-07-31T02:15:46.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:46 smithi045 conmon[33655]: : cluster [DBG] pgmap v1728: 9 pgs: 9 active+clean; 127 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2021-07-31T02:15:46.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:46 smithi154 conmon[34553]: cluster 2021-07-31T02:15:46.200622+0000 mgr.x (mgr.24542) 1224 : cluster [DBG] pgmap v1728: 9 pgs: 9 active+clean; 127 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2021-07-31T02:15:47.286 INFO:tasks.workunit.client.0.smithi045.stdout: 628: throughput=4.99MB/sec pending data=0 2021-07-31T02:15:47.287 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-6lkxRX7plUzqBeb off=3735631 len=1601217 2021-07-31T02:15:47.288 INFO:tasks.workunit.client.0.smithi045.stdout:op 3221 completed, throughput=5MB/sec 2021-07-31T02:15:47.288 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Cl5RKjC_WTO2E0C off=4878196 len=407097 2021-07-31T02:15:47.289 INFO:tasks.workunit.client.0.smithi045.stdout:op 3222 completed, throughput=5MB/sec 2021-07-31T02:15:47.290 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-YYUVz5IyTX6Oub7 off=8238370 len=389704 2021-07-31T02:15:47.292 INFO:tasks.workunit.client.0.smithi045.stdout:op 3223 completed, throughput=5MB/sec 2021-07-31T02:15:47.293 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-UFuF1Kdn8KCncib off=11434406 len=2058827 2021-07-31T02:15:47.294 INFO:tasks.workunit.client.0.smithi045.stdout:op 3224 completed, throughput=5MB/sec 2021-07-31T02:15:47.295 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-BUaSPLEwKuKE4l2 off=12321050 len=731789 2021-07-31T02:15:47.296 INFO:tasks.workunit.client.0.smithi045.stdout:op 3225 completed, throughput=5MB/sec 2021-07-31T02:15:47.524 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:15:47.800 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:15:47.801 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 30m ago 31m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:15:47.801 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 28m ago 31m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:15:47.801 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 28m ago 34m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:15:47.801 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 30m ago 36m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:15:47.802 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 30m ago 36m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:15:47.802 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 28m ago 34m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:15:47.802 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 30m ago 35m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:15:47.802 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 30m ago 31m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:15:47.802 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 28m ago 31m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:15:47.803 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 30m ago 34m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:15:47.803 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 30m ago 34m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:15:47.803 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 30m ago 33m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:15:47.803 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 30m ago 33m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:15:47.804 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 28m ago 33m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:15:47.804 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 28m ago 32m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:15:47.804 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 28m ago 32m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:15:47.804 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 28m ago 32m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:15:47.804 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 28m ago 31m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:15:48.135 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:15:48.135 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:15:48.136 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:15:48.136 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:15:48.136 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:15:48.137 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:15:48.137 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:15:48.137 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:15:48.138 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:15:48.138 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:15:48.138 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:15:48.139 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:15:48.139 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:15:48.139 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:15:48.139 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:15:48.140 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:15:48.140 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:15:48.297 INFO:tasks.workunit.client.0.smithi045.stdout: 629: throughput=4.99MB/sec pending data=0 2021-07-31T02:15:48.298 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-R_wOqlv08y-mEK2 off=7529618 len=744512 2021-07-31T02:15:48.301 INFO:tasks.workunit.client.0.smithi045.stdout:op 3226 completed, throughput=4.99MB/sec 2021-07-31T02:15:48.302 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-eQeWUXV9v4a2Ef6 off=1180002 len=232045 2021-07-31T02:15:48.302 INFO:tasks.workunit.client.0.smithi045.stdout:op 3227 completed, throughput=4.99MB/sec 2021-07-31T02:15:48.302 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-yjcEHxt8vnCnmEI off=10437072 len=1682973 2021-07-31T02:15:48.309 INFO:tasks.workunit.client.0.smithi045.stdout:op 3228 completed, throughput=5MB/sec 2021-07-31T02:15:48.309 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-TraVaHbb29CivLd off=6002875 len=1651210 2021-07-31T02:15:48.312 INFO:tasks.workunit.client.0.smithi045.stdout:op 3229 completed, throughput=5MB/sec 2021-07-31T02:15:48.312 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-3bIcuSw1iuE-VWk off=716908 len=577887 2021-07-31T02:15:48.313 INFO:tasks.workunit.client.0.smithi045.stdout:op 3230 completed, throughput=5MB/sec 2021-07-31T02:15:48.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:48 smithi154 conmon[34553]: audit 2021-07-31T02:15:48.134999+0000 mon.c (mon.1) 84 : audit [DBG] from='client.? 172.21.15.45:0/4276731580' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:15:48.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:48 smithi045 conmon[33655]: audit 2021-07-31T02:15:48 2021-07-31T02:15:48.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:48 smithi045 conmon[33655]: .134999+0000 mon.c (mon.1) 84 : audit [DBG] from='client.? 172.21.15.45:0/4276731580' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:15:48.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:48 smithi045 conmon[39718]: audit 2021-07-31T02:15:48.134999 2021-07-31T02:15:48.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:48 smithi045 conmon[39718]: +0000 mon.c (mon.1) 84 : audit [DBG] from='client.? 172.21.15.45:0/4276731580' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:15:49.314 INFO:tasks.workunit.client.0.smithi045.stdout: 630: throughput=4.99MB/sec pending data=0 2021-07-31T02:15:49.314 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ijEaneD053rNNAn off=15909594 len=1391265 2021-07-31T02:15:49.319 INFO:tasks.workunit.client.0.smithi045.stdout:op 3231 completed, throughput=4.99MB/sec 2021-07-31T02:15:49.320 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-xITWRBy-aUMEwW2 off=117638 len=1633127 2021-07-31T02:15:49.323 INFO:tasks.workunit.client.0.smithi045.stdout:op 3232 completed, throughput=5MB/sec 2021-07-31T02:15:49.323 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-N8loNVBEVfd50BS off=2403446 len=1030438 2021-07-31T02:15:49.330 INFO:tasks.workunit.client.0.smithi045.stdout:op 3233 completed, throughput=5MB/sec 2021-07-31T02:15:49.330 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-h2hd5stTmU5KMG3 off=472046 len=970692 2021-07-31T02:15:49.331 INFO:tasks.workunit.client.0.smithi045.stdout:op 3234 completed, throughput=5MB/sec 2021-07-31T02:15:49.331 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-dUDBoyRKaPcAaT6 off=1077057 len=1152556 2021-07-31T02:15:49.336 INFO:tasks.workunit.client.0.smithi045.stdout:op 3235 completed, throughput=5MB/sec 2021-07-31T02:15:49.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:49 smithi154 conmon[34553]: audit 2021-07-31T02:15:47.513111+0000 mgr.x (mgr.24542) 1225 : audit [DBG] from='client.16095 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:15:49.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:49 smithi154 conmon[34553]: audit 2021-07-31T02:15:47.792860+0000 mgr.x (mgr.24542) 1226 : audit [DBG] from='client.26008 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:15:49.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:49 smithi154 conmon[34553]: cluster 2021-07-31T02:15:48.201084+0000 mgr.x (mgr.24542) 1227 : cluster [DBG] 2021-07-31T02:15:49.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:49 smithi154 conmon[34553]: pgmap v1729: 9 pgs: 9 active+clean; 127 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2021-07-31T02:15:49.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:49 smithi045 conmon[33655]: audit 2021-07-31T02:15:47.513111+0000 mgr.x (mgr.24542) 1225 2021-07-31T02:15:49.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:49 smithi045 conmon[33655]: : audit [DBG] from='client.16095 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:15:49.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:49 smithi045 conmon[33655]: audit 2021-07-31T02:15:47.792860+0000 mgr.x (mgr.24542) 1226 : audit [DBG] from='client.26008 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:15:49.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:49 smithi045 conmon[33655]: cluster 2021-07-31T02: 2021-07-31T02:15:49.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:49 smithi045 conmon[33655]: 15:48.201084+0000 mgr.x (mgr.24542) 1227 : cluster [DBG] pgmap v1729: 9 pgs: 9 active+clean; 127 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2021-07-31T02:15:49.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:49 smithi045 conmon[39718]: audit 2021-07-31T02:15:47.513111+0000 mgr.x (mgr.24542) 1225 : audit [DBG] from='client.16095 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:15:49.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:49 smithi045 conmon[39718]: 2021-07-31T02:15:49.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:49 smithi045 conmon[39718]: audit 2021-07-31T02:15:47.792860 2021-07-31T02:15:49.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:49 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 1226 : audit [DBG] 2021-07-31T02:15:49.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:49 smithi045 conmon[39718]: from='client.26008 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:15:49.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:49 smithi045 conmon[39718]: cluster 2021-07-31T02:15:49.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:49 smithi045 conmon[39718]: 2021-07-31T02: 2021-07-31T02:15:49.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:49 smithi045 conmon[39718]: 15:48.201084+0000 mgr.x ( 2021-07-31T02:15:49.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:49 smithi045 conmon[39718]: mgr.24542) 1227 : cluster [DBG] 2021-07-31T02:15:49.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:49 smithi045 conmon[39718]: pgmap v1729: 9 pgs: 9 active+clean; 127 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2021-07-31T02:15:50.336 INFO:tasks.workunit.client.0.smithi045.stdout: 631: throughput=4.99MB/sec pending data=0 2021-07-31T02:15:50.336 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-GtWyzNYwHPqAyPj off=1616477 len=342603 2021-07-31T02:15:50.337 INFO:tasks.workunit.client.0.smithi045.stdout:op 3236 completed, throughput=4.99MB/sec 2021-07-31T02:15:50.337 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Mx3GAimzRBcR84L off=7356731 len=2048057 2021-07-31T02:15:50.347 INFO:tasks.workunit.client.0.smithi045.stdout:op 3237 completed, throughput=5MB/sec 2021-07-31T02:15:50.348 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-8yg-M88lWlVw8oO off=762425 len=664250 2021-07-31T02:15:50.349 INFO:tasks.workunit.client.0.smithi045.stdout:op 3238 completed, throughput=5MB/sec 2021-07-31T02:15:50.349 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-WXI-kg60goEpYR6 off=10922663 len=741911 2021-07-31T02:15:50.353 INFO:tasks.workunit.client.0.smithi045.stdout:op 3239 completed, throughput=5MB/sec 2021-07-31T02:15:50.354 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-F9kOd5Jhtsg0J6D off=9160824 len=1162034 2021-07-31T02:15:50.355 INFO:tasks.workunit.client.0.smithi045.stdout:op 3240 completed, throughput=5MB/sec 2021-07-31T02:15:50.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:50 smithi045 conmon[39718]: cluster 2021-07-31T02:15:50.201703+0000 mgr.x (mgr 2021-07-31T02:15:50.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:50 smithi045 conmon[39718]: .24542) 1228 : cluster [DBG] pgmap v1730: 9 pgs: 9 active+clean; 127 GiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 3.0 MiB/s rd, 3.4 MiB/s wr, 6 op/s 2021-07-31T02:15:50.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:50 smithi045 conmon[33655]: cluster 2021-07-31T02:15:50.201703+0000 mgr.x (mgr.24542) 1228 : cluster 2021-07-31T02:15:50.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:50 smithi045 conmon[33655]: [DBG] pgmap v1730: 9 pgs: 9 active+clean; 127 GiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 3.0 MiB/s rd, 3.4 MiB/s wr, 6 op/s 2021-07-31T02:15:50.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:50 smithi154 conmon[34553]: cluster 2021-07-31T02:15:50.201703+0000 mgr.x (mgr.24542) 1228 : cluster [DBG] pgmap v1730: 9 pgs: 9 active+clean; 127 GiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 3.0 MiB/s rd, 3.4 MiB/s wr, 6 op/s 2021-07-31T02:15:50.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:50 smithi154 conmon[34553]: 2021-07-31T02:15:51.356 INFO:tasks.workunit.client.0.smithi045.stdout: 632: throughput=4.99MB/sec pending data=0 2021-07-31T02:15:51.356 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ZaH3ANF5I31W42Z off=2052946 len=274572 2021-07-31T02:15:51.357 INFO:tasks.workunit.client.0.smithi045.stdout:op 3241 completed, throughput=4.99MB/sec 2021-07-31T02:15:51.357 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-RqlqxRawJo-NPoi off=5406886 len=1512936 2021-07-31T02:15:51.364 INFO:tasks.workunit.client.0.smithi045.stdout:op 3242 completed, throughput=5MB/sec 2021-07-31T02:15:51.364 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-oe5_1ESIWHo5gfE off=3299316 len=1208439 2021-07-31T02:15:51.367 INFO:tasks.workunit.client.0.smithi045.stdout:op 3243 completed, throughput=5MB/sec 2021-07-31T02:15:51.367 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-WNJRo72YxCspPg- off=2976867 len=989336 2021-07-31T02:15:51.369 INFO:tasks.workunit.client.0.smithi045.stdout:op 3244 completed, throughput=5MB/sec 2021-07-31T02:15:51.370 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-VU0-u-4XjuKkT4l off=2016390 len=825172 2021-07-31T02:15:51.372 INFO:tasks.workunit.client.0.smithi045.stdout:op 3245 completed, throughput=5MB/sec 2021-07-31T02:15:52.373 INFO:tasks.workunit.client.0.smithi045.stdout: 633: throughput=4.99MB/sec pending data=0 2021-07-31T02:15:52.373 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-WZC-Sl4ic5fQd87 off=7207007 len=1038992 2021-07-31T02:15:52.378 INFO:tasks.workunit.client.0.smithi045.stdout:op 3246 completed, throughput=4.99MB/sec 2021-07-31T02:15:52.379 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-YTMJfvET_4EjaMR off=20528420 len=1543407 2021-07-31T02:15:52.381 INFO:tasks.workunit.client.0.smithi045.stdout:op 3247 completed, throughput=5MB/sec 2021-07-31T02:15:52.382 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-2xMGUFe9qku31i5 off=20551247 len=1158327 2021-07-31T02:15:52.389 INFO:tasks.workunit.client.0.smithi045.stdout:op 3248 completed, throughput=5MB/sec 2021-07-31T02:15:52.389 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-h0ZpW2ZkjGPd0iS off=6735546 len=623212 2021-07-31T02:15:52.392 INFO:tasks.workunit.client.0.smithi045.stdout:op 3249 completed, throughput=5MB/sec 2021-07-31T02:15:52.393 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-uY9njNEHY542zue off=2888526 len=50224 2021-07-31T02:15:52.394 INFO:tasks.workunit.client.0.smithi045.stdout:op 3250 completed, throughput=5MB/sec 2021-07-31T02:15:52.395 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ETg0z_Z12hK-iFO off=16227714 len=2044016 2021-07-31T02:15:52.398 INFO:tasks.workunit.client.0.smithi045.stdout:op 3251 completed, throughput=5MB/sec 2021-07-31T02:15:53.398 INFO:tasks.workunit.client.0.smithi045.stdout: 634: throughput=4.99MB/sec pending data=0 2021-07-31T02:15:53.398 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-9ex8JMJk2KvMZMG off=9102866 len=1263988 2021-07-31T02:15:53.401 INFO:tasks.workunit.client.0.smithi045.stdout:op 3252 completed, throughput=5MB/sec 2021-07-31T02:15:53.402 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-2LvuM_N8mPEwdIc off=13401770 len=58138 2021-07-31T02:15:53.404 INFO:tasks.workunit.client.0.smithi045.stdout:op 3253 completed, throughput=5MB/sec 2021-07-31T02:15:53.404 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-fnoj_F1xIXnDlB2 off=2513216 len=611455 2021-07-31T02:15:53.408 INFO:tasks.workunit.client.0.smithi045.stdout:op 3254 completed, throughput=5MB/sec 2021-07-31T02:15:53.408 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-nQmqik6rSb-8yGB off=5819864 len=1442626 2021-07-31T02:15:53.416 INFO:tasks.workunit.client.0.smithi045.stdout:op 3255 completed, throughput=5MB/sec 2021-07-31T02:15:53.416 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Jrx1VC-SKzgh2W_ off=688749 len=653149 2021-07-31T02:15:53.420 INFO:tasks.workunit.client.0.smithi045.stdout:op 3256 completed, throughput=5MB/sec 2021-07-31T02:15:53.420 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-fnoj_F1xIXnDlB2 off=11673176 len=226319 2021-07-31T02:15:53.421 INFO:tasks.workunit.client.0.smithi045.stdout:op 3257 completed, throughput=5MB/sec 2021-07-31T02:15:53.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:53 smithi154 conmon[34553]: cluster 2021-07-31T02:15:52.202096+0000 mgr.x (mgr.24542) 1229 : cluster [DBG] 2021-07-31T02:15:53.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:53 smithi154 conmon[34553]: pgmap v1731: 9 pgs: 9 active+clean; 127 GiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2021-07-31T02:15:53.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:53 smithi045 conmon[39718]: cluster 2021-07-31T02:15:52.202096+0000 mgr.x (mgr.24542) 1229 : cluster [DBG] 2021-07-31T02:15:53.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:53 smithi045 conmon[39718]: pgmap v1731: 9 pgs: 9 active+clean; 127 GiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2021-07-31T02:15:53.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:53 smithi045 conmon[33655]: cluster 2021-07-31T02:15:52.202096+0000 mgr.x (mgr.24542) 1229 : cluster [DBG] pgmap v1731: 9 pgs: 9 active+clean; 127 GiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2021-07-31T02:15:53.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:53 smithi045 conmon[33655]: 2021-07-31T02:15:54.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:15:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:15:53.557Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:15:54.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:15:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:15:53.557Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:15:54.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:15:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:15:53.557Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:15:54.422 INFO:tasks.workunit.client.0.smithi045.stdout: 635: throughput=4.99MB/sec pending data=0 2021-07-31T02:15:54.422 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-1T-quRFnzDxcsHv off=6987056 len=1201453 2021-07-31T02:15:54.430 INFO:tasks.workunit.client.0.smithi045.stdout:op 3258 completed, throughput=4.99MB/sec 2021-07-31T02:15:54.430 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-fhY46r5j-cr-1L4 off=2456305 len=756958 2021-07-31T02:15:54.431 INFO:tasks.workunit.client.0.smithi045.stdout:op 3259 completed, throughput=5MB/sec 2021-07-31T02:15:54.431 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-OKwNGXk83JFwjGV off=15785610 len=925347 2021-07-31T02:15:54.438 INFO:tasks.workunit.client.0.smithi045.stdout:op 3260 completed, throughput=5MB/sec 2021-07-31T02:15:54.438 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-fB0MzUlyH4VJFCT off=6310009 len=1725645 2021-07-31T02:15:54.439 INFO:tasks.workunit.client.0.smithi045.stdout:op 3261 completed, throughput=5MB/sec 2021-07-31T02:15:54.439 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-lytt7eYBADYZO9r off=1456848 len=1023346 2021-07-31T02:15:54.441 INFO:tasks.workunit.client.0.smithi045.stdout:op 3262 completed, throughput=5MB/sec 2021-07-31T02:15:54.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:54 smithi045 conmon[39718]: cluster 2021-07-31T02:15:54.202558+0000 mgr.x (mgr. 2021-07-31T02:15:54.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:54 smithi045 conmon[39718]: 24542) 1230 : cluster [DBG] pgmap v1732: 9 pgs: 9 active+clean; 127 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 2.8 MiB/s rd, 3.4 MiB/s wr, 6 op/s 2021-07-31T02:15:54.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:54 smithi045 conmon[33655]: cluster 2021-07-31T02:15:54.202558+0000 mgr.x (mgr.24542) 1230 : cluster [DBG] pgmap v1732: 9 pgs: 9 active+clean; 127 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 2.8 MiB/s rd, 3.4 MiB/s wr, 6 op/s 2021-07-31T02:15:54.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:54 smithi154 conmon[34553]: cluster 2021-07-31T02:15:54.202558+0000 mgr.x (mgr.24542) 1230 : cluster [DBG] pgmap v1732: 9 pgs: 9 active+clean; 127 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 2.8 MiB/s rd, 3.4 MiB/s wr, 6 op/s 2021-07-31T02:15:54.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:54 smithi154 conmon[34553]: 2021-07-31T02:15:55.442 INFO:tasks.workunit.client.0.smithi045.stdout: 636: throughput=4.99MB/sec pending data=0 2021-07-31T02:15:55.442 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-4RLrOczlwo1CYKk off=9073193 len=875047 2021-07-31T02:15:55.444 INFO:tasks.workunit.client.0.smithi045.stdout:op 3263 completed, throughput=4.99MB/sec 2021-07-31T02:15:55.444 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-L6Ro03Jkqu0efNm off=4903152 len=1853392 2021-07-31T02:15:55.454 INFO:tasks.workunit.client.0.smithi045.stdout:op 3264 completed, throughput=5MB/sec 2021-07-31T02:15:55.454 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ECxIiBDiC6NsRKP off=141726 len=911910 2021-07-31T02:15:55.456 INFO:tasks.workunit.client.0.smithi045.stdout:op 3265 completed, throughput=5MB/sec 2021-07-31T02:15:55.456 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Db7c6LJHwQNocTj off=2775215 len=1428148 2021-07-31T02:15:55.462 INFO:tasks.workunit.client.0.smithi045.stdout:op 3266 completed, throughput=5MB/sec 2021-07-31T02:15:56.463 INFO:tasks.workunit.client.0.smithi045.stdout: 637: throughput=4.99MB/sec pending data=0 2021-07-31T02:15:56.463 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-a1kQCx6L4GCTi5s off=1508318 len=33230 2021-07-31T02:15:56.464 INFO:tasks.workunit.client.0.smithi045.stdout:op 3267 completed, throughput=4.99MB/sec 2021-07-31T02:15:56.464 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-KlOFFWBA7WIJ4X9 off=4826636 len=71991 2021-07-31T02:15:56.466 INFO:tasks.workunit.client.0.smithi045.stdout:op 3268 completed, throughput=4.99MB/sec 2021-07-31T02:15:56.466 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ycEigf6nDyN2zB0 off=5103826 len=1074177 2021-07-31T02:15:56.468 INFO:tasks.workunit.client.0.smithi045.stdout:op 3269 completed, throughput=4.99MB/sec 2021-07-31T02:15:56.469 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj--xYffa-7J2FyvV1 off=2011592 len=1659840 2021-07-31T02:15:56.470 INFO:tasks.workunit.client.0.smithi045.stdout:op 3270 completed, throughput=5MB/sec 2021-07-31T02:15:56.471 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-jmSZ0IPiXkYahUG off=5956536 len=1446019 2021-07-31T02:15:56.477 INFO:tasks.workunit.client.0.smithi045.stdout:op 3271 completed, throughput=5MB/sec 2021-07-31T02:15:56.478 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-VLU19E0lRr8MgiU off=4641838 len=140431 2021-07-31T02:15:56.479 INFO:tasks.workunit.client.0.smithi045.stdout:op 3272 completed, throughput=5MB/sec 2021-07-31T02:15:56.479 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-MrA5sXPGk29vzz9 off=8052858 len=958176 2021-07-31T02:15:56.480 INFO:tasks.workunit.client.0.smithi045.stdout:op 3273 completed, throughput=5MB/sec 2021-07-31T02:15:56.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:56 smithi045 conmon[39718]: cluster 2021-07-31T02:15:56.203005+0000 mgr.x ( 2021-07-31T02:15:56.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:56 smithi045 conmon[39718]: mgr.24542) 1231 : cluster [DBG] pgmap v1733: 9 pgs: 9 active+clean; 127 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2021-07-31T02:15:56.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:56 smithi045 conmon[33655]: cluster 2021-07-31T02:15:56.203005+0000 mgr.x (mgr.24542) 1231 : cluster [DBG] 2021-07-31T02:15:56.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:56 smithi045 conmon[33655]: pgmap v1733: 9 pgs: 9 active+clean; 127 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2021-07-31T02:15:56.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:56 smithi154 conmon[34553]: cluster 2021-07-31T02:15:56.203005+0000 mgr.x (mgr.24542) 1231 : cluster [DBG] pgmap v1733: 9 pgs: 9 active+clean; 127 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2021-07-31T02:15:57.480 INFO:tasks.workunit.client.0.smithi045.stdout: 638: throughput=4.99MB/sec pending data=0 2021-07-31T02:15:57.480 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-RTu8B-GdPoiYeGb off=1075506 len=173854 2021-07-31T02:15:57.481 INFO:tasks.workunit.client.0.smithi045.stdout:op 3274 completed, throughput=4.99MB/sec 2021-07-31T02:15:57.481 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-8BZApo4xCnLWIBn off=13025635 len=1919706 2021-07-31T02:15:57.485 INFO:tasks.workunit.client.0.smithi045.stdout:op 3275 completed, throughput=5MB/sec 2021-07-31T02:15:57.485 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-bOu5p-KhmfeyNAb off=1380493 len=406675 2021-07-31T02:15:57.485 INFO:tasks.workunit.client.0.smithi045.stdout:op 3276 completed, throughput=5MB/sec 2021-07-31T02:15:57.486 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-FjPh4zpO3t6Wogt off=12971602 len=104554 2021-07-31T02:15:57.486 INFO:tasks.workunit.client.0.smithi045.stdout:op 3277 completed, throughput=5MB/sec 2021-07-31T02:15:57.487 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-3hwaptNE3VtuR1T off=2304860 len=78027 2021-07-31T02:15:57.487 INFO:tasks.workunit.client.0.smithi045.stdout:op 3278 completed, throughput=5MB/sec 2021-07-31T02:15:57.487 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-DtUxmparsYsXsL4 off=2757573 len=1355514 2021-07-31T02:15:57.488 INFO:tasks.workunit.client.0.smithi045.stdout:op 3279 completed, throughput=5MB/sec 2021-07-31T02:15:57.489 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Xlz9lQ94vPNrzXS off=1094712 len=1569397 2021-07-31T02:15:57.492 INFO:tasks.workunit.client.0.smithi045.stdout:op 3280 completed, throughput=5MB/sec 2021-07-31T02:15:58.492 INFO:tasks.workunit.client.0.smithi045.stdout: 639: throughput=4.99MB/sec pending data=0 2021-07-31T02:15:58.492 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-n_ppTh1B8VLr25L off=6876589 len=751244 2021-07-31T02:15:58.493 INFO:tasks.workunit.client.0.smithi045.stdout:op 3281 completed, throughput=4.99MB/sec 2021-07-31T02:15:58.493 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-7uqAeB33vZYTu4c off=2856288 len=1571696 2021-07-31T02:15:58.496 INFO:tasks.workunit.client.0.smithi045.stdout:op 3282 completed, throughput=5MB/sec 2021-07-31T02:15:58.496 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-fvjUVANdcubjt5M off=7149090 len=2037837 2021-07-31T02:15:58.507 INFO:tasks.workunit.client.0.smithi045.stdout:op 3283 completed, throughput=5MB/sec 2021-07-31T02:15:58.507 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-eJ-5sglcHTPK2qQ off=10035498 len=1577585 2021-07-31T02:15:58.511 INFO:tasks.workunit.client.0.smithi045.stdout:op 3284 completed, throughput=5MB/sec 2021-07-31T02:15:58.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:15:58 smithi154 conmon[34553]: cluster 2021-07-31T02:15:58.203428+0000 mgr.x (mgr.24542) 1232 : cluster [DBG] pgmap v1734: 9 pgs: 9 active+clean; 127 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2021-07-31T02:15:58.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:58 smithi045 conmon[39718]: cluster 2021-07-31T02:15:58.203428+0000 mgr.x ( 2021-07-31T02:15:58.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:15:58 smithi045 conmon[39718]: mgr.24542) 1232 : cluster [DBG] pgmap v1734: 9 pgs: 9 active+clean; 127 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2021-07-31T02:15:58.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:58 smithi045 conmon[33655]: cluster 2021-07-31T02:15:58.203428 2021-07-31T02:15:58.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:15:58 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 1232 : cluster [DBG] pgmap v1734: 9 pgs: 9 active+clean; 127 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2021-07-31T02:15:59.512 INFO:tasks.workunit.client.0.smithi045.stdout: 640: throughput=4.99MB/sec pending data=0 2021-07-31T02:15:59.513 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-7RdpDNi6HcHrayU off=12617648 len=590319 2021-07-31T02:15:59.516 INFO:tasks.workunit.client.0.smithi045.stdout:op 3285 completed, throughput=4.99MB/sec 2021-07-31T02:15:59.516 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-_ZeAE4d7s1_PtIj off=1155980 len=5156 2021-07-31T02:15:59.519 INFO:tasks.workunit.client.0.smithi045.stdout:op 3286 completed, throughput=4.99MB/sec 2021-07-31T02:15:59.520 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-VlvWO0ioXujibSL off=13609259 len=1619603 2021-07-31T02:15:59.522 INFO:tasks.workunit.client.0.smithi045.stdout:op 3287 completed, throughput=5MB/sec 2021-07-31T02:15:59.522 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-2DnpW0jC9zwVP57 off=9930638 len=1454643 2021-07-31T02:15:59.531 INFO:tasks.workunit.client.0.smithi045.stdout:op 3288 completed, throughput=5MB/sec 2021-07-31T02:15:59.531 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-KYA5mwH6QvCyESB off=672892 len=1872580 2021-07-31T02:15:59.540 INFO:tasks.workunit.client.0.smithi045.stdout:op 3289 completed, throughput=5MB/sec 2021-07-31T02:16:00.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:00 smithi045 conmon[39718]: cluster 2021-07-31T02:16:00.203892+0000 mgr.x (mgr.24542) 2021-07-31T02:16:00.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:00 smithi045 conmon[39718]: 1233 : cluster [DBG] pgmap v1735: 9 pgs: 9 active+clean; 127 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 3.7 MiB/s rd, 2.7 MiB/s wr, 6 op/s 2021-07-31T02:16:00.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:00 smithi045 conmon[33655]: cluster 2021-07-31T02:16:00.203892+0000 mgr.x (mgr.24542) 1233 : cluster [DBG] pgmap v1735: 9 pgs: 9 active+clean; 127 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 3.7 MiB/s rd, 2.7 MiB/s wr, 6 op/s 2021-07-31T02:16:00.540 INFO:tasks.workunit.client.0.smithi045.stdout: 641: throughput=4.99MB/sec pending data=0 2021-07-31T02:16:00.541 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-UJiPNUKa9_NO-7Z off=5564072 len=1932904 2021-07-31T02:16:00.552 INFO:tasks.workunit.client.0.smithi045.stdout:op 3290 completed, throughput=5MB/sec 2021-07-31T02:16:00.553 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-_6NeCuNYBOvDS2i off=6540701 len=1428673 2021-07-31T02:16:00.554 INFO:tasks.workunit.client.0.smithi045.stdout:op 3291 completed, throughput=5MB/sec 2021-07-31T02:16:00.555 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-qtjMN0XcfSFafag off=16134958 len=1975121 2021-07-31T02:16:00.558 INFO:tasks.workunit.client.0.smithi045.stdout:op 3292 completed, throughput=5MB/sec 2021-07-31T02:16:00.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:00 smithi154 conmon[34553]: cluster 2021-07-31T02:16:00.203892+0000 mgr.x (mgr.24542) 1233 : cluster [DBG] pgmap v1735: 9 pgs: 9 active+clean; 127 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 3.7 MiB/s rd, 2.7 MiB/s wr, 6 op/s 2021-07-31T02:16:01.558 INFO:tasks.workunit.client.0.smithi045.stdout: 642: throughput=4.99MB/sec pending data=0 2021-07-31T02:16:01.559 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-kgfMax7q-dis-9p off=1387060 len=1069017 2021-07-31T02:16:01.565 INFO:tasks.workunit.client.0.smithi045.stdout:op 3293 completed, throughput=5MB/sec 2021-07-31T02:16:01.565 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-rHj42S-ZMkbKHzS off=1083874 len=1011785 2021-07-31T02:16:01.566 INFO:tasks.workunit.client.0.smithi045.stdout:op 3294 completed, throughput=5MB/sec 2021-07-31T02:16:01.567 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-kuFHCHuZwbXsPEy off=838370 len=2019826 2021-07-31T02:16:01.577 INFO:tasks.workunit.client.0.smithi045.stdout:op 3295 completed, throughput=5MB/sec 2021-07-31T02:16:01.578 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Lq1VxWuw7YRZPZ8 off=884272 len=1261672 2021-07-31T02:16:01.583 INFO:tasks.workunit.client.0.smithi045.stdout:op 3296 completed, throughput=5MB/sec 2021-07-31T02:16:02.583 INFO:tasks.workunit.client.0.smithi045.stdout: 643: throughput=4.99MB/sec pending data=0 2021-07-31T02:16:02.584 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-MfRNLyVAvRBh-3- off=803549 len=756195 2021-07-31T02:16:02.589 INFO:tasks.workunit.client.0.smithi045.stdout:op 3297 completed, throughput=5MB/sec 2021-07-31T02:16:02.589 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-h0Y5I_9O_LX1yNy off=9015690 len=1513196 2021-07-31T02:16:02.596 INFO:tasks.workunit.client.0.smithi045.stdout:op 3298 completed, throughput=5MB/sec 2021-07-31T02:16:02.597 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-x3pzq06oFSpeVaz off=208906 len=110457 2021-07-31T02:16:02.599 INFO:tasks.workunit.client.0.smithi045.stdout:op 3299 completed, throughput=5MB/sec 2021-07-31T02:16:02.599 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-riZFerXNoAxXai- off=7551091 len=1693606 2021-07-31T02:16:02.602 INFO:tasks.workunit.client.0.smithi045.stdout:op 3300 completed, throughput=5MB/sec 2021-07-31T02:16:02.602 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-9c0ErdU6pOxGaFU off=17690126 len=1319163 2021-07-31T02:16:02.608 INFO:tasks.workunit.client.0.smithi045.stdout:op 3301 completed, throughput=5MB/sec 2021-07-31T02:16:03.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:03 smithi154 conmon[34553]: cluster 2021-07-31T02:16:02.204432+0000 mgr.x (mgr.24542) 1234 : cluster [DBG] pgmap v1736: 9 pgs: 9 active+clean; 127 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2021-07-31T02:16:03.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:03 smithi045 conmon[39718]: cluster 2021-07-31T02:16:02.204432+0000 mgr.x (mgr.24542) 1234 : cluster [DBG] pgmap v1736: 9 pgs: 9 active+clean; 127 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2021-07-31T02:16:03.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:03 smithi045 conmon[33655]: cluster 2021-07-31T02:16:02.204432+0000 mgr.x (mgr.24542) 1234 : cluster [DBG] pgmap v1736: 9 pgs: 9 active+clean; 127 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2021-07-31T02:16:03.608 INFO:tasks.workunit.client.0.smithi045.stdout: 644: throughput=4.99MB/sec pending data=0 2021-07-31T02:16:03.608 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-QCgWXOkGJM7-E3s off=5695360 len=455411 2021-07-31T02:16:03.610 INFO:tasks.workunit.client.0.smithi045.stdout:op 3302 completed, throughput=4.99MB/sec 2021-07-31T02:16:03.610 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ndZAA4fm1x2jRvP off=19331731 len=1477007 2021-07-31T02:16:03.612 INFO:tasks.workunit.client.0.smithi045.stdout:op 3303 completed, throughput=5MB/sec 2021-07-31T02:16:03.612 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-vfaaQEV5hlxXDTc off=6845124 len=1529860 2021-07-31T02:16:03.614 INFO:tasks.workunit.client.0.smithi045.stdout:op 3304 completed, throughput=5MB/sec 2021-07-31T02:16:03.614 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-YN_t4g1ZYj95jn4 off=10457548 len=182920 2021-07-31T02:16:03.617 INFO:tasks.workunit.client.0.smithi045.stdout:op 3305 completed, throughput=5MB/sec 2021-07-31T02:16:03.618 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-v5savTkIu4d9GlV off=8534316 len=1971967 2021-07-31T02:16:03.621 INFO:tasks.workunit.client.0.smithi045.stdout:op 3306 completed, throughput=5MB/sec 2021-07-31T02:16:04.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:16:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:16:03.558Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:16:04.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:16:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:16:03.558Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:16:04.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:16:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:16:03.558Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:16:04.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:04 smithi045 conmon[33655]: cluster 2021-07-31T02:16:04.205011+0000 mgr.x (mgr.24542) 1235 : cluster [DBG] pgmap v1737: 9 pgs: 9 active+clean; 127 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 3.5 MiB/s rd, 3.0 MiB/s wr, 6 op/s 2021-07-31T02:16:04.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:04 smithi045 conmon[39718]: cluster 2021-07-31T02:16:04.205011+0000 mgr.x ( 2021-07-31T02:16:04.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:04 smithi045 conmon[39718]: mgr.24542) 1235 : cluster [DBG] pgmap v1737: 9 pgs: 9 active+clean; 127 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 3.5 MiB/s rd, 3.0 MiB/s wr, 6 op/s 2021-07-31T02:16:04.621 INFO:tasks.workunit.client.0.smithi045.stdout: 645: throughput=4.99MB/sec pending data=0 2021-07-31T02:16:04.622 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-zFI97HqZ9ICstVl off=1563636 len=1993509 2021-07-31T02:16:04.625 INFO:tasks.workunit.client.0.smithi045.stdout:op 3307 completed, throughput=5MB/sec 2021-07-31T02:16:04.625 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Xl6omTIU4eaJ-9l off=6906179 len=679039 2021-07-31T02:16:04.630 INFO:tasks.workunit.client.0.smithi045.stdout:op 3308 completed, throughput=5MB/sec 2021-07-31T02:16:04.630 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-lGVP9M1YkcAX6Ah off=291194 len=1942953 2021-07-31T02:16:04.633 INFO:tasks.workunit.client.0.smithi045.stdout:op 3309 completed, throughput=5MB/sec 2021-07-31T02:16:04.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:04 smithi154 conmon[34553]: cluster 2021-07-31T02:16:04.205011+0000 mgr.x (mgr.24542) 1235 : cluster [DBG] pgmap v1737: 9 pgs: 9 active+clean; 127 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 3.5 MiB/s rd, 3.0 MiB/s wr, 6 op/s 2021-07-31T02:16:05.633 INFO:tasks.workunit.client.0.smithi045.stdout: 646: throughput=4.99MB/sec pending data=0 2021-07-31T02:16:05.634 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-NAOxondR2h_fSQE off=12742245 len=1054990 2021-07-31T02:16:05.641 INFO:tasks.workunit.client.0.smithi045.stdout:op 3310 completed, throughput=4.99MB/sec 2021-07-31T02:16:05.641 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-eU6Od6-YC0HO-Et off=962881 len=1802831 2021-07-31T02:16:05.648 INFO:tasks.workunit.client.0.smithi045.stdout:op 3311 completed, throughput=5MB/sec 2021-07-31T02:16:05.648 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-d9acoecj9mASPu_ off=2434272 len=126352 2021-07-31T02:16:05.649 INFO:tasks.workunit.client.0.smithi045.stdout:op 3312 completed, throughput=5MB/sec 2021-07-31T02:16:05.649 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-VaqOcTfq22Junpc off=3657651 len=332358 2021-07-31T02:16:05.652 INFO:tasks.workunit.client.0.smithi045.stdout:op 3313 completed, throughput=5MB/sec 2021-07-31T02:16:05.652 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-TOMDlCO8LUfd0x9 off=2044822 len=34002 2021-07-31T02:16:05.653 INFO:tasks.workunit.client.0.smithi045.stdout:op 3314 completed, throughput=5MB/sec 2021-07-31T02:16:05.653 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj--K0VEPva9TXLIYS off=9981515 len=461900 2021-07-31T02:16:05.654 INFO:tasks.workunit.client.0.smithi045.stdout:op 3315 completed, throughput=5MB/sec 2021-07-31T02:16:05.654 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-70VooUW9a_oCSxU off=1772741 len=1821316 2021-07-31T02:16:05.655 INFO:tasks.workunit.client.0.smithi045.stdout:op 3316 completed, throughput=5MB/sec 2021-07-31T02:16:06.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:06 smithi045 conmon[33655]: cluster 2021-07-31T02:16:06.205445+0000 mgr.x (mgr.24542) 1236 2021-07-31T02:16:06.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:06 smithi045 conmon[33655]: : cluster [DBG] pgmap v1738: 9 pgs: 9 active+clean; 127 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.8 MiB/s wr, 4 op/s 2021-07-31T02:16:06.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:06 smithi045 conmon[39718]: cluster 2021-07-31T02:16:06.205445+0000 mgr.x (mgr.24542 2021-07-31T02:16:06.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:06 smithi045 conmon[39718]: ) 1236 : cluster [DBG] pgmap v1738: 9 pgs: 9 active+clean; 127 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.8 MiB/s wr, 4 op/s 2021-07-31T02:16:06.655 INFO:tasks.workunit.client.0.smithi045.stdout: 647: throughput=4.99MB/sec pending data=0 2021-07-31T02:16:06.656 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj--Ds4HgXl2EN9QRo off=2435475 len=1987771 2021-07-31T02:16:06.659 INFO:tasks.workunit.client.0.smithi045.stdout:op 3317 completed, throughput=5MB/sec 2021-07-31T02:16:06.659 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-rQUbSSwLMMyTI7H off=10933870 len=266761 2021-07-31T02:16:06.660 INFO:tasks.workunit.client.0.smithi045.stdout:op 3318 completed, throughput=5MB/sec 2021-07-31T02:16:06.661 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-zCOSAZSZS8waQOK off=215153 len=1759336 2021-07-31T02:16:06.664 INFO:tasks.workunit.client.0.smithi045.stdout:op 3319 completed, throughput=5MB/sec 2021-07-31T02:16:06.665 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-DSU74IjIGzeWUgN off=1256476 len=1572740 2021-07-31T02:16:06.667 INFO:tasks.workunit.client.0.smithi045.stdout:op 3320 completed, throughput=5MB/sec 2021-07-31T02:16:06.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:06 smithi154 conmon[34553]: cluster 2021-07-31T02:16:06.205445+0000 mgr.x (mgr.24542) 1236 : cluster [DBG] pgmap v1738: 9 pgs: 9 active+clean; 127 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.8 MiB/s wr, 4 op/s 2021-07-31T02:16:07.673 INFO:tasks.workunit.client.0.smithi045.stdout: 648: throughput=4.99MB/sec pending data=0 2021-07-31T02:16:07.673 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-hRmdqzj6W7WjUm- off=3500553 len=1542629 2021-07-31T02:16:07.676 INFO:tasks.workunit.client.0.smithi045.stdout:op 3321 completed, throughput=5MB/sec 2021-07-31T02:16:07.676 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-uB3WVt9AGIRY-Dd off=5580306 len=1950355 2021-07-31T02:16:07.684 INFO:tasks.workunit.client.0.smithi045.stdout:op 3322 completed, throughput=5MB/sec 2021-07-31T02:16:07.685 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-U_0cSJmqH1gZs4q off=5577905 len=1191589 2021-07-31T02:16:07.690 INFO:tasks.workunit.client.0.smithi045.stdout:op 3323 completed, throughput=5MB/sec 2021-07-31T02:16:08.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:08 smithi045 conmon[39718]: cluster 2021-07-31T02:16:08.205771+0000 mgr.x (mgr.24542 2021-07-31T02:16:08.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:08 smithi045 conmon[39718]: ) 1237 : cluster [DBG] pgmap v1739: 9 pgs: 9 active+clean; 127 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.8 MiB/s wr, 4 op/s 2021-07-31T02:16:08.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:08 smithi045 conmon[33655]: cluster 2021-07-31T02:16:08.205771+0000 mgr.x (mgr.24542) 1237 2021-07-31T02:16:08.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:08 smithi045 conmon[33655]: : cluster [DBG] pgmap v1739: 9 pgs: 9 active+clean; 127 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.8 MiB/s wr, 4 op/s 2021-07-31T02:16:08.691 INFO:tasks.workunit.client.0.smithi045.stdout: 649: throughput=4.99MB/sec pending data=0 2021-07-31T02:16:08.691 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-bv9HfNl7tTDueb7 off=1817568 len=52945 2021-07-31T02:16:08.692 INFO:tasks.workunit.client.0.smithi045.stdout:op 3324 completed, throughput=4.99MB/sec 2021-07-31T02:16:08.692 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-z7j-rx2n5Z3o99z off=8847015 len=1631203 2021-07-31T02:16:08.695 INFO:tasks.workunit.client.0.smithi045.stdout:op 3325 completed, throughput=5MB/sec 2021-07-31T02:16:08.695 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-9NcIBereEyhHb6c off=0 len=1843173 2021-07-31T02:16:08.703 INFO:tasks.workunit.client.0.smithi045.stdout:op 3326 completed, throughput=5MB/sec 2021-07-31T02:16:08.704 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-eXZBb3JrJ8kW_Gf off=11172913 len=806155 2021-07-31T02:16:08.706 INFO:tasks.workunit.client.0.smithi045.stdout:op 3327 completed, throughput=5MB/sec 2021-07-31T02:16:08.706 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Nhjaf8cfrj6GzqU off=10897328 len=1233533 2021-07-31T02:16:08.711 INFO:tasks.workunit.client.0.smithi045.stdout:op 3328 completed, throughput=5MB/sec 2021-07-31T02:16:08.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:08 smithi154 conmon[34553]: cluster 2021-07-31T02:16:08.205771+0000 mgr.x (mgr.24542) 1237 : cluster [DBG] pgmap v1739: 9 pgs: 9 active+clean; 127 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.8 MiB/s wr, 4 op/s 2021-07-31T02:16:09.711 INFO:tasks.workunit.client.0.smithi045.stdout: 650: throughput=4.99MB/sec pending data=0 2021-07-31T02:16:09.711 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-RTu8B-GdPoiYeGb off=9357581 len=106197 2021-07-31T02:16:09.714 INFO:tasks.workunit.client.0.smithi045.stdout:op 3329 completed, throughput=4.99MB/sec 2021-07-31T02:16:09.714 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-PQGHtA02LRCW7SO off=2555132 len=245494 2021-07-31T02:16:09.715 INFO:tasks.workunit.client.0.smithi045.stdout:op 3330 completed, throughput=4.99MB/sec 2021-07-31T02:16:09.715 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Ly-ROr9t-XXqS9M off=16532689 len=1832483 2021-07-31T02:16:09.725 INFO:tasks.workunit.client.0.smithi045.stdout:op 3331 completed, throughput=5MB/sec 2021-07-31T02:16:09.725 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-EUVwGMUjOEkG1gR off=2678907 len=1564197 2021-07-31T02:16:09.726 INFO:tasks.workunit.client.0.smithi045.stdout:op 3332 completed, throughput=5MB/sec 2021-07-31T02:16:09.727 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-h0ZpW2ZkjGPd0iS off=254628 len=691424 2021-07-31T02:16:09.728 INFO:tasks.workunit.client.0.smithi045.stdout:op 3333 completed, throughput=5MB/sec 2021-07-31T02:16:09.729 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-1at_q_u7G_kM1Nt off=2952991 len=429051 2021-07-31T02:16:09.730 INFO:tasks.workunit.client.0.smithi045.stdout:op 3334 completed, throughput=5MB/sec 2021-07-31T02:16:10.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:10 smithi045 conmon[39718]: cluster 2021-07-31T02:16:10.206386+0000 mgr.x (mgr.24542) 1238 : cluster [DBG] pgmap v1740: 9 pgs: 9 active+clean; 127 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 3.7 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2021-07-31T02:16:10.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:10 smithi045 conmon[33655]: cluster 2021-07-31T02:16:10.206386+0000 mgr.x ( 2021-07-31T02:16:10.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:10 smithi045 conmon[33655]: mgr.24542) 1238 : cluster [DBG] pgmap v1740: 9 pgs: 9 active+clean; 127 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 3.7 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2021-07-31T02:16:10.730 INFO:tasks.workunit.client.0.smithi045.stdout: 651: throughput=4.99MB/sec pending data=0 2021-07-31T02:16:10.731 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-sqkv0wPZ09jrgBq off=5485028 len=273731 2021-07-31T02:16:10.733 INFO:tasks.workunit.client.0.smithi045.stdout:op 3335 completed, throughput=4.99MB/sec 2021-07-31T02:16:10.734 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-FbZ5yAuIoN5BuTh off=2302175 len=198590 2021-07-31T02:16:10.737 INFO:tasks.workunit.client.0.smithi045.stdout:op 3336 completed, throughput=4.99MB/sec 2021-07-31T02:16:10.737 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-D1w5D7P6icRR67P off=8587824 len=680657 2021-07-31T02:16:10.739 INFO:tasks.workunit.client.0.smithi045.stdout:op 3337 completed, throughput=4.99MB/sec 2021-07-31T02:16:10.739 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-215gHqQJ5YLDMVT off=5997570 len=761841 2021-07-31T02:16:10.741 INFO:tasks.workunit.client.0.smithi045.stdout:op 3338 completed, throughput=5MB/sec 2021-07-31T02:16:10.741 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-EBlFVj-qpBhGQvO off=485550 len=749720 2021-07-31T02:16:10.745 INFO:tasks.workunit.client.0.smithi045.stdout:op 3339 completed, throughput=5MB/sec 2021-07-31T02:16:10.746 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-pLSrqGOhm4bghE- off=686529 len=429440 2021-07-31T02:16:10.746 INFO:tasks.workunit.client.0.smithi045.stdout:op 3340 completed, throughput=5MB/sec 2021-07-31T02:16:10.746 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ywGFplkzf-CmGBM off=2877907 len=1808917 2021-07-31T02:16:10.748 INFO:tasks.workunit.client.0.smithi045.stdout:op 3341 completed, throughput=5MB/sec 2021-07-31T02:16:10.748 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-az9lH6CiF2pAMl1 off=2722285 len=1717118 2021-07-31T02:16:10.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:10 smithi154 conmon[34553]: cluster 2021-07-31T02:16:10.206386+0000 mgr.x (mgr.24542) 1238 : cluster [DBG] pgmap v1740: 9 pgs: 9 active+clean; 127 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 3.7 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2021-07-31T02:16:10.757 INFO:tasks.workunit.client.0.smithi045.stdout:op 3342 completed, throughput=5MB/sec 2021-07-31T02:16:11.757 INFO:tasks.workunit.client.0.smithi045.stdout: 652: throughput=4.99MB/sec pending data=0 2021-07-31T02:16:11.758 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-TTrOmBZb85tb_mK off=13929042 len=114409 2021-07-31T02:16:11.758 INFO:tasks.workunit.client.0.smithi045.stdout:op 3343 completed, throughput=4.99MB/sec 2021-07-31T02:16:11.759 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-nhYpFBPexAIrfEi off=19865523 len=319682 2021-07-31T02:16:11.762 INFO:tasks.workunit.client.0.smithi045.stdout:op 3344 completed, throughput=5MB/sec 2021-07-31T02:16:11.762 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-kGljQU4a5KoBuPn off=1108169 len=430909 2021-07-31T02:16:11.765 INFO:tasks.workunit.client.0.smithi045.stdout:op 3345 completed, throughput=5MB/sec 2021-07-31T02:16:11.766 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ExPRInMq58CNXX2 off=7073429 len=1422951 2021-07-31T02:16:11.768 INFO:tasks.workunit.client.0.smithi045.stdout:op 3346 completed, throughput=5MB/sec 2021-07-31T02:16:11.768 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-XnnIGNjU-GbKQRi off=5517159 len=804255 2021-07-31T02:16:11.769 INFO:tasks.workunit.client.0.smithi045.stdout:op 3347 completed, throughput=5MB/sec 2021-07-31T02:16:11.770 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Yf7mUI0JWzeU49g off=887009 len=1304466 2021-07-31T02:16:11.776 INFO:tasks.workunit.client.0.smithi045.stdout:op 3348 completed, throughput=5MB/sec 2021-07-31T02:16:12.776 INFO:tasks.workunit.client.0.smithi045.stdout: 653: throughput=4.99MB/sec pending data=0 2021-07-31T02:16:12.776 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-p11oQ9sL4v_b89h off=5909171 len=1928227 2021-07-31T02:16:12.779 INFO:tasks.workunit.client.0.smithi045.stdout:op 3349 completed, throughput=5MB/sec 2021-07-31T02:16:12.780 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-kbNY1wh_zP4efkx off=3210900 len=1607610 2021-07-31T02:16:12.782 INFO:tasks.workunit.client.0.smithi045.stdout:op 3350 completed, throughput=5MB/sec 2021-07-31T02:16:12.783 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-DNKz6CHc9prdkb1 off=11744718 len=1964509 2021-07-31T02:16:12.794 INFO:tasks.workunit.client.0.smithi045.stdout:op 3351 completed, throughput=5MB/sec 2021-07-31T02:16:13.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:13 smithi154 conmon[34553]: cluster 2021-07-31T02:16:12.206832+0000 mgr.x (mgr.24542) 1239 : cluster [DBG] pgmap v1741: 9 pgs: 9 active+clean; 127 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 3 op/s 2021-07-31T02:16:13.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:13 smithi045 conmon[39718]: cluster 2021-07-31T02:16:12.206832+0000 mgr.x 2021-07-31T02:16:13.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:13 smithi045 conmon[39718]: (mgr.24542) 1239 : cluster [DBG] pgmap v1741: 9 pgs: 9 active+clean; 127 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 3 op/s 2021-07-31T02:16:13.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:13 smithi045 conmon[33655]: cluster 2021-07-31T02:16:12.206832+0000 mgr.x (mgr.24542 2021-07-31T02:16:13.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:13 smithi045 conmon[33655]: ) 1239 : cluster [DBG] pgmap v1741: 9 pgs: 9 active+clean; 127 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 3 op/s 2021-07-31T02:16:13.794 INFO:tasks.workunit.client.0.smithi045.stdout: 654: throughput=4.99MB/sec pending data=0 2021-07-31T02:16:13.794 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-nBBu-KNaRXhOLpI off=15362521 len=1887888 2021-07-31T02:16:13.797 INFO:tasks.workunit.client.0.smithi045.stdout:op 3352 completed, throughput=5MB/sec 2021-07-31T02:16:13.798 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-K7a5AcIEtCuIAew off=478859 len=1016727 2021-07-31T02:16:13.799 INFO:tasks.workunit.client.0.smithi045.stdout:op 3353 completed, throughput=5MB/sec 2021-07-31T02:16:13.799 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-gknhFYQ6V1AakWp off=766733 len=999513 2021-07-31T02:16:13.800 INFO:tasks.workunit.client.0.smithi045.stdout:op 3354 completed, throughput=5MB/sec 2021-07-31T02:16:13.801 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-C9PD4_BxG7F4OOc off=2517458 len=816289 2021-07-31T02:16:13.806 INFO:tasks.workunit.client.0.smithi045.stdout:op 3355 completed, throughput=5MB/sec 2021-07-31T02:16:14.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:16:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:16:13.558Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:16:14.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:16:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:16:13.558Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:16:14.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:16:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:16:13.558Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:16:14.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:14 smithi045 conmon[33655]: cluster 2021-07-31T02:16:14.207250+0000 mgr.x (mgr.24542) 1240 : cluster [DBG] pgmap v1742: 9 pgs: 9 active+clean; 127 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 3.5 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2021-07-31T02:16:14.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:14 smithi045 conmon[39718]: cluster 2021-07-31T02:16:14.207250+0000 mgr.x (mgr 2021-07-31T02:16:14.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:14 smithi045 conmon[39718]: .24542) 1240 : cluster [DBG] pgmap v1742: 9 pgs: 9 active+clean; 127 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 3.5 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2021-07-31T02:16:14.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:14 smithi154 conmon[34553]: cluster 2021-07-31T02:16:14.207250+0000 mgr.x (mgr.24542) 1240 : cluster 2021-07-31T02:16:14.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:14 smithi154 conmon[34553]: [DBG] pgmap v1742: 9 pgs: 9 active+clean; 127 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 3.5 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2021-07-31T02:16:14.806 INFO:tasks.workunit.client.0.smithi045.stdout: 655: throughput=4.99MB/sec pending data=0 2021-07-31T02:16:14.807 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-r0IdbCDyCMIkHRb off=665459 len=851284 2021-07-31T02:16:14.812 INFO:tasks.workunit.client.0.smithi045.stdout:op 3356 completed, throughput=4.99MB/sec 2021-07-31T02:16:14.812 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-NL_RPm8lXChx85S off=4300452 len=277736 2021-07-31T02:16:14.815 INFO:tasks.workunit.client.0.smithi045.stdout:op 3357 completed, throughput=4.99MB/sec 2021-07-31T02:16:14.815 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-QqfSULE37zoSrCT off=1559169 len=1606916 2021-07-31T02:16:14.823 INFO:tasks.workunit.client.0.smithi045.stdout:op 3358 completed, throughput=5MB/sec 2021-07-31T02:16:14.824 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-EL9peavCPOBu4Rt off=1709926 len=1064408 2021-07-31T02:16:14.826 INFO:tasks.workunit.client.0.smithi045.stdout:op 3359 completed, throughput=5MB/sec 2021-07-31T02:16:14.826 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-lVAcFkQzMwUexyB off=3111557 len=1379434 2021-07-31T02:16:14.833 INFO:tasks.workunit.client.0.smithi045.stdout:op 3360 completed, throughput=5MB/sec 2021-07-31T02:16:15.833 INFO:tasks.workunit.client.0.smithi045.stdout: 656: throughput=4.99MB/sec pending data=0 2021-07-31T02:16:15.834 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-CcHkewqpRHf8QUH off=1952694 len=1296511 2021-07-31T02:16:15.840 INFO:tasks.workunit.client.0.smithi045.stdout:op 3361 completed, throughput=4.99MB/sec 2021-07-31T02:16:15.840 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-zEIIW5Q096q4862 off=12273684 len=671686 2021-07-31T02:16:15.841 INFO:tasks.workunit.client.0.smithi045.stdout:op 3362 completed, throughput=5MB/sec 2021-07-31T02:16:15.842 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-PkI918PhCOjvpxB off=18100818 len=740702 2021-07-31T02:16:15.846 INFO:tasks.workunit.client.0.smithi045.stdout:op 3363 completed, throughput=5MB/sec 2021-07-31T02:16:15.847 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-KGn5c63ovJnwSRS off=3110549 len=580520 2021-07-31T02:16:15.850 INFO:tasks.workunit.client.0.smithi045.stdout:op 3364 completed, throughput=5MB/sec 2021-07-31T02:16:15.851 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-kR2tU1G_jve2rsx off=5233138 len=232958 2021-07-31T02:16:15.852 INFO:tasks.workunit.client.0.smithi045.stdout:op 3365 completed, throughput=5MB/sec 2021-07-31T02:16:15.852 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-YC1IzgobXyNi_DB off=758769 len=764601 2021-07-31T02:16:15.854 INFO:tasks.workunit.client.0.smithi045.stdout:op 3366 completed, throughput=5MB/sec 2021-07-31T02:16:15.854 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-KV86JuE4I_AB6Dw off=221253 len=1224225 2021-07-31T02:16:15.858 INFO:tasks.workunit.client.0.smithi045.stdout:op 3367 completed, throughput=5MB/sec 2021-07-31T02:16:16.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:16 smithi045 conmon[39718]: cluster 2021-07-31T02:16:16.207725+0000 mgr.x (mgr 2021-07-31T02:16:16.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:16 smithi045 conmon[39718]: .24542) 1241 : cluster [DBG] pgmap v1743: 9 pgs: 9 active+clean; 127 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.7 MiB/s wr, 4 op/s 2021-07-31T02:16:16.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:16 smithi045 conmon[33655]: cluster 2021-07-31T02:16:16.207725+0000 mgr.x (mgr.24542) 1241 : cluster [DBG] 2021-07-31T02:16:16.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:16 smithi045 conmon[33655]: pgmap v1743: 9 pgs: 9 active+clean; 127 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.7 MiB/s wr, 4 op/s 2021-07-31T02:16:16.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:16 smithi154 conmon[34553]: cluster 2021-07-31T02:16:16.207725+0000 mgr.x (mgr.24542) 1241 : cluster [DBG] pgmap v1743: 9 pgs: 9 active+clean; 127 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.7 MiB/s wr, 4 op/s 2021-07-31T02:16:16.859 INFO:tasks.workunit.client.0.smithi045.stdout: 657: throughput=4.99MB/sec pending data=0 2021-07-31T02:16:16.894 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-tcAT-1oDnv-06sY off=4143162 len=1174634 2021-07-31T02:16:16.896 INFO:tasks.workunit.client.0.smithi045.stdout:op 3368 completed, throughput=4.99MB/sec 2021-07-31T02:16:16.896 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-8NXpnFpsYa458Nd off=2442253 len=1193931 2021-07-31T02:16:16.896 INFO:tasks.workunit.client.0.smithi045.stdout:op 3369 completed, throughput=5MB/sec 2021-07-31T02:16:16.896 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-lO7xLuGK2qAjJY4 off=670628 len=813845 2021-07-31T02:16:16.897 INFO:tasks.workunit.client.0.smithi045.stdout:op 3370 completed, throughput=5MB/sec 2021-07-31T02:16:16.897 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-F3H_kKxa_FqGiEB off=17037853 len=1703862 2021-07-31T02:16:16.897 INFO:tasks.workunit.client.0.smithi045.stdout:op 3371 completed, throughput=5MB/sec 2021-07-31T02:16:16.898 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-yF3XEpoOz-C91Er off=12775078 len=435315 2021-07-31T02:16:16.898 INFO:tasks.workunit.client.0.smithi045.stdout:op 3372 completed, throughput=5MB/sec 2021-07-31T02:16:17.883 INFO:tasks.workunit.client.0.smithi045.stdout: 658: throughput=4.99MB/sec pending data=0 2021-07-31T02:16:17.883 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-j5NaeFcYG_WoVQp off=6711671 len=180682 2021-07-31T02:16:17.886 INFO:tasks.workunit.client.0.smithi045.stdout:op 3373 completed, throughput=4.99MB/sec 2021-07-31T02:16:17.886 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-aGKZ5huif2sVCKd off=985830 len=1756304 2021-07-31T02:16:17.890 INFO:tasks.workunit.client.0.smithi045.stdout:op 3374 completed, throughput=5MB/sec 2021-07-31T02:16:17.890 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-iqA5C0W_YsSb6Xd off=10636131 len=1531989 2021-07-31T02:16:17.891 INFO:tasks.workunit.client.0.smithi045.stdout:op 3375 completed, throughput=5MB/sec 2021-07-31T02:16:17.892 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-SPt3H_TLkT2gfDC off=6061190 len=1922320 2021-07-31T02:16:17.900 INFO:tasks.workunit.client.0.smithi045.stdout:op 3376 completed, throughput=5MB/sec 2021-07-31T02:16:18.451 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:16:18.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:18 smithi045 conmon[39718]: cluster 2021-07-31T02:16:18.208213+0000 mgr.x (mgr. 2021-07-31T02:16:18.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:18 smithi045 conmon[39718]: 24542) 1242 : cluster [DBG] pgmap v1744: 9 pgs: 9 active+clean; 127 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.7 MiB/s wr, 4 op/s 2021-07-31T02:16:18.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:18 smithi045 conmon[33655]: cluster 2021-07-31T02:16:18.208213+0000 mgr.x (mgr.24542) 1242 : cluster 2021-07-31T02:16:18.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:18 smithi045 conmon[33655]: [DBG] pgmap v1744: 9 pgs: 9 active+clean; 127 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.7 MiB/s wr, 4 op/s 2021-07-31T02:16:18.730 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:16:18.730 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 31m ago 31m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:16:18.731 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 28m ago 31m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:16:18.731 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 28m ago 35m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:16:18.731 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 31m ago 37m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:16:18.731 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 31m ago 37m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:16:18.732 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 28m ago 35m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:16:18.732 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 31m ago 35m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:16:18.732 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 31m ago 32m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:16:18.732 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 28m ago 32m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:16:18.733 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 31m ago 34m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:16:18.733 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 31m ago 34m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:16:18.733 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 31m ago 34m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:16:18.733 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 31m ago 33m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:16:18.734 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 28m ago 33m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:16:18.734 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 28m ago 33m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:16:18.734 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 28m ago 32m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:16:18.734 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 28m ago 32m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:16:18.735 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 28m ago 32m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:16:18.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:18 smithi154 conmon[34553]: cluster 2021-07-31T02:16:18.208213+0000 mgr.x (mgr.24542) 1242 : cluster [DBG] pgmap v1744: 9 pgs: 9 active+clean; 127 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.7 MiB/s wr, 4 op/s 2021-07-31T02:16:18.901 INFO:tasks.workunit.client.0.smithi045.stdout: 659: throughput=4.99MB/sec pending data=0 2021-07-31T02:16:18.901 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj--diP7rzoe1Vys-z off=8071755 len=1327964 2021-07-31T02:16:18.906 INFO:tasks.workunit.client.0.smithi045.stdout:op 3377 completed, throughput=4.99MB/sec 2021-07-31T02:16:18.906 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-bFDKz6W-jHj17mg off=2224165 len=429179 2021-07-31T02:16:18.908 INFO:tasks.workunit.client.0.smithi045.stdout:op 3378 completed, throughput=5MB/sec 2021-07-31T02:16:18.908 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-58YciV6DIb-gH0Z off=2141546 len=1396047 2021-07-31T02:16:18.915 INFO:tasks.workunit.client.0.smithi045.stdout:op 3379 completed, throughput=5MB/sec 2021-07-31T02:16:18.916 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-AePwuZqIXORN5dF off=6891833 len=1735264 2021-07-31T02:16:18.919 INFO:tasks.workunit.client.0.smithi045.stdout:op 3380 completed, throughput=5MB/sec 2021-07-31T02:16:18.919 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-cU0Bl4nmVNYHWLd off=1237384 len=1843779 2021-07-31T02:16:18.929 INFO:tasks.workunit.client.0.smithi045.stdout:op 3381 completed, throughput=5MB/sec 2021-07-31T02:16:19.075 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:16:19.075 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:16:19.075 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:16:19.075 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:16:19.076 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:16:19.076 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:16:19.076 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:16:19.076 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:16:19.076 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:16:19.077 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:16:19.077 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:16:19.077 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:16:19.077 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:16:19.078 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:16:19.078 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:16:19.078 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:16:19.078 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:16:19.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:19 smithi045 conmon[33655]: audit 2021-07-31T02:16:18.438760+0000 mgr.x (mgr 2021-07-31T02:16:19.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:19 smithi045 conmon[33655]: .24542) 1243 : audit [DBG] from='client.16113 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:16:19.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:19 smithi045 conmon[33655]: audit 2021-07-31T02:16:18.722822+0000 mgr.x (mgr.24542) 2021-07-31T02:16:19.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:19 smithi045 conmon[33655]: 1244 : audit [DBG] from='client.16119 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:16:19.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:19 smithi045 conmon[33655]: audit 2021-07-31T02:16:19.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:19 smithi045 conmon[33655]: 2021-07-31T02:16:19.075079+0000 mon.c ( 2021-07-31T02:16:19.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:19 smithi045 conmon[33655]: mon.1) 85 : audit [DBG] from='client.? 172.21.15.45:0/2116002133' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:16:19.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:19 smithi045 conmon[39718]: audit 2021-07-31T02:16:18 2021-07-31T02:16:19.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:19 smithi045 conmon[39718]: .438760+0000 mgr.x (mgr.24542) 1243 : audit [DBG] from='client.16113 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:16:19.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:19 smithi045 conmon[39718]: audit 2021-07-31T02:16:19.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:19 smithi045 conmon[39718]: 2021-07-31T02:16:18.722822 2021-07-31T02:16:19.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:19 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 1244 : audit 2021-07-31T02:16:19.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:19 smithi045 conmon[39718]: [DBG] from='client.16119 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:16:19.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:19 smithi045 conmon[39718]: audit 2021-07-31T02:16:19.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:19 smithi045 conmon[39718]: 2021-07-31T02 2021-07-31T02:16:19.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:19 smithi045 conmon[39718]: :16:19.075079+0000 mon.c ( 2021-07-31T02:16:19.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:19 smithi045 conmon[39718]: mon.1) 85 : audit [DBG] from='client.? 172.21.15.45:0/2116002133' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:16:19.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:19 smithi045 conmon[39718]: 2021-07-31T02:16:19.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:19 smithi154 conmon[34553]: audit 2021-07-31T02:16:18.438760+0000 mgr.x (mgr.24542) 1243 : audit [DBG] from='client.16113 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:16:19.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:19 smithi154 conmon[34553]: audit 2021-07-31T02:16:18.722822+0000 mgr.x (mgr.24542) 1244 : audit [DBG] from='client.16119 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:16:19.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:19 smithi154 conmon[34553]: audit 2021-07-31T02:16:19.075079+0000 mon.c (mon.1) 85 : audit [DBG] from='client.? 172.21.15.45:0/2116002133' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:16:19.930 INFO:tasks.workunit.client.0.smithi045.stdout: 660: throughput=4.99MB/sec pending data=0 2021-07-31T02:16:19.930 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-3YrFC7wKl8JmHFL off=12056263 len=1079554 2021-07-31T02:16:19.936 INFO:tasks.workunit.client.0.smithi045.stdout:op 3382 completed, throughput=5MB/sec 2021-07-31T02:16:19.936 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-2qwYeQx084owTdt off=10390801 len=832370 2021-07-31T02:16:19.938 INFO:tasks.workunit.client.0.smithi045.stdout:op 3383 completed, throughput=5MB/sec 2021-07-31T02:16:19.938 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-kyvrWB08w5BJDTI off=13432327 len=206280 2021-07-31T02:16:19.941 INFO:tasks.workunit.client.0.smithi045.stdout:op 3384 completed, throughput=5MB/sec 2021-07-31T02:16:19.941 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Fa_6DUfSax93QRR off=706768 len=1138963 2021-07-31T02:16:19.944 INFO:tasks.workunit.client.0.smithi045.stdout:op 3385 completed, throughput=5MB/sec 2021-07-31T02:16:19.944 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-lAMhYyD7IoN9NtI off=60383 len=804229 2021-07-31T02:16:19.945 INFO:tasks.workunit.client.0.smithi045.stdout:op 3386 completed, throughput=5MB/sec 2021-07-31T02:16:20.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:20 smithi045 conmon[33655]: cluster 2021-07-31T02:16:20.208732+0000 mgr.x (mgr.24542) 2021-07-31T02:16:20.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:20 smithi045 conmon[33655]: 1245 : cluster [DBG] pgmap v1745: 9 pgs: 9 active+clean; 127 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 3.3 MiB/s rd, 3.1 MiB/s wr, 6 op/s 2021-07-31T02:16:20.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:20 smithi045 conmon[39718]: cluster 2021-07-31T02:16:20.208732+0000 2021-07-31T02:16:20.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:20 smithi045 conmon[39718]: mgr.x (mgr.24542) 1245 : cluster [DBG] pgmap v1745: 9 pgs: 9 active+clean; 127 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 3.3 MiB/s rd, 3.1 MiB/s wr, 6 op/s 2021-07-31T02:16:20.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:20 smithi154 conmon[34553]: cluster 2021-07-31T02:16:20.208732+0000 mgr.x (mgr.24542) 1245 : cluster [DBG] pgmap v1745: 9 pgs: 9 active+clean; 127 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 3.3 MiB/s rd, 3.1 MiB/s wr, 6 op/s 2021-07-31T02:16:20.945 INFO:tasks.workunit.client.0.smithi045.stdout: 661: throughput=4.99MB/sec pending data=0 2021-07-31T02:16:20.945 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-L1T9zZGhgNB9d73 off=4571304 len=398683 2021-07-31T02:16:20.949 INFO:tasks.workunit.client.0.smithi045.stdout:op 3387 completed, throughput=4.99MB/sec 2021-07-31T02:16:20.949 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-LtX2zwca7RA_VVu off=9457622 len=537177 2021-07-31T02:16:20.953 INFO:tasks.workunit.client.0.smithi045.stdout:op 3388 completed, throughput=4.99MB/sec 2021-07-31T02:16:20.953 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-oESOrMbmbKwARbk off=20020309 len=1684135 2021-07-31T02:16:20.956 INFO:tasks.workunit.client.0.smithi045.stdout:op 3389 completed, throughput=5MB/sec 2021-07-31T02:16:20.956 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-YRZ5vHe2MuiRgwr off=9374682 len=141477 2021-07-31T02:16:20.959 INFO:tasks.workunit.client.0.smithi045.stdout:op 3390 completed, throughput=5MB/sec 2021-07-31T02:16:20.959 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-yyEP5zAZTbiijsJ off=887922 len=1413528 2021-07-31T02:16:20.967 INFO:tasks.workunit.client.0.smithi045.stdout:op 3391 completed, throughput=5MB/sec 2021-07-31T02:16:20.967 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-zYrS8vRhcT1MLTZ off=2108834 len=887438 2021-07-31T02:16:20.970 INFO:tasks.workunit.client.0.smithi045.stdout:op 3392 completed, throughput=5MB/sec 2021-07-31T02:16:20.970 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-LWPwSc20ktIonot off=3830511 len=1089371 2021-07-31T02:16:20.971 INFO:tasks.workunit.client.0.smithi045.stdout:op 3393 completed, throughput=5MB/sec 2021-07-31T02:16:21.972 INFO:tasks.workunit.client.0.smithi045.stdout: 662: throughput=4.99MB/sec pending data=0 2021-07-31T02:16:21.972 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-KC7FtRNGHuzHYdV off=5254707 len=1622013 2021-07-31T02:16:21.975 INFO:tasks.workunit.client.0.smithi045.stdout:op 3394 completed, throughput=5MB/sec 2021-07-31T02:16:21.976 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-q1IZ_aD0WumL3Dg off=995132 len=1267038 2021-07-31T02:16:21.981 INFO:tasks.workunit.client.0.smithi045.stdout:op 3395 completed, throughput=5MB/sec 2021-07-31T02:16:21.982 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-fZqiYVnEh683Rm1 off=14495019 len=441485 2021-07-31T02:16:21.983 INFO:tasks.workunit.client.0.smithi045.stdout:op 3396 completed, throughput=5MB/sec 2021-07-31T02:16:21.983 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-qeCzHXeiPFwJmcG off=21415170 len=1676293 2021-07-31T02:16:21.984 INFO:tasks.workunit.client.0.smithi045.stdout:op 3397 completed, throughput=5MB/sec 2021-07-31T02:16:22.984 INFO:tasks.workunit.client.0.smithi045.stdout: 663: throughput=4.99MB/sec pending data=0 2021-07-31T02:16:22.985 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-20oyDr0j61jGL03 off=1338429 len=861254 2021-07-31T02:16:22.991 INFO:tasks.workunit.client.0.smithi045.stdout:op 3398 completed, throughput=4.99MB/sec 2021-07-31T02:16:22.991 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ZEqb7CddQFqVxqf off=868015 len=123313 2021-07-31T02:16:22.992 INFO:tasks.workunit.client.0.smithi045.stdout:op 3399 completed, throughput=4.99MB/sec 2021-07-31T02:16:22.993 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-J6K3BjSPdYDpIfj off=2690467 len=303421 2021-07-31T02:16:22.994 INFO:tasks.workunit.client.0.smithi045.stdout:op 3400 completed, throughput=5MB/sec 2021-07-31T02:16:22.994 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-wPbHL68RbBUte6V off=1216758 len=106720 2021-07-31T02:16:22.994 INFO:tasks.workunit.client.0.smithi045.stdout:op 3401 completed, throughput=5MB/sec 2021-07-31T02:16:22.995 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ZHLMXcyYMO42gYs off=11396488 len=883817 2021-07-31T02:16:22.996 INFO:tasks.workunit.client.0.smithi045.stdout:op 3402 completed, throughput=5MB/sec 2021-07-31T02:16:22.996 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-zfYRDAYOeWZ0f5C off=8517599 len=1047489 2021-07-31T02:16:22.998 INFO:tasks.workunit.client.0.smithi045.stdout:op 3403 completed, throughput=5MB/sec 2021-07-31T02:16:22.998 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-GZbGzRwQPAE_dwo off=3479828 len=1749457 2021-07-31T02:16:23.009 INFO:tasks.workunit.client.0.smithi045.stdout:op 3404 completed, throughput=5MB/sec 2021-07-31T02:16:23.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:23 smithi154 conmon[34553]: cluster 2021-07-31T02:16:22.209226+0000 mgr.x (mgr.24542) 1246 : cluster [DBG] pgmap v1746: 9 pgs: 9 active+clean; 127 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2021-07-31T02:16:23.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:23 smithi154 conmon[34553]: 2021-07-31T02:16:23.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:23 smithi045 conmon[33655]: cluster 2021-07-31T02:16:22.209226+0000 mgr.x (mgr.24542) 1246 : cluster [DBG] pgmap v1746: 9 pgs: 9 active+clean; 127 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2021-07-31T02:16:23.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:23 smithi045 conmon[39718]: cluster 2021-07-31T02:16:22. 2021-07-31T02:16:23.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:23 smithi045 conmon[39718]: 209226+0000 mgr.x (mgr.24542) 1246 : cluster [DBG] pgmap v1746: 9 pgs: 9 active+clean; 127 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2021-07-31T02:16:24.010 INFO:tasks.workunit.client.0.smithi045.stdout: 664: throughput=4.99MB/sec pending data=0 2021-07-31T02:16:24.010 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-cV-Yq-KmUZBXuVY off=5295321 len=1786509 2021-07-31T02:16:24.012 INFO:tasks.workunit.client.0.smithi045.stdout:op 3405 completed, throughput=5MB/sec 2021-07-31T02:16:24.012 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-swaarWFpPEMPWAH off=2456057 len=275211 2021-07-31T02:16:24.013 INFO:tasks.workunit.client.0.smithi045.stdout:op 3406 completed, throughput=5MB/sec 2021-07-31T02:16:24.013 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-wgMkYuz1KBcyCsG off=5537532 len=665506 2021-07-31T02:16:24.018 INFO:tasks.workunit.client.0.smithi045.stdout:op 3407 completed, throughput=5MB/sec 2021-07-31T02:16:24.018 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-lTEJeTEYiOUPt8g off=13342121 len=695665 2021-07-31T02:16:24.019 INFO:tasks.workunit.client.0.smithi045.stdout:op 3408 completed, throughput=5MB/sec 2021-07-31T02:16:24.019 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Jlwqe4rpaWVvlb- off=17831971 len=1606900 2021-07-31T02:16:24.020 INFO:tasks.workunit.client.0.smithi045.stdout:op 3409 completed, throughput=5MB/sec 2021-07-31T02:16:24.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:16:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:16:23.558Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:16:24.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:16:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:16:23.558Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:16:24.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:16:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:16:23.558Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:16:24.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:24 smithi045 conmon[39718]: cluster 2021-07-31T02:16:24.209789+0000 mgr.x 2021-07-31T02:16:24.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:24 smithi045 conmon[39718]: (mgr.24542) 1247 : cluster [DBG] pgmap v1747: 9 pgs: 9 active+clean; 127 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.8 MiB/s wr, 6 op/s 2021-07-31T02:16:24.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:24 smithi045 conmon[33655]: cluster 2021-07-31T02:16:24.209789+0000 mgr.x (mgr.24542) 1247 : 2021-07-31T02:16:24.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:24 smithi045 conmon[33655]: cluster [DBG] pgmap v1747: 9 pgs: 9 active+clean; 127 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.8 MiB/s wr, 6 op/s 2021-07-31T02:16:24.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:24 smithi154 conmon[34553]: cluster 2021-07-31T02:16:24.209789+0000 mgr.x (mgr.24542) 1247 : cluster [DBG] pgmap v1747: 9 pgs: 9 active+clean; 127 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.8 MiB/s wr, 6 op/s 2021-07-31T02:16:25.021 INFO:tasks.workunit.client.0.smithi045.stdout: 665: throughput=4.99MB/sec pending data=0 2021-07-31T02:16:25.021 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-tflTZLF9Lz6_ADL off=4469240 len=1825939 2021-07-31T02:16:25.024 INFO:tasks.workunit.client.0.smithi045.stdout:op 3410 completed, throughput=5MB/sec 2021-07-31T02:16:25.024 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj--sXxAvJGAgZrG3b off=1307655 len=681492 2021-07-31T02:16:25.026 INFO:tasks.workunit.client.0.smithi045.stdout:op 3411 completed, throughput=5MB/sec 2021-07-31T02:16:25.026 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-6CcULOglucqOhnU off=10595476 len=1416093 2021-07-31T02:16:25.034 INFO:tasks.workunit.client.0.smithi045.stdout:op 3412 completed, throughput=5MB/sec 2021-07-31T02:16:25.035 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-JhePelO8enTbaV8 off=1542012 len=1930426 2021-07-31T02:16:25.041 INFO:tasks.workunit.client.0.smithi045.stdout:op 3413 completed, throughput=5MB/sec 2021-07-31T02:16:26.042 INFO:tasks.workunit.client.0.smithi045.stdout: 666: throughput=4.99MB/sec pending data=0 2021-07-31T02:16:26.042 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-xFkSMj7npiVR3Yb off=5189452 len=1943650 2021-07-31T02:16:26.045 INFO:tasks.workunit.client.0.smithi045.stdout:op 3414 completed, throughput=5MB/sec 2021-07-31T02:16:26.046 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-56rdzlhlKWdXd1X off=303497 len=1141382 2021-07-31T02:16:26.047 INFO:tasks.workunit.client.0.smithi045.stdout:op 3415 completed, throughput=5MB/sec 2021-07-31T02:16:26.047 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-hol8-mdrlChX6Yo off=18438762 len=1749910 2021-07-31T02:16:26.050 INFO:tasks.workunit.client.0.smithi045.stdout:op 3416 completed, throughput=5MB/sec 2021-07-31T02:16:26.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:26 smithi045 conmon[33655]: cluster 2021-07-31T02:16:26.210170+0000 mgr.x 2021-07-31T02:16:26.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:26 smithi045 conmon[33655]: (mgr.24542) 1248 : cluster [DBG] pgmap v1748: 9 pgs: 9 active+clean; 127 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2021-07-31T02:16:26.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:26 smithi045 conmon[39718]: cluster 2021-07-31T02:16:26.210170+0000 mgr.x ( 2021-07-31T02:16:26.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:26 smithi045 conmon[39718]: mgr.24542) 1248 : cluster [DBG] pgmap v1748: 9 pgs: 9 active+clean; 127 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2021-07-31T02:16:26.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:26 smithi154 conmon[34553]: cluster 2021-07-31T02:16:26.210170+0000 mgr.x (mgr.24542) 1248 : cluster [DBG] pgmap v1748: 9 pgs: 9 active+clean; 127 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2021-07-31T02:16:27.051 INFO:tasks.workunit.client.0.smithi045.stdout: 667: throughput=4.99MB/sec pending data=0 2021-07-31T02:16:27.052 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-lcSmzX8iW33oZmi off=12504349 len=1814649 2021-07-31T02:16:27.053 INFO:tasks.workunit.client.0.smithi045.stdout:op 3417 completed, throughput=5MB/sec 2021-07-31T02:16:27.053 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-CNhCkBdys-upuNw off=4735028 len=1971696 2021-07-31T02:16:27.063 INFO:tasks.workunit.client.0.smithi045.stdout:op 3418 completed, throughput=5MB/sec 2021-07-31T02:16:27.063 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-aVKAWUsZVAKyTwe off=12721076 len=1068484 2021-07-31T02:16:27.065 INFO:tasks.workunit.client.0.smithi045.stdout:op 3419 completed, throughput=5MB/sec 2021-07-31T02:16:27.065 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Lb8vFgjipV5oteL off=7589103 len=1618711 2021-07-31T02:16:27.066 INFO:tasks.workunit.client.0.smithi045.stdout:op 3420 completed, throughput=5MB/sec 2021-07-31T02:16:28.067 INFO:tasks.workunit.client.0.smithi045.stdout: 668: throughput=4.99MB/sec pending data=0 2021-07-31T02:16:28.067 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-EZDhq9qWhX8Sltb off=4570127 len=385890 2021-07-31T02:16:28.068 INFO:tasks.workunit.client.0.smithi045.stdout:op 3421 completed, throughput=5MB/sec 2021-07-31T02:16:28.068 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-KgMO3sJqezBUojZ off=13385586 len=1569594 2021-07-31T02:16:28.075 INFO:tasks.workunit.client.0.smithi045.stdout:op 3422 completed, throughput=5MB/sec 2021-07-31T02:16:28.075 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-RmVTngt5Xapef8x off=7299986 len=1052615 2021-07-31T02:16:28.081 INFO:tasks.workunit.client.0.smithi045.stdout:op 3423 completed, throughput=5MB/sec 2021-07-31T02:16:28.082 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-jsMqGcAcM6_rjIC off=5573192 len=224744 2021-07-31T02:16:28.082 INFO:tasks.workunit.client.0.smithi045.stdout:op 3424 completed, throughput=5MB/sec 2021-07-31T02:16:28.083 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Ig0R_4-W358BmSs off=6433095 len=1704285 2021-07-31T02:16:28.088 INFO:tasks.workunit.client.0.smithi045.stdout:op 3425 completed, throughput=5MB/sec 2021-07-31T02:16:28.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:28 smithi154 conmon[34553]: audit 2021-07-31T02:16:28.035018+0000 mon.b (mon.2) 213 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:16:28.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:28 smithi154 conmon[34553]: audit 2021-07-31T02:16:28.035716+0000 mon.a (mon.0) 1884 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:16:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:28 smithi154 conmon[34553]: audit 2021-07-31T02:16:28.056787+0000 mon.b (mon.2) 214 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:16:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:28 smithi154 conmon[34553]: audit 2021- 2021-07-31T02:16:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:28 smithi154 conmon[34553]: 07-31T02:16:28.057364 2021-07-31T02:16:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:28 smithi154 conmon[34553]: +0000 mon.a (mon.0) 1885 : 2021-07-31T02:16:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:28 smithi154 conmon[34553]: audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:16:28.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:28 smithi045 conmon[33655]: audit 2021-07-31T02:16:28.035018+0000 mon.b (mon.2) 213 2021-07-31T02:16:28.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:28 smithi045 conmon[33655]: : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:16:28.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:28 smithi045 conmon[33655]: audit 2021-07-31T02:16:28.035716+0000 mon.a (mon.0) 1884 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:16:28.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:28 smithi045 conmon[33655]: audit 2021-07-31T02:16:28.056787+0000 2021-07-31T02:16:28.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:28 smithi045 conmon[33655]: mon.b (mon.2) 214 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:16:28.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:28 smithi045 conmon[33655]: audit 2021-07-31T02:16 2021-07-31T02:16:28.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:28 smithi045 conmon[33655]: :28.057364+0000 mon.a (mon.0) 1885 : 2021-07-31T02:16:28.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:28 smithi045 conmon[33655]: audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:16:28.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:28 smithi045 conmon[39718]: audit 2021-07-31T02:16:28.035018+0000 2021-07-31T02:16:28.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:28 smithi045 conmon[39718]: mon.b (mon.2) 213 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:16:28.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:28 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T02:16:28.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:28 smithi045 conmon[39718]: T02:16:28.035716+0000 mon.a (mon.0) 1884 : audit 2021-07-31T02:16:28.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:28 smithi045 conmon[39718]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:16:28.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:28 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T02:16:28.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:28 smithi045 conmon[39718]: T02:16:28.056787+0000 mon.b 2021-07-31T02:16:28.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:28 smithi045 conmon[39718]: (mon.2) 214 : audit [INF] 2021-07-31T02:16:28.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:28 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:16:28.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:28 smithi045 conmon[39718]: audit 2021-07-31T02 2021-07-31T02:16:28.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:28 smithi045 conmon[39718]: :16:28.057364+0000 mon.a (mon. 2021-07-31T02:16:28.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:28 smithi045 conmon[39718]: 0) 1885 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:16:29.089 INFO:tasks.workunit.client.0.smithi045.stdout: 669: throughput=4.99MB/sec pending data=0 2021-07-31T02:16:29.089 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-zNQi0uIR-9L1d8- off=4123333 len=1329317 2021-07-31T02:16:29.095 INFO:tasks.workunit.client.0.smithi045.stdout:op 3426 completed, throughput=5MB/sec 2021-07-31T02:16:29.095 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ym-6Gj3VhsbpnoA off=9330490 len=46805 2021-07-31T02:16:29.098 INFO:tasks.workunit.client.0.smithi045.stdout:op 3427 completed, throughput=5MB/sec 2021-07-31T02:16:29.099 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-eq5ur-QSX-UzNBr off=9970011 len=1037920 2021-07-31T02:16:29.104 INFO:tasks.workunit.client.0.smithi045.stdout:op 3428 completed, throughput=5MB/sec 2021-07-31T02:16:29.104 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-_dt--LMBNR89_RG off=12510138 len=1623462 2021-07-31T02:16:29.106 INFO:tasks.workunit.client.0.smithi045.stdout:op 3429 completed, throughput=5MB/sec 2021-07-31T02:16:29.106 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-zPPbb_Ai8oQ74rj off=521157 len=627539 2021-07-31T02:16:29.107 INFO:tasks.workunit.client.0.smithi045.stdout:op 3430 completed, throughput=5MB/sec 2021-07-31T02:16:29.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:29 smithi154 conmon[34553]: cluster 2021-07-31T02:16:28.210565+0000 mgr.x (mgr.24542) 1249 : cluster [DBG] pgmap v1749: 9 pgs: 9 active+clean; 127 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2021-07-31T02:16:29.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:29 smithi045 conmon[39718]: cluster 2021-07-31T02:16:28.210565+0000 mgr.x 2021-07-31T02:16:29.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:29 smithi045 conmon[39718]: (mgr.24542) 1249 : cluster [DBG] pgmap v1749: 9 pgs: 9 active+clean; 127 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2021-07-31T02:16:29.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:29 smithi045 conmon[33655]: cluster 2021-07-31T02:16:28.210565+0000 mgr.x (mgr.24542) 1249 : cluster [DBG] pgmap v1749: 9 pgs: 9 active+clean; 127 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2021-07-31T02:16:30.108 INFO:tasks.workunit.client.0.smithi045.stdout: 670: throughput=4.99MB/sec pending data=0 2021-07-31T02:16:30.108 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-p7Mhn9SpsWx-Nq- off=11200690 len=1024517 2021-07-31T02:16:30.109 INFO:tasks.workunit.client.0.smithi045.stdout:op 3431 completed, throughput=4.99MB/sec 2021-07-31T02:16:30.109 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-q3zZzNDJBmBjSM7 off=334129 len=646027 2021-07-31T02:16:30.116 INFO:tasks.workunit.client.0.smithi045.stdout:op 3432 completed, throughput=5MB/sec 2021-07-31T02:16:30.116 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj--37WkLORrLr9bU4 off=986557 len=903084 2021-07-31T02:16:30.117 INFO:tasks.workunit.client.0.smithi045.stdout:op 3433 completed, throughput=5MB/sec 2021-07-31T02:16:30.117 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-SI0X4xGeF4LeBus off=9008011 len=1099808 2021-07-31T02:16:30.117 INFO:tasks.workunit.client.0.smithi045.stdout:op 3434 completed, throughput=5MB/sec 2021-07-31T02:16:30.117 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-RySeRmE6zN-jFiz off=4402851 len=1288609 2021-07-31T02:16:30.123 INFO:tasks.workunit.client.0.smithi045.stdout:op 3435 completed, throughput=5MB/sec 2021-07-31T02:16:30.123 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Hd-pWl07xS4qV5J off=255026 len=1852489 2021-07-31T02:16:30.126 INFO:tasks.workunit.client.0.smithi045.stdout:op 3436 completed, throughput=5MB/sec 2021-07-31T02:16:31.126 INFO:tasks.workunit.client.0.smithi045.stdout: 671: throughput=5MB/sec pending data=0 2021-07-31T02:16:31.127 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-gUhLloE8DuJCLrJ off=10541824 len=36039 2021-07-31T02:16:31.128 INFO:tasks.workunit.client.0.smithi045.stdout:op 3437 completed, throughput=5MB/sec 2021-07-31T02:16:31.129 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-2cQIbM_J33tqS9C off=5193119 len=128706 2021-07-31T02:16:31.130 INFO:tasks.workunit.client.0.smithi045.stdout:op 3438 completed, throughput=5MB/sec 2021-07-31T02:16:31.130 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ZFcZMKEqdTMibqV off=8777145 len=158807 2021-07-31T02:16:31.131 INFO:tasks.workunit.client.0.smithi045.stdout:op 3439 completed, throughput=5MB/sec 2021-07-31T02:16:31.132 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-lQlWGSL3zC5dnvM off=8008972 len=164973 2021-07-31T02:16:31.134 INFO:tasks.workunit.client.0.smithi045.stdout:op 3440 completed, throughput=5MB/sec 2021-07-31T02:16:31.134 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-XEaDSh1hQAWbJAP off=8400449 len=1713835 2021-07-31T02:16:31.137 INFO:tasks.workunit.client.0.smithi045.stdout:op 3441 completed, throughput=5MB/sec 2021-07-31T02:16:31.137 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-rjJ9_FRh5jYtIeK off=11212172 len=106944 2021-07-31T02:16:31.138 INFO:tasks.workunit.client.0.smithi045.stdout:op 3442 completed, throughput=5MB/sec 2021-07-31T02:16:31.139 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ytjI2O7Jqhgoqjy off=1657229 len=1017241 2021-07-31T02:16:31.139 INFO:tasks.workunit.client.0.smithi045.stdout:op 3443 completed, throughput=5MB/sec 2021-07-31T02:16:31.140 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-3ebRONC69QjizYv off=3874158 len=259493 2021-07-31T02:16:31.142 INFO:tasks.workunit.client.0.smithi045.stdout:op 3444 completed, throughput=5MB/sec 2021-07-31T02:16:31.142 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-nl1yv-bztXu6MWj off=598798 len=1115675 2021-07-31T02:16:31.148 INFO:tasks.workunit.client.0.smithi045.stdout:op 3445 completed, throughput=5MB/sec 2021-07-31T02:16:31.252 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: debug 2021-07-31T02:16:31.001+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:16:31.253 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:16:31.253 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:16:31.253 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: yield 2021-07-31T02:16:31.253 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:16:31.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:16:31.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:16:31.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:16:31.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:16:31.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:16:31.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:16:31.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:16:31.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:16:31.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: task.result() 2021-07-31T02:16:31.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:16:31.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: debug 2021-07-31T02:16:31.013+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:16:31.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:16:31.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:16:31.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: yield 2021-07-31T02:16:31.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:16:31.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:16:31.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:16:31.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:16:31.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:16:31.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:16:31.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:16:31.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:16:31.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:16:31.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: task.result() 2021-07-31T02:16:31.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:16:31.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: debug 2021-07-31T02:16:31.038+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:16:31.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:16:31.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:16:31.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: yield 2021-07-31T02:16:31.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:16:31.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:16:31.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:16:31.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:16:31.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:16:31.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:16:31.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:16:31.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:16:31.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:16:31.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: task.result() 2021-07-31T02:16:31.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:16:31.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: debug 2021-07-31T02:16:31.038+0000 7fcb9816a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:16:31.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:16:31.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:16:31.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: yield 2021-07-31T02:16:31.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:16:31.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:16:31.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:16:31.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:16:31.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:16:31.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:16:31.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:16:31.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:16:31.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:16:31.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: task.result() 2021-07-31T02:16:31.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:16:31.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: 2021-07-31T02:16:31.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:16:31.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: 2021-07-31T02:16:31.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:16:31.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:16:31.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:16:31.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:16:31.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:16:31.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:16:31.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:16:31.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:16:31.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:16:31.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:16:31.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:16:31.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:16:31.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:16:31.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:16:31.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:16:31.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:16:31.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:16:31.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:16:31.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: raise 2021-07-31T02:16:31.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:16:31.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:16:31.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:16:31.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:16:31.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:16:31.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: debug 2021-07-31T02:16:31.047+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:16:31.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:16:31.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:16:31.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: yield 2021-07-31T02:16:31.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:16:31.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:16:31.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:16:31.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:16:31.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:16:31.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:16:31.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:16:31.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:16:31.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:16:31.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: task.result() 2021-07-31T02:16:31.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:16:31.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: debug 2021-07-31T02:16:31.047+0000 7fcb97969700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:16:31.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:16:31.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:16:31.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: yield 2021-07-31T02:16:31.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:16:31.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:16:31.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:16:31.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:16:31.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:16:31.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:16:31.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:16:31.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:16:31.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:16:31.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: task.result() 2021-07-31T02:16:31.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:16:31.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: 2021-07-31T02:16:31.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:16:31.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: 2021-07-31T02:16:31.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:16:31.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:16:31.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:16:31.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:16:31.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:16:31.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:16:31.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:16:31.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:16:31.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:16:31.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:16:31.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:16:31.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:16:31.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:16:31.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:16:31.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:16:31.284 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:16:31.284 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:16:31.284 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:16:31.284 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: raise 2021-07-31T02:16:31.284 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:16:31.285 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:16:31.285 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:16:31.285 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:16:31.285 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:16:31.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: cluster 2021-07-31T02:16:30.211124+0000 mgr.x (mgr 2021-07-31T02:16:31.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: .24542) 1250 : cluster [DBG] pgmap v1750: 9 pgs: 9 active+clean; 127 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.9 MiB/s wr, 5 op/s 2021-07-31T02:16:31.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: audit 2021-07-31T02:16:31.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: 2021-07-31T02:16:30.956288+0000 mon.b (mon.2) 215 : audit 2021-07-31T02:16:31.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:16:31.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: audit 2021 2021-07-31T02:16:31.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: -07-31T02:16:30.957798+0000 mon.b ( 2021-07-31T02:16:31.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: mon.2) 216 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:16:31.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: audit 2021-07 2021-07-31T02:16:31.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: -31T02:16:30.958811+0000 mon.b (mon.2) 217 : audit 2021-07-31T02:16:31.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:16:31.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:16:31.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: 2021-07-31T02:16 2021-07-31T02:16:31.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: :30.959607+0000 mgr.x (mgr.24542) 1251 : 2021-07-31T02:16:31.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:16:31.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:16:31.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: 2021-07-31T02:16:31.002492+0000 mgr.x 2021-07-31T02:16:31.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: (mgr.24542) 1252 : cephadm [ERR] 2021-07-31T02:16:31.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:16:31.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:16:31.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: yield 2021-07-31T02:16:31.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:16:31.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:16:31.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:16:31.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:16:31.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:16:31.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:16:31.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:16:31.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:16:31.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:16:31.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: task.result() 2021-07-31T02:16:31.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:16:31.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:16:31.002937+0000 mgr.x (mgr.24542) 1253 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:16:31.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: cephadm 2021- 2021-07-31T02:16:31.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: 07-31T02:16:31.014107+0000 mgr.x (mgr.24542) 1254 : cephadm [ERR] 2021-07-31T02:16:31.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: 2021-07-31T02:16:31.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:16:31.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:16:31.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: yield 2021-07-31T02:16:31.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:16:31.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: cluster 2021-07-31T02:16:30.211124+0000 mgr.x (mgr.24542) 2021-07-31T02:16:31.539 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: 1250 : cluster [DBG] pgmap v1750: 9 pgs: 9 active+clean; 127 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.9 MiB/s wr, 5 op/s 2021-07-31T02:16:31.539 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: audit 2021-07-31T02:16:30.956288+0000 mon.b (mon.2) 215 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:16:31.539 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: audit 2021-07-31T 2021-07-31T02:16:31.540 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: 02:16:30.957798+0000 mon.b (mon.2) 216 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:16:31.540 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: audit 2021-07-31T02:16:30.958811+0000 mon.b (mon 2021-07-31T02:16:31.540 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: .2) 217 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:16:31.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: cephadm 2021 2021-07-31T02:16:31.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: -07-31T02:16:30.959607+0000 mgr.x (mgr.24542) 1251 : cephadm 2021-07-31T02:16:31.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:16:31.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:16 2021-07-31T02:16:31.543 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: :31.002492+0000 mgr.x (mgr.24542) 1252 : cephadm 2021-07-31T02:16:31.543 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: [ERR] 2021-07-31T02:16:31.544 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:16:31.544 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:16:31.544 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: yield 2021-07-31T02:16:31.545 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:16:31.545 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:16:31.545 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:16:31.546 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:16:31.546 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:16:31.546 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:16:31.547 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:16:31.547 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:16:31.547 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:16:31.548 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: task.result() 2021-07-31T02:16:31.548 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:16:31.548 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:16:31.002937+0000 mgr.x (mgr.24542) 1253 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:16:31.549 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:16:31.014107+0000 2021-07-31T02:16:31.549 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: mgr.x (mgr.24542) 1254 : cephadm [ERR] 2021-07-31T02:16:31.550 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:16:31.550 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:16:31.551 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: yield 2021-07-31T02:16:31.551 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:16:31.552 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:16:31.553 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:16:31.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:16:31.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:16:31.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:16:31.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:16:31.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:16:31.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:16:31.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:16:31.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:16:31.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: task.result() 2021-07-31T02:16:31.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:16:31.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:16:31.014961+0000 mgr.x (mgr.24542) 1255 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:16:31.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:16:31. 2021-07-31T02:16:31.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: 039317+0000 mgr.x (mgr.24542) 1256 : cephadm [ERR] 2021-07-31T02:16:31.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:16:31.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:16:31.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: yield 2021-07-31T02:16:31.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:16:31.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:16:31.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:16:31.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:16:31.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:16:31.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:16:31.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:16:31.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:16:31.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:16:31.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: task.result() 2021-07-31T02:16:31.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:16:31.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:16:31.039712+0000 mgr.x (mgr.24542) 1257 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:16:31.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:16:31.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:16:31.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: yield 2021-07-31T02:16:31.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:16:31.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:16:31.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:16:31.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:16:31.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:16:31.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:16:31.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:16:31.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:16:31.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:16:31.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:16:31.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:16:31.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:16:31.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:16:31.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:16:31.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: task.result() 2021-07-31T02:16:31.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:16:31.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:16:31.014961+0000 mgr.x (mgr.24542) 1255 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:16:31.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:16:31 2021-07-31T02:16:31.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: .039317+0000 mgr.x (mgr.24542) 1256 : cephadm [ERR] 2021-07-31T02:16:31.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:16:31.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:16:31.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: yield 2021-07-31T02:16:31.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:16:31.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:16:31.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:16:31.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:16:31.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:16:31.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:16:31.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:16:31.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:16:31.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:16:31.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: task.result() 2021-07-31T02:16:31.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:16:31.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:16:31.039712+0000 mgr.x (mgr.24542) 1257 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:16:31.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:16:31.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:16:31.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: yield 2021-07-31T02:16:31.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:16:31.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:16:31.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:16:31.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:16:31.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:16:31.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:16:31.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:16:31.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:16:31.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:16:31.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: task.result() 2021-07-31T02:16:31.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:16:31.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: 2021-07-31T02:16:31.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:16:31.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: task.result() 2021-07-31T02:16:31.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:16:31.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: 2021-07-31T02:16:31.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:16:31.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: 2021-07-31T02:16:31.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:16:31.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:16:31.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:16:31.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:16:31.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:16:31.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:16:31.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:16:31.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:16:31.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:16:31.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:16:31.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:16:31.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:16:31.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:16:31.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:16:31.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:16:31.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:16:31.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:16:31.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:16:31.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: raise 2021-07-31T02:16:31.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:16:31.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:16:31.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:16:31.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:16:31.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:16:31.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:16:31.048243+0000 mgr.x (mgr.24542) 1258 : cephadm [ERR] 2021-07-31T02:16:31.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:16:31.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:16:31.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: yield 2021-07-31T02:16:31.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:16:31.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:16:31.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:16:31.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:16:31.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:16:31.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:16:31.585 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:16:31.585 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:16:31.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:16:31.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: 2021-07-31T02:16:31.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:16:31.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:16:31.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:16:31.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:16:31.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:16:31.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:16:31.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:16:31.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:16:31.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:16:31.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:16:31.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:16:31.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:16:31.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:16:31.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:16:31.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:16:31.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:16:31.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:16:31.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:16:31.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: raise 2021-07-31T02:16:31.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:16:31.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:16:31.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:16:31.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:16:31.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:16:31.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: cephadm 2021 2021-07-31T02:16:31.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: -07-31T02:16:31.048243+0000 mgr.x (mgr.24542) 1258 : cephadm [ERR] 2021-07-31T02:16:31.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:16:31.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:16:31.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: yield 2021-07-31T02:16:31.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:16:31.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:16:31.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:16:31.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:16:31.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:16:31.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:16:31.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:16:31.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:16:31.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:16:31.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: task.result() 2021-07-31T02:16:31.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:16:31.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:16:31.048776+0000 mgr.x (mgr.24542) 1259 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:16:31.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:16:31.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:16:31.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: yield 2021-07-31T02:16:31.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:16:31.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:16:31.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:16:31.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:16:31.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:16:31.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:16:31.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:16:31.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:16:31.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:16:31.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: task.result() 2021-07-31T02:16:31.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:16:31.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: 2021-07-31T02:16:31.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:16:31.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: 2021-07-31T02:16:31.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:16:31.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:16:31.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:16:31.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:16:31.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:16:31.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:16:31.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:16:31.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:16:31.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:16:31.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:16:31.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:16:31.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:16:31.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:16:31.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:16:31.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:16:31.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:16:31.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:16:31.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:16:31.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: raise 2021-07-31T02:16:31.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:16:31.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:16:31.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:16:31.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:16:31.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:16:31.607 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:16:31.608 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: task.result() 2021-07-31T02:16:31.608 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:16:31.608 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:16:31.048776+0000 mgr.x (mgr.24542) 1259 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:16:31.608 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:16:31.608 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:16:31.609 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: yield 2021-07-31T02:16:31.609 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:16:31.610 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:16:31.610 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:16:31.610 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:16:31.610 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:16:31.611 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:16:31.611 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:16:31.611 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:16:31.612 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:16:31.612 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: task.result() 2021-07-31T02:16:31.612 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:16:31.612 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: 2021-07-31T02:16:31.612 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:16:31.613 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: 2021-07-31T02:16:31.613 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:16:31.613 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:16:31.613 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:16:31.614 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:16:31.614 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:16:31.614 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:16:31.614 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:16:31.614 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:16:31.615 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:16:31.615 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:16:31.615 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:16:31.615 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:16:31.616 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:16:31.616 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:16:31.616 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:16:31.616 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:16:31.616 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:16:31.617 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:16:31.617 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: raise 2021-07-31T02:16:31.617 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:16:31.617 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:16:31.617 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:16:31.618 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:16:31.618 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:31 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:16:31.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: cluster 2021-07-31T02:16:30.211124+0000 mgr.x (mgr.24542) 1250 : cluster 2021-07-31T02:16:31.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: [DBG] pgmap v1750: 9 pgs: 9 active+clean; 127 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.9 MiB/s wr, 5 op/s 2021-07-31T02:16:31.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: audit 2021-07-31T02:16:30.956288+0000 mon.b (mon.2) 215 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:16:31.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: audit 2021-07-31T02: 2021-07-31T02:16:31.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: 16:30.957798+0000 mon.b (mon. 2021-07-31T02:16:31.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: 2) 216 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:16:31.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: audit 2021- 2021-07-31T02:16:31.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: 07-31T02:16:30.958811+0000 2021-07-31T02:16:31.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: mon.b (mon.2) 217 : audit 2021-07-31T02:16:31.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:16:31.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: cephadm 2021 2021-07-31T02:16:31.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: -07-31T02:16:30. 2021-07-31T02:16:31.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: 959607+0000 mgr.x (mgr.24542) 2021-07-31T02:16:31.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: 1251 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:16:31.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: cephadm 2021- 2021-07-31T02:16:31.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: 07-31T02:16:31.002492 2021-07-31T02:16:31.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: +0000 mgr.x (mgr.24542) 1252 2021-07-31T02:16:31.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: : cephadm [ERR] 2021-07-31T02:16:31.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:16:31.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:16:31.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: yield 2021-07-31T02:16:31.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:16:31.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:16:31.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:16:31.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:16:31.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:16:31.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:16:31.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:16:31.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:16:31.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:16:31.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: task.result() 2021-07-31T02:16:31.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:16:31.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:16:31.002937+0000 mgr.x (mgr.24542) 1253 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:16:31.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: cephadm 2021- 2021-07-31T02:16:31.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: 07-31T02:16:31.014107+0000 2021-07-31T02:16:31.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: mgr.x (mgr.24542) 1254 : 2021-07-31T02:16:31.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: cephadm [ERR] 2021-07-31T02:16:31.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:16:31.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:16:31.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: yield 2021-07-31T02:16:31.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:16:31.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:16:31.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:16:31.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:16:31.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:16:31.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:16:31.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:16:31.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:16:31.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:16:31.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: task.result() 2021-07-31T02:16:31.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:16:31.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:16:31.014961+0000 mgr.x (mgr.24542) 1255 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:16:31.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: cephadm 2021- 2021-07-31T02:16:31.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: 07-31T02:16:31.039317 2021-07-31T02:16:31.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: +0000 mgr.x (mgr.24542) 1256 2021-07-31T02:16:31.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: : cephadm [ERR] 2021-07-31T02:16:31.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:16:31.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:16:31.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: yield 2021-07-31T02:16:31.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:16:31.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:16:31.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:16:31.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:16:31.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:16:31.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:16:31.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:16:31.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:16:31.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:16:31.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: task.result() 2021-07-31T02:16:31.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:16:31.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:16:31.039712+0000 mgr.x (mgr.24542) 1257 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:16:31.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:16:31.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:16:31.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: yield 2021-07-31T02:16:31.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:16:31.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:16:31.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:16:31.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:16:31.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:16:31.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:16:31.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:16:31.782 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:16:31.782 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:16:31.782 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: task.result() 2021-07-31T02:16:31.782 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:16:31.783 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: 2021-07-31T02:16:31.783 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:16:31.785 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: 2021-07-31T02:16:31.785 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:16:31.785 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:16:31.785 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:16:31.786 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:16:31.786 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:16:31.786 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:16:31.786 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:16:31.787 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:16:31.787 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:16:31.787 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:16:31.787 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:16:31.788 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:16:31.788 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:16:31.788 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:16:31.788 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:16:31.788 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:16:31.789 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:16:31.789 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:16:31.789 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: raise 2021-07-31T02:16:31.789 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:16:31.790 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:16:31.790 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:16:31.790 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:16:31.790 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:16:31.790 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:16:31.048243+0000 mgr.x (mgr.24542) 1258 : cephadm [ERR] 2021-07-31T02:16:31.791 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:16:31.791 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:16:31.791 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: yield 2021-07-31T02:16:31.791 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:16:31.792 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:16:31.792 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:16:31.792 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:16:31.792 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:16:31.793 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:16:31.793 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:16:31.793 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:16:31.793 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:16:31.793 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: task.result() 2021-07-31T02:16:31.794 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:16:31.794 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:16:31.048776+0000 mgr.x (mgr.24542) 1259 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:16:31.794 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:16:31.794 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:16:31.795 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: yield 2021-07-31T02:16:31.795 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:16:31.795 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:16:31.795 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:16:31.796 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:16:31.796 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:16:31.796 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:16:31.796 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:16:31.796 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:16:31.797 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:16:31.797 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: task.result() 2021-07-31T02:16:31.797 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:16:31.797 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: 2021-07-31T02:16:31.798 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:16:31.798 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: 2021-07-31T02:16:31.798 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:16:31.798 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:16:31.798 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:16:31.799 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:16:31.799 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:16:31.799 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:16:31.799 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:16:31.800 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:16:31.800 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:16:31.800 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:16:31.800 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:16:31.800 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:16:31.801 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:16:31.801 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:16:31.801 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:16:31.801 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:16:31.802 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:16:31.802 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:16:31.802 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: raise 2021-07-31T02:16:31.802 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:16:31.803 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:16:31.803 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:16:31.803 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:16:31.803 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:31 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:16:32.148 INFO:tasks.workunit.client.0.smithi045.stdout: 672: throughput=4.99MB/sec pending data=0 2021-07-31T02:16:32.149 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-FWeeATjWeMCFqua off=3090528 len=427212 2021-07-31T02:16:32.152 INFO:tasks.workunit.client.0.smithi045.stdout:op 3446 completed, throughput=4.99MB/sec 2021-07-31T02:16:32.152 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ZUyl5B0lHAig05H off=3596531 len=597241 2021-07-31T02:16:32.154 INFO:tasks.workunit.client.0.smithi045.stdout:op 3447 completed, throughput=5MB/sec 2021-07-31T02:16:32.154 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-doT4GfBOB_p68Bl off=1704339 len=1472534 2021-07-31T02:16:32.162 INFO:tasks.workunit.client.0.smithi045.stdout:op 3448 completed, throughput=5MB/sec 2021-07-31T02:16:32.162 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-U0X1NQ1hXKNiGR0 off=601357 len=784024 2021-07-31T02:16:32.163 INFO:tasks.workunit.client.0.smithi045.stdout:op 3449 completed, throughput=5MB/sec 2021-07-31T02:16:32.164 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-cBRXDruFhJPx4nX off=3167288 len=1886986 2021-07-31T02:16:32.166 INFO:tasks.workunit.client.0.smithi045.stdout:op 3450 completed, throughput=5MB/sec 2021-07-31T02:16:33.167 INFO:tasks.workunit.client.0.smithi045.stdout: 673: throughput=4.99MB/sec pending data=0 2021-07-31T02:16:33.167 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-hpEvZkq1d5iKLWd off=879075 len=1933815 2021-07-31T02:16:33.171 INFO:tasks.workunit.client.0.smithi045.stdout:op 3451 completed, throughput=5MB/sec 2021-07-31T02:16:33.171 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-eJD5LTJDweURF46 off=4087436 len=706800 2021-07-31T02:16:33.175 INFO:tasks.workunit.client.0.smithi045.stdout:op 3452 completed, throughput=5MB/sec 2021-07-31T02:16:33.175 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-swaarWFpPEMPWAH off=43296 len=765293 2021-07-31T02:16:33.181 INFO:tasks.workunit.client.0.smithi045.stdout:op 3453 completed, throughput=5MB/sec 2021-07-31T02:16:33.181 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-rn50sj0d0ZRdLOE off=2996630 len=1984019 2021-07-31T02:16:33.183 INFO:tasks.workunit.client.0.smithi045.stdout:op 3454 completed, throughput=5MB/sec 2021-07-31T02:16:34.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:16:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:16:33.559Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:16:34.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:16:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:16:33.559Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:16:34.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:16:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:16:33.559Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:16:34.184 INFO:tasks.workunit.client.0.smithi045.stdout: 674: throughput=4.99MB/sec pending data=0 2021-07-31T02:16:34.184 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-1aOprUMhSHPZ4mA off=10467933 len=1475216 2021-07-31T02:16:34.190 INFO:tasks.workunit.client.0.smithi045.stdout:op 3455 completed, throughput=5MB/sec 2021-07-31T02:16:34.191 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-9-a9-5JA2Tr9SWU off=14879883 len=1335988 2021-07-31T02:16:34.193 INFO:tasks.workunit.client.0.smithi045.stdout:op 3456 completed, throughput=5MB/sec 2021-07-31T02:16:34.193 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-gYTwLTmDklJRPOH off=580310 len=2076852 2021-07-31T02:16:34.196 INFO:tasks.workunit.client.0.smithi045.stdout:op 3457 completed, throughput=5MB/sec 2021-07-31T02:16:34.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:34 smithi154 conmon[34553]: cluster 2021-07-31T02:16:32.211651+0000 mgr.x (mgr.24542) 1260 : cluster [DBG] pgmap v1751: 9 pgs: 9 active+clean; 127 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 2.3 MiB/s rd, 1.4 MiB/s wr, 3 op/s 2021-07-31T02:16:34.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:34 smithi154 conmon[34553]: 2021-07-31T02:16:34.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:34 smithi045 conmon[33655]: cluster 2021-07-31T02:16:32.211651+0000 mgr.x (mgr.24542) 1260 : cluster [DBG] 2021-07-31T02:16:34.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:34 smithi045 conmon[33655]: pgmap v1751: 9 pgs: 9 active+clean; 127 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 2.3 MiB/s rd, 1.4 MiB/s wr, 3 op/s 2021-07-31T02:16:34.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:34 smithi045 conmon[39718]: cluster 2021-07-31T02:16:32.211651+0000 mgr.x 2021-07-31T02:16:34.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:34 smithi045 conmon[39718]: (mgr.24542) 1260 : cluster [DBG] pgmap v1751: 9 pgs: 9 active+clean; 127 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 2.3 MiB/s rd, 1.4 MiB/s wr, 3 op/s 2021-07-31T02:16:35.197 INFO:tasks.workunit.client.0.smithi045.stdout: 675: throughput=4.99MB/sec pending data=0 2021-07-31T02:16:35.197 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Jkxh5O7bT-lgsHG off=13918947 len=932328 2021-07-31T02:16:35.199 INFO:tasks.workunit.client.0.smithi045.stdout:op 3458 completed, throughput=4.99MB/sec 2021-07-31T02:16:35.199 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-GtAZN-QvvcZ6GZO off=10791645 len=2030417 2021-07-31T02:16:35.202 INFO:tasks.workunit.client.0.smithi045.stdout:op 3459 completed, throughput=5MB/sec 2021-07-31T02:16:35.202 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-fC9Anjxi8VV_NsD off=3357889 len=1948634 2021-07-31T02:16:35.205 INFO:tasks.workunit.client.0.smithi045.stdout:op 3460 completed, throughput=5MB/sec 2021-07-31T02:16:36.205 INFO:tasks.workunit.client.0.smithi045.stdout: 676: throughput=4.99MB/sec pending data=0 2021-07-31T02:16:36.205 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Bz1NqGLXEE-FUWd off=12133815 len=118875 2021-07-31T02:16:36.208 INFO:tasks.workunit.client.0.smithi045.stdout:op 3461 completed, throughput=4.99MB/sec 2021-07-31T02:16:36.208 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-fuP3ezUI6w35Nk- off=2327668 len=244013 2021-07-31T02:16:36.211 INFO:tasks.workunit.client.0.smithi045.stdout:op 3462 completed, throughput=4.99MB/sec 2021-07-31T02:16:36.211 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-4NtHp0_-s4wbuQ- off=10002083 len=416711 2021-07-31T02:16:36.213 INFO:tasks.workunit.client.0.smithi045.stdout:op 3463 completed, throughput=4.99MB/sec 2021-07-31T02:16:36.213 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-yf_9bCFMTbQlGWP off=2654711 len=1018330 2021-07-31T02:16:36.218 INFO:tasks.workunit.client.0.smithi045.stdout:op 3464 completed, throughput=5MB/sec 2021-07-31T02:16:36.219 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-X-rcF29YtPblU6Y off=468343 len=1288632 2021-07-31T02:16:36.225 INFO:tasks.workunit.client.0.smithi045.stdout:op 3465 completed, throughput=5MB/sec 2021-07-31T02:16:36.226 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-aQPaAY2MbEXj19K off=1105828 len=1014385 2021-07-31T02:16:36.231 INFO:tasks.workunit.client.0.smithi045.stdout:op 3466 completed, throughput=5MB/sec 2021-07-31T02:16:36.231 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-5uPLLYi9HVTJYYc off=10099848 len=76222 2021-07-31T02:16:36.232 INFO:tasks.workunit.client.0.smithi045.stdout:op 3467 completed, throughput=5MB/sec 2021-07-31T02:16:36.232 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-KTTH0FB57C5D0p2 off=4645385 len=1174481 2021-07-31T02:16:36.233 INFO:tasks.workunit.client.0.smithi045.stdout:op 3468 completed, throughput=5MB/sec 2021-07-31T02:16:36.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:36 smithi154 conmon[34553]: cluster 2021-07-31T02:16:34.212100+0000 mgr.x (mgr.24542) 1261 : cluster 2021-07-31T02:16:36.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:36 smithi154 conmon[34553]: [DBG] pgmap v1752: 9 pgs: 9 active+clean; 127 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 3.8 MiB/s rd, 2.1 MiB/s wr, 6 op/s 2021-07-31T02:16:36.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:36 smithi045 conmon[39718]: cluster 2021-07-31T02:16:34.212100+0000 mgr.x (mgr.24542) 1261 : cluster [DBG] pgmap v1752: 9 pgs: 9 active+clean; 127 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 3.8 MiB/s rd, 2.1 MiB/s wr, 6 op/s 2021-07-31T02:16:36.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:36 smithi045 conmon[33655]: cluster 2021-07-31T02:16:34.212100+0000 mgr.x (mgr.24542) 1261 : cluster 2021-07-31T02:16:36.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:36 smithi045 conmon[33655]: [DBG] pgmap v1752: 9 pgs: 9 active+clean; 127 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 3.8 MiB/s rd, 2.1 MiB/s wr, 6 op/s 2021-07-31T02:16:37.233 INFO:tasks.workunit.client.0.smithi045.stdout: 677: throughput=4.99MB/sec pending data=0 2021-07-31T02:16:37.234 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-PDtO49vc_Qx6ZqS off=5500328 len=376671 2021-07-31T02:16:37.237 INFO:tasks.workunit.client.0.smithi045.stdout:op 3469 completed, throughput=4.99MB/sec 2021-07-31T02:16:37.237 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-l1VF6t_A7C-LDAx off=9330548 len=1997034 2021-07-31T02:16:37.241 INFO:tasks.workunit.client.0.smithi045.stdout:op 3470 completed, throughput=5MB/sec 2021-07-31T02:16:37.241 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-SuA2_lk_pE60pMp off=3144259 len=1687727 2021-07-31T02:16:37.244 INFO:tasks.workunit.client.0.smithi045.stdout:op 3471 completed, throughput=5MB/sec 2021-07-31T02:16:37.245 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-rzc4G_rQbLgKht7 off=15648415 len=584599 2021-07-31T02:16:37.245 INFO:tasks.workunit.client.0.smithi045.stdout:op 3472 completed, throughput=5MB/sec 2021-07-31T02:16:37.246 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-61Y4Nffkf_R-LXm off=13743740 len=8337 2021-07-31T02:16:37.248 INFO:tasks.workunit.client.0.smithi045.stdout:op 3473 completed, throughput=5MB/sec 2021-07-31T02:16:37.248 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ou-lhgD3yYXRhhO off=15279848 len=1707607 2021-07-31T02:16:37.250 INFO:tasks.workunit.client.0.smithi045.stdout:op 3474 completed, throughput=5MB/sec 2021-07-31T02:16:38.250 INFO:tasks.workunit.client.0.smithi045.stdout: 678: throughput=4.99MB/sec pending data=0 2021-07-31T02:16:38.251 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-CmiS_6svk9jElVg off=8298664 len=251309 2021-07-31T02:16:38.255 INFO:tasks.workunit.client.0.smithi045.stdout:op 3475 completed, throughput=4.99MB/sec 2021-07-31T02:16:38.255 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-i1S_NMVu419NBN2 off=1625706 len=1186555 2021-07-31T02:16:38.255 INFO:tasks.workunit.client.0.smithi045.stdout:op 3476 completed, throughput=5MB/sec 2021-07-31T02:16:38.256 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-vZXWWE6AoahDgbU off=5334575 len=544387 2021-07-31T02:16:38.256 INFO:tasks.workunit.client.0.smithi045.stdout:op 3477 completed, throughput=5MB/sec 2021-07-31T02:16:38.257 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-M5GmLZMgx05sZIz off=10893150 len=72500 2021-07-31T02:16:38.258 INFO:tasks.workunit.client.0.smithi045.stdout:op 3478 completed, throughput=5MB/sec 2021-07-31T02:16:38.258 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-LlO7PeNSq0v5iQc off=2305171 len=1413304 2021-07-31T02:16:38.260 INFO:tasks.workunit.client.0.smithi045.stdout:op 3479 completed, throughput=5MB/sec 2021-07-31T02:16:38.261 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-me03T_hgLZxYY61 off=670101 len=237634 2021-07-31T02:16:38.261 INFO:tasks.workunit.client.0.smithi045.stdout:op 3480 completed, throughput=5MB/sec 2021-07-31T02:16:38.262 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-617g0UtHar4SLWx off=6870266 len=260919 2021-07-31T02:16:38.262 INFO:tasks.workunit.client.0.smithi045.stdout:op 3481 completed, throughput=5MB/sec 2021-07-31T02:16:38.262 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Vljt7Rx_-X3_44P off=1989225 len=1093991 2021-07-31T02:16:38.268 INFO:tasks.workunit.client.0.smithi045.stdout:op 3482 completed, throughput=5MB/sec 2021-07-31T02:16:38.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:38 smithi154 conmon[34553]: cluster 2021-07-31T02:16:36.212498+0000 mgr.x (mgr.24542) 1262 : cluster [DBG] pgmap v1753: 9 pgs: 9 active+clean; 127 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 2.8 MiB/s rd, 1.5 MiB/s wr, 4 op/s 2021-07-31T02:16:38.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:38 smithi045 conmon[39718]: cluster 2021-07-31T02:16:36.212498+0000 mgr.x (mgr. 2021-07-31T02:16:38.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:38 smithi045 conmon[39718]: 24542) 1262 : cluster [DBG] pgmap v1753: 9 pgs: 9 active+clean; 127 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 2.8 MiB/s rd, 1.5 MiB/s wr, 4 op/s 2021-07-31T02:16:38.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:38 smithi045 conmon[33655]: cluster 2021-07-31T02:16:36.212498+0000 mgr.x (mgr.24542) 1262 2021-07-31T02:16:38.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:38 smithi045 conmon[33655]: : cluster [DBG] pgmap v1753: 9 pgs: 9 active+clean; 127 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 2.8 MiB/s rd, 1.5 MiB/s wr, 4 op/s 2021-07-31T02:16:39.268 INFO:tasks.workunit.client.0.smithi045.stdout: 679: throughput=4.99MB/sec pending data=0 2021-07-31T02:16:39.268 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-wvYoyqcNG1rknmY off=21448527 len=484523 2021-07-31T02:16:39.269 INFO:tasks.workunit.client.0.smithi045.stdout:op 3483 completed, throughput=4.99MB/sec 2021-07-31T02:16:39.269 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Skr4JNzQSjP2CH_ off=11286940 len=37926 2021-07-31T02:16:39.270 INFO:tasks.workunit.client.0.smithi045.stdout:op 3484 completed, throughput=4.99MB/sec 2021-07-31T02:16:39.270 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-4hO2c_Hvfls4czm off=2120490 len=2074779 2021-07-31T02:16:39.272 INFO:tasks.workunit.client.0.smithi045.stdout:op 3485 completed, throughput=5MB/sec 2021-07-31T02:16:39.272 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-i4V29EOudGCX2qx off=1125400 len=371084 2021-07-31T02:16:39.277 INFO:tasks.workunit.client.0.smithi045.stdout:op 3486 completed, throughput=5MB/sec 2021-07-31T02:16:39.277 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-R7ghzqg-ymJ2HOQ off=3613285 len=2096121 2021-07-31T02:16:39.279 INFO:tasks.workunit.client.0.smithi045.stdout:op 3487 completed, throughput=5MB/sec 2021-07-31T02:16:40.280 INFO:tasks.workunit.client.0.smithi045.stdout: 680: throughput=4.99MB/sec pending data=0 2021-07-31T02:16:40.281 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ePb60XVRBd_RVKk off=18752633 len=1517696 2021-07-31T02:16:40.282 INFO:tasks.workunit.client.0.smithi045.stdout:op 3488 completed, throughput=5MB/sec 2021-07-31T02:16:40.282 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-1GQyzItT5s0iCrz off=1256688 len=906674 2021-07-31T02:16:40.288 INFO:tasks.workunit.client.0.smithi045.stdout:op 3489 completed, throughput=5MB/sec 2021-07-31T02:16:40.288 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-G0fH6kd5GvoWHyV off=1712861 len=1919002 2021-07-31T02:16:40.291 INFO:tasks.workunit.client.0.smithi045.stdout:op 3490 completed, throughput=5MB/sec 2021-07-31T02:16:40.292 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-n87tWTGKjCaWvOO off=1777264 len=1291994 2021-07-31T02:16:40.293 INFO:tasks.workunit.client.0.smithi045.stdout:op 3491 completed, throughput=5MB/sec 2021-07-31T02:16:40.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:40 smithi154 conmon[34553]: cluster 2021-07-31T02:16:38.213067+0000 mgr.x (mgr.24542) 1263 : cluster [DBG] pgmap v1754: 9 pgs: 9 active+clean; 127 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 2.8 MiB/s rd, 1.5 MiB/s wr, 4 op/s 2021-07-31T02:16:40.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:40 smithi045 conmon[33655]: cluster 2021-07-31T02:16: 2021-07-31T02:16:40.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:40 smithi045 conmon[33655]: 38.213067+0000 mgr.x (mgr.24542) 1263 : cluster [DBG] pgmap v1754: 9 pgs: 9 active+clean; 127 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 2.8 MiB/s rd, 1.5 MiB/s wr, 4 op/s 2021-07-31T02:16:40.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:40 smithi045 conmon[39718]: cluster 2021-07-31T02:16:38.213067+0000 mgr.x (mgr.24542) 2021-07-31T02:16:40.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:40 smithi045 conmon[39718]: 1263 : cluster [DBG] pgmap v1754: 9 pgs: 9 active+clean; 127 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 2.8 MiB/s rd, 1.5 MiB/s wr, 4 op/s 2021-07-31T02:16:41.293 INFO:tasks.workunit.client.0.smithi045.stdout: 681: throughput=4.99MB/sec pending data=0 2021-07-31T02:16:41.294 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-39Dz7RvgDiweR8c off=2243046 len=61999 2021-07-31T02:16:41.294 INFO:tasks.workunit.client.0.smithi045.stdout:op 3492 completed, throughput=4.99MB/sec 2021-07-31T02:16:41.295 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj--H6LsbaDe1EylGt off=21219526 len=177817 2021-07-31T02:16:41.295 INFO:tasks.workunit.client.0.smithi045.stdout:op 3493 completed, throughput=4.99MB/sec 2021-07-31T02:16:41.296 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-gnZX_U4H6Yu5yGM off=18854890 len=1024809 2021-07-31T02:16:41.297 INFO:tasks.workunit.client.0.smithi045.stdout:op 3494 completed, throughput=5MB/sec 2021-07-31T02:16:41.297 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Lw9kwCEk-b15vBP off=6018911 len=1652131 2021-07-31T02:16:41.306 INFO:tasks.workunit.client.0.smithi045.stdout:op 3495 completed, throughput=5MB/sec 2021-07-31T02:16:41.307 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-3uiyjJq_l2I5TBB off=13749343 len=1920973 2021-07-31T02:16:41.310 INFO:tasks.workunit.client.0.smithi045.stdout:op 3496 completed, throughput=5MB/sec 2021-07-31T02:16:42.310 INFO:tasks.workunit.client.0.smithi045.stdout: 682: throughput=4.99MB/sec pending data=0 2021-07-31T02:16:42.311 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-4fpWgPZApTEWcnO off=13977997 len=937087 2021-07-31T02:16:42.316 INFO:tasks.workunit.client.0.smithi045.stdout:op 3497 completed, throughput=4.99MB/sec 2021-07-31T02:16:42.316 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-A9Vx4JwuNH_3YRV off=6179460 len=1312121 2021-07-31T02:16:42.322 INFO:tasks.workunit.client.0.smithi045.stdout:op 3498 completed, throughput=5MB/sec 2021-07-31T02:16:42.322 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-by8kCB2UUXXF1J6 off=4587234 len=383533 2021-07-31T02:16:42.325 INFO:tasks.workunit.client.0.smithi045.stdout:op 3499 completed, throughput=5MB/sec 2021-07-31T02:16:42.326 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Ux1m15M9QVHiu-9 off=2406210 len=2094449 2021-07-31T02:16:42.335 INFO:tasks.workunit.client.0.smithi045.stdout:op 3500 completed, throughput=5MB/sec 2021-07-31T02:16:42.335 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Jdziy_x-FGyn3Hk off=2124687 len=455745 2021-07-31T02:16:42.336 INFO:tasks.workunit.client.0.smithi045.stdout:op 3501 completed, throughput=5MB/sec 2021-07-31T02:16:42.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:42 smithi154 conmon[34553]: cluster 2021-07-31T02:16:40.213648+0000 mgr.x (mgr.24542) 1264 : cluster [DBG] pgmap v1755: 9 pgs: 9 active+clean; 127 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 4.3 MiB/s rd, 2.1 MiB/s wr, 6 op/s 2021-07-31T02:16:42.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:42 smithi045 conmon[39718]: cluster 2021-07-31T02:16:40.213648+0000 mgr.x (mgr.24542) 2021-07-31T02:16:42.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:42 smithi045 conmon[39718]: 1264 : cluster [DBG] pgmap v1755: 9 pgs: 9 active+clean; 127 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 4.3 MiB/s rd, 2.1 MiB/s wr, 6 op/s 2021-07-31T02:16:42.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:42 smithi045 conmon[33655]: cluster 2021-07-31 2021-07-31T02:16:42.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:42 smithi045 conmon[33655]: T02:16:40.213648+0000 mgr.x (mgr.24542) 1264 : cluster [DBG] pgmap v1755: 9 pgs: 9 active+clean; 127 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 4.3 MiB/s rd, 2.1 MiB/s wr, 6 op/s 2021-07-31T02:16:43.336 INFO:tasks.workunit.client.0.smithi045.stdout: 683: throughput=4.99MB/sec pending data=0 2021-07-31T02:16:43.336 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-VwygEvEc3GsxOhy off=8335927 len=121525 2021-07-31T02:16:43.339 INFO:tasks.workunit.client.0.smithi045.stdout:op 3502 completed, throughput=4.99MB/sec 2021-07-31T02:16:43.339 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-GA0it3yCCrdmRzC off=9141613 len=699256 2021-07-31T02:16:43.340 INFO:tasks.workunit.client.0.smithi045.stdout:op 3503 completed, throughput=4.99MB/sec 2021-07-31T02:16:43.341 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-OecCeeNDO-AZUu4 off=15673444 len=245824 2021-07-31T02:16:43.344 INFO:tasks.workunit.client.0.smithi045.stdout:op 3504 completed, throughput=4.99MB/sec 2021-07-31T02:16:43.344 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-UleVHc6-Gc0jxnC off=2451909 len=40860 2021-07-31T02:16:43.345 INFO:tasks.workunit.client.0.smithi045.stdout:op 3505 completed, throughput=4.99MB/sec 2021-07-31T02:16:43.345 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-dcLhomXqvFhx-eF off=2197491 len=797042 2021-07-31T02:16:43.349 INFO:tasks.workunit.client.0.smithi045.stdout:op 3506 completed, throughput=5MB/sec 2021-07-31T02:16:43.350 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-e4h65FN5M1L52dP off=2120430 len=905070 2021-07-31T02:16:43.352 INFO:tasks.workunit.client.0.smithi045.stdout:op 3507 completed, throughput=5MB/sec 2021-07-31T02:16:43.352 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-_3cK5BMi8jYtzfc off=7202801 len=1778548 2021-07-31T02:16:43.363 INFO:tasks.workunit.client.0.smithi045.stdout:op 3508 completed, throughput=5MB/sec 2021-07-31T02:16:43.363 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-TyJNAxZbJpmikSt off=16980421 len=566362 2021-07-31T02:16:43.364 INFO:tasks.workunit.client.0.smithi045.stdout:op 3509 completed, throughput=5MB/sec 2021-07-31T02:16:44.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:16:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:16:43.559Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:16:44.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:16:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:16:43.559Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:16:44.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:16:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:16:43.559Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:16:44.364 INFO:tasks.workunit.client.0.smithi045.stdout: 684: throughput=4.99MB/sec pending data=0 2021-07-31T02:16:44.364 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-TJqKJgAVr-lFVUP off=6163587 len=320694 2021-07-31T02:16:44.365 INFO:tasks.workunit.client.0.smithi045.stdout:op 3510 completed, throughput=4.99MB/sec 2021-07-31T02:16:44.366 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-lprxmZh3kXHEAQ4 off=1535015 len=1748714 2021-07-31T02:16:44.374 INFO:tasks.workunit.client.0.smithi045.stdout:op 3511 completed, throughput=5MB/sec 2021-07-31T02:16:44.374 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-6uVKl3tVczrGXOc off=16690597 len=686267 2021-07-31T02:16:44.380 INFO:tasks.workunit.client.0.smithi045.stdout:op 3512 completed, throughput=5MB/sec 2021-07-31T02:16:44.380 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-QbVH4hW9ts-FmSc off=419709 len=879239 2021-07-31T02:16:44.381 INFO:tasks.workunit.client.0.smithi045.stdout:op 3513 completed, throughput=5MB/sec 2021-07-31T02:16:44.382 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-n2OSoKtUKLHYnlu off=409147 len=54724 2021-07-31T02:16:44.382 INFO:tasks.workunit.client.0.smithi045.stdout:op 3514 completed, throughput=5MB/sec 2021-07-31T02:16:44.383 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-KTTH0FB57C5D0p2 off=3247343 len=1296253 2021-07-31T02:16:44.384 INFO:tasks.workunit.client.0.smithi045.stdout:op 3515 completed, throughput=5MB/sec 2021-07-31T02:16:44.385 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-jVbGJEhVHHlwDSI off=15602299 len=1072734 2021-07-31T02:16:44.390 INFO:tasks.workunit.client.0.smithi045.stdout:op 3516 completed, throughput=5MB/sec 2021-07-31T02:16:44.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:44 smithi154 conmon[34553]: cluster 2021-07-31T02:16:42.214138+0000 mgr.x (mgr.24542) 1265 : cluster [DBG] pgmap v1756: 9 pgs: 9 active+clean; 127 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 3.0 MiB/s rd, 1.2 MiB/s wr, 4 op/s 2021-07-31T02:16:44.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:44 smithi045 conmon[33655]: cluster 2021-07-31T02:16:42.214138+0000 mgr.x (mgr.24542 2021-07-31T02:16:44.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:44 smithi045 conmon[33655]: ) 1265 : cluster [DBG] pgmap v1756: 9 pgs: 9 active+clean; 127 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 3.0 MiB/s rd, 1.2 MiB/s wr, 4 op/s 2021-07-31T02:16:44.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:44 smithi045 conmon[39718]: cluster 2021-07-31T02:16:42.214138+0000 mgr.x ( 2021-07-31T02:16:44.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:44 smithi045 conmon[39718]: mgr.24542) 1265 : cluster [DBG] pgmap v1756: 9 pgs: 9 active+clean; 127 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 3.0 MiB/s rd, 1.2 MiB/s wr, 4 op/s 2021-07-31T02:16:45.390 INFO:tasks.workunit.client.0.smithi045.stdout: 685: throughput=4.99MB/sec pending data=0 2021-07-31T02:16:45.391 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-A56qERtc66L5V-F off=939017 len=1913928 2021-07-31T02:16:45.395 INFO:tasks.workunit.client.0.smithi045.stdout:op 3517 completed, throughput=5MB/sec 2021-07-31T02:16:45.395 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj--diP7rzoe1Vys-z off=15204921 len=1680878 2021-07-31T02:16:45.403 INFO:tasks.workunit.client.0.smithi045.stdout:op 3518 completed, throughput=5MB/sec 2021-07-31T02:16:45.404 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Y3pX6OXITb4Ju4O off=2777124 len=1605972 2021-07-31T02:16:45.410 INFO:tasks.workunit.client.0.smithi045.stdout:op 3519 completed, throughput=5MB/sec 2021-07-31T02:16:46.411 INFO:tasks.workunit.client.0.smithi045.stdout: 686: throughput=4.99MB/sec pending data=0 2021-07-31T02:16:46.411 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-rXo3gDfx1V8TPXb off=12540152 len=1881156 2021-07-31T02:16:46.419 INFO:tasks.workunit.client.0.smithi045.stdout:op 3520 completed, throughput=5MB/sec 2021-07-31T02:16:46.419 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-N8loNVBEVfd50BS off=2921528 len=512356 2021-07-31T02:16:46.423 INFO:tasks.workunit.client.0.smithi045.stdout:op 3521 completed, throughput=5MB/sec 2021-07-31T02:16:46.424 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-yqmSsBhmQ6bvMeX off=8212458 len=1957144 2021-07-31T02:16:46.425 INFO:tasks.workunit.client.0.smithi045.stdout:op 3522 completed, throughput=5MB/sec 2021-07-31T02:16:46.426 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-AMBCjW_0hqAreHk off=12014397 len=625359 2021-07-31T02:16:46.431 INFO:tasks.workunit.client.0.smithi045.stdout:op 3523 completed, throughput=5MB/sec 2021-07-31T02:16:46.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:46 smithi154 conmon[34553]: cluster 2021-07-31T02:16:44.214665+0000 mgr.x (mgr.24542) 1266 : cluster [DBG] pgmap v1757: 9 pgs: 9 active+clean; 127 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 4.2 MiB/s rd, 2.1 MiB/s wr, 6 op/s 2021-07-31T02:16:46.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:46 smithi045 conmon[33655]: cluster 2021-07-31T02:16:44.214665+0000 mgr.x (mgr 2021-07-31T02:16:46.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:46 smithi045 conmon[33655]: .24542) 1266 : cluster [DBG] pgmap v1757: 9 pgs: 9 active+clean; 127 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 4.2 MiB/s rd, 2.1 MiB/s wr, 6 op/s 2021-07-31T02:16:46.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:46 smithi045 conmon[39718]: cluster 2021-07-31T02:16:44.214665 2021-07-31T02:16:46.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:46 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 1266 : cluster [DBG] pgmap v1757: 9 pgs: 9 active+clean; 127 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 4.2 MiB/s rd, 2.1 MiB/s wr, 6 op/s 2021-07-31T02:16:47.431 INFO:tasks.workunit.client.0.smithi045.stdout: 687: throughput=4.99MB/sec pending data=0 2021-07-31T02:16:47.432 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-tUalvqURPP-qKsb off=9809055 len=1420578 2021-07-31T02:16:47.439 INFO:tasks.workunit.client.0.smithi045.stdout:op 3524 completed, throughput=4.99MB/sec 2021-07-31T02:16:47.439 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-XGotu6msMaCmZF6 off=17259579 len=1922109 2021-07-31T02:16:47.443 INFO:tasks.workunit.client.0.smithi045.stdout:op 3525 completed, throughput=5MB/sec 2021-07-31T02:16:47.444 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-di0ApIs01Vhi1jN off=4459554 len=1693177 2021-07-31T02:16:47.450 INFO:tasks.workunit.client.0.smithi045.stdout:op 3526 completed, throughput=5MB/sec 2021-07-31T02:16:47.450 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-015kTsEEkq6UUYl off=4310953 len=792667 2021-07-31T02:16:47.452 INFO:tasks.workunit.client.0.smithi045.stdout:op 3527 completed, throughput=5MB/sec 2021-07-31T02:16:48.453 INFO:tasks.workunit.client.0.smithi045.stdout: 688: throughput=4.99MB/sec pending data=0 2021-07-31T02:16:48.453 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-wcZR0aSC38SpmxO off=5493977 len=42415 2021-07-31T02:16:48.454 INFO:tasks.workunit.client.0.smithi045.stdout:op 3528 completed, throughput=4.99MB/sec 2021-07-31T02:16:48.454 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-t0iTeGJScnezuJL off=17793530 len=437341 2021-07-31T02:16:48.455 INFO:tasks.workunit.client.0.smithi045.stdout:op 3529 completed, throughput=4.99MB/sec 2021-07-31T02:16:48.455 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-j-braWVE2TFcoP7 off=6894535 len=1285058 2021-07-31T02:16:48.461 INFO:tasks.workunit.client.0.smithi045.stdout:op 3530 completed, throughput=5MB/sec 2021-07-31T02:16:48.461 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-L0DwfUk29O_pvLD off=15837956 len=1454028 2021-07-31T02:16:48.469 INFO:tasks.workunit.client.0.smithi045.stdout:op 3531 completed, throughput=5MB/sec 2021-07-31T02:16:48.469 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-HZtzCyQmsecCGQ5 off=11268740 len=1616468 2021-07-31T02:16:48.471 INFO:tasks.workunit.client.0.smithi045.stdout:op 3532 completed, throughput=5MB/sec 2021-07-31T02:16:48.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:48 smithi154 conmon[34553]: cluster 2021-07-31T02:16:46.215070+0000 mgr.x (mgr.24542) 1267 : cluster [DBG] pgmap v1758: 9 pgs: 9 active+clean; 127 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 2.8 MiB/s rd, 1.4 MiB/s wr, 4 op/s 2021-07-31T02:16:48.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:48 smithi045 conmon[33655]: cluster 2021-07-31T02:16:46.215070+0000 mgr.x (mgr 2021-07-31T02:16:48.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:48 smithi045 conmon[33655]: .24542) 1267 : cluster [DBG] pgmap v1758: 9 pgs: 9 active+clean; 127 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 2.8 MiB/s rd, 1.4 MiB/s wr, 4 op/s 2021-07-31T02:16:48.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:48 smithi045 conmon[39718]: cluster 2021-07-31T02:16:46.215070+0000 mgr.x (mgr.24542) 1267 : 2021-07-31T02:16:48.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:48 smithi045 conmon[39718]: cluster [DBG] pgmap v1758: 9 pgs: 9 active+clean; 127 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 2.8 MiB/s rd, 1.4 MiB/s wr, 4 op/s 2021-07-31T02:16:49.388 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:16:49.472 INFO:tasks.workunit.client.0.smithi045.stdout: 689: throughput=4.99MB/sec pending data=0 2021-07-31T02:16:49.472 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-0yWBxy9ABci7Hkc off=829782 len=574525 2021-07-31T02:16:49.477 INFO:tasks.workunit.client.0.smithi045.stdout:op 3533 completed, throughput=4.99MB/sec 2021-07-31T02:16:49.477 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-5PwdPbQOarRLVL5 off=14948043 len=625256 2021-07-31T02:16:49.481 INFO:tasks.workunit.client.0.smithi045.stdout:op 3534 completed, throughput=4.99MB/sec 2021-07-31T02:16:49.481 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-eq5ur-QSX-UzNBr off=7665459 len=1259002 2021-07-31T02:16:49.487 INFO:tasks.workunit.client.0.smithi045.stdout:op 3535 completed, throughput=5MB/sec 2021-07-31T02:16:49.488 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-BrlTmek2_RvTG34 off=2694489 len=874598 2021-07-31T02:16:49.495 INFO:tasks.workunit.client.0.smithi045.stdout:op 3536 completed, throughput=5MB/sec 2021-07-31T02:16:49.496 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-3bIcuSw1iuE-VWk off=1081224 len=1785250 2021-07-31T02:16:49.504 INFO:tasks.workunit.client.0.smithi045.stdout:op 3537 completed, throughput=5MB/sec 2021-07-31T02:16:49.504 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Tu08iqyr-BT4oOT off=10857623 len=1162154 2021-07-31T02:16:49.506 INFO:tasks.workunit.client.0.smithi045.stdout:op 3538 completed, throughput=5MB/sec 2021-07-31T02:16:49.669 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:16:49.669 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 31m ago 32m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:16:49.670 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 29m ago 32m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:16:49.670 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 29m ago 35m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:16:49.670 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 31m ago 37m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:16:49.670 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 31m ago 37m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:16:49.671 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 29m ago 36m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:16:49.671 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 31m ago 36m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:16:49.671 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 31m ago 32m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:16:49.671 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 29m ago 32m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:16:49.671 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 31m ago 35m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:16:49.672 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 31m ago 35m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:16:49.672 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 31m ago 34m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:16:49.672 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 31m ago 34m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:16:49.672 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 29m ago 34m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:16:49.672 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 29m ago 33m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:16:49.673 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 29m ago 33m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:16:49.673 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 29m ago 33m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:16:49.673 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 29m ago 32m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:16:50.004 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:16:50.005 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:16:50.005 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:16:50.005 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:16:50.005 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:16:50.006 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:16:50.006 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:16:50.006 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:16:50.006 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:16:50.006 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:16:50.007 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:16:50.007 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:16:50.007 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:16:50.007 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:16:50.008 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:16:50.008 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:16:50.008 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:16:50.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:50 smithi154 conmon[34553]: cluster 2021-07-31T02:16:48.215431+0000 mgr.x (mgr.24542) 1268 : cluster [DBG] pgmap v1759: 9 pgs: 9 active+clean; 127 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 2.8 MiB/s rd, 1.4 MiB/s wr, 4 op/s 2021-07-31T02:16:50.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:50 smithi154 conmon[34553]: audit 2021-07-31T02:16:50.004752+0000 mon.a (mon.0) 1886 : audit [DBG] from='client.? 172.21.15.45:0/1286237226' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:16:50.510 INFO:tasks.workunit.client.0.smithi045.stdout: 690: throughput=4.99MB/sec pending data=0 2021-07-31T02:16:50.510 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-44hN1vXQgBjILZW off=9615387 len=1052027 2021-07-31T02:16:50.511 INFO:tasks.workunit.client.0.smithi045.stdout:op 3539 completed, throughput=5MB/sec 2021-07-31T02:16:50.511 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Z3VuY5tMR4Hj26j off=1225437 len=279291 2021-07-31T02:16:50.511 INFO:tasks.workunit.client.0.smithi045.stdout:op 3540 completed, throughput=5MB/sec 2021-07-31T02:16:50.511 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-PKQRRJWOWQG-Ecj off=8159973 len=1851164 2021-07-31T02:16:50.513 INFO:tasks.workunit.client.0.smithi045.stdout:op 3541 completed, throughput=5MB/sec 2021-07-31T02:16:50.513 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-oAWseaTpHXryReu off=6413449 len=493562 2021-07-31T02:16:50.516 INFO:tasks.workunit.client.0.smithi045.stdout:op 3542 completed, throughput=5MB/sec 2021-07-31T02:16:50.517 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-_NadweYEUAr_fhn off=13310920 len=96466 2021-07-31T02:16:50.518 INFO:tasks.workunit.client.0.smithi045.stdout:op 3543 completed, throughput=5MB/sec 2021-07-31T02:16:50.518 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-U1bHzm3E7Y31IKZ off=7927803 len=640217 2021-07-31T02:16:50.522 INFO:tasks.workunit.client.0.smithi045.stdout:op 3544 completed, throughput=5MB/sec 2021-07-31T02:16:50.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:50 smithi045 conmon[39718]: cluster 2021-07-31T02:16:48 2021-07-31T02:16:50.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:50 smithi045 conmon[39718]: .215431+0000 mgr.x (mgr.24542) 1268 : cluster [DBG] pgmap v1759: 9 pgs: 9 active+clean; 127 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 2.8 MiB/s rd, 1.4 MiB/s wr, 4 op/s 2021-07-31T02:16:50.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:50 smithi045 conmon[39718]: audit 2021-07-31T02:16:50. 2021-07-31T02:16:50.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:50 smithi045 conmon[39718]: 004752+0000 mon.a (mon.0) 1886 : audit 2021-07-31T02:16:50.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:50 smithi045 conmon[39718]: [DBG] from='client.? 172.21.15.45:0/1286237226' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:16:50.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:50 smithi045 conmon[33655]: cluster 2021-07-31T02:16:48.215431+0000 mgr.x (mgr.24542 2021-07-31T02:16:50.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:50 smithi045 conmon[33655]: ) 1268 : cluster [DBG] pgmap v1759: 9 pgs: 9 active+clean; 127 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 2.8 MiB/s rd, 1.4 MiB/s wr, 4 op/s 2021-07-31T02:16:50.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:50 smithi045 conmon[33655]: audit 2021-07-31T02:16:50.004752+0000 mon.a (mon.0) 1886 : audit [DBG] from='client.? 172.21.15.45:0/1286237226' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:16:51.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:51 smithi154 conmon[34553]: audit 2021-07-31T02:16:49.376806+0000 mgr.x (mgr.24542) 1269 : 2021-07-31T02:16:51.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:51 smithi154 conmon[34553]: audit [DBG] from='client.16131 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:16:51.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:51 smithi154 conmon[34553]: audit 2021-07-31T02:16:49.661362+0000 mgr.x (mgr.24542) 1270 : audit [DBG] from='client.26044 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:16:51.522 INFO:tasks.workunit.client.0.smithi045.stdout: 691: throughput=4.99MB/sec pending data=0 2021-07-31T02:16:51.523 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-FTR80Cik2n4JpoI off=14248164 len=1303183 2021-07-31T02:16:51.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:51 smithi045 conmon[39718]: audit 2021-07-31T02:16:49.376806 2021-07-31T02:16:51.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:51 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 1269 : audit [DBG] from='client.16131 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:16:51.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:51 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T02:16:51.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:51 smithi045 conmon[39718]: T02:16:49.661362+0000 mgr.x (mgr.24542) 2021-07-31T02:16:51.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:51 smithi045 conmon[39718]: 1270 : audit [DBG] from='client.26044 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:16:51.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:51 smithi045 conmon[33655]: audit 2021-07-31T02:16:49.376806+0000 mgr.x (mgr.24542) 1269 : audit 2021-07-31T02:16:51.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:51 smithi045 conmon[33655]: [DBG] from='client.16131 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:16:51.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:51 smithi045 conmon[33655]: audit 2021-07-31T02:16:49.661362+0000 mgr.x (mgr.24542) 1270 : audit [DBG] from='client.26044 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:16:51.528 INFO:tasks.workunit.client.0.smithi045.stdout:op 3545 completed, throughput=4.99MB/sec 2021-07-31T02:16:51.528 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-RYUIbdiQMr-Yb9k off=222862 len=1844368 2021-07-31T02:16:51.528 INFO:tasks.workunit.client.0.smithi045.stdout:op 3546 completed, throughput=5MB/sec 2021-07-31T02:16:51.528 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-5D6_wmi6K6iU_Us off=12917258 len=2058490 2021-07-31T02:16:51.540 INFO:tasks.workunit.client.0.smithi045.stdout:op 3547 completed, throughput=5MB/sec 2021-07-31T02:16:51.540 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-E14WJkE9j0cJysc off=484349 len=1025095 2021-07-31T02:16:51.545 INFO:tasks.workunit.client.0.smithi045.stdout:op 3548 completed, throughput=5MB/sec 2021-07-31T02:16:52.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:52 smithi154 conmon[34553]: cluster 2021-07-31T02:16:50.215967+0000 mgr.x (mgr.24542) 1271 : cluster [DBG] 2021-07-31T02:16:52.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:52 smithi154 conmon[34553]: pgmap v1760: 9 pgs: 9 active+clean; 127 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 3.7 MiB/s rd, 2.6 MiB/s wr, 6 op/s 2021-07-31T02:16:52.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:52 smithi045 conmon[39718]: cluster 2021-07-31T02:16:50.215967+0000 mgr.x (mgr.24542) 2021-07-31T02:16:52.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:52 smithi045 conmon[39718]: 1271 : cluster [DBG] pgmap v1760: 9 pgs: 9 active+clean; 127 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 3.7 MiB/s rd, 2.6 MiB/s wr, 6 op/s 2021-07-31T02:16:52.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:52 smithi045 conmon[33655]: cluster 2021-07-31T02:16:50.215967+0000 mgr.x (mgr.24542) 1271 : cluster [DBG] pgmap v1760: 9 pgs: 9 active+clean; 127 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 3.7 MiB/s rd, 2.6 MiB/s wr, 6 op/s 2021-07-31T02:16:52.545 INFO:tasks.workunit.client.0.smithi045.stdout: 692: throughput=4.99MB/sec pending data=0 2021-07-31T02:16:52.545 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Hakz9AUx5jPnNG3 off=8560726 len=1130971 2021-07-31T02:16:52.553 INFO:tasks.workunit.client.0.smithi045.stdout:op 3549 completed, throughput=5MB/sec 2021-07-31T02:16:52.553 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-o2iih3OsN8q3wk- off=20897809 len=1051396 2021-07-31T02:16:52.559 INFO:tasks.workunit.client.0.smithi045.stdout:op 3550 completed, throughput=5MB/sec 2021-07-31T02:16:52.559 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-gjwIa1syRNSrdM_ off=639269 len=371120 2021-07-31T02:16:52.560 INFO:tasks.workunit.client.0.smithi045.stdout:op 3551 completed, throughput=5MB/sec 2021-07-31T02:16:52.561 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-XUatAoyDM8glY6j off=1875130 len=1215505 2021-07-31T02:16:52.566 INFO:tasks.workunit.client.0.smithi045.stdout:op 3552 completed, throughput=5MB/sec 2021-07-31T02:16:52.566 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-6UuxXZovZ-c5NTj off=16898738 len=1836753 2021-07-31T02:16:52.575 INFO:tasks.workunit.client.0.smithi045.stdout:op 3553 completed, throughput=5MB/sec 2021-07-31T02:16:53.576 INFO:tasks.workunit.client.0.smithi045.stdout: 693: throughput=4.99MB/sec pending data=0 2021-07-31T02:16:53.576 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-A--oa_Zvg-maSkb off=399574 len=465453 2021-07-31T02:16:53.577 INFO:tasks.workunit.client.0.smithi045.stdout:op 3554 completed, throughput=5MB/sec 2021-07-31T02:16:53.577 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-o1w4GhD8F5bQD2r off=2983118 len=2027200 2021-07-31T02:16:53.579 INFO:tasks.workunit.client.0.smithi045.stdout:op 3555 completed, throughput=5MB/sec 2021-07-31T02:16:53.579 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-1TrkDBFVJCpt0Cj off=9485594 len=1517403 2021-07-31T02:16:53.587 INFO:tasks.workunit.client.0.smithi045.stdout:op 3556 completed, throughput=5MB/sec 2021-07-31T02:16:53.587 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-8UlfADz1UCDURQZ off=5955096 len=193720 2021-07-31T02:16:53.590 INFO:tasks.workunit.client.0.smithi045.stdout:op 3557 completed, throughput=5MB/sec 2021-07-31T02:16:54.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:16:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:16:53.559Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:16:54.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:16:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:16:53.559Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:16:54.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:16:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:16:53.559Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:16:54.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:54 smithi154 conmon[34553]: cluster 2021-07-31T02:16:52.216375+0000 mgr.x (mgr.24542) 1272 : cluster [DBG] 2021-07-31T02:16:54.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:54 smithi154 conmon[34553]: pgmap v1761: 9 pgs: 9 active+clean; 127 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2021-07-31T02:16:54.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:54 smithi045 conmon[39718]: cluster 2021-07-31T02:16:52.216375+0000 mgr.x ( 2021-07-31T02:16:54.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:54 smithi045 conmon[39718]: mgr.24542) 1272 : cluster [DBG] pgmap v1761: 9 pgs: 9 active+clean; 127 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2021-07-31T02:16:54.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:54 smithi045 conmon[33655]: cluster 2021-07-31T02:16:52.216375+0000 mgr.x (mgr.24542) 1272 : cluster [DBG] pgmap v1761: 9 pgs: 9 active+clean; 127 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2021-07-31T02:16:54.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:54 smithi045 conmon[33655]: 2021-07-31T02:16:54.590 INFO:tasks.workunit.client.0.smithi045.stdout: 694: throughput=4.99MB/sec pending data=0 2021-07-31T02:16:54.591 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-f95mfDejz5yHwyo off=4273829 len=213745 2021-07-31T02:16:54.591 INFO:tasks.workunit.client.0.smithi045.stdout:op 3558 completed, throughput=4.99MB/sec 2021-07-31T02:16:54.592 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-NKycwjhTwgIGp12 off=2750562 len=1517820 2021-07-31T02:16:54.593 INFO:tasks.workunit.client.0.smithi045.stdout:op 3559 completed, throughput=5MB/sec 2021-07-31T02:16:54.593 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-uNZs9MWQKUUwv2Y off=2627251 len=126633 2021-07-31T02:16:54.596 INFO:tasks.workunit.client.0.smithi045.stdout:op 3560 completed, throughput=5MB/sec 2021-07-31T02:16:54.596 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-NyMhyJtgsgKUcvK off=2582618 len=765727 2021-07-31T02:16:54.601 INFO:tasks.workunit.client.0.smithi045.stdout:op 3561 completed, throughput=5MB/sec 2021-07-31T02:16:54.601 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-HlQnJgrYxiiCTqj off=16643509 len=640982 2021-07-31T02:16:54.602 INFO:tasks.workunit.client.0.smithi045.stdout:op 3562 completed, throughput=5MB/sec 2021-07-31T02:16:54.602 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Czo08kHMF6NGFxp off=2450574 len=223724 2021-07-31T02:16:54.605 INFO:tasks.workunit.client.0.smithi045.stdout:op 3563 completed, throughput=5MB/sec 2021-07-31T02:16:54.606 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-hjkjc9iyaGcDbtk off=2516837 len=1620431 2021-07-31T02:16:54.613 INFO:tasks.workunit.client.0.smithi045.stdout:op 3564 completed, throughput=5MB/sec 2021-07-31T02:16:54.613 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-BUzLgFqf-aQPqDs off=352950 len=661354 2021-07-31T02:16:54.614 INFO:tasks.workunit.client.0.smithi045.stdout:op 3565 completed, throughput=5MB/sec 2021-07-31T02:16:55.615 INFO:tasks.workunit.client.0.smithi045.stdout: 695: throughput=4.99MB/sec pending data=0 2021-07-31T02:16:55.616 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-r_60zpxLFDslL76 off=8229827 len=41256 2021-07-31T02:16:55.618 INFO:tasks.workunit.client.0.smithi045.stdout:op 3566 completed, throughput=4.99MB/sec 2021-07-31T02:16:55.619 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-rKJa-aaaLZtWaTs off=4656880 len=1638703 2021-07-31T02:16:55.621 INFO:tasks.workunit.client.0.smithi045.stdout:op 3567 completed, throughput=5MB/sec 2021-07-31T02:16:55.622 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-F-AjKAo8r8nVz_o off=12354492 len=1823851 2021-07-31T02:16:55.630 INFO:tasks.workunit.client.0.smithi045.stdout:op 3568 completed, throughput=5MB/sec 2021-07-31T02:16:55.630 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-vXzCUBmMSc8MXOu off=14217789 len=779969 2021-07-31T02:16:55.635 INFO:tasks.workunit.client.0.smithi045.stdout:op 3569 completed, throughput=5MB/sec 2021-07-31T02:16:55.635 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-svNzi6CCKNghXjB off=834340 len=449990 2021-07-31T02:16:55.636 INFO:tasks.workunit.client.0.smithi045.stdout:op 3570 completed, throughput=5MB/sec 2021-07-31T02:16:55.636 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Ns9ZO9TWPTHH_N5 off=23247153 len=700149 2021-07-31T02:16:55.639 INFO:tasks.workunit.client.0.smithi045.stdout:op 3571 completed, throughput=5MB/sec 2021-07-31T02:16:56.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:56 smithi154 conmon[34553]: cluster 2021-07-31T02:16:54.216872+0000 mgr.x (mgr.24542) 1273 : cluster [DBG] pgmap v1762: 9 pgs: 9 active+clean; 127 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 2.9 MiB/s rd, 3.4 MiB/s wr, 6 op/s 2021-07-31T02:16:56.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:56 smithi045 conmon[39718]: cluster 2021-07-31T02:16:54.216872+0000 mgr.x ( 2021-07-31T02:16:56.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:56 smithi045 conmon[39718]: mgr.24542) 1273 : cluster [DBG] pgmap v1762: 9 pgs: 9 active+clean; 127 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 2.9 MiB/s rd, 3.4 MiB/s wr, 6 op/s 2021-07-31T02:16:56.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:56 smithi045 conmon[33655]: cluster 2021-07-31T02:16:54.216872+0000 mgr.x (mgr.24542) 2021-07-31T02:16:56.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:56 smithi045 conmon[33655]: 1273 : cluster [DBG] pgmap v1762: 9 pgs: 9 active+clean; 127 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 2.9 MiB/s rd, 3.4 MiB/s wr, 6 op/s 2021-07-31T02:16:56.640 INFO:tasks.workunit.client.0.smithi045.stdout: 696: throughput=4.99MB/sec pending data=0 2021-07-31T02:16:56.640 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-3BEtQU8JUMgBuhO off=13684955 len=868840 2021-07-31T02:16:56.645 INFO:tasks.workunit.client.0.smithi045.stdout:op 3572 completed, throughput=4.99MB/sec 2021-07-31T02:16:56.645 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-QKmOX2UJUY6nFw5 off=12877279 len=1231533 2021-07-31T02:16:56.654 INFO:tasks.workunit.client.0.smithi045.stdout:op 3573 completed, throughput=5MB/sec 2021-07-31T02:16:56.654 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-cN1yBohj3NL3WYl off=13746470 len=1439561 2021-07-31T02:16:56.661 INFO:tasks.workunit.client.0.smithi045.stdout:op 3574 completed, throughput=5MB/sec 2021-07-31T02:16:56.661 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-EXtA0i06Wi1_OAW off=1972905 len=2068739 2021-07-31T02:16:56.664 INFO:tasks.workunit.client.0.smithi045.stdout:op 3575 completed, throughput=5MB/sec 2021-07-31T02:16:57.665 INFO:tasks.workunit.client.0.smithi045.stdout: 697: throughput=4.99MB/sec pending data=0 2021-07-31T02:16:57.665 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-atEKW9K0DtSZmoO off=2475069 len=1227514 2021-07-31T02:16:57.667 INFO:tasks.workunit.client.0.smithi045.stdout:op 3576 completed, throughput=5MB/sec 2021-07-31T02:16:57.667 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ZXfMGpdC5MdOKU3 off=7896532 len=585223 2021-07-31T02:16:57.672 INFO:tasks.workunit.client.0.smithi045.stdout:op 3577 completed, throughput=5MB/sec 2021-07-31T02:16:57.672 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-HuC0_tvxFMQQV90 off=12018192 len=2089252 2021-07-31T02:16:57.676 INFO:tasks.workunit.client.0.smithi045.stdout:op 3578 completed, throughput=5MB/sec 2021-07-31T02:16:57.676 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-66bq0a3uXhx_C3_ off=2668826 len=297781 2021-07-31T02:16:57.679 INFO:tasks.workunit.client.0.smithi045.stdout:op 3579 completed, throughput=5MB/sec 2021-07-31T02:16:57.679 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-VfKb22Lb_dYfGUe off=4590640 len=1764306 2021-07-31T02:16:57.686 INFO:tasks.workunit.client.0.smithi045.stdout:op 3580 completed, throughput=5MB/sec 2021-07-31T02:16:58.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:16:58 smithi154 conmon[34553]: cluster 2021-07-31T02:16:56.217290+0000 mgr.x (mgr.24542) 1274 : cluster [DBG] pgmap v1763: 9 pgs: 9 active+clean; 127 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.5 MiB/s wr, 3 op/s 2021-07-31T02:16:58.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:58 smithi045 conmon[33655]: cluster 2021-07-31T02:16:56.217290+0000 mgr.x (mgr.24542) 1274 2021-07-31T02:16:58.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:16:58 smithi045 conmon[33655]: : cluster [DBG] pgmap v1763: 9 pgs: 9 active+clean; 127 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.5 MiB/s wr, 3 op/s 2021-07-31T02:16:58.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:58 smithi045 conmon[39718]: cluster 2021-07-31T02:16:56.217290+0000 mgr.x 2021-07-31T02:16:58.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:16:58 smithi045 conmon[39718]: (mgr.24542) 1274 : cluster [DBG] pgmap v1763: 9 pgs: 9 active+clean; 127 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.5 MiB/s wr, 3 op/s 2021-07-31T02:16:58.687 INFO:tasks.workunit.client.0.smithi045.stdout: 698: throughput=4.99MB/sec pending data=0 2021-07-31T02:16:58.687 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-MOkk9LOnYoLm4ov off=14125444 len=494133 2021-07-31T02:16:58.691 INFO:tasks.workunit.client.0.smithi045.stdout:op 3581 completed, throughput=5MB/sec 2021-07-31T02:16:58.691 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-3oTls6vFTqOAQKQ off=4276297 len=547349 2021-07-31T02:16:58.692 INFO:tasks.workunit.client.0.smithi045.stdout:op 3582 completed, throughput=5MB/sec 2021-07-31T02:16:58.692 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-VOA2WvRgCDvXOvE off=3167105 len=1949048 2021-07-31T02:16:58.701 INFO:tasks.workunit.client.0.smithi045.stdout:op 3583 completed, throughput=5MB/sec 2021-07-31T02:16:58.702 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-t-UDIjXBG-ddkEF off=7212037 len=1129698 2021-07-31T02:16:58.707 INFO:tasks.workunit.client.0.smithi045.stdout:op 3584 completed, throughput=5MB/sec 2021-07-31T02:16:59.707 INFO:tasks.workunit.client.0.smithi045.stdout: 699: throughput=4.99MB/sec pending data=0 2021-07-31T02:16:59.708 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-LkK7ntiklM2hePm off=2046120 len=1260383 2021-07-31T02:16:59.709 INFO:tasks.workunit.client.0.smithi045.stdout:op 3585 completed, throughput=4.99MB/sec 2021-07-31T02:16:59.710 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-v8fpJbbwy1jkeAt off=555486 len=392960 2021-07-31T02:16:59.711 INFO:tasks.workunit.client.0.smithi045.stdout:op 3586 completed, throughput=5MB/sec 2021-07-31T02:16:59.711 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-af-qGER6OfhjpJh off=0 len=1099051 2021-07-31T02:16:59.717 INFO:tasks.workunit.client.0.smithi045.stdout:op 3587 completed, throughput=5MB/sec 2021-07-31T02:16:59.717 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-j5suao-vFSnzFcg off=19882693 len=1271494 2021-07-31T02:16:59.719 INFO:tasks.workunit.client.0.smithi045.stdout:op 3588 completed, throughput=5MB/sec 2021-07-31T02:16:59.720 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Y2oo4nPEcIkTLeg off=153060 len=949220 2021-07-31T02:16:59.725 INFO:tasks.workunit.client.0.smithi045.stdout:op 3589 completed, throughput=5MB/sec 2021-07-31T02:16:59.725 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-uuONZP4Igvp31fY off=6215750 len=1374760 2021-07-31T02:16:59.730 INFO:tasks.workunit.client.0.smithi045.stdout:op 3590 completed, throughput=5MB/sec 2021-07-31T02:17:00.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:00 smithi154 conmon[34553]: cluster 2021-07-31T02:16:58.217623+0000 mgr.x (mgr.24542) 1275 : cluster [DBG] pgmap v1764: 9 pgs: 9 active+clean; 127 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.5 MiB/s wr, 3 op/s 2021-07-31T02:17:00.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:00 smithi045 conmon[33655]: cluster 2021-07-31T02:16:58.217623+0000 mgr.x (mgr.24542) 1275 : cluster [DBG] pgmap v1764: 9 pgs: 9 active+clean; 127 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.5 MiB/s wr, 3 op/s 2021-07-31T02:17:00.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:00 smithi045 conmon[39718]: cluster 2021-07-31T02:16:58.217623+0000 mgr.x (mgr.24542) 1275 : cluster [DBG] pgmap v1764: 9 pgs: 9 active+clean; 127 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.5 MiB/s wr, 3 op/s 2021-07-31T02:17:00.731 INFO:tasks.workunit.client.0.smithi045.stdout: 700: throughput=4.99MB/sec pending data=0 2021-07-31T02:17:00.731 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-4UEQNqeRGaUXP3l off=92905 len=1132931 2021-07-31T02:17:00.737 INFO:tasks.workunit.client.0.smithi045.stdout:op 3591 completed, throughput=5MB/sec 2021-07-31T02:17:00.737 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Uyq6_h1LGFbj1QU off=7010338 len=1292897 2021-07-31T02:17:00.740 INFO:tasks.workunit.client.0.smithi045.stdout:op 3592 completed, throughput=5MB/sec 2021-07-31T02:17:00.740 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-cHb5kdIz50evCSU off=322916 len=129579 2021-07-31T02:17:00.743 INFO:tasks.workunit.client.0.smithi045.stdout:op 3593 completed, throughput=5MB/sec 2021-07-31T02:17:00.743 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-b_4oiYLNsndho0V off=3100360 len=459351 2021-07-31T02:17:00.744 INFO:tasks.workunit.client.0.smithi045.stdout:op 3594 completed, throughput=5MB/sec 2021-07-31T02:17:00.745 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-sQROl6_wqmMS8ad off=17957933 len=157361 2021-07-31T02:17:00.747 INFO:tasks.workunit.client.0.smithi045.stdout:op 3595 completed, throughput=5MB/sec 2021-07-31T02:17:00.748 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-BEXFG8d2toofyiY off=24015656 len=1344384 2021-07-31T02:17:00.749 INFO:tasks.workunit.client.0.smithi045.stdout:op 3596 completed, throughput=5MB/sec 2021-07-31T02:17:01.749 INFO:tasks.workunit.client.0.smithi045.stdout: 701: throughput=4.99MB/sec pending data=0 2021-07-31T02:17:01.750 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-CbYT2oGCvvDoPm3 off=2593165 len=516251 2021-07-31T02:17:01.753 INFO:tasks.workunit.client.0.smithi045.stdout:op 3597 completed, throughput=4.99MB/sec 2021-07-31T02:17:01.753 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-qCVpTTukKAswMb3 off=149820 len=153754 2021-07-31T02:17:01.754 INFO:tasks.workunit.client.0.smithi045.stdout:op 3598 completed, throughput=4.99MB/sec 2021-07-31T02:17:01.754 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-XljygX5xR7VBQ_j off=1891236 len=606969 2021-07-31T02:17:01.755 INFO:tasks.workunit.client.0.smithi045.stdout:op 3599 completed, throughput=5MB/sec 2021-07-31T02:17:01.755 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-zuaclCPPtLrJWhj off=11966843 len=120388 2021-07-31T02:17:01.758 INFO:tasks.workunit.client.0.smithi045.stdout:op 3600 completed, throughput=5MB/sec 2021-07-31T02:17:01.758 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-qynU59K4tCOf-Mv off=3531314 len=924168 2021-07-31T02:17:01.763 INFO:tasks.workunit.client.0.smithi045.stdout:op 3601 completed, throughput=5MB/sec 2021-07-31T02:17:01.764 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-nqfJo7xm_1YEbbQ off=8279641 len=1829113 2021-07-31T02:17:01.771 INFO:tasks.workunit.client.0.smithi045.stdout:op 3602 completed, throughput=5MB/sec 2021-07-31T02:17:01.772 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-X--RYU5IHj1rXAp off=297812 len=1179934 2021-07-31T02:17:01.774 INFO:tasks.workunit.client.0.smithi045.stdout:op 3603 completed, throughput=5MB/sec 2021-07-31T02:17:02.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:02 smithi154 conmon[34553]: cluster 2021-07-31T02:17:00.218121+0000 mgr.x (mgr.24542) 1276 : cluster [DBG] pgmap v1765: 9 pgs: 9 active+clean; 127 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 2.6 MiB/s rd, 3.8 MiB/s wr, 6 op/s 2021-07-31T02:17:02.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:02 smithi045 conmon[33655]: cluster 2021-07-31T02:17:00.218121+0000 mgr.x (mgr.24542) 1276 : 2021-07-31T02:17:02.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:02 smithi045 conmon[33655]: cluster [DBG] pgmap v1765: 9 pgs: 9 active+clean; 127 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 2.6 MiB/s rd, 3.8 MiB/s wr, 6 op/s 2021-07-31T02:17:02.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:02 smithi045 conmon[39718]: cluster 2021-07-31T02:17:00.218121+0000 mgr.x 2021-07-31T02:17:02.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:02 smithi045 conmon[39718]: (mgr.24542) 1276 : cluster [DBG] pgmap v1765: 9 pgs: 9 active+clean; 127 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 2.6 MiB/s rd, 3.8 MiB/s wr, 6 op/s 2021-07-31T02:17:02.774 INFO:tasks.workunit.client.0.smithi045.stdout: 702: throughput=4.99MB/sec pending data=0 2021-07-31T02:17:02.774 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-bzXyHhMCzJhTgJr off=5043044 len=574420 2021-07-31T02:17:02.778 INFO:tasks.workunit.client.0.smithi045.stdout:op 3604 completed, throughput=4.99MB/sec 2021-07-31T02:17:02.778 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-54PZmK-2oHredxP off=12745165 len=1554336 2021-07-31T02:17:02.780 INFO:tasks.workunit.client.0.smithi045.stdout:op 3605 completed, throughput=5MB/sec 2021-07-31T02:17:02.780 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ue84fEQqPNjLsYp off=6021671 len=166472 2021-07-31T02:17:02.783 INFO:tasks.workunit.client.0.smithi045.stdout:op 3606 completed, throughput=5MB/sec 2021-07-31T02:17:02.783 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-lbqPLd3YodLcvSG off=1540309 len=521865 2021-07-31T02:17:02.784 INFO:tasks.workunit.client.0.smithi045.stdout:op 3607 completed, throughput=5MB/sec 2021-07-31T02:17:02.784 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-RXWtrSgcZ6k8TfS off=14015904 len=1055865 2021-07-31T02:17:02.790 INFO:tasks.workunit.client.0.smithi045.stdout:op 3608 completed, throughput=5MB/sec 2021-07-31T02:17:02.790 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Db7c6LJHwQNocTj off=507348 len=255203 2021-07-31T02:17:02.792 INFO:tasks.workunit.client.0.smithi045.stdout:op 3609 completed, throughput=5MB/sec 2021-07-31T02:17:02.793 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Jdziy_x-FGyn3Hk off=3794336 len=708818 2021-07-31T02:17:02.793 INFO:tasks.workunit.client.0.smithi045.stdout:op 3610 completed, throughput=5MB/sec 2021-07-31T02:17:02.794 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-MXVV_8-cYLdtHRo off=14276194 len=1669294 2021-07-31T02:17:02.796 INFO:tasks.workunit.client.0.smithi045.stdout:op 3611 completed, throughput=5MB/sec 2021-07-31T02:17:03.797 INFO:tasks.workunit.client.0.smithi045.stdout: 703: throughput=4.99MB/sec pending data=0 2021-07-31T02:17:03.797 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-msb4qtmjNCG7QG- off=10283858 len=848744 2021-07-31T02:17:03.802 INFO:tasks.workunit.client.0.smithi045.stdout:op 3612 completed, throughput=5MB/sec 2021-07-31T02:17:03.802 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-pUcIRYLtC2JWAs6 off=6030351 len=1618770 2021-07-31T02:17:03.810 INFO:tasks.workunit.client.0.smithi045.stdout:op 3613 completed, throughput=5MB/sec 2021-07-31T02:17:03.810 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-m8Z7_mdYCQkydMz off=14929063 len=989513 2021-07-31T02:17:03.814 INFO:tasks.workunit.client.0.smithi045.stdout:op 3614 completed, throughput=5MB/sec 2021-07-31T02:17:03.815 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ncEOOiTzOzMbGGj off=3590123 len=1250531 2021-07-31T02:17:03.817 INFO:tasks.workunit.client.0.smithi045.stdout:op 3615 completed, throughput=5MB/sec 2021-07-31T02:17:04.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:17:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:17:03.559Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:17:04.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:17:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:17:03.559Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:17:04.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:17:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:17:03.560Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:17:04.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:04 smithi154 conmon[34553]: cluster 2021-07-31T02:17:02.218546+0000 mgr.x (mgr.24542) 1277 : cluster [DBG] pgmap v1766: 9 pgs: 9 active+clean; 127 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2021-07-31T02:17:04.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:04 smithi045 conmon[39718]: cluster 2021-07-31T02:17:02.218546+0000 mgr.x (mgr.24542 2021-07-31T02:17:04.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:04 smithi045 conmon[39718]: ) 1277 : cluster [DBG] pgmap v1766: 9 pgs: 9 active+clean; 127 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2021-07-31T02:17:04.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:04 smithi045 conmon[33655]: cluster 2021-07-31T02:17:02.218546+0000 mgr.x (mgr 2021-07-31T02:17:04.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:04 smithi045 conmon[33655]: .24542) 1277 : cluster [DBG] pgmap v1766: 9 pgs: 9 active+clean; 127 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2021-07-31T02:17:04.818 INFO:tasks.workunit.client.0.smithi045.stdout: 704: throughput=4.99MB/sec pending data=0 2021-07-31T02:17:04.819 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-p3iUxT-3Bc0_GPc off=7730711 len=1004234 2021-07-31T02:17:04.819 INFO:tasks.workunit.client.0.smithi045.stdout:op 3616 completed, throughput=5MB/sec 2021-07-31T02:17:04.820 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-xburYaqCnqEH4o9 off=1604969 len=298133 2021-07-31T02:17:04.823 INFO:tasks.workunit.client.0.smithi045.stdout:op 3617 completed, throughput=5MB/sec 2021-07-31T02:17:04.823 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-q4R4OxZ-lnfx8_6 off=5845702 len=68727 2021-07-31T02:17:04.825 INFO:tasks.workunit.client.0.smithi045.stdout:op 3618 completed, throughput=5MB/sec 2021-07-31T02:17:04.825 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-fiwik-Aul9SOcvS off=508 len=1692396 2021-07-31T02:17:04.827 INFO:tasks.workunit.client.0.smithi045.stdout:op 3619 completed, throughput=5MB/sec 2021-07-31T02:17:04.828 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ee5pg5kDK7331Xa off=3975941 len=1942862 2021-07-31T02:17:04.829 INFO:tasks.workunit.client.0.smithi045.stdout:op 3620 completed, throughput=5MB/sec 2021-07-31T02:17:05.830 INFO:tasks.workunit.client.0.smithi045.stdout: 705: throughput=4.99MB/sec pending data=0 2021-07-31T02:17:05.830 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-4A6NIHAZlI1kU3Y off=9015298 len=193235 2021-07-31T02:17:05.833 INFO:tasks.workunit.client.0.smithi045.stdout:op 3621 completed, throughput=4.99MB/sec 2021-07-31T02:17:05.834 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-a1kQCx6L4GCTi5s off=3016382 len=700934 2021-07-31T02:17:05.839 INFO:tasks.workunit.client.0.smithi045.stdout:op 3622 completed, throughput=4.99MB/sec 2021-07-31T02:17:05.839 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-YHKH4N9bXpBc5u7 off=898032 len=566414 2021-07-31T02:17:05.842 INFO:tasks.workunit.client.0.smithi045.stdout:op 3623 completed, throughput=5MB/sec 2021-07-31T02:17:05.843 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-epdKd7Jx41Eh9zE off=4797652 len=481569 2021-07-31T02:17:05.844 INFO:tasks.workunit.client.0.smithi045.stdout:op 3624 completed, throughput=5MB/sec 2021-07-31T02:17:05.845 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-SrBumC4zogdfjgg off=2495054 len=749837 2021-07-31T02:17:05.846 INFO:tasks.workunit.client.0.smithi045.stdout:op 3625 completed, throughput=5MB/sec 2021-07-31T02:17:05.846 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-39bJ2bcMUUOoUyI off=67783 len=1986752 2021-07-31T02:17:05.849 INFO:tasks.workunit.client.0.smithi045.stdout:op 3626 completed, throughput=5MB/sec 2021-07-31T02:17:05.849 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-xc455aO7uoTOC_p off=20991383 len=793720 2021-07-31T02:17:05.850 INFO:tasks.workunit.client.0.smithi045.stdout:op 3627 completed, throughput=5MB/sec 2021-07-31T02:17:06.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:06 smithi154 conmon[34553]: cluster 2021-07-31T02:17:04.219015+0000 mgr.x (mgr.24542) 1278 : cluster [DBG] pgmap v1767: 9 pgs: 9 active+clean; 127 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 2.8 MiB/s rd, 3.6 MiB/s wr, 6 op/s 2021-07-31T02:17:06.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:06 smithi045 conmon[39718]: cluster 2021-07-31T02:17:04.219015+0000 mgr.x (mgr.24542) 1278 2021-07-31T02:17:06.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:06 smithi045 conmon[39718]: : cluster [DBG] pgmap v1767: 9 pgs: 9 active+clean; 127 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 2.8 MiB/s rd, 3.6 MiB/s wr, 6 op/s 2021-07-31T02:17:06.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:06 smithi045 conmon[33655]: cluster 2021-07-31T02:17:04.219015 2021-07-31T02:17:06.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:06 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 1278 : cluster [DBG] pgmap v1767: 9 pgs: 9 active+clean; 127 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 2.8 MiB/s rd, 3.6 MiB/s wr, 6 op/s 2021-07-31T02:17:06.850 INFO:tasks.workunit.client.0.smithi045.stdout: 706: throughput=4.99MB/sec pending data=0 2021-07-31T02:17:06.851 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-FyXNqxoGF97028h off=5246871 len=188882 2021-07-31T02:17:06.854 INFO:tasks.workunit.client.0.smithi045.stdout:op 3628 completed, throughput=4.99MB/sec 2021-07-31T02:17:06.854 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-92XN7f9HJ3xLGlD off=233971 len=1818049 2021-07-31T02:17:06.865 INFO:tasks.workunit.client.0.smithi045.stdout:op 3629 completed, throughput=5MB/sec 2021-07-31T02:17:06.865 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-sJWsdrkiJHLCciV off=9670868 len=753429 2021-07-31T02:17:06.870 INFO:tasks.workunit.client.0.smithi045.stdout:op 3630 completed, throughput=5MB/sec 2021-07-31T02:17:06.871 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-JLPK6UjvzH5gcak off=8339931 len=913862 2021-07-31T02:17:06.875 INFO:tasks.workunit.client.0.smithi045.stdout:op 3631 completed, throughput=5MB/sec 2021-07-31T02:17:06.875 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-idc-m6DkN8z0j2Z off=8653129 len=1969709 2021-07-31T02:17:06.878 INFO:tasks.workunit.client.0.smithi045.stdout:op 3632 completed, throughput=5MB/sec 2021-07-31T02:17:07.879 INFO:tasks.workunit.client.0.smithi045.stdout: 707: throughput=4.99MB/sec pending data=0 2021-07-31T02:17:07.879 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-xFkSMj7npiVR3Yb off=6534780 len=2058342 2021-07-31T02:17:07.883 INFO:tasks.workunit.client.0.smithi045.stdout:op 3633 completed, throughput=5MB/sec 2021-07-31T02:17:07.883 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-fEsDtbejrAallKG off=4772207 len=1656612 2021-07-31T02:17:07.891 INFO:tasks.workunit.client.0.smithi045.stdout:op 3634 completed, throughput=5MB/sec 2021-07-31T02:17:07.891 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-B3u4kcbm2AvYvOk off=1435778 len=1043059 2021-07-31T02:17:07.892 INFO:tasks.workunit.client.0.smithi045.stdout:op 3635 completed, throughput=5MB/sec 2021-07-31T02:17:08.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:08 smithi154 conmon[34553]: cluster 2021-07-31T02:17:06.219391+0000 mgr.x (mgr.24542) 1279 : cluster [DBG] pgmap v1768: 9 pgs: 9 active+clean; 127 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:17:08.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:08 smithi154 conmon[34553]: 2021-07-31T02:17:08.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:08 smithi045 conmon[39718]: cluster 2021-07-31T02: 2021-07-31T02:17:08.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:08 smithi045 conmon[39718]: 17:06.219391+0000 mgr.x (mgr.24542) 1279 : cluster [DBG] pgmap v1768: 9 pgs: 9 active+clean; 127 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:17:08.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:08 smithi045 conmon[33655]: cluster 2021-07-31T02:17: 2021-07-31T02:17:08.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:08 smithi045 conmon[33655]: 06.219391+0000 mgr.x (mgr.24542) 1279 : cluster [DBG] pgmap v1768: 9 pgs: 9 active+clean; 127 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:17:08.892 INFO:tasks.workunit.client.0.smithi045.stdout: 708: throughput=4.99MB/sec pending data=0 2021-07-31T02:17:08.893 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-d6lNhKb6pwBZW9z off=12330792 len=1229559 2021-07-31T02:17:08.899 INFO:tasks.workunit.client.0.smithi045.stdout:op 3636 completed, throughput=4.99MB/sec 2021-07-31T02:17:08.900 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-knOGqV9xHpDbxD8 off=13315908 len=1472883 2021-07-31T02:17:08.902 INFO:tasks.workunit.client.0.smithi045.stdout:op 3637 completed, throughput=5MB/sec 2021-07-31T02:17:08.902 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-sp034GNgSzJj52o off=25667 len=706523 2021-07-31T02:17:08.907 INFO:tasks.workunit.client.0.smithi045.stdout:op 3638 completed, throughput=5MB/sec 2021-07-31T02:17:08.907 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-qH-NRRI06krbDQO off=9894234 len=143012 2021-07-31T02:17:08.910 INFO:tasks.workunit.client.0.smithi045.stdout:op 3639 completed, throughput=5MB/sec 2021-07-31T02:17:08.910 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-KBpcJ0pEImzAGQw off=5451479 len=1412087 2021-07-31T02:17:08.912 INFO:tasks.workunit.client.0.smithi045.stdout:op 3640 completed, throughput=5MB/sec 2021-07-31T02:17:08.912 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ihQmyZtzaIFUi3C off=2231731 len=369770 2021-07-31T02:17:08.915 INFO:tasks.workunit.client.0.smithi045.stdout:op 3641 completed, throughput=5MB/sec 2021-07-31T02:17:09.915 INFO:tasks.workunit.client.0.smithi045.stdout: 709: throughput=4.99MB/sec pending data=0 2021-07-31T02:17:09.916 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-he3ABI7t-Rw5c7J off=6189795 len=1065297 2021-07-31T02:17:09.917 INFO:tasks.workunit.client.0.smithi045.stdout:op 3642 completed, throughput=4.99MB/sec 2021-07-31T02:17:09.917 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-47Ymr1QHF5wSB2F off=1405447 len=659405 2021-07-31T02:17:09.922 INFO:tasks.workunit.client.0.smithi045.stdout:op 3643 completed, throughput=5MB/sec 2021-07-31T02:17:09.922 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-UPN_TbId3GzYsmU off=9169801 len=225064 2021-07-31T02:17:09.923 INFO:tasks.workunit.client.0.smithi045.stdout:op 3644 completed, throughput=5MB/sec 2021-07-31T02:17:09.923 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-cXTgCZb36edi_g5 off=5124779 len=632708 2021-07-31T02:17:09.924 INFO:tasks.workunit.client.0.smithi045.stdout:op 3645 completed, throughput=5MB/sec 2021-07-31T02:17:09.924 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Wz2oWlfQG3trPLZ off=6509528 len=1557713 2021-07-31T02:17:09.932 INFO:tasks.workunit.client.0.smithi045.stdout:op 3646 completed, throughput=5MB/sec 2021-07-31T02:17:09.932 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Dbq61AA_FZVC2qp off=16224594 len=134157 2021-07-31T02:17:09.933 INFO:tasks.workunit.client.0.smithi045.stdout:op 3647 completed, throughput=5MB/sec 2021-07-31T02:17:09.933 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-jUV6RGGo55yOYxc off=2948370 len=1529879 2021-07-31T02:17:09.935 INFO:tasks.workunit.client.0.smithi045.stdout:op 3648 completed, throughput=5MB/sec 2021-07-31T02:17:10.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:10 smithi154 conmon[34553]: cluster 2021-07-31T02:17:08.219768+0000 mgr.x 2021-07-31T02:17:10.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:10 smithi154 conmon[34553]: (mgr.24542) 1280 : cluster [DBG] pgmap v1769: 9 pgs: 9 active+clean; 127 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:17:10.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:10 smithi045 conmon[33655]: cluster 2021-07-31T02:17:08.219768+0000 mgr.x (mgr.24542) 1280 : cluster [DBG] pgmap v1769: 9 pgs: 9 active+clean; 127 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:17:10.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:10 smithi045 conmon[39718]: cluster 2021-07-31T02:17:08.219768+0000 mgr.x (mgr.24542) 1280 : cluster [DBG] pgmap v1769: 9 pgs: 9 active+clean; 127 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:17:10.935 INFO:tasks.workunit.client.0.smithi045.stdout: 710: throughput=4.99MB/sec pending data=0 2021-07-31T02:17:10.936 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-JH8dbAh5jOPYAOf off=6937813 len=370969 2021-07-31T02:17:10.937 INFO:tasks.workunit.client.0.smithi045.stdout:op 3649 completed, throughput=4.99MB/sec 2021-07-31T02:17:10.937 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-S53fo-rGCG0Rbot off=24798657 len=307477 2021-07-31T02:17:10.938 INFO:tasks.workunit.client.0.smithi045.stdout:op 3650 completed, throughput=4.99MB/sec 2021-07-31T02:17:10.938 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-GdmGeOR8Ap33Vys off=2564787 len=269564 2021-07-31T02:17:10.939 INFO:tasks.workunit.client.0.smithi045.stdout:op 3651 completed, throughput=5MB/sec 2021-07-31T02:17:10.939 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-1QrWZd9IaDwjdMk off=6320335 len=1741017 2021-07-31T02:17:10.949 INFO:tasks.workunit.client.0.smithi045.stdout:op 3652 completed, throughput=5MB/sec 2021-07-31T02:17:10.949 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-MbOzC8BO8klGl_a off=1016577 len=24388 2021-07-31T02:17:10.949 INFO:tasks.workunit.client.0.smithi045.stdout:op 3653 completed, throughput=5MB/sec 2021-07-31T02:17:10.950 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-5SAynbWDSECecZl off=3142827 len=977649 2021-07-31T02:17:10.951 INFO:tasks.workunit.client.0.smithi045.stdout:op 3654 completed, throughput=5MB/sec 2021-07-31T02:17:10.951 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-B3n-H82kTP3iEvi off=381229 len=1733435 2021-07-31T02:17:10.954 INFO:tasks.workunit.client.0.smithi045.stdout:op 3655 completed, throughput=5MB/sec 2021-07-31T02:17:11.955 INFO:tasks.workunit.client.0.smithi045.stdout: 711: throughput=4.99MB/sec pending data=0 2021-07-31T02:17:11.955 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-oo9Y_x-4d2ns5EM off=11876269 len=383831 2021-07-31T02:17:11.959 INFO:tasks.workunit.client.0.smithi045.stdout:op 3656 completed, throughput=4.99MB/sec 2021-07-31T02:17:11.959 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Jgd-ABzpOU5tyRB off=1554228 len=1002631 2021-07-31T02:17:11.965 INFO:tasks.workunit.client.0.smithi045.stdout:op 3657 completed, throughput=5MB/sec 2021-07-31T02:17:11.965 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-vaow47CcEZUXfkh off=11863849 len=1461246 2021-07-31T02:17:11.973 INFO:tasks.workunit.client.0.smithi045.stdout:op 3658 completed, throughput=5MB/sec 2021-07-31T02:17:11.973 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-i17QSSuzLJPzcfF off=12718768 len=1125115 2021-07-31T02:17:11.979 INFO:tasks.workunit.client.0.smithi045.stdout:op 3659 completed, throughput=5MB/sec 2021-07-31T02:17:11.979 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-MnhPDokEhig9uad off=11202546 len=1703439 2021-07-31T02:17:11.987 INFO:tasks.workunit.client.0.smithi045.stdout:op 3660 completed, throughput=5MB/sec 2021-07-31T02:17:12.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:12 smithi154 conmon[34553]: cluster 2021-07-31T02:17:10.220318+0000 mgr.x (mgr.24542) 1281 : cluster [DBG] pgmap v1770: 9 pgs: 9 active+clean; 127 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 3.2 MiB/s rd, 3.1 MiB/s wr, 6 op/s 2021-07-31T02:17:12.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:12 smithi045 conmon[39718]: cluster 2021-07-31T02:17:10.220318+0000 mgr.x (mgr 2021-07-31T02:17:12.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:12 smithi045 conmon[39718]: .24542) 1281 : cluster [DBG] pgmap v1770: 9 pgs: 9 active+clean; 127 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 3.2 MiB/s rd, 3.1 MiB/s wr, 6 op/s 2021-07-31T02:17:12.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:12 smithi045 conmon[33655]: cluster 2021-07-31T02:17:10.220318+0000 mgr.x (mgr.24542) 1281 : cluster [DBG] pgmap v1770: 9 pgs: 9 active+clean; 127 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 3.2 MiB/s rd, 3.1 MiB/s wr, 6 op/s 2021-07-31T02:17:12.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:12 smithi045 conmon[33655]: 2021-07-31T02:17:12.987 INFO:tasks.workunit.client.0.smithi045.stdout: 712: throughput=4.99MB/sec pending data=0 2021-07-31T02:17:12.988 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-vKhuK6ZZhSX38JL off=1602961 len=681509 2021-07-31T02:17:12.992 INFO:tasks.workunit.client.0.smithi045.stdout:op 3661 completed, throughput=5MB/sec 2021-07-31T02:17:12.992 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-BQHiJgWT6u9Qf1O off=18023297 len=1921337 2021-07-31T02:17:12.996 INFO:tasks.workunit.client.0.smithi045.stdout:op 3662 completed, throughput=5MB/sec 2021-07-31T02:17:12.996 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-CGrOe9vYRnczhdB off=621782 len=152031 2021-07-31T02:17:12.997 INFO:tasks.workunit.client.0.smithi045.stdout:op 3663 completed, throughput=5MB/sec 2021-07-31T02:17:12.997 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ByTK2jNNLdAYS6Y off=531556 len=855262 2021-07-31T02:17:13.001 INFO:tasks.workunit.client.0.smithi045.stdout:op 3664 completed, throughput=5MB/sec 2021-07-31T02:17:13.002 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Z9eXhZUFnZJi-xC off=4664019 len=1923673 2021-07-31T02:17:13.011 INFO:tasks.workunit.client.0.smithi045.stdout:op 3665 completed, throughput=5MB/sec 2021-07-31T02:17:14.011 INFO:tasks.workunit.client.0.smithi045.stdout: 713: throughput=4.99MB/sec pending data=0 2021-07-31T02:17:14.012 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-YDHGjNZaLfonsTt off=893038 len=1232600 2021-07-31T02:17:14.014 INFO:tasks.workunit.client.0.smithi045.stdout:op 3666 completed, throughput=5MB/sec 2021-07-31T02:17:14.014 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-UxwKJq5W0Gedk9f off=1545708 len=636356 2021-07-31T02:17:14.019 INFO:tasks.workunit.client.0.smithi045.stdout:op 3667 completed, throughput=5MB/sec 2021-07-31T02:17:14.019 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-OAF4jbxUncGio0t off=0 len=1712553 2021-07-31T02:17:14.025 INFO:tasks.workunit.client.0.smithi045.stdout:op 3668 completed, throughput=5MB/sec 2021-07-31T02:17:14.025 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-bOu5p-KhmfeyNAb off=1167251 len=619917 2021-07-31T02:17:14.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:17:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:17:13.560Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:17:14.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:17:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:17:13.560Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:17:14.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:17:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:17:13.560Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:17:14.027 INFO:tasks.workunit.client.0.smithi045.stdout:op 3669 completed, throughput=5MB/sec 2021-07-31T02:17:14.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:14 smithi154 conmon[34553]: cluster 2021-07-31T02:17:12.220749+0000 mgr.x (mgr.24542) 1282 : cluster [DBG] pgmap v1771: 9 pgs: 9 active+clean; 127 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 2.3 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2021-07-31T02:17:14.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:14 smithi045 conmon[33655]: cluster 2021-07-31T02:17:12.220749+0000 mgr.x (mgr.24542) 1282 : cluster 2021-07-31T02:17:14.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:14 smithi045 conmon[33655]: [DBG] pgmap v1771: 9 pgs: 9 active+clean; 127 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 2.3 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2021-07-31T02:17:14.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:14 smithi045 conmon[39718]: cluster 2021-07-31T02:17:12.220749+0000 mgr.x 2021-07-31T02:17:14.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:14 smithi045 conmon[39718]: (mgr.24542) 1282 : cluster [DBG] pgmap v1771: 9 pgs: 9 active+clean; 127 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 2.3 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2021-07-31T02:17:15.027 INFO:tasks.workunit.client.0.smithi045.stdout: 714: throughput=4.99MB/sec pending data=0 2021-07-31T02:17:15.027 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-b44-1OF7ADWUV28 off=9054753 len=1035108 2021-07-31T02:17:15.032 INFO:tasks.workunit.client.0.smithi045.stdout:op 3670 completed, throughput=4.99MB/sec 2021-07-31T02:17:15.032 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-xyAzBEyIpT_Vw6g off=3436270 len=1502500 2021-07-31T02:17:15.039 INFO:tasks.workunit.client.0.smithi045.stdout:op 3671 completed, throughput=5MB/sec 2021-07-31T02:17:15.039 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-eefeIOexFigW3gJ off=23546053 len=1285486 2021-07-31T02:17:15.041 INFO:tasks.workunit.client.0.smithi045.stdout:op 3672 completed, throughput=5MB/sec 2021-07-31T02:17:15.041 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-zBIKrmrKbQjJxwW off=7022760 len=1155307 2021-07-31T02:17:15.043 INFO:tasks.workunit.client.0.smithi045.stdout:op 3673 completed, throughput=5MB/sec 2021-07-31T02:17:15.044 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-sqXyIlybWq0FNRS off=23037730 len=1888667 2021-07-31T02:17:15.045 INFO:tasks.workunit.client.0.smithi045.stdout:op 3674 completed, throughput=5MB/sec 2021-07-31T02:17:16.046 INFO:tasks.workunit.client.0.smithi045.stdout: 715: throughput=5MB/sec pending data=0 2021-07-31T02:17:16.046 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-46p1NP229JO0Fm8 off=4687560 len=1819455 2021-07-31T02:17:16.049 INFO:tasks.workunit.client.0.smithi045.stdout:op 3675 completed, throughput=5MB/sec 2021-07-31T02:17:16.049 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-7bRkEr2wFmb0p9I off=591388 len=313603 2021-07-31T02:17:16.051 INFO:tasks.workunit.client.0.smithi045.stdout:op 3676 completed, throughput=5MB/sec 2021-07-31T02:17:16.051 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-144Jq1UIB80iiJM off=11786247 len=283504 2021-07-31T02:17:16.054 INFO:tasks.workunit.client.0.smithi045.stdout:op 3677 completed, throughput=5MB/sec 2021-07-31T02:17:16.055 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-w9OYTIlVMYUaKan off=872652 len=826331 2021-07-31T02:17:16.060 INFO:tasks.workunit.client.0.smithi045.stdout:op 3678 completed, throughput=5MB/sec 2021-07-31T02:17:16.060 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-c9gSKAfbibW2oWI off=117685 len=1325674 2021-07-31T02:17:16.066 INFO:tasks.workunit.client.0.smithi045.stdout:op 3679 completed, throughput=5MB/sec 2021-07-31T02:17:16.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:16 smithi154 conmon[34553]: cluster 2021-07-31T02:17:14.221294+0000 mgr.x (mgr.24542) 1283 : cluster [DBG] pgmap v1772: 9 pgs: 9 active+clean; 127 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.9 MiB/s wr, 6 op/s 2021-07-31T02:17:16.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:16 smithi045 conmon[33655]: cluster 2021-07-31T02:17:14.221294+0000 mgr.x (mgr.24542) 1283 : cluster 2021-07-31T02:17:16.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:16 smithi045 conmon[33655]: [DBG] pgmap v1772: 9 pgs: 9 active+clean; 127 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.9 MiB/s wr, 6 op/s 2021-07-31T02:17:16.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:16 smithi045 conmon[39718]: cluster 2021-07-31T02:17:14.221294+0000 mgr.x ( 2021-07-31T02:17:16.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:16 smithi045 conmon[39718]: mgr.24542) 1283 : cluster [DBG] pgmap v1772: 9 pgs: 9 active+clean; 127 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.9 MiB/s wr, 6 op/s 2021-07-31T02:17:17.067 INFO:tasks.workunit.client.0.smithi045.stdout: 716: throughput=4.99MB/sec pending data=0 2021-07-31T02:17:17.067 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-jH-xcLlSuoqPhVT off=6777102 len=1717060 2021-07-31T02:17:17.070 INFO:tasks.workunit.client.0.smithi045.stdout:op 3680 completed, throughput=5MB/sec 2021-07-31T02:17:17.071 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-TJ6KcEffTwkdUXJ off=19835000 len=1616763 2021-07-31T02:17:17.078 INFO:tasks.workunit.client.0.smithi045.stdout:op 3681 completed, throughput=5MB/sec 2021-07-31T02:17:17.078 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-QCgWXOkGJM7-E3s off=8416237 len=1899273 2021-07-31T02:17:17.082 INFO:tasks.workunit.client.0.smithi045.stdout:op 3682 completed, throughput=5MB/sec 2021-07-31T02:17:18.082 INFO:tasks.workunit.client.0.smithi045.stdout: 717: throughput=4.99MB/sec pending data=0 2021-07-31T02:17:18.083 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-OuQplqeb2PUsf8Y off=14874924 len=2034114 2021-07-31T02:17:18.095 INFO:tasks.workunit.client.0.smithi045.stdout:op 3683 completed, throughput=5MB/sec 2021-07-31T02:17:18.095 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-4LMkU1gBetUBqrE off=19296169 len=1456792 2021-07-31T02:17:18.103 INFO:tasks.workunit.client.0.smithi045.stdout:op 3684 completed, throughput=5MB/sec 2021-07-31T02:17:18.103 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-GYXtY0JPJ61Jfk9 off=6899139 len=363964 2021-07-31T02:17:18.107 INFO:tasks.workunit.client.0.smithi045.stdout:op 3685 completed, throughput=5MB/sec 2021-07-31T02:17:18.107 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-57QagppET9QYLAH off=2996880 len=380366 2021-07-31T02:17:18.111 INFO:tasks.workunit.client.0.smithi045.stdout:op 3686 completed, throughput=5MB/sec 2021-07-31T02:17:18.111 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-u7blUDMYnPuV7xT off=11663493 len=1586155 2021-07-31T02:17:18.112 INFO:tasks.workunit.client.0.smithi045.stdout:op 3687 completed, throughput=5MB/sec 2021-07-31T02:17:18.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:18 smithi154 conmon[34553]: cluster 2021-07-31T02:17:16.221669+0000 mgr.x (mgr.24542) 1284 : cluster [DBG] pgmap v1773: 9 pgs: 9 active+clean; 127 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 2.1 MiB/s rd, 1.8 MiB/s wr, 4 op/s 2021-07-31T02:17:18.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:18 smithi045 conmon[39718]: cluster 2021-07-31T02:17:16.221669+0000 mgr.x (mgr. 2021-07-31T02:17:18.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:18 smithi045 conmon[39718]: 24542) 1284 : cluster [DBG] pgmap v1773: 9 pgs: 9 active+clean; 127 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 2.1 MiB/s rd, 1.8 MiB/s wr, 4 op/s 2021-07-31T02:17:18.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:18 smithi045 conmon[33655]: cluster 2021-07-31T02:17:16.221669 2021-07-31T02:17:18.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:18 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 1284 : cluster [DBG] pgmap v1773: 9 pgs: 9 active+clean; 127 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 2.1 MiB/s rd, 1.8 MiB/s wr, 4 op/s 2021-07-31T02:17:19.113 INFO:tasks.workunit.client.0.smithi045.stdout: 718: throughput=4.99MB/sec pending data=0 2021-07-31T02:17:19.113 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-opBW1wUUYWByhDn off=1562855 len=719424 2021-07-31T02:17:19.115 INFO:tasks.workunit.client.0.smithi045.stdout:op 3688 completed, throughput=5MB/sec 2021-07-31T02:17:19.115 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-LWPwSc20ktIonot off=8562411 len=1895976 2021-07-31T02:17:19.124 INFO:tasks.workunit.client.0.smithi045.stdout:op 3689 completed, throughput=5MB/sec 2021-07-31T02:17:19.125 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-5AsTYDSEnUB2OkG off=1734673 len=181512 2021-07-31T02:17:19.125 INFO:tasks.workunit.client.0.smithi045.stdout:op 3690 completed, throughput=5MB/sec 2021-07-31T02:17:19.126 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-vbrC7OSF56KiFGA off=11542563 len=1985456 2021-07-31T02:17:19.129 INFO:tasks.workunit.client.0.smithi045.stdout:op 3691 completed, throughput=5MB/sec 2021-07-31T02:17:20.129 INFO:tasks.workunit.client.0.smithi045.stdout: 719: throughput=4.99MB/sec pending data=0 2021-07-31T02:17:20.129 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-OaGZGfFL8PMhfqh off=1522741 len=359631 2021-07-31T02:17:20.130 INFO:tasks.workunit.client.0.smithi045.stdout:op 3692 completed, throughput=4.99MB/sec 2021-07-31T02:17:20.131 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-b2t3f7yNi-BhIyj off=2147700 len=1603587 2021-07-31T02:17:20.140 INFO:tasks.workunit.client.0.smithi045.stdout:op 3693 completed, throughput=5MB/sec 2021-07-31T02:17:20.140 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-yGNsMDw8-H4KFpo off=3236131 len=715981 2021-07-31T02:17:20.141 INFO:tasks.workunit.client.0.smithi045.stdout:op 3694 completed, throughput=5MB/sec 2021-07-31T02:17:20.141 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-dMmC-ILcyhKO-0q off=3911579 len=177141 2021-07-31T02:17:20.142 INFO:tasks.workunit.client.0.smithi045.stdout:op 3695 completed, throughput=5MB/sec 2021-07-31T02:17:20.142 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-4BeF221YJhLN6wd off=13758746 len=35809 2021-07-31T02:17:20.142 INFO:tasks.workunit.client.0.smithi045.stdout:op 3696 completed, throughput=5MB/sec 2021-07-31T02:17:20.143 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-0JqYV2AAOmTbtMO off=16517178 len=85582 2021-07-31T02:17:20.143 INFO:tasks.workunit.client.0.smithi045.stdout:op 3697 completed, throughput=5MB/sec 2021-07-31T02:17:20.144 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Ug4IlHTRCkzsjOW off=0 len=1233461 2021-07-31T02:17:20.147 INFO:tasks.workunit.client.0.smithi045.stdout:op 3698 completed, throughput=5MB/sec 2021-07-31T02:17:20.148 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-0fyu4u8LcIoEFjX off=575696 len=2052012 2021-07-31T02:17:20.158 INFO:tasks.workunit.client.0.smithi045.stdout:op 3699 completed, throughput=5MB/sec 2021-07-31T02:17:20.321 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:17:20.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:20 smithi154 conmon[34553]: cluster 2021-07-31T02:17:18.222053+0000 mgr.x (mgr.24542) 1285 : cluster [DBG] pgmap v1774: 9 pgs: 9 active+clean; 127 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 2.1 MiB/s rd, 1.8 MiB/s wr, 4 op/s 2021-07-31T02:17:20.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:20 smithi045 conmon[33655]: cluster 2021-07-31T02:17:18.222053+0000 mgr.x (mgr.24542) 1285 : cluster [DBG] pgmap v1774: 9 pgs: 9 active+clean; 127 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 2.1 MiB/s rd, 1.8 MiB/s wr, 4 op/s 2021-07-31T02:17:20.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:20 smithi045 conmon[39718]: cluster 2021-07-31T02:17: 2021-07-31T02:17:20.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:20 smithi045 conmon[39718]: 18.222053+0000 mgr.x (mgr.24542) 1285 : cluster [DBG] pgmap v1774: 9 pgs: 9 active+clean; 127 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 2.1 MiB/s rd, 1.8 MiB/s wr, 4 op/s 2021-07-31T02:17:20.599 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:17:20.599 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 32m ago 32m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:17:20.599 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 29m ago 32m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:17:20.600 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 29m ago 36m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:17:20.600 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 32m ago 38m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:17:20.600 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 32m ago 38m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:17:20.600 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 29m ago 36m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:17:20.601 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 32m ago 36m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:17:20.601 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 32m ago 33m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:17:20.601 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 29m ago 33m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:17:20.601 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 32m ago 35m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:17:20.602 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 32m ago 35m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:17:20.602 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 32m ago 35m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:17:20.602 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 32m ago 34m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:17:20.602 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 29m ago 34m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:17:20.602 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 29m ago 34m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:17:20.603 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 29m ago 33m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:17:20.603 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 29m ago 33m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:17:20.603 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 29m ago 33m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:17:20.934 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:17:20.935 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:17:20.935 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:17:20.935 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:17:20.935 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:17:20.936 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:17:20.936 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:17:20.936 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:17:20.936 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:17:20.937 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:17:20.937 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:17:20.937 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:17:20.937 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:17:20.938 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:17:20.938 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:17:20.938 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:17:20.938 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:17:21.158 INFO:tasks.workunit.client.0.smithi045.stdout: 720: throughput=5MB/sec pending data=0 2021-07-31T02:17:21.158 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-zcmznw_0Bx1fB9d off=3641339 len=132213 2021-07-31T02:17:21.161 INFO:tasks.workunit.client.0.smithi045.stdout:op 3700 completed, throughput=5MB/sec 2021-07-31T02:17:21.161 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-kV-js4JYYNYH0AT off=2190346 len=1909416 2021-07-31T02:17:21.164 INFO:tasks.workunit.client.0.smithi045.stdout:op 3701 completed, throughput=5MB/sec 2021-07-31T02:17:21.164 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-3HvjtlK-ms_CG5E off=3097721 len=666894 2021-07-31T02:17:21.168 INFO:tasks.workunit.client.0.smithi045.stdout:op 3702 completed, throughput=5MB/sec 2021-07-31T02:17:21.168 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ufce3sRszvE4DBR off=7070945 len=1525906 2021-07-31T02:17:21.174 INFO:tasks.workunit.client.0.smithi045.stdout:op 3703 completed, throughput=5MB/sec 2021-07-31T02:17:21.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:21 smithi154 conmon[34553]: audit 2021-07-31T02:17:20.934927+0000 mon.c (mon.1) 86 : audit [DBG] from='client.? 172.21.15.45:0/4071806181' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:17:21.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:21 smithi154 conmon[34553]: 2021-07-31T02:17:21.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:21 smithi045 conmon[33655]: audit 2021-07-31T02:17:20.934927+0000 mon.c (mon. 2021-07-31T02:17:21.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:21 smithi045 conmon[33655]: 1) 86 : audit [DBG] from='client.? 172.21.15.45:0/4071806181' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:17:21.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:21 smithi045 conmon[39718]: audit 2021-07-31T02:17: 2021-07-31T02:17:21.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:21 smithi045 conmon[39718]: 20.934927+0000 mon.c (mon.1) 86 : audit [DBG] from='client.? 172.21.15.45:0/4071806181' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:17:22.175 INFO:tasks.workunit.client.0.smithi045.stdout: 721: throughput=4.99MB/sec pending data=0 2021-07-31T02:17:22.175 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-9PqSwV-DwkC_REo off=6768433 len=1265188 2021-07-31T02:17:22.182 INFO:tasks.workunit.client.0.smithi045.stdout:op 3704 completed, throughput=5MB/sec 2021-07-31T02:17:22.182 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-XhEXR9sIqgJiZTZ off=14625109 len=666686 2021-07-31T02:17:22.183 INFO:tasks.workunit.client.0.smithi045.stdout:op 3705 completed, throughput=5MB/sec 2021-07-31T02:17:22.183 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-zf5yos0C0Bgv7_d off=1858668 len=1756182 2021-07-31T02:17:22.192 INFO:tasks.workunit.client.0.smithi045.stdout:op 3706 completed, throughput=5MB/sec 2021-07-31T02:17:22.193 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-88-82woaAPXscyo off=21672614 len=155745 2021-07-31T02:17:22.194 INFO:tasks.workunit.client.0.smithi045.stdout:op 3707 completed, throughput=5MB/sec 2021-07-31T02:17:22.194 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-kMPFcEdXVN3Z0-I off=13361 len=150883 2021-07-31T02:17:22.196 INFO:tasks.workunit.client.0.smithi045.stdout:op 3708 completed, throughput=5MB/sec 2021-07-31T02:17:22.196 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-EDwrBTXIwtRttSw off=667825 len=720606 2021-07-31T02:17:22.197 INFO:tasks.workunit.client.0.smithi045.stdout:op 3709 completed, throughput=5MB/sec 2021-07-31T02:17:22.197 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-nBDmfGmk6Mg2SbE off=3060096 len=33866 2021-07-31T02:17:22.199 INFO:tasks.workunit.client.0.smithi045.stdout:op 3710 completed, throughput=5MB/sec 2021-07-31T02:17:22.200 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-8ITLMwdS_tsZcuT off=7483186 len=1357058 2021-07-31T02:17:22.202 INFO:tasks.workunit.client.0.smithi045.stdout:op 3711 completed, throughput=5MB/sec 2021-07-31T02:17:22.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:22 smithi154 conmon[34553]: cluster 2021-07-31T02:17:20.222691+0000 mgr.x (mgr.24542) 1286 : cluster [DBG] pgmap v1775: 9 pgs: 9 active+clean; 127 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 3.3 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2021-07-31T02:17:22.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:22 smithi154 conmon[34553]: audit 2021-07-31T02:17:20.309183+0000 mgr.x 2021-07-31T02:17:22.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:22 smithi154 conmon[34553]: (mgr.24542) 1287 : audit [DBG] from='client.16149 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:17:22.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:22 smithi154 conmon[34553]: audit 2021-07-31T02:17:20.590991+0000 mgr.x (mgr.24542) 1288 : audit [DBG] from='client.26062 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:17:22.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:22 smithi045 conmon[33655]: cluster 2021-07-31T02:17:20.222691+0000 mgr.x (mgr.24542) 1286 : cluster [DBG] 2021-07-31T02:17:22.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:22 smithi045 conmon[33655]: pgmap v1775: 9 pgs: 9 active+clean; 127 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 3.3 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2021-07-31T02:17:22.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:22 smithi045 conmon[33655]: audit 2021-07-31T02:17:20.309183+0000 mgr.x (mgr.24542) 1287 : audit [DBG] from='client.16149 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:17:22.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:22 smithi045 conmon[33655]: audit 2021-07-31T02:17:20.590991+0000 mgr.x (mgr.24542) 1288 : audit [DBG] from='client.26062 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:17:22.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:22 smithi045 conmon[39718]: cluster 2021-07-31T02:17:20.222691+0000 mgr.x (mgr. 2021-07-31T02:17:22.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:22 smithi045 conmon[39718]: 24542) 1286 : cluster [DBG] pgmap v1775: 9 pgs: 9 active+clean; 127 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 3.3 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2021-07-31T02:17:22.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:22 smithi045 conmon[39718]: audit 2021-07-31T02:17:20.309183+0000 mgr.x (mgr.24542) 1287 2021-07-31T02:17:22.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:22 smithi045 conmon[39718]: : audit [DBG] from='client.16149 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:17:22.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:22 smithi045 conmon[39718]: audit 2021 2021-07-31T02:17:22.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:22 smithi045 conmon[39718]: -07-31T02:17:20.590991+0000 mgr.x (mgr. 2021-07-31T02:17:22.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:22 smithi045 conmon[39718]: 24542) 1288 : audit [DBG] from='client.26062 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:17:23.202 INFO:tasks.workunit.client.0.smithi045.stdout: 722: throughput=4.99MB/sec pending data=0 2021-07-31T02:17:23.203 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-y-6XIbgaXueC-5t off=1650162 len=120581 2021-07-31T02:17:23.205 INFO:tasks.workunit.client.0.smithi045.stdout:op 3712 completed, throughput=4.99MB/sec 2021-07-31T02:17:23.205 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-PI7UiWuajF-32Br off=466651 len=1201026 2021-07-31T02:17:23.208 INFO:tasks.workunit.client.0.smithi045.stdout:op 3713 completed, throughput=5MB/sec 2021-07-31T02:17:23.208 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj--OrsnB55oY4_fKD off=1381568 len=9917 2021-07-31T02:17:23.209 INFO:tasks.workunit.client.0.smithi045.stdout:op 3714 completed, throughput=5MB/sec 2021-07-31T02:17:23.209 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-FT23sZ4pfeRwBKX off=20007281 len=721733 2021-07-31T02:17:23.210 INFO:tasks.workunit.client.0.smithi045.stdout:op 3715 completed, throughput=5MB/sec 2021-07-31T02:17:23.211 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-FUuGaP79f1TzrO- off=1488869 len=1892426 2021-07-31T02:17:23.219 INFO:tasks.workunit.client.0.smithi045.stdout:op 3716 completed, throughput=5MB/sec 2021-07-31T02:17:23.219 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-lNwwfWioDWkck0M off=14988999 len=485612 2021-07-31T02:17:23.223 INFO:tasks.workunit.client.0.smithi045.stdout:op 3717 completed, throughput=5MB/sec 2021-07-31T02:17:24.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:17:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:17:23.560Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:17:24.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:17:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:17:23.560Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:17:24.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:17:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:17:23.560Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:17:24.224 INFO:tasks.workunit.client.0.smithi045.stdout: 723: throughput=4.99MB/sec pending data=0 2021-07-31T02:17:24.224 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-fUBuFnd3_Ik1V1y off=2101087 len=1825561 2021-07-31T02:17:24.235 INFO:tasks.workunit.client.0.smithi045.stdout:op 3718 completed, throughput=5MB/sec 2021-07-31T02:17:24.235 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-C-Dt6nilJVjnovI off=19268564 len=2034521 2021-07-31T02:17:24.239 INFO:tasks.workunit.client.0.smithi045.stdout:op 3719 completed, throughput=5MB/sec 2021-07-31T02:17:24.239 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-QlqChufxcu6MeGN off=3072627 len=1678980 2021-07-31T02:17:24.246 INFO:tasks.workunit.client.0.smithi045.stdout:op 3720 completed, throughput=5MB/sec 2021-07-31T02:17:24.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:24 smithi154 conmon[34553]: cluster 2021-07-31T02:17:22.223304+0000 mgr.x (mgr.24542) 1289 : cluster [DBG] pgmap v1776: 9 pgs: 9 active+clean; 127 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.0 MiB/s wr, 3 op/s 2021-07-31T02:17:24.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:24 smithi045 conmon[39718]: cluster 2021-07-31T02:17:22.223304+0000 mgr.x 2021-07-31T02:17:24.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:24 smithi045 conmon[39718]: (mgr.24542) 1289 : cluster [DBG] pgmap v1776: 9 pgs: 9 active+clean; 127 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.0 MiB/s wr, 3 op/s 2021-07-31T02:17:24.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:24 smithi045 conmon[33655]: cluster 2021-07-31T02:17:22.223304+0000 mgr.x (mgr.24542) 2021-07-31T02:17:24.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:24 smithi045 conmon[33655]: 1289 : cluster [DBG] pgmap v1776: 9 pgs: 9 active+clean; 127 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.0 MiB/s wr, 3 op/s 2021-07-31T02:17:25.247 INFO:tasks.workunit.client.0.smithi045.stdout: 724: throughput=4.99MB/sec pending data=0 2021-07-31T02:17:25.247 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-CFE5DbufgEtXZFg off=12471096 len=709887 2021-07-31T02:17:25.252 INFO:tasks.workunit.client.0.smithi045.stdout:op 3721 completed, throughput=4.99MB/sec 2021-07-31T02:17:25.252 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-LymoCatHC_3B6lH off=10687992 len=2041116 2021-07-31T02:17:25.262 INFO:tasks.workunit.client.0.smithi045.stdout:op 3722 completed, throughput=5MB/sec 2021-07-31T02:17:25.263 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-88vme0xS4bmg1PE off=1760868 len=1336326 2021-07-31T02:17:25.268 INFO:tasks.workunit.client.0.smithi045.stdout:op 3723 completed, throughput=5MB/sec 2021-07-31T02:17:25.268 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-inNpVjkQ4ncWmCh off=231836 len=704912 2021-07-31T02:17:25.269 INFO:tasks.workunit.client.0.smithi045.stdout:op 3724 completed, throughput=5MB/sec 2021-07-31T02:17:25.270 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-buqPkCHeKNH3M6c off=327957 len=877618 2021-07-31T02:17:25.270 INFO:tasks.workunit.client.0.smithi045.stdout:op 3725 completed, throughput=5MB/sec 2021-07-31T02:17:26.271 INFO:tasks.workunit.client.0.smithi045.stdout: 725: throughput=4.99MB/sec pending data=0 2021-07-31T02:17:26.272 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-TuRyDdI_6g3agTm off=2500773 len=495503 2021-07-31T02:17:26.273 INFO:tasks.workunit.client.0.smithi045.stdout:op 3726 completed, throughput=4.99MB/sec 2021-07-31T02:17:26.273 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ee5pg5kDK7331Xa off=4898537 len=574550 2021-07-31T02:17:26.278 INFO:tasks.workunit.client.0.smithi045.stdout:op 3727 completed, throughput=5MB/sec 2021-07-31T02:17:26.278 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-gGcaOK11sBg-ZZa off=7178041 len=323142 2021-07-31T02:17:26.281 INFO:tasks.workunit.client.0.smithi045.stdout:op 3728 completed, throughput=5MB/sec 2021-07-31T02:17:26.281 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-hNgwzoYWVVrSIbw off=1002498 len=357415 2021-07-31T02:17:26.284 INFO:tasks.workunit.client.0.smithi045.stdout:op 3729 completed, throughput=5MB/sec 2021-07-31T02:17:26.285 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-P5K93bBzZffOhEf off=4521752 len=245885 2021-07-31T02:17:26.287 INFO:tasks.workunit.client.0.smithi045.stdout:op 3730 completed, throughput=5MB/sec 2021-07-31T02:17:26.288 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-fMFBwp5B6lZR0X9 off=638967 len=606453 2021-07-31T02:17:26.289 INFO:tasks.workunit.client.0.smithi045.stdout:op 3731 completed, throughput=5MB/sec 2021-07-31T02:17:26.289 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-JY4WsIpyfhSi5Ip off=14235661 len=502754 2021-07-31T02:17:26.290 INFO:tasks.workunit.client.0.smithi045.stdout:op 3732 completed, throughput=5MB/sec 2021-07-31T02:17:26.290 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-_0X2vkb6uB0k0tH off=16086639 len=1749648 2021-07-31T02:17:26.298 INFO:tasks.workunit.client.0.smithi045.stdout:op 3733 completed, throughput=5MB/sec 2021-07-31T02:17:26.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:26 smithi154 conmon[34553]: cluster 2021-07-31T02:17:24.223729+0000 mgr.x (mgr.24542) 1290 : cluster [DBG] pgmap v1777: 9 pgs: 9 active+clean; 127 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 2.9 MiB/s rd, 3.1 MiB/s wr, 6 op/s 2021-07-31T02:17:26.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:26 smithi045 conmon[33655]: cluster 2021-07-31T02:17:24.223729+0000 mgr.x (mgr.24542) 1290 : 2021-07-31T02:17:26.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:26 smithi045 conmon[33655]: cluster [DBG] pgmap v1777: 9 pgs: 9 active+clean; 127 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 2.9 MiB/s rd, 3.1 MiB/s wr, 6 op/s 2021-07-31T02:17:26.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:26 smithi045 conmon[39718]: cluster 2021-07-31T02:17:24.223729+0000 mgr.x ( 2021-07-31T02:17:26.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:26 smithi045 conmon[39718]: mgr.24542) 1290 : cluster [DBG] pgmap v1777: 9 pgs: 9 active+clean; 127 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 2.9 MiB/s rd, 3.1 MiB/s wr, 6 op/s 2021-07-31T02:17:27.299 INFO:tasks.workunit.client.0.smithi045.stdout: 726: throughput=4.99MB/sec pending data=0 2021-07-31T02:17:27.299 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-AjWLwS5_T1dar4F off=2215888 len=515722 2021-07-31T02:17:27.300 INFO:tasks.workunit.client.0.smithi045.stdout:op 3734 completed, throughput=4.99MB/sec 2021-07-31T02:17:27.301 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-R_2tukePLtHJW48 off=3286838 len=1211530 2021-07-31T02:17:27.302 INFO:tasks.workunit.client.0.smithi045.stdout:op 3735 completed, throughput=5MB/sec 2021-07-31T02:17:27.302 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-xSubuR0KLJNlQTY off=4394857 len=1709512 2021-07-31T02:17:27.313 INFO:tasks.workunit.client.0.smithi045.stdout:op 3736 completed, throughput=5MB/sec 2021-07-31T02:17:27.313 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-qBEk8H5DFdW43h9 off=130709 len=771787 2021-07-31T02:17:27.318 INFO:tasks.workunit.client.0.smithi045.stdout:op 3737 completed, throughput=5MB/sec 2021-07-31T02:17:27.318 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-DitGDkJg-AA2e_c off=7914532 len=1613955 2021-07-31T02:17:27.321 INFO:tasks.workunit.client.0.smithi045.stdout:op 3738 completed, throughput=5MB/sec 2021-07-31T02:17:28.322 INFO:tasks.workunit.client.0.smithi045.stdout: 727: throughput=4.99MB/sec pending data=0 2021-07-31T02:17:28.322 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-3FMGLLf0fRBzPgm off=14751113 len=1883273 2021-07-31T02:17:28.325 INFO:tasks.workunit.client.0.smithi045.stdout:op 3739 completed, throughput=5MB/sec 2021-07-31T02:17:28.325 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-yaccHlHpk2JJNp3 off=1369739 len=1243602 2021-07-31T02:17:28.332 INFO:tasks.workunit.client.0.smithi045.stdout:op 3740 completed, throughput=5MB/sec 2021-07-31T02:17:28.332 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-f-v8QC6-47991rE off=10400737 len=169543 2021-07-31T02:17:28.333 INFO:tasks.workunit.client.0.smithi045.stdout:op 3741 completed, throughput=5MB/sec 2021-07-31T02:17:28.333 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-kOT-ROHZOH0E0Ud off=16385225 len=1095945 2021-07-31T02:17:28.339 INFO:tasks.workunit.client.0.smithi045.stdout:op 3742 completed, throughput=5MB/sec 2021-07-31T02:17:28.339 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-zVpq8kKUtbVL5zb off=1470584 len=872260 2021-07-31T02:17:28.340 INFO:tasks.workunit.client.0.smithi045.stdout:op 3743 completed, throughput=5MB/sec 2021-07-31T02:17:28.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:28 smithi154 conmon[34553]: cluster 2021-07-31T02:17:26.224153+0000 mgr.x (mgr.24542) 1291 : cluster [DBG] pgmap v1778: 9 pgs: 9 active+clean; 127 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2021-07-31T02:17:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:28 smithi154 conmon[34553]: audit 2021-07-31T02:17:28.038236+0000 mon.b (mon.2) 218 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:17:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:28 smithi154 conmon[34553]: audit 2021-07-31T02:17:28.038903+0000 mon.a (mon.0) 1887 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:17:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:28 smithi154 conmon[34553]: audit 2021-07-31T02:17:28.060079+0000 mon.b (mon.2 2021-07-31T02:17:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:28 smithi154 conmon[34553]: ) 219 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:17:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:28 smithi154 conmon[34553]: audit 2021-07- 2021-07-31T02:17:28.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:28 smithi154 conmon[34553]: 31T02:17:28.060640+0000 mon.a (mon. 2021-07-31T02:17:28.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:28 smithi154 conmon[34553]: 0) 1888 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:17:28.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:28 smithi045 conmon[39718]: cluster 2021-07-31T02:17:26.224153+0000 mgr.x (mgr.24542) 2021-07-31T02:17:28.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:28 smithi045 conmon[39718]: 1291 : cluster [DBG] pgmap v1778: 9 pgs: 9 active+clean; 127 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2021-07-31T02:17:28.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:28 smithi045 conmon[39718]: audit 2021-07-31T02:17:28.038236+0000 mon.b (mon.2) 218 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:17:28.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:28 smithi045 conmon[39718]: audit 2021-07-31T02:17:28.038903+0000 mon.a (mon.0) 1887 : audit [INF] 2021-07-31T02:17:28.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:28 smithi045 conmon[39718]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:17:28.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:28 smithi045 conmon[39718]: audit 2021 2021-07-31T02:17:28.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:28 smithi045 conmon[39718]: -07-31T02:17:28.060079+0000 mon.b (mon.2) 219 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:17:28.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:28 smithi045 conmon[39718]: audit 2021-07-31T02 2021-07-31T02:17:28.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:28 smithi045 conmon[39718]: :17:28.060640+0000 mon.a (mon.0) 1888 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:17:28.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:28 smithi045 conmon[33655]: cluster 2021-07-31T02:17:26.224153+0000 mgr.x (mgr.24542) 1291 : 2021-07-31T02:17:28.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:28 smithi045 conmon[33655]: cluster [DBG] pgmap v1778: 9 pgs: 9 active+clean; 127 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2021-07-31T02:17:28.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:28 smithi045 conmon[33655]: audit 2021-07-31T02:17:28.038236+0000 mon.b (mon.2) 218 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:17:28.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:28 smithi045 conmon[33655]: audit 2021-07- 2021-07-31T02:17:28.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:28 smithi045 conmon[33655]: 31T02:17:28.038903+0000 mon.a (mon.0) 1887 : 2021-07-31T02:17:28.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:28 smithi045 conmon[33655]: audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:17:28.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:28 smithi045 conmon[33655]: audit 2021-07-31T02:17:28.060079+0000 mon.b (mon 2021-07-31T02:17:28.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:28 smithi045 conmon[33655]: .2) 219 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:17:28.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:28 smithi045 conmon[33655]: audit 2021-07-31T02:17:28.060640+0000 2021-07-31T02:17:28.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:28 smithi045 conmon[33655]: mon.a (mon.0) 1888 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:17:29.340 INFO:tasks.workunit.client.0.smithi045.stdout: 728: throughput=4.99MB/sec pending data=0 2021-07-31T02:17:29.341 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-1Pug8gF5gdT_zSD off=2423364 len=255033 2021-07-31T02:17:29.344 INFO:tasks.workunit.client.0.smithi045.stdout:op 3744 completed, throughput=4.99MB/sec 2021-07-31T02:17:29.344 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj--yPNkJ0ChjdVIWs off=12792246 len=904544 2021-07-31T02:17:29.349 INFO:tasks.workunit.client.0.smithi045.stdout:op 3745 completed, throughput=5MB/sec 2021-07-31T02:17:29.349 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-PtCnlfY8MYBbo_n off=4605150 len=2004769 2021-07-31T02:17:29.351 INFO:tasks.workunit.client.0.smithi045.stdout:op 3746 completed, throughput=5MB/sec 2021-07-31T02:17:29.351 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-vRZklViEkGQk0w8 off=16249767 len=1092917 2021-07-31T02:17:29.354 INFO:tasks.workunit.client.0.smithi045.stdout:op 3747 completed, throughput=5MB/sec 2021-07-31T02:17:29.355 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-1Sn3uo7DvozeE5l off=2677336 len=1493822 2021-07-31T02:17:29.357 INFO:tasks.workunit.client.0.smithi045.stdout:op 3748 completed, throughput=5MB/sec 2021-07-31T02:17:30.357 INFO:tasks.workunit.client.0.smithi045.stdout: 729: throughput=4.99MB/sec pending data=0 2021-07-31T02:17:30.357 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-wKQjd-vo35F_EXW off=7878449 len=499063 2021-07-31T02:17:30.361 INFO:tasks.workunit.client.0.smithi045.stdout:op 3749 completed, throughput=5MB/sec 2021-07-31T02:17:30.362 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-qka3IgmO-HxqAKo off=4530289 len=1748386 2021-07-31T02:17:30.366 INFO:tasks.workunit.client.0.smithi045.stdout:op 3750 completed, throughput=5MB/sec 2021-07-31T02:17:30.366 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-j4W_loK7ijHnpAs off=17783876 len=1312410 2021-07-31T02:17:30.373 INFO:tasks.workunit.client.0.smithi045.stdout:op 3751 completed, throughput=5MB/sec 2021-07-31T02:17:30.374 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-TQiVdhbsmGFe2eh off=4047677 len=1652134 2021-07-31T02:17:30.382 INFO:tasks.workunit.client.0.smithi045.stdout:op 3752 completed, throughput=5MB/sec 2021-07-31T02:17:30.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:30 smithi154 conmon[34553]: cluster 2021-07-31T02:17:28.224497+0000 mgr.x (mgr.24542) 1292 : cluster [DBG] pgmap v1779: 9 pgs: 9 active+clean; 127 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2021-07-31T02:17:30.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:30 smithi154 conmon[34553]: 2021-07-31T02:17:30.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:30 smithi045 conmon[39718]: cluster 2021-07-31T02:17:28.224497+0000 mgr.x (mgr.24542) 1292 : cluster [DBG] pgmap v1779: 9 pgs: 9 active+clean; 127 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2021-07-31T02:17:30.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:30 smithi045 conmon[33655]: cluster 2021-07-31T02:17:28.224497+0000 mgr.x (mgr.24542) 1292 2021-07-31T02:17:30.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:30 smithi045 conmon[33655]: : cluster [DBG] pgmap v1779: 9 pgs: 9 active+clean; 127 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2021-07-31T02:17:31.382 INFO:tasks.workunit.client.0.smithi045.stdout: 730: throughput=4.99MB/sec pending data=0 2021-07-31T02:17:31.382 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-XiMNv6h41io275M off=150421 len=1108245 2021-07-31T02:17:31.384 INFO:tasks.workunit.client.0.smithi045.stdout:op 3753 completed, throughput=5MB/sec 2021-07-31T02:17:31.384 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ZuhwE9wp836OzOF off=397915 len=981803 2021-07-31T02:17:31.386 INFO:tasks.workunit.client.0.smithi045.stdout:op 3754 completed, throughput=5MB/sec 2021-07-31T02:17:31.386 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-6lkxRX7plUzqBeb off=7092660 len=1355809 2021-07-31T02:17:31.392 INFO:tasks.workunit.client.0.smithi045.stdout:op 3755 completed, throughput=5MB/sec 2021-07-31T02:17:31.393 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-3JRJUeoknwuE-2Q off=10387860 len=541873 2021-07-31T02:17:31.397 INFO:tasks.workunit.client.0.smithi045.stdout:op 3756 completed, throughput=5MB/sec 2021-07-31T02:17:31.397 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-0KnXzyu62-OHoqe off=10119494 len=725542 2021-07-31T02:17:31.398 INFO:tasks.workunit.client.0.smithi045.stdout:op 3757 completed, throughput=5MB/sec 2021-07-31T02:17:31.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[34553]: audit 2021-07-31T02:17:31.049588+0000 mon.b (mon.2) 220 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:17:31.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[34553]: audit 2021-07-31T02:17:31.051088+0000 mon.b (mon.2) 221 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:17:31.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[34553]: audit 2021-07- 2021-07-31T02:17:31.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[34553]: 31T02:17:31.051985+0000 mon.b ( 2021-07-31T02:17:31.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[34553]: mon.2) 222 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:17:31.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: debug 2021-07-31T02:17:31.095+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:17:31.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:17:31.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:17:31.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: yield 2021-07-31T02:17:31.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:17:31.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:17:31.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:17:31.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:17:31.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:17:31.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:17:31.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:17:31.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:17:31.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:17:31.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: task.result() 2021-07-31T02:17:31.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:17:31.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: debug 2021-07-31T02:17:31.106+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:17:31.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:17:31.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:17:31.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: yield 2021-07-31T02:17:31.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:17:31.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:17:31.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:17:31.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:17:31.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:17:31.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:17:31.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:17:31.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:17:31.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:17:31.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: task.result() 2021-07-31T02:17:31.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:17:31.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: debug 2021-07-31T02:17:31.131+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:17:31.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:17:31.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:17:31.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: yield 2021-07-31T02:17:31.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:17:31.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:17:31.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:17:31.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:17:31.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:17:31.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:17:31.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:17:31.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:17:31.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:17:31.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: task.result() 2021-07-31T02:17:31.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:17:31.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: debug 2021-07-31T02:17:31.132+0000 7fcb93961700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:17:31.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:17:31.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:17:31.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: yield 2021-07-31T02:17:31.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:17:31.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:17:31.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:17:31.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:17:31.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:17:31.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:17:31.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:17:31.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:17:31.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:17:31.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: task.result() 2021-07-31T02:17:31.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:17:31.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: 2021-07-31T02:17:31.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:17:31.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: 2021-07-31T02:17:31.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:17:31.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:17:31.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:17:31.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:17:31.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:17:31.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:17:31.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:17:31.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:17:31.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:17:31.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:17:31.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:17:31.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:17:31.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:17:31.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:17:31.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:17:31.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:17:31.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:17:31.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:17:31.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: raise 2021-07-31T02:17:31.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:17:31.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:17:31.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:17:31.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:17:31.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:17:31.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: debug 2021-07-31T02:17:31.140+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:17:31.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:17:31.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:17:31.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: yield 2021-07-31T02:17:31.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:17:31.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:17:31.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:17:31.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:17:31.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:17:31.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:17:31.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:17:31.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:17:31.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:17:31.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: task.result() 2021-07-31T02:17:31.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:17:31.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: debug 2021-07-31T02:17:31.141+0000 7fcb94162700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:17:31.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:17:31.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:17:31.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: yield 2021-07-31T02:17:31.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:17:31.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:17:31.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:17:31.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:17:31.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:17:31.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:17:31.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:17:31.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:17:31.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:17:31.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: task.result() 2021-07-31T02:17:31.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:17:31.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: 2021-07-31T02:17:31.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:17:31.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: 2021-07-31T02:17:31.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:17:31.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:17:31.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:17:31.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:17:31.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:17:31.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:17:31.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:17:31.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:17:31.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:17:31.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:17:31.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:17:31.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:17:31.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:17:31.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:17:31.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:17:31.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:17:31.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:17:31.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:17:31.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: raise 2021-07-31T02:17:31.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:17:31.535 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:17:31.535 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:17:31.535 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:17:31.535 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:17:31 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:17:31.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:31 smithi045 conmon[33655]: audit 2021-07-31T02:17:31.049588+0000 mon.b (mon.2) 220 2021-07-31T02:17:31.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:31 smithi045 conmon[33655]: : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:17:31.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:31 smithi045 conmon[33655]: audit 2021-07-31T02:17:31.051088+0000 mon.b (mon.2) 221 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:17:31.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:31 smithi045 conmon[33655]: audit 2021-07-31T02:17:31.051985+0000 mon.b (mon.2) 2021-07-31T02:17:31.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:31 smithi045 conmon[33655]: 222 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:17:31.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:31 smithi045 conmon[39718]: audit 2021-07-31T02:17 2021-07-31T02:17:31.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:31 smithi045 conmon[39718]: :31.049588+0000 mon.b (mon.2) 220 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:17:31.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:31 smithi045 conmon[39718]: audit 2021 2021-07-31T02:17:31.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:31 smithi045 conmon[39718]: -07-31T02:17:31.051088+0000 mon.b 2021-07-31T02:17:31.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:31 smithi045 conmon[39718]: (mon.2) 221 : audit [DBG] 2021-07-31T02:17:31.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:31 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:17:31.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:31 smithi045 conmon[39718]: audit 2021 2021-07-31T02:17:31.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:31 smithi045 conmon[39718]: -07-31T02:17:31.051985+0000 mon.b ( 2021-07-31T02:17:31.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:31 smithi045 conmon[39718]: mon.2) 222 : audit [INF] 2021-07-31T02:17:31.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:31 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:17:32.398 INFO:tasks.workunit.client.0.smithi045.stdout: 731: throughput=4.99MB/sec pending data=0 2021-07-31T02:17:32.399 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-wsQBwyhmBM1Xdd3 off=14844466 len=813423 2021-07-31T02:17:32.403 INFO:tasks.workunit.client.0.smithi045.stdout:op 3758 completed, throughput=4.99MB/sec 2021-07-31T02:17:32.404 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-5Lj53BLW3F43rOj off=1438475 len=678121 2021-07-31T02:17:32.405 INFO:tasks.workunit.client.0.smithi045.stdout:op 3759 completed, throughput=5MB/sec 2021-07-31T02:17:32.405 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-q_MheieW073W6V8 off=3891093 len=219616 2021-07-31T02:17:32.406 INFO:tasks.workunit.client.0.smithi045.stdout:op 3760 completed, throughput=5MB/sec 2021-07-31T02:17:32.406 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ePju3E0kDtoQAg1 off=11111686 len=468296 2021-07-31T02:17:32.407 INFO:tasks.workunit.client.0.smithi045.stdout:op 3761 completed, throughput=5MB/sec 2021-07-31T02:17:32.407 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-_7Xe_gU7IcDiq7w off=5999764 len=1636031 2021-07-31T02:17:32.416 INFO:tasks.workunit.client.0.smithi045.stdout:op 3762 completed, throughput=5MB/sec 2021-07-31T02:17:32.417 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-P9WuDAhE3YxGVfS off=2839878 len=1237733 2021-07-31T02:17:32.419 INFO:tasks.workunit.client.0.smithi045.stdout:op 3763 completed, throughput=5MB/sec 2021-07-31T02:17:32.419 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-w_DxJbp30QCTwyn off=3269806 len=133570 2021-07-31T02:17:32.420 INFO:tasks.workunit.client.0.smithi045.stdout:op 3764 completed, throughput=5MB/sec 2021-07-31T02:17:32.420 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-LxfEHBTN_mR8V_0 off=0 len=1954174 2021-07-31T02:17:32.430 INFO:tasks.workunit.client.0.smithi045.stdout:op 3765 completed, throughput=5MB/sec 2021-07-31T02:17:32.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: cluster 2021-07-31T02:17:30.224942+0000 mgr.x (mgr.24542) 1293 : cluster [DBG] pgmap v1780: 9 pgs: 9 active+clean; 127 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 3.0 MiB/s rd, 3.3 MiB/s wr, 6 op/s 2021-07-31T02:17:32.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: cephadm 2021-07-31T02:17:31.052745+0000 mgr.x (mgr.24542) 1294 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:17:32.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: cephadm 2021-07-31T02:17:31.096087+0000 mgr.x (mgr.24542) 1295 : cephadm [ERR] 2021-07-31T02:17:32.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:17:32.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:17:32.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: yield 2021-07-31T02:17:32.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:17:32.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:17:32.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:17:32.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:17:32.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:17:32.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:17:32.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:17:32.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:17:32.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:17:32.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: task.result() 2021-07-31T02:17:32.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:17:32.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: cephadm 2021-07-31T02:17:31.096706+0000 mgr.x (mgr.24542) 1296 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:17:32.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: cephadm 2021-07-31T02:17:31.107788+0000 mgr.x (mgr.24542) 1297 : cephadm [ERR] 2021-07-31T02:17:32.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:17:32.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:17:32.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: yield 2021-07-31T02:17:32.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:17:32.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:17:32.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:17:32.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:17:32.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:17:32.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:17:32.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:17:32.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:17:32.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:17:32.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: task.result() 2021-07-31T02:17:32.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:17:32.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: cephadm 2021-07 2021-07-31T02:17:32.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: -31T02:17:31.108193+0000 mgr.x (mgr.24542) 1298 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:17:32.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: cephadm 2021-07-31T02:17:31.132797+0000 mgr.x (mgr.24542) 1299 : cephadm [ERR] 2021-07-31T02:17:32.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:17:32.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:17:32.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: yield 2021-07-31T02:17:32.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:17:32.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:17:32.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:17:32.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:17:32.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:17:32.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:17:32.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:17:32.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:17:32.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:17:32.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: task.result() 2021-07-31T02:17:32.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:17:32.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: cephadm 2021-07-31T02:17:32.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: 2021-07-31T02:17:31.133268+0000 mgr.x (mgr.24542) 1300 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:17:32.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:17:32.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:17:32.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: yield 2021-07-31T02:17:32.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:17:32.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:17:32.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:17:32.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:17:32.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:17:32.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:17:32.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:17:32.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:17:32.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:17:32.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: task.result() 2021-07-31T02:17:32.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:17:32.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: 2021-07-31T02:17:32.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:17:32.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: 2021-07-31T02:17:32.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:17:32.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:17:32.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:17:32.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:17:32.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:17:32.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:17:32.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:17:32.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:17:32.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:17:32.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:17:32.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:17:32.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:17:32.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:17:32.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:17:32.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:17:32.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:17:32.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:17:32.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:17:32.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: raise 2021-07-31T02:17:32.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:17:32.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:17:32.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:17:32.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:17:32.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:17:32.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: cephadm 2021-07-31T02:17:31.141845+0000 mgr.x (mgr.24542) 1301 : cephadm [ERR] 2021-07-31T02:17:32.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:17:32.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:17:32.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: yield 2021-07-31T02:17:32.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:17:32.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:17:32.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:17:32.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:17:32.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:17:32.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:17:32.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:17:32.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:17:32.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:17:32.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: task.result() 2021-07-31T02:17:32.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:17:32.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: cephadm 2021-07-31T02:17:31.142232+0000 mgr.x (mgr.24542) 1302 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:17:32.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:17:32.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:17:32.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: yield 2021-07-31T02:17:32.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:17:32.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:17:32.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:17:32.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:17:32.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:17:32.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:17:32.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:17:32.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:17:32.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:17:32.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: task.result() 2021-07-31T02:17:32.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:17:32.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: 2021-07-31T02:17:32.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:17:32.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: 2021-07-31T02:17:32.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:17:32.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:17:32.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:17:32.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:17:32.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:17:32.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:17:32.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:17:32.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:17:32.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:17:32.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:17:32.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:17:32.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:17:32.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:17:32.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:17:32.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:17:32.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:17:32.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:17:32.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:17:32.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: raise 2021-07-31T02:17:32.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:17:32.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:17:32.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:17:32.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:17:32.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:32 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:17:32.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: cluster 2021-07-31T02:17:30.224942+0000 mgr.x (mgr.24542) 1293 2021-07-31T02:17:32.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: : cluster [DBG] pgmap v1780: 9 pgs: 9 active+clean; 127 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 3.0 MiB/s rd, 3.3 MiB/s wr, 6 op/s 2021-07-31T02:17:32.539 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: cephadm 2021-07-31T02:17:31.052745+0000 mgr.x (mgr.24542) 1294 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:17:32.539 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: cephadm 2021-07-31T 2021-07-31T02:17:32.539 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: 02:17:31.096087+0000 mgr.x (mgr.24542) 1295 2021-07-31T02:17:32.539 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: : cephadm [ERR] 2021-07-31T02:17:32.539 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:17:32.540 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:17:32.540 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: yield 2021-07-31T02:17:32.540 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:17:32.540 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:17:32.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:17:32.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:17:32.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:17:32.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:17:32.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:17:32.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:17:32.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:17:32.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: task.result() 2021-07-31T02:17:32.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:17:32.543 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: cephadm 2021-07-31T02:17:31.096706+0000 mgr.x (mgr. 2021-07-31T02:17:32.543 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: 24542) 1296 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:17:32.543 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: 2021-07-31T02:17:32.543 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: cephadm 2021 2021-07-31T02:17:32.544 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: -07-31T02:17:31.107788 2021-07-31T02:17:32.544 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542 2021-07-31T02:17:32.544 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: ) 1297 : cephadm [ERR] 2021-07-31T02:17:32.544 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:17:32.544 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:17:32.545 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: yield 2021-07-31T02:17:32.545 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:17:32.545 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:17:32.545 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:17:32.546 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:17:32.546 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:17:32.546 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:17:32.546 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:17:32.547 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:17:32.547 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:17:32.547 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: task.result() 2021-07-31T02:17:32.547 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:17:32.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: cluster 2021-07-31T02:17: 2021-07-31T02:17:32.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: 30.224942+0000 mgr.x (mgr.24542) 1293 : cluster [DBG] pgmap v1780: 9 pgs: 9 active+clean; 127 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 3.0 MiB/s rd, 3.3 MiB/s wr, 6 op/s 2021-07-31T02:17:32.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: cephadm 2021- 2021-07-31T02:17:32.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: 07-31T02:17:31.052745 2021-07-31T02:17:32.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 2021-07-31T02:17:32.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: 1294 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:17:32.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: cephadm 2021- 2021-07-31T02:17:32.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: 07-31T02:17:31. 2021-07-31T02:17:32.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: 096087+0000 mgr.x (mgr.24542 2021-07-31T02:17:32.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: ) 1295 : cephadm [ERR] 2021-07-31T02:17:32.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:17:32.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:17:32.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: yield 2021-07-31T02:17:32.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:17:32.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:17:32.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:17:32.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:17:32.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:17:32.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:17:32.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:17:32.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:17:32.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:17:32.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: task.result() 2021-07-31T02:17:32.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:17:32.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: cephadm 2021-07-31T02:17:31.096706+0000 mgr.x (mgr.24542) 1296 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:17:32.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: cephadm 2021- 2021-07-31T02:17:32.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: 07-31T02:17:31.107788 2021-07-31T02:17:32.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 2021-07-31T02:17:32.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: 1297 : cephadm [ERR] 2021-07-31T02:17:32.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:17:32.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:17:32.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: yield 2021-07-31T02:17:32.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:17:32.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:17:32.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:17:32.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:17:32.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:17:32.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:17:32.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:17:32.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:17:32.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:17:32.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: task.result() 2021-07-31T02:17:32.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: 2021-07-31T02:17:32.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: cephadm 2021-07-31T02:17:31.108193+0000 mgr.x (mgr.24542) 1298 : 2021-07-31T02:17:32.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:17:32.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: cephadm 2021 2021-07-31T02:17:32.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: -07-31T02:17:31.132797 2021-07-31T02:17:32.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542 2021-07-31T02:17:32.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: ) 1299 : cephadm [ERR] 2021-07-31T02:17:32.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:17:32.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:17:32.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: yield 2021-07-31T02:17:32.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:17:32.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:17:32.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:17:32.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:17:32.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:17:32.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:17:32.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:17:32.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:17:32.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:17:32.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: task.result() 2021-07-31T02:17:32.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:17:32.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: 2021-07-31T02:17:32.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: cephadm 2021-07-31T02: 2021-07-31T02:17:32.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: 17:31.133268+0000 mgr.x (mgr.24542) 1300 2021-07-31T02:17:32.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:17:32.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:17:32.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:17:32.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: yield 2021-07-31T02:17:32.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:17:32.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:17:32.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:17:32.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:17:32.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:17:32.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:17:32.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:17:32.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:17:32.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:17:32.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: task.result() 2021-07-31T02:17:32.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:17:32.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: 2021-07-31T02:17:32.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:17:32.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: 2021-07-31T02:17:32.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:17:32.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:17:32.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:17:32.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:17:32.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: cephadm 2021-07-31T02:17:31.108193+0000 mgr.x (mgr.24542) 1298 : cephadm 2021-07-31T02:17:32.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:17:32.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: cephadm 2021 2021-07-31T02:17:32.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: -07-31T02:17:31. 2021-07-31T02:17:32.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: 132797+0000 mgr.x (mgr.24542 2021-07-31T02:17:32.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: ) 1299 : cephadm [ERR] 2021-07-31T02:17:32.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:17:32.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:17:32.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: yield 2021-07-31T02:17:32.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:17:32.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:17:32.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:17:32.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:17:32.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:17:32.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:17:32.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:17:32.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:17:32.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:17:32.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: task.result() 2021-07-31T02:17:32.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:17:32.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: 2021-07-31T02:17:32.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: cephadm 2021-07-31T02:17:31.133268+0000 mgr.x ( 2021-07-31T02:17:32.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: mgr.24542) 1300 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:17:32.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:17:32.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:17:32.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: yield 2021-07-31T02:17:32.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:17:32.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:17:32.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:17:32.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:17:32.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:17:32.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:17:32.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:17:32.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:17:32.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:17:32.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: task.result() 2021-07-31T02:17:32.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:17:32.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: 2021-07-31T02:17:32.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:17:32.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: 2021-07-31T02:17:32.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:17:32.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:17:32.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:17:32.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:17:32.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:17:32.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:17:32.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:17:32.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:17:32.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:17:32.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:17:32.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:17:32.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:17:32.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:17:32.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:17:32.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:17:32.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:17:32.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:17:32.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:17:32.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: raise 2021-07-31T02:17:32.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:17:32.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:17:32.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:17:32.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:17:32.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:17:32.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: cephadm 2021-07-31T02:17:31.141845+0000 mgr.x (mgr.24542) 1301 : cephadm [ERR] 2021-07-31T02:17:32.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:17:32.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:17:32.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: yield 2021-07-31T02:17:32.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:17:32.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:17:32.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:17:32.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:17:32.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:17:32.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:17:32.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:17:32.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:17:32.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:17:32.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: task.result() 2021-07-31T02:17:32.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:17:32.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: cephadm 2021-07-31T02:17:31.142232+0000 mgr.x (mgr.24542) 1302 : cephadm [ERR] 2021-07-31T02:17:32.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:17:32.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:17:32.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:17:32.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:17:32.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:17:32.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:17:32.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:17:32.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:17:32.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:17:32.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:17:32.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:17:32.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:17:32.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:17:32.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:17:32.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:17:32.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:17:32.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:17:32.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: raise 2021-07-31T02:17:32.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:17:32.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:17:32.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:17:32.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:17:32.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:17:32.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: cephadm 2021-07-31T02:17:31.141845+0000 mgr.x (mgr.24542) 1301 : cephadm [ERR] 2021-07-31T02:17:32.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:17:32.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:17:32.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: yield 2021-07-31T02:17:32.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:17:32.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:17:32.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:17:32.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:17:32.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:17:32.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:17:32.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:17:32.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:17:32.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:17:32.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: task.result() 2021-07-31T02:17:32.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:17:32.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: cephadm 2021-07-31T02:17:31.142232+0000 mgr.x (mgr.24542) 1302 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:17:32.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:17:32.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:17:32.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:17:32.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: yield 2021-07-31T02:17:32.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:17:32.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:17:32.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:17:32.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:17:32.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:17:32.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:17:32.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:17:32.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:17:32.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:17:32.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: task.result() 2021-07-31T02:17:32.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:17:32.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: 2021-07-31T02:17:32.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:17:32.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: 2021-07-31T02:17:32.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:17:32.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:17:32.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:17:32.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:17:32.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:17:32.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:17:32.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:17:32.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:17:32.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:17:32.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:17:32.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:17:32.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:17:32.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:17:32.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:17:32.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:17:32.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:17:32.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:17:32.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:17:32.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: raise 2021-07-31T02:17:32.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:17:32.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:17:32.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:17:32.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:17:32.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:17:32.607 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: yield 2021-07-31T02:17:32.607 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:17:32.607 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:17:32.607 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:17:32.607 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:17:32.608 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:17:32.608 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:17:32.608 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:17:32.608 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:17:32.609 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:17:32.609 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: task.result() 2021-07-31T02:17:32.609 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:17:32.609 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: 2021-07-31T02:17:32.609 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:17:32.609 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: 2021-07-31T02:17:32.610 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:17:32.610 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:17:32.610 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:17:32.610 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:17:32.610 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:17:32.611 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:17:32.611 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:17:32.611 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:17:32.611 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:17:32.612 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:17:32.612 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:17:32.612 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:17:32.612 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:17:32.613 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:17:32.613 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:17:32.613 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:17:32.613 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:17:32.614 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:17:32.614 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: raise 2021-07-31T02:17:32.614 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:17:32.614 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:17:32.615 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:17:32.615 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:17:32.615 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:32 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:17:33.431 INFO:tasks.workunit.client.0.smithi045.stdout: 732: throughput=5MB/sec pending data=0 2021-07-31T02:17:33.431 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-zpG6MrLsaxB6j8U off=3855292 len=572535 2021-07-31T02:17:33.432 INFO:tasks.workunit.client.0.smithi045.stdout:op 3766 completed, throughput=5MB/sec 2021-07-31T02:17:33.433 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-UCr0xSDK4yIx5wf off=14397344 len=1954395 2021-07-31T02:17:33.443 INFO:tasks.workunit.client.0.smithi045.stdout:op 3767 completed, throughput=5MB/sec 2021-07-31T02:17:33.444 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-DPc4lMjqFxjGGbK off=2896347 len=544273 2021-07-31T02:17:33.444 INFO:tasks.workunit.client.0.smithi045.stdout:op 3768 completed, throughput=5MB/sec 2021-07-31T02:17:33.445 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-G7rdVnvvzzHKzi7 off=1016525 len=297105 2021-07-31T02:17:33.446 INFO:tasks.workunit.client.0.smithi045.stdout:op 3769 completed, throughput=5MB/sec 2021-07-31T02:17:33.446 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-9oZMu-DELMHHAQY off=14630781 len=1657440 2021-07-31T02:17:33.453 INFO:tasks.workunit.client.0.smithi045.stdout:op 3770 completed, throughput=5MB/sec 2021-07-31T02:17:34.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:17:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:17:33.560Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:17:34.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:17:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:17:33.560Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:17:34.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:17:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:17:33.561Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:17:34.453 INFO:tasks.workunit.client.0.smithi045.stdout: 733: throughput=5MB/sec pending data=0 2021-07-31T02:17:34.454 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-YN_t4g1ZYj95jn4 off=5585960 len=1475831 2021-07-31T02:17:34.457 INFO:tasks.workunit.client.0.smithi045.stdout:op 3771 completed, throughput=5MB/sec 2021-07-31T02:17:34.457 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-wFsl5bfNHeXvdKB off=13712539 len=724930 2021-07-31T02:17:34.461 INFO:tasks.workunit.client.0.smithi045.stdout:op 3772 completed, throughput=5MB/sec 2021-07-31T02:17:34.462 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-A6BnrEK1dZcs7BV off=2681023 len=894840 2021-07-31T02:17:34.467 INFO:tasks.workunit.client.0.smithi045.stdout:op 3773 completed, throughput=5MB/sec 2021-07-31T02:17:34.467 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-zIzQAjR01F7I-X- off=16182945 len=45784 2021-07-31T02:17:34.468 INFO:tasks.workunit.client.0.smithi045.stdout:op 3774 completed, throughput=5MB/sec 2021-07-31T02:17:34.468 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Q-TuY2Dr1r1CDuv off=8902715 len=1272243 2021-07-31T02:17:34.474 INFO:tasks.workunit.client.0.smithi045.stdout:op 3775 completed, throughput=5MB/sec 2021-07-31T02:17:34.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:34 smithi154 conmon[34553]: cluster 2021-07-31T02:17:32.225300+0000 mgr.x (mgr.24542) 1303 : cluster 2021-07-31T02:17:34.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:34 smithi154 conmon[34553]: [DBG] pgmap v1781: 9 pgs: 9 active+clean; 127 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:17:34.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:34 smithi045 conmon[39718]: cluster 2021-07-31T02:17:32.225300+0000 mgr.x (mgr.24542) 1303 : cluster [DBG] pgmap v1781: 9 pgs: 9 active+clean; 127 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:17:34.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:34 smithi045 conmon[33655]: cluster 2021-07-31T02:17:32.225300+0000 mgr.x (mgr. 2021-07-31T02:17:34.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:34 smithi045 conmon[33655]: 24542) 1303 : cluster [DBG] pgmap v1781: 9 pgs: 9 active+clean; 127 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:17:35.474 INFO:tasks.workunit.client.0.smithi045.stdout: 734: throughput=4.99MB/sec pending data=0 2021-07-31T02:17:35.475 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-xaDL8bTJ7KTw4oY off=12148599 len=1447685 2021-07-31T02:17:35.482 INFO:tasks.workunit.client.0.smithi045.stdout:op 3776 completed, throughput=5MB/sec 2021-07-31T02:17:35.483 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-YjahfzkMWgwGs6b off=0 len=1103473 2021-07-31T02:17:35.486 INFO:tasks.workunit.client.0.smithi045.stdout:op 3777 completed, throughput=5MB/sec 2021-07-31T02:17:35.486 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-qTVdeHaJA1bUkIC off=385979 len=192442 2021-07-31T02:17:35.489 INFO:tasks.workunit.client.0.smithi045.stdout:op 3778 completed, throughput=5MB/sec 2021-07-31T02:17:35.489 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-862fVxjfXcwmFvr off=9822437 len=2017022 2021-07-31T02:17:35.499 INFO:tasks.workunit.client.0.smithi045.stdout:op 3779 completed, throughput=5MB/sec 2021-07-31T02:17:35.499 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-jq8cyLUc7tmQdz1 off=4163650 len=1523344 2021-07-31T02:17:35.501 INFO:tasks.workunit.client.0.smithi045.stdout:op 3780 completed, throughput=5MB/sec 2021-07-31T02:17:36.501 INFO:tasks.workunit.client.0.smithi045.stdout: 735: throughput=5MB/sec pending data=0 2021-07-31T02:17:36.502 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-U5hZkXC9Rfmc1_L off=10282570 len=660302 2021-07-31T02:17:36.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:36 smithi154 conmon[34553]: cluster 2021-07-31T02:17:34.225791+0000 mgr.x (mgr.24542) 1304 : cluster [DBG] pgmap v1782: 9 pgs: 9 active+clean; 127 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 2.8 MiB/s rd, 3.6 MiB/s wr, 6 op/s 2021-07-31T02:17:36.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:36 smithi154 conmon[34553]: 2021-07-31T02:17:36.506 INFO:tasks.workunit.client.0.smithi045.stdout:op 3781 completed, throughput=5MB/sec 2021-07-31T02:17:36.506 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-zWcCUAIn5NqsZQ4 off=7972232 len=961142 2021-07-31T02:17:36.508 INFO:tasks.workunit.client.0.smithi045.stdout:op 3782 completed, throughput=5MB/sec 2021-07-31T02:17:36.508 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-3gs9PmhpoukdI26 off=660357 len=334436 2021-07-31T02:17:36.509 INFO:tasks.workunit.client.0.smithi045.stdout:op 3783 completed, throughput=5MB/sec 2021-07-31T02:17:36.509 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-AmuUlxb0ZOdXtpx off=2611426 len=377386 2021-07-31T02:17:36.510 INFO:tasks.workunit.client.0.smithi045.stdout:op 3784 completed, throughput=5MB/sec 2021-07-31T02:17:36.510 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-9B-Qb7v5eYmzR6w off=3103501 len=1021625 2021-07-31T02:17:36.512 INFO:tasks.workunit.client.0.smithi045.stdout:op 3785 completed, throughput=5MB/sec 2021-07-31T02:17:36.512 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-l4rFQQxwjHdHGhC off=3230466 len=1569543 2021-07-31T02:17:36.519 INFO:tasks.workunit.client.0.smithi045.stdout:op 3786 completed, throughput=5MB/sec 2021-07-31T02:17:36.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:36 smithi045 conmon[39718]: cluster 2021-07-31T02:17:34.225791+0000 mgr.x (mgr. 2021-07-31T02:17:36.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:36 smithi045 conmon[39718]: 24542) 1304 : cluster [DBG] pgmap v1782: 9 pgs: 9 active+clean; 127 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 2.8 MiB/s rd, 3.6 MiB/s wr, 6 op/s 2021-07-31T02:17:36.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:36 smithi045 conmon[33655]: cluster 2021-07-31T02:17:34.225791 2021-07-31T02:17:36.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:36 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 1304 : cluster [DBG] pgmap v1782: 9 pgs: 9 active+clean; 127 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 2.8 MiB/s rd, 3.6 MiB/s wr, 6 op/s 2021-07-31T02:17:37.520 INFO:tasks.workunit.client.0.smithi045.stdout: 736: throughput=4.99MB/sec pending data=0 2021-07-31T02:17:37.520 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-QbrbDDQ11Q1UFRj off=17478326 len=570622 2021-07-31T02:17:37.525 INFO:tasks.workunit.client.0.smithi045.stdout:op 3787 completed, throughput=5MB/sec 2021-07-31T02:17:37.525 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-NsFkGNBnDVyFFNw off=13461614 len=1010311 2021-07-31T02:17:37.530 INFO:tasks.workunit.client.0.smithi045.stdout:op 3788 completed, throughput=5MB/sec 2021-07-31T02:17:37.530 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-p_l3mwRkZUuFU0L off=1377439 len=1174363 2021-07-31T02:17:37.532 INFO:tasks.workunit.client.0.smithi045.stdout:op 3789 completed, throughput=5MB/sec 2021-07-31T02:17:37.533 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-WfkcZVbHyB1sYuG off=11427429 len=15747 2021-07-31T02:17:37.534 INFO:tasks.workunit.client.0.smithi045.stdout:op 3790 completed, throughput=5MB/sec 2021-07-31T02:17:37.534 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-qOGdC6-9VfLAa6j off=16733645 len=1531143 2021-07-31T02:17:37.541 INFO:tasks.workunit.client.0.smithi045.stdout:op 3791 completed, throughput=5MB/sec 2021-07-31T02:17:38.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:38 smithi154 conmon[34553]: cluster 2021-07-31T02:17:36.226356+0000 mgr.x (mgr.24542) 1305 : cluster [DBG] 2021-07-31T02:17:38.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:38 smithi154 conmon[34553]: pgmap v1783: 9 pgs: 9 active+clean; 127 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:17:38.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:38 smithi045 conmon[33655]: cluster 2021-07-31T02:17:36.226356+0000 mgr.x (mgr.24542 2021-07-31T02:17:38.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:38 smithi045 conmon[33655]: ) 1305 : cluster [DBG] pgmap v1783: 9 pgs: 9 active+clean; 127 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:17:38.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:38 smithi045 conmon[39718]: cluster 2021-07-31T02:17:36.226356+0000 2021-07-31T02:17:38.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:38 smithi045 conmon[39718]: mgr.x (mgr.24542) 1305 : cluster [DBG] pgmap v1783: 9 pgs: 9 active+clean; 127 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:17:38.541 INFO:tasks.workunit.client.0.smithi045.stdout: 737: throughput=4.99MB/sec pending data=0 2021-07-31T02:17:38.541 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-U5O19p5vJb_iFEr off=3301681 len=1633402 2021-07-31T02:17:38.550 INFO:tasks.workunit.client.0.smithi045.stdout:op 3792 completed, throughput=5MB/sec 2021-07-31T02:17:38.550 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-oyh5EmnVxAFlS5n off=6688596 len=1988800 2021-07-31T02:17:38.562 INFO:tasks.workunit.client.0.smithi045.stdout:op 3793 completed, throughput=5MB/sec 2021-07-31T02:17:38.562 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-lAMhYyD7IoN9NtI off=0 len=1179643 2021-07-31T02:17:38.563 INFO:tasks.workunit.client.0.smithi045.stdout:op 3794 completed, throughput=5MB/sec 2021-07-31T02:17:38.564 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-2qwYeQx084owTdt off=15499318 len=194787 2021-07-31T02:17:38.564 INFO:tasks.workunit.client.0.smithi045.stdout:op 3795 completed, throughput=5MB/sec 2021-07-31T02:17:38.565 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-m9aH0hX51l-WuOd off=5387110 len=466108 2021-07-31T02:17:38.565 INFO:tasks.workunit.client.0.smithi045.stdout:op 3796 completed, throughput=5MB/sec 2021-07-31T02:17:39.566 INFO:tasks.workunit.client.0.smithi045.stdout: 738: throughput=4.99MB/sec pending data=0 2021-07-31T02:17:39.566 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-vlxbByxzlvQNOIU off=8469102 len=1247635 2021-07-31T02:17:39.572 INFO:tasks.workunit.client.0.smithi045.stdout:op 3797 completed, throughput=5MB/sec 2021-07-31T02:17:39.573 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-qZkT2s2kMjAVg8h off=1136356 len=1475663 2021-07-31T02:17:39.575 INFO:tasks.workunit.client.0.smithi045.stdout:op 3798 completed, throughput=5MB/sec 2021-07-31T02:17:39.576 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-el-Sr63JiwsjQS4 off=0 len=1150528 2021-07-31T02:17:39.582 INFO:tasks.workunit.client.0.smithi045.stdout:op 3799 completed, throughput=5MB/sec 2021-07-31T02:17:39.582 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-JIb0L-SDfImjQHv off=1550099 len=1249866 2021-07-31T02:17:39.583 INFO:tasks.workunit.client.0.smithi045.stdout:op 3800 completed, throughput=5MB/sec 2021-07-31T02:17:39.584 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-7uqAeB33vZYTu4c off=3161243 len=1266741 2021-07-31T02:17:39.589 INFO:tasks.workunit.client.0.smithi045.stdout:op 3801 completed, throughput=5MB/sec 2021-07-31T02:17:40.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:40 smithi154 conmon[34553]: cluster 2021-07-31T02:17:38.226821+0000 mgr.x (mgr.24542) 1306 : cluster [DBG] 2021-07-31T02:17:40.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:40 smithi154 conmon[34553]: pgmap v1784: 9 pgs: 9 active+clean; 127 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:17:40.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:40 smithi045 conmon[33655]: cluster 2021-07-31T02:17:38.226821+0000 mgr.x (mgr.24542) 1306 2021-07-31T02:17:40.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:40 smithi045 conmon[33655]: : cluster [DBG] pgmap v1784: 9 pgs: 9 active+clean; 127 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:17:40.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:40 smithi045 conmon[39718]: cluster 2021-07-31T02:17:38.226821+0000 mgr.x ( 2021-07-31T02:17:40.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:40 smithi045 conmon[39718]: mgr.24542) 1306 : cluster [DBG] pgmap v1784: 9 pgs: 9 active+clean; 127 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:17:40.590 INFO:tasks.workunit.client.0.smithi045.stdout: 739: throughput=4.99MB/sec pending data=0 2021-07-31T02:17:40.590 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-8z2zEeJ6_STTO5E off=1718225 len=328090 2021-07-31T02:17:40.594 INFO:tasks.workunit.client.0.smithi045.stdout:op 3802 completed, throughput=5MB/sec 2021-07-31T02:17:40.594 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-xoLJsHHSmklxRXN off=1557776 len=891354 2021-07-31T02:17:40.599 INFO:tasks.workunit.client.0.smithi045.stdout:op 3803 completed, throughput=5MB/sec 2021-07-31T02:17:40.599 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-2bp6io-a8KRMCLP off=2781101 len=1982554 2021-07-31T02:17:40.601 INFO:tasks.workunit.client.0.smithi045.stdout:op 3804 completed, throughput=5MB/sec 2021-07-31T02:17:40.601 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-OeVprL6JBCuacaI off=772195 len=98362 2021-07-31T02:17:40.604 INFO:tasks.workunit.client.0.smithi045.stdout:op 3805 completed, throughput=5MB/sec 2021-07-31T02:17:40.604 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ySnNqmRTAy8Qsfq off=1700542 len=1851536 2021-07-31T02:17:40.611 INFO:tasks.workunit.client.0.smithi045.stdout:op 3806 completed, throughput=5MB/sec 2021-07-31T02:17:41.611 INFO:tasks.workunit.client.0.smithi045.stdout: 740: throughput=4.99MB/sec pending data=0 2021-07-31T02:17:41.611 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-LM4jbnD_c-fzeZX off=4292471 len=1914284 2021-07-31T02:17:41.616 INFO:tasks.workunit.client.0.smithi045.stdout:op 3807 completed, throughput=5MB/sec 2021-07-31T02:17:41.617 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-1ntkareiN3GCOQk off=15452570 len=1310438 2021-07-31T02:17:41.620 INFO:tasks.workunit.client.0.smithi045.stdout:op 3808 completed, throughput=5MB/sec 2021-07-31T02:17:41.620 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-l5CTyOi2iDFWLuE off=4333492 len=1143070 2021-07-31T02:17:41.621 INFO:tasks.workunit.client.0.smithi045.stdout:op 3809 completed, throughput=5MB/sec 2021-07-31T02:17:41.622 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-PJLlPNK7aRjRmiR off=1236652 len=538240 2021-07-31T02:17:41.626 INFO:tasks.workunit.client.0.smithi045.stdout:op 3810 completed, throughput=5MB/sec 2021-07-31T02:17:42.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:42 smithi154 conmon[34553]: cluster 2021-07-31T02:17:40.227293+0000 mgr.x (mgr.24542) 1307 : cluster [DBG] 2021-07-31T02:17:42.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:42 smithi154 conmon[34553]: pgmap v1785: 9 pgs: 9 active+clean; 127 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 2.8 MiB/s rd, 3.6 MiB/s wr, 6 op/s 2021-07-31T02:17:42.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:42 smithi045 conmon[33655]: cluster 2021-07-31T02:17:40.227293+0000 mgr.x (mgr.24542) 1307 : 2021-07-31T02:17:42.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:42 smithi045 conmon[33655]: cluster [DBG] pgmap v1785: 9 pgs: 9 active+clean; 127 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 2.8 MiB/s rd, 3.6 MiB/s wr, 6 op/s 2021-07-31T02:17:42.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:42 smithi045 conmon[39718]: cluster 2021-07-31T02:17:40.227293+0000 mgr.x ( 2021-07-31T02:17:42.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:42 smithi045 conmon[39718]: mgr.24542) 1307 : cluster [DBG] pgmap v1785: 9 pgs: 9 active+clean; 127 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 2.8 MiB/s rd, 3.6 MiB/s wr, 6 op/s 2021-07-31T02:17:42.626 INFO:tasks.workunit.client.0.smithi045.stdout: 741: throughput=4.99MB/sec pending data=0 2021-07-31T02:17:42.627 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-cA8kRsiFledNp80 off=572312 len=2011833 2021-07-31T02:17:42.629 INFO:tasks.workunit.client.0.smithi045.stdout:op 3811 completed, throughput=5MB/sec 2021-07-31T02:17:42.630 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-GxlJg5Cw_twV2eN off=12811816 len=617880 2021-07-31T02:17:42.635 INFO:tasks.workunit.client.0.smithi045.stdout:op 3812 completed, throughput=5MB/sec 2021-07-31T02:17:42.635 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-0y2GHz8m_MdiBkS off=21969708 len=1114162 2021-07-31T02:17:42.641 INFO:tasks.workunit.client.0.smithi045.stdout:op 3813 completed, throughput=5MB/sec 2021-07-31T02:17:42.641 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-zbkLLhdWQ-2MVw2 off=7902290 len=239391 2021-07-31T02:17:42.642 INFO:tasks.workunit.client.0.smithi045.stdout:op 3814 completed, throughput=5MB/sec 2021-07-31T02:17:42.642 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-j741i2ThI8C1pyo off=8532151 len=227826 2021-07-31T02:17:42.645 INFO:tasks.workunit.client.0.smithi045.stdout:op 3815 completed, throughput=5MB/sec 2021-07-31T02:17:42.645 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-E80r4o6uHpPeMKj off=685616 len=1900284 2021-07-31T02:17:42.646 INFO:tasks.workunit.client.0.smithi045.stdout:op 3816 completed, throughput=5MB/sec 2021-07-31T02:17:43.647 INFO:tasks.workunit.client.0.smithi045.stdout: 742: throughput=4.99MB/sec pending data=0 2021-07-31T02:17:43.647 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-M90zkSVbhRArPZQ off=11657932 len=1910729 2021-07-31T02:17:43.655 INFO:tasks.workunit.client.0.smithi045.stdout:op 3817 completed, throughput=5MB/sec 2021-07-31T02:17:43.656 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ipVRrVosfhyKsb2 off=1395691 len=1243414 2021-07-31T02:17:43.663 INFO:tasks.workunit.client.0.smithi045.stdout:op 3818 completed, throughput=5MB/sec 2021-07-31T02:17:43.663 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-7y0ggVXGlC6AGnG off=1209490 len=1316435 2021-07-31T02:17:43.669 INFO:tasks.workunit.client.0.smithi045.stdout:op 3819 completed, throughput=5MB/sec 2021-07-31T02:17:44.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:17:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:17:43.561Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:17:44.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:17:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:17:43.561Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:17:44.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:17:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:17:43.561Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:17:44.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:44 smithi154 conmon[34553]: cluster 2021-07-31T02:17:42.227781+0000 mgr.x (mgr.24542) 1308 : cluster [DBG] pgmap v1786: 9 pgs: 9 active+clean; 127 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:17:44.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:44 smithi045 conmon[33655]: cluster 2021-07-31T02:17:42.227781+0000 mgr.x (mgr.24542) 1308 : cluster 2021-07-31T02:17:44.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:44 smithi045 conmon[33655]: [DBG] pgmap v1786: 9 pgs: 9 active+clean; 127 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:17:44.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:44 smithi045 conmon[39718]: cluster 2021-07-31T02:17:42.227781+0000 mgr.x (mgr.24542) 2021-07-31T02:17:44.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:44 smithi045 conmon[39718]: 1308 : cluster [DBG] pgmap v1786: 9 pgs: 9 active+clean; 127 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:17:44.670 INFO:tasks.workunit.client.0.smithi045.stdout: 743: throughput=4.99MB/sec pending data=0 2021-07-31T02:17:44.670 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj--urtldA1KK3dY9e off=2847894 len=464486 2021-07-31T02:17:44.674 INFO:tasks.workunit.client.0.smithi045.stdout:op 3820 completed, throughput=4.99MB/sec 2021-07-31T02:17:44.674 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-mQD-Df0I1W06fe7 off=3381451 len=779549 2021-07-31T02:17:44.678 INFO:tasks.workunit.client.0.smithi045.stdout:op 3821 completed, throughput=5MB/sec 2021-07-31T02:17:44.679 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-lGVP9M1YkcAX6Ah off=13488799 len=1410510 2021-07-31T02:17:44.683 INFO:tasks.workunit.client.0.smithi045.stdout:op 3822 completed, throughput=5MB/sec 2021-07-31T02:17:44.683 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Sdupg3nWQPSaFAV off=8278173 len=1682131 2021-07-31T02:17:44.690 INFO:tasks.workunit.client.0.smithi045.stdout:op 3823 completed, throughput=5MB/sec 2021-07-31T02:17:44.691 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Mfv1nb24-b3mKKf off=368092 len=469583 2021-07-31T02:17:44.693 INFO:tasks.workunit.client.0.smithi045.stdout:op 3824 completed, throughput=5MB/sec 2021-07-31T02:17:44.693 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-r9yczMXvEei9OXV off=4810767 len=1005773 2021-07-31T02:17:44.695 INFO:tasks.workunit.client.0.smithi045.stdout:op 3825 completed, throughput=5MB/sec 2021-07-31T02:17:45.696 INFO:tasks.workunit.client.0.smithi045.stdout: 744: throughput=4.99MB/sec pending data=0 2021-07-31T02:17:45.696 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-bERL7J698xYYMWg off=1413178 len=1767297 2021-07-31T02:17:45.698 INFO:tasks.workunit.client.0.smithi045.stdout:op 3826 completed, throughput=5MB/sec 2021-07-31T02:17:45.698 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-7PeZ0fwc4jlWRaQ off=2364161 len=433306 2021-07-31T02:17:45.699 INFO:tasks.workunit.client.0.smithi045.stdout:op 3827 completed, throughput=5MB/sec 2021-07-31T02:17:45.699 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-KgMO3sJqezBUojZ off=4118623 len=1779274 2021-07-31T02:17:45.707 INFO:tasks.workunit.client.0.smithi045.stdout:op 3828 completed, throughput=5MB/sec 2021-07-31T02:17:45.708 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-xITWRBy-aUMEwW2 off=10018545 len=702877 2021-07-31T02:17:45.712 INFO:tasks.workunit.client.0.smithi045.stdout:op 3829 completed, throughput=5MB/sec 2021-07-31T02:17:46.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:46 smithi154 conmon[34553]: cluster 2021-07-31T02:17:44.228215+0000 mgr.x (mgr.24542) 1309 : cluster [DBG] pgmap v1787: 9 pgs: 9 active+clean; 127 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 2.9 MiB/s rd, 3.5 MiB/s wr, 6 op/s 2021-07-31T02:17:46.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:46 smithi045 conmon[33655]: cluster 2021-07-31T02:17:44.228215+0000 mgr.x (mgr.24542) 1309 : cluster [DBG] pgmap v1787: 9 pgs: 9 active+clean; 127 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 2.9 MiB/s rd, 3.5 MiB/s wr, 6 op/s 2021-07-31T02:17:46.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:46 smithi045 conmon[39718]: cluster 2021-07-31T02:17:44.228215+0000 mgr.x (mgr.24542) 1309 : cluster [DBG] pgmap v1787: 9 pgs: 9 active+clean; 127 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 2.9 MiB/s rd, 3.5 MiB/s wr, 6 op/s 2021-07-31T02:17:46.712 INFO:tasks.workunit.client.0.smithi045.stdout: 745: throughput=4.99MB/sec pending data=0 2021-07-31T02:17:46.713 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-_NV5kMf-faW1ihu off=1850835 len=795005 2021-07-31T02:17:46.714 INFO:tasks.workunit.client.0.smithi045.stdout:op 3830 completed, throughput=4.99MB/sec 2021-07-31T02:17:46.714 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-UcfJINbC7vN9N5r off=490542 len=964258 2021-07-31T02:17:46.720 INFO:tasks.workunit.client.0.smithi045.stdout:op 3831 completed, throughput=5MB/sec 2021-07-31T02:17:46.720 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-WuqHNrvuRCysxFx off=6879421 len=1671425 2021-07-31T02:17:46.728 INFO:tasks.workunit.client.0.smithi045.stdout:op 3832 completed, throughput=5MB/sec 2021-07-31T02:17:46.728 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-UluQt4Y16tU7KU9 off=6656435 len=2034715 2021-07-31T02:17:46.736 INFO:tasks.workunit.client.0.smithi045.stdout:op 3833 completed, throughput=5MB/sec 2021-07-31T02:17:47.736 INFO:tasks.workunit.client.0.smithi045.stdout: 746: throughput=4.99MB/sec pending data=0 2021-07-31T02:17:47.737 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-w81LBwh8NXR9UjI off=927437 len=2086340 2021-07-31T02:17:47.746 INFO:tasks.workunit.client.0.smithi045.stdout:op 3834 completed, throughput=5MB/sec 2021-07-31T02:17:47.746 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-U4QRETZpBRIADCW off=312878 len=1426247 2021-07-31T02:17:47.749 INFO:tasks.workunit.client.0.smithi045.stdout:op 3835 completed, throughput=5MB/sec 2021-07-31T02:17:47.749 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-kngxuiG1Y7uqyaH off=5809332 len=1530645 2021-07-31T02:17:47.752 INFO:tasks.workunit.client.0.smithi045.stdout:op 3836 completed, throughput=5MB/sec 2021-07-31T02:17:47.752 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-7xlv9Rfr6DtMQ44 off=7206802 len=667939 2021-07-31T02:17:47.753 INFO:tasks.workunit.client.0.smithi045.stdout:op 3837 completed, throughput=5MB/sec 2021-07-31T02:17:48.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:48 smithi154 conmon[34553]: cluster 2021-07-31T02:17:46.228659+0000 mgr.x (mgr.24542) 1310 : cluster [DBG] pgmap v1788: 9 pgs: 9 active+clean; 127 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2021-07-31T02:17:48.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:48 smithi045 conmon[39718]: cluster 2021-07-31T02:17:46.228659+0000 mgr.x ( 2021-07-31T02:17:48.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:48 smithi045 conmon[39718]: mgr.24542) 1310 : cluster [DBG] pgmap v1788: 9 pgs: 9 active+clean; 127 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2021-07-31T02:17:48.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:48 smithi045 conmon[33655]: cluster 2021-07-31T02:17:46.228659+0000 mgr.x (mgr.24542) 1310 2021-07-31T02:17:48.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:48 smithi045 conmon[33655]: : cluster [DBG] pgmap v1788: 9 pgs: 9 active+clean; 127 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2021-07-31T02:17:48.753 INFO:tasks.workunit.client.0.smithi045.stdout: 747: throughput=4.99MB/sec pending data=0 2021-07-31T02:17:48.754 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-kHPoAR1oKVwFUi6 off=13133350 len=692654 2021-07-31T02:17:48.758 INFO:tasks.workunit.client.0.smithi045.stdout:op 3838 completed, throughput=5MB/sec 2021-07-31T02:17:48.758 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-n_4pW42hdaqA55P off=3054790 len=204607 2021-07-31T02:17:48.761 INFO:tasks.workunit.client.0.smithi045.stdout:op 3839 completed, throughput=5MB/sec 2021-07-31T02:17:48.761 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-VctPA5CU15_qpTi off=8183694 len=888503 2021-07-31T02:17:48.766 INFO:tasks.workunit.client.0.smithi045.stdout:op 3840 completed, throughput=5MB/sec 2021-07-31T02:17:48.766 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Wx2PVd_mI2zvfms off=3655518 len=63192 2021-07-31T02:17:48.767 INFO:tasks.workunit.client.0.smithi045.stdout:op 3841 completed, throughput=5MB/sec 2021-07-31T02:17:48.767 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-XQyF0nBJws1QwNm off=5619410 len=749362 2021-07-31T02:17:48.768 INFO:tasks.workunit.client.0.smithi045.stdout:op 3842 completed, throughput=5MB/sec 2021-07-31T02:17:48.768 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-jrb-q-MX4Amsppc off=9570310 len=1511778 2021-07-31T02:17:48.775 INFO:tasks.workunit.client.0.smithi045.stdout:op 3843 completed, throughput=5MB/sec 2021-07-31T02:17:48.776 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-3nLokZ76j0F56qL off=8701220 len=402179 2021-07-31T02:17:48.779 INFO:tasks.workunit.client.0.smithi045.stdout:op 3844 completed, throughput=5MB/sec 2021-07-31T02:17:48.779 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-QjjCOVBJEBPtPaB off=663820 len=173032 2021-07-31T02:17:48.781 INFO:tasks.workunit.client.0.smithi045.stdout:op 3845 completed, throughput=5MB/sec 2021-07-31T02:17:48.782 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-lmt4J6X5R7gp8Nw off=1190721 len=473528 2021-07-31T02:17:48.783 INFO:tasks.workunit.client.0.smithi045.stdout:op 3846 completed, throughput=5MB/sec 2021-07-31T02:17:49.783 INFO:tasks.workunit.client.0.smithi045.stdout: 748: throughput=4.99MB/sec pending data=0 2021-07-31T02:17:49.784 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-9QV_q55rRMJqNrs off=6150940 len=9417 2021-07-31T02:17:49.786 INFO:tasks.workunit.client.0.smithi045.stdout:op 3847 completed, throughput=4.99MB/sec 2021-07-31T02:17:49.786 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-FRqBc52xrk478VW off=1393279 len=1503869 2021-07-31T02:17:49.793 INFO:tasks.workunit.client.0.smithi045.stdout:op 3848 completed, throughput=5MB/sec 2021-07-31T02:17:49.793 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-yX1tpYxZXyCq83a off=2213169 len=1680659 2021-07-31T02:17:49.801 INFO:tasks.workunit.client.0.smithi045.stdout:op 3849 completed, throughput=5MB/sec 2021-07-31T02:17:49.801 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-gFlItd5VB3juWsN off=2111250 len=978463 2021-07-31T02:17:49.803 INFO:tasks.workunit.client.0.smithi045.stdout:op 3850 completed, throughput=5MB/sec 2021-07-31T02:17:49.803 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-iBRadvV0jwSwNUj off=5725998 len=1007685 2021-07-31T02:17:49.808 INFO:tasks.workunit.client.0.smithi045.stdout:op 3851 completed, throughput=5MB/sec 2021-07-31T02:17:50.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:50 smithi154 conmon[34553]: cluster 2021-07-31T02:17:48.229138+0000 mgr.x (mgr.24542) 1311 : cluster [DBG] pgmap v1789: 9 pgs: 9 active+clean; 127 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2021-07-31T02:17:50.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:50 smithi045 conmon[33655]: cluster 2021-07-31T02:17:48.229138+0000 mgr.x (mgr.24542 2021-07-31T02:17:50.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:50 smithi045 conmon[33655]: ) 1311 : cluster [DBG] pgmap v1789: 9 pgs: 9 active+clean; 127 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2021-07-31T02:17:50.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:50 smithi045 conmon[39718]: cluster 2021-07-31T02:17:48.229138+0000 mgr.x (mgr.24542 2021-07-31T02:17:50.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:50 smithi045 conmon[39718]: ) 1311 : cluster [DBG] pgmap v1789: 9 pgs: 9 active+clean; 127 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2021-07-31T02:17:50.809 INFO:tasks.workunit.client.0.smithi045.stdout: 749: throughput=4.99MB/sec pending data=0 2021-07-31T02:17:50.809 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-NLj1_Xbtg7fuuD4 off=2283610 len=1550338 2021-07-31T02:17:50.819 INFO:tasks.workunit.client.0.smithi045.stdout:op 3852 completed, throughput=5MB/sec 2021-07-31T02:17:50.819 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-qSjuesWgzXkjxZV off=11592886 len=843484 2021-07-31T02:17:50.824 INFO:tasks.workunit.client.0.smithi045.stdout:op 3853 completed, throughput=5MB/sec 2021-07-31T02:17:50.824 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Qkc9qxTwMNCbOQJ off=8987532 len=445088 2021-07-31T02:17:50.833 INFO:tasks.workunit.client.0.smithi045.stdout:op 3854 completed, throughput=5MB/sec 2021-07-31T02:17:50.833 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-hhk_7sDjbyrKUz- off=10968960 len=1491116 2021-07-31T02:17:50.833 INFO:tasks.workunit.client.0.smithi045.stdout:op 3855 completed, throughput=5MB/sec 2021-07-31T02:17:50.833 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-7wKXdFWhMjBy9fn off=1543456 len=1816982 2021-07-31T02:17:50.841 INFO:tasks.workunit.client.0.smithi045.stdout:op 3856 completed, throughput=5MB/sec 2021-07-31T02:17:51.249 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:17:51.526 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:17:51.526 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 32m ago 33m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:17:51.526 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 30m ago 33m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:17:51.527 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 30m ago 36m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:17:51.527 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 32m ago 38m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:17:51.527 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 32m ago 39m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:17:51.527 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 30m ago 37m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:17:51.527 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 32m ago 37m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:17:51.528 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 32m ago 33m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:17:51.528 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 30m ago 33m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:17:51.528 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 32m ago 36m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:17:51.528 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 32m ago 36m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:17:51.529 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 32m ago 35m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:17:51.529 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 32m ago 35m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:17:51.529 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 30m ago 35m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:17:51.529 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 30m ago 34m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:17:51.529 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 30m ago 34m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:17:51.530 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 30m ago 34m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:17:51.530 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 30m ago 33m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:17:51.841 INFO:tasks.workunit.client.0.smithi045.stdout: 750: throughput=4.99MB/sec pending data=0 2021-07-31T02:17:51.841 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-JRi7y6qOOMFthqa off=8389118 len=2047988 2021-07-31T02:17:51.851 INFO:tasks.workunit.client.0.smithi045.stdout:op 3857 completed, throughput=5MB/sec 2021-07-31T02:17:51.851 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-k5xOLrSVMxc-AyI off=3575905 len=1321822 2021-07-31T02:17:51.857 INFO:tasks.workunit.client.0.smithi045.stdout:op 3858 completed, throughput=5MB/sec 2021-07-31T02:17:51.857 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-JzMHroS5tUclaEU off=1302574 len=1353145 2021-07-31T02:17:51.863 INFO:tasks.workunit.client.0.smithi045.stdout:op 3859 completed, throughput=5MB/sec 2021-07-31T02:17:51.864 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:17:51.864 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:17:51.865 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:17:51.865 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:17:51.865 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:17:51.865 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:17:51.866 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:17:51.866 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:17:51.866 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:17:51.866 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:17:51.867 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:17:51.867 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:17:51.867 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:17:51.867 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:17:51.867 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:17:51.868 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:17:51.868 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:17:52.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:52 smithi154 conmon[34553]: cluster 2021-07-31T02:17:50.229614+0000 mgr.x (mgr.24542) 1312 : cluster [DBG] pgmap v1790: 9 pgs: 9 active+clean; 127 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 2.7 MiB/s rd, 3.6 MiB/s wr, 6 op/s 2021-07-31T02:17:52.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:52 smithi154 conmon[34553]: audit 2021-07-31T02:17:51.864958+0000 mon.a (mon.0) 1889 : audit [DBG] from='client.? 172.21.15.45:0/477743785' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:17:52.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:52 smithi045 conmon[33655]: cluster 2021-07-31T02:17:50.229614+0000 mgr.x 2021-07-31T02:17:52.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:52 smithi045 conmon[33655]: (mgr.24542) 1312 : cluster [DBG] pgmap v1790: 9 pgs: 9 active+clean; 127 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 2.7 MiB/s rd, 3.6 MiB/s wr, 6 op/s 2021-07-31T02:17:52.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:52 smithi045 conmon[33655]: audit 2021-07-31T02:17:51.864958+0000 mon.a (mon.0) 1889 : audit [DBG] from='client.? 172.21.15.45:0/477743785' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:17:52.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:52 smithi045 conmon[39718]: cluster 2021-07-31T02:17:50.229614 2021-07-31T02:17:52.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:52 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 1312 : cluster [DBG] pgmap v1790: 9 pgs: 9 active+clean; 127 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 2.7 MiB/s rd, 3.6 MiB/s wr, 6 op/s 2021-07-31T02:17:52.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:52 smithi045 conmon[39718]: audit 2021- 2021-07-31T02:17:52.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:52 smithi045 conmon[39718]: 07-31T02:17:51.864958+0000 mon.a (mon.0) 1889 : audit [DBG] from='client.? 172.21.15.45:0/477743785' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:17:52.863 INFO:tasks.workunit.client.0.smithi045.stdout: 751: throughput=4.99MB/sec pending data=0 2021-07-31T02:17:52.863 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-zNQi0uIR-9L1d8- off=13911177 len=583703 2021-07-31T02:17:52.865 INFO:tasks.workunit.client.0.smithi045.stdout:op 3860 completed, throughput=4.99MB/sec 2021-07-31T02:17:52.865 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Rt-QIugNUGAGoHY off=461921 len=990395 2021-07-31T02:17:52.871 INFO:tasks.workunit.client.0.smithi045.stdout:op 3861 completed, throughput=5MB/sec 2021-07-31T02:17:52.871 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-WGlNjZ21vIVF6Gp off=13610123 len=876179 2021-07-31T02:17:52.875 INFO:tasks.workunit.client.0.smithi045.stdout:op 3862 completed, throughput=5MB/sec 2021-07-31T02:17:52.876 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-6lUMbtTrancL6DD off=61526 len=1426868 2021-07-31T02:17:52.882 INFO:tasks.workunit.client.0.smithi045.stdout:op 3863 completed, throughput=5MB/sec 2021-07-31T02:17:52.882 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-podpT7WomieOghG off=2394645 len=681604 2021-07-31T02:17:52.884 INFO:tasks.workunit.client.0.smithi045.stdout:op 3864 completed, throughput=5MB/sec 2021-07-31T02:17:52.884 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-mJV-yyH4BUTm2cJ off=10658754 len=1962264 2021-07-31T02:17:52.887 INFO:tasks.workunit.client.0.smithi045.stdout:op 3865 completed, throughput=5MB/sec 2021-07-31T02:17:53.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:53 smithi154 conmon[34553]: audit 2021-07-31T02:17:51.237352+0000 mgr.x (mgr.24542) 1313 : audit [DBG] 2021-07-31T02:17:53.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:53 smithi154 conmon[34553]: from='client.16167 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:17:53.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:53 smithi154 conmon[34553]: audit 2021-07-31T02:17:51.518002+0000 mgr.x (mgr.24542) 1314 : audit [DBG] from='client.16173 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:17:53.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:53 smithi045 conmon[33655]: audit 2021-07-31T02:17:51.237352+0000 mgr.x ( 2021-07-31T02:17:53.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:53 smithi045 conmon[33655]: mgr.24542) 1313 : audit [DBG] from='client.16167 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:17:53.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:53 smithi045 conmon[33655]: audit 2021-07-31T02:17:51.518002+0000 mgr.x (mgr.24542) 1314 : audit [DBG] from='client.16173 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:17:53.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:53 smithi045 conmon[39718]: audit 2021-07-31T02:17:51.237352+0000 mgr.x ( 2021-07-31T02:17:53.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:53 smithi045 conmon[39718]: mgr.24542) 1313 : audit [DBG] from='client.16167 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:17:53.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:53 smithi045 conmon[39718]: audit 2021-07-31T02:17:51.518002+0000 mgr.x (mgr.24542) 1314 : audit [DBG] from='client.16173 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:17:53.887 INFO:tasks.workunit.client.0.smithi045.stdout: 752: throughput=5MB/sec pending data=0 2021-07-31T02:17:53.888 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-pRKKeQ3XBBtEeGl off=18779294 len=1339131 2021-07-31T02:17:53.889 INFO:tasks.workunit.client.0.smithi045.stdout:op 3866 completed, throughput=5MB/sec 2021-07-31T02:17:53.889 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-2UHS8TUORrH2rpq off=11872694 len=863693 2021-07-31T02:17:53.892 INFO:tasks.workunit.client.0.smithi045.stdout:op 3867 completed, throughput=5MB/sec 2021-07-31T02:17:53.892 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-RvEvs2cLNfZ6t7C off=13689270 len=1698343 2021-07-31T02:17:53.895 INFO:tasks.workunit.client.0.smithi045.stdout:op 3868 completed, throughput=5MB/sec 2021-07-31T02:17:54.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:17:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:17:53.561Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:17:54.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:17:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:17:53.561Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:17:54.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:17:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:17:53.561Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:17:54.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:54 smithi154 conmon[34553]: cluster 2021-07-31T02:17:52.230182+0000 mgr.x (mgr.24542) 1315 : cluster [DBG] pgmap v1791: 9 pgs: 9 active+clean; 127 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:17:54.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:54 smithi045 conmon[39718]: cluster 2021-07-31T02:17:52.230182+0000 mgr.x ( 2021-07-31T02:17:54.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:54 smithi045 conmon[39718]: mgr.24542) 1315 : cluster [DBG] pgmap v1791: 9 pgs: 9 active+clean; 127 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:17:54.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:54 smithi045 conmon[33655]: cluster 2021-07-31T02:17:52.230182+0000 mgr.x (mgr.24542) 1315 : cluster [DBG] pgmap v1791: 9 pgs: 9 active+clean; 127 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:17:54.895 INFO:tasks.workunit.client.0.smithi045.stdout: 753: throughput=4.99MB/sec pending data=0 2021-07-31T02:17:54.895 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-bGVTzB-0scwRCB2 off=4461182 len=229837 2021-07-31T02:17:54.896 INFO:tasks.workunit.client.0.smithi045.stdout:op 3869 completed, throughput=4.99MB/sec 2021-07-31T02:17:54.896 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-JmJwNfdWmbujpGD off=12383225 len=840623 2021-07-31T02:17:54.898 INFO:tasks.workunit.client.0.smithi045.stdout:op 3870 completed, throughput=4.99MB/sec 2021-07-31T02:17:54.898 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Y-qhJP50Kdh-WqA off=610504 len=1743511 2021-07-31T02:17:54.909 INFO:tasks.workunit.client.0.smithi045.stdout:op 3871 completed, throughput=5MB/sec 2021-07-31T02:17:54.909 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-L-5IoZ-R_RrxUsC off=22983657 len=824184 2021-07-31T02:17:54.914 INFO:tasks.workunit.client.0.smithi045.stdout:op 3872 completed, throughput=5MB/sec 2021-07-31T02:17:54.914 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-DD0QdmylNJlJFCk off=897524 len=962072 2021-07-31T02:17:54.915 INFO:tasks.workunit.client.0.smithi045.stdout:op 3873 completed, throughput=5MB/sec 2021-07-31T02:17:54.916 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Tk3VG1L5RLOOw7o off=1913345 len=1189064 2021-07-31T02:17:54.918 INFO:tasks.workunit.client.0.smithi045.stdout:op 3874 completed, throughput=5MB/sec 2021-07-31T02:17:55.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:55 smithi154 conmon[34553]: cluster 2021-07-31T02:17:54.230615+0000 mgr.x (mgr.24542) 1316 : cluster [DBG] pgmap v1792: 9 pgs: 9 active+clean; 127 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 2.5 MiB/s rd, 3.7 MiB/s wr, 5 op/s 2021-07-31T02:17:55.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:55 smithi045 conmon[39718]: cluster 2021-07-31T02:17:54.230615+0000 mgr.x (mgr 2021-07-31T02:17:55.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:55 smithi045 conmon[39718]: .24542) 1316 : cluster [DBG] pgmap v1792: 9 pgs: 9 active+clean; 127 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 2.5 MiB/s rd, 3.7 MiB/s wr, 5 op/s 2021-07-31T02:17:55.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:55 smithi045 conmon[33655]: cluster 2021-07-31T02:17:54.230615+0000 mgr.x (mgr. 2021-07-31T02:17:55.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:55 smithi045 conmon[33655]: 24542) 1316 : cluster [DBG] pgmap v1792: 9 pgs: 9 active+clean; 127 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 2.5 MiB/s rd, 3.7 MiB/s wr, 5 op/s 2021-07-31T02:17:55.918 INFO:tasks.workunit.client.0.smithi045.stdout: 754: throughput=4.99MB/sec pending data=0 2021-07-31T02:17:55.919 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-cHOTssXsMLNEYPU off=2869296 len=496064 2021-07-31T02:17:55.919 INFO:tasks.workunit.client.0.smithi045.stdout:op 3875 completed, throughput=4.99MB/sec 2021-07-31T02:17:55.920 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-GtlZE6b6S2KYiu0 off=2557883 len=1831087 2021-07-31T02:17:55.931 INFO:tasks.workunit.client.0.smithi045.stdout:op 3876 completed, throughput=5MB/sec 2021-07-31T02:17:55.931 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-RZMLYMEEPvAj1wi off=18075900 len=945210 2021-07-31T02:17:55.933 INFO:tasks.workunit.client.0.smithi045.stdout:op 3877 completed, throughput=5MB/sec 2021-07-31T02:17:55.934 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-qggFAlWnTS3xKnc off=9158055 len=1786170 2021-07-31T02:17:55.942 INFO:tasks.workunit.client.0.smithi045.stdout:op 3878 completed, throughput=5MB/sec 2021-07-31T02:17:56.941 INFO:tasks.workunit.client.0.smithi045.stdout: 755: throughput=4.99MB/sec pending data=0 2021-07-31T02:17:56.942 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ThMVBzl5bbv8dQ3 off=11260849 len=1456792 2021-07-31T02:17:56.948 INFO:tasks.workunit.client.0.smithi045.stdout:op 3879 completed, throughput=5MB/sec 2021-07-31T02:17:56.948 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-8okU43eeHkaFrb_ off=10739829 len=250388 2021-07-31T02:17:56.952 INFO:tasks.workunit.client.0.smithi045.stdout:op 3880 completed, throughput=5MB/sec 2021-07-31T02:17:56.952 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-dNQsXabYAVpSH9A off=2608502 len=1162001 2021-07-31T02:17:56.953 INFO:tasks.workunit.client.0.smithi045.stdout:op 3881 completed, throughput=5MB/sec 2021-07-31T02:17:56.953 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-jsMqGcAcM6_rjIC off=5213548 len=691289 2021-07-31T02:17:56.955 INFO:tasks.workunit.client.0.smithi045.stdout:op 3882 completed, throughput=5MB/sec 2021-07-31T02:17:56.955 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Hu98jXjv3LG4FMO off=136033 len=950354 2021-07-31T02:17:56.960 INFO:tasks.workunit.client.0.smithi045.stdout:op 3883 completed, throughput=5MB/sec 2021-07-31T02:17:56.960 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-7ql01x52U0LM_yS off=856588 len=1875988 2021-07-31T02:17:56.968 INFO:tasks.workunit.client.0.smithi045.stdout:op 3884 completed, throughput=5MB/sec 2021-07-31T02:17:57.968 INFO:tasks.workunit.client.0.smithi045.stdout: 756: throughput=4.99MB/sec pending data=0 2021-07-31T02:17:57.969 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-TqTIJ1H4o2BUqSQ off=4779515 len=581148 2021-07-31T02:17:57.973 INFO:tasks.workunit.client.0.smithi045.stdout:op 3885 completed, throughput=5MB/sec 2021-07-31T02:17:57.973 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-nGX5zOYhLWz7Ja1 off=391848 len=305995 2021-07-31T02:17:57.974 INFO:tasks.workunit.client.0.smithi045.stdout:op 3886 completed, throughput=5MB/sec 2021-07-31T02:17:57.974 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Mk_eSa2M926452G off=1333695 len=791751 2021-07-31T02:17:57.979 INFO:tasks.workunit.client.0.smithi045.stdout:op 3887 completed, throughput=5MB/sec 2021-07-31T02:17:57.979 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-qjyDJNidHs-RFe1 off=15072940 len=652205 2021-07-31T02:17:57.984 INFO:tasks.workunit.client.0.smithi045.stdout:op 3888 completed, throughput=5MB/sec 2021-07-31T02:17:57.984 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-VlCeyFIy-HCPCdg off=11239478 len=1645055 2021-07-31T02:17:57.992 INFO:tasks.workunit.client.0.smithi045.stdout:op 3889 completed, throughput=5MB/sec 2021-07-31T02:17:57.992 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Z6Ec_sRtNtWWUoy off=17428737 len=268509 2021-07-31T02:17:57.995 INFO:tasks.workunit.client.0.smithi045.stdout:op 3890 completed, throughput=5MB/sec 2021-07-31T02:17:58.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:17:58 smithi154 conmon[34553]: cluster 2021-07-31T02:17:56.231010+0000 mgr.x (mgr.24542) 1317 : cluster [DBG] pgmap v1793: 9 pgs: 9 active+clean; 127 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 1.4 MiB/s rd, 2.6 MiB/s wr, 3 op/s 2021-07-31T02:17:58.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:58 smithi045 conmon[39718]: cluster 2021-07-31T02:17:56.231010+0000 mgr.x 2021-07-31T02:17:58.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:17:58 smithi045 conmon[39718]: (mgr.24542) 1317 : cluster [DBG] pgmap v1793: 9 pgs: 9 active+clean; 127 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 1.4 MiB/s rd, 2.6 MiB/s wr, 3 op/s 2021-07-31T02:17:58.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:58 smithi045 conmon[33655]: cluster 2021-07-31T02:17:56.231010+0000 mgr.x (mgr.24542 2021-07-31T02:17:58.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:17:58 smithi045 conmon[33655]: ) 1317 : cluster [DBG] pgmap v1793: 9 pgs: 9 active+clean; 127 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 1.4 MiB/s rd, 2.6 MiB/s wr, 3 op/s 2021-07-31T02:17:58.996 INFO:tasks.workunit.client.0.smithi045.stdout: 757: throughput=4.99MB/sec pending data=0 2021-07-31T02:17:58.996 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-YQIdZnR7UoBanOQ off=3140182 len=113627 2021-07-31T02:17:58.999 INFO:tasks.workunit.client.0.smithi045.stdout:op 3891 completed, throughput=4.99MB/sec 2021-07-31T02:17:58.999 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-g5xacuBqqZENuly off=6846703 len=1478959 2021-07-31T02:17:59.000 INFO:tasks.workunit.client.0.smithi045.stdout:op 3892 completed, throughput=5MB/sec 2021-07-31T02:17:59.001 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-YgWx9Z_eFB_jAkO off=992363 len=1147335 2021-07-31T02:17:59.003 INFO:tasks.workunit.client.0.smithi045.stdout:op 3893 completed, throughput=5MB/sec 2021-07-31T02:17:59.003 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-nF0ZX4A1jLuNaSq off=1875644 len=1454495 2021-07-31T02:17:59.010 INFO:tasks.workunit.client.0.smithi045.stdout:op 3894 completed, throughput=5MB/sec 2021-07-31T02:17:59.010 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-T5Iis3YInTZEDrs off=2487550 len=1264053 2021-07-31T02:17:59.012 INFO:tasks.workunit.client.0.smithi045.stdout:op 3895 completed, throughput=5MB/sec 2021-07-31T02:18:00.013 INFO:tasks.workunit.client.0.smithi045.stdout: 758: throughput=4.99MB/sec pending data=0 2021-07-31T02:18:00.013 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-lfeb6BFjUHsFW7a off=17069428 len=1116532 2021-07-31T02:18:00.015 INFO:tasks.workunit.client.0.smithi045.stdout:op 3896 completed, throughput=5MB/sec 2021-07-31T02:18:00.015 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-SFioxAi1idUHAN7 off=663655 len=1258936 2021-07-31T02:18:00.017 INFO:tasks.workunit.client.0.smithi045.stdout:op 3897 completed, throughput=5MB/sec 2021-07-31T02:18:00.017 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-T_3r7lXixgjOB79 off=1494650 len=331603 2021-07-31T02:18:00.018 INFO:tasks.workunit.client.0.smithi045.stdout:op 3898 completed, throughput=5MB/sec 2021-07-31T02:18:00.018 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-TSmzO_AJdO77myT off=18618718 len=701530 2021-07-31T02:18:00.020 INFO:tasks.workunit.client.0.smithi045.stdout:op 3899 completed, throughput=5MB/sec 2021-07-31T02:18:00.020 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-aQPaAY2MbEXj19K off=2830262 len=1107926 2021-07-31T02:18:00.022 INFO:tasks.workunit.client.0.smithi045.stdout:op 3900 completed, throughput=5MB/sec 2021-07-31T02:18:00.022 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-pvN9PKLtqJBrKBK off=1445166 len=18047 2021-07-31T02:18:00.023 INFO:tasks.workunit.client.0.smithi045.stdout:op 3901 completed, throughput=5MB/sec 2021-07-31T02:18:00.023 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-3a3xcI3WlcIsnlg off=6188532 len=273868 2021-07-31T02:18:00.026 INFO:tasks.workunit.client.0.smithi045.stdout:op 3902 completed, throughput=5MB/sec 2021-07-31T02:18:00.026 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-n7AwGbScsfUCmit off=3671476 len=1920885 2021-07-31T02:18:00.029 INFO:tasks.workunit.client.0.smithi045.stdout:op 3903 completed, throughput=5MB/sec 2021-07-31T02:18:00.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:00 smithi154 conmon[34553]: cluster 2021-07-31T02:17:58.231425+0000 mgr.x (mgr.24542) 1318 : cluster [DBG] pgmap v1794: 9 pgs: 9 active+clean; 127 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 1.4 MiB/s rd, 2.6 MiB/s wr, 3 op/s 2021-07-31T02:18:00.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:00 smithi045 conmon[33655]: cluster 2021-07-31T02:17:58. 2021-07-31T02:18:00.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:00 smithi045 conmon[33655]: 231425+0000 mgr.x (mgr.24542) 1318 : cluster [DBG] pgmap v1794: 9 pgs: 9 active+clean; 127 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 1.4 MiB/s rd, 2.6 MiB/s wr, 3 op/s 2021-07-31T02:18:00.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:00 smithi045 conmon[39718]: cluster 2021-07-31T02:17:58.231425+0000 mgr.x (mgr.24542) 1318 : cluster [DBG] pgmap v1794: 9 pgs: 9 active+clean; 127 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 1.4 MiB/s rd, 2.6 MiB/s wr, 3 op/s 2021-07-31T02:18:01.029 INFO:tasks.workunit.client.0.smithi045.stdout: 759: throughput=5MB/sec pending data=0 2021-07-31T02:18:01.030 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-6eaPNLm50uTPNZ1 off=11882234 len=130754 2021-07-31T02:18:01.032 INFO:tasks.workunit.client.0.smithi045.stdout:op 3904 completed, throughput=5MB/sec 2021-07-31T02:18:01.032 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-NNjI4Cua4QleFMs off=2294253 len=1304272 2021-07-31T02:18:01.037 INFO:tasks.workunit.client.0.smithi045.stdout:op 3905 completed, throughput=5MB/sec 2021-07-31T02:18:01.037 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-vGb93BWcsjZoK0E off=14449900 len=632850 2021-07-31T02:18:01.042 INFO:tasks.workunit.client.0.smithi045.stdout:op 3906 completed, throughput=5MB/sec 2021-07-31T02:18:01.042 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-MzXmxbfu3Dku_E2 off=5043531 len=625972 2021-07-31T02:18:01.043 INFO:tasks.workunit.client.0.smithi045.stdout:op 3907 completed, throughput=5MB/sec 2021-07-31T02:18:01.044 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-2L1QVJG_lQWTNtf off=16208234 len=267391 2021-07-31T02:18:01.046 INFO:tasks.workunit.client.0.smithi045.stdout:op 3908 completed, throughput=5MB/sec 2021-07-31T02:18:01.047 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-zskR3AzTvDtwrVf off=11256867 len=837386 2021-07-31T02:18:01.051 INFO:tasks.workunit.client.0.smithi045.stdout:op 3909 completed, throughput=5MB/sec 2021-07-31T02:18:02.052 INFO:tasks.workunit.client.0.smithi045.stdout: 760: throughput=4.99MB/sec pending data=0 2021-07-31T02:18:02.053 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ns6qz_VldJ_yh4p off=3608906 len=1134293 2021-07-31T02:18:02.054 INFO:tasks.workunit.client.0.smithi045.stdout:op 3910 completed, throughput=4.99MB/sec 2021-07-31T02:18:02.055 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-kn9afUizDNUM5nM off=2374822 len=1638064 2021-07-31T02:18:02.063 INFO:tasks.workunit.client.0.smithi045.stdout:op 3911 completed, throughput=5MB/sec 2021-07-31T02:18:02.063 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-yxMivLdHAwlMGrf off=6100412 len=499307 2021-07-31T02:18:02.064 INFO:tasks.workunit.client.0.smithi045.stdout:op 3912 completed, throughput=5MB/sec 2021-07-31T02:18:02.064 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-owJhpxcHSR1eyyc off=2858470 len=1253465 2021-07-31T02:18:02.071 INFO:tasks.workunit.client.0.smithi045.stdout:op 3913 completed, throughput=5MB/sec 2021-07-31T02:18:02.071 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj--SNsmheS1EDI_nt off=6123458 len=2076808 2021-07-31T02:18:02.075 INFO:tasks.workunit.client.0.smithi045.stdout:op 3914 completed, throughput=5MB/sec 2021-07-31T02:18:02.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:02 smithi154 conmon[34553]: cluster 2021-07-31T02:18:00.232046+0000 mgr.x (mgr.24542) 1319 : cluster [DBG] 2021-07-31T02:18:02.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:02 smithi154 conmon[34553]: pgmap v1795: 9 pgs: 9 active+clean; 127 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 2.2 MiB/s rd, 3.8 MiB/s wr, 5 op/s 2021-07-31T02:18:02.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:02 smithi045 conmon[33655]: cluster 2021-07-31T02:18:00.232046+0000 mgr.x (mgr.24542) 1319 : 2021-07-31T02:18:02.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:02 smithi045 conmon[33655]: cluster [DBG] pgmap v1795: 9 pgs: 9 active+clean; 127 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 2.2 MiB/s rd, 3.8 MiB/s wr, 5 op/s 2021-07-31T02:18:02.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:02 smithi045 conmon[39718]: cluster 2021-07-31T02:18:00.232046+0000 mgr.x (mgr.24542 2021-07-31T02:18:02.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:02 smithi045 conmon[39718]: ) 1319 : cluster [DBG] pgmap v1795: 9 pgs: 9 active+clean; 127 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 2.2 MiB/s rd, 3.8 MiB/s wr, 5 op/s 2021-07-31T02:18:03.075 INFO:tasks.workunit.client.0.smithi045.stdout: 761: throughput=5MB/sec pending data=0 2021-07-31T02:18:03.076 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-wlxTUh2KULmZR-p off=10468449 len=1054494 2021-07-31T02:18:03.081 INFO:tasks.workunit.client.0.smithi045.stdout:op 3915 completed, throughput=5MB/sec 2021-07-31T02:18:03.082 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Xxo_AiEB-xU7HnR off=866368 len=361702 2021-07-31T02:18:03.084 INFO:tasks.workunit.client.0.smithi045.stdout:op 3916 completed, throughput=5MB/sec 2021-07-31T02:18:03.084 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-zYrS8vRhcT1MLTZ off=43006 len=1799333 2021-07-31T02:18:03.093 INFO:tasks.workunit.client.0.smithi045.stdout:op 3917 completed, throughput=5MB/sec 2021-07-31T02:18:03.094 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-DLxkRaYK64-H8mh off=5483355 len=1120710 2021-07-31T02:18:03.100 INFO:tasks.workunit.client.0.smithi045.stdout:op 3918 completed, throughput=5MB/sec 2021-07-31T02:18:04.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:18:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:18:03.561Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:18:04.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:18:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:18:03.561Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:18:04.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:18:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:18:03.561Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:18:04.101 INFO:tasks.workunit.client.0.smithi045.stdout: 762: throughput=4.99MB/sec pending data=0 2021-07-31T02:18:04.101 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-0kkKDaJZ4TYy0Hv off=12943954 len=1682856 2021-07-31T02:18:04.104 INFO:tasks.workunit.client.0.smithi045.stdout:op 3919 completed, throughput=5MB/sec 2021-07-31T02:18:04.104 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-4_f7mLD24BP1eM5 off=18186152 len=435863 2021-07-31T02:18:04.108 INFO:tasks.workunit.client.0.smithi045.stdout:op 3920 completed, throughput=5MB/sec 2021-07-31T02:18:04.108 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-3UgndEQe7UwANvC off=363433 len=1256501 2021-07-31T02:18:04.111 INFO:tasks.workunit.client.0.smithi045.stdout:op 3921 completed, throughput=5MB/sec 2021-07-31T02:18:04.111 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-V4TqohJWC7exfy1 off=5778367 len=1892834 2021-07-31T02:18:04.114 INFO:tasks.workunit.client.0.smithi045.stdout:op 3922 completed, throughput=5MB/sec 2021-07-31T02:18:04.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:04 smithi154 conmon[34553]: cluster 2021-07-31T02:18:02.232599+0000 mgr.x (mgr.24542) 1320 : cluster [DBG] 2021-07-31T02:18:04.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:04 smithi154 conmon[34553]: pgmap v1796: 9 pgs: 9 active+clean; 127 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 3 op/s 2021-07-31T02:18:04.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:04 smithi045 conmon[39718]: cluster 2021-07-31T02:18:02.232599+0000 mgr.x ( 2021-07-31T02:18:04.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:04 smithi045 conmon[39718]: mgr.24542) 1320 : cluster [DBG] pgmap v1796: 9 pgs: 9 active+clean; 127 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 3 op/s 2021-07-31T02:18:04.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:04 smithi045 conmon[33655]: cluster 2021-07-31T02:18:02.232599+0000 mgr.x (mgr.24542) 2021-07-31T02:18:04.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:04 smithi045 conmon[33655]: 1320 : cluster [DBG] pgmap v1796: 9 pgs: 9 active+clean; 127 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 3 op/s 2021-07-31T02:18:05.115 INFO:tasks.workunit.client.0.smithi045.stdout: 763: throughput=4.99MB/sec pending data=0 2021-07-31T02:18:05.115 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-zyRVCOJg2xR3bgb off=11299340 len=1251890 2021-07-31T02:18:05.118 INFO:tasks.workunit.client.0.smithi045.stdout:op 3923 completed, throughput=5MB/sec 2021-07-31T02:18:05.118 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-gs5Z6ZhcCakqjHO off=3843896 len=1189227 2021-07-31T02:18:05.119 INFO:tasks.workunit.client.0.smithi045.stdout:op 3924 completed, throughput=5MB/sec 2021-07-31T02:18:05.120 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-oAoXbBUZndaVYB1 off=9637267 len=1615976 2021-07-31T02:18:05.123 INFO:tasks.workunit.client.0.smithi045.stdout:op 3925 completed, throughput=5MB/sec 2021-07-31T02:18:05.123 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-JQwrfJMaM85kMwA off=7631147 len=1713544 2021-07-31T02:18:05.126 INFO:tasks.workunit.client.0.smithi045.stdout:op 3926 completed, throughput=5MB/sec 2021-07-31T02:18:06.126 INFO:tasks.workunit.client.0.smithi045.stdout: 764: throughput=4.99MB/sec pending data=0 2021-07-31T02:18:06.127 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-EEYKyh_vXFRz7tD off=8453955 len=1373387 2021-07-31T02:18:06.134 INFO:tasks.workunit.client.0.smithi045.stdout:op 3927 completed, throughput=5MB/sec 2021-07-31T02:18:06.134 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-7iS7DFMwei52PFn off=22136054 len=461324 2021-07-31T02:18:06.135 INFO:tasks.workunit.client.0.smithi045.stdout:op 3928 completed, throughput=5MB/sec 2021-07-31T02:18:06.135 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ni-sydQwUTzxxww off=1546698 len=686292 2021-07-31T02:18:06.137 INFO:tasks.workunit.client.0.smithi045.stdout:op 3929 completed, throughput=5MB/sec 2021-07-31T02:18:06.137 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-aDUkonTXmCDeBwE off=5340287 len=758403 2021-07-31T02:18:06.142 INFO:tasks.workunit.client.0.smithi045.stdout:op 3930 completed, throughput=5MB/sec 2021-07-31T02:18:06.142 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-8dt3Z8p8UL_OkwO off=692146 len=1154511 2021-07-31T02:18:06.149 INFO:tasks.workunit.client.0.smithi045.stdout:op 3931 completed, throughput=5MB/sec 2021-07-31T02:18:06.149 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-FhTtv1l1Dcb9avn off=407827 len=1968166 2021-07-31T02:18:06.151 INFO:tasks.workunit.client.0.smithi045.stdout:op 3932 completed, throughput=5MB/sec 2021-07-31T02:18:06.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:06 smithi154 conmon[34553]: cluster 2021-07-31T02:18:04.233195+0000 mgr.x (mgr.24542) 1321 : cluster [DBG] pgmap v1797: 9 pgs: 9 active+clean; 127 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 2.5 MiB/s rd, 3.5 MiB/s wr, 5 op/s 2021-07-31T02:18:06.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:06 smithi045 conmon[39718]: cluster 2021-07-31T02:18:04.233195+0000 mgr.x (mgr. 2021-07-31T02:18:06.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:06 smithi045 conmon[39718]: 24542) 1321 : cluster [DBG] pgmap v1797: 9 pgs: 9 active+clean; 127 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 2.5 MiB/s rd, 3.5 MiB/s wr, 5 op/s 2021-07-31T02:18:06.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:06 smithi045 conmon[33655]: cluster 2021-07-31T02:18:04.233195+0000 mgr.x (mgr.24542) 1321 : cluster [DBG] pgmap v1797: 9 pgs: 9 active+clean; 127 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 2.5 MiB/s rd, 3.5 MiB/s wr, 5 op/s 2021-07-31T02:18:06.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:06 smithi045 conmon[33655]: 2021-07-31T02:18:07.152 INFO:tasks.workunit.client.0.smithi045.stdout: 765: throughput=5MB/sec pending data=0 2021-07-31T02:18:07.152 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-PIfQ82uN0ggIrJa off=5730379 len=379208 2021-07-31T02:18:07.155 INFO:tasks.workunit.client.0.smithi045.stdout:op 3933 completed, throughput=5MB/sec 2021-07-31T02:18:07.156 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-kurNzp0UWpObcdb off=1100363 len=1283271 2021-07-31T02:18:07.162 INFO:tasks.workunit.client.0.smithi045.stdout:op 3934 completed, throughput=5MB/sec 2021-07-31T02:18:07.163 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-lA3OODMbiahqJMd off=3325274 len=626744 2021-07-31T02:18:07.164 INFO:tasks.workunit.client.0.smithi045.stdout:op 3935 completed, throughput=5MB/sec 2021-07-31T02:18:07.164 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-9eUT9y-QYVY6F05 off=18701474 len=150002 2021-07-31T02:18:07.165 INFO:tasks.workunit.client.0.smithi045.stdout:op 3936 completed, throughput=5MB/sec 2021-07-31T02:18:07.165 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-JkjuPnTXLW531kV off=20972783 len=533684 2021-07-31T02:18:07.167 INFO:tasks.workunit.client.0.smithi045.stdout:op 3937 completed, throughput=5MB/sec 2021-07-31T02:18:07.167 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ilII2E07xCdpyBi off=4511997 len=68369 2021-07-31T02:18:07.169 INFO:tasks.workunit.client.0.smithi045.stdout:op 3938 completed, throughput=5MB/sec 2021-07-31T02:18:07.170 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-2Z6TFRriSmGvQUj off=1422189 len=1592587 2021-07-31T02:18:07.179 INFO:tasks.workunit.client.0.smithi045.stdout:op 3939 completed, throughput=5MB/sec 2021-07-31T02:18:08.179 INFO:tasks.workunit.client.0.smithi045.stdout: 766: throughput=4.99MB/sec pending data=0 2021-07-31T02:18:08.180 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Z6YG0f4ZiQpOvUB off=4648881 len=724055 2021-07-31T02:18:08.184 INFO:tasks.workunit.client.0.smithi045.stdout:op 3940 completed, throughput=5MB/sec 2021-07-31T02:18:08.184 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-0Rb0Jj5kBl7NwVb off=2019829 len=1173732 2021-07-31T02:18:08.190 INFO:tasks.workunit.client.0.smithi045.stdout:op 3941 completed, throughput=5MB/sec 2021-07-31T02:18:08.191 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-mRNBRZZCYowi1-W off=427584 len=727470 2021-07-31T02:18:08.195 INFO:tasks.workunit.client.0.smithi045.stdout:op 3942 completed, throughput=5MB/sec 2021-07-31T02:18:08.195 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-25CKU_OUPiBll-H off=8892416 len=558597 2021-07-31T02:18:08.196 INFO:tasks.workunit.client.0.smithi045.stdout:op 3943 completed, throughput=5MB/sec 2021-07-31T02:18:08.197 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-DjR96ZZrblDIiNP off=8806839 len=898251 2021-07-31T02:18:08.201 INFO:tasks.workunit.client.0.smithi045.stdout:op 3944 completed, throughput=5MB/sec 2021-07-31T02:18:08.201 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-p_l3mwRkZUuFU0L off=933261 len=49095 2021-07-31T02:18:08.202 INFO:tasks.workunit.client.0.smithi045.stdout:op 3945 completed, throughput=5MB/sec 2021-07-31T02:18:08.203 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-s202OmuUknJh65O off=3981183 len=601264 2021-07-31T02:18:08.206 INFO:tasks.workunit.client.0.smithi045.stdout:op 3946 completed, throughput=5MB/sec 2021-07-31T02:18:08.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:08 smithi154 conmon[34553]: cluster 2021-07-31T02:18:06.233596+0000 mgr.x (mgr.24542) 1322 : cluster [DBG] pgmap v1798: 9 pgs: 9 active+clean; 127 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2021-07-31T02:18:08.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:08 smithi154 conmon[34553]: 2021-07-31T02:18:08.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:08 smithi045 conmon[33655]: cluster 2021-07-31T02:18:06.233596+0000 mgr.x ( 2021-07-31T02:18:08.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:08 smithi045 conmon[33655]: mgr.24542) 1322 : cluster [DBG] pgmap v1798: 9 pgs: 9 active+clean; 127 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2021-07-31T02:18:08.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:08 smithi045 conmon[39718]: cluster 2021-07-31T02:18:06.233596+0000 mgr.x (mgr. 2021-07-31T02:18:08.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:08 smithi045 conmon[39718]: 24542) 1322 : cluster [DBG] pgmap v1798: 9 pgs: 9 active+clean; 127 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2021-07-31T02:18:09.206 INFO:tasks.workunit.client.0.smithi045.stdout: 767: throughput=4.99MB/sec pending data=0 2021-07-31T02:18:09.207 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Uyq6_h1LGFbj1QU off=12439715 len=1232037 2021-07-31T02:18:09.209 INFO:tasks.workunit.client.0.smithi045.stdout:op 3947 completed, throughput=5MB/sec 2021-07-31T02:18:09.209 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Ekl0hb8BhjPi50J off=1442881 len=1176233 2021-07-31T02:18:09.210 INFO:tasks.workunit.client.0.smithi045.stdout:op 3948 completed, throughput=5MB/sec 2021-07-31T02:18:09.211 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Y_QzKuvk3amgPdm off=6262523 len=813617 2021-07-31T02:18:09.212 INFO:tasks.workunit.client.0.smithi045.stdout:op 3949 completed, throughput=5MB/sec 2021-07-31T02:18:09.212 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-_qZ-EUeZ6q3Cb8p off=12284093 len=1794181 2021-07-31T02:18:09.220 INFO:tasks.workunit.client.0.smithi045.stdout:op 3950 completed, throughput=5MB/sec 2021-07-31T02:18:09.220 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-qz30Mp6pBEgdY3K off=678893 len=40132 2021-07-31T02:18:09.223 INFO:tasks.workunit.client.0.smithi045.stdout:op 3951 completed, throughput=5MB/sec 2021-07-31T02:18:09.223 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-6lAfOW1uIiQX2Th off=5593271 len=436573 2021-07-31T02:18:09.226 INFO:tasks.workunit.client.0.smithi045.stdout:op 3952 completed, throughput=5MB/sec 2021-07-31T02:18:10.227 INFO:tasks.workunit.client.0.smithi045.stdout: 768: throughput=4.99MB/sec pending data=0 2021-07-31T02:18:10.227 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-A--oa_Zvg-maSkb off=2025363 len=1020235 2021-07-31T02:18:10.228 INFO:tasks.workunit.client.0.smithi045.stdout:op 3953 completed, throughput=5MB/sec 2021-07-31T02:18:10.229 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-a1UDI-3_3DPg6-K off=2197374 len=1601826 2021-07-31T02:18:10.236 INFO:tasks.workunit.client.0.smithi045.stdout:op 3954 completed, throughput=5MB/sec 2021-07-31T02:18:10.237 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-zO2yUV29EtkSTOS off=3951014 len=1849069 2021-07-31T02:18:10.240 INFO:tasks.workunit.client.0.smithi045.stdout:op 3955 completed, throughput=5MB/sec 2021-07-31T02:18:10.240 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-xwnFUxjcgReb6rA off=2194807 len=1293666 2021-07-31T02:18:10.246 INFO:tasks.workunit.client.0.smithi045.stdout:op 3956 completed, throughput=5MB/sec 2021-07-31T02:18:10.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:10 smithi154 conmon[34553]: cluster 2021-07-31T02:18:08.233978+0000 mgr.x (mgr.24542) 1323 : cluster [DBG] pgmap v1799: 9 pgs: 9 active+clean; 127 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2021-07-31T02:18:10.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:10 smithi154 conmon[34553]: 2021-07-31T02:18:10.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:10 smithi045 conmon[39718]: cluster 2021-07-31T02:18:08.233978+0000 mgr.x (mgr.24542) 1323 : cluster [DBG] pgmap v1799: 9 pgs: 9 active+clean; 127 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2021-07-31T02:18:10.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:10 smithi045 conmon[33655]: cluster 2021-07-31T02:18:08.233978+0000 mgr.x (mgr.24542) 2021-07-31T02:18:10.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:10 smithi045 conmon[33655]: 1323 : cluster [DBG] pgmap v1799: 9 pgs: 9 active+clean; 127 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2021-07-31T02:18:11.247 INFO:tasks.workunit.client.0.smithi045.stdout: 769: throughput=4.99MB/sec pending data=0 2021-07-31T02:18:11.247 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Mli3Jc0dXcMvuvv off=4609949 len=1786776 2021-07-31T02:18:11.250 INFO:tasks.workunit.client.0.smithi045.stdout:op 3957 completed, throughput=5MB/sec 2021-07-31T02:18:11.250 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-qFxXtboiFKs0Kjb off=5682621 len=1534614 2021-07-31T02:18:11.253 INFO:tasks.workunit.client.0.smithi045.stdout:op 3958 completed, throughput=5MB/sec 2021-07-31T02:18:11.254 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-u9OurSbzq0oIOTl off=662924 len=137045 2021-07-31T02:18:11.256 INFO:tasks.workunit.client.0.smithi045.stdout:op 3959 completed, throughput=5MB/sec 2021-07-31T02:18:11.256 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-4EOmFKSpU5KdgQ6 off=5018683 len=154277 2021-07-31T02:18:11.257 INFO:tasks.workunit.client.0.smithi045.stdout:op 3960 completed, throughput=5MB/sec 2021-07-31T02:18:11.257 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-klBJ8ZtuhzkMc43 off=474505 len=152932 2021-07-31T02:18:11.258 INFO:tasks.workunit.client.0.smithi045.stdout:op 3961 completed, throughput=5MB/sec 2021-07-31T02:18:11.258 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-qMPErM-gdBxUCsh off=2976855 len=1667610 2021-07-31T02:18:11.266 INFO:tasks.workunit.client.0.smithi045.stdout:op 3962 completed, throughput=5MB/sec 2021-07-31T02:18:12.266 INFO:tasks.workunit.client.0.smithi045.stdout: 770: throughput=4.99MB/sec pending data=0 2021-07-31T02:18:12.267 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-EXqZVECPolrVMO8 off=1058927 len=72403 2021-07-31T02:18:12.269 INFO:tasks.workunit.client.0.smithi045.stdout:op 3963 completed, throughput=4.99MB/sec 2021-07-31T02:18:12.270 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-RWIo90_U7AISvD_ off=13849165 len=1702576 2021-07-31T02:18:12.272 INFO:tasks.workunit.client.0.smithi045.stdout:op 3964 completed, throughput=5MB/sec 2021-07-31T02:18:12.272 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Ibm4gBJaGNikh-a off=152568 len=1926602 2021-07-31T02:18:12.284 INFO:tasks.workunit.client.0.smithi045.stdout:op 3965 completed, throughput=5MB/sec 2021-07-31T02:18:12.284 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-_X-NJom1M8Pmbq4 off=3736718 len=1347027 2021-07-31T02:18:12.292 INFO:tasks.workunit.client.0.smithi045.stdout:op 3966 completed, throughput=5MB/sec 2021-07-31T02:18:12.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:12 smithi154 conmon[34553]: cluster 2021-07-31T02:18:10.234417+0000 mgr.x (mgr.24542) 1324 : cluster [DBG] 2021-07-31T02:18:12.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:12 smithi154 conmon[34553]: pgmap v1800: 9 pgs: 9 active+clean; 127 GiB data, 8.7 GiB used, 706 GiB / 715 GiB avail; 3.1 MiB/s rd, 3.0 MiB/s wr, 6 op/s 2021-07-31T02:18:12.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:12 smithi045 conmon[39718]: cluster 2021-07-31T02:18:10.234417+0000 mgr.x (mgr 2021-07-31T02:18:12.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:12 smithi045 conmon[39718]: .24542) 1324 : cluster [DBG] pgmap v1800: 9 pgs: 9 active+clean; 127 GiB data, 8.7 GiB used, 706 GiB / 715 GiB avail; 3.1 MiB/s rd, 3.0 MiB/s wr, 6 op/s 2021-07-31T02:18:12.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:12 smithi045 conmon[33655]: cluster 2021-07-31T02:18:10.234417+0000 mgr.x 2021-07-31T02:18:12.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:12 smithi045 conmon[33655]: (mgr.24542) 1324 : cluster [DBG] pgmap v1800: 9 pgs: 9 active+clean; 127 GiB data, 8.7 GiB used, 706 GiB / 715 GiB avail; 3.1 MiB/s rd, 3.0 MiB/s wr, 6 op/s 2021-07-31T02:18:13.292 INFO:tasks.workunit.client.0.smithi045.stdout: 771: throughput=4.99MB/sec pending data=0 2021-07-31T02:18:13.293 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-8xMOPQ1RKuYMqYw off=1217585 len=1799699 2021-07-31T02:18:13.302 INFO:tasks.workunit.client.0.smithi045.stdout:op 3967 completed, throughput=5MB/sec 2021-07-31T02:18:13.302 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-nyM8MF7EsPw0dmX off=10285750 len=653316 2021-07-31T02:18:13.303 INFO:tasks.workunit.client.0.smithi045.stdout:op 3968 completed, throughput=5MB/sec 2021-07-31T02:18:13.304 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-e6q6Mq8ygv25hA- off=298437 len=1010872 2021-07-31T02:18:13.305 INFO:tasks.workunit.client.0.smithi045.stdout:op 3969 completed, throughput=5MB/sec 2021-07-31T02:18:13.306 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-fRM_PR8xj0pImq5 off=10790733 len=66605 2021-07-31T02:18:13.306 INFO:tasks.workunit.client.0.smithi045.stdout:op 3970 completed, throughput=5MB/sec 2021-07-31T02:18:13.306 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-UrCJ1S4_NCNHCm7 off=14828296 len=1013677 2021-07-31T02:18:13.311 INFO:tasks.workunit.client.0.smithi045.stdout:op 3971 completed, throughput=5MB/sec 2021-07-31T02:18:13.312 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-HV3PKiqySxNaDwu off=761282 len=1717609 2021-07-31T02:18:13.319 INFO:tasks.workunit.client.0.smithi045.stdout:op 3972 completed, throughput=5MB/sec 2021-07-31T02:18:14.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:18:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:18:13.562Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:18:14.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:18:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:18:13.562Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:18:14.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:18:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:18:13.562Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:18:14.319 INFO:tasks.workunit.client.0.smithi045.stdout: 772: throughput=5MB/sec pending data=0 2021-07-31T02:18:14.320 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Ej76xB-uLFaVOYo off=7389498 len=81992 2021-07-31T02:18:14.322 INFO:tasks.workunit.client.0.smithi045.stdout:op 3973 completed, throughput=5MB/sec 2021-07-31T02:18:14.322 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-KSQD2u1b5-he6QZ off=892633 len=140480 2021-07-31T02:18:14.325 INFO:tasks.workunit.client.0.smithi045.stdout:op 3974 completed, throughput=5MB/sec 2021-07-31T02:18:14.325 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-bpoH2yp1NBlKkAO off=1484923 len=1110483 2021-07-31T02:18:14.327 INFO:tasks.workunit.client.0.smithi045.stdout:op 3975 completed, throughput=5MB/sec 2021-07-31T02:18:14.327 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-gYfRWzDzA8BjpYp off=7955916 len=1413833 2021-07-31T02:18:14.334 INFO:tasks.workunit.client.0.smithi045.stdout:op 3976 completed, throughput=5MB/sec 2021-07-31T02:18:14.334 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-QY22276hXRFQcGz off=364677 len=1975649 2021-07-31T02:18:14.343 INFO:tasks.workunit.client.0.smithi045.stdout:op 3977 completed, throughput=5MB/sec 2021-07-31T02:18:14.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:14 smithi154 conmon[34553]: cluster 2021-07-31T02:18:12.234962+0000 mgr.x (mgr.24542) 2021-07-31T02:18:14.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:14 smithi154 conmon[34553]: 1325 : cluster [DBG] pgmap v1801: 9 pgs: 9 active+clean; 127 GiB data, 8.7 GiB used, 706 GiB / 715 GiB avail; 2.4 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2021-07-31T02:18:14.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:14 smithi045 conmon[39718]: cluster 2021-07-31T02:18:12.234962+0000 2021-07-31T02:18:14.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:14 smithi045 conmon[39718]: mgr.x (mgr.24542) 1325 : cluster [DBG] pgmap v1801: 9 pgs: 9 active+clean; 127 GiB data, 8.7 GiB used, 706 GiB / 715 GiB avail; 2.4 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2021-07-31T02:18:14.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:14 smithi045 conmon[33655]: cluster 2021-07-31T02:18:12.234962+0000 2021-07-31T02:18:14.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:14 smithi045 conmon[33655]: mgr.x (mgr.24542) 1325 : cluster [DBG] pgmap v1801: 9 pgs: 9 active+clean; 127 GiB data, 8.7 GiB used, 706 GiB / 715 GiB avail; 2.4 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2021-07-31T02:18:15.344 INFO:tasks.workunit.client.0.smithi045.stdout: 773: throughput=4.99MB/sec pending data=0 2021-07-31T02:18:15.344 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Xc9Yb8AIyfdU7Dc off=2511437 len=1285137 2021-07-31T02:18:15.352 INFO:tasks.workunit.client.0.smithi045.stdout:op 3978 completed, throughput=5MB/sec 2021-07-31T02:18:15.352 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-gaofJgxNRIOI8mg off=15452259 len=980629 2021-07-31T02:18:15.353 INFO:tasks.workunit.client.0.smithi045.stdout:op 3979 completed, throughput=5MB/sec 2021-07-31T02:18:15.354 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-_350TkZiBlBkRGF off=1557840 len=1919207 2021-07-31T02:18:15.361 INFO:tasks.workunit.client.0.smithi045.stdout:op 3980 completed, throughput=5MB/sec 2021-07-31T02:18:15.362 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-HJ2U2PcJlfn1_Tf off=23366613 len=1903394 2021-07-31T02:18:15.369 INFO:tasks.workunit.client.0.smithi045.stdout:op 3981 completed, throughput=5MB/sec 2021-07-31T02:18:16.369 INFO:tasks.workunit.client.0.smithi045.stdout: 774: throughput=5MB/sec pending data=0 2021-07-31T02:18:16.370 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-XiMNv6h41io275M off=220206 len=536345 2021-07-31T02:18:16.370 INFO:tasks.workunit.client.0.smithi045.stdout:op 3982 completed, throughput=5MB/sec 2021-07-31T02:18:16.370 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-4v8lcTA-VCvSlt8 off=5246635 len=2086175 2021-07-31T02:18:16.373 INFO:tasks.workunit.client.0.smithi045.stdout:op 3983 completed, throughput=5MB/sec 2021-07-31T02:18:16.373 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-vJP_wWHFPt7itCo off=0 len=1936581 2021-07-31T02:18:16.374 INFO:tasks.workunit.client.0.smithi045.stdout:op 3984 completed, throughput=5MB/sec 2021-07-31T02:18:16.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:16 smithi154 conmon[34553]: cluster 2021-07-31T02:18:14.235458+0000 mgr.x (mgr.24542) 1326 : cluster [DBG] pgmap v1802: 9 pgs: 9 active+clean; 127 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 3.5 MiB/s rd, 3.0 MiB/s wr, 6 op/s 2021-07-31T02:18:16.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:16 smithi045 conmon[33655]: cluster 2021-07-31T02:18:14.235458+0000 2021-07-31T02:18:16.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:16 smithi045 conmon[33655]: mgr.x (mgr.24542) 1326 : cluster [DBG] pgmap v1802: 9 pgs: 9 active+clean; 127 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 3.5 MiB/s rd, 3.0 MiB/s wr, 6 op/s 2021-07-31T02:18:16.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:16 smithi045 conmon[39718]: cluster 2021-07-31T02:18:14. 2021-07-31T02:18:16.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:16 smithi045 conmon[39718]: 235458+0000 mgr.x (mgr.24542) 1326 : cluster [DBG] pgmap v1802: 9 pgs: 9 active+clean; 127 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 3.5 MiB/s rd, 3.0 MiB/s wr, 6 op/s 2021-07-31T02:18:17.375 INFO:tasks.workunit.client.0.smithi045.stdout: 775: throughput=4.99MB/sec pending data=0 2021-07-31T02:18:17.375 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-zB2bxEZRkZGGzeA off=373935 len=77666 2021-07-31T02:18:17.377 INFO:tasks.workunit.client.0.smithi045.stdout:op 3985 completed, throughput=4.99MB/sec 2021-07-31T02:18:17.378 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-1V6B6NdG9qOXGBr off=14851677 len=1231192 2021-07-31T02:18:17.387 INFO:tasks.workunit.client.0.smithi045.stdout:op 3986 completed, throughput=5MB/sec 2021-07-31T02:18:17.387 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-aHosV8cKSlk91Hq off=0 len=1092338 2021-07-31T02:18:17.394 INFO:tasks.workunit.client.0.smithi045.stdout:op 3987 completed, throughput=5MB/sec 2021-07-31T02:18:17.395 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ZHuzXvN05kYj1XJ off=4746189 len=1607092 2021-07-31T02:18:17.400 INFO:tasks.workunit.client.0.smithi045.stdout:op 3988 completed, throughput=5MB/sec 2021-07-31T02:18:17.402 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-5nSEqfB7JO2I9HE off=19514319 len=1067353 2021-07-31T02:18:17.402 INFO:tasks.workunit.client.0.smithi045.stdout:op 3989 completed, throughput=5MB/sec 2021-07-31T02:18:18.403 INFO:tasks.workunit.client.0.smithi045.stdout: 776: throughput=4.99MB/sec pending data=0 2021-07-31T02:18:18.403 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-e-gQgin17IrTlzX off=9521456 len=1504810 2021-07-31T02:18:18.412 INFO:tasks.workunit.client.0.smithi045.stdout:op 3990 completed, throughput=5MB/sec 2021-07-31T02:18:18.413 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Cd9A0VdQhnyOHwe off=10122005 len=523440 2021-07-31T02:18:18.416 INFO:tasks.workunit.client.0.smithi045.stdout:op 3991 completed, throughput=5MB/sec 2021-07-31T02:18:18.416 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ENzYzdSVQOHEPOX off=9660137 len=1398374 2021-07-31T02:18:18.423 INFO:tasks.workunit.client.0.smithi045.stdout:op 3992 completed, throughput=5MB/sec 2021-07-31T02:18:18.423 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-LsBFTAy39PYA_8d off=4332729 len=69886 2021-07-31T02:18:18.426 INFO:tasks.workunit.client.0.smithi045.stdout:op 3993 completed, throughput=5MB/sec 2021-07-31T02:18:18.426 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Avb2NfGJP0V6skb off=1774284 len=1263514 2021-07-31T02:18:18.429 INFO:tasks.workunit.client.0.smithi045.stdout:op 3994 completed, throughput=5MB/sec 2021-07-31T02:18:18.429 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-qH-NRRI06krbDQO off=297100 len=866865 2021-07-31T02:18:18.431 INFO:tasks.workunit.client.0.smithi045.stdout:op 3995 completed, throughput=5MB/sec 2021-07-31T02:18:18.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:18 smithi154 conmon[34553]: cluster 2021-07-31T02:18:16.235999+0000 mgr.x (mgr.24542) 1327 : cluster [DBG] pgmap v1803: 9 pgs: 9 active+clean; 127 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2021-07-31T02:18:18.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:18 smithi045 conmon[33655]: cluster 2021-07-31T02:18:16.235999+0000 mgr.x (mgr.24542) 1327 : cluster [DBG] pgmap v1803: 9 pgs: 9 active+clean; 127 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2021-07-31T02:18:18.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:18 smithi045 conmon[33655]: 2021-07-31T02:18:18.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:18 smithi045 conmon[39718]: cluster 2021-07-31T02:18:16.235999+0000 2021-07-31T02:18:18.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:18 smithi045 conmon[39718]: mgr.x (mgr.24542) 1327 : cluster [DBG] pgmap v1803: 9 pgs: 9 active+clean; 127 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2021-07-31T02:18:19.431 INFO:tasks.workunit.client.0.smithi045.stdout: 777: throughput=4.99MB/sec pending data=0 2021-07-31T02:18:19.432 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-tqd_LNWM1-jVTff off=273277 len=1060769 2021-07-31T02:18:19.438 INFO:tasks.workunit.client.0.smithi045.stdout:op 3996 completed, throughput=5MB/sec 2021-07-31T02:18:19.438 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-6POC3Nh7Lu594L7 off=3095670 len=1607985 2021-07-31T02:18:19.445 INFO:tasks.workunit.client.0.smithi045.stdout:op 3997 completed, throughput=5MB/sec 2021-07-31T02:18:19.446 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-NHwvfOKmsHfH05h off=136149 len=1691350 2021-07-31T02:18:19.455 INFO:tasks.workunit.client.0.smithi045.stdout:op 3998 completed, throughput=5MB/sec 2021-07-31T02:18:19.455 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-TQ9XufmQDrHPjcb off=5691765 len=1204423 2021-07-31T02:18:19.460 INFO:tasks.workunit.client.0.smithi045.stdout:op 3999 completed, throughput=5MB/sec 2021-07-31T02:18:20.461 INFO:tasks.workunit.client.0.smithi045.stdout: 778: throughput=4.99MB/sec pending data=0 2021-07-31T02:18:20.461 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ub5Dtl_n_T1Op4R off=1888143 len=1338717 2021-07-31T02:18:20.464 INFO:tasks.workunit.client.0.smithi045.stdout:op 4000 completed, throughput=5MB/sec 2021-07-31T02:18:20.464 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-tjCoVv-Zkz_w7s0 off=1162442 len=1408873 2021-07-31T02:18:20.471 INFO:tasks.workunit.client.0.smithi045.stdout:op 4001 completed, throughput=5MB/sec 2021-07-31T02:18:20.471 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-McrZlyJ-EkgYnVY off=2074365 len=1385354 2021-07-31T02:18:20.474 INFO:tasks.workunit.client.0.smithi045.stdout:op 4002 completed, throughput=5MB/sec 2021-07-31T02:18:20.474 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-c3BbOFHTlKNkY9b off=5478837 len=754006 2021-07-31T02:18:20.478 INFO:tasks.workunit.client.0.smithi045.stdout:op 4003 completed, throughput=5MB/sec 2021-07-31T02:18:20.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:20 smithi154 conmon[34553]: cluster 2021-07-31T02:18:18.236497+0000 mgr.x (mgr.24542) 1328 : cluster [DBG] pgmap v1804: 9 pgs: 9 active+clean; 127 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2021-07-31T02:18:20.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:20 smithi045 conmon[39718]: cluster 2021-07-31T02:18:18.236497+0000 mgr.x (mgr.24542 2021-07-31T02:18:20.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:20 smithi045 conmon[39718]: ) 1328 : cluster [DBG] pgmap v1804: 9 pgs: 9 active+clean; 127 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2021-07-31T02:18:20.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:20 smithi045 conmon[33655]: cluster 2021-07-31T02:18:18.236497+0000 mgr.x (mgr.24542) 1328 2021-07-31T02:18:20.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:20 smithi045 conmon[33655]: : cluster [DBG] pgmap v1804: 9 pgs: 9 active+clean; 127 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2021-07-31T02:18:21.479 INFO:tasks.workunit.client.0.smithi045.stdout: 779: throughput=4.99MB/sec pending data=0 2021-07-31T02:18:21.479 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-yK9NECZFLIuL3Sc off=670383 len=477919 2021-07-31T02:18:21.480 INFO:tasks.workunit.client.0.smithi045.stdout:op 4004 completed, throughput=4.99MB/sec 2021-07-31T02:18:21.480 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-zgQHJ8Gl6g5oiMH off=5009603 len=390688 2021-07-31T02:18:21.482 INFO:tasks.workunit.client.0.smithi045.stdout:op 4005 completed, throughput=5MB/sec 2021-07-31T02:18:21.482 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ByNxVIFM2y2guNb off=4546178 len=1962826 2021-07-31T02:18:21.495 INFO:tasks.workunit.client.0.smithi045.stdout:op 4006 completed, throughput=5MB/sec 2021-07-31T02:18:21.495 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Xt5zmoccWnrSKcF off=3502133 len=446503 2021-07-31T02:18:21.496 INFO:tasks.workunit.client.0.smithi045.stdout:op 4007 completed, throughput=5MB/sec 2021-07-31T02:18:21.496 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-wzsqSXDXEZDfP0q off=1092462 len=11621 2021-07-31T02:18:21.498 INFO:tasks.workunit.client.0.smithi045.stdout:op 4008 completed, throughput=5MB/sec 2021-07-31T02:18:21.498 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-f-v8QC6-47991rE off=8459038 len=1291164 2021-07-31T02:18:21.500 INFO:tasks.workunit.client.0.smithi045.stdout:op 4009 completed, throughput=5MB/sec 2021-07-31T02:18:21.501 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-r5NXg2PGtDxHWZN off=2744972 len=497634 2021-07-31T02:18:21.505 INFO:tasks.workunit.client.0.smithi045.stdout:op 4010 completed, throughput=5MB/sec 2021-07-31T02:18:22.179 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:18:22.455 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:18:22.456 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 33m ago 34m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:18:22.456 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 30m ago 33m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:18:22.456 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 30m ago 37m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:18:22.456 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 33m ago 39m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:18:22.457 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 33m ago 39m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:18:22.457 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 30m ago 37m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:18:22.457 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 33m ago 37m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:18:22.457 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 33m ago 34m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:18:22.457 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 30m ago 34m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:18:22.458 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 33m ago 36m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:18:22.458 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 33m ago 36m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:18:22.458 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 33m ago 36m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:18:22.458 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 33m ago 35m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:18:22.459 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 30m ago 35m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:18:22.459 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 30m ago 35m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:18:22.459 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 30m ago 35m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:18:22.459 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 30m ago 34m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:18:22.459 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 30m ago 34m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:18:22.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:22 smithi154 conmon[34553]: cluster 2021-07-31T02:18:20.237042+0000 mgr.x (mgr.24542) 1329 : cluster [DBG] pgmap v1805: 9 pgs: 9 active+clean; 127 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 3.1 MiB/s rd, 3.4 MiB/s wr, 6 op/s 2021-07-31T02:18:22.505 INFO:tasks.workunit.client.0.smithi045.stdout: 780: throughput=4.99MB/sec pending data=0 2021-07-31T02:18:22.505 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-zJAeqdqFzufIX75 off=3371420 len=1036856 2021-07-31T02:18:22.508 INFO:tasks.workunit.client.0.smithi045.stdout:op 4011 completed, throughput=5MB/sec 2021-07-31T02:18:22.509 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-cJDkURvisfe4FbB off=0 len=1492813 2021-07-31T02:18:22.514 INFO:tasks.workunit.client.0.smithi045.stdout:op 4012 completed, throughput=5MB/sec 2021-07-31T02:18:22.514 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-IbYPcOrkwsI-RkL off=998243 len=1608054 2021-07-31T02:18:22.515 INFO:tasks.workunit.client.0.smithi045.stdout:op 4013 completed, throughput=5MB/sec 2021-07-31T02:18:22.516 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-zHq7U7i_o2etogh off=10709223 len=2019683 2021-07-31T02:18:22.517 INFO:tasks.workunit.client.0.smithi045.stdout:op 4014 completed, throughput=5MB/sec 2021-07-31T02:18:22.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:22 smithi045 conmon[33655]: cluster 2021-07-31T02:18:20.237042+0000 mgr.x (mgr 2021-07-31T02:18:22.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:22 smithi045 conmon[33655]: .24542) 1329 : cluster [DBG] pgmap v1805: 9 pgs: 9 active+clean; 127 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 3.1 MiB/s rd, 3.4 MiB/s wr, 6 op/s 2021-07-31T02:18:22.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:22 smithi045 conmon[39718]: cluster 2021-07-31T02:18:20.237042 2021-07-31T02:18:22.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:22 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 1329 : cluster [DBG] pgmap v1805: 9 pgs: 9 active+clean; 127 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 3.1 MiB/s rd, 3.4 MiB/s wr, 6 op/s 2021-07-31T02:18:22.797 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:18:22.797 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:18:22.797 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:18:22.797 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:18:22.798 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:18:22.798 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:18:22.798 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:18:22.798 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:18:22.799 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:18:22.799 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:18:22.799 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:18:22.799 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:18:22.800 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:18:22.800 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:18:22.800 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:18:22.800 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:18:22.800 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:18:23.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:23 smithi154 conmon[34553]: audit 2021-07-31T02:18:22.167140+0000 mgr.x ( 2021-07-31T02:18:23.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:23 smithi154 conmon[34553]: mgr.24542) 1330 : audit [DBG] from='client.26092 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:18:23.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:23 smithi154 conmon[34553]: audit 2021-07-31T02:18:22.797138+0000 mon.a (mon.0) 1890 : audit [DBG] from='client.? 172.21.15.45:0/3162991093' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:18:23.517 INFO:tasks.workunit.client.0.smithi045.stdout: 781: throughput=4.99MB/sec pending data=0 2021-07-31T02:18:23.518 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-MH1HGfXCRf6m0IQ off=2077789 len=493181 2021-07-31T02:18:23.519 INFO:tasks.workunit.client.0.smithi045.stdout:op 4015 completed, throughput=5MB/sec 2021-07-31T02:18:23.520 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-rOb9wzen_B8s7C0 off=7076187 len=1481642 2021-07-31T02:18:23.522 INFO:tasks.workunit.client.0.smithi045.stdout:op 4016 completed, throughput=5MB/sec 2021-07-31T02:18:23.522 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-4djFz0IQdOuksZr off=7757022 len=1436078 2021-07-31T02:18:23.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:23 smithi045 conmon[33655]: audit 2021-07-31T02:18:22.167140+0000 mgr.x (mgr.24542) 1330 : audit [DBG] from='client.26092 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:18:23.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:23 smithi045 conmon[33655]: audit 2021-07-31T02:18:22.797138+0000 mon.a (mon.0) 1890 : audit [DBG] from='client.? 172.21.15.45:0/3162991093' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:18:23.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:23 smithi045 conmon[39718]: audit 2021-07-31T02:18:22.167140+0000 2021-07-31T02:18:23.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:23 smithi045 conmon[39718]: mgr.x (mgr.24542) 1330 : audit [DBG] from='client.26092 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:18:23.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:23 smithi045 conmon[39718]: audit 2021-07-31T02:18:22.797138+0000 2021-07-31T02:18:23.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:23 smithi045 conmon[39718]: mon.a (mon.0) 1890 : audit [DBG] from='client.? 172.21.15.45:0/3162991093' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:18:23.527 INFO:tasks.workunit.client.0.smithi045.stdout:op 4017 completed, throughput=5MB/sec 2021-07-31T02:18:23.528 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-OChmwnZzrFVHUCB off=4878916 len=250288 2021-07-31T02:18:23.528 INFO:tasks.workunit.client.0.smithi045.stdout:op 4018 completed, throughput=5MB/sec 2021-07-31T02:18:23.528 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-cC3vf55SVK3k2rx off=2351166 len=1808460 2021-07-31T02:18:23.529 INFO:tasks.workunit.client.0.smithi045.stdout:op 4019 completed, throughput=5MB/sec 2021-07-31T02:18:24.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:18:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:18:23.562Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:18:24.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:18:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:18:23.562Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:18:24.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:18:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:18:23.562Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:18:24.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:24 smithi154 conmon[34553]: cluster 2021-07-31T02:18:22.237469+0000 mgr.x (mgr.24542) 1331 : cluster [DBG] pgmap v1806: 9 pgs: 9 active+clean; 127 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2021-07-31T02:18:24.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:24 smithi154 conmon[34553]: audit 2021-07-31T02:18:22.447981+0000 mgr.x (mgr.24542) 1332 : audit [DBG] from='client.26098 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:18:24.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:24 smithi045 conmon[33655]: cluster 2021-07-31T02:18:22.237469+0000 mgr.x (mgr.24542) 1331 2021-07-31T02:18:24.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:24 smithi045 conmon[33655]: : cluster [DBG] pgmap v1806: 9 pgs: 9 active+clean; 127 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2021-07-31T02:18:24.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:24 smithi045 conmon[33655]: audit 2021-07-31T02:18:22.447981+0000 mgr.x 2021-07-31T02:18:24.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:24 smithi045 conmon[33655]: (mgr.24542) 1332 : audit [DBG] from='client.26098 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:18:24.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:24 smithi045 conmon[39718]: cluster 2021-07-31T02:18:22.237469+0000 2021-07-31T02:18:24.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:24 smithi045 conmon[39718]: mgr.x (mgr.24542) 1331 : cluster [DBG] pgmap v1806: 9 pgs: 9 active+clean; 127 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2021-07-31T02:18:24.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:24 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T02:18:24.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:24 smithi045 conmon[39718]: 31T02:18:22.447981+0000 mgr.x (mgr.24542) 2021-07-31T02:18:24.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:24 smithi045 conmon[39718]: 1332 : audit [DBG] from='client.26098 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:18:24.531 INFO:tasks.workunit.client.0.smithi045.stdout: 782: throughput=5MB/sec pending data=0 2021-07-31T02:18:24.531 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-NEFSZuqb7ybcshm off=7716019 len=1356607 2021-07-31T02:18:24.533 INFO:tasks.workunit.client.0.smithi045.stdout:op 4020 completed, throughput=5MB/sec 2021-07-31T02:18:24.533 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-RGHGO0YQBO4ocuV off=8503924 len=27423 2021-07-31T02:18:24.534 INFO:tasks.workunit.client.0.smithi045.stdout:op 4021 completed, throughput=5MB/sec 2021-07-31T02:18:24.534 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-CQVxMzGijLJ80pm off=19387733 len=1331883 2021-07-31T02:18:24.535 INFO:tasks.workunit.client.0.smithi045.stdout:op 4022 completed, throughput=5MB/sec 2021-07-31T02:18:24.536 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-3uc89e4YYavELmX off=9561989 len=1506324 2021-07-31T02:18:24.538 INFO:tasks.workunit.client.0.smithi045.stdout:op 4023 completed, throughput=5MB/sec 2021-07-31T02:18:25.538 INFO:tasks.workunit.client.0.smithi045.stdout: 783: throughput=4.99MB/sec pending data=0 2021-07-31T02:18:25.539 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-c9niTtetEm_xiNG off=5410339 len=192663 2021-07-31T02:18:25.540 INFO:tasks.workunit.client.0.smithi045.stdout:op 4024 completed, throughput=4.99MB/sec 2021-07-31T02:18:25.540 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-HmXfSEpq-jZupaH off=1610256 len=1731231 2021-07-31T02:18:25.547 INFO:tasks.workunit.client.0.smithi045.stdout:op 4025 completed, throughput=5MB/sec 2021-07-31T02:18:25.548 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-40ne0qOYkBQLdX- off=1721227 len=2018839 2021-07-31T02:18:25.558 INFO:tasks.workunit.client.0.smithi045.stdout:op 4026 completed, throughput=5MB/sec 2021-07-31T02:18:25.559 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-jX4X3t-vxEX3F-d off=9433644 len=1327277 2021-07-31T02:18:25.565 INFO:tasks.workunit.client.0.smithi045.stdout:op 4027 completed, throughput=5MB/sec 2021-07-31T02:18:25.565 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-AerQ9Iy6kFypPei off=17519115 len=1126986 2021-07-31T02:18:25.566 INFO:tasks.workunit.client.0.smithi045.stdout:op 4028 completed, throughput=5MB/sec 2021-07-31T02:18:26.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:26 smithi154 conmon[34553]: cluster 2021-07-31T02:18:24.237881+0000 mgr.x (mgr.24542) 1333 : cluster [DBG] pgmap v1807: 9 pgs: 9 active+clean; 127 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 3.1 MiB/s rd, 3.4 MiB/s wr, 6 op/s 2021-07-31T02:18:26.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:26 smithi045 conmon[33655]: cluster 2021-07-31T02:18:24.237881+0000 mgr.x (mgr.24542) 1333 : cluster [DBG] 2021-07-31T02:18:26.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:26 smithi045 conmon[33655]: pgmap v1807: 9 pgs: 9 active+clean; 127 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 3.1 MiB/s rd, 3.4 MiB/s wr, 6 op/s 2021-07-31T02:18:26.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:26 smithi045 conmon[39718]: cluster 2021-07-31T02:18:24.237881+0000 mgr.x 2021-07-31T02:18:26.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:26 smithi045 conmon[39718]: (mgr.24542) 1333 : cluster [DBG] pgmap v1807: 9 pgs: 9 active+clean; 127 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 3.1 MiB/s rd, 3.4 MiB/s wr, 6 op/s 2021-07-31T02:18:26.567 INFO:tasks.workunit.client.0.smithi045.stdout: 784: throughput=5MB/sec pending data=0 2021-07-31T02:18:26.567 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-QORCCETYgtFoF9C off=5616917 len=1084889 2021-07-31T02:18:26.568 INFO:tasks.workunit.client.0.smithi045.stdout:op 4029 completed, throughput=5MB/sec 2021-07-31T02:18:26.569 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-R_AFoEDKQUzzOtl off=177355 len=1245176 2021-07-31T02:18:26.576 INFO:tasks.workunit.client.0.smithi045.stdout:op 4030 completed, throughput=5MB/sec 2021-07-31T02:18:26.576 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-gjW_QS3PYY3by-3 off=12210911 len=1640482 2021-07-31T02:18:26.580 INFO:tasks.workunit.client.0.smithi045.stdout:op 4031 completed, throughput=5MB/sec 2021-07-31T02:18:26.580 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-TG7XX5aaA-OvHDz off=11796473 len=1535755 2021-07-31T02:18:26.582 INFO:tasks.workunit.client.0.smithi045.stdout:op 4032 completed, throughput=5MB/sec 2021-07-31T02:18:27.583 INFO:tasks.workunit.client.0.smithi045.stdout: 785: throughput=5MB/sec pending data=0 2021-07-31T02:18:27.583 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-evx4ZVta2-eng65 off=4438181 len=1225506 2021-07-31T02:18:27.590 INFO:tasks.workunit.client.0.smithi045.stdout:op 4033 completed, throughput=5MB/sec 2021-07-31T02:18:27.590 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-MqiD5nOCQyN8UQR off=2217092 len=122476 2021-07-31T02:18:27.591 INFO:tasks.workunit.client.0.smithi045.stdout:op 4034 completed, throughput=5MB/sec 2021-07-31T02:18:27.591 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-UgkkT0nw5xSrmJi off=1435005 len=493787 2021-07-31T02:18:27.595 INFO:tasks.workunit.client.0.smithi045.stdout:op 4035 completed, throughput=5MB/sec 2021-07-31T02:18:27.595 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-dugjptcGNeR07Cv off=16150096 len=109193 2021-07-31T02:18:27.596 INFO:tasks.workunit.client.0.smithi045.stdout:op 4036 completed, throughput=5MB/sec 2021-07-31T02:18:27.596 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-trLRVLDFJ9a-FPJ off=11431608 len=1280436 2021-07-31T02:18:27.597 INFO:tasks.workunit.client.0.smithi045.stdout:op 4037 completed, throughput=5MB/sec 2021-07-31T02:18:27.598 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-wr0KM8YVAtKPdXt off=1797522 len=352099 2021-07-31T02:18:27.600 INFO:tasks.workunit.client.0.smithi045.stdout:op 4038 completed, throughput=5MB/sec 2021-07-31T02:18:27.600 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-062xNccAVX2AuzS off=3979613 len=1425524 2021-07-31T02:18:27.602 INFO:tasks.workunit.client.0.smithi045.stdout:op 4039 completed, throughput=5MB/sec 2021-07-31T02:18:28.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:28 smithi154 conmon[34553]: cluster 2021-07-31T02:18:26.238291+0000 mgr.x (mgr.24542) 1334 : cluster [DBG] pgmap v1808: 9 pgs: 9 active+clean; 127 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.2 MiB/s wr, 3 op/s 2021-07-31T02:18:28.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:28 smithi154 conmon[34553]: audit 2021-07-31T02:18:28.041533+0000 mon.b (mon.2) 223 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:18:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:28 smithi154 conmon[34553]: audit 2021-07-31T02:18:28.042115+0000 mon.a (mon.0) 1891 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:18:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:28 smithi154 conmon[34553]: audit 2021-07-31T02:18:28.063460+0000 mon.b (mon.2) 224 : 2021-07-31T02:18:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:28 smithi154 conmon[34553]: audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:18:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:28 smithi154 conmon[34553]: audit 2021-07-31T02:18:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:28 smithi154 conmon[34553]: 2021-07-31T02:18:28.063890+0000 mon.a (mon.0) 1892 : audit [INF] 2021-07-31T02:18:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:28 smithi154 conmon[34553]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:18:28.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:28 smithi045 conmon[39718]: cluster 2021-07-31T02:18:26.238291+0000 mgr.x (mgr.24542) 1334 : cluster [DBG] pgmap v1808: 9 pgs: 9 active+clean; 127 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.2 MiB/s wr, 3 op/s 2021-07-31T02:18:28.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:28 smithi045 conmon[39718]: audit 2021-07-31T02:18:28.041533+0000 mon.b (mon.2) 223 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:18:28.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:28 smithi045 conmon[39718]: audit 2021-07-31T02:18:28.042115+0000 2021-07-31T02:18:28.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:28 smithi045 conmon[39718]: mon.a (mon.0) 1891 : audit [INF] 2021-07-31T02:18:28.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:28 smithi045 conmon[39718]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:18:28.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:28 smithi045 conmon[39718]: audit 2021-07-31T02 2021-07-31T02:18:28.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:28 smithi045 conmon[39718]: :18:28.063460+0000 mon.b (mon 2021-07-31T02:18:28.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:28 smithi045 conmon[39718]: .2) 224 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:18:28.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:28 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T02:18:28.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:28 smithi045 conmon[39718]: T02:18:28.063890+0000 mon.a (mon.0) 2021-07-31T02:18:28.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:28 smithi045 conmon[39718]: 1892 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:18:28.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:28 smithi045 conmon[33655]: cluster 2021-07-31T02:18:26.238291+0000 mgr.x (mgr.24542) 2021-07-31T02:18:28.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:28 smithi045 conmon[33655]: 1334 : cluster [DBG] pgmap v1808: 9 pgs: 9 active+clean; 127 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.2 MiB/s wr, 3 op/s 2021-07-31T02:18:28.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:28 smithi045 conmon[33655]: audit 2021-07-31T02:18:28.041533+0000 mon.b (mon 2021-07-31T02:18:28.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:28 smithi045 conmon[33655]: .2) 223 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:18:28.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:28 smithi045 conmon[33655]: audit 2021- 2021-07-31T02:18:28.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:28 smithi045 conmon[33655]: 07-31T02:18:28.042115+0000 mon.a (mon. 2021-07-31T02:18:28.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:28 smithi045 conmon[33655]: 0) 1891 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:18:28.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:28 smithi045 conmon[33655]: audit 2021- 2021-07-31T02:18:28.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:28 smithi045 conmon[33655]: 07-31T02:18:28.063460+0000 2021-07-31T02:18:28.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:28 smithi045 conmon[33655]: mon.b (mon.2) 224 2021-07-31T02:18:28.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:28 smithi045 conmon[33655]: : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:18:28.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:28 smithi045 conmon[33655]: audit 2021 2021-07-31T02:18:28.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:28 smithi045 conmon[33655]: -07-31T02:18:28 2021-07-31T02:18:28.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:28 smithi045 conmon[33655]: .063890+0000 mon.a (mon. 2021-07-31T02:18:28.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:28 smithi045 conmon[33655]: 0) 1892 : audit [INF] 2021-07-31T02:18:28.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:28 smithi045 conmon[33655]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:18:28.602 INFO:tasks.workunit.client.0.smithi045.stdout: 786: throughput=4.99MB/sec pending data=0 2021-07-31T02:18:28.602 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-sgN6Iy_on1Y7V4b off=7243422 len=1114707 2021-07-31T02:18:28.607 INFO:tasks.workunit.client.0.smithi045.stdout:op 4040 completed, throughput=5MB/sec 2021-07-31T02:18:28.608 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-UgkkT0nw5xSrmJi off=1695210 len=977680 2021-07-31T02:18:28.613 INFO:tasks.workunit.client.0.smithi045.stdout:op 4041 completed, throughput=5MB/sec 2021-07-31T02:18:28.613 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-jUV6RGGo55yOYxc off=939143 len=1735467 2021-07-31T02:18:28.616 INFO:tasks.workunit.client.0.smithi045.stdout:op 4042 completed, throughput=5MB/sec 2021-07-31T02:18:28.617 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Ks4W-4v1k42i2qk off=9934499 len=1861292 2021-07-31T02:18:28.618 INFO:tasks.workunit.client.0.smithi045.stdout:op 4043 completed, throughput=5MB/sec 2021-07-31T02:18:29.619 INFO:tasks.workunit.client.0.smithi045.stdout: 787: throughput=5MB/sec pending data=0 2021-07-31T02:18:29.619 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-iMWFYVcmDlvX77o off=6949963 len=1579280 2021-07-31T02:18:29.622 INFO:tasks.workunit.client.0.smithi045.stdout:op 4044 completed, throughput=5MB/sec 2021-07-31T02:18:29.622 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-N0OzNL2-0znPmlL off=6813205 len=1542733 2021-07-31T02:18:29.624 INFO:tasks.workunit.client.0.smithi045.stdout:op 4045 completed, throughput=5MB/sec 2021-07-31T02:18:29.624 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-LrS_tIiX-y9CpoE off=1501965 len=875809 2021-07-31T02:18:29.628 INFO:tasks.workunit.client.0.smithi045.stdout:op 4046 completed, throughput=5MB/sec 2021-07-31T02:18:29.629 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-QapBbz9b7kjJ53x off=18349912 len=1837859 2021-07-31T02:18:29.632 INFO:tasks.workunit.client.0.smithi045.stdout:op 4047 completed, throughput=5MB/sec 2021-07-31T02:18:30.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:30 smithi154 conmon[34553]: cluster 2021-07-31T02:18:28.238819+0000 mgr.x (mgr.24542) 1335 : cluster [DBG] pgmap v1809: 9 pgs: 9 active+clean; 127 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.2 MiB/s wr, 3 op/s 2021-07-31T02:18:30.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:30 smithi045 conmon[39718]: cluster 2021-07-31T02:18:28.238819+0000 mgr.x ( 2021-07-31T02:18:30.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:30 smithi045 conmon[39718]: mgr.24542) 1335 : cluster [DBG] pgmap v1809: 9 pgs: 9 active+clean; 127 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.2 MiB/s wr, 3 op/s 2021-07-31T02:18:30.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:30 smithi045 conmon[33655]: cluster 2021-07-31T02:18:28.238819+0000 2021-07-31T02:18:30.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:30 smithi045 conmon[33655]: mgr.x (mgr.24542) 1335 : cluster [DBG] pgmap v1809: 9 pgs: 9 active+clean; 127 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.2 MiB/s wr, 3 op/s 2021-07-31T02:18:30.632 INFO:tasks.workunit.client.0.smithi045.stdout: 788: throughput=5MB/sec pending data=0 2021-07-31T02:18:30.632 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-SVrSZ7hlh3ClH7M off=639812 len=1750599 2021-07-31T02:18:30.642 INFO:tasks.workunit.client.0.smithi045.stdout:op 4048 completed, throughput=5MB/sec 2021-07-31T02:18:30.642 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-6HdiVpys8EBNj5w off=3228038 len=252357 2021-07-31T02:18:30.643 INFO:tasks.workunit.client.0.smithi045.stdout:op 4049 completed, throughput=5MB/sec 2021-07-31T02:18:30.643 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-7rI9WjROqqzSluQ off=2124773 len=158509 2021-07-31T02:18:30.644 INFO:tasks.workunit.client.0.smithi045.stdout:op 4050 completed, throughput=5MB/sec 2021-07-31T02:18:30.644 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-tybwqN5z0m5SjX9 off=15327803 len=76604 2021-07-31T02:18:30.645 INFO:tasks.workunit.client.0.smithi045.stdout:op 4051 completed, throughput=5MB/sec 2021-07-31T02:18:30.645 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-3dS57Xz3gR_klUo off=3990489 len=11298 2021-07-31T02:18:30.647 INFO:tasks.workunit.client.0.smithi045.stdout:op 4052 completed, throughput=5MB/sec 2021-07-31T02:18:30.647 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-spvV2ny8g6lSfuZ off=10298045 len=1973454 2021-07-31T02:18:30.656 INFO:tasks.workunit.client.0.smithi045.stdout:op 4053 completed, throughput=5MB/sec 2021-07-31T02:18:31.503 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: debug 2021-07-31T02:18:31.193+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:18:31.503 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:18:31.503 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:18:31.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: yield 2021-07-31T02:18:31.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:18:31.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:18:31.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:18:31.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:18:31.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:18:31.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:18:31.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:18:31.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:18:31.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:18:31.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: task.result() 2021-07-31T02:18:31.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:18:31.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: debug 2021-07-31T02:18:31.204+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:18:31.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:18:31.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:18:31.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: yield 2021-07-31T02:18:31.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:18:31.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:18:31.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:18:31.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:18:31.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:18:31.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:18:31.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:18:31.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:18:31.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:18:31.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: task.result() 2021-07-31T02:18:31.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:18:31.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: debug 2021-07-31T02:18:31.231+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:18:31.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:18:31.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:18:31.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: yield 2021-07-31T02:18:31.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:18:31.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:18:31.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:18:31.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:18:31.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:18:31.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:18:31.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:18:31.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:18:31.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:18:31.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: task.result() 2021-07-31T02:18:31.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:18:31.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: debug 2021-07-31T02:18:31.231+0000 7fcb93160700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:18:31.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:18:31.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:18:31.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: yield 2021-07-31T02:18:31.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:18:31.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:18:31.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:18:31.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:18:31.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:18:31.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:18:31.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:18:31.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:18:31.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:18:31.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: task.result() 2021-07-31T02:18:31.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:18:31.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: 2021-07-31T02:18:31.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:18:31.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: 2021-07-31T02:18:31.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:18:31.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:18:31.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:18:31.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:18:31.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:18:31.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:18:31.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:18:31.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:18:31.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:18:31.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:18:31.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:18:31.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:18:31.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:18:31.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:18:31.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:18:31.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:18:31.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:18:31.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:18:31.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: raise 2021-07-31T02:18:31.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:18:31.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:18:31.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:18:31.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:18:31.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:18:31.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: debug 2021-07-31T02:18:31.239+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:18:31.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:18:31.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:18:31.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: yield 2021-07-31T02:18:31.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:18:31.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:18:31.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:18:31.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:18:31.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:18:31.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:18:31.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:18:31.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:18:31.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:18:31.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: task.result() 2021-07-31T02:18:31.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:18:31.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: debug 2021-07-31T02:18:31.239+0000 7fcb9295f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:18:31.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:18:31.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:18:31.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: yield 2021-07-31T02:18:31.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:18:31.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:18:31.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:18:31.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:18:31.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:18:31.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:18:31.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:18:31.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:18:31.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:18:31.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: task.result() 2021-07-31T02:18:31.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:18:31.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: 2021-07-31T02:18:31.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:18:31.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: 2021-07-31T02:18:31.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:18:31.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:18:31.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:18:31.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:18:31.535 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:18:31.535 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:18:31.536 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:18:31.536 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:18:31.536 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:18:31.536 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:18:31.536 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:18:31.537 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:18:31.537 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:18:31.537 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:18:31.537 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:18:31.538 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:18:31.538 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:18:31.538 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:18:31.538 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: raise 2021-07-31T02:18:31.539 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:18:31.539 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:18:31.539 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:18:31.540 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:18:31.540 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:18:31 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:18:31.656 INFO:tasks.workunit.client.0.smithi045.stdout: 789: throughput=4.99MB/sec pending data=0 2021-07-31T02:18:31.657 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-xef0FA04nkp4RFR off=12810272 len=1568439 2021-07-31T02:18:31.659 INFO:tasks.workunit.client.0.smithi045.stdout:op 4054 completed, throughput=5MB/sec 2021-07-31T02:18:31.659 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-N6dRQhdmcCoTN5j off=10821360 len=1762110 2021-07-31T02:18:31.662 INFO:tasks.workunit.client.0.smithi045.stdout:op 4055 completed, throughput=5MB/sec 2021-07-31T02:18:31.662 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-mRAwGbMU4Eq4dzZ off=918403 len=1716486 2021-07-31T02:18:31.665 INFO:tasks.workunit.client.0.smithi045.stdout:op 4056 completed, throughput=5MB/sec 2021-07-31T02:18:32.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:32 smithi154 conmon[34553]: cluster 2021-07-31T02:18:30.239343+0000 mgr.x (mgr.24542) 1336 : cluster [DBG] pgmap v1810: 9 pgs: 9 active+clean; 127 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 3.3 MiB/s rd, 3.1 MiB/s wr, 5 op/s 2021-07-31T02:18:32.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:32 smithi154 conmon[34553]: audit 2021-07-31T02:18:31.143053+0000 mon.b (mon.2) 225 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:18:32.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:32 smithi154 conmon[34553]: audit 2021-07-31T02:18:31.149489+0000 mon.b (mon.2) 226 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:18:32.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:32 smithi154 conmon[34553]: audit 2021-07-31T02 2021-07-31T02:18:32.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:32 smithi154 conmon[34553]: :18:31.150410+0000 mon.b (mon.2) 227 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:18:32.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:32 smithi154 conmon[34553]: cephadm 2021-07- 2021-07-31T02:18:32.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:32 smithi154 conmon[34553]: 31T02:18:31.151149+0000 mgr.x (mgr.24542) 1337 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:18:32.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:32 smithi154 conmon[34553]: cephadm 2021-07-31T02:18:31 2021-07-31T02:18:32.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:32 smithi154 conmon[34553]: .194291+0000 mgr.x (mgr.24542) 1338 : cephadm [ERR] 2021-07-31T02:18:32.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:32 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:18:32.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:18:32.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:32 smithi154 conmon[34553]: yield 2021-07-31T02:18:32.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:18:32.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:32 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:18:32.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:18:32.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:32 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:18:32.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:18:32.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:32 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:18:32.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:18:32.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:32 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:18:32.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:18:32.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:32 smithi154 conmon[34553]: task.result() 2021-07-31T02:18:32.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:32 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:18:32.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:32 smithi154 conmon[34553]: cephadm 2021-07-31T02:18:31.194708+0000 mgr.x (mgr.24542) 1339 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:18:32.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:32 smithi154 conmon[34553]: cephadm 2021-07-31T02:18:31.205811+0000 mgr.x (mgr.24542) 1340 : cephadm [ERR] 2021-07-31T02:18:32.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:32 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:18:32.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:18:32.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:32 smithi154 conmon[34553]: yield 2021-07-31T02:18:32.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:18:32.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:32 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:18:32.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:18:32.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:32 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:18:32.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:18:32.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:32 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:18:32.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:18:32.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:32 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:18:32.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:18:32.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:32 smithi154 conmon[34553]: task.result() 2021-07-31T02:18:32.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:32 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:18:32.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:32 smithi154 conmon[34553]: cephadm 2021-07-31T02:18:31.206779 2021-07-31T02:18:32.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:32 smithi154 conmon[34553]: +0000 mgr.x (mgr.24542) 1341 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:18:32.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:32 smithi154 conmon[34553]: cephadm 2021-07-31T02:18:31.232186+0000 mgr.x (mgr.24542) 1342 : cephadm [ERR] 2021-07-31T02:18:32.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:32 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:18:32.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:18:32.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:32 smithi154 conmon[34553]: yield 2021-07-31T02:18:32.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:18:32.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:32 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:18:32.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:18:32.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:32 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:18:32.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:18:32.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:32 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:18:32.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:18:32.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:32 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:18:32.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:18:32.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:32 smithi154 conmon[34553]: task.result() 2021-07-31T02:18:32.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:32 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:18:32.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[33655]: cluster 2021-07-31T02:18:30.239343+0000 mgr.x (mgr.24542) 1336 : cluster [DBG] pgmap v1810: 9 pgs: 9 active+clean; 127 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 3.3 MiB/s rd, 3.1 MiB/s wr, 5 op/s 2021-07-31T02:18:32.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[33655]: 2021-07-31T02:18:32.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[33655]: audit 2021-07-31T02:18:31.143053+0000 mon.b (mon.2) 225 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:18:32.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[33655]: audit 2021-07-31T02:18:31.149489+0000 mon.b (mon.2) 226 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:18:32.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[33655]: audit 2021-07-31T02:18:31.150410 2021-07-31T02:18:32.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[33655]: +0000 mon.b (mon.2) 227 : audit [INF] 2021-07-31T02:18:32.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[33655]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:18:32.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[33655]: cephadm 2021- 2021-07-31T02:18:32.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[33655]: 07-31T02:18:31 2021-07-31T02:18:32.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[33655]: .151149+0000 mgr.x (mgr.24542) 1337 : cephadm [WRN] 2021-07-31T02:18:32.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[33655]: unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:18:32.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[33655]: cephadm 2021-07-31T02:18:32.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[33655]: 2021-07-31T02:18:31.194291+0000 mgr.x (mgr.24542) 1338 : cephadm 2021-07-31T02:18:32.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[33655]: [ERR] 2021-07-31T02:18:32.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:18:32.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:18:32.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[33655]: yield 2021-07-31T02:18:32.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:18:32.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:18:32.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:18:32.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:18:32.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:18:32.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:18:32.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:18:32.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:18:32.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:18:32.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[33655]: task.result() 2021-07-31T02:18:32.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:18:32.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[33655]: cephadm 2021-07-31T02:18:31.194708+0000 mgr.x (mgr.24542) 1339 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:18:32.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[33655]: cephadm 2021-07-31T 2021-07-31T02:18:32.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[33655]: 02:18:31.205811+0000 mgr.x (mgr. 2021-07-31T02:18:32.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[33655]: 24542) 1340 : cephadm [ERR] 2021-07-31T02:18:32.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:18:32.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:18:32.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[33655]: yield 2021-07-31T02:18:32.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:18:32.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:18:32.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[39718]: cluster 2021-07-31T02:18:30.239343+0000 mgr.x ( 2021-07-31T02:18:32.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[39718]: mgr.24542) 1336 : cluster [DBG] pgmap v1810: 9 pgs: 9 active+clean; 127 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 3.3 MiB/s rd, 3.1 MiB/s wr, 5 op/s 2021-07-31T02:18:32.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[39718]: audit 2021-07-31T02:18:31 2021-07-31T02:18:32.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[39718]: .143053+0000 mon.b (mon.2) 225 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:18:32.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[39718]: audit 2021-07-31T02:18:32.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[39718]: 2021-07-31T02:18:31.149489+0000 mon.b (mon.2) 226 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:18:32.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[39718]: 2021-07-31T02:18:32.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[39718]: audit 2021-07 2021-07-31T02:18:32.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[39718]: -31T02:18:31.150410+0000 mon.b (mon.2) 227 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:18:32.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[39718]: 2021-07-31T02:18:32.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[39718]: cephadm 2021 2021-07-31T02:18:32.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[39718]: -07-31T02:18:31.151149+0000 mgr.x (mgr.24542) 1337 : cephadm [WRN] 2021-07-31T02:18:32.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[39718]: unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:18:32.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[39718]: cephadm 2021-07-31T02: 2021-07-31T02:18:32.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[39718]: 18:31.194291+0000 mgr.x (mgr.24542) 1338 : cephadm [ERR] 2021-07-31T02:18:32.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:18:32.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:18:32.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[39718]: yield 2021-07-31T02:18:32.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:18:32.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:18:32.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:18:32.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:18:32.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:18:32.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:18:32.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:18:32.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:18:32.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:18:32.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[39718]: task.result() 2021-07-31T02:18:32.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:18:32.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[39718]: cephadm 2021-07-31T02:18:31.194708+0000 mgr.x (mgr.24542) 1339 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:18:32.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[39718]: cephadm 2021-07-31T02:18:31.205811+0000 mgr.x (mgr.24542) 1340 : cephadm [ERR] 2021-07-31T02:18:32.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:18:32.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:18:32.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[39718]: yield 2021-07-31T02:18:32.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:18:32.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:18:32.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:18:32.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:18:32.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:18:32.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:18:32.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:18:32.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:18:32.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:18:32.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[39718]: task.result() 2021-07-31T02:18:32.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:18:32.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[39718]: cephadm 2021-07-31T02:18:31.206779+0000 mgr.x (mgr.24542) 1341 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:18:32.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[39718]: cephadm 2021-07-31T02:18:31.232186+0000 mgr.x (mgr.24542) 1342 : cephadm [ERR] 2021-07-31T02:18:32.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:18:32.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:18:32.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[39718]: yield 2021-07-31T02:18:32.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:18:32.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:18:32.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:18:32.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:18:32.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:18:32.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:18:32.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:18:32.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:18:32.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:18:32.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[39718]: task.result() 2021-07-31T02:18:32.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:18:32.548 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:18:32.548 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:18:32.548 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:18:32.548 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:18:32.548 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:18:32.549 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:18:32.549 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:18:32.549 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[33655]: task.result() 2021-07-31T02:18:32.549 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:18:32.550 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[33655]: cephadm 2021-07-31T02:18:31.206779+0000 mgr.x (mgr.24542) 1341 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:18:32.550 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[33655]: cephadm 2021-07-31T02:18:32.550 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[33655]: 2021-07-31T02:18: 2021-07-31T02:18:32.550 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[33655]: 31.232186+0000 mgr.x (mgr.24542 2021-07-31T02:18:32.550 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[33655]: ) 1342 : cephadm [ERR] 2021-07-31T02:18:32.551 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:18:32.551 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:18:32.551 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[33655]: yield 2021-07-31T02:18:32.551 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:18:32.552 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:18:32.552 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:18:32.552 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:18:32.552 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:18:32.553 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:18:32.553 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:18:32.553 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:18:32.553 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:18:32.553 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[33655]: task.result() 2021-07-31T02:18:32.554 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:32 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:18:32.665 INFO:tasks.workunit.client.0.smithi045.stdout: 790: throughput=4.99MB/sec pending data=0 2021-07-31T02:18:32.666 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Ygcl2FsJ7_EP3en off=2818439 len=2090222 2021-07-31T02:18:32.668 INFO:tasks.workunit.client.0.smithi045.stdout:op 4057 completed, throughput=5MB/sec 2021-07-31T02:18:32.668 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-xaDL8bTJ7KTw4oY off=6842349 len=649057 2021-07-31T02:18:32.672 INFO:tasks.workunit.client.0.smithi045.stdout:op 4058 completed, throughput=5MB/sec 2021-07-31T02:18:32.673 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-h5Exn9OIJZJ5ncc off=860963 len=616110 2021-07-31T02:18:32.674 INFO:tasks.workunit.client.0.smithi045.stdout:op 4059 completed, throughput=5MB/sec 2021-07-31T02:18:32.674 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-yEws7U_P9Tl2nuP off=11548295 len=1860695 2021-07-31T02:18:32.677 INFO:tasks.workunit.client.0.smithi045.stdout:op 4060 completed, throughput=5MB/sec 2021-07-31T02:18:33.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:18:31.232667+0000 mgr.x (mgr.24542) 1343 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:18:33.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:18:33.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:18:33.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: yield 2021-07-31T02:18:33.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:18:33.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:18:33.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:18:33.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:18:33.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:18:33.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:18:33.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:18:33.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:18:33.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:18:33.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:18:33.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:18:33.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: 2021-07-31T02:18:33.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:18:33.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: 2021-07-31T02:18:33.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:18:33.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:18:33.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:18:33.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:18:33.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:18:33.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:18:33.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:18:33.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:18:33.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:18:33.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:18:33.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:18:33.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:18:33.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:18:33.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:18:33.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:18:33.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:18:33.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:18:33.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:18:33.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: raise 2021-07-31T02:18:33.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:18:33.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:18:33.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:18:33.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:18:33.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:18:33.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: 2021-07-31T02:18:33.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:18:31.240070+0000 mgr.x (mgr.24542) 1344 : cephadm [ERR] 2021-07-31T02:18:33.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:18:33.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:18:33.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: yield 2021-07-31T02:18:33.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:18:33.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:18:33.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:18:33.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:18:33.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:18:33.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:18:33.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:18:33.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:18:33.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:18:33.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:18:33.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:18:33.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:18:31.240603+0000 mgr.x (mgr.24542) 1345 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:18:33.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:18:33.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:18:33.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: yield 2021-07-31T02:18:33.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:18:33.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:18:33.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:18:33.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:18:33.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:18:33.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:18:33.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:18:33.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:18:33.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:18:33.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:18:33.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:18:33.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: 2021-07-31T02:18:33.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:18:33.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: 2021-07-31T02:18:33.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:18:33.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:18:33.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:18:33.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:18:33.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:18:33.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:18:33.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:18:33.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:18:33.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:18:33.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:18:33.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:18:33.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:18:33.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:18:33.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:18:33.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:18:33.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:18:33.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:18:33.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:18:33.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: raise 2021-07-31T02:18:33.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:18:33.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:18:33.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:18:33.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:18:33.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:33 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:18:33.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:18:31.232667+0000 mgr.x (mgr.24542) 1343 : 2021-07-31T02:18:33.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:18:33.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:18:33.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:18:33.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: yield 2021-07-31T02:18:33.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:18:33.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:18:33.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:18:33.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:18:33.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:18:33.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:18:33.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:18:33.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:18:33.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:18:33.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:18:33.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:18:33.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: 2021-07-31T02:18:33.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:18:33.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: 2021-07-31T02:18:33.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:18:33.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:18:33.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:18:33.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:18:33.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:18:33.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:18:33.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:18:33.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:18:33.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:18:33.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:18:33.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:18:33.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:18:33.535 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:18:33.535 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:18:33.535 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:18:33.535 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:18:33.535 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:18:33.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:18:33.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: raise 2021-07-31T02:18:33.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:18:33.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:18:33.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:18:33.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: cephadm 2021-07-31T02: 2021-07-31T02:18:33.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: 18:31.232667+0000 mgr.x (mgr.24542) 1343 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:18:33.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:18:33.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:18:33.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: yield 2021-07-31T02:18:33.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:18:33.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:18:33.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:18:33.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:18:33.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:18:33.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:18:33.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:18:33.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:18:33.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:18:33.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:18:33.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:18:33.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: 2021-07-31T02:18:33.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:18:33.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: 2021-07-31T02:18:33.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:18:33.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:18:33.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:18:33.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:18:33.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:18:33.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:18:33.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:18:33.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:18:33.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:18:33.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:18:33.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:18:33.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:18:33.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:18:33.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:18:33.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:18:33.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:18:33.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:18:33.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:18:33.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: raise 2021-07-31T02:18:33.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:18:33.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:18:33.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:18:33.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:18:33.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:18:33.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:18:31.240070+0000 mgr.x (mgr.24542) 1344 : cephadm [ERR] 2021-07-31T02:18:33.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:18:33.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:18:33.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: yield 2021-07-31T02:18:33.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:18:33.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:18:33.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:18:33.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:18:33.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:18:33.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:18:33.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:18:33.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:18:33.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:18:33.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:18:33.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:18:33.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:18:31.240603+0000 mgr.x (mgr.24542) 1345 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:18:33.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:18:33.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:18:33.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: yield 2021-07-31T02:18:33.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:18:33.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:18:33.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:18:33.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:18:33.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:18:33.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:18:33.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:18:33.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:18:33.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:18:33.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:18:33.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:18:33.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: 2021-07-31T02:18:33.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:18:33.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: 2021-07-31T02:18:33.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:18:33.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:18:33.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:18:33.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:18:33.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:18:33.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:18:33.556 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:18:33.556 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:18:33.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:18:31.240070+0000 mgr.x (mgr.24542) 1344 : cephadm [ERR] 2021-07-31T02:18:33.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:18:33.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:18:33.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: yield 2021-07-31T02:18:33.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:18:33.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:18:33.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:18:33.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:18:33.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:18:33.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:18:33.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:18:33.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:18:33.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:18:33.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:18:33.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:18:33.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:18:31.240603+0000 mgr.x (mgr.24542) 1345 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:18:33.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:18:33.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:18:33.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: yield 2021-07-31T02:18:33.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:18:33.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:18:33.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:18:33.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:18:33.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:18:33.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:18:33.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:18:33.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:18:33.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:18:33.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:18:33.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:18:33.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: 2021-07-31T02:18:33.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:18:33.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: 2021-07-31T02:18:33.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:18:33.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:18:33.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:18:33.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:18:33.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:18:33.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:18:33.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:18:33.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:18:33.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:18:33.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:18:33.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:18:33.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:18:33.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:18:33.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:18:33.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:18:33.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:18:33.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:18:33.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:18:33.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: raise 2021-07-31T02:18:33.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:18:33.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:18:33.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:18:33.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:18:33.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:18:33.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:18:33.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:18:33.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:18:33.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:18:33.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:18:33.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:18:33.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:18:33.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:18:33.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:18:33.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:18:33.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:18:33.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:18:33.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: raise 2021-07-31T02:18:33.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:18:33.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:18:33.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:18:33.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:18:33.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:18:33.678 INFO:tasks.workunit.client.0.smithi045.stdout: 791: throughput=4.99MB/sec pending data=0 2021-07-31T02:18:33.678 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-GsPoW0CSc5k6y6I off=9320084 len=1782537 2021-07-31T02:18:33.691 INFO:tasks.workunit.client.0.smithi045.stdout:op 4061 completed, throughput=5MB/sec 2021-07-31T02:18:33.691 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-WPBG_WEpg6JoQQU off=10525874 len=1970603 2021-07-31T02:18:33.694 INFO:tasks.workunit.client.0.smithi045.stdout:op 4062 completed, throughput=5MB/sec 2021-07-31T02:18:33.694 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-bYK1xc6srLr248c off=5872672 len=592050 2021-07-31T02:18:33.697 INFO:tasks.workunit.client.0.smithi045.stdout:op 4063 completed, throughput=5MB/sec 2021-07-31T02:18:33.698 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-35FlEg3Yp5kmOXE off=2372124 len=1404568 2021-07-31T02:18:33.699 INFO:tasks.workunit.client.0.smithi045.stdout:op 4064 completed, throughput=5MB/sec 2021-07-31T02:18:34.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:18:33.562Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:18:34.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:18:33.562Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:18:34.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:18:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:18:33.562Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:18:34.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:34 smithi154 conmon[34553]: cluster 2021-07-31T02:18:32.239692+0000 mgr.x (mgr.24542) 1346 : cluster [DBG] pgmap v1811: 9 pgs: 9 active+clean; 127 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.8 MiB/s wr, 3 op/s 2021-07-31T02:18:34.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:34 smithi045 conmon[33655]: cluster 2021-07-31T02:18:32.239692+0000 mgr.x (mgr.24542) 1346 : cluster [DBG] pgmap v1811: 9 pgs: 9 active+clean; 127 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.8 MiB/s wr, 3 op/s 2021-07-31T02:18:34.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:34 smithi045 conmon[33655]: 2021-07-31T02:18:34.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:34 smithi045 conmon[39718]: cluster 2021-07-31T02:18:32.239692+0000 mgr.x (mgr.24542 2021-07-31T02:18:34.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:34 smithi045 conmon[39718]: ) 1346 : cluster [DBG] pgmap v1811: 9 pgs: 9 active+clean; 127 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.8 MiB/s wr, 3 op/s 2021-07-31T02:18:34.699 INFO:tasks.workunit.client.0.smithi045.stdout: 792: throughput=4.99MB/sec pending data=0 2021-07-31T02:18:34.699 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-tcAT-1oDnv-06sY off=3804303 len=970490 2021-07-31T02:18:34.706 INFO:tasks.workunit.client.0.smithi045.stdout:op 4065 completed, throughput=5MB/sec 2021-07-31T02:18:34.706 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-z7rc9o3-Ya94OZo off=1509037 len=1556476 2021-07-31T02:18:34.714 INFO:tasks.workunit.client.0.smithi045.stdout:op 4066 completed, throughput=5MB/sec 2021-07-31T02:18:34.714 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-jBLn3ft-YJwn9QQ off=15401065 len=1173803 2021-07-31T02:18:34.716 INFO:tasks.workunit.client.0.smithi045.stdout:op 4067 completed, throughput=5MB/sec 2021-07-31T02:18:34.716 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-nA8pIsyIin8hSFR off=892672 len=692733 2021-07-31T02:18:34.720 INFO:tasks.workunit.client.0.smithi045.stdout:op 4068 completed, throughput=5MB/sec 2021-07-31T02:18:34.721 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-HldAaR8Cyhdw_np off=2619876 len=1428755 2021-07-31T02:18:34.722 INFO:tasks.workunit.client.0.smithi045.stdout:op 4069 completed, throughput=5MB/sec 2021-07-31T02:18:35.723 INFO:tasks.workunit.client.0.smithi045.stdout: 793: throughput=5MB/sec pending data=0 2021-07-31T02:18:35.723 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-rGACQt5ylVxcJya off=12691936 len=2013666 2021-07-31T02:18:35.733 INFO:tasks.workunit.client.0.smithi045.stdout:op 4070 completed, throughput=5MB/sec 2021-07-31T02:18:35.733 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-KO_Gy0vS-z8T6f- off=11611287 len=1953677 2021-07-31T02:18:35.744 INFO:tasks.workunit.client.0.smithi045.stdout:op 4071 completed, throughput=5MB/sec 2021-07-31T02:18:35.746 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-PklXAkugO5wrPuX off=7328455 len=93713 2021-07-31T02:18:35.747 INFO:tasks.workunit.client.0.smithi045.stdout:op 4072 completed, throughput=5MB/sec 2021-07-31T02:18:35.747 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-mcGRAJbW_7ifZ2v off=8045845 len=251530 2021-07-31T02:18:35.747 INFO:tasks.workunit.client.0.smithi045.stdout:op 4073 completed, throughput=5MB/sec 2021-07-31T02:18:36.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:36 smithi154 conmon[34553]: cluster 2021-07-31T02:18:34.240117+0000 mgr.x (mgr.24542) 1347 : cluster [DBG] pgmap v1812: 9 pgs: 9 active+clean; 127 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 4.0 MiB/s rd, 2.3 MiB/s wr, 5 op/s 2021-07-31T02:18:36.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:36 smithi045 conmon[33655]: cluster 2021-07-31T02:18:34.240117+0000 mgr.x (mgr.24542 2021-07-31T02:18:36.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:36 smithi045 conmon[33655]: ) 1347 : cluster [DBG] pgmap v1812: 9 pgs: 9 active+clean; 127 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 4.0 MiB/s rd, 2.3 MiB/s wr, 5 op/s 2021-07-31T02:18:36.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:36 smithi045 conmon[39718]: cluster 2021-07-31T02:18:34.240117+0000 mgr.x (mgr. 2021-07-31T02:18:36.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:36 smithi045 conmon[39718]: 24542) 1347 : cluster [DBG] pgmap v1812: 9 pgs: 9 active+clean; 127 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 4.0 MiB/s rd, 2.3 MiB/s wr, 5 op/s 2021-07-31T02:18:36.746 INFO:tasks.workunit.client.0.smithi045.stdout: 794: throughput=4.99MB/sec pending data=0 2021-07-31T02:18:36.746 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-oXagn1k_HPEv6-N off=4843344 len=1566532 2021-07-31T02:18:36.753 INFO:tasks.workunit.client.0.smithi045.stdout:op 4074 completed, throughput=5MB/sec 2021-07-31T02:18:36.753 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-YUwHDqqqBTigkwK off=7694651 len=1332655 2021-07-31T02:18:36.755 INFO:tasks.workunit.client.0.smithi045.stdout:op 4075 completed, throughput=5MB/sec 2021-07-31T02:18:36.755 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Yq8Zt1iJLnQmkBV off=0 len=1063314 2021-07-31T02:18:36.760 INFO:tasks.workunit.client.0.smithi045.stdout:op 4076 completed, throughput=5MB/sec 2021-07-31T02:18:36.761 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-jkMH7bEjnMab6Mo off=9100664 len=1867070 2021-07-31T02:18:36.764 INFO:tasks.workunit.client.0.smithi045.stdout:op 4077 completed, throughput=5MB/sec 2021-07-31T02:18:37.764 INFO:tasks.workunit.client.0.smithi045.stdout: 795: throughput=4.99MB/sec pending data=0 2021-07-31T02:18:37.765 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-I-gmYA4Eg66LtWj off=1401212 len=604109 2021-07-31T02:18:37.770 INFO:tasks.workunit.client.0.smithi045.stdout:op 4078 completed, throughput=5MB/sec 2021-07-31T02:18:37.770 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-k-KcgXJkGB7jovB off=3346429 len=893349 2021-07-31T02:18:37.776 INFO:tasks.workunit.client.0.smithi045.stdout:op 4079 completed, throughput=5MB/sec 2021-07-31T02:18:37.776 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-gnEm5GHucdNH_I6 off=9108887 len=172334 2021-07-31T02:18:37.779 INFO:tasks.workunit.client.0.smithi045.stdout:op 4080 completed, throughput=5MB/sec 2021-07-31T02:18:37.780 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-yG08f1yU7cg-IZH off=57270 len=44447 2021-07-31T02:18:37.781 INFO:tasks.workunit.client.0.smithi045.stdout:op 4081 completed, throughput=5MB/sec 2021-07-31T02:18:37.781 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ATNndgNFC1RSoOu off=1009067 len=1195060 2021-07-31T02:18:37.786 INFO:tasks.workunit.client.0.smithi045.stdout:op 4082 completed, throughput=5MB/sec 2021-07-31T02:18:37.786 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-W7oMd4a0fP5B3Jm off=10147084 len=979760 2021-07-31T02:18:37.792 INFO:tasks.workunit.client.0.smithi045.stdout:op 4083 completed, throughput=5MB/sec 2021-07-31T02:18:37.793 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-lDWC97BEmZJyh4a off=7981103 len=943300 2021-07-31T02:18:37.794 INFO:tasks.workunit.client.0.smithi045.stdout:op 4084 completed, throughput=5MB/sec 2021-07-31T02:18:38.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:38 smithi154 conmon[34553]: cluster 2021-07-31T02:18:36.240557+0000 mgr.x (mgr.24542) 1348 : cluster [DBG] pgmap v1813: 9 pgs: 9 active+clean; 127 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 2.8 MiB/s rd, 1.4 MiB/s wr, 3 op/s 2021-07-31T02:18:38.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:38 smithi045 conmon[39718]: cluster 2021-07-31T02:18:36.240557+0000 mgr.x (mgr 2021-07-31T02:18:38.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:38 smithi045 conmon[39718]: .24542) 1348 : cluster [DBG] pgmap v1813: 9 pgs: 9 active+clean; 127 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 2.8 MiB/s rd, 1.4 MiB/s wr, 3 op/s 2021-07-31T02:18:38.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:38 smithi045 conmon[33655]: cluster 2021-07-31T02:18:36.240557+0000 mgr.x ( 2021-07-31T02:18:38.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:38 smithi045 conmon[33655]: mgr.24542) 1348 : cluster [DBG] pgmap v1813: 9 pgs: 9 active+clean; 127 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 2.8 MiB/s rd, 1.4 MiB/s wr, 3 op/s 2021-07-31T02:18:38.795 INFO:tasks.workunit.client.0.smithi045.stdout: 796: throughput=4.99MB/sec pending data=0 2021-07-31T02:18:38.795 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-2-ReXFXv8q07MCI off=4660603 len=345317 2021-07-31T02:18:38.799 INFO:tasks.workunit.client.0.smithi045.stdout:op 4085 completed, throughput=4.99MB/sec 2021-07-31T02:18:38.799 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-_mTGZbBKqmiakTX off=5027521 len=9369 2021-07-31T02:18:38.802 INFO:tasks.workunit.client.0.smithi045.stdout:op 4086 completed, throughput=4.99MB/sec 2021-07-31T02:18:38.802 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-sQk3l1HpPsY4ySM off=102214 len=165235 2021-07-31T02:18:38.804 INFO:tasks.workunit.client.0.smithi045.stdout:op 4087 completed, throughput=4.99MB/sec 2021-07-31T02:18:38.804 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-VIsRP0W7ICb4gH6 off=7375610 len=996316 2021-07-31T02:18:38.808 INFO:tasks.workunit.client.0.smithi045.stdout:op 4088 completed, throughput=5MB/sec 2021-07-31T02:18:38.808 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-DoIUjVfn0_QiJy8 off=19195355 len=348639 2021-07-31T02:18:38.809 INFO:tasks.workunit.client.0.smithi045.stdout:op 4089 completed, throughput=5MB/sec 2021-07-31T02:18:38.810 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-PYBPMGkrWmkCVDL off=1791483 len=1240957 2021-07-31T02:18:38.815 INFO:tasks.workunit.client.0.smithi045.stdout:op 4090 completed, throughput=5MB/sec 2021-07-31T02:18:38.815 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-WAWikBp8vyCTksD off=16545520 len=1136089 2021-07-31T02:18:38.817 INFO:tasks.workunit.client.0.smithi045.stdout:op 4091 completed, throughput=5MB/sec 2021-07-31T02:18:38.817 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Fcjlc5xeuAhi85R off=15213331 len=1798296 2021-07-31T02:18:38.818 INFO:tasks.workunit.client.0.smithi045.stdout:op 4092 completed, throughput=5MB/sec 2021-07-31T02:18:39.818 INFO:tasks.workunit.client.0.smithi045.stdout: 797: throughput=4.99MB/sec pending data=0 2021-07-31T02:18:39.818 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-FPv-dxrBiMUw_Pd off=14680356 len=953302 2021-07-31T02:18:39.819 INFO:tasks.workunit.client.0.smithi045.stdout:op 4093 completed, throughput=5MB/sec 2021-07-31T02:18:39.819 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-y6Ai0RAqXUBQzde off=16642331 len=1332141 2021-07-31T02:18:39.822 INFO:tasks.workunit.client.0.smithi045.stdout:op 4094 completed, throughput=5MB/sec 2021-07-31T02:18:39.823 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Ja893liHbUYg9bA off=2208238 len=1184021 2021-07-31T02:18:39.825 INFO:tasks.workunit.client.0.smithi045.stdout:op 4095 completed, throughput=5MB/sec 2021-07-31T02:18:39.825 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-BmCJeD91zY0FdiJ off=9393320 len=1483085 2021-07-31T02:18:39.828 INFO:tasks.workunit.client.0.smithi045.stdout:op 4096 completed, throughput=5MB/sec 2021-07-31T02:18:40.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:40 smithi154 conmon[34553]: cluster 2021-07-31T02:18:38.240981+0000 mgr.x (mgr.24542) 1349 : cluster [DBG] pgmap v1814: 9 pgs: 9 active+clean; 127 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 2.8 MiB/s rd, 1.4 MiB/s wr, 3 op/s 2021-07-31T02:18:40.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:40 smithi045 conmon[39718]: cluster 2021-07-31T02:18:38.240981+0000 2021-07-31T02:18:40.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:40 smithi045 conmon[39718]: mgr.x (mgr.24542) 1349 : cluster [DBG] pgmap v1814: 9 pgs: 9 active+clean; 127 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 2.8 MiB/s rd, 1.4 MiB/s wr, 3 op/s 2021-07-31T02:18:40.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:40 smithi045 conmon[33655]: cluster 2021-07-31T02:18:38.240981+0000 mgr.x 2021-07-31T02:18:40.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:40 smithi045 conmon[33655]: (mgr.24542) 1349 : cluster [DBG] pgmap v1814: 9 pgs: 9 active+clean; 127 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 2.8 MiB/s rd, 1.4 MiB/s wr, 3 op/s 2021-07-31T02:18:40.828 INFO:tasks.workunit.client.0.smithi045.stdout: 798: throughput=4.99MB/sec pending data=0 2021-07-31T02:18:40.829 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-DQHen5iukSa1rT1 off=17018120 len=1150100 2021-07-31T02:18:40.831 INFO:tasks.workunit.client.0.smithi045.stdout:op 4097 completed, throughput=5MB/sec 2021-07-31T02:18:40.831 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-KNaNpQqyci-faeS off=7668008 len=884685 2021-07-31T02:18:40.836 INFO:tasks.workunit.client.0.smithi045.stdout:op 4098 completed, throughput=5MB/sec 2021-07-31T02:18:40.837 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-qb_Exhjh8M_zm-o off=1663395 len=485555 2021-07-31T02:18:40.838 INFO:tasks.workunit.client.0.smithi045.stdout:op 4099 completed, throughput=5MB/sec 2021-07-31T02:18:40.838 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-xFf3fjFtggOYlUp off=14368816 len=1638212 2021-07-31T02:18:40.841 INFO:tasks.workunit.client.0.smithi045.stdout:op 4100 completed, throughput=5MB/sec 2021-07-31T02:18:40.841 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-m0kbI_a0_KOoFNC off=14077216 len=1348970 2021-07-31T02:18:40.847 INFO:tasks.workunit.client.0.smithi045.stdout:op 4101 completed, throughput=5MB/sec 2021-07-31T02:18:41.848 INFO:tasks.workunit.client.0.smithi045.stdout: 799: throughput=4.99MB/sec pending data=0 2021-07-31T02:18:41.848 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-8yg-M88lWlVw8oO off=420235 len=2005481 2021-07-31T02:18:41.851 INFO:tasks.workunit.client.0.smithi045.stdout:op 4102 completed, throughput=5MB/sec 2021-07-31T02:18:41.852 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-deqo38dpTYcIKXg off=5234755 len=473715 2021-07-31T02:18:41.856 INFO:tasks.workunit.client.0.smithi045.stdout:op 4103 completed, throughput=5MB/sec 2021-07-31T02:18:41.856 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-g5xacuBqqZENuly off=5125240 len=1843048 2021-07-31T02:18:41.867 INFO:tasks.workunit.client.0.smithi045.stdout:op 4104 completed, throughput=5MB/sec 2021-07-31T02:18:41.868 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-2Q8NiJfckZ_KOQg off=178554 len=400442 2021-07-31T02:18:41.872 INFO:tasks.workunit.client.0.smithi045.stdout:op 4105 completed, throughput=5MB/sec 2021-07-31T02:18:41.872 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-4ufwgivbkiTFOo5 off=1343841 len=556565 2021-07-31T02:18:41.873 INFO:tasks.workunit.client.0.smithi045.stdout:op 4106 completed, throughput=5MB/sec 2021-07-31T02:18:42.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:42 smithi154 conmon[34553]: cluster 2021-07-31T02:18:40.241402+0000 mgr.x (mgr.24542) 1350 : cluster 2021-07-31T02:18:42.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:42 smithi154 conmon[34553]: [DBG] pgmap v1815: 9 pgs: 9 active+clean; 127 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 3.5 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2021-07-31T02:18:42.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:42 smithi045 conmon[39718]: cluster 2021-07-31T02:18:40.241402+0000 mgr.x ( 2021-07-31T02:18:42.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:42 smithi045 conmon[39718]: mgr.24542) 1350 : cluster [DBG] pgmap v1815: 9 pgs: 9 active+clean; 127 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 3.5 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2021-07-31T02:18:42.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:42 smithi045 conmon[33655]: cluster 2021-07-31T02:18:40.241402+0000 mgr.x (mgr.24542) 1350 : 2021-07-31T02:18:42.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:42 smithi045 conmon[33655]: cluster [DBG] pgmap v1815: 9 pgs: 9 active+clean; 127 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 3.5 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2021-07-31T02:18:42.873 INFO:tasks.workunit.client.0.smithi045.stdout: 800: throughput=4.99MB/sec pending data=0 2021-07-31T02:18:42.874 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-OewACFv1ZTWhC7u off=11994198 len=1766198 2021-07-31T02:18:42.876 INFO:tasks.workunit.client.0.smithi045.stdout:op 4107 completed, throughput=5MB/sec 2021-07-31T02:18:42.877 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-1zd0Uhk6qKxgcl_ off=2564275 len=858664 2021-07-31T02:18:42.884 INFO:tasks.workunit.client.0.smithi045.stdout:op 4108 completed, throughput=5MB/sec 2021-07-31T02:18:42.884 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-M1nHfVn11b8SNWg off=11197263 len=266534 2021-07-31T02:18:42.887 INFO:tasks.workunit.client.0.smithi045.stdout:op 4109 completed, throughput=5MB/sec 2021-07-31T02:18:42.887 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-edNmQdJjyCtr6GH off=4583089 len=2014008 2021-07-31T02:18:42.896 INFO:tasks.workunit.client.0.smithi045.stdout:op 4110 completed, throughput=5MB/sec 2021-07-31T02:18:42.897 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-1Sn3uo7DvozeE5l off=2577048 len=1958890 2021-07-31T02:18:42.900 INFO:tasks.workunit.client.0.smithi045.stdout:op 4111 completed, throughput=5MB/sec 2021-07-31T02:18:43.900 INFO:tasks.workunit.client.0.smithi045.stdout: 801: throughput=5MB/sec pending data=0 2021-07-31T02:18:43.901 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-uKxEqP1S6JDtWOc off=8882650 len=1949131 2021-07-31T02:18:43.911 INFO:tasks.workunit.client.0.smithi045.stdout:op 4112 completed, throughput=5MB/sec 2021-07-31T02:18:43.911 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-n9hhpUBf_hr0HSS off=8459109 len=2008425 2021-07-31T02:18:43.914 INFO:tasks.workunit.client.0.smithi045.stdout:op 4113 completed, throughput=5MB/sec 2021-07-31T02:18:44.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:18:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:18:43.563Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:18:44.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:18:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:18:43.563Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:18:44.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:18:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:18:43.563Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:18:44.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:44 smithi154 conmon[34553]: cluster 2021-07-31T02:18:42.241973+0000 mgr.x (mgr.24542) 1351 : cluster 2021-07-31T02:18:44.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:44 smithi154 conmon[34553]: [DBG] pgmap v1816: 9 pgs: 9 active+clean; 127 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 2.2 MiB/s rd, 1.8 MiB/s wr, 3 op/s 2021-07-31T02:18:44.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:44 smithi045 conmon[39718]: cluster 2021-07-31T02:18:42.241973+0000 mgr.x (mgr.24542) 1351 : cluster [DBG] pgmap v1816: 9 pgs: 9 active+clean; 127 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 2.2 MiB/s rd, 1.8 MiB/s wr, 3 op/s 2021-07-31T02:18:44.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:44 smithi045 conmon[33655]: cluster 2021-07-31T02:18:42.241973+0000 mgr.x (mgr.24542) 1351 : cluster 2021-07-31T02:18:44.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:44 smithi045 conmon[33655]: [DBG] pgmap v1816: 9 pgs: 9 active+clean; 127 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 2.2 MiB/s rd, 1.8 MiB/s wr, 3 op/s 2021-07-31T02:18:44.916 INFO:tasks.workunit.client.0.smithi045.stdout: 802: throughput=4.99MB/sec pending data=0 2021-07-31T02:18:44.916 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-gqp0PjJMuRfleHV off=16393213 len=196544 2021-07-31T02:18:44.917 INFO:tasks.workunit.client.0.smithi045.stdout:op 4114 completed, throughput=4.99MB/sec 2021-07-31T02:18:44.917 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-3dgT42r_um4614n off=17753841 len=74124 2021-07-31T02:18:44.920 INFO:tasks.workunit.client.0.smithi045.stdout:op 4115 completed, throughput=4.99MB/sec 2021-07-31T02:18:44.920 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-7IqYcyGsylozf9T off=1428111 len=1170472 2021-07-31T02:18:44.926 INFO:tasks.workunit.client.0.smithi045.stdout:op 4116 completed, throughput=5MB/sec 2021-07-31T02:18:44.927 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-clTTtu0PPmxRRi9 off=856809 len=681369 2021-07-31T02:18:44.927 INFO:tasks.workunit.client.0.smithi045.stdout:op 4117 completed, throughput=5MB/sec 2021-07-31T02:18:44.928 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-CONKLo5A35HBKyQ off=7460167 len=1053721 2021-07-31T02:18:44.930 INFO:tasks.workunit.client.0.smithi045.stdout:op 4118 completed, throughput=5MB/sec 2021-07-31T02:18:44.930 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-am_N4QvyIVTfifN off=5847907 len=959659 2021-07-31T02:18:44.932 INFO:tasks.workunit.client.0.smithi045.stdout:op 4119 completed, throughput=5MB/sec 2021-07-31T02:18:44.932 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-EuOlxIuLkbHbtwx off=8325098 len=67218 2021-07-31T02:18:44.935 INFO:tasks.workunit.client.0.smithi045.stdout:op 4120 completed, throughput=5MB/sec 2021-07-31T02:18:44.935 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-UauuAQOk2pv7S0z off=21652516 len=702890 2021-07-31T02:18:44.938 INFO:tasks.workunit.client.0.smithi045.stdout:op 4121 completed, throughput=5MB/sec 2021-07-31T02:18:45.939 INFO:tasks.workunit.client.0.smithi045.stdout: 803: throughput=4.99MB/sec pending data=0 2021-07-31T02:18:45.939 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-9Je3z0f96YeKj97 off=16218390 len=1298802 2021-07-31T02:18:45.940 INFO:tasks.workunit.client.0.smithi045.stdout:op 4122 completed, throughput=5MB/sec 2021-07-31T02:18:45.941 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-akkJpcxWdVw9jVS off=1705629 len=778139 2021-07-31T02:18:45.946 INFO:tasks.workunit.client.0.smithi045.stdout:op 4123 completed, throughput=5MB/sec 2021-07-31T02:18:45.946 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-OGYFcN0m9f_CaXs off=1473244 len=858081 2021-07-31T02:18:45.952 INFO:tasks.workunit.client.0.smithi045.stdout:op 4124 completed, throughput=5MB/sec 2021-07-31T02:18:45.952 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-UVrwu22dj7o061j off=3151928 len=974312 2021-07-31T02:18:45.957 INFO:tasks.workunit.client.0.smithi045.stdout:op 4125 completed, throughput=5MB/sec 2021-07-31T02:18:45.957 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-vKhuK6ZZhSX38JL off=1436863 len=1459620 2021-07-31T02:18:45.961 INFO:tasks.workunit.client.0.smithi045.stdout:op 4126 completed, throughput=5MB/sec 2021-07-31T02:18:46.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:46 smithi154 conmon[34553]: cluster 2021-07-31T02:18:44.242366+0000 mgr.x (mgr.24542) 1352 : cluster [DBG] 2021-07-31T02:18:46.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:46 smithi154 conmon[34553]: pgmap v1817: 9 pgs: 9 active+clean; 127 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 3.5 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2021-07-31T02:18:46.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:46 smithi045 conmon[33655]: cluster 2021-07-31T02:18:44.242366+0000 mgr.x (mgr.24542) 2021-07-31T02:18:46.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:46 smithi045 conmon[33655]: 1352 : cluster [DBG] pgmap v1817: 9 pgs: 9 active+clean; 127 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 3.5 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2021-07-31T02:18:46.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:46 smithi045 conmon[39718]: cluster 2021-07-31T02:18:44.242366+0000 mgr.x ( 2021-07-31T02:18:46.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:46 smithi045 conmon[39718]: mgr.24542) 1352 : cluster [DBG] pgmap v1817: 9 pgs: 9 active+clean; 127 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 3.5 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2021-07-31T02:18:46.962 INFO:tasks.workunit.client.0.smithi045.stdout: 804: throughput=4.99MB/sec pending data=0 2021-07-31T02:18:46.962 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ijGTD1uJ6Lqb9fg off=1005711 len=724264 2021-07-31T02:18:46.967 INFO:tasks.workunit.client.0.smithi045.stdout:op 4127 completed, throughput=4.99MB/sec 2021-07-31T02:18:46.967 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-S6i1fkdZ_DEZ2hZ off=3112609 len=328208 2021-07-31T02:18:46.968 INFO:tasks.workunit.client.0.smithi045.stdout:op 4128 completed, throughput=5MB/sec 2021-07-31T02:18:46.968 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-6T53GqWz6xxb-YR off=578518 len=1704878 2021-07-31T02:18:46.971 INFO:tasks.workunit.client.0.smithi045.stdout:op 4129 completed, throughput=5MB/sec 2021-07-31T02:18:46.971 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-4Z8-t6PRyobQ-SH off=7590229 len=1609101 2021-07-31T02:18:46.979 INFO:tasks.workunit.client.0.smithi045.stdout:op 4130 completed, throughput=5MB/sec 2021-07-31T02:18:46.979 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-sDGJFeCmvnWvY0S off=3680095 len=1599764 2021-07-31T02:18:46.981 INFO:tasks.workunit.client.0.smithi045.stdout:op 4131 completed, throughput=5MB/sec 2021-07-31T02:18:47.982 INFO:tasks.workunit.client.0.smithi045.stdout: 805: throughput=4.99MB/sec pending data=0 2021-07-31T02:18:47.982 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-iq03bItaarHtOYJ off=5660786 len=804013 2021-07-31T02:18:47.986 INFO:tasks.workunit.client.0.smithi045.stdout:op 4132 completed, throughput=5MB/sec 2021-07-31T02:18:47.987 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-eRvhhBzbbHFmJON off=205698 len=1463304 2021-07-31T02:18:47.989 INFO:tasks.workunit.client.0.smithi045.stdout:op 4133 completed, throughput=5MB/sec 2021-07-31T02:18:47.990 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-pd5DFONjbFWpQJc off=6916541 len=942663 2021-07-31T02:18:47.991 INFO:tasks.workunit.client.0.smithi045.stdout:op 4134 completed, throughput=5MB/sec 2021-07-31T02:18:47.991 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-OQJbQWRDeQSNAr8 off=3990816 len=1799381 2021-07-31T02:18:47.993 INFO:tasks.workunit.client.0.smithi045.stdout:op 4135 completed, throughput=5MB/sec 2021-07-31T02:18:48.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:48 smithi154 conmon[34553]: cluster 2021-07-31T02:18:46.242772+0000 mgr.x (mgr.24542) 1353 : cluster [DBG] pgmap v1818: 9 pgs: 9 active+clean; 127 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 2.0 MiB/s rd, 1.9 MiB/s wr, 3 op/s 2021-07-31T02:18:48.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:48 smithi045 conmon[39718]: cluster 2021-07-31T02:18:46.242772+0000 mgr.x (mgr.24542 2021-07-31T02:18:48.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:48 smithi045 conmon[39718]: ) 1353 : cluster [DBG] pgmap v1818: 9 pgs: 9 active+clean; 127 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 2.0 MiB/s rd, 1.9 MiB/s wr, 3 op/s 2021-07-31T02:18:48.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:48 smithi045 conmon[33655]: cluster 2021-07-31T02:18:46.242772+0000 2021-07-31T02:18:48.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:48 smithi045 conmon[33655]: mgr.x (mgr.24542) 1353 : cluster [DBG] pgmap v1818: 9 pgs: 9 active+clean; 127 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 2.0 MiB/s rd, 1.9 MiB/s wr, 3 op/s 2021-07-31T02:18:48.994 INFO:tasks.workunit.client.0.smithi045.stdout: 806: throughput=4.99MB/sec pending data=0 2021-07-31T02:18:48.994 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-BK0EtxhLU1eXXs1 off=9230743 len=2084507 2021-07-31T02:18:48.996 INFO:tasks.workunit.client.0.smithi045.stdout:op 4136 completed, throughput=5MB/sec 2021-07-31T02:18:48.996 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-gQfHNgr-bL69Gtl off=7743493 len=869794 2021-07-31T02:18:48.997 INFO:tasks.workunit.client.0.smithi045.stdout:op 4137 completed, throughput=5MB/sec 2021-07-31T02:18:48.997 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-x9YKLrON4JEZQD3 off=5828677 len=2062232 2021-07-31T02:18:49.008 INFO:tasks.workunit.client.0.smithi045.stdout:op 4138 completed, throughput=5MB/sec 2021-07-31T02:18:50.008 INFO:tasks.workunit.client.0.smithi045.stdout: 807: throughput=4.99MB/sec pending data=0 2021-07-31T02:18:50.009 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-0TV5U3gmh6hWz5I off=7160901 len=2058581 2021-07-31T02:18:50.017 INFO:tasks.workunit.client.0.smithi045.stdout:op 4139 completed, throughput=5MB/sec 2021-07-31T02:18:50.018 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-2-gIQej5nV90LTD off=9393399 len=1508093 2021-07-31T02:18:50.023 INFO:tasks.workunit.client.0.smithi045.stdout:op 4140 completed, throughput=5MB/sec 2021-07-31T02:18:50.023 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-i1S_NMVu419NBN2 off=10041136 len=408247 2021-07-31T02:18:50.024 INFO:tasks.workunit.client.0.smithi045.stdout:op 4141 completed, throughput=5MB/sec 2021-07-31T02:18:50.024 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-hTkLpyql--YukR5 off=147853 len=916845 2021-07-31T02:18:50.027 INFO:tasks.workunit.client.0.smithi045.stdout:op 4142 completed, throughput=5MB/sec 2021-07-31T02:18:50.027 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-exN3tEkKPH-A7rF off=7260337 len=1675637 2021-07-31T02:18:50.036 INFO:tasks.workunit.client.0.smithi045.stdout:op 4143 completed, throughput=5MB/sec 2021-07-31T02:18:50.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:50 smithi154 conmon[34553]: cluster 2021-07-31T02:18:48.243278+0000 mgr.x (mgr.24542) 1354 : cluster [DBG] pgmap v1819: 9 pgs: 9 active+clean; 127 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 2.0 MiB/s rd, 1.9 MiB/s wr, 3 op/s 2021-07-31T02:18:50.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:50 smithi045 conmon[33655]: cluster 2021-07-31T02:18:48.243278+0000 mgr.x (mgr.24542) 1354 : cluster [DBG] pgmap v1819: 9 pgs: 9 active+clean; 127 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 2.0 MiB/s rd, 1.9 MiB/s wr, 3 op/s 2021-07-31T02:18:50.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:50 smithi045 conmon[39718]: cluster 2021-07-31T02:18:48.243278+0000 mgr.x (mgr.24542) 1354 : cluster [DBG] pgmap v1819: 9 pgs: 9 active+clean; 127 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 2.0 MiB/s rd, 1.9 MiB/s wr, 3 op/s 2021-07-31T02:18:51.036 INFO:tasks.workunit.client.0.smithi045.stdout: 808: throughput=5MB/sec pending data=0 2021-07-31T02:18:51.036 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-XRMecFRJj64EEtc off=9739627 len=623280 2021-07-31T02:18:51.038 INFO:tasks.workunit.client.0.smithi045.stdout:op 4144 completed, throughput=5MB/sec 2021-07-31T02:18:51.038 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-wdltX2BGdeUMnfY off=3935801 len=439395 2021-07-31T02:18:51.042 INFO:tasks.workunit.client.0.smithi045.stdout:op 4145 completed, throughput=5MB/sec 2021-07-31T02:18:51.042 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-06HjYjIOmLtwaD3 off=312200 len=1970289 2021-07-31T02:18:51.046 INFO:tasks.workunit.client.0.smithi045.stdout:op 4146 completed, throughput=5MB/sec 2021-07-31T02:18:51.047 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-YXsDfazcRLJe9vs off=553524 len=603367 2021-07-31T02:18:51.051 INFO:tasks.workunit.client.0.smithi045.stdout:op 4147 completed, throughput=5MB/sec 2021-07-31T02:18:51.051 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-DAYNVwkiV58o9CS off=9567226 len=1135684 2021-07-31T02:18:51.056 INFO:tasks.workunit.client.0.smithi045.stdout:op 4148 completed, throughput=5MB/sec 2021-07-31T02:18:52.056 INFO:tasks.workunit.client.0.smithi045.stdout: 809: throughput=4.99MB/sec pending data=0 2021-07-31T02:18:52.057 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-vyqFPqkY_i-Ra0o off=13788265 len=1042546 2021-07-31T02:18:52.059 INFO:tasks.workunit.client.0.smithi045.stdout:op 4149 completed, throughput=5MB/sec 2021-07-31T02:18:52.059 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-HTHRsb8-_nmjPwQ off=12871229 len=562965 2021-07-31T02:18:52.060 INFO:tasks.workunit.client.0.smithi045.stdout:op 4150 completed, throughput=5MB/sec 2021-07-31T02:18:52.060 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-GPFIj6_kV4dfSAa off=3137901 len=1243594 2021-07-31T02:18:52.061 INFO:tasks.workunit.client.0.smithi045.stdout:op 4151 completed, throughput=5MB/sec 2021-07-31T02:18:52.062 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-V7vo0ze4t0xoFZG off=11361251 len=2073440 2021-07-31T02:18:52.065 INFO:tasks.workunit.client.0.smithi045.stdout:op 4152 completed, throughput=5MB/sec 2021-07-31T02:18:52.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:52 smithi154 conmon[34553]: cluster 2021-07-31T02:18:50.243721+0000 mgr.x (mgr.24542) 1355 : cluster [DBG] pgmap v1820: 9 pgs: 9 active+clean; 127 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2021-07-31T02:18:52.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:52 smithi045 conmon[33655]: cluster 2021-07-31T02:18:50.243721+0000 mgr.x (mgr.24542) 1355 : 2021-07-31T02:18:52.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:52 smithi045 conmon[33655]: cluster [DBG] pgmap v1820: 9 pgs: 9 active+clean; 127 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2021-07-31T02:18:52.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:52 smithi045 conmon[39718]: cluster 2021-07-31T02:18:50.243721+0000 mgr.x 2021-07-31T02:18:52.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:52 smithi045 conmon[39718]: (mgr.24542) 1355 : cluster [DBG] pgmap v1820: 9 pgs: 9 active+clean; 127 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2021-07-31T02:18:53.065 INFO:tasks.workunit.client.0.smithi045.stdout: 810: throughput=4.99MB/sec pending data=0 2021-07-31T02:18:53.066 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-7ucbSfRW6E1u3Yy off=3272849 len=1713519 2021-07-31T02:18:53.067 INFO:tasks.workunit.client.0.smithi045.stdout:op 4153 completed, throughput=5MB/sec 2021-07-31T02:18:53.067 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-7QomePAvHxGiZod off=11823598 len=353771 2021-07-31T02:18:53.067 INFO:tasks.workunit.client.0.smithi045.stdout:op 4154 completed, throughput=5MB/sec 2021-07-31T02:18:53.068 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-abCXxpJVvC44Qgi off=2242349 len=546831 2021-07-31T02:18:53.068 INFO:tasks.workunit.client.0.smithi045.stdout:op 4155 completed, throughput=5MB/sec 2021-07-31T02:18:53.069 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-rJ2ZcOmDzM-uxvZ off=10503407 len=2026755 2021-07-31T02:18:53.076 INFO:tasks.workunit.client.0.smithi045.stdout:op 4156 completed, throughput=5MB/sec 2021-07-31T02:18:53.076 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-m-m2YGk57MaBmB4 off=7535297 len=965973 2021-07-31T02:18:53.080 INFO:tasks.workunit.client.0.smithi045.stdout:op 4157 completed, throughput=5MB/sec 2021-07-31T02:18:53.113 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:18:53.390 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:18:53.391 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 33m ago 34m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:18:53.391 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 31m ago 34m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:18:53.391 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 31m ago 37m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:18:53.392 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 33m ago 39m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:18:53.392 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 33m ago 40m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:18:53.392 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 31m ago 38m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:18:53.392 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 33m ago 38m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:18:53.392 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 33m ago 34m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:18:53.393 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 31m ago 34m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:18:53.393 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 33m ago 37m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:18:53.393 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 33m ago 37m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:18:53.393 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 33m ago 36m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:18:53.394 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 33m ago 36m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:18:53.394 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 31m ago 36m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:18:53.394 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 31m ago 35m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:18:53.394 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 31m ago 35m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:18:53.395 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 31m ago 35m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:18:53.395 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 31m ago 34m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:18:53.730 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:18:53.731 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:18:53.731 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:18:53.731 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:18:53.731 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:18:53.732 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:18:53.732 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:18:53.732 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:18:53.732 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:18:53.732 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:18:53.733 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:18:53.733 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:18:53.733 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:18:53.733 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:18:53.734 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:18:53.734 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:18:53.734 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:18:54.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:18:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:18:53.563Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:18:54.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:18:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:18:53.563Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:18:54.027 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:18:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:18:53.563Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:18:54.080 INFO:tasks.workunit.client.0.smithi045.stdout: 811: throughput=4.99MB/sec pending data=0 2021-07-31T02:18:54.081 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-DkAL3JIdBJLpaLf off=11473145 len=218223 2021-07-31T02:18:54.083 INFO:tasks.workunit.client.0.smithi045.stdout:op 4158 completed, throughput=4.99MB/sec 2021-07-31T02:18:54.084 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-hV_P0ar9IYJv-uM off=6879035 len=565597 2021-07-31T02:18:54.087 INFO:tasks.workunit.client.0.smithi045.stdout:op 4159 completed, throughput=5MB/sec 2021-07-31T02:18:54.088 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-5TZh2QsDaDrUJ2A off=5737102 len=1087136 2021-07-31T02:18:54.093 INFO:tasks.workunit.client.0.smithi045.stdout:op 4160 completed, throughput=5MB/sec 2021-07-31T02:18:54.093 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-B9NzWg4EuPiaCZd off=339262 len=1718854 2021-07-31T02:18:54.102 INFO:tasks.workunit.client.0.smithi045.stdout:op 4161 completed, throughput=5MB/sec 2021-07-31T02:18:54.102 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-yVwsPJxi99xmuGU off=15116369 len=352007 2021-07-31T02:18:54.103 INFO:tasks.workunit.client.0.smithi045.stdout:op 4162 completed, throughput=5MB/sec 2021-07-31T02:18:54.103 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-RdSa49muxDe2xN- off=2215596 len=698449 2021-07-31T02:18:54.107 INFO:tasks.workunit.client.0.smithi045.stdout:op 4163 completed, throughput=5MB/sec 2021-07-31T02:18:54.107 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-vSGiOOHoPpe9b3U off=264678 len=83510 2021-07-31T02:18:54.108 INFO:tasks.workunit.client.0.smithi045.stdout:op 4164 completed, throughput=5MB/sec 2021-07-31T02:18:54.108 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-PAZ8hPebgVwQGv3 off=11929351 len=269634 2021-07-31T02:18:54.111 INFO:tasks.workunit.client.0.smithi045.stdout:op 4165 completed, throughput=5MB/sec 2021-07-31T02:18:54.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:54 smithi154 conmon[34553]: cluster 2021-07-31T02:18:52.244273+0000 mgr.x (mgr.24542) 1356 : cluster [DBG] pgmap v1821: 9 pgs: 9 active+clean; 127 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.5 MiB/s wr, 3 op/s 2021-07-31T02:18:54.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:54 smithi154 conmon[34553]: audit 2021-07-31T02:18:53.101367 2021-07-31T02:18:54.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:54 smithi154 conmon[34553]: +0000 mgr.x (mgr.24542) 1357 : audit [DBG] from='client.16203 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:18:54.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:54 smithi154 conmon[34553]: audit 2021-07-31T02:18:53.730633+0000 mon.c (mon.1) 87 : audit [DBG] from='client.? 172.21.15.45:0/917231672' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:18:54.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:54 smithi045 conmon[33655]: cluster 2021-07-31T02:18:52.244273+0000 mgr.x (mgr.24542) 2021-07-31T02:18:54.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:54 smithi045 conmon[33655]: 1356 : cluster [DBG] pgmap v1821: 9 pgs: 9 active+clean; 127 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.5 MiB/s wr, 3 op/s 2021-07-31T02:18:54.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:54 smithi045 conmon[33655]: audit 2021-07-31T02:18:53.101367+0000 mgr.x (mgr.24542) 1357 : audit [DBG] from='client.16203 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:18:54.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:54 smithi045 conmon[33655]: audit 2021-07-31T02:18: 2021-07-31T02:18:54.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:54 smithi045 conmon[33655]: 53.730633+0000 mon.c (mon.1) 87 : audit [DBG] from='client.? 172.21.15.45:0/917231672' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:18:54.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:54 smithi045 conmon[39718]: cluster 2021-07-31T02:18:52.244273+0000 mgr.x (mgr.24542) 1356 : cluster 2021-07-31T02:18:54.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:54 smithi045 conmon[39718]: [DBG] pgmap v1821: 9 pgs: 9 active+clean; 127 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.5 MiB/s wr, 3 op/s 2021-07-31T02:18:54.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:54 smithi045 conmon[39718]: audit 2021-07-31T02:18:53.101367+0000 mgr.x (mgr.24542) 1357 : audit [DBG] from='client.16203 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:18:54.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:54 smithi045 conmon[39718]: audit 2021-07-31T02:18:53.730633+0000 mon.c (mon.1) 87 2021-07-31T02:18:54.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:54 smithi045 conmon[39718]: : audit [DBG] from='client.? 172.21.15.45:0/917231672' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:18:55.111 INFO:tasks.workunit.client.0.smithi045.stdout: 812: throughput=4.99MB/sec pending data=0 2021-07-31T02:18:55.112 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-uXgXzTPfumXhf-1 off=14878584 len=551121 2021-07-31T02:18:55.116 INFO:tasks.workunit.client.0.smithi045.stdout:op 4166 completed, throughput=4.99MB/sec 2021-07-31T02:18:55.116 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-VtSoUBQXJWyexFE off=18541060 len=1377063 2021-07-31T02:18:55.122 INFO:tasks.workunit.client.0.smithi045.stdout:op 4167 completed, throughput=5MB/sec 2021-07-31T02:18:55.123 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-XpMX0_sxpkIDpwu off=5633348 len=1514068 2021-07-31T02:18:55.125 INFO:tasks.workunit.client.0.smithi045.stdout:op 4168 completed, throughput=5MB/sec 2021-07-31T02:18:55.125 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-KKc9ktXY-0JvbWG off=1690072 len=1575243 2021-07-31T02:18:55.127 INFO:tasks.workunit.client.0.smithi045.stdout:op 4169 completed, throughput=5MB/sec 2021-07-31T02:18:55.127 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-mPDV-MR3GOwH0Dc off=10421479 len=1769083 2021-07-31T02:18:55.131 INFO:tasks.workunit.client.0.smithi045.stdout:op 4170 completed, throughput=5MB/sec 2021-07-31T02:18:55.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:55 smithi154 conmon[34553]: audit 2021-07-31T02:18:53.383420+0000 mgr.x (mgr.24542) 1358 : audit [DBG] from='client.16209 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:18:55.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:55 smithi045 conmon[33655]: audit 2021-07-31T02:18:53.383420 2021-07-31T02:18:55.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:55 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 1358 : audit [DBG] from='client.16209 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:18:55.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:55 smithi045 conmon[39718]: audit 2021-07-31T02:18:53.383420+0000 mgr.x (mgr.24542) 1358 : audit [DBG] from='client.16209 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:18:55.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:55 smithi045 conmon[39718]: 2021-07-31T02:18:56.130 INFO:tasks.workunit.client.0.smithi045.stdout: 813: throughput=5MB/sec pending data=0 2021-07-31T02:18:56.131 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-srvN0I5MF1mBkBb off=5014411 len=1296659 2021-07-31T02:18:56.137 INFO:tasks.workunit.client.0.smithi045.stdout:op 4171 completed, throughput=5MB/sec 2021-07-31T02:18:56.138 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ujS-1FxJayK5sJL off=14049297 len=2008780 2021-07-31T02:18:56.141 INFO:tasks.workunit.client.0.smithi045.stdout:op 4172 completed, throughput=5MB/sec 2021-07-31T02:18:56.141 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-4djFz0IQdOuksZr off=6759936 len=1602395 2021-07-31T02:18:56.154 INFO:tasks.workunit.client.0.smithi045.stdout:op 4173 completed, throughput=5MB/sec 2021-07-31T02:18:56.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:56 smithi154 conmon[34553]: cluster 2021-07-31T02:18:54.244892+0000 mgr.x (mgr.24542) 1359 : cluster [DBG] pgmap v1822: 9 pgs: 9 active+clean; 127 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 3.8 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2021-07-31T02:18:56.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:56 smithi045 conmon[39718]: cluster 2021-07-31T02:18:54.244892+0000 mgr.x (mgr.24542) 1359 : cluster [DBG] pgmap v1822: 9 pgs: 9 active+clean; 127 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 3.8 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2021-07-31T02:18:56.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:56 smithi045 conmon[33655]: cluster 2021-07-31T02:18:54.244892+0000 mgr.x (mgr.24542 2021-07-31T02:18:56.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:56 smithi045 conmon[33655]: ) 1359 : cluster [DBG] pgmap v1822: 9 pgs: 9 active+clean; 127 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 3.8 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2021-07-31T02:18:57.154 INFO:tasks.workunit.client.0.smithi045.stdout: 814: throughput=5MB/sec pending data=0 2021-07-31T02:18:57.155 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-X76783qSoNSJGgH off=6334141 len=800142 2021-07-31T02:18:57.159 INFO:tasks.workunit.client.0.smithi045.stdout:op 4174 completed, throughput=5MB/sec 2021-07-31T02:18:57.160 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-cVKjVvano0kqg9r off=17901355 len=908399 2021-07-31T02:18:57.165 INFO:tasks.workunit.client.0.smithi045.stdout:op 4175 completed, throughput=5MB/sec 2021-07-31T02:18:57.165 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-dGi2cwenskgpsy4 off=4414500 len=1852968 2021-07-31T02:18:57.167 INFO:tasks.workunit.client.0.smithi045.stdout:op 4176 completed, throughput=5MB/sec 2021-07-31T02:18:57.167 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-OHGLzbnlAvuD7q2 off=22025188 len=2096174 2021-07-31T02:18:57.172 INFO:tasks.workunit.client.0.smithi045.stdout:op 4177 completed, throughput=5MB/sec 2021-07-31T02:18:58.172 INFO:tasks.workunit.client.0.smithi045.stdout: 815: throughput=5MB/sec pending data=0 2021-07-31T02:18:58.173 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-0ZS6G3yF82pqN-H off=9190524 len=1226951 2021-07-31T02:18:58.175 INFO:tasks.workunit.client.0.smithi045.stdout:op 4178 completed, throughput=5MB/sec 2021-07-31T02:18:58.175 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-t0LekeKL1S0eSt1 off=21777726 len=1353233 2021-07-31T02:18:58.177 INFO:tasks.workunit.client.0.smithi045.stdout:op 4179 completed, throughput=5MB/sec 2021-07-31T02:18:58.177 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-LdTwCzQUPJszmCK off=14148366 len=1945169 2021-07-31T02:18:58.181 INFO:tasks.workunit.client.0.smithi045.stdout:op 4180 completed, throughput=5MB/sec 2021-07-31T02:18:58.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:18:58 smithi154 conmon[34553]: cluster 2021-07-31T02:18:56.245387+0000 mgr.x (mgr.24542) 1360 : cluster [DBG] pgmap v1823: 9 pgs: 9 active+clean; 127 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.7 MiB/s wr, 3 op/s 2021-07-31T02:18:58.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:18:58 smithi045 conmon[33655]: cluster 2021-07-31T02:18:56.245387+0000 mgr.x (mgr.24542) 1360 : cluster [DBG] pgmap v1823: 9 pgs: 9 active+clean; 127 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.7 MiB/s wr, 3 op/s 2021-07-31T02:18:58.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:58 smithi045 conmon[39718]: cluster 2021-07-31T02:18:56.245387+0000 mgr.x ( 2021-07-31T02:18:58.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:18:58 smithi045 conmon[39718]: mgr.24542) 1360 : cluster [DBG] pgmap v1823: 9 pgs: 9 active+clean; 127 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.7 MiB/s wr, 3 op/s 2021-07-31T02:18:59.181 INFO:tasks.workunit.client.0.smithi045.stdout: 816: throughput=4.99MB/sec pending data=0 2021-07-31T02:18:59.182 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-41_g0R-lDdF8BMq off=6949126 len=1793779 2021-07-31T02:18:59.184 INFO:tasks.workunit.client.0.smithi045.stdout:op 4181 completed, throughput=5MB/sec 2021-07-31T02:18:59.185 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-g6kk0nFLpf3GvSd off=201296 len=758185 2021-07-31T02:18:59.190 INFO:tasks.workunit.client.0.smithi045.stdout:op 4182 completed, throughput=5MB/sec 2021-07-31T02:18:59.190 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-cWyKOU36_9ZondI off=15503107 len=2051839 2021-07-31T02:18:59.199 INFO:tasks.workunit.client.0.smithi045.stdout:op 4183 completed, throughput=5MB/sec 2021-07-31T02:19:00.200 INFO:tasks.workunit.client.0.smithi045.stdout: 817: throughput=4.99MB/sec pending data=0 2021-07-31T02:19:00.200 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-IcbWGFoGtNObl5C off=2113066 len=1871194 2021-07-31T02:19:00.209 INFO:tasks.workunit.client.0.smithi045.stdout:op 4184 completed, throughput=5MB/sec 2021-07-31T02:19:00.209 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-BOyqlJAIvztyOXJ off=17635948 len=1922722 2021-07-31T02:19:00.219 INFO:tasks.workunit.client.0.smithi045.stdout:op 4185 completed, throughput=5MB/sec 2021-07-31T02:19:00.220 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-TIuG8fccIuf6-qk off=2669815 len=1697368 2021-07-31T02:19:00.227 INFO:tasks.workunit.client.0.smithi045.stdout:op 4186 completed, throughput=5MB/sec 2021-07-31T02:19:00.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:00 smithi154 conmon[34553]: cluster 2021-07-31T02:18:58.245859+0000 mgr.x (mgr.24542) 1361 : 2021-07-31T02:19:00.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:00 smithi154 conmon[34553]: cluster [DBG] pgmap v1824: 9 pgs: 9 active+clean; 127 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.7 MiB/s wr, 3 op/s 2021-07-31T02:19:00.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:00 smithi045 conmon[39718]: cluster 2021-07-31T02:18:58 2021-07-31T02:19:00.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:00 smithi045 conmon[39718]: .245859+0000 mgr.x (mgr.24542) 1361 : cluster [DBG] pgmap v1824: 9 pgs: 9 active+clean; 127 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.7 MiB/s wr, 3 op/s 2021-07-31T02:19:00.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:00 smithi045 conmon[33655]: cluster 2021-07-31T02:18:58.245859+0000 mgr.x (mgr.24542) 1361 : cluster [DBG] pgmap v1824: 9 pgs: 9 active+clean; 127 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.7 MiB/s wr, 3 op/s 2021-07-31T02:19:01.227 INFO:tasks.workunit.client.0.smithi045.stdout: 818: throughput=4.99MB/sec pending data=0 2021-07-31T02:19:01.228 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-yqmSsBhmQ6bvMeX off=7344872 len=1839842 2021-07-31T02:19:01.239 INFO:tasks.workunit.client.0.smithi045.stdout:op 4187 completed, throughput=5MB/sec 2021-07-31T02:19:01.240 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-NAOxondR2h_fSQE off=10461800 len=718322 2021-07-31T02:19:01.246 INFO:tasks.workunit.client.0.smithi045.stdout:op 4188 completed, throughput=5MB/sec 2021-07-31T02:19:01.246 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-OGYFcN0m9f_CaXs off=1944051 len=13231 2021-07-31T02:19:01.248 INFO:tasks.workunit.client.0.smithi045.stdout:op 4189 completed, throughput=5MB/sec 2021-07-31T02:19:01.249 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-9zRrMHBDgNeJ1PE off=3133705 len=151070 2021-07-31T02:19:01.249 INFO:tasks.workunit.client.0.smithi045.stdout:op 4190 completed, throughput=5MB/sec 2021-07-31T02:19:01.250 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-J-ktnLpCivqvueR off=3409531 len=697240 2021-07-31T02:19:01.251 INFO:tasks.workunit.client.0.smithi045.stdout:op 4191 completed, throughput=5MB/sec 2021-07-31T02:19:01.251 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-02b3Hk5iyseQfgQ off=265663 len=357143 2021-07-31T02:19:01.254 INFO:tasks.workunit.client.0.smithi045.stdout:op 4192 completed, throughput=5MB/sec 2021-07-31T02:19:01.255 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-76kr8GxBpXX4wuY off=8838554 len=384528 2021-07-31T02:19:01.258 INFO:tasks.workunit.client.0.smithi045.stdout:op 4193 completed, throughput=5MB/sec 2021-07-31T02:19:01.258 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-TNqYD7lX2YFurO7 off=1309326 len=1206521 2021-07-31T02:19:01.264 INFO:tasks.workunit.client.0.smithi045.stdout:op 4194 completed, throughput=5MB/sec 2021-07-31T02:19:02.264 INFO:tasks.workunit.client.0.smithi045.stdout: 819: throughput=4.99MB/sec pending data=0 2021-07-31T02:19:02.265 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-SYnktOdCyiaR9T3 off=2456601 len=953907 2021-07-31T02:19:02.266 INFO:tasks.workunit.client.0.smithi045.stdout:op 4195 completed, throughput=5MB/sec 2021-07-31T02:19:02.267 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-5YV24wfuoE0t-Ud off=2864697 len=205007 2021-07-31T02:19:02.267 INFO:tasks.workunit.client.0.smithi045.stdout:op 4196 completed, throughput=5MB/sec 2021-07-31T02:19:02.268 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-GN5XB9y6QMvWom5 off=8827334 len=596318 2021-07-31T02:19:02.269 INFO:tasks.workunit.client.0.smithi045.stdout:op 4197 completed, throughput=5MB/sec 2021-07-31T02:19:02.269 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-q-PTEHpWKGjuTi4 off=5526817 len=813379 2021-07-31T02:19:02.271 INFO:tasks.workunit.client.0.smithi045.stdout:op 4198 completed, throughput=5MB/sec 2021-07-31T02:19:02.271 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-XEaDSh1hQAWbJAP off=992597 len=565034 2021-07-31T02:19:02.272 INFO:tasks.workunit.client.0.smithi045.stdout:op 4199 completed, throughput=5MB/sec 2021-07-31T02:19:02.272 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-0ySURj-RFNGHsi- off=6241798 len=154658 2021-07-31T02:19:02.275 INFO:tasks.workunit.client.0.smithi045.stdout:op 4200 completed, throughput=5MB/sec 2021-07-31T02:19:02.276 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-vzO_X9QphdM3oxA off=842320 len=1031221 2021-07-31T02:19:02.281 INFO:tasks.workunit.client.0.smithi045.stdout:op 4201 completed, throughput=5MB/sec 2021-07-31T02:19:02.281 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-PZHnipfLh75zK-a off=8671185 len=768114 2021-07-31T02:19:02.283 INFO:tasks.workunit.client.0.smithi045.stdout:op 4202 completed, throughput=5MB/sec 2021-07-31T02:19:02.283 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-dYWZolVwaK6Y5He off=3568272 len=1621424 2021-07-31T02:19:02.284 INFO:tasks.workunit.client.0.smithi045.stdout:op 4203 completed, throughput=5MB/sec 2021-07-31T02:19:02.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:02 smithi154 conmon[34553]: cluster 2021-07-31T02:19:00.246379+0000 mgr.x (mgr.24542) 1362 : cluster [DBG] pgmap v1825: 9 pgs: 9 active+clean; 127 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 3.7 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2021-07-31T02:19:02.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:02 smithi045 conmon[33655]: cluster 2021-07-31T02:19:00.246379+0000 mgr.x (mgr.24542) 1362 : cluster 2021-07-31T02:19:02.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:02 smithi045 conmon[33655]: [DBG] pgmap v1825: 9 pgs: 9 active+clean; 127 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 3.7 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2021-07-31T02:19:02.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:02 smithi045 conmon[39718]: cluster 2021-07-31T02:19:00.246379+0000 mgr.x (mgr.24542) 1362 : cluster [DBG] pgmap v1825: 9 pgs: 9 active+clean; 127 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 3.7 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2021-07-31T02:19:03.285 INFO:tasks.workunit.client.0.smithi045.stdout: 820: throughput=5MB/sec pending data=0 2021-07-31T02:19:03.285 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-viOCICuXIQ7jzFS off=5338331 len=226283 2021-07-31T02:19:03.288 INFO:tasks.workunit.client.0.smithi045.stdout:op 4204 completed, throughput=5MB/sec 2021-07-31T02:19:03.289 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-F9kOd5Jhtsg0J6D off=14157624 len=1102304 2021-07-31T02:19:03.295 INFO:tasks.workunit.client.0.smithi045.stdout:op 4205 completed, throughput=5MB/sec 2021-07-31T02:19:03.295 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-3Q0vJa8JVBFjla6 off=8514747 len=1382390 2021-07-31T02:19:03.304 INFO:tasks.workunit.client.0.smithi045.stdout:op 4206 completed, throughput=5MB/sec 2021-07-31T02:19:03.304 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-8XKn9tchfLRJT9Z off=7208899 len=1051651 2021-07-31T02:19:03.306 INFO:tasks.workunit.client.0.smithi045.stdout:op 4207 completed, throughput=5MB/sec 2021-07-31T02:19:03.307 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-E7RXN-G8etsWC_p off=810732 len=1318066 2021-07-31T02:19:03.309 INFO:tasks.workunit.client.0.smithi045.stdout:op 4208 completed, throughput=5MB/sec 2021-07-31T02:19:04.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:19:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:19:03.563Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:19:04.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:19:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:19:03.563Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:19:04.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:19:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:19:03.563Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:19:04.310 INFO:tasks.workunit.client.0.smithi045.stdout: 821: throughput=5MB/sec pending data=0 2021-07-31T02:19:04.310 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-4aS0T0PPktSY7Tr off=2717428 len=1949083 2021-07-31T02:19:04.323 INFO:tasks.workunit.client.0.smithi045.stdout:op 4209 completed, throughput=5MB/sec 2021-07-31T02:19:04.323 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-js4kWNNxDUlJ2Pg off=2085759 len=1252893 2021-07-31T02:19:04.330 INFO:tasks.workunit.client.0.smithi045.stdout:op 4210 completed, throughput=5MB/sec 2021-07-31T02:19:04.330 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-XKTfCFeeWtnMwOC off=2300255 len=1208413 2021-07-31T02:19:04.335 INFO:tasks.workunit.client.0.smithi045.stdout:op 4211 completed, throughput=5MB/sec 2021-07-31T02:19:04.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:04 smithi154 conmon[34553]: cluster 2021-07-31T02:19:02.246826+0000 mgr.x (mgr.24542) 1363 : cluster [DBG] pgmap v1826: 9 pgs: 9 active+clean; 127 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.8 MiB/s wr, 3 op/s 2021-07-31T02:19:04.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:04 smithi045 conmon[39718]: cluster 2021-07-31T02:19:02.246826 2021-07-31T02:19:04.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:04 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 1363 : cluster [DBG] pgmap v1826: 9 pgs: 9 active+clean; 127 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.8 MiB/s wr, 3 op/s 2021-07-31T02:19:04.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:04 smithi045 conmon[33655]: cluster 2021-07-31T02:19:02.246826+0000 mgr.x 2021-07-31T02:19:04.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:04 smithi045 conmon[33655]: (mgr.24542) 1363 : cluster [DBG] pgmap v1826: 9 pgs: 9 active+clean; 127 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.8 MiB/s wr, 3 op/s 2021-07-31T02:19:05.335 INFO:tasks.workunit.client.0.smithi045.stdout: 822: throughput=4.99MB/sec pending data=0 2021-07-31T02:19:05.335 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-mMYMR2ezniPSey6 off=11878265 len=839107 2021-07-31T02:19:05.337 INFO:tasks.workunit.client.0.smithi045.stdout:op 4212 completed, throughput=5MB/sec 2021-07-31T02:19:05.338 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj--gH7UJfSGLlT0-s off=541978 len=1871181 2021-07-31T02:19:05.347 INFO:tasks.workunit.client.0.smithi045.stdout:op 4213 completed, throughput=5MB/sec 2021-07-31T02:19:05.347 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-BcEUax6z6Z3DIhJ off=14504919 len=530908 2021-07-31T02:19:05.348 INFO:tasks.workunit.client.0.smithi045.stdout:op 4214 completed, throughput=5MB/sec 2021-07-31T02:19:05.348 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Jxou8PjORsBIqhk off=4615261 len=1990142 2021-07-31T02:19:05.352 INFO:tasks.workunit.client.0.smithi045.stdout:op 4215 completed, throughput=5MB/sec 2021-07-31T02:19:06.352 INFO:tasks.workunit.client.0.smithi045.stdout: 823: throughput=4.99MB/sec pending data=0 2021-07-31T02:19:06.353 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-cENGrVgkAVpSt6S off=1952865 len=1219328 2021-07-31T02:19:06.355 INFO:tasks.workunit.client.0.smithi045.stdout:op 4216 completed, throughput=5MB/sec 2021-07-31T02:19:06.355 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-NlpsX0czr0xb0zh off=2378702 len=381117 2021-07-31T02:19:06.356 INFO:tasks.workunit.client.0.smithi045.stdout:op 4217 completed, throughput=5MB/sec 2021-07-31T02:19:06.357 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-BQojtFSdAyH6W5a off=142636 len=1927855 2021-07-31T02:19:06.368 INFO:tasks.workunit.client.0.smithi045.stdout:op 4218 completed, throughput=5MB/sec 2021-07-31T02:19:06.368 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-88BTvXPlA4VEEYW off=20124657 len=1342990 2021-07-31T02:19:06.371 INFO:tasks.workunit.client.0.smithi045.stdout:op 4219 completed, throughput=5MB/sec 2021-07-31T02:19:06.371 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-tNkGWqEwAQeujZt off=6189766 len=1876661 2021-07-31T02:19:06.373 INFO:tasks.workunit.client.0.smithi045.stdout:op 4220 completed, throughput=5MB/sec 2021-07-31T02:19:06.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:06 smithi154 conmon[34553]: cluster 2021-07-31T02:19:04.247353+0000 mgr.x (mgr.24542) 1364 : cluster [DBG] pgmap v1827: 9 pgs: 9 active+clean; 127 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 3.4 MiB/s rd, 3.1 MiB/s wr, 6 op/s 2021-07-31T02:19:06.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:06 smithi045 conmon[39718]: cluster 2021-07-31T02:19:04.247353+0000 mgr.x (mgr.24542 2021-07-31T02:19:06.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:06 smithi045 conmon[39718]: ) 1364 : cluster [DBG] pgmap v1827: 9 pgs: 9 active+clean; 127 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 3.4 MiB/s rd, 3.1 MiB/s wr, 6 op/s 2021-07-31T02:19:06.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:06 smithi045 conmon[33655]: cluster 2021-07-31T02:19:06.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:06 smithi045 conmon[33655]: 2021-07-31T02:19:04.247353+0000 mgr.x (mgr.24542) 1364 : cluster [DBG] 2021-07-31T02:19:06.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:06 smithi045 conmon[33655]: pgmap v1827: 9 pgs: 9 active+clean; 127 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 3.4 MiB/s rd, 3.1 MiB/s wr, 6 op/s 2021-07-31T02:19:07.373 INFO:tasks.workunit.client.0.smithi045.stdout: 824: throughput=5MB/sec pending data=0 2021-07-31T02:19:07.374 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-H90ANHaunm2XwSI off=9004579 len=300688 2021-07-31T02:19:07.374 INFO:tasks.workunit.client.0.smithi045.stdout:op 4221 completed, throughput=5MB/sec 2021-07-31T02:19:07.375 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-YUwHDqqqBTigkwK off=3017532 len=1835683 2021-07-31T02:19:07.378 INFO:tasks.workunit.client.0.smithi045.stdout:op 4222 completed, throughput=5MB/sec 2021-07-31T02:19:07.378 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-i8zkis1GuEV20zY off=2559267 len=1772291 2021-07-31T02:19:07.380 INFO:tasks.workunit.client.0.smithi045.stdout:op 4223 completed, throughput=5MB/sec 2021-07-31T02:19:08.380 INFO:tasks.workunit.client.0.smithi045.stdout: 825: throughput=4.99MB/sec pending data=0 2021-07-31T02:19:08.381 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-x3W9bR8aN7-Q3sq off=4714347 len=2026969 2021-07-31T02:19:08.391 INFO:tasks.workunit.client.0.smithi045.stdout:op 4224 completed, throughput=5MB/sec 2021-07-31T02:19:08.391 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-GwgruxmEzCB5O-T off=3866345 len=1497344 2021-07-31T02:19:08.393 INFO:tasks.workunit.client.0.smithi045.stdout:op 4225 completed, throughput=5MB/sec 2021-07-31T02:19:08.393 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-wqnABBqfSBKFNg4 off=3548092 len=1703527 2021-07-31T02:19:08.401 INFO:tasks.workunit.client.0.smithi045.stdout:op 4226 completed, throughput=5MB/sec 2021-07-31T02:19:08.401 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-TJZYYBdt-q7KrNi off=1749661 len=403325 2021-07-31T02:19:08.402 INFO:tasks.workunit.client.0.smithi045.stdout:op 4227 completed, throughput=5MB/sec 2021-07-31T02:19:08.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:08 smithi154 conmon[34553]: cluster 2021-07-31T02:19:06.247888+0000 mgr.x (mgr.24542) 1365 : cluster [DBG] pgmap v1828: 9 pgs: 9 active+clean; 127 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2021-07-31T02:19:08.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:08 smithi045 conmon[33655]: cluster 2021-07-31T02:19:06.247888+0000 mgr.x (mgr.24542) 1365 : cluster 2021-07-31T02:19:08.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:08 smithi045 conmon[33655]: [DBG] pgmap v1828: 9 pgs: 9 active+clean; 127 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2021-07-31T02:19:08.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:08 smithi045 conmon[39718]: cluster 2021-07-31T02:19:06.247888+0000 mgr.x (mgr. 2021-07-31T02:19:08.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:08 smithi045 conmon[39718]: 24542) 1365 : cluster [DBG] pgmap v1828: 9 pgs: 9 active+clean; 127 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2021-07-31T02:19:09.403 INFO:tasks.workunit.client.0.smithi045.stdout: 826: throughput=4.99MB/sec pending data=0 2021-07-31T02:19:09.403 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-L93DdCDLxw6A4lq off=5288954 len=721534 2021-07-31T02:19:09.406 INFO:tasks.workunit.client.0.smithi045.stdout:op 4228 completed, throughput=5MB/sec 2021-07-31T02:19:09.406 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-GdThmyeBEcCV02X off=1963795 len=247753 2021-07-31T02:19:09.411 INFO:tasks.workunit.client.0.smithi045.stdout:op 4229 completed, throughput=5MB/sec 2021-07-31T02:19:09.412 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-juz6_fsDzKFPsp7 off=9944559 len=265365 2021-07-31T02:19:09.412 INFO:tasks.workunit.client.0.smithi045.stdout:op 4230 completed, throughput=5MB/sec 2021-07-31T02:19:09.412 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-M585E0fCt2VFqqg off=1508296 len=1167336 2021-07-31T02:19:09.412 INFO:tasks.workunit.client.0.smithi045.stdout:op 4231 completed, throughput=5MB/sec 2021-07-31T02:19:09.413 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ETD9zOpyZNtsbl8 off=10318916 len=864920 2021-07-31T02:19:09.414 INFO:tasks.workunit.client.0.smithi045.stdout:op 4232 completed, throughput=5MB/sec 2021-07-31T02:19:09.414 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-CBbQre7N90r9xLm off=6074027 len=1563944 2021-07-31T02:19:09.422 INFO:tasks.workunit.client.0.smithi045.stdout:op 4233 completed, throughput=5MB/sec 2021-07-31T02:19:09.422 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-OwcuARl_2VS3qn1 off=3289454 len=196821 2021-07-31T02:19:09.425 INFO:tasks.workunit.client.0.smithi045.stdout:op 4234 completed, throughput=5MB/sec 2021-07-31T02:19:10.425 INFO:tasks.workunit.client.0.smithi045.stdout: 827: throughput=4.99MB/sec pending data=0 2021-07-31T02:19:10.426 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-08yrXYxLL6e2_ci off=15028854 len=1648202 2021-07-31T02:19:10.432 INFO:tasks.workunit.client.0.smithi045.stdout:op 4235 completed, throughput=5MB/sec 2021-07-31T02:19:10.433 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-9US_GG6IRjh1oLr off=1998207 len=1994388 2021-07-31T02:19:10.433 INFO:tasks.workunit.client.0.smithi045.stdout:op 4236 completed, throughput=5MB/sec 2021-07-31T02:19:10.434 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-u5gyWKny5iFn5pe off=2149837 len=838249 2021-07-31T02:19:10.435 INFO:tasks.workunit.client.0.smithi045.stdout:op 4237 completed, throughput=5MB/sec 2021-07-31T02:19:10.435 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-LFPZL2358oQgNAP off=694484 len=312851 2021-07-31T02:19:10.436 INFO:tasks.workunit.client.0.smithi045.stdout:op 4238 completed, throughput=5MB/sec 2021-07-31T02:19:10.436 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-dwCz9x2a483fg5b off=12571745 len=1963410 2021-07-31T02:19:10.439 INFO:tasks.workunit.client.0.smithi045.stdout:op 4239 completed, throughput=5MB/sec 2021-07-31T02:19:10.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:10 smithi154 conmon[34553]: cluster 2021-07-31T02:19:08.248345+0000 mgr.x (mgr.24542) 1366 : cluster [DBG] pgmap v1829: 9 pgs: 9 active+clean; 127 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2021-07-31T02:19:10.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:10 smithi045 conmon[39718]: cluster 2021-07-31T02:19:08.248345+0000 mgr.x (mgr 2021-07-31T02:19:10.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:10 smithi045 conmon[39718]: .24542) 1366 : cluster [DBG] pgmap v1829: 9 pgs: 9 active+clean; 127 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2021-07-31T02:19:10.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:10 smithi045 conmon[33655]: cluster 2021-07-31T02:19:08.248345+0000 mgr.x (mgr.24542) 1366 : cluster [DBG] 2021-07-31T02:19:10.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:10 smithi045 conmon[33655]: pgmap v1829: 9 pgs: 9 active+clean; 127 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2021-07-31T02:19:11.440 INFO:tasks.workunit.client.0.smithi045.stdout: 828: throughput=5MB/sec pending data=0 2021-07-31T02:19:11.440 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-GRbYUXfI_IaGDSX off=4130583 len=959625 2021-07-31T02:19:11.442 INFO:tasks.workunit.client.0.smithi045.stdout:op 4240 completed, throughput=5MB/sec 2021-07-31T02:19:11.442 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-nuP2pGc5NuB1UTr off=9712902 len=1654316 2021-07-31T02:19:11.445 INFO:tasks.workunit.client.0.smithi045.stdout:op 4241 completed, throughput=5MB/sec 2021-07-31T02:19:11.445 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-QWYmXhsFNee0STQ off=177899 len=1067877 2021-07-31T02:19:11.446 INFO:tasks.workunit.client.0.smithi045.stdout:op 4242 completed, throughput=5MB/sec 2021-07-31T02:19:11.447 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-NdwzCnpQJFsG531 off=16202900 len=1562974 2021-07-31T02:19:11.448 INFO:tasks.workunit.client.0.smithi045.stdout:op 4243 completed, throughput=5MB/sec 2021-07-31T02:19:12.449 INFO:tasks.workunit.client.0.smithi045.stdout: 829: throughput=5MB/sec pending data=0 2021-07-31T02:19:12.449 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-HefZVh47PbmVY7w off=624672 len=1957662 2021-07-31T02:19:12.451 INFO:tasks.workunit.client.0.smithi045.stdout:op 4244 completed, throughput=5MB/sec 2021-07-31T02:19:12.452 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-WpjjRETx5Dq8QdV off=4648407 len=707101 2021-07-31T02:19:12.453 INFO:tasks.workunit.client.0.smithi045.stdout:op 4245 completed, throughput=5MB/sec 2021-07-31T02:19:12.453 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-kVY5t3AlyBG0EJ4 off=1267771 len=1432327 2021-07-31T02:19:12.455 INFO:tasks.workunit.client.0.smithi045.stdout:op 4246 completed, throughput=5MB/sec 2021-07-31T02:19:12.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:12 smithi154 conmon[34553]: cluster 2021-07-31T02:19:10.249011+0000 mgr.x (mgr.24542) 1367 : cluster [DBG] pgmap v1830: 9 pgs: 9 active+clean; 127 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 3.2 MiB/s rd, 3.2 MiB/s wr, 5 op/s 2021-07-31T02:19:12.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:12 smithi045 conmon[33655]: cluster 2021-07-31T02:19:10.249011+0000 mgr.x (mgr.24542) 1367 : cluster [DBG] 2021-07-31T02:19:12.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:12 smithi045 conmon[33655]: pgmap v1830: 9 pgs: 9 active+clean; 127 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 3.2 MiB/s rd, 3.2 MiB/s wr, 5 op/s 2021-07-31T02:19:12.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:12 smithi045 conmon[39718]: cluster 2021-07-31T02:19:10.249011 2021-07-31T02:19:12.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:12 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 1367 : cluster [DBG] pgmap v1830: 9 pgs: 9 active+clean; 127 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 3.2 MiB/s rd, 3.2 MiB/s wr, 5 op/s 2021-07-31T02:19:13.456 INFO:tasks.workunit.client.0.smithi045.stdout: 830: throughput=4.99MB/sec pending data=0 2021-07-31T02:19:13.456 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-YVHw8qXu9AhMULM off=10068248 len=153938 2021-07-31T02:19:13.459 INFO:tasks.workunit.client.0.smithi045.stdout:op 4247 completed, throughput=4.99MB/sec 2021-07-31T02:19:13.460 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-q6vvCIDYwR3tINX off=0 len=1506357 2021-07-31T02:19:13.462 INFO:tasks.workunit.client.0.smithi045.stdout:op 4248 completed, throughput=5MB/sec 2021-07-31T02:19:13.462 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-lCLhrq-hlH7xKQL off=11401758 len=1625333 2021-07-31T02:19:13.465 INFO:tasks.workunit.client.0.smithi045.stdout:op 4249 completed, throughput=5MB/sec 2021-07-31T02:19:13.465 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-gojq97kEEKgqQzW off=1739133 len=234180 2021-07-31T02:19:13.466 INFO:tasks.workunit.client.0.smithi045.stdout:op 4250 completed, throughput=5MB/sec 2021-07-31T02:19:13.467 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-h0Y5I_9O_LX1yNy off=8082517 len=909881 2021-07-31T02:19:13.468 INFO:tasks.workunit.client.0.smithi045.stdout:op 4251 completed, throughput=5MB/sec 2021-07-31T02:19:13.470 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-qOynAE-4NRh-o7h off=3113737 len=1809388 2021-07-31T02:19:13.478 INFO:tasks.workunit.client.0.smithi045.stdout:op 4252 completed, throughput=5MB/sec 2021-07-31T02:19:14.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:19:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:19:13.563Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:19:14.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:19:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:19:13.563Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:19:14.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:19:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:19:13.563Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:19:14.478 INFO:tasks.workunit.client.0.smithi045.stdout: 831: throughput=5MB/sec pending data=0 2021-07-31T02:19:14.479 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-1nmK124TLQUOX4h off=9889392 len=2034136 2021-07-31T02:19:14.489 INFO:tasks.workunit.client.0.smithi045.stdout:op 4253 completed, throughput=5MB/sec 2021-07-31T02:19:14.489 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-npVe5GyIt6-fIEC off=8774984 len=1971328 2021-07-31T02:19:14.498 INFO:tasks.workunit.client.0.smithi045.stdout:op 4254 completed, throughput=5MB/sec 2021-07-31T02:19:14.498 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-1EYyBaaTsAawyva off=1793030 len=1842030 2021-07-31T02:19:14.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:14 smithi154 conmon[34553]: cluster 2021-07-31T02:19:12.249546+0000 mgr.x (mgr.24542) 1368 : cluster [DBG] pgmap v1831: 9 pgs: 9 active+clean; 127 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 3 op/s 2021-07-31T02:19:14.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:14 smithi154 conmon[34553]: 2021-07-31T02:19:14.507 INFO:tasks.workunit.client.0.smithi045.stdout:op 4255 completed, throughput=5MB/sec 2021-07-31T02:19:14.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:14 smithi045 conmon[33655]: cluster 2021-07-31T02:19:12.249546+0000 mgr.x (mgr.24542 2021-07-31T02:19:14.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:14 smithi045 conmon[33655]: ) 1368 : cluster [DBG] pgmap v1831: 9 pgs: 9 active+clean; 127 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 3 op/s 2021-07-31T02:19:14.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:14 smithi045 conmon[39718]: cluster 2021-07-31T02:19:12.249546+0000 mgr.x (mgr.24542) 1368 2021-07-31T02:19:14.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:14 smithi045 conmon[39718]: : cluster [DBG] pgmap v1831: 9 pgs: 9 active+clean; 127 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 3 op/s 2021-07-31T02:19:15.508 INFO:tasks.workunit.client.0.smithi045.stdout: 832: throughput=5MB/sec pending data=0 2021-07-31T02:19:15.508 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-df1wFR_PJUPwwSk off=5708766 len=398998 2021-07-31T02:19:15.509 INFO:tasks.workunit.client.0.smithi045.stdout:op 4256 completed, throughput=5MB/sec 2021-07-31T02:19:15.510 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-DSVUTF_txBFGJns off=11583766 len=1277237 2021-07-31T02:19:15.512 INFO:tasks.workunit.client.0.smithi045.stdout:op 4257 completed, throughput=5MB/sec 2021-07-31T02:19:15.512 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-cJDkURvisfe4FbB off=921048 len=563477 2021-07-31T02:19:15.516 INFO:tasks.workunit.client.0.smithi045.stdout:op 4258 completed, throughput=5MB/sec 2021-07-31T02:19:15.517 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-9MVgMW3qzq7tzw5 off=1029560 len=208720 2021-07-31T02:19:15.519 INFO:tasks.workunit.client.0.smithi045.stdout:op 4259 completed, throughput=5MB/sec 2021-07-31T02:19:15.520 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-R3O4vpocoI3_yu9 off=6577861 len=1153088 2021-07-31T02:19:15.526 INFO:tasks.workunit.client.0.smithi045.stdout:op 4260 completed, throughput=5MB/sec 2021-07-31T02:19:15.526 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-A8Uj5rJFHWoU9xU off=6876997 len=1191106 2021-07-31T02:19:15.532 INFO:tasks.workunit.client.0.smithi045.stdout:op 4261 completed, throughput=5MB/sec 2021-07-31T02:19:16.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:16 smithi154 conmon[34553]: cluster 2021-07-31T02:19:14.250168+0000 mgr.x (mgr.24542) 1369 : cluster [DBG] pgmap v1832: 9 pgs: 9 active+clean; 127 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 3.6 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2021-07-31T02:19:16.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:16 smithi045 conmon[39718]: cluster 2021-07-31T02:19:14.250168+0000 mgr.x (mgr 2021-07-31T02:19:16.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:16 smithi045 conmon[39718]: .24542) 1369 : cluster [DBG] pgmap v1832: 9 pgs: 9 active+clean; 127 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 3.6 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2021-07-31T02:19:16.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:16 smithi045 conmon[33655]: cluster 2021-07-31T02:19:14.250168+0000 mgr.x (mgr.24542) 2021-07-31T02:19:16.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:16 smithi045 conmon[33655]: 1369 : cluster [DBG] pgmap v1832: 9 pgs: 9 active+clean; 127 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 3.6 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2021-07-31T02:19:16.532 INFO:tasks.workunit.client.0.smithi045.stdout: 833: throughput=5MB/sec pending data=0 2021-07-31T02:19:16.532 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-JFU3RZwRymRfVRl off=2211329 len=1865529 2021-07-31T02:19:16.536 INFO:tasks.workunit.client.0.smithi045.stdout:op 4262 completed, throughput=5MB/sec 2021-07-31T02:19:16.536 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-MFcK1LOazb5C8JF off=13237914 len=420864 2021-07-31T02:19:16.537 INFO:tasks.workunit.client.0.smithi045.stdout:op 4263 completed, throughput=5MB/sec 2021-07-31T02:19:16.538 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-9B-Qb7v5eYmzR6w off=6968170 len=1886888 2021-07-31T02:19:16.540 INFO:tasks.workunit.client.0.smithi045.stdout:op 4264 completed, throughput=5MB/sec 2021-07-31T02:19:16.541 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-2UQmooSkCnMSLjT off=5975956 len=922462 2021-07-31T02:19:16.542 INFO:tasks.workunit.client.0.smithi045.stdout:op 4265 completed, throughput=5MB/sec 2021-07-31T02:19:17.542 INFO:tasks.workunit.client.0.smithi045.stdout: 834: throughput=4.99MB/sec pending data=0 2021-07-31T02:19:17.543 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-lcSmzX8iW33oZmi off=8761915 len=334873 2021-07-31T02:19:17.544 INFO:tasks.workunit.client.0.smithi045.stdout:op 4266 completed, throughput=5MB/sec 2021-07-31T02:19:17.544 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-jxE52zNWktJ6DEz off=14680923 len=520175 2021-07-31T02:19:17.545 INFO:tasks.workunit.client.0.smithi045.stdout:op 4267 completed, throughput=5MB/sec 2021-07-31T02:19:17.545 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-T9NUgUrta9B_9SH off=17135290 len=2062468 2021-07-31T02:19:17.548 INFO:tasks.workunit.client.0.smithi045.stdout:op 4268 completed, throughput=5MB/sec 2021-07-31T02:19:17.549 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Y-qhJP50Kdh-WqA off=681212 len=982567 2021-07-31T02:19:17.551 INFO:tasks.workunit.client.0.smithi045.stdout:op 4269 completed, throughput=5MB/sec 2021-07-31T02:19:17.551 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-chhLthGUuXNzjLz off=14848203 len=499836 2021-07-31T02:19:17.556 INFO:tasks.workunit.client.0.smithi045.stdout:op 4270 completed, throughput=5MB/sec 2021-07-31T02:19:17.556 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-jyDhpbSfqFExuSI off=11628509 len=1926831 2021-07-31T02:19:17.559 INFO:tasks.workunit.client.0.smithi045.stdout:op 4271 completed, throughput=5MB/sec 2021-07-31T02:19:18.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:18 smithi154 conmon[34553]: cluster 2021-07-31T02:19:16.250619+0000 mgr.x (mgr.24542) 1370 : cluster [DBG] pgmap v1833: 9 pgs: 9 active+clean; 127 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 2.7 MiB/s rd, 1.5 MiB/s wr, 3 op/s 2021-07-31T02:19:18.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:18 smithi045 conmon[39718]: cluster 2021-07-31T02:19:16.250619+0000 mgr.x (mgr.24542 2021-07-31T02:19:18.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:18 smithi045 conmon[39718]: ) 1370 : cluster [DBG] pgmap v1833: 9 pgs: 9 active+clean; 127 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 2.7 MiB/s rd, 1.5 MiB/s wr, 3 op/s 2021-07-31T02:19:18.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:18 smithi045 conmon[33655]: cluster 2021-07-31T02:19:16.250619+0000 mgr.x (mgr.24542) 1370 : cluster [DBG] pgmap v1833: 9 pgs: 9 active+clean; 127 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 2.7 MiB/s rd, 1.5 MiB/s wr, 3 op/s 2021-07-31T02:19:18.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:18 smithi045 conmon[33655]: 2021-07-31T02:19:18.559 INFO:tasks.workunit.client.0.smithi045.stdout: 835: throughput=5MB/sec pending data=0 2021-07-31T02:19:18.560 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-PIl7l4IYArGbXNM off=7494187 len=737387 2021-07-31T02:19:18.564 INFO:tasks.workunit.client.0.smithi045.stdout:op 4272 completed, throughput=5MB/sec 2021-07-31T02:19:18.565 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-M6N_cWyTSKEEWAM off=9425971 len=857310 2021-07-31T02:19:18.566 INFO:tasks.workunit.client.0.smithi045.stdout:op 4273 completed, throughput=5MB/sec 2021-07-31T02:19:18.567 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-cd1PuKFTw9-a7Yt off=3405984 len=814497 2021-07-31T02:19:18.572 INFO:tasks.workunit.client.0.smithi045.stdout:op 4274 completed, throughput=5MB/sec 2021-07-31T02:19:18.572 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-sfXA1YOf9avE3x4 off=1826464 len=967486 2021-07-31T02:19:18.573 INFO:tasks.workunit.client.0.smithi045.stdout:op 4275 completed, throughput=5MB/sec 2021-07-31T02:19:18.574 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-_wZN3Hq0jHOXN4R off=19030075 len=607189 2021-07-31T02:19:18.575 INFO:tasks.workunit.client.0.smithi045.stdout:op 4276 completed, throughput=5MB/sec 2021-07-31T02:19:19.575 INFO:tasks.workunit.client.0.smithi045.stdout: 836: throughput=4.99MB/sec pending data=0 2021-07-31T02:19:19.576 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-4jVbgslFS3slk51 off=5151755 len=1780399 2021-07-31T02:19:19.577 INFO:tasks.workunit.client.0.smithi045.stdout:op 4277 completed, throughput=5MB/sec 2021-07-31T02:19:19.577 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-3ImH4H4MDi2mSFF off=7817187 len=1531152 2021-07-31T02:19:19.580 INFO:tasks.workunit.client.0.smithi045.stdout:op 4278 completed, throughput=5MB/sec 2021-07-31T02:19:19.580 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-LsGt7pzkhCxOugz off=5448170 len=707309 2021-07-31T02:19:19.581 INFO:tasks.workunit.client.0.smithi045.stdout:op 4279 completed, throughput=5MB/sec 2021-07-31T02:19:19.581 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-JVYgLaP4BP-3Dqk off=336456 len=1311705 2021-07-31T02:19:19.587 INFO:tasks.workunit.client.0.smithi045.stdout:op 4280 completed, throughput=5MB/sec 2021-07-31T02:19:20.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:20 smithi154 conmon[34553]: cluster 2021-07-31T02:19:18.251142+0000 mgr.x (mgr.24542) 1371 : cluster [DBG] pgmap v1834: 9 pgs: 9 active+clean; 127 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 2.7 MiB/s rd, 1.5 MiB/s wr, 3 op/s 2021-07-31T02:19:20.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:20 smithi045 conmon[33655]: cluster 2021-07-31T02:19:18.251142+0000 mgr.x (mgr.24542) 1371 : cluster [DBG] pgmap v1834: 9 pgs: 9 active+clean; 127 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 2.7 MiB/s rd, 1.5 MiB/s wr, 3 op/s 2021-07-31T02:19:20.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:20 smithi045 conmon[39718]: cluster 2021-07-31T02:19:18.251142+0000 mgr.x (mgr. 2021-07-31T02:19:20.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:20 smithi045 conmon[39718]: 24542) 1371 : cluster [DBG] pgmap v1834: 9 pgs: 9 active+clean; 127 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 2.7 MiB/s rd, 1.5 MiB/s wr, 3 op/s 2021-07-31T02:19:20.587 INFO:tasks.workunit.client.0.smithi045.stdout: 837: throughput=4.99MB/sec pending data=0 2021-07-31T02:19:20.588 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-EoNGeSelxURWflt off=708567 len=154528 2021-07-31T02:19:20.591 INFO:tasks.workunit.client.0.smithi045.stdout:op 4281 completed, throughput=4.99MB/sec 2021-07-31T02:19:20.591 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-6EKTjX9cYW_t4eO off=2080654 len=411558 2021-07-31T02:19:20.592 INFO:tasks.workunit.client.0.smithi045.stdout:op 4282 completed, throughput=5MB/sec 2021-07-31T02:19:20.592 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-LHh6Y_J25WkhaaC off=263292 len=751203 2021-07-31T02:19:20.596 INFO:tasks.workunit.client.0.smithi045.stdout:op 4283 completed, throughput=5MB/sec 2021-07-31T02:19:20.597 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Fa_6DUfSax93QRR off=547085 len=1298646 2021-07-31T02:19:20.602 INFO:tasks.workunit.client.0.smithi045.stdout:op 4284 completed, throughput=5MB/sec 2021-07-31T02:19:20.602 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-W15fJfrLKcbb57O off=256078 len=1417547 2021-07-31T02:19:20.609 INFO:tasks.workunit.client.0.smithi045.stdout:op 4285 completed, throughput=5MB/sec 2021-07-31T02:19:20.609 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-1V6B6NdG9qOXGBr off=9875961 len=135977 2021-07-31T02:19:20.614 INFO:tasks.workunit.client.0.smithi045.stdout:op 4286 completed, throughput=5MB/sec 2021-07-31T02:19:20.614 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-nIry27XHyhql2X8 off=4202278 len=1505752 2021-07-31T02:19:20.621 INFO:tasks.workunit.client.0.smithi045.stdout:op 4287 completed, throughput=5MB/sec 2021-07-31T02:19:21.621 INFO:tasks.workunit.client.0.smithi045.stdout: 838: throughput=4.99MB/sec pending data=0 2021-07-31T02:19:21.621 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-62fxJ1T7LMYwU-8 off=5315267 len=871190 2021-07-31T02:19:21.626 INFO:tasks.workunit.client.0.smithi045.stdout:op 4288 completed, throughput=5MB/sec 2021-07-31T02:19:21.626 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-XUatAoyDM8glY6j off=4524567 len=1653619 2021-07-31T02:19:21.628 INFO:tasks.workunit.client.0.smithi045.stdout:op 4289 completed, throughput=5MB/sec 2021-07-31T02:19:21.628 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-C_GP364fKeBPeHO off=10264059 len=66480 2021-07-31T02:19:21.629 INFO:tasks.workunit.client.0.smithi045.stdout:op 4290 completed, throughput=5MB/sec 2021-07-31T02:19:21.629 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-x1T9Ve-_IoT8xJB off=5690549 len=1427964 2021-07-31T02:19:21.636 INFO:tasks.workunit.client.0.smithi045.stdout:op 4291 completed, throughput=5MB/sec 2021-07-31T02:19:21.636 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-h2363S7q_mHplma off=65189 len=1939085 2021-07-31T02:19:21.648 INFO:tasks.workunit.client.0.smithi045.stdout:op 4292 completed, throughput=5MB/sec 2021-07-31T02:19:21.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:21 smithi154 conmon[34553]: cluster 2021-07-31T02:19:20.251501+0000 mgr.x (mgr.24542) 1372 : cluster [DBG] pgmap v1835: 9 pgs: 9 active+clean; 127 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 3.9 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2021-07-31T02:19:21.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:21 smithi045 conmon[33655]: cluster 2021-07-31T02:19:20.251501 2021-07-31T02:19:21.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:21 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 1372 : cluster [DBG] pgmap v1835: 9 pgs: 9 active+clean; 127 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 3.9 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2021-07-31T02:19:21.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:21 smithi045 conmon[39718]: cluster 2021-07-31T02:19:20.251501+0000 mgr.x (mgr.24542) 2021-07-31T02:19:21.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:21 smithi045 conmon[39718]: 1372 : cluster [DBG] pgmap v1835: 9 pgs: 9 active+clean; 127 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 3.9 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2021-07-31T02:19:22.648 INFO:tasks.workunit.client.0.smithi045.stdout: 839: throughput=5MB/sec pending data=0 2021-07-31T02:19:22.648 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-tmg3oT4UBFrQI0b off=857309 len=1497499 2021-07-31T02:19:22.651 INFO:tasks.workunit.client.0.smithi045.stdout:op 4293 completed, throughput=5MB/sec 2021-07-31T02:19:22.651 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-M-ir1t9yuJmtxM4 off=1749069 len=95397 2021-07-31T02:19:22.651 INFO:tasks.workunit.client.0.smithi045.stdout:op 4294 completed, throughput=5MB/sec 2021-07-31T02:19:22.652 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-VRP6QTrNaaqsIwx off=2394397 len=1750556 2021-07-31T02:19:22.655 INFO:tasks.workunit.client.0.smithi045.stdout:op 4295 completed, throughput=5MB/sec 2021-07-31T02:19:22.655 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-oIpB1eYHJ-yjv90 off=5882660 len=2085797 2021-07-31T02:19:22.667 INFO:tasks.workunit.client.0.smithi045.stdout:op 4296 completed, throughput=5MB/sec 2021-07-31T02:19:23.667 INFO:tasks.workunit.client.0.smithi045.stdout: 840: throughput=5MB/sec pending data=0 2021-07-31T02:19:23.668 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-pRKKeQ3XBBtEeGl off=13605832 len=1506055 2021-07-31T02:19:23.674 INFO:tasks.workunit.client.0.smithi045.stdout:op 4297 completed, throughput=5MB/sec 2021-07-31T02:19:23.675 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-i4V29EOudGCX2qx off=532263 len=810381 2021-07-31T02:19:23.680 INFO:tasks.workunit.client.0.smithi045.stdout:op 4298 completed, throughput=5MB/sec 2021-07-31T02:19:23.681 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Y7DjHqZ6v8KY1F2 off=4823039 len=1079231 2021-07-31T02:19:23.682 INFO:tasks.workunit.client.0.smithi045.stdout:op 4299 completed, throughput=5MB/sec 2021-07-31T02:19:23.683 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-kXCI8rX-AX7Nhqv off=11664306 len=192929 2021-07-31T02:19:23.686 INFO:tasks.workunit.client.0.smithi045.stdout:op 4300 completed, throughput=5MB/sec 2021-07-31T02:19:23.686 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-9hBDMCslzZi24O7 off=4881849 len=461006 2021-07-31T02:19:23.689 INFO:tasks.workunit.client.0.smithi045.stdout:op 4301 completed, throughput=5MB/sec 2021-07-31T02:19:24.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:19:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:19:23.564Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:19:24.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:19:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:19:23.564Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:19:24.027 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:19:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:19:23.564Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:19:24.042 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:19:24.323 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:19:24.323 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 34m ago 35m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:19:24.323 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 31m ago 34m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:19:24.324 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 31m ago 38m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:19:24.324 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 34m ago 40m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:19:24.324 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 34m ago 40m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:19:24.324 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 31m ago 38m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:19:24.325 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 34m ago 38m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:19:24.325 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 34m ago 35m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:19:24.325 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 31m ago 35m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:19:24.325 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 34m ago 38m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:19:24.325 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 34m ago 37m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:19:24.326 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 34m ago 37m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:19:24.326 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 34m ago 37m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:19:24.326 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 31m ago 36m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:19:24.327 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 31m ago 36m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:19:24.327 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 31m ago 36m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:19:24.327 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 31m ago 35m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:19:24.328 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 31m ago 35m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:19:24.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:24 smithi154 conmon[34553]: cluster 2021-07-31T02:19:22.251972+0000 mgr.x (mgr.24542) 1373 : cluster [DBG] pgmap v1836: 9 pgs: 9 active+clean; 127 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 2.8 MiB/s rd, 1.4 MiB/s wr, 4 op/s 2021-07-31T02:19:24.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:24 smithi045 conmon[39718]: cluster 2021-07-31T02:19:22.251972+0000 mgr.x ( 2021-07-31T02:19:24.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:24 smithi045 conmon[39718]: mgr.24542) 1373 : cluster [DBG] pgmap v1836: 9 pgs: 9 active+clean; 127 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 2.8 MiB/s rd, 1.4 MiB/s wr, 4 op/s 2021-07-31T02:19:24.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:24 smithi045 conmon[33655]: cluster 2021-07-31T02:19:22.251972+0000 mgr.x (mgr 2021-07-31T02:19:24.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:24 smithi045 conmon[33655]: .24542) 1373 : cluster [DBG] pgmap v1836: 9 pgs: 9 active+clean; 127 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 2.8 MiB/s rd, 1.4 MiB/s wr, 4 op/s 2021-07-31T02:19:24.661 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:19:24.661 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:19:24.662 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:19:24.662 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:19:24.662 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:19:24.662 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:19:24.663 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:19:24.663 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:19:24.663 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:19:24.663 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:19:24.664 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:19:24.664 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:19:24.664 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:19:24.664 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:19:24.665 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:19:24.665 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:19:24.665 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:19:24.690 INFO:tasks.workunit.client.0.smithi045.stdout: 841: throughput=4.99MB/sec pending data=0 2021-07-31T02:19:24.690 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-zN7cSVX5yG1B0QI off=1971870 len=478814 2021-07-31T02:19:24.691 INFO:tasks.workunit.client.0.smithi045.stdout:op 4302 completed, throughput=4.99MB/sec 2021-07-31T02:19:24.691 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-VllGcOben1TSdrE off=14533246 len=1317383 2021-07-31T02:19:24.694 INFO:tasks.workunit.client.0.smithi045.stdout:op 4303 completed, throughput=5MB/sec 2021-07-31T02:19:24.695 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Mx3GAimzRBcR84L off=19786703 len=1439460 2021-07-31T02:19:24.701 INFO:tasks.workunit.client.0.smithi045.stdout:op 4304 completed, throughput=5MB/sec 2021-07-31T02:19:24.702 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj--sEmDj65WpyHEEY off=5001578 len=645033 2021-07-31T02:19:24.705 INFO:tasks.workunit.client.0.smithi045.stdout:op 4305 completed, throughput=5MB/sec 2021-07-31T02:19:24.705 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Gw-Y4Neey9s7Jh3 off=4235885 len=90267 2021-07-31T02:19:24.706 INFO:tasks.workunit.client.0.smithi045.stdout:op 4306 completed, throughput=5MB/sec 2021-07-31T02:19:24.706 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-aNUw73_8q-RRKk_ off=5287986 len=753018 2021-07-31T02:19:24.709 INFO:tasks.workunit.client.0.smithi045.stdout:op 4307 completed, throughput=5MB/sec 2021-07-31T02:19:24.709 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-3wbhLlDiEdST1Nj off=16827129 len=1912109 2021-07-31T02:19:24.713 INFO:tasks.workunit.client.0.smithi045.stdout:op 4308 completed, throughput=5MB/sec 2021-07-31T02:19:25.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:25 smithi154 conmon[34553]: audit 2021-07-31T02:19:24.029666+0000 mgr.x (mgr.24542) 1374 : audit [DBG] from='client.26128 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:19:25.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:25 smithi154 conmon[34553]: audit 2021-07-31T02:19:24.661466+0000 mon.c (mon.1) 88 : audit [DBG] from='client.? 172.21.15.45:0/3602942777' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:19:25.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:25 smithi045 conmon[33655]: audit 2021-07-31T02:19:24.029666+0000 mgr.x (mgr.24542) 1374 : audit [DBG] from='client.26128 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:19:25.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:25 smithi045 conmon[33655]: audit 2021-07-31T02:19:24.661466+0000 mon.c (mon.1) 88 : audit [DBG] from='client.? 172.21.15.45:0/3602942777' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:19:25.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:25 smithi045 conmon[39718]: audit 2021-07-31T02:19:24.029666+0000 mgr.x ( 2021-07-31T02:19:25.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:25 smithi045 conmon[39718]: mgr.24542) 1374 : audit [DBG] from='client.26128 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:19:25.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:25 smithi045 conmon[39718]: audit 2021-07-31T02:19:24 2021-07-31T02:19:25.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:25 smithi045 conmon[39718]: .661466+0000 mon.c (mon.1) 88 : audit [DBG] from='client.? 172.21.15.45:0/3602942777' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:19:25.713 INFO:tasks.workunit.client.0.smithi045.stdout: 842: throughput=5MB/sec pending data=0 2021-07-31T02:19:25.714 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-bs41xwfLFN-3N2i off=2030442 len=716159 2021-07-31T02:19:25.719 INFO:tasks.workunit.client.0.smithi045.stdout:op 4309 completed, throughput=5MB/sec 2021-07-31T02:19:25.719 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-bWRBGDjstOLoEkU off=960014 len=1484634 2021-07-31T02:19:25.721 INFO:tasks.workunit.client.0.smithi045.stdout:op 4310 completed, throughput=5MB/sec 2021-07-31T02:19:25.721 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-NueHoseG8XAWO9v off=7503342 len=452146 2021-07-31T02:19:25.722 INFO:tasks.workunit.client.0.smithi045.stdout:op 4311 completed, throughput=5MB/sec 2021-07-31T02:19:25.722 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-W_RGLRcQ-8SKzLc off=13881443 len=1334769 2021-07-31T02:19:25.730 INFO:tasks.workunit.client.0.smithi045.stdout:op 4312 completed, throughput=5MB/sec 2021-07-31T02:19:25.730 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-5wPzBUsO7u3wf2K off=1121006 len=539665 2021-07-31T02:19:25.732 INFO:tasks.workunit.client.0.smithi045.stdout:op 4313 completed, throughput=5MB/sec 2021-07-31T02:19:26.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:26 smithi154 conmon[34553]: cluster 2021-07-31T02:19:24.252393+0000 mgr.x (mgr.24542) 1375 : cluster [DBG] pgmap v1837: 9 pgs: 9 active+clean; 127 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 3.7 MiB/s rd, 2.7 MiB/s wr, 6 op/s 2021-07-31T02:19:26.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:26 smithi154 conmon[34553]: audit 2021-07-31T02:19:24.315655+0000 mgr.x (mgr.24542) 1376 : audit [DBG] from='client.16227 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:19:26.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:26 smithi045 conmon[39718]: cluster 2021-07-31T02:19:24.252393 2021-07-31T02:19:26.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:26 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 1375 : cluster [DBG] pgmap v1837: 9 pgs: 9 active+clean; 127 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 3.7 MiB/s rd, 2.7 MiB/s wr, 6 op/s 2021-07-31T02:19:26.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:26 smithi045 conmon[39718]: audit 2021-07-31T02: 2021-07-31T02:19:26.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:26 smithi045 conmon[39718]: 19:24.315655+0000 mgr.x ( 2021-07-31T02:19:26.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:26 smithi045 conmon[39718]: mgr.24542) 1376 : audit [DBG] 2021-07-31T02:19:26.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:26 smithi045 conmon[39718]: from='client.16227 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:19:26.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:26 smithi045 conmon[33655]: cluster 2021-07-31T02:19:24.252393+0000 mgr.x (mgr.24542) 1375 : cluster [DBG] pgmap v1837: 9 pgs: 9 active+clean; 127 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 3.7 MiB/s rd, 2.7 MiB/s wr, 6 op/s 2021-07-31T02:19:26.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:26 smithi045 conmon[33655]: 2021-07-31T02:19:26.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:26 smithi045 conmon[33655]: audit 2021-07-31T02:19:26.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:26 smithi045 conmon[33655]: 2021-07 2021-07-31T02:19:26.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:26 smithi045 conmon[33655]: -31T02:19:24.315655+0000 2021-07-31T02:19:26.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:26 smithi045 conmon[33655]: mgr.x (mgr.24542) 1376 : 2021-07-31T02:19:26.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:26 smithi045 conmon[33655]: audit [DBG] from='client.16227 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:19:26.733 INFO:tasks.workunit.client.0.smithi045.stdout: 843: throughput=4.99MB/sec pending data=0 2021-07-31T02:19:26.733 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-0fDjqFtfBJuWLwa off=132595 len=1120038 2021-07-31T02:19:26.736 INFO:tasks.workunit.client.0.smithi045.stdout:op 4314 completed, throughput=5MB/sec 2021-07-31T02:19:26.736 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-bHASt3v12s8jQ7i off=17096860 len=510827 2021-07-31T02:19:26.741 INFO:tasks.workunit.client.0.smithi045.stdout:op 4315 completed, throughput=5MB/sec 2021-07-31T02:19:26.741 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-dQSeikALP6I1TIe off=2244609 len=443763 2021-07-31T02:19:26.742 INFO:tasks.workunit.client.0.smithi045.stdout:op 4316 completed, throughput=5MB/sec 2021-07-31T02:19:26.742 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-KT2mWFzQ6WDh99Y off=5954771 len=729700 2021-07-31T02:19:26.748 INFO:tasks.workunit.client.0.smithi045.stdout:op 4317 completed, throughput=5MB/sec 2021-07-31T02:19:26.748 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-VphUQu_yt7gDA3A off=6745978 len=1908619 2021-07-31T02:19:26.756 INFO:tasks.workunit.client.0.smithi045.stdout:op 4318 completed, throughput=5MB/sec 2021-07-31T02:19:26.757 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Q6FbF1MGFMGTCge off=9048635 len=138522 2021-07-31T02:19:26.758 INFO:tasks.workunit.client.0.smithi045.stdout:op 4319 completed, throughput=5MB/sec 2021-07-31T02:19:26.758 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-kE2pkLntcgYswq0 off=1843406 len=709275 2021-07-31T02:19:26.759 INFO:tasks.workunit.client.0.smithi045.stdout:op 4320 completed, throughput=5MB/sec 2021-07-31T02:19:27.759 INFO:tasks.workunit.client.0.smithi045.stdout: 844: throughput=4.99MB/sec pending data=0 2021-07-31T02:19:27.759 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-w9OYTIlVMYUaKan off=2420119 len=405119 2021-07-31T02:19:27.760 INFO:tasks.workunit.client.0.smithi045.stdout:op 4321 completed, throughput=5MB/sec 2021-07-31T02:19:27.760 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-R2fVupmJD7yYRWv off=3908694 len=1037551 2021-07-31T02:19:27.764 INFO:tasks.workunit.client.0.smithi045.stdout:op 4322 completed, throughput=5MB/sec 2021-07-31T02:19:27.765 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-BCHzeqWCgbNe5Fo off=1831052 len=1624785 2021-07-31T02:19:27.773 INFO:tasks.workunit.client.0.smithi045.stdout:op 4323 completed, throughput=5MB/sec 2021-07-31T02:19:27.773 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-dshBWL-iieYb0dk off=45801 len=1347261 2021-07-31T02:19:27.780 INFO:tasks.workunit.client.0.smithi045.stdout:op 4324 completed, throughput=5MB/sec 2021-07-31T02:19:27.780 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-0xCnyIdmNzYZj_q off=154819 len=2032669 2021-07-31T02:19:27.788 INFO:tasks.workunit.client.0.smithi045.stdout:op 4325 completed, throughput=5MB/sec 2021-07-31T02:19:28.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:28 smithi154 conmon[34553]: cluster 2021-07-31T02:19:26.252903+0000 mgr.x (mgr.24542) 1377 : cluster [DBG] 2021-07-31T02:19:28.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:28 smithi154 conmon[34553]: pgmap v1838: 9 pgs: 9 active+clean; 127 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2021-07-31T02:19:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:28 smithi154 conmon[34553]: audit 2021-07-31T02:19:28.044733+0000 mon.b (mon.2) 228 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:19:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:28 smithi154 conmon[34553]: audit 2021-07-31T02:19:28.045334+0000 mon.a (mon 2021-07-31T02:19:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:28 smithi154 conmon[34553]: .0) 1893 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:19:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:28 smithi154 conmon[34553]: audit 2021-07-31T02: 2021-07-31T02:19:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:28 smithi154 conmon[34553]: 19:28.066451+0000 mon.b (mon.2) 229 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:19:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:28 smithi154 conmon[34553]: audit 2021-07-31T02:19 2021-07-31T02:19:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:28 smithi154 conmon[34553]: :28.066956+0000 mon.a (mon.0) 1894 : audit [INF] 2021-07-31T02:19:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:28 smithi154 conmon[34553]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:19:28.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:28 smithi045 conmon[33655]: cluster 2021-07-31T02:19:26.252903+0000 mgr.x (mgr.24542) 1377 : cluster [DBG] pgmap v1838: 9 pgs: 9 active+clean; 127 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2021-07-31T02:19:28.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:28 smithi045 conmon[33655]: audit 2021-07-31T02:19:28.044733+0000 mon.b (mon.2) 228 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:19:28.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:28 smithi045 conmon[33655]: audit 2021-07-31T02:19:28.045334+0000 mon.a (mon.0) 1893 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:19:28.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:28 smithi045 conmon[33655]: audit 2021-07-31T02:19:28.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:28 smithi045 conmon[33655]: 2021-07-31 2021-07-31T02:19:28.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:28 smithi045 conmon[33655]: T02:19:28.066451+0000 mon.b ( 2021-07-31T02:19:28.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:28 smithi045 conmon[33655]: mon.2) 229 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:19:28.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:28 smithi045 conmon[33655]: audit 2021-07- 2021-07-31T02:19:28.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:28 smithi045 conmon[33655]: 31T02:19:28.066956+0000 mon.a (mon.0) 1894 : 2021-07-31T02:19:28.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:28 smithi045 conmon[33655]: audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:19:28.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:28 smithi045 conmon[39718]: cluster 2021-07-31T02:19:26 2021-07-31T02:19:28.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:28 smithi045 conmon[39718]: .252903+0000 mgr.x (mgr.24542) 1377 : cluster [DBG] pgmap v1838: 9 pgs: 9 active+clean; 127 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2021-07-31T02:19:28.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:28 smithi045 conmon[39718]: audit 2021-07-31T02:19:28.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:28 smithi045 conmon[39718]: 2021-07-31T02: 2021-07-31T02:19:28.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:28 smithi045 conmon[39718]: 19:28.044733+0000 mon.b (mon. 2021-07-31T02:19:28.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:28 smithi045 conmon[39718]: 2) 228 : audit [INF] 2021-07-31T02:19:28.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:28 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:19:28.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:28 smithi045 conmon[39718]: audit 2021 2021-07-31T02:19:28.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:28 smithi045 conmon[39718]: -07-31T02:19:28. 2021-07-31T02:19:28.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:28 smithi045 conmon[39718]: 045334+0000 mon.a (mon.0 2021-07-31T02:19:28.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:28 smithi045 conmon[39718]: ) 1893 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:19:28.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:28 smithi045 conmon[39718]: 2021-07-31T02:19:28.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:28 smithi045 conmon[39718]: audit 2021- 2021-07-31T02:19:28.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:28 smithi045 conmon[39718]: 07-31T02:19:28. 2021-07-31T02:19:28.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:28 smithi045 conmon[39718]: 066451+0000 mon.b (mon.2 2021-07-31T02:19:28.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:28 smithi045 conmon[39718]: ) 229 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:19:28.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:28 smithi045 conmon[39718]: 2021-07-31T02:19:28.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:28 smithi045 conmon[39718]: audit 2021 2021-07-31T02:19:28.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:28 smithi045 conmon[39718]: -07-31T02:19:28 2021-07-31T02:19:28.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:28 smithi045 conmon[39718]: .066956+0000 mon.a (mon. 2021-07-31T02:19:28.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:28 smithi045 conmon[39718]: 0) 1894 : audit [INF] 2021-07-31T02:19:28.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:28 smithi045 conmon[39718]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:19:28.788 INFO:tasks.workunit.client.0.smithi045.stdout: 845: throughput=5MB/sec pending data=0 2021-07-31T02:19:28.788 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-QPRncLuNjugh2yh off=8976403 len=970547 2021-07-31T02:19:28.790 INFO:tasks.workunit.client.0.smithi045.stdout:op 4326 completed, throughput=5MB/sec 2021-07-31T02:19:28.791 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-70VooUW9a_oCSxU off=4769446 len=1669003 2021-07-31T02:19:28.798 INFO:tasks.workunit.client.0.smithi045.stdout:op 4327 completed, throughput=5MB/sec 2021-07-31T02:19:28.798 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-btjBFaW4sW_9Jte off=3903833 len=1820076 2021-07-31T02:19:28.801 INFO:tasks.workunit.client.0.smithi045.stdout:op 4328 completed, throughput=5MB/sec 2021-07-31T02:19:29.802 INFO:tasks.workunit.client.0.smithi045.stdout: 846: throughput=5MB/sec pending data=0 2021-07-31T02:19:29.802 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-X02JJSBhPlM5Uw4 off=6788779 len=538036 2021-07-31T02:19:29.803 INFO:tasks.workunit.client.0.smithi045.stdout:op 4329 completed, throughput=5MB/sec 2021-07-31T02:19:29.803 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-rEdAY_T69Spm4iQ off=3879842 len=1007691 2021-07-31T02:19:29.804 INFO:tasks.workunit.client.0.smithi045.stdout:op 4330 completed, throughput=5MB/sec 2021-07-31T02:19:29.804 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-EDQkILBLxXXh2Du off=6515860 len=1014301 2021-07-31T02:19:29.806 INFO:tasks.workunit.client.0.smithi045.stdout:op 4331 completed, throughput=5MB/sec 2021-07-31T02:19:29.806 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-qSveJU5tFAyfla7 off=4678272 len=976727 2021-07-31T02:19:29.812 INFO:tasks.workunit.client.0.smithi045.stdout:op 4332 completed, throughput=5MB/sec 2021-07-31T02:19:29.813 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-VQHiQcV4PKq8R2f off=7361057 len=1051214 2021-07-31T02:19:29.814 INFO:tasks.workunit.client.0.smithi045.stdout:op 4333 completed, throughput=5MB/sec 2021-07-31T02:19:30.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:30 smithi154 conmon[34553]: cluster 2021-07-31T02:19:28.253409+0000 mgr.x (mgr.24542) 1378 : cluster [DBG] pgmap v1839: 9 pgs: 9 active+clean; 127 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2021-07-31T02:19:30.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:30 smithi045 conmon[33655]: cluster 2021-07-31T02:19:28.253409+0000 mgr.x (mgr.24542) 1378 : cluster [DBG] pgmap v1839: 9 pgs: 9 active+clean; 127 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2021-07-31T02:19:30.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:30 smithi045 conmon[39718]: cluster 2021-07-31T02:19:28.253409 2021-07-31T02:19:30.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:30 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 1378 : cluster [DBG] pgmap v1839: 9 pgs: 9 active+clean; 127 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2021-07-31T02:19:30.815 INFO:tasks.workunit.client.0.smithi045.stdout: 847: throughput=4.99MB/sec pending data=0 2021-07-31T02:19:30.815 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-rYuOwGn6N0eBuX0 off=3986963 len=197473 2021-07-31T02:19:30.818 INFO:tasks.workunit.client.0.smithi045.stdout:op 4334 completed, throughput=4.99MB/sec 2021-07-31T02:19:30.818 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-EgILvhFKOxGMU_X off=7811318 len=967762 2021-07-31T02:19:30.823 INFO:tasks.workunit.client.0.smithi045.stdout:op 4335 completed, throughput=5MB/sec 2021-07-31T02:19:30.824 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-CAv90fYF362izDu off=531858 len=1870467 2021-07-31T02:19:30.825 INFO:tasks.workunit.client.0.smithi045.stdout:op 4336 completed, throughput=5MB/sec 2021-07-31T02:19:30.826 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-UQsO0INJ3vB_Unk off=11260733 len=770629 2021-07-31T02:19:30.827 INFO:tasks.workunit.client.0.smithi045.stdout:op 4337 completed, throughput=5MB/sec 2021-07-31T02:19:30.827 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-mZ1zOtqsQ1T5tto off=14184234 len=612433 2021-07-31T02:19:30.832 INFO:tasks.workunit.client.0.smithi045.stdout:op 4338 completed, throughput=5MB/sec 2021-07-31T02:19:30.832 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-5iFJOGKHJH1Jgt5 off=13168224 len=1760984 2021-07-31T02:19:30.839 INFO:tasks.workunit.client.0.smithi045.stdout:op 4339 completed, throughput=5MB/sec 2021-07-31T02:19:31.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: debug 2021-07-31T02:19:31.287+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:19:31.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:19:31.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:19:31.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: yield 2021-07-31T02:19:31.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:19:31.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:19:31.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:19:31.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:19:31.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:19:31.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:19:31.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:19:31.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:19:31.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:19:31.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: task.result() 2021-07-31T02:19:31.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:19:31.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: debug 2021-07-31T02:19:31.298+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:19:31.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:19:31.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:19:31.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: yield 2021-07-31T02:19:31.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:19:31.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:19:31.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:19:31.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:19:31.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:19:31.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:19:31.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:19:31.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:19:31.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:19:31.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: task.result() 2021-07-31T02:19:31.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:19:31.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: debug 2021-07-31T02:19:31.324+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:19:31.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:19:31.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:19:31.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: yield 2021-07-31T02:19:31.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:19:31.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:19:31.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:19:31.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:19:31.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:19:31.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:19:31.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:19:31.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:19:31.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:19:31.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: task.result() 2021-07-31T02:19:31.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:19:31.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: debug 2021-07-31T02:19:31.324+0000 7fcb9195d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:19:31.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:19:31.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:19:31.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: yield 2021-07-31T02:19:31.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:19:31.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:19:31.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:19:31.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:19:31.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:19:31.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:19:31.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:19:31.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:19:31.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:19:31.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: task.result() 2021-07-31T02:19:31.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:19:31.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: 2021-07-31T02:19:31.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:19:31.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: 2021-07-31T02:19:31.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:19:31.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:19:31.786 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:19:31.787 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:19:31.787 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:19:31.788 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:19:31.788 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:19:31.788 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:19:31.788 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:19:31.788 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:19:31.789 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:19:31.789 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:19:31.789 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:19:31.789 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:19:31.790 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:19:31.790 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:19:31.790 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:19:31.790 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:19:31.790 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: raise 2021-07-31T02:19:31.791 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:19:31.791 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:19:31.791 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:19:31.791 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:19:31.792 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:19:31.792 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: debug 2021-07-31T02:19:31.333+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:19:31.792 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:19:31.792 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:19:31.792 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: yield 2021-07-31T02:19:31.793 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:19:31.793 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:19:31.793 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:19:31.793 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:19:31.793 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:19:31.794 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:19:31.794 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:19:31.794 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:19:31.794 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:19:31.795 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: task.result() 2021-07-31T02:19:31.795 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:19:31.795 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: debug 2021-07-31T02:19:31.333+0000 7fcb9215e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:19:31.795 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:19:31.795 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:19:31.796 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: yield 2021-07-31T02:19:31.796 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:19:31.796 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:19:31.796 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:19:31.796 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:19:31.797 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:19:31.797 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:19:31.797 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:19:31.798 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:19:31.798 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:19:31.798 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: task.result() 2021-07-31T02:19:31.798 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:19:31.799 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: 2021-07-31T02:19:31.799 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:19:31.799 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: 2021-07-31T02:19:31.799 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:19:31.799 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:19:31.800 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:19:31.800 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:19:31.800 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:19:31.800 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:19:31.800 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:19:31.801 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:19:31.801 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:19:31.801 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:19:31.801 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:19:31.802 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:19:31.802 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:19:31.802 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:19:31.802 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:19:31.802 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:19:31.803 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:19:31.803 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:19:31.803 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: raise 2021-07-31T02:19:31.803 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:19:31.803 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:19:31.804 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:19:31.804 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:19:31.804 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:19:31 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:19:31.839 INFO:tasks.workunit.client.0.smithi045.stdout: 848: throughput=5MB/sec pending data=0 2021-07-31T02:19:31.840 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ydSw9V2AyhXhJX2 off=6104178 len=1056742 2021-07-31T02:19:31.845 INFO:tasks.workunit.client.0.smithi045.stdout:op 4340 completed, throughput=5MB/sec 2021-07-31T02:19:31.845 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ci9iXV1cFvWk0T2 off=3180211 len=494395 2021-07-31T02:19:31.845 INFO:tasks.workunit.client.0.smithi045.stdout:op 4341 completed, throughput=5MB/sec 2021-07-31T02:19:31.846 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-WZGC6mIlinM2RaW off=5525430 len=1453534 2021-07-31T02:19:31.855 INFO:tasks.workunit.client.0.smithi045.stdout:op 4342 completed, throughput=5MB/sec 2021-07-31T02:19:31.855 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-FMH7LKvHXPiVNhd off=1672369 len=1482829 2021-07-31T02:19:31.857 INFO:tasks.workunit.client.0.smithi045.stdout:op 4343 completed, throughput=5MB/sec 2021-07-31T02:19:32.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:32 smithi154 conmon[34553]: cluster 2021-07-31T02:19:30.254024+0000 mgr.x (mgr.24542) 1379 : cluster [DBG] pgmap v1840: 9 pgs: 9 active+clean; 127 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 3.0 MiB/s rd, 3.4 MiB/s wr, 6 op/s 2021-07-31T02:19:32.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:32 smithi154 conmon[34553]: 2021-07-31T02:19:32.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:32 smithi154 conmon[34553]: audit 2021-07-31T02:19:31.241329+0000 mon.b (mon.2) 230 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:19:32.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:32 smithi154 conmon[34553]: audit 2021-07-31T02:19:31.242690+0000 mon.b (mon.2) 231 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:19:32.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:32 smithi154 conmon[34553]: audit 2021-07-31T02:19:31.243698+0000 mon.b (mon.2) 232 2021-07-31T02:19:32.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:32 smithi154 conmon[34553]: : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:19:32.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:32 smithi045 conmon[33655]: cluster 2021-07-31T02:19:30.254024+0000 mgr.x (mgr.24542) 1379 : cluster [DBG] pgmap v1840: 9 pgs: 9 active+clean; 127 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 3.0 MiB/s rd, 3.4 MiB/s wr, 6 op/s 2021-07-31T02:19:32.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:32 smithi045 conmon[33655]: audit 2021-07-31T02:19:31.241329+0000 mon.b (mon.2) 230 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:19:32.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:32 smithi045 conmon[33655]: audit 2021-07-31T02:19:31.242690+0000 mon.b (mon.2) 231 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:19:32.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:32 smithi045 conmon[33655]: audit 2021-07-31T02:19:32.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:32 smithi045 conmon[33655]: 2021-07-31 2021-07-31T02:19:32.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:32 smithi045 conmon[33655]: T02:19:31.243698 2021-07-31T02:19:32.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:32 smithi045 conmon[33655]: +0000 mon.b (mon.2) 2021-07-31T02:19:32.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:32 smithi045 conmon[33655]: 232 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:19:32.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:32 smithi045 conmon[39718]: cluster 2021-07-31T02:19:30. 2021-07-31T02:19:32.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:32 smithi045 conmon[39718]: 254024+0000 mgr.x (mgr.24542) 1379 : cluster [DBG] pgmap v1840: 9 pgs: 9 active+clean; 127 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 3.0 MiB/s rd, 3.4 MiB/s wr, 6 op/s 2021-07-31T02:19:32.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:32 smithi045 conmon[39718]: audit 2021-07-31T02:19:31. 2021-07-31T02:19:32.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:32 smithi045 conmon[39718]: 241329+0000 mon.b (mon.2) 230 : audit [DBG] 2021-07-31T02:19:32.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:32 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:19:32.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:32 smithi045 conmon[39718]: audit 2021-07-31T02:19:32.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:32 smithi045 conmon[39718]: 2021-07-31T02:19:31.242690+0000 mon.b 2021-07-31T02:19:32.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:32 smithi045 conmon[39718]: (mon.2) 231 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:19:32.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:32 smithi045 conmon[39718]: audit 2021-07-31T02:19:32.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:32 smithi045 conmon[39718]: 2021-07-31T02:19: 2021-07-31T02:19:32.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:32 smithi045 conmon[39718]: 31.243698+0000 mon.b (mon.2 2021-07-31T02:19:32.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:32 smithi045 conmon[39718]: ) 232 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:19:32.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:32 smithi045 conmon[39718]: 2021-07-31T02:19:32.858 INFO:tasks.workunit.client.0.smithi045.stdout: 849: throughput=4.99MB/sec pending data=0 2021-07-31T02:19:32.859 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-cKPY8-OEuMKw49i off=5631537 len=162440 2021-07-31T02:19:32.861 INFO:tasks.workunit.client.0.smithi045.stdout:op 4344 completed, throughput=4.99MB/sec 2021-07-31T02:19:32.861 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-jKHvYOqoJuGnF1J off=95421 len=1856988 2021-07-31T02:19:32.870 INFO:tasks.workunit.client.0.smithi045.stdout:op 4345 completed, throughput=5MB/sec 2021-07-31T02:19:32.870 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-CPdOl94vhZtey9i off=0 len=1085185 2021-07-31T02:19:32.875 INFO:tasks.workunit.client.0.smithi045.stdout:op 4346 completed, throughput=5MB/sec 2021-07-31T02:19:32.875 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-eMcIi8GTO45p7uA off=13403142 len=1992606 2021-07-31T02:19:32.878 INFO:tasks.workunit.client.0.smithi045.stdout:op 4347 completed, throughput=5MB/sec 2021-07-31T02:19:32.878 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-yT_bPIpDN3LRUOc off=13845199 len=2078655 2021-07-31T02:19:32.880 INFO:tasks.workunit.client.0.smithi045.stdout:op 4348 completed, throughput=5MB/sec 2021-07-31T02:19:33.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:19:31.244502+0000 mgr.x (mgr.24542) 1380 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:19:33.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:19:31.287985+0000 mgr.x (mgr.24542) 1381 : cephadm [ERR] 2021-07-31T02:19:33.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:19:33.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:19:33.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: yield 2021-07-31T02:19:33.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:19:33.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:19:33.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:19:33.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:19:33.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:19:33.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:19:33.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:19:33.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:19:33.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:19:33.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:19:33.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:19:33.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:19:31.288391+0000 mgr.x (mgr.24542) 1382 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:19:33.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: cephadm 2021 2021-07-31T02:19:33.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: -07-31T02:19:31.299423+0000 mgr.x (mgr.24542) 1383 : cephadm [ERR] 2021-07-31T02:19:33.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:19:33.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:19:33.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: yield 2021-07-31T02:19:33.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:19:33.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:19:33.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:19:33.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:19:33.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:19:33.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:19:33.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:19:33.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:19:33.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:19:33.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:19:33.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:19:33.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:19:31.300455+0000 mgr.x (mgr.24542) 1384 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:19:33.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:19:31.325011+0000 mgr.x (mgr.24542) 1385 : cephadm [ERR] 2021-07-31T02:19:33.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:19:33.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:19:33.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: yield 2021-07-31T02:19:33.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:19:33.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:19:33.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:19:33.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:19:33.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:19:33.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:19:33.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:19:33.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:19:33.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:19:33.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:19:33.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:19:33.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:19:31.325621+0000 mgr.x (mgr.24542) 1386 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:19:33.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:19:33.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:19:33.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: yield 2021-07-31T02:19:33.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:19:33.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:19:33.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:19:33.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:19:33.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:19:33.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:19:33.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:19:33.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:19:33.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:19:33.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:19:33.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:19:33.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: 2021-07-31T02:19:33.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:19:33.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: 2021-07-31T02:19:33.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:19:33.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:19:33.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:19:33.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:19:33.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:19:33.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:19:33.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:19:33.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:19:33.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:19:33.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:19:33.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:19:33.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:19:33.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:19:33.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:19:33.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:19:33.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:19:33.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:19:33.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:19:33.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: raise 2021-07-31T02:19:33.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:19:33.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:19:33.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:19:33.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:19:33.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:19:33.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:19:31.334137+0000 mgr.x (mgr.24542) 1387 : cephadm [ERR] 2021-07-31T02:19:33.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:19:33.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:19:33.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: yield 2021-07-31T02:19:33.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:19:33.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:19:33.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:19:33.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:19:33.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:19:33.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:19:33.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:19:33.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:19:33.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:19:33.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:19:33.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:19:33.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:19:31.334551+0000 mgr.x (mgr.24542) 1388 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:19:33.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:19:33.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:19:33.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: yield 2021-07-31T02:19:33.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:19:33.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:19:33.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:19:33.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:19:33.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:19:33.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:19:33.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:19:33.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:19:33.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:19:33.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:19:33.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:19:33.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: 2021-07-31T02:19:33.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:19:33.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: 2021-07-31T02:19:33.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:19:33.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:19:33.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:19:33.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:19:33.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:19:33.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:19:33.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:19:33.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:19:33.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:19:33.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:19:33.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:19:33.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:19:33.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:19:33.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:19:33.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:19:33.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:19:33.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:19:33.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:19:33.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: raise 2021-07-31T02:19:33.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:19:33.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:19:33.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:19:33.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:19:33.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:33 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:19:33.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:19:31. 2021-07-31T02:19:33.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: 244502+0000 mgr.x (mgr.24542) 1380 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:19:33.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: cephadm 2021-07-31T02: 2021-07-31T02:19:33.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: 19:31.287985+0000 mgr.x (mgr. 2021-07-31T02:19:33.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: 24542) 1381 : cephadm [ERR] 2021-07-31T02:19:33.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:19:33.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:19:33.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: yield 2021-07-31T02:19:33.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:19:33.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:19:33.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:19:33.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:19:33.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:19:33.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:19:33.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:19:33.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:19:33.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:19:33.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:19:33.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:19:33.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:19:31.288391+0000 mgr.x (mgr.24542) 1382 2021-07-31T02:19:33.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:19:33.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: cephadm 2021- 2021-07-31T02:19:33.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: 07-31T02:19:31.299423 2021-07-31T02:19:33.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 1383 2021-07-31T02:19:33.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: : cephadm [ERR] 2021-07-31T02:19:33.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:19:33.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:19:33.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: yield 2021-07-31T02:19:33.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:19:33.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:19:33.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:19:33.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:19:33.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:19:33.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:19:33.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:19:33.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:19:33.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:19:33.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:19:33.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:19:33.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:19:31.300455+0000 mgr.x (mgr.24542) 1384 : cephadm [INF] 2021-07-31T02:19:33.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:19:33.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: cephadm 2021-07 2021-07-31T02:19:33.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: -31T02:19:31.325011+0000 mgr.x 2021-07-31T02:19:33.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: (mgr.24542) 1385 : cephadm [ERR] 2021-07-31T02:19:33.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:19:33.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:19:33.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: yield 2021-07-31T02:19:33.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:19:33.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:19:33.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:19:33.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:19:33.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:19:33.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:19:33.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:19:33.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:19:33.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:19:33.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:19:33.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:19:33.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: 2021-07-31T02:19:33.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:19:31.325621+0000 mgr.x (mgr.24542) 2021-07-31T02:19:33.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: 1386 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:19:33.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:19:33.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:19:33.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: yield 2021-07-31T02:19:33.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:19:33.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:19:33.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:19:33.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:19:33.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:19:33.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:19:33.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:19:33.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:19:33.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:19:33.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:19:33.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:19:33.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: 2021-07-31T02:19:33.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:19:33.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: 2021-07-31T02:19:33.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:19:33.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:19:33.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:19:33.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:19:33.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:19:33.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:19:33.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:19:33.556 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:19:31.244502+0000 mgr.x (mgr.24542) 1380 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:19:33.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:19:31.287985+0000 mgr.x (mgr.24542) 1381 : cephadm [ERR] 2021-07-31T02:19:33.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:19:33.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:19:33.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: yield 2021-07-31T02:19:33.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:19:33.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:19:33.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:19:33.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:19:33.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:19:33.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:19:33.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:19:33.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:19:33.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:19:33.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:19:33.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:19:33.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:19:31.288391+0000 mgr.x (mgr.24542) 1382 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:19:33.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:19:31.299423+0000 mgr.x (mgr.24542) 1383 : cephadm [ERR] 2021-07-31T02:19:33.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:19:33.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:19:33.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: yield 2021-07-31T02:19:33.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:19:33.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:19:33.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:19:33.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:19:33.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:19:33.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:19:33.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:19:33.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:19:33.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:19:33.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:19:33.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:19:33.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:19:31.300455+0000 mgr.x (mgr.24542) 1384 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:19:33.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: cephadm 2021 2021-07-31T02:19:33.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: -07-31T02:19:31.325011+0000 mgr.x (mgr 2021-07-31T02:19:33.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: .24542) 1385 : cephadm [ERR] 2021-07-31T02:19:33.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: 2021-07-31T02:19:33.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:19:33.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:19:33.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:19:33.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:19:33.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:19:33.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:19:33.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:19:33.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:19:33.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:19:33.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:19:33.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:19:33.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:19:33.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: raise 2021-07-31T02:19:33.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:19:33.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:19:33.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:19:33.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:19:33.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:19:33.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:19:31.334137+0000 mgr.x (mgr.24542) 1387 : cephadm [ERR] 2021-07-31T02:19:33.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:19:33.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:19:33.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: yield 2021-07-31T02:19:33.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:19:33.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:19:33.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:19:33.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:19:33.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:19:33.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:19:33.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:19:33.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:19:33.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:19:33.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:19:33.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:19:33.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:19:31.334551+0000 mgr.x (mgr.24542) 1388 : cephadm [ERR] 2021-07-31T02:19:33.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:19:33.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:19:33.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:19:33.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: yield 2021-07-31T02:19:33.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:19:33.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:19:33.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:19:33.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: yield 2021-07-31T02:19:33.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:19:33.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:19:33.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:19:33.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:19:33.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:19:33.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:19:33.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:19:33.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:19:33.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:19:33.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:19:33.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:19:33.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:19:31.325621+0000 mgr.x (mgr. 2021-07-31T02:19:33.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: 24542) 1386 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:19:33.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:19:33.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:19:33.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: yield 2021-07-31T02:19:33.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:19:33.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:19:33.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:19:33.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:19:33.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:19:33.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:19:33.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:19:33.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:19:33.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:19:33.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:19:33.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:19:33.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: 2021-07-31T02:19:33.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:19:33.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: 2021-07-31T02:19:33.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:19:33.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:19:33.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:19:33.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:19:33.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:19:33.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:19:33.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:19:33.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:19:33.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:19:33.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:19:33.585 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:19:33.585 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:19:33.585 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:19:33.586 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:19:33.586 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:19:33.586 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:19:33.586 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:19:33.586 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:19:33.587 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:19:33.587 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: 2021-07-31T02:19:33.587 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:19:33.587 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: 2021-07-31T02:19:33.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:19:33.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:19:33.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:19:33.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:19:33.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:19:33.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:19:33.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:19:33.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:19:33.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:19:33.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:19:33.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:19:33.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:19:33.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:19:33.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:19:33.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:19:33.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:19:33.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:19:33.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:19:33.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: raise 2021-07-31T02:19:33.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:19:33.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:19:33.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:19:33.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:19:33.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:19:33.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:19:33.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:19:33.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:19:33.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:19:33.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:19:33.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:19:33.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:19:33.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:19:33.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: raise 2021-07-31T02:19:33.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:19:33.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:19:33.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:19:33.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:19:33.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:19:33.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: 2021-07-31T02:19:33.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:19:31.334137+0000 mgr.x (mgr.24542) 1387 : cephadm [ERR] 2021-07-31T02:19:33.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:19:33.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:19:33.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: yield 2021-07-31T02:19:33.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:19:33.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:19:33.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:19:33.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:19:33.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:19:33.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:19:33.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:19:33.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:19:33.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:19:33.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:19:33.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:19:33.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:19:31.334551+0000 mgr.x (mgr.24542) 1388 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:19:33.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:19:33.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:19:33.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: yield 2021-07-31T02:19:33.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:19:33.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:19:33.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:19:33.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:19:33.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:19:33.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:19:33.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:19:33.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:19:33.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:19:33.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:19:33.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:19:33.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: 2021-07-31T02:19:33.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:19:33.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: 2021-07-31T02:19:33.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:19:33.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:19:33.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:19:33.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:19:33.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:19:33.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:19:33.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:19:33.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:19:33.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:19:33.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:19:33.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:19:33.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:19:33.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:19:33.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:19:33.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:19:33.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:19:33.608 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:19:33.608 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:19:33.608 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: raise 2021-07-31T02:19:33.608 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:19:33.609 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:19:33.609 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:19:33.609 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:19:33.609 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:19:33.881 INFO:tasks.workunit.client.0.smithi045.stdout: 850: throughput=5MB/sec pending data=0 2021-07-31T02:19:33.881 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-sxGR7gu1bFF6-s4 off=4557255 len=826608 2021-07-31T02:19:33.882 INFO:tasks.workunit.client.0.smithi045.stdout:op 4349 completed, throughput=5MB/sec 2021-07-31T02:19:33.882 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ENzYzdSVQOHEPOX off=385391 len=1962919 2021-07-31T02:19:33.884 INFO:tasks.workunit.client.0.smithi045.stdout:op 4350 completed, throughput=5MB/sec 2021-07-31T02:19:33.884 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-39bJ2bcMUUOoUyI off=416490 len=199696 2021-07-31T02:19:33.888 INFO:tasks.workunit.client.0.smithi045.stdout:op 4351 completed, throughput=5MB/sec 2021-07-31T02:19:33.888 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-wgrEg6h8Yl-KVdq off=5317485 len=1429557 2021-07-31T02:19:33.890 INFO:tasks.workunit.client.0.smithi045.stdout:op 4352 completed, throughput=5MB/sec 2021-07-31T02:19:34.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:19:33.564Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:19:34.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:19:33.564Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:19:34.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:19:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:19:33.564Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:19:34.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:34 smithi154 conmon[34553]: cluster 2021-07-31T02:19:32.254463+0000 mgr.x (mgr.24542) 1389 : cluster [DBG] pgmap v1841: 9 pgs: 9 active+clean; 127 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2021-07-31T02:19:34.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:34 smithi045 conmon[33655]: cluster 2021-07-31T02:19:32.254463+0000 mgr.x (mgr.24542) 1389 : cluster 2021-07-31T02:19:34.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:34 smithi045 conmon[33655]: [DBG] pgmap v1841: 9 pgs: 9 active+clean; 127 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2021-07-31T02:19:34.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:34 smithi045 conmon[39718]: cluster 2021-07-31T02:19:32. 2021-07-31T02:19:34.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:34 smithi045 conmon[39718]: 254463+0000 mgr.x (mgr.24542) 1389 : cluster [DBG] pgmap v1841: 9 pgs: 9 active+clean; 127 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2021-07-31T02:19:34.891 INFO:tasks.workunit.client.0.smithi045.stdout: 851: throughput=5MB/sec pending data=0 2021-07-31T02:19:34.891 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-sO5o4HT2IUXjyOC off=2403510 len=623689 2021-07-31T02:19:34.893 INFO:tasks.workunit.client.0.smithi045.stdout:op 4353 completed, throughput=5MB/sec 2021-07-31T02:19:34.893 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-JpGw935GD25s7wh off=7587652 len=755035 2021-07-31T02:19:34.897 INFO:tasks.workunit.client.0.smithi045.stdout:op 4354 completed, throughput=5MB/sec 2021-07-31T02:19:34.898 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-hiebpaiKgzSk42O off=18138395 len=1288332 2021-07-31T02:19:34.900 INFO:tasks.workunit.client.0.smithi045.stdout:op 4355 completed, throughput=5MB/sec 2021-07-31T02:19:34.900 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-TFjXlRF0G18ZqzY off=5155715 len=1180412 2021-07-31T02:19:34.902 INFO:tasks.workunit.client.0.smithi045.stdout:op 4356 completed, throughput=5MB/sec 2021-07-31T02:19:34.903 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-X3TglP9GD8jxdFn off=8472567 len=604382 2021-07-31T02:19:34.903 INFO:tasks.workunit.client.0.smithi045.stdout:op 4357 completed, throughput=5MB/sec 2021-07-31T02:19:35.903 INFO:tasks.workunit.client.0.smithi045.stdout: 852: throughput=4.99MB/sec pending data=0 2021-07-31T02:19:35.904 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-GZKPUpIh3NTjQDh off=391586 len=2037425 2021-07-31T02:19:35.906 INFO:tasks.workunit.client.0.smithi045.stdout:op 4358 completed, throughput=5MB/sec 2021-07-31T02:19:35.906 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-fCVvb-z66HXelRU off=6978608 len=1904181 2021-07-31T02:19:35.907 INFO:tasks.workunit.client.0.smithi045.stdout:op 4359 completed, throughput=5MB/sec 2021-07-31T02:19:35.907 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ZM2js9G23-V26j7 off=2738724 len=1698898 2021-07-31T02:19:35.910 INFO:tasks.workunit.client.0.smithi045.stdout:op 4360 completed, throughput=5MB/sec 2021-07-31T02:19:36.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:36 smithi154 conmon[34553]: cluster 2021-07-31T02:19:34.254958+0000 mgr.x (mgr.24542) 1390 : cluster [DBG] pgmap v1842: 9 pgs: 9 active+clean; 127 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 2.8 MiB/s rd, 3.3 MiB/s wr, 5 op/s 2021-07-31T02:19:36.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:36 smithi045 conmon[33655]: cluster 2021-07-31T02:19:34.254958+0000 2021-07-31T02:19:36.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:36 smithi045 conmon[33655]: mgr.x (mgr.24542) 1390 : cluster [DBG] pgmap v1842: 9 pgs: 9 active+clean; 127 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 2.8 MiB/s rd, 3.3 MiB/s wr, 5 op/s 2021-07-31T02:19:36.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:36 smithi045 conmon[39718]: cluster 2021-07-31T02:19:34.254958+0000 2021-07-31T02:19:36.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:36 smithi045 conmon[39718]: mgr.x (mgr.24542) 1390 : cluster [DBG] pgmap v1842: 9 pgs: 9 active+clean; 127 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 2.8 MiB/s rd, 3.3 MiB/s wr, 5 op/s 2021-07-31T02:19:36.911 INFO:tasks.workunit.client.0.smithi045.stdout: 853: throughput=4.99MB/sec pending data=0 2021-07-31T02:19:36.911 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-QNhmbVptiCA3KPN off=10049149 len=2081288 2021-07-31T02:19:36.915 INFO:tasks.workunit.client.0.smithi045.stdout:op 4361 completed, throughput=5MB/sec 2021-07-31T02:19:36.915 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-HTpyki-WMXdLy9f off=3182018 len=754778 2021-07-31T02:19:36.920 INFO:tasks.workunit.client.0.smithi045.stdout:op 4362 completed, throughput=5MB/sec 2021-07-31T02:19:36.920 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-wdlwtjKKJC-2yM6 off=4465539 len=1417078 2021-07-31T02:19:36.927 INFO:tasks.workunit.client.0.smithi045.stdout:op 4363 completed, throughput=5MB/sec 2021-07-31T02:19:36.928 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-p0kblXLDTMqevsd off=351030 len=393960 2021-07-31T02:19:36.931 INFO:tasks.workunit.client.0.smithi045.stdout:op 4364 completed, throughput=5MB/sec 2021-07-31T02:19:36.931 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-fml6HSsWmBvKuYR off=8420292 len=1385532 2021-07-31T02:19:36.934 INFO:tasks.workunit.client.0.smithi045.stdout:op 4365 completed, throughput=5MB/sec 2021-07-31T02:19:37.935 INFO:tasks.workunit.client.0.smithi045.stdout: 854: throughput=5MB/sec pending data=0 2021-07-31T02:19:37.935 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-WSRybxKpkchzko2 off=13722551 len=1148456 2021-07-31T02:19:37.941 INFO:tasks.workunit.client.0.smithi045.stdout:op 4366 completed, throughput=5MB/sec 2021-07-31T02:19:37.941 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-UcI_D0k2m46o872 off=2288777 len=1918509 2021-07-31T02:19:37.944 INFO:tasks.workunit.client.0.smithi045.stdout:op 4367 completed, throughput=5MB/sec 2021-07-31T02:19:37.944 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-QnmVTgpBWMBFDqZ off=1836554 len=1557960 2021-07-31T02:19:37.951 INFO:tasks.workunit.client.0.smithi045.stdout:op 4368 completed, throughput=5MB/sec 2021-07-31T02:19:38.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:38 smithi154 conmon[34553]: cluster 2021-07-31T02:19:36.255407+0000 mgr.x (mgr.24542) 1391 : cluster [DBG] pgmap v1843: 9 pgs: 9 active+clean; 127 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.1 MiB/s wr, 3 op/s 2021-07-31T02:19:38.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:38 smithi045 conmon[39718]: cluster 2021-07-31T02:19:36.255407+0000 mgr.x ( 2021-07-31T02:19:38.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:38 smithi045 conmon[39718]: mgr.24542) 1391 : cluster [DBG] pgmap v1843: 9 pgs: 9 active+clean; 127 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.1 MiB/s wr, 3 op/s 2021-07-31T02:19:38.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:38 smithi045 conmon[33655]: cluster 2021-07-31T02:19:36.255407+0000 mgr.x (mgr.24542) 1391 2021-07-31T02:19:38.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:38 smithi045 conmon[33655]: : cluster [DBG] pgmap v1843: 9 pgs: 9 active+clean; 127 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.1 MiB/s wr, 3 op/s 2021-07-31T02:19:38.952 INFO:tasks.workunit.client.0.smithi045.stdout: 855: throughput=4.99MB/sec pending data=0 2021-07-31T02:19:38.952 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-VOA2WvRgCDvXOvE off=311510 len=924641 2021-07-31T02:19:38.957 INFO:tasks.workunit.client.0.smithi045.stdout:op 4369 completed, throughput=5MB/sec 2021-07-31T02:19:38.957 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ryeFQWmX5QVRyQ1 off=4359373 len=847933 2021-07-31T02:19:38.962 INFO:tasks.workunit.client.0.smithi045.stdout:op 4370 completed, throughput=5MB/sec 2021-07-31T02:19:38.962 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-_XbBnzDWJW1lBD4 off=11670385 len=2086926 2021-07-31T02:19:38.964 INFO:tasks.workunit.client.0.smithi045.stdout:op 4371 completed, throughput=5MB/sec 2021-07-31T02:19:38.964 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-57XmIUH77WMahYu off=367130 len=663257 2021-07-31T02:19:38.965 INFO:tasks.workunit.client.0.smithi045.stdout:op 4372 completed, throughput=5MB/sec 2021-07-31T02:19:38.966 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-RUP4ETmjTQ9beP7 off=10056046 len=1584689 2021-07-31T02:19:38.969 INFO:tasks.workunit.client.0.smithi045.stdout:op 4373 completed, throughput=5MB/sec 2021-07-31T02:19:39.969 INFO:tasks.workunit.client.0.smithi045.stdout: 856: throughput=5MB/sec pending data=0 2021-07-31T02:19:39.970 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-XwU-IQwPuZirUlH off=11194812 len=1335227 2021-07-31T02:19:39.976 INFO:tasks.workunit.client.0.smithi045.stdout:op 4374 completed, throughput=5MB/sec 2021-07-31T02:19:39.976 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-qq20W5JxYWkxEZ7 off=5759850 len=1661614 2021-07-31T02:19:39.985 INFO:tasks.workunit.client.0.smithi045.stdout:op 4375 completed, throughput=5MB/sec 2021-07-31T02:19:39.985 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-TqsX1h5BOkFGlqx off=557551 len=989181 2021-07-31T02:19:39.990 INFO:tasks.workunit.client.0.smithi045.stdout:op 4376 completed, throughput=5MB/sec 2021-07-31T02:19:39.991 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-jVRKidYt9_37QRs off=0 len=1231158 2021-07-31T02:19:39.996 INFO:tasks.workunit.client.0.smithi045.stdout:op 4377 completed, throughput=5MB/sec 2021-07-31T02:19:40.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:40 smithi154 conmon[34553]: cluster 2021-07-31T02:19:38.255784+0000 mgr.x (mgr.24542) 1392 : cluster [DBG] pgmap v1844: 9 pgs: 9 active+clean; 127 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.1 MiB/s wr, 3 op/s 2021-07-31T02:19:40.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:40 smithi045 conmon[39718]: cluster 2021-07-31T02:19:38.255784 2021-07-31T02:19:40.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:40 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 1392 : cluster [DBG] pgmap v1844: 9 pgs: 9 active+clean; 127 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.1 MiB/s wr, 3 op/s 2021-07-31T02:19:40.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:40 smithi045 conmon[33655]: cluster 2021-07-31T02:19:38.255784+0000 mgr.x (mgr.24542) 1392 : 2021-07-31T02:19:40.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:40 smithi045 conmon[33655]: cluster [DBG] pgmap v1844: 9 pgs: 9 active+clean; 127 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.1 MiB/s wr, 3 op/s 2021-07-31T02:19:40.996 INFO:tasks.workunit.client.0.smithi045.stdout: 857: throughput=5MB/sec pending data=0 2021-07-31T02:19:40.996 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-A09BCSkN56IefPi off=7314520 len=236821 2021-07-31T02:19:40.999 INFO:tasks.workunit.client.0.smithi045.stdout:op 4378 completed, throughput=5MB/sec 2021-07-31T02:19:41.000 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-TEUr0zT4J9vQWLl off=2779118 len=790169 2021-07-31T02:19:41.004 INFO:tasks.workunit.client.0.smithi045.stdout:op 4379 completed, throughput=5MB/sec 2021-07-31T02:19:41.004 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-TwM-TJBGVluZc86 off=7884645 len=710328 2021-07-31T02:19:41.005 INFO:tasks.workunit.client.0.smithi045.stdout:op 4380 completed, throughput=5MB/sec 2021-07-31T02:19:41.006 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ugg2xMpau_8nN3H off=9552192 len=325415 2021-07-31T02:19:41.009 INFO:tasks.workunit.client.0.smithi045.stdout:op 4381 completed, throughput=5MB/sec 2021-07-31T02:19:41.009 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-cHOTssXsMLNEYPU off=7498869 len=393318 2021-07-31T02:19:41.010 INFO:tasks.workunit.client.0.smithi045.stdout:op 4382 completed, throughput=5MB/sec 2021-07-31T02:19:41.010 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-oQL8WlTKuMVmLrg off=8659755 len=1573334 2021-07-31T02:19:41.013 INFO:tasks.workunit.client.0.smithi045.stdout:op 4383 completed, throughput=5MB/sec 2021-07-31T02:19:41.013 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Pz9rnnHDALxyWEt off=18056762 len=115785 2021-07-31T02:19:41.014 INFO:tasks.workunit.client.0.smithi045.stdout:op 4384 completed, throughput=5MB/sec 2021-07-31T02:19:41.014 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-xG9CP1l5c8Lmk44 off=1905980 len=18132 2021-07-31T02:19:41.016 INFO:tasks.workunit.client.0.smithi045.stdout:op 4385 completed, throughput=5MB/sec 2021-07-31T02:19:41.016 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-89SB_cgBsgCNYaR off=3115928 len=628114 2021-07-31T02:19:41.018 INFO:tasks.workunit.client.0.smithi045.stdout:op 4386 completed, throughput=5MB/sec 2021-07-31T02:19:42.018 INFO:tasks.workunit.client.0.smithi045.stdout: 858: throughput=4.99MB/sec pending data=0 2021-07-31T02:19:42.018 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-qlA5IeLStGY48NT off=6344099 len=1378236 2021-07-31T02:19:42.025 INFO:tasks.workunit.client.0.smithi045.stdout:op 4387 completed, throughput=5MB/sec 2021-07-31T02:19:42.025 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-1gkvttVAUjDz4VM off=8371891 len=1152072 2021-07-31T02:19:42.031 INFO:tasks.workunit.client.0.smithi045.stdout:op 4388 completed, throughput=5MB/sec 2021-07-31T02:19:42.031 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-6dMaxRkuBSABett off=8057618 len=364188 2021-07-31T02:19:42.034 INFO:tasks.workunit.client.0.smithi045.stdout:op 4389 completed, throughput=5MB/sec 2021-07-31T02:19:42.034 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-04gTLi179YEbDcj off=6642884 len=1043173 2021-07-31T02:19:42.035 INFO:tasks.workunit.client.0.smithi045.stdout:op 4390 completed, throughput=5MB/sec 2021-07-31T02:19:42.035 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-8vm4a3zjB2X9sBr off=4249618 len=1210278 2021-07-31T02:19:42.041 INFO:tasks.workunit.client.0.smithi045.stdout:op 4391 completed, throughput=5MB/sec 2021-07-31T02:19:42.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:42 smithi154 conmon[34553]: cluster 2021-07-31T02:19:40.256198+0000 mgr.x (mgr.24542) 1393 : cluster [DBG] 2021-07-31T02:19:42.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:42 smithi154 conmon[34553]: pgmap v1845: 9 pgs: 9 active+clean; 127 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 3.4 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2021-07-31T02:19:42.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:42 smithi045 conmon[39718]: cluster 2021-07-31T02:19:40.256198+0000 mgr.x ( 2021-07-31T02:19:42.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:42 smithi045 conmon[39718]: mgr.24542) 1393 : cluster [DBG] pgmap v1845: 9 pgs: 9 active+clean; 127 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 3.4 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2021-07-31T02:19:42.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:42 smithi045 conmon[33655]: cluster 2021-07-31T02:19:40.256198+0000 mgr.x (mgr.24542) 1393 : cluster [DBG] pgmap v1845: 9 pgs: 9 active+clean; 127 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 3.4 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2021-07-31T02:19:43.042 INFO:tasks.workunit.client.0.smithi045.stdout: 859: throughput=4.99MB/sec pending data=0 2021-07-31T02:19:43.042 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-VJqR4LMct5ytUhr off=7350037 len=1706430 2021-07-31T02:19:43.049 INFO:tasks.workunit.client.0.smithi045.stdout:op 4392 completed, throughput=5MB/sec 2021-07-31T02:19:43.049 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-GPkOXwTtn2OVE-q off=1827296 len=1341405 2021-07-31T02:19:43.052 INFO:tasks.workunit.client.0.smithi045.stdout:op 4393 completed, throughput=5MB/sec 2021-07-31T02:19:43.052 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-aKntbW6O9RYtwVD off=2343543 len=446996 2021-07-31T02:19:43.055 INFO:tasks.workunit.client.0.smithi045.stdout:op 4394 completed, throughput=5MB/sec 2021-07-31T02:19:43.055 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-k_qyL8dwMcZZ0nN off=2940770 len=1767985 2021-07-31T02:19:43.064 INFO:tasks.workunit.client.0.smithi045.stdout:op 4395 completed, throughput=5MB/sec 2021-07-31T02:19:44.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:19:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:19:43.564Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:19:44.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:19:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:19:43.564Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:19:44.027 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:19:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:19:43.564Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:19:44.065 INFO:tasks.workunit.client.0.smithi045.stdout: 860: throughput=4.99MB/sec pending data=0 2021-07-31T02:19:44.065 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-0VCCewCcLv_IDKR off=6992922 len=389243 2021-07-31T02:19:44.067 INFO:tasks.workunit.client.0.smithi045.stdout:op 4396 completed, throughput=4.99MB/sec 2021-07-31T02:19:44.067 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-x7ELh3ax6iBZJgl off=5948586 len=2096475 2021-07-31T02:19:44.078 INFO:tasks.workunit.client.0.smithi045.stdout:op 4397 completed, throughput=5MB/sec 2021-07-31T02:19:44.078 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-OCmVthaqL-lLmLf off=1786898 len=1414655 2021-07-31T02:19:44.085 INFO:tasks.workunit.client.0.smithi045.stdout:op 4398 completed, throughput=5MB/sec 2021-07-31T02:19:44.085 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ptRY6ilBzMDbz6o off=11237163 len=1470141 2021-07-31T02:19:44.092 INFO:tasks.workunit.client.0.smithi045.stdout:op 4399 completed, throughput=5MB/sec 2021-07-31T02:19:44.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:44 smithi154 conmon[34553]: cluster 2021-07-31T02:19:42.256680+0000 mgr.x (mgr.24542) 1394 : cluster [DBG] pgmap v1846: 9 pgs: 9 active+clean; 127 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.3 MiB/s wr, 3 op/s 2021-07-31T02:19:44.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:44 smithi045 conmon[39718]: cluster 2021-07-31T02:19:42.256680+0000 mgr.x ( 2021-07-31T02:19:44.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:44 smithi045 conmon[39718]: mgr.24542) 1394 : cluster [DBG] pgmap v1846: 9 pgs: 9 active+clean; 127 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.3 MiB/s wr, 3 op/s 2021-07-31T02:19:44.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:44 smithi045 conmon[33655]: cluster 2021-07-31T02:19:42.256680+0000 mgr.x (mgr.24542) 1394 : 2021-07-31T02:19:44.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:44 smithi045 conmon[33655]: cluster [DBG] pgmap v1846: 9 pgs: 9 active+clean; 127 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.3 MiB/s wr, 3 op/s 2021-07-31T02:19:45.092 INFO:tasks.workunit.client.0.smithi045.stdout: 861: throughput=4.99MB/sec pending data=0 2021-07-31T02:19:45.093 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-dGi2cwenskgpsy4 off=1742120 len=31504 2021-07-31T02:19:45.095 INFO:tasks.workunit.client.0.smithi045.stdout:op 4400 completed, throughput=4.99MB/sec 2021-07-31T02:19:45.096 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-_Oj4XtonavSG9bO off=15034652 len=333544 2021-07-31T02:19:45.099 INFO:tasks.workunit.client.0.smithi045.stdout:op 4401 completed, throughput=4.99MB/sec 2021-07-31T02:19:45.099 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-QBsUJEH3M5dnsaV off=1374183 len=978212 2021-07-31T02:19:45.104 INFO:tasks.workunit.client.0.smithi045.stdout:op 4402 completed, throughput=5MB/sec 2021-07-31T02:19:45.105 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-tUe7-6ycNhmT-e4 off=22313184 len=640385 2021-07-31T02:19:45.106 INFO:tasks.workunit.client.0.smithi045.stdout:op 4403 completed, throughput=5MB/sec 2021-07-31T02:19:45.106 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Grt1xdt4mWcRsPb off=7376925 len=2084538 2021-07-31T02:19:45.116 INFO:tasks.workunit.client.0.smithi045.stdout:op 4404 completed, throughput=5MB/sec 2021-07-31T02:19:45.116 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-WhXJadgGJcEv1Yv off=0 len=1588316 2021-07-31T02:19:45.123 INFO:tasks.workunit.client.0.smithi045.stdout:op 4405 completed, throughput=5MB/sec 2021-07-31T02:19:46.124 INFO:tasks.workunit.client.0.smithi045.stdout: 862: throughput=4.99MB/sec pending data=0 2021-07-31T02:19:46.124 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-g2eqEVaw8s8_uAg off=3238206 len=1486654 2021-07-31T02:19:46.127 INFO:tasks.workunit.client.0.smithi045.stdout:op 4406 completed, throughput=5MB/sec 2021-07-31T02:19:46.127 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-3Hlfik-XkFUAvwS off=11704792 len=11352 2021-07-31T02:19:46.127 INFO:tasks.workunit.client.0.smithi045.stdout:op 4407 completed, throughput=5MB/sec 2021-07-31T02:19:46.128 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-0qOdOn_pg3U3y98 off=20328837 len=1513198 2021-07-31T02:19:46.130 INFO:tasks.workunit.client.0.smithi045.stdout:op 4408 completed, throughput=5MB/sec 2021-07-31T02:19:46.131 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Yrvj6B-mCGSz-3F off=731264 len=995276 2021-07-31T02:19:46.136 INFO:tasks.workunit.client.0.smithi045.stdout:op 4409 completed, throughput=5MB/sec 2021-07-31T02:19:46.136 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Q1_1zzYkd8s4MKU off=2011364 len=463595 2021-07-31T02:19:46.139 INFO:tasks.workunit.client.0.smithi045.stdout:op 4410 completed, throughput=5MB/sec 2021-07-31T02:19:46.140 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-dkWPhNDJ91y7N9y off=1842325 len=1924467 2021-07-31T02:19:46.143 INFO:tasks.workunit.client.0.smithi045.stdout:op 4411 completed, throughput=5MB/sec 2021-07-31T02:19:46.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:46 smithi154 conmon[34553]: cluster 2021-07-31T02:19:44.257107+0000 mgr.x (mgr.24542) 1395 : cluster [DBG] pgmap v1847: 9 pgs: 9 active+clean; 127 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 3.3 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2021-07-31T02:19:46.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:46 smithi045 conmon[39718]: cluster 2021-07-31T02:19:44.257107+0000 mgr.x (mgr.24542) 1395 : cluster [DBG] pgmap v1847: 9 pgs: 9 active+clean; 127 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 3.3 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2021-07-31T02:19:46.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:46 smithi045 conmon[33655]: cluster 2021-07-31T02:19:44.257107 2021-07-31T02:19:46.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:46 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 1395 : cluster [DBG] pgmap v1847: 9 pgs: 9 active+clean; 127 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 3.3 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2021-07-31T02:19:47.143 INFO:tasks.workunit.client.0.smithi045.stdout: 863: throughput=5MB/sec pending data=0 2021-07-31T02:19:47.143 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-2-gIQej5nV90LTD off=6264629 len=47013 2021-07-31T02:19:47.146 INFO:tasks.workunit.client.0.smithi045.stdout:op 4412 completed, throughput=5MB/sec 2021-07-31T02:19:47.146 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-OP5P1iwiNAAmsdj off=944158 len=117830 2021-07-31T02:19:47.148 INFO:tasks.workunit.client.0.smithi045.stdout:op 4413 completed, throughput=5MB/sec 2021-07-31T02:19:47.149 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-2JvEp9ZkDSAc53R off=7692678 len=1866670 2021-07-31T02:19:47.151 INFO:tasks.workunit.client.0.smithi045.stdout:op 4414 completed, throughput=5MB/sec 2021-07-31T02:19:47.151 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-n5RLEeqd8PA6rz- off=2357332 len=62976 2021-07-31T02:19:47.154 INFO:tasks.workunit.client.0.smithi045.stdout:op 4415 completed, throughput=5MB/sec 2021-07-31T02:19:47.154 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-yTx23hirIzDQHx8 off=633353 len=42852 2021-07-31T02:19:47.155 INFO:tasks.workunit.client.0.smithi045.stdout:op 4416 completed, throughput=5MB/sec 2021-07-31T02:19:47.155 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-GB2UbU2-Dsqzqub off=7816531 len=966733 2021-07-31T02:19:47.161 INFO:tasks.workunit.client.0.smithi045.stdout:op 4417 completed, throughput=5MB/sec 2021-07-31T02:19:47.161 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-d541QQ84EEx82jE off=5999194 len=610652 2021-07-31T02:19:47.162 INFO:tasks.workunit.client.0.smithi045.stdout:op 4418 completed, throughput=5MB/sec 2021-07-31T02:19:47.162 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Inntt5wDRR0UeZT off=1040372 len=1943605 2021-07-31T02:19:47.165 INFO:tasks.workunit.client.0.smithi045.stdout:op 4419 completed, throughput=5MB/sec 2021-07-31T02:19:48.166 INFO:tasks.workunit.client.0.smithi045.stdout: 864: throughput=5MB/sec pending data=0 2021-07-31T02:19:48.166 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-pPVlvdsML_wXxrl off=13373606 len=1352355 2021-07-31T02:19:48.168 INFO:tasks.workunit.client.0.smithi045.stdout:op 4420 completed, throughput=5MB/sec 2021-07-31T02:19:48.169 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-v2Hz-0y9r7apwPq off=1766502 len=1619389 2021-07-31T02:19:48.172 INFO:tasks.workunit.client.0.smithi045.stdout:op 4421 completed, throughput=5MB/sec 2021-07-31T02:19:48.172 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-w8B3ojABztIQfSE off=439259 len=28133 2021-07-31T02:19:48.173 INFO:tasks.workunit.client.0.smithi045.stdout:op 4422 completed, throughput=5MB/sec 2021-07-31T02:19:48.173 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-_UqK7Ce6NBvEtNh off=459646 len=1140510 2021-07-31T02:19:48.179 INFO:tasks.workunit.client.0.smithi045.stdout:op 4423 completed, throughput=5MB/sec 2021-07-31T02:19:48.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:48 smithi154 conmon[34553]: cluster 2021-07-31T02:19:46.257475+0000 mgr.x (mgr.24542) 1396 : cluster [DBG] pgmap v1848: 9 pgs: 9 active+clean; 127 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 2.3 MiB/s rd, 1.8 MiB/s wr, 3 op/s 2021-07-31T02:19:48.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:48 smithi154 conmon[34553]: 2021-07-31T02:19:48.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:48 smithi045 conmon[33655]: cluster 2021-07-31T02:19:46.257475+0000 mgr.x (mgr.24542) 1396 : 2021-07-31T02:19:48.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:48 smithi045 conmon[33655]: cluster [DBG] pgmap v1848: 9 pgs: 9 active+clean; 127 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 2.3 MiB/s rd, 1.8 MiB/s wr, 3 op/s 2021-07-31T02:19:48.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:48 smithi045 conmon[39718]: cluster 2021-07-31T02:19:46.257475+0000 mgr.x (mgr 2021-07-31T02:19:48.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:48 smithi045 conmon[39718]: .24542) 1396 : cluster [DBG] pgmap v1848: 9 pgs: 9 active+clean; 127 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 2.3 MiB/s rd, 1.8 MiB/s wr, 3 op/s 2021-07-31T02:19:49.179 INFO:tasks.workunit.client.0.smithi045.stdout: 865: throughput=5MB/sec pending data=0 2021-07-31T02:19:49.179 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-YDJabmycn9lWBEm off=1616786 len=1238883 2021-07-31T02:19:49.186 INFO:tasks.workunit.client.0.smithi045.stdout:op 4424 completed, throughput=5MB/sec 2021-07-31T02:19:49.186 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-RqRIy09u5pTWbBq off=172623 len=1106190 2021-07-31T02:19:49.187 INFO:tasks.workunit.client.0.smithi045.stdout:op 4425 completed, throughput=5MB/sec 2021-07-31T02:19:49.190 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-p25MlgT3AGP7FFQ off=6357132 len=1391150 2021-07-31T02:19:49.195 INFO:tasks.workunit.client.0.smithi045.stdout:op 4426 completed, throughput=5MB/sec 2021-07-31T02:19:49.195 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-V-gwh4YvIQzu1sf off=8996343 len=391517 2021-07-31T02:19:49.198 INFO:tasks.workunit.client.0.smithi045.stdout:op 4427 completed, throughput=5MB/sec 2021-07-31T02:19:49.198 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Pg8v0wHQ1pmEV8s off=4199754 len=618327 2021-07-31T02:19:49.199 INFO:tasks.workunit.client.0.smithi045.stdout:op 4428 completed, throughput=5MB/sec 2021-07-31T02:19:50.199 INFO:tasks.workunit.client.0.smithi045.stdout: 866: throughput=4.99MB/sec pending data=0 2021-07-31T02:19:50.200 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-0xlhC-zSWlS0q-v off=9457031 len=244013 2021-07-31T02:19:50.201 INFO:tasks.workunit.client.0.smithi045.stdout:op 4429 completed, throughput=4.99MB/sec 2021-07-31T02:19:50.201 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-TTrOmBZb85tb_mK off=15246265 len=277128 2021-07-31T02:19:50.204 INFO:tasks.workunit.client.0.smithi045.stdout:op 4430 completed, throughput=4.99MB/sec 2021-07-31T02:19:50.204 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-sK77wvRznjrCmTW off=9491968 len=14320 2021-07-31T02:19:50.204 INFO:tasks.workunit.client.0.smithi045.stdout:op 4431 completed, throughput=4.99MB/sec 2021-07-31T02:19:50.205 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-GHrdSUKz2r9lo3S off=18654463 len=1572950 2021-07-31T02:19:50.212 INFO:tasks.workunit.client.0.smithi045.stdout:op 4432 completed, throughput=5MB/sec 2021-07-31T02:19:50.212 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-_NV5kMf-faW1ihu off=5244018 len=1822336 2021-07-31T02:19:50.214 INFO:tasks.workunit.client.0.smithi045.stdout:op 4433 completed, throughput=5MB/sec 2021-07-31T02:19:50.214 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-0Cyq7Zs_6k42GIL off=11736200 len=332383 2021-07-31T02:19:50.216 INFO:tasks.workunit.client.0.smithi045.stdout:op 4434 completed, throughput=5MB/sec 2021-07-31T02:19:50.216 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-jCHPwY9X4D_rIL0 off=5223409 len=1212088 2021-07-31T02:19:50.222 INFO:tasks.workunit.client.0.smithi045.stdout:op 4435 completed, throughput=5MB/sec 2021-07-31T02:19:50.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:50 smithi154 conmon[34553]: cluster 2021-07-31T02:19:48.257975+0000 mgr.x (mgr.24542) 1397 : cluster [DBG] pgmap v1849: 9 pgs: 9 active+clean; 127 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 2.3 MiB/s rd, 1.8 MiB/s wr, 3 op/s 2021-07-31T02:19:50.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:50 smithi045 conmon[39718]: cluster 2021-07-31T02:19:48.257975+0000 mgr.x ( 2021-07-31T02:19:50.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:50 smithi045 conmon[39718]: mgr.24542) 1397 : cluster [DBG] pgmap v1849: 9 pgs: 9 active+clean; 127 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 2.3 MiB/s rd, 1.8 MiB/s wr, 3 op/s 2021-07-31T02:19:50.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:50 smithi045 conmon[33655]: cluster 2021-07-31T02:19:48.257975+0000 mgr.x (mgr. 2021-07-31T02:19:50.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:50 smithi045 conmon[33655]: 24542) 1397 : cluster [DBG] pgmap v1849: 9 pgs: 9 active+clean; 127 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 2.3 MiB/s rd, 1.8 MiB/s wr, 3 op/s 2021-07-31T02:19:51.222 INFO:tasks.workunit.client.0.smithi045.stdout: 867: throughput=4.99MB/sec pending data=0 2021-07-31T02:19:51.222 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-KKc9ktXY-0JvbWG off=13370096 len=1715476 2021-07-31T02:19:51.224 INFO:tasks.workunit.client.0.smithi045.stdout:op 4436 completed, throughput=5MB/sec 2021-07-31T02:19:51.225 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-XV18Q0K9x4UfIAb off=5923745 len=1692068 2021-07-31T02:19:51.227 INFO:tasks.workunit.client.0.smithi045.stdout:op 4437 completed, throughput=5MB/sec 2021-07-31T02:19:51.228 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-GZflFLPPgITXUKa off=2090767 len=1104752 2021-07-31T02:19:51.229 INFO:tasks.workunit.client.0.smithi045.stdout:op 4438 completed, throughput=5MB/sec 2021-07-31T02:19:51.229 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ft6OCPPQmw13hFU off=23495676 len=1522067 2021-07-31T02:19:51.239 INFO:tasks.workunit.client.0.smithi045.stdout:op 4439 completed, throughput=5MB/sec 2021-07-31T02:19:52.239 INFO:tasks.workunit.client.0.smithi045.stdout: 868: throughput=5MB/sec pending data=0 2021-07-31T02:19:52.239 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-HMQVmEVD1LUibGs off=2297538 len=135096 2021-07-31T02:19:52.242 INFO:tasks.workunit.client.0.smithi045.stdout:op 4440 completed, throughput=5MB/sec 2021-07-31T02:19:52.242 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-oZP7XOCPpJrhe3m off=1252108 len=800490 2021-07-31T02:19:52.244 INFO:tasks.workunit.client.0.smithi045.stdout:op 4441 completed, throughput=5MB/sec 2021-07-31T02:19:52.245 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-_T_xrC5YJ81uIaU off=23439525 len=174212 2021-07-31T02:19:52.248 INFO:tasks.workunit.client.0.smithi045.stdout:op 4442 completed, throughput=5MB/sec 2021-07-31T02:19:52.249 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-hrQz0O6bfLEOSha off=20771923 len=381826 2021-07-31T02:19:52.251 INFO:tasks.workunit.client.0.smithi045.stdout:op 4443 completed, throughput=5MB/sec 2021-07-31T02:19:52.252 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-7vyCpZFXnerqLS8 off=4315587 len=834481 2021-07-31T02:19:52.258 INFO:tasks.workunit.client.0.smithi045.stdout:op 4444 completed, throughput=5MB/sec 2021-07-31T02:19:52.259 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-dGgGMZJh_JuFkL7 off=14615627 len=1780440 2021-07-31T02:19:52.260 INFO:tasks.workunit.client.0.smithi045.stdout:op 4445 completed, throughput=5MB/sec 2021-07-31T02:19:52.260 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-6xEjormLNzwNV1O off=16043653 len=724588 2021-07-31T02:19:52.268 INFO:tasks.workunit.client.0.smithi045.stdout:op 4446 completed, throughput=5MB/sec 2021-07-31T02:19:52.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:52 smithi154 conmon[34553]: cluster 2021-07-31T02:19:50.258430+0000 mgr.x (mgr.24542) 1398 : cluster [DBG] pgmap v1850: 9 pgs: 9 active+clean; 127 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 3.4 MiB/s rd, 2.9 MiB/s wr, 6 op/s 2021-07-31T02:19:52.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:52 smithi045 conmon[39718]: cluster 2021-07-31T02:19:50.258430+0000 2021-07-31T02:19:52.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:52 smithi045 conmon[39718]: mgr.x (mgr.24542) 1398 : cluster [DBG] pgmap v1850: 9 pgs: 9 active+clean; 127 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 3.4 MiB/s rd, 2.9 MiB/s wr, 6 op/s 2021-07-31T02:19:52.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:52 smithi045 conmon[33655]: cluster 2021-07-31T02:19:50.258430+0000 mgr.x (mgr 2021-07-31T02:19:52.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:52 smithi045 conmon[33655]: .24542) 1398 : cluster [DBG] pgmap v1850: 9 pgs: 9 active+clean; 127 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 3.4 MiB/s rd, 2.9 MiB/s wr, 6 op/s 2021-07-31T02:19:53.266 INFO:tasks.workunit.client.0.smithi045.stdout: 869: throughput=4.99MB/sec pending data=0 2021-07-31T02:19:53.266 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-atEKW9K0DtSZmoO off=5825022 len=1053256 2021-07-31T02:19:53.272 INFO:tasks.workunit.client.0.smithi045.stdout:op 4447 completed, throughput=5MB/sec 2021-07-31T02:19:53.272 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Zj7uN3y7COYXOw6 off=2156542 len=1864990 2021-07-31T02:19:53.276 INFO:tasks.workunit.client.0.smithi045.stdout:op 4448 completed, throughput=5MB/sec 2021-07-31T02:19:53.276 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-DO2t0mVxe9ijPui off=13660739 len=1900633 2021-07-31T02:19:53.278 INFO:tasks.workunit.client.0.smithi045.stdout:op 4449 completed, throughput=5MB/sec 2021-07-31T02:19:53.278 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-_Dzq-eB12irTlFj off=5491702 len=1641889 2021-07-31T02:19:53.280 INFO:tasks.workunit.client.0.smithi045.stdout:op 4450 completed, throughput=5MB/sec 2021-07-31T02:19:54.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:19:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:19:53.565Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:19:54.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:19:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:19:53.565Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:19:54.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:19:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:19:53.565Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:19:54.281 INFO:tasks.workunit.client.0.smithi045.stdout: 870: throughput=5MB/sec pending data=0 2021-07-31T02:19:54.281 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ibEjXrvOeA5-INc off=13595251 len=520672 2021-07-31T02:19:54.282 INFO:tasks.workunit.client.0.smithi045.stdout:op 4451 completed, throughput=5MB/sec 2021-07-31T02:19:54.283 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-EDZocJC3pRABo2D off=10140052 len=1650522 2021-07-31T02:19:54.292 INFO:tasks.workunit.client.0.smithi045.stdout:op 4452 completed, throughput=5MB/sec 2021-07-31T02:19:54.292 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-OrjoM2ucoGoIIMI off=14632966 len=1063507 2021-07-31T02:19:54.296 INFO:tasks.workunit.client.0.smithi045.stdout:op 4453 completed, throughput=5MB/sec 2021-07-31T02:19:54.296 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj--PV7L_TWi0FP92p off=384679 len=554082 2021-07-31T02:19:54.301 INFO:tasks.workunit.client.0.smithi045.stdout:op 4454 completed, throughput=5MB/sec 2021-07-31T02:19:54.302 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-NdECYT2YU28GsYu off=3803766 len=157807 2021-07-31T02:19:54.303 INFO:tasks.workunit.client.0.smithi045.stdout:op 4455 completed, throughput=5MB/sec 2021-07-31T02:19:54.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:54 smithi154 conmon[34553]: cluster 2021-07-31T02:19:52.258849+0000 mgr.x (mgr.24542) 1399 : cluster [DBG] pgmap v1851: 9 pgs: 9 active+clean; 127 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:19:54.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:54 smithi045 conmon[33655]: cluster 2021-07-31T02:19:52.258849+0000 mgr.x (mgr.24542 2021-07-31T02:19:54.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:54 smithi045 conmon[33655]: ) 1399 : cluster [DBG] pgmap v1851: 9 pgs: 9 active+clean; 127 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:19:54.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:54 smithi045 conmon[39718]: cluster 2021-07-31T02:19:52.258849+0000 mgr.x (mgr 2021-07-31T02:19:54.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:54 smithi045 conmon[39718]: .24542) 1399 : cluster [DBG] pgmap v1851: 9 pgs: 9 active+clean; 127 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:19:54.975 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:19:55.253 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:19:55.253 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 34m ago 35m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:19:55.254 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 32m ago 35m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:19:55.254 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 32m ago 38m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:19:55.254 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 34m ago 40m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:19:55.255 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 34m ago 41m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:19:55.255 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 32m ago 39m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:19:55.255 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 34m ago 39m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:19:55.255 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 34m ago 35m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:19:55.255 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 32m ago 35m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:19:55.256 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 34m ago 38m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:19:55.256 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 34m ago 38m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:19:55.256 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 34m ago 37m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:19:55.256 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 34m ago 37m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:19:55.257 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 32m ago 37m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:19:55.257 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 32m ago 36m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:19:55.257 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 32m ago 36m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:19:55.257 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 32m ago 36m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:19:55.258 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 32m ago 35m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:19:55.304 INFO:tasks.workunit.client.0.smithi045.stdout: 871: throughput=4.99MB/sec pending data=0 2021-07-31T02:19:55.304 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-FwfjIMujwIAH_JT off=20189749 len=1839878 2021-07-31T02:19:55.305 INFO:tasks.workunit.client.0.smithi045.stdout:op 4456 completed, throughput=5MB/sec 2021-07-31T02:19:55.305 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-uv-AUS495Bp9o2U off=4183450 len=787542 2021-07-31T02:19:55.310 INFO:tasks.workunit.client.0.smithi045.stdout:op 4457 completed, throughput=5MB/sec 2021-07-31T02:19:55.310 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-QqPQkiO8n8dzikK off=14567098 len=1469240 2021-07-31T02:19:55.318 INFO:tasks.workunit.client.0.smithi045.stdout:op 4458 completed, throughput=5MB/sec 2021-07-31T02:19:55.318 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-IIqg7uZ9Ypo6d0W off=9827727 len=512918 2021-07-31T02:19:55.319 INFO:tasks.workunit.client.0.smithi045.stdout:op 4459 completed, throughput=5MB/sec 2021-07-31T02:19:55.319 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-OocAU1bHriKzI03 off=665898 len=3064 2021-07-31T02:19:55.321 INFO:tasks.workunit.client.0.smithi045.stdout:op 4460 completed, throughput=5MB/sec 2021-07-31T02:19:55.321 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-QB7W77eH2dakLFU off=23617909 len=630155 2021-07-31T02:19:55.322 INFO:tasks.workunit.client.0.smithi045.stdout:op 4461 completed, throughput=5MB/sec 2021-07-31T02:19:55.322 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj--mNDv06fFkBxLjh off=3425968 len=1758131 2021-07-31T02:19:55.325 INFO:tasks.workunit.client.0.smithi045.stdout:op 4462 completed, throughput=5MB/sec 2021-07-31T02:19:55.591 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:19:55.591 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:19:55.591 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:19:55.592 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:19:55.592 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:19:55.592 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:19:55.592 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:19:55.593 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:19:55.593 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:19:55.593 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:19:55.593 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:19:55.594 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:19:55.594 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:19:55.594 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:19:55.594 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:19:55.594 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:19:55.595 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:19:56.325 INFO:tasks.workunit.client.0.smithi045.stdout: 872: throughput=5MB/sec pending data=0 2021-07-31T02:19:56.326 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-9pkFMh9r4UsDbcX off=13231690 len=916870 2021-07-31T02:19:56.331 INFO:tasks.workunit.client.0.smithi045.stdout:op 4463 completed, throughput=5MB/sec 2021-07-31T02:19:56.331 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-4cLWlsnT-lhalfn off=6865187 len=1360519 2021-07-31T02:19:56.337 INFO:tasks.workunit.client.0.smithi045.stdout:op 4464 completed, throughput=5MB/sec 2021-07-31T02:19:56.338 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-5Pv47GDeT4RygLk off=2418592 len=926022 2021-07-31T02:19:56.342 INFO:tasks.workunit.client.0.smithi045.stdout:op 4465 completed, throughput=5MB/sec 2021-07-31T02:19:56.343 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj--mmTxqMnK49-ilb off=1536869 len=1448850 2021-07-31T02:19:56.352 INFO:tasks.workunit.client.0.smithi045.stdout:op 4466 completed, throughput=5MB/sec 2021-07-31T02:19:56.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:56 smithi154 conmon[34553]: cluster 2021-07-31T02:19:54.259224+0000 mgr.x (mgr.24542) 1400 : cluster [DBG] pgmap v1852: 9 pgs: 9 active+clean; 127 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 3.2 MiB/s rd, 3.3 MiB/s wr, 6 op/s 2021-07-31T02:19:56.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:56 smithi154 conmon[34553]: audit 2021-07-31T02:19:54.963334+0000 mgr.x (mgr.24542) 1401 : audit [DBG] from='client.26146 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:19:56.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:56 smithi154 conmon[34553]: audit 2021-07-31T02:19:55.245634+0000 2021-07-31T02:19:56.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:56 smithi154 conmon[34553]: mgr.x (mgr.24542) 1402 : audit [DBG] from='client.16245 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:19:56.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:56 smithi154 conmon[34553]: audit 2021-07- 2021-07-31T02:19:56.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:56 smithi154 conmon[34553]: 31T02:19:55.591303+0000 2021-07-31T02:19:56.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:56 smithi154 conmon[34553]: mon.c (mon.1) 89 : audit [DBG] 2021-07-31T02:19:56.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:56 smithi154 conmon[34553]: from='client.? 172.21.15.45:0/2721888767' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:19:56.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:56 smithi045 conmon[33655]: cluster 2021-07-31T02:19:54.259224+0000 mgr.x (mgr.24542) 1400 : 2021-07-31T02:19:56.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:56 smithi045 conmon[33655]: cluster [DBG] pgmap v1852: 9 pgs: 9 active+clean; 127 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 3.2 MiB/s rd, 3.3 MiB/s wr, 6 op/s 2021-07-31T02:19:56.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:56 smithi045 conmon[33655]: audit 2021-07-31T02:19:54.963334+0000 mgr.x (mgr.24542) 1401 : audit [DBG] from='client.26146 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:19:56.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:56 smithi045 conmon[33655]: audit 2021-07-31T02:19:55.245634 2021-07-31T02:19:56.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:56 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 1402 : audit [DBG] from='client.16245 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:19:56.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:56 smithi045 conmon[33655]: audit 2021- 2021-07-31T02:19:56.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:56 smithi045 conmon[33655]: 07-31T02:19:55.591303+0000 mon.c (mon 2021-07-31T02:19:56.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:56 smithi045 conmon[33655]: .1) 89 : audit [DBG] from='client.? 172.21.15.45:0/2721888767' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:19:56.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:56 smithi045 conmon[33655]: 2021-07-31T02:19:56.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:56 smithi045 conmon[39718]: cluster 2021-07-31T02:19: 2021-07-31T02:19:56.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:56 smithi045 conmon[39718]: 54.259224+0000 mgr.x (mgr.24542) 1400 : cluster [DBG] pgmap v1852: 9 pgs: 9 active+clean; 127 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 3.2 MiB/s rd, 3.3 MiB/s wr, 6 op/s 2021-07-31T02:19:56.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:56 smithi045 conmon[39718]: audit 2021-07 2021-07-31T02:19:56.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:56 smithi045 conmon[39718]: -31T02:19:54. 2021-07-31T02:19:56.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:56 smithi045 conmon[39718]: 963334+0000 mgr.x (mgr.24542 2021-07-31T02:19:56.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:56 smithi045 conmon[39718]: ) 1401 : audit [DBG] from='client.26146 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:19:56.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:56 smithi045 conmon[39718]: 2021-07-31T02:19:56.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:56 smithi045 conmon[39718]: audit 2021-07-31T02:19:56.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:56 smithi045 conmon[39718]: 2021-07- 2021-07-31T02:19:56.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:56 smithi045 conmon[39718]: 31T02:19:55.245634+0000 2021-07-31T02:19:56.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:56 smithi045 conmon[39718]: mgr.x (mgr.24542) 1402 2021-07-31T02:19:56.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:56 smithi045 conmon[39718]: : audit [DBG] from='client.16245 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:19:56.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:56 smithi045 conmon[39718]: audit 2021-07-31T02:19:56.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:56 smithi045 conmon[39718]: 2021-07 2021-07-31T02:19:56.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:56 smithi045 conmon[39718]: -31T02:19:55 2021-07-31T02:19:56.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:56 smithi045 conmon[39718]: .591303+0000 mon.c (mon. 2021-07-31T02:19:56.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:56 smithi045 conmon[39718]: 1) 89 : audit [DBG] 2021-07-31T02:19:56.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:56 smithi045 conmon[39718]: from='client.? 172.21.15.45:0/2721888767' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:19:57.352 INFO:tasks.workunit.client.0.smithi045.stdout: 873: throughput=5MB/sec pending data=0 2021-07-31T02:19:57.353 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-i2oiVef6H4enCIK off=12431721 len=1478021 2021-07-31T02:19:57.362 INFO:tasks.workunit.client.0.smithi045.stdout:op 4467 completed, throughput=5MB/sec 2021-07-31T02:19:57.362 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-0JqYV2AAOmTbtMO off=6703328 len=139187 2021-07-31T02:19:57.363 INFO:tasks.workunit.client.0.smithi045.stdout:op 4468 completed, throughput=5MB/sec 2021-07-31T02:19:57.363 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-O0s_M-_iEM4lMIx off=133463 len=1738569 2021-07-31T02:19:57.366 INFO:tasks.workunit.client.0.smithi045.stdout:op 4469 completed, throughput=5MB/sec 2021-07-31T02:19:57.366 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-GMQEUAqZbEyx1nL off=16757432 len=1175531 2021-07-31T02:19:57.367 INFO:tasks.workunit.client.0.smithi045.stdout:op 4470 completed, throughput=5MB/sec 2021-07-31T02:19:58.368 INFO:tasks.workunit.client.0.smithi045.stdout: 874: throughput=4.99MB/sec pending data=0 2021-07-31T02:19:58.368 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-I9oxKsF_oPJ0K76 off=4630599 len=2088479 2021-07-31T02:19:58.372 INFO:tasks.workunit.client.0.smithi045.stdout:op 4471 completed, throughput=5MB/sec 2021-07-31T02:19:58.372 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-t2nfwWf-IGsLD-A off=7605765 len=1854656 2021-07-31T02:19:58.374 INFO:tasks.workunit.client.0.smithi045.stdout:op 4472 completed, throughput=5MB/sec 2021-07-31T02:19:58.374 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-l1RnE-_X3a7MNGc off=1407416 len=1734748 2021-07-31T02:19:58.382 INFO:tasks.workunit.client.0.smithi045.stdout:op 4473 completed, throughput=5MB/sec 2021-07-31T02:19:58.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:19:58 smithi154 conmon[34553]: cluster 2021-07-31T02:19:56.259707+0000 mgr.x (mgr.24542) 1403 : cluster [DBG] pgmap v1853: 9 pgs: 9 active+clean; 127 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2021-07-31T02:19:58.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:58 smithi045 conmon[39718]: cluster 2021-07-31T02:19:56.259707 2021-07-31T02:19:58.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:19:58 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 1403 : cluster [DBG] pgmap v1853: 9 pgs: 9 active+clean; 127 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2021-07-31T02:19:58.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:58 smithi045 conmon[33655]: cluster 2021-07-31T02:19:56.259707 2021-07-31T02:19:58.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:19:58 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 1403 : cluster [DBG] pgmap v1853: 9 pgs: 9 active+clean; 127 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2021-07-31T02:19:59.383 INFO:tasks.workunit.client.0.smithi045.stdout: 875: throughput=4.99MB/sec pending data=0 2021-07-31T02:19:59.383 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-GvdpBbL2o9efBbt off=8009583 len=647334 2021-07-31T02:19:59.384 INFO:tasks.workunit.client.0.smithi045.stdout:op 4474 completed, throughput=5MB/sec 2021-07-31T02:19:59.384 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Imfe0JtG2PiMKvn off=5805246 len=539363 2021-07-31T02:19:59.389 INFO:tasks.workunit.client.0.smithi045.stdout:op 4475 completed, throughput=5MB/sec 2021-07-31T02:19:59.389 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-UfNa_rdxhEtZBKA off=3368634 len=1026300 2021-07-31T02:19:59.391 INFO:tasks.workunit.client.0.smithi045.stdout:op 4476 completed, throughput=5MB/sec 2021-07-31T02:19:59.392 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-HqytkLDPOphdOea off=15407874 len=73349 2021-07-31T02:19:59.394 INFO:tasks.workunit.client.0.smithi045.stdout:op 4477 completed, throughput=5MB/sec 2021-07-31T02:19:59.395 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-f1mYcrPSpI5eg5r off=17410475 len=1894285 2021-07-31T02:19:59.404 INFO:tasks.workunit.client.0.smithi045.stdout:op 4478 completed, throughput=5MB/sec 2021-07-31T02:19:59.405 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-GQ-y0XK-c7q596y off=9200261 len=1953731 2021-07-31T02:19:59.406 INFO:tasks.workunit.client.0.smithi045.stdout:op 4479 completed, throughput=5MB/sec 2021-07-31T02:20:00.407 INFO:tasks.workunit.client.0.smithi045.stdout: 876: throughput=5MB/sec pending data=0 2021-07-31T02:20:00.407 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-wdltX2BGdeUMnfY off=3855230 len=30121 2021-07-31T02:20:00.410 INFO:tasks.workunit.client.0.smithi045.stdout:op 4480 completed, throughput=5MB/sec 2021-07-31T02:20:00.410 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-tq4KWVT0xTnx45J off=1190765 len=1231970 2021-07-31T02:20:00.417 INFO:tasks.workunit.client.0.smithi045.stdout:op 4481 completed, throughput=5MB/sec 2021-07-31T02:20:00.417 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-MjUU8wXuxqhih3p off=6376104 len=1285279 2021-07-31T02:20:00.423 INFO:tasks.workunit.client.0.smithi045.stdout:op 4482 completed, throughput=5MB/sec 2021-07-31T02:20:00.424 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-f2-3rC711ajP1no off=21912151 len=1372580 2021-07-31T02:20:00.429 INFO:tasks.workunit.client.0.smithi045.stdout:op 4483 completed, throughput=5MB/sec 2021-07-31T02:20:00.429 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-AiV0wthVsRplX6Z off=0 len=1085908 2021-07-31T02:20:00.431 INFO:tasks.workunit.client.0.smithi045.stdout:op 4484 completed, throughput=5MB/sec 2021-07-31T02:20:00.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:00 smithi154 conmon[34553]: cluster 2021-07-31T02:19:58.260204+0000 mgr.x (mgr.24542) 1404 : cluster [DBG] pgmap v1854: 9 pgs: 9 active+clean; 127 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2021-07-31T02:20:00.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:00 smithi154 conmon[34553]: cluster 2021-07-31T02 2021-07-31T02:20:00.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:00 smithi154 conmon[34553]: :20:00.000174+0000 mon.a (mon.0) 1895 : cluster [INF] overall HEALTH_OK 2021-07-31T02:20:00.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:00 smithi045 conmon[39718]: cluster 2021-07-31T02:19:58. 2021-07-31T02:20:00.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:00 smithi045 conmon[39718]: 260204+0000 mgr.x (mgr.24542) 1404 : cluster [DBG] pgmap v1854: 9 pgs: 9 active+clean; 127 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2021-07-31T02:20:00.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:00 smithi045 conmon[39718]: cluster 2021-07-31T02:20:00.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:00 smithi045 conmon[39718]: 2021-07-31T02:20:00.000174+0000 mon.a (mon 2021-07-31T02:20:00.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:00 smithi045 conmon[39718]: .0) 1895 : cluster [INF] overall HEALTH_OK 2021-07-31T02:20:00.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:00 smithi045 conmon[33655]: cluster 2021-07-31T02:19:58.260204+0000 mgr.x (mgr. 2021-07-31T02:20:00.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:00 smithi045 conmon[33655]: 24542) 1404 : cluster [DBG] pgmap v1854: 9 pgs: 9 active+clean; 127 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2021-07-31T02:20:00.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:00 smithi045 conmon[33655]: cluster 2021-07-31T 2021-07-31T02:20:00.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:00 smithi045 conmon[33655]: 02:20:00.000174+0000 mon.a (mon.0) 1895 : cluster [INF] overall HEALTH_OK 2021-07-31T02:20:01.432 INFO:tasks.workunit.client.0.smithi045.stdout: 877: throughput=5MB/sec pending data=0 2021-07-31T02:20:01.433 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-V5eVYOxVkRCLrA6 off=2965432 len=844668 2021-07-31T02:20:01.434 INFO:tasks.workunit.client.0.smithi045.stdout:op 4485 completed, throughput=5MB/sec 2021-07-31T02:20:01.435 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-SRzLytsqmpMCoH4 off=6859403 len=615113 2021-07-31T02:20:01.436 INFO:tasks.workunit.client.0.smithi045.stdout:op 4486 completed, throughput=5MB/sec 2021-07-31T02:20:01.436 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-3ZkEklhlhKEmuTx off=1256970 len=299700 2021-07-31T02:20:01.439 INFO:tasks.workunit.client.0.smithi045.stdout:op 4487 completed, throughput=5MB/sec 2021-07-31T02:20:01.440 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-uuONZP4Igvp31fY off=454340 len=1851492 2021-07-31T02:20:01.448 INFO:tasks.workunit.client.0.smithi045.stdout:op 4488 completed, throughput=5MB/sec 2021-07-31T02:20:01.448 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-6oMo8wThX_yrJUz off=6512595 len=1186207 2021-07-31T02:20:01.449 INFO:tasks.workunit.client.0.smithi045.stdout:op 4489 completed, throughput=5MB/sec 2021-07-31T02:20:02.450 INFO:tasks.workunit.client.0.smithi045.stdout: 878: throughput=4.99MB/sec pending data=0 2021-07-31T02:20:02.450 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ncaKYKfTSNZl1Ku off=8325685 len=418525 2021-07-31T02:20:02.453 INFO:tasks.workunit.client.0.smithi045.stdout:op 4490 completed, throughput=5MB/sec 2021-07-31T02:20:02.454 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ZjaVv8izkhzdEVh off=7152508 len=453824 2021-07-31T02:20:02.458 INFO:tasks.workunit.client.0.smithi045.stdout:op 4491 completed, throughput=5MB/sec 2021-07-31T02:20:02.458 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-g6kk0nFLpf3GvSd off=1452092 len=407471 2021-07-31T02:20:02.459 INFO:tasks.workunit.client.0.smithi045.stdout:op 4492 completed, throughput=5MB/sec 2021-07-31T02:20:02.459 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Gr1nrNmHYqxcGT3 off=17495548 len=1694633 2021-07-31T02:20:02.469 INFO:tasks.workunit.client.0.smithi045.stdout:op 4493 completed, throughput=5MB/sec 2021-07-31T02:20:02.470 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-y8QdXiPP6O35t8j off=5128307 len=1440020 2021-07-31T02:20:02.472 INFO:tasks.workunit.client.0.smithi045.stdout:op 4494 completed, throughput=5MB/sec 2021-07-31T02:20:02.472 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-eq2oSyb6P5EMqOG off=761641 len=1781131 2021-07-31T02:20:02.474 INFO:tasks.workunit.client.0.smithi045.stdout:op 4495 completed, throughput=5MB/sec 2021-07-31T02:20:02.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:02 smithi154 conmon[34553]: cluster 2021-07-31T02:20:00.260649+0000 mgr.x (mgr.24542) 1405 : cluster [DBG] pgmap v1855: 9 pgs: 9 active+clean; 127 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 3.4 MiB/s rd, 3.0 MiB/s wr, 6 op/s 2021-07-31T02:20:02.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:02 smithi045 conmon[39718]: cluster 2021-07-31T02:20: 2021-07-31T02:20:02.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:02 smithi045 conmon[39718]: 00.260649+0000 mgr.x (mgr.24542) 1405 : cluster [DBG] pgmap v1855: 9 pgs: 9 active+clean; 127 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 3.4 MiB/s rd, 3.0 MiB/s wr, 6 op/s 2021-07-31T02:20:02.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:02 smithi045 conmon[33655]: cluster 2021-07-31T02:20:00.260649+0000 mgr.x (mgr.24542) 1405 2021-07-31T02:20:02.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:02 smithi045 conmon[33655]: : cluster [DBG] pgmap v1855: 9 pgs: 9 active+clean; 127 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 3.4 MiB/s rd, 3.0 MiB/s wr, 6 op/s 2021-07-31T02:20:03.479 INFO:tasks.workunit.client.0.smithi045.stdout: 879: throughput=5MB/sec pending data=0 2021-07-31T02:20:03.479 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-7xqlvf1EOGRIR_V off=2474831 len=1445628 2021-07-31T02:20:03.482 INFO:tasks.workunit.client.0.smithi045.stdout:op 4496 completed, throughput=5MB/sec 2021-07-31T02:20:03.482 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-g_xepwT1y5Wx7Jj off=14682230 len=77601 2021-07-31T02:20:03.485 INFO:tasks.workunit.client.0.smithi045.stdout:op 4497 completed, throughput=5MB/sec 2021-07-31T02:20:03.485 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-_-UTjSB83HCHTEp off=3633448 len=940376 2021-07-31T02:20:03.487 INFO:tasks.workunit.client.0.smithi045.stdout:op 4498 completed, throughput=5MB/sec 2021-07-31T02:20:03.487 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-fTtVTaCAy_N8S1j off=250541 len=1954654 2021-07-31T02:20:03.498 INFO:tasks.workunit.client.0.smithi045.stdout:op 4499 completed, throughput=5MB/sec 2021-07-31T02:20:04.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:20:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:20:03.565Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:20:04.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:20:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:20:03.565Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:20:04.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:20:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:20:03.565Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:20:04.499 INFO:tasks.workunit.client.0.smithi045.stdout: 880: throughput=4.99MB/sec pending data=0 2021-07-31T02:20:04.499 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-A8gQRA-62L9OkLL off=4072801 len=2047800 2021-07-31T02:20:04.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:04 smithi154 conmon[34553]: cluster 2021-07-31T02:20:02.261176+0000 mgr.x (mgr.24542) 1406 : cluster [DBG] pgmap v1856: 9 pgs: 9 active+clean; 127 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 2.3 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2021-07-31T02:20:04.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:04 smithi154 conmon[34553]: 2021-07-31T02:20:04.510 INFO:tasks.workunit.client.0.smithi045.stdout:op 4500 completed, throughput=5MB/sec 2021-07-31T02:20:04.511 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ZjaVv8izkhzdEVh off=5524268 len=233940 2021-07-31T02:20:04.514 INFO:tasks.workunit.client.0.smithi045.stdout:op 4501 completed, throughput=5MB/sec 2021-07-31T02:20:04.514 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Yc2IfW7gLVVciBp off=22247341 len=929860 2021-07-31T02:20:04.519 INFO:tasks.workunit.client.0.smithi045.stdout:op 4502 completed, throughput=5MB/sec 2021-07-31T02:20:04.519 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-If7vMeF2hisCmpq off=6574734 len=1120827 2021-07-31T02:20:04.524 INFO:tasks.workunit.client.0.smithi045.stdout:op 4503 completed, throughput=5MB/sec 2021-07-31T02:20:04.525 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-m0oy3ZTrArURMNS off=8915299 len=647484 2021-07-31T02:20:04.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:04 smithi045 conmon[39718]: cluster 2021-07-31T02:20:02.261176+0000 mgr.x (mgr.24542) 1406 : cluster [DBG] pgmap v1856: 9 pgs: 9 active+clean; 127 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 2.3 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2021-07-31T02:20:04.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:04 smithi045 conmon[33655]: cluster 2021-07-31T02:20:02.261176+0000 mgr.x (mgr.24542) 1406 : cluster 2021-07-31T02:20:04.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:04 smithi045 conmon[33655]: [DBG] pgmap v1856: 9 pgs: 9 active+clean; 127 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 2.3 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2021-07-31T02:20:04.527 INFO:tasks.workunit.client.0.smithi045.stdout:op 4504 completed, throughput=5MB/sec 2021-07-31T02:20:04.527 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-fZqiYVnEh683Rm1 off=5965699 len=1448627 2021-07-31T02:20:04.533 INFO:tasks.workunit.client.0.smithi045.stdout:op 4505 completed, throughput=5MB/sec 2021-07-31T02:20:05.535 INFO:tasks.workunit.client.0.smithi045.stdout: 881: throughput=5MB/sec pending data=0 2021-07-31T02:20:05.535 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-i2oiVef6H4enCIK off=9047886 len=407508 2021-07-31T02:20:05.536 INFO:tasks.workunit.client.0.smithi045.stdout:op 4506 completed, throughput=5MB/sec 2021-07-31T02:20:05.536 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-4fpWgPZApTEWcnO off=2944909 len=1377975 2021-07-31T02:20:05.542 INFO:tasks.workunit.client.0.smithi045.stdout:op 4507 completed, throughput=5MB/sec 2021-07-31T02:20:05.542 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj--FjmGllOdrmT1Oi off=1201995 len=1717986 2021-07-31T02:20:05.544 INFO:tasks.workunit.client.0.smithi045.stdout:op 4508 completed, throughput=5MB/sec 2021-07-31T02:20:05.545 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-h_expECW2rMPiCj off=2934366 len=928905 2021-07-31T02:20:05.546 INFO:tasks.workunit.client.0.smithi045.stdout:op 4509 completed, throughput=5MB/sec 2021-07-31T02:20:06.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:06 smithi154 conmon[34553]: cluster 2021-07-31T02:20:04.261804+0000 mgr.x (mgr.24542) 1407 : cluster [DBG] pgmap v1857: 9 pgs: 9 active+clean; 127 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 3.2 MiB/s rd, 3.1 MiB/s wr, 6 op/s 2021-07-31T02:20:06.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:06 smithi154 conmon[34553]: 2021-07-31T02:20:06.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:06 smithi045 conmon[39718]: cluster 2021-07-31T02:20:04 2021-07-31T02:20:06.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:06 smithi045 conmon[39718]: .261804+0000 mgr.x (mgr.24542) 1407 : cluster [DBG] pgmap v1857: 9 pgs: 9 active+clean; 127 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 3.2 MiB/s rd, 3.1 MiB/s wr, 6 op/s 2021-07-31T02:20:06.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:06 smithi045 conmon[33655]: cluster 2021-07-31T02:20:04.261804+0000 mgr.x (mgr.24542) 1407 : cluster 2021-07-31T02:20:06.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:06 smithi045 conmon[33655]: [DBG] pgmap v1857: 9 pgs: 9 active+clean; 127 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 3.2 MiB/s rd, 3.1 MiB/s wr, 6 op/s 2021-07-31T02:20:06.547 INFO:tasks.workunit.client.0.smithi045.stdout: 882: throughput=4.99MB/sec pending data=0 2021-07-31T02:20:06.547 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Y--VqpfrPyqW1Ev off=4902859 len=817550 2021-07-31T02:20:06.553 INFO:tasks.workunit.client.0.smithi045.stdout:op 4510 completed, throughput=5MB/sec 2021-07-31T02:20:06.553 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-MusnFuhxmdATLaO off=143199 len=1325550 2021-07-31T02:20:06.559 INFO:tasks.workunit.client.0.smithi045.stdout:op 4511 completed, throughput=5MB/sec 2021-07-31T02:20:06.560 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-BHQ_QE2vYNJps1u off=6031325 len=1187394 2021-07-31T02:20:06.565 INFO:tasks.workunit.client.0.smithi045.stdout:op 4512 completed, throughput=5MB/sec 2021-07-31T02:20:06.566 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-IpyF5hEFjsPRiGx off=2882699 len=143537 2021-07-31T02:20:06.570 INFO:tasks.workunit.client.0.smithi045.stdout:op 4513 completed, throughput=5MB/sec 2021-07-31T02:20:06.570 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-WooGvr_t-SAp8Zl off=3817454 len=1451107 2021-07-31T02:20:06.575 INFO:tasks.workunit.client.0.smithi045.stdout:op 4514 completed, throughput=5MB/sec 2021-07-31T02:20:06.575 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-lZwxqCTEfvX-F4w off=5922019 len=66688 2021-07-31T02:20:06.577 INFO:tasks.workunit.client.0.smithi045.stdout:op 4515 completed, throughput=5MB/sec 2021-07-31T02:20:06.577 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-L6j16PuWTAkNv4U off=630494 len=665363 2021-07-31T02:20:06.578 INFO:tasks.workunit.client.0.smithi045.stdout:op 4516 completed, throughput=5MB/sec 2021-07-31T02:20:07.579 INFO:tasks.workunit.client.0.smithi045.stdout: 883: throughput=5MB/sec pending data=0 2021-07-31T02:20:07.580 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-YnK3OdVxonqVhmo off=3861083 len=818133 2021-07-31T02:20:07.581 INFO:tasks.workunit.client.0.smithi045.stdout:op 4517 completed, throughput=5MB/sec 2021-07-31T02:20:07.581 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-NzRTQLlxDhFF9j8 off=4492796 len=1971392 2021-07-31T02:20:07.591 INFO:tasks.workunit.client.0.smithi045.stdout:op 4518 completed, throughput=5MB/sec 2021-07-31T02:20:07.592 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-o_5URweAS_YyyKk off=8149123 len=1688605 2021-07-31T02:20:07.600 INFO:tasks.workunit.client.0.smithi045.stdout:op 4519 completed, throughput=5MB/sec 2021-07-31T02:20:07.600 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-7QomePAvHxGiZod off=5545751 len=523777 2021-07-31T02:20:07.604 INFO:tasks.workunit.client.0.smithi045.stdout:op 4520 completed, throughput=5MB/sec 2021-07-31T02:20:08.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:08 smithi154 conmon[34553]: cluster 2021-07-31T02:20:06.262210+0000 mgr.x (mgr.24542) 1408 : cluster [DBG] pgmap v1858: 9 pgs: 9 active+clean; 127 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2021-07-31T02:20:08.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:08 smithi045 conmon[39718]: cluster 2021-07-31T02:20:06.262210+0000 mgr.x ( 2021-07-31T02:20:08.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:08 smithi045 conmon[39718]: mgr.24542) 1408 : cluster [DBG] pgmap v1858: 9 pgs: 9 active+clean; 127 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2021-07-31T02:20:08.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:08 smithi045 conmon[33655]: cluster 2021-07-31T02:20:06.262210+0000 mgr.x (mgr 2021-07-31T02:20:08.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:08 smithi045 conmon[33655]: .24542) 1408 : cluster [DBG] pgmap v1858: 9 pgs: 9 active+clean; 127 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2021-07-31T02:20:08.604 INFO:tasks.workunit.client.0.smithi045.stdout: 884: throughput=4.99MB/sec pending data=0 2021-07-31T02:20:08.605 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-7Lf6dWTWef4HBys off=4061562 len=1392078 2021-07-31T02:20:08.612 INFO:tasks.workunit.client.0.smithi045.stdout:op 4521 completed, throughput=5MB/sec 2021-07-31T02:20:08.613 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-GGW8byt2_9iHujH off=3582865 len=248730 2021-07-31T02:20:08.613 INFO:tasks.workunit.client.0.smithi045.stdout:op 4522 completed, throughput=5MB/sec 2021-07-31T02:20:08.614 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-iuCyqHc3SvbLAi1 off=10009220 len=1257592 2021-07-31T02:20:08.620 INFO:tasks.workunit.client.0.smithi045.stdout:op 4523 completed, throughput=5MB/sec 2021-07-31T02:20:08.623 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-tHgGIK6NKykdNRh off=10518196 len=1823216 2021-07-31T02:20:08.623 INFO:tasks.workunit.client.0.smithi045.stdout:op 4524 completed, throughput=5MB/sec 2021-07-31T02:20:08.624 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-R2cpbEdL12emhVP off=4733618 len=1236105 2021-07-31T02:20:08.624 INFO:tasks.workunit.client.0.smithi045.stdout:op 4525 completed, throughput=5MB/sec 2021-07-31T02:20:09.624 INFO:tasks.workunit.client.0.smithi045.stdout: 885: throughput=5MB/sec pending data=0 2021-07-31T02:20:09.625 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-2RS-14CTvdAc-Dp off=1851148 len=241254 2021-07-31T02:20:09.626 INFO:tasks.workunit.client.0.smithi045.stdout:op 4526 completed, throughput=5MB/sec 2021-07-31T02:20:09.626 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-vfvgg1TCzW4y2-L off=17792837 len=1553888 2021-07-31T02:20:09.633 INFO:tasks.workunit.client.0.smithi045.stdout:op 4527 completed, throughput=5MB/sec 2021-07-31T02:20:09.633 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-DaULm6W74aTTZjz off=6777251 len=1375644 2021-07-31T02:20:09.636 INFO:tasks.workunit.client.0.smithi045.stdout:op 4528 completed, throughput=5MB/sec 2021-07-31T02:20:09.636 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-G8kAbr4fjxpMiQL off=2485080 len=426419 2021-07-31T02:20:09.640 INFO:tasks.workunit.client.0.smithi045.stdout:op 4529 completed, throughput=5MB/sec 2021-07-31T02:20:09.641 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-4LgNtFgvDttT66V off=2460919 len=1821582 2021-07-31T02:20:09.643 INFO:tasks.workunit.client.0.smithi045.stdout:op 4530 completed, throughput=5MB/sec 2021-07-31T02:20:10.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:10 smithi154 conmon[34553]: cluster 2021-07-31T02:20:08.262720+0000 mgr.x (mgr.24542) 1409 : cluster [DBG] pgmap v1859: 9 pgs: 9 active+clean; 127 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2021-07-31T02:20:10.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:10 smithi045 conmon[33655]: cluster 2021-07-31T02:20:08.262720+0000 mgr.x (mgr.24542) 2021-07-31T02:20:10.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:10 smithi045 conmon[33655]: 1409 : cluster [DBG] pgmap v1859: 9 pgs: 9 active+clean; 127 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2021-07-31T02:20:10.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:10 smithi045 conmon[39718]: cluster 2021-07-31T02:20:08.262720+0000 mgr.x (mgr.24542) 1409 : cluster 2021-07-31T02:20:10.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:10 smithi045 conmon[39718]: [DBG] pgmap v1859: 9 pgs: 9 active+clean; 127 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2021-07-31T02:20:10.643 INFO:tasks.workunit.client.0.smithi045.stdout: 886: throughput=5MB/sec pending data=0 2021-07-31T02:20:10.644 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-zIj66QbM1TkikLE off=14161441 len=1170146 2021-07-31T02:20:10.645 INFO:tasks.workunit.client.0.smithi045.stdout:op 4531 completed, throughput=5MB/sec 2021-07-31T02:20:10.645 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-lC2Okc2xkY5fsYz off=20370703 len=2081697 2021-07-31T02:20:10.649 INFO:tasks.workunit.client.0.smithi045.stdout:op 4532 completed, throughput=5MB/sec 2021-07-31T02:20:10.649 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-8SMip5mRoIRA76m off=4677706 len=1099366 2021-07-31T02:20:10.650 INFO:tasks.workunit.client.0.smithi045.stdout:op 4533 completed, throughput=5MB/sec 2021-07-31T02:20:10.651 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-nx5vNLzyvpIIdkp off=8764075 len=144740 2021-07-31T02:20:10.654 INFO:tasks.workunit.client.0.smithi045.stdout:op 4534 completed, throughput=5MB/sec 2021-07-31T02:20:11.654 INFO:tasks.workunit.client.0.smithi045.stdout: 887: throughput=4.99MB/sec pending data=0 2021-07-31T02:20:11.654 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-syOpv_jxV8dkhUz off=6072180 len=310095 2021-07-31T02:20:11.655 INFO:tasks.workunit.client.0.smithi045.stdout:op 4535 completed, throughput=4.99MB/sec 2021-07-31T02:20:11.656 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-YYAj3Cl7qhuULH3 off=2509934 len=1012003 2021-07-31T02:20:11.657 INFO:tasks.workunit.client.0.smithi045.stdout:op 4536 completed, throughput=5MB/sec 2021-07-31T02:20:11.658 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-RQGVf8O16UeUcIs off=1043970 len=1175236 2021-07-31T02:20:11.664 INFO:tasks.workunit.client.0.smithi045.stdout:op 4537 completed, throughput=5MB/sec 2021-07-31T02:20:11.664 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-7ZJfCUBVRH8UWMw off=3529253 len=2036646 2021-07-31T02:20:11.667 INFO:tasks.workunit.client.0.smithi045.stdout:op 4538 completed, throughput=5MB/sec 2021-07-31T02:20:11.667 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-lzjgC0oRJLYfir2 off=8191941 len=911093 2021-07-31T02:20:11.670 INFO:tasks.workunit.client.0.smithi045.stdout:op 4539 completed, throughput=5MB/sec 2021-07-31T02:20:12.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:12 smithi154 conmon[34553]: cluster 2021-07-31T02:20:10.263329+0000 mgr.x (mgr.24542) 1410 : cluster [DBG] pgmap v1860: 9 pgs: 9 active+clean; 127 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 2.6 MiB/s rd, 3.7 MiB/s wr, 6 op/s 2021-07-31T02:20:12.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:12 smithi154 conmon[34553]: 2021-07-31T02:20:12.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:12 smithi045 conmon[33655]: cluster 2021-07-31T02:20:10.263329+0000 mgr.x (mgr.24542) 1410 : cluster [DBG] pgmap v1860: 9 pgs: 9 active+clean; 127 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 2.6 MiB/s rd, 3.7 MiB/s wr, 6 op/s 2021-07-31T02:20:12.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:12 smithi045 conmon[39718]: cluster 2021-07-31T02:20:10.263329+0000 mgr.x ( 2021-07-31T02:20:12.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:12 smithi045 conmon[39718]: mgr.24542) 1410 : cluster [DBG] pgmap v1860: 9 pgs: 9 active+clean; 127 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 2.6 MiB/s rd, 3.7 MiB/s wr, 6 op/s 2021-07-31T02:20:12.670 INFO:tasks.workunit.client.0.smithi045.stdout: 888: throughput=4.99MB/sec pending data=0 2021-07-31T02:20:12.670 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-fD1h1HXeOXxE2xQ off=14301664 len=1914980 2021-07-31T02:20:12.674 INFO:tasks.workunit.client.0.smithi045.stdout:op 4540 completed, throughput=5MB/sec 2021-07-31T02:20:12.674 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-F3H_kKxa_FqGiEB off=13472988 len=1545494 2021-07-31T02:20:12.682 INFO:tasks.workunit.client.0.smithi045.stdout:op 4541 completed, throughput=5MB/sec 2021-07-31T02:20:12.682 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-y-OKoXcgkv4yZzr off=2442543 len=562613 2021-07-31T02:20:12.685 INFO:tasks.workunit.client.0.smithi045.stdout:op 4542 completed, throughput=5MB/sec 2021-07-31T02:20:12.686 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-_K0eOhSyyQVs70Z off=8600949 len=1710170 2021-07-31T02:20:12.687 INFO:tasks.workunit.client.0.smithi045.stdout:op 4543 completed, throughput=5MB/sec 2021-07-31T02:20:13.688 INFO:tasks.workunit.client.0.smithi045.stdout: 889: throughput=5MB/sec pending data=0 2021-07-31T02:20:13.688 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-F7HsmLNZ8BpvTy3 off=4126020 len=1924509 2021-07-31T02:20:13.697 INFO:tasks.workunit.client.0.smithi045.stdout:op 4544 completed, throughput=5MB/sec 2021-07-31T02:20:13.697 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-NA9J56ZicyHnz9U off=5697062 len=83573 2021-07-31T02:20:13.697 INFO:tasks.workunit.client.0.smithi045.stdout:op 4545 completed, throughput=5MB/sec 2021-07-31T02:20:13.698 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-GwgruxmEzCB5O-T off=5030134 len=333555 2021-07-31T02:20:13.698 INFO:tasks.workunit.client.0.smithi045.stdout:op 4546 completed, throughput=5MB/sec 2021-07-31T02:20:13.698 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-3ZkEklhlhKEmuTx off=484639 len=1440314 2021-07-31T02:20:13.705 INFO:tasks.workunit.client.0.smithi045.stdout:op 4547 completed, throughput=5MB/sec 2021-07-31T02:20:13.705 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-uJxv87NELZyapOe off=5997329 len=1554055 2021-07-31T02:20:13.707 INFO:tasks.workunit.client.0.smithi045.stdout:op 4548 completed, throughput=5MB/sec 2021-07-31T02:20:14.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:20:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:20:13.565Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:20:14.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:20:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:20:13.565Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:20:14.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:20:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:20:13.565Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:20:14.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:14 smithi154 conmon[34553]: cluster 2021-07-31T02:20:12.263836+0000 mgr.x (mgr.24542) 1411 : cluster 2021-07-31T02:20:14.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:14 smithi154 conmon[34553]: [DBG] pgmap v1861: 9 pgs: 9 active+clean; 127 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1.6 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2021-07-31T02:20:14.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:14 smithi045 conmon[39718]: cluster 2021-07-31T02:20:12.263836+0000 mgr.x ( 2021-07-31T02:20:14.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:14 smithi045 conmon[39718]: mgr.24542) 1411 : cluster [DBG] pgmap v1861: 9 pgs: 9 active+clean; 127 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1.6 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2021-07-31T02:20:14.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:14 smithi045 conmon[33655]: cluster 2021-07-31T02:20:12.263836+0000 mgr.x (mgr. 2021-07-31T02:20:14.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:14 smithi045 conmon[33655]: 24542) 1411 : cluster [DBG] pgmap v1861: 9 pgs: 9 active+clean; 127 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1.6 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2021-07-31T02:20:14.708 INFO:tasks.workunit.client.0.smithi045.stdout: 890: throughput=5MB/sec pending data=0 2021-07-31T02:20:14.708 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Q0mqAs9UPjZt0LQ off=2735500 len=1783401 2021-07-31T02:20:14.719 INFO:tasks.workunit.client.0.smithi045.stdout:op 4549 completed, throughput=5MB/sec 2021-07-31T02:20:14.719 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-lkibUvCTBqdA5jR off=1070752 len=776286 2021-07-31T02:20:14.721 INFO:tasks.workunit.client.0.smithi045.stdout:op 4550 completed, throughput=5MB/sec 2021-07-31T02:20:14.721 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-arw3PabfXeMl8M2 off=4296491 len=632177 2021-07-31T02:20:14.722 INFO:tasks.workunit.client.0.smithi045.stdout:op 4551 completed, throughput=5MB/sec 2021-07-31T02:20:14.723 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ZPDhc1srauLuX-m off=8239379 len=1109795 2021-07-31T02:20:14.725 INFO:tasks.workunit.client.0.smithi045.stdout:op 4552 completed, throughput=5MB/sec 2021-07-31T02:20:14.725 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-qLdVNo7st__W3pC off=479960 len=100353 2021-07-31T02:20:14.726 INFO:tasks.workunit.client.0.smithi045.stdout:op 4553 completed, throughput=5MB/sec 2021-07-31T02:20:14.726 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-DghOCcn2oC04Qcg off=7012880 len=1384408 2021-07-31T02:20:14.732 INFO:tasks.workunit.client.0.smithi045.stdout:op 4554 completed, throughput=5MB/sec 2021-07-31T02:20:15.733 INFO:tasks.workunit.client.0.smithi045.stdout: 891: throughput=5MB/sec pending data=0 2021-07-31T02:20:15.733 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-BhjN5t1NUA8f3w7 off=18176470 len=736712 2021-07-31T02:20:15.734 INFO:tasks.workunit.client.0.smithi045.stdout:op 4555 completed, throughput=5MB/sec 2021-07-31T02:20:15.735 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Ztwrfcc1QqZ7P2G off=12346414 len=306669 2021-07-31T02:20:15.735 INFO:tasks.workunit.client.0.smithi045.stdout:op 4556 completed, throughput=5MB/sec 2021-07-31T02:20:15.736 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-chhIx09kkNz-AZf off=10063648 len=1798914 2021-07-31T02:20:15.745 INFO:tasks.workunit.client.0.smithi045.stdout:op 4557 completed, throughput=5MB/sec 2021-07-31T02:20:15.746 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-KNaNpQqyci-faeS off=915316 len=1252814 2021-07-31T02:20:15.748 INFO:tasks.workunit.client.0.smithi045.stdout:op 4558 completed, throughput=5MB/sec 2021-07-31T02:20:15.748 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-q9uTbPIP_9AgqY- off=3619478 len=1400685 2021-07-31T02:20:15.750 INFO:tasks.workunit.client.0.smithi045.stdout:op 4559 completed, throughput=5MB/sec 2021-07-31T02:20:16.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:16 smithi154 conmon[34553]: cluster 2021-07-31T02:20:14.264411+0000 mgr.x (mgr.24542) 1412 : cluster [DBG] pgmap v1862: 9 pgs: 9 active+clean; 127 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 3.0 MiB/s rd, 3.4 MiB/s wr, 6 op/s 2021-07-31T02:20:16.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:16 smithi154 conmon[34553]: 2021-07-31T02:20:16.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:16 smithi045 conmon[39718]: cluster 2021-07-31T02:20:14.264411+0000 mgr.x (mgr.24542 2021-07-31T02:20:16.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:16 smithi045 conmon[39718]: ) 1412 : cluster [DBG] pgmap v1862: 9 pgs: 9 active+clean; 127 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 3.0 MiB/s rd, 3.4 MiB/s wr, 6 op/s 2021-07-31T02:20:16.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:16 smithi045 conmon[33655]: cluster 2021-07-31T02:20:14.264411+0000 mgr.x (mgr.24542) 1412 : cluster [DBG] 2021-07-31T02:20:16.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:16 smithi045 conmon[33655]: pgmap v1862: 9 pgs: 9 active+clean; 127 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 3.0 MiB/s rd, 3.4 MiB/s wr, 6 op/s 2021-07-31T02:20:16.751 INFO:tasks.workunit.client.0.smithi045.stdout: 892: throughput=5MB/sec pending data=0 2021-07-31T02:20:16.751 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-aJ0ijNVxYTa2qnD off=4674055 len=1510467 2021-07-31T02:20:16.755 INFO:tasks.workunit.client.0.smithi045.stdout:op 4560 completed, throughput=5MB/sec 2021-07-31T02:20:16.755 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-hzf13za_nku6PKW off=1744144 len=765433 2021-07-31T02:20:16.756 INFO:tasks.workunit.client.0.smithi045.stdout:op 4561 completed, throughput=5MB/sec 2021-07-31T02:20:16.756 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-7_IBBnlWIDAoPyQ off=7581325 len=347757 2021-07-31T02:20:16.759 INFO:tasks.workunit.client.0.smithi045.stdout:op 4562 completed, throughput=5MB/sec 2021-07-31T02:20:16.759 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-_ivv-kLi0zhtwQn off=1193568 len=39439 2021-07-31T02:20:16.760 INFO:tasks.workunit.client.0.smithi045.stdout:op 4563 completed, throughput=5MB/sec 2021-07-31T02:20:16.761 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-KKWuI0aq_5Q9On9 off=1142853 len=1166060 2021-07-31T02:20:16.763 INFO:tasks.workunit.client.0.smithi045.stdout:op 4564 completed, throughput=5MB/sec 2021-07-31T02:20:16.763 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-y-OKoXcgkv4yZzr off=78604 len=1343268 2021-07-31T02:20:16.764 INFO:tasks.workunit.client.0.smithi045.stdout:op 4565 completed, throughput=5MB/sec 2021-07-31T02:20:17.765 INFO:tasks.workunit.client.0.smithi045.stdout: 893: throughput=5MB/sec pending data=0 2021-07-31T02:20:17.765 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-XRdDs-xTaEEX0uU off=9091245 len=1727185 2021-07-31T02:20:17.774 INFO:tasks.workunit.client.0.smithi045.stdout:op 4566 completed, throughput=5MB/sec 2021-07-31T02:20:17.775 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-eJD5LTJDweURF46 off=2489491 len=613686 2021-07-31T02:20:17.778 INFO:tasks.workunit.client.0.smithi045.stdout:op 4567 completed, throughput=5MB/sec 2021-07-31T02:20:17.779 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-HMQVmEVD1LUibGs off=318466 len=1938093 2021-07-31T02:20:17.787 INFO:tasks.workunit.client.0.smithi045.stdout:op 4568 completed, throughput=5MB/sec 2021-07-31T02:20:17.787 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-NH_pkDD7ZTTcQWi off=21653947 len=266914 2021-07-31T02:20:17.790 INFO:tasks.workunit.client.0.smithi045.stdout:op 4569 completed, throughput=5MB/sec 2021-07-31T02:20:18.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:18 smithi154 conmon[34553]: cluster 2021-07-31T02:20:16.264982+0000 mgr.x (mgr.24542) 1413 : cluster [DBG] pgmap v1863: 9 pgs: 9 active+clean; 127 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2021-07-31T02:20:18.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:18 smithi045 conmon[39718]: cluster 2021-07-31T02:20:16.264982+0000 mgr.x (mgr.24542 2021-07-31T02:20:18.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:18 smithi045 conmon[39718]: ) 1413 : cluster [DBG] pgmap v1863: 9 pgs: 9 active+clean; 127 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2021-07-31T02:20:18.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:18 smithi045 conmon[33655]: cluster 2021-07-31T02:20:16.264982+0000 mgr.x (mgr.24542) 1413 : cluster [DBG] pgmap v1863: 9 pgs: 9 active+clean; 127 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2021-07-31T02:20:18.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:18 smithi045 conmon[33655]: 2021-07-31T02:20:18.790 INFO:tasks.workunit.client.0.smithi045.stdout: 894: throughput=4.99MB/sec pending data=0 2021-07-31T02:20:18.791 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-WGlNjZ21vIVF6Gp off=8211961 len=1536972 2021-07-31T02:20:18.799 INFO:tasks.workunit.client.0.smithi045.stdout:op 4570 completed, throughput=5MB/sec 2021-07-31T02:20:18.799 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-cURaK0ObcHRux_S off=5499601 len=1302617 2021-07-31T02:20:18.805 INFO:tasks.workunit.client.0.smithi045.stdout:op 4571 completed, throughput=5MB/sec 2021-07-31T02:20:18.805 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-0GqiYJjXNS55oYG off=898140 len=122998 2021-07-31T02:20:18.808 INFO:tasks.workunit.client.0.smithi045.stdout:op 4572 completed, throughput=5MB/sec 2021-07-31T02:20:18.808 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ms0HS47OxOTXUix off=1736168 len=1395331 2021-07-31T02:20:18.815 INFO:tasks.workunit.client.0.smithi045.stdout:op 4573 completed, throughput=5MB/sec 2021-07-31T02:20:18.815 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-yk7PBzZmp29C16_ off=2773663 len=718272 2021-07-31T02:20:18.819 INFO:tasks.workunit.client.0.smithi045.stdout:op 4574 completed, throughput=5MB/sec 2021-07-31T02:20:18.819 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ph41k_bufIA3End off=1653932 len=2016953 2021-07-31T02:20:18.820 INFO:tasks.workunit.client.0.smithi045.stdout:op 4575 completed, throughput=5MB/sec 2021-07-31T02:20:19.820 INFO:tasks.workunit.client.0.smithi045.stdout: 895: throughput=5MB/sec pending data=0 2021-07-31T02:20:19.821 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Eh9uGVJEhNlmOqY off=861589 len=433574 2021-07-31T02:20:19.822 INFO:tasks.workunit.client.0.smithi045.stdout:op 4576 completed, throughput=5MB/sec 2021-07-31T02:20:19.822 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-vXnDWj5fyCC5rsG off=4367322 len=1589387 2021-07-31T02:20:19.825 INFO:tasks.workunit.client.0.smithi045.stdout:op 4577 completed, throughput=5MB/sec 2021-07-31T02:20:19.825 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-MSvcOk96wIg6VrK off=19609808 len=738210 2021-07-31T02:20:19.830 INFO:tasks.workunit.client.0.smithi045.stdout:op 4578 completed, throughput=5MB/sec 2021-07-31T02:20:19.830 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-tjCoVv-Zkz_w7s0 off=1035405 len=1535910 2021-07-31T02:20:19.837 INFO:tasks.workunit.client.0.smithi045.stdout:op 4579 completed, throughput=5MB/sec 2021-07-31T02:20:20.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:20 smithi154 conmon[34553]: cluster 2021-07-31T02:20:18.265349+0000 mgr.x (mgr.24542) 1414 : cluster [DBG] pgmap v1864: 9 pgs: 9 active+clean; 127 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2021-07-31T02:20:20.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:20 smithi045 conmon[39718]: cluster 2021-07-31T02:20:18.265349+0000 2021-07-31T02:20:20.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:20 smithi045 conmon[39718]: mgr.x (mgr.24542) 1414 : cluster [DBG] pgmap v1864: 9 pgs: 9 active+clean; 127 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2021-07-31T02:20:20.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:20 smithi045 conmon[33655]: cluster 2021-07-31T02:20: 2021-07-31T02:20:20.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:20 smithi045 conmon[33655]: 18.265349+0000 mgr.x (mgr.24542) 1414 : cluster [DBG] pgmap v1864: 9 pgs: 9 active+clean; 127 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2021-07-31T02:20:20.837 INFO:tasks.workunit.client.0.smithi045.stdout: 896: throughput=5MB/sec pending data=0 2021-07-31T02:20:20.838 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-_dt--LMBNR89_RG off=1149336 len=585118 2021-07-31T02:20:20.842 INFO:tasks.workunit.client.0.smithi045.stdout:op 4580 completed, throughput=5MB/sec 2021-07-31T02:20:20.842 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-w97KF_mLVJwmYXV off=92730 len=238487 2021-07-31T02:20:20.843 INFO:tasks.workunit.client.0.smithi045.stdout:op 4581 completed, throughput=5MB/sec 2021-07-31T02:20:20.843 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-AcoOSdqzMDAR8Qt off=5568359 len=1886741 2021-07-31T02:20:20.846 INFO:tasks.workunit.client.0.smithi045.stdout:op 4582 completed, throughput=5MB/sec 2021-07-31T02:20:20.847 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-iv1iTwoeUAwuIkt off=9300592 len=1850794 2021-07-31T02:20:20.854 INFO:tasks.workunit.client.0.smithi045.stdout:op 4583 completed, throughput=5MB/sec 2021-07-31T02:20:21.855 INFO:tasks.workunit.client.0.smithi045.stdout: 897: throughput=4.99MB/sec pending data=0 2021-07-31T02:20:21.855 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-JhiD6W8JPCtsJMq off=15538657 len=581731 2021-07-31T02:20:21.857 INFO:tasks.workunit.client.0.smithi045.stdout:op 4584 completed, throughput=5MB/sec 2021-07-31T02:20:21.857 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-LIQbplckBKMY9fw off=9293514 len=1142196 2021-07-31T02:20:21.865 INFO:tasks.workunit.client.0.smithi045.stdout:op 4585 completed, throughput=5MB/sec 2021-07-31T02:20:21.865 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-HiVRhOjE1BUBs7_ off=314163 len=1762789 2021-07-31T02:20:21.875 INFO:tasks.workunit.client.0.smithi045.stdout:op 4586 completed, throughput=5MB/sec 2021-07-31T02:20:21.875 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-zA3teTflJdvz18E off=14561152 len=3871 2021-07-31T02:20:21.876 INFO:tasks.workunit.client.0.smithi045.stdout:op 4587 completed, throughput=5MB/sec 2021-07-31T02:20:21.876 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-YcwjsPiOD4fekr2 off=4104679 len=1803144 2021-07-31T02:20:21.878 INFO:tasks.workunit.client.0.smithi045.stdout:op 4588 completed, throughput=5MB/sec 2021-07-31T02:20:22.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:22 smithi154 conmon[34553]: cluster 2021-07-31T02:20:20.265820+0000 mgr.x (mgr.24542) 1415 : cluster [DBG] pgmap v1865: 9 pgs: 9 active+clean; 127 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 3.1 MiB/s rd, 3.4 MiB/s wr, 6 op/s 2021-07-31T02:20:22.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:22 smithi045 conmon[39718]: cluster 2021-07-31T02:20:20.265820+0000 mgr.x (mgr.24542) 1415 : cluster [DBG] 2021-07-31T02:20:22.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:22 smithi045 conmon[39718]: pgmap v1865: 9 pgs: 9 active+clean; 127 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 3.1 MiB/s rd, 3.4 MiB/s wr, 6 op/s 2021-07-31T02:20:22.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:22 smithi045 conmon[33655]: cluster 2021-07-31T02:20:20.265820+0000 mgr.x (mgr 2021-07-31T02:20:22.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:22 smithi045 conmon[33655]: .24542) 1415 : cluster [DBG] pgmap v1865: 9 pgs: 9 active+clean; 127 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 3.1 MiB/s rd, 3.4 MiB/s wr, 6 op/s 2021-07-31T02:20:22.878 INFO:tasks.workunit.client.0.smithi045.stdout: 898: throughput=4.99MB/sec pending data=0 2021-07-31T02:20:22.878 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ibr_zPl-0vCAni- off=3342462 len=1206672 2021-07-31T02:20:22.885 INFO:tasks.workunit.client.0.smithi045.stdout:op 4589 completed, throughput=5MB/sec 2021-07-31T02:20:22.885 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Mrau5zDjosnRBAs off=14417360 len=1162315 2021-07-31T02:20:22.891 INFO:tasks.workunit.client.0.smithi045.stdout:op 4590 completed, throughput=5MB/sec 2021-07-31T02:20:22.891 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-K0mvfMPRNwFmg4V off=18227374 len=1136494 2021-07-31T02:20:22.896 INFO:tasks.workunit.client.0.smithi045.stdout:op 4591 completed, throughput=5MB/sec 2021-07-31T02:20:22.896 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-D6pp5Cj-m9kpi39 off=17600801 len=707110 2021-07-31T02:20:22.898 INFO:tasks.workunit.client.0.smithi045.stdout:op 4592 completed, throughput=5MB/sec 2021-07-31T02:20:22.899 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-oupZ8hEyNI6iZOO off=406960 len=1844889 2021-07-31T02:20:22.905 INFO:tasks.workunit.client.0.smithi045.stdout:op 4593 completed, throughput=5MB/sec 2021-07-31T02:20:23.906 INFO:tasks.workunit.client.0.smithi045.stdout: 899: throughput=5MB/sec pending data=0 2021-07-31T02:20:23.907 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-TWF3NPfgEq_V_Cc off=20852537 len=1512962 2021-07-31T02:20:23.908 INFO:tasks.workunit.client.0.smithi045.stdout:op 4594 completed, throughput=5MB/sec 2021-07-31T02:20:23.908 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-7qcpDBPECLtshbk off=4606669 len=618078 2021-07-31T02:20:23.912 INFO:tasks.workunit.client.0.smithi045.stdout:op 4595 completed, throughput=5MB/sec 2021-07-31T02:20:23.913 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-EcyYsAuXQse79se off=9703103 len=1708427 2021-07-31T02:20:23.920 INFO:tasks.workunit.client.0.smithi045.stdout:op 4596 completed, throughput=5MB/sec 2021-07-31T02:20:23.921 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Zy1hKKHzaCk_1xN off=2541067 len=997144 2021-07-31T02:20:23.923 INFO:tasks.workunit.client.0.smithi045.stdout:op 4597 completed, throughput=5MB/sec 2021-07-31T02:20:24.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:20:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:20:23.566Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:20:24.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:20:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:20:23.566Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:20:24.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:20:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:20:23.566Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:20:24.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:24 smithi154 conmon[34553]: cluster 2021-07-31T02:20:22.266319+0000 mgr.x (mgr.24542) 1416 : cluster [DBG] pgmap v1866: 9 pgs: 9 active+clean; 127 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 2.4 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2021-07-31T02:20:24.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:24 smithi045 conmon[39718]: cluster 2021-07-31T02:20:22.266319+0000 mgr.x (mgr.24542 2021-07-31T02:20:24.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:24 smithi045 conmon[39718]: ) 1416 : cluster [DBG] pgmap v1866: 9 pgs: 9 active+clean; 127 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 2.4 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2021-07-31T02:20:24.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:24 smithi045 conmon[33655]: cluster 2021-07-31T02:20:22.266319+0000 mgr.x (mgr.24542) 1416 : 2021-07-31T02:20:24.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:24 smithi045 conmon[33655]: cluster [DBG] pgmap v1866: 9 pgs: 9 active+clean; 127 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 2.4 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2021-07-31T02:20:24.923 INFO:tasks.workunit.client.0.smithi045.stdout: 900: throughput=4.99MB/sec pending data=0 2021-07-31T02:20:24.923 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-QyOcnO9aPCKRm5z off=8479272 len=1744250 2021-07-31T02:20:24.926 INFO:tasks.workunit.client.0.smithi045.stdout:op 4598 completed, throughput=5MB/sec 2021-07-31T02:20:24.927 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-gBCZZmTEiY0Cnyi off=10529668 len=668951 2021-07-31T02:20:24.937 INFO:tasks.workunit.client.0.smithi045.stdout:op 4599 completed, throughput=5MB/sec 2021-07-31T02:20:24.937 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-tQAaa4VzaZAwf3v off=9502025 len=1820099 2021-07-31T02:20:24.940 INFO:tasks.workunit.client.0.smithi045.stdout:op 4600 completed, throughput=5MB/sec 2021-07-31T02:20:24.940 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-1HjjS9eWtH6wxix off=4892140 len=1095248 2021-07-31T02:20:24.943 INFO:tasks.workunit.client.0.smithi045.stdout:op 4601 completed, throughput=5MB/sec 2021-07-31T02:20:25.906 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:20:25.943 INFO:tasks.workunit.client.0.smithi045.stdout: 901: throughput=4.99MB/sec pending data=0 2021-07-31T02:20:25.943 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-PDcORlt_KJzyw6a off=12525051 len=1992061 2021-07-31T02:20:25.954 INFO:tasks.workunit.client.0.smithi045.stdout:op 4602 completed, throughput=5MB/sec 2021-07-31T02:20:25.955 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-xLd3tIVNUge0Ru0 off=4187179 len=693485 2021-07-31T02:20:25.955 INFO:tasks.workunit.client.0.smithi045.stdout:op 4603 completed, throughput=5MB/sec 2021-07-31T02:20:25.955 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-2RS-14CTvdAc-Dp off=5745359 len=1458419 2021-07-31T02:20:25.961 INFO:tasks.workunit.client.0.smithi045.stdout:op 4604 completed, throughput=5MB/sec 2021-07-31T02:20:25.961 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-SxASCKiEmpJ3mUE off=10485782 len=892375 2021-07-31T02:20:25.963 INFO:tasks.workunit.client.0.smithi045.stdout:op 4605 completed, throughput=5MB/sec 2021-07-31T02:20:25.964 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-yuRvTX6rOs7y2QV off=7707303 len=375724 2021-07-31T02:20:25.967 INFO:tasks.workunit.client.0.smithi045.stdout:op 4606 completed, throughput=5MB/sec 2021-07-31T02:20:26.189 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:20:26.190 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 35m ago 36m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:20:26.190 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 32m ago 35m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:20:26.190 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 32m ago 39m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:20:26.191 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 35m ago 41m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:20:26.191 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 35m ago 41m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:20:26.191 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 32m ago 39m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:20:26.191 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 35m ago 39m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:20:26.192 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 35m ago 36m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:20:26.193 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 32m ago 36m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:20:26.193 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 35m ago 39m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:20:26.193 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 35m ago 38m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:20:26.193 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 35m ago 38m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:20:26.194 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 35m ago 38m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:20:26.194 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 32m ago 37m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:20:26.194 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 32m ago 37m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:20:26.194 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 32m ago 37m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:20:26.195 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 32m ago 36m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:20:26.195 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 32m ago 36m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:20:26.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:26 smithi154 conmon[34553]: cluster 2021-07-31T02:20:24.266974+0000 mgr.x (mgr.24542) 1417 : cluster [DBG] pgmap v1867: 9 pgs: 9 active+clean; 127 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.9 MiB/s wr, 5 op/s 2021-07-31T02:20:26.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:26 smithi154 conmon[34553]: 2021-07-31T02:20:26.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:26 smithi045 conmon[33655]: cluster 2021-07-31T02:20:24. 2021-07-31T02:20:26.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:26 smithi045 conmon[33655]: 266974+0000 mgr.x (mgr.24542) 1417 : cluster [DBG] pgmap v1867: 9 pgs: 9 active+clean; 127 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.9 MiB/s wr, 5 op/s 2021-07-31T02:20:26.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:26 smithi045 conmon[39718]: cluster 2021-07-31T02:20:24.266974 2021-07-31T02:20:26.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:26 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 1417 : cluster [DBG] pgmap v1867: 9 pgs: 9 active+clean; 127 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.9 MiB/s wr, 5 op/s 2021-07-31T02:20:26.528 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:20:26.528 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:20:26.528 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:20:26.529 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:20:26.529 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:20:26.529 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:20:26.529 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:20:26.530 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:20:26.530 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:20:26.530 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:20:26.530 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:20:26.530 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:20:26.531 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:20:26.531 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:20:26.531 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:20:26.531 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:20:26.532 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:20:26.967 INFO:tasks.workunit.client.0.smithi045.stdout: 902: throughput=4.99MB/sec pending data=0 2021-07-31T02:20:26.967 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-c8d5Soildfa31lb off=13548475 len=2089394 2021-07-31T02:20:26.975 INFO:tasks.workunit.client.0.smithi045.stdout:op 4607 completed, throughput=5MB/sec 2021-07-31T02:20:26.976 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-4KuSfoEtEkgZx52 off=5147849 len=407625 2021-07-31T02:20:26.979 INFO:tasks.workunit.client.0.smithi045.stdout:op 4608 completed, throughput=5MB/sec 2021-07-31T02:20:26.979 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ohxUaLE1TsFNLG1 off=5668859 len=99506 2021-07-31T02:20:26.980 INFO:tasks.workunit.client.0.smithi045.stdout:op 4609 completed, throughput=5MB/sec 2021-07-31T02:20:26.980 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-bkka_5lZcPQrXNC off=15051823 len=1582922 2021-07-31T02:20:26.989 INFO:tasks.workunit.client.0.smithi045.stdout:op 4610 completed, throughput=5MB/sec 2021-07-31T02:20:26.989 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-vvIgGohRLs4kNBx off=37859 len=1507225 2021-07-31T02:20:26.990 INFO:tasks.workunit.client.0.smithi045.stdout:op 4611 completed, throughput=5MB/sec 2021-07-31T02:20:27.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:27 smithi154 conmon[34553]: audit 2021-07-31T02:20:25.893075+0000 mgr.x (mgr.24542) 1418 : audit [DBG] from='client.26164 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:20:27.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:27 smithi154 conmon[34553]: audit 2021-07-31T02:20:26.181256+0000 mgr.x (mgr.24542) 1419 : audit [DBG] from='client.26170 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:20:27.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:27 smithi154 conmon[34553]: audit 2021-07-31T02:20:26.528685+0000 mon.c (mon.1) 90 : audit [DBG] from='client.? 172.21.15.45:0/1075057914' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:20:27.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:27 smithi045 conmon[33655]: audit 2021-07-31T02:20:25.893075+0000 mgr.x (mgr.24542) 1418 : audit 2021-07-31T02:20:27.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:27 smithi045 conmon[33655]: [DBG] from='client.26164 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:20:27.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:27 smithi045 conmon[33655]: audit 2021-07-31T02:20:26.181256+0000 mgr.x (mgr.24542) 1419 : audit [DBG] from='client.26170 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:20:27.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:27 smithi045 conmon[33655]: audit 2021-07-31T02:20:26.528685+0000 mon.c (mon.1) 90 : audit [DBG] from='client.? 172.21.15.45:0/1075057914' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:20:27.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:27 smithi045 conmon[39718]: audit 2021-07-31T02:20:25.893075+0000 mgr.x (mgr.24542 2021-07-31T02:20:27.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:27 smithi045 conmon[39718]: ) 1418 : audit [DBG] from='client.26164 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:20:27.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:27 smithi045 conmon[39718]: audit 2021-07-31T02:20:26.181256+0000 mgr.x 2021-07-31T02:20:27.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:27 smithi045 conmon[39718]: (mgr.24542) 1419 : audit [DBG] from='client.26170 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:20:27.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:27 smithi045 conmon[39718]: audit 2021-07-31T02: 2021-07-31T02:20:27.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:27 smithi045 conmon[39718]: 20:26.528685+0000 mon.c ( 2021-07-31T02:20:27.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:27 smithi045 conmon[39718]: mon.1) 90 : audit [DBG] 2021-07-31T02:20:27.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:27 smithi045 conmon[39718]: from='client.? 172.21.15.45:0/1075057914' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:20:27.990 INFO:tasks.workunit.client.0.smithi045.stdout: 903: throughput=5MB/sec pending data=0 2021-07-31T02:20:27.991 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-bCbB2EbYbmpxhRn off=8129568 len=1366844 2021-07-31T02:20:27.998 INFO:tasks.workunit.client.0.smithi045.stdout:op 4612 completed, throughput=5MB/sec 2021-07-31T02:20:27.998 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-WxhbE8560tacXlj off=282156 len=484542 2021-07-31T02:20:28.002 INFO:tasks.workunit.client.0.smithi045.stdout:op 4613 completed, throughput=5MB/sec 2021-07-31T02:20:28.002 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-k3yv2C-wW87fwzZ off=15238166 len=1033286 2021-07-31T02:20:28.008 INFO:tasks.workunit.client.0.smithi045.stdout:op 4614 completed, throughput=5MB/sec 2021-07-31T02:20:28.008 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-JOfQ7YlcbgBrGeV off=1217745 len=822106 2021-07-31T02:20:28.013 INFO:tasks.workunit.client.0.smithi045.stdout:op 4615 completed, throughput=5MB/sec 2021-07-31T02:20:28.013 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-TOMDlCO8LUfd0x9 off=4722573 len=1313405 2021-07-31T02:20:28.019 INFO:tasks.workunit.client.0.smithi045.stdout:op 4616 completed, throughput=5MB/sec 2021-07-31T02:20:28.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:28 smithi154 conmon[34553]: cluster 2021-07-31T02:20:26.267449+0000 mgr.x (mgr.24542) 1420 : cluster [DBG] 2021-07-31T02:20:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:28 smithi154 conmon[34553]: pgmap v1868: 9 pgs: 9 active+clean; 127 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1.6 MiB/s rd, 2.2 MiB/s wr, 3 op/s 2021-07-31T02:20:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:28 smithi154 conmon[34553]: audit 2021-07-31T02:20:28.047913+0000 mon.b (mon.2) 233 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:20:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:28 smithi154 conmon[34553]: audit 2021-07-31T02:20:28.048669+0000 mon.a (mon.0) 1896 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:20:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:28 smithi154 conmon[34553]: audit 2021-07-31T02:20:28.069954+0000 mon.b (mon.2) 234 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:20:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:28 smithi154 conmon[34553]: audit 2021-07-31T02:20:28.070484+0000 mon.a (mon.0) 1897 2021-07-31T02:20:28.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:28 smithi154 conmon[34553]: : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:20:28.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:28 smithi045 conmon[33655]: cluster 2021-07-31T02:20:26.267449+0000 mgr.x 2021-07-31T02:20:28.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:28 smithi045 conmon[33655]: (mgr.24542) 1420 : cluster [DBG] pgmap v1868: 9 pgs: 9 active+clean; 127 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1.6 MiB/s rd, 2.2 MiB/s wr, 3 op/s 2021-07-31T02:20:28.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:28 smithi045 conmon[33655]: audit 2021-07-31T02:20:28.047913+0000 mon.b (mon.2) 233 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:20:28.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:28 smithi045 conmon[33655]: audit 2021-07-31T02:20:28.048669+0000 mon.a (mon.0) 1896 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:20:28.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:28 smithi045 conmon[33655]: audit 2021-07-31T02:20:28.069954+0000 mon.b (mon.2 2021-07-31T02:20:28.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:28 smithi045 conmon[33655]: ) 234 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:20:28.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:28 smithi045 conmon[33655]: audit 2021-07-31T02:20:28. 2021-07-31T02:20:28.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:28 smithi045 conmon[33655]: 070484+0000 mon.a (mon.0) 1897 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:20:28.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:28 smithi045 conmon[39718]: cluster 2021-07-31T02:20:26.267449+0000 mgr.x (mgr 2021-07-31T02:20:28.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:28 smithi045 conmon[39718]: .24542) 1420 : cluster [DBG] pgmap v1868: 9 pgs: 9 active+clean; 127 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1.6 MiB/s rd, 2.2 MiB/s wr, 3 op/s 2021-07-31T02:20:28.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:28 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T02:20:28.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:28 smithi045 conmon[39718]: 31T02:20:28.047913+0000 mon.b (mon.2) 233 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:20:28.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:28 smithi045 conmon[39718]: audit 2021- 2021-07-31T02:20:28.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:28 smithi045 conmon[39718]: 07-31T02:20:28.048669+0000 mon.a (mon.0) 1896 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:20:28.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:28 smithi045 conmon[39718]: audit 2021- 2021-07-31T02:20:28.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:28 smithi045 conmon[39718]: 07-31T02:20:28.069954+0000 mon.b (mon.2) 234 : audit 2021-07-31T02:20:28.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:28 smithi045 conmon[39718]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:20:28.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:28 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T02:20:28.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:28 smithi045 conmon[39718]: 31T02:20:28.070484+0000 mon.a (mon. 2021-07-31T02:20:28.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:28 smithi045 conmon[39718]: 0) 1897 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:20:29.019 INFO:tasks.workunit.client.0.smithi045.stdout: 904: throughput=4.99MB/sec pending data=0 2021-07-31T02:20:29.019 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ilnJxBzeCa7t0T7 off=7762062 len=331731 2021-07-31T02:20:29.020 INFO:tasks.workunit.client.0.smithi045.stdout:op 4617 completed, throughput=5MB/sec 2021-07-31T02:20:29.020 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-6Eo2G1XNKnKAfGF off=2975639 len=594843 2021-07-31T02:20:29.022 INFO:tasks.workunit.client.0.smithi045.stdout:op 4618 completed, throughput=5MB/sec 2021-07-31T02:20:29.022 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-KsI7kbeU-_5hNVT off=13330776 len=1596779 2021-07-31T02:20:29.030 INFO:tasks.workunit.client.0.smithi045.stdout:op 4619 completed, throughput=5MB/sec 2021-07-31T02:20:29.031 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-coSBANNUeXmerNB off=1867383 len=946736 2021-07-31T02:20:29.032 INFO:tasks.workunit.client.0.smithi045.stdout:op 4620 completed, throughput=5MB/sec 2021-07-31T02:20:29.033 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-bHydncoON19FU3d off=3379974 len=912704 2021-07-31T02:20:29.038 INFO:tasks.workunit.client.0.smithi045.stdout:op 4621 completed, throughput=5MB/sec 2021-07-31T02:20:29.038 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-QSlXBRy11P1N3W5 off=13026895 len=371101 2021-07-31T02:20:29.039 INFO:tasks.workunit.client.0.smithi045.stdout:op 4622 completed, throughput=5MB/sec 2021-07-31T02:20:29.039 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-9jCbLF5GrfRslj3 off=5412800 len=838077 2021-07-31T02:20:29.044 INFO:tasks.workunit.client.0.smithi045.stdout:op 4623 completed, throughput=5MB/sec 2021-07-31T02:20:30.044 INFO:tasks.workunit.client.0.smithi045.stdout: 905: throughput=4.99MB/sec pending data=0 2021-07-31T02:20:30.044 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ZWiQyXnT-00DGX- off=3686940 len=1540355 2021-07-31T02:20:30.052 INFO:tasks.workunit.client.0.smithi045.stdout:op 4624 completed, throughput=5MB/sec 2021-07-31T02:20:30.052 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-AsT7B44VfG_rMpP off=8761835 len=1293578 2021-07-31T02:20:30.055 INFO:tasks.workunit.client.0.smithi045.stdout:op 4625 completed, throughput=5MB/sec 2021-07-31T02:20:30.055 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-EB3PHRuD6d467__ off=0 len=1123422 2021-07-31T02:20:30.057 INFO:tasks.workunit.client.0.smithi045.stdout:op 4626 completed, throughput=5MB/sec 2021-07-31T02:20:30.058 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Ru0xJHLkWSM2LDs off=2112870 len=352045 2021-07-31T02:20:30.059 INFO:tasks.workunit.client.0.smithi045.stdout:op 4627 completed, throughput=5MB/sec 2021-07-31T02:20:30.059 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-jUC5pBQIyaLucje off=1966023 len=1383384 2021-07-31T02:20:30.069 INFO:tasks.workunit.client.0.smithi045.stdout:op 4628 completed, throughput=5MB/sec 2021-07-31T02:20:30.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:30 smithi154 conmon[34553]: cluster 2021-07-31T02:20:28.267848+0000 mgr.x (mgr.24542) 1421 : cluster [DBG] pgmap v1869: 9 pgs: 9 active+clean; 127 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1.6 MiB/s rd, 2.2 MiB/s wr, 3 op/s 2021-07-31T02:20:30.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:30 smithi045 conmon[39718]: cluster 2021-07-31T02:20:28.267848+0000 mgr.x ( 2021-07-31T02:20:30.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:30 smithi045 conmon[39718]: mgr.24542) 1421 : cluster [DBG] pgmap v1869: 9 pgs: 9 active+clean; 127 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1.6 MiB/s rd, 2.2 MiB/s wr, 3 op/s 2021-07-31T02:20:30.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:30 smithi045 conmon[33655]: cluster 2021-07-31T02:20:28.267848+0000 mgr.x (mgr.24542) 1421 : cluster 2021-07-31T02:20:30.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:30 smithi045 conmon[33655]: [DBG] pgmap v1869: 9 pgs: 9 active+clean; 127 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1.6 MiB/s rd, 2.2 MiB/s wr, 3 op/s 2021-07-31T02:20:31.068 INFO:tasks.workunit.client.0.smithi045.stdout: 906: throughput=5MB/sec pending data=0 2021-07-31T02:20:31.068 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-T8OaU0I3Rwc1xa0 off=12545620 len=1570338 2021-07-31T02:20:31.078 INFO:tasks.workunit.client.0.smithi045.stdout:op 4629 completed, throughput=5MB/sec 2021-07-31T02:20:31.078 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ugg2xMpau_8nN3H off=8400076 len=1746132 2021-07-31T02:20:31.080 INFO:tasks.workunit.client.0.smithi045.stdout:op 4630 completed, throughput=5MB/sec 2021-07-31T02:20:31.080 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-TZUxY2npid1TITR off=203040 len=1089720 2021-07-31T02:20:31.082 INFO:tasks.workunit.client.0.smithi045.stdout:op 4631 completed, throughput=5MB/sec 2021-07-31T02:20:31.083 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-CjBCW-Hy2qk76iC off=4980646 len=1357095 2021-07-31T02:20:31.085 INFO:tasks.workunit.client.0.smithi045.stdout:op 4632 completed, throughput=5MB/sec 2021-07-31T02:20:31.753 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: debug 2021-07-31T02:20:31.380+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:20:31.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:20:31.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:20:31.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: yield 2021-07-31T02:20:31.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:20:31.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:20:31.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:20:31.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:20:31.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:20:31.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:20:31.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:20:31.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:20:31.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:20:31.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: task.result() 2021-07-31T02:20:31.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:20:31.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: debug 2021-07-31T02:20:31.391+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:20:31.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:20:31.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:20:31.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: yield 2021-07-31T02:20:31.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:20:31.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:20:31.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:20:31.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:20:31.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:20:31.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:20:31.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:20:31.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:20:31.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:20:31.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: task.result() 2021-07-31T02:20:31.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:20:31.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: debug 2021-07-31T02:20:31.417+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:20:31.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:20:31.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:20:31.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: yield 2021-07-31T02:20:31.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:20:31.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:20:31.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:20:31.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:20:31.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:20:31.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:20:31.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:20:31.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:20:31.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:20:31.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: task.result() 2021-07-31T02:20:31.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:20:31.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: debug 2021-07-31T02:20:31.418+0000 7fcb9095b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:20:31.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:20:31.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:20:31.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: yield 2021-07-31T02:20:31.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:20:31.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:20:31.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:20:31.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:20:31.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:20:31.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:20:31.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:20:31.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:20:31.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:20:31.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: task.result() 2021-07-31T02:20:31.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:20:31.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: 2021-07-31T02:20:31.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:20:31.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: 2021-07-31T02:20:31.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:20:31.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:20:31.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:20:31.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:20:31.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:20:31.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:20:31.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:20:31.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:20:31.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:20:31.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:20:31.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:20:31.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:20:31.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:20:31.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:20:31.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:20:31.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:20:31.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:20:31.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:20:31.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: raise 2021-07-31T02:20:31.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:20:31.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:20:31.786 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:20:31.786 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:20:31.786 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:20:31.787 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: debug 2021-07-31T02:20:31.426+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:20:31.787 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:20:31.787 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:20:31.787 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: yield 2021-07-31T02:20:31.788 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:20:31.788 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:20:31.788 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:20:31.788 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:20:31.789 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:20:31.789 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:20:31.789 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:20:31.789 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:20:31.790 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:20:31.790 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: task.result() 2021-07-31T02:20:31.790 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:20:31.790 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: debug 2021-07-31T02:20:31.426+0000 7fcb9115c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:20:31.791 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:20:31.791 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:20:31.791 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: yield 2021-07-31T02:20:31.791 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:20:31.791 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:20:31.792 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:20:31.792 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:20:31.792 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:20:31.792 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:20:31.792 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:20:31.793 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:20:31.793 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:20:31.793 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: task.result() 2021-07-31T02:20:31.793 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:20:31.794 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: 2021-07-31T02:20:31.794 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:20:31.794 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: 2021-07-31T02:20:31.794 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:20:31.794 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:20:31.795 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:20:31.795 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:20:31.795 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:20:31.795 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:20:31.796 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:20:31.796 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:20:31.796 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:20:31.796 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:20:31.796 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:20:31.797 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:20:31.797 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:20:31.797 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:20:31.798 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:20:31.798 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:20:31.798 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:20:31.799 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:20:31.799 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: raise 2021-07-31T02:20:31.800 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:20:31.800 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:20:31.801 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:20:31.801 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:20:31.801 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:20:31 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:20:32.085 INFO:tasks.workunit.client.0.smithi045.stdout: 907: throughput=5MB/sec pending data=0 2021-07-31T02:20:32.086 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-lVz17DX_1d-NJlz off=366491 len=871514 2021-07-31T02:20:32.088 INFO:tasks.workunit.client.0.smithi045.stdout:op 4633 completed, throughput=5MB/sec 2021-07-31T02:20:32.088 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-56rdzlhlKWdXd1X off=2216466 len=1466605 2021-07-31T02:20:32.089 INFO:tasks.workunit.client.0.smithi045.stdout:op 4634 completed, throughput=5MB/sec 2021-07-31T02:20:32.090 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-FcBVKoAebiJOIgJ off=6476845 len=1856939 2021-07-31T02:20:32.093 INFO:tasks.workunit.client.0.smithi045.stdout:op 4635 completed, throughput=5MB/sec 2021-07-31T02:20:32.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:32 smithi154 conmon[34553]: cluster 2021-07-31T02:20:30.268322+0000 mgr.x (mgr.24542) 1422 : cluster [DBG] pgmap v1870: 9 pgs: 9 active+clean; 127 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 2.4 MiB/s rd, 3.6 MiB/s wr, 5 op/s 2021-07-31T02:20:32.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:32 smithi154 conmon[34553]: audit 2021-07-31T02:20:31.335353+0000 mon.b (mon.2) 235 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:20:32.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:32 smithi154 conmon[34553]: audit 2021-07-31T02:20:31.336645+0000 mon.b (mon.2) 236 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:20:32.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:32 smithi154 conmon[34553]: audit 2021-07-31T 2021-07-31T02:20:32.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:32 smithi154 conmon[34553]: 02:20:31.337466+0000 mon.b (mon.2) 237 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:20:32.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:32 smithi045 conmon[33655]: cluster 2021-07-31T02:20:30.268322+0000 mgr.x (mgr.24542) 1422 : cluster [DBG] pgmap v1870: 9 pgs: 9 active+clean; 127 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 2.4 MiB/s rd, 3.6 MiB/s wr, 5 op/s 2021-07-31T02:20:32.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:32 smithi045 conmon[33655]: audit 2021-07-31T02:20:31.335353+0000 mon.b (mon.2) 235 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:20:32.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:32 smithi045 conmon[33655]: audit 2021- 2021-07-31T02:20:32.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:32 smithi045 conmon[33655]: 07-31T02:20:31.336645+0000 2021-07-31T02:20:32.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:32 smithi045 conmon[33655]: mon.b (mon.2) 236 : 2021-07-31T02:20:32.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:32 smithi045 conmon[33655]: audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:20:32.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:32 smithi045 conmon[33655]: audit 2021-07-31T02:20:32.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:32 smithi045 conmon[33655]: 2021-07-31T02:20 2021-07-31T02:20:32.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:32 smithi045 conmon[33655]: :31.337466+0000 mon.b (mon.2) 237 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:20:32.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:32 smithi045 conmon[39718]: cluster 2021-07-31T02:20:30.268322+0000 2021-07-31T02:20:32.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:32 smithi045 conmon[39718]: mgr.x (mgr.24542) 1422 : cluster [DBG] pgmap v1870: 9 pgs: 9 active+clean; 127 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 2.4 MiB/s rd, 3.6 MiB/s wr, 5 op/s 2021-07-31T02:20:32.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:32 smithi045 conmon[39718]: audit 2021-07-31T02:20:32.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:32 smithi045 conmon[39718]: 2021-07-31T02:20:31 2021-07-31T02:20:32.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:32 smithi045 conmon[39718]: .335353+0000 mon.b (mon. 2021-07-31T02:20:32.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:32 smithi045 conmon[39718]: 2) 235 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:20:32.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:32 smithi045 conmon[39718]: audit 2021- 2021-07-31T02:20:32.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:32 smithi045 conmon[39718]: 07-31T02:20:31.336645+0000 2021-07-31T02:20:32.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:32 smithi045 conmon[39718]: mon.b (mon.2) 236 : audit [DBG] 2021-07-31T02:20:32.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:32 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:20:32.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:32 smithi045 conmon[39718]: audit 2021-07-31T02:20:32.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:32 smithi045 conmon[39718]: 2021-07 2021-07-31T02:20:32.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:32 smithi045 conmon[39718]: -31T02:20:31.337466+0000 2021-07-31T02:20:32.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:32 smithi045 conmon[39718]: mon.b (mon.2) 237 : audit 2021-07-31T02:20:32.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:32 smithi045 conmon[39718]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:20:33.094 INFO:tasks.workunit.client.0.smithi045.stdout: 908: throughput=4.99MB/sec pending data=0 2021-07-31T02:20:33.094 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-eJ-5sglcHTPK2qQ off=2543271 len=1810339 2021-07-31T02:20:33.097 INFO:tasks.workunit.client.0.smithi045.stdout:op 4636 completed, throughput=5MB/sec 2021-07-31T02:20:33.097 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-mdxXWtwe2voZwMT off=9174894 len=951436 2021-07-31T02:20:33.103 INFO:tasks.workunit.client.0.smithi045.stdout:op 4637 completed, throughput=5MB/sec 2021-07-31T02:20:33.103 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-7vxKQAVSRtPRkO0 off=3591 len=219760 2021-07-31T02:20:33.104 INFO:tasks.workunit.client.0.smithi045.stdout:op 4638 completed, throughput=5MB/sec 2021-07-31T02:20:33.105 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-r99Nza49oHFChBd off=136597 len=1450936 2021-07-31T02:20:33.106 INFO:tasks.workunit.client.0.smithi045.stdout:op 4639 completed, throughput=5MB/sec 2021-07-31T02:20:33.106 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-QWFDl0oJiySXr1W off=9601074 len=551642 2021-07-31T02:20:33.108 INFO:tasks.workunit.client.0.smithi045.stdout:op 4640 completed, throughput=5MB/sec 2021-07-31T02:20:33.108 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-NASXaOrxPTFwvNw off=1768369 len=548447 2021-07-31T02:20:33.108 INFO:tasks.workunit.client.0.smithi045.stdout:op 4641 completed, throughput=5MB/sec 2021-07-31T02:20:33.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:20:31.338182+0000 mgr.x (mgr.24542) 1423 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:20:33.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:20:31.381240+0000 mgr.x (mgr.24542) 1424 : cephadm [ERR] 2021-07-31T02:20:33.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:20:33.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:20:33.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: yield 2021-07-31T02:20:33.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:20:33.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:20:33.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:20:33.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:20:33.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:20:33.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:20:33.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:20:33.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:20:33.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:20:33.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:20:33.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:20:33.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:20:31.381668+0000 mgr.x (mgr.24542) 1425 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:20:33.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:20:31.392700+0000 mgr.x (mgr.24542 2021-07-31T02:20:33.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: ) 1426 : cephadm [ERR] 2021-07-31T02:20:33.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:20:33.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:20:33.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: yield 2021-07-31T02:20:33.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:20:33.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:20:33.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:20:33.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:20:33.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:20:33.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:20:33.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:20:33.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:20:33.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:20:33.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:20:33.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:20:33.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:20:31.393616+0000 mgr.x (mgr.24542) 1427 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:20:33.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:20:31.418558+0000 mgr.x (mgr.24542) 1428 : cephadm [ERR] 2021-07-31T02:20:33.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:20:33.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:20:33.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: yield 2021-07-31T02:20:33.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:20:33.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:20:33.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:20:33.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:20:33.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:20:33.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:20:33.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:20:33.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:20:33.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:20:33.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:20:33.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:20:33.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:20:31.418999+0000 mgr.x (mgr.24542) 2021-07-31T02:20:33.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: 1429 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:20:33.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:20:33.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:20:33.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: yield 2021-07-31T02:20:33.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:20:33.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:20:33.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:20:33.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:20:33.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:20:33.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:20:33.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:20:33.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:20:33.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:20:33.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:20:33.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:20:33.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: 2021-07-31T02:20:33.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:20:33.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: 2021-07-31T02:20:33.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:20:33.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:20:33.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:20:33.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:20:33.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:20:33.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:20:33.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:20:33.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:20:33.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:20:33.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:20:33.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:20:33.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:20:33.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:20:33.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:20:33.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:20:33.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:20:33.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:20:33.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:20:33.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: raise 2021-07-31T02:20:33.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:20:33.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:20:33.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:20:33.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:20:33.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:20:33.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:20:31.427004+0000 mgr.x (mgr.24542) 1430 : cephadm [ERR] 2021-07-31T02:20:33.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:20:33.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:20:33.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: yield 2021-07-31T02:20:33.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:20:33.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:20:33.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:20:33.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:20:33.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:20:33.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:20:33.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:20:33.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:20:33.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:20:33.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:20:33.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:20:33.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:20:31.427400+0000 mgr.x (mgr.24542) 1431 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:20:33.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:20:33.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:20:33.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: yield 2021-07-31T02:20:33.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:20:33.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:20:33.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:20:33.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:20:33.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:20:33.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:20:33.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:20:33.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:20:33.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:20:33.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:20:33.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:20:33.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: 2021-07-31T02:20:33.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:20:33.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: 2021-07-31T02:20:33.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:20:33.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:20:33.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:20:33.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:20:33.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:20:33.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:20:33.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:20:33.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:20:33.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:20:33.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:20:33.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:20:33.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:20:33.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:20:33.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:20:33.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:20:33.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:20:33.541 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:20:33.541 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:20:33.541 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: raise 2021-07-31T02:20:33.542 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:20:33.542 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:20:33.543 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:20:33.543 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:20:33.543 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:33 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:20:33.544 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:20:31.338182+0000 mgr.x (mgr.24542) 1423 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:20:33.544 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:20:31.381240+0000 mgr.x (mgr.24542) 1424 : cephadm [ERR] 2021-07-31T02:20:33.545 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:20:33.545 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:20:33.545 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: yield 2021-07-31T02:20:33.545 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:20:33.546 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:20:33.546 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:20:33.546 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:20:33.546 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:20:33.547 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:20:33.547 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:20:33.547 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:20:33.547 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:20:33.548 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:20:33.548 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:20:33.548 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: cephadm 2021-07-31 2021-07-31T02:20:33.548 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: T02:20:31.381668+0000 mgr.x (mgr.24542) 1425 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:20:33.549 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:20:31.392700+0000 mgr.x (mgr.24542) 1426 : cephadm [ERR] 2021-07-31T02:20:33.549 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:20:33.549 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:20:33.549 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: yield 2021-07-31T02:20:33.550 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:20:33.550 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:20:33.550 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:20:33.550 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:20:33.551 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:20:33.551 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:20:33.551 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:20:33.551 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:20:33.551 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:20:33.552 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:20:33.552 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:20:33.552 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:20:31.393616+0000 mgr.x (mgr.24542) 1427 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:20:33.553 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:20:31.418558+0000 mgr.x (mgr.24542) 1428 : cephadm [ERR] 2021-07-31T02:20:33.553 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:20:33.553 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:20:33.554 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: yield 2021-07-31T02:20:33.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:20:31.338182+0000 mgr.x (mgr. 2021-07-31T02:20:33.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: 24542) 1423 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:20:33.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: cephadm 2021-07-31T 2021-07-31T02:20:33.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: 02:20:31.381240+0000 mgr.x (mgr.24542) 1424 : cephadm [ERR] 2021-07-31T02:20:33.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:20:33.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:20:33.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: yield 2021-07-31T02:20:33.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:20:33.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:20:33.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:20:33.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:20:33.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:20:33.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:20:33.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:20:33.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:20:33.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:20:33.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:20:33.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:20:33.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:20:31.381668+0000 mgr.x (mgr.24542) 1425 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:20:33.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: cephadm 2021- 2021-07-31T02:20:33.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: 07-31T02:20:31.392700+0000 mgr.x (mgr.24542) 1426 : cephadm [ERR] 2021-07-31T02:20:33.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:20:33.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:20:33.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: yield 2021-07-31T02:20:33.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:20:33.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:20:33.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:20:33.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:20:33.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:20:33.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:20:33.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:20:33.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:20:33.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:20:33.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:20:33.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:20:33.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:20:31.393616+0000 mgr.x (mgr.24542) 1427 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:20:33.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:20:31.418558+0000 2021-07-31T02:20:33.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: mgr.x (mgr.24542) 1428 : cephadm [ERR] 2021-07-31T02:20:33.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:20:33.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:20:33.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:20:33.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:20:33.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:20:33.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:20:33.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:20:33.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:20:33.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:20:33.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:20:33.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:20:33.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:20:33.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:20:31.418999+0000 mgr.x (mgr.24542) 1429 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:20:33.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:20:33.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:20:33.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: yield 2021-07-31T02:20:33.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:20:33.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:20:33.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:20:33.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:20:33.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:20:33.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:20:33.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:20:33.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:20:33.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:20:33.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:20:33.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:20:33.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: 2021-07-31T02:20:33.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:20:33.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: 2021-07-31T02:20:33.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:20:33.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:20:33.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:20:33.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:20:33.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:20:33.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:20:33.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:20:33.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:20:33.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:20:33.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:20:33.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:20:33.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:20:33.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: yield 2021-07-31T02:20:33.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:20:33.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:20:33.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:20:33.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:20:33.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:20:33.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:20:33.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:20:33.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:20:33.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:20:33.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:20:33.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:20:33.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:20:31.418999+0000 mgr.x (mgr.24542) 1429 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:20:33.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:20:33.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:20:33.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: yield 2021-07-31T02:20:33.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:20:33.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:20:33.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:20:33.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:20:33.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:20:33.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:20:33.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:20:33.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:20:33.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:20:33.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:20:33.585 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:20:33.585 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: 2021-07-31T02:20:33.585 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:20:33.585 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: 2021-07-31T02:20:33.586 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:20:33.586 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:20:33.586 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:20:33.586 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:20:33.586 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:20:33.587 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:20:33.587 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:20:33.587 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:20:33.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:20:33.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:20:33.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:20:33.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:20:33.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:20:33.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:20:33.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:20:33.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:20:33.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:20:33.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: raise 2021-07-31T02:20:33.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:20:33.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:20:33.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:20:33.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:20:33.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:20:33.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:20:33.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: 2021-07-31T02:20:31.427004+0000 mgr.x (mgr.24542) 1430 : cephadm [ERR] 2021-07-31T02:20:33.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:20:33.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:20:33.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: yield 2021-07-31T02:20:33.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:20:33.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:20:33.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:20:33.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:20:33.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:20:33.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:20:33.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:20:33.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:20:33.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:20:33.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:20:33.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:20:33.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:20:31.427400+0000 mgr.x (mgr.24542) 1431 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:20:33.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:20:33.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:20:33.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: yield 2021-07-31T02:20:33.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:20:33.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:20:33.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:20:33.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:20:33.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:20:33.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:20:33.603 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:20:33.603 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:20:33.604 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:20:33.604 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:20:33.604 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:20:33.604 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:20:33.605 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:20:33.605 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:20:33.605 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: raise 2021-07-31T02:20:33.605 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:20:33.606 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:20:33.606 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:20:33.606 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:20:33.606 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:20:33.607 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:20: 2021-07-31T02:20:33.607 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: 31.427004+0000 mgr.x (mgr.24542) 1430 : cephadm [ERR] 2021-07-31T02:20:33.607 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:20:33.607 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:20:33.607 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: yield 2021-07-31T02:20:33.608 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:20:33.608 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:20:33.608 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:20:33.608 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:20:33.609 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:20:33.609 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:20:33.609 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:20:33.610 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:20:33.610 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:20:33.610 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:20:33.611 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:20:33.611 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:20:31.427400+0000 mgr.x (mgr.24542) 1431 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:20:33.611 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:20:33.614 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:20:33.615 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: yield 2021-07-31T02:20:33.615 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:20:33.615 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:20:33.615 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:20:33.616 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:20:33.616 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:20:33.616 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:20:33.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:20:33.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:20:33.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:20:33.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:20:33.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: 2021-07-31T02:20:33.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:20:33.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: 2021-07-31T02:20:33.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:20:33.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:20:33.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:20:33.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:20:33.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:20:33.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:20:33.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:20:33.622 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:20:33.622 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:20:33.622 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:20:33.623 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:20:33.623 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:20:33.624 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:20:33.624 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:20:33.624 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:20:33.625 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:20:33.625 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:20:33.625 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:20:33.625 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: raise 2021-07-31T02:20:33.626 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:20:33.626 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:20:33.626 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:20:33.626 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:20:33.626 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:20:33.627 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:20:33.627 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:20:33.627 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:20:33.628 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:20:33.628 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:20:33.628 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:20:33.628 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: 2021-07-31T02:20:33.628 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:20:33.629 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: 2021-07-31T02:20:33.629 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:20:33.629 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:20:33.629 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:20:33.630 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:20:33.630 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:20:33.630 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:20:33.631 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:20:33.631 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:20:33.631 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:20:33.632 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:20:33.632 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:20:33.632 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:20:33.632 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:20:33.633 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:20:33.633 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:20:33.633 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:20:33.633 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:20:33.634 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:20:33.634 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: raise 2021-07-31T02:20:33.634 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:20:33.635 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:20:33.635 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:20:33.635 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:20:33.636 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:20:34.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:20:33.566Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:20:34.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:20:33.566Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:20:34.027 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:20:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:20:33.566Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:20:34.108 INFO:tasks.workunit.client.0.smithi045.stdout: 909: throughput=4.99MB/sec pending data=0 2021-07-31T02:20:34.108 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-JM6ii6rnaajceWL off=17238393 len=868965 2021-07-31T02:20:34.114 INFO:tasks.workunit.client.0.smithi045.stdout:op 4642 completed, throughput=5MB/sec 2021-07-31T02:20:34.115 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Q0mqAs9UPjZt0LQ off=438331 len=2030303 2021-07-31T02:20:34.127 INFO:tasks.workunit.client.0.smithi045.stdout:op 4643 completed, throughput=5MB/sec 2021-07-31T02:20:34.128 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-pTlGtk89dvV4sVM off=21316862 len=1651957 2021-07-31T02:20:34.132 INFO:tasks.workunit.client.0.smithi045.stdout:op 4644 completed, throughput=5MB/sec 2021-07-31T02:20:34.133 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-iYPI9LIbPXcCQSy off=10090583 len=2007558 2021-07-31T02:20:34.135 INFO:tasks.workunit.client.0.smithi045.stdout:op 4645 completed, throughput=5MB/sec 2021-07-31T02:20:34.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:34 smithi154 conmon[34553]: cluster 2021-07-31T02:20:32.268842+0000 mgr.x (mgr.24542) 1432 : cluster [DBG] pgmap v1871: 9 pgs: 9 active+clean; 127 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 1.3 MiB/s rd, 2.4 MiB/s wr, 3 op/s 2021-07-31T02:20:34.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:34 smithi045 conmon[33655]: cluster 2021-07-31T02:20:32.268842+0000 mgr.x (mgr.24542) 1432 : 2021-07-31T02:20:34.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:34 smithi045 conmon[33655]: cluster [DBG] pgmap v1871: 9 pgs: 9 active+clean; 127 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 1.3 MiB/s rd, 2.4 MiB/s wr, 3 op/s 2021-07-31T02:20:34.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:34 smithi045 conmon[39718]: cluster 2021-07-31T02:20:32.268842+0000 mgr.x (mgr 2021-07-31T02:20:34.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:34 smithi045 conmon[39718]: .24542) 1432 : cluster [DBG] pgmap v1871: 9 pgs: 9 active+clean; 127 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 1.3 MiB/s rd, 2.4 MiB/s wr, 3 op/s 2021-07-31T02:20:35.136 INFO:tasks.workunit.client.0.smithi045.stdout: 910: throughput=5MB/sec pending data=0 2021-07-31T02:20:35.136 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-wOIO_ZE0-M5qndF off=637977 len=1553766 2021-07-31T02:20:35.139 INFO:tasks.workunit.client.0.smithi045.stdout:op 4646 completed, throughput=5MB/sec 2021-07-31T02:20:35.139 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-afSzKaVmlhnOv7w off=11645933 len=310456 2021-07-31T02:20:35.143 INFO:tasks.workunit.client.0.smithi045.stdout:op 4647 completed, throughput=5MB/sec 2021-07-31T02:20:35.143 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-qx4M9htr1DIAFFx off=1820442 len=1890899 2021-07-31T02:20:35.151 INFO:tasks.workunit.client.0.smithi045.stdout:op 4648 completed, throughput=5MB/sec 2021-07-31T02:20:35.151 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-9B5zt0U1_PSW0jy off=10584069 len=502951 2021-07-31T02:20:35.155 INFO:tasks.workunit.client.0.smithi045.stdout:op 4649 completed, throughput=5MB/sec 2021-07-31T02:20:36.155 INFO:tasks.workunit.client.0.smithi045.stdout: 911: throughput=4.99MB/sec pending data=0 2021-07-31T02:20:36.156 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-B7URPBbpXsVjieo off=4797403 len=51498 2021-07-31T02:20:36.157 INFO:tasks.workunit.client.0.smithi045.stdout:op 4650 completed, throughput=5MB/sec 2021-07-31T02:20:36.157 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-6RifZ-nunTtwiYm off=6609028 len=1967491 2021-07-31T02:20:36.167 INFO:tasks.workunit.client.0.smithi045.stdout:op 4651 completed, throughput=5MB/sec 2021-07-31T02:20:36.167 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-as_yEFiFrnZXmyG off=18685581 len=795586 2021-07-31T02:20:36.171 INFO:tasks.workunit.client.0.smithi045.stdout:op 4652 completed, throughput=5MB/sec 2021-07-31T02:20:36.171 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-oVajVmLfAIbOQe4 off=15949703 len=1832606 2021-07-31T02:20:36.172 INFO:tasks.workunit.client.0.smithi045.stdout:op 4653 completed, throughput=5MB/sec 2021-07-31T02:20:36.172 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-5kqCPgePXQTuGxJ off=8353783 len=1078286 2021-07-31T02:20:36.178 INFO:tasks.workunit.client.0.smithi045.stdout:op 4654 completed, throughput=5MB/sec 2021-07-31T02:20:36.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:36 smithi154 conmon[34553]: cluster 2021-07-31T02:20:34.269327+0000 mgr.x (mgr.24542) 1433 : cluster [DBG] pgmap v1872: 9 pgs: 9 active+clean; 127 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 2.7 MiB/s rd, 3.1 MiB/s wr, 5 op/s 2021-07-31T02:20:36.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:36 smithi154 conmon[34553]: 2021-07-31T02:20:36.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:36 smithi045 conmon[33655]: cluster 2021-07-31T02:20:34.269327 2021-07-31T02:20:36.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:36 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 1433 : cluster [DBG] pgmap v1872: 9 pgs: 9 active+clean; 127 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 2.7 MiB/s rd, 3.1 MiB/s wr, 5 op/s 2021-07-31T02:20:36.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:36 smithi045 conmon[39718]: cluster 2021-07-31T02:20:34.269327+0000 mgr.x 2021-07-31T02:20:36.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:36 smithi045 conmon[39718]: (mgr.24542) 1433 : cluster [DBG] pgmap v1872: 9 pgs: 9 active+clean; 127 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 2.7 MiB/s rd, 3.1 MiB/s wr, 5 op/s 2021-07-31T02:20:37.178 INFO:tasks.workunit.client.0.smithi045.stdout: 912: throughput=5MB/sec pending data=0 2021-07-31T02:20:37.179 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-RuTA9trBcVYcZxq off=1258655 len=1554025 2021-07-31T02:20:37.180 INFO:tasks.workunit.client.0.smithi045.stdout:op 4655 completed, throughput=5MB/sec 2021-07-31T02:20:37.180 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Yf7mUI0JWzeU49g off=3596910 len=13902 2021-07-31T02:20:37.181 INFO:tasks.workunit.client.0.smithi045.stdout:op 4656 completed, throughput=5MB/sec 2021-07-31T02:20:37.181 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-nWgBQ7ngbID2inp off=3916110 len=1467584 2021-07-31T02:20:37.182 INFO:tasks.workunit.client.0.smithi045.stdout:op 4657 completed, throughput=5MB/sec 2021-07-31T02:20:37.182 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-9PTMp3e1LCgilXs off=9465543 len=529508 2021-07-31T02:20:37.187 INFO:tasks.workunit.client.0.smithi045.stdout:op 4658 completed, throughput=5MB/sec 2021-07-31T02:20:37.188 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ySrjSpdKlfg6mGI off=3951657 len=1463417 2021-07-31T02:20:37.189 INFO:tasks.workunit.client.0.smithi045.stdout:op 4659 completed, throughput=5MB/sec 2021-07-31T02:20:38.189 INFO:tasks.workunit.client.0.smithi045.stdout: 913: throughput=5MB/sec pending data=0 2021-07-31T02:20:38.189 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-SI0X4xGeF4LeBus off=2164788 len=86398 2021-07-31T02:20:38.192 INFO:tasks.workunit.client.0.smithi045.stdout:op 4660 completed, throughput=5MB/sec 2021-07-31T02:20:38.192 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-RNetz4nenJdBive off=5234291 len=910478 2021-07-31T02:20:38.193 INFO:tasks.workunit.client.0.smithi045.stdout:op 4661 completed, throughput=5MB/sec 2021-07-31T02:20:38.194 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Z_eSeNkDClKCc7W off=6828061 len=851556 2021-07-31T02:20:38.198 INFO:tasks.workunit.client.0.smithi045.stdout:op 4662 completed, throughput=5MB/sec 2021-07-31T02:20:38.198 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-gMH5i7i0QVGNNTG off=7842311 len=1872451 2021-07-31T02:20:38.200 INFO:tasks.workunit.client.0.smithi045.stdout:op 4663 completed, throughput=5MB/sec 2021-07-31T02:20:38.201 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-O0xmyhVZPWBzL4C off=10392662 len=2001775 2021-07-31T02:20:38.212 INFO:tasks.workunit.client.0.smithi045.stdout:op 4664 completed, throughput=5MB/sec 2021-07-31T02:20:38.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:38 smithi154 conmon[34553]: cluster 2021-07-31T02:20:36.269760+0000 mgr.x (mgr.24542) 1434 : cluster [DBG] pgmap v1873: 9 pgs: 9 active+clean; 127 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.1 MiB/s wr, 3 op/s 2021-07-31T02:20:38.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:38 smithi045 conmon[39718]: cluster 2021-07-31T02:20:36.269760+0000 2021-07-31T02:20:38.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:38 smithi045 conmon[39718]: mgr.x (mgr.24542) 1434 : cluster [DBG] pgmap v1873: 9 pgs: 9 active+clean; 127 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.1 MiB/s wr, 3 op/s 2021-07-31T02:20:38.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:38 smithi045 conmon[33655]: cluster 2021-07-31T02:20:36.269760+0000 mgr.x (mgr.24542) 1434 : 2021-07-31T02:20:38.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:38 smithi045 conmon[33655]: cluster [DBG] pgmap v1873: 9 pgs: 9 active+clean; 127 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.1 MiB/s wr, 3 op/s 2021-07-31T02:20:39.212 INFO:tasks.workunit.client.0.smithi045.stdout: 914: throughput=5MB/sec pending data=0 2021-07-31T02:20:39.213 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-CO8GIgKZ8C3FJCx off=594471 len=866011 2021-07-31T02:20:39.219 INFO:tasks.workunit.client.0.smithi045.stdout:op 4665 completed, throughput=5MB/sec 2021-07-31T02:20:39.219 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-NfKRmGIVn6KupYo off=7359092 len=492732 2021-07-31T02:20:39.220 INFO:tasks.workunit.client.0.smithi045.stdout:op 4666 completed, throughput=5MB/sec 2021-07-31T02:20:39.220 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-SMnZnz6o4UhONRD off=14595732 len=1069328 2021-07-31T02:20:39.225 INFO:tasks.workunit.client.0.smithi045.stdout:op 4667 completed, throughput=5MB/sec 2021-07-31T02:20:39.225 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-K7JAHGzmbpiTlVl off=4002259 len=468657 2021-07-31T02:20:39.227 INFO:tasks.workunit.client.0.smithi045.stdout:op 4668 completed, throughput=5MB/sec 2021-07-31T02:20:39.227 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-AHaTDowvW6C0R2W off=4031909 len=765623 2021-07-31T02:20:39.228 INFO:tasks.workunit.client.0.smithi045.stdout:op 4669 completed, throughput=5MB/sec 2021-07-31T02:20:39.229 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-1MP5BiBGMb041Y5 off=668882 len=1768266 2021-07-31T02:20:39.236 INFO:tasks.workunit.client.0.smithi045.stdout:op 4670 completed, throughput=5MB/sec 2021-07-31T02:20:39.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:39 smithi154 conmon[34553]: cluster 2021-07-31T02:20:38.270193+0000 mgr.x (mgr.24542) 1435 : cluster [DBG] pgmap v1874: 9 pgs: 9 active+clean; 127 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.1 MiB/s wr, 3 op/s 2021-07-31T02:20:39.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:39 smithi045 conmon[39718]: cluster 2021-07-31T02:20:38.270193+0000 2021-07-31T02:20:39.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:39 smithi045 conmon[39718]: mgr.x (mgr.24542) 1435 : cluster [DBG] pgmap v1874: 9 pgs: 9 active+clean; 127 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.1 MiB/s wr, 3 op/s 2021-07-31T02:20:39.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:39 smithi045 conmon[33655]: cluster 2021-07-31T02:20:38.270193+0000 mgr.x (mgr.24542) 1435 : cluster [DBG] pgmap v1874: 9 pgs: 9 active+clean; 127 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.1 MiB/s wr, 3 op/s 2021-07-31T02:20:40.237 INFO:tasks.workunit.client.0.smithi045.stdout: 915: throughput=5MB/sec pending data=0 2021-07-31T02:20:40.237 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-CSbmItpLVjk8Rup off=359950 len=255456 2021-07-31T02:20:40.240 INFO:tasks.workunit.client.0.smithi045.stdout:op 4671 completed, throughput=5MB/sec 2021-07-31T02:20:40.241 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-TWF3NPfgEq_V_Cc off=7624791 len=1621190 2021-07-31T02:20:40.249 INFO:tasks.workunit.client.0.smithi045.stdout:op 4672 completed, throughput=5MB/sec 2021-07-31T02:20:40.249 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-zzQ-SQmOSfQZOJJ off=344721 len=758930 2021-07-31T02:20:40.250 INFO:tasks.workunit.client.0.smithi045.stdout:op 4673 completed, throughput=5MB/sec 2021-07-31T02:20:40.251 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-3P22yi9n9--xqwK off=12346319 len=199526 2021-07-31T02:20:40.253 INFO:tasks.workunit.client.0.smithi045.stdout:op 4674 completed, throughput=5MB/sec 2021-07-31T02:20:40.254 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-YyBWRA8ezneJPSZ off=1063402 len=1805580 2021-07-31T02:20:40.262 INFO:tasks.workunit.client.0.smithi045.stdout:op 4675 completed, throughput=5MB/sec 2021-07-31T02:20:41.262 INFO:tasks.workunit.client.0.smithi045.stdout: 916: throughput=4.99MB/sec pending data=0 2021-07-31T02:20:41.262 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-YMFwKov-jVFoZZl off=5481293 len=819487 2021-07-31T02:20:41.268 INFO:tasks.workunit.client.0.smithi045.stdout:op 4676 completed, throughput=5MB/sec 2021-07-31T02:20:41.268 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj--sXxAvJGAgZrG3b off=1531760 len=464511 2021-07-31T02:20:41.271 INFO:tasks.workunit.client.0.smithi045.stdout:op 4677 completed, throughput=5MB/sec 2021-07-31T02:20:41.272 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-3FRGNPx-UzxGgn6 off=183160 len=1032569 2021-07-31T02:20:41.273 INFO:tasks.workunit.client.0.smithi045.stdout:op 4678 completed, throughput=5MB/sec 2021-07-31T02:20:41.273 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Rd46JM_QVLJSqqA off=18481009 len=1304457 2021-07-31T02:20:41.281 INFO:tasks.workunit.client.0.smithi045.stdout:op 4679 completed, throughput=5MB/sec 2021-07-31T02:20:41.281 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Kypqh-YfiUD3D42 off=2641377 len=1184742 2021-07-31T02:20:41.283 INFO:tasks.workunit.client.0.smithi045.stdout:op 4680 completed, throughput=5MB/sec 2021-07-31T02:20:41.283 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-LP_m7uikDyArDA4 off=3332494 len=1991027 2021-07-31T02:20:41.287 INFO:tasks.workunit.client.0.smithi045.stdout:op 4681 completed, throughput=5MB/sec 2021-07-31T02:20:41.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:41 smithi154 conmon[34553]: cluster 2021-07-31T02:20:40.270647+0000 mgr.x (mgr.24542) 1436 : cluster [DBG] 2021-07-31T02:20:41.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:41 smithi154 conmon[34553]: pgmap v1875: 9 pgs: 9 active+clean; 127 GiB data, 9.7 GiB used, 705 GiB / 715 GiB avail; 3.2 MiB/s rd, 3.2 MiB/s wr, 5 op/s 2021-07-31T02:20:41.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:41 smithi045 conmon[39718]: cluster 2021-07-31T02:20:40.270647+0000 mgr.x 2021-07-31T02:20:41.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:41 smithi045 conmon[39718]: (mgr.24542) 1436 : cluster [DBG] pgmap v1875: 9 pgs: 9 active+clean; 127 GiB data, 9.7 GiB used, 705 GiB / 715 GiB avail; 3.2 MiB/s rd, 3.2 MiB/s wr, 5 op/s 2021-07-31T02:20:41.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:41 smithi045 conmon[33655]: cluster 2021-07-31T02:20:40.270647+0000 mgr.x (mgr. 2021-07-31T02:20:41.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:41 smithi045 conmon[33655]: 24542) 1436 : cluster [DBG] pgmap v1875: 9 pgs: 9 active+clean; 127 GiB data, 9.7 GiB used, 705 GiB / 715 GiB avail; 3.2 MiB/s rd, 3.2 MiB/s wr, 5 op/s 2021-07-31T02:20:42.287 INFO:tasks.workunit.client.0.smithi045.stdout: 917: throughput=5MB/sec pending data=0 2021-07-31T02:20:42.288 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Fn-fKMoC0QGHwXG off=7867058 len=1404756 2021-07-31T02:20:42.295 INFO:tasks.workunit.client.0.smithi045.stdout:op 4682 completed, throughput=5MB/sec 2021-07-31T02:20:42.295 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-eTdVUnpZqdGmg3X off=11398753 len=1130364 2021-07-31T02:20:42.297 INFO:tasks.workunit.client.0.smithi045.stdout:op 4683 completed, throughput=5MB/sec 2021-07-31T02:20:42.298 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-vbduIc3MsqX2Hs0 off=10675346 len=899008 2021-07-31T02:20:42.299 INFO:tasks.workunit.client.0.smithi045.stdout:op 4684 completed, throughput=5MB/sec 2021-07-31T02:20:42.299 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-LrGpzyu0IN_m5FO off=7511940 len=1847416 2021-07-31T02:20:42.300 INFO:tasks.workunit.client.0.smithi045.stdout:op 4685 completed, throughput=5MB/sec 2021-07-31T02:20:43.301 INFO:tasks.workunit.client.0.smithi045.stdout: 918: throughput=5MB/sec pending data=0 2021-07-31T02:20:43.301 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-lQlWGSL3zC5dnvM off=6325198 len=608162 2021-07-31T02:20:43.305 INFO:tasks.workunit.client.0.smithi045.stdout:op 4686 completed, throughput=5MB/sec 2021-07-31T02:20:43.306 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-kevv5qVmwbTIWj- off=1213186 len=691933 2021-07-31T02:20:43.309 INFO:tasks.workunit.client.0.smithi045.stdout:op 4687 completed, throughput=5MB/sec 2021-07-31T02:20:43.310 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-vl0_q5xjfalvxAs off=2218222 len=1411137 2021-07-31T02:20:43.312 INFO:tasks.workunit.client.0.smithi045.stdout:op 4688 completed, throughput=5MB/sec 2021-07-31T02:20:43.312 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-bTInmVkgQ2NgGv9 off=6410847 len=579161 2021-07-31T02:20:43.316 INFO:tasks.workunit.client.0.smithi045.stdout:op 4689 completed, throughput=5MB/sec 2021-07-31T02:20:43.317 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-FT23sZ4pfeRwBKX off=14761454 len=1529145 2021-07-31T02:20:43.323 INFO:tasks.workunit.client.0.smithi045.stdout:op 4690 completed, throughput=5MB/sec 2021-07-31T02:20:44.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:20:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:20:43.566Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:20:44.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:20:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:20:43.566Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:20:44.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:20:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:20:43.566Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:20:44.323 INFO:tasks.workunit.client.0.smithi045.stdout: 919: throughput=5MB/sec pending data=0 2021-07-31T02:20:44.324 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-B-orp9Bp6Hn0bIn off=3413885 len=110862 2021-07-31T02:20:44.326 INFO:tasks.workunit.client.0.smithi045.stdout:op 4691 completed, throughput=5MB/sec 2021-07-31T02:20:44.326 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-sINioyK7aps0ZKw off=4780891 len=251631 2021-07-31T02:20:44.328 INFO:tasks.workunit.client.0.smithi045.stdout:op 4692 completed, throughput=5MB/sec 2021-07-31T02:20:44.328 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ujV7GObTpP20-7N off=3173222 len=1286807 2021-07-31T02:20:44.334 INFO:tasks.workunit.client.0.smithi045.stdout:op 4693 completed, throughput=5MB/sec 2021-07-31T02:20:44.335 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-HV3PKiqySxNaDwu off=11988832 len=108076 2021-07-31T02:20:44.335 INFO:tasks.workunit.client.0.smithi045.stdout:op 4694 completed, throughput=5MB/sec 2021-07-31T02:20:44.335 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-he3ABI7t-Rw5c7J off=18191928 len=1578583 2021-07-31T02:20:44.337 INFO:tasks.workunit.client.0.smithi045.stdout:op 4695 completed, throughput=5MB/sec 2021-07-31T02:20:44.337 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-dugjptcGNeR07Cv off=16454767 len=1141363 2021-07-31T02:20:44.342 INFO:tasks.workunit.client.0.smithi045.stdout:op 4696 completed, throughput=5MB/sec 2021-07-31T02:20:44.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:44 smithi154 conmon[34553]: cluster 2021-07-31T02:20:42.271140+0000 mgr.x (mgr.24542) 1437 : cluster [DBG] pgmap v1876: 9 pgs: 9 active+clean; 127 GiB data, 9.7 GiB used, 705 GiB / 715 GiB avail; 2.4 MiB/s rd, 1.9 MiB/s wr, 3 op/s 2021-07-31T02:20:44.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:44 smithi045 conmon[33655]: cluster 2021-07-31T02:20:42.271140+0000 mgr.x (mgr.24542) 1437 : cluster [DBG] pgmap v1876: 9 pgs: 9 active+clean; 127 GiB data, 9.7 GiB used, 705 GiB / 715 GiB avail; 2.4 MiB/s rd, 1.9 MiB/s wr, 3 op/s 2021-07-31T02:20:44.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:44 smithi045 conmon[39718]: cluster 2021-07-31T02:20:42.271140+0000 mgr.x (mgr.24542) 2021-07-31T02:20:44.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:44 smithi045 conmon[39718]: 1437 : cluster [DBG] pgmap v1876: 9 pgs: 9 active+clean; 127 GiB data, 9.7 GiB used, 705 GiB / 715 GiB avail; 2.4 MiB/s rd, 1.9 MiB/s wr, 3 op/s 2021-07-31T02:20:45.343 INFO:tasks.workunit.client.0.smithi045.stdout: 920: throughput=4.99MB/sec pending data=0 2021-07-31T02:20:45.343 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-IMLOggbWTWdNKxo off=2172710 len=1559900 2021-07-31T02:20:45.351 INFO:tasks.workunit.client.0.smithi045.stdout:op 4697 completed, throughput=5MB/sec 2021-07-31T02:20:45.351 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-jrrX3WLV8QPYkDr off=1566575 len=169802 2021-07-31T02:20:45.352 INFO:tasks.workunit.client.0.smithi045.stdout:op 4698 completed, throughput=5MB/sec 2021-07-31T02:20:45.352 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-wWL-LqRlke7s5N_ off=3295290 len=1252972 2021-07-31T02:20:45.353 INFO:tasks.workunit.client.0.smithi045.stdout:op 4699 completed, throughput=5MB/sec 2021-07-31T02:20:45.354 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-vBinCRZikhe_VUD off=4723755 len=215693 2021-07-31T02:20:45.355 INFO:tasks.workunit.client.0.smithi045.stdout:op 4700 completed, throughput=5MB/sec 2021-07-31T02:20:45.355 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-9LfHnYe52PyK_yS off=0 len=1238740 2021-07-31T02:20:45.358 INFO:tasks.workunit.client.0.smithi045.stdout:op 4701 completed, throughput=5MB/sec 2021-07-31T02:20:45.358 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-FPEsCjBN9PU6epc off=2515439 len=1519970 2021-07-31T02:20:45.366 INFO:tasks.workunit.client.0.smithi045.stdout:op 4702 completed, throughput=5MB/sec 2021-07-31T02:20:46.367 INFO:tasks.workunit.client.0.smithi045.stdout: 921: throughput=5MB/sec pending data=0 2021-07-31T02:20:46.367 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-IAyKPq3ZUh2muL1 off=1274787 len=1759694 2021-07-31T02:20:46.375 INFO:tasks.workunit.client.0.smithi045.stdout:op 4703 completed, throughput=5MB/sec 2021-07-31T02:20:46.375 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-JxEvVoqwcPyJno6 off=2839899 len=1330584 2021-07-31T02:20:46.377 INFO:tasks.workunit.client.0.smithi045.stdout:op 4704 completed, throughput=5MB/sec 2021-07-31T02:20:46.378 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-yVwsPJxi99xmuGU off=16707479 len=1000868 2021-07-31T02:20:46.380 INFO:tasks.workunit.client.0.smithi045.stdout:op 4705 completed, throughput=5MB/sec 2021-07-31T02:20:46.380 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-CgoMlzc_FbEjVP7 off=6830597 len=1996525 2021-07-31T02:20:46.381 INFO:tasks.workunit.client.0.smithi045.stdout:op 4706 completed, throughput=5MB/sec 2021-07-31T02:20:46.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:46 smithi154 conmon[34553]: cluster 2021-07-31T02:20:44.271566+0000 mgr.x (mgr.24542) 1438 : cluster [DBG] pgmap v1877: 9 pgs: 9 active+clean; 127 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 3.4 MiB/s rd, 3.0 MiB/s wr, 6 op/s 2021-07-31T02:20:46.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:46 smithi045 conmon[33655]: cluster 2021-07-31T02:20:44.271566+0000 mgr.x (mgr.24542) 1438 : cluster [DBG] pgmap v1877: 9 pgs: 9 active+clean; 127 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 3.4 MiB/s rd, 3.0 MiB/s wr, 6 op/s 2021-07-31T02:20:46.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:46 smithi045 conmon[39718]: cluster 2021-07-31T02:20:44.271566+0000 mgr.x (mgr.24542) 1438 : cluster 2021-07-31T02:20:46.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:46 smithi045 conmon[39718]: [DBG] pgmap v1877: 9 pgs: 9 active+clean; 127 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 3.4 MiB/s rd, 3.0 MiB/s wr, 6 op/s 2021-07-31T02:20:47.381 INFO:tasks.workunit.client.0.smithi045.stdout: 922: throughput=5MB/sec pending data=0 2021-07-31T02:20:47.382 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-3CBGvyFltRVcA1n off=16961953 len=1939802 2021-07-31T02:20:47.390 INFO:tasks.workunit.client.0.smithi045.stdout:op 4707 completed, throughput=5MB/sec 2021-07-31T02:20:47.391 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-If7vMeF2hisCmpq off=1121060 len=1565378 2021-07-31T02:20:47.395 INFO:tasks.workunit.client.0.smithi045.stdout:op 4708 completed, throughput=5MB/sec 2021-07-31T02:20:47.395 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-iBcIdqpmQkxmime off=1414242 len=86285 2021-07-31T02:20:47.396 INFO:tasks.workunit.client.0.smithi045.stdout:op 4709 completed, throughput=5MB/sec 2021-07-31T02:20:47.396 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-SWUSqI05q5B08iu off=386709 len=1455944 2021-07-31T02:20:47.403 INFO:tasks.workunit.client.0.smithi045.stdout:op 4710 completed, throughput=5MB/sec 2021-07-31T02:20:48.403 INFO:tasks.workunit.client.0.smithi045.stdout: 923: throughput=5MB/sec pending data=0 2021-07-31T02:20:48.403 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-zamMXSaUhwpPxkX off=10298064 len=2003924 2021-07-31T02:20:48.406 INFO:tasks.workunit.client.0.smithi045.stdout:op 4711 completed, throughput=5MB/sec 2021-07-31T02:20:48.407 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-IdFBjujaqT0fxSO off=953063 len=2029804 2021-07-31T02:20:48.416 INFO:tasks.workunit.client.0.smithi045.stdout:op 4712 completed, throughput=5MB/sec 2021-07-31T02:20:48.416 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-t9uiNXMfnsIdQZr off=5262273 len=2079124 2021-07-31T02:20:48.430 INFO:tasks.workunit.client.0.smithi045.stdout:op 4713 completed, throughput=5MB/sec 2021-07-31T02:20:48.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:48 smithi154 conmon[34553]: cluster 2021-07-31T02:20:46.271996+0000 mgr.x (mgr.24542) 1439 : cluster [DBG] pgmap v1878: 9 pgs: 9 active+clean; 127 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2021-07-31T02:20:48.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:48 smithi045 conmon[33655]: cluster 2021-07-31T02:20:46.271996+0000 mgr.x (mgr.24542) 1439 : cluster [DBG] pgmap v1878: 9 pgs: 9 active+clean; 127 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2021-07-31T02:20:48.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:48 smithi045 conmon[33655]: 2021-07-31T02:20:48.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:48 smithi045 conmon[39718]: cluster 2021-07-31T02:20:46.271996+0000 mgr.x 2021-07-31T02:20:48.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:48 smithi045 conmon[39718]: (mgr.24542) 1439 : cluster [DBG] pgmap v1878: 9 pgs: 9 active+clean; 127 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2021-07-31T02:20:49.425 INFO:tasks.workunit.client.0.smithi045.stdout: 924: throughput=5MB/sec pending data=0 2021-07-31T02:20:49.426 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-6uDqyokO5oOv9Tt off=6442985 len=1239553 2021-07-31T02:20:49.432 INFO:tasks.workunit.client.0.smithi045.stdout:op 4714 completed, throughput=5MB/sec 2021-07-31T02:20:49.432 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-v2Hz-0y9r7apwPq off=1124268 len=517149 2021-07-31T02:20:49.436 INFO:tasks.workunit.client.0.smithi045.stdout:op 4715 completed, throughput=5MB/sec 2021-07-31T02:20:49.437 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-559YixShNkOFG1c off=18139245 len=1604416 2021-07-31T02:20:49.439 INFO:tasks.workunit.client.0.smithi045.stdout:op 4716 completed, throughput=5MB/sec 2021-07-31T02:20:49.439 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-RCg4xLuhIo8fa_Q off=8208267 len=952254 2021-07-31T02:20:49.440 INFO:tasks.workunit.client.0.smithi045.stdout:op 4717 completed, throughput=5MB/sec 2021-07-31T02:20:50.442 INFO:tasks.workunit.client.0.smithi045.stdout: 925: throughput=5MB/sec pending data=0 2021-07-31T02:20:50.443 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-uQvg0LRgMkXOoTi off=2213685 len=455356 2021-07-31T02:20:50.445 INFO:tasks.workunit.client.0.smithi045.stdout:op 4718 completed, throughput=5MB/sec 2021-07-31T02:20:50.445 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ucdb0P6NsLj2ppf off=4286544 len=1236663 2021-07-31T02:20:50.449 INFO:tasks.workunit.client.0.smithi045.stdout:op 4719 completed, throughput=5MB/sec 2021-07-31T02:20:50.449 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ZsBu9CJyt1BrZvl off=20734905 len=861229 2021-07-31T02:20:50.456 INFO:tasks.workunit.client.0.smithi045.stdout:op 4720 completed, throughput=5MB/sec 2021-07-31T02:20:50.456 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-eNjo41nON66byyZ off=4657837 len=597880 2021-07-31T02:20:50.457 INFO:tasks.workunit.client.0.smithi045.stdout:op 4721 completed, throughput=5MB/sec 2021-07-31T02:20:50.457 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-7l5gFccfBHlwQ6F off=241843 len=1945854 2021-07-31T02:20:50.465 INFO:tasks.workunit.client.0.smithi045.stdout:op 4722 completed, throughput=5MB/sec 2021-07-31T02:20:50.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:50 smithi154 conmon[34553]: cluster 2021-07-31T02:20:48.272388+0000 mgr.x (mgr.24542) 1440 : cluster [DBG] pgmap v1879: 9 pgs: 9 active+clean; 127 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2021-07-31T02:20:50.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:50 smithi154 conmon[34553]: 2021-07-31T02:20:50.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:50 smithi045 conmon[33655]: cluster 2021-07-31T02:20:48.272388+0000 mgr.x (mgr.24542) 1440 : 2021-07-31T02:20:50.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:50 smithi045 conmon[33655]: cluster [DBG] pgmap v1879: 9 pgs: 9 active+clean; 127 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2021-07-31T02:20:50.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:50 smithi045 conmon[39718]: cluster 2021-07-31T02:20:48.272388+0000 2021-07-31T02:20:50.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:50 smithi045 conmon[39718]: mgr.x (mgr.24542) 1440 : cluster [DBG] pgmap v1879: 9 pgs: 9 active+clean; 127 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2021-07-31T02:20:51.465 INFO:tasks.workunit.client.0.smithi045.stdout: 926: throughput=5MB/sec pending data=0 2021-07-31T02:20:51.465 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-bbkKPcPsc2Z3eeS off=11898063 len=481237 2021-07-31T02:20:51.467 INFO:tasks.workunit.client.0.smithi045.stdout:op 4723 completed, throughput=5MB/sec 2021-07-31T02:20:51.467 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-etXERW2iUXnQ1yT off=1174669 len=1644617 2021-07-31T02:20:51.470 INFO:tasks.workunit.client.0.smithi045.stdout:op 4724 completed, throughput=5MB/sec 2021-07-31T02:20:51.470 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-dQpCWXPyVYUhhau off=19270677 len=1600036 2021-07-31T02:20:51.477 INFO:tasks.workunit.client.0.smithi045.stdout:op 4725 completed, throughput=5MB/sec 2021-07-31T02:20:51.477 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-tLVWZMeUD8Zrriz off=7624939 len=1785698 2021-07-31T02:20:51.485 INFO:tasks.workunit.client.0.smithi045.stdout:op 4726 completed, throughput=5MB/sec 2021-07-31T02:20:52.486 INFO:tasks.workunit.client.0.smithi045.stdout: 927: throughput=5MB/sec pending data=0 2021-07-31T02:20:52.487 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-1SUZDUIALPfp4fA off=3010550 len=1528850 2021-07-31T02:20:52.493 INFO:tasks.workunit.client.0.smithi045.stdout:op 4727 completed, throughput=5MB/sec 2021-07-31T02:20:52.493 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-25XGBhJ601VqeMd off=3592383 len=694868 2021-07-31T02:20:52.498 INFO:tasks.workunit.client.0.smithi045.stdout:op 4728 completed, throughput=5MB/sec 2021-07-31T02:20:52.498 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-HvaUmmPxvC9wqk3 off=5527172 len=1682282 2021-07-31T02:20:52.501 INFO:tasks.workunit.client.0.smithi045.stdout:op 4729 completed, throughput=5MB/sec 2021-07-31T02:20:52.501 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-C2iOqjDLTfIqLfh off=7836177 len=365955 2021-07-31T02:20:52.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:52 smithi154 conmon[34553]: cluster 2021-07-31T02:20:50.272860+0000 mgr.x (mgr.24542) 1441 : cluster [DBG] pgmap v1880: 9 pgs: 9 active+clean; 127 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 3.0 MiB/s rd, 3.5 MiB/s wr, 5 op/s 2021-07-31T02:20:52.503 INFO:tasks.workunit.client.0.smithi045.stdout:op 4730 completed, throughput=5MB/sec 2021-07-31T02:20:52.503 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-pfs57yCWOKZSgKp off=9473380 len=192403 2021-07-31T02:20:52.503 INFO:tasks.workunit.client.0.smithi045.stdout:op 4731 completed, throughput=5MB/sec 2021-07-31T02:20:52.503 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-sillLU3Zsog3lB0 off=23929123 len=319452 2021-07-31T02:20:52.505 INFO:tasks.workunit.client.0.smithi045.stdout:op 4732 completed, throughput=5MB/sec 2021-07-31T02:20:52.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:52 smithi045 conmon[33655]: cluster 2021-07-31T02:20:50.272860+0000 mgr.x (mgr.24542 2021-07-31T02:20:52.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:52 smithi045 conmon[33655]: ) 1441 : cluster [DBG] pgmap v1880: 9 pgs: 9 active+clean; 127 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 3.0 MiB/s rd, 3.5 MiB/s wr, 5 op/s 2021-07-31T02:20:52.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:52 smithi045 conmon[39718]: cluster 2021-07-31T02:20:50.272860+0000 mgr.x (mgr.24542 2021-07-31T02:20:52.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:52 smithi045 conmon[39718]: ) 1441 : cluster [DBG] pgmap v1880: 9 pgs: 9 active+clean; 127 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 3.0 MiB/s rd, 3.5 MiB/s wr, 5 op/s 2021-07-31T02:20:53.506 INFO:tasks.workunit.client.0.smithi045.stdout: 928: throughput=4.99MB/sec pending data=0 2021-07-31T02:20:53.506 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-9hpDWJSxgEWEkB2 off=10125883 len=1244170 2021-07-31T02:20:53.514 INFO:tasks.workunit.client.0.smithi045.stdout:op 4733 completed, throughput=5MB/sec 2021-07-31T02:20:53.516 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-5NLCdZGs0dYutbf off=1132390 len=935917 2021-07-31T02:20:53.516 INFO:tasks.workunit.client.0.smithi045.stdout:op 4734 completed, throughput=5MB/sec 2021-07-31T02:20:53.517 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-s_z5SJxdtadLsRH off=5514446 len=383764 2021-07-31T02:20:53.519 INFO:tasks.workunit.client.0.smithi045.stdout:op 4735 completed, throughput=5MB/sec 2021-07-31T02:20:53.520 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-9PwtoQxgJq3bmG8 off=39084 len=1299095 2021-07-31T02:20:53.521 INFO:tasks.workunit.client.0.smithi045.stdout:op 4736 completed, throughput=5MB/sec 2021-07-31T02:20:53.522 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-w97KF_mLVJwmYXV off=484443 len=671759 2021-07-31T02:20:53.522 INFO:tasks.workunit.client.0.smithi045.stdout:op 4737 completed, throughput=5MB/sec 2021-07-31T02:20:53.523 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-HfMLjd0rF_phNEa off=10368946 len=1864791 2021-07-31T02:20:53.526 INFO:tasks.workunit.client.0.smithi045.stdout:op 4738 completed, throughput=5MB/sec 2021-07-31T02:20:54.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:20:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:20:53.566Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:20:54.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:20:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:20:53.566Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:20:54.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:20:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:20:53.566Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:20:54.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:54 smithi154 conmon[34553]: cluster 2021-07-31T02:20:52.273340+0000 mgr.x (mgr.24542) 1442 : cluster [DBG] pgmap v1881: 9 pgs: 9 active+clean; 127 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:20:54.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:54 smithi045 conmon[39718]: cluster 2021-07-31T02:20:52.273340+0000 mgr.x (mgr.24542) 1442 2021-07-31T02:20:54.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:54 smithi045 conmon[39718]: : cluster [DBG] pgmap v1881: 9 pgs: 9 active+clean; 127 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:20:54.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:54 smithi045 conmon[33655]: cluster 2021-07-31T02:20:52.273340+0000 mgr.x (mgr.24542) 1442 : 2021-07-31T02:20:54.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:54 smithi045 conmon[33655]: cluster [DBG] pgmap v1881: 9 pgs: 9 active+clean; 127 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:20:54.527 INFO:tasks.workunit.client.0.smithi045.stdout: 929: throughput=5MB/sec pending data=0 2021-07-31T02:20:54.527 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-hHKtqDTVji05HP4 off=3170211 len=1990299 2021-07-31T02:20:54.537 INFO:tasks.workunit.client.0.smithi045.stdout:op 4739 completed, throughput=5MB/sec 2021-07-31T02:20:54.537 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-BhjN5t1NUA8f3w7 off=10244116 len=210968 2021-07-31T02:20:54.540 INFO:tasks.workunit.client.0.smithi045.stdout:op 4740 completed, throughput=5MB/sec 2021-07-31T02:20:54.541 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-jxhvBnwWNHeyDQm off=391798 len=1145218 2021-07-31T02:20:54.543 INFO:tasks.workunit.client.0.smithi045.stdout:op 4741 completed, throughput=5MB/sec 2021-07-31T02:20:54.543 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-zcmznw_0Bx1fB9d off=5545077 len=1332357 2021-07-31T02:20:54.545 INFO:tasks.workunit.client.0.smithi045.stdout:op 4742 completed, throughput=5MB/sec 2021-07-31T02:20:55.545 INFO:tasks.workunit.client.0.smithi045.stdout: 930: throughput=5MB/sec pending data=0 2021-07-31T02:20:55.546 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-l94whW_ktU1KeLt off=5282648 len=242056 2021-07-31T02:20:55.549 INFO:tasks.workunit.client.0.smithi045.stdout:op 4743 completed, throughput=5MB/sec 2021-07-31T02:20:55.549 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-aFk0GhiiG5PpBC- off=3351080 len=861569 2021-07-31T02:20:55.554 INFO:tasks.workunit.client.0.smithi045.stdout:op 4744 completed, throughput=5MB/sec 2021-07-31T02:20:55.554 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-AcUqu09ilfo0-In off=5044310 len=1697671 2021-07-31T02:20:55.562 INFO:tasks.workunit.client.0.smithi045.stdout:op 4745 completed, throughput=5MB/sec 2021-07-31T02:20:55.562 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-WN9mW7olw3-VTWX off=3131690 len=1986318 2021-07-31T02:20:55.566 INFO:tasks.workunit.client.0.smithi045.stdout:op 4746 completed, throughput=5MB/sec 2021-07-31T02:20:56.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:56 smithi154 conmon[34553]: cluster 2021-07-31T02:20:54.273892+0000 mgr.x (mgr.24542) 1443 : cluster [DBG] pgmap v1882: 9 pgs: 9 active+clean; 127 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 3.0 MiB/s rd, 3.4 MiB/s wr, 6 op/s 2021-07-31T02:20:56.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:56 smithi045 conmon[33655]: cluster 2021-07-31T02:20:54.273892+0000 mgr.x (mgr.24542) 1443 : cluster [DBG] pgmap v1882: 9 pgs: 9 active+clean; 127 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 3.0 MiB/s rd, 3.4 MiB/s wr, 6 op/s 2021-07-31T02:20:56.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:56 smithi045 conmon[39718]: cluster 2021-07-31T02:20: 2021-07-31T02:20:56.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:56 smithi045 conmon[39718]: 54.273892+0000 mgr.x (mgr.24542) 1443 : cluster [DBG] pgmap v1882: 9 pgs: 9 active+clean; 127 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 3.0 MiB/s rd, 3.4 MiB/s wr, 6 op/s 2021-07-31T02:20:56.566 INFO:tasks.workunit.client.0.smithi045.stdout: 931: throughput=4.99MB/sec pending data=0 2021-07-31T02:20:56.567 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-RWIo90_U7AISvD_ off=5986350 len=1833802 2021-07-31T02:20:56.569 INFO:tasks.workunit.client.0.smithi045.stdout:op 4747 completed, throughput=5MB/sec 2021-07-31T02:20:56.569 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-DPzd_B46g0TDe_I off=3553549 len=261122 2021-07-31T02:20:56.573 INFO:tasks.workunit.client.0.smithi045.stdout:op 4748 completed, throughput=5MB/sec 2021-07-31T02:20:56.573 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-k577Ih0VQ9CYfAy off=8956327 len=1840071 2021-07-31T02:20:56.575 INFO:tasks.workunit.client.0.smithi045.stdout:op 4749 completed, throughput=5MB/sec 2021-07-31T02:20:56.575 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-yhoiudgJiJO2VcY off=12733002 len=1029569 2021-07-31T02:20:56.577 INFO:tasks.workunit.client.0.smithi045.stdout:op 4750 completed, throughput=5MB/sec 2021-07-31T02:20:56.578 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-L6gmbCb0vJRhQnS off=5553614 len=798175 2021-07-31T02:20:56.579 INFO:tasks.workunit.client.0.smithi045.stdout:op 4751 completed, throughput=5MB/sec 2021-07-31T02:20:56.842 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:20:57.119 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:20:57.119 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 35m ago 36m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:20:57.120 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 33m ago 36m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:20:57.120 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 33m ago 39m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:20:57.121 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 35m ago 41m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:20:57.121 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 35m ago 42m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:20:57.122 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 33m ago 40m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:20:57.122 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 35m ago 40m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:20:57.123 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 35m ago 36m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:20:57.123 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 33m ago 36m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:20:57.123 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 35m ago 39m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:20:57.124 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 35m ago 39m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:20:57.124 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 35m ago 38m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:20:57.125 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 35m ago 38m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:20:57.125 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 33m ago 38m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:20:57.125 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 33m ago 37m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:20:57.126 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 33m ago 37m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:20:57.126 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 33m ago 37m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:20:57.127 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 33m ago 36m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:20:57.456 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:20:57.457 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:20:57.457 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:20:57.457 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:20:57.458 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:20:57.458 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:20:57.458 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:20:57.458 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:20:57.458 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:20:57.459 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:20:57.459 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:20:57.459 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:20:57.459 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:20:57.460 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:20:57.460 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:20:57.461 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:20:57.461 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:20:57.580 INFO:tasks.workunit.client.0.smithi045.stdout: 932: throughput=5MB/sec pending data=0 2021-07-31T02:20:57.580 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-52wyaGKlmrdvqm8 off=11454746 len=1776231 2021-07-31T02:20:57.587 INFO:tasks.workunit.client.0.smithi045.stdout:op 4752 completed, throughput=5MB/sec 2021-07-31T02:20:57.588 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-o36Kr78_Ryr_Pmj off=1780068 len=1790860 2021-07-31T02:20:57.589 INFO:tasks.workunit.client.0.smithi045.stdout:op 4753 completed, throughput=5MB/sec 2021-07-31T02:20:57.589 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ptRY6ilBzMDbz6o off=5586461 len=1913429 2021-07-31T02:20:57.590 INFO:tasks.workunit.client.0.smithi045.stdout:op 4754 completed, throughput=5MB/sec 2021-07-31T02:20:58.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:58 smithi154 conmon[34553]: cluster 2021-07-31T02:20:56.274367+0000 mgr.x (mgr.24542) 1444 : cluster [DBG] pgmap v1883: 9 pgs: 9 active+clean; 127 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.2 MiB/s wr, 3 op/s 2021-07-31T02:20:58.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:58 smithi154 conmon[34553]: audit 2021-07-31T02:20:56.830718+0000 2021-07-31T02:20:58.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:58 smithi154 conmon[34553]: mgr.x (mgr.24542) 1445 : audit [DBG] from='client.16275 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:20:58.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:58 smithi154 conmon[34553]: audit 2021-07-31T02:20:57.111081+0000 mgr.x (mgr.24542) 1446 : audit [DBG] from='client.26188 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:20:58.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:20:58 smithi154 conmon[34553]: audit 2021-07-31T02:20:57.456982+0000 mon.a (mon.0) 1898 : audit [DBG] from='client.? 172.21.15.45:0/3300590419' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:20:58.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:58 smithi045 conmon[33655]: cluster 2021-07-31T02:20:56.274367+0000 mgr.x (mgr.24542) 1444 : cluster [DBG] pgmap v1883: 9 pgs: 9 active+clean; 127 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.2 MiB/s wr, 3 op/s 2021-07-31T02:20:58.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:58 smithi045 conmon[33655]: audit 2021-07-31T02:20:56.830718+0000 mgr.x (mgr.24542) 1445 : audit [DBG] from='client.16275 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:20:58.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:58 smithi045 conmon[33655]: audit 2021-07-31T02: 2021-07-31T02:20:58.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:58 smithi045 conmon[33655]: 20:57.111081+0000 mgr.x (mgr.24542) 1446 : audit [DBG] from='client.26188 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:20:58.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:58 smithi045 conmon[33655]: audit 2021-07 2021-07-31T02:20:58.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:58 smithi045 conmon[33655]: -31T02:20:57.456982+0000 2021-07-31T02:20:58.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:58 smithi045 conmon[33655]: mon.a (mon.0) 1898 : 2021-07-31T02:20:58.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:20:58 smithi045 conmon[33655]: audit [DBG] from='client.? 172.21.15.45:0/3300590419' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:20:58.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:58 smithi045 conmon[39718]: cluster 2021-07-31T02:20:56.274367+0000 mgr.x (mgr 2021-07-31T02:20:58.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:58 smithi045 conmon[39718]: .24542) 1444 : cluster [DBG] pgmap v1883: 9 pgs: 9 active+clean; 127 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.2 MiB/s wr, 3 op/s 2021-07-31T02:20:58.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:58 smithi045 conmon[39718]: audit 2021-07-31T02:20:56.830718+0000 mgr.x 2021-07-31T02:20:58.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:58 smithi045 conmon[39718]: (mgr.24542) 1445 : audit [DBG] from='client.16275 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:20:58.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:58 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T02:20:58.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:58 smithi045 conmon[39718]: 02:20:57.111081+0000 mgr.x (mgr 2021-07-31T02:20:58.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:58 smithi045 conmon[39718]: .24542) 1446 : audit [DBG] from='client.26188 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:20:58.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:58 smithi045 conmon[39718]: audit 2021 2021-07-31T02:20:58.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:58 smithi045 conmon[39718]: -07-31T02:20:57. 2021-07-31T02:20:58.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:58 smithi045 conmon[39718]: 456982+0000 mon.a (mon.0) 1898 : 2021-07-31T02:20:58.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:20:58 smithi045 conmon[39718]: audit [DBG] from='client.? 172.21.15.45:0/3300590419' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:20:58.591 INFO:tasks.workunit.client.0.smithi045.stdout: 933: throughput=5MB/sec pending data=0 2021-07-31T02:20:58.591 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Y8juEXwZE7wE95C off=18001662 len=1519060 2021-07-31T02:20:58.598 INFO:tasks.workunit.client.0.smithi045.stdout:op 4755 completed, throughput=5MB/sec 2021-07-31T02:20:58.599 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-YZV5976H38O5wSc off=6390758 len=1796656 2021-07-31T02:20:58.602 INFO:tasks.workunit.client.0.smithi045.stdout:op 4756 completed, throughput=5MB/sec 2021-07-31T02:20:58.602 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Hu98jXjv3LG4FMO off=583093 len=510335 2021-07-31T02:20:58.606 INFO:tasks.workunit.client.0.smithi045.stdout:op 4757 completed, throughput=5MB/sec 2021-07-31T02:20:58.606 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-z7rc9o3-Ya94OZo off=811614 len=1966442 2021-07-31T02:20:58.611 INFO:tasks.workunit.client.0.smithi045.stdout:op 4758 completed, throughput=5MB/sec 2021-07-31T02:20:59.611 INFO:tasks.workunit.client.0.smithi045.stdout: 934: throughput=5MB/sec pending data=0 2021-07-31T02:20:59.612 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-noHmvD6qT9p4_l1 off=1012137 len=41458 2021-07-31T02:20:59.613 INFO:tasks.workunit.client.0.smithi045.stdout:op 4759 completed, throughput=5MB/sec 2021-07-31T02:20:59.613 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-4H62YZXJeTdqhjy off=3311998 len=1669219 2021-07-31T02:20:59.620 INFO:tasks.workunit.client.0.smithi045.stdout:op 4760 completed, throughput=5MB/sec 2021-07-31T02:20:59.621 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-np1FoUFwC7Jotac off=496191 len=1119247 2021-07-31T02:20:59.626 INFO:tasks.workunit.client.0.smithi045.stdout:op 4761 completed, throughput=5MB/sec 2021-07-31T02:20:59.627 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj--FgOCfILbFKqxNV off=176451 len=1949316 2021-07-31T02:20:59.636 INFO:tasks.workunit.client.0.smithi045.stdout:op 4762 completed, throughput=5MB/sec 2021-07-31T02:21:00.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:00 smithi154 conmon[34553]: cluster 2021-07-31T02:20:58.274910+0000 mgr.x (mgr.24542) 1447 : cluster [DBG] pgmap v1884: 9 pgs: 9 active+clean; 127 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.2 MiB/s wr, 3 op/s 2021-07-31T02:21:00.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:00 smithi045 conmon[33655]: cluster 2021-07-31T02:20:58.274910+0000 mgr.x (mgr.24542) 1447 : cluster 2021-07-31T02:21:00.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:00 smithi045 conmon[33655]: [DBG] pgmap v1884: 9 pgs: 9 active+clean; 127 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.2 MiB/s wr, 3 op/s 2021-07-31T02:21:00.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:00 smithi045 conmon[39718]: cluster 2021-07-31T02:20:58.274910 2021-07-31T02:21:00.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:00 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 1447 : cluster [DBG] pgmap v1884: 9 pgs: 9 active+clean; 127 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.2 MiB/s wr, 3 op/s 2021-07-31T02:21:00.636 INFO:tasks.workunit.client.0.smithi045.stdout: 935: throughput=5MB/sec pending data=0 2021-07-31T02:21:00.636 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-8U8YjdwJX4MYRKU off=555991 len=1879585 2021-07-31T02:21:00.638 INFO:tasks.workunit.client.0.smithi045.stdout:op 4763 completed, throughput=5MB/sec 2021-07-31T02:21:00.639 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-kiBU4DmXj_bO0fp off=9749747 len=1260970 2021-07-31T02:21:00.641 INFO:tasks.workunit.client.0.smithi045.stdout:op 4764 completed, throughput=5MB/sec 2021-07-31T02:21:00.641 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-xt09DkSh_BbpiuV off=9958609 len=1996225 2021-07-31T02:21:00.643 INFO:tasks.workunit.client.0.smithi045.stdout:op 4765 completed, throughput=5MB/sec 2021-07-31T02:21:01.644 INFO:tasks.workunit.client.0.smithi045.stdout: 936: throughput=5MB/sec pending data=0 2021-07-31T02:21:01.645 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-IBCAFYqKQWRb8g8 off=1286174 len=624650 2021-07-31T02:21:01.645 INFO:tasks.workunit.client.0.smithi045.stdout:op 4766 completed, throughput=5MB/sec 2021-07-31T02:21:01.645 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-UfNa_rdxhEtZBKA off=2569970 len=1429436 2021-07-31T02:21:01.648 INFO:tasks.workunit.client.0.smithi045.stdout:op 4767 completed, throughput=5MB/sec 2021-07-31T02:21:01.649 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-IET4h92A73x2Np_ off=14058549 len=277006 2021-07-31T02:21:01.649 INFO:tasks.workunit.client.0.smithi045.stdout:op 4768 completed, throughput=5MB/sec 2021-07-31T02:21:01.650 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-bs41xwfLFN-3N2i off=3582371 len=1093137 2021-07-31T02:21:01.652 INFO:tasks.workunit.client.0.smithi045.stdout:op 4769 completed, throughput=5MB/sec 2021-07-31T02:21:01.652 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-PYBPMGkrWmkCVDL off=22348244 len=78877 2021-07-31T02:21:01.655 INFO:tasks.workunit.client.0.smithi045.stdout:op 4770 completed, throughput=5MB/sec 2021-07-31T02:21:01.655 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-bT8BkhYKB2-fkp- off=12295051 len=1165568 2021-07-31T02:21:01.657 INFO:tasks.workunit.client.0.smithi045.stdout:op 4771 completed, throughput=5MB/sec 2021-07-31T02:21:01.657 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-8AKWa5OeO6MVjj8 off=8342150 len=657181 2021-07-31T02:21:01.661 INFO:tasks.workunit.client.0.smithi045.stdout:op 4772 completed, throughput=5MB/sec 2021-07-31T02:21:02.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:02 smithi154 conmon[34553]: cluster 2021-07-31T02:21:00.275359+0000 mgr.x (mgr.24542) 1448 : cluster [DBG] pgmap v1885: 9 pgs: 9 active+clean; 127 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 3.4 MiB/s rd, 3.0 MiB/s wr, 5 op/s 2021-07-31T02:21:02.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:02 smithi154 conmon[34553]: 2021-07-31T02:21:02.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:02 smithi045 conmon[33655]: cluster 2021-07-31T02:21:00.275359+0000 mgr.x (mgr.24542) 1448 2021-07-31T02:21:02.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:02 smithi045 conmon[33655]: : cluster [DBG] pgmap v1885: 9 pgs: 9 active+clean; 127 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 3.4 MiB/s rd, 3.0 MiB/s wr, 5 op/s 2021-07-31T02:21:02.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:02 smithi045 conmon[39718]: cluster 2021-07-31T02:21:00.275359 2021-07-31T02:21:02.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:02 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 1448 : cluster [DBG] pgmap v1885: 9 pgs: 9 active+clean; 127 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 3.4 MiB/s rd, 3.0 MiB/s wr, 5 op/s 2021-07-31T02:21:02.662 INFO:tasks.workunit.client.0.smithi045.stdout: 937: throughput=5MB/sec pending data=0 2021-07-31T02:21:02.662 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-kiaWOyoSUD_PdoI off=1364621 len=935612 2021-07-31T02:21:02.664 INFO:tasks.workunit.client.0.smithi045.stdout:op 4773 completed, throughput=5MB/sec 2021-07-31T02:21:02.664 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-CVYXGkwgQ-Y0zHL off=12064261 len=1922088 2021-07-31T02:21:02.673 INFO:tasks.workunit.client.0.smithi045.stdout:op 4774 completed, throughput=5MB/sec 2021-07-31T02:21:02.673 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-3exurqFdidI7Jxa off=20304303 len=1515447 2021-07-31T02:21:02.676 INFO:tasks.workunit.client.0.smithi045.stdout:op 4775 completed, throughput=5MB/sec 2021-07-31T02:21:02.676 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-cgLuiMTOuEN_MMq off=9707630 len=836781 2021-07-31T02:21:02.678 INFO:tasks.workunit.client.0.smithi045.stdout:op 4776 completed, throughput=5MB/sec 2021-07-31T02:21:03.678 INFO:tasks.workunit.client.0.smithi045.stdout: 938: throughput=5MB/sec pending data=0 2021-07-31T02:21:03.679 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-4KT6bZJSojrj952 off=4343091 len=2017349 2021-07-31T02:21:03.688 INFO:tasks.workunit.client.0.smithi045.stdout:op 4777 completed, throughput=5MB/sec 2021-07-31T02:21:03.689 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-X_osljzjEQM2FN5 off=6334180 len=471845 2021-07-31T02:21:03.692 INFO:tasks.workunit.client.0.smithi045.stdout:op 4778 completed, throughput=5MB/sec 2021-07-31T02:21:03.693 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-phPKa1PWaGBpBEQ off=14090075 len=326416 2021-07-31T02:21:03.696 INFO:tasks.workunit.client.0.smithi045.stdout:op 4779 completed, throughput=5MB/sec 2021-07-31T02:21:03.696 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-FPv-dxrBiMUw_Pd off=4504222 len=1958702 2021-07-31T02:21:03.703 INFO:tasks.workunit.client.0.smithi045.stdout:op 4780 completed, throughput=5MB/sec 2021-07-31T02:21:03.704 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ggkUzyjbYkBV9WO off=3349368 len=1730380 2021-07-31T02:21:03.711 INFO:tasks.workunit.client.0.smithi045.stdout:op 4781 completed, throughput=5MB/sec 2021-07-31T02:21:04.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:21:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:21:03.567Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:21:04.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:21:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:21:03.567Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:21:04.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:21:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:21:03.567Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:21:04.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:04 smithi154 conmon[34553]: cluster 2021-07-31T02:21:02.275987+0000 mgr.x (mgr.24542) 1449 : cluster [DBG] pgmap v1886: 9 pgs: 9 active+clean; 127 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 2.4 MiB/s rd, 1.8 MiB/s wr, 3 op/s 2021-07-31T02:21:04.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:04 smithi045 conmon[33655]: cluster 2021-07-31T02:21:02.275987+0000 2021-07-31T02:21:04.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:04 smithi045 conmon[33655]: mgr.x (mgr.24542) 1449 : cluster [DBG] pgmap v1886: 9 pgs: 9 active+clean; 127 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 2.4 MiB/s rd, 1.8 MiB/s wr, 3 op/s 2021-07-31T02:21:04.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:04 smithi045 conmon[39718]: cluster 2021-07-31T02:21:02.275987+0000 mgr.x 2021-07-31T02:21:04.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:04 smithi045 conmon[39718]: (mgr.24542) 1449 : cluster [DBG] pgmap v1886: 9 pgs: 9 active+clean; 127 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 2.4 MiB/s rd, 1.8 MiB/s wr, 3 op/s 2021-07-31T02:21:04.711 INFO:tasks.workunit.client.0.smithi045.stdout: 939: throughput=5MB/sec pending data=0 2021-07-31T02:21:04.712 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-o_5URweAS_YyyKk off=4319109 len=1143411 2021-07-31T02:21:04.718 INFO:tasks.workunit.client.0.smithi045.stdout:op 4782 completed, throughput=5MB/sec 2021-07-31T02:21:04.718 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj--PV7L_TWi0FP92p off=345543 len=671993 2021-07-31T02:21:04.719 INFO:tasks.workunit.client.0.smithi045.stdout:op 4783 completed, throughput=5MB/sec 2021-07-31T02:21:04.720 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-0wpX5oqPsWnTWGM off=8716790 len=996318 2021-07-31T02:21:04.725 INFO:tasks.workunit.client.0.smithi045.stdout:op 4784 completed, throughput=5MB/sec 2021-07-31T02:21:04.725 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-hXH7DaEecWWlqi2 off=6594587 len=1976989 2021-07-31T02:21:04.729 INFO:tasks.workunit.client.0.smithi045.stdout:op 4785 completed, throughput=5MB/sec 2021-07-31T02:21:05.729 INFO:tasks.workunit.client.0.smithi045.stdout: 940: throughput=5MB/sec pending data=0 2021-07-31T02:21:05.729 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-0nu8FhSFIiCPs1V off=6572121 len=64122 2021-07-31T02:21:05.732 INFO:tasks.workunit.client.0.smithi045.stdout:op 4786 completed, throughput=5MB/sec 2021-07-31T02:21:05.732 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-1QrWZd9IaDwjdMk off=8467160 len=498626 2021-07-31T02:21:05.733 INFO:tasks.workunit.client.0.smithi045.stdout:op 4787 completed, throughput=5MB/sec 2021-07-31T02:21:05.734 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-9h9Vs191JVTUf9J off=3653817 len=972699 2021-07-31T02:21:05.735 INFO:tasks.workunit.client.0.smithi045.stdout:op 4788 completed, throughput=5MB/sec 2021-07-31T02:21:05.735 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-S1bOZv9SizZ7wBp off=15936748 len=1812783 2021-07-31T02:21:05.736 INFO:tasks.workunit.client.0.smithi045.stdout:op 4789 completed, throughput=5MB/sec 2021-07-31T02:21:05.737 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ilII2E07xCdpyBi off=2132856 len=1761316 2021-07-31T02:21:05.740 INFO:tasks.workunit.client.0.smithi045.stdout:op 4790 completed, throughput=5MB/sec 2021-07-31T02:21:06.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:06 smithi154 conmon[34553]: cluster 2021-07-31T02:21:04.276490+0000 mgr.x (mgr.24542) 1450 : cluster [DBG] pgmap v1887: 9 pgs: 9 active+clean; 127 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 3.4 MiB/s rd, 2.9 MiB/s wr, 5 op/s 2021-07-31T02:21:06.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:06 smithi045 conmon[33655]: cluster 2021-07-31T02:21:04.276490+0000 mgr.x (mgr.24542) 1450 : cluster [DBG] pgmap v1887: 9 pgs: 9 active+clean; 127 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 3.4 MiB/s rd, 2.9 MiB/s wr, 5 op/s 2021-07-31T02:21:06.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:06 smithi045 conmon[39718]: cluster 2021-07-31T02:21:04 2021-07-31T02:21:06.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:06 smithi045 conmon[39718]: .276490+0000 mgr.x (mgr.24542) 1450 : cluster [DBG] pgmap v1887: 9 pgs: 9 active+clean; 127 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 3.4 MiB/s rd, 2.9 MiB/s wr, 5 op/s 2021-07-31T02:21:06.740 INFO:tasks.workunit.client.0.smithi045.stdout: 941: throughput=5MB/sec pending data=0 2021-07-31T02:21:06.741 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-myiTl1r0Xdv3bhz off=13148062 len=630506 2021-07-31T02:21:06.742 INFO:tasks.workunit.client.0.smithi045.stdout:op 4791 completed, throughput=5MB/sec 2021-07-31T02:21:06.742 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-XljygX5xR7VBQ_j off=9393158 len=655716 2021-07-31T02:21:06.742 INFO:tasks.workunit.client.0.smithi045.stdout:op 4792 completed, throughput=5MB/sec 2021-07-31T02:21:06.743 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-xFF0hmawuIsJCWo off=13568919 len=788740 2021-07-31T02:21:06.745 INFO:tasks.workunit.client.0.smithi045.stdout:op 4793 completed, throughput=5MB/sec 2021-07-31T02:21:06.745 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-yjgOZKTKifsBKTl off=1975481 len=1994398 2021-07-31T02:21:06.755 INFO:tasks.workunit.client.0.smithi045.stdout:op 4794 completed, throughput=5MB/sec 2021-07-31T02:21:06.756 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-NDU_h2mNTmnER-S off=1790759 len=1760070 2021-07-31T02:21:06.760 INFO:tasks.workunit.client.0.smithi045.stdout:op 4795 completed, throughput=5MB/sec 2021-07-31T02:21:07.760 INFO:tasks.workunit.client.0.smithi045.stdout: 942: throughput=5MB/sec pending data=0 2021-07-31T02:21:07.761 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-bKs4xjkjyaNGQ7Z off=15810198 len=1837749 2021-07-31T02:21:07.769 INFO:tasks.workunit.client.0.smithi045.stdout:op 4796 completed, throughput=5MB/sec 2021-07-31T02:21:07.769 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-RUhbY7b-0qNlueq off=767980 len=434633 2021-07-31T02:21:07.770 INFO:tasks.workunit.client.0.smithi045.stdout:op 4797 completed, throughput=5MB/sec 2021-07-31T02:21:07.770 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-VQqtBX05ZWB81kG off=10866122 len=1796902 2021-07-31T02:21:07.772 INFO:tasks.workunit.client.0.smithi045.stdout:op 4798 completed, throughput=5MB/sec 2021-07-31T02:21:07.772 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-5DCO2Gr8kEeEF_r off=6189723 len=1252985 2021-07-31T02:21:07.775 INFO:tasks.workunit.client.0.smithi045.stdout:op 4799 completed, throughput=5MB/sec 2021-07-31T02:21:08.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:08 smithi154 conmon[34553]: cluster 2021-07-31T02:21:06.276933+0000 mgr.x (mgr.24542) 1451 : cluster [DBG] pgmap v1888: 9 pgs: 9 active+clean; 127 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 2.4 MiB/s rd, 1.8 MiB/s wr, 3 op/s 2021-07-31T02:21:08.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:08 smithi045 conmon[39718]: cluster 2021-07-31T02:21:06.276933+0000 mgr.x (mgr.24542) 1451 : 2021-07-31T02:21:08.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:08 smithi045 conmon[39718]: cluster [DBG] pgmap v1888: 9 pgs: 9 active+clean; 127 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 2.4 MiB/s rd, 1.8 MiB/s wr, 3 op/s 2021-07-31T02:21:08.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:08 smithi045 conmon[33655]: cluster 2021-07-31T02:21:06.276933+0000 mgr.x (mgr.24542) 1451 : cluster [DBG] pgmap v1888: 9 pgs: 9 active+clean; 127 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 2.4 MiB/s rd, 1.8 MiB/s wr, 3 op/s 2021-07-31T02:21:08.775 INFO:tasks.workunit.client.0.smithi045.stdout: 943: throughput=5MB/sec pending data=0 2021-07-31T02:21:08.776 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-9MVgMW3qzq7tzw5 off=202849 len=1539805 2021-07-31T02:21:08.777 INFO:tasks.workunit.client.0.smithi045.stdout:op 4800 completed, throughput=5MB/sec 2021-07-31T02:21:08.777 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-WahAyTNcRKpGCeG off=2309232 len=922793 2021-07-31T02:21:08.783 INFO:tasks.workunit.client.0.smithi045.stdout:op 4801 completed, throughput=5MB/sec 2021-07-31T02:21:08.783 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-iUXFo2bQda8Yp_T off=3708684 len=1734724 2021-07-31T02:21:08.785 INFO:tasks.workunit.client.0.smithi045.stdout:op 4802 completed, throughput=5MB/sec 2021-07-31T02:21:08.785 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-UEPv9dckWr9KLtq off=6814233 len=730140 2021-07-31T02:21:08.787 INFO:tasks.workunit.client.0.smithi045.stdout:op 4803 completed, throughput=5MB/sec 2021-07-31T02:21:09.787 INFO:tasks.workunit.client.0.smithi045.stdout: 944: throughput=5MB/sec pending data=0 2021-07-31T02:21:09.788 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ojM3n2kqEGT99nf off=6418135 len=1561503 2021-07-31T02:21:09.795 INFO:tasks.workunit.client.0.smithi045.stdout:op 4804 completed, throughput=5MB/sec 2021-07-31T02:21:09.795 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-NQaPYkcwtpdvAoj off=8514175 len=583682 2021-07-31T02:21:09.796 INFO:tasks.workunit.client.0.smithi045.stdout:op 4805 completed, throughput=5MB/sec 2021-07-31T02:21:09.796 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ua1jwMhfFk4SwnY off=0 len=1315984 2021-07-31T02:21:09.797 INFO:tasks.workunit.client.0.smithi045.stdout:op 4806 completed, throughput=5MB/sec 2021-07-31T02:21:09.798 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-xJR6i6olMjmxXzo off=3232957 len=819982 2021-07-31T02:21:09.798 INFO:tasks.workunit.client.0.smithi045.stdout:op 4807 completed, throughput=5MB/sec 2021-07-31T02:21:09.799 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-y3aD2qYo449EAC2 off=8690936 len=1914721 2021-07-31T02:21:09.803 INFO:tasks.workunit.client.0.smithi045.stdout:op 4808 completed, throughput=5MB/sec 2021-07-31T02:21:10.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:10 smithi154 conmon[34553]: cluster 2021-07-31T02:21:08.277445+0000 mgr.x (mgr.24542) 1452 : cluster [DBG] pgmap v1889: 9 pgs: 9 active+clean; 127 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 2.4 MiB/s rd, 1.8 MiB/s wr, 3 op/s 2021-07-31T02:21:10.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:10 smithi045 conmon[33655]: cluster 2021-07-31T02:21:08.277445+0000 mgr.x (mgr.24542) 1452 : cluster 2021-07-31T02:21:10.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:10 smithi045 conmon[33655]: [DBG] pgmap v1889: 9 pgs: 9 active+clean; 127 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 2.4 MiB/s rd, 1.8 MiB/s wr, 3 op/s 2021-07-31T02:21:10.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:10 smithi045 conmon[39718]: cluster 2021-07-31T02:21:08.277445+0000 mgr.x (mgr.24542) 1452 : cluster 2021-07-31T02:21:10.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:10 smithi045 conmon[39718]: [DBG] pgmap v1889: 9 pgs: 9 active+clean; 127 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 2.4 MiB/s rd, 1.8 MiB/s wr, 3 op/s 2021-07-31T02:21:10.801 INFO:tasks.workunit.client.0.smithi045.stdout: 945: throughput=5MB/sec pending data=0 2021-07-31T02:21:10.802 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-eHbpOx-7yru3sET off=7003358 len=2022345 2021-07-31T02:21:10.810 INFO:tasks.workunit.client.0.smithi045.stdout:op 4809 completed, throughput=5MB/sec 2021-07-31T02:21:10.811 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-DjR96ZZrblDIiNP off=4518561 len=284057 2021-07-31T02:21:10.812 INFO:tasks.workunit.client.0.smithi045.stdout:op 4810 completed, throughput=5MB/sec 2021-07-31T02:21:10.812 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-mrtnxBGkFNwd84H off=5241919 len=714330 2021-07-31T02:21:10.813 INFO:tasks.workunit.client.0.smithi045.stdout:op 4811 completed, throughput=5MB/sec 2021-07-31T02:21:10.814 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-jgJuRe3WPZrptfR off=1148135 len=704438 2021-07-31T02:21:10.817 INFO:tasks.workunit.client.0.smithi045.stdout:op 4812 completed, throughput=5MB/sec 2021-07-31T02:21:10.817 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-sTRH4C198L4fW-T off=10240744 len=69760 2021-07-31T02:21:10.820 INFO:tasks.workunit.client.0.smithi045.stdout:op 4813 completed, throughput=5MB/sec 2021-07-31T02:21:11.820 INFO:tasks.workunit.client.0.smithi045.stdout: 946: throughput=4.99MB/sec pending data=0 2021-07-31T02:21:11.821 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-yQIqLROkMb7_0Bw off=1498005 len=1006151 2021-07-31T02:21:11.826 INFO:tasks.workunit.client.0.smithi045.stdout:op 4814 completed, throughput=5MB/sec 2021-07-31T02:21:11.827 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-I4nJkJdASX0cFAC off=7002531 len=1353318 2021-07-31T02:21:11.829 INFO:tasks.workunit.client.0.smithi045.stdout:op 4815 completed, throughput=5MB/sec 2021-07-31T02:21:11.829 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-zWhbE-xfVrK-Ukc off=2896523 len=931812 2021-07-31T02:21:11.831 INFO:tasks.workunit.client.0.smithi045.stdout:op 4816 completed, throughput=5MB/sec 2021-07-31T02:21:11.832 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-uqUMAUtkAbwPg4M off=1119355 len=1902544 2021-07-31T02:21:11.843 INFO:tasks.workunit.client.0.smithi045.stdout:op 4817 completed, throughput=5MB/sec 2021-07-31T02:21:11.843 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-FNbHsy74-OQpNUc off=7122227 len=2002147 2021-07-31T02:21:11.853 INFO:tasks.workunit.client.0.smithi045.stdout:op 4818 completed, throughput=5MB/sec 2021-07-31T02:21:12.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:12 smithi154 conmon[34553]: cluster 2021-07-31T02:21:10.277996+0000 mgr.x (mgr.24542) 1453 : cluster [DBG] pgmap v1890: 9 pgs: 9 active+clean; 127 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 3.9 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2021-07-31T02:21:12.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:12 smithi045 conmon[39718]: cluster 2021-07-31T02:21:10.277996+0000 mgr.x (mgr.24542) 1453 : cluster 2021-07-31T02:21:12.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:12 smithi045 conmon[39718]: [DBG] pgmap v1890: 9 pgs: 9 active+clean; 127 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 3.9 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2021-07-31T02:21:12.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:12 smithi045 conmon[33655]: cluster 2021-07-31T02:21:10.277996+0000 mgr.x (mgr.24542) 1453 : cluster [DBG] 2021-07-31T02:21:12.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:12 smithi045 conmon[33655]: pgmap v1890: 9 pgs: 9 active+clean; 127 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 3.9 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2021-07-31T02:21:12.853 INFO:tasks.workunit.client.0.smithi045.stdout: 947: throughput=5MB/sec pending data=0 2021-07-31T02:21:12.853 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-llFMZ_mHErL-NAD off=2495287 len=1087660 2021-07-31T02:21:12.860 INFO:tasks.workunit.client.0.smithi045.stdout:op 4819 completed, throughput=5MB/sec 2021-07-31T02:21:12.860 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-8Gw8MJe3mI4yjl6 off=3249806 len=674595 2021-07-31T02:21:12.864 INFO:tasks.workunit.client.0.smithi045.stdout:op 4820 completed, throughput=5MB/sec 2021-07-31T02:21:12.865 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-sLFpDAUDBXzsSgl off=6127221 len=849596 2021-07-31T02:21:12.869 INFO:tasks.workunit.client.0.smithi045.stdout:op 4821 completed, throughput=5MB/sec 2021-07-31T02:21:12.869 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-APrKPajdxkq-k8q off=7578316 len=1126593 2021-07-31T02:21:12.877 INFO:tasks.workunit.client.0.smithi045.stdout:op 4822 completed, throughput=5MB/sec 2021-07-31T02:21:13.876 INFO:tasks.workunit.client.0.smithi045.stdout: 948: throughput=4.99MB/sec pending data=0 2021-07-31T02:21:13.877 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-CHels6O5D9hmpWN off=375657 len=199059 2021-07-31T02:21:13.877 INFO:tasks.workunit.client.0.smithi045.stdout:op 4823 completed, throughput=5MB/sec 2021-07-31T02:21:13.878 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-AHdKZ6XqLvlBUT2 off=6637357 len=1603842 2021-07-31T02:21:13.885 INFO:tasks.workunit.client.0.smithi045.stdout:op 4824 completed, throughput=5MB/sec 2021-07-31T02:21:13.886 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-GJrWSPjo4A6YVDV off=6509248 len=88038 2021-07-31T02:21:13.886 INFO:tasks.workunit.client.0.smithi045.stdout:op 4825 completed, throughput=5MB/sec 2021-07-31T02:21:13.887 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-KuRAYuVyXDeR3Za off=1748753 len=1969417 2021-07-31T02:21:13.896 INFO:tasks.workunit.client.0.smithi045.stdout:op 4826 completed, throughput=5MB/sec 2021-07-31T02:21:13.896 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-4ufwgivbkiTFOo5 off=3981879 len=1650303 2021-07-31T02:21:13.897 INFO:tasks.workunit.client.0.smithi045.stdout:op 4827 completed, throughput=5MB/sec 2021-07-31T02:21:14.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:21:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:21:13.567Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:21:14.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:21:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:21:13.567Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:21:14.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:21:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:21:13.567Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:21:14.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:14 smithi154 conmon[34553]: cluster 2021-07-31T02:21:12.278445+0000 mgr.x (mgr.24542) 1454 : cluster [DBG] pgmap v1891: 9 pgs: 9 active+clean; 127 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.7 MiB/s wr, 3 op/s 2021-07-31T02:21:14.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:14 smithi045 conmon[39718]: cluster 2021-07-31T02:21:12.278445+0000 mgr.x (mgr 2021-07-31T02:21:14.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:14 smithi045 conmon[39718]: .24542) 1454 : cluster [DBG] pgmap v1891: 9 pgs: 9 active+clean; 127 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.7 MiB/s wr, 3 op/s 2021-07-31T02:21:14.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:14 smithi045 conmon[33655]: cluster 2021-07-31T02:21:12.278445+0000 2021-07-31T02:21:14.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:14 smithi045 conmon[33655]: mgr.x (mgr.24542) 1454 : cluster [DBG] pgmap v1891: 9 pgs: 9 active+clean; 127 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.7 MiB/s wr, 3 op/s 2021-07-31T02:21:14.897 INFO:tasks.workunit.client.0.smithi045.stdout: 949: throughput=5MB/sec pending data=0 2021-07-31T02:21:14.898 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-5fPWihOkBFu0s2V off=8166644 len=245639 2021-07-31T02:21:14.899 INFO:tasks.workunit.client.0.smithi045.stdout:op 4828 completed, throughput=5MB/sec 2021-07-31T02:21:14.899 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ihovyqIStkJJgL_ off=14656214 len=1307935 2021-07-31T02:21:14.901 INFO:tasks.workunit.client.0.smithi045.stdout:op 4829 completed, throughput=5MB/sec 2021-07-31T02:21:14.901 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-u_KNvaSorTZmwwq off=6749180 len=127211 2021-07-31T02:21:14.904 INFO:tasks.workunit.client.0.smithi045.stdout:op 4830 completed, throughput=5MB/sec 2021-07-31T02:21:14.904 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-VeS3O1EWgqlX1Gd off=20784799 len=509101 2021-07-31T02:21:14.908 INFO:tasks.workunit.client.0.smithi045.stdout:op 4831 completed, throughput=5MB/sec 2021-07-31T02:21:14.908 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-FhDFANoVMVOFrPa off=10193213 len=1948302 2021-07-31T02:21:14.913 INFO:tasks.workunit.client.0.smithi045.stdout:op 4832 completed, throughput=5MB/sec 2021-07-31T02:21:14.913 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-dm8ZT5H-reFLcaO off=11949334 len=1083319 2021-07-31T02:21:14.918 INFO:tasks.workunit.client.0.smithi045.stdout:op 4833 completed, throughput=5MB/sec 2021-07-31T02:21:15.918 INFO:tasks.workunit.client.0.smithi045.stdout: 950: throughput=5MB/sec pending data=0 2021-07-31T02:21:15.918 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-MowuEa5EuK08jXK off=192889 len=1755338 2021-07-31T02:21:15.927 INFO:tasks.workunit.client.0.smithi045.stdout:op 4834 completed, throughput=5MB/sec 2021-07-31T02:21:15.927 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Sdupg3nWQPSaFAV off=2202550 len=62813 2021-07-31T02:21:15.928 INFO:tasks.workunit.client.0.smithi045.stdout:op 4835 completed, throughput=5MB/sec 2021-07-31T02:21:15.928 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-bm5KAF2PMUa_pGV off=2319069 len=1418718 2021-07-31T02:21:15.937 INFO:tasks.workunit.client.0.smithi045.stdout:op 4836 completed, throughput=5MB/sec 2021-07-31T02:21:15.937 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-o0yNZ9ZPtxAUnK0 off=22653764 len=1752911 2021-07-31T02:21:15.945 INFO:tasks.workunit.client.0.smithi045.stdout:op 4837 completed, throughput=5MB/sec 2021-07-31T02:21:15.946 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-GeCN4Wsnv17TvQp off=9930708 len=1545254 2021-07-31T02:21:15.947 INFO:tasks.workunit.client.0.smithi045.stdout:op 4838 completed, throughput=5MB/sec 2021-07-31T02:21:16.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:16 smithi154 conmon[34553]: cluster 2021-07-31T02:21:14.278959+0000 mgr.x (mgr.24542) 1455 : cluster [DBG] pgmap v1892: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 3.2 MiB/s rd, 3.0 MiB/s wr, 5 op/s 2021-07-31T02:21:16.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:16 smithi045 conmon[39718]: cluster 2021-07-31T02:21:14.278959+0000 2021-07-31T02:21:16.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:16 smithi045 conmon[39718]: mgr.x (mgr.24542) 1455 : cluster [DBG] pgmap v1892: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 3.2 MiB/s rd, 3.0 MiB/s wr, 5 op/s 2021-07-31T02:21:16.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:16 smithi045 conmon[33655]: cluster 2021-07-31T02:21:14.278959 2021-07-31T02:21:16.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:16 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 1455 : cluster [DBG] pgmap v1892: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 3.2 MiB/s rd, 3.0 MiB/s wr, 5 op/s 2021-07-31T02:21:16.947 INFO:tasks.workunit.client.0.smithi045.stdout: 951: throughput=5MB/sec pending data=0 2021-07-31T02:21:16.947 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-gbVJalfHyiOCJsz off=5423167 len=794056 2021-07-31T02:21:16.948 INFO:tasks.workunit.client.0.smithi045.stdout:op 4839 completed, throughput=5MB/sec 2021-07-31T02:21:16.949 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-X_osljzjEQM2FN5 off=6735921 len=246622 2021-07-31T02:21:16.950 INFO:tasks.workunit.client.0.smithi045.stdout:op 4840 completed, throughput=5MB/sec 2021-07-31T02:21:16.951 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-AJg8Ck7NexIHGQI off=1864716 len=1660612 2021-07-31T02:21:16.953 INFO:tasks.workunit.client.0.smithi045.stdout:op 4841 completed, throughput=5MB/sec 2021-07-31T02:21:16.953 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-3AqxU8iTPDCeIxA off=9103219 len=772515 2021-07-31T02:21:16.954 INFO:tasks.workunit.client.0.smithi045.stdout:op 4842 completed, throughput=5MB/sec 2021-07-31T02:21:16.954 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-KO_Gy0vS-z8T6f- off=5333528 len=532254 2021-07-31T02:21:16.955 INFO:tasks.workunit.client.0.smithi045.stdout:op 4843 completed, throughput=5MB/sec 2021-07-31T02:21:17.956 INFO:tasks.workunit.client.0.smithi045.stdout: 952: throughput=4.99MB/sec pending data=0 2021-07-31T02:21:17.956 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-L1g7PoEZ3eRKz_8 off=6014258 len=1685323 2021-07-31T02:21:17.965 INFO:tasks.workunit.client.0.smithi045.stdout:op 4844 completed, throughput=5MB/sec 2021-07-31T02:21:17.966 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-MOqcIfoZZrteIcQ off=1609951 len=27901 2021-07-31T02:21:17.968 INFO:tasks.workunit.client.0.smithi045.stdout:op 4845 completed, throughput=5MB/sec 2021-07-31T02:21:17.968 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-M0ZbZIBvnTN267E off=4423092 len=1373867 2021-07-31T02:21:17.974 INFO:tasks.workunit.client.0.smithi045.stdout:op 4846 completed, throughput=5MB/sec 2021-07-31T02:21:17.974 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-VOA2WvRgCDvXOvE off=1419023 len=178992 2021-07-31T02:21:17.975 INFO:tasks.workunit.client.0.smithi045.stdout:op 4847 completed, throughput=5MB/sec 2021-07-31T02:21:17.975 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Dwij56PWqC39e1t off=1418705 len=119446 2021-07-31T02:21:17.978 INFO:tasks.workunit.client.0.smithi045.stdout:op 4848 completed, throughput=5MB/sec 2021-07-31T02:21:17.978 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-O0bTzVs8HAlXAam off=1336227 len=627194 2021-07-31T02:21:17.982 INFO:tasks.workunit.client.0.smithi045.stdout:op 4849 completed, throughput=5MB/sec 2021-07-31T02:21:17.983 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-b0zjj4ccgRgu8o4 off=322962 len=1039685 2021-07-31T02:21:17.984 INFO:tasks.workunit.client.0.smithi045.stdout:op 4850 completed, throughput=5MB/sec 2021-07-31T02:21:17.985 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-kYwGHDnu_93hdGF off=13309481 len=439084 2021-07-31T02:21:17.988 INFO:tasks.workunit.client.0.smithi045.stdout:op 4851 completed, throughput=5MB/sec 2021-07-31T02:21:18.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:18 smithi154 conmon[34553]: cluster 2021-07-31T02:21:16.279594+0000 mgr.x (mgr.24542) 1456 : cluster [DBG] pgmap v1893: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 2.1 MiB/s rd, 1.9 MiB/s wr, 3 op/s 2021-07-31T02:21:18.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:18 smithi045 conmon[33655]: cluster 2021-07-31T02:21:16.279594 2021-07-31T02:21:18.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:18 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 1456 : cluster [DBG] pgmap v1893: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 2.1 MiB/s rd, 1.9 MiB/s wr, 3 op/s 2021-07-31T02:21:18.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:18 smithi045 conmon[39718]: cluster 2021-07-31T02:21:16.279594+0000 mgr.x (mgr. 2021-07-31T02:21:18.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:18 smithi045 conmon[39718]: 24542) 1456 : cluster [DBG] pgmap v1893: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 2.1 MiB/s rd, 1.9 MiB/s wr, 3 op/s 2021-07-31T02:21:18.989 INFO:tasks.workunit.client.0.smithi045.stdout: 953: throughput=4.99MB/sec pending data=0 2021-07-31T02:21:18.989 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-o9srov6v_B_7iZw off=12754081 len=1435040 2021-07-31T02:21:18.996 INFO:tasks.workunit.client.0.smithi045.stdout:op 4852 completed, throughput=5MB/sec 2021-07-31T02:21:18.996 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-JwLSvZdCjWwsZw9 off=4018511 len=1494398 2021-07-31T02:21:19.004 INFO:tasks.workunit.client.0.smithi045.stdout:op 4853 completed, throughput=5MB/sec 2021-07-31T02:21:19.004 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-2oDxJp4TfQRYC-m off=7614370 len=223807 2021-07-31T02:21:19.005 INFO:tasks.workunit.client.0.smithi045.stdout:op 4854 completed, throughput=5MB/sec 2021-07-31T02:21:19.005 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ruMiaCFsvN1gSq_ off=3426022 len=2025711 2021-07-31T02:21:19.014 INFO:tasks.workunit.client.0.smithi045.stdout:op 4855 completed, throughput=5MB/sec 2021-07-31T02:21:19.014 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-iVT0m7QU-Ukwx5u off=19340000 len=986753 2021-07-31T02:21:19.019 INFO:tasks.workunit.client.0.smithi045.stdout:op 4856 completed, throughput=5MB/sec 2021-07-31T02:21:20.019 INFO:tasks.workunit.client.0.smithi045.stdout: 954: throughput=5MB/sec pending data=0 2021-07-31T02:21:20.019 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-w1qwwnbv2fUsnzB off=11731538 len=628599 2021-07-31T02:21:20.024 INFO:tasks.workunit.client.0.smithi045.stdout:op 4857 completed, throughput=5MB/sec 2021-07-31T02:21:20.024 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj--E2-IuAhI_T3pcy off=8590040 len=697747 2021-07-31T02:21:20.026 INFO:tasks.workunit.client.0.smithi045.stdout:op 4858 completed, throughput=5MB/sec 2021-07-31T02:21:20.026 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-UgpYju6Z7gMJb20 off=1072146 len=1912525 2021-07-31T02:21:20.028 INFO:tasks.workunit.client.0.smithi045.stdout:op 4859 completed, throughput=5MB/sec 2021-07-31T02:21:20.028 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-gFjRVRJmKUC26qj off=10517352 len=519551 2021-07-31T02:21:20.029 INFO:tasks.workunit.client.0.smithi045.stdout:op 4860 completed, throughput=5MB/sec 2021-07-31T02:21:20.030 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-wSCzbgmDJib-h7m off=12020683 len=107151 2021-07-31T02:21:20.030 INFO:tasks.workunit.client.0.smithi045.stdout:op 4861 completed, throughput=5MB/sec 2021-07-31T02:21:20.030 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-q_kSlH7C5y5e49c off=8748831 len=1897488 2021-07-31T02:21:20.033 INFO:tasks.workunit.client.0.smithi045.stdout:op 4862 completed, throughput=5MB/sec 2021-07-31T02:21:20.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:20 smithi154 conmon[34553]: cluster 2021-07-31T02:21:18.280056+0000 mgr.x (mgr.24542) 1457 : cluster [DBG] pgmap v1894: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 2.1 MiB/s rd, 1.9 MiB/s wr, 3 op/s 2021-07-31T02:21:20.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:20 smithi045 conmon[33655]: cluster 2021-07-31T02: 2021-07-31T02:21:20.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:20 smithi045 conmon[33655]: 21:18.280056+0000 mgr.x (mgr.24542) 1457 : cluster [DBG] pgmap v1894: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 2.1 MiB/s rd, 1.9 MiB/s wr, 3 op/s 2021-07-31T02:21:20.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:20 smithi045 conmon[39718]: cluster 2021-07-31T02:21:18 2021-07-31T02:21:20.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:20 smithi045 conmon[39718]: .280056+0000 mgr.x (mgr.24542) 1457 : cluster [DBG] pgmap v1894: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 2.1 MiB/s rd, 1.9 MiB/s wr, 3 op/s 2021-07-31T02:21:21.033 INFO:tasks.workunit.client.0.smithi045.stdout: 955: throughput=5MB/sec pending data=0 2021-07-31T02:21:21.034 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-VQqtBX05ZWB81kG off=12754514 len=167931 2021-07-31T02:21:21.035 INFO:tasks.workunit.client.0.smithi045.stdout:op 4863 completed, throughput=5MB/sec 2021-07-31T02:21:21.035 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-xyAzBEyIpT_Vw6g off=8706490 len=107254 2021-07-31T02:21:21.035 INFO:tasks.workunit.client.0.smithi045.stdout:op 4864 completed, throughput=5MB/sec 2021-07-31T02:21:21.035 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Q-iIQN79B4EAk4K off=9893944 len=846988 2021-07-31T02:21:21.040 INFO:tasks.workunit.client.0.smithi045.stdout:op 4865 completed, throughput=5MB/sec 2021-07-31T02:21:21.040 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-RySeRmE6zN-jFiz off=4303798 len=1387662 2021-07-31T02:21:21.048 INFO:tasks.workunit.client.0.smithi045.stdout:op 4866 completed, throughput=5MB/sec 2021-07-31T02:21:21.048 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-esSoEkbiF-5z_Vm off=5063979 len=280207 2021-07-31T02:21:21.049 INFO:tasks.workunit.client.0.smithi045.stdout:op 4867 completed, throughput=5MB/sec 2021-07-31T02:21:21.049 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Q5kh1bi-N9lHDzO off=5913432 len=1197461 2021-07-31T02:21:21.054 INFO:tasks.workunit.client.0.smithi045.stdout:op 4868 completed, throughput=5MB/sec 2021-07-31T02:21:21.054 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-svNzi6CCKNghXjB off=5700100 len=1965525 2021-07-31T02:21:21.063 INFO:tasks.workunit.client.0.smithi045.stdout:op 4869 completed, throughput=5MB/sec 2021-07-31T02:21:22.064 INFO:tasks.workunit.client.0.smithi045.stdout: 956: throughput=5MB/sec pending data=0 2021-07-31T02:21:22.064 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ETD9zOpyZNtsbl8 off=22277686 len=969425 2021-07-31T02:21:22.068 INFO:tasks.workunit.client.0.smithi045.stdout:op 4870 completed, throughput=5MB/sec 2021-07-31T02:21:22.069 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-yaccHlHpk2JJNp3 off=2939778 len=1271967 2021-07-31T02:21:22.070 INFO:tasks.workunit.client.0.smithi045.stdout:op 4871 completed, throughput=5MB/sec 2021-07-31T02:21:22.070 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-bYmt6_FdVOQgmE7 off=17111215 len=911013 2021-07-31T02:21:22.073 INFO:tasks.workunit.client.0.smithi045.stdout:op 4872 completed, throughput=5MB/sec 2021-07-31T02:21:22.073 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-JBGG-5FMgD3KX8R off=7079794 len=1273210 2021-07-31T02:21:22.075 INFO:tasks.workunit.client.0.smithi045.stdout:op 4873 completed, throughput=5MB/sec 2021-07-31T02:21:22.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:22 smithi154 conmon[34553]: cluster 2021-07-31T02:21:20.280518+0000 mgr.x (mgr.24542) 1458 : cluster [DBG] pgmap v1895: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2021-07-31T02:21:22.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:22 smithi045 conmon[33655]: cluster 2021-07-31T02:21:20.280518+0000 mgr.x (mgr.24542) 1458 : cluster [DBG] 2021-07-31T02:21:22.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:22 smithi045 conmon[33655]: pgmap v1895: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2021-07-31T02:21:22.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:22 smithi045 conmon[39718]: cluster 2021-07-31T02:21:20 2021-07-31T02:21:22.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:22 smithi045 conmon[39718]: .280518+0000 mgr.x (mgr.24542) 1458 : cluster [DBG] pgmap v1895: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2021-07-31T02:21:23.075 INFO:tasks.workunit.client.0.smithi045.stdout: 957: throughput=5MB/sec pending data=0 2021-07-31T02:21:23.075 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-2clbUlcqiX7xyh8 off=3987103 len=815530 2021-07-31T02:21:23.081 INFO:tasks.workunit.client.0.smithi045.stdout:op 4874 completed, throughput=5MB/sec 2021-07-31T02:21:23.081 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-XT04ipFEOAObExl off=5935539 len=204458 2021-07-31T02:21:23.084 INFO:tasks.workunit.client.0.smithi045.stdout:op 4875 completed, throughput=5MB/sec 2021-07-31T02:21:23.084 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-by3bFa8hEMus_w- off=17405518 len=863334 2021-07-31T02:21:23.088 INFO:tasks.workunit.client.0.smithi045.stdout:op 4876 completed, throughput=5MB/sec 2021-07-31T02:21:23.089 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ZPe6ni67A1OHxgJ off=2144426 len=248112 2021-07-31T02:21:23.091 INFO:tasks.workunit.client.0.smithi045.stdout:op 4877 completed, throughput=5MB/sec 2021-07-31T02:21:23.092 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-7qJ-NBXBoE1oyUK off=8851897 len=187561 2021-07-31T02:21:23.093 INFO:tasks.workunit.client.0.smithi045.stdout:op 4878 completed, throughput=5MB/sec 2021-07-31T02:21:23.093 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-nvXh_RiFzRqdWj5 off=1013431 len=1406909 2021-07-31T02:21:23.094 INFO:tasks.workunit.client.0.smithi045.stdout:op 4879 completed, throughput=5MB/sec 2021-07-31T02:21:23.094 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-StGbvqLrqU_emxO off=4980723 len=1061700 2021-07-31T02:21:23.096 INFO:tasks.workunit.client.0.smithi045.stdout:op 4880 completed, throughput=5MB/sec 2021-07-31T02:21:24.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:21:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:21:23.567Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:21:24.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:21:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:21:23.567Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:21:24.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:21:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:21:23.567Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:21:24.097 INFO:tasks.workunit.client.0.smithi045.stdout: 958: throughput=5MB/sec pending data=0 2021-07-31T02:21:24.097 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-QkD5nPXOH3Xs9RA off=962554 len=1009144 2021-07-31T02:21:24.103 INFO:tasks.workunit.client.0.smithi045.stdout:op 4881 completed, throughput=5MB/sec 2021-07-31T02:21:24.103 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-vbrC7OSF56KiFGA off=16700296 len=644373 2021-07-31T02:21:24.104 INFO:tasks.workunit.client.0.smithi045.stdout:op 4882 completed, throughput=5MB/sec 2021-07-31T02:21:24.105 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-0GqiYJjXNS55oYG off=528704 len=197713 2021-07-31T02:21:24.108 INFO:tasks.workunit.client.0.smithi045.stdout:op 4883 completed, throughput=5MB/sec 2021-07-31T02:21:24.108 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-e8zTMQPLAiD3o43 off=20237851 len=1160951 2021-07-31T02:21:24.115 INFO:tasks.workunit.client.0.smithi045.stdout:op 4884 completed, throughput=5MB/sec 2021-07-31T02:21:24.115 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ZVIUkCjXjT3806t off=3221509 len=1593993 2021-07-31T02:21:24.117 INFO:tasks.workunit.client.0.smithi045.stdout:op 4885 completed, throughput=5MB/sec 2021-07-31T02:21:24.118 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-L180F-RsDxn9OOQ off=953877 len=1218120 2021-07-31T02:21:24.124 INFO:tasks.workunit.client.0.smithi045.stdout:op 4886 completed, throughput=5MB/sec 2021-07-31T02:21:24.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:24 smithi154 conmon[34553]: cluster 2021-07-31T02:21:22.280963+0000 mgr.x (mgr.24542) 1459 : cluster [DBG] pgmap v1896: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.6 MiB/s rd, 2.2 MiB/s wr, 3 op/s 2021-07-31T02:21:24.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:24 smithi045 conmon[33655]: cluster 2021-07-31T02:21:22.280963 2021-07-31T02:21:24.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:24 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 1459 : cluster [DBG] pgmap v1896: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.6 MiB/s rd, 2.2 MiB/s wr, 3 op/s 2021-07-31T02:21:24.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:24 smithi045 conmon[39718]: cluster 2021-07-31T02:21:22.280963+0000 mgr.x (mgr 2021-07-31T02:21:24.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:24 smithi045 conmon[39718]: .24542) 1459 : cluster [DBG] pgmap v1896: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.6 MiB/s rd, 2.2 MiB/s wr, 3 op/s 2021-07-31T02:21:25.125 INFO:tasks.workunit.client.0.smithi045.stdout: 959: throughput=5MB/sec pending data=0 2021-07-31T02:21:25.125 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj--46UkTWlZ_HuUbA off=1206290 len=1066378 2021-07-31T02:21:25.127 INFO:tasks.workunit.client.0.smithi045.stdout:op 4887 completed, throughput=5MB/sec 2021-07-31T02:21:25.127 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-B6CcaVlMEpmiABM off=11608903 len=1507055 2021-07-31T02:21:25.134 INFO:tasks.workunit.client.0.smithi045.stdout:op 4888 completed, throughput=5MB/sec 2021-07-31T02:21:25.134 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-mwHeCxpvtQ3NyJh off=375163 len=1723000 2021-07-31T02:21:25.150 INFO:tasks.workunit.client.0.smithi045.stdout:op 4889 completed, throughput=5MB/sec 2021-07-31T02:21:25.151 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-XnnIGNjU-GbKQRi off=10357635 len=1836326 2021-07-31T02:21:25.152 INFO:tasks.workunit.client.0.smithi045.stdout:op 4890 completed, throughput=5MB/sec 2021-07-31T02:21:26.152 INFO:tasks.workunit.client.0.smithi045.stdout: 960: throughput=5MB/sec pending data=0 2021-07-31T02:21:26.153 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-93vXm5l9B_VrSLs off=9931303 len=1171624 2021-07-31T02:21:26.159 INFO:tasks.workunit.client.0.smithi045.stdout:op 4891 completed, throughput=5MB/sec 2021-07-31T02:21:26.159 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ThwyY-7yuEwAcDu off=10806565 len=1587011 2021-07-31T02:21:26.162 INFO:tasks.workunit.client.0.smithi045.stdout:op 4892 completed, throughput=5MB/sec 2021-07-31T02:21:26.162 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-U6PX6QAf6q9xJve off=12865690 len=1092922 2021-07-31T02:21:26.163 INFO:tasks.workunit.client.0.smithi045.stdout:op 4893 completed, throughput=5MB/sec 2021-07-31T02:21:26.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:26 smithi154 conmon[34553]: cluster 2021-07-31T02:21:24.281586+0000 mgr.x (mgr.24542) 1460 : cluster [DBG] pgmap v1897: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 2.6 MiB/s rd, 3.4 MiB/s wr, 6 op/s 2021-07-31T02:21:26.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:26 smithi045 conmon[33655]: cluster 2021-07-31T02:21:24.281586 2021-07-31T02:21:26.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:26 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 1460 : cluster [DBG] pgmap v1897: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 2.6 MiB/s rd, 3.4 MiB/s wr, 6 op/s 2021-07-31T02:21:26.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:26 smithi045 conmon[39718]: cluster 2021-07-31T02:21:24.281586+0000 2021-07-31T02:21:26.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:26 smithi045 conmon[39718]: mgr.x (mgr.24542) 1460 : cluster [DBG] pgmap v1897: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 2.6 MiB/s rd, 3.4 MiB/s wr, 6 op/s 2021-07-31T02:21:27.165 INFO:tasks.workunit.client.0.smithi045.stdout: 961: throughput=5MB/sec pending data=0 2021-07-31T02:21:27.165 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-F-quC22BYrIPVfx off=109405 len=235358 2021-07-31T02:21:27.166 INFO:tasks.workunit.client.0.smithi045.stdout:op 4894 completed, throughput=5MB/sec 2021-07-31T02:21:27.166 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-1dhluxOwXaO6Uoe off=16534717 len=1431466 2021-07-31T02:21:27.168 INFO:tasks.workunit.client.0.smithi045.stdout:op 4895 completed, throughput=5MB/sec 2021-07-31T02:21:27.169 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-SuA2_lk_pE60pMp off=1571897 len=1554303 2021-07-31T02:21:27.179 INFO:tasks.workunit.client.0.smithi045.stdout:op 4896 completed, throughput=5MB/sec 2021-07-31T02:21:27.180 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-o36Kr78_Ryr_Pmj off=691921 len=1430491 2021-07-31T02:21:27.186 INFO:tasks.workunit.client.0.smithi045.stdout:op 4897 completed, throughput=5MB/sec 2021-07-31T02:21:27.187 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-idc-m6DkN8z0j2Z off=94393 len=1558509 2021-07-31T02:21:27.188 INFO:tasks.workunit.client.0.smithi045.stdout:op 4898 completed, throughput=5MB/sec 2021-07-31T02:21:27.768 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:21:28.044 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:21:28.044 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 36m ago 37m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:21:28.044 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 34m ago 36m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:21:28.044 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 34m ago 40m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:21:28.045 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 36m ago 42m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:21:28.045 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 36m ago 42m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:21:28.045 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 34m ago 40m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:21:28.045 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 36m ago 40m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:21:28.045 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 36m ago 37m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:21:28.046 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 34m ago 37m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:21:28.046 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 36m ago 40m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:21:28.046 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 36m ago 39m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:21:28.046 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 36m ago 39m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:21:28.047 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 36m ago 39m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:21:28.047 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 34m ago 38m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:21:28.047 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 34m ago 38m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:21:28.047 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 34m ago 38m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:21:28.047 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 34m ago 37m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:21:28.048 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 34m ago 37m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:21:28.188 INFO:tasks.workunit.client.0.smithi045.stdout: 962: throughput=5MB/sec pending data=0 2021-07-31T02:21:28.189 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-OW5JGhiUxYeINdR off=882709 len=1745084 2021-07-31T02:21:28.196 INFO:tasks.workunit.client.0.smithi045.stdout:op 4899 completed, throughput=5MB/sec 2021-07-31T02:21:28.196 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-IwOiML-DhV0dkkG off=5503675 len=244825 2021-07-31T02:21:28.197 INFO:tasks.workunit.client.0.smithi045.stdout:op 4900 completed, throughput=5MB/sec 2021-07-31T02:21:28.197 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-noHmvD6qT9p4_l1 off=6379103 len=540482 2021-07-31T02:21:28.200 INFO:tasks.workunit.client.0.smithi045.stdout:op 4901 completed, throughput=5MB/sec 2021-07-31T02:21:28.200 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-8bx__hrhnG0-wx5 off=4038089 len=107022 2021-07-31T02:21:28.201 INFO:tasks.workunit.client.0.smithi045.stdout:op 4902 completed, throughput=5MB/sec 2021-07-31T02:21:28.201 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-UCbOO_1A1Y3Q4Qn off=876719 len=1806952 2021-07-31T02:21:28.212 INFO:tasks.workunit.client.0.smithi045.stdout:op 4903 completed, throughput=5MB/sec 2021-07-31T02:21:28.381 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:21:28.381 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:21:28.382 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:21:28.382 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:21:28.383 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:21:28.383 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:21:28.383 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:21:28.383 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:21:28.384 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:21:28.384 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:21:28.384 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:21:28.384 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:21:28.384 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:21:28.385 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:21:28.385 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:21:28.385 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:21:28.385 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:21:28.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:28 smithi154 conmon[34553]: cluster 2021-07-31T02:21:26.282024+0000 mgr.x (mgr.24542) 1461 : cluster [DBG] pgmap v1898: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2021-07-31T02:21:28.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:28 smithi154 conmon[34553]: audit 2021-07-31T02:21:28.051212+0000 mon.b (mon.2) 238 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:21:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:28 smithi154 conmon[34553]: audit 2021-07-31T02:21:28.051609+0000 mon.a (mon.0) 1899 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:21:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:28 smithi154 conmon[34553]: audit 2021-07-31T02:21:28.073325+0000 mon.b (mon.2) 239 2021-07-31T02:21:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:28 smithi154 conmon[34553]: : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:21:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:28 smithi154 conmon[34553]: audit 2021-07-31 2021-07-31T02:21:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:28 smithi154 conmon[34553]: T02:21:28.073748+0000 mon.a (mon. 2021-07-31T02:21:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:28 smithi154 conmon[34553]: 0) 1900 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:21:28.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:28 smithi045 conmon[33655]: cluster 2021-07-31T02:21:26.282024+0000 mgr.x (mgr.24542) 1461 : 2021-07-31T02:21:28.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:28 smithi045 conmon[33655]: cluster [DBG] pgmap v1898: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2021-07-31T02:21:28.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:28 smithi045 conmon[33655]: audit 2021-07-31T02:21:28.051212+0000 mon.b (mon.2) 238 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:21:28.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:28 smithi045 conmon[33655]: audit 2021-07-31T02:21:28.051609+0000 2021-07-31T02:21:28.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:28 smithi045 conmon[33655]: mon.a (mon.0) 1899 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:21:28.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:28 smithi045 conmon[33655]: audit 2021-07-31T02:21:28.073325+0000 2021-07-31T02:21:28.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:28 smithi045 conmon[33655]: mon.b (mon.2) 239 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:21:28.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:28 smithi045 conmon[33655]: audit 2021-07-31T02 2021-07-31T02:21:28.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:28 smithi045 conmon[33655]: :21:28.073748+0000 mon.a (mon.0) 1900 : audit [INF] 2021-07-31T02:21:28.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:28 smithi045 conmon[33655]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:21:28.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:28 smithi045 conmon[39718]: cluster 2021-07-31T02 2021-07-31T02:21:28.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:28 smithi045 conmon[39718]: :21:26.282024+0000 mgr.x (mgr.24542) 1461 : cluster [DBG] pgmap v1898: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2021-07-31T02:21:28.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:28 smithi045 conmon[39718]: audit 2021 2021-07-31T02:21:28.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:28 smithi045 conmon[39718]: -07-31T02:21: 2021-07-31T02:21:28.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:28 smithi045 conmon[39718]: 28.051212+0000 mon.b ( 2021-07-31T02:21:28.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:28 smithi045 conmon[39718]: mon.2) 238 : audit 2021-07-31T02:21:28.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:28 smithi045 conmon[39718]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:21:28.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:28 smithi045 conmon[39718]: audit 2021 2021-07-31T02:21:28.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:28 smithi045 conmon[39718]: -07-31T02:21:28.051609+0000 2021-07-31T02:21:28.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:28 smithi045 conmon[39718]: mon.a (mon.0) 1899 : audit 2021-07-31T02:21:28.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:28 smithi045 conmon[39718]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:21:28.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:28 smithi045 conmon[39718]: audit 2021- 2021-07-31T02:21:28.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:28 smithi045 conmon[39718]: 07-31T02:21:28.073325+0000 mon.b (mon 2021-07-31T02:21:28.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:28 smithi045 conmon[39718]: .2) 239 : audit [INF] 2021-07-31T02:21:28.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:28 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:21:28.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:28 smithi045 conmon[39718]: audit 2021- 2021-07-31T02:21:28.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:28 smithi045 conmon[39718]: 07-31T02:21:28.073748+0000 2021-07-31T02:21:28.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:28 smithi045 conmon[39718]: mon.a (mon.0) 1900 : audit 2021-07-31T02:21:28.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:28 smithi045 conmon[39718]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:21:29.211 INFO:tasks.workunit.client.0.smithi045.stdout: 963: throughput=5MB/sec pending data=0 2021-07-31T02:21:29.212 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-HZnR6CkMR3RcKY8 off=767988 len=1258060 2021-07-31T02:21:29.214 INFO:tasks.workunit.client.0.smithi045.stdout:op 4904 completed, throughput=5MB/sec 2021-07-31T02:21:29.215 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-LsBFTAy39PYA_8d off=1105539 len=1094984 2021-07-31T02:21:29.221 INFO:tasks.workunit.client.0.smithi045.stdout:op 4905 completed, throughput=5MB/sec 2021-07-31T02:21:29.221 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-fjG0Ms6s95mSLcK off=507303 len=1771768 2021-07-31T02:21:29.224 INFO:tasks.workunit.client.0.smithi045.stdout:op 4906 completed, throughput=5MB/sec 2021-07-31T02:21:29.224 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-hjrRXapuRJkCWp_ off=6460413 len=1698721 2021-07-31T02:21:29.226 INFO:tasks.workunit.client.0.smithi045.stdout:op 4907 completed, throughput=5MB/sec 2021-07-31T02:21:29.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:29 smithi154 conmon[34553]: audit 2021-07-31T02:21:27.756956+0000 mgr.x (mgr.24542) 1462 : audit [DBG] 2021-07-31T02:21:29.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:29 smithi154 conmon[34553]: from='client.16293 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:21:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:29 smithi154 conmon[34553]: audit 2021-07-31T02:21:28.036530+0000 mgr.x (mgr.24542) 1463 : audit [DBG] from='client.16299 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:21:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:29 smithi154 conmon[34553]: audit 2021-07-31T02:21:28.381647+0000 mon.c (mon.1) 91 : audit [DBG] from='client.? 172.21.15.45:0/3784822862' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:21:29.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:29 smithi045 conmon[39718]: audit 2021-07-31T02:21: 2021-07-31T02:21:29.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:29 smithi045 conmon[39718]: 27.756956+0000 mgr.x (mgr.24542) 1462 : audit [DBG] from='client.16293 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:21:29.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:29 smithi045 conmon[39718]: audit 2021-07-31T02: 2021-07-31T02:21:29.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:29 smithi045 conmon[39718]: 21:28.036530+0000 mgr.x (mgr.24542) 1463 : audit 2021-07-31T02:21:29.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:29 smithi045 conmon[39718]: [DBG] from='client.16299 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:21:29.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:29 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T02:21:29.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:29 smithi045 conmon[39718]: T02:21:28.381647+0000 mon.c (mon.1 2021-07-31T02:21:29.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:29 smithi045 conmon[39718]: ) 91 : audit [DBG] from='client.? 172.21.15.45:0/3784822862' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:21:29.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:29 smithi045 conmon[33655]: audit 2021-07-31T02:21:27.756956+0000 mgr.x (mgr.24542) 1462 : 2021-07-31T02:21:29.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:29 smithi045 conmon[33655]: audit [DBG] from='client.16293 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:21:29.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:29 smithi045 conmon[33655]: audit 2021-07-31T02:21:28.036530+0000 mgr.x (mgr.24542) 1463 : audit [DBG] from='client.16299 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:21:29.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:29 smithi045 conmon[33655]: audit 2021-07-31T02:21:29.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:29 smithi045 conmon[33655]: 2021-07-31T02:21:28.381647+0000 mon.c (mon.1) 91 : audit [DBG] from='client.? 172.21.15.45:0/3784822862' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:21:30.227 INFO:tasks.workunit.client.0.smithi045.stdout: 964: throughput=5MB/sec pending data=0 2021-07-31T02:21:30.227 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-SqLaImsQmbKuyfK off=14863406 len=717859 2021-07-31T02:21:30.232 INFO:tasks.workunit.client.0.smithi045.stdout:op 4908 completed, throughput=5MB/sec 2021-07-31T02:21:30.233 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-BCHzeqWCgbNe5Fo off=8246834 len=634236 2021-07-31T02:21:30.233 INFO:tasks.workunit.client.0.smithi045.stdout:op 4909 completed, throughput=5MB/sec 2021-07-31T02:21:30.234 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-K1CIooXTasNjEdO off=5260573 len=1835228 2021-07-31T02:21:30.245 INFO:tasks.workunit.client.0.smithi045.stdout:op 4910 completed, throughput=5MB/sec 2021-07-31T02:21:30.245 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-mZ1zOtqsQ1T5tto off=13022360 len=1030805 2021-07-31T02:21:30.247 INFO:tasks.workunit.client.0.smithi045.stdout:op 4911 completed, throughput=5MB/sec 2021-07-31T02:21:30.247 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj--wFBE17hplnBbNQ off=19644761 len=1695251 2021-07-31T02:21:30.255 INFO:tasks.workunit.client.0.smithi045.stdout:op 4912 completed, throughput=5MB/sec 2021-07-31T02:21:30.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:30 smithi154 conmon[34553]: cluster 2021-07-31T02:21:28.282567+0000 mgr.x (mgr.24542) 1464 : cluster [DBG] pgmap v1899: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2021-07-31T02:21:30.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:30 smithi045 conmon[39718]: cluster 2021-07-31T02:21:28.282567+0000 mgr.x ( 2021-07-31T02:21:30.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:30 smithi045 conmon[39718]: mgr.24542) 1464 : cluster [DBG] pgmap v1899: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2021-07-31T02:21:30.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:30 smithi045 conmon[33655]: cluster 2021-07-31T02:21:28.282567+0000 mgr.x (mgr.24542) 1464 : cluster 2021-07-31T02:21:30.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:30 smithi045 conmon[33655]: [DBG] pgmap v1899: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2021-07-31T02:21:31.256 INFO:tasks.workunit.client.0.smithi045.stdout: 965: throughput=5MB/sec pending data=0 2021-07-31T02:21:31.256 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-7abcL89bryy_Wsx off=1386193 len=62109 2021-07-31T02:21:31.258 INFO:tasks.workunit.client.0.smithi045.stdout:op 4913 completed, throughput=5MB/sec 2021-07-31T02:21:31.259 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-pP-OeMIveuCJJ67 off=1225165 len=1273495 2021-07-31T02:21:31.267 INFO:tasks.workunit.client.0.smithi045.stdout:op 4914 completed, throughput=5MB/sec 2021-07-31T02:21:31.267 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-8QbkfW6hT9cGSA4 off=7877437 len=1091877 2021-07-31T02:21:31.275 INFO:tasks.workunit.client.0.smithi045.stdout:op 4915 completed, throughput=5MB/sec 2021-07-31T02:21:31.276 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-m88R3IRlGX8XuMs off=3331567 len=2074382 2021-07-31T02:21:31.282 INFO:tasks.workunit.client.0.smithi045.stdout:op 4916 completed, throughput=5MB/sec 2021-07-31T02:21:31.753 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: debug 2021-07-31T02:21:31.473+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:21:31.753 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:21:31.753 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:21:31.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: yield 2021-07-31T02:21:31.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:21:31.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:21:31.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:21:31.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:21:31.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:21:31.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:21:31.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:21:31.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:21:31.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:21:31.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: task.result() 2021-07-31T02:21:31.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:21:31.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: debug 2021-07-31T02:21:31.485+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:21:31.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:21:31.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:21:31.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: yield 2021-07-31T02:21:31.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:21:31.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:21:31.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:21:31.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:21:31.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:21:31.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:21:31.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:21:31.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:21:31.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:21:31.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: task.result() 2021-07-31T02:21:31.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:21:31.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: debug 2021-07-31T02:21:31.510+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:21:31.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:21:31.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:21:31.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: yield 2021-07-31T02:21:31.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:21:31.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:21:31.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:21:31.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:21:31.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:21:31.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:21:31.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:21:31.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:21:31.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:21:31.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: task.result() 2021-07-31T02:21:31.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:21:31.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: debug 2021-07-31T02:21:31.511+0000 7fcb9816a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:21:31.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:21:31.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:21:31.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: yield 2021-07-31T02:21:31.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:21:31.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:21:31.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:21:31.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:21:31.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:21:31.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:21:31.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:21:31.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:21:31.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:21:31.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: task.result() 2021-07-31T02:21:31.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:21:31.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: 2021-07-31T02:21:31.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:21:31.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: 2021-07-31T02:21:31.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:21:31.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:21:31.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:21:31.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:21:31.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:21:31.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:21:31.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:21:31.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:21:31.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:21:31.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:21:31.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:21:31.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:21:31.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:21:31.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:21:31.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:21:31.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:21:31.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:21:31.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:21:31.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: raise 2021-07-31T02:21:31.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:21:31.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:21:31.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:21:31.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:21:31.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:21:31.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: debug 2021-07-31T02:21:31.520+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:21:31.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:21:31.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:21:31.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: yield 2021-07-31T02:21:31.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:21:31.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:21:31.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:21:31.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:21:31.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:21:31.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:21:31.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:21:31.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:21:31.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:21:31.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: task.result() 2021-07-31T02:21:31.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:21:31.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: debug 2021-07-31T02:21:31.520+0000 7fcb97969700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:21:31.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:21:31.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:21:31.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: yield 2021-07-31T02:21:31.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:21:31.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:21:31.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:21:31.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:21:31.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:21:31.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:21:31.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:21:31.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:21:31.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:21:31.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: task.result() 2021-07-31T02:21:31.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:21:31.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: 2021-07-31T02:21:31.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:21:31.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: 2021-07-31T02:21:31.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:21:31.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:21:31.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:21:31.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:21:31.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:21:31.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:21:31.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:21:31.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:21:31.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:21:31.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:21:31.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:21:31.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:21:31.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:21:31.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:21:31.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:21:31.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:21:31.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:21:31.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:21:31.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: raise 2021-07-31T02:21:31.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:21:31.786 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:21:31.786 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:21:31.786 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:21:31.786 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:21:31 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:21:32.283 INFO:tasks.workunit.client.0.smithi045.stdout: 966: throughput=5MB/sec pending data=0 2021-07-31T02:21:32.283 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-6qxT2mOjfEIw_PT off=43159 len=1236052 2021-07-31T02:21:32.284 INFO:tasks.workunit.client.0.smithi045.stdout:op 4917 completed, throughput=5MB/sec 2021-07-31T02:21:32.284 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-hEOAXqmA7DeX64h off=1649703 len=468218 2021-07-31T02:21:32.288 INFO:tasks.workunit.client.0.smithi045.stdout:op 4918 completed, throughput=5MB/sec 2021-07-31T02:21:32.288 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-_srP4fRYJ6q2JF1 off=15596518 len=748689 2021-07-31T02:21:32.290 INFO:tasks.workunit.client.0.smithi045.stdout:op 4919 completed, throughput=5MB/sec 2021-07-31T02:21:32.290 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-7RdpDNi6HcHrayU off=172735 len=1634749 2021-07-31T02:21:32.297 INFO:tasks.workunit.client.0.smithi045.stdout:op 4920 completed, throughput=5MB/sec 2021-07-31T02:21:32.298 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-wKe46Pw8EDr_E_G off=4997008 len=236651 2021-07-31T02:21:32.298 INFO:tasks.workunit.client.0.smithi045.stdout:op 4921 completed, throughput=5MB/sec 2021-07-31T02:21:32.299 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-hbduplg0YAi3KNC off=7741600 len=251188 2021-07-31T02:21:32.301 INFO:tasks.workunit.client.0.smithi045.stdout:op 4922 completed, throughput=5MB/sec 2021-07-31T02:21:32.301 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-AnRZv_jr_lwc0jt off=7093275 len=718195 2021-07-31T02:21:32.305 INFO:tasks.workunit.client.0.smithi045.stdout:op 4923 completed, throughput=5MB/sec 2021-07-31T02:21:32.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:32 smithi154 conmon[34553]: cluster 2021-07-31T02:21:30.283260+0000 mgr.x (mgr.24542) 1465 : cluster [DBG] pgmap v1900: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 2.8 MiB/s rd, 3.3 MiB/s wr, 6 op/s 2021-07-31T02:21:32.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:32 smithi154 conmon[34553]: 2021-07-31T02:21:32.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:32 smithi154 conmon[34553]: audit 2021-07-31T02:21:31.428347+0000 mon.b (mon.2) 240 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:21:32.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:32 smithi154 conmon[34553]: audit 2021-07-31T02:21:31.429841+0000 mon.b (mon.2) 241 : 2021-07-31T02:21:32.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:32 smithi154 conmon[34553]: audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:21:32.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:32 smithi154 conmon[34553]: audit 2021-07-31T02:21:31.430823+0000 2021-07-31T02:21:32.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:32 smithi154 conmon[34553]: mon.b (mon.2) 242 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:21:32.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:32 smithi045 conmon[39718]: cluster 2021-07-31T02:21:30.283260+0000 mgr.x (mgr 2021-07-31T02:21:32.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:32 smithi045 conmon[39718]: .24542) 1465 : cluster [DBG] pgmap v1900: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 2.8 MiB/s rd, 3.3 MiB/s wr, 6 op/s 2021-07-31T02:21:32.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:32 smithi045 conmon[39718]: audit 2021-07-31T02:21:31.428347+0000 2021-07-31T02:21:32.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:32 smithi045 conmon[39718]: mon.b (mon.2) 240 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:21:32.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:32 smithi045 conmon[39718]: audit 2021- 2021-07-31T02:21:32.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:32 smithi045 conmon[39718]: 07-31T02:21:31 2021-07-31T02:21:32.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:32 smithi045 conmon[39718]: .429841+0000 mon.b (mon.2) 241 : 2021-07-31T02:21:32.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:32 smithi045 conmon[39718]: audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:21:32.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:32 smithi045 conmon[39718]: audit 2021-07-31T02:21:32.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:32 smithi045 conmon[39718]: 2021-07-31T02 2021-07-31T02:21:32.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:32 smithi045 conmon[39718]: :21:31.430823+0000 mon.b 2021-07-31T02:21:32.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:32 smithi045 conmon[39718]: (mon.2) 242 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:21:32.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:32 smithi045 conmon[33655]: cluster 2021-07-31T02:21:30.283260+0000 mgr.x (mgr.24542) 1465 : cluster [DBG] pgmap v1900: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 2.8 MiB/s rd, 3.3 MiB/s wr, 6 op/s 2021-07-31T02:21:32.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:32 smithi045 conmon[33655]: audit 2021-07-31T02:21:31.428347+0000 mon.b (mon.2) 240 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:21:32.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:32 smithi045 conmon[33655]: audit 2021-07-31T02:21:31.429841+0000 2021-07-31T02:21:32.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:32 smithi045 conmon[33655]: mon.b (mon.2) 241 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:21:32.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:32 smithi045 conmon[33655]: audit 2021-07 2021-07-31T02:21:32.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:32 smithi045 conmon[33655]: -31T02:21:31.430823+0000 mon.b (mon.2) 242 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:21:33.246 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:21:31. 2021-07-31T02:21:33.247 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: 431615+0000 mgr.x (mgr.24542) 1466 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:21:33.247 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: cephadm 2021- 2021-07-31T02:21:33.247 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: 07-31T02:21:31.474690+0000 mgr.x (mgr.24542) 1467 : cephadm [ERR] 2021-07-31T02:21:33.247 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:21:33.247 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:21:33.248 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: yield 2021-07-31T02:21:33.248 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:21:33.248 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:21:33.248 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:21:33.249 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:21:33.249 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:21:33.249 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:21:33.249 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:21:33.249 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:21:33.250 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:21:33.250 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:21:33.250 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:21:33.250 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: 2021-07-31T02:21:33.250 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:21:31.475361+0000 mgr.x (mgr.24542) 1468 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:21:33.251 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: cephadm 2021-07-31T 2021-07-31T02:21:33.251 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: 02:21:31.486419+0000 mgr.x (mgr.24542) 2021-07-31T02:21:33.251 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: 1469 : cephadm [ERR] 2021-07-31T02:21:33.251 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:21:33.252 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:21:33.252 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: yield 2021-07-31T02:21:33.252 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:21:33.252 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:21:33.253 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:21:33.253 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:21:33.253 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:21:33.253 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:21:33.254 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:21:33.254 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:21:33.254 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:21:33.254 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:21:33.255 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:21:33.255 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:21:31.487469+0000 mgr.x (mgr.24542) 1470 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:21:33.255 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: cephadm 2021-07 2021-07-31T02:21:33.255 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: -31T02:21:31.511867+0000 mgr.x (mgr. 2021-07-31T02:21:33.256 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: 24542) 1471 : cephadm [ERR] 2021-07-31T02:21:33.256 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:21:33.256 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:21:33.257 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: yield 2021-07-31T02:21:33.257 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:21:33.257 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:21:33.257 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:21:33.258 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:21:33.258 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:21:33.258 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:21:33.258 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:21:33.258 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:21:33.259 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:21:33.259 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:21:33.259 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:21:33.259 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:21:31.512747+0000 mgr.x (mgr.24542) 1472 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:21:33.260 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:21:33.260 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:21:33.260 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: yield 2021-07-31T02:21:33.260 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:21:33.260 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:21:33.261 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:21:33.261 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:21:33.261 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:21:33.261 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:21:33.261 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:21:33.262 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:21:33.263 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:21:31.431615+0000 mgr.x (mgr 2021-07-31T02:21:33.263 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: .24542) 1466 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:21:33.263 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:21:31.474690+0000 mgr.x (mgr 2021-07-31T02:21:33.264 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: .24542) 1467 : cephadm [ERR] 2021-07-31T02:21:33.264 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:21:33.264 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:21:33.264 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: yield 2021-07-31T02:21:33.264 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:21:33.265 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:21:33.265 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:21:33.265 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:21:33.266 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:21:33.266 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:21:33.266 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:21:33.266 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:21:33.267 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:21:33.267 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:21:33.267 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:21:33.267 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:21:31.475361+0000 mgr.x (mgr.24542) 1468 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:21:33.268 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: cephadm 2021 2021-07-31T02:21:33.268 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: -07-31T02:21:31 2021-07-31T02:21:33.268 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: .486419+0000 mgr.x (mgr. 2021-07-31T02:21:33.268 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: 24542) 1469 : cephadm [ERR] 2021-07-31T02:21:33.269 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: 2021-07-31T02:21:33.269 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:21:33.269 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:21:33.270 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: yield 2021-07-31T02:21:33.270 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:21:33.270 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:21:33.270 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:21:33.271 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:21:33.271 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:21:33.271 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:21:33.271 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:21:33.272 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:21:33.272 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:21:33.272 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:21:33.272 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:21:33.273 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:21:31.487469+0000 mgr.x (mgr.24542) 1470 2021-07-31T02:21:33.273 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:21:33.273 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: cephadm 2021 2021-07-31T02:21:33.273 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: -07-31T02:21:31 2021-07-31T02:21:33.274 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: .511867+0000 mgr.x (mgr. 2021-07-31T02:21:33.274 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: 24542) 1471 : cephadm [ERR] 2021-07-31T02:21:33.274 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:21:33.274 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:21:33.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: yield 2021-07-31T02:21:33.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:21:33.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:21:33.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:21:33.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:21:33.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:21:33.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:21:33.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:21:33.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:21:33.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:21:33.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:21:33.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:21:33.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: 2021-07-31T02:21:33.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: cephadm 2021-07-31T 2021-07-31T02:21:33.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: 02:21:31.512747+0000 mgr.x ( 2021-07-31T02:21:33.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: mgr.24542) 1472 : cephadm 2021-07-31T02:21:33.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:21:33.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:21:33.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:21:33.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: yield 2021-07-31T02:21:33.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:21:33.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:21:33.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:21:33.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:21:33.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:21:33.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:21:33.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:21:33.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:21:33.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:21:33.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:21:33.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:21:33.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: 2021-07-31T02:21:33.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:21:33.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: 2021-07-31T02:21:33.283 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:21:33.283 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:21:33.283 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:21:33.283 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:21:33.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:21:33.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:21:33.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:21:33.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:21:33.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:21:33.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:21:33.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:21:33.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:21:33.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:21:33.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:21:33.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:21:33.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:21:33.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:21:33.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:21:33.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: raise 2021-07-31T02:21:33.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:21:33.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:21:33.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:21:33.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:21:33.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:21:33.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:21:31.521059+0000 mgr.x (mgr.24542) 1473 : cephadm [ERR] 2021-07-31T02:21:33.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:21:33.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:21:33.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: yield 2021-07-31T02:21:33.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:21:33.290 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:21:33.290 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:21:33.290 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:21:33.290 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:21:33.291 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:21:33.291 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:21:33.291 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:21:33.291 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:21:33.292 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:21:33.292 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:21:33.292 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:21:31.521460+0000 mgr.x (mgr.24542) 1474 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:21:33.292 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:21:33.292 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:21:33.293 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: yield 2021-07-31T02:21:33.293 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:21:33.293 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:21:33.293 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:21:33.294 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:21:33.294 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:21:33.294 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:21:33.294 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:21:33.295 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:21:33.295 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:21:33.295 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:21:33.295 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:21:33.295 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: 2021-07-31T02:21:33.296 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:21:33.296 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: 2021-07-31T02:21:33.297 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:21:33.297 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:21:33.297 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:21:33.297 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:21:33.298 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:21:33.298 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:21:33.298 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:21:33.298 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:21:33.299 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:21:33.299 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:21:33.299 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:21:33.299 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:21:33.300 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:21:33.300 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:21:33.300 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:21:33.300 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:21:33.300 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:21:33.301 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:21:33.301 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: raise 2021-07-31T02:21:33.301 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:21:33.301 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:21:33.301 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:21:33.302 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:21:33.302 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:21:33.302 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[33655]: 2021-07-31T02:21:33.306 INFO:tasks.workunit.client.0.smithi045.stdout: 967: throughput=5MB/sec pending data=0 2021-07-31T02:21:33.306 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-JzMHroS5tUclaEU off=5784892 len=1704612 2021-07-31T02:21:33.309 INFO:tasks.workunit.client.0.smithi045.stdout:op 4924 completed, throughput=5MB/sec 2021-07-31T02:21:33.309 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Vdgfwt0GsQtXOaw off=10959756 len=470138 2021-07-31T02:21:33.313 INFO:tasks.workunit.client.0.smithi045.stdout:op 4925 completed, throughput=5MB/sec 2021-07-31T02:21:33.313 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-KHBegEendxMIXmh off=866550 len=1394931 2021-07-31T02:21:33.316 INFO:tasks.workunit.client.0.smithi045.stdout:op 4926 completed, throughput=5MB/sec 2021-07-31T02:21:33.316 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-k577Ih0VQ9CYfAy off=6822739 len=1352422 2021-07-31T02:21:33.322 INFO:tasks.workunit.client.0.smithi045.stdout:op 4927 completed, throughput=5MB/sec 2021-07-31T02:21:33.323 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-6tUAkDuKrFubsCf off=4435628 len=930183 2021-07-31T02:21:33.324 INFO:tasks.workunit.client.0.smithi045.stdout:op 4928 completed, throughput=5MB/sec 2021-07-31T02:21:33.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:21:31.431615+0000 mgr.x (mgr.24542) 1466 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:21:33.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:21:31.474690+0000 mgr.x (mgr.24542) 1467 : cephadm [ERR] 2021-07-31T02:21:33.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:21:33.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:21:33.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: yield 2021-07-31T02:21:33.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:21:33.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:21:33.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:21:33.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:21:33.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:21:33.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:21:33.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:21:33.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:21:33.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:21:33.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:21:33.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:21:33.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:21:31.475361+0000 mgr.x (mgr.24542) 1468 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:21:33.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:21:31.486419+0000 mgr.x (mgr.24542) 1469 : cephadm [ERR] 2021-07-31T02:21:33.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:21:33.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:21:33.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: yield 2021-07-31T02:21:33.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:21:33.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:21:33.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:21:33.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:21:33.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:21:33.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:21:33.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:21:33.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:21:33.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:21:33.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:21:33.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:21:33.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:21:31.487469+0000 mgr.x (mgr.24542) 1470 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:21:33.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:21:31.511867+0000 mgr.x (mgr.24542) 1471 : cephadm [ERR] 2021-07-31T02:21:33.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:21:33.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:21:33.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: yield 2021-07-31T02:21:33.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:21:33.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:21:33.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:21:33.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:21:33.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:21:33.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:21:33.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:21:33.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:21:33.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:21:33.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:21:33.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:21:33.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:21:31.512747+0000 mgr.x ( 2021-07-31T02:21:33.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: mgr.24542) 1472 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:21:33.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:21:33.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:21:33.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: yield 2021-07-31T02:21:33.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:21:33.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:21:33.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:21:33.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:21:33.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:21:33.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:21:33.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:21:33.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:21:33.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:21:33.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:21:33.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:21:33.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: 2021-07-31T02:21:33.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:21:33.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: 2021-07-31T02:21:33.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:21:33.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:21:33.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:21:33.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:21:33.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:21:33.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:21:33.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:21:33.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:21:33.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:21:33.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:21:33.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:21:33.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:21:33.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:21:33.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:21:33.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:21:33.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:21:33.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:21:33.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:21:33.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: raise 2021-07-31T02:21:33.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:21:33.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:21:33.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:21:33.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:21:33.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:21:33.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:21:31.521059+0000 mgr.x (mgr.24542) 1473 : cephadm [ERR] 2021-07-31T02:21:33.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:21:33.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:21:33.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: yield 2021-07-31T02:21:33.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:21:33.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:21:33.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:21:33.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:21:33.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:21:33.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:21:33.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:21:33.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:21:33.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:21:33.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:21:33.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:21:33.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: cephadm 2021 2021-07-31T02:21:33.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: -07-31T02:21:31.521460+0000 mgr.x (mgr.24542) 1474 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:21:33.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:21:33.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:21:33.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: yield 2021-07-31T02:21:33.541 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:21:33.541 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:21:33.541 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:21:33.541 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:21:33.541 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:21:33.542 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:21:33.542 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:21:33.542 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:21:33.542 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:21:33.543 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:21:33.544 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:21:33.544 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: 2021-07-31T02:21:33.544 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:21:33.544 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: 2021-07-31T02:21:33.545 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:21:33.545 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:21:33.545 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:21:33.545 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:21:33.546 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:21:33.546 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:21:33.546 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:21:33.546 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:21:33.547 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:21:33.547 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:21:33.547 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:21:33.548 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:21:33.548 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:21:33.548 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:21:33.548 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:21:33.549 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:21:33.549 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:21:33.549 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:21:33.549 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: raise 2021-07-31T02:21:33.550 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:21:33.550 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:21:33.550 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:21:33.550 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:21:33.551 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:33 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:21:33.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:21:33.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:21:33.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:21:33.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: 2021-07-31T02:21:33.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:21:33.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: 2021-07-31T02:21:33.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:21:33.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:21:33.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:21:33.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:21:33.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:21:33.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:21:33.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:21:33.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:21:33.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:21:33.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:21:33.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:21:33.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:21:33.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:21:33.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:21:33.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:21:33.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:21:33.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:21:33.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:21:33.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: raise 2021-07-31T02:21:33.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:21:33.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:21:33.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:21:33.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:21:33.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:21:33.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:21:31.521059+0000 mgr.x (mgr.24542) 1473 : cephadm [ERR] 2021-07-31T02:21:33.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:21:33.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:21:33.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: yield 2021-07-31T02:21:33.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:21:33.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:21:33.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:21:33.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:21:33.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:21:33.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:21:33.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:21:33.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:21:33.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:21:33.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:21:33.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:21:33.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:21:31.521460+0000 mgr.x (mgr.24542) 1474 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:21:33.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:21:33.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:21:33.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: yield 2021-07-31T02:21:33.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:21:33.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:21:33.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:21:33.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:21:33.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:21:33.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:21:33.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:21:33.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:21:33.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:21:33.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:21:33.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:21:33.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: 2021-07-31T02:21:33.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:21:33.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: 2021-07-31T02:21:33.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:21:33.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:21:33.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:21:33.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:21:33.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:21:33.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:21:33.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:21:33.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:21:33.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:21:33.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:21:33.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:21:33.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:21:33.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:21:33.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:21:33.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:21:33.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:21:33.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:21:33.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:21:33.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: raise 2021-07-31T02:21:33.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:21:33.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:21:33.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:21:33.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:21:33.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:21:34.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:21:33.568Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:21:34.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:21:33.568Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:21:34.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:21:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:21:33.568Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:21:34.324 INFO:tasks.workunit.client.0.smithi045.stdout: 968: throughput=5MB/sec pending data=0 2021-07-31T02:21:34.325 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-MceFvQhpoovL4R- off=13337712 len=1688387 2021-07-31T02:21:34.326 INFO:tasks.workunit.client.0.smithi045.stdout:op 4929 completed, throughput=5MB/sec 2021-07-31T02:21:34.327 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-QUEOIL_C2mW7c2k off=3964478 len=74605 2021-07-31T02:21:34.327 INFO:tasks.workunit.client.0.smithi045.stdout:op 4930 completed, throughput=5MB/sec 2021-07-31T02:21:34.327 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Huf_yXcv8PINBh6 off=9442770 len=701703 2021-07-31T02:21:34.329 INFO:tasks.workunit.client.0.smithi045.stdout:op 4931 completed, throughput=5MB/sec 2021-07-31T02:21:34.329 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-owJhpxcHSR1eyyc off=6206837 len=2069064 2021-07-31T02:21:34.332 INFO:tasks.workunit.client.0.smithi045.stdout:op 4932 completed, throughput=5MB/sec 2021-07-31T02:21:34.332 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Q6VwEYV7-3CXSVe off=3977520 len=544967 2021-07-31T02:21:34.333 INFO:tasks.workunit.client.0.smithi045.stdout:op 4933 completed, throughput=5MB/sec 2021-07-31T02:21:34.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:34 smithi154 conmon[34553]: cluster 2021-07-31T02:21:32.283750+0000 mgr.x (mgr.24542) 1475 : cluster [DBG] pgmap v1901: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:21:34.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:34 smithi045 conmon[39718]: cluster 2021-07-31T02:21:32.283750+0000 mgr.x ( 2021-07-31T02:21:34.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:34 smithi045 conmon[39718]: mgr.24542) 1475 : cluster [DBG] pgmap v1901: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:21:34.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:34 smithi045 conmon[33655]: cluster 2021-07-31T02:21:32.283750+0000 mgr.x (mgr.24542) 1475 2021-07-31T02:21:34.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:34 smithi045 conmon[33655]: : cluster [DBG] pgmap v1901: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:21:35.333 INFO:tasks.workunit.client.0.smithi045.stdout: 969: throughput=5MB/sec pending data=0 2021-07-31T02:21:35.334 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-oXKHTAHxh5t1-7T off=4554944 len=209339 2021-07-31T02:21:35.337 INFO:tasks.workunit.client.0.smithi045.stdout:op 4934 completed, throughput=5MB/sec 2021-07-31T02:21:35.337 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-oe5l2QwrOGjnZ-0 off=7761677 len=1053718 2021-07-31T02:21:35.339 INFO:tasks.workunit.client.0.smithi045.stdout:op 4935 completed, throughput=5MB/sec 2021-07-31T02:21:35.339 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-9RW43DdazU2XH_9 off=9199356 len=1285937 2021-07-31T02:21:35.346 INFO:tasks.workunit.client.0.smithi045.stdout:op 4936 completed, throughput=5MB/sec 2021-07-31T02:21:35.346 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-obKRXdd8YaQ2kXT off=3569322 len=518917 2021-07-31T02:21:35.346 INFO:tasks.workunit.client.0.smithi045.stdout:op 4937 completed, throughput=5MB/sec 2021-07-31T02:21:35.347 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Q-iIQN79B4EAk4K off=9282676 len=1106397 2021-07-31T02:21:35.351 INFO:tasks.workunit.client.0.smithi045.stdout:op 4938 completed, throughput=5MB/sec 2021-07-31T02:21:35.351 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-SrBumC4zogdfjgg off=8092806 len=100404 2021-07-31T02:21:35.352 INFO:tasks.workunit.client.0.smithi045.stdout:op 4939 completed, throughput=5MB/sec 2021-07-31T02:21:35.352 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-lrl78fQA1a4NY3s off=6297078 len=866768 2021-07-31T02:21:35.357 INFO:tasks.workunit.client.0.smithi045.stdout:op 4940 completed, throughput=5MB/sec 2021-07-31T02:21:36.358 INFO:tasks.workunit.client.0.smithi045.stdout: 970: throughput=5MB/sec pending data=0 2021-07-31T02:21:36.358 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-GCKHDXrK8ZSv-du off=18749307 len=2051621 2021-07-31T02:21:36.370 INFO:tasks.workunit.client.0.smithi045.stdout:op 4941 completed, throughput=5MB/sec 2021-07-31T02:21:36.371 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-n_4pW42hdaqA55P off=3163538 len=950536 2021-07-31T02:21:36.378 INFO:tasks.workunit.client.0.smithi045.stdout:op 4942 completed, throughput=5MB/sec 2021-07-31T02:21:36.378 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-qSjuesWgzXkjxZV off=7302922 len=1736618 2021-07-31T02:21:36.385 INFO:tasks.workunit.client.0.smithi045.stdout:op 4943 completed, throughput=5MB/sec 2021-07-31T02:21:36.385 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-OocAU1bHriKzI03 off=1312242 len=575093 2021-07-31T02:21:36.388 INFO:tasks.workunit.client.0.smithi045.stdout:op 4944 completed, throughput=5MB/sec 2021-07-31T02:21:36.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:36 smithi154 conmon[34553]: cluster 2021-07-31T02:21:34.284116+0000 mgr.x (mgr.24542) 1476 : cluster [DBG] pgmap v1902: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 2.9 MiB/s rd, 3.6 MiB/s wr, 6 op/s 2021-07-31T02:21:36.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:36 smithi045 conmon[33655]: cluster 2021-07-31T02:21:34.284116 2021-07-31T02:21:36.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:36 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 1476 : cluster [DBG] pgmap v1902: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 2.9 MiB/s rd, 3.6 MiB/s wr, 6 op/s 2021-07-31T02:21:36.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:36 smithi045 conmon[39718]: cluster 2021-07-31T02:21:34.284116+0000 mgr.x ( 2021-07-31T02:21:36.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:36 smithi045 conmon[39718]: mgr.24542) 1476 : cluster [DBG] pgmap v1902: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 2.9 MiB/s rd, 3.6 MiB/s wr, 6 op/s 2021-07-31T02:21:37.389 INFO:tasks.workunit.client.0.smithi045.stdout: 971: throughput=5MB/sec pending data=0 2021-07-31T02:21:37.389 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-QjO9cE60oukb8Q5 off=3274134 len=638037 2021-07-31T02:21:37.390 INFO:tasks.workunit.client.0.smithi045.stdout:op 4945 completed, throughput=5MB/sec 2021-07-31T02:21:37.390 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-EuOlxIuLkbHbtwx off=12407992 len=1419098 2021-07-31T02:21:37.399 INFO:tasks.workunit.client.0.smithi045.stdout:op 4946 completed, throughput=5MB/sec 2021-07-31T02:21:37.399 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-0b_9-hViKhSpdjl off=16099238 len=506338 2021-07-31T02:21:37.403 INFO:tasks.workunit.client.0.smithi045.stdout:op 4947 completed, throughput=5MB/sec 2021-07-31T02:21:37.403 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-mUjCRva0Q9KwaEk off=1608125 len=1147310 2021-07-31T02:21:37.408 INFO:tasks.workunit.client.0.smithi045.stdout:op 4948 completed, throughput=5MB/sec 2021-07-31T02:21:37.409 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-WCCtM2rXFaq3Y_l off=12929726 len=1389874 2021-07-31T02:21:37.415 INFO:tasks.workunit.client.0.smithi045.stdout:op 4949 completed, throughput=5MB/sec 2021-07-31T02:21:37.415 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-0rEXbE1T_X3XFDX off=14593630 len=627748 2021-07-31T02:21:37.416 INFO:tasks.workunit.client.0.smithi045.stdout:op 4950 completed, throughput=5MB/sec 2021-07-31T02:21:38.416 INFO:tasks.workunit.client.0.smithi045.stdout: 972: throughput=5MB/sec pending data=0 2021-07-31T02:21:38.417 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-9cV8QExIqOX4AOW off=5959893 len=239831 2021-07-31T02:21:38.417 INFO:tasks.workunit.client.0.smithi045.stdout:op 4951 completed, throughput=5MB/sec 2021-07-31T02:21:38.417 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ugKF1432057lGif off=9186871 len=163798 2021-07-31T02:21:38.418 INFO:tasks.workunit.client.0.smithi045.stdout:op 4952 completed, throughput=5MB/sec 2021-07-31T02:21:38.418 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-oXUYjKr4dWmwnXO off=2054707 len=1150041 2021-07-31T02:21:38.420 INFO:tasks.workunit.client.0.smithi045.stdout:op 4953 completed, throughput=5MB/sec 2021-07-31T02:21:38.421 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-LAPDn1FeQOuHTg2 off=10449073 len=754509 2021-07-31T02:21:38.422 INFO:tasks.workunit.client.0.smithi045.stdout:op 4954 completed, throughput=5MB/sec 2021-07-31T02:21:38.422 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-CKucD8vZexUyzcm off=5128529 len=1512532 2021-07-31T02:21:38.430 INFO:tasks.workunit.client.0.smithi045.stdout:op 4955 completed, throughput=5MB/sec 2021-07-31T02:21:38.430 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-d-krXKtNOuGikeg off=3055592 len=780414 2021-07-31T02:21:38.431 INFO:tasks.workunit.client.0.smithi045.stdout:op 4956 completed, throughput=5MB/sec 2021-07-31T02:21:38.431 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-tUqH4F9vP0nvGhM off=1211600 len=1555573 2021-07-31T02:21:38.441 INFO:tasks.workunit.client.0.smithi045.stdout:op 4957 completed, throughput=5MB/sec 2021-07-31T02:21:38.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:38 smithi154 conmon[34553]: cluster 2021-07-31T02:21:36.284495+0000 mgr.x (mgr.24542) 1477 : cluster [DBG] pgmap v1903: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.4 MiB/s wr, 3 op/s 2021-07-31T02:21:38.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:38 smithi045 conmon[39718]: cluster 2021-07-31T02:21:36.284495+0000 mgr.x (mgr.24542) 1477 : cluster [DBG] pgmap v1903: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.4 MiB/s wr, 3 op/s 2021-07-31T02:21:38.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:38 smithi045 conmon[33655]: cluster 2021-07-31T02:21:36.284495+0000 mgr.x (mgr.24542) 1477 : cluster [DBG] 2021-07-31T02:21:38.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:38 smithi045 conmon[33655]: pgmap v1903: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.4 MiB/s wr, 3 op/s 2021-07-31T02:21:39.442 INFO:tasks.workunit.client.0.smithi045.stdout: 973: throughput=5MB/sec pending data=0 2021-07-31T02:21:39.442 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-LJF2ETV_iyydpm2 off=1506822 len=865087 2021-07-31T02:21:39.447 INFO:tasks.workunit.client.0.smithi045.stdout:op 4958 completed, throughput=5MB/sec 2021-07-31T02:21:39.448 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-0LWY10OxgMk3IRG off=22940515 len=475555 2021-07-31T02:21:39.452 INFO:tasks.workunit.client.0.smithi045.stdout:op 4959 completed, throughput=5MB/sec 2021-07-31T02:21:39.452 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-xKP5C3TEeGjryqP off=18636550 len=1971881 2021-07-31T02:21:39.461 INFO:tasks.workunit.client.0.smithi045.stdout:op 4960 completed, throughput=5MB/sec 2021-07-31T02:21:39.461 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-oZXOlKyWniRfcOr off=10976351 len=1079226 2021-07-31T02:21:39.462 INFO:tasks.workunit.client.0.smithi045.stdout:op 4961 completed, throughput=5MB/sec 2021-07-31T02:21:40.463 INFO:tasks.workunit.client.0.smithi045.stdout: 974: throughput=5MB/sec pending data=0 2021-07-31T02:21:40.463 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-o_iS_eLRfAp_dd2 off=8480440 len=1589996 2021-07-31T02:21:40.465 INFO:tasks.workunit.client.0.smithi045.stdout:op 4962 completed, throughput=5MB/sec 2021-07-31T02:21:40.465 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-rH4eWtFpE5JW3JZ off=18393562 len=2023673 2021-07-31T02:21:40.474 INFO:tasks.workunit.client.0.smithi045.stdout:op 4963 completed, throughput=5MB/sec 2021-07-31T02:21:40.474 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-BdAIan9I0-h0YLv off=8389840 len=215821 2021-07-31T02:21:40.477 INFO:tasks.workunit.client.0.smithi045.stdout:op 4964 completed, throughput=5MB/sec 2021-07-31T02:21:40.477 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-fyM0xIR4hURn67H off=16679768 len=1544521 2021-07-31T02:21:40.485 INFO:tasks.workunit.client.0.smithi045.stdout:op 4965 completed, throughput=5MB/sec 2021-07-31T02:21:40.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:40 smithi045 conmon[33655]: cluster 2021-07-31T02:21:38.285097+0000 2021-07-31T02:21:40.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:40 smithi045 conmon[33655]: mgr.x (mgr.24542) 1478 : cluster [DBG] pgmap v1904: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.4 MiB/s wr, 3 op/s 2021-07-31T02:21:40.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:40 smithi045 conmon[39718]: cluster 2021-07-31T02:21:38.285097+0000 mgr.x 2021-07-31T02:21:40.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:40 smithi045 conmon[39718]: (mgr.24542) 1478 : cluster [DBG] pgmap v1904: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.4 MiB/s wr, 3 op/s 2021-07-31T02:21:40.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:40 smithi154 conmon[34553]: cluster 2021-07-31T02:21:38.285097+0000 mgr.x (mgr.24542) 1478 : cluster [DBG] pgmap v1904: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.4 MiB/s wr, 3 op/s 2021-07-31T02:21:41.485 INFO:tasks.workunit.client.0.smithi045.stdout: 975: throughput=5MB/sec pending data=0 2021-07-31T02:21:41.485 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-h93vaGvNFeus6oI off=2316954 len=1507574 2021-07-31T02:21:41.494 INFO:tasks.workunit.client.0.smithi045.stdout:op 4966 completed, throughput=5MB/sec 2021-07-31T02:21:41.495 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-u5gyWKny5iFn5pe off=1686959 len=498957 2021-07-31T02:21:41.495 INFO:tasks.workunit.client.0.smithi045.stdout:op 4967 completed, throughput=5MB/sec 2021-07-31T02:21:41.495 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ZVIUkCjXjT3806t off=6090134 len=1645943 2021-07-31T02:21:41.503 INFO:tasks.workunit.client.0.smithi045.stdout:op 4968 completed, throughput=5MB/sec 2021-07-31T02:21:41.503 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-izzZiUiC7GWaDBK off=11144226 len=1092460 2021-07-31T02:21:41.505 INFO:tasks.workunit.client.0.smithi045.stdout:op 4969 completed, throughput=5MB/sec 2021-07-31T02:21:41.505 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-3lQNORn8YTV1xHx off=4519014 len=1555982 2021-07-31T02:21:41.514 INFO:tasks.workunit.client.0.smithi045.stdout:op 4970 completed, throughput=5MB/sec 2021-07-31T02:21:42.514 INFO:tasks.workunit.client.0.smithi045.stdout: 976: throughput=5MB/sec pending data=0 2021-07-31T02:21:42.515 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-b-Qb9k8vhlUFced off=5763910 len=1654174 2021-07-31T02:21:42.517 INFO:tasks.workunit.client.0.smithi045.stdout:op 4971 completed, throughput=5MB/sec 2021-07-31T02:21:42.518 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj--cuFS6ZJPRMdBz2 off=55699 len=501512 2021-07-31T02:21:42.519 INFO:tasks.workunit.client.0.smithi045.stdout:op 4972 completed, throughput=5MB/sec 2021-07-31T02:21:42.519 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Yos72vol2FCZrJd off=790416 len=1642748 2021-07-31T02:21:42.521 INFO:tasks.workunit.client.0.smithi045.stdout:op 4973 completed, throughput=5MB/sec 2021-07-31T02:21:42.521 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Q6dJY00QBGoJIhb off=10790469 len=827916 2021-07-31T02:21:42.522 INFO:tasks.workunit.client.0.smithi045.stdout:op 4974 completed, throughput=5MB/sec 2021-07-31T02:21:42.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:42 smithi045 conmon[33655]: cluster 2021-07-31T02:21:40.285669+0000 mgr.x (mgr.24542) 1479 : cluster [DBG] pgmap v1905: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 2.9 MiB/s rd, 3.6 MiB/s wr, 6 op/s 2021-07-31T02:21:42.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:42 smithi045 conmon[33655]: 2021-07-31T02:21:42.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:42 smithi045 conmon[39718]: cluster 2021-07-31T02:21:40.285669+0000 mgr.x (mgr. 2021-07-31T02:21:42.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:42 smithi045 conmon[39718]: 24542) 1479 : cluster [DBG] pgmap v1905: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 2.9 MiB/s rd, 3.6 MiB/s wr, 6 op/s 2021-07-31T02:21:42.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:42 smithi154 conmon[34553]: cluster 2021-07-31T02:21:40.285669+0000 mgr.x (mgr.24542) 1479 : cluster [DBG] pgmap v1905: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 2.9 MiB/s rd, 3.6 MiB/s wr, 6 op/s 2021-07-31T02:21:43.523 INFO:tasks.workunit.client.0.smithi045.stdout: 977: throughput=5MB/sec pending data=0 2021-07-31T02:21:43.523 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-tjCoVv-Zkz_w7s0 off=665240 len=1906075 2021-07-31T02:21:43.527 INFO:tasks.workunit.client.0.smithi045.stdout:op 4975 completed, throughput=5MB/sec 2021-07-31T02:21:43.527 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-7gax0YZQ8piqBYl off=1849476 len=1632657 2021-07-31T02:21:43.535 INFO:tasks.workunit.client.0.smithi045.stdout:op 4976 completed, throughput=5MB/sec 2021-07-31T02:21:43.536 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-U2e50fIhpJ3J-Lw off=3632849 len=598863 2021-07-31T02:21:43.537 INFO:tasks.workunit.client.0.smithi045.stdout:op 4977 completed, throughput=5MB/sec 2021-07-31T02:21:43.537 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-gv3h9-_MyV7SRrr off=2153027 len=1386468 2021-07-31T02:21:43.543 INFO:tasks.workunit.client.0.smithi045.stdout:op 4978 completed, throughput=5MB/sec 2021-07-31T02:21:44.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:21:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:21:43.568Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:21:44.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:21:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:21:43.568Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:21:44.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:21:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:21:43.568Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:21:44.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:44 smithi045 conmon[39718]: cluster 2021-07-31T02:21:42.286155+0000 mgr.x (mgr. 2021-07-31T02:21:44.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:44 smithi045 conmon[39718]: 24542) 1480 : cluster [DBG] pgmap v1906: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:21:44.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:44 smithi045 conmon[33655]: cluster 2021-07-31T02:21:42.286155 2021-07-31T02:21:44.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:44 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 1480 : cluster [DBG] pgmap v1906: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:21:44.544 INFO:tasks.workunit.client.0.smithi045.stdout: 978: throughput=5MB/sec pending data=0 2021-07-31T02:21:44.544 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-6zSkLFVRMB5jbfu off=345012 len=659735 2021-07-31T02:21:44.545 INFO:tasks.workunit.client.0.smithi045.stdout:op 4979 completed, throughput=5MB/sec 2021-07-31T02:21:44.545 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Urp_2ZmWnDmt9wo off=16270156 len=1758614 2021-07-31T02:21:44.556 INFO:tasks.workunit.client.0.smithi045.stdout:op 4980 completed, throughput=5MB/sec 2021-07-31T02:21:44.556 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-uTC2pg6bLElw6lj off=2392207 len=307281 2021-07-31T02:21:44.559 INFO:tasks.workunit.client.0.smithi045.stdout:op 4981 completed, throughput=5MB/sec 2021-07-31T02:21:44.560 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-rumtZNI0xf05MeG off=844004 len=1833034 2021-07-31T02:21:44.569 INFO:tasks.workunit.client.0.smithi045.stdout:op 4982 completed, throughput=5MB/sec 2021-07-31T02:21:44.569 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-tVNcyjhcQhEgyge off=6787606 len=218548 2021-07-31T02:21:44.570 INFO:tasks.workunit.client.0.smithi045.stdout:op 4983 completed, throughput=5MB/sec 2021-07-31T02:21:44.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:44 smithi154 conmon[34553]: cluster 2021-07-31T02:21:42.286155+0000 mgr.x (mgr.24542) 1480 : cluster [DBG] pgmap v1906: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:21:45.570 INFO:tasks.workunit.client.0.smithi045.stdout: 979: throughput=5MB/sec pending data=0 2021-07-31T02:21:45.571 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-plUqXgdtli-vgSo off=8470541 len=1588440 2021-07-31T02:21:45.574 INFO:tasks.workunit.client.0.smithi045.stdout:op 4984 completed, throughput=5MB/sec 2021-07-31T02:21:45.574 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-OJccfo6H3Ot3Gdf off=8633738 len=90960 2021-07-31T02:21:45.575 INFO:tasks.workunit.client.0.smithi045.stdout:op 4985 completed, throughput=5MB/sec 2021-07-31T02:21:45.575 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-podpT7WomieOghG off=856603 len=529849 2021-07-31T02:21:45.576 INFO:tasks.workunit.client.0.smithi045.stdout:op 4986 completed, throughput=5MB/sec 2021-07-31T02:21:45.577 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-zAPunYytZWzL6Ww off=2267096 len=2033824 2021-07-31T02:21:45.580 INFO:tasks.workunit.client.0.smithi045.stdout:op 4987 completed, throughput=5MB/sec 2021-07-31T02:21:45.581 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-UinL1LvnwXwCcS2 off=2436995 len=1811896 2021-07-31T02:21:45.587 INFO:tasks.workunit.client.0.smithi045.stdout:op 4988 completed, throughput=5MB/sec 2021-07-31T02:21:46.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:46 smithi045 conmon[39718]: cluster 2021-07-31T02:21:44.286659+0000 mgr.x 2021-07-31T02:21:46.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:46 smithi045 conmon[39718]: (mgr.24542) 1481 : cluster [DBG] pgmap v1907: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 2.9 MiB/s rd, 3.6 MiB/s wr, 6 op/s 2021-07-31T02:21:46.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:46 smithi045 conmon[33655]: cluster 2021-07-31T02:21:44.286659+0000 mgr.x (mgr.24542) 1481 : cluster 2021-07-31T02:21:46.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:46 smithi045 conmon[33655]: [DBG] pgmap v1907: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 2.9 MiB/s rd, 3.6 MiB/s wr, 6 op/s 2021-07-31T02:21:46.588 INFO:tasks.workunit.client.0.smithi045.stdout: 980: throughput=5MB/sec pending data=0 2021-07-31T02:21:46.588 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-KMbGQmpNJozIwxG off=10919910 len=642254 2021-07-31T02:21:46.592 INFO:tasks.workunit.client.0.smithi045.stdout:op 4989 completed, throughput=5MB/sec 2021-07-31T02:21:46.592 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-0yCV9YmGoUV5qnB off=6206290 len=760486 2021-07-31T02:21:46.596 INFO:tasks.workunit.client.0.smithi045.stdout:op 4990 completed, throughput=5MB/sec 2021-07-31T02:21:46.596 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-avqxfyVAGfm5SlU off=21618216 len=71880 2021-07-31T02:21:46.596 INFO:tasks.workunit.client.0.smithi045.stdout:op 4991 completed, throughput=5MB/sec 2021-07-31T02:21:46.596 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-TlsWAuPmyzdi_DH off=14137365 len=1555606 2021-07-31T02:21:46.597 INFO:tasks.workunit.client.0.smithi045.stdout:op 4992 completed, throughput=5MB/sec 2021-07-31T02:21:46.597 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-kRn71Fzi81qmnDR off=9179377 len=1051338 2021-07-31T02:21:46.601 INFO:tasks.workunit.client.0.smithi045.stdout:op 4993 completed, throughput=5MB/sec 2021-07-31T02:21:46.602 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-9-d4c2SPXNwqUZ1 off=295659 len=1578363 2021-07-31T02:21:46.609 INFO:tasks.workunit.client.0.smithi045.stdout:op 4994 completed, throughput=5MB/sec 2021-07-31T02:21:46.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:46 smithi154 conmon[34553]: cluster 2021-07-31T02:21:44.286659+0000 mgr.x (mgr.24542) 1481 : cluster [DBG] pgmap v1907: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 2.9 MiB/s rd, 3.6 MiB/s wr, 6 op/s 2021-07-31T02:21:47.610 INFO:tasks.workunit.client.0.smithi045.stdout: 981: throughput=5MB/sec pending data=0 2021-07-31T02:21:47.610 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-A9w-2fmHUFH1kyn off=6177260 len=173126 2021-07-31T02:21:47.611 INFO:tasks.workunit.client.0.smithi045.stdout:op 4995 completed, throughput=5MB/sec 2021-07-31T02:21:47.611 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-3gs9PmhpoukdI26 off=10081423 len=934184 2021-07-31T02:21:47.616 INFO:tasks.workunit.client.0.smithi045.stdout:op 4996 completed, throughput=5MB/sec 2021-07-31T02:21:47.616 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-tP1rk1Z4bu8PeNW off=2219073 len=620337 2021-07-31T02:21:47.617 INFO:tasks.workunit.client.0.smithi045.stdout:op 4997 completed, throughput=5MB/sec 2021-07-31T02:21:47.618 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-FYtIbTH42MoUGGA off=1250944 len=832103 2021-07-31T02:21:47.622 INFO:tasks.workunit.client.0.smithi045.stdout:op 4998 completed, throughput=5MB/sec 2021-07-31T02:21:47.622 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-2R_fcp1XWr1BCBZ off=1625122 len=2072610 2021-07-31T02:21:47.625 INFO:tasks.workunit.client.0.smithi045.stdout:op 4999 completed, throughput=5MB/sec 2021-07-31T02:21:48.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:48 smithi045 conmon[33655]: cluster 2021-07-31T02:21:46.287148+0000 mgr.x (mgr.24542) 1482 : cluster 2021-07-31T02:21:48.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:48 smithi045 conmon[33655]: [DBG] pgmap v1908: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:21:48.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:48 smithi045 conmon[39718]: cluster 2021-07-31T02:21:46.287148+0000 mgr.x ( 2021-07-31T02:21:48.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:48 smithi045 conmon[39718]: mgr.24542) 1482 : cluster [DBG] pgmap v1908: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:21:48.626 INFO:tasks.workunit.client.0.smithi045.stdout: 982: throughput=5MB/sec pending data=0 2021-07-31T02:21:48.626 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-mL_PlTNm62Z8uk4 off=20180761 len=1073719 2021-07-31T02:21:48.627 INFO:tasks.workunit.client.0.smithi045.stdout:op 5000 completed, throughput=5MB/sec 2021-07-31T02:21:48.628 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-G-_dPSAAN1xp5J6 off=76287 len=570763 2021-07-31T02:21:48.629 INFO:tasks.workunit.client.0.smithi045.stdout:op 5001 completed, throughput=5MB/sec 2021-07-31T02:21:48.630 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-HhwVQXRzXCLtXny off=2326552 len=1313712 2021-07-31T02:21:48.631 INFO:tasks.workunit.client.0.smithi045.stdout:op 5002 completed, throughput=5MB/sec 2021-07-31T02:21:48.632 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-XRdDs-xTaEEX0uU off=4826280 len=1102746 2021-07-31T02:21:48.637 INFO:tasks.workunit.client.0.smithi045.stdout:op 5003 completed, throughput=5MB/sec 2021-07-31T02:21:48.638 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-i0rEoszjoEC0v-1 off=3254282 len=904516 2021-07-31T02:21:48.639 INFO:tasks.workunit.client.0.smithi045.stdout:op 5004 completed, throughput=5MB/sec 2021-07-31T02:21:48.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:48 smithi154 conmon[34553]: cluster 2021-07-31T02:21:46.287148+0000 mgr.x (mgr.24542) 1482 : cluster [DBG] pgmap v1908: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:21:49.639 INFO:tasks.workunit.client.0.smithi045.stdout: 983: throughput=5MB/sec pending data=0 2021-07-31T02:21:49.640 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-3VauABfTpEzJ7nR off=634087 len=319838 2021-07-31T02:21:49.643 INFO:tasks.workunit.client.0.smithi045.stdout:op 5005 completed, throughput=5MB/sec 2021-07-31T02:21:49.643 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-3-pBIaKZkvNaxsN off=638313 len=1878592 2021-07-31T02:21:49.650 INFO:tasks.workunit.client.0.smithi045.stdout:op 5006 completed, throughput=5MB/sec 2021-07-31T02:21:49.651 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-T18aUeT5sMkqPjX off=11597629 len=973357 2021-07-31T02:21:49.656 INFO:tasks.workunit.client.0.smithi045.stdout:op 5007 completed, throughput=5MB/sec 2021-07-31T02:21:49.657 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-4JcEL8zGwyXJoi_ off=6435043 len=332106 2021-07-31T02:21:49.659 INFO:tasks.workunit.client.0.smithi045.stdout:op 5008 completed, throughput=5MB/sec 2021-07-31T02:21:49.660 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-VOA2WvRgCDvXOvE off=1658110 len=1867625 2021-07-31T02:21:49.663 INFO:tasks.workunit.client.0.smithi045.stdout:op 5009 completed, throughput=5MB/sec 2021-07-31T02:21:50.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:50 smithi045 conmon[39718]: cluster 2021-07-31T02:21:48.287524+0000 mgr.x (mgr 2021-07-31T02:21:50.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:50 smithi045 conmon[39718]: .24542) 1483 : cluster [DBG] pgmap v1909: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:21:50.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:50 smithi045 conmon[33655]: cluster 2021-07-31T02:21:48.287524+0000 mgr.x (mgr.24542) 1483 : cluster [DBG] pgmap v1909: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:21:50.663 INFO:tasks.workunit.client.0.smithi045.stdout: 984: throughput=5MB/sec pending data=0 2021-07-31T02:21:50.664 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-GBHJI6PMIgDsUeM off=2230459 len=444997 2021-07-31T02:21:50.667 INFO:tasks.workunit.client.0.smithi045.stdout:op 5010 completed, throughput=5MB/sec 2021-07-31T02:21:50.667 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-fjjvZo5lEMwjpP3 off=1179859 len=1828434 2021-07-31T02:21:50.671 INFO:tasks.workunit.client.0.smithi045.stdout:op 5011 completed, throughput=5MB/sec 2021-07-31T02:21:50.671 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-g6E1QvkCOoi4Mik off=3572208 len=1747194 2021-07-31T02:21:50.681 INFO:tasks.workunit.client.0.smithi045.stdout:op 5012 completed, throughput=5MB/sec 2021-07-31T02:21:50.681 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj--_NzIT5TSpg8osP off=11165454 len=1113483 2021-07-31T02:21:50.682 INFO:tasks.workunit.client.0.smithi045.stdout:op 5013 completed, throughput=5MB/sec 2021-07-31T02:21:50.682 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-0wCck9QEy3oa_pA off=8011633 len=48154 2021-07-31T02:21:50.685 INFO:tasks.workunit.client.0.smithi045.stdout:op 5014 completed, throughput=5MB/sec 2021-07-31T02:21:50.685 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-RNetz4nenJdBive off=6349431 len=705856 2021-07-31T02:21:50.685 INFO:tasks.workunit.client.0.smithi045.stdout:op 5015 completed, throughput=5MB/sec 2021-07-31T02:21:50.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:50 smithi154 conmon[34553]: cluster 2021-07-31T02:21:48.287524+0000 mgr.x (mgr.24542) 1483 : cluster [DBG] pgmap v1909: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:21:51.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:51 smithi045 conmon[39718]: cluster 2021-07-31T02:21:50.288020+0000 mgr.x (mgr 2021-07-31T02:21:51.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:51 smithi045 conmon[39718]: .24542) 1484 : cluster [DBG] pgmap v1910: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 3.0 MiB/s rd, 3.3 MiB/s wr, 6 op/s 2021-07-31T02:21:51.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:51 smithi045 conmon[33655]: cluster 2021-07-31T02:21:50.288020+0000 mgr.x (mgr. 2021-07-31T02:21:51.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:51 smithi045 conmon[33655]: 24542) 1484 : cluster [DBG] pgmap v1910: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 3.0 MiB/s rd, 3.3 MiB/s wr, 6 op/s 2021-07-31T02:21:51.686 INFO:tasks.workunit.client.0.smithi045.stdout: 985: throughput=5MB/sec pending data=0 2021-07-31T02:21:51.686 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-jiJ9N4gVn9pl37L off=2908280 len=832369 2021-07-31T02:21:51.687 INFO:tasks.workunit.client.0.smithi045.stdout:op 5016 completed, throughput=5MB/sec 2021-07-31T02:21:51.687 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-FhTtv1l1Dcb9avn off=910041 len=219284 2021-07-31T02:21:51.688 INFO:tasks.workunit.client.0.smithi045.stdout:op 5017 completed, throughput=5MB/sec 2021-07-31T02:21:51.688 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-JIcX3dJn3EWCjZV off=10499141 len=2031786 2021-07-31T02:21:51.696 INFO:tasks.workunit.client.0.smithi045.stdout:op 5018 completed, throughput=5MB/sec 2021-07-31T02:21:51.696 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-wns1qrDHOGU_BDY off=9526377 len=1263213 2021-07-31T02:21:51.698 INFO:tasks.workunit.client.0.smithi045.stdout:op 5019 completed, throughput=5MB/sec 2021-07-31T02:21:51.699 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-cujeh9GTRKF4_Fr off=3615332 len=1158748 2021-07-31T02:21:51.705 INFO:tasks.workunit.client.0.smithi045.stdout:op 5020 completed, throughput=5MB/sec 2021-07-31T02:21:51.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:51 smithi154 conmon[34553]: cluster 2021-07-31T02:21:50.288020+0000 mgr.x (mgr.24542) 1484 : cluster [DBG] pgmap v1910: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 3.0 MiB/s rd, 3.3 MiB/s wr, 6 op/s 2021-07-31T02:21:52.705 INFO:tasks.workunit.client.0.smithi045.stdout: 986: throughput=5MB/sec pending data=0 2021-07-31T02:21:52.705 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-v-w9R5DM5RRrd4L off=338360 len=399103 2021-07-31T02:21:52.707 INFO:tasks.workunit.client.0.smithi045.stdout:op 5021 completed, throughput=5MB/sec 2021-07-31T02:21:52.707 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-58YciV6DIb-gH0Z off=1672444 len=1865149 2021-07-31T02:21:52.718 INFO:tasks.workunit.client.0.smithi045.stdout:op 5022 completed, throughput=5MB/sec 2021-07-31T02:21:52.718 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-uQQUEMj42nhwnAr off=14901083 len=1778072 2021-07-31T02:21:52.726 INFO:tasks.workunit.client.0.smithi045.stdout:op 5023 completed, throughput=5MB/sec 2021-07-31T02:21:52.726 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-VO3OcPFnlm2nVDP off=0 len=1169557 2021-07-31T02:21:52.732 INFO:tasks.workunit.client.0.smithi045.stdout:op 5024 completed, throughput=5MB/sec 2021-07-31T02:21:53.733 INFO:tasks.workunit.client.0.smithi045.stdout: 987: throughput=5MB/sec pending data=0 2021-07-31T02:21:53.733 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-3dPDtI5Nf_dKj9h off=14040848 len=703235 2021-07-31T02:21:53.738 INFO:tasks.workunit.client.0.smithi045.stdout:op 5025 completed, throughput=5MB/sec 2021-07-31T02:21:53.739 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-sjGaxFi0TZhVyLK off=3055832 len=1085174 2021-07-31T02:21:53.744 INFO:tasks.workunit.client.0.smithi045.stdout:op 5026 completed, throughput=5MB/sec 2021-07-31T02:21:53.744 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-m4b_D6jhmpfH7ZY off=9654398 len=555741 2021-07-31T02:21:53.745 INFO:tasks.workunit.client.0.smithi045.stdout:op 5027 completed, throughput=5MB/sec 2021-07-31T02:21:53.746 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-K0zFKxHvoNqU9DC off=14164449 len=1759506 2021-07-31T02:21:53.753 INFO:tasks.workunit.client.0.smithi045.stdout:op 5028 completed, throughput=5MB/sec 2021-07-31T02:21:53.754 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-F4y_6Y-D1rjjRwE off=15218925 len=128145 2021-07-31T02:21:53.756 INFO:tasks.workunit.client.0.smithi045.stdout:op 5029 completed, throughput=5MB/sec 2021-07-31T02:21:53.757 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-WPTcUmPT6YJCQ7x off=8860045 len=753126 2021-07-31T02:21:53.760 INFO:tasks.workunit.client.0.smithi045.stdout:op 5030 completed, throughput=5MB/sec 2021-07-31T02:21:54.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:21:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:21:53.568Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:21:54.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:21:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:21:53.568Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:21:54.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:21:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:21:53.568Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:21:54.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:54 smithi154 conmon[34553]: cluster 2021-07-31T02:21:52.288425+0000 mgr.x (mgr.24542) 1485 : cluster [DBG] pgmap v1911: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.1 MiB/s wr, 3 op/s 2021-07-31T02:21:54.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:54 smithi154 conmon[34553]: 2021-07-31T02:21:54.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:54 smithi045 conmon[39718]: cluster 2021-07-31T02:21:52.288425+0000 mgr.x 2021-07-31T02:21:54.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:54 smithi045 conmon[39718]: (mgr.24542) 1485 : cluster [DBG] pgmap v1911: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.1 MiB/s wr, 3 op/s 2021-07-31T02:21:54.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:54 smithi045 conmon[33655]: cluster 2021-07-31T02:21:52.288425+0000 mgr.x (mgr.24542) 1485 2021-07-31T02:21:54.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:54 smithi045 conmon[33655]: : cluster [DBG] pgmap v1911: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.1 MiB/s wr, 3 op/s 2021-07-31T02:21:54.760 INFO:tasks.workunit.client.0.smithi045.stdout: 988: throughput=5MB/sec pending data=0 2021-07-31T02:21:54.761 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-8SrjIBzOwHmdeiB off=5185612 len=1466024 2021-07-31T02:21:54.762 INFO:tasks.workunit.client.0.smithi045.stdout:op 5031 completed, throughput=5MB/sec 2021-07-31T02:21:54.763 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-yl8RPiUUuMNIw-P off=771584 len=883839 2021-07-31T02:21:54.774 INFO:tasks.workunit.client.0.smithi045.stdout:op 5032 completed, throughput=5MB/sec 2021-07-31T02:21:54.774 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj--K0VEPva9TXLIYS off=1064374 len=251337 2021-07-31T02:21:54.775 INFO:tasks.workunit.client.0.smithi045.stdout:op 5033 completed, throughput=5MB/sec 2021-07-31T02:21:54.775 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-8cnE4hwdwvfJqVf off=608330 len=2092082 2021-07-31T02:21:54.780 INFO:tasks.workunit.client.0.smithi045.stdout:op 5034 completed, throughput=5MB/sec 2021-07-31T02:21:54.780 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-armW6Et0c8hYSq9 off=325070 len=533206 2021-07-31T02:21:54.781 INFO:tasks.workunit.client.0.smithi045.stdout:op 5035 completed, throughput=5MB/sec 2021-07-31T02:21:55.782 INFO:tasks.workunit.client.0.smithi045.stdout: 989: throughput=5MB/sec pending data=0 2021-07-31T02:21:55.782 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-KsI7kbeU-_5hNVT off=2855039 len=1410915 2021-07-31T02:21:55.784 INFO:tasks.workunit.client.0.smithi045.stdout:op 5036 completed, throughput=5MB/sec 2021-07-31T02:21:55.784 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-AKTRX6L2b8jF5pE off=12475189 len=1391394 2021-07-31T02:21:55.792 INFO:tasks.workunit.client.0.smithi045.stdout:op 5037 completed, throughput=5MB/sec 2021-07-31T02:21:55.793 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ZjIQpAx4dVCECdl off=7077888 len=1724644 2021-07-31T02:21:55.795 INFO:tasks.workunit.client.0.smithi045.stdout:op 5038 completed, throughput=5MB/sec 2021-07-31T02:21:55.796 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Dv45_m6MLjwFJnt off=19615891 len=546401 2021-07-31T02:21:55.799 INFO:tasks.workunit.client.0.smithi045.stdout:op 5039 completed, throughput=5MB/sec 2021-07-31T02:21:55.799 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-BUaSPLEwKuKE4l2 off=18850293 len=155719 2021-07-31T02:21:55.800 INFO:tasks.workunit.client.0.smithi045.stdout:op 5040 completed, throughput=5MB/sec 2021-07-31T02:21:55.800 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-eRvhhBzbbHFmJON off=7143182 len=1171759 2021-07-31T02:21:55.802 INFO:tasks.workunit.client.0.smithi045.stdout:op 5041 completed, throughput=5MB/sec 2021-07-31T02:21:56.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:56 smithi154 conmon[34553]: cluster 2021-07-31T02:21:54.289058+0000 mgr.x (mgr.24542) 1486 : cluster [DBG] pgmap v1912: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 2.7 MiB/s rd, 3.6 MiB/s wr, 6 op/s 2021-07-31T02:21:56.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:56 smithi045 conmon[33655]: cluster 2021-07-31T02:21:54.289058+0000 mgr.x (mgr.24542 2021-07-31T02:21:56.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:56 smithi045 conmon[33655]: ) 1486 : cluster [DBG] pgmap v1912: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 2.7 MiB/s rd, 3.6 MiB/s wr, 6 op/s 2021-07-31T02:21:56.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:56 smithi045 conmon[39718]: cluster 2021-07-31T02:21:54. 2021-07-31T02:21:56.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:56 smithi045 conmon[39718]: 289058+0000 mgr.x (mgr.24542) 1486 : cluster [DBG] pgmap v1912: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 2.7 MiB/s rd, 3.6 MiB/s wr, 6 op/s 2021-07-31T02:21:56.802 INFO:tasks.workunit.client.0.smithi045.stdout: 990: throughput=5MB/sec pending data=0 2021-07-31T02:21:56.803 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-AksugRhbKh_rXTE off=1620684 len=1898046 2021-07-31T02:21:56.806 INFO:tasks.workunit.client.0.smithi045.stdout:op 5042 completed, throughput=5MB/sec 2021-07-31T02:21:56.806 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-EtJ23b2lztaTs78 off=6378832 len=1082036 2021-07-31T02:21:56.807 INFO:tasks.workunit.client.0.smithi045.stdout:op 5043 completed, throughput=5MB/sec 2021-07-31T02:21:56.808 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-jm-_N2JOW9bgKmF off=5302040 len=1167123 2021-07-31T02:21:56.810 INFO:tasks.workunit.client.0.smithi045.stdout:op 5044 completed, throughput=5MB/sec 2021-07-31T02:21:56.810 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-anJMRKAgXFzGwiW off=0 len=1399189 2021-07-31T02:21:56.818 INFO:tasks.workunit.client.0.smithi045.stdout:op 5045 completed, throughput=5MB/sec 2021-07-31T02:21:57.818 INFO:tasks.workunit.client.0.smithi045.stdout: 991: throughput=5MB/sec pending data=0 2021-07-31T02:21:57.819 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ayMMO8pYMwopXty off=3118281 len=377455 2021-07-31T02:21:57.822 INFO:tasks.workunit.client.0.smithi045.stdout:op 5046 completed, throughput=5MB/sec 2021-07-31T02:21:57.822 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-qDtIs3BjIc7PtH4 off=4670306 len=1088699 2021-07-31T02:21:57.825 INFO:tasks.workunit.client.0.smithi045.stdout:op 5047 completed, throughput=5MB/sec 2021-07-31T02:21:57.826 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-k9SMdZ8dDA0r-oy off=16343955 len=676413 2021-07-31T02:21:57.829 INFO:tasks.workunit.client.0.smithi045.stdout:op 5048 completed, throughput=5MB/sec 2021-07-31T02:21:57.829 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-_jAb4cPFTKc6deF off=1043651 len=507806 2021-07-31T02:21:57.831 INFO:tasks.workunit.client.0.smithi045.stdout:op 5049 completed, throughput=5MB/sec 2021-07-31T02:21:57.831 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-nri7oaEbvh7GmIP off=8064707 len=1841938 2021-07-31T02:21:57.832 INFO:tasks.workunit.client.0.smithi045.stdout:op 5050 completed, throughput=5MB/sec 2021-07-31T02:21:58.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:21:58 smithi154 conmon[34553]: cluster 2021-07-31T02:21:56.289600+0000 mgr.x (mgr.24542) 1487 : cluster [DBG] pgmap v1913: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:21:58.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:58 smithi045 conmon[33655]: cluster 2021-07-31T02:21:56. 2021-07-31T02:21:58.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:21:58 smithi045 conmon[33655]: 289600+0000 mgr.x (mgr.24542) 1487 : cluster [DBG] pgmap v1913: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:21:58.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:58 smithi045 conmon[39718]: cluster 2021-07-31T02:21:56.289600+0000 2021-07-31T02:21:58.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:21:58 smithi045 conmon[39718]: mgr.x (mgr.24542) 1487 : cluster [DBG] pgmap v1913: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:21:58.696 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:21:58.832 INFO:tasks.workunit.client.0.smithi045.stdout: 992: throughput=5MB/sec pending data=0 2021-07-31T02:21:58.832 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-QH17SbVd74A5x5P off=5031166 len=1357447 2021-07-31T02:21:58.835 INFO:tasks.workunit.client.0.smithi045.stdout:op 5051 completed, throughput=5MB/sec 2021-07-31T02:21:58.835 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-bB0lCGOlm28mQdT off=11804920 len=781734 2021-07-31T02:21:58.839 INFO:tasks.workunit.client.0.smithi045.stdout:op 5052 completed, throughput=5MB/sec 2021-07-31T02:21:58.840 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-6ZB0nVtN3uSCW9x off=5010151 len=130821 2021-07-31T02:21:58.840 INFO:tasks.workunit.client.0.smithi045.stdout:op 5053 completed, throughput=5MB/sec 2021-07-31T02:21:58.841 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-dPo1VUQLmBrlniA off=14347159 len=738704 2021-07-31T02:21:58.841 INFO:tasks.workunit.client.0.smithi045.stdout:op 5054 completed, throughput=5MB/sec 2021-07-31T02:21:58.841 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-b0HVAvtr-o2IUoh off=4097845 len=586014 2021-07-31T02:21:58.845 INFO:tasks.workunit.client.0.smithi045.stdout:op 5055 completed, throughput=5MB/sec 2021-07-31T02:21:58.845 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-qoxPK2TQQZtFyMy off=3682760 len=395643 2021-07-31T02:21:58.845 INFO:tasks.workunit.client.0.smithi045.stdout:op 5056 completed, throughput=5MB/sec 2021-07-31T02:21:58.846 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ydyTschpRdh55Q3 off=10760169 len=1621705 2021-07-31T02:21:58.853 INFO:tasks.workunit.client.0.smithi045.stdout:op 5057 completed, throughput=5MB/sec 2021-07-31T02:21:58.976 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:21:58.977 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 36m ago 37m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:21:58.977 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 34m ago 37m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:21:58.978 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 34m ago 40m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:21:58.979 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 36m ago 43m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:21:58.979 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 36m ago 43m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:21:58.979 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 34m ago 41m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:21:58.979 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 36m ago 41m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:21:58.980 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 36m ago 37m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:21:58.980 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 34m ago 37m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:21:58.980 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 36m ago 40m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:21:58.980 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 36m ago 40m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:21:58.981 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 36m ago 39m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:21:58.981 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 36m ago 39m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:21:58.981 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 34m ago 39m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:21:58.981 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 34m ago 38m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:21:58.981 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 34m ago 38m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:21:58.982 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 34m ago 38m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:21:58.982 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 34m ago 37m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:21:59.311 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:21:59.312 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:21:59.312 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:21:59.312 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:21:59.312 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:21:59.313 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:21:59.313 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:21:59.313 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:21:59.313 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:21:59.314 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:21:59.314 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:21:59.314 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:21:59.314 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:21:59.314 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:21:59.315 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:21:59.315 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:21:59.315 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:21:59.853 INFO:tasks.workunit.client.0.smithi045.stdout: 993: throughput=5MB/sec pending data=0 2021-07-31T02:21:59.853 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-j1rmNezSU7G3ZRX off=10223558 len=276972 2021-07-31T02:21:59.854 INFO:tasks.workunit.client.0.smithi045.stdout:op 5058 completed, throughput=5MB/sec 2021-07-31T02:21:59.855 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-w80XqxAV6-xuLXS off=169878 len=1654597 2021-07-31T02:21:59.857 INFO:tasks.workunit.client.0.smithi045.stdout:op 5059 completed, throughput=5MB/sec 2021-07-31T02:21:59.857 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-R2de_FbTYVD3peU off=2563053 len=758694 2021-07-31T02:21:59.858 INFO:tasks.workunit.client.0.smithi045.stdout:op 5060 completed, throughput=5MB/sec 2021-07-31T02:21:59.859 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-cSSrFV9rXgiSNv8 off=6135950 len=625717 2021-07-31T02:21:59.859 INFO:tasks.workunit.client.0.smithi045.stdout:op 5061 completed, throughput=5MB/sec 2021-07-31T02:21:59.859 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-iQviKseAd1hBSbV off=10147313 len=1037859 2021-07-31T02:21:59.864 INFO:tasks.workunit.client.0.smithi045.stdout:op 5062 completed, throughput=5MB/sec 2021-07-31T02:21:59.864 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-KHBegEendxMIXmh off=7921184 len=1867398 2021-07-31T02:21:59.867 INFO:tasks.workunit.client.0.smithi045.stdout:op 5063 completed, throughput=5MB/sec 2021-07-31T02:22:00.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:00 smithi154 conmon[34553]: cluster 2021-07-31T02:21:58.290106+0000 mgr.x (mgr.24542) 1488 : cluster [DBG] pgmap v1914: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:22:00.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:00 smithi154 conmon[34553]: audit 2021-07-31T02:21:58.684563+0000 mgr.x (mgr.24542) 1489 : audit [DBG] from='client.16311 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:22:00.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:00 smithi154 conmon[34553]: audit 2021-07-31T02:21:58.969436+0000 mgr.x (mgr.24542) 1490 : audit [DBG] from='client.26224 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:22:00.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:00 smithi154 conmon[34553]: audit 2021-07-31T02:21:59. 2021-07-31T02:22:00.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:00 smithi154 conmon[34553]: 312028+0000 mon.c (mon.1) 92 : audit [DBG] 2021-07-31T02:22:00.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:00 smithi154 conmon[34553]: from='client.? 172.21.15.45:0/3986728646' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:22:00.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:00 smithi045 conmon[33655]: cluster 2021-07-31T02:21:58.290106+0000 2021-07-31T02:22:00.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:00 smithi045 conmon[33655]: mgr.x (mgr.24542) 1488 : cluster [DBG] pgmap v1914: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:22:00.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:00 smithi045 conmon[33655]: audit 2021-07 2021-07-31T02:22:00.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:00 smithi045 conmon[33655]: -31T02:21:58.684563+0000 mgr.x (mgr.24542) 1489 : audit [DBG] 2021-07-31T02:22:00.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:00 smithi045 conmon[33655]: from='client.16311 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:22:00.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:00 smithi045 conmon[33655]: audit 2021-07-31T02:21:58.969436+0000 mgr.x (mgr. 2021-07-31T02:22:00.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:00 smithi045 conmon[33655]: 24542) 1490 : audit [DBG] from='client.26224 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:22:00.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:00 smithi045 conmon[33655]: audit 2021- 2021-07-31T02:22:00.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:00 smithi045 conmon[33655]: 07-31T02:21:59.312028+0000 2021-07-31T02:22:00.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:00 smithi045 conmon[33655]: mon.c (mon.1) 92 : audit [DBG] from='client.? 172.21.15.45:0/3986728646' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:22:00.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:00 smithi045 conmon[33655]: 2021-07-31T02:22:00.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:00 smithi045 conmon[39718]: cluster 2021-07-31T02:21: 2021-07-31T02:22:00.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:00 smithi045 conmon[39718]: 58.290106+0000 mgr.x (mgr.24542) 1488 : cluster [DBG] pgmap v1914: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:22:00.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:00 smithi045 conmon[39718]: audit 2021-07-31T02:22:00.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:00 smithi045 conmon[39718]: 2021-07-31 2021-07-31T02:22:00.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:00 smithi045 conmon[39718]: T02:21:58.684563+0000 2021-07-31T02:22:00.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:00 smithi045 conmon[39718]: mgr.x (mgr.24542) 1489 : 2021-07-31T02:22:00.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:00 smithi045 conmon[39718]: audit [DBG] from='client.16311 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:22:00.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:00 smithi045 conmon[39718]: audit 2021- 2021-07-31T02:22:00.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:00 smithi045 conmon[39718]: 07-31T02:21:58.969436 2021-07-31T02:22:00.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:00 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 1490 2021-07-31T02:22:00.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:00 smithi045 conmon[39718]: : audit [DBG] from='client.26224 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:22:00.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:00 smithi045 conmon[39718]: audit 2021 2021-07-31T02:22:00.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:00 smithi045 conmon[39718]: -07-31T02:21:59 2021-07-31T02:22:00.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:00 smithi045 conmon[39718]: .312028+0000 mon.c (mon. 2021-07-31T02:22:00.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:00 smithi045 conmon[39718]: 1) 92 : audit [DBG] from='client.? 172.21.15.45:0/3986728646' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:22:00.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:00 smithi045 conmon[39718]: 2021-07-31T02:22:00.868 INFO:tasks.workunit.client.0.smithi045.stdout: 994: throughput=5MB/sec pending data=0 2021-07-31T02:22:00.869 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-fZxfHyECeAaZymQ off=4873183 len=1850907 2021-07-31T02:22:00.875 INFO:tasks.workunit.client.0.smithi045.stdout:op 5064 completed, throughput=5MB/sec 2021-07-31T02:22:00.876 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-doT4GfBOB_p68Bl off=1733154 len=1404849 2021-07-31T02:22:00.883 INFO:tasks.workunit.client.0.smithi045.stdout:op 5065 completed, throughput=5MB/sec 2021-07-31T02:22:00.883 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-d5ldimMYHnHR7le off=707800 len=2041659 2021-07-31T02:22:00.890 INFO:tasks.workunit.client.0.smithi045.stdout:op 5066 completed, throughput=5MB/sec 2021-07-31T02:22:01.890 INFO:tasks.workunit.client.0.smithi045.stdout: 995: throughput=5MB/sec pending data=0 2021-07-31T02:22:01.891 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-1qijOc_m1DRxWhQ off=2160656 len=1555551 2021-07-31T02:22:01.892 INFO:tasks.workunit.client.0.smithi045.stdout:op 5067 completed, throughput=5MB/sec 2021-07-31T02:22:01.893 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-NM9JuqEjrvXV4Ys off=20163057 len=180059 2021-07-31T02:22:01.896 INFO:tasks.workunit.client.0.smithi045.stdout:op 5068 completed, throughput=5MB/sec 2021-07-31T02:22:01.896 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-DCNEaaqtVtnOac- off=4202187 len=1565979 2021-07-31T02:22:01.900 INFO:tasks.workunit.client.0.smithi045.stdout:op 5069 completed, throughput=5MB/sec 2021-07-31T02:22:01.901 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-_jAb4cPFTKc6deF off=1700164 len=455809 2021-07-31T02:22:01.902 INFO:tasks.workunit.client.0.smithi045.stdout:op 5070 completed, throughput=5MB/sec 2021-07-31T02:22:02.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:02 smithi154 conmon[34553]: cluster 2021-07-31T02:22:00.290595+0000 mgr.x (mgr.24542) 1491 : cluster [DBG] pgmap v1915: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 2.9 MiB/s rd, 3.1 MiB/s wr, 6 op/s 2021-07-31T02:22:02.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:02 smithi045 conmon[33655]: cluster 2021-07-31T02:22:00.290595 2021-07-31T02:22:02.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:02 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 1491 : cluster [DBG] pgmap v1915: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 2.9 MiB/s rd, 3.1 MiB/s wr, 6 op/s 2021-07-31T02:22:02.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:02 smithi045 conmon[39718]: cluster 2021-07-31T02:22:00.290595+0000 mgr.x 2021-07-31T02:22:02.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:02 smithi045 conmon[39718]: (mgr.24542) 1491 : cluster [DBG] pgmap v1915: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 2.9 MiB/s rd, 3.1 MiB/s wr, 6 op/s 2021-07-31T02:22:02.903 INFO:tasks.workunit.client.0.smithi045.stdout: 996: throughput=5MB/sec pending data=0 2021-07-31T02:22:02.903 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-8lTZ_zNyYPoLDe1 off=10794342 len=150358 2021-07-31T02:22:02.904 INFO:tasks.workunit.client.0.smithi045.stdout:op 5071 completed, throughput=5MB/sec 2021-07-31T02:22:02.904 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-HEd-2YYQKVuWWBx off=4407764 len=1334349 2021-07-31T02:22:02.906 INFO:tasks.workunit.client.0.smithi045.stdout:op 5072 completed, throughput=5MB/sec 2021-07-31T02:22:02.907 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-VehwZ3xXjmZUjeh off=2991189 len=1834050 2021-07-31T02:22:02.910 INFO:tasks.workunit.client.0.smithi045.stdout:op 5073 completed, throughput=5MB/sec 2021-07-31T02:22:02.910 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Eh9uGVJEhNlmOqY off=0 len=1581207 2021-07-31T02:22:02.918 INFO:tasks.workunit.client.0.smithi045.stdout:op 5074 completed, throughput=5MB/sec 2021-07-31T02:22:02.918 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Hey-fWZfZFL0EFc off=0 len=1469304 2021-07-31T02:22:02.925 INFO:tasks.workunit.client.0.smithi045.stdout:op 5075 completed, throughput=5MB/sec 2021-07-31T02:22:03.925 INFO:tasks.workunit.client.0.smithi045.stdout: 997: throughput=5MB/sec pending data=0 2021-07-31T02:22:03.926 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-SB_wudqhJCjtf55 off=6483553 len=1305947 2021-07-31T02:22:03.928 INFO:tasks.workunit.client.0.smithi045.stdout:op 5076 completed, throughput=5MB/sec 2021-07-31T02:22:03.929 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Dkklr5eXD2IL8Yy off=1623285 len=634797 2021-07-31T02:22:03.930 INFO:tasks.workunit.client.0.smithi045.stdout:op 5077 completed, throughput=5MB/sec 2021-07-31T02:22:03.930 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-3W_XG60DiRaDGVz off=1956712 len=675713 2021-07-31T02:22:03.934 INFO:tasks.workunit.client.0.smithi045.stdout:op 5078 completed, throughput=5MB/sec 2021-07-31T02:22:03.935 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-enP-d_SFYbNjKWy off=3121269 len=1956826 2021-07-31T02:22:03.942 INFO:tasks.workunit.client.0.smithi045.stdout:op 5079 completed, throughput=5MB/sec 2021-07-31T02:22:04.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:22:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:22:03.569Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:22:04.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:22:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:22:03.569Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:22:04.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:22:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:22:03.569Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:22:04.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:04 smithi154 conmon[34553]: cluster 2021-07-31T02:22:02.291145+0000 mgr.x (mgr.24542) 1492 : cluster [DBG] pgmap v1916: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.2 MiB/s wr, 3 op/s 2021-07-31T02:22:04.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:04 smithi045 conmon[33655]: cluster 2021-07-31T02:22:02.291145+0000 mgr.x (mgr.24542) 1492 2021-07-31T02:22:04.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:04 smithi045 conmon[33655]: : cluster [DBG] pgmap v1916: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.2 MiB/s wr, 3 op/s 2021-07-31T02:22:04.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:04 smithi045 conmon[39718]: cluster 2021-07-31T02:22:02. 2021-07-31T02:22:04.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:04 smithi045 conmon[39718]: 291145+0000 mgr.x (mgr.24542) 1492 : cluster [DBG] pgmap v1916: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.2 MiB/s wr, 3 op/s 2021-07-31T02:22:04.943 INFO:tasks.workunit.client.0.smithi045.stdout: 998: throughput=5MB/sec pending data=0 2021-07-31T02:22:04.943 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-_XbBnzDWJW1lBD4 off=4834770 len=356062 2021-07-31T02:22:04.946 INFO:tasks.workunit.client.0.smithi045.stdout:op 5080 completed, throughput=5MB/sec 2021-07-31T02:22:04.946 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-xef0FA04nkp4RFR off=18767295 len=735951 2021-07-31T02:22:04.948 INFO:tasks.workunit.client.0.smithi045.stdout:op 5081 completed, throughput=5MB/sec 2021-07-31T02:22:04.948 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj--tRlPKiESfXsg_W off=9129880 len=740694 2021-07-31T02:22:04.950 INFO:tasks.workunit.client.0.smithi045.stdout:op 5082 completed, throughput=5MB/sec 2021-07-31T02:22:04.950 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-sgN6Iy_on1Y7V4b off=12670104 len=472874 2021-07-31T02:22:04.954 INFO:tasks.workunit.client.0.smithi045.stdout:op 5083 completed, throughput=5MB/sec 2021-07-31T02:22:04.954 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-eBQIUNTdnGDc2D9 off=15759759 len=1265370 2021-07-31T02:22:04.960 INFO:tasks.workunit.client.0.smithi045.stdout:op 5084 completed, throughput=5MB/sec 2021-07-31T02:22:04.960 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-eFl7nrKurBeYPR0 off=74892 len=1974646 2021-07-31T02:22:04.962 INFO:tasks.workunit.client.0.smithi045.stdout:op 5085 completed, throughput=5MB/sec 2021-07-31T02:22:05.963 INFO:tasks.workunit.client.0.smithi045.stdout: 999: throughput=5MB/sec pending data=0 2021-07-31T02:22:05.963 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-f4jJ10nN4UG_3Bl off=8463376 len=2071302 2021-07-31T02:22:05.971 INFO:tasks.workunit.client.0.smithi045.stdout:op 5086 completed, throughput=5MB/sec 2021-07-31T02:22:05.971 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-SRzLytsqmpMCoH4 off=6617101 len=957129 2021-07-31T02:22:05.977 INFO:tasks.workunit.client.0.smithi045.stdout:op 5087 completed, throughput=5MB/sec 2021-07-31T02:22:05.977 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Fcf0Z0X96rfzCiJ off=6961270 len=1985167 2021-07-31T02:22:05.980 INFO:tasks.workunit.client.0.smithi045.stdout:op 5088 completed, throughput=5MB/sec 2021-07-31T02:22:06.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:06 smithi154 conmon[34553]: cluster 2021-07-31T02:22:04.291678+0000 mgr.x (mgr.24542) 1493 : cluster [DBG] pgmap v1917: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 2.9 MiB/s rd, 3.1 MiB/s wr, 5 op/s 2021-07-31T02:22:06.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:06 smithi045 conmon[39718]: cluster 2021-07-31T02:22: 2021-07-31T02:22:06.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:06 smithi045 conmon[39718]: 04.291678+0000 mgr.x (mgr.24542) 1493 : cluster [DBG] pgmap v1917: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 2.9 MiB/s rd, 3.1 MiB/s wr, 5 op/s 2021-07-31T02:22:06.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:06 smithi045 conmon[33655]: cluster 2021-07-31T02:22:04.291678+0000 mgr.x (mgr.24542) 1493 : cluster [DBG] pgmap v1917: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 2.9 MiB/s rd, 3.1 MiB/s wr, 5 op/s 2021-07-31T02:22:06.981 INFO:tasks.workunit.client.0.smithi045.stdout: 1000: throughput=5MB/sec pending data=0 2021-07-31T02:22:06.981 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-VKib5qt8YnWuR-8 off=703466 len=128692 2021-07-31T02:22:06.982 INFO:tasks.workunit.client.0.smithi045.stdout:op 5089 completed, throughput=5MB/sec 2021-07-31T02:22:06.982 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-uAT95zbNWE_Ux0U off=4988619 len=464539 2021-07-31T02:22:06.983 INFO:tasks.workunit.client.0.smithi045.stdout:op 5090 completed, throughput=5MB/sec 2021-07-31T02:22:06.983 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-AF90ZoWnB3hlVJ2 off=5122416 len=1280172 2021-07-31T02:22:06.986 INFO:tasks.workunit.client.0.smithi045.stdout:op 5091 completed, throughput=5MB/sec 2021-07-31T02:22:06.986 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-HtRJJ96ID2KTxHl off=13462739 len=1654938 2021-07-31T02:22:06.994 INFO:tasks.workunit.client.0.smithi045.stdout:op 5092 completed, throughput=5MB/sec 2021-07-31T02:22:06.994 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-gyYGqYzT98slMZr off=8638833 len=1300154 2021-07-31T02:22:07.001 INFO:tasks.workunit.client.0.smithi045.stdout:op 5093 completed, throughput=5MB/sec 2021-07-31T02:22:07.001 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-K0PmXOZWCP7_nfj off=1297054 len=844444 2021-07-31T02:22:07.006 INFO:tasks.workunit.client.0.smithi045.stdout:op 5094 completed, throughput=5MB/sec 2021-07-31T02:22:08.006 INFO:tasks.workunit.client.0.smithi045.stdout: 1001: throughput=5MB/sec pending data=0 2021-07-31T02:22:08.006 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-WSvZXJ7bzA-KLyx off=4711478 len=1430545 2021-07-31T02:22:08.015 INFO:tasks.workunit.client.0.smithi045.stdout:op 5095 completed, throughput=5MB/sec 2021-07-31T02:22:08.016 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-8kQlEqmM7Zaa1XZ off=5162574 len=1618555 2021-07-31T02:22:08.023 INFO:tasks.workunit.client.0.smithi045.stdout:op 5096 completed, throughput=5MB/sec 2021-07-31T02:22:08.024 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-VqM55H-KAl4E2vu off=928002 len=1018434 2021-07-31T02:22:08.028 INFO:tasks.workunit.client.0.smithi045.stdout:op 5097 completed, throughput=5MB/sec 2021-07-31T02:22:08.029 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-nhsZspEQMrfHgUR off=3946556 len=799148 2021-07-31T02:22:08.033 INFO:tasks.workunit.client.0.smithi045.stdout:op 5098 completed, throughput=5MB/sec 2021-07-31T02:22:08.033 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-IzoDw2jUVWrThqi off=2020361 len=374748 2021-07-31T02:22:08.037 INFO:tasks.workunit.client.0.smithi045.stdout:op 5099 completed, throughput=5MB/sec 2021-07-31T02:22:08.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:08 smithi154 conmon[34553]: cluster 2021-07-31T02:22:06.292253+0000 mgr.x (mgr.24542) 1494 : cluster [DBG] pgmap v1918: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 2.2 MiB/s rd, 1.6 MiB/s wr, 3 op/s 2021-07-31T02:22:08.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:08 smithi045 conmon[39718]: cluster 2021-07-31T02:22:06 2021-07-31T02:22:08.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:08 smithi045 conmon[39718]: .292253+0000 mgr.x (mgr.24542) 1494 : cluster [DBG] pgmap v1918: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 2.2 MiB/s rd, 1.6 MiB/s wr, 3 op/s 2021-07-31T02:22:08.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:08 smithi045 conmon[33655]: cluster 2021-07-31T02:22:06.292253+0000 mgr.x (mgr.24542) 1494 : cluster [DBG] pgmap v1918: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 2.2 MiB/s rd, 1.6 MiB/s wr, 3 op/s 2021-07-31T02:22:09.037 INFO:tasks.workunit.client.0.smithi045.stdout: 1002: throughput=5MB/sec pending data=0 2021-07-31T02:22:09.038 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-OaGZGfFL8PMhfqh off=465325 len=1417047 2021-07-31T02:22:09.040 INFO:tasks.workunit.client.0.smithi045.stdout:op 5100 completed, throughput=5MB/sec 2021-07-31T02:22:09.040 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-H60rJMG7fB2ZNn- off=801541 len=53885 2021-07-31T02:22:09.043 INFO:tasks.workunit.client.0.smithi045.stdout:op 5101 completed, throughput=5MB/sec 2021-07-31T02:22:09.044 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-3UTJW9hQD_8i2Ud off=8393930 len=571871 2021-07-31T02:22:09.045 INFO:tasks.workunit.client.0.smithi045.stdout:op 5102 completed, throughput=5MB/sec 2021-07-31T02:22:09.045 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-9RM0E7N8FgaqBV0 off=922206 len=192355 2021-07-31T02:22:09.048 INFO:tasks.workunit.client.0.smithi045.stdout:op 5103 completed, throughput=5MB/sec 2021-07-31T02:22:09.049 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-JLPK6UjvzH5gcak off=1451644 len=96244 2021-07-31T02:22:09.049 INFO:tasks.workunit.client.0.smithi045.stdout:op 5104 completed, throughput=5MB/sec 2021-07-31T02:22:09.050 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-k9SMdZ8dDA0r-oy off=22619528 len=292131 2021-07-31T02:22:09.052 INFO:tasks.workunit.client.0.smithi045.stdout:op 5105 completed, throughput=5MB/sec 2021-07-31T02:22:09.052 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-atjTYx6M6KRXZ8F off=321890 len=1742811 2021-07-31T02:22:09.055 INFO:tasks.workunit.client.0.smithi045.stdout:op 5106 completed, throughput=5MB/sec 2021-07-31T02:22:09.055 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-4fG43xyaVk4lDBI off=29915 len=526842 2021-07-31T02:22:09.060 INFO:tasks.workunit.client.0.smithi045.stdout:op 5107 completed, throughput=5MB/sec 2021-07-31T02:22:09.060 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-VKspf-c7xufM9Mf off=12893918 len=400139 2021-07-31T02:22:09.061 INFO:tasks.workunit.client.0.smithi045.stdout:op 5108 completed, throughput=5MB/sec 2021-07-31T02:22:10.061 INFO:tasks.workunit.client.0.smithi045.stdout: 1003: throughput=5MB/sec pending data=0 2021-07-31T02:22:10.062 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-9-UtbbU8XWDyOiz off=201400 len=1095724 2021-07-31T02:22:10.067 INFO:tasks.workunit.client.0.smithi045.stdout:op 5109 completed, throughput=5MB/sec 2021-07-31T02:22:10.068 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-NEtkQEltnRKLVK9 off=5433585 len=1884658 2021-07-31T02:22:10.076 INFO:tasks.workunit.client.0.smithi045.stdout:op 5110 completed, throughput=5MB/sec 2021-07-31T02:22:10.076 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-wW03wooWI9yMPOX off=3593949 len=689827 2021-07-31T02:22:10.080 INFO:tasks.workunit.client.0.smithi045.stdout:op 5111 completed, throughput=5MB/sec 2021-07-31T02:22:10.080 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-AigDaVCcLOlBEQr off=6189503 len=2039091 2021-07-31T02:22:10.089 INFO:tasks.workunit.client.0.smithi045.stdout:op 5112 completed, throughput=5MB/sec 2021-07-31T02:22:10.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:10 smithi154 conmon[34553]: cluster 2021-07-31T02:22:08.292687+0000 mgr.x (mgr.24542) 1495 : cluster [DBG] pgmap v1919: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 2.2 MiB/s rd, 1.6 MiB/s wr, 3 op/s 2021-07-31T02:22:10.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:10 smithi045 conmon[33655]: cluster 2021-07-31T02:22:08.292687+0000 2021-07-31T02:22:10.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:10 smithi045 conmon[33655]: mgr.x (mgr.24542) 1495 : cluster [DBG] pgmap v1919: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 2.2 MiB/s rd, 1.6 MiB/s wr, 3 op/s 2021-07-31T02:22:10.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:10 smithi045 conmon[39718]: cluster 2021-07-31T02:22:08 2021-07-31T02:22:10.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:10 smithi045 conmon[39718]: .292687+0000 mgr.x (mgr.24542) 1495 : cluster [DBG] pgmap v1919: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 2.2 MiB/s rd, 1.6 MiB/s wr, 3 op/s 2021-07-31T02:22:11.089 INFO:tasks.workunit.client.0.smithi045.stdout: 1004: throughput=5MB/sec pending data=0 2021-07-31T02:22:11.090 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-gzHV-kaUrdqbfAY off=13653550 len=1642230 2021-07-31T02:22:11.099 INFO:tasks.workunit.client.0.smithi045.stdout:op 5113 completed, throughput=5MB/sec 2021-07-31T02:22:11.099 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-4Y-4umrYiFljPuR off=6736992 len=1959230 2021-07-31T02:22:11.102 INFO:tasks.workunit.client.0.smithi045.stdout:op 5114 completed, throughput=5MB/sec 2021-07-31T02:22:11.102 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-7X3D1SekWRRYZGb off=1030484 len=231047 2021-07-31T02:22:11.103 INFO:tasks.workunit.client.0.smithi045.stdout:op 5115 completed, throughput=5MB/sec 2021-07-31T02:22:11.103 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-zbQITGONNkFy70w off=6309072 len=560851 2021-07-31T02:22:11.107 INFO:tasks.workunit.client.0.smithi045.stdout:op 5116 completed, throughput=5MB/sec 2021-07-31T02:22:11.107 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-nAz1QS_YzGYsslH off=62213 len=784136 2021-07-31T02:22:11.112 INFO:tasks.workunit.client.0.smithi045.stdout:op 5117 completed, throughput=5MB/sec 2021-07-31T02:22:12.113 INFO:tasks.workunit.client.0.smithi045.stdout: 1005: throughput=5MB/sec pending data=0 2021-07-31T02:22:12.113 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ou-lhgD3yYXRhhO off=11942918 len=919106 2021-07-31T02:22:12.114 INFO:tasks.workunit.client.0.smithi045.stdout:op 5118 completed, throughput=5MB/sec 2021-07-31T02:22:12.115 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Axh6ZlPHwboLmzN off=15654251 len=941581 2021-07-31T02:22:12.120 INFO:tasks.workunit.client.0.smithi045.stdout:op 5119 completed, throughput=5MB/sec 2021-07-31T02:22:12.120 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-7YP8BzUZhgm6ip4 off=2267031 len=1021650 2021-07-31T02:22:12.122 INFO:tasks.workunit.client.0.smithi045.stdout:op 5120 completed, throughput=5MB/sec 2021-07-31T02:22:12.122 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-2Z6TFRriSmGvQUj off=9666686 len=1789692 2021-07-31T02:22:12.133 INFO:tasks.workunit.client.0.smithi045.stdout:op 5121 completed, throughput=5MB/sec 2021-07-31T02:22:12.133 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-6BOY9ovnBCN_EDd off=12151333 len=727733 2021-07-31T02:22:12.137 INFO:tasks.workunit.client.0.smithi045.stdout:op 5122 completed, throughput=5MB/sec 2021-07-31T02:22:12.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:12 smithi154 conmon[34553]: cluster 2021-07-31T02:22:10.293272+0000 mgr.x (mgr.24542) 1496 : cluster [DBG] pgmap v1920: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 3.0 MiB/s rd, 3.0 MiB/s wr, 5 op/s 2021-07-31T02:22:12.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:12 smithi045 conmon[33655]: cluster 2021-07-31T02: 2021-07-31T02:22:12.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:12 smithi045 conmon[33655]: 22:10.293272+0000 mgr.x (mgr.24542) 1496 : cluster [DBG] pgmap v1920: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 3.0 MiB/s rd, 3.0 MiB/s wr, 5 op/s 2021-07-31T02:22:12.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:12 smithi045 conmon[39718]: cluster 2021-07-31T02:22:10.293272+0000 2021-07-31T02:22:12.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:12 smithi045 conmon[39718]: mgr.x (mgr.24542) 1496 : cluster [DBG] pgmap v1920: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 3.0 MiB/s rd, 3.0 MiB/s wr, 5 op/s 2021-07-31T02:22:13.137 INFO:tasks.workunit.client.0.smithi045.stdout: 1006: throughput=5MB/sec pending data=0 2021-07-31T02:22:13.138 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Mk_eSa2M926452G off=10353660 len=107011 2021-07-31T02:22:13.140 INFO:tasks.workunit.client.0.smithi045.stdout:op 5123 completed, throughput=5MB/sec 2021-07-31T02:22:13.141 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-YT0x1eRpgDPn5ad off=112823 len=131793 2021-07-31T02:22:13.141 INFO:tasks.workunit.client.0.smithi045.stdout:op 5124 completed, throughput=5MB/sec 2021-07-31T02:22:13.142 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-4ATy5p-X7blKQH1 off=4739825 len=2007669 2021-07-31T02:22:13.149 INFO:tasks.workunit.client.0.smithi045.stdout:op 5125 completed, throughput=5MB/sec 2021-07-31T02:22:13.149 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj--xrLxyvjEyKkatP off=1132841 len=41795 2021-07-31T02:22:13.152 INFO:tasks.workunit.client.0.smithi045.stdout:op 5126 completed, throughput=5MB/sec 2021-07-31T02:22:13.153 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Xp4x8FJYxpVSNEx off=3300392 len=300478 2021-07-31T02:22:13.154 INFO:tasks.workunit.client.0.smithi045.stdout:op 5127 completed, throughput=5MB/sec 2021-07-31T02:22:13.154 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-nhzO6eRkhbI7uoY off=292682 len=90741 2021-07-31T02:22:13.155 INFO:tasks.workunit.client.0.smithi045.stdout:op 5128 completed, throughput=5MB/sec 2021-07-31T02:22:13.156 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-buqPkCHeKNH3M6c off=166617 len=1242957 2021-07-31T02:22:13.156 INFO:tasks.workunit.client.0.smithi045.stdout:op 5129 completed, throughput=5MB/sec 2021-07-31T02:22:13.157 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-n5RLEeqd8PA6rz- off=3177547 len=214354 2021-07-31T02:22:13.159 INFO:tasks.workunit.client.0.smithi045.stdout:op 5130 completed, throughput=5MB/sec 2021-07-31T02:22:13.159 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-bOu5p-KhmfeyNAb off=250660 len=1536508 2021-07-31T02:22:13.160 INFO:tasks.workunit.client.0.smithi045.stdout:op 5131 completed, throughput=5MB/sec 2021-07-31T02:22:14.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:22:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:22:13.569Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:22:14.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:22:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:22:13.569Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:22:14.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:22:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:22:13.569Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:22:14.161 INFO:tasks.workunit.client.0.smithi045.stdout: 1007: throughput=5MB/sec pending data=0 2021-07-31T02:22:14.161 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-EbjM5v9j4_VI23M off=1024909 len=1282881 2021-07-31T02:22:14.167 INFO:tasks.workunit.client.0.smithi045.stdout:op 5132 completed, throughput=5MB/sec 2021-07-31T02:22:14.167 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ptRY6ilBzMDbz6o off=7649961 len=596784 2021-07-31T02:22:14.168 INFO:tasks.workunit.client.0.smithi045.stdout:op 5133 completed, throughput=5MB/sec 2021-07-31T02:22:14.168 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-6n1P01-7_aZrZmp off=5007592 len=1966190 2021-07-31T02:22:14.185 INFO:tasks.workunit.client.0.smithi045.stdout:op 5134 completed, throughput=5MB/sec 2021-07-31T02:22:14.186 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj--f-iuOaH4ONPNJi off=14399535 len=247454 2021-07-31T02:22:14.186 INFO:tasks.workunit.client.0.smithi045.stdout:op 5135 completed, throughput=5MB/sec 2021-07-31T02:22:14.187 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-N4ffiUqzP1CzU-V off=229762 len=2033012 2021-07-31T02:22:14.188 INFO:tasks.workunit.client.0.smithi045.stdout:op 5136 completed, throughput=5MB/sec 2021-07-31T02:22:14.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:14 smithi154 conmon[34553]: cluster 2021-07-31T02:22:12.293836+0000 mgr.x (mgr.24542) 1497 : cluster [DBG] pgmap v1921: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.2 MiB/s wr, 3 op/s 2021-07-31T02:22:14.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:14 smithi045 conmon[39718]: cluster 2021-07-31T02:22: 2021-07-31T02:22:14.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:14 smithi045 conmon[39718]: 12.293836+0000 mgr.x (mgr.24542) 1497 : cluster [DBG] pgmap v1921: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.2 MiB/s wr, 3 op/s 2021-07-31T02:22:14.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:14 smithi045 conmon[33655]: cluster 2021-07-31T02:22:12.293836+0000 mgr.x (mgr.24542 2021-07-31T02:22:14.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:14 smithi045 conmon[33655]: ) 1497 : cluster [DBG] pgmap v1921: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.2 MiB/s wr, 3 op/s 2021-07-31T02:22:15.188 INFO:tasks.workunit.client.0.smithi045.stdout: 1008: throughput=5MB/sec pending data=0 2021-07-31T02:22:15.189 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-vynQj6x2GypWISH off=4346801 len=1626306 2021-07-31T02:22:15.192 INFO:tasks.workunit.client.0.smithi045.stdout:op 5137 completed, throughput=5MB/sec 2021-07-31T02:22:15.192 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-k9SMdZ8dDA0r-oy off=19429139 len=309955 2021-07-31T02:22:15.196 INFO:tasks.workunit.client.0.smithi045.stdout:op 5138 completed, throughput=5MB/sec 2021-07-31T02:22:15.196 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-L1g7PoEZ3eRKz_8 off=15888470 len=872954 2021-07-31T02:22:15.197 INFO:tasks.workunit.client.0.smithi045.stdout:op 5139 completed, throughput=5MB/sec 2021-07-31T02:22:15.197 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-oBzBrBWkXGbwqYY off=1517253 len=2028186 2021-07-31T02:22:15.200 INFO:tasks.workunit.client.0.smithi045.stdout:op 5140 completed, throughput=5MB/sec 2021-07-31T02:22:16.201 INFO:tasks.workunit.client.0.smithi045.stdout: 1009: throughput=5MB/sec pending data=0 2021-07-31T02:22:16.201 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-KVvzl6gx6fQPbOd off=2308176 len=1298488 2021-07-31T02:22:16.204 INFO:tasks.workunit.client.0.smithi045.stdout:op 5141 completed, throughput=5MB/sec 2021-07-31T02:22:16.204 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-UktezRh55BMhrSM off=2559682 len=819385 2021-07-31T02:22:16.206 INFO:tasks.workunit.client.0.smithi045.stdout:op 5142 completed, throughput=5MB/sec 2021-07-31T02:22:16.206 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-qFxXtboiFKs0Kjb off=7375250 len=604909 2021-07-31T02:22:16.211 INFO:tasks.workunit.client.0.smithi045.stdout:op 5143 completed, throughput=5MB/sec 2021-07-31T02:22:16.211 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-TP-0VmSrY3b-w1W off=6675553 len=558216 2021-07-31T02:22:16.216 INFO:tasks.workunit.client.0.smithi045.stdout:op 5144 completed, throughput=5MB/sec 2021-07-31T02:22:16.216 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-m3qjk1YV2FInWHX off=11847638 len=1567864 2021-07-31T02:22:16.217 INFO:tasks.workunit.client.0.smithi045.stdout:op 5145 completed, throughput=5MB/sec 2021-07-31T02:22:16.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:16 smithi154 conmon[34553]: cluster 2021-07-31T02:22:14.294476+0000 mgr.x (mgr.24542) 1498 : cluster [DBG] pgmap v1922: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 2.8 MiB/s rd, 3.5 MiB/s wr, 6 op/s 2021-07-31T02:22:16.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:16 smithi045 conmon[33655]: cluster 2021-07-31T02:22:14.294476+0000 mgr.x (mgr.24542) 1498 : cluster [DBG] 2021-07-31T02:22:16.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:16 smithi045 conmon[33655]: pgmap v1922: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 2.8 MiB/s rd, 3.5 MiB/s wr, 6 op/s 2021-07-31T02:22:16.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:16 smithi045 conmon[39718]: cluster 2021-07-31T02:22:14.294476+0000 mgr.x (mgr.24542) 1498 2021-07-31T02:22:16.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:16 smithi045 conmon[39718]: : cluster [DBG] pgmap v1922: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 2.8 MiB/s rd, 3.5 MiB/s wr, 6 op/s 2021-07-31T02:22:17.218 INFO:tasks.workunit.client.0.smithi045.stdout: 1010: throughput=5MB/sec pending data=0 2021-07-31T02:22:17.218 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Zlgdi55JabRRxwa off=11511675 len=1692459 2021-07-31T02:22:17.226 INFO:tasks.workunit.client.0.smithi045.stdout:op 5146 completed, throughput=5MB/sec 2021-07-31T02:22:17.227 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Iw7iSn3jWRGq6_y off=5182359 len=2054619 2021-07-31T02:22:17.236 INFO:tasks.workunit.client.0.smithi045.stdout:op 5147 completed, throughput=5MB/sec 2021-07-31T02:22:17.236 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-pAJY8uA7ROi-H-P off=6092010 len=827120 2021-07-31T02:22:17.238 INFO:tasks.workunit.client.0.smithi045.stdout:op 5148 completed, throughput=5MB/sec 2021-07-31T02:22:17.239 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-gYH7XpFzcGyDXP- off=12881879 len=1712548 2021-07-31T02:22:17.245 INFO:tasks.workunit.client.0.smithi045.stdout:op 5149 completed, throughput=5MB/sec 2021-07-31T02:22:18.245 INFO:tasks.workunit.client.0.smithi045.stdout: 1011: throughput=5MB/sec pending data=0 2021-07-31T02:22:18.246 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-JRmmERy38De9Gpc off=21521158 len=1126022 2021-07-31T02:22:18.247 INFO:tasks.workunit.client.0.smithi045.stdout:op 5150 completed, throughput=5MB/sec 2021-07-31T02:22:18.248 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ScgV6zhlnc0BDAJ off=1934745 len=1386844 2021-07-31T02:22:18.250 INFO:tasks.workunit.client.0.smithi045.stdout:op 5151 completed, throughput=5MB/sec 2021-07-31T02:22:18.251 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-MLQc7VD8VsZsGIS off=4341280 len=1692105 2021-07-31T02:22:18.254 INFO:tasks.workunit.client.0.smithi045.stdout:op 5152 completed, throughput=5MB/sec 2021-07-31T02:22:18.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:18 smithi154 conmon[34553]: cluster 2021-07-31T02:22:16.294985+0000 mgr.x (mgr.24542) 1499 : cluster [DBG] pgmap v1923: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2021-07-31T02:22:18.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:18 smithi045 conmon[39718]: cluster 2021-07-31T02:22:16.294985+0000 mgr.x (mgr 2021-07-31T02:22:18.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:18 smithi045 conmon[39718]: .24542) 1499 : cluster [DBG] pgmap v1923: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2021-07-31T02:22:18.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:18 smithi045 conmon[33655]: cluster 2021-07-31T02:22:16.294985+0000 mgr.x 2021-07-31T02:22:18.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:18 smithi045 conmon[33655]: (mgr.24542) 1499 : cluster [DBG] pgmap v1923: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2021-07-31T02:22:19.254 INFO:tasks.workunit.client.0.smithi045.stdout: 1012: throughput=5MB/sec pending data=0 2021-07-31T02:22:19.254 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj--xdr_Dw28fWTJ2s off=8815259 len=1972202 2021-07-31T02:22:19.257 INFO:tasks.workunit.client.0.smithi045.stdout:op 5153 completed, throughput=5MB/sec 2021-07-31T02:22:19.257 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-BUzLgFqf-aQPqDs off=2947959 len=727135 2021-07-31T02:22:19.262 INFO:tasks.workunit.client.0.smithi045.stdout:op 5154 completed, throughput=5MB/sec 2021-07-31T02:22:19.262 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Yq8Zt1iJLnQmkBV off=0 len=1063314 2021-07-31T02:22:19.264 INFO:tasks.workunit.client.0.smithi045.stdout:op 5155 completed, throughput=5MB/sec 2021-07-31T02:22:19.264 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-hWz8emgWx0F8zTj off=3344216 len=295807 2021-07-31T02:22:19.267 INFO:tasks.workunit.client.0.smithi045.stdout:op 5156 completed, throughput=5MB/sec 2021-07-31T02:22:19.267 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-O9ImbqorBtjn0gj off=2924977 len=658413 2021-07-31T02:22:19.268 INFO:tasks.workunit.client.0.smithi045.stdout:op 5157 completed, throughput=5MB/sec 2021-07-31T02:22:19.269 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-3qg1ET8SvUvJ_AS off=2558248 len=228383 2021-07-31T02:22:19.272 INFO:tasks.workunit.client.0.smithi045.stdout:op 5158 completed, throughput=5MB/sec 2021-07-31T02:22:19.272 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-3HvjtlK-ms_CG5E off=5192284 len=1958262 2021-07-31T02:22:19.275 INFO:tasks.workunit.client.0.smithi045.stdout:op 5159 completed, throughput=5MB/sec 2021-07-31T02:22:20.276 INFO:tasks.workunit.client.0.smithi045.stdout: 1013: throughput=5MB/sec pending data=0 2021-07-31T02:22:20.276 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-O2xb36KqSz3mQhk off=255561 len=110845 2021-07-31T02:22:20.277 INFO:tasks.workunit.client.0.smithi045.stdout:op 5160 completed, throughput=5MB/sec 2021-07-31T02:22:20.277 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-oGe8lBhlVs2QkJi off=8128869 len=472993 2021-07-31T02:22:20.281 INFO:tasks.workunit.client.0.smithi045.stdout:op 5161 completed, throughput=5MB/sec 2021-07-31T02:22:20.281 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-7B-NJDNPMAWzWmo off=2417219 len=1898339 2021-07-31T02:22:20.294 INFO:tasks.workunit.client.0.smithi045.stdout:op 5162 completed, throughput=5MB/sec 2021-07-31T02:22:20.294 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-m_fh0epu7QWFL4C off=6068646 len=900834 2021-07-31T02:22:20.300 INFO:tasks.workunit.client.0.smithi045.stdout:op 5163 completed, throughput=5MB/sec 2021-07-31T02:22:20.300 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-q1IZ_aD0WumL3Dg off=5617211 len=37403 2021-07-31T02:22:20.301 INFO:tasks.workunit.client.0.smithi045.stdout:op 5164 completed, throughput=5MB/sec 2021-07-31T02:22:20.301 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-hlGPA52sHCFKVtL off=12132221 len=743700 2021-07-31T02:22:20.305 INFO:tasks.workunit.client.0.smithi045.stdout:op 5165 completed, throughput=5MB/sec 2021-07-31T02:22:20.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:20 smithi154 conmon[34553]: cluster 2021-07-31T02:22:18.295404+0000 mgr.x (mgr.24542) 1500 : cluster [DBG] pgmap v1924: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2021-07-31T02:22:20.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:20 smithi045 conmon[33655]: cluster 2021-07-31T02:22:18.295404+0000 mgr.x (mgr.24542) 2021-07-31T02:22:20.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:20 smithi045 conmon[33655]: 1500 : cluster [DBG] pgmap v1924: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2021-07-31T02:22:20.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:20 smithi045 conmon[39718]: cluster 2021-07-31T02:22:18.295404+0000 mgr.x ( 2021-07-31T02:22:20.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:20 smithi045 conmon[39718]: mgr.24542) 1500 : cluster [DBG] pgmap v1924: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2021-07-31T02:22:21.306 INFO:tasks.workunit.client.0.smithi045.stdout: 1014: throughput=5MB/sec pending data=0 2021-07-31T02:22:21.306 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-VXChwy-OdXabMHK off=7254846 len=568719 2021-07-31T02:22:21.307 INFO:tasks.workunit.client.0.smithi045.stdout:op 5166 completed, throughput=5MB/sec 2021-07-31T02:22:21.307 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-v5vsFhWf4y6iYFt off=4736270 len=312219 2021-07-31T02:22:21.308 INFO:tasks.workunit.client.0.smithi045.stdout:op 5167 completed, throughput=5MB/sec 2021-07-31T02:22:21.309 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Z9FGVLKU1v9evkS off=5132098 len=1864880 2021-07-31T02:22:21.312 INFO:tasks.workunit.client.0.smithi045.stdout:op 5168 completed, throughput=5MB/sec 2021-07-31T02:22:21.313 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-oESOrMbmbKwARbk off=21045483 len=1951290 2021-07-31T02:22:21.315 INFO:tasks.workunit.client.0.smithi045.stdout:op 5169 completed, throughput=5MB/sec 2021-07-31T02:22:22.316 INFO:tasks.workunit.client.0.smithi045.stdout: 1015: throughput=5MB/sec pending data=0 2021-07-31T02:22:22.316 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-r9xAJGmnWwVcKYR off=10370046 len=1928929 2021-07-31T02:22:22.327 INFO:tasks.workunit.client.0.smithi045.stdout:op 5170 completed, throughput=5MB/sec 2021-07-31T02:22:22.328 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-o3XMaoflyKVr7sU off=1917665 len=1219746 2021-07-31T02:22:22.334 INFO:tasks.workunit.client.0.smithi045.stdout:op 5171 completed, throughput=5MB/sec 2021-07-31T02:22:22.334 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-7KyfgQK1CMhTDTk off=13403046 len=96683 2021-07-31T02:22:22.337 INFO:tasks.workunit.client.0.smithi045.stdout:op 5172 completed, throughput=5MB/sec 2021-07-31T02:22:22.337 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-b0uKqDUGVSr8mEs off=10183817 len=1715423 2021-07-31T02:22:22.339 INFO:tasks.workunit.client.0.smithi045.stdout:op 5173 completed, throughput=5MB/sec 2021-07-31T02:22:22.339 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-2owHL22ldw7n1pr off=13466929 len=404001 2021-07-31T02:22:22.341 INFO:tasks.workunit.client.0.smithi045.stdout:op 5174 completed, throughput=5MB/sec 2021-07-31T02:22:22.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:22 smithi154 conmon[34553]: cluster 2021-07-31T02:22:20.295855+0000 mgr.x (mgr.24542) 1501 : cluster [DBG] pgmap v1925: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 2.9 MiB/s rd, 3.4 MiB/s wr, 6 op/s 2021-07-31T02:22:22.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:22 smithi045 conmon[39718]: cluster 2021-07-31T02:22:20. 2021-07-31T02:22:22.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:22 smithi045 conmon[39718]: 295855+0000 mgr.x (mgr.24542) 1501 : cluster [DBG] pgmap v1925: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 2.9 MiB/s rd, 3.4 MiB/s wr, 6 op/s 2021-07-31T02:22:22.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:22 smithi045 conmon[33655]: cluster 2021-07-31T02:22:20.295855+0000 mgr.x (mgr.24542) 1501 2021-07-31T02:22:22.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:22 smithi045 conmon[33655]: : cluster [DBG] pgmap v1925: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 2.9 MiB/s rd, 3.4 MiB/s wr, 6 op/s 2021-07-31T02:22:23.341 INFO:tasks.workunit.client.0.smithi045.stdout: 1016: throughput=5MB/sec pending data=0 2021-07-31T02:22:23.341 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-xA2kON5oDeV1pZ_ off=13828699 len=1389673 2021-07-31T02:22:23.344 INFO:tasks.workunit.client.0.smithi045.stdout:op 5175 completed, throughput=5MB/sec 2021-07-31T02:22:23.344 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-fXc6DSM2YzkHJCt off=1450673 len=906070 2021-07-31T02:22:23.345 INFO:tasks.workunit.client.0.smithi045.stdout:op 5176 completed, throughput=5MB/sec 2021-07-31T02:22:23.345 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-w1qwwnbv2fUsnzB off=12045405 len=1565936 2021-07-31T02:22:23.354 INFO:tasks.workunit.client.0.smithi045.stdout:op 5177 completed, throughput=5MB/sec 2021-07-31T02:22:23.354 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-LhejbDfvNBzmEbZ off=892237 len=1821685 2021-07-31T02:22:23.362 INFO:tasks.workunit.client.0.smithi045.stdout:op 5178 completed, throughput=5MB/sec 2021-07-31T02:22:24.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:22:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:22:23.569Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:22:24.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:22:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:22:23.569Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:22:24.027 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:22:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:22:23.569Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:22:24.362 INFO:tasks.workunit.client.0.smithi045.stdout: 1017: throughput=5MB/sec pending data=0 2021-07-31T02:22:24.362 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-_PhSYGqIiusCmdu off=386854 len=232241 2021-07-31T02:22:24.365 INFO:tasks.workunit.client.0.smithi045.stdout:op 5179 completed, throughput=5MB/sec 2021-07-31T02:22:24.366 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-uiC5fU_LLTFb1xk off=5922832 len=1913700 2021-07-31T02:22:24.368 INFO:tasks.workunit.client.0.smithi045.stdout:op 5180 completed, throughput=5MB/sec 2021-07-31T02:22:24.368 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-opal_2WFGch0ekW off=8366547 len=1994427 2021-07-31T02:22:24.378 INFO:tasks.workunit.client.0.smithi045.stdout:op 5181 completed, throughput=5MB/sec 2021-07-31T02:22:24.378 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-55cr09C3wBCrWbg off=8358198 len=1417452 2021-07-31T02:22:24.379 INFO:tasks.workunit.client.0.smithi045.stdout:op 5182 completed, throughput=5MB/sec 2021-07-31T02:22:24.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:24 smithi154 conmon[34553]: cluster 2021-07-31T02:22:22.296222+0000 mgr.x (mgr.24542) 1502 : cluster [DBG] pgmap v1926: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2021-07-31T02:22:24.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:24 smithi045 conmon[33655]: cluster 2021-07-31T02:22:22.296222+0000 mgr.x (mgr.24542) 1502 2021-07-31T02:22:24.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:24 smithi045 conmon[33655]: : cluster [DBG] pgmap v1926: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2021-07-31T02:22:24.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:24 smithi045 conmon[39718]: cluster 2021-07-31T02:22: 2021-07-31T02:22:24.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:24 smithi045 conmon[39718]: 22.296222+0000 mgr.x (mgr.24542) 1502 : cluster [DBG] pgmap v1926: 9 pgs: 9 active+clean; 127 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2021-07-31T02:22:25.379 INFO:tasks.workunit.client.0.smithi045.stdout: 1018: throughput=5MB/sec pending data=0 2021-07-31T02:22:25.380 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-fn1UFioeX4eKULC off=15312282 len=1138596 2021-07-31T02:22:25.382 INFO:tasks.workunit.client.0.smithi045.stdout:op 5183 completed, throughput=5MB/sec 2021-07-31T02:22:25.383 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-K7eJaLzANAsZ5PT off=2602701 len=1829954 2021-07-31T02:22:25.384 INFO:tasks.workunit.client.0.smithi045.stdout:op 5184 completed, throughput=5MB/sec 2021-07-31T02:22:25.384 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-kY3Cbo76Xva1C1y off=10228339 len=892404 2021-07-31T02:22:25.386 INFO:tasks.workunit.client.0.smithi045.stdout:op 5185 completed, throughput=5MB/sec 2021-07-31T02:22:25.387 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-RrZiPA2bIztOylL off=18348588 len=1866314 2021-07-31T02:22:25.389 INFO:tasks.workunit.client.0.smithi045.stdout:op 5186 completed, throughput=5MB/sec 2021-07-31T02:22:26.390 INFO:tasks.workunit.client.0.smithi045.stdout: 1019: throughput=5MB/sec pending data=0 2021-07-31T02:22:26.390 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-9wMb7cimJ4NZ5JG off=7787977 len=792306 2021-07-31T02:22:26.391 INFO:tasks.workunit.client.0.smithi045.stdout:op 5187 completed, throughput=5MB/sec 2021-07-31T02:22:26.391 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Q-iIQN79B4EAk4K off=2902724 len=1389211 2021-07-31T02:22:26.394 INFO:tasks.workunit.client.0.smithi045.stdout:op 5188 completed, throughput=5MB/sec 2021-07-31T02:22:26.394 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-lZ1v8GUqsRFOZru off=8538405 len=335846 2021-07-31T02:22:26.395 INFO:tasks.workunit.client.0.smithi045.stdout:op 5189 completed, throughput=5MB/sec 2021-07-31T02:22:26.395 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-WNqjQHR9QNtpU4_ off=4088186 len=775268 2021-07-31T02:22:26.399 INFO:tasks.workunit.client.0.smithi045.stdout:op 5190 completed, throughput=5MB/sec 2021-07-31T02:22:26.400 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-R8M_nvQaKTsz7ER off=3033107 len=717578 2021-07-31T02:22:26.401 INFO:tasks.workunit.client.0.smithi045.stdout:op 5191 completed, throughput=5MB/sec 2021-07-31T02:22:26.402 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-SG0qy2OHmt54HXd off=465844 len=59324 2021-07-31T02:22:26.402 INFO:tasks.workunit.client.0.smithi045.stdout:op 5192 completed, throughput=5MB/sec 2021-07-31T02:22:26.403 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-p2WK-fHhycH_Fus off=823591 len=1109205 2021-07-31T02:22:26.404 INFO:tasks.workunit.client.0.smithi045.stdout:op 5193 completed, throughput=5MB/sec 2021-07-31T02:22:26.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:26 smithi154 conmon[34553]: cluster 2021-07-31T02:22:24.296750+0000 mgr.x (mgr.24542) 1503 : cluster 2021-07-31T02:22:26.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:26 smithi154 conmon[34553]: [DBG] pgmap v1927: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 3.4 MiB/s rd, 3.0 MiB/s wr, 6 op/s 2021-07-31T02:22:26.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:26 smithi045 conmon[39718]: cluster 2021-07-31T02:22:24.296750 2021-07-31T02:22:26.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:26 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 1503 : cluster [DBG] pgmap v1927: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 3.4 MiB/s rd, 3.0 MiB/s wr, 6 op/s 2021-07-31T02:22:26.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:26 smithi045 conmon[33655]: cluster 2021-07-31T02:22:24.296750+0000 mgr.x (mgr.24542) 1503 2021-07-31T02:22:26.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:26 smithi045 conmon[33655]: : cluster [DBG] pgmap v1927: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 3.4 MiB/s rd, 3.0 MiB/s wr, 6 op/s 2021-07-31T02:22:27.405 INFO:tasks.workunit.client.0.smithi045.stdout: 1020: throughput=5MB/sec pending data=0 2021-07-31T02:22:27.405 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-d6Rmn0klVn_SNIM off=2357064 len=1207783 2021-07-31T02:22:27.413 INFO:tasks.workunit.client.0.smithi045.stdout:op 5194 completed, throughput=5MB/sec 2021-07-31T02:22:27.414 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-YMFwKov-jVFoZZl off=6051006 len=816123 2021-07-31T02:22:27.419 INFO:tasks.workunit.client.0.smithi045.stdout:op 5195 completed, throughput=5MB/sec 2021-07-31T02:22:27.419 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-TjU2qUOxnSGD45p off=7470125 len=8834 2021-07-31T02:22:27.419 INFO:tasks.workunit.client.0.smithi045.stdout:op 5196 completed, throughput=5MB/sec 2021-07-31T02:22:27.420 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-I4EIaWf7X3lvjZO off=2168079 len=496194 2021-07-31T02:22:27.420 INFO:tasks.workunit.client.0.smithi045.stdout:op 5197 completed, throughput=5MB/sec 2021-07-31T02:22:27.421 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ZwIwLC798Qbcjd7 off=8590863 len=574867 2021-07-31T02:22:27.424 INFO:tasks.workunit.client.0.smithi045.stdout:op 5198 completed, throughput=5MB/sec 2021-07-31T02:22:27.425 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj--mqqbg3twZjo4z_ off=0 len=1947727 2021-07-31T02:22:27.435 INFO:tasks.workunit.client.0.smithi045.stdout:op 5199 completed, throughput=5MB/sec 2021-07-31T02:22:28.436 INFO:tasks.workunit.client.0.smithi045.stdout: 1021: throughput=5MB/sec pending data=0 2021-07-31T02:22:28.436 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-h2u5A6kGWChPtp6 off=12607395 len=764980 2021-07-31T02:22:28.441 INFO:tasks.workunit.client.0.smithi045.stdout:op 5200 completed, throughput=5MB/sec 2021-07-31T02:22:28.441 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-MeNV3F2JV570IsP off=19568757 len=431232 2021-07-31T02:22:28.445 INFO:tasks.workunit.client.0.smithi045.stdout:op 5201 completed, throughput=5MB/sec 2021-07-31T02:22:28.445 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-XgsinZ4luIG4SXY off=489195 len=543438 2021-07-31T02:22:28.448 INFO:tasks.workunit.client.0.smithi045.stdout:op 5202 completed, throughput=5MB/sec 2021-07-31T02:22:28.449 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-409JHJk55Sjo_dq off=6213699 len=1312432 2021-07-31T02:22:28.450 INFO:tasks.workunit.client.0.smithi045.stdout:op 5203 completed, throughput=5MB/sec 2021-07-31T02:22:28.451 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-l3w-K_Ge6V1QMlF off=6988676 len=398480 2021-07-31T02:22:28.451 INFO:tasks.workunit.client.0.smithi045.stdout:op 5204 completed, throughput=5MB/sec 2021-07-31T02:22:28.452 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-smR2g0MMQbHjgCl off=806684 len=1356905 2021-07-31T02:22:28.460 INFO:tasks.workunit.client.0.smithi045.stdout:op 5205 completed, throughput=5MB/sec 2021-07-31T02:22:28.460 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-XyY4ctSYoFc70ki off=1934819 len=1611937 2021-07-31T02:22:28.463 INFO:tasks.workunit.client.0.smithi045.stdout:op 5206 completed, throughput=5MB/sec 2021-07-31T02:22:28.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:28 smithi154 conmon[34553]: cluster 2021-07-31T02:22:26.297112+0000 mgr.x (mgr.24542) 1504 : cluster [DBG] 2021-07-31T02:22:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:28 smithi154 conmon[34553]: pgmap v1928: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.8 MiB/s wr, 3 op/s 2021-07-31T02:22:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:28 smithi154 conmon[34553]: audit 2021-07-31T02:22:28.053954+0000 mon.b (mon.2) 243 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:22:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:28 smithi154 conmon[34553]: audit 2021-07-31T02:22:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:28 smithi154 conmon[34553]: 2021-07-31T02:22:28.054636+0000 mon.a (mon.0) 1901 : audit [INF] 2021-07-31T02:22:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:28 smithi154 conmon[34553]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:22:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:28 smithi154 conmon[34553]: audit 2021-07-31T02:22: 2021-07-31T02:22:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:28 smithi154 conmon[34553]: 28.076860+0000 mon.b (mon.2) 244 : audit 2021-07-31T02:22:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:28 smithi154 conmon[34553]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:22:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:28 smithi154 conmon[34553]: audit 2021-07-31T02: 2021-07-31T02:22:28.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:28 smithi154 conmon[34553]: 22:28.077371+0000 mon.a (mon.0) 1902 : 2021-07-31T02:22:28.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:28 smithi154 conmon[34553]: audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:22:28.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:28 smithi045 conmon[39718]: cluster 2021-07-31T02:22:26 2021-07-31T02:22:28.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:28 smithi045 conmon[39718]: .297112+0000 mgr.x (mgr.24542) 1504 : cluster [DBG] pgmap v1928: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.8 MiB/s wr, 3 op/s 2021-07-31T02:22:28.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:28 smithi045 conmon[39718]: audit 2021- 2021-07-31T02:22:28.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:28 smithi045 conmon[39718]: 07-31T02:22:28.053954+0000 mon.b (mon.2 2021-07-31T02:22:28.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:28 smithi045 conmon[39718]: ) 243 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:22:28.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:28 smithi045 conmon[39718]: audit 2021-07 2021-07-31T02:22:28.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:28 smithi045 conmon[39718]: -31T02:22:28.054636+0000 mon.a (mon.0 2021-07-31T02:22:28.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:28 smithi045 conmon[39718]: ) 1901 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:22:28.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:28 smithi045 conmon[39718]: 2021-07-31T02:22:28.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:28 smithi045 conmon[39718]: audit 2021- 2021-07-31T02:22:28.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:28 smithi045 conmon[39718]: 07-31T02:22:28.076860+0000 mon.b (mon 2021-07-31T02:22:28.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:28 smithi045 conmon[39718]: .2) 244 : audit [INF] 2021-07-31T02:22:28.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:28 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:22:28.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:28 smithi045 conmon[39718]: audit 2021-07 2021-07-31T02:22:28.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:28 smithi045 conmon[39718]: -31T02:22:28.077371+0000 mon.a (mon. 2021-07-31T02:22:28.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:28 smithi045 conmon[39718]: 0) 1902 : audit [INF] 2021-07-31T02:22:28.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:28 smithi045 conmon[39718]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:22:28.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:28 smithi045 conmon[33655]: cluster 2021-07-31T02:22:26.297112+0000 mgr.x (mgr.24542) 2021-07-31T02:22:28.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:28 smithi045 conmon[33655]: 1504 : cluster [DBG] pgmap v1928: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.8 MiB/s wr, 3 op/s 2021-07-31T02:22:28.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:28 smithi045 conmon[33655]: audit 2021-07-31T02:22:28.053954+0000 mon.b (mon.2) 243 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:22:28.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:28 smithi045 conmon[33655]: audit 2021-07-31T02:22:28.054636 2021-07-31T02:22:28.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:28 smithi045 conmon[33655]: +0000 mon.a (mon.0) 1901 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:22:28.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:28 smithi045 conmon[33655]: audit 2021-07-31T02:22:28 2021-07-31T02:22:28.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:28 smithi045 conmon[33655]: .076860+0000 mon.b (mon.2) 244 : audit 2021-07-31T02:22:28.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:28 smithi045 conmon[33655]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:22:28.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:28 smithi045 conmon[33655]: audit 2021- 2021-07-31T02:22:28.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:28 smithi045 conmon[33655]: 07-31T02:22:28.077371 2021-07-31T02:22:28.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:28 smithi045 conmon[33655]: +0000 mon.a (mon.0) 1902 : 2021-07-31T02:22:28.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:28 smithi045 conmon[33655]: audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:22:29.464 INFO:tasks.workunit.client.0.smithi045.stdout: 1022: throughput=5MB/sec pending data=0 2021-07-31T02:22:29.464 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-BgE9HJTs48HaQq_ off=11209015 len=1609227 2021-07-31T02:22:29.465 INFO:tasks.workunit.client.0.smithi045.stdout:op 5207 completed, throughput=5MB/sec 2021-07-31T02:22:29.466 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-MAIokndfVZW7tFh off=5448224 len=1116537 2021-07-31T02:22:29.467 INFO:tasks.workunit.client.0.smithi045.stdout:op 5208 completed, throughput=5MB/sec 2021-07-31T02:22:29.467 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-jk0ymowo6OMxvTW off=10722621 len=2082660 2021-07-31T02:22:29.469 INFO:tasks.workunit.client.0.smithi045.stdout:op 5209 completed, throughput=5MB/sec 2021-07-31T02:22:29.626 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:22:29.901 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:22:29.902 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 37m ago 38m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:22:29.902 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 35m ago 37m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:22:29.903 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 35m ago 41m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:22:29.903 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 37m ago 43m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:22:29.903 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 37m ago 43m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:22:29.903 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 35m ago 41m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:22:29.904 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 37m ago 41m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:22:29.904 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 37m ago 38m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:22:29.904 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 35m ago 38m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:22:29.904 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 37m ago 41m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:22:29.904 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 37m ago 40m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:22:29.905 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 37m ago 40m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:22:29.905 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 37m ago 40m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:22:29.905 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 35m ago 39m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:22:29.905 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 35m ago 39m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:22:29.906 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 35m ago 39m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:22:29.906 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 35m ago 38m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:22:29.906 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 35m ago 38m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:22:30.244 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:22:30.244 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:22:30.245 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:22:30.245 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:22:30.245 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:22:30.245 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:22:30.246 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:22:30.246 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:22:30.247 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:22:30.247 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:22:30.247 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:22:30.247 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:22:30.248 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:22:30.249 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:22:30.249 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:22:30.249 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:22:30.249 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:22:30.469 INFO:tasks.workunit.client.0.smithi045.stdout: 1023: throughput=5MB/sec pending data=0 2021-07-31T02:22:30.469 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-hsxotLXNMhIMctd off=116744 len=1256150 2021-07-31T02:22:30.470 INFO:tasks.workunit.client.0.smithi045.stdout:op 5210 completed, throughput=5MB/sec 2021-07-31T02:22:30.470 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-K7eJaLzANAsZ5PT off=948850 len=1800099 2021-07-31T02:22:30.477 INFO:tasks.workunit.client.0.smithi045.stdout:op 5211 completed, throughput=5MB/sec 2021-07-31T02:22:30.477 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-QR4YOG5z1djhkX4 off=9058710 len=675128 2021-07-31T02:22:30.481 INFO:tasks.workunit.client.0.smithi045.stdout:op 5212 completed, throughput=5MB/sec 2021-07-31T02:22:30.482 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-_X-NJom1M8Pmbq4 off=15475275 len=523796 2021-07-31T02:22:30.485 INFO:tasks.workunit.client.0.smithi045.stdout:op 5213 completed, throughput=5MB/sec 2021-07-31T02:22:30.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:30 smithi154 conmon[34553]: cluster 2021-07-31T02:22:28.297462+0000 mgr.x (mgr.24542) 1505 : cluster [DBG] pgmap v1929: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.8 MiB/s wr, 3 op/s 2021-07-31T02:22:30.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:30 smithi045 conmon[33655]: cluster 2021-07-31T02:22:28.297462+0000 mgr.x ( 2021-07-31T02:22:30.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:30 smithi045 conmon[33655]: mgr.24542) 1505 : cluster [DBG] pgmap v1929: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.8 MiB/s wr, 3 op/s 2021-07-31T02:22:30.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:30 smithi045 conmon[39718]: cluster 2021-07-31T02:22:28 2021-07-31T02:22:30.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:30 smithi045 conmon[39718]: .297462+0000 mgr.x (mgr.24542) 1505 : cluster [DBG] pgmap v1929: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.8 MiB/s wr, 3 op/s 2021-07-31T02:22:31.485 INFO:tasks.workunit.client.0.smithi045.stdout: 1024: throughput=5MB/sec pending data=0 2021-07-31T02:22:31.486 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ENfEMnanxb2EZA7 off=11265958 len=1786431 2021-07-31T02:22:31.489 INFO:tasks.workunit.client.0.smithi045.stdout:op 5214 completed, throughput=5MB/sec 2021-07-31T02:22:31.489 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-1YS2cSof80dGSim off=1928344 len=1765168 2021-07-31T02:22:31.497 INFO:tasks.workunit.client.0.smithi045.stdout:op 5215 completed, throughput=5MB/sec 2021-07-31T02:22:31.497 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-lgSt7f-aMtUL1Wk off=1046065 len=1579484 2021-07-31T02:22:31.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[34553]: audit 2021-07-31T02:22:29.613667+0000 mgr.x (mgr.24542) 1506 : audit [DBG] from='client.26236 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:22:31.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[34553]: audit 2021-07-31T02:22:29.894109+0000 mgr.x (mgr.24542) 1507 : audit [DBG] from='client.16335 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:22:31.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[34553]: audit 2021-07-31T02:22:30.244206+0000 mon.c (mon.1) 93 : audit [DBG] from='client.? 172.21.15.45:0/651931954' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:22:31.504 INFO:tasks.workunit.client.0.smithi045.stdout:op 5216 completed, throughput=5MB/sec 2021-07-31T02:22:31.504 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-S6i1fkdZ_DEZ2hZ off=11886682 len=402868 2021-07-31T02:22:31.505 INFO:tasks.workunit.client.0.smithi045.stdout:op 5217 completed, throughput=5MB/sec 2021-07-31T02:22:31.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:31 smithi045 conmon[39718]: audit 2021-07-31T02:22:29.613667+0000 mgr.x (mgr 2021-07-31T02:22:31.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:31 smithi045 conmon[39718]: .24542) 1506 : audit [DBG] from='client.26236 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:22:31.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:31 smithi045 conmon[39718]: audit 2021-07-31T02:22:29.894109+0000 mgr.x (mgr.24542) 1507 : audit [DBG] 2021-07-31T02:22:31.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:31 smithi045 conmon[39718]: from='client.16335 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:22:31.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:31 smithi045 conmon[39718]: audit 2021-07-31T02:22:30.244206+0000 mon.c (mon.1 2021-07-31T02:22:31.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:31 smithi045 conmon[39718]: ) 93 : audit [DBG] from='client.? 172.21.15.45:0/651931954' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:22:31.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:31 smithi045 conmon[33655]: audit 2021-07-31T02:22:29.613667 2021-07-31T02:22:31.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:31 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 1506 : audit [DBG] from='client.26236 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:22:31.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:31 smithi045 conmon[33655]: audit 2021-07-31T02:22:29.894109+0000 mgr.x (mgr.24542) 1507 : audit [DBG] from='client.16335 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:22:31.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:31 smithi045 conmon[33655]: audit 2021-07-31T02:22:30. 2021-07-31T02:22:31.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:31 smithi045 conmon[33655]: 244206+0000 mon.c (mon.1) 93 : audit [DBG] from='client.? 172.21.15.45:0/651931954' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:22:32.003 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: debug 2021-07-31T02:22:31.565+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:22:32.003 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:22:32.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:22:32.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: yield 2021-07-31T02:22:32.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:22:32.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:22:32.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:22:32.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:22:32.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:22:32.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:22:32.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:22:32.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:22:32.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:22:32.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: task.result() 2021-07-31T02:22:32.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:22:32.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: debug 2021-07-31T02:22:31.576+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:22:32.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:22:32.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:22:32.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: yield 2021-07-31T02:22:32.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:22:32.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:22:32.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:22:32.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:22:32.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:22:32.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:22:32.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:22:32.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:22:32.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:22:32.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: task.result() 2021-07-31T02:22:32.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:22:32.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: debug 2021-07-31T02:22:31.601+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:22:32.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:22:32.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:22:32.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: yield 2021-07-31T02:22:32.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:22:32.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:22:32.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:22:32.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:22:32.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:22:32.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:22:32.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:22:32.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:22:32.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:22:32.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: task.result() 2021-07-31T02:22:32.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:22:32.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: debug 2021-07-31T02:22:31.602+0000 7fcb93961700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:22:32.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:22:32.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:22:32.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: yield 2021-07-31T02:22:32.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:22:32.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:22:32.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:22:32.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:22:32.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:22:32.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:22:32.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:22:32.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:22:32.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:22:32.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: task.result() 2021-07-31T02:22:32.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:22:32.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: 2021-07-31T02:22:32.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:22:32.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: 2021-07-31T02:22:32.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:22:32.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:22:32.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:22:32.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:22:32.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:22:32.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:22:32.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:22:32.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:22:32.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:22:32.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:22:32.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:22:32.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:22:32.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:22:32.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:22:32.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:22:32.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:22:32.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:22:32.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:22:32.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: raise 2021-07-31T02:22:32.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:22:32.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:22:32.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:22:32.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:22:32.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:22:32.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: debug 2021-07-31T02:22:31.610+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:22:32.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:22:32.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:22:32.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: yield 2021-07-31T02:22:32.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:22:32.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:22:32.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:22:32.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:22:32.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:22:32.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:22:32.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:22:32.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:22:32.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:22:32.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: task.result() 2021-07-31T02:22:32.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:22:32.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: debug 2021-07-31T02:22:31.610+0000 7fcb94162700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:22:32.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:22:32.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:22:32.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: yield 2021-07-31T02:22:32.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:22:32.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:22:32.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:22:32.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:22:32.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:22:32.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:22:32.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:22:32.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:22:32.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:22:32.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: task.result() 2021-07-31T02:22:32.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:22:32.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: 2021-07-31T02:22:32.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:22:32.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: 2021-07-31T02:22:32.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:22:32.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:22:32.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:22:32.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:22:32.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:22:32.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:22:32.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:22:32.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:22:32.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:22:32.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:22:32.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:22:32.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:22:32.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:22:32.035 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:22:32.035 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:22:32.035 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:22:32.035 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:22:32.035 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:22:32.036 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: raise 2021-07-31T02:22:32.036 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:22:32.036 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:22:32.036 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:22:32.036 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:22:32.037 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:22:31 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:22:32.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:32 smithi154 conmon[34553]: cluster 2021-07-31T02:22:30.298025+0000 mgr.x (mgr.24542) 1508 : cluster [DBG] 2021-07-31T02:22:32.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:32 smithi154 conmon[34553]: pgmap v1930: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 3.8 MiB/s rd, 2.6 MiB/s wr, 6 op/s 2021-07-31T02:22:32.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:32 smithi154 conmon[34553]: audit 2021-07-31T02:22:31.522215+0000 mon.b (mon.2) 245 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:22:32.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:32 smithi154 conmon[34553]: audit 2021-07-31T02:22:31.523482+0000 mon.b ( 2021-07-31T02:22:32.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:32 smithi154 conmon[34553]: mon.2) 246 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:22:32.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:32 smithi154 conmon[34553]: audit 2021-07-31T02:22:31.524233+0000 mon.b (mon.2) 247 2021-07-31T02:22:32.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:32 smithi154 conmon[34553]: : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:22:32.505 INFO:tasks.workunit.client.0.smithi045.stdout: 1025: throughput=5MB/sec pending data=0 2021-07-31T02:22:32.505 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-HNlFo4tKeLBfuk0 off=844807 len=1340789 2021-07-31T02:22:32.507 INFO:tasks.workunit.client.0.smithi045.stdout:op 5218 completed, throughput=5MB/sec 2021-07-31T02:22:32.507 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-OLBBCuxCLVRCGZJ off=15716386 len=1313430 2021-07-31T02:22:32.510 INFO:tasks.workunit.client.0.smithi045.stdout:op 5219 completed, throughput=5MB/sec 2021-07-31T02:22:32.510 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ayMMO8pYMwopXty off=1085515 len=1592483 2021-07-31T02:22:32.519 INFO:tasks.workunit.client.0.smithi045.stdout:op 5220 completed, throughput=5MB/sec 2021-07-31T02:22:32.520 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-PtuOIImT2uPRBLH off=22370079 len=444980 2021-07-31T02:22:32.520 INFO:tasks.workunit.client.0.smithi045.stdout:op 5221 completed, throughput=5MB/sec 2021-07-31T02:22:32.521 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-aQxoqyRA-rf9Hoi off=3622408 len=580639 2021-07-31T02:22:32.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:32 smithi045 conmon[39718]: cluster 2021-07-31T02:22:30.298025+0000 mgr.x (mgr.24542) 1508 : cluster [DBG] pgmap v1930: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 3.8 MiB/s rd, 2.6 MiB/s wr, 6 op/s 2021-07-31T02:22:32.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:32 smithi045 conmon[39718]: audit 2021-07-31T02:22:31.522215+0000 mon.b (mon.2) 245 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:22:32.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:32 smithi045 conmon[39718]: audit 2021-07-31T02:22:31.523482 2021-07-31T02:22:32.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:32 smithi045 conmon[39718]: +0000 mon.b (mon.2) 246 : audit [DBG] 2021-07-31T02:22:32.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:32 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:22:32.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:32 smithi045 conmon[39718]: audit 2021-07-31T02 2021-07-31T02:22:32.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:32 smithi045 conmon[39718]: :22:31.524233+0000 mon.b (mon. 2021-07-31T02:22:32.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:32 smithi045 conmon[39718]: 2) 247 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:22:32.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:32 smithi045 conmon[33655]: cluster 2021-07-31T02:22:30.298025+0000 2021-07-31T02:22:32.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:32 smithi045 conmon[33655]: mgr.x (mgr.24542) 1508 : cluster [DBG] pgmap v1930: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 3.8 MiB/s rd, 2.6 MiB/s wr, 6 op/s 2021-07-31T02:22:32.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:32 smithi045 conmon[33655]: audit 2021- 2021-07-31T02:22:32.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:32 smithi045 conmon[33655]: 07-31T02:22:31. 2021-07-31T02:22:32.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:32 smithi045 conmon[33655]: 522215+0000 mon.b 2021-07-31T02:22:32.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:32 smithi045 conmon[33655]: (mon.2) 245 : audit 2021-07-31T02:22:32.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:32 smithi045 conmon[33655]: [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:22:32.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:32 smithi045 conmon[33655]: audit 2021-07-31T02:22 2021-07-31T02:22:32.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:32 smithi045 conmon[33655]: :31.523482+0000 mon.b (mon.2) 2021-07-31T02:22:32.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:32 smithi045 conmon[33655]: 246 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:22:32.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:32 smithi045 conmon[33655]: audit 2021- 2021-07-31T02:22:32.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:32 smithi045 conmon[33655]: 07-31T02:22:31.524233 2021-07-31T02:22:32.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:32 smithi045 conmon[33655]: +0000 mon.b (mon.2) 2021-07-31T02:22:32.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:32 smithi045 conmon[33655]: 247 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:22:32.531 INFO:tasks.workunit.client.0.smithi045.stdout:op 5222 completed, throughput=5MB/sec 2021-07-31T02:22:33.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:22:31.524858+0000 mgr.x (mgr.24542) 1509 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:22:33.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: 2021-07-31T02:22:33.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:22:31.566485+0000 mgr.x (mgr.24542) 1510 : cephadm [ERR] 2021-07-31T02:22:33.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:22:33.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:22:33.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: yield 2021-07-31T02:22:33.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:22:33.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:22:33.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:22:33.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:22:33.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:22:33.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:22:33.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:22:33.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:22:33.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:22:33.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:22:33.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:22:33.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:22:31.566876+0000 mgr.x (mgr.24542) 1511 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:22:33.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:22:31.577695+0000 mgr.x (mgr.24542) 1512 : cephadm [ERR] 2021-07-31T02:22:33.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:22:33.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:22:33.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: yield 2021-07-31T02:22:33.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:22:33.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:22:33.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:22:33.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:22:33.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:22:33.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:22:33.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:22:33.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:22:33.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:22:33.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:22:33.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:22:33.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:22:31.578335+0000 mgr.x (mgr.24542) 1513 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:22:33.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:22:31.602632+0000 mgr.x (mgr.24542) 1514 : cephadm [ERR] 2021-07-31T02:22:33.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:22:33.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:22:33.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: yield 2021-07-31T02:22:33.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:22:33.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:22:33.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:22:33.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:22:33.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:22:33.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:22:33.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:22:33.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:22:33.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:22:33.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:22:33.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:22:33.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:22:31.603414+0000 mgr.x (mgr.24542) 1515 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:22:33.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:22:33.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:22:33.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: yield 2021-07-31T02:22:33.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:22:33.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:22:33.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:22:33.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:22:33.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:22:33.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:22:33.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:22:33.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:22:33.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:22:33.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:22:33.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:22:33.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: 2021-07-31T02:22:33.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:22:33.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: 2021-07-31T02:22:33.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:22:33.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:22:33.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:22:33.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:22:33.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:22:33.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:22:33.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:22:33.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:22:33.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:22:33.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:22:33.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:22:33.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:22:33.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:22:33.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:22:33.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:22:33.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:22:33.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:22:33.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:22:33.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: raise 2021-07-31T02:22:33.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:22:33.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:22:33.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:22:33.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:22:33.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:22:33.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:22:31.611455+0000 mgr.x (mgr.24542) 1516 : cephadm [ERR] 2021-07-31T02:22:33.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:22:33.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:22:33.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: yield 2021-07-31T02:22:33.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:22:33.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:22:33.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:22:33.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:22:33.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:22:33.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:22:33.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:22:33.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:22:33.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:22:33.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:22:33.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:22:33.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:22:31.611826+0000 mgr.x (mgr.24542) 1517 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:22:33.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:22:33.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:22:33.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: yield 2021-07-31T02:22:33.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:22:33.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:22:33.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:22:33.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:22:33.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:22:33.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:22:33.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:22:33.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:22:33.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:22:33.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:22:33.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:22:33.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: 2021-07-31T02:22:33.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:22:33.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: 2021-07-31T02:22:33.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:22:33.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:22:33.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:22:33.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:22:33.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:22:33.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:22:33.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:22:33.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:22:33.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:22:33.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:22:33.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:22:33.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:22:33.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:22:33.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:22:33.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:22:33.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:22:33.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:22:33.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:22:33.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: raise 2021-07-31T02:22:33.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:22:33.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:22:33.541 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:22:33.541 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:22:33.541 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:33 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:22:33.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:22:31.524858+0000 mgr.x 2021-07-31T02:22:33.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: (mgr.24542) 1509 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:22:33.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: cephadm 2021-07-31T02 2021-07-31T02:22:33.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: :22:31.566485+0000 mgr.x (mgr.24542) 1510 : cephadm [ERR] 2021-07-31T02:22:33.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:22:33.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:22:33.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: yield 2021-07-31T02:22:33.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:22:33.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:22:33.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:22:33.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:22:33.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:22:33.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:22:33.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:22:33.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:22:33.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:22:33.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:22:33.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:22:33.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:22:31.566876+0000 mgr.x (mgr.24542) 1511 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:22:33.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:22:31.577695+0000 mgr.x (mgr.24542) 1512 : cephadm [ERR] 2021-07-31T02:22:33.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:22:33.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:22:33.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: yield 2021-07-31T02:22:33.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:22:33.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:22:33.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:22:33.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:22:33.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:22:33.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:22:33.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:22:33.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:22:33.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:22:33.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:22:33.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:22:33.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:22:31.578335+0000 mgr.x (mgr.24542) 1513 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:22:33.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:22:31.602632+0000 mgr.x (mgr.24542) 1514 : cephadm [ERR] 2021-07-31T02:22:33.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:22:33.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:22:33.552 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:22:31.524858+0000 mgr.x (mgr.24542 2021-07-31T02:22:33.553 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: ) 1509 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:22:33.553 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:22:31.566485+0000 mgr.x (mgr.24542) 1510 : cephadm [ERR] 2021-07-31T02:22:33.554 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:22:33.554 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:22:33.555 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: yield 2021-07-31T02:22:33.555 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:22:33.555 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:22:33.555 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:22:33.556 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:22:33.556 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:22:33.556 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:22:33.556 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:22:33.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:22:33.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:22:33.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:22:33.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:22:33.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:22:33.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: 2021-07-31T02:22:31.566876+0000 mgr.x (mgr.24542) 1511 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:22:33.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:22:31.577695+0000 mgr.x (mgr.24542) 1512 : cephadm [ERR] 2021-07-31T02:22:33.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:22:33.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:22:33.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: yield 2021-07-31T02:22:33.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:22:33.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:22:33.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:22:33.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:22:33.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:22:33.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:22:33.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:22:33.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:22:33.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:22:33.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:22:33.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:22:33.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:22:31.578335+0000 mgr.x (mgr.24542) 1513 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:22:33.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: 2021-07-31T02:22:33.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: cephadm 2021-07- 2021-07-31T02:22:33.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: 31T02:22:31.602632+0000 2021-07-31T02:22:33.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: mgr.x (mgr.24542) 1514 : 2021-07-31T02:22:33.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: cephadm [ERR] 2021-07-31T02:22:33.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:22:33.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:22:33.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: yield 2021-07-31T02:22:33.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:22:33.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:22:33.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:22:33.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:22:33.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:22:33.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:22:33.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:22:33.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:22:33.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:22:33.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:22:33.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:22:33.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:22:31.603414+0000 mgr.x (mgr.24542) 1515 : 2021-07-31T02:22:33.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:22:33.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:22:33.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:22:33.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: yield 2021-07-31T02:22:33.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:22:33.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:22:33.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:22:33.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:22:33.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:22:33.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:22:33.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:22:33.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:22:33.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:22:33.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:22:33.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:22:33.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: 2021-07-31T02:22:33.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:22:33.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: 2021-07-31T02:22:33.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:22:33.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:22:33.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:22:33.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:22:33.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:22:33.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:22:33.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:22:33.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:22:33.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: yield 2021-07-31T02:22:33.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:22:33.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:22:33.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:22:33.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:22:33.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:22:33.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:22:33.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:22:33.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:22:33.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:22:33.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:22:33.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:22:33.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:22:33.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: 2021-07-31T02:22:31.603414+0000 mgr.x (mgr.24542) 1515 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:22:33.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:22:33.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:22:33.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: yield 2021-07-31T02:22:33.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:22:33.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:22:33.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:22:33.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:22:33.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:22:33.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:22:33.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:22:33.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:22:33.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:22:33.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:22:33.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:22:33.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: 2021-07-31T02:22:33.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:22:33.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: 2021-07-31T02:22:33.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:22:33.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:22:33.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:22:33.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:22:33.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:22:33.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:22:33.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:22:33.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:22:33.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:22:33.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:22:33.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:22:33.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:22:33.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:22:33.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:22:33.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:22:33.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:22:33.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:22:33.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:22:33.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:22:33.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:22:33.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: raise 2021-07-31T02:22:33.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:22:33.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:22:33.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:22:33.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:22:33.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:22:33.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:22:31.611455+0000 mgr.x (mgr.24542) 1516 : cephadm [ERR] 2021-07-31T02:22:33.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:22:33.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:22:33.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: yield 2021-07-31T02:22:33.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:22:33.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:22:33.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:22:33.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:22:33.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:22:33.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:22:33.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:22:33.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:22:33.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:22:33.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:22:33.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:22:33.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:22:31.611826+0000 mgr.x (mgr.24542) 1517 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:22:33.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:22:33.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:22:33.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: yield 2021-07-31T02:22:33.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:22:33.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:22:33.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:22:33.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:22:33.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:22:33.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:22:33.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:22:33.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:22:33.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:22:33.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:22:33.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:22:33.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:22:33.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: raise 2021-07-31T02:22:33.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:22:33.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:22:33.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:22:33.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:22:33.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:22:33.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:22:31.611455+0000 mgr.x (mgr.24542) 1516 : cephadm [ERR] 2021-07-31T02:22:33.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:22:33.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:22:33.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: yield 2021-07-31T02:22:33.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:22:33.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:22:33.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:22:33.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:22:33.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:22:33.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:22:33.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:22:33.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:22:33.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:22:33.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:22:33.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:22:33.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: cephadm 2021-07-31T 2021-07-31T02:22:33.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: 02:22:31.611826+0000 mgr.x (mgr.24542) 1517 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:22:33.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:22:33.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:22:33.603 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: yield 2021-07-31T02:22:33.603 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:22:33.603 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:22:33.603 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:22:33.603 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:22:33.604 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:22:33.604 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:22:33.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:22:33.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:22:33.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:22:33.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:22:33.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:22:33.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:22:33.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:22:33.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: 2021-07-31T02:22:33.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:22:33.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: 2021-07-31T02:22:33.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:22:33.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:22:33.608 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:22:33.608 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:22:33.608 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:22:33.608 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:22:33.609 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:22:33.609 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:22:33.609 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:22:33.609 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:22:33.610 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:22:33.610 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:22:33.610 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:22:33.610 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:22:33.611 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:22:33.611 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:22:33.611 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:22:33.611 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:22:33.611 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: raise 2021-07-31T02:22:33.612 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:22:33.612 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:22:33.612 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:22:33.612 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:22:33.613 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:22:33.613 INFO:tasks.workunit.client.0.smithi045.stdout: 1026: throughput=5MB/sec pending data=0 2021-07-31T02:22:33.613 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-DTOmcK_OINhY09H off=2451763 len=1981353 2021-07-31T02:22:33.614 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:22:33.614 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:22:33.614 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:22:33.615 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:22:33.615 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:22:33.615 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: 2021-07-31T02:22:33.615 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:22:33.616 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: 2021-07-31T02:22:33.616 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:22:33.616 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:22:33.616 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:22:33.617 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:22:33.617 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:22:33.617 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:22:33.617 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:22:33.618 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:22:33.618 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:22:33.618 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:22:33.618 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:22:33.618 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:22:33.619 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:22:33.619 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:22:33.619 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:22:33.619 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:22:33.620 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:22:33.620 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:22:33.620 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: raise 2021-07-31T02:22:33.620 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:22:33.621 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:22:33.621 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:22:33.621 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:22:33.621 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:22:33.622 INFO:tasks.workunit.client.0.smithi045.stdout:op 5223 completed, throughput=5MB/sec 2021-07-31T02:22:33.622 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-1T-quRFnzDxcsHv off=4947078 len=1520244 2021-07-31T02:22:33.623 INFO:tasks.workunit.client.0.smithi045.stdout:op 5224 completed, throughput=5MB/sec 2021-07-31T02:22:33.623 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-xwemAWbX8kFS0ST off=1355008 len=627045 2021-07-31T02:22:33.623 INFO:tasks.workunit.client.0.smithi045.stdout:op 5225 completed, throughput=5MB/sec 2021-07-31T02:22:33.623 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Bpm547gP1ZFgCxw off=1776519 len=871011 2021-07-31T02:22:33.624 INFO:tasks.workunit.client.0.smithi045.stdout:op 5226 completed, throughput=5MB/sec 2021-07-31T02:22:33.624 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-K-Xr66VDnAPud3x off=15846458 len=1389928 2021-07-31T02:22:33.624 INFO:tasks.workunit.client.0.smithi045.stdout:op 5227 completed, throughput=5MB/sec 2021-07-31T02:22:34.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:22:33.570Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:22:34.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:22:33.570Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:22:34.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:22:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:22:33.570Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:22:34.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:34 smithi154 conmon[34553]: cluster 2021-07-31T02:22:32.298422+0000 mgr.x (mgr.24542) 1518 : cluster [DBG] pgmap v1931: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 2.6 MiB/s rd, 1.8 MiB/s wr, 4 op/s 2021-07-31T02:22:34.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:34 smithi154 conmon[34553]: 2021-07-31T02:22:34.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:34 smithi045 conmon[39718]: cluster 2021-07-31T02:22:32.298422+0000 mgr.x (mgr 2021-07-31T02:22:34.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:34 smithi045 conmon[39718]: .24542) 1518 : cluster [DBG] pgmap v1931: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 2.6 MiB/s rd, 1.8 MiB/s wr, 4 op/s 2021-07-31T02:22:34.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:34 smithi045 conmon[33655]: cluster 2021-07-31T02:22:32.298422+0000 mgr.x ( 2021-07-31T02:22:34.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:34 smithi045 conmon[33655]: mgr.24542) 1518 : cluster [DBG] pgmap v1931: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 2.6 MiB/s rd, 1.8 MiB/s wr, 4 op/s 2021-07-31T02:22:34.557 INFO:tasks.workunit.client.0.smithi045.stdout: 1027: throughput=5MB/sec pending data=0 2021-07-31T02:22:34.557 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Y6SYA4Wdwm-HkKW off=3579066 len=466525 2021-07-31T02:22:34.558 INFO:tasks.workunit.client.0.smithi045.stdout:op 5228 completed, throughput=5MB/sec 2021-07-31T02:22:34.558 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-VctPA5CU15_qpTi off=14557480 len=518703 2021-07-31T02:22:34.559 INFO:tasks.workunit.client.0.smithi045.stdout:op 5229 completed, throughput=5MB/sec 2021-07-31T02:22:34.559 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-TQ9XufmQDrHPjcb off=15775382 len=2062089 2021-07-31T02:22:34.561 INFO:tasks.workunit.client.0.smithi045.stdout:op 5230 completed, throughput=5MB/sec 2021-07-31T02:22:34.561 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-BIfGs38Muh9TzAa off=3963238 len=818604 2021-07-31T02:22:34.563 INFO:tasks.workunit.client.0.smithi045.stdout:op 5231 completed, throughput=5MB/sec 2021-07-31T02:22:34.564 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-W2oQvAUTGNfGq2c off=2866266 len=640659 2021-07-31T02:22:34.564 INFO:tasks.workunit.client.0.smithi045.stdout:op 5232 completed, throughput=5MB/sec 2021-07-31T02:22:35.565 INFO:tasks.workunit.client.0.smithi045.stdout: 1028: throughput=5MB/sec pending data=0 2021-07-31T02:22:35.566 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-NSf5pECiPc8wkBl off=16140234 len=1582824 2021-07-31T02:22:35.567 INFO:tasks.workunit.client.0.smithi045.stdout:op 5233 completed, throughput=5MB/sec 2021-07-31T02:22:35.567 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-hXflMp-avCwKJiH off=19682881 len=567496 2021-07-31T02:22:35.569 INFO:tasks.workunit.client.0.smithi045.stdout:op 5234 completed, throughput=5MB/sec 2021-07-31T02:22:35.569 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ti2YK0OeaXLftF5 off=4551335 len=2007991 2021-07-31T02:22:35.572 INFO:tasks.workunit.client.0.smithi045.stdout:op 5235 completed, throughput=5MB/sec 2021-07-31T02:22:35.572 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-icBtYEDLS9mBCbd off=5239901 len=1336217 2021-07-31T02:22:35.578 INFO:tasks.workunit.client.0.smithi045.stdout:op 5236 completed, throughput=5MB/sec 2021-07-31T02:22:36.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:36 smithi154 conmon[34553]: cluster 2021-07-31T02:22:34.299016+0000 mgr.x (mgr.24542) 1519 : cluster [DBG] pgmap v1932: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 3.6 MiB/s rd, 2.8 MiB/s wr, 6 op/s 2021-07-31T02:22:36.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:36 smithi045 conmon[33655]: cluster 2021-07-31T02:22:34.299016+0000 mgr.x (mgr.24542) 1519 : cluster [DBG] pgmap v1932: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 3.6 MiB/s rd, 2.8 MiB/s wr, 6 op/s 2021-07-31T02:22:36.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:36 smithi045 conmon[39718]: cluster 2021-07-31T02:22:34 2021-07-31T02:22:36.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:36 smithi045 conmon[39718]: .299016+0000 mgr.x (mgr.24542) 1519 : cluster [DBG] pgmap v1932: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 3.6 MiB/s rd, 2.8 MiB/s wr, 6 op/s 2021-07-31T02:22:36.579 INFO:tasks.workunit.client.0.smithi045.stdout: 1029: throughput=5MB/sec pending data=0 2021-07-31T02:22:36.579 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-VbPVLygsjGAiJI4 off=6121552 len=2060118 2021-07-31T02:22:36.589 INFO:tasks.workunit.client.0.smithi045.stdout:op 5237 completed, throughput=5MB/sec 2021-07-31T02:22:36.589 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-rTGIhq1EfbiWTE0 off=243465 len=877597 2021-07-31T02:22:36.591 INFO:tasks.workunit.client.0.smithi045.stdout:op 5238 completed, throughput=5MB/sec 2021-07-31T02:22:36.591 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-RFY5lZv68_rAKUR off=2446009 len=340582 2021-07-31T02:22:36.592 INFO:tasks.workunit.client.0.smithi045.stdout:op 5239 completed, throughput=5MB/sec 2021-07-31T02:22:36.592 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-jrrX3WLV8QPYkDr off=1234903 len=1280954 2021-07-31T02:22:36.595 INFO:tasks.workunit.client.0.smithi045.stdout:op 5240 completed, throughput=5MB/sec 2021-07-31T02:22:36.595 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-QYjusKGejy9Mw8M off=11204267 len=290337 2021-07-31T02:22:36.596 INFO:tasks.workunit.client.0.smithi045.stdout:op 5241 completed, throughput=5MB/sec 2021-07-31T02:22:37.596 INFO:tasks.workunit.client.0.smithi045.stdout: 1030: throughput=5MB/sec pending data=0 2021-07-31T02:22:37.597 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-U8uKjXBh_yWtJFJ off=8557708 len=106045 2021-07-31T02:22:37.601 INFO:tasks.workunit.client.0.smithi045.stdout:op 5242 completed, throughput=5MB/sec 2021-07-31T02:22:37.601 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-NCXLU_sUAm8sZWl off=1627906 len=1036974 2021-07-31T02:22:37.605 INFO:tasks.workunit.client.0.smithi045.stdout:op 5243 completed, throughput=5MB/sec 2021-07-31T02:22:37.605 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-3RcbjaogV24Z58b off=2662068 len=202751 2021-07-31T02:22:37.606 INFO:tasks.workunit.client.0.smithi045.stdout:op 5244 completed, throughput=5MB/sec 2021-07-31T02:22:37.606 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-lTEJeTEYiOUPt8g off=2232284 len=1649741 2021-07-31T02:22:37.613 INFO:tasks.workunit.client.0.smithi045.stdout:op 5245 completed, throughput=5MB/sec 2021-07-31T02:22:37.613 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-U5vlDVTkJJQn9Qv off=12136745 len=9838 2021-07-31T02:22:37.614 INFO:tasks.workunit.client.0.smithi045.stdout:op 5246 completed, throughput=5MB/sec 2021-07-31T02:22:37.614 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-pK54vbouFrrqDOY off=14473483 len=1534880 2021-07-31T02:22:37.621 INFO:tasks.workunit.client.0.smithi045.stdout:op 5247 completed, throughput=5MB/sec 2021-07-31T02:22:37.621 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-WAWikBp8vyCTksD off=24713585 len=735840 2021-07-31T02:22:37.623 INFO:tasks.workunit.client.0.smithi045.stdout:op 5248 completed, throughput=5MB/sec 2021-07-31T02:22:37.624 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-32qWWZz7yX9n7H2 off=76619 len=1668583 2021-07-31T02:22:37.625 INFO:tasks.workunit.client.0.smithi045.stdout:op 5249 completed, throughput=5MB/sec 2021-07-31T02:22:38.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:38 smithi154 conmon[34553]: cluster 2021-07-31T02:22:36.299401+0000 mgr.x (mgr.24542) 1520 : cluster [DBG] pgmap v1933: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 2.4 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2021-07-31T02:22:38.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:38 smithi045 conmon[39718]: cluster 2021-07-31T02:22:36.299401+0000 2021-07-31T02:22:38.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:38 smithi045 conmon[39718]: mgr.x (mgr.24542) 1520 : cluster [DBG] pgmap v1933: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 2.4 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2021-07-31T02:22:38.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:38 smithi045 conmon[33655]: cluster 2021-07-31T02:22:36.299401+0000 mgr.x (mgr.24542 2021-07-31T02:22:38.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:38 smithi045 conmon[33655]: ) 1520 : cluster [DBG] pgmap v1933: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 2.4 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2021-07-31T02:22:38.625 INFO:tasks.workunit.client.0.smithi045.stdout: 1031: throughput=5MB/sec pending data=0 2021-07-31T02:22:38.626 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-exN3tEkKPH-A7rF off=3484478 len=277699 2021-07-31T02:22:38.627 INFO:tasks.workunit.client.0.smithi045.stdout:op 5250 completed, throughput=5MB/sec 2021-07-31T02:22:38.627 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-j1VmyoHWp2jdEAL off=18354514 len=1497962 2021-07-31T02:22:38.629 INFO:tasks.workunit.client.0.smithi045.stdout:op 5251 completed, throughput=5MB/sec 2021-07-31T02:22:38.629 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-NW0atou29hOh8Lv off=592040 len=1920373 2021-07-31T02:22:38.633 INFO:tasks.workunit.client.0.smithi045.stdout:op 5252 completed, throughput=5MB/sec 2021-07-31T02:22:38.633 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-34F3QEu8Dy42b2z off=6138930 len=1875308 2021-07-31T02:22:38.641 INFO:tasks.workunit.client.0.smithi045.stdout:op 5253 completed, throughput=5MB/sec 2021-07-31T02:22:39.641 INFO:tasks.workunit.client.0.smithi045.stdout: 1032: throughput=5MB/sec pending data=0 2021-07-31T02:22:39.642 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-UdGlRtbpLiukAY7 off=1807226 len=1934872 2021-07-31T02:22:39.650 INFO:tasks.workunit.client.0.smithi045.stdout:op 5254 completed, throughput=5MB/sec 2021-07-31T02:22:39.650 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-tflTZLF9Lz6_ADL off=11205368 len=1806768 2021-07-31T02:22:39.657 INFO:tasks.workunit.client.0.smithi045.stdout:op 5255 completed, throughput=5MB/sec 2021-07-31T02:22:40.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:40 smithi045 conmon[33655]: cluster 2021-07-31T02:22:38.299803+0000 mgr.x (mgr.24542 2021-07-31T02:22:40.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:40 smithi045 conmon[33655]: ) 1521 : cluster [DBG] pgmap v1934: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 2.4 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2021-07-31T02:22:40.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:40 smithi045 conmon[39718]: cluster 2021-07-31T02:22:38.299803+0000 mgr.x 2021-07-31T02:22:40.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:40 smithi045 conmon[39718]: (mgr.24542) 1521 : cluster [DBG] pgmap v1934: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 2.4 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2021-07-31T02:22:40.658 INFO:tasks.workunit.client.0.smithi045.stdout: 1033: throughput=5MB/sec pending data=0 2021-07-31T02:22:40.658 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-cr1UGEzkvKGOHNW off=1796381 len=110787 2021-07-31T02:22:40.659 INFO:tasks.workunit.client.0.smithi045.stdout:op 5256 completed, throughput=5MB/sec 2021-07-31T02:22:40.659 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-nb4BirArwy9Ctsb off=18491678 len=539060 2021-07-31T02:22:40.660 INFO:tasks.workunit.client.0.smithi045.stdout:op 5257 completed, throughput=5MB/sec 2021-07-31T02:22:40.661 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-1iHtBF0-MhXsPxU off=1545418 len=740242 2021-07-31T02:22:40.662 INFO:tasks.workunit.client.0.smithi045.stdout:op 5258 completed, throughput=5MB/sec 2021-07-31T02:22:40.662 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ih8ReNQZEHHCErj off=3267997 len=51011 2021-07-31T02:22:40.663 INFO:tasks.workunit.client.0.smithi045.stdout:op 5259 completed, throughput=5MB/sec 2021-07-31T02:22:40.663 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-QwGTq0miJ-G5WvK off=15310292 len=1930712 2021-07-31T02:22:40.666 INFO:tasks.workunit.client.0.smithi045.stdout:op 5260 completed, throughput=5MB/sec 2021-07-31T02:22:40.666 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-iAyJ1ESL4Bc32Xo off=10679249 len=1072681 2021-07-31T02:22:40.672 INFO:tasks.workunit.client.0.smithi045.stdout:op 5261 completed, throughput=5MB/sec 2021-07-31T02:22:40.672 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-L1T9zZGhgNB9d73 off=2918881 len=2051106 2021-07-31T02:22:40.680 INFO:tasks.workunit.client.0.smithi045.stdout:op 5262 completed, throughput=5MB/sec 2021-07-31T02:22:40.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:40 smithi154 conmon[34553]: cluster 2021-07-31T02:22:38.299803+0000 mgr.x (mgr.24542) 1521 : cluster [DBG] pgmap v1934: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 2.4 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2021-07-31T02:22:41.680 INFO:tasks.workunit.client.0.smithi045.stdout: 1034: throughput=5MB/sec pending data=0 2021-07-31T02:22:41.680 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-1y-yS1pqDjCB-2x off=605650 len=828545 2021-07-31T02:22:41.685 INFO:tasks.workunit.client.0.smithi045.stdout:op 5263 completed, throughput=5MB/sec 2021-07-31T02:22:41.685 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-iT1c1tsgIvoccHD off=8936555 len=1232382 2021-07-31T02:22:41.692 INFO:tasks.workunit.client.0.smithi045.stdout:op 5264 completed, throughput=5MB/sec 2021-07-31T02:22:41.692 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-0k0qPLQEmZtdlJW off=10971968 len=483275 2021-07-31T02:22:41.693 INFO:tasks.workunit.client.0.smithi045.stdout:op 5265 completed, throughput=5MB/sec 2021-07-31T02:22:41.693 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-iWHV3ngz-mtvpVj off=5596559 len=441055 2021-07-31T02:22:41.696 INFO:tasks.workunit.client.0.smithi045.stdout:op 5266 completed, throughput=5MB/sec 2021-07-31T02:22:41.696 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-n5RLEeqd8PA6rz- off=535464 len=468083 2021-07-31T02:22:41.697 INFO:tasks.workunit.client.0.smithi045.stdout:op 5267 completed, throughput=5MB/sec 2021-07-31T02:22:41.698 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ev_glmWNKn0wBPP off=10738864 len=1638688 2021-07-31T02:22:41.700 INFO:tasks.workunit.client.0.smithi045.stdout:op 5268 completed, throughput=5MB/sec 2021-07-31T02:22:42.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:42 smithi045 conmon[33655]: cluster 2021-07-31T02:22:40.300281+0000 mgr.x (mgr.24542) 1522 : cluster 2021-07-31T02:22:42.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:42 smithi045 conmon[33655]: [DBG] pgmap v1935: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 3.8 MiB/s rd, 2.6 MiB/s wr, 6 op/s 2021-07-31T02:22:42.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:42 smithi045 conmon[39718]: cluster 2021-07-31T02:22:40.300281+0000 mgr.x (mgr 2021-07-31T02:22:42.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:42 smithi045 conmon[39718]: .24542) 1522 : cluster [DBG] pgmap v1935: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 3.8 MiB/s rd, 2.6 MiB/s wr, 6 op/s 2021-07-31T02:22:42.702 INFO:tasks.workunit.client.0.smithi045.stdout: 1035: throughput=5MB/sec pending data=0 2021-07-31T02:22:42.702 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-2R-FBxaEWFFLLii off=10297661 len=1792778 2021-07-31T02:22:42.704 INFO:tasks.workunit.client.0.smithi045.stdout:op 5269 completed, throughput=5MB/sec 2021-07-31T02:22:42.704 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-VLU19E0lRr8MgiU off=12815621 len=372988 2021-07-31T02:22:42.705 INFO:tasks.workunit.client.0.smithi045.stdout:op 5270 completed, throughput=5MB/sec 2021-07-31T02:22:42.706 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-SwAY_4e31BGE35L off=13581926 len=517846 2021-07-31T02:22:42.707 INFO:tasks.workunit.client.0.smithi045.stdout:op 5271 completed, throughput=5MB/sec 2021-07-31T02:22:42.707 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-0X3XfDjw5xmVXsT off=0 len=1406122 2021-07-31T02:22:42.709 INFO:tasks.workunit.client.0.smithi045.stdout:op 5272 completed, throughput=5MB/sec 2021-07-31T02:22:42.710 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-CBp1Vipf4x29K9k off=12187416 len=1179416 2021-07-31T02:22:42.711 INFO:tasks.workunit.client.0.smithi045.stdout:op 5273 completed, throughput=5MB/sec 2021-07-31T02:22:42.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:42 smithi154 conmon[34553]: cluster 2021-07-31T02:22:40.300281+0000 mgr.x (mgr.24542) 1522 : cluster [DBG] pgmap v1935: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 3.8 MiB/s rd, 2.6 MiB/s wr, 6 op/s 2021-07-31T02:22:43.712 INFO:tasks.workunit.client.0.smithi045.stdout: 1036: throughput=5MB/sec pending data=0 2021-07-31T02:22:43.712 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-h1OklJ_1oZ9BElH off=8706996 len=104838 2021-07-31T02:22:43.713 INFO:tasks.workunit.client.0.smithi045.stdout:op 5274 completed, throughput=5MB/sec 2021-07-31T02:22:43.714 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-dFd8Vb4OhFD4j39 off=9334879 len=1249511 2021-07-31T02:22:43.716 INFO:tasks.workunit.client.0.smithi045.stdout:op 5275 completed, throughput=5MB/sec 2021-07-31T02:22:43.716 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ffMeTG4lwlKRp6f off=8208048 len=375052 2021-07-31T02:22:43.720 INFO:tasks.workunit.client.0.smithi045.stdout:op 5276 completed, throughput=5MB/sec 2021-07-31T02:22:43.720 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-CsfT54kQP1AMQsU off=15861984 len=701256 2021-07-31T02:22:43.722 INFO:tasks.workunit.client.0.smithi045.stdout:op 5277 completed, throughput=5MB/sec 2021-07-31T02:22:43.722 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-AcVvC9jZ6E_8dUK off=16789797 len=1659435 2021-07-31T02:22:43.729 INFO:tasks.workunit.client.0.smithi045.stdout:op 5278 completed, throughput=5MB/sec 2021-07-31T02:22:43.730 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-QIuzvWVEemQ6Glb off=6895878 len=2090744 2021-07-31T02:22:43.738 INFO:tasks.workunit.client.0.smithi045.stdout:op 5279 completed, throughput=5MB/sec 2021-07-31T02:22:44.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:22:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:22:43.570Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:22:44.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:22:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:22:43.570Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:22:44.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:22:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:22:43.570Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:22:44.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:44 smithi045 conmon[39718]: cluster 2021-07-31T02:22:42.300830+0000 mgr.x 2021-07-31T02:22:44.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:44 smithi045 conmon[39718]: (mgr.24542) 1523 : cluster [DBG] pgmap v1936: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.8 MiB/s wr, 3 op/s 2021-07-31T02:22:44.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:44 smithi045 conmon[33655]: cluster 2021-07-31T02:22:42 2021-07-31T02:22:44.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:44 smithi045 conmon[33655]: .300830+0000 mgr.x (mgr.24542) 1523 : cluster [DBG] pgmap v1936: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.8 MiB/s wr, 3 op/s 2021-07-31T02:22:44.739 INFO:tasks.workunit.client.0.smithi045.stdout: 1037: throughput=5MB/sec pending data=0 2021-07-31T02:22:44.739 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-owJhpxcHSR1eyyc off=3497682 len=932669 2021-07-31T02:22:44.740 INFO:tasks.workunit.client.0.smithi045.stdout:op 5280 completed, throughput=5MB/sec 2021-07-31T02:22:44.740 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-uuxAXEF2hQ9OIeR off=3480645 len=677107 2021-07-31T02:22:44.745 INFO:tasks.workunit.client.0.smithi045.stdout:op 5281 completed, throughput=5MB/sec 2021-07-31T02:22:44.745 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-gA2zqHYgXfHHOIk off=14661612 len=826766 2021-07-31T02:22:44.747 INFO:tasks.workunit.client.0.smithi045.stdout:op 5282 completed, throughput=5MB/sec 2021-07-31T02:22:44.747 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ud3mXbnTOLu96jh off=948812 len=1866365 2021-07-31T02:22:44.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:44 smithi154 conmon[34553]: cluster 2021-07-31T02:22:42.300830+0000 mgr.x (mgr.24542) 1523 : cluster [DBG] pgmap v1936: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.8 MiB/s wr, 3 op/s 2021-07-31T02:22:44.757 INFO:tasks.workunit.client.0.smithi045.stdout:op 5283 completed, throughput=5MB/sec 2021-07-31T02:22:45.757 INFO:tasks.workunit.client.0.smithi045.stdout: 1038: throughput=5MB/sec pending data=0 2021-07-31T02:22:45.758 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-87c7EKm6GA6mDX9 off=4285987 len=28372 2021-07-31T02:22:45.758 INFO:tasks.workunit.client.0.smithi045.stdout:op 5284 completed, throughput=5MB/sec 2021-07-31T02:22:45.758 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ybPE41z7O2l7IVy off=7012623 len=432935 2021-07-31T02:22:45.761 INFO:tasks.workunit.client.0.smithi045.stdout:op 5285 completed, throughput=5MB/sec 2021-07-31T02:22:45.762 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-yCESK1_jKhoP3_x off=15917047 len=1578472 2021-07-31T02:22:45.769 INFO:tasks.workunit.client.0.smithi045.stdout:op 5286 completed, throughput=5MB/sec 2021-07-31T02:22:45.769 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ijEaneD053rNNAn off=8138189 len=2068272 2021-07-31T02:22:45.771 INFO:tasks.workunit.client.0.smithi045.stdout:op 5287 completed, throughput=5MB/sec 2021-07-31T02:22:45.772 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-RqMMnU0dMHTf9SF off=8057926 len=758633 2021-07-31T02:22:45.777 INFO:tasks.workunit.client.0.smithi045.stdout:op 5288 completed, throughput=5MB/sec 2021-07-31T02:22:46.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:46 smithi045 conmon[33655]: cluster 2021-07-31T02:22:44.301274+0000 mgr.x (mgr.24542) 2021-07-31T02:22:46.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:46 smithi045 conmon[33655]: 1524 : cluster [DBG] pgmap v1937: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 3.5 MiB/s rd, 2.9 MiB/s wr, 6 op/s 2021-07-31T02:22:46.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:46 smithi045 conmon[39718]: cluster 2021-07-31T02:22:44.301274+0000 mgr.x ( 2021-07-31T02:22:46.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:46 smithi045 conmon[39718]: mgr.24542) 1524 : cluster [DBG] pgmap v1937: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 3.5 MiB/s rd, 2.9 MiB/s wr, 6 op/s 2021-07-31T02:22:46.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:46 smithi154 conmon[34553]: cluster 2021-07-31T02:22:44.301274+0000 mgr.x (mgr.24542) 1524 : cluster [DBG] pgmap v1937: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 3.5 MiB/s rd, 2.9 MiB/s wr, 6 op/s 2021-07-31T02:22:46.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:46 smithi154 conmon[34553]: 2021-07-31T02:22:46.777 INFO:tasks.workunit.client.0.smithi045.stdout: 1039: throughput=5MB/sec pending data=0 2021-07-31T02:22:46.778 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Ek-oH3q1P0Wz8wL off=3085710 len=229636 2021-07-31T02:22:46.778 INFO:tasks.workunit.client.0.smithi045.stdout:op 5289 completed, throughput=5MB/sec 2021-07-31T02:22:46.778 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-0k0CUrjCirWzdPE off=1041254 len=254716 2021-07-31T02:22:46.779 INFO:tasks.workunit.client.0.smithi045.stdout:op 5290 completed, throughput=5MB/sec 2021-07-31T02:22:46.780 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-HH_YbDtxBp1PyK1 off=966217 len=250349 2021-07-31T02:22:46.780 INFO:tasks.workunit.client.0.smithi045.stdout:op 5291 completed, throughput=5MB/sec 2021-07-31T02:22:46.780 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-mEM3vbd1GnOnao0 off=12268402 len=1310385 2021-07-31T02:22:46.783 INFO:tasks.workunit.client.0.smithi045.stdout:op 5292 completed, throughput=5MB/sec 2021-07-31T02:22:46.783 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Ev0twc2-GlWMIbs off=322578 len=1401924 2021-07-31T02:22:46.784 INFO:tasks.workunit.client.0.smithi045.stdout:op 5293 completed, throughput=5MB/sec 2021-07-31T02:22:46.784 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-kurNzp0UWpObcdb off=4807205 len=960937 2021-07-31T02:22:46.787 INFO:tasks.workunit.client.0.smithi045.stdout:op 5294 completed, throughput=5MB/sec 2021-07-31T02:22:46.787 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Za9e3ZD80gBACuA off=9212738 len=31150 2021-07-31T02:22:46.789 INFO:tasks.workunit.client.0.smithi045.stdout:op 5295 completed, throughput=5MB/sec 2021-07-31T02:22:46.789 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-LnlC9PDpo7onf54 off=18232851 len=1320680 2021-07-31T02:22:46.796 INFO:tasks.workunit.client.0.smithi045.stdout:op 5296 completed, throughput=5MB/sec 2021-07-31T02:22:47.796 INFO:tasks.workunit.client.0.smithi045.stdout: 1040: throughput=5MB/sec pending data=0 2021-07-31T02:22:47.796 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-1nmK124TLQUOX4h off=6464225 len=2078024 2021-07-31T02:22:47.800 INFO:tasks.workunit.client.0.smithi045.stdout:op 5297 completed, throughput=5MB/sec 2021-07-31T02:22:47.800 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-XGJSRSA1VPCt93f off=8819161 len=1173557 2021-07-31T02:22:47.802 INFO:tasks.workunit.client.0.smithi045.stdout:op 5298 completed, throughput=5MB/sec 2021-07-31T02:22:47.803 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-s1Wp4s13Es5IFCF off=13774308 len=664667 2021-07-31T02:22:47.804 INFO:tasks.workunit.client.0.smithi045.stdout:op 5299 completed, throughput=5MB/sec 2021-07-31T02:22:47.805 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-IH3xhXzJVr8VkDX off=19963245 len=401799 2021-07-31T02:22:47.805 INFO:tasks.workunit.client.0.smithi045.stdout:op 5300 completed, throughput=5MB/sec 2021-07-31T02:22:47.805 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-DlH0RszJiAclbJX off=4747802 len=550475 2021-07-31T02:22:47.809 INFO:tasks.workunit.client.0.smithi045.stdout:op 5301 completed, throughput=5MB/sec 2021-07-31T02:22:48.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:48 smithi045 conmon[33655]: cluster 2021-07-31T02:22:46.301719+0000 2021-07-31T02:22:48.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:48 smithi045 conmon[33655]: mgr.x (mgr.24542) 1525 : cluster [DBG] pgmap v1938: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.8 MiB/s wr, 4 op/s 2021-07-31T02:22:48.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:48 smithi045 conmon[39718]: cluster 2021-07-31T02:22:46.301719+0000 mgr.x (mgr. 2021-07-31T02:22:48.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:48 smithi045 conmon[39718]: 24542) 1525 : cluster [DBG] pgmap v1938: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.8 MiB/s wr, 4 op/s 2021-07-31T02:22:48.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:48 smithi154 conmon[34553]: cluster 2021-07-31T02:22:46.301719+0000 mgr.x (mgr.24542) 1525 : cluster [DBG] pgmap v1938: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.8 MiB/s wr, 4 op/s 2021-07-31T02:22:48.810 INFO:tasks.workunit.client.0.smithi045.stdout: 1041: throughput=5MB/sec pending data=0 2021-07-31T02:22:48.810 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ZuhwE9wp836OzOF off=111616 len=1313200 2021-07-31T02:22:48.817 INFO:tasks.workunit.client.0.smithi045.stdout:op 5302 completed, throughput=5MB/sec 2021-07-31T02:22:48.817 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-LLI1nemDFbtVDX7 off=3057378 len=1572425 2021-07-31T02:22:48.820 INFO:tasks.workunit.client.0.smithi045.stdout:op 5303 completed, throughput=5MB/sec 2021-07-31T02:22:48.820 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-rDkC-GlJkRocyCr off=337461 len=1954788 2021-07-31T02:22:48.829 INFO:tasks.workunit.client.0.smithi045.stdout:op 5304 completed, throughput=5MB/sec 2021-07-31T02:22:48.830 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-1SUZDUIALPfp4fA off=4637448 len=770218 2021-07-31T02:22:48.834 INFO:tasks.workunit.client.0.smithi045.stdout:op 5305 completed, throughput=5MB/sec 2021-07-31T02:22:49.834 INFO:tasks.workunit.client.0.smithi045.stdout: 1042: throughput=5MB/sec pending data=0 2021-07-31T02:22:49.835 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-hm31JCRTC4oJYEY off=2119554 len=1185067 2021-07-31T02:22:49.837 INFO:tasks.workunit.client.0.smithi045.stdout:op 5306 completed, throughput=5MB/sec 2021-07-31T02:22:49.837 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-cBRXDruFhJPx4nX off=154306 len=1641364 2021-07-31T02:22:49.845 INFO:tasks.workunit.client.0.smithi045.stdout:op 5307 completed, throughput=5MB/sec 2021-07-31T02:22:49.845 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-pU5LEYhbqEoAt6Z off=430105 len=126683 2021-07-31T02:22:49.846 INFO:tasks.workunit.client.0.smithi045.stdout:op 5308 completed, throughput=5MB/sec 2021-07-31T02:22:49.846 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-7RdpDNi6HcHrayU off=9566249 len=1397364 2021-07-31T02:22:49.853 INFO:tasks.workunit.client.0.smithi045.stdout:op 5309 completed, throughput=5MB/sec 2021-07-31T02:22:49.853 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-7-TKJoWBqmZiSm3 off=12994977 len=2031124 2021-07-31T02:22:49.862 INFO:tasks.workunit.client.0.smithi045.stdout:op 5310 completed, throughput=5MB/sec 2021-07-31T02:22:50.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:50 smithi045 conmon[39718]: cluster 2021-07-31T02:22:48.302255+0000 mgr.x ( 2021-07-31T02:22:50.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:50 smithi045 conmon[39718]: mgr.24542) 1526 : cluster [DBG] pgmap v1939: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.8 MiB/s wr, 4 op/s 2021-07-31T02:22:50.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:50 smithi045 conmon[33655]: cluster 2021-07-31T02:22:48.302255+0000 2021-07-31T02:22:50.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:50 smithi045 conmon[33655]: mgr.x (mgr.24542) 1526 : cluster [DBG] pgmap v1939: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.8 MiB/s wr, 4 op/s 2021-07-31T02:22:50.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:50 smithi154 conmon[34553]: cluster 2021-07-31T02:22:48.302255+0000 mgr.x (mgr.24542) 1526 : cluster [DBG] pgmap v1939: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.8 MiB/s wr, 4 op/s 2021-07-31T02:22:50.862 INFO:tasks.workunit.client.0.smithi045.stdout: 1043: throughput=5MB/sec pending data=0 2021-07-31T02:22:50.863 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-_id-02UQer3q3Ek off=8749905 len=410037 2021-07-31T02:22:50.863 INFO:tasks.workunit.client.0.smithi045.stdout:op 5311 completed, throughput=5MB/sec 2021-07-31T02:22:50.863 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-m36VxIUkNpQRRkd off=7751590 len=54170 2021-07-31T02:22:50.866 INFO:tasks.workunit.client.0.smithi045.stdout:op 5312 completed, throughput=5MB/sec 2021-07-31T02:22:50.867 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-cZLdiHnsf3GeClS off=3274107 len=1571883 2021-07-31T02:22:50.874 INFO:tasks.workunit.client.0.smithi045.stdout:op 5313 completed, throughput=5MB/sec 2021-07-31T02:22:50.874 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-CnqjKtG15bzVGn3 off=10444108 len=1174290 2021-07-31T02:22:50.881 INFO:tasks.workunit.client.0.smithi045.stdout:op 5314 completed, throughput=5MB/sec 2021-07-31T02:22:50.881 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-sZp1a58x9GkbDQ- off=8960486 len=1752209 2021-07-31T02:22:50.889 INFO:tasks.workunit.client.0.smithi045.stdout:op 5315 completed, throughput=5MB/sec 2021-07-31T02:22:51.889 INFO:tasks.workunit.client.0.smithi045.stdout: 1044: throughput=5MB/sec pending data=0 2021-07-31T02:22:51.890 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-5i0GWwnvt_kbRGg off=5770244 len=1115706 2021-07-31T02:22:51.891 INFO:tasks.workunit.client.0.smithi045.stdout:op 5316 completed, throughput=5MB/sec 2021-07-31T02:22:51.891 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-jFiwGMBZ70SQol6 off=6720666 len=2080916 2021-07-31T02:22:51.901 INFO:tasks.workunit.client.0.smithi045.stdout:op 5317 completed, throughput=5MB/sec 2021-07-31T02:22:51.902 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-X1OiYHw9_KXAXyR off=10618347 len=170316 2021-07-31T02:22:51.902 INFO:tasks.workunit.client.0.smithi045.stdout:op 5318 completed, throughput=5MB/sec 2021-07-31T02:22:51.903 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-a9h0EvnEdkclqeA off=7658679 len=24541 2021-07-31T02:22:51.903 INFO:tasks.workunit.client.0.smithi045.stdout:op 5319 completed, throughput=5MB/sec 2021-07-31T02:22:51.904 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-sDkbyAotT7i27hJ off=776574 len=1896462 2021-07-31T02:22:51.914 INFO:tasks.workunit.client.0.smithi045.stdout:op 5320 completed, throughput=5MB/sec 2021-07-31T02:22:52.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:52 smithi045 conmon[33655]: cluster 2021-07-31T02:22:50.302845+0000 mgr.x (mgr.24542) 1527 : cluster [DBG] pgmap v1940: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 3.6 MiB/s rd, 2.7 MiB/s wr, 6 op/s 2021-07-31T02:22:52.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:52 smithi045 conmon[39718]: cluster 2021-07-31T02:22:50.302845+0000 mgr.x (mgr.24542 2021-07-31T02:22:52.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:52 smithi045 conmon[39718]: ) 1527 : cluster [DBG] pgmap v1940: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 3.6 MiB/s rd, 2.7 MiB/s wr, 6 op/s 2021-07-31T02:22:52.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:52 smithi154 conmon[34553]: cluster 2021-07-31T02:22:50.302845+0000 mgr.x (mgr.24542) 1527 : cluster [DBG] pgmap v1940: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 3.6 MiB/s rd, 2.7 MiB/s wr, 6 op/s 2021-07-31T02:22:52.915 INFO:tasks.workunit.client.0.smithi045.stdout: 1045: throughput=5MB/sec pending data=0 2021-07-31T02:22:52.915 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-GG78cu9FozeVmU8 off=10470262 len=1579031 2021-07-31T02:22:52.923 INFO:tasks.workunit.client.0.smithi045.stdout:op 5321 completed, throughput=5MB/sec 2021-07-31T02:22:52.923 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-SWUSqI05q5B08iu off=11615300 len=1008606 2021-07-31T02:22:52.925 INFO:tasks.workunit.client.0.smithi045.stdout:op 5322 completed, throughput=5MB/sec 2021-07-31T02:22:52.925 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-S3i9qvisDFRjbMS off=22961904 len=806334 2021-07-31T02:22:52.930 INFO:tasks.workunit.client.0.smithi045.stdout:op 5323 completed, throughput=5MB/sec 2021-07-31T02:22:52.931 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Es8tdWt_gUGt52P off=2714288 len=1561004 2021-07-31T02:22:52.933 INFO:tasks.workunit.client.0.smithi045.stdout:op 5324 completed, throughput=5MB/sec 2021-07-31T02:22:53.934 INFO:tasks.workunit.client.0.smithi045.stdout: 1046: throughput=5MB/sec pending data=0 2021-07-31T02:22:53.934 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-osfkcYGPHJwoDS- off=327497 len=64792 2021-07-31T02:22:53.938 INFO:tasks.workunit.client.0.smithi045.stdout:op 5325 completed, throughput=5MB/sec 2021-07-31T02:22:53.938 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-teyUKzEVMzOVl_v off=99921 len=869483 2021-07-31T02:22:53.943 INFO:tasks.workunit.client.0.smithi045.stdout:op 5326 completed, throughput=5MB/sec 2021-07-31T02:22:53.943 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-jJdsvoCXs3RAnCa off=5964796 len=1397462 2021-07-31T02:22:53.952 INFO:tasks.workunit.client.0.smithi045.stdout:op 5327 completed, throughput=5MB/sec 2021-07-31T02:22:53.952 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-GGW8byt2_9iHujH off=9861903 len=1020148 2021-07-31T02:22:53.961 INFO:tasks.workunit.client.0.smithi045.stdout:op 5328 completed, throughput=5MB/sec 2021-07-31T02:22:53.962 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-4mQfGRlZKW57Sbv off=4868666 len=821243 2021-07-31T02:22:53.962 INFO:tasks.workunit.client.0.smithi045.stdout:op 5329 completed, throughput=5MB/sec 2021-07-31T02:22:53.963 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-FMH7LKvHXPiVNhd off=1609885 len=127482 2021-07-31T02:22:53.964 INFO:tasks.workunit.client.0.smithi045.stdout:op 5330 completed, throughput=5MB/sec 2021-07-31T02:22:53.965 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-FMv2ZldVrfB3jAu off=11759008 len=53518 2021-07-31T02:22:53.967 INFO:tasks.workunit.client.0.smithi045.stdout:op 5331 completed, throughput=5MB/sec 2021-07-31T02:22:53.967 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-9zRrMHBDgNeJ1PE off=2825435 len=767220 2021-07-31T02:22:53.971 INFO:tasks.workunit.client.0.smithi045.stdout:op 5332 completed, throughput=5MB/sec 2021-07-31T02:22:54.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:22:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:22:53.570Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:22:54.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:22:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:22:53.570Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:22:54.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:22:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:22:53.570Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:22:54.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:54 smithi154 conmon[34553]: cluster 2021-07-31T02:22:52.303271+0000 mgr.x (mgr.24542) 1528 : cluster [DBG] pgmap v1941: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 2.2 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2021-07-31T02:22:54.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:54 smithi154 conmon[34553]: 2021-07-31T02:22:54.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:54 smithi045 conmon[33655]: cluster 2021-07-31T02:22:52.303271+0000 mgr.x (mgr.24542) 2021-07-31T02:22:54.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:54 smithi045 conmon[33655]: 1528 : cluster [DBG] pgmap v1941: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 2.2 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2021-07-31T02:22:54.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:54 smithi045 conmon[39718]: cluster 2021-07-31T02:22:52.303271+0000 mgr.x ( 2021-07-31T02:22:54.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:54 smithi045 conmon[39718]: mgr.24542) 1528 : cluster [DBG] pgmap v1941: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 2.2 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2021-07-31T02:22:54.971 INFO:tasks.workunit.client.0.smithi045.stdout: 1047: throughput=5MB/sec pending data=0 2021-07-31T02:22:54.972 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Lbrytz60YN820-e off=7110045 len=306890 2021-07-31T02:22:54.974 INFO:tasks.workunit.client.0.smithi045.stdout:op 5333 completed, throughput=5MB/sec 2021-07-31T02:22:54.974 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-02b3Hk5iyseQfgQ off=15331855 len=456574 2021-07-31T02:22:54.974 INFO:tasks.workunit.client.0.smithi045.stdout:op 5334 completed, throughput=5MB/sec 2021-07-31T02:22:54.974 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-bX0gtZ1E5C26qP5 off=11761233 len=1441334 2021-07-31T02:22:54.981 INFO:tasks.workunit.client.0.smithi045.stdout:op 5335 completed, throughput=5MB/sec 2021-07-31T02:22:54.982 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ICNAdYR_Q4T7Ndk off=4022216 len=1579031 2021-07-31T02:22:54.989 INFO:tasks.workunit.client.0.smithi045.stdout:op 5336 completed, throughput=5MB/sec 2021-07-31T02:22:54.989 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-qPgV4LPeVKWcfcc off=731937 len=928881 2021-07-31T02:22:54.990 INFO:tasks.workunit.client.0.smithi045.stdout:op 5337 completed, throughput=5MB/sec 2021-07-31T02:22:54.991 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-pvCZmjo70PwfVBq off=1465453 len=1930898 2021-07-31T02:22:54.999 INFO:tasks.workunit.client.0.smithi045.stdout:op 5338 completed, throughput=5MB/sec 2021-07-31T02:22:55.999 INFO:tasks.workunit.client.0.smithi045.stdout: 1048: throughput=5MB/sec pending data=0 2021-07-31T02:22:56.000 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-6N_yhGlYfQ06bZs off=5787570 len=869018 2021-07-31T02:22:56.001 INFO:tasks.workunit.client.0.smithi045.stdout:op 5339 completed, throughput=5MB/sec 2021-07-31T02:22:56.001 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj--Ms6UbQjMQ8QRqK off=9833508 len=979101 2021-07-31T02:22:56.007 INFO:tasks.workunit.client.0.smithi045.stdout:op 5340 completed, throughput=5MB/sec 2021-07-31T02:22:56.007 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-7lgvvcm5Wth0282 off=8721143 len=825299 2021-07-31T02:22:56.008 INFO:tasks.workunit.client.0.smithi045.stdout:op 5341 completed, throughput=5MB/sec 2021-07-31T02:22:56.008 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Iv-O04kUtVDGqjW off=20252408 len=29323 2021-07-31T02:22:56.011 INFO:tasks.workunit.client.0.smithi045.stdout:op 5342 completed, throughput=5MB/sec 2021-07-31T02:22:56.011 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-d1ptEIqRzFE8LYZ off=9274634 len=33909 2021-07-31T02:22:56.013 INFO:tasks.workunit.client.0.smithi045.stdout:op 5343 completed, throughput=5MB/sec 2021-07-31T02:22:56.014 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-zutZJncMyXWS_bi off=8821624 len=1945810 2021-07-31T02:22:56.022 INFO:tasks.workunit.client.0.smithi045.stdout:op 5344 completed, throughput=5MB/sec 2021-07-31T02:22:56.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:56 smithi154 conmon[34553]: cluster 2021-07-31T02:22:54.303814+0000 mgr.x (mgr.24542) 2021-07-31T02:22:56.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:56 smithi154 conmon[34553]: 1529 : cluster [DBG] pgmap v1942: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 2.6 MiB/s rd, 3.2 MiB/s wr, 5 op/s 2021-07-31T02:22:56.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:56 smithi045 conmon[39718]: cluster 2021-07-31T02:22:54.303814+0000 mgr.x (mgr.24542) 1529 : cluster 2021-07-31T02:22:56.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:56 smithi045 conmon[39718]: [DBG] pgmap v1942: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 2.6 MiB/s rd, 3.2 MiB/s wr, 5 op/s 2021-07-31T02:22:56.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:56 smithi045 conmon[33655]: cluster 2021-07-31T02:22:54.303814+0000 mgr.x (mgr.24542) 2021-07-31T02:22:56.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:56 smithi045 conmon[33655]: 1529 : cluster [DBG] pgmap v1942: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 2.6 MiB/s rd, 3.2 MiB/s wr, 5 op/s 2021-07-31T02:22:57.023 INFO:tasks.workunit.client.0.smithi045.stdout: 1049: throughput=5MB/sec pending data=0 2021-07-31T02:22:57.023 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-DfHY-Iw_ntruKVf off=6156242 len=1410715 2021-07-31T02:22:57.030 INFO:tasks.workunit.client.0.smithi045.stdout:op 5345 completed, throughput=5MB/sec 2021-07-31T02:22:57.030 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-NzsP6RcATzGNEIz off=1270976 len=387719 2021-07-31T02:22:57.033 INFO:tasks.workunit.client.0.smithi045.stdout:op 5346 completed, throughput=5MB/sec 2021-07-31T02:22:57.034 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-M0MsctH3Z2tBH_l off=1041063 len=2019848 2021-07-31T02:22:57.036 INFO:tasks.workunit.client.0.smithi045.stdout:op 5347 completed, throughput=5MB/sec 2021-07-31T02:22:57.037 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-DMl4w6S8AbTAOG0 off=5277248 len=2058315 2021-07-31T02:22:57.047 INFO:tasks.workunit.client.0.smithi045.stdout:op 5348 completed, throughput=5MB/sec 2021-07-31T02:22:58.047 INFO:tasks.workunit.client.0.smithi045.stdout: 1050: throughput=5MB/sec pending data=0 2021-07-31T02:22:58.047 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-0O_bHEr7MY9LhiV off=8643137 len=1866254 2021-07-31T02:22:58.049 INFO:tasks.workunit.client.0.smithi045.stdout:op 5349 completed, throughput=5MB/sec 2021-07-31T02:22:58.049 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-JVYgLaP4BP-3Dqk off=2982576 len=1481465 2021-07-31T02:22:58.057 INFO:tasks.workunit.client.0.smithi045.stdout:op 5350 completed, throughput=5MB/sec 2021-07-31T02:22:58.057 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-gHGaqNFaxdKKncF off=11340979 len=1017583 2021-07-31T02:22:58.058 INFO:tasks.workunit.client.0.smithi045.stdout:op 5351 completed, throughput=5MB/sec 2021-07-31T02:22:58.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:58 smithi154 conmon[34553]: cluster 2021-07-31T02:22:56.304326+0000 mgr.x (mgr.24542) 1530 : cluster [DBG] pgmap v1943: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.6 MiB/s rd, 2.2 MiB/s wr, 3 op/s 2021-07-31T02:22:58.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:58 smithi045 conmon[39718]: cluster 2021-07-31T02:22:56.304326+0000 mgr.x 2021-07-31T02:22:58.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:58 smithi045 conmon[39718]: (mgr.24542) 1530 : cluster [DBG] pgmap v1943: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.6 MiB/s rd, 2.2 MiB/s wr, 3 op/s 2021-07-31T02:22:58.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:58 smithi045 conmon[33655]: cluster 2021-07-31T02:22:56.304326+0000 mgr.x (mgr.24542) 2021-07-31T02:22:58.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:58 smithi045 conmon[33655]: 1530 : cluster [DBG] pgmap v1943: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.6 MiB/s rd, 2.2 MiB/s wr, 3 op/s 2021-07-31T02:22:59.059 INFO:tasks.workunit.client.0.smithi045.stdout: 1051: throughput=5MB/sec pending data=0 2021-07-31T02:22:59.059 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-J_K4SPbeGA18vPK off=3770338 len=333388 2021-07-31T02:22:59.062 INFO:tasks.workunit.client.0.smithi045.stdout:op 5352 completed, throughput=5MB/sec 2021-07-31T02:22:59.063 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-cKPY8-OEuMKw49i off=6091547 len=68737 2021-07-31T02:22:59.065 INFO:tasks.workunit.client.0.smithi045.stdout:op 5353 completed, throughput=5MB/sec 2021-07-31T02:22:59.065 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-_7Xe_gU7IcDiq7w off=712190 len=665473 2021-07-31T02:22:59.066 INFO:tasks.workunit.client.0.smithi045.stdout:op 5354 completed, throughput=5MB/sec 2021-07-31T02:22:59.066 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-9cV8QExIqOX4AOW off=4150381 len=1537994 2021-07-31T02:22:59.073 INFO:tasks.workunit.client.0.smithi045.stdout:op 5355 completed, throughput=5MB/sec 2021-07-31T02:22:59.074 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-biIlv-02LK7Ckem off=14109463 len=2032114 2021-07-31T02:22:59.076 INFO:tasks.workunit.client.0.smithi045.stdout:op 5356 completed, throughput=5MB/sec 2021-07-31T02:22:59.076 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-NNVeKhuxX8St2tf off=13141719 len=837490 2021-07-31T02:22:59.078 INFO:tasks.workunit.client.0.smithi045.stdout:op 5357 completed, throughput=5MB/sec 2021-07-31T02:22:59.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:22:59 smithi154 conmon[34553]: cluster 2021-07-31T02:22:58.304799+0000 mgr.x (mgr.24542) 1531 : cluster [DBG] pgmap v1944: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.6 MiB/s rd, 2.2 MiB/s wr, 3 op/s 2021-07-31T02:22:59.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:59 smithi045 conmon[39718]: cluster 2021-07-31T02:22:58.304799+0000 mgr.x (mgr.24542) 1531 : cluster [DBG] 2021-07-31T02:22:59.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:22:59 smithi045 conmon[39718]: pgmap v1944: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.6 MiB/s rd, 2.2 MiB/s wr, 3 op/s 2021-07-31T02:22:59.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:59 smithi045 conmon[33655]: cluster 2021-07-31T02:22:58.304799+0000 2021-07-31T02:22:59.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:22:59 smithi045 conmon[33655]: mgr.x (mgr.24542) 1531 : cluster [DBG] pgmap v1944: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.6 MiB/s rd, 2.2 MiB/s wr, 3 op/s 2021-07-31T02:23:00.078 INFO:tasks.workunit.client.0.smithi045.stdout: 1052: throughput=5MB/sec pending data=0 2021-07-31T02:23:00.078 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-PqVusJ8Lwi0CMW_ off=13045367 len=2027378 2021-07-31T02:23:00.087 INFO:tasks.workunit.client.0.smithi045.stdout:op 5358 completed, throughput=5MB/sec 2021-07-31T02:23:00.087 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-XqnVRSN9qkUR9kn off=818470 len=354869 2021-07-31T02:23:00.088 INFO:tasks.workunit.client.0.smithi045.stdout:op 5359 completed, throughput=5MB/sec 2021-07-31T02:23:00.089 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-afSzKaVmlhnOv7w off=1377651 len=324836 2021-07-31T02:23:00.092 INFO:tasks.workunit.client.0.smithi045.stdout:op 5360 completed, throughput=5MB/sec 2021-07-31T02:23:00.093 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj--E2-IuAhI_T3pcy off=7280700 len=707841 2021-07-31T02:23:00.096 INFO:tasks.workunit.client.0.smithi045.stdout:op 5361 completed, throughput=5MB/sec 2021-07-31T02:23:00.096 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-nzKesKTYSKGYshN off=19849925 len=1182404 2021-07-31T02:23:00.098 INFO:tasks.workunit.client.0.smithi045.stdout:op 5362 completed, throughput=5MB/sec 2021-07-31T02:23:00.098 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ZsqPy2j0RTGK4HA off=6037979 len=31522 2021-07-31T02:23:00.099 INFO:tasks.workunit.client.0.smithi045.stdout:op 5363 completed, throughput=5MB/sec 2021-07-31T02:23:00.099 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-TEKIIC26Uc3IWRb off=12443076 len=198771 2021-07-31T02:23:00.100 INFO:tasks.workunit.client.0.smithi045.stdout:op 5364 completed, throughput=5MB/sec 2021-07-31T02:23:00.560 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:23:00.841 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:23:00.842 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 37m ago 38m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:23:00.842 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 35m ago 38m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:23:00.843 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 35m ago 41m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:23:00.843 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 37m ago 44m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:23:00.843 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 37m ago 44m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:23:00.843 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 35m ago 42m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:23:00.844 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 37m ago 42m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:23:00.844 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 37m ago 38m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:23:00.844 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 35m ago 38m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:23:00.844 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 37m ago 41m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:23:00.845 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 37m ago 41m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:23:00.845 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 37m ago 40m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:23:00.845 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 37m ago 40m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:23:00.845 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 35m ago 40m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:23:00.845 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 35m ago 39m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:23:00.846 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 35m ago 39m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:23:00.846 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 35m ago 39m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:23:00.846 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 35m ago 38m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:23:01.100 INFO:tasks.workunit.client.0.smithi045.stdout: 1053: throughput=5MB/sec pending data=0 2021-07-31T02:23:01.100 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-sCF0cc3pBkObnpM off=4342450 len=1387610 2021-07-31T02:23:01.101 INFO:tasks.workunit.client.0.smithi045.stdout:op 5365 completed, throughput=5MB/sec 2021-07-31T02:23:01.101 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-wiZuqcsWfG7pwyb off=6668196 len=1748920 2021-07-31T02:23:01.105 INFO:tasks.workunit.client.0.smithi045.stdout:op 5366 completed, throughput=5MB/sec 2021-07-31T02:23:01.105 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-9zRrMHBDgNeJ1PE off=2591506 len=930675 2021-07-31T02:23:01.109 INFO:tasks.workunit.client.0.smithi045.stdout:op 5367 completed, throughput=5MB/sec 2021-07-31T02:23:01.110 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-xdNpNCcqH7ymv4T off=8690536 len=510614 2021-07-31T02:23:01.113 INFO:tasks.workunit.client.0.smithi045.stdout:op 5368 completed, throughput=5MB/sec 2021-07-31T02:23:01.114 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-oe5l2QwrOGjnZ-0 off=15010275 len=147388 2021-07-31T02:23:01.116 INFO:tasks.workunit.client.0.smithi045.stdout:op 5369 completed, throughput=5MB/sec 2021-07-31T02:23:01.116 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ZCQnCdE6p2ykBdQ off=4311067 len=432710 2021-07-31T02:23:01.120 INFO:tasks.workunit.client.0.smithi045.stdout:op 5370 completed, throughput=5MB/sec 2021-07-31T02:23:01.120 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ou2VcDY1OvSdo_o off=13923345 len=1048384 2021-07-31T02:23:01.122 INFO:tasks.workunit.client.0.smithi045.stdout:op 5371 completed, throughput=5MB/sec 2021-07-31T02:23:01.180 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:23:01.180 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:23:01.180 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:23:01.181 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:23:01.181 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:23:01.181 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:23:01.182 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:23:01.182 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:23:01.183 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:23:01.183 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:23:01.183 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:23:01.183 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:23:01.183 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:23:01.184 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:23:01.184 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:23:01.184 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:23:01.185 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:23:01.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:01 smithi154 conmon[34553]: cluster 2021-07-31T02:23:00.305297+0000 mgr.x (mgr.24542) 1532 : cluster [DBG] pgmap v1945: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 2.2 MiB/s rd, 3.6 MiB/s wr, 5 op/s 2021-07-31T02:23:01.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:01 smithi154 conmon[34553]: audit 2021-07-31T02:23:00.548911+0000 mgr.x (mgr.24542) 1533 : audit [DBG] from='client.26254 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:23:01.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:01 smithi154 conmon[34553]: audit 2021-07-31T02:23:00.834149+0000 mgr.x (mgr.24542) 1534 : audit [DBG] from='client.16353 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:23:01.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:01 smithi154 conmon[34553]: audit 2021-07-31T02:23:01.180153+0000 mon.a (mon.0) 1903 2021-07-31T02:23:01.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:01 smithi154 conmon[34553]: : audit [DBG] from='client.? 172.21.15.45:0/2269415043' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:23:01.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:01 smithi045 conmon[33655]: cluster 2021-07-31T02:23:00.305297+0000 mgr.x (mgr.24542) 1532 : cluster 2021-07-31T02:23:01.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:01 smithi045 conmon[33655]: [DBG] pgmap v1945: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 2.2 MiB/s rd, 3.6 MiB/s wr, 5 op/s 2021-07-31T02:23:01.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:01 smithi045 conmon[33655]: audit 2021-07-31T02:23:00.548911+0000 mgr.x (mgr.24542) 1533 : audit [DBG] from='client.26254 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:23:01.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:01 smithi045 conmon[33655]: audit 2021-07-31T02:23:00.834149+0000 mgr.x ( 2021-07-31T02:23:01.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:01 smithi045 conmon[33655]: mgr.24542) 1534 : audit [DBG] from='client.16353 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:23:01.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:01 smithi045 conmon[33655]: audit 2021-07-31T02:23:01.180153+0000 mon.a (mon.0) 1903 : audit [DBG] from='client.? 172.21.15.45:0/2269415043' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:23:01.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:01 smithi045 conmon[39718]: cluster 2021-07-31T02:23:00.305297+0000 mgr.x (mgr.24542 2021-07-31T02:23:01.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:01 smithi045 conmon[39718]: ) 1532 : cluster [DBG] pgmap v1945: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 2.2 MiB/s rd, 3.6 MiB/s wr, 5 op/s 2021-07-31T02:23:01.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:01 smithi045 conmon[39718]: audit 2021-07-31T02:23:00.548911+0000 mgr.x (mgr.24542) 1533 2021-07-31T02:23:01.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:01 smithi045 conmon[39718]: : audit [DBG] from='client.26254 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:23:01.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:01 smithi045 conmon[39718]: audit 2021-07-31T02:23:00 2021-07-31T02:23:01.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:01 smithi045 conmon[39718]: .834149+0000 mgr.x (mgr.24542) 1534 : audit [DBG] from='client.16353 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:23:01.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:01 smithi045 conmon[39718]: audit 2021-07-31T02:23:01.180153+0000 2021-07-31T02:23:01.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:01 smithi045 conmon[39718]: mon.a (mon.0) 1903 : audit [DBG] from='client.? 172.21.15.45:0/2269415043' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:23:02.122 INFO:tasks.workunit.client.0.smithi045.stdout: 1054: throughput=5MB/sec pending data=0 2021-07-31T02:23:02.123 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-LcNugv35I7ghClc off=15365678 len=1302665 2021-07-31T02:23:02.125 INFO:tasks.workunit.client.0.smithi045.stdout:op 5372 completed, throughput=5MB/sec 2021-07-31T02:23:02.125 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-7v3KyJF-23liV9C off=5658906 len=375154 2021-07-31T02:23:02.126 INFO:tasks.workunit.client.0.smithi045.stdout:op 5373 completed, throughput=5MB/sec 2021-07-31T02:23:02.126 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-kH54pxyvRFa_DAU off=11260491 len=1756989 2021-07-31T02:23:02.128 INFO:tasks.workunit.client.0.smithi045.stdout:op 5374 completed, throughput=5MB/sec 2021-07-31T02:23:02.128 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-iAh-RmEawGwMY5p off=14241148 len=1347620 2021-07-31T02:23:02.134 INFO:tasks.workunit.client.0.smithi045.stdout:op 5375 completed, throughput=5MB/sec 2021-07-31T02:23:03.134 INFO:tasks.workunit.client.0.smithi045.stdout: 1055: throughput=5MB/sec pending data=0 2021-07-31T02:23:03.135 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-gjwIa1syRNSrdM_ off=18379330 len=2031648 2021-07-31T02:23:03.144 INFO:tasks.workunit.client.0.smithi045.stdout:op 5376 completed, throughput=5MB/sec 2021-07-31T02:23:03.144 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-GeCN4Wsnv17TvQp off=5370453 len=364431 2021-07-31T02:23:03.145 INFO:tasks.workunit.client.0.smithi045.stdout:op 5377 completed, throughput=5MB/sec 2021-07-31T02:23:03.145 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ZaYTcj-CmFNuPG3 off=15169714 len=2039495 2021-07-31T02:23:03.154 INFO:tasks.workunit.client.0.smithi045.stdout:op 5378 completed, throughput=5MB/sec 2021-07-31T02:23:03.154 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-3LZcg6HiTgKPHsX off=8306462 len=1713325 2021-07-31T02:23:03.161 INFO:tasks.workunit.client.0.smithi045.stdout:op 5379 completed, throughput=5MB/sec 2021-07-31T02:23:04.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:23:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:23:03.571Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:23:04.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:23:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:23:03.571Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:23:04.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:23:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:23:03.571Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:23:04.162 INFO:tasks.workunit.client.0.smithi045.stdout: 1056: throughput=5MB/sec pending data=0 2021-07-31T02:23:04.162 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-EPYwVgZhFG4MnXP off=212982 len=1721721 2021-07-31T02:23:04.165 INFO:tasks.workunit.client.0.smithi045.stdout:op 5380 completed, throughput=5MB/sec 2021-07-31T02:23:04.165 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-8nNJoEXle1pXcZH off=644056 len=413298 2021-07-31T02:23:04.166 INFO:tasks.workunit.client.0.smithi045.stdout:op 5381 completed, throughput=5MB/sec 2021-07-31T02:23:04.166 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-wOIO_ZE0-M5qndF off=474669 len=1348159 2021-07-31T02:23:04.175 INFO:tasks.workunit.client.0.smithi045.stdout:op 5382 completed, throughput=5MB/sec 2021-07-31T02:23:04.175 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-0-SGHoYCX6__MEb off=8867178 len=362500 2021-07-31T02:23:04.176 INFO:tasks.workunit.client.0.smithi045.stdout:op 5383 completed, throughput=5MB/sec 2021-07-31T02:23:04.176 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-5TUA3YGewRdEFKn off=1478012 len=745695 2021-07-31T02:23:04.180 INFO:tasks.workunit.client.0.smithi045.stdout:op 5384 completed, throughput=5MB/sec 2021-07-31T02:23:04.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:04 smithi154 conmon[34553]: cluster 2021-07-31T02:23:02.305825+0000 mgr.x (mgr.24542) 1535 : cluster [DBG] 2021-07-31T02:23:04.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:04 smithi154 conmon[34553]: pgmap v1946: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.1 MiB/s rd, 2.8 MiB/s wr, 3 op/s 2021-07-31T02:23:04.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:04 smithi045 conmon[39718]: cluster 2021-07-31T02:23:02.305825+0000 mgr.x (mgr 2021-07-31T02:23:04.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:04 smithi045 conmon[39718]: .24542) 1535 : cluster [DBG] pgmap v1946: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.1 MiB/s rd, 2.8 MiB/s wr, 3 op/s 2021-07-31T02:23:04.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:04 smithi045 conmon[33655]: cluster 2021-07-31T02:23:02.305825+0000 2021-07-31T02:23:04.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:04 smithi045 conmon[33655]: mgr.x (mgr.24542) 1535 : cluster [DBG] pgmap v1946: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.1 MiB/s rd, 2.8 MiB/s wr, 3 op/s 2021-07-31T02:23:05.180 INFO:tasks.workunit.client.0.smithi045.stdout: 1057: throughput=5MB/sec pending data=0 2021-07-31T02:23:05.181 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-fNLteZzB9gZ77x6 off=422362 len=1545464 2021-07-31T02:23:05.190 INFO:tasks.workunit.client.0.smithi045.stdout:op 5385 completed, throughput=5MB/sec 2021-07-31T02:23:05.190 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-uGcecmdCSk70d2b off=1099121 len=1124539 2021-07-31T02:23:05.192 INFO:tasks.workunit.client.0.smithi045.stdout:op 5386 completed, throughput=5MB/sec 2021-07-31T02:23:05.193 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-M6rFKwxOFNqkD2Q off=2204561 len=191618 2021-07-31T02:23:05.193 INFO:tasks.workunit.client.0.smithi045.stdout:op 5387 completed, throughput=5MB/sec 2021-07-31T02:23:05.193 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-K2LSEHa3qJHnVXA off=23092937 len=936855 2021-07-31T02:23:05.199 INFO:tasks.workunit.client.0.smithi045.stdout:op 5388 completed, throughput=5MB/sec 2021-07-31T02:23:05.200 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-B3a0lpfdXxbAH6t off=13211548 len=1362963 2021-07-31T02:23:05.206 INFO:tasks.workunit.client.0.smithi045.stdout:op 5389 completed, throughput=5MB/sec 2021-07-31T02:23:06.206 INFO:tasks.workunit.client.0.smithi045.stdout: 1058: throughput=5MB/sec pending data=0 2021-07-31T02:23:06.207 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-_TDRrJyq55Ik3gf off=5485151 len=1605401 2021-07-31T02:23:06.213 INFO:tasks.workunit.client.0.smithi045.stdout:op 5390 completed, throughput=5MB/sec 2021-07-31T02:23:06.213 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-YT0x1eRpgDPn5ad off=3698865 len=1421192 2021-07-31T02:23:06.222 INFO:tasks.workunit.client.0.smithi045.stdout:op 5391 completed, throughput=5MB/sec 2021-07-31T02:23:06.222 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-cMZve3iYAUL_hFZ off=15753884 len=735071 2021-07-31T02:23:06.228 INFO:tasks.workunit.client.0.smithi045.stdout:op 5392 completed, throughput=5MB/sec 2021-07-31T02:23:06.228 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-QsvdZvRLxALQO8Z off=14839746 len=265526 2021-07-31T02:23:06.231 INFO:tasks.workunit.client.0.smithi045.stdout:op 5393 completed, throughput=5MB/sec 2021-07-31T02:23:06.231 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-X9lLW7CxmzAxviU off=9185290 len=30902 2021-07-31T02:23:06.232 INFO:tasks.workunit.client.0.smithi045.stdout:op 5394 completed, throughput=5MB/sec 2021-07-31T02:23:06.232 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-3RsQys1lxtf7n8c off=3975446 len=204432 2021-07-31T02:23:06.234 INFO:tasks.workunit.client.0.smithi045.stdout:op 5395 completed, throughput=5MB/sec 2021-07-31T02:23:06.235 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Wy1FnE2HnvL36Il off=3597572 len=918429 2021-07-31T02:23:06.236 INFO:tasks.workunit.client.0.smithi045.stdout:op 5396 completed, throughput=5MB/sec 2021-07-31T02:23:06.236 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-lJH9GR-W_SFxDbo off=9809960 len=1771805 2021-07-31T02:23:06.244 INFO:tasks.workunit.client.0.smithi045.stdout:op 5397 completed, throughput=5MB/sec 2021-07-31T02:23:06.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:06 smithi154 conmon[34553]: cluster 2021-07-31T02:23:04.306293+0000 mgr.x (mgr.24542) 1536 : cluster [DBG] pgmap v1947: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 2.2 MiB/s rd, 3.9 MiB/s wr, 6 op/s 2021-07-31T02:23:06.659 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:06 smithi045 conmon[39718]: cluster 2021-07-31T02:23:04.306293+0000 2021-07-31T02:23:06.660 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:06 smithi045 conmon[39718]: mgr.x (mgr.24542) 1536 : cluster [DBG] pgmap v1947: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 2.2 MiB/s rd, 3.9 MiB/s wr, 6 op/s 2021-07-31T02:23:06.660 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:06 smithi045 conmon[33655]: cluster 2021-07-31T02:23:04.306293+0000 mgr.x (mgr. 2021-07-31T02:23:06.660 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:06 smithi045 conmon[33655]: 24542) 1536 : cluster [DBG] pgmap v1947: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 2.2 MiB/s rd, 3.9 MiB/s wr, 6 op/s 2021-07-31T02:23:07.244 INFO:tasks.workunit.client.0.smithi045.stdout: 1059: throughput=5MB/sec pending data=0 2021-07-31T02:23:07.244 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-l1r7PVsadnEjsPr off=6378924 len=1552556 2021-07-31T02:23:07.251 INFO:tasks.workunit.client.0.smithi045.stdout:op 5398 completed, throughput=5MB/sec 2021-07-31T02:23:07.251 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-WnTWIRVnmoK7qEI off=2610944 len=524487 2021-07-31T02:23:07.251 INFO:tasks.workunit.client.0.smithi045.stdout:op 5399 completed, throughput=5MB/sec 2021-07-31T02:23:07.252 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-0l84aTGu72NWven off=10332464 len=1279449 2021-07-31T02:23:07.254 INFO:tasks.workunit.client.0.smithi045.stdout:op 5400 completed, throughput=5MB/sec 2021-07-31T02:23:07.254 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-nJqqiUNK9JemKRx off=158304 len=347784 2021-07-31T02:23:07.257 INFO:tasks.workunit.client.0.smithi045.stdout:op 5401 completed, throughput=5MB/sec 2021-07-31T02:23:08.258 INFO:tasks.workunit.client.0.smithi045.stdout: 1060: throughput=5MB/sec pending data=0 2021-07-31T02:23:08.259 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj--cuFS6ZJPRMdBz2 off=3537266 len=593195 2021-07-31T02:23:08.262 INFO:tasks.workunit.client.0.smithi045.stdout:op 5402 completed, throughput=5MB/sec 2021-07-31T02:23:08.263 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-SNJAJU_E43Cf8fo off=3792926 len=1721996 2021-07-31T02:23:08.270 INFO:tasks.workunit.client.0.smithi045.stdout:op 5403 completed, throughput=5MB/sec 2021-07-31T02:23:08.270 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-9UOucgyXVL7FwVc off=984654 len=1980769 2021-07-31T02:23:08.273 INFO:tasks.workunit.client.0.smithi045.stdout:op 5404 completed, throughput=5MB/sec 2021-07-31T02:23:08.274 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-rA9_iRKnBvUpk4R off=7900060 len=1428812 2021-07-31T02:23:08.278 INFO:tasks.workunit.client.0.smithi045.stdout:op 5405 completed, throughput=5MB/sec 2021-07-31T02:23:08.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:08 smithi154 conmon[34553]: cluster 2021-07-31T02:23:06.306717+0000 mgr.x (mgr.24542) 1537 : cluster [DBG] pgmap v1948: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2021-07-31T02:23:08.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:08 smithi045 conmon[39718]: cluster 2021-07-31T02:23:06.306717+0000 mgr.x (mgr.24542) 1537 : cluster [DBG] 2021-07-31T02:23:08.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:08 smithi045 conmon[39718]: pgmap v1948: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2021-07-31T02:23:08.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:08 smithi045 conmon[33655]: cluster 2021-07-31T02:23:06.306717 2021-07-31T02:23:08.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:08 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 1537 : cluster [DBG] pgmap v1948: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2021-07-31T02:23:09.279 INFO:tasks.workunit.client.0.smithi045.stdout: 1061: throughput=5MB/sec pending data=0 2021-07-31T02:23:09.279 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-muP13TQ-QZUcKE0 off=893810 len=1717506 2021-07-31T02:23:09.282 INFO:tasks.workunit.client.0.smithi045.stdout:op 5406 completed, throughput=5MB/sec 2021-07-31T02:23:09.282 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-R2cpbEdL12emhVP off=12700598 len=759617 2021-07-31T02:23:09.284 INFO:tasks.workunit.client.0.smithi045.stdout:op 5407 completed, throughput=5MB/sec 2021-07-31T02:23:09.284 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj--cqs3u9viAMoY1K off=1297156 len=732681 2021-07-31T02:23:09.291 INFO:tasks.workunit.client.0.smithi045.stdout:op 5408 completed, throughput=5MB/sec 2021-07-31T02:23:09.291 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-_snqsucI2Ad-nO4 off=5389327 len=182439 2021-07-31T02:23:09.292 INFO:tasks.workunit.client.0.smithi045.stdout:op 5409 completed, throughput=5MB/sec 2021-07-31T02:23:09.292 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-zp5W8qLnG50lQV4 off=13544299 len=1889524 2021-07-31T02:23:09.294 INFO:tasks.workunit.client.0.smithi045.stdout:op 5410 completed, throughput=5MB/sec 2021-07-31T02:23:10.294 INFO:tasks.workunit.client.0.smithi045.stdout: 1062: throughput=5MB/sec pending data=0 2021-07-31T02:23:10.295 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Rlet7D4V_uDPmsG off=14229603 len=137497 2021-07-31T02:23:10.297 INFO:tasks.workunit.client.0.smithi045.stdout:op 5411 completed, throughput=5MB/sec 2021-07-31T02:23:10.297 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-E4ze8HIzAFLDO2K off=5474302 len=1747501 2021-07-31T02:23:10.305 INFO:tasks.workunit.client.0.smithi045.stdout:op 5412 completed, throughput=5MB/sec 2021-07-31T02:23:10.306 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-DHibxBQFauigtoV off=4481872 len=1021251 2021-07-31T02:23:10.307 INFO:tasks.workunit.client.0.smithi045.stdout:op 5413 completed, throughput=5MB/sec 2021-07-31T02:23:10.308 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-JWm4tl7W0YWV1N3 off=14476289 len=241805 2021-07-31T02:23:10.310 INFO:tasks.workunit.client.0.smithi045.stdout:op 5414 completed, throughput=5MB/sec 2021-07-31T02:23:10.311 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-YTMJfvET_4EjaMR off=8800386 len=527139 2021-07-31T02:23:10.315 INFO:tasks.workunit.client.0.smithi045.stdout:op 5415 completed, throughput=5MB/sec 2021-07-31T02:23:10.315 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-53MTPLC1AEI64LY off=9361875 len=1968708 2021-07-31T02:23:10.323 INFO:tasks.workunit.client.0.smithi045.stdout:op 5416 completed, throughput=5MB/sec 2021-07-31T02:23:10.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:10 smithi154 conmon[34553]: cluster 2021-07-31T02:23:08.307253+0000 mgr.x (mgr.24542) 1538 : cluster [DBG] pgmap v1949: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2021-07-31T02:23:10.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:10 smithi045 conmon[33655]: cluster 2021-07-31T02:23:08.307253+0000 mgr.x (mgr.24542) 1538 : 2021-07-31T02:23:10.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:10 smithi045 conmon[33655]: cluster [DBG] pgmap v1949: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2021-07-31T02:23:10.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:10 smithi045 conmon[39718]: cluster 2021-07-31T02:23:08.307253+0000 mgr.x (mgr.24542) 2021-07-31T02:23:10.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:10 smithi045 conmon[39718]: 1538 : cluster [DBG] pgmap v1949: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2021-07-31T02:23:11.324 INFO:tasks.workunit.client.0.smithi045.stdout: 1063: throughput=5MB/sec pending data=0 2021-07-31T02:23:11.324 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-urZAg5TuwX5G-Dz off=11650933 len=759120 2021-07-31T02:23:11.325 INFO:tasks.workunit.client.0.smithi045.stdout:op 5417 completed, throughput=5MB/sec 2021-07-31T02:23:11.326 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-lZBw4DVaHKLtIvR off=272500 len=517613 2021-07-31T02:23:11.327 INFO:tasks.workunit.client.0.smithi045.stdout:op 5418 completed, throughput=5MB/sec 2021-07-31T02:23:11.327 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-nIBewmdhVDNSn4m off=12266458 len=1084964 2021-07-31T02:23:11.335 INFO:tasks.workunit.client.0.smithi045.stdout:op 5419 completed, throughput=5MB/sec 2021-07-31T02:23:11.335 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-5697S50DIpDrQ7u off=1217881 len=296882 2021-07-31T02:23:11.336 INFO:tasks.workunit.client.0.smithi045.stdout:op 5420 completed, throughput=5MB/sec 2021-07-31T02:23:11.336 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-e_ht0CSAw5G9yyf off=14274099 len=730432 2021-07-31T02:23:11.337 INFO:tasks.workunit.client.0.smithi045.stdout:op 5421 completed, throughput=5MB/sec 2021-07-31T02:23:11.337 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-T1fCcz8lWmOOZf0 off=0 len=1571539 2021-07-31T02:23:11.344 INFO:tasks.workunit.client.0.smithi045.stdout:op 5422 completed, throughput=5MB/sec 2021-07-31T02:23:12.344 INFO:tasks.workunit.client.0.smithi045.stdout: 1064: throughput=5MB/sec pending data=0 2021-07-31T02:23:12.345 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-xrLNns70nxdGAP_ off=729136 len=1111061 2021-07-31T02:23:12.346 INFO:tasks.workunit.client.0.smithi045.stdout:op 5423 completed, throughput=5MB/sec 2021-07-31T02:23:12.346 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-6TpOrS5kN4U4pKE off=13416810 len=1519039 2021-07-31T02:23:12.354 INFO:tasks.workunit.client.0.smithi045.stdout:op 5424 completed, throughput=5MB/sec 2021-07-31T02:23:12.354 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-9ERjAqkhkxvwm59 off=9467576 len=1790399 2021-07-31T02:23:12.358 INFO:tasks.workunit.client.0.smithi045.stdout:op 5425 completed, throughput=5MB/sec 2021-07-31T02:23:12.358 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-5XktsW3ltlSy1kb off=1927395 len=689203 2021-07-31T02:23:12.359 INFO:tasks.workunit.client.0.smithi045.stdout:op 5426 completed, throughput=5MB/sec 2021-07-31T02:23:12.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:12 smithi154 conmon[34553]: cluster 2021-07-31T02:23:10.307643+0000 mgr.x (mgr.24542) 1539 : cluster [DBG] 2021-07-31T02:23:12.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:12 smithi154 conmon[34553]: pgmap v1950: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 2.4 MiB/s rd, 4.0 MiB/s wr, 6 op/s 2021-07-31T02:23:12.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:12 smithi045 conmon[33655]: cluster 2021-07-31T02:23:10.307643+0000 mgr.x (mgr.24542) 1539 : cluster 2021-07-31T02:23:12.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:12 smithi045 conmon[33655]: [DBG] pgmap v1950: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 2.4 MiB/s rd, 4.0 MiB/s wr, 6 op/s 2021-07-31T02:23:12.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:12 smithi045 conmon[39718]: cluster 2021-07-31T02:23:10.307643+0000 mgr.x ( 2021-07-31T02:23:12.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:12 smithi045 conmon[39718]: mgr.24542) 1539 : cluster [DBG] pgmap v1950: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 2.4 MiB/s rd, 4.0 MiB/s wr, 6 op/s 2021-07-31T02:23:13.360 INFO:tasks.workunit.client.0.smithi045.stdout: 1065: throughput=5MB/sec pending data=0 2021-07-31T02:23:13.361 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-fHYHmdupnlTkeSQ off=1855988 len=1417270 2021-07-31T02:23:13.367 INFO:tasks.workunit.client.0.smithi045.stdout:op 5427 completed, throughput=5MB/sec 2021-07-31T02:23:13.368 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-TV53ZVS5oTLrnru off=9261724 len=160578 2021-07-31T02:23:13.368 INFO:tasks.workunit.client.0.smithi045.stdout:op 5428 completed, throughput=5MB/sec 2021-07-31T02:23:13.368 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-7xlv9Rfr6DtMQ44 off=5659392 len=891520 2021-07-31T02:23:13.370 INFO:tasks.workunit.client.0.smithi045.stdout:op 5429 completed, throughput=5MB/sec 2021-07-31T02:23:13.370 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-G_Keuq161s1j9_q off=1156637 len=875279 2021-07-31T02:23:13.371 INFO:tasks.workunit.client.0.smithi045.stdout:op 5430 completed, throughput=5MB/sec 2021-07-31T02:23:13.371 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-CqcNEvgaA2S1eh0 off=1712251 len=1538349 2021-07-31T02:23:13.372 INFO:tasks.workunit.client.0.smithi045.stdout:op 5431 completed, throughput=5MB/sec 2021-07-31T02:23:13.373 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-aZC_g8aQjYQAxTk off=23390 len=1884181 2021-07-31T02:23:13.380 INFO:tasks.workunit.client.0.smithi045.stdout:op 5432 completed, throughput=5MB/sec 2021-07-31T02:23:14.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:23:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:23:13.571Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:23:14.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:23:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:23:13.571Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:23:14.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:23:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:23:13.571Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:23:14.381 INFO:tasks.workunit.client.0.smithi045.stdout: 1066: throughput=5MB/sec pending data=0 2021-07-31T02:23:14.381 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj--cqs3u9viAMoY1K off=652447 len=1377390 2021-07-31T02:23:14.390 INFO:tasks.workunit.client.0.smithi045.stdout:op 5433 completed, throughput=5MB/sec 2021-07-31T02:23:14.390 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-_ntCJm_AUl74ZLr off=4882581 len=1920684 2021-07-31T02:23:14.394 INFO:tasks.workunit.client.0.smithi045.stdout:op 5434 completed, throughput=5MB/sec 2021-07-31T02:23:14.395 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-80JkLeVB629zI4J off=408824 len=1938091 2021-07-31T02:23:14.397 INFO:tasks.workunit.client.0.smithi045.stdout:op 5435 completed, throughput=5MB/sec 2021-07-31T02:23:14.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:14 smithi154 conmon[34553]: cluster 2021-07-31T02:23:12.308168+0000 mgr.x (mgr.24542) 1540 : cluster [DBG] pgmap v1951: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2021-07-31T02:23:14.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:14 smithi045 conmon[33655]: cluster 2021-07-31T02: 2021-07-31T02:23:14.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:14 smithi045 conmon[33655]: 23:12.308168+0000 mgr.x (mgr.24542) 1540 : cluster [DBG] pgmap v1951: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2021-07-31T02:23:14.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:14 smithi045 conmon[39718]: cluster 2021-07-31T02:23:12.308168+0000 mgr.x ( 2021-07-31T02:23:14.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:14 smithi045 conmon[39718]: mgr.24542) 1540 : cluster [DBG] pgmap v1951: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2021-07-31T02:23:15.398 INFO:tasks.workunit.client.0.smithi045.stdout: 1067: throughput=5MB/sec pending data=0 2021-07-31T02:23:15.398 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-qwmTvwV5Ch7f5bq off=4011164 len=942020 2021-07-31T02:23:15.403 INFO:tasks.workunit.client.0.smithi045.stdout:op 5436 completed, throughput=5MB/sec 2021-07-31T02:23:15.404 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-S0Lyf2nTsc-7Es4 off=8981433 len=1700430 2021-07-31T02:23:15.405 INFO:tasks.workunit.client.0.smithi045.stdout:op 5437 completed, throughput=5MB/sec 2021-07-31T02:23:15.406 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-wwuk5BifyE-Sgt5 off=11579567 len=1888764 2021-07-31T02:23:15.407 INFO:tasks.workunit.client.0.smithi045.stdout:op 5438 completed, throughput=5MB/sec 2021-07-31T02:23:16.408 INFO:tasks.workunit.client.0.smithi045.stdout: 1068: throughput=5MB/sec pending data=0 2021-07-31T02:23:16.408 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-j9F31oET05vaiki off=1385334 len=563421 2021-07-31T02:23:16.410 INFO:tasks.workunit.client.0.smithi045.stdout:op 5439 completed, throughput=5MB/sec 2021-07-31T02:23:16.410 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-wjDTRBJXb7SyACU off=4921161 len=461948 2021-07-31T02:23:16.415 INFO:tasks.workunit.client.0.smithi045.stdout:op 5440 completed, throughput=5MB/sec 2021-07-31T02:23:16.415 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ZtBnvncePtyennA off=287341 len=1438614 2021-07-31T02:23:16.422 INFO:tasks.workunit.client.0.smithi045.stdout:op 5441 completed, throughput=5MB/sec 2021-07-31T02:23:16.422 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-o6QGY6cRt3rVCE5 off=2367835 len=581307 2021-07-31T02:23:16.427 INFO:tasks.workunit.client.0.smithi045.stdout:op 5442 completed, throughput=5MB/sec 2021-07-31T02:23:16.427 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-qFxXtboiFKs0Kjb off=2533390 len=72799 2021-07-31T02:23:16.430 INFO:tasks.workunit.client.0.smithi045.stdout:op 5443 completed, throughput=5MB/sec 2021-07-31T02:23:16.430 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-whEuFPOtAr9eWhU off=11617444 len=1434694 2021-07-31T02:23:16.436 INFO:tasks.workunit.client.0.smithi045.stdout:op 5444 completed, throughput=5MB/sec 2021-07-31T02:23:16.437 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-jks5SpasLWORVOi off=12287652 len=458176 2021-07-31T02:23:16.440 INFO:tasks.workunit.client.0.smithi045.stdout:op 5445 completed, throughput=5MB/sec 2021-07-31T02:23:16.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:16 smithi154 conmon[34553]: cluster 2021-07-31T02:23:14.308745+0000 mgr.x (mgr.24542) 1541 : cluster [DBG] pgmap v1952: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 2.9 MiB/s rd, 3.5 MiB/s wr, 6 op/s 2021-07-31T02:23:16.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:16 smithi045 conmon[39718]: cluster 2021-07-31T02:23:14.308745+0000 mgr.x (mgr.24542) 1541 : cluster [DBG] pgmap v1952: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 2.9 MiB/s rd, 3.5 MiB/s wr, 6 op/s 2021-07-31T02:23:16.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:16 smithi045 conmon[33655]: cluster 2021-07-31T02:23:14.308745+0000 mgr.x (mgr.24542) 2021-07-31T02:23:16.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:16 smithi045 conmon[33655]: 1541 : cluster [DBG] pgmap v1952: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 2.9 MiB/s rd, 3.5 MiB/s wr, 6 op/s 2021-07-31T02:23:17.441 INFO:tasks.workunit.client.0.smithi045.stdout: 1069: throughput=5MB/sec pending data=0 2021-07-31T02:23:17.441 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-d5ldimMYHnHR7le off=3338134 len=2045431 2021-07-31T02:23:17.449 INFO:tasks.workunit.client.0.smithi045.stdout:op 5446 completed, throughput=5MB/sec 2021-07-31T02:23:17.449 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-409JHJk55Sjo_dq off=5504665 len=1145957 2021-07-31T02:23:17.450 INFO:tasks.workunit.client.0.smithi045.stdout:op 5447 completed, throughput=5MB/sec 2021-07-31T02:23:17.450 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-HQJc5b_Wulyn09P off=2945950 len=133578 2021-07-31T02:23:17.451 INFO:tasks.workunit.client.0.smithi045.stdout:op 5448 completed, throughput=5MB/sec 2021-07-31T02:23:17.451 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-7rI9WjROqqzSluQ off=3588852 len=1685933 2021-07-31T02:23:17.460 INFO:tasks.workunit.client.0.smithi045.stdout:op 5449 completed, throughput=5MB/sec 2021-07-31T02:23:17.461 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-hYxf191OEJO3bXB off=628269 len=738107 2021-07-31T02:23:17.462 INFO:tasks.workunit.client.0.smithi045.stdout:op 5450 completed, throughput=5MB/sec 2021-07-31T02:23:18.462 INFO:tasks.workunit.client.0.smithi045.stdout: 1070: throughput=5MB/sec pending data=0 2021-07-31T02:23:18.463 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-d-krXKtNOuGikeg off=11931215 len=1109152 2021-07-31T02:23:18.465 INFO:tasks.workunit.client.0.smithi045.stdout:op 5451 completed, throughput=5MB/sec 2021-07-31T02:23:18.466 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-l-F6p6h9gNP3HTU off=91586 len=1125653 2021-07-31T02:23:18.473 INFO:tasks.workunit.client.0.smithi045.stdout:op 5452 completed, throughput=5MB/sec 2021-07-31T02:23:18.473 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-tSC0RB3WK-iCmMK off=1031911 len=84344 2021-07-31T02:23:18.476 INFO:tasks.workunit.client.0.smithi045.stdout:op 5453 completed, throughput=5MB/sec 2021-07-31T02:23:18.476 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-YYl_7w91fwd34sC off=1989114 len=1324728 2021-07-31T02:23:18.478 INFO:tasks.workunit.client.0.smithi045.stdout:op 5454 completed, throughput=5MB/sec 2021-07-31T02:23:18.479 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-DarC72gu78dgUA0 off=245173 len=820573 2021-07-31T02:23:18.479 INFO:tasks.workunit.client.0.smithi045.stdout:op 5455 completed, throughput=5MB/sec 2021-07-31T02:23:18.480 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-o864C5EWS1Q3z8t off=4223089 len=715543 2021-07-31T02:23:18.484 INFO:tasks.workunit.client.0.smithi045.stdout:op 5456 completed, throughput=5MB/sec 2021-07-31T02:23:18.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:18 smithi154 conmon[34553]: cluster 2021-07-31T02:23:16.309217+0000 mgr.x (mgr.24542) 1542 : cluster [DBG] pgmap v1953: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:23:18.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:18 smithi045 conmon[33655]: cluster 2021-07-31T02:23:16.309217+0000 mgr.x (mgr.24542) 1542 2021-07-31T02:23:18.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:18 smithi045 conmon[33655]: : cluster [DBG] pgmap v1953: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:23:18.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:18 smithi045 conmon[39718]: cluster 2021-07-31T02:23:16.309217+0000 mgr.x (mgr. 2021-07-31T02:23:18.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:18 smithi045 conmon[39718]: 24542) 1542 : cluster [DBG] pgmap v1953: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:23:19.484 INFO:tasks.workunit.client.0.smithi045.stdout: 1071: throughput=5MB/sec pending data=0 2021-07-31T02:23:19.485 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-IdFBjujaqT0fxSO off=3550453 len=1331312 2021-07-31T02:23:19.486 INFO:tasks.workunit.client.0.smithi045.stdout:op 5457 completed, throughput=5MB/sec 2021-07-31T02:23:19.486 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-B125fNl1cpYf17u off=1209094 len=33724 2021-07-31T02:23:19.489 INFO:tasks.workunit.client.0.smithi045.stdout:op 5458 completed, throughput=5MB/sec 2021-07-31T02:23:19.489 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-_B1t4AwZYZCNsQO off=1459316 len=1623031 2021-07-31T02:23:19.497 INFO:tasks.workunit.client.0.smithi045.stdout:op 5459 completed, throughput=5MB/sec 2021-07-31T02:23:19.497 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-4H2jdJu1kWyMJ6w off=1112142 len=1791725 2021-07-31T02:23:19.500 INFO:tasks.workunit.client.0.smithi045.stdout:op 5460 completed, throughput=5MB/sec 2021-07-31T02:23:19.500 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Xn7VyPwAfOVCTwM off=2210274 len=1385603 2021-07-31T02:23:19.503 INFO:tasks.workunit.client.0.smithi045.stdout:op 5461 completed, throughput=5MB/sec 2021-07-31T02:23:20.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:20 smithi154 conmon[34553]: cluster 2021-07-31T02:23:18.309589+0000 mgr.x (mgr.24542) 1543 : cluster [DBG] pgmap v1954: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:23:20.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:20 smithi154 conmon[34553]: 2021-07-31T02:23:20.504 INFO:tasks.workunit.client.0.smithi045.stdout: 1072: throughput=5MB/sec pending data=0 2021-07-31T02:23:20.504 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-aLmXehjqrlK4VGi off=7478741 len=1055822 2021-07-31T02:23:20.506 INFO:tasks.workunit.client.0.smithi045.stdout:op 5462 completed, throughput=5MB/sec 2021-07-31T02:23:20.506 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Lb8vFgjipV5oteL off=9613926 len=1872277 2021-07-31T02:23:20.518 INFO:tasks.workunit.client.0.smithi045.stdout:op 5463 completed, throughput=5MB/sec 2021-07-31T02:23:20.519 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-SbTD1-bUYMr3aRS off=1700373 len=1898043 2021-07-31T02:23:20.523 INFO:tasks.workunit.client.0.smithi045.stdout:op 5464 completed, throughput=5MB/sec 2021-07-31T02:23:20.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:20 smithi045 conmon[39718]: cluster 2021-07-31T02:23:18.309589+0000 mgr.x (mgr.24542) 1543 : cluster [DBG] pgmap v1954: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:23:20.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:20 smithi045 conmon[33655]: cluster 2021-07-31T02:23:18.309589+0000 mgr.x (mgr.24542) 1543 : cluster [DBG] 2021-07-31T02:23:20.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:20 smithi045 conmon[33655]: pgmap v1954: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:23:21.523 INFO:tasks.workunit.client.0.smithi045.stdout: 1073: throughput=5MB/sec pending data=0 2021-07-31T02:23:21.524 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-U0ShOLlBPb4pMwh off=12317973 len=416525 2021-07-31T02:23:21.524 INFO:tasks.workunit.client.0.smithi045.stdout:op 5465 completed, throughput=5MB/sec 2021-07-31T02:23:21.524 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-vjXTiN0e_pgyUbE off=6701561 len=179428 2021-07-31T02:23:21.527 INFO:tasks.workunit.client.0.smithi045.stdout:op 5466 completed, throughput=5MB/sec 2021-07-31T02:23:21.528 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-S5amb7c9-Bmz9ds off=14283941 len=1455425 2021-07-31T02:23:21.534 INFO:tasks.workunit.client.0.smithi045.stdout:op 5467 completed, throughput=5MB/sec 2021-07-31T02:23:21.534 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-kByHflCSSSUFB7S off=2568431 len=1160744 2021-07-31T02:23:21.540 INFO:tasks.workunit.client.0.smithi045.stdout:op 5468 completed, throughput=5MB/sec 2021-07-31T02:23:21.542 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-2xMGUFe9qku31i5 off=17511377 len=138746 2021-07-31T02:23:21.543 INFO:tasks.workunit.client.0.smithi045.stdout:op 5469 completed, throughput=5MB/sec 2021-07-31T02:23:21.543 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-CjrBK4Psa9z6ez6 off=227917 len=1670135 2021-07-31T02:23:21.550 INFO:tasks.workunit.client.0.smithi045.stdout:op 5470 completed, throughput=5MB/sec 2021-07-31T02:23:22.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:22 smithi154 conmon[34553]: cluster 2021-07-31T02:23:20.310035+0000 mgr.x (mgr.24542) 1544 : cluster 2021-07-31T02:23:22.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:22 smithi154 conmon[34553]: [DBG] pgmap v1955: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 2.9 MiB/s rd, 3.4 MiB/s wr, 6 op/s 2021-07-31T02:23:22.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:22 smithi045 conmon[33655]: cluster 2021-07-31T02:23:20.310035+0000 mgr.x (mgr.24542 2021-07-31T02:23:22.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:22 smithi045 conmon[33655]: ) 1544 : cluster [DBG] pgmap v1955: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 2.9 MiB/s rd, 3.4 MiB/s wr, 6 op/s 2021-07-31T02:23:22.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:22 smithi045 conmon[39718]: cluster 2021-07-31T02:23:20.310035+0000 2021-07-31T02:23:22.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:22 smithi045 conmon[39718]: mgr.x (mgr.24542) 1544 : cluster [DBG] pgmap v1955: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 2.9 MiB/s rd, 3.4 MiB/s wr, 6 op/s 2021-07-31T02:23:22.550 INFO:tasks.workunit.client.0.smithi045.stdout: 1074: throughput=5MB/sec pending data=0 2021-07-31T02:23:22.551 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-1w951G-djxOzbKa off=4341924 len=275184 2021-07-31T02:23:22.551 INFO:tasks.workunit.client.0.smithi045.stdout:op 5471 completed, throughput=5MB/sec 2021-07-31T02:23:22.551 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-np0Wo9s6wfdJyTf off=4707415 len=854300 2021-07-31T02:23:22.556 INFO:tasks.workunit.client.0.smithi045.stdout:op 5472 completed, throughput=5MB/sec 2021-07-31T02:23:22.556 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-GZ9w18NqGZowdHF off=676591 len=1683966 2021-07-31T02:23:22.564 INFO:tasks.workunit.client.0.smithi045.stdout:op 5473 completed, throughput=5MB/sec 2021-07-31T02:23:22.565 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-QuK9vaSM-XPGOMu off=4350976 len=1675602 2021-07-31T02:23:22.572 INFO:tasks.workunit.client.0.smithi045.stdout:op 5474 completed, throughput=5MB/sec 2021-07-31T02:23:22.572 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-fBCunYw1I9ZdN-T off=2738474 len=847933 2021-07-31T02:23:22.578 INFO:tasks.workunit.client.0.smithi045.stdout:op 5475 completed, throughput=5MB/sec 2021-07-31T02:23:23.578 INFO:tasks.workunit.client.0.smithi045.stdout: 1075: throughput=5MB/sec pending data=0 2021-07-31T02:23:23.578 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-9o02a4RLt08tkK9 off=2043836 len=1694184 2021-07-31T02:23:23.586 INFO:tasks.workunit.client.0.smithi045.stdout:op 5476 completed, throughput=5MB/sec 2021-07-31T02:23:23.586 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-G8kAbr4fjxpMiQL off=9220510 len=1534024 2021-07-31T02:23:23.589 INFO:tasks.workunit.client.0.smithi045.stdout:op 5477 completed, throughput=5MB/sec 2021-07-31T02:23:23.589 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-FLwjeyXfWv6E-dv off=635086 len=1258459 2021-07-31T02:23:23.590 INFO:tasks.workunit.client.0.smithi045.stdout:op 5478 completed, throughput=5MB/sec 2021-07-31T02:23:23.590 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-L5VDmMU_b5XZuOY off=2903240 len=1084907 2021-07-31T02:23:23.591 INFO:tasks.workunit.client.0.smithi045.stdout:op 5479 completed, throughput=5MB/sec 2021-07-31T02:23:24.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:23:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:23:23.571Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:23:24.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:23:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:23:23.571Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:23:24.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:23:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:23:23.571Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:23:24.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:24 smithi154 conmon[34553]: cluster 2021-07-31T02:23:22.310471+0000 mgr.x (mgr.24542) 1545 : cluster [DBG] 2021-07-31T02:23:24.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:24 smithi154 conmon[34553]: pgmap v1956: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2021-07-31T02:23:24.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:24 smithi045 conmon[33655]: cluster 2021-07-31T02:23:22.310471+0000 mgr.x (mgr.24542) 1545 : cluster [DBG] 2021-07-31T02:23:24.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:24 smithi045 conmon[33655]: pgmap v1956: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2021-07-31T02:23:24.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:24 smithi045 conmon[39718]: cluster 2021-07-31T02:23:22.310471+0000 mgr.x (mgr 2021-07-31T02:23:24.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:24 smithi045 conmon[39718]: .24542) 1545 : cluster [DBG] pgmap v1956: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2021-07-31T02:23:24.592 INFO:tasks.workunit.client.0.smithi045.stdout: 1076: throughput=5MB/sec pending data=0 2021-07-31T02:23:24.592 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-odwa6d1_84s-ob8 off=18234524 len=56795 2021-07-31T02:23:24.595 INFO:tasks.workunit.client.0.smithi045.stdout:op 5480 completed, throughput=5MB/sec 2021-07-31T02:23:24.595 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-PzFaG8xhMs59pVp off=2694275 len=1464031 2021-07-31T02:23:24.598 INFO:tasks.workunit.client.0.smithi045.stdout:op 5481 completed, throughput=5MB/sec 2021-07-31T02:23:24.598 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-tPDd0kBILAVFfwk off=2689901 len=576783 2021-07-31T02:23:24.602 INFO:tasks.workunit.client.0.smithi045.stdout:op 5482 completed, throughput=5MB/sec 2021-07-31T02:23:24.603 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-a61NGCEqK5wuL2X off=4178243 len=59007 2021-07-31T02:23:24.603 INFO:tasks.workunit.client.0.smithi045.stdout:op 5483 completed, throughput=5MB/sec 2021-07-31T02:23:24.604 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-NGh0UA0UIezgVcd off=1321228 len=769988 2021-07-31T02:23:24.606 INFO:tasks.workunit.client.0.smithi045.stdout:op 5484 completed, throughput=5MB/sec 2021-07-31T02:23:24.606 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-zB2bxEZRkZGGzeA off=203711 len=1417187 2021-07-31T02:23:24.613 INFO:tasks.workunit.client.0.smithi045.stdout:op 5485 completed, throughput=5MB/sec 2021-07-31T02:23:24.613 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-lY972B9-s0SsSHJ off=7896116 len=670303 2021-07-31T02:23:24.614 INFO:tasks.workunit.client.0.smithi045.stdout:op 5486 completed, throughput=5MB/sec 2021-07-31T02:23:25.614 INFO:tasks.workunit.client.0.smithi045.stdout: 1077: throughput=5MB/sec pending data=0 2021-07-31T02:23:25.614 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-XTlefbHOHcyv5Hn off=14195291 len=754052 2021-07-31T02:23:25.620 INFO:tasks.workunit.client.0.smithi045.stdout:op 5487 completed, throughput=5MB/sec 2021-07-31T02:23:25.620 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-FnAkMzJw01DzqG7 off=20865732 len=715367 2021-07-31T02:23:25.625 INFO:tasks.workunit.client.0.smithi045.stdout:op 5488 completed, throughput=5MB/sec 2021-07-31T02:23:25.625 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-kiBU4DmXj_bO0fp off=1127611 len=49569 2021-07-31T02:23:25.626 INFO:tasks.workunit.client.0.smithi045.stdout:op 5489 completed, throughput=5MB/sec 2021-07-31T02:23:25.626 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-k-KcgXJkGB7jovB off=10211300 len=480652 2021-07-31T02:23:25.627 INFO:tasks.workunit.client.0.smithi045.stdout:op 5490 completed, throughput=5MB/sec 2021-07-31T02:23:25.627 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-uozWcmoHeIOIkJZ off=3211098 len=2072506 2021-07-31T02:23:25.636 INFO:tasks.workunit.client.0.smithi045.stdout:op 5491 completed, throughput=5MB/sec 2021-07-31T02:23:25.636 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-tAnoTRiMTQIO_S- off=12508961 len=1930918 2021-07-31T02:23:25.638 INFO:tasks.workunit.client.0.smithi045.stdout:op 5492 completed, throughput=5MB/sec 2021-07-31T02:23:26.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:26 smithi154 conmon[34553]: cluster 2021-07-31T02:23:24.310984+0000 mgr.x (mgr.24542) 1546 : cluster [DBG] 2021-07-31T02:23:26.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:26 smithi154 conmon[34553]: pgmap v1957: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 3.2 MiB/s rd, 3.2 MiB/s wr, 6 op/s 2021-07-31T02:23:26.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:26 smithi045 conmon[39718]: cluster 2021-07-31T02:23:24.310984+0000 mgr.x (mgr.24542) 1546 : cluster [DBG] pgmap v1957: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 3.2 MiB/s rd, 3.2 MiB/s wr, 6 op/s 2021-07-31T02:23:26.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:26 smithi045 conmon[33655]: cluster 2021-07-31T02:23:24.310984+0000 mgr.x (mgr 2021-07-31T02:23:26.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:26 smithi045 conmon[33655]: .24542) 1546 : cluster [DBG] pgmap v1957: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 3.2 MiB/s rd, 3.2 MiB/s wr, 6 op/s 2021-07-31T02:23:26.638 INFO:tasks.workunit.client.0.smithi045.stdout: 1078: throughput=5MB/sec pending data=0 2021-07-31T02:23:26.639 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-JgKxW92pUuVNz7a off=11221876 len=1257231 2021-07-31T02:23:26.640 INFO:tasks.workunit.client.0.smithi045.stdout:op 5493 completed, throughput=5MB/sec 2021-07-31T02:23:26.640 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-XPoKbgf56hbikKi off=12702169 len=1651059 2021-07-31T02:23:26.647 INFO:tasks.workunit.client.0.smithi045.stdout:op 5494 completed, throughput=5MB/sec 2021-07-31T02:23:26.647 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj--J_0rv1lf7N0t0N off=20573562 len=1897513 2021-07-31T02:23:26.648 INFO:tasks.workunit.client.0.smithi045.stdout:op 5495 completed, throughput=5MB/sec 2021-07-31T02:23:27.649 INFO:tasks.workunit.client.0.smithi045.stdout: 1079: throughput=5MB/sec pending data=0 2021-07-31T02:23:27.649 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-vl0_q5xjfalvxAs off=1349323 len=262481 2021-07-31T02:23:27.650 INFO:tasks.workunit.client.0.smithi045.stdout:op 5496 completed, throughput=5MB/sec 2021-07-31T02:23:27.650 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-fZxfHyECeAaZymQ off=10592862 len=759270 2021-07-31T02:23:27.655 INFO:tasks.workunit.client.0.smithi045.stdout:op 5497 completed, throughput=5MB/sec 2021-07-31T02:23:27.655 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-yESBP3Qgk4VkeIt off=5715332 len=1235537 2021-07-31T02:23:27.657 INFO:tasks.workunit.client.0.smithi045.stdout:op 5498 completed, throughput=5MB/sec 2021-07-31T02:23:27.658 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-tHgGIK6NKykdNRh off=3869955 len=1487404 2021-07-31T02:23:27.658 INFO:tasks.workunit.client.0.smithi045.stdout:op 5499 completed, throughput=5MB/sec 2021-07-31T02:23:27.659 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-osQwWlO3YXjQVpd off=6602892 len=360172 2021-07-31T02:23:27.661 INFO:tasks.workunit.client.0.smithi045.stdout:op 5500 completed, throughput=5MB/sec 2021-07-31T02:23:27.662 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-2x6FNHz5YXbK2MI off=4798890 len=1490185 2021-07-31T02:23:27.669 INFO:tasks.workunit.client.0.smithi045.stdout:op 5501 completed, throughput=5MB/sec 2021-07-31T02:23:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:28 smithi154 conmon[34553]: cluster 2021-07-31T02:23:26.311391+0000 mgr.x (mgr.24542) 1547 : cluster [DBG] pgmap v1958: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.2 MiB/s wr, 3 op/s 2021-07-31T02:23:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:28 smithi154 conmon[34553]: 2021-07-31T02:23:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:28 smithi154 conmon[34553]: audit 2021-07-31T02:23:28.057088+0000 mon.b (mon.2) 248 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:23:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:28 smithi154 conmon[34553]: audit 2021-07-31T02:23:28.057774+0000 mon.a (mon.0) 2021-07-31T02:23:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:28 smithi154 conmon[34553]: 1904 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:23:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:28 smithi154 conmon[34553]: audit 2021-07-31T02:23:28.080106+0000 mon.b (mon 2021-07-31T02:23:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:28 smithi154 conmon[34553]: .2) 249 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:23:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:28 smithi154 conmon[34553]: 2021-07-31T02:23:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:28 smithi154 conmon[34553]: audit 2021-07-31 2021-07-31T02:23:28.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:28 smithi154 conmon[34553]: T02:23:28.080712+0000 mon.a (mon 2021-07-31T02:23:28.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:28 smithi154 conmon[34553]: .0) 1905 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:23:28.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:28 smithi154 conmon[34553]: 2021-07-31T02:23:28.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:28 smithi045 conmon[33655]: cluster 2021-07-31T02:23:26.311391+0000 mgr.x (mgr.24542) 1547 : cluster 2021-07-31T02:23:28.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:28 smithi045 conmon[33655]: [DBG] pgmap v1958: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.2 MiB/s wr, 3 op/s 2021-07-31T02:23:28.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:28 smithi045 conmon[33655]: audit 2021-07-31T02:23:28.057088+0000 mon.b (mon.2) 248 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:23:28.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:28 smithi045 conmon[33655]: audit 2021-07-31T02:23:28.057774+0000 mon.a (mon.0) 1904 : audit [INF] 2021-07-31T02:23:28.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:28 smithi045 conmon[33655]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:23:28.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:28 smithi045 conmon[33655]: audit 2021-07-31T02:23:28 2021-07-31T02:23:28.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:28 smithi045 conmon[33655]: .080106+0000 mon.b (mon.2) 249 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:23:28.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:28 smithi045 conmon[33655]: audit 2021 2021-07-31T02:23:28.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:28 smithi045 conmon[33655]: -07-31T02:23:28.080712 2021-07-31T02:23:28.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:28 smithi045 conmon[33655]: +0000 mon.a (mon.0) 1905 : audit [INF] 2021-07-31T02:23:28.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:28 smithi045 conmon[33655]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:23:28.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:28 smithi045 conmon[39718]: cluster 2021-07-31T02:23:26.311391+0000 mgr.x 2021-07-31T02:23:28.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:28 smithi045 conmon[39718]: (mgr.24542) 1547 : cluster [DBG] pgmap v1958: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.2 MiB/s wr, 3 op/s 2021-07-31T02:23:28.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:28 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T02:23:28.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:28 smithi045 conmon[39718]: 31T02:23:28.057088+0000 mon.b (mon.2) 248 : 2021-07-31T02:23:28.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:28 smithi045 conmon[39718]: audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:23:28.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:28 smithi045 conmon[39718]: audit 2021-07 2021-07-31T02:23:28.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:28 smithi045 conmon[39718]: -31T02:23:28.057774+0000 2021-07-31T02:23:28.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:28 smithi045 conmon[39718]: mon.a (mon.0) 1904 : audit [INF] 2021-07-31T02:23:28.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:28 smithi045 conmon[39718]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:23:28.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:28 smithi045 conmon[39718]: audit 2021-07-31T02: 2021-07-31T02:23:28.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:28 smithi045 conmon[39718]: 23:28.080106+0000 mon.b (mon.2 2021-07-31T02:23:28.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:28 smithi045 conmon[39718]: ) 249 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:23:28.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:28 smithi045 conmon[39718]: audit 2021- 2021-07-31T02:23:28.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:28 smithi045 conmon[39718]: 07-31T02:23:28.080712+0000 2021-07-31T02:23:28.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:28 smithi045 conmon[39718]: mon.a (mon.0) 1905 : audit 2021-07-31T02:23:28.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:28 smithi045 conmon[39718]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:23:28.669 INFO:tasks.workunit.client.0.smithi045.stdout: 1080: throughput=5MB/sec pending data=0 2021-07-31T02:23:28.669 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Vdgfwt0GsQtXOaw off=11878196 len=398396 2021-07-31T02:23:28.673 INFO:tasks.workunit.client.0.smithi045.stdout:op 5502 completed, throughput=5MB/sec 2021-07-31T02:23:28.674 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-cWaRiff9Yc_pSfC off=15596088 len=1970400 2021-07-31T02:23:28.683 INFO:tasks.workunit.client.0.smithi045.stdout:op 5503 completed, throughput=5MB/sec 2021-07-31T02:23:28.683 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-YvHJyvd-lZAGP0X off=5541182 len=923366 2021-07-31T02:23:28.685 INFO:tasks.workunit.client.0.smithi045.stdout:op 5504 completed, throughput=5MB/sec 2021-07-31T02:23:28.685 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-HkqxcAd8uokLe0Y off=4252134 len=616601 2021-07-31T02:23:28.689 INFO:tasks.workunit.client.0.smithi045.stdout:op 5505 completed, throughput=5MB/sec 2021-07-31T02:23:28.689 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-uqqz7y-j_f_3RS0 off=5386961 len=442767 2021-07-31T02:23:28.690 INFO:tasks.workunit.client.0.smithi045.stdout:op 5506 completed, throughput=5MB/sec 2021-07-31T02:23:28.690 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ldwJi3O9Xv__eb1 off=18407362 len=428616 2021-07-31T02:23:28.693 INFO:tasks.workunit.client.0.smithi045.stdout:op 5507 completed, throughput=5MB/sec 2021-07-31T02:23:29.694 INFO:tasks.workunit.client.0.smithi045.stdout: 1081: throughput=5MB/sec pending data=0 2021-07-31T02:23:29.694 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-lqyFQQ3tawcvxOq off=3089322 len=1550046 2021-07-31T02:23:29.697 INFO:tasks.workunit.client.0.smithi045.stdout:op 5508 completed, throughput=5MB/sec 2021-07-31T02:23:29.697 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-mTSwDZUHFv4bWN_ off=2658829 len=1370317 2021-07-31T02:23:29.699 INFO:tasks.workunit.client.0.smithi045.stdout:op 5509 completed, throughput=5MB/sec 2021-07-31T02:23:29.699 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-yc0mFR4-STRCZKr off=11866065 len=163211 2021-07-31T02:23:29.702 INFO:tasks.workunit.client.0.smithi045.stdout:op 5510 completed, throughput=5MB/sec 2021-07-31T02:23:29.702 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-o8PxBdaErLwOAKJ off=4722030 len=1754562 2021-07-31T02:23:29.711 INFO:tasks.workunit.client.0.smithi045.stdout:op 5511 completed, throughput=5MB/sec 2021-07-31T02:23:29.711 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-xW-WerfjTzN2zSF off=12174007 len=1458534 2021-07-31T02:23:29.713 INFO:tasks.workunit.client.0.smithi045.stdout:op 5512 completed, throughput=5MB/sec 2021-07-31T02:23:30.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:30 smithi154 conmon[34553]: cluster 2021-07-31T02:23:28.311966+0000 mgr.x (mgr.24542) 1548 : cluster [DBG] 2021-07-31T02:23:30.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:30 smithi154 conmon[34553]: pgmap v1959: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.2 MiB/s wr, 3 op/s 2021-07-31T02:23:30.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:30 smithi045 conmon[33655]: cluster 2021-07-31T02:23:28.311966+0000 mgr.x (mgr 2021-07-31T02:23:30.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:30 smithi045 conmon[33655]: .24542) 1548 : cluster [DBG] pgmap v1959: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.2 MiB/s wr, 3 op/s 2021-07-31T02:23:30.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:30 smithi045 conmon[39718]: cluster 2021-07-31T02:23:28.311966+0000 mgr.x (mgr.24542) 1548 : cluster [DBG] 2021-07-31T02:23:30.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:30 smithi045 conmon[39718]: pgmap v1959: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.2 MiB/s wr, 3 op/s 2021-07-31T02:23:30.714 INFO:tasks.workunit.client.0.smithi045.stdout: 1082: throughput=5MB/sec pending data=0 2021-07-31T02:23:30.714 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-lVvMvHkkQi8-Xpt off=16449993 len=1731182 2021-07-31T02:23:30.723 INFO:tasks.workunit.client.0.smithi045.stdout:op 5513 completed, throughput=5MB/sec 2021-07-31T02:23:30.723 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-2ODqDUv9bAKif-a off=14011022 len=486904 2021-07-31T02:23:30.724 INFO:tasks.workunit.client.0.smithi045.stdout:op 5514 completed, throughput=5MB/sec 2021-07-31T02:23:30.725 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Y4lVTc_UAkQguQf off=975374 len=451982 2021-07-31T02:23:30.728 INFO:tasks.workunit.client.0.smithi045.stdout:op 5515 completed, throughput=5MB/sec 2021-07-31T02:23:30.728 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-CN_jfqXXLYZITft off=4203117 len=1007461 2021-07-31T02:23:30.730 INFO:tasks.workunit.client.0.smithi045.stdout:op 5516 completed, throughput=5MB/sec 2021-07-31T02:23:30.730 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-pRKKeQ3XBBtEeGl off=21158878 len=366248 2021-07-31T02:23:30.733 INFO:tasks.workunit.client.0.smithi045.stdout:op 5517 completed, throughput=5MB/sec 2021-07-31T02:23:30.734 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ROuMMgiaspMuMjE off=8341691 len=766755 2021-07-31T02:23:30.737 INFO:tasks.workunit.client.0.smithi045.stdout:op 5518 completed, throughput=5MB/sec 2021-07-31T02:23:31.497 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:23:31.738 INFO:tasks.workunit.client.0.smithi045.stdout: 1083: throughput=5MB/sec pending data=0 2021-07-31T02:23:31.738 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-7aiO_TSwYpncme9 off=5396260 len=1626791 2021-07-31T02:23:31.745 INFO:tasks.workunit.client.0.smithi045.stdout:op 5519 completed, throughput=5MB/sec 2021-07-31T02:23:31.745 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-634iXE_8Da6cG8A off=1532560 len=1475420 2021-07-31T02:23:31.747 INFO:tasks.workunit.client.0.smithi045.stdout:op 5520 completed, throughput=5MB/sec 2021-07-31T02:23:31.747 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Vo0ld3LGm7fR_8S off=19073133 len=526065 2021-07-31T02:23:31.748 INFO:tasks.workunit.client.0.smithi045.stdout:op 5521 completed, throughput=5MB/sec 2021-07-31T02:23:31.748 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-8-is6bb2SgjajmN off=475298 len=1918517 2021-07-31T02:23:31.758 INFO:tasks.workunit.client.0.smithi045.stdout:op 5522 completed, throughput=5MB/sec 2021-07-31T02:23:31.777 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:23:31.778 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 38m ago 39m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:23:31.778 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 36m ago 38m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:23:31.778 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 36m ago 42m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:23:31.779 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 38m ago 44m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:23:31.779 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 38m ago 44m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:23:31.779 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 36m ago 42m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:23:31.779 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 38m ago 42m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:23:31.779 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 38m ago 39m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:23:31.780 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 36m ago 39m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:23:31.780 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 38m ago 42m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:23:31.780 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 38m ago 41m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:23:31.780 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 38m ago 41m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:23:31.781 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 38m ago 41m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:23:31.781 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 36m ago 40m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:23:31.781 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 36m ago 40m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:23:31.781 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 36m ago 40m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:23:31.781 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 36m ago 39m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:23:31.782 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 36m ago 39m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:23:32.002 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: debug 2021-07-31T02:23:31.656+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:23:32.003 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:23:32.003 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:23:32.003 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: yield 2021-07-31T02:23:32.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:23:32.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:23:32.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:23:32.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:23:32.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:23:32.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:23:32.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:23:32.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:23:32.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:23:32.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: task.result() 2021-07-31T02:23:32.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:23:32.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: debug 2021-07-31T02:23:31.668+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:23:32.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:23:32.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:23:32.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: yield 2021-07-31T02:23:32.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:23:32.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:23:32.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:23:32.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:23:32.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:23:32.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:23:32.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:23:32.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:23:32.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:23:32.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: task.result() 2021-07-31T02:23:32.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:23:32.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: debug 2021-07-31T02:23:31.694+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:23:32.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:23:32.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:23:32.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: yield 2021-07-31T02:23:32.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:23:32.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:23:32.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:23:32.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:23:32.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:23:32.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:23:32.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:23:32.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:23:32.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:23:32.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: task.result() 2021-07-31T02:23:32.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:23:32.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: debug 2021-07-31T02:23:31.694+0000 7fcb93160700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:23:32.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:23:32.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:23:32.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: yield 2021-07-31T02:23:32.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:23:32.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:23:32.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:23:32.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:23:32.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:23:32.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:23:32.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:23:32.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:23:32.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:23:32.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: task.result() 2021-07-31T02:23:32.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:23:32.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: 2021-07-31T02:23:32.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:23:32.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: 2021-07-31T02:23:32.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:23:32.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:23:32.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:23:32.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:23:32.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:23:32.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:23:32.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:23:32.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:23:32.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:23:32.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:23:32.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:23:32.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:23:32.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:23:32.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:23:32.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:23:32.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:23:32.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:23:32.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:23:32.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: raise 2021-07-31T02:23:32.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:23:32.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:23:32.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:23:32.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:23:32.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:23:32.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: debug 2021-07-31T02:23:31.703+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:23:32.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:23:32.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:23:32.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: yield 2021-07-31T02:23:32.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:23:32.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:23:32.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:23:32.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:23:32.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:23:32.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:23:32.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:23:32.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:23:32.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:23:32.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: task.result() 2021-07-31T02:23:32.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:23:32.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: debug 2021-07-31T02:23:31.703+0000 7fcb9295f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:23:32.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:23:32.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:23:32.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: yield 2021-07-31T02:23:32.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:23:32.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:23:32.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:23:32.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:23:32.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:23:32.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:23:32.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:23:32.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:23:32.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:23:32.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: task.result() 2021-07-31T02:23:32.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:23:32.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: 2021-07-31T02:23:32.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:23:32.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: 2021-07-31T02:23:32.035 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:23:32.035 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:23:32.035 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:23:32.035 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:23:32.036 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:23:32.036 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:23:32.036 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:23:32.036 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:23:32.037 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:23:32.037 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:23:32.037 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:23:32.037 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:23:32.037 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:23:32.038 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:23:32.038 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:23:32.038 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:23:32.038 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:23:32.038 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:23:32.039 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: raise 2021-07-31T02:23:32.039 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:23:32.039 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:23:32.039 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:23:32.040 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:23:32.040 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:23:31 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:23:32.118 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:23:32.118 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:23:32.119 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:23:32.119 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:23:32.119 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:23:32.120 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:23:32.120 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:23:32.120 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:23:32.121 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:23:32.121 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:23:32.121 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:23:32.122 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:23:32.122 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:23:32.123 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:23:32.123 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:23:32.123 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:23:32.124 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:23:32.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:32 smithi154 conmon[34553]: cluster 2021-07-31T02:23:30.312454+0000 mgr.x (mgr.24542) 1549 : cluster [DBG] pgmap v1960: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 3.0 MiB/s rd, 3.2 MiB/s wr, 6 op/s 2021-07-31T02:23:32.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:32 smithi154 conmon[34553]: audit 2021-07-31T02:23: 2021-07-31T02:23:32.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:32 smithi154 conmon[34553]: 31.612517+0000 mon.b (mon.2) 250 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:23:32.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:32 smithi154 conmon[34553]: audit 2021-07-31T02:23:31.613938+0000 mon.b (mon.2) 251 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:23:32.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:32 smithi154 conmon[34553]: audit 2021-07- 2021-07-31T02:23:32.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:32 smithi154 conmon[34553]: 31T02:23:31.614854+0000 mon.b (mon.2) 252 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:23:32.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:32 smithi154 conmon[34553]: audit 2021- 2021-07-31T02:23:32.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:32 smithi154 conmon[34553]: 07-31T02:23:32.118128+0000 2021-07-31T02:23:32.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:32 smithi154 conmon[34553]: mon.a (mon.0) 1906 : audit [DBG] from='client.? 172.21.15.45:0/1023479230' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:23:32.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:32 smithi045 conmon[39718]: cluster 2021-07-31T02 2021-07-31T02:23:32.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:32 smithi045 conmon[39718]: :23:30.312454+0000 mgr.x (mgr.24542) 1549 : cluster [DBG] pgmap v1960: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 3.0 MiB/s rd, 3.2 MiB/s wr, 6 op/s 2021-07-31T02:23:32.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:32 smithi045 conmon[39718]: audit 2021 2021-07-31T02:23:32.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:32 smithi045 conmon[39718]: -07-31T02:23: 2021-07-31T02:23:32.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:32 smithi045 conmon[39718]: 31.612517+0000 mon.b (mon 2021-07-31T02:23:32.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:32 smithi045 conmon[39718]: .2) 250 : audit [DBG] 2021-07-31T02:23:32.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:32 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:23:32.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:32 smithi045 conmon[39718]: audit 2021-07-31T02:23:32.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:32 smithi045 conmon[39718]: 2021-07-31T02:23 2021-07-31T02:23:32.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:32 smithi045 conmon[39718]: :31.613938+0000 mon.b 2021-07-31T02:23:32.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:32 smithi045 conmon[39718]: (mon.2) 251 : audit 2021-07-31T02:23:32.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:32 smithi045 conmon[39718]: [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:23:32.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:32 smithi045 conmon[39718]: audit 2021 2021-07-31T02:23:32.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:32 smithi045 conmon[39718]: -07-31T02:23: 2021-07-31T02:23:32.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:32 smithi045 conmon[39718]: 31.614854+0000 mon.b (mon. 2021-07-31T02:23:32.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:32 smithi045 conmon[39718]: 2) 252 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:23:32.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:32 smithi045 conmon[39718]: 2021-07-31T02:23:32.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:32 smithi045 conmon[39718]: audit 2021 2021-07-31T02:23:32.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:32 smithi045 conmon[39718]: -07-31T02:23: 2021-07-31T02:23:32.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:32 smithi045 conmon[39718]: 32.118128+0000 mon.a (mon 2021-07-31T02:23:32.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:32 smithi045 conmon[39718]: .0) 1906 : audit [DBG] 2021-07-31T02:23:32.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:32 smithi045 conmon[39718]: from='client.? 172.21.15.45:0/1023479230' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:23:32.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:32 smithi045 conmon[33655]: cluster 2021-07-31T02:23:30.312454+0000 mgr.x 2021-07-31T02:23:32.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:32 smithi045 conmon[33655]: (mgr.24542) 1549 : cluster [DBG] pgmap v1960: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 3.0 MiB/s rd, 3.2 MiB/s wr, 6 op/s 2021-07-31T02:23:32.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:32 smithi045 conmon[33655]: audit 2021-07-31T02:23:31.612517+0000 2021-07-31T02:23:32.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:32 smithi045 conmon[33655]: mon.b (mon.2) 250 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:23:32.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:32 smithi045 conmon[33655]: audit 2021-07-31T02:23:32.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:32 smithi045 conmon[33655]: 2021-07-31T02:23:31.613938 2021-07-31T02:23:32.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:32 smithi045 conmon[33655]: +0000 mon.b (mon.2) 251 : audit [DBG] 2021-07-31T02:23:32.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:32 smithi045 conmon[33655]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:23:32.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:32 smithi045 conmon[33655]: audit 2021-07-31T02:23: 2021-07-31T02:23:32.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:32 smithi045 conmon[33655]: 31.614854+0000 mon.b (mon.2) 252 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:23:32.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:32 smithi045 conmon[33655]: audit 2021-07-31T02:23:32 2021-07-31T02:23:32.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:32 smithi045 conmon[33655]: .118128+0000 mon.a (mon.0) 1906 : audit [DBG] from='client.? 172.21.15.45:0/1023479230' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:23:32.758 INFO:tasks.workunit.client.0.smithi045.stdout: 1084: throughput=5MB/sec pending data=0 2021-07-31T02:23:32.759 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-bcFH6UYkMCUUlut off=10591782 len=1192159 2021-07-31T02:23:32.761 INFO:tasks.workunit.client.0.smithi045.stdout:op 5523 completed, throughput=5MB/sec 2021-07-31T02:23:32.761 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-da-pdq-qqsTQdxL off=8227684 len=1554088 2021-07-31T02:23:32.769 INFO:tasks.workunit.client.0.smithi045.stdout:op 5524 completed, throughput=5MB/sec 2021-07-31T02:23:32.769 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-JkfKrHeVXrdzqIq off=13753153 len=111675 2021-07-31T02:23:32.772 INFO:tasks.workunit.client.0.smithi045.stdout:op 5525 completed, throughput=5MB/sec 2021-07-31T02:23:32.773 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Ud_thX_63dt5K0n off=14994946 len=602553 2021-07-31T02:23:32.776 INFO:tasks.workunit.client.0.smithi045.stdout:op 5526 completed, throughput=5MB/sec 2021-07-31T02:23:32.776 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Drxe0cyYZYEXmMC off=15288304 len=909380 2021-07-31T02:23:32.778 INFO:tasks.workunit.client.0.smithi045.stdout:op 5527 completed, throughput=5MB/sec 2021-07-31T02:23:32.778 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-iAh-RmEawGwMY5p off=2176201 len=318976 2021-07-31T02:23:32.779 INFO:tasks.workunit.client.0.smithi045.stdout:op 5528 completed, throughput=5MB/sec 2021-07-31T02:23:32.779 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-FSvxA5rhkgtY6Pf off=1371135 len=1360398 2021-07-31T02:23:32.780 INFO:tasks.workunit.client.0.smithi045.stdout:op 5529 completed, throughput=5MB/sec 2021-07-31T02:23:33.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: audit 2021-07-31T02:23:31.485682+0000 mgr.x (mgr.24542) 1550 : audit [DBG] from='client.26272 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:23:33.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:23:31.615636+0000 mgr.x (mgr.24542) 1551 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:23:33.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:23:31.657641+0000 mgr.x (mgr.24542) 1552 : cephadm [ERR] 2021-07-31T02:23:33.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:23:33.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:23:33.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: yield 2021-07-31T02:23:33.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:23:33.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:23:33.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:23:33.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:23:33.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:23:33.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:23:33.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:23:33.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:23:33.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:23:33.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:23:33.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:23:33.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:23:31.658069+0000 mgr.x (mgr.24542) 1553 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:23:33.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:23:31.669663+0000 mgr.x (mgr.24542) 1554 : cephadm [ERR] 2021-07-31T02:23:33.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:23:33.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:23:33.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: yield 2021-07-31T02:23:33.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:23:33.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:23:33.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:23:33.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:23:33.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:23:33.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:23:33.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:23:33.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:23:33.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:23:33.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:23:33.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:23:33.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:23:31.670473+0000 mgr.x (mgr.24542) 1555 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:23:33.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:23:31.695133+0000 mgr.x (mgr.24542) 1556 : cephadm [ERR] 2021-07-31T02:23:33.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:23:33.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:23:33.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: yield 2021-07-31T02:23:33.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:23:33.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:23:33.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:23:33.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:23:33.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:23:33.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:23:33.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:23:33.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:23:33.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:23:33.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:23:33.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:23:33.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: 2021-07-31T02:23:33.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:23:31.695759+0000 mgr.x (mgr.24542) 1557 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:23:33.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:23:33.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:23:33.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: yield 2021-07-31T02:23:33.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:23:33.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:23:33.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:23:33.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:23:33.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:23:33.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:23:33.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:23:33.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:23:33.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:23:33.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:23:33.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:23:33.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: 2021-07-31T02:23:33.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:23:33.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: 2021-07-31T02:23:33.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:23:33.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:23:33.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:23:33.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:23:33.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:23:33.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:23:33.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:23:33.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:23:33.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:23:33.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:23:33.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:23:33.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:23:33.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:23:33.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:23:33.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:23:33.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:23:33.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:23:33.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:23:33.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: raise 2021-07-31T02:23:33.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:23:33.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:23:33.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:23:33.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:23:33.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:23:33.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:23:31.704182+0000 mgr.x ( 2021-07-31T02:23:33.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: mgr.24542) 1558 : cephadm [ERR] 2021-07-31T02:23:33.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:23:33.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:23:33.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: yield 2021-07-31T02:23:33.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:23:33.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:23:33.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:23:33.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:23:33.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:23:33.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:23:33.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:23:33.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:23:33.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:23:33.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:23:33.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:23:33.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:23:31.704582+0000 mgr.x (mgr.24542) 1559 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:23:33.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:23:33.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:23:33.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: yield 2021-07-31T02:23:33.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:23:33.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:23:33.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:23:33.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:23:33.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:23:33.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:23:33.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:23:33.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:23:33.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:23:33.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:23:33.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:23:33.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: 2021-07-31T02:23:33.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:23:33.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: 2021-07-31T02:23:33.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:23:33.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:23:33.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:23:33.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:23:33.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:23:33.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:23:33.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:23:33.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:23:33.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:23:33.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:23:33.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:23:33.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:23:33.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:23:33.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:23:33.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:23:33.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:23:33.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:23:33.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:23:33.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: raise 2021-07-31T02:23:33.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:23:33.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:23:33.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:23:33.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:23:33.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:23:33.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:33 smithi154 conmon[34553]: audit 2021-07-31T02:23:31.771080+0000 mgr.x (mgr.24542) 1560 : audit [DBG] from='client.26278 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:23:33.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: audit 2021-07-31T02:23:31.485682+0000 mgr.x (mgr 2021-07-31T02:23:33.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: .24542) 1550 : audit [DBG] from='client.26272 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:23:33.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: cephadm 2021-07- 2021-07-31T02:23:33.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: 31T02:23:31.615636+0000 mgr.x (mgr 2021-07-31T02:23:33.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: .24542) 1551 : cephadm [WRN] 2021-07-31T02:23:33.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:23:33.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: cephadm 2021- 2021-07-31T02:23:33.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: 07-31T02:23:31.657641 2021-07-31T02:23:33.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 1552 : cephadm 2021-07-31T02:23:33.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: [ERR] 2021-07-31T02:23:33.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:23:33.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:23:33.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: yield 2021-07-31T02:23:33.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:23:33.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:23:33.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:23:33.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:23:33.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:23:33.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:23:33.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:23:33.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:23:33.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:23:33.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:23:33.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:23:33.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:23:31.658069+0000 mgr.x (mgr.24542) 1553 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:23:33.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:23:31.669663+0000 2021-07-31T02:23:33.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: mgr.x (mgr.24542) 1554 : cephadm [ERR] 2021-07-31T02:23:33.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:23:33.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:23:33.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: yield 2021-07-31T02:23:33.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:23:33.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:23:33.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:23:33.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:23:33.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:23:33.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:23:33.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:23:33.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:23:33.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:23:33.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:23:33.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:23:33.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:23:31.670473+0000 mgr.x (mgr.24542) 1555 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:23:33.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:23:33.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: 2021-07-31T02:23 2021-07-31T02:23:33.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: :31.695133+0000 mgr.x (mgr.24542 2021-07-31T02:23:33.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: ) 1556 : cephadm [ERR] 2021-07-31T02:23:33.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:23:33.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:23:33.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: yield 2021-07-31T02:23:33.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:23:33.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:23:33.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:23:33.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:23:33.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:23:33.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:23:33.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:23:33.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:23:33.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:23:33.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:23:33.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:23:33.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: 2021-07-31T02:23:33.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:23:31.695759+0000 mgr.x (mgr.24542) 1557 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:23:33.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:23:33.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:23:33.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: yield 2021-07-31T02:23:33.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:23:33.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:23:33.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:23:33.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:23:33.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:23:33.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:23:33.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:23:33.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:23:33.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:23:33.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:23:33.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:23:33.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: 2021-07-31T02:23:33.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:23:33.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: 2021-07-31T02:23:33.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:23:33.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:23:33.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:23:33.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:23:33.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:23:33.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: audit 2021-07-31T02:23:31.485682+0000 mgr.x (mgr.24542) 1550 : audit 2021-07-31T02:23:33.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: [DBG] from='client.26272 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:23:33.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:23:31.615636+0000 mgr.x (mgr.24542) 1551 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:23:33.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: cephadm 2021-07-31T02: 2021-07-31T02:23:33.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: 23:31.657641+0000 mgr.x (mgr.24542) 1552 : cephadm [ERR] 2021-07-31T02:23:33.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:23:33.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:23:33.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: yield 2021-07-31T02:23:33.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:23:33.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:23:33.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:23:33.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:23:33.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:23:33.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:23:33.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:23:33.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:23:33.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:23:33.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:23:33.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:23:33.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:23:31.658069+0000 mgr.x (mgr.24542) 1553 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:23:33.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:23:31.669663+0000 mgr.x (mgr.24542) 1554 : 2021-07-31T02:23:33.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: cephadm [ERR] 2021-07-31T02:23:33.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:23:33.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:23:33.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: yield 2021-07-31T02:23:33.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:23:33.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:23:33.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:23:33.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:23:33.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:23:33.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:23:33.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:23:33.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:23:33.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:23:33.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:23:33.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:23:33.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:23:31.670473+0000 2021-07-31T02:23:33.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: mgr.x (mgr.24542) 1555 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:23:33.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:23:33.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:23:33.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:23:33.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:23:33.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:23:33.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:23:33.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:23:33.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:23:33.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:23:33.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:23:33.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:23:33.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:23:33.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:23:33.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: raise 2021-07-31T02:23:33.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:23:33.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:23:33.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:23:33.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:23:33.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:23:33.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:23:31.704182+0000 mgr.x (mgr.24542) 1558 : cephadm [ERR] 2021-07-31T02:23:33.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:23:33.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:23:33.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: yield 2021-07-31T02:23:33.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:23:33.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:23:33.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:23:33.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:23:33.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:23:33.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:23:33.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:23:33.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:23:33.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:23:33.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:23:33.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:23:33.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:23:31.704582+0000 mgr.x (mgr.24542) 1559 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:23:33.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:23:33.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:23:33.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: yield 2021-07-31T02:23:33.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:23:33.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:23:31.695133+0000 mgr.x (mgr.24542) 1556 : cephadm [ERR] 2021-07-31T02:23:33.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:23:33.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:23:33.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: yield 2021-07-31T02:23:33.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:23:33.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:23:33.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:23:33.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:23:33.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:23:33.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:23:33.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:23:33.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:23:33.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:23:33.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:23:33.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:23:33.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: cephadm 2021-07- 2021-07-31T02:23:33.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: 31T02:23:31.695759+0000 mgr.x (mgr.24542) 1557 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:23:33.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:23:33.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:23:33.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: yield 2021-07-31T02:23:33.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:23:33.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:23:33.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:23:33.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:23:33.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:23:33.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:23:33.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:23:33.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:23:33.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:23:33.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:23:33.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:23:33.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: 2021-07-31T02:23:33.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:23:33.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: 2021-07-31T02:23:33.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:23:33.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:23:33.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:23:33.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:23:33.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:23:33.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:23:33.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:23:33.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:23:33.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:23:33.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:23:33.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:23:33.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:23:33.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:23:33.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:23:33.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:23:33.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:23:33.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:23:33.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: 2021-07-31T02:23:33.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:23:33.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: 2021-07-31T02:23:33.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:23:33.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:23:33.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:23:33.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:23:33.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:23:33.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:23:33.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:23:33.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:23:33.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:23:33.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:23:33.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:23:33.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:23:33.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:23:33.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:23:33.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:23:33.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:23:33.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:23:33.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:23:33.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: raise 2021-07-31T02:23:33.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:23:33.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:23:33.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:23:33.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:23:33.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:23:33.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[39718]: audit 2021-07-31T02:23:31.771080+0000 mgr.x (mgr.24542) 1560 : audit [DBG] from='client.26278 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:23:33.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:23:33.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:23:33.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:23:33.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:23:33.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:23:33.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:23:33.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:23:33.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:23:33.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:23:33.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:23:33.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:23:33.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: raise 2021-07-31T02:23:33.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:23:33.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:23:33.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:23:33.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:23:33.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:23:33.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:23:31.704182+0000 mgr.x (mgr.24542) 1558 : cephadm 2021-07-31T02:23:33.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: [ERR] 2021-07-31T02:23:33.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:23:33.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:23:33.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: yield 2021-07-31T02:23:33.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:23:33.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:23:33.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:23:33.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:23:33.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:23:33.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:23:33.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:23:33.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:23:33.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:23:33.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:23:33.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:23:33.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:23:31.704582+0000 mgr.x (mgr.24542) 1559 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:23:33.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:23:33.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:23:33.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: yield 2021-07-31T02:23:33.608 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:23:33.608 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:23:33.608 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:23:33.608 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:23:33.608 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:23:33.609 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:23:33.609 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:23:33.609 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:23:33.609 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:23:33.609 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:23:33.610 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:23:33.610 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: 2021-07-31T02:23:33.610 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:23:33.610 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: 2021-07-31T02:23:33.611 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:23:33.611 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:23:33.611 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:23:33.611 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:23:33.611 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:23:33.612 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:23:33.612 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:23:33.612 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:23:33.612 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:23:33.613 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:23:33.613 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:23:33.613 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:23:33.613 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:23:33.613 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:23:33.614 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:23:33.614 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:23:33.614 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:23:33.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:23:33.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: raise 2021-07-31T02:23:33.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:23:33.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:23:33.616 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:23:33.616 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:23:33.616 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:23:33.616 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[33655]: audit 2021-07-31T02:23:31.771080+0000 mgr.x (mgr.24542) 1560 : audit [DBG] from='client.26278 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:23:33.780 INFO:tasks.workunit.client.0.smithi045.stdout: 1085: throughput=5MB/sec pending data=0 2021-07-31T02:23:33.781 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-PQOvnu8XGYxKagn off=15648500 len=102950 2021-07-31T02:23:33.783 INFO:tasks.workunit.client.0.smithi045.stdout:op 5530 completed, throughput=5MB/sec 2021-07-31T02:23:33.784 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-maLhorhRnKLw5lM off=173502 len=1357915 2021-07-31T02:23:33.786 INFO:tasks.workunit.client.0.smithi045.stdout:op 5531 completed, throughput=5MB/sec 2021-07-31T02:23:33.787 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-C44GUulS7QmdH3I off=20046289 len=656471 2021-07-31T02:23:33.791 INFO:tasks.workunit.client.0.smithi045.stdout:op 5532 completed, throughput=5MB/sec 2021-07-31T02:23:33.791 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-my_nCvzjc__wO1t off=4423286 len=1206709 2021-07-31T02:23:33.792 INFO:tasks.workunit.client.0.smithi045.stdout:op 5533 completed, throughput=5MB/sec 2021-07-31T02:23:33.793 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-jZYIG2ZbJ68K-yu off=9014300 len=1064965 2021-07-31T02:23:33.794 INFO:tasks.workunit.client.0.smithi045.stdout:op 5534 completed, throughput=5MB/sec 2021-07-31T02:23:34.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:23:33.572Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:23:34.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:23:33.572Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:23:34.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:23:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:23:33.572Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:23:34.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:34 smithi154 conmon[34553]: cluster 2021-07-31T02:23:32.312997+0000 mgr.x (mgr.24542) 1561 : cluster [DBG] pgmap v1961: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2021-07-31T02:23:34.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:34 smithi045 conmon[33655]: cluster 2021-07-31T02:23:32.312997+0000 mgr.x (mgr.24542) 1561 : cluster 2021-07-31T02:23:34.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:34 smithi045 conmon[33655]: [DBG] pgmap v1961: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2021-07-31T02:23:34.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:34 smithi045 conmon[39718]: cluster 2021-07-31T02:23:32.312997+0000 mgr.x 2021-07-31T02:23:34.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:34 smithi045 conmon[39718]: (mgr.24542) 1561 : cluster [DBG] pgmap v1961: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2021-07-31T02:23:34.795 INFO:tasks.workunit.client.0.smithi045.stdout: 1086: throughput=5MB/sec pending data=0 2021-07-31T02:23:34.795 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-YaJaiIcXcwfO3-F off=5468785 len=1293906 2021-07-31T02:23:34.804 INFO:tasks.workunit.client.0.smithi045.stdout:op 5535 completed, throughput=5MB/sec 2021-07-31T02:23:34.804 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-JIcX3dJn3EWCjZV off=11767208 len=262551 2021-07-31T02:23:34.805 INFO:tasks.workunit.client.0.smithi045.stdout:op 5536 completed, throughput=5MB/sec 2021-07-31T02:23:34.805 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Z2wfU8is6UBycQn off=388022 len=1591703 2021-07-31T02:23:34.807 INFO:tasks.workunit.client.0.smithi045.stdout:op 5537 completed, throughput=5MB/sec 2021-07-31T02:23:34.807 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-oKqzql7oLSoeTjX off=4039924 len=451782 2021-07-31T02:23:34.811 INFO:tasks.workunit.client.0.smithi045.stdout:op 5538 completed, throughput=5MB/sec 2021-07-31T02:23:34.811 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-7Xol3ngVn87QxMI off=52620 len=98476 2021-07-31T02:23:34.812 INFO:tasks.workunit.client.0.smithi045.stdout:op 5539 completed, throughput=5MB/sec 2021-07-31T02:23:34.812 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-5s7883YgCfw_K9- off=3807480 len=1641896 2021-07-31T02:23:34.814 INFO:tasks.workunit.client.0.smithi045.stdout:op 5540 completed, throughput=5MB/sec 2021-07-31T02:23:35.814 INFO:tasks.workunit.client.0.smithi045.stdout: 1087: throughput=5MB/sec pending data=0 2021-07-31T02:23:35.815 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj--EOSdDZpWd9HVtp off=12588039 len=1678003 2021-07-31T02:23:35.817 INFO:tasks.workunit.client.0.smithi045.stdout:op 5541 completed, throughput=5MB/sec 2021-07-31T02:23:35.817 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-KRfHIqOYSbLQw4_ off=3073683 len=876742 2021-07-31T02:23:35.818 INFO:tasks.workunit.client.0.smithi045.stdout:op 5542 completed, throughput=5MB/sec 2021-07-31T02:23:35.819 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-y-OKoXcgkv4yZzr off=2307697 len=1229936 2021-07-31T02:23:35.825 INFO:tasks.workunit.client.0.smithi045.stdout:op 5543 completed, throughput=5MB/sec 2021-07-31T02:23:35.825 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-QOUmlSQ_rGhSVKu off=18054506 len=138017 2021-07-31T02:23:35.828 INFO:tasks.workunit.client.0.smithi045.stdout:op 5544 completed, throughput=5MB/sec 2021-07-31T02:23:35.828 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-kd3_H6wVTIzZHWr off=0 len=1262547 2021-07-31T02:23:35.829 INFO:tasks.workunit.client.0.smithi045.stdout:op 5545 completed, throughput=5MB/sec 2021-07-31T02:23:36.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:36 smithi154 conmon[34553]: cluster 2021-07-31T02:23:34.313455+0000 mgr.x (mgr.24542) 1562 : cluster [DBG] pgmap v1962: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 3.2 MiB/s rd, 3.2 MiB/s wr, 6 op/s 2021-07-31T02:23:36.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:36 smithi045 conmon[33655]: cluster 2021-07-31T02:23:34.313455+0000 mgr.x (mgr.24542) 1562 : cluster [DBG] pgmap v1962: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 3.2 MiB/s rd, 3.2 MiB/s wr, 6 op/s 2021-07-31T02:23:36.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:36 smithi045 conmon[33655]: 2021-07-31T02:23:36.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:36 smithi045 conmon[39718]: cluster 2021-07-31T02:23:34 2021-07-31T02:23:36.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:36 smithi045 conmon[39718]: .313455+0000 mgr.x (mgr.24542) 1562 : cluster [DBG] pgmap v1962: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 3.2 MiB/s rd, 3.2 MiB/s wr, 6 op/s 2021-07-31T02:23:36.830 INFO:tasks.workunit.client.0.smithi045.stdout: 1088: throughput=5MB/sec pending data=0 2021-07-31T02:23:36.830 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-OXfmcT7cuwOIqq- off=343941 len=899454 2021-07-31T02:23:36.835 INFO:tasks.workunit.client.0.smithi045.stdout:op 5546 completed, throughput=5MB/sec 2021-07-31T02:23:36.836 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-NEtkQEltnRKLVK9 off=5743891 len=1574352 2021-07-31T02:23:36.837 INFO:tasks.workunit.client.0.smithi045.stdout:op 5547 completed, throughput=5MB/sec 2021-07-31T02:23:36.838 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Sg753MkkaunRyEs off=6838392 len=1743875 2021-07-31T02:23:36.849 INFO:tasks.workunit.client.0.smithi045.stdout:op 5548 completed, throughput=5MB/sec 2021-07-31T02:23:36.849 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-t8jP_NCcpvSY_6N off=474529 len=407824 2021-07-31T02:23:36.850 INFO:tasks.workunit.client.0.smithi045.stdout:op 5549 completed, throughput=5MB/sec 2021-07-31T02:23:36.850 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-1at_q_u7G_kM1Nt off=3351539 len=1158136 2021-07-31T02:23:36.856 INFO:tasks.workunit.client.0.smithi045.stdout:op 5550 completed, throughput=5MB/sec 2021-07-31T02:23:37.856 INFO:tasks.workunit.client.0.smithi045.stdout: 1089: throughput=5MB/sec pending data=0 2021-07-31T02:23:37.856 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-_cOZBY8jzJj3ISc off=15846550 len=385831 2021-07-31T02:23:37.860 INFO:tasks.workunit.client.0.smithi045.stdout:op 5551 completed, throughput=5MB/sec 2021-07-31T02:23:37.860 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-o2_W3xznTlEmqM_ off=5911990 len=252368 2021-07-31T02:23:37.861 INFO:tasks.workunit.client.0.smithi045.stdout:op 5552 completed, throughput=5MB/sec 2021-07-31T02:23:37.861 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ia26eqiBSBPtcqM off=729979 len=1334213 2021-07-31T02:23:37.863 INFO:tasks.workunit.client.0.smithi045.stdout:op 5553 completed, throughput=5MB/sec 2021-07-31T02:23:37.863 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-H8OcsMs4KR6r5R0 off=2495217 len=1027740 2021-07-31T02:23:37.864 INFO:tasks.workunit.client.0.smithi045.stdout:op 5554 completed, throughput=5MB/sec 2021-07-31T02:23:37.865 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-c1vLU_vCn6H-7b2 off=5692261 len=1052699 2021-07-31T02:23:37.870 INFO:tasks.workunit.client.0.smithi045.stdout:op 5555 completed, throughput=5MB/sec 2021-07-31T02:23:37.870 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-VJqR4LMct5ytUhr off=1877644 len=1644621 2021-07-31T02:23:37.871 INFO:tasks.workunit.client.0.smithi045.stdout:op 5556 completed, throughput=5MB/sec 2021-07-31T02:23:38.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:38 smithi154 conmon[34553]: cluster 2021-07-31T02:23:36.314018+0000 mgr.x (mgr.24542) 1563 : cluster [DBG] pgmap v1963: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2021-07-31T02:23:38.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:38 smithi045 conmon[39718]: cluster 2021-07-31T02:23:36.314018+0000 mgr.x (mgr.24542) 1563 : cluster 2021-07-31T02:23:38.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:38 smithi045 conmon[39718]: [DBG] pgmap v1963: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2021-07-31T02:23:38.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:38 smithi045 conmon[33655]: cluster 2021-07-31T02:23:36.314018+0000 mgr.x (mgr.24542) 1563 : cluster [DBG] 2021-07-31T02:23:38.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:38 smithi045 conmon[33655]: pgmap v1963: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2021-07-31T02:23:38.872 INFO:tasks.workunit.client.0.smithi045.stdout: 1090: throughput=5MB/sec pending data=0 2021-07-31T02:23:38.873 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Hb7dTrD9Z8Ci0o7 off=10368661 len=2034216 2021-07-31T02:23:38.884 INFO:tasks.workunit.client.0.smithi045.stdout:op 5557 completed, throughput=5MB/sec 2021-07-31T02:23:38.884 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-sbfdeI_WvhwtpWt off=9192734 len=510063 2021-07-31T02:23:38.885 INFO:tasks.workunit.client.0.smithi045.stdout:op 5558 completed, throughput=5MB/sec 2021-07-31T02:23:38.886 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-M1Cw6CIQrAnKr1i off=2491831 len=1032593 2021-07-31T02:23:38.888 INFO:tasks.workunit.client.0.smithi045.stdout:op 5559 completed, throughput=5MB/sec 2021-07-31T02:23:38.888 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-NSsgiAOAloDhBH6 off=11461604 len=1817756 2021-07-31T02:23:38.891 INFO:tasks.workunit.client.0.smithi045.stdout:op 5560 completed, throughput=5MB/sec 2021-07-31T02:23:39.892 INFO:tasks.workunit.client.0.smithi045.stdout: 1091: throughput=5MB/sec pending data=0 2021-07-31T02:23:39.892 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-mjjk1xsanXSHb0Q off=16867093 len=654485 2021-07-31T02:23:39.897 INFO:tasks.workunit.client.0.smithi045.stdout:op 5561 completed, throughput=5MB/sec 2021-07-31T02:23:39.897 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj--y9JncaqFnYuNuY off=4343506 len=1526621 2021-07-31T02:23:39.904 INFO:tasks.workunit.client.0.smithi045.stdout:op 5562 completed, throughput=5MB/sec 2021-07-31T02:23:39.905 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-RxqdKWFVXKgcwgi off=17184375 len=1177796 2021-07-31T02:23:39.906 INFO:tasks.workunit.client.0.smithi045.stdout:op 5563 completed, throughput=5MB/sec 2021-07-31T02:23:39.906 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-QY22276hXRFQcGz off=6262440 len=441846 2021-07-31T02:23:39.910 INFO:tasks.workunit.client.0.smithi045.stdout:op 5564 completed, throughput=5MB/sec 2021-07-31T02:23:39.911 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-kawgpE4id5sifHt off=8809461 len=1078911 2021-07-31T02:23:39.915 INFO:tasks.workunit.client.0.smithi045.stdout:op 5565 completed, throughput=5MB/sec 2021-07-31T02:23:40.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:40 smithi154 conmon[34553]: cluster 2021-07-31T02:23:38.314483+0000 mgr.x (mgr.24542) 1564 : cluster [DBG] pgmap v1964: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2021-07-31T02:23:40.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:40 smithi045 conmon[33655]: cluster 2021-07-31T02:23:38.314483+0000 mgr.x (mgr.24542) 1564 : cluster 2021-07-31T02:23:40.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:40 smithi045 conmon[33655]: [DBG] pgmap v1964: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2021-07-31T02:23:40.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:40 smithi045 conmon[39718]: cluster 2021-07-31T02:23:38.314483+0000 mgr.x ( 2021-07-31T02:23:40.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:40 smithi045 conmon[39718]: mgr.24542) 1564 : cluster [DBG] pgmap v1964: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2021-07-31T02:23:40.916 INFO:tasks.workunit.client.0.smithi045.stdout: 1092: throughput=5MB/sec pending data=0 2021-07-31T02:23:40.916 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-JR-ZOKAqIvUDFZr off=909122 len=458175 2021-07-31T02:23:40.920 INFO:tasks.workunit.client.0.smithi045.stdout:op 5566 completed, throughput=5MB/sec 2021-07-31T02:23:40.920 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-5849yibTVgytwSx off=14275210 len=426885 2021-07-31T02:23:40.923 INFO:tasks.workunit.client.0.smithi045.stdout:op 5567 completed, throughput=5MB/sec 2021-07-31T02:23:40.924 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-qzQP_uLjP0WE11e off=8148131 len=2092259 2021-07-31T02:23:40.935 INFO:tasks.workunit.client.0.smithi045.stdout:op 5568 completed, throughput=5MB/sec 2021-07-31T02:23:40.936 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-L39eW2PpLJARbHx off=3161164 len=401471 2021-07-31T02:23:40.936 INFO:tasks.workunit.client.0.smithi045.stdout:op 5569 completed, throughput=5MB/sec 2021-07-31T02:23:40.936 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Xz3ByaET0A-yXoY off=8276960 len=1465590 2021-07-31T02:23:40.943 INFO:tasks.workunit.client.0.smithi045.stdout:op 5570 completed, throughput=5MB/sec 2021-07-31T02:23:41.943 INFO:tasks.workunit.client.0.smithi045.stdout: 1093: throughput=5MB/sec pending data=0 2021-07-31T02:23:41.944 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-_I15xJHuKZSdjsI off=3729480 len=1766432 2021-07-31T02:23:41.951 INFO:tasks.workunit.client.0.smithi045.stdout:op 5571 completed, throughput=5MB/sec 2021-07-31T02:23:41.952 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-tZydApzbUxPxruD off=6497961 len=1687738 2021-07-31T02:23:41.959 INFO:tasks.workunit.client.0.smithi045.stdout:op 5572 completed, throughput=5MB/sec 2021-07-31T02:23:41.959 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-kyvrWB08w5BJDTI off=3644822 len=1075649 2021-07-31T02:23:41.961 INFO:tasks.workunit.client.0.smithi045.stdout:op 5573 completed, throughput=5MB/sec 2021-07-31T02:23:41.961 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-9PwtoQxgJq3bmG8 off=4163215 len=331625 2021-07-31T02:23:41.962 INFO:tasks.workunit.client.0.smithi045.stdout:op 5574 completed, throughput=5MB/sec 2021-07-31T02:23:41.962 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-6QiO0poWodldu0q off=9981035 len=745560 2021-07-31T02:23:41.963 INFO:tasks.workunit.client.0.smithi045.stdout:op 5575 completed, throughput=5MB/sec 2021-07-31T02:23:42.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:42 smithi154 conmon[34553]: cluster 2021-07-31T02:23:40.314909+0000 mgr.x (mgr.24542) 1565 : cluster [DBG] pgmap v1965: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 3.4 MiB/s rd, 2.7 MiB/s wr, 6 op/s 2021-07-31T02:23:42.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:42 smithi045 conmon[33655]: cluster 2021-07-31T02:23:40.314909+0000 mgr.x (mgr.24542) 2021-07-31T02:23:42.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:42 smithi045 conmon[33655]: 1565 : cluster [DBG] pgmap v1965: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 3.4 MiB/s rd, 2.7 MiB/s wr, 6 op/s 2021-07-31T02:23:42.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:42 smithi045 conmon[39718]: cluster 2021-07-31T02:23:40.314909+0000 mgr.x (mgr.24542) 1565 : cluster 2021-07-31T02:23:42.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:42 smithi045 conmon[39718]: [DBG] pgmap v1965: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 3.4 MiB/s rd, 2.7 MiB/s wr, 6 op/s 2021-07-31T02:23:42.965 INFO:tasks.workunit.client.0.smithi045.stdout: 1094: throughput=5MB/sec pending data=0 2021-07-31T02:23:42.965 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-TvZ5CK_ZNsptNDY off=3131846 len=1599203 2021-07-31T02:23:42.972 INFO:tasks.workunit.client.0.smithi045.stdout:op 5576 completed, throughput=5MB/sec 2021-07-31T02:23:42.972 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-dhUH2b1y7bJ7kkR off=9523214 len=1461132 2021-07-31T02:23:42.974 INFO:tasks.workunit.client.0.smithi045.stdout:op 5577 completed, throughput=5MB/sec 2021-07-31T02:23:42.975 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-7HNItv6lYtRTEEi off=9606947 len=419751 2021-07-31T02:23:42.975 INFO:tasks.workunit.client.0.smithi045.stdout:op 5578 completed, throughput=5MB/sec 2021-07-31T02:23:42.976 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-WbGWw1wofCVspss off=19768161 len=1751789 2021-07-31T02:23:42.979 INFO:tasks.workunit.client.0.smithi045.stdout:op 5579 completed, throughput=5MB/sec 2021-07-31T02:23:43.979 INFO:tasks.workunit.client.0.smithi045.stdout: 1095: throughput=5MB/sec pending data=0 2021-07-31T02:23:43.980 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ltEZ2iWaDHLPIVP off=9612055 len=687774 2021-07-31T02:23:43.980 INFO:tasks.workunit.client.0.smithi045.stdout:op 5580 completed, throughput=5MB/sec 2021-07-31T02:23:43.981 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-W6ou6xzEbaFsKtA off=34907 len=69902 2021-07-31T02:23:43.981 INFO:tasks.workunit.client.0.smithi045.stdout:op 5581 completed, throughput=5MB/sec 2021-07-31T02:23:43.982 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-T-fHvKuq2LcitrF off=7324265 len=249805 2021-07-31T02:23:43.982 INFO:tasks.workunit.client.0.smithi045.stdout:op 5582 completed, throughput=5MB/sec 2021-07-31T02:23:43.983 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-rtsRnavVXgffbIz off=82345 len=1948524 2021-07-31T02:23:43.986 INFO:tasks.workunit.client.0.smithi045.stdout:op 5583 completed, throughput=5MB/sec 2021-07-31T02:23:43.986 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-McrZlyJ-EkgYnVY off=834340 len=503397 2021-07-31T02:23:43.996 INFO:tasks.workunit.client.0.smithi045.stdout:op 5584 completed, throughput=5MB/sec 2021-07-31T02:23:43.996 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-FXxminL2vPN4VyH off=17729223 len=234956 2021-07-31T02:23:43.997 INFO:tasks.workunit.client.0.smithi045.stdout:op 5585 completed, throughput=5MB/sec 2021-07-31T02:23:43.997 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-FCbiwZSAyXSl6AP off=1799836 len=493925 2021-07-31T02:23:43.998 INFO:tasks.workunit.client.0.smithi045.stdout:op 5586 completed, throughput=5MB/sec 2021-07-31T02:23:43.999 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj--bZe8Wk8NIQEq31 off=1266495 len=552434 2021-07-31T02:23:44.001 INFO:tasks.workunit.client.0.smithi045.stdout:op 5587 completed, throughput=5MB/sec 2021-07-31T02:23:44.002 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj--2Uid3rN8YSuOOJ off=8899732 len=1517908 2021-07-31T02:23:44.009 INFO:tasks.workunit.client.0.smithi045.stdout:op 5588 completed, throughput=5MB/sec 2021-07-31T02:23:44.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:23:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:23:43.572Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:23:44.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:23:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:23:43.572Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:23:44.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:23:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:23:43.572Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:23:44.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:44 smithi154 conmon[34553]: cluster 2021-07-31T02:23:42.315299+0000 mgr.x (mgr.24542) 1566 : cluster [DBG] pgmap v1966: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 2.3 MiB/s rd, 1.6 MiB/s wr, 4 op/s 2021-07-31T02:23:44.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:44 smithi045 conmon[39718]: cluster 2021-07-31T02:23:42.315299+0000 mgr.x ( 2021-07-31T02:23:44.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:44 smithi045 conmon[39718]: mgr.24542) 1566 : cluster [DBG] pgmap v1966: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 2.3 MiB/s rd, 1.6 MiB/s wr, 4 op/s 2021-07-31T02:23:44.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:44 smithi045 conmon[33655]: cluster 2021-07-31T02:23:42.315299+0000 mgr.x 2021-07-31T02:23:44.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:44 smithi045 conmon[33655]: (mgr.24542) 1566 : cluster [DBG] pgmap v1966: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 2.3 MiB/s rd, 1.6 MiB/s wr, 4 op/s 2021-07-31T02:23:45.009 INFO:tasks.workunit.client.0.smithi045.stdout: 1096: throughput=5MB/sec pending data=0 2021-07-31T02:23:45.009 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-v-w9R5DM5RRrd4L off=0 len=1173269 2021-07-31T02:23:45.012 INFO:tasks.workunit.client.0.smithi045.stdout:op 5589 completed, throughput=5MB/sec 2021-07-31T02:23:45.012 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-XskjTYC7zvAmcC1 off=2686670 len=477479 2021-07-31T02:23:45.016 INFO:tasks.workunit.client.0.smithi045.stdout:op 5590 completed, throughput=5MB/sec 2021-07-31T02:23:45.016 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-s6MYXHikjwUfhnN off=16085311 len=1866957 2021-07-31T02:23:45.027 INFO:tasks.workunit.client.0.smithi045.stdout:op 5591 completed, throughput=5MB/sec 2021-07-31T02:23:45.027 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-4XBlK-lLjo52Lva off=10411623 len=682992 2021-07-31T02:23:45.028 INFO:tasks.workunit.client.0.smithi045.stdout:op 5592 completed, throughput=5MB/sec 2021-07-31T02:23:45.028 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-h4uXx4-HzSwnU4f off=13648262 len=1386146 2021-07-31T02:23:45.030 INFO:tasks.workunit.client.0.smithi045.stdout:op 5593 completed, throughput=5MB/sec 2021-07-31T02:23:46.031 INFO:tasks.workunit.client.0.smithi045.stdout: 1097: throughput=5MB/sec pending data=0 2021-07-31T02:23:46.031 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Tf2KHTjPraZnm1T off=13290848 len=348020 2021-07-31T02:23:46.036 INFO:tasks.workunit.client.0.smithi045.stdout:op 5594 completed, throughput=5MB/sec 2021-07-31T02:23:46.036 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-1QrWZd9IaDwjdMk off=2777576 len=872762 2021-07-31T02:23:46.037 INFO:tasks.workunit.client.0.smithi045.stdout:op 5595 completed, throughput=5MB/sec 2021-07-31T02:23:46.038 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-IxHzq-aqiPjfI08 off=1390023 len=1872748 2021-07-31T02:23:46.047 INFO:tasks.workunit.client.0.smithi045.stdout:op 5596 completed, throughput=5MB/sec 2021-07-31T02:23:46.048 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-g0KcH_bjZkvz7Qk off=3220399 len=893331 2021-07-31T02:23:46.049 INFO:tasks.workunit.client.0.smithi045.stdout:op 5597 completed, throughput=5MB/sec 2021-07-31T02:23:46.050 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-6egBtQB_4Nue0md off=9963273 len=1457365 2021-07-31T02:23:46.055 INFO:tasks.workunit.client.0.smithi045.stdout:op 5598 completed, throughput=5MB/sec 2021-07-31T02:23:46.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:46 smithi154 conmon[34553]: cluster 2021-07-31T02:23:44.315939+0000 mgr.x (mgr.24542) 1567 : cluster [DBG] pgmap v1967: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2021-07-31T02:23:46.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:46 smithi045 conmon[33655]: cluster 2021-07-31T02:23:44.315939+0000 mgr.x (mgr 2021-07-31T02:23:46.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:46 smithi045 conmon[33655]: .24542) 1567 : cluster [DBG] pgmap v1967: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2021-07-31T02:23:46.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:46 smithi045 conmon[39718]: cluster 2021-07-31T02:23:44.315939+0000 mgr.x (mgr 2021-07-31T02:23:46.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:46 smithi045 conmon[39718]: .24542) 1567 : cluster [DBG] pgmap v1967: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2021-07-31T02:23:47.056 INFO:tasks.workunit.client.0.smithi045.stdout: 1098: throughput=5MB/sec pending data=0 2021-07-31T02:23:47.056 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-7n8d4xPIkC3b7br off=6659745 len=316374 2021-07-31T02:23:47.057 INFO:tasks.workunit.client.0.smithi045.stdout:op 5599 completed, throughput=5MB/sec 2021-07-31T02:23:47.057 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Pxn2NS5Fae3kPbt off=14191023 len=1898882 2021-07-31T02:23:47.066 INFO:tasks.workunit.client.0.smithi045.stdout:op 5600 completed, throughput=5MB/sec 2021-07-31T02:23:47.066 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-BITVg8S21xIHV-B off=1796568 len=1573639 2021-07-31T02:23:47.074 INFO:tasks.workunit.client.0.smithi045.stdout:op 5601 completed, throughput=5MB/sec 2021-07-31T02:23:47.075 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-EHdm5huup9oRfZk off=22455842 len=1376450 2021-07-31T02:23:47.076 INFO:tasks.workunit.client.0.smithi045.stdout:op 5602 completed, throughput=5MB/sec 2021-07-31T02:23:48.076 INFO:tasks.workunit.client.0.smithi045.stdout: 1099: throughput=5MB/sec pending data=0 2021-07-31T02:23:48.077 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-WtLIWSt1rJbKN5x off=6023163 len=954744 2021-07-31T02:23:48.079 INFO:tasks.workunit.client.0.smithi045.stdout:op 5603 completed, throughput=5MB/sec 2021-07-31T02:23:48.079 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-QTB--hGjEDIRWVK off=2069928 len=1829686 2021-07-31T02:23:48.088 INFO:tasks.workunit.client.0.smithi045.stdout:op 5604 completed, throughput=5MB/sec 2021-07-31T02:23:48.088 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-eWwxOFoBa0gJuYl off=1495461 len=961851 2021-07-31T02:23:48.089 INFO:tasks.workunit.client.0.smithi045.stdout:op 5605 completed, throughput=5MB/sec 2021-07-31T02:23:48.089 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Y5tW-WvQ1Ij0hHE off=6490685 len=467122 2021-07-31T02:23:48.092 INFO:tasks.workunit.client.0.smithi045.stdout:op 5606 completed, throughput=5MB/sec 2021-07-31T02:23:48.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:48 smithi045 conmon[33655]: cluster 2021-07-31T02:23:46.316362+0000 mgr.x (mgr.24542) 1568 : cluster [DBG] pgmap v1968: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.9 MiB/s rd, 1.9 MiB/s wr, 3 op/s 2021-07-31T02:23:48.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:48 smithi045 conmon[39718]: cluster 2021-07-31T02:23:46 2021-07-31T02:23:48.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:48 smithi045 conmon[39718]: .316362+0000 mgr.x (mgr.24542) 1568 : cluster [DBG] pgmap v1968: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.9 MiB/s rd, 1.9 MiB/s wr, 3 op/s 2021-07-31T02:23:48.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:48 smithi154 conmon[34553]: cluster 2021-07-31T02:23:46.316362+0000 mgr.x (mgr.24542) 1568 : cluster [DBG] pgmap v1968: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.9 MiB/s rd, 1.9 MiB/s wr, 3 op/s 2021-07-31T02:23:49.093 INFO:tasks.workunit.client.0.smithi045.stdout: 1100: throughput=5MB/sec pending data=0 2021-07-31T02:23:49.093 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-sEUgXEAtgogu_Qt off=8234527 len=44380 2021-07-31T02:23:49.094 INFO:tasks.workunit.client.0.smithi045.stdout:op 5607 completed, throughput=5MB/sec 2021-07-31T02:23:49.094 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-HZnR6CkMR3RcKY8 off=3249342 len=1682863 2021-07-31T02:23:49.102 INFO:tasks.workunit.client.0.smithi045.stdout:op 5608 completed, throughput=5MB/sec 2021-07-31T02:23:49.102 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-uNZs9MWQKUUwv2Y off=1302093 len=1266193 2021-07-31T02:23:49.104 INFO:tasks.workunit.client.0.smithi045.stdout:op 5609 completed, throughput=5MB/sec 2021-07-31T02:23:49.105 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-KV86JuE4I_AB6Dw off=618318 len=827160 2021-07-31T02:23:49.107 INFO:tasks.workunit.client.0.smithi045.stdout:op 5610 completed, throughput=5MB/sec 2021-07-31T02:23:49.107 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-8HRAFuEiOuabR8u off=2255267 len=1793278 2021-07-31T02:23:49.115 INFO:tasks.workunit.client.0.smithi045.stdout:op 5611 completed, throughput=5MB/sec 2021-07-31T02:23:50.115 INFO:tasks.workunit.client.0.smithi045.stdout: 1101: throughput=5MB/sec pending data=0 2021-07-31T02:23:50.116 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-PkI918PhCOjvpxB off=2120778 len=64300 2021-07-31T02:23:50.116 INFO:tasks.workunit.client.0.smithi045.stdout:op 5612 completed, throughput=5MB/sec 2021-07-31T02:23:50.117 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-taNJpiSfEkvanBS off=15266015 len=1414358 2021-07-31T02:23:50.118 INFO:tasks.workunit.client.0.smithi045.stdout:op 5613 completed, throughput=5MB/sec 2021-07-31T02:23:50.119 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-PlDjQI358_1ld1x off=9003910 len=768742 2021-07-31T02:23:50.123 INFO:tasks.workunit.client.0.smithi045.stdout:op 5614 completed, throughput=5MB/sec 2021-07-31T02:23:50.124 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-CFI3l7JXS-ytQc3 off=1927556 len=811347 2021-07-31T02:23:50.125 INFO:tasks.workunit.client.0.smithi045.stdout:op 5615 completed, throughput=5MB/sec 2021-07-31T02:23:50.125 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Did7CPSL8spKtaZ off=267415 len=1293399 2021-07-31T02:23:50.132 INFO:tasks.workunit.client.0.smithi045.stdout:op 5616 completed, throughput=5MB/sec 2021-07-31T02:23:50.132 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-JYqQXEfsgobBIMI off=17533934 len=2057369 2021-07-31T02:23:50.141 INFO:tasks.workunit.client.0.smithi045.stdout:op 5617 completed, throughput=5MB/sec 2021-07-31T02:23:50.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:50 smithi045 conmon[39718]: cluster 2021-07-31T02:23:48 2021-07-31T02:23:50.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:50 smithi045 conmon[39718]: .316741+0000 mgr.x (mgr.24542) 1569 : cluster [DBG] pgmap v1969: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.9 MiB/s rd, 1.9 MiB/s wr, 3 op/s 2021-07-31T02:23:50.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:50 smithi045 conmon[33655]: cluster 2021-07-31T02:23:48.316741+0000 mgr.x ( 2021-07-31T02:23:50.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:50 smithi045 conmon[33655]: mgr.24542) 1569 : cluster [DBG] pgmap v1969: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.9 MiB/s rd, 1.9 MiB/s wr, 3 op/s 2021-07-31T02:23:50.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:50 smithi154 conmon[34553]: cluster 2021-07-31T02:23:48.316741+0000 mgr.x (mgr.24542) 1569 : cluster [DBG] pgmap v1969: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.9 MiB/s rd, 1.9 MiB/s wr, 3 op/s 2021-07-31T02:23:51.142 INFO:tasks.workunit.client.0.smithi045.stdout: 1102: throughput=5MB/sec pending data=0 2021-07-31T02:23:51.142 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-LlNUu5OVB6rKz0s off=3074191 len=980810 2021-07-31T02:23:51.148 INFO:tasks.workunit.client.0.smithi045.stdout:op 5618 completed, throughput=5MB/sec 2021-07-31T02:23:51.149 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-i4V29EOudGCX2qx off=872274 len=260884 2021-07-31T02:23:51.152 INFO:tasks.workunit.client.0.smithi045.stdout:op 5619 completed, throughput=5MB/sec 2021-07-31T02:23:51.153 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-TIuG8fccIuf6-qk off=2238582 len=563090 2021-07-31T02:23:51.154 INFO:tasks.workunit.client.0.smithi045.stdout:op 5620 completed, throughput=5MB/sec 2021-07-31T02:23:51.154 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-cq073oVzV4lK10h off=17493075 len=1399404 2021-07-31T02:23:51.161 INFO:tasks.workunit.client.0.smithi045.stdout:op 5621 completed, throughput=5MB/sec 2021-07-31T02:23:51.161 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-RCWW-CBfVzPpVGt off=3750739 len=558201 2021-07-31T02:23:51.165 INFO:tasks.workunit.client.0.smithi045.stdout:op 5622 completed, throughput=5MB/sec 2021-07-31T02:23:51.165 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-xmsHePgQJDgYHYs off=77637 len=1233836 2021-07-31T02:23:51.166 INFO:tasks.workunit.client.0.smithi045.stdout:op 5623 completed, throughput=5MB/sec 2021-07-31T02:23:52.167 INFO:tasks.workunit.client.0.smithi045.stdout: 1103: throughput=5MB/sec pending data=0 2021-07-31T02:23:52.167 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-eJD5LTJDweURF46 off=1984954 len=106226 2021-07-31T02:23:52.168 INFO:tasks.workunit.client.0.smithi045.stdout:op 5624 completed, throughput=5MB/sec 2021-07-31T02:23:52.168 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-wrYYROSnxXHHs8X off=2634685 len=308620 2021-07-31T02:23:52.171 INFO:tasks.workunit.client.0.smithi045.stdout:op 5625 completed, throughput=5MB/sec 2021-07-31T02:23:52.171 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-esSoEkbiF-5z_Vm off=3405974 len=1155386 2021-07-31T02:23:52.177 INFO:tasks.workunit.client.0.smithi045.stdout:op 5626 completed, throughput=5MB/sec 2021-07-31T02:23:52.177 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-9rkQXmKYGR48AVC off=1015574 len=1647579 2021-07-31T02:23:52.180 INFO:tasks.workunit.client.0.smithi045.stdout:op 5627 completed, throughput=5MB/sec 2021-07-31T02:23:52.180 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-OeA1PPb00Fa_-_h off=7097511 len=797311 2021-07-31T02:23:52.186 INFO:tasks.workunit.client.0.smithi045.stdout:op 5628 completed, throughput=5MB/sec 2021-07-31T02:23:52.186 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-duGJgktSruP4pL3 off=8340215 len=411971 2021-07-31T02:23:52.189 INFO:tasks.workunit.client.0.smithi045.stdout:op 5629 completed, throughput=5MB/sec 2021-07-31T02:23:52.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:52 smithi045 conmon[33655]: cluster 2021-07-31T02:23:50.317330+0000 mgr.x (mgr.24542) 1570 : cluster 2021-07-31T02:23:52.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:52 smithi045 conmon[33655]: [DBG] pgmap v1970: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 2.9 MiB/s rd, 3.1 MiB/s wr, 5 op/s 2021-07-31T02:23:52.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:52 smithi045 conmon[39718]: cluster 2021-07-31T02:23:50.317330+0000 2021-07-31T02:23:52.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:52 smithi045 conmon[39718]: mgr.x (mgr.24542) 1570 : cluster [DBG] pgmap v1970: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 2.9 MiB/s rd, 3.1 MiB/s wr, 5 op/s 2021-07-31T02:23:52.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:52 smithi154 conmon[34553]: cluster 2021-07-31T02:23:50.317330+0000 mgr.x (mgr.24542) 1570 : cluster [DBG] pgmap v1970: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 2.9 MiB/s rd, 3.1 MiB/s wr, 5 op/s 2021-07-31T02:23:53.189 INFO:tasks.workunit.client.0.smithi045.stdout: 1104: throughput=5MB/sec pending data=0 2021-07-31T02:23:53.190 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-91cFKtNZpqeD_RK off=1848489 len=1329550 2021-07-31T02:23:53.197 INFO:tasks.workunit.client.0.smithi045.stdout:op 5630 completed, throughput=5MB/sec 2021-07-31T02:23:53.197 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-SrBumC4zogdfjgg off=16569411 len=1303453 2021-07-31T02:23:53.203 INFO:tasks.workunit.client.0.smithi045.stdout:op 5631 completed, throughput=5MB/sec 2021-07-31T02:23:53.204 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-0Qm2GwD8Jd4718R off=5409694 len=1925289 2021-07-31T02:23:53.211 INFO:tasks.workunit.client.0.smithi045.stdout:op 5632 completed, throughput=5MB/sec 2021-07-31T02:23:53.212 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-S-C5AGbn2VoT6b6 off=6556957 len=642947 2021-07-31T02:23:53.216 INFO:tasks.workunit.client.0.smithi045.stdout:op 5633 completed, throughput=5MB/sec 2021-07-31T02:23:53.216 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-O0xmyhVZPWBzL4C off=6457927 len=1291890 2021-07-31T02:23:53.218 INFO:tasks.workunit.client.0.smithi045.stdout:op 5634 completed, throughput=5MB/sec 2021-07-31T02:23:54.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:23:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:23:53.572Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:23:54.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:23:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:23:53.572Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:23:54.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:23:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:23:53.572Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:23:54.219 INFO:tasks.workunit.client.0.smithi045.stdout: 1105: throughput=5MB/sec pending data=0 2021-07-31T02:23:54.219 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-_wBtAf_D6m0xhuC off=638924 len=2024562 2021-07-31T02:23:54.223 INFO:tasks.workunit.client.0.smithi045.stdout:op 5635 completed, throughput=5MB/sec 2021-07-31T02:23:54.223 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-GGQFbDuGX-qDFeS off=4440907 len=1522044 2021-07-31T02:23:54.225 INFO:tasks.workunit.client.0.smithi045.stdout:op 5636 completed, throughput=5MB/sec 2021-07-31T02:23:54.225 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Wbt-dyLNkPaUGDO off=16545647 len=761013 2021-07-31T02:23:54.227 INFO:tasks.workunit.client.0.smithi045.stdout:op 5637 completed, throughput=5MB/sec 2021-07-31T02:23:54.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:54 smithi045 conmon[39718]: cluster 2021-07-31T02:23:52.317879+0000 mgr.x ( 2021-07-31T02:23:54.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:54 smithi045 conmon[39718]: mgr.24542) 1571 : cluster [DBG] pgmap v1971: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:23:54.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:54 smithi045 conmon[33655]: cluster 2021-07-31T02:23:52.317879+0000 mgr.x (mgr.24542) 1571 : cluster [DBG] 2021-07-31T02:23:54.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:54 smithi045 conmon[33655]: pgmap v1971: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:23:54.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:54 smithi154 conmon[34553]: cluster 2021-07-31T02:23:52.317879+0000 mgr.x (mgr.24542) 1571 : cluster [DBG] pgmap v1971: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:23:55.227 INFO:tasks.workunit.client.0.smithi045.stdout: 1106: throughput=5MB/sec pending data=0 2021-07-31T02:23:55.228 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-5zY_Wkr2aoMDNC5 off=4296837 len=234281 2021-07-31T02:23:55.228 INFO:tasks.workunit.client.0.smithi045.stdout:op 5638 completed, throughput=5MB/sec 2021-07-31T02:23:55.228 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-MjUU8wXuxqhih3p off=1363553 len=240356 2021-07-31T02:23:55.231 INFO:tasks.workunit.client.0.smithi045.stdout:op 5639 completed, throughput=5MB/sec 2021-07-31T02:23:55.232 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-EmcjEN9HKZ4vCaq off=1287247 len=1176700 2021-07-31T02:23:55.238 INFO:tasks.workunit.client.0.smithi045.stdout:op 5640 completed, throughput=5MB/sec 2021-07-31T02:23:55.238 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Og_5dozBBcmYcz7 off=1242588 len=460221 2021-07-31T02:23:55.239 INFO:tasks.workunit.client.0.smithi045.stdout:op 5641 completed, throughput=5MB/sec 2021-07-31T02:23:55.239 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-xlDbxfOin4GSwwO off=1724079 len=1730096 2021-07-31T02:23:55.247 INFO:tasks.workunit.client.0.smithi045.stdout:op 5642 completed, throughput=5MB/sec 2021-07-31T02:23:55.247 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-K7JAHGzmbpiTlVl off=1935930 len=501054 2021-07-31T02:23:55.250 INFO:tasks.workunit.client.0.smithi045.stdout:op 5643 completed, throughput=5MB/sec 2021-07-31T02:23:55.250 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-lHe-PQVmQWCW85l off=20708575 len=662344 2021-07-31T02:23:55.251 INFO:tasks.workunit.client.0.smithi045.stdout:op 5644 completed, throughput=5MB/sec 2021-07-31T02:23:55.252 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-YjahfzkMWgwGs6b off=25657 len=1077816 2021-07-31T02:23:55.256 INFO:tasks.workunit.client.0.smithi045.stdout:op 5645 completed, throughput=5MB/sec 2021-07-31T02:23:56.257 INFO:tasks.workunit.client.0.smithi045.stdout: 1107: throughput=5MB/sec pending data=0 2021-07-31T02:23:56.257 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-6rMnxRevJOQ3jHp off=12108972 len=802313 2021-07-31T02:23:56.258 INFO:tasks.workunit.client.0.smithi045.stdout:op 5646 completed, throughput=5MB/sec 2021-07-31T02:23:56.259 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-sk8iMbBr0T2OBsc off=3018385 len=822460 2021-07-31T02:23:56.264 INFO:tasks.workunit.client.0.smithi045.stdout:op 5647 completed, throughput=5MB/sec 2021-07-31T02:23:56.264 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-TFjXlRF0G18ZqzY off=4418042 len=1918085 2021-07-31T02:23:56.267 INFO:tasks.workunit.client.0.smithi045.stdout:op 5648 completed, throughput=5MB/sec 2021-07-31T02:23:56.267 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-a67wOEany1r42a9 off=2051530 len=1490413 2021-07-31T02:23:56.277 INFO:tasks.workunit.client.0.smithi045.stdout:op 5649 completed, throughput=5MB/sec 2021-07-31T02:23:56.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:56 smithi045 conmon[33655]: cluster 2021-07-31T02:23:54.318375+0000 mgr.x (mgr.24542) 1572 : 2021-07-31T02:23:56.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:56 smithi045 conmon[33655]: cluster [DBG] pgmap v1972: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 2.5 MiB/s rd, 3.9 MiB/s wr, 6 op/s 2021-07-31T02:23:56.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:56 smithi045 conmon[39718]: cluster 2021-07-31T02:23:54.318375+0000 mgr.x (mgr 2021-07-31T02:23:56.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:56 smithi045 conmon[39718]: .24542) 1572 : cluster [DBG] pgmap v1972: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 2.5 MiB/s rd, 3.9 MiB/s wr, 6 op/s 2021-07-31T02:23:56.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:56 smithi154 conmon[34553]: cluster 2021-07-31T02:23:54.318375+0000 mgr.x (mgr.24542) 1572 : cluster [DBG] pgmap v1972: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 2.5 MiB/s rd, 3.9 MiB/s wr, 6 op/s 2021-07-31T02:23:57.277 INFO:tasks.workunit.client.0.smithi045.stdout: 1108: throughput=5MB/sec pending data=0 2021-07-31T02:23:57.277 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-YIovmR1CC7rxwPq off=1200963 len=1110486 2021-07-31T02:23:57.279 INFO:tasks.workunit.client.0.smithi045.stdout:op 5650 completed, throughput=5MB/sec 2021-07-31T02:23:57.279 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-fcFFDWEWVunm8uL off=13307449 len=1844091 2021-07-31T02:23:57.282 INFO:tasks.workunit.client.0.smithi045.stdout:op 5651 completed, throughput=5MB/sec 2021-07-31T02:23:57.282 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Ji7lyFomuV3n8hG off=11407859 len=372798 2021-07-31T02:23:57.283 INFO:tasks.workunit.client.0.smithi045.stdout:op 5652 completed, throughput=5MB/sec 2021-07-31T02:23:57.284 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-K1xiZkKeXCXfwre off=15546124 len=97847 2021-07-31T02:23:57.285 INFO:tasks.workunit.client.0.smithi045.stdout:op 5653 completed, throughput=5MB/sec 2021-07-31T02:23:57.285 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-YA7a6PHQhtb7nUU off=9152556 len=424741 2021-07-31T02:23:57.285 INFO:tasks.workunit.client.0.smithi045.stdout:op 5654 completed, throughput=5MB/sec 2021-07-31T02:23:57.285 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-8RuWlD9E1ttv-Es off=14525836 len=1814009 2021-07-31T02:23:57.296 INFO:tasks.workunit.client.0.smithi045.stdout:op 5655 completed, throughput=5MB/sec 2021-07-31T02:23:58.296 INFO:tasks.workunit.client.0.smithi045.stdout: 1109: throughput=5MB/sec pending data=0 2021-07-31T02:23:58.297 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-N5v8CcDD5c2HwFW off=10842243 len=983234 2021-07-31T02:23:58.304 INFO:tasks.workunit.client.0.smithi045.stdout:op 5656 completed, throughput=5MB/sec 2021-07-31T02:23:58.304 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-7v3KyJF-23liV9C off=81571 len=1763146 2021-07-31T02:23:58.311 INFO:tasks.workunit.client.0.smithi045.stdout:op 5657 completed, throughput=5MB/sec 2021-07-31T02:23:58.311 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-5QBouV7PTQKyyNV off=713633 len=1182313 2021-07-31T02:23:58.319 INFO:tasks.workunit.client.0.smithi045.stdout:op 5658 completed, throughput=5MB/sec 2021-07-31T02:23:58.319 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Y6WFsS5SeWgQ0MQ off=10309661 len=593050 2021-07-31T02:23:58.322 INFO:tasks.workunit.client.0.smithi045.stdout:op 5659 completed, throughput=5MB/sec 2021-07-31T02:23:58.322 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-FUofMmKr3_whjM5 off=1167624 len=941739 2021-07-31T02:23:58.327 INFO:tasks.workunit.client.0.smithi045.stdout:op 5660 completed, throughput=5MB/sec 2021-07-31T02:23:58.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:58 smithi045 conmon[33655]: cluster 2021-07-31T02:23:56.318815+0000 mgr.x (mgr.24542) 1573 2021-07-31T02:23:58.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:23:58 smithi045 conmon[33655]: : cluster [DBG] pgmap v1973: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2021-07-31T02:23:58.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:58 smithi045 conmon[39718]: cluster 2021-07-31T02:23:56.318815 2021-07-31T02:23:58.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:23:58 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 1573 : cluster [DBG] pgmap v1973: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2021-07-31T02:23:58.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:23:58 smithi154 conmon[34553]: cluster 2021-07-31T02:23:56.318815+0000 mgr.x (mgr.24542) 1573 : cluster [DBG] pgmap v1973: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2021-07-31T02:23:59.327 INFO:tasks.workunit.client.0.smithi045.stdout: 1110: throughput=5MB/sec pending data=0 2021-07-31T02:23:59.327 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ZjHbj9ffD2KVl2I off=7655579 len=1724564 2021-07-31T02:23:59.338 INFO:tasks.workunit.client.0.smithi045.stdout:op 5661 completed, throughput=5MB/sec 2021-07-31T02:23:59.339 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ZT9lGVYgM-TJOx9 off=4628800 len=1375749 2021-07-31T02:23:59.345 INFO:tasks.workunit.client.0.smithi045.stdout:op 5662 completed, throughput=5MB/sec 2021-07-31T02:23:59.346 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-EdH1WfokDhBYxSl off=4054100 len=1015312 2021-07-31T02:23:59.348 INFO:tasks.workunit.client.0.smithi045.stdout:op 5663 completed, throughput=5MB/sec 2021-07-31T02:23:59.348 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-bOnEfcwVujcQS5S off=2259290 len=1545117 2021-07-31T02:23:59.358 INFO:tasks.workunit.client.0.smithi045.stdout:op 5664 completed, throughput=5MB/sec 2021-07-31T02:24:00.359 INFO:tasks.workunit.client.0.smithi045.stdout: 1111: throughput=5MB/sec pending data=0 2021-07-31T02:24:00.359 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-KfrTSdgW7wpV-Ah off=6382852 len=819103 2021-07-31T02:24:00.360 INFO:tasks.workunit.client.0.smithi045.stdout:op 5665 completed, throughput=5MB/sec 2021-07-31T02:24:00.361 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-2zSrHpe8qyRGsIr off=15796688 len=478674 2021-07-31T02:24:00.362 INFO:tasks.workunit.client.0.smithi045.stdout:op 5666 completed, throughput=5MB/sec 2021-07-31T02:24:00.362 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ZUyl5B0lHAig05H off=15452895 len=762784 2021-07-31T02:24:00.366 INFO:tasks.workunit.client.0.smithi045.stdout:op 5667 completed, throughput=5MB/sec 2021-07-31T02:24:00.366 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-sglPYHGFNbINZha off=1288375 len=403734 2021-07-31T02:24:00.367 INFO:tasks.workunit.client.0.smithi045.stdout:op 5668 completed, throughput=5MB/sec 2021-07-31T02:24:00.368 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-rPgBkglfnGtyQui off=8333791 len=1588385 2021-07-31T02:24:00.369 INFO:tasks.workunit.client.0.smithi045.stdout:op 5669 completed, throughput=5MB/sec 2021-07-31T02:24:00.369 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-SFioxAi1idUHAN7 off=722553 len=34112 2021-07-31T02:24:00.370 INFO:tasks.workunit.client.0.smithi045.stdout:op 5670 completed, throughput=5MB/sec 2021-07-31T02:24:00.370 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-MjUU8wXuxqhih3p off=9670775 len=282702 2021-07-31T02:24:00.371 INFO:tasks.workunit.client.0.smithi045.stdout:op 5671 completed, throughput=5MB/sec 2021-07-31T02:24:00.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:00 smithi045 conmon[39718]: cluster 2021-07-31T02:23:58.319280+0000 mgr.x (mgr.24542) 1574 : cluster [DBG] pgmap v1974: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2021-07-31T02:24:00.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:00 smithi045 conmon[33655]: cluster 2021-07-31T02:23:58.319280+0000 mgr.x (mgr.24542) 1574 : cluster 2021-07-31T02:24:00.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:00 smithi045 conmon[33655]: [DBG] pgmap v1974: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2021-07-31T02:24:00.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:00 smithi154 conmon[34553]: cluster 2021-07-31T02:23:58.319280+0000 mgr.x (mgr.24542) 1574 : cluster [DBG] pgmap v1974: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2021-07-31T02:24:01.371 INFO:tasks.workunit.client.0.smithi045.stdout: 1112: throughput=5MB/sec pending data=0 2021-07-31T02:24:01.372 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-v2xotfB_fPEYfFZ off=247576 len=210904 2021-07-31T02:24:01.373 INFO:tasks.workunit.client.0.smithi045.stdout:op 5672 completed, throughput=5MB/sec 2021-07-31T02:24:01.373 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-D3LyBxver2GX3JF off=4948668 len=1686483 2021-07-31T02:24:01.375 INFO:tasks.workunit.client.0.smithi045.stdout:op 5673 completed, throughput=5MB/sec 2021-07-31T02:24:01.375 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-lIWxsYX2LVXl9fx off=4104007 len=1500653 2021-07-31T02:24:01.383 INFO:tasks.workunit.client.0.smithi045.stdout:op 5674 completed, throughput=5MB/sec 2021-07-31T02:24:01.383 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-LnlC9PDpo7onf54 off=4034656 len=1686806 2021-07-31T02:24:01.391 INFO:tasks.workunit.client.0.smithi045.stdout:op 5675 completed, throughput=5MB/sec 2021-07-31T02:24:01.391 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-va5bBaHtK2oH3QM off=1561928 len=901029 2021-07-31T02:24:01.397 INFO:tasks.workunit.client.0.smithi045.stdout:op 5676 completed, throughput=5MB/sec 2021-07-31T02:24:02.397 INFO:tasks.workunit.client.0.smithi045.stdout: 1113: throughput=5MB/sec pending data=0 2021-07-31T02:24:02.398 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-B9El7i0dKfR8KFb off=287106 len=1309714 2021-07-31T02:24:02.400 INFO:tasks.workunit.client.0.smithi045.stdout:op 5677 completed, throughput=5MB/sec 2021-07-31T02:24:02.401 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-sa6Wdwji7265iJj off=4912639 len=1386036 2021-07-31T02:24:02.401 INFO:tasks.workunit.client.0.smithi045.stdout:op 5678 completed, throughput=5MB/sec 2021-07-31T02:24:02.402 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-rDllOlWsBRPUlz6 off=579861 len=1092413 2021-07-31T02:24:02.405 INFO:tasks.workunit.client.0.smithi045.stdout:op 5679 completed, throughput=5MB/sec 2021-07-31T02:24:02.406 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-7uvyIzPd2fXy3rR off=2788855 len=256102 2021-07-31T02:24:02.407 INFO:tasks.workunit.client.0.smithi045.stdout:op 5680 completed, throughput=5MB/sec 2021-07-31T02:24:02.407 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-w1O_RZs5SIZznUc off=2278614 len=148917 2021-07-31T02:24:02.408 INFO:tasks.workunit.client.0.smithi045.stdout:op 5681 completed, throughput=5MB/sec 2021-07-31T02:24:02.408 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-5o-_UBK-1ZP8OXC off=11681324 len=938358 2021-07-31T02:24:02.412 INFO:tasks.workunit.client.0.smithi045.stdout:op 5682 completed, throughput=5MB/sec 2021-07-31T02:24:02.436 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:24:02.712 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:24:02.712 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 38m ago 39m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:24:02.712 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 36m ago 39m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:24:02.713 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 36m ago 42m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:24:02.713 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 38m ago 45m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:24:02.713 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 38m ago 45m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:24:02.713 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 36m ago 43m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:24:02.714 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 38m ago 43m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:24:02.714 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 38m ago 39m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:24:02.714 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 36m ago 39m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:24:02.714 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 38m ago 42m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:24:02.714 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 38m ago 42m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:24:02.715 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 38m ago 42m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:24:02.715 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 38m ago 41m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:24:02.715 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 36m ago 41m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:24:02.715 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 36m ago 41m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:24:02.716 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 36m ago 40m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:24:02.716 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 36m ago 40m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:24:02.716 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 36m ago 40m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:24:02.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:02 smithi154 conmon[34553]: cluster 2021-07-31T02:24:00.319812+0000 mgr.x (mgr.24542) 1575 : cluster [DBG] pgmap v1975: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 2.6 MiB/s rd, 3.8 MiB/s wr, 6 op/s 2021-07-31T02:24:02.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:02 smithi045 conmon[33655]: cluster 2021-07-31T02:24:00 2021-07-31T02:24:02.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:02 smithi045 conmon[33655]: .319812+0000 mgr.x (mgr.24542) 1575 : cluster [DBG] pgmap v1975: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 2.6 MiB/s rd, 3.8 MiB/s wr, 6 op/s 2021-07-31T02:24:02.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:02 smithi045 conmon[39718]: cluster 2021-07-31T02:24:00.319812 2021-07-31T02:24:02.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:02 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 1575 : cluster [DBG] pgmap v1975: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 2.6 MiB/s rd, 3.8 MiB/s wr, 6 op/s 2021-07-31T02:24:03.047 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:24:03.047 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:24:03.048 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:24:03.048 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:24:03.048 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:24:03.048 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:24:03.049 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:24:03.049 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:24:03.049 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:24:03.050 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:24:03.050 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:24:03.050 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:24:03.050 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:24:03.051 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:24:03.051 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:24:03.051 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:24:03.052 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:24:03.413 INFO:tasks.workunit.client.0.smithi045.stdout: 1114: throughput=5MB/sec pending data=0 2021-07-31T02:24:03.413 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-etjmhQsllpTYvc_ off=6002441 len=987853 2021-07-31T02:24:03.418 INFO:tasks.workunit.client.0.smithi045.stdout:op 5683 completed, throughput=5MB/sec 2021-07-31T02:24:03.418 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-zdWQI3m3PzEDoXH off=5295923 len=1333375 2021-07-31T02:24:03.420 INFO:tasks.workunit.client.0.smithi045.stdout:op 5684 completed, throughput=5MB/sec 2021-07-31T02:24:03.420 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-l3w-K_Ge6V1QMlF off=21617294 len=548416 2021-07-31T02:24:03.424 INFO:tasks.workunit.client.0.smithi045.stdout:op 5685 completed, throughput=5MB/sec 2021-07-31T02:24:03.425 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-_ur6JqfGe5QArqc off=5964188 len=435181 2021-07-31T02:24:03.425 INFO:tasks.workunit.client.0.smithi045.stdout:op 5686 completed, throughput=5MB/sec 2021-07-31T02:24:03.426 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-vFOf28Wk9zOdO21 off=1988891 len=842291 2021-07-31T02:24:03.426 INFO:tasks.workunit.client.0.smithi045.stdout:op 5687 completed, throughput=5MB/sec 2021-07-31T02:24:03.427 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-d0GMWdayRGji0kW off=12595592 len=1469280 2021-07-31T02:24:03.433 INFO:tasks.workunit.client.0.smithi045.stdout:op 5688 completed, throughput=5MB/sec 2021-07-31T02:24:03.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:03 smithi045 conmon[39718]: audit 2021-07-31T02:24:03.046545+0000 mon.c (mon.1) 94 : audit [DBG] from='client.? 172.21.15.45:0/2445064990' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:24:03.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:03 smithi045 conmon[33655]: audit 2021-07-31T02:24:03.046545+0000 mon.c (mon. 2021-07-31T02:24:03.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:03 smithi045 conmon[33655]: 1) 94 : audit [DBG] from='client.? 172.21.15.45:0/2445064990' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:24:03.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:03 smithi154 conmon[34553]: audit 2021-07-31T02:24:03.046545+0000 mon.c (mon.1) 94 : audit [DBG] from='client.? 172.21.15.45:0/2445064990' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:24:04.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:24:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:24:03.573Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:24:04.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:24:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:24:03.573Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:24:04.027 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:24:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:24:03.573Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:24:04.433 INFO:tasks.workunit.client.0.smithi045.stdout: 1115: throughput=5MB/sec pending data=0 2021-07-31T02:24:04.434 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-cz6dWWdGsZt_gX1 off=8073627 len=1492707 2021-07-31T02:24:04.441 INFO:tasks.workunit.client.0.smithi045.stdout:op 5689 completed, throughput=5MB/sec 2021-07-31T02:24:04.441 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-DpEKukBdcYSIyYL off=990082 len=612477 2021-07-31T02:24:04.443 INFO:tasks.workunit.client.0.smithi045.stdout:op 5690 completed, throughput=5MB/sec 2021-07-31T02:24:04.443 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-6TpOrS5kN4U4pKE off=1220983 len=107842 2021-07-31T02:24:04.446 INFO:tasks.workunit.client.0.smithi045.stdout:op 5691 completed, throughput=5MB/sec 2021-07-31T02:24:04.446 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-K38moGzQRlgBuup off=4829066 len=1104478 2021-07-31T02:24:04.447 INFO:tasks.workunit.client.0.smithi045.stdout:op 5692 completed, throughput=5MB/sec 2021-07-31T02:24:04.448 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-IvY3Ak4egxzIgbu off=865558 len=888488 2021-07-31T02:24:04.454 INFO:tasks.workunit.client.0.smithi045.stdout:op 5693 completed, throughput=5MB/sec 2021-07-31T02:24:04.454 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-smR2g0MMQbHjgCl off=1515802 len=577447 2021-07-31T02:24:04.459 INFO:tasks.workunit.client.0.smithi045.stdout:op 5694 completed, throughput=5MB/sec 2021-07-31T02:24:04.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:04 smithi154 conmon[34553]: cluster 2021-07-31T02:24:02.320293+0000 mgr.x (mgr.24542) 1576 : cluster [DBG] pgmap v1976: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2021-07-31T02:24:04.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:04 smithi154 conmon[34553]: audit 2021-07-31T02:24:02.424564+0000 mgr.x (mgr.24542) 1577 : audit [DBG] from='client.26290 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:24:04.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:04 smithi154 conmon[34553]: audit 2021-07-31T02:24:02.704414+0000 mgr.x (mgr.24542) 1578 : audit [DBG] from='client.16389 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:24:04.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:04 smithi045 conmon[39718]: cluster 2021-07-31T02:24:02.320293+0000 mgr.x (mgr.24542) 1576 : cluster 2021-07-31T02:24:04.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:04 smithi045 conmon[39718]: [DBG] pgmap v1976: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2021-07-31T02:24:04.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:04 smithi045 conmon[39718]: audit 2021-07-31T02:24:02.424564+0000 mgr.x (mgr.24542) 1577 : 2021-07-31T02:24:04.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:04 smithi045 conmon[39718]: audit [DBG] from='client.26290 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:24:04.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:04 smithi045 conmon[39718]: audit 2021-07-31T02:24:02.704414+0000 2021-07-31T02:24:04.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:04 smithi045 conmon[39718]: mgr.x (mgr.24542) 1578 : audit [DBG] from='client.16389 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:24:04.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:04 smithi045 conmon[33655]: cluster 2021-07-31T02:24:02.320293+0000 mgr.x (mgr.24542) 1576 2021-07-31T02:24:04.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:04 smithi045 conmon[33655]: : cluster [DBG] pgmap v1976: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2021-07-31T02:24:04.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:04 smithi045 conmon[33655]: audit 2021-07-31T02:24:02.424564+0000 mgr.x (mgr.24542) 1577 : audit [DBG] from='client.26290 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:24:04.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:04 smithi045 conmon[33655]: audit 2021-07-31T02:24: 2021-07-31T02:24:04.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:04 smithi045 conmon[33655]: 02.704414+0000 mgr.x (mgr.24542) 1578 : audit [DBG] from='client.16389 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:24:05.460 INFO:tasks.workunit.client.0.smithi045.stdout: 1116: throughput=5MB/sec pending data=0 2021-07-31T02:24:05.461 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-4BwsD4LMbMe3bFI off=14427900 len=1368448 2021-07-31T02:24:05.462 INFO:tasks.workunit.client.0.smithi045.stdout:op 5695 completed, throughput=5MB/sec 2021-07-31T02:24:05.462 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-VO75wD47tXNx65v off=22063837 len=2057361 2021-07-31T02:24:05.472 INFO:tasks.workunit.client.0.smithi045.stdout:op 5696 completed, throughput=5MB/sec 2021-07-31T02:24:05.473 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-RgOlSfGHVMtt3oi off=1278569 len=2072237 2021-07-31T02:24:05.475 INFO:tasks.workunit.client.0.smithi045.stdout:op 5697 completed, throughput=5MB/sec 2021-07-31T02:24:06.476 INFO:tasks.workunit.client.0.smithi045.stdout: 1117: throughput=5MB/sec pending data=0 2021-07-31T02:24:06.476 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-7vRGmHUKog9dnaw off=311173 len=228254 2021-07-31T02:24:06.477 INFO:tasks.workunit.client.0.smithi045.stdout:op 5698 completed, throughput=5MB/sec 2021-07-31T02:24:06.477 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-sUhCTNxU22q64Z3 off=37733 len=398568 2021-07-31T02:24:06.481 INFO:tasks.workunit.client.0.smithi045.stdout:op 5699 completed, throughput=5MB/sec 2021-07-31T02:24:06.481 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-lJH9GR-W_SFxDbo off=7545120 len=823601 2021-07-31T02:24:06.483 INFO:tasks.workunit.client.0.smithi045.stdout:op 5700 completed, throughput=5MB/sec 2021-07-31T02:24:06.483 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-WGlNo1tIakA3kvM off=5718975 len=1074726 2021-07-31T02:24:06.491 INFO:tasks.workunit.client.0.smithi045.stdout:op 5701 completed, throughput=5MB/sec 2021-07-31T02:24:06.491 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ZeCn5gS-tngfawL off=1713891 len=748885 2021-07-31T02:24:06.493 INFO:tasks.workunit.client.0.smithi045.stdout:op 5702 completed, throughput=5MB/sec 2021-07-31T02:24:06.493 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Hd-pWl07xS4qV5J off=11405458 len=1774710 2021-07-31T02:24:06.500 INFO:tasks.workunit.client.0.smithi045.stdout:op 5703 completed, throughput=5MB/sec 2021-07-31T02:24:06.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:06 smithi154 conmon[34553]: cluster 2021-07-31T02:24:04.320781+0000 mgr.x (mgr.24542) 1579 : cluster [DBG] pgmap v1977: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 2.7 MiB/s rd, 3.8 MiB/s wr, 6 op/s 2021-07-31T02:24:06.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:06 smithi045 conmon[33655]: cluster 2021-07-31T02:24:04.320781+0000 mgr.x (mgr.24542) 1579 : cluster [DBG] pgmap v1977: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 2.7 MiB/s rd, 3.8 MiB/s wr, 6 op/s 2021-07-31T02:24:06.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:06 smithi045 conmon[33655]: 2021-07-31T02:24:06.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:06 smithi045 conmon[39718]: cluster 2021-07-31T02:24:04.320781+0000 mgr.x (mgr.24542) 1579 : cluster [DBG] pgmap v1977: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 2.7 MiB/s rd, 3.8 MiB/s wr, 6 op/s 2021-07-31T02:24:07.501 INFO:tasks.workunit.client.0.smithi045.stdout: 1118: throughput=5MB/sec pending data=0 2021-07-31T02:24:07.501 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-4xHHLbQ_9M9-eHz off=16065262 len=2023795 2021-07-31T02:24:07.510 INFO:tasks.workunit.client.0.smithi045.stdout:op 5704 completed, throughput=5MB/sec 2021-07-31T02:24:07.510 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-8yoRpbeGrG0tddL off=4842977 len=285300 2021-07-31T02:24:07.511 INFO:tasks.workunit.client.0.smithi045.stdout:op 5705 completed, throughput=5MB/sec 2021-07-31T02:24:07.511 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-sDGXGqtrC3I8ofB off=6426786 len=410582 2021-07-31T02:24:07.515 INFO:tasks.workunit.client.0.smithi045.stdout:op 5706 completed, throughput=5MB/sec 2021-07-31T02:24:07.515 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-L1SQBcu51notra3 off=14366437 len=1534030 2021-07-31T02:24:07.523 INFO:tasks.workunit.client.0.smithi045.stdout:op 5707 completed, throughput=5MB/sec 2021-07-31T02:24:07.523 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-zSh22mmF0l966nE off=4004753 len=1920515 2021-07-31T02:24:07.532 INFO:tasks.workunit.client.0.smithi045.stdout:op 5708 completed, throughput=5MB/sec 2021-07-31T02:24:08.532 INFO:tasks.workunit.client.0.smithi045.stdout: 1119: throughput=5MB/sec pending data=0 2021-07-31T02:24:08.533 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-YzRIXVIqWPAGa0g off=11590301 len=1644167 2021-07-31T02:24:08.541 INFO:tasks.workunit.client.0.smithi045.stdout:op 5709 completed, throughput=5MB/sec 2021-07-31T02:24:08.542 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-BoUpwFcA3jNVTeF off=47578 len=1212748 2021-07-31T02:24:08.549 INFO:tasks.workunit.client.0.smithi045.stdout:op 5710 completed, throughput=5MB/sec 2021-07-31T02:24:08.550 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-B_4w32QqEQEnoBO off=2796111 len=1997830 2021-07-31T02:24:08.551 INFO:tasks.workunit.client.0.smithi045.stdout:op 5711 completed, throughput=5MB/sec 2021-07-31T02:24:08.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:08 smithi154 conmon[34553]: cluster 2021-07-31T02:24:06.321263+0000 mgr.x (mgr.24542) 1580 : cluster [DBG] pgmap v1978: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2021-07-31T02:24:08.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:08 smithi045 conmon[39718]: cluster 2021-07-31T02:24:06.321263+0000 2021-07-31T02:24:08.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:08 smithi045 conmon[39718]: mgr.x (mgr.24542) 1580 : cluster [DBG] pgmap v1978: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2021-07-31T02:24:08.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:08 smithi045 conmon[33655]: cluster 2021-07-31T02:24:06.321263+0000 mgr.x (mgr.24542 2021-07-31T02:24:08.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:08 smithi045 conmon[33655]: ) 1580 : cluster [DBG] pgmap v1978: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2021-07-31T02:24:09.551 INFO:tasks.workunit.client.0.smithi045.stdout: 1120: throughput=5MB/sec pending data=0 2021-07-31T02:24:09.552 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-fXc6DSM2YzkHJCt off=3700949 len=692243 2021-07-31T02:24:09.552 INFO:tasks.workunit.client.0.smithi045.stdout:op 5712 completed, throughput=5MB/sec 2021-07-31T02:24:09.553 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-j5suao-vFSnzFcg off=21101106 len=1230822 2021-07-31T02:24:09.559 INFO:tasks.workunit.client.0.smithi045.stdout:op 5713 completed, throughput=5MB/sec 2021-07-31T02:24:09.559 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-WnTWIRVnmoK7qEI off=8328313 len=535632 2021-07-31T02:24:09.560 INFO:tasks.workunit.client.0.smithi045.stdout:op 5714 completed, throughput=5MB/sec 2021-07-31T02:24:09.560 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-fpOgTibnjJSpnB3 off=2177204 len=752968 2021-07-31T02:24:09.562 INFO:tasks.workunit.client.0.smithi045.stdout:op 5715 completed, throughput=5MB/sec 2021-07-31T02:24:09.562 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-wDI5cKFQk_nOdFE off=16090934 len=855839 2021-07-31T02:24:09.563 INFO:tasks.workunit.client.0.smithi045.stdout:op 5716 completed, throughput=5MB/sec 2021-07-31T02:24:09.564 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-mESGi0iGeddSKBv off=4838766 len=1066040 2021-07-31T02:24:09.564 INFO:tasks.workunit.client.0.smithi045.stdout:op 5717 completed, throughput=5MB/sec 2021-07-31T02:24:09.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:09 smithi154 conmon[34553]: cluster 2021-07-31T02:24:08.321691+0000 mgr.x (mgr.24542) 1581 : cluster [DBG] pgmap v1979: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2021-07-31T02:24:09.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:09 smithi154 conmon[34553]: 2021-07-31T02:24:09.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:09 smithi045 conmon[39718]: cluster 2021-07-31T02:24:08.321691 2021-07-31T02:24:09.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:09 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 1581 : cluster [DBG] pgmap v1979: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2021-07-31T02:24:09.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:09 smithi045 conmon[33655]: cluster 2021-07-31T02:24:08.321691+0000 mgr.x (mgr 2021-07-31T02:24:09.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:09 smithi045 conmon[33655]: .24542) 1581 : cluster [DBG] pgmap v1979: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2021-07-31T02:24:10.564 INFO:tasks.workunit.client.0.smithi045.stdout: 1121: throughput=5MB/sec pending data=0 2021-07-31T02:24:10.564 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-PVWKX8Kv69jwAk9 off=6101583 len=1114727 2021-07-31T02:24:10.567 INFO:tasks.workunit.client.0.smithi045.stdout:op 5718 completed, throughput=5MB/sec 2021-07-31T02:24:10.567 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-b-Qb9k8vhlUFced off=3361034 len=875521 2021-07-31T02:24:10.568 INFO:tasks.workunit.client.0.smithi045.stdout:op 5719 completed, throughput=5MB/sec 2021-07-31T02:24:10.569 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Vl-RyQ037cWCJMB off=1224733 len=479470 2021-07-31T02:24:10.570 INFO:tasks.workunit.client.0.smithi045.stdout:op 5720 completed, throughput=5MB/sec 2021-07-31T02:24:10.571 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-onApNpElKw2lCbd off=3437161 len=592517 2021-07-31T02:24:10.573 INFO:tasks.workunit.client.0.smithi045.stdout:op 5721 completed, throughput=5MB/sec 2021-07-31T02:24:10.573 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-_orqsdLeUpb0Qng off=18919390 len=1190755 2021-07-31T02:24:10.579 INFO:tasks.workunit.client.0.smithi045.stdout:op 5722 completed, throughput=5MB/sec 2021-07-31T02:24:10.579 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-f3NbEJszqNXdnNV off=3890755 len=1910416 2021-07-31T02:24:10.589 INFO:tasks.workunit.client.0.smithi045.stdout:op 5723 completed, throughput=5MB/sec 2021-07-31T02:24:11.589 INFO:tasks.workunit.client.0.smithi045.stdout: 1122: throughput=5MB/sec pending data=0 2021-07-31T02:24:11.589 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-e49fwL93uBe6XWY off=875492 len=1733080 2021-07-31T02:24:11.599 INFO:tasks.workunit.client.0.smithi045.stdout:op 5724 completed, throughput=5MB/sec 2021-07-31T02:24:11.599 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-MYH3KHQxJNlNpgC off=4441560 len=591623 2021-07-31T02:24:11.600 INFO:tasks.workunit.client.0.smithi045.stdout:op 5725 completed, throughput=5MB/sec 2021-07-31T02:24:11.600 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-N5I4S3bIb5-UFzZ off=284651 len=494154 2021-07-31T02:24:11.600 INFO:tasks.workunit.client.0.smithi045.stdout:op 5726 completed, throughput=5MB/sec 2021-07-31T02:24:11.601 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-azbJZQCx7AAYd9r off=4299343 len=591636 2021-07-31T02:24:11.604 INFO:tasks.workunit.client.0.smithi045.stdout:op 5727 completed, throughput=5MB/sec 2021-07-31T02:24:11.605 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-M8oTpWd_E2yBC2H off=6164184 len=1745404 2021-07-31T02:24:11.612 INFO:tasks.workunit.client.0.smithi045.stdout:op 5728 completed, throughput=5MB/sec 2021-07-31T02:24:11.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:11 smithi154 conmon[34553]: cluster 2021-07-31T02:24:10.322198+0000 mgr.x (mgr.24542) 1582 : cluster [DBG] pgmap v1980: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 2.7 MiB/s rd, 3.7 MiB/s wr, 6 op/s 2021-07-31T02:24:11.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:11 smithi045 conmon[33655]: cluster 2021-07-31T02:24:10.322198+0000 mgr.x (mgr 2021-07-31T02:24:11.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:11 smithi045 conmon[33655]: .24542) 1582 : cluster [DBG] pgmap v1980: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 2.7 MiB/s rd, 3.7 MiB/s wr, 6 op/s 2021-07-31T02:24:11.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:11 smithi045 conmon[39718]: cluster 2021-07-31T02:24:10.322198+0000 mgr.x (mgr 2021-07-31T02:24:11.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:11 smithi045 conmon[39718]: .24542) 1582 : cluster [DBG] pgmap v1980: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 2.7 MiB/s rd, 3.7 MiB/s wr, 6 op/s 2021-07-31T02:24:12.613 INFO:tasks.workunit.client.0.smithi045.stdout: 1123: throughput=5MB/sec pending data=0 2021-07-31T02:24:12.613 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-xVn9arnbRpEREaq off=3374719 len=785911 2021-07-31T02:24:12.614 INFO:tasks.workunit.client.0.smithi045.stdout:op 5729 completed, throughput=5MB/sec 2021-07-31T02:24:12.614 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-FBAZB_e0mh_GYNa off=13445495 len=380326 2021-07-31T02:24:12.618 INFO:tasks.workunit.client.0.smithi045.stdout:op 5730 completed, throughput=5MB/sec 2021-07-31T02:24:12.618 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-hHKtqDTVji05HP4 off=3434101 len=987085 2021-07-31T02:24:12.626 INFO:tasks.workunit.client.0.smithi045.stdout:op 5731 completed, throughput=5MB/sec 2021-07-31T02:24:12.627 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-jHFi_i6s9-UJAHm off=538667 len=1049665 2021-07-31T02:24:12.628 INFO:tasks.workunit.client.0.smithi045.stdout:op 5732 completed, throughput=5MB/sec 2021-07-31T02:24:12.628 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-KOx2y_6sBb6ynSv off=578226 len=361141 2021-07-31T02:24:12.629 INFO:tasks.workunit.client.0.smithi045.stdout:op 5733 completed, throughput=5MB/sec 2021-07-31T02:24:12.629 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-SYnktOdCyiaR9T3 off=163595 len=1269402 2021-07-31T02:24:12.642 INFO:tasks.workunit.client.0.smithi045.stdout:op 5734 completed, throughput=5MB/sec 2021-07-31T02:24:13.636 INFO:tasks.workunit.client.0.smithi045.stdout: 1124: throughput=5MB/sec pending data=0 2021-07-31T02:24:13.637 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-SMuW-Ex1RSrd8cd off=11490914 len=1428930 2021-07-31T02:24:13.638 INFO:tasks.workunit.client.0.smithi045.stdout:op 5735 completed, throughput=5MB/sec 2021-07-31T02:24:13.638 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-6kqAjfH7vzq0m-z off=16227707 len=2055622 2021-07-31T02:24:13.642 INFO:tasks.workunit.client.0.smithi045.stdout:op 5736 completed, throughput=5MB/sec 2021-07-31T02:24:13.642 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Ekl0hb8BhjPi50J off=2120651 len=563218 2021-07-31T02:24:13.643 INFO:tasks.workunit.client.0.smithi045.stdout:op 5737 completed, throughput=5MB/sec 2021-07-31T02:24:13.643 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-oql-R3ZYAGBrQa2 off=7630390 len=697593 2021-07-31T02:24:13.644 INFO:tasks.workunit.client.0.smithi045.stdout:op 5738 completed, throughput=5MB/sec 2021-07-31T02:24:13.644 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Puk9dMQiGt9BdZ8 off=5534530 len=1161263 2021-07-31T02:24:13.650 INFO:tasks.workunit.client.0.smithi045.stdout:op 5739 completed, throughput=5MB/sec 2021-07-31T02:24:14.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:24:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:24:13.573Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:24:14.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:24:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:24:13.573Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:24:14.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:24:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:24:13.573Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:24:14.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:14 smithi154 conmon[34553]: cluster 2021-07-31T02:24:12.322683+0000 mgr.x (mgr.24542) 1583 : cluster [DBG] pgmap v1981: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2021-07-31T02:24:14.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:14 smithi045 conmon[39718]: cluster 2021-07-31T02:24:12.322683+0000 mgr.x (mgr.24542) 1583 : cluster [DBG] pgmap v1981: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2021-07-31T02:24:14.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:14 smithi045 conmon[33655]: cluster 2021-07-31T02 2021-07-31T02:24:14.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:14 smithi045 conmon[33655]: :24:12.322683+0000 mgr.x (mgr.24542) 1583 : cluster [DBG] pgmap v1981: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2021-07-31T02:24:14.650 INFO:tasks.workunit.client.0.smithi045.stdout: 1125: throughput=5MB/sec pending data=0 2021-07-31T02:24:14.651 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ill7QBEg9tcOilA off=14521919 len=481492 2021-07-31T02:24:14.654 INFO:tasks.workunit.client.0.smithi045.stdout:op 5740 completed, throughput=5MB/sec 2021-07-31T02:24:14.655 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-qQygdfPgDD651pM off=14350434 len=461541 2021-07-31T02:24:14.659 INFO:tasks.workunit.client.0.smithi045.stdout:op 5741 completed, throughput=5MB/sec 2021-07-31T02:24:14.659 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-gfBPpBBj0mzY9tn off=9848892 len=499626 2021-07-31T02:24:14.663 INFO:tasks.workunit.client.0.smithi045.stdout:op 5742 completed, throughput=5MB/sec 2021-07-31T02:24:14.665 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-sTRH4C198L4fW-T off=1902874 len=1171715 2021-07-31T02:24:14.666 INFO:tasks.workunit.client.0.smithi045.stdout:op 5743 completed, throughput=5MB/sec 2021-07-31T02:24:14.666 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-0SffUXo0D6Hn5vu off=1403105 len=435261 2021-07-31T02:24:14.666 INFO:tasks.workunit.client.0.smithi045.stdout:op 5744 completed, throughput=5MB/sec 2021-07-31T02:24:14.666 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-KZwSb0T2AW-6Qp5 off=3412362 len=1397473 2021-07-31T02:24:14.668 INFO:tasks.workunit.client.0.smithi045.stdout:op 5745 completed, throughput=5MB/sec 2021-07-31T02:24:14.669 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-5ktGtML11lwuMlO off=574711 len=1178314 2021-07-31T02:24:14.673 INFO:tasks.workunit.client.0.smithi045.stdout:op 5746 completed, throughput=5MB/sec 2021-07-31T02:24:15.673 INFO:tasks.workunit.client.0.smithi045.stdout: 1126: throughput=5MB/sec pending data=0 2021-07-31T02:24:15.674 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-KuzJ31ZUFdtLzB8 off=5567211 len=2046721 2021-07-31T02:24:15.686 INFO:tasks.workunit.client.0.smithi045.stdout:op 5747 completed, throughput=5MB/sec 2021-07-31T02:24:15.687 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-bthAZXHrcQXUuBr off=9823810 len=2057205 2021-07-31T02:24:15.696 INFO:tasks.workunit.client.0.smithi045.stdout:op 5748 completed, throughput=5MB/sec 2021-07-31T02:24:15.697 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Xh9vah62XwWp6FE off=3048110 len=1257452 2021-07-31T02:24:15.702 INFO:tasks.workunit.client.0.smithi045.stdout:op 5749 completed, throughput=5MB/sec 2021-07-31T02:24:16.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:16 smithi154 conmon[34553]: cluster 2021-07-31T02:24:14.323163+0000 mgr.x (mgr.24542) 1584 : cluster [DBG] pgmap v1982: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 2.9 MiB/s rd, 3.5 MiB/s wr, 6 op/s 2021-07-31T02:24:16.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:16 smithi045 conmon[39718]: cluster 2021-07-31T02:24:14.323163+0000 mgr.x (mgr.24542) 1584 : cluster [DBG] pgmap v1982: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 2.9 MiB/s rd, 3.5 MiB/s wr, 6 op/s 2021-07-31T02:24:16.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:16 smithi045 conmon[33655]: cluster 2021-07-31T02:24: 2021-07-31T02:24:16.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:16 smithi045 conmon[33655]: 14.323163+0000 mgr.x (mgr.24542) 1584 : cluster [DBG] pgmap v1982: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 2.9 MiB/s rd, 3.5 MiB/s wr, 6 op/s 2021-07-31T02:24:16.703 INFO:tasks.workunit.client.0.smithi045.stdout: 1127: throughput=5MB/sec pending data=0 2021-07-31T02:24:16.703 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-l7UZa-TZPQ_Eu37 off=5833856 len=1233785 2021-07-31T02:24:16.705 INFO:tasks.workunit.client.0.smithi045.stdout:op 5750 completed, throughput=5MB/sec 2021-07-31T02:24:16.705 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj--Ms6UbQjMQ8QRqK off=649138 len=1348511 2021-07-31T02:24:16.707 INFO:tasks.workunit.client.0.smithi045.stdout:op 5751 completed, throughput=5MB/sec 2021-07-31T02:24:16.708 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-vNbzflJ1frR_EVW off=492751 len=808967 2021-07-31T02:24:16.712 INFO:tasks.workunit.client.0.smithi045.stdout:op 5752 completed, throughput=5MB/sec 2021-07-31T02:24:16.713 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-FCNoxEE1tRq_ZoA off=11639946 len=1967524 2021-07-31T02:24:16.716 INFO:tasks.workunit.client.0.smithi045.stdout:op 5753 completed, throughput=5MB/sec 2021-07-31T02:24:17.717 INFO:tasks.workunit.client.0.smithi045.stdout: 1128: throughput=5MB/sec pending data=0 2021-07-31T02:24:17.717 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-MLQc7VD8VsZsGIS off=4288396 len=1160174 2021-07-31T02:24:17.723 INFO:tasks.workunit.client.0.smithi045.stdout:op 5754 completed, throughput=5MB/sec 2021-07-31T02:24:17.723 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-pd3doCQCK-ndp1w off=4358332 len=1021359 2021-07-31T02:24:17.726 INFO:tasks.workunit.client.0.smithi045.stdout:op 5755 completed, throughput=5MB/sec 2021-07-31T02:24:17.727 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-rUe36qjjJakoJwn off=17748906 len=61883 2021-07-31T02:24:17.729 INFO:tasks.workunit.client.0.smithi045.stdout:op 5756 completed, throughput=5MB/sec 2021-07-31T02:24:17.729 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-o3jy8acXMtXst20 off=5451233 len=1810017 2021-07-31T02:24:17.739 INFO:tasks.workunit.client.0.smithi045.stdout:op 5757 completed, throughput=5MB/sec 2021-07-31T02:24:17.739 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-cklxGBfUSSXsvcI off=8102330 len=1804942 2021-07-31T02:24:17.746 INFO:tasks.workunit.client.0.smithi045.stdout:op 5758 completed, throughput=5MB/sec 2021-07-31T02:24:18.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:18 smithi154 conmon[34553]: cluster 2021-07-31T02:24:16.323572+0000 mgr.x (mgr.24542) 1585 : cluster [DBG] 2021-07-31T02:24:18.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:18 smithi154 conmon[34553]: pgmap v1983: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2021-07-31T02:24:18.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:18 smithi045 conmon[33655]: cluster 2021-07-31T02:24:16.323572+0000 mgr.x (mgr.24542) 1585 : cluster [DBG] 2021-07-31T02:24:18.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:18 smithi045 conmon[33655]: pgmap v1983: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2021-07-31T02:24:18.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:18 smithi045 conmon[39718]: cluster 2021-07-31T02:24:16.323572+0000 mgr.x (mgr.24542) 1585 : 2021-07-31T02:24:18.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:18 smithi045 conmon[39718]: cluster [DBG] pgmap v1983: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2021-07-31T02:24:18.746 INFO:tasks.workunit.client.0.smithi045.stdout: 1129: throughput=5MB/sec pending data=0 2021-07-31T02:24:18.746 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-aP9-sub5JxDv5ZK off=10632297 len=23089 2021-07-31T02:24:18.748 INFO:tasks.workunit.client.0.smithi045.stdout:op 5759 completed, throughput=5MB/sec 2021-07-31T02:24:18.749 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-SdUoaDayXVex9y9 off=4897850 len=1438954 2021-07-31T02:24:18.752 INFO:tasks.workunit.client.0.smithi045.stdout:op 5760 completed, throughput=5MB/sec 2021-07-31T02:24:18.752 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-gk-q7_mJNGxDCMg off=7618034 len=1699985 2021-07-31T02:24:18.760 INFO:tasks.workunit.client.0.smithi045.stdout:op 5761 completed, throughput=5MB/sec 2021-07-31T02:24:18.760 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Kmibl1ImQVD9EEq off=15297064 len=448336 2021-07-31T02:24:18.761 INFO:tasks.workunit.client.0.smithi045.stdout:op 5762 completed, throughput=5MB/sec 2021-07-31T02:24:18.761 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-YwmHSw3m9h7QGBX off=19078125 len=88487 2021-07-31T02:24:18.762 INFO:tasks.workunit.client.0.smithi045.stdout:op 5763 completed, throughput=5MB/sec 2021-07-31T02:24:18.762 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj--2ZCR9qbK5ru_hP off=7182 len=1304335 2021-07-31T02:24:18.768 INFO:tasks.workunit.client.0.smithi045.stdout:op 5764 completed, throughput=5MB/sec 2021-07-31T02:24:19.768 INFO:tasks.workunit.client.0.smithi045.stdout: 1130: throughput=5MB/sec pending data=0 2021-07-31T02:24:19.768 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-V3SsoMuxZ7CdI0K off=3146794 len=763493 2021-07-31T02:24:19.773 INFO:tasks.workunit.client.0.smithi045.stdout:op 5765 completed, throughput=5MB/sec 2021-07-31T02:24:19.773 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-LSJNUKrnFKkfbXF off=13490658 len=1594930 2021-07-31T02:24:19.781 INFO:tasks.workunit.client.0.smithi045.stdout:op 5766 completed, throughput=5MB/sec 2021-07-31T02:24:19.781 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-0DR2WZ-_mJZ5OLG off=1149709 len=677386 2021-07-31T02:24:19.782 INFO:tasks.workunit.client.0.smithi045.stdout:op 5767 completed, throughput=5MB/sec 2021-07-31T02:24:19.782 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj--cpeJAQotKs8aYH off=2784771 len=1529495 2021-07-31T02:24:19.789 INFO:tasks.workunit.client.0.smithi045.stdout:op 5768 completed, throughput=5MB/sec 2021-07-31T02:24:20.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:20 smithi154 conmon[34553]: cluster 2021-07-31T02:24:18.324159+0000 mgr.x (mgr.24542) 1586 : cluster [DBG] pgmap v1984: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2021-07-31T02:24:20.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:20 smithi045 conmon[33655]: cluster 2021-07-31T02:24:18.324159+0000 mgr.x (mgr.24542) 1586 : cluster 2021-07-31T02:24:20.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:20 smithi045 conmon[33655]: [DBG] pgmap v1984: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2021-07-31T02:24:20.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:20 smithi045 conmon[39718]: cluster 2021-07-31T02:24:18.324159+0000 mgr.x (mgr.24542) 2021-07-31T02:24:20.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:20 smithi045 conmon[39718]: 1586 : cluster [DBG] pgmap v1984: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2021-07-31T02:24:20.789 INFO:tasks.workunit.client.0.smithi045.stdout: 1131: throughput=5MB/sec pending data=0 2021-07-31T02:24:20.790 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-4oaH9JrQjW-qZMy off=21539587 len=517594 2021-07-31T02:24:20.795 INFO:tasks.workunit.client.0.smithi045.stdout:op 5769 completed, throughput=5MB/sec 2021-07-31T02:24:20.795 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-uqkI0lOYyo_D2hK off=496954 len=1688282 2021-07-31T02:24:20.802 INFO:tasks.workunit.client.0.smithi045.stdout:op 5770 completed, throughput=5MB/sec 2021-07-31T02:24:20.802 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-zutZJncMyXWS_bi off=7545622 len=1759240 2021-07-31T02:24:20.811 INFO:tasks.workunit.client.0.smithi045.stdout:op 5771 completed, throughput=5MB/sec 2021-07-31T02:24:20.811 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-DjR96ZZrblDIiNP off=11102407 len=2046232 2021-07-31T02:24:20.814 INFO:tasks.workunit.client.0.smithi045.stdout:op 5772 completed, throughput=5MB/sec 2021-07-31T02:24:21.815 INFO:tasks.workunit.client.0.smithi045.stdout: 1132: throughput=5MB/sec pending data=0 2021-07-31T02:24:21.815 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-YApemt-k6rZuHZ1 off=16149607 len=759791 2021-07-31T02:24:21.820 INFO:tasks.workunit.client.0.smithi045.stdout:op 5773 completed, throughput=5MB/sec 2021-07-31T02:24:21.821 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-tlKWbS76-UyF6HZ off=4781202 len=661061 2021-07-31T02:24:21.822 INFO:tasks.workunit.client.0.smithi045.stdout:op 5774 completed, throughput=5MB/sec 2021-07-31T02:24:21.822 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-y5mAnAiYd-D4edH off=0 len=1150363 2021-07-31T02:24:21.828 INFO:tasks.workunit.client.0.smithi045.stdout:op 5775 completed, throughput=5MB/sec 2021-07-31T02:24:21.828 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-NkfReI0ZUiwAT6J off=16569407 len=1598538 2021-07-31T02:24:21.835 INFO:tasks.workunit.client.0.smithi045.stdout:op 5776 completed, throughput=5MB/sec 2021-07-31T02:24:21.835 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-CAk6WLKNJvNc312 off=4834147 len=1278160 2021-07-31T02:24:21.837 INFO:tasks.workunit.client.0.smithi045.stdout:op 5777 completed, throughput=5MB/sec 2021-07-31T02:24:22.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:22 smithi154 conmon[34553]: cluster 2021-07-31T02:24:20.324609+0000 mgr.x (mgr.24542) 1587 : cluster [DBG] pgmap v1985: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 2.8 MiB/s rd, 3.7 MiB/s wr, 6 op/s 2021-07-31T02:24:22.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:22 smithi154 conmon[34553]: 2021-07-31T02:24:22.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:22 smithi045 conmon[33655]: cluster 2021-07-31T02:24:20.324609+0000 2021-07-31T02:24:22.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:22 smithi045 conmon[33655]: mgr.x (mgr.24542) 1587 : cluster [DBG] pgmap v1985: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 2.8 MiB/s rd, 3.7 MiB/s wr, 6 op/s 2021-07-31T02:24:22.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:22 smithi045 conmon[39718]: cluster 2021-07-31T02:24:20.324609+0000 mgr.x (mgr.24542) 1587 : cluster [DBG] 2021-07-31T02:24:22.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:22 smithi045 conmon[39718]: pgmap v1985: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 2.8 MiB/s rd, 3.7 MiB/s wr, 6 op/s 2021-07-31T02:24:22.837 INFO:tasks.workunit.client.0.smithi045.stdout: 1133: throughput=5MB/sec pending data=0 2021-07-31T02:24:22.838 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-JiVHIqyrz3qxkBm off=11229899 len=642473 2021-07-31T02:24:22.839 INFO:tasks.workunit.client.0.smithi045.stdout:op 5778 completed, throughput=5MB/sec 2021-07-31T02:24:22.840 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-f9zCLY4vfS447_4 off=18516391 len=549773 2021-07-31T02:24:22.840 INFO:tasks.workunit.client.0.smithi045.stdout:op 5779 completed, throughput=5MB/sec 2021-07-31T02:24:22.841 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-KBpcJ0pEImzAGQw off=3908434 len=2008883 2021-07-31T02:24:22.849 INFO:tasks.workunit.client.0.smithi045.stdout:op 5780 completed, throughput=5MB/sec 2021-07-31T02:24:22.849 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-HyfFNAF0SmfplOD off=16788619 len=263905 2021-07-31T02:24:22.850 INFO:tasks.workunit.client.0.smithi045.stdout:op 5781 completed, throughput=5MB/sec 2021-07-31T02:24:22.850 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-J0p4FWB8w2PdXcq off=13044079 len=848478 2021-07-31T02:24:22.855 INFO:tasks.workunit.client.0.smithi045.stdout:op 5782 completed, throughput=5MB/sec 2021-07-31T02:24:23.855 INFO:tasks.workunit.client.0.smithi045.stdout: 1134: throughput=5MB/sec pending data=0 2021-07-31T02:24:23.856 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-KV86JuE4I_AB6Dw off=690804 len=95166 2021-07-31T02:24:23.859 INFO:tasks.workunit.client.0.smithi045.stdout:op 5783 completed, throughput=5MB/sec 2021-07-31T02:24:23.859 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-UolCfIm-DiexUQX off=2883644 len=1614961 2021-07-31T02:24:23.866 INFO:tasks.workunit.client.0.smithi045.stdout:op 5784 completed, throughput=5MB/sec 2021-07-31T02:24:23.867 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-s1Wp4s13Es5IFCF off=4614424 len=1067903 2021-07-31T02:24:23.869 INFO:tasks.workunit.client.0.smithi045.stdout:op 5785 completed, throughput=5MB/sec 2021-07-31T02:24:23.869 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-9vW9DhVDKJcEar0 off=3118459 len=455252 2021-07-31T02:24:23.873 INFO:tasks.workunit.client.0.smithi045.stdout:op 5786 completed, throughput=5MB/sec 2021-07-31T02:24:23.873 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-MEaVUjU-0LMQe4V off=20833015 len=454213 2021-07-31T02:24:23.876 INFO:tasks.workunit.client.0.smithi045.stdout:op 5787 completed, throughput=5MB/sec 2021-07-31T02:24:23.876 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-pmLrtDEQ2iLcHz1 off=4865251 len=1641663 2021-07-31T02:24:23.877 INFO:tasks.workunit.client.0.smithi045.stdout:op 5788 completed, throughput=5MB/sec 2021-07-31T02:24:24.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:24:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:24:23.573Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:24:24.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:24:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:24:23.573Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:24:24.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:24:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:24:23.573Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:24:24.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:24 smithi154 conmon[34553]: cluster 2021-07-31T02:24:22.325097+0000 mgr.x (mgr.24542) 1588 : cluster [DBG] pgmap v1986: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2021-07-31T02:24:24.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:24 smithi045 conmon[39718]: cluster 2021-07-31T02:24:22.325097+0000 mgr.x (mgr 2021-07-31T02:24:24.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:24 smithi045 conmon[39718]: .24542) 1588 : cluster [DBG] pgmap v1986: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2021-07-31T02:24:24.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:24 smithi045 conmon[33655]: cluster 2021-07-31T02:24:22.325097+0000 mgr.x (mgr.24542) 1588 : cluster [DBG] 2021-07-31T02:24:24.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:24 smithi045 conmon[33655]: pgmap v1986: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2021-07-31T02:24:24.878 INFO:tasks.workunit.client.0.smithi045.stdout: 1135: throughput=5MB/sec pending data=0 2021-07-31T02:24:24.878 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-oMnKybkOl7oT_0a off=11449907 len=1148943 2021-07-31T02:24:24.879 INFO:tasks.workunit.client.0.smithi045.stdout:op 5789 completed, throughput=5MB/sec 2021-07-31T02:24:24.880 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-avqxfyVAGfm5SlU off=12509583 len=1525754 2021-07-31T02:24:24.888 INFO:tasks.workunit.client.0.smithi045.stdout:op 5790 completed, throughput=5MB/sec 2021-07-31T02:24:24.888 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-YB_bmfJZu3_J_aN off=1486894 len=2087578 2021-07-31T02:24:24.889 INFO:tasks.workunit.client.0.smithi045.stdout:op 5791 completed, throughput=5MB/sec 2021-07-31T02:24:24.889 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-xoLJsHHSmklxRXN off=94197 len=1755910 2021-07-31T02:24:24.895 INFO:tasks.workunit.client.0.smithi045.stdout:op 5792 completed, throughput=5MB/sec 2021-07-31T02:24:25.895 INFO:tasks.workunit.client.0.smithi045.stdout: 1136: throughput=5MB/sec pending data=0 2021-07-31T02:24:25.895 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-qggFAlWnTS3xKnc off=2284034 len=1526332 2021-07-31T02:24:25.902 INFO:tasks.workunit.client.0.smithi045.stdout:op 5793 completed, throughput=5MB/sec 2021-07-31T02:24:25.903 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-GpUeJCY87ihsDC6 off=14124904 len=1519896 2021-07-31T02:24:25.910 INFO:tasks.workunit.client.0.smithi045.stdout:op 5794 completed, throughput=5MB/sec 2021-07-31T02:24:25.910 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-4M9V24CnhGjuZmK off=5176648 len=1924465 2021-07-31T02:24:25.921 INFO:tasks.workunit.client.0.smithi045.stdout:op 5795 completed, throughput=5MB/sec 2021-07-31T02:24:26.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:26 smithi154 conmon[34553]: cluster 2021-07-31T02:24:24.325705+0000 mgr.x (mgr.24542) 1589 : cluster [DBG] pgmap v1987: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 2.5 MiB/s rd, 3.5 MiB/s wr, 5 op/s 2021-07-31T02:24:26.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:26 smithi045 conmon[33655]: cluster 2021-07-31T02:24:24.325705+0000 2021-07-31T02:24:26.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:26 smithi045 conmon[33655]: mgr.x (mgr.24542) 1589 : cluster [DBG] pgmap v1987: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 2.5 MiB/s rd, 3.5 MiB/s wr, 5 op/s 2021-07-31T02:24:26.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:26 smithi045 conmon[39718]: cluster 2021-07-31T02:24:24.325705+0000 mgr.x ( 2021-07-31T02:24:26.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:26 smithi045 conmon[39718]: mgr.24542) 1589 : cluster [DBG] pgmap v1987: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 2.5 MiB/s rd, 3.5 MiB/s wr, 5 op/s 2021-07-31T02:24:26.922 INFO:tasks.workunit.client.0.smithi045.stdout: 1137: throughput=5MB/sec pending data=0 2021-07-31T02:24:26.923 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-hATmpHv74V9zpA2 off=2273324 len=1000048 2021-07-31T02:24:26.927 INFO:tasks.workunit.client.0.smithi045.stdout:op 5796 completed, throughput=5MB/sec 2021-07-31T02:24:26.927 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-FtNxHUoLRfs_Eoo off=15785155 len=280986 2021-07-31T02:24:26.930 INFO:tasks.workunit.client.0.smithi045.stdout:op 5797 completed, throughput=5MB/sec 2021-07-31T02:24:26.931 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-psty6UeuMxt1hK8 off=2599203 len=1401142 2021-07-31T02:24:26.933 INFO:tasks.workunit.client.0.smithi045.stdout:op 5798 completed, throughput=5MB/sec 2021-07-31T02:24:26.934 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-gCUhAbSE9-BFVlC off=18335924 len=1066607 2021-07-31T02:24:26.936 INFO:tasks.workunit.client.0.smithi045.stdout:op 5799 completed, throughput=5MB/sec 2021-07-31T02:24:26.936 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-dshBWL-iieYb0dk off=2427815 len=161727 2021-07-31T02:24:26.938 INFO:tasks.workunit.client.0.smithi045.stdout:op 5800 completed, throughput=5MB/sec 2021-07-31T02:24:26.939 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-OHEe927VWUeIrzR off=7871234 len=1381869 2021-07-31T02:24:26.945 INFO:tasks.workunit.client.0.smithi045.stdout:op 5801 completed, throughput=5MB/sec 2021-07-31T02:24:27.945 INFO:tasks.workunit.client.0.smithi045.stdout: 1138: throughput=5MB/sec pending data=0 2021-07-31T02:24:27.946 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-36f7dNnbWf5EBzM off=2077510 len=428035 2021-07-31T02:24:27.946 INFO:tasks.workunit.client.0.smithi045.stdout:op 5802 completed, throughput=5MB/sec 2021-07-31T02:24:27.947 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-5o5IvOj9E6fYjTm off=4396977 len=708685 2021-07-31T02:24:27.952 INFO:tasks.workunit.client.0.smithi045.stdout:op 5803 completed, throughput=5MB/sec 2021-07-31T02:24:27.953 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-h4uXx4-HzSwnU4f off=4274011 len=160890 2021-07-31T02:24:27.953 INFO:tasks.workunit.client.0.smithi045.stdout:op 5804 completed, throughput=5MB/sec 2021-07-31T02:24:27.953 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-6rKpCWlLdJ4ntlP off=7684658 len=1734627 2021-07-31T02:24:27.963 INFO:tasks.workunit.client.0.smithi045.stdout:op 5805 completed, throughput=5MB/sec 2021-07-31T02:24:27.964 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-oESOrMbmbKwARbk off=1188325 len=2096809 2021-07-31T02:24:27.973 INFO:tasks.workunit.client.0.smithi045.stdout:op 5806 completed, throughput=5MB/sec 2021-07-31T02:24:28.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:28 smithi154 conmon[34553]: cluster 2021-07-31T02:24:26.326261+0000 mgr.x (mgr.24542) 1590 : cluster [DBG] pgmap v1988: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 3 op/s 2021-07-31T02:24:28.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:28 smithi154 conmon[34553]: audit 2021-07-31T02:24:28.060271+0000 mon.b (mon.2) 253 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:24:28.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:28 smithi154 conmon[34553]: audit 2021-07-31T02:24:28.060771+0000 mon.a (mon.0) 1907 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:24:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:28 smithi154 conmon[34553]: audit 2021-07-31T02:24:28.083707+0000 mon.b (mon.2) 2021-07-31T02:24:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:28 smithi154 conmon[34553]: 254 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:24:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:28 smithi154 conmon[34553]: audit 2021-07- 2021-07-31T02:24:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:28 smithi154 conmon[34553]: 31T02:24:28.084414+0000 mon.a (mon 2021-07-31T02:24:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:28 smithi154 conmon[34553]: .0) 1908 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:24:28.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:28 smithi045 conmon[39718]: cluster 2021-07-31T02:24:26.326261+0000 mgr.x (mgr.24542) 1590 : 2021-07-31T02:24:28.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:28 smithi045 conmon[39718]: cluster [DBG] pgmap v1988: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 3 op/s 2021-07-31T02:24:28.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:28 smithi045 conmon[39718]: audit 2021-07-31T02:24:28.060271 2021-07-31T02:24:28.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:28 smithi045 conmon[39718]: +0000 mon.b (mon.2) 253 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:24:28.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:28 smithi045 conmon[39718]: audit 2021-07-31T02:24:28.060771+0000 2021-07-31T02:24:28.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:28 smithi045 conmon[39718]: mon.a (mon.0) 1907 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:24:28.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:28 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T02:24:28.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:28 smithi045 conmon[39718]: 31T02:24:28.083707+0000 mon.b (mon 2021-07-31T02:24:28.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:28 smithi045 conmon[39718]: .2) 254 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:24:28.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:28 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T02:24:28.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:28 smithi045 conmon[39718]: 02:24:28.084414+0000 mon.a (mon.0) 1908 : audit 2021-07-31T02:24:28.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:28 smithi045 conmon[39718]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:24:28.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:28 smithi045 conmon[33655]: cluster 2021-07-31T02:24:26.326261+0000 mgr.x (mgr.24542) 1590 : cluster 2021-07-31T02:24:28.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:28 smithi045 conmon[33655]: [DBG] pgmap v1988: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 3 op/s 2021-07-31T02:24:28.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:28 smithi045 conmon[33655]: audit 2021-07-31T02:24:28.060271+0000 mon.b (mon.2) 253 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:24:28.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:28 smithi045 conmon[33655]: audit 2021-07-31T02:24:28.060771+0000 mon.a (mon.0) 1907 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:24:28.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:28 smithi045 conmon[33655]: 2021-07-31T02:24:28.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:28 smithi045 conmon[33655]: audit 2021-07-31T02 2021-07-31T02:24:28.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:28 smithi045 conmon[33655]: :24:28.083707+0000 mon.b (mon.2) 254 2021-07-31T02:24:28.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:28 smithi045 conmon[33655]: : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:24:28.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:28 smithi045 conmon[33655]: audit 2021-07-31T02:24:28.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:28 smithi045 conmon[33655]: 2021-07- 2021-07-31T02:24:28.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:28 smithi045 conmon[33655]: 31T02:24:28.084414+0000 mon.a (mon.0 2021-07-31T02:24:28.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:28 smithi045 conmon[33655]: ) 1908 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:24:28.973 INFO:tasks.workunit.client.0.smithi045.stdout: 1139: throughput=5MB/sec pending data=0 2021-07-31T02:24:28.973 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-qHhbde6P2gQQG3Z off=11627897 len=910399 2021-07-31T02:24:28.979 INFO:tasks.workunit.client.0.smithi045.stdout:op 5807 completed, throughput=5MB/sec 2021-07-31T02:24:28.979 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-WeqaMWMDV6agFXS off=1273735 len=468828 2021-07-31T02:24:28.983 INFO:tasks.workunit.client.0.smithi045.stdout:op 5808 completed, throughput=5MB/sec 2021-07-31T02:24:28.983 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-_OjftXqAymoTo5e off=2469484 len=807112 2021-07-31T02:24:28.988 INFO:tasks.workunit.client.0.smithi045.stdout:op 5809 completed, throughput=5MB/sec 2021-07-31T02:24:28.988 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-h1OklJ_1oZ9BElH off=7005664 len=479024 2021-07-31T02:24:28.989 INFO:tasks.workunit.client.0.smithi045.stdout:op 5810 completed, throughput=5MB/sec 2021-07-31T02:24:28.989 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-XT9azZtEE5a6AoV off=368238 len=1307138 2021-07-31T02:24:28.991 INFO:tasks.workunit.client.0.smithi045.stdout:op 5811 completed, throughput=5MB/sec 2021-07-31T02:24:28.991 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-epiQMENPQuGhzfr off=12799917 len=1632211 2021-07-31T02:24:28.997 INFO:tasks.workunit.client.0.smithi045.stdout:op 5812 completed, throughput=5MB/sec 2021-07-31T02:24:29.997 INFO:tasks.workunit.client.0.smithi045.stdout: 1140: throughput=5MB/sec pending data=0 2021-07-31T02:24:30.001 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-7Of0gruegbVq-zA off=2915444 len=1430982 2021-07-31T02:24:30.002 INFO:tasks.workunit.client.0.smithi045.stdout:op 5813 completed, throughput=5MB/sec 2021-07-31T02:24:30.002 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-E6twaoz6PsvxqD9 off=7042773 len=1373505 2021-07-31T02:24:30.010 INFO:tasks.workunit.client.0.smithi045.stdout:op 5814 completed, throughput=5MB/sec 2021-07-31T02:24:30.010 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-cBYxsQtaC9t_Jds off=0 len=1438451 2021-07-31T02:24:30.014 INFO:tasks.workunit.client.0.smithi045.stdout:op 5815 completed, throughput=5MB/sec 2021-07-31T02:24:30.015 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-b_9Rr08ybsvQ_SO off=4567707 len=749734 2021-07-31T02:24:30.019 INFO:tasks.workunit.client.0.smithi045.stdout:op 5816 completed, throughput=5MB/sec 2021-07-31T02:24:30.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:30 smithi154 conmon[34553]: cluster 2021-07-31T02:24:28.326736+0000 mgr.x (mgr.24542) 1591 : cluster [DBG] 2021-07-31T02:24:30.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:30 smithi154 conmon[34553]: pgmap v1989: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 3 op/s 2021-07-31T02:24:30.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:30 smithi045 conmon[39718]: cluster 2021-07-31T02:24:28.326736+0000 mgr.x (mgr 2021-07-31T02:24:30.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:30 smithi045 conmon[39718]: .24542) 1591 : cluster [DBG] pgmap v1989: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 3 op/s 2021-07-31T02:24:30.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:30 smithi045 conmon[33655]: cluster 2021-07-31T02:24:28.326736+0000 mgr.x (mgr.24542 2021-07-31T02:24:30.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:30 smithi045 conmon[33655]: ) 1591 : cluster [DBG] pgmap v1989: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 3 op/s 2021-07-31T02:24:31.020 INFO:tasks.workunit.client.0.smithi045.stdout: 1141: throughput=5MB/sec pending data=0 2021-07-31T02:24:31.020 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Z-GZCJm1s5rbgPJ off=17896 len=579249 2021-07-31T02:24:31.021 INFO:tasks.workunit.client.0.smithi045.stdout:op 5817 completed, throughput=5MB/sec 2021-07-31T02:24:31.022 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-MNrOXmKHJN4yfus off=12260903 len=1672652 2021-07-31T02:24:31.025 INFO:tasks.workunit.client.0.smithi045.stdout:op 5818 completed, throughput=5MB/sec 2021-07-31T02:24:31.025 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-DdDdDS4JeNKdMyq off=4431887 len=1644215 2021-07-31T02:24:31.028 INFO:tasks.workunit.client.0.smithi045.stdout:op 5819 completed, throughput=5MB/sec 2021-07-31T02:24:31.028 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-dGNUKtInK85bDRC off=4099399 len=12672 2021-07-31T02:24:31.030 INFO:tasks.workunit.client.0.smithi045.stdout:op 5820 completed, throughput=5MB/sec 2021-07-31T02:24:31.030 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-WHQhCjS9wqQVe2Q off=7165725 len=1105004 2021-07-31T02:24:31.036 INFO:tasks.workunit.client.0.smithi045.stdout:op 5821 completed, throughput=5MB/sec 2021-07-31T02:24:32.003 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: debug 2021-07-31T02:24:31.750+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:24:32.003 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:24:32.003 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:24:32.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: yield 2021-07-31T02:24:32.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:24:32.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:24:32.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:24:32.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:24:32.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:24:32.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:24:32.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:24:32.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:24:32.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:24:32.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: task.result() 2021-07-31T02:24:32.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:24:32.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: debug 2021-07-31T02:24:31.762+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:24:32.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:24:32.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:24:32.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: yield 2021-07-31T02:24:32.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:24:32.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:24:32.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:24:32.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:24:32.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:24:32.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:24:32.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:24:32.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:24:32.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:24:32.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: task.result() 2021-07-31T02:24:32.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:24:32.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: debug 2021-07-31T02:24:31.788+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:24:32.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:24:32.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:24:32.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: yield 2021-07-31T02:24:32.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:24:32.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:24:32.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:24:32.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:24:32.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:24:32.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:24:32.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:24:32.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:24:32.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:24:32.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: task.result() 2021-07-31T02:24:32.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:24:32.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: debug 2021-07-31T02:24:31.788+0000 7fcb9195d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:24:32.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:24:32.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:24:32.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: yield 2021-07-31T02:24:32.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:24:32.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:24:32.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:24:32.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:24:32.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:24:32.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:24:32.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:24:32.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:24:32.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:24:32.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: task.result() 2021-07-31T02:24:32.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:24:32.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: 2021-07-31T02:24:32.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:24:32.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: 2021-07-31T02:24:32.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:24:32.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:24:32.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:24:32.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:24:32.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:24:32.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:24:32.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:24:32.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:24:32.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:24:32.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:24:32.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:24:32.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:24:32.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:24:32.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:24:32.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:24:32.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:24:32.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:24:32.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:24:32.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: raise 2021-07-31T02:24:32.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:24:32.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:24:32.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:24:32.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:24:32.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:24:32.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: debug 2021-07-31T02:24:31.797+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:24:32.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:24:32.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:24:32.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: yield 2021-07-31T02:24:32.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:24:32.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:24:32.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:24:32.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:24:32.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:24:32.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:24:32.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:24:32.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:24:32.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:24:32.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: task.result() 2021-07-31T02:24:32.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:24:32.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: debug 2021-07-31T02:24:31.797+0000 7fcb9215e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:24:32.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:24:32.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:24:32.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: yield 2021-07-31T02:24:32.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:24:32.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:24:32.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:24:32.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:24:32.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:24:32.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:24:32.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:24:32.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:24:32.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:24:32.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: task.result() 2021-07-31T02:24:32.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:24:32.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: 2021-07-31T02:24:32.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:24:32.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: 2021-07-31T02:24:32.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:24:32.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:24:32.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:24:32.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:24:32.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:24:32.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:24:32.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:24:32.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:24:32.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:24:32.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:24:32.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:24:32.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:24:32.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:24:32.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:24:32.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:24:32.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:24:32.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:24:32.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:24:32.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: raise 2021-07-31T02:24:32.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:24:32.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:24:32.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:24:32.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:24:32.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:24:31 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:24:32.036 INFO:tasks.workunit.client.0.smithi045.stdout: 1142: throughput=5MB/sec pending data=0 2021-07-31T02:24:32.037 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-L_nDQBocBM2ZHeo off=7514714 len=2085610 2021-07-31T02:24:32.040 INFO:tasks.workunit.client.0.smithi045.stdout:op 5822 completed, throughput=5MB/sec 2021-07-31T02:24:32.040 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-AbDqQIHoZV5lAWp off=10916414 len=1610126 2021-07-31T02:24:32.048 INFO:tasks.workunit.client.0.smithi045.stdout:op 5823 completed, throughput=5MB/sec 2021-07-31T02:24:32.048 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-08Ux6LTtRHS7sPC off=6505045 len=2065784 2021-07-31T02:24:32.051 INFO:tasks.workunit.client.0.smithi045.stdout:op 5824 completed, throughput=5MB/sec 2021-07-31T02:24:32.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:32 smithi154 conmon[34553]: cluster 2021-07-31T02:24:30.327321+0000 mgr.x (mgr.24542) 1592 : cluster [DBG] pgmap v1990: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 2.2 MiB/s rd, 3.8 MiB/s wr, 5 op/s 2021-07-31T02:24:32.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:32 smithi154 conmon[34553]: audit 2021-07-31T02:24:31.705284+0000 mon.b (mon.2) 255 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:24:32.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:32 smithi154 conmon[34553]: audit 2021-07-31T02:24:31.706672+0000 mon.b (mon.2) 256 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:24:32.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:32 smithi154 conmon[34553]: audit 2021-07-31T02:24:31.707611+0000 mon.b 2021-07-31T02:24:32.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:32 smithi154 conmon[34553]: (mon.2) 257 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:24:32.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:32 smithi154 conmon[34553]: 2021-07-31T02:24:32.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:32 smithi045 conmon[33655]: cluster 2021-07-31T02:24:30.327321+0000 mgr.x (mgr.24542) 1592 : cluster [DBG] pgmap v1990: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 2.2 MiB/s rd, 3.8 MiB/s wr, 5 op/s 2021-07-31T02:24:32.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:32 smithi045 conmon[33655]: 2021-07-31T02:24:32.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:32 smithi045 conmon[33655]: audit 2021-07-31T02:24:31.705284+0000 mon.b (mon.2) 255 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:24:32.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:32 smithi045 conmon[33655]: audit 2021-07-31T02:24:31.706672+0000 mon.b 2021-07-31T02:24:32.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:32 smithi045 conmon[33655]: (mon.2) 256 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:24:32.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:32 smithi045 conmon[33655]: audit 2021-07-31T02:24:32.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:32 smithi045 conmon[33655]: 2021-07-31T02:24:31.707611 2021-07-31T02:24:32.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:32 smithi045 conmon[33655]: +0000 mon.b (mon.2) 257 : audit [INF] 2021-07-31T02:24:32.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:32 smithi045 conmon[33655]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:24:32.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:32 smithi045 conmon[39718]: cluster 2021-07-31T02:24:30.327321+0000 mgr.x ( 2021-07-31T02:24:32.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:32 smithi045 conmon[39718]: mgr.24542) 1592 : cluster [DBG] pgmap v1990: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 2.2 MiB/s rd, 3.8 MiB/s wr, 5 op/s 2021-07-31T02:24:32.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:32 smithi045 conmon[39718]: audit 2021-07-31T02:24:31.705284+0000 mon.b (mon.2) 255 2021-07-31T02:24:32.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:32 smithi045 conmon[39718]: : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:24:32.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:32 smithi045 conmon[39718]: audit 2021-07-31T02:24:31.706672+0000 mon.b 2021-07-31T02:24:32.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:32 smithi045 conmon[39718]: (mon.2) 256 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:24:32.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:32 smithi045 conmon[39718]: audit 2021-07-31T02:24:32.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:32 smithi045 conmon[39718]: 2021-07-31T02:24:31.707611+0000 mon.b (mon.2) 257 : audit [INF] 2021-07-31T02:24:32.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:32 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:24:33.052 INFO:tasks.workunit.client.0.smithi045.stdout: 1143: throughput=5MB/sec pending data=0 2021-07-31T02:24:33.052 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-tpd-nGoIKrqgWnv off=2391889 len=819353 2021-07-31T02:24:33.057 INFO:tasks.workunit.client.0.smithi045.stdout:op 5825 completed, throughput=5MB/sec 2021-07-31T02:24:33.057 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-6ubYPBMCjDkU3-B off=1092810 len=1233813 2021-07-31T02:24:33.064 INFO:tasks.workunit.client.0.smithi045.stdout:op 5826 completed, throughput=5MB/sec 2021-07-31T02:24:33.064 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-jH-xcLlSuoqPhVT off=4912047 len=521659 2021-07-31T02:24:33.065 INFO:tasks.workunit.client.0.smithi045.stdout:op 5827 completed, throughput=5MB/sec 2021-07-31T02:24:33.065 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-YvMn0IvfUtDKP-2 off=3548400 len=1123493 2021-07-31T02:24:33.071 INFO:tasks.workunit.client.0.smithi045.stdout:op 5828 completed, throughput=5MB/sec 2021-07-31T02:24:33.071 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-VTQz_7xMBmc5RI1 off=7739844 len=271533 2021-07-31T02:24:33.072 INFO:tasks.workunit.client.0.smithi045.stdout:op 5829 completed, throughput=5MB/sec 2021-07-31T02:24:33.073 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-HRGRredg-V_WQP0 off=66975 len=77640 2021-07-31T02:24:33.073 INFO:tasks.workunit.client.0.smithi045.stdout:op 5830 completed, throughput=5MB/sec 2021-07-31T02:24:33.073 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-m25DnnxSQ356-nS off=10118218 len=656332 2021-07-31T02:24:33.076 INFO:tasks.workunit.client.0.smithi045.stdout:op 5831 completed, throughput=5MB/sec 2021-07-31T02:24:33.077 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-eBcymEPMMg3XJpF off=15093478 len=567226 2021-07-31T02:24:33.080 INFO:tasks.workunit.client.0.smithi045.stdout:op 5832 completed, throughput=5MB/sec 2021-07-31T02:24:33.363 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:24:33.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:24:31.708367+0000 mgr.x (mgr.24542) 1593 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:24:33.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:24:31.751358+0000 mgr.x 2021-07-31T02:24:33.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: (mgr.24542) 1594 : cephadm [ERR] 2021-07-31T02:24:33.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:24:33.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:24:33.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: yield 2021-07-31T02:24:33.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:24:33.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:24:33.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:24:33.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:24:33.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:24:33.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:24:33.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:24:33.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:24:33.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:24:33.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:24:33.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:24:33.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:24:31.751885+0000 mgr.x (mgr.24542) 1595 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:24:33.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:24:31.762997+0000 mgr.x (mgr.24542) 1596 : cephadm 2021-07-31T02:24:33.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: [ERR] 2021-07-31T02:24:33.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:24:33.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:24:33.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: yield 2021-07-31T02:24:33.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:24:33.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:24:33.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:24:33.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:24:33.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:24:33.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:24:33.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:24:33.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:24:33.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:24:33.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:24:33.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:24:33.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:24:31.763415+0000 mgr.x (mgr.24542) 1597 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:24:33.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:24:31.789345+0000 mgr.x (mgr.24542) 1598 : cephadm [ERR] 2021-07-31T02:24:33.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:24:33.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:24:33.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: yield 2021-07-31T02:24:33.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:24:33.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:24:33.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:24:33.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:24:33.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:24:33.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:24:33.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:24:33.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:24:33.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:24:33.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:24:33.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:24:33.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:24:31.789776+0000 mgr.x (mgr.24542) 1599 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:24:33.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:24:33.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:24:33.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: yield 2021-07-31T02:24:33.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:24:33.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:24:33.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:24:33.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:24:33.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:24:33.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:24:33.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:24:33.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:24:33.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:24:33.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:24:33.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:24:33.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: 2021-07-31T02:24:33.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:24:33.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: 2021-07-31T02:24:33.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:24:33.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:24:33.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:24:33.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:24:33.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:24:33.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:24:33.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:24:33.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:24:33.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:24:33.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:24:33.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:24:33.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:24:33.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:24:33.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:24:33.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:24:33.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:24:33.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:24:33.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:24:33.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: raise 2021-07-31T02:24:33.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:24:33.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:24:33.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:24:33.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:24:33.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:24:33.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:24:33.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: 2021-07-31T02:24:31.797991+0000 mgr.x (mgr.24542) 1600 : cephadm [ERR] 2021-07-31T02:24:33.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:24:33.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:24:33.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: yield 2021-07-31T02:24:33.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:24:33.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:24:33.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:24:33.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:24:33.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:24:33.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:24:33.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:24:33.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:24:33.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:24:33.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:24:33.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:24:33.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:24:31.798382+0000 mgr.x (mgr.24542) 1601 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:24:33.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:24:33.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:24:33.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: yield 2021-07-31T02:24:33.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:24:33.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:24:33.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:24:33.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:24:33.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:24:33.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:24:33.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:24:33.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:24:33.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:24:33.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:24:33.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:24:33.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: 2021-07-31T02:24:33.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:24:33.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: 2021-07-31T02:24:33.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:24:33.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:24:33.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:24:33.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:24:33.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:24:33.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:24:33.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:24:33.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:24:33.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:24:33.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:24:33.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:24:33.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:24:33.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:24:33.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:24:33.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:24:33.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:24:33.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:24:33.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:24:33.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: raise 2021-07-31T02:24:33.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:24:33.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:24:33.541 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:24:33.541 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:24:33.541 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:33 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:24:33.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: cephadm 2021-07-31T02: 2021-07-31T02:24:33.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: 24:31.708367+0000 mgr.x (mgr.24542) 1593 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:24:33.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: cephadm 2021-07 2021-07-31T02:24:33.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: -31T02:24:31. 2021-07-31T02:24:33.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: 751358+0000 mgr.x (mgr.24542) 2021-07-31T02:24:33.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: 1594 : cephadm [ERR] 2021-07-31T02:24:33.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:24:33.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:24:33.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: yield 2021-07-31T02:24:33.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:24:33.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:24:33.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:24:33.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:24:33.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:24:33.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:24:33.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:24:33.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:24:33.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:24:33.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:24:33.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:24:33.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: 2021-07-31T02:24:33.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:24:31.751885+0000 mgr.x (mgr.24542) 1595 : cephadm 2021-07-31T02:24:33.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:24:33.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: cephadm 2021 2021-07-31T02:24:33.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: -07-31T02:24: 2021-07-31T02:24:33.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: 31.762997+0000 mgr.x ( 2021-07-31T02:24:33.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: mgr.24542) 1596 : cephadm 2021-07-31T02:24:33.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: [ERR] 2021-07-31T02:24:33.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:24:33.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:24:33.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: yield 2021-07-31T02:24:33.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:24:33.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:24:33.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:24:33.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:24:33.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:24:33.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:24:33.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:24:33.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:24:33.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:24:33.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:24:33.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:24:33.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:24:31.763415+0000 mgr.x (mgr.24542) 1597 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:24:33.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: 2021-07-31T02:24:33.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: cephadm 2021- 2021-07-31T02:24:33.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: 07-31T02:24: 2021-07-31T02:24:33.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: 31.789345+0000 mgr.x (mgr 2021-07-31T02:24:33.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: .24542) 1598 : cephadm 2021-07-31T02:24:33.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: [ERR] 2021-07-31T02:24:33.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:24:33.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:24:33.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: yield 2021-07-31T02:24:33.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:24:33.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:24:33.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:24:33.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:24:33.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:24:33.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:24:33.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:24:33.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:24:33.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:24:33.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:24:33.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:24:33.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:24:31.789776+0000 mgr.x (mgr.24542) 1599 2021-07-31T02:24:33.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:24:33.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:24:33.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:24:33.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: yield 2021-07-31T02:24:33.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:24:33.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:24:33.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:24:33.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:24:33.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:24:33.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:24:33.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:24:33.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:24:33.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:24:33.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:24:33.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:24:33.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: 2021-07-31T02:24:33.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:24:33.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: 2021-07-31T02:24:33.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:24:33.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:24:33.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:24:33.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:24:33.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:24:33.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:24:31. 2021-07-31T02:24:33.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: 708367+0000 mgr.x (mgr.24542) 1593 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:24:33.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:24:31.751358+0000 mgr.x (mgr. 2021-07-31T02:24:33.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: 24542) 1594 : cephadm [ERR] 2021-07-31T02:24:33.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:24:33.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:24:33.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: yield 2021-07-31T02:24:33.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:24:33.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:24:33.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:24:33.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:24:33.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:24:33.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:24:33.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:24:33.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:24:33.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:24:33.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:24:33.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:24:33.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:24:31.751885+0000 mgr.x (mgr.24542) 1595 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:24:33.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:24:33.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: 2021-07-31T02: 2021-07-31T02:24:33.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: 24:31.762997+0000 mgr.x 2021-07-31T02:24:33.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: (mgr.24542) 1596 : cephadm 2021-07-31T02:24:33.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: [ERR] 2021-07-31T02:24:33.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:24:33.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:24:33.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: yield 2021-07-31T02:24:33.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:24:33.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:24:33.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:24:33.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:24:33.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:24:33.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:24:33.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:24:33.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:24:33.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:24:33.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:24:33.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:24:33.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:24:31.763415+0000 mgr.x (mgr.24542) 1597 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:24:33.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: cephadm 2021- 2021-07-31T02:24:33.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: 07-31T02:24:31.789345+0000 2021-07-31T02:24:33.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:24:33.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:24:33.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:24:33.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:24:33.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:24:33.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:24:33.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:24:33.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:24:33.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:24:33.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:24:33.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:24:33.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:24:33.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:24:33.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: raise 2021-07-31T02:24:33.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:24:33.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:24:33.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:24:33.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:24:33.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:24:33.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:24:31.797991+0000 mgr.x (mgr.24542) 1600 : cephadm [ERR] 2021-07-31T02:24:33.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:24:33.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:24:33.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: yield 2021-07-31T02:24:33.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:24:33.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:24:33.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:24:33.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:24:33.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:24:33.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:24:33.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:24:33.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:24:33.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:24:33.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:24:33.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:24:33.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:24:31.798382+0000 mgr.x (mgr.24542) 1601 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:24:33.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:24:33.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:24:33.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: yield 2021-07-31T02:24:33.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:24:33.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: mgr.x (mgr.24542) 1598 : cephadm 2021-07-31T02:24:33.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: [ERR] 2021-07-31T02:24:33.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:24:33.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:24:33.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: yield 2021-07-31T02:24:33.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:24:33.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:24:33.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:24:33.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:24:33.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:24:33.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:24:33.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:24:33.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:24:33.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:24:33.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:24:33.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:24:33.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:24:31.789776+0000 mgr.x (mgr.24542) 1599 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:24:33.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:24:33.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:24:33.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: yield 2021-07-31T02:24:33.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:24:33.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:24:33.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:24:33.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:24:33.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:24:33.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:24:33.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:24:33.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:24:33.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:24:33.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:24:33.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:24:33.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: 2021-07-31T02:24:33.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:24:33.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: 2021-07-31T02:24:33.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:24:33.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:24:33.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:24:33.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:24:33.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:24:33.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:24:33.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:24:33.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:24:33.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:24:33.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:24:33.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:24:33.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:24:33.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:24:33.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:24:33.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:24:33.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:24:33.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:24:33.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:24:33.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: 2021-07-31T02:24:33.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:24:33.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: 2021-07-31T02:24:33.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:24:33.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:24:33.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:24:33.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:24:33.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:24:33.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:24:33.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:24:33.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:24:33.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:24:33.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:24:33.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:24:33.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:24:33.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:24:33.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:24:33.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:24:33.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:24:33.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:24:33.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:24:33.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: raise 2021-07-31T02:24:33.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:24:33.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:24:33.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:24:33.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:24:33.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:24:33.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:24:33.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:24:33.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:24:33.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:24:33.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:24:33.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:24:33.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:24:33.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:24:33.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:24:33.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:24:33.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: raise 2021-07-31T02:24:33.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:24:33.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:24:33.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:24:33.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:24:33.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:24:33.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:24:31.797991+0000 mgr.x (mgr.24542) 1600 : cephadm [ERR] 2021-07-31T02:24:33.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:24:33.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:24:33.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: yield 2021-07-31T02:24:33.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:24:33.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:24:33.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:24:33.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:24:33.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:24:33.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:24:33.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:24:33.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:24:33.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:24:33.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:24:33.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:24:33.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:24:31.798382+0000 mgr.x (mgr.24542) 1601 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:24:33.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:24:33.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:24:33.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: yield 2021-07-31T02:24:33.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:24:33.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:24:33.608 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:24:33.608 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:24:33.608 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:24:33.608 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:24:33.608 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:24:33.609 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:24:33.609 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:24:33.609 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:24:33.609 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:24:33.610 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: 2021-07-31T02:24:33.610 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:24:33.610 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: 2021-07-31T02:24:33.610 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:24:33.611 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:24:33.611 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:24:33.611 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:24:33.611 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:24:33.611 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:24:33.612 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:24:33.612 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:24:33.612 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:24:33.612 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:24:33.613 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:24:33.613 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:24:33.613 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:24:33.613 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:24:33.613 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:24:33.614 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:24:33.614 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:24:33.614 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:24:33.614 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: raise 2021-07-31T02:24:33.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:24:33.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:24:33.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:24:33.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:24:33.616 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:24:33.650 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:24:33.651 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 39m ago 40m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:24:33.651 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 37m ago 39m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:24:33.651 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 37m ago 43m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:24:33.651 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 39m ago 45m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:24:33.652 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 39m ago 45m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:24:33.652 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 37m ago 43m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:24:33.652 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 39m ago 43m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:24:33.652 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 39m ago 40m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:24:33.653 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 37m ago 40m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:24:33.653 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 39m ago 43m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:24:33.653 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 39m ago 42m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:24:33.653 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 39m ago 42m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:24:33.653 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 39m ago 42m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:24:33.654 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 37m ago 41m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:24:33.654 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 37m ago 41m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:24:33.654 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 37m ago 41m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:24:33.655 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 37m ago 40m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:24:33.655 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 37m ago 40m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:24:33.985 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:24:33.985 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:24:33.985 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:24:33.986 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:24:33.986 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:24:33.986 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:24:33.986 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:24:33.986 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:24:33.987 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:24:33.987 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:24:33.987 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:24:33.988 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:24:33.988 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:24:33.988 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:24:33.989 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:24:33.989 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:24:33.989 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:24:34.033 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:24:33.574Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:24:34.034 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:24:33.574Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:24:34.034 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:24:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:24:33.574Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:24:34.081 INFO:tasks.workunit.client.0.smithi045.stdout: 1144: throughput=5MB/sec pending data=0 2021-07-31T02:24:34.081 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-9C3CypJ1AGdiqnG off=149370 len=915415 2021-07-31T02:24:34.087 INFO:tasks.workunit.client.0.smithi045.stdout:op 5833 completed, throughput=5MB/sec 2021-07-31T02:24:34.087 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-spuewV_C8Ri1O60 off=6035213 len=371201 2021-07-31T02:24:34.088 INFO:tasks.workunit.client.0.smithi045.stdout:op 5834 completed, throughput=5MB/sec 2021-07-31T02:24:34.088 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-xBWupweci5KdOVg off=3685557 len=1621041 2021-07-31T02:24:34.091 INFO:tasks.workunit.client.0.smithi045.stdout:op 5835 completed, throughput=5MB/sec 2021-07-31T02:24:34.091 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Wz2oWlfQG3trPLZ off=4991378 len=1467230 2021-07-31T02:24:34.093 INFO:tasks.workunit.client.0.smithi045.stdout:op 5836 completed, throughput=5MB/sec 2021-07-31T02:24:34.094 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-lfeb6BFjUHsFW7a off=14292849 len=1363076 2021-07-31T02:24:34.095 INFO:tasks.workunit.client.0.smithi045.stdout:op 5837 completed, throughput=5MB/sec 2021-07-31T02:24:34.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:34 smithi154 conmon[34553]: cluster 2021-07-31T02:24:32.327830+0000 mgr.x (mgr.24542) 1602 : cluster [DBG] pgmap v1991: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.3 MiB/s rd, 2.4 MiB/s wr, 3 op/s 2021-07-31T02:24:34.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:34 smithi154 conmon[34553]: 2021-07-31T02:24:34.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:34 smithi154 conmon[34553]: audit 2021-07-31T02:24:33.985403+0000 mon.a (mon.0) 1909 : audit [DBG] from='client.? 172.21.15.45:0/4230257667' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:24:34.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:34 smithi045 conmon[33655]: cluster 2021-07-31T02:24:32.327830+0000 mgr.x (mgr.24542) 2021-07-31T02:24:34.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:34 smithi045 conmon[33655]: 1602 : cluster [DBG] pgmap v1991: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.3 MiB/s rd, 2.4 MiB/s wr, 3 op/s 2021-07-31T02:24:34.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:34 smithi045 conmon[33655]: audit 2021-07-31T02:24:33.985403+0000 mon.a (mon.0) 1909 : audit [DBG] from='client.? 172.21.15.45:0/4230257667' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:24:34.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:34 smithi045 conmon[39718]: cluster 2021-07-31T02:24:32.327830+0000 2021-07-31T02:24:34.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:34 smithi045 conmon[39718]: mgr.x (mgr.24542) 1602 : cluster [DBG] pgmap v1991: 9 pgs: 9 active+clean; 127 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.3 MiB/s rd, 2.4 MiB/s wr, 3 op/s 2021-07-31T02:24:34.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:34 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T02:24:34.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:34 smithi045 conmon[39718]: T02:24:33.985403+0000 mon.a (mon.0) 1909 : audit 2021-07-31T02:24:34.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:34 smithi045 conmon[39718]: [DBG] from='client.? 172.21.15.45:0/4230257667' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:24:35.096 INFO:tasks.workunit.client.0.smithi045.stdout: 1145: throughput=5MB/sec pending data=0 2021-07-31T02:24:35.096 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-DAUGZQTtSyIUOL_ off=1669742 len=60303 2021-07-31T02:24:35.097 INFO:tasks.workunit.client.0.smithi045.stdout:op 5838 completed, throughput=5MB/sec 2021-07-31T02:24:35.097 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-cyj-FlO1UMuvREl off=5827527 len=272147 2021-07-31T02:24:35.098 INFO:tasks.workunit.client.0.smithi045.stdout:op 5839 completed, throughput=5MB/sec 2021-07-31T02:24:35.098 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-baQcxNTkuiSRWZa off=23615675 len=691663 2021-07-31T02:24:35.099 INFO:tasks.workunit.client.0.smithi045.stdout:op 5840 completed, throughput=5MB/sec 2021-07-31T02:24:35.100 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-b_lBYavSv754Uwt off=18394875 len=194923 2021-07-31T02:24:35.100 INFO:tasks.workunit.client.0.smithi045.stdout:op 5841 completed, throughput=5MB/sec 2021-07-31T02:24:35.101 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-LjBMok7J1NpeONp off=2062518 len=1321234 2021-07-31T02:24:35.102 INFO:tasks.workunit.client.0.smithi045.stdout:op 5842 completed, throughput=5MB/sec 2021-07-31T02:24:35.102 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-GiJqIbFdgcVz2c5 off=1195758 len=254493 2021-07-31T02:24:35.103 INFO:tasks.workunit.client.0.smithi045.stdout:op 5843 completed, throughput=5MB/sec 2021-07-31T02:24:35.103 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-fOSvWvwmw7iJSUl off=9177279 len=328132 2021-07-31T02:24:35.106 INFO:tasks.workunit.client.0.smithi045.stdout:op 5844 completed, throughput=5MB/sec 2021-07-31T02:24:35.106 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-bmWlcpnQG34924g off=19508337 len=2018313 2021-07-31T02:24:35.116 INFO:tasks.workunit.client.0.smithi045.stdout:op 5845 completed, throughput=5MB/sec 2021-07-31T02:24:35.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:35 smithi154 conmon[34553]: audit 2021-07-31T02:24:33.351844+0000 mgr.x (mgr.24542) 1603 : audit [DBG] from='client.26308 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:24:35.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:35 smithi154 conmon[34553]: audit 2021-07-31T02:24:33.642833+0000 mgr.x (mgr.24542) 1604 : audit [DBG] from='client.26314 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:24:35.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:35 smithi045 conmon[33655]: audit 2021-07-31T02:24:33.351844+0000 mgr.x (mgr.24542 2021-07-31T02:24:35.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:35 smithi045 conmon[33655]: ) 1603 : audit [DBG] from='client.26308 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:24:35.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:35 smithi045 conmon[33655]: audit 2021-07-31T02:24:33.642833+0000 mgr.x (mgr.24542) 1604 : audit [DBG] from='client.26314 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:24:35.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:35 smithi045 conmon[39718]: audit 2021-07-31T02:24:33.351844+0000 mgr.x (mgr 2021-07-31T02:24:35.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:35 smithi045 conmon[39718]: .24542) 1603 : audit [DBG] from='client.26308 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:24:35.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:35 smithi045 conmon[39718]: audit 2021-07-31T02:24:33.642833+0000 mgr.x (mgr.24542) 1604 : audit [DBG] from='client.26314 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:24:36.117 INFO:tasks.workunit.client.0.smithi045.stdout: 1146: throughput=5MB/sec pending data=0 2021-07-31T02:24:36.117 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-EwrI6Y2XiFND8ho off=13635805 len=751029 2021-07-31T02:24:36.118 INFO:tasks.workunit.client.0.smithi045.stdout:op 5846 completed, throughput=5MB/sec 2021-07-31T02:24:36.118 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-xD4bQDcu_H6jLk9 off=1287311 len=363514 2021-07-31T02:24:36.120 INFO:tasks.workunit.client.0.smithi045.stdout:op 5847 completed, throughput=5MB/sec 2021-07-31T02:24:36.120 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-eU6Od6-YC0HO-Et off=17976705 len=579121 2021-07-31T02:24:36.123 INFO:tasks.workunit.client.0.smithi045.stdout:op 5848 completed, throughput=5MB/sec 2021-07-31T02:24:36.124 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-zQEkMkPtNF_0r1j off=8851330 len=1118473 2021-07-31T02:24:36.131 INFO:tasks.workunit.client.0.smithi045.stdout:op 5849 completed, throughput=5MB/sec 2021-07-31T02:24:36.131 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-l8LJ4mBmQxJHt7N off=4068334 len=1795898 2021-07-31T02:24:36.134 INFO:tasks.workunit.client.0.smithi045.stdout:op 5850 completed, throughput=5MB/sec 2021-07-31T02:24:36.134 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-xdJWITyeoBZ9cMo off=16505202 len=1537010 2021-07-31T02:24:36.136 INFO:tasks.workunit.client.0.smithi045.stdout:op 5851 completed, throughput=5MB/sec 2021-07-31T02:24:36.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:36 smithi154 conmon[34553]: cluster 2021-07-31T02:24:34.328235+0000 mgr.x (mgr.24542) 1605 : cluster [DBG] pgmap v1992: 9 pgs: 9 active+clean; 127 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 2.4 MiB/s rd, 3.5 MiB/s wr, 5 op/s 2021-07-31T02:24:36.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:36 smithi045 conmon[39718]: cluster 2021-07-31T02:24:34.328235+0000 mgr.x (mgr 2021-07-31T02:24:36.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:36 smithi045 conmon[39718]: .24542) 1605 : cluster [DBG] pgmap v1992: 9 pgs: 9 active+clean; 127 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 2.4 MiB/s rd, 3.5 MiB/s wr, 5 op/s 2021-07-31T02:24:36.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:36 smithi045 conmon[33655]: cluster 2021-07-31T02:24:34.328235+0000 mgr.x (mgr.24542) 1605 : cluster [DBG] pgmap v1992: 9 pgs: 9 active+clean; 127 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 2.4 MiB/s rd, 3.5 MiB/s wr, 5 op/s 2021-07-31T02:24:37.137 INFO:tasks.workunit.client.0.smithi045.stdout: 1147: throughput=5MB/sec pending data=0 2021-07-31T02:24:37.137 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-6mjfQNVy1SwHDIA off=5689807 len=703851 2021-07-31T02:24:37.142 INFO:tasks.workunit.client.0.smithi045.stdout:op 5852 completed, throughput=5MB/sec 2021-07-31T02:24:37.142 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-yv3zh9dlDNsSiah off=2032312 len=768970 2021-07-31T02:24:37.147 INFO:tasks.workunit.client.0.smithi045.stdout:op 5853 completed, throughput=5MB/sec 2021-07-31T02:24:37.147 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-iLs4AeX8qtG2JOJ off=13782814 len=593467 2021-07-31T02:24:37.151 INFO:tasks.workunit.client.0.smithi045.stdout:op 5854 completed, throughput=5MB/sec 2021-07-31T02:24:37.152 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-4cLWlsnT-lhalfn off=250127 len=971302 2021-07-31T02:24:37.153 INFO:tasks.workunit.client.0.smithi045.stdout:op 5855 completed, throughput=5MB/sec 2021-07-31T02:24:37.154 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ExPRInMq58CNXX2 off=7544975 len=274869 2021-07-31T02:24:37.154 INFO:tasks.workunit.client.0.smithi045.stdout:op 5856 completed, throughput=5MB/sec 2021-07-31T02:24:37.155 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-7gax0YZQ8piqBYl off=2775893 len=243762 2021-07-31T02:24:37.155 INFO:tasks.workunit.client.0.smithi045.stdout:op 5857 completed, throughput=5MB/sec 2021-07-31T02:24:37.156 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-rkX2jASw41ohNOF off=12727040 len=539784 2021-07-31T02:24:37.157 INFO:tasks.workunit.client.0.smithi045.stdout:op 5858 completed, throughput=5MB/sec 2021-07-31T02:24:38.157 INFO:tasks.workunit.client.0.smithi045.stdout: 1148: throughput=5MB/sec pending data=0 2021-07-31T02:24:38.157 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-QXDB-SQgRwXFlUd off=402900 len=1383332 2021-07-31T02:24:38.165 INFO:tasks.workunit.client.0.smithi045.stdout:op 5859 completed, throughput=5MB/sec 2021-07-31T02:24:38.165 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-DHFQ7UpkOgQ16C2 off=15319838 len=275150 2021-07-31T02:24:38.166 INFO:tasks.workunit.client.0.smithi045.stdout:op 5860 completed, throughput=5MB/sec 2021-07-31T02:24:38.166 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-8y06hMcXagfxUiM off=21760726 len=580480 2021-07-31T02:24:38.170 INFO:tasks.workunit.client.0.smithi045.stdout:op 5861 completed, throughput=5MB/sec 2021-07-31T02:24:38.170 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-3VauABfTpEzJ7nR off=389229 len=1051288 2021-07-31T02:24:38.171 INFO:tasks.workunit.client.0.smithi045.stdout:op 5862 completed, throughput=5MB/sec 2021-07-31T02:24:38.171 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-IV1uHlS6jLkR000 off=1996980 len=1400306 2021-07-31T02:24:38.174 INFO:tasks.workunit.client.0.smithi045.stdout:op 5863 completed, throughput=5MB/sec 2021-07-31T02:24:38.174 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-se7AgWyWhJED85Y off=583801 len=646885 2021-07-31T02:24:38.176 INFO:tasks.workunit.client.0.smithi045.stdout:op 5864 completed, throughput=5MB/sec 2021-07-31T02:24:38.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:38 smithi154 conmon[34553]: cluster 2021-07-31T02:24:36.328623+0000 mgr.x (mgr.24542) 1606 : cluster [DBG] pgmap v1993: 9 pgs: 9 active+clean; 127 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2021-07-31T02:24:38.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:38 smithi045 conmon[39718]: cluster 2021-07-31T02:24:36.328623+0000 mgr.x (mgr 2021-07-31T02:24:38.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:38 smithi045 conmon[39718]: .24542) 1606 : cluster [DBG] pgmap v1993: 9 pgs: 9 active+clean; 127 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2021-07-31T02:24:38.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:38 smithi045 conmon[33655]: cluster 2021-07-31T02:24:36.328623+0000 mgr.x (mgr.24542) 1606 : cluster [DBG] pgmap v1993: 9 pgs: 9 active+clean; 127 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2021-07-31T02:24:39.176 INFO:tasks.workunit.client.0.smithi045.stdout: 1149: throughput=5MB/sec pending data=0 2021-07-31T02:24:39.176 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Pr7R1vI62rcSpoV off=1460465 len=270554 2021-07-31T02:24:39.177 INFO:tasks.workunit.client.0.smithi045.stdout:op 5865 completed, throughput=5MB/sec 2021-07-31T02:24:39.177 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-cHb5kdIz50evCSU off=0 len=1200808 2021-07-31T02:24:39.180 INFO:tasks.workunit.client.0.smithi045.stdout:op 5866 completed, throughput=5MB/sec 2021-07-31T02:24:39.181 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-_Vq2P7tpbszGmYN off=583856 len=995398 2021-07-31T02:24:39.181 INFO:tasks.workunit.client.0.smithi045.stdout:op 5867 completed, throughput=5MB/sec 2021-07-31T02:24:39.182 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-wyMIElXH1BzZKFk off=18707548 len=627763 2021-07-31T02:24:39.186 INFO:tasks.workunit.client.0.smithi045.stdout:op 5868 completed, throughput=5MB/sec 2021-07-31T02:24:39.186 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-z7nyJxtEYA028HK off=5314800 len=1770971 2021-07-31T02:24:39.189 INFO:tasks.workunit.client.0.smithi045.stdout:op 5869 completed, throughput=5MB/sec 2021-07-31T02:24:39.189 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-KKc9ktXY-0JvbWG off=11106283 len=1569955 2021-07-31T02:24:39.196 INFO:tasks.workunit.client.0.smithi045.stdout:op 5870 completed, throughput=5MB/sec 2021-07-31T02:24:40.196 INFO:tasks.workunit.client.0.smithi045.stdout: 1150: throughput=5MB/sec pending data=0 2021-07-31T02:24:40.197 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-eMcIi8GTO45p7uA off=2311283 len=1175956 2021-07-31T02:24:40.199 INFO:tasks.workunit.client.0.smithi045.stdout:op 5871 completed, throughput=5MB/sec 2021-07-31T02:24:40.199 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-uLc5hnwz1v0sKMb off=534532 len=1309704 2021-07-31T02:24:40.201 INFO:tasks.workunit.client.0.smithi045.stdout:op 5872 completed, throughput=5MB/sec 2021-07-31T02:24:40.201 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-BjTIFxv-6OFqDpG off=4716584 len=675678 2021-07-31T02:24:40.206 INFO:tasks.workunit.client.0.smithi045.stdout:op 5873 completed, throughput=5MB/sec 2021-07-31T02:24:40.206 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-cBRXDruFhJPx4nX off=5571031 len=889678 2021-07-31T02:24:40.207 INFO:tasks.workunit.client.0.smithi045.stdout:op 5874 completed, throughput=5MB/sec 2021-07-31T02:24:40.208 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj--xYffa-7J2FyvV1 off=3583485 len=810482 2021-07-31T02:24:40.212 INFO:tasks.workunit.client.0.smithi045.stdout:op 5875 completed, throughput=5MB/sec 2021-07-31T02:24:40.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:40 smithi154 conmon[34553]: cluster 2021-07-31T02:24:38.329197+0000 mgr.x (mgr.24542) 1607 : cluster [DBG] pgmap v1994: 9 pgs: 9 active+clean; 127 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2021-07-31T02:24:40.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:40 smithi045 conmon[39718]: cluster 2021-07-31T02:24:38.329197+0000 mgr.x (mgr.24542) 1607 : 2021-07-31T02:24:40.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:40 smithi045 conmon[39718]: cluster [DBG] pgmap v1994: 9 pgs: 9 active+clean; 127 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2021-07-31T02:24:40.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:40 smithi045 conmon[33655]: cluster 2021-07-31T02:24:38.329197+0000 mgr.x (mgr.24542 2021-07-31T02:24:40.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:40 smithi045 conmon[33655]: ) 1607 : cluster [DBG] pgmap v1994: 9 pgs: 9 active+clean; 127 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2021-07-31T02:24:41.213 INFO:tasks.workunit.client.0.smithi045.stdout: 1151: throughput=5MB/sec pending data=0 2021-07-31T02:24:41.213 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-e4h65FN5M1L52dP off=4864799 len=523283 2021-07-31T02:24:41.217 INFO:tasks.workunit.client.0.smithi045.stdout:op 5876 completed, throughput=5MB/sec 2021-07-31T02:24:41.217 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-egDrP5WjB2VdZqp off=7283688 len=875758 2021-07-31T02:24:41.218 INFO:tasks.workunit.client.0.smithi045.stdout:op 5877 completed, throughput=5MB/sec 2021-07-31T02:24:41.219 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-FFv_Ysq6trWh1Q1 off=968562 len=1879109 2021-07-31T02:24:41.226 INFO:tasks.workunit.client.0.smithi045.stdout:op 5878 completed, throughput=5MB/sec 2021-07-31T02:24:41.227 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-MdZzTA4AHLB4GCD off=159916 len=6652 2021-07-31T02:24:41.229 INFO:tasks.workunit.client.0.smithi045.stdout:op 5879 completed, throughput=5MB/sec 2021-07-31T02:24:41.229 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-kEb_MSIlJPCmlYe off=21210510 len=2093546 2021-07-31T02:24:41.231 INFO:tasks.workunit.client.0.smithi045.stdout:op 5880 completed, throughput=5MB/sec 2021-07-31T02:24:42.232 INFO:tasks.workunit.client.0.smithi045.stdout: 1152: throughput=5MB/sec pending data=0 2021-07-31T02:24:42.232 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-I8Cbovf1yclX0Dg off=11259790 len=1303170 2021-07-31T02:24:42.239 INFO:tasks.workunit.client.0.smithi045.stdout:op 5881 completed, throughput=5MB/sec 2021-07-31T02:24:42.239 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-_iXOHgc3VahchXa off=2047296 len=27923 2021-07-31T02:24:42.241 INFO:tasks.workunit.client.0.smithi045.stdout:op 5882 completed, throughput=5MB/sec 2021-07-31T02:24:42.242 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-YoGAb26a0jPBiC8 off=676613 len=1994135 2021-07-31T02:24:42.252 INFO:tasks.workunit.client.0.smithi045.stdout:op 5883 completed, throughput=5MB/sec 2021-07-31T02:24:42.252 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-KMHzKm0ucFksB1f off=14111404 len=220988 2021-07-31T02:24:42.252 INFO:tasks.workunit.client.0.smithi045.stdout:op 5884 completed, throughput=5MB/sec 2021-07-31T02:24:42.253 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-EI3j92un-CanWcn off=12182308 len=1748031 2021-07-31T02:24:42.263 INFO:tasks.workunit.client.0.smithi045.stdout:op 5885 completed, throughput=5MB/sec 2021-07-31T02:24:42.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:42 smithi154 conmon[34553]: cluster 2021-07-31T02:24:40.329639+0000 mgr.x (mgr.24542) 1608 : cluster [DBG] pgmap v1995: 9 pgs: 9 active+clean; 127 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 3.3 MiB/s rd, 3.0 MiB/s wr, 6 op/s 2021-07-31T02:24:42.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:42 smithi045 conmon[33655]: cluster 2021-07-31T02:24:40.329639+0000 mgr.x (mgr.24542) 1608 : cluster [DBG] 2021-07-31T02:24:42.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:42 smithi045 conmon[33655]: pgmap v1995: 9 pgs: 9 active+clean; 127 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 3.3 MiB/s rd, 3.0 MiB/s wr, 6 op/s 2021-07-31T02:24:42.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:42 smithi045 conmon[39718]: cluster 2021-07-31T02:24:40.329639+0000 mgr.x 2021-07-31T02:24:42.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:42 smithi045 conmon[39718]: (mgr.24542) 1608 : cluster [DBG] pgmap v1995: 9 pgs: 9 active+clean; 127 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 3.3 MiB/s rd, 3.0 MiB/s wr, 6 op/s 2021-07-31T02:24:43.264 INFO:tasks.workunit.client.0.smithi045.stdout: 1153: throughput=5MB/sec pending data=0 2021-07-31T02:24:43.264 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-f5OrC3D2IQvmIAa off=16731337 len=924097 2021-07-31T02:24:43.266 INFO:tasks.workunit.client.0.smithi045.stdout:op 5886 completed, throughput=5MB/sec 2021-07-31T02:24:43.266 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-RCgwizbieefxOgK off=2474973 len=651773 2021-07-31T02:24:43.267 INFO:tasks.workunit.client.0.smithi045.stdout:op 5887 completed, throughput=5MB/sec 2021-07-31T02:24:43.267 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-4--elFJlnnp3tnc off=12024356 len=481996 2021-07-31T02:24:43.272 INFO:tasks.workunit.client.0.smithi045.stdout:op 5888 completed, throughput=5MB/sec 2021-07-31T02:24:43.272 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-IGBd3kGOCNCefxq off=8987582 len=374589 2021-07-31T02:24:43.273 INFO:tasks.workunit.client.0.smithi045.stdout:op 5889 completed, throughput=5MB/sec 2021-07-31T02:24:43.273 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-7KyfgQK1CMhTDTk off=16323687 len=312567 2021-07-31T02:24:43.274 INFO:tasks.workunit.client.0.smithi045.stdout:op 5890 completed, throughput=5MB/sec 2021-07-31T02:24:43.274 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-VIsRP0W7ICb4gH6 off=6826429 len=78067 2021-07-31T02:24:43.276 INFO:tasks.workunit.client.0.smithi045.stdout:op 5891 completed, throughput=5MB/sec 2021-07-31T02:24:43.276 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-dOvzKICEXbdjQD_ off=578150 len=500392 2021-07-31T02:24:43.277 INFO:tasks.workunit.client.0.smithi045.stdout:op 5892 completed, throughput=5MB/sec 2021-07-31T02:24:43.278 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Ly-ROr9t-XXqS9M off=6314976 len=869639 2021-07-31T02:24:43.283 INFO:tasks.workunit.client.0.smithi045.stdout:op 5893 completed, throughput=5MB/sec 2021-07-31T02:24:43.283 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ms_QlPFVWKr8nXk off=15617847 len=607429 2021-07-31T02:24:43.284 INFO:tasks.workunit.client.0.smithi045.stdout:op 5894 completed, throughput=5MB/sec 2021-07-31T02:24:44.029 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:24:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:24:43.574Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:24:44.034 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:24:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:24:43.574Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:24:44.034 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:24:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:24:43.574Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:24:44.285 INFO:tasks.workunit.client.0.smithi045.stdout: 1154: throughput=5MB/sec pending data=0 2021-07-31T02:24:44.285 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-AcVvC9jZ6E_8dUK off=12705202 len=1057603 2021-07-31T02:24:44.287 INFO:tasks.workunit.client.0.smithi045.stdout:op 5895 completed, throughput=5MB/sec 2021-07-31T02:24:44.287 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-UnbYIZ3_2zPflcr off=13011012 len=460654 2021-07-31T02:24:44.291 INFO:tasks.workunit.client.0.smithi045.stdout:op 5896 completed, throughput=5MB/sec 2021-07-31T02:24:44.292 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ZqQxC9NJFKN2WsH off=4341837 len=554610 2021-07-31T02:24:44.296 INFO:tasks.workunit.client.0.smithi045.stdout:op 5897 completed, throughput=5MB/sec 2021-07-31T02:24:44.296 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-OQhr9tCuYhyuDV_ off=1885048 len=1154251 2021-07-31T02:24:44.305 INFO:tasks.workunit.client.0.smithi045.stdout:op 5898 completed, throughput=5MB/sec 2021-07-31T02:24:44.306 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-4H62YZXJeTdqhjy off=9922674 len=1925953 2021-07-31T02:24:44.314 INFO:tasks.workunit.client.0.smithi045.stdout:op 5899 completed, throughput=5MB/sec 2021-07-31T02:24:44.314 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-UGhQ7EuGW6uhrQ3 off=2636669 len=165311 2021-07-31T02:24:44.316 INFO:tasks.workunit.client.0.smithi045.stdout:op 5900 completed, throughput=5MB/sec 2021-07-31T02:24:44.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:44 smithi154 conmon[34553]: cluster 2021-07-31T02:24:42.330246+0000 mgr.x (mgr.24542) 1609 : cluster [DBG] pgmap v1996: 9 pgs: 9 active+clean; 127 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.8 MiB/s wr, 4 op/s 2021-07-31T02:24:44.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:44 smithi154 conmon[34553]: 2021-07-31T02:24:44.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:44 smithi045 conmon[39718]: cluster 2021-07-31T02:24:42.330246+0000 mgr.x 2021-07-31T02:24:44.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:44 smithi045 conmon[39718]: (mgr.24542) 1609 : cluster [DBG] pgmap v1996: 9 pgs: 9 active+clean; 127 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.8 MiB/s wr, 4 op/s 2021-07-31T02:24:44.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:44 smithi045 conmon[33655]: cluster 2021-07-31T02:24:42.330246+0000 mgr.x (mgr.24542) 1609 : cluster 2021-07-31T02:24:44.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:44 smithi045 conmon[33655]: [DBG] pgmap v1996: 9 pgs: 9 active+clean; 127 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.8 MiB/s wr, 4 op/s 2021-07-31T02:24:45.316 INFO:tasks.workunit.client.0.smithi045.stdout: 1155: throughput=5MB/sec pending data=0 2021-07-31T02:24:45.317 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-q-PTEHpWKGjuTi4 off=6990587 len=1014784 2021-07-31T02:24:45.322 INFO:tasks.workunit.client.0.smithi045.stdout:op 5901 completed, throughput=5MB/sec 2021-07-31T02:24:45.322 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-YTMJfvET_4EjaMR off=20276246 len=1795581 2021-07-31T02:24:45.333 INFO:tasks.workunit.client.0.smithi045.stdout:op 5902 completed, throughput=5MB/sec 2021-07-31T02:24:45.333 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-v1EIG1BQo4pFHhF off=5060524 len=1509830 2021-07-31T02:24:45.341 INFO:tasks.workunit.client.0.smithi045.stdout:op 5903 completed, throughput=5MB/sec 2021-07-31T02:24:45.341 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-DLxkRaYK64-H8mh off=6289977 len=560960 2021-07-31T02:24:45.345 INFO:tasks.workunit.client.0.smithi045.stdout:op 5904 completed, throughput=5MB/sec 2021-07-31T02:24:45.345 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ZKd3tdyyCM8qsJE off=595699 len=84636 2021-07-31T02:24:45.347 INFO:tasks.workunit.client.0.smithi045.stdout:op 5905 completed, throughput=5MB/sec 2021-07-31T02:24:45.348 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-PCQWa7EPKJvkXK8 off=12785642 len=1559333 2021-07-31T02:24:45.353 INFO:tasks.workunit.client.0.smithi045.stdout:op 5906 completed, throughput=5MB/sec 2021-07-31T02:24:46.354 INFO:tasks.workunit.client.0.smithi045.stdout: 1156: throughput=5MB/sec pending data=0 2021-07-31T02:24:46.354 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-HS0LIRJJIEiIVAX off=1754522 len=884607 2021-07-31T02:24:46.356 INFO:tasks.workunit.client.0.smithi045.stdout:op 5907 completed, throughput=5MB/sec 2021-07-31T02:24:46.356 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-eTpsQFl7jmwEjHO off=597831 len=532105 2021-07-31T02:24:46.357 INFO:tasks.workunit.client.0.smithi045.stdout:op 5908 completed, throughput=5MB/sec 2021-07-31T02:24:46.358 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-i2oiVef6H4enCIK off=13448662 len=461080 2021-07-31T02:24:46.359 INFO:tasks.workunit.client.0.smithi045.stdout:op 5909 completed, throughput=5MB/sec 2021-07-31T02:24:46.359 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-6jPQs-8Dhq0pXrt off=5378793 len=1277863 2021-07-31T02:24:46.361 INFO:tasks.workunit.client.0.smithi045.stdout:op 5910 completed, throughput=5MB/sec 2021-07-31T02:24:46.361 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-w7YlkJGJrn0CSLF off=20002627 len=89482 2021-07-31T02:24:46.361 INFO:tasks.workunit.client.0.smithi045.stdout:op 5911 completed, throughput=5MB/sec 2021-07-31T02:24:46.362 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-3sYZgF2DYf0ljfc off=209363 len=1497002 2021-07-31T02:24:46.369 INFO:tasks.workunit.client.0.smithi045.stdout:op 5912 completed, throughput=5MB/sec 2021-07-31T02:24:46.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:46 smithi045 conmon[39718]: cluster 2021-07-31T02:24:44.330796+0000 mgr.x 2021-07-31T02:24:46.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:46 smithi045 conmon[39718]: (mgr.24542) 1610 : cluster [DBG] pgmap v1997: 9 pgs: 9 active+clean; 127 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 3.6 MiB/s rd, 2.8 MiB/s wr, 7 op/s 2021-07-31T02:24:46.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:46 smithi045 conmon[33655]: cluster 2021-07-31T02:24:44.330796 2021-07-31T02:24:46.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:46 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 1610 : cluster [DBG] pgmap v1997: 9 pgs: 9 active+clean; 127 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 3.6 MiB/s rd, 2.8 MiB/s wr, 7 op/s 2021-07-31T02:24:46.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:46 smithi154 conmon[34553]: cluster 2021-07-31T02:24:44.330796+0000 mgr.x (mgr.24542) 1610 : cluster [DBG] pgmap v1997: 9 pgs: 9 active+clean; 127 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 3.6 MiB/s rd, 2.8 MiB/s wr, 7 op/s 2021-07-31T02:24:47.369 INFO:tasks.workunit.client.0.smithi045.stdout: 1157: throughput=5MB/sec pending data=0 2021-07-31T02:24:47.369 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-kdy58Lv1iKqQAW4 off=17353339 len=44938 2021-07-31T02:24:47.370 INFO:tasks.workunit.client.0.smithi045.stdout:op 5913 completed, throughput=5MB/sec 2021-07-31T02:24:47.370 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ahz3miqaOVyKUvE off=6242577 len=642341 2021-07-31T02:24:47.372 INFO:tasks.workunit.client.0.smithi045.stdout:op 5914 completed, throughput=5MB/sec 2021-07-31T02:24:47.372 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-5Vi8lAXys3ZJOCU off=2495477 len=954045 2021-07-31T02:24:47.373 INFO:tasks.workunit.client.0.smithi045.stdout:op 5915 completed, throughput=5MB/sec 2021-07-31T02:24:47.373 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-tI2yYNPFS0n6g22 off=11842562 len=1867188 2021-07-31T02:24:47.382 INFO:tasks.workunit.client.0.smithi045.stdout:op 5916 completed, throughput=5MB/sec 2021-07-31T02:24:47.383 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-UaJbYXdP3a7aZs- off=3724430 len=709914 2021-07-31T02:24:47.383 INFO:tasks.workunit.client.0.smithi045.stdout:op 5917 completed, throughput=5MB/sec 2021-07-31T02:24:47.384 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-R6NrF8m_sfhMwx9 off=2469221 len=2021833 2021-07-31T02:24:47.385 INFO:tasks.workunit.client.0.smithi045.stdout:op 5918 completed, throughput=5MB/sec 2021-07-31T02:24:48.386 INFO:tasks.workunit.client.0.smithi045.stdout: 1158: throughput=5MB/sec pending data=0 2021-07-31T02:24:48.386 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-oq263vd337S5rIc off=1212516 len=1367747 2021-07-31T02:24:48.393 INFO:tasks.workunit.client.0.smithi045.stdout:op 5919 completed, throughput=5MB/sec 2021-07-31T02:24:48.393 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-9G8dsXZtrNE13p3 off=10838477 len=1985466 2021-07-31T02:24:48.402 INFO:tasks.workunit.client.0.smithi045.stdout:op 5920 completed, throughput=5MB/sec 2021-07-31T02:24:48.402 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-LXFv23R-j63_fQz off=7243373 len=137079 2021-07-31T02:24:48.404 INFO:tasks.workunit.client.0.smithi045.stdout:op 5921 completed, throughput=5MB/sec 2021-07-31T02:24:48.405 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-cRmfIvn6FsHQwgH off=3623830 len=1889410 2021-07-31T02:24:48.408 INFO:tasks.workunit.client.0.smithi045.stdout:op 5922 completed, throughput=5MB/sec 2021-07-31T02:24:48.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:48 smithi045 conmon[33655]: cluster 2021-07-31T02:24:46.331254+0000 mgr.x (mgr.24542) 1611 : 2021-07-31T02:24:48.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:48 smithi045 conmon[33655]: cluster [DBG] pgmap v1998: 9 pgs: 9 active+clean; 127 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.7 MiB/s wr, 5 op/s 2021-07-31T02:24:48.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:48 smithi045 conmon[39718]: cluster 2021-07-31T02:24:46.331254+0000 mgr.x (mgr 2021-07-31T02:24:48.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:48 smithi045 conmon[39718]: .24542) 1611 : cluster [DBG] pgmap v1998: 9 pgs: 9 active+clean; 127 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.7 MiB/s wr, 5 op/s 2021-07-31T02:24:48.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:48 smithi154 conmon[34553]: cluster 2021-07-31T02:24:46.331254+0000 mgr.x (mgr.24542) 1611 : cluster [DBG] pgmap v1998: 9 pgs: 9 active+clean; 127 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.7 MiB/s wr, 5 op/s 2021-07-31T02:24:49.408 INFO:tasks.workunit.client.0.smithi045.stdout: 1159: throughput=5MB/sec pending data=0 2021-07-31T02:24:49.409 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-lbTEsYCvydJWMtK off=8120881 len=508548 2021-07-31T02:24:49.412 INFO:tasks.workunit.client.0.smithi045.stdout:op 5923 completed, throughput=5MB/sec 2021-07-31T02:24:49.413 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-_k-sQdvSpPdnIsj off=21516874 len=946838 2021-07-31T02:24:49.418 INFO:tasks.workunit.client.0.smithi045.stdout:op 5924 completed, throughput=5MB/sec 2021-07-31T02:24:49.418 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-AqBoRuqDg47mhPl off=8905540 len=1087284 2021-07-31T02:24:49.426 INFO:tasks.workunit.client.0.smithi045.stdout:op 5925 completed, throughput=5MB/sec 2021-07-31T02:24:49.426 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-7gZA5dZxhKAaVbj off=2406001 len=214284 2021-07-31T02:24:49.429 INFO:tasks.workunit.client.0.smithi045.stdout:op 5926 completed, throughput=5MB/sec 2021-07-31T02:24:49.429 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-bv9HfNl7tTDueb7 off=1909023 len=554431 2021-07-31T02:24:49.430 INFO:tasks.workunit.client.0.smithi045.stdout:op 5927 completed, throughput=5MB/sec 2021-07-31T02:24:49.431 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-wIvS6U-rzBJDVCU off=7883129 len=1988079 2021-07-31T02:24:49.438 INFO:tasks.workunit.client.0.smithi045.stdout:op 5928 completed, throughput=5MB/sec 2021-07-31T02:24:50.439 INFO:tasks.workunit.client.0.smithi045.stdout: 1160: throughput=5MB/sec pending data=0 2021-07-31T02:24:50.439 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-eTpsQFl7jmwEjHO off=1390803 len=232169 2021-07-31T02:24:50.440 INFO:tasks.workunit.client.0.smithi045.stdout:op 5929 completed, throughput=5MB/sec 2021-07-31T02:24:50.440 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-ZDoCFW_0WYtvNFC off=495962 len=1467146 2021-07-31T02:24:50.447 INFO:tasks.workunit.client.0.smithi045.stdout:op 5930 completed, throughput=5MB/sec 2021-07-31T02:24:50.448 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-RQGVf8O16UeUcIs off=913488 len=1859074 2021-07-31T02:24:50.456 INFO:tasks.workunit.client.0.smithi045.stdout:op 5931 completed, throughput=5MB/sec 2021-07-31T02:24:50.456 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-5ih8l5a7eAuAYbb off=13482688 len=1923303 2021-07-31T02:24:50.459 INFO:tasks.workunit.client.0.smithi045.stdout:op 5932 completed, throughput=5MB/sec 2021-07-31T02:24:50.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:50 smithi045 conmon[39718]: cluster 2021-07-31T02:24:48.331697+0000 mgr.x (mgr.24542 2021-07-31T02:24:50.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:50 smithi045 conmon[39718]: ) 1612 : cluster [DBG] pgmap v1999: 9 pgs: 9 active+clean; 127 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.7 MiB/s wr, 5 op/s 2021-07-31T02:24:50.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:50 smithi045 conmon[33655]: cluster 2021-07-31T02:24:48.331697+0000 mgr.x (mgr.24542) 1612 : cluster [DBG] 2021-07-31T02:24:50.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:50 smithi045 conmon[33655]: pgmap v1999: 9 pgs: 9 active+clean; 127 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.7 MiB/s wr, 5 op/s 2021-07-31T02:24:50.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:50 smithi154 conmon[34553]: cluster 2021-07-31T02:24:48.331697+0000 mgr.x (mgr.24542) 1612 : cluster [DBG] pgmap v1999: 9 pgs: 9 active+clean; 127 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.7 MiB/s wr, 5 op/s 2021-07-31T02:24:51.460 INFO:tasks.workunit.client.0.smithi045.stdout: 1161: throughput=5MB/sec pending data=0 2021-07-31T02:24:51.460 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-8GeRf0zA1L7VZNw off=338876 len=1887925 2021-07-31T02:24:51.462 INFO:tasks.workunit.client.0.smithi045.stdout:op 5933 completed, throughput=5MB/sec 2021-07-31T02:24:51.462 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-nNeold9ZH8ZMuWl off=19918512 len=667199 2021-07-31T02:24:51.467 INFO:tasks.workunit.client.0.smithi045.stdout:op 5934 completed, throughput=5MB/sec 2021-07-31T02:24:51.467 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-eeu5DGEz2PYV_T3 off=20952145 len=1540643 2021-07-31T02:24:51.468 INFO:tasks.workunit.client.0.smithi045.stdout:op 5935 completed, throughput=5MB/sec 2021-07-31T02:24:52.469 INFO:tasks.workunit.client.0.smithi045.stdout: 1162: throughput=5MB/sec pending data=0 2021-07-31T02:24:52.469 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-znimR4ve8loSFgv off=217759 len=2031092 2021-07-31T02:24:52.471 INFO:tasks.workunit.client.0.smithi045.stdout:op 5936 completed, throughput=5MB/sec 2021-07-31T02:24:52.472 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-OqX0SAZfL1SQ_R8 off=3003554 len=483809 2021-07-31T02:24:52.473 INFO:tasks.workunit.client.0.smithi045.stdout:op 5937 completed, throughput=5MB/sec 2021-07-31T02:24:52.473 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-GfoJF7inNG8pIv- off=8261139 len=1319231 2021-07-31T02:24:52.479 INFO:tasks.workunit.client.0.smithi045.stdout:op 5938 completed, throughput=5MB/sec 2021-07-31T02:24:52.480 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-n8NLoV5h6kQeKp2 off=7706252 len=996188 2021-07-31T02:24:52.485 INFO:tasks.workunit.client.0.smithi045.stdout:op 5939 completed, throughput=5MB/sec 2021-07-31T02:24:52.485 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Fa_6DUfSax93QRR off=419933 len=596869 2021-07-31T02:24:52.487 INFO:tasks.workunit.client.0.smithi045.stdout:op 5940 completed, throughput=5MB/sec 2021-07-31T02:24:52.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:52 smithi045 conmon[33655]: cluster 2021-07-31T02:24:50.332265 2021-07-31T02:24:52.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:52 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 1613 : cluster [DBG] pgmap v2000: 9 pgs: 9 active+clean; 127 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 3.4 MiB/s rd, 3.1 MiB/s wr, 7 op/s 2021-07-31T02:24:52.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:52 smithi045 conmon[39718]: cluster 2021-07-31T02:24:50.332265+0000 mgr.x 2021-07-31T02:24:52.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:52 smithi045 conmon[39718]: (mgr.24542) 1613 : cluster [DBG] pgmap v2000: 9 pgs: 9 active+clean; 127 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 3.4 MiB/s rd, 3.1 MiB/s wr, 7 op/s 2021-07-31T02:24:52.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:52 smithi154 conmon[34553]: cluster 2021-07-31T02:24:50.332265+0000 mgr.x (mgr.24542) 1613 : cluster [DBG] pgmap v2000: 9 pgs: 9 active+clean; 127 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 3.4 MiB/s rd, 3.1 MiB/s wr, 7 op/s 2021-07-31T02:24:53.488 INFO:tasks.workunit.client.0.smithi045.stdout: 1163: throughput=5MB/sec pending data=0 2021-07-31T02:24:53.488 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-m25DnnxSQ356-nS off=4442723 len=871146 2021-07-31T02:24:53.489 INFO:tasks.workunit.client.0.smithi045.stdout:op 5941 completed, throughput=5MB/sec 2021-07-31T02:24:53.490 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-f5iXSqNe2Q3VLnZ off=4480537 len=268598 2021-07-31T02:24:53.493 INFO:tasks.workunit.client.0.smithi045.stdout:op 5942 completed, throughput=5MB/sec 2021-07-31T02:24:53.493 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-AcoOSdqzMDAR8Qt off=4041832 len=142320 2021-07-31T02:24:53.496 INFO:tasks.workunit.client.0.smithi045.stdout:op 5943 completed, throughput=5MB/sec 2021-07-31T02:24:53.496 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj--HRh8NVuTxEXJWm off=389974 len=1785625 2021-07-31T02:24:53.499 INFO:tasks.workunit.client.0.smithi045.stdout:op 5944 completed, throughput=5MB/sec 2021-07-31T02:24:53.499 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-I8Cbovf1yclX0Dg off=15209496 len=1627624 2021-07-31T02:24:53.501 INFO:tasks.workunit.client.0.smithi045.stdout:op 5945 completed, throughput=5MB/sec 2021-07-31T02:24:53.501 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-MSBYYJz2qMA_VCY off=14235192 len=1585580 2021-07-31T02:24:53.515 INFO:tasks.workunit.client.0.smithi045.stdout:op 5946 completed, throughput=5MB/sec 2021-07-31T02:24:54.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:24:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:24:53.574Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:24:54.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:24:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:24:53.574Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:24:54.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:24:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:24:53.574Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:24:54.516 INFO:tasks.workunit.client.0.smithi045.stdout: 1164: throughput=5MB/sec pending data=0 2021-07-31T02:24:54.517 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-wrUiJJ8s0sXE6wo off=1143347 len=591876 2021-07-31T02:24:54.523 INFO:tasks.workunit.client.0.smithi045.stdout:op 5947 completed, throughput=5MB/sec 2021-07-31T02:24:54.523 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-NtTevqeTEJ5P_4Z off=3480860 len=1317719 2021-07-31T02:24:54.529 INFO:tasks.workunit.client.0.smithi045.stdout:op 5948 completed, throughput=5MB/sec 2021-07-31T02:24:54.532 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-efNTIIZF86uzM0g off=395327 len=240230 2021-07-31T02:24:54.533 INFO:tasks.workunit.client.0.smithi045.stdout:op 5949 completed, throughput=5MB/sec 2021-07-31T02:24:54.534 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-nWZ6F_F2f0DTtpY off=16237738 len=636249 2021-07-31T02:24:54.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:54 smithi045 conmon[39718]: cluster 2021-07-31T02:24:52.332687+0000 mgr.x ( 2021-07-31T02:24:54.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:54 smithi045 conmon[39718]: mgr.24542) 1614 : cluster [DBG] pgmap v2001: 9 pgs: 9 active+clean; 127 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:24:54.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:54 smithi045 conmon[33655]: cluster 2021-07-31T02:24:52.332687+0000 mgr.x (mgr.24542 2021-07-31T02:24:54.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:54 smithi045 conmon[33655]: ) 1614 : cluster [DBG] pgmap v2001: 9 pgs: 9 active+clean; 127 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:24:54.542 INFO:tasks.workunit.client.0.smithi045.stdout:op 5950 completed, throughput=5MB/sec 2021-07-31T02:24:54.545 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-nkcKndNkofBbPzN off=2770534 len=708589 2021-07-31T02:24:54.547 INFO:tasks.workunit.client.0.smithi045.stdout:op 5951 completed, throughput=5MB/sec 2021-07-31T02:24:54.547 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-r7HIQGfJtiXTpw- off=18847130 len=1478743 2021-07-31T02:24:54.550 INFO:tasks.workunit.client.0.smithi045.stdout:op 5952 completed, throughput=5MB/sec 2021-07-31T02:24:54.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:54 smithi154 conmon[34553]: cluster 2021-07-31T02:24:52.332687+0000 mgr.x (mgr.24542) 1614 : cluster [DBG] pgmap v2001: 9 pgs: 9 active+clean; 127 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:24:55.537 INFO:tasks.workunit.client.0.smithi045.stdout: 1165: throughput=5MB/sec pending data=0 2021-07-31T02:24:55.537 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-QGmH9aOepv9JQmm off=979348 len=732738 2021-07-31T02:24:55.539 INFO:tasks.workunit.client.0.smithi045.stdout:op 5953 completed, throughput=5MB/sec 2021-07-31T02:24:55.539 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-JBdyv84UCFd_gnI off=6144022 len=310911 2021-07-31T02:24:55.542 INFO:tasks.workunit.client.0.smithi045.stdout:op 5954 completed, throughput=5MB/sec 2021-07-31T02:24:55.542 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj--ns2tgCIC12S-pe off=10174381 len=1052593 2021-07-31T02:24:55.545 INFO:tasks.workunit.client.0.smithi045.stdout:op 5955 completed, throughput=5MB/sec 2021-07-31T02:24:55.546 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-z6V_Li1d0Zfvh-e off=20211432 len=960058 2021-07-31T02:24:55.547 INFO:tasks.workunit.client.0.smithi045.stdout:op 5956 completed, throughput=5MB/sec 2021-07-31T02:24:55.547 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-8MlmKcF_zUZsQYf off=4565924 len=724145 2021-07-31T02:24:55.553 INFO:tasks.workunit.client.0.smithi045.stdout:op 5957 completed, throughput=5MB/sec 2021-07-31T02:24:55.553 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-HldAaR8Cyhdw_np off=4387938 len=1289014 2021-07-31T02:24:55.555 INFO:tasks.workunit.client.0.smithi045.stdout:op 5958 completed, throughput=5MB/sec 2021-07-31T02:24:56.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:56 smithi045 conmon[39718]: cluster 2021-07-31T02:24:54.333249+0000 mgr.x (mgr. 2021-07-31T02:24:56.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:56 smithi045 conmon[39718]: 24542) 1615 : cluster [DBG] pgmap v2002: 9 pgs: 9 active+clean; 127 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 3.0 MiB/s rd, 3.4 MiB/s wr, 6 op/s 2021-07-31T02:24:56.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:56 smithi045 conmon[33655]: cluster 2021-07-31T02:24:54.333249+0000 mgr.x (mgr.24542) 1615 : cluster 2021-07-31T02:24:56.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:56 smithi045 conmon[33655]: [DBG] pgmap v2002: 9 pgs: 9 active+clean; 127 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 3.0 MiB/s rd, 3.4 MiB/s wr, 6 op/s 2021-07-31T02:24:56.555 INFO:tasks.workunit.client.0.smithi045.stdout: 1166: throughput=5MB/sec pending data=0 2021-07-31T02:24:56.555 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-DMl4w6S8AbTAOG0 off=4852695 len=2055066 2021-07-31T02:24:56.560 INFO:tasks.workunit.client.0.smithi045.stdout:op 5959 completed, throughput=5MB/sec 2021-07-31T02:24:56.560 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-YrMobObki__JHoT off=16849834 len=843027 2021-07-31T02:24:56.565 INFO:tasks.workunit.client.0.smithi045.stdout:op 5960 completed, throughput=5MB/sec 2021-07-31T02:24:56.565 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-nBpAXCqJ5VP7Kum off=8471001 len=431446 2021-07-31T02:24:56.569 INFO:tasks.workunit.client.0.smithi045.stdout:op 5961 completed, throughput=5MB/sec 2021-07-31T02:24:56.569 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Yrvj6B-mCGSz-3F off=350284 len=2000857 2021-07-31T02:24:56.577 INFO:tasks.workunit.client.0.smithi045.stdout:op 5962 completed, throughput=5MB/sec 2021-07-31T02:24:56.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:56 smithi154 conmon[34553]: cluster 2021-07-31T02:24:54.333249+0000 mgr.x (mgr.24542) 1615 : cluster [DBG] pgmap v2002: 9 pgs: 9 active+clean; 127 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 3.0 MiB/s rd, 3.4 MiB/s wr, 6 op/s 2021-07-31T02:24:57.577 INFO:tasks.workunit.client.0.smithi045.stdout: 1167: throughput=5MB/sec pending data=0 2021-07-31T02:24:57.577 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-HhQLjs-lfNP9XyG off=2516373 len=569154 2021-07-31T02:24:57.579 INFO:tasks.workunit.client.0.smithi045.stdout:op 5963 completed, throughput=5MB/sec 2021-07-31T02:24:57.579 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-3uc89e4YYavELmX off=8434689 len=777107 2021-07-31T02:24:57.581 INFO:tasks.workunit.client.0.smithi045.stdout:op 5964 completed, throughput=5MB/sec 2021-07-31T02:24:57.581 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-9JESFRVqWcxlDM9 off=13095280 len=1627973 2021-07-31T02:24:57.585 INFO:tasks.workunit.client.0.smithi045.stdout:op 5965 completed, throughput=5MB/sec 2021-07-31T02:24:57.585 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-sQVP-1RcJwGs9mE off=4797259 len=766509 2021-07-31T02:24:57.586 INFO:tasks.workunit.client.0.smithi045.stdout:op 5966 completed, throughput=5MB/sec 2021-07-31T02:24:57.586 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-XQ6eUa_4DTDH6r7 off=4335828 len=6910 2021-07-31T02:24:57.587 INFO:tasks.workunit.client.0.smithi045.stdout:op 5967 completed, throughput=5MB/sec 2021-07-31T02:24:57.587 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-fPLQESO6ceYoRuf off=15874100 len=287121 2021-07-31T02:24:57.590 INFO:tasks.workunit.client.0.smithi045.stdout:op 5968 completed, throughput=5MB/sec 2021-07-31T02:24:57.590 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-QgxWNVpRStzoDVv off=1518521 len=111211 2021-07-31T02:24:57.593 INFO:tasks.workunit.client.0.smithi045.stdout:op 5969 completed, throughput=5MB/sec 2021-07-31T02:24:57.593 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-6WyNSE-onQJ_oOi off=7119372 len=353448 2021-07-31T02:24:57.594 INFO:tasks.workunit.client.0.smithi045.stdout:op 5970 completed, throughput=5MB/sec 2021-07-31T02:24:57.594 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-1DyLwVWhxe6AbBU off=15625999 len=332022 2021-07-31T02:24:57.594 INFO:tasks.workunit.client.0.smithi045.stdout:op 5971 completed, throughput=5MB/sec 2021-07-31T02:24:58.595 INFO:tasks.workunit.client.0.smithi045.stdout: 1168: throughput=5MB/sec pending data=0 2021-07-31T02:24:58.596 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-MMJ2lLOdHRo1sD9 off=2674317 len=541458 2021-07-31T02:24:58.599 INFO:tasks.workunit.client.0.smithi045.stdout:op 5972 completed, throughput=5MB/sec 2021-07-31T02:24:58.599 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-ph41k_bufIA3End off=2116069 len=489417 2021-07-31T02:24:58.600 INFO:tasks.workunit.client.0.smithi045.stdout:op 5973 completed, throughput=5MB/sec 2021-07-31T02:24:58.600 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-XTgvvlVNRHSVhyU off=5734693 len=1874263 2021-07-31T02:24:58.609 INFO:tasks.workunit.client.0.smithi045.stdout:op 5974 completed, throughput=5MB/sec 2021-07-31T02:24:58.609 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-vXnDWj5fyCC5rsG off=7830802 len=1551144 2021-07-31T02:24:58.619 INFO:tasks.workunit.client.0.smithi045.stdout:op 5975 completed, throughput=5MB/sec 2021-07-31T02:24:58.619 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-WeVfnQzUgE-p7DM off=7583000 len=9107 2021-07-31T02:24:58.619 INFO:tasks.workunit.client.0.smithi045.stdout:op 5976 completed, throughput=5MB/sec 2021-07-31T02:24:58.620 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-6Iyt6Nj3HdnXp0p off=703046 len=2037915 2021-07-31T02:24:58.623 INFO:tasks.workunit.client.0.smithi045.stdout:op 5977 completed, throughput=5MB/sec 2021-07-31T02:24:58.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:24:58 smithi154 conmon[34553]: cluster 2021-07-31T02:24:56.333776+0000 mgr.x (mgr.24542) 1616 : cluster [DBG] pgmap v2003: 9 pgs: 9 active+clean; 127 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:24:58.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:58 smithi045 conmon[39718]: cluster 2021-07-31T02:24:56.333776+0000 mgr.x 2021-07-31T02:24:58.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:24:58 smithi045 conmon[39718]: (mgr.24542) 1616 : cluster [DBG] pgmap v2003: 9 pgs: 9 active+clean; 127 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:24:58.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:58 smithi045 conmon[33655]: cluster 2021-07-31T02:24:56.333776+0000 mgr.x (mgr.24542) 1616 : cluster 2021-07-31T02:24:58.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:24:58 smithi045 conmon[33655]: [DBG] pgmap v2003: 9 pgs: 9 active+clean; 127 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:24:59.623 INFO:tasks.workunit.client.0.smithi045.stdout: 1169: throughput=5MB/sec pending data=0 2021-07-31T02:24:59.624 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-jrrX3WLV8QPYkDr off=1251963 len=272104 2021-07-31T02:24:59.627 INFO:tasks.workunit.client.0.smithi045.stdout:op 5978 completed, throughput=5MB/sec 2021-07-31T02:24:59.627 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-pPyR5gP8Zg32r-J off=1435893 len=703918 2021-07-31T02:24:59.631 INFO:tasks.workunit.client.0.smithi045.stdout:op 5979 completed, throughput=5MB/sec 2021-07-31T02:24:59.632 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-C9PD4_BxG7F4OOc off=1237181 len=329982 2021-07-31T02:24:59.635 INFO:tasks.workunit.client.0.smithi045.stdout:op 5980 completed, throughput=5MB/sec 2021-07-31T02:24:59.635 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-r5UA9wkL9NJDam7 off=8266501 len=657819 2021-07-31T02:24:59.639 INFO:tasks.workunit.client.0.smithi045.stdout:op 5981 completed, throughput=5MB/sec 2021-07-31T02:24:59.640 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-YrMobObki__JHoT off=2314026 len=2094823 2021-07-31T02:24:59.642 INFO:tasks.workunit.client.0.smithi045.stdout:op 5982 completed, throughput=5MB/sec 2021-07-31T02:24:59.642 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-GZRcyGO-RyUil0x off=14641389 len=61166 2021-07-31T02:24:59.646 INFO:tasks.workunit.client.0.smithi045.stdout:op 5983 completed, throughput=5MB/sec 2021-07-31T02:24:59.646 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-P4cXf9PuHFUWOWz off=4054219 len=1491552 2021-07-31T02:24:59.652 INFO:tasks.workunit.client.0.smithi045.stdout:op 5984 completed, throughput=5MB/sec 2021-07-31T02:25:00.653 INFO:tasks.workunit.client.0.smithi045.stdout: 1170: throughput=5MB/sec pending data=0 2021-07-31T02:25:00.653 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-8gbk6s4PF9dvGRR off=2165568 len=1004052 2021-07-31T02:25:00.659 INFO:tasks.workunit.client.0.smithi045.stdout:op 5985 completed, throughput=5MB/sec 2021-07-31T02:25:00.659 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-55cr09C3wBCrWbg off=7116283 len=412476 2021-07-31T02:25:00.660 INFO:tasks.workunit.client.0.smithi045.stdout:op 5986 completed, throughput=5MB/sec 2021-07-31T02:25:00.661 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-CN_jfqXXLYZITft off=4278722 len=992381 2021-07-31T02:25:00.666 INFO:tasks.workunit.client.0.smithi045.stdout:op 5987 completed, throughput=5MB/sec 2021-07-31T02:25:00.666 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-CH1J3vv1SA057rV off=15595946 len=615145 2021-07-31T02:25:00.671 INFO:tasks.workunit.client.0.smithi045.stdout:op 5988 completed, throughput=5MB/sec 2021-07-31T02:25:00.671 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-m36VxIUkNpQRRkd off=1947023 len=739367 2021-07-31T02:25:00.674 INFO:tasks.workunit.client.0.smithi045.stdout:op 5989 completed, throughput=5MB/sec 2021-07-31T02:25:00.675 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-uGcecmdCSk70d2b off=980049 len=384602 2021-07-31T02:25:00.678 INFO:tasks.workunit.client.0.smithi045.stdout:op 5990 completed, throughput=5MB/sec 2021-07-31T02:25:00.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:00 smithi154 conmon[34553]: cluster 2021-07-31T02:24:58.334169+0000 mgr.x (mgr.24542) 1617 : cluster [DBG] pgmap v2004: 9 pgs: 9 active+clean; 127 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:25:00.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:00 smithi045 conmon[39718]: cluster 2021-07-31T02:24:58.334169+0000 mgr.x (mgr 2021-07-31T02:25:00.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:00 smithi045 conmon[39718]: .24542) 1617 : cluster [DBG] pgmap v2004: 9 pgs: 9 active+clean; 127 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:25:00.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:00 smithi045 conmon[33655]: cluster 2021-07-31T02:24:58.334169+0000 mgr.x (mgr.24542) 1617 : cluster 2021-07-31T02:25:00.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:00 smithi045 conmon[33655]: [DBG] pgmap v2004: 9 pgs: 9 active+clean; 127 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2021-07-31T02:25:01.678 INFO:tasks.workunit.client.0.smithi045.stdout: 1171: throughput=5MB/sec pending data=0 2021-07-31T02:25:01.678 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-lF5jsqAdNBXBU8w off=5778307 len=308280 2021-07-31T02:25:01.682 INFO:tasks.workunit.client.0.smithi045.stdout:op 5991 completed, throughput=5MB/sec 2021-07-31T02:25:01.682 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-QbVH4hW9ts-FmSc off=1373882 len=305812 2021-07-31T02:25:01.686 INFO:tasks.workunit.client.0.smithi045.stdout:op 5992 completed, throughput=5MB/sec 2021-07-31T02:25:01.686 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-rTGIhq1EfbiWTE0 off=4083828 len=11213 2021-07-31T02:25:01.688 INFO:tasks.workunit.client.0.smithi045.stdout:op 5993 completed, throughput=5MB/sec 2021-07-31T02:25:01.688 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-KVA9JeVtvf8yXF4 off=5863652 len=1634172 2021-07-31T02:25:01.697 INFO:tasks.workunit.client.0.smithi045.stdout:op 5994 completed, throughput=5MB/sec 2021-07-31T02:25:01.697 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-4EFkbjJMsoxiWzO off=3771989 len=21791 2021-07-31T02:25:01.699 INFO:tasks.workunit.client.0.smithi045.stdout:op 5995 completed, throughput=5MB/sec 2021-07-31T02:25:01.700 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-v9kEt_J6J-qRfZb off=495347 len=1545291 2021-07-31T02:25:01.700 INFO:tasks.workunit.client.0.smithi045.stdout:op 5996 completed, throughput=5MB/sec 2021-07-31T02:25:01.701 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-TsdUvZSQKhe94Ka off=9290035 len=1110295 2021-07-31T02:25:01.703 INFO:tasks.workunit.client.0.smithi045.stdout:op 5997 completed, throughput=5MB/sec 2021-07-31T02:25:01.703 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-HlQnJgrYxiiCTqj off=7483100 len=1080525 2021-07-31T02:25:01.704 INFO:tasks.workunit.client.0.smithi045.stdout:op 5998 completed, throughput=5MB/sec 2021-07-31T02:25:02.705 INFO:tasks.workunit.client.0.smithi045.stdout: 1172: throughput=5MB/sec pending data=0 2021-07-31T02:25:02.706 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-kurNzp0UWpObcdb off=2857513 len=1286179 2021-07-31T02:25:02.708 INFO:tasks.workunit.client.0.smithi045.stdout:op 5999 completed, throughput=5MB/sec 2021-07-31T02:25:02.708 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-76MXbNtLGU5HFgi off=2487744 len=692962 2021-07-31T02:25:02.713 INFO:tasks.workunit.client.0.smithi045.stdout:op 6000 completed, throughput=5MB/sec 2021-07-31T02:25:02.713 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-G-GR2ADlsbXwEVP off=6362591 len=1189771 2021-07-31T02:25:02.715 INFO:tasks.workunit.client.0.smithi045.stdout:op 6001 completed, throughput=5MB/sec 2021-07-31T02:25:02.715 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-9orRXpWwvFzAin4 off=14997762 len=839234 2021-07-31T02:25:02.721 INFO:tasks.workunit.client.0.smithi045.stdout:op 6002 completed, throughput=5MB/sec 2021-07-31T02:25:02.721 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-kDmRaeL5voMt4-E off=3631182 len=117001 2021-07-31T02:25:02.722 INFO:tasks.workunit.client.0.smithi045.stdout:op 6003 completed, throughput=5MB/sec 2021-07-31T02:25:02.722 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-4br2bhDiT1lP4m2 off=4301028 len=459292 2021-07-31T02:25:02.725 INFO:tasks.workunit.client.0.smithi045.stdout:op 6004 completed, throughput=5MB/sec 2021-07-31T02:25:02.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:02 smithi154 conmon[34553]: cluster 2021-07-31T02:25:00.334595+0000 mgr.x (mgr.24542) 1618 : cluster [DBG] pgmap v2005: 9 pgs: 9 active+clean; 127 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 3.1 MiB/s rd, 3.3 MiB/s wr, 6 op/s 2021-07-31T02:25:02.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:02 smithi045 conmon[39718]: cluster 2021-07-31T02:25:00.334595+0000 mgr.x (mgr 2021-07-31T02:25:02.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:02 smithi045 conmon[39718]: .24542) 1618 : cluster [DBG] pgmap v2005: 9 pgs: 9 active+clean; 127 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 3.1 MiB/s rd, 3.3 MiB/s wr, 6 op/s 2021-07-31T02:25:02.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:02 smithi045 conmon[33655]: cluster 2021-07-31T02:25:00.334595+0000 mgr.x (mgr.24542) 1618 : cluster 2021-07-31T02:25:02.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:02 smithi045 conmon[33655]: [DBG] pgmap v2005: 9 pgs: 9 active+clean; 127 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 3.1 MiB/s rd, 3.3 MiB/s wr, 6 op/s 2021-07-31T02:25:03.725 INFO:tasks.workunit.client.0.smithi045.stdout: 1173: throughput=5MB/sec pending data=0 2021-07-31T02:25:03.726 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-vyQ-kqHeSorTEcu off=9581199 len=1750482 2021-07-31T02:25:03.729 INFO:tasks.workunit.client.0.smithi045.stdout:op 6005 completed, throughput=5MB/sec 2021-07-31T02:25:03.730 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-jHFi_i6s9-UJAHm off=4568062 len=1480049 2021-07-31T02:25:03.736 INFO:tasks.workunit.client.0.smithi045.stdout:op 6006 completed, throughput=5MB/sec 2021-07-31T02:25:03.737 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-dG-p_HYIVCMBjdY off=10051381 len=827756 2021-07-31T02:25:03.742 INFO:tasks.workunit.client.0.smithi045.stdout:op 6007 completed, throughput=5MB/sec 2021-07-31T02:25:03.742 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Dy8gR_9t41SzcDP off=1766708 len=242539 2021-07-31T02:25:03.745 INFO:tasks.workunit.client.0.smithi045.stdout:op 6008 completed, throughput=5MB/sec 2021-07-31T02:25:03.745 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-u5LTPUk9Bhqy_aq off=1155270 len=534087 2021-07-31T02:25:03.746 INFO:tasks.workunit.client.0.smithi045.stdout:op 6009 completed, throughput=5MB/sec 2021-07-31T02:25:03.746 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-R2f-pwLVZLsmgzY off=3135525 len=1802179 2021-07-31T02:25:03.749 INFO:tasks.workunit.client.0.smithi045.stdout:op 6010 completed, throughput=5MB/sec 2021-07-31T02:25:04.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:25:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:25:03.574Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:25:04.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:25:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:25:03.575Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:25:04.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:25:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:25:03.575Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:25:04.300 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:25:04.578 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:25:04.579 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 39m ago 40m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:25:04.579 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 37m ago 40m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:25:04.579 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 37m ago 43m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:25:04.579 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 39m ago 46m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:25:04.580 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 39m ago 46m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:25:04.580 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 37m ago 44m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:25:04.580 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 39m ago 44m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:25:04.580 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 39m ago 40m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:25:04.580 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 37m ago 40m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:25:04.581 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 39m ago 43m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:25:04.581 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 39m ago 43m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:25:04.581 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 39m ago 43m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:25:04.581 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 39m ago 42m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:25:04.582 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 37m ago 42m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:25:04.582 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 37m ago 42m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:25:04.582 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 37m ago 41m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:25:04.583 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 37m ago 41m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:25:04.583 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 37m ago 41m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:25:04.749 INFO:tasks.workunit.client.0.smithi045.stdout: 1174: throughput=5MB/sec pending data=0 2021-07-31T02:25:04.750 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-qHhbde6P2gQQG3Z off=1209916 len=1228481 2021-07-31T02:25:04.750 INFO:tasks.workunit.client.0.smithi045.stdout:op 6011 completed, throughput=5MB/sec 2021-07-31T02:25:04.750 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-SYYlPtiDwcEMIe6 off=11486242 len=1092283 2021-07-31T02:25:04.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:04 smithi154 conmon[34553]: cluster 2021-07-31T02:25:02.335137+0000 mgr.x (mgr.24542) 1619 : cluster [DBG] pgmap v2006: 9 pgs: 9 active+clean; 127 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 2.3 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2021-07-31T02:25:04.755 INFO:tasks.workunit.client.0.smithi045.stdout:op 6012 completed, throughput=5MB/sec 2021-07-31T02:25:04.756 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-DWs_sFhGMcobnez off=199858 len=1864093 2021-07-31T02:25:04.765 INFO:tasks.workunit.client.0.smithi045.stdout:op 6013 completed, throughput=5MB/sec 2021-07-31T02:25:04.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:04 smithi045 conmon[39718]: cluster 2021-07-31T02:25:02.335137 2021-07-31T02:25:04.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:04 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 1619 : cluster [DBG] pgmap v2006: 9 pgs: 9 active+clean; 127 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 2.3 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2021-07-31T02:25:04.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:04 smithi045 conmon[33655]: cluster 2021-07-31T02:25:02.335137+0000 2021-07-31T02:25:04.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:04 smithi045 conmon[33655]: mgr.x (mgr.24542) 1619 : cluster [DBG] pgmap v2006: 9 pgs: 9 active+clean; 127 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 2.3 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2021-07-31T02:25:04.915 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:25:04.915 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:25:04.916 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:25:04.916 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:25:04.916 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:25:04.916 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:25:04.916 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:25:04.917 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:25:04.917 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:25:04.917 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:25:04.917 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:25:04.917 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:25:04.918 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:25:04.918 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:25:04.918 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:25:04.918 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:25:04.919 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:25:05.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:05 smithi154 conmon[34553]: audit 2021-07-31T02:25:04.286726+0000 mgr.x (mgr.24542) 1620 : audit [DBG] from='client.16419 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:25:05.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:05 smithi154 conmon[34553]: audit 2021-07-31T02:25:04.914965+0000 mon.c (mon.1) 95 : audit [DBG] from='client.? 172.21.15.45:0/577902277' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:25:05.765 INFO:tasks.workunit.client.0.smithi045.stdout: 1175: throughput=5MB/sec pending data=0 2021-07-31T02:25:05.765 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-jmSZ0IPiXkYahUG off=2252528 len=570411 2021-07-31T02:25:05.766 INFO:tasks.workunit.client.0.smithi045.stdout:op 6014 completed, throughput=5MB/sec 2021-07-31T02:25:05.767 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-M45Q7khzHekUWji off=2948768 len=563310 2021-07-31T02:25:05.771 INFO:tasks.workunit.client.0.smithi045.stdout:op 6015 completed, throughput=5MB/sec 2021-07-31T02:25:05.771 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Avb2NfGJP0V6skb off=1011415 len=192725 2021-07-31T02:25:05.772 INFO:tasks.workunit.client.0.smithi045.stdout:op 6016 completed, throughput=5MB/sec 2021-07-31T02:25:05.772 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-Zwn9QWvA0hNyJU- off=3296816 len=1423515 2021-07-31T02:25:05.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:05 smithi045 conmon[33655]: audit 2021-07-31T02:25:04.286726 2021-07-31T02:25:05.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:05 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 1620 : audit [DBG] from='client.16419 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:25:05.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:05 smithi045 conmon[33655]: audit 2021-07-31T02:25 2021-07-31T02:25:05.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:05 smithi045 conmon[33655]: :04.914965+0000 mon.c (mon.1) 95 : audit [DBG] from='client.? 172.21.15.45:0/577902277' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:25:05.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:05 smithi045 conmon[39718]: audit 2021-07-31T02:25:04.286726+0000 mgr.x ( 2021-07-31T02:25:05.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:05 smithi045 conmon[39718]: mgr.24542) 1620 : audit [DBG] from='client.16419 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:25:05.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:05 smithi045 conmon[39718]: audit 2021-07-31T02:25:04.914965+0000 mon.c ( 2021-07-31T02:25:05.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:05 smithi045 conmon[39718]: mon.1) 95 : audit [DBG] from='client.? 172.21.15.45:0/577902277' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:25:05.778 INFO:tasks.workunit.client.0.smithi045.stdout:op 6017 completed, throughput=5MB/sec 2021-07-31T02:25:05.778 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-M45Q7khzHekUWji off=6880200 len=436831 2021-07-31T02:25:05.778 INFO:tasks.workunit.client.0.smithi045.stdout:op 6018 completed, throughput=5MB/sec 2021-07-31T02:25:05.778 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-Wy1FnE2HnvL36Il off=437713 len=1232217 2021-07-31T02:25:05.785 INFO:tasks.workunit.client.0.smithi045.stdout:op 6019 completed, throughput=5MB/sec 2021-07-31T02:25:05.785 INFO:tasks.workunit.client.0.smithi045.stdout:WRITE : oid=obj-344JD2CQ8-PIKwN off=6702848 len=43372 2021-07-31T02:25:05.787 INFO:tasks.workunit.client.0.smithi045.stdout:op 6020 completed, throughput=5MB/sec 2021-07-31T02:25:05.787 INFO:tasks.workunit.client.0.smithi045.stdout:READ : oid=obj-N6100Xv0nKhw-xM off=5924340 len=1655366 2021-07-31T02:25:05.788 INFO:tasks.workunit.client.0.smithi045.stdout:op 6021 completed, throughput=5MB/sec 2021-07-31T02:25:06.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:06 smithi154 conmon[34553]: cluster 2021-07-31T02:25:04.335554+0000 mgr.x (mgr.24542) 1621 : cluster [DBG] pgmap v2007: 9 pgs: 9 active+clean; 127 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 3.3 MiB/s rd, 3.1 MiB/s wr, 7 op/s 2021-07-31T02:25:06.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:06 smithi154 conmon[34553]: audit 2021-07-31T02:25:04.570934+0000 mgr.x (mgr.24542) 1622 : audit [DBG] from='client.16425 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:25:06.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:06 smithi045 conmon[39718]: cluster 2021-07-31T02:25:04.335554+0000 mgr.x 2021-07-31T02:25:06.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:06 smithi045 conmon[39718]: (mgr.24542) 1621 : cluster [DBG] pgmap v2007: 9 pgs: 9 active+clean; 127 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 3.3 MiB/s rd, 3.1 MiB/s wr, 7 op/s 2021-07-31T02:25:06.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:06 smithi045 conmon[39718]: audit 2021-07 2021-07-31T02:25:06.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:06 smithi045 conmon[39718]: -31T02:25:04.570934+0000 mgr.x (mgr.24542) 1622 : audit [DBG] from='client.16425 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:25:06.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:06 smithi045 conmon[33655]: cluster 2021-07-31T02:25:04.335554+0000 2021-07-31T02:25:06.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:06 smithi045 conmon[33655]: mgr.x (mgr.24542) 1621 : cluster [DBG] pgmap v2007: 9 pgs: 9 active+clean; 127 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 3.3 MiB/s rd, 3.1 MiB/s wr, 7 op/s 2021-07-31T02:25:06.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:06 smithi045 conmon[33655]: audit 2021-07-31T 2021-07-31T02:25:06.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:06 smithi045 conmon[33655]: 02:25:04.570934 2021-07-31T02:25:06.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:06 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542 2021-07-31T02:25:06.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:06 smithi045 conmon[33655]: ) 1622 : audit [DBG] 2021-07-31T02:25:06.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:06 smithi045 conmon[33655]: from='client.16425 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:25:06.788 INFO:tasks.workunit.client.0.smithi045.stdout:waiting for all operations to complete 2021-07-31T02:25:06.789 INFO:tasks.workunit.client.0.smithi045.stdout:cleaning up objects 2021-07-31T02:25:08.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:08 smithi154 conmon[34553]: cluster 2021-07-31T02:25:06.336119+0000 mgr.x (mgr. 2021-07-31T02:25:08.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:08 smithi154 conmon[34553]: 24542) 1623 : cluster [DBG] pgmap v2008: 9 pgs: 9 active+clean; 127 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 5 op/s 2021-07-31T02:25:08.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:08 smithi045 conmon[39718]: cluster 2021-07-31T02:25:06.336119 2021-07-31T02:25:08.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:08 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 1623 : cluster [DBG] pgmap v2008: 9 pgs: 9 active+clean; 127 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 5 op/s 2021-07-31T02:25:08.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:08 smithi045 conmon[33655]: cluster 2021-07-31T02:25:06.336119+0000 2021-07-31T02:25:08.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:08 smithi045 conmon[33655]: mgr.x (mgr.24542) 1623 : cluster [DBG] pgmap v2008: 9 pgs: 9 active+clean; 127 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 5 op/s 2021-07-31T02:25:10.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:10 smithi154 conmon[34553]: cluster 2021-07-31T02:25:08.336486+0000 mgr.x (mgr.24542 2021-07-31T02:25:10.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:10 smithi154 conmon[34553]: ) 1624 : cluster [DBG] pgmap v2009: 9 pgs: 9 active+clean; 127 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 5 op/s 2021-07-31T02:25:10.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:10 smithi045 conmon[39718]: cluster 2021-07-31T02:25:10.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:10 smithi045 conmon[39718]: 2021-07-31T02 2021-07-31T02:25:10.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:10 smithi045 conmon[39718]: :25:08.336486+0000 mgr.x (mgr.24542) 1624 : cluster [DBG] pgmap v2009: 9 pgs: 9 active+clean; 127 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 5 op/s 2021-07-31T02:25:10.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:10 smithi045 conmon[33655]: cluster 2021-07-31T02:25:08.336486+0000 2021-07-31T02:25:10.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:10 smithi045 conmon[33655]: mgr.x (mgr.24542) 1624 : cluster [DBG] pgmap v2009: 9 pgs: 9 active+clean; 127 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 5 op/s 2021-07-31T02:25:12.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:12 smithi154 conmon[34553]: cluster 2021-07-31T02:25:10.336842+0000 mgr.x (mgr. 2021-07-31T02:25:12.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:12 smithi154 conmon[34553]: 24542) 1625 : cluster [DBG] pgmap v2010: 9 pgs: 9 active+clean; 110 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.4 MiB/s wr, 120 op/s 2021-07-31T02:25:12.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:12 smithi045 conmon[39718]: cluster 2021-07-31T02:25:10.336842+0000 mgr.x ( 2021-07-31T02:25:12.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:12 smithi045 conmon[39718]: mgr.24542) 1625 : cluster [DBG] pgmap v2010: 9 pgs: 9 active+clean; 110 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.4 MiB/s wr, 120 op/s 2021-07-31T02:25:12.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:12 smithi045 conmon[33655]: cluster 2021-07-31T02 2021-07-31T02:25:12.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:12 smithi045 conmon[33655]: :25:10.336842+0000 mgr.x (mgr.24542) 1625 : cluster [DBG] pgmap v2010: 9 pgs: 9 active+clean; 110 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.4 MiB/s wr, 120 op/s 2021-07-31T02:25:14.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:25:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:25:13.575Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:25:14.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:25:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:25:13.575Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:25:14.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:25:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:25:13.575Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:25:14.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:14 smithi154 conmon[34553]: cluster 2021-07-31T02:25:12.337144+0000 mgr.x ( 2021-07-31T02:25:14.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:14 smithi154 conmon[34553]: mgr.24542) 1626 : cluster [DBG] pgmap v2011: 9 pgs: 9 active+clean; 110 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 117 op/s 2021-07-31T02:25:14.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:14 smithi045 conmon[33655]: cluster 2021-07-31T02:25:12.337144+0000 2021-07-31T02:25:14.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:14 smithi045 conmon[33655]: mgr.x (mgr.24542) 1626 : cluster [DBG] pgmap v2011: 9 pgs: 9 active+clean; 110 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 117 op/s 2021-07-31T02:25:14.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:14 smithi045 conmon[39718]: cluster 2021-07-31T02:25:12.337144 2021-07-31T02:25:14.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:14 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 1626 : cluster [DBG] pgmap v2011: 9 pgs: 9 active+clean; 110 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 117 op/s 2021-07-31T02:25:15.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:15 smithi154 conmon[34553]: cluster 2021-07-31T02:25:14.337508+0000 mgr.x (mgr. 2021-07-31T02:25:15.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:15 smithi154 conmon[34553]: 24542) 1627 : cluster [DBG] pgmap v2012: 9 pgs: 9 active+clean; 71 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 380 op/s 2021-07-31T02:25:15.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:15 smithi045 conmon[33655]: cluster 2021-07-31T02:25:14.337508+0000 mgr.x ( 2021-07-31T02:25:15.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:15 smithi045 conmon[33655]: mgr.24542) 1627 : cluster [DBG] pgmap v2012: 9 pgs: 9 active+clean; 71 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 380 op/s 2021-07-31T02:25:15.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:15 smithi045 conmon[39718]: cluster 2021-07-31T02:25:14.337508+0000 mgr.x 2021-07-31T02:25:15.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:15 smithi045 conmon[39718]: (mgr.24542) 1627 : cluster [DBG] pgmap v2012: 9 pgs: 9 active+clean; 71 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 380 op/s 2021-07-31T02:25:18.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:18 smithi154 conmon[34553]: cluster 2021-07-31T02:25:16.337816+0000 mgr.x (mgr. 2021-07-31T02:25:18.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:18 smithi154 conmon[34553]: 24542) 1628 : cluster [DBG] pgmap v2013: 9 pgs: 9 active+clean; 71 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 448 KiB/s rd, 390 KiB/s wr, 378 op/s 2021-07-31T02:25:18.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:18 smithi045 conmon[33655]: cluster 2021-07-31T02:25:16. 2021-07-31T02:25:18.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:18 smithi045 conmon[33655]: 337816+0000 mgr.x (mgr.24542) 1628 : cluster [DBG] pgmap v2013: 9 pgs: 9 active+clean; 71 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 448 KiB/s rd, 390 KiB/s wr, 378 op/s 2021-07-31T02:25:18.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:18 smithi045 conmon[39718]: cluster 2021-07-31T02:25:16.337816+0000 2021-07-31T02:25:18.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:18 smithi045 conmon[39718]: mgr.x (mgr.24542) 1628 : cluster [DBG] pgmap v2013: 9 pgs: 9 active+clean; 71 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 448 KiB/s rd, 390 KiB/s wr, 378 op/s 2021-07-31T02:25:20.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:20 smithi154 conmon[34553]: cluster 2021-07-31T02:25:18.338106+0000 mgr.x (mgr.24542 2021-07-31T02:25:20.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:20 smithi154 conmon[34553]: ) 1629 : cluster [DBG] pgmap v2014: 9 pgs: 9 active+clean; 71 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 448 KiB/s rd, 390 KiB/s wr, 378 op/s 2021-07-31T02:25:20.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:20 smithi045 conmon[33655]: cluster 2021-07- 2021-07-31T02:25:20.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:20 smithi045 conmon[33655]: 31T02:25:18.338106+0000 mgr.x (mgr.24542) 1629 : cluster [DBG] pgmap v2014: 9 pgs: 9 active+clean; 71 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 448 KiB/s rd, 390 KiB/s wr, 378 op/s 2021-07-31T02:25:20.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:20 smithi045 conmon[39718]: cluster 2021-07-31T02:25:18. 2021-07-31T02:25:20.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:20 smithi045 conmon[39718]: 338106+0000 mgr.x (mgr.24542) 1629 : cluster [DBG] pgmap v2014: 9 pgs: 9 active+clean; 71 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 448 KiB/s rd, 390 KiB/s wr, 378 op/s 2021-07-31T02:25:22.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:22 smithi154 conmon[34553]: cluster 2021-07-31T02:25:20.338644+0000 mgr.x (mgr. 2021-07-31T02:25:22.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:22 smithi154 conmon[34553]: 24542) 1630 : cluster [DBG] pgmap v2015: 9 pgs: 9 active+clean; 35 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 448 KiB/s rd, 390 KiB/s wr, 616 op/s 2021-07-31T02:25:22.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:22 smithi045 conmon[33655]: cluster 2021-07-31T02:25:20.338644+0000 mgr.x (mgr 2021-07-31T02:25:22.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:22 smithi045 conmon[33655]: .24542) 1630 : cluster [DBG] pgmap v2015: 9 pgs: 9 active+clean; 35 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 448 KiB/s rd, 390 KiB/s wr, 616 op/s 2021-07-31T02:25:22.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:22 smithi045 conmon[39718]: cluster 2021-07-31T02:25 2021-07-31T02:25:22.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:22 smithi045 conmon[39718]: :20.338644+0000 mgr.x (mgr.24542) 1630 : cluster [DBG] pgmap v2015: 9 pgs: 9 active+clean; 35 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 448 KiB/s rd, 390 KiB/s wr, 616 op/s 2021-07-31T02:25:24.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:25:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:25:23.575Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:25:24.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:25:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:25:23.575Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:25:24.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:25:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:25:23.575Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:25:24.178 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-07-31T02:25:24.179 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2021-07-31T02:25:24.219 INFO:tasks.workunit:Stopping ['rados/load-gen-big.sh'] on client.0... 2021-07-31T02:25:24.220 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2021-07-31T02:25:24.443 DEBUG:teuthology.parallel:result is None 2021-07-31T02:25:24.444 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2021-07-31T02:25:24.465 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:24 smithi045 conmon[33655]: cluster 2021-07-31T02:25:22.338980+0000 mgr.x (mgr.24542 2021-07-31T02:25:24.466 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:24 smithi045 conmon[33655]: ) 1631 : cluster [DBG] pgmap v2016: 9 pgs: 9 active+clean; 35 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 501 op/s 2021-07-31T02:25:24.466 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:24 smithi045 conmon[39718]: cluster 2021-07-31T02:25:22.338980+0000 mgr.x ( 2021-07-31T02:25:24.466 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:24 smithi045 conmon[39718]: mgr.24542) 1631 : cluster [DBG] pgmap v2016: 9 pgs: 9 active+clean; 35 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 501 op/s 2021-07-31T02:25:24.468 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2021-07-31T02:25:24.469 DEBUG:teuthology.orchestra.run.smithi045:> rmdir -- /home/ubuntu/cephtest/mnt.0 2021-07-31T02:25:24.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:24 smithi154 conmon[34553]: cluster 2021-07-31T02:25:22.338980+0000 mgr.x (mgr.24542) 2021-07-31T02:25:24.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:24 smithi154 conmon[34553]: 1631 : cluster [DBG] pgmap v2016: 9 pgs: 9 active+clean; 35 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 501 op/s 2021-07-31T02:25:24.524 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2021-07-31T02:25:24.524 INFO:teuthology.task.full_sequential:In full_sequential, running task print... 2021-07-31T02:25:24.525 INFO:teuthology.task.print:**** done end rados_loadgenbig.yaml 2021-07-31T02:25:24.525 INFO:teuthology.task.full_sequential:In full_sequential, running task print... 2021-07-31T02:25:24.525 INFO:teuthology.task.print:**** done start rbd_import_export.yaml 2021-07-31T02:25:24.525 INFO:teuthology.task.full_sequential:In full_sequential, running task workunit... 2021-07-31T02:25:24.526 INFO:tasks.workunit:Pulling workunits from ref pacific 2021-07-31T02:25:24.527 INFO:tasks.workunit:Making a separate scratch dir for every client... 2021-07-31T02:25:24.527 DEBUG:teuthology.orchestra.run.smithi154:> stat -- /home/ubuntu/cephtest/mnt.1 2021-07-31T02:25:24.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-07-31T02:25:24.546 INFO:teuthology.orchestra.run.smithi154.stderr:stat: cannot stat '/home/ubuntu/cephtest/mnt.1': No such file or directory 2021-07-31T02:25:24.547 DEBUG:teuthology.orchestra.run.smithi154:> mkdir -- /home/ubuntu/cephtest/mnt.1 2021-07-31T02:25:24.602 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.1 2021-07-31T02:25:24.602 DEBUG:teuthology.orchestra.run.smithi154:> cd -- /home/ubuntu/cephtest/mnt.1 && mkdir -- client.1 2021-07-31T02:25:24.657 INFO:tasks.workunit:timeout=3h 2021-07-31T02:25:24.657 INFO:tasks.workunit:cleanup=True 2021-07-31T02:25:24.658 DEBUG:teuthology.orchestra.run.smithi154:> rm -rf /home/ubuntu/cephtest/clone.client.1 && git clone --depth 1 --branch pacific git://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.1 && cd /home/ubuntu/cephtest/clone.client.1 2021-07-31T02:25:24.712 INFO:tasks.workunit.client.1.smithi154.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.1'... 2021-07-31T02:25:24.737 INFO:tasks.workunit.client.1.smithi154.stderr:warning: Could not find remote branch pacific to clone. 2021-07-31T02:25:24.737 INFO:tasks.workunit.client.1.smithi154.stderr:fatal: Remote branch pacific not found in upstream origin 2021-07-31T02:25:24.738 DEBUG:teuthology.orchestra.run:got remote process result: 128 2021-07-31T02:25:24.739 INFO:tasks.workunit:failed to check out 'pacific' from git://git.ceph.com/ceph-ci.git; will also try in git://git.ceph.com/ceph.git 2021-07-31T02:25:24.739 DEBUG:teuthology.orchestra.run.smithi154:> rm -rf /home/ubuntu/cephtest/clone.client.1 && git clone --depth 1 --branch pacific git://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.1 && cd /home/ubuntu/cephtest/clone.client.1 2021-07-31T02:25:24.794 INFO:tasks.workunit.client.1.smithi154.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.1'... 2021-07-31T02:25:26.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:26 smithi154 conmon[34553]: cluster 2021-07-31T02:25:24.339510+0000 mgr.x (mgr.24542) 1632 : cluster [DBG] pgmap v2017: 9 pgs: 9 active+clean; 2.0 GiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 726 op/s 2021-07-31T02:25:26.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:26 smithi045 conmon[39718]: cluster 2021-07-31T02:25:24.339510+0000 mgr.x (mgr.24542) 1632 : cluster [DBG] pgmap v2017: 9 pgs: 9 active+clean; 2.0 GiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 726 op/s 2021-07-31T02:25:26.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:26 smithi045 conmon[33655]: cluster 2021-07-31T02:25:24.339510+0000 mgr.x (mgr.24542) 1632 : cluster [DBG] pgmap v2017: 9 pgs: 9 active+clean; 2.0 GiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 726 op/s 2021-07-31T02:25:28.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:28 smithi154 conmon[34553]: cluster 2021-07-31T02:25:26.339849+0000 mgr.x (mgr.24542) 1633 : cluster [DBG] pgmap v2018: 9 pgs: 9 active+clean; 2.0 GiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 462 op/s 2021-07-31T02:25:28.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:28 smithi154 conmon[34553]: audit 2021-07-31T02 2021-07-31T02:25:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:28 smithi154 conmon[34553]: :25:28.063274+0000 mon.b (mon.2) 258 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:25:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:28 smithi154 conmon[34553]: audit 2021-07-31T02:25:28.063805+0000 mon.a (mon.0) 1910 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:25:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:28 smithi154 conmon[34553]: audit 2021-07-31T02:25:28.087774+0000 mon.b 2021-07-31T02:25:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:28 smithi154 conmon[34553]: (mon.2) 259 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:25:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:28 smithi154 conmon[34553]: audit 2021- 2021-07-31T02:25:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:28 smithi154 conmon[34553]: 07-31T02:25:28.088312+0000 2021-07-31T02:25:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:28 smithi154 conmon[34553]: mon.a (mon.0) 1911 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:25:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:28 smithi154 conmon[34553]: 2021-07-31T02:25:28.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:28 smithi045 conmon[33655]: cluster 2021-07-31T02:25:26.339849+0000 2021-07-31T02:25:28.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:28 smithi045 conmon[33655]: mgr.x (mgr.24542) 1633 : cluster [DBG] pgmap v2018: 9 pgs: 9 active+clean; 2.0 GiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 462 op/s 2021-07-31T02:25:28.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:28 smithi045 conmon[33655]: audit 2021-07-31T02:25:28.063274+0000 mon.b (mon. 2021-07-31T02:25:28.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:28 smithi045 conmon[33655]: 2) 258 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:25:28.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:28 smithi045 conmon[33655]: audit 2021-07-31T02:25:28. 2021-07-31T02:25:28.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:28 smithi045 conmon[33655]: 063805+0000 mon.a (mon.0) 1910 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:25:28.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:28 smithi045 conmon[33655]: audit 2021-07-31T02: 2021-07-31T02:25:28.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:28 smithi045 conmon[33655]: 25:28.087774+0000 mon.b (mon.2) 259 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:25:28.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:28 smithi045 conmon[33655]: audit 2021-07-31T02:25:28. 2021-07-31T02:25:28.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:28 smithi045 conmon[33655]: 088312+0000 mon.a (mon.0) 1911 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:25:28.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:28 smithi045 conmon[39718]: cluster 2021-07-31T02:25:26.339849+0000 mgr.x ( 2021-07-31T02:25:28.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:28 smithi045 conmon[39718]: mgr.24542) 1633 : cluster [DBG] pgmap v2018: 9 pgs: 9 active+clean; 2.0 GiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 462 op/s 2021-07-31T02:25:28.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:28 smithi045 conmon[39718]: audit 2021-07-31T02:25:28.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:28 smithi045 conmon[39718]: 2021-07-31T02:25:28.063274+0000 mon.b (mon.2 2021-07-31T02:25:28.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:28 smithi045 conmon[39718]: ) 258 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:25:28.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:28 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T02:25:28.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:28 smithi045 conmon[39718]: T02:25:28.063805+0000 mon.a (mon.0) 2021-07-31T02:25:28.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:28 smithi045 conmon[39718]: 1910 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:25:28.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:28 smithi045 conmon[39718]: audit 2021 2021-07-31T02:25:28.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:28 smithi045 conmon[39718]: -07-31T02:25:28.087774 2021-07-31T02:25:28.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:28 smithi045 conmon[39718]: +0000 mon.b (mon.2) 259 : audit 2021-07-31T02:25:28.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:28 smithi045 conmon[39718]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:25:28.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:28 smithi045 conmon[39718]: audit 2021 2021-07-31T02:25:28.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:28 smithi045 conmon[39718]: -07-31T02:25:28.088312 2021-07-31T02:25:28.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:28 smithi045 conmon[39718]: +0000 mon.a (mon.0) 1911 : audit 2021-07-31T02:25:28.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:28 smithi045 conmon[39718]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:25:29.862 DEBUG:teuthology.orchestra.run.smithi154:> cd -- /home/ubuntu/cephtest/clone.client.1/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.1 2021-07-31T02:25:29.931 INFO:tasks.workunit.client.1.smithi154.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2021-07-31T02:25:29.933 INFO:tasks.workunit.client.1.smithi154.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2021-07-31T02:25:29.934 INFO:tasks.workunit.client.1.smithi154.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2021-07-31T02:25:30.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:30 smithi154 conmon[34553]: cluster 2021-07-31T02:25:28.340283+0000 mgr.x (mgr.24542) 1634 : 2021-07-31T02:25:30.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:30 smithi154 conmon[34553]: cluster [DBG] pgmap v2019: 9 pgs: 9 active+clean; 2.0 GiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 462 op/s 2021-07-31T02:25:30.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:30 smithi045 conmon[33655]: cluster 2021-07-31T02:25:28.340283+0000 mgr.x (mgr.24542) 1634 : cluster 2021-07-31T02:25:30.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:30 smithi045 conmon[33655]: [DBG] pgmap v2019: 9 pgs: 9 active+clean; 2.0 GiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 462 op/s 2021-07-31T02:25:30.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:30 smithi045 conmon[39718]: cluster 2021-07-31T02:25:28.340283+0000 mgr.x (mgr.24542) 1634 : cluster 2021-07-31T02:25:30.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:30 smithi045 conmon[39718]: [DBG] pgmap v2019: 9 pgs: 9 active+clean; 2.0 GiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 462 op/s 2021-07-31T02:25:30.624 INFO:tasks.workunit.client.1.smithi154.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2021-07-31T02:25:30.671 INFO:tasks.workunit.client.1.smithi154.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2021-07-31T02:25:30.707 INFO:tasks.workunit.client.1.smithi154.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2021-07-31T02:25:30.708 INFO:tasks.workunit.client.1.smithi154.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2021-07-31T02:25:30.708 INFO:tasks.workunit.client.1.smithi154.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2021-07-31T02:25:30.743 INFO:tasks.workunit.client.1.smithi154.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2021-07-31T02:25:30.746 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2021-07-31T02:25:30.746 DEBUG:teuthology.orchestra.run.smithi154:> dd if=/home/ubuntu/cephtest/workunits.list.client.1 of=/dev/stdout 2021-07-31T02:25:30.801 INFO:tasks.workunit:Running workunits matching rbd/import_export.sh on client.1... 2021-07-31T02:25:30.802 INFO:tasks.workunit:Running workunit rbd/import_export.sh... 2021-07-31T02:25:30.803 DEBUG:teuthology.orchestra.run.smithi154:workunit test rbd/import_export.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=pacific TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 RBD_CREATE_ARGS=--new-format adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/rbd/import_export.sh 2021-07-31T02:25:30.870 INFO:tasks.workunit.client.1.smithi154.stderr:+ export RBD_FORCE_ALLOW_V1=1 2021-07-31T02:25:30.871 INFO:tasks.workunit.client.1.smithi154.stderr:+ RBD_FORCE_ALLOW_V1=1 2021-07-31T02:25:30.871 INFO:tasks.workunit.client.1.smithi154.stderr:+ TMPDIR=/tmp/rbd_import_export_59341 2021-07-31T02:25:30.871 INFO:tasks.workunit.client.1.smithi154.stderr:+ rm -rf /tmp/rbd_import_export_59341 2021-07-31T02:25:30.872 INFO:tasks.workunit.client.1.smithi154.stderr:+ mkdir /tmp/rbd_import_export_59341 2021-07-31T02:25:30.872 INFO:tasks.workunit.client.1.smithi154.stderr:+ trap 'rm -rf /tmp/rbd_import_export_59341' INT TERM EXIT 2021-07-31T02:25:30.872 INFO:tasks.workunit.client.1.smithi154.stderr:+ mkdir foo.59341 2021-07-31T02:25:30.873 INFO:tasks.workunit.client.1.smithi154.stderr:+ rbd import foo.59341 foo.dir 2021-07-31T02:25:30.898 INFO:tasks.workunit.client.1.smithi154.stderr:rbd: cannot import a directory 2021-07-31T02:25:30.898 INFO:tasks.workunit.client.1.smithi154.stderr: Importing image: 0% complete...failed. 2021-07-31T02:25:30.899 INFO:tasks.workunit.client.1.smithi154.stderr:rbd: import failed: (21) Is a directory 2021-07-31T02:25:30.900 INFO:tasks.workunit.client.1.smithi154.stderr:+ true 2021-07-31T02:25:30.900 INFO:tasks.workunit.client.1.smithi154.stderr:+ rmdir foo.59341 2021-07-31T02:25:30.901 INFO:tasks.workunit.client.1.smithi154.stderr:+ dd if=/bin/sh of=/tmp/rbd_import_export_59341/img bs=1k count=1 seek=10 2021-07-31T02:25:30.902 INFO:tasks.workunit.client.1.smithi154.stderr:1+0 records in 2021-07-31T02:25:30.902 INFO:tasks.workunit.client.1.smithi154.stderr:1+0 records out 2021-07-31T02:25:30.902 INFO:tasks.workunit.client.1.smithi154.stderr:1024 bytes (1.0 kB, 1.0 KiB) copied, 4.0906e-05 s, 25.0 MB/s 2021-07-31T02:25:30.902 INFO:tasks.workunit.client.1.smithi154.stderr:+ dd if=/bin/dd of=/tmp/rbd_import_export_59341/img bs=1k count=10 seek=100 2021-07-31T02:25:30.903 INFO:tasks.workunit.client.1.smithi154.stderr:10+0 records in 2021-07-31T02:25:30.904 INFO:tasks.workunit.client.1.smithi154.stderr:10+0 records out 2021-07-31T02:25:30.904 INFO:tasks.workunit.client.1.smithi154.stderr:10240 bytes (10 kB, 10 KiB) copied, 4.8138e-05 s, 213 MB/s 2021-07-31T02:25:30.904 INFO:tasks.workunit.client.1.smithi154.stderr:+ dd if=/bin/rm of=/tmp/rbd_import_export_59341/img bs=1k count=100 seek=1000 2021-07-31T02:25:30.904 INFO:tasks.workunit.client.1.smithi154.stderr:70+1 records in 2021-07-31T02:25:30.905 INFO:tasks.workunit.client.1.smithi154.stderr:70+1 records out 2021-07-31T02:25:30.905 INFO:tasks.workunit.client.1.smithi154.stderr:72216 bytes (72 kB, 71 KiB) copied, 0.000181456 s, 398 MB/s 2021-07-31T02:25:30.905 INFO:tasks.workunit.client.1.smithi154.stderr:+ dd if=/bin/ls of=/tmp/rbd_import_export_59341/img bs=1k seek=10000 2021-07-31T02:25:30.906 INFO:tasks.workunit.client.1.smithi154.stderr:140+1 records in 2021-07-31T02:25:30.907 INFO:tasks.workunit.client.1.smithi154.stderr:140+1 records out 2021-07-31T02:25:30.907 INFO:tasks.workunit.client.1.smithi154.stderr:143368 bytes (143 kB, 140 KiB) copied, 0.000310603 s, 462 MB/s 2021-07-31T02:25:30.907 INFO:tasks.workunit.client.1.smithi154.stderr:+ dd if=/bin/ln of=/tmp/rbd_import_export_59341/img bs=1k seek=100000 2021-07-31T02:25:30.907 INFO:tasks.workunit.client.1.smithi154.stderr:70+1 records in 2021-07-31T02:25:30.908 INFO:tasks.workunit.client.1.smithi154.stderr:70+1 records out 2021-07-31T02:25:30.908 INFO:tasks.workunit.client.1.smithi154.stderr:72320 bytes (72 kB, 71 KiB) copied, 0.000202429 s, 357 MB/s 2021-07-31T02:25:30.908 INFO:tasks.workunit.client.1.smithi154.stderr:+ dd if=/bin/grep of=/tmp/rbd_import_export_59341/img bs=1k seek=1000000 2021-07-31T02:25:30.908 INFO:tasks.workunit.client.1.smithi154.stderr:189+1 records in 2021-07-31T02:25:30.908 INFO:tasks.workunit.client.1.smithi154.stderr:189+1 records out 2021-07-31T02:25:30.909 INFO:tasks.workunit.client.1.smithi154.stderr:193768 bytes (194 kB, 189 KiB) copied, 0.000403587 s, 480 MB/s 2021-07-31T02:25:30.909 INFO:tasks.workunit.client.1.smithi154.stderr:+ rbd rm testimg 2021-07-31T02:25:30.957 INFO:tasks.workunit.client.1.smithi154.stderr: Removing image: 0% complete...failed. 2021-07-31T02:25:30.958 INFO:tasks.workunit.client.1.smithi154.stderr:rbd: delete error: (2) No such file or directory 2021-07-31T02:25:30.960 INFO:tasks.workunit.client.1.smithi154.stderr:+ true 2021-07-31T02:25:30.960 INFO:tasks.workunit.client.1.smithi154.stderr:+ rbd import --new-format /tmp/rbd_import_export_59341/img testimg 2021-07-31T02:25:32.221 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: debug 2021-07-31T02:25:31.846+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:25:32.222 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:25:32.222 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:25:32.222 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: yield 2021-07-31T02:25:32.222 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:25:32.223 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:25:32.223 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:25:32.223 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:25:32.223 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:25:32.223 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:25:32.224 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:25:32.224 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:25:32.224 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:25:32.224 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: task.result() 2021-07-31T02:25:32.224 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:25:32.225 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: debug 2021-07-31T02:25:31.857+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:25:32.225 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:25:32.225 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:25:32.225 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: yield 2021-07-31T02:25:32.225 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:25:32.226 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:25:32.226 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:25:32.226 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:25:32.226 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:25:32.227 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:25:32.227 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:25:32.227 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:25:32.228 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:25:32.228 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: task.result() 2021-07-31T02:25:32.228 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:25:32.228 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: debug 2021-07-31T02:25:31.884+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:25:32.228 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:25:32.229 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:25:32.229 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: yield 2021-07-31T02:25:32.229 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:25:32.229 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:25:32.230 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:25:32.230 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:25:32.230 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:25:32.231 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:25:32.231 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:25:32.231 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:25:32.231 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:25:32.232 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: task.result() 2021-07-31T02:25:32.232 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:25:32.232 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: debug 2021-07-31T02:25:31.884+0000 7fcb9095b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:25:32.232 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:25:32.233 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:25:32.233 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: yield 2021-07-31T02:25:32.233 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:25:32.233 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:25:32.233 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:25:32.234 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:25:32.234 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:25:32.234 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:25:32.235 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:25:32.235 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:25:32.235 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:25:32.236 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: task.result() 2021-07-31T02:25:32.236 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:25:32.236 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: 2021-07-31T02:25:32.236 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:25:32.236 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: 2021-07-31T02:25:32.237 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:25:32.237 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:25:32.237 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:25:32.237 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:25:32.237 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:25:32.238 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:25:32.238 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:25:32.238 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:25:32.238 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:25:32.239 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:25:32.239 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:25:32.239 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:25:32.239 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:25:32.240 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:25:32.240 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:25:32.240 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:25:32.240 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:25:32.240 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:25:32.241 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: raise 2021-07-31T02:25:32.241 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:25:32.241 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:25:32.241 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:25:32.241 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:25:32.242 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:25:32.242 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: debug 2021-07-31T02:25:31.893+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:25:32.242 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:25:32.242 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:25:32.242 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: yield 2021-07-31T02:25:32.243 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:25:32.243 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:25:32.243 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:25:32.243 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:25:32.244 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:25:32.244 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:25:32.244 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:25:32.244 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:25:32.244 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:25:32.249 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: task.result() 2021-07-31T02:25:32.249 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:25:32.249 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: debug 2021-07-31T02:25:31.893+0000 7fcb9115c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:25:32.250 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:25:32.250 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:25:32.250 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: yield 2021-07-31T02:25:32.251 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:25:32.251 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:25:32.252 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:25:32.252 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:25:32.252 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:25:32.253 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:25:32.253 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:25:32.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:25:32.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:25:32.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: task.result() 2021-07-31T02:25:32.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:25:32.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: 2021-07-31T02:25:32.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:25:32.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: 2021-07-31T02:25:32.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:25:32.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:25:32.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:25:32.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:25:32.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:25:32.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:25:32.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:25:32.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:25:32.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:25:32.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:25:32.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:25:32.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:25:32.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:25:32.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:25:32.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:25:32.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:25:32.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:25:32.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:25:32.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: raise 2021-07-31T02:25:32.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:25:32.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:25:32.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:25:32.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:25:32.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:25:31 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:25:32.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:32 smithi154 conmon[34553]: cluster 2021-07-31T02:25:30.340599+0000 mgr.x (mgr.24542) 2021-07-31T02:25:32.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:32 smithi154 conmon[34553]: 1635 : cluster [DBG] pgmap v2020: 9 pgs: 9 active+clean; 577 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 476 op/s 2021-07-31T02:25:32.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:32 smithi154 conmon[34553]: cluster 2021-07-31T02:25:31.221670+0000 mon.a (mon.0) 1912 : cluster [DBG] osdmap e1026: 8 total, 8 up, 8 in 2021-07-31T02:25:32.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:32 smithi154 conmon[34553]: audit 2021-07-31T02:25:32.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:32 smithi154 conmon[34553]: 2021-07-31T02:25: 2021-07-31T02:25:32.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:32 smithi154 conmon[34553]: 31.799117+0000 mon.b (mon.2) 2021-07-31T02:25:32.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:32 smithi154 conmon[34553]: 260 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:25:32.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:32 smithi154 conmon[34553]: audit 2021-07-31T02:25:31.800675+0000 mon.b (mon.2) 261 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:25:32.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:32 smithi154 conmon[34553]: audit 2021-07-31T02 2021-07-31T02:25:32.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:32 smithi154 conmon[34553]: :25:31.801641+0000 mon.b (mon.2) 262 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:25:32.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:32 smithi045 conmon[33655]: cluster 2021-07-31T02:25:30.340599+0000 mgr.x (mgr.24542) 1635 : 2021-07-31T02:25:32.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:32 smithi045 conmon[33655]: cluster [DBG] pgmap v2020: 9 pgs: 9 active+clean; 577 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 476 op/s 2021-07-31T02:25:32.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:32 smithi045 conmon[33655]: cluster 2021-07-31T02:25:31.221670+0000 mon.a (mon.0) 1912 : cluster [DBG] osdmap e1026: 8 total, 8 up, 8 in 2021-07-31T02:25:32.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:32 smithi045 conmon[33655]: audit 2021-07- 2021-07-31T02:25:32.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:32 smithi045 conmon[33655]: 31T02:25:31.799117+0000 mon.b (mon.2) 260 : 2021-07-31T02:25:32.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:32 smithi045 conmon[33655]: audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:25:32.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:32 smithi045 conmon[33655]: audit 2021-07-31T02:25:32.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:32 smithi045 conmon[33655]: 2021-07-31T02:25:31.800675+0000 2021-07-31T02:25:32.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:32 smithi045 conmon[33655]: mon.b (mon.2) 261 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:25:32.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:32 smithi045 conmon[33655]: 2021-07-31T02:25:32.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:32 smithi045 conmon[33655]: audit 2021-07-31 2021-07-31T02:25:32.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:32 smithi045 conmon[33655]: T02:25:31.801641+0000 mon.b (mon.2) 262 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:25:32.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:32 smithi045 conmon[39718]: cluster 2021-07-31T02:25:30.340599+0000 mgr.x ( 2021-07-31T02:25:32.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:32 smithi045 conmon[39718]: mgr.24542) 1635 : cluster [DBG] pgmap v2020: 9 pgs: 9 active+clean; 577 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 476 op/s 2021-07-31T02:25:32.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:32 smithi045 conmon[39718]: cluster 2021-07-31T02:25:31.221670 2021-07-31T02:25:32.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:32 smithi045 conmon[39718]: +0000 mon.a (mon.0) 1912 : cluster [DBG] osdmap e1026: 8 total, 8 up, 8 in 2021-07-31T02:25:32.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:32 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T02:25:32.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:32 smithi045 conmon[39718]: 31T02:25:31.799117+0000 mon.b (mon.2) 260 : audit 2021-07-31T02:25:32.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:32 smithi045 conmon[39718]: [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:25:32.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:32 smithi045 conmon[39718]: audit 2021 2021-07-31T02:25:32.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:32 smithi045 conmon[39718]: -07-31T02:25:31.800675+0000 mon.b (mon 2021-07-31T02:25:32.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:32 smithi045 conmon[39718]: .2) 261 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:25:32.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:32 smithi045 conmon[39718]: audit 2021-07-31T02:25:32.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:32 smithi045 conmon[39718]: 2021-07-31T02:25:31. 2021-07-31T02:25:32.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:32 smithi045 conmon[39718]: 801641+0000 mon.b (mon.2) 262 : audit 2021-07-31T02:25:32.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:32 smithi045 conmon[39718]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:25:32.751 INFO:tasks.workunit.client.1.smithi154.stderr: Importing image: 1% complete... Importing image: 2% complete... Importing image: 3% complete... Importing image: 4% complete... Importing image: 5% complete... Importing image: 6% complete... Importing image: 7% complete... Importing image: 8% complete... Importing image: 9% complete... Importing image: 10% complete... Importing image: 11% complete... Importing image: 12% complete... Importing image: 13% complete... Importing image: 14% complete... Importing image: 15% complete... Importing image: 16% complete... Importing image: 17% complete... Importing image: 18% complete... Importing image: 19% complete... Importing image: 20% complete... Importing image: 21% complete... Importing image: 22% complete... Importing image: 23% complete... Importing image: 24% complete... Importing image: 25% complete... Importing image: 26% complete... Importing image: 27% complete... Importing image: 28% complete... Importing image: 29% complete... Importing image: 30% complete... Importing image: 31% complete... Importing image: 32% complete... Importing image: 33% complete... Importing image: 34% complete... Importing image: 35% complete... Importing image: 36% complete... Importing image: 37% complete... Importing image: 38% complete... Importing image: 39% complete... Importing image: 40% complete... Importing image: 41% complete... Importing image: 42% complete... Importing image: 43% complete... Importing image: 44% complete... Importing image: 45% complete... Importing image: 46% complete... Importing image: 47% complete... Importing image: 48% complete... Importing image: 49% complete... Importing image: 50% complete... Importing image: 51% complete... Importing image: 52% complete... Importing image: 53% complete... Importing image: 54% complete... Importing image: 55% complete... Importing image: 56% complete... Importing image: 57% complete... Importing image: 58% complete... Importing image: 59% complete... Importing image: 60% complete... Importing image: 61% complete... Importing image: 62% complete... Importing image: 63% complete... Importing image: 64% complete... Importing image: 65% complete... Importing image: 66% complete... Importing image: 67% complete... Importing image: 68% complete... Importing image: 69% complete... Importing image: 70% complete... Importing image: 71% complete... Importing image: 72% complete... Importing image: 73% complete... Importing image: 74% complete... Importing image: 75% complete... Importing image: 76% complete... Importing image: 77% complete... Importing image: 78% complete... Importing image: 79% complete... Importing image: 80% complete... Importing image: 81% complete... Importing image: 82% complete... Importing image: 83% complete... Importing image: 84% complete... Importing image: 85% complete... Importing image: 86% complete... Importing image: 87% complete... Importing image: 88% complete... Importing image: 89% complete... Importing image: 90% complete... Importing image: 91% complete... Importing image: 92% complete... Importing image: 93% complete... Importing image: 94% complete... Importing image: 95% complete... Importing image: 96% complete... Importing image: 97% complete... Importing image: 98% complete... Importing image: 99% complete... Importing image: 100% complete...done. 2021-07-31T02:25:32.751 INFO:tasks.workunit.client.1.smithi154.stderr:rbd: --new-format is deprecated, use --image-format 2 2021-07-31T02:25:32.754 INFO:tasks.workunit.client.1.smithi154.stderr:+ rbd export testimg /tmp/rbd_import_export_59341/img2 2021-07-31T02:25:33.018 INFO:tasks.workunit.client.1.smithi154.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2021-07-31T02:25:33.020 INFO:tasks.workunit.client.1.smithi154.stderr:+ rbd export testimg - 2021-07-31T02:25:33.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:25:31.802422+0000 mgr.x (mgr.24542) 1636 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:25:33.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:25:31.847180+0000 mgr.x (mgr.24542) 1637 : cephadm [ERR] 2021-07-31T02:25:33.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:25:33.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:25:33.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: yield 2021-07-31T02:25:33.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:25:33.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:25:33.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:25:33.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:25:33.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:25:33.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:25:33.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:25:33.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:25:33.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:25:33.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:25:33.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:25:33.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:25:31.847648+0000 mgr.x (mgr.24542) 1638 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:25:33.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:25:31.858666+0000 mgr.x (mgr.24542) 1639 : cephadm [ERR] 2021-07-31T02:25:33.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:25:33.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:25:33.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: yield 2021-07-31T02:25:33.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:25:33.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:25:33.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:25:33.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:25:33.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:25:33.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:25:33.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:25:33.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:25:33.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:25:33.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:25:33.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:25:33.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:25:31.859605+0000 mgr.x (mgr.24542) 1640 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:25:33.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:25:31.885080+0000 mgr.x (mgr.24542) 1641 : cephadm [ERR] 2021-07-31T02:25:33.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:25:33.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:25:33.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: yield 2021-07-31T02:25:33.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:25:33.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:25:33.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:25:33.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:25:33.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:25:33.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:25:33.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:25:33.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:25:33.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:25:33.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:25:33.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:25:33.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:25:31.885529+0000 mgr.x (mgr.24542) 2021-07-31T02:25:33.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: 1642 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:25:33.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:25:33.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:25:33.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: yield 2021-07-31T02:25:33.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:25:33.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:25:33.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:25:33.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:25:33.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:25:33.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:25:33.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:25:33.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:25:33.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:25:33.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:25:33.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:25:33.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: 2021-07-31T02:25:33.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:25:33.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: 2021-07-31T02:25:33.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:25:33.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:25:33.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:25:33.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:25:33.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:25:33.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:25:33.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:25:33.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:25:33.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:25:33.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:25:33.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:25:33.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:25:33.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:25:33.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:25:33.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:25:33.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:25:33.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:25:33.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:25:33.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: raise 2021-07-31T02:25:33.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:25:33.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:25:33.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:25:33.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:25:33.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:25:33.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:25:31.894232+0000 mgr.x (mgr.24542) 1643 : cephadm [ERR] 2021-07-31T02:25:33.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:25:33.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:25:33.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: yield 2021-07-31T02:25:33.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:25:33.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:25:33.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:25:33.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:25:33.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:25:33.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:25:33.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:25:33.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:25:33.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:25:33.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:25:33.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:25:33.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:25:31.894610 2021-07-31T02:25:33.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: +0000 mgr.x (mgr.24542) 1644 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:25:33.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:25:33.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:25:33.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: yield 2021-07-31T02:25:33.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:25:33.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:25:33.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:25:33.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:25:33.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:25:33.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:25:33.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:25:33.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:25:33.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:25:33.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:25:33.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:25:33.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: 2021-07-31T02:25:33.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:25:33.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: 2021-07-31T02:25:33.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:25:33.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:25:33.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:25:33.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:25:33.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:25:33.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:25:33.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:25:33.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:25:33.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:25:33.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:25:33.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:25:33.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:25:33.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:25:33.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:25:33.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:25:33.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:25:33.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:25:33.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:25:33.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: raise 2021-07-31T02:25:33.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:25:33.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:25:33.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:25:33.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:25:33.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:25:33.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:33 smithi154 conmon[34553]: cluster 2021-07-31T02:25:32.227903+0000 mon.a (mon.0) 1913 : cluster [DBG] osdmap e1027: 8 total, 8 up, 8 in 2021-07-31T02:25:33.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:25:31.802422+0000 mgr.x (mgr.24542) 1636 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:25:33.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: 2021-07-31T02:25:33.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:25:31.847180+0000 mgr.x (mgr.24542) 1637 : cephadm [ERR] 2021-07-31T02:25:33.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:25:33.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:25:33.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: yield 2021-07-31T02:25:33.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:25:33.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:25:33.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:25:33.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:25:33.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:25:33.539 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:25:33.539 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:25:33.539 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:25:33.539 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:25:33.540 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:25:33.540 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:25:33.540 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:25:31.847648+0000 mgr.x (mgr.24542) 1638 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:25:33.540 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: cephadm 2021-07 2021-07-31T02:25:33.540 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: -31T02:25:31 2021-07-31T02:25:33.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: .858666+0000 mgr.x (mgr.24542 2021-07-31T02:25:33.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: ) 1639 : cephadm [ERR] 2021-07-31T02:25:33.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:25:33.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:25:33.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: yield 2021-07-31T02:25:33.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:25:33.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:25:33.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:25:33.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:25:33.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:25:33.543 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:25:33.543 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:25:33.543 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:25:33.543 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:25:33.544 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:25:33.544 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:25:33.544 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: 2021-07-31T02:25:33.544 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:25:31.859605+0000 mgr.x ( 2021-07-31T02:25:33.544 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: mgr.24542) 1640 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:25:33.545 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: cephadm 2021- 2021-07-31T02:25:33.545 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: 07-31T02:25:31.885080 2021-07-31T02:25:33.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:25:31.802422+0000 mgr.x ( 2021-07-31T02:25:33.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: mgr.24542) 1636 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:25:33.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:25:31.847180+0000 2021-07-31T02:25:33.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: mgr.x (mgr.24542) 1637 : cephadm [ERR] 2021-07-31T02:25:33.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:25:33.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:25:33.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: yield 2021-07-31T02:25:33.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:25:33.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:25:33.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:25:33.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:25:33.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:25:33.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:25:33.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:25:33.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:25:33.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:25:33.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:25:33.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:25:33.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:25:31.847648+0000 mgr.x (mgr.24542) 1638 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:25:33.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:25:31.858666 2021-07-31T02:25:33.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 1639 : cephadm [ERR] 2021-07-31T02:25:33.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:25:33.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:25:33.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: yield 2021-07-31T02:25:33.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:25:33.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:25:33.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:25:33.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:25:33.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:25:33.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:25:33.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:25:33.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:25:33.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:25:33.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:25:33.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:25:33.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:25:31.859605+0000 mgr.x (mgr.24542) 1640 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:25:33.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: cephadm 2021 2021-07-31T02:25:33.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: -07-31T02:25:31.885080+0000 mgr.x 2021-07-31T02:25:33.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: (mgr.24542) 1641 : cephadm [ERR] 2021-07-31T02:25:33.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:25:33.555 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 1641 2021-07-31T02:25:33.555 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: : cephadm [ERR] 2021-07-31T02:25:33.555 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:25:33.555 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:25:33.556 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: yield 2021-07-31T02:25:33.556 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:25:33.556 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:25:33.556 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:25:33.556 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:25:33.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:25:33.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:25:33.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:25:33.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:25:33.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:25:33.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:25:33.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:25:33.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:25:31. 2021-07-31T02:25:33.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: 885529+0000 mgr.x (mgr.24542 2021-07-31T02:25:33.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: ) 1642 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:25:33.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:25:33.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:25:33.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: yield 2021-07-31T02:25:33.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:25:33.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:25:33.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:25:33.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:25:33.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:25:33.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:25:33.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:25:33.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:25:33.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:25:33.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:25:33.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:25:33.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: 2021-07-31T02:25:33.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:25:33.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: 2021-07-31T02:25:33.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:25:33.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:25:33.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:25:33.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:25:33.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:25:33.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:25:33.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:25:33.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:25:33.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: yield 2021-07-31T02:25:33.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:25:33.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:25:33.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:25:33.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:25:33.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:25:33.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:25:33.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:25:33.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:25:33.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:25:33.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:25:33.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:25:33.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: 2021-07-31T02:25:33.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:25:31.885529+0000 mgr.x (mgr.24542) 1642 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:25:33.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:25:33.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:25:33.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: yield 2021-07-31T02:25:33.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:25:33.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:25:33.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:25:33.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:25:33.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:25:33.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:25:33.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:25:33.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:25:33.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:25:33.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:25:33.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:25:33.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: 2021-07-31T02:25:33.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:25:33.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: 2021-07-31T02:25:33.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:25:33.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:25:33.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:25:33.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:25:33.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:25:33.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:25:33.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:25:33.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:25:33.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:25:33.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:25:33.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:25:33.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:25:33.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:25:33.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:25:33.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:25:33.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:25:33.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:25:33.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:25:33.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:25:33.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:25:33.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: raise 2021-07-31T02:25:33.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:25:33.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:25:33.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:25:33.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:25:33.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:25:33.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: 2021-07-31T02:25:33.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:25:31.894232+0000 mgr.x (mgr.24542) 1643 : cephadm [ERR] 2021-07-31T02:25:33.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:25:33.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:25:33.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: yield 2021-07-31T02:25:33.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:25:33.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:25:33.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:25:33.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:25:33.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:25:33.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:25:33.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:25:33.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:25:33.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:25:33.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:25:33.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:25:33.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:25:31.894610+0000 mgr.x (mgr.24542) 1644 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:25:33.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:25:33.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:25:33.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: yield 2021-07-31T02:25:33.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:25:33.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:25:33.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:25:33.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:25:33.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:25:33.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:25:33.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:25:33.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:25:33.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:25:33.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:25:33.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:25:33.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: raise 2021-07-31T02:25:33.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:25:33.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:25:33.585 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:25:33.585 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:25:33.585 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:25:33.585 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:25:31.894232+0000 mgr.x (mgr.24542) 1643 : cephadm [ERR] 2021-07-31T02:25:33.585 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:25:33.586 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:25:33.586 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: yield 2021-07-31T02:25:33.586 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:25:33.586 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:25:33.587 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:25:33.587 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:25:33.587 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:25:33.587 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:25:33.587 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:25:33.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:25:33.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:25:33.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:25:33.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:25:33.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:25:31.894610+0000 mgr.x (mgr.24542) 1644 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:25:33.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:25:33.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:25:33.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: yield 2021-07-31T02:25:33.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:25:33.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:25:33.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:25:33.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:25:33.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:25:33.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:25:33.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:25:33.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:25:33.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:25:33.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:25:33.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:25:33.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:25:33.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:25:33.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:25:33.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: 2021-07-31T02:25:33.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:25:33.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: 2021-07-31T02:25:33.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:25:33.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:25:33.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:25:33.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:25:33.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:25:33.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:25:33.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:25:33.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:25:33.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:25:33.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:25:33.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:25:33.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:25:33.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:25:33.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:25:33.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:25:33.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:25:33.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:25:33.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:25:33.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: raise 2021-07-31T02:25:33.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:25:33.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:25:33.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:25:33.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:25:33.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:25:33.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: 2021-07-31T02:25:33.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[33655]: cluster 2021-07-31T02:25:32.227903+0000 mon.a (mon.0) 1913 : cluster [DBG] osdmap e1027: 8 total, 8 up, 8 in 2021-07-31T02:25:33.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:25:33.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:25:33.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:25:33.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:25:33.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:25:33.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:25:33.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: 2021-07-31T02:25:33.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:25:33.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: 2021-07-31T02:25:33.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:25:33.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:25:33.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:25:33.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:25:33.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:25:33.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:25:33.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:25:33.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:25:33.603 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:25:33.603 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:25:33.603 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:25:33.603 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:25:33.604 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:25:33.604 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:25:33.604 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:25:33.604 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:25:33.604 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:25:33.605 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:25:33.605 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: raise 2021-07-31T02:25:33.605 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:25:33.605 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:25:33.605 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:25:33.606 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:25:33.606 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:25:33.606 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[39718]: cluster 2021-07-31T02:25:32.227903+0000 mon.a (mon.0) 1913 : cluster [DBG] osdmap e1027: 8 total, 8 up, 8 in 2021-07-31T02:25:33.696 INFO:tasks.workunit.client.1.smithi154.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2021-07-31T02:25:33.699 INFO:tasks.workunit.client.1.smithi154.stderr:+ rbd rm testimg 2021-07-31T02:25:33.817 INFO:tasks.workunit.client.1.smithi154.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2021-07-31T02:25:33.820 INFO:tasks.workunit.client.1.smithi154.stderr:+ cmp /tmp/rbd_import_export_59341/img /tmp/rbd_import_export_59341/img2 2021-07-31T02:25:34.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:25:33.575Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:25:34.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:25:33.575Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:25:34.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:25:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:25:33.575Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:25:34.364 INFO:tasks.workunit.client.1.smithi154.stderr:+ cmp /tmp/rbd_import_export_59341/img /tmp/rbd_import_export_59341/img3 2021-07-31T02:25:34.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:34 smithi154 conmon[34553]: cluster 2021-07-31T02:25:32.341010+0000 mgr.x (mgr.24542) 1645 2021-07-31T02:25:34.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:34 smithi154 conmon[34553]: : cluster [DBG] pgmap v2023: 9 pgs: 9 active+clean; 577 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 19 op/s 2021-07-31T02:25:34.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:34 smithi045 conmon[33655]: cluster 2021-07-31 2021-07-31T02:25:34.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:34 smithi045 conmon[33655]: T02:25:32.341010+0000 mgr.x (mgr.24542) 1645 : cluster [DBG] pgmap v2023: 9 pgs: 9 active+clean; 577 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 19 op/s 2021-07-31T02:25:34.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:34 smithi045 conmon[39718]: cluster 2021-07-31T02:25:32.341010+0000 mgr.x ( 2021-07-31T02:25:34.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:34 smithi045 conmon[39718]: mgr.24542) 1645 : cluster [DBG] pgmap v2023: 9 pgs: 9 active+clean; 577 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 19 op/s 2021-07-31T02:25:34.767 INFO:tasks.workunit.client.1.smithi154.stderr:+ rm /tmp/rbd_import_export_59341/img2 /tmp/rbd_import_export_59341/img3 2021-07-31T02:25:34.933 INFO:tasks.workunit.client.1.smithi154.stderr:+ rbd import --new-format - testimg 2021-07-31T02:25:35.252 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:25:35.395 INFO:tasks.workunit.client.1.smithi154.stderr: Importing image: 100% complete...done. 2021-07-31T02:25:35.396 INFO:tasks.workunit.client.1.smithi154.stderr:rbd: --new-format is deprecated, use --image-format 2 2021-07-31T02:25:35.398 INFO:tasks.workunit.client.1.smithi154.stderr:+ rbd export testimg /tmp/rbd_import_export_59341/img2 2021-07-31T02:25:35.536 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:25:35.536 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 40m ago 41m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:25:35.537 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 38m ago 40m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:25:35.537 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 38m ago 44m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:25:35.537 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 40m ago 46m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:25:35.537 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 40m ago 46m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:25:35.538 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 38m ago 44m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:25:35.538 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 40m ago 44m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:25:35.538 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 40m ago 41m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:25:35.539 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 38m ago 41m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:25:35.539 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 40m ago 44m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:25:35.539 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 40m ago 43m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:25:35.539 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 40m ago 43m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:25:35.540 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 40m ago 43m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:25:35.540 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 38m ago 42m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:25:35.540 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 38m ago 42m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:25:35.540 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 38m ago 42m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:25:35.541 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 38m ago 41m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:25:35.541 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 38m ago 41m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:25:35.678 INFO:tasks.workunit.client.1.smithi154.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2021-07-31T02:25:35.681 INFO:tasks.workunit.client.1.smithi154.stderr:+ rbd export testimg - 2021-07-31T02:25:35.875 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:25:35.876 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:25:35.876 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:25:35.876 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:25:35.876 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:25:35.877 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:25:35.877 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:25:35.877 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:25:35.877 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:25:35.878 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:25:35.878 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:25:35.878 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:25:35.878 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:25:35.879 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:25:35.879 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:25:35.879 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:25:35.879 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:25:36.377 INFO:tasks.workunit.client.1.smithi154.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2021-07-31T02:25:36.380 INFO:tasks.workunit.client.1.smithi154.stderr:+ rbd rm testimg 2021-07-31T02:25:36.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:36 smithi154 conmon[34553]: cluster 2021-07-31T02:25:34.341384+0000 mgr.x (mgr.24542) 1646 : cluster [DBG] pgmap v2024: 9 pgs: 9 active+clean; 577 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 148 KiB/s rd, 64 KiB/s wr, 58 op/s 2021-07-31T02:25:36.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:36 smithi154 conmon[34553]: audit 2021-07-31T02:25:35. 2021-07-31T02:25:36.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:36 smithi154 conmon[34553]: 240340+0000 mgr.x (mgr.24542) 1647 : audit [DBG] from='client.26371 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:25:36.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:36 smithi154 conmon[34553]: audit 2021-07-31T02:25:35.876169+0000 mon.a (mon.0) 1914 : audit [DBG] from='client.? 172.21.15.45:0/505434814' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:25:36.513 INFO:tasks.workunit.client.1.smithi154.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2021-07-31T02:25:36.515 INFO:tasks.workunit.client.1.smithi154.stderr:+ cmp /tmp/rbd_import_export_59341/img /tmp/rbd_import_export_59341/img2 2021-07-31T02:25:36.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:36 smithi045 conmon[33655]: cluster 2021-07-31T02:25:34.341384+0000 mgr.x (mgr 2021-07-31T02:25:36.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:36 smithi045 conmon[33655]: .24542) 1646 : cluster [DBG] pgmap v2024: 9 pgs: 9 active+clean; 577 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 148 KiB/s rd, 64 KiB/s wr, 58 op/s 2021-07-31T02:25:36.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:36 smithi045 conmon[33655]: audit 2021-07-31T02:25:35.240340+0000 mgr.x (mgr.24542) 1647 : audit 2021-07-31T02:25:36.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:36 smithi045 conmon[33655]: [DBG] from='client.26371 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:25:36.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:36 smithi045 conmon[33655]: audit 2021-07-31T02:25:36.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:36 smithi045 conmon[33655]: 2021-07-31 2021-07-31T02:25:36.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:36 smithi045 conmon[33655]: T02:25:35.876169+0000 2021-07-31T02:25:36.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:36 smithi045 conmon[33655]: mon.a (mon.0) 2021-07-31T02:25:36.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:36 smithi045 conmon[33655]: 1914 : audit [DBG] from='client.? 172.21.15.45:0/505434814' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:25:36.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:36 smithi045 conmon[33655]: 2021-07-31T02:25:36.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:36 smithi045 conmon[39718]: cluster 2021-07-31T02:25: 2021-07-31T02:25:36.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:36 smithi045 conmon[39718]: 34.341384+0000 mgr.x (mgr.24542) 1646 : cluster [DBG] pgmap v2024: 9 pgs: 9 active+clean; 577 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 148 KiB/s rd, 64 KiB/s wr, 58 op/s 2021-07-31T02:25:36.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:36 smithi045 conmon[39718]: audit 2021- 2021-07-31T02:25:36.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:36 smithi045 conmon[39718]: 07-31T02:25: 2021-07-31T02:25:36.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:36 smithi045 conmon[39718]: 35.240340+0000 mgr.x (mgr 2021-07-31T02:25:36.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:36 smithi045 conmon[39718]: .24542) 1647 : audit [DBG] 2021-07-31T02:25:36.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:36 smithi045 conmon[39718]: from='client.26371 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:25:36.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:36 smithi045 conmon[39718]: audit 2021-07-31T02:25:36.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:36 smithi045 conmon[39718]: 2021-07-31 2021-07-31T02:25:36.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:36 smithi045 conmon[39718]: T02:25:35.876169 2021-07-31T02:25:36.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:36 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2021-07-31T02:25:36.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:36 smithi045 conmon[39718]: 1914 : audit [DBG] from='client.? 172.21.15.45:0/505434814' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:25:37.035 INFO:tasks.workunit.client.1.smithi154.stderr:+ cmp /tmp/rbd_import_export_59341/img /tmp/rbd_import_export_59341/img3 2021-07-31T02:25:37.440 INFO:tasks.workunit.client.1.smithi154.stderr:+ rm /tmp/rbd_import_export_59341/img /tmp/rbd_import_export_59341/img2 /tmp/rbd_import_export_59341/img3 2021-07-31T02:25:37.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:37 smithi154 conmon[34553]: audit 2021-07-31T02:25:35.526498+0000 2021-07-31T02:25:37.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:37 smithi154 conmon[34553]: mgr.x (mgr.24542) 1648 : audit [DBG] from='client.16467 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:25:37.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:37 smithi154 conmon[34553]: cluster 2021-07-31T02 2021-07-31T02:25:37.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:37 smithi154 conmon[34553]: :25:37.204042+0000 mon.a (mon.0) 1915 : cluster [DBG] osdmap e1028: 8 total, 8 up, 8 in 2021-07-31T02:25:37.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:37 smithi045 conmon[39718]: audit 2021-07-31T02:25:35.526498+0000 mgr.x (mgr. 2021-07-31T02:25:37.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:37 smithi045 conmon[39718]: 24542) 1648 : audit [DBG] from='client.16467 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:25:37.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:37 smithi045 conmon[39718]: cluster 2021-07-31T02:25:37.204042+0000 mon.a ( 2021-07-31T02:25:37.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:37 smithi045 conmon[39718]: mon.0) 1915 : cluster [DBG] osdmap e1028: 8 total, 8 up, 8 in 2021-07-31T02:25:37.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:37 smithi045 conmon[33655]: audit 2021-07-31T02:25:35.526498+0000 2021-07-31T02:25:37.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:37 smithi045 conmon[33655]: mgr.x (mgr.24542) 1648 : audit [DBG] from='client.16467 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:25:37.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:37 smithi045 conmon[33655]: cluster 2021-07-31T02:25:37.204042+0000 mon.a (mon.0) 1915 : cluster 2021-07-31T02:25:37.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:37 smithi045 conmon[33655]: [DBG] osdmap e1028: 8 total, 8 up, 8 in 2021-07-31T02:25:37.662 INFO:tasks.workunit.client.1.smithi154.stderr:+ rbd help export 2021-07-31T02:25:37.662 INFO:tasks.workunit.client.1.smithi154.stderr:+ grep -q export-format 2021-07-31T02:25:37.681 INFO:tasks.workunit.client.1.smithi154.stderr:+ dd if=/bin/dd of=/tmp/rbd_import_export_59341/img bs=1k count=10 seek=100 2021-07-31T02:25:37.681 INFO:tasks.workunit.client.1.smithi154.stderr:10+0 records in 2021-07-31T02:25:37.682 INFO:tasks.workunit.client.1.smithi154.stderr:10+0 records out 2021-07-31T02:25:37.682 INFO:tasks.workunit.client.1.smithi154.stderr:10240 bytes (10 kB, 10 KiB) copied, 5.9878e-05 s, 171 MB/s 2021-07-31T02:25:37.682 INFO:tasks.workunit.client.1.smithi154.stderr:+ rbd import --new-format /tmp/rbd_import_export_59341/img testimg 2021-07-31T02:25:37.748 INFO:tasks.workunit.client.1.smithi154.stderr: Importing image: 100% complete...done. 2021-07-31T02:25:37.748 INFO:tasks.workunit.client.1.smithi154.stderr:rbd: --new-format is deprecated, use --image-format 2 2021-07-31T02:25:37.750 INFO:tasks.workunit.client.1.smithi154.stderr:+ rbd snap create testimg@snap 2021-07-31T02:25:38.261 INFO:tasks.workunit.client.1.smithi154.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2021-07-31T02:25:38.270 INFO:tasks.workunit.client.1.smithi154.stderr:+ rbd image-meta set testimg key1 value1 2021-07-31T02:25:38.322 INFO:tasks.workunit.client.1.smithi154.stderr:++ rbd image-meta list testimg 2021-07-31T02:25:38.362 INFO:tasks.workunit.client.1.smithi154.stderr:+ IMAGEMETA_BEFORE='There is 1 metadatum on this image: 2021-07-31T02:25:38.362 INFO:tasks.workunit.client.1.smithi154.stderr: 2021-07-31T02:25:38.362 INFO:tasks.workunit.client.1.smithi154.stderr:Key Value 2021-07-31T02:25:38.363 INFO:tasks.workunit.client.1.smithi154.stderr:key1 value1' 2021-07-31T02:25:38.363 INFO:tasks.workunit.client.1.smithi154.stderr:+ rbd export --export-format 2 testimg /tmp/rbd_import_export_59341/img_v2 2021-07-31T02:25:38.401 INFO:tasks.workunit.client.1.smithi154.stderr: Exporting image: 100% complete... Exporting image: 100% complete...done. 2021-07-31T02:25:38.404 INFO:tasks.workunit.client.1.smithi154.stderr:+ rbd import --export-format 2 /tmp/rbd_import_export_59341/img_v2 testimg_import 2021-07-31T02:25:38.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:38 smithi154 conmon[34553]: cluster 2021-07-31T02:25:36.341781+0000 mgr.x (mgr.24542) 2021-07-31T02:25:38.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:38 smithi154 conmon[34553]: 1649 : cluster [DBG] pgmap v2025: 9 pgs: 9 active+clean; 577 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 148 KiB/s rd, 64 KiB/s wr, 58 op/s 2021-07-31T02:25:38.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:38 smithi045 conmon[33655]: cluster 2021-07-31T02:25:36.341781+0000 mgr.x (mgr.24542) 1649 : cluster [DBG] 2021-07-31T02:25:38.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:38 smithi045 conmon[33655]: pgmap v2025: 9 pgs: 9 active+clean; 577 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 148 KiB/s rd, 64 KiB/s wr, 58 op/s 2021-07-31T02:25:38.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:38 smithi045 conmon[39718]: cluster 2021-07-31T02:25:36.341781+0000 mgr.x (mgr.24542) 1649 2021-07-31T02:25:38.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:38 smithi045 conmon[39718]: : cluster [DBG] pgmap v2025: 9 pgs: 9 active+clean; 577 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 148 KiB/s rd, 64 KiB/s wr, 58 op/s 2021-07-31T02:25:39.280 INFO:tasks.workunit.client.1.smithi154.stderr: Importing image: 50% complete... Importing image: 100% complete... Importing image: 100% complete...done. 2021-07-31T02:25:39.283 INFO:tasks.workunit.client.1.smithi154.stderr:+ rbd info testimg_import 2021-07-31T02:25:39.319 INFO:tasks.workunit.client.1.smithi154.stdout:rbd image 'testimg_import': 2021-07-31T02:25:39.319 INFO:tasks.workunit.client.1.smithi154.stdout: size 110 KiB in 1 objects 2021-07-31T02:25:39.319 INFO:tasks.workunit.client.1.smithi154.stdout: order 22 (4 MiB objects) 2021-07-31T02:25:39.320 INFO:tasks.workunit.client.1.smithi154.stdout: snapshot_count: 1 2021-07-31T02:25:39.320 INFO:tasks.workunit.client.1.smithi154.stdout: id: 6521e2089e64 2021-07-31T02:25:39.320 INFO:tasks.workunit.client.1.smithi154.stdout: block_name_prefix: rbd_data.6521e2089e64 2021-07-31T02:25:39.320 INFO:tasks.workunit.client.1.smithi154.stdout: format: 2 2021-07-31T02:25:39.321 INFO:tasks.workunit.client.1.smithi154.stdout: features: layering, exclusive-lock, object-map, fast-diff, deep-flatten 2021-07-31T02:25:39.321 INFO:tasks.workunit.client.1.smithi154.stdout: op_features: 2021-07-31T02:25:39.321 INFO:tasks.workunit.client.1.smithi154.stdout: flags: 2021-07-31T02:25:39.321 INFO:tasks.workunit.client.1.smithi154.stdout: create_timestamp: Sat Jul 31 02:25:38 2021 2021-07-31T02:25:39.322 INFO:tasks.workunit.client.1.smithi154.stdout: access_timestamp: Sat Jul 31 02:25:38 2021 2021-07-31T02:25:39.322 INFO:tasks.workunit.client.1.smithi154.stdout: modify_timestamp: Sat Jul 31 02:25:38 2021 2021-07-31T02:25:39.323 INFO:tasks.workunit.client.1.smithi154.stderr:+ rbd info testimg_import@snap 2021-07-31T02:25:39.356 INFO:tasks.workunit.client.1.smithi154.stdout:rbd image 'testimg_import': 2021-07-31T02:25:39.356 INFO:tasks.workunit.client.1.smithi154.stdout: size 110 KiB in 1 objects 2021-07-31T02:25:39.356 INFO:tasks.workunit.client.1.smithi154.stdout: order 22 (4 MiB objects) 2021-07-31T02:25:39.357 INFO:tasks.workunit.client.1.smithi154.stdout: snapshot_count: 1 2021-07-31T02:25:39.357 INFO:tasks.workunit.client.1.smithi154.stdout: id: 6521e2089e64 2021-07-31T02:25:39.357 INFO:tasks.workunit.client.1.smithi154.stdout: block_name_prefix: rbd_data.6521e2089e64 2021-07-31T02:25:39.357 INFO:tasks.workunit.client.1.smithi154.stdout: format: 2 2021-07-31T02:25:39.358 INFO:tasks.workunit.client.1.smithi154.stdout: features: layering, exclusive-lock, object-map, fast-diff, deep-flatten 2021-07-31T02:25:39.358 INFO:tasks.workunit.client.1.smithi154.stdout: op_features: 2021-07-31T02:25:39.359 INFO:tasks.workunit.client.1.smithi154.stdout: flags: 2021-07-31T02:25:39.359 INFO:tasks.workunit.client.1.smithi154.stdout: create_timestamp: Sat Jul 31 02:25:38 2021 2021-07-31T02:25:39.359 INFO:tasks.workunit.client.1.smithi154.stdout: access_timestamp: Sat Jul 31 02:25:38 2021 2021-07-31T02:25:39.360 INFO:tasks.workunit.client.1.smithi154.stdout: modify_timestamp: Sat Jul 31 02:25:38 2021 2021-07-31T02:25:39.360 INFO:tasks.workunit.client.1.smithi154.stdout: protected: False 2021-07-31T02:25:39.361 INFO:tasks.workunit.client.1.smithi154.stderr:++ rbd image-meta list testimg_import 2021-07-31T02:25:39.398 INFO:tasks.workunit.client.1.smithi154.stderr:+ IMAGEMETA_AFTER='There is 1 metadatum on this image: 2021-07-31T02:25:39.399 INFO:tasks.workunit.client.1.smithi154.stderr: 2021-07-31T02:25:39.399 INFO:tasks.workunit.client.1.smithi154.stderr:Key Value 2021-07-31T02:25:39.399 INFO:tasks.workunit.client.1.smithi154.stderr:key1 value1' 2021-07-31T02:25:39.399 INFO:tasks.workunit.client.1.smithi154.stderr:+ '[' 'There is 1 metadatum on this image: 2021-07-31T02:25:39.400 INFO:tasks.workunit.client.1.smithi154.stderr: 2021-07-31T02:25:39.400 INFO:tasks.workunit.client.1.smithi154.stderr:Key Value 2021-07-31T02:25:39.400 INFO:tasks.workunit.client.1.smithi154.stderr:key1 value1' = 'There is 1 metadatum on this image: 2021-07-31T02:25:39.400 INFO:tasks.workunit.client.1.smithi154.stderr: 2021-07-31T02:25:39.401 INFO:tasks.workunit.client.1.smithi154.stderr:Key Value 2021-07-31T02:25:39.401 INFO:tasks.workunit.client.1.smithi154.stderr:key1 value1' ']' 2021-07-31T02:25:39.401 INFO:tasks.workunit.client.1.smithi154.stderr:+ rbd export testimg_import /tmp/rbd_import_export_59341/img_import 2021-07-31T02:25:39.431 INFO:tasks.workunit.client.1.smithi154.stderr: Exporting image: 100% complete...done. 2021-07-31T02:25:39.433 INFO:tasks.workunit.client.1.smithi154.stderr:+ compare_files_and_ondisk_sizes /tmp/rbd_import_export_59341/img /tmp/rbd_import_export_59341/img_import 2021-07-31T02:25:39.434 INFO:tasks.workunit.client.1.smithi154.stderr:+ cmp -l /tmp/rbd_import_export_59341/img /tmp/rbd_import_export_59341/img_import 2021-07-31T02:25:39.434 INFO:tasks.workunit.client.1.smithi154.stderr:++ stat /tmp/rbd_import_export_59341/img --format %b 2021-07-31T02:25:39.435 INFO:tasks.workunit.client.1.smithi154.stderr:+ origsize=24 2021-07-31T02:25:39.435 INFO:tasks.workunit.client.1.smithi154.stderr:++ stat /tmp/rbd_import_export_59341/img_import --format %b 2021-07-31T02:25:39.436 INFO:tasks.workunit.client.1.smithi154.stderr:+ exportsize=224 2021-07-31T02:25:39.437 INFO:tasks.workunit.client.1.smithi154.stderr:+ difference=200 2021-07-31T02:25:39.437 INFO:tasks.workunit.client.1.smithi154.stderr:+ difference=200 2021-07-31T02:25:39.437 INFO:tasks.workunit.client.1.smithi154.stderr:+ test 200 -ge 0 -a 200 -lt 4096 2021-07-31T02:25:39.437 INFO:tasks.workunit.client.1.smithi154.stderr:+ rbd export testimg@snap /tmp/rbd_import_export_59341/img_snap 2021-07-31T02:25:39.471 INFO:tasks.workunit.client.1.smithi154.stderr: Exporting image: 100% complete...done. 2021-07-31T02:25:39.473 INFO:tasks.workunit.client.1.smithi154.stderr:+ rbd export testimg_import@snap /tmp/rbd_import_export_59341/img_snap_import 2021-07-31T02:25:39.507 INFO:tasks.workunit.client.1.smithi154.stderr: Exporting image: 100% complete...done. 2021-07-31T02:25:39.509 INFO:tasks.workunit.client.1.smithi154.stderr:+ compare_files_and_ondisk_sizes /tmp/rbd_import_export_59341/img_snap /tmp/rbd_import_export_59341/img_snap_import 2021-07-31T02:25:39.509 INFO:tasks.workunit.client.1.smithi154.stderr:+ cmp -l /tmp/rbd_import_export_59341/img_snap /tmp/rbd_import_export_59341/img_snap_import 2021-07-31T02:25:39.510 INFO:tasks.workunit.client.1.smithi154.stderr:++ stat /tmp/rbd_import_export_59341/img_snap --format %b 2021-07-31T02:25:39.511 INFO:tasks.workunit.client.1.smithi154.stderr:+ origsize=224 2021-07-31T02:25:39.511 INFO:tasks.workunit.client.1.smithi154.stderr:++ stat /tmp/rbd_import_export_59341/img_snap_import --format %b 2021-07-31T02:25:39.512 INFO:tasks.workunit.client.1.smithi154.stderr:+ exportsize=224 2021-07-31T02:25:39.513 INFO:tasks.workunit.client.1.smithi154.stderr:+ difference=0 2021-07-31T02:25:39.513 INFO:tasks.workunit.client.1.smithi154.stderr:+ difference=0 2021-07-31T02:25:39.513 INFO:tasks.workunit.client.1.smithi154.stderr:+ test 0 -ge 0 -a 0 -lt 4096 2021-07-31T02:25:39.513 INFO:tasks.workunit.client.1.smithi154.stderr:+ rm /tmp/rbd_import_export_59341/img_v2 2021-07-31T02:25:39.514 INFO:tasks.workunit.client.1.smithi154.stderr:+ rm /tmp/rbd_import_export_59341/img_import 2021-07-31T02:25:39.514 INFO:tasks.workunit.client.1.smithi154.stderr:+ rm /tmp/rbd_import_export_59341/img_snap 2021-07-31T02:25:39.515 INFO:tasks.workunit.client.1.smithi154.stderr:+ rm /tmp/rbd_import_export_59341/img_snap_import 2021-07-31T02:25:39.515 INFO:tasks.workunit.client.1.smithi154.stderr:+ rbd snap rm testimg_import@snap 2021-07-31T02:25:39.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:39 smithi045 conmon[33655]: cluster 2021-07-31T02:25:38.249923+0000 mon.a (mon.0) 1916 : cluster [DBG] 2021-07-31T02:25:39.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:39 smithi045 conmon[33655]: osdmap e1029: 8 total, 8 up, 8 in 2021-07-31T02:25:39.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:39 smithi045 conmon[39718]: cluster 2021-07-31T02:25:38.249923+0000 mon.a (mon 2021-07-31T02:25:39.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:39 smithi045 conmon[39718]: .0) 1916 : cluster [DBG] osdmap e1029: 8 total, 8 up, 8 in 2021-07-31T02:25:39.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:39 smithi154 conmon[34553]: cluster 2021-07-31T02:25:38.249923+0000 mon.a (mon.0) 1916 : cluster [DBG] osdmap e1029: 8 total, 8 up, 8 in 2021-07-31T02:25:40.265 INFO:tasks.workunit.client.1.smithi154.stderr: Removing snap: 100% complete...done. 2021-07-31T02:25:40.275 INFO:tasks.workunit.client.1.smithi154.stderr:+ rbd remove testimg_import 2021-07-31T02:25:40.406 INFO:tasks.workunit.client.1.smithi154.stderr: Removing image: 100% complete...done. 2021-07-31T02:25:40.408 INFO:tasks.workunit.client.1.smithi154.stderr:+ rbd snap rm testimg@snap 2021-07-31T02:25:40.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:40 smithi045 conmon[39718]: cluster 2021-07-31T02:25:38.342128+0000 mgr.x (mgr.24542 2021-07-31T02:25:40.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:40 smithi045 conmon[39718]: ) 1650 : cluster [DBG] pgmap v2028: 9 pgs: 9 active+clean; 577 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 194 KiB/s rd, 84 KiB/s wr, 50 op/s 2021-07-31T02:25:40.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:40 smithi045 conmon[39718]: cluster 2021-07-31T02:25:39.255608+0000 mon.a (mon.0) 1917 : cluster 2021-07-31T02:25:40.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:40 smithi045 conmon[39718]: [DBG] osdmap e1030: 8 total, 8 up, 8 in 2021-07-31T02:25:40.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:40 smithi045 conmon[33655]: cluster 2021-07-31T02:25:38.342128 2021-07-31T02:25:40.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:40 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 1650 : cluster [DBG] pgmap v2028: 9 pgs: 9 active+clean; 577 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 194 KiB/s rd, 84 KiB/s wr, 50 op/s 2021-07-31T02:25:40.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:40 smithi045 conmon[33655]: cluster 2021-07 2021-07-31T02:25:40.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:40 smithi045 conmon[33655]: -31T02:25:39.255608+0000 mon.a (mon.0) 1917 : cluster [DBG] osdmap e1030: 8 total, 8 up, 8 in 2021-07-31T02:25:40.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:40 smithi045 conmon[33655]: 2021-07-31T02:25:40.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:40 smithi154 conmon[34553]: cluster 2021-07-31T02:25:38.342128+0000 mgr.x (mgr.24542) 1650 : cluster [DBG] pgmap v2028: 9 pgs: 9 active+clean; 577 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 194 KiB/s rd, 84 KiB/s wr, 50 op/s 2021-07-31T02:25:40.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:40 smithi154 conmon[34553]: cluster 2021-07-31T02:25:39.255608+0000 mon.a (mon.0) 1917 : cluster [DBG] osdmap e1030: 8 total, 8 up, 8 in 2021-07-31T02:25:41.278 INFO:tasks.workunit.client.1.smithi154.stderr: Removing snap: 100% complete...done. 2021-07-31T02:25:41.289 INFO:tasks.workunit.client.1.smithi154.stderr:+ rbd rm testimg 2021-07-31T02:25:41.416 INFO:tasks.workunit.client.1.smithi154.stderr: Removing image: 100% complete...done. 2021-07-31T02:25:41.418 INFO:tasks.workunit.client.1.smithi154.stderr:+ rbd import --order 20 /tmp/rbd_import_export_59341/img testimg 2021-07-31T02:25:41.494 INFO:tasks.workunit.client.1.smithi154.stderr: Importing image: 100% complete...done. 2021-07-31T02:25:41.495 INFO:tasks.workunit.client.1.smithi154.stderr:rbd: --order is deprecated, use --object-size 2021-07-31T02:25:41.496 INFO:tasks.workunit.client.1.smithi154.stderr:+ rbd export --export-format 2 testimg /tmp/rbd_import_export_59341/img_v2 2021-07-31T02:25:41.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:41 smithi045 conmon[39718]: cluster 2021-07-31T02:25:40.261217+0000 mon.a (mon 2021-07-31T02:25:41.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:41 smithi045 conmon[39718]: .0) 1918 : cluster [DBG] osdmap e1031: 8 total, 8 up, 8 in 2021-07-31T02:25:41.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:41 smithi045 conmon[33655]: cluster 2021-07-31T02:25:40.261217+0000 mon.a (mon.0) 1918 : cluster 2021-07-31T02:25:41.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:41 smithi045 conmon[33655]: [DBG] osdmap e1031: 8 total, 8 up, 8 in 2021-07-31T02:25:41.543 INFO:tasks.workunit.client.1.smithi154.stderr: Exporting image: 100% complete... Exporting image: 100% complete...done. 2021-07-31T02:25:41.546 INFO:tasks.workunit.client.1.smithi154.stderr:+ rbd import --export-format 2 /tmp/rbd_import_export_59341/img_v2 testimg_import 2021-07-31T02:25:41.635 INFO:tasks.workunit.client.1.smithi154.stderr: Importing image: 100% complete... Importing image: 100% complete...done. 2021-07-31T02:25:41.638 INFO:tasks.workunit.client.1.smithi154.stderr:+ rbd info testimg_import 2021-07-31T02:25:41.639 INFO:tasks.workunit.client.1.smithi154.stderr:+ grep order 2021-07-31T02:25:41.639 INFO:tasks.workunit.client.1.smithi154.stderr:+ awk '{print $2}' 2021-07-31T02:25:41.639 INFO:tasks.workunit.client.1.smithi154.stderr:+ grep 20 2021-07-31T02:25:41.681 INFO:tasks.workunit.client.1.smithi154.stdout:20 2021-07-31T02:25:41.682 INFO:tasks.workunit.client.1.smithi154.stderr:+ rm /tmp/rbd_import_export_59341/img_v2 2021-07-31T02:25:41.682 INFO:tasks.workunit.client.1.smithi154.stderr:+ rbd remove testimg_import 2021-07-31T02:25:41.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:41 smithi154 conmon[34553]: cluster 2021-07-31T02:25:40.261217+0000 mon.a (mon.0) 1918 : cluster [DBG] osdmap e1031: 8 total, 8 up, 8 in 2021-07-31T02:25:41.824 INFO:tasks.workunit.client.1.smithi154.stderr: Removing image: 100% complete...done. 2021-07-31T02:25:41.826 INFO:tasks.workunit.client.1.smithi154.stderr:+ rbd remove testimg 2021-07-31T02:25:41.976 INFO:tasks.workunit.client.1.smithi154.stderr: Removing image: 100% complete...done. 2021-07-31T02:25:41.978 INFO:tasks.workunit.client.1.smithi154.stderr:+ rbd import --image-feature layering /tmp/rbd_import_export_59341/img testimg 2021-07-31T02:25:42.027 INFO:tasks.workunit.client.1.smithi154.stderr: Importing image: 100% complete...done. 2021-07-31T02:25:42.030 INFO:tasks.workunit.client.1.smithi154.stderr:++ rbd info testimg 2021-07-31T02:25:42.030 INFO:tasks.workunit.client.1.smithi154.stderr:++ grep features 2021-07-31T02:25:42.065 INFO:tasks.workunit.client.1.smithi154.stderr:+ FEATURES_BEFORE=' features: layering 2021-07-31T02:25:42.065 INFO:tasks.workunit.client.1.smithi154.stderr: op_features: ' 2021-07-31T02:25:42.065 INFO:tasks.workunit.client.1.smithi154.stderr:+ rbd export --export-format 2 testimg /tmp/rbd_import_export_59341/img_v2 2021-07-31T02:25:42.099 INFO:tasks.workunit.client.1.smithi154.stderr: Exporting image: 100% complete... Exporting image: 100% complete...done. 2021-07-31T02:25:42.101 INFO:tasks.workunit.client.1.smithi154.stderr:+ rbd import --export-format 2 /tmp/rbd_import_export_59341/img_v2 testimg_import 2021-07-31T02:25:42.178 INFO:tasks.workunit.client.1.smithi154.stderr: Importing image: 100% complete... Importing image: 100% complete...done. 2021-07-31T02:25:42.181 INFO:tasks.workunit.client.1.smithi154.stderr:++ rbd info testimg_import 2021-07-31T02:25:42.182 INFO:tasks.workunit.client.1.smithi154.stderr:++ grep features 2021-07-31T02:25:42.230 INFO:tasks.workunit.client.1.smithi154.stderr:+ FEATURES_AFTER=' features: layering 2021-07-31T02:25:42.230 INFO:tasks.workunit.client.1.smithi154.stderr: op_features: ' 2021-07-31T02:25:42.230 INFO:tasks.workunit.client.1.smithi154.stderr:+ '[' ' features: layering 2021-07-31T02:25:42.231 INFO:tasks.workunit.client.1.smithi154.stderr: op_features: ' '!=' ' features: layering 2021-07-31T02:25:42.231 INFO:tasks.workunit.client.1.smithi154.stderr: op_features: ' ']' 2021-07-31T02:25:42.231 INFO:tasks.workunit.client.1.smithi154.stderr:+ rm /tmp/rbd_import_export_59341/img_v2 2021-07-31T02:25:42.231 INFO:tasks.workunit.client.1.smithi154.stderr:+ rbd remove testimg_import 2021-07-31T02:25:42.422 INFO:tasks.workunit.client.1.smithi154.stderr: Removing image: 100% complete...done. 2021-07-31T02:25:42.424 INFO:tasks.workunit.client.1.smithi154.stderr:+ rbd remove testimg 2021-07-31T02:25:42.508 INFO:tasks.workunit.client.1.smithi154.stderr: Removing image: 100% complete...done. 2021-07-31T02:25:42.511 INFO:tasks.workunit.client.1.smithi154.stderr:+ rbd import --stripe-count 1000 --stripe-unit 4096 /tmp/rbd_import_export_59341/img testimg 2021-07-31T02:25:42.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:42 smithi045 conmon[39718]: cluster 2021-07-31T02:25 2021-07-31T02:25:42.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:42 smithi045 conmon[39718]: :40.342492+0000 mgr.x (mgr.24542) 1651 : cluster [DBG] pgmap v2031: 9 pgs: 9 active+clean; 797 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 397 KiB/s rd, 142 KiB/s wr, 208 op/s 2021-07-31T02:25:42.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:42 smithi045 conmon[39718]: cluster 2021 2021-07-31T02:25:42.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:42 smithi045 conmon[39718]: -07-31T02:25:41.266863 2021-07-31T02:25:42.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:42 smithi045 conmon[39718]: +0000 mon.a (mon.0) 1919 : 2021-07-31T02:25:42.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:42 smithi045 conmon[39718]: cluster [DBG] osdmap e1032: 8 total, 8 up, 8 in 2021-07-31T02:25:42.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:42 smithi045 conmon[33655]: cluster 2021-07-31T02:25:40.342492 2021-07-31T02:25:42.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:42 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 1651 : cluster [DBG] pgmap v2031: 9 pgs: 9 active+clean; 797 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 397 KiB/s rd, 142 KiB/s wr, 208 op/s 2021-07-31T02:25:42.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:42 smithi045 conmon[33655]: cluster 2021-07 2021-07-31T02:25:42.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:42 smithi045 conmon[33655]: -31T02:25:41.266863+0000 mon.a (mon.0) 1919 2021-07-31T02:25:42.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:42 smithi045 conmon[33655]: : cluster [DBG] osdmap e1032: 8 total, 8 up, 8 in 2021-07-31T02:25:42.583 INFO:tasks.workunit.client.1.smithi154.stderr: Importing image: 100% complete...done. 2021-07-31T02:25:42.585 INFO:tasks.workunit.client.1.smithi154.stderr:+ rbd export --export-format 2 testimg /tmp/rbd_import_export_59341/img_v2 2021-07-31T02:25:42.622 INFO:tasks.workunit.client.1.smithi154.stderr: Exporting image: 100% complete... Exporting image: 100% complete...done. 2021-07-31T02:25:42.625 INFO:tasks.workunit.client.1.smithi154.stderr:+ rbd import --export-format 2 /tmp/rbd_import_export_59341/img_v2 testimg_import 2021-07-31T02:25:42.725 INFO:tasks.workunit.client.1.smithi154.stderr: Importing image: 100% complete... Importing image: 100% complete...done. 2021-07-31T02:25:42.728 INFO:tasks.workunit.client.1.smithi154.stderr:+ rbd info testimg_import 2021-07-31T02:25:42.728 INFO:tasks.workunit.client.1.smithi154.stderr:+ grep 'stripe unit' 2021-07-31T02:25:42.728 INFO:tasks.workunit.client.1.smithi154.stderr:+ grep -Ei '(4 KiB|4096)' 2021-07-31T02:25:42.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:42 smithi154 conmon[34553]: cluster 2021-07-31T02:25:40.342492+0000 mgr.x (mgr.24542) 1651 : cluster [DBG] pgmap v2031: 9 pgs: 9 active+clean; 797 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 397 KiB/s rd, 142 KiB/s wr, 208 op/s 2021-07-31T02:25:42.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:42 smithi154 conmon[34553]: cluster 2021-07-31T02:25:41 2021-07-31T02:25:42.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:42 smithi154 conmon[34553]: .266863+0000 mon.a (mon.0) 1919 : cluster [DBG] osdmap e1032: 8 total, 8 up, 8 in 2021-07-31T02:25:42.771 INFO:tasks.workunit.client.1.smithi154.stdout: stripe unit: 4 KiB 2021-07-31T02:25:42.771 INFO:tasks.workunit.client.1.smithi154.stderr:+ rbd info testimg_import 2021-07-31T02:25:42.772 INFO:tasks.workunit.client.1.smithi154.stderr:+ grep 'stripe count' 2021-07-31T02:25:42.772 INFO:tasks.workunit.client.1.smithi154.stderr:+ awk '{print $3}' 2021-07-31T02:25:42.772 INFO:tasks.workunit.client.1.smithi154.stderr:+ grep 1000 2021-07-31T02:25:42.813 INFO:tasks.workunit.client.1.smithi154.stdout:1000 2021-07-31T02:25:42.814 INFO:tasks.workunit.client.1.smithi154.stderr:+ rm /tmp/rbd_import_export_59341/img_v2 2021-07-31T02:25:42.814 INFO:tasks.workunit.client.1.smithi154.stderr:+ rbd remove testimg_import 2021-07-31T02:25:42.965 INFO:tasks.workunit.client.1.smithi154.stderr: Removing image: 3% complete... Removing image: 7% complete... Removing image: 10% complete... Removing image: 14% complete... Removing image: 17% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 32% complete... Removing image: 35% complete... Removing image: 39% complete... Removing image: 42% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 57% complete... Removing image: 60% complete... Removing image: 64% complete... Removing image: 67% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 82% complete... Removing image: 85% complete... Removing image: 89% complete... Removing image: 92% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2021-07-31T02:25:42.967 INFO:tasks.workunit.client.1.smithi154.stderr:+ rbd remove testimg 2021-07-31T02:25:43.085 INFO:tasks.workunit.client.1.smithi154.stderr: Removing image: 3% complete... Removing image: 7% complete... Removing image: 10% complete... Removing image: 14% complete... Removing image: 17% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 32% complete... Removing image: 35% complete... Removing image: 39% complete... Removing image: 42% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 57% complete... Removing image: 60% complete... Removing image: 64% complete... Removing image: 67% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 82% complete... Removing image: 85% complete... Removing image: 89% complete... Removing image: 92% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2021-07-31T02:25:43.088 INFO:tasks.workunit.client.1.smithi154.stderr:+ rbd import --image-format=2 /tmp/rbd_import_export_59341/img testimg 2021-07-31T02:25:43.151 INFO:tasks.workunit.client.1.smithi154.stderr: Importing image: 100% complete...done. 2021-07-31T02:25:43.153 INFO:tasks.workunit.client.1.smithi154.stderr:+ rbd snap create testimg@snap1 2021-07-31T02:25:43.287 INFO:tasks.workunit.client.1.smithi154.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2021-07-31T02:25:43.296 INFO:tasks.workunit.client.1.smithi154.stderr:+ rbd snap create testimg@snap2 2021-07-31T02:25:44.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:25:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:25:43.575Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:25:44.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:25:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:25:43.575Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:25:44.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:25:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:25:43.576Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:25:44.295 INFO:tasks.workunit.client.1.smithi154.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2021-07-31T02:25:44.304 INFO:tasks.workunit.client.1.smithi154.stderr:+ rbd snap protect testimg@snap2 2021-07-31T02:25:44.353 INFO:tasks.workunit.client.1.smithi154.stderr:+ rbd export --export-format 2 testimg /tmp/rbd_import_export_59341/snap_protect 2021-07-31T02:25:44.392 INFO:tasks.workunit.client.1.smithi154.stderr: Exporting image: 33% complete... Exporting image: 100% complete... Exporting image: 100% complete...done. 2021-07-31T02:25:44.395 INFO:tasks.workunit.client.1.smithi154.stderr:+ rbd import --export-format 2 /tmp/rbd_import_export_59341/snap_protect testimg_import 2021-07-31T02:25:44.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:44 smithi045 conmon[39718]: cluster 2021-07-31T02:25:42.342841+0000 mgr.x (mgr.24542) 1652 : 2021-07-31T02:25:44.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:44 smithi045 conmon[39718]: cluster [DBG] pgmap v2033: 9 pgs: 9 active+clean; 797 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 388 KiB/s rd, 138 KiB/s wr, 203 op/s 2021-07-31T02:25:44.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:44 smithi045 conmon[39718]: cluster 2021-07-31T02:25:43.271419+0000 mon.a (mon.0) 1920 : cluster [DBG] osdmap e1033: 8 total, 8 up, 8 in 2021-07-31T02:25:44.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:44 smithi045 conmon[33655]: cluster 2021-07-31T02:25:42.342841+0000 mgr.x (mgr.24542) 1652 : cluster [DBG] 2021-07-31T02:25:44.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:44 smithi045 conmon[33655]: pgmap v2033: 9 pgs: 9 active+clean; 797 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 388 KiB/s rd, 138 KiB/s wr, 203 op/s 2021-07-31T02:25:44.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:44 smithi045 conmon[33655]: cluster 2021-07-31T02:25:43.271419 2021-07-31T02:25:44.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:44 smithi045 conmon[33655]: +0000 mon.a (mon.0) 1920 : cluster [DBG] osdmap e1033: 8 total, 8 up, 8 in 2021-07-31T02:25:44.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:44 smithi154 conmon[34553]: cluster 2021-07-31T02:25:42.342841+0000 mgr.x (mgr.24542) 1652 : cluster [DBG] pgmap v2033: 9 pgs: 9 active+clean; 797 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 388 KiB/s rd, 138 KiB/s wr, 203 op/s 2021-07-31T02:25:44.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:44 smithi154 conmon[34553]: cluster 2021-07-31T02:25:43.271419+0000 mon.a (mon.0) 1920 : cluster [DBG] osdmap e1033: 8 total, 8 up, 8 in 2021-07-31T02:25:45.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:45 smithi154 conmon[34553]: cluster 2021-07-31T02:25:44.277792+0000 mon.a (mon.0) 1921 : cluster [DBG] osdmap e1034: 8 total, 8 up, 8 in 2021-07-31T02:25:45.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:45 smithi154 conmon[34553]: 2021-07-31T02:25:45.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:45 smithi045 conmon[33655]: cluster 2021-07-31T02:25:44.277792+0000 mon.a (mon.0 2021-07-31T02:25:45.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:45 smithi045 conmon[33655]: ) 1921 : cluster [DBG] osdmap e1034: 8 total, 8 up, 8 in 2021-07-31T02:25:45.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:45 smithi045 conmon[39718]: cluster 2021-07-31T02:25:44.277792+0000 mon.a (mon.0) 1921 : 2021-07-31T02:25:45.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:45 smithi045 conmon[39718]: cluster [DBG] osdmap e1034: 8 total, 8 up, 8 in 2021-07-31T02:25:46.334 INFO:tasks.workunit.client.1.smithi154.stderr: Importing image: 33% complete... Importing image: 66% complete... Importing image: 100% complete... Importing image: 100% complete...done. 2021-07-31T02:25:46.337 INFO:tasks.workunit.client.1.smithi154.stderr:+ rbd info testimg_import@snap1 2021-07-31T02:25:46.337 INFO:tasks.workunit.client.1.smithi154.stderr:+ grep 'protected: False' 2021-07-31T02:25:46.382 INFO:tasks.workunit.client.1.smithi154.stdout: protected: False 2021-07-31T02:25:46.383 INFO:tasks.workunit.client.1.smithi154.stderr:+ rbd info testimg_import@snap2 2021-07-31T02:25:46.383 INFO:tasks.workunit.client.1.smithi154.stderr:+ grep 'protected: True' 2021-07-31T02:25:46.426 INFO:tasks.workunit.client.1.smithi154.stdout: protected: True 2021-07-31T02:25:46.427 INFO:tasks.workunit.client.1.smithi154.stderr:+ rm /tmp/rbd_import_export_59341/snap_protect 2021-07-31T02:25:46.427 INFO:tasks.workunit.client.1.smithi154.stderr:+ rbd snap unprotect testimg@snap2 2021-07-31T02:25:46.470 INFO:tasks.workunit.client.1.smithi154.stderr:+ rbd snap unprotect testimg_import@snap2 2021-07-31T02:25:46.520 INFO:tasks.workunit.client.1.smithi154.stderr:+ rbd snap purge testimg 2021-07-31T02:25:46.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:46 smithi154 conmon[34553]: cluster 2021-07-31T02:25:44.343144+0000 mgr.x (mgr.24542) 1653 : cluster [DBG] pgmap v2036: 9 pgs: 9 active+clean; 687 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 442 KiB/s rd, 46 KiB/s wr, 581 op/s 2021-07-31T02:25:46.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:46 smithi154 conmon[34553]: cluster 2021-07-31T02:25:45.283367+0000 mon.a (mon.0) 1922 : cluster [DBG] osdmap e1035: 8 total, 8 up, 8 in 2021-07-31T02:25:46.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:46 smithi045 conmon[33655]: cluster 2021-07-31T02:25:44.343144 2021-07-31T02:25:46.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:46 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 1653 : cluster [DBG] pgmap v2036: 9 pgs: 9 active+clean; 687 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 442 KiB/s rd, 46 KiB/s wr, 581 op/s 2021-07-31T02:25:46.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:46 smithi045 conmon[33655]: cluster 2021-07-31T02:25:45.283367+0000 2021-07-31T02:25:46.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:46 smithi045 conmon[33655]: mon.a (mon.0) 1922 : cluster [DBG] osdmap e1035: 8 total, 8 up, 8 in 2021-07-31T02:25:46.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:46 smithi045 conmon[39718]: cluster 2021-07-31T02:25:44.343144+0000 mgr.x (mgr.24542) 2021-07-31T02:25:46.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:46 smithi045 conmon[39718]: 1653 : cluster [DBG] pgmap v2036: 9 pgs: 9 active+clean; 687 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 442 KiB/s rd, 46 KiB/s wr, 581 op/s 2021-07-31T02:25:46.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:46 smithi045 conmon[39718]: cluster 2021-07-31T02:25:45.283367+0000 mon.a (mon.0) 1922 : cluster [DBG] osdmap e1035: 8 total, 8 up, 8 in 2021-07-31T02:25:47.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:47 smithi154 conmon[34553]: cluster 2021-07-31T02:25:46.284956+0000 mon.a (mon.0) 1923 : cluster [DBG] osdmap e1036: 8 total, 8 up, 8 in 2021-07-31T02:25:47.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:47 smithi154 conmon[34553]: 2021-07-31T02:25:47.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:47 smithi154 conmon[34553]: cluster 2021-07-31T02:25:47.211936+0000 mon.a (mon.0) 1924 : cluster [DBG] osdmap e1037: 8 total, 8 up, 8 in 2021-07-31T02:25:47.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:47 smithi045 conmon[39718]: cluster 2021-07-31T02:25:46.284956+0000 mon.a ( 2021-07-31T02:25:47.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:47 smithi045 conmon[39718]: mon.0) 1923 : cluster [DBG] osdmap e1036: 8 total, 8 up, 8 in 2021-07-31T02:25:47.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:47 smithi045 conmon[39718]: cluster 2021-07-31T02:25:47.211936+0000 2021-07-31T02:25:47.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:47 smithi045 conmon[39718]: mon.a (mon.0) 1924 : cluster [DBG] osdmap e1037: 8 total, 8 up, 8 in 2021-07-31T02:25:47.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:47 smithi045 conmon[33655]: cluster 2021-07-31T02:25:46.284956+0000 mon.a (mon.0) 1923 : cluster 2021-07-31T02:25:47.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:47 smithi045 conmon[33655]: [DBG] osdmap e1036: 8 total, 8 up, 8 in 2021-07-31T02:25:47.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:47 smithi045 conmon[33655]: cluster 2021-07-31T02:25:47.211936+0000 mon.a (mon.0) 1924 : cluster [DBG] osdmap e1037: 8 total, 8 up, 8 in 2021-07-31T02:25:48.226 INFO:tasks.workunit.client.1.smithi154.stderr: Removing all snapshots: 50% complete... Removing all snapshots: 100% complete... Removing all snapshots: 100% complete...done. 2021-07-31T02:25:48.237 INFO:tasks.workunit.client.1.smithi154.stderr:+ rbd snap purge testimg_import 2021-07-31T02:25:48.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:48 smithi154 conmon[34553]: cluster 2021-07-31T02:25:46.343523+0000 mgr.x (mgr.24542) 2021-07-31T02:25:48.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:48 smithi154 conmon[34553]: 1654 : cluster [DBG] pgmap v2039: 9 pgs: 9 active+clean; 687 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 451 KiB/s rd, 47 KiB/s wr, 593 op/s 2021-07-31T02:25:48.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:48 smithi154 conmon[34553]: cluster 2021-07-31T02:25:48.217670+0000 mon.a (mon.0) 1925 : cluster [DBG] osdmap e1038: 8 total, 8 up, 8 in 2021-07-31T02:25:48.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:48 smithi045 conmon[39718]: cluster 2021-07-31T02:25:46. 2021-07-31T02:25:48.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:48 smithi045 conmon[39718]: 343523+0000 mgr.x (mgr.24542) 1654 : cluster [DBG] pgmap v2039: 9 pgs: 9 active+clean; 687 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 451 KiB/s rd, 47 KiB/s wr, 593 op/s 2021-07-31T02:25:48.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:48 smithi045 conmon[39718]: cluster 2021-07-31T02:25 2021-07-31T02:25:48.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:48 smithi045 conmon[39718]: :48.217670+0000 mon.a (mon 2021-07-31T02:25:48.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:48 smithi045 conmon[39718]: .0) 1925 : cluster [DBG] 2021-07-31T02:25:48.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:48 smithi045 conmon[39718]: osdmap e1038: 8 total, 8 up, 8 in 2021-07-31T02:25:48.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:48 smithi045 conmon[33655]: cluster 2021-07-31T02:25:46.343523 2021-07-31T02:25:48.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:48 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 1654 : cluster [DBG] pgmap v2039: 9 pgs: 9 active+clean; 687 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 451 KiB/s rd, 47 KiB/s wr, 593 op/s 2021-07-31T02:25:48.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:48 smithi045 conmon[33655]: cluster 2021-07-31T02:25:48.217670+0000 mon.a (mon. 2021-07-31T02:25:48.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:48 smithi045 conmon[33655]: 0) 1925 : cluster [DBG] osdmap e1038: 8 total, 8 up, 8 in 2021-07-31T02:25:50.325 INFO:tasks.workunit.client.1.smithi154.stderr: Removing all snapshots: 50% complete... Removing all snapshots: 100% complete... Removing all snapshots: 100% complete...done. 2021-07-31T02:25:50.338 INFO:tasks.workunit.client.1.smithi154.stderr:+ rbd remove testimg 2021-07-31T02:25:50.466 INFO:tasks.workunit.client.1.smithi154.stderr: Removing image: 100% complete...done. 2021-07-31T02:25:50.468 INFO:tasks.workunit.client.1.smithi154.stderr:+ rbd remove testimg_import 2021-07-31T02:25:50.603 INFO:tasks.workunit.client.1.smithi154.stderr: Removing image: 100% complete...done. 2021-07-31T02:25:50.606 INFO:tasks.workunit.client.1.smithi154.stderr:+ tiered=0 2021-07-31T02:25:50.606 INFO:tasks.workunit.client.1.smithi154.stderr:+ ceph osd dump 2021-07-31T02:25:50.606 INFO:tasks.workunit.client.1.smithi154.stderr:+ grep '^pool' 2021-07-31T02:25:50.606 INFO:tasks.workunit.client.1.smithi154.stderr:+ grep ''\''rbd'\''' 2021-07-31T02:25:50.607 INFO:tasks.workunit.client.1.smithi154.stderr:+ grep tier 2021-07-31T02:25:50.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:50 smithi154 conmon[34553]: cluster 2021-07-31T02:25:48.343895+0000 mgr.x (mgr.24542) 1655 : cluster [DBG] pgmap v2042: 9 pgs: 9 active+clean; 687 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:25:50.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:50 smithi154 conmon[34553]: cluster 2021-07-31T02 2021-07-31T02:25:50.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:50 smithi154 conmon[34553]: :25:49.306751+0000 mon.a (mon.0) 1926 : cluster [DBG] osdmap e1039: 8 total, 8 up, 8 in 2021-07-31T02:25:50.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:50 smithi045 conmon[33655]: cluster 2021-07-31T02:25:50.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:50 smithi045 conmon[33655]: 2021-07-31T02:25:48.343895+0000 mgr.x (mgr.24542) 1655 : cluster [DBG] pgmap v2042: 9 pgs: 9 active+clean; 687 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:25:50.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:50 smithi045 conmon[33655]: cluster 2021-07-31T02:25: 2021-07-31T02:25:50.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:50 smithi045 conmon[33655]: 49.306751+0000 mon.a (mon.0) 1926 : cluster [DBG] osdmap e1039: 8 total, 8 up, 8 in 2021-07-31T02:25:50.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:50 smithi045 conmon[39718]: cluster 2021-07-31T02:25:48.343895+0000 mgr.x (mgr.24542 2021-07-31T02:25:50.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:50 smithi045 conmon[39718]: ) 1655 : cluster [DBG] pgmap v2042: 9 pgs: 9 active+clean; 687 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:25:50.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:50 smithi045 conmon[39718]: cluster 2021-07-31T02:25:49.306751+0000 mon.a (mon. 2021-07-31T02:25:50.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:50 smithi045 conmon[39718]: 0) 1926 : cluster [DBG] osdmap e1039: 8 total, 8 up, 8 in 2021-07-31T02:25:50.973 INFO:tasks.workunit.client.1.smithi154.stderr:+ dd if=/dev/urandom bs=1M seek=1 count=1 of=/tmp/rbd_import_export_59341/sparse1 2021-07-31T02:25:50.979 INFO:tasks.workunit.client.1.smithi154.stderr:1+0 records in 2021-07-31T02:25:50.979 INFO:tasks.workunit.client.1.smithi154.stderr:1+0 records out 2021-07-31T02:25:50.979 INFO:tasks.workunit.client.1.smithi154.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00481415 s, 218 MB/s 2021-07-31T02:25:50.980 INFO:tasks.workunit.client.1.smithi154.stderr:+ dd if=/dev/urandom bs=1M count=1 of=/tmp/rbd_import_export_59341/sparse2 2021-07-31T02:25:50.984 INFO:tasks.workunit.client.1.smithi154.stderr:1+0 records in 2021-07-31T02:25:50.984 INFO:tasks.workunit.client.1.smithi154.stderr:1+0 records out 2021-07-31T02:25:50.985 INFO:tasks.workunit.client.1.smithi154.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00481876 s, 218 MB/s 2021-07-31T02:25:50.985 INFO:tasks.workunit.client.1.smithi154.stderr:+ truncate /tmp/rbd_import_export_59341/sparse2 -s 2M 2021-07-31T02:25:51.057 INFO:tasks.workunit.client.1.smithi154.stderr:+ rbd rm sparse1 2021-07-31T02:25:51.111 INFO:tasks.workunit.client.1.smithi154.stderr: Removing image: 0% complete...failed. 2021-07-31T02:25:51.111 INFO:tasks.workunit.client.1.smithi154.stderr:rbd: delete error: (2) No such file or directory 2021-07-31T02:25:51.113 INFO:tasks.workunit.client.1.smithi154.stderr:+ true 2021-07-31T02:25:51.113 INFO:tasks.workunit.client.1.smithi154.stderr:+ rbd import --new-format --order 20 /tmp/rbd_import_export_59341/sparse1 2021-07-31T02:25:51.185 INFO:tasks.workunit.client.1.smithi154.stderr: Importing image: 50% complete... Importing image: 100% complete...done. 2021-07-31T02:25:51.186 INFO:tasks.workunit.client.1.smithi154.stderr:rbd: --new-format is deprecated, use --image-format 2 2021-07-31T02:25:51.187 INFO:tasks.workunit.client.1.smithi154.stderr:rbd: --order is deprecated, use --object-size 2021-07-31T02:25:51.188 INFO:tasks.workunit.client.1.smithi154.stderr:+ rbd ls -l 2021-07-31T02:25:51.189 INFO:tasks.workunit.client.1.smithi154.stderr:+ grep sparse1 2021-07-31T02:25:51.189 INFO:tasks.workunit.client.1.smithi154.stderr:+ grep -Ei '(2 MiB|2048k)' 2021-07-31T02:25:51.225 INFO:tasks.workunit.client.1.smithi154.stdout:sparse1 2 MiB 2 2021-07-31T02:25:51.226 INFO:tasks.workunit.client.1.smithi154.stderr:++ objects sparse1 2021-07-31T02:25:51.227 INFO:tasks.workunit.client.1.smithi154.stderr:++ image=sparse1 2021-07-31T02:25:51.227 INFO:tasks.workunit.client.1.smithi154.stderr:+++ rbd info sparse1 2021-07-31T02:25:51.227 INFO:tasks.workunit.client.1.smithi154.stderr:+++ grep block_name_prefix 2021-07-31T02:25:51.227 INFO:tasks.workunit.client.1.smithi154.stderr:+++ awk '{print $NF;}' 2021-07-31T02:25:51.259 INFO:tasks.workunit.client.1.smithi154.stderr:++ prefix=rbd_data.663b326f8048 2021-07-31T02:25:51.260 INFO:tasks.workunit.client.1.smithi154.stderr:+++ grep rbd_data.663b326f8048 2021-07-31T02:25:51.260 INFO:tasks.workunit.client.1.smithi154.stderr:+++ sed -e 's/rbd_data.663b326f8048\.//' -e 's/^0*\([0-9a-f]\)/\1/' 2021-07-31T02:25:51.260 INFO:tasks.workunit.client.1.smithi154.stderr:++++ get_image_data_pool sparse1 2021-07-31T02:25:51.261 INFO:tasks.workunit.client.1.smithi154.stderr:++++ image=sparse1 2021-07-31T02:25:51.261 INFO:tasks.workunit.client.1.smithi154.stderr:+++ sort -u 2021-07-31T02:25:51.261 INFO:tasks.workunit.client.1.smithi154.stderr:+++++ rbd info sparse1 2021-07-31T02:25:51.261 INFO:tasks.workunit.client.1.smithi154.stderr:+++++ grep 'data_pool: ' 2021-07-31T02:25:51.261 INFO:tasks.workunit.client.1.smithi154.stderr:+++++ awk -F: '{ print $NF }' 2021-07-31T02:25:51.295 INFO:tasks.workunit.client.1.smithi154.stderr:++++ data_pool= 2021-07-31T02:25:51.295 INFO:tasks.workunit.client.1.smithi154.stderr:++++ '[' -z ']' 2021-07-31T02:25:51.295 INFO:tasks.workunit.client.1.smithi154.stderr:++++ data_pool=rbd 2021-07-31T02:25:51.296 INFO:tasks.workunit.client.1.smithi154.stderr:++++ echo rbd 2021-07-31T02:25:51.296 INFO:tasks.workunit.client.1.smithi154.stderr:+++ rados ls -p rbd 2021-07-31T02:25:51.350 INFO:tasks.workunit.client.1.smithi154.stderr:++ objects=1 2021-07-31T02:25:51.350 INFO:tasks.workunit.client.1.smithi154.stderr:++ echo 1 2021-07-31T02:25:51.351 INFO:tasks.workunit.client.1.smithi154.stderr:+ '[' 0 -eq 1 -o 1 = 1 ']' 2021-07-31T02:25:51.351 INFO:tasks.workunit.client.1.smithi154.stderr:+ rbd export sparse1 /tmp/rbd_import_export_59341/sparse1.out 2021-07-31T02:25:51.384 INFO:tasks.workunit.client.1.smithi154.stderr: Exporting image: 50% complete... Exporting image: 100% complete...done. 2021-07-31T02:25:51.387 INFO:tasks.workunit.client.1.smithi154.stderr:+ compare_files_and_ondisk_sizes /tmp/rbd_import_export_59341/sparse1 /tmp/rbd_import_export_59341/sparse1.out 2021-07-31T02:25:51.387 INFO:tasks.workunit.client.1.smithi154.stderr:+ cmp -l /tmp/rbd_import_export_59341/sparse1 /tmp/rbd_import_export_59341/sparse1.out 2021-07-31T02:25:51.389 INFO:tasks.workunit.client.1.smithi154.stderr:++ stat /tmp/rbd_import_export_59341/sparse1 --format %b 2021-07-31T02:25:51.389 INFO:tasks.workunit.client.1.smithi154.stderr:+ origsize=2048 2021-07-31T02:25:51.391 INFO:tasks.workunit.client.1.smithi154.stderr:++ stat /tmp/rbd_import_export_59341/sparse1.out --format %b 2021-07-31T02:25:51.391 INFO:tasks.workunit.client.1.smithi154.stderr:+ exportsize=2048 2021-07-31T02:25:51.391 INFO:tasks.workunit.client.1.smithi154.stderr:+ difference=0 2021-07-31T02:25:51.392 INFO:tasks.workunit.client.1.smithi154.stderr:+ difference=0 2021-07-31T02:25:51.392 INFO:tasks.workunit.client.1.smithi154.stderr:+ test 0 -ge 0 -a 0 -lt 4096 2021-07-31T02:25:51.392 INFO:tasks.workunit.client.1.smithi154.stderr:+ rm /tmp/rbd_import_export_59341/sparse1.out 2021-07-31T02:25:51.393 INFO:tasks.workunit.client.1.smithi154.stderr:+ rbd rm sparse1 2021-07-31T02:25:51.510 INFO:tasks.workunit.client.1.smithi154.stderr: Removing image: 50% complete... Removing image: 100% complete...done. 2021-07-31T02:25:51.513 INFO:tasks.workunit.client.1.smithi154.stderr:+ rbd rm sparse2 2021-07-31T02:25:51.565 INFO:tasks.workunit.client.1.smithi154.stderr: Removing image: 0% complete...failed. 2021-07-31T02:25:51.566 INFO:tasks.workunit.client.1.smithi154.stderr:rbd: delete error: (2) No such file or directory 2021-07-31T02:25:51.567 INFO:tasks.workunit.client.1.smithi154.stderr:+ true 2021-07-31T02:25:51.567 INFO:tasks.workunit.client.1.smithi154.stderr:+ rbd import --new-format --order 20 /tmp/rbd_import_export_59341/sparse2 2021-07-31T02:25:51.637 INFO:tasks.workunit.client.1.smithi154.stderr: Importing image: 50% complete... Importing image: 100% complete...done. 2021-07-31T02:25:51.638 INFO:tasks.workunit.client.1.smithi154.stderr:rbd: --new-format is deprecated, use --image-format 2 2021-07-31T02:25:51.638 INFO:tasks.workunit.client.1.smithi154.stderr:rbd: --order is deprecated, use --object-size 2021-07-31T02:25:51.640 INFO:tasks.workunit.client.1.smithi154.stderr:+ rbd ls -l 2021-07-31T02:25:51.640 INFO:tasks.workunit.client.1.smithi154.stderr:+ grep sparse2 2021-07-31T02:25:51.640 INFO:tasks.workunit.client.1.smithi154.stderr:+ grep -Ei '(2 MiB|2048k)' 2021-07-31T02:25:51.681 INFO:tasks.workunit.client.1.smithi154.stdout:sparse2 2 MiB 2 2021-07-31T02:25:51.682 INFO:tasks.workunit.client.1.smithi154.stderr:++ objects sparse2 2021-07-31T02:25:51.683 INFO:tasks.workunit.client.1.smithi154.stderr:++ image=sparse2 2021-07-31T02:25:51.683 INFO:tasks.workunit.client.1.smithi154.stderr:+++ rbd info sparse2 2021-07-31T02:25:51.683 INFO:tasks.workunit.client.1.smithi154.stderr:+++ grep block_name_prefix 2021-07-31T02:25:51.683 INFO:tasks.workunit.client.1.smithi154.stderr:+++ awk '{print $NF;}' 2021-07-31T02:25:51.717 INFO:tasks.workunit.client.1.smithi154.stderr:++ prefix=rbd_data.666e27128061 2021-07-31T02:25:51.718 INFO:tasks.workunit.client.1.smithi154.stderr:+++ grep rbd_data.666e27128061 2021-07-31T02:25:51.718 INFO:tasks.workunit.client.1.smithi154.stderr:+++ sed -e 's/rbd_data.666e27128061\.//' -e 's/^0*\([0-9a-f]\)/\1/' 2021-07-31T02:25:51.718 INFO:tasks.workunit.client.1.smithi154.stderr:++++ get_image_data_pool sparse2 2021-07-31T02:25:51.718 INFO:tasks.workunit.client.1.smithi154.stderr:+++ sort -u 2021-07-31T02:25:51.719 INFO:tasks.workunit.client.1.smithi154.stderr:++++ image=sparse2 2021-07-31T02:25:51.719 INFO:tasks.workunit.client.1.smithi154.stderr:+++++ rbd info sparse2 2021-07-31T02:25:51.719 INFO:tasks.workunit.client.1.smithi154.stderr:+++++ grep 'data_pool: ' 2021-07-31T02:25:51.719 INFO:tasks.workunit.client.1.smithi154.stderr:+++++ awk -F: '{ print $NF }' 2021-07-31T02:25:51.750 INFO:tasks.workunit.client.1.smithi154.stderr:++++ data_pool= 2021-07-31T02:25:51.751 INFO:tasks.workunit.client.1.smithi154.stderr:++++ '[' -z ']' 2021-07-31T02:25:51.751 INFO:tasks.workunit.client.1.smithi154.stderr:++++ data_pool=rbd 2021-07-31T02:25:51.751 INFO:tasks.workunit.client.1.smithi154.stderr:++++ echo rbd 2021-07-31T02:25:51.752 INFO:tasks.workunit.client.1.smithi154.stderr:+++ rados ls -p rbd 2021-07-31T02:25:51.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:51 smithi154 conmon[34553]: cluster 2021-07-31T02:25:50.312772+0000 mon.a (mon. 2021-07-31T02:25:51.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:51 smithi154 conmon[34553]: 0) 1927 : cluster [DBG] osdmap e1040: 8 total, 8 up, 8 in 2021-07-31T02:25:51.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:51 smithi154 conmon[34553]: cluster 2021-07-31T02:25:50.344239+0000 mgr.x (mgr.24542) 1656 : cluster [DBG] pgmap v2045: 9 pgs: 9 active+clean; 797 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 128 KiB/s rd, 12 KiB/s wr, 171 op/s 2021-07-31T02:25:51.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:51 smithi154 conmon[34553]: audit 2021-07-31T02:25:50.860152+0000 2021-07-31T02:25:51.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:51 smithi154 conmon[34553]: mon.b (mon.2) 263 : audit [DBG] from='client.? 172.21.15.154:0/1095024303' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2021-07-31T02:25:51.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:51 smithi154 conmon[34553]: 2021-07-31T02:25:51.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:51 smithi154 conmon[34553]: audit 2021-07-31T02: 2021-07-31T02:25:51.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:51 smithi154 conmon[34553]: 25:50.961901+0000 mon.b (mon.2) 2021-07-31T02:25:51.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:51 smithi154 conmon[34553]: 264 : audit [DBG] from='client.? 172.21.15.154:0/1095024303' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2021-07-31T02:25:51.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:51 smithi045 conmon[33655]: cluster 2021-07-31T02:25:50.312772+0000 mon.a (mon.0) 1927 : cluster [DBG] osdmap e1040: 8 total, 8 up, 8 in 2021-07-31T02:25:51.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:51 smithi045 conmon[33655]: cluster 2021- 2021-07-31T02:25:51.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:51 smithi045 conmon[33655]: 07-31T02:25:50.344239+0000 mgr.x (mgr.24542) 1656 : cluster [DBG] pgmap v2045: 9 pgs: 9 active+clean; 797 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 128 KiB/s rd, 12 KiB/s wr, 171 op/s 2021-07-31T02:25:51.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:51 smithi045 conmon[33655]: audit 2021-07-31T02:25:50.860152+0000 mon.b (mon.2) 263 : audit [DBG] from='client.? 172.21.15.154:0/1095024303' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2021-07-31T02:25:51.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:51 smithi045 conmon[33655]: audit 2021-07-31T02:25:50.961901+0000 mon.b (mon.2) 264 : audit [DBG] from='client.? 172.21.15.154:0/1095024303' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2021-07-31T02:25:51.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:51 smithi045 conmon[33655]: 2021-07-31T02:25:51.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:51 smithi045 conmon[39718]: cluster 2021-07-31T02:25:50.312772+0000 mon.a 2021-07-31T02:25:51.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:51 smithi045 conmon[39718]: (mon.0) 1927 : cluster [DBG] osdmap e1040: 8 total, 8 up, 8 in 2021-07-31T02:25:51.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:51 smithi045 conmon[39718]: cluster 2021-07-31T 2021-07-31T02:25:51.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:51 smithi045 conmon[39718]: 02:25:50.344239+0000 mgr.x (mgr.24542) 1656 : 2021-07-31T02:25:51.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:51 smithi045 conmon[39718]: cluster [DBG] pgmap v2045: 9 pgs: 9 active+clean; 797 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 128 KiB/s rd, 12 KiB/s wr, 171 op/s 2021-07-31T02:25:51.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:51 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T02:25:51.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:51 smithi045 conmon[39718]: T02:25:50.860152+0000 mon.b ( 2021-07-31T02:25:51.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:51 smithi045 conmon[39718]: mon.2) 263 : audit [DBG] from='client.? 172.21.15.154:0/1095024303' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2021-07-31T02:25:51.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:51 smithi045 conmon[39718]: 2021-07-31T02:25:51.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:51 smithi045 conmon[39718]: audit 2021-07-31T02:25 2021-07-31T02:25:51.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:51 smithi045 conmon[39718]: :50.961901+0000 mon.b (mon.2 2021-07-31T02:25:51.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:51 smithi045 conmon[39718]: ) 264 : audit [DBG] from='client.? 172.21.15.154:0/1095024303' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2021-07-31T02:25:51.799 INFO:tasks.workunit.client.1.smithi154.stderr:++ objects=0 2021-07-31T02:25:51.799 INFO:tasks.workunit.client.1.smithi154.stderr:++ echo 0 2021-07-31T02:25:51.799 INFO:tasks.workunit.client.1.smithi154.stderr:+ '[' 0 -eq 1 -o 0 = 0 ']' 2021-07-31T02:25:51.799 INFO:tasks.workunit.client.1.smithi154.stderr:+ rbd export sparse2 /tmp/rbd_import_export_59341/sparse2.out 2021-07-31T02:25:51.833 INFO:tasks.workunit.client.1.smithi154.stderr: Exporting image: 50% complete... Exporting image: 100% complete...done. 2021-07-31T02:25:51.836 INFO:tasks.workunit.client.1.smithi154.stderr:+ compare_files_and_ondisk_sizes /tmp/rbd_import_export_59341/sparse2 /tmp/rbd_import_export_59341/sparse2.out 2021-07-31T02:25:51.837 INFO:tasks.workunit.client.1.smithi154.stderr:+ cmp -l /tmp/rbd_import_export_59341/sparse2 /tmp/rbd_import_export_59341/sparse2.out 2021-07-31T02:25:51.838 INFO:tasks.workunit.client.1.smithi154.stderr:++ stat /tmp/rbd_import_export_59341/sparse2 --format %b 2021-07-31T02:25:51.839 INFO:tasks.workunit.client.1.smithi154.stderr:+ origsize=2048 2021-07-31T02:25:51.840 INFO:tasks.workunit.client.1.smithi154.stderr:++ stat /tmp/rbd_import_export_59341/sparse2.out --format %b 2021-07-31T02:25:51.840 INFO:tasks.workunit.client.1.smithi154.stderr:+ exportsize=2048 2021-07-31T02:25:51.841 INFO:tasks.workunit.client.1.smithi154.stderr:+ difference=0 2021-07-31T02:25:51.841 INFO:tasks.workunit.client.1.smithi154.stderr:+ difference=0 2021-07-31T02:25:51.841 INFO:tasks.workunit.client.1.smithi154.stderr:+ test 0 -ge 0 -a 0 -lt 4096 2021-07-31T02:25:51.841 INFO:tasks.workunit.client.1.smithi154.stderr:+ rm /tmp/rbd_import_export_59341/sparse2.out 2021-07-31T02:25:51.842 INFO:tasks.workunit.client.1.smithi154.stderr:+ rbd rm sparse2 2021-07-31T02:25:51.958 INFO:tasks.workunit.client.1.smithi154.stderr: Removing image: 50% complete... Removing image: 100% complete...done. 2021-07-31T02:25:51.961 INFO:tasks.workunit.client.1.smithi154.stderr:+ truncate /tmp/rbd_import_export_59341/sparse1 -s 10M 2021-07-31T02:25:51.961 INFO:tasks.workunit.client.1.smithi154.stderr:+ rbd import --new-format --order 20 - sparse1 2021-07-31T02:25:52.082 INFO:tasks.workunit.client.1.smithi154.stderr: Importing image: 100% complete...done. 2021-07-31T02:25:52.083 INFO:tasks.workunit.client.1.smithi154.stderr:rbd: --new-format is deprecated, use --image-format 2 2021-07-31T02:25:52.083 INFO:tasks.workunit.client.1.smithi154.stderr:rbd: --order is deprecated, use --object-size 2021-07-31T02:25:52.085 INFO:tasks.workunit.client.1.smithi154.stderr:+ rbd ls -l 2021-07-31T02:25:52.085 INFO:tasks.workunit.client.1.smithi154.stderr:+ grep sparse1 2021-07-31T02:25:52.086 INFO:tasks.workunit.client.1.smithi154.stderr:+ grep -Ei '(10 MiB|10240k)' 2021-07-31T02:25:52.129 INFO:tasks.workunit.client.1.smithi154.stdout:sparse1 10 MiB 2 2021-07-31T02:25:52.130 INFO:tasks.workunit.client.1.smithi154.stderr:++ objects sparse1 2021-07-31T02:25:52.130 INFO:tasks.workunit.client.1.smithi154.stderr:++ image=sparse1 2021-07-31T02:25:52.130 INFO:tasks.workunit.client.1.smithi154.stderr:+++ rbd info sparse1 2021-07-31T02:25:52.130 INFO:tasks.workunit.client.1.smithi154.stderr:+++ grep block_name_prefix 2021-07-31T02:25:52.131 INFO:tasks.workunit.client.1.smithi154.stderr:+++ awk '{print $NF;}' 2021-07-31T02:25:52.165 INFO:tasks.workunit.client.1.smithi154.stderr:++ prefix=rbd_data.669be56e1c54 2021-07-31T02:25:52.167 INFO:tasks.workunit.client.1.smithi154.stderr:+++ grep rbd_data.669be56e1c54 2021-07-31T02:25:52.167 INFO:tasks.workunit.client.1.smithi154.stderr:+++ sed -e 's/rbd_data.669be56e1c54\.//' -e 's/^0*\([0-9a-f]\)/\1/' 2021-07-31T02:25:52.167 INFO:tasks.workunit.client.1.smithi154.stderr:++++ get_image_data_pool sparse1 2021-07-31T02:25:52.167 INFO:tasks.workunit.client.1.smithi154.stderr:+++ sort -u 2021-07-31T02:25:52.168 INFO:tasks.workunit.client.1.smithi154.stderr:++++ image=sparse1 2021-07-31T02:25:52.168 INFO:tasks.workunit.client.1.smithi154.stderr:+++++ rbd info sparse1 2021-07-31T02:25:52.168 INFO:tasks.workunit.client.1.smithi154.stderr:+++++ grep 'data_pool: ' 2021-07-31T02:25:52.168 INFO:tasks.workunit.client.1.smithi154.stderr:+++++ awk -F: '{ print $NF }' 2021-07-31T02:25:52.206 INFO:tasks.workunit.client.1.smithi154.stderr:++++ data_pool= 2021-07-31T02:25:52.206 INFO:tasks.workunit.client.1.smithi154.stderr:++++ '[' -z ']' 2021-07-31T02:25:52.207 INFO:tasks.workunit.client.1.smithi154.stderr:++++ data_pool=rbd 2021-07-31T02:25:52.207 INFO:tasks.workunit.client.1.smithi154.stderr:++++ echo rbd 2021-07-31T02:25:52.207 INFO:tasks.workunit.client.1.smithi154.stderr:+++ rados ls -p rbd 2021-07-31T02:25:52.261 INFO:tasks.workunit.client.1.smithi154.stderr:++ objects=1 2021-07-31T02:25:52.261 INFO:tasks.workunit.client.1.smithi154.stderr:++ echo 1 2021-07-31T02:25:52.261 INFO:tasks.workunit.client.1.smithi154.stderr:+ '[' 0 -eq 1 -o 1 = 1 ']' 2021-07-31T02:25:52.262 INFO:tasks.workunit.client.1.smithi154.stderr:+ rbd export sparse1 /tmp/rbd_import_export_59341/sparse1.out 2021-07-31T02:25:52.298 INFO:tasks.workunit.client.1.smithi154.stderr: Exporting image: 10% complete... Exporting image: 20% complete... Exporting image: 30% complete... Exporting image: 40% complete... Exporting image: 50% complete... Exporting image: 60% complete... Exporting image: 70% complete... Exporting image: 80% complete... Exporting image: 90% complete... Exporting image: 100% complete...done. 2021-07-31T02:25:52.301 INFO:tasks.workunit.client.1.smithi154.stderr:+ compare_files_and_ondisk_sizes /tmp/rbd_import_export_59341/sparse1 /tmp/rbd_import_export_59341/sparse1.out 2021-07-31T02:25:52.301 INFO:tasks.workunit.client.1.smithi154.stderr:+ cmp -l /tmp/rbd_import_export_59341/sparse1 /tmp/rbd_import_export_59341/sparse1.out 2021-07-31T02:25:52.308 INFO:tasks.workunit.client.1.smithi154.stderr:++ stat /tmp/rbd_import_export_59341/sparse1 --format %b 2021-07-31T02:25:52.309 INFO:tasks.workunit.client.1.smithi154.stderr:+ origsize=2048 2021-07-31T02:25:52.309 INFO:tasks.workunit.client.1.smithi154.stderr:++ stat /tmp/rbd_import_export_59341/sparse1.out --format %b 2021-07-31T02:25:52.310 INFO:tasks.workunit.client.1.smithi154.stderr:+ exportsize=2048 2021-07-31T02:25:52.310 INFO:tasks.workunit.client.1.smithi154.stderr:+ difference=0 2021-07-31T02:25:52.310 INFO:tasks.workunit.client.1.smithi154.stderr:+ difference=0 2021-07-31T02:25:52.311 INFO:tasks.workunit.client.1.smithi154.stderr:+ test 0 -ge 0 -a 0 -lt 4096 2021-07-31T02:25:52.311 INFO:tasks.workunit.client.1.smithi154.stderr:+ rm /tmp/rbd_import_export_59341/sparse1.out 2021-07-31T02:25:52.311 INFO:tasks.workunit.client.1.smithi154.stderr:+ rbd rm sparse1 2021-07-31T02:25:52.438 INFO:tasks.workunit.client.1.smithi154.stderr: Removing image: 10% complete... Removing image: 20% complete... Removing image: 30% complete... Removing image: 40% complete... Removing image: 50% complete... Removing image: 60% complete... Removing image: 70% complete... Removing image: 80% complete... Removing image: 90% complete... Removing image: 100% complete...done. 2021-07-31T02:25:52.440 INFO:tasks.workunit.client.1.smithi154.stderr:+ dd if=/dev/urandom bs=2M count=1 of=/tmp/rbd_import_export_59341/sparse2 oflag=append conv=notrunc 2021-07-31T02:25:52.451 INFO:tasks.workunit.client.1.smithi154.stderr:1+0 records in 2021-07-31T02:25:52.451 INFO:tasks.workunit.client.1.smithi154.stderr:1+0 records out 2021-07-31T02:25:52.451 INFO:tasks.workunit.client.1.smithi154.stderr:2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.00971801 s, 216 MB/s 2021-07-31T02:25:52.452 INFO:tasks.workunit.client.1.smithi154.stderr:+ rbd import --new-format --order 20 - sparse2 2021-07-31T02:25:52.569 INFO:tasks.workunit.client.1.smithi154.stderr: Importing image: 100% complete...done. 2021-07-31T02:25:52.570 INFO:tasks.workunit.client.1.smithi154.stderr:rbd: --new-format is deprecated, use --image-format 2 2021-07-31T02:25:52.570 INFO:tasks.workunit.client.1.smithi154.stderr:rbd: --order is deprecated, use --object-size 2021-07-31T02:25:52.572 INFO:tasks.workunit.client.1.smithi154.stderr:+ rbd ls -l 2021-07-31T02:25:52.572 INFO:tasks.workunit.client.1.smithi154.stderr:+ grep sparse2 2021-07-31T02:25:52.573 INFO:tasks.workunit.client.1.smithi154.stderr:+ grep -Ei '(4 MiB|4096k)' 2021-07-31T02:25:52.619 INFO:tasks.workunit.client.1.smithi154.stdout:sparse2 4 MiB 2 2021-07-31T02:25:52.620 INFO:tasks.workunit.client.1.smithi154.stderr:++ objects sparse2 2021-07-31T02:25:52.621 INFO:tasks.workunit.client.1.smithi154.stderr:++ image=sparse2 2021-07-31T02:25:52.621 INFO:tasks.workunit.client.1.smithi154.stderr:+++ rbd info sparse2 2021-07-31T02:25:52.621 INFO:tasks.workunit.client.1.smithi154.stderr:+++ grep block_name_prefix 2021-07-31T02:25:52.621 INFO:tasks.workunit.client.1.smithi154.stderr:+++ awk '{print $NF;}' 2021-07-31T02:25:52.661 INFO:tasks.workunit.client.1.smithi154.stderr:++ prefix=rbd_data.66c848165061 2021-07-31T02:25:52.662 INFO:tasks.workunit.client.1.smithi154.stderr:+++ grep rbd_data.66c848165061 2021-07-31T02:25:52.663 INFO:tasks.workunit.client.1.smithi154.stderr:+++ sed -e 's/rbd_data.66c848165061\.//' -e 's/^0*\([0-9a-f]\)/\1/' 2021-07-31T02:25:52.663 INFO:tasks.workunit.client.1.smithi154.stderr:++++ get_image_data_pool sparse2 2021-07-31T02:25:52.663 INFO:tasks.workunit.client.1.smithi154.stderr:++++ image=sparse2 2021-07-31T02:25:52.663 INFO:tasks.workunit.client.1.smithi154.stderr:+++ sort -u 2021-07-31T02:25:52.663 INFO:tasks.workunit.client.1.smithi154.stderr:+++++ rbd info sparse2 2021-07-31T02:25:52.664 INFO:tasks.workunit.client.1.smithi154.stderr:+++++ grep 'data_pool: ' 2021-07-31T02:25:52.664 INFO:tasks.workunit.client.1.smithi154.stderr:+++++ awk -F: '{ print $NF }' 2021-07-31T02:25:52.703 INFO:tasks.workunit.client.1.smithi154.stderr:++++ data_pool= 2021-07-31T02:25:52.703 INFO:tasks.workunit.client.1.smithi154.stderr:++++ '[' -z ']' 2021-07-31T02:25:52.704 INFO:tasks.workunit.client.1.smithi154.stderr:++++ data_pool=rbd 2021-07-31T02:25:52.704 INFO:tasks.workunit.client.1.smithi154.stderr:++++ echo rbd 2021-07-31T02:25:52.704 INFO:tasks.workunit.client.1.smithi154.stderr:+++ rados ls -p rbd 2021-07-31T02:25:52.748 INFO:tasks.workunit.client.1.smithi154.stderr:++ objects='0 2021-07-31T02:25:52.749 INFO:tasks.workunit.client.1.smithi154.stderr:2 2021-07-31T02:25:52.749 INFO:tasks.workunit.client.1.smithi154.stderr:3' 2021-07-31T02:25:52.749 INFO:tasks.workunit.client.1.smithi154.stderr:++ echo 0 2 3 2021-07-31T02:25:52.749 INFO:tasks.workunit.client.1.smithi154.stderr:+ '[' 0 -eq 1 -o '0 2 3' = '0 2 3' ']' 2021-07-31T02:25:52.750 INFO:tasks.workunit.client.1.smithi154.stderr:+ rbd export sparse2 /tmp/rbd_import_export_59341/sparse2.out 2021-07-31T02:25:52.789 INFO:tasks.workunit.client.1.smithi154.stderr: Exporting image: 25% complete... Exporting image: 50% complete... Exporting image: 75% complete... Exporting image: 100% complete...done. 2021-07-31T02:25:52.792 INFO:tasks.workunit.client.1.smithi154.stderr:+ compare_files_and_ondisk_sizes /tmp/rbd_import_export_59341/sparse2 /tmp/rbd_import_export_59341/sparse2.out 2021-07-31T02:25:52.793 INFO:tasks.workunit.client.1.smithi154.stderr:+ cmp -l /tmp/rbd_import_export_59341/sparse2 /tmp/rbd_import_export_59341/sparse2.out 2021-07-31T02:25:52.795 INFO:tasks.workunit.client.1.smithi154.stderr:++ stat /tmp/rbd_import_export_59341/sparse2 --format %b 2021-07-31T02:25:52.796 INFO:tasks.workunit.client.1.smithi154.stderr:+ origsize=6144 2021-07-31T02:25:52.796 INFO:tasks.workunit.client.1.smithi154.stderr:++ stat /tmp/rbd_import_export_59341/sparse2.out --format %b 2021-07-31T02:25:52.797 INFO:tasks.workunit.client.1.smithi154.stderr:+ exportsize=6144 2021-07-31T02:25:52.798 INFO:tasks.workunit.client.1.smithi154.stderr:+ difference=0 2021-07-31T02:25:52.798 INFO:tasks.workunit.client.1.smithi154.stderr:+ difference=0 2021-07-31T02:25:52.798 INFO:tasks.workunit.client.1.smithi154.stderr:+ test 0 -ge 0 -a 0 -lt 4096 2021-07-31T02:25:52.798 INFO:tasks.workunit.client.1.smithi154.stderr:+ rm /tmp/rbd_import_export_59341/sparse2.out 2021-07-31T02:25:52.799 INFO:tasks.workunit.client.1.smithi154.stderr:+ rbd rm sparse2 2021-07-31T02:25:52.935 INFO:tasks.workunit.client.1.smithi154.stderr: Removing image: 25% complete... Removing image: 50% complete... Removing image: 75% complete... Removing image: 100% complete...done. 2021-07-31T02:25:52.938 INFO:tasks.workunit.client.1.smithi154.stderr:+ echo 'partially-sparse file imports to partially-sparse image' 2021-07-31T02:25:52.938 INFO:tasks.workunit.client.1.smithi154.stdout:partially-sparse file imports to partially-sparse image 2021-07-31T02:25:52.939 INFO:tasks.workunit.client.1.smithi154.stderr:+ rbd import --new-format --order 20 /tmp/rbd_import_export_59341/sparse1 sparse 2021-07-31T02:25:53.016 INFO:tasks.workunit.client.1.smithi154.stderr: Importing image: 10% complete... Importing image: 20% complete... Importing image: 30% complete... Importing image: 40% complete... Importing image: 50% complete... Importing image: 60% complete... Importing image: 70% complete... Importing image: 80% complete... Importing image: 90% complete... Importing image: 100% complete...done. 2021-07-31T02:25:53.017 INFO:tasks.workunit.client.1.smithi154.stderr:rbd: --new-format is deprecated, use --image-format 2 2021-07-31T02:25:53.017 INFO:tasks.workunit.client.1.smithi154.stderr:rbd: --order is deprecated, use --object-size 2021-07-31T02:25:53.019 INFO:tasks.workunit.client.1.smithi154.stderr:++ objects sparse 2021-07-31T02:25:53.019 INFO:tasks.workunit.client.1.smithi154.stderr:++ image=sparse 2021-07-31T02:25:53.019 INFO:tasks.workunit.client.1.smithi154.stderr:+++ rbd info sparse 2021-07-31T02:25:53.020 INFO:tasks.workunit.client.1.smithi154.stderr:+++ grep block_name_prefix 2021-07-31T02:25:53.020 INFO:tasks.workunit.client.1.smithi154.stderr:+++ awk '{print $NF;}' 2021-07-31T02:25:53.062 INFO:tasks.workunit.client.1.smithi154.stderr:++ prefix=rbd_data.66f52655a94c 2021-07-31T02:25:53.062 INFO:tasks.workunit.client.1.smithi154.stderr:+++ grep rbd_data.66f52655a94c 2021-07-31T02:25:53.064 INFO:tasks.workunit.client.1.smithi154.stderr:+++ sed -e 's/rbd_data.66f52655a94c\.//' -e 's/^0*\([0-9a-f]\)/\1/' 2021-07-31T02:25:53.065 INFO:tasks.workunit.client.1.smithi154.stderr:++++ get_image_data_pool sparse 2021-07-31T02:25:53.065 INFO:tasks.workunit.client.1.smithi154.stderr:+++ sort -u 2021-07-31T02:25:53.065 INFO:tasks.workunit.client.1.smithi154.stderr:++++ image=sparse 2021-07-31T02:25:53.065 INFO:tasks.workunit.client.1.smithi154.stderr:+++++ rbd info sparse 2021-07-31T02:25:53.066 INFO:tasks.workunit.client.1.smithi154.stderr:+++++ grep 'data_pool: ' 2021-07-31T02:25:53.066 INFO:tasks.workunit.client.1.smithi154.stderr:+++++ awk -F: '{ print $NF }' 2021-07-31T02:25:53.103 INFO:tasks.workunit.client.1.smithi154.stderr:++++ data_pool= 2021-07-31T02:25:53.103 INFO:tasks.workunit.client.1.smithi154.stderr:++++ '[' -z ']' 2021-07-31T02:25:53.104 INFO:tasks.workunit.client.1.smithi154.stderr:++++ data_pool=rbd 2021-07-31T02:25:53.104 INFO:tasks.workunit.client.1.smithi154.stderr:++++ echo rbd 2021-07-31T02:25:53.104 INFO:tasks.workunit.client.1.smithi154.stderr:+++ rados ls -p rbd 2021-07-31T02:25:53.152 INFO:tasks.workunit.client.1.smithi154.stderr:++ objects=1 2021-07-31T02:25:53.153 INFO:tasks.workunit.client.1.smithi154.stderr:++ echo 1 2021-07-31T02:25:53.153 INFO:tasks.workunit.client.1.smithi154.stderr:+ '[' 0 -eq 1 -o 1 = 1 ']' 2021-07-31T02:25:53.153 INFO:tasks.workunit.client.1.smithi154.stderr:+ rbd rm sparse 2021-07-31T02:25:53.294 INFO:tasks.workunit.client.1.smithi154.stderr: Removing image: 10% complete... Removing image: 20% complete... Removing image: 30% complete... Removing image: 40% complete... Removing image: 50% complete... Removing image: 60% complete... Removing image: 70% complete... Removing image: 80% complete... Removing image: 90% complete... Removing image: 100% complete...done. 2021-07-31T02:25:53.297 INFO:tasks.workunit.client.1.smithi154.stdout:zeros import through stdin to sparse image 2021-07-31T02:25:53.298 INFO:tasks.workunit.client.1.smithi154.stderr:+ echo 'zeros import through stdin to sparse image' 2021-07-31T02:25:53.298 INFO:tasks.workunit.client.1.smithi154.stderr:+ dd if=/dev/zero bs=1M count=4 2021-07-31T02:25:53.298 INFO:tasks.workunit.client.1.smithi154.stderr:+ rbd import --new-format - sparse 2021-07-31T02:25:53.353 INFO:tasks.workunit.client.1.smithi154.stderr:4+0 records in 2021-07-31T02:25:53.353 INFO:tasks.workunit.client.1.smithi154.stderr:4+0 records out 2021-07-31T02:25:53.353 INFO:tasks.workunit.client.1.smithi154.stderr:4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.0503844 s, 83.2 MB/s 2021-07-31T02:25:53.362 INFO:tasks.workunit.client.1.smithi154.stderr: Importing image: 100% complete...done. 2021-07-31T02:25:53.363 INFO:tasks.workunit.client.1.smithi154.stderr:rbd: --new-format is deprecated, use --image-format 2 2021-07-31T02:25:53.365 INFO:tasks.workunit.client.1.smithi154.stderr:++ objects sparse 2021-07-31T02:25:53.365 INFO:tasks.workunit.client.1.smithi154.stderr:++ image=sparse 2021-07-31T02:25:53.366 INFO:tasks.workunit.client.1.smithi154.stderr:+++ rbd info sparse 2021-07-31T02:25:53.366 INFO:tasks.workunit.client.1.smithi154.stderr:+++ grep block_name_prefix 2021-07-31T02:25:53.366 INFO:tasks.workunit.client.1.smithi154.stderr:+++ awk '{print $NF;}' 2021-07-31T02:25:53.401 INFO:tasks.workunit.client.1.smithi154.stderr:++ prefix=rbd_data.67164575d5d4 2021-07-31T02:25:53.402 INFO:tasks.workunit.client.1.smithi154.stderr:+++ grep rbd_data.67164575d5d4 2021-07-31T02:25:53.402 INFO:tasks.workunit.client.1.smithi154.stderr:+++ sed -e 's/rbd_data.67164575d5d4\.//' -e 's/^0*\([0-9a-f]\)/\1/' 2021-07-31T02:25:53.402 INFO:tasks.workunit.client.1.smithi154.stderr:++++ get_image_data_pool sparse 2021-07-31T02:25:53.403 INFO:tasks.workunit.client.1.smithi154.stderr:+++ sort -u 2021-07-31T02:25:53.403 INFO:tasks.workunit.client.1.smithi154.stderr:++++ image=sparse 2021-07-31T02:25:53.403 INFO:tasks.workunit.client.1.smithi154.stderr:+++++ rbd info sparse 2021-07-31T02:25:53.403 INFO:tasks.workunit.client.1.smithi154.stderr:+++++ grep 'data_pool: ' 2021-07-31T02:25:53.404 INFO:tasks.workunit.client.1.smithi154.stderr:+++++ awk -F: '{ print $NF }' 2021-07-31T02:25:53.438 INFO:tasks.workunit.client.1.smithi154.stderr:++++ data_pool= 2021-07-31T02:25:53.438 INFO:tasks.workunit.client.1.smithi154.stderr:++++ '[' -z ']' 2021-07-31T02:25:53.439 INFO:tasks.workunit.client.1.smithi154.stderr:++++ data_pool=rbd 2021-07-31T02:25:53.440 INFO:tasks.workunit.client.1.smithi154.stderr:++++ echo rbd 2021-07-31T02:25:53.440 INFO:tasks.workunit.client.1.smithi154.stderr:+++ rados ls -p rbd 2021-07-31T02:25:53.485 INFO:tasks.workunit.client.1.smithi154.stderr:++ objects= 2021-07-31T02:25:53.486 INFO:tasks.workunit.client.1.smithi154.stderr:++ echo 2021-07-31T02:25:53.486 INFO:tasks.workunit.client.1.smithi154.stderr:+ '[' 0 -eq 1 -o '' = '' ']' 2021-07-31T02:25:53.486 INFO:tasks.workunit.client.1.smithi154.stderr:+ rbd rm sparse 2021-07-31T02:25:53.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:53 smithi154 conmon[34553]: cluster 2021-07-31T02:25:52.212210+0000 mon.a (mon. 2021-07-31T02:25:53.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:53 smithi154 conmon[34553]: 0) 1928 : cluster [DBG] osdmap e1041: 8 total, 8 up, 8 in 2021-07-31T02:25:53.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:53 smithi045 conmon[39718]: cluster 2021-07-31T02:25:52.212210 2021-07-31T02:25:53.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:53 smithi045 conmon[39718]: +0000 mon.a (mon.0) 1928 : cluster [DBG] osdmap e1041: 8 total, 8 up, 8 in 2021-07-31T02:25:53.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:53 smithi045 conmon[33655]: cluster 2021-07-31T02:25:52.212210+0000 mon.a (mon.0) 1928 : cluster 2021-07-31T02:25:53.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:53 smithi045 conmon[33655]: [DBG] osdmap e1041: 8 total, 8 up, 8 in 2021-07-31T02:25:53.598 INFO:tasks.workunit.client.1.smithi154.stderr: Removing image: 100% complete...done. 2021-07-31T02:25:53.601 INFO:tasks.workunit.client.1.smithi154.stderr:+ echo 'zeros export to sparse file' 2021-07-31T02:25:53.601 INFO:tasks.workunit.client.1.smithi154.stdout:zeros export to sparse file 2021-07-31T02:25:53.602 INFO:tasks.workunit.client.1.smithi154.stderr:+ rbd create --new-format sparse --size 4 2021-07-31T02:25:53.641 INFO:tasks.workunit.client.1.smithi154.stderr:rbd: --new-format is deprecated, use --image-format 2 2021-07-31T02:25:53.643 INFO:tasks.workunit.client.1.smithi154.stderr:++ rbd info sparse 2021-07-31T02:25:53.643 INFO:tasks.workunit.client.1.smithi154.stderr:++ grep block_name_prefix 2021-07-31T02:25:53.644 INFO:tasks.workunit.client.1.smithi154.stderr:++ awk '{print $NF;}' 2021-07-31T02:25:53.684 INFO:tasks.workunit.client.1.smithi154.stderr:+ prefix=rbd_data.67371f9033e2 2021-07-31T02:25:53.684 INFO:tasks.workunit.client.1.smithi154.stderr:+ dd if=/dev/zero bs=4M count=1 2021-07-31T02:25:53.685 INFO:tasks.workunit.client.1.smithi154.stderr:++ get_image_data_pool sparse 2021-07-31T02:25:53.685 INFO:tasks.workunit.client.1.smithi154.stderr:++ image=sparse 2021-07-31T02:25:53.685 INFO:tasks.workunit.client.1.smithi154.stderr:+++ rbd info sparse 2021-07-31T02:25:53.685 INFO:tasks.workunit.client.1.smithi154.stderr:+++ grep 'data_pool: ' 2021-07-31T02:25:53.685 INFO:tasks.workunit.client.1.smithi154.stderr:+++ awk -F: '{ print $NF }' 2021-07-31T02:25:53.725 INFO:tasks.workunit.client.1.smithi154.stderr:++ data_pool= 2021-07-31T02:25:53.725 INFO:tasks.workunit.client.1.smithi154.stderr:++ '[' -z ']' 2021-07-31T02:25:53.726 INFO:tasks.workunit.client.1.smithi154.stderr:++ data_pool=rbd 2021-07-31T02:25:53.726 INFO:tasks.workunit.client.1.smithi154.stderr:++ echo rbd 2021-07-31T02:25:53.726 INFO:tasks.workunit.client.1.smithi154.stderr:+ rados -p rbd put rbd_data.67371f9033e2.000000000000 - 2021-07-31T02:25:53.752 INFO:tasks.workunit.client.1.smithi154.stderr:1+0 records in 2021-07-31T02:25:53.752 INFO:tasks.workunit.client.1.smithi154.stderr:1+0 records out 2021-07-31T02:25:53.752 INFO:tasks.workunit.client.1.smithi154.stderr:4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.0680171 s, 61.7 MB/s 2021-07-31T02:25:53.771 INFO:tasks.workunit.client.1.smithi154.stderr:++ objects sparse 2021-07-31T02:25:53.771 INFO:tasks.workunit.client.1.smithi154.stderr:++ image=sparse 2021-07-31T02:25:53.772 INFO:tasks.workunit.client.1.smithi154.stderr:+++ rbd info sparse 2021-07-31T02:25:53.772 INFO:tasks.workunit.client.1.smithi154.stderr:+++ grep block_name_prefix 2021-07-31T02:25:53.772 INFO:tasks.workunit.client.1.smithi154.stderr:+++ awk '{print $NF;}' 2021-07-31T02:25:53.811 INFO:tasks.workunit.client.1.smithi154.stderr:++ prefix=rbd_data.67371f9033e2 2021-07-31T02:25:53.813 INFO:tasks.workunit.client.1.smithi154.stderr:+++ grep rbd_data.67371f9033e2 2021-07-31T02:25:53.813 INFO:tasks.workunit.client.1.smithi154.stderr:+++ sed -e 's/rbd_data.67371f9033e2\.//' -e 's/^0*\([0-9a-f]\)/\1/' 2021-07-31T02:25:53.813 INFO:tasks.workunit.client.1.smithi154.stderr:++++ get_image_data_pool sparse 2021-07-31T02:25:53.813 INFO:tasks.workunit.client.1.smithi154.stderr:++++ image=sparse 2021-07-31T02:25:53.813 INFO:tasks.workunit.client.1.smithi154.stderr:+++ sort -u 2021-07-31T02:25:53.814 INFO:tasks.workunit.client.1.smithi154.stderr:+++++ rbd info sparse 2021-07-31T02:25:53.814 INFO:tasks.workunit.client.1.smithi154.stderr:+++++ grep 'data_pool: ' 2021-07-31T02:25:53.814 INFO:tasks.workunit.client.1.smithi154.stderr:+++++ awk -F: '{ print $NF }' 2021-07-31T02:25:53.854 INFO:tasks.workunit.client.1.smithi154.stderr:++++ data_pool= 2021-07-31T02:25:53.855 INFO:tasks.workunit.client.1.smithi154.stderr:++++ '[' -z ']' 2021-07-31T02:25:53.855 INFO:tasks.workunit.client.1.smithi154.stderr:++++ data_pool=rbd 2021-07-31T02:25:53.855 INFO:tasks.workunit.client.1.smithi154.stderr:++++ echo rbd 2021-07-31T02:25:53.855 INFO:tasks.workunit.client.1.smithi154.stderr:+++ rados ls -p rbd 2021-07-31T02:25:53.902 INFO:tasks.workunit.client.1.smithi154.stderr:++ objects=0 2021-07-31T02:25:53.903 INFO:tasks.workunit.client.1.smithi154.stderr:++ echo 0 2021-07-31T02:25:53.903 INFO:tasks.workunit.client.1.smithi154.stderr:+ '[' 0 -eq 1 -o 0 = 0 ']' 2021-07-31T02:25:53.903 INFO:tasks.workunit.client.1.smithi154.stderr:+ rm /tmp/rbd_import_export_59341/sparse 2021-07-31T02:25:53.904 INFO:tasks.workunit.client.1.smithi154.stderr:rm: cannot remove '/tmp/rbd_import_export_59341/sparse': No such file or directory 2021-07-31T02:25:53.904 INFO:tasks.workunit.client.1.smithi154.stderr:+ true 2021-07-31T02:25:53.905 INFO:tasks.workunit.client.1.smithi154.stderr:+ rbd export sparse /tmp/rbd_import_export_59341/sparse 2021-07-31T02:25:53.943 INFO:tasks.workunit.client.1.smithi154.stderr: Exporting image: 100% complete...done. 2021-07-31T02:25:53.946 INFO:tasks.workunit.client.1.smithi154.stderr:++ stat /tmp/rbd_import_export_59341/sparse --format=%b 2021-07-31T02:25:53.947 INFO:tasks.workunit.client.1.smithi154.stderr:+ '[' 0 = 0 ']' 2021-07-31T02:25:53.947 INFO:tasks.workunit.client.1.smithi154.stderr:+ rbd rm sparse 2021-07-31T02:25:54.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:25:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:25:53.576Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:25:54.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:25:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:25:53.576Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:25:54.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:25:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:25:53.576Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:25:54.077 INFO:tasks.workunit.client.1.smithi154.stderr: Removing image: 100% complete...done. 2021-07-31T02:25:54.080 INFO:tasks.workunit.client.1.smithi154.stderr:+ rm /tmp/rbd_import_export_59341/sparse /tmp/rbd_import_export_59341/sparse1 /tmp/rbd_import_export_59341/sparse2 /tmp/rbd_import_export_59341/sparse3 2021-07-31T02:25:54.082 INFO:tasks.workunit.client.1.smithi154.stderr:rm: cannot remove '/tmp/rbd_import_export_59341/sparse3': No such file or directory 2021-07-31T02:25:54.082 INFO:tasks.workunit.client.1.smithi154.stderr:+ true 2021-07-31T02:25:54.083 INFO:tasks.workunit.client.1.smithi154.stdout:OK 2021-07-31T02:25:54.084 INFO:tasks.workunit.client.1.smithi154.stderr:+ echo OK 2021-07-31T02:25:54.084 INFO:tasks.workunit.client.1.smithi154.stderr:+ rm -rf /tmp/rbd_import_export_59341 2021-07-31T02:25:54.085 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-07-31T02:25:54.085 DEBUG:teuthology.orchestra.run.smithi154:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.1/client.1/tmp 2021-07-31T02:25:54.159 INFO:tasks.workunit:Stopping ['rbd/import_export.sh'] on client.1... 2021-07-31T02:25:54.160 DEBUG:teuthology.orchestra.run.smithi154:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.1 /home/ubuntu/cephtest/clone.client.1 2021-07-31T02:25:54.371 DEBUG:teuthology.parallel:result is None 2021-07-31T02:25:54.372 DEBUG:teuthology.orchestra.run.smithi154:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.1/client.1 2021-07-31T02:25:54.396 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.1/client.1 2021-07-31T02:25:54.396 DEBUG:teuthology.orchestra.run.smithi154:> rmdir -- /home/ubuntu/cephtest/mnt.1 2021-07-31T02:25:54.450 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.1/client.1 2021-07-31T02:25:54.451 INFO:teuthology.task.full_sequential:In full_sequential, running task print... 2021-07-31T02:25:54.451 INFO:teuthology.task.print:**** done end rbd_import_export.yaml 2021-07-31T02:25:54.451 INFO:teuthology.task.full_sequential:In full_sequential, running task print... 2021-07-31T02:25:54.452 INFO:teuthology.task.print:**** done start test_rbd_api.yaml 2021-07-31T02:25:54.452 INFO:teuthology.task.full_sequential:In full_sequential, running task workunit... 2021-07-31T02:25:54.452 INFO:tasks.workunit:Pulling workunits from ref pacific 2021-07-31T02:25:54.453 INFO:tasks.workunit:Making a separate scratch dir for every client... 2021-07-31T02:25:54.453 DEBUG:teuthology.orchestra.run.smithi045:> stat -- /home/ubuntu/cephtest/mnt.0 2021-07-31T02:25:54.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-07-31T02:25:54.471 INFO:teuthology.orchestra.run.smithi045.stderr:stat: cannot stat '/home/ubuntu/cephtest/mnt.0': No such file or directory 2021-07-31T02:25:54.471 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -- /home/ubuntu/cephtest/mnt.0 2021-07-31T02:25:54.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:54 smithi154 conmon[34553]: cluster 2021-07-31T02:25:52.344549+0000 mgr.x (mgr.24542) 1657 : cluster [DBG] pgmap v2047: 9 pgs: 9 active+clean; 797 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 124 KiB/s rd, 12 KiB/s wr, 166 op/s 2021-07-31T02:25:54.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:54 smithi154 conmon[34553]: 2021-07-31T02:25:54.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:54 smithi045 conmon[33655]: cluster 2021-07-31T02:25:52.344549+0000 mgr.x (mgr 2021-07-31T02:25:54.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:54 smithi045 conmon[33655]: .24542) 1657 : cluster [DBG] pgmap v2047: 9 pgs: 9 active+clean; 797 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 124 KiB/s rd, 12 KiB/s wr, 166 op/s 2021-07-31T02:25:54.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:54 smithi045 conmon[39718]: cluster 2021-07-31T02:25:52.344549+0000 mgr.x (mgr.24542) 1657 : cluster [DBG] pgmap v2047: 9 pgs: 9 active+clean; 797 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 124 KiB/s rd, 12 KiB/s wr, 166 op/s 2021-07-31T02:25:54.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:54 smithi045 conmon[39718]: 2021-07-31T02:25:54.528 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2021-07-31T02:25:54.528 DEBUG:teuthology.orchestra.run.smithi045:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2021-07-31T02:25:54.583 INFO:tasks.workunit:timeout=3h 2021-07-31T02:25:54.584 INFO:tasks.workunit:cleanup=True 2021-07-31T02:25:54.584 DEBUG:teuthology.orchestra.run.smithi045:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone --depth 1 --branch pacific git://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 2021-07-31T02:25:54.639 INFO:tasks.workunit.client.0.smithi045.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2021-07-31T02:25:54.664 INFO:tasks.workunit.client.0.smithi045.stderr:warning: Could not find remote branch pacific to clone. 2021-07-31T02:25:54.665 INFO:tasks.workunit.client.0.smithi045.stderr:fatal: Remote branch pacific not found in upstream origin 2021-07-31T02:25:54.665 DEBUG:teuthology.orchestra.run:got remote process result: 128 2021-07-31T02:25:54.666 INFO:tasks.workunit:failed to check out 'pacific' from git://git.ceph.com/ceph-ci.git; will also try in git://git.ceph.com/ceph.git 2021-07-31T02:25:54.667 DEBUG:teuthology.orchestra.run.smithi045:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone --depth 1 --branch pacific git://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 2021-07-31T02:25:54.722 INFO:tasks.workunit.client.0.smithi045.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2021-07-31T02:25:56.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:56 smithi154 conmon[34553]: cluster 2021-07-31T02:25:54.344928+0000 mgr.x (mgr.24542) 1658 : cluster [DBG] pgmap v2048: 9 pgs: 9 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.4 MiB/s rd, 1.9 MiB/s wr, 529 op/s 2021-07-31T02:25:56.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:56 smithi045 conmon[33655]: cluster 2021-07-31T02:25:54.344928+0000 mgr.x (mgr.24542) 1658 2021-07-31T02:25:56.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:56 smithi045 conmon[33655]: : cluster [DBG] pgmap v2048: 9 pgs: 9 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.4 MiB/s rd, 1.9 MiB/s wr, 529 op/s 2021-07-31T02:25:56.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:56 smithi045 conmon[39718]: cluster 2021-07-31T02:25:54.344928 2021-07-31T02:25:56.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:56 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 1658 : cluster [DBG] pgmap v2048: 9 pgs: 9 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.4 MiB/s rd, 1.9 MiB/s wr, 529 op/s 2021-07-31T02:25:58.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:58 smithi154 conmon[34553]: cluster 2021-07-31T02:25:56.345382+0000 mgr.x (mgr.24542) 1659 : cluster [DBG] pgmap v2049: 9 pgs: 9 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.1 MiB/s rd, 1.6 MiB/s wr, 354 op/s 2021-07-31T02:25:58.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:58 smithi154 conmon[34553]: cluster 2021-07-31T02:25: 2021-07-31T02:25:58.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:25:58 smithi154 conmon[34553]: 57.213826+0000 mon.a (mon.0) 1929 : cluster [DBG] osdmap e1042: 8 total, 8 up, 8 in 2021-07-31T02:25:58.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:58 smithi045 conmon[33655]: cluster 2021-07-31T02:25:56.345382+0000 mgr.x 2021-07-31T02:25:58.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:58 smithi045 conmon[33655]: (mgr.24542) 1659 : cluster [DBG] pgmap v2049: 9 pgs: 9 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.1 MiB/s rd, 1.6 MiB/s wr, 354 op/s 2021-07-31T02:25:58.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:25:58 smithi045 conmon[33655]: cluster 2021-07-31T02:25:57.213826+0000 mon.a (mon.0) 1929 : cluster [DBG] osdmap e1042: 8 total, 8 up, 8 in 2021-07-31T02:25:58.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:58 smithi045 conmon[39718]: cluster 2021-07-31T 2021-07-31T02:25:58.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:58 smithi045 conmon[39718]: 02:25:56.345382+0000 mgr.x (mgr.24542) 1659 : cluster 2021-07-31T02:25:58.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:58 smithi045 conmon[39718]: [DBG] 2021-07-31T02:25:58.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:58 smithi045 conmon[39718]: pgmap v2049: 9 pgs: 9 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.1 MiB/s rd, 1.6 MiB/s wr, 354 op/s 2021-07-31T02:25:58.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:58 smithi045 conmon[39718]: cluster 2021 2021-07-31T02:25:58.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:58 smithi045 conmon[39718]: -07-31T02:25: 2021-07-31T02:25:58.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:58 smithi045 conmon[39718]: 57.213826+0000 mon.a (mon 2021-07-31T02:25:58.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:58 smithi045 conmon[39718]: .0) 1929 : cluster [DBG] 2021-07-31T02:25:58.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:25:58 smithi045 conmon[39718]: osdmap e1042: 8 total, 8 up, 8 in 2021-07-31T02:25:59.716 DEBUG:teuthology.orchestra.run.smithi045:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2021-07-31T02:25:59.771 INFO:tasks.workunit.client.0.smithi045.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2021-07-31T02:25:59.773 INFO:tasks.workunit.client.0.smithi045.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2021-07-31T02:25:59.773 INFO:tasks.workunit.client.0.smithi045.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2021-07-31T02:25:59.820 INFO:tasks.workunit.client.0.smithi045.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2021-07-31T02:25:59.868 INFO:tasks.workunit.client.0.smithi045.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2021-07-31T02:25:59.906 INFO:tasks.workunit.client.0.smithi045.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2021-07-31T02:25:59.907 INFO:tasks.workunit.client.0.smithi045.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2021-07-31T02:25:59.908 INFO:tasks.workunit.client.0.smithi045.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2021-07-31T02:25:59.947 INFO:tasks.workunit.client.0.smithi045.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2021-07-31T02:25:59.948 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2021-07-31T02:25:59.948 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2021-07-31T02:26:00.004 INFO:tasks.workunit:Running workunits matching rbd/test_librbd.sh on client.0... 2021-07-31T02:26:00.005 INFO:tasks.workunit:Running workunit rbd/test_librbd.sh... 2021-07-31T02:26:00.005 DEBUG:teuthology.orchestra.run.smithi045:workunit test rbd/test_librbd.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=pacific TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_librbd.sh 2021-07-31T02:26:00.089 INFO:tasks.workunit.client.0.smithi045.stdout:seed 78694 2021-07-31T02:26:00.090 INFO:tasks.workunit.client.0.smithi045.stdout:[==========] Running 359 tests from 15 test suites. 2021-07-31T02:26:00.090 INFO:tasks.workunit.client.0.smithi045.stdout:[----------] Global test environment set-up. 2021-07-31T02:26:00.090 INFO:tasks.workunit.client.0.smithi045.stdout:[----------] 124 tests from TestLibRBD 2021-07-31T02:26:00.116 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.CreateAndStat 2021-07-31T02:26:01.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:00 smithi154 conmon[34553]: cluster 2021-07-31T02:25:58.345913+0000 mgr.x (mgr.24542) 1660 : cluster [DBG] pgmap v2051: 9 pgs: 9 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1001 KiB/s rd, 1.4 MiB/s wr, 311 op/s 2021-07-31T02:26:01.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:00 smithi045 conmon[33655]: cluster 2021-07-31T02:25:58.345913+0000 2021-07-31T02:26:01.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:00 smithi045 conmon[33655]: mgr.x (mgr.24542) 1660 : cluster [DBG] pgmap v2051: 9 pgs: 9 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1001 KiB/s rd, 1.4 MiB/s wr, 311 op/s 2021-07-31T02:26:01.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:00 smithi045 conmon[39718]: cluster 2021-07-31T02:25:58.345913+0000 2021-07-31T02:26:01.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:00 smithi045 conmon[39718]: mgr.x (mgr.24542) 1660 : cluster [DBG] pgmap v2051: 9 pgs: 9 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1001 KiB/s rd, 1.4 MiB/s wr, 311 op/s 2021-07-31T02:26:01.653 INFO:tasks.workunit.client.0.smithi045.stdout:using old format 2021-07-31T02:26:01.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:01 smithi045 conmon[33655]: cluster 2021-07-31T02:26:01.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:01 smithi045 conmon[33655]: 2021-07-31T02:26:01.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:01 smithi045 conmon[33655]: 2021-07-31T02:26:00.346509+0000 mgr.x (mgr.24542) 1661 : cluster [DBG] pgmap v2052: 9 pgs: 9 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 998 KiB/s rd, 1.4 MiB/s wr, 324 op/s 2021-07-31T02:26:01.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:01 smithi045 conmon[33655]: cluster 2021-07-31T02:26:00.636095+0000 mon.a (mon.0) 1930 : cluster [DBG] osdmap e1043: 8 total, 8 up, 8 in 2021-07-31T02:26:01.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:01 smithi045 conmon[33655]: audit 2021-07-31T02:26:00.637106+0000 mon.a (mon.0) 1931 : audit [INF] from='client.? 172.21.15.45:0/3324205734' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:26:01.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:01 smithi045 conmon[39718]: cluster 2021-07-31T02:26:01.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:01 smithi045 conmon[39718]: 2021-07-31T02 2021-07-31T02:26:01.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:01 smithi045 conmon[39718]: :26:00.346509+0000 mgr.x (mgr.24542) 1661 : cluster [DBG] pgmap v2052: 9 pgs: 9 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 998 KiB/s rd, 1.4 MiB/s wr, 324 op/s 2021-07-31T02:26:01.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:01 smithi045 conmon[39718]: cluster 2021-07-31T02:26:01.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:01 smithi045 conmon[39718]: 2021-07-31T 2021-07-31T02:26:01.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:01 smithi045 conmon[39718]: 02:26:00.636095+0000 mon.a (mon.0 2021-07-31T02:26:01.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:01 smithi045 conmon[39718]: ) 1930 : cluster [DBG] osdmap e1043: 8 total, 8 up, 8 in 2021-07-31T02:26:01.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:01 smithi045 conmon[39718]: audit 2021-07-31T02:26:00.637106+0000 2021-07-31T02:26:01.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:01 smithi045 conmon[39718]: mon.a (mon.0) 1931 : audit [INF] from='client.? 172.21.15.45:0/3324205734' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:26:02.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:01 smithi154 conmon[34553]: cluster 2021-07-31T02:26:00.346509+0000 mgr.x (mgr.24542) 1661 : cluster [DBG] 2021-07-31T02:26:02.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:01 smithi154 conmon[34553]: pgmap v2052: 9 pgs: 9 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 998 KiB/s rd, 1.4 MiB/s wr, 324 op/s 2021-07-31T02:26:02.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:01 smithi154 conmon[34553]: cluster 2021-07-31T02:26:00.636095+0000 mon.a (mon.0) 1930 : cluster [DBG] osdmap e1043: 8 total, 8 up, 8 in 2021-07-31T02:26:02.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:01 smithi154 conmon[34553]: audit 2021-07-31T02:26:00.637106+0000 2021-07-31T02:26:02.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:01 smithi154 conmon[34553]: mon.a (mon.0) 1931 : audit [INF] from='client.? 172.21.15.45:0/3324205734' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:26:03.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:02 smithi154 conmon[34553]: audit 2021-07-31T02:26:01.638866+0000 mon.a (mon.0) 1932 : audit [INF] from='client.? 172.21.15.45:0/3324205734' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-1","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:26:03.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:02 smithi154 conmon[34553]: cluster 2021-07-31T02:26:01.639130+0000 mon.a (mon.0) 1933 : cluster [DBG] osdmap e1044: 8 total, 8 up, 8 in 2021-07-31T02:26:03.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:02 smithi045 conmon[33655]: audit 2021-07-31T02:26:01.638866+0000 mon.a (mon.0) 1932 : audit [INF] from='client.? 172.21.15.45:0/3324205734' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-1","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:26:03.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:02 smithi045 conmon[33655]: cluster 2021-07-31T02:26:01.639130+0000 mon.a (mon.0) 1933 : cluster [DBG] osdmap e1044: 8 total, 8 up, 8 in 2021-07-31T02:26:03.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:02 smithi045 conmon[39718]: audit 2021-07-31T02:26:01.638866+0000 mon.a (mon.0) 1932 : audit [INF] 2021-07-31T02:26:03.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:02 smithi045 conmon[39718]: from='client.? 172.21.15.45:0/3324205734' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-1","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:26:03.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:02 smithi045 conmon[39718]: cluster 2021-07-31T02:26:01.639130+0000 mon.a (mon.0) 1933 : cluster [DBG] osdmap e1044: 8 total, 8 up, 8 in 2021-07-31T02:26:03.680 INFO:tasks.workunit.client.0.smithi045.stdout:image has size 2097152 and order 22 2021-07-31T02:26:03.681 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.CreateAndStat (3565 ms) 2021-07-31T02:26:03.681 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.CreateWithSameDataPool 2021-07-31T02:26:03.681 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:26:03.682 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.CreateWithSameDataPool (0 ms) 2021-07-31T02:26:03.682 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.CreateAndStatPP 2021-07-31T02:26:03.682 INFO:tasks.workunit.client.0.smithi045.stdout:using old format 2021-07-31T02:26:03.691 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.CreateAndStatPP (11 ms) 2021-07-31T02:26:03.692 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.GetId 2021-07-31T02:26:03.692 INFO:tasks.workunit.client.0.smithi045.stdout:using old format 2021-07-31T02:26:03.706 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.GetId (15 ms) 2021-07-31T02:26:03.706 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.GetIdPP 2021-07-31T02:26:03.706 INFO:tasks.workunit.client.0.smithi045.stdout:using old format 2021-07-31T02:26:03.718 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.GetIdPP (12 ms) 2021-07-31T02:26:03.719 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.GetBlockNamePrefix 2021-07-31T02:26:03.719 INFO:tasks.workunit.client.0.smithi045.stdout:using old format 2021-07-31T02:26:03.728 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.GetBlockNamePrefix (10 ms) 2021-07-31T02:26:03.728 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.GetBlockNamePrefixPP 2021-07-31T02:26:03.728 INFO:tasks.workunit.client.0.smithi045.stdout:using old format 2021-07-31T02:26:03.738 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.GetBlockNamePrefixPP (9 ms) 2021-07-31T02:26:03.738 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.TestGetCreateTimestamp 2021-07-31T02:26:03.738 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:26:03.738 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.TestGetCreateTimestamp (0 ms) 2021-07-31T02:26:03.739 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.GetCreateTimestampPP 2021-07-31T02:26:03.739 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:26:03.739 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.GetCreateTimestampPP (0 ms) 2021-07-31T02:26:03.739 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.OpenAio 2021-07-31T02:26:03.739 INFO:tasks.workunit.client.0.smithi045.stdout:using old format 2021-07-31T02:26:03.747 INFO:tasks.workunit.client.0.smithi045.stdout:image has size 2097152 and order 22 2021-07-31T02:26:03.748 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.OpenAio (10 ms) 2021-07-31T02:26:03.748 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.OpenAioFail 2021-07-31T02:26:03.750 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.OpenAioFail (3 ms) 2021-07-31T02:26:03.750 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.OpenAioPP 2021-07-31T02:26:03.750 INFO:tasks.workunit.client.0.smithi045.stdout:using old format 2021-07-31T02:26:03.765 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.OpenAioPP (15 ms) 2021-07-31T02:26:03.766 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.OpenAioFailPP 2021-07-31T02:26:03.769 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.OpenAioFailPP (3 ms) 2021-07-31T02:26:03.769 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.ResizeAndStat 2021-07-31T02:26:03.769 INFO:tasks.workunit.client.0.smithi045.stdout:using old format 2021-07-31T02:26:03.789 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.ResizeAndStat (21 ms) 2021-07-31T02:26:03.789 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.ResizeAndStatPP 2021-07-31T02:26:03.789 INFO:tasks.workunit.client.0.smithi045.stdout:using old format 2021-07-31T02:26:03.807 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.ResizeAndStatPP (18 ms) 2021-07-31T02:26:03.807 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.UpdateWatchAndResize 2021-07-31T02:26:03.808 INFO:tasks.workunit.client.0.smithi045.stdout:using old format 2021-07-31T02:26:03.824 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.UpdateWatchAndResize (17 ms) 2021-07-31T02:26:03.825 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.UpdateWatchAndResizePP 2021-07-31T02:26:03.825 INFO:tasks.workunit.client.0.smithi045.stdout:using old format 2021-07-31T02:26:03.844 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.UpdateWatchAndResizePP (20 ms) 2021-07-31T02:26:03.844 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.TestCreateLsDelete 2021-07-31T02:26:04.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:03 smithi154 conmon[34553]: cluster 2021-07-31T02:26:04.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:03 smithi154 conmon[34553]: 2021-07-31T02:26:02.346986 2021-07-31T02:26:04.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:03 smithi154 conmon[34553]: +0000 mgr.x (mgr.24542) 1662 : cluster [DBG] pgmap v2055: 41 pgs: 32 unknown, 9 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 18 KiB/s rd, 511 B/s wr, 24 op/s 2021-07-31T02:26:04.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:03 smithi154 conmon[34553]: cluster 2021-07-31 2021-07-31T02:26:04.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:03 smithi154 conmon[34553]: T02:26:02.651778+0000 mon.a (mon.0) 1934 : cluster [DBG] osdmap e1045: 8 total, 8 up, 8 in 2021-07-31T02:26:04.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:26:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:26:03.576Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:26:04.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:26:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:26:03.576Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:26:04.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:26:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:26:03.576Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:26:04.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:03 smithi045 conmon[39718]: cluster 2021-07-31T02:26:02.346986+0000 mgr.x (mgr 2021-07-31T02:26:04.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:03 smithi045 conmon[39718]: .24542) 1662 : cluster [DBG] pgmap v2055: 41 pgs: 32 unknown, 9 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 18 KiB/s rd, 511 B/s wr, 24 op/s 2021-07-31T02:26:04.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:03 smithi045 conmon[39718]: cluster 2021-07-31T02:26:02.651778+0000 mon.a (mon.0) 1934 : cluster [DBG] osdmap e1045: 8 total, 8 up, 8 in 2021-07-31T02:26:04.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:03 smithi045 conmon[33655]: cluster 2021-07-31T02:26:02.346986+0000 mgr.x ( 2021-07-31T02:26:04.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:03 smithi045 conmon[33655]: mgr.24542) 1662 : cluster [DBG] pgmap v2055: 41 pgs: 32 unknown, 9 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 18 KiB/s rd, 511 B/s wr, 24 op/s 2021-07-31T02:26:04.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:03 smithi045 conmon[33655]: cluster 2021-07-31T02:26:04.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:03 smithi045 conmon[33655]: 2021 2021-07-31T02:26:04.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:03 smithi045 conmon[33655]: -07-31T02:26 2021-07-31T02:26:04.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:03 smithi045 conmon[33655]: :02.651778+0000 mon.a (mon.0) 1934 : cluster [DBG] osdmap e1045: 8 total, 8 up, 8 in 2021-07-31T02:26:05.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:04 smithi154 conmon[34553]: cluster 2021-07-31T02:26:05.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:04 smithi154 conmon[34553]: 2021-07-31T02:26:05.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:04 smithi154 conmon[34553]: 2021 2021-07-31T02:26:05.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:04 smithi154 conmon[34553]: - 2021-07-31T02:26:05.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:04 smithi154 conmon[34553]: 07 2021-07-31T02:26:05.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:04 smithi154 conmon[34553]: -31T02:26:03.660170+0000 mon.a (mon.0) 1935 : cluster 2021-07-31T02:26:05.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:04 smithi154 conmon[34553]: [DBG] osdmap e1046: 8 total, 8 up, 8 in 2021-07-31T02:26:05.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:04 smithi045 conmon[39718]: cluster 2021-07-31T02:26:03.660170+0000 mon.a 2021-07-31T02:26:05.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:04 smithi045 conmon[39718]: (mon.0) 1935 : cluster [DBG] osdmap e1046: 8 total, 8 up, 8 in 2021-07-31T02:26:05.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:04 smithi045 conmon[33655]: cluster 2021-07-31T02:26:03.660170 2021-07-31T02:26:05.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:04 smithi045 conmon[33655]: +0000 mon.a (mon.0) 1935 : cluster [DBG] osdmap e1046: 8 total, 8 up, 8 in 2021-07-31T02:26:05.698 INFO:tasks.workunit.client.0.smithi045.stdout:using old format 2021-07-31T02:26:06.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:05 smithi154 conmon[34553]: cluster 2021-07-31T02:26:04.347598+0000 mgr.x (mgr.24542) 1663 : cluster 2021-07-31T02:26:06.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:05 smithi154 conmon[34553]: [DBG] pgmap v2058: 41 pgs: 41 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 16 KiB/s rd, 8.2 KiB/s wr, 36 op/s 2021-07-31T02:26:06.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:05 smithi154 conmon[34553]: cluster 2021-07-31T02:26:04.674406+0000 mon.a (mon.0) 1936 : cluster [DBG] osdmap e1047: 8 total, 8 up, 8 in 2021-07-31T02:26:06.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:05 smithi154 conmon[34553]: audit 2021-07-31T02:26:04.675854+0000 mon.c (mon.1 2021-07-31T02:26:06.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:05 smithi154 conmon[34553]: ) 96 : audit [INF] from='client.? 172.21.15.45:0/837641511' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:26:06.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:05 smithi154 conmon[34553]: audit 2021-07-31T02:26:06.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:05 smithi154 conmon[34553]: 2021-07- 2021-07-31T02:26:06.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:05 smithi154 conmon[34553]: 31T02:26:04.676655+0000 mon.a (mon.0) 1937 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:26:06.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[39718]: cluster 2021-07-31T02:26:04.347598+0000 mgr.x (mgr. 2021-07-31T02:26:06.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[39718]: 24542) 1663 : cluster [DBG] pgmap v2058: 41 pgs: 41 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 16 KiB/s rd, 8.2 KiB/s wr, 36 op/s 2021-07-31T02:26:06.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[39718]: cluster 2021-07-31T02:26:04.674406+0000 mon.a (mon.0) 1936 : 2021-07-31T02:26:06.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[39718]: cluster [DBG] osdmap e1047: 8 total, 8 up, 8 in 2021-07-31T02:26:06.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[39718]: audit 2021-07-31T02:26:04.675854+0000 2021-07-31T02:26:06.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[39718]: mon.c (mon.1) 96 : audit [INF] from='client.? 172.21.15.45:0/837641511' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:26:06.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[39718]: audit 2021-07-31T02:26 2021-07-31T02:26:06.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[39718]: :04.676655+0000 mon.a (mon.0) 1937 : audit 2021-07-31T02:26:06.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[39718]: [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:26:06.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: cluster 2021-07-31T02:26:06.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: 2021-07-31T02:26:06.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: 2021 2021-07-31T02:26:06.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: - 2021-07-31T02:26:06.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: 07 2021-07-31T02:26:06.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: - 2021-07-31T02:26:06.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: 31 2021-07-31T02:26:06.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: T 2021-07-31T02:26:06.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: 02 2021-07-31T02:26:06.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: : 2021-07-31T02:26:06.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: 26 2021-07-31T02:26:06.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: : 2021-07-31T02:26:06.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: 04 2021-07-31T02:26:06.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: . 2021-07-31T02:26:06.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: 347598 2021-07-31T02:26:06.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: +0000 2021-07-31T02:26:06.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: 2021-07-31T02:26:06.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: mgr.x 2021-07-31T02:26:06.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: ( 2021-07-31T02:26:06.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: mgr 2021-07-31T02:26:06.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: . 2021-07-31T02:26:06.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: 24542 2021-07-31T02:26:06.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: ) 2021-07-31T02:26:06.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: 1663 2021-07-31T02:26:06.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: : 2021-07-31T02:26:06.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: cluster 2021-07-31T02:26:06.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: 2021-07-31T02:26:06.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: [DBG] 2021-07-31T02:26:06.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: 2021-07-31T02:26:06.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: pgmap v2058: 41 pgs: 41 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 16 KiB/s rd, 8.2 KiB/s wr, 36 op/s 2021-07-31T02:26:06.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: 2021-07-31T02:26:06.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: cluster 2021-07-31T02:26:06.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: 2021-07-31T02:26:06.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: 2021 2021-07-31T02:26:06.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: - 2021-07-31T02:26:06.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: 07 2021-07-31T02:26:06.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: - 2021-07-31T02:26:06.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: 31 2021-07-31T02:26:06.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: T 2021-07-31T02:26:06.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: 02 2021-07-31T02:26:06.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: : 2021-07-31T02:26:06.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: 26 2021-07-31T02:26:06.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: : 2021-07-31T02:26:06.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: 04 2021-07-31T02:26:06.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: . 2021-07-31T02:26:06.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: 674406 2021-07-31T02:26:06.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: +0000 2021-07-31T02:26:06.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: 2021-07-31T02:26:06.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: mon.a 2021-07-31T02:26:06.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: ( 2021-07-31T02:26:06.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: mon 2021-07-31T02:26:06.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: . 2021-07-31T02:26:06.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: 0 2021-07-31T02:26:06.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: ) 2021-07-31T02:26:06.040 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: 1936 2021-07-31T02:26:06.040 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: : 2021-07-31T02:26:06.040 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: cluster 2021-07-31T02:26:06.040 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: 2021-07-31T02:26:06.040 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: [DBG] 2021-07-31T02:26:06.041 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: 2021-07-31T02:26:06.041 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: osdmap e1047: 8 total, 8 up, 8 in 2021-07-31T02:26:06.041 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: 2021-07-31T02:26:06.041 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: audit 2021-07-31T02:26:06.041 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: 2021-07-31T02:26:06.042 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: 2021 2021-07-31T02:26:06.042 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: - 2021-07-31T02:26:06.043 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: 07 2021-07-31T02:26:06.043 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: - 2021-07-31T02:26:06.043 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: 31 2021-07-31T02:26:06.043 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: T 2021-07-31T02:26:06.044 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: 02 2021-07-31T02:26:06.044 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: : 2021-07-31T02:26:06.044 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: 26 2021-07-31T02:26:06.044 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: : 2021-07-31T02:26:06.044 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: 04 2021-07-31T02:26:06.045 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: . 2021-07-31T02:26:06.045 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: 675854 2021-07-31T02:26:06.045 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: +0000 2021-07-31T02:26:06.045 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: 2021-07-31T02:26:06.045 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: mon.c 2021-07-31T02:26:06.046 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: ( 2021-07-31T02:26:06.046 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: mon 2021-07-31T02:26:06.046 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: . 2021-07-31T02:26:06.046 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: 1 2021-07-31T02:26:06.047 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: ) 2021-07-31T02:26:06.047 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: 96 2021-07-31T02:26:06.047 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: : 2021-07-31T02:26:06.047 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: audit 2021-07-31T02:26:06.048 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: [INF] from='client.? 172.21.15.45:0/837641511' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:26:06.048 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: audit 2021- 2021-07-31T02:26:06.048 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: 07-31T02:26:04.676655+0000 mon.a (mon.0) 1937 : audit 2021-07-31T02:26:06.048 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:05 smithi045 conmon[33655]: [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:26:06.189 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:26:06.466 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:26:06.466 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 40m ago 41m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:26:06.466 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 38m ago 41m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:26:06.467 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 38m ago 45m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:26:06.467 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 40m ago 47m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:26:06.467 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 40m ago 47m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:26:06.467 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 38m ago 45m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:26:06.468 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 40m ago 45m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:26:06.468 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 40m ago 41m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:26:06.468 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 38m ago 41m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:26:06.468 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 40m ago 44m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:26:06.468 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 40m ago 44m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:26:06.469 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 40m ago 44m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:26:06.469 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 40m ago 43m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:26:06.469 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 38m ago 43m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:26:06.469 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 38m ago 43m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:26:06.470 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 38m ago 42m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:26:06.470 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 38m ago 42m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:26:06.470 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 38m ago 42m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:26:06.806 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:26:06.806 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:26:06.807 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:26:06.807 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:26:06.807 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:26:06.808 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:26:06.808 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:26:06.808 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:26:06.808 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:26:06.808 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:26:06.809 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:26:06.809 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:26:06.810 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:26:06.810 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:26:06.810 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:26:06.810 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:26:06.811 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:26:07.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:06 smithi154 conmon[34553]: audit 2021-07-31T02:26:05.676818+0000 mon.a (mon.0) 1938 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-2","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:26:07.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:06 smithi154 conmon[34553]: cluster 2021-07-31T02:26:05.677140+0000 mon.a 2021-07-31T02:26:07.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:06 smithi154 conmon[34553]: (mon.0) 1939 : cluster [DBG] osdmap e1048: 8 total, 8 up, 8 in 2021-07-31T02:26:07.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:06 smithi154 conmon[34553]: audit 2021-07-31T02:26:06.177691+0000 mgr.x (mgr.24542) 1664 : audit [DBG] from='client.17046 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:26:07.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:06 smithi154 conmon[34553]: cluster 2021-07-31T02:26:06.677038+0000 mon.a (mon.0) 1940 : cluster [DBG] osdmap e1049: 8 total, 8 up, 8 in 2021-07-31T02:26:07.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:06 smithi045 conmon[33655]: audit 2021-07-31T02:26:05.676818+0000 mon.a (mon 2021-07-31T02:26:07.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:06 smithi045 conmon[33655]: .0) 1938 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-2","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:26:07.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:06 smithi045 conmon[33655]: cluster 2021-07-31T02:26:05.677140+0000 mon.a (mon.0) 1939 : cluster [DBG] osdmap e1048: 8 total, 8 up, 8 in 2021-07-31T02:26:07.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:06 smithi045 conmon[33655]: audit 2021-07-31T02:26:06.177691+0000 2021-07-31T02:26:07.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:06 smithi045 conmon[33655]: mgr.x (mgr.24542) 1664 : audit [DBG] from='client.17046 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:26:07.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:06 smithi045 conmon[33655]: cluster 2021-07-31T02:26:07.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:06 smithi045 conmon[33655]: 2021-07-31T02: 2021-07-31T02:26:07.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:06 smithi045 conmon[33655]: 26:06.677038+0000 mon.a 2021-07-31T02:26:07.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:06 smithi045 conmon[33655]: (mon.0) 1940 : 2021-07-31T02:26:07.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:06 smithi045 conmon[33655]: cluster [DBG] osdmap e1049: 8 total, 8 up, 8 in 2021-07-31T02:26:07.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:06 smithi045 conmon[39718]: audit 2021-07-31T02:26:05.676818 2021-07-31T02:26:07.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:06 smithi045 conmon[39718]: +0000 mon.a (mon.0) 1938 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-2","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:26:07.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:06 smithi045 conmon[39718]: cluster 2021-07-31T02:26:07.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:06 smithi045 conmon[39718]: 2021-07-31T02:26:05.677140+0000 mon.a (mon.0) 1939 : cluster 2021-07-31T02:26:07.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:06 smithi045 conmon[39718]: [DBG] osdmap e1048: 8 total, 8 up, 8 in 2021-07-31T02:26:07.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:06 smithi045 conmon[39718]: audit 2021-07-31T02:26:06. 2021-07-31T02:26:07.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:06 smithi045 conmon[39718]: 177691+0000 mgr.x (mgr.24542) 1664 : audit 2021-07-31T02:26:07.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:06 smithi045 conmon[39718]: [DBG] from='client.17046 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:26:07.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:06 smithi045 conmon[39718]: cluster 2021-07- 2021-07-31T02:26:07.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:06 smithi045 conmon[39718]: 31T02:26:06.677038+0000 mon.a ( 2021-07-31T02:26:07.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:06 smithi045 conmon[39718]: mon.0) 1940 : cluster [DBG] osdmap e1049: 8 total, 8 up, 8 in 2021-07-31T02:26:07.236 INFO:tasks.workunit.client.0.smithi045.stdout:image: image15 2021-07-31T02:26:07.236 INFO:tasks.workunit.client.0.smithi045.stdout:expected = image15 2021-07-31T02:26:07.236 INFO:tasks.workunit.client.0.smithi045.stdout:found image15 2021-07-31T02:26:07.236 INFO:tasks.workunit.client.0.smithi045.stdout:erased image15 2021-07-31T02:26:07.237 INFO:tasks.workunit.client.0.smithi045.stdout:using old format 2021-07-31T02:26:07.269 INFO:tasks.workunit.client.0.smithi045.stdout:image: image15 2021-07-31T02:26:07.270 INFO:tasks.workunit.client.0.smithi045.stdout:image: image16 2021-07-31T02:26:07.270 INFO:tasks.workunit.client.0.smithi045.stdout:expected = image15 2021-07-31T02:26:07.270 INFO:tasks.workunit.client.0.smithi045.stdout:found image15 2021-07-31T02:26:07.271 INFO:tasks.workunit.client.0.smithi045.stdout:erased image15 2021-07-31T02:26:07.271 INFO:tasks.workunit.client.0.smithi045.stdout:expected = image16 2021-07-31T02:26:07.271 INFO:tasks.workunit.client.0.smithi045.stdout:found image16 2021-07-31T02:26:07.271 INFO:tasks.workunit.client.0.smithi045.stdout:erased image16 2021-07-31T02:26:07.271 INFO:tasks.workunit.client.0.smithi045.stdout:image: image16 2021-07-31T02:26:07.272 INFO:tasks.workunit.client.0.smithi045.stdout:expected = image16 2021-07-31T02:26:07.272 INFO:tasks.workunit.client.0.smithi045.stdout:found image16 2021-07-31T02:26:07.272 INFO:tasks.workunit.client.0.smithi045.stdout:erased image16 2021-07-31T02:26:07.272 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.TestCreateLsDelete (3425 ms) 2021-07-31T02:26:07.272 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.TestCreateLsDeletePP 2021-07-31T02:26:08.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:07 smithi154 conmon[34553]: audit 2021-07-31T02:26:06.806594+0000 mon.a (mon.0) 1941 : audit [DBG] 2021-07-31T02:26:08.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:07 smithi154 conmon[34553]: from='client.? 172.21.15.45:0/1580261200' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:26:08.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:07 smithi154 conmon[34553]: cluster 2021-07-31T02:26:07.216727+0000 mon.a (mon.0) 1942 : cluster [DBG] osdmap e1050: 8 total, 8 up, 8 in 2021-07-31T02:26:08.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:07 smithi045 conmon[33655]: audit 2021-07-31T02:26:06.806594 2021-07-31T02:26:08.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:07 smithi045 conmon[33655]: +0000 mon.a (mon.0) 1941 : audit [DBG] from='client.? 172.21.15.45:0/1580261200' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:26:08.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:07 smithi045 conmon[33655]: cluster 2021-07-31T02:26:07 2021-07-31T02:26:08.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:07 smithi045 conmon[33655]: .216727+0000 mon.a (mon.0) 1942 : cluster [DBG] osdmap e1050: 8 total, 8 up, 8 in 2021-07-31T02:26:08.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:07 smithi045 conmon[39718]: audit 2021-07-31T02:26:06.806594 2021-07-31T02:26:08.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:07 smithi045 conmon[39718]: +0000 mon.a (mon.0) 1941 : audit [DBG] from='client.? 172.21.15.45:0/1580261200' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:26:08.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:07 smithi045 conmon[39718]: cluster 2021-07-31T02:26:07 2021-07-31T02:26:08.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:07 smithi045 conmon[39718]: .216727+0000 mon.a (mon.0) 1942 : 2021-07-31T02:26:08.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:07 smithi045 conmon[39718]: cluster [DBG] osdmap e1050: 8 total, 8 up, 8 in 2021-07-31T02:26:09.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:08 smithi154 conmon[34553]: cluster 2021-07-31T02:26:06.348104+0000 mgr.x (mgr.24542) 1665 : cluster [DBG] pgmap v2061: 73 pgs: 32 unknown, 41 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 16 KiB/s rd, 8.2 KiB/s wr, 36 op/s 2021-07-31T02:26:09.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:08 smithi154 conmon[34553]: audit 2021-07-31T02:26:06.458372+0000 mgr.x (mgr.24542) 1666 : audit [DBG] from='client.17052 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:26:09.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:08 smithi154 conmon[34553]: cluster 2021-07-31T02:26:08.223141+0000 mon.a (mon.0) 1943 : cluster [DBG] osdmap e1051: 8 total, 8 up, 8 in 2021-07-31T02:26:09.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:08 smithi154 conmon[34553]: audit 2021 2021-07-31T02:26:09.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:08 smithi154 conmon[34553]: -07-31T02:26:08.225193+0000 mon.a (mon.0) 1944 : audit [INF] from='client.? 172.21.15.45:0/526721409' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:26:09.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:08 smithi045 conmon[39718]: cluster 2021-07-31T02:26:06.348104+0000 2021-07-31T02:26:09.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:08 smithi045 conmon[39718]: mgr.x (mgr.24542) 1665 : cluster [DBG] pgmap v2061: 73 pgs: 32 unknown, 41 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 16 KiB/s rd, 8.2 KiB/s wr, 36 op/s 2021-07-31T02:26:09.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:08 smithi045 conmon[39718]: audit 2021-07-31T02:26:09.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:08 smithi045 conmon[39718]: 2021-07-31T02:26:06. 2021-07-31T02:26:09.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:08 smithi045 conmon[39718]: 458372+0000 mgr.x (mgr.24542) 2021-07-31T02:26:09.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:08 smithi045 conmon[39718]: 1666 : audit [DBG] from='client.17052 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:26:09.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:08 smithi045 conmon[39718]: cluster 2021-07- 2021-07-31T02:26:09.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:08 smithi045 conmon[39718]: 31T02:26:08.223141+0000 2021-07-31T02:26:09.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:08 smithi045 conmon[39718]: mon.a (mon.0) 1943 2021-07-31T02:26:09.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:08 smithi045 conmon[39718]: : cluster [DBG] osdmap e1051: 8 total, 8 up, 8 in 2021-07-31T02:26:09.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:08 smithi045 conmon[39718]: audit 2021- 2021-07-31T02:26:09.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:08 smithi045 conmon[39718]: 07-31T02:26:08. 2021-07-31T02:26:09.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:08 smithi045 conmon[39718]: 225193+0000 mon.a (mon.0) 1944 2021-07-31T02:26:09.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:08 smithi045 conmon[39718]: : audit [INF] from='client.? 172.21.15.45:0/526721409' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:26:09.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:08 smithi045 conmon[33655]: cluster 2021-07-31T02:26:06.348104+0000 mgr.x (mgr.24542) 1665 : cluster 2021-07-31T02:26:09.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:08 smithi045 conmon[33655]: [DBG] pgmap v2061: 73 pgs: 32 unknown, 41 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 16 KiB/s rd, 8.2 KiB/s wr, 36 op/s 2021-07-31T02:26:09.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:08 smithi045 conmon[33655]: audit 2021-07-31T02:26:06.458372+0000 mgr.x (mgr.24542) 1666 : audit [DBG] from='client.17052 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:26:09.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:08 smithi045 conmon[33655]: cluster 2021-07-31T02 2021-07-31T02:26:09.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:08 smithi045 conmon[33655]: :26:08.223141+0000 mon.a (mon.0) 1943 : cluster [DBG] 2021-07-31T02:26:09.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:08 smithi045 conmon[33655]: osdmap e1051: 8 total, 8 up, 8 in 2021-07-31T02:26:09.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:08 smithi045 conmon[33655]: audit 2021-07-31T02:26:09.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:08 smithi045 conmon[33655]: 2021-07 2021-07-31T02:26:09.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:08 smithi045 conmon[33655]: -31T02:26:08.225193 2021-07-31T02:26:09.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:08 smithi045 conmon[33655]: +0000 mon.a (mon.0) 1944 : audit 2021-07-31T02:26:09.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:08 smithi045 conmon[33655]: [INF] from='client.? 172.21.15.45:0/526721409' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:26:09.232 INFO:tasks.workunit.client.0.smithi045.stdout:using old format 2021-07-31T02:26:10.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:10 smithi154 conmon[34553]: cluster 2021-07-31T02:26:08.348661+0000 mgr.x (mgr.24542) 1667 : cluster [DBG] 2021-07-31T02:26:10.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:10 smithi154 conmon[34553]: pgmap v2065: 105 pgs: 64 unknown, 41 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:26:10.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:10 smithi154 conmon[34553]: audit 2021-07-31T02:26:09.227927+0000 mon.a (mon.0) 1945 : audit [INF] from='client.? 172.21.15.45:0/526721409' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-3","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:26:10.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:10 smithi154 conmon[34553]: cluster 2021-07-31T02:26:10.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:10 smithi154 conmon[34553]: 2021-07-31T02:26:09.228290+0000 mon.a (mon.0) 1946 : cluster [DBG] osdmap e1052: 8 total, 8 up, 8 in 2021-07-31T02:26:10.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:10 smithi045 conmon[39718]: cluster 2021-07-31T02:26:08.348661+0000 mgr.x (mgr.24542) 1667 : cluster [DBG] pgmap v2065: 105 pgs: 64 unknown, 41 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:26:10.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:10 smithi045 conmon[39718]: audit 2021-07-31T02:26:09.227927+0000 mon.a (mon.0) 1945 : audit [INF] from='client.? 172.21.15.45:0/526721409' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-3","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:26:10.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:10 smithi045 conmon[39718]: cluster 2021- 2021-07-31T02:26:10.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:10 smithi045 conmon[39718]: 07-31T02:26:09.228290+0000 mon.a (mon.0) 2021-07-31T02:26:10.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:10 smithi045 conmon[39718]: 1946 : cluster [DBG] osdmap e1052: 8 total, 8 up, 8 in 2021-07-31T02:26:10.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:10 smithi045 conmon[33655]: cluster 2021-07-31T02:26:08. 2021-07-31T02:26:10.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:10 smithi045 conmon[33655]: 348661+0000 mgr.x (mgr.24542) 1667 : cluster [DBG] pgmap v2065: 105 pgs: 64 unknown, 41 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:26:10.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:10 smithi045 conmon[33655]: audit 2021-07-31T02:26:09.227927+0000 mon.a (mon.0) 1945 : audit [INF] from='client.? 172.21.15.45:0/526721409' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-3","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:26:10.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:10 smithi045 conmon[33655]: cluster 2021-07-31T02:26:09.228290+0000 mon.a (mon 2021-07-31T02:26:10.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:10 smithi045 conmon[33655]: .0) 1946 : cluster [DBG] osdmap e1052: 8 total, 8 up, 8 in 2021-07-31T02:26:11.249 INFO:tasks.workunit.client.0.smithi045.stdout:num images is: 1 2021-07-31T02:26:11.250 INFO:tasks.workunit.client.0.smithi045.stdout:expected: 1 2021-07-31T02:26:11.250 INFO:tasks.workunit.client.0.smithi045.stdout:image: image17 2021-07-31T02:26:11.250 INFO:tasks.workunit.client.0.smithi045.stdout:expected = image17 2021-07-31T02:26:11.251 INFO:tasks.workunit.client.0.smithi045.stdout:using old format 2021-07-31T02:26:11.257 INFO:tasks.workunit.client.0.smithi045.stdout:num images is: 2 2021-07-31T02:26:11.258 INFO:tasks.workunit.client.0.smithi045.stdout:expected: 2 2021-07-31T02:26:11.258 INFO:tasks.workunit.client.0.smithi045.stdout:image: image17 2021-07-31T02:26:11.258 INFO:tasks.workunit.client.0.smithi045.stdout:image: image18 2021-07-31T02:26:11.259 INFO:tasks.workunit.client.0.smithi045.stdout:expected = image17 2021-07-31T02:26:11.259 INFO:tasks.workunit.client.0.smithi045.stdout:expected = image18 2021-07-31T02:26:11.272 INFO:tasks.workunit.client.0.smithi045.stdout:num images is: 1 2021-07-31T02:26:11.273 INFO:tasks.workunit.client.0.smithi045.stdout:expected: 1 2021-07-31T02:26:11.273 INFO:tasks.workunit.client.0.smithi045.stdout:image: image18 2021-07-31T02:26:11.273 INFO:tasks.workunit.client.0.smithi045.stdout:expected = image18 2021-07-31T02:26:11.273 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.TestCreateLsDeletePP (4003 ms) 2021-07-31T02:26:11.274 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.TestCopy 2021-07-31T02:26:11.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:11 smithi154 conmon[34553]: cluster 2021-07-31T02:26:11.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:11 smithi154 conmon[34553]: 2021-07-31T02:26:11.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:11 smithi154 conmon[34553]: 2021 2021-07-31T02:26:11.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:11 smithi154 conmon[34553]: - 2021-07-31T02:26:11.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:11 smithi154 conmon[34553]: 07 2021-07-31T02:26:11.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:11 smithi154 conmon[34553]: - 2021-07-31T02:26:11.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:11 smithi154 conmon[34553]: 31 2021-07-31T02:26:11.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:11 smithi154 conmon[34553]: T 2021-07-31T02:26:11.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:11 smithi154 conmon[34553]: 02 2021-07-31T02:26:11.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:11 smithi154 conmon[34553]: : 2021-07-31T02:26:11.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:11 smithi154 conmon[34553]: 26 2021-07-31T02:26:11.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:11 smithi154 conmon[34553]: : 2021-07-31T02:26:11.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:11 smithi154 conmon[34553]: 10 2021-07-31T02:26:11.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:11 smithi154 conmon[34553]: . 2021-07-31T02:26:11.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:11 smithi154 conmon[34553]: 230070 2021-07-31T02:26:11.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:11 smithi154 conmon[34553]: +0000 2021-07-31T02:26:11.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:11 smithi154 conmon[34553]: 2021-07-31T02:26:11.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:11 smithi154 conmon[34553]: mon.a 2021-07-31T02:26:11.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:11 smithi154 conmon[34553]: ( 2021-07-31T02:26:11.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:11 smithi154 conmon[34553]: mon 2021-07-31T02:26:11.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:11 smithi154 conmon[34553]: . 2021-07-31T02:26:11.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:11 smithi154 conmon[34553]: 0 2021-07-31T02:26:11.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:11 smithi154 conmon[34553]: ) 2021-07-31T02:26:11.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:11 smithi154 conmon[34553]: 1947 2021-07-31T02:26:11.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:11 smithi154 conmon[34553]: : 2021-07-31T02:26:11.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:11 smithi154 conmon[34553]: cluster 2021-07-31T02:26:11.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:11 smithi154 conmon[34553]: 2021-07-31T02:26:11.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:11 smithi154 conmon[34553]: [DBG] 2021-07-31T02:26:11.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:11 smithi154 conmon[34553]: 2021-07-31T02:26:11.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:11 smithi154 conmon[34553]: osdmap e1053: 8 total, 8 up, 8 in 2021-07-31T02:26:11.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:11 smithi154 conmon[34553]: 2021-07-31T02:26:11.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:11 smithi045 conmon[39718]: cluster 2021-07-31T02:26:10.230070+0000 mon.a (mon 2021-07-31T02:26:11.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:11 smithi045 conmon[39718]: .0) 1947 : cluster [DBG] osdmap e1053: 8 total, 8 up, 8 in 2021-07-31T02:26:11.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:11 smithi045 conmon[33655]: cluster 2021-07-31T 2021-07-31T02:26:11.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:11 smithi045 conmon[33655]: 02:26:10.230070+0000 mon.a (mon.0) 1947 : cluster [DBG] osdmap e1053: 8 total, 8 up, 8 in 2021-07-31T02:26:12.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:12 smithi154 conmon[34553]: cluster 2021-07-31T02:26:10.349245+0000 mgr.x (mgr.24542) 1668 : cluster [DBG] pgmap v2068: 105 pgs: 32 creating+peering, 73 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.4 KiB/s wr, 5 op/s 2021-07-31T02:26:12.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:12 smithi154 conmon[34553]: cluster 2021-07-31T02:26:11.231808+0000 mon.a (mon.0) 1948 : cluster [DBG] osdmap e1054: 8 total, 8 up, 8 in 2021-07-31T02:26:12.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:12 smithi154 conmon[34553]: cluster 2021-07-31T02:26:12.218535+0000 mon.a (mon.0) 1949 : cluster [DBG] osdmap e1055: 8 total, 8 up, 8 in 2021-07-31T02:26:12.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:12 smithi154 conmon[34553]: audit 2021-07-31T02:26:12.219753+0000 mon.a (mon.0 2021-07-31T02:26:12.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:12 smithi154 conmon[34553]: ) 1950 : audit [INF] from='client.? 172.21.15.45:0/564481794' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:26:12.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:12 smithi045 conmon[33655]: cluster 2021-07-31T02:26:10.349245+0000 mgr.x ( 2021-07-31T02:26:12.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:12 smithi045 conmon[33655]: mgr.24542) 1668 : cluster [DBG] pgmap v2068: 105 pgs: 32 creating+peering, 73 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.4 KiB/s wr, 5 op/s 2021-07-31T02:26:12.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:12 smithi045 conmon[33655]: cluster 2021-07-31T02:26:11.231808+0000 mon.a (mon.0) 1948 : cluster [DBG] osdmap e1054: 8 total, 8 up, 8 in 2021-07-31T02:26:12.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:12 smithi045 conmon[33655]: cluster 2021-07-31T02:26:12.218535+0000 mon.a (mon.0) 1949 : cluster 2021-07-31T02:26:12.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:12 smithi045 conmon[33655]: [DBG] osdmap e1055: 8 total, 8 up, 8 in 2021-07-31T02:26:12.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:12 smithi045 conmon[33655]: audit 2021-07 2021-07-31T02:26:12.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:12 smithi045 conmon[33655]: -31T02:26:12.219753+0000 mon.a (mon.0 2021-07-31T02:26:12.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:12 smithi045 conmon[33655]: ) 1950 : audit [INF] from='client.? 172.21.15.45:0/564481794' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:26:12.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:12 smithi045 conmon[39718]: cluster 2021-07-31T02:26:10.349245 2021-07-31T02:26:12.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:12 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 1668 : cluster [DBG] pgmap v2068: 105 pgs: 32 creating+peering, 73 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.4 KiB/s wr, 5 op/s 2021-07-31T02:26:12.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:12 smithi045 conmon[39718]: cluster 2021-07-31T02:26:12.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:12 smithi045 conmon[39718]: 2021- 2021-07-31T02:26:12.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:12 smithi045 conmon[39718]: 07-31T02:26: 2021-07-31T02:26:12.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:12 smithi045 conmon[39718]: 11.231808+0000 mon.a ( 2021-07-31T02:26:12.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:12 smithi045 conmon[39718]: mon.0) 1948 : cluster [DBG] osdmap e1054: 8 total, 8 up, 8 in 2021-07-31T02:26:12.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:12 smithi045 conmon[39718]: 2021-07-31T02:26:12.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:12 smithi045 conmon[39718]: cluster 2021-07-31T02:26:12.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:12 smithi045 conmon[39718]: 2021-07- 2021-07-31T02:26:12.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:12 smithi045 conmon[39718]: 31T02:26:12.218535 2021-07-31T02:26:12.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:12 smithi045 conmon[39718]: +0000 mon.a (mon.0) 1949 : cluster 2021-07-31T02:26:12.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:12 smithi045 conmon[39718]: [DBG] osdmap e1055: 8 total, 8 up, 8 in 2021-07-31T02:26:12.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:12 smithi045 conmon[39718]: audit 2021-07-31T02:26:12.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:12 smithi045 conmon[39718]: 2021-07-31T02:26 2021-07-31T02:26:12.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:12 smithi045 conmon[39718]: :12.219753+0000 mon.a (mon 2021-07-31T02:26:12.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:12 smithi045 conmon[39718]: .0) 1950 : audit [INF] from='client.? 172.21.15.45:0/564481794' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:26:13.228 INFO:tasks.workunit.client.0.smithi045.stdout:using old format 2021-07-31T02:26:14.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:26:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:26:13.576Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:26:14.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:26:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:26:13.576Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:26:14.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:26:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:26:13.576Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:26:14.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:14 smithi154 conmon[34553]: cluster 2021-07-31T02:26:12.349677+0000 mgr.x (mgr.24542) 1669 : cluster [DBG] pgmap v2071: 137 pgs: 32 unknown, 32 creating+peering, 73 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2021-07-31T02:26:14.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:14 smithi154 conmon[34553]: audit 2021-07-31T02:26:13.220612 2021-07-31T02:26:14.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:14 smithi154 conmon[34553]: +0000 mon.a (mon.0) 1951 : audit [INF] from='client.? 172.21.15.45:0/564481794' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-4","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:26:14.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:14 smithi154 conmon[34553]: cluster 2021-07-31T02:26:13.220866+0000 mon.a (mon.0) 1952 : cluster [DBG] osdmap e1056: 8 total, 8 up, 8 in 2021-07-31T02:26:14.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:14 smithi045 conmon[39718]: cluster 2021-07-31T02:26:12.349677+0000 mgr.x (mgr. 2021-07-31T02:26:14.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:14 smithi045 conmon[39718]: 24542) 1669 : cluster [DBG] pgmap v2071: 137 pgs: 32 unknown, 32 creating+peering, 73 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2021-07-31T02:26:14.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:14 smithi045 conmon[39718]: audit 2021-07-31T02:26:13.220612+0000 mon.a (mon 2021-07-31T02:26:14.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:14 smithi045 conmon[39718]: .0) 1951 : audit [INF] from='client.? 172.21.15.45:0/564481794' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-4","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:26:14.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:14 smithi045 conmon[39718]: cluster 2021-07-31T02:26 2021-07-31T02:26:14.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:14 smithi045 conmon[39718]: :13.220866+0000 mon.a (mon.0) 1952 2021-07-31T02:26:14.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:14 smithi045 conmon[39718]: : cluster [DBG] osdmap e1056: 8 total, 8 up, 8 in 2021-07-31T02:26:14.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:14 smithi045 conmon[33655]: cluster 2021-07-31T02:26:12.349677+0000 2021-07-31T02:26:14.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:14 smithi045 conmon[33655]: mgr.x (mgr.24542) 1669 : cluster [DBG] pgmap v2071: 137 pgs: 32 unknown, 32 creating+peering, 73 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2021-07-31T02:26:14.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:14 smithi045 conmon[33655]: audit 2021-07-31T02:26:13. 2021-07-31T02:26:14.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:14 smithi045 conmon[33655]: 220612+0000 mon.a (mon.0) 1951 : audit [INF] from='client.? 172.21.15.45:0/564481794' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-4","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:26:14.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:14 smithi045 conmon[33655]: cluster 2021-07-31T02:26:14.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:14 smithi045 conmon[33655]: 2021-07-31T02: 2021-07-31T02:26:14.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:14 smithi045 conmon[33655]: 26:13.220866+0000 mon.a 2021-07-31T02:26:14.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:14 smithi045 conmon[33655]: (mon.0) 1952 : cluster [DBG] 2021-07-31T02:26:14.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:14 smithi045 conmon[33655]: osdmap e1056: 8 total, 8 up, 8 in 2021-07-31T02:26:15.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:15 smithi154 conmon[34553]: cluster 2021-07-31T02:26:14.226648+0000 mon.a (mon.0) 1953 : 2021-07-31T02:26:15.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:15 smithi154 conmon[34553]: cluster [DBG] osdmap e1057: 8 total, 8 up, 8 in 2021-07-31T02:26:15.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:15 smithi045 conmon[39718]: cluster 2021-07-31T02:26:15.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:15 smithi045 conmon[39718]: 2021-07-31T02:26:14.226648+0000 mon.a ( 2021-07-31T02:26:15.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:15 smithi045 conmon[39718]: mon.0) 1953 : cluster [DBG] osdmap e1057: 8 total, 8 up, 8 in 2021-07-31T02:26:15.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:15 smithi045 conmon[33655]: cluster 2021-07-31T02:26:14.226648+0000 mon.a (mon.0 2021-07-31T02:26:15.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:15 smithi045 conmon[33655]: ) 1953 : cluster [DBG] osdmap e1057: 8 total, 8 up, 8 in 2021-07-31T02:26:16.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:16 smithi154 conmon[34553]: cluster 2021-07-31T02:26:14.350129+0000 mgr.x (mgr.24542) 1670 : cluster [DBG] pgmap v2074: 137 pgs: 137 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2021-07-31T02:26:16.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:16 smithi154 conmon[34553]: 2021-07-31T02:26:16.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:16 smithi154 conmon[34553]: cluster 2021-07-31T02:26:15.232902+0000 mon.a (mon.0) 1954 : cluster [DBG] osdmap e1058: 8 total, 8 up, 8 in 2021-07-31T02:26:16.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:16 smithi045 conmon[39718]: cluster 2021-07-31T02:26:14.350129+0000 mgr.x (mgr.24542) 1670 : cluster 2021-07-31T02:26:16.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:16 smithi045 conmon[39718]: [DBG] pgmap v2074: 137 pgs: 137 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2021-07-31T02:26:16.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:16 smithi045 conmon[39718]: cluster 2021-07-31T02:26:15.232902+0000 mon.a (mon 2021-07-31T02:26:16.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:16 smithi045 conmon[39718]: .0) 1954 : cluster [DBG] osdmap e1058: 8 total, 8 up, 8 in 2021-07-31T02:26:16.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:16 smithi045 conmon[33655]: cluster 2021-07-31T02:26:14.350129+0000 mgr.x (mgr.24542 2021-07-31T02:26:16.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:16 smithi045 conmon[33655]: ) 1670 : cluster [DBG] pgmap v2074: 137 pgs: 137 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2021-07-31T02:26:16.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:16 smithi045 conmon[33655]: cluster 2021-07-31T02:26:15.232902+0000 mon.a (mon.0) 1954 : cluster [DBG] osdmap e1058: 8 total, 8 up, 8 in 2021-07-31T02:26:17.375 INFO:tasks.workunit.client.0.smithi045.stdout:image: image19 2021-07-31T02:26:17.375 INFO:tasks.workunit.client.0.smithi045.stdout:expected = image19 2021-07-31T02:26:17.376 INFO:tasks.workunit.client.0.smithi045.stdout:found image19 2021-07-31T02:26:17.376 INFO:tasks.workunit.client.0.smithi045.stdout:erased image19 2021-07-31T02:26:17.376 INFO:tasks.workunit.client.0.smithi045.stdout:image: image19 2021-07-31T02:26:17.376 INFO:tasks.workunit.client.0.smithi045.stdout:image: image20 2021-07-31T02:26:17.376 INFO:tasks.workunit.client.0.smithi045.stdout:expected = image19 2021-07-31T02:26:17.377 INFO:tasks.workunit.client.0.smithi045.stdout:found image19 2021-07-31T02:26:17.377 INFO:tasks.workunit.client.0.smithi045.stdout:erased image19 2021-07-31T02:26:17.377 INFO:tasks.workunit.client.0.smithi045.stdout:expected = image20 2021-07-31T02:26:17.377 INFO:tasks.workunit.client.0.smithi045.stdout:found image20 2021-07-31T02:26:17.378 INFO:tasks.workunit.client.0.smithi045.stdout:erased image20 2021-07-31T02:26:17.378 INFO:tasks.workunit.client.0.smithi045.stdout:0.00% done 2021-07-31T02:26:17.378 INFO:tasks.workunit.client.0.smithi045.stdout:100.00% done 2021-07-31T02:26:17.378 INFO:tasks.workunit.client.0.smithi045.stdout:image: image19 2021-07-31T02:26:17.378 INFO:tasks.workunit.client.0.smithi045.stdout:image: image20 2021-07-31T02:26:17.379 INFO:tasks.workunit.client.0.smithi045.stdout:image: image21 2021-07-31T02:26:17.379 INFO:tasks.workunit.client.0.smithi045.stdout:expected = image19 2021-07-31T02:26:17.379 INFO:tasks.workunit.client.0.smithi045.stdout:found image19 2021-07-31T02:26:17.379 INFO:tasks.workunit.client.0.smithi045.stdout:erased image19 2021-07-31T02:26:17.379 INFO:tasks.workunit.client.0.smithi045.stdout:expected = image20 2021-07-31T02:26:17.380 INFO:tasks.workunit.client.0.smithi045.stdout:found image20 2021-07-31T02:26:17.380 INFO:tasks.workunit.client.0.smithi045.stdout:erased image20 2021-07-31T02:26:17.380 INFO:tasks.workunit.client.0.smithi045.stdout:expected = image21 2021-07-31T02:26:17.380 INFO:tasks.workunit.client.0.smithi045.stdout:found image21 2021-07-31T02:26:17.380 INFO:tasks.workunit.client.0.smithi045.stdout:erased image21 2021-07-31T02:26:17.381 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.TestCopy (6103 ms) 2021-07-31T02:26:17.381 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.TestCopyPP 2021-07-31T02:26:17.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:17 smithi154 conmon[34553]: cluster 2021-07-31T02:26:17.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:17 smithi154 conmon[34553]: 2021-07-31T02:26:17.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:17 smithi154 conmon[34553]: 2021 2021-07-31T02:26:17.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:17 smithi154 conmon[34553]: -07-31T02:26:16.234454+0000 2021-07-31T02:26:17.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:17 smithi154 conmon[34553]: mon.a (mon.0) 1955 : cluster [DBG] osdmap e1059: 8 total, 8 up, 8 in 2021-07-31T02:26:17.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:17 smithi154 conmon[34553]: cluster 2021 2021-07-31T02:26:17.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:17 smithi154 conmon[34553]: -07-31T02:26:17.220914 2021-07-31T02:26:17.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:17 smithi154 conmon[34553]: +0000 mon.a (mon.0) 1956 2021-07-31T02:26:17.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:17 smithi154 conmon[34553]: : cluster [DBG] osdmap e1060: 8 total, 8 up, 8 in 2021-07-31T02:26:17.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:17 smithi045 conmon[39718]: cluster 2021-07-31T02:26:17.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:17 smithi045 conmon[39718]: 2021-07-31T02:26:17.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:17 smithi045 conmon[39718]: 2021-07-31T02:26 2021-07-31T02:26:17.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:17 smithi045 conmon[39718]: :16.234454+0000 mon.a (mon.0) 1955 : cluster [DBG] osdmap e1059: 8 total, 8 up, 8 in 2021-07-31T02:26:17.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:17 smithi045 conmon[39718]: cluster 2021-07-31T02:26:17.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:17 smithi045 conmon[39718]: 2021-07-31T02:26:17.220914+0000 mon.a ( 2021-07-31T02:26:17.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:17 smithi045 conmon[39718]: mon.0) 1956 : cluster [DBG] osdmap e1060: 8 total, 8 up, 8 in 2021-07-31T02:26:17.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:17 smithi045 conmon[33655]: cluster 2021-07-31T02:26:16.234454+0000 mon.a (mon. 2021-07-31T02:26:17.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:17 smithi045 conmon[33655]: 0) 1955 : cluster [DBG] osdmap e1059: 8 total, 8 up, 8 in 2021-07-31T02:26:17.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:17 smithi045 conmon[33655]: cluster 2021-07-31T02:26:17.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:17 smithi045 conmon[33655]: 2021-07-31T02:26 2021-07-31T02:26:17.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:17 smithi045 conmon[33655]: :17.220914+0000 mon.a (mon.0) 1956 : cluster [DBG] osdmap e1060: 8 total, 8 up, 8 in 2021-07-31T02:26:18.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:18 smithi154 conmon[34553]: cluster 2021-07-31T02:26:16.350737+0000 mgr.x (mgr.24542) 1671 : cluster [DBG] pgmap v2077: 137 pgs: 137 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2021-07-31T02:26:18.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:18 smithi045 conmon[39718]: cluster 2021-07-31T02:26:16.350737+0000 mgr.x (mgr.24542) 1671 : 2021-07-31T02:26:18.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:18 smithi045 conmon[39718]: cluster [DBG] pgmap v2077: 137 pgs: 137 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2021-07-31T02:26:18.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:18 smithi045 conmon[33655]: cluster 2021-07-31T02:26:16.350737+0000 mgr.x (mgr. 2021-07-31T02:26:18.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:18 smithi045 conmon[33655]: 24542) 1671 : cluster [DBG] pgmap v2077: 137 pgs: 137 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2021-07-31T02:26:19.265 INFO:tasks.workunit.client.0.smithi045.stdout:using old format 2021-07-31T02:26:19.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:19 smithi045 conmon[33655]: cluster 2021-07-31T02:26:18.251920+0000 mon.a (mon.0) 1957 : 2021-07-31T02:26:19.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:19 smithi045 conmon[33655]: cluster [DBG] osdmap e1061: 8 total, 8 up, 8 in 2021-07-31T02:26:19.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:19 smithi045 conmon[33655]: audit 2021-07-31T02:26:18.253804+0000 mon.c (mon.1) 97 : audit [INF] from='client.? 172.21.15.45:0/261281869' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:26:19.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:19 smithi045 conmon[33655]: audit 2021-07-31T02:26:18.255345+0000 mon.a (mon.0) 1958 2021-07-31T02:26:19.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:19 smithi045 conmon[33655]: : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:26:19.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:19 smithi045 conmon[39718]: cluster 2021-07-31T02:26:18.251920+0000 2021-07-31T02:26:19.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:19 smithi045 conmon[39718]: mon.a (mon.0) 1957 : cluster [DBG] osdmap e1061: 8 total, 8 up, 8 in 2021-07-31T02:26:19.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:19 smithi045 conmon[39718]: audit 2021-07-31T02: 2021-07-31T02:26:19.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:19 smithi045 conmon[39718]: 26:18.253804+0000 mon.c (mon.1) 97 : audit [INF] from='client.? 172.21.15.45:0/261281869' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:26:19.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:19 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T02:26:19.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:19 smithi045 conmon[39718]: 31T02:26:18.255345+0000 mon.a ( 2021-07-31T02:26:19.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:19 smithi045 conmon[39718]: mon.0) 1958 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:26:19.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:19 smithi154 conmon[34553]: cluster 2021-07-31T02:26:18.251920+0000 mon.a (mon.0 2021-07-31T02:26:19.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:19 smithi154 conmon[34553]: ) 1957 : cluster [DBG] osdmap e1061: 8 total, 8 up, 8 in 2021-07-31T02:26:19.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:19 smithi154 conmon[34553]: audit 2021-07-31T02:26:18.253804+0000 mon.c (mon.1) 97 : audit [INF] from='client.? 172.21.15.45:0/261281869' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:26:19.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:19 smithi154 conmon[34553]: audit 2021-07-31T02:26:18.255345+0000 mon.a (mon 2021-07-31T02:26:19.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:19 smithi154 conmon[34553]: .0) 1958 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:26:20.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:20 smithi045 conmon[33655]: cluster 2021-07-31T02:26:18.351300+0000 2021-07-31T02:26:20.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:20 smithi045 conmon[33655]: mgr.x (mgr.24542) 1672 : cluster [DBG] pgmap v2080: 169 pgs: 32 unknown, 137 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:26:20.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:20 smithi045 conmon[33655]: audit 2021-07-31T02:26:19.259553 2021-07-31T02:26:20.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:20 smithi045 conmon[33655]: +0000 mon.a (mon.0) 1959 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-5","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:26:20.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:20 smithi045 conmon[33655]: cluster 2021-07-31T02:26: 2021-07-31T02:26:20.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:20 smithi045 conmon[33655]: 19.259857+0000 mon.a (mon.0) 1960 : cluster [DBG] 2021-07-31T02:26:20.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:20 smithi045 conmon[33655]: osdmap e1062: 8 total, 8 up, 8 in 2021-07-31T02:26:20.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:20 smithi045 conmon[39718]: cluster 2021-07-31T02:26:18.351300+0000 mgr.x ( 2021-07-31T02:26:20.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:20 smithi045 conmon[39718]: mgr.24542) 1672 : cluster [DBG] pgmap v2080: 169 pgs: 32 unknown, 137 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:26:20.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:20 smithi045 conmon[39718]: audit 2021-07-31T02:26:19.259553+0000 mon.a (mon.0) 1959 : audit [INF] 2021-07-31T02:26:20.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:20 smithi045 conmon[39718]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-5","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:26:20.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:20 smithi045 conmon[39718]: cluster 2021-07-31T02:26:19.259857+0000 mon.a 2021-07-31T02:26:20.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:20 smithi045 conmon[39718]: (mon.0) 1960 : cluster [DBG] osdmap e1062: 8 total, 8 up, 8 in 2021-07-31T02:26:20.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:20 smithi154 conmon[34553]: cluster 2021-07-31T02:26:18.351300+0000 mgr.x (mgr.24542) 1672 : 2021-07-31T02:26:20.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:20 smithi154 conmon[34553]: cluster [DBG] pgmap v2080: 169 pgs: 32 unknown, 137 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:26:20.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:20 smithi154 conmon[34553]: audit 2021-07-31T02:26:19.259553+0000 mon.a (mon.0) 1959 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-5","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:26:20.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:20 smithi154 conmon[34553]: cluster 2021-07-31 2021-07-31T02:26:20.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:20 smithi154 conmon[34553]: T02:26:19.259857+0000 mon.a (mon.0) 1960 : cluster [DBG] osdmap e1062: 8 total, 8 up, 8 in 2021-07-31T02:26:21.475 INFO:tasks.workunit.client.0.smithi045.stdout:num images is: 1 2021-07-31T02:26:21.476 INFO:tasks.workunit.client.0.smithi045.stdout:expected: 1 2021-07-31T02:26:21.476 INFO:tasks.workunit.client.0.smithi045.stdout:image: image22 2021-07-31T02:26:21.476 INFO:tasks.workunit.client.0.smithi045.stdout:expected = image22 2021-07-31T02:26:21.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:21 smithi045 conmon[39718]: cluster 2021-07-31T02:26:20.260262+0000 mon.a 2021-07-31T02:26:21.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:21 smithi045 conmon[39718]: (mon.0) 1961 : cluster [DBG] osdmap e1063: 8 total, 8 up, 8 in 2021-07-31T02:26:21.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:21 smithi045 conmon[39718]: cluster 2021-07-31T02:26:21.261676+0000 mon.a (mon. 2021-07-31T02:26:21.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:21 smithi045 conmon[39718]: 0) 1962 : cluster [DBG] osdmap e1064: 8 total, 8 up, 8 in 2021-07-31T02:26:21.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:21 smithi045 conmon[33655]: cluster 2021-07-31T02:26:20.260262+0000 mon.a (mon.0) 1961 2021-07-31T02:26:21.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:21 smithi045 conmon[33655]: : cluster [DBG] osdmap e1063: 8 total, 8 up, 8 in 2021-07-31T02:26:21.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:21 smithi045 conmon[33655]: cluster 2021-07-31T02:26:21.261676+0000 mon.a (mon.0) 1962 : cluster [DBG] osdmap e1064: 8 total, 8 up, 8 in 2021-07-31T02:26:21.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:21 smithi154 conmon[34553]: cluster 2021-07-31T02:26:21.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:21 smithi154 conmon[34553]: 2021-07-31T02:26:21.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:21 smithi154 conmon[34553]: 2021-07-31T02:26:20.260262+0000 mon.a (mon 2021-07-31T02:26:21.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:21 smithi154 conmon[34553]: .0) 1961 : cluster [DBG] osdmap e1063: 8 total, 8 up, 8 in 2021-07-31T02:26:21.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:21 smithi154 conmon[34553]: cluster 2021-07 2021-07-31T02:26:21.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:21 smithi154 conmon[34553]: -31T02:26:21.261676+0000 mon.a (mon.0) 1962 : cluster [DBG] osdmap e1064: 8 total, 8 up, 8 in 2021-07-31T02:26:22.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:22 smithi045 conmon[39718]: cluster 2021-07-31T02:26:20.351909+0000 mgr.x 2021-07-31T02:26:22.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:22 smithi045 conmon[39718]: (mgr.24542) 1673 : cluster [DBG] pgmap v2083: 169 pgs: 32 creating+peering, 137 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 98 KiB/s rd, 21 KiB/s wr, 141 op/s 2021-07-31T02:26:22.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:22 smithi045 conmon[39718]: cluster 2021-07-31T02:26:22. 2021-07-31T02:26:22.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:22 smithi045 conmon[39718]: 221121+0000 mon.a (mon.0) 1963 : cluster [DBG] osdmap e1065: 8 total, 8 up, 8 in 2021-07-31T02:26:22.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:22 smithi045 conmon[33655]: cluster 2021-07-31T02:26:20.351909+0000 mgr.x (mgr.24542) 1673 : cluster [DBG] pgmap v2083: 169 pgs: 32 creating+peering, 137 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 98 KiB/s rd, 21 KiB/s wr, 141 op/s 2021-07-31T02:26:22.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:22 smithi045 conmon[33655]: cluster 2021-07-31T02:26:22.221121+0000 mon.a (mon.0) 1963 : cluster [DBG] osdmap e1065: 8 total, 8 up, 8 in 2021-07-31T02:26:22.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:22 smithi154 conmon[34553]: cluster 2021-07-31T02:26:20.351909+0000 mgr.x (mgr.24542) 1673 : cluster [DBG] pgmap v2083: 169 pgs: 32 creating+peering, 137 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 98 KiB/s rd, 21 KiB/s wr, 141 op/s 2021-07-31T02:26:22.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:22 smithi154 conmon[34553]: cluster 2021-07-31T02:26:22.221121+0000 mon.a (mon.0) 1963 : cluster [DBG] osdmap e1065: 8 total, 8 up, 8 in 2021-07-31T02:26:23.271 INFO:tasks.workunit.client.0.smithi045.stdout:num images is: 2 2021-07-31T02:26:23.272 INFO:tasks.workunit.client.0.smithi045.stdout:expected: 2 2021-07-31T02:26:23.272 INFO:tasks.workunit.client.0.smithi045.stdout:image: image22 2021-07-31T02:26:23.272 INFO:tasks.workunit.client.0.smithi045.stdout:image: image23 2021-07-31T02:26:23.272 INFO:tasks.workunit.client.0.smithi045.stdout:expected = image22 2021-07-31T02:26:23.272 INFO:tasks.workunit.client.0.smithi045.stdout:expected = image23 2021-07-31T02:26:23.344 INFO:tasks.workunit.client.0.smithi045.stdout:0.00% done 2021-07-31T02:26:23.344 INFO:tasks.workunit.client.0.smithi045.stdout:100.00% done 2021-07-31T02:26:23.344 INFO:tasks.workunit.client.0.smithi045.stdout:num images is: 3 2021-07-31T02:26:23.345 INFO:tasks.workunit.client.0.smithi045.stdout:expected: 3 2021-07-31T02:26:23.345 INFO:tasks.workunit.client.0.smithi045.stdout:image: image22 2021-07-31T02:26:23.345 INFO:tasks.workunit.client.0.smithi045.stdout:image: image23 2021-07-31T02:26:23.345 INFO:tasks.workunit.client.0.smithi045.stdout:image: image24 2021-07-31T02:26:23.345 INFO:tasks.workunit.client.0.smithi045.stdout:expected = image22 2021-07-31T02:26:23.346 INFO:tasks.workunit.client.0.smithi045.stdout:expected = image23 2021-07-31T02:26:23.346 INFO:tasks.workunit.client.0.smithi045.stdout:expected = image24 2021-07-31T02:26:23.387 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.TestCopyPP (6011 ms) 2021-07-31T02:26:23.387 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.TestDeepCopy 2021-07-31T02:26:23.387 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:26:23.387 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.TestDeepCopy (0 ms) 2021-07-31T02:26:23.388 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.TestDeepCopyPP 2021-07-31T02:26:23.388 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:26:23.388 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.TestDeepCopyPP (0 ms) 2021-07-31T02:26:23.388 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.TestCreateLsDeleteSnap 2021-07-31T02:26:23.389 INFO:tasks.workunit.client.0.smithi045.stdout:using old format 2021-07-31T02:26:24.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:26:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:26:23.577Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:26:24.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:26:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:26:23.577Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:26:24.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:26:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:26:23.577Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:26:24.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:24 smithi154 conmon[34553]: cluster 2021-07-31T02:26:22.352402+0000 mgr.x (mgr.24542) 1674 : cluster [DBG] pgmap v2086: 169 pgs: 32 creating+peering, 137 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 98 KiB/s rd, 21 KiB/s wr, 141 op/s 2021-07-31T02:26:24.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:24 smithi154 conmon[34553]: cluster 2021-07-31T02:26:23.228017+0000 mon.a (mon.0) 1964 : cluster [DBG] osdmap e1066: 8 total, 8 up, 8 in 2021-07-31T02:26:24.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:24 smithi045 conmon[39718]: cluster 2021-07-31T02:26:22.352402 2021-07-31T02:26:24.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:24 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 1674 : cluster [DBG] pgmap v2086: 169 pgs: 32 creating+peering, 137 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 98 KiB/s rd, 21 KiB/s wr, 141 op/s 2021-07-31T02:26:24.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:24 smithi045 conmon[39718]: cluster 2021-07-31T02:26:23.228017+0000 mon.a (mon. 2021-07-31T02:26:24.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:24 smithi045 conmon[39718]: 0) 1964 : cluster [DBG] osdmap e1066: 8 total, 8 up, 8 in 2021-07-31T02:26:24.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:24 smithi045 conmon[33655]: cluster 2021-07-31T02:26:22.352402+0000 mgr.x (mgr.24542) 2021-07-31T02:26:24.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:24 smithi045 conmon[33655]: 1674 : cluster [DBG] pgmap v2086: 169 pgs: 32 creating+peering, 137 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 98 KiB/s rd, 21 KiB/s wr, 141 op/s 2021-07-31T02:26:24.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:24 smithi045 conmon[33655]: cluster 2021-07-31T02:26:23.228017+0000 mon.a (mon.0) 1964 : cluster [DBG] 2021-07-31T02:26:24.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:24 smithi045 conmon[33655]: osdmap e1066: 8 total, 8 up, 8 in 2021-07-31T02:26:25.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:25 smithi154 conmon[34553]: cluster 2021-07-31T02:26:24.278998+0000 mon.a (mon.0) 1965 2021-07-31T02:26:25.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:25 smithi154 conmon[34553]: : cluster [DBG] osdmap e1067: 8 total, 8 up, 8 in 2021-07-31T02:26:25.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:25 smithi045 conmon[39718]: cluster 2021-07-31T02:26:24.278998+0000 mon.a 2021-07-31T02:26:25.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:25 smithi045 conmon[39718]: (mon.0) 1965 : cluster [DBG] osdmap e1067: 8 total, 8 up, 8 in 2021-07-31T02:26:25.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:25 smithi045 conmon[33655]: cluster 2021-07-31T02:26:24.278998+0000 mon.a (mon. 2021-07-31T02:26:25.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:25 smithi045 conmon[33655]: 0) 1965 : cluster [DBG] osdmap e1067: 8 total, 8 up, 8 in 2021-07-31T02:26:26.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:26 smithi154 conmon[34553]: cluster 2021-07-31T02:26:24.352847+0000 mgr.x (mgr.24542) 1675 : cluster [DBG] pgmap v2089: 169 pgs: 169 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 99 KiB/s rd, 22 KiB/s wr, 143 op/s 2021-07-31T02:26:26.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:26 smithi154 conmon[34553]: cluster 2021-07-31T02:26:25.283795+0000 mon.a (mon.0) 1966 : cluster [DBG] osdmap e1068: 8 total, 8 up, 8 in 2021-07-31T02:26:26.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:26 smithi045 conmon[39718]: cluster 2021-07-31T02:26:24.352847+0000 mgr.x (mgr. 2021-07-31T02:26:26.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:26 smithi045 conmon[39718]: 24542) 1675 : cluster [DBG] pgmap v2089: 169 pgs: 169 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 99 KiB/s rd, 22 KiB/s wr, 143 op/s 2021-07-31T02:26:26.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:26 smithi045 conmon[39718]: cluster 2021-07-31T02:26:25.283795+0000 mon.a (mon.0) 1966 : cluster [DBG] osdmap e1068: 8 total, 8 up, 8 in 2021-07-31T02:26:26.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:26 smithi045 conmon[33655]: cluster 2021-07-31T02:26:24.352847+0000 mgr.x (mgr.24542) 1675 : cluster [DBG] pgmap v2089: 169 pgs: 169 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 99 KiB/s rd, 22 KiB/s wr, 143 op/s 2021-07-31T02:26:26.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:26 smithi045 conmon[33655]: cluster 2021-07-31T02:26:25.283795+0000 mon.a (mon.0) 1966 : cluster [DBG] osdmap e1068: 8 total, 8 up, 8 in 2021-07-31T02:26:27.243 INFO:tasks.workunit.client.0.smithi045.stdout:num snaps is: 1 2021-07-31T02:26:27.244 INFO:tasks.workunit.client.0.smithi045.stdout:expected: 1 2021-07-31T02:26:27.244 INFO:tasks.workunit.client.0.smithi045.stdout:snap: snap1 2021-07-31T02:26:27.244 INFO:tasks.workunit.client.0.smithi045.stdout:found snap1 with size 2097152 2021-07-31T02:26:27.244 INFO:tasks.workunit.client.0.smithi045.stdout:num snaps is: 2 2021-07-31T02:26:27.245 INFO:tasks.workunit.client.0.smithi045.stdout:expected: 2 2021-07-31T02:26:27.245 INFO:tasks.workunit.client.0.smithi045.stdout:snap: snap1 2021-07-31T02:26:27.245 INFO:tasks.workunit.client.0.smithi045.stdout:snap: snap2 2021-07-31T02:26:27.245 INFO:tasks.workunit.client.0.smithi045.stdout:found snap1 with size 2097152 2021-07-31T02:26:27.245 INFO:tasks.workunit.client.0.smithi045.stdout:found snap2 with size 4194304 2021-07-31T02:26:27.246 INFO:tasks.workunit.client.0.smithi045.stdout:num snaps is: 1 2021-07-31T02:26:27.246 INFO:tasks.workunit.client.0.smithi045.stdout:expected: 1 2021-07-31T02:26:27.246 INFO:tasks.workunit.client.0.smithi045.stdout:snap: snap2 2021-07-31T02:26:27.246 INFO:tasks.workunit.client.0.smithi045.stdout:found snap2 with size 4194304 2021-07-31T02:26:27.247 INFO:tasks.workunit.client.0.smithi045.stdout:num snaps is: 0 2021-07-31T02:26:27.247 INFO:tasks.workunit.client.0.smithi045.stdout:expected: 0 2021-07-31T02:26:27.247 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.TestCreateLsDeleteSnap (3857 ms) 2021-07-31T02:26:27.247 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.TestGetSnapShotTimeStamp 2021-07-31T02:26:27.247 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:26:27.248 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.TestGetSnapShotTimeStamp (0 ms) 2021-07-31T02:26:27.248 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.TestCreateLsDeleteSnapPP 2021-07-31T02:26:27.249 INFO:tasks.workunit.client.0.smithi045.stdout:using old format 2021-07-31T02:26:27.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:27 smithi154 conmon[34553]: cluster 2021-07-31T02:26:26.291158+0000 mon.a (mon.0) 1967 : cluster [DBG] osdmap e1069: 8 total, 8 up, 8 in 2021-07-31T02:26:27.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:27 smithi154 conmon[34553]: 2021-07-31T02:26:27.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:27 smithi154 conmon[34553]: cluster 2021-07-31T02:26:27.222958+0000 mon.a (mon.0) 1968 : cluster [DBG] osdmap e1070: 8 total, 8 up, 8 in 2021-07-31T02:26:27.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:27 smithi045 conmon[39718]: cluster 2021-07-31T02:26:26.291158+0000 mon.a (mon.0) 1967 : cluster 2021-07-31T02:26:27.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:27 smithi045 conmon[39718]: [DBG] osdmap e1069: 8 total, 8 up, 8 in 2021-07-31T02:26:27.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:27 smithi045 conmon[39718]: cluster 2021-07-31T02:26:27.222958+0000 mon.a (mon.0 2021-07-31T02:26:27.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:27 smithi045 conmon[39718]: ) 1968 : cluster [DBG] osdmap e1070: 8 total, 8 up, 8 in 2021-07-31T02:26:27.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:27 smithi045 conmon[33655]: cluster 2021-07-31T02:26:26.291158+0000 mon.a (mon.0) 1967 : cluster [DBG] osdmap e1069: 8 total, 8 up, 8 in 2021-07-31T02:26:27.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:27 smithi045 conmon[33655]: cluster 2021-07-31T02:26:27.222958+0000 mon.a (mon.0) 1968 : 2021-07-31T02:26:27.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:27 smithi045 conmon[33655]: cluster [DBG] osdmap e1070: 8 total, 8 up, 8 in 2021-07-31T02:26:28.242 INFO:tasks.workunit.client.0.smithi045.stdout:num snaps is: 1 2021-07-31T02:26:28.243 INFO:tasks.workunit.client.0.smithi045.stdout:expected: 1 2021-07-31T02:26:28.243 INFO:tasks.workunit.client.0.smithi045.stdout:snap: snap1 2021-07-31T02:26:28.244 INFO:tasks.workunit.client.0.smithi045.stdout:found snap1 with size 2097152 2021-07-31T02:26:28.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:28 smithi154 conmon[34553]: cluster 2021-07-31T02:26:26.353392+0000 mgr.x (mgr.24542) 1676 : cluster [DBG] pgmap v2092: 169 pgs: 169 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 99 KiB/s rd, 22 KiB/s wr, 143 op/s 2021-07-31T02:26:28.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:28 smithi154 conmon[34553]: audit 2021-07-31T02:26:28.065960+0000 2021-07-31T02:26:28.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:28 smithi154 conmon[34553]: mon.b (mon.2) 265 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:26:28.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:28 smithi154 conmon[34553]: audit 2021-07-31T02:26:28.066709+0000 mon.a (mon.0) 1969 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:26:28.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:28 smithi154 conmon[34553]: audit 2021-07-31T02:26:28.091437+0000 mon.b (mon 2021-07-31T02:26:28.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:28 smithi154 conmon[34553]: .2) 266 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:26:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:28 smithi154 conmon[34553]: audit 2021-07-31T02:26:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:28 smithi154 conmon[34553]: 2021-07-31T02:26:28.092007+0000 2021-07-31T02:26:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:28 smithi154 conmon[34553]: mon.a (mon.0) 1970 : audit [INF] 2021-07-31T02:26:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:28 smithi154 conmon[34553]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:26:28.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:28 smithi154 conmon[34553]: audit 2021-07- 2021-07-31T02:26:28.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:28 smithi154 conmon[34553]: 31T02:26:28.092394+0000 2021-07-31T02:26:28.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:28 smithi154 conmon[34553]: mon.b (mon.2) 267 : audit [INF] 2021-07-31T02:26:28.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:28 smithi154 conmon[34553]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "63.1", "id": [1, 3]}]: dispatch 2021-07-31T02:26:28.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:28 smithi154 conmon[34553]: audit 2021-07-31T02:26:28.092816+0000 mon.b 2021-07-31T02:26:28.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:28 smithi154 conmon[34553]: (mon.2) 268 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "63.18", "id": [1, 3]}]: dispatch 2021-07-31T02:26:28.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:28 smithi154 conmon[34553]: audit 2021-07-31T02:26:28.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:28 smithi154 conmon[34553]: 2021-07-31T02: 2021-07-31T02:26:28.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:28 smithi154 conmon[34553]: 26:28.092826+0000 mon.a (mon.0 2021-07-31T02:26:28.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:28 smithi154 conmon[34553]: ) 1971 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "63.1", "id": [1, 3]}]: dispatch 2021-07-31T02:26:28.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:28 smithi154 conmon[34553]: audit 2021-07-31T02:26:28.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:28 smithi154 conmon[34553]: 2021-07-31T02:26 2021-07-31T02:26:28.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:28 smithi154 conmon[34553]: :28.093200+0000 mon.a (mon. 2021-07-31T02:26:28.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:28 smithi154 conmon[34553]: 0) 1972 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "63.18", "id": [1, 3]}]: dispatch 2021-07-31T02:26:28.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:28 smithi154 conmon[34553]: audit 2021-07-31T02:26:28.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:28 smithi154 conmon[34553]: 2021-07-31T02:26:28 2021-07-31T02:26:28.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:28 smithi154 conmon[34553]: .232110+0000 mon.a (mon.0 2021-07-31T02:26:28.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:28 smithi154 conmon[34553]: ) 1973 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "63.1", "id": [1, 3]}]': finished 2021-07-31T02:26:28.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:28 smithi154 conmon[34553]: audit 2021-07 2021-07-31T02:26:28.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:28 smithi154 conmon[34553]: -31T02:26:28.232379+0000 mon.a 2021-07-31T02:26:28.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:28 smithi154 conmon[34553]: (mon.0) 1974 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "63.18", "id": [1, 3]}]': finished 2021-07-31T02:26:28.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:28 smithi154 conmon[34553]: 2021-07-31T02:26:28.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:28 smithi154 conmon[34553]: cluster 2021-07 2021-07-31T02:26:28.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:28 smithi154 conmon[34553]: -31T02:26:28. 2021-07-31T02:26:28.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:28 smithi154 conmon[34553]: 232492+0000 mon.a (mon.0) 1975 : cluster [DBG] 2021-07-31T02:26:28.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:28 smithi154 conmon[34553]: osdmap e1071: 8 total, 8 up, 8 in 2021-07-31T02:26:28.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:28 smithi045 conmon[39718]: cluster 2021-07-31T02:26:26.353392+0000 mgr.x (mgr. 2021-07-31T02:26:28.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:28 smithi045 conmon[39718]: 24542) 1676 : cluster [DBG] pgmap v2092: 169 pgs: 169 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 99 KiB/s rd, 22 KiB/s wr, 143 op/s 2021-07-31T02:26:28.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:28 smithi045 conmon[39718]: audit 2021-07-31T02:26:28.065960+0000 2021-07-31T02:26:28.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:28 smithi045 conmon[39718]: mon.b (mon.2) 265 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:26:28.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:28 smithi045 conmon[39718]: audit 2021-07-31T02:26:28.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:28 smithi045 conmon[39718]: 2021-07-31T02:26:28 2021-07-31T02:26:28.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:28 smithi045 conmon[39718]: .066709+0000 mon.a (mon.0) 1969 : audit 2021-07-31T02:26:28.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:28 smithi045 conmon[39718]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:26:28.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:28 smithi045 conmon[39718]: audit 2021- 2021-07-31T02:26:28.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:28 smithi045 conmon[39718]: 07-31T02:26:28.091437+0000 mon.b ( 2021-07-31T02:26:28.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:28 smithi045 conmon[39718]: mon.2) 266 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:26:28.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:28 smithi045 conmon[39718]: audit 2021-07-31T02:26:28.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:28 smithi045 conmon[39718]: 2021-07-31T02:26:28.092007+0000 mon.a 2021-07-31T02:26:28.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:28 smithi045 conmon[39718]: (mon.0) 1970 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:26:28.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:28 smithi045 conmon[39718]: audit 2021-07-31T02:26:28.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:28 smithi045 conmon[39718]: 2021-07-31T02:26:28.092394+0000 mon.b ( 2021-07-31T02:26:28.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:28 smithi045 conmon[39718]: mon.2) 267 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "63.1", "id": [1, 3]}]: dispatch 2021-07-31T02:26:28.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:28 smithi045 conmon[39718]: audit 2021-07-31T02:26:28.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:28 smithi045 conmon[39718]: 2021-07-31T02:26:28.092816+0000 2021-07-31T02:26:28.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:28 smithi045 conmon[39718]: mon.b (mon.2) 268 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "63.18", "id": [1, 3]}]: dispatch 2021-07-31T02:26:28.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:28 smithi045 conmon[39718]: audit 2021-07-31T02:26:28.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:28 smithi045 conmon[39718]: 2021-07-31T02:26:28.092826+0000 mon.a ( 2021-07-31T02:26:28.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:28 smithi045 conmon[39718]: mon.0) 1971 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "63.1", "id": [1, 3]}]: dispatch 2021-07-31T02:26:28.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:28 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T02:26:28.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:28 smithi045 conmon[39718]: T02:26:28.093200+0000 mon.a (mon.0) 1972 : audit [INF] 2021-07-31T02:26:28.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:28 smithi045 conmon[39718]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "63.18", "id": [1, 3]}]: dispatch 2021-07-31T02:26:28.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:28 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T02:26:28.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:28 smithi045 conmon[39718]: 02:26:28.232110+0000 mon.a (mon.0 2021-07-31T02:26:28.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:28 smithi045 conmon[39718]: ) 1973 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "63.1", "id": [1, 3]}]': finished 2021-07-31T02:26:28.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:28 smithi045 conmon[39718]: audit 2021-07-31T02:26:28.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:28 smithi045 conmon[39718]: 2021-07-31T02:26:28.232379 2021-07-31T02:26:28.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:28 smithi045 conmon[39718]: +0000 mon.a (mon.0) 1974 : audit [INF] 2021-07-31T02:26:28.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:28 smithi045 conmon[39718]: from='mgr.24542 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "63.18", "id": [1, 3]}]': finished 2021-07-31T02:26:28.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:28 smithi045 conmon[39718]: cluster 2021-07-31T02:26:28.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:28 smithi045 conmon[39718]: 2021-07-31T02:26:28.232492+0000 mon.a 2021-07-31T02:26:28.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:28 smithi045 conmon[39718]: (mon.0) 1975 : cluster [DBG] osdmap e1071: 8 total, 8 up, 8 in 2021-07-31T02:26:28.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:28 smithi045 conmon[33655]: cluster 2021-07-31T02:26:26.353392+0000 mgr.x 2021-07-31T02:26:28.788 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:28 smithi045 conmon[33655]: (mgr.24542) 1676 : cluster [DBG] pgmap v2092: 169 pgs: 169 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 99 KiB/s rd, 22 KiB/s wr, 143 op/s 2021-07-31T02:26:28.788 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:28 smithi045 conmon[33655]: audit 2021-07-31T02:26 2021-07-31T02:26:28.788 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:28 smithi045 conmon[33655]: :28.065960+0000 mon.b (mon.2) 265 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:26:28.789 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:28 smithi045 conmon[33655]: audit 2021-07-31T02:26:28.789 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:28 smithi045 conmon[33655]: 2021-07 2021-07-31T02:26:28.789 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:28 smithi045 conmon[33655]: -31T02:26:28.066709+0000 2021-07-31T02:26:28.790 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:28 smithi045 conmon[33655]: mon.a (mon.0) 1969 : audit 2021-07-31T02:26:28.790 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:28 smithi045 conmon[33655]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:26:28.790 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:28 smithi045 conmon[33655]: audit 2021-07-31T02:26: 2021-07-31T02:26:28.790 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:28 smithi045 conmon[33655]: 28.091437+0000 mon.b (mon.2) 266 2021-07-31T02:26:28.790 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:28 smithi045 conmon[33655]: : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:26:28.791 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:28 smithi045 conmon[33655]: audit 2021-07- 2021-07-31T02:26:28.791 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:28 smithi045 conmon[33655]: 31T02:26:28.092007+0000 2021-07-31T02:26:28.791 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:28 smithi045 conmon[33655]: mon.a (mon.0) 1970 : audit [INF] 2021-07-31T02:26:28.791 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:28 smithi045 conmon[33655]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:26:28.792 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:28 smithi045 conmon[33655]: audit 2021-07-31 2021-07-31T02:26:28.792 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:28 smithi045 conmon[33655]: T02 2021-07-31T02:26:28.792 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:28 smithi045 conmon[33655]: :26:28.092394+0000 2021-07-31T02:26:28.793 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:28 smithi045 conmon[33655]: mon.b (mon.2) 267 : audit 2021-07-31T02:26:28.793 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:28 smithi045 conmon[33655]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "63.1", "id": [1, 3]}]: dispatch 2021-07-31T02:26:28.793 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:28 smithi045 conmon[33655]: audit 2021-07-31T02:26:28.793 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:28 smithi045 conmon[33655]: 2021-07-31T02:26:28 2021-07-31T02:26:28.793 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:28 smithi045 conmon[33655]: .092816+0000 mon.b (mon.2) 2021-07-31T02:26:28.794 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:28 smithi045 conmon[33655]: 268 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "63.18", "id": [1, 3]}]: dispatch 2021-07-31T02:26:28.794 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:28 smithi045 conmon[33655]: audit 2021-07- 2021-07-31T02:26:28.794 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:28 smithi045 conmon[33655]: 31T02:26:28.092826+0000 mon.a (mon 2021-07-31T02:26:28.794 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:28 smithi045 conmon[33655]: .0) 1971 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "63.1", "id": [1, 3]}]: dispatch 2021-07-31T02:26:28.794 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:28 smithi045 conmon[33655]: audit 2021-07-31T02:26:28.795 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:28 smithi045 conmon[33655]: 2021-07-31T02:26:28 2021-07-31T02:26:28.795 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:28 smithi045 conmon[33655]: .093200+0000 mon.a (mon.0) 1972 2021-07-31T02:26:28.795 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:28 smithi045 conmon[33655]: : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "63.18", "id": [1, 3]}]: dispatch 2021-07-31T02:26:28.795 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:28 smithi045 conmon[33655]: audit 2021-07-31T02:26 2021-07-31T02:26:28.796 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:28 smithi045 conmon[33655]: :28.232110+0000 mon.a (mon 2021-07-31T02:26:28.796 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:28 smithi045 conmon[33655]: .0) 1973 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "63.1", "id": [1, 3]}]': finished 2021-07-31T02:26:28.796 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:28 smithi045 conmon[33655]: audit 2021- 2021-07-31T02:26:28.796 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:28 smithi045 conmon[33655]: 07-31T02:26:28. 2021-07-31T02:26:28.796 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:28 smithi045 conmon[33655]: 232379+0000 mon.a (mon.0) 1974 : 2021-07-31T02:26:28.797 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:28 smithi045 conmon[33655]: audit [INF] from='mgr.24542 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "63.18", "id": [1, 3]}]': finished 2021-07-31T02:26:28.797 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:28 smithi045 conmon[33655]: cluster 2021- 2021-07-31T02:26:28.797 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:28 smithi045 conmon[33655]: 07-31T02:26:28.232492 2021-07-31T02:26:28.797 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:28 smithi045 conmon[33655]: +0000 mon.a (mon.0) 1975 : cluster 2021-07-31T02:26:28.797 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:28 smithi045 conmon[33655]: [DBG] osdmap e1071: 8 total, 8 up, 8 in 2021-07-31T02:26:29.246 INFO:tasks.workunit.client.0.smithi045.stdout:num snaps is: 2 2021-07-31T02:26:29.247 INFO:tasks.workunit.client.0.smithi045.stdout:expected: 2 2021-07-31T02:26:29.247 INFO:tasks.workunit.client.0.smithi045.stdout:snap: snap1 2021-07-31T02:26:29.248 INFO:tasks.workunit.client.0.smithi045.stdout:snap: snap2 2021-07-31T02:26:29.248 INFO:tasks.workunit.client.0.smithi045.stdout:found snap1 with size 2097152 2021-07-31T02:26:29.248 INFO:tasks.workunit.client.0.smithi045.stdout:found snap2 with size 4194304 2021-07-31T02:26:30.253 INFO:tasks.workunit.client.0.smithi045.stdout:num snaps is: 1 2021-07-31T02:26:30.254 INFO:tasks.workunit.client.0.smithi045.stdout:expected: 1 2021-07-31T02:26:30.254 INFO:tasks.workunit.client.0.smithi045.stdout:snap: snap2 2021-07-31T02:26:30.254 INFO:tasks.workunit.client.0.smithi045.stdout:found snap2 with size 4194304 2021-07-31T02:26:30.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:30 smithi154 conmon[34553]: cluster 2021-07-31T02:26:28.354008+0000 mgr.x (mgr.24542) 1677 : cluster [DBG] pgmap v2095: 169 pgs: 169 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:26:30.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:30 smithi154 conmon[34553]: 2021-07-31T02:26:30.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:30 smithi154 conmon[34553]: cluster 2021-07-31T02:26:29.233953+0000 mon.a (mon.0) 1976 : cluster [DBG] osdmap e1072: 8 total, 8 up, 8 in 2021-07-31T02:26:30.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:30 smithi045 conmon[33655]: cluster 2021-07-31T02:26 2021-07-31T02:26:30.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:30 smithi045 conmon[33655]: :28.354008+0000 mgr.x (mgr.24542) 1677 : cluster [DBG] pgmap v2095: 169 pgs: 169 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:26:30.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:30 smithi045 conmon[33655]: cluster 2021-07-31T02:26:30.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:30 smithi045 conmon[33655]: 2021-07-31T02:26:29.233953+0000 2021-07-31T02:26:30.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:30 smithi045 conmon[33655]: mon.a (mon.0) 1976 : cluster [DBG] osdmap e1072: 8 total, 8 up, 8 in 2021-07-31T02:26:30.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:30 smithi045 conmon[39718]: cluster 2021-07-31T02:26:28.354008+0000 mgr.x ( 2021-07-31T02:26:30.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:30 smithi045 conmon[39718]: mgr.24542) 1677 : cluster [DBG] pgmap v2095: 169 pgs: 169 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:26:30.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:30 smithi045 conmon[39718]: cluster 2021-07-31T02:26:30.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:30 smithi045 conmon[39718]: 2021-07-31T02:26:29.233953+0000 mon.a (mon.0) 1976 : 2021-07-31T02:26:30.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:30 smithi045 conmon[39718]: cluster [DBG] osdmap e1072: 8 total, 8 up, 8 in 2021-07-31T02:26:31.256 INFO:tasks.workunit.client.0.smithi045.stdout:num snaps is: 0 2021-07-31T02:26:31.256 INFO:tasks.workunit.client.0.smithi045.stdout:expected: 0 2021-07-31T02:26:31.258 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.TestCreateLsDeleteSnapPP (4015 ms) 2021-07-31T02:26:31.259 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.TestGetNameIdSnapPP 2021-07-31T02:26:31.259 INFO:tasks.workunit.client.0.smithi045.stdout:using old format 2021-07-31T02:26:31.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[34553]: cluster 2021-07-31T02:26:30.239658+0000 mon.a (mon.0) 1977 2021-07-31T02:26:31.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[34553]: : cluster [DBG] osdmap e1073: 8 total, 8 up, 8 in 2021-07-31T02:26:31.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:31 smithi045 conmon[39718]: cluster 2021-07-31T02:26:30.239658+0000 mon.a (mon.0) 2021-07-31T02:26:31.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:31 smithi045 conmon[39718]: 1977 : cluster [DBG] osdmap e1073: 8 total, 8 up, 8 in 2021-07-31T02:26:31.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:31 smithi045 conmon[33655]: cluster 2021-07-31T02:26:30.239658+0000 2021-07-31T02:26:31.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:31 smithi045 conmon[33655]: mon.a (mon.0) 1977 : cluster [DBG] osdmap e1073: 8 total, 8 up, 8 in 2021-07-31T02:26:32.245 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: debug 2021-07-31T02:26:31.942+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:26:32.246 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:26:32.246 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:26:32.246 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: yield 2021-07-31T02:26:32.246 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:26:32.247 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:26:32.247 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:26:32.247 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:26:32.247 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:26:32.247 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:26:32.248 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:26:32.248 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:26:32.248 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:26:32.248 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: task.result() 2021-07-31T02:26:32.249 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:26:32.249 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: debug 2021-07-31T02:26:31.953+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:26:32.249 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:26:32.249 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:26:32.249 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: yield 2021-07-31T02:26:32.250 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:26:32.250 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:26:32.250 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:26:32.250 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:26:32.251 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:26:32.251 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:26:32.251 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:26:32.251 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:26:32.251 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:26:32.252 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: task.result() 2021-07-31T02:26:32.252 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:26:32.252 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: debug 2021-07-31T02:26:31.979+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:26:32.252 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:26:32.252 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:26:32.253 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: yield 2021-07-31T02:26:32.253 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:26:32.253 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:26:32.253 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:26:32.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:26:32.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:26:32.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:26:32.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:26:32.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:26:32.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:26:32.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: task.result() 2021-07-31T02:26:32.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:26:32.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: debug 2021-07-31T02:26:31.980+0000 7fcb97969700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:26:32.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:26:32.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:26:32.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: yield 2021-07-31T02:26:32.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:26:32.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:26:32.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:26:32.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:26:32.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:26:32.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:26:32.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:26:32.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:26:32.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:26:32.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: task.result() 2021-07-31T02:26:32.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:26:32.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: 2021-07-31T02:26:32.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:26:32.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: 2021-07-31T02:26:32.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:26:32.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:26:32.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:26:32.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:26:32.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:26:32.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:26:32.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:26:32.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:26:32.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:26:32.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:26:32.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:26:32.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:26:32.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:26:32.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:26:32.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:26:32.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:26:32.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:26:32.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:26:32.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: raise 2021-07-31T02:26:32.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:26:32.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:26:32.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:26:32.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:26:32.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:26:32.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: debug 2021-07-31T02:26:31.987+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:26:32.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:26:32.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:26:32.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: yield 2021-07-31T02:26:32.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:26:32.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:26:32.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:26:32.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:26:32.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:26:32.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:26:32.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:26:32.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:26:32.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:26:32.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: task.result() 2021-07-31T02:26:32.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:26:32.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: debug 2021-07-31T02:26:31.987+0000 7fcb9816a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:26:32.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:26:32.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:26:32.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: yield 2021-07-31T02:26:32.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:26:32.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:26:32.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:26:32.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:26:32.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:26:32.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:26:32.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:26:32.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:26:32.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:26:32.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: task.result() 2021-07-31T02:26:32.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:26:32.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: 2021-07-31T02:26:32.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:26:32.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: 2021-07-31T02:26:32.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:26:32.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:26:32.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:26:32.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:26:32.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:26:32.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:26:32.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:26:32.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:26:32.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:26:32.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:26:32.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:26:32.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:26:32.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:26:32.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:26:32.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:26:32.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:26:32.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:26:32.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:26:32.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: raise 2021-07-31T02:26:32.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:26:32.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:26:32.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:26:32.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:26:32.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:26:31 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:26:32.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:32 smithi154 conmon[34553]: cluster 2021-07-31T02:26:30.354646+0000 mgr.x (mgr.24542) 1678 : cluster [DBG] pgmap v2098: 169 pgs: 2 peering, 167 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 8.5 KiB/s rd, 2.2 KiB/s wr, 13 op/s 2021-07-31T02:26:32.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:32 smithi154 conmon[34553]: cluster 2021-07-31T02:26:31.245990+0000 mon.a (mon.0) 1978 : cluster [DBG] osdmap e1074: 8 total, 8 up, 8 in 2021-07-31T02:26:32.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:32 smithi154 conmon[34553]: audit 2021-07-31T02:26:31.895431+0000 mon.b (mon.2) 269 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:26:32.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:32 smithi154 conmon[34553]: audit 2021-07-31T02:26:32.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:32 smithi154 conmon[34553]: 2021-07-31T02:26:31.896911+0000 mon.b (mon.2) 270 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:26:32.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:32 smithi154 conmon[34553]: audit 2021-07-31T02:26:32.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:32 smithi154 conmon[34553]: 2021-07-31T02:26:31.897824+0000 mon.b (mon.2) 271 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:26:32.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:32 smithi154 conmon[34553]: 2021-07-31T02:26:32.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:32 smithi154 conmon[34553]: cluster 2021-07-31T 2021-07-31T02:26:32.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:32 smithi154 conmon[34553]: 02:26:32.224437+0000 mon.a (mon.0) 2021-07-31T02:26:32.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:32 smithi154 conmon[34553]: 1979 : cluster [DBG] osdmap e1075: 8 total, 8 up, 8 in 2021-07-31T02:26:32.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:32 smithi045 conmon[39718]: cluster 2021-07-31T02:26:30.354646+0000 mgr.x ( 2021-07-31T02:26:32.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:32 smithi045 conmon[39718]: mgr.24542) 1678 : cluster [DBG] pgmap v2098: 169 pgs: 2 peering, 167 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 8.5 KiB/s rd, 2.2 KiB/s wr, 13 op/s 2021-07-31T02:26:32.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:32 smithi045 conmon[39718]: cluster 2021-07-31T02:26:31.245990 2021-07-31T02:26:32.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:32 smithi045 conmon[39718]: +0000 mon.a (mon.0) 1978 : cluster [DBG] osdmap e1074: 8 total, 8 up, 8 in 2021-07-31T02:26:32.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:32 smithi045 conmon[39718]: audit 2021- 2021-07-31T02:26:32.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:32 smithi045 conmon[39718]: 07-31T02:26:31.895431+0000 mon.b (mon.2) 269 : audit [DBG] 2021-07-31T02:26:32.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:32 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:26:32.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:32 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T02:26:32.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:32 smithi045 conmon[39718]: T02:26:31.896911+0000 mon.b (mon.2) 270 : audit [DBG] 2021-07-31T02:26:32.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:32 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:26:32.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:32 smithi045 conmon[39718]: audit 2021-07-31T02:26:31 2021-07-31T02:26:32.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:32 smithi045 conmon[39718]: .897824+0000 mon.b (mon.2) 271 : audit 2021-07-31T02:26:32.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:32 smithi045 conmon[39718]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:26:32.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:32 smithi045 conmon[39718]: cluster 2021-07-31T02:26:32.224437+0000 2021-07-31T02:26:32.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:32 smithi045 conmon[39718]: mon.a (mon.0) 1979 : cluster [DBG] osdmap e1075: 8 total, 8 up, 8 in 2021-07-31T02:26:32.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:32 smithi045 conmon[33655]: cluster 2021-07-31T02:26:30.354646+0000 mgr.x (mgr.24542) 1678 : cluster [DBG] pgmap v2098: 169 pgs: 2 peering, 167 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 8.5 KiB/s rd, 2.2 KiB/s wr, 13 op/s 2021-07-31T02:26:32.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:32 smithi045 conmon[33655]: cluster 2021-07-31T02:26:31.245990+0000 mon.a (mon.0) 1978 : cluster 2021-07-31T02:26:32.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:32 smithi045 conmon[33655]: [DBG] osdmap e1074: 8 total, 8 up, 8 in 2021-07-31T02:26:32.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:32 smithi045 conmon[33655]: audit 2021- 2021-07-31T02:26:32.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:32 smithi045 conmon[33655]: 07-31T02:26:31.895431+0000 mon.b ( 2021-07-31T02:26:32.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:32 smithi045 conmon[33655]: mon.2) 269 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:26:32.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:32 smithi045 conmon[33655]: audit 2021-07 2021-07-31T02:26:32.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:32 smithi045 conmon[33655]: -31T02:26:31.896911+0000 mon.b (mon.2) 270 2021-07-31T02:26:32.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:32 smithi045 conmon[33655]: : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:26:32.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:32 smithi045 conmon[33655]: audit 2021 2021-07-31T02:26:32.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:32 smithi045 conmon[33655]: -07-31T02:26:31.897824+0000 mon.b (mon.2) 271 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:26:32.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:32 smithi045 conmon[33655]: cluster 2021-07-31T02:26:32.224437+0000 mon.a (mon.0) 1979 2021-07-31T02:26:32.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:32 smithi045 conmon[33655]: : cluster [DBG] osdmap e1075: 8 total, 8 up, 8 in 2021-07-31T02:26:33.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:26:31.898599+0000 mgr.x (mgr.24542) 1679 : cephadm [WRN] 2021-07-31T02:26:33.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:26:33.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:26:31.943166+0000 mgr.x (mgr.24542) 1680 : cephadm [ERR] 2021-07-31T02:26:33.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:26:33.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:26:33.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: yield 2021-07-31T02:26:33.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:26:33.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:26:33.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:26:33.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:26:33.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:26:33.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:26:33.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:26:33.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:26:33.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:26:33.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:26:33.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:26:33.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:26:31.943744+0000 mgr.x (mgr.24542) 1681 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:26:33.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:26:31.954659+0000 mgr.x (mgr.24542) 1682 : cephadm [ERR] 2021-07-31T02:26:33.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:26:33.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:26:33.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: yield 2021-07-31T02:26:33.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:26:33.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:26:33.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:26:33.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:26:33.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:26:33.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:26:33.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:26:33.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:26:33.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:26:33.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:26:33.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:26:33.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:26:31.955082+0000 mgr.x (mgr.24542) 1683 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:26:33.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:26:31.980824+0000 mgr.x (mgr.24542) 1684 : cephadm [ERR] 2021-07-31T02:26:33.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:26:33.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:26:33.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: yield 2021-07-31T02:26:33.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:26:33.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:26:33.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:26:33.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:26:33.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:26:33.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:26:33.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:26:33.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:26:33.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:26:33.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:26:33.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:26:33.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: cephadm 2021- 2021-07-31T02:26:33.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: 07-31T02:26:31.981289+0000 mgr.x (mgr.24542) 1685 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:26:33.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:26:33.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:26:33.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: yield 2021-07-31T02:26:33.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:26:33.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:26:33.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:26:33.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:26:33.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:26:33.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:26:33.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:26:33.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:26:33.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:26:33.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:26:33.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:26:33.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: 2021-07-31T02:26:33.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:26:33.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: 2021-07-31T02:26:33.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:26:33.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:26:33.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:26:33.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:26:33.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:26:33.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:26:33.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:26:33.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:26:33.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:26:33.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:26:33.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:26:33.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:26:33.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:26:33.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:26:33.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:26:33.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:26:33.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:26:33.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:26:33.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: raise 2021-07-31T02:26:33.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:26:33.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:26:33.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:26:33.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:26:33.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:26:33.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:26:31.988169+0000 mgr.x (mgr.24542) 1686 : cephadm [ERR] 2021-07-31T02:26:33.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:26:33.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:26:33.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: yield 2021-07-31T02:26:33.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:26:33.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:26:33.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:26:33.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:26:33.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:26:33.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:26:33.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:26:33.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:26:33.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:26:33.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:26:33.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:26:33.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:26:31.988618+0000 mgr.x (mgr.24542) 1687 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:26:33.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:26:33.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:26:33.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: yield 2021-07-31T02:26:33.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:26:33.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:26:33.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:26:33.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:26:33.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:26:33.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:26:33.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:26:31. 2021-07-31T02:26:33.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: 898599+0000 mgr.x (mgr.24542) 1679 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:26:33.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: cephadm 2021-07-31T 2021-07-31T02:26:33.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: 02:26:31.943166+0000 mgr.x (mgr.24542) 1680 : 2021-07-31T02:26:33.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: cephadm [ERR] 2021-07-31T02:26:33.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:26:33.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:26:33.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: yield 2021-07-31T02:26:33.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:26:33.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:26:33.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:26:33.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:26:33.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:26:33.535 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:26:33.535 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:26:33.535 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:26:33.535 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:26:33.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:26:33.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:26:33.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:26:31.943744+0000 mgr.x (mgr.24542) 1681 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:26:33.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:26:31.954659+0000 mgr.x (mgr.24542) 1682 : cephadm [ERR] 2021-07-31T02:26:33.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:26:33.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:26:33.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: yield 2021-07-31T02:26:33.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:26:33.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:26:33.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:26:33.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:26:33.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:26:33.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:26:33.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:26:33.539 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:26:33.539 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:26:33.539 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:26:33.539 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:26:33.539 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:26:31.955082+0000 mgr.x (mgr.24542) 1683 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:26:33.540 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:26:31.980824+0000 mgr.x (mgr.24542) 1684 : cephadm [ERR] 2021-07-31T02:26:33.540 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:26:33.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:26:33.541 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:26:33.541 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:26:33.541 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:26:33.541 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:26:33.541 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: 2021-07-31T02:26:33.542 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:26:33.542 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: 2021-07-31T02:26:33.542 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:26:33.542 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:26:33.543 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:26:33.543 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:26:33.543 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:26:33.543 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:26:33.543 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:26:33.544 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:26:33.544 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:26:33.544 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:26:33.544 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:26:33.545 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:26:33.545 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:26:33.545 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:26:33.545 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:26:33.545 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:26:33.546 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:26:33.546 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:26:33.546 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: raise 2021-07-31T02:26:33.546 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:26:33.547 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:26:33.547 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:26:33.547 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:26:33.547 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:26:33.548 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:33 smithi154 conmon[34553]: cluster 2021-07-31T02:26:33.233914+0000 mon.a (mon.0) 1980 : cluster [DBG] osdmap e1076: 8 total, 8 up, 8 in 2021-07-31T02:26:33.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:26:31.898599+0000 2021-07-31T02:26:33.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: mgr.x (mgr.24542) 1679 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:26:33.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: cephadm 2021- 2021-07-31T02:26:33.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: 07-31T02:26:31.943166+0000 mgr.x (mgr.24542) 1680 : cephadm [ERR] 2021-07-31T02:26:33.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:26:33.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:26:33.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: yield 2021-07-31T02:26:33.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:26:33.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:26:33.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:26:33.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:26:33.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:26:33.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:26:33.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:26:33.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:26:33.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:26:33.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:26:33.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:26:33.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:26:31.943744+0000 mgr.x (mgr.24542) 1681 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:26:33.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:26:31.954659+0000 mgr.x (mgr.24542) 1682 : cephadm [ERR] 2021-07-31T02:26:33.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:26:33.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:26:33.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: yield 2021-07-31T02:26:33.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:26:33.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:26:33.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:26:33.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:26:33.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:26:33.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:26:33.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:26:33.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:26:33.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:26:33.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:26:33.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:26:33.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: cephadm 2021-07 2021-07-31T02:26:33.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: -31T02:26:31.955082+0000 mgr.x (mgr.24542) 1683 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:26:33.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:26:31.980824+0000 mgr.x (mgr.24542) 1684 : cephadm [ERR] 2021-07-31T02:26:33.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:26:33.556 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:26:33.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: yield 2021-07-31T02:26:33.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:26:33.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:26:33.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:26:33.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:26:33.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:26:33.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:26:33.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:26:33.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:26:33.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:26:33.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:26:33.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:26:33.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:26:31.981289+0000 mgr.x (mgr.24542) 1685 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:26:33.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:26:33.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:26:33.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: yield 2021-07-31T02:26:33.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:26:33.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:26:33.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:26:33.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:26:33.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:26:33.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:26:33.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:26:33.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:26:33.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:26:33.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:26:33.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:26:33.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: 2021-07-31T02:26:33.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:26:33.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: 2021-07-31T02:26:33.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:26:33.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:26:33.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:26:33.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:26:33.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:26:33.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:26:33.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:26:33.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:26:33.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:26:33.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:26:33.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:26:33.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: yield 2021-07-31T02:26:33.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:26:33.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:26:33.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:26:33.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:26:33.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:26:33.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:26:33.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:26:33.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:26:33.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:26:33.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:26:33.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:26:33.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: cephadm 2021-07-31T02: 2021-07-31T02:26:33.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: 26:31.981289+0000 mgr.x (mgr.24542) 1685 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:26:33.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:26:33.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:26:33.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: yield 2021-07-31T02:26:33.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:26:33.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:26:33.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:26:33.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:26:33.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:26:33.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:26:33.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:26:33.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:26:33.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:26:33.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:26:33.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:26:33.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: 2021-07-31T02:26:33.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:26:33.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: 2021-07-31T02:26:33.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:26:33.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:26:33.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:26:33.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:26:33.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:26:33.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:26:33.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:26:33.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:26:33.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:26:33.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:26:33.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:26:33.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:26:33.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:26:33.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:26:33.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:26:33.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:26:33.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:26:33.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:26:33.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: raise 2021-07-31T02:26:33.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:26:33.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:26:33.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:26:33.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:26:33.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:26:33.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:26:31.988169+0000 mgr.x 2021-07-31T02:26:33.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: (mgr.24542) 1686 : cephadm [ERR] 2021-07-31T02:26:33.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:26:33.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:26:33.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: yield 2021-07-31T02:26:33.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:26:33.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:26:33.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:26:33.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:26:33.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:26:33.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:26:33.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:26:33.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:26:33.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:26:33.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:26:33.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:26:33.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:26:31.988618+0000 mgr.x (mgr.24542) 1687 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:26:33.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:26:33.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:26:33.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: yield 2021-07-31T02:26:33.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:26:33.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:26:33.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:26:33.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:26:33.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:26:33.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:26:33.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:26:33.585 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:26:33.585 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:26:33.585 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:26:33.585 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:26:33.586 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:26:33.586 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:26:33.586 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:26:33.586 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: raise 2021-07-31T02:26:33.586 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:26:33.587 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:26:33.587 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:26:33.587 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:26:33.587 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:26:33.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:26:31.988169+0000 mgr.x (mgr.24542) 1686 : cephadm [ERR] 2021-07-31T02:26:33.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:26:33.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:26:33.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: yield 2021-07-31T02:26:33.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:26:33.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:26:33.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:26:33.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:26:33.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:26:33.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:26:33.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:26:33.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:26:33.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:26:33.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:26:33.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:26:33.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:26:31.988618+0000 mgr.x (mgr 2021-07-31T02:26:33.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: .24542) 1687 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:26:33.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:26:33.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:26:33.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: yield 2021-07-31T02:26:33.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:26:33.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:26:33.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:26:33.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:26:33.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:26:33.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:26:33.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:26:33.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:26:33.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:26:33.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:26:33.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: 2021-07-31T02:26:33.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:26:33.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: 2021-07-31T02:26:33.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:26:33.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:26:33.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:26:33.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:26:33.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:26:33.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:26:33.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:26:33.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:26:33.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:26:33.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:26:33.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:26:33.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:26:33.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:26:33.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:26:33.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:26:33.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:26:33.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:26:33.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:26:33.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: raise 2021-07-31T02:26:33.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:26:33.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:26:33.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:26:33.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:26:33.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:26:33.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[33655]: cluster 2021-07-31T02:26:33.233914+0000 mon.a (mon.0) 1980 : cluster [DBG] osdmap e1076: 8 total, 8 up, 8 in 2021-07-31T02:26:33.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:26:33.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:26:33.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:26:33.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:26:33.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:26:33.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:26:33.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: 2021-07-31T02:26:33.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:26:33.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: 2021-07-31T02:26:33.603 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:26:33.603 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:26:33.603 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:26:33.603 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:26:33.603 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:26:33.604 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:26:33.604 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:26:33.604 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:26:33.604 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:26:33.604 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:26:33.605 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:26:33.605 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:26:33.605 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:26:33.605 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:26:33.606 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:26:33.606 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:26:33.606 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:26:33.606 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:26:33.606 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: raise 2021-07-31T02:26:33.607 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:26:33.607 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:26:33.607 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:26:33.607 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:26:33.607 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:26:33.608 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[39718]: cluster 2021-07-31T02:26:33.233914+0000 mon.a (mon.0) 1980 : cluster [DBG] osdmap e1076: 8 total, 8 up, 8 in 2021-07-31T02:26:34.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:26:33.577Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:26:34.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:26:33.577Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:26:34.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:26:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:26:33.577Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:26:34.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:34 smithi045 conmon[39718]: cluster 2021-07-31T02:26:32.355217+0000 mgr.x (mgr 2021-07-31T02:26:34.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:34 smithi045 conmon[39718]: .24542) 1688 : cluster [DBG] pgmap v2101: 169 pgs: 2 peering, 167 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 8.5 KiB/s rd, 2.2 KiB/s wr, 13 op/s 2021-07-31T02:26:34.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:34 smithi045 conmon[39718]: cluster 2021-07-31T02 2021-07-31T02:26:34.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:34 smithi045 conmon[39718]: :26:34.235788+0000 mon.a (mon.0) 1981 : cluster [DBG] osdmap e1077: 8 total, 8 up, 8 in 2021-07-31T02:26:34.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:34 smithi045 conmon[33655]: cluster 2021-07-31T02: 2021-07-31T02:26:34.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:34 smithi045 conmon[33655]: 26:32.355217+0000 mgr.x (mgr.24542) 1688 : cluster [DBG] pgmap v2101: 169 pgs: 2 peering, 167 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 8.5 KiB/s rd, 2.2 KiB/s wr, 13 op/s 2021-07-31T02:26:34.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:34 smithi045 conmon[33655]: cluster 2021-07-31T02:26:34.235788+0000 mon.a (mon. 2021-07-31T02:26:34.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:34 smithi045 conmon[33655]: 0) 1981 : cluster [DBG] osdmap e1077: 8 total, 8 up, 8 in 2021-07-31T02:26:34.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:34 smithi154 conmon[34553]: cluster 2021-07-31T02:26:32.355217+0000 mgr.x (mgr.24542) 1688 : cluster [DBG] pgmap v2101: 169 pgs: 2 peering, 167 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 8.5 KiB/s rd, 2.2 KiB/s wr, 13 op/s 2021-07-31T02:26:34.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:34 smithi154 conmon[34553]: cluster 2021-07-31T02:26:34.235788+0000 mon.a (mon.0) 1981 : cluster [DBG] osdmap e1077: 8 total, 8 up, 8 in 2021-07-31T02:26:36.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:36 smithi154 conmon[34553]: cluster 2021-07-31T02:26:34.355661+0000 mgr.x (mgr.24542) 1689 2021-07-31T02:26:36.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:36 smithi154 conmon[34553]: : cluster [DBG] pgmap v2104: 169 pgs: 169 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2021-07-31T02:26:36.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:36 smithi154 conmon[34553]: cluster 2021-07-31T02:26:35.237292+0000 mon.a (mon.0) 1982 : cluster [DBG] osdmap e1078: 8 total, 8 up, 8 in 2021-07-31T02:26:36.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:36 smithi045 conmon[39718]: cluster 2021-07-31T02:26:34.355661+0000 mgr.x (mgr. 2021-07-31T02:26:36.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:36 smithi045 conmon[39718]: 24542) 1689 : cluster [DBG] pgmap v2104: 169 pgs: 169 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2021-07-31T02:26:36.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:36 smithi045 conmon[39718]: cluster 2021-07-31T02:26:35.237292+0000 mon.a 2021-07-31T02:26:36.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:36 smithi045 conmon[39718]: (mon.0) 1982 : cluster [DBG] osdmap e1078: 8 total, 8 up, 8 in 2021-07-31T02:26:36.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:36 smithi045 conmon[33655]: cluster 2021-07-31T02:26:34.355661+0000 mgr.x (mgr.24542 2021-07-31T02:26:36.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:36 smithi045 conmon[33655]: ) 1689 : cluster [DBG] pgmap v2104: 169 pgs: 169 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2021-07-31T02:26:36.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:36 smithi045 conmon[33655]: cluster 2021-07-31T02:26:35.237292+0000 mon.a (mon.0) 1982 : cluster [DBG] osdmap e1078: 8 total, 8 up, 8 in 2021-07-31T02:26:37.120 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:26:37.240 INFO:tasks.workunit.client.0.smithi045.stdout:num snaps is: 0 2021-07-31T02:26:37.240 INFO:tasks.workunit.client.0.smithi045.stdout:expected: 0 2021-07-31T02:26:37.244 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.TestGetNameIdSnapPP (5986 ms) 2021-07-31T02:26:37.244 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.TestCreateLsRenameSnapPP 2021-07-31T02:26:37.244 INFO:tasks.workunit.client.0.smithi045.stdout:using old format 2021-07-31T02:26:37.412 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:26:37.412 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 41m ago 42m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:26:37.412 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 39m ago 41m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:26:37.412 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 39m ago 45m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:26:37.413 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 41m ago 47m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:26:37.413 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 41m ago 47m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:26:37.413 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 39m ago 45m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:26:37.413 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 41m ago 45m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:26:37.414 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 41m ago 42m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:26:37.414 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 39m ago 42m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:26:37.414 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 41m ago 45m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:26:37.414 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 41m ago 44m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:26:37.414 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 41m ago 44m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:26:37.415 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 41m ago 44m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:26:37.415 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 39m ago 43m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:26:37.415 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 39m ago 43m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:26:37.415 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 39m ago 43m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:26:37.415 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 39m ago 42m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:26:37.416 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 39m ago 42m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:26:37.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:37 smithi154 conmon[34553]: cluster 2021-07-31T02:26:37.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:37 smithi154 conmon[34553]: 2021-07-31T02:26:37.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:37 smithi154 conmon[34553]: 2021-07-31T02:26:36.243606 2021-07-31T02:26:37.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:37 smithi154 conmon[34553]: +0000 mon.a (mon.0) 1983 : cluster [DBG] osdmap e1079: 8 total, 8 up, 8 in 2021-07-31T02:26:37.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:37 smithi154 conmon[34553]: cluster 2021-07-31T02:26:37.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:37 smithi154 conmon[34553]: 2021-07-31T02:26 2021-07-31T02:26:37.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:37 smithi154 conmon[34553]: :37.224726+0000 mon.a ( 2021-07-31T02:26:37.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:37 smithi154 conmon[34553]: mon.0) 1984 : cluster [DBG] osdmap e1080: 8 total, 8 up, 8 in 2021-07-31T02:26:37.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:37 smithi045 conmon[39718]: cluster 2021-07-31T02:26:36.243606+0000 mon.a (mon. 2021-07-31T02:26:37.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:37 smithi045 conmon[39718]: 0) 1983 : cluster [DBG] osdmap e1079: 8 total, 8 up, 8 in 2021-07-31T02:26:37.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:37 smithi045 conmon[39718]: cluster 2021-07-31T02:26:37.224726+0000 mon.a ( 2021-07-31T02:26:37.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:37 smithi045 conmon[39718]: mon.0) 1984 : cluster [DBG] osdmap e1080: 8 total, 8 up, 8 in 2021-07-31T02:26:37.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:37 smithi045 conmon[33655]: cluster 2021-07-31T02:26:36.243606+0000 mon.a 2021-07-31T02:26:37.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:37 smithi045 conmon[33655]: (mon.0) 1983 : cluster [DBG] osdmap e1079: 8 total, 8 up, 8 in 2021-07-31T02:26:37.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:37 smithi045 conmon[33655]: cluster 2021-07-31T02:26:37.224726+0000 mon.a (mon.0) 1984 : cluster [DBG] osdmap e1080: 8 total, 8 up, 8 in 2021-07-31T02:26:37.752 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:26:37.753 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:26:37.753 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:26:37.753 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:26:37.754 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:26:37.754 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:26:37.754 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:26:37.754 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:26:37.755 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:26:37.755 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:26:37.755 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:26:37.755 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:26:37.755 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:26:37.756 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:26:37.756 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:26:37.756 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:26:37.756 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:26:38.263 INFO:tasks.workunit.client.0.smithi045.stdout:num snaps is: 1 2021-07-31T02:26:38.263 INFO:tasks.workunit.client.0.smithi045.stdout:expected: 1 2021-07-31T02:26:38.263 INFO:tasks.workunit.client.0.smithi045.stdout:snap: snap1 2021-07-31T02:26:38.264 INFO:tasks.workunit.client.0.smithi045.stdout:found snap1 with size 2097152 2021-07-31T02:26:38.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:38 smithi154 conmon[34553]: cluster 2021-07-31T02:26:36.356252+0000 mgr.x (mgr.24542) 1690 : cluster [DBG] pgmap v2107: 169 pgs: 169 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2021-07-31T02:26:38.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:38 smithi154 conmon[34553]: audit 2021-07-31T02:26:37.108577+0000 mgr.x (mgr.24542) 1691 : audit [DBG] from='client.17082 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:26:38.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:38 smithi154 conmon[34553]: audit 2021-07-31T02:26:37.752988+0000 mon.c (mon.1) 98 : 2021-07-31T02:26:38.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:38 smithi154 conmon[34553]: audit [DBG] from='client.? 172.21.15.45:0/3078540722' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:26:38.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:38 smithi045 conmon[33655]: cluster 2021-07-31T02:26:36.356252+0000 mgr.x ( 2021-07-31T02:26:38.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:38 smithi045 conmon[33655]: mgr.24542) 1690 : cluster [DBG] pgmap v2107: 169 pgs: 169 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2021-07-31T02:26:38.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:38 smithi045 conmon[33655]: audit 2021-07-31T02:26:37.108577+0000 mgr.x (mgr.24542) 1691 : audit [DBG] from='client.17082 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:26:38.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:38 smithi045 conmon[33655]: audit 2021-07-31T02:26:37.752988+0000 mon.c (mon. 2021-07-31T02:26:38.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:38 smithi045 conmon[33655]: 1) 98 : audit [DBG] from='client.? 172.21.15.45:0/3078540722' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:26:38.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:38 smithi045 conmon[39718]: cluster 2021-07-31T02:26:36 2021-07-31T02:26:38.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:38 smithi045 conmon[39718]: .356252+0000 mgr.x (mgr.24542) 1690 : cluster [DBG] pgmap v2107: 169 pgs: 169 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2021-07-31T02:26:38.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:38 smithi045 conmon[39718]: audit 2021-07 2021-07-31T02:26:38.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:38 smithi045 conmon[39718]: -31T02:26:37. 2021-07-31T02:26:38.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:38 smithi045 conmon[39718]: 108577+0000 mgr.x (mgr. 2021-07-31T02:26:38.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:38 smithi045 conmon[39718]: 24542) 1691 : audit [DBG] from='client.17082 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:26:38.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:38 smithi045 conmon[39718]: 2021-07-31T02:26:38.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:38 smithi045 conmon[39718]: audit 2021-07 2021-07-31T02:26:38.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:38 smithi045 conmon[39718]: -31T02:26:37.752988+0000 2021-07-31T02:26:38.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:38 smithi045 conmon[39718]: mon.c (mon.1) 98 2021-07-31T02:26:38.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:38 smithi045 conmon[39718]: : audit [DBG] from='client.? 172.21.15.45:0/3078540722' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:26:39.268 INFO:tasks.workunit.client.0.smithi045.stdout:num snaps is: 2 2021-07-31T02:26:39.268 INFO:tasks.workunit.client.0.smithi045.stdout:expected: 2 2021-07-31T02:26:39.269 INFO:tasks.workunit.client.0.smithi045.stdout:snap: snap1 2021-07-31T02:26:39.269 INFO:tasks.workunit.client.0.smithi045.stdout:snap: snap2 2021-07-31T02:26:39.269 INFO:tasks.workunit.client.0.smithi045.stdout:found snap1 with size 2097152 2021-07-31T02:26:39.269 INFO:tasks.workunit.client.0.smithi045.stdout:found snap2 with size 4194304 2021-07-31T02:26:39.271 INFO:tasks.workunit.client.0.smithi045.stdout:num snaps is: 2 2021-07-31T02:26:39.271 INFO:tasks.workunit.client.0.smithi045.stdout:expected: 2 2021-07-31T02:26:39.272 INFO:tasks.workunit.client.0.smithi045.stdout:snap: snap1-rename 2021-07-31T02:26:39.272 INFO:tasks.workunit.client.0.smithi045.stdout:snap: snap2 2021-07-31T02:26:39.272 INFO:tasks.workunit.client.0.smithi045.stdout:found snap1-rename with size 2097152 2021-07-31T02:26:39.272 INFO:tasks.workunit.client.0.smithi045.stdout:found snap2 with size 4194304 2021-07-31T02:26:39.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:39 smithi045 conmon[39718]: audit 2021-07-31T02:26:37.404187+0000 mgr.x (mgr.24542 2021-07-31T02:26:39.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:39 smithi045 conmon[39718]: ) 1692 : audit [DBG] from='client.17088 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:26:39.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:39 smithi045 conmon[39718]: cluster 2021-07-31T02:26:38.255544+0000 mon.a (mon.0) 1985 : cluster [DBG] osdmap e1081: 8 total, 8 up, 8 in 2021-07-31T02:26:39.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:39 smithi045 conmon[33655]: audit 2021-07-31T02:26:37.404187+0000 mgr.x (mgr.24542) 2021-07-31T02:26:39.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:39 smithi045 conmon[33655]: 1692 : audit [DBG] from='client.17088 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:26:39.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:39 smithi045 conmon[33655]: cluster 2021-07-31T02:26:38.255544+0000 mon.a (mon.0) 1985 : cluster [DBG] osdmap e1081: 8 total, 8 up, 8 in 2021-07-31T02:26:39.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:39 smithi154 conmon[34553]: audit 2021-07-31T02:26:37.404187+0000 mgr.x (mgr.24542 2021-07-31T02:26:39.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:39 smithi154 conmon[34553]: ) 1692 : audit [DBG] from='client.17088 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:26:39.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:39 smithi154 conmon[34553]: cluster 2021-07-31T02:26:38.255544+0000 mon.a (mon.0) 1985 : cluster [DBG] osdmap e1081: 8 total, 8 up, 8 in 2021-07-31T02:26:40.275 INFO:tasks.workunit.client.0.smithi045.stdout:num snaps is: 1 2021-07-31T02:26:40.276 INFO:tasks.workunit.client.0.smithi045.stdout:expected: 1 2021-07-31T02:26:40.276 INFO:tasks.workunit.client.0.smithi045.stdout:snap: snap2-rename 2021-07-31T02:26:40.276 INFO:tasks.workunit.client.0.smithi045.stdout:found snap2-rename with size 4194304 2021-07-31T02:26:40.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:40 smithi045 conmon[39718]: cluster 2021-07-31T02:26:38.356735+0000 2021-07-31T02:26:40.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:40 smithi045 conmon[39718]: mgr.x (mgr.24542) 1693 : cluster [DBG] pgmap v2110: 169 pgs: 169 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:26:40.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:40 smithi045 conmon[39718]: cluster 2021-07-31T02:26:40.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:40 smithi045 conmon[39718]: 2021-07-31T02:26:39.261675+0000 mon.a (mon.0) 1986 : cluster 2021-07-31T02:26:40.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:40 smithi045 conmon[39718]: [DBG] osdmap e1082: 8 total, 8 up, 8 in 2021-07-31T02:26:40.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:40 smithi045 conmon[33655]: cluster 2021-07-31T02:26:38.356735+0000 mgr.x (mgr.24542) 2021-07-31T02:26:40.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:40 smithi045 conmon[33655]: 1693 : cluster [DBG] pgmap v2110: 169 pgs: 169 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:26:40.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:40 smithi045 conmon[33655]: cluster 2021-07-31T02:26:39.261675+0000 mon.a (mon.0) 1986 : cluster [DBG] osdmap e1082: 8 total, 8 up, 8 in 2021-07-31T02:26:40.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:40 smithi154 conmon[34553]: cluster 2021-07-31T02:26:40.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:40 smithi154 conmon[34553]: 2021-07-31T02:26:40.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:40 smithi154 conmon[34553]: 2021-07-31T02:26:38.356735+0000 mgr.x 2021-07-31T02:26:40.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:40 smithi154 conmon[34553]: (mgr.24542) 1693 : cluster [DBG] pgmap v2110: 169 pgs: 169 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:26:40.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:40 smithi154 conmon[34553]: cluster 2021-07-31T02:26:40.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:40 smithi154 conmon[34553]: 2021-07-31T02:26:39.261675+0000 mon.a (mon.0) 1986 : cluster [DBG] 2021-07-31T02:26:40.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:40 smithi154 conmon[34553]: osdmap e1082: 8 total, 8 up, 8 in 2021-07-31T02:26:41.286 INFO:tasks.workunit.client.0.smithi045.stdout:num snaps is: 0 2021-07-31T02:26:41.287 INFO:tasks.workunit.client.0.smithi045.stdout:expected: 0 2021-07-31T02:26:41.289 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.TestCreateLsRenameSnapPP (4043 ms) 2021-07-31T02:26:41.289 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.TestIO 2021-07-31T02:26:41.290 INFO:tasks.workunit.client.0.smithi045.stdout:using old format 2021-07-31T02:26:41.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:41 smithi045 conmon[33655]: cluster 2021-07-31T02:26:40.263269+0000 mon.a 2021-07-31T02:26:41.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:41 smithi045 conmon[33655]: (mon.0) 1987 : cluster [DBG] osdmap e1083: 8 total, 8 up, 8 in 2021-07-31T02:26:41.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:41 smithi045 conmon[39718]: cluster 2021-07-31T02:26:40.263269+0000 mon.a 2021-07-31T02:26:41.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:41 smithi045 conmon[39718]: (mon.0) 1987 : cluster [DBG] osdmap e1083: 8 total, 8 up, 8 in 2021-07-31T02:26:41.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:41 smithi154 conmon[34553]: cluster 2021-07-31T02:26:41.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:41 smithi154 conmon[34553]: 2021-07-31T02:26:40. 2021-07-31T02:26:41.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:41 smithi154 conmon[34553]: 263269+0000 mon.a (mon.0) 1987 : cluster [DBG] osdmap e1083: 8 total, 8 up, 8 in 2021-07-31T02:26:42.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:42 smithi045 conmon[39718]: cluster 2021-07-31T02:26:40.357317+0000 mgr.x 2021-07-31T02:26:42.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:42 smithi045 conmon[39718]: (mgr.24542) 1694 : cluster [DBG] pgmap v2113: 169 pgs: 169 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 2.0 KiB/s wr, 11 op/s 2021-07-31T02:26:42.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:42 smithi045 conmon[39718]: cluster 2021-07-31T02: 2021-07-31T02:26:42.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:42 smithi045 conmon[39718]: 26:41.277038+0000 mon.a (mon.0) 1988 : cluster [DBG] osdmap e1084: 8 total, 8 up, 8 in 2021-07-31T02:26:42.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:42 smithi045 conmon[39718]: cluster 2021-07-31T02:26:42.225437+0000 2021-07-31T02:26:42.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:42 smithi045 conmon[39718]: mon.a (mon.0) 1989 : cluster [DBG] 2021-07-31T02:26:42.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:42 smithi045 conmon[39718]: osdmap e1085: 8 total, 8 up, 8 in 2021-07-31T02:26:42.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:42 smithi045 conmon[33655]: cluster 2021-07-31T02:26:40.357317 2021-07-31T02:26:42.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:42 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 1694 : cluster [DBG] pgmap v2113: 169 pgs: 169 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 2.0 KiB/s wr, 11 op/s 2021-07-31T02:26:42.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:42 smithi045 conmon[33655]: cluster 2021-07-31T02:26:41.277038+0000 mon.a 2021-07-31T02:26:42.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:42 smithi045 conmon[33655]: (mon.0) 1988 : cluster [DBG] osdmap e1084: 8 total, 8 up, 8 in 2021-07-31T02:26:42.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:42 smithi045 conmon[33655]: cluster 2021-07- 2021-07-31T02:26:42.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:42 smithi045 conmon[33655]: 31T02:26:42. 2021-07-31T02:26:42.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:42 smithi045 conmon[33655]: 225437+0000 mon.a (mon.0) 1989 : cluster [DBG] 2021-07-31T02:26:42.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:42 smithi045 conmon[33655]: osdmap e1085: 8 total, 8 up, 8 in 2021-07-31T02:26:42.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:42 smithi154 conmon[34553]: cluster 2021-07-31T02:26:40.357317+0000 mgr.x ( 2021-07-31T02:26:42.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:42 smithi154 conmon[34553]: mgr.24542) 1694 : cluster [DBG] pgmap v2113: 169 pgs: 169 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 2.0 KiB/s wr, 11 op/s 2021-07-31T02:26:42.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:42 smithi154 conmon[34553]: cluster 2021-07-31T02:26:41.277038+0000 mon.a (mon.0) 1988 : cluster [DBG] osdmap e1084: 8 total, 8 up, 8 in 2021-07-31T02:26:42.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:42 smithi154 conmon[34553]: cluster 2021-07-31T02:26:42.225437+0000 mon.a (mon.0) 1989 : cluster [DBG] osdmap e1085: 8 total, 8 up, 8 in 2021-07-31T02:26:42.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:42 smithi154 conmon[34553]: 2021-07-31T02:26:43.836 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 512 2021-07-31T02:26:43.836 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 512 2021-07-31T02:26:43.836 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 512 2021-07-31T02:26:43.837 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 512 2021-07-31T02:26:43.837 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 512 2021-07-31T02:26:43.837 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:43.837 INFO:tasks.workunit.client.0.smithi045.stdout:started write 2021-07-31T02:26:43.837 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:26:43.838 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:26:43.838 INFO:tasks.workunit.client.0.smithi045.stdout:finished write 2021-07-31T02:26:43.838 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:43.838 INFO:tasks.workunit.client.0.smithi045.stdout:started write 2021-07-31T02:26:43.839 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:26:43.839 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:26:43.839 INFO:tasks.workunit.client.0.smithi045.stdout:finished write 2021-07-31T02:26:43.839 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:43.839 INFO:tasks.workunit.client.0.smithi045.stdout:started write 2021-07-31T02:26:43.840 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:26:43.840 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:26:43.840 INFO:tasks.workunit.client.0.smithi045.stdout:finished write 2021-07-31T02:26:43.840 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:43.840 INFO:tasks.workunit.client.0.smithi045.stdout:started write 2021-07-31T02:26:43.841 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:26:43.841 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:26:43.841 INFO:tasks.workunit.client.0.smithi045.stdout:finished write 2021-07-31T02:26:43.841 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:43.841 INFO:tasks.workunit.client.0.smithi045.stdout:started write 2021-07-31T02:26:43.842 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:26:43.842 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:26:43.842 INFO:tasks.workunit.client.0.smithi045.stdout:finished write 2021-07-31T02:26:43.842 INFO:tasks.workunit.client.0.smithi045.stdout:start compare and write 2021-07-31T02:26:43.843 INFO:tasks.workunit.client.0.smithi045.stdout:compare and wrote: 512 2021-07-31T02:26:43.843 INFO:tasks.workunit.client.0.smithi045.stdout:start compare and write 2021-07-31T02:26:43.843 INFO:tasks.workunit.client.0.smithi045.stdout:compare and wrote: 512 2021-07-31T02:26:43.843 INFO:tasks.workunit.client.0.smithi045.stdout:start compare and write 2021-07-31T02:26:43.843 INFO:tasks.workunit.client.0.smithi045.stdout:compare and wrote: 512 2021-07-31T02:26:43.844 INFO:tasks.workunit.client.0.smithi045.stdout:start compare and write 2021-07-31T02:26:43.844 INFO:tasks.workunit.client.0.smithi045.stdout:compare and wrote: 512 2021-07-31T02:26:43.844 INFO:tasks.workunit.client.0.smithi045.stdout:start compare and write 2021-07-31T02:26:43.844 INFO:tasks.workunit.client.0.smithi045.stdout:compare and wrote: 512 2021-07-31T02:26:43.844 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:43.845 INFO:tasks.workunit.client.0.smithi045.stdout:started aio compare and write 2021-07-31T02:26:43.845 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:26:43.845 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:26:43.845 INFO:tasks.workunit.client.0.smithi045.stdout:finished aio compare and write 2021-07-31T02:26:43.846 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:43.846 INFO:tasks.workunit.client.0.smithi045.stdout:started aio compare and write 2021-07-31T02:26:43.846 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:26:43.846 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:26:43.846 INFO:tasks.workunit.client.0.smithi045.stdout:finished aio compare and write 2021-07-31T02:26:43.847 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:43.847 INFO:tasks.workunit.client.0.smithi045.stdout:started aio compare and write 2021-07-31T02:26:43.847 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:26:43.847 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:26:43.847 INFO:tasks.workunit.client.0.smithi045.stdout:finished aio compare and write 2021-07-31T02:26:43.848 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:43.848 INFO:tasks.workunit.client.0.smithi045.stdout:started aio compare and write 2021-07-31T02:26:43.848 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:26:43.848 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:26:43.848 INFO:tasks.workunit.client.0.smithi045.stdout:finished aio compare and write 2021-07-31T02:26:43.849 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:43.849 INFO:tasks.workunit.client.0.smithi045.stdout:started aio compare and write 2021-07-31T02:26:43.849 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:26:43.850 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:26:43.850 INFO:tasks.workunit.client.0.smithi045.stdout:finished aio compare and write 2021-07-31T02:26:43.851 INFO:tasks.workunit.client.0.smithi045.stdout:read: 512 2021-07-31T02:26:43.851 INFO:tasks.workunit.client.0.smithi045.stdout:read: 512 2021-07-31T02:26:43.851 INFO:tasks.workunit.client.0.smithi045.stdout:read: 512 2021-07-31T02:26:43.851 INFO:tasks.workunit.client.0.smithi045.stdout:read: 512 2021-07-31T02:26:43.851 INFO:tasks.workunit.client.0.smithi045.stdout:read: 512 2021-07-31T02:26:43.852 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:43.852 INFO:tasks.workunit.client.0.smithi045.stdout:started read 2021-07-31T02:26:43.852 INFO:tasks.workunit.client.0.smithi045.stdout:read completion cb called! 2021-07-31T02:26:43.852 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 512 2021-07-31T02:26:43.853 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:43.853 INFO:tasks.workunit.client.0.smithi045.stdout:started read 2021-07-31T02:26:43.853 INFO:tasks.workunit.client.0.smithi045.stdout:read completion cb called! 2021-07-31T02:26:43.853 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 512 2021-07-31T02:26:43.854 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:43.854 INFO:tasks.workunit.client.0.smithi045.stdout:started read 2021-07-31T02:26:43.854 INFO:tasks.workunit.client.0.smithi045.stdout:read completion cb called! 2021-07-31T02:26:43.854 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 512 2021-07-31T02:26:43.855 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:43.855 INFO:tasks.workunit.client.0.smithi045.stdout:started read 2021-07-31T02:26:43.855 INFO:tasks.workunit.client.0.smithi045.stdout:read completion cb called! 2021-07-31T02:26:43.855 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 512 2021-07-31T02:26:43.855 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:43.856 INFO:tasks.workunit.client.0.smithi045.stdout:started read 2021-07-31T02:26:43.856 INFO:tasks.workunit.client.0.smithi045.stdout:read completion cb called! 2021-07-31T02:26:43.856 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 512 2021-07-31T02:26:43.856 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 512~512 = 512 2021-07-31T02:26:43.856 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:26:43.857 INFO:tasks.workunit.client.0.smithi045.stdout:aio discard: 1536~512 = 0 2021-07-31T02:26:43.857 INFO:tasks.workunit.client.0.smithi045.stdout:read: 512 2021-07-31T02:26:43.857 INFO:tasks.workunit.client.0.smithi045.stdout:read: 512 2021-07-31T02:26:43.858 INFO:tasks.workunit.client.0.smithi045.stdout:read: 512 2021-07-31T02:26:43.858 INFO:tasks.workunit.client.0.smithi045.stdout:read: 512 2021-07-31T02:26:43.858 INFO:tasks.workunit.client.0.smithi045.stdout:read: 512 2021-07-31T02:26:43.858 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 0 2021-07-31T02:26:43.858 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:26:43.859 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:26:43.859 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 0 2021-07-31T02:26:43.859 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:26:43.859 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:26:43.859 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 16384 2021-07-31T02:26:43.860 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:26:43.860 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:26:43.860 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 16384 2021-07-31T02:26:43.860 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:26:43.861 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:26:43.861 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:26:43.861 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:26:43.861 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 49152 2021-07-31T02:26:43.861 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:26:43.862 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:26:43.862 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 49152 2021-07-31T02:26:43.862 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:26:43.862 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:26:43.862 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 65536 2021-07-31T02:26:43.863 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:26:43.863 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:26:43.863 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 65536 2021-07-31T02:26:43.867 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:26:43.867 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:26:43.867 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:26:43.867 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:26:43.868 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 98304 2021-07-31T02:26:43.868 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:26:43.868 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:26:43.868 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 98304 2021-07-31T02:26:43.868 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:26:43.869 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:26:43.869 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 114688 2021-07-31T02:26:43.869 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:26:43.869 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:26:43.870 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 114688 2021-07-31T02:26:43.870 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:26:43.870 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:26:43.871 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:26:43.871 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:26:43.871 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 147456 2021-07-31T02:26:43.872 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:26:43.872 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:26:43.872 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 147456 2021-07-31T02:26:43.872 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:26:43.872 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:26:43.873 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 163840 2021-07-31T02:26:43.873 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:26:43.873 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:26:43.873 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 163840 2021-07-31T02:26:43.873 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:26:43.874 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:26:43.874 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:26:43.874 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:26:43.874 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 196608 2021-07-31T02:26:43.875 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:26:43.875 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:26:43.875 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 196608 2021-07-31T02:26:43.875 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:26:43.876 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:26:43.876 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 212992 2021-07-31T02:26:43.876 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:26:43.877 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:26:43.877 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 212992 2021-07-31T02:26:43.877 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:26:43.877 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:26:43.878 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:26:43.878 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:26:43.878 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:43.878 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:26:43.879 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:26:43.879 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:26:43.879 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:26:43.879 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:26:43.879 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:26:43.880 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:43.880 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:26:43.880 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:26:43.880 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:26:43.880 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:26:43.881 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:26:43.881 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:26:43.881 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:43.881 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:26:43.882 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:26:43.882 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:26:43.882 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:26:43.882 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:26:43.882 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:26:43.883 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:43.883 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:26:43.884 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:26:43.884 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:26:43.884 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:26:43.884 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:26:43.885 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:26:43.885 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:43.885 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:26:43.886 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:26:43.886 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:43.886 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:26:43.886 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:26:43.886 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:43.887 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:26:43.887 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:26:43.887 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:26:43.887 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:26:43.887 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:26:43.888 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:26:43.888 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:43.888 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:26:43.888 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:26:43.888 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:26:43.889 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:26:43.889 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:26:43.889 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:26:43.889 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:44.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:26:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:26:43.577Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:26:44.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:26:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:26:43.577Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:26:44.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:26:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:26:43.577Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:26:44.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:44 smithi154 conmon[34553]: cluster 2021-07-31T02:26:42.357787+0000 mgr.x (mgr.24542) 1695 : cluster [DBG] 2021-07-31T02:26:44.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:44 smithi154 conmon[34553]: pgmap v2116: 169 pgs: 169 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 2.0 KiB/s wr, 11 op/s 2021-07-31T02:26:44.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:44 smithi045 conmon[33655]: cluster 2021-07-31T02:26:42.357787+0000 mgr.x ( 2021-07-31T02:26:44.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:44 smithi045 conmon[33655]: mgr.24542) 1695 : cluster [DBG] pgmap v2116: 169 pgs: 169 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 2.0 KiB/s wr, 11 op/s 2021-07-31T02:26:44.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:44 smithi045 conmon[39718]: cluster 2021-07-31T02:26:42.357787 2021-07-31T02:26:44.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:44 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 1695 : cluster [DBG] pgmap v2116: 169 pgs: 169 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 2.0 KiB/s wr, 11 op/s 2021-07-31T02:26:45.916 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:26:45.916 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:26:45.917 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:26:45.917 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:26:45.917 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:26:45.917 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:26:45.918 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:45.918 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:26:45.918 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:26:45.919 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:26:45.919 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:26:45.919 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:26:45.920 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:26:45.920 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:45.920 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:26:45.921 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:26:45.921 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:45.921 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:26:45.921 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:26:45.922 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:45.922 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:26:45.922 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:26:45.922 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:26:45.922 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:26:45.923 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:26:45.923 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:26:45.923 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:45.923 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:26:45.924 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:26:45.924 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:26:45.924 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:26:45.924 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:26:45.924 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:26:45.925 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:45.925 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:26:45.925 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:26:45.925 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:26:45.925 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:26:45.926 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:26:45.926 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:26:45.926 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:45.926 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:26:45.927 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:26:45.927 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:26:45.927 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:26:45.928 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:26:45.928 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:26:45.928 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:45.929 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:26:45.929 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:26:45.929 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:45.930 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:26:45.930 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:26:45.930 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:45.930 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:26:45.930 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:26:45.931 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:26:45.931 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:26:45.931 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:26:45.931 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:26:45.932 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:45.932 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:26:45.932 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:26:45.932 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:26:45.933 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:26:45.933 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:26:45.933 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:26:45.933 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:45.934 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:26:45.934 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:26:45.934 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:26:45.934 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:26:45.934 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:26:45.935 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:26:45.935 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:45.935 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:26:45.936 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:26:45.936 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:26:45.936 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:26:45.936 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:26:45.937 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:26:45.937 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:45.937 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:26:45.937 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:26:45.938 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:45.938 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:26:45.938 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:26:45.939 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:45.939 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:26:45.939 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:26:45.939 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:26:45.940 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:26:45.940 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:26:45.940 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:26:45.940 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:45.941 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:26:45.941 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:26:45.941 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:26:45.941 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:26:45.942 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:26:45.942 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:26:45.942 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:45.942 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:26:45.942 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:26:45.943 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:26:45.943 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:26:45.943 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:26:45.943 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:26:45.944 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:45.944 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:26:45.944 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:26:45.944 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:26:45.945 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:26:45.945 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:26:45.945 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:26:45.945 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:45.946 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:26:45.946 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:26:45.946 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:45.947 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:26:45.947 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:26:45.947 INFO:tasks.workunit.client.0.smithi045.stdout:read completion cb called! 2021-07-31T02:26:45.947 INFO:tasks.workunit.client.0.smithi045.stdout:read completion cb called! 2021-07-31T02:26:45.948 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 512 2021-07-31T02:26:45.948 INFO:tasks.workunit.client.0.smithi045.stdout:read completion cb called! 2021-07-31T02:26:45.948 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.TestIO (4628 ms) 2021-07-31T02:26:45.948 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.TestEncryptionLUKS1 2021-07-31T02:26:45.949 INFO:tasks.workunit.client.0.smithi045.stdout:using old format 2021-07-31T02:26:46.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:46 smithi154 conmon[34553]: cluster 2021-07-31T02:26:44.358541+0000 mgr.x (mgr.24542) 1696 2021-07-31T02:26:46.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:46 smithi154 conmon[34553]: : cluster [DBG] pgmap v2117: 169 pgs: 169 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 469 KiB/s rd, 235 KiB/s wr, 925 op/s 2021-07-31T02:26:46.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:46 smithi045 conmon[33655]: cluster 2021-07-31T02:26:44. 2021-07-31T02:26:46.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:46 smithi045 conmon[33655]: 358541+0000 mgr.x (mgr.24542) 1696 : cluster [DBG] pgmap v2117: 169 pgs: 169 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 469 KiB/s rd, 235 KiB/s wr, 925 op/s 2021-07-31T02:26:46.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:46 smithi045 conmon[39718]: cluster 2021-07-31T02:26:44.358541 2021-07-31T02:26:46.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:46 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 1696 : cluster [DBG] pgmap v2117: 169 pgs: 169 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 469 KiB/s rd, 235 KiB/s wr, 925 op/s 2021-07-31T02:26:48.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:48 smithi154 conmon[34553]: cluster 2021-07-31T02:26:46.359090+0000 mgr.x (mgr.24542) 1697 : cluster [DBG] pgmap v2118: 169 pgs: 169 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 392 KiB/s rd, 196 KiB/s wr, 774 op/s 2021-07-31T02:26:48.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:48 smithi154 conmon[34553]: cluster 2021-07-31T02:26: 2021-07-31T02:26:48.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:48 smithi154 conmon[34553]: 47.227919+0000 mon.a (mon.0) 1990 : cluster [DBG] osdmap e1086: 8 total, 8 up, 8 in 2021-07-31T02:26:48.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:48 smithi045 conmon[39718]: cluster 2021-07-31T02:26:46.359090+0000 2021-07-31T02:26:48.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:48 smithi045 conmon[39718]: mgr.x (mgr.24542) 1697 : cluster [DBG] pgmap v2118: 169 pgs: 169 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 392 KiB/s rd, 196 KiB/s wr, 774 op/s 2021-07-31T02:26:48.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:48 smithi045 conmon[39718]: cluster 2021-07-31T02:26:47.227919+0000 2021-07-31T02:26:48.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:48 smithi045 conmon[39718]: mon.a (mon.0) 1990 : cluster [DBG] osdmap e1086: 8 total, 8 up, 8 in 2021-07-31T02:26:48.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:48 smithi045 conmon[33655]: cluster 2021-07-31T02:26:46.359090 2021-07-31T02:26:48.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:48 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 1697 : cluster [DBG] pgmap v2118: 169 pgs: 169 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 392 KiB/s rd, 196 KiB/s wr, 774 op/s 2021-07-31T02:26:48.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:48 smithi045 conmon[33655]: cluster 2021-07-31T02:26:48.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:48 smithi045 conmon[33655]: 2021-07-31T02:26:47.227919+0000 mon.a ( 2021-07-31T02:26:48.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:48 smithi045 conmon[33655]: mon.0) 1990 : cluster [DBG] osdmap e1086: 8 total, 8 up, 8 in 2021-07-31T02:26:50.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:50 smithi154 conmon[34553]: cluster 2021-07-31T02:26:48.359567+0000 mgr.x (mgr.24542) 2021-07-31T02:26:50.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:50 smithi154 conmon[34553]: 1698 : cluster [DBG] pgmap v2120: 169 pgs: 169 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 337 KiB/s rd, 169 KiB/s wr, 666 op/s 2021-07-31T02:26:50.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:50 smithi045 conmon[33655]: cluster 2021-07-31T02:26:48.359567+0000 mgr.x (mgr.24542 2021-07-31T02:26:50.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:50 smithi045 conmon[33655]: ) 1698 : cluster [DBG] pgmap v2120: 169 pgs: 169 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 337 KiB/s rd, 169 KiB/s wr, 666 op/s 2021-07-31T02:26:50.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:50 smithi045 conmon[39718]: cluster 2021-07-31T02:26:48.359567+0000 2021-07-31T02:26:50.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:50 smithi045 conmon[39718]: mgr.x (mgr.24542) 1698 : cluster [DBG] pgmap v2120: 169 pgs: 169 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 337 KiB/s rd, 169 KiB/s wr, 666 op/s 2021-07-31T02:26:51.890 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 512 2021-07-31T02:26:51.891 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 512 2021-07-31T02:26:51.891 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 512 2021-07-31T02:26:51.892 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 512 2021-07-31T02:26:51.892 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 512 2021-07-31T02:26:51.892 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:51.892 INFO:tasks.workunit.client.0.smithi045.stdout:started write 2021-07-31T02:26:51.893 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:26:51.893 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:26:51.893 INFO:tasks.workunit.client.0.smithi045.stdout:finished write 2021-07-31T02:26:51.894 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:51.894 INFO:tasks.workunit.client.0.smithi045.stdout:started write 2021-07-31T02:26:51.894 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:26:51.895 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:26:51.895 INFO:tasks.workunit.client.0.smithi045.stdout:finished write 2021-07-31T02:26:51.895 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:51.896 INFO:tasks.workunit.client.0.smithi045.stdout:started write 2021-07-31T02:26:51.896 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:26:51.896 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:26:51.896 INFO:tasks.workunit.client.0.smithi045.stdout:finished write 2021-07-31T02:26:51.896 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:51.897 INFO:tasks.workunit.client.0.smithi045.stdout:started write 2021-07-31T02:26:51.897 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:26:51.897 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:26:51.897 INFO:tasks.workunit.client.0.smithi045.stdout:finished write 2021-07-31T02:26:51.897 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:51.898 INFO:tasks.workunit.client.0.smithi045.stdout:started write 2021-07-31T02:26:51.898 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:26:51.898 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:26:51.898 INFO:tasks.workunit.client.0.smithi045.stdout:finished write 2021-07-31T02:26:51.899 INFO:tasks.workunit.client.0.smithi045.stdout:start compare and write 2021-07-31T02:26:51.899 INFO:tasks.workunit.client.0.smithi045.stdout:compare and wrote: 512 2021-07-31T02:26:51.899 INFO:tasks.workunit.client.0.smithi045.stdout:start compare and write 2021-07-31T02:26:51.899 INFO:tasks.workunit.client.0.smithi045.stdout:compare and wrote: 512 2021-07-31T02:26:51.900 INFO:tasks.workunit.client.0.smithi045.stdout:start compare and write 2021-07-31T02:26:51.900 INFO:tasks.workunit.client.0.smithi045.stdout:compare and wrote: 512 2021-07-31T02:26:51.900 INFO:tasks.workunit.client.0.smithi045.stdout:start compare and write 2021-07-31T02:26:51.900 INFO:tasks.workunit.client.0.smithi045.stdout:compare and wrote: 512 2021-07-31T02:26:51.901 INFO:tasks.workunit.client.0.smithi045.stdout:start compare and write 2021-07-31T02:26:51.901 INFO:tasks.workunit.client.0.smithi045.stdout:compare and wrote: 512 2021-07-31T02:26:51.901 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:51.901 INFO:tasks.workunit.client.0.smithi045.stdout:started aio compare and write 2021-07-31T02:26:51.902 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:26:51.902 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:26:51.902 INFO:tasks.workunit.client.0.smithi045.stdout:finished aio compare and write 2021-07-31T02:26:51.903 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:51.903 INFO:tasks.workunit.client.0.smithi045.stdout:started aio compare and write 2021-07-31T02:26:51.903 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:26:51.903 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:26:51.903 INFO:tasks.workunit.client.0.smithi045.stdout:finished aio compare and write 2021-07-31T02:26:51.904 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:51.904 INFO:tasks.workunit.client.0.smithi045.stdout:started aio compare and write 2021-07-31T02:26:51.904 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:26:51.904 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:26:51.905 INFO:tasks.workunit.client.0.smithi045.stdout:finished aio compare and write 2021-07-31T02:26:51.905 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:51.905 INFO:tasks.workunit.client.0.smithi045.stdout:started aio compare and write 2021-07-31T02:26:51.905 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:26:51.905 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:26:51.906 INFO:tasks.workunit.client.0.smithi045.stdout:finished aio compare and write 2021-07-31T02:26:51.906 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:51.906 INFO:tasks.workunit.client.0.smithi045.stdout:started aio compare and write 2021-07-31T02:26:51.906 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:26:51.907 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:26:51.907 INFO:tasks.workunit.client.0.smithi045.stdout:finished aio compare and write 2021-07-31T02:26:51.907 INFO:tasks.workunit.client.0.smithi045.stdout:read: 512 2021-07-31T02:26:51.907 INFO:tasks.workunit.client.0.smithi045.stdout:read: 512 2021-07-31T02:26:51.907 INFO:tasks.workunit.client.0.smithi045.stdout:read: 512 2021-07-31T02:26:51.908 INFO:tasks.workunit.client.0.smithi045.stdout:read: 512 2021-07-31T02:26:51.908 INFO:tasks.workunit.client.0.smithi045.stdout:read: 512 2021-07-31T02:26:51.908 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:51.908 INFO:tasks.workunit.client.0.smithi045.stdout:started read 2021-07-31T02:26:51.909 INFO:tasks.workunit.client.0.smithi045.stdout:read completion cb called! 2021-07-31T02:26:51.909 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 512 2021-07-31T02:26:51.909 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:51.909 INFO:tasks.workunit.client.0.smithi045.stdout:started read 2021-07-31T02:26:51.909 INFO:tasks.workunit.client.0.smithi045.stdout:read completion cb called! 2021-07-31T02:26:51.910 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 512 2021-07-31T02:26:51.910 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:51.910 INFO:tasks.workunit.client.0.smithi045.stdout:started read 2021-07-31T02:26:51.910 INFO:tasks.workunit.client.0.smithi045.stdout:read completion cb called! 2021-07-31T02:26:51.911 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 512 2021-07-31T02:26:51.911 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:51.912 INFO:tasks.workunit.client.0.smithi045.stdout:started read 2021-07-31T02:26:51.912 INFO:tasks.workunit.client.0.smithi045.stdout:read completion cb called! 2021-07-31T02:26:51.912 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 512 2021-07-31T02:26:51.913 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:51.913 INFO:tasks.workunit.client.0.smithi045.stdout:started read 2021-07-31T02:26:51.914 INFO:tasks.workunit.client.0.smithi045.stdout:read completion cb called! 2021-07-31T02:26:51.914 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 512 2021-07-31T02:26:51.915 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 512~512 = 512 2021-07-31T02:26:51.915 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:26:51.915 INFO:tasks.workunit.client.0.smithi045.stdout:aio discard: 1536~512 = 0 2021-07-31T02:26:51.916 INFO:tasks.workunit.client.0.smithi045.stdout:read: 512 2021-07-31T02:26:51.916 INFO:tasks.workunit.client.0.smithi045.stdout:read: 512 2021-07-31T02:26:51.917 INFO:tasks.workunit.client.0.smithi045.stdout:read: 512 2021-07-31T02:26:51.917 INFO:tasks.workunit.client.0.smithi045.stdout:read: 512 2021-07-31T02:26:51.918 INFO:tasks.workunit.client.0.smithi045.stdout:read: 512 2021-07-31T02:26:51.918 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 0 2021-07-31T02:26:51.918 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:26:51.919 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:26:51.919 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 0 2021-07-31T02:26:51.920 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:26:51.920 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:26:51.920 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 16384 2021-07-31T02:26:51.921 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:26:51.921 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:26:51.922 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 16384 2021-07-31T02:26:51.922 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:26:51.923 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:26:51.923 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:26:51.923 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:26:51.924 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 49152 2021-07-31T02:26:51.924 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:26:51.925 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:26:51.925 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 49152 2021-07-31T02:26:51.926 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:26:51.926 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:26:51.926 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 65536 2021-07-31T02:26:51.927 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:26:51.927 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:26:51.928 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 65536 2021-07-31T02:26:51.928 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:26:51.928 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:26:51.929 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:26:51.929 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:26:51.930 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 98304 2021-07-31T02:26:51.930 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:26:51.931 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:26:51.931 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 98304 2021-07-31T02:26:51.931 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:26:51.932 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:26:51.932 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 114688 2021-07-31T02:26:51.933 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:26:51.933 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:26:51.934 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 114688 2021-07-31T02:26:51.934 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:26:51.934 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:26:51.935 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:26:51.935 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:26:51.936 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 147456 2021-07-31T02:26:51.936 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:26:51.937 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:26:51.937 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 147456 2021-07-31T02:26:51.937 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:26:51.938 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:26:51.938 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 163840 2021-07-31T02:26:51.939 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:26:51.939 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:26:51.940 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 163840 2021-07-31T02:26:51.940 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:26:51.940 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:26:51.941 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:26:51.941 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:26:51.942 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 196608 2021-07-31T02:26:51.942 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:26:51.943 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:26:51.943 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 196608 2021-07-31T02:26:51.943 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:26:51.944 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:26:51.944 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 212992 2021-07-31T02:26:51.945 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:26:51.945 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:26:51.946 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 212992 2021-07-31T02:26:51.946 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:26:51.946 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:26:51.947 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:26:51.947 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:26:51.948 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:51.948 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:26:51.948 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:26:51.949 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:26:51.949 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:26:51.950 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:26:51.950 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:26:51.951 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:51.951 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:26:51.951 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:26:51.952 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:26:51.952 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:26:51.953 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:26:51.953 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:26:51.954 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:51.954 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:26:51.954 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:26:51.955 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:26:51.955 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:26:51.956 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:26:51.956 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:26:51.956 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:51.957 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:26:51.957 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:26:51.958 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:26:51.958 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:26:51.959 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:26:51.959 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:26:51.959 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:51.960 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:26:51.960 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:26:51.961 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:51.961 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:26:51.961 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:26:51.962 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:51.962 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:26:51.963 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:26:51.963 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:26:51.964 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:26:51.964 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:26:51.964 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:26:51.965 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:51.965 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:26:51.966 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:26:51.966 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:26:51.966 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:26:51.967 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:26:51.967 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:26:51.968 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:52.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:52 smithi154 conmon[34553]: cluster 2021-07-31T02:26:50.360313+0000 mgr.x (mgr. 2021-07-31T02:26:52.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:52 smithi154 conmon[34553]: 24542) 1699 : cluster [DBG] pgmap v2121: 169 pgs: 169 active+clean; 6.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 518 KiB/s rd, 260 KiB/s wr, 1.04k op/s 2021-07-31T02:26:52.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:52 smithi045 conmon[39718]: cluster 2021-07- 2021-07-31T02:26:52.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:52 smithi045 conmon[39718]: 31T02:26:50.360313+0000 mgr.x (mgr.24542 2021-07-31T02:26:52.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:52 smithi045 conmon[39718]: ) 1699 : cluster [DBG] pgmap v2121: 169 pgs: 169 active+clean; 6.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 518 KiB/s rd, 260 KiB/s wr, 1.04k op/s 2021-07-31T02:26:52.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:52 smithi045 conmon[39718]: 2021-07-31T02:26:52.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:52 smithi045 conmon[33655]: cluster 2021-07-31T02:26:50.360313+0000 mgr.x ( 2021-07-31T02:26:52.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:52 smithi045 conmon[33655]: mgr.24542) 1699 : cluster [DBG] pgmap v2121: 169 pgs: 169 active+clean; 6.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 518 KiB/s rd, 260 KiB/s wr, 1.04k op/s 2021-07-31T02:26:54.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:26:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:26:53.577Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:26:54.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:26:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:26:53.578Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:26:54.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:26:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:26:53.578Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:26:54.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:54 smithi154 conmon[34553]: cluster 2021-07-31T02:26:52.360662+0000 mgr.x (mgr.24542) 1700 : cluster [DBG] pgmap v2122: 169 pgs: 169 active+clean; 6.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 421 KiB/s rd, 211 KiB/s wr, 845 op/s 2021-07-31T02:26:54.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:54 smithi045 conmon[39718]: cluster 2021-07-31T02:26:52 2021-07-31T02:26:54.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:54 smithi045 conmon[39718]: .360662+0000 mgr.x (mgr.24542) 1700 : cluster [DBG] pgmap v2122: 169 pgs: 169 active+clean; 6.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 421 KiB/s rd, 211 KiB/s wr, 845 op/s 2021-07-31T02:26:54.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:54 smithi045 conmon[33655]: cluster 2021-07-31T02:26:52.360662+0000 mgr.x 2021-07-31T02:26:54.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:54 smithi045 conmon[33655]: (mgr.24542) 1700 : cluster [DBG] pgmap v2122: 169 pgs: 169 active+clean; 6.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 421 KiB/s rd, 211 KiB/s wr, 845 op/s 2021-07-31T02:26:56.156 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:26:56.156 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:26:56.156 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:26:56.157 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:26:56.157 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:26:56.157 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:26:56.157 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:56.157 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:26:56.158 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:26:56.158 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:26:56.158 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:26:56.159 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:26:56.159 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:26:56.159 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:56.160 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:26:56.160 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:26:56.160 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:56.160 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:26:56.161 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:26:56.161 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:56.161 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:26:56.162 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:26:56.162 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:26:56.162 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:26:56.162 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:26:56.163 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:26:56.163 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:56.163 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:26:56.163 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:26:56.164 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:26:56.164 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:26:56.164 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:26:56.164 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:26:56.165 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:56.165 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:26:56.165 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:26:56.165 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:26:56.165 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:26:56.166 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:26:56.166 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:26:56.166 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:56.166 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:26:56.167 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:26:56.167 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:26:56.167 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:26:56.167 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:26:56.167 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:26:56.168 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:56.168 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:26:56.168 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:26:56.168 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:56.168 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:26:56.169 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:26:56.169 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:56.169 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:26:56.170 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:26:56.170 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:26:56.170 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:26:56.170 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:26:56.171 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:26:56.171 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:56.171 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:26:56.171 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:26:56.172 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:26:56.172 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:26:56.172 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:26:56.172 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:26:56.172 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:56.173 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:26:56.173 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:26:56.173 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:26:56.173 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:26:56.174 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:26:56.174 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:26:56.174 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:56.174 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:26:56.175 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:26:56.175 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:26:56.175 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:26:56.175 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:26:56.176 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:26:56.176 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:56.176 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:26:56.176 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:26:56.176 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:56.177 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:26:56.177 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:26:56.177 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:56.177 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:26:56.178 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:26:56.178 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:26:56.178 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:26:56.178 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:26:56.179 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:26:56.179 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:56.179 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:26:56.179 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:26:56.180 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:26:56.180 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:26:56.180 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:26:56.180 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:26:56.180 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:56.181 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:26:56.181 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:26:56.181 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:26:56.181 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:26:56.181 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:26:56.182 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:26:56.182 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:56.182 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:26:56.182 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:26:56.183 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:26:56.183 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:26:56.183 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:26:56.183 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:26:56.183 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:56.184 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:26:56.184 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:26:56.184 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:26:56.184 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:26:56.185 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:26:56.185 INFO:tasks.workunit.client.0.smithi045.stdout:read completion cb called! 2021-07-31T02:26:56.185 INFO:tasks.workunit.client.0.smithi045.stdout:read completion cb called! 2021-07-31T02:26:56.185 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 512 2021-07-31T02:26:56.185 INFO:tasks.workunit.client.0.smithi045.stdout:read completion cb called! 2021-07-31T02:26:56.186 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 4 2021-07-31T02:26:56.186 INFO:tasks.workunit.client.0.smithi045.stdout:read: 4 2021-07-31T02:26:56.186 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.TestEncryptionLUKS1 (10238 ms) 2021-07-31T02:26:56.186 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.TestEncryptionLUKS2 2021-07-31T02:26:56.187 INFO:tasks.workunit.client.0.smithi045.stdout:using old format 2021-07-31T02:26:56.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:56 smithi154 conmon[34553]: cluster 2021-07-31T02:26:54.361659+0000 mgr.x (mgr.24542) 1701 : cluster [DBG] pgmap v2123: 169 pgs: 169 active+clean; 11 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 995 KiB/s rd, 921 KiB/s wr, 1.19k op/s 2021-07-31T02:26:56.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:56 smithi045 conmon[39718]: cluster 2021-07-31T02:26 2021-07-31T02:26:56.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:56 smithi045 conmon[39718]: :54.361659+0000 mgr.x (mgr.24542) 1701 : cluster [DBG] pgmap v2123: 169 pgs: 169 active+clean; 11 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 995 KiB/s rd, 921 KiB/s wr, 1.19k op/s 2021-07-31T02:26:56.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:56 smithi045 conmon[33655]: cluster 2021-07-31T02:26:54.361659+0000 mgr.x ( 2021-07-31T02:26:56.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:56 smithi045 conmon[33655]: mgr.24542) 1701 : cluster [DBG] pgmap v2123: 169 pgs: 169 active+clean; 11 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 995 KiB/s rd, 921 KiB/s wr, 1.19k op/s 2021-07-31T02:26:58.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:26:58 smithi154 conmon[34553]: cluster 2021-07-31T02:26:56.362318+0000 mgr.x (mgr.24542) 1702 : cluster [DBG] pgmap v2124: 169 pgs: 169 active+clean; 11 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 995 KiB/s rd, 921 KiB/s wr, 1.19k op/s 2021-07-31T02:26:58.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:58 smithi045 conmon[33655]: cluster 2021-07-31T02:26:56.362318+0000 mgr.x 2021-07-31T02:26:58.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:26:58 smithi045 conmon[33655]: (mgr.24542) 1702 : cluster [DBG] pgmap v2124: 169 pgs: 169 active+clean; 11 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 995 KiB/s rd, 921 KiB/s wr, 1.19k op/s 2021-07-31T02:26:58.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:58 smithi045 conmon[39718]: cluster 2021-07-31T02:26:56.362318 2021-07-31T02:26:58.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:26:58 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 1702 : cluster [DBG] pgmap v2124: 169 pgs: 169 active+clean; 11 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 995 KiB/s rd, 921 KiB/s wr, 1.19k op/s 2021-07-31T02:27:00.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:00 smithi154 conmon[34553]: cluster 2021-07-31T02:26:58.362776+0000 mgr.x (mgr.24542) 1703 : cluster [DBG] pgmap v2125: 169 pgs: 169 active+clean; 11 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 894 KiB/s rd, 827 KiB/s wr, 1.07k op/s 2021-07-31T02:27:00.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:00 smithi045 conmon[33655]: cluster 2021-07-31T02:26:58.362776+0000 mgr.x ( 2021-07-31T02:27:00.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:00 smithi045 conmon[33655]: mgr.24542) 1703 : cluster [DBG] pgmap v2125: 169 pgs: 169 active+clean; 11 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 894 KiB/s rd, 827 KiB/s wr, 1.07k op/s 2021-07-31T02:27:00.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:00 smithi045 conmon[39718]: cluster 2021-07-31T02:27:00.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:00 smithi045 conmon[39718]: 2021-07-31T02: 2021-07-31T02:27:00.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:00 smithi045 conmon[39718]: 26:58.362776+0000 mgr.x (mgr.24542) 1703 : cluster [DBG] pgmap v2125: 169 pgs: 169 active+clean; 11 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 894 KiB/s rd, 827 KiB/s wr, 1.07k op/s 2021-07-31T02:27:02.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:02 smithi154 conmon[34553]: cluster 2021-07-31T02:27:00.363751+0000 mgr.x (mgr.24542) 1704 : cluster [DBG] pgmap v2126: 169 pgs: 169 active+clean; 15 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 875 KiB/s rd, 768 KiB/s wr, 1.02k op/s 2021-07-31T02:27:02.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:02 smithi045 conmon[33655]: cluster 2021-07-31T02:27:00.363751+0000 mgr.x (mgr.24542) 1704 : cluster 2021-07-31T02:27:02.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:02 smithi045 conmon[33655]: [DBG] pgmap v2126: 169 pgs: 169 active+clean; 15 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 875 KiB/s rd, 768 KiB/s wr, 1.02k op/s 2021-07-31T02:27:02.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:02 smithi045 conmon[39718]: cluster 2021-07-31T02:27:00 2021-07-31T02:27:02.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:02 smithi045 conmon[39718]: .363751+0000 mgr.x (mgr.24542) 1704 : cluster [DBG] pgmap v2126: 169 pgs: 169 active+clean; 15 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 875 KiB/s rd, 768 KiB/s wr, 1.02k op/s 2021-07-31T02:27:04.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:27:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:27:03.578Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:27:04.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:27:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:27:03.578Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:27:04.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:27:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:27:03.578Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:27:04.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:04 smithi045 conmon[33655]: cluster 2021-07-31T02: 2021-07-31T02:27:04.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:04 smithi045 conmon[33655]: 27:02.364247+0000 mgr.x (mgr.24542) 1705 : cluster [DBG] pgmap v2127: 169 pgs: 169 active+clean; 15 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 723 KiB/s rd, 692 KiB/s wr, 705 op/s 2021-07-31T02:27:04.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:04 smithi045 conmon[39718]: cluster 2021-07-31T02:27:02.364247 2021-07-31T02:27:04.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:04 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 1705 : cluster [DBG] pgmap v2127: 169 pgs: 169 active+clean; 15 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 723 KiB/s rd, 692 KiB/s wr, 705 op/s 2021-07-31T02:27:04.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:04 smithi154 conmon[34553]: cluster 2021-07-31T02:27:02.364247+0000 mgr.x (mgr.24542) 2021-07-31T02:27:04.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:04 smithi154 conmon[34553]: 1705 : cluster [DBG] pgmap v2127: 169 pgs: 169 active+clean; 15 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 723 KiB/s rd, 692 KiB/s wr, 705 op/s 2021-07-31T02:27:06.110 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 512 2021-07-31T02:27:06.111 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 512 2021-07-31T02:27:06.111 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 512 2021-07-31T02:27:06.111 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 512 2021-07-31T02:27:06.112 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 512 2021-07-31T02:27:06.112 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:06.112 INFO:tasks.workunit.client.0.smithi045.stdout:started write 2021-07-31T02:27:06.112 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:06.113 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:06.113 INFO:tasks.workunit.client.0.smithi045.stdout:finished write 2021-07-31T02:27:06.113 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:06.113 INFO:tasks.workunit.client.0.smithi045.stdout:started write 2021-07-31T02:27:06.113 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:06.114 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:06.114 INFO:tasks.workunit.client.0.smithi045.stdout:finished write 2021-07-31T02:27:06.114 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:06.114 INFO:tasks.workunit.client.0.smithi045.stdout:started write 2021-07-31T02:27:06.114 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:06.115 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:06.115 INFO:tasks.workunit.client.0.smithi045.stdout:finished write 2021-07-31T02:27:06.115 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:06.115 INFO:tasks.workunit.client.0.smithi045.stdout:started write 2021-07-31T02:27:06.116 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:06.116 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:06.116 INFO:tasks.workunit.client.0.smithi045.stdout:finished write 2021-07-31T02:27:06.116 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:06.116 INFO:tasks.workunit.client.0.smithi045.stdout:started write 2021-07-31T02:27:06.117 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:06.117 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:06.117 INFO:tasks.workunit.client.0.smithi045.stdout:finished write 2021-07-31T02:27:06.117 INFO:tasks.workunit.client.0.smithi045.stdout:start compare and write 2021-07-31T02:27:06.117 INFO:tasks.workunit.client.0.smithi045.stdout:compare and wrote: 512 2021-07-31T02:27:06.118 INFO:tasks.workunit.client.0.smithi045.stdout:start compare and write 2021-07-31T02:27:06.118 INFO:tasks.workunit.client.0.smithi045.stdout:compare and wrote: 512 2021-07-31T02:27:06.118 INFO:tasks.workunit.client.0.smithi045.stdout:start compare and write 2021-07-31T02:27:06.118 INFO:tasks.workunit.client.0.smithi045.stdout:compare and wrote: 512 2021-07-31T02:27:06.119 INFO:tasks.workunit.client.0.smithi045.stdout:start compare and write 2021-07-31T02:27:06.119 INFO:tasks.workunit.client.0.smithi045.stdout:compare and wrote: 512 2021-07-31T02:27:06.119 INFO:tasks.workunit.client.0.smithi045.stdout:start compare and write 2021-07-31T02:27:06.119 INFO:tasks.workunit.client.0.smithi045.stdout:compare and wrote: 512 2021-07-31T02:27:06.120 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:06.120 INFO:tasks.workunit.client.0.smithi045.stdout:started aio compare and write 2021-07-31T02:27:06.120 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:06.120 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:06.121 INFO:tasks.workunit.client.0.smithi045.stdout:finished aio compare and write 2021-07-31T02:27:06.121 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:06.121 INFO:tasks.workunit.client.0.smithi045.stdout:started aio compare and write 2021-07-31T02:27:06.121 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:06.122 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:06.122 INFO:tasks.workunit.client.0.smithi045.stdout:finished aio compare and write 2021-07-31T02:27:06.122 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:06.122 INFO:tasks.workunit.client.0.smithi045.stdout:started aio compare and write 2021-07-31T02:27:06.123 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:06.123 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:06.123 INFO:tasks.workunit.client.0.smithi045.stdout:finished aio compare and write 2021-07-31T02:27:06.123 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:06.124 INFO:tasks.workunit.client.0.smithi045.stdout:started aio compare and write 2021-07-31T02:27:06.124 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:06.124 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:06.124 INFO:tasks.workunit.client.0.smithi045.stdout:finished aio compare and write 2021-07-31T02:27:06.125 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:06.125 INFO:tasks.workunit.client.0.smithi045.stdout:started aio compare and write 2021-07-31T02:27:06.125 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:06.125 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:06.126 INFO:tasks.workunit.client.0.smithi045.stdout:finished aio compare and write 2021-07-31T02:27:06.126 INFO:tasks.workunit.client.0.smithi045.stdout:read: 512 2021-07-31T02:27:06.126 INFO:tasks.workunit.client.0.smithi045.stdout:read: 512 2021-07-31T02:27:06.126 INFO:tasks.workunit.client.0.smithi045.stdout:read: 512 2021-07-31T02:27:06.127 INFO:tasks.workunit.client.0.smithi045.stdout:read: 512 2021-07-31T02:27:06.127 INFO:tasks.workunit.client.0.smithi045.stdout:read: 512 2021-07-31T02:27:06.127 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:06.127 INFO:tasks.workunit.client.0.smithi045.stdout:started read 2021-07-31T02:27:06.128 INFO:tasks.workunit.client.0.smithi045.stdout:read completion cb called! 2021-07-31T02:27:06.128 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 512 2021-07-31T02:27:06.128 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:06.128 INFO:tasks.workunit.client.0.smithi045.stdout:started read 2021-07-31T02:27:06.129 INFO:tasks.workunit.client.0.smithi045.stdout:read completion cb called! 2021-07-31T02:27:06.129 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 512 2021-07-31T02:27:06.129 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:06.129 INFO:tasks.workunit.client.0.smithi045.stdout:started read 2021-07-31T02:27:06.129 INFO:tasks.workunit.client.0.smithi045.stdout:read completion cb called! 2021-07-31T02:27:06.130 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 512 2021-07-31T02:27:06.130 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:06.130 INFO:tasks.workunit.client.0.smithi045.stdout:started read 2021-07-31T02:27:06.130 INFO:tasks.workunit.client.0.smithi045.stdout:read completion cb called! 2021-07-31T02:27:06.130 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 512 2021-07-31T02:27:06.131 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:06.131 INFO:tasks.workunit.client.0.smithi045.stdout:started read 2021-07-31T02:27:06.131 INFO:tasks.workunit.client.0.smithi045.stdout:read completion cb called! 2021-07-31T02:27:06.131 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 512 2021-07-31T02:27:06.132 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 512~512 = 512 2021-07-31T02:27:06.132 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:06.132 INFO:tasks.workunit.client.0.smithi045.stdout:aio discard: 1536~512 = 0 2021-07-31T02:27:06.132 INFO:tasks.workunit.client.0.smithi045.stdout:read: 512 2021-07-31T02:27:06.132 INFO:tasks.workunit.client.0.smithi045.stdout:read: 512 2021-07-31T02:27:06.133 INFO:tasks.workunit.client.0.smithi045.stdout:read: 512 2021-07-31T02:27:06.133 INFO:tasks.workunit.client.0.smithi045.stdout:read: 512 2021-07-31T02:27:06.133 INFO:tasks.workunit.client.0.smithi045.stdout:read: 512 2021-07-31T02:27:06.133 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 0 2021-07-31T02:27:06.134 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:06.134 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:06.134 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 0 2021-07-31T02:27:06.134 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:06.135 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:06.135 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 16384 2021-07-31T02:27:06.135 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:06.135 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:06.136 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 16384 2021-07-31T02:27:06.136 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:06.136 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:06.136 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:27:06.137 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:27:06.137 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 49152 2021-07-31T02:27:06.137 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:06.137 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:06.137 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 49152 2021-07-31T02:27:06.138 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:06.138 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:06.138 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 65536 2021-07-31T02:27:06.138 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:06.139 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:06.139 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 65536 2021-07-31T02:27:06.139 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:06.139 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:06.139 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:27:06.140 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:27:06.140 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 98304 2021-07-31T02:27:06.140 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:06.140 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:06.141 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 98304 2021-07-31T02:27:06.141 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:06.141 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:06.141 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 114688 2021-07-31T02:27:06.142 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:06.142 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:06.142 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 114688 2021-07-31T02:27:06.142 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:06.143 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:06.143 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:27:06.143 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:27:06.143 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 147456 2021-07-31T02:27:06.143 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:06.144 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:06.144 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 147456 2021-07-31T02:27:06.144 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:06.145 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:06.145 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 163840 2021-07-31T02:27:06.145 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:06.145 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:06.145 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 163840 2021-07-31T02:27:06.146 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:06.146 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:06.146 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:27:06.146 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:27:06.146 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 196608 2021-07-31T02:27:06.147 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:06.147 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:06.147 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 196608 2021-07-31T02:27:06.147 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:06.148 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:06.148 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 212992 2021-07-31T02:27:06.148 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:06.148 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:06.148 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 212992 2021-07-31T02:27:06.149 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:06.149 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:06.149 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:27:06.149 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:27:06.149 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:06.150 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:06.150 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:06.150 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:06.150 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:27:06.150 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:06.151 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:06.151 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:06.151 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:06.151 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:06.152 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:06.152 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:27:06.152 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:06.152 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:06.152 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:06.153 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:06.153 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:06.153 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:06.154 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:27:06.154 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:06.154 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:06.154 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:06.154 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:06.155 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:06.155 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:06.155 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:27:06.155 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:06.155 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:06.156 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:06.156 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:06.156 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:27:06.156 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:06.156 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:06.157 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:27:06.157 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:06.157 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:06.157 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:06.157 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:06.158 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:27:06.158 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:06.158 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:06.158 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:06.159 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:06.159 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:06.159 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:06.159 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:27:06.159 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:06.160 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:06.160 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:06.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:06 smithi045 conmon[33655]: cluster 2021-07-31T02:27 2021-07-31T02:27:06.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:06 smithi045 conmon[33655]: :04.364972+0000 mgr.x (mgr.24542) 1706 : cluster [DBG] pgmap v2128: 169 pgs: 169 active+clean; 31 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 980 KiB/s rd, 2.0 MiB/s wr, 769 op/s 2021-07-31T02:27:06.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:06 smithi045 conmon[39718]: cluster 2021-07-31T02:27:04.364972+0000 2021-07-31T02:27:06.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:06 smithi045 conmon[39718]: mgr.x (mgr.24542) 1706 : cluster [DBG] pgmap v2128: 169 pgs: 169 active+clean; 31 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 980 KiB/s rd, 2.0 MiB/s wr, 769 op/s 2021-07-31T02:27:06.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:06 smithi154 conmon[34553]: cluster 2021-07-31T02:27:04.364972+0000 mgr.x (mgr.24542 2021-07-31T02:27:06.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:06 smithi154 conmon[34553]: ) 1706 : cluster [DBG] pgmap v2128: 169 pgs: 169 active+clean; 31 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 980 KiB/s rd, 2.0 MiB/s wr, 769 op/s 2021-07-31T02:27:08.079 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:27:08.375 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:27:08.375 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 42m ago 42m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:27:08.375 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 39m ago 42m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:27:08.376 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 39m ago 46m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:27:08.376 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 42m ago 48m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:27:08.376 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 42m ago 48m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:27:08.376 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 39m ago 46m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:27:08.377 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 42m ago 46m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:27:08.377 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 42m ago 43m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:27:08.377 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 39m ago 42m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:27:08.377 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 42m ago 45m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:27:08.378 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 42m ago 45m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:27:08.378 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 42m ago 45m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:27:08.378 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 42m ago 44m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:27:08.378 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 39m ago 44m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:27:08.379 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 39m ago 44m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:27:08.379 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 39m ago 43m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:27:08.379 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 39m ago 43m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:27:08.379 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 39m ago 43m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:27:08.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:08 smithi045 conmon[33655]: cluster 2021-07-31T02:27:06. 2021-07-31T02:27:08.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:08 smithi045 conmon[33655]: 365476+0000 mgr.x (mgr.24542) 1707 : cluster [DBG] pgmap v2129: 169 pgs: 169 active+clean; 31 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 303 KiB/s rd, 1.4 MiB/s wr, 87 op/s 2021-07-31T02:27:08.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:08 smithi045 conmon[39718]: cluster 2021-07-31T02: 2021-07-31T02:27:08.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:08 smithi045 conmon[39718]: 27:06.365476+0000 mgr.x (mgr.24542) 1707 : cluster [DBG] pgmap v2129: 169 pgs: 169 active+clean; 31 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 303 KiB/s rd, 1.4 MiB/s wr, 87 op/s 2021-07-31T02:27:08.751 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:27:08.751 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:27:08.751 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:27:08.751 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:27:08.752 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:27:08.752 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:27:08.752 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:27:08.752 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:27:08.753 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:27:08.753 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:27:08.753 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:27:08.753 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:27:08.753 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:27:08.754 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:27:08.754 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:27:08.754 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:27:08.754 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:27:08.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:08 smithi154 conmon[34553]: cluster 2021-07-31T02:27:06.365476+0000 mgr.x (mgr.24542) 2021-07-31T02:27:08.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:08 smithi154 conmon[34553]: 1707 : cluster [DBG] pgmap v2129: 169 pgs: 169 active+clean; 31 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 303 KiB/s rd, 1.4 MiB/s wr, 87 op/s 2021-07-31T02:27:09.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:09 smithi045 conmon[33655]: audit 2021-07-31T02:27:08.067731+0000 mgr.x 2021-07-31T02:27:09.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:09 smithi045 conmon[33655]: (mgr.24542) 1708 : audit [DBG] from='client.27034 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:27:09.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:09 smithi045 conmon[33655]: audit 2021-07-31T02:27:08.750616+0000 mon.b (mon.2) 272 : audit 2021-07-31T02:27:09.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:09 smithi045 conmon[33655]: [DBG] from='client.? 172.21.15.45:0/2783745832' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:27:09.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:09 smithi045 conmon[39718]: audit 2021-07-31T02:27 2021-07-31T02:27:09.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:09 smithi045 conmon[39718]: :08.067731+0000 mgr.x (mgr.24542) 1708 : audit [DBG] from='client.27034 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:27:09.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:09 smithi045 conmon[39718]: audit 2021 2021-07-31T02:27:09.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:09 smithi045 conmon[39718]: -07-31T02:27:08.750616+0000 2021-07-31T02:27:09.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:09 smithi045 conmon[39718]: mon.b (mon.2) 272 : audit 2021-07-31T02:27:09.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:09 smithi045 conmon[39718]: [DBG] from='client.? 172.21.15.45:0/2783745832' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:27:09.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:09 smithi154 conmon[34553]: audit 2021-07-31T02:27:08.067731+0000 mgr.x (mgr.24542) 1708 : audit [DBG] from='client.27034 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:27:09.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:09 smithi154 conmon[34553]: audit 2021-07-31T02:27:08.750616+0000 mon.b (mon.2) 272 : audit [DBG] from='client.? 172.21.15.45:0/2783745832' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:27:10.378 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:10.379 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:10.379 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:10.379 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:27:10.379 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:10.380 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:10.380 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:10.380 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:10.380 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:10.380 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:10.381 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:27:10.381 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:10.381 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:10.381 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:10.381 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:10.382 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:27:10.382 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:10.382 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:10.382 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:27:10.383 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:10.383 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:10.383 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:10.383 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:10.383 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:27:10.384 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:10.384 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:10.384 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:10.384 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:10.384 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:10.385 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:10.385 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:27:10.385 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:10.385 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:10.385 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:10.386 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:10.386 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:10.386 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:10.386 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:27:10.387 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:10.387 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:10.387 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:10.387 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:10.387 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:10.388 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:10.388 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:27:10.388 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:10.388 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:10.388 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:10.389 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:10.389 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:27:10.389 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:10.389 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:10.390 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:27:10.390 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:10.390 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:10.390 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:10.390 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:10.391 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:27:10.391 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:10.391 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:10.391 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:10.392 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:10.392 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:10.392 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:10.392 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:27:10.392 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:10.393 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:10.393 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:10.393 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:10.393 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:10.393 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:10.394 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:27:10.394 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:10.394 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:10.394 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:10.395 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:10.395 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:10.395 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:10.395 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:27:10.395 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:10.396 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:10.396 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:10.396 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:10.396 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:27:10.396 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:10.397 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:10.397 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:27:10.397 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:10.397 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:10.398 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:10.398 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:10.398 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:27:10.398 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:10.398 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:10.399 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:10.399 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:10.399 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:10.399 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:10.399 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:27:10.400 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:10.400 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:10.400 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:10.400 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:10.400 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:10.401 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:10.401 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:27:10.401 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:10.401 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:10.402 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:10.402 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:10.402 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:10.402 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:10.402 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:27:10.403 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:10.403 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:10.403 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:10.403 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:10.403 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:27:10.404 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:10.404 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:10.404 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:27:10.404 INFO:tasks.workunit.client.0.smithi045.stdout:read completion cb called! 2021-07-31T02:27:10.404 INFO:tasks.workunit.client.0.smithi045.stdout:read completion cb called! 2021-07-31T02:27:10.405 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 512 2021-07-31T02:27:10.405 INFO:tasks.workunit.client.0.smithi045.stdout:read completion cb called! 2021-07-31T02:27:10.405 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 4 2021-07-31T02:27:10.405 INFO:tasks.workunit.client.0.smithi045.stdout:read: 4 2021-07-31T02:27:10.405 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.TestEncryptionLUKS2 (14224 ms) 2021-07-31T02:27:10.406 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.TestIOWithIOHint 2021-07-31T02:27:10.406 INFO:tasks.workunit.client.0.smithi045.stdout:using old format 2021-07-31T02:27:10.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:10 smithi045 conmon[33655]: cluster 2021-07-31T02:27:08.365876+0000 mgr.x (mgr.24542 2021-07-31T02:27:10.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:10 smithi045 conmon[33655]: ) 1709 : cluster [DBG] pgmap v2130: 169 pgs: 169 active+clean; 31 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 303 KiB/s rd, 1.4 MiB/s wr, 87 op/s 2021-07-31T02:27:10.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:10 smithi045 conmon[33655]: audit 2021-07-31T02:27:08.368335+0000 mgr.x (mgr.24542) 1710 : audit [DBG] from='client.17106 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:27:10.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:10 smithi045 conmon[39718]: cluster 2021-07-31T02:27:08.365876+0000 mgr.x (mgr 2021-07-31T02:27:10.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:10 smithi045 conmon[39718]: .24542) 1709 : cluster [DBG] pgmap v2130: 169 pgs: 169 active+clean; 31 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 303 KiB/s rd, 1.4 MiB/s wr, 87 op/s 2021-07-31T02:27:10.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:10 smithi045 conmon[39718]: audit 2021-07-31T02:27:08.368335+0000 mgr.x (mgr.24542 2021-07-31T02:27:10.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:10 smithi045 conmon[39718]: ) 1710 : audit [DBG] from='client.17106 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:27:10.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:10 smithi154 conmon[34553]: cluster 2021-07-31T02:27:08.365876+0000 mgr.x (mgr.24542) 1709 : cluster [DBG] pgmap v2130: 169 pgs: 169 active+clean; 31 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 303 KiB/s rd, 1.4 MiB/s wr, 87 op/s 2021-07-31T02:27:10.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:10 smithi154 conmon[34553]: 2021-07-31T02:27:10.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:10 smithi154 conmon[34553]: audit 2021-07-31T02:27:08.368335+0000 mgr.x (mgr.24542) 1710 : audit [DBG] from='client.17106 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:27:12.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:12 smithi045 conmon[33655]: cluster 2021-07-31T02:27:10.366873+0000 mgr.x (mgr. 2021-07-31T02:27:12.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:12 smithi045 conmon[33655]: 24542) 1711 : cluster [DBG] pgmap v2131: 169 pgs: 169 active+clean; 35 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 3.0 MiB/s rd, 1.7 MiB/s wr, 737 op/s 2021-07-31T02:27:12.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:12 smithi045 conmon[39718]: cluster 2021-07-31T02:27:10.366873 2021-07-31T02:27:12.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:12 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 1711 : cluster [DBG] pgmap v2131: 169 pgs: 169 active+clean; 35 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 3.0 MiB/s rd, 1.7 MiB/s wr, 737 op/s 2021-07-31T02:27:12.708 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 512 2021-07-31T02:27:12.709 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 512 2021-07-31T02:27:12.709 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 512 2021-07-31T02:27:12.709 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 512 2021-07-31T02:27:12.709 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 512 2021-07-31T02:27:12.710 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:12.710 INFO:tasks.workunit.client.0.smithi045.stdout:started write 2021-07-31T02:27:12.710 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:12.710 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:12.711 INFO:tasks.workunit.client.0.smithi045.stdout:finished write 2021-07-31T02:27:12.711 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:12.711 INFO:tasks.workunit.client.0.smithi045.stdout:started write 2021-07-31T02:27:12.711 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:12.712 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:12.712 INFO:tasks.workunit.client.0.smithi045.stdout:finished write 2021-07-31T02:27:12.712 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:12.712 INFO:tasks.workunit.client.0.smithi045.stdout:started write 2021-07-31T02:27:12.712 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:12.713 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:12.713 INFO:tasks.workunit.client.0.smithi045.stdout:finished write 2021-07-31T02:27:12.713 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:12.714 INFO:tasks.workunit.client.0.smithi045.stdout:started write 2021-07-31T02:27:12.714 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:12.715 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:12.715 INFO:tasks.workunit.client.0.smithi045.stdout:finished write 2021-07-31T02:27:12.716 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:12.716 INFO:tasks.workunit.client.0.smithi045.stdout:started write 2021-07-31T02:27:12.716 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:12.717 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:12.717 INFO:tasks.workunit.client.0.smithi045.stdout:finished write 2021-07-31T02:27:12.717 INFO:tasks.workunit.client.0.smithi045.stdout:start compare and write 2021-07-31T02:27:12.717 INFO:tasks.workunit.client.0.smithi045.stdout:compare and wrote: 512 2021-07-31T02:27:12.718 INFO:tasks.workunit.client.0.smithi045.stdout:start compare and write 2021-07-31T02:27:12.718 INFO:tasks.workunit.client.0.smithi045.stdout:compare and wrote: 512 2021-07-31T02:27:12.718 INFO:tasks.workunit.client.0.smithi045.stdout:start compare and write 2021-07-31T02:27:12.718 INFO:tasks.workunit.client.0.smithi045.stdout:compare and wrote: 512 2021-07-31T02:27:12.718 INFO:tasks.workunit.client.0.smithi045.stdout:start compare and write 2021-07-31T02:27:12.719 INFO:tasks.workunit.client.0.smithi045.stdout:compare and wrote: 512 2021-07-31T02:27:12.719 INFO:tasks.workunit.client.0.smithi045.stdout:start compare and write 2021-07-31T02:27:12.719 INFO:tasks.workunit.client.0.smithi045.stdout:compare and wrote: 512 2021-07-31T02:27:12.719 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:12.719 INFO:tasks.workunit.client.0.smithi045.stdout:started aio compare and write 2021-07-31T02:27:12.720 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:12.720 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:12.720 INFO:tasks.workunit.client.0.smithi045.stdout:finished aio compare and write 2021-07-31T02:27:12.721 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:12.721 INFO:tasks.workunit.client.0.smithi045.stdout:started aio compare and write 2021-07-31T02:27:12.721 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:12.721 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:12.722 INFO:tasks.workunit.client.0.smithi045.stdout:finished aio compare and write 2021-07-31T02:27:12.722 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:12.723 INFO:tasks.workunit.client.0.smithi045.stdout:started aio compare and write 2021-07-31T02:27:12.723 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:12.723 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:12.724 INFO:tasks.workunit.client.0.smithi045.stdout:finished aio compare and write 2021-07-31T02:27:12.724 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:12.724 INFO:tasks.workunit.client.0.smithi045.stdout:started aio compare and write 2021-07-31T02:27:12.724 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:12.724 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:12.725 INFO:tasks.workunit.client.0.smithi045.stdout:finished aio compare and write 2021-07-31T02:27:12.725 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:12.725 INFO:tasks.workunit.client.0.smithi045.stdout:started aio compare and write 2021-07-31T02:27:12.725 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:12.726 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:12.726 INFO:tasks.workunit.client.0.smithi045.stdout:finished aio compare and write 2021-07-31T02:27:12.726 INFO:tasks.workunit.client.0.smithi045.stdout:read: 512 2021-07-31T02:27:12.726 INFO:tasks.workunit.client.0.smithi045.stdout:read: 512 2021-07-31T02:27:12.726 INFO:tasks.workunit.client.0.smithi045.stdout:read: 512 2021-07-31T02:27:12.727 INFO:tasks.workunit.client.0.smithi045.stdout:read: 512 2021-07-31T02:27:12.727 INFO:tasks.workunit.client.0.smithi045.stdout:read: 512 2021-07-31T02:27:12.727 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:12.727 INFO:tasks.workunit.client.0.smithi045.stdout:started read 2021-07-31T02:27:12.728 INFO:tasks.workunit.client.0.smithi045.stdout:read completion cb called! 2021-07-31T02:27:12.728 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 512 2021-07-31T02:27:12.728 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:12.729 INFO:tasks.workunit.client.0.smithi045.stdout:started read 2021-07-31T02:27:12.729 INFO:tasks.workunit.client.0.smithi045.stdout:read completion cb called! 2021-07-31T02:27:12.729 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 512 2021-07-31T02:27:12.731 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:12.731 INFO:tasks.workunit.client.0.smithi045.stdout:started read 2021-07-31T02:27:12.731 INFO:tasks.workunit.client.0.smithi045.stdout:read completion cb called! 2021-07-31T02:27:12.732 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 512 2021-07-31T02:27:12.732 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:12.733 INFO:tasks.workunit.client.0.smithi045.stdout:started read 2021-07-31T02:27:12.733 INFO:tasks.workunit.client.0.smithi045.stdout:read completion cb called! 2021-07-31T02:27:12.733 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 512 2021-07-31T02:27:12.733 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:12.733 INFO:tasks.workunit.client.0.smithi045.stdout:started read 2021-07-31T02:27:12.734 INFO:tasks.workunit.client.0.smithi045.stdout:read completion cb called! 2021-07-31T02:27:12.734 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 512 2021-07-31T02:27:12.734 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 512~512 = 512 2021-07-31T02:27:12.734 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:12.735 INFO:tasks.workunit.client.0.smithi045.stdout:aio discard: 1536~512 = 0 2021-07-31T02:27:12.735 INFO:tasks.workunit.client.0.smithi045.stdout:read: 512 2021-07-31T02:27:12.735 INFO:tasks.workunit.client.0.smithi045.stdout:read: 512 2021-07-31T02:27:12.735 INFO:tasks.workunit.client.0.smithi045.stdout:read: 512 2021-07-31T02:27:12.735 INFO:tasks.workunit.client.0.smithi045.stdout:read: 512 2021-07-31T02:27:12.736 INFO:tasks.workunit.client.0.smithi045.stdout:read: 512 2021-07-31T02:27:12.736 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 0 2021-07-31T02:27:12.736 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:12.736 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:12.736 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 0 2021-07-31T02:27:12.737 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:12.737 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:12.737 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 16384 2021-07-31T02:27:12.737 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:12.737 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:12.738 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 16384 2021-07-31T02:27:12.738 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:12.738 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:12.738 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:27:12.739 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:27:12.739 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 49152 2021-07-31T02:27:12.739 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:12.739 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:12.739 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 49152 2021-07-31T02:27:12.740 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:12.740 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:12.740 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 65536 2021-07-31T02:27:12.741 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:12.741 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:12.741 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 65536 2021-07-31T02:27:12.741 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:12.741 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:12.742 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:27:12.742 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:27:12.742 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 98304 2021-07-31T02:27:12.742 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:12.743 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:12.743 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 98304 2021-07-31T02:27:12.743 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:12.743 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:12.744 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 114688 2021-07-31T02:27:12.744 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:12.745 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:12.745 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 114688 2021-07-31T02:27:12.745 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:12.746 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:12.746 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:27:12.747 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:27:12.747 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 147456 2021-07-31T02:27:12.747 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:12.747 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:12.748 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 147456 2021-07-31T02:27:12.748 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:12.748 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:12.748 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 163840 2021-07-31T02:27:12.748 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:12.749 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:12.749 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 163840 2021-07-31T02:27:12.750 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:12.750 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:12.750 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:27:12.751 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:27:12.751 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 196608 2021-07-31T02:27:12.752 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:12.752 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:12.753 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 196608 2021-07-31T02:27:12.753 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:12.754 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:12.754 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 212992 2021-07-31T02:27:12.755 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:12.755 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:12.755 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 212992 2021-07-31T02:27:12.756 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:12.756 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:12.757 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:27:12.757 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:27:12.757 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:12.758 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:12.758 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:12.759 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:12.759 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:27:12.759 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:12.760 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:12.760 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:12.760 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:12.760 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:12.760 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:12.761 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:27:12.761 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:12.761 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:12.761 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:12.762 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:12.762 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:12.762 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:12.762 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:27:12.763 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:12.763 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:12.763 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:12.763 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:12.763 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:12.764 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:12.764 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:27:12.764 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:12.764 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:12.764 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:12.765 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:12.765 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:27:12.765 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:12.765 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:12.765 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:27:12.766 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:12.766 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:12.766 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:12.766 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:12.767 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:27:12.767 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:12.767 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:12.767 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:12.767 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:12.768 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:12.768 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:12.768 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:27:12.768 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:12.768 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:12.769 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:12.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:12 smithi154 conmon[34553]: cluster 2021-07-31T02:27:10.366873+0000 mgr.x (mgr. 2021-07-31T02:27:12.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:12 smithi154 conmon[34553]: 24542) 1711 : cluster [DBG] pgmap v2131: 169 pgs: 169 active+clean; 35 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 3.0 MiB/s rd, 1.7 MiB/s wr, 737 op/s 2021-07-31T02:27:14.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:27:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:27:13.578Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:27:14.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:27:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:27:13.578Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:27:14.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:27:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:27:13.578Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:27:14.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:14 smithi045 conmon[33655]: cluster 2021-07-31T02:27:12. 2021-07-31T02:27:14.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:14 smithi045 conmon[33655]: 367251+0000 mgr.x (mgr.24542) 1712 : cluster [DBG] pgmap v2132: 169 pgs: 169 active+clean; 35 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 3.0 MiB/s rd, 1.7 MiB/s wr, 714 op/s 2021-07-31T02:27:14.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:14 smithi045 conmon[39718]: cluster 2021-07-31T02:27:12.367251+0000 2021-07-31T02:27:14.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:14 smithi045 conmon[39718]: mgr.x (mgr.24542) 1712 : cluster [DBG] pgmap v2132: 169 pgs: 169 active+clean; 35 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 3.0 MiB/s rd, 1.7 MiB/s wr, 714 op/s 2021-07-31T02:27:14.668 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:14.668 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:14.669 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:14.669 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:27:14.669 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:14.669 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:14.670 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:14.670 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:14.670 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:14.670 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:14.670 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:27:14.671 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:14.671 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:14.671 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:14.671 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:14.672 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:27:14.672 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:14.672 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:14.672 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:27:14.672 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:14.673 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:14.673 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:14.673 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:14.673 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:27:14.673 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:14.674 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:14.674 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:14.674 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:14.674 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:14.675 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:14.675 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:27:14.675 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:14.675 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:14.676 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:14.676 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:14.676 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:14.676 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:14.677 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:27:14.677 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:14.677 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:14.677 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:14.678 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:14.678 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:14.678 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:14.678 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:27:14.679 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:14.679 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:14.679 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:14.679 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:14.679 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:27:14.680 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:14.680 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:14.680 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:27:14.680 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:14.681 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:14.681 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:14.681 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:14.681 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:27:14.681 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:14.682 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:14.682 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:14.682 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:14.682 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:14.683 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:14.683 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:27:14.683 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:14.683 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:14.683 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:14.684 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:14.684 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:14.684 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:14.684 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:27:14.684 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:14.685 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:14.685 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:14.685 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:14.685 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:14.685 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:14.686 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:27:14.686 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:14.686 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:14.687 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:14.687 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:14.687 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:27:14.687 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:14.687 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:14.688 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:27:14.688 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:14.688 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:14.688 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:14.689 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:14.689 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:27:14.689 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:14.689 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:14.689 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:14.690 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:14.690 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:14.690 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:14.690 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:27:14.691 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:14.691 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:14.691 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:14.691 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:14.691 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:14.692 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:14.692 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:27:14.692 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:14.692 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:14.692 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:14.693 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:14.693 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:14.693 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:14.693 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:27:14.693 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:14.694 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:14.694 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:14.694 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:14.694 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:27:14.695 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:14.695 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:14.695 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:27:14.695 INFO:tasks.workunit.client.0.smithi045.stdout:read completion cb called! 2021-07-31T02:27:14.695 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 512 2021-07-31T02:27:14.696 INFO:tasks.workunit.client.0.smithi045.stdout:read completion cb called! 2021-07-31T02:27:14.696 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.TestIOWithIOHint (4290 ms) 2021-07-31T02:27:14.696 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.TestDataPoolIO 2021-07-31T02:27:14.696 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:27:14.696 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.TestDataPoolIO (0 ms) 2021-07-31T02:27:14.697 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.TestScatterGatherIO 2021-07-31T02:27:14.697 INFO:tasks.workunit.client.0.smithi045.stdout:using old format 2021-07-31T02:27:14.698 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.TestScatterGatherIO (11 ms) 2021-07-31T02:27:14.698 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.TestEmptyDiscard 2021-07-31T02:27:14.698 INFO:tasks.workunit.client.0.smithi045.stdout:using old format 2021-07-31T02:27:14.698 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:14.699 INFO:tasks.workunit.client.0.smithi045.stdout:aio discard: 0~1048576 = 0 2021-07-31T02:27:14.699 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:14.699 INFO:tasks.workunit.client.0.smithi045.stdout:aio discard: 0~4194304 = 0 2021-07-31T02:27:14.699 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:14.699 INFO:tasks.workunit.client.0.smithi045.stdout:aio discard: 3145728~1048576 = 0 2021-07-31T02:27:14.700 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.TestEmptyDiscard (16 ms) 2021-07-31T02:27:14.700 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.TestFUA 2021-07-31T02:27:14.700 INFO:tasks.workunit.client.0.smithi045.stdout:using old format 2021-07-31T02:27:14.736 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 512 2021-07-31T02:27:14.736 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 512 2021-07-31T02:27:14.736 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 512 2021-07-31T02:27:14.737 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 512 2021-07-31T02:27:14.737 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 512 2021-07-31T02:27:14.737 INFO:tasks.workunit.client.0.smithi045.stdout:read: 512 2021-07-31T02:27:14.737 INFO:tasks.workunit.client.0.smithi045.stdout:read: 512 2021-07-31T02:27:14.738 INFO:tasks.workunit.client.0.smithi045.stdout:read: 512 2021-07-31T02:27:14.738 INFO:tasks.workunit.client.0.smithi045.stdout:read: 512 2021-07-31T02:27:14.738 INFO:tasks.workunit.client.0.smithi045.stdout:read: 512 2021-07-31T02:27:14.738 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:14.738 INFO:tasks.workunit.client.0.smithi045.stdout:started write 2021-07-31T02:27:14.739 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:14.739 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:14.739 INFO:tasks.workunit.client.0.smithi045.stdout:finished write 2021-07-31T02:27:14.739 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:14.739 INFO:tasks.workunit.client.0.smithi045.stdout:started write 2021-07-31T02:27:14.740 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:14.740 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:14.740 INFO:tasks.workunit.client.0.smithi045.stdout:finished write 2021-07-31T02:27:14.740 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:14.741 INFO:tasks.workunit.client.0.smithi045.stdout:started write 2021-07-31T02:27:14.741 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:14.741 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:14.741 INFO:tasks.workunit.client.0.smithi045.stdout:finished write 2021-07-31T02:27:14.741 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:14.742 INFO:tasks.workunit.client.0.smithi045.stdout:started write 2021-07-31T02:27:14.742 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:14.742 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:14.742 INFO:tasks.workunit.client.0.smithi045.stdout:finished write 2021-07-31T02:27:14.742 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:14.743 INFO:tasks.workunit.client.0.smithi045.stdout:started write 2021-07-31T02:27:14.743 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:14.743 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:14.743 INFO:tasks.workunit.client.0.smithi045.stdout:finished write 2021-07-31T02:27:14.744 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:14.744 INFO:tasks.workunit.client.0.smithi045.stdout:started read 2021-07-31T02:27:14.744 INFO:tasks.workunit.client.0.smithi045.stdout:read completion cb called! 2021-07-31T02:27:14.744 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 512 2021-07-31T02:27:14.744 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:14.745 INFO:tasks.workunit.client.0.smithi045.stdout:started read 2021-07-31T02:27:14.745 INFO:tasks.workunit.client.0.smithi045.stdout:read completion cb called! 2021-07-31T02:27:14.745 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 512 2021-07-31T02:27:14.745 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:14.745 INFO:tasks.workunit.client.0.smithi045.stdout:started read 2021-07-31T02:27:14.746 INFO:tasks.workunit.client.0.smithi045.stdout:read completion cb called! 2021-07-31T02:27:14.746 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 512 2021-07-31T02:27:14.746 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:14.746 INFO:tasks.workunit.client.0.smithi045.stdout:started read 2021-07-31T02:27:14.746 INFO:tasks.workunit.client.0.smithi045.stdout:read completion cb called! 2021-07-31T02:27:14.747 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 512 2021-07-31T02:27:14.747 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:14.747 INFO:tasks.workunit.client.0.smithi045.stdout:started read 2021-07-31T02:27:14.747 INFO:tasks.workunit.client.0.smithi045.stdout:read completion cb called! 2021-07-31T02:27:14.748 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 512 2021-07-31T02:27:14.748 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.TestFUA (41 ms) 2021-07-31T02:27:14.748 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.TestIOPP 2021-07-31T02:27:14.748 INFO:tasks.workunit.client.0.smithi045.stdout:using old format 2021-07-31T02:27:14.749 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 512 2021-07-31T02:27:14.749 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 512 2021-07-31T02:27:14.750 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 512 2021-07-31T02:27:14.750 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 512 2021-07-31T02:27:14.750 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 512 2021-07-31T02:27:14.750 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:14.751 INFO:tasks.workunit.client.0.smithi045.stdout:started write 2021-07-31T02:27:14.751 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:14.751 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:14.751 INFO:tasks.workunit.client.0.smithi045.stdout:finished write 2021-07-31T02:27:14.751 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:14.752 INFO:tasks.workunit.client.0.smithi045.stdout:started write 2021-07-31T02:27:14.752 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:14.752 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:14.752 INFO:tasks.workunit.client.0.smithi045.stdout:finished write 2021-07-31T02:27:14.752 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:14.753 INFO:tasks.workunit.client.0.smithi045.stdout:started write 2021-07-31T02:27:14.753 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:14.753 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:14.753 INFO:tasks.workunit.client.0.smithi045.stdout:finished write 2021-07-31T02:27:14.753 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:14.754 INFO:tasks.workunit.client.0.smithi045.stdout:started write 2021-07-31T02:27:14.754 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:14.754 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:14.754 INFO:tasks.workunit.client.0.smithi045.stdout:finished write 2021-07-31T02:27:14.755 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:14.755 INFO:tasks.workunit.client.0.smithi045.stdout:started write 2021-07-31T02:27:14.755 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:14.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:14 smithi154 conmon[34553]: cluster 2021-07-31T02:27:12.367251+0000 mgr.x (mgr.24542) 1712 : cluster 2021-07-31T02:27:14.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:14 smithi154 conmon[34553]: [DBG] pgmap v2132: 169 pgs: 169 active+clean; 35 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 3.0 MiB/s rd, 1.7 MiB/s wr, 714 op/s 2021-07-31T02:27:14.756 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:14.757 INFO:tasks.workunit.client.0.smithi045.stdout:finished write 2021-07-31T02:27:14.757 INFO:tasks.workunit.client.0.smithi045.stdout:start compare and write 2021-07-31T02:27:14.757 INFO:tasks.workunit.client.0.smithi045.stdout:compare and wrote: 512 2021-07-31T02:27:14.757 INFO:tasks.workunit.client.0.smithi045.stdout:start compare and write 2021-07-31T02:27:14.758 INFO:tasks.workunit.client.0.smithi045.stdout:compare and wrote: 512 2021-07-31T02:27:14.758 INFO:tasks.workunit.client.0.smithi045.stdout:start compare and write 2021-07-31T02:27:14.758 INFO:tasks.workunit.client.0.smithi045.stdout:compare and wrote: 512 2021-07-31T02:27:14.758 INFO:tasks.workunit.client.0.smithi045.stdout:start compare and write 2021-07-31T02:27:14.758 INFO:tasks.workunit.client.0.smithi045.stdout:compare and wrote: 512 2021-07-31T02:27:14.759 INFO:tasks.workunit.client.0.smithi045.stdout:start compare and write 2021-07-31T02:27:14.759 INFO:tasks.workunit.client.0.smithi045.stdout:compare and wrote: 512 2021-07-31T02:27:14.759 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:14.759 INFO:tasks.workunit.client.0.smithi045.stdout:started aio compare and write 2021-07-31T02:27:14.759 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:14.760 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:14.760 INFO:tasks.workunit.client.0.smithi045.stdout:finished aio compare and write 2021-07-31T02:27:14.761 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:14.761 INFO:tasks.workunit.client.0.smithi045.stdout:started aio compare and write 2021-07-31T02:27:14.761 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:14.761 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:14.762 INFO:tasks.workunit.client.0.smithi045.stdout:finished aio compare and write 2021-07-31T02:27:14.762 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:14.762 INFO:tasks.workunit.client.0.smithi045.stdout:started aio compare and write 2021-07-31T02:27:14.762 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:14.763 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:14.763 INFO:tasks.workunit.client.0.smithi045.stdout:finished aio compare and write 2021-07-31T02:27:14.763 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:14.764 INFO:tasks.workunit.client.0.smithi045.stdout:started aio compare and write 2021-07-31T02:27:14.764 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:14.765 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:14.765 INFO:tasks.workunit.client.0.smithi045.stdout:finished aio compare and write 2021-07-31T02:27:14.765 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:14.766 INFO:tasks.workunit.client.0.smithi045.stdout:started aio compare and write 2021-07-31T02:27:14.766 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:14.766 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:14.766 INFO:tasks.workunit.client.0.smithi045.stdout:finished aio compare and write 2021-07-31T02:27:14.767 INFO:tasks.workunit.client.0.smithi045.stdout:read: 512 2021-07-31T02:27:14.767 INFO:tasks.workunit.client.0.smithi045.stdout:read: 512 2021-07-31T02:27:14.767 INFO:tasks.workunit.client.0.smithi045.stdout:read: 512 2021-07-31T02:27:14.767 INFO:tasks.workunit.client.0.smithi045.stdout:read: 512 2021-07-31T02:27:14.767 INFO:tasks.workunit.client.0.smithi045.stdout:read: 512 2021-07-31T02:27:14.768 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:14.768 INFO:tasks.workunit.client.0.smithi045.stdout:started read 2021-07-31T02:27:14.768 INFO:tasks.workunit.client.0.smithi045.stdout:read completion cb called! 2021-07-31T02:27:14.768 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 512 2021-07-31T02:27:14.768 INFO:tasks.workunit.client.0.smithi045.stdout:finished read 2021-07-31T02:27:14.769 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:14.769 INFO:tasks.workunit.client.0.smithi045.stdout:started read 2021-07-31T02:27:14.769 INFO:tasks.workunit.client.0.smithi045.stdout:read completion cb called! 2021-07-31T02:27:14.769 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 512 2021-07-31T02:27:14.769 INFO:tasks.workunit.client.0.smithi045.stdout:finished read 2021-07-31T02:27:14.770 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:14.770 INFO:tasks.workunit.client.0.smithi045.stdout:started read 2021-07-31T02:27:14.770 INFO:tasks.workunit.client.0.smithi045.stdout:read completion cb called! 2021-07-31T02:27:14.770 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 512 2021-07-31T02:27:14.771 INFO:tasks.workunit.client.0.smithi045.stdout:finished read 2021-07-31T02:27:14.771 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:14.771 INFO:tasks.workunit.client.0.smithi045.stdout:started read 2021-07-31T02:27:14.771 INFO:tasks.workunit.client.0.smithi045.stdout:read completion cb called! 2021-07-31T02:27:14.771 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 512 2021-07-31T02:27:14.772 INFO:tasks.workunit.client.0.smithi045.stdout:finished read 2021-07-31T02:27:14.772 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:14.772 INFO:tasks.workunit.client.0.smithi045.stdout:started read 2021-07-31T02:27:14.772 INFO:tasks.workunit.client.0.smithi045.stdout:read completion cb called! 2021-07-31T02:27:14.772 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 512 2021-07-31T02:27:14.773 INFO:tasks.workunit.client.0.smithi045.stdout:finished read 2021-07-31T02:27:14.773 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 512~512 2021-07-31T02:27:14.773 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:16.152 INFO:tasks.workunit.client.0.smithi045.stdout:read: 512 2021-07-31T02:27:16.152 INFO:tasks.workunit.client.0.smithi045.stdout:read: 512 2021-07-31T02:27:16.153 INFO:tasks.workunit.client.0.smithi045.stdout:read: 512 2021-07-31T02:27:16.153 INFO:tasks.workunit.client.0.smithi045.stdout:read: 512 2021-07-31T02:27:16.153 INFO:tasks.workunit.client.0.smithi045.stdout:read: 512 2021-07-31T02:27:16.153 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 0 2021-07-31T02:27:16.153 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:16.154 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:16.154 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 0 2021-07-31T02:27:16.154 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:16.154 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:16.155 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 16384 2021-07-31T02:27:16.155 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:16.155 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:16.155 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 16384 2021-07-31T02:27:16.155 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:16.156 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:16.156 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:27:16.156 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:27:16.156 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 49152 2021-07-31T02:27:16.156 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:16.157 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:16.157 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 49152 2021-07-31T02:27:16.157 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:16.157 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:16.158 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 65536 2021-07-31T02:27:16.159 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:16.159 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:16.159 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 65536 2021-07-31T02:27:16.159 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:16.160 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:16.160 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:27:16.160 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:27:16.161 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 98304 2021-07-31T02:27:16.161 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:16.161 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:16.161 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 98304 2021-07-31T02:27:16.161 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:16.162 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:16.162 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 114688 2021-07-31T02:27:16.162 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:16.162 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:16.163 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 114688 2021-07-31T02:27:16.163 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:16.163 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:16.163 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:27:16.164 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:27:16.164 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 147456 2021-07-31T02:27:16.164 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:16.164 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:16.164 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 147456 2021-07-31T02:27:16.165 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:16.165 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:16.165 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 163840 2021-07-31T02:27:16.165 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:16.165 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:16.166 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 163840 2021-07-31T02:27:16.166 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:16.166 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:16.166 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:27:16.167 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:27:16.167 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 196608 2021-07-31T02:27:16.167 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:16.167 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:16.168 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 196608 2021-07-31T02:27:16.168 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:16.168 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:16.168 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 212992 2021-07-31T02:27:16.168 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:16.169 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:16.169 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 212992 2021-07-31T02:27:16.169 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:16.169 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:16.170 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:27:16.170 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:27:16.170 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:16.170 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:16.170 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:16.171 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:16.171 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:27:16.171 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:16.171 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:16.172 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:16.172 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:16.172 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:16.172 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:16.172 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:27:16.173 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:16.173 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:16.173 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:16.173 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:16.173 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:16.174 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:16.175 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:27:16.175 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:16.175 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:16.175 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:16.175 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:16.176 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:16.176 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:16.176 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:27:16.176 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:16.177 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:16.177 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:16.177 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:16.177 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:27:16.177 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:16.178 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:16.178 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:27:16.178 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:16.178 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:16.179 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:16.209 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:16.210 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:27:16.210 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:16.210 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:16.210 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:16.210 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:16.211 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:16.235 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:16.236 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:27:16.236 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:16.236 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:16.236 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:16.237 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:16.237 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:16.290 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:16.290 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:27:16.290 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:16.290 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:16.291 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:16.291 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:16.291 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:16.336 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:16.337 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:27:16.337 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:16.337 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:16.337 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:16.337 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:16.339 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:27:16.339 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:16.339 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:16.339 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:27:16.340 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:16.340 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:16.340 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:16.390 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:16.391 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:27:16.391 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:16.391 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:16.392 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:16.392 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:16.392 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:16.434 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:16.435 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:27:16.435 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:16.435 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:16.435 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:16.436 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:16.436 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:16.503 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:16.503 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:27:16.503 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:16.504 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:16.504 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:16.504 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:16.505 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:16.568 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:16.569 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:27:16.569 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:16.569 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:16.569 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:16.570 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:16.570 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:27:16.570 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:16.570 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:16.570 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:27:16.571 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:16.571 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:16.571 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:16.681 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:16.682 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:27:16.682 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:16.682 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:16.683 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:16.683 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:16.683 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:16.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:16 smithi154 conmon[34553]: cluster 2021-07-31T02:27:14.368018+0000 mgr.x (mgr.24542 2021-07-31T02:27:16.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:16 smithi154 conmon[34553]: ) 1713 : cluster [DBG] pgmap v2133: 169 pgs: 169 active+clean; 35 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 3.2 MiB/s rd, 1.8 MiB/s wr, 1.28k op/s 2021-07-31T02:27:16.758 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:16.758 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:27:16.758 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:16.758 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:16.759 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:16.759 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:16.759 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:16.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:16 smithi045 conmon[33655]: cluster 2021-07-31T02:27:14.368018+0000 2021-07-31T02:27:16.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:16 smithi045 conmon[33655]: mgr.x (mgr.24542) 1713 : cluster [DBG] pgmap v2133: 169 pgs: 169 active+clean; 35 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 3.2 MiB/s rd, 1.8 MiB/s wr, 1.28k op/s 2021-07-31T02:27:16.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:16 smithi045 conmon[39718]: cluster 2021-07-31T02:27:14.368018 2021-07-31T02:27:16.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:16 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 1713 : cluster [DBG] pgmap v2133: 169 pgs: 169 active+clean; 35 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 3.2 MiB/s rd, 1.8 MiB/s wr, 1.28k op/s 2021-07-31T02:27:16.871 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:16.871 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:27:16.872 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:16.872 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:16.872 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:16.872 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:16.872 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:16.958 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:16.959 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:27:16.959 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:16.959 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:16.959 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:16.960 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:16.960 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:27:16.960 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:16.960 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:16.960 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:27:16.961 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:16.961 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:16.961 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:17.081 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:17.081 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:27:17.082 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:17.082 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:17.082 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:17.083 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:17.083 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:17.206 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:17.207 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:27:17.207 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:17.207 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:17.207 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:17.208 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:17.208 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:17.364 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:17.364 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:27:17.364 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:17.364 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:17.365 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:17.365 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:17.365 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:17.477 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:17.478 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:27:17.478 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:17.478 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:17.478 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:17.479 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:17.479 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:27:17.479 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:17.479 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:17.480 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:27:17.480 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.TestIOPP (2741 ms) 2021-07-31T02:27:17.480 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.TestIOPPWithIOHint 2021-07-31T02:27:17.480 INFO:tasks.workunit.client.0.smithi045.stdout:using old format 2021-07-31T02:27:17.492 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 512 2021-07-31T02:27:17.493 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 512 2021-07-31T02:27:17.493 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 512 2021-07-31T02:27:17.493 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 512 2021-07-31T02:27:17.493 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 512 2021-07-31T02:27:17.494 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:17.494 INFO:tasks.workunit.client.0.smithi045.stdout:started write 2021-07-31T02:27:17.494 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:17.495 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:17.495 INFO:tasks.workunit.client.0.smithi045.stdout:finished write 2021-07-31T02:27:17.495 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:17.496 INFO:tasks.workunit.client.0.smithi045.stdout:started write 2021-07-31T02:27:17.496 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:17.496 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:17.496 INFO:tasks.workunit.client.0.smithi045.stdout:finished write 2021-07-31T02:27:17.497 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:17.497 INFO:tasks.workunit.client.0.smithi045.stdout:started write 2021-07-31T02:27:17.497 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:17.497 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:17.497 INFO:tasks.workunit.client.0.smithi045.stdout:finished write 2021-07-31T02:27:17.498 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:17.498 INFO:tasks.workunit.client.0.smithi045.stdout:started write 2021-07-31T02:27:17.498 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:17.499 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:17.500 INFO:tasks.workunit.client.0.smithi045.stdout:finished write 2021-07-31T02:27:17.500 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:17.500 INFO:tasks.workunit.client.0.smithi045.stdout:started write 2021-07-31T02:27:17.500 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:17.500 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:17.501 INFO:tasks.workunit.client.0.smithi045.stdout:finished write 2021-07-31T02:27:17.501 INFO:tasks.workunit.client.0.smithi045.stdout:read: 512 2021-07-31T02:27:17.501 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:17.501 INFO:tasks.workunit.client.0.smithi045.stdout:started read 2021-07-31T02:27:17.502 INFO:tasks.workunit.client.0.smithi045.stdout:read completion cb called! 2021-07-31T02:27:17.502 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 512 2021-07-31T02:27:17.502 INFO:tasks.workunit.client.0.smithi045.stdout:finished read 2021-07-31T02:27:17.502 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:17.502 INFO:tasks.workunit.client.0.smithi045.stdout:started read 2021-07-31T02:27:17.503 INFO:tasks.workunit.client.0.smithi045.stdout:read completion cb called! 2021-07-31T02:27:17.503 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 512 2021-07-31T02:27:17.503 INFO:tasks.workunit.client.0.smithi045.stdout:finished read 2021-07-31T02:27:17.503 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:17.504 INFO:tasks.workunit.client.0.smithi045.stdout:started read 2021-07-31T02:27:17.504 INFO:tasks.workunit.client.0.smithi045.stdout:read completion cb called! 2021-07-31T02:27:17.504 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 512 2021-07-31T02:27:17.504 INFO:tasks.workunit.client.0.smithi045.stdout:finished read 2021-07-31T02:27:17.504 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:17.505 INFO:tasks.workunit.client.0.smithi045.stdout:started read 2021-07-31T02:27:17.505 INFO:tasks.workunit.client.0.smithi045.stdout:read completion cb called! 2021-07-31T02:27:17.505 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 512 2021-07-31T02:27:17.505 INFO:tasks.workunit.client.0.smithi045.stdout:finished read 2021-07-31T02:27:17.506 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:17.506 INFO:tasks.workunit.client.0.smithi045.stdout:started read 2021-07-31T02:27:17.506 INFO:tasks.workunit.client.0.smithi045.stdout:read completion cb called! 2021-07-31T02:27:18.484 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 512 2021-07-31T02:27:18.485 INFO:tasks.workunit.client.0.smithi045.stdout:finished read 2021-07-31T02:27:18.485 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 0 2021-07-31T02:27:18.485 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:18.485 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:18.486 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 0 2021-07-31T02:27:18.486 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:18.486 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:18.486 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 16384 2021-07-31T02:27:18.487 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:18.487 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:18.487 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 16384 2021-07-31T02:27:18.487 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:18.488 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:18.489 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:27:18.489 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:27:18.489 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 49152 2021-07-31T02:27:18.490 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:18.490 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:18.490 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 49152 2021-07-31T02:27:18.490 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:18.491 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:18.491 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 65536 2021-07-31T02:27:18.491 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:18.492 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:18.492 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 65536 2021-07-31T02:27:18.492 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:18.492 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:18.493 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:27:18.493 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:27:18.493 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 98304 2021-07-31T02:27:18.493 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:18.494 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:18.494 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 98304 2021-07-31T02:27:18.494 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:18.495 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:18.495 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 114688 2021-07-31T02:27:18.495 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:18.495 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:18.496 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 114688 2021-07-31T02:27:18.496 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:18.496 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:18.496 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:27:18.496 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:27:18.497 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 147456 2021-07-31T02:27:18.497 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:18.497 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:18.497 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 147456 2021-07-31T02:27:18.498 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:18.498 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:18.498 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 163840 2021-07-31T02:27:18.498 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:18.498 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:18.499 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 163840 2021-07-31T02:27:18.499 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:18.499 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:18.499 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:27:18.499 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:27:18.500 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 196608 2021-07-31T02:27:18.500 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:18.502 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:18.502 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 196608 2021-07-31T02:27:18.502 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:18.502 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:18.503 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 212992 2021-07-31T02:27:18.503 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:18.503 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:18.503 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 212992 2021-07-31T02:27:18.504 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:18.504 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:18.504 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:27:18.504 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:27:18.505 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:18.505 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:18.505 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:18.505 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:18.506 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:27:18.506 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:18.506 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:18.506 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:18.506 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:18.507 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:18.508 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:18.508 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:27:18.508 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:18.508 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:18.509 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:18.509 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:18.509 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:18.509 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:18.509 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:27:18.510 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:18.510 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:18.510 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:18.510 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:18.511 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:18.511 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:18.511 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:27:18.511 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:18.511 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:18.512 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:18.512 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:18.512 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:27:18.512 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:18.512 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:18.513 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:27:18.513 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:18.513 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:18.514 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:18.528 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:18.529 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:27:18.529 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:18.529 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:18.529 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:18.529 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:18.530 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:18.550 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:18.551 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:27:18.551 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:18.551 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:18.551 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:18.552 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:18.552 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:18.581 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:18.582 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:27:18.582 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:18.582 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:18.583 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:18.583 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:18.583 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:18.610 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:18.611 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:27:18.611 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:18.611 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:18.611 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:18.612 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:18.612 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:27:18.612 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:18.612 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:18.613 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:27:18.613 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:18.613 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:18.613 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:18.661 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:18.661 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:27:18.662 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:18.662 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:18.662 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:18.662 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:18.663 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:18.703 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:18.704 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:27:18.704 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:18.704 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:18.705 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:18.705 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:18.706 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:18.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:18 smithi154 conmon[34553]: cluster 2021-07-31T02:27:16.368359+0000 mgr.x (mgr.24542) 1714 : cluster [DBG] pgmap v2134: 169 pgs: 169 active+clean; 35 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 3.0 MiB/s rd, 486 KiB/s wr, 1.22k op/s 2021-07-31T02:27:18.756 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:18.757 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:27:18.757 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:18.757 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:18.757 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:18.758 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:18.758 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:18.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:18 smithi045 conmon[39718]: cluster 2021-07-31T02:27:16 2021-07-31T02:27:18.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:18 smithi045 conmon[39718]: .368359+0000 mgr.x (mgr.24542) 1714 : cluster [DBG] pgmap v2134: 169 pgs: 169 active+clean; 35 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 3.0 MiB/s rd, 486 KiB/s wr, 1.22k op/s 2021-07-31T02:27:18.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:18 smithi045 conmon[33655]: cluster 2021-07-31T02:27:16.368359+0000 2021-07-31T02:27:18.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:18 smithi045 conmon[33655]: mgr.x (mgr.24542) 1714 : cluster [DBG] pgmap v2134: 169 pgs: 169 active+clean; 35 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 3.0 MiB/s rd, 486 KiB/s wr, 1.22k op/s 2021-07-31T02:27:18.805 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:18.806 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:27:18.806 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:18.806 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:18.806 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:18.807 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:18.807 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:27:18.807 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:18.807 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:18.808 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:27:18.808 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:18.808 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:18.808 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:18.875 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:18.875 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:27:18.876 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:18.876 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:18.876 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:18.876 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:18.877 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:18.941 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:18.941 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:27:18.941 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:18.942 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:18.942 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:18.942 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:18.942 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:19.024 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:19.025 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:27:19.025 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:19.025 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:19.026 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:19.026 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:19.026 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:19.096 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:19.096 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:27:19.096 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:19.097 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:19.097 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:19.097 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:19.097 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:27:19.098 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:19.098 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:19.098 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:27:19.098 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:19.099 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:19.099 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:19.196 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:19.197 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:27:19.197 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:19.197 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:19.197 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:19.198 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:19.198 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:19.295 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:19.295 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:27:19.295 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:19.296 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:19.296 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:19.296 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:19.296 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:19.391 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:19.392 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:27:19.392 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:19.393 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:19.393 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:19.393 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:19.393 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:19.493 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:19.494 INFO:tasks.workunit.client.0.smithi045.stdout:finished writesame 2021-07-31T02:27:19.494 INFO:tasks.workunit.client.0.smithi045.stdout:to verify the data 2021-07-31T02:27:19.495 INFO:tasks.workunit.client.0.smithi045.stdout:verified 2021-07-31T02:27:19.495 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:19.495 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:19.495 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:27:19.496 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:19.496 INFO:tasks.workunit.client.0.smithi045.stdout:started writesame 2021-07-31T02:27:19.496 INFO:tasks.workunit.client.0.smithi045.stdout:expected fail, finished writesame 2021-07-31T02:27:19.496 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.TestIOPPWithIOHint (2016 ms) 2021-07-31T02:27:19.497 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.TestIOToSnapshot 2021-07-31T02:27:19.497 INFO:tasks.workunit.client.0.smithi045.stdout:using old format 2021-07-31T02:27:20.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:20 smithi154 conmon[34553]: cluster 2021-07-31T02:27:18.368770+0000 mgr.x (mgr.24542) 1715 : cluster [DBG] pgmap v2135: 169 pgs: 169 active+clean; 35 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 3.0 MiB/s rd, 486 KiB/s wr, 1.22k op/s 2021-07-31T02:27:20.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:20 smithi045 conmon[33655]: cluster 2021-07-31T02:27:18.368770+0000 mgr.x (mgr.24542) 1715 2021-07-31T02:27:20.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:20 smithi045 conmon[33655]: : cluster [DBG] pgmap v2135: 169 pgs: 169 active+clean; 35 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 3.0 MiB/s rd, 486 KiB/s wr, 1.22k op/s 2021-07-31T02:27:20.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:20 smithi045 conmon[39718]: cluster 2021-07-31T02:27:18.368770+0000 mgr.x 2021-07-31T02:27:20.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:20 smithi045 conmon[39718]: (mgr.24542) 1715 : cluster [DBG] pgmap v2135: 169 pgs: 169 active+clean; 35 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 3.0 MiB/s rd, 486 KiB/s wr, 1.22k op/s 2021-07-31T02:27:21.308 INFO:tasks.workunit.client.0.smithi045.stdout:num snaps is: 0 2021-07-31T02:27:21.309 INFO:tasks.workunit.client.0.smithi045.stdout:expected: 0 2021-07-31T02:27:21.309 INFO:tasks.workunit.client.0.smithi045.stdout:num snaps is: 1 2021-07-31T02:27:21.309 INFO:tasks.workunit.client.0.smithi045.stdout:expected: 1 2021-07-31T02:27:21.309 INFO:tasks.workunit.client.0.smithi045.stdout:snap: orig 2021-07-31T02:27:21.310 INFO:tasks.workunit.client.0.smithi045.stdout:found orig with size 2097152 2021-07-31T02:27:21.310 INFO:tasks.workunit.client.0.smithi045.stdout:read: 80 2021-07-31T02:27:21.310 INFO:tasks.workunit.client.0.smithi045.stdout:write test data! 2021-07-31T02:27:21.310 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 80 2021-07-31T02:27:21.310 INFO:tasks.workunit.client.0.smithi045.stdout:num snaps is: 2 2021-07-31T02:27:21.311 INFO:tasks.workunit.client.0.smithi045.stdout:expected: 2 2021-07-31T02:27:21.311 INFO:tasks.workunit.client.0.smithi045.stdout:snap: orig 2021-07-31T02:27:21.311 INFO:tasks.workunit.client.0.smithi045.stdout:snap: written 2021-07-31T02:27:21.311 INFO:tasks.workunit.client.0.smithi045.stdout:found orig with size 2097152 2021-07-31T02:27:21.312 INFO:tasks.workunit.client.0.smithi045.stdout:found written with size 2097152 2021-07-31T02:27:21.312 INFO:tasks.workunit.client.0.smithi045.stdout:read: 80 2021-07-31T02:27:21.312 INFO:tasks.workunit.client.0.smithi045.stdout:read: 80 2021-07-31T02:27:21.312 INFO:tasks.workunit.client.0.smithi045.stdout:read: 80 2021-07-31T02:27:21.312 INFO:tasks.workunit.client.0.smithi045.stdout:write to snapshot returned -30 2021-07-31T02:27:21.313 INFO:tasks.workunit.client.0.smithi045.stdout:Read-only file system 2021-07-31T02:27:21.324 INFO:tasks.workunit.client.0.smithi045.stdout:read: 80 2021-07-31T02:27:21.325 INFO:tasks.workunit.client.0.smithi045.stdout:read: 80 2021-07-31T02:27:21.325 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 80 2021-07-31T02:27:21.325 INFO:tasks.workunit.client.0.smithi045.stdout:opening testimg@orig 2021-07-31T02:27:21.325 INFO:tasks.workunit.client.0.smithi045.stdout:read: 80 2021-07-31T02:27:21.326 INFO:tasks.workunit.client.0.smithi045.stdout:write to snapshot returned -30 2021-07-31T02:27:21.326 INFO:tasks.workunit.client.0.smithi045.stdout:Read-only file system 2021-07-31T02:27:21.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:21 smithi154 conmon[34553]: cluster 2021-07-31T02:27:21.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:21 smithi154 conmon[34553]: 2021-07-31T02:27:20.291171 2021-07-31T02:27:21.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:21 smithi154 conmon[34553]: +0000 mon.a (mon.0) 1991 : cluster [DBG] osdmap e1087: 8 total, 8 up, 8 in 2021-07-31T02:27:21.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:21 smithi045 conmon[33655]: cluster 2021-07-31T02:27:20.291171+0000 mon.a (mon.0) 1991 : 2021-07-31T02:27:21.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:21 smithi045 conmon[33655]: cluster [DBG] osdmap e1087: 8 total, 8 up, 8 in 2021-07-31T02:27:21.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:21 smithi045 conmon[39718]: cluster 2021-07-31T02:27:21.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:21 smithi045 conmon[39718]: 2021-07-31T02:27:21.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:21 smithi045 conmon[39718]: 2021-07-31T02:27:20.291171+0000 2021-07-31T02:27:21.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:21 smithi045 conmon[39718]: mon.a (mon.0) 1991 : cluster [DBG] osdmap e1087: 8 total, 8 up, 8 in 2021-07-31T02:27:22.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:22 smithi154 conmon[34553]: cluster 2021-07-31T02:27:20.369347+0000 mgr.x (mgr.24542) 1716 : cluster [DBG] pgmap v2137: 169 pgs: 169 active+clean; 37 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 1.1 MiB/s rd, 577 KiB/s wr, 2.26k op/s 2021-07-31T02:27:22.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:22 smithi154 conmon[34553]: cluster 2021-07-31T02:27:21.297128+0000 mon.a (mon.0) 1992 : cluster [DBG] osdmap e1088: 8 total, 8 up, 8 in 2021-07-31T02:27:22.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:22 smithi045 conmon[33655]: cluster 2021-07-31T02:27:20.369347+0000 mgr.x (mgr.24542) 1716 : cluster 2021-07-31T02:27:22.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:22 smithi045 conmon[33655]: [DBG] pgmap v2137: 169 pgs: 169 active+clean; 37 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 1.1 MiB/s rd, 577 KiB/s wr, 2.26k op/s 2021-07-31T02:27:22.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:22 smithi045 conmon[33655]: cluster 2021-07-31T02:27:21.297128+0000 mon.a (mon.0) 1992 : cluster [DBG] osdmap e1088: 8 total, 8 up, 8 in 2021-07-31T02:27:22.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:22 smithi045 conmon[39718]: cluster 2021-07-31T02:27:20.369347+0000 mgr.x (mgr.24542) 1716 : cluster [DBG] pgmap v2137: 169 pgs: 169 active+clean; 37 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 1.1 MiB/s rd, 577 KiB/s wr, 2.26k op/s 2021-07-31T02:27:22.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:22 smithi045 conmon[39718]: cluster 2021-07-31T02:27:21.297128+0000 mon.a (mon.0) 1992 : cluster [DBG] osdmap e1088: 8 total, 8 up, 8 in 2021-07-31T02:27:23.317 INFO:tasks.workunit.client.0.smithi045.stdout:num snaps is: 2 2021-07-31T02:27:23.318 INFO:tasks.workunit.client.0.smithi045.stdout:expected: 2 2021-07-31T02:27:23.318 INFO:tasks.workunit.client.0.smithi045.stdout:snap: orig 2021-07-31T02:27:23.318 INFO:tasks.workunit.client.0.smithi045.stdout:snap: written 2021-07-31T02:27:23.318 INFO:tasks.workunit.client.0.smithi045.stdout:found orig with size 2097152 2021-07-31T02:27:23.319 INFO:tasks.workunit.client.0.smithi045.stdout:found written with size 2097152 2021-07-31T02:27:23.319 INFO:tasks.workunit.client.0.smithi045.stdout:num snaps is: 1 2021-07-31T02:27:23.319 INFO:tasks.workunit.client.0.smithi045.stdout:expected: 1 2021-07-31T02:27:23.319 INFO:tasks.workunit.client.0.smithi045.stdout:snap: orig 2021-07-31T02:27:23.320 INFO:tasks.workunit.client.0.smithi045.stdout:found orig with size 2097152 2021-07-31T02:27:23.320 INFO:tasks.workunit.client.0.smithi045.stdout:num snaps is: 0 2021-07-31T02:27:23.320 INFO:tasks.workunit.client.0.smithi045.stdout:expected: 0 2021-07-31T02:27:23.320 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.TestIOToSnapshot (3824 ms) 2021-07-31T02:27:23.320 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.TestSnapshotDeletedIo 2021-07-31T02:27:23.321 INFO:tasks.workunit.client.0.smithi045.stdout:using old format 2021-07-31T02:27:23.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:23 smithi045 conmon[33655]: cluster 2021-07-31T02:27:22.302181+0000 mon.a (mon.0) 1993 : cluster 2021-07-31T02:27:23.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:23 smithi045 conmon[33655]: [DBG] osdmap e1089: 8 total, 8 up, 8 in 2021-07-31T02:27:23.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:23 smithi045 conmon[39718]: cluster 2021-07-31T02:27:23.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:23 smithi045 conmon[39718]: 2021-07-31T02: 2021-07-31T02:27:23.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:23 smithi045 conmon[39718]: 27:22.302181+0000 mon.a (mon.0) 1993 : cluster [DBG] osdmap e1089: 8 total, 8 up, 8 in 2021-07-31T02:27:23.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:23 smithi154 conmon[34553]: cluster 2021-07-31T02:27:23.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:23 smithi154 conmon[34553]: 2021-07-31T02:27:22.302181+0000 mon.a (mon.0 2021-07-31T02:27:23.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:23 smithi154 conmon[34553]: ) 1993 : cluster [DBG] osdmap e1089: 8 total, 8 up, 8 in 2021-07-31T02:27:24.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:27:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:27:23.578Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:27:24.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:27:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:27:23.578Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:27:24.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:27:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:27:23.578Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:27:24.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:24 smithi154 conmon[34553]: cluster 2021-07-31T02:27:22.369873+0000 mgr.x (mgr.24542) 1717 : cluster [DBG] pgmap v2140: 169 pgs: 169 active+clean; 37 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 1.3 MiB/s rd, 645 KiB/s wr, 2.64k op/s 2021-07-31T02:27:24.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:24 smithi154 conmon[34553]: 2021-07-31T02:27:24.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:24 smithi154 conmon[34553]: cluster 2021-07-31T02:27:23.308704+0000 mon.a (mon.0) 1994 : cluster [DBG] osdmap e1090: 8 total, 8 up, 8 in 2021-07-31T02:27:24.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:24 smithi045 conmon[39718]: cluster 2021-07-31T02:27:22.369873+0000 mgr.x (mgr.24542) 1717 2021-07-31T02:27:24.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:24 smithi045 conmon[39718]: : cluster [DBG] pgmap v2140: 169 pgs: 169 active+clean; 37 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 1.3 MiB/s rd, 645 KiB/s wr, 2.64k op/s 2021-07-31T02:27:24.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:24 smithi045 conmon[39718]: cluster 2021-07-31T02:27:23.308704+0000 mon.a (mon.0) 1994 : cluster [DBG] osdmap e1090: 8 total, 8 up, 8 in 2021-07-31T02:27:24.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:24 smithi045 conmon[33655]: cluster 2021-07-31T02:27:22.369873+0000 mgr.x (mgr.24542) 1717 : 2021-07-31T02:27:24.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:24 smithi045 conmon[33655]: cluster [DBG] pgmap v2140: 169 pgs: 169 active+clean; 37 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 1.3 MiB/s rd, 645 KiB/s wr, 2.64k op/s 2021-07-31T02:27:24.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:24 smithi045 conmon[33655]: cluster 2021-07-31T02:27:23.308704+0000 mon.a (mon.0) 1994 : cluster [DBG] osdmap e1090: 8 total, 8 up, 8 in 2021-07-31T02:27:25.339 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.TestSnapshotDeletedIo (2021 ms) 2021-07-31T02:27:25.339 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.TestClone 2021-07-31T02:27:25.339 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:27:25.340 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.TestClone (0 ms) 2021-07-31T02:27:25.340 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.TestClone2 2021-07-31T02:27:25.340 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:27:25.340 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.TestClone2 (0 ms) 2021-07-31T02:27:25.341 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.ListChildren 2021-07-31T02:27:25.341 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:27:25.341 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.ListChildren (0 ms) 2021-07-31T02:27:25.341 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.ListChildrenTiered 2021-07-31T02:27:25.342 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:27:25.342 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.ListChildrenTiered (0 ms) 2021-07-31T02:27:25.342 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.LockingPP 2021-07-31T02:27:25.342 INFO:tasks.workunit.client.0.smithi045.stdout:using old format 2021-07-31T02:27:25.385 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.LockingPP (47 ms) 2021-07-31T02:27:25.393 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.FlushAio 2021-07-31T02:27:25.393 INFO:tasks.workunit.client.0.smithi045.stdout:using old format 2021-07-31T02:27:25.588 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.FlushAio (202 ms) 2021-07-31T02:27:25.588 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.FlushAioPP 2021-07-31T02:27:25.588 INFO:tasks.workunit.client.0.smithi045.stdout:using old format 2021-07-31T02:27:25.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:25 smithi154 conmon[34553]: cluster 2021-07-31T02:27:24.310345+0000 mon.a (mon 2021-07-31T02:27:25.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:25 smithi154 conmon[34553]: .0) 1995 : cluster [DBG] osdmap e1091: 8 total, 8 up, 8 in 2021-07-31T02:27:25.766 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.FlushAioPP (178 ms) 2021-07-31T02:27:25.766 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.ZeroLengthWrite 2021-07-31T02:27:25.766 INFO:tasks.workunit.client.0.smithi045.stdout:using old format 2021-07-31T02:27:25.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:25 smithi045 conmon[33655]: cluster 2021-07-31T02:27:24.310345+0000 mon.a (mon.0) 1995 : cluster 2021-07-31T02:27:25.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:25 smithi045 conmon[33655]: [DBG] osdmap e1091: 8 total, 8 up, 8 in 2021-07-31T02:27:25.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:25 smithi045 conmon[39718]: cluster 2021-07-31T02:27:24.310345+0000 mon.a (mon.0) 1995 : cluster [DBG] 2021-07-31T02:27:25.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:25 smithi045 conmon[39718]: osdmap e1091: 8 total, 8 up, 8 in 2021-07-31T02:27:25.777 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.ZeroLengthWrite (11 ms) 2021-07-31T02:27:25.777 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.ZeroLengthDiscard 2021-07-31T02:27:25.777 INFO:tasks.workunit.client.0.smithi045.stdout:using old format 2021-07-31T02:27:25.787 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.ZeroLengthDiscard (11 ms) 2021-07-31T02:27:25.787 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.ZeroLengthRead 2021-07-31T02:27:25.787 INFO:tasks.workunit.client.0.smithi045.stdout:using old format 2021-07-31T02:27:25.800 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.ZeroLengthRead (13 ms) 2021-07-31T02:27:25.801 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.LargeCacheRead 2021-07-31T02:27:25.801 INFO:tasks.workunit.client.0.smithi045.stdout:using old format 2021-07-31T02:27:25.827 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.LargeCacheRead (26 ms) 2021-07-31T02:27:25.828 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.TestPendingAio 2021-07-31T02:27:25.828 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:27:25.829 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.TestPendingAio (0 ms) 2021-07-31T02:27:25.829 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.CompareAndWriteCopyup 2021-07-31T02:27:25.829 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:27:25.829 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.CompareAndWriteCopyup (0 ms) 2021-07-31T02:27:25.829 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.CompareAndWriteCopyupMismatch 2021-07-31T02:27:25.830 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:27:25.830 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.CompareAndWriteCopyupMismatch (0 ms) 2021-07-31T02:27:25.830 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.Flatten 2021-07-31T02:27:25.830 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:27:25.831 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.Flatten (0 ms) 2021-07-31T02:27:25.831 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.Sparsify 2021-07-31T02:27:25.831 INFO:tasks.workunit.client.0.smithi045.stdout:using old format 2021-07-31T02:27:25.845 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 32768 2021-07-31T02:27:25.845 INFO:tasks.workunit.client.0.smithi045.stdout:read: 32768 2021-07-31T02:27:25.845 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.Sparsify (18 ms) 2021-07-31T02:27:25.845 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.SparsifyPP 2021-07-31T02:27:25.846 INFO:tasks.workunit.client.0.smithi045.stdout:using old format 2021-07-31T02:27:25.863 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.SparsifyPP (18 ms) 2021-07-31T02:27:25.863 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.SnapshotLimit 2021-07-31T02:27:25.863 INFO:tasks.workunit.client.0.smithi045.stdout:using old format 2021-07-31T02:27:26.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:26 smithi154 conmon[34553]: cluster 2021-07-31T02:27:24.370410+0000 mgr.x (mgr.24542) 2021-07-31T02:27:26.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:26 smithi154 conmon[34553]: 1718 : cluster [DBG] pgmap v2143: 169 pgs: 169 active+clean; 37 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 313 KiB/s rd, 143 KiB/s wr, 673 op/s 2021-07-31T02:27:26.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:26 smithi154 conmon[34553]: cluster 2021-07-31T02:27:25.323681+0000 mon.a (mon.0) 1996 : cluster [DBG] osdmap e1092: 8 total, 8 up, 8 in 2021-07-31T02:27:26.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:26 smithi045 conmon[39718]: cluster 2021-07-31T02:27:26.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:26 smithi045 conmon[39718]: 2021-07 2021-07-31T02:27:26.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:26 smithi045 conmon[39718]: -31T02:27:24.370410+0000 mgr.x (mgr.24542) 1718 : cluster [DBG] pgmap v2143: 169 pgs: 169 active+clean; 37 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 313 KiB/s rd, 143 KiB/s wr, 673 op/s 2021-07-31T02:27:26.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:26 smithi045 conmon[39718]: cluster 2021-07-31T02:27:25.323681+0000 mon.a (mon.0) 1996 : cluster [DBG] osdmap e1092: 8 total, 8 up, 8 in 2021-07-31T02:27:26.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:26 smithi045 conmon[33655]: cluster 2021-07-31T02:27:24.370410+0000 mgr.x (mgr.24542) 1718 : cluster [DBG] pgmap v2143: 169 pgs: 169 active+clean; 37 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 313 KiB/s rd, 143 KiB/s wr, 673 op/s 2021-07-31T02:27:26.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:26 smithi045 conmon[33655]: cluster 2021-07-31T02:27:26.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:26 smithi045 conmon[33655]: 2021-07-31T02:27:25.323681+0000 mon.a (mon.0) 1996 : cluster [DBG] osdmap e1092: 8 total, 8 up, 8 in 2021-07-31T02:27:27.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:27 smithi154 conmon[34553]: cluster 2021-07-31T02:27:26.324453+0000 mon.a (mon.0) 1997 : cluster [DBG] 2021-07-31T02:27:27.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:27 smithi154 conmon[34553]: osdmap e1093: 8 total, 8 up, 8 in 2021-07-31T02:27:27.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:27 smithi154 conmon[34553]: cluster 2021-07-31T02:27:26.371077+0000 mgr.x (mgr.24542) 1719 : cluster [DBG] pgmap v2146: 169 pgs: 169 active+clean; 37 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 313 KiB/s rd, 143 KiB/s wr, 673 op/s 2021-07-31T02:27:27.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:27 smithi154 conmon[34553]: cluster 2021-07-31T02:27:27.236039+0000 2021-07-31T02:27:27.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:27 smithi154 conmon[34553]: mon.a (mon.0) 1998 : cluster [DBG] osdmap e1094: 8 total, 8 up, 8 in 2021-07-31T02:27:27.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:27 smithi045 conmon[39718]: cluster 2021-07-31T02:27:26.324453+0000 2021-07-31T02:27:27.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:27 smithi045 conmon[39718]: mon.a (mon.0) 1997 : cluster [DBG] osdmap e1093: 8 total, 8 up, 8 in 2021-07-31T02:27:27.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:27 smithi045 conmon[39718]: cluster 2021-07-31 2021-07-31T02:27:27.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:27 smithi045 conmon[39718]: T02:27:26.371077+0000 mgr.x (mgr.24542) 1719 : 2021-07-31T02:27:27.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:27 smithi045 conmon[39718]: cluster [DBG] pgmap v2146: 169 pgs: 169 active+clean; 37 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 313 KiB/s rd, 143 KiB/s wr, 673 op/s 2021-07-31T02:27:27.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:27 smithi045 conmon[39718]: cluster 2021-07-31T02:27 2021-07-31T02:27:27.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:27 smithi045 conmon[39718]: :27.236039+0000 mon.a (mon.0 2021-07-31T02:27:27.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:27 smithi045 conmon[39718]: ) 1998 : cluster [DBG] osdmap e1094: 8 total, 8 up, 8 in 2021-07-31T02:27:27.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:27 smithi045 conmon[33655]: cluster 2021-07-31T02:27:26.324453+0000 mon.a (mon.0) 1997 : 2021-07-31T02:27:27.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:27 smithi045 conmon[33655]: cluster [DBG] osdmap e1093: 8 total, 8 up, 8 in 2021-07-31T02:27:27.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:27 smithi045 conmon[33655]: cluster 2021-07-31T02:27:26.371077+0000 mgr.x (mgr.24542) 1719 : cluster [DBG] pgmap v2146: 169 pgs: 169 active+clean; 37 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 313 KiB/s rd, 143 KiB/s wr, 673 op/s 2021-07-31T02:27:27.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:27 smithi045 conmon[33655]: cluster 2021-07-31 2021-07-31T02:27:27.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:27 smithi045 conmon[33655]: T02:27:27.236039+0000 mon.a (mon.0) 1998 : cluster [DBG] osdmap e1094: 8 total, 8 up, 8 in 2021-07-31T02:27:28.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:28 smithi154 conmon[34553]: audit 2021-07-31T02:27:28.070030+0000 mon.b (mon.2) 273 : audit [INF] 2021-07-31T02:27:28.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:28 smithi154 conmon[34553]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:27:28.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:28 smithi154 conmon[34553]: audit 2021-07-31T02:27:28.070746+0000 mon.a (mon.0) 1999 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:27:28.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:28 smithi154 conmon[34553]: audit 2021-07-31T02:27: 2021-07-31T02:27:28.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:28 smithi154 conmon[34553]: 28.095487+0000 mon.b (mon.2) 274 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:27:28.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:28 smithi154 conmon[34553]: audit 2021-07-31T02:27:28.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:28 smithi154 conmon[34553]: 2021-07-31T02:27:28.096125+0000 mon.a (mon.0) 2000 : audit [INF] 2021-07-31T02:27:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:28 smithi154 conmon[34553]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:27:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:28 smithi154 conmon[34553]: cluster 2021-07-31T02 2021-07-31T02:27:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:28 smithi154 conmon[34553]: :27:28.243512+0000 mon.a (mon.0) 2001 2021-07-31T02:27:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:28 smithi154 conmon[34553]: : cluster [DBG] osdmap e1095: 8 total, 8 up, 8 in 2021-07-31T02:27:28.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:28 smithi045 conmon[33655]: audit 2021-07-31T02:27:28.070030+0000 mon.b (mon.2) 273 : 2021-07-31T02:27:28.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:28 smithi045 conmon[33655]: audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:27:28.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:28 smithi045 conmon[33655]: audit 2021-07-31T02:27:28.070746+0000 mon.a (mon.0) 1999 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:27:28.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:28 smithi045 conmon[33655]: audit 2021-07-31T02:27:28.095487+0000 2021-07-31T02:27:28.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:28 smithi045 conmon[33655]: mon.b (mon.2) 274 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:27:28.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:28 smithi045 conmon[33655]: audit 2021-07-31T02 2021-07-31T02:27:28.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:28 smithi045 conmon[33655]: :27:28.096125+0000 mon.a (mon.0 2021-07-31T02:27:28.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:28 smithi045 conmon[33655]: ) 2000 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:27:28.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:28 smithi045 conmon[33655]: cluster 2021- 2021-07-31T02:27:28.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:28 smithi045 conmon[33655]: 07-31T02:27:28.243512+0000 2021-07-31T02:27:28.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:28 smithi045 conmon[33655]: mon.a (mon.0) 2001 : 2021-07-31T02:27:28.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:28 smithi045 conmon[33655]: cluster [DBG] osdmap e1095: 8 total, 8 up, 8 in 2021-07-31T02:27:28.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:28 smithi045 conmon[39718]: audit 2021-07-31T02:27:28.070030+0000 mon.b 2021-07-31T02:27:28.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:28 smithi045 conmon[39718]: (mon.2) 273 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:27:28.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:28 smithi045 conmon[39718]: audit 2021-07-31T02:27:28.070746+0000 mon.a ( 2021-07-31T02:27:28.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:28 smithi045 conmon[39718]: mon.0) 1999 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:27:28.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:28 smithi045 conmon[39718]: audit 2021-07-31T02:27:28.095487+0000 2021-07-31T02:27:28.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:28 smithi045 conmon[39718]: mon.b (mon.2) 274 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:27:28.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:28 smithi045 conmon[39718]: audit 2021-07 2021-07-31T02:27:28.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:28 smithi045 conmon[39718]: -31T02:27:28.096125+0000 mon.a (mon.0) 2000 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:27:28.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:28 smithi045 conmon[39718]: cluster 2021-07-31T02:27:28.243512+0000 2021-07-31T02:27:28.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:28 smithi045 conmon[39718]: mon.a (mon.0) 2001 : cluster [DBG] osdmap e1095: 8 total, 8 up, 8 in 2021-07-31T02:27:30.259 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.SnapshotLimit (4396 ms) 2021-07-31T02:27:30.259 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.SnapshotLimitPP 2021-07-31T02:27:30.259 INFO:tasks.workunit.client.0.smithi045.stdout:using old format 2021-07-31T02:27:30.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:30 smithi154 conmon[34553]: cluster 2021-07-31T02:27:28.371708+0000 mgr.x (mgr.24542) 1720 : cluster [DBG] pgmap v2149: 169 pgs: 169 active+clean; 37 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:27:30.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:30 smithi154 conmon[34553]: cluster 2021-07-31T02: 2021-07-31T02:27:30.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:30 smithi154 conmon[34553]: 27:29.245447+0000 mon.a (mon.0) 2002 : cluster [DBG] osdmap e1096: 8 total, 8 up, 8 in 2021-07-31T02:27:30.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:30 smithi045 conmon[39718]: cluster 2021-07-31T02:27:28.371708+0000 mgr.x (mgr 2021-07-31T02:27:30.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:30 smithi045 conmon[39718]: .24542) 1720 : cluster [DBG] pgmap v2149: 169 pgs: 169 active+clean; 37 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:27:30.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:30 smithi045 conmon[39718]: cluster 2021-07-31T02:27:29. 2021-07-31T02:27:30.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:30 smithi045 conmon[39718]: 245447+0000 mon.a (mon.0) 2002 : cluster [DBG] osdmap e1096: 8 total, 8 up, 8 in 2021-07-31T02:27:30.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:30 smithi045 conmon[39718]: 2021-07-31T02:27:30.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:30 smithi045 conmon[33655]: cluster 2021-07-31T02:27:28.371708+0000 mgr.x (mgr.24542) 1720 : cluster [DBG] pgmap v2149: 169 pgs: 169 active+clean; 37 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:27:30.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:30 smithi045 conmon[33655]: cluster 2021-07-31T02:27:29.245447+0000 mon.a (mon.0) 2002 : cluster [DBG] osdmap e1096: 8 total, 8 up, 8 in 2021-07-31T02:27:31.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:31 smithi045 conmon[33655]: cluster 2021-07-31T02:27:30.251029+0000 mon.a ( 2021-07-31T02:27:31.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:31 smithi045 conmon[33655]: mon.0) 2003 : cluster [DBG] osdmap e1097: 8 total, 8 up, 8 in 2021-07-31T02:27:31.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:31 smithi045 conmon[39718]: cluster 2021-07-31T02:27:30.251029 2021-07-31T02:27:31.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:31 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2003 : cluster [DBG] osdmap e1097: 8 total, 8 up, 8 in 2021-07-31T02:27:31.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:31 smithi154 conmon[34553]: cluster 2021-07-31T02:27:30.251029+0000 mon.a (mon.0) 2003 : cluster 2021-07-31T02:27:31.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:31 smithi154 conmon[34553]: [DBG] osdmap e1097: 8 total, 8 up, 8 in 2021-07-31T02:27:32.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:32 smithi045 conmon[33655]: cluster 2021-07-31T02:27:30.372145+0000 2021-07-31T02:27:32.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:32 smithi045 conmon[33655]: mgr.x (mgr.24542) 1721 : cluster [DBG] pgmap v2152: 169 pgs: 169 active+clean; 41 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 552 KiB/s rd, 666 KiB/s wr, 175 op/s 2021-07-31T02:27:32.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:32 smithi045 conmon[33655]: cluster 2021-07-31T02:27:31.258763+0000 mon.a (mon.0) 2004 : cluster [DBG] osdmap e1098: 8 total, 8 up, 8 in 2021-07-31T02:27:32.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:32 smithi045 conmon[33655]: audit 2021-07-31T02:27:31.989433+0000 mon.b (mon.2) 275 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:27:32.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:32 smithi045 conmon[33655]: 2021-07-31T02:27:32.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:32 smithi045 conmon[33655]: audit 2021-07-31T 2021-07-31T02:27:32.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:32 smithi045 conmon[33655]: 02:27:31.990949+0000 mon.b (mon.2) 276 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:27:32.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:32 smithi045 conmon[33655]: audit 2021-07-31T02:27:32.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:32 smithi045 conmon[33655]: 2021-07-31T02:27 2021-07-31T02:27:32.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:32 smithi045 conmon[33655]: :31.992046+0000 mon.b (mon.2) 2021-07-31T02:27:32.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:32 smithi045 conmon[33655]: 277 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:27:32.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:32 smithi045 conmon[33655]: cluster 2021- 2021-07-31T02:27:32.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:32 smithi045 conmon[33655]: 07-31T02:27:32.238722 2021-07-31T02:27:32.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:32 smithi045 conmon[33655]: +0000 mon.a (mon.0) 2021-07-31T02:27:32.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:32 smithi045 conmon[33655]: 2005 : cluster [DBG] osdmap e1099: 8 total, 8 up, 8 in 2021-07-31T02:27:32.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:32 smithi045 conmon[39718]: cluster 2021-07-31T02:27:30.372145+0000 2021-07-31T02:27:32.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:32 smithi045 conmon[39718]: mgr.x (mgr.24542) 1721 : cluster [DBG] pgmap v2152: 169 pgs: 169 active+clean; 41 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 552 KiB/s rd, 666 KiB/s wr, 175 op/s 2021-07-31T02:27:32.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:32 smithi045 conmon[39718]: cluster 2021-07-31T02:27 2021-07-31T02:27:32.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:32 smithi045 conmon[39718]: :31.258763+0000 mon.a (mon.0) 2004 : cluster [DBG] osdmap e1098: 8 total, 8 up, 8 in 2021-07-31T02:27:32.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:32 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T02:27:32.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:32 smithi045 conmon[39718]: 31T02:27:31.989433+0000 mon.b (mon. 2021-07-31T02:27:32.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:32 smithi045 conmon[39718]: 2) 275 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:27:32.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:32 smithi045 conmon[39718]: audit 2021- 2021-07-31T02:27:32.282 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:32 smithi045 conmon[39718]: 07-31T02:27:31.990949 2021-07-31T02:27:32.282 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:32 smithi045 conmon[39718]: +0000 mon.b (mon.2) 276 : audit 2021-07-31T02:27:32.282 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:32 smithi045 conmon[39718]: [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:27:32.282 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:32 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T02:27:32.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:32 smithi045 conmon[39718]: T02:27:31.992046+0000 mon.b ( 2021-07-31T02:27:32.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:32 smithi045 conmon[39718]: mon.2) 277 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:27:32.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:32 smithi045 conmon[39718]: 2021-07-31T02:27:32.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:32 smithi045 conmon[39718]: cluster 2021-07-31T02:27 2021-07-31T02:27:32.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:32 smithi045 conmon[39718]: :32.238722+0000 mon.a (mon.0 2021-07-31T02:27:32.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:32 smithi045 conmon[39718]: ) 2005 : cluster [DBG] osdmap e1099: 8 total, 8 up, 8 in 2021-07-31T02:27:32.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[34553]: cluster 2021-07-31T02:27:30.372145+0000 mgr.x (mgr.24542) 1721 2021-07-31T02:27:32.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[34553]: : cluster [DBG] pgmap v2152: 169 pgs: 169 active+clean; 41 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 552 KiB/s rd, 666 KiB/s wr, 175 op/s 2021-07-31T02:27:32.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[34553]: cluster 2021-07-31T02:27:31.258763+0000 mon.a (mon.0) 2004 : cluster [DBG] osdmap e1098: 8 total, 8 up, 8 in 2021-07-31T02:27:32.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[34553]: audit 2021- 2021-07-31T02:27:32.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[34553]: 07-31T02:27:31.989433+0000 mon.b (mon.2) 275 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:27:32.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[34553]: audit 2021-07-31T02:27:32.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[34553]: 2021-07-31T02:27:31.990949+0000 2021-07-31T02:27:32.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[34553]: mon.b (mon.2) 276 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:27:32.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[34553]: 2021-07-31T02:27:32.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[34553]: audit 2021-07-31T02:27:32.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[34553]: 2021-07-31T02:27:31.992046 2021-07-31T02:27:32.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[34553]: +0000 mon.b (mon.2) 277 : audit [INF] 2021-07-31T02:27:32.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[34553]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:27:32.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[34553]: cluster 2021-07-31T02:27:32.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[34553]: 2021-07-31T02:27:32. 2021-07-31T02:27:32.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[34553]: 238722+0000 mon.a (mon.0) 2005 : cluster 2021-07-31T02:27:32.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[34553]: [DBG] osdmap e1099: 8 total, 8 up, 8 in 2021-07-31T02:27:32.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: debug 2021-07-31T02:27:32.034+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:27:32.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:27:32.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:27:32.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: yield 2021-07-31T02:27:32.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:27:32.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:27:32.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:27:32.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:27:32.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:27:32.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:27:32.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:27:32.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:27:32.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:27:32.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: task.result() 2021-07-31T02:27:32.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:27:32.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: debug 2021-07-31T02:27:32.046+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:27:32.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:27:32.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:27:32.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: yield 2021-07-31T02:27:32.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:27:32.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:27:32.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:27:32.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:27:32.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:27:32.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:27:32.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:27:32.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:27:32.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:27:32.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: task.result() 2021-07-31T02:27:32.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:27:32.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: debug 2021-07-31T02:27:32.072+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:27:32.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:27:32.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:27:32.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: yield 2021-07-31T02:27:32.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:27:32.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:27:32.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:27:32.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:27:32.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:27:32.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:27:32.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:27:32.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:27:32.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:27:32.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: task.result() 2021-07-31T02:27:32.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:27:32.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: debug 2021-07-31T02:27:32.072+0000 7fcb93961700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:27:32.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:27:32.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:27:32.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: yield 2021-07-31T02:27:32.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:27:32.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:27:32.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:27:32.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:27:32.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:27:32.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:27:32.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:27:32.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:27:32.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:27:32.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: task.result() 2021-07-31T02:27:32.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:27:32.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: 2021-07-31T02:27:32.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:27:32.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: 2021-07-31T02:27:32.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:27:32.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:27:32.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:27:32.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:27:32.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:27:32.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:27:32.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:27:32.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:27:32.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:27:32.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:27:32.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:27:32.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:27:32.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:27:32.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:27:32.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:27:32.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:27:32.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:27:32.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:27:32.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: raise 2021-07-31T02:27:32.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:27:32.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:27:32.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:27:32.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:27:32.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:27:32.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: debug 2021-07-31T02:27:32.081+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:27:32.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:27:32.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:27:32.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: yield 2021-07-31T02:27:32.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:27:32.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:27:32.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:27:32.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:27:32.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:27:32.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:27:32.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:27:32.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:27:32.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:27:32.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: task.result() 2021-07-31T02:27:32.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:27:32.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: debug 2021-07-31T02:27:32.081+0000 7fcb94162700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:27:32.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:27:32.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:27:32.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: yield 2021-07-31T02:27:32.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:27:32.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:27:32.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:27:32.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:27:32.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:27:32.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:27:32.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:27:32.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:27:32.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:27:32.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: task.result() 2021-07-31T02:27:32.535 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:27:32.535 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: 2021-07-31T02:27:32.535 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:27:32.535 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: 2021-07-31T02:27:32.535 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:27:32.536 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:27:32.536 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:27:32.536 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:27:32.536 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:27:32.537 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:27:32.537 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:27:32.537 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:27:32.537 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:27:32.537 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:27:32.538 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:27:32.538 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:27:32.538 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:27:32.538 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:27:32.538 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:27:32.539 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:27:32.539 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:27:32.539 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:27:32.539 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: raise 2021-07-31T02:27:32.540 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:27:32.540 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:27:32.540 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:27:32.540 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:27:32.540 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:27:32 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:27:33.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:27:31.992829+0000 mgr.x (mgr.24542) 1722 : cephadm 2021-07-31T02:27:33.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:27:33.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:27:32.035887+0000 mgr.x (mgr.24542) 1723 : cephadm [ERR] 2021-07-31T02:27:33.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:27:33.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:27:33.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: yield 2021-07-31T02:27:33.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:27:33.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:27:33.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:27:33.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:27:33.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:27:33.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:27:33.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:27:33.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:27:33.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:27:33.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:27:33.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:27:33.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:27:32.036396+0000 mgr.x (mgr.24542) 1724 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:27:33.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:27:32.047409+0000 mgr.x (mgr.24542) 1725 : cephadm [ERR] 2021-07-31T02:27:33.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:27:33.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:27:33.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: yield 2021-07-31T02:27:33.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:27:33.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:27:33.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:27:33.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:27:33.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:27:33.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:27:33.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:27:33.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:27:33.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:27:33.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:27:33.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:27:33.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:27:32.048349+0000 mgr.x (mgr. 2021-07-31T02:27:33.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: 24542) 1726 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:27:33.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:27:32.073048+0000 mgr.x (mgr.24542) 1727 : cephadm [ERR] 2021-07-31T02:27:33.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:27:33.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:27:33.535 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:27:31.992829+0000 mgr.x (mgr. 2021-07-31T02:27:33.535 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: 24542) 1722 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:27:33.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:27:32.035887+0000 mgr.x (mgr.24542) 1723 : cephadm [ERR] 2021-07-31T02:27:33.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: 2021-07-31T02:27:33.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:27:33.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:27:33.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: yield 2021-07-31T02:27:33.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:27:33.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:27:33.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:27:33.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:27:33.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:27:33.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:27:33.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:27:33.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:27:33.539 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:27:33.539 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:27:33.539 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:27:33.539 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:27:32.036396+0000 mgr.x (mgr.24542) 1724 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:27:33.540 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:27:33.540 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: 2021-07-31T02:27: 2021-07-31T02:27:33.540 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: 32.047409+0000 mgr.x (mgr. 2021-07-31T02:27:33.540 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: 24542) 1725 : cephadm [ERR] 2021-07-31T02:27:33.540 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:27:33.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:27:33.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: yield 2021-07-31T02:27:33.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:27:33.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:27:33.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:27:33.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:27:33.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:27:33.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:27:33.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:27:33.543 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:27:33.543 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:27:33.543 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:27:33.543 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:27:33.543 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: 2021-07-31T02:27:33.544 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:27:32.048349+0000 mgr.x (mgr.24542) 1726 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:27:33.544 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: cephadm 2021-07-31 2021-07-31T02:27:33.544 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: T02:27:32.073048+0000 2021-07-31T02:27:33.544 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: mgr.x (mgr.24542) 1727 : cephadm [ERR] 2021-07-31T02:27:33.545 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: 2021-07-31T02:27:33.545 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:27:33.545 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:27:33.545 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: yield 2021-07-31T02:27:33.545 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:27:33.546 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:27:33.546 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:27:33.546 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:27:33.546 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:27:33.547 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:27:33.547 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:27:33.547 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:27:33.547 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:27:33.547 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:27:33.548 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:27:33.548 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:27:32.073483+0000 mgr.x (mgr.24542) 1728 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:27:33.548 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:27:33.548 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:27:33.549 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: yield 2021-07-31T02:27:33.549 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:27:33.549 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:27:33.549 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:27:33.549 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:27:33.550 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:27:33.550 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:27:33.550 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:27:33.550 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:27:33.551 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:27:33.551 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:27:33.551 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:27:33.551 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: 2021-07-31T02:27:33.551 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:27:33.552 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: 2021-07-31T02:27:33.552 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:27:33.552 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:27:33.552 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:27:33.553 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:27:33.553 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:27:33.553 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:27:33.553 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:27:33.553 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:27:33.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: yield 2021-07-31T02:27:33.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:27:33.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:27:33.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:27:33.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:27:33.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:27:33.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:27:33.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:27:33.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:27:33.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:27:33.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:27:33.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:27:33.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:27:32.073483+0000 mgr.x (mgr.24542) 1728 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:27:33.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:27:33.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:27:33.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: yield 2021-07-31T02:27:33.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:27:33.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:27:33.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:27:33.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:27:33.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:27:33.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:27:33.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:27:33.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:27:33.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:27:33.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:27:33.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:27:33.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: 2021-07-31T02:27:33.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:27:33.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: 2021-07-31T02:27:33.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:27:33.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:27:33.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:27:33.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:27:33.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:27:33.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:27:33.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:27:33.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:27:33.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:27:33.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:27:33.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:27:33.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:27:33.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:27:33.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:27:33.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:27:33.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:27:33.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:27:33.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:27:33.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:27:33.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:27:33.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:27:33.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: raise 2021-07-31T02:27:33.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:27:33.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:27:33.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:27:33.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:27:33.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:27:33.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:27:32.082159+0000 mgr.x (mgr.24542) 1729 : cephadm [ERR] 2021-07-31T02:27:33.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:27:33.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:27:33.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: yield 2021-07-31T02:27:33.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:27:33.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:27:33.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:27:33.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:27:33.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:27:33.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:27:33.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:27:33.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:27:33.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:27:33.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:27:33.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:27:33.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:27:32.082555+0000 mgr.x (mgr.24542) 1730 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:27:33.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:27:33.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:27:33.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: yield 2021-07-31T02:27:33.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:27:33.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:27:33.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:27:33.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:27:33.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:27:33.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:27:33.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:27:33.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:27:33.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:27:33.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:27:33.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:27:33.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: raise 2021-07-31T02:27:33.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:27:33.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:27:33.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:27:33.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:27:33.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:27:33.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: cephadm 2021 2021-07-31T02:27:33.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: -07-31T02:27:32.082159+0000 mgr.x (mgr.24542) 1729 : cephadm [ERR] 2021-07-31T02:27:33.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:27:33.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:27:33.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: yield 2021-07-31T02:27:33.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:27:33.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:27:33.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:27:33.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:27:33.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:27:33.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:27:33.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:27:33.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:27:33.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:27:33.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:27:33.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:27:33.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:27:32.082555+0000 mgr.x (mgr.24542) 1730 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:27:33.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:27:33.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:27:33.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: yield 2021-07-31T02:27:33.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:27:33.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:27:33.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:27:33.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:27:33.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:27:33.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:27:33.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:27:33.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:27:33.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:27:33.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:27:33.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:27:33.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:27:33.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:27:33.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: 2021-07-31T02:27:33.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:27:33.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: 2021-07-31T02:27:33.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:27:33.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:27:33.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:27:33.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:27:33.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:27:33.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:27:33.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:27:33.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:27:33.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:27:33.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:27:33.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:27:33.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:27:33.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:27:33.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:27:33.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:27:33.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:27:33.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:27:33.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:27:33.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: raise 2021-07-31T02:27:33.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:27:33.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:27:33.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:27:33.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:27:33.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:27:33.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: cluster 2021-07-31T02:27:33.246098 2021-07-31T02:27:33.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[33655]: +0000 mon.a (mon.0) 2006 : cluster [DBG] osdmap e1100: 8 total, 8 up, 8 in 2021-07-31T02:27:33.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:27:33.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:27:33.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:27:33.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:27:33.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:27:33.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: 2021-07-31T02:27:33.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:27:33.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: 2021-07-31T02:27:33.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:27:33.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:27:33.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:27:33.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:27:33.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:27:33.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:27:33.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:27:33.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:27:33.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:27:33.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:27:33.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:27:33.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:27:33.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:27:33.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:27:33.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:27:33.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:27:33.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:27:33.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:27:33.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: raise 2021-07-31T02:27:33.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:27:33.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:27:33.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:27:33.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:27:33.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:27:33.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[39718]: cluster 2021-07-31T02:27:33.246098+0000 mon.a (mon.0) 2006 : cluster [DBG] osdmap e1100: 8 total, 8 up, 8 in 2021-07-31T02:27:33.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:27:31.992829+0000 mgr.x (mgr.24542) 1722 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:27:33.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:27:32.035887+0000 mgr.x (mgr.24542) 1723 : cephadm [ERR] 2021-07-31T02:27:33.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:27:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:27:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: yield 2021-07-31T02:27:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:27:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:27:33.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:27:33.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:27:33.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:27:33.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:27:33.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:27:33.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:27:33.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:27:33.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:27:33.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:27:33.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:27:32.036396+0000 mgr.x (mgr.24542) 1724 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:27:33.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:27:32.047409+0000 mgr.x (mgr.24542) 1725 : cephadm [ERR] 2021-07-31T02:27:33.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:27:33.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:27:33.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: yield 2021-07-31T02:27:33.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:27:33.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:27:33.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:27:33.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:27:33.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:27:33.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:27:33.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:27:33.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:27:33.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:27:33.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:27:33.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:27:33.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:27:32.048349+0000 mgr.x (mgr.24542) 1726 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:27:33.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:27:32.073048+0000 mgr.x (mgr.24542) 1727 : cephadm [ERR] 2021-07-31T02:27:33.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:27:33.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:27:33.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: yield 2021-07-31T02:27:33.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:27:33.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:27:33.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:27:33.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:27:33.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:27:33.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:27:33.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:27:33.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:27:33.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:27:33.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:27:33.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:27:33.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:27:32.073483+0000 mgr.x (mgr.24542) 1728 : cephadm [ERR] 2021-07-31T02:27:33.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:27:33.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:27:33.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:27:33.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: yield 2021-07-31T02:27:33.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:27:33.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:27:33.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:27:33.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:27:33.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:27:33.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:27:33.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:27:33.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:27:33.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:27:33.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:27:33.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:27:33.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: 2021-07-31T02:27:33.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:27:33.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: 2021-07-31T02:27:33.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:27:33.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:27:33.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:27:33.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:27:33.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:27:33.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:27:33.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:27:33.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:27:33.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:27:33.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:27:33.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:27:33.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:27:33.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:27:33.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:27:33.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:27:33.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:27:33.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:27:33.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:27:33.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: raise 2021-07-31T02:27:33.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:27:33.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:27:33.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:27:33.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:27:33.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:27:33.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:27:32.082159+0000 mgr.x (mgr.24542) 1729 : cephadm [ERR] 2021-07-31T02:27:33.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:27:33.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:27:33.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: yield 2021-07-31T02:27:33.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:27:33.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:27:33.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:27:33.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:27:33.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:27:33.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:27:33.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:27:33.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:27:33.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:27:33.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:27:33.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:27:33.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:27:32.082555+0000 mgr.x (mgr.24542) 1730 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:27:33.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:27:33.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:27:33.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: yield 2021-07-31T02:27:33.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:27:33.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:27:33.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:27:33.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:27:33.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:27:33.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:27:33.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:27:33.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:27:33.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:27:33.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:27:33.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:27:33.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: 2021-07-31T02:27:33.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:27:33.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: 2021-07-31T02:27:33.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:27:33.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:27:33.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:27:33.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:27:33.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:27:33.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:27:33.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:27:33.782 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:27:33.782 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:27:33.782 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:27:33.782 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:27:33.783 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:27:33.783 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:27:33.783 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:27:33.783 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:27:33.783 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:27:33.784 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:27:33.784 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:27:33.784 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: raise 2021-07-31T02:27:33.784 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:27:33.784 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:27:33.785 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:27:33.785 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:27:33.785 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:27:33.785 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:33 smithi154 conmon[34553]: cluster 2021-07-31T02:27:33.246098+0000 mon.a (mon.0) 2006 : cluster [DBG] osdmap e1100: 8 total, 8 up, 8 in 2021-07-31T02:27:34.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:27:33.578Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:27:34.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:27:33.579Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:27:34.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:27:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:27:33.579Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:27:34.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:34 smithi045 conmon[33655]: cluster 2021-07-31T02:27:32.372734+0000 2021-07-31T02:27:34.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:34 smithi045 conmon[33655]: mgr.x (mgr.24542) 1731 : cluster [DBG] pgmap v2155: 169 pgs: 169 active+clean; 41 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 552 KiB/s rd, 666 KiB/s wr, 175 op/s 2021-07-31T02:27:34.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:34 smithi045 conmon[33655]: cluster 2021-07-31T02:27:34.248349+0000 mon.a (mon.0) 2007 : cluster [DBG] osdmap e1101: 8 total, 8 up, 8 in 2021-07-31T02:27:34.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:34 smithi045 conmon[39718]: cluster 2021-07-31T02:27:32.372734+0000 mgr.x 2021-07-31T02:27:34.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:34 smithi045 conmon[39718]: (mgr.24542) 1731 : cluster [DBG] pgmap v2155: 169 pgs: 169 active+clean; 41 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 552 KiB/s rd, 666 KiB/s wr, 175 op/s 2021-07-31T02:27:34.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:34 smithi045 conmon[39718]: cluster 2021-07-31 2021-07-31T02:27:34.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:34 smithi045 conmon[39718]: T02:27:34.248349+0000 mon.a (mon.0) 2007 : cluster 2021-07-31T02:27:34.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:34 smithi045 conmon[39718]: [DBG] osdmap e1101: 8 total, 8 up, 8 in 2021-07-31T02:27:34.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:34 smithi154 conmon[34553]: cluster 2021-07-31T02:27:32.372734+0000 mgr.x (mgr.24542) 1731 : cluster [DBG] pgmap v2155: 169 pgs: 169 active+clean; 41 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 552 KiB/s rd, 666 KiB/s wr, 175 op/s 2021-07-31T02:27:34.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:34 smithi154 conmon[34553]: 2021-07-31T02:27:34.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:34 smithi154 conmon[34553]: cluster 2021-07-31T02:27:34.248349+0000 mon.a (mon.0) 2007 : cluster [DBG] osdmap e1101: 8 total, 8 up, 8 in 2021-07-31T02:27:35.259 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.SnapshotLimitPP (5000 ms) 2021-07-31T02:27:35.259 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.RebuildObjectMapViaLockOwner 2021-07-31T02:27:35.260 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:27:35.260 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.RebuildObjectMapViaLockOwner (0 ms) 2021-07-31T02:27:35.260 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.RenameViaLockOwner 2021-07-31T02:27:35.261 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:27:35.261 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.RenameViaLockOwner (0 ms) 2021-07-31T02:27:35.261 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.SnapCreateViaLockOwner 2021-07-31T02:27:35.261 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:27:35.261 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.SnapCreateViaLockOwner (0 ms) 2021-07-31T02:27:35.262 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.SnapRemoveViaLockOwner 2021-07-31T02:27:35.262 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:27:35.262 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.SnapRemoveViaLockOwner (0 ms) 2021-07-31T02:27:35.262 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.EnableJournalingViaLockOwner 2021-07-31T02:27:35.263 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:27:35.263 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.EnableJournalingViaLockOwner (0 ms) 2021-07-31T02:27:35.263 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.SnapRemove2 2021-07-31T02:27:35.263 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:27:35.263 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.SnapRemove2 (0 ms) 2021-07-31T02:27:35.264 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.SnapRenameViaLockOwner 2021-07-31T02:27:35.264 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:27:35.264 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.SnapRenameViaLockOwner (0 ms) 2021-07-31T02:27:35.264 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.SnapProtectViaLockOwner 2021-07-31T02:27:35.264 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:27:35.265 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.SnapProtectViaLockOwner (0 ms) 2021-07-31T02:27:35.265 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.SnapUnprotectViaLockOwner 2021-07-31T02:27:35.265 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:27:35.265 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.SnapUnprotectViaLockOwner (0 ms) 2021-07-31T02:27:35.266 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.FlattenViaLockOwner 2021-07-31T02:27:35.266 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:27:35.266 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.FlattenViaLockOwner (0 ms) 2021-07-31T02:27:35.266 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.ResizeViaLockOwner 2021-07-31T02:27:35.266 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:27:35.267 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.ResizeViaLockOwner (0 ms) 2021-07-31T02:27:35.267 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.SparsifyViaLockOwner 2021-07-31T02:27:35.267 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:27:35.267 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.SparsifyViaLockOwner (0 ms) 2021-07-31T02:27:35.267 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.ObjectMapConsistentSnap 2021-07-31T02:27:35.268 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:27:35.268 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.ObjectMapConsistentSnap (0 ms) 2021-07-31T02:27:35.268 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.Metadata 2021-07-31T02:27:35.268 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:27:35.268 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.Metadata (0 ms) 2021-07-31T02:27:35.269 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.MetadataPP 2021-07-31T02:27:35.269 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:27:35.269 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.MetadataPP (0 ms) 2021-07-31T02:27:35.269 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.UpdateFeatures 2021-07-31T02:27:35.270 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:27:35.270 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.UpdateFeatures (0 ms) 2021-07-31T02:27:35.270 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.FeaturesBitmaskString 2021-07-31T02:27:35.270 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.FeaturesBitmaskString (0 ms) 2021-07-31T02:27:35.270 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.RebuildObjectMap 2021-07-31T02:27:35.271 INFO:tasks.workunit.client.0.smithi045.stdout:using old format 2021-07-31T02:27:35.275 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.RebuildObjectMap (17 ms) 2021-07-31T02:27:35.275 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.RebuildNewObjectMap 2021-07-31T02:27:35.276 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:27:35.276 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.RebuildNewObjectMap (0 ms) 2021-07-31T02:27:35.276 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.CheckObjectMap 2021-07-31T02:27:35.276 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:27:35.277 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.CheckObjectMap (0 ms) 2021-07-31T02:27:35.277 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.BlockingAIO 2021-07-31T02:27:35.277 INFO:tasks.workunit.client.0.smithi045.stdout:using old format 2021-07-31T02:27:35.296 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.BlockingAIO (19 ms) 2021-07-31T02:27:35.296 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.ExclusiveLockTransition 2021-07-31T02:27:35.297 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:27:35.297 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.ExclusiveLockTransition (0 ms) 2021-07-31T02:27:35.298 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.ExclusiveLockReadTransition 2021-07-31T02:27:35.298 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:27:35.298 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.ExclusiveLockReadTransition (0 ms) 2021-07-31T02:27:35.298 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.CacheMayCopyOnWrite 2021-07-31T02:27:35.299 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:27:35.300 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.CacheMayCopyOnWrite (0 ms) 2021-07-31T02:27:35.300 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.FlushEmptyOpsOnExternalSnapshot 2021-07-31T02:27:35.301 INFO:tasks.workunit.client.0.smithi045.stdout:using old format 2021-07-31T02:27:36.284 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.FlushEmptyOpsOnExternalSnapshot (990 ms) 2021-07-31T02:27:36.284 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.TestImageOptions 2021-07-31T02:27:36.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:36 smithi154 conmon[34553]: cluster 2021-07-31T02:27:34.373363+0000 mgr.x (mgr.24542) 1732 : cluster [DBG] pgmap v2158: 169 pgs: 169 active+clean; 41 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.5 KiB/s wr, 7 op/s 2021-07-31T02:27:36.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:36 smithi154 conmon[34553]: cluster 2021-07-31T02:27:35.249892+0000 mon.a (mon.0) 2008 : cluster [DBG] osdmap e1102: 8 total, 8 up, 8 in 2021-07-31T02:27:36.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:36 smithi045 conmon[39718]: cluster 2021-07-31T02:27:34.373363+0000 mgr.x 2021-07-31T02:27:36.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:36 smithi045 conmon[39718]: (mgr.24542) 1732 : cluster [DBG] pgmap v2158: 169 pgs: 169 active+clean; 41 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.5 KiB/s wr, 7 op/s 2021-07-31T02:27:36.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:36 smithi045 conmon[39718]: cluster 2021-07-31T02 2021-07-31T02:27:36.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:36 smithi045 conmon[39718]: :27:35.249892+0000 mon.a (mon.0) 2008 : 2021-07-31T02:27:36.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:36 smithi045 conmon[39718]: cluster [DBG] osdmap e1102: 8 total, 8 up, 8 in 2021-07-31T02:27:36.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:36 smithi045 conmon[33655]: cluster 2021-07-31T02:27:34. 2021-07-31T02:27:36.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:36 smithi045 conmon[33655]: 373363+0000 mgr.x (mgr.24542) 1732 : cluster [DBG] pgmap v2158: 169 pgs: 169 active+clean; 41 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.5 KiB/s wr, 7 op/s 2021-07-31T02:27:36.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:36 smithi045 conmon[33655]: cluster 2021-07-31T02:27:36.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:36 smithi045 conmon[33655]: 2021-07-31T 2021-07-31T02:27:36.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:36 smithi045 conmon[33655]: 02:27: 2021-07-31T02:27:36.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:36 smithi045 conmon[33655]: 35.249892+0000 mon.a (mon.0) 2008 : 2021-07-31T02:27:36.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:36 smithi045 conmon[33655]: cluster [DBG] osdmap e1102: 8 total, 8 up, 8 in 2021-07-31T02:27:37.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:37 smithi045 conmon[39718]: cluster 2021-07-31T02:27:36.269821+0000 mon.a 2021-07-31T02:27:37.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:37 smithi045 conmon[39718]: (mon.0) 2009 : cluster [DBG] osdmap e1103: 8 total, 8 up, 8 in 2021-07-31T02:27:37.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:37 smithi045 conmon[39718]: cluster 2021-07-31T02:27:37.239126 2021-07-31T02:27:37.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:37 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2010 : cluster [DBG] osdmap e1104: 8 total, 8 up, 8 in 2021-07-31T02:27:37.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:37 smithi045 conmon[33655]: cluster 2021-07-31T02:27:36.269821+0000 mon.a (mon.0) 2009 2021-07-31T02:27:37.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:37 smithi045 conmon[33655]: : cluster [DBG] osdmap e1103: 8 total, 8 up, 8 in 2021-07-31T02:27:37.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:37 smithi045 conmon[33655]: cluster 2021-07-31T02:27:37.239126+0000 mon.a (mon.0) 2010 : cluster [DBG] osdmap e1104: 8 total, 8 up, 8 in 2021-07-31T02:27:37.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:37 smithi154 conmon[34553]: cluster 2021-07-31T02:27:36.269821+0000 mon.a (mon.0) 2009 : cluster [DBG] osdmap e1103: 8 total, 8 up, 8 in 2021-07-31T02:27:37.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:37 smithi154 conmon[34553]: cluster 2021-07-31T02:27:37.239126+0000 mon.a (mon.0) 2010 : cluster [DBG] osdmap e1104: 8 total, 8 up, 8 in 2021-07-31T02:27:38.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:38 smithi045 conmon[39718]: cluster 2021-07-31T02:27:36 2021-07-31T02:27:38.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:38 smithi045 conmon[39718]: .374015+0000 mgr.x (mgr.24542) 1733 : cluster [DBG] pgmap v2161: 169 pgs: 169 active+clean; 41 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.5 KiB/s wr, 7 op/s 2021-07-31T02:27:38.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:38 smithi045 conmon[39718]: cluster 2021-07- 2021-07-31T02:27:38.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:38 smithi045 conmon[39718]: 31T02:27:38.246054+0000 mon.a (mon.0 2021-07-31T02:27:38.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:38 smithi045 conmon[39718]: ) 2011 : cluster [DBG] osdmap e1105: 8 total, 8 up, 8 in 2021-07-31T02:27:38.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:38 smithi045 conmon[33655]: cluster 2021-07-31T02:27:36.374015+0000 mgr.x 2021-07-31T02:27:38.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:38 smithi045 conmon[33655]: (mgr.24542) 1733 : cluster [DBG] pgmap v2161: 169 pgs: 169 active+clean; 41 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.5 KiB/s wr, 7 op/s 2021-07-31T02:27:38.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:38 smithi045 conmon[33655]: cluster 2021-07-31T02:27:38.246054+0000 mon.a (mon.0) 2011 : cluster 2021-07-31T02:27:38.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:38 smithi045 conmon[33655]: [DBG] osdmap e1105: 8 total, 8 up, 8 in 2021-07-31T02:27:38.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:38 smithi154 conmon[34553]: cluster 2021-07-31T02:27:36.374015+0000 mgr.x (mgr.24542) 1733 : cluster [DBG] pgmap v2161: 169 pgs: 169 active+clean; 41 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.5 KiB/s wr, 7 op/s 2021-07-31T02:27:38.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:38 smithi154 conmon[34553]: 2021-07-31T02:27:38.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:38 smithi154 conmon[34553]: cluster 2021-07-31T02:27:38.246054+0000 mon.a (mon.0) 2011 : cluster [DBG] osdmap e1105: 8 total, 8 up, 8 in 2021-07-31T02:27:39.070 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:27:39.367 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:27:39.368 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 42m ago 43m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:27:39.368 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 40m ago 43m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:27:39.368 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 40m ago 46m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:27:39.368 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 42m ago 48m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:27:39.369 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 42m ago 48m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:27:39.369 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 40m ago 46m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:27:39.369 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 42m ago 46m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:27:39.369 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 42m ago 43m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:27:39.370 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 40m ago 43m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:27:39.370 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 42m ago 46m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:27:39.370 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 42m ago 45m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:27:39.370 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 42m ago 45m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:27:39.371 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 42m ago 45m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:27:39.371 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 40m ago 44m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:27:39.371 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 40m ago 44m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:27:39.371 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 40m ago 44m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:27:39.372 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 40m ago 43m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:27:39.372 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 40m ago 43m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:27:39.399 INFO:tasks.workunit.client.0.smithi045.stdout:0.00% done 2021-07-31T02:27:39.400 INFO:tasks.workunit.client.0.smithi045.stdout:100.00% done 2021-07-31T02:27:39.400 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.TestImageOptions (3115 ms) 2021-07-31T02:27:39.401 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.TestImageOptionsPP 2021-07-31T02:27:39.711 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:27:39.711 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:27:39.711 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:27:39.712 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:27:39.712 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:27:39.712 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:27:39.712 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:27:39.712 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:27:39.713 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:27:39.713 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:27:39.713 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:27:39.713 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:27:39.714 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:27:39.714 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:27:39.714 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:27:39.714 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:27:39.715 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:27:40.411 INFO:tasks.workunit.client.0.smithi045.stdout:0.00% done 2021-07-31T02:27:40.411 INFO:tasks.workunit.client.0.smithi045.stdout:100.00% done 2021-07-31T02:27:40.411 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.TestImageOptionsPP (1011 ms) 2021-07-31T02:27:40.412 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.EventSocketPipe 2021-07-31T02:27:40.412 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.EventSocketPipe (0 ms) 2021-07-31T02:27:40.412 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.EventSocketEventfd 2021-07-31T02:27:40.412 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.EventSocketEventfd (0 ms) 2021-07-31T02:27:40.412 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.ImagePollIO 2021-07-31T02:27:40.413 INFO:tasks.workunit.client.0.smithi045.stdout:using old format 2021-07-31T02:27:40.445 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 512 2021-07-31T02:27:40.445 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 512 2021-07-31T02:27:40.446 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 512 2021-07-31T02:27:40.446 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 512 2021-07-31T02:27:40.446 INFO:tasks.workunit.client.0.smithi045.stdout:wrote: 512 2021-07-31T02:27:40.446 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:40.447 INFO:tasks.workunit.client.0.smithi045.stdout:started write 2021-07-31T02:27:40.447 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:40.447 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:40.447 INFO:tasks.workunit.client.0.smithi045.stdout:finished write 2021-07-31T02:27:40.447 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:40.448 INFO:tasks.workunit.client.0.smithi045.stdout:started write 2021-07-31T02:27:40.448 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:40.448 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:40.448 INFO:tasks.workunit.client.0.smithi045.stdout:finished write 2021-07-31T02:27:40.449 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:40.449 INFO:tasks.workunit.client.0.smithi045.stdout:started write 2021-07-31T02:27:40.449 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:40.449 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:40.449 INFO:tasks.workunit.client.0.smithi045.stdout:finished write 2021-07-31T02:27:40.450 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:40.450 INFO:tasks.workunit.client.0.smithi045.stdout:started write 2021-07-31T02:27:40.450 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:40.450 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:40.450 INFO:tasks.workunit.client.0.smithi045.stdout:finished write 2021-07-31T02:27:40.451 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:40.451 INFO:tasks.workunit.client.0.smithi045.stdout:started write 2021-07-31T02:27:40.451 INFO:tasks.workunit.client.0.smithi045.stdout:write completion cb called! 2021-07-31T02:27:40.451 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 0 2021-07-31T02:27:40.451 INFO:tasks.workunit.client.0.smithi045.stdout:finished write 2021-07-31T02:27:40.452 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:40.452 INFO:tasks.workunit.client.0.smithi045.stdout:started read 2021-07-31T02:27:40.454 INFO:tasks.workunit.client.0.smithi045.stdout:read completion cb called! 2021-07-31T02:27:40.454 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 512 2021-07-31T02:27:40.454 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:40.455 INFO:tasks.workunit.client.0.smithi045.stdout:started read 2021-07-31T02:27:40.455 INFO:tasks.workunit.client.0.smithi045.stdout:read completion cb called! 2021-07-31T02:27:40.455 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 512 2021-07-31T02:27:40.455 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:40.456 INFO:tasks.workunit.client.0.smithi045.stdout:started read 2021-07-31T02:27:40.456 INFO:tasks.workunit.client.0.smithi045.stdout:read completion cb called! 2021-07-31T02:27:40.456 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 512 2021-07-31T02:27:40.456 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:40.456 INFO:tasks.workunit.client.0.smithi045.stdout:started read 2021-07-31T02:27:40.457 INFO:tasks.workunit.client.0.smithi045.stdout:read completion cb called! 2021-07-31T02:27:40.457 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 512 2021-07-31T02:27:40.457 INFO:tasks.workunit.client.0.smithi045.stdout:created completion 2021-07-31T02:27:40.458 INFO:tasks.workunit.client.0.smithi045.stdout:started read 2021-07-31T02:27:40.458 INFO:tasks.workunit.client.0.smithi045.stdout:read completion cb called! 2021-07-31T02:27:40.458 INFO:tasks.workunit.client.0.smithi045.stdout:return value is: 512 2021-07-31T02:27:40.458 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.ImagePollIO (35 ms) 2021-07-31T02:27:40.459 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.Mirror 2021-07-31T02:27:40.459 INFO:tasks.workunit.client.0.smithi045.stdout:using old format 2021-07-31T02:27:40.460 INFO:tasks.workunit.client.0.smithi045.stdout:using old format 2021-07-31T02:27:40.493 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.Mirror (47 ms) 2021-07-31T02:27:40.493 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.MirrorPeerAttributes 2021-07-31T02:27:40.507 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.MirrorPeerAttributes (15 ms) 2021-07-31T02:27:40.508 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.CreateWithMirrorEnabled 2021-07-31T02:27:40.508 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:27:40.508 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.CreateWithMirrorEnabled (0 ms) 2021-07-31T02:27:40.508 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.FlushCacheWithCopyupOnExternalSnapshot 2021-07-31T02:27:40.509 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:27:40.509 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.FlushCacheWithCopyupOnExternalSnapshot (0 ms) 2021-07-31T02:27:40.509 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.ExclusiveLock 2021-07-31T02:27:40.509 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:27:40.509 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.ExclusiveLock (0 ms) 2021-07-31T02:27:40.510 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.BreakLock 2021-07-31T02:27:40.510 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:27:40.510 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.BreakLock (0 ms) 2021-07-31T02:27:40.510 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.DiscardAfterWrite 2021-07-31T02:27:40.511 INFO:tasks.workunit.client.0.smithi045.stdout:using old format 2021-07-31T02:27:40.519 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.DiscardAfterWrite (11 ms) 2021-07-31T02:27:40.519 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.DefaultFeatures 2021-07-31T02:27:40.519 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.DefaultFeatures (0 ms) 2021-07-31T02:27:40.519 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.TestTrashMoveAndPurge 2021-07-31T02:27:40.520 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:27:40.520 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.TestTrashMoveAndPurge (0 ms) 2021-07-31T02:27:40.520 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.TestTrashMoveAndPurgeNonExpiredDelay 2021-07-31T02:27:40.520 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:27:40.520 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.TestTrashMoveAndPurgeNonExpiredDelay (0 ms) 2021-07-31T02:27:40.521 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.TestTrashPurge 2021-07-31T02:27:40.521 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:27:40.521 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.TestTrashPurge (0 ms) 2021-07-31T02:27:40.521 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.TestTrashMoveAndRestore 2021-07-31T02:27:40.522 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:27:40.522 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.TestTrashMoveAndRestore (0 ms) 2021-07-31T02:27:40.522 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.TestListWatchers 2021-07-31T02:27:40.523 INFO:tasks.workunit.client.0.smithi045.stdout:using old format 2021-07-31T02:27:40.533 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.TestListWatchers (15 ms) 2021-07-31T02:27:40.534 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.TestSetSnapById 2021-07-31T02:27:40.534 INFO:tasks.workunit.client.0.smithi045.stdout:using old format 2021-07-31T02:27:40.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:40 smithi154 conmon[34553]: cluster 2021-07-31T02:27:38.374553+0000 mgr.x (mgr.24542) 1734 : cluster [DBG] pgmap v2164: 169 pgs: 169 active+clean; 41 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:27:40.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:40 smithi154 conmon[34553]: audit 2021-07-31T02:27:39.057963+0000 mgr.x (mgr.24542) 1735 : audit [DBG] from='client.17115 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:27:40.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:40 smithi154 conmon[34553]: cluster 2021-07-31T02:27:39.284099+0000 mon.a (mon.0) 2012 : cluster [DBG] osdmap e1106: 8 total, 8 up, 8 in 2021-07-31T02:27:40.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:40 smithi154 conmon[34553]: audit 2021-07-31T02:27:39 2021-07-31T02:27:40.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:40 smithi154 conmon[34553]: .711202+0000 mon.c (mon.1) 99 : audit [DBG] from='client.? 172.21.15.45:0/2860783713' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:27:40.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:40 smithi045 conmon[39718]: cluster 2021-07-31T02:27:38.374553+0000 mgr.x (mgr.24542) 1734 : cluster [DBG] pgmap v2164: 169 pgs: 169 active+clean; 41 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:27:40.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:40 smithi045 conmon[39718]: audit 2021-07-31T02:27:39.057963+0000 mgr.x (mgr.24542) 1735 : audit [DBG] from='client.17115 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:27:40.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:40 smithi045 conmon[39718]: cluster 2021-07-31T02:27:40.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:40 smithi045 conmon[39718]: 2021-07-31T02:27:39.284099+0000 mon.a (mon. 2021-07-31T02:27:40.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:40 smithi045 conmon[39718]: 0) 2012 : cluster [DBG] osdmap e1106: 8 total, 8 up, 8 in 2021-07-31T02:27:40.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:40 smithi045 conmon[39718]: audit 2021- 2021-07-31T02:27:40.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:40 smithi045 conmon[39718]: 07-31T02:27:39.711202+0000 2021-07-31T02:27:40.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:40 smithi045 conmon[39718]: mon.c (mon.1) 99 : audit 2021-07-31T02:27:40.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:40 smithi045 conmon[39718]: [DBG] from='client.? 172.21.15.45:0/2860783713' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:27:40.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:40 smithi045 conmon[33655]: cluster 2021-07-31T02:27:38.374553+0000 mgr.x (mgr.24542) 1734 : cluster [DBG] 2021-07-31T02:27:40.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:40 smithi045 conmon[33655]: pgmap v2164: 169 pgs: 169 active+clean; 41 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:27:40.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:40 smithi045 conmon[33655]: audit 2021-07-31T02:27:39.057963+0000 mgr.x (mgr.24542) 1735 : audit [DBG] from='client.17115 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:27:40.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:40 smithi045 conmon[33655]: cluster 2021-07-31T02:27:39.284099+0000 mon.a (mon. 2021-07-31T02:27:40.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:40 smithi045 conmon[33655]: 0) 2012 : cluster [DBG] osdmap e1106: 8 total, 8 up, 8 in 2021-07-31T02:27:40.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:40 smithi045 conmon[33655]: audit 2021-07-31T02:27:40.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:40 smithi045 conmon[33655]: 2021-07-31T02:27:39.711202+0000 mon.c 2021-07-31T02:27:40.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:40 smithi045 conmon[33655]: (mon.1) 99 : audit [DBG] from='client.? 172.21.15.45:0/2860783713' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:27:41.523 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.TestSetSnapById (982 ms) 2021-07-31T02:27:41.524 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.Namespaces 2021-07-31T02:27:41.547 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.Namespaces (31 ms) 2021-07-31T02:27:41.547 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.NamespacesPP 2021-07-31T02:27:41.578 INFO:tasks.workunit.client.0.smithi045.stdout:using old format 2021-07-31T02:27:41.579 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.NamespacesPP (32 ms) 2021-07-31T02:27:41.579 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.Migration 2021-07-31T02:27:41.579 INFO:tasks.workunit.client.0.smithi045.stdout:using old format 2021-07-31T02:27:41.580 INFO:tasks.workunit.client.0.smithi045.stdout:using old format 2021-07-31T02:27:41.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:41 smithi154 conmon[34553]: audit 2021-07-31T02:27:39.355538+0000 mgr.x (mgr.24542) 1736 : audit [DBG] from='client.17121 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:27:41.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:41 smithi154 conmon[34553]: cluster 2021-07-31T02:27:40.292302+0000 mon.a (mon.0) 2013 : cluster [DBG] osdmap e1107: 8 total, 8 up, 8 in 2021-07-31T02:27:41.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:41 smithi154 conmon[34553]: audit 2021-07-31T02:27:40.465795+0000 mon.a (mon.0) 2014 : audit [INF] from='client.? 172.21.15.45:0/32709565' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/63/uuid4"}]: dispatch 2021-07-31T02:27:41.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:41 smithi154 conmon[34553]: 2021-07-31T02:27:41.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:41 smithi154 conmon[34553]: audit 2021-07-31T02:27: 2021-07-31T02:27:41.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:41 smithi154 conmon[34553]: 40.468706+0000 mon.a (mon.0) 2015 : audit [INF] from='client.? 172.21.15.45:0/32709565' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/63/uuid4"}]': finished 2021-07-31T02:27:41.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:41 smithi154 conmon[34553]: audit 2021- 2021-07-31T02:27:41.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:41 smithi154 conmon[34553]: 07-31T02:27:40.470770+0000 mon.a (mon.0) 2016 : 2021-07-31T02:27:41.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:41 smithi154 conmon[34553]: audit [INF] from='client.? 172.21.15.45:0/32709565' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/63/6a0f8bf2-1571-4344-95ba-d9e1c08f02f4"}]: dispatch 2021-07-31T02:27:41.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:41 smithi154 conmon[34553]: audit 2021-07-31T02:27:41.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:41 smithi154 conmon[34553]: 2021-07-31T02:27:40.473991+0000 2021-07-31T02:27:41.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:41 smithi154 conmon[34553]: mon.a (mon.0) 2017 : audit [INF] 2021-07-31T02:27:41.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:41 smithi154 conmon[34553]: from='client.? 172.21.15.45:0/32709565' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/63/6a0f8bf2-1571-4344-95ba-d9e1c08f02f4"}]': finished 2021-07-31T02:27:41.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:41 smithi154 conmon[34553]: audit 2021-07- 2021-07-31T02:27:41.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:41 smithi154 conmon[34553]: 31T02:27:40.480937+0000 mon.a (mon.0) 2018 2021-07-31T02:27:41.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:41 smithi154 conmon[34553]: : audit [INF] from='client.? 172.21.15.45:0/32709565' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/63/5977bee1-3229-463c-bf67-5b58105e83c3"}]: dispatch 2021-07-31T02:27:41.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:41 smithi154 conmon[34553]: audit 2021-07- 2021-07-31T02:27:41.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:41 smithi154 conmon[34553]: 31T02:27:40.484289+0000 mon.a (mon.0) 2019 : audit [INF] 2021-07-31T02:27:41.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:41 smithi154 conmon[34553]: from='client.? 172.21.15.45:0/32709565' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/63/5977bee1-3229-463c-bf67-5b58105e83c3"}]': finished 2021-07-31T02:27:41.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:41 smithi154 conmon[34553]: audit 2021-07-31T02:27:41.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:41 smithi154 conmon[34553]: 2021-07-31T02:27:40.486285+0000 mon.a (mon 2021-07-31T02:27:41.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:41 smithi154 conmon[34553]: .0) 2020 : audit [INF] from='client.? 172.21.15.45:0/32709565' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/63/44246c88-eca9-43fd-b59d-a12c90ac280d"}]: dispatch 2021-07-31T02:27:41.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:41 smithi154 conmon[34553]: audit 2021-07 2021-07-31T02:27:41.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:41 smithi154 conmon[34553]: -31T02:27:40.488695+0000 mon.a (mon.0) 2021-07-31T02:27:41.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:41 smithi154 conmon[34553]: 2021 : audit [INF] from='client.? 172.21.15.45:0/32709565' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/63/44246c88-eca9-43fd-b59d-a12c90ac280d"}]': finished 2021-07-31T02:27:41.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:41 smithi154 conmon[34553]: audit 2021-07- 2021-07-31T02:27:41.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:41 smithi154 conmon[34553]: 31T02:27:40.497720+0000 mon.a (mon.0) 2022 2021-07-31T02:27:41.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:41 smithi154 conmon[34553]: : audit [DBG] from='client.? 172.21.15.45:0/32709565' entity='client.admin' cmd=[{"prefix": "config-key get", "key": "rbd/mirror/peer/63/2cc6f59e-6db2-491b-a549-fb8f3df81bde"}]: dispatch 2021-07-31T02:27:41.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:41 smithi154 conmon[34553]: audit 2021-07- 2021-07-31T02:27:41.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:41 smithi154 conmon[34553]: 31T02:27:40.501250+0000 mon.a (mon.0) 2021-07-31T02:27:41.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:41 smithi154 conmon[34553]: 2023 : audit [INF] from='client.? 172.21.15.45:0/32709565' entity='client.admin' 2021-07-31T02:27:41.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:41 smithi154 conmon[34553]: audit 2021-07- 2021-07-31T02:27:41.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:41 smithi154 conmon[34553]: 31T02:27:40.501553+0000 mon.a (mon.0) 2024 2021-07-31T02:27:41.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:41 smithi154 conmon[34553]: : audit [DBG] from='client.? 172.21.15.45:0/32709565' entity='client.admin' cmd=[{"prefix": "config-key get", "key": "rbd/mirror/peer/63/2cc6f59e-6db2-491b-a549-fb8f3df81bde"}]: dispatch 2021-07-31T02:27:41.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:41 smithi154 conmon[34553]: audit 2021-07-31T02:27:40.501856 2021-07-31T02:27:41.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:41 smithi154 conmon[34553]: +0000 mon.a (mon.0) 2025 : audit [INF] from='client.? 172.21.15.45:0/32709565' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/63/2cc6f59e-6db2-491b-a549-fb8f3df81bde"}]: dispatch 2021-07-31T02:27:41.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:41 smithi154 conmon[34553]: audit 2021-07 2021-07-31T02:27:41.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:41 smithi154 conmon[34553]: -31T02:27:40.504012+0000 mon.a (mon.0 2021-07-31T02:27:41.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:41 smithi154 conmon[34553]: ) 2026 : audit [INF] from='client.? 172.21.15.45:0/32709565' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/63/2cc6f59e-6db2-491b-a549-fb8f3df81bde"}]': finished 2021-07-31T02:27:41.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[33655]: audit 2021-07-31T02:27:39.355538+0000 mgr.x (mgr.24542) 2021-07-31T02:27:41.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[33655]: 1736 : audit [DBG] from='client.17121 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:27:41.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[33655]: cluster 2021-07-31T02:27:40.292302+0000 mon.a (mon.0) 2013 : cluster [DBG] osdmap e1107: 8 total, 8 up, 8 in 2021-07-31T02:27:41.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[33655]: audit 2021-07-31T02:27: 2021-07-31T02:27:41.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[33655]: 40.465795+0000 mon.a (mon.0) 2014 : audit [INF] 2021-07-31T02:27:41.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[33655]: from='client.? 172.21.15.45:0/32709565' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/63/uuid4"}]: dispatch 2021-07-31T02:27:41.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[33655]: audit 2021 2021-07-31T02:27:41.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[33655]: -07-31T02:27:40. 2021-07-31T02:27:41.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[33655]: 468706+0000 mon.a (mon.0 2021-07-31T02:27:41.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[33655]: ) 2015 : audit [INF] 2021-07-31T02:27:41.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[33655]: from='client.? 172.21.15.45:0/32709565' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/63/uuid4"}]': finished 2021-07-31T02:27:41.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[33655]: audit 2021 2021-07-31T02:27:41.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[33655]: -07-31T02:27:40. 2021-07-31T02:27:41.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[33655]: 470770+0000 mon.a (mon.0) 2021-07-31T02:27:41.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[33655]: 2016 : audit [INF] from='client.? 172.21.15.45:0/32709565' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/63/6a0f8bf2-1571-4344-95ba-d9e1c08f02f4"}]: dispatch 2021-07-31T02:27:41.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[33655]: audit 2021 2021-07-31T02:27:41.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[33655]: -07-31T02:27:40. 2021-07-31T02:27:41.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[33655]: 473991+0000 mon.a (mon. 2021-07-31T02:27:41.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[33655]: 0) 2017 : audit [INF] 2021-07-31T02:27:41.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[33655]: from='client.? 172.21.15.45:0/32709565' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/63/6a0f8bf2-1571-4344-95ba-d9e1c08f02f4"}]': finished 2021-07-31T02:27:41.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[33655]: audit 2021 2021-07-31T02:27:41.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[33655]: -07-31T02:27: 2021-07-31T02:27:41.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[33655]: 40.480937+0000 mon.a ( 2021-07-31T02:27:41.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[33655]: mon.0) 2018 : audit [INF] 2021-07-31T02:27:41.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[33655]: from='client.? 172.21.15.45:0/32709565' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/63/5977bee1-3229-463c-bf67-5b58105e83c3"}]: dispatch 2021-07-31T02:27:41.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[33655]: audit 2021 2021-07-31T02:27:41.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[33655]: -07-31T02:27:40 2021-07-31T02:27:41.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[33655]: .484289+0000 mon.a (mon 2021-07-31T02:27:41.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[33655]: .0) 2019 : audit [INF] 2021-07-31T02:27:41.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[33655]: from='client.? 172.21.15.45:0/32709565' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/63/5977bee1-3229-463c-bf67-5b58105e83c3"}]': finished 2021-07-31T02:27:41.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[33655]: audit 2021-07 2021-07-31T02:27:41.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[33655]: -31T02:27:40.486285+0000 mon.a ( 2021-07-31T02:27:41.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[33655]: mon.0) 2020 : audit [INF] from='client.? 172.21.15.45:0/32709565' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/63/44246c88-eca9-43fd-b59d-a12c90ac280d"}]: dispatch 2021-07-31T02:27:41.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[33655]: 2021-07-31T02:27:41.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[33655]: audit 2021- 2021-07-31T02:27:41.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[33655]: 07-31T02:27:40 2021-07-31T02:27:41.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[33655]: .488695+0000 mon.a (mon.0 2021-07-31T02:27:41.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[33655]: ) 2021 : audit [INF] from='client.? 172.21.15.45:0/32709565' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/63/44246c88-eca9-43fd-b59d-a12c90ac280d"}]': finished 2021-07-31T02:27:41.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[33655]: 2021-07-31T02:27:41.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[33655]: audit 2021-07-31 2021-07-31T02:27:41.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[33655]: T02:27:40.497720+0000 mon.a (mon.0) 2022 : audit 2021-07-31T02:27:41.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[39718]: audit 2021-07-31T02:27: 2021-07-31T02:27:41.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[39718]: 39.355538+0000 mgr.x (mgr.24542) 1736 : audit [DBG] from='client.17121 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:27:41.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[39718]: cluster 2021 2021-07-31T02:27:41.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[39718]: -07-31T02:27:40. 2021-07-31T02:27:41.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[39718]: 292302+0000 mon.a (mon. 2021-07-31T02:27:41.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[39718]: 0) 2013 : cluster [DBG] osdmap e1107: 8 total, 8 up, 8 in 2021-07-31T02:27:41.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[39718]: 2021-07-31T02:27:41.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[39718]: audit 2021 2021-07-31T02:27:41.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[39718]: -07-31T02:27:40 2021-07-31T02:27:41.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[39718]: .465795+0000 mon.a (mon. 2021-07-31T02:27:41.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[39718]: 0) 2014 : audit [INF] 2021-07-31T02:27:41.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[39718]: from='client.? 172.21.15.45:0/32709565' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/63/uuid4"}]: dispatch 2021-07-31T02:27:41.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[39718]: audit 2021- 2021-07-31T02:27:41.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[39718]: 07-31T02:27:40 2021-07-31T02:27:41.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[39718]: .468706+0000 mon.a (mon. 2021-07-31T02:27:41.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[39718]: 0) 2015 : audit [INF] 2021-07-31T02:27:41.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[39718]: from='client.? 172.21.15.45:0/32709565' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/63/uuid4"}]': finished 2021-07-31T02:27:41.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[39718]: audit 2021-07 2021-07-31T02:27:41.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[39718]: -31T02:27:40.470770 2021-07-31T02:27:41.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[39718]: +0000 mon.a (mon.0 2021-07-31T02:27:41.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[39718]: ) 2016 : audit [INF] from='client.? 172.21.15.45:0/32709565' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/63/6a0f8bf2-1571-4344-95ba-d9e1c08f02f4"}]: dispatch 2021-07-31T02:27:41.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[39718]: 2021-07-31T02:27:41.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[39718]: audit 2021-07-31T02:27:41.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[39718]: 2021-07-31T02: 2021-07-31T02:27:41.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[39718]: 27:40.473991+0000 mon.a (mon. 2021-07-31T02:27:41.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[39718]: 0) 2017 : audit [INF] 2021-07-31T02:27:41.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[39718]: from='client.? 172.21.15.45:0/32709565' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/63/6a0f8bf2-1571-4344-95ba-d9e1c08f02f4"}]': finished 2021-07-31T02:27:41.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[39718]: audit 2021- 2021-07-31T02:27:41.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[39718]: 07-31T02:27:40. 2021-07-31T02:27:41.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[39718]: 480937+0000 mon.a (mon.0 2021-07-31T02:27:41.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[39718]: ) 2018 : audit [INF] from='client.? 172.21.15.45:0/32709565' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/63/5977bee1-3229-463c-bf67-5b58105e83c3"}]: dispatch 2021-07-31T02:27:41.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[39718]: 2021-07-31T02:27:41.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[39718]: audit 2021 2021-07-31T02:27:41.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[39718]: -07-31T02:27: 2021-07-31T02:27:41.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[39718]: 40.484289+0000 mon.a (mon 2021-07-31T02:27:41.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[39718]: .0) 2019 : audit [INF] 2021-07-31T02:27:41.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[39718]: from='client.? 172.21.15.45:0/32709565' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/63/5977bee1-3229-463c-bf67-5b58105e83c3"}]': finished 2021-07-31T02:27:41.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[39718]: audit 2021-07-31T02:27:41.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[39718]: 2021- 2021-07-31T02:27:41.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[39718]: 07-31T02:27:40 2021-07-31T02:27:41.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[39718]: .486285+0000 mon.a (mon. 2021-07-31T02:27:41.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[39718]: 0) 2020 : audit [INF] from='client.? 172.21.15.45:0/32709565' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/63/44246c88-eca9-43fd-b59d-a12c90ac280d"}]: dispatch 2021-07-31T02:27:41.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[39718]: 2021-07-31T02:27:41.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[39718]: audit 2021- 2021-07-31T02:27:41.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[39718]: 07-31T02:27:40. 2021-07-31T02:27:41.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[39718]: 488695+0000 mon.a (mon.0) 2021-07-31T02:27:41.798 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[33655]: [DBG] from='client.? 172.21.15.45:0/32709565' entity='client.admin' cmd=[{"prefix": "config-key get", "key": "rbd/mirror/peer/63/2cc6f59e-6db2-491b-a549-fb8f3df81bde"}]: dispatch 2021-07-31T02:27:41.798 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[33655]: audit 2021- 2021-07-31T02:27:41.799 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[33655]: 07-31T02:27:40.501250+0000 mon.a ( 2021-07-31T02:27:41.799 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[33655]: mon.0) 2023 : audit 2021-07-31T02:27:41.799 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[33655]: [INF] from='client.? 172.21.15.45:0/32709565' entity='client.admin' 2021-07-31T02:27:41.799 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[33655]: audit 2021-07 2021-07-31T02:27:41.800 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[33655]: -31T02:27:40.501553+0000 mon.a (mon 2021-07-31T02:27:41.800 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[33655]: .0) 2024 : audit [DBG] 2021-07-31T02:27:41.800 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[33655]: from='client.? 172.21.15.45:0/32709565' entity='client.admin' cmd=[{"prefix": "config-key get", "key": "rbd/mirror/peer/63/2cc6f59e-6db2-491b-a549-fb8f3df81bde"}]: dispatch 2021-07-31T02:27:41.801 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[33655]: audit 2021-07 2021-07-31T02:27:41.801 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[33655]: -31T02:27:40.501856+0000 mon.a (mon.0 2021-07-31T02:27:41.801 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[33655]: ) 2025 : audit [INF] from='client.? 172.21.15.45:0/32709565' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/63/2cc6f59e-6db2-491b-a549-fb8f3df81bde"}]: dispatch 2021-07-31T02:27:41.801 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[33655]: audit 2021-07 2021-07-31T02:27:41.802 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[33655]: -31T02:27:40.504012+0000 mon.a (mon. 2021-07-31T02:27:41.802 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[33655]: 0) 2026 : audit [INF] 2021-07-31T02:27:41.802 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[33655]: from='client.? 172.21.15.45:0/32709565' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/63/2cc6f59e-6db2-491b-a549-fb8f3df81bde"}]': finished 2021-07-31T02:27:41.803 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[39718]: 2021 : audit [INF] from='client.? 172.21.15.45:0/32709565' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/63/44246c88-eca9-43fd-b59d-a12c90ac280d"}]': finished 2021-07-31T02:27:41.803 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[39718]: audit 2021 2021-07-31T02:27:41.803 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[39718]: -07-31T02:27:40 2021-07-31T02:27:41.804 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[39718]: .497720+0000 mon.a (mon. 2021-07-31T02:27:41.804 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[39718]: 0) 2022 : audit [DBG] 2021-07-31T02:27:41.804 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[39718]: from='client.? 172.21.15.45:0/32709565' entity='client.admin' cmd=[{"prefix": "config-key get", "key": "rbd/mirror/peer/63/2cc6f59e-6db2-491b-a549-fb8f3df81bde"}]: dispatch 2021-07-31T02:27:41.804 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[39718]: audit 2021 2021-07-31T02:27:41.805 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[39718]: -07-31T02:27:40 2021-07-31T02:27:41.805 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[39718]: .501250+0000 mon.a (mon. 2021-07-31T02:27:41.805 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[39718]: 0) 2023 : audit [INF] 2021-07-31T02:27:41.805 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[39718]: from='client.? 172.21.15.45:0/32709565' entity='client.admin' 2021-07-31T02:27:41.806 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[39718]: audit 2021- 2021-07-31T02:27:41.806 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[39718]: 07-31T02:27:40. 2021-07-31T02:27:41.806 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[39718]: 501553+0000 mon.a (mon.0 2021-07-31T02:27:41.806 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[39718]: ) 2024 : audit [DBG] from='client.? 172.21.15.45:0/32709565' entity='client.admin' cmd=[{"prefix": "config-key get", "key": "rbd/mirror/peer/63/2cc6f59e-6db2-491b-a549-fb8f3df81bde"}]: dispatch 2021-07-31T02:27:41.807 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[39718]: 2021-07-31T02:27:41.807 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[39718]: audit 2021-07-31T02:27:41.807 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[39718]: 2021-07-31T02: 2021-07-31T02:27:41.807 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[39718]: 27:40.501856+0000 mon.a 2021-07-31T02:27:41.807 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[39718]: (mon.0) 2025 : audit 2021-07-31T02:27:41.808 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[39718]: [INF] from='client.? 172.21.15.45:0/32709565' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/63/2cc6f59e-6db2-491b-a549-fb8f3df81bde"}]: dispatch 2021-07-31T02:27:41.808 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[39718]: audit 2021-07-31T02:27:41.808 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[39718]: 2021-07-31T02:27: 2021-07-31T02:27:41.809 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[39718]: 40.504012+0000 mon.a (mon 2021-07-31T02:27:41.809 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[39718]: .0) 2026 : audit [INF] 2021-07-31T02:27:41.809 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:41 smithi045 conmon[39718]: from='client.? 172.21.15.45:0/32709565' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/63/2cc6f59e-6db2-491b-a549-fb8f3df81bde"}]': finished 2021-07-31T02:27:41.890 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.Migration (312 ms) 2021-07-31T02:27:41.891 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.MigrationPP 2021-07-31T02:27:41.891 INFO:tasks.workunit.client.0.smithi045.stdout:using old format 2021-07-31T02:27:41.891 INFO:tasks.workunit.client.0.smithi045.stdout:using old format 2021-07-31T02:27:42.201 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.MigrationPP (310 ms) 2021-07-31T02:27:42.202 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.TestGetAccessTimestamp 2021-07-31T02:27:42.202 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:27:42.202 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.TestGetAccessTimestamp (0 ms) 2021-07-31T02:27:42.202 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.TestGetModifyTimestamp 2021-07-31T02:27:42.203 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:27:42.203 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.TestGetModifyTimestamp (0 ms) 2021-07-31T02:27:42.203 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.ZeroOverlapFlatten 2021-07-31T02:27:42.203 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:27:42.204 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.ZeroOverlapFlatten (0 ms) 2021-07-31T02:27:42.204 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.PoolMetadata 2021-07-31T02:27:42.204 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:27:42.204 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.PoolMetadata (0 ms) 2021-07-31T02:27:42.204 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.PoolMetadataPP 2021-07-31T02:27:42.205 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:27:42.205 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.PoolMetadataPP (0 ms) 2021-07-31T02:27:42.205 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.Config 2021-07-31T02:27:42.205 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:27:42.205 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.Config (0 ms) 2021-07-31T02:27:42.206 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.ConfigPP 2021-07-31T02:27:42.206 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:27:42.206 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.ConfigPP (0 ms) 2021-07-31T02:27:42.206 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.PoolStatsPP 2021-07-31T02:27:42.207 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:27:42.207 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.PoolStatsPP (0 ms) 2021-07-31T02:27:42.207 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.ImageSpec 2021-07-31T02:27:42.207 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:27:42.208 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.ImageSpec (0 ms) 2021-07-31T02:27:42.208 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.SnapRemoveWithChildMissing 2021-07-31T02:27:42.208 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:27:42.208 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.SnapRemoveWithChildMissing (0 ms) 2021-07-31T02:27:42.209 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.QuiesceWatch 2021-07-31T02:27:42.209 INFO:tasks.workunit.client.0.smithi045.stdout:using old format 2021-07-31T02:27:42.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:42 smithi154 conmon[34553]: cluster 2021-07-31T02:27:40.374964+0000 mgr.x (mgr.24542) 1737 : 2021-07-31T02:27:42.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:42 smithi154 conmon[34553]: cluster [DBG] pgmap v2167: 169 pgs: 169 active+clean; 41 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 13 KiB/s rd, 5.2 KiB/s wr, 24 op/s 2021-07-31T02:27:42.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:42 smithi154 conmon[34553]: cluster 2021-07-31T02:27:41.507687+0000 mon.a (mon.0) 2027 : cluster [DBG] osdmap e1108: 8 total, 8 up, 8 in 2021-07-31T02:27:42.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:42 smithi154 conmon[34553]: cluster 2021-07-31T02:27:42.252228+0000 mon.a (mon.0) 2028 : cluster 2021-07-31T02:27:42.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:42 smithi154 conmon[34553]: [DBG] osdmap e1109: 8 total, 8 up, 8 in 2021-07-31T02:27:42.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:42 smithi045 conmon[33655]: cluster 2021-07-31T02:27:40.374964+0000 mgr.x (mgr 2021-07-31T02:27:42.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:42 smithi045 conmon[33655]: .24542) 1737 : cluster [DBG] pgmap v2167: 169 pgs: 169 active+clean; 41 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 13 KiB/s rd, 5.2 KiB/s wr, 24 op/s 2021-07-31T02:27:42.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:42 smithi045 conmon[33655]: cluster 2021-07-31T02:27:41.507687+0000 mon.a (mon.0) 2027 : cluster [DBG] osdmap e1108: 8 total, 8 up, 8 in 2021-07-31T02:27:42.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:42 smithi045 conmon[33655]: cluster 2021-07-31T02:27:42.252228+0000 mon.a (mon.0) 2028 : cluster [DBG] 2021-07-31T02:27:42.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:42 smithi045 conmon[33655]: osdmap e1109: 8 total, 8 up, 8 in 2021-07-31T02:27:42.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:42 smithi045 conmon[39718]: cluster 2021-07-31T02:27:40.374964+0000 mgr.x (mgr.24542 2021-07-31T02:27:42.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:42 smithi045 conmon[39718]: ) 1737 : cluster [DBG] pgmap v2167: 169 pgs: 169 active+clean; 41 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 13 KiB/s rd, 5.2 KiB/s wr, 24 op/s 2021-07-31T02:27:42.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:42 smithi045 conmon[39718]: cluster 2021-07-31T02:27:41 2021-07-31T02:27:42.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:42 smithi045 conmon[39718]: .507687+0000 mon.a (mon.0) 2027 : cluster [DBG] osdmap e1108: 8 total, 8 up, 8 in 2021-07-31T02:27:42.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:42 smithi045 conmon[39718]: cluster 2021-07-31T02:27: 2021-07-31T02:27:42.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:42 smithi045 conmon[39718]: 42.252228+0000 mon.a (mon.0) 2028 : cluster [DBG] 2021-07-31T02:27:42.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:42 smithi045 conmon[39718]: osdmap e1109: 8 total, 8 up, 8 in 2021-07-31T02:27:44.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:27:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:27:43.579Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:27:44.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:27:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:27:43.579Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:27:44.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:27:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:27:43.579Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:27:44.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:44 smithi045 conmon[33655]: cluster 2021-07-31T02:27:42. 2021-07-31T02:27:44.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:44 smithi045 conmon[33655]: 375384+0000 mgr.x (mgr.24542) 1738 : cluster [DBG] pgmap v2170: 169 pgs: 169 active+clean; 41 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 13 KiB/s rd, 5.2 KiB/s wr, 24 op/s 2021-07-31T02:27:44.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:44 smithi045 conmon[33655]: cluster 2021- 2021-07-31T02:27:44.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:44 smithi045 conmon[33655]: 07-31T02:27:43.260369+0000 mon.a (mon.0) 2029 : cluster [DBG] osdmap e1110: 8 total, 8 up, 8 in 2021-07-31T02:27:44.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:44 smithi045 conmon[33655]: 2021-07-31T02:27:44.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:44 smithi045 conmon[39718]: cluster 2021-07-31T02:27:42.375384+0000 2021-07-31T02:27:44.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:44 smithi045 conmon[39718]: mgr.x (mgr.24542) 1738 : cluster [DBG] pgmap v2170: 169 pgs: 169 active+clean; 41 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 13 KiB/s rd, 5.2 KiB/s wr, 24 op/s 2021-07-31T02:27:44.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:44 smithi045 conmon[39718]: cluster 2021 2021-07-31T02:27:44.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:44 smithi045 conmon[39718]: -07-31T02:27:43.260369+0000 mon.a (mon.0) 2029 : cluster [DBG] 2021-07-31T02:27:44.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:44 smithi045 conmon[39718]: osdmap e1110: 8 total, 8 up, 8 in 2021-07-31T02:27:44.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:44 smithi154 conmon[34553]: cluster 2021-07-31T02:27:42.375384+0000 mgr.x (mgr.24542) 1738 : cluster [DBG] 2021-07-31T02:27:44.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:44 smithi154 conmon[34553]: pgmap v2170: 169 pgs: 169 active+clean; 41 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 13 KiB/s rd, 5.2 KiB/s wr, 24 op/s 2021-07-31T02:27:44.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:44 smithi154 conmon[34553]: cluster 2021-07-31T02:27:43.260369+0000 mon.a (mon.0) 2029 : cluster [DBG] osdmap e1110: 8 total, 8 up, 8 in 2021-07-31T02:27:45.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:45 smithi045 conmon[33655]: cluster 2021-07-31T02:27:44.261870+0000 mon.a (mon 2021-07-31T02:27:45.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:45 smithi045 conmon[33655]: .0) 2030 : cluster [DBG] osdmap e1111: 8 total, 8 up, 8 in 2021-07-31T02:27:45.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:45 smithi045 conmon[39718]: cluster 2021-07-31T02:27:45.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:45 smithi045 conmon[39718]: 2021-07- 2021-07-31T02:27:45.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:45 smithi045 conmon[39718]: 31T02:27:44.261870+0000 mon.a (mon.0) 2030 : cluster [DBG] osdmap e1111: 8 total, 8 up, 8 in 2021-07-31T02:27:45.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:45 smithi154 conmon[34553]: cluster 2021-07-31T02:27:45.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:45 smithi154 conmon[34553]: 2021-07-31T02:27:44.261870+0000 mon.a (mon. 2021-07-31T02:27:45.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:45 smithi154 conmon[34553]: 0) 2030 : cluster [DBG] osdmap e1111: 8 total, 8 up, 8 in 2021-07-31T02:27:46.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:46 smithi045 conmon[39718]: cluster 2021-07-31T02:27:44.376029+0000 mgr.x (mgr. 2021-07-31T02:27:46.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:46 smithi045 conmon[39718]: 24542) 1739 : cluster [DBG] pgmap v2173: 169 pgs: 169 active+clean; 49 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 444 KiB/s rd, 2.0 MiB/s wr, 603 op/s 2021-07-31T02:27:46.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:46 smithi045 conmon[39718]: cluster 2021-07-31T02:27:45.272468+0000 mon.a (mon.0 2021-07-31T02:27:46.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:46 smithi045 conmon[39718]: ) 2031 : cluster [DBG] osdmap e1112: 8 total, 8 up, 8 in 2021-07-31T02:27:46.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:46 smithi045 conmon[33655]: cluster 2021- 2021-07-31T02:27:46.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:46 smithi045 conmon[33655]: 07-31T02:27:44.376029+0000 mgr.x (mgr.24542) 1739 : cluster [DBG] pgmap v2173: 169 pgs: 169 active+clean; 49 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 444 KiB/s rd, 2.0 MiB/s wr, 603 op/s 2021-07-31T02:27:46.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:46 smithi045 conmon[33655]: cluster 2021-07-31T 2021-07-31T02:27:46.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:46 smithi045 conmon[33655]: 02:27:45.272468+0000 mon.a (mon.0) 2021-07-31T02:27:46.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:46 smithi045 conmon[33655]: 2031 : cluster [DBG] osdmap e1112: 8 total, 8 up, 8 in 2021-07-31T02:27:46.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:46 smithi154 conmon[34553]: cluster 2021-07-31T02:27:44.376029+0000 mgr.x (mgr.24542) 1739 : cluster [DBG] pgmap v2173: 169 pgs: 169 active+clean; 49 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 444 KiB/s rd, 2.0 MiB/s wr, 603 op/s 2021-07-31T02:27:46.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:46 smithi154 conmon[34553]: cluster 2021-07-31T02:27:45.272468+0000 2021-07-31T02:27:46.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:46 smithi154 conmon[34553]: mon.a (mon.0) 2031 : cluster [DBG] osdmap e1112: 8 total, 8 up, 8 in 2021-07-31T02:27:47.314 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.QuiesceWatch (5113 ms) 2021-07-31T02:27:47.314 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.QuiesceWatchPP 2021-07-31T02:27:47.314 INFO:tasks.workunit.client.0.smithi045.stdout:using old format 2021-07-31T02:27:47.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:47 smithi154 conmon[34553]: cluster 2021-07-31T02:27:46.278435+0000 mon.a (mon.0) 2021-07-31T02:27:47.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:47 smithi154 conmon[34553]: 2032 : cluster [DBG] osdmap e1113: 8 total, 8 up, 8 in 2021-07-31T02:27:47.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:47 smithi045 conmon[33655]: cluster 2021-07-31T02:27:46.278435+0000 mon.a (mon.0) 2032 2021-07-31T02:27:47.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:47 smithi045 conmon[33655]: : cluster [DBG] osdmap e1113: 8 total, 8 up, 8 in 2021-07-31T02:27:47.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:47 smithi045 conmon[39718]: cluster 2021-07-31T02:27:47.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:47 smithi045 conmon[39718]: 2021-07-31T02:27:47.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:47 smithi045 conmon[39718]: 2021-07-31T02:27:46.278435+0000 2021-07-31T02:27:47.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:47 smithi045 conmon[39718]: mon.a (mon.0) 2032 : cluster [DBG] osdmap e1113: 8 total, 8 up, 8 in 2021-07-31T02:27:48.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:48 smithi154 conmon[34553]: cluster 2021-07-31T02:27:46.376595+0000 mgr.x (mgr.24542) 1740 : cluster [DBG] pgmap v2176: 169 pgs: 169 active+clean; 49 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 444 KiB/s rd, 2.0 MiB/s wr, 603 op/s 2021-07-31T02:27:48.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:48 smithi154 conmon[34553]: cluster 2021-07-31T02:27:47 2021-07-31T02:27:48.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:48 smithi154 conmon[34553]: .284061+0000 mon.a (mon.0) 2033 : cluster [DBG] osdmap e1114: 8 total, 8 up, 8 in 2021-07-31T02:27:48.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:48 smithi045 conmon[39718]: cluster 2021-07-31T02:27:46.376595+0000 mgr.x (mgr.24542) 1740 : cluster 2021-07-31T02:27:48.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:48 smithi045 conmon[39718]: [DBG] pgmap v2176: 169 pgs: 169 active+clean; 49 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 444 KiB/s rd, 2.0 MiB/s wr, 603 op/s 2021-07-31T02:27:48.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:48 smithi045 conmon[39718]: cluster 2021-07-31T02:27:47.284061+0000 mon.a (mon.0) 2033 : cluster [DBG] osdmap e1114: 8 total, 8 up, 8 in 2021-07-31T02:27:48.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:48 smithi045 conmon[33655]: cluster 2021-07-31T02:27:46.376595+0000 mgr.x (mgr.24542 2021-07-31T02:27:48.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:48 smithi045 conmon[33655]: ) 1740 : cluster [DBG] pgmap v2176: 169 pgs: 169 active+clean; 49 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 444 KiB/s rd, 2.0 MiB/s wr, 603 op/s 2021-07-31T02:27:48.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:48 smithi045 conmon[33655]: cluster 2021-07-31T02:27:47.284061+0000 mon.a (mon.0) 2033 : cluster [DBG] osdmap e1114: 8 total, 8 up, 8 in 2021-07-31T02:27:49.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:49 smithi154 conmon[34553]: cluster 2021-07-31T02:27:49.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:49 smithi154 conmon[34553]: 2021-07-31T02:27:48.289343+0000 2021-07-31T02:27:49.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:49 smithi154 conmon[34553]: mon.a (mon.0) 2034 : cluster [DBG] osdmap e1115: 8 total, 8 up, 8 in 2021-07-31T02:27:49.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:49 smithi045 conmon[33655]: cluster 2021-07-31T02:27:48.289343+0000 mon.a (mon.0) 2021-07-31T02:27:49.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:49 smithi045 conmon[33655]: 2034 : cluster [DBG] osdmap e1115: 8 total, 8 up, 8 in 2021-07-31T02:27:49.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:49 smithi045 conmon[39718]: cluster 2021-07-31T02:27:49.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:49 smithi045 conmon[39718]: 2021-07-31T02: 2021-07-31T02:27:49.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:49 smithi045 conmon[39718]: 27:48.289343+0000 mon.a (mon.0) 2034 : cluster [DBG] osdmap e1115: 8 total, 8 up, 8 in 2021-07-31T02:27:50.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:50 smithi154 conmon[34553]: cluster 2021-07-31T02:27:48.377147+0000 mgr.x (mgr.24542) 1741 : cluster [DBG] pgmap v2179: 169 pgs: 169 active+clean; 49 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:27:50.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:50 smithi154 conmon[34553]: cluster 2021-07-31T02:27:49.295136+0000 mon.a (mon.0) 2035 : cluster [DBG] osdmap e1116: 8 total, 8 up, 8 in 2021-07-31T02:27:50.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:50 smithi045 conmon[39718]: cluster 2021-07-31T02:27:48.377147+0000 mgr.x (mgr 2021-07-31T02:27:50.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:50 smithi045 conmon[39718]: .24542) 1741 : cluster [DBG] pgmap v2179: 169 pgs: 169 active+clean; 49 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:27:50.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:50 smithi045 conmon[39718]: cluster 2021-07-31T02:27:49.295136+0000 mon.a 2021-07-31T02:27:50.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:50 smithi045 conmon[39718]: (mon.0) 2035 : cluster [DBG] osdmap e1116: 8 total, 8 up, 8 in 2021-07-31T02:27:50.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:50 smithi045 conmon[33655]: cluster 2021-07-31T02:27:48.377147+0000 mgr.x (mgr.24542 2021-07-31T02:27:50.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:50 smithi045 conmon[33655]: ) 1741 : cluster [DBG] pgmap v2179: 169 pgs: 169 active+clean; 49 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:27:50.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:50 smithi045 conmon[33655]: cluster 2021-07-31T02:27:49.295136+0000 mon.a (mon.0) 2035 : cluster [DBG] osdmap e1116: 8 total, 8 up, 8 in 2021-07-31T02:27:51.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:51 smithi154 conmon[34553]: cluster 2021-07-31T02:27:51.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:51 smithi154 conmon[34553]: 2021-07-31T02:27:50.300940 2021-07-31T02:27:51.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:51 smithi154 conmon[34553]: +0000 mon.a (mon.0) 2036 : cluster [DBG] osdmap e1117: 8 total, 8 up, 8 in 2021-07-31T02:27:51.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:51 smithi045 conmon[33655]: cluster 2021-07-31T02:27:50.300940+0000 mon.a (mon.0) 2036 : cluster [DBG] 2021-07-31T02:27:51.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:51 smithi045 conmon[33655]: osdmap e1117: 8 total, 8 up, 8 in 2021-07-31T02:27:51.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:51 smithi045 conmon[39718]: cluster 2021-07-31T02:27:50.300940+0000 mon.a (mon.0) 2036 : cluster 2021-07-31T02:27:51.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:51 smithi045 conmon[39718]: [DBG] osdmap e1117: 8 total, 8 up, 8 in 2021-07-31T02:27:52.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:52 smithi154 conmon[34553]: cluster 2021-07-31T02:27:50.377683+0000 mgr.x (mgr.24542) 1742 : cluster [DBG] pgmap v2182: 169 pgs: 169 active+clean; 49 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 8.0 KiB/s rd, 2.0 KiB/s wr, 13 op/s 2021-07-31T02:27:52.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:52 smithi154 conmon[34553]: cluster 2021-07-31T02:27:51.306981+0000 mon.a (mon.0) 2037 : cluster [DBG] osdmap e1118: 8 total, 8 up, 8 in 2021-07-31T02:27:52.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:52 smithi154 conmon[34553]: cluster 2021-07-31T02:27:52.256495+0000 mon.a (mon.0) 2038 : cluster [DBG] osdmap e1119: 8 total, 8 up, 8 in 2021-07-31T02:27:52.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:52 smithi045 conmon[39718]: cluster 2021-07-31T02:27:50.377683+0000 mgr.x (mgr.24542) 1742 2021-07-31T02:27:52.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:52 smithi045 conmon[39718]: : cluster [DBG] pgmap v2182: 169 pgs: 169 active+clean; 49 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 8.0 KiB/s rd, 2.0 KiB/s wr, 13 op/s 2021-07-31T02:27:52.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:52 smithi045 conmon[39718]: cluster 2021-07-31T02:27:51.306981+0000 mon.a (mon.0) 2037 : cluster 2021-07-31T02:27:52.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:52 smithi045 conmon[39718]: [DBG] osdmap e1118: 8 total, 8 up, 8 in 2021-07-31T02:27:52.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:52 smithi045 conmon[39718]: cluster 2021-07-31T02:27:52.256495+0000 2021-07-31T02:27:52.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:52 smithi045 conmon[39718]: mon.a (mon.0) 2038 : cluster [DBG] osdmap e1119: 8 total, 8 up, 8 in 2021-07-31T02:27:52.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:52 smithi045 conmon[33655]: cluster 2021-07-31T02:27:50.377683+0000 mgr.x (mgr.24542) 1742 2021-07-31T02:27:52.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:52 smithi045 conmon[33655]: : cluster [DBG] pgmap v2182: 169 pgs: 169 active+clean; 49 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 8.0 KiB/s rd, 2.0 KiB/s wr, 13 op/s 2021-07-31T02:27:52.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:52 smithi045 conmon[33655]: cluster 2021-07-31T02:27:51.306981+0000 mon.a (mon.0) 2037 : cluster [DBG] osdmap e1118: 8 total, 8 up, 8 in 2021-07-31T02:27:52.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:52 smithi045 conmon[33655]: cluster 2021 2021-07-31T02:27:52.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:52 smithi045 conmon[33655]: -07-31T02:27:52.256495+0000 mon.a (mon.0) 2038 : cluster [DBG] osdmap e1119: 8 total, 8 up, 8 in 2021-07-31T02:27:53.296 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.QuiesceWatchPP (5983 ms) 2021-07-31T02:27:53.296 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.QuiesceWatchError 2021-07-31T02:27:53.296 INFO:tasks.workunit.client.0.smithi045.stdout:using old format 2021-07-31T02:27:54.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:27:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:27:53.579Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:27:54.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:27:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:27:53.579Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:27:54.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:27:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:27:53.579Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:27:54.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:54 smithi045 conmon[39718]: cluster 2021-07-31T02:27:52.378072+0000 mgr.x (mgr 2021-07-31T02:27:54.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:54 smithi045 conmon[39718]: .24542) 1743 : cluster [DBG] pgmap v2185: 169 pgs: 169 active+clean; 49 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 8.0 KiB/s rd, 2.0 KiB/s wr, 13 op/s 2021-07-31T02:27:54.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:54 smithi045 conmon[39718]: cluster 2021-07-31T02:27:53.264223+0000 mon.a (mon 2021-07-31T02:27:54.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:54 smithi045 conmon[39718]: .0) 2039 : cluster [DBG] osdmap e1120: 8 total, 8 up, 8 in 2021-07-31T02:27:54.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:54 smithi045 conmon[33655]: cluster 2021-07-31T02:27:52.378072+0000 mgr.x (mgr.24542) 1743 : 2021-07-31T02:27:54.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:54 smithi045 conmon[33655]: cluster [DBG] pgmap v2185: 169 pgs: 169 active+clean; 49 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 8.0 KiB/s rd, 2.0 KiB/s wr, 13 op/s 2021-07-31T02:27:54.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:54 smithi045 conmon[33655]: cluster 2021-07-31T02:27:53.264223+0000 mon.a (mon.0) 2039 : cluster [DBG] osdmap e1120: 8 total, 8 up, 8 in 2021-07-31T02:27:54.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:54 smithi154 conmon[34553]: cluster 2021-07-31T02:27:52.378072+0000 mgr.x (mgr.24542) 1743 : cluster [DBG] pgmap v2185: 169 pgs: 169 active+clean; 49 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 8.0 KiB/s rd, 2.0 KiB/s wr, 13 op/s 2021-07-31T02:27:54.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:54 smithi154 conmon[34553]: cluster 2021-07-31T02:27:53.264223+0000 mon.a (mon.0) 2039 : cluster [DBG] osdmap e1120: 8 total, 8 up, 8 in 2021-07-31T02:27:55.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:55 smithi154 conmon[34553]: cluster 2021-07-31T02:27:55.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:55 smithi154 conmon[34553]: 2021-07 2021-07-31T02:27:55.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:55 smithi154 conmon[34553]: -31T02:27:54.310900+0000 mon.a (mon.0) 2040 : cluster [DBG] osdmap e1121: 8 total, 8 up, 8 in 2021-07-31T02:27:55.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:55 smithi045 conmon[39718]: cluster 2021-07-31T02:27:55.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:55 smithi045 conmon[39718]: 2021-07-31T02:27:55.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:55 smithi045 conmon[39718]: 2021 2021-07-31T02:27:55.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:55 smithi045 conmon[39718]: - 2021-07-31T02:27:55.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:55 smithi045 conmon[39718]: 07 2021-07-31T02:27:55.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:55 smithi045 conmon[39718]: -31T02:27:54.310900+0000 mon.a (mon. 2021-07-31T02:27:55.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:55 smithi045 conmon[39718]: 0) 2040 : cluster [DBG] osdmap e1121: 8 total, 8 up, 8 in 2021-07-31T02:27:55.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:55 smithi045 conmon[33655]: cluster 2021-07-31T02:27:55.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:55 smithi045 conmon[33655]: 2021-07-31T02:27:55.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:55 smithi045 conmon[33655]: 2021-07-31T02:27:54.310900+0000 2021-07-31T02:27:55.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:55 smithi045 conmon[33655]: mon.a (mon.0) 2040 : cluster [DBG] osdmap e1121: 8 total, 8 up, 8 in 2021-07-31T02:27:56.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:56 smithi154 conmon[34553]: cluster 2021-07-31T02:27:54.378580+0000 mgr.x (mgr.24542) 1744 : cluster [DBG] pgmap v2188: 169 pgs: 169 active+clean; 49 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 2.0 KiB/s wr, 13 op/s 2021-07-31T02:27:56.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:56 smithi154 conmon[34553]: 2021-07-31T02:27:56.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:56 smithi154 conmon[34553]: cluster 2021-07-31T02:27:55.317149+0000 mon.a (mon.0) 2041 : cluster [DBG] osdmap e1122: 8 total, 8 up, 8 in 2021-07-31T02:27:56.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:56 smithi045 conmon[33655]: cluster 2021-07-31T02:27:54.378580+0000 mgr.x (mgr.24542) 1744 : cluster [DBG] pgmap v2188: 169 pgs: 169 active+clean; 49 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 2.0 KiB/s wr, 13 op/s 2021-07-31T02:27:56.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:56 smithi045 conmon[33655]: cluster 2021-07-31T02:27:55.317149+0000 mon.a (mon.0) 2041 : cluster [DBG] osdmap e1122: 8 total, 8 up, 8 in 2021-07-31T02:27:56.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:56 smithi045 conmon[39718]: cluster 2021-07-31T02:27:54.378580+0000 mgr.x ( 2021-07-31T02:27:56.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:56 smithi045 conmon[39718]: mgr.24542) 1744 : cluster [DBG] pgmap v2188: 169 pgs: 169 active+clean; 49 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 2.0 KiB/s wr, 13 op/s 2021-07-31T02:27:56.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:56 smithi045 conmon[39718]: cluster 2021-07-31T02:27:55.317149+0000 mon.a (mon. 2021-07-31T02:27:56.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:56 smithi045 conmon[39718]: 0) 2041 : cluster [DBG] osdmap e1122: 8 total, 8 up, 8 in 2021-07-31T02:27:57.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:57 smithi154 conmon[34553]: cluster 2021-07-31T02:27:56.323328+0000 mon.a (mon.0) 2042 : cluster [DBG] osdmap e1123: 8 total, 8 up, 8 in 2021-07-31T02:27:57.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:57 smithi154 conmon[34553]: cluster 2021-07-31T02: 2021-07-31T02:27:57.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:57 smithi154 conmon[34553]: 27:57.258729+0000 mon.a (mon.0) 2043 : cluster [DBG] osdmap e1124: 8 total, 8 up, 8 in 2021-07-31T02:27:57.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:57 smithi045 conmon[39718]: cluster 2021-07-31T02:27:56.323328+0000 mon.a (mon.0 2021-07-31T02:27:57.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:57 smithi045 conmon[39718]: ) 2042 : cluster [DBG] osdmap e1123: 8 total, 8 up, 8 in 2021-07-31T02:27:57.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:57 smithi045 conmon[39718]: cluster 2021-07-31T02:27:57.258729+0000 mon.a (mon.0) 2043 : cluster 2021-07-31T02:27:57.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:57 smithi045 conmon[39718]: [DBG] osdmap e1124: 8 total, 8 up, 8 in 2021-07-31T02:27:57.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:57 smithi045 conmon[33655]: cluster 2021-07-31T02:27:56.323328 2021-07-31T02:27:57.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:57 smithi045 conmon[33655]: +0000 mon.a (mon.0) 2042 : cluster [DBG] osdmap e1123: 8 total, 8 up, 8 in 2021-07-31T02:27:57.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:57 smithi045 conmon[33655]: cluster 2021 2021-07-31T02:27:57.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:57 smithi045 conmon[33655]: -07-31T02:27:57.258729+0000 mon.a (mon. 2021-07-31T02:27:57.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:57 smithi045 conmon[33655]: 0) 2043 : cluster [DBG] osdmap e1124: 8 total, 8 up, 8 in 2021-07-31T02:27:58.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:58 smithi154 conmon[34553]: cluster 2021-07-31T02:27:56.379127+0000 mgr.x (mgr.24542) 1745 : cluster [DBG] pgmap v2191: 169 pgs: 169 active+clean; 49 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 2.0 KiB/s wr, 13 op/s 2021-07-31T02:27:58.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:27:58 smithi154 conmon[34553]: cluster 2021-07-31T02:27:58.265996+0000 mon.a (mon.0) 2044 : cluster [DBG] osdmap e1125: 8 total, 8 up, 8 in 2021-07-31T02:27:58.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:58 smithi045 conmon[39718]: cluster 2021-07-31T02:27:56.379127+0000 mgr.x (mgr. 2021-07-31T02:27:58.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:58 smithi045 conmon[39718]: 24542) 1745 : cluster [DBG] pgmap v2191: 169 pgs: 169 active+clean; 49 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 2.0 KiB/s wr, 13 op/s 2021-07-31T02:27:58.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:58 smithi045 conmon[39718]: cluster 2021-07-31T02:27:58.265996+0000 mon.a (mon.0 2021-07-31T02:27:58.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:27:58 smithi045 conmon[39718]: ) 2044 : cluster [DBG] osdmap e1125: 8 total, 8 up, 8 in 2021-07-31T02:27:58.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:58 smithi045 conmon[33655]: cluster 2021-07-31T02:27:56.379127+0000 mgr.x (mgr.24542) 1745 2021-07-31T02:27:58.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:58 smithi045 conmon[33655]: : cluster [DBG] pgmap v2191: 169 pgs: 169 active+clean; 49 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 2.0 KiB/s wr, 13 op/s 2021-07-31T02:27:58.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:27:58 smithi045 conmon[33655]: cluster 2021-07-31T02:27:58.265996+0000 mon.a (mon.0) 2044 : cluster [DBG] osdmap e1125: 8 total, 8 up, 8 in 2021-07-31T02:27:59.300 INFO:tasks.workunit.client.0.smithi045.stdout:10.00% done 2021-07-31T02:27:59.300 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.QuiesceWatchError (6003 ms) 2021-07-31T02:27:59.301 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.QuiesceWatchTimeout 2021-07-31T02:27:59.301 INFO:tasks.workunit.client.0.smithi045.stdout:using old format 2021-07-31T02:27:59.310 INFO:tasks.workunit.client.0.smithi045.stderr:test quiesce is not long enough to time out 2021-07-31T02:28:00.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:00 smithi045 conmon[33655]: cluster 2021-07-31T02:27:58.379638+0000 2021-07-31T02:28:00.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:00 smithi045 conmon[33655]: mgr.x (mgr.24542) 1746 : cluster [DBG] pgmap v2194: 169 pgs: 169 active+clean; 49 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:28:00.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:00 smithi045 conmon[33655]: cluster 2021-07-31T02:27 2021-07-31T02:28:00.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:00 smithi045 conmon[33655]: :59.268228+0000 mon.a (mon.0) 2045 : cluster [DBG] 2021-07-31T02:28:00.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:00 smithi045 conmon[33655]: osdmap e1126: 8 total, 8 up, 8 in 2021-07-31T02:28:00.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:00 smithi045 conmon[39718]: cluster 2021-07-31T02:27:58. 2021-07-31T02:28:00.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:00 smithi045 conmon[39718]: 379638+0000 mgr.x (mgr.24542) 1746 : cluster [DBG] pgmap v2194: 169 pgs: 169 active+clean; 49 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:28:00.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:00 smithi045 conmon[39718]: cluster 2021-07-31T 2021-07-31T02:28:00.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:00 smithi045 conmon[39718]: 02:27:59.268228+0000 mon.a (mon.0) 2045 : 2021-07-31T02:28:00.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:00 smithi045 conmon[39718]: cluster [DBG] osdmap e1126: 8 total, 8 up, 8 in 2021-07-31T02:28:00.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:00 smithi154 conmon[34553]: cluster 2021-07-31T02:27:58.379638+0000 mgr.x (mgr.24542) 1746 : cluster [DBG] pgmap v2194: 169 pgs: 169 active+clean; 49 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:28:00.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:00 smithi154 conmon[34553]: cluster 2021-07-31T02:27:59.268228+0000 mon.a (mon.0) 2045 : cluster [DBG] osdmap e1126: 8 total, 8 up, 8 in 2021-07-31T02:28:02.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:02 smithi045 conmon[33655]: cluster 2021-07-31T02:28:00.380240+0000 mgr.x (mgr.24542) 1747 : cluster [DBG] pgmap v2196: 169 pgs: 169 active+clean; 49 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2021-07-31T02:28:02.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:02 smithi045 conmon[33655]: cluster 2021-07-31T02:28:02.259507+0000 mon.a (mon.0) 2046 : cluster [DBG] osdmap e1127: 8 total, 8 up, 8 in 2021-07-31T02:28:02.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:02 smithi045 conmon[39718]: cluster 2021-07-31T02:28:00.380240+0000 mgr.x (mgr.24542) 1747 2021-07-31T02:28:02.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:02 smithi045 conmon[39718]: : cluster [DBG] pgmap v2196: 169 pgs: 169 active+clean; 49 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2021-07-31T02:28:02.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:02 smithi045 conmon[39718]: cluster 2021-07-31T02:28:02.259507+0000 mon.a (mon.0) 2046 2021-07-31T02:28:02.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:02 smithi045 conmon[39718]: : cluster [DBG] osdmap e1127: 8 total, 8 up, 8 in 2021-07-31T02:28:02.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:02 smithi154 conmon[34553]: cluster 2021-07-31T02:28:00.380240+0000 mgr.x (mgr.24542) 1747 : cluster [DBG] pgmap v2196: 169 pgs: 169 active+clean; 49 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2021-07-31T02:28:02.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:02 smithi154 conmon[34553]: cluster 2021- 2021-07-31T02:28:02.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:02 smithi154 conmon[34553]: 07-31T02:28:02.259507+0000 mon.a (mon.0) 2046 : cluster [DBG] osdmap e1127: 8 total, 8 up, 8 in 2021-07-31T02:28:04.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:28:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:28:03.580Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:28:04.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:28:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:28:03.580Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:28:04.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:28:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:28:03.580Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:28:04.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:04 smithi045 conmon[33655]: cluster 2021-07-31T02:28:02.380705+0000 mgr.x (mgr.24542 2021-07-31T02:28:04.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:04 smithi045 conmon[33655]: ) 1748 : cluster [DBG] pgmap v2198: 169 pgs: 169 active+clean; 49 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.3 KiB/s rd, 999 B/s wr, 7 op/s 2021-07-31T02:28:04.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:04 smithi045 conmon[39718]: cluster 2021-07-31T02:28:02.380705+0000 mgr.x (mgr. 2021-07-31T02:28:04.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:04 smithi045 conmon[39718]: 24542) 1748 : cluster [DBG] pgmap v2198: 169 pgs: 169 active+clean; 49 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.3 KiB/s rd, 999 B/s wr, 7 op/s 2021-07-31T02:28:04.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:04 smithi154 conmon[34553]: cluster 2021-07-31T02:28:02.380705+0000 mgr.x (mgr.24542) 1748 : cluster [DBG] 2021-07-31T02:28:04.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:04 smithi154 conmon[34553]: pgmap v2198: 169 pgs: 169 active+clean; 49 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.3 KiB/s rd, 999 B/s wr, 7 op/s 2021-07-31T02:28:06.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:06 smithi154 conmon[34553]: cluster 2021-07-31T02:28:04.381619+0000 mgr.x (mgr.24542) 1749 : cluster [DBG] pgmap v2199: 169 pgs: 169 active+clean; 49 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.9 KiB/s rd, 1.5 KiB/s wr, 9 op/s 2021-07-31T02:28:06.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:06 smithi154 conmon[34553]: 2021-07-31T02:28:06.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:06 smithi045 conmon[33655]: cluster 2021-07-31T02:28:04.381619+0000 mgr.x (mgr.24542) 1749 : cluster [DBG] pgmap v2199: 169 pgs: 169 active+clean; 49 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.9 KiB/s rd, 1.5 KiB/s wr, 9 op/s 2021-07-31T02:28:06.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:06 smithi045 conmon[33655]: 2021-07-31T02:28:06.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:06 smithi045 conmon[39718]: cluster 2021-07-31T02:28:04.381619+0000 mgr.x 2021-07-31T02:28:06.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:06 smithi045 conmon[39718]: (mgr.24542) 1749 : cluster [DBG] pgmap v2199: 169 pgs: 169 active+clean; 49 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.9 KiB/s rd, 1.5 KiB/s wr, 9 op/s 2021-07-31T02:28:08.291 INFO:tasks.workunit.client.0.smithi045.stderr:test quiesce is timed out 2021-07-31T02:28:08.291 INFO:tasks.workunit.client.0.smithi045.stderr:waiting for timed out ... 0 2021-07-31T02:28:08.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:08 smithi045 conmon[33655]: cluster 2021-07-31T02:28:06.382150+0000 mgr.x 2021-07-31T02:28:08.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:08 smithi045 conmon[33655]: (mgr.24542) 1750 : cluster [DBG] pgmap v2200: 169 pgs: 169 active+clean; 49 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 4.5 KiB/s rd, 1.1 KiB/s wr, 7 op/s 2021-07-31T02:28:08.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:08 smithi045 conmon[33655]: cluster 2021-07-31T02:28:07.261015+0000 mon.a (mon.0) 2047 : cluster [DBG] osdmap e1128: 8 total, 8 up, 8 in 2021-07-31T02:28:08.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:08 smithi045 conmon[39718]: cluster 2021-07-31T02:28:06 2021-07-31T02:28:08.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:08 smithi045 conmon[39718]: .382150+0000 mgr.x (mgr.24542) 1750 : cluster [DBG] pgmap v2200: 169 pgs: 169 active+clean; 49 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 4.5 KiB/s rd, 1.1 KiB/s wr, 7 op/s 2021-07-31T02:28:08.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:08 smithi045 conmon[39718]: cluster 2021-07-31T02:28:08.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:08 smithi045 conmon[39718]: 2021-07-31T 2021-07-31T02:28:08.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:08 smithi045 conmon[39718]: 02:28:07.261015 2021-07-31T02:28:08.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:08 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2021-07-31T02:28:08.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:08 smithi045 conmon[39718]: 2047 : cluster [DBG] osdmap e1128: 8 total, 8 up, 8 in 2021-07-31T02:28:08.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:08 smithi154 conmon[34553]: cluster 2021-07-31T02:28:06.382150+0000 mgr.x (mgr.24542) 1750 : cluster [DBG] pgmap v2200: 169 pgs: 169 active+clean; 49 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 4.5 KiB/s rd, 1.1 KiB/s wr, 7 op/s 2021-07-31T02:28:08.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:08 smithi154 conmon[34553]: cluster 2021-07-31T02:28:07.261015+0000 mon.a (mon.0) 2047 : cluster [DBG] osdmap e1128: 8 total, 8 up, 8 in 2021-07-31T02:28:09.292 INFO:tasks.workunit.client.0.smithi045.stderr:waiting for timed out ... 1 2021-07-31T02:28:09.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:09 smithi154 conmon[34553]: cluster 2021-07-31T02:28:08.284035+0000 mon.a (mon.0) 2048 : cluster 2021-07-31T02:28:09.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:09 smithi154 conmon[34553]: [DBG] osdmap e1129: 8 total, 8 up, 8 in 2021-07-31T02:28:09.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:09 smithi045 conmon[39718]: cluster 2021-07-31T02:28:08.284035+0000 mon.a 2021-07-31T02:28:09.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:09 smithi045 conmon[39718]: (mon.0) 2048 : cluster [DBG] osdmap e1129: 8 total, 8 up, 8 in 2021-07-31T02:28:09.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:09 smithi045 conmon[33655]: cluster 2021-07-31T02:28:08.284035+0000 mon.a (mon.0) 2048 : cluster [DBG] 2021-07-31T02:28:09.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:09 smithi045 conmon[33655]: osdmap e1129: 8 total, 8 up, 8 in 2021-07-31T02:28:10.024 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:28:10.292 INFO:tasks.workunit.client.0.smithi045.stderr:waiting for timed out ... 2 2021-07-31T02:28:10.302 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:28:10.303 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 43m ago 43m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:28:10.303 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 40m ago 43m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:28:10.304 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 40m ago 47m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:28:10.304 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 43m ago 49m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:28:10.304 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 43m ago 49m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:28:10.304 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 40m ago 47m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:28:10.305 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 43m ago 47m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:28:10.305 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 43m ago 44m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:28:10.305 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 40m ago 43m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:28:10.305 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 43m ago 46m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:28:10.306 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 43m ago 46m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:28:10.306 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 43m ago 46m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:28:10.306 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 43m ago 45m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:28:10.306 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 40m ago 45m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:28:10.307 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 40m ago 45m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:28:10.307 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 40m ago 44m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:28:10.307 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 40m ago 44m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:28:10.308 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 40m ago 44m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:28:10.644 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:28:10.644 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:28:10.644 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:28:10.645 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:28:10.645 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:28:10.645 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:28:10.645 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:28:10.646 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:28:10.646 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:28:10.646 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:28:10.646 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:28:10.647 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:28:10.647 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:28:10.647 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:28:10.648 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:28:10.648 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:28:10.648 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:28:10.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:10 smithi154 conmon[34553]: cluster 2021-07-31T02:28:08.382688+0000 mgr.x (mgr.24542) 1751 : cluster [DBG] pgmap v2203: 169 pgs: 169 active+clean; 49 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 1.5 KiB/s rd, 668 B/s wr, 3 op/s 2021-07-31T02:28:10.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:10 smithi154 conmon[34553]: 2021-07-31T02:28:10.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:10 smithi045 conmon[39718]: cluster 2021-07-31T02:28:08.382688 2021-07-31T02:28:10.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:10 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 1751 : cluster [DBG] pgmap v2203: 169 pgs: 169 active+clean; 49 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 1.5 KiB/s rd, 668 B/s wr, 3 op/s 2021-07-31T02:28:10.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:10 smithi045 conmon[33655]: cluster 2021-07-31T02:28:08.382688+0000 mgr.x (mgr. 2021-07-31T02:28:10.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:10 smithi045 conmon[33655]: 24542) 1751 : cluster [DBG] pgmap v2203: 169 pgs: 169 active+clean; 49 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 1.5 KiB/s rd, 668 B/s wr, 3 op/s 2021-07-31T02:28:11.293 INFO:tasks.workunit.client.0.smithi045.stderr:waiting for timed out ... 3 2021-07-31T02:28:11.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:11 smithi154 conmon[34553]: audit 2021-07-31T02:28:10.013602+0000 mgr.x (mgr.24542) 1752 : audit [DBG] 2021-07-31T02:28:11.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:11 smithi154 conmon[34553]: from='client.17133 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:28:11.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:11 smithi154 conmon[34553]: audit 2021-07-31T02:28:10.294515+0000 mgr.x (mgr.24542) 1753 : audit [DBG] from='client.17139 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:28:11.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:11 smithi154 conmon[34553]: audit 2021 2021-07-31T02:28:11.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:11 smithi154 conmon[34553]: -07-31T02:28:10.644488+0000 mon.a (mon.0) 2049 : audit [DBG] from='client.? 172.21.15.45:0/1044743630' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:28:11.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:11 smithi045 conmon[33655]: audit 2021-07-31T02:28:10.013602+0000 mgr.x (mgr.24542) 1752 : 2021-07-31T02:28:11.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:11 smithi045 conmon[33655]: audit [DBG] from='client.17133 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:28:11.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:11 smithi045 conmon[33655]: audit 2021-07-31T02:28:10.294515+0000 mgr.x (mgr.24542) 1753 : audit [DBG] from='client.17139 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:28:11.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:11 smithi045 conmon[33655]: audit 2021-07-31T02:28:10.644488+0000 mon.a (mon.0) 2049 : audit [DBG] 2021-07-31T02:28:11.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:11 smithi045 conmon[33655]: from='client.? 172.21.15.45:0/1044743630' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:28:11.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:11 smithi045 conmon[39718]: audit 2021-07-31T02:28:10.013602+0000 mgr.x (mgr 2021-07-31T02:28:11.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:11 smithi045 conmon[39718]: .24542) 1752 : audit [DBG] from='client.17133 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:28:11.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:11 smithi045 conmon[39718]: audit 2021-07-31T02:28:10.294515 2021-07-31T02:28:11.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:11 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 1753 : audit [DBG] from='client.17139 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:28:11.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:11 smithi045 conmon[39718]: audit 2021-07-31T02: 2021-07-31T02:28:11.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:11 smithi045 conmon[39718]: 28:10.644488+0000 mon.a (mon.0) 2049 2021-07-31T02:28:11.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:11 smithi045 conmon[39718]: : audit [DBG] from='client.? 172.21.15.45:0/1044743630' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:28:12.292 INFO:tasks.workunit.client.0.smithi045.stderr:waiting for timed out ... 4 2021-07-31T02:28:12.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:12 smithi154 conmon[34553]: cluster 2021-07-31T02:28:10.383744+0000 mgr.x (mgr.24542) 1754 : cluster 2021-07-31T02:28:12.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:12 smithi154 conmon[34553]: [DBG] pgmap v2204: 169 pgs: 169 active+clean; 49 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 1.7 KiB/s rd, 639 B/s wr, 3 op/s 2021-07-31T02:28:12.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:12 smithi045 conmon[33655]: cluster 2021-07-31T02:28:10.383744+0000 mgr.x (mgr.24542) 1754 : cluster [DBG] pgmap v2204: 169 pgs: 169 active+clean; 49 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 1.7 KiB/s rd, 639 B/s wr, 3 op/s 2021-07-31T02:28:12.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:12 smithi045 conmon[33655]: 2021-07-31T02:28:12.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:12 smithi045 conmon[39718]: cluster 2021-07-31T02:28:10.383744+0000 mgr.x (mgr. 2021-07-31T02:28:12.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:12 smithi045 conmon[39718]: 24542) 1754 : cluster [DBG] pgmap v2204: 169 pgs: 169 active+clean; 49 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 1.7 KiB/s rd, 639 B/s wr, 3 op/s 2021-07-31T02:28:13.292 INFO:tasks.workunit.client.0.smithi045.stderr:waiting for timed out ... 5 2021-07-31T02:28:14.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:28:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:28:13.580Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:28:14.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:28:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:28:13.580Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:28:14.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:28:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:28:13.580Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:28:14.292 INFO:tasks.workunit.client.0.smithi045.stderr:waiting for timed out ... 6 2021-07-31T02:28:14.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:14 smithi154 conmon[34553]: cluster 2021-07-31T02:28:12.384315+0000 mgr.x (mgr.24542) 1755 : cluster [DBG] 2021-07-31T02:28:14.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:14 smithi154 conmon[34553]: pgmap v2205: 169 pgs: 169 active+clean; 49 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 127 B/s wr, 0 op/s 2021-07-31T02:28:14.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:14 smithi045 conmon[33655]: cluster 2021-07-31T02:28:12.384315+0000 mgr.x (mgr.24542) 1755 : 2021-07-31T02:28:14.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:14 smithi045 conmon[33655]: cluster [DBG] pgmap v2205: 169 pgs: 169 active+clean; 49 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 127 B/s wr, 0 op/s 2021-07-31T02:28:14.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:14 smithi045 conmon[39718]: cluster 2021-07-31T02:28: 2021-07-31T02:28:14.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:14 smithi045 conmon[39718]: 12.384315+0000 mgr.x (mgr.24542) 1755 : cluster [DBG] pgmap v2205: 169 pgs: 169 active+clean; 49 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 127 B/s wr, 0 op/s 2021-07-31T02:28:15.292 INFO:tasks.workunit.client.0.smithi045.stderr:waiting for timed out ... 7 2021-07-31T02:28:16.293 INFO:tasks.workunit.client.0.smithi045.stderr:waiting for timed out ... 8 2021-07-31T02:28:16.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:16 smithi154 conmon[34553]: cluster 2021-07-31T02:28:16.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:16 smithi154 conmon[34553]: 2021-07-31T02:28:14.385338+0000 mgr.x (mgr.24542) 1756 : cluster [DBG] pgmap v2206: 169 pgs: 169 active+clean; 49 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 127 B/s wr, 0 op/s 2021-07-31T02:28:16.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:16 smithi045 conmon[33655]: cluster 2021-07-31T02:28:14.385338+0000 mgr.x (mgr.24542) 1756 : cluster [DBG] pgmap v2206: 169 pgs: 169 active+clean; 49 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 127 B/s wr, 0 op/s 2021-07-31T02:28:16.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:16 smithi045 conmon[33655]: 2021-07-31T02:28:16.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:16 smithi045 conmon[39718]: cluster 2021-07-31T02:28:14.385338+0000 mgr.x ( 2021-07-31T02:28:16.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:16 smithi045 conmon[39718]: mgr.24542) 1756 : cluster [DBG] pgmap v2206: 169 pgs: 169 active+clean; 49 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 127 B/s wr, 0 op/s 2021-07-31T02:28:17.293 INFO:tasks.workunit.client.0.smithi045.stderr:waiting for timed out ... 9 2021-07-31T02:28:18.293 INFO:tasks.workunit.client.0.smithi045.stderr:waiting for timed out ... 10 2021-07-31T02:28:18.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:18 smithi154 conmon[34553]: cluster 2021-07-31T02:28:16.385807+0000 mgr.x (mgr.24542) 1757 : cluster [DBG] 2021-07-31T02:28:18.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:18 smithi154 conmon[34553]: pgmap v2207: 169 pgs: 169 active+clean; 49 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 561 B/s rd, 112 B/s wr, 0 op/s 2021-07-31T02:28:18.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:18 smithi045 conmon[33655]: cluster 2021-07-31T02:28:16.385807+0000 mgr.x (mgr.24542) 1757 : cluster [DBG] pgmap v2207: 169 pgs: 169 active+clean; 49 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 561 B/s rd, 112 B/s wr, 0 op/s 2021-07-31T02:28:18.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:18 smithi045 conmon[39718]: cluster 2021-07-31T02:28:16.385807+0000 2021-07-31T02:28:18.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:18 smithi045 conmon[39718]: mgr.x (mgr.24542) 1757 : cluster [DBG] pgmap v2207: 169 pgs: 169 active+clean; 49 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 561 B/s rd, 112 B/s wr, 0 op/s 2021-07-31T02:28:20.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:20 smithi154 conmon[34553]: cluster 2021-07-31T02:28:18.386355+0000 mgr.x (mgr.24542) 1758 : cluster [DBG] 2021-07-31T02:28:20.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:20 smithi154 conmon[34553]: pgmap v2208: 169 pgs: 169 active+clean; 49 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 506 B/s rd, 101 B/s wr, 0 op/s 2021-07-31T02:28:20.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:20 smithi045 conmon[33655]: cluster 2021-07-31T02:28:18.386355+0000 mgr.x (mgr.24542) 1758 : cluster [DBG] pgmap v2208: 169 pgs: 169 active+clean; 49 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 506 B/s rd, 101 B/s wr, 0 op/s 2021-07-31T02:28:20.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:20 smithi045 conmon[39718]: cluster 2021-07-31T02:28:18.386355+0000 mgr.x ( 2021-07-31T02:28:20.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:20 smithi045 conmon[39718]: mgr.24542) 1758 : cluster [DBG] pgmap v2208: 169 pgs: 169 active+clean; 49 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 506 B/s rd, 101 B/s wr, 0 op/s 2021-07-31T02:28:22.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:22 smithi154 conmon[34553]: cluster 2021-07-31T02:28:20.387210+0000 mgr.x (mgr.24542) 1759 : cluster [DBG] 2021-07-31T02:28:22.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:22 smithi154 conmon[34553]: pgmap v2209: 169 pgs: 169 active+clean; 49 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 426 B/s rd, 85 B/s wr, 0 op/s 2021-07-31T02:28:22.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:22 smithi045 conmon[33655]: cluster 2021-07-31T02:28:20.387210+0000 mgr.x 2021-07-31T02:28:22.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:22 smithi045 conmon[33655]: (mgr.24542) 1759 : cluster [DBG] pgmap v2209: 169 pgs: 169 active+clean; 49 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 426 B/s rd, 85 B/s wr, 0 op/s 2021-07-31T02:28:22.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:22 smithi045 conmon[39718]: cluster 2021-07-31T02:28:20.387210+0000 2021-07-31T02:28:22.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:22 smithi045 conmon[39718]: mgr.x (mgr.24542) 1759 : cluster [DBG] pgmap v2209: 169 pgs: 169 active+clean; 49 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 426 B/s rd, 85 B/s wr, 0 op/s 2021-07-31T02:28:24.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:28:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:28:23.580Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:28:24.027 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:28:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:28:23.580Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:28:24.027 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:28:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:28:23.580Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:28:24.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:24 smithi154 conmon[34553]: cluster 2021-07-31T02:28:22.387769+0000 mgr.x (mgr.24542) 1760 : cluster [DBG] pgmap v2210: 169 pgs: 169 active+clean; 49 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:28:24.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:24 smithi045 conmon[39718]: cluster 2021-07-31T02:28:22.387769+0000 2021-07-31T02:28:24.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:24 smithi045 conmon[39718]: mgr.x (mgr.24542) 1760 : cluster [DBG] pgmap v2210: 169 pgs: 169 active+clean; 49 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:28:24.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:24 smithi045 conmon[33655]: cluster 2021-07-31T02:28:22.387769+0000 mgr.x (mgr.24542 2021-07-31T02:28:24.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:24 smithi045 conmon[33655]: ) 1760 : cluster [DBG] pgmap v2210: 169 pgs: 169 active+clean; 49 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:28:26.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:26 smithi154 conmon[34553]: cluster 2021-07-31T02:28:24.388838+0000 mgr.x (mgr.24542) 1761 : cluster [DBG] pgmap v2211: 169 pgs: 169 active+clean; 49 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:28:26.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:26 smithi045 conmon[39718]: cluster 2021-07-31T02:28:24.388838+0000 mgr.x (mgr.24542) 1761 : 2021-07-31T02:28:26.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:26 smithi045 conmon[39718]: cluster [DBG] pgmap v2211: 169 pgs: 169 active+clean; 49 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:28:26.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:26 smithi045 conmon[33655]: cluster 2021-07-31T02:28:24.388838+0000 mgr.x (mgr.24542) 1761 : cluster [DBG] pgmap v2211: 169 pgs: 169 active+clean; 49 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:28:28.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:28 smithi154 conmon[34553]: cluster 2021-07-31T02:28:26.389328+0000 mgr.x (mgr.24542) 1762 : cluster [DBG] pgmap v2212: 169 pgs: 169 active+clean; 49 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:28:28.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:28 smithi154 conmon[34553]: audit 2021-07-31T02:28:28.074204+0000 mon.b (mon.2) 278 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:28:28.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:28 smithi154 conmon[34553]: audit 2021-07-31T02:28:28.074683+0000 mon.a (mon.0) 2050 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:28:28.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:28 smithi154 conmon[34553]: audit 2021-07-31T02:28:28.099367+0000 mon.b (mon.2) 279 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:28:28.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:28 smithi154 conmon[34553]: audit 2021-07 2021-07-31T02:28:28.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:28 smithi154 conmon[34553]: -31T02:28:28.099979+0000 mon.a (mon.0) 2051 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:28:28.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:28 smithi045 conmon[33655]: cluster 2021-07-31T02:28:26.389328+0000 mgr.x (mgr.24542) 1762 2021-07-31T02:28:28.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:28 smithi045 conmon[33655]: : cluster [DBG] pgmap v2212: 169 pgs: 169 active+clean; 49 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:28:28.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:28 smithi045 conmon[33655]: audit 2021-07-31T02:28:28.074204+0000 mon.b (mon.2) 278 : audit 2021-07-31T02:28:28.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:28 smithi045 conmon[33655]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:28:28.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:28 smithi045 conmon[33655]: audit 2021-07-31T02 2021-07-31T02:28:28.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:28 smithi045 conmon[33655]: :28:28.074683+0000 mon.a (mon. 2021-07-31T02:28:28.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:28 smithi045 conmon[33655]: 0) 2050 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:28:28.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:28 smithi045 conmon[33655]: audit 2021-07- 2021-07-31T02:28:28.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:28 smithi045 conmon[33655]: 31T02:28:28.099367+0000 mon.b 2021-07-31T02:28:28.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:28 smithi045 conmon[33655]: (mon.2) 279 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:28:28.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:28 smithi045 conmon[33655]: 2021-07-31T02:28:28.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:28 smithi045 conmon[33655]: audit 2021-07-31T02:28 2021-07-31T02:28:28.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:28 smithi045 conmon[33655]: :28.099979+0000 mon.a (mon.0) 2021-07-31T02:28:28.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:28 smithi045 conmon[33655]: 2051 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:28:28.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:28 smithi045 conmon[39718]: cluster 2021-07-31T02:28:26.389328+0000 mgr.x (mgr.24542) 1762 : cluster [DBG] pgmap v2212: 169 pgs: 169 active+clean; 49 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:28:28.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:28 smithi045 conmon[39718]: 2021-07-31T02:28:28.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:28 smithi045 conmon[39718]: audit 2021-07-31T02:28 2021-07-31T02:28:28.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:28 smithi045 conmon[39718]: :28.074204+0000 mon.b (mon.2) 2021-07-31T02:28:28.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:28 smithi045 conmon[39718]: 278 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:28:28.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:28 smithi045 conmon[39718]: audit 2021- 2021-07-31T02:28:28.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:28 smithi045 conmon[39718]: 07-31T02:28:28.074683 2021-07-31T02:28:28.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:28 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2050 2021-07-31T02:28:28.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:28 smithi045 conmon[39718]: : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:28:28.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:28 smithi045 conmon[39718]: audit 2021 2021-07-31T02:28:28.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:28 smithi045 conmon[39718]: -07-31T02:28: 2021-07-31T02:28:28.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:28 smithi045 conmon[39718]: 28.099367+0000 mon.b ( 2021-07-31T02:28:28.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:28 smithi045 conmon[39718]: mon.2) 279 : audit 2021-07-31T02:28:28.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:28 smithi045 conmon[39718]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:28:28.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:28 smithi045 conmon[39718]: audit 2021 2021-07-31T02:28:28.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:28 smithi045 conmon[39718]: -07-31T02:28:28.099979+0000 mon.a 2021-07-31T02:28:28.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:28 smithi045 conmon[39718]: (mon.0) 2051 : audit 2021-07-31T02:28:28.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:28 smithi045 conmon[39718]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:28:29.297 INFO:tasks.workunit.client.0.smithi045.stderr:test retry succeeds 2021-07-31T02:28:30.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:30 smithi154 conmon[34553]: cluster 2021-07-31T02:28:28.389684+0000 mgr.x (mgr.24542) 1763 : cluster [DBG] pgmap v2213: 169 pgs: 169 active+clean; 49 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:28:30.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:30 smithi154 conmon[34553]: cluster 2021-07-31T02:28:29.331649+0000 mon.a (mon.0) 2052 : cluster [DBG] osdmap e1130: 8 total, 8 up, 8 in 2021-07-31T02:28:30.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:30 smithi045 conmon[33655]: cluster 2021-07-31T02:28:28.389684+0000 mgr.x (mgr.24542) 1763 : cluster [DBG] 2021-07-31T02:28:30.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:30 smithi045 conmon[33655]: pgmap v2213: 169 pgs: 169 active+clean; 49 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:28:30.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:30 smithi045 conmon[33655]: cluster 2021-07-31T02:28:29.331649+0000 mon.a (mon.0) 2052 : cluster [DBG] osdmap e1130: 8 total, 8 up, 8 in 2021-07-31T02:28:30.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:30 smithi045 conmon[39718]: cluster 2021-07-31T02:28:30.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:30 smithi045 conmon[39718]: 2021-07-31T02:28:30.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:30 smithi045 conmon[39718]: 2021 2021-07-31T02:28:30.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:30 smithi045 conmon[39718]: - 2021-07-31T02:28:30.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:30 smithi045 conmon[39718]: 07 2021-07-31T02:28:30.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:30 smithi045 conmon[39718]: - 2021-07-31T02:28:30.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:30 smithi045 conmon[39718]: 31 2021-07-31T02:28:30.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:30 smithi045 conmon[39718]: T 2021-07-31T02:28:30.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:30 smithi045 conmon[39718]: 02 2021-07-31T02:28:30.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:30 smithi045 conmon[39718]: : 2021-07-31T02:28:30.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:30 smithi045 conmon[39718]: 28 2021-07-31T02:28:30.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:30 smithi045 conmon[39718]: : 2021-07-31T02:28:30.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:30 smithi045 conmon[39718]: 28 2021-07-31T02:28:30.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:30 smithi045 conmon[39718]: . 2021-07-31T02:28:30.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:30 smithi045 conmon[39718]: 389684 2021-07-31T02:28:30.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:30 smithi045 conmon[39718]: +0000 2021-07-31T02:28:30.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:30 smithi045 conmon[39718]: 2021-07-31T02:28:30.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:30 smithi045 conmon[39718]: mgr.x (mgr.24542) 1763 : cluster [DBG] 2021-07-31T02:28:30.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:30 smithi045 conmon[39718]: pgmap v2213: 169 pgs: 169 active+clean; 49 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:28:30.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:30 smithi045 conmon[39718]: cluster 2021-07-31T02:28:30.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:30 smithi045 conmon[39718]: 2021-07-31T02:28:29.331649+0000 2021-07-31T02:28:30.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:30 smithi045 conmon[39718]: mon.a (mon.0) 2052 2021-07-31T02:28:30.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:30 smithi045 conmon[39718]: : cluster [DBG] osdmap e1130: 8 total, 8 up, 8 in 2021-07-31T02:28:31.374 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.QuiesceWatchTimeout (32075 ms) 2021-07-31T02:28:31.375 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.WriteZeroes 2021-07-31T02:28:31.375 INFO:tasks.workunit.client.0.smithi045.stdout:using old format 2021-07-31T02:28:31.428 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.WriteZeroes (53 ms) 2021-07-31T02:28:31.428 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.WriteZeroesThickProvision 2021-07-31T02:28:31.428 INFO:tasks.workunit.client.0.smithi045.stdout:using old format 2021-07-31T02:28:31.480 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.WriteZeroesThickProvision (53 ms) 2021-07-31T02:28:31.480 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestLibRBD.ConcurentOperations 2021-07-31T02:28:31.481 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:28:31.481 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestLibRBD.ConcurentOperations (0 ms) 2021-07-31T02:28:31.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:31 smithi154 conmon[34553]: cluster 2021-07-31T02:28:30.334348+0000 mon.a (mon.0) 2053 : 2021-07-31T02:28:31.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:31 smithi154 conmon[34553]: cluster [DBG] osdmap e1131: 8 total, 8 up, 8 in 2021-07-31T02:28:31.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:31 smithi154 conmon[34553]: cluster 2021-07-31T02:28:30.390265+0000 mgr.x (mgr.24542) 1764 : cluster [DBG] pgmap v2216: 169 pgs: 169 active+clean; 49 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:28:31.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:31 smithi045 conmon[33655]: cluster 2021-07-31T02:28:30.334348+0000 mon.a (mon.0) 2053 : cluster 2021-07-31T02:28:31.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:31 smithi045 conmon[33655]: [DBG] osdmap e1131: 8 total, 8 up, 8 in 2021-07-31T02:28:31.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:31 smithi045 conmon[33655]: cluster 2021-07-31T02:28:30.390265+0000 mgr.x (mgr.24542) 1764 : cluster [DBG] pgmap v2216: 169 pgs: 169 active+clean; 49 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:28:31.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:31 smithi045 conmon[39718]: cluster 2021-07-31T02:28:30.334348+0000 mon.a (mon.0) 2021-07-31T02:28:31.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:31 smithi045 conmon[39718]: 2053 : cluster [DBG] osdmap e1131: 8 total, 8 up, 8 in 2021-07-31T02:28:31.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:31 smithi045 conmon[39718]: cluster 2021-07-31T02:28:30.390265+0000 mgr.x (mgr.24542) 1764 : cluster [DBG] pgmap v2216: 169 pgs: 169 active+clean; 49 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:28:32.361 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: cluster 2021-07-31T02:28:32.361 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: 2021-07-31T02:28:31.347704+0000 mon.a (mon. 2021-07-31T02:28:32.361 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: 0) 2054 : cluster [DBG] osdmap e1132: 8 total, 8 up, 8 in 2021-07-31T02:28:32.361 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T02:28:32.362 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: 02:28:32.083352+0000 mon.b (mon.2) 280 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:28:32.362 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: 2021-07-31T02:28:32.362 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: audit 2021- 2021-07-31T02:28:32.362 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: 07-31T02:28:32.084945+0000 mon.b ( 2021-07-31T02:28:32.363 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: mon.2) 281 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:28:32.363 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T02:28:32.364 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: cluster 2021-07-31T02:28:32.364 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: 2021-07-31T02:28:31. 2021-07-31T02:28:32.365 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: 347704+0000 mon.a (mon.0) 2054 : cluster [DBG] osdmap e1132: 8 total, 8 up, 8 in 2021-07-31T02:28:32.366 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: audit 2021-07 2021-07-31T02:28:32.366 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: -31T02:28:32.083352+0000 mon.b 2021-07-31T02:28:32.366 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: (mon.2) 280 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:28:32.367 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: audit 2021-07-31T02:28:32.367 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: 2021-07-31T02:28:32.084945 2021-07-31T02:28:32.367 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: +0000 mon.b (mon.2) 281 : audit [DBG] 2021-07-31T02:28:32.367 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:28:32.367 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: audit 2021- 2021-07-31T02:28:32.368 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: 07-31T02:28:32.085943+0000 mon.b (mon.2) 2021-07-31T02:28:32.368 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: 282 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:28:32.368 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: cephadm 2021-07-31T02:28:32.086693+0000 mgr.x ( 2021-07-31T02:28:32.368 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: mgr.24542) 1765 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:28:32.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: cephadm 2021-07-31T02:28:32. 2021-07-31T02:28:32.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: 130183+0000 mgr.x (mgr.24542) 1766 : cephadm [ERR] 2021-07-31T02:28:32.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:28:32.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:28:32.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: yield 2021-07-31T02:28:32.370 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:28:32.370 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:28:32.370 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:28:32.370 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:28:32.370 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:28:32.371 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:28:32.371 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:28:32.371 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:28:32.371 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:28:32.372 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: task.result() 2021-07-31T02:28:32.372 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:28:32.372 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: 2021-07-31T02:28:32.372 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: cephadm 2021-07-31T02:28:32.130596+0000 mgr.x (mgr.24542) 1767 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:28:32.372 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: cephadm 2021-07-31T02:28:32.141812+0000 mgr.x (mgr.24542) 1768 : cephadm [ERR] 2021-07-31T02:28:32.373 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:28:32.373 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:28:32.373 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: yield 2021-07-31T02:28:32.373 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:28:32.374 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:28:32.374 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:28:32.374 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:28:32.374 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:28:32.375 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:28:32.375 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:28:32.375 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:28:32.376 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:28:32.376 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: task.result() 2021-07-31T02:28:32.376 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:28:32.376 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: cephadm 2021-07-31T02:28:32.142749+0000 mgr.x (mgr.24542) 2021-07-31T02:28:32.377 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: 1769 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:28:32.377 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: cephadm 2021-07-31T02:28:32.167182+0000 mgr.x (mgr.24542) 1770 : cephadm [ERR] 2021-07-31T02:28:32.377 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:28:32.377 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:28:32.377 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: yield 2021-07-31T02:28:32.378 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:28:32.378 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:28:32.378 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:28:32.378 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:28:32.379 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:28:32.379 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:28:32.379 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:28:32.379 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:28:32.379 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:28:32.380 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: task.result() 2021-07-31T02:28:32.380 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:28:32.380 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: cephadm 2021-07-31T02:28:32.167828+0000 mgr.x (mgr.24542) 1771 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:28:32.381 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:28:32.381 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:28:32.381 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: yield 2021-07-31T02:28:32.382 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:28:32.382 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:28:32.382 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:28:32.382 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:28:32.383 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:28:32.383 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:28:32.383 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:28:32.383 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:28:32.383 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:28:32.384 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: task.result() 2021-07-31T02:28:32.384 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:28:32.384 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: 2021-07-31T02:28:32.385 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:28:32.385 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: 2021-07-31T02:28:32.385 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:28:32.385 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:28:32.386 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:28:32.386 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:28:32.386 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:28:32.387 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:28:32.387 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:28:32.387 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:28:32.387 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:28:32.387 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:28:32.388 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:28:32.388 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:28:32.388 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:28:32.388 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:28:32.388 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:28:32.389 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:28:32.389 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:28:32.389 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:28:32.389 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: raise 2021-07-31T02:28:32.390 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:28:32.390 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:28:32.390 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:28:32.390 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:28:32.390 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:28:32.391 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: cephadm 2021-07-31T02:28:32.176091+0000 mgr.x (mgr.24542) 1772 : cephadm [ERR] 2021-07-31T02:28:32.391 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:28:32.391 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:28:32.391 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: yield 2021-07-31T02:28:32.391 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:28:32.392 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:28:32.392 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:28:32.392 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:28:32.392 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:28:32.392 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:28:32.393 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:28:32.393 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:28:32.393 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:28:32.393 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: task.result() 2021-07-31T02:28:32.394 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:28:32.394 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: cephadm 2021-07-31T02:28:32.176539+0000 mgr.x (mgr.24542) 1773 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:28:32.394 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:28:32.395 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:28:32.395 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: yield 2021-07-31T02:28:32.395 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:28:32.395 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:28:32.395 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:28:32.396 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:28:32.396 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:28:32.396 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:28:32.397 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:28:32.397 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:28:32.397 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:28:32.397 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: task.result() 2021-07-31T02:28:32.398 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:28:32.398 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: 2021-07-31T02:28:32.398 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:28:32.398 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: 2021-07-31T02:28:32.398 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:28:32.399 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:28:32.399 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:28:32.399 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:28:32.399 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:28:32.399 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:28:32.400 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:28:32.400 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:28:32.400 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:28:32.400 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:28:32.401 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:28:32.401 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:28:32.401 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:28:32.401 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:28:32.401 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:28:32.402 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:28:32.402 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:28:32.402 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:28:32.402 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: raise 2021-07-31T02:28:32.402 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:28:32.403 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:28:32.403 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:28:32.403 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:28:32.403 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:28:32.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: cluster 2021-07-31T02:28:31.347704+0000 2021-07-31T02:28:32.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: mon.a (mon.0) 2054 : cluster [DBG] osdmap e1132: 8 total, 8 up, 8 in 2021-07-31T02:28:32.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: audit 2021-07-31T02:28:32.083352+0000 mon.b (mon.2) 280 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:28:32.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: audit 2021-07-31T02: 2021-07-31T02:28:32.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: 28:32.084945+0000 mon.b (mon. 2021-07-31T02:28:32.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: 2) 281 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:28:32.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: audit 2021-07 2021-07-31T02:28:32.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: -31T02:28:32.085943+0000 mon.b (mon.2) 282 : audit [INF] 2021-07-31T02:28:32.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:28:32.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: cephadm 2021-07-31T02:28:32.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: 2021-07-31T02:28:32.086693+0000 2021-07-31T02:28:32.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: mgr.x (mgr.24542) 1765 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:28:32.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: cephadm 2021-07-31T02:28:32.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: 2021-07-31T02:28:32.130183+0000 mgr.x ( 2021-07-31T02:28:32.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: mgr.24542) 1766 : cephadm [ERR] 2021-07-31T02:28:32.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:28:32.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:28:32.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: yield 2021-07-31T02:28:32.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:28:32.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:28:32.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:28:32.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:28:32.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:28:32.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:28:32.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:28:32.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:28:32.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:28:32.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: task.result() 2021-07-31T02:28:32.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:28:32.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: cephadm 2021-07-31T02:28:32.130596+0000 mgr.x (mgr.24542) 1767 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:28:32.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: cephadm 2021-07-31T02:28:32.141812+0000 mgr.x (mgr.24542) 1768 : cephadm [ERR] 2021-07-31T02:28:32.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:28:32.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:28:32.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: yield 2021-07-31T02:28:32.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:28:32.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:28:32.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:28:32.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:28:32.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: debug 2021-07-31T02:28:32.129+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:28:32.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:28:32.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:28:32.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: yield 2021-07-31T02:28:32.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:28:32.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:28:32.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:28:32.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:28:32.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:28:32.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:28:32.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:28:32.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:28:32.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:28:32.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: task.result() 2021-07-31T02:28:32.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:28:32.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: debug 2021-07-31T02:28:32.140+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:28:32.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:28:32.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:28:32.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: yield 2021-07-31T02:28:32.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:28:32.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:28:32.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:28:32.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:28:32.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:28:32.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:28:32.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:28:32.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:28:32.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:28:32.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: task.result() 2021-07-31T02:28:32.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:28:32.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: debug 2021-07-31T02:28:32.166+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:28:32.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:28:32.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:28:32.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: yield 2021-07-31T02:28:32.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:28:32.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:28:32.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:28:32.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:28:32.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:28:32.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:28:32.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:28:32.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:28:32.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:28:32.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:28:32.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: task.result() 2021-07-31T02:28:32.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:28:32.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: cephadm 2021-07-31T02:28:32.142749+0000 mgr.x (mgr.24542) 1769 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:28:32.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: cephadm 2021-07-31T02:28:32.167182+0000 mgr.x (mgr. 2021-07-31T02:28:32.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: 24542) 1770 : cephadm [ERR] 2021-07-31T02:28:32.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:28:32.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:28:32.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: yield 2021-07-31T02:28:32.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:28:32.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:28:32.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:28:32.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:28:32.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:28:32.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:28:32.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:28:32.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:28:32.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:28:32.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: task.result() 2021-07-31T02:28:32.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:28:32.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: cephadm 2021-07-31T02:28:32.167828+0000 mgr.x (mgr.24542) 1771 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:28:32.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:28:32.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:28:32.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: yield 2021-07-31T02:28:32.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:28:32.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:28:32.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:28:32.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:28:32.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:28:32.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:28:32.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:28:32.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:28:32.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:28:32.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: task.result() 2021-07-31T02:28:32.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:28:32.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: 2021-07-31T02:28:32.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:28:32.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: 2021-07-31T02:28:32.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:28:32.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:28:32.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:28:32.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:28:32.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:28:32.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:28:32.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:28:32.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:28:32.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:28:32.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:28:32.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:28:32.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:28:32.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:28:32.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:28:32.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:28:32.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:28:32.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:28:32.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:28:32.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: raise 2021-07-31T02:28:32.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:28:32.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:28:32.541 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:28:32.541 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:28:32.541 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:28:32.541 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: cephadm 2021-07-31T02:28:32.176091+0000 2021-07-31T02:28:32.542 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: mgr.x (mgr.24542) 1772 : cephadm [ERR] 2021-07-31T02:28:32.542 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:28:32.542 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:28:32.542 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: yield 2021-07-31T02:28:32.543 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:28:32.543 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:28:32.543 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:28:32.543 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:28:32.544 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:28:32.544 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:28:32.544 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:28:32.545 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:28:32.545 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:28:32.545 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: task.result() 2021-07-31T02:28:32.545 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:28:32.546 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:28:32.546 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:28:32.547 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:28:32.547 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:28:32.547 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: task.result() 2021-07-31T02:28:32.547 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:28:32.548 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: debug 2021-07-31T02:28:32.166+0000 7fcb9295f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:28:32.548 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:28:32.548 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:28:32.548 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: yield 2021-07-31T02:28:32.549 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:28:32.549 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:28:32.549 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:28:32.549 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:28:32.550 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:28:32.550 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:28:32.550 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:28:32.550 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:28:32.551 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:28:32.551 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: task.result() 2021-07-31T02:28:32.551 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:28:32.551 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: 2021-07-31T02:28:32.552 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:28:32.552 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: 2021-07-31T02:28:32.552 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:28:32.552 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:28:32.552 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:28:32.553 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:28:32.553 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:28:32.553 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:28:32.554 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:28:32.554 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:28:32.554 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:28:32.555 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:28:32.555 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:28:32.555 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:28:32.555 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:28:32.556 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:28:32.556 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:28:32.556 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:28:32.557 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:28:32.558 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: cephadm 2021-07-31T02:28:32.176539+0000 mgr.x (mgr.24542) 1773 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:28:32.558 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:28:32.558 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:28:32.559 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: yield 2021-07-31T02:28:32.559 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:28:32.559 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:28:32.560 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:28:32.560 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:28:32.560 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:28:32.560 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:28:32.561 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:28:32.561 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:28:32.561 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:28:32.562 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: task.result() 2021-07-31T02:28:32.562 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:28:32.562 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: 2021-07-31T02:28:32.563 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:28:32.563 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: 2021-07-31T02:28:32.563 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:28:32.563 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:28:32.564 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:28:32.564 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:28:32.564 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:28:32.565 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:28:32.565 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:28:32.565 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:28:32.566 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:28:32.566 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:28:32.566 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:28:32.566 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:28:32.567 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:28:32.567 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:28:32.567 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:28:32.568 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:28:32.568 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:28:32.568 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:28:32.568 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: raise 2021-07-31T02:28:32.568 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:28:32.569 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:28:32.569 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:28:32.569 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:28:32.569 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:28:32.570 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:28:32.571 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: raise 2021-07-31T02:28:32.571 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:28:32.571 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:28:32.572 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:28:32.572 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:28:32.572 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:28:32.573 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: debug 2021-07-31T02:28:32.175+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:28:32.573 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:28:32.573 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:28:32.573 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: yield 2021-07-31T02:28:32.574 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:28:32.574 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:28:32.574 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:28:32.575 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:28:32.575 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:28:32.575 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:28:32.575 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:28:32.576 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:28:32.576 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:28:32.576 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: task.result() 2021-07-31T02:28:32.577 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:28:32.577 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: debug 2021-07-31T02:28:32.175+0000 7fcb93160700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:28:32.577 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:28:32.577 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:28:32.578 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: yield 2021-07-31T02:28:32.578 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:28:32.578 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:28:32.578 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:28:32.579 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:28:32.579 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:28:32.579 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:28:32.579 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:28:32.580 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:28:32.580 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:28:32.580 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: task.result() 2021-07-31T02:28:32.580 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:28:32.581 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: 2021-07-31T02:28:32.581 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:28:32.581 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: 2021-07-31T02:28:32.581 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:28:32.582 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:28:32.582 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:28:32.582 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:28:32.582 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:28:32.583 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:28:32.583 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:28:32.583 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:28:32.583 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:28:32.584 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:28:32.584 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:28:32.584 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:28:32.584 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:28:32.585 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:28:32.585 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:28:32.585 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:28:32.585 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:28:32.585 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:28:32.586 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: raise 2021-07-31T02:28:32.586 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:28:32.586 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:28:32.586 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:28:32.587 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:28:32.587 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:28:32 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:28:32.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: 31T02:28:32.085943+0000 mon.b (mon.2) 2021-07-31T02:28:32.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: 282 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:28:32.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: cephadm 2021-07-31T02:28:32.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: 2021-07-31T02:28:32.086693+0000 mgr.x 2021-07-31T02:28:32.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: (mgr.24542) 1765 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:28:32.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: cephadm 2021- 2021-07-31T02:28:32.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: 07-31T02:28:32.130183+0000 mgr.x ( 2021-07-31T02:28:32.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: mgr.24542) 1766 : cephadm [ERR] 2021-07-31T02:28:32.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:28:32.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:28:32.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: yield 2021-07-31T02:28:32.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:28:32.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:28:32.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:28:32.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:28:32.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:28:32.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:28:32.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:28:32.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:28:32.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:28:32.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: task.result() 2021-07-31T02:28:32.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:28:32.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: cephadm 2021-07-31T02:28:32.130596+0000 mgr.x (mgr.24542) 1767 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:28:32.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: cephadm 2021-07-31T 2021-07-31T02:28:32.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: 02:28:32.141812+0000 mgr.x (mgr.24542) 1768 : 2021-07-31T02:28:32.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: cephadm [ERR] 2021-07-31T02:28:32.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:28:32.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:28:32.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: yield 2021-07-31T02:28:32.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:28:32.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:28:32.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:28:32.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:28:32.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:28:32.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:28:32.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:28:32.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:28:32.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:28:32.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: task.result() 2021-07-31T02:28:32.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:28:32.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: cephadm 2021-07-31T02:28:32.142749+0000 mgr.x (mgr.24542) 1769 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:28:32.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: cephadm 2021 2021-07-31T02:28:32.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: -07-31T02:28:32.167182+0000 mgr.x (mgr.24542) 1770 2021-07-31T02:28:32.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: : cephadm [ERR] 2021-07-31T02:28:32.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:28:32.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:28:32.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: yield 2021-07-31T02:28:32.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:28:32.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:28:32.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:28:32.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:28:32.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:28:32.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:28:32.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:28:32.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:28:32.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:28:32.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: task.result() 2021-07-31T02:28:32.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:28:32.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: cephadm 2021-07-31T02:28:32.167828+0000 mgr.x (mgr.24542) 1771 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:28:32.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:28:32.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:28:32.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: yield 2021-07-31T02:28:32.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:28:32.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:28:32.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:28:32.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:28:32.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:28:32.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:28:32.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:28:32.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:28:32.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:28:32.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: task.result() 2021-07-31T02:28:32.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:28:32.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: 2021-07-31T02:28:32.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:28:32.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: 2021-07-31T02:28:32.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:28:32.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:28:32.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:28:32.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:28:32.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:28:32.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:28:32.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:28:32.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:28:32.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:28:32.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:28:32.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:28:32.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:28:32.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:28:32.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:28:32.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:28:32.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:28:32.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:28:32.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:28:32.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: raise 2021-07-31T02:28:32.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:28:32.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:28:32.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:28:32.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:28:32.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:28:32.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: cephadm 2021-07-31T02:28:32.176091+0000 mgr.x (mgr.24542) 1772 : cephadm [ERR] 2021-07-31T02:28:32.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:28:32.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:28:32.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: yield 2021-07-31T02:28:32.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:28:32.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:28:32.802 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:28:32.802 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:28:32.802 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:28:32.802 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:28:32.802 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:28:32.803 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:28:32.803 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:28:32.803 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: task.result() 2021-07-31T02:28:32.803 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:28:32.804 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: cephadm 2021-07-31T02:28:32.176539+0000 mgr.x (mgr.24542) 1773 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:28:32.804 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:28:32.804 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:28:32.804 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: yield 2021-07-31T02:28:32.804 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:28:32.805 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:28:32.805 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:28:32.806 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:28:32.806 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:28:32.806 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:28:32.806 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:28:32.806 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:28:32.807 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:28:32.807 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: task.result() 2021-07-31T02:28:32.807 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:28:32.807 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: 2021-07-31T02:28:32.808 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:28:32.808 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: 2021-07-31T02:28:32.808 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:28:32.808 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:28:32.808 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:28:32.809 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:28:32.809 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:28:32.809 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:28:32.809 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:28:32.809 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:28:32.810 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:28:32.810 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:28:32.810 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:28:32.810 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:28:32.810 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:28:32.811 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:28:32.811 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:28:32.811 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:28:32.811 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:28:32.812 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:28:32.812 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: raise 2021-07-31T02:28:32.812 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:28:32.812 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:28:32.812 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:28:32.813 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:28:32.813 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:32 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:28:33.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:33 smithi154 conmon[34553]: cluster 2021-07-31T02:28:32.348261+0000 mon.a (mon.0) 2055 : cluster [DBG] osdmap e1133: 8 total, 8 up, 8 in 2021-07-31T02:28:33.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:33 smithi154 conmon[34553]: cluster 2021-07-31T02: 2021-07-31T02:28:33.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:33 smithi154 conmon[34553]: 28:32.390665+0000 mgr.x (mgr.24542) 1774 : cluster [DBG] pgmap v2219: 137 pgs: 137 active+clean; 49 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:28:33.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:33 smithi045 conmon[39718]: cluster 2021-07-31T02:28:32.348261+0000 mon.a (mon.0 2021-07-31T02:28:33.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:33 smithi045 conmon[39718]: ) 2055 : cluster [DBG] osdmap e1133: 8 total, 8 up, 8 in 2021-07-31T02:28:33.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:33 smithi045 conmon[39718]: cluster 2021-07-31T02:28:32.390665+0000 2021-07-31T02:28:33.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:33 smithi045 conmon[39718]: mgr.x (mgr.24542) 1774 : cluster [DBG] pgmap v2219: 137 pgs: 137 active+clean; 49 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:28:33.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:28:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:28:33.581Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:28:33.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:28:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:28:33.581Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:28:33.778 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:28:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:28:33.581Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:28:33.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:33 smithi045 conmon[33655]: cluster 2021-07-31T02:28:32.348261+0000 mon.a (mon.0) 2055 : cluster [DBG] osdmap e1133: 8 total, 8 up, 8 in 2021-07-31T02:28:33.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:33 smithi045 conmon[33655]: cluster 2021-07-31T02 2021-07-31T02:28:33.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:33 smithi045 conmon[33655]: :28:32.390665+0000 mgr.x (mgr.24542) 1774 : cluster [DBG] pgmap v2219: 137 pgs: 137 active+clean; 49 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:28:34.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:34 smithi154 conmon[34553]: cluster 2021-07-31T02:28:33.351405+0000 mon.a (mon.0) 2056 : cluster [DBG] osdmap e1134: 8 total, 8 up, 8 in 2021-07-31T02:28:34.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:34 smithi154 conmon[34553]: cluster 2021-07-31T02:28:34.352187 2021-07-31T02:28:34.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:34 smithi154 conmon[34553]: +0000 mon.a (mon.0) 2057 : cluster [DBG] osdmap e1135: 8 total, 8 up, 8 in 2021-07-31T02:28:34.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:34 smithi045 conmon[39718]: cluster 2021-07-31T02:28:33.351405+0000 mon.a (mon.0) 2021-07-31T02:28:34.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:34 smithi045 conmon[39718]: 2056 : cluster [DBG] osdmap e1134: 8 total, 8 up, 8 in 2021-07-31T02:28:34.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:34 smithi045 conmon[39718]: cluster 2021-07-31T02:28:34.352187+0000 2021-07-31T02:28:34.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:34 smithi045 conmon[39718]: mon.a (mon.0) 2057 : cluster [DBG] osdmap e1135: 8 total, 8 up, 8 in 2021-07-31T02:28:34.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:34 smithi045 conmon[33655]: cluster 2021-07-31T02:28:33.351405+0000 mon.a (mon.0) 2056 : cluster 2021-07-31T02:28:34.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:34 smithi045 conmon[33655]: [DBG] osdmap e1134: 8 total, 8 up, 8 in 2021-07-31T02:28:34.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:34 smithi045 conmon[33655]: cluster 2021-07-31T02:28:34.352187+0000 mon.a (mon.0) 2057 : cluster [DBG] osdmap e1135: 8 total, 8 up, 8 in 2021-07-31T02:28:35.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:35 smithi154 conmon[34553]: cluster 2021-07-31T02:28:34.391205+0000 mgr.x (mgr.24542) 1775 : cluster [DBG] pgmap v2222: 73 pgs: 73 active+clean; 49 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:28:35.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:35 smithi154 conmon[34553]: 2021-07-31T02:28:35.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:35 smithi154 conmon[34553]: cluster 2021-07-31T02:28:35.354036+0000 mon.a (mon.0) 2058 : cluster [DBG] osdmap e1136: 8 total, 8 up, 8 in 2021-07-31T02:28:35.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:35 smithi045 conmon[33655]: cluster 2021-07-31T02:28:34.391205+0000 mgr.x ( 2021-07-31T02:28:35.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:35 smithi045 conmon[33655]: mgr.24542) 1775 : cluster [DBG] pgmap v2222: 73 pgs: 73 active+clean; 49 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:28:35.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:35 smithi045 conmon[33655]: cluster 2021-07-31T02:28:35.354036+0000 mon.a (mon.0) 2058 : cluster [DBG] osdmap e1136: 8 total, 8 up, 8 in 2021-07-31T02:28:35.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:35 smithi045 conmon[39718]: cluster 2021-07-31T02:28:34. 2021-07-31T02:28:35.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:35 smithi045 conmon[39718]: 391205+0000 mgr.x (mgr.24542) 1775 : cluster [DBG] pgmap v2222: 73 pgs: 73 active+clean; 49 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:28:35.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:35 smithi045 conmon[39718]: cluster 2021-07-31T02:28:35.354036+0000 mon.a 2021-07-31T02:28:35.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:35 smithi045 conmon[39718]: (mon.0) 2058 : cluster [DBG] osdmap e1136: 8 total, 8 up, 8 in 2021-07-31T02:28:36.359 INFO:tasks.workunit.client.0.smithi045.stdout:[----------] 124 tests from TestLibRBD (151365 ms total) 2021-07-31T02:28:36.359 INFO:tasks.workunit.client.0.smithi045.stdout: 2021-07-31T02:28:36.359 INFO:tasks.workunit.client.0.smithi045.stdout:[----------] 7 tests from DiffIterateTest/0, where TypeParam = DiffIterateParams 2021-07-31T02:28:36.390 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] DiffIterateTest/0.DiffIterate 2021-07-31T02:28:37.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:37 smithi154 conmon[34553]: cluster 2021-07-31T02:28:36.356370+0000 mon.a (mon.0) 2059 : cluster 2021-07-31T02:28:37.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:37 smithi154 conmon[34553]: [DBG] osdmap e1137: 8 total, 8 up, 8 in 2021-07-31T02:28:37.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:37 smithi154 conmon[34553]: cluster 2021-07-31T02:28:36.391689+0000 mgr.x (mgr.24542) 1776 : cluster [DBG] pgmap v2225: 9 pgs: 9 active+clean; 4.6 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:28:37.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:37 smithi045 conmon[33655]: cluster 2021-07-31T02:28:36.356370 2021-07-31T02:28:37.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:37 smithi045 conmon[33655]: +0000 mon.a (mon.0) 2059 : cluster [DBG] osdmap e1137: 8 total, 8 up, 8 in 2021-07-31T02:28:37.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:37 smithi045 conmon[33655]: cluster 2021-07-31T02:28:36.391689+0000 mgr.x (mgr.24542) 1776 : cluster 2021-07-31T02:28:37.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:37 smithi045 conmon[33655]: [DBG] pgmap v2225: 9 pgs: 9 active+clean; 4.6 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:28:37.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:37 smithi045 conmon[39718]: cluster 2021-07-31T02:28:36 2021-07-31T02:28:37.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:37 smithi045 conmon[39718]: .356370+0000 mon.a (mon.0) 2059 : cluster [DBG] osdmap e1137: 8 total, 8 up, 8 in 2021-07-31T02:28:37.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:37 smithi045 conmon[39718]: cluster 2021-07-31T02:28: 2021-07-31T02:28:37.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:37 smithi045 conmon[39718]: 36.391689+0000 mgr.x (mgr.24542) 1776 : cluster [DBG] pgmap v2225: 9 pgs: 9 active+clean; 4.6 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:28:38.375 INFO:tasks.workunit.client.0.smithi045.stdout:using old format 2021-07-31T02:28:38.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:38 smithi154 conmon[34553]: cluster 2021-07-31T02:28:38.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:38 smithi154 conmon[34553]: 2021-07-31T02:28:38.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:38 smithi154 conmon[34553]: 2021 2021-07-31T02:28:38.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:38 smithi154 conmon[34553]: -07-31T02:28:37.361591+0000 mon.a ( 2021-07-31T02:28:38.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:38 smithi154 conmon[34553]: mon.0) 2060 : cluster [DBG] osdmap e1138: 8 total, 8 up, 8 in 2021-07-31T02:28:38.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:38 smithi154 conmon[34553]: audit 2021-07-31T02:28:37.362852+0000 mon.c ( 2021-07-31T02:28:38.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:38 smithi154 conmon[34553]: mon.1) 100 : audit [INF] from='client.? 172.21.15.45:0/950621650' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:28:38.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:38 smithi154 conmon[34553]: audit 2021-07- 2021-07-31T02:28:38.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:38 smithi154 conmon[34553]: 31T02:28:37.364312+0000 mon.a (mon.0 2021-07-31T02:28:38.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:38 smithi154 conmon[34553]: ) 2061 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:28:38.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:38 smithi045 conmon[39718]: cluster 2021-07-31T02:28:37.361591+0000 mon.a (mon.0) 2060 : cluster [DBG] osdmap e1138: 8 total, 8 up, 8 in 2021-07-31T02:28:38.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:38 smithi045 conmon[39718]: audit 2021-07-31T02:28 2021-07-31T02:28:38.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:38 smithi045 conmon[39718]: :37.362852+0000 mon.c (mon.1) 100 : audit [INF] from='client.? 172.21.15.45:0/950621650' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:28:38.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:38 smithi045 conmon[39718]: audit 2021-07-31T02:28:37.364312+0000 mon.a (mon.0) 2061 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:28:38.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:38 smithi045 conmon[33655]: cluster 2021-07-31T02:28:37.361591+0000 mon.a 2021-07-31T02:28:38.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:38 smithi045 conmon[33655]: (mon.0) 2060 : cluster [DBG] osdmap e1138: 8 total, 8 up, 8 in 2021-07-31T02:28:38.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:38 smithi045 conmon[33655]: audit 2021-07-31T02:28:37.362852+0000 mon.c (mon.1) 100 : audit [INF] from='client.? 172.21.15.45:0/950621650' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:28:38.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:38 smithi045 conmon[33655]: 2021-07-31T02:28:38.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:38 smithi045 conmon[33655]: audit 2021-07-31 2021-07-31T02:28:38.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:38 smithi045 conmon[33655]: T02:28:37.364312+0000 mon.a (mon.0) 2061 2021-07-31T02:28:38.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:38 smithi045 conmon[33655]: : 2021-07-31T02:28:38.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:38 smithi045 conmon[33655]: audit 2021-07-31T02:28:38.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:38 smithi045 conmon[33655]: 2021-07-31T02:28:38.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:38 smithi045 conmon[33655]: [INF] 2021-07-31T02:28:38.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:38 smithi045 conmon[33655]: 2021-07-31T02:28:38.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:38 smithi045 conmon[33655]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:28:38.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:38 smithi045 conmon[33655]: 2021-07-31T02:28:39.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:39 smithi154 conmon[34553]: audit 2021-07-31T02:28:38.363892+0000 mon.a (mon.0) 2062 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-6","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:28:39.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:39 smithi154 conmon[34553]: cluster 2021-07-31T02:28:38.364327+0000 mon.a (mon.0) 2063 : cluster [DBG] osdmap e1139: 8 total, 8 up, 8 in 2021-07-31T02:28:39.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:39 smithi154 conmon[34553]: cluster 2021-07-31T02:28:38.392164+0000 mgr.x (mgr.24542) 1777 : cluster [DBG] pgmap v2228: 41 pgs: 32 unknown, 9 active+clean; 4.6 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:28:39.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:39 smithi045 conmon[33655]: audit 2021-07-31T02:28:38.363892+0000 2021-07-31T02:28:39.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:39 smithi045 conmon[33655]: mon.a (mon.0) 2062 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-6","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:28:39.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:39 smithi045 conmon[33655]: cluster 2021-07-31T02:28:38.364327+0000 mon.a 2021-07-31T02:28:39.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:39 smithi045 conmon[33655]: (mon.0) 2063 : cluster [DBG] osdmap e1139: 8 total, 8 up, 8 in 2021-07-31T02:28:39.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:39 smithi045 conmon[33655]: cluster 2021 2021-07-31T02:28:39.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:39 smithi045 conmon[33655]: -07-31T02:28:38.392164+0000 mgr.x (mgr.24542) 1777 : cluster [DBG] pgmap v2228: 41 pgs: 32 unknown, 9 active+clean; 4.6 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:28:39.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:39 smithi045 conmon[39718]: audit 2021-07-31T02:28:38.363892+0000 2021-07-31T02:28:39.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:39 smithi045 conmon[39718]: mon.a (mon.0) 2062 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-6","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:28:39.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:39 smithi045 conmon[39718]: cluster 2021-07-31T02:28:39.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:39 smithi045 conmon[39718]: 2021-07-31T02:28:38.364327+0000 mon.a (mon.0) 2063 2021-07-31T02:28:39.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:39 smithi045 conmon[39718]: : cluster [DBG] osdmap e1139: 8 total, 8 up, 8 in 2021-07-31T02:28:39.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:39 smithi045 conmon[39718]: cluster 2021-07 2021-07-31T02:28:39.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:39 smithi045 conmon[39718]: -31T02:28:38.392164+0000 mgr.x (mgr.24542) 2021-07-31T02:28:39.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:39 smithi045 conmon[39718]: 1777 : cluster [DBG] pgmap v2228: 41 pgs: 32 unknown, 9 active+clean; 4.6 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:28:40.400 INFO:tasks.workunit.client.0.smithi045.stdout: wrote [246316~71616,1040217~61287,8185375~53000,11665221~5002,12270132~10296,13269891~49600,14001286~8561,14895675~49865,19875549~73168,20865495~1187] 2021-07-31T02:28:40.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:40 smithi154 conmon[34553]: cluster 2021-07-31T02:28:39.376882+0000 mon.a (mon.0) 2064 : cluster [DBG] osdmap e1140: 8 total, 8 up, 8 in 2021-07-31T02:28:40.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:40 smithi045 conmon[39718]: cluster 2021-07-31T02:28:39.376882+0000 mon.a (mon 2021-07-31T02:28:40.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:40 smithi045 conmon[39718]: .0) 2064 : cluster [DBG] osdmap e1140: 8 total, 8 up, 8 in 2021-07-31T02:28:40.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:40 smithi045 conmon[33655]: cluster 2021-07-31T02:28:39. 2021-07-31T02:28:40.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:40 smithi045 conmon[33655]: 376882+0000 mon.a (mon.0) 2064 : cluster [DBG] osdmap e1140: 8 total, 8 up, 8 in 2021-07-31T02:28:40.958 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:28:41.238 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:28:41.239 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 43m ago 44m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:28:41.239 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 41m ago 44m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:28:41.239 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 41m ago 47m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:28:41.239 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 43m ago 49m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:28:41.240 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 43m ago 49m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:28:41.240 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 41m ago 47m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:28:41.240 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 43m ago 48m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:28:41.240 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 43m ago 44m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:28:41.241 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 41m ago 44m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:28:41.241 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 43m ago 47m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:28:41.241 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 43m ago 46m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:28:41.241 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 43m ago 46m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:28:41.241 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 43m ago 46m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:28:41.242 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 41m ago 45m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:28:41.242 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 41m ago 45m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:28:41.242 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 41m ago 45m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:28:41.242 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 41m ago 44m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:28:41.243 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 41m ago 44m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:28:41.396 INFO:tasks.workunit.client.0.smithi045.stdout: wrote [78851~147119,1425002~96861,2497999~91127,3200951~40015,8059987~35074,8289289~91408,11191554~23715,15348116~83573,20138951~91380] 2021-07-31T02:28:41.400 INFO:tasks.workunit.client.0.smithi045.stdout: diff was [78851~147119,1101504~2139462,8059987~35074,8238375~142322,11191554~23715,14945540~486149,20138951~91380] 2021-07-31T02:28:41.402 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] DiffIterateTest/0.DiffIterate (5012 ms) 2021-07-31T02:28:41.402 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] DiffIterateTest/0.DiffIterateDiscard 2021-07-31T02:28:41.403 INFO:tasks.workunit.client.0.smithi045.stdout:using old format 2021-07-31T02:28:41.415 INFO:tasks.workunit.client.0.smithi045.stdout:iterate_cb 0~256 2021-07-31T02:28:41.581 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:28:41.581 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:28:41.581 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:28:41.582 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:28:41.582 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:28:41.582 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:28:41.582 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:28:41.583 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:28:41.583 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:28:41.583 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:28:41.583 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:28:41.583 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:28:41.584 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:28:41.584 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:28:41.584 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:28:41.584 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:28:41.585 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:28:41.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:41 smithi154 conmon[34553]: cluster 2021-07-31T02:28:40.382825+0000 mon.a (mon.0) 2065 : cluster [DBG] 2021-07-31T02:28:41.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:41 smithi154 conmon[34553]: osdmap e1141: 8 total, 8 up, 8 in 2021-07-31T02:28:41.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:41 smithi154 conmon[34553]: cluster 2021-07-31T02:28:40.392706+0000 mgr.x (mgr.24542) 1778 : cluster [DBG] pgmap v2231: 41 pgs: 41 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:28:41.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:41 smithi154 conmon[34553]: audit 2021-07-31T02:28:40 2021-07-31T02:28:41.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:41 smithi154 conmon[34553]: .946444+0000 mgr.x (mgr.24542) 1779 : audit [DBG] from='client.27106 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:28:41.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:41 smithi154 conmon[34553]: audit 2021-07-31T02:28:41.230732+0000 mgr.x (mgr. 2021-07-31T02:28:41.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:41 smithi154 conmon[34553]: 24542) 1780 : audit [DBG] from='client.17175 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:28:41.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:41 smithi045 conmon[33655]: cluster 2021-07-31T02:28:40.382825+0000 2021-07-31T02:28:41.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:41 smithi045 conmon[33655]: mon.a (mon.0) 2065 : cluster [DBG] osdmap e1141: 8 total, 8 up, 8 in 2021-07-31T02:28:41.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:41 smithi045 conmon[33655]: cluster 2021-07-31T02:28:40.392706+0000 mgr.x (mgr.24542) 1778 : cluster [DBG] 2021-07-31T02:28:41.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:41 smithi045 conmon[33655]: pgmap v2231: 41 pgs: 41 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:28:41.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:41 smithi045 conmon[33655]: audit 2021 2021-07-31T02:28:41.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:41 smithi045 conmon[33655]: -07-31T02:28:40.946444+0000 mgr.x (mgr 2021-07-31T02:28:41.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:41 smithi045 conmon[33655]: .24542) 1779 : audit [DBG] from='client.27106 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:28:41.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:41 smithi045 conmon[33655]: audit 2021-07-31T02:28:41 2021-07-31T02:28:41.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:41 smithi045 conmon[33655]: .230732+0000 mgr.x (mgr.24542) 1780 : audit [DBG] from='client.17175 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:28:41.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:41 smithi045 conmon[33655]: 2021-07-31T02:28:41.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:41 smithi045 conmon[39718]: cluster 2021-07-31T02:28:40. 2021-07-31T02:28:41.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:41 smithi045 conmon[39718]: 382825+0000 mon.a (mon.0) 2065 : cluster [DBG] osdmap e1141: 8 total, 8 up, 8 in 2021-07-31T02:28:41.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:41 smithi045 conmon[39718]: cluster 2021-07-31T02:28:41.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:41 smithi045 conmon[39718]: 2021-07-31T02:28:40.392706 2021-07-31T02:28:41.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:41 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 1778 : cluster [DBG] 2021-07-31T02:28:41.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:41 smithi045 conmon[39718]: pgmap v2231: 41 pgs: 41 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:28:41.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:41 smithi045 conmon[39718]: audit 2021-07-31T02:28:40 2021-07-31T02:28:41.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:41 smithi045 conmon[39718]: .946444+0000 mgr.x (mgr.24542) 1779 : 2021-07-31T02:28:41.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:41 smithi045 conmon[39718]: audit [DBG] from='client.27106 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:28:41.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:41 smithi045 conmon[39718]: audit 2021-07-31T02:28:41 2021-07-31T02:28:41.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:41 smithi045 conmon[39718]: .230732+0000 mgr.x (mgr.24542) 1780 : audit [DBG] 2021-07-31T02:28:41.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:41 smithi045 conmon[39718]: from='client.17175 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:28:42.407 INFO:tasks.workunit.client.0.smithi045.stdout:iterate_cb 0~256 2021-07-31T02:28:42.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:42 smithi154 conmon[34553]: cluster 2021-07-31T02:28:41.387450+0000 mon.a (mon.0) 2066 : cluster 2021-07-31T02:28:42.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:42 smithi154 conmon[34553]: [DBG] osdmap e1142: 8 total, 8 up, 8 in 2021-07-31T02:28:42.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:42 smithi154 conmon[34553]: audit 2021-07-31T02:28:41.581475+0000 mon.a (mon.0) 2067 : audit [DBG] from='client.? 172.21.15.45:0/2266906555' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:28:42.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:42 smithi045 conmon[33655]: cluster 2021-07-31T02:28:41.387450+0000 mon.a (mon.0) 2066 : cluster [DBG] osdmap e1142: 8 total, 8 up, 8 in 2021-07-31T02:28:42.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:42 smithi045 conmon[33655]: audit 2021-07-31T02:28:41.581475+0000 mon.a (mon.0) 2067 : audit [DBG] from='client.? 172.21.15.45:0/2266906555' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:28:42.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:42 smithi045 conmon[39718]: cluster 2021-07-31T02:28:41.387450+0000 mon.a (mon.0) 2066 : 2021-07-31T02:28:42.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:42 smithi045 conmon[39718]: cluster [DBG] osdmap e1142: 8 total, 8 up, 8 in 2021-07-31T02:28:42.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:42 smithi045 conmon[39718]: audit 2021-07-31T02:28:41.581475+0000 mon.a (mon.0) 2067 : audit [DBG] from='client.? 172.21.15.45:0/2266906555' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:28:43.415 INFO:tasks.workunit.client.0.smithi045.stdout:iterate_cb 0~256 2021-07-31T02:28:43.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:43 smithi154 conmon[34553]: cluster 2021-07-31T02:28:42.393112+0000 mgr.x (mgr.24542) 1781 : cluster [DBG] pgmap v2233: 41 pgs: 41 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:28:43.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:43 smithi154 conmon[34553]: cluster 2021-07-31T02:28:42.394898+0000 mon.a (mon.0) 2068 : cluster [DBG] osdmap e1143: 8 total, 8 up, 8 in 2021-07-31T02:28:43.775 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:28:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:28:43.581Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:28:43.775 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:28:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:28:43.581Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:28:43.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:28:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:28:43.581Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:28:43.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:43 smithi045 conmon[33655]: cluster 2021-07-31T02:28:42.393112+0000 2021-07-31T02:28:43.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:43 smithi045 conmon[33655]: mgr.x (mgr.24542) 1781 : cluster [DBG] pgmap v2233: 41 pgs: 41 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:28:43.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:43 smithi045 conmon[33655]: cluster 2021-07-31T02:28: 2021-07-31T02:28:43.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:43 smithi045 conmon[33655]: 42.394898+0000 mon.a (mon.0) 2068 : cluster [DBG] osdmap e1143: 8 total, 8 up, 8 in 2021-07-31T02:28:43.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:43 smithi045 conmon[39718]: cluster 2021-07-31T02:28:42 2021-07-31T02:28:43.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:43 smithi045 conmon[39718]: .393112+0000 mgr.x (mgr.24542) 1781 : cluster [DBG] pgmap v2233: 41 pgs: 41 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:28:43.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:43 smithi045 conmon[39718]: cluster 2021-07-31T02:28:43.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:43 smithi045 conmon[39718]: 2021-07-31T02:28:42.394898+0000 mon.a (mon.0) 2068 : cluster [DBG] osdmap e1143: 8 total, 8 up, 8 in 2021-07-31T02:28:44.417 INFO:tasks.workunit.client.0.smithi045.stdout:iterate_cb 0~256 2021-07-31T02:28:44.421 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] DiffIterateTest/0.DiffIterateDiscard (3019 ms) 2021-07-31T02:28:44.421 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] DiffIterateTest/0.DiffIterateStress 2021-07-31T02:28:44.421 INFO:tasks.workunit.client.0.smithi045.stdout:using old format 2021-07-31T02:28:44.472 INFO:tasks.workunit.client.0.smithi045.stdout: i=0 exists [28700819~7542195,54069806~7818859,70135526~2486670,113255843~3168203,244488868~73526,301213740~1522530,308947723~4847144,353065352~1140030,372845341~1074280] wrote [28700819~7542195,54069806~7818859,70135526~2486670,113255843~3168203,244488868~73526,301213740~1522530,308947723~4847144,353065352~1140030,372845341~1074280] 2021-07-31T02:28:44.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:44 smithi154 conmon[34553]: cluster 2021-07-31T02:28:43.402630+0000 mon.a (mon.0) 2069 : cluster 2021-07-31T02:28:44.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:44 smithi154 conmon[34553]: [DBG] osdmap e1144: 8 total, 8 up, 8 in 2021-07-31T02:28:44.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:44 smithi045 conmon[33655]: cluster 2021-07-31T02:28:43.402630+0000 mon.a (mon.0) 2069 : cluster 2021-07-31T02:28:44.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:44 smithi045 conmon[33655]: [DBG] osdmap e1144: 8 total, 8 up, 8 in 2021-07-31T02:28:44.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:44 smithi045 conmon[39718]: cluster 2021-07-31T02:28:43. 2021-07-31T02:28:44.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:44 smithi045 conmon[39718]: 402630+0000 mon.a (mon.0) 2069 : cluster [DBG] osdmap e1144: 8 total, 8 up, 8 in 2021-07-31T02:28:45.483 INFO:tasks.workunit.client.0.smithi045.stdout: i=1 exists [28700819~7542195,54069806~12336304,70135526~2486670,97713673~7130343,113255843~3168203,168270518~1029788,192512896~3147079,236085312~4054340,244488868~73526,264644458~6400300,301213740~1522530,308947723~4847144,351131954~7035272,372845341~1074280,407624981~5893133] wrote [31927003~3593700,54287633~1005145,61127569~5278541,97713673~7130343,168270518~1029788,192512896~3147079,236085312~4054340,264644458~6400300,351131954~7035272,407624981~5893133] 2021-07-31T02:28:45.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:45 smithi154 conmon[34553]: cluster 2021-07-31T02:28:44.393857+0000 mgr.x (mgr.24542) 1782 : cluster [DBG] pgmap v2236: 41 pgs: 41 active+clean; 22 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.2 KiB/s rd, 270 KiB/s wr, 18 op/s 2021-07-31T02:28:45.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:45 smithi154 conmon[34553]: cluster 2021-07-31T02:28:44.406941+0000 mon.a (mon.0) 2070 : cluster [DBG] osdmap e1145: 8 total, 8 up, 8 in 2021-07-31T02:28:45.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:45 smithi045 conmon[33655]: cluster 2021-07-31T02:28:44.393857+0000 mgr.x (mgr. 2021-07-31T02:28:45.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:45 smithi045 conmon[33655]: 24542) 1782 : cluster [DBG] pgmap v2236: 41 pgs: 41 active+clean; 22 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.2 KiB/s rd, 270 KiB/s wr, 18 op/s 2021-07-31T02:28:45.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:45 smithi045 conmon[33655]: cluster 2021-07-31T02:28:44.406941+0000 mon.a (mon.0) 2070 : 2021-07-31T02:28:45.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:45 smithi045 conmon[33655]: cluster [DBG] osdmap e1145: 8 total, 8 up, 8 in 2021-07-31T02:28:45.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:45 smithi045 conmon[39718]: cluster 2021-07-31T02:28:44.393857+0000 mgr.x (mgr.24542) 1782 : cluster [DBG] pgmap v2236: 41 pgs: 41 active+clean; 22 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.2 KiB/s rd, 270 KiB/s wr, 18 op/s 2021-07-31T02:28:45.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:45 smithi045 conmon[39718]: cluster 2021-07-31T02:28:44.406941+0000 mon.a (mon.0) 2070 : cluster [DBG] osdmap e1145: 8 total, 8 up, 8 in 2021-07-31T02:28:46.430 INFO:tasks.workunit.client.0.smithi045.stdout: i=2 exists [28700819~7542195,54069806~12336304,70135526~2486670,97713673~7130343,113255843~3168203,151774456~3015875,154848333~968134,168270518~1029788,192512896~4411164,202609068~1036125,211528859~7869203,236085312~4054340,244488868~73526,249937343~665922,264644458~6400300,292729238~310220,301213740~1522530,308947723~4847144,328597020~2837810,351131954~7035272,372845341~1074280,389494265~1241584,407624981~5893133] wrote [151774456~3015875,154848333~968134,193394214~3529846,202609068~1036125,211528859~7869203,249937343~665922,265869750~2844406,292729238~310220,328597020~2837810,389494265~1241584] 2021-07-31T02:28:46.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:46 smithi154 conmon[34553]: cluster 2021-07-31T02:28:45.412761+0000 mon.a ( 2021-07-31T02:28:46.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:46 smithi154 conmon[34553]: mon.0) 2071 : cluster [DBG] osdmap e1146: 8 total, 8 up, 8 in 2021-07-31T02:28:46.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:46 smithi045 conmon[33655]: cluster 2021-07-31T02:28:45.412761+0000 mon.a (mon. 2021-07-31T02:28:46.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:46 smithi045 conmon[33655]: 0) 2071 : cluster [DBG] osdmap e1146: 8 total, 8 up, 8 in 2021-07-31T02:28:46.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:46 smithi045 conmon[39718]: cluster 2021-07-31T02:28:45.412761+0000 mon.a (mon.0) 2071 2021-07-31T02:28:46.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:46 smithi045 conmon[39718]: : cluster [DBG] osdmap e1146: 8 total, 8 up, 8 in 2021-07-31T02:28:47.332 INFO:tasks.workunit.client.0.smithi045.stdout: i=3 exists [24697164~11545850,47779655~367499,48770053~17636057,70135526~2486670,89659479~3999082,97713673~7130343,113255843~3168203,151774456~3015875,154848333~968134,168270518~1029788,191717187~6799895,202609068~1036125,211528859~7869203,225566155~7953768,236085312~4054340,244488868~73526,249937343~665922,264644458~6400300,292729238~310220,298647376~7010387,308947723~4847144,328597020~2837810,351131954~7035272,372845341~1074280,389494265~1241584,407624981~5893133] wrote [24697164~11073145,47779655~367499,48770053~5969751,89659479~3999082,191717187~6799895,225566155~7953768,298647376~7010387] 2021-07-31T02:28:47.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:47 smithi154 conmon[34553]: cluster 2021-07-31T02:28:46.394302+0000 mgr.x (mgr.24542) 1783 : cluster [DBG] pgmap v2239: 41 pgs: 41 active+clean; 22 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.2 KiB/s rd, 270 KiB/s wr, 18 op/s 2021-07-31T02:28:47.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:47 smithi154 conmon[34553]: cluster 2021-07-31T02:28:46.418082+0000 mon.a (mon.0) 2072 : cluster [DBG] osdmap e1147: 8 total, 8 up, 8 in 2021-07-31T02:28:47.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:47 smithi154 conmon[34553]: cluster 2021-07-31T 2021-07-31T02:28:47.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:47 smithi154 conmon[34553]: 02:28:47.269127+0000 mon.a (mon.0) 2073 : cluster [DBG] osdmap e1148: 8 total, 8 up, 8 in 2021-07-31T02:28:47.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:47 smithi045 conmon[33655]: cluster 2021-07-31T02:28: 2021-07-31T02:28:47.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:47 smithi045 conmon[33655]: 46.394302+0000 mgr.x (mgr.24542) 1783 : cluster [DBG] pgmap v2239: 41 pgs: 41 active+clean; 22 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.2 KiB/s rd, 270 KiB/s wr, 18 op/s 2021-07-31T02:28:47.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:47 smithi045 conmon[33655]: cluster 2021-07 2021-07-31T02:28:47.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:47 smithi045 conmon[33655]: -31T02:28:46.418082+0000 mon.a (mon.0) 2072 : cluster [DBG] osdmap e1147: 8 total, 8 up, 8 in 2021-07-31T02:28:47.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:47 smithi045 conmon[33655]: 2021-07-31T02:28:47.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:47 smithi045 conmon[33655]: cluster 2021-07-31T02:28:47.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:47 smithi045 conmon[33655]: 2021-07-31T02 2021-07-31T02:28:47.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:47 smithi045 conmon[33655]: :28:47.269127+0000 2021-07-31T02:28:47.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:47 smithi045 conmon[33655]: mon.a (mon.0) 2021-07-31T02:28:47.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:47 smithi045 conmon[33655]: 2073 : cluster [DBG] osdmap e1148: 8 total, 8 up, 8 in 2021-07-31T02:28:47.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:47 smithi045 conmon[33655]: 2021-07-31T02:28:47.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:47 smithi045 conmon[39718]: cluster 2021-07-31T02: 2021-07-31T02:28:47.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:47 smithi045 conmon[39718]: 28:46.394302+0000 mgr.x (mgr.24542) 1783 : cluster [DBG] pgmap v2239: 41 pgs: 41 active+clean; 22 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.2 KiB/s rd, 270 KiB/s wr, 18 op/s 2021-07-31T02:28:47.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:47 smithi045 conmon[39718]: cluster 2021-07 2021-07-31T02:28:47.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:47 smithi045 conmon[39718]: -31T02:28:46.418082+0000 mon.a (mon. 2021-07-31T02:28:47.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:47 smithi045 conmon[39718]: 0) 2072 : cluster [DBG] osdmap e1147: 8 total, 8 up, 8 in 2021-07-31T02:28:47.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:47 smithi045 conmon[39718]: 2021-07-31T02:28:47.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:47 smithi045 conmon[39718]: cluster 2021-07-31T02:28:47.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:47 smithi045 conmon[39718]: 2021-07-31T02:28:47.269127+0000 2021-07-31T02:28:47.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:47 smithi045 conmon[39718]: mon.a (mon.0) 2073 : 2021-07-31T02:28:47.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:47 smithi045 conmon[39718]: cluster [DBG] osdmap e1148: 8 total, 8 up, 8 in 2021-07-31T02:28:48.329 INFO:tasks.workunit.client.0.smithi045.stdout: i=4 exists [677713~11779869,24697164~11545850,47779655~367499,48159474~18246636,70135526~2486670,75265061~7871002,89659479~3999082,97713673~7130343,113255843~3168203,125404837~2998498,151774456~3015875,154848333~968134,162635090~2222342,168270518~1029788,191717187~6799895,202609068~1036125,211528859~7869203,225566155~7953768,236085312~4054340,244488868~73526,249937343~665922,264644458~6400300,276020820~3015231,291252461~6684351,298647376~7010387,308947723~4847144,328597020~2837810,344188514~47344,351131954~7035272,360795176~5362515,372845341~1074280,389494265~1241584,407624981~5893133] wrote [677713~11779869,48159474~2619597,75265061~7871002,125404837~2998498,162635090~2222342,276020820~3015231,291252461~6684351,344188514~47344,360795176~5362515] 2021-07-31T02:28:49.338 INFO:tasks.workunit.client.0.smithi045.stdout: i=5 exists [677713~11779869,24697164~11545850,47779655~367499,48159474~18246636,70135526~2486670,75265061~7871002,89659479~3999082,97713673~7130343,113255843~3168203,125404837~2998498,128426182~8003404,151774456~5750484,158077298~3138373,162635090~2222342,168270518~1029788,175704140~5416148,191717187~6799895,202609068~1036125,211528859~7869203,220279147~2389400,225566155~7953768,236085312~4054340,242667678~8137835,264644458~6400300,276020820~3015231,291252461~14405302,308947723~4847144,327653500~3781330,344188514~47344,351131954~7035272,360795176~5362515,367021723~5371848,372845341~1074280,389494265~1241584,407624981~5893133] wrote [128426182~8003404,154322932~3202008,158077298~3138373,175704140~5416148,220279147~2389400,242667678~8137835,296245411~8076833,327653500~3214988,354571565~2088938,367021723~5371848] 2021-07-31T02:28:49.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:49 smithi154 conmon[34553]: cluster 2021-07-31T02:28:48.278922+0000 mon.a (mon.0) 2074 : cluster [DBG] osdmap e1149: 8 total, 8 up, 8 in 2021-07-31T02:28:49.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:49 smithi045 conmon[39718]: cluster 2021-07-31T02:28:48.278922 2021-07-31T02:28:49.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:49 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2074 : cluster [DBG] osdmap e1149: 8 total, 8 up, 8 in 2021-07-31T02:28:49.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:49 smithi045 conmon[33655]: cluster 2021-07-31T02:28:48.278922+0000 2021-07-31T02:28:49.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:49 smithi045 conmon[33655]: mon.a (mon.0) 2074 : cluster [DBG] osdmap e1149: 8 total, 8 up, 8 in 2021-07-31T02:28:50.339 INFO:tasks.workunit.client.0.smithi045.stdout: i=6 exists [677713~11779869,24697164~11545850,39062249~828950,47779655~367499,48159474~18246636,70135526~2486670,75265061~7871002,89659479~3999082,97713673~7130343,107771043~3686162,113255843~3168203,116694067~5293712,125404837~2998498,128426182~8003404,146393293~4680383,151774456~5750484,158077298~3138373,162635090~2222342,168270518~1029788,175704140~5416148,191717187~6799895,202609068~1036125,211528859~7869203,220279147~2389400,225566155~9257544,236085312~4054340,242667678~8137835,264644458~6400300,275231912~5088107,291252461~14405302,308947723~4847144,327653500~3781330,342292118~6052493,351131954~7035272,360795176~5362515,367021723~5371848,372845341~1074280,389494265~1241584,392344733~459593,407624981~5893133] wrote [39062249~828950,107771043~3686162,116694067~5293712,133012892~915417,146393293~4680383,229781426~5042273,275231912~5088107,292264750~4687170,342292118~6052493,392344733~459593] 2021-07-31T02:28:50.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:50 smithi154 conmon[34553]: cluster 2021-07-31T02:28:48.394709+0000 mgr.x (mgr.24542) 1784 : cluster [DBG] pgmap v2243: 41 pgs: 41 active+clean; 22 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:28:50.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:50 smithi154 conmon[34553]: cluster 2021-07-31T02:28:49.285399+0000 mon.a (mon.0) 2075 : cluster [DBG] osdmap e1150: 8 total, 8 up, 8 in 2021-07-31T02:28:50.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:50 smithi045 conmon[39718]: cluster 2021-07-31T02:28:48.394709+0000 mgr.x ( 2021-07-31T02:28:50.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:50 smithi045 conmon[39718]: mgr.24542) 1784 : cluster [DBG] pgmap v2243: 41 pgs: 41 active+clean; 22 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:28:50.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:50 smithi045 conmon[39718]: cluster 2021-07-31T02:28:49.285399 2021-07-31T02:28:50.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:50 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2075 : cluster [DBG] osdmap e1150: 8 total, 8 up, 8 in 2021-07-31T02:28:50.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:50 smithi045 conmon[33655]: cluster 2021-07-31T02:28:48 2021-07-31T02:28:50.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:50 smithi045 conmon[33655]: .394709+0000 mgr.x (mgr.24542) 1784 : cluster [DBG] pgmap v2243: 41 pgs: 41 active+clean; 22 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:28:50.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:50 smithi045 conmon[33655]: cluster 2021-07-31T02:28:50.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:50 smithi045 conmon[33655]: 2021-07-31T02:28:49.285399+0000 mon.a (mon 2021-07-31T02:28:50.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:50 smithi045 conmon[33655]: .0) 2075 : cluster [DBG] osdmap e1150: 8 total, 8 up, 8 in 2021-07-31T02:28:51.333 INFO:tasks.workunit.client.0.smithi045.stdout: i=7 exists [677713~11779869,24697164~11545850,36574959~6759647,47779655~367499,48159474~18246636,69906101~13229962,89659479~3999082,97713673~7130343,107771043~3686162,113255843~3168203,116694067~5293712,125404837~2998498,128426182~8003404,146393293~4680383,151774456~9441215,162635090~2222342,168270518~1029788,175704140~5416148,191717187~6799895,202609068~1036125,211528859~7869203,220279147~2389400,223825249~10998450,236085312~4054340,242667678~8137835,253559427~288594,260438339~10606419,275231912~5088107,291252461~14405302,308947723~4847144,327653500~3781330,331638058~718760,342292118~6052493,351131954~7035272,360795176~5362515,367021723~5371848,372845341~1074280,389494265~1241584,392344733~459593,407624981~5893133] wrote [33435101~869368,36574959~6759647,69906101~6581740,121752631~132598,157303826~1384259,223825249~6528699,253559427~288594,260438339~4519598,331638058~718760] 2021-07-31T02:28:51.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:51 smithi154 conmon[34553]: cluster 2021-07-31T02:28:50.290589+0000 mon.a (mon.0) 2076 : cluster [DBG] 2021-07-31T02:28:51.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:51 smithi154 conmon[34553]: osdmap e1151: 8 total, 8 up, 8 in 2021-07-31T02:28:51.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:51 smithi045 conmon[39718]: cluster 2021-07-31T02:28:50.290589 2021-07-31T02:28:51.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:51 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2076 : cluster [DBG] osdmap e1151: 8 total, 8 up, 8 in 2021-07-31T02:28:51.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:51 smithi045 conmon[33655]: cluster 2021-07-31T02:28:50.290589+0000 2021-07-31T02:28:51.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:51 smithi045 conmon[33655]: mon.a (mon.0) 2076 : cluster [DBG] osdmap e1151: 8 total, 8 up, 8 in 2021-07-31T02:28:52.371 INFO:tasks.workunit.client.0.smithi045.stdout: i=8 exists [677713~17405775,24697164~11545850,36574959~6759647,47779655~367499,48159474~18246636,69906101~13229962,89659479~3999082,97713673~8454237,107771043~3686162,113255843~3168203,116694067~5293712,124734943~11694643,146393293~4680383,151774456~9441215,162635090~2222342,168270518~1029788,175704140~5416148,191717187~6799895,202609068~1036125,211528859~7869203,220279147~2389400,223825249~10998450,236085312~4054340,241317308~9488205,253559427~288594,260438339~10606419,275231912~5088107,291252461~14405302,308947723~4847144,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,342292118~6052493,351131954~15025737,367021723~5371848,372845341~1074280,384392456~2082977,389494265~1241584,392344733~459593,407624981~5893133] wrote [11422525~6660963,102437886~3730024,124734943~4595042,241317308~7450669,318433435~4174606,322615554~4664316,355654064~8384557,384392456~2082977] 2021-07-31T02:28:52.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:52 smithi154 conmon[34553]: cluster 2021-07-31T02:28:50.395157+0000 mgr.x (mgr.24542) 1785 : cluster [DBG] pgmap v2246: 41 pgs: 41 active+clean; 267 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 8.0 KiB/s rd, 47 MiB/s wr, 37 op/s 2021-07-31T02:28:52.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:52 smithi154 conmon[34553]: cluster 2021-07-31T02:28:51.296719+0000 mon.a (mon.0) 2077 : cluster [DBG] osdmap e1152: 8 total, 8 up, 8 in 2021-07-31T02:28:52.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:52 smithi045 conmon[33655]: cluster 2021- 2021-07-31T02:28:52.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:52 smithi045 conmon[33655]: 07-31T02:28:50.395157+0000 mgr.x (mgr.24542) 1785 : cluster [DBG] pgmap v2246: 41 pgs: 41 active+clean; 267 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 8.0 KiB/s rd, 47 MiB/s wr, 37 op/s 2021-07-31T02:28:52.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:52 smithi045 conmon[33655]: cluster 2021-07-31T02:28:51. 2021-07-31T02:28:52.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:52 smithi045 conmon[33655]: 296719+0000 mon.a (mon.0) 2077 : cluster [DBG] osdmap e1152: 8 total, 8 up, 8 in 2021-07-31T02:28:52.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:52 smithi045 conmon[39718]: cluster 2021-07-31T02:28:50.395157 2021-07-31T02:28:52.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:52 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 1785 : cluster [DBG] pgmap v2246: 41 pgs: 41 active+clean; 267 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 8.0 KiB/s rd, 47 MiB/s wr, 37 op/s 2021-07-31T02:28:52.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:52 smithi045 conmon[39718]: cluster 2021-07 2021-07-31T02:28:52.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:52 smithi045 conmon[39718]: -31T02:28:51.296719+0000 mon.a (mon.0) 2077 : 2021-07-31T02:28:52.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:52 smithi045 conmon[39718]: cluster [DBG] osdmap e1152: 8 total, 8 up, 8 in 2021-07-31T02:28:53.375 INFO:tasks.workunit.client.0.smithi045.stdout: i=9 exists [677713~17405775,24697164~18637442,47779655~367499,48159474~18246636,69906101~13229962,89659479~3999082,97713673~8454237,107771043~3686162,113255843~3168203,116694067~5293712,124734943~11694643,146393293~4680383,151774456~9559283,162635090~2222342,167531438~7281081,175704140~5416148,191717187~6799895,201478817~2166376,211528859~7869203,220279147~2389400,223825249~10998450,236085312~4054340,241317308~9488205,253559427~288594,260438339~10606419,275231912~5088107,291252461~14405302,308947723~4847144,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,342292118~23865573,367021723~5371848,372845341~1074280,384392456~2082977,389494265~1241584,392344733~459593,403808658~9709456] wrote [872289~3582298,34802352~7512067,132705403~3160212,156774204~4559535,167531438~7281081,201478817~1199557,347819082~6186929,403808658~5668404] 2021-07-31T02:28:53.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:53 smithi045 conmon[33655]: cluster 2021-07-31T02:28:52.303352+0000 mon.a (mon.0 2021-07-31T02:28:53.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:53 smithi045 conmon[33655]: ) 2078 : cluster [DBG] osdmap e1153: 8 total, 8 up, 8 in 2021-07-31T02:28:53.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:53 smithi045 conmon[39718]: cluster 2021-07-31T02:28:53.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:53 smithi045 conmon[39718]: 2021-07-31T02:28:52. 2021-07-31T02:28:53.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:53 smithi045 conmon[39718]: 303352+0000 mon.a (mon.0) 2078 : cluster [DBG] osdmap e1153: 8 total, 8 up, 8 in 2021-07-31T02:28:53.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:53 smithi154 conmon[34553]: cluster 2021-07-31T02:28:52.303352+0000 mon.a (mon.0) 2078 : cluster [DBG] osdmap e1153: 8 total, 8 up, 8 in 2021-07-31T02:28:54.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:28:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:28:53.581Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:28:54.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:28:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:28:53.581Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:28:54.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:28:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:28:53.581Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:28:54.350 INFO:tasks.workunit.client.0.smithi045.stdout: i=10 exists [677713~17405775,24697164~18637442,47779655~367499,48159474~18246636,69906101~13229962,89659479~3999082,97713673~8454237,107771043~3686162,113255843~3168203,116694067~5293712,124734943~11694643,138274647~3401402,146393293~14940446,162635090~2222342,167531438~7281081,175704140~5416148,191717187~6799895,201478817~2166376,211528859~7869203,220279147~2389400,223825249~10998450,236085312~4054340,241317308~11430657,253559427~288594,260438339~10606419,275231912~5088107,291252461~14405302,308947723~4847144,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,342292118~23865573,367021723~5371848,372845341~3901715,384392456~2082977,389494265~1241584,392344733~459593,403808658~9709456] wrote [24886295~2243069,36634458~572921,79627966~536224,126900852~6720050,138274647~3401402,147600507~7259852,248659714~4088251,262527487~598269,373726074~3020982] 2021-07-31T02:28:54.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:54 smithi154 conmon[34553]: cluster 2021-07-31T02:28:52.395570+0000 mgr.x (mgr.24542) 1786 : cluster [DBG] 2021-07-31T02:28:54.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:54 smithi154 conmon[34553]: pgmap v2249: 41 pgs: 41 active+clean; 267 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 8.0 KiB/s rd, 47 MiB/s wr, 36 op/s 2021-07-31T02:28:54.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:54 smithi154 conmon[34553]: cluster 2021-07-31T02:28:53.310449+0000 mon.a (mon.0) 2079 : cluster [DBG] osdmap e1154: 8 total, 8 up, 8 in 2021-07-31T02:28:54.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:54 smithi045 conmon[39718]: cluster 2021-07-31T02:28:52.395570+0000 mgr.x 2021-07-31T02:28:54.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:54 smithi045 conmon[39718]: (mgr.24542) 1786 : cluster [DBG] pgmap v2249: 41 pgs: 41 active+clean; 267 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 8.0 KiB/s rd, 47 MiB/s wr, 36 op/s 2021-07-31T02:28:54.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:54 smithi045 conmon[39718]: cluster 2021-07-31T02:28:53.310449 2021-07-31T02:28:54.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:54 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2079 : cluster [DBG] osdmap e1154: 8 total, 8 up, 8 in 2021-07-31T02:28:54.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:54 smithi045 conmon[33655]: cluster 2021-07-31T02:28:52.395570+0000 mgr.x 2021-07-31T02:28:54.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:54 smithi045 conmon[33655]: (mgr.24542) 1786 : cluster [DBG] pgmap v2249: 41 pgs: 41 active+clean; 267 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 8.0 KiB/s rd, 47 MiB/s wr, 36 op/s 2021-07-31T02:28:54.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:54 smithi045 conmon[33655]: cluster 2021-07-31T02:28:53.310449+0000 mon.a (mon.0) 2021-07-31T02:28:54.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:54 smithi045 conmon[33655]: 2079 : cluster [DBG] osdmap e1154: 8 total, 8 up, 8 in 2021-07-31T02:28:55.409 INFO:tasks.workunit.client.0.smithi045.stdout: i=11 exists [677713~19484113,24697164~18637442,47779655~367499,48159474~18246636,69906101~13229962,89659479~3999082,97713673~8454237,107771043~3686162,113255843~3168203,116694067~5293712,124734943~11694643,138274647~3401402,146393293~14940446,162635090~2222342,167531438~13588850,191717187~6799895,201478817~2166376,206474279~12923783,220279147~2389400,223825249~10998450,236085312~4054340,241317308~11430657,253559427~288594,260438339~10606419,275231912~5088107,291252461~22542406,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,342292118~30101453,372845341~3901715,381820951~6416356,389494265~6599994,403808658~9998078] wrote [14598685~5563141,78244364~2928921,167792807~7953009,206474279~7964043,242763340~3666547,302959050~7238531,363754003~7447014,381820951~6416356,389902499~6191760,408916890~4889846] 2021-07-31T02:28:55.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:55 smithi154 conmon[34553]: cluster 2021-07-31T02:28:54.315954+0000 mon.a (mon.0) 2080 : cluster 2021-07-31T02:28:55.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:55 smithi154 conmon[34553]: [DBG] osdmap e1155: 8 total, 8 up, 8 in 2021-07-31T02:28:55.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:55 smithi045 conmon[39718]: cluster 2021-07-31T02:28:54.315954+0000 mon.a (mon.0) 2080 2021-07-31T02:28:55.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:55 smithi045 conmon[39718]: : cluster [DBG] osdmap e1155: 8 total, 8 up, 8 in 2021-07-31T02:28:55.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:55 smithi045 conmon[33655]: cluster 2021-07-31T02:28:54.315954+0000 mon.a (mon.0) 2021-07-31T02:28:55.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:55 smithi045 conmon[33655]: 2080 : cluster [DBG] osdmap e1155: 8 total, 8 up, 8 in 2021-07-31T02:28:56.381 INFO:tasks.workunit.client.0.smithi045.stdout: i=12 exists [677713~19484113,24697164~18637442,47779655~367499,48159474~34976589,89659479~3999082,97713673~8454237,107771043~3686162,113255843~3168203,116694067~5293712,124734943~11694643,138274647~3401402,143078160~18255579,162635090~2222342,167531438~13588850,181972347~5415105,191717187~6799895,201478817~2166376,206474279~12923783,220279147~2389400,223825249~10998450,236085312~4054340,241317308~11430657,253559427~288594,260438339~10606419,275231912~5088107,291252461~22542406,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~5080550,342292118~30101453,372845341~6415061,381820951~6416356,389494265~6599994,403808658~9998078] wrote [10475392~4329833,66056524~7829858,143078160~7988495,168673724~4017167,181972347~5415105,296242351~493439,334588993~5080550,343702186~3839518,373441908~5818494,404169452~1052236] 2021-07-31T02:28:56.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:56 smithi154 conmon[34553]: cluster 2021-07-31T02:28:54.396100+0000 mgr.x (mgr.24542) 1787 : cluster 2021-07-31T02:28:56.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:56 smithi154 conmon[34553]: [DBG] pgmap v2252: 41 pgs: 41 active+clean; 486 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 7.5 KiB/s rd, 49 MiB/s wr, 36 op/s 2021-07-31T02:28:56.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:56 smithi154 conmon[34553]: cluster 2021-07-31T02:28:55.322819+0000 mon.a (mon.0) 2081 : cluster [DBG] osdmap e1156: 8 total, 8 up, 8 in 2021-07-31T02:28:56.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:56 smithi045 conmon[33655]: cluster 2021-07-31T02:28:54.396100+0000 mgr.x (mgr.24542) 1787 : cluster 2021-07-31T02:28:56.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:56 smithi045 conmon[33655]: [DBG] pgmap v2252: 41 pgs: 41 active+clean; 486 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 7.5 KiB/s rd, 49 MiB/s wr, 36 op/s 2021-07-31T02:28:56.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:56 smithi045 conmon[33655]: cluster 2021-07-31T02:28:55.322819+0000 mon.a (mon.0) 2081 : cluster [DBG] osdmap e1156: 8 total, 8 up, 8 in 2021-07-31T02:28:56.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:56 smithi045 conmon[39718]: cluster 2021-07-31T02:28:54.396100+0000 mgr.x 2021-07-31T02:28:56.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:56 smithi045 conmon[39718]: (mgr.24542) 1787 : cluster [DBG] pgmap v2252: 41 pgs: 41 active+clean; 486 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 7.5 KiB/s rd, 49 MiB/s wr, 36 op/s 2021-07-31T02:28:56.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:56 smithi045 conmon[39718]: cluster 2021-07-31T02:28:55.322819+0000 mon.a 2021-07-31T02:28:56.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:56 smithi045 conmon[39718]: (mon.0) 2081 : cluster [DBG] osdmap e1156: 8 total, 8 up, 8 in 2021-07-31T02:28:57.409 INFO:tasks.workunit.client.0.smithi045.stdout: i=13 exists [677713~19484113,24697164~18637442,47779655~367499,48159474~34976589,89659479~3999082,97713673~8454237,107771043~3686162,113255843~3168203,116694067~5293712,124734943~11694643,138274647~3401402,143078160~18255579,162635090~2222342,167531438~13588850,181972347~5415105,191717187~6799895,201478817~17919245,220279147~2389400,223825249~10998450,236085312~4054340,241317308~11430657,253324513~6186025,260438339~10606419,275231912~5088107,291252461~22542406,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~5080550,342292118~30101453,372845341~6415061,381820951~6416356,389494265~9114581,403808658~9998078] wrote [27353411~2246694,89752687~3175987,126020102~8880466,174713264~2975701,202180755~9035885,253324513~6186025,362777592~2806618,390462350~8146496] 2021-07-31T02:28:57.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:57 smithi154 conmon[34553]: cluster 2021-07-31T02:28:56.328933+0000 mon.a (mon.0) 2082 : cluster 2021-07-31T02:28:57.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:57 smithi154 conmon[34553]: [DBG] osdmap e1157: 8 total, 8 up, 8 in 2021-07-31T02:28:57.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:57 smithi045 conmon[39718]: cluster 2021-07-31T02:28:56.328933+0000 mon.a ( 2021-07-31T02:28:57.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:57 smithi045 conmon[39718]: mon.0) 2082 : cluster [DBG] osdmap e1157: 8 total, 8 up, 8 in 2021-07-31T02:28:57.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:57 smithi045 conmon[33655]: cluster 2021-07-31T02:28:56.328933+0000 mon.a (mon.0) 2082 2021-07-31T02:28:57.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:57 smithi045 conmon[33655]: : cluster [DBG] osdmap e1157: 8 total, 8 up, 8 in 2021-07-31T02:28:58.414 INFO:tasks.workunit.client.0.smithi045.stdout: i=14 exists [677713~19484113,24697164~18637442,47779655~367499,48159474~34976589,89659479~3999082,97713673~8454237,107771043~3686162,113255843~3168203,116694067~5293712,124734943~11694643,138274647~3401402,143078160~18255579,162635090~2222342,167531438~13588850,181972347~5415105,191717187~6799895,201478817~17919245,220279147~2389400,223825249~10998450,236085312~4054340,241317308~11430657,253324513~6186025,260438339~10606419,275231912~5088107,291252461~22542406,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~5080550,342292118~30101453,372845341~6415061,381820951~6416356,389494265~9114581,403808658~9998078] wrote [25410155~3524489,59300692~1156219,143369911~6779467,154729104~3694539,207917770~6023057,264698001~5703680,301341353~4978213,344585688~1191257] 2021-07-31T02:28:58.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:58 smithi154 conmon[34553]: cluster 2021-07-31T02:28:56.396416+0000 mgr.x (mgr.24542) 1788 : cluster [DBG] pgmap v2255: 41 pgs: 41 active+clean; 486 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 7.5 KiB/s rd, 49 MiB/s wr, 36 op/s 2021-07-31T02:28:58.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:58 smithi154 conmon[34553]: cluster 2021-07-31T02:28:57.334317+0000 mon.a (mon.0) 2083 : cluster [DBG] osdmap e1158: 8 total, 8 up, 8 in 2021-07-31T02:28:58.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:58 smithi045 conmon[33655]: cluster 2021-07-31T02:28:56.396416+0000 mgr.x (mgr.24542) 2021-07-31T02:28:58.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:58 smithi045 conmon[33655]: 1788 : cluster [DBG] pgmap v2255: 41 pgs: 41 active+clean; 486 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 7.5 KiB/s rd, 49 MiB/s wr, 36 op/s 2021-07-31T02:28:58.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:58 smithi045 conmon[33655]: cluster 2021-07-31T02:28:57.334317+0000 mon.a (mon.0) 2083 : cluster [DBG] osdmap e1158: 8 total, 8 up, 8 in 2021-07-31T02:28:58.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:58 smithi045 conmon[39718]: cluster 2021-07-31T02:28:56.396416+0000 2021-07-31T02:28:58.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:58 smithi045 conmon[39718]: mgr.x (mgr.24542) 1788 : cluster [DBG] pgmap v2255: 41 pgs: 41 active+clean; 486 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 7.5 KiB/s rd, 49 MiB/s wr, 36 op/s 2021-07-31T02:28:58.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:58 smithi045 conmon[39718]: cluster 2021-07-31T02:28 2021-07-31T02:28:58.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:58 smithi045 conmon[39718]: :57.334317+0000 mon.a (mon.0) 2083 : cluster [DBG] 2021-07-31T02:28:58.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:58 smithi045 conmon[39718]: osdmap e1158: 8 total, 8 up, 8 in 2021-07-31T02:28:59.415 INFO:tasks.workunit.client.0.smithi045.stdout: i=15 exists [677713~19484113,24697164~18637442,47779655~367499,48159474~34976589,89659479~3999082,97713673~8454237,107771043~3686162,113255843~3168203,116694067~5293712,124734943~11694643,138274647~3401402,143078160~18255579,162635090~2222342,167531438~13588850,181972347~5415105,191717187~6799895,201478817~17919245,220279147~2389400,223825249~10998450,236085312~4054340,241317308~11430657,253324513~6186025,260438339~12248801,275231912~5088107,291252461~22542406,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~37804578,372845341~6415061,381820951~6416356,389494265~9114581,403808658~9998078] wrote [1823868~7597255,147626899~6973069,265182709~7504431,307774986~2757682,335687900~14016330,359181831~3015778,406364931~1096263] 2021-07-31T02:28:59.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:59 smithi154 conmon[34553]: cluster 2021-07-31T02:28:58.340509+0000 mon.a (mon.0) 2084 : cluster [DBG] 2021-07-31T02:28:59.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:28:59 smithi154 conmon[34553]: osdmap e1159: 8 total, 8 up, 8 in 2021-07-31T02:28:59.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:59 smithi045 conmon[39718]: cluster 2021-07-31T02:28:59.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:59 smithi045 conmon[39718]: 2021-07-31T02:28:58.340509 2021-07-31T02:28:59.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:28:59 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2084 : cluster [DBG] osdmap e1159: 8 total, 8 up, 8 in 2021-07-31T02:28:59.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:59 smithi045 conmon[33655]: cluster 2021-07-31T02:28:58.340509+0000 mon.a (mon.0) 2084 : cluster 2021-07-31T02:28:59.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:28:59 smithi045 conmon[33655]: [DBG] osdmap e1159: 8 total, 8 up, 8 in 2021-07-31T02:29:00.429 INFO:tasks.workunit.client.0.smithi045.stdout: i=16 exists [677713~19484113,24372582~18962024,47779655~367499,48159474~34976589,86327888~2219507,89659479~3999082,97713673~8454237,107771043~3686162,113255843~3168203,116694067~7911258,124734943~11694643,138274647~3401402,143078160~18255579,162635090~2222342,167531438~13588850,181972347~5415105,191717187~6799895,201478817~17919245,220279147~2389400,223825249~10998450,236085312~4054340,241317308~11430657,253324513~6186025,260438339~12248801,275231912~5088107,291252461~22542406,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~37804578,372845341~6415061,381820951~6416356,389494265~13486636,403808658~9998078] wrote [24372582~11685962,53817318~2949139,86327888~2219507,121407098~3198227,207946866~2990158,225052374~5309220,391467418~11513483] 2021-07-31T02:29:00.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:00 smithi154 conmon[34553]: cluster 2021-07-31T02:28:58.396804+0000 mgr.x (mgr.24542) 1789 : cluster [DBG] pgmap v2258: 41 pgs: 41 active+clean; 486 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail 2021-07-31T02:29:00.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:00 smithi154 conmon[34553]: cluster 2021-07-31T02:28:59.346160+0000 mon.a (mon.0) 2085 : cluster [DBG] osdmap e1160: 8 total, 8 up, 8 in 2021-07-31T02:29:00.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:00 smithi045 conmon[33655]: cluster 2021-07-31T02:28:58.396804+0000 mgr.x (mgr.24542) 1789 : cluster [DBG] 2021-07-31T02:29:00.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:00 smithi045 conmon[33655]: pgmap v2258: 41 pgs: 41 active+clean; 486 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail 2021-07-31T02:29:00.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:00 smithi045 conmon[33655]: cluster 2021-07-31T02:28:59.346160+0000 mon.a (mon.0) 2085 : cluster [DBG] osdmap e1160: 8 total, 8 up, 8 in 2021-07-31T02:29:00.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:00 smithi045 conmon[39718]: cluster 2021-07-31T02:28:58. 2021-07-31T02:29:00.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:00 smithi045 conmon[39718]: 396804+0000 mgr.x (mgr.24542) 1789 : cluster [DBG] pgmap v2258: 41 pgs: 41 active+clean; 486 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail 2021-07-31T02:29:00.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:00 smithi045 conmon[39718]: cluster 2021-07-31T02:28:59 2021-07-31T02:29:00.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:00 smithi045 conmon[39718]: .346160+0000 mon.a (mon.0) 2085 : 2021-07-31T02:29:00.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:00 smithi045 conmon[39718]: cluster [DBG] osdmap e1160: 8 total, 8 up, 8 in 2021-07-31T02:29:01.393 INFO:tasks.workunit.client.0.smithi045.stdout: i=17 exists [677713~19484113,24372582~18962024,47779655~367499,48159474~34976589,86327888~2219507,89659479~3999082,97713673~8454237,107771043~3686162,113255843~3168203,116694067~7911258,124734943~11694643,137433861~4421821,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~17919245,220279147~2389400,223825249~10998450,236085312~4054340,241317308~11430657,253324513~6186025,260438339~12248801,275231912~5088107,291252461~22542406,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~37804578,372845341~6415061,381820951~6416356,389494265~13486636,403808658~9998078] wrote [9457761~5303041,71563215~1562492,89929518~1882381,99285857~789121,137433861~4421821,157325666~6076679,186487148~881866,335055096~7851089,351585344~2625399,375687458~1062344] 2021-07-31T02:29:01.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:01 smithi154 conmon[34553]: cluster 2021-07-31T02:29:00.352687+0000 mon.a (mon.0) 2086 : cluster [DBG] 2021-07-31T02:29:01.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:01 smithi154 conmon[34553]: osdmap e1161: 8 total, 8 up, 8 in 2021-07-31T02:29:01.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:01 smithi154 conmon[34553]: cluster 2021-07-31T02:29:00.397275+0000 mgr.x (mgr.24542) 1790 : cluster [DBG] pgmap v2261: 41 pgs: 41 active+clean; 693 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 7.5 KiB/s rd, 53 MiB/s wr, 34 op/s 2021-07-31T02:29:01.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:01 smithi045 conmon[39718]: cluster 2021-07-31T02:29:00.352687+0000 mon.a ( 2021-07-31T02:29:01.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:01 smithi045 conmon[39718]: mon.0) 2086 : cluster [DBG] osdmap e1161: 8 total, 8 up, 8 in 2021-07-31T02:29:01.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:01 smithi045 conmon[39718]: cluster 2021-07-31T02:29:00.397275+0000 mgr.x (mgr.24542) 1790 2021-07-31T02:29:01.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:01 smithi045 conmon[39718]: : cluster [DBG] pgmap v2261: 41 pgs: 41 active+clean; 693 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 7.5 KiB/s rd, 53 MiB/s wr, 34 op/s 2021-07-31T02:29:01.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:01 smithi045 conmon[33655]: cluster 2021-07-31T02:29:00.352687+0000 mon.a (mon.0) 2086 2021-07-31T02:29:01.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:01 smithi045 conmon[33655]: : cluster [DBG] osdmap e1161: 8 total, 8 up, 8 in 2021-07-31T02:29:01.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:01 smithi045 conmon[33655]: cluster 2021-07-31T02:29:00.397275+0000 mgr.x (mgr.24542) 1790 : cluster [DBG] pgmap v2261: 41 pgs: 41 active+clean; 693 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 7.5 KiB/s rd, 53 MiB/s wr, 34 op/s 2021-07-31T02:29:02.415 INFO:tasks.workunit.client.0.smithi045.stdout: i=18 exists [677713~19484113,24372582~18962024,47779655~367499,48159474~34976589,86327888~2219507,89659479~3999082,97713673~8454237,107771043~3686162,113255843~3168203,116694067~7911258,124734943~11694643,137433861~4421821,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~17919245,220279147~2389400,223825249~10998450,236085312~4054340,241317308~18193230,260438339~12248801,275231912~5088107,291252461~22542406,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~37804578,372845341~6415061,381820951~21159950,403808658~9998078] wrote [9296466~2171054,31521345~7039855,55515079~1195728,248682238~8485363,264809820~4477895,351737416~6640011,359544279~7960967,383830156~6295455] 2021-07-31T02:29:02.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:02 smithi154 conmon[34553]: cluster 2021-07-31T02:29:01.358841+0000 mon.a (mon.0) 2087 : cluster [DBG] 2021-07-31T02:29:03.173 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:02 smithi154 conmon[34553]: osdmap e1162: 8 total, 8 up, 8 in 2021-07-31T02:29:03.174 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:02 smithi045 conmon[39718]: cluster 2021-07-31T02:29:01.358841+0000 mon.a (mon.0) 2021-07-31T02:29:03.174 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:02 smithi045 conmon[39718]: 2087 : cluster [DBG] osdmap e1162: 8 total, 8 up, 8 in 2021-07-31T02:29:03.175 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:02 smithi045 conmon[33655]: cluster 2021-07-31T02:29:01.358841+0000 mon.a (mon.0) 2087 : cluster 2021-07-31T02:29:03.175 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:02 smithi045 conmon[33655]: [DBG] osdmap e1162: 8 total, 8 up, 8 in 2021-07-31T02:29:03.425 INFO:tasks.workunit.client.0.smithi045.stdout: i=19 exists [677713~20930344,24372582~18962024,47779655~367499,48159474~34976589,86327888~2219507,89659479~3999082,97713673~8454237,107771043~3686162,113255843~3168203,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,236085312~4054340,241317308~18193230,260438339~12248801,275231912~5088107,291252461~22542406,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~37804578,372845341~6415061,381820951~21159950,403808658~9998078] wrote [17882092~3725965,32852874~482850,60609981~5053700,135550801~7285470,169186192~7553729,213575464~7116676,221470799~1383607,305074438~5008577,360539103~2306471] 2021-07-31T02:29:03.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:03 smithi154 conmon[34553]: cluster 2021-07-31T02:29:02.366201+0000 mon.a (mon.0) 2088 : cluster 2021-07-31T02:29:03.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:03 smithi154 conmon[34553]: [DBG] osdmap e1163: 8 total, 8 up, 8 in 2021-07-31T02:29:03.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:03 smithi154 conmon[34553]: cluster 2021-07-31T02:29:02.397608+0000 mgr.x (mgr.24542) 1791 : cluster [DBG] pgmap v2264: 41 pgs: 41 active+clean; 693 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 7.5 KiB/s rd, 53 MiB/s wr, 34 op/s 2021-07-31T02:29:03.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:03 smithi045 conmon[33655]: cluster 2021-07-31T02:29:02.366201+0000 mon.a (mon.0) 2088 2021-07-31T02:29:03.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:03 smithi045 conmon[33655]: : cluster [DBG] osdmap e1163: 8 total, 8 up, 8 in 2021-07-31T02:29:03.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:03 smithi045 conmon[33655]: cluster 2021-07-31T02:29:02.397608+0000 mgr.x (mgr.24542) 1791 : cluster [DBG] pgmap v2264: 41 pgs: 41 active+clean; 693 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 7.5 KiB/s rd, 53 MiB/s wr, 34 op/s 2021-07-31T02:29:03.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:29:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:29:03.582Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:29:03.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:29:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:29:03.582Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:29:03.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:29:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:29:03.582Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:29:03.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:03 smithi045 conmon[39718]: cluster 2021-07-31T02:29:02.366201+0000 2021-07-31T02:29:03.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:03 smithi045 conmon[39718]: mon.a (mon.0) 2088 : cluster [DBG] osdmap e1163: 8 total, 8 up, 8 in 2021-07-31T02:29:03.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:03 smithi045 conmon[39718]: cluster 2021- 2021-07-31T02:29:03.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:03 smithi045 conmon[39718]: 07-31T02:29:02.397608+0000 mgr.x ( 2021-07-31T02:29:03.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:03 smithi045 conmon[39718]: mgr.24542) 1791 : cluster [DBG] 2021-07-31T02:29:03.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:03 smithi045 conmon[39718]: pgmap v2264: 41 pgs: 41 active+clean; 693 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 7.5 KiB/s rd, 53 MiB/s wr, 34 op/s 2021-07-31T02:29:04.385 INFO:tasks.workunit.client.0.smithi045.stdout:from 0 to 1 diff [31927003~3593700,54287633~1005145,61127569~5278541,97713673~7130343,168270518~1029788,192512896~3147079,236085312~4054340,264644458~6400300,351131954~7035272,407624981~5893133] 2021-07-31T02:29:04.385 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [31927003~3593700,54287633~1005145,61127569~5278541,97713673~7130343,168270518~1029788,192512896~3147079,236085312~4054340,264644458~6400300,351131954~7035272,407624981~5893133] 2021-07-31T02:29:04.396 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [31927003~3593700,54287633~1005145,61127569~5278541,96468992~8375024,167772160~1528146,188743680~6916295,234881024~5258628,264241152~6803606,348127232~10039994,406847488~6670626] 2021-07-31T02:29:04.396 INFO:tasks.workunit.client.0.smithi045.stdout:from 0 to 2 diff [31927003~3593700,54287633~1005145,61127569~5278541,97713673~7130343,151774456~3015875,154848333~968134,168270518~1029788,192512896~4411164,202609068~1036125,211528859~7869203,236085312~4054340,249937343~665922,264644458~6400300,292729238~310220,328597020~2837810,351131954~7035272,389494265~1241584,407624981~5893133] 2021-07-31T02:29:04.397 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [31927003~3593700,54287633~1005145,61127569~5278541,97713673~7130343,151774456~3015875,154848333~968134,168270518~1029788,192512896~4411164,202609068~1036125,211528859~7869203,236085312~4054340,249937343~665922,264644458~6400300,292729238~310220,328597020~2837810,351131954~7035272,389494265~1241584,407624981~5893133] 2021-07-31T02:29:04.403 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [31927003~3593700,54287633~1005145,61127569~5278541,96468992~8375024,150994944~4821523,167772160~1528146,188743680~8180380,201326592~2318601,209715200~9682862,234881024~5258628,247463936~3139329,264241152~6803606,289406976~3632482,327155712~4279118,348127232~10039994,385875968~4859881,406847488~6670626] 2021-07-31T02:29:04.403 INFO:tasks.workunit.client.0.smithi045.stdout:from 0 to 3 diff [24697164~11073145,47779655~367499,48770053~6522725,61127569~5278541,89659479~3999082,97713673~7130343,151774456~3015875,154848333~968134,168270518~1029788,191717187~6799895,202609068~1036125,211528859~7869203,225566155~7953768,236085312~4054340,249937343~665922,264644458~6400300,292729238~310220,298647376~7010387,328597020~2837810,351131954~7035272,389494265~1241584,407624981~5893133] 2021-07-31T02:29:04.403 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [24697164~11073145,47779655~367499,48770053~6522725,61127569~5278541,89659479~3999082,97713673~7130343,151774456~3015875,154848333~968134,168270518~1029788,191717187~6799895,202609068~1036125,211528859~7869203,225566155~7953768,236085312~4054340,249937343~665922,264644458~6400300,292729238~310220,298647376~7010387,328597020~2837810,351131954~7035272,389494265~1241584,407624981~5893133] 2021-07-31T02:29:04.410 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [20971520~14798789,46137344~9155434,61127569~5278541,88080384~5578177,96468992~8375024,150994944~4821523,167772160~1528146,188743680~9773402,201326592~2318601,209715200~9682862,222298112~11221811,234881024~5258628,247463936~3139329,264241152~6803606,289406976~3632482,298647376~7010387,327155712~4279118,348127232~10039994,385875968~4859881,406847488~6670626] 2021-07-31T02:29:04.410 INFO:tasks.workunit.client.0.smithi045.stdout:from 0 to 4 diff [677713~11779869,24697164~11073145,47779655~367499,48159474~7133304,61127569~5278541,75265061~7871002,89659479~3999082,97713673~7130343,125404837~2998498,151774456~3015875,154848333~968134,162635090~2222342,168270518~1029788,191717187~6799895,202609068~1036125,211528859~7869203,225566155~7953768,236085312~4054340,249937343~665922,264644458~6400300,276020820~3015231,291252461~6684351,298647376~7010387,328597020~2837810,344188514~47344,351131954~7035272,360795176~5362515,389494265~1241584,407624981~5893133] 2021-07-31T02:29:04.411 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~11779869,24697164~11073145,47779655~367499,48159474~7133304,61127569~5278541,75265061~7871002,89659479~3999082,97713673~7130343,125404837~2998498,151774456~3015875,154848333~968134,162635090~2222342,168270518~1029788,191717187~6799895,202609068~1036125,211528859~7869203,225566155~7953768,236085312~4054340,249937343~665922,264644458~6400300,276020820~3015231,291252461~6684351,298647376~7010387,328597020~2837810,344188514~47344,351131954~7035272,360795176~5362515,389494265~1241584,407624981~5893133] 2021-07-31T02:29:04.416 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~12457582,20971520~14798789,46137344~9155434,61127569~5278541,72622196~10513867,88080384~5578177,96468992~8375024,121634816~6768519,150994944~4821523,159383552~5473880,167772160~1528146,188743680~9773402,201326592~2318601,209715200~9682862,222298112~11221811,234881024~5258628,247463936~3139329,264241152~6803606,272629760~6406291,289406976~8529836,298647376~7010387,327155712~4279118,343932928~302930,348127232~10039994,360710144~5447547,385875968~4859881,406847488~6670626] 2021-07-31T02:29:04.417 INFO:tasks.workunit.client.0.smithi045.stdout:from 0 to 5 diff [677713~11779869,24697164~11073145,47779655~367499,48159474~7133304,61127569~5278541,75265061~7871002,89659479~3999082,97713673~7130343,125404837~2998498,128426182~8003404,151774456~5750484,158077298~3138373,162635090~2222342,168270518~1029788,175704140~5416148,191717187~6799895,202609068~1036125,211528859~7869203,220279147~2389400,225566155~7953768,236085312~4054340,242667678~8137835,264644458~6400300,276020820~3015231,291252461~14405302,327653500~3781330,344188514~47344,351131954~7035272,360795176~5362515,367021723~5371848,389494265~1241584,407624981~5893133] 2021-07-31T02:29:04.417 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~11779869,24697164~11073145,47779655~367499,48159474~7133304,61127569~5278541,75265061~7871002,89659479~3999082,97713673~7130343,125404837~2998498,128426182~8003404,151774456~5750484,158077298~3138373,162635090~2222342,168270518~1029788,175704140~5416148,191717187~6799895,202609068~1036125,211528859~7869203,220279147~2389400,225566155~7953768,236085312~4054340,242667678~8137835,264644458~6400300,276020820~3015231,291252461~14405302,327653500~3781330,344188514~47344,351131954~7035272,360795176~5362515,367021723~5371848,389494265~1241584,407624981~5893133] 2021-07-31T02:29:04.423 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~12457582,20971520~14798789,46137344~9155434,61127569~5278541,72622196~10513867,88080384~5578177,96468992~8375024,121634816~14794770,150994944~13862488,167772160~1528146,171966464~9153824,188743680~9773402,201326592~2318601,209715200~23804723,234881024~15924489,264241152~6803606,272629760~6406291,289406976~16250787,327155712~4279118,343932928~302930,348127232~10039994,360710144~11683427,385875968~4859881,406847488~6670626] 2021-07-31T02:29:04.424 INFO:tasks.workunit.client.0.smithi045.stdout:from 0 to 6 diff [677713~11779869,24697164~11073145,39062249~828950,47779655~367499,48159474~7133304,61127569~5278541,75265061~7871002,89659479~3999082,97713673~7130343,107771043~3686162,116694067~5293712,125404837~2998498,128426182~8003404,146393293~4680383,151774456~5750484,158077298~3138373,162635090~2222342,168270518~1029788,175704140~5416148,191717187~6799895,202609068~1036125,211528859~7869203,220279147~2389400,225566155~9257544,236085312~4054340,242667678~8137835,264644458~6400300,275231912~5088107,291252461~14405302,327653500~3781330,342292118~6052493,351131954~7035272,360795176~5362515,367021723~5371848,389494265~1241584,392344733~459593,407624981~5893133] 2021-07-31T02:29:04.424 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~11779869,24697164~11073145,39062249~828950,47779655~367499,48159474~7133304,61127569~5278541,75265061~7871002,89659479~3999082,97713673~7130343,107771043~3686162,116694067~5293712,125404837~2998498,128426182~8003404,146393293~4680383,151774456~5750484,158077298~3138373,162635090~2222342,168270518~1029788,175704140~5416148,191717187~6799895,202609068~1036125,211528859~7869203,220279147~2389400,225566155~9257544,236085312~4054340,242667678~8137835,264644458~6400300,275231912~5088107,291252461~14405302,327653500~3781330,342292118~6052493,351131954~7035272,360795176~5362515,367021723~5371848,389494265~1241584,392344733~459593,407624981~5893133] 2021-07-31T02:29:04.429 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~12457582,20971520~14798789,37748736~2142463,46137344~9155434,61127569~5278541,72622196~10513867,88080384~5578177,96468992~8375024,104857600~6599605,116424046~20005540,142606336~22251096,167772160~1528146,171966464~9153824,188743680~9773402,201326592~2318601,209715200~25108499,234881024~15924489,264241152~6803606,272629760~7690259,289406976~16250787,327155712~4279118,339738624~18428602,360710144~11683427,385875968~6928358,406847488~6670626] 2021-07-31T02:29:04.429 INFO:tasks.workunit.client.0.smithi045.stdout:from 0 to 7 diff [677713~11779869,24697164~11073145,36574959~6759647,47779655~367499,48159474~7133304,61127569~5278541,69906101~13229962,89659479~3999082,97713673~7130343,107771043~3686162,116694067~5293712,125404837~2998498,128426182~8003404,146393293~4680383,151774456~9441215,162635090~2222342,168270518~1029788,175704140~5416148,191717187~6799895,202609068~1036125,211528859~7869203,220279147~2389400,223825249~10998450,236085312~4054340,242667678~8137835,253559427~288594,260438339~10606419,275231912~5088107,291252461~14405302,327653500~3781330,331638058~718760,342292118~6052493,351131954~7035272,360795176~5362515,367021723~5371848,389494265~1241584,392344733~459593,407624981~5893133] 2021-07-31T02:29:04.430 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~11779869,24697164~11073145,36574959~6759647,47779655~367499,48159474~7133304,61127569~5278541,69906101~13229962,89659479~3999082,97713673~7130343,107771043~3686162,116694067~5293712,125404837~2998498,128426182~8003404,146393293~4680383,151774456~9441215,162635090~2222342,168270518~1029788,175704140~5416148,191717187~6799895,202609068~1036125,211528859~7869203,220279147~2389400,223825249~10998450,236085312~4054340,242667678~8137835,253559427~288594,260438339~10606419,275231912~5088107,291252461~14405302,327653500~3781330,331638058~718760,342292118~6052493,351131954~7035272,360795176~5362515,367021723~5371848,389494265~1241584,392344733~459593,407624981~5893133] 2021-07-31T02:29:04.436 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~12457582,20971520~14798789,36243014~7091592,46137344~9155434,61127569~5278541,69906101~13229962,88080384~5578177,96468992~8375024,104857600~6599605,116424046~20005540,142606336~22251096,167772160~1528146,171966464~9153824,188743680~9773402,201326592~2318601,209715200~25108499,234881024~15924489,251658240~2189781,260046848~10997910,272629760~7690259,289406976~16250787,327155712~5201106,339738624~18428602,360710144~11683427,385875968~6928358,406847488~6670626] 2021-07-31T02:29:04.436 INFO:tasks.workunit.client.0.smithi045.stdout:from 0 to 8 diff [677713~17405775,24697164~11073145,36574959~6759647,47779655~367499,48159474~7133304,61127569~5278541,69906101~13229962,89659479~3999082,97713673~8454237,107771043~3686162,116694067~5293712,124734943~11694643,146393293~4680383,151774456~9441215,162635090~2222342,168270518~1029788,175704140~5416148,191717187~6799895,202609068~1036125,211528859~7869203,220279147~2389400,223825249~10998450,236085312~4054340,241317308~9488205,253559427~288594,260438339~10606419,275231912~5088107,291252461~14405302,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,342292118~6052493,351131954~15025737,367021723~5371848,384392456~2082977,389494265~1241584,392344733~459593,407624981~5893133] 2021-07-31T02:29:04.436 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~17405775,24697164~11073145,36574959~6759647,47779655~367499,48159474~7133304,61127569~5278541,69906101~13229962,89659479~3999082,97713673~8454237,107771043~3686162,116694067~5293712,124734943~11694643,146393293~4680383,151774456~9441215,162635090~2222342,168270518~1029788,175704140~5416148,191717187~6799895,202609068~1036125,211528859~7869203,220279147~2389400,223825249~10998450,236085312~4054340,241317308~9488205,253559427~288594,260438339~10606419,275231912~5088107,291252461~14405302,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,342292118~6052493,351131954~15025737,367021723~5371848,384392456~2082977,389494265~1241584,392344733~459593,407624981~5893133] 2021-07-31T02:29:04.442 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~18083488,20971520~14798789,36243014~7091592,46137344~9155434,61127569~5278541,69906101~13229962,88080384~5578177,96468992~14988213,116424046~20005540,142606336~22251096,167772160~1528146,171966464~9153824,188743680~9773402,201326592~2318601,209715200~25108499,234881024~15924489,251658240~2189781,260046848~10997910,272629760~7690259,289406976~16250787,314572800~17784018,339738624~32654947,381681664~11122662,406847488~6670626] 2021-07-31T02:29:04.444 INFO:tasks.workunit.client.0.smithi045.stdout:from 0 to 9 diff [677713~17405775,24697164~18637442,47779655~367499,48159474~7133304,61127569~5278541,69906101~13229962,89659479~3999082,97713673~8454237,107771043~3686162,116694067~5293712,124734943~11694643,146393293~4680383,151774456~9559283,162635090~2222342,167531438~7281081,175704140~5416148,191717187~6799895,201478817~2166376,211528859~7869203,220279147~2389400,223825249~10998450,236085312~4054340,241317308~9488205,253559427~288594,260438339~10606419,275231912~5088107,291252461~14405302,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,342292118~23865573,367021723~5371848,384392456~2082977,389494265~1241584,392344733~459593,403808658~9709456] 2021-07-31T02:29:04.444 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~17405775,24697164~18637442,47779655~367499,48159474~7133304,61127569~5278541,69906101~13229962,89659479~3999082,97713673~8454237,107771043~3686162,116694067~5293712,124734943~11694643,146393293~4680383,151774456~9559283,162635090~2222342,167531438~7281081,175704140~5416148,191717187~6799895,201478817~2166376,211528859~7869203,220279147~2389400,223825249~10998450,236085312~4054340,241317308~9488205,253559427~288594,260438339~10606419,275231912~5088107,291252461~14405302,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,342292118~23865573,367021723~5371848,384392456~2082977,389494265~1241584,392344733~459593,403808658~9709456] 2021-07-31T02:29:04.449 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~18083488,20971520~22363086,46137344~9155434,61127569~5278541,69906101~13229962,88080384~5578177,96468992~14988213,116424046~20005540,142606336~38513952,188743680~9773402,201326592~2318601,209715200~25108499,234881024~15924489,251658240~2189781,260046848~10997910,272629760~7690259,289406976~16250787,314572800~17784018,339738624~32654947,381681664~11122662,402653184~10864930] 2021-07-31T02:29:04.449 INFO:tasks.workunit.client.0.smithi045.stdout:from 0 to 10 diff [677713~17405775,24697164~18637442,47779655~367499,48159474~7133304,61127569~5278541,69906101~13229962,89659479~3999082,97713673~8454237,107771043~3686162,116694067~5293712,124734943~11694643,138274647~3401402,146393293~14940446,162635090~2222342,167531438~7281081,175704140~5416148,191717187~6799895,201478817~2166376,211528859~7869203,220279147~2389400,223825249~10998450,236085312~4054340,241317308~11430657,253559427~288594,260438339~10606419,275231912~5088107,291252461~14405302,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,342292118~23865573,367021723~5371848,373726074~3020982,384392456~2082977,389494265~1241584,392344733~459593,403808658~9709456] 2021-07-31T02:29:04.449 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~17405775,24697164~18637442,47779655~367499,48159474~7133304,61127569~5278541,69906101~13229962,89659479~3999082,97713673~8454237,107771043~3686162,116694067~5293712,124734943~11694643,138274647~3401402,146393293~14940446,162635090~2222342,167531438~7281081,175704140~5416148,191717187~6799895,201478817~2166376,211528859~7869203,220279147~2389400,223825249~10998450,236085312~4054340,241317308~11430657,253559427~288594,260438339~10606419,275231912~5088107,291252461~14405302,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,342292118~23865573,367021723~5371848,373726074~3020982,384392456~2082977,389494265~1241584,392344733~459593,403808658~9709456] 2021-07-31T02:29:04.455 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~18083488,20971520~22363086,46137344~9155434,61127569~5278541,69906101~13229962,88080384~5578177,96468992~14988213,116424046~25252003,142606336~38513952,188743680~9773402,201326592~2318601,209715200~25108499,234881024~18966997,260046848~10997910,272629760~7690259,289406976~16250787,314572800~17784018,339738624~32654947,373726074~3020982,381681664~11122662,402653184~10864930] 2021-07-31T02:29:04.455 INFO:tasks.workunit.client.0.smithi045.stdout:from 0 to 11 diff [677713~19484113,24697164~18637442,47779655~367499,48159474~7133304,61127569~5278541,69906101~13229962,89659479~3999082,97713673~8454237,107771043~3686162,116694067~5293712,124734943~11694643,138274647~3401402,146393293~14940446,162635090~2222342,167531438~13588850,191717187~6799895,201478817~2166376,206474279~12923783,220279147~2389400,223825249~10998450,236085312~4054340,241317308~11430657,253559427~288594,260438339~10606419,275231912~5088107,291252461~18945120,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,342292118~30101453,373726074~3020982,381820951~6416356,389494265~6599994,403808658~9998078] 2021-07-31T02:29:04.456 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~19484113,24697164~18637442,47779655~367499,48159474~7133304,61127569~5278541,69906101~13229962,89659479~3999082,97713673~8454237,107771043~3686162,116694067~5293712,124734943~11694643,138274647~3401402,146393293~14940446,162635090~2222342,167531438~13588850,191717187~6799895,201478817~2166376,206474279~12923783,220279147~2389400,223825249~10998450,236085312~4054340,241317308~11430657,253559427~288594,260438339~10606419,275231912~5088107,291252461~18945120,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,342292118~30101453,373726074~3020982,381820951~6416356,389494265~6599994,403808658~9998078] 2021-07-31T02:29:04.461 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~20161826,20971520~22363086,46137344~9155434,61127569~5278541,69906101~13229962,88080384~5578177,96468992~14988213,116424046~25252003,142606336~38513952,188743680~9773402,201326592~2318601,205520896~29302803,234881024~18966997,260046848~10997910,272629760~7690259,289406976~20790605,314572800~17784018,339738624~32654947,373726074~3020982,381681664~14412595,402653184~11153552] 2021-07-31T02:29:04.461 INFO:tasks.workunit.client.0.smithi045.stdout:from 0 to 12 diff [677713~19484113,24697164~18637442,47779655~367499,48159474~7133304,61127569~22008494,89659479~3999082,97713673~8454237,107771043~3686162,116694067~5293712,124734943~11694643,138274647~3401402,143078160~18255579,162635090~2222342,167531438~13588850,181972347~5415105,191717187~6799895,201478817~2166376,206474279~12923783,220279147~2389400,223825249~10998450,236085312~4054340,241317308~11430657,253559427~288594,260438339~10606419,275231912~5088107,291252461~18945120,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~5080550,342292118~30101453,373441908~5818494,381820951~6416356,389494265~6599994,403808658~9998078] 2021-07-31T02:29:04.461 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~19484113,24697164~18637442,47779655~367499,48159474~7133304,61127569~22008494,89659479~3999082,97713673~8454237,107771043~3686162,116694067~5293712,124734943~11694643,138274647~3401402,143078160~18255579,162635090~2222342,167531438~13588850,181972347~5415105,191717187~6799895,201478817~2166376,206474279~12923783,220279147~2389400,223825249~10998450,236085312~4054340,241317308~11430657,253559427~288594,260438339~10606419,275231912~5088107,291252461~18945120,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~5080550,342292118~30101453,373441908~5818494,381820951~6416356,389494265~6599994,403808658~9998078] 2021-07-31T02:29:04.467 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~20161826,20971520~22363086,46137344~9155434,61127569~22008494,88080384~5578177,96468992~14988213,116424046~25252003,142606336~44781116,188743680~9773402,201326592~2318601,205520896~29302803,234881024~18966997,260046848~10997910,272629760~7690259,289406976~20790605,314572800~25096743,339738624~32654947,373441908~5818494,381681664~14412595,402653184~11153552] 2021-07-31T02:29:04.467 INFO:tasks.workunit.client.0.smithi045.stdout:from 0 to 13 diff [677713~19484113,24697164~18637442,47779655~367499,48159474~7133304,61127569~22008494,89659479~3999082,97713673~8454237,107771043~3686162,116694067~5293712,124734943~11694643,138274647~3401402,143078160~18255579,162635090~2222342,167531438~13588850,181972347~5415105,191717187~6799895,201478817~17919245,220279147~2389400,223825249~10998450,236085312~4054340,241317308~11430657,253324513~6186025,260438339~10606419,275231912~5088107,291252461~18945120,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~5080550,342292118~30101453,373441908~5818494,381820951~6416356,389494265~9114581,403808658~9998078] 2021-07-31T02:29:04.468 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~19484113,24697164~18637442,47779655~367499,48159474~7133304,61127569~22008494,89659479~3999082,97713673~8454237,107771043~3686162,116694067~5293712,124734943~11694643,138274647~3401402,143078160~18255579,162635090~2222342,167531438~13588850,181972347~5415105,191717187~6799895,201478817~17919245,220279147~2389400,223825249~10998450,236085312~4054340,241317308~11430657,253324513~6186025,260438339~10606419,275231912~5088107,291252461~18945120,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~5080550,342292118~30101453,373441908~5818494,381820951~6416356,389494265~9114581,403808658~9998078] 2021-07-31T02:29:04.473 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~20161826,20971520~22363086,46137344~9155434,61127569~22008494,88080384~5578177,96468992~14988213,116424046~25252003,142606336~44781116,188743680~9773402,201326592~33497107,234881024~24629514,260046848~10997910,272629760~7690259,289406976~20790605,314572800~25096743,339738624~32654947,373441908~5818494,381681664~16927182,402653184~11153552] 2021-07-31T02:29:04.473 INFO:tasks.workunit.client.0.smithi045.stdout:from 0 to 14 diff [677713~19484113,24697164~18637442,47779655~367499,48159474~7133304,59300692~1156219,61127569~22008494,89659479~3999082,97713673~8454237,107771043~3686162,116694067~5293712,124734943~11694643,138274647~3401402,143078160~18255579,162635090~2222342,167531438~13588850,181972347~5415105,191717187~6799895,201478817~17919245,220279147~2389400,223825249~10998450,236085312~4054340,241317308~11430657,253324513~6186025,260438339~10606419,275231912~5088107,291252461~18945120,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~5080550,342292118~30101453,373441908~5818494,381820951~6416356,389494265~9114581,403808658~9998078] 2021-07-31T02:29:04.474 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~19484113,24697164~18637442,47779655~367499,48159474~7133304,59300692~1156219,61127569~22008494,89659479~3999082,97713673~8454237,107771043~3686162,116694067~5293712,124734943~11694643,138274647~3401402,143078160~18255579,162635090~2222342,167531438~13588850,181972347~5415105,191717187~6799895,201478817~17919245,220279147~2389400,223825249~10998450,236085312~4054340,241317308~11430657,253324513~6186025,260438339~10606419,275231912~5088107,291252461~18945120,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~5080550,342292118~30101453,373441908~5818494,381820951~6416356,389494265~9114581,403808658~9998078] 2021-07-31T02:29:04.480 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~20161826,20971520~22363086,46137344~9155434,59300692~1156219,61127569~22008494,88080384~5578177,96468992~14988213,116424046~25252003,142606336~44781116,188743680~9773402,201326592~33497107,234881024~24629514,260046848~10997910,272629760~7690259,289406976~20790605,314572800~25096743,339738624~32654947,373441908~5818494,381681664~16927182,402653184~11153552] 2021-07-31T02:29:04.480 INFO:tasks.workunit.client.0.smithi045.stdout:from 0 to 15 diff [677713~19484113,24697164~18637442,47779655~367499,48159474~7133304,59300692~1156219,61127569~22008494,89659479~3999082,97713673~8454237,107771043~3686162,116694067~5293712,124734943~11694643,138274647~3401402,143078160~18255579,162635090~2222342,167531438~13588850,181972347~5415105,191717187~6799895,201478817~17919245,220279147~2389400,223825249~10998450,236085312~4054340,241317308~11430657,253324513~6186025,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~37804578,373441908~5818494,381820951~6416356,389494265~9114581,403808658~9998078] 2021-07-31T02:29:04.480 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~19484113,24697164~18637442,47779655~367499,48159474~7133304,59300692~1156219,61127569~22008494,89659479~3999082,97713673~8454237,107771043~3686162,116694067~5293712,124734943~11694643,138274647~3401402,143078160~18255579,162635090~2222342,167531438~13588850,181972347~5415105,191717187~6799895,201478817~17919245,220279147~2389400,223825249~10998450,236085312~4054340,241317308~11430657,253324513~6186025,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~37804578,373441908~5818494,381820951~6416356,389494265~9114581,403808658~9998078] 2021-07-31T02:29:04.487 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~20161826,20971520~22363086,46137344~9155434,59300692~1156219,61127569~22008494,88080384~5578177,96468992~14988213,116424046~25252003,142606336~44781116,188743680~9773402,201326592~33497107,234881024~24629514,260046848~20273171,289406976~21125692,314572800~57820771,373441908~5818494,381681664~16927182,402653184~11153552] 2021-07-31T02:29:04.487 INFO:tasks.workunit.client.0.smithi045.stdout:from 0 to 16 diff [677713~19484113,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,61127569~22008494,86327888~2219507,89659479~3999082,97713673~8454237,107771043~3686162,116694067~7911258,124734943~11694643,138274647~3401402,143078160~18255579,162635090~2222342,167531438~13588850,181972347~5415105,191717187~6799895,201478817~17919245,220279147~2389400,223825249~10998450,236085312~4054340,241317308~11430657,253324513~6186025,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~37804578,373441908~5818494,381820951~6416356,389494265~13486636,403808658~9998078] 2021-07-31T02:29:04.488 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~19484113,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,61127569~22008494,86327888~2219507,89659479~3999082,97713673~8454237,107771043~3686162,116694067~7911258,124734943~11694643,138274647~3401402,143078160~18255579,162635090~2222342,167531438~13588850,181972347~5415105,191717187~6799895,201478817~17919245,220279147~2389400,223825249~10998450,236085312~4054340,241317308~11430657,253324513~6186025,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~37804578,373441908~5818494,381820951~6416356,389494265~13486636,403808658~9998078] 2021-07-31T02:29:04.491 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~20161826,20971520~22363086,46137344~10629113,59300692~1156219,61127569~22008494,83886080~9772481,96468992~14988213,116424046~25252003,142606336~44781116,188743680~9773402,201326592~33497107,234881024~24629514,260046848~20273171,289406976~21125692,314572800~57820771,373441908~5818494,381681664~32125072] 2021-07-31T02:29:04.492 INFO:tasks.workunit.client.0.smithi045.stdout:from 0 to 17 diff [677713~19484113,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,61127569~22008494,86327888~2219507,89659479~3999082,97713673~8454237,107771043~3686162,116694067~7911258,124734943~11694643,137433861~4421821,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~17919245,220279147~2389400,223825249~10998450,236085312~4054340,241317308~11430657,253324513~6186025,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~37804578,373441908~5818494,381820951~6416356,389494265~13486636,403808658~9998078] 2021-07-31T02:29:04.492 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~19484113,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,61127569~22008494,86327888~2219507,89659479~3999082,97713673~8454237,107771043~3686162,116694067~7911258,124734943~11694643,137433861~4421821,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~17919245,220279147~2389400,223825249~10998450,236085312~4054340,241317308~11430657,253324513~6186025,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~37804578,373441908~5818494,381820951~6416356,389494265~13486636,403808658~9998078] 2021-07-31T02:29:04.497 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~20161826,20971520~22363086,46137344~10629113,59300692~1156219,61127569~22008494,83886080~9772481,96468992~14988213,116424046~25431636,142606336~44781116,188743680~9773402,201326592~33497107,234881024~24629514,260046848~20273171,289406976~21125692,314572800~57820771,373441908~5818494,381681664~32125072] 2021-07-31T02:29:04.497 INFO:tasks.workunit.client.0.smithi045.stdout:from 0 to 18 diff [677713~19484113,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,61127569~22008494,86327888~2219507,89659479~3999082,97713673~8454237,107771043~3686162,116694067~7911258,124734943~11694643,137433861~4421821,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~17919245,220279147~2389400,223825249~10998450,236085312~4054340,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:04.497 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~19484113,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,61127569~22008494,86327888~2219507,89659479~3999082,97713673~8454237,107771043~3686162,116694067~7911258,124734943~11694643,137433861~4421821,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~17919245,220279147~2389400,223825249~10998450,236085312~4054340,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:04.503 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~20161826,20971520~22363086,46137344~10629113,59300692~1156219,61127569~22008494,83886080~9772481,96468992~14988213,116424046~25431636,142606336~44781116,188743680~9773402,201326592~33497107,234881024~24629514,260046848~20273171,289406976~21125692,314572800~57820771,373441908~5818494,381681664~32125072] 2021-07-31T02:29:04.503 INFO:tasks.workunit.client.0.smithi045.stdout:from 0 to 19 diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~22526082,86327888~2219507,89659479~3999082,97713673~8454237,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,236085312~4054340,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:04.504 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~22526082,86327888~2219507,89659479~3999082,97713673~8454237,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,236085312~4054340,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:04.509 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~43334606,46137344~10629113,59300692~1156219,60609981~22526082,83886080~9772481,96468992~14988213,116424046~70963406,188743680~9773402,201326592~33497107,234881024~24629514,260046848~20273171,289406976~21125692,314572800~57820771,373441908~5818494,381681664~32125072] 2021-07-31T02:29:04.509 INFO:tasks.workunit.client.0.smithi045.stdout:from 1 to 2 diff [151774456~3015875,154848333~968134,193394214~3529846,202609068~1036125,211528859~7869203,249937343~665922,265869750~2844406,292729238~310220,328597020~2837810,389494265~1241584] 2021-07-31T02:29:04.509 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [151774456~3015875,154848333~968134,193394214~3529846,202609068~1036125,211528859~7869203,249937343~665922,265869750~2844406,292729238~310220,328597020~2837810,389494265~1241584] 2021-07-31T02:29:04.516 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [150994944~4821523,193394214~3529846,201326592~2318601,209715200~9682862,247463936~3139329,265869750~2844406,289406976~3632482,327155712~4279118,385875968~4859881] 2021-07-31T02:29:04.516 INFO:tasks.workunit.client.0.smithi045.stdout:from 1 to 3 diff [24697164~11073145,47779655~367499,48770053~5969751,89659479~3999082,151774456~3015875,154848333~968134,191717187~6799895,202609068~1036125,211528859~7869203,225566155~7953768,249937343~665922,265869750~2844406,292729238~310220,298647376~7010387,328597020~2837810,389494265~1241584] 2021-07-31T02:29:04.516 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [24697164~11073145,47779655~367499,48770053~5969751,89659479~3999082,151774456~3015875,154848333~968134,191717187~6799895,202609068~1036125,211528859~7869203,225566155~7953768,249937343~665922,265869750~2844406,292729238~310220,298647376~7010387,328597020~2837810,389494265~1241584] 2021-07-31T02:29:04.522 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [20971520~14798789,46137344~8602460,88080384~5578177,150994944~4821523,191717187~6799895,201326592~2318601,209715200~9682862,222298112~11221811,247463936~3139329,265869750~2844406,289406976~3632482,298647376~7010387,327155712~4279118,385875968~4859881] 2021-07-31T02:29:04.523 INFO:tasks.workunit.client.0.smithi045.stdout:from 1 to 4 diff [677713~11779869,24697164~11073145,47779655~367499,48159474~6580330,75265061~7871002,89659479~3999082,125404837~2998498,151774456~3015875,154848333~968134,162635090~2222342,191717187~6799895,202609068~1036125,211528859~7869203,225566155~7953768,249937343~665922,265869750~2844406,276020820~3015231,291252461~6684351,298647376~7010387,328597020~2837810,344188514~47344,360795176~5362515,389494265~1241584] 2021-07-31T02:29:04.523 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~11779869,24697164~11073145,47779655~367499,48159474~6580330,75265061~7871002,89659479~3999082,125404837~2998498,151774456~3015875,154848333~968134,162635090~2222342,191717187~6799895,202609068~1036125,211528859~7869203,225566155~7953768,249937343~665922,265869750~2844406,276020820~3015231,291252461~6684351,298647376~7010387,328597020~2837810,344188514~47344,360795176~5362515,389494265~1241584] 2021-07-31T02:29:04.528 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~12457582,20971520~14798789,46137344~8602460,72622196~10513867,88080384~5578177,121634816~6768519,150994944~4821523,159383552~5473880,191717187~6799895,201326592~2318601,209715200~9682862,222298112~11221811,247463936~3139329,265869750~2844406,272629760~6406291,289406976~8529836,298647376~7010387,327155712~4279118,343932928~302930,360710144~5447547,385875968~4859881] 2021-07-31T02:29:04.528 INFO:tasks.workunit.client.0.smithi045.stdout:from 1 to 5 diff [677713~11779869,24697164~11073145,47779655~367499,48159474~6580330,75265061~7871002,89659479~3999082,125404837~2998498,128426182~8003404,151774456~5750484,158077298~3138373,162635090~2222342,175704140~5416148,191717187~6799895,202609068~1036125,211528859~7869203,220279147~2389400,225566155~7953768,242667678~8137835,265869750~2844406,276020820~3015231,291252461~14405302,327653500~3781330,344188514~47344,354571565~2088938,360795176~5362515,367021723~5371848,389494265~1241584] 2021-07-31T02:29:04.529 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~11779869,24697164~11073145,47779655~367499,48159474~6580330,75265061~7871002,89659479~3999082,125404837~2998498,128426182~8003404,151774456~5750484,158077298~3138373,162635090~2222342,175704140~5416148,191717187~6799895,202609068~1036125,211528859~7869203,220279147~2389400,225566155~7953768,242667678~8137835,265869750~2844406,276020820~3015231,291252461~14405302,327653500~3781330,344188514~47344,354571565~2088938,360795176~5362515,367021723~5371848,389494265~1241584] 2021-07-31T02:29:04.532 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~12457582,20971520~14798789,46137344~8602460,72622196~10513867,88080384~5578177,121634816~14794770,150994944~13862488,171966464~9153824,191717187~6799895,201326592~2318601,209715200~23804723,240139652~10665861,265869750~2844406,272629760~6406291,289406976~16250787,327155712~4279118,343932928~302930,354571565~2088938,360710144~11683427,385875968~4859881] 2021-07-31T02:29:04.532 INFO:tasks.workunit.client.0.smithi045.stdout:from 1 to 6 diff [677713~11779869,24697164~11073145,39062249~828950,47779655~367499,48159474~6580330,75265061~7871002,89659479~3999082,107771043~3686162,116694067~5293712,125404837~2998498,128426182~8003404,146393293~4680383,151774456~5750484,158077298~3138373,162635090~2222342,175704140~5416148,191717187~6799895,202609068~1036125,211528859~7869203,220279147~2389400,225566155~9257544,242667678~8137835,265869750~2844406,275231912~5088107,291252461~14405302,327653500~3781330,342292118~6052493,354571565~2088938,360795176~5362515,367021723~5371848,389494265~1241584,392344733~459593] 2021-07-31T02:29:04.533 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~11779869,24697164~11073145,39062249~828950,47779655~367499,48159474~6580330,75265061~7871002,89659479~3999082,107771043~3686162,116694067~5293712,125404837~2998498,128426182~8003404,146393293~4680383,151774456~5750484,158077298~3138373,162635090~2222342,175704140~5416148,191717187~6799895,202609068~1036125,211528859~7869203,220279147~2389400,225566155~9257544,242667678~8137835,265869750~2844406,275231912~5088107,291252461~14405302,327653500~3781330,342292118~6052493,354571565~2088938,360795176~5362515,367021723~5371848,389494265~1241584,392344733~459593] 2021-07-31T02:29:04.538 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~12457582,20971520~14798789,37748736~2142463,46137344~8602460,72622196~10513867,88080384~5578177,104857600~6599605,116424046~20005540,142606336~22251096,171966464~9153824,191717187~6799895,201326592~2318601,209715200~25108499,240139652~10665861,265869750~2844406,272629760~7690259,289406976~16250787,327155712~4279118,339738624~8605987,354571565~2088938,360710144~11683427,385875968~6928358] 2021-07-31T02:29:04.538 INFO:tasks.workunit.client.0.smithi045.stdout:from 1 to 7 diff [677713~11779869,24697164~11073145,36574959~6759647,47779655~367499,48159474~6580330,69906101~13229962,89659479~3999082,107771043~3686162,116694067~5293712,125404837~2998498,128426182~8003404,146393293~4680383,151774456~9441215,162635090~2222342,175704140~5416148,191717187~6799895,202609068~1036125,211528859~7869203,220279147~2389400,223825249~10998450,242667678~8137835,253559427~288594,260438339~4519598,265869750~2844406,275231912~5088107,291252461~14405302,327653500~3781330,331638058~718760,342292118~6052493,354571565~2088938,360795176~5362515,367021723~5371848,389494265~1241584,392344733~459593] 2021-07-31T02:29:04.539 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~11779869,24697164~11073145,36574959~6759647,47779655~367499,48159474~6580330,69906101~13229962,89659479~3999082,107771043~3686162,116694067~5293712,125404837~2998498,128426182~8003404,146393293~4680383,151774456~9441215,162635090~2222342,175704140~5416148,191717187~6799895,202609068~1036125,211528859~7869203,220279147~2389400,223825249~10998450,242667678~8137835,253559427~288594,260438339~4519598,265869750~2844406,275231912~5088107,291252461~14405302,327653500~3781330,331638058~718760,342292118~6052493,354571565~2088938,360795176~5362515,367021723~5371848,389494265~1241584,392344733~459593] 2021-07-31T02:29:04.545 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~12457582,20971520~14798789,36243014~7091592,46137344~8602460,69906101~13229962,88080384~5578177,104857600~6599605,116424046~20005540,142606336~22251096,171966464~9153824,191717187~6799895,201326592~2318601,209715200~25108499,240139652~10665861,251658240~2189781,260046848~4911089,265869750~2844406,272629760~7690259,289406976~16250787,327155712~5201106,339738624~8605987,354571565~2088938,360710144~11683427,385875968~6928358] 2021-07-31T02:29:04.546 INFO:tasks.workunit.client.0.smithi045.stdout:from 1 to 8 diff [677713~17405775,24697164~11073145,36574959~6759647,47779655~367499,48159474~6580330,69906101~13229962,89659479~3999082,102437886~3730024,107771043~3686162,116694067~5293712,124734943~11694643,146393293~4680383,151774456~9441215,162635090~2222342,175704140~5416148,191717187~6799895,202609068~1036125,211528859~7869203,220279147~2389400,223825249~10998450,241317308~9488205,253559427~288594,260438339~4519598,265869750~2844406,275231912~5088107,291252461~14405302,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,342292118~6052493,354571565~11586126,367021723~5371848,384392456~2082977,389494265~1241584,392344733~459593] 2021-07-31T02:29:04.547 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~17405775,24697164~11073145,36574959~6759647,47779655~367499,48159474~6580330,69906101~13229962,89659479~3999082,102437886~3730024,107771043~3686162,116694067~5293712,124734943~11694643,146393293~4680383,151774456~9441215,162635090~2222342,175704140~5416148,191717187~6799895,202609068~1036125,211528859~7869203,220279147~2389400,223825249~10998450,241317308~9488205,253559427~288594,260438339~4519598,265869750~2844406,275231912~5088107,291252461~14405302,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,342292118~6052493,354571565~11586126,367021723~5371848,384392456~2082977,389494265~1241584,392344733~459593] 2021-07-31T02:29:04.550 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~18083488,20971520~14798789,36243014~7091592,46137344~8602460,69906101~13229962,88080384~5578177,102437886~9019319,116424046~20005540,142606336~22251096,171966464~9153824,191717187~6799895,201326592~2318601,209715200~25108499,240139652~10665861,251658240~2189781,260046848~4911089,265869750~2844406,272629760~7690259,289406976~16250787,314572800~17784018,339738624~8605987,354571565~17822006,381681664~11122662] 2021-07-31T02:29:04.550 INFO:tasks.workunit.client.0.smithi045.stdout:from 1 to 9 diff [677713~17405775,24697164~18637442,47779655~367499,48159474~6580330,69906101~13229962,89659479~3999082,102437886~3730024,107771043~3686162,116694067~5293712,124734943~11694643,146393293~4680383,151774456~9559283,162635090~2222342,167531438~7281081,175704140~5416148,191717187~6799895,201478817~2166376,211528859~7869203,220279147~2389400,223825249~10998450,241317308~9488205,253559427~288594,260438339~4519598,265869750~2844406,275231912~5088107,291252461~14405302,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,342292118~11713893,354571565~11586126,367021723~5371848,384392456~2082977,389494265~1241584,392344733~459593,403808658~5668404] 2021-07-31T02:29:04.551 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~17405775,24697164~18637442,47779655~367499,48159474~6580330,69906101~13229962,89659479~3999082,102437886~3730024,107771043~3686162,116694067~5293712,124734943~11694643,146393293~4680383,151774456~9559283,162635090~2222342,167531438~7281081,175704140~5416148,191717187~6799895,201478817~2166376,211528859~7869203,220279147~2389400,223825249~10998450,241317308~9488205,253559427~288594,260438339~4519598,265869750~2844406,275231912~5088107,291252461~14405302,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,342292118~11713893,354571565~11586126,367021723~5371848,384392456~2082977,389494265~1241584,392344733~459593,403808658~5668404] 2021-07-31T02:29:04.556 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~18083488,20971520~22363086,46137344~8602460,69906101~13229962,88080384~5578177,102437886~9019319,116424046~20005540,142606336~38513952,191717187~6799895,201326592~2318601,209715200~25108499,240139652~10665861,251658240~2189781,260046848~4911089,265869750~2844406,272629760~7690259,289406976~16250787,314572800~17784018,339738624~14267387,354571565~17822006,381681664~11122662,402653184~6823878] 2021-07-31T02:29:04.556 INFO:tasks.workunit.client.0.smithi045.stdout:from 1 to 10 diff [677713~17405775,24697164~18637442,47779655~367499,48159474~6580330,69906101~13229962,89659479~3999082,102437886~3730024,107771043~3686162,116694067~5293712,124734943~11694643,138274647~3401402,146393293~14940446,162635090~2222342,167531438~7281081,175704140~5416148,191717187~6799895,201478817~2166376,211528859~7869203,220279147~2389400,223825249~10998450,241317308~11430657,253559427~288594,260438339~4519598,265869750~2844406,275231912~5088107,291252461~14405302,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,342292118~11713893,354571565~11586126,367021723~5371848,373726074~3020982,384392456~2082977,389494265~1241584,392344733~459593,403808658~5668404] 2021-07-31T02:29:04.556 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~17405775,24697164~18637442,47779655~367499,48159474~6580330,69906101~13229962,89659479~3999082,102437886~3730024,107771043~3686162,116694067~5293712,124734943~11694643,138274647~3401402,146393293~14940446,162635090~2222342,167531438~7281081,175704140~5416148,191717187~6799895,201478817~2166376,211528859~7869203,220279147~2389400,223825249~10998450,241317308~11430657,253559427~288594,260438339~4519598,265869750~2844406,275231912~5088107,291252461~14405302,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,342292118~11713893,354571565~11586126,367021723~5371848,373726074~3020982,384392456~2082977,389494265~1241584,392344733~459593,403808658~5668404] 2021-07-31T02:29:04.561 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~18083488,20971520~22363086,46137344~8602460,69906101~13229962,88080384~5578177,102437886~9019319,116424046~25252003,142606336~38513952,191717187~6799895,201326592~2318601,209715200~25108499,240139652~13708369,260046848~4911089,265869750~2844406,272629760~7690259,289406976~16250787,314572800~17784018,339738624~14267387,354571565~17822006,373726074~3020982,381681664~11122662,402653184~6823878] 2021-07-31T02:29:04.561 INFO:tasks.workunit.client.0.smithi045.stdout:from 1 to 11 diff [677713~19484113,24697164~18637442,47779655~367499,48159474~6580330,69906101~13229962,89659479~3999082,102437886~3730024,107771043~3686162,116694067~5293712,124734943~11694643,138274647~3401402,146393293~14940446,162635090~2222342,167531438~13588850,191717187~6799895,201478817~2166376,206474279~12923783,220279147~2389400,223825249~10998450,241317308~11430657,253559427~288594,260438339~4519598,265869750~2844406,275231912~5088107,291252461~18945120,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,342292118~11713893,354571565~17822006,373726074~3020982,381820951~6416356,389494265~6599994,403808658~9998078] 2021-07-31T02:29:04.562 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~19484113,24697164~18637442,47779655~367499,48159474~6580330,69906101~13229962,89659479~3999082,102437886~3730024,107771043~3686162,116694067~5293712,124734943~11694643,138274647~3401402,146393293~14940446,162635090~2222342,167531438~13588850,191717187~6799895,201478817~2166376,206474279~12923783,220279147~2389400,223825249~10998450,241317308~11430657,253559427~288594,260438339~4519598,265869750~2844406,275231912~5088107,291252461~18945120,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,342292118~11713893,354571565~17822006,373726074~3020982,381820951~6416356,389494265~6599994,403808658~9998078] 2021-07-31T02:29:04.567 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~20161826,20971520~22363086,46137344~8602460,69906101~13229962,88080384~5578177,102437886~9019319,116424046~25252003,142606336~38513952,191717187~6799895,201326592~2318601,205520896~29302803,240139652~13708369,260046848~4911089,265869750~2844406,272629760~7690259,289406976~20790605,314572800~17784018,339738624~14267387,354571565~17822006,373726074~3020982,381681664~14412595,402653184~11153552] 2021-07-31T02:29:04.567 INFO:tasks.workunit.client.0.smithi045.stdout:from 1 to 12 diff [677713~19484113,24697164~18637442,47779655~367499,48159474~6580330,66056524~17079539,89659479~3999082,102437886~3730024,107771043~3686162,116694067~5293712,124734943~11694643,138274647~3401402,143078160~18255579,162635090~2222342,167531438~13588850,181972347~5415105,191717187~6799895,201478817~2166376,206474279~12923783,220279147~2389400,223825249~10998450,241317308~11430657,253559427~288594,260438339~4519598,265869750~2844406,275231912~5088107,291252461~18945120,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~5080550,342292118~11713893,354571565~17822006,373441908~5818494,381820951~6416356,389494265~6599994,403808658~9998078] 2021-07-31T02:29:04.567 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~19484113,24697164~18637442,47779655~367499,48159474~6580330,66056524~17079539,89659479~3999082,102437886~3730024,107771043~3686162,116694067~5293712,124734943~11694643,138274647~3401402,143078160~18255579,162635090~2222342,167531438~13588850,181972347~5415105,191717187~6799895,201478817~2166376,206474279~12923783,220279147~2389400,223825249~10998450,241317308~11430657,253559427~288594,260438339~4519598,265869750~2844406,275231912~5088107,291252461~18945120,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~5080550,342292118~11713893,354571565~17822006,373441908~5818494,381820951~6416356,389494265~6599994,403808658~9998078] 2021-07-31T02:29:04.576 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~20161826,20971520~22363086,46137344~8602460,66056524~17079539,88080384~5578177,102437886~9019319,116424046~25252003,142606336~44781116,191717187~6799895,201326592~2318601,205520896~29302803,240139652~13708369,260046848~4911089,265869750~2844406,272629760~7690259,289406976~20790605,314572800~25096743,339738624~14267387,354571565~17822006,373441908~5818494,381681664~14412595,402653184~11153552] 2021-07-31T02:29:04.576 INFO:tasks.workunit.client.0.smithi045.stdout:from 1 to 13 diff [677713~19484113,24697164~18637442,47779655~367499,48159474~6580330,66056524~17079539,89659479~3999082,102437886~3730024,107771043~3686162,116694067~5293712,124734943~11694643,138274647~3401402,143078160~18255579,162635090~2222342,167531438~13588850,181972347~5415105,191717187~6799895,201478817~17919245,220279147~2389400,223825249~10998450,241317308~11430657,253324513~6186025,260438339~4519598,265869750~2844406,275231912~5088107,291252461~18945120,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~5080550,342292118~11713893,354571565~17822006,373441908~5818494,381820951~6416356,389494265~9114581,403808658~9998078] 2021-07-31T02:29:04.576 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~19484113,24697164~18637442,47779655~367499,48159474~6580330,66056524~17079539,89659479~3999082,102437886~3730024,107771043~3686162,116694067~5293712,124734943~11694643,138274647~3401402,143078160~18255579,162635090~2222342,167531438~13588850,181972347~5415105,191717187~6799895,201478817~17919245,220279147~2389400,223825249~10998450,241317308~11430657,253324513~6186025,260438339~4519598,265869750~2844406,275231912~5088107,291252461~18945120,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~5080550,342292118~11713893,354571565~17822006,373441908~5818494,381820951~6416356,389494265~9114581,403808658~9998078] 2021-07-31T02:29:04.579 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~20161826,20971520~22363086,46137344~8602460,66056524~17079539,88080384~5578177,102437886~9019319,116424046~25252003,142606336~44781116,191717187~6799895,201326592~33497107,240139652~19370886,260046848~4911089,265869750~2844406,272629760~7690259,289406976~20790605,314572800~25096743,339738624~14267387,354571565~17822006,373441908~5818494,381681664~16927182,402653184~11153552] 2021-07-31T02:29:04.579 INFO:tasks.workunit.client.0.smithi045.stdout:from 1 to 14 diff [677713~19484113,24697164~18637442,47779655~367499,48159474~6580330,59300692~1156219,66056524~17079539,89659479~3999082,102437886~3730024,107771043~3686162,116694067~5293712,124734943~11694643,138274647~3401402,143078160~18255579,162635090~2222342,167531438~13588850,181972347~5415105,191717187~6799895,201478817~17919245,220279147~2389400,223825249~10998450,241317308~11430657,253324513~6186025,260438339~9963342,275231912~5088107,291252461~18945120,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~5080550,342292118~11713893,354571565~17822006,373441908~5818494,381820951~6416356,389494265~9114581,403808658~9998078] 2021-07-31T02:29:04.579 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~19484113,24697164~18637442,47779655~367499,48159474~6580330,59300692~1156219,66056524~17079539,89659479~3999082,102437886~3730024,107771043~3686162,116694067~5293712,124734943~11694643,138274647~3401402,143078160~18255579,162635090~2222342,167531438~13588850,181972347~5415105,191717187~6799895,201478817~17919245,220279147~2389400,223825249~10998450,241317308~11430657,253324513~6186025,260438339~9963342,275231912~5088107,291252461~18945120,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~5080550,342292118~11713893,354571565~17822006,373441908~5818494,381820951~6416356,389494265~9114581,403808658~9998078] 2021-07-31T02:29:04.585 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~20161826,20971520~22363086,46137344~8602460,59300692~1156219,66056524~17079539,88080384~5578177,102437886~9019319,116424046~25252003,142606336~44781116,191717187~6799895,201326592~33497107,240139652~19370886,260046848~10354833,272629760~7690259,289406976~20790605,314572800~25096743,339738624~14267387,354571565~17822006,373441908~5818494,381681664~16927182,402653184~11153552] 2021-07-31T02:29:04.586 INFO:tasks.workunit.client.0.smithi045.stdout:from 1 to 15 diff [677713~19484113,24697164~18637442,47779655~367499,48159474~6580330,59300692~1156219,66056524~17079539,89659479~3999082,102437886~3730024,107771043~3686162,116694067~5293712,124734943~11694643,138274647~3401402,143078160~18255579,162635090~2222342,167531438~13588850,181972347~5415105,191717187~6799895,201478817~17919245,220279147~2389400,223825249~10998450,241317308~11430657,253324513~6186025,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~19417018,354571565~17822006,373441908~5818494,381820951~6416356,389494265~9114581,403808658~9998078] 2021-07-31T02:29:04.586 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~19484113,24697164~18637442,47779655~367499,48159474~6580330,59300692~1156219,66056524~17079539,89659479~3999082,102437886~3730024,107771043~3686162,116694067~5293712,124734943~11694643,138274647~3401402,143078160~18255579,162635090~2222342,167531438~13588850,181972347~5415105,191717187~6799895,201478817~17919245,220279147~2389400,223825249~10998450,241317308~11430657,253324513~6186025,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~19417018,354571565~17822006,373441908~5818494,381820951~6416356,389494265~9114581,403808658~9998078] 2021-07-31T02:29:04.590 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~20161826,20971520~22363086,46137344~8602460,59300692~1156219,66056524~17079539,88080384~5578177,102437886~9019319,116424046~25252003,142606336~44781116,191717187~6799895,201326592~33497107,240139652~19370886,260046848~20273171,289406976~21125692,314572800~39433211,354571565~17822006,373441908~5818494,381681664~16927182,402653184~11153552] 2021-07-31T02:29:04.591 INFO:tasks.workunit.client.0.smithi045.stdout:from 1 to 16 diff [677713~19484113,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,66056524~17079539,86327888~2219507,89659479~3999082,102437886~3730024,107771043~3686162,116694067~7911258,124734943~11694643,138274647~3401402,143078160~18255579,162635090~2222342,167531438~13588850,181972347~5415105,191717187~6799895,201478817~17919245,220279147~2389400,223825249~10998450,241317308~11430657,253324513~6186025,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~19417018,354571565~17822006,373441908~5818494,381820951~6416356,389494265~13486636,403808658~9998078] 2021-07-31T02:29:04.591 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~19484113,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,66056524~17079539,86327888~2219507,89659479~3999082,102437886~3730024,107771043~3686162,116694067~7911258,124734943~11694643,138274647~3401402,143078160~18255579,162635090~2222342,167531438~13588850,181972347~5415105,191717187~6799895,201478817~17919245,220279147~2389400,223825249~10998450,241317308~11430657,253324513~6186025,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~19417018,354571565~17822006,373441908~5818494,381820951~6416356,389494265~13486636,403808658~9998078] 2021-07-31T02:29:04.596 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~20161826,20971520~22363086,46137344~10629113,59300692~1156219,66056524~17079539,83886080~9772481,102437886~9019319,116424046~25252003,142606336~44781116,191717187~6799895,201326592~33497107,240139652~19370886,260046848~20273171,289406976~21125692,314572800~39433211,354571565~17822006,373441908~5818494,381681664~32125072] 2021-07-31T02:29:04.597 INFO:tasks.workunit.client.0.smithi045.stdout:from 1 to 17 diff [677713~19484113,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,66056524~17079539,86327888~2219507,89659479~3999082,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~11694643,137433861~4421821,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~17919245,220279147~2389400,223825249~10998450,241317308~11430657,253324513~6186025,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~19621750,354571565~17822006,373441908~5818494,381820951~6416356,389494265~13486636,403808658~9998078] 2021-07-31T02:29:04.597 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~19484113,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,66056524~17079539,86327888~2219507,89659479~3999082,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~11694643,137433861~4421821,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~17919245,220279147~2389400,223825249~10998450,241317308~11430657,253324513~6186025,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~19621750,354571565~17822006,373441908~5818494,381820951~6416356,389494265~13486636,403808658~9998078] 2021-07-31T02:29:04.602 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~20161826,20971520~22363086,46137344~10629113,59300692~1156219,66056524~17079539,83886080~9772481,99285857~789121,102437886~9019319,116424046~25431636,142606336~44781116,191717187~6799895,201326592~33497107,240139652~19370886,260046848~20273171,289406976~21125692,314572800~39637943,354571565~17822006,373441908~5818494,381681664~32125072] 2021-07-31T02:29:04.602 INFO:tasks.workunit.client.0.smithi045.stdout:from 1 to 18 diff [677713~19484113,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,66056524~17079539,86327888~2219507,89659479~3999082,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~11694643,137433861~4421821,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~17919245,220279147~2389400,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:04.602 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~19484113,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,66056524~17079539,86327888~2219507,89659479~3999082,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~11694643,137433861~4421821,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~17919245,220279147~2389400,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:04.607 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~20161826,20971520~22363086,46137344~10629113,59300692~1156219,66056524~17079539,83886080~9772481,99285857~789121,102437886~9019319,116424046~25431636,142606336~44781116,191717187~6799895,201326592~33497107,240139652~19370886,260046848~20273171,289406976~21125692,314572800~57820771,373441908~5818494,381681664~32125072] 2021-07-31T02:29:04.608 INFO:tasks.workunit.client.0.smithi045.stdout:from 1 to 19 diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89659479~3999082,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:04.608 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89659479~3999082,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:04.613 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~43334606,46137344~10629113,59300692~1156219,60609981~5053700,66056524~17079539,83886080~9772481,99285857~789121,102437886~9019319,116424046~70963406,191717187~6799895,201326592~33497107,240139652~19370886,260046848~20273171,289406976~21125692,314572800~57820771,373441908~5818494,381681664~32125072] 2021-07-31T02:29:04.613 INFO:tasks.workunit.client.0.smithi045.stdout:from 2 to 3 diff [24697164~11073145,47779655~367499,48770053~5969751,89659479~3999082,191717187~6799895,225566155~7953768,298647376~7010387] 2021-07-31T02:29:04.613 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [24697164~11073145,47779655~367499,48770053~5969751,89659479~3999082,191717187~6799895,225566155~7953768,298647376~7010387] 2021-07-31T02:29:04.619 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [20971520~14798789,46137344~8602460,88080384~5578177,191717187~6799895,222298112~11221811,298647376~7010387] 2021-07-31T02:29:04.620 INFO:tasks.workunit.client.0.smithi045.stdout:from 2 to 4 diff [677713~11779869,24697164~11073145,47779655~367499,48159474~6580330,75265061~7871002,89659479~3999082,125404837~2998498,162635090~2222342,191717187~6799895,225566155~7953768,276020820~3015231,291252461~6684351,298647376~7010387,344188514~47344,360795176~5362515] 2021-07-31T02:29:04.620 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~11779869,24697164~11073145,47779655~367499,48159474~6580330,75265061~7871002,89659479~3999082,125404837~2998498,162635090~2222342,191717187~6799895,225566155~7953768,276020820~3015231,291252461~6684351,298647376~7010387,344188514~47344,360795176~5362515] 2021-07-31T02:29:04.628 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~12457582,20971520~14798789,46137344~8602460,72622196~10513867,88080384~5578177,121634816~6768519,159383552~5473880,191717187~6799895,222298112~11221811,272629760~6406291,291252461~6684351,298647376~7010387,343932928~302930,360710144~5447547] 2021-07-31T02:29:04.629 INFO:tasks.workunit.client.0.smithi045.stdout:from 2 to 5 diff [677713~11779869,24697164~11073145,47779655~367499,48159474~6580330,75265061~7871002,89659479~3999082,125404837~2998498,128426182~8003404,154322932~3202008,158077298~3138373,162635090~2222342,175704140~5416148,191717187~6799895,220279147~2389400,225566155~7953768,242667678~8137835,276020820~3015231,291252461~14405302,327653500~3214988,344188514~47344,354571565~2088938,360795176~5362515,367021723~5371848] 2021-07-31T02:29:04.629 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~11779869,24697164~11073145,47779655~367499,48159474~6580330,75265061~7871002,89659479~3999082,125404837~2998498,128426182~8003404,154322932~3202008,158077298~3138373,162635090~2222342,175704140~5416148,191717187~6799895,220279147~2389400,225566155~7953768,242667678~8137835,276020820~3015231,291252461~14405302,327653500~3214988,344188514~47344,354571565~2088938,360795176~5362515,367021723~5371848] 2021-07-31T02:29:04.632 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~12457582,20971520~14798789,46137344~8602460,72622196~10513867,88080384~5578177,121634816~14794770,154322932~10534500,171966464~9153824,191717187~6799895,219398062~14121861,240139652~10665861,272629760~6406291,291252461~14405302,327653500~3214988,343932928~302930,354571565~2088938,360710144~11683427] 2021-07-31T02:29:04.632 INFO:tasks.workunit.client.0.smithi045.stdout:from 2 to 6 diff [677713~11779869,24697164~11073145,39062249~828950,47779655~367499,48159474~6580330,75265061~7871002,89659479~3999082,107771043~3686162,116694067~5293712,125404837~2998498,128426182~8003404,146393293~4680383,154322932~3202008,158077298~3138373,162635090~2222342,175704140~5416148,191717187~6799895,220279147~2389400,225566155~9257544,242667678~8137835,275231912~5088107,291252461~14405302,327653500~3214988,342292118~6052493,354571565~2088938,360795176~5362515,367021723~5371848,392344733~459593] 2021-07-31T02:29:04.632 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~11779869,24697164~11073145,39062249~828950,47779655~367499,48159474~6580330,75265061~7871002,89659479~3999082,107771043~3686162,116694067~5293712,125404837~2998498,128426182~8003404,146393293~4680383,154322932~3202008,158077298~3138373,162635090~2222342,175704140~5416148,191717187~6799895,220279147~2389400,225566155~9257544,242667678~8137835,275231912~5088107,291252461~14405302,327653500~3214988,342292118~6052493,354571565~2088938,360795176~5362515,367021723~5371848,392344733~459593] 2021-07-31T02:29:04.636 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~12457582,20971520~14798789,37748736~2142463,46137344~8602460,72622196~10513867,88080384~5578177,104857600~6599605,116424046~20005540,142606336~8467340,154322932~10534500,171966464~9153824,191717187~6799895,219398062~15425637,240139652~10665861,272629760~7690259,291252461~14405302,327653500~3214988,339738624~8605987,354571565~2088938,360710144~11683427,390735849~2068477] 2021-07-31T02:29:04.637 INFO:tasks.workunit.client.0.smithi045.stdout:from 2 to 7 diff [677713~11779869,24697164~11073145,36574959~6759647,47779655~367499,48159474~6580330,69906101~13229962,89659479~3999082,107771043~3686162,116694067~5293712,125404837~2998498,128426182~8003404,146393293~4680383,154322932~6892739,162635090~2222342,175704140~5416148,191717187~6799895,220279147~2389400,223825249~10998450,242667678~8137835,253559427~288594,260438339~4519598,275231912~5088107,291252461~14405302,327653500~3214988,331638058~718760,342292118~6052493,354571565~2088938,360795176~5362515,367021723~5371848,392344733~459593] 2021-07-31T02:29:04.637 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~11779869,24697164~11073145,36574959~6759647,47779655~367499,48159474~6580330,69906101~13229962,89659479~3999082,107771043~3686162,116694067~5293712,125404837~2998498,128426182~8003404,146393293~4680383,154322932~6892739,162635090~2222342,175704140~5416148,191717187~6799895,220279147~2389400,223825249~10998450,242667678~8137835,253559427~288594,260438339~4519598,275231912~5088107,291252461~14405302,327653500~3214988,331638058~718760,342292118~6052493,354571565~2088938,360795176~5362515,367021723~5371848,392344733~459593] 2021-07-31T02:29:04.642 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~12457582,20971520~14798789,36243014~7091592,46137344~8602460,69906101~13229962,88080384~5578177,104857600~6599605,116424046~20005540,142606336~8467340,154322932~10534500,171966464~9153824,191717187~6799895,219398062~15425637,240139652~10665861,251658240~2189781,260046848~4911089,272629760~7690259,291252461~14405302,327653500~3214988,331434830~921988,339738624~8605987,354571565~2088938,360710144~11683427,390735849~2068477] 2021-07-31T02:29:04.643 INFO:tasks.workunit.client.0.smithi045.stdout:from 2 to 8 diff [677713~17405775,24697164~11073145,36574959~6759647,47779655~367499,48159474~6580330,69906101~13229962,89659479~3999082,102437886~3730024,107771043~3686162,116694067~5293712,124734943~11694643,146393293~4680383,154322932~6892739,162635090~2222342,175704140~5416148,191717187~6799895,220279147~2389400,223825249~10998450,241317308~9488205,253559427~288594,260438339~4519598,275231912~5088107,291252461~14405302,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,342292118~6052493,354571565~11586126,367021723~5371848,384392456~2082977,392344733~459593] 2021-07-31T02:29:04.643 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~17405775,24697164~11073145,36574959~6759647,47779655~367499,48159474~6580330,69906101~13229962,89659479~3999082,102437886~3730024,107771043~3686162,116694067~5293712,124734943~11694643,146393293~4680383,154322932~6892739,162635090~2222342,175704140~5416148,191717187~6799895,220279147~2389400,223825249~10998450,241317308~9488205,253559427~288594,260438339~4519598,275231912~5088107,291252461~14405302,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,342292118~6052493,354571565~11586126,367021723~5371848,384392456~2082977,392344733~459593] 2021-07-31T02:29:04.648 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~18083488,20971520~14798789,36243014~7091592,46137344~8602460,69906101~13229962,88080384~5578177,102437886~9019319,116424046~20005540,142606336~8467340,154322932~10534500,171966464~9153824,191717187~6799895,219398062~15425637,240139652~10665861,251658240~2189781,260046848~4911089,272629760~7690259,291252461~14405302,314572800~12707070,327653500~3214988,331434830~921988,339738624~8605987,354571565~17822006,381681664~4793769,390735849~2068477] 2021-07-31T02:29:04.648 INFO:tasks.workunit.client.0.smithi045.stdout:from 2 to 9 diff [677713~17405775,24697164~18637442,47779655~367499,48159474~6580330,69906101~13229962,89659479~3999082,102437886~3730024,107771043~3686162,116694067~5293712,124734943~11694643,146393293~4680383,154322932~7010807,162635090~2222342,167531438~7281081,175704140~5416148,191717187~6799895,201478817~1199557,220279147~2389400,223825249~10998450,241317308~9488205,253559427~288594,260438339~4519598,275231912~5088107,291252461~14405302,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,342292118~11713893,354571565~11586126,367021723~5371848,384392456~2082977,392344733~459593,403808658~5668404] 2021-07-31T02:29:04.648 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~17405775,24697164~18637442,47779655~367499,48159474~6580330,69906101~13229962,89659479~3999082,102437886~3730024,107771043~3686162,116694067~5293712,124734943~11694643,146393293~4680383,154322932~7010807,162635090~2222342,167531438~7281081,175704140~5416148,191717187~6799895,201478817~1199557,220279147~2389400,223825249~10998450,241317308~9488205,253559427~288594,260438339~4519598,275231912~5088107,291252461~14405302,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,342292118~11713893,354571565~11586126,367021723~5371848,384392456~2082977,392344733~459593,403808658~5668404] 2021-07-31T02:29:04.657 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~18083488,20971520~22363086,46137344~8602460,69906101~13229962,88080384~5578177,102437886~9019319,116424046~20005540,142606336~8467340,154322932~26797356,191717187~6799895,201478817~1199557,219398062~15425637,240139652~10665861,251658240~2189781,260046848~4911089,272629760~7690259,291252461~14405302,314572800~12707070,327653500~3214988,331434830~921988,339738624~14267387,354571565~17822006,381681664~4793769,390735849~2068477,402653184~6823878] 2021-07-31T02:29:04.657 INFO:tasks.workunit.client.0.smithi045.stdout:from 2 to 10 diff [677713~17405775,24697164~18637442,47779655~367499,48159474~6580330,69906101~13229962,89659479~3999082,102437886~3730024,107771043~3686162,116694067~5293712,124734943~11694643,138274647~3401402,146393293~14940446,162635090~2222342,167531438~7281081,175704140~5416148,191717187~6799895,201478817~1199557,220279147~2389400,223825249~10998450,241317308~11430657,253559427~288594,260438339~4519598,275231912~5088107,291252461~14405302,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,342292118~11713893,354571565~11586126,367021723~5371848,373726074~3020982,384392456~2082977,392344733~459593,403808658~5668404] 2021-07-31T02:29:04.658 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~17405775,24697164~18637442,47779655~367499,48159474~6580330,69906101~13229962,89659479~3999082,102437886~3730024,107771043~3686162,116694067~5293712,124734943~11694643,138274647~3401402,146393293~14940446,162635090~2222342,167531438~7281081,175704140~5416148,191717187~6799895,201478817~1199557,220279147~2389400,223825249~10998450,241317308~11430657,253559427~288594,260438339~4519598,275231912~5088107,291252461~14405302,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,342292118~11713893,354571565~11586126,367021723~5371848,373726074~3020982,384392456~2082977,392344733~459593,403808658~5668404] 2021-07-31T02:29:04.663 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~18083488,20971520~22363086,46137344~8602460,69906101~13229962,88080384~5578177,102437886~9019319,116424046~25252003,142606336~38513952,191717187~6799895,201478817~1199557,219398062~15425637,240139652~13708369,260046848~4911089,272629760~7690259,291252461~14405302,314572800~12707070,327653500~3214988,331434830~921988,339738624~14267387,354571565~17822006,373726074~3020982,381681664~4793769,390735849~2068477,402653184~6823878] 2021-07-31T02:29:04.664 INFO:tasks.workunit.client.0.smithi045.stdout:from 2 to 11 diff [677713~19484113,24697164~18637442,47779655~367499,48159474~6580330,69906101~13229962,89659479~3999082,102437886~3730024,107771043~3686162,116694067~5293712,124734943~11694643,138274647~3401402,146393293~14940446,162635090~2222342,167531438~13588850,191717187~6799895,201478817~1199557,206474279~7964043,220279147~2389400,223825249~10998450,241317308~11430657,253559427~288594,260438339~4519598,275231912~5088107,291252461~18945120,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,342292118~11713893,354571565~17822006,373726074~3020982,381820951~6416356,389902499~6191760,403808658~9998078] 2021-07-31T02:29:04.664 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~19484113,24697164~18637442,47779655~367499,48159474~6580330,69906101~13229962,89659479~3999082,102437886~3730024,107771043~3686162,116694067~5293712,124734943~11694643,138274647~3401402,146393293~14940446,162635090~2222342,167531438~13588850,191717187~6799895,201478817~1199557,206474279~7964043,220279147~2389400,223825249~10998450,241317308~11430657,253559427~288594,260438339~4519598,275231912~5088107,291252461~18945120,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,342292118~11713893,354571565~17822006,373726074~3020982,381820951~6416356,389902499~6191760,403808658~9998078] 2021-07-31T02:29:04.665 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~20161826,20971520~22363086,46137344~8602460,69906101~13229962,88080384~5578177,102437886~9019319,116424046~25252003,142606336~38513952,191717187~6799895,201478817~1199557,205520896~8917426,219398062~15425637,240139652~13708369,260046848~4911089,272629760~7690259,291252461~18945120,314572800~12707070,327653500~3214988,331434830~921988,339738624~14267387,354571565~17822006,373726074~3020982,381681664~6555643,389902499~6191760,402653184~11153552] 2021-07-31T02:29:04.665 INFO:tasks.workunit.client.0.smithi045.stdout:from 2 to 12 diff [677713~19484113,24697164~18637442,47779655~367499,48159474~6580330,66056524~17079539,89659479~3999082,102437886~3730024,107771043~3686162,116694067~5293712,124734943~11694643,138274647~3401402,143078160~18255579,162635090~2222342,167531438~13588850,181972347~5415105,191717187~6799895,201478817~1199557,206474279~7964043,220279147~2389400,223825249~10998450,241317308~11430657,253559427~288594,260438339~4519598,275231912~5088107,291252461~18945120,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~5080550,342292118~11713893,354571565~17822006,373441908~5818494,381820951~6416356,389902499~6191760,403808658~9998078] 2021-07-31T02:29:04.665 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~19484113,24697164~18637442,47779655~367499,48159474~6580330,66056524~17079539,89659479~3999082,102437886~3730024,107771043~3686162,116694067~5293712,124734943~11694643,138274647~3401402,143078160~18255579,162635090~2222342,167531438~13588850,181972347~5415105,191717187~6799895,201478817~1199557,206474279~7964043,220279147~2389400,223825249~10998450,241317308~11430657,253559427~288594,260438339~4519598,275231912~5088107,291252461~18945120,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~5080550,342292118~11713893,354571565~17822006,373441908~5818494,381820951~6416356,389902499~6191760,403808658~9998078] 2021-07-31T02:29:04.672 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~20161826,20971520~22363086,46137344~8602460,66056524~17079539,88080384~5578177,102437886~9019319,116424046~25252003,142606336~44781116,191717187~6799895,201478817~1199557,205520896~8917426,219398062~15425637,240139652~13708369,260046848~4911089,272629760~7690259,291252461~18945120,314572800~12707070,327653500~3214988,331434830~8234713,339738624~14267387,354571565~17822006,373441908~5818494,381681664~6555643,389902499~6191760,402653184~11153552] 2021-07-31T02:29:04.672 INFO:tasks.workunit.client.0.smithi045.stdout:from 2 to 13 diff [677713~19484113,24697164~18637442,47779655~367499,48159474~6580330,66056524~17079539,89659479~3999082,102437886~3730024,107771043~3686162,116694067~5293712,124734943~11694643,138274647~3401402,143078160~18255579,162635090~2222342,167531438~13588850,181972347~5415105,191717187~6799895,201478817~12959505,220279147~2389400,223825249~10998450,241317308~11430657,253324513~6186025,260438339~4519598,275231912~5088107,291252461~18945120,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~5080550,342292118~11713893,354571565~17822006,373441908~5818494,381820951~6416356,389902499~8706347,403808658~9998078] 2021-07-31T02:29:04.672 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~19484113,24697164~18637442,47779655~367499,48159474~6580330,66056524~17079539,89659479~3999082,102437886~3730024,107771043~3686162,116694067~5293712,124734943~11694643,138274647~3401402,143078160~18255579,162635090~2222342,167531438~13588850,181972347~5415105,191717187~6799895,201478817~12959505,220279147~2389400,223825249~10998450,241317308~11430657,253324513~6186025,260438339~4519598,275231912~5088107,291252461~18945120,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~5080550,342292118~11713893,354571565~17822006,373441908~5818494,381820951~6416356,389902499~8706347,403808658~9998078] 2021-07-31T02:29:04.677 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~20161826,20971520~22363086,46137344~8602460,66056524~17079539,88080384~5578177,102437886~9019319,116424046~25252003,142606336~44781116,191717187~6799895,201478817~12959505,219398062~15425637,240139652~19370886,260046848~4911089,272629760~7690259,291252461~18945120,314572800~12707070,327653500~3214988,331434830~8234713,339738624~14267387,354571565~17822006,373441908~5818494,381681664~6555643,389902499~8706347,402653184~11153552] 2021-07-31T02:29:04.678 INFO:tasks.workunit.client.0.smithi045.stdout:from 2 to 14 diff [677713~19484113,24697164~18637442,47779655~367499,48159474~6580330,59300692~1156219,66056524~17079539,89659479~3999082,102437886~3730024,107771043~3686162,116694067~5293712,124734943~11694643,138274647~3401402,143078160~18255579,162635090~2222342,167531438~13588850,181972347~5415105,191717187~6799895,201478817~12959505,220279147~2389400,223825249~10998450,241317308~11430657,253324513~6186025,260438339~9963342,275231912~5088107,291252461~18945120,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~5080550,342292118~11713893,354571565~17822006,373441908~5818494,381820951~6416356,389902499~8706347,403808658~9998078] 2021-07-31T02:29:04.678 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~19484113,24697164~18637442,47779655~367499,48159474~6580330,59300692~1156219,66056524~17079539,89659479~3999082,102437886~3730024,107771043~3686162,116694067~5293712,124734943~11694643,138274647~3401402,143078160~18255579,162635090~2222342,167531438~13588850,181972347~5415105,191717187~6799895,201478817~12959505,220279147~2389400,223825249~10998450,241317308~11430657,253324513~6186025,260438339~9963342,275231912~5088107,291252461~18945120,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~5080550,342292118~11713893,354571565~17822006,373441908~5818494,381820951~6416356,389902499~8706347,403808658~9998078] 2021-07-31T02:29:04.683 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~20161826,20971520~22363086,46137344~8602460,59300692~1156219,66056524~17079539,88080384~5578177,102437886~9019319,116424046~25252003,142606336~44781116,191717187~6799895,201478817~12959505,219398062~15425637,240139652~19370886,260046848~10354833,272629760~7690259,291252461~18945120,314572800~12707070,327653500~3214988,331434830~8234713,339738624~14267387,354571565~17822006,373441908~5818494,381681664~6555643,389902499~8706347,402653184~11153552] 2021-07-31T02:29:04.683 INFO:tasks.workunit.client.0.smithi045.stdout:from 2 to 15 diff [677713~19484113,24697164~18637442,47779655~367499,48159474~6580330,59300692~1156219,66056524~17079539,89659479~3999082,102437886~3730024,107771043~3686162,116694067~5293712,124734943~11694643,138274647~3401402,143078160~18255579,162635090~2222342,167531438~13588850,181972347~5415105,191717187~6799895,201478817~12959505,220279147~2389400,223825249~10998450,241317308~11430657,253324513~6186025,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~19417018,354571565~17822006,373441908~5818494,381820951~6416356,389902499~8706347,403808658~9998078] 2021-07-31T02:29:04.683 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~19484113,24697164~18637442,47779655~367499,48159474~6580330,59300692~1156219,66056524~17079539,89659479~3999082,102437886~3730024,107771043~3686162,116694067~5293712,124734943~11694643,138274647~3401402,143078160~18255579,162635090~2222342,167531438~13588850,181972347~5415105,191717187~6799895,201478817~12959505,220279147~2389400,223825249~10998450,241317308~11430657,253324513~6186025,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~19417018,354571565~17822006,373441908~5818494,381820951~6416356,389902499~8706347,403808658~9998078] 2021-07-31T02:29:04.689 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~20161826,20971520~22363086,46137344~8602460,59300692~1156219,66056524~17079539,88080384~5578177,102437886~9019319,116424046~25252003,142606336~44781116,191717187~6799895,201478817~12959505,219398062~15425637,240139652~19370886,260046848~20273171,291252461~19280207,314572800~12707070,327653500~3214988,331434830~22571181,354571565~17822006,373441908~5818494,381681664~6555643,389902499~8706347,402653184~11153552] 2021-07-31T02:29:04.689 INFO:tasks.workunit.client.0.smithi045.stdout:from 2 to 16 diff [677713~19484113,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,66056524~17079539,86327888~2219507,89659479~3999082,102437886~3730024,107771043~3686162,116694067~7911258,124734943~11694643,138274647~3401402,143078160~18255579,162635090~2222342,167531438~13588850,181972347~5415105,191717187~6799895,201478817~12959505,220279147~2389400,223825249~10998450,241317308~11430657,253324513~6186025,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~19417018,354571565~17822006,373441908~5818494,381820951~6416356,389902499~13078402,403808658~9998078] 2021-07-31T02:29:04.689 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~19484113,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,66056524~17079539,86327888~2219507,89659479~3999082,102437886~3730024,107771043~3686162,116694067~7911258,124734943~11694643,138274647~3401402,143078160~18255579,162635090~2222342,167531438~13588850,181972347~5415105,191717187~6799895,201478817~12959505,220279147~2389400,223825249~10998450,241317308~11430657,253324513~6186025,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~19417018,354571565~17822006,373441908~5818494,381820951~6416356,389902499~13078402,403808658~9998078] 2021-07-31T02:29:04.694 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~20161826,20971520~22363086,46137344~10629113,59300692~1156219,66056524~17079539,83886080~9772481,102437886~9019319,116424046~25252003,142606336~44781116,191717187~6799895,201478817~12959505,219398062~15425637,240139652~19370886,260046848~20273171,291252461~19280207,314572800~12707070,327653500~3214988,331434830~22571181,354571565~17822006,373441908~5818494,381681664~6555643,389902499~23904237] 2021-07-31T02:29:04.695 INFO:tasks.workunit.client.0.smithi045.stdout:from 2 to 17 diff [677713~19484113,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,66056524~17079539,86327888~2219507,89659479~3999082,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~11694643,137433861~4421821,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~12959505,220279147~2389400,223825249~10998450,241317308~11430657,253324513~6186025,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~19621750,354571565~17822006,373441908~5818494,381820951~6416356,389902499~13078402,403808658~9998078] 2021-07-31T02:29:04.695 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~19484113,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,66056524~17079539,86327888~2219507,89659479~3999082,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~11694643,137433861~4421821,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~12959505,220279147~2389400,223825249~10998450,241317308~11430657,253324513~6186025,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~19621750,354571565~17822006,373441908~5818494,381820951~6416356,389902499~13078402,403808658~9998078] 2021-07-31T02:29:04.700 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~20161826,20971520~22363086,46137344~10629113,59300692~1156219,66056524~17079539,83886080~9772481,99285857~789121,102437886~9019319,116424046~25431636,142606336~44781116,191717187~6799895,201478817~12959505,219398062~15425637,240139652~19370886,260046848~20273171,291252461~19280207,314572800~12707070,327653500~3214988,331434830~22775913,354571565~17822006,373441908~5818494,381681664~6555643,389902499~23904237] 2021-07-31T02:29:04.700 INFO:tasks.workunit.client.0.smithi045.stdout:from 2 to 18 diff [677713~19484113,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,66056524~17079539,86327888~2219507,89659479~3999082,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~11694643,137433861~4421821,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~12959505,220279147~2389400,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:04.700 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~19484113,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,66056524~17079539,86327888~2219507,89659479~3999082,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~11694643,137433861~4421821,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~12959505,220279147~2389400,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:04.705 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~20161826,20971520~22363086,46137344~10629113,59300692~1156219,66056524~17079539,83886080~9772481,99285857~789121,102437886~9019319,116424046~25431636,142606336~44781116,191717187~6799895,201478817~12959505,219398062~15425637,240139652~19370886,260046848~20273171,291252461~19280207,314572800~12707070,327653500~3214988,331434830~40958741,373441908~5818494,381681664~32125072] 2021-07-31T02:29:04.706 INFO:tasks.workunit.client.0.smithi045.stdout:from 2 to 19 diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89659479~3999082,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:04.706 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89659479~3999082,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:04.711 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~43334606,46137344~10629113,59300692~1156219,60609981~5053700,66056524~17079539,83886080~9772481,99285857~789121,102437886~9019319,116424046~70963406,191717187~6799895,201478817~33344882,240139652~19370886,260046848~20273171,291252461~19280207,314572800~12707070,327653500~3214988,331434830~40958741,373441908~5818494,381681664~32125072] 2021-07-31T02:29:04.712 INFO:tasks.workunit.client.0.smithi045.stdout:from 3 to 4 diff [677713~11779869,48159474~2619597,75265061~7871002,125404837~2998498,162635090~2222342,276020820~3015231,291252461~6684351,344188514~47344,360795176~5362515] 2021-07-31T02:29:04.712 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~11779869,48159474~2619597,75265061~7871002,125404837~2998498,162635090~2222342,276020820~3015231,291252461~6684351,344188514~47344,360795176~5362515] 2021-07-31T02:29:04.717 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~12457582,48159474~2619597,72622196~10513867,121634816~6768519,159383552~5473880,272629760~6406291,291252461~6684351,343932928~302930,360710144~5447547] 2021-07-31T02:29:04.718 INFO:tasks.workunit.client.0.smithi045.stdout:from 3 to 5 diff [677713~11779869,48159474~2619597,75265061~7871002,125404837~2998498,128426182~8003404,154322932~3202008,158077298~3138373,162635090~2222342,175704140~5416148,220279147~2389400,242667678~8137835,276020820~3015231,291252461~13069783,327653500~3214988,344188514~47344,354571565~2088938,360795176~5362515,367021723~5371848] 2021-07-31T02:29:04.718 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~11779869,48159474~2619597,75265061~7871002,125404837~2998498,128426182~8003404,154322932~3202008,158077298~3138373,162635090~2222342,175704140~5416148,220279147~2389400,242667678~8137835,276020820~3015231,291252461~13069783,327653500~3214988,344188514~47344,354571565~2088938,360795176~5362515,367021723~5371848] 2021-07-31T02:29:04.723 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~12457582,48159474~2619597,72622196~10513867,121634816~14794770,154322932~10534500,171966464~9153824,219398062~3270485,240139652~10665861,272629760~6406291,291252461~13069783,327653500~3214988,343932928~302930,354571565~2088938,360710144~11683427] 2021-07-31T02:29:04.723 INFO:tasks.workunit.client.0.smithi045.stdout:from 3 to 6 diff [677713~11779869,39062249~828950,48159474~2619597,75265061~7871002,107771043~3686162,116694067~5293712,125404837~2998498,128426182~8003404,146393293~4680383,154322932~3202008,158077298~3138373,162635090~2222342,175704140~5416148,220279147~2389400,229781426~5042273,242667678~8137835,275231912~5088107,291252461~13069783,327653500~3214988,342292118~6052493,354571565~2088938,360795176~5362515,367021723~5371848,392344733~459593] 2021-07-31T02:29:04.724 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~11779869,39062249~828950,48159474~2619597,75265061~7871002,107771043~3686162,116694067~5293712,125404837~2998498,128426182~8003404,146393293~4680383,154322932~3202008,158077298~3138373,162635090~2222342,175704140~5416148,220279147~2389400,229781426~5042273,242667678~8137835,275231912~5088107,291252461~13069783,327653500~3214988,342292118~6052493,354571565~2088938,360795176~5362515,367021723~5371848,392344733~459593] 2021-07-31T02:29:04.729 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~12457582,37748736~2142463,48159474~2619597,72622196~10513867,104857600~6599605,116424046~20005540,142606336~8467340,154322932~10534500,171966464~9153824,219398062~3270485,229781426~5042273,240139652~10665861,272629760~7690259,291252461~13069783,327653500~3214988,339738624~8605987,354571565~2088938,360710144~11683427,390735849~2068477] 2021-07-31T02:29:04.729 INFO:tasks.workunit.client.0.smithi045.stdout:from 3 to 7 diff [677713~11779869,33435101~869368,36574959~6759647,48159474~2619597,69906101~13229962,107771043~3686162,116694067~5293712,125404837~2998498,128426182~8003404,146393293~4680383,154322932~6892739,162635090~2222342,175704140~5416148,220279147~2389400,223825249~10998450,242667678~8137835,253559427~288594,260438339~4519598,275231912~5088107,291252461~13069783,327653500~3214988,331638058~718760,342292118~6052493,354571565~2088938,360795176~5362515,367021723~5371848,392344733~459593] 2021-07-31T02:29:04.729 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~11779869,33435101~869368,36574959~6759647,48159474~2619597,69906101~13229962,107771043~3686162,116694067~5293712,125404837~2998498,128426182~8003404,146393293~4680383,154322932~6892739,162635090~2222342,175704140~5416148,220279147~2389400,223825249~10998450,242667678~8137835,253559427~288594,260438339~4519598,275231912~5088107,291252461~13069783,327653500~3214988,331638058~718760,342292118~6052493,354571565~2088938,360795176~5362515,367021723~5371848,392344733~459593] 2021-07-31T02:29:04.734 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~12457582,33435101~869368,36243014~7091592,48159474~2619597,69906101~13229962,104857600~6599605,116424046~20005540,142606336~8467340,154322932~10534500,171966464~9153824,219398062~3270485,223825249~10998450,240139652~10665861,251658240~2189781,260046848~4911089,272629760~7690259,291252461~13069783,327653500~3214988,331434830~921988,339738624~8605987,354571565~2088938,360710144~11683427,390735849~2068477] 2021-07-31T02:29:04.735 INFO:tasks.workunit.client.0.smithi045.stdout:from 3 to 8 diff [677713~17405775,33435101~869368,36574959~6759647,48159474~2619597,69906101~13229962,102437886~3730024,107771043~3686162,116694067~5293712,124734943~11694643,146393293~4680383,154322932~6892739,162635090~2222342,175704140~5416148,220279147~2389400,223825249~10998450,241317308~9488205,253559427~288594,260438339~4519598,275231912~5088107,291252461~13069783,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,342292118~6052493,354571565~11586126,367021723~5371848,384392456~2082977,392344733~459593] 2021-07-31T02:29:04.735 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~17405775,33435101~869368,36574959~6759647,48159474~2619597,69906101~13229962,102437886~3730024,107771043~3686162,116694067~5293712,124734943~11694643,146393293~4680383,154322932~6892739,162635090~2222342,175704140~5416148,220279147~2389400,223825249~10998450,241317308~9488205,253559427~288594,260438339~4519598,275231912~5088107,291252461~13069783,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,342292118~6052493,354571565~11586126,367021723~5371848,384392456~2082977,392344733~459593] 2021-07-31T02:29:04.745 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~18083488,33435101~869368,36243014~7091592,48159474~2619597,69906101~13229962,102437886~9019319,116424046~20005540,142606336~8467340,154322932~10534500,171966464~9153824,219398062~3270485,223825249~10998450,240139652~10665861,251658240~2189781,260046848~4911089,272629760~7690259,291252461~13069783,314572800~12707070,327653500~3214988,331434830~921988,339738624~8605987,354571565~17822006,381681664~4793769,390735849~2068477] 2021-07-31T02:29:04.747 INFO:tasks.workunit.client.0.smithi045.stdout:from 3 to 9 diff [677713~17405775,33435101~869368,34802352~8532254,48159474~2619597,69906101~13229962,102437886~3730024,107771043~3686162,116694067~5293712,124734943~11694643,146393293~4680383,154322932~7010807,162635090~2222342,167531438~7281081,175704140~5416148,201478817~1199557,220279147~2389400,223825249~10998450,241317308~9488205,253559427~288594,260438339~4519598,275231912~5088107,291252461~13069783,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,342292118~11713893,354571565~11586126,367021723~5371848,384392456~2082977,392344733~459593,403808658~5668404] 2021-07-31T02:29:04.747 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~17405775,33435101~869368,34802352~8532254,48159474~2619597,69906101~13229962,102437886~3730024,107771043~3686162,116694067~5293712,124734943~11694643,146393293~4680383,154322932~7010807,162635090~2222342,167531438~7281081,175704140~5416148,201478817~1199557,220279147~2389400,223825249~10998450,241317308~9488205,253559427~288594,260438339~4519598,275231912~5088107,291252461~13069783,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,342292118~11713893,354571565~11586126,367021723~5371848,384392456~2082977,392344733~459593,403808658~5668404] 2021-07-31T02:29:04.749 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~18083488,33435101~869368,34802352~8532254,48159474~2619597,69906101~13229962,102437886~9019319,116424046~20005540,142606336~8467340,154322932~26797356,201478817~1199557,219398062~3270485,223825249~10998450,240139652~10665861,251658240~2189781,260046848~4911089,272629760~7690259,291252461~13069783,314572800~12707070,327653500~3214988,331434830~921988,339738624~14267387,354571565~17822006,381681664~4793769,390735849~2068477,402653184~6823878] 2021-07-31T02:29:04.750 INFO:tasks.workunit.client.0.smithi045.stdout:from 3 to 10 diff [677713~17405775,24886295~2243069,33435101~869368,34802352~8532254,48159474~2619597,69906101~13229962,102437886~3730024,107771043~3686162,116694067~5293712,124734943~11694643,138274647~3401402,146393293~14940446,162635090~2222342,167531438~7281081,175704140~5416148,201478817~1199557,220279147~2389400,223825249~10998450,241317308~11430657,253559427~288594,260438339~4519598,275231912~5088107,291252461~13069783,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,342292118~11713893,354571565~11586126,367021723~5371848,373726074~3020982,384392456~2082977,392344733~459593,403808658~5668404] 2021-07-31T02:29:04.751 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~17405775,24886295~2243069,33435101~869368,34802352~8532254,48159474~2619597,69906101~13229962,102437886~3730024,107771043~3686162,116694067~5293712,124734943~11694643,138274647~3401402,146393293~14940446,162635090~2222342,167531438~7281081,175704140~5416148,201478817~1199557,220279147~2389400,223825249~10998450,241317308~11430657,253559427~288594,260438339~4519598,275231912~5088107,291252461~13069783,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,342292118~11713893,354571565~11586126,367021723~5371848,373726074~3020982,384392456~2082977,392344733~459593,403808658~5668404] 2021-07-31T02:29:04.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:04 smithi154 conmon[34553]: cluster 2021-07-31T02:29:03.372007+0000 mon.a (mon.0) 2089 : cluster [DBG] osdmap e1164: 8 total, 8 up, 8 in 2021-07-31T02:29:04.763 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~18083488,24886295~2243069,33435101~869368,34802352~8532254,48159474~2619597,69906101~13229962,102437886~9019319,116424046~25252003,142606336~38513952,201478817~1199557,219398062~3270485,223825249~10998450,240139652~13708369,260046848~4911089,272629760~7690259,291252461~13069783,314572800~12707070,327653500~3214988,331434830~921988,339738624~14267387,354571565~17822006,373726074~3020982,381681664~4793769,390735849~2068477,402653184~6823878] 2021-07-31T02:29:04.764 INFO:tasks.workunit.client.0.smithi045.stdout:from 3 to 11 diff [677713~19484113,24886295~2243069,33435101~869368,34802352~8532254,48159474~2619597,69906101~13229962,102437886~3730024,107771043~3686162,116694067~5293712,124734943~11694643,138274647~3401402,146393293~14940446,162635090~2222342,167531438~13588850,201478817~1199557,206474279~7964043,220279147~2389400,223825249~10998450,241317308~11430657,253559427~288594,260438339~4519598,275231912~5088107,291252461~18945120,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,342292118~11713893,354571565~17822006,373726074~3020982,381820951~6416356,389902499~6191760,403808658~9998078] 2021-07-31T02:29:04.764 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~19484113,24886295~2243069,33435101~869368,34802352~8532254,48159474~2619597,69906101~13229962,102437886~3730024,107771043~3686162,116694067~5293712,124734943~11694643,138274647~3401402,146393293~14940446,162635090~2222342,167531438~13588850,201478817~1199557,206474279~7964043,220279147~2389400,223825249~10998450,241317308~11430657,253559427~288594,260438339~4519598,275231912~5088107,291252461~18945120,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,342292118~11713893,354571565~17822006,373726074~3020982,381820951~6416356,389902499~6191760,403808658~9998078] 2021-07-31T02:29:04.764 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~20161826,24886295~2243069,33435101~869368,34802352~8532254,48159474~2619597,69906101~13229962,102437886~9019319,116424046~25252003,142606336~38513952,201478817~1199557,205520896~8917426,219398062~3270485,223825249~10998450,240139652~13708369,260046848~4911089,272629760~7690259,291252461~18945120,314572800~12707070,327653500~3214988,331434830~921988,339738624~14267387,354571565~17822006,373726074~3020982,381681664~6555643,389902499~6191760,402653184~11153552] 2021-07-31T02:29:04.765 INFO:tasks.workunit.client.0.smithi045.stdout:from 3 to 12 diff [677713~19484113,24886295~2243069,33435101~869368,34802352~8532254,48159474~2619597,66056524~17079539,102437886~3730024,107771043~3686162,116694067~5293712,124734943~11694643,138274647~3401402,143078160~18255579,162635090~2222342,167531438~13588850,181972347~5415105,201478817~1199557,206474279~7964043,220279147~2389400,223825249~10998450,241317308~11430657,253559427~288594,260438339~4519598,275231912~5088107,291252461~18945120,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~5080550,342292118~11713893,354571565~17822006,373441908~5818494,381820951~6416356,389902499~6191760,403808658~9998078] 2021-07-31T02:29:04.765 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~19484113,24886295~2243069,33435101~869368,34802352~8532254,48159474~2619597,66056524~17079539,102437886~3730024,107771043~3686162,116694067~5293712,124734943~11694643,138274647~3401402,143078160~18255579,162635090~2222342,167531438~13588850,181972347~5415105,201478817~1199557,206474279~7964043,220279147~2389400,223825249~10998450,241317308~11430657,253559427~288594,260438339~4519598,275231912~5088107,291252461~18945120,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~5080550,342292118~11713893,354571565~17822006,373441908~5818494,381820951~6416356,389902499~6191760,403808658~9998078] 2021-07-31T02:29:04.765 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~20161826,24886295~2243069,33435101~869368,34802352~8532254,48159474~2619597,66056524~17079539,102437886~9019319,116424046~25252003,142606336~44781116,201478817~1199557,205520896~8917426,219398062~3270485,223825249~10998450,240139652~13708369,260046848~4911089,272629760~7690259,291252461~18945120,314572800~12707070,327653500~3214988,331434830~8234713,339738624~14267387,354571565~17822006,373441908~5818494,381681664~6555643,389902499~6191760,402653184~11153552] 2021-07-31T02:29:04.765 INFO:tasks.workunit.client.0.smithi045.stdout:from 3 to 13 diff [677713~19484113,24886295~2243069,27353411~2246694,33435101~869368,34802352~8532254,48159474~2619597,66056524~17079539,89752687~3175987,102437886~3730024,107771043~3686162,116694067~5293712,124734943~11694643,138274647~3401402,143078160~18255579,162635090~2222342,167531438~13588850,181972347~5415105,201478817~12959505,220279147~2389400,223825249~10998450,241317308~11430657,253324513~6186025,260438339~4519598,275231912~5088107,291252461~18945120,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~5080550,342292118~11713893,354571565~17822006,373441908~5818494,381820951~6416356,389902499~8706347,403808658~9998078] 2021-07-31T02:29:04.765 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~19484113,24886295~2243069,27353411~2246694,33435101~869368,34802352~8532254,48159474~2619597,66056524~17079539,89752687~3175987,102437886~3730024,107771043~3686162,116694067~5293712,124734943~11694643,138274647~3401402,143078160~18255579,162635090~2222342,167531438~13588850,181972347~5415105,201478817~12959505,220279147~2389400,223825249~10998450,241317308~11430657,253324513~6186025,260438339~4519598,275231912~5088107,291252461~18945120,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~5080550,342292118~11713893,354571565~17822006,373441908~5818494,381820951~6416356,389902499~8706347,403808658~9998078] 2021-07-31T02:29:04.769 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~20161826,24886295~2243069,27353411~2246694,33435101~869368,34802352~8532254,48159474~2619597,66056524~17079539,89752687~3175987,102437886~9019319,116424046~25252003,142606336~44781116,201478817~12959505,219398062~3270485,223825249~10998450,240139652~19370886,260046848~4911089,272629760~7690259,291252461~18945120,314572800~12707070,327653500~3214988,331434830~8234713,339738624~14267387,354571565~17822006,373441908~5818494,381681664~6555643,389902499~8706347,402653184~11153552] 2021-07-31T02:29:04.769 INFO:tasks.workunit.client.0.smithi045.stdout:from 3 to 14 diff [677713~19484113,24886295~4713810,33435101~869368,34802352~8532254,48159474~2619597,59300692~1156219,66056524~17079539,89752687~3175987,102437886~3730024,107771043~3686162,116694067~5293712,124734943~11694643,138274647~3401402,143078160~18255579,162635090~2222342,167531438~13588850,181972347~5415105,201478817~12959505,220279147~2389400,223825249~10998450,241317308~11430657,253324513~6186025,260438339~9963342,275231912~5088107,291252461~18945120,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~5080550,342292118~11713893,354571565~17822006,373441908~5818494,381820951~6416356,389902499~8706347,403808658~9998078] 2021-07-31T02:29:04.770 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~19484113,24886295~4713810,33435101~869368,34802352~8532254,48159474~2619597,59300692~1156219,66056524~17079539,89752687~3175987,102437886~3730024,107771043~3686162,116694067~5293712,124734943~11694643,138274647~3401402,143078160~18255579,162635090~2222342,167531438~13588850,181972347~5415105,201478817~12959505,220279147~2389400,223825249~10998450,241317308~11430657,253324513~6186025,260438339~9963342,275231912~5088107,291252461~18945120,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~5080550,342292118~11713893,354571565~17822006,373441908~5818494,381820951~6416356,389902499~8706347,403808658~9998078] 2021-07-31T02:29:04.775 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~20161826,24886295~4713810,33435101~869368,34802352~8532254,48159474~2619597,59300692~1156219,66056524~17079539,89752687~3175987,102437886~9019319,116424046~25252003,142606336~44781116,201478817~12959505,219398062~3270485,223825249~10998450,240139652~19370886,260046848~10354833,272629760~7690259,291252461~18945120,314572800~12707070,327653500~3214988,331434830~8234713,339738624~14267387,354571565~17822006,373441908~5818494,381681664~6555643,389902499~8706347,402653184~11153552] 2021-07-31T02:29:04.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:04 smithi045 conmon[39718]: cluster 2021-07-31T02:29:03.372007+0000 mon.a (mon.0) 2089 : cluster [DBG] osdmap e1164: 8 total, 8 up, 8 in 2021-07-31T02:29:04.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:04 smithi045 conmon[33655]: cluster 2021-07-31T02:29:03.372007+0000 mon.a (mon.0 2021-07-31T02:29:04.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:04 smithi045 conmon[33655]: ) 2089 : cluster [DBG] osdmap e1164: 8 total, 8 up, 8 in 2021-07-31T02:29:04.776 INFO:tasks.workunit.client.0.smithi045.stdout:from 3 to 15 diff [677713~19484113,24886295~4713810,33435101~869368,34802352~8532254,48159474~2619597,59300692~1156219,66056524~17079539,89752687~3175987,102437886~3730024,107771043~3686162,116694067~5293712,124734943~11694643,138274647~3401402,143078160~18255579,162635090~2222342,167531438~13588850,181972347~5415105,201478817~12959505,220279147~2389400,223825249~10998450,241317308~11430657,253324513~6186025,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~19417018,354571565~17822006,373441908~5818494,381820951~6416356,389902499~8706347,403808658~9998078] 2021-07-31T02:29:04.777 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~19484113,24886295~4713810,33435101~869368,34802352~8532254,48159474~2619597,59300692~1156219,66056524~17079539,89752687~3175987,102437886~3730024,107771043~3686162,116694067~5293712,124734943~11694643,138274647~3401402,143078160~18255579,162635090~2222342,167531438~13588850,181972347~5415105,201478817~12959505,220279147~2389400,223825249~10998450,241317308~11430657,253324513~6186025,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~19417018,354571565~17822006,373441908~5818494,381820951~6416356,389902499~8706347,403808658~9998078] 2021-07-31T02:29:04.781 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~20161826,24886295~4713810,33435101~869368,34802352~8532254,48159474~2619597,59300692~1156219,66056524~17079539,89752687~3175987,102437886~9019319,116424046~25252003,142606336~44781116,201478817~12959505,219398062~3270485,223825249~10998450,240139652~19370886,260046848~20273171,291252461~19280207,314572800~12707070,327653500~3214988,331434830~22571181,354571565~17822006,373441908~5818494,381681664~6555643,389902499~8706347,402653184~11153552] 2021-07-31T02:29:04.781 INFO:tasks.workunit.client.0.smithi045.stdout:from 3 to 16 diff [677713~19484113,24372582~18962024,48159474~2619597,53817318~2949139,59300692~1156219,66056524~17079539,86327888~2219507,89752687~3175987,102437886~3730024,107771043~3686162,116694067~7911258,124734943~11694643,138274647~3401402,143078160~18255579,162635090~2222342,167531438~13588850,181972347~5415105,201478817~12959505,220279147~2389400,223825249~10998450,241317308~11430657,253324513~6186025,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~19417018,354571565~17822006,373441908~5818494,381820951~6416356,389902499~13078402,403808658~9998078] 2021-07-31T02:29:04.781 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~19484113,24372582~18962024,48159474~2619597,53817318~2949139,59300692~1156219,66056524~17079539,86327888~2219507,89752687~3175987,102437886~3730024,107771043~3686162,116694067~7911258,124734943~11694643,138274647~3401402,143078160~18255579,162635090~2222342,167531438~13588850,181972347~5415105,201478817~12959505,220279147~2389400,223825249~10998450,241317308~11430657,253324513~6186025,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~19417018,354571565~17822006,373441908~5818494,381820951~6416356,389902499~13078402,403808658~9998078] 2021-07-31T02:29:04.787 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~20161826,24372582~18962024,48159474~2619597,53817318~2949139,59300692~1156219,66056524~17079539,83886080~4661315,89752687~3175987,102437886~9019319,116424046~25252003,142606336~44781116,201478817~12959505,219398062~3270485,223825249~10998450,240139652~19370886,260046848~20273171,291252461~19280207,314572800~12707070,327653500~3214988,331434830~22571181,354571565~17822006,373441908~5818494,381681664~6555643,389902499~23904237] 2021-07-31T02:29:04.787 INFO:tasks.workunit.client.0.smithi045.stdout:from 3 to 17 diff [677713~19484113,24372582~18962024,48159474~2619597,53817318~2949139,59300692~1156219,66056524~17079539,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~11694643,137433861~4421821,143078160~21779272,167531438~13588850,181972347~5415105,201478817~12959505,220279147~2389400,223825249~10998450,241317308~11430657,253324513~6186025,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~19621750,354571565~17822006,373441908~5818494,381820951~6416356,389902499~13078402,403808658~9998078] 2021-07-31T02:29:04.787 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~19484113,24372582~18962024,48159474~2619597,53817318~2949139,59300692~1156219,66056524~17079539,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~11694643,137433861~4421821,143078160~21779272,167531438~13588850,181972347~5415105,201478817~12959505,220279147~2389400,223825249~10998450,241317308~11430657,253324513~6186025,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~19621750,354571565~17822006,373441908~5818494,381820951~6416356,389902499~13078402,403808658~9998078] 2021-07-31T02:29:04.793 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~20161826,24372582~18962024,48159474~2619597,53817318~2949139,59300692~1156219,66056524~17079539,83886080~4661315,89752687~3175987,99285857~789121,102437886~9019319,116424046~25431636,142606336~44781116,201478817~12959505,219398062~3270485,223825249~10998450,240139652~19370886,260046848~20273171,291252461~19280207,314572800~12707070,327653500~3214988,331434830~22775913,354571565~17822006,373441908~5818494,381681664~6555643,389902499~23904237] 2021-07-31T02:29:04.793 INFO:tasks.workunit.client.0.smithi045.stdout:from 3 to 18 diff [677713~19484113,24372582~18962024,48159474~2619597,53817318~2949139,59300692~1156219,66056524~17079539,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~11694643,137433861~4421821,143078160~21779272,167531438~13588850,181972347~5415105,201478817~12959505,220279147~2389400,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:04.793 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~19484113,24372582~18962024,48159474~2619597,53817318~2949139,59300692~1156219,66056524~17079539,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~11694643,137433861~4421821,143078160~21779272,167531438~13588850,181972347~5415105,201478817~12959505,220279147~2389400,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:04.798 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~20161826,24372582~18962024,48159474~2619597,53817318~2949139,59300692~1156219,66056524~17079539,83886080~4661315,89752687~3175987,99285857~789121,102437886~9019319,116424046~25431636,142606336~44781116,201478817~12959505,219398062~3270485,223825249~10998450,240139652~19370886,260046848~20273171,291252461~19280207,314572800~12707070,327653500~3214988,331434830~40958741,373441908~5818494,381681664~32125072] 2021-07-31T02:29:04.798 INFO:tasks.workunit.client.0.smithi045.stdout:from 3 to 19 diff [677713~20930344,24372582~18962024,48159474~2619597,53817318~2949139,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:04.799 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~20930344,24372582~18962024,48159474~2619597,53817318~2949139,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:04.804 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~21608057,24372582~18962024,48159474~2619597,53817318~2949139,59300692~1156219,60609981~5053700,66056524~17079539,83886080~4661315,89752687~3175987,99285857~789121,102437886~9019319,116424046~70963406,201478817~21375589,223825249~10998450,240139652~19370886,260046848~20273171,291252461~19280207,314572800~12707070,327653500~3214988,331434830~40958741,373441908~5818494,381681664~32125072] 2021-07-31T02:29:04.804 INFO:tasks.workunit.client.0.smithi045.stdout:from 4 to 5 diff [128426182~8003404,154322932~3202008,158077298~3138373,175704140~5416148,220279147~2389400,242667678~8137835,296245411~8076833,327653500~3214988,354571565~2088938,367021723~5371848] 2021-07-31T02:29:04.804 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [128426182~8003404,154322932~3202008,158077298~3138373,175704140~5416148,220279147~2389400,242667678~8137835,296245411~8076833,327653500~3214988,354571565~2088938,367021723~5371848] 2021-07-31T02:29:04.810 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [128403335~8026251,154322932~6892739,171966464~9153824,219398062~3270485,240139652~10665861,296245411~8076833,327653500~3214988,354571565~2088938,366157691~6235880] 2021-07-31T02:29:04.810 INFO:tasks.workunit.client.0.smithi045.stdout:from 4 to 6 diff [39062249~828950,107771043~3686162,116694067~5293712,128426182~8003404,146393293~4680383,154322932~3202008,158077298~3138373,175704140~5416148,220279147~2389400,229781426~5042273,242667678~8137835,275231912~5088107,292264750~12057494,327653500~3214988,342292118~6052493,354571565~2088938,367021723~5371848,392344733~459593] 2021-07-31T02:29:04.811 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [39062249~828950,107771043~3686162,116694067~5293712,128426182~8003404,146393293~4680383,154322932~3202008,158077298~3138373,175704140~5416148,220279147~2389400,229781426~5042273,242667678~8137835,275231912~5088107,292264750~12057494,327653500~3214988,342292118~6052493,354571565~2088938,367021723~5371848,392344733~459593] 2021-07-31T02:29:04.816 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [37748736~2142463,104857600~6599605,116424046~5563733,128403335~8026251,142606336~8467340,154322932~6892739,171966464~9153824,219398062~3270485,229781426~5042273,240139652~10665861,275231912~5088107,292264750~12057494,327653500~3214988,339738624~8605987,354571565~2088938,366157691~6235880,390735849~2068477] 2021-07-31T02:29:04.816 INFO:tasks.workunit.client.0.smithi045.stdout:from 4 to 7 diff [33435101~869368,36574959~6759647,69906101~6581740,107771043~3686162,116694067~5293712,128426182~8003404,146393293~4680383,154322932~6892739,175704140~5416148,220279147~2389400,223825249~10998450,242667678~8137835,253559427~288594,260438339~4519598,275231912~5088107,292264750~12057494,327653500~3214988,331638058~718760,342292118~6052493,354571565~2088938,367021723~5371848,392344733~459593] 2021-07-31T02:29:04.816 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [33435101~869368,36574959~6759647,69906101~6581740,107771043~3686162,116694067~5293712,128426182~8003404,146393293~4680383,154322932~6892739,175704140~5416148,220279147~2389400,223825249~10998450,242667678~8137835,253559427~288594,260438339~4519598,275231912~5088107,292264750~12057494,327653500~3214988,331638058~718760,342292118~6052493,354571565~2088938,367021723~5371848,392344733~459593] 2021-07-31T02:29:04.821 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [33435101~869368,36243014~7091592,69906101~6581740,104857600~6599605,116424046~5563733,128403335~8026251,142606336~8467340,154322932~6892739,171966464~9153824,219398062~3270485,223825249~10998450,240139652~10665861,251658240~2189781,260046848~4911089,275231912~5088107,292264750~12057494,327653500~3214988,331434830~921988,339738624~8605987,354571565~2088938,366157691~6235880,390735849~2068477] 2021-07-31T02:29:04.822 INFO:tasks.workunit.client.0.smithi045.stdout:from 4 to 8 diff [11422525~6660963,33435101~869368,36574959~6759647,69906101~6581740,102437886~3730024,107771043~3686162,116694067~5293712,124734943~11694643,146393293~4680383,154322932~6892739,175704140~5416148,220279147~2389400,223825249~10998450,241317308~9488205,253559427~288594,260438339~4519598,275231912~5088107,292264750~12057494,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,342292118~6052493,354571565~9467056,367021723~5371848,384392456~2082977,392344733~459593] 2021-07-31T02:29:04.822 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [11422525~6660963,33435101~869368,36574959~6759647,69906101~6581740,102437886~3730024,107771043~3686162,116694067~5293712,124734943~11694643,146393293~4680383,154322932~6892739,175704140~5416148,220279147~2389400,223825249~10998450,241317308~9488205,253559427~288594,260438339~4519598,275231912~5088107,292264750~12057494,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,342292118~6052493,354571565~9467056,367021723~5371848,384392456~2082977,392344733~459593] 2021-07-31T02:29:04.827 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [11422525~6660963,33435101~869368,36243014~7091592,69906101~6581740,102437886~9019319,116424046~5563733,124734943~11694643,142606336~8467340,154322932~6892739,171966464~9153824,219398062~3270485,223825249~10998450,240139652~10665861,251658240~2189781,260046848~4911089,275231912~5088107,292264750~12057494,314572800~12707070,327653500~3214988,331434830~921988,339738624~8605987,354571565~9467056,366157691~6235880,381681664~4793769,390735849~2068477] 2021-07-31T02:29:04.827 INFO:tasks.workunit.client.0.smithi045.stdout:from 4 to 9 diff [872289~3582298,11422525~6660963,33435101~869368,34802352~8532254,69906101~6581740,102437886~3730024,107771043~3686162,116694067~5293712,124734943~11694643,146393293~4680383,154322932~7010807,167531438~7281081,175704140~5416148,201478817~1199557,220279147~2389400,223825249~10998450,241317308~9488205,253559427~288594,260438339~4519598,275231912~5088107,292264750~12057494,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,342292118~11713893,354571565~9467056,367021723~5371848,384392456~2082977,392344733~459593,403808658~5668404] 2021-07-31T02:29:04.828 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~3582298,11422525~6660963,33435101~869368,34802352~8532254,69906101~6581740,102437886~3730024,107771043~3686162,116694067~5293712,124734943~11694643,146393293~4680383,154322932~7010807,167531438~7281081,175704140~5416148,201478817~1199557,220279147~2389400,223825249~10998450,241317308~9488205,253559427~288594,260438339~4519598,275231912~5088107,292264750~12057494,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,342292118~11713893,354571565~9467056,367021723~5371848,384392456~2082977,392344733~459593,403808658~5668404] 2021-07-31T02:29:04.833 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~3582298,11422525~6660963,33435101~869368,34802352~8532254,69906101~6581740,102437886~9019319,116424046~5563733,124734943~11694643,142606336~8467340,154322932~7010807,164857432~16262856,201478817~1199557,219398062~3270485,223825249~10998450,240139652~10665861,251658240~2189781,260046848~4911089,275231912~5088107,292264750~12057494,314572800~12707070,327653500~3214988,331434830~921988,339738624~14267387,354571565~9467056,366157691~6235880,381681664~4793769,390735849~2068477,402653184~6823878] 2021-07-31T02:29:04.834 INFO:tasks.workunit.client.0.smithi045.stdout:from 4 to 10 diff [872289~3582298,11422525~6660963,24886295~2243069,33435101~869368,34802352~8532254,69906101~6581740,79627966~536224,102437886~3730024,107771043~3686162,116694067~5293712,124734943~11694643,138274647~3401402,146393293~14940446,167531438~7281081,175704140~5416148,201478817~1199557,220279147~2389400,223825249~10998450,241317308~11430657,253559427~288594,260438339~4519598,275231912~5088107,292264750~12057494,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,342292118~11713893,354571565~9467056,367021723~5371848,373726074~3020982,384392456~2082977,392344733~459593,403808658~5668404] 2021-07-31T02:29:04.834 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~3582298,11422525~6660963,24886295~2243069,33435101~869368,34802352~8532254,69906101~6581740,79627966~536224,102437886~3730024,107771043~3686162,116694067~5293712,124734943~11694643,138274647~3401402,146393293~14940446,167531438~7281081,175704140~5416148,201478817~1199557,220279147~2389400,223825249~10998450,241317308~11430657,253559427~288594,260438339~4519598,275231912~5088107,292264750~12057494,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,342292118~11713893,354571565~9467056,367021723~5371848,373726074~3020982,384392456~2082977,392344733~459593,403808658~5668404] 2021-07-31T02:29:04.839 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~3582298,11422525~6660963,24886295~2243069,33435101~869368,34802352~8532254,69906101~6581740,79627966~536224,102437886~9019319,116424046~5563733,124734943~16941106,142606336~18727403,164857432~16262856,201478817~1199557,219398062~3270485,223825249~10998450,240139652~13708369,260046848~4911089,275231912~5088107,292264750~12057494,314572800~12707070,327653500~3214988,331434830~921988,339738624~14267387,354571565~9467056,366157691~6235880,373726074~3020982,381681664~4793769,390735849~2068477,402653184~6823878] 2021-07-31T02:29:04.839 INFO:tasks.workunit.client.0.smithi045.stdout:from 4 to 11 diff [872289~3582298,11422525~8739301,24886295~2243069,33435101~869368,34802352~8532254,69906101~6581740,78244364~2928921,102437886~3730024,107771043~3686162,116694067~5293712,124734943~11694643,138274647~3401402,146393293~14940446,167531438~13588850,201478817~1199557,206474279~7964043,220279147~2389400,223825249~10998450,241317308~11430657,253559427~288594,260438339~4519598,275231912~5088107,292264750~17932831,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,342292118~11713893,354571565~17822006,373726074~3020982,381820951~6416356,389902499~6191760,403808658~9998078] 2021-07-31T02:29:04.839 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~3582298,11422525~8739301,24886295~2243069,33435101~869368,34802352~8532254,69906101~6581740,78244364~2928921,102437886~3730024,107771043~3686162,116694067~5293712,124734943~11694643,138274647~3401402,146393293~14940446,167531438~13588850,201478817~1199557,206474279~7964043,220279147~2389400,223825249~10998450,241317308~11430657,253559427~288594,260438339~4519598,275231912~5088107,292264750~17932831,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,342292118~11713893,354571565~17822006,373726074~3020982,381820951~6416356,389902499~6191760,403808658~9998078] 2021-07-31T02:29:04.844 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~3582298,11422525~8739301,24886295~2243069,33435101~869368,34802352~8532254,69906101~6581740,78244364~2928921,102437886~9019319,116424046~5563733,124734943~16941106,142606336~18727403,164857432~16262856,201478817~1199557,205520896~8917426,219398062~3270485,223825249~10998450,240139652~13708369,260046848~4911089,275231912~5088107,292264750~17932831,314572800~12707070,327653500~3214988,331434830~921988,339738624~14267387,354571565~17822006,373726074~3020982,381681664~6555643,389902499~6191760,402653184~11153552] 2021-07-31T02:29:04.845 INFO:tasks.workunit.client.0.smithi045.stdout:from 4 to 12 diff [872289~3582298,10475392~9686434,24886295~2243069,33435101~869368,34802352~8532254,66056524~10431317,78244364~2928921,102437886~3730024,107771043~3686162,116694067~5293712,124734943~11694643,138274647~3401402,143078160~18255579,167531438~13588850,181972347~5415105,201478817~1199557,206474279~7964043,220279147~2389400,223825249~10998450,241317308~11430657,253559427~288594,260438339~4519598,275231912~5088107,292264750~17932831,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~5080550,342292118~11713893,354571565~17822006,373441908~5818494,381820951~6416356,389902499~6191760,403808658~9998078] 2021-07-31T02:29:04.845 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~3582298,10475392~9686434,24886295~2243069,33435101~869368,34802352~8532254,66056524~10431317,78244364~2928921,102437886~3730024,107771043~3686162,116694067~5293712,124734943~11694643,138274647~3401402,143078160~18255579,167531438~13588850,181972347~5415105,201478817~1199557,206474279~7964043,220279147~2389400,223825249~10998450,241317308~11430657,253559427~288594,260438339~4519598,275231912~5088107,292264750~17932831,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~5080550,342292118~11713893,354571565~17822006,373441908~5818494,381820951~6416356,389902499~6191760,403808658~9998078] 2021-07-31T02:29:04.853 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~3582298,10475392~9686434,24886295~2243069,33435101~869368,34802352~8532254,66056524~10431317,78244364~2928921,102437886~9019319,116424046~5563733,124734943~16941106,142606336~18727403,164857432~22530020,201478817~1199557,205520896~8917426,219398062~3270485,223825249~10998450,240139652~13708369,260046848~4911089,275231912~5088107,292264750~17932831,314572800~12707070,327653500~3214988,331434830~8234713,339738624~14267387,354571565~17822006,373441908~5818494,381681664~6555643,389902499~6191760,402653184~11153552] 2021-07-31T02:29:04.854 INFO:tasks.workunit.client.0.smithi045.stdout:from 4 to 13 diff [872289~3582298,10475392~9686434,24886295~2243069,27353411~2246694,33435101~869368,34802352~8532254,66056524~10431317,78244364~2928921,89752687~3175987,102437886~3730024,107771043~3686162,116694067~5293712,124734943~11694643,138274647~3401402,143078160~18255579,167531438~13588850,181972347~5415105,201478817~12959505,220279147~2389400,223825249~10998450,241317308~11430657,253324513~6186025,260438339~4519598,275231912~5088107,292264750~17932831,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~5080550,342292118~11713893,354571565~17822006,373441908~5818494,381820951~6416356,389902499~8706347,403808658~9998078] 2021-07-31T02:29:04.854 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~3582298,10475392~9686434,24886295~2243069,27353411~2246694,33435101~869368,34802352~8532254,66056524~10431317,78244364~2928921,89752687~3175987,102437886~3730024,107771043~3686162,116694067~5293712,124734943~11694643,138274647~3401402,143078160~18255579,167531438~13588850,181972347~5415105,201478817~12959505,220279147~2389400,223825249~10998450,241317308~11430657,253324513~6186025,260438339~4519598,275231912~5088107,292264750~17932831,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~5080550,342292118~11713893,354571565~17822006,373441908~5818494,381820951~6416356,389902499~8706347,403808658~9998078] 2021-07-31T02:29:04.857 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~3582298,10475392~9686434,24886295~2243069,27353411~2246694,33435101~869368,34802352~8532254,66056524~10431317,78244364~2928921,89752687~3175987,102437886~9019319,116424046~5563733,124734943~16941106,142606336~18727403,164857432~22530020,201478817~12959505,219398062~3270485,223825249~10998450,240139652~19370886,260046848~4911089,275231912~5088107,292264750~17932831,314572800~12707070,327653500~3214988,331434830~8234713,339738624~14267387,354571565~17822006,373441908~5818494,381681664~6555643,389902499~8706347,402653184~11153552] 2021-07-31T02:29:04.857 INFO:tasks.workunit.client.0.smithi045.stdout:from 4 to 14 diff [872289~3582298,10475392~9686434,24886295~4713810,33435101~869368,34802352~8532254,59300692~1156219,66056524~10431317,78244364~2928921,89752687~3175987,102437886~3730024,107771043~3686162,116694067~5293712,124734943~11694643,138274647~3401402,143078160~18255579,167531438~13588850,181972347~5415105,201478817~12959505,220279147~2389400,223825249~10998450,241317308~11430657,253324513~6186025,260438339~9963342,275231912~5088107,292264750~17932831,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~5080550,342292118~11713893,354571565~17822006,373441908~5818494,381820951~6416356,389902499~8706347,403808658~9998078] 2021-07-31T02:29:04.857 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~3582298,10475392~9686434,24886295~4713810,33435101~869368,34802352~8532254,59300692~1156219,66056524~10431317,78244364~2928921,89752687~3175987,102437886~3730024,107771043~3686162,116694067~5293712,124734943~11694643,138274647~3401402,143078160~18255579,167531438~13588850,181972347~5415105,201478817~12959505,220279147~2389400,223825249~10998450,241317308~11430657,253324513~6186025,260438339~9963342,275231912~5088107,292264750~17932831,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~5080550,342292118~11713893,354571565~17822006,373441908~5818494,381820951~6416356,389902499~8706347,403808658~9998078] 2021-07-31T02:29:04.864 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~3582298,10475392~9686434,24886295~4713810,33435101~869368,34802352~8532254,59300692~1156219,66056524~10431317,78244364~2928921,89752687~3175987,102437886~9019319,116424046~5563733,124734943~16941106,142606336~18727403,164857432~22530020,201478817~12959505,219398062~3270485,223825249~10998450,240139652~19370886,260046848~10354833,275231912~5088107,292264750~17932831,314572800~12707070,327653500~3214988,331434830~8234713,339738624~14267387,354571565~17822006,373441908~5818494,381681664~6555643,389902499~8706347,402653184~11153552] 2021-07-31T02:29:04.864 INFO:tasks.workunit.client.0.smithi045.stdout:from 4 to 15 diff [872289~8548834,10475392~9686434,24886295~4713810,33435101~869368,34802352~8532254,59300692~1156219,66056524~10431317,78244364~2928921,89752687~3175987,102437886~3730024,107771043~3686162,116694067~5293712,124734943~11694643,138274647~3401402,143078160~18255579,167531438~13588850,181972347~5415105,201478817~12959505,220279147~2389400,223825249~10998450,241317308~11430657,253324513~6186025,260438339~12248801,275231912~5088107,292264750~18267918,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~19417018,354571565~17822006,373441908~5818494,381820951~6416356,389902499~8706347,403808658~9998078] 2021-07-31T02:29:04.865 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~8548834,10475392~9686434,24886295~4713810,33435101~869368,34802352~8532254,59300692~1156219,66056524~10431317,78244364~2928921,89752687~3175987,102437886~3730024,107771043~3686162,116694067~5293712,124734943~11694643,138274647~3401402,143078160~18255579,167531438~13588850,181972347~5415105,201478817~12959505,220279147~2389400,223825249~10998450,241317308~11430657,253324513~6186025,260438339~12248801,275231912~5088107,292264750~18267918,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~19417018,354571565~17822006,373441908~5818494,381820951~6416356,389902499~8706347,403808658~9998078] 2021-07-31T02:29:04.868 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~8548834,10475392~9686434,24886295~4713810,33435101~869368,34802352~8532254,59300692~1156219,66056524~10431317,78244364~2928921,89752687~3175987,102437886~9019319,116424046~5563733,124734943~16941106,142606336~18727403,164857432~22530020,201478817~12959505,219398062~3270485,223825249~10998450,240139652~19370886,260046848~12640292,275231912~5088107,292264750~18267918,314572800~12707070,327653500~3214988,331434830~22571181,354571565~17822006,373441908~5818494,381681664~6555643,389902499~8706347,402653184~11153552] 2021-07-31T02:29:04.868 INFO:tasks.workunit.client.0.smithi045.stdout:from 4 to 16 diff [872289~8548834,10475392~9686434,24372582~18962024,53817318~2949139,59300692~1156219,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,102437886~3730024,107771043~3686162,116694067~7911258,124734943~11694643,138274647~3401402,143078160~18255579,167531438~13588850,181972347~5415105,201478817~12959505,220279147~2389400,223825249~10998450,241317308~11430657,253324513~6186025,260438339~12248801,275231912~5088107,292264750~18267918,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~19417018,354571565~17822006,373441908~5818494,381820951~6416356,389902499~13078402,403808658~9998078] 2021-07-31T02:29:04.869 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~8548834,10475392~9686434,24372582~18962024,53817318~2949139,59300692~1156219,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,102437886~3730024,107771043~3686162,116694067~7911258,124734943~11694643,138274647~3401402,143078160~18255579,167531438~13588850,181972347~5415105,201478817~12959505,220279147~2389400,223825249~10998450,241317308~11430657,253324513~6186025,260438339~12248801,275231912~5088107,292264750~18267918,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~19417018,354571565~17822006,373441908~5818494,381820951~6416356,389902499~13078402,403808658~9998078] 2021-07-31T02:29:04.874 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~8548834,10475392~9686434,24372582~18962024,53817318~2949139,59300692~1156219,66056524~10431317,78244364~2928921,83886080~4661315,89752687~3175987,102437886~9019319,116424046~8181279,124734943~16941106,142606336~18727403,164857432~22530020,201478817~12959505,219398062~3270485,223825249~10998450,240139652~19370886,260046848~12640292,275231912~5088107,292264750~18267918,314572800~12707070,327653500~3214988,331434830~22571181,354571565~17822006,373441908~5818494,381681664~6555643,389902499~23904237] 2021-07-31T02:29:04.874 INFO:tasks.workunit.client.0.smithi045.stdout:from 4 to 17 diff [872289~8548834,9457761~10704065,24372582~18962024,53817318~2949139,59300692~1156219,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~11694643,137433861~4421821,143078160~20324185,167531438~13588850,181972347~5415105,201478817~12959505,220279147~2389400,223825249~10998450,241317308~11430657,253324513~6186025,260438339~12248801,275231912~5088107,292264750~18267918,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~19621750,354571565~17822006,373441908~5818494,381820951~6416356,389902499~13078402,403808658~9998078] 2021-07-31T02:29:04.875 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~8548834,9457761~10704065,24372582~18962024,53817318~2949139,59300692~1156219,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~11694643,137433861~4421821,143078160~20324185,167531438~13588850,181972347~5415105,201478817~12959505,220279147~2389400,223825249~10998450,241317308~11430657,253324513~6186025,260438339~12248801,275231912~5088107,292264750~18267918,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~19621750,354571565~17822006,373441908~5818494,381820951~6416356,389902499~13078402,403808658~9998078] 2021-07-31T02:29:04.881 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~8548834,9457761~10704065,24372582~18962024,53817318~2949139,59300692~1156219,66056524~10431317,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,102437886~9019319,116424046~8181279,124734943~17120739,142606336~20796009,164857432~22530020,201478817~12959505,219398062~3270485,223825249~10998450,240139652~19370886,260046848~12640292,275231912~5088107,292264750~18267918,314572800~12707070,327653500~3214988,331434830~22775913,354571565~17822006,373441908~5818494,381681664~6555643,389902499~23904237] 2021-07-31T02:29:04.882 INFO:tasks.workunit.client.0.smithi045.stdout:from 4 to 18 diff [872289~19289537,24372582~18962024,53817318~2949139,59300692~1156219,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~11694643,137433861~4421821,143078160~20324185,167531438~13588850,181972347~5415105,201478817~12959505,220279147~2389400,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,292264750~18267918,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:04.882 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~19289537,24372582~18962024,53817318~2949139,59300692~1156219,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~11694643,137433861~4421821,143078160~20324185,167531438~13588850,181972347~5415105,201478817~12959505,220279147~2389400,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,292264750~18267918,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:04.885 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~19289537,24372582~18962024,53817318~2949139,59300692~1156219,66056524~10431317,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,102437886~9019319,116424046~8181279,124734943~17120739,142606336~20796009,164857432~22530020,201478817~12959505,219398062~3270485,223825249~10998450,240139652~19370886,260046848~12640292,275231912~5088107,292264750~18267918,314572800~12707070,327653500~3214988,331434830~40958741,373441908~5818494,381681664~32125072] 2021-07-31T02:29:04.886 INFO:tasks.workunit.client.0.smithi045.stdout:from 4 to 19 diff [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~20324185,167531438~13588850,181972347~5415105,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,292264750~18267918,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:04.886 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~20324185,167531438~13588850,181972347~5415105,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,292264750~18267918,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:04.891 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,102437886~9019319,116424046~8181279,124734943~38667402,164857432~22530020,201478817~21375589,223825249~10998450,240139652~19370886,260046848~12640292,275231912~5088107,292264750~18267918,314572800~12707070,327653500~3214988,331434830~40958741,373441908~5818494,381681664~32125072] 2021-07-31T02:29:04.892 INFO:tasks.workunit.client.0.smithi045.stdout:from 5 to 6 diff [39062249~828950,107771043~3686162,116694067~5293712,133012892~915417,146393293~4680383,229781426~5042273,275231912~5088107,292264750~4687170,342292118~6052493,392344733~459593] 2021-07-31T02:29:04.892 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [39062249~828950,107771043~3686162,116694067~5293712,133012892~915417,146393293~4680383,229781426~5042273,275231912~5088107,292264750~4687170,342292118~6052493,392344733~459593] 2021-07-31T02:29:04.899 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [37748736~2142463,104857600~6599605,116424046~5563733,133012892~915417,142606336~8467340,229781426~5042273,275231912~5088107,292264750~4687170,339738624~8605987,390735849~2068477] 2021-07-31T02:29:04.900 INFO:tasks.workunit.client.0.smithi045.stdout:from 5 to 7 diff [33435101~869368,36574959~6759647,69906101~6581740,107771043~3686162,116694067~5293712,133012892~915417,146393293~4680383,157303826~1384259,223825249~10998450,253559427~288594,260438339~4519598,275231912~5088107,292264750~4687170,331638058~718760,342292118~6052493,392344733~459593] 2021-07-31T02:29:04.901 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [33435101~869368,36574959~6759647,69906101~6581740,107771043~3686162,116694067~5293712,133012892~915417,146393293~4680383,157303826~1384259,223825249~10998450,253559427~288594,260438339~4519598,275231912~5088107,292264750~4687170,331638058~718760,342292118~6052493,392344733~459593] 2021-07-31T02:29:04.904 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [33435101~869368,36243014~7091592,69906101~6581740,104857600~6599605,116424046~5563733,133012892~915417,142606336~8467340,157303826~1384259,223825249~10998450,251658240~2189781,260046848~4911089,275231912~5088107,292264750~4687170,331434830~921988,339738624~8605987,390735849~2068477] 2021-07-31T02:29:04.905 INFO:tasks.workunit.client.0.smithi045.stdout:from 5 to 8 diff [11422525~6660963,33435101~869368,36574959~6759647,69906101~6581740,102437886~3730024,107771043~3686162,116694067~5293712,124734943~4595042,133012892~915417,146393293~4680383,157303826~1384259,223825249~10998450,241317308~7450669,253559427~288594,260438339~4519598,275231912~5088107,292264750~4687170,318433435~4174606,322615554~4664316,331638058~718760,342292118~6052493,355654064~8384557,384392456~2082977,392344733~459593] 2021-07-31T02:29:04.905 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [11422525~6660963,33435101~869368,36574959~6759647,69906101~6581740,102437886~3730024,107771043~3686162,116694067~5293712,124734943~4595042,133012892~915417,146393293~4680383,157303826~1384259,223825249~10998450,241317308~7450669,253559427~288594,260438339~4519598,275231912~5088107,292264750~4687170,318433435~4174606,322615554~4664316,331638058~718760,342292118~6052493,355654064~8384557,384392456~2082977,392344733~459593] 2021-07-31T02:29:04.908 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [11422525~6660963,33435101~869368,36243014~7091592,69906101~6581740,102437886~9019319,116424046~5563733,124734943~4595042,133012892~915417,142606336~8467340,157303826~1384259,223825249~10998450,241317308~7450669,251658240~2189781,260046848~4911089,275231912~5088107,292264750~4687170,314572800~12707070,331434830~921988,339738624~8605987,355654064~8384557,381681664~4793769,390735849~2068477] 2021-07-31T02:29:04.909 INFO:tasks.workunit.client.0.smithi045.stdout:from 5 to 9 diff [872289~3582298,11422525~6660963,33435101~869368,34802352~8532254,69906101~6581740,102437886~3730024,107771043~3686162,116694067~5293712,124734943~4595042,132705403~3160212,146393293~4680383,156774204~4559535,167531438~7281081,201478817~1199557,223825249~10998450,241317308~7450669,253559427~288594,260438339~4519598,275231912~5088107,292264750~4687170,318433435~4174606,322615554~4664316,331638058~718760,342292118~11713893,355654064~8384557,384392456~2082977,392344733~459593,403808658~5668404] 2021-07-31T02:29:04.909 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~3582298,11422525~6660963,33435101~869368,34802352~8532254,69906101~6581740,102437886~3730024,107771043~3686162,116694067~5293712,124734943~4595042,132705403~3160212,146393293~4680383,156774204~4559535,167531438~7281081,201478817~1199557,223825249~10998450,241317308~7450669,253559427~288594,260438339~4519598,275231912~5088107,292264750~4687170,318433435~4174606,322615554~4664316,331638058~718760,342292118~11713893,355654064~8384557,384392456~2082977,392344733~459593,403808658~5668404] 2021-07-31T02:29:04.914 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~3582298,11422525~6660963,33435101~869368,34802352~8532254,69906101~6581740,102437886~9019319,116424046~5563733,124734943~4595042,132705403~3160212,142606336~8467340,156774204~4559535,164857432~9955087,201478817~1199557,223825249~10998450,241317308~7450669,251658240~2189781,260046848~4911089,275231912~5088107,292264750~4687170,314572800~12707070,331434830~921988,339738624~14267387,355654064~8384557,381681664~4793769,390735849~2068477,402653184~6823878] 2021-07-31T02:29:04.914 INFO:tasks.workunit.client.0.smithi045.stdout:from 5 to 10 diff [872289~3582298,11422525~6660963,24886295~2243069,33435101~869368,34802352~8532254,69906101~6581740,79627966~536224,102437886~3730024,107771043~3686162,116694067~5293712,124734943~11130672,138274647~3401402,146393293~8467066,156774204~4559535,167531438~7281081,201478817~1199557,223825249~10998450,241317308~11430657,253559427~288594,260438339~4519598,275231912~5088107,292264750~4687170,318433435~4174606,322615554~4664316,331638058~718760,342292118~11713893,355654064~8384557,373726074~3020982,384392456~2082977,392344733~459593,403808658~5668404] 2021-07-31T02:29:04.914 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~3582298,11422525~6660963,24886295~2243069,33435101~869368,34802352~8532254,69906101~6581740,79627966~536224,102437886~3730024,107771043~3686162,116694067~5293712,124734943~11130672,138274647~3401402,146393293~8467066,156774204~4559535,167531438~7281081,201478817~1199557,223825249~10998450,241317308~11430657,253559427~288594,260438339~4519598,275231912~5088107,292264750~4687170,318433435~4174606,322615554~4664316,331638058~718760,342292118~11713893,355654064~8384557,373726074~3020982,384392456~2082977,392344733~459593,403808658~5668404] 2021-07-31T02:29:04.920 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~3582298,11422525~6660963,24886295~2243069,33435101~869368,34802352~8532254,69906101~6581740,79627966~536224,102437886~9019319,116424046~5563733,124734943~11130672,136429586~5246463,142606336~12254023,156774204~4559535,164857432~9955087,201478817~1199557,223825249~10998450,241317308~12530713,260046848~4911089,275231912~5088107,292264750~4687170,314572800~12707070,331434830~921988,339738624~14267387,355654064~8384557,373726074~3020982,381681664~4793769,390735849~2068477,402653184~6823878] 2021-07-31T02:29:04.920 INFO:tasks.workunit.client.0.smithi045.stdout:from 5 to 11 diff [872289~3582298,11422525~8739301,24886295~2243069,33435101~869368,34802352~8532254,69906101~6581740,78244364~2928921,102437886~3730024,107771043~3686162,116694067~5293712,124734943~11130672,138274647~3401402,146393293~8467066,156774204~4559535,167531438~8214378,201478817~1199557,206474279~7964043,223825249~10998450,241317308~11430657,253559427~288594,260438339~4519598,275231912~5088107,292264750~4687170,302959050~7238531,318433435~4174606,322615554~4664316,331638058~718760,342292118~11713893,355654064~15546953,373726074~3020982,381820951~6416356,389902499~6191760,403808658~9998078] 2021-07-31T02:29:04.920 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~3582298,11422525~8739301,24886295~2243069,33435101~869368,34802352~8532254,69906101~6581740,78244364~2928921,102437886~3730024,107771043~3686162,116694067~5293712,124734943~11130672,138274647~3401402,146393293~8467066,156774204~4559535,167531438~8214378,201478817~1199557,206474279~7964043,223825249~10998450,241317308~11430657,253559427~288594,260438339~4519598,275231912~5088107,292264750~4687170,302959050~7238531,318433435~4174606,322615554~4664316,331638058~718760,342292118~11713893,355654064~15546953,373726074~3020982,381820951~6416356,389902499~6191760,403808658~9998078] 2021-07-31T02:29:04.925 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~3582298,11422525~8739301,24886295~2243069,33435101~869368,34802352~8532254,69906101~6581740,78244364~2928921,102437886~9019319,116424046~5563733,124734943~11130672,136429586~5246463,142606336~12254023,156774204~4559535,164857432~10888384,201478817~1199557,205520896~8917426,223825249~10998450,241317308~12530713,260046848~4911089,275231912~5088107,292264750~4687170,302959050~7238531,314572800~12707070,331434830~921988,339738624~14267387,355654064~15546953,373726074~3020982,381681664~6555643,389902499~6191760,402653184~11153552] 2021-07-31T02:29:04.925 INFO:tasks.workunit.client.0.smithi045.stdout:from 5 to 12 diff [872289~3582298,10475392~9686434,24886295~2243069,33435101~869368,34802352~8532254,66056524~10431317,78244364~2928921,102437886~3730024,107771043~3686162,116694067~5293712,124734943~11130672,138274647~3401402,143078160~11782199,156774204~4559535,167531438~8214378,181972347~5415105,201478817~1199557,206474279~7964043,223825249~10998450,241317308~11430657,253559427~288594,260438339~4519598,275231912~5088107,292264750~4687170,302959050~7238531,318433435~4174606,322615554~4664316,331638058~718760,334588993~5080550,342292118~11713893,355654064~15546953,373441908~5818494,381820951~6416356,389902499~6191760,403808658~9998078] 2021-07-31T02:29:04.926 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~3582298,10475392~9686434,24886295~2243069,33435101~869368,34802352~8532254,66056524~10431317,78244364~2928921,102437886~3730024,107771043~3686162,116694067~5293712,124734943~11130672,138274647~3401402,143078160~11782199,156774204~4559535,167531438~8214378,181972347~5415105,201478817~1199557,206474279~7964043,223825249~10998450,241317308~11430657,253559427~288594,260438339~4519598,275231912~5088107,292264750~4687170,302959050~7238531,318433435~4174606,322615554~4664316,331638058~718760,334588993~5080550,342292118~11713893,355654064~15546953,373441908~5818494,381820951~6416356,389902499~6191760,403808658~9998078] 2021-07-31T02:29:04.931 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~3582298,10475392~9686434,24886295~2243069,33435101~869368,34802352~8532254,66056524~10431317,78244364~2928921,102437886~9019319,116424046~5563733,124734943~11130672,136429586~5246463,142606336~12254023,156774204~4559535,164857432~10888384,181120288~6267164,201478817~1199557,205520896~8917426,223825249~10998450,241317308~12530713,260046848~4911089,275231912~5088107,292264750~4687170,302959050~7238531,314572800~12707070,331434830~8234713,339738624~14267387,355654064~15546953,373441908~5818494,381681664~6555643,389902499~6191760,402653184~11153552] 2021-07-31T02:29:04.933 INFO:tasks.workunit.client.0.smithi045.stdout:from 5 to 13 diff [872289~3582298,10475392~9686434,24886295~2243069,27353411~2246694,33435101~869368,34802352~8532254,66056524~10431317,78244364~2928921,89752687~3175987,102437886~3730024,107771043~3686162,116694067~5293712,124734943~11130672,138274647~3401402,143078160~11782199,156774204~4559535,167531438~10157527,181972347~5415105,201478817~12959505,223825249~10998450,241317308~11430657,253324513~6186025,260438339~4519598,275231912~5088107,292264750~4687170,302959050~7238531,318433435~4174606,322615554~4664316,331638058~718760,334588993~5080550,342292118~11713893,355654064~15546953,373441908~5818494,381820951~6416356,389902499~8706347,403808658~9998078] 2021-07-31T02:29:04.934 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~3582298,10475392~9686434,24886295~2243069,27353411~2246694,33435101~869368,34802352~8532254,66056524~10431317,78244364~2928921,89752687~3175987,102437886~3730024,107771043~3686162,116694067~5293712,124734943~11130672,138274647~3401402,143078160~11782199,156774204~4559535,167531438~10157527,181972347~5415105,201478817~12959505,223825249~10998450,241317308~11430657,253324513~6186025,260438339~4519598,275231912~5088107,292264750~4687170,302959050~7238531,318433435~4174606,322615554~4664316,331638058~718760,334588993~5080550,342292118~11713893,355654064~15546953,373441908~5818494,381820951~6416356,389902499~8706347,403808658~9998078] 2021-07-31T02:29:04.937 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~3582298,10475392~9686434,24886295~2243069,27353411~2246694,33435101~869368,34802352~8532254,66056524~10431317,78244364~2928921,89752687~3175987,102437886~9019319,116424046~5563733,124734943~11130672,136429586~5246463,142606336~12254023,156774204~4559535,164857432~12831533,181120288~6267164,201478817~12959505,223825249~10998450,241317308~18193230,260046848~4911089,275231912~5088107,292264750~4687170,302959050~7238531,314572800~12707070,331434830~8234713,339738624~14267387,355654064~15546953,373441908~5818494,381681664~6555643,389902499~8706347,402653184~11153552] 2021-07-31T02:29:04.937 INFO:tasks.workunit.client.0.smithi045.stdout:from 5 to 14 diff [872289~3582298,10475392~9686434,24886295~4713810,33435101~869368,34802352~8532254,59300692~1156219,66056524~10431317,78244364~2928921,89752687~3175987,102437886~3730024,107771043~3686162,116694067~5293712,124734943~11130672,138274647~3401402,143078160~18255579,167531438~10157527,181972347~5415105,201478817~12959505,223825249~10998450,241317308~11430657,253324513~6186025,260438339~9963342,275231912~5088107,292264750~4687170,301341353~8856228,318433435~4174606,322615554~4664316,331638058~718760,334588993~5080550,342292118~11713893,355654064~15546953,373441908~5818494,381820951~6416356,389902499~8706347,403808658~9998078] 2021-07-31T02:29:04.938 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~3582298,10475392~9686434,24886295~4713810,33435101~869368,34802352~8532254,59300692~1156219,66056524~10431317,78244364~2928921,89752687~3175987,102437886~3730024,107771043~3686162,116694067~5293712,124734943~11130672,138274647~3401402,143078160~18255579,167531438~10157527,181972347~5415105,201478817~12959505,223825249~10998450,241317308~11430657,253324513~6186025,260438339~9963342,275231912~5088107,292264750~4687170,301341353~8856228,318433435~4174606,322615554~4664316,331638058~718760,334588993~5080550,342292118~11713893,355654064~15546953,373441908~5818494,381820951~6416356,389902499~8706347,403808658~9998078] 2021-07-31T02:29:04.942 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~3582298,10475392~9686434,24886295~4713810,33435101~869368,34802352~8532254,59300692~1156219,66056524~10431317,78244364~2928921,89752687~3175987,102437886~9019319,116424046~5563733,124734943~11130672,136429586~5246463,142606336~18727403,164857432~12831533,181120288~6267164,201478817~12959505,223825249~10998450,241317308~18193230,260046848~10354833,275231912~5088107,292264750~4687170,301341353~8856228,314572800~12707070,331434830~8234713,339738624~14267387,355654064~15546953,373441908~5818494,381681664~6555643,389902499~8706347,402653184~11153552] 2021-07-31T02:29:04.942 INFO:tasks.workunit.client.0.smithi045.stdout:from 5 to 15 diff [872289~8548834,10475392~9686434,24886295~4713810,33435101~869368,34802352~8532254,59300692~1156219,66056524~10431317,78244364~2928921,89752687~3175987,102437886~3730024,107771043~3686162,116694067~5293712,124734943~11130672,138274647~3401402,143078160~18255579,167531438~10157527,181972347~5415105,201478817~12959505,223825249~10998450,241317308~11430657,253324513~6186025,260438339~12248801,275231912~5088107,292264750~4687170,301341353~9191315,318433435~4174606,322615554~4664316,331638058~718760,334588993~19417018,355654064~15546953,373441908~5818494,381820951~6416356,389902499~8706347,403808658~9998078] 2021-07-31T02:29:04.942 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~8548834,10475392~9686434,24886295~4713810,33435101~869368,34802352~8532254,59300692~1156219,66056524~10431317,78244364~2928921,89752687~3175987,102437886~3730024,107771043~3686162,116694067~5293712,124734943~11130672,138274647~3401402,143078160~18255579,167531438~10157527,181972347~5415105,201478817~12959505,223825249~10998450,241317308~11430657,253324513~6186025,260438339~12248801,275231912~5088107,292264750~4687170,301341353~9191315,318433435~4174606,322615554~4664316,331638058~718760,334588993~19417018,355654064~15546953,373441908~5818494,381820951~6416356,389902499~8706347,403808658~9998078] 2021-07-31T02:29:04.948 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~8548834,10475392~9686434,24886295~4713810,33435101~869368,34802352~8532254,59300692~1156219,66056524~10431317,78244364~2928921,89752687~3175987,102437886~9019319,116424046~5563733,124734943~11130672,136429586~5246463,142606336~18727403,164857432~12831533,181120288~6267164,201478817~12959505,223825249~10998450,241317308~18193230,260046848~12640292,275231912~5088107,292264750~4687170,301341353~9191315,314572800~12707070,331434830~22571181,355654064~15546953,373441908~5818494,381681664~6555643,389902499~8706347,402653184~11153552] 2021-07-31T02:29:04.949 INFO:tasks.workunit.client.0.smithi045.stdout:from 5 to 16 diff [872289~8548834,10475392~9686434,24372582~18962024,53817318~2949139,59300692~1156219,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,102437886~3730024,107771043~3686162,116694067~7911258,124734943~11130672,138274647~3401402,143078160~18255579,167531438~10157527,181972347~5415105,201478817~12959505,223825249~10998450,241317308~11430657,253324513~6186025,260438339~12248801,275231912~5088107,292264750~4687170,301341353~9191315,318433435~4174606,322615554~4664316,331638058~718760,334588993~19417018,355654064~15546953,373441908~5818494,381820951~6416356,389902499~13078402,403808658~9998078] 2021-07-31T02:29:04.950 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~8548834,10475392~9686434,24372582~18962024,53817318~2949139,59300692~1156219,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,102437886~3730024,107771043~3686162,116694067~7911258,124734943~11130672,138274647~3401402,143078160~18255579,167531438~10157527,181972347~5415105,201478817~12959505,223825249~10998450,241317308~11430657,253324513~6186025,260438339~12248801,275231912~5088107,292264750~4687170,301341353~9191315,318433435~4174606,322615554~4664316,331638058~718760,334588993~19417018,355654064~15546953,373441908~5818494,381820951~6416356,389902499~13078402,403808658~9998078] 2021-07-31T02:29:04.954 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~8548834,10475392~9686434,24372582~18962024,53817318~2949139,59300692~1156219,66056524~10431317,78244364~2928921,83886080~4661315,89752687~3175987,102437886~9019319,116424046~8181279,124734943~11130672,136429586~5246463,142606336~18727403,164857432~12831533,181120288~6267164,201478817~12959505,223825249~10998450,241317308~18193230,260046848~12640292,275231912~5088107,292264750~4687170,301341353~9191315,314572800~12707070,331434830~22571181,355654064~15546953,373441908~5818494,381681664~6555643,389902499~23904237] 2021-07-31T02:29:04.954 INFO:tasks.workunit.client.0.smithi045.stdout:from 5 to 17 diff [872289~8548834,9457761~10704065,24372582~18962024,53817318~2949139,59300692~1156219,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~11130672,137433861~4421821,143078160~20324185,167531438~10157527,181972347~5415105,201478817~12959505,223825249~10998450,241317308~11430657,253324513~6186025,260438339~12248801,275231912~5088107,292264750~4687170,301341353~9191315,318433435~4174606,322615554~4664316,331638058~718760,334588993~19621750,355654064~15546953,373441908~5818494,381820951~6416356,389902499~13078402,403808658~9998078] 2021-07-31T02:29:04.954 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~8548834,9457761~10704065,24372582~18962024,53817318~2949139,59300692~1156219,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~11130672,137433861~4421821,143078160~20324185,167531438~10157527,181972347~5415105,201478817~12959505,223825249~10998450,241317308~11430657,253324513~6186025,260438339~12248801,275231912~5088107,292264750~4687170,301341353~9191315,318433435~4174606,322615554~4664316,331638058~718760,334588993~19621750,355654064~15546953,373441908~5818494,381820951~6416356,389902499~13078402,403808658~9998078] 2021-07-31T02:29:04.959 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~8548834,9457761~10704065,24372582~18962024,53817318~2949139,59300692~1156219,66056524~10431317,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,102437886~9019319,116424046~8181279,124734943~11130672,136429586~5426096,142606336~20796009,164857432~12831533,181120288~6267164,201478817~12959505,223825249~10998450,241317308~18193230,260046848~12640292,275231912~5088107,292264750~4687170,301341353~9191315,314572800~12707070,331434830~22775913,355654064~15546953,373441908~5818494,381681664~6555643,389902499~23904237] 2021-07-31T02:29:04.960 INFO:tasks.workunit.client.0.smithi045.stdout:from 5 to 18 diff [872289~19289537,24372582~18962024,53817318~2949139,59300692~1156219,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~11130672,137433861~4421821,143078160~20324185,167531438~10157527,181972347~5415105,201478817~12959505,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,292264750~4687170,301341353~9191315,318433435~4174606,322615554~4664316,331638058~718760,334588993~36612024,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:04.960 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~19289537,24372582~18962024,53817318~2949139,59300692~1156219,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~11130672,137433861~4421821,143078160~20324185,167531438~10157527,181972347~5415105,201478817~12959505,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,292264750~4687170,301341353~9191315,318433435~4174606,322615554~4664316,331638058~718760,334588993~36612024,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:04.965 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~19289537,24372582~18962024,53817318~2949139,59300692~1156219,66056524~10431317,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,102437886~9019319,116424046~8181279,124734943~11130672,136429586~5426096,142606336~20796009,164857432~12831533,181120288~6267164,201478817~12959505,223825249~10998450,241317308~18193230,260046848~12640292,275231912~5088107,292264750~4687170,301341353~9191315,314572800~12707070,331434830~39766187,373441908~5818494,381681664~32125072] 2021-07-31T02:29:04.966 INFO:tasks.workunit.client.0.smithi045.stdout:from 5 to 19 diff [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,292264750~4687170,301341353~9191315,318433435~4174606,322615554~4664316,331638058~718760,334588993~36612024,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:04.966 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,292264750~4687170,301341353~9191315,318433435~4174606,322615554~4664316,331638058~718760,334588993~36612024,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:04.971 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,102437886~9019319,116424046~8181279,124734943~38667402,164857432~12831533,181120288~6267164,201478817~19213323,221470799~1383607,223825249~10998450,241317308~18193230,260046848~12640292,275231912~5088107,292264750~4687170,301341353~9191315,314572800~12707070,331434830~39766187,373441908~5818494,381681664~32125072] 2021-07-31T02:29:04.972 INFO:tasks.workunit.client.0.smithi045.stdout:from 6 to 7 diff [33435101~869368,36574959~6759647,69906101~6581740,121752631~132598,157303826~1384259,223825249~6528699,253559427~288594,260438339~4519598,331638058~718760] 2021-07-31T02:29:04.972 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [33435101~869368,36574959~6759647,69906101~6581740,121752631~132598,157303826~1384259,223825249~6528699,253559427~288594,260438339~4519598,331638058~718760] 2021-07-31T02:29:04.977 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [33435101~869368,36243014~7091592,69906101~6581740,121752631~132598,157303826~1384259,223825249~6528699,251658240~2189781,260046848~4911089,331434830~921988] 2021-07-31T02:29:04.978 INFO:tasks.workunit.client.0.smithi045.stdout:from 6 to 8 diff [11422525~6660963,33435101~869368,36574959~6759647,69906101~6581740,102437886~3730024,121752631~132598,124734943~4595042,157303826~1384259,223825249~6528699,241317308~7450669,253559427~288594,260438339~4519598,318433435~4174606,322615554~4664316,331638058~718760,355654064~8384557,384392456~2082977] 2021-07-31T02:29:04.978 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [11422525~6660963,33435101~869368,36574959~6759647,69906101~6581740,102437886~3730024,121752631~132598,124734943~4595042,157303826~1384259,223825249~6528699,241317308~7450669,253559427~288594,260438339~4519598,318433435~4174606,322615554~4664316,331638058~718760,355654064~8384557,384392456~2082977] 2021-07-31T02:29:04.984 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [11422525~6660963,33435101~869368,36243014~7091592,69906101~6581740,102437886~3730024,121752631~132598,124734943~4595042,157303826~1384259,223825249~6528699,241317308~7450669,251658240~2189781,260046848~4911089,314572800~12707070,331434830~921988,355654064~8384557,381681664~4793769] 2021-07-31T02:29:04.985 INFO:tasks.workunit.client.0.smithi045.stdout:from 6 to 9 diff [872289~3582298,11422525~6660963,33435101~869368,34802352~8532254,69906101~6581740,102437886~3730024,121752631~132598,124734943~4595042,132705403~3160212,156774204~4559535,167531438~7281081,201478817~1199557,223825249~6528699,241317308~7450669,253559427~288594,260438339~4519598,318433435~4174606,322615554~4664316,331638058~718760,347819082~6186929,355654064~8384557,384392456~2082977,403808658~5668404] 2021-07-31T02:29:04.985 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~3582298,11422525~6660963,33435101~869368,34802352~8532254,69906101~6581740,102437886~3730024,121752631~132598,124734943~4595042,132705403~3160212,156774204~4559535,167531438~7281081,201478817~1199557,223825249~6528699,241317308~7450669,253559427~288594,260438339~4519598,318433435~4174606,322615554~4664316,331638058~718760,347819082~6186929,355654064~8384557,384392456~2082977,403808658~5668404] 2021-07-31T02:29:04.989 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~3582298,11422525~6660963,33435101~869368,34802352~8532254,69906101~6581740,102437886~3730024,121752631~132598,124734943~4595042,132705403~3160212,156774204~4559535,164857432~9955087,201478817~1199557,223825249~6528699,241317308~7450669,251658240~2189781,260046848~4911089,314572800~12707070,331434830~921988,347819082~6186929,355654064~8384557,381681664~4793769,402653184~6823878] 2021-07-31T02:29:04.990 INFO:tasks.workunit.client.0.smithi045.stdout:from 6 to 10 diff [872289~3582298,11422525~6660963,24886295~2243069,33435101~869368,34802352~8532254,69906101~6581740,79627966~536224,102437886~3730024,121752631~132598,124734943~11130672,138274647~3401402,147600507~7259852,156774204~4559535,167531438~7281081,201478817~1199557,223825249~6528699,241317308~11430657,253559427~288594,260438339~4519598,318433435~4174606,322615554~4664316,331638058~718760,347819082~6186929,355654064~8384557,373726074~3020982,384392456~2082977,403808658~5668404] 2021-07-31T02:29:04.990 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~3582298,11422525~6660963,24886295~2243069,33435101~869368,34802352~8532254,69906101~6581740,79627966~536224,102437886~3730024,121752631~132598,124734943~11130672,138274647~3401402,147600507~7259852,156774204~4559535,167531438~7281081,201478817~1199557,223825249~6528699,241317308~11430657,253559427~288594,260438339~4519598,318433435~4174606,322615554~4664316,331638058~718760,347819082~6186929,355654064~8384557,373726074~3020982,384392456~2082977,403808658~5668404] 2021-07-31T02:29:04.995 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~3582298,11422525~6660963,24886295~2243069,33435101~869368,34802352~8532254,69906101~6581740,79627966~536224,102437886~3730024,121752631~132598,124734943~11130672,136429586~5246463,147600507~7259852,156774204~4559535,164857432~9955087,201478817~1199557,223825249~6528699,241317308~12530713,260046848~4911089,314572800~12707070,331434830~921988,347819082~6186929,355654064~8384557,373726074~3020982,381681664~4793769,402653184~6823878] 2021-07-31T02:29:04.995 INFO:tasks.workunit.client.0.smithi045.stdout:from 6 to 11 diff [872289~3582298,11422525~8739301,24886295~2243069,33435101~869368,34802352~8532254,69906101~6581740,78244364~2928921,102437886~3730024,121752631~132598,124734943~11130672,138274647~3401402,147600507~7259852,156774204~4559535,167531438~8214378,201478817~1199557,206474279~7964043,223825249~6528699,241317308~11430657,253559427~288594,260438339~4519598,302959050~7238531,318433435~4174606,322615554~4664316,331638058~718760,347819082~6186929,355654064~15546953,373726074~3020982,381820951~6416356,389902499~6191760,403808658~9998078] 2021-07-31T02:29:04.996 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~3582298,11422525~8739301,24886295~2243069,33435101~869368,34802352~8532254,69906101~6581740,78244364~2928921,102437886~3730024,121752631~132598,124734943~11130672,138274647~3401402,147600507~7259852,156774204~4559535,167531438~8214378,201478817~1199557,206474279~7964043,223825249~6528699,241317308~11430657,253559427~288594,260438339~4519598,302959050~7238531,318433435~4174606,322615554~4664316,331638058~718760,347819082~6186929,355654064~15546953,373726074~3020982,381820951~6416356,389902499~6191760,403808658~9998078] 2021-07-31T02:29:05.001 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~3582298,11422525~8739301,24886295~2243069,33435101~869368,34802352~8532254,69906101~6581740,78244364~2928921,102437886~3730024,121752631~132598,124734943~11130672,136429586~5246463,147600507~7259852,156774204~4559535,164857432~10888384,201478817~1199557,205520896~8917426,223825249~6528699,241317308~12530713,260046848~4911089,302959050~7238531,314572800~12707070,331434830~921988,347819082~6186929,355654064~15546953,373726074~3020982,381681664~6555643,389902499~6191760,402653184~11153552] 2021-07-31T02:29:05.001 INFO:tasks.workunit.client.0.smithi045.stdout:from 6 to 12 diff [872289~3582298,10475392~9686434,24886295~2243069,33435101~869368,34802352~8532254,66056524~10431317,78244364~2928921,102437886~3730024,121752631~132598,124734943~11130672,138274647~3401402,143078160~11782199,156774204~4559535,167531438~8214378,181972347~5415105,201478817~1199557,206474279~7964043,223825249~6528699,241317308~11430657,253559427~288594,260438339~4519598,296242351~493439,302959050~7238531,318433435~4174606,322615554~4664316,331638058~718760,334588993~5080550,343702186~3839518,347819082~6186929,355654064~15546953,373441908~5818494,381820951~6416356,389902499~6191760,403808658~9998078] 2021-07-31T02:29:05.001 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~3582298,10475392~9686434,24886295~2243069,33435101~869368,34802352~8532254,66056524~10431317,78244364~2928921,102437886~3730024,121752631~132598,124734943~11130672,138274647~3401402,143078160~11782199,156774204~4559535,167531438~8214378,181972347~5415105,201478817~1199557,206474279~7964043,223825249~6528699,241317308~11430657,253559427~288594,260438339~4519598,296242351~493439,302959050~7238531,318433435~4174606,322615554~4664316,331638058~718760,334588993~5080550,343702186~3839518,347819082~6186929,355654064~15546953,373441908~5818494,381820951~6416356,389902499~6191760,403808658~9998078] 2021-07-31T02:29:05.006 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~3582298,10475392~9686434,24886295~2243069,33435101~869368,34802352~8532254,66056524~10431317,78244364~2928921,102437886~3730024,121752631~132598,124734943~11130672,136429586~5246463,143078160~11782199,156774204~4559535,164857432~10888384,181120288~6267164,201478817~1199557,205520896~8917426,223825249~6528699,241317308~12530713,260046848~4911089,296242351~493439,302959050~7238531,314572800~12707070,331434830~8234713,343702186~3839518,347819082~6186929,355654064~15546953,373441908~5818494,381681664~6555643,389902499~6191760,402653184~11153552] 2021-07-31T02:29:05.007 INFO:tasks.workunit.client.0.smithi045.stdout:from 6 to 13 diff [872289~3582298,10475392~9686434,24886295~2243069,27353411~2246694,33435101~869368,34802352~8532254,66056524~10431317,78244364~2928921,89752687~3175987,102437886~3730024,121752631~132598,124734943~11130672,138274647~3401402,143078160~11782199,156774204~4559535,167531438~10157527,181972347~5415105,201478817~12959505,223825249~6528699,241317308~11430657,253324513~6186025,260438339~4519598,296242351~493439,302959050~7238531,318433435~4174606,322615554~4664316,331638058~718760,334588993~5080550,343702186~3839518,347819082~6186929,355654064~15546953,373441908~5818494,381820951~6416356,389902499~8706347,403808658~9998078] 2021-07-31T02:29:05.007 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~3582298,10475392~9686434,24886295~2243069,27353411~2246694,33435101~869368,34802352~8532254,66056524~10431317,78244364~2928921,89752687~3175987,102437886~3730024,121752631~132598,124734943~11130672,138274647~3401402,143078160~11782199,156774204~4559535,167531438~10157527,181972347~5415105,201478817~12959505,223825249~6528699,241317308~11430657,253324513~6186025,260438339~4519598,296242351~493439,302959050~7238531,318433435~4174606,322615554~4664316,331638058~718760,334588993~5080550,343702186~3839518,347819082~6186929,355654064~15546953,373441908~5818494,381820951~6416356,389902499~8706347,403808658~9998078] 2021-07-31T02:29:05.015 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~3582298,10475392~9686434,24886295~2243069,27353411~2246694,33435101~869368,34802352~8532254,66056524~10431317,78244364~2928921,89752687~3175987,102437886~3730024,121752631~132598,124734943~11130672,136429586~5246463,143078160~11782199,156774204~4559535,164857432~12831533,181120288~6267164,201478817~12959505,223825249~6528699,241317308~18193230,260046848~4911089,296242351~493439,302959050~7238531,314572800~12707070,331434830~8234713,343702186~3839518,347819082~6186929,355654064~15546953,373441908~5818494,381681664~6555643,389902499~8706347,402653184~11153552] 2021-07-31T02:29:05.016 INFO:tasks.workunit.client.0.smithi045.stdout:from 6 to 14 diff [872289~3582298,10475392~9686434,24886295~4713810,33435101~869368,34802352~8532254,59300692~1156219,66056524~10431317,78244364~2928921,89752687~3175987,102437886~3730024,121752631~132598,124734943~11130672,138274647~3401402,143078160~18255579,167531438~10157527,181972347~5415105,201478817~12959505,223825249~6528699,241317308~11430657,253324513~6186025,260438339~9963342,296242351~493439,301341353~8856228,318433435~4174606,322615554~4664316,331638058~718760,334588993~5080550,343702186~3839518,347819082~6186929,355654064~15546953,373441908~5818494,381820951~6416356,389902499~8706347,403808658~9998078] 2021-07-31T02:29:05.016 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~3582298,10475392~9686434,24886295~4713810,33435101~869368,34802352~8532254,59300692~1156219,66056524~10431317,78244364~2928921,89752687~3175987,102437886~3730024,121752631~132598,124734943~11130672,138274647~3401402,143078160~18255579,167531438~10157527,181972347~5415105,201478817~12959505,223825249~6528699,241317308~11430657,253324513~6186025,260438339~9963342,296242351~493439,301341353~8856228,318433435~4174606,322615554~4664316,331638058~718760,334588993~5080550,343702186~3839518,347819082~6186929,355654064~15546953,373441908~5818494,381820951~6416356,389902499~8706347,403808658~9998078] 2021-07-31T02:29:05.018 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~3582298,10475392~9686434,24886295~4713810,33435101~869368,34802352~8532254,59300692~1156219,66056524~10431317,78244364~2928921,89752687~3175987,102437886~3730024,121752631~132598,124734943~11130672,136429586~5246463,143078160~18255579,164857432~12831533,181120288~6267164,201478817~12959505,223825249~6528699,241317308~18193230,260046848~10354833,296242351~493439,301341353~8856228,314572800~12707070,331434830~8234713,343702186~3839518,347819082~6186929,355654064~15546953,373441908~5818494,381681664~6555643,389902499~8706347,402653184~11153552] 2021-07-31T02:29:05.019 INFO:tasks.workunit.client.0.smithi045.stdout:from 6 to 15 diff [872289~8548834,10475392~9686434,24886295~4713810,33435101~869368,34802352~8532254,59300692~1156219,66056524~10431317,78244364~2928921,89752687~3175987,102437886~3730024,121752631~132598,124734943~11130672,138274647~3401402,143078160~18255579,167531438~10157527,181972347~5415105,201478817~12959505,223825249~6528699,241317308~11430657,253324513~6186025,260438339~12248801,296242351~493439,301341353~9191315,318433435~4174606,322615554~4664316,331638058~718760,334588993~19417018,355654064~15546953,373441908~5818494,381820951~6416356,389902499~8706347,403808658~9998078] 2021-07-31T02:29:05.019 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~8548834,10475392~9686434,24886295~4713810,33435101~869368,34802352~8532254,59300692~1156219,66056524~10431317,78244364~2928921,89752687~3175987,102437886~3730024,121752631~132598,124734943~11130672,138274647~3401402,143078160~18255579,167531438~10157527,181972347~5415105,201478817~12959505,223825249~6528699,241317308~11430657,253324513~6186025,260438339~12248801,296242351~493439,301341353~9191315,318433435~4174606,322615554~4664316,331638058~718760,334588993~19417018,355654064~15546953,373441908~5818494,381820951~6416356,389902499~8706347,403808658~9998078] 2021-07-31T02:29:05.023 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~8548834,10475392~9686434,24886295~4713810,33435101~869368,34802352~8532254,59300692~1156219,66056524~10431317,78244364~2928921,89752687~3175987,102437886~3730024,121752631~132598,124734943~11130672,136429586~5246463,143078160~18255579,164857432~12831533,181120288~6267164,201478817~12959505,223825249~6528699,241317308~18193230,260046848~12640292,296242351~493439,301341353~9191315,314572800~12707070,331434830~22571181,355654064~15546953,373441908~5818494,381681664~6555643,389902499~8706347,402653184~11153552] 2021-07-31T02:29:05.024 INFO:tasks.workunit.client.0.smithi045.stdout:from 6 to 16 diff [872289~8548834,10475392~9686434,24372582~18962024,53817318~2949139,59300692~1156219,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,102437886~3730024,121407098~3198227,124734943~11130672,138274647~3401402,143078160~18255579,167531438~10157527,181972347~5415105,201478817~12959505,223825249~6536345,241317308~11430657,253324513~6186025,260438339~12248801,296242351~493439,301341353~9191315,318433435~4174606,322615554~4664316,331638058~718760,334588993~19417018,355654064~15546953,373441908~5818494,381820951~6416356,389902499~13078402,403808658~9998078] 2021-07-31T02:29:05.024 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~8548834,10475392~9686434,24372582~18962024,53817318~2949139,59300692~1156219,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,102437886~3730024,121407098~3198227,124734943~11130672,138274647~3401402,143078160~18255579,167531438~10157527,181972347~5415105,201478817~12959505,223825249~6536345,241317308~11430657,253324513~6186025,260438339~12248801,296242351~493439,301341353~9191315,318433435~4174606,322615554~4664316,331638058~718760,334588993~19417018,355654064~15546953,373441908~5818494,381820951~6416356,389902499~13078402,403808658~9998078] 2021-07-31T02:29:05.029 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~8548834,10475392~9686434,24372582~18962024,53817318~2949139,59300692~1156219,66056524~10431317,78244364~2928921,83886080~4661315,89752687~3175987,102437886~3730024,121407098~3198227,124734943~11130672,136429586~5246463,143078160~18255579,164857432~12831533,181120288~6267164,201478817~12959505,223825249~6536345,241317308~18193230,260046848~12640292,296242351~493439,301341353~9191315,314572800~12707070,331434830~22571181,355654064~15546953,373441908~5818494,381681664~6555643,389902499~23904237] 2021-07-31T02:29:05.030 INFO:tasks.workunit.client.0.smithi045.stdout:from 6 to 17 diff [872289~8548834,9457761~10704065,24372582~18962024,53817318~2949139,59300692~1156219,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,121407098~3198227,124734943~11130672,137433861~4421821,143078160~20324185,167531438~10157527,181972347~5415105,201478817~12959505,223825249~6536345,241317308~11430657,253324513~6186025,260438339~12248801,296242351~493439,301341353~9191315,318433435~4174606,322615554~4664316,331638058~718760,334588993~19621750,355654064~15546953,373441908~5818494,381820951~6416356,389902499~13078402,403808658~9998078] 2021-07-31T02:29:05.030 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~8548834,9457761~10704065,24372582~18962024,53817318~2949139,59300692~1156219,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,121407098~3198227,124734943~11130672,137433861~4421821,143078160~20324185,167531438~10157527,181972347~5415105,201478817~12959505,223825249~6536345,241317308~11430657,253324513~6186025,260438339~12248801,296242351~493439,301341353~9191315,318433435~4174606,322615554~4664316,331638058~718760,334588993~19621750,355654064~15546953,373441908~5818494,381820951~6416356,389902499~13078402,403808658~9998078] 2021-07-31T02:29:05.035 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~8548834,9457761~10704065,24372582~18962024,53817318~2949139,59300692~1156219,66056524~10431317,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,102437886~3730024,121407098~3198227,124734943~11130672,136429586~5426096,143078160~20324185,164857432~12831533,181120288~6267164,201478817~12959505,223825249~6536345,241317308~18193230,260046848~12640292,296242351~493439,301341353~9191315,314572800~12707070,331434830~22775913,355654064~15546953,373441908~5818494,381681664~6555643,389902499~23904237] 2021-07-31T02:29:05.036 INFO:tasks.workunit.client.0.smithi045.stdout:from 6 to 18 diff [872289~19289537,24372582~18962024,53817318~2949139,59300692~1156219,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,121407098~3198227,124734943~11130672,137433861~4421821,143078160~20324185,167531438~10157527,181972347~5415105,201478817~12959505,223825249~6536345,241317308~18193230,260438339~12248801,296242351~493439,301341353~9191315,318433435~4174606,322615554~4664316,331638058~718760,334588993~36612024,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:05.036 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~19289537,24372582~18962024,53817318~2949139,59300692~1156219,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,121407098~3198227,124734943~11130672,137433861~4421821,143078160~20324185,167531438~10157527,181972347~5415105,201478817~12959505,223825249~6536345,241317308~18193230,260438339~12248801,296242351~493439,301341353~9191315,318433435~4174606,322615554~4664316,331638058~718760,334588993~36612024,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:05.042 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~19289537,24372582~18962024,53817318~2949139,59300692~1156219,66056524~10431317,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,102437886~3730024,121407098~3198227,124734943~11130672,136429586~5426096,143078160~20324185,164857432~12831533,181120288~6267164,201478817~12959505,223825249~6536345,241317308~18193230,260046848~12640292,296242351~493439,301341353~9191315,314572800~12707070,331434830~39766187,373441908~5818494,381681664~32125072] 2021-07-31T02:29:05.042 INFO:tasks.workunit.client.0.smithi045.stdout:from 6 to 19 diff [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,121407098~3198227,124734943~18101328,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,223825249~6536345,241317308~18193230,260438339~12248801,296242351~493439,301341353~9191315,318433435~4174606,322615554~4664316,331638058~718760,334588993~36612024,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:05.042 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,121407098~3198227,124734943~18101328,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,223825249~6536345,241317308~18193230,260438339~12248801,296242351~493439,301341353~9191315,318433435~4174606,322615554~4664316,331638058~718760,334588993~36612024,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:05.047 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,102437886~3730024,121407098~3198227,124734943~18101328,143078160~20324185,164857432~12831533,181120288~6267164,201478817~19213323,221470799~1383607,223825249~6536345,241317308~18193230,260046848~12640292,296242351~493439,301341353~9191315,314572800~12707070,331434830~39766187,373441908~5818494,381681664~32125072] 2021-07-31T02:29:05.047 INFO:tasks.workunit.client.0.smithi045.stdout:from 7 to 8 diff [11422525~6660963,102437886~3730024,124734943~4595042,241317308~7450669,318433435~4174606,322615554~4664316,355654064~8384557,384392456~2082977] 2021-07-31T02:29:05.047 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [11422525~6660963,102437886~3730024,124734943~4595042,241317308~7450669,318433435~4174606,322615554~4664316,355654064~8384557,384392456~2082977] 2021-07-31T02:29:05.053 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [11422525~6660963,102437886~3730024,124734943~4595042,241317308~7450669,314572800~12707070,355654064~8384557,381681664~4793769] 2021-07-31T02:29:05.053 INFO:tasks.workunit.client.0.smithi045.stdout:from 7 to 9 diff [872289~3582298,11422525~6660963,34802352~7512067,102437886~3730024,124734943~4595042,132705403~3160212,156774204~4559535,167531438~7281081,201478817~1199557,241317308~7450669,318433435~4174606,322615554~4664316,347819082~6186929,355654064~8384557,384392456~2082977,403808658~5668404] 2021-07-31T02:29:05.054 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~3582298,11422525~6660963,34802352~7512067,102437886~3730024,124734943~4595042,132705403~3160212,156774204~4559535,167531438~7281081,201478817~1199557,241317308~7450669,318433435~4174606,322615554~4664316,347819082~6186929,355654064~8384557,384392456~2082977,403808658~5668404] 2021-07-31T02:29:05.059 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~3582298,11422525~6660963,34802352~7512067,102437886~3730024,124734943~4595042,132705403~3160212,156774204~4559535,164857432~9955087,201478817~1199557,241317308~7450669,314572800~12707070,347819082~6186929,355654064~8384557,381681664~4793769,402653184~6823878] 2021-07-31T02:29:05.060 INFO:tasks.workunit.client.0.smithi045.stdout:from 7 to 10 diff [872289~3582298,11422525~6660963,24886295~2243069,34802352~7512067,79627966~536224,102437886~3730024,124734943~11130672,138274647~3401402,147600507~7259852,156774204~4559535,167531438~7281081,201478817~1199557,241317308~11430657,262527487~598269,318433435~4174606,322615554~4664316,347819082~6186929,355654064~8384557,373726074~3020982,384392456~2082977,403808658~5668404] 2021-07-31T02:29:05.060 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~3582298,11422525~6660963,24886295~2243069,34802352~7512067,79627966~536224,102437886~3730024,124734943~11130672,138274647~3401402,147600507~7259852,156774204~4559535,167531438~7281081,201478817~1199557,241317308~11430657,262527487~598269,318433435~4174606,322615554~4664316,347819082~6186929,355654064~8384557,373726074~3020982,384392456~2082977,403808658~5668404] 2021-07-31T02:29:05.065 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~3582298,11422525~6660963,24886295~2243069,34802352~7512067,79627966~536224,102437886~3730024,124734943~11130672,136429586~5246463,147600507~7259852,156774204~4559535,164857432~9955087,201478817~1199557,241317308~11430657,262527487~598269,314572800~12707070,347819082~6186929,355654064~8384557,373726074~3020982,381681664~4793769,402653184~6823878] 2021-07-31T02:29:05.065 INFO:tasks.workunit.client.0.smithi045.stdout:from 7 to 11 diff [872289~3582298,11422525~8739301,24886295~2243069,34802352~7512067,78244364~2928921,102437886~3730024,124734943~11130672,138274647~3401402,147600507~7259852,156774204~4559535,167531438~8214378,201478817~1199557,206474279~7964043,241317308~11430657,262527487~598269,302959050~7238531,318433435~4174606,322615554~4664316,347819082~6186929,355654064~15546953,373726074~3020982,381820951~6416356,389902499~6191760,403808658~9998078] 2021-07-31T02:29:05.066 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~3582298,11422525~8739301,24886295~2243069,34802352~7512067,78244364~2928921,102437886~3730024,124734943~11130672,138274647~3401402,147600507~7259852,156774204~4559535,167531438~8214378,201478817~1199557,206474279~7964043,241317308~11430657,262527487~598269,302959050~7238531,318433435~4174606,322615554~4664316,347819082~6186929,355654064~15546953,373726074~3020982,381820951~6416356,389902499~6191760,403808658~9998078] 2021-07-31T02:29:05.070 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~3582298,11422525~8739301,24886295~2243069,34802352~7512067,78244364~2928921,102437886~3730024,124734943~11130672,136429586~5246463,147600507~7259852,156774204~4559535,164857432~10888384,201478817~1199557,205520896~8917426,241317308~11430657,262527487~598269,302959050~7238531,314572800~12707070,347819082~6186929,355654064~15546953,373726074~3020982,381681664~6555643,389902499~6191760,402653184~11153552] 2021-07-31T02:29:05.071 INFO:tasks.workunit.client.0.smithi045.stdout:from 7 to 12 diff [872289~3582298,10475392~9686434,24886295~2243069,34802352~7512067,66056524~7829858,78244364~2928921,102437886~3730024,124734943~11130672,138274647~3401402,143078160~11782199,156774204~4559535,167531438~8214378,181972347~5415105,201478817~1199557,206474279~7964043,241317308~11430657,262527487~598269,296242351~493439,302959050~7238531,318433435~4174606,322615554~4664316,334588993~5080550,343702186~3839518,347819082~6186929,355654064~15546953,373441908~5818494,381820951~6416356,389902499~6191760,403808658~9998078] 2021-07-31T02:29:05.071 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~3582298,10475392~9686434,24886295~2243069,34802352~7512067,66056524~7829858,78244364~2928921,102437886~3730024,124734943~11130672,138274647~3401402,143078160~11782199,156774204~4559535,167531438~8214378,181972347~5415105,201478817~1199557,206474279~7964043,241317308~11430657,262527487~598269,296242351~493439,302959050~7238531,318433435~4174606,322615554~4664316,334588993~5080550,343702186~3839518,347819082~6186929,355654064~15546953,373441908~5818494,381820951~6416356,389902499~6191760,403808658~9998078] 2021-07-31T02:29:05.077 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~3582298,10475392~9686434,24886295~2243069,34802352~7512067,66056524~7829858,78244364~2928921,102437886~3730024,124734943~11130672,136429586~5246463,143078160~11782199,156774204~4559535,164857432~10888384,181120288~6267164,201478817~1199557,205520896~8917426,241317308~11430657,262527487~598269,296242351~493439,302959050~7238531,314572800~12707070,332356818~7312725,343702186~3839518,347819082~6186929,355654064~15546953,373441908~5818494,381681664~6555643,389902499~6191760,402653184~11153552] 2021-07-31T02:29:05.077 INFO:tasks.workunit.client.0.smithi045.stdout:from 7 to 13 diff [872289~3582298,10475392~9686434,24886295~2243069,27353411~2246694,34802352~7512067,66056524~7829858,78244364~2928921,89752687~3175987,102437886~3730024,124734943~11130672,138274647~3401402,143078160~11782199,156774204~4559535,167531438~10157527,181972347~5415105,201478817~12959505,241317308~11430657,253324513~6186025,262527487~598269,296242351~493439,302959050~7238531,318433435~4174606,322615554~4664316,334588993~5080550,343702186~3839518,347819082~6186929,355654064~15546953,373441908~5818494,381820951~6416356,389902499~8706347,403808658~9998078] 2021-07-31T02:29:05.077 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~3582298,10475392~9686434,24886295~2243069,27353411~2246694,34802352~7512067,66056524~7829858,78244364~2928921,89752687~3175987,102437886~3730024,124734943~11130672,138274647~3401402,143078160~11782199,156774204~4559535,167531438~10157527,181972347~5415105,201478817~12959505,241317308~11430657,253324513~6186025,262527487~598269,296242351~493439,302959050~7238531,318433435~4174606,322615554~4664316,334588993~5080550,343702186~3839518,347819082~6186929,355654064~15546953,373441908~5818494,381820951~6416356,389902499~8706347,403808658~9998078] 2021-07-31T02:29:05.083 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~3582298,10475392~9686434,24886295~2243069,27353411~2246694,34802352~7512067,66056524~7829858,78244364~2928921,89752687~3175987,102437886~3730024,124734943~11130672,136429586~5246463,143078160~11782199,156774204~4559535,164857432~12831533,181120288~6267164,201478817~12959505,241317308~11430657,253324513~6186025,262527487~598269,296242351~493439,302959050~7238531,314572800~12707070,332356818~7312725,343702186~3839518,347819082~6186929,355654064~15546953,373441908~5818494,381681664~6555643,389902499~8706347,402653184~11153552] 2021-07-31T02:29:05.084 INFO:tasks.workunit.client.0.smithi045.stdout:from 7 to 14 diff [872289~3582298,10475392~9686434,24886295~4713810,34802352~7512067,59300692~1156219,66056524~7829858,78244364~2928921,89752687~3175987,102437886~3730024,124734943~11130672,138274647~3401402,143078160~18255579,167531438~10157527,181972347~5415105,201478817~12959505,241317308~11430657,253324513~6186025,262527487~598269,264698001~5703680,296242351~493439,301341353~8856228,318433435~4174606,322615554~4664316,334588993~5080550,343702186~3839518,347819082~6186929,355654064~15546953,373441908~5818494,381820951~6416356,389902499~8706347,403808658~9998078] 2021-07-31T02:29:05.084 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~3582298,10475392~9686434,24886295~4713810,34802352~7512067,59300692~1156219,66056524~7829858,78244364~2928921,89752687~3175987,102437886~3730024,124734943~11130672,138274647~3401402,143078160~18255579,167531438~10157527,181972347~5415105,201478817~12959505,241317308~11430657,253324513~6186025,262527487~598269,264698001~5703680,296242351~493439,301341353~8856228,318433435~4174606,322615554~4664316,334588993~5080550,343702186~3839518,347819082~6186929,355654064~15546953,373441908~5818494,381820951~6416356,389902499~8706347,403808658~9998078] 2021-07-31T02:29:05.088 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~3582298,10475392~9686434,24886295~4713810,34802352~7512067,59300692~1156219,66056524~7829858,78244364~2928921,89752687~3175987,102437886~3730024,124734943~11130672,136429586~5246463,143078160~18255579,164857432~12831533,181120288~6267164,201478817~12959505,241317308~11430657,253324513~6186025,262527487~598269,264698001~5703680,296242351~493439,301341353~8856228,314572800~12707070,332356818~7312725,343702186~3839518,347819082~6186929,355654064~15546953,373441908~5818494,381681664~6555643,389902499~8706347,402653184~11153552] 2021-07-31T02:29:05.088 INFO:tasks.workunit.client.0.smithi045.stdout:from 7 to 15 diff [872289~8548834,10475392~9686434,24886295~4713810,34802352~7512067,59300692~1156219,66056524~7829858,78244364~2928921,89752687~3175987,102437886~3730024,124734943~11130672,138274647~3401402,143078160~18255579,167531438~10157527,181972347~5415105,201478817~12959505,241317308~11430657,253324513~6186025,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,318433435~4174606,322615554~4664316,334588993~19417018,355654064~15546953,373441908~5818494,381820951~6416356,389902499~8706347,403808658~9998078] 2021-07-31T02:29:05.088 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~8548834,10475392~9686434,24886295~4713810,34802352~7512067,59300692~1156219,66056524~7829858,78244364~2928921,89752687~3175987,102437886~3730024,124734943~11130672,138274647~3401402,143078160~18255579,167531438~10157527,181972347~5415105,201478817~12959505,241317308~11430657,253324513~6186025,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,318433435~4174606,322615554~4664316,334588993~19417018,355654064~15546953,373441908~5818494,381820951~6416356,389902499~8706347,403808658~9998078] 2021-07-31T02:29:05.093 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~8548834,10475392~9686434,24886295~4713810,34802352~7512067,59300692~1156219,66056524~7829858,78244364~2928921,89752687~3175987,102437886~3730024,124734943~11130672,136429586~5246463,143078160~18255579,164857432~12831533,181120288~6267164,201478817~12959505,241317308~11430657,253324513~6186025,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,314572800~12707070,332356818~21649193,355654064~15546953,373441908~5818494,381681664~6555643,389902499~8706347,402653184~11153552] 2021-07-31T02:29:05.094 INFO:tasks.workunit.client.0.smithi045.stdout:from 7 to 16 diff [872289~8548834,10475392~9686434,24372582~17941837,53817318~2949139,59300692~1156219,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,102437886~3730024,121407098~3198227,124734943~11130672,138274647~3401402,143078160~18255579,167531438~10157527,181972347~5415105,201478817~12959505,225052374~5309220,241317308~11430657,253324513~6186025,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,318433435~4174606,322615554~4664316,334588993~19417018,355654064~15546953,373441908~5818494,381820951~6416356,389902499~13078402,403808658~9998078] 2021-07-31T02:29:05.095 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~8548834,10475392~9686434,24372582~17941837,53817318~2949139,59300692~1156219,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,102437886~3730024,121407098~3198227,124734943~11130672,138274647~3401402,143078160~18255579,167531438~10157527,181972347~5415105,201478817~12959505,225052374~5309220,241317308~11430657,253324513~6186025,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,318433435~4174606,322615554~4664316,334588993~19417018,355654064~15546953,373441908~5818494,381820951~6416356,389902499~13078402,403808658~9998078] 2021-07-31T02:29:05.099 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~8548834,10475392~9686434,24372582~17941837,53817318~2949139,59300692~1156219,66056524~7829858,78244364~2928921,83886080~4661315,89752687~3175987,102437886~3730024,121407098~3198227,124734943~11130672,136429586~5246463,143078160~18255579,164857432~12831533,181120288~6267164,201478817~12959505,225052374~5309220,241317308~11430657,253324513~6186025,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,314572800~12707070,332356818~21649193,355654064~15546953,373441908~5818494,381681664~6555643,389902499~23904237] 2021-07-31T02:29:05.100 INFO:tasks.workunit.client.0.smithi045.stdout:from 7 to 17 diff [872289~8548834,9457761~10704065,24372582~17941837,53817318~2949139,59300692~1156219,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,121407098~3198227,124734943~11130672,137433861~4421821,143078160~20324185,167531438~10157527,181972347~5415105,201478817~12959505,225052374~5309220,241317308~11430657,253324513~6186025,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,318433435~4174606,322615554~4664316,334588993~19621750,355654064~15546953,373441908~5818494,381820951~6416356,389902499~13078402,403808658~9998078] 2021-07-31T02:29:05.100 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~8548834,9457761~10704065,24372582~17941837,53817318~2949139,59300692~1156219,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,121407098~3198227,124734943~11130672,137433861~4421821,143078160~20324185,167531438~10157527,181972347~5415105,201478817~12959505,225052374~5309220,241317308~11430657,253324513~6186025,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,318433435~4174606,322615554~4664316,334588993~19621750,355654064~15546953,373441908~5818494,381820951~6416356,389902499~13078402,403808658~9998078] 2021-07-31T02:29:05.105 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~8548834,9457761~10704065,24372582~17941837,53817318~2949139,59300692~1156219,66056524~7829858,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,102437886~3730024,121407098~3198227,124734943~11130672,136429586~5426096,143078160~20324185,164857432~12831533,181120288~6267164,201478817~12959505,225052374~5309220,241317308~11430657,253324513~6186025,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,314572800~12707070,332356818~21853925,355654064~15546953,373441908~5818494,381681664~6555643,389902499~23904237] 2021-07-31T02:29:05.106 INFO:tasks.workunit.client.0.smithi045.stdout:from 7 to 18 diff [872289~19289537,24372582~17941837,53817318~2949139,59300692~1156219,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,121407098~3198227,124734943~11130672,137433861~4421821,143078160~20324185,167531438~10157527,181972347~5415105,201478817~12959505,225052374~5309220,241317308~18193230,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,318433435~4174606,322615554~4664316,334588993~36612024,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:05.106 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~19289537,24372582~17941837,53817318~2949139,59300692~1156219,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,121407098~3198227,124734943~11130672,137433861~4421821,143078160~20324185,167531438~10157527,181972347~5415105,201478817~12959505,225052374~5309220,241317308~18193230,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,318433435~4174606,322615554~4664316,334588993~36612024,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:05.111 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~19289537,24372582~17941837,53817318~2949139,59300692~1156219,66056524~7829858,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,102437886~3730024,121407098~3198227,124734943~11130672,136429586~5426096,143078160~20324185,164857432~12831533,181120288~6267164,201478817~12959505,225052374~5309220,241317308~18193230,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,314572800~12707070,332356818~38844199,373441908~5818494,381681664~32125072] 2021-07-31T02:29:05.112 INFO:tasks.workunit.client.0.smithi045.stdout:from 7 to 19 diff [872289~20735768,24372582~17941837,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,121407098~3198227,124734943~18101328,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,225052374~5309220,241317308~18193230,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,318433435~4174606,322615554~4664316,334588993~36612024,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:05.112 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~20735768,24372582~17941837,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,121407098~3198227,124734943~18101328,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,225052374~5309220,241317308~18193230,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,318433435~4174606,322615554~4664316,334588993~36612024,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:05.119 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~20735768,24372582~17941837,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,102437886~3730024,121407098~3198227,124734943~18101328,143078160~20324185,164857432~12831533,181120288~6267164,201478817~19213323,221470799~1383607,225052374~5309220,241317308~18193230,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,314572800~12707070,332356818~38844199,373441908~5818494,381681664~32125072] 2021-07-31T02:29:05.119 INFO:tasks.workunit.client.0.smithi045.stdout:from 8 to 9 diff [872289~3582298,34802352~7512067,132705403~3160212,156774204~4559535,167531438~7281081,201478817~1199557,347819082~6186929,403808658~5668404] 2021-07-31T02:29:05.120 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~3582298,34802352~7512067,132705403~3160212,156774204~4559535,167531438~7281081,201478817~1199557,347819082~6186929,403808658~5668404] 2021-07-31T02:29:05.123 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~3582298,34802352~7512067,132705403~3160212,156774204~4559535,164857432~9955087,201478817~1199557,347819082~6186929,402653184~6823878] 2021-07-31T02:29:05.123 INFO:tasks.workunit.client.0.smithi045.stdout:from 8 to 10 diff [872289~3582298,24886295~2243069,34802352~7512067,79627966~536224,126900852~8964763,138274647~3401402,147600507~7259852,156774204~4559535,167531438~7281081,201478817~1199557,248659714~4088251,262527487~598269,347819082~6186929,373726074~3020982,403808658~5668404] 2021-07-31T02:29:05.123 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~3582298,24886295~2243069,34802352~7512067,79627966~536224,126900852~8964763,138274647~3401402,147600507~7259852,156774204~4559535,167531438~7281081,201478817~1199557,248659714~4088251,262527487~598269,347819082~6186929,373726074~3020982,403808658~5668404] 2021-07-31T02:29:05.129 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~3582298,24886295~2243069,34802352~7512067,79627966~536224,126900852~8964763,136429586~5246463,147600507~7259852,156774204~4559535,164857432~9955087,201478817~1199557,248659714~4088251,262527487~598269,347819082~6186929,373726074~3020982,402653184~6823878] 2021-07-31T02:29:05.129 INFO:tasks.workunit.client.0.smithi045.stdout:from 8 to 11 diff [872289~3582298,14598685~5563141,24886295~2243069,34802352~7512067,78244364~2928921,126900852~8964763,138274647~3401402,147600507~7259852,156774204~4559535,167531438~8214378,201478817~1199557,206474279~7964043,242763340~3666547,248659714~4088251,262527487~598269,302959050~7238531,347819082~6186929,363754003~7447014,373726074~3020982,381820951~6416356,389902499~6191760,403808658~9998078] 2021-07-31T02:29:05.129 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~3582298,14598685~5563141,24886295~2243069,34802352~7512067,78244364~2928921,126900852~8964763,138274647~3401402,147600507~7259852,156774204~4559535,167531438~8214378,201478817~1199557,206474279~7964043,242763340~3666547,248659714~4088251,262527487~598269,302959050~7238531,347819082~6186929,363754003~7447014,373726074~3020982,381820951~6416356,389902499~6191760,403808658~9998078] 2021-07-31T02:29:05.135 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~3582298,14598685~5563141,24886295~2243069,34802352~7512067,78244364~2928921,126900852~8964763,136429586~5246463,147600507~7259852,156774204~4559535,164857432~10888384,201478817~1199557,205520896~8917426,242763340~3666547,248659714~4088251,262527487~598269,302959050~7238531,347819082~6186929,363754003~7447014,373726074~3020982,381820951~6416356,389902499~6191760,402653184~11153552] 2021-07-31T02:29:05.135 INFO:tasks.workunit.client.0.smithi045.stdout:from 8 to 12 diff [872289~3582298,10475392~9686434,24886295~2243069,34802352~7512067,66056524~7829858,78244364~2928921,126900852~8964763,138274647~3401402,143078160~11782199,156774204~4559535,167531438~8214378,181972347~5415105,201478817~1199557,206474279~7964043,242763340~3666547,248659714~4088251,262527487~598269,296242351~493439,302959050~7238531,334588993~5080550,343702186~3839518,347819082~6186929,363754003~7447014,373441908~5818494,381820951~6416356,389902499~6191760,403808658~9998078] 2021-07-31T02:29:05.135 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~3582298,10475392~9686434,24886295~2243069,34802352~7512067,66056524~7829858,78244364~2928921,126900852~8964763,138274647~3401402,143078160~11782199,156774204~4559535,167531438~8214378,181972347~5415105,201478817~1199557,206474279~7964043,242763340~3666547,248659714~4088251,262527487~598269,296242351~493439,302959050~7238531,334588993~5080550,343702186~3839518,347819082~6186929,363754003~7447014,373441908~5818494,381820951~6416356,389902499~6191760,403808658~9998078] 2021-07-31T02:29:05.141 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~3582298,10475392~9686434,24886295~2243069,34802352~7512067,66056524~7829858,78244364~2928921,126900852~8964763,136429586~5246463,143078160~11782199,156774204~4559535,164857432~10888384,181120288~6267164,201478817~1199557,205520896~8917426,242763340~3666547,248659714~4088251,262527487~598269,296242351~493439,302959050~7238531,332356818~7312725,343702186~3839518,347819082~6186929,363754003~7447014,373441908~5818494,381820951~6416356,389902499~6191760,402653184~11153552] 2021-07-31T02:29:05.141 INFO:tasks.workunit.client.0.smithi045.stdout:from 8 to 13 diff [872289~3582298,10475392~9686434,24886295~2243069,27353411~2246694,34802352~7512067,66056524~7829858,78244364~2928921,89752687~3175987,126020102~9845513,138274647~3401402,143078160~11782199,156774204~4559535,167531438~10157527,181972347~5415105,201478817~12959505,242763340~3666547,248659714~4088251,253324513~6186025,262527487~598269,296242351~493439,302959050~7238531,334588993~5080550,343702186~3839518,347819082~6186929,362777592~8423425,373441908~5818494,381820951~6416356,389902499~8706347,403808658~9998078] 2021-07-31T02:29:05.141 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~3582298,10475392~9686434,24886295~2243069,27353411~2246694,34802352~7512067,66056524~7829858,78244364~2928921,89752687~3175987,126020102~9845513,138274647~3401402,143078160~11782199,156774204~4559535,167531438~10157527,181972347~5415105,201478817~12959505,242763340~3666547,248659714~4088251,253324513~6186025,262527487~598269,296242351~493439,302959050~7238531,334588993~5080550,343702186~3839518,347819082~6186929,362777592~8423425,373441908~5818494,381820951~6416356,389902499~8706347,403808658~9998078] 2021-07-31T02:29:05.146 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~3582298,10475392~9686434,24886295~2243069,27353411~2246694,34802352~7512067,66056524~7829858,78244364~2928921,89752687~3175987,126020102~9845513,136429586~5246463,143078160~11782199,156774204~4559535,164857432~12831533,181120288~6267164,201478817~12959505,242763340~3666547,248659714~4088251,253324513~6186025,262527487~598269,296242351~493439,302959050~7238531,332356818~7312725,343702186~3839518,347819082~6186929,362777592~8423425,373441908~5818494,381820951~6416356,389902499~8706347,402653184~11153552] 2021-07-31T02:29:05.147 INFO:tasks.workunit.client.0.smithi045.stdout:from 8 to 14 diff [872289~3582298,10475392~9686434,24886295~4713810,34802352~7512067,59300692~1156219,66056524~7829858,78244364~2928921,89752687~3175987,126020102~9845513,138274647~3401402,143078160~18255579,167531438~10157527,181972347~5415105,201478817~12959505,242763340~3666547,248659714~4088251,253324513~6186025,262527487~598269,264698001~5703680,296242351~493439,301341353~8856228,334588993~5080550,343702186~3839518,347819082~6186929,362777592~8423425,373441908~5818494,381820951~6416356,389902499~8706347,403808658~9998078] 2021-07-31T02:29:05.147 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~3582298,10475392~9686434,24886295~4713810,34802352~7512067,59300692~1156219,66056524~7829858,78244364~2928921,89752687~3175987,126020102~9845513,138274647~3401402,143078160~18255579,167531438~10157527,181972347~5415105,201478817~12959505,242763340~3666547,248659714~4088251,253324513~6186025,262527487~598269,264698001~5703680,296242351~493439,301341353~8856228,334588993~5080550,343702186~3839518,347819082~6186929,362777592~8423425,373441908~5818494,381820951~6416356,389902499~8706347,403808658~9998078] 2021-07-31T02:29:05.152 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~3582298,10475392~9686434,24886295~4713810,34802352~7512067,59300692~1156219,66056524~7829858,78244364~2928921,89752687~3175987,126020102~9845513,136429586~5246463,143078160~18255579,164857432~12831533,181120288~6267164,201478817~12959505,242763340~3666547,248659714~4088251,253324513~6186025,262527487~598269,264698001~5703680,296242351~493439,301341353~8856228,332356818~7312725,343702186~3839518,347819082~6186929,362777592~8423425,373441908~5818494,381820951~6416356,389902499~8706347,402653184~11153552] 2021-07-31T02:29:05.152 INFO:tasks.workunit.client.0.smithi045.stdout:from 8 to 15 diff [872289~8548834,10475392~9686434,24886295~4713810,34802352~7512067,59300692~1156219,66056524~7829858,78244364~2928921,89752687~3175987,126020102~9845513,138274647~3401402,143078160~18255579,167531438~10157527,181972347~5415105,201478817~12959505,242763340~3666547,248659714~4088251,253324513~6186025,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,334588993~19417018,359181831~3015778,362777592~8423425,373441908~5818494,381820951~6416356,389902499~8706347,403808658~9998078] 2021-07-31T02:29:05.152 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~8548834,10475392~9686434,24886295~4713810,34802352~7512067,59300692~1156219,66056524~7829858,78244364~2928921,89752687~3175987,126020102~9845513,138274647~3401402,143078160~18255579,167531438~10157527,181972347~5415105,201478817~12959505,242763340~3666547,248659714~4088251,253324513~6186025,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,334588993~19417018,359181831~3015778,362777592~8423425,373441908~5818494,381820951~6416356,389902499~8706347,403808658~9998078] 2021-07-31T02:29:05.158 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~8548834,10475392~9686434,24886295~4713810,34802352~7512067,59300692~1156219,66056524~7829858,78244364~2928921,89752687~3175987,126020102~9845513,136429586~5246463,143078160~18255579,164857432~12831533,181120288~6267164,201478817~12959505,242763340~3666547,248659714~4088251,253324513~6186025,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,332356818~21649193,359181831~3015778,362777592~8423425,373441908~5818494,381820951~6416356,389902499~8706347,402653184~11153552] 2021-07-31T02:29:05.158 INFO:tasks.workunit.client.0.smithi045.stdout:from 8 to 16 diff [872289~8548834,10475392~9686434,24372582~17941837,53817318~2949139,59300692~1156219,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,121407098~3198227,126020102~9845513,138274647~3401402,143078160~18255579,167531438~10157527,181972347~5415105,201478817~12959505,225052374~5309220,242763340~3666547,248659714~4088251,253324513~6186025,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,334588993~19417018,359181831~3015778,362777592~8423425,373441908~5818494,381820951~6416356,389902499~13078402,403808658~9998078] 2021-07-31T02:29:05.158 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~8548834,10475392~9686434,24372582~17941837,53817318~2949139,59300692~1156219,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,121407098~3198227,126020102~9845513,138274647~3401402,143078160~18255579,167531438~10157527,181972347~5415105,201478817~12959505,225052374~5309220,242763340~3666547,248659714~4088251,253324513~6186025,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,334588993~19417018,359181831~3015778,362777592~8423425,373441908~5818494,381820951~6416356,389902499~13078402,403808658~9998078] 2021-07-31T02:29:05.164 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~8548834,10475392~9686434,24372582~17941837,53817318~2949139,59300692~1156219,66056524~7829858,78244364~2928921,83886080~4661315,89752687~3175987,121407098~3198227,126020102~9845513,136429586~5246463,143078160~18255579,164857432~12831533,181120288~6267164,201478817~12959505,225052374~5309220,242763340~3666547,248659714~4088251,253324513~6186025,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,332356818~21649193,359181831~3015778,362777592~8423425,373441908~5818494,381820951~6416356,389902499~23904237] 2021-07-31T02:29:05.164 INFO:tasks.workunit.client.0.smithi045.stdout:from 8 to 17 diff [872289~8548834,9457761~10704065,24372582~17941837,53817318~2949139,59300692~1156219,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~9845513,137433861~4421821,143078160~20324185,167531438~10157527,181972347~5415105,201478817~12959505,225052374~5309220,242763340~3666547,248659714~4088251,253324513~6186025,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,334588993~19621750,359181831~3015778,362777592~8423425,373441908~5818494,381820951~6416356,389902499~13078402,403808658~9998078] 2021-07-31T02:29:05.164 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~8548834,9457761~10704065,24372582~17941837,53817318~2949139,59300692~1156219,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~9845513,137433861~4421821,143078160~20324185,167531438~10157527,181972347~5415105,201478817~12959505,225052374~5309220,242763340~3666547,248659714~4088251,253324513~6186025,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,334588993~19621750,359181831~3015778,362777592~8423425,373441908~5818494,381820951~6416356,389902499~13078402,403808658~9998078] 2021-07-31T02:29:05.169 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~8548834,9457761~10704065,24372582~17941837,53817318~2949139,59300692~1156219,66056524~7829858,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,121407098~3198227,126020102~9845513,136429586~5426096,143078160~20324185,164857432~12831533,181120288~6267164,201478817~12959505,225052374~5309220,242763340~3666547,248659714~4088251,253324513~6186025,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,332356818~21853925,359181831~3015778,362777592~8423425,373441908~5818494,381820951~6416356,389902499~23904237] 2021-07-31T02:29:05.170 INFO:tasks.workunit.client.0.smithi045.stdout:from 8 to 18 diff [872289~19289537,24372582~17941837,53817318~2949139,59300692~1156219,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~9845513,137433861~4421821,143078160~20324185,167531438~10157527,181972347~5415105,201478817~12959505,225052374~5309220,242763340~3666547,248659714~10850824,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,334588993~23788434,359181831~12019186,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:05.170 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~19289537,24372582~17941837,53817318~2949139,59300692~1156219,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~9845513,137433861~4421821,143078160~20324185,167531438~10157527,181972347~5415105,201478817~12959505,225052374~5309220,242763340~3666547,248659714~10850824,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,334588993~23788434,359181831~12019186,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:05.175 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~19289537,24372582~17941837,53817318~2949139,59300692~1156219,66056524~7829858,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,121407098~3198227,126020102~9845513,136429586~5426096,143078160~20324185,164857432~12831533,181120288~6267164,201478817~12959505,225052374~5309220,242763340~3666547,248659714~10850824,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,332356818~26020609,359181831~12019186,373441908~5818494,381820951~31985785] 2021-07-31T02:29:05.175 INFO:tasks.workunit.client.0.smithi045.stdout:from 8 to 19 diff [872289~20735768,24372582~17941837,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~16816169,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,225052374~5309220,242763340~3666547,248659714~10850824,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,334588993~23788434,359181831~12019186,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:05.175 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~20735768,24372582~17941837,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~16816169,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,225052374~5309220,242763340~3666547,248659714~10850824,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,334588993~23788434,359181831~12019186,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:05.181 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~20735768,24372582~17941837,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,121407098~3198227,126020102~16816169,143078160~20324185,164857432~12831533,181120288~6267164,201478817~19213323,221470799~1383607,225052374~5309220,242763340~3666547,248659714~10850824,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,332356818~26020609,359181831~12019186,373441908~5818494,381820951~31985785] 2021-07-31T02:29:05.181 INFO:tasks.workunit.client.0.smithi045.stdout:from 9 to 10 diff [24886295~2243069,36634458~572921,79627966~536224,126900852~6720050,138274647~3401402,147600507~7259852,248659714~4088251,262527487~598269,373726074~3020982] 2021-07-31T02:29:05.181 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [24886295~2243069,36634458~572921,79627966~536224,126900852~6720050,138274647~3401402,147600507~7259852,248659714~4088251,262527487~598269,373726074~3020982] 2021-07-31T02:29:05.187 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [24886295~2243069,36634458~572921,79627966~536224,126900852~6720050,136429586~5246463,147600507~7259852,248659714~4088251,262527487~598269,373726074~3020982] 2021-07-31T02:29:05.187 INFO:tasks.workunit.client.0.smithi045.stdout:from 9 to 11 diff [14598685~5563141,24886295~2243069,36634458~572921,78244364~2928921,126900852~6720050,138274647~3401402,147600507~7259852,167792807~7953009,206474279~7964043,242763340~3666547,248659714~4088251,262527487~598269,302959050~7238531,363754003~7447014,373726074~3020982,381820951~6416356,389902499~6191760,408916890~4889846] 2021-07-31T02:29:05.188 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [14598685~5563141,24886295~2243069,36634458~572921,78244364~2928921,126900852~6720050,138274647~3401402,147600507~7259852,167792807~7953009,206474279~7964043,242763340~3666547,248659714~4088251,262527487~598269,302959050~7238531,363754003~7447014,373726074~3020982,381820951~6416356,389902499~6191760,408916890~4889846] 2021-07-31T02:29:05.193 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [14598685~5563141,24886295~2243069,36634458~572921,78244364~2928921,126900852~6720050,136429586~5246463,147600507~7259852,167792807~7953009,205520896~8917426,242763340~3666547,248659714~4088251,262527487~598269,302959050~7238531,363754003~7447014,373726074~3020982,381820951~6416356,389902499~6191760,408916890~4889846] 2021-07-31T02:29:05.194 INFO:tasks.workunit.client.0.smithi045.stdout:from 9 to 12 diff [10475392~9686434,24886295~2243069,36634458~572921,66056524~7829858,78244364~2928921,126900852~6720050,138274647~3401402,143078160~11782199,167792807~7953009,181972347~5415105,206474279~7964043,242763340~3666547,248659714~4088251,262527487~598269,296242351~493439,302959050~7238531,334588993~5080550,343702186~3839518,363754003~7447014,373441908~5818494,381820951~6416356,389902499~6191760,404169452~1052236,408916890~4889846] 2021-07-31T02:29:05.195 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [10475392~9686434,24886295~2243069,36634458~572921,66056524~7829858,78244364~2928921,126900852~6720050,138274647~3401402,143078160~11782199,167792807~7953009,181972347~5415105,206474279~7964043,242763340~3666547,248659714~4088251,262527487~598269,296242351~493439,302959050~7238531,334588993~5080550,343702186~3839518,363754003~7447014,373441908~5818494,381820951~6416356,389902499~6191760,404169452~1052236,408916890~4889846] 2021-07-31T02:29:05.198 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [10475392~9686434,24886295~2243069,36634458~572921,66056524~7829858,78244364~2928921,126900852~6720050,136429586~5246463,143078160~11782199,167792807~7953009,181120288~6267164,205520896~8917426,242763340~3666547,248659714~4088251,262527487~598269,296242351~493439,302959050~7238531,332356818~7312725,343702186~3839518,363754003~7447014,373441908~5818494,381820951~6416356,389902499~6191760,404169452~1052236,408916890~4889846] 2021-07-31T02:29:05.199 INFO:tasks.workunit.client.0.smithi045.stdout:from 9 to 13 diff [10475392~9686434,24886295~2243069,27353411~2246694,36634458~572921,66056524~7829858,78244364~2928921,89752687~3175987,126020102~8880466,138274647~3401402,143078160~11782199,167792807~9896158,181972347~5415105,202180755~12257567,242763340~3666547,248659714~4088251,253324513~6186025,262527487~598269,296242351~493439,302959050~7238531,334588993~5080550,343702186~3839518,362777592~8423425,373441908~5818494,381820951~6416356,389902499~8706347,404169452~1052236,408916890~4889846] 2021-07-31T02:29:05.199 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [10475392~9686434,24886295~2243069,27353411~2246694,36634458~572921,66056524~7829858,78244364~2928921,89752687~3175987,126020102~8880466,138274647~3401402,143078160~11782199,167792807~9896158,181972347~5415105,202180755~12257567,242763340~3666547,248659714~4088251,253324513~6186025,262527487~598269,296242351~493439,302959050~7238531,334588993~5080550,343702186~3839518,362777592~8423425,373441908~5818494,381820951~6416356,389902499~8706347,404169452~1052236,408916890~4889846] 2021-07-31T02:29:05.203 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [10475392~9686434,24886295~2243069,27353411~2246694,36634458~572921,66056524~7829858,78244364~2928921,89752687~3175987,126020102~8880466,136429586~5246463,143078160~11782199,167792807~9896158,181120288~6267164,202180755~12257567,242763340~3666547,248659714~4088251,253324513~6186025,262527487~598269,296242351~493439,302959050~7238531,332356818~7312725,343702186~3839518,362777592~8423425,373441908~5818494,381820951~6416356,389902499~8706347,404169452~1052236,408916890~4889846] 2021-07-31T02:29:05.203 INFO:tasks.workunit.client.0.smithi045.stdout:from 9 to 14 diff [10475392~9686434,24886295~4713810,36634458~572921,59300692~1156219,66056524~7829858,78244364~2928921,89752687~3175987,126020102~8880466,138274647~3401402,143078160~15345483,167792807~9896158,181972347~5415105,202180755~12257567,242763340~3666547,248659714~4088251,253324513~6186025,262527487~598269,264698001~5703680,296242351~493439,301341353~8856228,334588993~5080550,343702186~3839518,362777592~8423425,373441908~5818494,381820951~6416356,389902499~8706347,404169452~1052236,408916890~4889846] 2021-07-31T02:29:05.204 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [10475392~9686434,24886295~4713810,36634458~572921,59300692~1156219,66056524~7829858,78244364~2928921,89752687~3175987,126020102~8880466,138274647~3401402,143078160~15345483,167792807~9896158,181972347~5415105,202180755~12257567,242763340~3666547,248659714~4088251,253324513~6186025,262527487~598269,264698001~5703680,296242351~493439,301341353~8856228,334588993~5080550,343702186~3839518,362777592~8423425,373441908~5818494,381820951~6416356,389902499~8706347,404169452~1052236,408916890~4889846] 2021-07-31T02:29:05.209 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [10475392~9686434,24886295~4713810,36634458~572921,59300692~1156219,66056524~7829858,78244364~2928921,89752687~3175987,126020102~8880466,136429586~5246463,143078160~15345483,167792807~9896158,181120288~6267164,202180755~12257567,242763340~3666547,248659714~4088251,253324513~6186025,262527487~598269,264698001~5703680,296242351~493439,301341353~8856228,332356818~7312725,343702186~3839518,362777592~8423425,373441908~5818494,381820951~6416356,389902499~8706347,404169452~1052236,408916890~4889846] 2021-07-31T02:29:05.209 INFO:tasks.workunit.client.0.smithi045.stdout:from 9 to 15 diff [1823868~7597255,10475392~9686434,24886295~4713810,36634458~572921,59300692~1156219,66056524~7829858,78244364~2928921,89752687~3175987,126020102~8880466,138274647~3401402,143078160~15345483,167792807~9896158,181972347~5415105,202180755~12257567,242763340~3666547,248659714~4088251,253324513~6186025,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,334588993~15115237,359181831~3015778,362777592~8423425,373441908~5818494,381820951~6416356,389902499~8706347,404169452~1052236,406364931~1096263,408916890~4889846] 2021-07-31T02:29:05.209 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [1823868~7597255,10475392~9686434,24886295~4713810,36634458~572921,59300692~1156219,66056524~7829858,78244364~2928921,89752687~3175987,126020102~8880466,138274647~3401402,143078160~15345483,167792807~9896158,181972347~5415105,202180755~12257567,242763340~3666547,248659714~4088251,253324513~6186025,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,334588993~15115237,359181831~3015778,362777592~8423425,373441908~5818494,381820951~6416356,389902499~8706347,404169452~1052236,406364931~1096263,408916890~4889846] 2021-07-31T02:29:05.215 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [1823868~7597255,10475392~9686434,24886295~4713810,36634458~572921,59300692~1156219,66056524~7829858,78244364~2928921,89752687~3175987,126020102~8880466,136429586~5246463,143078160~15345483,167792807~9896158,181120288~6267164,202180755~12257567,242763340~3666547,248659714~4088251,253324513~6186025,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,332356818~17347412,359181831~3015778,362777592~8423425,373441908~5818494,381820951~6416356,389902499~8706347,404169452~1052236,406364931~1096263,408916890~4889846] 2021-07-31T02:29:05.215 INFO:tasks.workunit.client.0.smithi045.stdout:from 9 to 16 diff [1823868~7597255,10475392~9686434,24372582~11685962,36634458~572921,53817318~2949139,59300692~1156219,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,121407098~3198227,126020102~8880466,138274647~3401402,143078160~15345483,167792807~9896158,181972347~5415105,202180755~12257567,225052374~5309220,242763340~3666547,248659714~4088251,253324513~6186025,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,334588993~15115237,359181831~3015778,362777592~8423425,373441908~5818494,381820951~6416356,389902499~13078402,404169452~1052236,406364931~1096263,408916890~4889846] 2021-07-31T02:29:05.216 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [1823868~7597255,10475392~9686434,24372582~11685962,36634458~572921,53817318~2949139,59300692~1156219,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,121407098~3198227,126020102~8880466,138274647~3401402,143078160~15345483,167792807~9896158,181972347~5415105,202180755~12257567,225052374~5309220,242763340~3666547,248659714~4088251,253324513~6186025,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,334588993~15115237,359181831~3015778,362777592~8423425,373441908~5818494,381820951~6416356,389902499~13078402,404169452~1052236,406364931~1096263,408916890~4889846] 2021-07-31T02:29:05.221 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [1823868~7597255,10475392~9686434,24372582~11685962,36634458~572921,53817318~2949139,59300692~1156219,66056524~7829858,78244364~2928921,83886080~4661315,89752687~3175987,121407098~3198227,126020102~8880466,136429586~5246463,143078160~15345483,167792807~9896158,181120288~6267164,202180755~12257567,225052374~5309220,242763340~3666547,248659714~4088251,253324513~6186025,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,332356818~17347412,359181831~3015778,362777592~8423425,373441908~5818494,381820951~6416356,389902499~13078402,404169452~1052236,406364931~1096263,408916890~4889846] 2021-07-31T02:29:05.222 INFO:tasks.workunit.client.0.smithi045.stdout:from 9 to 17 diff [1823868~7597255,9457761~10704065,24372582~11685962,36634458~572921,53817318~2949139,59300692~1156219,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,137433861~4421821,143078160~20324185,167792807~9896158,181972347~5415105,202180755~12257567,225052374~5309220,242763340~3666547,248659714~4088251,253324513~6186025,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,334588993~15115237,351585344~2625399,359181831~3015778,362777592~8423425,373441908~5818494,381820951~6416356,389902499~13078402,404169452~1052236,406364931~1096263,408916890~4889846] 2021-07-31T02:29:05.222 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [1823868~7597255,9457761~10704065,24372582~11685962,36634458~572921,53817318~2949139,59300692~1156219,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,137433861~4421821,143078160~20324185,167792807~9896158,181972347~5415105,202180755~12257567,225052374~5309220,242763340~3666547,248659714~4088251,253324513~6186025,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,334588993~15115237,351585344~2625399,359181831~3015778,362777592~8423425,373441908~5818494,381820951~6416356,389902499~13078402,404169452~1052236,406364931~1096263,408916890~4889846] 2021-07-31T02:29:05.227 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [1823868~7597255,9457761~10704065,24372582~11685962,36634458~572921,53817318~2949139,59300692~1156219,66056524~7829858,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,136429586~5426096,143078160~20324185,167792807~9896158,181120288~6267164,202180755~12257567,225052374~5309220,242763340~3666547,248659714~4088251,253324513~6186025,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,332356818~17347412,351585344~2625399,359181831~3015778,362777592~8423425,373441908~5818494,381820951~6416356,389902499~13078402,404169452~1052236,406364931~1096263,408916890~4889846] 2021-07-31T02:29:05.228 INFO:tasks.workunit.client.0.smithi045.stdout:from 9 to 18 diff [1823868~18337958,24372582~14188618,53817318~2949139,59300692~1156219,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,137433861~4421821,143078160~20324185,167792807~9896158,181972347~5415105,202180755~12257567,225052374~5309220,242763340~3666547,248659714~10850824,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,334588993~15115237,351585344~6792083,359181831~12019186,373441908~5818494,381820951~21159950,404169452~1052236,406364931~1096263,408916890~4889846] 2021-07-31T02:29:05.228 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [1823868~18337958,24372582~14188618,53817318~2949139,59300692~1156219,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,137433861~4421821,143078160~20324185,167792807~9896158,181972347~5415105,202180755~12257567,225052374~5309220,242763340~3666547,248659714~10850824,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,334588993~15115237,351585344~6792083,359181831~12019186,373441908~5818494,381820951~21159950,404169452~1052236,406364931~1096263,408916890~4889846] 2021-07-31T02:29:05.233 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [1823868~18337958,24372582~14188618,53817318~2949139,59300692~1156219,66056524~7829858,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,136429586~5426096,143078160~20324185,167792807~9896158,181120288~6267164,202180755~12257567,225052374~5309220,242763340~3666547,248659714~10850824,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,332356818~17347412,351585344~6792083,359181831~12019186,373441908~5818494,381820951~21159950,404169452~1052236,406364931~1096263,408916890~4889846] 2021-07-31T02:29:05.233 INFO:tasks.workunit.client.0.smithi045.stdout:from 9 to 19 diff [1823868~19784189,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143078160~20324185,167792807~9896158,181972347~5415105,202180755~18511385,221470799~1383607,225052374~5309220,242763340~3666547,248659714~10850824,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,334588993~15115237,351585344~6792083,359181831~12019186,373441908~5818494,381820951~21159950,404169452~1052236,406364931~1096263,408916890~4889846] 2021-07-31T02:29:05.233 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [1823868~19784189,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143078160~20324185,167792807~9896158,181972347~5415105,202180755~18511385,221470799~1383607,225052374~5309220,242763340~3666547,248659714~10850824,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,334588993~15115237,351585344~6792083,359181831~12019186,373441908~5818494,381820951~21159950,404169452~1052236,406364931~1096263,408916890~4889846] 2021-07-31T02:29:05.238 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [1823868~19784189,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143078160~20324185,167792807~9896158,181120288~6267164,202180755~18511385,221470799~1383607,225052374~5309220,242763340~3666547,248659714~10850824,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,332356818~17347412,351585344~6792083,359181831~12019186,373441908~5818494,381820951~21159950,404169452~1052236,406364931~1096263,408916890~4889846] 2021-07-31T02:29:05.239 INFO:tasks.workunit.client.0.smithi045.stdout:from 10 to 11 diff [14598685~5563141,78244364~2928921,167792807~7953009,206474279~7964043,242763340~3666547,302959050~7238531,363754003~7447014,381820951~6416356,389902499~6191760,408916890~4889846] 2021-07-31T02:29:05.239 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [14598685~5563141,78244364~2928921,167792807~7953009,206474279~7964043,242763340~3666547,302959050~7238531,363754003~7447014,381820951~6416356,389902499~6191760,408916890~4889846] 2021-07-31T02:29:05.244 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [14598685~5563141,78244364~2928921,167792807~7953009,205520896~8917426,242763340~3666547,302959050~7238531,363754003~7447014,381820951~6416356,389902499~6191760,408916890~4889846] 2021-07-31T02:29:05.244 INFO:tasks.workunit.client.0.smithi045.stdout:from 10 to 12 diff [10475392~9686434,66056524~7829858,78244364~2928921,143078160~7988495,167792807~7953009,181972347~5415105,206474279~7964043,242763340~3666547,296242351~493439,302959050~7238531,334588993~5080550,343702186~3839518,363754003~7447014,373441908~5818494,381820951~6416356,389902499~6191760,404169452~1052236,408916890~4889846] 2021-07-31T02:29:05.245 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [10475392~9686434,66056524~7829858,78244364~2928921,143078160~7988495,167792807~7953009,181972347~5415105,206474279~7964043,242763340~3666547,296242351~493439,302959050~7238531,334588993~5080550,343702186~3839518,363754003~7447014,373441908~5818494,381820951~6416356,389902499~6191760,404169452~1052236,408916890~4889846] 2021-07-31T02:29:05.250 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [10475392~9686434,66056524~7829858,78244364~2928921,143078160~7988495,167792807~7953009,181120288~6267164,205520896~8917426,242763340~3666547,296242351~493439,302959050~7238531,332356818~7312725,343702186~3839518,363754003~7447014,373441908~5818494,381820951~6416356,389902499~6191760,404169452~1052236,408916890~4889846] 2021-07-31T02:29:05.250 INFO:tasks.workunit.client.0.smithi045.stdout:from 10 to 13 diff [10475392~9686434,27353411~2246694,66056524~7829858,78244364~2928921,89752687~3175987,126020102~8880466,143078160~7988495,167792807~9896158,181972347~5415105,202180755~12257567,242763340~3666547,253324513~6186025,296242351~493439,302959050~7238531,334588993~5080550,343702186~3839518,362777592~8423425,373441908~5818494,381820951~6416356,389902499~8706347,404169452~1052236,408916890~4889846] 2021-07-31T02:29:05.251 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [10475392~9686434,27353411~2246694,66056524~7829858,78244364~2928921,89752687~3175987,126020102~8880466,143078160~7988495,167792807~9896158,181972347~5415105,202180755~12257567,242763340~3666547,253324513~6186025,296242351~493439,302959050~7238531,334588993~5080550,343702186~3839518,362777592~8423425,373441908~5818494,381820951~6416356,389902499~8706347,404169452~1052236,408916890~4889846] 2021-07-31T02:29:05.255 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [10475392~9686434,27353411~2246694,66056524~7829858,78244364~2928921,89752687~3175987,126020102~8880466,143078160~7988495,167792807~9896158,181120288~6267164,202180755~12257567,242763340~3666547,253324513~6186025,296242351~493439,302959050~7238531,332356818~7312725,343702186~3839518,362777592~8423425,373441908~5818494,381820951~6416356,389902499~8706347,404169452~1052236,408916890~4889846] 2021-07-31T02:29:05.256 INFO:tasks.workunit.client.0.smithi045.stdout:from 10 to 14 diff [10475392~9686434,25410155~4189950,59300692~1156219,66056524~7829858,78244364~2928921,89752687~3175987,126020102~8880466,143078160~7988495,154729104~3694539,167792807~9896158,181972347~5415105,202180755~12257567,242763340~3666547,253324513~6186025,264698001~5703680,296242351~493439,301341353~8856228,334588993~5080550,343702186~3839518,362777592~8423425,373441908~5818494,381820951~6416356,389902499~8706347,404169452~1052236,408916890~4889846] 2021-07-31T02:29:05.256 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [10475392~9686434,25410155~4189950,59300692~1156219,66056524~7829858,78244364~2928921,89752687~3175987,126020102~8880466,143078160~7988495,154729104~3694539,167792807~9896158,181972347~5415105,202180755~12257567,242763340~3666547,253324513~6186025,264698001~5703680,296242351~493439,301341353~8856228,334588993~5080550,343702186~3839518,362777592~8423425,373441908~5818494,381820951~6416356,389902499~8706347,404169452~1052236,408916890~4889846] 2021-07-31T02:29:05.265 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [10475392~9686434,25410155~4189950,59300692~1156219,66056524~7829858,78244364~2928921,89752687~3175987,126020102~8880466,143078160~7988495,154729104~3694539,167792807~9896158,181120288~6267164,202180755~12257567,242763340~3666547,253324513~6186025,264698001~5703680,296242351~493439,301341353~8856228,332356818~7312725,343702186~3839518,362777592~8423425,373441908~5818494,381820951~6416356,389902499~8706347,404169452~1052236,408916890~4889846] 2021-07-31T02:29:05.266 INFO:tasks.workunit.client.0.smithi045.stdout:from 10 to 15 diff [1823868~7597255,10475392~9686434,25410155~4189950,59300692~1156219,66056524~7829858,78244364~2928921,89752687~3175987,126020102~8880466,143078160~11521808,154729104~3694539,167792807~9896158,181972347~5415105,202180755~12257567,242763340~3666547,253324513~6186025,264698001~7989139,296242351~493439,301341353~9191315,334588993~15115237,359181831~3015778,362777592~8423425,373441908~5818494,381820951~6416356,389902499~8706347,404169452~1052236,406364931~1096263,408916890~4889846] 2021-07-31T02:29:05.267 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [1823868~7597255,10475392~9686434,25410155~4189950,59300692~1156219,66056524~7829858,78244364~2928921,89752687~3175987,126020102~8880466,143078160~11521808,154729104~3694539,167792807~9896158,181972347~5415105,202180755~12257567,242763340~3666547,253324513~6186025,264698001~7989139,296242351~493439,301341353~9191315,334588993~15115237,359181831~3015778,362777592~8423425,373441908~5818494,381820951~6416356,389902499~8706347,404169452~1052236,406364931~1096263,408916890~4889846] 2021-07-31T02:29:05.271 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [1823868~7597255,10475392~9686434,25410155~4189950,59300692~1156219,66056524~7829858,78244364~2928921,89752687~3175987,126020102~8880466,143078160~11521808,154729104~3694539,167792807~9896158,181120288~6267164,202180755~12257567,242763340~3666547,253324513~6186025,264698001~7989139,296242351~493439,301341353~9191315,332356818~17347412,359181831~3015778,362777592~8423425,373441908~5818494,381820951~6416356,389902499~8706347,404169452~1052236,406364931~1096263,408916890~4889846] 2021-07-31T02:29:05.271 INFO:tasks.workunit.client.0.smithi045.stdout:from 10 to 16 diff [1823868~7597255,10475392~9686434,24372582~11685962,53817318~2949139,59300692~1156219,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,121407098~3198227,126020102~8880466,143078160~11521808,154729104~3694539,167792807~9896158,181972347~5415105,202180755~12257567,225052374~5309220,242763340~3666547,253324513~6186025,264698001~7989139,296242351~493439,301341353~9191315,334588993~15115237,359181831~3015778,362777592~8423425,373441908~5818494,381820951~6416356,389902499~13078402,404169452~1052236,406364931~1096263,408916890~4889846] 2021-07-31T02:29:05.273 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [1823868~7597255,10475392~9686434,24372582~11685962,53817318~2949139,59300692~1156219,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,121407098~3198227,126020102~8880466,143078160~11521808,154729104~3694539,167792807~9896158,181972347~5415105,202180755~12257567,225052374~5309220,242763340~3666547,253324513~6186025,264698001~7989139,296242351~493439,301341353~9191315,334588993~15115237,359181831~3015778,362777592~8423425,373441908~5818494,381820951~6416356,389902499~13078402,404169452~1052236,406364931~1096263,408916890~4889846] 2021-07-31T02:29:05.273 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [1823868~7597255,10475392~9686434,24372582~11685962,53817318~2949139,59300692~1156219,66056524~7829858,78244364~2928921,83886080~4661315,89752687~3175987,121407098~3198227,126020102~8880466,143078160~11521808,154729104~3694539,167792807~9896158,181120288~6267164,202180755~12257567,225052374~5309220,242763340~3666547,253324513~6186025,264698001~7989139,296242351~493439,301341353~9191315,332356818~17347412,359181831~3015778,362777592~8423425,373441908~5818494,381820951~6416356,389902499~13078402,404169452~1052236,406364931~1096263,408916890~4889846] 2021-07-31T02:29:05.274 INFO:tasks.workunit.client.0.smithi045.stdout:from 10 to 17 diff [1823868~7597255,9457761~10704065,24372582~11685962,53817318~2949139,59300692~1156219,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,137433861~4421821,143078160~11521808,154729104~8673241,167792807~9896158,181972347~5415105,202180755~12257567,225052374~5309220,242763340~3666547,253324513~6186025,264698001~7989139,296242351~493439,301341353~9191315,334588993~15115237,351585344~2625399,359181831~3015778,362777592~8423425,373441908~5818494,381820951~6416356,389902499~13078402,404169452~1052236,406364931~1096263,408916890~4889846] 2021-07-31T02:29:05.274 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [1823868~7597255,9457761~10704065,24372582~11685962,53817318~2949139,59300692~1156219,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,137433861~4421821,143078160~11521808,154729104~8673241,167792807~9896158,181972347~5415105,202180755~12257567,225052374~5309220,242763340~3666547,253324513~6186025,264698001~7989139,296242351~493439,301341353~9191315,334588993~15115237,351585344~2625399,359181831~3015778,362777592~8423425,373441908~5818494,381820951~6416356,389902499~13078402,404169452~1052236,406364931~1096263,408916890~4889846] 2021-07-31T02:29:05.279 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [1823868~7597255,9457761~10704065,24372582~11685962,53817318~2949139,59300692~1156219,66056524~7829858,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,137433861~4421821,143078160~11521808,154729104~8673241,167792807~9896158,181120288~6267164,202180755~12257567,225052374~5309220,242763340~3666547,253324513~6186025,264698001~7989139,296242351~493439,301341353~9191315,332356818~17347412,351585344~2625399,359181831~3015778,362777592~8423425,373441908~5818494,381820951~6416356,389902499~13078402,404169452~1052236,406364931~1096263,408916890~4889846] 2021-07-31T02:29:05.279 INFO:tasks.workunit.client.0.smithi045.stdout:from 10 to 18 diff [1823868~18337958,24372582~14188618,53817318~2949139,59300692~1156219,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,137433861~4421821,143078160~11521808,154729104~8673241,167792807~9896158,181972347~5415105,202180755~12257567,225052374~5309220,242763340~3666547,248682238~10828300,264698001~7989139,296242351~493439,301341353~9191315,334588993~15115237,351585344~6792083,359181831~12019186,373441908~5818494,381820951~21159950,404169452~1052236,406364931~1096263,408916890~4889846] 2021-07-31T02:29:05.280 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [1823868~18337958,24372582~14188618,53817318~2949139,59300692~1156219,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,137433861~4421821,143078160~11521808,154729104~8673241,167792807~9896158,181972347~5415105,202180755~12257567,225052374~5309220,242763340~3666547,248682238~10828300,264698001~7989139,296242351~493439,301341353~9191315,334588993~15115237,351585344~6792083,359181831~12019186,373441908~5818494,381820951~21159950,404169452~1052236,406364931~1096263,408916890~4889846] 2021-07-31T02:29:05.287 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [1823868~18337958,24372582~14188618,53817318~2949139,59300692~1156219,66056524~7829858,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,137433861~4421821,143078160~11521808,154729104~8673241,167792807~9896158,181120288~6267164,202180755~12257567,225052374~5309220,242763340~3666547,248682238~10828300,264698001~7989139,296242351~493439,301341353~9191315,332356818~17347412,351585344~6792083,359181831~12019186,373441908~5818494,381820951~21159950,404169452~1052236,406364931~1096263,408916890~4889846] 2021-07-31T02:29:05.288 INFO:tasks.workunit.client.0.smithi045.stdout:from 10 to 19 diff [1823868~19784189,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143078160~11521808,154729104~8673241,167792807~9896158,181972347~5415105,202180755~18511385,221470799~1383607,225052374~5309220,242763340~3666547,248682238~10828300,264698001~7989139,296242351~493439,301341353~9191315,334588993~15115237,351585344~6792083,359181831~12019186,373441908~5818494,381820951~21159950,404169452~1052236,406364931~1096263,408916890~4889846] 2021-07-31T02:29:05.288 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [1823868~19784189,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143078160~11521808,154729104~8673241,167792807~9896158,181972347~5415105,202180755~18511385,221470799~1383607,225052374~5309220,242763340~3666547,248682238~10828300,264698001~7989139,296242351~493439,301341353~9191315,334588993~15115237,351585344~6792083,359181831~12019186,373441908~5818494,381820951~21159950,404169452~1052236,406364931~1096263,408916890~4889846] 2021-07-31T02:29:05.290 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [1823868~19784189,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143078160~11521808,154729104~8673241,167792807~9896158,181120288~6267164,202180755~18511385,221470799~1383607,225052374~5309220,242763340~3666547,248682238~10828300,264698001~7989139,296242351~493439,301341353~9191315,332356818~17347412,351585344~6792083,359181831~12019186,373441908~5818494,381820951~21159950,404169452~1052236,406364931~1096263,408916890~4889846] 2021-07-31T02:29:05.291 INFO:tasks.workunit.client.0.smithi045.stdout:from 11 to 12 diff [10475392~4329833,66056524~7829858,143078160~7988495,168673724~4017167,181972347~5415105,296242351~493439,334588993~5080550,343702186~3839518,373441908~5818494,404169452~1052236] 2021-07-31T02:29:05.291 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [10475392~4329833,66056524~7829858,143078160~7988495,168673724~4017167,181972347~5415105,296242351~493439,334588993~5080550,343702186~3839518,373441908~5818494,404169452~1052236] 2021-07-31T02:29:05.303 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [10475392~4329833,66056524~7829858,143078160~7988495,168673724~4017167,181120288~6267164,296242351~493439,332356818~7312725,343702186~3839518,373441908~5818494,404169452~1052236] 2021-07-31T02:29:05.304 INFO:tasks.workunit.client.0.smithi045.stdout:from 11 to 13 diff [10475392~4329833,27353411~2246694,66056524~7829858,89752687~3175987,126020102~8880466,143078160~7988495,168673724~4017167,174713264~2975701,181972347~5415105,202180755~9035885,253324513~6186025,296242351~493439,334588993~5080550,343702186~3839518,362777592~2806618,373441908~5818494,390462350~8146496,404169452~1052236] 2021-07-31T02:29:05.304 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [10475392~4329833,27353411~2246694,66056524~7829858,89752687~3175987,126020102~8880466,143078160~7988495,168673724~4017167,174713264~2975701,181972347~5415105,202180755~9035885,253324513~6186025,296242351~493439,334588993~5080550,343702186~3839518,362777592~2806618,373441908~5818494,390462350~8146496,404169452~1052236] 2021-07-31T02:29:05.305 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [10475392~4329833,27353411~2246694,66056524~7829858,89752687~3175987,126020102~8880466,143078160~7988495,168673724~4017167,174713264~2975701,181120288~6267164,202180755~9035885,253324513~6186025,296242351~493439,332356818~7312725,343702186~3839518,362777592~2806618,373441908~5818494,390462350~8146496,404169452~1052236] 2021-07-31T02:29:05.305 INFO:tasks.workunit.client.0.smithi045.stdout:from 11 to 14 diff [10475392~4329833,25410155~4189950,59300692~1156219,66056524~7829858,89752687~3175987,126020102~8880466,143078160~7988495,154729104~3694539,168673724~4017167,174713264~2975701,181972347~5415105,202180755~11760072,253324513~6186025,264698001~5703680,296242351~493439,301341353~4978213,334588993~5080550,343702186~3839518,362777592~2806618,373441908~5818494,390462350~8146496,404169452~1052236] 2021-07-31T02:29:05.305 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [10475392~4329833,25410155~4189950,59300692~1156219,66056524~7829858,89752687~3175987,126020102~8880466,143078160~7988495,154729104~3694539,168673724~4017167,174713264~2975701,181972347~5415105,202180755~11760072,253324513~6186025,264698001~5703680,296242351~493439,301341353~4978213,334588993~5080550,343702186~3839518,362777592~2806618,373441908~5818494,390462350~8146496,404169452~1052236] 2021-07-31T02:29:05.307 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [10475392~4329833,25410155~4189950,59300692~1156219,66056524~7829858,89752687~3175987,126020102~8880466,143078160~7988495,154729104~3694539,168673724~4017167,174713264~2975701,181120288~6267164,202180755~11760072,253324513~6186025,264698001~5703680,296242351~493439,301341353~4978213,332356818~7312725,343702186~3839518,362777592~2806618,373441908~5818494,390462350~8146496,404169452~1052236] 2021-07-31T02:29:05.308 INFO:tasks.workunit.client.0.smithi045.stdout:from 11 to 15 diff [1823868~7597255,10475392~4329833,25410155~4189950,59300692~1156219,66056524~7829858,89752687~3175987,126020102~8880466,143078160~11521808,154729104~3694539,168673724~4017167,174713264~2975701,181972347~5415105,202180755~11760072,253324513~6186025,264698001~7989139,296242351~493439,301341353~4978213,307774986~2757682,334588993~15115237,359181831~3015778,362777592~2806618,373441908~5818494,390462350~8146496,404169452~1052236,406364931~1096263] 2021-07-31T02:29:05.308 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [1823868~7597255,10475392~4329833,25410155~4189950,59300692~1156219,66056524~7829858,89752687~3175987,126020102~8880466,143078160~11521808,154729104~3694539,168673724~4017167,174713264~2975701,181972347~5415105,202180755~11760072,253324513~6186025,264698001~7989139,296242351~493439,301341353~4978213,307774986~2757682,334588993~15115237,359181831~3015778,362777592~2806618,373441908~5818494,390462350~8146496,404169452~1052236,406364931~1096263] 2021-07-31T02:29:05.313 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [1823868~7597255,10475392~4329833,25410155~4189950,59300692~1156219,66056524~7829858,89752687~3175987,126020102~8880466,143078160~11521808,154729104~3694539,168673724~4017167,174713264~2975701,181120288~6267164,202180755~11760072,253324513~6186025,264698001~7989139,296242351~493439,301341353~4978213,307774986~2757682,332356818~17347412,359181831~3015778,362777592~2806618,373441908~5818494,390462350~8146496,404169452~1052236,406364931~1096263] 2021-07-31T02:29:05.314 INFO:tasks.workunit.client.0.smithi045.stdout:from 11 to 16 diff [1823868~7597255,10475392~4329833,24372582~11685962,53817318~2949139,59300692~1156219,66056524~7829858,86327888~2219507,89752687~3175987,121407098~3198227,126020102~8880466,143078160~11521808,154729104~3694539,168673724~4017167,174713264~2975701,181972347~5415105,202180755~11760072,225052374~5309220,253324513~6186025,264698001~7989139,296242351~493439,301341353~4978213,307774986~2757682,334588993~15115237,359181831~3015778,362777592~2806618,373441908~5818494,390462350~12518551,404169452~1052236,406364931~1096263] 2021-07-31T02:29:05.314 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [1823868~7597255,10475392~4329833,24372582~11685962,53817318~2949139,59300692~1156219,66056524~7829858,86327888~2219507,89752687~3175987,121407098~3198227,126020102~8880466,143078160~11521808,154729104~3694539,168673724~4017167,174713264~2975701,181972347~5415105,202180755~11760072,225052374~5309220,253324513~6186025,264698001~7989139,296242351~493439,301341353~4978213,307774986~2757682,334588993~15115237,359181831~3015778,362777592~2806618,373441908~5818494,390462350~12518551,404169452~1052236,406364931~1096263] 2021-07-31T02:29:05.319 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [1823868~7597255,10475392~4329833,24372582~11685962,53817318~2949139,59300692~1156219,66056524~7829858,83886080~4661315,89752687~3175987,121407098~3198227,126020102~8880466,143078160~11521808,154729104~3694539,168673724~4017167,174713264~2975701,181120288~6267164,202180755~11760072,225052374~5309220,253324513~6186025,264698001~7989139,296242351~493439,301341353~4978213,307774986~2757682,332356818~17347412,359181831~3015778,362777592~2806618,373441908~5818494,390462350~12518551,404169452~1052236,406364931~1096263] 2021-07-31T02:29:05.319 INFO:tasks.workunit.client.0.smithi045.stdout:from 11 to 17 diff [1823868~7597255,9457761~5347464,24372582~11685962,53817318~2949139,59300692~1156219,66056524~7829858,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,137433861~4421821,143078160~11521808,154729104~8673241,168673724~4017167,174713264~2975701,181972347~5415105,202180755~11760072,225052374~5309220,253324513~6186025,264698001~7989139,296242351~493439,301341353~4978213,307774986~2757682,334588993~15115237,351585344~2625399,359181831~3015778,362777592~2806618,373441908~5818494,390462350~12518551,404169452~1052236,406364931~1096263] 2021-07-31T02:29:05.319 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [1823868~7597255,9457761~5347464,24372582~11685962,53817318~2949139,59300692~1156219,66056524~7829858,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,137433861~4421821,143078160~11521808,154729104~8673241,168673724~4017167,174713264~2975701,181972347~5415105,202180755~11760072,225052374~5309220,253324513~6186025,264698001~7989139,296242351~493439,301341353~4978213,307774986~2757682,334588993~15115237,351585344~2625399,359181831~3015778,362777592~2806618,373441908~5818494,390462350~12518551,404169452~1052236,406364931~1096263] 2021-07-31T02:29:05.324 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [1823868~7597255,9457761~5347464,24372582~11685962,53817318~2949139,59300692~1156219,66056524~7829858,83886080~4661315,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,137433861~4421821,143078160~11521808,154729104~8673241,168673724~4017167,174713264~2975701,181120288~6267164,202180755~11760072,225052374~5309220,253324513~6186025,264698001~7989139,296242351~493439,301341353~4978213,307774986~2757682,332356818~17347412,351585344~2625399,359181831~3015778,362777592~2806618,373441908~5818494,390462350~12518551,404169452~1052236,406364931~1096263] 2021-07-31T02:29:05.325 INFO:tasks.workunit.client.0.smithi045.stdout:from 11 to 18 diff [1823868~12981357,24372582~14188618,53817318~2949139,59300692~1156219,66056524~7829858,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,137433861~4421821,143078160~11521808,154729104~8673241,168673724~4017167,174713264~2975701,181972347~5415105,202180755~11760072,225052374~5309220,248682238~10828300,264698001~7989139,296242351~493439,301341353~4978213,307774986~2757682,334588993~15115237,351585344~6792083,359181831~8323415,373441908~5818494,383830156~6295455,390462350~12518551,404169452~1052236,406364931~1096263] 2021-07-31T02:29:05.325 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [1823868~12981357,24372582~14188618,53817318~2949139,59300692~1156219,66056524~7829858,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,137433861~4421821,143078160~11521808,154729104~8673241,168673724~4017167,174713264~2975701,181972347~5415105,202180755~11760072,225052374~5309220,248682238~10828300,264698001~7989139,296242351~493439,301341353~4978213,307774986~2757682,334588993~15115237,351585344~6792083,359181831~8323415,373441908~5818494,383830156~6295455,390462350~12518551,404169452~1052236,406364931~1096263] 2021-07-31T02:29:05.330 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [1823868~12981357,24372582~14188618,53817318~2949139,59300692~1156219,66056524~7829858,83886080~4661315,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,137433861~4421821,143078160~11521808,154729104~8673241,168673724~4017167,174713264~2975701,181120288~6267164,202180755~11760072,225052374~5309220,248682238~10828300,264698001~7989139,296242351~493439,301341353~4978213,307774986~2757682,332356818~17347412,351585344~6792083,359181831~8323415,373441908~5818494,383830156~6295455,390462350~12518551,404169452~1052236,406364931~1096263] 2021-07-31T02:29:05.331 INFO:tasks.workunit.client.0.smithi045.stdout:from 11 to 19 diff [1823868~12981357,17882092~3725965,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143078160~11521808,154729104~8673241,168673724~9015241,181972347~5415105,202180755~18511385,221470799~1383607,225052374~5309220,248682238~10828300,264698001~7989139,296242351~493439,301341353~9191315,334588993~15115237,351585344~6792083,359181831~8323415,373441908~5818494,383830156~6295455,390462350~12518551,404169452~1052236,406364931~1096263] 2021-07-31T02:29:05.331 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [1823868~12981357,17882092~3725965,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143078160~11521808,154729104~8673241,168673724~9015241,181972347~5415105,202180755~18511385,221470799~1383607,225052374~5309220,248682238~10828300,264698001~7989139,296242351~493439,301341353~9191315,334588993~15115237,351585344~6792083,359181831~8323415,373441908~5818494,383830156~6295455,390462350~12518551,404169452~1052236,406364931~1096263] 2021-07-31T02:29:05.337 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [1823868~12981357,17882092~3725965,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,83886080~4661315,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143078160~11521808,154729104~8673241,168673724~9015241,181120288~6267164,202180755~18511385,221470799~1383607,225052374~5309220,248682238~10828300,264698001~7989139,296242351~493439,301341353~9191315,332356818~17347412,351585344~6792083,359181831~8323415,373441908~5818494,383830156~6295455,390462350~12518551,404169452~1052236,406364931~1096263] 2021-07-31T02:29:05.337 INFO:tasks.workunit.client.0.smithi045.stdout:from 12 to 13 diff [27353411~2246694,89752687~3175987,126020102~8880466,174713264~2975701,202180755~9035885,253324513~6186025,362777592~2806618,390462350~8146496] 2021-07-31T02:29:05.337 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [27353411~2246694,89752687~3175987,126020102~8880466,174713264~2975701,202180755~9035885,253324513~6186025,362777592~2806618,390462350~8146496] 2021-07-31T02:29:05.342 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [27353411~2246694,89752687~3175987,126020102~8880466,174713264~2975701,202180755~9035885,253324513~6186025,362777592~2806618,390462350~8146496] 2021-07-31T02:29:05.342 INFO:tasks.workunit.client.0.smithi045.stdout:from 12 to 14 diff [25410155~4189950,59300692~1156219,89752687~3175987,126020102~8880466,143369911~6779467,154729104~3694539,174713264~2975701,202180755~11760072,253324513~6186025,264698001~5703680,301341353~4978213,344585688~1191257,362777592~2806618,390462350~8146496] 2021-07-31T02:29:05.342 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [25410155~4189950,59300692~1156219,89752687~3175987,126020102~8880466,143369911~6779467,154729104~3694539,174713264~2975701,202180755~11760072,253324513~6186025,264698001~5703680,301341353~4978213,344585688~1191257,362777592~2806618,390462350~8146496] 2021-07-31T02:29:05.347 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [25410155~4189950,59300692~1156219,89752687~3175987,126020102~8880466,143369911~6779467,154729104~3694539,174713264~2975701,202180755~11760072,253324513~6186025,264698001~5703680,301341353~4978213,344585688~1191257,362777592~2806618,390462350~8146496] 2021-07-31T02:29:05.348 INFO:tasks.workunit.client.0.smithi045.stdout:from 12 to 15 diff [1823868~7597255,25410155~4189950,59300692~1156219,89752687~3175987,126020102~8880466,143369911~11230057,154729104~3694539,174713264~2975701,202180755~11760072,253324513~6186025,264698001~7989139,301341353~4978213,307774986~2757682,335687900~14016330,359181831~3015778,362777592~2806618,390462350~8146496,406364931~1096263] 2021-07-31T02:29:05.348 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [1823868~7597255,25410155~4189950,59300692~1156219,89752687~3175987,126020102~8880466,143369911~11230057,154729104~3694539,174713264~2975701,202180755~11760072,253324513~6186025,264698001~7989139,301341353~4978213,307774986~2757682,335687900~14016330,359181831~3015778,362777592~2806618,390462350~8146496,406364931~1096263] 2021-07-31T02:29:05.353 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [1823868~7597255,25410155~4189950,59300692~1156219,89752687~3175987,126020102~8880466,143369911~11230057,154729104~3694539,174713264~2975701,202180755~11760072,253324513~6186025,264698001~7989139,301341353~4978213,307774986~2757682,335687900~14016330,359181831~3015778,362777592~2806618,390462350~8146496,406364931~1096263] 2021-07-31T02:29:05.353 INFO:tasks.workunit.client.0.smithi045.stdout:from 12 to 16 diff [1823868~7597255,24372582~11685962,53817318~2949139,59300692~1156219,86327888~2219507,89752687~3175987,121407098~3198227,126020102~8880466,143369911~11230057,154729104~3694539,174713264~2975701,202180755~11760072,225052374~5309220,253324513~6186025,264698001~7989139,301341353~4978213,307774986~2757682,335687900~14016330,359181831~3015778,362777592~2806618,390462350~12518551,406364931~1096263] 2021-07-31T02:29:05.353 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [1823868~7597255,24372582~11685962,53817318~2949139,59300692~1156219,86327888~2219507,89752687~3175987,121407098~3198227,126020102~8880466,143369911~11230057,154729104~3694539,174713264~2975701,202180755~11760072,225052374~5309220,253324513~6186025,264698001~7989139,301341353~4978213,307774986~2757682,335687900~14016330,359181831~3015778,362777592~2806618,390462350~12518551,406364931~1096263] 2021-07-31T02:29:05.360 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [1823868~7597255,24372582~11685962,53817318~2949139,59300692~1156219,83886080~4661315,89752687~3175987,121407098~3198227,126020102~8880466,143369911~11230057,154729104~3694539,174713264~2975701,202180755~11760072,225052374~5309220,253324513~6186025,264698001~7989139,301341353~4978213,307774986~2757682,335687900~14016330,359181831~3015778,362777592~2806618,390462350~12518551,406364931~1096263] 2021-07-31T02:29:05.361 INFO:tasks.workunit.client.0.smithi045.stdout:from 12 to 17 diff [1823868~7597255,9457761~5303041,24372582~11685962,53817318~2949139,59300692~1156219,71563215~1562492,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,137433861~4421821,143369911~11230057,154729104~8673241,174713264~2975701,186487148~881866,202180755~11760072,225052374~5309220,253324513~6186025,264698001~7989139,301341353~4978213,307774986~2757682,335055096~14649134,351585344~2625399,359181831~3015778,362777592~2806618,375687458~1062344,390462350~12518551,406364931~1096263] 2021-07-31T02:29:05.363 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [1823868~7597255,9457761~5303041,24372582~11685962,53817318~2949139,59300692~1156219,71563215~1562492,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,137433861~4421821,143369911~11230057,154729104~8673241,174713264~2975701,186487148~881866,202180755~11760072,225052374~5309220,253324513~6186025,264698001~7989139,301341353~4978213,307774986~2757682,335055096~14649134,351585344~2625399,359181831~3015778,362777592~2806618,375687458~1062344,390462350~12518551,406364931~1096263] 2021-07-31T02:29:05.367 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [1823868~7597255,9457761~5303041,24372582~11685962,53817318~2949139,59300692~1156219,71563215~1562492,83886080~4661315,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,137433861~4421821,143369911~11230057,154729104~8673241,174713264~2975701,186487148~881866,202180755~11760072,225052374~5309220,253324513~6186025,264698001~7989139,301341353~4978213,307774986~2757682,335055096~14649134,351585344~2625399,359181831~3015778,362777592~2806618,375687458~1062344,390462350~12518551,406364931~1096263] 2021-07-31T02:29:05.367 INFO:tasks.workunit.client.0.smithi045.stdout:from 12 to 18 diff [1823868~12936934,24372582~14188618,53817318~2949139,59300692~1156219,71563215~1562492,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,137433861~4421821,143369911~11230057,154729104~8673241,174713264~2975701,186487148~881866,202180755~11760072,225052374~5309220,248682238~10828300,264698001~7989139,301341353~4978213,307774986~2757682,335055096~14649134,351585344~6792083,359181831~8323415,375687458~1062344,383830156~6295455,390462350~12518551,406364931~1096263] 2021-07-31T02:29:05.367 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [1823868~12936934,24372582~14188618,53817318~2949139,59300692~1156219,71563215~1562492,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,137433861~4421821,143369911~11230057,154729104~8673241,174713264~2975701,186487148~881866,202180755~11760072,225052374~5309220,248682238~10828300,264698001~7989139,301341353~4978213,307774986~2757682,335055096~14649134,351585344~6792083,359181831~8323415,375687458~1062344,383830156~6295455,390462350~12518551,406364931~1096263] 2021-07-31T02:29:05.372 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [1823868~12936934,24372582~14188618,53817318~2949139,59300692~1156219,71563215~1562492,83886080~4661315,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,137433861~4421821,143369911~11230057,154729104~8673241,174713264~2975701,186487148~881866,202180755~11760072,225052374~5309220,248682238~10828300,264698001~7989139,301341353~4978213,307774986~2757682,335055096~14649134,351585344~6792083,359181831~8323415,375687458~1062344,383830156~6295455,390462350~12518551,406364931~1096263] 2021-07-31T02:29:05.375 INFO:tasks.workunit.client.0.smithi045.stdout:from 12 to 19 diff [1823868~12936934,17882092~3725965,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,71563215~1562492,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143369911~11230057,154729104~8673241,169186192~8502773,186487148~881866,202180755~18511385,221470799~1383607,225052374~5309220,248682238~10828300,264698001~7989139,301341353~9191315,335055096~14649134,351585344~6792083,359181831~8323415,375687458~1062344,383830156~6295455,390462350~12518551,406364931~1096263] 2021-07-31T02:29:05.375 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [1823868~12936934,17882092~3725965,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,71563215~1562492,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143369911~11230057,154729104~8673241,169186192~8502773,186487148~881866,202180755~18511385,221470799~1383607,225052374~5309220,248682238~10828300,264698001~7989139,301341353~9191315,335055096~14649134,351585344~6792083,359181831~8323415,375687458~1062344,383830156~6295455,390462350~12518551,406364931~1096263] 2021-07-31T02:29:05.377 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [1823868~12936934,17882092~3725965,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,71563215~1562492,83886080~4661315,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143369911~11230057,154729104~8673241,169186192~8502773,186487148~881866,202180755~18511385,221470799~1383607,225052374~5309220,248682238~10828300,264698001~7989139,301341353~9191315,335055096~14649134,351585344~6792083,359181831~8323415,375687458~1062344,383830156~6295455,390462350~12518551,406364931~1096263] 2021-07-31T02:29:05.377 INFO:tasks.workunit.client.0.smithi045.stdout:from 13 to 14 diff [25410155~3524489,59300692~1156219,143369911~6779467,154729104~3694539,207917770~6023057,264698001~5703680,301341353~4978213,344585688~1191257] 2021-07-31T02:29:05.377 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [25410155~3524489,59300692~1156219,143369911~6779467,154729104~3694539,207917770~6023057,264698001~5703680,301341353~4978213,344585688~1191257] 2021-07-31T02:29:05.387 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [25410155~3524489,59300692~1156219,143369911~6779467,154729104~3694539,207917770~6023057,264698001~5703680,301341353~4978213,344585688~1191257] 2021-07-31T02:29:05.387 INFO:tasks.workunit.client.0.smithi045.stdout:from 13 to 15 diff [1823868~7597255,25410155~3524489,59300692~1156219,143369911~11230057,154729104~3694539,207917770~6023057,264698001~7989139,301341353~4978213,307774986~2757682,335687900~14016330,359181831~3015778,406364931~1096263] 2021-07-31T02:29:05.388 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [1823868~7597255,25410155~3524489,59300692~1156219,143369911~11230057,154729104~3694539,207917770~6023057,264698001~7989139,301341353~4978213,307774986~2757682,335687900~14016330,359181831~3015778,406364931~1096263] 2021-07-31T02:29:05.394 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [1823868~7597255,25410155~3524489,59300692~1156219,143369911~11230057,154729104~3694539,207917770~6023057,264698001~7989139,301341353~4978213,307774986~2757682,335687900~14016330,359181831~3015778,406364931~1096263] 2021-07-31T02:29:05.394 INFO:tasks.workunit.client.0.smithi045.stdout:from 13 to 16 diff [1823868~7597255,24372582~11685962,53817318~2949139,59300692~1156219,86327888~2219507,121407098~3198227,143369911~11230057,154729104~3694539,207917770~6023057,225052374~5309220,264698001~7989139,301341353~4978213,307774986~2757682,335687900~14016330,359181831~3015778,391467418~11513483,406364931~1096263] 2021-07-31T02:29:05.394 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [1823868~7597255,24372582~11685962,53817318~2949139,59300692~1156219,86327888~2219507,121407098~3198227,143369911~11230057,154729104~3694539,207917770~6023057,225052374~5309220,264698001~7989139,301341353~4978213,307774986~2757682,335687900~14016330,359181831~3015778,391467418~11513483,406364931~1096263] 2021-07-31T02:29:05.397 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [1823868~7597255,24372582~11685962,53817318~2949139,59300692~1156219,83886080~4661315,121407098~3198227,143369911~11230057,154729104~3694539,207917770~6023057,225052374~5309220,264698001~7989139,301341353~4978213,307774986~2757682,335687900~14016330,359181831~3015778,391467418~11513483,406364931~1096263] 2021-07-31T02:29:05.398 INFO:tasks.workunit.client.0.smithi045.stdout:from 13 to 17 diff [1823868~7597255,9457761~5303041,24372582~11685962,53817318~2949139,59300692~1156219,71563215~1562492,86327888~2219507,89929518~1882381,99285857~789121,121407098~3198227,137433861~4421821,143369911~11230057,154729104~8673241,186487148~881866,207917770~6023057,225052374~5309220,264698001~7989139,301341353~4978213,307774986~2757682,335055096~14649134,351585344~2625399,359181831~3015778,375687458~1062344,391467418~11513483,406364931~1096263] 2021-07-31T02:29:05.398 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [1823868~7597255,9457761~5303041,24372582~11685962,53817318~2949139,59300692~1156219,71563215~1562492,86327888~2219507,89929518~1882381,99285857~789121,121407098~3198227,137433861~4421821,143369911~11230057,154729104~8673241,186487148~881866,207917770~6023057,225052374~5309220,264698001~7989139,301341353~4978213,307774986~2757682,335055096~14649134,351585344~2625399,359181831~3015778,375687458~1062344,391467418~11513483,406364931~1096263] 2021-07-31T02:29:05.403 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [1823868~7597255,9457761~5303041,24372582~11685962,53817318~2949139,59300692~1156219,71563215~1562492,83886080~4661315,89929518~1882381,99285857~789121,121407098~3198227,137433861~4421821,143369911~11230057,154729104~8673241,186487148~881866,207917770~6023057,225052374~5309220,264698001~7989139,301341353~4978213,307774986~2757682,335055096~14649134,351585344~2625399,359181831~3015778,375687458~1062344,391467418~11513483,406364931~1096263] 2021-07-31T02:29:05.404 INFO:tasks.workunit.client.0.smithi045.stdout:from 13 to 18 diff [1823868~12936934,24372582~14188618,53817318~2949139,59300692~1156219,71563215~1562492,86327888~2219507,89929518~1882381,99285857~789121,121407098~3198227,137433861~4421821,143369911~11230057,154729104~8673241,186487148~881866,207917770~6023057,225052374~5309220,248682238~8485363,264698001~7989139,301341353~4978213,307774986~2757682,335055096~14649134,351585344~6792083,359181831~8323415,375687458~1062344,383830156~6295455,391467418~11513483,406364931~1096263] 2021-07-31T02:29:05.404 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [1823868~12936934,24372582~14188618,53817318~2949139,59300692~1156219,71563215~1562492,86327888~2219507,89929518~1882381,99285857~789121,121407098~3198227,137433861~4421821,143369911~11230057,154729104~8673241,186487148~881866,207917770~6023057,225052374~5309220,248682238~8485363,264698001~7989139,301341353~4978213,307774986~2757682,335055096~14649134,351585344~6792083,359181831~8323415,375687458~1062344,383830156~6295455,391467418~11513483,406364931~1096263] 2021-07-31T02:29:05.409 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [1823868~12936934,24372582~14188618,53817318~2949139,59300692~1156219,71563215~1562492,83886080~4661315,89929518~1882381,99285857~789121,121407098~3198227,137433861~4421821,143369911~11230057,154729104~8673241,186487148~881866,207917770~6023057,225052374~5309220,248682238~8485363,264698001~7989139,301341353~4978213,307774986~2757682,335055096~14649134,351585344~6792083,359181831~8323415,375687458~1062344,383830156~6295455,391467418~11513483,406364931~1096263] 2021-07-31T02:29:05.410 INFO:tasks.workunit.client.0.smithi045.stdout:from 13 to 19 diff [1823868~12936934,17882092~3725965,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,71563215~1562492,86327888~2219507,89929518~1882381,99285857~789121,121407098~3198227,135550801~7285470,143369911~11230057,154729104~8673241,169186192~7553729,186487148~881866,207917770~12774370,221470799~1383607,225052374~5309220,248682238~8485363,264698001~7989139,301341353~9191315,335055096~14649134,351585344~6792083,359181831~8323415,375687458~1062344,383830156~6295455,391467418~11513483,406364931~1096263] 2021-07-31T02:29:05.410 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [1823868~12936934,17882092~3725965,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,71563215~1562492,86327888~2219507,89929518~1882381,99285857~789121,121407098~3198227,135550801~7285470,143369911~11230057,154729104~8673241,169186192~7553729,186487148~881866,207917770~12774370,221470799~1383607,225052374~5309220,248682238~8485363,264698001~7989139,301341353~9191315,335055096~14649134,351585344~6792083,359181831~8323415,375687458~1062344,383830156~6295455,391467418~11513483,406364931~1096263] 2021-07-31T02:29:05.415 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [1823868~12936934,17882092~3725965,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,71563215~1562492,83886080~4661315,89929518~1882381,99285857~789121,121407098~3198227,135550801~7285470,143369911~11230057,154729104~8673241,169186192~7553729,186487148~881866,207917770~12774370,221470799~1383607,225052374~5309220,248682238~8485363,264698001~7989139,301341353~9191315,335055096~14649134,351585344~6792083,359181831~8323415,375687458~1062344,383830156~6295455,391467418~11513483,406364931~1096263] 2021-07-31T02:29:05.415 INFO:tasks.workunit.client.0.smithi045.stdout:from 14 to 15 diff [1823868~7597255,147626899~6973069,265182709~7504431,307774986~2757682,335687900~14016330,359181831~3015778,406364931~1096263] 2021-07-31T02:29:05.416 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [1823868~7597255,147626899~6973069,265182709~7504431,307774986~2757682,335687900~14016330,359181831~3015778,406364931~1096263] 2021-07-31T02:29:05.421 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [1823868~7597255,147626899~6973069,265182709~7504431,307774986~2757682,335687900~14016330,359181831~3015778,406364931~1096263] 2021-07-31T02:29:05.421 INFO:tasks.workunit.client.0.smithi045.stdout:from 14 to 16 diff [1823868~7597255,24372582~11685962,53817318~2949139,86327888~2219507,121407098~3198227,147626899~6973069,207946866~2990158,225052374~5309220,265182709~7504431,307774986~2757682,335687900~14016330,359181831~3015778,391467418~11513483,406364931~1096263] 2021-07-31T02:29:05.422 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [1823868~7597255,24372582~11685962,53817318~2949139,86327888~2219507,121407098~3198227,147626899~6973069,207946866~2990158,225052374~5309220,265182709~7504431,307774986~2757682,335687900~14016330,359181831~3015778,391467418~11513483,406364931~1096263] 2021-07-31T02:29:05.429 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [1823868~7597255,24372582~11685962,53817318~2949139,83886080~4661315,121407098~3198227,147626899~6973069,207946866~2990158,225052374~5309220,265182709~7504431,307774986~2757682,335687900~14016330,359181831~3015778,391467418~11513483,406364931~1096263] 2021-07-31T02:29:05.430 INFO:tasks.workunit.client.0.smithi045.stdout:from 14 to 17 diff [1823868~7597255,9457761~5303041,24372582~11685962,53817318~2949139,71563215~1562492,86327888~2219507,89929518~1882381,99285857~789121,121407098~3198227,137433861~4421821,147626899~6973069,157325666~6076679,186487148~881866,207946866~2990158,225052374~5309220,265182709~7504431,307774986~2757682,335055096~14649134,351585344~2625399,359181831~3015778,375687458~1062344,391467418~11513483,406364931~1096263] 2021-07-31T02:29:05.430 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [1823868~7597255,9457761~5303041,24372582~11685962,53817318~2949139,71563215~1562492,86327888~2219507,89929518~1882381,99285857~789121,121407098~3198227,137433861~4421821,147626899~6973069,157325666~6076679,186487148~881866,207946866~2990158,225052374~5309220,265182709~7504431,307774986~2757682,335055096~14649134,351585344~2625399,359181831~3015778,375687458~1062344,391467418~11513483,406364931~1096263] 2021-07-31T02:29:05.433 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [1823868~7597255,9457761~5303041,24372582~11685962,53817318~2949139,71563215~1562492,83886080~4661315,89929518~1882381,99285857~789121,121407098~3198227,137433861~4421821,147626899~6973069,157325666~6076679,186487148~881866,207946866~2990158,225052374~5309220,265182709~7504431,307774986~2757682,335055096~14649134,351585344~2625399,359181831~3015778,375687458~1062344,391467418~11513483,406364931~1096263] 2021-07-31T02:29:05.433 INFO:tasks.workunit.client.0.smithi045.stdout:from 14 to 18 diff [1823868~12936934,24372582~14188618,53817318~2949139,71563215~1562492,86327888~2219507,89929518~1882381,99285857~789121,121407098~3198227,137433861~4421821,147626899~6973069,157325666~6076679,186487148~881866,207946866~2990158,225052374~5309220,248682238~8485363,264809820~7877320,307774986~2757682,335055096~14649134,351585344~6792083,359181831~8323415,375687458~1062344,383830156~6295455,391467418~11513483,406364931~1096263] 2021-07-31T02:29:05.433 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [1823868~12936934,24372582~14188618,53817318~2949139,71563215~1562492,86327888~2219507,89929518~1882381,99285857~789121,121407098~3198227,137433861~4421821,147626899~6973069,157325666~6076679,186487148~881866,207946866~2990158,225052374~5309220,248682238~8485363,264809820~7877320,307774986~2757682,335055096~14649134,351585344~6792083,359181831~8323415,375687458~1062344,383830156~6295455,391467418~11513483,406364931~1096263] 2021-07-31T02:29:05.439 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [1823868~12936934,24372582~14188618,53817318~2949139,71563215~1562492,83886080~4661315,89929518~1882381,99285857~789121,121407098~3198227,137433861~4421821,147626899~6973069,157325666~6076679,186487148~881866,207946866~2990158,225052374~5309220,248682238~8485363,264809820~7877320,307774986~2757682,335055096~14649134,351585344~6792083,359181831~8323415,375687458~1062344,383830156~6295455,391467418~11513483,406364931~1096263] 2021-07-31T02:29:05.439 INFO:tasks.workunit.client.0.smithi045.stdout:from 14 to 19 diff [1823868~12936934,17882092~3725965,24372582~14188618,53817318~2949139,60609981~5053700,71563215~1562492,86327888~2219507,89929518~1882381,99285857~789121,121407098~3198227,135550801~7285470,147626899~6973069,157325666~6076679,169186192~7553729,186487148~881866,207946866~2990158,213575464~7116676,221470799~1383607,225052374~5309220,248682238~8485363,264809820~7877320,305074438~5458230,335055096~14649134,351585344~6792083,359181831~8323415,375687458~1062344,383830156~6295455,391467418~11513483,406364931~1096263] 2021-07-31T02:29:05.439 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [1823868~12936934,17882092~3725965,24372582~14188618,53817318~2949139,60609981~5053700,71563215~1562492,86327888~2219507,89929518~1882381,99285857~789121,121407098~3198227,135550801~7285470,147626899~6973069,157325666~6076679,169186192~7553729,186487148~881866,207946866~2990158,213575464~7116676,221470799~1383607,225052374~5309220,248682238~8485363,264809820~7877320,305074438~5458230,335055096~14649134,351585344~6792083,359181831~8323415,375687458~1062344,383830156~6295455,391467418~11513483,406364931~1096263] 2021-07-31T02:29:05.445 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [1823868~12936934,17882092~3725965,24372582~14188618,53817318~2949139,60609981~5053700,71563215~1562492,83886080~4661315,89929518~1882381,99285857~789121,121407098~3198227,135550801~7285470,147626899~6973069,157325666~6076679,169186192~7553729,186487148~881866,207946866~2990158,213575464~7116676,221470799~1383607,225052374~5309220,248682238~8485363,264809820~7877320,305074438~5458230,335055096~14649134,351585344~6792083,359181831~8323415,375687458~1062344,383830156~6295455,391467418~11513483,406364931~1096263] 2021-07-31T02:29:05.445 INFO:tasks.workunit.client.0.smithi045.stdout:from 15 to 16 diff [24372582~11685962,53817318~2949139,86327888~2219507,121407098~3198227,207946866~2990158,225052374~5309220,391467418~11513483] 2021-07-31T02:29:05.445 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [24372582~11685962,53817318~2949139,86327888~2219507,121407098~3198227,207946866~2990158,225052374~5309220,391467418~11513483] 2021-07-31T02:29:05.450 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [24372582~11685962,53817318~2949139,83886080~4661315,121407098~3198227,207946866~2990158,225052374~5309220,391467418~11513483] 2021-07-31T02:29:05.451 INFO:tasks.workunit.client.0.smithi045.stdout:from 15 to 17 diff [9457761~5303041,24372582~11685962,53817318~2949139,71563215~1562492,86327888~2219507,89929518~1882381,99285857~789121,121407098~3198227,137433861~4421821,157325666~6076679,186487148~881866,207946866~2990158,225052374~5309220,335055096~7851089,351585344~2625399,375687458~1062344,391467418~11513483] 2021-07-31T02:29:05.451 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [9457761~5303041,24372582~11685962,53817318~2949139,71563215~1562492,86327888~2219507,89929518~1882381,99285857~789121,121407098~3198227,137433861~4421821,157325666~6076679,186487148~881866,207946866~2990158,225052374~5309220,335055096~7851089,351585344~2625399,375687458~1062344,391467418~11513483] 2021-07-31T02:29:05.456 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [9457761~5303041,24372582~11685962,53817318~2949139,71563215~1562492,83886080~4661315,89929518~1882381,99285857~789121,121407098~3198227,137433861~4421821,157325666~6076679,186487148~881866,207946866~2990158,225052374~5309220,335055096~7851089,351585344~2625399,375687458~1062344,391467418~11513483] 2021-07-31T02:29:05.456 INFO:tasks.workunit.client.0.smithi045.stdout:from 15 to 18 diff [9296466~5464336,24372582~14188618,53817318~2949139,71563215~1562492,86327888~2219507,89929518~1882381,99285857~789121,121407098~3198227,137433861~4421821,157325666~6076679,186487148~881866,207946866~2990158,225052374~5309220,248682238~8485363,264809820~4477895,335055096~7851089,351585344~6792083,359544279~7960967,375687458~1062344,383830156~6295455,391467418~11513483] 2021-07-31T02:29:05.456 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [9296466~5464336,24372582~14188618,53817318~2949139,71563215~1562492,86327888~2219507,89929518~1882381,99285857~789121,121407098~3198227,137433861~4421821,157325666~6076679,186487148~881866,207946866~2990158,225052374~5309220,248682238~8485363,264809820~4477895,335055096~7851089,351585344~6792083,359544279~7960967,375687458~1062344,383830156~6295455,391467418~11513483] 2021-07-31T02:29:05.462 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [9296466~5464336,24372582~14188618,53817318~2949139,71563215~1562492,83886080~4661315,89929518~1882381,99285857~789121,121407098~3198227,137433861~4421821,157325666~6076679,186487148~881866,207946866~2990158,225052374~5309220,248682238~8485363,264809820~4477895,335055096~7851089,351585344~6792083,359544279~7960967,375687458~1062344,383830156~6295455,391467418~11513483] 2021-07-31T02:29:05.462 INFO:tasks.workunit.client.0.smithi045.stdout:from 15 to 19 diff [9296466~5464336,17882092~3725965,24372582~14188618,53817318~2949139,60609981~5053700,71563215~1562492,86327888~2219507,89929518~1882381,99285857~789121,121407098~3198227,135550801~7285470,157325666~6076679,169186192~7553729,186487148~881866,207946866~2990158,213575464~7116676,221470799~1383607,225052374~5309220,248682238~8485363,264809820~4477895,305074438~5008577,335055096~7851089,351585344~6792083,359544279~7960967,375687458~1062344,383830156~6295455,391467418~11513483] 2021-07-31T02:29:05.462 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [9296466~5464336,17882092~3725965,24372582~14188618,53817318~2949139,60609981~5053700,71563215~1562492,86327888~2219507,89929518~1882381,99285857~789121,121407098~3198227,135550801~7285470,157325666~6076679,169186192~7553729,186487148~881866,207946866~2990158,213575464~7116676,221470799~1383607,225052374~5309220,248682238~8485363,264809820~4477895,305074438~5008577,335055096~7851089,351585344~6792083,359544279~7960967,375687458~1062344,383830156~6295455,391467418~11513483] 2021-07-31T02:29:05.467 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [9296466~5464336,17882092~3725965,24372582~14188618,53817318~2949139,60609981~5053700,71563215~1562492,83886080~4661315,89929518~1882381,99285857~789121,121407098~3198227,135550801~7285470,157325666~6076679,169186192~7553729,186487148~881866,207946866~2990158,213575464~7116676,221470799~1383607,225052374~5309220,248682238~8485363,264809820~4477895,305074438~5008577,335055096~7851089,351585344~6792083,359544279~7960967,375687458~1062344,383830156~6295455,391467418~11513483] 2021-07-31T02:29:05.467 INFO:tasks.workunit.client.0.smithi045.stdout:from 16 to 17 diff [9457761~5303041,71563215~1562492,89929518~1882381,99285857~789121,137433861~4421821,157325666~6076679,186487148~881866,335055096~7851089,351585344~2625399,375687458~1062344] 2021-07-31T02:29:05.468 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [9457761~5303041,71563215~1562492,89929518~1882381,99285857~789121,137433861~4421821,157325666~6076679,186487148~881866,335055096~7851089,351585344~2625399,375687458~1062344] 2021-07-31T02:29:05.473 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [9457761~5303041,71563215~1562492,89929518~1882381,99285857~789121,137433861~4421821,157325666~6076679,186487148~881866,335055096~7851089,351585344~2625399,375687458~1062344] 2021-07-31T02:29:05.473 INFO:tasks.workunit.client.0.smithi045.stdout:from 16 to 18 diff [9296466~5464336,31521345~7039855,55515079~1195728,71563215~1562492,89929518~1882381,99285857~789121,137433861~4421821,157325666~6076679,186487148~881866,248682238~8485363,264809820~4477895,335055096~7851089,351585344~6792083,359544279~7960967,375687458~1062344,383830156~6295455] 2021-07-31T02:29:05.474 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [9296466~5464336,31521345~7039855,55515079~1195728,71563215~1562492,89929518~1882381,99285857~789121,137433861~4421821,157325666~6076679,186487148~881866,248682238~8485363,264809820~4477895,335055096~7851089,351585344~6792083,359544279~7960967,375687458~1062344,383830156~6295455] 2021-07-31T02:29:05.479 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [9296466~5464336,31521345~7039855,55515079~1195728,71563215~1562492,89929518~1882381,99285857~789121,137433861~4421821,157325666~6076679,186487148~881866,248682238~8485363,264809820~4477895,335055096~7851089,351585344~6792083,359544279~7960967,375687458~1062344,383830156~6295455] 2021-07-31T02:29:05.480 INFO:tasks.workunit.client.0.smithi045.stdout:from 16 to 19 diff [9296466~5464336,17882092~3725965,31521345~7039855,55515079~1195728,60609981~5053700,71563215~1562492,89929518~1882381,99285857~789121,135550801~7285470,157325666~6076679,169186192~7553729,186487148~881866,213575464~7116676,221470799~1383607,248682238~8485363,264809820~4477895,305074438~5008577,335055096~7851089,351585344~6792083,359544279~7960967,375687458~1062344,383830156~6295455] 2021-07-31T02:29:05.480 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [9296466~5464336,17882092~3725965,31521345~7039855,55515079~1195728,60609981~5053700,71563215~1562492,89929518~1882381,99285857~789121,135550801~7285470,157325666~6076679,169186192~7553729,186487148~881866,213575464~7116676,221470799~1383607,248682238~8485363,264809820~4477895,305074438~5008577,335055096~7851089,351585344~6792083,359544279~7960967,375687458~1062344,383830156~6295455] 2021-07-31T02:29:05.485 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [9296466~5464336,17882092~3725965,31521345~7039855,55515079~1195728,60609981~5053700,71563215~1562492,89929518~1882381,99285857~789121,135550801~7285470,157325666~6076679,169186192~7553729,186487148~881866,213575464~7116676,221470799~1383607,248682238~8485363,264809820~4477895,305074438~5008577,335055096~7851089,351585344~6792083,359544279~7960967,375687458~1062344,383830156~6295455] 2021-07-31T02:29:05.485 INFO:tasks.workunit.client.0.smithi045.stdout:from 17 to 18 diff [9296466~2171054,31521345~7039855,55515079~1195728,248682238~8485363,264809820~4477895,351737416~6640011,359544279~7960967,383830156~6295455] 2021-07-31T02:29:05.486 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [9296466~2171054,31521345~7039855,55515079~1195728,248682238~8485363,264809820~4477895,351737416~6640011,359544279~7960967,383830156~6295455] 2021-07-31T02:29:05.491 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [9296466~2171054,31521345~7039855,55515079~1195728,248682238~8485363,264809820~4477895,351737416~6640011,359544279~7960967,383830156~6295455] 2021-07-31T02:29:05.492 INFO:tasks.workunit.client.0.smithi045.stdout:from 17 to 19 diff [9296466~2171054,17882092~3725965,31521345~7039855,55515079~1195728,60609981~5053700,135550801~7285470,169186192~7553729,213575464~7116676,221470799~1383607,248682238~8485363,264809820~4477895,305074438~5008577,351737416~6640011,359544279~7960967,383830156~6295455] 2021-07-31T02:29:05.492 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [9296466~2171054,17882092~3725965,31521345~7039855,55515079~1195728,60609981~5053700,135550801~7285470,169186192~7553729,213575464~7116676,221470799~1383607,248682238~8485363,264809820~4477895,305074438~5008577,351737416~6640011,359544279~7960967,383830156~6295455] 2021-07-31T02:29:05.497 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [9296466~2171054,17882092~3725965,31521345~7039855,55515079~1195728,60609981~5053700,135550801~7285470,169186192~7553729,213575464~7116676,221470799~1383607,248682238~8485363,264809820~4477895,305074438~5008577,351737416~6640011,359544279~7960967,383830156~6295455] 2021-07-31T02:29:05.497 INFO:tasks.workunit.client.0.smithi045.stdout:from 18 to 19 diff [17882092~3725965,32852874~482850,60609981~5053700,135550801~7285470,169186192~7553729,213575464~7116676,221470799~1383607,305074438~5008577,360539103~2306471] 2021-07-31T02:29:05.498 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [17882092~3725965,32852874~482850,60609981~5053700,135550801~7285470,169186192~7553729,213575464~7116676,221470799~1383607,305074438~5008577,360539103~2306471] 2021-07-31T02:29:05.502 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [17882092~3725965,32852874~482850,60609981~5053700,135550801~7285470,169186192~7553729,213575464~7116676,221470799~1383607,305074438~5008577,360539103~2306471] 2021-07-31T02:29:05.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:05 smithi154 conmon[34553]: cluster 2021-07-31T02:29:04.378946+0000 2021-07-31T02:29:05.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:05 smithi154 conmon[34553]: mon.a (mon.0) 2090 : cluster [DBG] osdmap e1165: 8 total, 8 up, 8 in 2021-07-31T02:29:05.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:05 smithi154 conmon[34553]: cluster 2021-07-31T02:29:04.397977+0000 mgr.x (mgr.24542) 1792 : cluster [DBG] pgmap v2267: 41 pgs: 41 active+clean; 885 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 7.5 KiB/s rd, 51 MiB/s wr, 35 op/s 2021-07-31T02:29:05.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:05 smithi045 conmon[33655]: cluster 2021-07-31T02:29:04.378946+0000 mon.a (mon.0) 2090 : cluster [DBG] osdmap e1165: 8 total, 8 up, 8 in 2021-07-31T02:29:05.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:05 smithi045 conmon[33655]: cluster 2021-07-31T02:29:04.397977+0000 mgr.x (mgr.24542) 1792 : cluster [DBG] 2021-07-31T02:29:05.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:05 smithi045 conmon[33655]: pgmap v2267: 41 pgs: 41 active+clean; 885 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 7.5 KiB/s rd, 51 MiB/s wr, 35 op/s 2021-07-31T02:29:05.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:05 smithi045 conmon[39718]: cluster 2021-07-31T02:29:04.378946 2021-07-31T02:29:05.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:05 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2090 : cluster [DBG] osdmap e1165: 8 total, 8 up, 8 in 2021-07-31T02:29:05.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:05 smithi045 conmon[39718]: cluster 2021- 2021-07-31T02:29:05.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:05 smithi045 conmon[39718]: 07-31T02:29:04.397977+0000 mgr.x (mgr.24542) 1792 2021-07-31T02:29:05.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:05 smithi045 conmon[39718]: : cluster [DBG] pgmap v2267: 41 pgs: 41 active+clean; 885 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 7.5 KiB/s rd, 51 MiB/s wr, 35 op/s 2021-07-31T02:29:06.393 INFO:tasks.workunit.client.0.smithi045.stdout:from 0 to HEAD diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~22526082,86327888~2219507,89659479~3999082,97713673~8454237,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,236085312~4054340,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:06.393 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~22526082,86327888~2219507,89659479~3999082,97713673~8454237,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,236085312~4054340,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:06.410 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~43334606,46137344~10629113,59300692~1156219,60609981~22526082,83886080~9772481,96468992~14988213,116424046~70963406,188743680~9773402,201326592~33497107,234881024~24629514,260046848~20273171,289406976~21125692,314572800~57820771,373441908~5818494,381681664~32125072] 2021-07-31T02:29:06.411 INFO:tasks.workunit.client.0.smithi045.stdout:from 1 to HEAD diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89659479~3999082,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:06.412 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89659479~3999082,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:06.416 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~43334606,46137344~10629113,59300692~1156219,60609981~5053700,66056524~17079539,83886080~9772481,99285857~789121,102437886~9019319,116424046~70963406,191717187~6799895,201326592~33497107,240139652~19370886,260046848~20273171,289406976~21125692,314572800~57820771,373441908~5818494,381681664~32125072] 2021-07-31T02:29:06.416 INFO:tasks.workunit.client.0.smithi045.stdout:from 2 to HEAD diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89659479~3999082,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:06.417 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89659479~3999082,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:06.423 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~43334606,46137344~10629113,59300692~1156219,60609981~5053700,66056524~17079539,83886080~9772481,99285857~789121,102437886~9019319,116424046~70963406,191717187~6799895,201478817~33344882,240139652~19370886,260046848~20273171,291252461~19280207,314572800~12707070,327653500~3214988,331434830~40958741,373441908~5818494,381681664~32125072] 2021-07-31T02:29:06.424 INFO:tasks.workunit.client.0.smithi045.stdout:from 3 to HEAD diff [677713~20930344,24372582~18962024,48159474~2619597,53817318~2949139,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:06.424 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~20930344,24372582~18962024,48159474~2619597,53817318~2949139,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:06.430 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~21608057,24372582~18962024,48159474~2619597,53817318~2949139,59300692~1156219,60609981~5053700,66056524~17079539,83886080~4661315,89752687~3175987,99285857~789121,102437886~9019319,116424046~70963406,201478817~21375589,223825249~10998450,240139652~19370886,260046848~20273171,291252461~19280207,314572800~12707070,327653500~3214988,331434830~40958741,373441908~5818494,381681664~32125072] 2021-07-31T02:29:06.431 INFO:tasks.workunit.client.0.smithi045.stdout:from 4 to HEAD diff [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~20324185,167531438~13588850,181972347~5415105,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,292264750~18267918,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:06.431 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~20324185,167531438~13588850,181972347~5415105,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,292264750~18267918,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:06.436 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,102437886~9019319,116424046~8181279,124734943~38667402,164857432~22530020,201478817~21375589,223825249~10998450,240139652~19370886,260046848~12640292,275231912~5088107,292264750~18267918,314572800~12707070,327653500~3214988,331434830~40958741,373441908~5818494,381681664~32125072] 2021-07-31T02:29:06.437 INFO:tasks.workunit.client.0.smithi045.stdout:from 5 to HEAD diff [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,292264750~4687170,301341353~9191315,318433435~4174606,322615554~4664316,331638058~718760,334588993~36612024,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:06.437 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,292264750~4687170,301341353~9191315,318433435~4174606,322615554~4664316,331638058~718760,334588993~36612024,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:06.442 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,102437886~9019319,116424046~8181279,124734943~38667402,164857432~12831533,181120288~6267164,201478817~19213323,221470799~1383607,223825249~10998450,241317308~18193230,260046848~12640292,275231912~5088107,292264750~4687170,301341353~9191315,314572800~12707070,331434830~39766187,373441908~5818494,381681664~32125072] 2021-07-31T02:29:06.443 INFO:tasks.workunit.client.0.smithi045.stdout:from 6 to HEAD diff [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,121407098~3198227,124734943~18101328,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,223825249~6536345,241317308~18193230,260438339~12248801,296242351~493439,301341353~9191315,318433435~4174606,322615554~4664316,331638058~718760,334588993~36612024,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:06.443 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,121407098~3198227,124734943~18101328,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,223825249~6536345,241317308~18193230,260438339~12248801,296242351~493439,301341353~9191315,318433435~4174606,322615554~4664316,331638058~718760,334588993~36612024,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:06.451 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,102437886~3730024,121407098~3198227,124734943~18101328,143078160~20324185,164857432~12831533,181120288~6267164,201478817~19213323,221470799~1383607,223825249~6536345,241317308~18193230,260046848~12640292,296242351~493439,301341353~9191315,314572800~12707070,331434830~39766187,373441908~5818494,381681664~32125072] 2021-07-31T02:29:06.451 INFO:tasks.workunit.client.0.smithi045.stdout:from 7 to HEAD diff [872289~20735768,24372582~17941837,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,121407098~3198227,124734943~18101328,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,225052374~5309220,241317308~18193230,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,318433435~4174606,322615554~4664316,334588993~36612024,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:06.452 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~20735768,24372582~17941837,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,121407098~3198227,124734943~18101328,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,225052374~5309220,241317308~18193230,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,318433435~4174606,322615554~4664316,334588993~36612024,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:06.458 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~20735768,24372582~17941837,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,102437886~3730024,121407098~3198227,124734943~18101328,143078160~20324185,164857432~12831533,181120288~6267164,201478817~19213323,221470799~1383607,225052374~5309220,241317308~18193230,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,314572800~12707070,332356818~38844199,373441908~5818494,381681664~32125072] 2021-07-31T02:29:06.458 INFO:tasks.workunit.client.0.smithi045.stdout:from 8 to HEAD diff [872289~20735768,24372582~17941837,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~16816169,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,225052374~5309220,242763340~3666547,248659714~10850824,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,334588993~23788434,359181831~12019186,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:06.458 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~20735768,24372582~17941837,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~16816169,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,225052374~5309220,242763340~3666547,248659714~10850824,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,334588993~23788434,359181831~12019186,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:06.464 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~20735768,24372582~17941837,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,121407098~3198227,126020102~16816169,143078160~20324185,164857432~12831533,181120288~6267164,201478817~19213323,221470799~1383607,225052374~5309220,242763340~3666547,248659714~10850824,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,332356818~26020609,359181831~12019186,373441908~5818494,381820951~31985785] 2021-07-31T02:29:06.464 INFO:tasks.workunit.client.0.smithi045.stdout:from 9 to HEAD diff [1823868~19784189,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143078160~20324185,167792807~9896158,181972347~5415105,202180755~18511385,221470799~1383607,225052374~5309220,242763340~3666547,248659714~10850824,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,334588993~15115237,351585344~6792083,359181831~12019186,373441908~5818494,381820951~21159950,404169452~1052236,406364931~1096263,408916890~4889846] 2021-07-31T02:29:06.464 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [1823868~19784189,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143078160~20324185,167792807~9896158,181972347~5415105,202180755~18511385,221470799~1383607,225052374~5309220,242763340~3666547,248659714~10850824,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,334588993~15115237,351585344~6792083,359181831~12019186,373441908~5818494,381820951~21159950,404169452~1052236,406364931~1096263,408916890~4889846] 2021-07-31T02:29:06.470 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [1823868~19784189,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143078160~20324185,167792807~9896158,181120288~6267164,202180755~18511385,221470799~1383607,225052374~5309220,242763340~3666547,248659714~10850824,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,332356818~17347412,351585344~6792083,359181831~12019186,373441908~5818494,381820951~21159950,404169452~1052236,406364931~1096263,408916890~4889846] 2021-07-31T02:29:06.470 INFO:tasks.workunit.client.0.smithi045.stdout:from 10 to HEAD diff [1823868~19784189,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143078160~11521808,154729104~8673241,167792807~9896158,181972347~5415105,202180755~18511385,221470799~1383607,225052374~5309220,242763340~3666547,248682238~10828300,264698001~7989139,296242351~493439,301341353~9191315,334588993~15115237,351585344~6792083,359181831~12019186,373441908~5818494,381820951~21159950,404169452~1052236,406364931~1096263,408916890~4889846] 2021-07-31T02:29:06.471 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [1823868~19784189,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143078160~11521808,154729104~8673241,167792807~9896158,181972347~5415105,202180755~18511385,221470799~1383607,225052374~5309220,242763340~3666547,248682238~10828300,264698001~7989139,296242351~493439,301341353~9191315,334588993~15115237,351585344~6792083,359181831~12019186,373441908~5818494,381820951~21159950,404169452~1052236,406364931~1096263,408916890~4889846] 2021-07-31T02:29:06.476 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [1823868~19784189,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143078160~11521808,154729104~8673241,167792807~9896158,181120288~6267164,202180755~18511385,221470799~1383607,225052374~5309220,242763340~3666547,248682238~10828300,264698001~7989139,296242351~493439,301341353~9191315,332356818~17347412,351585344~6792083,359181831~12019186,373441908~5818494,381820951~21159950,404169452~1052236,406364931~1096263,408916890~4889846] 2021-07-31T02:29:06.476 INFO:tasks.workunit.client.0.smithi045.stdout:from 11 to HEAD diff [1823868~12981357,17882092~3725965,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143078160~11521808,154729104~8673241,168673724~9015241,181972347~5415105,202180755~18511385,221470799~1383607,225052374~5309220,248682238~10828300,264698001~7989139,296242351~493439,301341353~9191315,334588993~15115237,351585344~6792083,359181831~8323415,373441908~5818494,383830156~6295455,390462350~12518551,404169452~1052236,406364931~1096263] 2021-07-31T02:29:06.477 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [1823868~12981357,17882092~3725965,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143078160~11521808,154729104~8673241,168673724~9015241,181972347~5415105,202180755~18511385,221470799~1383607,225052374~5309220,248682238~10828300,264698001~7989139,296242351~493439,301341353~9191315,334588993~15115237,351585344~6792083,359181831~8323415,373441908~5818494,383830156~6295455,390462350~12518551,404169452~1052236,406364931~1096263] 2021-07-31T02:29:06.483 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [1823868~12981357,17882092~3725965,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,83886080~4661315,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143078160~11521808,154729104~8673241,168673724~9015241,181120288~6267164,202180755~18511385,221470799~1383607,225052374~5309220,248682238~10828300,264698001~7989139,296242351~493439,301341353~9191315,332356818~17347412,351585344~6792083,359181831~8323415,373441908~5818494,383830156~6295455,390462350~12518551,404169452~1052236,406364931~1096263] 2021-07-31T02:29:06.483 INFO:tasks.workunit.client.0.smithi045.stdout:from 12 to HEAD diff [1823868~12936934,17882092~3725965,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,71563215~1562492,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143369911~11230057,154729104~8673241,169186192~8502773,186487148~881866,202180755~18511385,221470799~1383607,225052374~5309220,248682238~10828300,264698001~7989139,301341353~9191315,335055096~14649134,351585344~6792083,359181831~8323415,375687458~1062344,383830156~6295455,390462350~12518551,406364931~1096263] 2021-07-31T02:29:06.484 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [1823868~12936934,17882092~3725965,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,71563215~1562492,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143369911~11230057,154729104~8673241,169186192~8502773,186487148~881866,202180755~18511385,221470799~1383607,225052374~5309220,248682238~10828300,264698001~7989139,301341353~9191315,335055096~14649134,351585344~6792083,359181831~8323415,375687458~1062344,383830156~6295455,390462350~12518551,406364931~1096263] 2021-07-31T02:29:06.491 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [1823868~12936934,17882092~3725965,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,71563215~1562492,83886080~4661315,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143369911~11230057,154729104~8673241,169186192~8502773,186487148~881866,202180755~18511385,221470799~1383607,225052374~5309220,248682238~10828300,264698001~7989139,301341353~9191315,335055096~14649134,351585344~6792083,359181831~8323415,375687458~1062344,383830156~6295455,390462350~12518551,406364931~1096263] 2021-07-31T02:29:06.492 INFO:tasks.workunit.client.0.smithi045.stdout:from 13 to HEAD diff [1823868~12936934,17882092~3725965,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,71563215~1562492,86327888~2219507,89929518~1882381,99285857~789121,121407098~3198227,135550801~7285470,143369911~11230057,154729104~8673241,169186192~7553729,186487148~881866,207917770~12774370,221470799~1383607,225052374~5309220,248682238~8485363,264698001~7989139,301341353~9191315,335055096~14649134,351585344~6792083,359181831~8323415,375687458~1062344,383830156~6295455,391467418~11513483,406364931~1096263] 2021-07-31T02:29:06.492 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [1823868~12936934,17882092~3725965,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,71563215~1562492,86327888~2219507,89929518~1882381,99285857~789121,121407098~3198227,135550801~7285470,143369911~11230057,154729104~8673241,169186192~7553729,186487148~881866,207917770~12774370,221470799~1383607,225052374~5309220,248682238~8485363,264698001~7989139,301341353~9191315,335055096~14649134,351585344~6792083,359181831~8323415,375687458~1062344,383830156~6295455,391467418~11513483,406364931~1096263] 2021-07-31T02:29:06.495 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [1823868~12936934,17882092~3725965,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,71563215~1562492,83886080~4661315,89929518~1882381,99285857~789121,121407098~3198227,135550801~7285470,143369911~11230057,154729104~8673241,169186192~7553729,186487148~881866,207917770~12774370,221470799~1383607,225052374~5309220,248682238~8485363,264698001~7989139,301341353~9191315,335055096~14649134,351585344~6792083,359181831~8323415,375687458~1062344,383830156~6295455,391467418~11513483,406364931~1096263] 2021-07-31T02:29:06.496 INFO:tasks.workunit.client.0.smithi045.stdout:from 14 to HEAD diff [1823868~12936934,17882092~3725965,24372582~14188618,53817318~2949139,60609981~5053700,71563215~1562492,86327888~2219507,89929518~1882381,99285857~789121,121407098~3198227,135550801~7285470,147626899~6973069,157325666~6076679,169186192~7553729,186487148~881866,207946866~2990158,213575464~7116676,221470799~1383607,225052374~5309220,248682238~8485363,264809820~7877320,305074438~5458230,335055096~14649134,351585344~6792083,359181831~8323415,375687458~1062344,383830156~6295455,391467418~11513483,406364931~1096263] 2021-07-31T02:29:06.496 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [1823868~12936934,17882092~3725965,24372582~14188618,53817318~2949139,60609981~5053700,71563215~1562492,86327888~2219507,89929518~1882381,99285857~789121,121407098~3198227,135550801~7285470,147626899~6973069,157325666~6076679,169186192~7553729,186487148~881866,207946866~2990158,213575464~7116676,221470799~1383607,225052374~5309220,248682238~8485363,264809820~7877320,305074438~5458230,335055096~14649134,351585344~6792083,359181831~8323415,375687458~1062344,383830156~6295455,391467418~11513483,406364931~1096263] 2021-07-31T02:29:06.501 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [1823868~12936934,17882092~3725965,24372582~14188618,53817318~2949139,60609981~5053700,71563215~1562492,83886080~4661315,89929518~1882381,99285857~789121,121407098~3198227,135550801~7285470,147626899~6973069,157325666~6076679,169186192~7553729,186487148~881866,207946866~2990158,213575464~7116676,221470799~1383607,225052374~5309220,248682238~8485363,264809820~7877320,305074438~5458230,335055096~14649134,351585344~6792083,359181831~8323415,375687458~1062344,383830156~6295455,391467418~11513483,406364931~1096263] 2021-07-31T02:29:06.502 INFO:tasks.workunit.client.0.smithi045.stdout:from 15 to HEAD diff [9296466~5464336,17882092~3725965,24372582~14188618,53817318~2949139,60609981~5053700,71563215~1562492,86327888~2219507,89929518~1882381,99285857~789121,121407098~3198227,135550801~7285470,157325666~6076679,169186192~7553729,186487148~881866,207946866~2990158,213575464~7116676,221470799~1383607,225052374~5309220,248682238~8485363,264809820~4477895,305074438~5008577,335055096~7851089,351585344~6792083,359544279~7960967,375687458~1062344,383830156~6295455,391467418~11513483] 2021-07-31T02:29:06.502 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [9296466~5464336,17882092~3725965,24372582~14188618,53817318~2949139,60609981~5053700,71563215~1562492,86327888~2219507,89929518~1882381,99285857~789121,121407098~3198227,135550801~7285470,157325666~6076679,169186192~7553729,186487148~881866,207946866~2990158,213575464~7116676,221470799~1383607,225052374~5309220,248682238~8485363,264809820~4477895,305074438~5008577,335055096~7851089,351585344~6792083,359544279~7960967,375687458~1062344,383830156~6295455,391467418~11513483] 2021-07-31T02:29:06.507 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [9296466~5464336,17882092~3725965,24372582~14188618,53817318~2949139,60609981~5053700,71563215~1562492,83886080~4661315,89929518~1882381,99285857~789121,121407098~3198227,135550801~7285470,157325666~6076679,169186192~7553729,186487148~881866,207946866~2990158,213575464~7116676,221470799~1383607,225052374~5309220,248682238~8485363,264809820~4477895,305074438~5008577,335055096~7851089,351585344~6792083,359544279~7960967,375687458~1062344,383830156~6295455,391467418~11513483] 2021-07-31T02:29:06.507 INFO:tasks.workunit.client.0.smithi045.stdout:from 16 to HEAD diff [9296466~5464336,17882092~3725965,31521345~7039855,55515079~1195728,60609981~5053700,71563215~1562492,89929518~1882381,99285857~789121,135550801~7285470,157325666~6076679,169186192~7553729,186487148~881866,213575464~7116676,221470799~1383607,248682238~8485363,264809820~4477895,305074438~5008577,335055096~7851089,351585344~6792083,359544279~7960967,375687458~1062344,383830156~6295455] 2021-07-31T02:29:06.508 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [9296466~5464336,17882092~3725965,31521345~7039855,55515079~1195728,60609981~5053700,71563215~1562492,89929518~1882381,99285857~789121,135550801~7285470,157325666~6076679,169186192~7553729,186487148~881866,213575464~7116676,221470799~1383607,248682238~8485363,264809820~4477895,305074438~5008577,335055096~7851089,351585344~6792083,359544279~7960967,375687458~1062344,383830156~6295455] 2021-07-31T02:29:06.515 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [9296466~5464336,17882092~3725965,31521345~7039855,55515079~1195728,60609981~5053700,71563215~1562492,89929518~1882381,99285857~789121,135550801~7285470,157325666~6076679,169186192~7553729,186487148~881866,213575464~7116676,221470799~1383607,248682238~8485363,264809820~4477895,305074438~5008577,335055096~7851089,351585344~6792083,359544279~7960967,375687458~1062344,383830156~6295455] 2021-07-31T02:29:06.515 INFO:tasks.workunit.client.0.smithi045.stdout:from 17 to HEAD diff [9296466~2171054,17882092~3725965,31521345~7039855,55515079~1195728,60609981~5053700,135550801~7285470,169186192~7553729,213575464~7116676,221470799~1383607,248682238~8485363,264809820~4477895,305074438~5008577,351737416~6640011,359544279~7960967,383830156~6295455] 2021-07-31T02:29:06.515 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [9296466~2171054,17882092~3725965,31521345~7039855,55515079~1195728,60609981~5053700,135550801~7285470,169186192~7553729,213575464~7116676,221470799~1383607,248682238~8485363,264809820~4477895,305074438~5008577,351737416~6640011,359544279~7960967,383830156~6295455] 2021-07-31T02:29:06.520 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [9296466~2171054,17882092~3725965,31521345~7039855,55515079~1195728,60609981~5053700,135550801~7285470,169186192~7553729,213575464~7116676,221470799~1383607,248682238~8485363,264809820~4477895,305074438~5008577,351737416~6640011,359544279~7960967,383830156~6295455] 2021-07-31T02:29:07.401 INFO:tasks.workunit.client.0.smithi045.stdout:from 0 to HEAD diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~22526082,86327888~2219507,89659479~3999082,97713673~8454237,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,236085312~4054340,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:07.401 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~22526082,86327888~2219507,89659479~3999082,97713673~8454237,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,236085312~4054340,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:07.432 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~43334606,46137344~10629113,59300692~1156219,60609981~22526082,83886080~9772481,96468992~14988213,116424046~70963406,188743680~9773402,201326592~33497107,234881024~24629514,260046848~20273171,289406976~21125692,314572800~57820771,373441908~5818494,381681664~32125072] 2021-07-31T02:29:07.433 INFO:tasks.workunit.client.0.smithi045.stdout:from 1 to HEAD diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89659479~3999082,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:07.433 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89659479~3999082,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:07.439 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~43334606,46137344~10629113,59300692~1156219,60609981~5053700,66056524~17079539,83886080~9772481,99285857~789121,102437886~9019319,116424046~70963406,191717187~6799895,201326592~33497107,240139652~19370886,260046848~20273171,289406976~21125692,314572800~57820771,373441908~5818494,381681664~32125072] 2021-07-31T02:29:07.440 INFO:tasks.workunit.client.0.smithi045.stdout:from 2 to HEAD diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89659479~3999082,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:07.440 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89659479~3999082,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:07.445 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~43334606,46137344~10629113,59300692~1156219,60609981~5053700,66056524~17079539,83886080~9772481,99285857~789121,102437886~9019319,116424046~70963406,191717187~6799895,201478817~33344882,240139652~19370886,260046848~20273171,291252461~19280207,314572800~12707070,327653500~3214988,331434830~40958741,373441908~5818494,381681664~32125072] 2021-07-31T02:29:07.446 INFO:tasks.workunit.client.0.smithi045.stdout:from 3 to HEAD diff [677713~20930344,24372582~18962024,48159474~2619597,53817318~2949139,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:07.447 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~20930344,24372582~18962024,48159474~2619597,53817318~2949139,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:07.451 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~21608057,24372582~18962024,48159474~2619597,53817318~2949139,59300692~1156219,60609981~5053700,66056524~17079539,83886080~4661315,89752687~3175987,99285857~789121,102437886~9019319,116424046~70963406,201478817~21375589,223825249~10998450,240139652~19370886,260046848~20273171,291252461~19280207,314572800~12707070,327653500~3214988,331434830~40958741,373441908~5818494,381681664~32125072] 2021-07-31T02:29:07.452 INFO:tasks.workunit.client.0.smithi045.stdout:from 4 to HEAD diff [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~20324185,167531438~13588850,181972347~5415105,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,292264750~18267918,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:07.452 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~20324185,167531438~13588850,181972347~5415105,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,292264750~18267918,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:07.458 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,102437886~9019319,116424046~8181279,124734943~38667402,164857432~22530020,201478817~21375589,223825249~10998450,240139652~19370886,260046848~12640292,275231912~5088107,292264750~18267918,314572800~12707070,327653500~3214988,331434830~40958741,373441908~5818494,381681664~32125072] 2021-07-31T02:29:07.458 INFO:tasks.workunit.client.0.smithi045.stdout:from 5 to HEAD diff [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,292264750~4687170,301341353~9191315,318433435~4174606,322615554~4664316,331638058~718760,334588993~36612024,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:07.458 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,292264750~4687170,301341353~9191315,318433435~4174606,322615554~4664316,331638058~718760,334588993~36612024,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:07.464 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,102437886~9019319,116424046~8181279,124734943~38667402,164857432~12831533,181120288~6267164,201478817~19213323,221470799~1383607,223825249~10998450,241317308~18193230,260046848~12640292,275231912~5088107,292264750~4687170,301341353~9191315,314572800~12707070,331434830~39766187,373441908~5818494,381681664~32125072] 2021-07-31T02:29:07.464 INFO:tasks.workunit.client.0.smithi045.stdout:from 6 to HEAD diff [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,121407098~3198227,124734943~18101328,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,223825249~6536345,241317308~18193230,260438339~12248801,296242351~493439,301341353~9191315,318433435~4174606,322615554~4664316,331638058~718760,334588993~36612024,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:07.464 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,121407098~3198227,124734943~18101328,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,223825249~6536345,241317308~18193230,260438339~12248801,296242351~493439,301341353~9191315,318433435~4174606,322615554~4664316,331638058~718760,334588993~36612024,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:07.470 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,102437886~3730024,121407098~3198227,124734943~18101328,143078160~20324185,164857432~12831533,181120288~6267164,201478817~19213323,221470799~1383607,223825249~6536345,241317308~18193230,260046848~12640292,296242351~493439,301341353~9191315,314572800~12707070,331434830~39766187,373441908~5818494,381681664~32125072] 2021-07-31T02:29:07.470 INFO:tasks.workunit.client.0.smithi045.stdout:from 7 to HEAD diff [872289~20735768,24372582~17941837,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,121407098~3198227,124734943~18101328,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,225052374~5309220,241317308~18193230,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,318433435~4174606,322615554~4664316,334588993~36612024,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:07.470 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~20735768,24372582~17941837,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,121407098~3198227,124734943~18101328,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,225052374~5309220,241317308~18193230,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,318433435~4174606,322615554~4664316,334588993~36612024,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:07.476 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~20735768,24372582~17941837,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,102437886~3730024,121407098~3198227,124734943~18101328,143078160~20324185,164857432~12831533,181120288~6267164,201478817~19213323,221470799~1383607,225052374~5309220,241317308~18193230,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,314572800~12707070,332356818~38844199,373441908~5818494,381681664~32125072] 2021-07-31T02:29:07.477 INFO:tasks.workunit.client.0.smithi045.stdout:from 8 to HEAD diff [872289~20735768,24372582~17941837,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~16816169,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,225052374~5309220,242763340~3666547,248659714~10850824,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,334588993~23788434,359181831~12019186,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:07.477 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~20735768,24372582~17941837,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~16816169,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,225052374~5309220,242763340~3666547,248659714~10850824,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,334588993~23788434,359181831~12019186,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:07.482 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~20735768,24372582~17941837,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,121407098~3198227,126020102~16816169,143078160~20324185,164857432~12831533,181120288~6267164,201478817~19213323,221470799~1383607,225052374~5309220,242763340~3666547,248659714~10850824,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,332356818~26020609,359181831~12019186,373441908~5818494,381820951~31985785] 2021-07-31T02:29:07.483 INFO:tasks.workunit.client.0.smithi045.stdout:from 9 to HEAD diff [1823868~19784189,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143078160~20324185,167792807~9896158,181972347~5415105,202180755~18511385,221470799~1383607,225052374~5309220,242763340~3666547,248659714~10850824,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,334588993~15115237,351585344~6792083,359181831~12019186,373441908~5818494,381820951~21159950,404169452~1052236,406364931~1096263,408916890~4889846] 2021-07-31T02:29:07.483 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [1823868~19784189,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143078160~20324185,167792807~9896158,181972347~5415105,202180755~18511385,221470799~1383607,225052374~5309220,242763340~3666547,248659714~10850824,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,334588993~15115237,351585344~6792083,359181831~12019186,373441908~5818494,381820951~21159950,404169452~1052236,406364931~1096263,408916890~4889846] 2021-07-31T02:29:07.489 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [1823868~19784189,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143078160~20324185,167792807~9896158,181120288~6267164,202180755~18511385,221470799~1383607,225052374~5309220,242763340~3666547,248659714~10850824,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,332356818~17347412,351585344~6792083,359181831~12019186,373441908~5818494,381820951~21159950,404169452~1052236,406364931~1096263,408916890~4889846] 2021-07-31T02:29:07.489 INFO:tasks.workunit.client.0.smithi045.stdout:from 10 to HEAD diff [1823868~19784189,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143078160~11521808,154729104~8673241,167792807~9896158,181972347~5415105,202180755~18511385,221470799~1383607,225052374~5309220,242763340~3666547,248682238~10828300,264698001~7989139,296242351~493439,301341353~9191315,334588993~15115237,351585344~6792083,359181831~12019186,373441908~5818494,381820951~21159950,404169452~1052236,406364931~1096263,408916890~4889846] 2021-07-31T02:29:07.489 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [1823868~19784189,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143078160~11521808,154729104~8673241,167792807~9896158,181972347~5415105,202180755~18511385,221470799~1383607,225052374~5309220,242763340~3666547,248682238~10828300,264698001~7989139,296242351~493439,301341353~9191315,334588993~15115237,351585344~6792083,359181831~12019186,373441908~5818494,381820951~21159950,404169452~1052236,406364931~1096263,408916890~4889846] 2021-07-31T02:29:07.494 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [1823868~19784189,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143078160~11521808,154729104~8673241,167792807~9896158,181120288~6267164,202180755~18511385,221470799~1383607,225052374~5309220,242763340~3666547,248682238~10828300,264698001~7989139,296242351~493439,301341353~9191315,332356818~17347412,351585344~6792083,359181831~12019186,373441908~5818494,381820951~21159950,404169452~1052236,406364931~1096263,408916890~4889846] 2021-07-31T02:29:07.495 INFO:tasks.workunit.client.0.smithi045.stdout:from 11 to HEAD diff [1823868~12981357,17882092~3725965,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143078160~11521808,154729104~8673241,168673724~9015241,181972347~5415105,202180755~18511385,221470799~1383607,225052374~5309220,248682238~10828300,264698001~7989139,296242351~493439,301341353~9191315,334588993~15115237,351585344~6792083,359181831~8323415,373441908~5818494,383830156~6295455,390462350~12518551,404169452~1052236,406364931~1096263] 2021-07-31T02:29:07.495 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [1823868~12981357,17882092~3725965,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143078160~11521808,154729104~8673241,168673724~9015241,181972347~5415105,202180755~18511385,221470799~1383607,225052374~5309220,248682238~10828300,264698001~7989139,296242351~493439,301341353~9191315,334588993~15115237,351585344~6792083,359181831~8323415,373441908~5818494,383830156~6295455,390462350~12518551,404169452~1052236,406364931~1096263] 2021-07-31T02:29:07.501 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [1823868~12981357,17882092~3725965,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,83886080~4661315,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143078160~11521808,154729104~8673241,168673724~9015241,181120288~6267164,202180755~18511385,221470799~1383607,225052374~5309220,248682238~10828300,264698001~7989139,296242351~493439,301341353~9191315,332356818~17347412,351585344~6792083,359181831~8323415,373441908~5818494,383830156~6295455,390462350~12518551,404169452~1052236,406364931~1096263] 2021-07-31T02:29:07.501 INFO:tasks.workunit.client.0.smithi045.stdout:from 12 to HEAD diff [1823868~12936934,17882092~3725965,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,71563215~1562492,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143369911~11230057,154729104~8673241,169186192~8502773,186487148~881866,202180755~18511385,221470799~1383607,225052374~5309220,248682238~10828300,264698001~7989139,301341353~9191315,335055096~14649134,351585344~6792083,359181831~8323415,375687458~1062344,383830156~6295455,390462350~12518551,406364931~1096263] 2021-07-31T02:29:07.501 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [1823868~12936934,17882092~3725965,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,71563215~1562492,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143369911~11230057,154729104~8673241,169186192~8502773,186487148~881866,202180755~18511385,221470799~1383607,225052374~5309220,248682238~10828300,264698001~7989139,301341353~9191315,335055096~14649134,351585344~6792083,359181831~8323415,375687458~1062344,383830156~6295455,390462350~12518551,406364931~1096263] 2021-07-31T02:29:07.507 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [1823868~12936934,17882092~3725965,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,71563215~1562492,83886080~4661315,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143369911~11230057,154729104~8673241,169186192~8502773,186487148~881866,202180755~18511385,221470799~1383607,225052374~5309220,248682238~10828300,264698001~7989139,301341353~9191315,335055096~14649134,351585344~6792083,359181831~8323415,375687458~1062344,383830156~6295455,390462350~12518551,406364931~1096263] 2021-07-31T02:29:07.507 INFO:tasks.workunit.client.0.smithi045.stdout:from 13 to HEAD diff [1823868~12936934,17882092~3725965,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,71563215~1562492,86327888~2219507,89929518~1882381,99285857~789121,121407098~3198227,135550801~7285470,143369911~11230057,154729104~8673241,169186192~7553729,186487148~881866,207917770~12774370,221470799~1383607,225052374~5309220,248682238~8485363,264698001~7989139,301341353~9191315,335055096~14649134,351585344~6792083,359181831~8323415,375687458~1062344,383830156~6295455,391467418~11513483,406364931~1096263] 2021-07-31T02:29:07.508 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [1823868~12936934,17882092~3725965,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,71563215~1562492,86327888~2219507,89929518~1882381,99285857~789121,121407098~3198227,135550801~7285470,143369911~11230057,154729104~8673241,169186192~7553729,186487148~881866,207917770~12774370,221470799~1383607,225052374~5309220,248682238~8485363,264698001~7989139,301341353~9191315,335055096~14649134,351585344~6792083,359181831~8323415,375687458~1062344,383830156~6295455,391467418~11513483,406364931~1096263] 2021-07-31T02:29:07.513 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [1823868~12936934,17882092~3725965,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,71563215~1562492,83886080~4661315,89929518~1882381,99285857~789121,121407098~3198227,135550801~7285470,143369911~11230057,154729104~8673241,169186192~7553729,186487148~881866,207917770~12774370,221470799~1383607,225052374~5309220,248682238~8485363,264698001~7989139,301341353~9191315,335055096~14649134,351585344~6792083,359181831~8323415,375687458~1062344,383830156~6295455,391467418~11513483,406364931~1096263] 2021-07-31T02:29:07.513 INFO:tasks.workunit.client.0.smithi045.stdout:from 14 to HEAD diff [1823868~12936934,17882092~3725965,24372582~14188618,53817318~2949139,60609981~5053700,71563215~1562492,86327888~2219507,89929518~1882381,99285857~789121,121407098~3198227,135550801~7285470,147626899~6973069,157325666~6076679,169186192~7553729,186487148~881866,207946866~2990158,213575464~7116676,221470799~1383607,225052374~5309220,248682238~8485363,264809820~7877320,305074438~5458230,335055096~14649134,351585344~6792083,359181831~8323415,375687458~1062344,383830156~6295455,391467418~11513483,406364931~1096263] 2021-07-31T02:29:07.514 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [1823868~12936934,17882092~3725965,24372582~14188618,53817318~2949139,60609981~5053700,71563215~1562492,86327888~2219507,89929518~1882381,99285857~789121,121407098~3198227,135550801~7285470,147626899~6973069,157325666~6076679,169186192~7553729,186487148~881866,207946866~2990158,213575464~7116676,221470799~1383607,225052374~5309220,248682238~8485363,264809820~7877320,305074438~5458230,335055096~14649134,351585344~6792083,359181831~8323415,375687458~1062344,383830156~6295455,391467418~11513483,406364931~1096263] 2021-07-31T02:29:07.519 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [1823868~12936934,17882092~3725965,24372582~14188618,53817318~2949139,60609981~5053700,71563215~1562492,83886080~4661315,89929518~1882381,99285857~789121,121407098~3198227,135550801~7285470,147626899~6973069,157325666~6076679,169186192~7553729,186487148~881866,207946866~2990158,213575464~7116676,221470799~1383607,225052374~5309220,248682238~8485363,264809820~7877320,305074438~5458230,335055096~14649134,351585344~6792083,359181831~8323415,375687458~1062344,383830156~6295455,391467418~11513483,406364931~1096263] 2021-07-31T02:29:07.519 INFO:tasks.workunit.client.0.smithi045.stdout:from 15 to HEAD diff [9296466~5464336,17882092~3725965,24372582~14188618,53817318~2949139,60609981~5053700,71563215~1562492,86327888~2219507,89929518~1882381,99285857~789121,121407098~3198227,135550801~7285470,157325666~6076679,169186192~7553729,186487148~881866,207946866~2990158,213575464~7116676,221470799~1383607,225052374~5309220,248682238~8485363,264809820~4477895,305074438~5008577,335055096~7851089,351585344~6792083,359544279~7960967,375687458~1062344,383830156~6295455,391467418~11513483] 2021-07-31T02:29:07.519 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [9296466~5464336,17882092~3725965,24372582~14188618,53817318~2949139,60609981~5053700,71563215~1562492,86327888~2219507,89929518~1882381,99285857~789121,121407098~3198227,135550801~7285470,157325666~6076679,169186192~7553729,186487148~881866,207946866~2990158,213575464~7116676,221470799~1383607,225052374~5309220,248682238~8485363,264809820~4477895,305074438~5008577,335055096~7851089,351585344~6792083,359544279~7960967,375687458~1062344,383830156~6295455,391467418~11513483] 2021-07-31T02:29:07.525 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [9296466~5464336,17882092~3725965,24372582~14188618,53817318~2949139,60609981~5053700,71563215~1562492,83886080~4661315,89929518~1882381,99285857~789121,121407098~3198227,135550801~7285470,157325666~6076679,169186192~7553729,186487148~881866,207946866~2990158,213575464~7116676,221470799~1383607,225052374~5309220,248682238~8485363,264809820~4477895,305074438~5008577,335055096~7851089,351585344~6792083,359544279~7960967,375687458~1062344,383830156~6295455,391467418~11513483] 2021-07-31T02:29:07.525 INFO:tasks.workunit.client.0.smithi045.stdout:from 16 to HEAD diff [9296466~5464336,17882092~3725965,31521345~7039855,55515079~1195728,60609981~5053700,71563215~1562492,89929518~1882381,99285857~789121,135550801~7285470,157325666~6076679,169186192~7553729,186487148~881866,213575464~7116676,221470799~1383607,248682238~8485363,264809820~4477895,305074438~5008577,335055096~7851089,351585344~6792083,359544279~7960967,375687458~1062344,383830156~6295455] 2021-07-31T02:29:07.525 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [9296466~5464336,17882092~3725965,31521345~7039855,55515079~1195728,60609981~5053700,71563215~1562492,89929518~1882381,99285857~789121,135550801~7285470,157325666~6076679,169186192~7553729,186487148~881866,213575464~7116676,221470799~1383607,248682238~8485363,264809820~4477895,305074438~5008577,335055096~7851089,351585344~6792083,359544279~7960967,375687458~1062344,383830156~6295455] 2021-07-31T02:29:07.530 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [9296466~5464336,17882092~3725965,31521345~7039855,55515079~1195728,60609981~5053700,71563215~1562492,89929518~1882381,99285857~789121,135550801~7285470,157325666~6076679,169186192~7553729,186487148~881866,213575464~7116676,221470799~1383607,248682238~8485363,264809820~4477895,305074438~5008577,335055096~7851089,351585344~6792083,359544279~7960967,375687458~1062344,383830156~6295455] 2021-07-31T02:29:07.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:07 smithi154 conmon[34553]: cluster 2021-07-31T02:29:06.387060+0000 mon.a (mon.0) 2091 2021-07-31T02:29:07.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:07 smithi154 conmon[34553]: : cluster [DBG] osdmap e1166: 8 total, 8 up, 8 in 2021-07-31T02:29:07.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:07 smithi154 conmon[34553]: cluster 2021-07-31T02:29:06.398341+0000 mgr.x (mgr.24542) 1793 : cluster [DBG] pgmap v2269: 41 pgs: 41 active+clean; 885 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 7.4 KiB/s rd, 50 MiB/s wr, 35 op/s 2021-07-31T02:29:07.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:07 smithi154 conmon[34553]: 2021-07-31T02:29:07.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:07 smithi045 conmon[39718]: cluster 2021-07-31T02:29:06.387060+0000 2021-07-31T02:29:07.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:07 smithi045 conmon[39718]: mon.a (mon.0) 2091 : cluster [DBG] osdmap e1166: 8 total, 8 up, 8 in 2021-07-31T02:29:07.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:07 smithi045 conmon[39718]: cluster 2021-07-31T02:29: 2021-07-31T02:29:07.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:07 smithi045 conmon[39718]: 06.398341+0000 mgr.x (mgr.24542) 1793 : cluster [DBG] pgmap v2269: 41 pgs: 41 active+clean; 885 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 7.4 KiB/s rd, 50 MiB/s wr, 35 op/s 2021-07-31T02:29:07.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:07 smithi045 conmon[33655]: cluster 2021-07-31T02:29:06.387060+0000 mon.a ( 2021-07-31T02:29:07.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:07 smithi045 conmon[33655]: mon.0) 2091 : cluster [DBG] osdmap e1166: 8 total, 8 up, 8 in 2021-07-31T02:29:07.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:07 smithi045 conmon[33655]: cluster 2021-07-31T02:29:06.398341+0000 mgr.x (mgr.24542) 1793 : cluster [DBG] pgmap v2269: 41 pgs: 41 active+clean; 885 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 7.4 KiB/s rd, 50 MiB/s wr, 35 op/s 2021-07-31T02:29:08.416 INFO:tasks.workunit.client.0.smithi045.stdout:from 0 to HEAD diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~22526082,86327888~2219507,89659479~3999082,97713673~8454237,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,236085312~4054340,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:08.417 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~22526082,86327888~2219507,89659479~3999082,97713673~8454237,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,236085312~4054340,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:08.464 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~43334606,46137344~10629113,59300692~1156219,60609981~22526082,83886080~9772481,96468992~14988213,116424046~70963406,188743680~9773402,201326592~33497107,234881024~24629514,260046848~20273171,289406976~21125692,314572800~57820771,373441908~5818494,381681664~32125072] 2021-07-31T02:29:08.465 INFO:tasks.workunit.client.0.smithi045.stdout:from 1 to HEAD diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89659479~3999082,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:08.465 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89659479~3999082,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:08.471 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~43334606,46137344~10629113,59300692~1156219,60609981~5053700,66056524~17079539,83886080~9772481,99285857~789121,102437886~9019319,116424046~70963406,191717187~6799895,201326592~33497107,240139652~19370886,260046848~20273171,289406976~21125692,314572800~57820771,373441908~5818494,381681664~32125072] 2021-07-31T02:29:08.471 INFO:tasks.workunit.client.0.smithi045.stdout:from 2 to HEAD diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89659479~3999082,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:08.472 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89659479~3999082,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:08.479 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~43334606,46137344~10629113,59300692~1156219,60609981~5053700,66056524~17079539,83886080~9772481,99285857~789121,102437886~9019319,116424046~70963406,191717187~6799895,201478817~33344882,240139652~19370886,260046848~20273171,291252461~19280207,314572800~12707070,327653500~3214988,331434830~40958741,373441908~5818494,381681664~32125072] 2021-07-31T02:29:08.479 INFO:tasks.workunit.client.0.smithi045.stdout:from 3 to HEAD diff [677713~20930344,24372582~18962024,48159474~2619597,53817318~2949139,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:08.479 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~20930344,24372582~18962024,48159474~2619597,53817318~2949139,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:08.484 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~21608057,24372582~18962024,48159474~2619597,53817318~2949139,59300692~1156219,60609981~5053700,66056524~17079539,83886080~4661315,89752687~3175987,99285857~789121,102437886~9019319,116424046~70963406,201478817~21375589,223825249~10998450,240139652~19370886,260046848~20273171,291252461~19280207,314572800~12707070,327653500~3214988,331434830~40958741,373441908~5818494,381681664~32125072] 2021-07-31T02:29:08.484 INFO:tasks.workunit.client.0.smithi045.stdout:from 4 to HEAD diff [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~20324185,167531438~13588850,181972347~5415105,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,292264750~18267918,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:08.485 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~20324185,167531438~13588850,181972347~5415105,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,292264750~18267918,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:08.490 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,102437886~9019319,116424046~8181279,124734943~38667402,164857432~22530020,201478817~21375589,223825249~10998450,240139652~19370886,260046848~12640292,275231912~5088107,292264750~18267918,314572800~12707070,327653500~3214988,331434830~40958741,373441908~5818494,381681664~32125072] 2021-07-31T02:29:08.491 INFO:tasks.workunit.client.0.smithi045.stdout:from 5 to HEAD diff [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,292264750~4687170,301341353~9191315,318433435~4174606,322615554~4664316,331638058~718760,334588993~36612024,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:08.491 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,292264750~4687170,301341353~9191315,318433435~4174606,322615554~4664316,331638058~718760,334588993~36612024,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:08.497 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,102437886~9019319,116424046~8181279,124734943~38667402,164857432~12831533,181120288~6267164,201478817~19213323,221470799~1383607,223825249~10998450,241317308~18193230,260046848~12640292,275231912~5088107,292264750~4687170,301341353~9191315,314572800~12707070,331434830~39766187,373441908~5818494,381681664~32125072] 2021-07-31T02:29:08.497 INFO:tasks.workunit.client.0.smithi045.stdout:from 6 to HEAD diff [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,121407098~3198227,124734943~18101328,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,223825249~6536345,241317308~18193230,260438339~12248801,296242351~493439,301341353~9191315,318433435~4174606,322615554~4664316,331638058~718760,334588993~36612024,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:08.497 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,121407098~3198227,124734943~18101328,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,223825249~6536345,241317308~18193230,260438339~12248801,296242351~493439,301341353~9191315,318433435~4174606,322615554~4664316,331638058~718760,334588993~36612024,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:08.502 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,102437886~3730024,121407098~3198227,124734943~18101328,143078160~20324185,164857432~12831533,181120288~6267164,201478817~19213323,221470799~1383607,223825249~6536345,241317308~18193230,260046848~12640292,296242351~493439,301341353~9191315,314572800~12707070,331434830~39766187,373441908~5818494,381681664~32125072] 2021-07-31T02:29:08.503 INFO:tasks.workunit.client.0.smithi045.stdout:from 7 to HEAD diff [872289~20735768,24372582~17941837,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,121407098~3198227,124734943~18101328,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,225052374~5309220,241317308~18193230,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,318433435~4174606,322615554~4664316,334588993~36612024,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:08.503 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~20735768,24372582~17941837,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,121407098~3198227,124734943~18101328,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,225052374~5309220,241317308~18193230,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,318433435~4174606,322615554~4664316,334588993~36612024,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:08.508 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~20735768,24372582~17941837,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,102437886~3730024,121407098~3198227,124734943~18101328,143078160~20324185,164857432~12831533,181120288~6267164,201478817~19213323,221470799~1383607,225052374~5309220,241317308~18193230,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,314572800~12707070,332356818~38844199,373441908~5818494,381681664~32125072] 2021-07-31T02:29:08.509 INFO:tasks.workunit.client.0.smithi045.stdout:from 8 to HEAD diff [872289~20735768,24372582~17941837,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~16816169,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,225052374~5309220,242763340~3666547,248659714~10850824,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,334588993~23788434,359181831~12019186,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:08.509 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~20735768,24372582~17941837,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~16816169,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,225052374~5309220,242763340~3666547,248659714~10850824,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,334588993~23788434,359181831~12019186,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:08.514 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~20735768,24372582~17941837,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,121407098~3198227,126020102~16816169,143078160~20324185,164857432~12831533,181120288~6267164,201478817~19213323,221470799~1383607,225052374~5309220,242763340~3666547,248659714~10850824,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,332356818~26020609,359181831~12019186,373441908~5818494,381820951~31985785] 2021-07-31T02:29:08.514 INFO:tasks.workunit.client.0.smithi045.stdout:from 9 to HEAD diff [1823868~19784189,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143078160~20324185,167792807~9896158,181972347~5415105,202180755~18511385,221470799~1383607,225052374~5309220,242763340~3666547,248659714~10850824,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,334588993~15115237,351585344~6792083,359181831~12019186,373441908~5818494,381820951~21159950,404169452~1052236,406364931~1096263,408916890~4889846] 2021-07-31T02:29:08.515 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [1823868~19784189,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143078160~20324185,167792807~9896158,181972347~5415105,202180755~18511385,221470799~1383607,225052374~5309220,242763340~3666547,248659714~10850824,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,334588993~15115237,351585344~6792083,359181831~12019186,373441908~5818494,381820951~21159950,404169452~1052236,406364931~1096263,408916890~4889846] 2021-07-31T02:29:08.522 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [1823868~19784189,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143078160~20324185,167792807~9896158,181120288~6267164,202180755~18511385,221470799~1383607,225052374~5309220,242763340~3666547,248659714~10850824,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,332356818~17347412,351585344~6792083,359181831~12019186,373441908~5818494,381820951~21159950,404169452~1052236,406364931~1096263,408916890~4889846] 2021-07-31T02:29:08.524 INFO:tasks.workunit.client.0.smithi045.stdout:from 10 to HEAD diff [1823868~19784189,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143078160~11521808,154729104~8673241,167792807~9896158,181972347~5415105,202180755~18511385,221470799~1383607,225052374~5309220,242763340~3666547,248682238~10828300,264698001~7989139,296242351~493439,301341353~9191315,334588993~15115237,351585344~6792083,359181831~12019186,373441908~5818494,381820951~21159950,404169452~1052236,406364931~1096263,408916890~4889846] 2021-07-31T02:29:08.525 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [1823868~19784189,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143078160~11521808,154729104~8673241,167792807~9896158,181972347~5415105,202180755~18511385,221470799~1383607,225052374~5309220,242763340~3666547,248682238~10828300,264698001~7989139,296242351~493439,301341353~9191315,334588993~15115237,351585344~6792083,359181831~12019186,373441908~5818494,381820951~21159950,404169452~1052236,406364931~1096263,408916890~4889846] 2021-07-31T02:29:08.526 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [1823868~19784189,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143078160~11521808,154729104~8673241,167792807~9896158,181120288~6267164,202180755~18511385,221470799~1383607,225052374~5309220,242763340~3666547,248682238~10828300,264698001~7989139,296242351~493439,301341353~9191315,332356818~17347412,351585344~6792083,359181831~12019186,373441908~5818494,381820951~21159950,404169452~1052236,406364931~1096263,408916890~4889846] 2021-07-31T02:29:08.526 INFO:tasks.workunit.client.0.smithi045.stdout:from 11 to HEAD diff [1823868~12981357,17882092~3725965,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143078160~11521808,154729104~8673241,168673724~9015241,181972347~5415105,202180755~18511385,221470799~1383607,225052374~5309220,248682238~10828300,264698001~7989139,296242351~493439,301341353~9191315,334588993~15115237,351585344~6792083,359181831~8323415,373441908~5818494,383830156~6295455,390462350~12518551,404169452~1052236,406364931~1096263] 2021-07-31T02:29:08.527 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [1823868~12981357,17882092~3725965,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143078160~11521808,154729104~8673241,168673724~9015241,181972347~5415105,202180755~18511385,221470799~1383607,225052374~5309220,248682238~10828300,264698001~7989139,296242351~493439,301341353~9191315,334588993~15115237,351585344~6792083,359181831~8323415,373441908~5818494,383830156~6295455,390462350~12518551,404169452~1052236,406364931~1096263] 2021-07-31T02:29:08.534 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [1823868~12981357,17882092~3725965,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,83886080~4661315,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143078160~11521808,154729104~8673241,168673724~9015241,181120288~6267164,202180755~18511385,221470799~1383607,225052374~5309220,248682238~10828300,264698001~7989139,296242351~493439,301341353~9191315,332356818~17347412,351585344~6792083,359181831~8323415,373441908~5818494,383830156~6295455,390462350~12518551,404169452~1052236,406364931~1096263] 2021-07-31T02:29:08.534 INFO:tasks.workunit.client.0.smithi045.stdout:from 12 to HEAD diff [1823868~12936934,17882092~3725965,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,71563215~1562492,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143369911~11230057,154729104~8673241,169186192~8502773,186487148~881866,202180755~18511385,221470799~1383607,225052374~5309220,248682238~10828300,264698001~7989139,301341353~9191315,335055096~14649134,351585344~6792083,359181831~8323415,375687458~1062344,383830156~6295455,390462350~12518551,406364931~1096263] 2021-07-31T02:29:08.535 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [1823868~12936934,17882092~3725965,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,71563215~1562492,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143369911~11230057,154729104~8673241,169186192~8502773,186487148~881866,202180755~18511385,221470799~1383607,225052374~5309220,248682238~10828300,264698001~7989139,301341353~9191315,335055096~14649134,351585344~6792083,359181831~8323415,375687458~1062344,383830156~6295455,390462350~12518551,406364931~1096263] 2021-07-31T02:29:08.539 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [1823868~12936934,17882092~3725965,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,71563215~1562492,83886080~4661315,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143369911~11230057,154729104~8673241,169186192~8502773,186487148~881866,202180755~18511385,221470799~1383607,225052374~5309220,248682238~10828300,264698001~7989139,301341353~9191315,335055096~14649134,351585344~6792083,359181831~8323415,375687458~1062344,383830156~6295455,390462350~12518551,406364931~1096263] 2021-07-31T02:29:08.540 INFO:tasks.workunit.client.0.smithi045.stdout:from 13 to HEAD diff [1823868~12936934,17882092~3725965,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,71563215~1562492,86327888~2219507,89929518~1882381,99285857~789121,121407098~3198227,135550801~7285470,143369911~11230057,154729104~8673241,169186192~7553729,186487148~881866,207917770~12774370,221470799~1383607,225052374~5309220,248682238~8485363,264698001~7989139,301341353~9191315,335055096~14649134,351585344~6792083,359181831~8323415,375687458~1062344,383830156~6295455,391467418~11513483,406364931~1096263] 2021-07-31T02:29:08.540 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [1823868~12936934,17882092~3725965,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,71563215~1562492,86327888~2219507,89929518~1882381,99285857~789121,121407098~3198227,135550801~7285470,143369911~11230057,154729104~8673241,169186192~7553729,186487148~881866,207917770~12774370,221470799~1383607,225052374~5309220,248682238~8485363,264698001~7989139,301341353~9191315,335055096~14649134,351585344~6792083,359181831~8323415,375687458~1062344,383830156~6295455,391467418~11513483,406364931~1096263] 2021-07-31T02:29:08.545 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [1823868~12936934,17882092~3725965,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,71563215~1562492,83886080~4661315,89929518~1882381,99285857~789121,121407098~3198227,135550801~7285470,143369911~11230057,154729104~8673241,169186192~7553729,186487148~881866,207917770~12774370,221470799~1383607,225052374~5309220,248682238~8485363,264698001~7989139,301341353~9191315,335055096~14649134,351585344~6792083,359181831~8323415,375687458~1062344,383830156~6295455,391467418~11513483,406364931~1096263] 2021-07-31T02:29:08.545 INFO:tasks.workunit.client.0.smithi045.stdout:from 14 to HEAD diff [1823868~12936934,17882092~3725965,24372582~14188618,53817318~2949139,60609981~5053700,71563215~1562492,86327888~2219507,89929518~1882381,99285857~789121,121407098~3198227,135550801~7285470,147626899~6973069,157325666~6076679,169186192~7553729,186487148~881866,207946866~2990158,213575464~7116676,221470799~1383607,225052374~5309220,248682238~8485363,264809820~7877320,305074438~5458230,335055096~14649134,351585344~6792083,359181831~8323415,375687458~1062344,383830156~6295455,391467418~11513483,406364931~1096263] 2021-07-31T02:29:08.545 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [1823868~12936934,17882092~3725965,24372582~14188618,53817318~2949139,60609981~5053700,71563215~1562492,86327888~2219507,89929518~1882381,99285857~789121,121407098~3198227,135550801~7285470,147626899~6973069,157325666~6076679,169186192~7553729,186487148~881866,207946866~2990158,213575464~7116676,221470799~1383607,225052374~5309220,248682238~8485363,264809820~7877320,305074438~5458230,335055096~14649134,351585344~6792083,359181831~8323415,375687458~1062344,383830156~6295455,391467418~11513483,406364931~1096263] 2021-07-31T02:29:08.550 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [1823868~12936934,17882092~3725965,24372582~14188618,53817318~2949139,60609981~5053700,71563215~1562492,83886080~4661315,89929518~1882381,99285857~789121,121407098~3198227,135550801~7285470,147626899~6973069,157325666~6076679,169186192~7553729,186487148~881866,207946866~2990158,213575464~7116676,221470799~1383607,225052374~5309220,248682238~8485363,264809820~7877320,305074438~5458230,335055096~14649134,351585344~6792083,359181831~8323415,375687458~1062344,383830156~6295455,391467418~11513483,406364931~1096263] 2021-07-31T02:29:08.552 INFO:tasks.workunit.client.0.smithi045.stdout:from 15 to HEAD diff [9296466~5464336,17882092~3725965,24372582~14188618,53817318~2949139,60609981~5053700,71563215~1562492,86327888~2219507,89929518~1882381,99285857~789121,121407098~3198227,135550801~7285470,157325666~6076679,169186192~7553729,186487148~881866,207946866~2990158,213575464~7116676,221470799~1383607,225052374~5309220,248682238~8485363,264809820~4477895,305074438~5008577,335055096~7851089,351585344~6792083,359544279~7960967,375687458~1062344,383830156~6295455,391467418~11513483] 2021-07-31T02:29:08.552 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [9296466~5464336,17882092~3725965,24372582~14188618,53817318~2949139,60609981~5053700,71563215~1562492,86327888~2219507,89929518~1882381,99285857~789121,121407098~3198227,135550801~7285470,157325666~6076679,169186192~7553729,186487148~881866,207946866~2990158,213575464~7116676,221470799~1383607,225052374~5309220,248682238~8485363,264809820~4477895,305074438~5008577,335055096~7851089,351585344~6792083,359544279~7960967,375687458~1062344,383830156~6295455,391467418~11513483] 2021-07-31T02:29:08.556 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [9296466~5464336,17882092~3725965,24372582~14188618,53817318~2949139,60609981~5053700,71563215~1562492,83886080~4661315,89929518~1882381,99285857~789121,121407098~3198227,135550801~7285470,157325666~6076679,169186192~7553729,186487148~881866,207946866~2990158,213575464~7116676,221470799~1383607,225052374~5309220,248682238~8485363,264809820~4477895,305074438~5008577,335055096~7851089,351585344~6792083,359544279~7960967,375687458~1062344,383830156~6295455,391467418~11513483] 2021-07-31T02:29:08.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:08 smithi154 conmon[34553]: cluster 2021-07-31T02:29:07.390934+0000 mon.a (mon.0) 2092 : cluster [DBG] osdmap e1167: 8 total, 8 up, 8 in 2021-07-31T02:29:08.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:08 smithi045 conmon[39718]: cluster 2021-07-31T02:29:07 2021-07-31T02:29:08.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:08 smithi045 conmon[39718]: .390934+0000 mon.a (mon.0) 2092 : cluster [DBG] osdmap e1167: 8 total, 8 up, 8 in 2021-07-31T02:29:08.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:08 smithi045 conmon[33655]: cluster 2021-07-31T02:29:07.390934+0000 mon.a (mon.0) 2092 2021-07-31T02:29:08.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:08 smithi045 conmon[33655]: : cluster [DBG] osdmap e1167: 8 total, 8 up, 8 in 2021-07-31T02:29:09.414 INFO:tasks.workunit.client.0.smithi045.stdout:from 0 to HEAD diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~22526082,86327888~2219507,89659479~3999082,97713673~8454237,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,236085312~4054340,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:09.414 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~22526082,86327888~2219507,89659479~3999082,97713673~8454237,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,236085312~4054340,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:09.472 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~43334606,46137344~10629113,59300692~1156219,60609981~22526082,83886080~9772481,96468992~14988213,116424046~70963406,188743680~9773402,201326592~33497107,234881024~24629514,260046848~20273171,289406976~21125692,314572800~57820771,373441908~5818494,381681664~32125072] 2021-07-31T02:29:09.472 INFO:tasks.workunit.client.0.smithi045.stdout:from 1 to HEAD diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89659479~3999082,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:09.472 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89659479~3999082,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:09.478 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~43334606,46137344~10629113,59300692~1156219,60609981~5053700,66056524~17079539,83886080~9772481,99285857~789121,102437886~9019319,116424046~70963406,191717187~6799895,201326592~33497107,240139652~19370886,260046848~20273171,289406976~21125692,314572800~57820771,373441908~5818494,381681664~32125072] 2021-07-31T02:29:09.479 INFO:tasks.workunit.client.0.smithi045.stdout:from 2 to HEAD diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89659479~3999082,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:09.479 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89659479~3999082,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:09.485 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~43334606,46137344~10629113,59300692~1156219,60609981~5053700,66056524~17079539,83886080~9772481,99285857~789121,102437886~9019319,116424046~70963406,191717187~6799895,201478817~33344882,240139652~19370886,260046848~20273171,291252461~19280207,314572800~12707070,327653500~3214988,331434830~40958741,373441908~5818494,381681664~32125072] 2021-07-31T02:29:09.485 INFO:tasks.workunit.client.0.smithi045.stdout:from 3 to HEAD diff [677713~20930344,24372582~18962024,48159474~2619597,53817318~2949139,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:09.486 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~20930344,24372582~18962024,48159474~2619597,53817318~2949139,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:09.490 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~21608057,24372582~18962024,48159474~2619597,53817318~2949139,59300692~1156219,60609981~5053700,66056524~17079539,83886080~4661315,89752687~3175987,99285857~789121,102437886~9019319,116424046~70963406,201478817~21375589,223825249~10998450,240139652~19370886,260046848~20273171,291252461~19280207,314572800~12707070,327653500~3214988,331434830~40958741,373441908~5818494,381681664~32125072] 2021-07-31T02:29:09.490 INFO:tasks.workunit.client.0.smithi045.stdout:from 4 to HEAD diff [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~20324185,167531438~13588850,181972347~5415105,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,292264750~18267918,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:09.491 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~20324185,167531438~13588850,181972347~5415105,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,292264750~18267918,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:09.500 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,102437886~9019319,116424046~8181279,124734943~38667402,164857432~22530020,201478817~21375589,223825249~10998450,240139652~19370886,260046848~12640292,275231912~5088107,292264750~18267918,314572800~12707070,327653500~3214988,331434830~40958741,373441908~5818494,381681664~32125072] 2021-07-31T02:29:09.501 INFO:tasks.workunit.client.0.smithi045.stdout:from 5 to HEAD diff [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,292264750~4687170,301341353~9191315,318433435~4174606,322615554~4664316,331638058~718760,334588993~36612024,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:09.501 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,292264750~4687170,301341353~9191315,318433435~4174606,322615554~4664316,331638058~718760,334588993~36612024,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:09.503 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,102437886~9019319,116424046~8181279,124734943~38667402,164857432~12831533,181120288~6267164,201478817~19213323,221470799~1383607,223825249~10998450,241317308~18193230,260046848~12640292,275231912~5088107,292264750~4687170,301341353~9191315,314572800~12707070,331434830~39766187,373441908~5818494,381681664~32125072] 2021-07-31T02:29:09.503 INFO:tasks.workunit.client.0.smithi045.stdout:from 6 to HEAD diff [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,121407098~3198227,124734943~18101328,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,223825249~6536345,241317308~18193230,260438339~12248801,296242351~493439,301341353~9191315,318433435~4174606,322615554~4664316,331638058~718760,334588993~36612024,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:09.503 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,121407098~3198227,124734943~18101328,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,223825249~6536345,241317308~18193230,260438339~12248801,296242351~493439,301341353~9191315,318433435~4174606,322615554~4664316,331638058~718760,334588993~36612024,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:09.509 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,102437886~3730024,121407098~3198227,124734943~18101328,143078160~20324185,164857432~12831533,181120288~6267164,201478817~19213323,221470799~1383607,223825249~6536345,241317308~18193230,260046848~12640292,296242351~493439,301341353~9191315,314572800~12707070,331434830~39766187,373441908~5818494,381681664~32125072] 2021-07-31T02:29:09.509 INFO:tasks.workunit.client.0.smithi045.stdout:from 7 to HEAD diff [872289~20735768,24372582~17941837,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,121407098~3198227,124734943~18101328,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,225052374~5309220,241317308~18193230,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,318433435~4174606,322615554~4664316,334588993~36612024,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:09.510 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~20735768,24372582~17941837,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,121407098~3198227,124734943~18101328,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,225052374~5309220,241317308~18193230,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,318433435~4174606,322615554~4664316,334588993~36612024,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:09.515 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~20735768,24372582~17941837,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,102437886~3730024,121407098~3198227,124734943~18101328,143078160~20324185,164857432~12831533,181120288~6267164,201478817~19213323,221470799~1383607,225052374~5309220,241317308~18193230,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,314572800~12707070,332356818~38844199,373441908~5818494,381681664~32125072] 2021-07-31T02:29:09.515 INFO:tasks.workunit.client.0.smithi045.stdout:from 8 to HEAD diff [872289~20735768,24372582~17941837,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~16816169,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,225052374~5309220,242763340~3666547,248659714~10850824,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,334588993~23788434,359181831~12019186,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:09.515 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~20735768,24372582~17941837,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~16816169,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,225052374~5309220,242763340~3666547,248659714~10850824,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,334588993~23788434,359181831~12019186,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:09.521 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~20735768,24372582~17941837,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,121407098~3198227,126020102~16816169,143078160~20324185,164857432~12831533,181120288~6267164,201478817~19213323,221470799~1383607,225052374~5309220,242763340~3666547,248659714~10850824,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,332356818~26020609,359181831~12019186,373441908~5818494,381820951~31985785] 2021-07-31T02:29:09.521 INFO:tasks.workunit.client.0.smithi045.stdout:from 9 to HEAD diff [1823868~19784189,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143078160~20324185,167792807~9896158,181972347~5415105,202180755~18511385,221470799~1383607,225052374~5309220,242763340~3666547,248659714~10850824,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,334588993~15115237,351585344~6792083,359181831~12019186,373441908~5818494,381820951~21159950,404169452~1052236,406364931~1096263,408916890~4889846] 2021-07-31T02:29:09.521 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [1823868~19784189,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143078160~20324185,167792807~9896158,181972347~5415105,202180755~18511385,221470799~1383607,225052374~5309220,242763340~3666547,248659714~10850824,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,334588993~15115237,351585344~6792083,359181831~12019186,373441908~5818494,381820951~21159950,404169452~1052236,406364931~1096263,408916890~4889846] 2021-07-31T02:29:09.527 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [1823868~19784189,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143078160~20324185,167792807~9896158,181120288~6267164,202180755~18511385,221470799~1383607,225052374~5309220,242763340~3666547,248659714~10850824,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,332356818~17347412,351585344~6792083,359181831~12019186,373441908~5818494,381820951~21159950,404169452~1052236,406364931~1096263,408916890~4889846] 2021-07-31T02:29:09.527 INFO:tasks.workunit.client.0.smithi045.stdout:from 10 to HEAD diff [1823868~19784189,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143078160~11521808,154729104~8673241,167792807~9896158,181972347~5415105,202180755~18511385,221470799~1383607,225052374~5309220,242763340~3666547,248682238~10828300,264698001~7989139,296242351~493439,301341353~9191315,334588993~15115237,351585344~6792083,359181831~12019186,373441908~5818494,381820951~21159950,404169452~1052236,406364931~1096263,408916890~4889846] 2021-07-31T02:29:09.527 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [1823868~19784189,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143078160~11521808,154729104~8673241,167792807~9896158,181972347~5415105,202180755~18511385,221470799~1383607,225052374~5309220,242763340~3666547,248682238~10828300,264698001~7989139,296242351~493439,301341353~9191315,334588993~15115237,351585344~6792083,359181831~12019186,373441908~5818494,381820951~21159950,404169452~1052236,406364931~1096263,408916890~4889846] 2021-07-31T02:29:09.533 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [1823868~19784189,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143078160~11521808,154729104~8673241,167792807~9896158,181120288~6267164,202180755~18511385,221470799~1383607,225052374~5309220,242763340~3666547,248682238~10828300,264698001~7989139,296242351~493439,301341353~9191315,332356818~17347412,351585344~6792083,359181831~12019186,373441908~5818494,381820951~21159950,404169452~1052236,406364931~1096263,408916890~4889846] 2021-07-31T02:29:09.533 INFO:tasks.workunit.client.0.smithi045.stdout:from 11 to HEAD diff [1823868~12981357,17882092~3725965,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143078160~11521808,154729104~8673241,168673724~9015241,181972347~5415105,202180755~18511385,221470799~1383607,225052374~5309220,248682238~10828300,264698001~7989139,296242351~493439,301341353~9191315,334588993~15115237,351585344~6792083,359181831~8323415,373441908~5818494,383830156~6295455,390462350~12518551,404169452~1052236,406364931~1096263] 2021-07-31T02:29:09.534 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [1823868~12981357,17882092~3725965,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143078160~11521808,154729104~8673241,168673724~9015241,181972347~5415105,202180755~18511385,221470799~1383607,225052374~5309220,248682238~10828300,264698001~7989139,296242351~493439,301341353~9191315,334588993~15115237,351585344~6792083,359181831~8323415,373441908~5818494,383830156~6295455,390462350~12518551,404169452~1052236,406364931~1096263] 2021-07-31T02:29:09.539 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [1823868~12981357,17882092~3725965,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,83886080~4661315,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143078160~11521808,154729104~8673241,168673724~9015241,181120288~6267164,202180755~18511385,221470799~1383607,225052374~5309220,248682238~10828300,264698001~7989139,296242351~493439,301341353~9191315,332356818~17347412,351585344~6792083,359181831~8323415,373441908~5818494,383830156~6295455,390462350~12518551,404169452~1052236,406364931~1096263] 2021-07-31T02:29:09.539 INFO:tasks.workunit.client.0.smithi045.stdout:from 12 to HEAD diff [1823868~12936934,17882092~3725965,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,71563215~1562492,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143369911~11230057,154729104~8673241,169186192~8502773,186487148~881866,202180755~18511385,221470799~1383607,225052374~5309220,248682238~10828300,264698001~7989139,301341353~9191315,335055096~14649134,351585344~6792083,359181831~8323415,375687458~1062344,383830156~6295455,390462350~12518551,406364931~1096263] 2021-07-31T02:29:09.539 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [1823868~12936934,17882092~3725965,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,71563215~1562492,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143369911~11230057,154729104~8673241,169186192~8502773,186487148~881866,202180755~18511385,221470799~1383607,225052374~5309220,248682238~10828300,264698001~7989139,301341353~9191315,335055096~14649134,351585344~6792083,359181831~8323415,375687458~1062344,383830156~6295455,390462350~12518551,406364931~1096263] 2021-07-31T02:29:09.544 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [1823868~12936934,17882092~3725965,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,71563215~1562492,83886080~4661315,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143369911~11230057,154729104~8673241,169186192~8502773,186487148~881866,202180755~18511385,221470799~1383607,225052374~5309220,248682238~10828300,264698001~7989139,301341353~9191315,335055096~14649134,351585344~6792083,359181831~8323415,375687458~1062344,383830156~6295455,390462350~12518551,406364931~1096263] 2021-07-31T02:29:09.545 INFO:tasks.workunit.client.0.smithi045.stdout:from 13 to HEAD diff [1823868~12936934,17882092~3725965,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,71563215~1562492,86327888~2219507,89929518~1882381,99285857~789121,121407098~3198227,135550801~7285470,143369911~11230057,154729104~8673241,169186192~7553729,186487148~881866,207917770~12774370,221470799~1383607,225052374~5309220,248682238~8485363,264698001~7989139,301341353~9191315,335055096~14649134,351585344~6792083,359181831~8323415,375687458~1062344,383830156~6295455,391467418~11513483,406364931~1096263] 2021-07-31T02:29:09.545 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [1823868~12936934,17882092~3725965,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,71563215~1562492,86327888~2219507,89929518~1882381,99285857~789121,121407098~3198227,135550801~7285470,143369911~11230057,154729104~8673241,169186192~7553729,186487148~881866,207917770~12774370,221470799~1383607,225052374~5309220,248682238~8485363,264698001~7989139,301341353~9191315,335055096~14649134,351585344~6792083,359181831~8323415,375687458~1062344,383830156~6295455,391467418~11513483,406364931~1096263] 2021-07-31T02:29:09.550 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [1823868~12936934,17882092~3725965,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,71563215~1562492,83886080~4661315,89929518~1882381,99285857~789121,121407098~3198227,135550801~7285470,143369911~11230057,154729104~8673241,169186192~7553729,186487148~881866,207917770~12774370,221470799~1383607,225052374~5309220,248682238~8485363,264698001~7989139,301341353~9191315,335055096~14649134,351585344~6792083,359181831~8323415,375687458~1062344,383830156~6295455,391467418~11513483,406364931~1096263] 2021-07-31T02:29:09.551 INFO:tasks.workunit.client.0.smithi045.stdout:from 14 to HEAD diff [1823868~12936934,17882092~3725965,24372582~14188618,53817318~2949139,60609981~5053700,71563215~1562492,86327888~2219507,89929518~1882381,99285857~789121,121407098~3198227,135550801~7285470,147626899~6973069,157325666~6076679,169186192~7553729,186487148~881866,207946866~2990158,213575464~7116676,221470799~1383607,225052374~5309220,248682238~8485363,264809820~7877320,305074438~5458230,335055096~14649134,351585344~6792083,359181831~8323415,375687458~1062344,383830156~6295455,391467418~11513483,406364931~1096263] 2021-07-31T02:29:09.551 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [1823868~12936934,17882092~3725965,24372582~14188618,53817318~2949139,60609981~5053700,71563215~1562492,86327888~2219507,89929518~1882381,99285857~789121,121407098~3198227,135550801~7285470,147626899~6973069,157325666~6076679,169186192~7553729,186487148~881866,207946866~2990158,213575464~7116676,221470799~1383607,225052374~5309220,248682238~8485363,264809820~7877320,305074438~5458230,335055096~14649134,351585344~6792083,359181831~8323415,375687458~1062344,383830156~6295455,391467418~11513483,406364931~1096263] 2021-07-31T02:29:09.556 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [1823868~12936934,17882092~3725965,24372582~14188618,53817318~2949139,60609981~5053700,71563215~1562492,83886080~4661315,89929518~1882381,99285857~789121,121407098~3198227,135550801~7285470,147626899~6973069,157325666~6076679,169186192~7553729,186487148~881866,207946866~2990158,213575464~7116676,221470799~1383607,225052374~5309220,248682238~8485363,264809820~7877320,305074438~5458230,335055096~14649134,351585344~6792083,359181831~8323415,375687458~1062344,383830156~6295455,391467418~11513483,406364931~1096263] 2021-07-31T02:29:09.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:09 smithi154 conmon[34553]: cluster 2021-07-31T02:29:08.397093+0000 mon.a (mon.0) 2093 : cluster [DBG] osdmap e1168: 8 total, 8 up, 8 in 2021-07-31T02:29:09.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:09 smithi154 conmon[34553]: cluster 2021-07-31T02:29:08.398670+0000 mgr.x (mgr.24542) 1794 : cluster [DBG] pgmap v2272: 41 pgs: 41 active+clean; 885 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail 2021-07-31T02:29:09.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:09 smithi045 conmon[39718]: cluster 2021-07-31T02:29:08. 2021-07-31T02:29:09.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:09 smithi045 conmon[39718]: 397093+0000 mon.a (mon.0) 2093 : cluster [DBG] osdmap e1168: 8 total, 8 up, 8 in 2021-07-31T02:29:09.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:09 smithi045 conmon[39718]: cluster 2021-07 2021-07-31T02:29:09.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:09 smithi045 conmon[39718]: -31T02:29:08.398670+0000 mgr.x (mgr.24542) 2021-07-31T02:29:09.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:09 smithi045 conmon[39718]: 1794 : cluster [DBG] pgmap v2272: 41 pgs: 41 active+clean; 885 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail 2021-07-31T02:29:09.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:09 smithi045 conmon[33655]: cluster 2021-07-31T02:29:08.397093+0000 mon.a (mon.0) 2021-07-31T02:29:09.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:09 smithi045 conmon[33655]: 2093 : cluster [DBG] osdmap e1168: 8 total, 8 up, 8 in 2021-07-31T02:29:09.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:09 smithi045 conmon[33655]: cluster 2021-07-31T02:29:08.398670+0000 mgr.x (mgr.24542) 1794 : cluster [DBG] pgmap v2272: 41 pgs: 41 active+clean; 885 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail 2021-07-31T02:29:10.421 INFO:tasks.workunit.client.0.smithi045.stdout:from 0 to HEAD diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~22526082,86327888~2219507,89659479~3999082,97713673~8454237,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,236085312~4054340,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:10.421 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~22526082,86327888~2219507,89659479~3999082,97713673~8454237,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,236085312~4054340,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:10.484 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~43334606,46137344~10629113,59300692~1156219,60609981~22526082,83886080~9772481,96468992~14988213,116424046~70963406,188743680~9773402,201326592~33497107,234881024~24629514,260046848~20273171,289406976~21125692,314572800~57820771,373441908~5818494,381681664~32125072] 2021-07-31T02:29:10.484 INFO:tasks.workunit.client.0.smithi045.stdout:from 1 to HEAD diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89659479~3999082,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:10.484 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89659479~3999082,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:10.491 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~43334606,46137344~10629113,59300692~1156219,60609981~5053700,66056524~17079539,83886080~9772481,99285857~789121,102437886~9019319,116424046~70963406,191717187~6799895,201326592~33497107,240139652~19370886,260046848~20273171,289406976~21125692,314572800~57820771,373441908~5818494,381681664~32125072] 2021-07-31T02:29:10.491 INFO:tasks.workunit.client.0.smithi045.stdout:from 2 to HEAD diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89659479~3999082,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:10.492 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89659479~3999082,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:10.497 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~43334606,46137344~10629113,59300692~1156219,60609981~5053700,66056524~17079539,83886080~9772481,99285857~789121,102437886~9019319,116424046~70963406,191717187~6799895,201478817~33344882,240139652~19370886,260046848~20273171,291252461~19280207,314572800~12707070,327653500~3214988,331434830~40958741,373441908~5818494,381681664~32125072] 2021-07-31T02:29:10.498 INFO:tasks.workunit.client.0.smithi045.stdout:from 3 to HEAD diff [677713~20930344,24372582~18962024,48159474~2619597,53817318~2949139,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:10.498 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~20930344,24372582~18962024,48159474~2619597,53817318~2949139,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:10.503 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~21608057,24372582~18962024,48159474~2619597,53817318~2949139,59300692~1156219,60609981~5053700,66056524~17079539,83886080~4661315,89752687~3175987,99285857~789121,102437886~9019319,116424046~70963406,201478817~21375589,223825249~10998450,240139652~19370886,260046848~20273171,291252461~19280207,314572800~12707070,327653500~3214988,331434830~40958741,373441908~5818494,381681664~32125072] 2021-07-31T02:29:10.504 INFO:tasks.workunit.client.0.smithi045.stdout:from 4 to HEAD diff [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~20324185,167531438~13588850,181972347~5415105,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,292264750~18267918,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:10.504 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~20324185,167531438~13588850,181972347~5415105,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,292264750~18267918,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:10.509 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,102437886~9019319,116424046~8181279,124734943~38667402,164857432~22530020,201478817~21375589,223825249~10998450,240139652~19370886,260046848~12640292,275231912~5088107,292264750~18267918,314572800~12707070,327653500~3214988,331434830~40958741,373441908~5818494,381681664~32125072] 2021-07-31T02:29:10.510 INFO:tasks.workunit.client.0.smithi045.stdout:from 5 to HEAD diff [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,292264750~4687170,301341353~9191315,318433435~4174606,322615554~4664316,331638058~718760,334588993~36612024,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:10.510 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,292264750~4687170,301341353~9191315,318433435~4174606,322615554~4664316,331638058~718760,334588993~36612024,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:10.515 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,102437886~9019319,116424046~8181279,124734943~38667402,164857432~12831533,181120288~6267164,201478817~19213323,221470799~1383607,223825249~10998450,241317308~18193230,260046848~12640292,275231912~5088107,292264750~4687170,301341353~9191315,314572800~12707070,331434830~39766187,373441908~5818494,381681664~32125072] 2021-07-31T02:29:10.516 INFO:tasks.workunit.client.0.smithi045.stdout:from 6 to HEAD diff [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,121407098~3198227,124734943~18101328,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,223825249~6536345,241317308~18193230,260438339~12248801,296242351~493439,301341353~9191315,318433435~4174606,322615554~4664316,331638058~718760,334588993~36612024,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:10.516 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,121407098~3198227,124734943~18101328,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,223825249~6536345,241317308~18193230,260438339~12248801,296242351~493439,301341353~9191315,318433435~4174606,322615554~4664316,331638058~718760,334588993~36612024,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:10.521 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,102437886~3730024,121407098~3198227,124734943~18101328,143078160~20324185,164857432~12831533,181120288~6267164,201478817~19213323,221470799~1383607,223825249~6536345,241317308~18193230,260046848~12640292,296242351~493439,301341353~9191315,314572800~12707070,331434830~39766187,373441908~5818494,381681664~32125072] 2021-07-31T02:29:10.521 INFO:tasks.workunit.client.0.smithi045.stdout:from 7 to HEAD diff [872289~20735768,24372582~17941837,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,121407098~3198227,124734943~18101328,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,225052374~5309220,241317308~18193230,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,318433435~4174606,322615554~4664316,334588993~36612024,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:10.522 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~20735768,24372582~17941837,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,121407098~3198227,124734943~18101328,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,225052374~5309220,241317308~18193230,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,318433435~4174606,322615554~4664316,334588993~36612024,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:10.527 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~20735768,24372582~17941837,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,102437886~3730024,121407098~3198227,124734943~18101328,143078160~20324185,164857432~12831533,181120288~6267164,201478817~19213323,221470799~1383607,225052374~5309220,241317308~18193230,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,314572800~12707070,332356818~38844199,373441908~5818494,381681664~32125072] 2021-07-31T02:29:10.527 INFO:tasks.workunit.client.0.smithi045.stdout:from 8 to HEAD diff [872289~20735768,24372582~17941837,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~16816169,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,225052374~5309220,242763340~3666547,248659714~10850824,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,334588993~23788434,359181831~12019186,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:10.527 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~20735768,24372582~17941837,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~16816169,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,225052374~5309220,242763340~3666547,248659714~10850824,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,334588993~23788434,359181831~12019186,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:10.533 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~20735768,24372582~17941837,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,121407098~3198227,126020102~16816169,143078160~20324185,164857432~12831533,181120288~6267164,201478817~19213323,221470799~1383607,225052374~5309220,242763340~3666547,248659714~10850824,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,332356818~26020609,359181831~12019186,373441908~5818494,381820951~31985785] 2021-07-31T02:29:10.533 INFO:tasks.workunit.client.0.smithi045.stdout:from 9 to HEAD diff [1823868~19784189,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143078160~20324185,167792807~9896158,181972347~5415105,202180755~18511385,221470799~1383607,225052374~5309220,242763340~3666547,248659714~10850824,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,334588993~15115237,351585344~6792083,359181831~12019186,373441908~5818494,381820951~21159950,404169452~1052236,406364931~1096263,408916890~4889846] 2021-07-31T02:29:10.533 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [1823868~19784189,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143078160~20324185,167792807~9896158,181972347~5415105,202180755~18511385,221470799~1383607,225052374~5309220,242763340~3666547,248659714~10850824,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,334588993~15115237,351585344~6792083,359181831~12019186,373441908~5818494,381820951~21159950,404169452~1052236,406364931~1096263,408916890~4889846] 2021-07-31T02:29:10.538 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [1823868~19784189,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143078160~20324185,167792807~9896158,181120288~6267164,202180755~18511385,221470799~1383607,225052374~5309220,242763340~3666547,248659714~10850824,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,332356818~17347412,351585344~6792083,359181831~12019186,373441908~5818494,381820951~21159950,404169452~1052236,406364931~1096263,408916890~4889846] 2021-07-31T02:29:10.539 INFO:tasks.workunit.client.0.smithi045.stdout:from 10 to HEAD diff [1823868~19784189,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143078160~11521808,154729104~8673241,167792807~9896158,181972347~5415105,202180755~18511385,221470799~1383607,225052374~5309220,242763340~3666547,248682238~10828300,264698001~7989139,296242351~493439,301341353~9191315,334588993~15115237,351585344~6792083,359181831~12019186,373441908~5818494,381820951~21159950,404169452~1052236,406364931~1096263,408916890~4889846] 2021-07-31T02:29:10.539 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [1823868~19784189,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143078160~11521808,154729104~8673241,167792807~9896158,181972347~5415105,202180755~18511385,221470799~1383607,225052374~5309220,242763340~3666547,248682238~10828300,264698001~7989139,296242351~493439,301341353~9191315,334588993~15115237,351585344~6792083,359181831~12019186,373441908~5818494,381820951~21159950,404169452~1052236,406364931~1096263,408916890~4889846] 2021-07-31T02:29:10.544 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [1823868~19784189,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143078160~11521808,154729104~8673241,167792807~9896158,181120288~6267164,202180755~18511385,221470799~1383607,225052374~5309220,242763340~3666547,248682238~10828300,264698001~7989139,296242351~493439,301341353~9191315,332356818~17347412,351585344~6792083,359181831~12019186,373441908~5818494,381820951~21159950,404169452~1052236,406364931~1096263,408916890~4889846] 2021-07-31T02:29:10.544 INFO:tasks.workunit.client.0.smithi045.stdout:from 11 to HEAD diff [1823868~12981357,17882092~3725965,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143078160~11521808,154729104~8673241,168673724~9015241,181972347~5415105,202180755~18511385,221470799~1383607,225052374~5309220,248682238~10828300,264698001~7989139,296242351~493439,301341353~9191315,334588993~15115237,351585344~6792083,359181831~8323415,373441908~5818494,383830156~6295455,390462350~12518551,404169452~1052236,406364931~1096263] 2021-07-31T02:29:10.545 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [1823868~12981357,17882092~3725965,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143078160~11521808,154729104~8673241,168673724~9015241,181972347~5415105,202180755~18511385,221470799~1383607,225052374~5309220,248682238~10828300,264698001~7989139,296242351~493439,301341353~9191315,334588993~15115237,351585344~6792083,359181831~8323415,373441908~5818494,383830156~6295455,390462350~12518551,404169452~1052236,406364931~1096263] 2021-07-31T02:29:10.550 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [1823868~12981357,17882092~3725965,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,83886080~4661315,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143078160~11521808,154729104~8673241,168673724~9015241,181120288~6267164,202180755~18511385,221470799~1383607,225052374~5309220,248682238~10828300,264698001~7989139,296242351~493439,301341353~9191315,332356818~17347412,351585344~6792083,359181831~8323415,373441908~5818494,383830156~6295455,390462350~12518551,404169452~1052236,406364931~1096263] 2021-07-31T02:29:10.550 INFO:tasks.workunit.client.0.smithi045.stdout:from 12 to HEAD diff [1823868~12936934,17882092~3725965,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,71563215~1562492,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143369911~11230057,154729104~8673241,169186192~8502773,186487148~881866,202180755~18511385,221470799~1383607,225052374~5309220,248682238~10828300,264698001~7989139,301341353~9191315,335055096~14649134,351585344~6792083,359181831~8323415,375687458~1062344,383830156~6295455,390462350~12518551,406364931~1096263] 2021-07-31T02:29:10.550 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [1823868~12936934,17882092~3725965,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,71563215~1562492,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143369911~11230057,154729104~8673241,169186192~8502773,186487148~881866,202180755~18511385,221470799~1383607,225052374~5309220,248682238~10828300,264698001~7989139,301341353~9191315,335055096~14649134,351585344~6792083,359181831~8323415,375687458~1062344,383830156~6295455,390462350~12518551,406364931~1096263] 2021-07-31T02:29:10.556 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [1823868~12936934,17882092~3725965,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,71563215~1562492,83886080~4661315,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143369911~11230057,154729104~8673241,169186192~8502773,186487148~881866,202180755~18511385,221470799~1383607,225052374~5309220,248682238~10828300,264698001~7989139,301341353~9191315,335055096~14649134,351585344~6792083,359181831~8323415,375687458~1062344,383830156~6295455,390462350~12518551,406364931~1096263] 2021-07-31T02:29:10.556 INFO:tasks.workunit.client.0.smithi045.stdout:from 13 to HEAD diff [1823868~12936934,17882092~3725965,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,71563215~1562492,86327888~2219507,89929518~1882381,99285857~789121,121407098~3198227,135550801~7285470,143369911~11230057,154729104~8673241,169186192~7553729,186487148~881866,207917770~12774370,221470799~1383607,225052374~5309220,248682238~8485363,264698001~7989139,301341353~9191315,335055096~14649134,351585344~6792083,359181831~8323415,375687458~1062344,383830156~6295455,391467418~11513483,406364931~1096263] 2021-07-31T02:29:10.556 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [1823868~12936934,17882092~3725965,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,71563215~1562492,86327888~2219507,89929518~1882381,99285857~789121,121407098~3198227,135550801~7285470,143369911~11230057,154729104~8673241,169186192~7553729,186487148~881866,207917770~12774370,221470799~1383607,225052374~5309220,248682238~8485363,264698001~7989139,301341353~9191315,335055096~14649134,351585344~6792083,359181831~8323415,375687458~1062344,383830156~6295455,391467418~11513483,406364931~1096263] 2021-07-31T02:29:10.561 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [1823868~12936934,17882092~3725965,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,71563215~1562492,83886080~4661315,89929518~1882381,99285857~789121,121407098~3198227,135550801~7285470,143369911~11230057,154729104~8673241,169186192~7553729,186487148~881866,207917770~12774370,221470799~1383607,225052374~5309220,248682238~8485363,264698001~7989139,301341353~9191315,335055096~14649134,351585344~6792083,359181831~8323415,375687458~1062344,383830156~6295455,391467418~11513483,406364931~1096263] 2021-07-31T02:29:10.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:10 smithi154 conmon[34553]: cluster 2021-07-31T02:29:09.403082+0000 mon.a (mon.0) 2094 : cluster 2021-07-31T02:29:10.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:10 smithi154 conmon[34553]: [DBG] osdmap e1169: 8 total, 8 up, 8 in 2021-07-31T02:29:10.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:10 smithi045 conmon[33655]: cluster 2021-07-31T02:29:09.403082+0000 mon.a (mon 2021-07-31T02:29:10.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:10 smithi045 conmon[33655]: .0) 2094 : cluster [DBG] osdmap e1169: 8 total, 8 up, 8 in 2021-07-31T02:29:10.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:10 smithi045 conmon[39718]: cluster 2021-07-31T02:29:09.403082+0000 mon.a (mon. 2021-07-31T02:29:10.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:10 smithi045 conmon[39718]: 0) 2094 : cluster [DBG] osdmap e1169: 8 total, 8 up, 8 in 2021-07-31T02:29:11.468 INFO:tasks.workunit.client.0.smithi045.stdout:from 0 to HEAD diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~22526082,86327888~2219507,89659479~3999082,97713673~8454237,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,236085312~4054340,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:11.469 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~22526082,86327888~2219507,89659479~3999082,97713673~8454237,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,236085312~4054340,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:11.519 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~43334606,46137344~10629113,59300692~1156219,60609981~22526082,83886080~9772481,96468992~14988213,116424046~70963406,188743680~9773402,201326592~33497107,234881024~24629514,260046848~20273171,289406976~21125692,314572800~57820771,373441908~5818494,381681664~32125072] 2021-07-31T02:29:11.519 INFO:tasks.workunit.client.0.smithi045.stdout:from 1 to HEAD diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89659479~3999082,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:11.519 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89659479~3999082,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:11.544 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~43334606,46137344~10629113,59300692~1156219,60609981~5053700,66056524~17079539,83886080~9772481,99285857~789121,102437886~9019319,116424046~70963406,191717187~6799895,201326592~33497107,240139652~19370886,260046848~20273171,289406976~21125692,314572800~57820771,373441908~5818494,381681664~32125072] 2021-07-31T02:29:11.544 INFO:tasks.workunit.client.0.smithi045.stdout:from 2 to HEAD diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89659479~3999082,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:11.545 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89659479~3999082,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:11.550 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~43334606,46137344~10629113,59300692~1156219,60609981~5053700,66056524~17079539,83886080~9772481,99285857~789121,102437886~9019319,116424046~70963406,191717187~6799895,201478817~33344882,240139652~19370886,260046848~20273171,291252461~19280207,314572800~12707070,327653500~3214988,331434830~40958741,373441908~5818494,381681664~32125072] 2021-07-31T02:29:11.554 INFO:tasks.workunit.client.0.smithi045.stdout:from 3 to HEAD diff [677713~20930344,24372582~18962024,48159474~2619597,53817318~2949139,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:11.554 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~20930344,24372582~18962024,48159474~2619597,53817318~2949139,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:11.557 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~21608057,24372582~18962024,48159474~2619597,53817318~2949139,59300692~1156219,60609981~5053700,66056524~17079539,83886080~4661315,89752687~3175987,99285857~789121,102437886~9019319,116424046~70963406,201478817~21375589,223825249~10998450,240139652~19370886,260046848~20273171,291252461~19280207,314572800~12707070,327653500~3214988,331434830~40958741,373441908~5818494,381681664~32125072] 2021-07-31T02:29:11.558 INFO:tasks.workunit.client.0.smithi045.stdout:from 4 to HEAD diff [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~20324185,167531438~13588850,181972347~5415105,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,292264750~18267918,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:11.558 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~20324185,167531438~13588850,181972347~5415105,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,292264750~18267918,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:11.563 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,102437886~9019319,116424046~8181279,124734943~38667402,164857432~22530020,201478817~21375589,223825249~10998450,240139652~19370886,260046848~12640292,275231912~5088107,292264750~18267918,314572800~12707070,327653500~3214988,331434830~40958741,373441908~5818494,381681664~32125072] 2021-07-31T02:29:11.563 INFO:tasks.workunit.client.0.smithi045.stdout:from 5 to HEAD diff [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,292264750~4687170,301341353~9191315,318433435~4174606,322615554~4664316,331638058~718760,334588993~36612024,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:11.563 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,292264750~4687170,301341353~9191315,318433435~4174606,322615554~4664316,331638058~718760,334588993~36612024,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:11.568 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,102437886~9019319,116424046~8181279,124734943~38667402,164857432~12831533,181120288~6267164,201478817~19213323,221470799~1383607,223825249~10998450,241317308~18193230,260046848~12640292,275231912~5088107,292264750~4687170,301341353~9191315,314572800~12707070,331434830~39766187,373441908~5818494,381681664~32125072] 2021-07-31T02:29:11.568 INFO:tasks.workunit.client.0.smithi045.stdout:from 6 to HEAD diff [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,121407098~3198227,124734943~18101328,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,223825249~6536345,241317308~18193230,260438339~12248801,296242351~493439,301341353~9191315,318433435~4174606,322615554~4664316,331638058~718760,334588993~36612024,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:11.569 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,121407098~3198227,124734943~18101328,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,223825249~6536345,241317308~18193230,260438339~12248801,296242351~493439,301341353~9191315,318433435~4174606,322615554~4664316,331638058~718760,334588993~36612024,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:11.574 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,102437886~3730024,121407098~3198227,124734943~18101328,143078160~20324185,164857432~12831533,181120288~6267164,201478817~19213323,221470799~1383607,223825249~6536345,241317308~18193230,260046848~12640292,296242351~493439,301341353~9191315,314572800~12707070,331434830~39766187,373441908~5818494,381681664~32125072] 2021-07-31T02:29:11.574 INFO:tasks.workunit.client.0.smithi045.stdout:from 7 to HEAD diff [872289~20735768,24372582~17941837,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,121407098~3198227,124734943~18101328,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,225052374~5309220,241317308~18193230,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,318433435~4174606,322615554~4664316,334588993~36612024,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:11.574 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~20735768,24372582~17941837,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,121407098~3198227,124734943~18101328,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,225052374~5309220,241317308~18193230,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,318433435~4174606,322615554~4664316,334588993~36612024,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:11.580 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~20735768,24372582~17941837,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,102437886~3730024,121407098~3198227,124734943~18101328,143078160~20324185,164857432~12831533,181120288~6267164,201478817~19213323,221470799~1383607,225052374~5309220,241317308~18193230,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,314572800~12707070,332356818~38844199,373441908~5818494,381681664~32125072] 2021-07-31T02:29:11.580 INFO:tasks.workunit.client.0.smithi045.stdout:from 8 to HEAD diff [872289~20735768,24372582~17941837,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~16816169,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,225052374~5309220,242763340~3666547,248659714~10850824,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,334588993~23788434,359181831~12019186,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:11.580 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~20735768,24372582~17941837,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~16816169,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,225052374~5309220,242763340~3666547,248659714~10850824,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,334588993~23788434,359181831~12019186,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:11.586 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~20735768,24372582~17941837,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,121407098~3198227,126020102~16816169,143078160~20324185,164857432~12831533,181120288~6267164,201478817~19213323,221470799~1383607,225052374~5309220,242763340~3666547,248659714~10850824,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,332356818~26020609,359181831~12019186,373441908~5818494,381820951~31985785] 2021-07-31T02:29:11.586 INFO:tasks.workunit.client.0.smithi045.stdout:from 9 to HEAD diff [1823868~19784189,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143078160~20324185,167792807~9896158,181972347~5415105,202180755~18511385,221470799~1383607,225052374~5309220,242763340~3666547,248659714~10850824,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,334588993~15115237,351585344~6792083,359181831~12019186,373441908~5818494,381820951~21159950,404169452~1052236,406364931~1096263,408916890~4889846] 2021-07-31T02:29:11.586 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [1823868~19784189,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143078160~20324185,167792807~9896158,181972347~5415105,202180755~18511385,221470799~1383607,225052374~5309220,242763340~3666547,248659714~10850824,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,334588993~15115237,351585344~6792083,359181831~12019186,373441908~5818494,381820951~21159950,404169452~1052236,406364931~1096263,408916890~4889846] 2021-07-31T02:29:11.592 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [1823868~19784189,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143078160~20324185,167792807~9896158,181120288~6267164,202180755~18511385,221470799~1383607,225052374~5309220,242763340~3666547,248659714~10850824,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,332356818~17347412,351585344~6792083,359181831~12019186,373441908~5818494,381820951~21159950,404169452~1052236,406364931~1096263,408916890~4889846] 2021-07-31T02:29:11.592 INFO:tasks.workunit.client.0.smithi045.stdout:from 10 to HEAD diff [1823868~19784189,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143078160~11521808,154729104~8673241,167792807~9896158,181972347~5415105,202180755~18511385,221470799~1383607,225052374~5309220,242763340~3666547,248682238~10828300,264698001~7989139,296242351~493439,301341353~9191315,334588993~15115237,351585344~6792083,359181831~12019186,373441908~5818494,381820951~21159950,404169452~1052236,406364931~1096263,408916890~4889846] 2021-07-31T02:29:11.592 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [1823868~19784189,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143078160~11521808,154729104~8673241,167792807~9896158,181972347~5415105,202180755~18511385,221470799~1383607,225052374~5309220,242763340~3666547,248682238~10828300,264698001~7989139,296242351~493439,301341353~9191315,334588993~15115237,351585344~6792083,359181831~12019186,373441908~5818494,381820951~21159950,404169452~1052236,406364931~1096263,408916890~4889846] 2021-07-31T02:29:11.599 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [1823868~19784189,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143078160~11521808,154729104~8673241,167792807~9896158,181120288~6267164,202180755~18511385,221470799~1383607,225052374~5309220,242763340~3666547,248682238~10828300,264698001~7989139,296242351~493439,301341353~9191315,332356818~17347412,351585344~6792083,359181831~12019186,373441908~5818494,381820951~21159950,404169452~1052236,406364931~1096263,408916890~4889846] 2021-07-31T02:29:11.599 INFO:tasks.workunit.client.0.smithi045.stdout:from 11 to HEAD diff [1823868~12981357,17882092~3725965,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143078160~11521808,154729104~8673241,168673724~9015241,181972347~5415105,202180755~18511385,221470799~1383607,225052374~5309220,248682238~10828300,264698001~7989139,296242351~493439,301341353~9191315,334588993~15115237,351585344~6792083,359181831~8323415,373441908~5818494,383830156~6295455,390462350~12518551,404169452~1052236,406364931~1096263] 2021-07-31T02:29:11.599 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [1823868~12981357,17882092~3725965,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143078160~11521808,154729104~8673241,168673724~9015241,181972347~5415105,202180755~18511385,221470799~1383607,225052374~5309220,248682238~10828300,264698001~7989139,296242351~493439,301341353~9191315,334588993~15115237,351585344~6792083,359181831~8323415,373441908~5818494,383830156~6295455,390462350~12518551,404169452~1052236,406364931~1096263] 2021-07-31T02:29:11.605 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [1823868~12981357,17882092~3725965,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,83886080~4661315,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143078160~11521808,154729104~8673241,168673724~9015241,181120288~6267164,202180755~18511385,221470799~1383607,225052374~5309220,248682238~10828300,264698001~7989139,296242351~493439,301341353~9191315,332356818~17347412,351585344~6792083,359181831~8323415,373441908~5818494,383830156~6295455,390462350~12518551,404169452~1052236,406364931~1096263] 2021-07-31T02:29:11.606 INFO:tasks.workunit.client.0.smithi045.stdout:from 12 to HEAD diff [1823868~12936934,17882092~3725965,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,71563215~1562492,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143369911~11230057,154729104~8673241,169186192~8502773,186487148~881866,202180755~18511385,221470799~1383607,225052374~5309220,248682238~10828300,264698001~7989139,301341353~9191315,335055096~14649134,351585344~6792083,359181831~8323415,375687458~1062344,383830156~6295455,390462350~12518551,406364931~1096263] 2021-07-31T02:29:11.606 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [1823868~12936934,17882092~3725965,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,71563215~1562492,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143369911~11230057,154729104~8673241,169186192~8502773,186487148~881866,202180755~18511385,221470799~1383607,225052374~5309220,248682238~10828300,264698001~7989139,301341353~9191315,335055096~14649134,351585344~6792083,359181831~8323415,375687458~1062344,383830156~6295455,390462350~12518551,406364931~1096263] 2021-07-31T02:29:11.612 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [1823868~12936934,17882092~3725965,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,71563215~1562492,83886080~4661315,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143369911~11230057,154729104~8673241,169186192~8502773,186487148~881866,202180755~18511385,221470799~1383607,225052374~5309220,248682238~10828300,264698001~7989139,301341353~9191315,335055096~14649134,351585344~6792083,359181831~8323415,375687458~1062344,383830156~6295455,390462350~12518551,406364931~1096263] 2021-07-31T02:29:11.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:11 smithi154 conmon[34553]: cluster 2021-07-31T02:29:10.399255+0000 mgr.x (mgr.24542) 1795 : cluster [DBG] pgmap v2274: 41 pgs: 41 active+clean; 874 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 5.2 KiB/s rd, 1020 B/s wr, 5.83k op/s 2021-07-31T02:29:11.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:11 smithi154 conmon[34553]: cluster 2021-07-31T02:29:10.404619+0000 mon.a (mon.0) 2095 : cluster [DBG] osdmap e1170: 8 total, 8 up, 8 in 2021-07-31T02:29:11.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:11 smithi045 conmon[39718]: cluster 2021-07-31T02:29:10.399255+0000 mgr.x (mgr. 2021-07-31T02:29:11.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:11 smithi045 conmon[39718]: 24542) 1795 : cluster [DBG] pgmap v2274: 41 pgs: 41 active+clean; 874 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 5.2 KiB/s rd, 1020 B/s wr, 5.83k op/s 2021-07-31T02:29:11.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:11 smithi045 conmon[39718]: cluster 2021-07-31T 2021-07-31T02:29:11.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:11 smithi045 conmon[39718]: 02:29:10.404619+0000 mon.a (mon.0) 2095 : cluster [DBG] osdmap e1170: 8 total, 8 up, 8 in 2021-07-31T02:29:11.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:11 smithi045 conmon[33655]: cluster 2021-07-31T02:29:10.399255+0000 mgr.x (mgr.24542) 1795 : cluster [DBG] pgmap v2274: 41 pgs: 41 active+clean; 874 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 5.2 KiB/s rd, 1020 B/s wr, 5.83k op/s 2021-07-31T02:29:11.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:11 smithi045 conmon[33655]: 2021-07-31T02:29:11.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:11 smithi045 conmon[33655]: cluster 2021-07-31T02:29:10.404619+0000 mon.a (mon.0) 2095 : cluster [DBG] osdmap e1170: 8 total, 8 up, 8 in 2021-07-31T02:29:11.887 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:29:12.166 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:29:12.166 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 44m ago 44m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:29:12.166 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 41m ago 44m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:29:12.167 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 41m ago 48m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:29:12.167 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 44m ago 50m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:29:12.168 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 44m ago 50m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:29:12.168 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 41m ago 48m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:29:12.168 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 44m ago 48m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:29:12.169 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 44m ago 45m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:29:12.169 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 41m ago 45m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:29:12.169 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 44m ago 47m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:29:12.170 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 44m ago 47m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:29:12.170 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 44m ago 47m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:29:12.171 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 44m ago 46m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:29:12.171 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 41m ago 46m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:29:12.171 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 41m ago 46m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:29:12.172 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 41m ago 45m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:29:12.172 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 41m ago 45m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:29:12.172 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 41m ago 45m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:29:12.292 INFO:tasks.workunit.client.0.smithi045.stdout:from 0 to HEAD diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~22526082,86327888~2219507,89659479~3999082,97713673~8454237,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,236085312~4054340,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:12.293 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~22526082,86327888~2219507,89659479~3999082,97713673~8454237,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,236085312~4054340,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:12.347 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~43334606,46137344~10629113,59300692~1156219,60609981~22526082,83886080~9772481,96468992~14988213,116424046~70963406,188743680~9773402,201326592~33497107,234881024~24629514,260046848~20273171,289406976~21125692,314572800~57820771,373441908~5818494,381681664~32125072] 2021-07-31T02:29:12.348 INFO:tasks.workunit.client.0.smithi045.stdout:from 1 to HEAD diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89659479~3999082,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:12.348 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89659479~3999082,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:12.373 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~43334606,46137344~10629113,59300692~1156219,60609981~5053700,66056524~17079539,83886080~9772481,99285857~789121,102437886~9019319,116424046~70963406,191717187~6799895,201326592~33497107,240139652~19370886,260046848~20273171,289406976~21125692,314572800~57820771,373441908~5818494,381681664~32125072] 2021-07-31T02:29:12.374 INFO:tasks.workunit.client.0.smithi045.stdout:from 2 to HEAD diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89659479~3999082,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:12.374 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89659479~3999082,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:12.379 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~43334606,46137344~10629113,59300692~1156219,60609981~5053700,66056524~17079539,83886080~9772481,99285857~789121,102437886~9019319,116424046~70963406,191717187~6799895,201478817~33344882,240139652~19370886,260046848~20273171,291252461~19280207,314572800~12707070,327653500~3214988,331434830~40958741,373441908~5818494,381681664~32125072] 2021-07-31T02:29:12.379 INFO:tasks.workunit.client.0.smithi045.stdout:from 3 to HEAD diff [677713~20930344,24372582~18962024,48159474~2619597,53817318~2949139,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:12.379 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~20930344,24372582~18962024,48159474~2619597,53817318~2949139,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:12.386 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~21608057,24372582~18962024,48159474~2619597,53817318~2949139,59300692~1156219,60609981~5053700,66056524~17079539,83886080~4661315,89752687~3175987,99285857~789121,102437886~9019319,116424046~70963406,201478817~21375589,223825249~10998450,240139652~19370886,260046848~20273171,291252461~19280207,314572800~12707070,327653500~3214988,331434830~40958741,373441908~5818494,381681664~32125072] 2021-07-31T02:29:12.386 INFO:tasks.workunit.client.0.smithi045.stdout:from 4 to HEAD diff [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~20324185,167531438~13588850,181972347~5415105,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,292264750~18267918,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:12.386 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~20324185,167531438~13588850,181972347~5415105,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,292264750~18267918,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:12.392 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,102437886~9019319,116424046~8181279,124734943~38667402,164857432~22530020,201478817~21375589,223825249~10998450,240139652~19370886,260046848~12640292,275231912~5088107,292264750~18267918,314572800~12707070,327653500~3214988,331434830~40958741,373441908~5818494,381681664~32125072] 2021-07-31T02:29:12.392 INFO:tasks.workunit.client.0.smithi045.stdout:from 5 to HEAD diff [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,292264750~4687170,301341353~9191315,318433435~4174606,322615554~4664316,331638058~718760,334588993~36612024,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:12.392 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,292264750~4687170,301341353~9191315,318433435~4174606,322615554~4664316,331638058~718760,334588993~36612024,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:12.398 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,102437886~9019319,116424046~8181279,124734943~38667402,164857432~12831533,181120288~6267164,201478817~19213323,221470799~1383607,223825249~10998450,241317308~18193230,260046848~12640292,275231912~5088107,292264750~4687170,301341353~9191315,314572800~12707070,331434830~39766187,373441908~5818494,381681664~32125072] 2021-07-31T02:29:12.398 INFO:tasks.workunit.client.0.smithi045.stdout:from 6 to HEAD diff [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,121407098~3198227,124734943~18101328,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,223825249~6536345,241317308~18193230,260438339~12248801,296242351~493439,301341353~9191315,318433435~4174606,322615554~4664316,331638058~718760,334588993~36612024,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:12.398 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,121407098~3198227,124734943~18101328,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,223825249~6536345,241317308~18193230,260438339~12248801,296242351~493439,301341353~9191315,318433435~4174606,322615554~4664316,331638058~718760,334588993~36612024,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:12.404 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,102437886~3730024,121407098~3198227,124734943~18101328,143078160~20324185,164857432~12831533,181120288~6267164,201478817~19213323,221470799~1383607,223825249~6536345,241317308~18193230,260046848~12640292,296242351~493439,301341353~9191315,314572800~12707070,331434830~39766187,373441908~5818494,381681664~32125072] 2021-07-31T02:29:12.404 INFO:tasks.workunit.client.0.smithi045.stdout:from 7 to HEAD diff [872289~20735768,24372582~17941837,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,121407098~3198227,124734943~18101328,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,225052374~5309220,241317308~18193230,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,318433435~4174606,322615554~4664316,334588993~36612024,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:12.405 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~20735768,24372582~17941837,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,121407098~3198227,124734943~18101328,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,225052374~5309220,241317308~18193230,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,318433435~4174606,322615554~4664316,334588993~36612024,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:12.411 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~20735768,24372582~17941837,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,102437886~3730024,121407098~3198227,124734943~18101328,143078160~20324185,164857432~12831533,181120288~6267164,201478817~19213323,221470799~1383607,225052374~5309220,241317308~18193230,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,314572800~12707070,332356818~38844199,373441908~5818494,381681664~32125072] 2021-07-31T02:29:12.411 INFO:tasks.workunit.client.0.smithi045.stdout:from 8 to HEAD diff [872289~20735768,24372582~17941837,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~16816169,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,225052374~5309220,242763340~3666547,248659714~10850824,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,334588993~23788434,359181831~12019186,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:12.411 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~20735768,24372582~17941837,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~16816169,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,225052374~5309220,242763340~3666547,248659714~10850824,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,334588993~23788434,359181831~12019186,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:12.417 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~20735768,24372582~17941837,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,121407098~3198227,126020102~16816169,143078160~20324185,164857432~12831533,181120288~6267164,201478817~19213323,221470799~1383607,225052374~5309220,242763340~3666547,248659714~10850824,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,332356818~26020609,359181831~12019186,373441908~5818494,381820951~31985785] 2021-07-31T02:29:12.417 INFO:tasks.workunit.client.0.smithi045.stdout:from 9 to HEAD diff [1823868~19784189,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143078160~20324185,167792807~9896158,181972347~5415105,202180755~18511385,221470799~1383607,225052374~5309220,242763340~3666547,248659714~10850824,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,334588993~15115237,351585344~6792083,359181831~12019186,373441908~5818494,381820951~21159950,404169452~1052236,406364931~1096263,408916890~4889846] 2021-07-31T02:29:12.418 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [1823868~19784189,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143078160~20324185,167792807~9896158,181972347~5415105,202180755~18511385,221470799~1383607,225052374~5309220,242763340~3666547,248659714~10850824,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,334588993~15115237,351585344~6792083,359181831~12019186,373441908~5818494,381820951~21159950,404169452~1052236,406364931~1096263,408916890~4889846] 2021-07-31T02:29:12.424 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [1823868~19784189,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143078160~20324185,167792807~9896158,181120288~6267164,202180755~18511385,221470799~1383607,225052374~5309220,242763340~3666547,248659714~10850824,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,332356818~17347412,351585344~6792083,359181831~12019186,373441908~5818494,381820951~21159950,404169452~1052236,406364931~1096263,408916890~4889846] 2021-07-31T02:29:12.424 INFO:tasks.workunit.client.0.smithi045.stdout:from 10 to HEAD diff [1823868~19784189,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143078160~11521808,154729104~8673241,167792807~9896158,181972347~5415105,202180755~18511385,221470799~1383607,225052374~5309220,242763340~3666547,248682238~10828300,264698001~7989139,296242351~493439,301341353~9191315,334588993~15115237,351585344~6792083,359181831~12019186,373441908~5818494,381820951~21159950,404169452~1052236,406364931~1096263,408916890~4889846] 2021-07-31T02:29:12.425 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [1823868~19784189,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143078160~11521808,154729104~8673241,167792807~9896158,181972347~5415105,202180755~18511385,221470799~1383607,225052374~5309220,242763340~3666547,248682238~10828300,264698001~7989139,296242351~493439,301341353~9191315,334588993~15115237,351585344~6792083,359181831~12019186,373441908~5818494,381820951~21159950,404169452~1052236,406364931~1096263,408916890~4889846] 2021-07-31T02:29:12.429 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [1823868~19784189,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143078160~11521808,154729104~8673241,167792807~9896158,181120288~6267164,202180755~18511385,221470799~1383607,225052374~5309220,242763340~3666547,248682238~10828300,264698001~7989139,296242351~493439,301341353~9191315,332356818~17347412,351585344~6792083,359181831~12019186,373441908~5818494,381820951~21159950,404169452~1052236,406364931~1096263,408916890~4889846] 2021-07-31T02:29:12.430 INFO:tasks.workunit.client.0.smithi045.stdout:from 11 to HEAD diff [1823868~12981357,17882092~3725965,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143078160~11521808,154729104~8673241,168673724~9015241,181972347~5415105,202180755~18511385,221470799~1383607,225052374~5309220,248682238~10828300,264698001~7989139,296242351~493439,301341353~9191315,334588993~15115237,351585344~6792083,359181831~8323415,373441908~5818494,383830156~6295455,390462350~12518551,404169452~1052236,406364931~1096263] 2021-07-31T02:29:12.431 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [1823868~12981357,17882092~3725965,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143078160~11521808,154729104~8673241,168673724~9015241,181972347~5415105,202180755~18511385,221470799~1383607,225052374~5309220,248682238~10828300,264698001~7989139,296242351~493439,301341353~9191315,334588993~15115237,351585344~6792083,359181831~8323415,373441908~5818494,383830156~6295455,390462350~12518551,404169452~1052236,406364931~1096263] 2021-07-31T02:29:12.436 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [1823868~12981357,17882092~3725965,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,83886080~4661315,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143078160~11521808,154729104~8673241,168673724~9015241,181120288~6267164,202180755~18511385,221470799~1383607,225052374~5309220,248682238~10828300,264698001~7989139,296242351~493439,301341353~9191315,332356818~17347412,351585344~6792083,359181831~8323415,373441908~5818494,383830156~6295455,390462350~12518551,404169452~1052236,406364931~1096263] 2021-07-31T02:29:12.574 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:29:12.574 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:29:12.575 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:29:12.575 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:29:12.575 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:29:12.575 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:29:12.576 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:29:12.576 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:29:12.577 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:29:12.578 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:29:12.578 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:29:12.578 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:29:12.579 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:29:12.579 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:29:12.579 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:29:12.579 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:29:12.580 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:29:12.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:12 smithi154 conmon[34553]: cluster 2021-07-31T02:29:11.456912+0000 mon.a (mon.0 2021-07-31T02:29:12.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:12 smithi154 conmon[34553]: ) 2096 : cluster [DBG] osdmap e1171: 8 total, 8 up, 8 in 2021-07-31T02:29:12.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:12 smithi154 conmon[34553]: audit 2021-07-31T02:29:11.875669+0000 mgr.x (mgr.24542) 1796 : audit [DBG] from='client.17187 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:29:12.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:12 smithi154 conmon[34553]: audit 2021-07-31T02:29:12.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:12 smithi154 conmon[34553]: 2021-07-31T02:29:12.158620+0000 mgr.x (mgr.24542) 1797 : audit [DBG] from='client.27130 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:29:12.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:12 smithi154 conmon[34553]: cluster 2021-07-31T02:29:12.277187+0000 mon.a ( 2021-07-31T02:29:12.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:12 smithi154 conmon[34553]: mon.0) 2097 : cluster [DBG] osdmap e1172: 8 total, 8 up, 8 in 2021-07-31T02:29:12.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:12 smithi045 conmon[33655]: cluster 2021-07-31T02:29:11.456912+0000 mon.a 2021-07-31T02:29:12.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:12 smithi045 conmon[33655]: (mon.0) 2096 : cluster [DBG] osdmap e1171: 8 total, 8 up, 8 in 2021-07-31T02:29:12.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:12 smithi045 conmon[33655]: audit 2021-07-31T02:29:11.875669+0000 mgr.x (mgr.24542) 1796 : audit 2021-07-31T02:29:12.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:12 smithi045 conmon[33655]: [DBG] from='client.17187 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:29:12.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:12 smithi045 conmon[33655]: audit 2021-07-31 2021-07-31T02:29:12.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:12 smithi045 conmon[33655]: T02:29:12.158620 2021-07-31T02:29:12.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:12 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 2021-07-31T02:29:12.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:12 smithi045 conmon[33655]: 1797 : audit [DBG] from='client.27130 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:29:12.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:12 smithi045 conmon[33655]: cluster 2021-07-31T 2021-07-31T02:29:12.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:12 smithi045 conmon[33655]: 02:29:12.277187+0000 mon.a (mon. 2021-07-31T02:29:12.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:12 smithi045 conmon[33655]: 0) 2097 : cluster [DBG] osdmap e1172: 8 total, 8 up, 8 in 2021-07-31T02:29:12.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:12 smithi045 conmon[33655]: 2021-07-31T02:29:12.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:12 smithi045 conmon[39718]: cluster 2021-07-31T02:29 2021-07-31T02:29:12.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:12 smithi045 conmon[39718]: :11.456912+0000 mon.a (mon.0) 2096 : cluster [DBG] osdmap e1171: 8 total, 8 up, 8 in 2021-07-31T02:29:12.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:12 smithi045 conmon[39718]: audit 2021-07 2021-07-31T02:29:12.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:12 smithi045 conmon[39718]: -31T02:29:11.875669+0000 mgr.x ( 2021-07-31T02:29:12.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:12 smithi045 conmon[39718]: mgr.24542) 1796 : audit [DBG] from='client.17187 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:29:12.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:12 smithi045 conmon[39718]: audit 2021- 2021-07-31T02:29:12.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:12 smithi045 conmon[39718]: 07-31T02:29:12 2021-07-31T02:29:12.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:12 smithi045 conmon[39718]: .158620+0000 mgr.x (mgr.24542) 2021-07-31T02:29:12.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:12 smithi045 conmon[39718]: 1797 : audit [DBG] from='client.27130 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:29:12.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:12 smithi045 conmon[39718]: cluster 2021-07-31T02:29:12.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:12 smithi045 conmon[39718]: 2021-07-31 2021-07-31T02:29:12.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:12 smithi045 conmon[39718]: T02:29:12.277187+0000 mon.a (mon 2021-07-31T02:29:12.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:12 smithi045 conmon[39718]: .0) 2097 : cluster [DBG] osdmap e1172: 8 total, 8 up, 8 in 2021-07-31T02:29:13.299 INFO:tasks.workunit.client.0.smithi045.stdout:from 0 to HEAD diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~22526082,86327888~2219507,89659479~3999082,97713673~8454237,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,236085312~4054340,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:13.299 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~22526082,86327888~2219507,89659479~3999082,97713673~8454237,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,236085312~4054340,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:13.344 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~43334606,46137344~10629113,59300692~1156219,60609981~22526082,83886080~9772481,96468992~14988213,116424046~70963406,188743680~9773402,201326592~33497107,234881024~24629514,260046848~20273171,289406976~21125692,314572800~57820771,373441908~5818494,381681664~32125072] 2021-07-31T02:29:13.345 INFO:tasks.workunit.client.0.smithi045.stdout:from 1 to HEAD diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89659479~3999082,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:13.345 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89659479~3999082,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:13.375 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~43334606,46137344~10629113,59300692~1156219,60609981~5053700,66056524~17079539,83886080~9772481,99285857~789121,102437886~9019319,116424046~70963406,191717187~6799895,201326592~33497107,240139652~19370886,260046848~20273171,289406976~21125692,314572800~57820771,373441908~5818494,381681664~32125072] 2021-07-31T02:29:13.375 INFO:tasks.workunit.client.0.smithi045.stdout:from 2 to HEAD diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89659479~3999082,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:13.375 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89659479~3999082,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:13.380 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~43334606,46137344~10629113,59300692~1156219,60609981~5053700,66056524~17079539,83886080~9772481,99285857~789121,102437886~9019319,116424046~70963406,191717187~6799895,201478817~33344882,240139652~19370886,260046848~20273171,291252461~19280207,314572800~12707070,327653500~3214988,331434830~40958741,373441908~5818494,381681664~32125072] 2021-07-31T02:29:13.381 INFO:tasks.workunit.client.0.smithi045.stdout:from 3 to HEAD diff [677713~20930344,24372582~18962024,48159474~2619597,53817318~2949139,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:13.381 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~20930344,24372582~18962024,48159474~2619597,53817318~2949139,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:13.387 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~21608057,24372582~18962024,48159474~2619597,53817318~2949139,59300692~1156219,60609981~5053700,66056524~17079539,83886080~4661315,89752687~3175987,99285857~789121,102437886~9019319,116424046~70963406,201478817~21375589,223825249~10998450,240139652~19370886,260046848~20273171,291252461~19280207,314572800~12707070,327653500~3214988,331434830~40958741,373441908~5818494,381681664~32125072] 2021-07-31T02:29:13.387 INFO:tasks.workunit.client.0.smithi045.stdout:from 4 to HEAD diff [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~20324185,167531438~13588850,181972347~5415105,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,292264750~18267918,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:13.388 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~20324185,167531438~13588850,181972347~5415105,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,292264750~18267918,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:13.393 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,102437886~9019319,116424046~8181279,124734943~38667402,164857432~22530020,201478817~21375589,223825249~10998450,240139652~19370886,260046848~12640292,275231912~5088107,292264750~18267918,314572800~12707070,327653500~3214988,331434830~40958741,373441908~5818494,381681664~32125072] 2021-07-31T02:29:13.394 INFO:tasks.workunit.client.0.smithi045.stdout:from 5 to HEAD diff [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,292264750~4687170,301341353~9191315,318433435~4174606,322615554~4664316,331638058~718760,334588993~36612024,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:13.394 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,292264750~4687170,301341353~9191315,318433435~4174606,322615554~4664316,331638058~718760,334588993~36612024,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:13.399 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,102437886~9019319,116424046~8181279,124734943~38667402,164857432~12831533,181120288~6267164,201478817~19213323,221470799~1383607,223825249~10998450,241317308~18193230,260046848~12640292,275231912~5088107,292264750~4687170,301341353~9191315,314572800~12707070,331434830~39766187,373441908~5818494,381681664~32125072] 2021-07-31T02:29:13.399 INFO:tasks.workunit.client.0.smithi045.stdout:from 6 to HEAD diff [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,121407098~3198227,124734943~18101328,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,223825249~6536345,241317308~18193230,260438339~12248801,296242351~493439,301341353~9191315,318433435~4174606,322615554~4664316,331638058~718760,334588993~36612024,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:13.400 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,121407098~3198227,124734943~18101328,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,223825249~6536345,241317308~18193230,260438339~12248801,296242351~493439,301341353~9191315,318433435~4174606,322615554~4664316,331638058~718760,334588993~36612024,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:13.405 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,102437886~3730024,121407098~3198227,124734943~18101328,143078160~20324185,164857432~12831533,181120288~6267164,201478817~19213323,221470799~1383607,223825249~6536345,241317308~18193230,260046848~12640292,296242351~493439,301341353~9191315,314572800~12707070,331434830~39766187,373441908~5818494,381681664~32125072] 2021-07-31T02:29:13.405 INFO:tasks.workunit.client.0.smithi045.stdout:from 7 to HEAD diff [872289~20735768,24372582~17941837,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,121407098~3198227,124734943~18101328,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,225052374~5309220,241317308~18193230,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,318433435~4174606,322615554~4664316,334588993~36612024,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:13.406 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~20735768,24372582~17941837,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,121407098~3198227,124734943~18101328,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,225052374~5309220,241317308~18193230,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,318433435~4174606,322615554~4664316,334588993~36612024,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:13.411 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~20735768,24372582~17941837,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,102437886~3730024,121407098~3198227,124734943~18101328,143078160~20324185,164857432~12831533,181120288~6267164,201478817~19213323,221470799~1383607,225052374~5309220,241317308~18193230,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,314572800~12707070,332356818~38844199,373441908~5818494,381681664~32125072] 2021-07-31T02:29:13.411 INFO:tasks.workunit.client.0.smithi045.stdout:from 8 to HEAD diff [872289~20735768,24372582~17941837,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~16816169,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,225052374~5309220,242763340~3666547,248659714~10850824,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,334588993~23788434,359181831~12019186,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:13.411 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~20735768,24372582~17941837,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~16816169,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,225052374~5309220,242763340~3666547,248659714~10850824,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,334588993~23788434,359181831~12019186,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:13.417 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~20735768,24372582~17941837,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,121407098~3198227,126020102~16816169,143078160~20324185,164857432~12831533,181120288~6267164,201478817~19213323,221470799~1383607,225052374~5309220,242763340~3666547,248659714~10850824,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,332356818~26020609,359181831~12019186,373441908~5818494,381820951~31985785] 2021-07-31T02:29:13.417 INFO:tasks.workunit.client.0.smithi045.stdout:from 9 to HEAD diff [1823868~19784189,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143078160~20324185,167792807~9896158,181972347~5415105,202180755~18511385,221470799~1383607,225052374~5309220,242763340~3666547,248659714~10850824,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,334588993~15115237,351585344~6792083,359181831~12019186,373441908~5818494,381820951~21159950,404169452~1052236,406364931~1096263,408916890~4889846] 2021-07-31T02:29:13.417 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [1823868~19784189,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143078160~20324185,167792807~9896158,181972347~5415105,202180755~18511385,221470799~1383607,225052374~5309220,242763340~3666547,248659714~10850824,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,334588993~15115237,351585344~6792083,359181831~12019186,373441908~5818494,381820951~21159950,404169452~1052236,406364931~1096263,408916890~4889846] 2021-07-31T02:29:13.423 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [1823868~19784189,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143078160~20324185,167792807~9896158,181120288~6267164,202180755~18511385,221470799~1383607,225052374~5309220,242763340~3666547,248659714~10850824,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,332356818~17347412,351585344~6792083,359181831~12019186,373441908~5818494,381820951~21159950,404169452~1052236,406364931~1096263,408916890~4889846] 2021-07-31T02:29:13.423 INFO:tasks.workunit.client.0.smithi045.stdout:from 10 to HEAD diff [1823868~19784189,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143078160~11521808,154729104~8673241,167792807~9896158,181972347~5415105,202180755~18511385,221470799~1383607,225052374~5309220,242763340~3666547,248682238~10828300,264698001~7989139,296242351~493439,301341353~9191315,334588993~15115237,351585344~6792083,359181831~12019186,373441908~5818494,381820951~21159950,404169452~1052236,406364931~1096263,408916890~4889846] 2021-07-31T02:29:13.423 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [1823868~19784189,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143078160~11521808,154729104~8673241,167792807~9896158,181972347~5415105,202180755~18511385,221470799~1383607,225052374~5309220,242763340~3666547,248682238~10828300,264698001~7989139,296242351~493439,301341353~9191315,334588993~15115237,351585344~6792083,359181831~12019186,373441908~5818494,381820951~21159950,404169452~1052236,406364931~1096263,408916890~4889846] 2021-07-31T02:29:13.428 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [1823868~19784189,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143078160~11521808,154729104~8673241,167792807~9896158,181120288~6267164,202180755~18511385,221470799~1383607,225052374~5309220,242763340~3666547,248682238~10828300,264698001~7989139,296242351~493439,301341353~9191315,332356818~17347412,351585344~6792083,359181831~12019186,373441908~5818494,381820951~21159950,404169452~1052236,406364931~1096263,408916890~4889846] 2021-07-31T02:29:13.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:13 smithi154 conmon[34553]: cluster 2021-07-31T02:29:12.399602+0000 mgr.x (mgr.24542) 1798 : cluster [DBG] pgmap v2278: 41 pgs: 41 active+clean; 874 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 5.2 KiB/s rd, 1023 B/s wr, 5.85k op/s 2021-07-31T02:29:13.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:13 smithi154 conmon[34553]: audit 2021-07-31T02:29:12.574775+0000 mon.c (mon.1) 101 : audit [DBG] from='client.? 172.21.15.45:0/561443991' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:29:13.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:13 smithi154 conmon[34553]: cluster 2021-07-31T02:29:13.287481+0000 mon.a (mon.0) 2098 : cluster [DBG] osdmap e1173: 8 total, 8 up, 8 in 2021-07-31T02:29:13.775 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:29:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:29:13.582Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:29:13.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:29:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:29:13.582Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:29:13.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:29:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:29:13.582Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:29:13.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:13 smithi045 conmon[39718]: cluster 2021-07-31T02:29:12.399602+0000 mgr.x (mgr.24542) 1798 : cluster [DBG] pgmap v2278: 41 pgs: 41 active+clean; 874 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 5.2 KiB/s rd, 1023 B/s wr, 5.85k op/s 2021-07-31T02:29:13.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:13 smithi045 conmon[39718]: audit 2021-07-31T02:29:12.574775+0000 2021-07-31T02:29:13.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:13 smithi045 conmon[39718]: mon.c (mon.1) 101 : audit [DBG] from='client.? 172.21.15.45:0/561443991' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:29:13.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:13 smithi045 conmon[39718]: 2021-07-31T02:29:13.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:13 smithi045 conmon[39718]: cluster 2021-07-31T02:29:13.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:13 smithi045 conmon[39718]: 2021-07-31T02:29:13. 2021-07-31T02:29:13.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:13 smithi045 conmon[39718]: 287481+0000 mon.a (mon.0) 2098 : 2021-07-31T02:29:13.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:13 smithi045 conmon[39718]: cluster [DBG] osdmap e1173: 8 total, 8 up, 8 in 2021-07-31T02:29:13.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:13 smithi045 conmon[33655]: cluster 2021-07-31T02:29:12.399602+0000 mgr.x ( 2021-07-31T02:29:13.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:13 smithi045 conmon[33655]: mgr.24542) 1798 : cluster [DBG] pgmap v2278: 41 pgs: 41 active+clean; 874 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 5.2 KiB/s rd, 1023 B/s wr, 5.85k op/s 2021-07-31T02:29:13.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:13 smithi045 conmon[33655]: audit 2021-07-31 2021-07-31T02:29:13.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:13 smithi045 conmon[33655]: T02:29:12.574775+0000 mon.c (mon.1 2021-07-31T02:29:13.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:13 smithi045 conmon[33655]: ) 101 : audit [DBG] from='client.? 172.21.15.45:0/561443991' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:29:13.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:13 smithi045 conmon[33655]: cluster 2021- 2021-07-31T02:29:13.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:13 smithi045 conmon[33655]: 07-31T02:29:13. 2021-07-31T02:29:13.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:13 smithi045 conmon[33655]: 287481+0000 mon.a (mon.0 2021-07-31T02:29:13.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:13 smithi045 conmon[33655]: ) 2098 : cluster [DBG] osdmap e1173: 8 total, 8 up, 8 in 2021-07-31T02:29:14.302 INFO:tasks.workunit.client.0.smithi045.stdout:from 0 to HEAD diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~22526082,86327888~2219507,89659479~3999082,97713673~8454237,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,236085312~4054340,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:14.303 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~22526082,86327888~2219507,89659479~3999082,97713673~8454237,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,236085312~4054340,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:14.353 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~43334606,46137344~10629113,59300692~1156219,60609981~22526082,83886080~9772481,96468992~14988213,116424046~70963406,188743680~9773402,201326592~33497107,234881024~24629514,260046848~20273171,289406976~21125692,314572800~57820771,373441908~5818494,381681664~32125072] 2021-07-31T02:29:14.354 INFO:tasks.workunit.client.0.smithi045.stdout:from 1 to HEAD diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89659479~3999082,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:14.354 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89659479~3999082,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:14.370 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~43334606,46137344~10629113,59300692~1156219,60609981~5053700,66056524~17079539,83886080~9772481,99285857~789121,102437886~9019319,116424046~70963406,191717187~6799895,201326592~33497107,240139652~19370886,260046848~20273171,289406976~21125692,314572800~57820771,373441908~5818494,381681664~32125072] 2021-07-31T02:29:14.370 INFO:tasks.workunit.client.0.smithi045.stdout:from 2 to HEAD diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89659479~3999082,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:14.371 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89659479~3999082,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:14.376 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~43334606,46137344~10629113,59300692~1156219,60609981~5053700,66056524~17079539,83886080~9772481,99285857~789121,102437886~9019319,116424046~70963406,191717187~6799895,201478817~33344882,240139652~19370886,260046848~20273171,291252461~19280207,314572800~12707070,327653500~3214988,331434830~40958741,373441908~5818494,381681664~32125072] 2021-07-31T02:29:14.376 INFO:tasks.workunit.client.0.smithi045.stdout:from 3 to HEAD diff [677713~20930344,24372582~18962024,48159474~2619597,53817318~2949139,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:14.377 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~20930344,24372582~18962024,48159474~2619597,53817318~2949139,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:14.383 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~21608057,24372582~18962024,48159474~2619597,53817318~2949139,59300692~1156219,60609981~5053700,66056524~17079539,83886080~4661315,89752687~3175987,99285857~789121,102437886~9019319,116424046~70963406,201478817~21375589,223825249~10998450,240139652~19370886,260046848~20273171,291252461~19280207,314572800~12707070,327653500~3214988,331434830~40958741,373441908~5818494,381681664~32125072] 2021-07-31T02:29:14.383 INFO:tasks.workunit.client.0.smithi045.stdout:from 4 to HEAD diff [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~20324185,167531438~13588850,181972347~5415105,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,292264750~18267918,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:14.383 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~20324185,167531438~13588850,181972347~5415105,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,292264750~18267918,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:14.388 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,102437886~9019319,116424046~8181279,124734943~38667402,164857432~22530020,201478817~21375589,223825249~10998450,240139652~19370886,260046848~12640292,275231912~5088107,292264750~18267918,314572800~12707070,327653500~3214988,331434830~40958741,373441908~5818494,381681664~32125072] 2021-07-31T02:29:14.388 INFO:tasks.workunit.client.0.smithi045.stdout:from 5 to HEAD diff [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,292264750~4687170,301341353~9191315,318433435~4174606,322615554~4664316,331638058~718760,334588993~36612024,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:14.389 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,292264750~4687170,301341353~9191315,318433435~4174606,322615554~4664316,331638058~718760,334588993~36612024,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:14.394 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,102437886~9019319,116424046~8181279,124734943~38667402,164857432~12831533,181120288~6267164,201478817~19213323,221470799~1383607,223825249~10998450,241317308~18193230,260046848~12640292,275231912~5088107,292264750~4687170,301341353~9191315,314572800~12707070,331434830~39766187,373441908~5818494,381681664~32125072] 2021-07-31T02:29:14.395 INFO:tasks.workunit.client.0.smithi045.stdout:from 6 to HEAD diff [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,121407098~3198227,124734943~18101328,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,223825249~6536345,241317308~18193230,260438339~12248801,296242351~493439,301341353~9191315,318433435~4174606,322615554~4664316,331638058~718760,334588993~36612024,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:14.395 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,121407098~3198227,124734943~18101328,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,223825249~6536345,241317308~18193230,260438339~12248801,296242351~493439,301341353~9191315,318433435~4174606,322615554~4664316,331638058~718760,334588993~36612024,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:14.400 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,102437886~3730024,121407098~3198227,124734943~18101328,143078160~20324185,164857432~12831533,181120288~6267164,201478817~19213323,221470799~1383607,223825249~6536345,241317308~18193230,260046848~12640292,296242351~493439,301341353~9191315,314572800~12707070,331434830~39766187,373441908~5818494,381681664~32125072] 2021-07-31T02:29:14.401 INFO:tasks.workunit.client.0.smithi045.stdout:from 7 to HEAD diff [872289~20735768,24372582~17941837,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,121407098~3198227,124734943~18101328,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,225052374~5309220,241317308~18193230,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,318433435~4174606,322615554~4664316,334588993~36612024,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:14.401 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~20735768,24372582~17941837,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,121407098~3198227,124734943~18101328,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,225052374~5309220,241317308~18193230,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,318433435~4174606,322615554~4664316,334588993~36612024,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:14.406 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~20735768,24372582~17941837,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,102437886~3730024,121407098~3198227,124734943~18101328,143078160~20324185,164857432~12831533,181120288~6267164,201478817~19213323,221470799~1383607,225052374~5309220,241317308~18193230,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,314572800~12707070,332356818~38844199,373441908~5818494,381681664~32125072] 2021-07-31T02:29:14.407 INFO:tasks.workunit.client.0.smithi045.stdout:from 8 to HEAD diff [872289~20735768,24372582~17941837,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~16816169,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,225052374~5309220,242763340~3666547,248659714~10850824,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,334588993~23788434,359181831~12019186,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:14.407 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~20735768,24372582~17941837,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~16816169,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,225052374~5309220,242763340~3666547,248659714~10850824,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,334588993~23788434,359181831~12019186,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:14.412 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~20735768,24372582~17941837,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,121407098~3198227,126020102~16816169,143078160~20324185,164857432~12831533,181120288~6267164,201478817~19213323,221470799~1383607,225052374~5309220,242763340~3666547,248659714~10850824,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,332356818~26020609,359181831~12019186,373441908~5818494,381820951~31985785] 2021-07-31T02:29:14.413 INFO:tasks.workunit.client.0.smithi045.stdout:from 9 to HEAD diff [1823868~19784189,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143078160~20324185,167792807~9896158,181972347~5415105,202180755~18511385,221470799~1383607,225052374~5309220,242763340~3666547,248659714~10850824,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,334588993~15115237,351585344~6792083,359181831~12019186,373441908~5818494,381820951~21159950,404169452~1052236,406364931~1096263,408916890~4889846] 2021-07-31T02:29:14.413 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [1823868~19784189,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143078160~20324185,167792807~9896158,181972347~5415105,202180755~18511385,221470799~1383607,225052374~5309220,242763340~3666547,248659714~10850824,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,334588993~15115237,351585344~6792083,359181831~12019186,373441908~5818494,381820951~21159950,404169452~1052236,406364931~1096263,408916890~4889846] 2021-07-31T02:29:14.418 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [1823868~19784189,24372582~14188618,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,121407098~3198227,126020102~8880466,135550801~7285470,143078160~20324185,167792807~9896158,181120288~6267164,202180755~18511385,221470799~1383607,225052374~5309220,242763340~3666547,248659714~10850824,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,332356818~17347412,351585344~6792083,359181831~12019186,373441908~5818494,381820951~21159950,404169452~1052236,406364931~1096263,408916890~4889846] 2021-07-31T02:29:15.309 INFO:tasks.workunit.client.0.smithi045.stdout:from 0 to HEAD diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~22526082,86327888~2219507,89659479~3999082,97713673~8454237,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,236085312~4054340,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:15.309 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~22526082,86327888~2219507,89659479~3999082,97713673~8454237,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,236085312~4054340,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:15.376 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~43334606,46137344~10629113,59300692~1156219,60609981~22526082,83886080~9772481,96468992~14988213,116424046~70963406,188743680~9773402,201326592~33497107,234881024~24629514,260046848~20273171,289406976~21125692,314572800~57820771,373441908~5818494,381681664~32125072] 2021-07-31T02:29:15.376 INFO:tasks.workunit.client.0.smithi045.stdout:from 1 to HEAD diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89659479~3999082,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:15.376 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89659479~3999082,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:15.394 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~43334606,46137344~10629113,59300692~1156219,60609981~5053700,66056524~17079539,83886080~9772481,99285857~789121,102437886~9019319,116424046~70963406,191717187~6799895,201326592~33497107,240139652~19370886,260046848~20273171,289406976~21125692,314572800~57820771,373441908~5818494,381681664~32125072] 2021-07-31T02:29:15.395 INFO:tasks.workunit.client.0.smithi045.stdout:from 2 to HEAD diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89659479~3999082,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:15.395 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89659479~3999082,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:15.400 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~43334606,46137344~10629113,59300692~1156219,60609981~5053700,66056524~17079539,83886080~9772481,99285857~789121,102437886~9019319,116424046~70963406,191717187~6799895,201478817~33344882,240139652~19370886,260046848~20273171,291252461~19280207,314572800~12707070,327653500~3214988,331434830~40958741,373441908~5818494,381681664~32125072] 2021-07-31T02:29:15.400 INFO:tasks.workunit.client.0.smithi045.stdout:from 3 to HEAD diff [677713~20930344,24372582~18962024,48159474~2619597,53817318~2949139,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:15.401 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~20930344,24372582~18962024,48159474~2619597,53817318~2949139,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:15.406 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~21608057,24372582~18962024,48159474~2619597,53817318~2949139,59300692~1156219,60609981~5053700,66056524~17079539,83886080~4661315,89752687~3175987,99285857~789121,102437886~9019319,116424046~70963406,201478817~21375589,223825249~10998450,240139652~19370886,260046848~20273171,291252461~19280207,314572800~12707070,327653500~3214988,331434830~40958741,373441908~5818494,381681664~32125072] 2021-07-31T02:29:15.406 INFO:tasks.workunit.client.0.smithi045.stdout:from 4 to HEAD diff [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~20324185,167531438~13588850,181972347~5415105,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,292264750~18267918,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:15.407 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~20324185,167531438~13588850,181972347~5415105,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,292264750~18267918,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:15.412 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,102437886~9019319,116424046~8181279,124734943~38667402,164857432~22530020,201478817~21375589,223825249~10998450,240139652~19370886,260046848~12640292,275231912~5088107,292264750~18267918,314572800~12707070,327653500~3214988,331434830~40958741,373441908~5818494,381681664~32125072] 2021-07-31T02:29:15.413 INFO:tasks.workunit.client.0.smithi045.stdout:from 5 to HEAD diff [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,292264750~4687170,301341353~9191315,318433435~4174606,322615554~4664316,331638058~718760,334588993~36612024,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:15.413 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,292264750~4687170,301341353~9191315,318433435~4174606,322615554~4664316,331638058~718760,334588993~36612024,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:15.418 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,102437886~9019319,116424046~8181279,124734943~38667402,164857432~12831533,181120288~6267164,201478817~19213323,221470799~1383607,223825249~10998450,241317308~18193230,260046848~12640292,275231912~5088107,292264750~4687170,301341353~9191315,314572800~12707070,331434830~39766187,373441908~5818494,381681664~32125072] 2021-07-31T02:29:15.420 INFO:tasks.workunit.client.0.smithi045.stdout:from 6 to HEAD diff [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,121407098~3198227,124734943~18101328,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,223825249~6536345,241317308~18193230,260438339~12248801,296242351~493439,301341353~9191315,318433435~4174606,322615554~4664316,331638058~718760,334588993~36612024,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:15.421 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,121407098~3198227,124734943~18101328,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,223825249~6536345,241317308~18193230,260438339~12248801,296242351~493439,301341353~9191315,318433435~4174606,322615554~4664316,331638058~718760,334588993~36612024,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:15.425 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,102437886~3730024,121407098~3198227,124734943~18101328,143078160~20324185,164857432~12831533,181120288~6267164,201478817~19213323,221470799~1383607,223825249~6536345,241317308~18193230,260046848~12640292,296242351~493439,301341353~9191315,314572800~12707070,331434830~39766187,373441908~5818494,381681664~32125072] 2021-07-31T02:29:15.425 INFO:tasks.workunit.client.0.smithi045.stdout:from 7 to HEAD diff [872289~20735768,24372582~17941837,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,121407098~3198227,124734943~18101328,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,225052374~5309220,241317308~18193230,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,318433435~4174606,322615554~4664316,334588993~36612024,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:15.426 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~20735768,24372582~17941837,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,121407098~3198227,124734943~18101328,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,225052374~5309220,241317308~18193230,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,318433435~4174606,322615554~4664316,334588993~36612024,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:15.430 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~20735768,24372582~17941837,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,102437886~3730024,121407098~3198227,124734943~18101328,143078160~20324185,164857432~12831533,181120288~6267164,201478817~19213323,221470799~1383607,225052374~5309220,241317308~18193230,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,314572800~12707070,332356818~38844199,373441908~5818494,381681664~32125072] 2021-07-31T02:29:15.430 INFO:tasks.workunit.client.0.smithi045.stdout:from 8 to HEAD diff [872289~20735768,24372582~17941837,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~16816169,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,225052374~5309220,242763340~3666547,248659714~10850824,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,334588993~23788434,359181831~12019186,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:15.430 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~20735768,24372582~17941837,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,121407098~3198227,126020102~16816169,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,225052374~5309220,242763340~3666547,248659714~10850824,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,334588993~23788434,359181831~12019186,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:15.435 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~20735768,24372582~17941837,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,121407098~3198227,126020102~16816169,143078160~20324185,164857432~12831533,181120288~6267164,201478817~19213323,221470799~1383607,225052374~5309220,242763340~3666547,248659714~10850824,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,332356818~26020609,359181831~12019186,373441908~5818494,381820951~31985785] 2021-07-31T02:29:15.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:15 smithi154 conmon[34553]: cluster 2021-07-31T02:29:15.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:15 smithi154 conmon[34553]: 2021-07-31T02:29:15.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:15 smithi154 conmon[34553]: 2021 2021-07-31T02:29:15.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:15 smithi154 conmon[34553]: - 2021-07-31T02:29:15.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:15 smithi154 conmon[34553]: 07 2021-07-31T02:29:15.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:15 smithi154 conmon[34553]: - 2021-07-31T02:29:15.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:15 smithi154 conmon[34553]: 31 2021-07-31T02:29:15.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:15 smithi154 conmon[34553]: T 2021-07-31T02:29:15.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:15 smithi154 conmon[34553]: 02 2021-07-31T02:29:15.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:15 smithi154 conmon[34553]: : 2021-07-31T02:29:15.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:15 smithi154 conmon[34553]: 29 2021-07-31T02:29:15.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:15 smithi154 conmon[34553]: : 2021-07-31T02:29:15.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:15 smithi154 conmon[34553]: 14 2021-07-31T02:29:15.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:15 smithi154 conmon[34553]: .290631+0000 mon.a (mon.0) 2099 : cluster [DBG] 2021-07-31T02:29:15.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:15 smithi154 conmon[34553]: osdmap e1174: 8 total, 8 up, 8 in 2021-07-31T02:29:15.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:15 smithi045 conmon[33655]: cluster 2021-07-31T02:29:14.290631+0000 mon.a (mon.0) 2099 : 2021-07-31T02:29:15.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:15 smithi045 conmon[33655]: cluster [DBG] osdmap e1174: 8 total, 8 up, 8 in 2021-07-31T02:29:15.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:15 smithi045 conmon[39718]: cluster 2021-07-31T02:29:14.290631+0000 mon.a (mon 2021-07-31T02:29:15.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:15 smithi045 conmon[39718]: .0) 2099 : cluster [DBG] osdmap e1174: 8 total, 8 up, 8 in 2021-07-31T02:29:16.309 INFO:tasks.workunit.client.0.smithi045.stdout:from 0 to HEAD diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~22526082,86327888~2219507,89659479~3999082,97713673~8454237,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,236085312~4054340,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:16.309 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~22526082,86327888~2219507,89659479~3999082,97713673~8454237,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,236085312~4054340,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:16.369 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~43334606,46137344~10629113,59300692~1156219,60609981~22526082,83886080~9772481,96468992~14988213,116424046~70963406,188743680~9773402,201326592~33497107,234881024~24629514,260046848~20273171,289406976~21125692,314572800~57820771,373441908~5818494,381681664~32125072] 2021-07-31T02:29:16.369 INFO:tasks.workunit.client.0.smithi045.stdout:from 1 to HEAD diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89659479~3999082,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:16.369 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89659479~3999082,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:16.406 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~43334606,46137344~10629113,59300692~1156219,60609981~5053700,66056524~17079539,83886080~9772481,99285857~789121,102437886~9019319,116424046~70963406,191717187~6799895,201326592~33497107,240139652~19370886,260046848~20273171,289406976~21125692,314572800~57820771,373441908~5818494,381681664~32125072] 2021-07-31T02:29:16.406 INFO:tasks.workunit.client.0.smithi045.stdout:from 2 to HEAD diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89659479~3999082,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:16.406 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89659479~3999082,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:16.411 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~43334606,46137344~10629113,59300692~1156219,60609981~5053700,66056524~17079539,83886080~9772481,99285857~789121,102437886~9019319,116424046~70963406,191717187~6799895,201478817~33344882,240139652~19370886,260046848~20273171,291252461~19280207,314572800~12707070,327653500~3214988,331434830~40958741,373441908~5818494,381681664~32125072] 2021-07-31T02:29:16.412 INFO:tasks.workunit.client.0.smithi045.stdout:from 3 to HEAD diff [677713~20930344,24372582~18962024,48159474~2619597,53817318~2949139,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:16.412 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~20930344,24372582~18962024,48159474~2619597,53817318~2949139,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:16.417 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~21608057,24372582~18962024,48159474~2619597,53817318~2949139,59300692~1156219,60609981~5053700,66056524~17079539,83886080~4661315,89752687~3175987,99285857~789121,102437886~9019319,116424046~70963406,201478817~21375589,223825249~10998450,240139652~19370886,260046848~20273171,291252461~19280207,314572800~12707070,327653500~3214988,331434830~40958741,373441908~5818494,381681664~32125072] 2021-07-31T02:29:16.418 INFO:tasks.workunit.client.0.smithi045.stdout:from 4 to HEAD diff [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~20324185,167531438~13588850,181972347~5415105,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,292264750~18267918,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:16.418 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~20324185,167531438~13588850,181972347~5415105,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,292264750~18267918,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:16.423 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,102437886~9019319,116424046~8181279,124734943~38667402,164857432~22530020,201478817~21375589,223825249~10998450,240139652~19370886,260046848~12640292,275231912~5088107,292264750~18267918,314572800~12707070,327653500~3214988,331434830~40958741,373441908~5818494,381681664~32125072] 2021-07-31T02:29:16.423 INFO:tasks.workunit.client.0.smithi045.stdout:from 5 to HEAD diff [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,292264750~4687170,301341353~9191315,318433435~4174606,322615554~4664316,331638058~718760,334588993~36612024,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:16.424 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,292264750~4687170,301341353~9191315,318433435~4174606,322615554~4664316,331638058~718760,334588993~36612024,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:16.429 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,102437886~9019319,116424046~8181279,124734943~38667402,164857432~12831533,181120288~6267164,201478817~19213323,221470799~1383607,223825249~10998450,241317308~18193230,260046848~12640292,275231912~5088107,292264750~4687170,301341353~9191315,314572800~12707070,331434830~39766187,373441908~5818494,381681664~32125072] 2021-07-31T02:29:16.429 INFO:tasks.workunit.client.0.smithi045.stdout:from 6 to HEAD diff [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,121407098~3198227,124734943~18101328,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,223825249~6536345,241317308~18193230,260438339~12248801,296242351~493439,301341353~9191315,318433435~4174606,322615554~4664316,331638058~718760,334588993~36612024,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:16.429 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,121407098~3198227,124734943~18101328,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,223825249~6536345,241317308~18193230,260438339~12248801,296242351~493439,301341353~9191315,318433435~4174606,322615554~4664316,331638058~718760,334588993~36612024,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:16.435 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,102437886~3730024,121407098~3198227,124734943~18101328,143078160~20324185,164857432~12831533,181120288~6267164,201478817~19213323,221470799~1383607,223825249~6536345,241317308~18193230,260046848~12640292,296242351~493439,301341353~9191315,314572800~12707070,331434830~39766187,373441908~5818494,381681664~32125072] 2021-07-31T02:29:16.435 INFO:tasks.workunit.client.0.smithi045.stdout:from 7 to HEAD diff [872289~20735768,24372582~17941837,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,121407098~3198227,124734943~18101328,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,225052374~5309220,241317308~18193230,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,318433435~4174606,322615554~4664316,334588993~36612024,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:16.435 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~20735768,24372582~17941837,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,121407098~3198227,124734943~18101328,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,225052374~5309220,241317308~18193230,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,318433435~4174606,322615554~4664316,334588993~36612024,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:16.440 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~20735768,24372582~17941837,53817318~2949139,59300692~1156219,60609981~5053700,66056524~7829858,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,102437886~3730024,121407098~3198227,124734943~18101328,143078160~20324185,164857432~12831533,181120288~6267164,201478817~19213323,221470799~1383607,225052374~5309220,241317308~18193230,262527487~598269,264698001~7989139,296242351~493439,301341353~9191315,314572800~12707070,332356818~38844199,373441908~5818494,381681664~32125072] 2021-07-31T02:29:16.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:16 smithi154 conmon[34553]: cluster 2021-07-31T02:29:14.399963+0000 mgr.x (mgr.24542) 1799 : cluster [DBG] pgmap v2281: 41 pgs: 41 active+clean; 787 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 6.3 KiB/s rd, 1.2 KiB/s wr, 1.59k op/s 2021-07-31T02:29:16.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:16 smithi154 conmon[34553]: 2021-07-31T02:29:16.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:16 smithi154 conmon[34553]: cluster 2021-07-31T02:29:15.295898+0000 mon.a (mon.0) 2100 : cluster [DBG] osdmap e1175: 8 total, 8 up, 8 in 2021-07-31T02:29:16.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:16 smithi045 conmon[39718]: cluster 2021-07-31T02:29:14.399963+0000 mgr.x (mgr.24542 2021-07-31T02:29:16.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:16 smithi045 conmon[39718]: ) 1799 : cluster [DBG] pgmap v2281: 41 pgs: 41 active+clean; 787 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 6.3 KiB/s rd, 1.2 KiB/s wr, 1.59k op/s 2021-07-31T02:29:16.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:16 smithi045 conmon[39718]: cluster 2021-07-31T02:29:15.295898+0000 mon.a (mon 2021-07-31T02:29:16.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:16 smithi045 conmon[39718]: .0) 2100 : cluster [DBG] osdmap e1175: 8 total, 8 up, 8 in 2021-07-31T02:29:16.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:16 smithi045 conmon[33655]: cluster 2021-07-31T02:29:14.399963+0000 mgr.x (mgr.24542) 1799 : cluster [DBG] pgmap v2281: 41 pgs: 41 active+clean; 787 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 6.3 KiB/s rd, 1.2 KiB/s wr, 1.59k op/s 2021-07-31T02:29:16.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:16 smithi045 conmon[33655]: cluster 2021-07-31T02:29:15.295898+0000 mon.a (mon.0) 2100 : cluster [DBG] osdmap e1175: 8 total, 8 up, 8 in 2021-07-31T02:29:17.290 INFO:tasks.workunit.client.0.smithi045.stdout:from 0 to HEAD diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~22526082,86327888~2219507,89659479~3999082,97713673~8454237,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,236085312~4054340,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:17.290 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~22526082,86327888~2219507,89659479~3999082,97713673~8454237,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,236085312~4054340,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:17.352 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~43334606,46137344~10629113,59300692~1156219,60609981~22526082,83886080~9772481,96468992~14988213,116424046~70963406,188743680~9773402,201326592~33497107,234881024~24629514,260046848~20273171,289406976~21125692,314572800~57820771,373441908~5818494,381681664~32125072] 2021-07-31T02:29:17.353 INFO:tasks.workunit.client.0.smithi045.stdout:from 1 to HEAD diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89659479~3999082,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:17.353 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89659479~3999082,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:17.376 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~43334606,46137344~10629113,59300692~1156219,60609981~5053700,66056524~17079539,83886080~9772481,99285857~789121,102437886~9019319,116424046~70963406,191717187~6799895,201326592~33497107,240139652~19370886,260046848~20273171,289406976~21125692,314572800~57820771,373441908~5818494,381681664~32125072] 2021-07-31T02:29:17.376 INFO:tasks.workunit.client.0.smithi045.stdout:from 2 to HEAD diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89659479~3999082,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:17.377 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89659479~3999082,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:17.382 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~43334606,46137344~10629113,59300692~1156219,60609981~5053700,66056524~17079539,83886080~9772481,99285857~789121,102437886~9019319,116424046~70963406,191717187~6799895,201478817~33344882,240139652~19370886,260046848~20273171,291252461~19280207,314572800~12707070,327653500~3214988,331434830~40958741,373441908~5818494,381681664~32125072] 2021-07-31T02:29:17.382 INFO:tasks.workunit.client.0.smithi045.stdout:from 3 to HEAD diff [677713~20930344,24372582~18962024,48159474~2619597,53817318~2949139,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:17.383 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~20930344,24372582~18962024,48159474~2619597,53817318~2949139,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:17.388 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~21608057,24372582~18962024,48159474~2619597,53817318~2949139,59300692~1156219,60609981~5053700,66056524~17079539,83886080~4661315,89752687~3175987,99285857~789121,102437886~9019319,116424046~70963406,201478817~21375589,223825249~10998450,240139652~19370886,260046848~20273171,291252461~19280207,314572800~12707070,327653500~3214988,331434830~40958741,373441908~5818494,381681664~32125072] 2021-07-31T02:29:17.388 INFO:tasks.workunit.client.0.smithi045.stdout:from 4 to HEAD diff [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~20324185,167531438~13588850,181972347~5415105,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,292264750~18267918,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:17.388 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~20324185,167531438~13588850,181972347~5415105,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,292264750~18267918,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:17.393 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,102437886~9019319,116424046~8181279,124734943~38667402,164857432~22530020,201478817~21375589,223825249~10998450,240139652~19370886,260046848~12640292,275231912~5088107,292264750~18267918,314572800~12707070,327653500~3214988,331434830~40958741,373441908~5818494,381681664~32125072] 2021-07-31T02:29:17.393 INFO:tasks.workunit.client.0.smithi045.stdout:from 5 to HEAD diff [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,292264750~4687170,301341353~9191315,318433435~4174606,322615554~4664316,331638058~718760,334588993~36612024,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:17.394 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,292264750~4687170,301341353~9191315,318433435~4174606,322615554~4664316,331638058~718760,334588993~36612024,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:17.399 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,102437886~9019319,116424046~8181279,124734943~38667402,164857432~12831533,181120288~6267164,201478817~19213323,221470799~1383607,223825249~10998450,241317308~18193230,260046848~12640292,275231912~5088107,292264750~4687170,301341353~9191315,314572800~12707070,331434830~39766187,373441908~5818494,381681664~32125072] 2021-07-31T02:29:17.399 INFO:tasks.workunit.client.0.smithi045.stdout:from 6 to HEAD diff [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,121407098~3198227,124734943~18101328,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,223825249~6536345,241317308~18193230,260438339~12248801,296242351~493439,301341353~9191315,318433435~4174606,322615554~4664316,331638058~718760,334588993~36612024,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:17.399 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,121407098~3198227,124734943~18101328,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,223825249~6536345,241317308~18193230,260438339~12248801,296242351~493439,301341353~9191315,318433435~4174606,322615554~4664316,331638058~718760,334588993~36612024,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:17.405 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,102437886~3730024,121407098~3198227,124734943~18101328,143078160~20324185,164857432~12831533,181120288~6267164,201478817~19213323,221470799~1383607,223825249~6536345,241317308~18193230,260046848~12640292,296242351~493439,301341353~9191315,314572800~12707070,331434830~39766187,373441908~5818494,381681664~32125072] 2021-07-31T02:29:17.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:17 smithi154 conmon[34553]: cluster 2021-07-31T02:29:16.303416+0000 mon.a (mon.0) 2101 : cluster 2021-07-31T02:29:17.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:17 smithi154 conmon[34553]: [DBG] osdmap e1176: 8 total, 8 up, 8 in 2021-07-31T02:29:17.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:17 smithi154 conmon[34553]: cluster 2021-07-31T02:29:17.276965+0000 mon.a (mon.0) 2102 : cluster [DBG] osdmap e1177: 8 total, 8 up, 8 in 2021-07-31T02:29:17.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:17 smithi045 conmon[39718]: cluster 2021-07-31T02:29:17.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:17 smithi045 conmon[39718]: 2021-07-31T02:29:17.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:17 smithi045 conmon[39718]: 2021 2021-07-31T02:29:17.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:17 smithi045 conmon[39718]: - 2021-07-31T02:29:17.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:17 smithi045 conmon[39718]: 07 2021-07-31T02:29:17.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:17 smithi045 conmon[39718]: - 2021-07-31T02:29:17.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:17 smithi045 conmon[39718]: 31 2021-07-31T02:29:17.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:17 smithi045 conmon[39718]: T 2021-07-31T02:29:17.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:17 smithi045 conmon[39718]: 02 2021-07-31T02:29:17.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:17 smithi045 conmon[39718]: : 2021-07-31T02:29:17.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:17 smithi045 conmon[39718]: 29 2021-07-31T02:29:17.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:17 smithi045 conmon[39718]: : 2021-07-31T02:29:17.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:17 smithi045 conmon[39718]: 16 2021-07-31T02:29:17.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:17 smithi045 conmon[39718]: . 2021-07-31T02:29:17.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:17 smithi045 conmon[39718]: 303416 2021-07-31T02:29:17.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:17 smithi045 conmon[39718]: +0000 2021-07-31T02:29:17.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:17 smithi045 conmon[39718]: 2021-07-31T02:29:17.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:17 smithi045 conmon[39718]: mon.a (mon.0) 2101 : cluster [DBG] 2021-07-31T02:29:17.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:17 smithi045 conmon[39718]: osdmap e1176: 8 total, 8 up, 8 in 2021-07-31T02:29:17.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:17 smithi045 conmon[39718]: cluster 2021-07-31T02:29:17.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:17 smithi045 conmon[39718]: 2021-07-31T02:29:17.276965+0000 2021-07-31T02:29:17.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:17 smithi045 conmon[39718]: mon.a (mon 2021-07-31T02:29:17.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:17 smithi045 conmon[39718]: .0) 2102 : cluster [DBG] osdmap e1177: 8 total, 8 up, 8 in 2021-07-31T02:29:17.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:17 smithi045 conmon[33655]: cluster 2021-07-31T02:29:17.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:17 smithi045 conmon[33655]: 2021-07-31T02:29:17.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:17 smithi045 conmon[33655]: 2021 2021-07-31T02:29:17.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:17 smithi045 conmon[33655]: - 2021-07-31T02:29:17.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:17 smithi045 conmon[33655]: 07 2021-07-31T02:29:17.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:17 smithi045 conmon[33655]: - 2021-07-31T02:29:17.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:17 smithi045 conmon[33655]: 31 2021-07-31T02:29:17.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:17 smithi045 conmon[33655]: T 2021-07-31T02:29:17.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:17 smithi045 conmon[33655]: 02 2021-07-31T02:29:17.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:17 smithi045 conmon[33655]: : 2021-07-31T02:29:17.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:17 smithi045 conmon[33655]: 29 2021-07-31T02:29:17.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:17 smithi045 conmon[33655]: : 2021-07-31T02:29:17.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:17 smithi045 conmon[33655]: 16 2021-07-31T02:29:17.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:17 smithi045 conmon[33655]: . 2021-07-31T02:29:17.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:17 smithi045 conmon[33655]: 303416 2021-07-31T02:29:17.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:17 smithi045 conmon[33655]: +0000 2021-07-31T02:29:17.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:17 smithi045 conmon[33655]: 2021-07-31T02:29:17.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:17 smithi045 conmon[33655]: mon.a 2021-07-31T02:29:17.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:17 smithi045 conmon[33655]: (mon.0) 2101 : cluster [DBG] osdmap e1176: 8 total, 8 up, 8 in 2021-07-31T02:29:17.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:17 smithi045 conmon[33655]: cluster 2021-07-31T02:29:17.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:17 smithi045 conmon[33655]: 2021-07-31T02:29:17.276965+0000 2021-07-31T02:29:17.788 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:17 smithi045 conmon[33655]: mon.a (mon.0) 2102 : cluster [DBG] osdmap e1177: 8 total, 8 up, 8 in 2021-07-31T02:29:18.330 INFO:tasks.workunit.client.0.smithi045.stdout:from 0 to HEAD diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~22526082,86327888~2219507,89659479~3999082,97713673~8454237,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,236085312~4054340,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:18.330 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~22526082,86327888~2219507,89659479~3999082,97713673~8454237,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,236085312~4054340,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:18.388 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~43334606,46137344~10629113,59300692~1156219,60609981~22526082,83886080~9772481,96468992~14988213,116424046~70963406,188743680~9773402,201326592~33497107,234881024~24629514,260046848~20273171,289406976~21125692,314572800~57820771,373441908~5818494,381681664~32125072] 2021-07-31T02:29:18.388 INFO:tasks.workunit.client.0.smithi045.stdout:from 1 to HEAD diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89659479~3999082,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:18.389 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89659479~3999082,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:18.422 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~43334606,46137344~10629113,59300692~1156219,60609981~5053700,66056524~17079539,83886080~9772481,99285857~789121,102437886~9019319,116424046~70963406,191717187~6799895,201326592~33497107,240139652~19370886,260046848~20273171,289406976~21125692,314572800~57820771,373441908~5818494,381681664~32125072] 2021-07-31T02:29:18.422 INFO:tasks.workunit.client.0.smithi045.stdout:from 2 to HEAD diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89659479~3999082,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:18.422 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89659479~3999082,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:18.424 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~43334606,46137344~10629113,59300692~1156219,60609981~5053700,66056524~17079539,83886080~9772481,99285857~789121,102437886~9019319,116424046~70963406,191717187~6799895,201478817~33344882,240139652~19370886,260046848~20273171,291252461~19280207,314572800~12707070,327653500~3214988,331434830~40958741,373441908~5818494,381681664~32125072] 2021-07-31T02:29:18.424 INFO:tasks.workunit.client.0.smithi045.stdout:from 3 to HEAD diff [677713~20930344,24372582~18962024,48159474~2619597,53817318~2949139,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:18.424 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~20930344,24372582~18962024,48159474~2619597,53817318~2949139,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:18.431 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~21608057,24372582~18962024,48159474~2619597,53817318~2949139,59300692~1156219,60609981~5053700,66056524~17079539,83886080~4661315,89752687~3175987,99285857~789121,102437886~9019319,116424046~70963406,201478817~21375589,223825249~10998450,240139652~19370886,260046848~20273171,291252461~19280207,314572800~12707070,327653500~3214988,331434830~40958741,373441908~5818494,381681664~32125072] 2021-07-31T02:29:18.432 INFO:tasks.workunit.client.0.smithi045.stdout:from 4 to HEAD diff [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~20324185,167531438~13588850,181972347~5415105,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,292264750~18267918,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:18.432 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~20324185,167531438~13588850,181972347~5415105,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,292264750~18267918,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:18.486 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,102437886~9019319,116424046~8181279,124734943~38667402,164857432~22530020,201478817~21375589,223825249~10998450,240139652~19370886,260046848~12640292,275231912~5088107,292264750~18267918,314572800~12707070,327653500~3214988,331434830~40958741,373441908~5818494,381681664~32125072] 2021-07-31T02:29:18.486 INFO:tasks.workunit.client.0.smithi045.stdout:from 5 to HEAD diff [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,292264750~4687170,301341353~9191315,318433435~4174606,322615554~4664316,331638058~718760,334588993~36612024,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:18.486 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~20324185,167531438~10157527,181972347~5415105,201478817~19213323,221470799~1383607,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,292264750~4687170,301341353~9191315,318433435~4174606,322615554~4664316,331638058~718760,334588993~36612024,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:18.491 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,102437886~9019319,116424046~8181279,124734943~38667402,164857432~12831533,181120288~6267164,201478817~19213323,221470799~1383607,223825249~10998450,241317308~18193230,260046848~12640292,275231912~5088107,292264750~4687170,301341353~9191315,314572800~12707070,331434830~39766187,373441908~5818494,381681664~32125072] 2021-07-31T02:29:18.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:18 smithi154 conmon[34553]: cluster 2021-07-31T02:29:16.400379+0000 mgr.x (mgr.24542) 1800 : cluster [DBG] 2021-07-31T02:29:18.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:18 smithi154 conmon[34553]: pgmap v2284: 41 pgs: 41 active+clean; 787 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 6.2 KiB/s rd, 1.2 KiB/s wr, 1.58k op/s 2021-07-31T02:29:18.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:18 smithi045 conmon[33655]: cluster 2021-07-31T02:29:16.400379+0000 mgr.x 2021-07-31T02:29:18.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:18 smithi045 conmon[33655]: (mgr.24542) 1800 : cluster [DBG] pgmap v2284: 41 pgs: 41 active+clean; 787 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 6.2 KiB/s rd, 1.2 KiB/s wr, 1.58k op/s 2021-07-31T02:29:18.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:18 smithi045 conmon[39718]: cluster 2021-07-31T02:29:16.400379+0000 mgr.x (mgr.24542) 1800 : cluster 2021-07-31T02:29:18.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:18 smithi045 conmon[39718]: [DBG] pgmap v2284: 41 pgs: 41 active+clean; 787 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 6.2 KiB/s rd, 1.2 KiB/s wr, 1.58k op/s 2021-07-31T02:29:19.342 INFO:tasks.workunit.client.0.smithi045.stdout:from 0 to HEAD diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~22526082,86327888~2219507,89659479~3999082,97713673~8454237,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,236085312~4054340,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:19.342 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~22526082,86327888~2219507,89659479~3999082,97713673~8454237,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,236085312~4054340,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:19.381 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~43334606,46137344~10629113,59300692~1156219,60609981~22526082,83886080~9772481,96468992~14988213,116424046~70963406,188743680~9773402,201326592~33497107,234881024~24629514,260046848~20273171,289406976~21125692,314572800~57820771,373441908~5818494,381681664~32125072] 2021-07-31T02:29:19.381 INFO:tasks.workunit.client.0.smithi045.stdout:from 1 to HEAD diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89659479~3999082,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:19.381 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89659479~3999082,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:19.424 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~43334606,46137344~10629113,59300692~1156219,60609981~5053700,66056524~17079539,83886080~9772481,99285857~789121,102437886~9019319,116424046~70963406,191717187~6799895,201326592~33497107,240139652~19370886,260046848~20273171,289406976~21125692,314572800~57820771,373441908~5818494,381681664~32125072] 2021-07-31T02:29:19.424 INFO:tasks.workunit.client.0.smithi045.stdout:from 2 to HEAD diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89659479~3999082,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:19.425 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89659479~3999082,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:19.443 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~43334606,46137344~10629113,59300692~1156219,60609981~5053700,66056524~17079539,83886080~9772481,99285857~789121,102437886~9019319,116424046~70963406,191717187~6799895,201478817~33344882,240139652~19370886,260046848~20273171,291252461~19280207,314572800~12707070,327653500~3214988,331434830~40958741,373441908~5818494,381681664~32125072] 2021-07-31T02:29:19.443 INFO:tasks.workunit.client.0.smithi045.stdout:from 3 to HEAD diff [677713~20930344,24372582~18962024,48159474~2619597,53817318~2949139,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:19.444 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~20930344,24372582~18962024,48159474~2619597,53817318~2949139,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:19.449 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~21608057,24372582~18962024,48159474~2619597,53817318~2949139,59300692~1156219,60609981~5053700,66056524~17079539,83886080~4661315,89752687~3175987,99285857~789121,102437886~9019319,116424046~70963406,201478817~21375589,223825249~10998450,240139652~19370886,260046848~20273171,291252461~19280207,314572800~12707070,327653500~3214988,331434830~40958741,373441908~5818494,381681664~32125072] 2021-07-31T02:29:19.449 INFO:tasks.workunit.client.0.smithi045.stdout:from 4 to HEAD diff [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~20324185,167531438~13588850,181972347~5415105,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,292264750~18267918,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:19.449 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~20324185,167531438~13588850,181972347~5415105,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,292264750~18267918,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:19.455 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [872289~20735768,24372582~18962024,53817318~2949139,59300692~1156219,60609981~5053700,66056524~10431317,78244364~2928921,83886080~4661315,89752687~3175987,99285857~789121,102437886~9019319,116424046~8181279,124734943~38667402,164857432~22530020,201478817~21375589,223825249~10998450,240139652~19370886,260046848~12640292,275231912~5088107,292264750~18267918,314572800~12707070,327653500~3214988,331434830~40958741,373441908~5818494,381681664~32125072] 2021-07-31T02:29:19.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:19 smithi154 conmon[34553]: cluster 2021-07-31T02:29:18.313215+0000 mon.a (mon.0) 2103 : cluster [DBG] osdmap e1178: 8 total, 8 up, 8 in 2021-07-31T02:29:19.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:19 smithi045 conmon[33655]: cluster 2021-07-31T02:29:18.313215+0000 mon.a (mon. 2021-07-31T02:29:19.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:19 smithi045 conmon[33655]: 0) 2103 : cluster [DBG] osdmap e1178: 8 total, 8 up, 8 in 2021-07-31T02:29:19.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:19 smithi045 conmon[39718]: cluster 2021-07-31T02:29:18 2021-07-31T02:29:19.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:19 smithi045 conmon[39718]: .313215+0000 mon.a (mon.0) 2103 : cluster [DBG] osdmap e1178: 8 total, 8 up, 8 in 2021-07-31T02:29:20.348 INFO:tasks.workunit.client.0.smithi045.stdout:from 0 to HEAD diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~22526082,86327888~2219507,89659479~3999082,97713673~8454237,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,236085312~4054340,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:20.348 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~22526082,86327888~2219507,89659479~3999082,97713673~8454237,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,236085312~4054340,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:20.406 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~43334606,46137344~10629113,59300692~1156219,60609981~22526082,83886080~9772481,96468992~14988213,116424046~70963406,188743680~9773402,201326592~33497107,234881024~24629514,260046848~20273171,289406976~21125692,314572800~57820771,373441908~5818494,381681664~32125072] 2021-07-31T02:29:20.406 INFO:tasks.workunit.client.0.smithi045.stdout:from 1 to HEAD diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89659479~3999082,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:20.407 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89659479~3999082,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:20.431 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~43334606,46137344~10629113,59300692~1156219,60609981~5053700,66056524~17079539,83886080~9772481,99285857~789121,102437886~9019319,116424046~70963406,191717187~6799895,201326592~33497107,240139652~19370886,260046848~20273171,289406976~21125692,314572800~57820771,373441908~5818494,381681664~32125072] 2021-07-31T02:29:20.431 INFO:tasks.workunit.client.0.smithi045.stdout:from 2 to HEAD diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89659479~3999082,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:20.432 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89659479~3999082,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:20.436 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~43334606,46137344~10629113,59300692~1156219,60609981~5053700,66056524~17079539,83886080~9772481,99285857~789121,102437886~9019319,116424046~70963406,191717187~6799895,201478817~33344882,240139652~19370886,260046848~20273171,291252461~19280207,314572800~12707070,327653500~3214988,331434830~40958741,373441908~5818494,381681664~32125072] 2021-07-31T02:29:20.437 INFO:tasks.workunit.client.0.smithi045.stdout:from 3 to HEAD diff [677713~20930344,24372582~18962024,48159474~2619597,53817318~2949139,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:20.437 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~20930344,24372582~18962024,48159474~2619597,53817318~2949139,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89752687~3175987,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:20.442 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~21608057,24372582~18962024,48159474~2619597,53817318~2949139,59300692~1156219,60609981~5053700,66056524~17079539,83886080~4661315,89752687~3175987,99285857~789121,102437886~9019319,116424046~70963406,201478817~21375589,223825249~10998450,240139652~19370886,260046848~20273171,291252461~19280207,314572800~12707070,327653500~3214988,331434830~40958741,373441908~5818494,381681664~32125072] 2021-07-31T02:29:20.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:20 smithi154 conmon[34553]: cluster 2021-07-31T02:29:18.400763+0000 mgr.x (mgr.24542) 1801 : cluster [DBG] pgmap v2287: 41 pgs: 41 active+clean; 787 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail 2021-07-31T02:29:20.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:20 smithi154 conmon[34553]: cluster 2021-07-31T02:29:19.325597+0000 mon.a (mon.0) 2104 : cluster [DBG] osdmap e1179: 8 total, 8 up, 8 in 2021-07-31T02:29:20.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:20 smithi045 conmon[39718]: cluster 2021-07-31T02:29:18.400763+0000 2021-07-31T02:29:20.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:20 smithi045 conmon[39718]: mgr.x (mgr.24542) 1801 : cluster [DBG] pgmap v2287: 41 pgs: 41 active+clean; 787 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail 2021-07-31T02:29:20.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:20 smithi045 conmon[39718]: cluster 2021-07-31T02:29:20.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:20 smithi045 conmon[39718]: 2021-07-31T02:29:19.325597+0000 2021-07-31T02:29:20.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:20 smithi045 conmon[39718]: mon.a (mon.0) 2104 : cluster [DBG] 2021-07-31T02:29:20.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:20 smithi045 conmon[39718]: osdmap e1179: 8 total, 8 up, 8 in 2021-07-31T02:29:20.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:20 smithi045 conmon[33655]: cluster 2021-07-31T02:29:18.400763+0000 2021-07-31T02:29:20.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:20 smithi045 conmon[33655]: mgr.x (mgr.24542) 1801 : cluster [DBG] pgmap v2287: 41 pgs: 41 active+clean; 787 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail 2021-07-31T02:29:20.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:20 smithi045 conmon[33655]: cluster 2021-07-31T02:29 2021-07-31T02:29:20.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:20 smithi045 conmon[33655]: :19.325597+0000 mon.a ( 2021-07-31T02:29:20.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:20 smithi045 conmon[33655]: mon.0) 2104 : cluster [DBG] osdmap e1179: 8 total, 8 up, 8 in 2021-07-31T02:29:20.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:20 smithi045 conmon[33655]: 2021-07-31T02:29:21.352 INFO:tasks.workunit.client.0.smithi045.stdout:from 0 to HEAD diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~22526082,86327888~2219507,89659479~3999082,97713673~8454237,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,236085312~4054340,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:21.352 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~22526082,86327888~2219507,89659479~3999082,97713673~8454237,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,236085312~4054340,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:21.408 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~43334606,46137344~10629113,59300692~1156219,60609981~22526082,83886080~9772481,96468992~14988213,116424046~70963406,188743680~9773402,201326592~33497107,234881024~24629514,260046848~20273171,289406976~21125692,314572800~57820771,373441908~5818494,381681664~32125072] 2021-07-31T02:29:21.408 INFO:tasks.workunit.client.0.smithi045.stdout:from 1 to HEAD diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89659479~3999082,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:21.409 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89659479~3999082,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:21.433 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~43334606,46137344~10629113,59300692~1156219,60609981~5053700,66056524~17079539,83886080~9772481,99285857~789121,102437886~9019319,116424046~70963406,191717187~6799895,201326592~33497107,240139652~19370886,260046848~20273171,289406976~21125692,314572800~57820771,373441908~5818494,381681664~32125072] 2021-07-31T02:29:21.434 INFO:tasks.workunit.client.0.smithi045.stdout:from 2 to HEAD diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89659479~3999082,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:21.434 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89659479~3999082,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3214988,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:21.440 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~43334606,46137344~10629113,59300692~1156219,60609981~5053700,66056524~17079539,83886080~9772481,99285857~789121,102437886~9019319,116424046~70963406,191717187~6799895,201478817~33344882,240139652~19370886,260046848~20273171,291252461~19280207,314572800~12707070,327653500~3214988,331434830~40958741,373441908~5818494,381681664~32125072] 2021-07-31T02:29:21.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:21 smithi154 conmon[34553]: cluster 2021-07-31T02:29:20.331271+0000 mon.a (mon.0) 2105 : cluster 2021-07-31T02:29:21.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:21 smithi154 conmon[34553]: [DBG] osdmap e1180: 8 total, 8 up, 8 in 2021-07-31T02:29:21.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:21 smithi045 conmon[33655]: cluster 2021-07-31T02:29:20.331271+0000 2021-07-31T02:29:21.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:21 smithi045 conmon[33655]: mon.a (mon.0) 2105 : cluster [DBG] osdmap e1180: 8 total, 8 up, 8 in 2021-07-31T02:29:21.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:21 smithi045 conmon[39718]: cluster 2021-07-31T02:29:20.331271 2021-07-31T02:29:21.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:21 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2105 : cluster [DBG] osdmap e1180: 8 total, 8 up, 8 in 2021-07-31T02:29:22.296 INFO:tasks.workunit.client.0.smithi045.stdout:from 0 to HEAD diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~22526082,86327888~2219507,89659479~3999082,97713673~8454237,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,236085312~4054340,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:22.297 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~22526082,86327888~2219507,89659479~3999082,97713673~8454237,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,236085312~4054340,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:22.349 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~43334606,46137344~10629113,59300692~1156219,60609981~22526082,83886080~9772481,96468992~14988213,116424046~70963406,188743680~9773402,201326592~33497107,234881024~24629514,260046848~20273171,289406976~21125692,314572800~57820771,373441908~5818494,381681664~32125072] 2021-07-31T02:29:22.349 INFO:tasks.workunit.client.0.smithi045.stdout:from 1 to HEAD diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89659479~3999082,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:22.349 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~5053700,66056524~17079539,86327888~2219507,89659479~3999082,99285857~789121,102437886~3730024,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:22.377 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~43334606,46137344~10629113,59300692~1156219,60609981~5053700,66056524~17079539,83886080~9772481,99285857~789121,102437886~9019319,116424046~70963406,191717187~6799895,201326592~33497107,240139652~19370886,260046848~20273171,289406976~21125692,314572800~57820771,373441908~5818494,381681664~32125072] 2021-07-31T02:29:22.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:22 smithi154 conmon[34553]: cluster 2021-07-31T02:29:20.401279+0000 mgr.x (mgr.24542) 1802 : cluster 2021-07-31T02:29:23.319 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:22 smithi154 conmon[34553]: [DBG] pgmap v2290: 41 pgs: 41 active+clean; 649 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 6.2 KiB/s rd, 1.2 KiB/s wr, 978 op/s 2021-07-31T02:29:23.320 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:22 smithi154 conmon[34553]: cluster 2021-07-31T02:29:21.336983+0000 mon.a (mon.0) 2106 : cluster [DBG] osdmap e1181: 8 total, 8 up, 8 in 2021-07-31T02:29:23.320 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:22 smithi154 conmon[34553]: cluster 2021-07-31T02:29:22.277871+0000 2021-07-31T02:29:23.320 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:22 smithi154 conmon[34553]: mon.a (mon.0) 2107 : cluster [DBG] osdmap e1182: 8 total, 8 up, 8 in 2021-07-31T02:29:23.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:22 smithi045 conmon[33655]: cluster 2021-07-31T02:29:23.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:22 smithi045 conmon[33655]: 2021-07-31T 2021-07-31T02:29:23.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:22 smithi045 conmon[33655]: 02:29:20.401279+0000 mgr.x (mgr.24542) 1802 : cluster [DBG] pgmap v2290: 41 pgs: 41 active+clean; 649 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 6.2 KiB/s rd, 1.2 KiB/s wr, 978 op/s 2021-07-31T02:29:23.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:22 smithi045 conmon[33655]: cluster 2021-07-31T02:29:23.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:22 smithi045 conmon[33655]: 2021-07-31T02:29:21.336983+0000 mon.a (mon.0) 2106 2021-07-31T02:29:23.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:22 smithi045 conmon[33655]: : cluster [DBG] osdmap e1181: 8 total, 8 up, 8 in 2021-07-31T02:29:23.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:22 smithi045 conmon[33655]: cluster 2021- 2021-07-31T02:29:23.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:22 smithi045 conmon[33655]: 07-31T02:29:22.277871+0000 mon.a (mon 2021-07-31T02:29:23.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:22 smithi045 conmon[33655]: .0) 2107 : cluster [DBG] osdmap e1182: 8 total, 8 up, 8 in 2021-07-31T02:29:23.324 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:22 smithi045 conmon[39718]: cluster 2021-07-31T02:29:20.401279 2021-07-31T02:29:23.324 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:22 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 1802 : cluster [DBG] pgmap v2290: 41 pgs: 41 active+clean; 649 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 6.2 KiB/s rd, 1.2 KiB/s wr, 978 op/s 2021-07-31T02:29:23.324 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:22 smithi045 conmon[39718]: cluster 2021- 2021-07-31T02:29:23.324 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:22 smithi045 conmon[39718]: 07-31T02:29:21.336983+0000 mon.a (mon.0) 2021-07-31T02:29:23.324 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:22 smithi045 conmon[39718]: 2106 : cluster [DBG] osdmap e1181: 8 total, 8 up, 8 in 2021-07-31T02:29:23.325 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:22 smithi045 conmon[39718]: cluster 2021-07-31T02:29:23.325 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:22 smithi045 conmon[39718]: 2021-07-31T02:29:22. 2021-07-31T02:29:23.326 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:22 smithi045 conmon[39718]: 277871+0000 mon.a (mon.0) 2107 : cluster 2021-07-31T02:29:23.326 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:22 smithi045 conmon[39718]: [DBG] osdmap e1182: 8 total, 8 up, 8 in 2021-07-31T02:29:23.355 INFO:tasks.workunit.client.0.smithi045.stdout:from 0 to HEAD diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~22526082,86327888~2219507,89659479~3999082,97713673~8454237,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,236085312~4054340,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:23.355 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [677713~20930344,24372582~18962024,47779655~367499,48159474~8606983,59300692~1156219,60609981~22526082,86327888~2219507,89659479~3999082,97713673~8454237,107771043~3686162,116694067~7911258,124734943~18101328,143078160~21779272,167531438~13588850,181972347~5415105,191717187~6799895,201478817~21375589,223825249~10998450,236085312~4054340,241317308~18193230,260438339~12248801,275231912~5088107,291252461~19280207,318433435~4174606,322615554~4664316,327653500~3781330,331638058~718760,334588993~37804578,373441908~5818494,381820951~21159950,403808658~9998078] 2021-07-31T02:29:23.407 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~43334606,46137344~10629113,59300692~1156219,60609981~22526082,83886080~9772481,96468992~14988213,116424046~70963406,188743680~9773402,201326592~33497107,234881024~24629514,260046848~20273171,289406976~21125692,314572800~57820771,373441908~5818494,381681664~32125072] 2021-07-31T02:29:24.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:29:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:29:23.582Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:29:24.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:29:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:29:23.582Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:29:24.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:29:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:29:23.582Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:29:24.369 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] DiffIterateTest/0.DiffIterateStress (39947 ms) 2021-07-31T02:29:24.369 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] DiffIterateTest/0.DiffIterateRegression6926 2021-07-31T02:29:24.369 INFO:tasks.workunit.client.0.smithi045.stdout:using old format 2021-07-31T02:29:24.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:24 smithi154 conmon[34553]: cluster 2021-07-31T02:29:22.401716+0000 mgr.x (mgr.24542) 1803 : cluster [DBG] pgmap v2293: 41 pgs: 41 active+clean; 649 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 6.2 KiB/s rd, 1.2 KiB/s wr, 978 op/s 2021-07-31T02:29:24.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:24 smithi154 conmon[34553]: cluster 2021-07-31T02:29 2021-07-31T02:29:24.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:24 smithi154 conmon[34553]: :23.343297+0000 mon.a (mon.0) 2108 : cluster [DBG] osdmap e1183: 8 total, 8 up, 8 in 2021-07-31T02:29:24.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:24 smithi045 conmon[33655]: cluster 2021-07-31T02:29:22.401716+0000 mgr.x (mgr.24542) 1803 : cluster 2021-07-31T02:29:24.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:24 smithi045 conmon[33655]: [DBG] pgmap v2293: 41 pgs: 41 active+clean; 649 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 6.2 KiB/s rd, 1.2 KiB/s wr, 978 op/s 2021-07-31T02:29:24.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:24 smithi045 conmon[33655]: cluster 2021-07-31T02:29:23.343297+0000 mon.a (mon.0) 2108 : cluster [DBG] osdmap e1183: 8 total, 8 up, 8 in 2021-07-31T02:29:24.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:24 smithi045 conmon[39718]: cluster 2021-07-31T02:29:22.401716+0000 mgr.x (mgr 2021-07-31T02:29:24.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:24 smithi045 conmon[39718]: .24542) 1803 : cluster [DBG] pgmap v2293: 41 pgs: 41 active+clean; 649 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 6.2 KiB/s rd, 1.2 KiB/s wr, 978 op/s 2021-07-31T02:29:24.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:24 smithi045 conmon[39718]: cluster 2021-07-31T02:29:23.343297+0000 mon.a (mon.0) 2108 : cluster [DBG] osdmap e1183: 8 total, 8 up, 8 in 2021-07-31T02:29:25.364 INFO:tasks.workunit.client.0.smithi045.stdout:iterate_cb 0~256 2021-07-31T02:29:25.368 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] DiffIterateTest/0.DiffIterateRegression6926 (999 ms) 2021-07-31T02:29:25.368 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] DiffIterateTest/0.DiffIterateIgnoreParent 2021-07-31T02:29:25.369 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:29:25.369 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] DiffIterateTest/0.DiffIterateIgnoreParent (0 ms) 2021-07-31T02:29:25.369 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] DiffIterateTest/0.DiffIterateCallbackError 2021-07-31T02:29:25.369 INFO:tasks.workunit.client.0.smithi045.stdout:using old format 2021-07-31T02:29:25.380 INFO:tasks.workunit.client.0.smithi045.stdout: wrote [1922022~5858,4459330~91050,4920122~33488,7423895~78649,9314694~89797,15189015~17512,15866854~96603,16339794~84828,19194521~100548,19964459~53317] 2021-07-31T02:29:25.387 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] DiffIterateTest/0.DiffIterateCallbackError (19 ms) 2021-07-31T02:29:25.387 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] DiffIterateTest/0.DiffIterateParentDiscard 2021-07-31T02:29:25.388 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:29:25.388 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] DiffIterateTest/0.DiffIterateParentDiscard (0 ms) 2021-07-31T02:29:25.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:25 smithi154 conmon[34553]: cluster 2021-07-31T02:29:24.349480+0000 mon.a (mon.0) 2109 : cluster 2021-07-31T02:29:25.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:25 smithi154 conmon[34553]: [DBG] osdmap e1184: 8 total, 8 up, 8 in 2021-07-31T02:29:25.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:25 smithi154 conmon[34553]: cluster 2021-07-31T02:29:24.402145+0000 mgr.x (mgr.24542) 1804 : cluster [DBG] pgmap v2296: 41 pgs: 41 active+clean; 472 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 6.2 KiB/s rd, 1.2 KiB/s wr, 372 op/s 2021-07-31T02:29:25.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:25 smithi045 conmon[33655]: cluster 2021-07-31T02:29:24.349480+0000 2021-07-31T02:29:25.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:25 smithi045 conmon[33655]: mon.a (mon.0) 2109 : cluster [DBG] osdmap e1184: 8 total, 8 up, 8 in 2021-07-31T02:29:25.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:25 smithi045 conmon[33655]: cluster 2021-07-31T02:29:24 2021-07-31T02:29:25.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:25 smithi045 conmon[33655]: .402145+0000 mgr.x (mgr.24542) 1804 : cluster [DBG] 2021-07-31T02:29:25.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:25 smithi045 conmon[33655]: pgmap v2296: 41 pgs: 41 active+clean; 472 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 6.2 KiB/s rd, 1.2 KiB/s wr, 372 op/s 2021-07-31T02:29:25.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:25 smithi045 conmon[39718]: cluster 2021-07-31T02:29:24.349480+0000 mon.a (mon.0 2021-07-31T02:29:25.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:25 smithi045 conmon[39718]: ) 2109 : cluster [DBG] osdmap e1184: 8 total, 8 up, 8 in 2021-07-31T02:29:25.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:25 smithi045 conmon[39718]: cluster 2021-07-31T 2021-07-31T02:29:25.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:25 smithi045 conmon[39718]: 02:29:24.402145+0000 mgr.x (mgr.24542) 1804 : cluster [DBG] pgmap v2296: 41 pgs: 41 active+clean; 472 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 6.2 KiB/s rd, 1.2 KiB/s wr, 372 op/s 2021-07-31T02:29:26.362 INFO:tasks.workunit.client.0.smithi045.stdout:[----------] 7 tests from DiffIterateTest/0 (48996 ms total) 2021-07-31T02:29:26.363 INFO:tasks.workunit.client.0.smithi045.stdout: 2021-07-31T02:29:26.363 INFO:tasks.workunit.client.0.smithi045.stdout:[----------] 7 tests from DiffIterateTest/1, where TypeParam = DiffIterateParams 2021-07-31T02:29:26.396 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] DiffIterateTest/1.DiffIterate 2021-07-31T02:29:26.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:26 smithi154 conmon[34553]: cluster 2021-07-31T02:29:25.356100+0000 mon.a (mon.0) 2110 : cluster [DBG] osdmap e1185: 8 total, 8 up, 8 in 2021-07-31T02:29:26.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:26 smithi154 conmon[34553]: 2021-07-31T02:29:26.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:26 smithi045 conmon[39718]: cluster 2021-07-31T02:29:25 2021-07-31T02:29:26.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:26 smithi045 conmon[39718]: .356100+0000 mon.a (mon.0) 2110 : cluster [DBG] osdmap e1185: 8 total, 8 up, 8 in 2021-07-31T02:29:26.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:26 smithi045 conmon[33655]: cluster 2021-07-31T02:29:26.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:26 smithi045 conmon[33655]: 2021-07-31T02:29:26.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:26 smithi045 conmon[33655]: 2021-07-31T02:29:25.356100+0000 2021-07-31T02:29:26.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:26 smithi045 conmon[33655]: mon.a (mon.0) 2110 : cluster [DBG] osdmap e1185: 8 total, 8 up, 8 in 2021-07-31T02:29:27.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:27 smithi154 conmon[34553]: cluster 2021-07-31T02:29:26.357663+0000 mon.a (mon.0) 2111 : cluster [DBG] 2021-07-31T02:29:27.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:27 smithi154 conmon[34553]: osdmap e1186: 8 total, 8 up, 8 in 2021-07-31T02:29:27.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:27 smithi154 conmon[34553]: cluster 2021-07-31T02:29:26.402562+0000 mgr.x (mgr.24542) 1805 : cluster [DBG] pgmap v2299: 9 pgs: 9 active+clean; 4.6 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2021-07-31T02:29:27.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:27 smithi045 conmon[39718]: cluster 2021-07-31T02:29:26.357663+0000 2021-07-31T02:29:27.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:27 smithi045 conmon[39718]: mon.a (mon.0) 2111 : cluster [DBG] osdmap e1186: 8 total, 8 up, 8 in 2021-07-31T02:29:27.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:27 smithi045 conmon[39718]: cluster 2021-07-31T 2021-07-31T02:29:27.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:27 smithi045 conmon[39718]: 02:29:26.402562+0000 mgr.x (mgr.24542) 1805 : cluster [DBG] 2021-07-31T02:29:27.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:27 smithi045 conmon[39718]: pgmap v2299: 9 pgs: 9 active+clean; 4.6 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2021-07-31T02:29:27.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:27 smithi045 conmon[33655]: cluster 2021-07-31T02:29:27.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:27 smithi045 conmon[33655]: 2021-07 2021-07-31T02:29:27.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:27 smithi045 conmon[33655]: -31T02:29:26.357663+0000 mon.a (mon.0) 2111 : cluster [DBG] osdmap e1186: 8 total, 8 up, 8 in 2021-07-31T02:29:27.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:27 smithi045 conmon[33655]: cluster 2021- 2021-07-31T02:29:27.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:27 smithi045 conmon[33655]: 07-31T02:29:26.402562+0000 mgr.x (mgr.24542) 1805 2021-07-31T02:29:27.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:27 smithi045 conmon[33655]: : cluster [DBG] pgmap v2299: 9 pgs: 9 active+clean; 4.6 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2021-07-31T02:29:27.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:27 smithi045 conmon[33655]: 2021-07-31T02:29:28.385 INFO:tasks.workunit.client.0.smithi045.stdout:using old format 2021-07-31T02:29:28.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:28 smithi154 conmon[34553]: cluster 2021-07-31T02:29:27.371115+0000 mon.a (mon.0) 2112 : cluster [DBG] osdmap e1187: 8 total, 8 up, 8 in 2021-07-31T02:29:28.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:28 smithi154 conmon[34553]: audit 2021-07-31T02:29:27.372116+0000 mon.c (mon.1) 102 : audit [INF] from='client.? 172.21.15.45:0/4033991907' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:29:28.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:28 smithi154 conmon[34553]: audit 2021-07-31T02:29:27.372696+0000 mon.a (mon.0) 2113 : audit 2021-07-31T02:29:28.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:28 smithi154 conmon[34553]: [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:29:28.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:28 smithi154 conmon[34553]: audit 2021- 2021-07-31T02:29:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:28 smithi154 conmon[34553]: 07-31T02:29:28.079443+0000 mon.b (mon.2) 2021-07-31T02:29:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:28 smithi154 conmon[34553]: 283 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:29:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:28 smithi154 conmon[34553]: audit 2021-07- 2021-07-31T02:29:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:28 smithi154 conmon[34553]: 31T02:29:28.080135+0000 mon.a ( 2021-07-31T02:29:28.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:28 smithi154 conmon[34553]: mon.0) 2114 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:29:28.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:28 smithi154 conmon[34553]: audit 2021-07-31T 2021-07-31T02:29:28.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:28 smithi154 conmon[34553]: 02:29:28.103059+0000 mon.b (mon.2) 284 : audit 2021-07-31T02:29:28.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:28 smithi154 conmon[34553]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:29:28.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:28 smithi154 conmon[34553]: audit 2021-07- 2021-07-31T02:29:28.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:28 smithi154 conmon[34553]: 31T02:29:28.103597+0000 mon.a ( 2021-07-31T02:29:28.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:28 smithi154 conmon[34553]: mon.0) 2115 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:29:28.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:28 smithi045 conmon[33655]: cluster 2021-07-31T02:29:27.371115+0000 2021-07-31T02:29:28.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:28 smithi045 conmon[33655]: mon.a (mon.0) 2112 : cluster [DBG] osdmap e1187: 8 total, 8 up, 8 in 2021-07-31T02:29:28.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:28 smithi045 conmon[33655]: audit 2021-07-31T02:29:27.372116+0000 mon.c (mon.1) 102 : audit [INF] from='client.? 172.21.15.45:0/4033991907' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:29:28.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:28 smithi045 conmon[33655]: 2021-07-31T02:29:28.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:28 smithi045 conmon[33655]: audit 2021-07-31 2021-07-31T02:29:28.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:28 smithi045 conmon[33655]: T02:29:27.372696+0000 mon.a (mon.0) 2113 : audit [INF] 2021-07-31T02:29:28.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:28 smithi045 conmon[33655]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:29:28.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:28 smithi045 conmon[33655]: audit 2021-07 2021-07-31T02:29:28.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:28 smithi045 conmon[33655]: -31T02:29:28.079443+0000 mon.b (mon 2021-07-31T02:29:28.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:28 smithi045 conmon[33655]: .2) 283 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:29:28.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:28 smithi045 conmon[33655]: 2021-07-31T02:29:28.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:28 smithi045 conmon[33655]: audit 2021-07-31T02:29:28.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:28 smithi045 conmon[33655]: 2021-07-31T02:29:28.080135+0000 2021-07-31T02:29:28.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:28 smithi045 conmon[33655]: mon.a (mon.0) 2114 : audit [INF] 2021-07-31T02:29:28.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:28 smithi045 conmon[33655]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:29:28.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:28 smithi045 conmon[33655]: audit 2021-07-31T02:29:28.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:28 smithi045 conmon[33655]: 2021-07-31T02:29: 2021-07-31T02:29:28.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:28 smithi045 conmon[33655]: 28.103059+0000 mon.b (mon. 2021-07-31T02:29:28.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:28 smithi045 conmon[33655]: 2) 284 : audit [INF] 2021-07-31T02:29:28.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:28 smithi045 conmon[33655]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:29:28.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:28 smithi045 conmon[33655]: audit 2021-07-31T02:29:28.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:28 smithi045 conmon[33655]: 2021-07-31T02:29:28. 2021-07-31T02:29:28.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:28 smithi045 conmon[33655]: 103597+0000 mon.a (mon.0) 2021-07-31T02:29:28.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:28 smithi045 conmon[33655]: 2115 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:29:28.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:28 smithi045 conmon[39718]: cluster 2021-07-31T02:29:27.371115+0000 2021-07-31T02:29:28.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:28 smithi045 conmon[39718]: mon.a (mon.0) 2112 : cluster [DBG] osdmap e1187: 8 total, 8 up, 8 in 2021-07-31T02:29:28.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:28 smithi045 conmon[39718]: audit 2021- 2021-07-31T02:29:28.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:28 smithi045 conmon[39718]: 07-31T02:29:27.372116+0000 mon.c (mon. 2021-07-31T02:29:28.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:28 smithi045 conmon[39718]: 1) 102 : audit [INF] from='client.? 172.21.15.45:0/4033991907' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:29:28.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:28 smithi045 conmon[39718]: audit 2021-07-31T02:29:28.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:28 smithi045 conmon[39718]: 2021-07-31T02:29:27 2021-07-31T02:29:28.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:28 smithi045 conmon[39718]: .372696+0000 mon.a (mon.0) 2113 2021-07-31T02:29:28.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:28 smithi045 conmon[39718]: : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:29:28.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:28 smithi045 conmon[39718]: audit 2021-07 2021-07-31T02:29:28.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:28 smithi045 conmon[39718]: -31T02:29:28.079443+0000 2021-07-31T02:29:28.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:28 smithi045 conmon[39718]: mon.b (mon.2) 283 : audit 2021-07-31T02:29:28.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:28 smithi045 conmon[39718]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:29:28.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:28 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T02:29:28.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:28 smithi045 conmon[39718]: 02:29:28.080135+0000 mon.a ( 2021-07-31T02:29:28.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:28 smithi045 conmon[39718]: mon.0) 2114 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:29:28.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:28 smithi045 conmon[39718]: audit 2021-07-31T02:29:28.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:28 smithi045 conmon[39718]: 2021-07-31T02:29: 2021-07-31T02:29:28.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:28 smithi045 conmon[39718]: 28.103059+0000 mon.b (mon.2) 284 2021-07-31T02:29:28.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:28 smithi045 conmon[39718]: : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:29:28.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:28 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T02:29:28.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:28 smithi045 conmon[39718]: 02:29:28.103597+0000 mon.a ( 2021-07-31T02:29:28.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:28 smithi045 conmon[39718]: mon.0) 2115 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:29:28.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:28 smithi045 conmon[39718]: 2021-07-31T02:29:29.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:29 smithi154 conmon[34553]: audit 2021-07-31T02:29:28.376948+0000 mon.a (mon.0) 2116 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-7","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:29:29.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:29 smithi154 conmon[34553]: cluster 2021-07-31T02:29:28.377247+0000 mon.a (mon.0) 2117 : 2021-07-31T02:29:29.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:29 smithi154 conmon[34553]: cluster [DBG] osdmap e1188: 8 total, 8 up, 8 in 2021-07-31T02:29:29.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:29 smithi154 conmon[34553]: cluster 2021-07-31T02:29:28.403041+0000 mgr.x (mgr.24542) 1806 : cluster [DBG] pgmap v2302: 41 pgs: 32 unknown, 9 active+clean; 4.6 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2021-07-31T02:29:29.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:29 smithi045 conmon[33655]: audit 2021-07-31T02:29:28.376948+0000 mon.a (mon.0) 2116 : audit 2021-07-31T02:29:29.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:29 smithi045 conmon[33655]: [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-7","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:29:29.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:29 smithi045 conmon[33655]: cluster 2021-07-31T02:29:28.377247+0000 mon.a (mon.0) 2117 : cluster [DBG] osdmap e1188: 8 total, 8 up, 8 in 2021-07-31T02:29:29.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:29 smithi045 conmon[33655]: cluster 2021-07-31T02:29:28.403041+0000 mgr.x (mgr.24542) 1806 : cluster [DBG] pgmap v2302: 41 pgs: 32 unknown, 9 active+clean; 4.6 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2021-07-31T02:29:29.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:29 smithi045 conmon[33655]: 2021-07-31T02:29:29.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:29 smithi045 conmon[39718]: audit 2021-07-31T02:29:28.376948+0000 mon.a (mon.0) 2116 2021-07-31T02:29:29.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:29 smithi045 conmon[39718]: : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-7","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:29:29.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:29 smithi045 conmon[39718]: cluster 2021-07-31T02:29:28.377247+0000 mon.a (mon.0) 2117 : cluster [DBG] osdmap e1188: 8 total, 8 up, 8 in 2021-07-31T02:29:29.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:29 smithi045 conmon[39718]: cluster 2021-07-31T02:29:28.403041+0000 mgr.x (mgr.24542) 1806 : cluster [DBG] pgmap v2302: 41 pgs: 32 unknown, 9 active+clean; 4.6 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2021-07-31T02:29:30.405 INFO:tasks.workunit.client.0.smithi045.stdout: wrote [731545~94551,9617137~87744,10341541~87580,11378541~55249,13556063~71302,14037258~96347,16155400~96284,16639803~34633,17498941~68426,19815860~3906] 2021-07-31T02:29:30.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:30 smithi154 conmon[34553]: cluster 2021-07-31T02:29:29.384787+0000 mon.a (mon.0) 2118 : cluster [DBG] osdmap e1189: 8 total, 8 up, 8 in 2021-07-31T02:29:30.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:30 smithi045 conmon[39718]: cluster 2021-07-31T02:29:29.384787+0000 mon.a ( 2021-07-31T02:29:30.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:30 smithi045 conmon[39718]: mon.0) 2118 : cluster [DBG] osdmap e1189: 8 total, 8 up, 8 in 2021-07-31T02:29:30.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:30 smithi045 conmon[33655]: cluster 2021-07-31T02:29:29.384787+0000 mon.a ( 2021-07-31T02:29:30.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:30 smithi045 conmon[33655]: mon.0) 2118 : cluster [DBG] osdmap e1189: 8 total, 8 up, 8 in 2021-07-31T02:29:31.407 INFO:tasks.workunit.client.0.smithi045.stdout: wrote [0~20971520] 2021-07-31T02:29:31.415 INFO:tasks.workunit.client.0.smithi045.stdout: diff was [0~20971520] 2021-07-31T02:29:31.416 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] DiffIterateTest/1.DiffIterate (5021 ms) 2021-07-31T02:29:31.417 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] DiffIterateTest/1.DiffIterateDiscard 2021-07-31T02:29:31.417 INFO:tasks.workunit.client.0.smithi045.stdout:using old format 2021-07-31T02:29:31.430 INFO:tasks.workunit.client.0.smithi045.stdout:iterate_cb 0~4194304 2021-07-31T02:29:31.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:31 smithi154 conmon[34553]: cluster 2021-07-31T02:29:30.386570+0000 mon.a (mon.0) 2119 : cluster [DBG] osdmap e1190: 8 total, 8 up, 8 in 2021-07-31T02:29:31.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:31 smithi154 conmon[34553]: 2021-07-31T02:29:31.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:31 smithi154 conmon[34553]: cluster 2021-07-31T02:29:30.403404+0000 mgr.x (mgr.24542) 1807 : cluster [DBG] pgmap v2305: 41 pgs: 41 active+clean; 4.6 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:29:31.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:31 smithi045 conmon[33655]: cluster 2021-07-31T02:29:30.386570+0000 mon.a (mon.0) 2119 2021-07-31T02:29:31.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:31 smithi045 conmon[33655]: : cluster [DBG] osdmap e1190: 8 total, 8 up, 8 in 2021-07-31T02:29:31.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:31 smithi045 conmon[33655]: cluster 2021-07-31T02:29:30.403404+0000 mgr.x (mgr.24542) 1807 : cluster [DBG] pgmap v2305: 41 pgs: 41 active+clean; 4.6 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:29:31.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:31 smithi045 conmon[39718]: cluster 2021-07-31T02:29:30.386570+0000 mon.a (mon 2021-07-31T02:29:31.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:31 smithi045 conmon[39718]: .0) 2119 : cluster [DBG] osdmap e1190: 8 total, 8 up, 8 in 2021-07-31T02:29:31.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:31 smithi045 conmon[39718]: cluster 2021- 2021-07-31T02:29:31.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:31 smithi045 conmon[39718]: 07-31T02:29:30.403404+0000 mgr.x (mgr.24542) 1807 : cluster [DBG] pgmap v2305: 41 pgs: 41 active+clean; 4.6 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:29:31.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:31 smithi045 conmon[39718]: 2021-07-31T02:29:32.413 INFO:tasks.workunit.client.0.smithi045.stdout:iterate_cb 0~4194304 2021-07-31T02:29:32.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: debug 2021-07-31T02:29:32.222+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:29:32.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:29:32.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:29:32.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: yield 2021-07-31T02:29:32.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:29:32.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:29:32.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:29:32.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:29:32.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:29:32.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:29:32.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:29:32.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:29:32.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:29:32.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: task.result() 2021-07-31T02:29:32.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:29:32.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: debug 2021-07-31T02:29:32.234+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:29:32.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:29:32.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:29:32.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: yield 2021-07-31T02:29:32.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:29:32.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:29:32.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:29:32.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:29:32.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:29:32.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:29:32.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:29:32.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:29:32.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:29:32.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: task.result() 2021-07-31T02:29:32.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:29:32.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: debug 2021-07-31T02:29:32.260+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:29:32.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:29:32.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:29:32.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: yield 2021-07-31T02:29:32.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:29:32.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:29:32.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:29:32.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:29:32.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:29:32.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: cluster 2021-07-31T02:29:31.399380+0000 mon.a (mon.0) 2120 : cluster [DBG] 2021-07-31T02:29:32.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: osdmap e1191: 8 total, 8 up, 8 in 2021-07-31T02:29:32.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: audit 2021-07-31T02:29:32.177360+0000 mon.b (mon.2) 285 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:29:32.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: audit 2021-07-31T02:29:32.178770+0000 mon.b (mon.2) 286 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:29:32.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: 2021-07-31T02:29:32.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: audit 2021-07-31T02:29:32.179810+0000 mon.b (mon.2) 287 : audit [INF] 2021-07-31T02:29:32.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:29:32.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: cephadm 2021-07-31T02: 2021-07-31T02:29:32.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: 29:32.180571+0000 mgr.x (mgr.24542) 1808 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:29:32.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: cephadm 2021-07 2021-07-31T02:29:32.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: -31T02:29:32.223679+0000 mgr.x (mgr.24542) 1809 2021-07-31T02:29:32.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: : cephadm [ERR] 2021-07-31T02:29:32.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:29:32.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:29:32.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: yield 2021-07-31T02:29:32.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:29:32.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:29:32.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:29:32.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:29:32.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:29:32.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:29:32.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:29:32.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:29:32.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:29:32.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: task.result() 2021-07-31T02:29:32.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:29:32.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: cephadm 2021-07-31T02:29:32.224190+0000 mgr.x (mgr.24542) 1810 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:29:32.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: cephadm 2021-07-31T02:29:32.235158+0000 mgr.x (mgr.24542) 1811 : cephadm [ERR] 2021-07-31T02:29:32.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:29:32.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:29:32.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: yield 2021-07-31T02:29:32.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:29:32.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:29:32.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:29:32.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:29:32.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:29:32.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:29:32.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:29:32.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:29:32.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:29:32.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: task.result() 2021-07-31T02:29:32.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:29:32.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: debug 2021-07-31T02:29:32.260+0000 7fcb9195d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:29:32.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:29:32.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:29:32.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: yield 2021-07-31T02:29:32.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:29:32.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:29:32.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:29:32.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:29:32.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:29:32.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:29:32.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:29:32.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:29:32.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:29:32.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: task.result() 2021-07-31T02:29:32.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:29:32.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: 2021-07-31T02:29:32.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:29:32.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: 2021-07-31T02:29:32.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:29:32.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:29:32.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:29:32.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:29:32.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:29:32.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:29:32.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:29:32.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:29:32.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:29:32.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:29:32.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:29:32.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:29:32.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:29:32.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:29:32.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:29:32.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:29:32.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:29:32.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:29:32.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:29:32.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:29:32.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:29:32.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: task.result() 2021-07-31T02:29:32.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:29:32.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: cephadm 2021-07-31T02:29:32.236137+0000 mgr.x (mgr.24542) 1812 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:29:32.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: cephadm 2021-07-31T02:29:32.261072+0000 mgr.x (mgr.24542) 1813 : cephadm [ERR] 2021-07-31T02:29:32.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:29:32.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:29:32.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: yield 2021-07-31T02:29:32.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:29:32.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:29:32.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:29:32.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:29:32.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:29:32.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:29:32.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:29:32.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:29:32.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:29:32.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: task.result() 2021-07-31T02:29:32.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:29:32.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: cephadm 2021-07-31T02:29:32.261505+0000 mgr.x (mgr.24542) 1814 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:29:32.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:29:32.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:29:32.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: yield 2021-07-31T02:29:32.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:29:32.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:29:32.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:29:32.541 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:29:32.541 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:29:32.541 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:29:32.541 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:29:32.542 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:29:32.542 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:29:32.542 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: task.result() 2021-07-31T02:29:32.542 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:29:32.542 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: 2021-07-31T02:29:32.543 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:29:32.543 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: 2021-07-31T02:29:32.543 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:29:32.544 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:29:32.544 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: raise 2021-07-31T02:29:32.544 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:29:32.544 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:29:32.545 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:29:32.545 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:29:32.545 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:29:32.545 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: debug 2021-07-31T02:29:32.269+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:29:32.546 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:29:32.546 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:29:32.546 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: yield 2021-07-31T02:29:32.546 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:29:32.546 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:29:32.547 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:29:32.547 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:29:32.547 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:29:32.547 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:29:32.548 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:29:32.548 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:29:32.548 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:29:32.548 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: task.result() 2021-07-31T02:29:32.548 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:29:32.549 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: debug 2021-07-31T02:29:32.269+0000 7fcb9215e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:29:32.549 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:29:32.549 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:29:32.549 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: yield 2021-07-31T02:29:32.550 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:29:32.550 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:29:32.550 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:29:32.550 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:29:32.551 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:29:32.551 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:29:32.551 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:29:32.551 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:29:32.551 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:29:32.552 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: task.result() 2021-07-31T02:29:32.552 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:29:32.552 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: 2021-07-31T02:29:32.552 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:29:32.552 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: 2021-07-31T02:29:32.553 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:29:32.553 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:29:32.553 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:29:32.554 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:29:32.554 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:29:32.554 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:29:32.554 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:29:32.555 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:29:32.555 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:29:32.555 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:29:32.555 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:29:32.555 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:29:32.556 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:29:32.556 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:29:32.556 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:29:32.556 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:29:32.556 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:29:32.557 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:29:32.557 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: raise 2021-07-31T02:29:32.557 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:29:32.557 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:29:32.558 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:29:32.558 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:29:32.558 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:29:32.558 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: cephadm 2021-07-31T02 2021-07-31T02:29:32.559 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: :29:32.270178+0000 mgr.x (mgr.24542) 1815 : cephadm [ERR] 2021-07-31T02:29:32.559 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:29:32.559 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:29:32.559 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: yield 2021-07-31T02:29:32.560 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:29:32.560 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:29:32.560 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:29:32.560 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:29:32.560 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:29:32.561 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:29:32.561 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:29:32.561 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:29:32.561 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:29:32.562 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: task.result() 2021-07-31T02:29:32.562 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:29:32.562 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:29:32.563 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:29:32.563 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:29:32.563 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:29:32.563 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:29:32.564 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:29:32.564 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:29:32.564 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:29:32.564 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:29:32.565 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:29:32.565 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:29:32.565 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:29:32.565 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:29:32.565 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:29:32.566 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:29:32.566 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:29:32.566 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:29:32.566 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: raise 2021-07-31T02:29:32.566 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:29:32.567 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:29:32.567 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:29:32.567 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:29:32.567 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:29:32.568 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: cephadm 2021-07-31T02:29:32.270586+0000 mgr.x (mgr.24542) 1816 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:29:32.568 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:29:32.568 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:29:32.568 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: yield 2021-07-31T02:29:32.569 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:29:32.569 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:29:32.569 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:29:32.570 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:29:32.570 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:29:32.570 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:29:32.571 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:29:32.571 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:29:32.571 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:29:32.571 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: task.result() 2021-07-31T02:29:32.572 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:29:32.572 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: 2021-07-31T02:29:32.572 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:29:32.572 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: 2021-07-31T02:29:32.572 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:29:32.573 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:29:32.573 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:29:32.573 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:29:32.573 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:29:32.573 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:29:32.574 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:29:32.574 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:29:32.574 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:29:32.574 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:29:32.575 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:29:32.575 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:29:32.575 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:29:32.575 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:29:32.575 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:29:32.576 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:29:32.576 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:29:32.576 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:29:32.576 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: raise 2021-07-31T02:29:32.576 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:29:32.577 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:29:32.577 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:29:32.577 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:29:32.577 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:32 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:29:32.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: cluster 2021-07-31T02:29:32.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: 2021-07-31T02:29:32.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: 2021-07-31T02:29:31.399380+0000 2021-07-31T02:29:32.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: mon.a (mon.0) 2120 : cluster [DBG] osdmap e1191: 8 total, 8 up, 8 in 2021-07-31T02:29:32.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: audit 2021-07-31T02:29:32.177360+0000 mon.b 2021-07-31T02:29:32.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: (mon.2) 285 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:29:32.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: audit 2021-07- 2021-07-31T02:29:32.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: 31T02:29:32.178770+0000 mon.b (mon 2021-07-31T02:29:32.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: .2) 286 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:29:32.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: audit 2021- 2021-07-31T02:29:32.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: 07-31T02:29:32.179810+0000 mon.b (mon.2) 287 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:29:32.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: cephadm 2021-07-31T02:29:32.180571+0000 mgr.x (mgr. 2021-07-31T02:29:32.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: 24542) 1808 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:29:32.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: cephadm 2021-07 2021-07-31T02:29:32.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: -31T02:29:32.223679+0000 mgr.x ( 2021-07-31T02:29:32.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: mgr.24542) 1809 : cephadm [ERR] 2021-07-31T02:29:32.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:29:32.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:29:32.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: yield 2021-07-31T02:29:32.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:29:32.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:29:32.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:29:32.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:29:32.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:29:32.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:29:32.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:29:32.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:29:32.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:29:32.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: task.result() 2021-07-31T02:29:32.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:29:32.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: cephadm 2021-07-31T02:29:32.224190+0000 mgr.x (mgr.24542) 1810 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:29:32.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: cephadm 2021-07-31T02:29 2021-07-31T02:29:32.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: :32.235158+0000 mgr.x (mgr.24542) 1811 : cephadm [ERR] 2021-07-31T02:29:32.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:29:32.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:29:32.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: yield 2021-07-31T02:29:32.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:29:32.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:29:32.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:29:32.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:29:32.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:29:32.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:29:32.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:29:32.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:29:32.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:29:32.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: task.result() 2021-07-31T02:29:32.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:29:32.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: cephadm 2021-07-31T02:29:32.236137+0000 mgr.x (mgr.24542) 1812 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:29:32.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: cephadm 2021-07-31T02:29:32.261072+0000 mgr.x (mgr.24542) 1813 : cephadm [ERR] 2021-07-31T02:29:32.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: 2021-07-31T02:29:32.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:29:32.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:29:32.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: yield 2021-07-31T02:29:32.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:29:32.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:29:32.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:29:32.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:29:32.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:29:32.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:29:32.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:29:32.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:29:32.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:29:32.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: task.result() 2021-07-31T02:29:32.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:29:32.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: cephadm 2021 2021-07-31T02:29:32.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: -07-31T02:29:32.261505+0000 mgr.x (mgr.24542) 1814 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:29:32.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:29:32.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:29:32.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: yield 2021-07-31T02:29:32.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:29:32.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:29:32.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:29:32.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:29:32.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:29:32.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:29:32.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:29:32.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:29:32.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:29:32.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: task.result() 2021-07-31T02:29:32.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:29:32.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: cluster 2021-07-31T02:29:31.399380+0000 mon.a (mon 2021-07-31T02:29:32.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: .0) 2120 : cluster [DBG] osdmap e1191: 8 total, 8 up, 8 in 2021-07-31T02:29:32.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: audit 2021-07-31T02:29: 2021-07-31T02:29:32.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: 32.177360+0000 mon.b (mon.2) 285 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:29:32.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T02:29:32.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: 31T02:29:32.178770+0000 mon.b (mon.2 2021-07-31T02:29:32.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: ) 286 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:29:32.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: audit 2021-07-31T02:29:32.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: 2021-07-31T02:29:32.179810+0000 mon.b ( 2021-07-31T02:29:32.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: mon.2) 287 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:29:32.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: cephadm 2021-07-31T02 2021-07-31T02:29:32.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: :29:32.180571+0000 mgr.x (mgr.24542) 1808 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:29:32.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: 2021-07-31T02:29:32.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: cephadm 2021-07-31T 2021-07-31T02:29:32.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: 02:29:32.223679+0000 mgr.x (mgr.24542 2021-07-31T02:29:32.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: ) 1809 : cephadm [ERR] 2021-07-31T02:29:32.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:29:32.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:29:32.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: yield 2021-07-31T02:29:32.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:29:32.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:29:32.603 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:29:32.603 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:29:32.603 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:29:32.603 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:29:32.603 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:29:32.604 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:29:32.604 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:29:32.604 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: task.result() 2021-07-31T02:29:32.604 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:29:32.604 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: cephadm 2021-07-31T02:29:32.224190+0000 mgr.x (mgr.24542) 1810 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:29:32.605 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: cephadm 2021-07-31T02:29:32.235158+0000 mgr.x (mgr.24542) 1811 2021-07-31T02:29:32.605 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: : cephadm [ERR] 2021-07-31T02:29:32.605 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:29:32.605 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:29:32.606 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: yield 2021-07-31T02:29:32.606 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:29:32.606 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:29:32.606 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:29:32.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: 2021-07-31T02:29:32.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:29:32.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: 2021-07-31T02:29:32.608 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:29:32.608 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:29:32.608 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:29:32.608 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:29:32.608 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:29:32.609 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:29:32.609 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:29:32.609 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:29:32.609 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:29:32.610 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:29:32.610 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:29:32.610 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:29:32.610 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:29:32.611 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:29:32.611 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:29:32.611 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:29:32.611 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:29:32.611 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:29:32.612 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: raise 2021-07-31T02:29:32.612 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:29:32.612 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:29:32.612 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:29:32.612 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:29:32.613 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:29:32.613 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: cephadm 2021-07-31T02:29:32.270178+0000 mgr.x (mgr.24542) 1815 : cephadm [ERR] 2021-07-31T02:29:32.613 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:29:32.613 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:29:32.614 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: yield 2021-07-31T02:29:32.614 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:29:32.614 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:29:32.614 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:29:32.614 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:29:32.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:29:32.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:29:32.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:29:32.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:29:32.616 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:29:32.616 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:29:32.616 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:29:32.617 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:29:32.617 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:29:32.617 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:29:32.617 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:29:32.617 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: task.result() 2021-07-31T02:29:32.618 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:29:32.618 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: cephadm 2021-07-31T02:29:32.236137+0000 mgr.x (mgr.24542) 1812 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:29:32.618 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: cephadm 2021-07-31T02:29:32.261072+0000 mgr.x (mgr.24542) 1813 : cephadm [ERR] 2021-07-31T02:29:32.618 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:29:32.619 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:29:32.619 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: yield 2021-07-31T02:29:32.619 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:29:32.619 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:29:32.619 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:29:32.620 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:29:32.620 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:29:32.620 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:29:32.620 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:29:32.620 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:29:32.621 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:29:32.621 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: task.result() 2021-07-31T02:29:32.621 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:29:32.621 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: cephadm 2021-07-31T02:29:32.261505+0000 mgr.x (mgr.24542) 1814 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:29:32.622 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:29:32.622 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:29:32.622 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: yield 2021-07-31T02:29:32.622 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:29:32.622 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:29:32.623 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:29:32.623 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:29:32.623 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:29:32.623 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:29:32.624 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:29:32.624 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:29:32.624 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:29:32.624 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: task.result() 2021-07-31T02:29:32.624 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:29:32.625 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: 2021-07-31T02:29:32.625 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: task.result() 2021-07-31T02:29:32.626 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:29:32.626 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: cephadm 2021-07-31T02:29:32.626 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: 2021-07-31T02:29:32.270586+0000 mgr.x (mgr.24542) 1816 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:29:32.627 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:29:32.627 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:29:32.627 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: yield 2021-07-31T02:29:32.627 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:29:32.627 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:29:32.628 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:29:32.628 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:29:32.628 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:29:32.628 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:29:32.629 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:29:32.629 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:29:32.629 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:29:32.629 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: task.result() 2021-07-31T02:29:32.630 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:29:32.630 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: 2021-07-31T02:29:32.630 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:29:32.630 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: 2021-07-31T02:29:32.631 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:29:32.631 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:29:32.631 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:29:32.631 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:29:32.631 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:29:32.632 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:29:32.632 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:29:32.632 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:29:32.632 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:29:32.632 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:29:32.633 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:29:32.633 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:29:32.633 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:29:32.634 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:29:32.634 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:29:32.634 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:29:32.635 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:29:32.636 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:29:32.636 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: raise 2021-07-31T02:29:32.636 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:29:32.636 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:29:32.637 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:29:32.637 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:29:32.637 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:29:32.638 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:29:32.638 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: 2021-07-31T02:29:32.639 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:29:32.639 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:29:32.639 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:29:32.639 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:29:32.640 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:29:32.640 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:29:32.640 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:29:32.640 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:29:32.641 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:29:32.641 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:29:32.641 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:29:32.641 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:29:32.641 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:29:32.642 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:29:32.642 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:29:32.642 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:29:32.642 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:29:32.643 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:29:32.643 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: raise 2021-07-31T02:29:32.643 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:29:32.643 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:29:32.643 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:29:32.644 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:29:32.644 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:29:32.644 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: cephadm 2021-07-31T 2021-07-31T02:29:32.644 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: 02:29:32.270178+0000 mgr.x (mgr.24542) 1815 : cephadm [ERR] 2021-07-31T02:29:32.645 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:29:32.645 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:29:32.645 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: yield 2021-07-31T02:29:32.645 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:29:32.645 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:29:32.646 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:29:32.646 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:29:32.646 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:29:32.646 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:29:32.647 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:29:32.647 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:29:32.647 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:29:32.647 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: task.result() 2021-07-31T02:29:32.648 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:29:32.648 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: cephadm 2021-07-31T02:29:32.270586+0000 mgr.x (mgr.24542) 1816 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:29:32.648 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:29:32.648 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:29:32.648 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: yield 2021-07-31T02:29:32.649 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:29:32.649 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:29:32.649 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:29:32.649 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:29:32.649 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:29:32.650 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:29:32.650 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:29:32.650 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:29:32.650 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:29:32.651 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: task.result() 2021-07-31T02:29:32.651 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:29:32.651 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: 2021-07-31T02:29:32.651 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:29:32.651 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: 2021-07-31T02:29:32.652 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:29:32.652 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:29:32.652 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:29:32.652 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:29:32.653 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:29:32.653 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:29:32.653 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:29:32.653 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:29:32.654 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:29:32.654 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:29:32.654 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:29:32.654 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:29:32.655 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:29:32.655 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:29:32.655 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:29:32.655 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:29:32.656 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:29:32.656 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:29:32.656 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: raise 2021-07-31T02:29:32.656 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:29:32.656 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:29:32.657 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:29:32.658 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:29:32.658 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:32 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:29:33.420 INFO:tasks.workunit.client.0.smithi045.stdout:iterate_cb 0~4194304 2021-07-31T02:29:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:33 smithi154 conmon[34553]: cluster 2021-07-31T02:29:32.400422+0000 mon.a (mon.0) 2121 : cluster [DBG] 2021-07-31T02:29:33.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:33 smithi154 conmon[34553]: osdmap e1192: 8 total, 8 up, 8 in 2021-07-31T02:29:33.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:33 smithi154 conmon[34553]: cluster 2021-07-31T02:29:32.403775+0000 mgr.x (mgr.24542) 1817 : cluster [DBG] pgmap v2308: 41 pgs: 41 active+clean; 4.6 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:29:33.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:33 smithi045 conmon[33655]: cluster 2021-07-31T02:29:32.400422+0000 mon.a 2021-07-31T02:29:33.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:33 smithi045 conmon[33655]: (mon.0) 2121 : cluster [DBG] osdmap e1192: 8 total, 8 up, 8 in 2021-07-31T02:29:33.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:33 smithi045 conmon[33655]: cluster 2021-07-31T02:29:32.403775+0000 mgr.x (mgr.24542) 1817 : cluster [DBG] pgmap v2308: 41 pgs: 41 active+clean; 4.6 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:29:33.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:33 smithi045 conmon[33655]: 2021-07-31T02:29:33.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:29:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:29:33.582Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:29:33.778 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:29:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:29:33.582Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:29:33.778 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:29:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:29:33.582Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:29:33.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:33 smithi045 conmon[39718]: cluster 2021-07-31T02:29:32.400422+0000 mon.a ( 2021-07-31T02:29:33.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:33 smithi045 conmon[39718]: mon.0) 2121 : cluster [DBG] osdmap e1192: 8 total, 8 up, 8 in 2021-07-31T02:29:33.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:33 smithi045 conmon[39718]: cluster 2021-07-31T02: 2021-07-31T02:29:33.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:33 smithi045 conmon[39718]: 29:32.403775+0000 mgr.x (mgr.24542) 1817 : cluster [DBG] 2021-07-31T02:29:33.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:33 smithi045 conmon[39718]: pgmap v2308: 41 pgs: 41 active+clean; 4.6 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:29:34.429 INFO:tasks.workunit.client.0.smithi045.stdout:iterate_cb 0~4194304 2021-07-31T02:29:34.432 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] DiffIterateTest/1.DiffIterateDiscard (3015 ms) 2021-07-31T02:29:34.432 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] DiffIterateTest/1.DiffIterateStress 2021-07-31T02:29:34.432 INFO:tasks.workunit.client.0.smithi045.stdout:using old format 2021-07-31T02:29:34.487 INFO:tasks.workunit.client.0.smithi045.stdout: i=0 exists [60903552~566944,106443276~5938646,132323939~1132981,299015817~4822889,318639955~6984409,329190096~5766166,341603328~6155127,377630547~5045887,389025719~5856037,399101982~1412477] wrote [60903552~566944,106443276~5938646,132323939~1132981,299015817~4822889,318639955~6984409,329190096~5766166,341603328~6155127,377630547~5045887,389025719~5856037,399101982~1412477] 2021-07-31T02:29:34.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:34 smithi154 conmon[34553]: cluster 2021-07-31T02:29:33.410925+0000 mon.a (mon.0) 2122 : cluster [DBG] 2021-07-31T02:29:34.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:34 smithi154 conmon[34553]: osdmap e1193: 8 total, 8 up, 8 in 2021-07-31T02:29:34.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:34 smithi045 conmon[33655]: cluster 2021-07-31T02:29:33.410925+0000 mon.a (mon.0) 2122 2021-07-31T02:29:34.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:34 smithi045 conmon[33655]: : cluster [DBG] osdmap e1193: 8 total, 8 up, 8 in 2021-07-31T02:29:34.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:34 smithi045 conmon[39718]: cluster 2021-07-31T02:29:33.410925+0000 mon.a (mon.0) 2122 : 2021-07-31T02:29:34.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:34 smithi045 conmon[39718]: cluster [DBG] osdmap e1193: 8 total, 8 up, 8 in 2021-07-31T02:29:35.464 INFO:tasks.workunit.client.0.smithi045.stdout: i=1 exists [16872282~2127957,60903552~566944,99407305~1655876,106443276~5938646,131940663~2851606,149733912~3458894,193138829~5959135,235395410~3378458,272159228~7769835,283712552~6641625,299015817~4822889,318639955~6984409,329190096~6280397,341603328~6155127,377630547~5045887,389025719~5856037,399101982~1412477] wrote [16872282~2127957,99407305~1655876,131940663~2851606,149733912~3458894,193138829~5959135,235395410~3378458,272159228~7769835,283712552~6641625,334025591~201047,334387978~1082515] 2021-07-31T02:29:35.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:35 smithi154 conmon[34553]: cluster 2021-07-31T02:29:34.404332+0000 mgr.x (mgr.24542) 1818 : 2021-07-31T02:29:35.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:35 smithi154 conmon[34553]: cluster [DBG] pgmap v2310: 41 pgs: 41 active+clean; 19 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.2 KiB/s rd, 325 KiB/s wr, 18 op/s 2021-07-31T02:29:35.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:35 smithi154 conmon[34553]: cluster 2021-07-31T02:29:34.421417+0000 mon.a (mon.0) 2123 : cluster [DBG] osdmap e1194: 8 total, 8 up, 8 in 2021-07-31T02:29:35.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:35 smithi045 conmon[33655]: cluster 2021-07-31T02:29:34.404332+0000 mgr.x (mgr.24542) 1818 2021-07-31T02:29:35.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:35 smithi045 conmon[33655]: : cluster [DBG] pgmap v2310: 41 pgs: 41 active+clean; 19 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.2 KiB/s rd, 325 KiB/s wr, 18 op/s 2021-07-31T02:29:35.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:35 smithi045 conmon[33655]: cluster 2021-07-31T02:29:34.421417+0000 mon.a (mon.0) 2123 : cluster [DBG] osdmap e1194: 8 total, 8 up, 8 in 2021-07-31T02:29:35.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:35 smithi045 conmon[39718]: cluster 2021-07-31T02:29:34.404332+0000 mgr.x 2021-07-31T02:29:35.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:35 smithi045 conmon[39718]: (mgr.24542) 1818 : cluster [DBG] pgmap v2310: 41 pgs: 41 active+clean; 19 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.2 KiB/s rd, 325 KiB/s wr, 18 op/s 2021-07-31T02:29:35.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:35 smithi045 conmon[39718]: cluster 2021-07-31T02:29:35.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:35 smithi045 conmon[39718]: 2021-07-31T02:29:34.421417+0000 mon.a ( 2021-07-31T02:29:35.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:35 smithi045 conmon[39718]: mon.0) 2123 : cluster [DBG] 2021-07-31T02:29:35.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:35 smithi045 conmon[39718]: osdmap e1194: 8 total, 8 up, 8 in 2021-07-31T02:29:36.502 INFO:tasks.workunit.client.0.smithi045.stdout: i=2 exists [5606294~4571378,16872282~2127957,21188995~6712523,45082762~8536712,60903552~566944,99407305~1655876,106443276~5938646,128819852~5972417,149733912~3458894,169554965~8455986,193138829~5959135,235395410~10838444,258954181~2599657,272159228~7769835,283712552~6641625,299015817~4822889,309596768~8066536,318639955~6984409,329190096~6280397,341603328~6155127,377630547~5045887,389025719~5856037,399101982~1412477] wrote [5606294~4571378,21188995~6712523,45082762~8536712,128819852~3166277,169554965~8455986,238198801~8035053,258954181~2599657,309596768~8066536] 2021-07-31T02:29:36.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:36 smithi154 conmon[34553]: cluster 2021-07-31T02:29:35.423040+0000 mon.a (mon.0) 2124 : cluster [DBG] 2021-07-31T02:29:36.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:36 smithi154 conmon[34553]: osdmap e1195: 8 total, 8 up, 8 in 2021-07-31T02:29:36.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:36 smithi045 conmon[33655]: cluster 2021-07-31T02:29:35.423040+0000 mon.a (mon.0) 2124 2021-07-31T02:29:36.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:36 smithi045 conmon[33655]: : cluster [DBG] osdmap e1195: 8 total, 8 up, 8 in 2021-07-31T02:29:36.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:36 smithi045 conmon[39718]: cluster 2021-07-31T02:29:35.423040+0000 mon.a (mon. 2021-07-31T02:29:36.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:36 smithi045 conmon[39718]: 0) 2124 : cluster [DBG] osdmap e1195: 8 total, 8 up, 8 in 2021-07-31T02:29:37.331 INFO:tasks.workunit.client.0.smithi045.stdout: i=3 exists [5606294~4571378,16872282~2127957,21188995~6712523,45082762~9063554,58811632~3451278,72290679~5049348,83937430~5723037,99407305~1655876,106443276~5938646,128819852~5972417,149733912~3458894,158823321~1926466,169554965~8455986,183119390~3236677,193138829~5959135,225143787~6150319,235395410~10838444,258954181~2599657,272159228~7769835,283712552~6641625,299015817~4822889,309596768~8066536,318639955~6984409,329190096~6280397,341603328~6155127,377630547~9050775,389025719~5856037,399101982~1412477] wrote [25671399~1051292,49060091~5086225,58811632~3451278,72290679~5049348,83937430~5723037,158823321~1926466,183119390~3236677,225143787~6150319,331500821~3362951,381782508~4898814] 2021-07-31T02:29:37.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:37 smithi154 conmon[34553]: cluster 2021-07-31T02:29:36.404758+0000 mgr.x (mgr.24542) 1819 : cluster [DBG] pgmap v2313: 41 pgs: 41 active+clean; 19 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.2 KiB/s rd, 326 KiB/s wr, 18 op/s 2021-07-31T02:29:37.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:37 smithi154 conmon[34553]: cluster 2021-07-31T 2021-07-31T02:29:37.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:37 smithi154 conmon[34553]: 02:29:36.435309+0000 mon.a (mon.0) 2125 : cluster [DBG] osdmap e1196: 8 total, 8 up, 8 in 2021-07-31T02:29:37.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:37 smithi154 conmon[34553]: cluster 2021-07-31T02:29:37.281107+0000 mon.a (mon.0) 2126 : cluster [DBG] osdmap e1197: 8 total, 8 up, 8 in 2021-07-31T02:29:37.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:37 smithi045 conmon[39718]: cluster 2021-07-31T02:29 2021-07-31T02:29:37.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:37 smithi045 conmon[39718]: :36.404758+0000 mgr.x (mgr.24542) 1819 : cluster [DBG] pgmap v2313: 41 pgs: 41 active+clean; 19 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.2 KiB/s rd, 326 KiB/s wr, 18 op/s 2021-07-31T02:29:37.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:37 smithi045 conmon[39718]: cluster 2021-07-31T02:29:37.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:37 smithi045 conmon[39718]: 2021-07-31T02: 2021-07-31T02:29:37.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:37 smithi045 conmon[39718]: 29:36.435309+0000 mon.a 2021-07-31T02:29:37.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:37 smithi045 conmon[39718]: (mon.0) 2125 : cluster 2021-07-31T02:29:37.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:37 smithi045 conmon[39718]: [DBG] osdmap e1196: 8 total, 8 up, 8 in 2021-07-31T02:29:37.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:37 smithi045 conmon[39718]: cluster 2021-07 2021-07-31T02:29:37.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:37 smithi045 conmon[39718]: -31T02:29:37.281107 2021-07-31T02:29:37.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:37 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2126 2021-07-31T02:29:37.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:37 smithi045 conmon[39718]: : cluster [DBG] osdmap e1197: 8 total, 8 up, 8 in 2021-07-31T02:29:37.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:37 smithi045 conmon[33655]: cluster 2021-07-31T02:29:36.404758+0000 mgr.x ( 2021-07-31T02:29:37.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:37 smithi045 conmon[33655]: mgr.24542) 1819 : cluster [DBG] pgmap v2313: 41 pgs: 41 active+clean; 19 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.2 KiB/s rd, 326 KiB/s wr, 18 op/s 2021-07-31T02:29:37.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:37 smithi045 conmon[33655]: cluster 2021-07-31T02:29:36.435309+0000 mon.a (mon.0) 2021-07-31T02:29:37.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:37 smithi045 conmon[33655]: 2125 : cluster [DBG] osdmap e1196: 8 total, 8 up, 8 in 2021-07-31T02:29:37.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:37 smithi045 conmon[33655]: cluster 2021-07- 2021-07-31T02:29:37.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:37 smithi045 conmon[33655]: 31T02:29:37.281107 2021-07-31T02:29:37.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:37 smithi045 conmon[33655]: +0000 mon.a (mon.0) 2126 : cluster 2021-07-31T02:29:37.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:37 smithi045 conmon[33655]: [DBG] osdmap e1197: 8 total, 8 up, 8 in 2021-07-31T02:29:38.366 INFO:tasks.workunit.client.0.smithi045.stdout: i=4 exists [5606294~6675943,16872282~2127957,21188995~9448254,34955369~7628581,45082762~9063554,58811632~3451278,72290679~9072303,83937430~5723037,99407305~1655876,106443276~5938646,126876605~7915664,148572550~4620256,158823321~1926466,169554965~8455986,183119390~3236677,193138829~5959135,218420359~6472794,225143787~6150319,235395410~10838444,258954181~2599657,272159228~7769835,283712552~6641625,299015817~4822889,309596768~8066536,318639955~6984409,328314017~7156476,341603328~6155127,377630547~9050775,389025719~5856037,399101982~1412477] wrote [6584165~5698072,23869329~6767920,34955369~7628581,74388130~6974852,126876605~7585545,148572550~1443955,218420359~6472794,328314017~6733473] 2021-07-31T02:29:39.349 INFO:tasks.workunit.client.0.smithi045.stdout: i=5 exists [5606294~6675943,13468904~2970358,16872282~2127957,21188995~12257359,34955369~7628581,45082762~9063554,58811632~3451278,72290679~9072303,83937430~8661233,99407305~1655876,106443276~5938646,123520397~11271872,148572550~4620256,158823321~1926466,169554965~8455986,183119390~3236677,189947221~9150743,218420359~6472794,225143787~6495485,235395410~10838444,258954181~2599657,267583087~12345976,283712552~6641625,299015817~4822889,309596768~8066536,318639955~6984409,328314017~7185057,341603328~6155127,377630547~9050775,389025719~5856037,399101982~1412477] wrote [13468904~2970358,28855707~4590647,84859790~7738873,123520397~3526901,189947221~7513901,229444717~2194555,267583087~7976115,328439221~7059853,389089295~3252481] 2021-07-31T02:29:39.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:39 smithi154 conmon[34553]: cluster 2021-07-31T02:29:39.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:39 smithi154 conmon[34553]: 2021-07-31T02:29: 2021-07-31T02:29:39.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:39 smithi154 conmon[34553]: 38.289938+0000 mon.a (mon.0) 2127 : cluster [DBG] osdmap e1198: 8 total, 8 up, 8 in 2021-07-31T02:29:39.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:39 smithi045 conmon[39718]: cluster 2021-07-31T02:29:38.289938+0000 mon.a (mon.0 2021-07-31T02:29:39.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:39 smithi045 conmon[39718]: ) 2127 : cluster [DBG] osdmap e1198: 8 total, 8 up, 8 in 2021-07-31T02:29:39.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:39 smithi045 conmon[33655]: cluster 2021-07-31T02:29:38.289938+0000 mon.a (mon.0 2021-07-31T02:29:39.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:39 smithi045 conmon[33655]: ) 2127 : cluster [DBG] osdmap e1198: 8 total, 8 up, 8 in 2021-07-31T02:29:40.352 INFO:tasks.workunit.client.0.smithi045.stdout: i=6 exists [5606294~7780689,13468904~2970358,16872282~2127957,21188995~12257359,34955369~7628581,45082762~9063554,55311859~6951051,72290679~9072303,83937430~8661233,99407305~1655876,104767910~7614012,123520397~11271872,148572550~9707130,158823321~1926466,169554965~8455986,183119390~3236677,189947221~10343013,210894309~4216555,218420359~6472794,225143787~6495485,235395410~10838444,251968052~4035487,258954181~2599657,267583087~12345976,283712552~6641625,299015817~4822889,309596768~8066536,318639955~6984409,328314017~7185057,341603328~6155127,350254010~6042664,371853644~5453194,377630547~9050775,389025719~5856037,399101982~1412477] wrote [11452040~1934943,26294986~5821958,55311859~5879995,104767910~3809014,152178802~6100878,196864129~3426105,210894309~4216555,251968052~4035487,350254010~6042664,371853644~5453194] 2021-07-31T02:29:40.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:40 smithi154 conmon[34553]: cluster 2021-07-31T02:29:38.405253+0000 mgr.x (mgr.24542) 1820 : cluster [DBG] pgmap v2317: 41 pgs: 41 active+clean; 19 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:29:40.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:40 smithi154 conmon[34553]: cluster 2021-07-31T02:29 2021-07-31T02:29:40.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:40 smithi154 conmon[34553]: :39.294728+0000 mon.a (mon.0) 2128 : cluster [DBG] osdmap e1199: 8 total, 8 up, 8 in 2021-07-31T02:29:40.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:40 smithi045 conmon[39718]: cluster 2021-07-31T02:29:38.405253+0000 mgr.x (mgr.24542) 2021-07-31T02:29:40.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:40 smithi045 conmon[39718]: 1820 : cluster [DBG] pgmap v2317: 41 pgs: 41 active+clean; 19 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:29:40.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:40 smithi045 conmon[39718]: cluster 2021-07-31T02:29:39.294728+0000 mon.a (mon.0) 2128 : cluster [DBG] osdmap e1199: 8 total, 8 up, 8 in 2021-07-31T02:29:40.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:40 smithi045 conmon[33655]: cluster 2021-07-31T02:29:38.405253+0000 mgr.x (mgr.24542) 1820 : cluster 2021-07-31T02:29:40.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:40 smithi045 conmon[33655]: [DBG] pgmap v2317: 41 pgs: 41 active+clean; 19 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:29:40.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:40 smithi045 conmon[33655]: cluster 2021-07-31T02:29:39.294728+0000 mon.a (mon.0) 2128 : cluster [DBG] osdmap e1199: 8 total, 8 up, 8 in 2021-07-31T02:29:41.358 INFO:tasks.workunit.client.0.smithi045.stdout: i=7 exists [5606294~7780689,13468904~2970358,16872282~2127957,21188995~12257359,34955369~7628581,45082762~9063554,55311859~6951051,72290679~9072303,83937430~8661233,99407305~1655876,104767910~7614012,121862248~595321,123520397~11271872,148572550~9707130,158823321~1926466,169554965~8455986,183119390~3236677,189947221~10343013,210894309~4216555,218420359~6472794,225143787~6495485,235395410~11746621,251968052~4035487,258954181~2599657,267583087~15254392,283712552~6641625,299015817~4822889,309201969~26297105,341603328~7006481,350254010~6042664,371853644~14827678,389025719~5856037,399101982~1412477] wrote [30150567~2461101,121862248~595321,131226927~2283373,242307709~4834322,276370439~6467040,309201969~4378168,316665131~4461057,323951090~4970510,343303830~5305979,373167876~5858624] 2021-07-31T02:29:41.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:41 smithi154 conmon[34553]: cluster 2021-07-31T02:29:40.300754+0000 mon.a (mon.0) 2129 : 2021-07-31T02:29:41.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:41 smithi154 conmon[34553]: cluster [DBG] osdmap e1200: 8 total, 8 up, 8 in 2021-07-31T02:29:41.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:41 smithi045 conmon[33655]: cluster 2021-07-31T02:29:40.300754+0000 mon.a (mon.0) 2129 : cluster [DBG] 2021-07-31T02:29:41.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:41 smithi045 conmon[33655]: osdmap e1200: 8 total, 8 up, 8 in 2021-07-31T02:29:41.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:41 smithi045 conmon[39718]: cluster 2021-07-31T02:29:40. 2021-07-31T02:29:41.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:41 smithi045 conmon[39718]: 300754+0000 mon.a (mon.0) 2129 : cluster [DBG] osdmap e1200: 8 total, 8 up, 8 in 2021-07-31T02:29:42.389 INFO:tasks.workunit.client.0.smithi045.stdout: i=8 exists [5606294~10832968,16872282~2127957,21188995~12257359,34955369~7628581,45082762~9063554,55311859~6951051,72290679~9072303,83937430~8661233,99407305~1655876,104767910~8483109,121862248~595321,123520397~11271872,148572550~9707130,158823321~1926466,169554965~12655371,183119390~3236677,189947221~10343013,210894309~4216555,218420359~6472794,225143787~21998244,251968052~4035487,258954181~2599657,267583087~22771090,299015817~4822889,309201969~26297105,341603328~7006481,350254010~6042664,371853644~14827678,389025719~5856037,399101982~1412477] wrote [8395080~6632173,75744729~2479958,84603480~5923746,108280977~4970042,175459682~6750654,228280200~7431300,279214629~6522465,325017034~7003364] 2021-07-31T02:29:42.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:42 smithi154 conmon[34553]: cluster 2021-07-31T02:29:40.405849+0000 mgr.x (mgr.24542) 1821 : cluster [DBG] pgmap v2320: 41 pgs: 41 active+clean; 285 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.1 KiB/s rd, 56 MiB/s wr, 38 op/s 2021-07-31T02:29:42.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:42 smithi154 conmon[34553]: cluster 2021-07-31T02:29:41.306860+0000 mon.a (mon.0) 2130 : cluster [DBG] osdmap e1201: 8 total, 8 up, 8 in 2021-07-31T02:29:42.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:42 smithi045 conmon[33655]: cluster 2021-07-31T02:29:40.405849+0000 mgr.x (mgr.24542) 1821 : cluster [DBG] pgmap v2320: 41 pgs: 41 active+clean; 285 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.1 KiB/s rd, 56 MiB/s wr, 38 op/s 2021-07-31T02:29:42.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:42 smithi045 conmon[33655]: cluster 2021-07-31T02:29:41.306860+0000 mon.a (mon.0) 2130 : cluster [DBG] osdmap e1201: 8 total, 8 up, 8 in 2021-07-31T02:29:42.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:42 smithi045 conmon[39718]: cluster 2021-07-31T02:29:40.405849+0000 mgr.x (mgr 2021-07-31T02:29:42.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:42 smithi045 conmon[39718]: .24542) 1821 : cluster [DBG] pgmap v2320: 41 pgs: 41 active+clean; 285 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.1 KiB/s rd, 56 MiB/s wr, 38 op/s 2021-07-31T02:29:42.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:42 smithi045 conmon[39718]: cluster 2021-07-31T02:29:41.306860+0000 mon.a (mon.0) 2130 : cluster [DBG] osdmap e1201: 8 total, 8 up, 8 in 2021-07-31T02:29:42.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:42 smithi045 conmon[39718]: 2021-07-31T02:29:42.880 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:29:43.160 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:29:43.160 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 44m ago 45m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:29:43.160 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 42m ago 45m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:29:43.161 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 42m ago 48m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:29:43.161 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 44m ago 50m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:29:43.161 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 44m ago 50m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:29:43.161 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 42m ago 48m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:29:43.161 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 44m ago 49m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:29:43.162 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 44m ago 45m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:29:43.162 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 42m ago 45m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:29:43.162 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 44m ago 48m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:29:43.162 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 44m ago 48m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:29:43.163 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 44m ago 47m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:29:43.163 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 44m ago 47m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:29:43.163 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 42m ago 47m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:29:43.163 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 42m ago 46m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:29:43.164 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 42m ago 46m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:29:43.164 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 42m ago 46m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:29:43.164 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 42m ago 45m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:29:43.368 INFO:tasks.workunit.client.0.smithi045.stdout: i=9 exists [1659779~14779483,16872282~2127957,21188995~12257359,34955369~7628581,45082762~10225917,55311859~6951051,72290679~9072303,83937430~8661233,99407305~1655876,104767910~8483109,121862248~595321,123520397~15032370,148572550~9707130,158823321~1926466,169554965~12655371,183119390~3236677,189947221~10343013,206899772~8211092,218420359~6472794,225143787~21998244,251968052~4035487,258954181~2599657,267583087~22771090,294412067~9426639,309201969~26297105,341603328~7006481,350254010~6042664,371853644~23028112,399101982~1412477] wrote [1659779~5974936,47472257~7836422,134375509~4177258,192668722~3390075,206899772~4640500,294412067~6778064,346764150~262474,385011769~6042995] 2021-07-31T02:29:43.525 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:29:43.526 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:29:43.526 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:29:43.526 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:29:43.527 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:29:43.527 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:29:43.527 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:29:43.527 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:29:43.528 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:29:43.528 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:29:43.528 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:29:43.528 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:29:43.528 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:29:43.529 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:29:43.529 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:29:43.530 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:29:43.530 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:29:43.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:43 smithi045 conmon[39718]: cluster 2021-07-31T02:29:42.312264+0000 mon.a (mon.0 2021-07-31T02:29:43.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:43 smithi045 conmon[39718]: ) 2131 : cluster [DBG] osdmap e1202: 8 total, 8 up, 8 in 2021-07-31T02:29:43.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:43 smithi045 conmon[33655]: cluster 2021-07-31T02:29:42.312264+0000 mon.a (mon. 2021-07-31T02:29:43.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:43 smithi045 conmon[33655]: 0) 2131 : cluster [DBG] osdmap e1202: 8 total, 8 up, 8 in 2021-07-31T02:29:43.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:43 smithi154 conmon[34553]: cluster 2021-07-31T02:29:42.312264+0000 mon.a (mon.0) 2131 : cluster [DBG] osdmap e1202: 8 total, 8 up, 8 in 2021-07-31T02:29:44.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:29:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:29:43.582Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:29:44.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:29:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:29:43.583Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:29:44.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:29:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:29:43.583Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:29:44.393 INFO:tasks.workunit.client.0.smithi045.stdout: i=10 exists [1659779~14779483,16872282~2127957,21188995~12257359,34955369~7628581,45082762~17180148,71632315~9730667,83937430~8661233,99407305~1655876,104767910~8483109,121862248~595321,123520397~15032370,140004714~6751159,148572550~9707130,158823321~1926466,169554965~12655371,183119390~3236677,189947221~10343013,206899772~8211092,218420359~6472794,225143787~21998244,251968052~4035487,258954181~2599657,267583087~22771090,294412067~9426639,309201969~26297105,340467595~8142214,350254010~6042664,371853644~23028112,399101982~1412477] wrote [54858074~5368980,71632315~5318479,124022427~3922090,140004714~6751159,153260896~3418572,234598398~6780723,272581398~8997136,340467595~8003206,373168485~1520761] 2021-07-31T02:29:44.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:44 smithi154 conmon[34553]: cluster 2021-07-31T02:29:42.406161+0000 mgr.x (mgr.24542) 1822 : cluster [DBG] pgmap v2323: 41 pgs: 41 active+clean; 285 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.0 KiB/s rd, 56 MiB/s wr, 37 op/s 2021-07-31T02:29:44.802 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:44 smithi154 conmon[34553]: audit 2021-07-31T02:29:42. 2021-07-31T02:29:44.803 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:44 smithi154 conmon[34553]: 868841+0000 mgr.x (mgr.24542) 1823 : audit [DBG] from='client.17223 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:29:44.803 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:44 smithi154 conmon[34553]: audit 2021-07-31T02:29:43.152973+0000 mgr.x (mgr.24542) 1824 : audit [DBG] from='client.27166 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:29:44.803 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:44 smithi154 conmon[34553]: cluster 2021-07-31T02:29:43 2021-07-31T02:29:44.803 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:44 smithi154 conmon[34553]: .316798+0000 mon.a (mon.0) 2132 : cluster [DBG] osdmap e1203: 8 total, 8 up, 8 in 2021-07-31T02:29:44.804 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:44 smithi154 conmon[34553]: audit 2021-07-31 2021-07-31T02:29:44.804 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:44 smithi154 conmon[34553]: T02:29:43.526070+0000 mon.c (mon.1) 103 : audit [DBG] from='client.? 172.21.15.45:0/1292348274' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:29:44.805 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:44 smithi045 conmon[33655]: cluster 2021-07-31T02:29:42.406161+0000 mgr.x (mgr.24542) 1822 : cluster 2021-07-31T02:29:44.805 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:44 smithi045 conmon[33655]: [DBG] pgmap v2323: 41 pgs: 41 active+clean; 285 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.0 KiB/s rd, 56 MiB/s wr, 37 op/s 2021-07-31T02:29:44.805 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:44 smithi045 conmon[33655]: audit 2021-07-31T02:29:42.868841+0000 mgr.x (mgr.24542) 1823 : audit [DBG] from='client.17223 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:29:44.805 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:44 smithi045 conmon[33655]: audit 2021-07-31T02:29:43.152973+0000 mgr.x (mgr.24542 2021-07-31T02:29:44.806 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:44 smithi045 conmon[33655]: ) 1824 : audit [DBG] from='client.27166 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:29:44.806 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:44 smithi045 conmon[33655]: cluster 2021-07-31T02:29:44.806 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:44 smithi045 conmon[33655]: 2021-07-31T 2021-07-31T02:29:44.806 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:44 smithi045 conmon[33655]: 02:29:43.316798+0000 mon.a ( 2021-07-31T02:29:44.807 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:44 smithi045 conmon[33655]: mon.0) 2132 : cluster [DBG] osdmap e1203: 8 total, 8 up, 8 in 2021-07-31T02:29:44.807 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:44 smithi045 conmon[33655]: audit 2021-07 2021-07-31T02:29:44.807 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:44 smithi045 conmon[33655]: -31T02:29:43.526070+0000 2021-07-31T02:29:44.807 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:44 smithi045 conmon[33655]: mon.c (mon.1) 103 : audit [DBG] 2021-07-31T02:29:44.807 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:44 smithi045 conmon[33655]: from='client.? 172.21.15.45:0/1292348274' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:29:44.808 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:44 smithi045 conmon[39718]: cluster 2021-07-31T02:29:42.406161+0000 mgr.x 2021-07-31T02:29:44.808 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:44 smithi045 conmon[39718]: (mgr.24542) 1822 : cluster [DBG] pgmap v2323: 41 pgs: 41 active+clean; 285 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.0 KiB/s rd, 56 MiB/s wr, 37 op/s 2021-07-31T02:29:44.808 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:44 smithi045 conmon[39718]: audit 2021-07-31T02: 2021-07-31T02:29:44.808 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:44 smithi045 conmon[39718]: 29:42.868841+0000 mgr.x (mgr.24542) 1823 : audit [DBG] 2021-07-31T02:29:44.809 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:44 smithi045 conmon[39718]: from='client.17223 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:29:44.809 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:44 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T02:29:44.809 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:44 smithi045 conmon[39718]: 31T02:29:43.152973+0000 mgr.x (mgr.24542) 1824 : audit 2021-07-31T02:29:44.809 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:44 smithi045 conmon[39718]: [DBG] from='client.27166 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:29:44.810 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:44 smithi045 conmon[39718]: cluster 2021-07-31T02:29:44.810 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:44 smithi045 conmon[39718]: 2021-07-31T02:29:43.316798+0000 2021-07-31T02:29:44.810 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:44 smithi045 conmon[39718]: mon.a (mon.0) 2132 : cluster [DBG] 2021-07-31T02:29:44.810 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:44 smithi045 conmon[39718]: osdmap e1203: 8 total, 8 up, 8 in 2021-07-31T02:29:44.810 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:44 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T02:29:44.811 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:44 smithi045 conmon[39718]: 02:29:43.526070+0000 mon.c ( 2021-07-31T02:29:44.811 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:44 smithi045 conmon[39718]: mon.1) 103 : audit [DBG] from='client.? 172.21.15.45:0/1292348274' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:29:44.811 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:44 smithi045 conmon[39718]: 2021-07-31T02:29:45.373 INFO:tasks.workunit.client.0.smithi045.stdout: i=11 exists [1659779~14779483,16872282~2127957,21188995~12257359,34767791~7816159,45082762~17180148,68778895~12584087,83937430~8661233,99407305~1655876,104767910~10590041,121862248~595321,123520397~15032370,140004714~6751159,148572550~9707130,158823321~1926466,163138692~1222581,169554965~12655371,183119390~5368199,189947221~10343013,206899772~8211092,218420359~28721672,251968052~4035487,258954181~2599657,267583087~22771090,294412067~9426639,309201969~26297105,340467595~8166428,350254010~6042664,371853644~23028112,399101982~5969618] wrote [34767791~5854240,68778895~5972207,109715521~5642430,163138692~1222581,185223309~3264280,221699832~5583794,281033397~1813122,294907254~1001287,347327314~1306709,400440293~4631307] 2021-07-31T02:29:45.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:45 smithi154 conmon[34553]: cluster 2021-07-31T02:29:44.323763+0000 mon.a (mon.0) 2133 : cluster [DBG] osdmap e1204: 8 total, 8 up, 8 in 2021-07-31T02:29:45.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:45 smithi045 conmon[39718]: cluster 2021-07-31T02:29:44.323763+0000 mon.a (mon.0) 2133 : cluster 2021-07-31T02:29:45.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:45 smithi045 conmon[39718]: [DBG] osdmap e1204: 8 total, 8 up, 8 in 2021-07-31T02:29:45.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:45 smithi045 conmon[33655]: cluster 2021-07-31T02:29:44.323763+0000 mon.a (mon.0) 2133 2021-07-31T02:29:45.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:45 smithi045 conmon[33655]: : cluster [DBG] osdmap e1204: 8 total, 8 up, 8 in 2021-07-31T02:29:46.384 INFO:tasks.workunit.client.0.smithi045.stdout: i=12 exists [1659779~14779483,16872282~2127957,21188995~12257359,34767791~7816159,45082762~17180148,68778895~12584087,83937430~8661233,99407305~1655876,104767910~10590041,121862248~595321,123520397~15032370,140004714~6751159,148572550~9707130,158823321~1926466,163138692~1222581,169554965~12655371,183119390~31991474,218420359~28721672,251968052~9585786,267583087~22771090,294412067~9426639,309201969~26297105,337835909~10798114,350254010~6042664,371853644~23028112,399101982~5969618] wrote [128363108~7889873,136500229~448154,184426156~7816206,199400107~8149653,253725353~6736540,315025695~2843426,324727817~1008480,337835909~2749176,350474172~3189307] 2021-07-31T02:29:46.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:46 smithi154 conmon[34553]: cluster 2021-07-31T02:29:44.406532+0000 mgr.x (mgr.24542) 1825 : cluster [DBG] 2021-07-31T02:29:46.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:46 smithi154 conmon[34553]: pgmap v2326: 41 pgs: 41 active+clean; 518 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 7.5 KiB/s rd, 55 MiB/s wr, 35 op/s 2021-07-31T02:29:46.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:46 smithi154 conmon[34553]: cluster 2021-07-31T02:29:45.329871+0000 mon.a (mon.0) 2134 : cluster [DBG] osdmap e1205: 8 total, 8 up, 8 in 2021-07-31T02:29:46.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:46 smithi045 conmon[39718]: cluster 2021-07-31T02:29:44.406532+0000 mgr.x (mgr.24542) 1825 : cluster [DBG] pgmap v2326: 41 pgs: 41 active+clean; 518 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 7.5 KiB/s rd, 55 MiB/s wr, 35 op/s 2021-07-31T02:29:46.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:46 smithi045 conmon[39718]: cluster 2021-07-31T02:29:45.329871+0000 mon.a (mon.0) 2134 : cluster [DBG] osdmap e1205: 8 total, 8 up, 8 in 2021-07-31T02:29:46.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:46 smithi045 conmon[33655]: cluster 2021-07-31T02:29:44.406532+0000 mgr.x (mgr.24542) 1825 : 2021-07-31T02:29:46.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:46 smithi045 conmon[33655]: cluster [DBG] pgmap v2326: 41 pgs: 41 active+clean; 518 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 7.5 KiB/s rd, 55 MiB/s wr, 35 op/s 2021-07-31T02:29:46.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:46 smithi045 conmon[33655]: cluster 2021-07-31T02:29:45.329871+0000 mon.a (mon.0) 2134 : cluster [DBG] osdmap e1205: 8 total, 8 up, 8 in 2021-07-31T02:29:47.375 INFO:tasks.workunit.client.0.smithi045.stdout: i=13 exists [1659779~14779483,16872282~2127957,21188995~12257359,34767791~7816159,45082762~17180148,68778895~12584087,83937430~10989158,99407305~1655876,104767910~10590041,117763863~3659110,121862248~595321,123520397~15032370,140004714~6751159,148572550~9707130,158823321~1926466,163138692~1222581,169554965~12655371,183119390~31991474,218420359~28721672,251968052~9585786,267583087~22771090,294412067~9426639,309201969~26297105,337835909~10798114,350254010~6042664,371853644~23735162,399101982~5969618] wrote [89899565~5027023,117763863~3659110,129575299~2745359,137505311~74244,141757840~1558637,175565980~1741298,191288418~50882,195560724~3437259,330606201~3405015,391601927~3986879] 2021-07-31T02:29:47.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:47 smithi154 conmon[34553]: cluster 2021-07-31T02:29:46.335525+0000 mon.a (mon.0) 2135 : cluster 2021-07-31T02:29:47.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:47 smithi154 conmon[34553]: [DBG] osdmap e1206: 8 total, 8 up, 8 in 2021-07-31T02:29:47.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:47 smithi045 conmon[33655]: cluster 2021-07-31T02:29:46.335525+0000 mon.a (mon.0) 2135 : cluster [DBG] osdmap e1206: 8 total, 8 up, 8 in 2021-07-31T02:29:47.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:47 smithi045 conmon[39718]: cluster 2021-07-31T02:29:46.335525+0000 mon.a ( 2021-07-31T02:29:47.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:47 smithi045 conmon[39718]: mon.0) 2135 : cluster [DBG] osdmap e1206: 8 total, 8 up, 8 in 2021-07-31T02:29:48.392 INFO:tasks.workunit.client.0.smithi045.stdout: i=14 exists [1659779~14779483,16872282~2127957,20506603~12939751,34767791~7816159,45082762~17180148,68427994~12934988,83937430~10989158,99407305~1655876,104767910~10590041,117763863~3659110,121862248~595321,123520397~15032370,140004714~6751159,148572550~9707130,158823321~1926466,163138692~1222581,169554965~12655371,183119390~31991474,218420359~28721672,251968052~9585786,267583087~22771090,294412067~9426639,309201969~26297105,337835909~10798114,350254010~6042664,371853644~23735162,399101982~5969618] wrote [20506603~3419423,68427994~4569455,86375397~644869,107123314~762196,186756180~7249788,196666634~2551883,199510478~6040584,230851236~7316448,284426883~131326,350287119~5452558] 2021-07-31T02:29:48.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:48 smithi154 conmon[34553]: cluster 2021-07-31T02:29:46.407148+0000 mgr.x (mgr.24542) 1826 : cluster [DBG] pgmap v2329: 41 pgs: 41 active+clean; 518 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 7.5 KiB/s rd, 55 MiB/s wr, 35 op/s 2021-07-31T02:29:48.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:48 smithi154 conmon[34553]: cluster 2021-07-31T02:29:47.341008+0000 mon.a (mon.0) 2136 : cluster [DBG] osdmap e1207: 8 total, 8 up, 8 in 2021-07-31T02:29:48.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:48 smithi045 conmon[39718]: cluster 2021-07-31T02:29:46.407148+0000 mgr.x (mgr.24542) 1826 : cluster [DBG] pgmap v2329: 41 pgs: 41 active+clean; 518 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 7.5 KiB/s rd, 55 MiB/s wr, 35 op/s 2021-07-31T02:29:48.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:48 smithi045 conmon[39718]: cluster 2021-07-31T02:29:47.341008+0000 mon.a (mon.0) 2136 : cluster [DBG] osdmap e1207: 8 total, 8 up, 8 in 2021-07-31T02:29:48.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:48 smithi045 conmon[33655]: cluster 2021-07-31T02:29:46.407148+0000 mgr.x (mgr.24542) 1826 : cluster 2021-07-31T02:29:48.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:48 smithi045 conmon[33655]: [DBG] pgmap v2329: 41 pgs: 41 active+clean; 518 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 7.5 KiB/s rd, 55 MiB/s wr, 35 op/s 2021-07-31T02:29:48.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:48 smithi045 conmon[33655]: cluster 2021-07-31T02:29:47.341008+0000 mon.a (mon.0) 2136 : cluster [DBG] osdmap e1207: 8 total, 8 up, 8 in 2021-07-31T02:29:49.400 INFO:tasks.workunit.client.0.smithi045.stdout: i=15 exists [1659779~14779483,16872282~2127957,20506603~12939751,34767791~7816159,45082762~17180148,68427994~12934988,81494486~1302829,83937430~10989158,99407305~1655876,104767910~10590041,117763863~3659110,121862248~595321,123520397~15032370,140004714~6751159,148572550~9707130,158823321~1926466,163138692~1222581,169108228~13102108,183119390~31991474,218420359~28721672,251968052~9585786,267583087~22771090,294412067~9516381,309201969~26297105,337835909~10798114,350254010~6042664,371853644~23735162,399101982~14161689] wrote [51581644~510738,70478475~3910324,81494486~1302829,141118469~19189,169108228~7208192,299192434~4736014,337892860~5174531,376616714~929395,402822647~10441024] 2021-07-31T02:29:49.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:49 smithi154 conmon[34553]: cluster 2021-07-31T02:29:48.342954+0000 mon.a (mon.0) 2137 : cluster [DBG] 2021-07-31T02:29:49.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:49 smithi154 conmon[34553]: osdmap e1208: 8 total, 8 up, 8 in 2021-07-31T02:29:49.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:49 smithi045 conmon[39718]: cluster 2021-07-31T02:29: 2021-07-31T02:29:49.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:49 smithi045 conmon[39718]: 48.342954+0000 mon.a (mon.0) 2137 : cluster [DBG] osdmap e1208: 8 total, 8 up, 8 in 2021-07-31T02:29:49.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:49 smithi045 conmon[33655]: cluster 2021-07-31T02:29:48.342954+0000 mon.a (mon.0) 2021-07-31T02:29:49.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:49 smithi045 conmon[33655]: 2137 : cluster [DBG] osdmap e1208: 8 total, 8 up, 8 in 2021-07-31T02:29:50.411 INFO:tasks.workunit.client.0.smithi045.stdout: i=16 exists [1659779~14779483,16872282~2127957,20506603~22077347,45082762~17180148,68427994~12934988,81494486~1302829,83937430~11416044,99407305~1655876,104767910~10590041,117763863~3659110,121862248~595321,123520397~15032370,140004714~6751159,148572550~9707130,158823321~5537952,169108228~13102108,183119390~31991474,218420359~28721672,251968052~9585786,267583087~22771090,294412067~9516381,309201969~26297105,337835909~10798114,350254010~6042664,360077475~3681783,364816374~5007680,371853644~23735162,399101982~14161689] wrote [6899945~3566536,29422664~5830765,52452368~6257030,72886811~3033071,88991588~6361886,133707134~4490059,159296841~4622916,236593655~142497,360077475~3681783,364816374~5007680] 2021-07-31T02:29:50.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:50 smithi154 conmon[34553]: cluster 2021-07-31T02:29:48.407461+0000 mgr.x (mgr.24542) 1827 : cluster [DBG] pgmap v2332: 41 pgs: 41 active+clean; 518 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2021-07-31T02:29:50.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:50 smithi154 conmon[34553]: 2021-07-31T02:29:50.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:50 smithi154 conmon[34553]: cluster 2021-07-31T02:29:49.354028+0000 mon.a (mon.0) 2138 : cluster [DBG] osdmap e1209: 8 total, 8 up, 8 in 2021-07-31T02:29:50.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:50 smithi045 conmon[39718]: cluster 2021-07-31T02:29:48.407461+0000 mgr.x (mgr.24542) 1827 : cluster [DBG] pgmap v2332: 41 pgs: 41 active+clean; 518 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2021-07-31T02:29:50.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:50 smithi045 conmon[39718]: cluster 2021-07-31T02:29:49.354028+0000 2021-07-31T02:29:50.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:50 smithi045 conmon[39718]: mon.a (mon.0) 2138 : cluster [DBG] 2021-07-31T02:29:50.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:50 smithi045 conmon[39718]: osdmap e1209: 8 total, 8 up, 8 in 2021-07-31T02:29:50.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:50 smithi045 conmon[33655]: cluster 2021-07-31T02:29:48.407461+0000 2021-07-31T02:29:50.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:50 smithi045 conmon[33655]: mgr.x (mgr.24542) 1827 : cluster [DBG] pgmap v2332: 41 pgs: 41 active+clean; 518 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2021-07-31T02:29:50.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:50 smithi045 conmon[33655]: cluster 2021-07-31T02:29:49.354028+0000 mon.a (mon.0) 2138 : cluster [DBG] 2021-07-31T02:29:50.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:50 smithi045 conmon[33655]: osdmap e1209: 8 total, 8 up, 8 in 2021-07-31T02:29:51.408 INFO:tasks.workunit.client.0.smithi045.stdout: i=17 exists [1659779~14779483,16872282~2127957,20506603~22077347,45082762~17180148,68427994~12934988,81494486~1302829,83937430~11416044,99407305~1655876,104767910~10590041,117763863~3659110,121862248~595321,123520397~15032370,140004714~6751159,148572550~9707130,158823321~5537952,169108228~46002636,218420359~28721672,251968052~9585786,264768676~25585501,294412067~9516381,309201969~26297105,337098215~11535808,350254010~13505248,364816374~5007680,371853644~23735162,399101982~14161689] wrote [5944690~3661829,176112436~7628109,221609797~5523538,241043318~3389673,264768676~4730600,337098215~892431,341466332~4104008,356209158~4287107] 2021-07-31T02:29:51.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:51 smithi154 conmon[34553]: cluster 2021-07-31T02:29:50.360189+0000 mon.a (mon.0) 2021-07-31T02:29:51.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:51 smithi154 conmon[34553]: 2139 : cluster [DBG] osdmap e1210: 8 total, 8 up, 8 in 2021-07-31T02:29:51.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:51 smithi154 conmon[34553]: cluster 2021-07-31T02:29:50.407812+0000 mgr.x (mgr.24542) 1828 : cluster [DBG] pgmap v2335: 41 pgs: 41 active+clean; 709 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 7.5 KiB/s rd, 46 MiB/s wr, 34 op/s 2021-07-31T02:29:51.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:51 smithi045 conmon[33655]: cluster 2021-07-31T02:29:50.360189+0000 mon.a (mon.0) 2139 2021-07-31T02:29:51.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:51 smithi045 conmon[33655]: : cluster [DBG] osdmap e1210: 8 total, 8 up, 8 in 2021-07-31T02:29:51.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:51 smithi045 conmon[33655]: cluster 2021-07-31T02:29:50.407812+0000 mgr.x (mgr.24542) 1828 : cluster [DBG] pgmap v2335: 41 pgs: 41 active+clean; 709 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 7.5 KiB/s rd, 46 MiB/s wr, 34 op/s 2021-07-31T02:29:51.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:51 smithi045 conmon[39718]: cluster 2021-07-31T02:29:50.360189+0000 mon.a (mon 2021-07-31T02:29:51.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:51 smithi045 conmon[39718]: .0) 2139 : cluster [DBG] osdmap e1210: 8 total, 8 up, 8 in 2021-07-31T02:29:51.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:51 smithi045 conmon[39718]: cluster 2021-07-31T02:29:50.407812+0000 mgr.x (mgr.24542) 1828 : cluster [DBG] 2021-07-31T02:29:51.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:51 smithi045 conmon[39718]: pgmap v2335: 41 pgs: 41 active+clean; 709 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 7.5 KiB/s rd, 46 MiB/s wr, 34 op/s 2021-07-31T02:29:52.425 INFO:tasks.workunit.client.0.smithi045.stdout: i=18 exists [1659779~14779483,16872282~2127957,20506603~22077347,45082762~17180148,68427994~12934988,81494486~1302829,83937430~11416044,98413952~23009021,121862248~595321,123520397~15032370,140004714~6751159,148572550~9707130,158823321~5537952,169108228~46002636,218420359~28721672,251968052~9585786,264768676~25585501,294412067~9516381,309201969~26297105,337098215~11535808,350254010~13505248,364816374~5007680,371853644~23735162,399101982~14161689] wrote [48291533~6389052,72056628~2120398,89646425~2337170,98413952~7579310,114241774~4240566,155103843~2328266,161078807~3256374,265842433~5675844,299760899~2239604] 2021-07-31T02:29:52.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:52 smithi154 conmon[34553]: cluster 2021-07-31T02:29:51.362458+0000 mon.a 2021-07-31T02:29:52.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:52 smithi154 conmon[34553]: (mon.0) 2140 : cluster [DBG] osdmap e1211: 8 total, 8 up, 8 in 2021-07-31T02:29:52.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:52 smithi045 conmon[39718]: cluster 2021-07-31T02:29:51.362458+0000 mon.a 2021-07-31T02:29:52.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:52 smithi045 conmon[39718]: (mon.0) 2140 : cluster [DBG] osdmap e1211: 8 total, 8 up, 8 in 2021-07-31T02:29:52.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:52 smithi045 conmon[33655]: cluster 2021-07-31T02:29:51.362458+0000 mon.a (mon 2021-07-31T02:29:52.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:52 smithi045 conmon[33655]: .0) 2140 : cluster [DBG] osdmap e1211: 8 total, 8 up, 8 in 2021-07-31T02:29:53.431 INFO:tasks.workunit.client.0.smithi045.stdout: i=19 exists [1659779~14779483,16872282~2127957,20506603~22077347,45082762~17180148,64161263~92264,68427994~12934988,81494486~1302829,83937430~11416044,98413952~23009021,121862248~595321,123520397~15032370,140004714~6751159,148572550~9707130,158823321~5537952,169108228~46002636,218420359~28721672,251968052~9585786,264768676~25585501,294412067~14692164,309201969~39432054,350254010~13505248,364816374~5007680,371853644~23735162,399101982~14161689] wrote [64161263~92264,70989687~643593,177623761~5876515,233889380~6540299,301858205~7246026,310046006~7387488,331742183~7253244,352410754~3227422,403457951~271481] 2021-07-31T02:29:53.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:53 smithi154 conmon[34553]: cluster 2021-07-31T02:29:52.368550+0000 mon.a (mon.0) 2141 : cluster [DBG] osdmap e1212: 8 total, 8 up, 8 in 2021-07-31T02:29:53.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:53 smithi154 conmon[34553]: cluster 2021-07-31T02:29:52.408163+0000 mgr.x (mgr.24542) 1829 : cluster [DBG] pgmap v2338: 41 pgs: 41 active+clean; 709 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 7.5 KiB/s rd, 46 MiB/s wr, 34 op/s 2021-07-31T02:29:53.775 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:29:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:29:53.583Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:29:53.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:29:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:29:53.583Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:29:53.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:29:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:29:53.583Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:29:53.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:53 smithi045 conmon[33655]: cluster 2021-07-31T02:29:52.368550+0000 2021-07-31T02:29:53.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:53 smithi045 conmon[33655]: mon.a (mon.0) 2141 : cluster [DBG] osdmap e1212: 8 total, 8 up, 8 in 2021-07-31T02:29:53.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:53 smithi045 conmon[33655]: cluster 2021-07-31T02:29:52.408163+0000 mgr.x ( 2021-07-31T02:29:53.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:53 smithi045 conmon[33655]: mgr.24542) 1829 : cluster [DBG] pgmap v2338: 41 pgs: 41 active+clean; 709 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 7.5 KiB/s rd, 46 MiB/s wr, 34 op/s 2021-07-31T02:29:53.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:53 smithi045 conmon[39718]: cluster 2021-07-31T02:29:52.368550+0000 mon.a (mon 2021-07-31T02:29:53.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:53 smithi045 conmon[39718]: .0) 2141 : cluster [DBG] osdmap e1212: 8 total, 8 up, 8 in 2021-07-31T02:29:53.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:53 smithi045 conmon[39718]: cluster 2021-07-31T02:29:52.408163+0000 2021-07-31T02:29:53.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:53 smithi045 conmon[39718]: mgr.x (mgr.24542) 1829 : cluster [DBG] pgmap v2338: 41 pgs: 41 active+clean; 709 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 7.5 KiB/s rd, 46 MiB/s wr, 34 op/s 2021-07-31T02:29:54.390 INFO:tasks.workunit.client.0.smithi045.stdout:from 0 to 1 diff [16777216~4194304,96468992~4194304,130023424~4194304,146800640~4194304,192937984~8388608,234881024~4194304,268435456~8388608,281018368~8388608,331350016~4194304] 2021-07-31T02:29:54.391 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [16777216~4194304,96468992~4194304,130023424~4194304,146800640~4194304,192937984~8388608,234881024~4194304,268435456~8388608,281018368~8388608,331350016~4194304] 2021-07-31T02:29:54.400 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [16777216~4194304,61470496~1444064,96468992~8388608,112381922~864286,130023424~8388608,146800640~8388608,192937984~8388608,234881024~4194304,268435456~25165824,303838706~2345486,325624364~1531348,331350016~4194304,347758455~368777,382676434~3199534,394881756~3577124,400514459~2138725] 2021-07-31T02:29:54.400 INFO:tasks.workunit.client.0.smithi045.stdout:from 0 to 2 diff [4194304~8388608,16777216~12582912,41943040~12582912,96468992~4194304,125829120~8388608,146800640~4194304,167772160~12582912,192937984~8388608,234881024~12582912,255852544~4194304,268435456~8388608,281018368~8388608,306184192~8388608,331350016~4194304] 2021-07-31T02:29:54.400 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~8388608,16777216~12582912,41943040~12582912,96468992~4194304,125829120~8388608,146800640~4194304,167772160~12582912,192937984~8388608,234881024~12582912,255852544~4194304,268435456~8388608,281018368~8388608,306184192~8388608,331350016~4194304] 2021-07-31T02:29:54.407 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~8388608,16777216~12582912,41943040~12582912,61470496~1444064,96468992~8388608,112381922~864286,125829120~12582912,146800640~8388608,167772160~12582912,192937984~8388608,234881024~12582912,255852544~8388608,268435456~25165824,303838706~14928398,325624364~1531348,331350016~4194304,347758455~368777,382676434~3199534,394881756~3577124,400514459~2138725] 2021-07-31T02:29:54.407 INFO:tasks.workunit.client.0.smithi045.stdout:from 0 to 3 diff [4194304~8388608,16777216~12582912,41943040~12582912,58720256~4194304,71303168~8388608,83886080~8388608,96468992~4194304,125829120~8388608,146800640~4194304,155189248~4194304,167772160~16777216,192937984~8388608,222298112~8388608,234881024~12582912,255852544~4194304,268435456~8388608,281018368~8388608,306184192~8388608,331350016~4194304,381681664~8388608] 2021-07-31T02:29:54.407 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~8388608,16777216~12582912,41943040~12582912,58720256~4194304,71303168~8388608,83886080~8388608,96468992~4194304,125829120~8388608,146800640~4194304,155189248~4194304,167772160~16777216,192937984~8388608,222298112~8388608,234881024~12582912,255852544~4194304,268435456~8388608,281018368~8388608,306184192~8388608,331350016~4194304,381681664~8388608] 2021-07-31T02:29:54.414 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~8388608,16777216~12582912,41943040~12582912,58720256~4194304,71303168~8388608,83886080~8388608,96468992~8388608,112381922~864286,125829120~12582912,146800640~16777216,167772160~20971520,192937984~8388608,222298112~25165824,255852544~8388608,268435456~25165824,303838706~14928398,325624364~1531348,331350016~4194304,347758455~368777,381681664~8388608,394881756~3577124,400514459~2138725] 2021-07-31T02:29:54.414 INFO:tasks.workunit.client.0.smithi045.stdout:from 0 to 4 diff [4194304~8388608,16777216~37748736,58720256~4194304,71303168~20971520,96468992~4194304,125829120~8388608,146800640~12582912,167772160~16777216,192937984~8388608,218103808~12582912,234881024~12582912,255852544~4194304,268435456~8388608,281018368~8388608,306184192~8388608,327155712~8388608,381681664~8388608] 2021-07-31T02:29:54.414 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~8388608,16777216~37748736,58720256~4194304,71303168~20971520,96468992~4194304,125829120~8388608,146800640~12582912,167772160~16777216,192937984~8388608,218103808~12582912,234881024~12582912,255852544~4194304,268435456~8388608,281018368~8388608,306184192~8388608,327155712~8388608,381681664~8388608] 2021-07-31T02:29:54.421 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~8388608,16777216~37748736,58720256~4194304,71303168~20971520,96468992~8388608,112381922~864286,125829120~12582912,146800640~16777216,167772160~20971520,192937984~8388608,218103808~29360128,255852544~8388608,268435456~25165824,303838706~14928398,325624364~9919956,347758455~368777,381681664~8388608,394881756~3577124,400514459~2138725] 2021-07-31T02:29:54.421 INFO:tasks.workunit.client.0.smithi045.stdout:from 0 to 5 diff [4194304~50331648,58720256~4194304,71303168~29360128,121634816~12582912,146800640~12582912,167772160~16777216,188743680~12582912,218103808~12582912,234881024~12582912,255852544~4194304,264241152~12582912,281018368~8388608,306184192~8388608,327155712~8388608,381681664~8388608] 2021-07-31T02:29:54.421 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~50331648,58720256~4194304,71303168~29360128,121634816~12582912,146800640~12582912,167772160~16777216,188743680~12582912,218103808~12582912,234881024~12582912,255852544~4194304,264241152~12582912,281018368~8388608,306184192~8388608,327155712~8388608,381681664~8388608] 2021-07-31T02:29:54.428 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~50331648,58720256~4194304,71303168~33554432,112381922~864286,121634816~16777216,146800640~16777216,167772160~33554432,218103808~29360128,255852544~37748736,303838706~14928398,325624364~9919956,347758455~368777,381681664~12582912,394881756~3577124,400514459~2138725] 2021-07-31T02:29:54.428 INFO:tasks.workunit.client.0.smithi045.stdout:from 0 to 6 diff [4194304~58720256,71303168~33554432,121634816~12582912,146800640~12582912,167772160~16777216,188743680~12582912,209715200~20971520,234881024~12582912,251658240~8388608,264241152~12582912,281018368~8388608,306184192~8388608,327155712~8388608,348127232~8388608,369098752~8388608,381681664~8388608] 2021-07-31T02:29:54.428 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~58720256,71303168~33554432,121634816~12582912,146800640~12582912,167772160~16777216,188743680~12582912,209715200~20971520,234881024~12582912,251658240~8388608,264241152~12582912,281018368~8388608,306184192~8388608,327155712~8388608,348127232~8388608,369098752~8388608,381681664~8388608] 2021-07-31T02:29:54.434 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~58720256,71303168~37748736,112381922~864286,121634816~16777216,146800640~16777216,167772160~33554432,209715200~37748736,251658240~41943040,303838706~14928398,325624364~9919956,347758455~8757385,369098752~8388608,381681664~12582912,394881756~3577124,400514459~2138725] 2021-07-31T02:29:54.435 INFO:tasks.workunit.client.0.smithi045.stdout:from 0 to 7 diff [4194304~58720256,71303168~33554432,121634816~12582912,146800640~12582912,167772160~16777216,188743680~12582912,209715200~20971520,234881024~12582912,251658240~8388608,264241152~25165824,306184192~12582912,322961408~12582912,339738624~16777216,369098752~8388608,381681664~8388608] 2021-07-31T02:29:54.435 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~58720256,71303168~33554432,121634816~12582912,146800640~12582912,167772160~16777216,188743680~12582912,209715200~20971520,234881024~12582912,251658240~8388608,264241152~25165824,306184192~12582912,322961408~12582912,339738624~16777216,369098752~8388608,381681664~8388608] 2021-07-31T02:29:54.441 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~58720256,71303168~37748736,112381922~864286,121634816~16777216,146800640~16777216,167772160~33554432,209715200~37748736,251658240~41943040,303838706~31705614,339738624~16777216,369098752~25165824,394881756~3577124,400514459~2138725] 2021-07-31T02:29:54.441 INFO:tasks.workunit.client.0.smithi045.stdout:from 0 to 8 diff [4194304~58720256,71303168~41943040,121634816~12582912,146800640~12582912,167772160~16777216,188743680~12582912,209715200~37748736,251658240~8388608,264241152~25165824,306184192~12582912,322961408~12582912,339738624~16777216,369098752~8388608,381681664~8388608] 2021-07-31T02:29:54.442 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~58720256,71303168~41943040,121634816~12582912,146800640~12582912,167772160~16777216,188743680~12582912,209715200~37748736,251658240~8388608,264241152~25165824,306184192~12582912,322961408~12582912,339738624~16777216,369098752~8388608,381681664~8388608] 2021-07-31T02:29:54.448 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~58720256,71303168~46137344,121634816~16777216,146800640~16777216,167772160~33554432,209715200~37748736,251658240~41943040,303838706~31705614,339738624~16777216,369098752~25165824,394881756~3577124,400514459~2138725] 2021-07-31T02:29:54.448 INFO:tasks.workunit.client.0.smithi045.stdout:from 0 to 9 diff [0~62914560,71303168~41943040,121634816~16777216,146800640~12582912,167772160~16777216,188743680~12582912,205520896~8388608,218103808~29360128,251658240~8388608,264241152~25165824,293601280~8388608,306184192~12582912,322961408~12582912,339738624~16777216,369098752~8388608,381681664~12582912] 2021-07-31T02:29:54.448 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~62914560,71303168~41943040,121634816~16777216,146800640~12582912,167772160~16777216,188743680~12582912,205520896~8388608,218103808~29360128,251658240~8388608,264241152~25165824,293601280~8388608,306184192~12582912,322961408~12582912,339738624~16777216,369098752~8388608,381681664~12582912] 2021-07-31T02:29:54.455 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~62914560,71303168~46137344,121634816~20971520,146800640~16777216,167772160~33554432,205520896~41943040,251658240~50331648,303838706~31705614,339738624~16777216,369098752~25165824,394881756~3577124,400514459~2138725] 2021-07-31T02:29:54.456 INFO:tasks.workunit.client.0.smithi045.stdout:from 0 to 10 diff [0~62914560,71303168~41943040,121634816~37748736,167772160~16777216,188743680~12582912,205520896~8388608,218103808~29360128,251658240~8388608,264241152~25165824,293601280~8388608,306184192~12582912,322961408~12582912,339738624~16777216,369098752~8388608,381681664~12582912] 2021-07-31T02:29:54.456 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~62914560,71303168~41943040,121634816~37748736,167772160~16777216,188743680~12582912,205520896~8388608,218103808~29360128,251658240~8388608,264241152~25165824,293601280~8388608,306184192~12582912,322961408~12582912,339738624~16777216,369098752~8388608,381681664~12582912] 2021-07-31T02:29:54.462 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~62914560,71303168~46137344,121634816~41943040,167772160~33554432,205520896~41943040,251658240~50331648,303838706~31705614,339738624~16777216,369098752~25165824,394881756~3577124,400514459~2138725] 2021-07-31T02:29:54.462 INFO:tasks.workunit.client.0.smithi045.stdout:from 0 to 11 diff [0~62914560,67108864~46137344,121634816~41943040,167772160~33554432,205520896~8388608,218103808~29360128,251658240~8388608,264241152~25165824,293601280~8388608,306184192~12582912,322961408~12582912,339738624~16777216,369098752~8388608,381681664~12582912,398458880~8388608] 2021-07-31T02:29:54.462 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~62914560,67108864~46137344,121634816~41943040,167772160~33554432,205520896~8388608,218103808~29360128,251658240~8388608,264241152~25165824,293601280~8388608,306184192~12582912,322961408~12582912,339738624~16777216,369098752~8388608,381681664~12582912,398458880~8388608] 2021-07-31T02:29:54.469 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~62914560,67108864~50331648,121634816~79691776,205520896~41943040,251658240~50331648,303838706~31705614,339738624~16777216,369098752~25165824,394881756~11965732] 2021-07-31T02:29:54.469 INFO:tasks.workunit.client.0.smithi045.stdout:from 0 to 12 diff [0~62914560,67108864~46137344,121634816~41943040,167772160~46137344,218103808~29360128,251658240~37748736,293601280~8388608,306184192~12582912,322961408~33554432,369098752~8388608,381681664~12582912,398458880~8388608] 2021-07-31T02:29:54.470 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~62914560,67108864~46137344,121634816~41943040,167772160~46137344,218103808~29360128,251658240~37748736,293601280~8388608,306184192~12582912,322961408~33554432,369098752~8388608,381681664~12582912,398458880~8388608] 2021-07-31T02:29:54.475 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~62914560,67108864~50331648,121634816~125829120,251658240~50331648,303838706~52677134,369098752~25165824,394881756~11965732] 2021-07-31T02:29:54.476 INFO:tasks.workunit.client.0.smithi045.stdout:from 0 to 13 diff [0~62914560,67108864~46137344,117440512~46137344,167772160~46137344,218103808~29360128,251658240~37748736,293601280~8388608,306184192~12582912,322961408~33554432,369098752~8388608,381681664~25165824] 2021-07-31T02:29:54.476 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~62914560,67108864~46137344,117440512~46137344,167772160~46137344,218103808~29360128,251658240~37748736,293601280~8388608,306184192~12582912,322961408~33554432,369098752~8388608,381681664~25165824] 2021-07-31T02:29:54.483 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~62914560,67108864~180355072,251658240~50331648,303838706~52677134,369098752~37748736] 2021-07-31T02:29:54.483 INFO:tasks.workunit.client.0.smithi045.stdout:from 0 to 14 diff [0~62914560,67108864~46137344,117440512~46137344,167772160~46137344,218103808~29360128,251658240~37748736,293601280~8388608,306184192~12582912,322961408~33554432,369098752~8388608,381681664~25165824] 2021-07-31T02:29:54.484 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~62914560,67108864~46137344,117440512~46137344,167772160~46137344,218103808~29360128,251658240~37748736,293601280~8388608,306184192~12582912,322961408~33554432,369098752~8388608,381681664~25165824] 2021-07-31T02:29:54.489 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~62914560,67108864~180355072,251658240~50331648,303838706~52677134,369098752~37748736] 2021-07-31T02:29:54.489 INFO:tasks.workunit.client.0.smithi045.stdout:from 0 to 15 diff [0~62914560,67108864~46137344,117440512~46137344,167772160~46137344,218103808~29360128,251658240~37748736,293601280~25165824,322961408~33554432,369098752~8388608,381681664~33554432] 2021-07-31T02:29:54.490 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~62914560,67108864~46137344,117440512~46137344,167772160~46137344,218103808~29360128,251658240~37748736,293601280~25165824,322961408~33554432,369098752~8388608,381681664~33554432] 2021-07-31T02:29:54.495 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~62914560,67108864~180355072,251658240~104857600,369098752~46137344] 2021-07-31T02:29:54.495 INFO:tasks.workunit.client.0.smithi045.stdout:from 0 to 16 diff [0~62914560,67108864~46137344,117440512~46137344,167772160~46137344,218103808~29360128,251658240~37748736,293601280~25165824,322961408~54525952,381681664~33554432] 2021-07-31T02:29:54.495 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~62914560,67108864~46137344,117440512~46137344,167772160~46137344,218103808~29360128,251658240~37748736,293601280~25165824,322961408~54525952,381681664~33554432] 2021-07-31T02:29:54.501 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~62914560,67108864~180355072,251658240~163577856] 2021-07-31T02:29:54.501 INFO:tasks.workunit.client.0.smithi045.stdout:from 0 to 17 diff [0~62914560,67108864~46137344,117440512~46137344,167772160~46137344,218103808~29360128,251658240~67108864,322961408~54525952,381681664~33554432] 2021-07-31T02:29:54.502 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~62914560,67108864~46137344,117440512~46137344,167772160~46137344,218103808~29360128,251658240~67108864,322961408~54525952,381681664~33554432] 2021-07-31T02:29:54.507 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~62914560,67108864~180355072,251658240~163577856] 2021-07-31T02:29:54.507 INFO:tasks.workunit.client.0.smithi045.stdout:from 0 to 18 diff [0~62914560,67108864~96468992,167772160~46137344,218103808~29360128,251658240~67108864,322961408~54525952,381681664~33554432] 2021-07-31T02:29:54.508 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~62914560,67108864~96468992,167772160~46137344,218103808~29360128,251658240~67108864,322961408~54525952,381681664~33554432] 2021-07-31T02:29:54.514 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~62914560,67108864~180355072,251658240~163577856] 2021-07-31T02:29:54.514 INFO:tasks.workunit.client.0.smithi045.stdout:from 0 to 19 diff [0~163577856,167772160~46137344,218103808~29360128,251658240~67108864,322961408~54525952,381681664~33554432] 2021-07-31T02:29:54.514 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~163577856,167772160~46137344,218103808~29360128,251658240~67108864,322961408~54525952,381681664~33554432] 2021-07-31T02:29:54.520 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~247463936,251658240~163577856] 2021-07-31T02:29:54.520 INFO:tasks.workunit.client.0.smithi045.stdout:from 1 to 2 diff [4194304~8388608,20971520~8388608,41943040~12582912,125829120~4194304,167772160~12582912,234881024~8388608,255852544~4194304,306184192~8388608] 2021-07-31T02:29:54.521 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~8388608,20971520~8388608,41943040~12582912,125829120~4194304,167772160~12582912,234881024~8388608,255852544~4194304,306184192~8388608] 2021-07-31T02:29:54.526 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~8388608,19000239~10359889,41943040~12582912,61470496~1444064,101063181~3794419,112381922~864286,125829120~8388608,134792269~3619763,153192806~1996442,167772160~12582912,199097964~2228628,234881024~12582912,255852544~8388608,279929063~1089305,290354177~3247103,303838706~14928398,325624364~1531348,335470493~73827,347758455~368777,382676434~3199534,394881756~3577124,400514459~2138725] 2021-07-31T02:29:54.526 INFO:tasks.workunit.client.0.smithi045.stdout:from 1 to 3 diff [4194304~8388608,20971520~8388608,41943040~12582912,58720256~4194304,71303168~8388608,83886080~8388608,125829120~4194304,155189248~4194304,167772160~16777216,222298112~8388608,234881024~8388608,255852544~4194304,306184192~8388608,331350016~4194304,381681664~8388608] 2021-07-31T02:29:54.527 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~8388608,20971520~8388608,41943040~12582912,58720256~4194304,71303168~8388608,83886080~8388608,125829120~4194304,155189248~4194304,167772160~16777216,222298112~8388608,234881024~8388608,255852544~4194304,306184192~8388608,331350016~4194304,381681664~8388608] 2021-07-31T02:29:54.532 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~8388608,19000239~10359889,41943040~12582912,58720256~4194304,71303168~8388608,83886080~8388608,101063181~3794419,112381922~864286,125829120~8388608,134792269~3619763,153192806~10385050,167772160~20971520,199097964~2228628,222298112~25165824,255852544~8388608,279929063~1089305,290354177~3247103,303838706~14928398,325624364~1531348,331350016~4194304,347758455~368777,381681664~8388608,394881756~3577124,400514459~2138725] 2021-07-31T02:29:54.533 INFO:tasks.workunit.client.0.smithi045.stdout:from 1 to 4 diff [4194304~8388608,20971520~33554432,58720256~4194304,71303168~20971520,125829120~8388608,146800640~4194304,155189248~4194304,167772160~16777216,218103808~12582912,234881024~8388608,255852544~4194304,306184192~8388608,327155712~8388608,381681664~8388608] 2021-07-31T02:29:54.533 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~8388608,20971520~33554432,58720256~4194304,71303168~20971520,125829120~8388608,146800640~4194304,155189248~4194304,167772160~16777216,218103808~12582912,234881024~8388608,255852544~4194304,306184192~8388608,327155712~8388608,381681664~8388608] 2021-07-31T02:29:54.539 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~8388608,19000239~35525713,58720256~4194304,71303168~20971520,101063181~3794419,112381922~864286,125829120~12582912,146800640~4194304,153192806~10385050,167772160~20971520,199097964~2228628,218103808~29360128,255852544~8388608,279929063~1089305,290354177~3247103,303838706~14928398,325624364~9919956,347758455~368777,381681664~8388608,394881756~3577124,400514459~2138725] 2021-07-31T02:29:54.539 INFO:tasks.workunit.client.0.smithi045.stdout:from 1 to 5 diff [4194304~12582912,20971520~33554432,58720256~4194304,71303168~25165824,121634816~12582912,146800640~4194304,155189248~4194304,167772160~16777216,188743680~8388608,218103808~12582912,234881024~8388608,255852544~4194304,264241152~8388608,306184192~8388608,327155712~8388608,381681664~8388608] 2021-07-31T02:29:54.540 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~12582912,20971520~33554432,58720256~4194304,71303168~25165824,121634816~12582912,146800640~4194304,155189248~4194304,167772160~16777216,188743680~8388608,218103808~12582912,234881024~8388608,255852544~4194304,264241152~8388608,306184192~8388608,327155712~8388608,381681664~8388608] 2021-07-31T02:29:54.545 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~12582912,19000239~35525713,58720256~4194304,71303168~25165824,101063181~3794419,112381922~864286,121634816~16777216,146800640~4194304,153192806~10385050,167772160~33554432,218103808~29360128,255852544~20971520,279929063~1089305,290354177~3247103,303838706~14928398,325624364~9919956,347758455~368777,381681664~12582912,394881756~3577124,400514459~2138725] 2021-07-31T02:29:54.545 INFO:tasks.workunit.client.0.smithi045.stdout:from 1 to 6 diff [4194304~12582912,20971520~41943040,71303168~25165824,100663296~4194304,121634816~12582912,146800640~12582912,167772160~16777216,188743680~12582912,209715200~20971520,234881024~8388608,251658240~8388608,264241152~8388608,306184192~8388608,327155712~8388608,348127232~8388608,369098752~8388608,381681664~8388608] 2021-07-31T02:29:54.545 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~12582912,20971520~41943040,71303168~25165824,100663296~4194304,121634816~12582912,146800640~12582912,167772160~16777216,188743680~12582912,209715200~20971520,234881024~8388608,251658240~8388608,264241152~8388608,306184192~8388608,327155712~8388608,348127232~8388608,369098752~8388608,381681664~8388608] 2021-07-31T02:29:54.552 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~12582912,19000239~43914321,71303168~25165824,100663296~8388608,112381922~864286,121634816~16777216,146800640~16777216,167772160~33554432,209715200~37748736,251658240~25165824,279929063~1089305,290354177~3247103,303838706~14928398,325624364~9919956,347758455~8757385,369098752~8388608,381681664~12582912,394881756~3577124,400514459~2138725] 2021-07-31T02:29:54.552 INFO:tasks.workunit.client.0.smithi045.stdout:from 1 to 7 diff [4194304~12582912,20971520~41943040,71303168~25165824,100663296~4194304,121634816~12582912,146800640~12582912,167772160~16777216,188743680~12582912,209715200~20971520,234881024~12582912,251658240~8388608,264241152~16777216,306184192~12582912,322961408~12582912,339738624~16777216,369098752~8388608,381681664~8388608] 2021-07-31T02:29:54.552 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~12582912,20971520~41943040,71303168~25165824,100663296~4194304,121634816~12582912,146800640~12582912,167772160~16777216,188743680~12582912,209715200~20971520,234881024~12582912,251658240~8388608,264241152~16777216,306184192~12582912,322961408~12582912,339738624~16777216,369098752~8388608,381681664~8388608] 2021-07-31T02:29:54.557 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~12582912,19000239~43914321,71303168~25165824,100663296~8388608,112381922~864286,121634816~16777216,146800640~16777216,167772160~33554432,209715200~37748736,251658240~33554432,290354177~3247103,303838706~31705614,339738624~16777216,369098752~25165824,394881756~3577124,400514459~2138725] 2021-07-31T02:29:54.558 INFO:tasks.workunit.client.0.smithi045.stdout:from 1 to 8 diff [4194304~12582912,20971520~41943040,71303168~25165824,100663296~12582912,121634816~12582912,146800640~12582912,167772160~16777216,188743680~12582912,209715200~37748736,251658240~8388608,264241152~20971520,306184192~12582912,322961408~12582912,339738624~16777216,369098752~8388608,381681664~8388608] 2021-07-31T02:29:54.558 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~12582912,20971520~41943040,71303168~25165824,100663296~12582912,121634816~12582912,146800640~12582912,167772160~16777216,188743680~12582912,209715200~37748736,251658240~8388608,264241152~20971520,306184192~12582912,322961408~12582912,339738624~16777216,369098752~8388608,381681664~8388608] 2021-07-31T02:29:54.563 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~12582912,19000239~43914321,71303168~25165824,100663296~16777216,121634816~16777216,146800640~16777216,167772160~33554432,209715200~37748736,251658240~37748736,290354177~3247103,303838706~31705614,339738624~16777216,369098752~25165824,394881756~3577124,400514459~2138725] 2021-07-31T02:29:54.564 INFO:tasks.workunit.client.0.smithi045.stdout:from 1 to 9 diff [0~16777216,20971520~41943040,71303168~25165824,100663296~12582912,121634816~16777216,146800640~12582912,167772160~16777216,188743680~12582912,205520896~8388608,218103808~29360128,251658240~8388608,264241152~20971520,293601280~8388608,306184192~12582912,322961408~12582912,339738624~16777216,369098752~8388608,381681664~12582912] 2021-07-31T02:29:54.564 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~16777216,20971520~41943040,71303168~25165824,100663296~12582912,121634816~16777216,146800640~12582912,167772160~16777216,188743680~12582912,205520896~8388608,218103808~29360128,251658240~8388608,264241152~20971520,293601280~8388608,306184192~12582912,322961408~12582912,339738624~16777216,369098752~8388608,381681664~12582912] 2021-07-31T02:29:54.570 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~16777216,19000239~43914321,71303168~25165824,100663296~16777216,121634816~20971520,146800640~16777216,167772160~33554432,205520896~41943040,251658240~37748736,290354177~11635711,303838706~31705614,339738624~16777216,369098752~25165824,394881756~3577124,400514459~2138725] 2021-07-31T02:29:54.570 INFO:tasks.workunit.client.0.smithi045.stdout:from 1 to 10 diff [0~16777216,20971520~41943040,71303168~25165824,100663296~12582912,121634816~37748736,167772160~16777216,188743680~12582912,205520896~8388608,218103808~29360128,251658240~8388608,264241152~20971520,293601280~8388608,306184192~12582912,322961408~12582912,339738624~16777216,369098752~8388608,381681664~12582912] 2021-07-31T02:29:54.570 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~16777216,20971520~41943040,71303168~25165824,100663296~12582912,121634816~37748736,167772160~16777216,188743680~12582912,205520896~8388608,218103808~29360128,251658240~8388608,264241152~20971520,293601280~8388608,306184192~12582912,322961408~12582912,339738624~16777216,369098752~8388608,381681664~12582912] 2021-07-31T02:29:54.578 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~16777216,19000239~43914321,71303168~25165824,100663296~16777216,121634816~41943040,167772160~33554432,205520896~41943040,251658240~37748736,290354177~11635711,303838706~31705614,339738624~16777216,369098752~25165824,394881756~3577124,400514459~2138725] 2021-07-31T02:29:54.578 INFO:tasks.workunit.client.0.smithi045.stdout:from 1 to 11 diff [0~16777216,20971520~41943040,67108864~29360128,100663296~12582912,121634816~41943040,167772160~33554432,205520896~8388608,218103808~29360128,251658240~8388608,264241152~20971520,293601280~8388608,306184192~12582912,322961408~12582912,339738624~16777216,369098752~8388608,381681664~12582912,398458880~8388608] 2021-07-31T02:29:54.579 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~16777216,20971520~41943040,67108864~29360128,100663296~12582912,121634816~41943040,167772160~33554432,205520896~8388608,218103808~29360128,251658240~8388608,264241152~20971520,293601280~8388608,306184192~12582912,322961408~12582912,339738624~16777216,369098752~8388608,381681664~12582912,398458880~8388608] 2021-07-31T02:29:54.582 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~16777216,19000239~43914321,67108864~29360128,100663296~16777216,121634816~79691776,205520896~41943040,251658240~37748736,290354177~11635711,303838706~31705614,339738624~16777216,369098752~25165824,394881756~11965732] 2021-07-31T02:29:54.582 INFO:tasks.workunit.client.0.smithi045.stdout:from 1 to 12 diff [0~16777216,20971520~41943040,67108864~29360128,100663296~12582912,121634816~41943040,167772160~46137344,218103808~29360128,251658240~33554432,293601280~8388608,306184192~12582912,322961408~33554432,369098752~8388608,381681664~12582912,398458880~8388608] 2021-07-31T02:29:54.582 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~16777216,20971520~41943040,67108864~29360128,100663296~12582912,121634816~41943040,167772160~46137344,218103808~29360128,251658240~33554432,293601280~8388608,306184192~12582912,322961408~33554432,369098752~8388608,381681664~12582912,398458880~8388608] 2021-07-31T02:29:54.588 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~16777216,19000239~43914321,67108864~29360128,100663296~16777216,121634816~125829120,251658240~37748736,290354177~11635711,303838706~52677134,369098752~25165824,394881756~11965732] 2021-07-31T02:29:54.589 INFO:tasks.workunit.client.0.smithi045.stdout:from 1 to 13 diff [0~16777216,20971520~41943040,67108864~29360128,100663296~12582912,117440512~46137344,167772160~46137344,218103808~29360128,251658240~33554432,293601280~8388608,306184192~12582912,322961408~33554432,369098752~8388608,381681664~25165824] 2021-07-31T02:29:54.589 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~16777216,20971520~41943040,67108864~29360128,100663296~12582912,117440512~46137344,167772160~46137344,218103808~29360128,251658240~33554432,293601280~8388608,306184192~12582912,322961408~33554432,369098752~8388608,381681664~25165824] 2021-07-31T02:29:54.595 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~16777216,19000239~43914321,67108864~29360128,100663296~146800640,251658240~37748736,290354177~11635711,303838706~52677134,369098752~37748736] 2021-07-31T02:29:54.595 INFO:tasks.workunit.client.0.smithi045.stdout:from 1 to 14 diff [0~62914560,67108864~29360128,100663296~12582912,117440512~46137344,167772160~46137344,218103808~29360128,251658240~33554432,293601280~8388608,306184192~12582912,322961408~33554432,369098752~8388608,381681664~25165824] 2021-07-31T02:29:54.595 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~62914560,67108864~29360128,100663296~12582912,117440512~46137344,167772160~46137344,218103808~29360128,251658240~33554432,293601280~8388608,306184192~12582912,322961408~33554432,369098752~8388608,381681664~25165824] 2021-07-31T02:29:54.600 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~62914560,67108864~29360128,100663296~146800640,251658240~37748736,290354177~11635711,303838706~52677134,369098752~37748736] 2021-07-31T02:29:54.601 INFO:tasks.workunit.client.0.smithi045.stdout:from 1 to 15 diff [0~62914560,67108864~29360128,100663296~12582912,117440512~46137344,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~33554432,369098752~8388608,381681664~33554432] 2021-07-31T02:29:54.601 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~62914560,67108864~29360128,100663296~12582912,117440512~46137344,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~33554432,369098752~8388608,381681664~33554432] 2021-07-31T02:29:54.607 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~62914560,67108864~29360128,100663296~146800640,251658240~37748736,290354177~66161663,369098752~46137344] 2021-07-31T02:29:54.607 INFO:tasks.workunit.client.0.smithi045.stdout:from 1 to 16 diff [0~62914560,67108864~29360128,100663296~12582912,117440512~46137344,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~33554432] 2021-07-31T02:29:54.607 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~62914560,67108864~29360128,100663296~12582912,117440512~46137344,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~33554432] 2021-07-31T02:29:54.614 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~62914560,67108864~29360128,100663296~146800640,251658240~37748736,290354177~124881919] 2021-07-31T02:29:54.614 INFO:tasks.workunit.client.0.smithi045.stdout:from 1 to 17 diff [0~62914560,67108864~29360128,100663296~12582912,117440512~46137344,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~33554432] 2021-07-31T02:29:54.614 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~62914560,67108864~29360128,100663296~12582912,117440512~46137344,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~33554432] 2021-07-31T02:29:54.619 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~62914560,67108864~29360128,100663296~146800640,251658240~37748736,290354177~124881919] 2021-07-31T02:29:54.620 INFO:tasks.workunit.client.0.smithi045.stdout:from 1 to 18 diff [0~62914560,67108864~96468992,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~33554432] 2021-07-31T02:29:54.620 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~62914560,67108864~96468992,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~33554432] 2021-07-31T02:29:54.625 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~62914560,67108864~180355072,251658240~37748736,290354177~124881919] 2021-07-31T02:29:54.626 INFO:tasks.workunit.client.0.smithi045.stdout:from 1 to 19 diff [0~163577856,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~33554432] 2021-07-31T02:29:54.626 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~163577856,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~33554432] 2021-07-31T02:29:54.631 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~247463936,251658240~37748736,290354177~124881919] 2021-07-31T02:29:54.632 INFO:tasks.workunit.client.0.smithi045.stdout:from 2 to 3 diff [25165824~4194304,46137344~8388608,58720256~4194304,71303168~8388608,83886080~8388608,155189248~4194304,180355072~4194304,222298112~8388608,331350016~4194304,381681664~8388608] 2021-07-31T02:29:54.632 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [25165824~4194304,46137344~8388608,58720256~4194304,71303168~8388608,83886080~8388608,155189248~4194304,180355072~4194304,222298112~8388608,331350016~4194304,381681664~8388608] 2021-07-31T02:29:54.638 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [10177672~2405240,19000239~1971281,25165824~4194304,46137344~8388608,58720256~4194304,71303168~8388608,83886080~8388608,101063181~3794419,112381922~864286,134792269~3619763,153192806~10385050,178010951~10732729,199097964~2228628,222298112~12582912,246233854~1230082,261553838~2687314,279929063~1089305,290354177~3247103,303838706~2345486,325624364~1531348,331350016~4194304,347758455~368777,381681664~8388608,394881756~3577124,400514459~2138725] 2021-07-31T02:29:54.638 INFO:tasks.workunit.client.0.smithi045.stdout:from 2 to 4 diff [4194304~8388608,20971520~8388608,33554432~8388608,46137344~8388608,58720256~4194304,71303168~20971520,125829120~8388608,146800640~4194304,155189248~4194304,180355072~4194304,218103808~12582912,327155712~8388608,381681664~8388608] 2021-07-31T02:29:54.638 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~8388608,20971520~8388608,33554432~8388608,46137344~8388608,58720256~4194304,71303168~20971520,125829120~8388608,146800640~4194304,155189248~4194304,180355072~4194304,218103808~12582912,327155712~8388608,381681664~8388608] 2021-07-31T02:29:54.644 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~8388608,19000239~35525713,58720256~4194304,71303168~20971520,101063181~3794419,112381922~864286,125829120~12582912,146800640~4194304,153192806~10385050,178010951~10732729,199097964~2228628,218103808~16777216,246233854~1230082,261553838~2687314,279929063~1089305,290354177~3247103,303838706~2345486,325624364~9919956,347758455~368777,381681664~8388608,394881756~3577124,400514459~2138725] 2021-07-31T02:29:54.644 INFO:tasks.workunit.client.0.smithi045.stdout:from 2 to 5 diff [4194304~12582912,20971520~20971520,46137344~8388608,58720256~4194304,71303168~25165824,121634816~12582912,146800640~4194304,155189248~4194304,180355072~4194304,188743680~8388608,218103808~12582912,264241152~8388608,327155712~8388608,381681664~8388608] 2021-07-31T02:29:54.644 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~12582912,20971520~20971520,46137344~8388608,58720256~4194304,71303168~25165824,121634816~12582912,146800640~4194304,155189248~4194304,180355072~4194304,188743680~8388608,218103808~12582912,264241152~8388608,327155712~8388608,381681664~8388608] 2021-07-31T02:29:54.650 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~12582912,19000239~35525713,58720256~4194304,71303168~25165824,101063181~3794419,112381922~864286,121634816~16777216,146800640~4194304,153192806~10385050,178010951~23315641,218103808~16777216,246233854~1230082,261553838~15270226,279929063~1089305,290354177~3247103,303838706~2345486,325624364~9919956,347758455~368777,381681664~12582912,394881756~3577124,400514459~2138725] 2021-07-31T02:29:54.651 INFO:tasks.workunit.client.0.smithi045.stdout:from 2 to 6 diff [4194304~12582912,20971520~20971520,46137344~16777216,71303168~25165824,100663296~4194304,121634816~12582912,146800640~12582912,180355072~4194304,188743680~12582912,209715200~20971520,251658240~4194304,264241152~8388608,327155712~8388608,348127232~8388608,369098752~8388608,381681664~8388608] 2021-07-31T02:29:54.651 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~12582912,20971520~20971520,46137344~16777216,71303168~25165824,100663296~4194304,121634816~12582912,146800640~12582912,180355072~4194304,188743680~12582912,209715200~20971520,251658240~4194304,264241152~8388608,327155712~8388608,348127232~8388608,369098752~8388608,381681664~8388608] 2021-07-31T02:29:54.656 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~12582912,19000239~43914321,71303168~25165824,100663296~8388608,112381922~864286,121634816~16777216,146800640~16777216,178010951~23315641,209715200~25165824,246233854~1230082,251658240~8388608,261553838~15270226,279929063~1089305,290354177~3247103,303838706~2345486,325624364~9919956,347758455~8757385,369098752~8388608,381681664~12582912,394881756~3577124,400514459~2138725] 2021-07-31T02:29:54.657 INFO:tasks.workunit.client.0.smithi045.stdout:from 2 to 7 diff [4194304~12582912,20971520~20971520,46137344~16777216,71303168~25165824,100663296~4194304,121634816~12582912,146800640~12582912,180355072~4194304,188743680~12582912,209715200~20971520,239075328~8388608,251658240~4194304,264241152~16777216,306184192~29360128,339738624~16777216,369098752~8388608,381681664~8388608] 2021-07-31T02:29:54.657 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~12582912,20971520~20971520,46137344~16777216,71303168~25165824,100663296~4194304,121634816~12582912,146800640~12582912,180355072~4194304,188743680~12582912,209715200~20971520,239075328~8388608,251658240~4194304,264241152~16777216,306184192~29360128,339738624~16777216,369098752~8388608,381681664~8388608] 2021-07-31T02:29:54.663 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~12582912,19000239~43914321,71303168~25165824,100663296~8388608,112381922~864286,121634816~16777216,146800640~16777216,178010951~23315641,209715200~25165824,239075328~8388608,251658240~8388608,261553838~23658834,290354177~3247103,303838706~31705614,339738624~16777216,369098752~25165824,394881756~3577124,400514459~2138725] 2021-07-31T02:29:54.663 INFO:tasks.workunit.client.0.smithi045.stdout:from 2 to 8 diff [4194304~12582912,20971520~20971520,46137344~16777216,71303168~25165824,100663296~12582912,121634816~12582912,146800640~12582912,171966464~12582912,188743680~12582912,209715200~25165824,239075328~8388608,251658240~4194304,264241152~20971520,306184192~29360128,339738624~16777216,369098752~8388608,381681664~8388608] 2021-07-31T02:29:54.664 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~12582912,20971520~20971520,46137344~16777216,71303168~25165824,100663296~12582912,121634816~12582912,146800640~12582912,171966464~12582912,188743680~12582912,209715200~25165824,239075328~8388608,251658240~4194304,264241152~20971520,306184192~29360128,339738624~16777216,369098752~8388608,381681664~8388608] 2021-07-31T02:29:54.669 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~12582912,19000239~43914321,71303168~25165824,100663296~16777216,121634816~16777216,146800640~16777216,171966464~29360128,209715200~37748736,251658240~8388608,261553838~27853138,290354177~3247103,303838706~31705614,339738624~16777216,369098752~25165824,394881756~3577124,400514459~2138725] 2021-07-31T02:29:54.670 INFO:tasks.workunit.client.0.smithi045.stdout:from 2 to 9 diff [0~16777216,20971520~20971520,46137344~16777216,71303168~25165824,100663296~12582912,121634816~16777216,146800640~12582912,171966464~12582912,188743680~12582912,205520896~8388608,218103808~16777216,239075328~8388608,251658240~4194304,264241152~20971520,293601280~8388608,306184192~29360128,339738624~16777216,369098752~8388608,381681664~12582912] 2021-07-31T02:29:54.670 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~16777216,20971520~20971520,46137344~16777216,71303168~25165824,100663296~12582912,121634816~16777216,146800640~12582912,171966464~12582912,188743680~12582912,205520896~8388608,218103808~16777216,239075328~8388608,251658240~4194304,264241152~20971520,293601280~8388608,306184192~29360128,339738624~16777216,369098752~8388608,381681664~12582912] 2021-07-31T02:29:54.675 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~16777216,19000239~43914321,71303168~25165824,100663296~16777216,121634816~20971520,146800640~16777216,171966464~29360128,205520896~41943040,251658240~8388608,261553838~27853138,290354177~11635711,303838706~31705614,339738624~16777216,369098752~25165824,394881756~3577124,400514459~2138725] 2021-07-31T02:29:54.675 INFO:tasks.workunit.client.0.smithi045.stdout:from 2 to 10 diff [0~16777216,20971520~20971520,46137344~16777216,71303168~25165824,100663296~12582912,121634816~37748736,171966464~12582912,188743680~12582912,205520896~8388608,218103808~29360128,251658240~4194304,264241152~20971520,293601280~8388608,306184192~29360128,339738624~16777216,369098752~8388608,381681664~12582912] 2021-07-31T02:29:54.675 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~16777216,20971520~20971520,46137344~16777216,71303168~25165824,100663296~12582912,121634816~37748736,171966464~12582912,188743680~12582912,205520896~8388608,218103808~29360128,251658240~4194304,264241152~20971520,293601280~8388608,306184192~29360128,339738624~16777216,369098752~8388608,381681664~12582912] 2021-07-31T02:29:54.681 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~16777216,19000239~43914321,71303168~25165824,100663296~16777216,121634816~41943040,171966464~29360128,205520896~41943040,251658240~8388608,261553838~27853138,290354177~11635711,303838706~31705614,339738624~16777216,369098752~25165824,394881756~3577124,400514459~2138725] 2021-07-31T02:29:54.682 INFO:tasks.workunit.client.0.smithi045.stdout:from 2 to 11 diff [0~16777216,20971520~20971520,46137344~16777216,67108864~29360128,100663296~12582912,121634816~41943040,171966464~29360128,205520896~8388608,218103808~29360128,251658240~4194304,264241152~20971520,293601280~8388608,306184192~29360128,339738624~16777216,369098752~8388608,381681664~12582912,398458880~8388608] 2021-07-31T02:29:54.682 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~16777216,20971520~20971520,46137344~16777216,67108864~29360128,100663296~12582912,121634816~41943040,171966464~29360128,205520896~8388608,218103808~29360128,251658240~4194304,264241152~20971520,293601280~8388608,306184192~29360128,339738624~16777216,369098752~8388608,381681664~12582912,398458880~8388608] 2021-07-31T02:29:54.687 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~16777216,19000239~43914321,67108864~29360128,100663296~16777216,121634816~46137344,171966464~29360128,205520896~41943040,251658240~8388608,261553838~27853138,290354177~11635711,303838706~31705614,339738624~16777216,369098752~25165824,394881756~11965732] 2021-07-31T02:29:54.687 INFO:tasks.workunit.client.0.smithi045.stdout:from 2 to 12 diff [0~16777216,20971520~20971520,46137344~16777216,67108864~29360128,100663296~12582912,121634816~41943040,171966464~41943040,218103808~29360128,251658240~33554432,293601280~8388608,306184192~50331648,369098752~8388608,381681664~12582912,398458880~8388608] 2021-07-31T02:29:54.687 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~16777216,20971520~20971520,46137344~16777216,67108864~29360128,100663296~12582912,121634816~41943040,171966464~41943040,218103808~29360128,251658240~33554432,293601280~8388608,306184192~50331648,369098752~8388608,381681664~12582912,398458880~8388608] 2021-07-31T02:29:54.694 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~16777216,19000239~43914321,67108864~29360128,100663296~16777216,121634816~46137344,171966464~75497472,251658240~37748736,290354177~11635711,303838706~52677134,369098752~25165824,394881756~11965732] 2021-07-31T02:29:54.694 INFO:tasks.workunit.client.0.smithi045.stdout:from 2 to 13 diff [0~16777216,20971520~20971520,46137344~16777216,67108864~29360128,100663296~12582912,117440512~46137344,171966464~41943040,218103808~29360128,251658240~33554432,293601280~8388608,306184192~50331648,369098752~8388608,381681664~25165824] 2021-07-31T02:29:54.694 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~16777216,20971520~20971520,46137344~16777216,67108864~29360128,100663296~12582912,117440512~46137344,171966464~41943040,218103808~29360128,251658240~33554432,293601280~8388608,306184192~50331648,369098752~8388608,381681664~25165824] 2021-07-31T02:29:54.700 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~16777216,19000239~43914321,67108864~29360128,100663296~67108864,171966464~75497472,251658240~37748736,290354177~11635711,303838706~52677134,369098752~37748736] 2021-07-31T02:29:54.700 INFO:tasks.workunit.client.0.smithi045.stdout:from 2 to 14 diff [0~41943040,46137344~16777216,67108864~29360128,100663296~12582912,117440512~46137344,171966464~41943040,218103808~29360128,251658240~33554432,293601280~8388608,306184192~50331648,369098752~8388608,381681664~25165824] 2021-07-31T02:29:54.700 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~41943040,46137344~16777216,67108864~29360128,100663296~12582912,117440512~46137344,171966464~41943040,218103808~29360128,251658240~33554432,293601280~8388608,306184192~50331648,369098752~8388608,381681664~25165824] 2021-07-31T02:29:54.705 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~62914560,67108864~29360128,100663296~67108864,171966464~75497472,251658240~37748736,290354177~11635711,303838706~52677134,369098752~37748736] 2021-07-31T02:29:54.706 INFO:tasks.workunit.client.0.smithi045.stdout:from 2 to 15 diff [0~41943040,46137344~16777216,67108864~29360128,100663296~12582912,117440512~46137344,167772160~46137344,218103808~29360128,251658240~33554432,293601280~62914560,369098752~8388608,381681664~33554432] 2021-07-31T02:29:54.706 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~41943040,46137344~16777216,67108864~29360128,100663296~12582912,117440512~46137344,167772160~46137344,218103808~29360128,251658240~33554432,293601280~62914560,369098752~8388608,381681664~33554432] 2021-07-31T02:29:54.712 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~62914560,67108864~29360128,100663296~146800640,251658240~37748736,290354177~66161663,369098752~46137344] 2021-07-31T02:29:54.712 INFO:tasks.workunit.client.0.smithi045.stdout:from 2 to 16 diff [0~41943040,46137344~16777216,67108864~29360128,100663296~12582912,117440512~46137344,167772160~46137344,218103808~29360128,251658240~33554432,293601280~83886080,381681664~33554432] 2021-07-31T02:29:54.713 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~41943040,46137344~16777216,67108864~29360128,100663296~12582912,117440512~46137344,167772160~46137344,218103808~29360128,251658240~33554432,293601280~83886080,381681664~33554432] 2021-07-31T02:29:54.718 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~62914560,67108864~29360128,100663296~146800640,251658240~37748736,290354177~124881919] 2021-07-31T02:29:54.718 INFO:tasks.workunit.client.0.smithi045.stdout:from 2 to 17 diff [0~41943040,46137344~16777216,67108864~29360128,100663296~12582912,117440512~46137344,167772160~46137344,218103808~29360128,251658240~33554432,293601280~83886080,381681664~33554432] 2021-07-31T02:29:54.719 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~41943040,46137344~16777216,67108864~29360128,100663296~12582912,117440512~46137344,167772160~46137344,218103808~29360128,251658240~33554432,293601280~83886080,381681664~33554432] 2021-07-31T02:29:54.724 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~62914560,67108864~29360128,100663296~146800640,251658240~37748736,290354177~124881919] 2021-07-31T02:29:54.724 INFO:tasks.workunit.client.0.smithi045.stdout:from 2 to 18 diff [0~41943040,46137344~16777216,67108864~96468992,167772160~46137344,218103808~29360128,251658240~33554432,293601280~83886080,381681664~33554432] 2021-07-31T02:29:54.725 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~41943040,46137344~16777216,67108864~96468992,167772160~46137344,218103808~29360128,251658240~33554432,293601280~83886080,381681664~33554432] 2021-07-31T02:29:54.730 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~62914560,67108864~180355072,251658240~37748736,290354177~124881919] 2021-07-31T02:29:54.730 INFO:tasks.workunit.client.0.smithi045.stdout:from 2 to 19 diff [0~41943040,46137344~117440512,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~33554432] 2021-07-31T02:29:54.731 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~41943040,46137344~117440512,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~33554432] 2021-07-31T02:29:54.736 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~247463936,251658240~37748736,290354177~124881919] 2021-07-31T02:29:54.737 INFO:tasks.workunit.client.0.smithi045.stdout:from 3 to 4 diff [4194304~8388608,20971520~8388608,33554432~8388608,71303168~8388608,125829120~8388608,146800640~4194304,218103808~8388608,327155712~8388608] 2021-07-31T02:29:54.737 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~8388608,20971520~8388608,33554432~8388608,71303168~8388608,125829120~8388608,146800640~4194304,218103808~8388608,327155712~8388608] 2021-07-31T02:29:54.743 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~8388608,19000239~27137105,54146316~379636,62262910~651650,71303168~12582912,89660467~2614221,101063181~3794419,112381922~864286,125829120~12582912,146800640~4194304,153192806~1996442,160749787~2828069,178010951~2344121,186356067~2387613,199097964~2228628,218103808~8388608,231294106~3586918,246233854~1230082,261553838~2687314,279929063~1089305,290354177~3247103,303838706~2345486,325624364~9919956,347758455~368777,394881756~3577124,400514459~2138725] 2021-07-31T02:29:54.743 INFO:tasks.workunit.client.0.smithi045.stdout:from 3 to 5 diff [4194304~12582912,20971520~20971520,71303168~8388608,83886080~8388608,121634816~12582912,146800640~4194304,188743680~8388608,218103808~12582912,264241152~8388608,327155712~8388608,385875968~4194304] 2021-07-31T02:29:54.743 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~12582912,20971520~20971520,71303168~8388608,83886080~8388608,121634816~12582912,146800640~4194304,188743680~8388608,218103808~12582912,264241152~8388608,327155712~8388608,385875968~4194304] 2021-07-31T02:29:54.749 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~12582912,19000239~27137105,54146316~379636,62262910~651650,71303168~25165824,101063181~3794419,112381922~864286,121634816~16777216,146800640~4194304,153192806~1996442,160749787~2828069,178010951~2344121,186356067~14970525,218103808~16777216,246233854~1230082,261553838~15270226,279929063~1089305,290354177~3247103,303838706~2345486,325624364~9919956,347758455~368777,385875968~8388608,394881756~3577124,400514459~2138725] 2021-07-31T02:29:54.749 INFO:tasks.workunit.client.0.smithi045.stdout:from 3 to 6 diff [4194304~12582912,20971520~20971520,54525952~8388608,71303168~8388608,83886080~8388608,100663296~4194304,121634816~12582912,146800640~12582912,188743680~12582912,209715200~20971520,251658240~4194304,264241152~8388608,327155712~8388608,348127232~8388608,369098752~8388608,385875968~4194304] 2021-07-31T02:29:54.750 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~12582912,20971520~20971520,54525952~8388608,71303168~8388608,83886080~8388608,100663296~4194304,121634816~12582912,146800640~12582912,188743680~12582912,209715200~20971520,251658240~4194304,264241152~8388608,327155712~8388608,348127232~8388608,369098752~8388608,385875968~4194304] 2021-07-31T02:29:54.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:54 smithi154 conmon[34553]: cluster 2021-07-31T02:29:53.379275+0000 mon.a (mon.0) 2142 : cluster [DBG] osdmap e1213: 8 total, 8 up, 8 in 2021-07-31T02:29:54.755 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~12582912,19000239~27137105,54146316~8768244,71303168~25165824,100663296~8388608,112381922~864286,121634816~16777216,146800640~12582912,160749787~2828069,178010951~2344121,186356067~14970525,209715200~25165824,246233854~1230082,251658240~8388608,261553838~15270226,279929063~1089305,290354177~3247103,303838706~2345486,325624364~9919956,347758455~8757385,369098752~8388608,385875968~8388608,394881756~3577124,400514459~2138725] 2021-07-31T02:29:54.755 INFO:tasks.workunit.client.0.smithi045.stdout:from 3 to 7 diff [4194304~12582912,20971520~20971520,54525952~8388608,71303168~8388608,83886080~8388608,100663296~4194304,121634816~12582912,146800640~12582912,188743680~12582912,209715200~20971520,239075328~8388608,251658240~4194304,264241152~16777216,306184192~29360128,339738624~16777216,369098752~8388608,385875968~4194304] 2021-07-31T02:29:54.755 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~12582912,20971520~20971520,54525952~8388608,71303168~8388608,83886080~8388608,100663296~4194304,121634816~12582912,146800640~12582912,188743680~12582912,209715200~20971520,239075328~8388608,251658240~4194304,264241152~16777216,306184192~29360128,339738624~16777216,369098752~8388608,385875968~4194304] 2021-07-31T02:29:54.761 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~12582912,19000239~27137105,54146316~8768244,71303168~25165824,100663296~8388608,112381922~864286,121634816~16777216,146800640~12582912,160749787~2828069,178010951~2344121,186356067~14970525,209715200~25165824,239075328~8388608,251658240~8388608,261553838~23658834,290354177~3247103,303838706~31705614,339738624~16777216,369098752~12582912,385875968~8388608,394881756~3577124,400514459~2138725] 2021-07-31T02:29:54.761 INFO:tasks.workunit.client.0.smithi045.stdout:from 3 to 8 diff [4194304~12582912,20971520~20971520,54525952~8388608,71303168~8388608,83886080~8388608,100663296~12582912,121634816~12582912,146800640~12582912,171966464~8388608,188743680~12582912,209715200~25165824,239075328~8388608,251658240~4194304,264241152~20971520,306184192~29360128,339738624~16777216,369098752~8388608,385875968~4194304] 2021-07-31T02:29:54.761 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~12582912,20971520~20971520,54525952~8388608,71303168~8388608,83886080~8388608,100663296~12582912,121634816~12582912,146800640~12582912,171966464~8388608,188743680~12582912,209715200~25165824,239075328~8388608,251658240~4194304,264241152~20971520,306184192~29360128,339738624~16777216,369098752~8388608,385875968~4194304] 2021-07-31T02:29:54.767 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~12582912,19000239~27137105,54146316~8768244,71303168~25165824,100663296~16777216,121634816~16777216,146800640~12582912,160749787~2828069,171966464~12582912,186356067~14970525,209715200~37748736,251658240~8388608,261553838~27853138,290354177~3247103,303838706~31705614,339738624~16777216,369098752~12582912,385875968~8388608,394881756~3577124,400514459~2138725] 2021-07-31T02:29:54.768 INFO:tasks.workunit.client.0.smithi045.stdout:from 3 to 9 diff [0~16777216,20971520~20971520,46137344~16777216,71303168~8388608,83886080~8388608,100663296~12582912,121634816~16777216,146800640~12582912,171966464~8388608,188743680~12582912,205520896~8388608,218103808~16777216,239075328~8388608,251658240~4194304,264241152~20971520,293601280~8388608,306184192~29360128,339738624~16777216,369098752~8388608,381681664~8388608] 2021-07-31T02:29:54.768 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~16777216,20971520~20971520,46137344~16777216,71303168~8388608,83886080~8388608,100663296~12582912,121634816~16777216,146800640~12582912,171966464~8388608,188743680~12582912,205520896~8388608,218103808~16777216,239075328~8388608,251658240~4194304,264241152~20971520,293601280~8388608,306184192~29360128,339738624~16777216,369098752~8388608,381681664~8388608] 2021-07-31T02:29:54.773 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~16777216,19000239~43914321,71303168~25165824,100663296~16777216,121634816~20971520,146800640~12582912,160749787~2828069,171966464~12582912,186356067~14970525,205520896~41943040,251658240~8388608,261553838~27853138,290354177~11635711,303838706~31705614,339738624~16777216,369098752~25165824,394881756~3577124,400514459~2138725] 2021-07-31T02:29:54.773 INFO:tasks.workunit.client.0.smithi045.stdout:from 3 to 10 diff [0~16777216,20971520~20971520,46137344~16777216,71303168~20971520,100663296~12582912,121634816~37748736,171966464~8388608,188743680~12582912,205520896~8388608,218103808~29360128,251658240~4194304,264241152~20971520,293601280~8388608,306184192~29360128,339738624~16777216,369098752~8388608,381681664~8388608] 2021-07-31T02:29:54.774 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~16777216,20971520~20971520,46137344~16777216,71303168~20971520,100663296~12582912,121634816~37748736,171966464~8388608,188743680~12582912,205520896~8388608,218103808~29360128,251658240~4194304,264241152~20971520,293601280~8388608,306184192~29360128,339738624~16777216,369098752~8388608,381681664~8388608] 2021-07-31T02:29:54.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:54 smithi045 conmon[39718]: cluster 2021-07-31T02:29:53.379275+0000 mon.a (mon.0) 2142 : cluster [DBG] osdmap e1213: 8 total, 8 up, 8 in 2021-07-31T02:29:54.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:54 smithi045 conmon[33655]: cluster 2021-07-31T02:29:53.379275 2021-07-31T02:29:54.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:54 smithi045 conmon[33655]: +0000 mon.a (mon.0) 2142 : cluster [DBG] osdmap e1213: 8 total, 8 up, 8 in 2021-07-31T02:29:54.780 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~16777216,19000239~43914321,71303168~25165824,100663296~16777216,121634816~37748736,160749787~2828069,171966464~12582912,186356067~14970525,205520896~41943040,251658240~8388608,261553838~27853138,290354177~11635711,303838706~31705614,339738624~16777216,369098752~25165824,394881756~3577124,400514459~2138725] 2021-07-31T02:29:54.780 INFO:tasks.workunit.client.0.smithi045.stdout:from 3 to 11 diff [0~16777216,20971520~20971520,46137344~16777216,67108864~25165824,100663296~12582912,121634816~41943040,171966464~8388608,184549376~16777216,205520896~8388608,218103808~29360128,251658240~4194304,264241152~20971520,293601280~8388608,306184192~29360128,339738624~16777216,369098752~8388608,381681664~8388608,398458880~8388608] 2021-07-31T02:29:54.780 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~16777216,20971520~20971520,46137344~16777216,67108864~25165824,100663296~12582912,121634816~41943040,171966464~8388608,184549376~16777216,205520896~8388608,218103808~29360128,251658240~4194304,264241152~20971520,293601280~8388608,306184192~29360128,339738624~16777216,369098752~8388608,381681664~8388608,398458880~8388608] 2021-07-31T02:29:54.787 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~16777216,19000239~43914321,67108864~29360128,100663296~16777216,121634816~46137344,171966464~29360128,205520896~41943040,251658240~8388608,261553838~27853138,290354177~11635711,303838706~31705614,339738624~16777216,369098752~25165824,394881756~11965732] 2021-07-31T02:29:54.787 INFO:tasks.workunit.client.0.smithi045.stdout:from 3 to 12 diff [0~16777216,20971520~20971520,46137344~16777216,67108864~25165824,100663296~12582912,121634816~41943040,171966464~41943040,218103808~29360128,251658240~33554432,293601280~8388608,306184192~50331648,369098752~8388608,381681664~8388608,398458880~8388608] 2021-07-31T02:29:54.787 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~16777216,20971520~20971520,46137344~16777216,67108864~25165824,100663296~12582912,121634816~41943040,171966464~41943040,218103808~29360128,251658240~33554432,293601280~8388608,306184192~50331648,369098752~8388608,381681664~8388608,398458880~8388608] 2021-07-31T02:29:54.792 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~16777216,19000239~43914321,67108864~29360128,100663296~16777216,121634816~46137344,171966464~75497472,251658240~37748736,290354177~11635711,303838706~52677134,369098752~25165824,394881756~11965732] 2021-07-31T02:29:54.792 INFO:tasks.workunit.client.0.smithi045.stdout:from 3 to 13 diff [0~16777216,20971520~20971520,46137344~16777216,67108864~29360128,100663296~12582912,117440512~46137344,171966464~41943040,218103808~29360128,251658240~33554432,293601280~8388608,306184192~50331648,369098752~8388608,381681664~12582912,398458880~8388608] 2021-07-31T02:29:54.792 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~16777216,20971520~20971520,46137344~16777216,67108864~29360128,100663296~12582912,117440512~46137344,171966464~41943040,218103808~29360128,251658240~33554432,293601280~8388608,306184192~50331648,369098752~8388608,381681664~12582912,398458880~8388608] 2021-07-31T02:29:54.799 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~16777216,19000239~43914321,67108864~29360128,100663296~67108864,171966464~75497472,251658240~37748736,290354177~11635711,303838706~52677134,369098752~37748736] 2021-07-31T02:29:54.799 INFO:tasks.workunit.client.0.smithi045.stdout:from 3 to 14 diff [0~41943040,46137344~16777216,67108864~29360128,100663296~12582912,117440512~46137344,171966464~41943040,218103808~29360128,251658240~33554432,293601280~8388608,306184192~50331648,369098752~8388608,381681664~12582912,398458880~8388608] 2021-07-31T02:29:54.799 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~41943040,46137344~16777216,67108864~29360128,100663296~12582912,117440512~46137344,171966464~41943040,218103808~29360128,251658240~33554432,293601280~8388608,306184192~50331648,369098752~8388608,381681664~12582912,398458880~8388608] 2021-07-31T02:29:54.805 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~62914560,67108864~29360128,100663296~67108864,171966464~75497472,251658240~37748736,290354177~11635711,303838706~52677134,369098752~37748736] 2021-07-31T02:29:54.805 INFO:tasks.workunit.client.0.smithi045.stdout:from 3 to 15 diff [0~41943040,46137344~16777216,67108864~29360128,100663296~12582912,117440512~46137344,167772160~46137344,218103808~29360128,251658240~33554432,293601280~62914560,369098752~8388608,381681664~12582912,398458880~16777216] 2021-07-31T02:29:54.805 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~41943040,46137344~16777216,67108864~29360128,100663296~12582912,117440512~46137344,167772160~46137344,218103808~29360128,251658240~33554432,293601280~62914560,369098752~8388608,381681664~12582912,398458880~16777216] 2021-07-31T02:29:54.811 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~62914560,67108864~29360128,100663296~146800640,251658240~37748736,290354177~66161663,369098752~46137344] 2021-07-31T02:29:54.811 INFO:tasks.workunit.client.0.smithi045.stdout:from 3 to 16 diff [0~41943040,46137344~16777216,67108864~29360128,100663296~12582912,117440512~46137344,167772160~46137344,218103808~29360128,251658240~33554432,293601280~83886080,381681664~12582912,398458880~16777216] 2021-07-31T02:29:54.811 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~41943040,46137344~16777216,67108864~29360128,100663296~12582912,117440512~46137344,167772160~46137344,218103808~29360128,251658240~33554432,293601280~83886080,381681664~12582912,398458880~16777216] 2021-07-31T02:29:54.818 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~62914560,67108864~29360128,100663296~146800640,251658240~37748736,290354177~124881919] 2021-07-31T02:29:54.818 INFO:tasks.workunit.client.0.smithi045.stdout:from 3 to 17 diff [0~41943040,46137344~16777216,67108864~29360128,100663296~12582912,117440512~46137344,167772160~46137344,218103808~29360128,251658240~33554432,293601280~83886080,381681664~12582912,398458880~16777216] 2021-07-31T02:29:54.818 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~41943040,46137344~16777216,67108864~29360128,100663296~12582912,117440512~46137344,167772160~46137344,218103808~29360128,251658240~33554432,293601280~83886080,381681664~12582912,398458880~16777216] 2021-07-31T02:29:54.823 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~62914560,67108864~29360128,100663296~146800640,251658240~37748736,290354177~124881919] 2021-07-31T02:29:54.824 INFO:tasks.workunit.client.0.smithi045.stdout:from 3 to 18 diff [0~41943040,46137344~16777216,67108864~96468992,167772160~46137344,218103808~29360128,251658240~33554432,293601280~83886080,381681664~12582912,398458880~16777216] 2021-07-31T02:29:54.824 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~41943040,46137344~16777216,67108864~96468992,167772160~46137344,218103808~29360128,251658240~33554432,293601280~83886080,381681664~12582912,398458880~16777216] 2021-07-31T02:29:54.829 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~62914560,67108864~180355072,251658240~37748736,290354177~124881919] 2021-07-31T02:29:54.831 INFO:tasks.workunit.client.0.smithi045.stdout:from 3 to 19 diff [0~41943040,46137344~117440512,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:29:54.831 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~41943040,46137344~117440512,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:29:54.835 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~247463936,251658240~37748736,290354177~124881919] 2021-07-31T02:29:54.836 INFO:tasks.workunit.client.0.smithi045.stdout:from 4 to 5 diff [12582912~4194304,25165824~8388608,83886080~8388608,121634816~4194304,188743680~8388608,226492416~4194304,264241152~8388608,327155712~8388608,385875968~4194304] 2021-07-31T02:29:54.836 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [12582912~4194304,25165824~8388608,83886080~8388608,121634816~4194304,188743680~8388608,226492416~4194304,264241152~8388608,327155712~8388608,385875968~4194304] 2021-07-31T02:29:54.842 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [12282237~4494979,19000239~1971281,25165824~8388608,54146316~379636,62262910~651650,81362982~15106010,101063181~3794419,112381922~864286,121634816~8388608,134792269~3619763,153192806~1996442,160749787~2828069,178010951~2344121,186356067~14970525,226492416~8388608,246233854~1230082,261553838~15270226,279929063~1089305,290354177~3247103,303838706~2345486,325624364~9919956,347758455~368777,385875968~8388608,394881756~3577124,400514459~2138725] 2021-07-31T02:29:54.842 INFO:tasks.workunit.client.0.smithi045.stdout:from 4 to 6 diff [8388608~8388608,25165824~8388608,54525952~8388608,83886080~8388608,100663296~4194304,121634816~4194304,150994944~8388608,188743680~12582912,209715200~8388608,226492416~4194304,251658240~4194304,264241152~8388608,327155712~8388608,348127232~8388608,369098752~8388608,385875968~4194304] 2021-07-31T02:29:54.842 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [8388608~8388608,25165824~8388608,54525952~8388608,83886080~8388608,100663296~4194304,121634816~4194304,150994944~8388608,188743680~12582912,209715200~8388608,226492416~4194304,251658240~4194304,264241152~8388608,327155712~8388608,348127232~8388608,369098752~8388608,385875968~4194304] 2021-07-31T02:29:54.848 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [8388608~8388608,19000239~1971281,25165824~8388608,54146316~8768244,81362982~15106010,100663296~8388608,112381922~864286,121634816~8388608,134792269~3619763,150994944~8388608,160749787~2828069,178010951~2344121,186356067~14970525,209715200~8388608,226492416~8388608,246233854~1230082,251658240~8388608,261553838~15270226,279929063~1089305,290354177~3247103,303838706~2345486,325624364~9919956,347758455~8757385,369098752~8388608,385875968~8388608,394881756~3577124,400514459~2138725] 2021-07-31T02:29:54.848 INFO:tasks.workunit.client.0.smithi045.stdout:from 4 to 7 diff [8388608~8388608,25165824~8388608,54525952~8388608,83886080~8388608,100663296~4194304,121634816~4194304,130023424~4194304,150994944~8388608,188743680~12582912,209715200~8388608,226492416~4194304,239075328~8388608,251658240~4194304,264241152~16777216,306184192~29360128,339738624~16777216,369098752~8388608,385875968~4194304] 2021-07-31T02:29:54.848 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [8388608~8388608,25165824~8388608,54525952~8388608,83886080~8388608,100663296~4194304,121634816~4194304,130023424~4194304,150994944~8388608,188743680~12582912,209715200~8388608,226492416~4194304,239075328~8388608,251658240~4194304,264241152~16777216,306184192~29360128,339738624~16777216,369098752~8388608,385875968~4194304] 2021-07-31T02:29:54.854 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [8388608~8388608,19000239~1971281,25165824~8388608,54146316~8768244,81362982~15106010,100663296~8388608,112381922~864286,121634816~12582912,134792269~3619763,150994944~8388608,160749787~2828069,178010951~2344121,186356067~14970525,209715200~8388608,226492416~8388608,239075328~8388608,251658240~8388608,261553838~23658834,290354177~3247103,303838706~31705614,339738624~16777216,369098752~12582912,385875968~8388608,394881756~3577124,400514459~2138725] 2021-07-31T02:29:54.855 INFO:tasks.workunit.client.0.smithi045.stdout:from 4 to 8 diff [8388608~8388608,25165824~8388608,54525952~8388608,75497472~4194304,83886080~8388608,100663296~12582912,121634816~4194304,130023424~4194304,150994944~8388608,171966464~8388608,188743680~12582912,209715200~8388608,226492416~8388608,239075328~8388608,251658240~4194304,264241152~20971520,306184192~29360128,339738624~16777216,369098752~8388608,385875968~4194304] 2021-07-31T02:29:54.855 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [8388608~8388608,25165824~8388608,54525952~8388608,75497472~4194304,83886080~8388608,100663296~12582912,121634816~4194304,130023424~4194304,150994944~8388608,171966464~8388608,188743680~12582912,209715200~8388608,226492416~8388608,239075328~8388608,251658240~4194304,264241152~20971520,306184192~29360128,339738624~16777216,369098752~8388608,385875968~4194304] 2021-07-31T02:29:54.860 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [8388608~8388608,19000239~1971281,25165824~8388608,54146316~8768244,75497472~4194304,81362982~15106010,100663296~16777216,121634816~12582912,134792269~3619763,150994944~8388608,160749787~2828069,171966464~12582912,186356067~14970525,209715200~8388608,226492416~20971520,251658240~8388608,261553838~27853138,290354177~3247103,303838706~31705614,339738624~16777216,369098752~12582912,385875968~8388608,394881756~3577124,400514459~2138725] 2021-07-31T02:29:54.860 INFO:tasks.workunit.client.0.smithi045.stdout:from 4 to 9 diff [0~16777216,25165824~8388608,46137344~16777216,75497472~4194304,83886080~8388608,100663296~12582912,121634816~4194304,130023424~8388608,150994944~8388608,171966464~8388608,188743680~12582912,205520896~8388608,226492416~8388608,239075328~8388608,251658240~4194304,264241152~20971520,293601280~8388608,306184192~29360128,339738624~16777216,369098752~8388608,381681664~8388608] 2021-07-31T02:29:54.861 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~16777216,25165824~8388608,46137344~16777216,75497472~4194304,83886080~8388608,100663296~12582912,121634816~4194304,130023424~8388608,150994944~8388608,171966464~8388608,188743680~12582912,205520896~8388608,226492416~8388608,239075328~8388608,251658240~4194304,264241152~20971520,293601280~8388608,306184192~29360128,339738624~16777216,369098752~8388608,381681664~8388608] 2021-07-31T02:29:54.866 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~16777216,19000239~1971281,25165824~8388608,46137344~16777216,75497472~4194304,81362982~15106010,100663296~16777216,121634816~20971520,150994944~8388608,160749787~2828069,171966464~12582912,186356067~14970525,205520896~12582912,226492416~20971520,251658240~8388608,261553838~27853138,290354177~11635711,303838706~31705614,339738624~16777216,369098752~25165824,394881756~3577124,400514459~2138725] 2021-07-31T02:29:54.866 INFO:tasks.workunit.client.0.smithi045.stdout:from 4 to 10 diff [0~16777216,25165824~8388608,46137344~16777216,71303168~8388608,83886080~8388608,100663296~12582912,121634816~25165824,150994944~8388608,171966464~8388608,188743680~12582912,205520896~8388608,226492416~20971520,251658240~4194304,264241152~20971520,293601280~8388608,306184192~29360128,339738624~16777216,369098752~8388608,381681664~8388608] 2021-07-31T02:29:54.867 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~16777216,25165824~8388608,46137344~16777216,71303168~8388608,83886080~8388608,100663296~12582912,121634816~25165824,150994944~8388608,171966464~8388608,188743680~12582912,205520896~8388608,226492416~20971520,251658240~4194304,264241152~20971520,293601280~8388608,306184192~29360128,339738624~16777216,369098752~8388608,381681664~8388608] 2021-07-31T02:29:54.872 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~16777216,19000239~1971281,25165824~8388608,46137344~16777216,71303168~8388608,81362982~15106010,100663296~16777216,121634816~25165824,150994944~8388608,160749787~2828069,171966464~12582912,186356067~14970525,205520896~12582912,226492416~20971520,251658240~8388608,261553838~27853138,290354177~11635711,303838706~31705614,339738624~16777216,369098752~25165824,394881756~3577124,400514459~2138725] 2021-07-31T02:29:54.873 INFO:tasks.workunit.client.0.smithi045.stdout:from 4 to 11 diff [0~16777216,25165824~16777216,46137344~16777216,67108864~12582912,83886080~8388608,100663296~12582912,121634816~25165824,150994944~12582912,171966464~8388608,184549376~16777216,205520896~8388608,218103808~29360128,251658240~4194304,264241152~20971520,293601280~8388608,306184192~29360128,339738624~16777216,369098752~8388608,381681664~8388608,398458880~8388608] 2021-07-31T02:29:54.873 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~16777216,25165824~16777216,46137344~16777216,67108864~12582912,83886080~8388608,100663296~12582912,121634816~25165824,150994944~12582912,171966464~8388608,184549376~16777216,205520896~8388608,218103808~29360128,251658240~4194304,264241152~20971520,293601280~8388608,306184192~29360128,339738624~16777216,369098752~8388608,381681664~8388608,398458880~8388608] 2021-07-31T02:29:54.879 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~16777216,19000239~1971281,25165824~16777216,46137344~16777216,67108864~12582912,81362982~15106010,100663296~16777216,121634816~25165824,150994944~16777216,171966464~29360128,205520896~41943040,251658240~8388608,261553838~27853138,290354177~11635711,303838706~31705614,339738624~16777216,369098752~25165824,394881756~11965732] 2021-07-31T02:29:54.879 INFO:tasks.workunit.client.0.smithi045.stdout:from 4 to 12 diff [0~16777216,25165824~16777216,46137344~16777216,67108864~12582912,83886080~8388608,100663296~12582912,121634816~25165824,150994944~12582912,171966464~41943040,218103808~29360128,251658240~33554432,293601280~8388608,306184192~50331648,369098752~8388608,381681664~8388608,398458880~8388608] 2021-07-31T02:29:54.879 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~16777216,25165824~16777216,46137344~16777216,67108864~12582912,83886080~8388608,100663296~12582912,121634816~25165824,150994944~12582912,171966464~41943040,218103808~29360128,251658240~33554432,293601280~8388608,306184192~50331648,369098752~8388608,381681664~8388608,398458880~8388608] 2021-07-31T02:29:54.885 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~16777216,19000239~1971281,25165824~16777216,46137344~16777216,67108864~12582912,81362982~15106010,100663296~16777216,121634816~25165824,150994944~16777216,171966464~75497472,251658240~37748736,290354177~11635711,303838706~52677134,369098752~25165824,394881756~11965732] 2021-07-31T02:29:54.885 INFO:tasks.workunit.client.0.smithi045.stdout:from 4 to 13 diff [0~16777216,25165824~16777216,46137344~16777216,67108864~12582912,83886080~12582912,100663296~12582912,117440512~29360128,150994944~12582912,171966464~41943040,218103808~29360128,251658240~33554432,293601280~8388608,306184192~50331648,369098752~8388608,381681664~12582912,398458880~8388608] 2021-07-31T02:29:54.885 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~16777216,25165824~16777216,46137344~16777216,67108864~12582912,83886080~12582912,100663296~12582912,117440512~29360128,150994944~12582912,171966464~41943040,218103808~29360128,251658240~33554432,293601280~8388608,306184192~50331648,369098752~8388608,381681664~12582912,398458880~8388608] 2021-07-31T02:29:54.891 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~16777216,19000239~1971281,25165824~16777216,46137344~16777216,67108864~12582912,81362982~15106010,100663296~46137344,150994944~16777216,171966464~75497472,251658240~37748736,290354177~11635711,303838706~52677134,369098752~37748736] 2021-07-31T02:29:54.891 INFO:tasks.workunit.client.0.smithi045.stdout:from 4 to 14 diff [0~20971520,25165824~16777216,46137344~16777216,67108864~12582912,83886080~12582912,100663296~12582912,117440512~29360128,150994944~12582912,171966464~41943040,218103808~29360128,251658240~33554432,293601280~8388608,306184192~50331648,369098752~8388608,381681664~12582912,398458880~8388608] 2021-07-31T02:29:54.891 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~20971520,25165824~16777216,46137344~16777216,67108864~12582912,83886080~12582912,100663296~12582912,117440512~29360128,150994944~12582912,171966464~41943040,218103808~29360128,251658240~33554432,293601280~8388608,306184192~50331648,369098752~8388608,381681664~12582912,398458880~8388608] 2021-07-31T02:29:54.897 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~41943040,46137344~16777216,67108864~12582912,81362982~15106010,100663296~46137344,150994944~16777216,171966464~75497472,251658240~37748736,290354177~11635711,303838706~52677134,369098752~37748736] 2021-07-31T02:29:54.897 INFO:tasks.workunit.client.0.smithi045.stdout:from 4 to 15 diff [0~20971520,25165824~16777216,46137344~16777216,67108864~29360128,100663296~12582912,117440512~29360128,150994944~12582912,167772160~46137344,218103808~29360128,251658240~33554432,293601280~62914560,369098752~8388608,381681664~12582912,398458880~16777216] 2021-07-31T02:29:54.897 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~20971520,25165824~16777216,46137344~16777216,67108864~29360128,100663296~12582912,117440512~29360128,150994944~12582912,167772160~46137344,218103808~29360128,251658240~33554432,293601280~62914560,369098752~8388608,381681664~12582912,398458880~16777216] 2021-07-31T02:29:54.903 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~41943040,46137344~16777216,67108864~29360128,100663296~46137344,150994944~96468992,251658240~37748736,290354177~66161663,369098752~46137344] 2021-07-31T02:29:54.903 INFO:tasks.workunit.client.0.smithi045.stdout:from 4 to 16 diff [0~20971520,25165824~16777216,46137344~16777216,67108864~29360128,100663296~12582912,117440512~29360128,150994944~12582912,167772160~46137344,218103808~29360128,251658240~33554432,293601280~83886080,381681664~12582912,398458880~16777216] 2021-07-31T02:29:54.903 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~20971520,25165824~16777216,46137344~16777216,67108864~29360128,100663296~12582912,117440512~29360128,150994944~12582912,167772160~46137344,218103808~29360128,251658240~33554432,293601280~83886080,381681664~12582912,398458880~16777216] 2021-07-31T02:29:54.909 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~41943040,46137344~16777216,67108864~29360128,100663296~46137344,150994944~96468992,251658240~37748736,290354177~124881919] 2021-07-31T02:29:54.910 INFO:tasks.workunit.client.0.smithi045.stdout:from 4 to 17 diff [0~20971520,25165824~16777216,46137344~16777216,67108864~29360128,100663296~12582912,117440512~29360128,150994944~12582912,167772160~46137344,218103808~29360128,251658240~33554432,293601280~83886080,381681664~12582912,398458880~16777216] 2021-07-31T02:29:54.910 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~20971520,25165824~16777216,46137344~16777216,67108864~29360128,100663296~12582912,117440512~29360128,150994944~12582912,167772160~46137344,218103808~29360128,251658240~33554432,293601280~83886080,381681664~12582912,398458880~16777216] 2021-07-31T02:29:54.915 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~41943040,46137344~16777216,67108864~29360128,100663296~46137344,150994944~96468992,251658240~37748736,290354177~124881919] 2021-07-31T02:29:54.915 INFO:tasks.workunit.client.0.smithi045.stdout:from 4 to 18 diff [0~20971520,25165824~16777216,46137344~16777216,67108864~79691776,150994944~12582912,167772160~46137344,218103808~29360128,251658240~33554432,293601280~83886080,381681664~12582912,398458880~16777216] 2021-07-31T02:29:54.916 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~20971520,25165824~16777216,46137344~16777216,67108864~79691776,150994944~12582912,167772160~46137344,218103808~29360128,251658240~33554432,293601280~83886080,381681664~12582912,398458880~16777216] 2021-07-31T02:29:54.921 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~41943040,46137344~16777216,67108864~79691776,150994944~96468992,251658240~37748736,290354177~124881919] 2021-07-31T02:29:54.922 INFO:tasks.workunit.client.0.smithi045.stdout:from 4 to 19 diff [0~20971520,25165824~16777216,46137344~100663296,150994944~12582912,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:29:54.922 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~20971520,25165824~16777216,46137344~100663296,150994944~12582912,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:29:54.927 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~41943040,46137344~100663296,150994944~96468992,251658240~37748736,290354177~124881919] 2021-07-31T02:29:54.928 INFO:tasks.workunit.client.0.smithi045.stdout:from 5 to 6 diff [8388608~4194304,25165824~8388608,54525952~8388608,100663296~4194304,150994944~8388608,192937984~4194304,209715200~8388608,251658240~4194304,348127232~8388608,369098752~8388608] 2021-07-31T02:29:54.928 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [8388608~4194304,25165824~8388608,54525952~8388608,100663296~4194304,150994944~8388608,192937984~4194304,209715200~8388608,251658240~4194304,348127232~8388608,369098752~8388608] 2021-07-31T02:29:54.933 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [8388608~8388608,19000239~1971281,25165824~8388608,54146316~8768244,81362982~2523098,92598663~3870329,100663296~8388608,112381922~864286,134792269~3619763,150994944~8388608,160749787~2828069,178010951~2344121,186356067~2387613,192937984~8388608,209715200~8388608,231639272~3241752,246233854~1230082,251658240~8388608,261553838~2687314,279929063~1089305,290354177~3247103,303838706~2345486,325624364~1531348,335499074~45246,347758455~8757385,369098752~8388608,394881756~3577124,400514459~2138725] 2021-07-31T02:29:54.934 INFO:tasks.workunit.client.0.smithi045.stdout:from 5 to 7 diff [8388608~4194304,25165824~8388608,54525952~8388608,100663296~4194304,121634816~4194304,130023424~4194304,150994944~8388608,192937984~4194304,209715200~8388608,239075328~8388608,251658240~4194304,272629760~8388608,306184192~25165824,339738624~16777216,369098752~8388608] 2021-07-31T02:29:54.934 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [8388608~4194304,25165824~8388608,54525952~8388608,100663296~4194304,121634816~4194304,130023424~4194304,150994944~8388608,192937984~4194304,209715200~8388608,239075328~8388608,251658240~4194304,272629760~8388608,306184192~25165824,339738624~16777216,369098752~8388608] 2021-07-31T02:29:54.939 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [8388608~8388608,19000239~1971281,25165824~8388608,54146316~8768244,81362982~2523098,92598663~3870329,100663296~8388608,112381922~864286,121634816~4194304,130023424~4194304,134792269~3619763,150994944~8388608,160749787~2828069,178010951~2344121,186356067~2387613,192937984~8388608,209715200~8388608,231639272~3241752,239075328~8388608,251658240~8388608,261553838~2687314,272629760~12582912,290354177~3247103,303838706~27511310,335499074~45246,339738624~16777216,369098752~12582912,394881756~3577124,400514459~2138725] 2021-07-31T02:29:54.940 INFO:tasks.workunit.client.0.smithi045.stdout:from 5 to 8 diff [8388608~8388608,25165824~8388608,54525952~8388608,75497472~4194304,83886080~8388608,100663296~12582912,121634816~4194304,130023424~4194304,150994944~8388608,171966464~8388608,192937984~4194304,209715200~8388608,226492416~8388608,239075328~8388608,251658240~4194304,272629760~12582912,306184192~25165824,339738624~16777216,369098752~8388608] 2021-07-31T02:29:54.940 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [8388608~8388608,25165824~8388608,54525952~8388608,75497472~4194304,83886080~8388608,100663296~12582912,121634816~4194304,130023424~4194304,150994944~8388608,171966464~8388608,192937984~4194304,209715200~8388608,226492416~8388608,239075328~8388608,251658240~4194304,272629760~12582912,306184192~25165824,339738624~16777216,369098752~8388608] 2021-07-31T02:29:54.946 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [8388608~8388608,19000239~1971281,25165824~8388608,54146316~8768244,75497472~4194304,81362982~10911706,92598663~3870329,100663296~16777216,121634816~4194304,130023424~4194304,134792269~3619763,150994944~8388608,160749787~2828069,171966464~12582912,186356067~2387613,192937984~8388608,209715200~8388608,226492416~20971520,251658240~8388608,261553838~2687314,272629760~16777216,290354177~3247103,303838706~31705614,339738624~16777216,369098752~12582912,394881756~3577124,400514459~2138725] 2021-07-31T02:29:54.946 INFO:tasks.workunit.client.0.smithi045.stdout:from 5 to 9 diff [0~16777216,25165824~8388608,46137344~16777216,75497472~4194304,83886080~8388608,100663296~12582912,121634816~4194304,130023424~8388608,150994944~8388608,171966464~8388608,188743680~8388608,205520896~8388608,226492416~8388608,239075328~8388608,251658240~4194304,272629760~12582912,293601280~8388608,306184192~25165824,339738624~16777216,369098752~8388608,381681664~8388608] 2021-07-31T02:29:54.946 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~16777216,25165824~8388608,46137344~16777216,75497472~4194304,83886080~8388608,100663296~12582912,121634816~4194304,130023424~8388608,150994944~8388608,171966464~8388608,188743680~8388608,205520896~8388608,226492416~8388608,239075328~8388608,251658240~4194304,272629760~12582912,293601280~8388608,306184192~25165824,339738624~16777216,369098752~8388608,381681664~8388608] 2021-07-31T02:29:54.952 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~16777216,19000239~1971281,25165824~8388608,46137344~16777216,75497472~4194304,81362982~10911706,92598663~3870329,100663296~16777216,121634816~4194304,130023424~12582912,150994944~8388608,160749787~2828069,171966464~12582912,186356067~14970525,205520896~12582912,226492416~20971520,251658240~8388608,261553838~2687314,272629760~16777216,290354177~11635711,303838706~31705614,339738624~16777216,369098752~25165824,394881756~3577124,400514459~2138725] 2021-07-31T02:29:54.952 INFO:tasks.workunit.client.0.smithi045.stdout:from 5 to 10 diff [0~16777216,25165824~8388608,46137344~16777216,71303168~8388608,83886080~8388608,100663296~12582912,121634816~4194304,130023424~16777216,150994944~8388608,171966464~8388608,188743680~8388608,205520896~8388608,226492416~20971520,251658240~4194304,268435456~16777216,293601280~8388608,306184192~25165824,339738624~16777216,369098752~8388608,381681664~8388608] 2021-07-31T02:29:54.952 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~16777216,25165824~8388608,46137344~16777216,71303168~8388608,83886080~8388608,100663296~12582912,121634816~4194304,130023424~16777216,150994944~8388608,171966464~8388608,188743680~8388608,205520896~8388608,226492416~20971520,251658240~4194304,268435456~16777216,293601280~8388608,306184192~25165824,339738624~16777216,369098752~8388608,381681664~8388608] 2021-07-31T02:29:54.958 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~16777216,19000239~1971281,25165824~8388608,46137344~16777216,71303168~8388608,81362982~10911706,92598663~3870329,100663296~16777216,121634816~25165824,150994944~8388608,160749787~2828069,171966464~12582912,186356067~14970525,205520896~12582912,226492416~20971520,251658240~8388608,261553838~2687314,268435456~20971520,290354177~11635711,303838706~31705614,339738624~16777216,369098752~25165824,394881756~3577124,400514459~2138725] 2021-07-31T02:29:54.958 INFO:tasks.workunit.client.0.smithi045.stdout:from 5 to 11 diff [0~16777216,25165824~16777216,46137344~16777216,67108864~12582912,83886080~8388608,100663296~12582912,121634816~4194304,130023424~16777216,150994944~12582912,171966464~8388608,184549376~12582912,205520896~8388608,218103808~29360128,251658240~4194304,268435456~16777216,293601280~8388608,306184192~25165824,339738624~16777216,369098752~8388608,381681664~8388608,398458880~8388608] 2021-07-31T02:29:54.958 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~16777216,25165824~16777216,46137344~16777216,67108864~12582912,83886080~8388608,100663296~12582912,121634816~4194304,130023424~16777216,150994944~12582912,171966464~8388608,184549376~12582912,205520896~8388608,218103808~29360128,251658240~4194304,268435456~16777216,293601280~8388608,306184192~25165824,339738624~16777216,369098752~8388608,381681664~8388608,398458880~8388608] 2021-07-31T02:29:54.964 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~16777216,19000239~1971281,25165824~16777216,46137344~16777216,67108864~12582912,81362982~10911706,92598663~3870329,100663296~16777216,121634816~25165824,150994944~16777216,171966464~29360128,205520896~41943040,251658240~8388608,261553838~2687314,268435456~20971520,290354177~11635711,303838706~31705614,339738624~16777216,369098752~25165824,394881756~11965732] 2021-07-31T02:29:54.965 INFO:tasks.workunit.client.0.smithi045.stdout:from 5 to 12 diff [0~16777216,25165824~16777216,46137344~16777216,67108864~12582912,83886080~8388608,100663296~12582912,121634816~25165824,150994944~12582912,171966464~41943040,218103808~29360128,251658240~12582912,268435456~16777216,293601280~8388608,306184192~25165824,335544320~20971520,369098752~8388608,381681664~8388608,398458880~8388608] 2021-07-31T02:29:54.965 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~16777216,25165824~16777216,46137344~16777216,67108864~12582912,83886080~8388608,100663296~12582912,121634816~25165824,150994944~12582912,171966464~41943040,218103808~29360128,251658240~12582912,268435456~16777216,293601280~8388608,306184192~25165824,335544320~20971520,369098752~8388608,381681664~8388608,398458880~8388608] 2021-07-31T02:29:54.971 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~16777216,19000239~1971281,25165824~16777216,46137344~16777216,67108864~12582912,81362982~10911706,92598663~3870329,100663296~16777216,121634816~25165824,150994944~16777216,171966464~75497472,251658240~12582912,268435456~20971520,290354177~11635711,303838706~52677134,369098752~25165824,394881756~11965732] 2021-07-31T02:29:54.971 INFO:tasks.workunit.client.0.smithi045.stdout:from 5 to 13 diff [0~16777216,25165824~16777216,46137344~16777216,67108864~12582912,83886080~12582912,100663296~12582912,117440512~29360128,150994944~12582912,171966464~41943040,218103808~29360128,251658240~12582912,268435456~16777216,293601280~8388608,306184192~50331648,369098752~8388608,381681664~12582912,398458880~8388608] 2021-07-31T02:29:54.971 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~16777216,25165824~16777216,46137344~16777216,67108864~12582912,83886080~12582912,100663296~12582912,117440512~29360128,150994944~12582912,171966464~41943040,218103808~29360128,251658240~12582912,268435456~16777216,293601280~8388608,306184192~50331648,369098752~8388608,381681664~12582912,398458880~8388608] 2021-07-31T02:29:54.977 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~16777216,19000239~1971281,25165824~16777216,46137344~16777216,67108864~12582912,81362982~15106010,100663296~46137344,150994944~16777216,171966464~75497472,251658240~12582912,268435456~20971520,290354177~11635711,303838706~52677134,369098752~37748736] 2021-07-31T02:29:54.977 INFO:tasks.workunit.client.0.smithi045.stdout:from 5 to 14 diff [0~20971520,25165824~16777216,46137344~16777216,67108864~12582912,83886080~12582912,100663296~12582912,117440512~29360128,150994944~12582912,171966464~41943040,218103808~29360128,251658240~12582912,268435456~16777216,293601280~8388608,306184192~50331648,369098752~8388608,381681664~12582912,398458880~8388608] 2021-07-31T02:29:54.978 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~20971520,25165824~16777216,46137344~16777216,67108864~12582912,83886080~12582912,100663296~12582912,117440512~29360128,150994944~12582912,171966464~41943040,218103808~29360128,251658240~12582912,268435456~16777216,293601280~8388608,306184192~50331648,369098752~8388608,381681664~12582912,398458880~8388608] 2021-07-31T02:29:54.983 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~41943040,46137344~16777216,67108864~12582912,81362982~15106010,100663296~46137344,150994944~16777216,171966464~75497472,251658240~12582912,268435456~20971520,290354177~11635711,303838706~52677134,369098752~37748736] 2021-07-31T02:29:54.984 INFO:tasks.workunit.client.0.smithi045.stdout:from 5 to 15 diff [0~20971520,25165824~16777216,46137344~16777216,67108864~29360128,100663296~12582912,117440512~29360128,150994944~12582912,167772160~46137344,218103808~29360128,251658240~12582912,268435456~16777216,293601280~62914560,369098752~8388608,381681664~12582912,398458880~16777216] 2021-07-31T02:29:54.985 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~20971520,25165824~16777216,46137344~16777216,67108864~29360128,100663296~12582912,117440512~29360128,150994944~12582912,167772160~46137344,218103808~29360128,251658240~12582912,268435456~16777216,293601280~62914560,369098752~8388608,381681664~12582912,398458880~16777216] 2021-07-31T02:29:54.989 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~41943040,46137344~16777216,67108864~29360128,100663296~46137344,150994944~96468992,251658240~12582912,268435456~20971520,290354177~66161663,369098752~46137344] 2021-07-31T02:29:54.989 INFO:tasks.workunit.client.0.smithi045.stdout:from 5 to 16 diff [0~20971520,25165824~16777216,46137344~16777216,67108864~29360128,100663296~12582912,117440512~29360128,150994944~12582912,167772160~46137344,218103808~29360128,251658240~12582912,268435456~16777216,293601280~83886080,381681664~12582912,398458880~16777216] 2021-07-31T02:29:54.989 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~20971520,25165824~16777216,46137344~16777216,67108864~29360128,100663296~12582912,117440512~29360128,150994944~12582912,167772160~46137344,218103808~29360128,251658240~12582912,268435456~16777216,293601280~83886080,381681664~12582912,398458880~16777216] 2021-07-31T02:29:54.995 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~41943040,46137344~16777216,67108864~29360128,100663296~46137344,150994944~96468992,251658240~12582912,268435456~20971520,290354177~124881919] 2021-07-31T02:29:54.995 INFO:tasks.workunit.client.0.smithi045.stdout:from 5 to 17 diff [0~20971520,25165824~16777216,46137344~16777216,67108864~29360128,100663296~12582912,117440512~29360128,150994944~12582912,167772160~46137344,218103808~29360128,251658240~33554432,293601280~83886080,381681664~12582912,398458880~16777216] 2021-07-31T02:29:54.996 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~20971520,25165824~16777216,46137344~16777216,67108864~29360128,100663296~12582912,117440512~29360128,150994944~12582912,167772160~46137344,218103808~29360128,251658240~33554432,293601280~83886080,381681664~12582912,398458880~16777216] 2021-07-31T02:29:55.001 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~41943040,46137344~16777216,67108864~29360128,100663296~46137344,150994944~96468992,251658240~37748736,290354177~124881919] 2021-07-31T02:29:55.002 INFO:tasks.workunit.client.0.smithi045.stdout:from 5 to 18 diff [0~20971520,25165824~16777216,46137344~16777216,67108864~79691776,150994944~12582912,167772160~46137344,218103808~29360128,251658240~33554432,293601280~83886080,381681664~12582912,398458880~16777216] 2021-07-31T02:29:55.002 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~20971520,25165824~16777216,46137344~16777216,67108864~79691776,150994944~12582912,167772160~46137344,218103808~29360128,251658240~33554432,293601280~83886080,381681664~12582912,398458880~16777216] 2021-07-31T02:29:55.008 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~41943040,46137344~16777216,67108864~79691776,150994944~96468992,251658240~37748736,290354177~124881919] 2021-07-31T02:29:55.008 INFO:tasks.workunit.client.0.smithi045.stdout:from 5 to 19 diff [0~20971520,25165824~16777216,46137344~100663296,150994944~12582912,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:29:55.008 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~20971520,25165824~16777216,46137344~100663296,150994944~12582912,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:29:55.014 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~41943040,46137344~100663296,150994944~96468992,251658240~37748736,290354177~124881919] 2021-07-31T02:29:55.014 INFO:tasks.workunit.client.0.smithi045.stdout:from 6 to 7 diff [29360128~4194304,121634816~4194304,130023424~4194304,239075328~8388608,272629760~8388608,306184192~25165824,339738624~8388608,369098752~8388608] 2021-07-31T02:29:55.014 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [29360128~4194304,121634816~4194304,130023424~4194304,239075328~8388608,272629760~8388608,306184192~25165824,339738624~8388608,369098752~8388608] 2021-07-31T02:29:55.020 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [16439262~337954,19000239~1971281,29360128~4194304,54146316~379636,62262910~651650,81362982~2523098,92598663~3870329,112381922~864286,121634816~4194304,130023424~4194304,134792269~3619763,160749787~2828069,178010951~2344121,186356067~2387613,200290234~1036358,215110864~2992944,231639272~3241752,239075328~8388608,261553838~2687314,272629760~12582912,290354177~3247103,303838706~27511310,335499074~45246,339738624~12582912,356296674~219166,369098752~12582912,394881756~3577124,400514459~2138725] 2021-07-31T02:29:55.021 INFO:tasks.workunit.client.0.smithi045.stdout:from 6 to 8 diff [8388608~8388608,29360128~4194304,75497472~4194304,83886080~8388608,104857600~8388608,121634816~4194304,130023424~4194304,171966464~8388608,226492416~8388608,239075328~8388608,272629760~12582912,306184192~25165824,339738624~8388608,369098752~8388608] 2021-07-31T02:29:55.021 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [8388608~8388608,29360128~4194304,75497472~4194304,83886080~8388608,104857600~8388608,121634816~4194304,130023424~4194304,171966464~8388608,226492416~8388608,239075328~8388608,272629760~12582912,306184192~25165824,339738624~8388608,369098752~8388608] 2021-07-31T02:29:55.026 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [8388608~8388608,19000239~1971281,29360128~4194304,54146316~379636,62262910~651650,75497472~4194304,81362982~10911706,92598663~3870329,104857600~12582912,121634816~4194304,130023424~4194304,134792269~3619763,160749787~2828069,171966464~12582912,186356067~2387613,200290234~1036358,215110864~2992944,226492416~20971520,261553838~2687314,272629760~16777216,290354177~3247103,303838706~31705614,339738624~12582912,356296674~219166,369098752~12582912,394881756~3577124,400514459~2138725] 2021-07-31T02:29:55.027 INFO:tasks.workunit.client.0.smithi045.stdout:from 6 to 9 diff [0~16777216,29360128~4194304,46137344~8388608,75497472~4194304,83886080~8388608,104857600~8388608,121634816~4194304,130023424~8388608,171966464~8388608,188743680~4194304,205520896~8388608,226492416~8388608,239075328~8388608,272629760~12582912,293601280~8388608,306184192~25165824,339738624~8388608,369098752~8388608,381681664~8388608] 2021-07-31T02:29:55.028 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~16777216,29360128~4194304,46137344~8388608,75497472~4194304,83886080~8388608,104857600~8388608,121634816~4194304,130023424~8388608,171966464~8388608,188743680~4194304,205520896~8388608,226492416~8388608,239075328~8388608,272629760~12582912,293601280~8388608,306184192~25165824,339738624~8388608,369098752~8388608,381681664~8388608] 2021-07-31T02:29:55.032 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~16777216,19000239~1971281,29360128~4194304,46137344~12582912,62262910~651650,75497472~4194304,81362982~10911706,92598663~3870329,104857600~12582912,121634816~4194304,130023424~12582912,160749787~2828069,171966464~12582912,186356067~10776221,200290234~1036358,205520896~8388608,215110864~2992944,226492416~20971520,261553838~2687314,272629760~16777216,290354177~11635711,303838706~31705614,339738624~12582912,356296674~219166,369098752~25165824,394881756~3577124,400514459~2138725] 2021-07-31T02:29:55.032 INFO:tasks.workunit.client.0.smithi045.stdout:from 6 to 10 diff [0~16777216,29360128~4194304,46137344~16777216,71303168~8388608,83886080~8388608,104857600~8388608,121634816~4194304,130023424~16777216,150994944~4194304,171966464~8388608,188743680~4194304,205520896~8388608,226492416~20971520,268435456~16777216,293601280~8388608,306184192~25165824,339738624~8388608,369098752~8388608,381681664~8388608] 2021-07-31T02:29:55.032 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~16777216,29360128~4194304,46137344~16777216,71303168~8388608,83886080~8388608,104857600~8388608,121634816~4194304,130023424~16777216,150994944~4194304,171966464~8388608,188743680~4194304,205520896~8388608,226492416~20971520,268435456~16777216,293601280~8388608,306184192~25165824,339738624~8388608,369098752~8388608,381681664~8388608] 2021-07-31T02:29:55.038 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~16777216,19000239~1971281,29360128~4194304,46137344~16777216,71303168~8388608,81362982~10911706,92598663~3870329,104857600~12582912,121634816~25165824,150994944~8388608,160749787~2828069,171966464~12582912,186356067~10776221,200290234~1036358,205520896~8388608,215110864~2992944,226492416~20971520,261553838~2687314,268435456~20971520,290354177~11635711,303838706~31705614,339738624~12582912,356296674~219166,369098752~25165824,394881756~3577124,400514459~2138725] 2021-07-31T02:29:55.039 INFO:tasks.workunit.client.0.smithi045.stdout:from 6 to 11 diff [0~16777216,29360128~12582912,46137344~16777216,67108864~12582912,83886080~8388608,104857600~8388608,121634816~4194304,130023424~16777216,150994944~4194304,159383552~4194304,171966464~8388608,184549376~8388608,205520896~8388608,218103808~29360128,268435456~16777216,293601280~8388608,306184192~25165824,339738624~8388608,369098752~8388608,381681664~8388608,398458880~8388608] 2021-07-31T02:29:55.039 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~16777216,29360128~12582912,46137344~16777216,67108864~12582912,83886080~8388608,104857600~8388608,121634816~4194304,130023424~16777216,150994944~4194304,159383552~4194304,171966464~8388608,184549376~8388608,205520896~8388608,218103808~29360128,268435456~16777216,293601280~8388608,306184192~25165824,339738624~8388608,369098752~8388608,381681664~8388608,398458880~8388608] 2021-07-31T02:29:55.045 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~16777216,19000239~1971281,29360128~12582912,46137344~16777216,67108864~12582912,81362982~10911706,92598663~3870329,104857600~12582912,121634816~25165824,150994944~16777216,171966464~25165824,200290234~1036358,205520896~8388608,215110864~32353072,261553838~2687314,268435456~20971520,290354177~11635711,303838706~31705614,339738624~12582912,356296674~219166,369098752~25165824,394881756~11965732] 2021-07-31T02:29:55.045 INFO:tasks.workunit.client.0.smithi045.stdout:from 6 to 12 diff [0~16777216,29360128~12582912,46137344~16777216,67108864~12582912,83886080~8388608,104857600~8388608,121634816~25165824,150994944~4194304,159383552~4194304,171966464~20971520,197132288~12582912,218103808~29360128,251658240~8388608,268435456~16777216,293601280~8388608,306184192~25165824,335544320~16777216,369098752~8388608,381681664~8388608,398458880~8388608] 2021-07-31T02:29:55.046 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~16777216,29360128~12582912,46137344~16777216,67108864~12582912,83886080~8388608,104857600~8388608,121634816~25165824,150994944~4194304,159383552~4194304,171966464~20971520,197132288~12582912,218103808~29360128,251658240~8388608,268435456~16777216,293601280~8388608,306184192~25165824,335544320~16777216,369098752~8388608,381681664~8388608,398458880~8388608] 2021-07-31T02:29:55.050 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~16777216,19000239~1971281,29360128~12582912,46137344~16777216,67108864~12582912,81362982~10911706,92598663~3870329,104857600~12582912,121634816~25165824,150994944~16777216,171966464~41943040,215110864~32353072,251658240~12582912,268435456~20971520,290354177~11635711,303838706~52677134,369098752~25165824,394881756~11965732] 2021-07-31T02:29:55.051 INFO:tasks.workunit.client.0.smithi045.stdout:from 6 to 13 diff [0~16777216,29360128~12582912,46137344~16777216,67108864~12582912,83886080~12582912,104857600~8388608,117440512~29360128,150994944~4194304,159383552~4194304,171966464~37748736,218103808~29360128,251658240~8388608,268435456~16777216,293601280~8388608,306184192~46137344,369098752~8388608,381681664~12582912,398458880~8388608] 2021-07-31T02:29:55.051 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~16777216,29360128~12582912,46137344~16777216,67108864~12582912,83886080~12582912,104857600~8388608,117440512~29360128,150994944~4194304,159383552~4194304,171966464~37748736,218103808~29360128,251658240~8388608,268435456~16777216,293601280~8388608,306184192~46137344,369098752~8388608,381681664~12582912,398458880~8388608] 2021-07-31T02:29:55.056 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~16777216,19000239~1971281,29360128~12582912,46137344~16777216,67108864~12582912,81362982~15106010,104857600~41943040,150994944~16777216,171966464~41943040,215110864~32353072,251658240~12582912,268435456~20971520,290354177~11635711,303838706~52677134,369098752~37748736] 2021-07-31T02:29:55.057 INFO:tasks.workunit.client.0.smithi045.stdout:from 6 to 14 diff [0~20971520,29360128~12582912,46137344~16777216,67108864~12582912,83886080~12582912,104857600~8388608,117440512~29360128,150994944~4194304,159383552~4194304,171966464~37748736,218103808~29360128,251658240~8388608,268435456~16777216,293601280~8388608,306184192~50331648,369098752~8388608,381681664~12582912,398458880~8388608] 2021-07-31T02:29:55.057 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~20971520,29360128~12582912,46137344~16777216,67108864~12582912,83886080~12582912,104857600~8388608,117440512~29360128,150994944~4194304,159383552~4194304,171966464~37748736,218103808~29360128,251658240~8388608,268435456~16777216,293601280~8388608,306184192~50331648,369098752~8388608,381681664~12582912,398458880~8388608] 2021-07-31T02:29:55.063 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~25165824,29360128~12582912,46137344~16777216,67108864~12582912,81362982~15106010,104857600~41943040,150994944~16777216,171966464~41943040,215110864~32353072,251658240~12582912,268435456~20971520,290354177~11635711,303838706~52677134,369098752~37748736] 2021-07-31T02:29:55.063 INFO:tasks.workunit.client.0.smithi045.stdout:from 6 to 15 diff [0~20971520,29360128~12582912,46137344~16777216,67108864~29360128,104857600~8388608,117440512~29360128,150994944~4194304,159383552~4194304,167772160~41943040,218103808~29360128,251658240~8388608,268435456~16777216,293601280~62914560,369098752~8388608,381681664~12582912,398458880~16777216] 2021-07-31T02:29:55.063 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~20971520,29360128~12582912,46137344~16777216,67108864~29360128,104857600~8388608,117440512~29360128,150994944~4194304,159383552~4194304,167772160~41943040,218103808~29360128,251658240~8388608,268435456~16777216,293601280~62914560,369098752~8388608,381681664~12582912,398458880~16777216] 2021-07-31T02:29:55.069 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~25165824,29360128~12582912,46137344~16777216,67108864~29360128,104857600~41943040,150994944~62914560,215110864~32353072,251658240~12582912,268435456~20971520,290354177~66161663,369098752~46137344] 2021-07-31T02:29:55.069 INFO:tasks.workunit.client.0.smithi045.stdout:from 6 to 16 diff [0~20971520,29360128~12582912,46137344~16777216,67108864~29360128,104857600~8388608,117440512~29360128,150994944~12582912,167772160~41943040,218103808~29360128,251658240~8388608,268435456~16777216,293601280~83886080,381681664~12582912,398458880~16777216] 2021-07-31T02:29:55.069 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~20971520,29360128~12582912,46137344~16777216,67108864~29360128,104857600~8388608,117440512~29360128,150994944~12582912,167772160~41943040,218103808~29360128,251658240~8388608,268435456~16777216,293601280~83886080,381681664~12582912,398458880~16777216] 2021-07-31T02:29:55.075 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~25165824,29360128~12582912,46137344~16777216,67108864~29360128,104857600~41943040,150994944~62914560,215110864~32353072,251658240~12582912,268435456~20971520,290354177~124881919] 2021-07-31T02:29:55.075 INFO:tasks.workunit.client.0.smithi045.stdout:from 6 to 17 diff [0~20971520,29360128~12582912,46137344~16777216,67108864~29360128,104857600~8388608,117440512~29360128,150994944~12582912,167772160~41943040,218103808~29360128,251658240~8388608,264241152~20971520,293601280~83886080,381681664~12582912,398458880~16777216] 2021-07-31T02:29:55.075 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~20971520,29360128~12582912,46137344~16777216,67108864~29360128,104857600~8388608,117440512~29360128,150994944~12582912,167772160~41943040,218103808~29360128,251658240~8388608,264241152~20971520,293601280~83886080,381681664~12582912,398458880~16777216] 2021-07-31T02:29:55.081 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~25165824,29360128~12582912,46137344~16777216,67108864~29360128,104857600~41943040,150994944~62914560,215110864~32353072,251658240~37748736,290354177~124881919] 2021-07-31T02:29:55.081 INFO:tasks.workunit.client.0.smithi045.stdout:from 6 to 18 diff [0~20971520,29360128~12582912,46137344~16777216,67108864~79691776,150994944~12582912,167772160~41943040,218103808~29360128,251658240~8388608,264241152~20971520,293601280~83886080,381681664~12582912,398458880~16777216] 2021-07-31T02:29:55.082 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~20971520,29360128~12582912,46137344~16777216,67108864~79691776,150994944~12582912,167772160~41943040,218103808~29360128,251658240~8388608,264241152~20971520,293601280~83886080,381681664~12582912,398458880~16777216] 2021-07-31T02:29:55.088 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~25165824,29360128~12582912,46137344~16777216,67108864~79691776,150994944~62914560,215110864~32353072,251658240~37748736,290354177~124881919] 2021-07-31T02:29:55.088 INFO:tasks.workunit.client.0.smithi045.stdout:from 6 to 19 diff [0~20971520,29360128~12582912,46137344~100663296,150994944~12582912,167772160~41943040,218103808~29360128,251658240~8388608,264241152~20971520,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:29:55.088 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~20971520,29360128~12582912,46137344~100663296,150994944~12582912,167772160~41943040,218103808~29360128,251658240~8388608,264241152~20971520,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:29:55.094 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~25165824,29360128~12582912,46137344~100663296,150994944~62914560,215110864~32353072,251658240~37748736,290354177~124881919] 2021-07-31T02:29:55.094 INFO:tasks.workunit.client.0.smithi045.stdout:from 7 to 8 diff [8388608~8388608,75497472~4194304,83886080~8388608,104857600~8388608,171966464~8388608,226492416~8388608,276824064~8388608,322961408~8388608] 2021-07-31T02:29:55.094 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [8388608~8388608,75497472~4194304,83886080~8388608,104857600~8388608,171966464~8388608,226492416~8388608,276824064~8388608,322961408~8388608] 2021-07-31T02:29:55.100 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [8388608~8388608,19000239~1971281,33446354~108078,54146316~379636,62262910~651650,75497472~4194304,81362982~10911706,92598663~3870329,104857600~12582912,134792269~3619763,160749787~2828069,171966464~12582912,186356067~2387613,200290234~1036358,215110864~2992944,226492416~12582912,247142031~321905,261553838~2687314,276824064~12582912,290354177~3247103,303838706~2345486,322961408~12582912,356296674~219166,394881756~3577124,400514459~2138725] 2021-07-31T02:29:55.100 INFO:tasks.workunit.client.0.smithi045.stdout:from 7 to 9 diff [0~16777216,46137344~8388608,75497472~4194304,83886080~8388608,104857600~8388608,134217728~4194304,171966464~8388608,188743680~4194304,205520896~8388608,226492416~8388608,276824064~8388608,293601280~8388608,322961408~8388608,343932928~4194304,381681664~8388608] 2021-07-31T02:29:55.101 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~16777216,46137344~8388608,75497472~4194304,83886080~8388608,104857600~8388608,134217728~4194304,171966464~8388608,188743680~4194304,205520896~8388608,226492416~8388608,276824064~8388608,293601280~8388608,322961408~8388608,343932928~4194304,381681664~8388608] 2021-07-31T02:29:55.107 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~16777216,19000239~1971281,33446354~108078,46137344~12582912,62262910~651650,75497472~4194304,81362982~10911706,92598663~3870329,104857600~12582912,134217728~8388608,160749787~2828069,171966464~12582912,186356067~10776221,200290234~1036358,205520896~8388608,215110864~2992944,226492416~12582912,247142031~321905,261553838~2687314,276824064~12582912,290354177~11635711,303838706~2345486,322961408~12582912,343932928~4194304,356296674~219166,381681664~12582912,394881756~3577124,400514459~2138725] 2021-07-31T02:29:55.107 INFO:tasks.workunit.client.0.smithi045.stdout:from 7 to 10 diff [0~16777216,46137344~16777216,71303168~8388608,83886080~8388608,104857600~8388608,121634816~4194304,134217728~12582912,150994944~4194304,171966464~8388608,188743680~4194304,205520896~8388608,226492416~16777216,268435456~16777216,293601280~8388608,322961408~8388608,339738624~8388608,369098752~4194304,381681664~8388608] 2021-07-31T02:29:55.107 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~16777216,46137344~16777216,71303168~8388608,83886080~8388608,104857600~8388608,121634816~4194304,134217728~12582912,150994944~4194304,171966464~8388608,188743680~4194304,205520896~8388608,226492416~16777216,268435456~16777216,293601280~8388608,322961408~8388608,339738624~8388608,369098752~4194304,381681664~8388608] 2021-07-31T02:29:55.112 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~16777216,19000239~1971281,33446354~108078,46137344~16777216,71303168~8388608,81362982~10911706,92598663~3870329,104857600~12582912,121634816~8388608,134217728~12582912,150994944~8388608,160749787~2828069,171966464~12582912,186356067~10776221,200290234~1036358,205520896~8388608,215110864~2992944,226492416~16777216,247142031~321905,261553838~2687314,268435456~20971520,290354177~11635711,303838706~2345486,322961408~12582912,339738624~12582912,356296674~219166,369098752~8388608,381681664~12582912,394881756~3577124,400514459~2138725] 2021-07-31T02:29:55.112 INFO:tasks.workunit.client.0.smithi045.stdout:from 7 to 11 diff [0~16777216,33554432~8388608,46137344~16777216,67108864~12582912,83886080~8388608,104857600~8388608,121634816~4194304,134217728~12582912,150994944~4194304,159383552~4194304,171966464~8388608,184549376~8388608,205520896~8388608,218103808~25165824,268435456~16777216,293601280~8388608,322961408~8388608,339738624~8388608,369098752~4194304,381681664~8388608,398458880~8388608] 2021-07-31T02:29:55.113 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~16777216,33554432~8388608,46137344~16777216,67108864~12582912,83886080~8388608,104857600~8388608,121634816~4194304,134217728~12582912,150994944~4194304,159383552~4194304,171966464~8388608,184549376~8388608,205520896~8388608,218103808~25165824,268435456~16777216,293601280~8388608,322961408~8388608,339738624~8388608,369098752~4194304,381681664~8388608,398458880~8388608] 2021-07-31T02:29:55.119 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~16777216,19000239~1971281,33446354~8496686,46137344~16777216,67108864~12582912,81362982~10911706,92598663~3870329,104857600~12582912,121634816~8388608,134217728~12582912,150994944~16777216,171966464~25165824,200290234~1036358,205520896~8388608,215110864~28158768,247142031~321905,261553838~2687314,268435456~20971520,290354177~11635711,303838706~2345486,322961408~12582912,339738624~12582912,356296674~219166,369098752~8388608,381681664~12582912,394881756~11965732] 2021-07-31T02:29:55.119 INFO:tasks.workunit.client.0.smithi045.stdout:from 7 to 12 diff [0~16777216,33554432~8388608,46137344~16777216,67108864~12582912,83886080~8388608,104857600~8388608,121634816~25165824,150994944~4194304,159383552~4194304,171966464~20971520,197132288~12582912,218103808~25165824,251658240~8388608,268435456~16777216,293601280~8388608,314572800~4194304,322961408~8388608,335544320~16777216,369098752~4194304,381681664~8388608,398458880~8388608] 2021-07-31T02:29:55.120 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~16777216,33554432~8388608,46137344~16777216,67108864~12582912,83886080~8388608,104857600~8388608,121634816~25165824,150994944~4194304,159383552~4194304,171966464~20971520,197132288~12582912,218103808~25165824,251658240~8388608,268435456~16777216,293601280~8388608,314572800~4194304,322961408~8388608,335544320~16777216,369098752~4194304,381681664~8388608,398458880~8388608] 2021-07-31T02:29:55.126 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~16777216,19000239~1971281,33446354~8496686,46137344~16777216,67108864~12582912,81362982~10911706,92598663~3870329,104857600~12582912,121634816~25165824,150994944~16777216,171966464~41943040,215110864~28158768,247142031~321905,251658240~12582912,268435456~20971520,290354177~11635711,303838706~2345486,314572800~4194304,322961408~33554432,369098752~8388608,381681664~12582912,394881756~11965732] 2021-07-31T02:29:55.126 INFO:tasks.workunit.client.0.smithi045.stdout:from 7 to 13 diff [0~16777216,33554432~8388608,46137344~16777216,67108864~12582912,83886080~12582912,104857600~8388608,117440512~29360128,150994944~4194304,159383552~4194304,171966464~37748736,218103808~25165824,251658240~8388608,268435456~16777216,293601280~8388608,314572800~4194304,322961408~29360128,369098752~4194304,381681664~12582912,398458880~8388608] 2021-07-31T02:29:55.126 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~16777216,33554432~8388608,46137344~16777216,67108864~12582912,83886080~12582912,104857600~8388608,117440512~29360128,150994944~4194304,159383552~4194304,171966464~37748736,218103808~25165824,251658240~8388608,268435456~16777216,293601280~8388608,314572800~4194304,322961408~29360128,369098752~4194304,381681664~12582912,398458880~8388608] 2021-07-31T02:29:55.131 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~16777216,19000239~1971281,33446354~8496686,46137344~16777216,67108864~12582912,81362982~15106010,104857600~41943040,150994944~16777216,171966464~41943040,215110864~28158768,247142031~321905,251658240~12582912,268435456~20971520,290354177~11635711,303838706~2345486,314572800~4194304,322961408~33554432,369098752~8388608,381681664~25165824] 2021-07-31T02:29:55.131 INFO:tasks.workunit.client.0.smithi045.stdout:from 7 to 14 diff [0~20971520,33554432~8388608,46137344~16777216,67108864~12582912,83886080~12582912,104857600~8388608,117440512~29360128,150994944~4194304,159383552~4194304,171966464~37748736,218103808~25165824,251658240~8388608,268435456~16777216,293601280~8388608,314572800~4194304,322961408~33554432,369098752~4194304,381681664~12582912,398458880~8388608] 2021-07-31T02:29:55.132 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~20971520,33554432~8388608,46137344~16777216,67108864~12582912,83886080~12582912,104857600~8388608,117440512~29360128,150994944~4194304,159383552~4194304,171966464~37748736,218103808~25165824,251658240~8388608,268435456~16777216,293601280~8388608,314572800~4194304,322961408~33554432,369098752~4194304,381681664~12582912,398458880~8388608] 2021-07-31T02:29:55.137 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~25165824,33446354~8496686,46137344~16777216,67108864~12582912,81362982~15106010,104857600~41943040,150994944~16777216,171966464~41943040,215110864~28158768,247142031~321905,251658240~12582912,268435456~20971520,290354177~11635711,303838706~2345486,314572800~4194304,322961408~33554432,369098752~8388608,381681664~25165824] 2021-07-31T02:29:55.137 INFO:tasks.workunit.client.0.smithi045.stdout:from 7 to 15 diff [0~20971520,33554432~8388608,46137344~16777216,67108864~29360128,104857600~8388608,117440512~29360128,150994944~4194304,159383552~4194304,167772160~41943040,218103808~25165824,251658240~8388608,268435456~16777216,293601280~12582912,314572800~4194304,322961408~33554432,369098752~8388608,381681664~12582912,398458880~16777216] 2021-07-31T02:29:55.137 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~20971520,33554432~8388608,46137344~16777216,67108864~29360128,104857600~8388608,117440512~29360128,150994944~4194304,159383552~4194304,167772160~41943040,218103808~25165824,251658240~8388608,268435456~16777216,293601280~12582912,314572800~4194304,322961408~33554432,369098752~8388608,381681664~12582912,398458880~16777216] 2021-07-31T02:29:55.143 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~25165824,33446354~8496686,46137344~16777216,67108864~29360128,104857600~41943040,150994944~62914560,215110864~28158768,247142031~321905,251658240~12582912,268435456~20971520,290354177~15830015,314572800~4194304,322961408~33554432,369098752~46137344] 2021-07-31T02:29:55.144 INFO:tasks.workunit.client.0.smithi045.stdout:from 7 to 16 diff [0~20971520,29360128~12582912,46137344~16777216,67108864~29360128,104857600~8388608,117440512~29360128,150994944~12582912,167772160~41943040,218103808~25165824,251658240~8388608,268435456~16777216,293601280~12582912,314572800~4194304,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:29:55.144 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~20971520,29360128~12582912,46137344~16777216,67108864~29360128,104857600~8388608,117440512~29360128,150994944~12582912,167772160~41943040,218103808~25165824,251658240~8388608,268435456~16777216,293601280~12582912,314572800~4194304,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:29:55.149 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~25165824,29360128~12582912,46137344~16777216,67108864~29360128,104857600~41943040,150994944~62914560,215110864~28158768,247142031~321905,251658240~12582912,268435456~20971520,290354177~15830015,314572800~4194304,322961408~92274688] 2021-07-31T02:29:55.150 INFO:tasks.workunit.client.0.smithi045.stdout:from 7 to 17 diff [0~20971520,29360128~12582912,46137344~16777216,67108864~29360128,104857600~8388608,117440512~29360128,150994944~12582912,167772160~41943040,218103808~25165824,251658240~8388608,264241152~20971520,293601280~12582912,314572800~4194304,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:29:55.150 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~20971520,29360128~12582912,46137344~16777216,67108864~29360128,104857600~8388608,117440512~29360128,150994944~12582912,167772160~41943040,218103808~25165824,251658240~8388608,264241152~20971520,293601280~12582912,314572800~4194304,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:29:55.156 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~25165824,29360128~12582912,46137344~16777216,67108864~29360128,104857600~41943040,150994944~62914560,215110864~32353072,251658240~37748736,290354177~15830015,314572800~4194304,322961408~92274688] 2021-07-31T02:29:55.156 INFO:tasks.workunit.client.0.smithi045.stdout:from 7 to 18 diff [0~20971520,29360128~12582912,46137344~16777216,67108864~79691776,150994944~12582912,167772160~41943040,218103808~25165824,251658240~8388608,264241152~20971520,293601280~12582912,314572800~4194304,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:29:55.156 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~20971520,29360128~12582912,46137344~16777216,67108864~79691776,150994944~12582912,167772160~41943040,218103808~25165824,251658240~8388608,264241152~20971520,293601280~12582912,314572800~4194304,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:29:55.162 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~25165824,29360128~12582912,46137344~16777216,67108864~79691776,150994944~62914560,215110864~32353072,251658240~37748736,290354177~15830015,314572800~4194304,322961408~92274688] 2021-07-31T02:29:55.162 INFO:tasks.workunit.client.0.smithi045.stdout:from 7 to 19 diff [0~20971520,29360128~12582912,46137344~100663296,150994944~12582912,167772160~41943040,218103808~25165824,251658240~8388608,264241152~20971520,293601280~20971520,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:29:55.163 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~20971520,29360128~12582912,46137344~100663296,150994944~12582912,167772160~41943040,218103808~25165824,251658240~8388608,264241152~20971520,293601280~20971520,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:29:55.168 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~25165824,29360128~12582912,46137344~100663296,150994944~62914560,215110864~32353072,251658240~37748736,290354177~28412927,322961408~92274688] 2021-07-31T02:29:55.169 INFO:tasks.workunit.client.0.smithi045.stdout:from 8 to 9 diff [0~8388608,46137344~8388608,134217728~4194304,188743680~4194304,205520896~8388608,293601280~8388608,343932928~4194304,381681664~8388608] 2021-07-31T02:29:55.169 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~8388608,46137344~8388608,134217728~4194304,188743680~4194304,205520896~8388608,293601280~8388608,343932928~4194304,381681664~8388608] 2021-07-31T02:29:55.174 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~8388608,16439262~337954,19000239~1971281,33446354~108078,46137344~12582912,62262910~651650,81362982~2523098,92598663~3870329,113251019~4189493,134217728~8388608,160749787~2828069,186356067~10776221,200290234~1036358,205520896~8388608,215110864~2992944,247142031~321905,261553838~2687314,290354177~11635711,303838706~2345486,335499074~45246,343932928~4194304,356296674~219166,381681664~12582912,394881756~3577124,400514459~2138725] 2021-07-31T02:29:55.175 INFO:tasks.workunit.client.0.smithi045.stdout:from 8 to 10 diff [0~8388608,46137344~16777216,71303168~8388608,121634816~4194304,134217728~12582912,150994944~4194304,188743680~4194304,205520896~8388608,230686720~8388608,268435456~12582912,293601280~8388608,339738624~8388608,369098752~4194304,381681664~8388608] 2021-07-31T02:29:55.175 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~8388608,46137344~16777216,71303168~8388608,121634816~4194304,134217728~12582912,150994944~4194304,188743680~4194304,205520896~8388608,230686720~8388608,268435456~12582912,293601280~8388608,339738624~8388608,369098752~4194304,381681664~8388608] 2021-07-31T02:29:55.180 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~8388608,16439262~337954,19000239~1971281,33446354~108078,46137344~16777216,71303168~8388608,81362982~2523098,92598663~3870329,113251019~4189493,121634816~8388608,134217728~12582912,150994944~8388608,160749787~2828069,186356067~10776221,200290234~1036358,205520896~8388608,215110864~2992944,230686720~12582912,247142031~321905,261553838~2687314,268435456~16777216,290354177~11635711,303838706~2345486,335499074~45246,339738624~12582912,356296674~219166,369098752~8388608,381681664~12582912,394881756~3577124,400514459~2138725] 2021-07-31T02:29:55.181 INFO:tasks.workunit.client.0.smithi045.stdout:from 8 to 11 diff [0~8388608,33554432~8388608,46137344~16777216,67108864~8388608,109051904~8388608,121634816~4194304,134217728~12582912,150994944~4194304,159383552~4194304,184549376~8388608,205520896~8388608,218103808~8388608,230686720~8388608,268435456~12582912,293601280~8388608,339738624~8388608,369098752~4194304,381681664~8388608,398458880~8388608] 2021-07-31T02:29:55.181 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~8388608,33554432~8388608,46137344~16777216,67108864~8388608,109051904~8388608,121634816~4194304,134217728~12582912,150994944~4194304,159383552~4194304,184549376~8388608,205520896~8388608,218103808~8388608,230686720~8388608,268435456~12582912,293601280~8388608,339738624~8388608,369098752~4194304,381681664~8388608,398458880~8388608] 2021-07-31T02:29:55.187 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~8388608,16439262~337954,19000239~1971281,33446354~8496686,46137344~16777216,67108864~12582912,81362982~2523098,92598663~3870329,109051904~8388608,121634816~8388608,134217728~12582912,150994944~16777216,184549376~12582912,200290234~1036358,205520896~8388608,215110864~28158768,247142031~321905,261553838~2687314,268435456~16777216,290354177~11635711,303838706~2345486,335499074~45246,339738624~12582912,356296674~219166,369098752~8388608,381681664~12582912,394881756~11965732] 2021-07-31T02:29:55.187 INFO:tasks.workunit.client.0.smithi045.stdout:from 8 to 12 diff [0~8388608,33554432~8388608,46137344~16777216,67108864~8388608,109051904~8388608,121634816~25165824,150994944~4194304,159383552~4194304,180355072~12582912,197132288~12582912,218103808~8388608,230686720~8388608,251658240~8388608,268435456~12582912,293601280~8388608,314572800~4194304,322961408~4194304,335544320~16777216,369098752~4194304,381681664~8388608,398458880~8388608] 2021-07-31T02:29:55.187 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~8388608,33554432~8388608,46137344~16777216,67108864~8388608,109051904~8388608,121634816~25165824,150994944~4194304,159383552~4194304,180355072~12582912,197132288~12582912,218103808~8388608,230686720~8388608,251658240~8388608,268435456~12582912,293601280~8388608,314572800~4194304,322961408~4194304,335544320~16777216,369098752~4194304,381681664~8388608,398458880~8388608] 2021-07-31T02:29:55.193 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~8388608,16439262~337954,19000239~1971281,33446354~8496686,46137344~16777216,67108864~12582912,81362982~2523098,92598663~3870329,109051904~8388608,121634816~25165824,150994944~16777216,180355072~33554432,215110864~28158768,247142031~321905,251658240~12582912,268435456~16777216,290354177~11635711,303838706~2345486,314572800~4194304,322961408~4194304,335499074~21016766,369098752~8388608,381681664~12582912,394881756~11965732] 2021-07-31T02:29:55.193 INFO:tasks.workunit.client.0.smithi045.stdout:from 8 to 13 diff [0~8388608,33554432~8388608,46137344~16777216,67108864~8388608,88080384~8388608,109051904~37748736,150994944~4194304,159383552~4194304,171966464~4194304,180355072~29360128,218103808~8388608,230686720~8388608,251658240~8388608,268435456~12582912,293601280~8388608,314572800~4194304,322961408~8388608,335544320~16777216,369098752~4194304,381681664~12582912,398458880~8388608] 2021-07-31T02:29:55.193 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~8388608,33554432~8388608,46137344~16777216,67108864~8388608,88080384~8388608,109051904~37748736,150994944~4194304,159383552~4194304,171966464~4194304,180355072~29360128,218103808~8388608,230686720~8388608,251658240~8388608,268435456~12582912,293601280~8388608,314572800~4194304,322961408~8388608,335544320~16777216,369098752~4194304,381681664~12582912,398458880~8388608] 2021-07-31T02:29:55.199 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~8388608,16439262~337954,19000239~1971281,33446354~8496686,46137344~16777216,67108864~12582912,81362982~2523098,88080384~8388608,109051904~37748736,150994944~16777216,171966464~41943040,215110864~28158768,247142031~321905,251658240~12582912,268435456~16777216,290354177~11635711,303838706~2345486,314572800~4194304,322961408~33554432,369098752~8388608,381681664~25165824] 2021-07-31T02:29:55.199 INFO:tasks.workunit.client.0.smithi045.stdout:from 8 to 14 diff [0~8388608,16777216~4194304,33554432~8388608,46137344~16777216,67108864~12582912,83886080~12582912,104857600~41943040,150994944~4194304,159383552~4194304,171966464~4194304,180355072~29360128,218103808~8388608,230686720~12582912,251658240~8388608,268435456~16777216,293601280~8388608,314572800~4194304,322961408~8388608,335544320~20971520,369098752~4194304,381681664~12582912,398458880~8388608] 2021-07-31T02:29:55.199 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~8388608,16777216~4194304,33554432~8388608,46137344~16777216,67108864~12582912,83886080~12582912,104857600~41943040,150994944~4194304,159383552~4194304,171966464~4194304,180355072~29360128,218103808~8388608,230686720~12582912,251658240~8388608,268435456~16777216,293601280~8388608,314572800~4194304,322961408~8388608,335544320~20971520,369098752~4194304,381681664~12582912,398458880~8388608] 2021-07-31T02:29:55.205 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~8388608,16439262~8726562,33446354~8496686,46137344~16777216,67108864~12582912,81362982~15106010,104857600~41943040,150994944~16777216,171966464~41943040,215110864~28158768,247142031~321905,251658240~12582912,268435456~16777216,290354177~11635711,303838706~2345486,314572800~4194304,322961408~33554432,369098752~8388608,381681664~25165824] 2021-07-31T02:29:55.205 INFO:tasks.workunit.client.0.smithi045.stdout:from 8 to 15 diff [0~8388608,16777216~4194304,33554432~8388608,46137344~16777216,67108864~29360128,104857600~41943040,150994944~4194304,159383552~4194304,167772160~8388608,180355072~29360128,218103808~8388608,230686720~12582912,251658240~8388608,268435456~16777216,293601280~12582912,314572800~4194304,322961408~8388608,335544320~20971520,369098752~8388608,381681664~12582912,398458880~16777216] 2021-07-31T02:29:55.206 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~8388608,16777216~4194304,33554432~8388608,46137344~16777216,67108864~29360128,104857600~41943040,150994944~4194304,159383552~4194304,167772160~8388608,180355072~29360128,218103808~8388608,230686720~12582912,251658240~8388608,268435456~16777216,293601280~12582912,314572800~4194304,322961408~8388608,335544320~20971520,369098752~8388608,381681664~12582912,398458880~16777216] 2021-07-31T02:29:55.211 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~8388608,16439262~8726562,33446354~8496686,46137344~16777216,67108864~29360128,104857600~41943040,150994944~62914560,215110864~28158768,247142031~321905,251658240~12582912,268435456~16777216,290354177~15830015,314572800~4194304,322961408~33554432,369098752~46137344] 2021-07-31T02:29:55.211 INFO:tasks.workunit.client.0.smithi045.stdout:from 8 to 16 diff [0~12582912,16777216~4194304,29360128~12582912,46137344~16777216,67108864~29360128,104857600~41943040,150994944~12582912,167772160~8388608,180355072~29360128,218103808~8388608,230686720~12582912,251658240~8388608,268435456~16777216,293601280~12582912,314572800~4194304,322961408~8388608,335544320~41943040,381681664~12582912,398458880~16777216] 2021-07-31T02:29:55.212 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~12582912,16777216~4194304,29360128~12582912,46137344~16777216,67108864~29360128,104857600~41943040,150994944~12582912,167772160~8388608,180355072~29360128,218103808~8388608,230686720~12582912,251658240~8388608,268435456~16777216,293601280~12582912,314572800~4194304,322961408~8388608,335544320~41943040,381681664~12582912,398458880~16777216] 2021-07-31T02:29:55.217 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~12582912,16439262~8726562,29360128~12582912,46137344~16777216,67108864~29360128,104857600~41943040,150994944~62914560,215110864~28158768,247142031~321905,251658240~12582912,268435456~16777216,290354177~15830015,314572800~4194304,322961408~92274688] 2021-07-31T02:29:55.217 INFO:tasks.workunit.client.0.smithi045.stdout:from 8 to 17 diff [0~12582912,16777216~4194304,29360128~12582912,46137344~16777216,67108864~29360128,104857600~41943040,150994944~12582912,167772160~41943040,218103808~8388608,230686720~16777216,251658240~8388608,264241152~20971520,293601280~12582912,314572800~4194304,322961408~8388608,335544320~41943040,381681664~12582912,398458880~16777216] 2021-07-31T02:29:55.218 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~12582912,16777216~4194304,29360128~12582912,46137344~16777216,67108864~29360128,104857600~41943040,150994944~12582912,167772160~41943040,218103808~8388608,230686720~16777216,251658240~8388608,264241152~20971520,293601280~12582912,314572800~4194304,322961408~8388608,335544320~41943040,381681664~12582912,398458880~16777216] 2021-07-31T02:29:55.223 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~12582912,16439262~8726562,29360128~12582912,46137344~16777216,67108864~29360128,104857600~41943040,150994944~62914560,215110864~32353072,251658240~33554432,290354177~15830015,314572800~4194304,322961408~92274688] 2021-07-31T02:29:55.224 INFO:tasks.workunit.client.0.smithi045.stdout:from 8 to 18 diff [0~12582912,16777216~4194304,29360128~12582912,46137344~16777216,67108864~79691776,150994944~12582912,167772160~41943040,218103808~8388608,230686720~16777216,251658240~8388608,264241152~20971520,293601280~12582912,314572800~4194304,322961408~8388608,335544320~41943040,381681664~12582912,398458880~16777216] 2021-07-31T02:29:55.224 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~12582912,16777216~4194304,29360128~12582912,46137344~16777216,67108864~79691776,150994944~12582912,167772160~41943040,218103808~8388608,230686720~16777216,251658240~8388608,264241152~20971520,293601280~12582912,314572800~4194304,322961408~8388608,335544320~41943040,381681664~12582912,398458880~16777216] 2021-07-31T02:29:55.230 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~12582912,16439262~8726562,29360128~12582912,46137344~16777216,67108864~79691776,150994944~62914560,215110864~32353072,251658240~33554432,290354177~15830015,314572800~4194304,322961408~92274688] 2021-07-31T02:29:55.230 INFO:tasks.workunit.client.0.smithi045.stdout:from 8 to 19 diff [0~12582912,16777216~4194304,29360128~12582912,46137344~100663296,150994944~12582912,167772160~41943040,218103808~8388608,230686720~16777216,251658240~8388608,264241152~20971520,293601280~20971520,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:29:55.230 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~12582912,16777216~4194304,29360128~12582912,46137344~100663296,150994944~12582912,167772160~41943040,218103808~8388608,230686720~16777216,251658240~8388608,264241152~20971520,293601280~20971520,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:29:55.236 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~12582912,16439262~8726562,29360128~12582912,46137344~100663296,150994944~62914560,215110864~32353072,251658240~33554432,290354177~28412927,322961408~92274688] 2021-07-31T02:29:55.237 INFO:tasks.workunit.client.0.smithi045.stdout:from 9 to 10 diff [54525952~8388608,71303168~8388608,121634816~4194304,138412032~8388608,150994944~4194304,230686720~8388608,268435456~12582912,339738624~8388608,369098752~4194304] 2021-07-31T02:29:55.237 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [54525952~8388608,71303168~8388608,121634816~4194304,138412032~8388608,150994944~4194304,230686720~8388608,268435456~12582912,339738624~8388608,369098752~4194304] 2021-07-31T02:29:55.242 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [16439262~337954,19000239~1971281,33446354~108078,54525952~8388608,71303168~8388608,81362982~2523098,92598663~3870329,113251019~4189493,121634816~8388608,138412032~8388608,150994944~8388608,160749787~2828069,186356067~2387613,200290234~1036358,215110864~2992944,230686720~12582912,247142031~321905,261553838~2687314,268435456~16777216,290354177~3247103,303838706~2345486,335499074~45246,339738624~12582912,356296674~219166,369098752~8388608,394881756~3577124,400514459~2138725] 2021-07-31T02:29:55.243 INFO:tasks.workunit.client.0.smithi045.stdout:from 9 to 11 diff [33554432~8388608,54525952~8388608,67108864~8388608,109051904~8388608,121634816~4194304,138412032~8388608,150994944~4194304,159383552~4194304,184549376~4194304,218103808~8388608,230686720~8388608,268435456~12582912,293601280~4194304,339738624~8388608,369098752~4194304,398458880~8388608] 2021-07-31T02:29:55.243 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [33554432~8388608,54525952~8388608,67108864~8388608,109051904~8388608,121634816~4194304,138412032~8388608,150994944~4194304,159383552~4194304,184549376~4194304,218103808~8388608,230686720~8388608,268435456~12582912,293601280~4194304,339738624~8388608,369098752~4194304,398458880~8388608] 2021-07-31T02:29:55.248 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [16439262~337954,19000239~1971281,33446354~8496686,54525952~8388608,67108864~12582912,81362982~2523098,92598663~3870329,109051904~8388608,121634816~8388608,138412032~8388608,150994944~16777216,184549376~4194304,200290234~1036358,215110864~28158768,247142031~321905,261553838~2687314,268435456~16777216,290354177~7441407,303838706~2345486,335499074~45246,339738624~12582912,356296674~219166,369098752~8388608,394881756~11965732] 2021-07-31T02:29:55.249 INFO:tasks.workunit.client.0.smithi045.stdout:from 9 to 12 diff [33554432~8388608,54525952~8388608,67108864~8388608,109051904~8388608,121634816~25165824,150994944~4194304,159383552~4194304,180355072~8388608,197132288~8388608,218103808~8388608,230686720~8388608,251658240~8388608,268435456~12582912,293601280~4194304,314572800~4194304,322961408~4194304,335544320~16777216,369098752~4194304,398458880~8388608] 2021-07-31T02:29:55.249 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [33554432~8388608,54525952~8388608,67108864~8388608,109051904~8388608,121634816~25165824,150994944~4194304,159383552~4194304,180355072~8388608,197132288~8388608,218103808~8388608,230686720~8388608,251658240~8388608,268435456~12582912,293601280~4194304,314572800~4194304,322961408~4194304,335544320~16777216,369098752~4194304,398458880~8388608] 2021-07-31T02:29:55.255 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [16439262~337954,19000239~1971281,33446354~8496686,54525952~8388608,67108864~12582912,81362982~2523098,92598663~3870329,109051904~8388608,121634816~25165824,150994944~16777216,180355072~12582912,197132288~12582912,215110864~28158768,247142031~321905,251658240~12582912,268435456~16777216,290354177~7441407,303838706~2345486,314572800~4194304,322961408~4194304,335499074~21016766,369098752~8388608,394881756~11965732] 2021-07-31T02:29:55.255 INFO:tasks.workunit.client.0.smithi045.stdout:from 9 to 13 diff [33554432~8388608,54525952~8388608,67108864~8388608,88080384~8388608,109051904~37748736,150994944~4194304,159383552~4194304,171966464~4194304,180355072~8388608,192937984~12582912,218103808~8388608,230686720~8388608,251658240~8388608,268435456~12582912,293601280~4194304,314572800~4194304,322961408~8388608,335544320~16777216,369098752~4194304,390070272~4194304,398458880~8388608] 2021-07-31T02:29:55.255 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [33554432~8388608,54525952~8388608,67108864~8388608,88080384~8388608,109051904~37748736,150994944~4194304,159383552~4194304,171966464~4194304,180355072~8388608,192937984~12582912,218103808~8388608,230686720~8388608,251658240~8388608,268435456~12582912,293601280~4194304,314572800~4194304,322961408~8388608,335544320~16777216,369098752~4194304,390070272~4194304,398458880~8388608] 2021-07-31T02:29:55.261 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [16439262~337954,19000239~1971281,33446354~8496686,54525952~8388608,67108864~12582912,81362982~2523098,88080384~8388608,109051904~37748736,150994944~16777216,171966464~37748736,215110864~28158768,247142031~321905,251658240~12582912,268435456~16777216,290354177~7441407,303838706~2345486,314572800~4194304,322961408~33554432,369098752~8388608,390070272~16777216] 2021-07-31T02:29:55.261 INFO:tasks.workunit.client.0.smithi045.stdout:from 9 to 14 diff [16777216~4194304,33554432~8388608,54525952~8388608,67108864~12582912,83886080~12582912,104857600~41943040,150994944~4194304,159383552~4194304,171966464~4194304,180355072~25165824,218103808~8388608,230686720~12582912,251658240~8388608,268435456~16777216,293601280~4194304,314572800~4194304,322961408~8388608,335544320~20971520,369098752~4194304,390070272~4194304,398458880~8388608] 2021-07-31T02:29:55.262 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [16777216~4194304,33554432~8388608,54525952~8388608,67108864~12582912,83886080~12582912,104857600~41943040,150994944~4194304,159383552~4194304,171966464~4194304,180355072~25165824,218103808~8388608,230686720~12582912,251658240~8388608,268435456~16777216,293601280~4194304,314572800~4194304,322961408~8388608,335544320~20971520,369098752~4194304,390070272~4194304,398458880~8388608] 2021-07-31T02:29:55.267 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [16439262~8726562,33446354~8496686,54525952~8388608,67108864~12582912,81362982~15106010,104857600~41943040,150994944~16777216,171966464~37748736,215110864~28158768,247142031~321905,251658240~12582912,268435456~16777216,290354177~7441407,303838706~2345486,314572800~4194304,322961408~33554432,369098752~8388608,390070272~16777216] 2021-07-31T02:29:55.267 INFO:tasks.workunit.client.0.smithi045.stdout:from 9 to 15 diff [16777216~4194304,33554432~8388608,50331648~12582912,67108864~29360128,104857600~41943040,150994944~4194304,159383552~4194304,167772160~8388608,180355072~25165824,218103808~8388608,230686720~12582912,251658240~8388608,268435456~16777216,293601280~12582912,314572800~4194304,322961408~8388608,335544320~20971520,369098752~8388608,390070272~4194304,398458880~16777216] 2021-07-31T02:29:55.267 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [16777216~4194304,33554432~8388608,50331648~12582912,67108864~29360128,104857600~41943040,150994944~4194304,159383552~4194304,167772160~8388608,180355072~25165824,218103808~8388608,230686720~12582912,251658240~8388608,268435456~16777216,293601280~12582912,314572800~4194304,322961408~8388608,335544320~20971520,369098752~8388608,390070272~4194304,398458880~16777216] 2021-07-31T02:29:55.273 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [16439262~8726562,33446354~8496686,50331648~12582912,67108864~29360128,104857600~41943040,150994944~58720256,215110864~28158768,247142031~321905,251658240~12582912,268435456~16777216,290354177~15830015,314572800~4194304,322961408~33554432,369098752~12582912,390070272~25165824] 2021-07-31T02:29:55.274 INFO:tasks.workunit.client.0.smithi045.stdout:from 9 to 16 diff [4194304~4194304,16777216~4194304,29360128~12582912,50331648~8388608,67108864~29360128,104857600~41943040,150994944~12582912,167772160~8388608,180355072~25165824,218103808~8388608,230686720~12582912,251658240~8388608,268435456~16777216,293601280~12582912,314572800~4194304,322961408~8388608,335544320~41943040,390070272~4194304,398458880~16777216] 2021-07-31T02:29:55.274 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~4194304,16777216~4194304,29360128~12582912,50331648~8388608,67108864~29360128,104857600~41943040,150994944~12582912,167772160~8388608,180355072~25165824,218103808~8388608,230686720~12582912,251658240~8388608,268435456~16777216,293601280~12582912,314572800~4194304,322961408~8388608,335544320~41943040,390070272~4194304,398458880~16777216] 2021-07-31T02:29:55.280 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~8388608,16439262~8726562,29360128~12582912,50331648~12582912,67108864~29360128,104857600~41943040,150994944~58720256,215110864~28158768,247142031~321905,251658240~12582912,268435456~16777216,290354177~15830015,314572800~4194304,322961408~58720256,390070272~25165824] 2021-07-31T02:29:55.280 INFO:tasks.workunit.client.0.smithi045.stdout:from 9 to 17 diff [4194304~8388608,16777216~4194304,29360128~12582912,50331648~8388608,67108864~29360128,104857600~41943040,150994944~12582912,167772160~37748736,218103808~8388608,230686720~16777216,251658240~8388608,264241152~20971520,293601280~12582912,314572800~4194304,322961408~8388608,335544320~41943040,390070272~4194304,398458880~16777216] 2021-07-31T02:29:55.280 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~8388608,16777216~4194304,29360128~12582912,50331648~8388608,67108864~29360128,104857600~41943040,150994944~12582912,167772160~37748736,218103808~8388608,230686720~16777216,251658240~8388608,264241152~20971520,293601280~12582912,314572800~4194304,322961408~8388608,335544320~41943040,390070272~4194304,398458880~16777216] 2021-07-31T02:29:55.286 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~8388608,16439262~8726562,29360128~12582912,50331648~12582912,67108864~29360128,104857600~41943040,150994944~58720256,215110864~32353072,251658240~33554432,290354177~15830015,314572800~4194304,322961408~58720256,390070272~25165824] 2021-07-31T02:29:55.286 INFO:tasks.workunit.client.0.smithi045.stdout:from 9 to 18 diff [4194304~8388608,16777216~4194304,29360128~12582912,46137344~12582912,67108864~79691776,150994944~12582912,167772160~37748736,218103808~8388608,230686720~16777216,251658240~8388608,264241152~20971520,293601280~12582912,314572800~4194304,322961408~8388608,335544320~41943040,390070272~4194304,398458880~16777216] 2021-07-31T02:29:55.286 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~8388608,16777216~4194304,29360128~12582912,46137344~12582912,67108864~79691776,150994944~12582912,167772160~37748736,218103808~8388608,230686720~16777216,251658240~8388608,264241152~20971520,293601280~12582912,314572800~4194304,322961408~8388608,335544320~41943040,390070272~4194304,398458880~16777216] 2021-07-31T02:29:55.291 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~8388608,16439262~8726562,29360128~12582912,46137344~16777216,67108864~79691776,150994944~58720256,215110864~32353072,251658240~33554432,290354177~15830015,314572800~4194304,322961408~58720256,390070272~25165824] 2021-07-31T02:29:55.292 INFO:tasks.workunit.client.0.smithi045.stdout:from 9 to 19 diff [4194304~8388608,16777216~4194304,29360128~12582912,46137344~12582912,62914560~83886080,150994944~12582912,167772160~37748736,218103808~8388608,230686720~16777216,251658240~8388608,264241152~20971520,293601280~20971520,322961408~54525952,390070272~4194304,398458880~16777216] 2021-07-31T02:29:55.292 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~8388608,16777216~4194304,29360128~12582912,46137344~12582912,62914560~83886080,150994944~12582912,167772160~37748736,218103808~8388608,230686720~16777216,251658240~8388608,264241152~20971520,293601280~20971520,322961408~54525952,390070272~4194304,398458880~16777216] 2021-07-31T02:29:55.298 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~8388608,16439262~8726562,29360128~12582912,46137344~100663296,150994944~58720256,215110864~32353072,251658240~33554432,290354177~28412927,322961408~58720256,390070272~25165824] 2021-07-31T02:29:55.298 INFO:tasks.workunit.client.0.smithi045.stdout:from 10 to 11 diff [33554432~8388608,67108864~8388608,109051904~8388608,159383552~4194304,184549376~4194304,218103808~8388608,281018368~4194304,293601280~4194304,343932928~4194304,398458880~8388608] 2021-07-31T02:29:55.298 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [33554432~8388608,67108864~8388608,109051904~8388608,159383552~4194304,184549376~4194304,218103808~8388608,281018368~4194304,293601280~4194304,343932928~4194304,398458880~8388608] 2021-07-31T02:29:55.304 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [16439262~337954,19000239~1971281,33446354~8496686,62262910~651650,67108864~8388608,81362982~2523098,92598663~3870329,109051904~8388608,146755873~44767,159383552~8388608,184549376~4194304,200290234~1036358,215110864~15575856,247142031~321905,261553838~2687314,281018368~4194304,290354177~7441407,303838706~2345486,335499074~45246,343932928~8388608,356296674~219166,394881756~11965732] 2021-07-31T02:29:55.304 INFO:tasks.workunit.client.0.smithi045.stdout:from 10 to 12 diff [33554432~8388608,67108864~8388608,109051904~8388608,125829120~12582912,159383552~4194304,180355072~8388608,197132288~8388608,218103808~8388608,251658240~8388608,281018368~4194304,293601280~4194304,314572800~4194304,322961408~4194304,335544320~4194304,343932928~8388608,398458880~8388608] 2021-07-31T02:29:55.304 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [33554432~8388608,67108864~8388608,109051904~8388608,125829120~12582912,159383552~4194304,180355072~8388608,197132288~8388608,218103808~8388608,251658240~8388608,281018368~4194304,293601280~4194304,314572800~4194304,322961408~4194304,335544320~4194304,343932928~8388608,398458880~8388608] 2021-07-31T02:29:55.310 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [16439262~337954,19000239~1971281,33446354~8496686,62262910~651650,67108864~8388608,81362982~2523098,92598663~3870329,109051904~8388608,125829120~12582912,146755873~44767,159383552~8388608,180355072~12582912,197132288~12582912,215110864~15575856,247142031~321905,251658240~12582912,281018368~4194304,290354177~7441407,303838706~2345486,314572800~4194304,322961408~4194304,335499074~21016766,394881756~11965732] 2021-07-31T02:29:55.311 INFO:tasks.workunit.client.0.smithi045.stdout:from 10 to 13 diff [33554432~8388608,67108864~8388608,88080384~8388608,109051904~12582912,125829120~16777216,159383552~4194304,171966464~4194304,180355072~8388608,192937984~12582912,218103808~8388608,251658240~8388608,281018368~4194304,293601280~4194304,314572800~4194304,322961408~8388608,335544320~4194304,343932928~8388608,390070272~4194304,398458880~8388608] 2021-07-31T02:29:55.311 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [33554432~8388608,67108864~8388608,88080384~8388608,109051904~12582912,125829120~16777216,159383552~4194304,171966464~4194304,180355072~8388608,192937984~12582912,218103808~8388608,251658240~8388608,281018368~4194304,293601280~4194304,314572800~4194304,322961408~8388608,335544320~4194304,343932928~8388608,390070272~4194304,398458880~8388608] 2021-07-31T02:29:55.316 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [16439262~337954,19000239~1971281,33446354~8496686,62262910~651650,67108864~8388608,81362982~2523098,88080384~8388608,109051904~12582912,125829120~20971520,159383552~8388608,171966464~37748736,215110864~15575856,247142031~321905,251658240~12582912,281018368~4194304,290354177~7441407,303838706~2345486,314572800~4194304,322961408~33554432,390070272~16777216] 2021-07-31T02:29:55.316 INFO:tasks.workunit.client.0.smithi045.stdout:from 10 to 14 diff [16777216~4194304,33554432~8388608,67108864~8388608,83886080~12582912,104857600~16777216,125829120~16777216,159383552~4194304,171966464~4194304,180355072~25165824,218103808~8388608,230686720~8388608,251658240~8388608,281018368~4194304,293601280~4194304,314572800~4194304,322961408~8388608,335544320~4194304,343932928~12582912,390070272~4194304,398458880~8388608] 2021-07-31T02:29:55.316 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [16777216~4194304,33554432~8388608,67108864~8388608,83886080~12582912,104857600~16777216,125829120~16777216,159383552~4194304,171966464~4194304,180355072~25165824,218103808~8388608,230686720~8388608,251658240~8388608,281018368~4194304,293601280~4194304,314572800~4194304,322961408~8388608,335544320~4194304,343932928~12582912,390070272~4194304,398458880~8388608] 2021-07-31T02:29:55.323 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [16439262~8726562,33446354~8496686,62262910~651650,67108864~8388608,81362982~15106010,104857600~16777216,125829120~20971520,159383552~8388608,171966464~37748736,215110864~23964464,247142031~321905,251658240~12582912,281018368~4194304,290354177~7441407,303838706~2345486,314572800~4194304,322961408~33554432,390070272~16777216] 2021-07-31T02:29:55.324 INFO:tasks.workunit.client.0.smithi045.stdout:from 10 to 15 diff [16777216~4194304,33554432~8388608,50331648~4194304,67108864~8388608,79691776~16777216,104857600~16777216,125829120~16777216,159383552~4194304,167772160~8388608,180355072~25165824,218103808~8388608,230686720~8388608,251658240~8388608,281018368~4194304,293601280~12582912,314572800~4194304,322961408~8388608,335544320~20971520,373293056~4194304,390070272~4194304,398458880~16777216] 2021-07-31T02:29:55.324 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [16777216~4194304,33554432~8388608,50331648~4194304,67108864~8388608,79691776~16777216,104857600~16777216,125829120~16777216,159383552~4194304,167772160~8388608,180355072~25165824,218103808~8388608,230686720~8388608,251658240~8388608,281018368~4194304,293601280~12582912,314572800~4194304,322961408~8388608,335544320~20971520,373293056~4194304,390070272~4194304,398458880~16777216] 2021-07-31T02:29:55.329 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [16439262~8726562,33446354~8496686,50331648~4194304,62262910~651650,67108864~8388608,79691776~16777216,104857600~16777216,125829120~20971520,159383552~50331648,215110864~23964464,247142031~321905,251658240~12582912,281018368~4194304,290354177~15830015,314572800~4194304,322961408~33554432,373293056~8388608,390070272~25165824] 2021-07-31T02:29:55.330 INFO:tasks.workunit.client.0.smithi045.stdout:from 10 to 16 diff [4194304~4194304,16777216~4194304,29360128~12582912,50331648~8388608,67108864~8388608,79691776~16777216,104857600~16777216,125829120~16777216,155189248~8388608,167772160~8388608,180355072~25165824,218103808~8388608,230686720~8388608,251658240~8388608,281018368~4194304,293601280~12582912,314572800~4194304,322961408~8388608,335544320~33554432,373293056~4194304,390070272~4194304,398458880~16777216] 2021-07-31T02:29:55.330 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~4194304,16777216~4194304,29360128~12582912,50331648~8388608,67108864~8388608,79691776~16777216,104857600~16777216,125829120~16777216,155189248~8388608,167772160~8388608,180355072~25165824,218103808~8388608,230686720~8388608,251658240~8388608,281018368~4194304,293601280~12582912,314572800~4194304,322961408~8388608,335544320~33554432,373293056~4194304,390070272~4194304,398458880~16777216] 2021-07-31T02:29:55.336 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~8388608,16439262~8726562,29360128~12582912,50331648~8388608,62262910~651650,67108864~29360128,104857600~16777216,125829120~20971520,155189248~54525952,215110864~23964464,247142031~321905,251658240~12582912,281018368~4194304,290354177~15830015,314572800~4194304,322961408~58720256,390070272~25165824] 2021-07-31T02:29:55.336 INFO:tasks.workunit.client.0.smithi045.stdout:from 10 to 17 diff [4194304~8388608,16777216~4194304,29360128~12582912,50331648~8388608,67108864~8388608,79691776~16777216,104857600~16777216,125829120~16777216,155189248~8388608,167772160~37748736,218103808~8388608,230686720~12582912,251658240~8388608,264241152~8388608,281018368~4194304,293601280~12582912,314572800~4194304,322961408~8388608,335544320~33554432,373293056~4194304,390070272~4194304,398458880~16777216] 2021-07-31T02:29:55.337 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~8388608,16777216~4194304,29360128~12582912,50331648~8388608,67108864~8388608,79691776~16777216,104857600~16777216,125829120~16777216,155189248~8388608,167772160~37748736,218103808~8388608,230686720~12582912,251658240~8388608,264241152~8388608,281018368~4194304,293601280~12582912,314572800~4194304,322961408~8388608,335544320~33554432,373293056~4194304,390070272~4194304,398458880~16777216] 2021-07-31T02:29:55.342 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~8388608,16439262~8726562,29360128~12582912,50331648~8388608,62262910~651650,67108864~29360128,104857600~16777216,125829120~20971520,155189248~54525952,215110864~32353072,251658240~20971520,281018368~4194304,290354177~15830015,314572800~4194304,322961408~58720256,390070272~25165824] 2021-07-31T02:29:55.342 INFO:tasks.workunit.client.0.smithi045.stdout:from 10 to 18 diff [4194304~8388608,16777216~4194304,29360128~12582912,46137344~12582912,67108864~8388608,79691776~41943040,125829120~16777216,150994944~12582912,167772160~37748736,218103808~8388608,230686720~12582912,251658240~8388608,264241152~8388608,281018368~4194304,293601280~12582912,314572800~4194304,322961408~8388608,335544320~33554432,373293056~4194304,390070272~4194304,398458880~16777216] 2021-07-31T02:29:55.342 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~8388608,16777216~4194304,29360128~12582912,46137344~12582912,67108864~8388608,79691776~41943040,125829120~16777216,150994944~12582912,167772160~37748736,218103808~8388608,230686720~12582912,251658240~8388608,264241152~8388608,281018368~4194304,293601280~12582912,314572800~4194304,322961408~8388608,335544320~33554432,373293056~4194304,390070272~4194304,398458880~16777216] 2021-07-31T02:29:55.348 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~8388608,16439262~8726562,29360128~12582912,46137344~12582912,62262910~651650,67108864~54525952,125829120~20971520,150994944~58720256,215110864~32353072,251658240~20971520,281018368~4194304,290354177~15830015,314572800~4194304,322961408~58720256,390070272~25165824] 2021-07-31T02:29:55.348 INFO:tasks.workunit.client.0.smithi045.stdout:from 10 to 19 diff [4194304~8388608,16777216~4194304,29360128~12582912,46137344~12582912,62914560~12582912,79691776~41943040,125829120~16777216,150994944~12582912,167772160~37748736,218103808~8388608,230686720~12582912,251658240~8388608,264241152~8388608,281018368~4194304,293601280~20971520,322961408~46137344,373293056~4194304,390070272~4194304,398458880~16777216] 2021-07-31T02:29:55.348 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~8388608,16777216~4194304,29360128~12582912,46137344~12582912,62914560~12582912,79691776~41943040,125829120~16777216,150994944~12582912,167772160~37748736,218103808~8388608,230686720~12582912,251658240~8388608,264241152~8388608,281018368~4194304,293601280~20971520,322961408~46137344,373293056~4194304,390070272~4194304,398458880~16777216] 2021-07-31T02:29:55.354 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~8388608,16439262~8726562,29360128~12582912,46137344~12582912,62262910~59371906,125829120~20971520,150994944~58720256,215110864~32353072,251658240~20971520,281018368~4194304,290354177~28412927,322961408~58720256,390070272~25165824] 2021-07-31T02:29:55.354 INFO:tasks.workunit.client.0.smithi045.stdout:from 11 to 12 diff [125829120~12582912,180355072~8388608,197132288~8388608,251658240~8388608,314572800~4194304,322961408~4194304,335544320~4194304,348127232~4194304] 2021-07-31T02:29:55.355 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [125829120~12582912,180355072~8388608,197132288~8388608,251658240~8388608,314572800~4194304,322961408~4194304,335544320~4194304,348127232~4194304] 2021-07-31T02:29:55.360 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [16439262~337954,19000239~1971281,33446354~108078,62262910~651650,81362982~2523098,92598663~3870329,115357951~2082561,125829120~12582912,146755873~44767,164361273~3410887,180355072~12582912,197132288~12582912,215110864~2992944,247142031~321905,251658240~12582912,290354177~3247103,303838706~2345486,314572800~4194304,322961408~4194304,335499074~8433854,348127232~8388608,394881756~3577124,405071600~1775888] 2021-07-31T02:29:55.360 INFO:tasks.workunit.client.0.smithi045.stdout:from 11 to 13 diff [88080384~8388608,117440512~4194304,125829120~16777216,171966464~4194304,180355072~8388608,192937984~12582912,251658240~8388608,314572800~4194304,322961408~8388608,335544320~4194304,348127232~4194304,390070272~4194304] 2021-07-31T02:29:55.361 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [88080384~8388608,117440512~4194304,125829120~16777216,171966464~4194304,180355072~8388608,192937984~12582912,251658240~8388608,314572800~4194304,322961408~8388608,335544320~4194304,348127232~4194304,390070272~4194304] 2021-07-31T02:29:55.367 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [16439262~337954,19000239~1971281,33446354~108078,62262910~651650,81362982~2523098,88080384~8388608,115357951~6276865,125829120~20971520,164361273~3410887,171966464~37748736,215110864~2992944,247142031~321905,251658240~12582912,290354177~3247103,303838706~2345486,314572800~4194304,322961408~20971520,348127232~8388608,390070272~8388608,405071600~1775888] 2021-07-31T02:29:55.367 INFO:tasks.workunit.client.0.smithi045.stdout:from 11 to 14 diff [16777216~4194304,67108864~8388608,83886080~12582912,104857600~4194304,117440512~4194304,125829120~16777216,171966464~4194304,180355072~25165824,230686720~8388608,251658240~8388608,281018368~4194304,314572800~4194304,322961408~8388608,335544320~4194304,348127232~8388608,390070272~4194304] 2021-07-31T02:29:55.368 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [16777216~4194304,67108864~8388608,83886080~12582912,104857600~4194304,117440512~4194304,125829120~16777216,171966464~4194304,180355072~25165824,230686720~8388608,251658240~8388608,281018368~4194304,314572800~4194304,322961408~8388608,335544320~4194304,348127232~8388608,390070272~4194304] 2021-07-31T02:29:55.371 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [16439262~8726562,33446354~108078,62262910~651650,67108864~8388608,81362982~15106010,104857600~4194304,115357951~6276865,125829120~20971520,164361273~3410887,171966464~37748736,215110864~2992944,230686720~8388608,247142031~321905,251658240~12582912,281018368~4194304,290354177~3247103,303838706~2345486,314572800~4194304,322961408~20971520,348127232~8388608,390070272~8388608,405071600~1775888] 2021-07-31T02:29:55.372 INFO:tasks.workunit.client.0.smithi045.stdout:from 11 to 15 diff [16777216~4194304,50331648~4194304,67108864~8388608,79691776~16777216,104857600~4194304,117440512~4194304,125829120~16777216,167772160~8388608,180355072~25165824,230686720~8388608,251658240~8388608,281018368~4194304,297795584~8388608,314572800~4194304,322961408~8388608,335544320~8388608,348127232~8388608,373293056~4194304,390070272~4194304,402653184~12582912] 2021-07-31T02:29:55.372 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [16777216~4194304,50331648~4194304,67108864~8388608,79691776~16777216,104857600~4194304,117440512~4194304,125829120~16777216,167772160~8388608,180355072~25165824,230686720~8388608,251658240~8388608,281018368~4194304,297795584~8388608,314572800~4194304,322961408~8388608,335544320~8388608,348127232~8388608,373293056~4194304,390070272~4194304,402653184~12582912] 2021-07-31T02:29:55.377 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [16439262~8726562,33446354~108078,50331648~4194304,62262910~651650,67108864~8388608,79691776~16777216,104857600~4194304,115357951~6276865,125829120~20971520,164361273~45353927,215110864~2992944,230686720~8388608,247142031~321905,251658240~12582912,281018368~4194304,290354177~3247103,297795584~8388608,314572800~4194304,322961408~20971520,348127232~8388608,373293056~8388608,390070272~8388608,402653184~12582912] 2021-07-31T02:29:55.378 INFO:tasks.workunit.client.0.smithi045.stdout:from 11 to 16 diff [4194304~4194304,16777216~4194304,29360128~8388608,50331648~8388608,67108864~8388608,79691776~16777216,104857600~4194304,117440512~4194304,125829120~16777216,155189248~8388608,167772160~8388608,180355072~25165824,230686720~8388608,251658240~8388608,281018368~4194304,297795584~8388608,314572800~4194304,322961408~8388608,335544320~8388608,348127232~20971520,373293056~4194304,390070272~4194304,402653184~12582912] 2021-07-31T02:29:55.378 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~4194304,16777216~4194304,29360128~8388608,50331648~8388608,67108864~8388608,79691776~16777216,104857600~4194304,117440512~4194304,125829120~16777216,155189248~8388608,167772160~8388608,180355072~25165824,230686720~8388608,251658240~8388608,281018368~4194304,297795584~8388608,314572800~4194304,322961408~8388608,335544320~8388608,348127232~20971520,373293056~4194304,390070272~4194304,402653184~12582912] 2021-07-31T02:29:55.384 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~8388608,16439262~8726562,29360128~8388608,50331648~8388608,62262910~651650,67108864~29360128,104857600~4194304,115357951~6276865,125829120~20971520,155189248~54525952,215110864~2992944,230686720~8388608,247142031~321905,251658240~12582912,281018368~4194304,290354177~3247103,297795584~8388608,314572800~4194304,322961408~20971520,348127232~33554432,390070272~8388608,402653184~12582912] 2021-07-31T02:29:55.384 INFO:tasks.workunit.client.0.smithi045.stdout:from 11 to 17 diff [4194304~8388608,16777216~4194304,29360128~8388608,50331648~8388608,67108864~8388608,79691776~16777216,104857600~4194304,117440512~4194304,125829120~16777216,155189248~8388608,167772160~37748736,218103808~8388608,230686720~12582912,251658240~8388608,264241152~8388608,281018368~4194304,297795584~8388608,314572800~4194304,322961408~8388608,335544320~33554432,373293056~4194304,390070272~4194304,402653184~12582912] 2021-07-31T02:29:55.386 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~8388608,16777216~4194304,29360128~8388608,50331648~8388608,67108864~8388608,79691776~16777216,104857600~4194304,117440512~4194304,125829120~16777216,155189248~8388608,167772160~37748736,218103808~8388608,230686720~12582912,251658240~8388608,264241152~8388608,281018368~4194304,297795584~8388608,314572800~4194304,322961408~8388608,335544320~33554432,373293056~4194304,390070272~4194304,402653184~12582912] 2021-07-31T02:29:55.391 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~8388608,16439262~8726562,29360128~8388608,50331648~8388608,62262910~651650,67108864~29360128,104857600~4194304,115357951~6276865,125829120~20971520,155189248~54525952,215110864~32353072,251658240~20971520,281018368~4194304,290354177~3247103,297795584~8388608,314572800~4194304,322961408~58720256,390070272~8388608,402653184~12582912] 2021-07-31T02:29:55.392 INFO:tasks.workunit.client.0.smithi045.stdout:from 11 to 18 diff [4194304~8388608,16777216~4194304,29360128~8388608,46137344~12582912,67108864~8388608,79691776~29360128,113246208~8388608,125829120~16777216,150994944~12582912,167772160~37748736,218103808~8388608,230686720~12582912,251658240~8388608,264241152~8388608,281018368~4194304,297795584~8388608,314572800~4194304,322961408~8388608,335544320~33554432,373293056~4194304,390070272~4194304,402653184~12582912] 2021-07-31T02:29:55.392 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~8388608,16777216~4194304,29360128~8388608,46137344~12582912,67108864~8388608,79691776~29360128,113246208~8388608,125829120~16777216,150994944~12582912,167772160~37748736,218103808~8388608,230686720~12582912,251658240~8388608,264241152~8388608,281018368~4194304,297795584~8388608,314572800~4194304,322961408~8388608,335544320~33554432,373293056~4194304,390070272~4194304,402653184~12582912] 2021-07-31T02:29:55.398 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~8388608,16439262~8726562,29360128~8388608,46137344~12582912,62262910~651650,67108864~41943040,113246208~8388608,125829120~20971520,150994944~58720256,215110864~32353072,251658240~20971520,281018368~4194304,290354177~3247103,297795584~8388608,314572800~4194304,322961408~58720256,390070272~8388608,402653184~12582912] 2021-07-31T02:29:55.399 INFO:tasks.workunit.client.0.smithi045.stdout:from 11 to 19 diff [4194304~8388608,16777216~4194304,29360128~8388608,46137344~12582912,62914560~12582912,79691776~29360128,113246208~8388608,125829120~16777216,150994944~12582912,167772160~37748736,218103808~8388608,230686720~12582912,251658240~8388608,264241152~8388608,281018368~4194304,297795584~16777216,322961408~20971520,348127232~20971520,373293056~4194304,390070272~4194304,402653184~12582912] 2021-07-31T02:29:55.399 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~8388608,16777216~4194304,29360128~8388608,46137344~12582912,62914560~12582912,79691776~29360128,113246208~8388608,125829120~16777216,150994944~12582912,167772160~37748736,218103808~8388608,230686720~12582912,251658240~8388608,264241152~8388608,281018368~4194304,297795584~16777216,322961408~20971520,348127232~20971520,373293056~4194304,390070272~4194304,402653184~12582912] 2021-07-31T02:29:55.405 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~8388608,16439262~8726562,29360128~8388608,46137344~12582912,62262910~46788994,113246208~8388608,125829120~20971520,150994944~58720256,215110864~32353072,251658240~20971520,281018368~4194304,290354177~3247103,297795584~20971520,322961408~58720256,390070272~8388608,402653184~12582912] 2021-07-31T02:29:55.405 INFO:tasks.workunit.client.0.smithi045.stdout:from 12 to 13 diff [88080384~8388608,117440512~4194304,125829120~4194304,134217728~8388608,171966464~4194304,188743680~8388608,327155712~4194304,390070272~4194304] 2021-07-31T02:29:55.405 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [88080384~8388608,117440512~4194304,125829120~4194304,134217728~8388608,171966464~4194304,188743680~8388608,327155712~4194304,390070272~4194304] 2021-07-31T02:29:55.411 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [16439262~337954,19000239~1971281,33446354~108078,62262910~651650,81362982~2523098,88080384~8388608,115357951~6276865,125829120~20971520,164361273~3410887,171966464~8388608,188743680~12582912,215110864~2992944,247142031~321905,261553838~2687314,290354177~3247103,303838706~2345486,327155712~8388608,356296674~219166,390070272~8388608,405071600~1775888] 2021-07-31T02:29:55.411 INFO:tasks.workunit.client.0.smithi045.stdout:from 12 to 14 diff [16777216~4194304,67108864~8388608,83886080~12582912,104857600~4194304,117440512~4194304,125829120~4194304,134217728~8388608,171966464~4194304,184549376~20971520,230686720~8388608,281018368~4194304,327155712~4194304,348127232~8388608,390070272~4194304] 2021-07-31T02:29:55.411 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [16777216~4194304,67108864~8388608,83886080~12582912,104857600~4194304,117440512~4194304,125829120~4194304,134217728~8388608,171966464~4194304,184549376~20971520,230686720~8388608,281018368~4194304,327155712~4194304,348127232~8388608,390070272~4194304] 2021-07-31T02:29:55.417 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [16439262~8726562,33446354~108078,62262910~651650,67108864~8388608,81362982~15106010,104857600~4194304,115357951~6276865,125829120~20971520,164361273~3410887,171966464~8388608,184549376~25165824,215110864~2992944,230686720~8388608,247142031~321905,261553838~2687314,281018368~4194304,290354177~3247103,303838706~2345486,327155712~8388608,348127232~8388608,390070272~8388608,405071600~1775888] 2021-07-31T02:29:55.417 INFO:tasks.workunit.client.0.smithi045.stdout:from 12 to 15 diff [16777216~4194304,50331648~4194304,67108864~8388608,79691776~16777216,104857600~4194304,117440512~4194304,125829120~4194304,134217728~8388608,167772160~8388608,184549376~20971520,230686720~8388608,281018368~4194304,297795584~8388608,327155712~4194304,335544320~8388608,348127232~8388608,373293056~4194304,390070272~4194304,402653184~12582912] 2021-07-31T02:29:55.417 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [16777216~4194304,50331648~4194304,67108864~8388608,79691776~16777216,104857600~4194304,117440512~4194304,125829120~4194304,134217728~8388608,167772160~8388608,184549376~20971520,230686720~8388608,281018368~4194304,297795584~8388608,327155712~4194304,335544320~8388608,348127232~8388608,373293056~4194304,390070272~4194304,402653184~12582912] 2021-07-31T02:29:55.423 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [16439262~8726562,33446354~108078,50331648~4194304,62262910~651650,67108864~8388608,79691776~16777216,104857600~4194304,115357951~6276865,125829120~20971520,164361273~15993799,184549376~25165824,215110864~2992944,230686720~8388608,247142031~321905,261553838~2687314,281018368~4194304,290354177~3247103,297795584~8388608,327155712~16777216,348127232~8388608,373293056~8388608,390070272~8388608,402653184~12582912] 2021-07-31T02:29:55.423 INFO:tasks.workunit.client.0.smithi045.stdout:from 12 to 16 diff [4194304~4194304,16777216~4194304,29360128~8388608,50331648~8388608,67108864~8388608,79691776~16777216,104857600~4194304,117440512~4194304,125829120~16777216,155189248~8388608,167772160~8388608,184549376~20971520,230686720~8388608,281018368~4194304,297795584~8388608,327155712~4194304,335544320~8388608,348127232~20971520,373293056~4194304,390070272~4194304,402653184~12582912] 2021-07-31T02:29:55.424 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~4194304,16777216~4194304,29360128~8388608,50331648~8388608,67108864~8388608,79691776~16777216,104857600~4194304,117440512~4194304,125829120~16777216,155189248~8388608,167772160~8388608,184549376~20971520,230686720~8388608,281018368~4194304,297795584~8388608,327155712~4194304,335544320~8388608,348127232~20971520,373293056~4194304,390070272~4194304,402653184~12582912] 2021-07-31T02:29:55.430 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~8388608,16439262~8726562,29360128~8388608,50331648~8388608,62262910~651650,67108864~29360128,104857600~4194304,115357951~6276865,125829120~20971520,155189248~25165824,184549376~25165824,215110864~2992944,230686720~8388608,247142031~321905,261553838~2687314,281018368~4194304,290354177~3247103,297795584~8388608,327155712~16777216,348127232~33554432,390070272~8388608,402653184~12582912] 2021-07-31T02:29:55.430 INFO:tasks.workunit.client.0.smithi045.stdout:from 12 to 17 diff [4194304~8388608,16777216~4194304,29360128~8388608,50331648~8388608,67108864~8388608,79691776~16777216,104857600~4194304,117440512~4194304,125829120~16777216,155189248~8388608,167772160~37748736,218103808~8388608,230686720~12582912,264241152~8388608,281018368~4194304,297795584~8388608,327155712~4194304,335544320~33554432,373293056~4194304,390070272~4194304,402653184~12582912] 2021-07-31T02:29:55.430 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~8388608,16777216~4194304,29360128~8388608,50331648~8388608,67108864~8388608,79691776~16777216,104857600~4194304,117440512~4194304,125829120~16777216,155189248~8388608,167772160~37748736,218103808~8388608,230686720~12582912,264241152~8388608,281018368~4194304,297795584~8388608,327155712~4194304,335544320~33554432,373293056~4194304,390070272~4194304,402653184~12582912] 2021-07-31T02:29:55.435 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~8388608,16439262~8726562,29360128~8388608,50331648~8388608,62262910~651650,67108864~29360128,104857600~4194304,115357951~6276865,125829120~20971520,155189248~54525952,215110864~32353072,261553838~11075922,281018368~4194304,290354177~3247103,297795584~8388608,327155712~54525952,390070272~8388608,402653184~12582912] 2021-07-31T02:29:55.435 INFO:tasks.workunit.client.0.smithi045.stdout:from 12 to 18 diff [4194304~8388608,16777216~4194304,29360128~8388608,46137344~12582912,67108864~8388608,79691776~29360128,113246208~8388608,125829120~16777216,150994944~12582912,167772160~37748736,218103808~8388608,230686720~12582912,264241152~8388608,281018368~4194304,297795584~8388608,327155712~4194304,335544320~33554432,373293056~4194304,390070272~4194304,402653184~12582912] 2021-07-31T02:29:55.435 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~8388608,16777216~4194304,29360128~8388608,46137344~12582912,67108864~8388608,79691776~29360128,113246208~8388608,125829120~16777216,150994944~12582912,167772160~37748736,218103808~8388608,230686720~12582912,264241152~8388608,281018368~4194304,297795584~8388608,327155712~4194304,335544320~33554432,373293056~4194304,390070272~4194304,402653184~12582912] 2021-07-31T02:29:55.441 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~8388608,16439262~8726562,29360128~8388608,46137344~12582912,62262910~651650,67108864~41943040,113246208~8388608,125829120~20971520,150994944~58720256,215110864~32353072,261553838~11075922,281018368~4194304,290354177~3247103,297795584~8388608,327155712~54525952,390070272~8388608,402653184~12582912] 2021-07-31T02:29:55.441 INFO:tasks.workunit.client.0.smithi045.stdout:from 12 to 19 diff [4194304~8388608,16777216~4194304,29360128~8388608,46137344~12582912,62914560~12582912,79691776~29360128,113246208~8388608,125829120~16777216,150994944~12582912,167772160~37748736,218103808~8388608,230686720~12582912,264241152~8388608,281018368~4194304,297795584~16777216,327155712~16777216,348127232~20971520,373293056~4194304,390070272~4194304,402653184~12582912] 2021-07-31T02:29:55.441 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~8388608,16777216~4194304,29360128~8388608,46137344~12582912,62914560~12582912,79691776~29360128,113246208~8388608,125829120~16777216,150994944~12582912,167772160~37748736,218103808~8388608,230686720~12582912,264241152~8388608,281018368~4194304,297795584~16777216,327155712~16777216,348127232~20971520,373293056~4194304,390070272~4194304,402653184~12582912] 2021-07-31T02:29:55.448 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~8388608,16439262~8726562,29360128~8388608,46137344~12582912,62262910~46788994,113246208~8388608,125829120~20971520,150994944~58720256,215110864~32353072,261553838~11075922,281018368~4194304,290354177~3247103,297795584~20971520,327155712~54525952,390070272~8388608,402653184~12582912] 2021-07-31T02:29:55.448 INFO:tasks.workunit.client.0.smithi045.stdout:from 13 to 14 diff [16777216~4194304,67108864~8388608,83886080~4194304,104857600~4194304,184549376~20971520,230686720~8388608,281018368~4194304,348127232~8388608] 2021-07-31T02:29:55.448 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [16777216~4194304,67108864~8388608,83886080~4194304,104857600~4194304,184549376~20971520,230686720~8388608,281018368~4194304,348127232~8388608] 2021-07-31T02:29:55.453 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [16439262~8726562,33446354~108078,62262910~651650,67108864~8388608,81362982~6717402,94926588~1542404,104857600~4194304,115357951~2082561,121422973~211843,146755873~44767,164361273~3410887,184549376~25165824,215110864~2992944,230686720~8388608,247142031~321905,261553838~2687314,281018368~4194304,290354177~3247103,303838706~2345486,335499074~45246,348127232~8388608,395588806~2870074,405071600~1775888] 2021-07-31T02:29:55.454 INFO:tasks.workunit.client.0.smithi045.stdout:from 13 to 15 diff [16777216~4194304,50331648~4194304,67108864~8388608,79691776~8388608,104857600~4194304,138412032~4194304,167772160~8388608,184549376~20971520,230686720~8388608,281018368~4194304,297795584~8388608,335544320~8388608,348127232~8388608,373293056~4194304,402653184~12582912] 2021-07-31T02:29:55.454 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [16777216~4194304,50331648~4194304,67108864~8388608,79691776~8388608,104857600~4194304,138412032~4194304,167772160~8388608,184549376~20971520,230686720~8388608,281018368~4194304,297795584~8388608,335544320~8388608,348127232~8388608,373293056~4194304,402653184~12582912] 2021-07-31T02:29:55.460 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [16439262~8726562,33446354~108078,50331648~4194304,62262910~651650,67108864~8388608,79691776~8388608,94926588~1542404,104857600~4194304,115357951~2082561,121422973~211843,138412032~4194304,146755873~44767,164361273~15993799,184549376~25165824,215110864~2992944,230686720~8388608,247142031~321905,261553838~2687314,281018368~4194304,290354177~3247103,297795584~8388608,335499074~8433854,348127232~8388608,373293056~8388608,395588806~2870074,402653184~12582912] 2021-07-31T02:29:55.460 INFO:tasks.workunit.client.0.smithi045.stdout:from 13 to 16 diff [4194304~4194304,16777216~4194304,29360128~8388608,50331648~8388608,67108864~8388608,79691776~16777216,104857600~4194304,130023424~12582912,155189248~8388608,167772160~8388608,184549376~20971520,230686720~8388608,281018368~4194304,297795584~8388608,335544320~8388608,348127232~20971520,373293056~4194304,402653184~12582912] 2021-07-31T02:29:55.461 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~4194304,16777216~4194304,29360128~8388608,50331648~8388608,67108864~8388608,79691776~16777216,104857600~4194304,130023424~12582912,155189248~8388608,167772160~8388608,184549376~20971520,230686720~8388608,281018368~4194304,297795584~8388608,335544320~8388608,348127232~20971520,373293056~4194304,402653184~12582912] 2021-07-31T02:29:55.466 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~8388608,16439262~8726562,29360128~8388608,50331648~8388608,62262910~651650,67108864~29360128,104857600~4194304,115357951~2082561,121422973~211843,130023424~12582912,146755873~44767,155189248~25165824,184549376~25165824,215110864~2992944,230686720~8388608,247142031~321905,261553838~2687314,281018368~4194304,290354177~3247103,297795584~8388608,335499074~8433854,348127232~33554432,395588806~2870074,402653184~12582912] 2021-07-31T02:29:55.466 INFO:tasks.workunit.client.0.smithi045.stdout:from 13 to 17 diff [4194304~8388608,16777216~4194304,29360128~8388608,50331648~8388608,67108864~8388608,79691776~16777216,104857600~4194304,130023424~12582912,155189248~8388608,167772160~37748736,218103808~8388608,230686720~12582912,264241152~8388608,281018368~4194304,297795584~8388608,335544320~33554432,373293056~4194304,402653184~12582912] 2021-07-31T02:29:55.466 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~8388608,16777216~4194304,29360128~8388608,50331648~8388608,67108864~8388608,79691776~16777216,104857600~4194304,130023424~12582912,155189248~8388608,167772160~37748736,218103808~8388608,230686720~12582912,264241152~8388608,281018368~4194304,297795584~8388608,335544320~33554432,373293056~4194304,402653184~12582912] 2021-07-31T02:29:55.472 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~8388608,16439262~8726562,29360128~8388608,50331648~8388608,62262910~651650,67108864~29360128,104857600~4194304,115357951~2082561,121422973~211843,130023424~12582912,146755873~44767,155189248~54525952,215110864~32353072,261553838~11075922,281018368~4194304,290354177~3247103,297795584~8388608,335499074~46182590,395588806~2870074,402653184~12582912] 2021-07-31T02:29:55.472 INFO:tasks.workunit.client.0.smithi045.stdout:from 13 to 18 diff [4194304~8388608,16777216~4194304,29360128~8388608,46137344~12582912,67108864~8388608,79691776~29360128,113246208~8388608,130023424~12582912,150994944~12582912,167772160~37748736,218103808~8388608,230686720~12582912,264241152~8388608,281018368~4194304,297795584~8388608,335544320~33554432,373293056~4194304,402653184~12582912] 2021-07-31T02:29:55.473 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~8388608,16777216~4194304,29360128~8388608,46137344~12582912,67108864~8388608,79691776~29360128,113246208~8388608,130023424~12582912,150994944~12582912,167772160~37748736,218103808~8388608,230686720~12582912,264241152~8388608,281018368~4194304,297795584~8388608,335544320~33554432,373293056~4194304,402653184~12582912] 2021-07-31T02:29:55.479 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~8388608,16439262~8726562,29360128~8388608,46137344~12582912,62262910~651650,67108864~41943040,113246208~8388608,130023424~12582912,146755873~44767,150994944~58720256,215110864~32353072,261553838~11075922,281018368~4194304,290354177~3247103,297795584~8388608,335499074~46182590,395588806~2870074,402653184~12582912] 2021-07-31T02:29:55.479 INFO:tasks.workunit.client.0.smithi045.stdout:from 13 to 19 diff [4194304~8388608,16777216~4194304,29360128~8388608,46137344~12582912,62914560~12582912,79691776~29360128,113246208~8388608,130023424~12582912,150994944~12582912,167772160~37748736,218103808~8388608,230686720~12582912,264241152~8388608,281018368~4194304,297795584~16777216,331350016~37748736,373293056~4194304,402653184~12582912] 2021-07-31T02:29:55.480 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~8388608,16777216~4194304,29360128~8388608,46137344~12582912,62914560~12582912,79691776~29360128,113246208~8388608,130023424~12582912,150994944~12582912,167772160~37748736,218103808~8388608,230686720~12582912,264241152~8388608,281018368~4194304,297795584~16777216,331350016~37748736,373293056~4194304,402653184~12582912] 2021-07-31T02:29:55.484 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~8388608,16439262~8726562,29360128~8388608,46137344~12582912,62262910~46788994,113246208~8388608,130023424~12582912,146755873~44767,150994944~58720256,215110864~32353072,261553838~11075922,281018368~4194304,290354177~3247103,297795584~20971520,331350016~50331648,395588806~2870074,402653184~12582912] 2021-07-31T02:29:55.485 INFO:tasks.workunit.client.0.smithi045.stdout:from 14 to 15 diff [50331648~4194304,67108864~4194304,79691776~4194304,138412032~4194304,167772160~8388608,297795584~8388608,335544320~8388608,373293056~4194304,402653184~12582912] 2021-07-31T02:29:55.485 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [50331648~4194304,67108864~4194304,79691776~4194304,138412032~4194304,167772160~8388608,297795584~8388608,335544320~8388608,373293056~4194304,402653184~12582912] 2021-07-31T02:29:55.491 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [16439262~337954,33446354~108078,50331648~4194304,62262910~651650,67108864~8388608,79691776~4194304,94926588~1542404,115357951~2082561,121422973~211843,138412032~4194304,146755873~44767,164361273~15993799,215110864~2992944,247142031~321905,261553838~2687314,290354177~3247103,297795584~8388608,335499074~8433854,356296674~219166,373293056~8388608,395588806~2870074,402653184~12582912] 2021-07-31T02:29:55.491 INFO:tasks.workunit.client.0.smithi045.stdout:from 14 to 16 diff [4194304~4194304,29360128~8388608,50331648~8388608,67108864~8388608,79691776~4194304,88080384~8388608,130023424~12582912,155189248~8388608,167772160~8388608,234881024~4194304,297795584~8388608,335544320~8388608,356515840~12582912,373293056~4194304,402653184~12582912] 2021-07-31T02:29:55.491 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~4194304,29360128~8388608,50331648~8388608,67108864~8388608,79691776~4194304,88080384~8388608,130023424~12582912,155189248~8388608,167772160~8388608,234881024~4194304,297795584~8388608,335544320~8388608,356515840~12582912,373293056~4194304,402653184~12582912] 2021-07-31T02:29:55.497 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~8388608,16439262~337954,29360128~8388608,50331648~8388608,62262910~651650,67108864~16777216,88080384~8388608,115357951~2082561,121422973~211843,130023424~12582912,146755873~44767,155189248~25165824,215110864~2992944,234881024~4194304,247142031~321905,261553838~2687314,290354177~3247103,297795584~8388608,335499074~8433854,356296674~25384990,395588806~2870074,402653184~12582912] 2021-07-31T02:29:55.498 INFO:tasks.workunit.client.0.smithi045.stdout:from 14 to 17 diff [4194304~8388608,29360128~8388608,50331648~8388608,67108864~8388608,79691776~4194304,88080384~8388608,130023424~12582912,155189248~8388608,167772160~16777216,218103808~8388608,234881024~8388608,264241152~8388608,297795584~8388608,335544320~12582912,352321536~16777216,373293056~4194304,402653184~12582912] 2021-07-31T02:29:55.498 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~8388608,29360128~8388608,50331648~8388608,67108864~8388608,79691776~4194304,88080384~8388608,130023424~12582912,155189248~8388608,167772160~16777216,218103808~8388608,234881024~8388608,264241152~8388608,297795584~8388608,335544320~12582912,352321536~16777216,373293056~4194304,402653184~12582912] 2021-07-31T02:29:55.502 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~8388608,16439262~337954,29360128~8388608,50331648~8388608,62262910~651650,67108864~16777216,88080384~8388608,115357951~2082561,121422973~211843,130023424~12582912,146755873~44767,155189248~29360128,215110864~15575856,234881024~12582912,261553838~11075922,290354177~3247103,297795584~8388608,335499074~12628158,352321536~29360128,395588806~2870074,402653184~12582912] 2021-07-31T02:29:55.503 INFO:tasks.workunit.client.0.smithi045.stdout:from 14 to 18 diff [4194304~8388608,29360128~8388608,46137344~12582912,67108864~8388608,79691776~4194304,88080384~16777216,113246208~8388608,130023424~12582912,150994944~12582912,167772160~16777216,218103808~8388608,234881024~8388608,264241152~8388608,297795584~8388608,335544320~12582912,352321536~16777216,373293056~4194304,402653184~12582912] 2021-07-31T02:29:55.503 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~8388608,29360128~8388608,46137344~12582912,67108864~8388608,79691776~4194304,88080384~16777216,113246208~8388608,130023424~12582912,150994944~12582912,167772160~16777216,218103808~8388608,234881024~8388608,264241152~8388608,297795584~8388608,335544320~12582912,352321536~16777216,373293056~4194304,402653184~12582912] 2021-07-31T02:29:55.510 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~8388608,16439262~337954,29360128~8388608,46137344~12582912,62262910~651650,67108864~16777216,88080384~20971520,113246208~8388608,130023424~12582912,146755873~44767,150994944~33554432,215110864~15575856,234881024~12582912,261553838~11075922,290354177~3247103,297795584~8388608,335499074~12628158,352321536~29360128,395588806~2870074,402653184~12582912] 2021-07-31T02:29:55.511 INFO:tasks.workunit.client.0.smithi045.stdout:from 14 to 19 diff [4194304~8388608,29360128~8388608,46137344~12582912,62914560~12582912,79691776~4194304,88080384~16777216,113246208~8388608,130023424~12582912,150994944~12582912,167772160~16777216,218103808~8388608,230686720~12582912,264241152~8388608,297795584~16777216,331350016~16777216,352321536~16777216,373293056~4194304,402653184~12582912] 2021-07-31T02:29:55.511 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~8388608,29360128~8388608,46137344~12582912,62914560~12582912,79691776~4194304,88080384~16777216,113246208~8388608,130023424~12582912,150994944~12582912,167772160~16777216,218103808~8388608,230686720~12582912,264241152~8388608,297795584~16777216,331350016~16777216,352321536~16777216,373293056~4194304,402653184~12582912] 2021-07-31T02:29:55.514 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~8388608,16439262~337954,29360128~8388608,46137344~12582912,62262910~21623170,88080384~20971520,113246208~8388608,130023424~12582912,146755873~44767,150994944~33554432,215110864~32353072,261553838~11075922,290354177~3247103,297795584~20971520,331350016~16777216,352321536~29360128,395588806~2870074,402653184~12582912] 2021-07-31T02:29:55.514 INFO:tasks.workunit.client.0.smithi045.stdout:from 15 to 16 diff [4194304~4194304,29360128~8388608,50331648~8388608,71303168~4194304,88080384~8388608,130023424~8388608,155189248~8388608,234881024~4194304,356515840~12582912] 2021-07-31T02:29:55.515 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~4194304,29360128~8388608,50331648~8388608,71303168~4194304,88080384~8388608,130023424~8388608,155189248~8388608,234881024~4194304,356515840~12582912] 2021-07-31T02:29:55.520 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~8388608,16439262~337954,29360128~8388608,50331648~8388608,62262910~651650,71303168~8388608,82797315~1088765,88080384~8388608,115357951~2082561,121422973~211843,130023424~8388608,146755873~44767,155189248~12582912,215110864~2992944,234881024~4194304,247142031~321905,261553838~2687314,290354177~3247103,303928448~2255744,335499074~45246,356296674~16996382,395588806~2870074,413263671~1972425] 2021-07-31T02:29:55.520 INFO:tasks.workunit.client.0.smithi045.stdout:from 15 to 17 diff [4194304~8388608,29360128~8388608,50331648~8388608,71303168~4194304,88080384~8388608,130023424~8388608,155189248~8388608,171966464~8388608,218103808~8388608,234881024~8388608,264241152~8388608,335544320~8388608,352321536~16777216] 2021-07-31T02:29:55.521 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~8388608,29360128~8388608,50331648~8388608,71303168~4194304,88080384~8388608,130023424~8388608,155189248~8388608,171966464~8388608,218103808~8388608,234881024~8388608,264241152~8388608,335544320~8388608,352321536~16777216] 2021-07-31T02:29:55.527 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~8388608,16439262~337954,29360128~8388608,50331648~8388608,62262910~651650,71303168~8388608,82797315~1088765,88080384~8388608,115357951~2082561,121422973~211843,130023424~8388608,146755873~44767,155189248~12582912,171966464~12582912,215110864~15575856,234881024~12582912,261553838~11075922,290354177~3247103,303928448~2255744,335499074~12628158,352321536~20971520,395588806~2870074,413263671~1972425] 2021-07-31T02:29:55.527 INFO:tasks.workunit.client.0.smithi045.stdout:from 15 to 18 diff [4194304~8388608,29360128~8388608,46137344~12582912,71303168~4194304,88080384~16777216,113246208~8388608,130023424~8388608,150994944~12582912,171966464~8388608,218103808~8388608,234881024~8388608,264241152~8388608,297795584~4194304,335544320~8388608,352321536~16777216] 2021-07-31T02:29:55.527 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~8388608,29360128~8388608,46137344~12582912,71303168~4194304,88080384~16777216,113246208~8388608,130023424~8388608,150994944~12582912,171966464~8388608,218103808~8388608,234881024~8388608,264241152~8388608,297795584~4194304,335544320~8388608,352321536~16777216] 2021-07-31T02:29:55.533 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~8388608,16439262~337954,29360128~8388608,46137344~12582912,62262910~651650,71303168~8388608,82797315~1088765,88080384~20971520,113246208~8388608,130023424~8388608,146755873~44767,150994944~16777216,171966464~12582912,215110864~15575856,234881024~12582912,261553838~11075922,290354177~3247103,297795584~8388608,335499074~12628158,352321536~20971520,395588806~2870074,413263671~1972425] 2021-07-31T02:29:55.533 INFO:tasks.workunit.client.0.smithi045.stdout:from 15 to 19 diff [4194304~8388608,29360128~8388608,46137344~12582912,62914560~12582912,88080384~16777216,113246208~8388608,130023424~8388608,150994944~12582912,171966464~8388608,218103808~8388608,230686720~12582912,264241152~8388608,297795584~16777216,331350016~12582912,352321536~16777216,402653184~4194304] 2021-07-31T02:29:55.533 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~8388608,29360128~8388608,46137344~12582912,62914560~12582912,88080384~16777216,113246208~8388608,130023424~8388608,150994944~12582912,171966464~8388608,218103808~8388608,230686720~12582912,264241152~8388608,297795584~16777216,331350016~12582912,352321536~16777216,402653184~4194304] 2021-07-31T02:29:55.539 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~8388608,16439262~337954,29360128~8388608,46137344~12582912,62262910~17428866,82797315~1088765,88080384~20971520,113246208~8388608,130023424~8388608,146755873~44767,150994944~16777216,171966464~12582912,215110864~32353072,261553838~11075922,290354177~3247103,297795584~20971520,331350016~16777216,352321536~20971520,395588806~2870074,402653184~4194304,413263671~1972425] 2021-07-31T02:29:55.539 INFO:tasks.workunit.client.0.smithi045.stdout:from 16 to 17 diff [4194304~4194304,171966464~8388608,218103808~8388608,239075328~4194304,264241152~8388608,335544320~8388608,352321536~8388608] 2021-07-31T02:29:55.540 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~4194304,171966464~8388608,218103808~8388608,239075328~4194304,264241152~8388608,335544320~8388608,352321536~8388608] 2021-07-31T02:29:55.544 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~8388608,16439262~337954,62262910~651650,82797315~1088765,95353474~1115518,115357951~2082561,121422973~211843,146755873~44767,164361273~3410887,171966464~12582912,215110864~15575856,239075328~8388608,261553838~11075922,290354177~3247103,303928448~2255744,335499074~12628158,352321536~8388608,395588806~2870074,413263671~1972425] 2021-07-31T02:29:55.545 INFO:tasks.workunit.client.0.smithi045.stdout:from 16 to 18 diff [4194304~4194304,46137344~8388608,71303168~4194304,88080384~4194304,96468992~8388608,113246208~8388608,150994944~4194304,159383552~4194304,171966464~8388608,218103808~8388608,239075328~4194304,264241152~8388608,297795584~4194304,335544320~8388608,352321536~8388608] 2021-07-31T02:29:55.545 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~4194304,46137344~8388608,71303168~4194304,88080384~4194304,96468992~8388608,113246208~8388608,150994944~4194304,159383552~4194304,171966464~8388608,218103808~8388608,239075328~4194304,264241152~8388608,297795584~4194304,335544320~8388608,352321536~8388608] 2021-07-31T02:29:55.550 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~8388608,16439262~337954,46137344~12582912,62262910~651650,71303168~4194304,82797315~1088765,88080384~4194304,95353474~13698430,113246208~8388608,146755873~44767,150994944~16777216,171966464~12582912,215110864~15575856,239075328~8388608,261553838~11075922,290354177~3247103,297795584~8388608,335499074~12628158,352321536~8388608,395588806~2870074,413263671~1972425] 2021-07-31T02:29:55.551 INFO:tasks.workunit.client.0.smithi045.stdout:from 16 to 19 diff [4194304~4194304,46137344~8388608,62914560~12582912,88080384~4194304,96468992~8388608,113246208~8388608,150994944~4194304,159383552~4194304,171966464~8388608,218103808~8388608,230686720~12582912,264241152~8388608,297795584~16777216,331350016~12582912,352321536~8388608,402653184~4194304] 2021-07-31T02:29:55.551 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~4194304,46137344~8388608,62914560~12582912,88080384~4194304,96468992~8388608,113246208~8388608,150994944~4194304,159383552~4194304,171966464~8388608,218103808~8388608,230686720~12582912,264241152~8388608,297795584~16777216,331350016~12582912,352321536~8388608,402653184~4194304] 2021-07-31T02:29:55.557 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~8388608,16439262~337954,46137344~12582912,62262910~13234562,82797315~1088765,88080384~4194304,95353474~13698430,113246208~8388608,146755873~44767,150994944~16777216,171966464~12582912,215110864~32353072,261553838~11075922,290354177~3247103,297795584~20971520,331350016~16777216,352321536~8388608,395588806~2870074,402653184~4194304,413263671~1972425] 2021-07-31T02:29:55.557 INFO:tasks.workunit.client.0.smithi045.stdout:from 17 to 18 diff [46137344~8388608,71303168~4194304,88080384~4194304,96468992~8388608,113246208~8388608,150994944~4194304,159383552~4194304,264241152~8388608,297795584~4194304] 2021-07-31T02:29:55.558 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [46137344~8388608,71303168~4194304,88080384~4194304,96468992~8388608,113246208~8388608,150994944~4194304,159383552~4194304,264241152~8388608,297795584~4194304] 2021-07-31T02:29:55.563 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [16439262~337954,46137344~12582912,62262910~651650,71303168~4194304,82797315~1088765,88080384~4194304,95353474~13698430,113246208~8388608,146755873~44767,150994944~16777216,215110864~2992944,247142031~321905,261553838~11075922,290354177~3247103,297795584~8388608,335499074~45246,395588806~2870074,413263671~1972425] 2021-07-31T02:29:55.563 INFO:tasks.workunit.client.0.smithi045.stdout:from 17 to 19 diff [46137344~8388608,62914560~12582912,88080384~4194304,96468992~8388608,113246208~8388608,150994944~4194304,159383552~4194304,176160768~8388608,230686720~8388608,264241152~8388608,297795584~16777216,331350016~8388608,352321536~4194304,402653184~4194304] 2021-07-31T02:29:55.564 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [46137344~8388608,62914560~12582912,88080384~4194304,96468992~8388608,113246208~8388608,150994944~4194304,159383552~4194304,176160768~8388608,230686720~8388608,264241152~8388608,297795584~16777216,331350016~8388608,352321536~4194304,402653184~4194304] 2021-07-31T02:29:55.569 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [16439262~337954,46137344~12582912,62262910~13234562,82797315~1088765,88080384~4194304,95353474~13698430,113246208~8388608,146755873~44767,150994944~16777216,176160768~8388608,215110864~2992944,230686720~12582912,247142031~321905,261553838~11075922,290354177~3247103,297795584~20971520,331350016~8388608,352321536~4194304,395588806~2870074,402653184~4194304,413263671~1972425] 2021-07-31T02:29:55.570 INFO:tasks.workunit.client.0.smithi045.stdout:from 18 to 19 diff [62914560~8388608,176160768~8388608,230686720~8388608,297795584~16777216,331350016~8388608,352321536~4194304,402653184~4194304] 2021-07-31T02:29:55.570 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [62914560~8388608,176160768~8388608,230686720~8388608,297795584~16777216,331350016~8388608,352321536~4194304,402653184~4194304] 2021-07-31T02:29:55.575 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [16439262~337954,62262910~13234562,82797315~1088765,95353474~1115518,121422973~211843,146755873~44767,164361273~3410887,176160768~8388608,215110864~2992944,230686720~12582912,247142031~321905,261553838~2687314,290354177~3247103,297795584~20971520,331350016~8388608,352321536~4194304,395588806~2870074,402653184~4194304,413263671~1972425] 2021-07-31T02:29:55.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:55 smithi154 conmon[34553]: cluster 2021-07-31T02:29:54.384581+0000 mon.a (mon.0 2021-07-31T02:29:55.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:55 smithi154 conmon[34553]: ) 2143 : cluster [DBG] osdmap e1214: 8 total, 8 up, 8 in 2021-07-31T02:29:55.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:55 smithi154 conmon[34553]: cluster 2021-07-31T02:29:54.408560+0000 mgr.x (mgr.24542) 1830 : cluster [DBG] pgmap v2341: 41 pgs: 41 active+clean; 892 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 7.5 KiB/s rd, 47 MiB/s wr, 34 op/s 2021-07-31T02:29:55.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:55 smithi045 conmon[33655]: cluster 2021-07-31T02:29:54.384581+0000 mon.a (mon.0) 2143 : cluster [DBG] osdmap e1214: 8 total, 8 up, 8 in 2021-07-31T02:29:55.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:55 smithi045 conmon[33655]: cluster 2021- 2021-07-31T02:29:55.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:55 smithi045 conmon[33655]: 07-31T02:29:54.408560+0000 mgr.x (mgr.24542) 1830 : cluster [DBG] pgmap v2341: 41 pgs: 41 active+clean; 892 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 7.5 KiB/s rd, 47 MiB/s wr, 34 op/s 2021-07-31T02:29:55.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:55 smithi045 conmon[39718]: cluster 2021-07-31T02:29:54.384581+0000 mon.a (mon.0) 2143 : cluster [DBG] osdmap e1214: 8 total, 8 up, 8 in 2021-07-31T02:29:55.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:55 smithi045 conmon[39718]: cluster 2021-07-31T02:29:54.408560+0000 mgr.x (mgr.24542) 1830 : cluster [DBG] pgmap v2341: 41 pgs: 41 active+clean; 892 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 7.5 KiB/s rd, 47 MiB/s wr, 34 op/s 2021-07-31T02:29:56.402 INFO:tasks.workunit.client.0.smithi045.stdout:from 0 to HEAD diff [0~163577856,167772160~46137344,218103808~29360128,251658240~67108864,322961408~54525952,381681664~33554432] 2021-07-31T02:29:56.402 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~163577856,167772160~46137344,218103808~29360128,251658240~67108864,322961408~54525952,381681664~33554432] 2021-07-31T02:29:56.417 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~247463936,251658240~163577856] 2021-07-31T02:29:56.417 INFO:tasks.workunit.client.0.smithi045.stdout:from 1 to HEAD diff [0~163577856,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~33554432] 2021-07-31T02:29:56.417 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~163577856,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~33554432] 2021-07-31T02:29:56.424 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~247463936,251658240~37748736,290354177~124881919] 2021-07-31T02:29:56.424 INFO:tasks.workunit.client.0.smithi045.stdout:from 2 to HEAD diff [0~41943040,46137344~117440512,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~33554432] 2021-07-31T02:29:56.424 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~41943040,46137344~117440512,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~33554432] 2021-07-31T02:29:56.431 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~247463936,251658240~37748736,290354177~124881919] 2021-07-31T02:29:56.431 INFO:tasks.workunit.client.0.smithi045.stdout:from 3 to HEAD diff [0~41943040,46137344~117440512,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:29:56.431 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~41943040,46137344~117440512,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:29:56.438 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~247463936,251658240~37748736,290354177~124881919] 2021-07-31T02:29:56.438 INFO:tasks.workunit.client.0.smithi045.stdout:from 4 to HEAD diff [0~20971520,25165824~16777216,46137344~100663296,150994944~12582912,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:29:56.439 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~20971520,25165824~16777216,46137344~100663296,150994944~12582912,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:29:56.445 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~41943040,46137344~100663296,150994944~96468992,251658240~37748736,290354177~124881919] 2021-07-31T02:29:56.445 INFO:tasks.workunit.client.0.smithi045.stdout:from 5 to HEAD diff [0~20971520,25165824~16777216,46137344~100663296,150994944~12582912,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:29:56.445 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~20971520,25165824~16777216,46137344~100663296,150994944~12582912,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:29:56.452 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~41943040,46137344~100663296,150994944~96468992,251658240~37748736,290354177~124881919] 2021-07-31T02:29:56.452 INFO:tasks.workunit.client.0.smithi045.stdout:from 6 to HEAD diff [0~20971520,29360128~12582912,46137344~100663296,150994944~12582912,167772160~41943040,218103808~29360128,251658240~8388608,264241152~20971520,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:29:56.452 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~20971520,29360128~12582912,46137344~100663296,150994944~12582912,167772160~41943040,218103808~29360128,251658240~8388608,264241152~20971520,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:29:56.459 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~25165824,29360128~12582912,46137344~100663296,150994944~62914560,215110864~32353072,251658240~37748736,290354177~124881919] 2021-07-31T02:29:56.459 INFO:tasks.workunit.client.0.smithi045.stdout:from 7 to HEAD diff [0~20971520,29360128~12582912,46137344~100663296,150994944~12582912,167772160~41943040,218103808~25165824,251658240~8388608,264241152~20971520,293601280~20971520,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:29:56.460 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~20971520,29360128~12582912,46137344~100663296,150994944~12582912,167772160~41943040,218103808~25165824,251658240~8388608,264241152~20971520,293601280~20971520,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:29:56.465 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~25165824,29360128~12582912,46137344~100663296,150994944~62914560,215110864~32353072,251658240~37748736,290354177~28412927,322961408~92274688] 2021-07-31T02:29:56.465 INFO:tasks.workunit.client.0.smithi045.stdout:from 8 to HEAD diff [0~12582912,16777216~4194304,29360128~12582912,46137344~100663296,150994944~12582912,167772160~41943040,218103808~8388608,230686720~16777216,251658240~8388608,264241152~20971520,293601280~20971520,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:29:56.465 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~12582912,16777216~4194304,29360128~12582912,46137344~100663296,150994944~12582912,167772160~41943040,218103808~8388608,230686720~16777216,251658240~8388608,264241152~20971520,293601280~20971520,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:29:56.472 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~12582912,16439262~8726562,29360128~12582912,46137344~100663296,150994944~62914560,215110864~32353072,251658240~33554432,290354177~28412927,322961408~92274688] 2021-07-31T02:29:56.472 INFO:tasks.workunit.client.0.smithi045.stdout:from 9 to HEAD diff [4194304~8388608,16777216~4194304,29360128~12582912,46137344~12582912,62914560~83886080,150994944~12582912,167772160~37748736,218103808~8388608,230686720~16777216,251658240~8388608,264241152~20971520,293601280~20971520,322961408~54525952,390070272~4194304,398458880~16777216] 2021-07-31T02:29:56.472 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~8388608,16777216~4194304,29360128~12582912,46137344~12582912,62914560~83886080,150994944~12582912,167772160~37748736,218103808~8388608,230686720~16777216,251658240~8388608,264241152~20971520,293601280~20971520,322961408~54525952,390070272~4194304,398458880~16777216] 2021-07-31T02:29:56.479 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~8388608,16439262~8726562,29360128~12582912,46137344~100663296,150994944~58720256,215110864~32353072,251658240~33554432,290354177~28412927,322961408~58720256,390070272~25165824] 2021-07-31T02:29:56.479 INFO:tasks.workunit.client.0.smithi045.stdout:from 10 to HEAD diff [4194304~8388608,16777216~4194304,29360128~12582912,46137344~12582912,62914560~12582912,79691776~41943040,125829120~16777216,150994944~12582912,167772160~37748736,218103808~8388608,230686720~12582912,251658240~8388608,264241152~8388608,281018368~4194304,293601280~20971520,322961408~46137344,373293056~4194304,390070272~4194304,398458880~16777216] 2021-07-31T02:29:56.479 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~8388608,16777216~4194304,29360128~12582912,46137344~12582912,62914560~12582912,79691776~41943040,125829120~16777216,150994944~12582912,167772160~37748736,218103808~8388608,230686720~12582912,251658240~8388608,264241152~8388608,281018368~4194304,293601280~20971520,322961408~46137344,373293056~4194304,390070272~4194304,398458880~16777216] 2021-07-31T02:29:56.485 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~8388608,16439262~8726562,29360128~12582912,46137344~12582912,62262910~59371906,125829120~20971520,150994944~58720256,215110864~32353072,251658240~20971520,281018368~4194304,290354177~28412927,322961408~58720256,390070272~25165824] 2021-07-31T02:29:56.485 INFO:tasks.workunit.client.0.smithi045.stdout:from 11 to HEAD diff [4194304~8388608,16777216~4194304,29360128~8388608,46137344~12582912,62914560~12582912,79691776~29360128,113246208~8388608,125829120~16777216,150994944~12582912,167772160~37748736,218103808~8388608,230686720~12582912,251658240~8388608,264241152~8388608,281018368~4194304,297795584~16777216,322961408~20971520,348127232~20971520,373293056~4194304,390070272~4194304,402653184~12582912] 2021-07-31T02:29:56.486 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~8388608,16777216~4194304,29360128~8388608,46137344~12582912,62914560~12582912,79691776~29360128,113246208~8388608,125829120~16777216,150994944~12582912,167772160~37748736,218103808~8388608,230686720~12582912,251658240~8388608,264241152~8388608,281018368~4194304,297795584~16777216,322961408~20971520,348127232~20971520,373293056~4194304,390070272~4194304,402653184~12582912] 2021-07-31T02:29:56.492 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~8388608,16439262~8726562,29360128~8388608,46137344~12582912,62262910~46788994,113246208~8388608,125829120~20971520,150994944~58720256,215110864~32353072,251658240~20971520,281018368~4194304,290354177~3247103,297795584~20971520,322961408~58720256,390070272~8388608,402653184~12582912] 2021-07-31T02:29:56.492 INFO:tasks.workunit.client.0.smithi045.stdout:from 12 to HEAD diff [4194304~8388608,16777216~4194304,29360128~8388608,46137344~12582912,62914560~12582912,79691776~29360128,113246208~8388608,125829120~16777216,150994944~12582912,167772160~37748736,218103808~8388608,230686720~12582912,264241152~8388608,281018368~4194304,297795584~16777216,327155712~16777216,348127232~20971520,373293056~4194304,390070272~4194304,402653184~12582912] 2021-07-31T02:29:56.492 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~8388608,16777216~4194304,29360128~8388608,46137344~12582912,62914560~12582912,79691776~29360128,113246208~8388608,125829120~16777216,150994944~12582912,167772160~37748736,218103808~8388608,230686720~12582912,264241152~8388608,281018368~4194304,297795584~16777216,327155712~16777216,348127232~20971520,373293056~4194304,390070272~4194304,402653184~12582912] 2021-07-31T02:29:56.498 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~8388608,16439262~8726562,29360128~8388608,46137344~12582912,62262910~46788994,113246208~8388608,125829120~20971520,150994944~58720256,215110864~32353072,261553838~11075922,281018368~4194304,290354177~3247103,297795584~20971520,327155712~54525952,390070272~8388608,402653184~12582912] 2021-07-31T02:29:56.498 INFO:tasks.workunit.client.0.smithi045.stdout:from 13 to HEAD diff [4194304~8388608,16777216~4194304,29360128~8388608,46137344~12582912,62914560~12582912,79691776~29360128,113246208~8388608,130023424~12582912,150994944~12582912,167772160~37748736,218103808~8388608,230686720~12582912,264241152~8388608,281018368~4194304,297795584~16777216,331350016~37748736,373293056~4194304,402653184~12582912] 2021-07-31T02:29:56.498 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~8388608,16777216~4194304,29360128~8388608,46137344~12582912,62914560~12582912,79691776~29360128,113246208~8388608,130023424~12582912,150994944~12582912,167772160~37748736,218103808~8388608,230686720~12582912,264241152~8388608,281018368~4194304,297795584~16777216,331350016~37748736,373293056~4194304,402653184~12582912] 2021-07-31T02:29:56.504 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~8388608,16439262~8726562,29360128~8388608,46137344~12582912,62262910~46788994,113246208~8388608,130023424~12582912,146755873~44767,150994944~58720256,215110864~32353072,261553838~11075922,281018368~4194304,290354177~3247103,297795584~20971520,331350016~50331648,395588806~2870074,402653184~12582912] 2021-07-31T02:29:56.504 INFO:tasks.workunit.client.0.smithi045.stdout:from 14 to HEAD diff [4194304~8388608,29360128~8388608,46137344~12582912,62914560~12582912,79691776~4194304,88080384~16777216,113246208~8388608,130023424~12582912,150994944~12582912,167772160~16777216,218103808~8388608,230686720~12582912,264241152~8388608,297795584~16777216,331350016~16777216,352321536~16777216,373293056~4194304,402653184~12582912] 2021-07-31T02:29:56.505 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~8388608,29360128~8388608,46137344~12582912,62914560~12582912,79691776~4194304,88080384~16777216,113246208~8388608,130023424~12582912,150994944~12582912,167772160~16777216,218103808~8388608,230686720~12582912,264241152~8388608,297795584~16777216,331350016~16777216,352321536~16777216,373293056~4194304,402653184~12582912] 2021-07-31T02:29:56.511 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~8388608,16439262~337954,29360128~8388608,46137344~12582912,62262910~21623170,88080384~20971520,113246208~8388608,130023424~12582912,146755873~44767,150994944~33554432,215110864~32353072,261553838~11075922,290354177~3247103,297795584~20971520,331350016~16777216,352321536~29360128,395588806~2870074,402653184~12582912] 2021-07-31T02:29:56.511 INFO:tasks.workunit.client.0.smithi045.stdout:from 15 to HEAD diff [4194304~8388608,29360128~8388608,46137344~12582912,62914560~12582912,88080384~16777216,113246208~8388608,130023424~8388608,150994944~12582912,171966464~8388608,218103808~8388608,230686720~12582912,264241152~8388608,297795584~16777216,331350016~12582912,352321536~16777216,402653184~4194304] 2021-07-31T02:29:56.511 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~8388608,29360128~8388608,46137344~12582912,62914560~12582912,88080384~16777216,113246208~8388608,130023424~8388608,150994944~12582912,171966464~8388608,218103808~8388608,230686720~12582912,264241152~8388608,297795584~16777216,331350016~12582912,352321536~16777216,402653184~4194304] 2021-07-31T02:29:56.517 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~8388608,16439262~337954,29360128~8388608,46137344~12582912,62262910~17428866,82797315~1088765,88080384~20971520,113246208~8388608,130023424~8388608,146755873~44767,150994944~16777216,171966464~12582912,215110864~32353072,261553838~11075922,290354177~3247103,297795584~20971520,331350016~16777216,352321536~20971520,395588806~2870074,402653184~4194304,413263671~1972425] 2021-07-31T02:29:56.517 INFO:tasks.workunit.client.0.smithi045.stdout:from 16 to HEAD diff [4194304~4194304,46137344~8388608,62914560~12582912,88080384~4194304,96468992~8388608,113246208~8388608,150994944~4194304,159383552~4194304,171966464~8388608,218103808~8388608,230686720~12582912,264241152~8388608,297795584~16777216,331350016~12582912,352321536~8388608,402653184~4194304] 2021-07-31T02:29:56.517 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~4194304,46137344~8388608,62914560~12582912,88080384~4194304,96468992~8388608,113246208~8388608,150994944~4194304,159383552~4194304,171966464~8388608,218103808~8388608,230686720~12582912,264241152~8388608,297795584~16777216,331350016~12582912,352321536~8388608,402653184~4194304] 2021-07-31T02:29:56.523 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~8388608,16439262~337954,46137344~12582912,62262910~13234562,82797315~1088765,88080384~4194304,95353474~13698430,113246208~8388608,146755873~44767,150994944~16777216,171966464~12582912,215110864~32353072,261553838~11075922,290354177~3247103,297795584~20971520,331350016~16777216,352321536~8388608,395588806~2870074,402653184~4194304,413263671~1972425] 2021-07-31T02:29:56.523 INFO:tasks.workunit.client.0.smithi045.stdout:from 17 to HEAD diff [46137344~8388608,62914560~12582912,88080384~4194304,96468992~8388608,113246208~8388608,150994944~4194304,159383552~4194304,176160768~8388608,230686720~8388608,264241152~8388608,297795584~16777216,331350016~8388608,352321536~4194304,402653184~4194304] 2021-07-31T02:29:56.524 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [46137344~8388608,62914560~12582912,88080384~4194304,96468992~8388608,113246208~8388608,150994944~4194304,159383552~4194304,176160768~8388608,230686720~8388608,264241152~8388608,297795584~16777216,331350016~8388608,352321536~4194304,402653184~4194304] 2021-07-31T02:29:56.529 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [16439262~337954,46137344~12582912,62262910~13234562,82797315~1088765,88080384~4194304,95353474~13698430,113246208~8388608,146755873~44767,150994944~16777216,176160768~8388608,215110864~2992944,230686720~12582912,247142031~321905,261553838~11075922,290354177~3247103,297795584~20971520,331350016~8388608,352321536~4194304,395588806~2870074,402653184~4194304,413263671~1972425] 2021-07-31T02:29:57.444 INFO:tasks.workunit.client.0.smithi045.stdout:from 0 to HEAD diff [0~163577856,167772160~46137344,218103808~29360128,251658240~67108864,322961408~54525952,381681664~33554432] 2021-07-31T02:29:57.444 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~163577856,167772160~46137344,218103808~29360128,251658240~67108864,322961408~54525952,381681664~33554432] 2021-07-31T02:29:57.477 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~247463936,251658240~163577856] 2021-07-31T02:29:57.477 INFO:tasks.workunit.client.0.smithi045.stdout:from 1 to HEAD diff [0~163577856,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~33554432] 2021-07-31T02:29:57.478 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~163577856,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~33554432] 2021-07-31T02:29:57.483 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~247463936,251658240~37748736,290354177~124881919] 2021-07-31T02:29:57.483 INFO:tasks.workunit.client.0.smithi045.stdout:from 2 to HEAD diff [0~41943040,46137344~117440512,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~33554432] 2021-07-31T02:29:57.484 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~41943040,46137344~117440512,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~33554432] 2021-07-31T02:29:57.489 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~247463936,251658240~37748736,290354177~124881919] 2021-07-31T02:29:57.490 INFO:tasks.workunit.client.0.smithi045.stdout:from 3 to HEAD diff [0~41943040,46137344~117440512,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:29:57.490 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~41943040,46137344~117440512,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:29:57.496 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~247463936,251658240~37748736,290354177~124881919] 2021-07-31T02:29:57.496 INFO:tasks.workunit.client.0.smithi045.stdout:from 4 to HEAD diff [0~20971520,25165824~16777216,46137344~100663296,150994944~12582912,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:29:57.497 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~20971520,25165824~16777216,46137344~100663296,150994944~12582912,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:29:57.502 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~41943040,46137344~100663296,150994944~96468992,251658240~37748736,290354177~124881919] 2021-07-31T02:29:57.503 INFO:tasks.workunit.client.0.smithi045.stdout:from 5 to HEAD diff [0~20971520,25165824~16777216,46137344~100663296,150994944~12582912,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:29:57.503 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~20971520,25165824~16777216,46137344~100663296,150994944~12582912,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:29:57.507 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~41943040,46137344~100663296,150994944~96468992,251658240~37748736,290354177~124881919] 2021-07-31T02:29:57.508 INFO:tasks.workunit.client.0.smithi045.stdout:from 6 to HEAD diff [0~20971520,29360128~12582912,46137344~100663296,150994944~12582912,167772160~41943040,218103808~29360128,251658240~8388608,264241152~20971520,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:29:57.508 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~20971520,29360128~12582912,46137344~100663296,150994944~12582912,167772160~41943040,218103808~29360128,251658240~8388608,264241152~20971520,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:29:57.513 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~25165824,29360128~12582912,46137344~100663296,150994944~62914560,215110864~32353072,251658240~37748736,290354177~124881919] 2021-07-31T02:29:57.514 INFO:tasks.workunit.client.0.smithi045.stdout:from 7 to HEAD diff [0~20971520,29360128~12582912,46137344~100663296,150994944~12582912,167772160~41943040,218103808~25165824,251658240~8388608,264241152~20971520,293601280~20971520,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:29:57.514 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~20971520,29360128~12582912,46137344~100663296,150994944~12582912,167772160~41943040,218103808~25165824,251658240~8388608,264241152~20971520,293601280~20971520,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:29:57.520 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~25165824,29360128~12582912,46137344~100663296,150994944~62914560,215110864~32353072,251658240~37748736,290354177~28412927,322961408~92274688] 2021-07-31T02:29:57.520 INFO:tasks.workunit.client.0.smithi045.stdout:from 8 to HEAD diff [0~12582912,16777216~4194304,29360128~12582912,46137344~100663296,150994944~12582912,167772160~41943040,218103808~8388608,230686720~16777216,251658240~8388608,264241152~20971520,293601280~20971520,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:29:57.520 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~12582912,16777216~4194304,29360128~12582912,46137344~100663296,150994944~12582912,167772160~41943040,218103808~8388608,230686720~16777216,251658240~8388608,264241152~20971520,293601280~20971520,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:29:57.526 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~12582912,16439262~8726562,29360128~12582912,46137344~100663296,150994944~62914560,215110864~32353072,251658240~33554432,290354177~28412927,322961408~92274688] 2021-07-31T02:29:57.526 INFO:tasks.workunit.client.0.smithi045.stdout:from 9 to HEAD diff [4194304~8388608,16777216~4194304,29360128~12582912,46137344~12582912,62914560~83886080,150994944~12582912,167772160~37748736,218103808~8388608,230686720~16777216,251658240~8388608,264241152~20971520,293601280~20971520,322961408~54525952,390070272~4194304,398458880~16777216] 2021-07-31T02:29:57.527 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~8388608,16777216~4194304,29360128~12582912,46137344~12582912,62914560~83886080,150994944~12582912,167772160~37748736,218103808~8388608,230686720~16777216,251658240~8388608,264241152~20971520,293601280~20971520,322961408~54525952,390070272~4194304,398458880~16777216] 2021-07-31T02:29:57.532 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~8388608,16439262~8726562,29360128~12582912,46137344~100663296,150994944~58720256,215110864~32353072,251658240~33554432,290354177~28412927,322961408~58720256,390070272~25165824] 2021-07-31T02:29:57.532 INFO:tasks.workunit.client.0.smithi045.stdout:from 10 to HEAD diff [4194304~8388608,16777216~4194304,29360128~12582912,46137344~12582912,62914560~12582912,79691776~41943040,125829120~16777216,150994944~12582912,167772160~37748736,218103808~8388608,230686720~12582912,251658240~8388608,264241152~8388608,281018368~4194304,293601280~20971520,322961408~46137344,373293056~4194304,390070272~4194304,398458880~16777216] 2021-07-31T02:29:57.533 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~8388608,16777216~4194304,29360128~12582912,46137344~12582912,62914560~12582912,79691776~41943040,125829120~16777216,150994944~12582912,167772160~37748736,218103808~8388608,230686720~12582912,251658240~8388608,264241152~8388608,281018368~4194304,293601280~20971520,322961408~46137344,373293056~4194304,390070272~4194304,398458880~16777216] 2021-07-31T02:29:57.538 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~8388608,16439262~8726562,29360128~12582912,46137344~12582912,62262910~59371906,125829120~20971520,150994944~58720256,215110864~32353072,251658240~20971520,281018368~4194304,290354177~28412927,322961408~58720256,390070272~25165824] 2021-07-31T02:29:57.539 INFO:tasks.workunit.client.0.smithi045.stdout:from 11 to HEAD diff [4194304~8388608,16777216~4194304,29360128~8388608,46137344~12582912,62914560~12582912,79691776~29360128,113246208~8388608,125829120~16777216,150994944~12582912,167772160~37748736,218103808~8388608,230686720~12582912,251658240~8388608,264241152~8388608,281018368~4194304,297795584~16777216,322961408~20971520,348127232~20971520,373293056~4194304,390070272~4194304,402653184~12582912] 2021-07-31T02:29:57.539 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~8388608,16777216~4194304,29360128~8388608,46137344~12582912,62914560~12582912,79691776~29360128,113246208~8388608,125829120~16777216,150994944~12582912,167772160~37748736,218103808~8388608,230686720~12582912,251658240~8388608,264241152~8388608,281018368~4194304,297795584~16777216,322961408~20971520,348127232~20971520,373293056~4194304,390070272~4194304,402653184~12582912] 2021-07-31T02:29:57.545 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~8388608,16439262~8726562,29360128~8388608,46137344~12582912,62262910~46788994,113246208~8388608,125829120~20971520,150994944~58720256,215110864~32353072,251658240~20971520,281018368~4194304,290354177~3247103,297795584~20971520,322961408~58720256,390070272~8388608,402653184~12582912] 2021-07-31T02:29:57.545 INFO:tasks.workunit.client.0.smithi045.stdout:from 12 to HEAD diff [4194304~8388608,16777216~4194304,29360128~8388608,46137344~12582912,62914560~12582912,79691776~29360128,113246208~8388608,125829120~16777216,150994944~12582912,167772160~37748736,218103808~8388608,230686720~12582912,264241152~8388608,281018368~4194304,297795584~16777216,327155712~16777216,348127232~20971520,373293056~4194304,390070272~4194304,402653184~12582912] 2021-07-31T02:29:57.545 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~8388608,16777216~4194304,29360128~8388608,46137344~12582912,62914560~12582912,79691776~29360128,113246208~8388608,125829120~16777216,150994944~12582912,167772160~37748736,218103808~8388608,230686720~12582912,264241152~8388608,281018368~4194304,297795584~16777216,327155712~16777216,348127232~20971520,373293056~4194304,390070272~4194304,402653184~12582912] 2021-07-31T02:29:57.551 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~8388608,16439262~8726562,29360128~8388608,46137344~12582912,62262910~46788994,113246208~8388608,125829120~20971520,150994944~58720256,215110864~32353072,261553838~11075922,281018368~4194304,290354177~3247103,297795584~20971520,327155712~54525952,390070272~8388608,402653184~12582912] 2021-07-31T02:29:57.551 INFO:tasks.workunit.client.0.smithi045.stdout:from 13 to HEAD diff [4194304~8388608,16777216~4194304,29360128~8388608,46137344~12582912,62914560~12582912,79691776~29360128,113246208~8388608,130023424~12582912,150994944~12582912,167772160~37748736,218103808~8388608,230686720~12582912,264241152~8388608,281018368~4194304,297795584~16777216,331350016~37748736,373293056~4194304,402653184~12582912] 2021-07-31T02:29:57.551 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~8388608,16777216~4194304,29360128~8388608,46137344~12582912,62914560~12582912,79691776~29360128,113246208~8388608,130023424~12582912,150994944~12582912,167772160~37748736,218103808~8388608,230686720~12582912,264241152~8388608,281018368~4194304,297795584~16777216,331350016~37748736,373293056~4194304,402653184~12582912] 2021-07-31T02:29:57.557 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~8388608,16439262~8726562,29360128~8388608,46137344~12582912,62262910~46788994,113246208~8388608,130023424~12582912,146755873~44767,150994944~58720256,215110864~32353072,261553838~11075922,281018368~4194304,290354177~3247103,297795584~20971520,331350016~50331648,395588806~2870074,402653184~12582912] 2021-07-31T02:29:57.557 INFO:tasks.workunit.client.0.smithi045.stdout:from 14 to HEAD diff [4194304~8388608,29360128~8388608,46137344~12582912,62914560~12582912,79691776~4194304,88080384~16777216,113246208~8388608,130023424~12582912,150994944~12582912,167772160~16777216,218103808~8388608,230686720~12582912,264241152~8388608,297795584~16777216,331350016~16777216,352321536~16777216,373293056~4194304,402653184~12582912] 2021-07-31T02:29:57.557 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~8388608,29360128~8388608,46137344~12582912,62914560~12582912,79691776~4194304,88080384~16777216,113246208~8388608,130023424~12582912,150994944~12582912,167772160~16777216,218103808~8388608,230686720~12582912,264241152~8388608,297795584~16777216,331350016~16777216,352321536~16777216,373293056~4194304,402653184~12582912] 2021-07-31T02:29:57.563 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~8388608,16439262~337954,29360128~8388608,46137344~12582912,62262910~21623170,88080384~20971520,113246208~8388608,130023424~12582912,146755873~44767,150994944~33554432,215110864~32353072,261553838~11075922,290354177~3247103,297795584~20971520,331350016~16777216,352321536~29360128,395588806~2870074,402653184~12582912] 2021-07-31T02:29:57.563 INFO:tasks.workunit.client.0.smithi045.stdout:from 15 to HEAD diff [4194304~8388608,29360128~8388608,46137344~12582912,62914560~12582912,88080384~16777216,113246208~8388608,130023424~8388608,150994944~12582912,171966464~8388608,218103808~8388608,230686720~12582912,264241152~8388608,297795584~16777216,331350016~12582912,352321536~16777216,402653184~4194304] 2021-07-31T02:29:57.563 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~8388608,29360128~8388608,46137344~12582912,62914560~12582912,88080384~16777216,113246208~8388608,130023424~8388608,150994944~12582912,171966464~8388608,218103808~8388608,230686720~12582912,264241152~8388608,297795584~16777216,331350016~12582912,352321536~16777216,402653184~4194304] 2021-07-31T02:29:57.569 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~8388608,16439262~337954,29360128~8388608,46137344~12582912,62262910~17428866,82797315~1088765,88080384~20971520,113246208~8388608,130023424~8388608,146755873~44767,150994944~16777216,171966464~12582912,215110864~32353072,261553838~11075922,290354177~3247103,297795584~20971520,331350016~16777216,352321536~20971520,395588806~2870074,402653184~4194304,413263671~1972425] 2021-07-31T02:29:57.569 INFO:tasks.workunit.client.0.smithi045.stdout:from 16 to HEAD diff [4194304~4194304,46137344~8388608,62914560~12582912,88080384~4194304,96468992~8388608,113246208~8388608,150994944~4194304,159383552~4194304,171966464~8388608,218103808~8388608,230686720~12582912,264241152~8388608,297795584~16777216,331350016~12582912,352321536~8388608,402653184~4194304] 2021-07-31T02:29:57.569 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~4194304,46137344~8388608,62914560~12582912,88080384~4194304,96468992~8388608,113246208~8388608,150994944~4194304,159383552~4194304,171966464~8388608,218103808~8388608,230686720~12582912,264241152~8388608,297795584~16777216,331350016~12582912,352321536~8388608,402653184~4194304] 2021-07-31T02:29:57.575 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~8388608,16439262~337954,46137344~12582912,62262910~13234562,82797315~1088765,88080384~4194304,95353474~13698430,113246208~8388608,146755873~44767,150994944~16777216,171966464~12582912,215110864~32353072,261553838~11075922,290354177~3247103,297795584~20971520,331350016~16777216,352321536~8388608,395588806~2870074,402653184~4194304,413263671~1972425] 2021-07-31T02:29:57.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:57 smithi154 conmon[34553]: cluster 2021-07-31T02:29:56.395186+0000 mon.a (mon.0) 2144 : cluster [DBG] osdmap e1215: 8 total, 8 up, 8 in 2021-07-31T02:29:57.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:57 smithi154 conmon[34553]: cluster 2021-07-31T02:29: 2021-07-31T02:29:57.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:57 smithi154 conmon[34553]: 56.408943+0000 mgr.x (mgr.24542) 1831 : cluster [DBG] pgmap v2343: 41 pgs: 41 active+clean; 892 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 7.4 KiB/s rd, 46 MiB/s wr, 34 op/s 2021-07-31T02:29:57.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:57 smithi045 conmon[39718]: cluster 2021-07-31T02:29:56.395186+0000 mon.a (mon 2021-07-31T02:29:57.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:57 smithi045 conmon[39718]: .0) 2144 : cluster [DBG] osdmap e1215: 8 total, 8 up, 8 in 2021-07-31T02:29:57.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:57 smithi045 conmon[39718]: cluster 2021-07-31T02:29:56.408943 2021-07-31T02:29:57.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:57 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 1831 : cluster [DBG] pgmap v2343: 41 pgs: 41 active+clean; 892 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 7.4 KiB/s rd, 46 MiB/s wr, 34 op/s 2021-07-31T02:29:57.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:57 smithi045 conmon[39718]: 2021-07-31T02:29:57.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:57 smithi045 conmon[33655]: cluster 2021-07-31T02:29:56.395186+0000 mon.a (mon 2021-07-31T02:29:57.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:57 smithi045 conmon[33655]: .0) 2144 : cluster [DBG] osdmap e1215: 8 total, 8 up, 8 in 2021-07-31T02:29:57.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:57 smithi045 conmon[33655]: cluster 2021-07-31T02:29:56.408943+0000 mgr.x (mgr.24542) 1831 : cluster [DBG] pgmap v2343: 41 pgs: 41 active+clean; 892 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 7.4 KiB/s rd, 46 MiB/s wr, 34 op/s 2021-07-31T02:29:58.456 INFO:tasks.workunit.client.0.smithi045.stdout:from 0 to HEAD diff [0~163577856,167772160~46137344,218103808~29360128,251658240~67108864,322961408~54525952,381681664~33554432] 2021-07-31T02:29:58.457 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~163577856,167772160~46137344,218103808~29360128,251658240~67108864,322961408~54525952,381681664~33554432] 2021-07-31T02:29:58.504 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~247463936,251658240~163577856] 2021-07-31T02:29:58.504 INFO:tasks.workunit.client.0.smithi045.stdout:from 1 to HEAD diff [0~163577856,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~33554432] 2021-07-31T02:29:58.505 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~163577856,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~33554432] 2021-07-31T02:29:58.521 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~247463936,251658240~37748736,290354177~124881919] 2021-07-31T02:29:58.522 INFO:tasks.workunit.client.0.smithi045.stdout:from 2 to HEAD diff [0~41943040,46137344~117440512,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~33554432] 2021-07-31T02:29:58.522 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~41943040,46137344~117440512,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~33554432] 2021-07-31T02:29:58.528 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~247463936,251658240~37748736,290354177~124881919] 2021-07-31T02:29:58.528 INFO:tasks.workunit.client.0.smithi045.stdout:from 3 to HEAD diff [0~41943040,46137344~117440512,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:29:58.528 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~41943040,46137344~117440512,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:29:58.535 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~247463936,251658240~37748736,290354177~124881919] 2021-07-31T02:29:58.535 INFO:tasks.workunit.client.0.smithi045.stdout:from 4 to HEAD diff [0~20971520,25165824~16777216,46137344~100663296,150994944~12582912,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:29:58.535 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~20971520,25165824~16777216,46137344~100663296,150994944~12582912,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:29:58.541 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~41943040,46137344~100663296,150994944~96468992,251658240~37748736,290354177~124881919] 2021-07-31T02:29:58.542 INFO:tasks.workunit.client.0.smithi045.stdout:from 5 to HEAD diff [0~20971520,25165824~16777216,46137344~100663296,150994944~12582912,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:29:58.542 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~20971520,25165824~16777216,46137344~100663296,150994944~12582912,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:29:58.548 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~41943040,46137344~100663296,150994944~96468992,251658240~37748736,290354177~124881919] 2021-07-31T02:29:58.548 INFO:tasks.workunit.client.0.smithi045.stdout:from 6 to HEAD diff [0~20971520,29360128~12582912,46137344~100663296,150994944~12582912,167772160~41943040,218103808~29360128,251658240~8388608,264241152~20971520,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:29:58.548 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~20971520,29360128~12582912,46137344~100663296,150994944~12582912,167772160~41943040,218103808~29360128,251658240~8388608,264241152~20971520,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:29:58.555 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~25165824,29360128~12582912,46137344~100663296,150994944~62914560,215110864~32353072,251658240~37748736,290354177~124881919] 2021-07-31T02:29:58.556 INFO:tasks.workunit.client.0.smithi045.stdout:from 7 to HEAD diff [0~20971520,29360128~12582912,46137344~100663296,150994944~12582912,167772160~41943040,218103808~25165824,251658240~8388608,264241152~20971520,293601280~20971520,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:29:58.556 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~20971520,29360128~12582912,46137344~100663296,150994944~12582912,167772160~41943040,218103808~25165824,251658240~8388608,264241152~20971520,293601280~20971520,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:29:58.563 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~25165824,29360128~12582912,46137344~100663296,150994944~62914560,215110864~32353072,251658240~37748736,290354177~28412927,322961408~92274688] 2021-07-31T02:29:58.564 INFO:tasks.workunit.client.0.smithi045.stdout:from 8 to HEAD diff [0~12582912,16777216~4194304,29360128~12582912,46137344~100663296,150994944~12582912,167772160~41943040,218103808~8388608,230686720~16777216,251658240~8388608,264241152~20971520,293601280~20971520,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:29:58.564 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~12582912,16777216~4194304,29360128~12582912,46137344~100663296,150994944~12582912,167772160~41943040,218103808~8388608,230686720~16777216,251658240~8388608,264241152~20971520,293601280~20971520,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:29:58.568 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~12582912,16439262~8726562,29360128~12582912,46137344~100663296,150994944~62914560,215110864~32353072,251658240~33554432,290354177~28412927,322961408~92274688] 2021-07-31T02:29:58.568 INFO:tasks.workunit.client.0.smithi045.stdout:from 9 to HEAD diff [4194304~8388608,16777216~4194304,29360128~12582912,46137344~12582912,62914560~83886080,150994944~12582912,167772160~37748736,218103808~8388608,230686720~16777216,251658240~8388608,264241152~20971520,293601280~20971520,322961408~54525952,390070272~4194304,398458880~16777216] 2021-07-31T02:29:58.569 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~8388608,16777216~4194304,29360128~12582912,46137344~12582912,62914560~83886080,150994944~12582912,167772160~37748736,218103808~8388608,230686720~16777216,251658240~8388608,264241152~20971520,293601280~20971520,322961408~54525952,390070272~4194304,398458880~16777216] 2021-07-31T02:29:58.575 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~8388608,16439262~8726562,29360128~12582912,46137344~100663296,150994944~58720256,215110864~32353072,251658240~33554432,290354177~28412927,322961408~58720256,390070272~25165824] 2021-07-31T02:29:58.575 INFO:tasks.workunit.client.0.smithi045.stdout:from 10 to HEAD diff [4194304~8388608,16777216~4194304,29360128~12582912,46137344~12582912,62914560~12582912,79691776~41943040,125829120~16777216,150994944~12582912,167772160~37748736,218103808~8388608,230686720~12582912,251658240~8388608,264241152~8388608,281018368~4194304,293601280~20971520,322961408~46137344,373293056~4194304,390070272~4194304,398458880~16777216] 2021-07-31T02:29:58.575 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~8388608,16777216~4194304,29360128~12582912,46137344~12582912,62914560~12582912,79691776~41943040,125829120~16777216,150994944~12582912,167772160~37748736,218103808~8388608,230686720~12582912,251658240~8388608,264241152~8388608,281018368~4194304,293601280~20971520,322961408~46137344,373293056~4194304,390070272~4194304,398458880~16777216] 2021-07-31T02:29:58.580 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~8388608,16439262~8726562,29360128~12582912,46137344~12582912,62262910~59371906,125829120~20971520,150994944~58720256,215110864~32353072,251658240~20971520,281018368~4194304,290354177~28412927,322961408~58720256,390070272~25165824] 2021-07-31T02:29:58.580 INFO:tasks.workunit.client.0.smithi045.stdout:from 11 to HEAD diff [4194304~8388608,16777216~4194304,29360128~8388608,46137344~12582912,62914560~12582912,79691776~29360128,113246208~8388608,125829120~16777216,150994944~12582912,167772160~37748736,218103808~8388608,230686720~12582912,251658240~8388608,264241152~8388608,281018368~4194304,297795584~16777216,322961408~20971520,348127232~20971520,373293056~4194304,390070272~4194304,402653184~12582912] 2021-07-31T02:29:58.581 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~8388608,16777216~4194304,29360128~8388608,46137344~12582912,62914560~12582912,79691776~29360128,113246208~8388608,125829120~16777216,150994944~12582912,167772160~37748736,218103808~8388608,230686720~12582912,251658240~8388608,264241152~8388608,281018368~4194304,297795584~16777216,322961408~20971520,348127232~20971520,373293056~4194304,390070272~4194304,402653184~12582912] 2021-07-31T02:29:58.586 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~8388608,16439262~8726562,29360128~8388608,46137344~12582912,62262910~46788994,113246208~8388608,125829120~20971520,150994944~58720256,215110864~32353072,251658240~20971520,281018368~4194304,290354177~3247103,297795584~20971520,322961408~58720256,390070272~8388608,402653184~12582912] 2021-07-31T02:29:58.586 INFO:tasks.workunit.client.0.smithi045.stdout:from 12 to HEAD diff [4194304~8388608,16777216~4194304,29360128~8388608,46137344~12582912,62914560~12582912,79691776~29360128,113246208~8388608,125829120~16777216,150994944~12582912,167772160~37748736,218103808~8388608,230686720~12582912,264241152~8388608,281018368~4194304,297795584~16777216,327155712~16777216,348127232~20971520,373293056~4194304,390070272~4194304,402653184~12582912] 2021-07-31T02:29:58.587 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~8388608,16777216~4194304,29360128~8388608,46137344~12582912,62914560~12582912,79691776~29360128,113246208~8388608,125829120~16777216,150994944~12582912,167772160~37748736,218103808~8388608,230686720~12582912,264241152~8388608,281018368~4194304,297795584~16777216,327155712~16777216,348127232~20971520,373293056~4194304,390070272~4194304,402653184~12582912] 2021-07-31T02:29:58.592 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~8388608,16439262~8726562,29360128~8388608,46137344~12582912,62262910~46788994,113246208~8388608,125829120~20971520,150994944~58720256,215110864~32353072,261553838~11075922,281018368~4194304,290354177~3247103,297795584~20971520,327155712~54525952,390070272~8388608,402653184~12582912] 2021-07-31T02:29:58.592 INFO:tasks.workunit.client.0.smithi045.stdout:from 13 to HEAD diff [4194304~8388608,16777216~4194304,29360128~8388608,46137344~12582912,62914560~12582912,79691776~29360128,113246208~8388608,130023424~12582912,150994944~12582912,167772160~37748736,218103808~8388608,230686720~12582912,264241152~8388608,281018368~4194304,297795584~16777216,331350016~37748736,373293056~4194304,402653184~12582912] 2021-07-31T02:29:58.593 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~8388608,16777216~4194304,29360128~8388608,46137344~12582912,62914560~12582912,79691776~29360128,113246208~8388608,130023424~12582912,150994944~12582912,167772160~37748736,218103808~8388608,230686720~12582912,264241152~8388608,281018368~4194304,297795584~16777216,331350016~37748736,373293056~4194304,402653184~12582912] 2021-07-31T02:29:58.598 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~8388608,16439262~8726562,29360128~8388608,46137344~12582912,62262910~46788994,113246208~8388608,130023424~12582912,146755873~44767,150994944~58720256,215110864~32353072,261553838~11075922,281018368~4194304,290354177~3247103,297795584~20971520,331350016~50331648,395588806~2870074,402653184~12582912] 2021-07-31T02:29:58.600 INFO:tasks.workunit.client.0.smithi045.stdout:from 14 to HEAD diff [4194304~8388608,29360128~8388608,46137344~12582912,62914560~12582912,79691776~4194304,88080384~16777216,113246208~8388608,130023424~12582912,150994944~12582912,167772160~16777216,218103808~8388608,230686720~12582912,264241152~8388608,297795584~16777216,331350016~16777216,352321536~16777216,373293056~4194304,402653184~12582912] 2021-07-31T02:29:58.600 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~8388608,29360128~8388608,46137344~12582912,62914560~12582912,79691776~4194304,88080384~16777216,113246208~8388608,130023424~12582912,150994944~12582912,167772160~16777216,218103808~8388608,230686720~12582912,264241152~8388608,297795584~16777216,331350016~16777216,352321536~16777216,373293056~4194304,402653184~12582912] 2021-07-31T02:29:58.613 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~8388608,16439262~337954,29360128~8388608,46137344~12582912,62262910~21623170,88080384~20971520,113246208~8388608,130023424~12582912,146755873~44767,150994944~33554432,215110864~32353072,261553838~11075922,290354177~3247103,297795584~20971520,331350016~16777216,352321536~29360128,395588806~2870074,402653184~12582912] 2021-07-31T02:29:58.613 INFO:tasks.workunit.client.0.smithi045.stdout:from 15 to HEAD diff [4194304~8388608,29360128~8388608,46137344~12582912,62914560~12582912,88080384~16777216,113246208~8388608,130023424~8388608,150994944~12582912,171966464~8388608,218103808~8388608,230686720~12582912,264241152~8388608,297795584~16777216,331350016~12582912,352321536~16777216,402653184~4194304] 2021-07-31T02:29:58.614 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~8388608,29360128~8388608,46137344~12582912,62914560~12582912,88080384~16777216,113246208~8388608,130023424~8388608,150994944~12582912,171966464~8388608,218103808~8388608,230686720~12582912,264241152~8388608,297795584~16777216,331350016~12582912,352321536~16777216,402653184~4194304] 2021-07-31T02:29:58.614 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~8388608,16439262~337954,29360128~8388608,46137344~12582912,62262910~17428866,82797315~1088765,88080384~20971520,113246208~8388608,130023424~8388608,146755873~44767,150994944~16777216,171966464~12582912,215110864~32353072,261553838~11075922,290354177~3247103,297795584~20971520,331350016~16777216,352321536~20971520,395588806~2870074,402653184~4194304,413263671~1972425] 2021-07-31T02:29:58.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:58 smithi154 conmon[34553]: cluster 2021-07-31T02:29:57.437827+0000 mon.a (mon.0) 2145 : cluster [DBG] osdmap e1216: 8 total, 8 up, 8 in 2021-07-31T02:29:58.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:58 smithi045 conmon[39718]: cluster 2021-07-31T02:29: 2021-07-31T02:29:58.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:58 smithi045 conmon[39718]: 57.437827+0000 mon.a (mon.0) 2145 : cluster [DBG] osdmap e1216: 8 total, 8 up, 8 in 2021-07-31T02:29:58.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:58 smithi045 conmon[33655]: cluster 2021-07-31T02:29:57.437827+0000 mon.a ( 2021-07-31T02:29:58.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:58 smithi045 conmon[33655]: mon.0) 2145 : cluster [DBG] osdmap e1216: 8 total, 8 up, 8 in 2021-07-31T02:29:59.459 INFO:tasks.workunit.client.0.smithi045.stdout:from 0 to HEAD diff [0~163577856,167772160~46137344,218103808~29360128,251658240~67108864,322961408~54525952,381681664~33554432] 2021-07-31T02:29:59.460 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~163577856,167772160~46137344,218103808~29360128,251658240~67108864,322961408~54525952,381681664~33554432] 2021-07-31T02:29:59.501 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~247463936,251658240~163577856] 2021-07-31T02:29:59.502 INFO:tasks.workunit.client.0.smithi045.stdout:from 1 to HEAD diff [0~163577856,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~33554432] 2021-07-31T02:29:59.502 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~163577856,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~33554432] 2021-07-31T02:29:59.520 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~247463936,251658240~37748736,290354177~124881919] 2021-07-31T02:29:59.520 INFO:tasks.workunit.client.0.smithi045.stdout:from 2 to HEAD diff [0~41943040,46137344~117440512,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~33554432] 2021-07-31T02:29:59.521 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~41943040,46137344~117440512,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~33554432] 2021-07-31T02:29:59.527 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~247463936,251658240~37748736,290354177~124881919] 2021-07-31T02:29:59.527 INFO:tasks.workunit.client.0.smithi045.stdout:from 3 to HEAD diff [0~41943040,46137344~117440512,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:29:59.527 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~41943040,46137344~117440512,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:29:59.533 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~247463936,251658240~37748736,290354177~124881919] 2021-07-31T02:29:59.534 INFO:tasks.workunit.client.0.smithi045.stdout:from 4 to HEAD diff [0~20971520,25165824~16777216,46137344~100663296,150994944~12582912,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:29:59.534 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~20971520,25165824~16777216,46137344~100663296,150994944~12582912,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:29:59.540 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~41943040,46137344~100663296,150994944~96468992,251658240~37748736,290354177~124881919] 2021-07-31T02:29:59.541 INFO:tasks.workunit.client.0.smithi045.stdout:from 5 to HEAD diff [0~20971520,25165824~16777216,46137344~100663296,150994944~12582912,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:29:59.541 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~20971520,25165824~16777216,46137344~100663296,150994944~12582912,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:29:59.547 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~41943040,46137344~100663296,150994944~96468992,251658240~37748736,290354177~124881919] 2021-07-31T02:29:59.547 INFO:tasks.workunit.client.0.smithi045.stdout:from 6 to HEAD diff [0~20971520,29360128~12582912,46137344~100663296,150994944~12582912,167772160~41943040,218103808~29360128,251658240~8388608,264241152~20971520,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:29:59.547 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~20971520,29360128~12582912,46137344~100663296,150994944~12582912,167772160~41943040,218103808~29360128,251658240~8388608,264241152~20971520,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:29:59.554 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~25165824,29360128~12582912,46137344~100663296,150994944~62914560,215110864~32353072,251658240~37748736,290354177~124881919] 2021-07-31T02:29:59.554 INFO:tasks.workunit.client.0.smithi045.stdout:from 7 to HEAD diff [0~20971520,29360128~12582912,46137344~100663296,150994944~12582912,167772160~41943040,218103808~25165824,251658240~8388608,264241152~20971520,293601280~20971520,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:29:59.554 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~20971520,29360128~12582912,46137344~100663296,150994944~12582912,167772160~41943040,218103808~25165824,251658240~8388608,264241152~20971520,293601280~20971520,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:29:59.560 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~25165824,29360128~12582912,46137344~100663296,150994944~62914560,215110864~32353072,251658240~37748736,290354177~28412927,322961408~92274688] 2021-07-31T02:29:59.560 INFO:tasks.workunit.client.0.smithi045.stdout:from 8 to HEAD diff [0~12582912,16777216~4194304,29360128~12582912,46137344~100663296,150994944~12582912,167772160~41943040,218103808~8388608,230686720~16777216,251658240~8388608,264241152~20971520,293601280~20971520,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:29:59.560 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~12582912,16777216~4194304,29360128~12582912,46137344~100663296,150994944~12582912,167772160~41943040,218103808~8388608,230686720~16777216,251658240~8388608,264241152~20971520,293601280~20971520,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:29:59.566 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~12582912,16439262~8726562,29360128~12582912,46137344~100663296,150994944~62914560,215110864~32353072,251658240~33554432,290354177~28412927,322961408~92274688] 2021-07-31T02:29:59.566 INFO:tasks.workunit.client.0.smithi045.stdout:from 9 to HEAD diff [4194304~8388608,16777216~4194304,29360128~12582912,46137344~12582912,62914560~83886080,150994944~12582912,167772160~37748736,218103808~8388608,230686720~16777216,251658240~8388608,264241152~20971520,293601280~20971520,322961408~54525952,390070272~4194304,398458880~16777216] 2021-07-31T02:29:59.567 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~8388608,16777216~4194304,29360128~12582912,46137344~12582912,62914560~83886080,150994944~12582912,167772160~37748736,218103808~8388608,230686720~16777216,251658240~8388608,264241152~20971520,293601280~20971520,322961408~54525952,390070272~4194304,398458880~16777216] 2021-07-31T02:29:59.573 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~8388608,16439262~8726562,29360128~12582912,46137344~100663296,150994944~58720256,215110864~32353072,251658240~33554432,290354177~28412927,322961408~58720256,390070272~25165824] 2021-07-31T02:29:59.573 INFO:tasks.workunit.client.0.smithi045.stdout:from 10 to HEAD diff [4194304~8388608,16777216~4194304,29360128~12582912,46137344~12582912,62914560~12582912,79691776~41943040,125829120~16777216,150994944~12582912,167772160~37748736,218103808~8388608,230686720~12582912,251658240~8388608,264241152~8388608,281018368~4194304,293601280~20971520,322961408~46137344,373293056~4194304,390070272~4194304,398458880~16777216] 2021-07-31T02:29:59.574 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~8388608,16777216~4194304,29360128~12582912,46137344~12582912,62914560~12582912,79691776~41943040,125829120~16777216,150994944~12582912,167772160~37748736,218103808~8388608,230686720~12582912,251658240~8388608,264241152~8388608,281018368~4194304,293601280~20971520,322961408~46137344,373293056~4194304,390070272~4194304,398458880~16777216] 2021-07-31T02:29:59.580 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~8388608,16439262~8726562,29360128~12582912,46137344~12582912,62262910~59371906,125829120~20971520,150994944~58720256,215110864~32353072,251658240~20971520,281018368~4194304,290354177~28412927,322961408~58720256,390070272~25165824] 2021-07-31T02:29:59.580 INFO:tasks.workunit.client.0.smithi045.stdout:from 11 to HEAD diff [4194304~8388608,16777216~4194304,29360128~8388608,46137344~12582912,62914560~12582912,79691776~29360128,113246208~8388608,125829120~16777216,150994944~12582912,167772160~37748736,218103808~8388608,230686720~12582912,251658240~8388608,264241152~8388608,281018368~4194304,297795584~16777216,322961408~20971520,348127232~20971520,373293056~4194304,390070272~4194304,402653184~12582912] 2021-07-31T02:29:59.580 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~8388608,16777216~4194304,29360128~8388608,46137344~12582912,62914560~12582912,79691776~29360128,113246208~8388608,125829120~16777216,150994944~12582912,167772160~37748736,218103808~8388608,230686720~12582912,251658240~8388608,264241152~8388608,281018368~4194304,297795584~16777216,322961408~20971520,348127232~20971520,373293056~4194304,390070272~4194304,402653184~12582912] 2021-07-31T02:29:59.586 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~8388608,16439262~8726562,29360128~8388608,46137344~12582912,62262910~46788994,113246208~8388608,125829120~20971520,150994944~58720256,215110864~32353072,251658240~20971520,281018368~4194304,290354177~3247103,297795584~20971520,322961408~58720256,390070272~8388608,402653184~12582912] 2021-07-31T02:29:59.586 INFO:tasks.workunit.client.0.smithi045.stdout:from 12 to HEAD diff [4194304~8388608,16777216~4194304,29360128~8388608,46137344~12582912,62914560~12582912,79691776~29360128,113246208~8388608,125829120~16777216,150994944~12582912,167772160~37748736,218103808~8388608,230686720~12582912,264241152~8388608,281018368~4194304,297795584~16777216,327155712~16777216,348127232~20971520,373293056~4194304,390070272~4194304,402653184~12582912] 2021-07-31T02:29:59.586 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~8388608,16777216~4194304,29360128~8388608,46137344~12582912,62914560~12582912,79691776~29360128,113246208~8388608,125829120~16777216,150994944~12582912,167772160~37748736,218103808~8388608,230686720~12582912,264241152~8388608,281018368~4194304,297795584~16777216,327155712~16777216,348127232~20971520,373293056~4194304,390070272~4194304,402653184~12582912] 2021-07-31T02:29:59.592 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~8388608,16439262~8726562,29360128~8388608,46137344~12582912,62262910~46788994,113246208~8388608,125829120~20971520,150994944~58720256,215110864~32353072,261553838~11075922,281018368~4194304,290354177~3247103,297795584~20971520,327155712~54525952,390070272~8388608,402653184~12582912] 2021-07-31T02:29:59.592 INFO:tasks.workunit.client.0.smithi045.stdout:from 13 to HEAD diff [4194304~8388608,16777216~4194304,29360128~8388608,46137344~12582912,62914560~12582912,79691776~29360128,113246208~8388608,130023424~12582912,150994944~12582912,167772160~37748736,218103808~8388608,230686720~12582912,264241152~8388608,281018368~4194304,297795584~16777216,331350016~37748736,373293056~4194304,402653184~12582912] 2021-07-31T02:29:59.593 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~8388608,16777216~4194304,29360128~8388608,46137344~12582912,62914560~12582912,79691776~29360128,113246208~8388608,130023424~12582912,150994944~12582912,167772160~37748736,218103808~8388608,230686720~12582912,264241152~8388608,281018368~4194304,297795584~16777216,331350016~37748736,373293056~4194304,402653184~12582912] 2021-07-31T02:29:59.598 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~8388608,16439262~8726562,29360128~8388608,46137344~12582912,62262910~46788994,113246208~8388608,130023424~12582912,146755873~44767,150994944~58720256,215110864~32353072,261553838~11075922,281018368~4194304,290354177~3247103,297795584~20971520,331350016~50331648,395588806~2870074,402653184~12582912] 2021-07-31T02:29:59.598 INFO:tasks.workunit.client.0.smithi045.stdout:from 14 to HEAD diff [4194304~8388608,29360128~8388608,46137344~12582912,62914560~12582912,79691776~4194304,88080384~16777216,113246208~8388608,130023424~12582912,150994944~12582912,167772160~16777216,218103808~8388608,230686720~12582912,264241152~8388608,297795584~16777216,331350016~16777216,352321536~16777216,373293056~4194304,402653184~12582912] 2021-07-31T02:29:59.598 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~8388608,29360128~8388608,46137344~12582912,62914560~12582912,79691776~4194304,88080384~16777216,113246208~8388608,130023424~12582912,150994944~12582912,167772160~16777216,218103808~8388608,230686720~12582912,264241152~8388608,297795584~16777216,331350016~16777216,352321536~16777216,373293056~4194304,402653184~12582912] 2021-07-31T02:29:59.604 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~8388608,16439262~337954,29360128~8388608,46137344~12582912,62262910~21623170,88080384~20971520,113246208~8388608,130023424~12582912,146755873~44767,150994944~33554432,215110864~32353072,261553838~11075922,290354177~3247103,297795584~20971520,331350016~16777216,352321536~29360128,395588806~2870074,402653184~12582912] 2021-07-31T02:29:59.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:59 smithi154 conmon[34553]: cluster 2021-07-31T02:29:58.409466+0000 mgr.x (mgr.24542) 1832 : cluster [DBG] pgmap v2345: 41 pgs: 41 active+clean; 892 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 6.0 KiB/s rd, 37 MiB/s wr, 27 op/s 2021-07-31T02:29:59.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:29:59 smithi154 conmon[34553]: cluster 2021-07-31T02:29:58.444845+0000 mon.a (mon.0) 2146 : cluster [DBG] osdmap e1217: 8 total, 8 up, 8 in 2021-07-31T02:29:59.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:59 smithi045 conmon[33655]: cluster 2021-07-31T02:29:58.409466+0000 mgr.x (mgr. 2021-07-31T02:29:59.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:59 smithi045 conmon[33655]: 24542) 1832 : cluster [DBG] pgmap v2345: 41 pgs: 41 active+clean; 892 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 6.0 KiB/s rd, 37 MiB/s wr, 27 op/s 2021-07-31T02:29:59.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:29:59 smithi045 conmon[33655]: cluster 2021-07-31T02:29:58.444845+0000 mon.a (mon.0) 2146 : cluster [DBG] osdmap e1217: 8 total, 8 up, 8 in 2021-07-31T02:29:59.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:59 smithi045 conmon[39718]: cluster 2021-07-31T02:29:58.409466+0000 mgr.x ( 2021-07-31T02:29:59.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:59 smithi045 conmon[39718]: mgr.24542) 1832 : cluster [DBG] pgmap v2345: 41 pgs: 41 active+clean; 892 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 6.0 KiB/s rd, 37 MiB/s wr, 27 op/s 2021-07-31T02:29:59.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:59 smithi045 conmon[39718]: cluster 2021-07-31T02:29:58.444845+0000 mon.a (mon. 2021-07-31T02:29:59.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:29:59 smithi045 conmon[39718]: 0) 2146 : cluster [DBG] osdmap e1217: 8 total, 8 up, 8 in 2021-07-31T02:30:00.472 INFO:tasks.workunit.client.0.smithi045.stdout:from 0 to HEAD diff [0~163577856,167772160~46137344,218103808~29360128,251658240~67108864,322961408~54525952,381681664~33554432] 2021-07-31T02:30:00.472 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~163577856,167772160~46137344,218103808~29360128,251658240~67108864,322961408~54525952,381681664~33554432] 2021-07-31T02:30:00.517 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~247463936,251658240~163577856] 2021-07-31T02:30:00.517 INFO:tasks.workunit.client.0.smithi045.stdout:from 1 to HEAD diff [0~163577856,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~33554432] 2021-07-31T02:30:00.517 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~163577856,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~33554432] 2021-07-31T02:30:00.544 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~247463936,251658240~37748736,290354177~124881919] 2021-07-31T02:30:00.544 INFO:tasks.workunit.client.0.smithi045.stdout:from 2 to HEAD diff [0~41943040,46137344~117440512,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~33554432] 2021-07-31T02:30:00.544 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~41943040,46137344~117440512,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~33554432] 2021-07-31T02:30:00.551 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~247463936,251658240~37748736,290354177~124881919] 2021-07-31T02:30:00.551 INFO:tasks.workunit.client.0.smithi045.stdout:from 3 to HEAD diff [0~41943040,46137344~117440512,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:00.551 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~41943040,46137344~117440512,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:00.557 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~247463936,251658240~37748736,290354177~124881919] 2021-07-31T02:30:00.557 INFO:tasks.workunit.client.0.smithi045.stdout:from 4 to HEAD diff [0~20971520,25165824~16777216,46137344~100663296,150994944~12582912,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:00.558 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~20971520,25165824~16777216,46137344~100663296,150994944~12582912,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:00.563 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~41943040,46137344~100663296,150994944~96468992,251658240~37748736,290354177~124881919] 2021-07-31T02:30:00.564 INFO:tasks.workunit.client.0.smithi045.stdout:from 5 to HEAD diff [0~20971520,25165824~16777216,46137344~100663296,150994944~12582912,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:00.564 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~20971520,25165824~16777216,46137344~100663296,150994944~12582912,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:00.570 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~41943040,46137344~100663296,150994944~96468992,251658240~37748736,290354177~124881919] 2021-07-31T02:30:00.570 INFO:tasks.workunit.client.0.smithi045.stdout:from 6 to HEAD diff [0~20971520,29360128~12582912,46137344~100663296,150994944~12582912,167772160~41943040,218103808~29360128,251658240~8388608,264241152~20971520,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:00.570 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~20971520,29360128~12582912,46137344~100663296,150994944~12582912,167772160~41943040,218103808~29360128,251658240~8388608,264241152~20971520,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:00.576 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~25165824,29360128~12582912,46137344~100663296,150994944~62914560,215110864~32353072,251658240~37748736,290354177~124881919] 2021-07-31T02:30:00.576 INFO:tasks.workunit.client.0.smithi045.stdout:from 7 to HEAD diff [0~20971520,29360128~12582912,46137344~100663296,150994944~12582912,167772160~41943040,218103808~25165824,251658240~8388608,264241152~20971520,293601280~20971520,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:00.577 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~20971520,29360128~12582912,46137344~100663296,150994944~12582912,167772160~41943040,218103808~25165824,251658240~8388608,264241152~20971520,293601280~20971520,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:00.582 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~25165824,29360128~12582912,46137344~100663296,150994944~62914560,215110864~32353072,251658240~37748736,290354177~28412927,322961408~92274688] 2021-07-31T02:30:00.583 INFO:tasks.workunit.client.0.smithi045.stdout:from 8 to HEAD diff [0~12582912,16777216~4194304,29360128~12582912,46137344~100663296,150994944~12582912,167772160~41943040,218103808~8388608,230686720~16777216,251658240~8388608,264241152~20971520,293601280~20971520,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:00.583 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~12582912,16777216~4194304,29360128~12582912,46137344~100663296,150994944~12582912,167772160~41943040,218103808~8388608,230686720~16777216,251658240~8388608,264241152~20971520,293601280~20971520,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:00.589 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~12582912,16439262~8726562,29360128~12582912,46137344~100663296,150994944~62914560,215110864~32353072,251658240~33554432,290354177~28412927,322961408~92274688] 2021-07-31T02:30:00.589 INFO:tasks.workunit.client.0.smithi045.stdout:from 9 to HEAD diff [4194304~8388608,16777216~4194304,29360128~12582912,46137344~12582912,62914560~83886080,150994944~12582912,167772160~37748736,218103808~8388608,230686720~16777216,251658240~8388608,264241152~20971520,293601280~20971520,322961408~54525952,390070272~4194304,398458880~16777216] 2021-07-31T02:30:00.589 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~8388608,16777216~4194304,29360128~12582912,46137344~12582912,62914560~83886080,150994944~12582912,167772160~37748736,218103808~8388608,230686720~16777216,251658240~8388608,264241152~20971520,293601280~20971520,322961408~54525952,390070272~4194304,398458880~16777216] 2021-07-31T02:30:00.595 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~8388608,16439262~8726562,29360128~12582912,46137344~100663296,150994944~58720256,215110864~32353072,251658240~33554432,290354177~28412927,322961408~58720256,390070272~25165824] 2021-07-31T02:30:00.596 INFO:tasks.workunit.client.0.smithi045.stdout:from 10 to HEAD diff [4194304~8388608,16777216~4194304,29360128~12582912,46137344~12582912,62914560~12582912,79691776~41943040,125829120~16777216,150994944~12582912,167772160~37748736,218103808~8388608,230686720~12582912,251658240~8388608,264241152~8388608,281018368~4194304,293601280~20971520,322961408~46137344,373293056~4194304,390070272~4194304,398458880~16777216] 2021-07-31T02:30:00.596 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~8388608,16777216~4194304,29360128~12582912,46137344~12582912,62914560~12582912,79691776~41943040,125829120~16777216,150994944~12582912,167772160~37748736,218103808~8388608,230686720~12582912,251658240~8388608,264241152~8388608,281018368~4194304,293601280~20971520,322961408~46137344,373293056~4194304,390070272~4194304,398458880~16777216] 2021-07-31T02:30:00.602 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~8388608,16439262~8726562,29360128~12582912,46137344~12582912,62262910~59371906,125829120~20971520,150994944~58720256,215110864~32353072,251658240~20971520,281018368~4194304,290354177~28412927,322961408~58720256,390070272~25165824] 2021-07-31T02:30:00.602 INFO:tasks.workunit.client.0.smithi045.stdout:from 11 to HEAD diff [4194304~8388608,16777216~4194304,29360128~8388608,46137344~12582912,62914560~12582912,79691776~29360128,113246208~8388608,125829120~16777216,150994944~12582912,167772160~37748736,218103808~8388608,230686720~12582912,251658240~8388608,264241152~8388608,281018368~4194304,297795584~16777216,322961408~20971520,348127232~20971520,373293056~4194304,390070272~4194304,402653184~12582912] 2021-07-31T02:30:00.602 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~8388608,16777216~4194304,29360128~8388608,46137344~12582912,62914560~12582912,79691776~29360128,113246208~8388608,125829120~16777216,150994944~12582912,167772160~37748736,218103808~8388608,230686720~12582912,251658240~8388608,264241152~8388608,281018368~4194304,297795584~16777216,322961408~20971520,348127232~20971520,373293056~4194304,390070272~4194304,402653184~12582912] 2021-07-31T02:30:00.608 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~8388608,16439262~8726562,29360128~8388608,46137344~12582912,62262910~46788994,113246208~8388608,125829120~20971520,150994944~58720256,215110864~32353072,251658240~20971520,281018368~4194304,290354177~3247103,297795584~20971520,322961408~58720256,390070272~8388608,402653184~12582912] 2021-07-31T02:30:00.608 INFO:tasks.workunit.client.0.smithi045.stdout:from 12 to HEAD diff [4194304~8388608,16777216~4194304,29360128~8388608,46137344~12582912,62914560~12582912,79691776~29360128,113246208~8388608,125829120~16777216,150994944~12582912,167772160~37748736,218103808~8388608,230686720~12582912,264241152~8388608,281018368~4194304,297795584~16777216,327155712~16777216,348127232~20971520,373293056~4194304,390070272~4194304,402653184~12582912] 2021-07-31T02:30:00.608 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~8388608,16777216~4194304,29360128~8388608,46137344~12582912,62914560~12582912,79691776~29360128,113246208~8388608,125829120~16777216,150994944~12582912,167772160~37748736,218103808~8388608,230686720~12582912,264241152~8388608,281018368~4194304,297795584~16777216,327155712~16777216,348127232~20971520,373293056~4194304,390070272~4194304,402653184~12582912] 2021-07-31T02:30:00.614 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~8388608,16439262~8726562,29360128~8388608,46137344~12582912,62262910~46788994,113246208~8388608,125829120~20971520,150994944~58720256,215110864~32353072,261553838~11075922,281018368~4194304,290354177~3247103,297795584~20971520,327155712~54525952,390070272~8388608,402653184~12582912] 2021-07-31T02:30:00.614 INFO:tasks.workunit.client.0.smithi045.stdout:from 13 to HEAD diff [4194304~8388608,16777216~4194304,29360128~8388608,46137344~12582912,62914560~12582912,79691776~29360128,113246208~8388608,130023424~12582912,150994944~12582912,167772160~37748736,218103808~8388608,230686720~12582912,264241152~8388608,281018368~4194304,297795584~16777216,331350016~37748736,373293056~4194304,402653184~12582912] 2021-07-31T02:30:00.615 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~8388608,16777216~4194304,29360128~8388608,46137344~12582912,62914560~12582912,79691776~29360128,113246208~8388608,130023424~12582912,150994944~12582912,167772160~37748736,218103808~8388608,230686720~12582912,264241152~8388608,281018368~4194304,297795584~16777216,331350016~37748736,373293056~4194304,402653184~12582912] 2021-07-31T02:30:00.626 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~8388608,16439262~8726562,29360128~8388608,46137344~12582912,62262910~46788994,113246208~8388608,130023424~12582912,146755873~44767,150994944~58720256,215110864~32353072,261553838~11075922,281018368~4194304,290354177~3247103,297795584~20971520,331350016~50331648,395588806~2870074,402653184~12582912] 2021-07-31T02:30:00.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:00 smithi154 conmon[34553]: cluster 2021-07-31T02:29:59.450810+0000 mon.a (mon.0) 2147 : cluster [DBG] osdmap e1218: 8 total, 8 up, 8 in 2021-07-31T02:30:00.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:00 smithi154 conmon[34553]: cluster 2021-07-31T02:30:00.000189+0000 mon.a (mon.0) 2148 : cluster [INF] overall HEALTH_OK 2021-07-31T02:30:00.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:00 smithi045 conmon[39718]: cluster 2021-07-31T02:29:59.450810+0000 mon.a (mon 2021-07-31T02:30:00.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:00 smithi045 conmon[39718]: .0) 2147 : cluster [DBG] osdmap e1218: 8 total, 8 up, 8 in 2021-07-31T02:30:00.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:00 smithi045 conmon[39718]: cluster 2021-07-31T02:30:00.000189+0000 mon.a (mon. 2021-07-31T02:30:00.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:00 smithi045 conmon[39718]: 0) 2148 : cluster [INF] overall HEALTH_OK 2021-07-31T02:30:00.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:00 smithi045 conmon[33655]: cluster 2021-07-31T02:29:59.450810+0000 mon.a (mon.0) 2147 : cluster 2021-07-31T02:30:00.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:00 smithi045 conmon[33655]: [DBG] osdmap e1218: 8 total, 8 up, 8 in 2021-07-31T02:30:00.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:00 smithi045 conmon[33655]: cluster 2021-07-31T02:30:00.000189+0000 mon.a (mon.0) 2148 : cluster [INF] overall HEALTH_OK 2021-07-31T02:30:01.467 INFO:tasks.workunit.client.0.smithi045.stdout:from 0 to HEAD diff [0~163577856,167772160~46137344,218103808~29360128,251658240~67108864,322961408~54525952,381681664~33554432] 2021-07-31T02:30:01.467 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~163577856,167772160~46137344,218103808~29360128,251658240~67108864,322961408~54525952,381681664~33554432] 2021-07-31T02:30:01.515 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~247463936,251658240~163577856] 2021-07-31T02:30:01.516 INFO:tasks.workunit.client.0.smithi045.stdout:from 1 to HEAD diff [0~163577856,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~33554432] 2021-07-31T02:30:01.516 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~163577856,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~33554432] 2021-07-31T02:30:01.538 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~247463936,251658240~37748736,290354177~124881919] 2021-07-31T02:30:01.538 INFO:tasks.workunit.client.0.smithi045.stdout:from 2 to HEAD diff [0~41943040,46137344~117440512,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~33554432] 2021-07-31T02:30:01.539 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~41943040,46137344~117440512,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~33554432] 2021-07-31T02:30:01.553 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~247463936,251658240~37748736,290354177~124881919] 2021-07-31T02:30:01.553 INFO:tasks.workunit.client.0.smithi045.stdout:from 3 to HEAD diff [0~41943040,46137344~117440512,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:01.553 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~41943040,46137344~117440512,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:01.560 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~247463936,251658240~37748736,290354177~124881919] 2021-07-31T02:30:01.560 INFO:tasks.workunit.client.0.smithi045.stdout:from 4 to HEAD diff [0~20971520,25165824~16777216,46137344~100663296,150994944~12582912,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:01.560 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~20971520,25165824~16777216,46137344~100663296,150994944~12582912,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:01.567 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~41943040,46137344~100663296,150994944~96468992,251658240~37748736,290354177~124881919] 2021-07-31T02:30:01.567 INFO:tasks.workunit.client.0.smithi045.stdout:from 5 to HEAD diff [0~20971520,25165824~16777216,46137344~100663296,150994944~12582912,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:01.567 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~20971520,25165824~16777216,46137344~100663296,150994944~12582912,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:01.572 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~41943040,46137344~100663296,150994944~96468992,251658240~37748736,290354177~124881919] 2021-07-31T02:30:01.572 INFO:tasks.workunit.client.0.smithi045.stdout:from 6 to HEAD diff [0~20971520,29360128~12582912,46137344~100663296,150994944~12582912,167772160~41943040,218103808~29360128,251658240~8388608,264241152~20971520,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:01.572 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~20971520,29360128~12582912,46137344~100663296,150994944~12582912,167772160~41943040,218103808~29360128,251658240~8388608,264241152~20971520,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:01.578 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~25165824,29360128~12582912,46137344~100663296,150994944~62914560,215110864~32353072,251658240~37748736,290354177~124881919] 2021-07-31T02:30:01.579 INFO:tasks.workunit.client.0.smithi045.stdout:from 7 to HEAD diff [0~20971520,29360128~12582912,46137344~100663296,150994944~12582912,167772160~41943040,218103808~25165824,251658240~8388608,264241152~20971520,293601280~20971520,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:01.579 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~20971520,29360128~12582912,46137344~100663296,150994944~12582912,167772160~41943040,218103808~25165824,251658240~8388608,264241152~20971520,293601280~20971520,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:01.585 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~25165824,29360128~12582912,46137344~100663296,150994944~62914560,215110864~32353072,251658240~37748736,290354177~28412927,322961408~92274688] 2021-07-31T02:30:01.585 INFO:tasks.workunit.client.0.smithi045.stdout:from 8 to HEAD diff [0~12582912,16777216~4194304,29360128~12582912,46137344~100663296,150994944~12582912,167772160~41943040,218103808~8388608,230686720~16777216,251658240~8388608,264241152~20971520,293601280~20971520,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:01.585 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~12582912,16777216~4194304,29360128~12582912,46137344~100663296,150994944~12582912,167772160~41943040,218103808~8388608,230686720~16777216,251658240~8388608,264241152~20971520,293601280~20971520,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:01.591 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~12582912,16439262~8726562,29360128~12582912,46137344~100663296,150994944~62914560,215110864~32353072,251658240~33554432,290354177~28412927,322961408~92274688] 2021-07-31T02:30:01.591 INFO:tasks.workunit.client.0.smithi045.stdout:from 9 to HEAD diff [4194304~8388608,16777216~4194304,29360128~12582912,46137344~12582912,62914560~83886080,150994944~12582912,167772160~37748736,218103808~8388608,230686720~16777216,251658240~8388608,264241152~20971520,293601280~20971520,322961408~54525952,390070272~4194304,398458880~16777216] 2021-07-31T02:30:01.592 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~8388608,16777216~4194304,29360128~12582912,46137344~12582912,62914560~83886080,150994944~12582912,167772160~37748736,218103808~8388608,230686720~16777216,251658240~8388608,264241152~20971520,293601280~20971520,322961408~54525952,390070272~4194304,398458880~16777216] 2021-07-31T02:30:01.598 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~8388608,16439262~8726562,29360128~12582912,46137344~100663296,150994944~58720256,215110864~32353072,251658240~33554432,290354177~28412927,322961408~58720256,390070272~25165824] 2021-07-31T02:30:01.598 INFO:tasks.workunit.client.0.smithi045.stdout:from 10 to HEAD diff [4194304~8388608,16777216~4194304,29360128~12582912,46137344~12582912,62914560~12582912,79691776~41943040,125829120~16777216,150994944~12582912,167772160~37748736,218103808~8388608,230686720~12582912,251658240~8388608,264241152~8388608,281018368~4194304,293601280~20971520,322961408~46137344,373293056~4194304,390070272~4194304,398458880~16777216] 2021-07-31T02:30:01.599 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~8388608,16777216~4194304,29360128~12582912,46137344~12582912,62914560~12582912,79691776~41943040,125829120~16777216,150994944~12582912,167772160~37748736,218103808~8388608,230686720~12582912,251658240~8388608,264241152~8388608,281018368~4194304,293601280~20971520,322961408~46137344,373293056~4194304,390070272~4194304,398458880~16777216] 2021-07-31T02:30:01.604 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~8388608,16439262~8726562,29360128~12582912,46137344~12582912,62262910~59371906,125829120~20971520,150994944~58720256,215110864~32353072,251658240~20971520,281018368~4194304,290354177~28412927,322961408~58720256,390070272~25165824] 2021-07-31T02:30:01.604 INFO:tasks.workunit.client.0.smithi045.stdout:from 11 to HEAD diff [4194304~8388608,16777216~4194304,29360128~8388608,46137344~12582912,62914560~12582912,79691776~29360128,113246208~8388608,125829120~16777216,150994944~12582912,167772160~37748736,218103808~8388608,230686720~12582912,251658240~8388608,264241152~8388608,281018368~4194304,297795584~16777216,322961408~20971520,348127232~20971520,373293056~4194304,390070272~4194304,402653184~12582912] 2021-07-31T02:30:01.605 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~8388608,16777216~4194304,29360128~8388608,46137344~12582912,62914560~12582912,79691776~29360128,113246208~8388608,125829120~16777216,150994944~12582912,167772160~37748736,218103808~8388608,230686720~12582912,251658240~8388608,264241152~8388608,281018368~4194304,297795584~16777216,322961408~20971520,348127232~20971520,373293056~4194304,390070272~4194304,402653184~12582912] 2021-07-31T02:30:01.609 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~8388608,16439262~8726562,29360128~8388608,46137344~12582912,62262910~46788994,113246208~8388608,125829120~20971520,150994944~58720256,215110864~32353072,251658240~20971520,281018368~4194304,290354177~3247103,297795584~20971520,322961408~58720256,390070272~8388608,402653184~12582912] 2021-07-31T02:30:01.610 INFO:tasks.workunit.client.0.smithi045.stdout:from 12 to HEAD diff [4194304~8388608,16777216~4194304,29360128~8388608,46137344~12582912,62914560~12582912,79691776~29360128,113246208~8388608,125829120~16777216,150994944~12582912,167772160~37748736,218103808~8388608,230686720~12582912,264241152~8388608,281018368~4194304,297795584~16777216,327155712~16777216,348127232~20971520,373293056~4194304,390070272~4194304,402653184~12582912] 2021-07-31T02:30:01.610 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~8388608,16777216~4194304,29360128~8388608,46137344~12582912,62914560~12582912,79691776~29360128,113246208~8388608,125829120~16777216,150994944~12582912,167772160~37748736,218103808~8388608,230686720~12582912,264241152~8388608,281018368~4194304,297795584~16777216,327155712~16777216,348127232~20971520,373293056~4194304,390070272~4194304,402653184~12582912] 2021-07-31T02:30:01.615 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~8388608,16439262~8726562,29360128~8388608,46137344~12582912,62262910~46788994,113246208~8388608,125829120~20971520,150994944~58720256,215110864~32353072,261553838~11075922,281018368~4194304,290354177~3247103,297795584~20971520,327155712~54525952,390070272~8388608,402653184~12582912] 2021-07-31T02:30:01.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:01 smithi154 conmon[34553]: cluster 2021-07-31T02:30:00.409894+0000 mgr.x ( 2021-07-31T02:30:01.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:01 smithi154 conmon[34553]: mgr.24542) 1833 : cluster [DBG] pgmap v2348: 41 pgs: 41 active+clean; 882 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 5.2 KiB/s rd, 1019 B/s wr, 5.89k op/s 2021-07-31T02:30:01.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:01 smithi154 conmon[34553]: cluster 2021-07-31T02:30:00.456843+0000 mon.a (mon.0) 2149 : cluster [DBG] osdmap e1219: 8 total, 8 up, 8 in 2021-07-31T02:30:01.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:01 smithi045 conmon[39718]: cluster 2021-07-31T02:30:00 2021-07-31T02:30:01.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:01 smithi045 conmon[39718]: .409894+0000 mgr.x (mgr.24542) 1833 : cluster [DBG] pgmap v2348: 41 pgs: 41 active+clean; 882 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 5.2 KiB/s rd, 1019 B/s wr, 5.89k op/s 2021-07-31T02:30:01.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:01 smithi045 conmon[39718]: cluster 2021- 2021-07-31T02:30:01.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:01 smithi045 conmon[39718]: 07-31T02:30:00.456843+0000 mon.a (mon 2021-07-31T02:30:01.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:01 smithi045 conmon[39718]: .0) 2149 : cluster [DBG] osdmap e1219: 8 total, 8 up, 8 in 2021-07-31T02:30:01.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:01 smithi045 conmon[33655]: cluster 2021-07-31T02:30:00.409894+0000 mgr.x (mgr 2021-07-31T02:30:01.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:01 smithi045 conmon[33655]: .24542) 1833 : cluster [DBG] pgmap v2348: 41 pgs: 41 active+clean; 882 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 5.2 KiB/s rd, 1019 B/s wr, 5.89k op/s 2021-07-31T02:30:01.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:01 smithi045 conmon[33655]: cluster 2021-07-31T02:30:00.456843+0000 mon.a (mon.0) 2149 : cluster [DBG] osdmap e1219: 8 total, 8 up, 8 in 2021-07-31T02:30:02.295 INFO:tasks.workunit.client.0.smithi045.stdout:from 0 to HEAD diff [0~163577856,167772160~46137344,218103808~29360128,251658240~67108864,322961408~54525952,381681664~33554432] 2021-07-31T02:30:02.295 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~163577856,167772160~46137344,218103808~29360128,251658240~67108864,322961408~54525952,381681664~33554432] 2021-07-31T02:30:02.366 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~247463936,251658240~163577856] 2021-07-31T02:30:02.366 INFO:tasks.workunit.client.0.smithi045.stdout:from 1 to HEAD diff [0~163577856,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~33554432] 2021-07-31T02:30:02.367 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~163577856,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~33554432] 2021-07-31T02:30:02.390 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~247463936,251658240~37748736,290354177~124881919] 2021-07-31T02:30:02.390 INFO:tasks.workunit.client.0.smithi045.stdout:from 2 to HEAD diff [0~41943040,46137344~117440512,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~33554432] 2021-07-31T02:30:02.390 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~41943040,46137344~117440512,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~33554432] 2021-07-31T02:30:02.396 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~247463936,251658240~37748736,290354177~124881919] 2021-07-31T02:30:02.396 INFO:tasks.workunit.client.0.smithi045.stdout:from 3 to HEAD diff [0~41943040,46137344~117440512,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:02.398 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~41943040,46137344~117440512,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:02.403 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~247463936,251658240~37748736,290354177~124881919] 2021-07-31T02:30:02.403 INFO:tasks.workunit.client.0.smithi045.stdout:from 4 to HEAD diff [0~20971520,25165824~16777216,46137344~100663296,150994944~12582912,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:02.403 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~20971520,25165824~16777216,46137344~100663296,150994944~12582912,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:02.411 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~41943040,46137344~100663296,150994944~96468992,251658240~37748736,290354177~124881919] 2021-07-31T02:30:02.411 INFO:tasks.workunit.client.0.smithi045.stdout:from 5 to HEAD diff [0~20971520,25165824~16777216,46137344~100663296,150994944~12582912,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:02.411 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~20971520,25165824~16777216,46137344~100663296,150994944~12582912,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:02.415 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~41943040,46137344~100663296,150994944~96468992,251658240~37748736,290354177~124881919] 2021-07-31T02:30:02.416 INFO:tasks.workunit.client.0.smithi045.stdout:from 6 to HEAD diff [0~20971520,29360128~12582912,46137344~100663296,150994944~12582912,167772160~41943040,218103808~29360128,251658240~8388608,264241152~20971520,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:02.416 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~20971520,29360128~12582912,46137344~100663296,150994944~12582912,167772160~41943040,218103808~29360128,251658240~8388608,264241152~20971520,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:02.421 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~25165824,29360128~12582912,46137344~100663296,150994944~62914560,215110864~32353072,251658240~37748736,290354177~124881919] 2021-07-31T02:30:02.422 INFO:tasks.workunit.client.0.smithi045.stdout:from 7 to HEAD diff [0~20971520,29360128~12582912,46137344~100663296,150994944~12582912,167772160~41943040,218103808~25165824,251658240~8388608,264241152~20971520,293601280~20971520,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:02.422 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~20971520,29360128~12582912,46137344~100663296,150994944~12582912,167772160~41943040,218103808~25165824,251658240~8388608,264241152~20971520,293601280~20971520,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:02.428 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~25165824,29360128~12582912,46137344~100663296,150994944~62914560,215110864~32353072,251658240~37748736,290354177~28412927,322961408~92274688] 2021-07-31T02:30:02.428 INFO:tasks.workunit.client.0.smithi045.stdout:from 8 to HEAD diff [0~12582912,16777216~4194304,29360128~12582912,46137344~100663296,150994944~12582912,167772160~41943040,218103808~8388608,230686720~16777216,251658240~8388608,264241152~20971520,293601280~20971520,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:02.429 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~12582912,16777216~4194304,29360128~12582912,46137344~100663296,150994944~12582912,167772160~41943040,218103808~8388608,230686720~16777216,251658240~8388608,264241152~20971520,293601280~20971520,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:02.434 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~12582912,16439262~8726562,29360128~12582912,46137344~100663296,150994944~62914560,215110864~32353072,251658240~33554432,290354177~28412927,322961408~92274688] 2021-07-31T02:30:02.435 INFO:tasks.workunit.client.0.smithi045.stdout:from 9 to HEAD diff [4194304~8388608,16777216~4194304,29360128~12582912,46137344~12582912,62914560~83886080,150994944~12582912,167772160~37748736,218103808~8388608,230686720~16777216,251658240~8388608,264241152~20971520,293601280~20971520,322961408~54525952,390070272~4194304,398458880~16777216] 2021-07-31T02:30:02.435 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~8388608,16777216~4194304,29360128~12582912,46137344~12582912,62914560~83886080,150994944~12582912,167772160~37748736,218103808~8388608,230686720~16777216,251658240~8388608,264241152~20971520,293601280~20971520,322961408~54525952,390070272~4194304,398458880~16777216] 2021-07-31T02:30:02.441 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~8388608,16439262~8726562,29360128~12582912,46137344~100663296,150994944~58720256,215110864~32353072,251658240~33554432,290354177~28412927,322961408~58720256,390070272~25165824] 2021-07-31T02:30:02.441 INFO:tasks.workunit.client.0.smithi045.stdout:from 10 to HEAD diff [4194304~8388608,16777216~4194304,29360128~12582912,46137344~12582912,62914560~12582912,79691776~41943040,125829120~16777216,150994944~12582912,167772160~37748736,218103808~8388608,230686720~12582912,251658240~8388608,264241152~8388608,281018368~4194304,293601280~20971520,322961408~46137344,373293056~4194304,390070272~4194304,398458880~16777216] 2021-07-31T02:30:02.441 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~8388608,16777216~4194304,29360128~12582912,46137344~12582912,62914560~12582912,79691776~41943040,125829120~16777216,150994944~12582912,167772160~37748736,218103808~8388608,230686720~12582912,251658240~8388608,264241152~8388608,281018368~4194304,293601280~20971520,322961408~46137344,373293056~4194304,390070272~4194304,398458880~16777216] 2021-07-31T02:30:02.447 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~8388608,16439262~8726562,29360128~12582912,46137344~12582912,62262910~59371906,125829120~20971520,150994944~58720256,215110864~32353072,251658240~20971520,281018368~4194304,290354177~28412927,322961408~58720256,390070272~25165824] 2021-07-31T02:30:02.447 INFO:tasks.workunit.client.0.smithi045.stdout:from 11 to HEAD diff [4194304~8388608,16777216~4194304,29360128~8388608,46137344~12582912,62914560~12582912,79691776~29360128,113246208~8388608,125829120~16777216,150994944~12582912,167772160~37748736,218103808~8388608,230686720~12582912,251658240~8388608,264241152~8388608,281018368~4194304,297795584~16777216,322961408~20971520,348127232~20971520,373293056~4194304,390070272~4194304,402653184~12582912] 2021-07-31T02:30:02.447 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~8388608,16777216~4194304,29360128~8388608,46137344~12582912,62914560~12582912,79691776~29360128,113246208~8388608,125829120~16777216,150994944~12582912,167772160~37748736,218103808~8388608,230686720~12582912,251658240~8388608,264241152~8388608,281018368~4194304,297795584~16777216,322961408~20971520,348127232~20971520,373293056~4194304,390070272~4194304,402653184~12582912] 2021-07-31T02:30:02.453 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~8388608,16439262~8726562,29360128~8388608,46137344~12582912,62262910~46788994,113246208~8388608,125829120~20971520,150994944~58720256,215110864~32353072,251658240~20971520,281018368~4194304,290354177~3247103,297795584~20971520,322961408~58720256,390070272~8388608,402653184~12582912] 2021-07-31T02:30:02.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:02 smithi154 conmon[34553]: cluster 2021-07-31T02:30:01.458293+0000 mon.a (mon.0) 2150 : cluster [DBG] osdmap e1220: 8 total, 8 up, 8 in 2021-07-31T02:30:02.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:02 smithi154 conmon[34553]: cluster 2021-07-31T02:30:02 2021-07-31T02:30:02.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:02 smithi154 conmon[34553]: .287706+0000 mon.a (mon.0) 2151 : cluster [DBG] osdmap e1221: 8 total, 8 up, 8 in 2021-07-31T02:30:02.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:02 smithi045 conmon[39718]: cluster 2021-07-31T02:30:01.458293+0000 2021-07-31T02:30:02.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:02 smithi045 conmon[39718]: mon.a (mon.0) 2150 : cluster [DBG] osdmap e1220: 8 total, 8 up, 8 in 2021-07-31T02:30:02.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:02 smithi045 conmon[39718]: cluster 2021-07-31T02:30: 2021-07-31T02:30:02.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:02 smithi045 conmon[39718]: 02.287706+0000 mon.a (mon.0) 2151 : cluster [DBG] osdmap e1221: 8 total, 8 up, 8 in 2021-07-31T02:30:02.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:02 smithi045 conmon[33655]: cluster 2021-07-31T02:30:01.458293+0000 mon.a (mon. 2021-07-31T02:30:02.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:02 smithi045 conmon[33655]: 0) 2150 : cluster [DBG] osdmap e1220: 8 total, 8 up, 8 in 2021-07-31T02:30:02.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:02 smithi045 conmon[33655]: cluster 2021-07-31T02:30:02.287706+0000 mon.a (mon.0) 2151 : cluster [DBG] osdmap e1221: 8 total, 8 up, 8 in 2021-07-31T02:30:03.315 INFO:tasks.workunit.client.0.smithi045.stdout:from 0 to HEAD diff [0~163577856,167772160~46137344,218103808~29360128,251658240~67108864,322961408~54525952,381681664~33554432] 2021-07-31T02:30:03.316 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~163577856,167772160~46137344,218103808~29360128,251658240~67108864,322961408~54525952,381681664~33554432] 2021-07-31T02:30:03.372 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~247463936,251658240~163577856] 2021-07-31T02:30:03.373 INFO:tasks.workunit.client.0.smithi045.stdout:from 1 to HEAD diff [0~163577856,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~33554432] 2021-07-31T02:30:03.373 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~163577856,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~33554432] 2021-07-31T02:30:03.391 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~247463936,251658240~37748736,290354177~124881919] 2021-07-31T02:30:03.391 INFO:tasks.workunit.client.0.smithi045.stdout:from 2 to HEAD diff [0~41943040,46137344~117440512,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~33554432] 2021-07-31T02:30:03.392 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~41943040,46137344~117440512,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~33554432] 2021-07-31T02:30:03.403 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~247463936,251658240~37748736,290354177~124881919] 2021-07-31T02:30:03.403 INFO:tasks.workunit.client.0.smithi045.stdout:from 3 to HEAD diff [0~41943040,46137344~117440512,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:03.404 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~41943040,46137344~117440512,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:03.411 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~247463936,251658240~37748736,290354177~124881919] 2021-07-31T02:30:03.412 INFO:tasks.workunit.client.0.smithi045.stdout:from 4 to HEAD diff [0~20971520,25165824~16777216,46137344~100663296,150994944~12582912,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:03.412 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~20971520,25165824~16777216,46137344~100663296,150994944~12582912,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:03.415 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~41943040,46137344~100663296,150994944~96468992,251658240~37748736,290354177~124881919] 2021-07-31T02:30:03.416 INFO:tasks.workunit.client.0.smithi045.stdout:from 5 to HEAD diff [0~20971520,25165824~16777216,46137344~100663296,150994944~12582912,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:03.416 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~20971520,25165824~16777216,46137344~100663296,150994944~12582912,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:03.422 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~41943040,46137344~100663296,150994944~96468992,251658240~37748736,290354177~124881919] 2021-07-31T02:30:03.422 INFO:tasks.workunit.client.0.smithi045.stdout:from 6 to HEAD diff [0~20971520,29360128~12582912,46137344~100663296,150994944~12582912,167772160~41943040,218103808~29360128,251658240~8388608,264241152~20971520,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:03.422 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~20971520,29360128~12582912,46137344~100663296,150994944~12582912,167772160~41943040,218103808~29360128,251658240~8388608,264241152~20971520,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:03.429 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~25165824,29360128~12582912,46137344~100663296,150994944~62914560,215110864~32353072,251658240~37748736,290354177~124881919] 2021-07-31T02:30:03.429 INFO:tasks.workunit.client.0.smithi045.stdout:from 7 to HEAD diff [0~20971520,29360128~12582912,46137344~100663296,150994944~12582912,167772160~41943040,218103808~25165824,251658240~8388608,264241152~20971520,293601280~20971520,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:03.429 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~20971520,29360128~12582912,46137344~100663296,150994944~12582912,167772160~41943040,218103808~25165824,251658240~8388608,264241152~20971520,293601280~20971520,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:03.435 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~25165824,29360128~12582912,46137344~100663296,150994944~62914560,215110864~32353072,251658240~37748736,290354177~28412927,322961408~92274688] 2021-07-31T02:30:03.435 INFO:tasks.workunit.client.0.smithi045.stdout:from 8 to HEAD diff [0~12582912,16777216~4194304,29360128~12582912,46137344~100663296,150994944~12582912,167772160~41943040,218103808~8388608,230686720~16777216,251658240~8388608,264241152~20971520,293601280~20971520,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:03.436 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~12582912,16777216~4194304,29360128~12582912,46137344~100663296,150994944~12582912,167772160~41943040,218103808~8388608,230686720~16777216,251658240~8388608,264241152~20971520,293601280~20971520,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:03.441 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~12582912,16439262~8726562,29360128~12582912,46137344~100663296,150994944~62914560,215110864~32353072,251658240~33554432,290354177~28412927,322961408~92274688] 2021-07-31T02:30:03.442 INFO:tasks.workunit.client.0.smithi045.stdout:from 9 to HEAD diff [4194304~8388608,16777216~4194304,29360128~12582912,46137344~12582912,62914560~83886080,150994944~12582912,167772160~37748736,218103808~8388608,230686720~16777216,251658240~8388608,264241152~20971520,293601280~20971520,322961408~54525952,390070272~4194304,398458880~16777216] 2021-07-31T02:30:03.443 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~8388608,16777216~4194304,29360128~12582912,46137344~12582912,62914560~83886080,150994944~12582912,167772160~37748736,218103808~8388608,230686720~16777216,251658240~8388608,264241152~20971520,293601280~20971520,322961408~54525952,390070272~4194304,398458880~16777216] 2021-07-31T02:30:03.447 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~8388608,16439262~8726562,29360128~12582912,46137344~100663296,150994944~58720256,215110864~32353072,251658240~33554432,290354177~28412927,322961408~58720256,390070272~25165824] 2021-07-31T02:30:03.447 INFO:tasks.workunit.client.0.smithi045.stdout:from 10 to HEAD diff [4194304~8388608,16777216~4194304,29360128~12582912,46137344~12582912,62914560~12582912,79691776~41943040,125829120~16777216,150994944~12582912,167772160~37748736,218103808~8388608,230686720~12582912,251658240~8388608,264241152~8388608,281018368~4194304,293601280~20971520,322961408~46137344,373293056~4194304,390070272~4194304,398458880~16777216] 2021-07-31T02:30:03.448 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~8388608,16777216~4194304,29360128~12582912,46137344~12582912,62914560~12582912,79691776~41943040,125829120~16777216,150994944~12582912,167772160~37748736,218103808~8388608,230686720~12582912,251658240~8388608,264241152~8388608,281018368~4194304,293601280~20971520,322961408~46137344,373293056~4194304,390070272~4194304,398458880~16777216] 2021-07-31T02:30:03.454 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~8388608,16439262~8726562,29360128~12582912,46137344~12582912,62262910~59371906,125829120~20971520,150994944~58720256,215110864~32353072,251658240~20971520,281018368~4194304,290354177~28412927,322961408~58720256,390070272~25165824] 2021-07-31T02:30:04.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:30:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:30:03.583Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:30:04.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:30:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:30:03.583Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:30:04.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:30:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:30:03.583Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:30:04.318 INFO:tasks.workunit.client.0.smithi045.stdout:from 0 to HEAD diff [0~163577856,167772160~46137344,218103808~29360128,251658240~67108864,322961408~54525952,381681664~33554432] 2021-07-31T02:30:04.319 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~163577856,167772160~46137344,218103808~29360128,251658240~67108864,322961408~54525952,381681664~33554432] 2021-07-31T02:30:04.376 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~247463936,251658240~163577856] 2021-07-31T02:30:04.377 INFO:tasks.workunit.client.0.smithi045.stdout:from 1 to HEAD diff [0~163577856,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~33554432] 2021-07-31T02:30:04.377 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~163577856,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~33554432] 2021-07-31T02:30:04.401 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~247463936,251658240~37748736,290354177~124881919] 2021-07-31T02:30:04.402 INFO:tasks.workunit.client.0.smithi045.stdout:from 2 to HEAD diff [0~41943040,46137344~117440512,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~33554432] 2021-07-31T02:30:04.402 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~41943040,46137344~117440512,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~33554432] 2021-07-31T02:30:04.408 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~247463936,251658240~37748736,290354177~124881919] 2021-07-31T02:30:04.408 INFO:tasks.workunit.client.0.smithi045.stdout:from 3 to HEAD diff [0~41943040,46137344~117440512,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:04.408 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~41943040,46137344~117440512,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:04.416 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~247463936,251658240~37748736,290354177~124881919] 2021-07-31T02:30:04.416 INFO:tasks.workunit.client.0.smithi045.stdout:from 4 to HEAD diff [0~20971520,25165824~16777216,46137344~100663296,150994944~12582912,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:04.416 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~20971520,25165824~16777216,46137344~100663296,150994944~12582912,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:04.421 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~41943040,46137344~100663296,150994944~96468992,251658240~37748736,290354177~124881919] 2021-07-31T02:30:04.421 INFO:tasks.workunit.client.0.smithi045.stdout:from 5 to HEAD diff [0~20971520,25165824~16777216,46137344~100663296,150994944~12582912,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:04.421 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~20971520,25165824~16777216,46137344~100663296,150994944~12582912,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:04.428 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~41943040,46137344~100663296,150994944~96468992,251658240~37748736,290354177~124881919] 2021-07-31T02:30:04.428 INFO:tasks.workunit.client.0.smithi045.stdout:from 6 to HEAD diff [0~20971520,29360128~12582912,46137344~100663296,150994944~12582912,167772160~41943040,218103808~29360128,251658240~8388608,264241152~20971520,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:04.428 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~20971520,29360128~12582912,46137344~100663296,150994944~12582912,167772160~41943040,218103808~29360128,251658240~8388608,264241152~20971520,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:04.434 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~25165824,29360128~12582912,46137344~100663296,150994944~62914560,215110864~32353072,251658240~37748736,290354177~124881919] 2021-07-31T02:30:04.434 INFO:tasks.workunit.client.0.smithi045.stdout:from 7 to HEAD diff [0~20971520,29360128~12582912,46137344~100663296,150994944~12582912,167772160~41943040,218103808~25165824,251658240~8388608,264241152~20971520,293601280~20971520,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:04.434 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~20971520,29360128~12582912,46137344~100663296,150994944~12582912,167772160~41943040,218103808~25165824,251658240~8388608,264241152~20971520,293601280~20971520,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:04.440 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~25165824,29360128~12582912,46137344~100663296,150994944~62914560,215110864~32353072,251658240~37748736,290354177~28412927,322961408~92274688] 2021-07-31T02:30:04.440 INFO:tasks.workunit.client.0.smithi045.stdout:from 8 to HEAD diff [0~12582912,16777216~4194304,29360128~12582912,46137344~100663296,150994944~12582912,167772160~41943040,218103808~8388608,230686720~16777216,251658240~8388608,264241152~20971520,293601280~20971520,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:04.441 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~12582912,16777216~4194304,29360128~12582912,46137344~100663296,150994944~12582912,167772160~41943040,218103808~8388608,230686720~16777216,251658240~8388608,264241152~20971520,293601280~20971520,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:04.446 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~12582912,16439262~8726562,29360128~12582912,46137344~100663296,150994944~62914560,215110864~32353072,251658240~33554432,290354177~28412927,322961408~92274688] 2021-07-31T02:30:04.447 INFO:tasks.workunit.client.0.smithi045.stdout:from 9 to HEAD diff [4194304~8388608,16777216~4194304,29360128~12582912,46137344~12582912,62914560~83886080,150994944~12582912,167772160~37748736,218103808~8388608,230686720~16777216,251658240~8388608,264241152~20971520,293601280~20971520,322961408~54525952,390070272~4194304,398458880~16777216] 2021-07-31T02:30:04.447 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [4194304~8388608,16777216~4194304,29360128~12582912,46137344~12582912,62914560~83886080,150994944~12582912,167772160~37748736,218103808~8388608,230686720~16777216,251658240~8388608,264241152~20971520,293601280~20971520,322961408~54525952,390070272~4194304,398458880~16777216] 2021-07-31T02:30:04.453 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [4194304~8388608,16439262~8726562,29360128~12582912,46137344~100663296,150994944~58720256,215110864~32353072,251658240~33554432,290354177~28412927,322961408~58720256,390070272~25165824] 2021-07-31T02:30:04.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:04 smithi154 conmon[34553]: cluster 2021-07-31T02:30:02.410275+0000 mgr.x (mgr.24542) 1834 : cluster [DBG] pgmap v2352: 41 pgs: 41 active+clean; 882 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 5.3 KiB/s rd, 1.0 KiB/s wr, 5.96k op/s 2021-07-31T02:30:04.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:04 smithi154 conmon[34553]: cluster 2021-07-31T02:30:03.298731 2021-07-31T02:30:04.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:04 smithi154 conmon[34553]: +0000 mon.a (mon.0) 2152 : cluster [DBG] osdmap e1222: 8 total, 8 up, 8 in 2021-07-31T02:30:04.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:04 smithi045 conmon[39718]: cluster 2021-07-31T02:30:02.410275+0000 mgr.x (mgr.24542) 1834 : cluster [DBG] pgmap v2352: 41 pgs: 41 active+clean; 882 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 5.3 KiB/s rd, 1.0 KiB/s wr, 5.96k op/s 2021-07-31T02:30:04.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:04 smithi045 conmon[39718]: cluster 2021-07-31T02:30:03.298731+0000 mon.a (mon.0) 2152 : cluster [DBG] osdmap e1222: 8 total, 8 up, 8 in 2021-07-31T02:30:04.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:04 smithi045 conmon[33655]: cluster 2021-07-31T02:30:02.410275 2021-07-31T02:30:04.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:04 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 1834 : cluster [DBG] pgmap v2352: 41 pgs: 41 active+clean; 882 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 5.3 KiB/s rd, 1.0 KiB/s wr, 5.96k op/s 2021-07-31T02:30:04.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:04 smithi045 conmon[33655]: cluster 2021-07-31T02:30:04.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:04 smithi045 conmon[33655]: 2021-07-31T02:30:03.298731 2021-07-31T02:30:04.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:04 smithi045 conmon[33655]: +0000 mon.a (mon.0) 2152 2021-07-31T02:30:04.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:04 smithi045 conmon[33655]: : cluster [DBG] osdmap e1222: 8 total, 8 up, 8 in 2021-07-31T02:30:05.326 INFO:tasks.workunit.client.0.smithi045.stdout:from 0 to HEAD diff [0~163577856,167772160~46137344,218103808~29360128,251658240~67108864,322961408~54525952,381681664~33554432] 2021-07-31T02:30:05.327 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~163577856,167772160~46137344,218103808~29360128,251658240~67108864,322961408~54525952,381681664~33554432] 2021-07-31T02:30:05.384 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~247463936,251658240~163577856] 2021-07-31T02:30:05.384 INFO:tasks.workunit.client.0.smithi045.stdout:from 1 to HEAD diff [0~163577856,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~33554432] 2021-07-31T02:30:05.384 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~163577856,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~33554432] 2021-07-31T02:30:05.407 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~247463936,251658240~37748736,290354177~124881919] 2021-07-31T02:30:05.407 INFO:tasks.workunit.client.0.smithi045.stdout:from 2 to HEAD diff [0~41943040,46137344~117440512,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~33554432] 2021-07-31T02:30:05.408 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~41943040,46137344~117440512,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~33554432] 2021-07-31T02:30:05.413 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~247463936,251658240~37748736,290354177~124881919] 2021-07-31T02:30:05.413 INFO:tasks.workunit.client.0.smithi045.stdout:from 3 to HEAD diff [0~41943040,46137344~117440512,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:05.413 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~41943040,46137344~117440512,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:05.419 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~247463936,251658240~37748736,290354177~124881919] 2021-07-31T02:30:05.419 INFO:tasks.workunit.client.0.smithi045.stdout:from 4 to HEAD diff [0~20971520,25165824~16777216,46137344~100663296,150994944~12582912,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:05.419 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~20971520,25165824~16777216,46137344~100663296,150994944~12582912,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:05.425 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~41943040,46137344~100663296,150994944~96468992,251658240~37748736,290354177~124881919] 2021-07-31T02:30:05.425 INFO:tasks.workunit.client.0.smithi045.stdout:from 5 to HEAD diff [0~20971520,25165824~16777216,46137344~100663296,150994944~12582912,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:05.426 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~20971520,25165824~16777216,46137344~100663296,150994944~12582912,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:05.432 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~41943040,46137344~100663296,150994944~96468992,251658240~37748736,290354177~124881919] 2021-07-31T02:30:05.432 INFO:tasks.workunit.client.0.smithi045.stdout:from 6 to HEAD diff [0~20971520,29360128~12582912,46137344~100663296,150994944~12582912,167772160~41943040,218103808~29360128,251658240~8388608,264241152~20971520,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:05.433 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~20971520,29360128~12582912,46137344~100663296,150994944~12582912,167772160~41943040,218103808~29360128,251658240~8388608,264241152~20971520,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:05.438 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~25165824,29360128~12582912,46137344~100663296,150994944~62914560,215110864~32353072,251658240~37748736,290354177~124881919] 2021-07-31T02:30:05.439 INFO:tasks.workunit.client.0.smithi045.stdout:from 7 to HEAD diff [0~20971520,29360128~12582912,46137344~100663296,150994944~12582912,167772160~41943040,218103808~25165824,251658240~8388608,264241152~20971520,293601280~20971520,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:05.439 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~20971520,29360128~12582912,46137344~100663296,150994944~12582912,167772160~41943040,218103808~25165824,251658240~8388608,264241152~20971520,293601280~20971520,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:05.444 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~25165824,29360128~12582912,46137344~100663296,150994944~62914560,215110864~32353072,251658240~37748736,290354177~28412927,322961408~92274688] 2021-07-31T02:30:05.444 INFO:tasks.workunit.client.0.smithi045.stdout:from 8 to HEAD diff [0~12582912,16777216~4194304,29360128~12582912,46137344~100663296,150994944~12582912,167772160~41943040,218103808~8388608,230686720~16777216,251658240~8388608,264241152~20971520,293601280~20971520,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:05.445 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~12582912,16777216~4194304,29360128~12582912,46137344~100663296,150994944~12582912,167772160~41943040,218103808~8388608,230686720~16777216,251658240~8388608,264241152~20971520,293601280~20971520,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:05.450 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~12582912,16439262~8726562,29360128~12582912,46137344~100663296,150994944~62914560,215110864~32353072,251658240~33554432,290354177~28412927,322961408~92274688] 2021-07-31T02:30:05.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:05 smithi154 conmon[34553]: cluster 2021-07-31T02:30:04.305001+0000 mon.a (mon.0) 2153 : cluster [DBG] 2021-07-31T02:30:05.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:05 smithi154 conmon[34553]: osdmap e1223: 8 total, 8 up, 8 in 2021-07-31T02:30:05.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:05 smithi045 conmon[33655]: cluster 2021-07-31T02:30:04.305001+0000 mon.a (mon.0) 2153 : cluster [DBG] 2021-07-31T02:30:05.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:05 smithi045 conmon[33655]: osdmap e1223: 8 total, 8 up, 8 in 2021-07-31T02:30:05.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:05 smithi045 conmon[39718]: cluster 2021-07-31T02:30:04.305001+0000 mon.a (mon.0 2021-07-31T02:30:05.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:05 smithi045 conmon[39718]: ) 2153 : cluster [DBG] osdmap e1223: 8 total, 8 up, 8 in 2021-07-31T02:30:06.322 INFO:tasks.workunit.client.0.smithi045.stdout:from 0 to HEAD diff [0~163577856,167772160~46137344,218103808~29360128,251658240~67108864,322961408~54525952,381681664~33554432] 2021-07-31T02:30:06.323 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~163577856,167772160~46137344,218103808~29360128,251658240~67108864,322961408~54525952,381681664~33554432] 2021-07-31T02:30:06.381 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~247463936,251658240~163577856] 2021-07-31T02:30:06.381 INFO:tasks.workunit.client.0.smithi045.stdout:from 1 to HEAD diff [0~163577856,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~33554432] 2021-07-31T02:30:06.381 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~163577856,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~33554432] 2021-07-31T02:30:06.419 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~247463936,251658240~37748736,290354177~124881919] 2021-07-31T02:30:06.419 INFO:tasks.workunit.client.0.smithi045.stdout:from 2 to HEAD diff [0~41943040,46137344~117440512,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~33554432] 2021-07-31T02:30:06.419 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~41943040,46137344~117440512,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~33554432] 2021-07-31T02:30:06.425 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~247463936,251658240~37748736,290354177~124881919] 2021-07-31T02:30:06.425 INFO:tasks.workunit.client.0.smithi045.stdout:from 3 to HEAD diff [0~41943040,46137344~117440512,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:06.425 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~41943040,46137344~117440512,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:06.431 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~247463936,251658240~37748736,290354177~124881919] 2021-07-31T02:30:06.431 INFO:tasks.workunit.client.0.smithi045.stdout:from 4 to HEAD diff [0~20971520,25165824~16777216,46137344~100663296,150994944~12582912,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:06.432 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~20971520,25165824~16777216,46137344~100663296,150994944~12582912,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:06.438 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~41943040,46137344~100663296,150994944~96468992,251658240~37748736,290354177~124881919] 2021-07-31T02:30:06.438 INFO:tasks.workunit.client.0.smithi045.stdout:from 5 to HEAD diff [0~20971520,25165824~16777216,46137344~100663296,150994944~12582912,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:06.438 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~20971520,25165824~16777216,46137344~100663296,150994944~12582912,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:06.444 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~41943040,46137344~100663296,150994944~96468992,251658240~37748736,290354177~124881919] 2021-07-31T02:30:06.445 INFO:tasks.workunit.client.0.smithi045.stdout:from 6 to HEAD diff [0~20971520,29360128~12582912,46137344~100663296,150994944~12582912,167772160~41943040,218103808~29360128,251658240~8388608,264241152~20971520,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:06.445 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~20971520,29360128~12582912,46137344~100663296,150994944~12582912,167772160~41943040,218103808~29360128,251658240~8388608,264241152~20971520,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:06.454 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~25165824,29360128~12582912,46137344~100663296,150994944~62914560,215110864~32353072,251658240~37748736,290354177~124881919] 2021-07-31T02:30:06.454 INFO:tasks.workunit.client.0.smithi045.stdout:from 7 to HEAD diff [0~20971520,29360128~12582912,46137344~100663296,150994944~12582912,167772160~41943040,218103808~25165824,251658240~8388608,264241152~20971520,293601280~20971520,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:06.454 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~20971520,29360128~12582912,46137344~100663296,150994944~12582912,167772160~41943040,218103808~25165824,251658240~8388608,264241152~20971520,293601280~20971520,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:06.460 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~25165824,29360128~12582912,46137344~100663296,150994944~62914560,215110864~32353072,251658240~37748736,290354177~28412927,322961408~92274688] 2021-07-31T02:30:06.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:06 smithi154 conmon[34553]: cluster 2021-07-31T02:30:06.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:06 smithi154 conmon[34553]: 2021-07-31T02:30:06.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:06 smithi154 conmon[34553]: 2021 2021-07-31T02:30:06.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:06 smithi154 conmon[34553]: - 2021-07-31T02:30:06.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:06 smithi154 conmon[34553]: 07 2021-07-31T02:30:06.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:06 smithi154 conmon[34553]: - 2021-07-31T02:30:06.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:06 smithi154 conmon[34553]: 31 2021-07-31T02:30:06.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:06 smithi154 conmon[34553]: T 2021-07-31T02:30:06.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:06 smithi154 conmon[34553]: 02 2021-07-31T02:30:06.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:06 smithi154 conmon[34553]: : 2021-07-31T02:30:06.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:06 smithi154 conmon[34553]: 30 2021-07-31T02:30:06.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:06 smithi154 conmon[34553]: : 2021-07-31T02:30:06.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:06 smithi154 conmon[34553]: 04 2021-07-31T02:30:06.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:06 smithi154 conmon[34553]: .410632+0000 mgr.x (mgr.24542) 1835 : cluster [DBG] pgmap v2355: 41 pgs: 41 active+clean; 809 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 6.3 KiB/s rd, 1.3 KiB/s wr, 1.62k op/s 2021-07-31T02:30:06.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:06 smithi154 conmon[34553]: cluster 2021-07-31T02:30:05.311125+0000 mon.a (mon.0) 2154 2021-07-31T02:30:06.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:06 smithi154 conmon[34553]: : cluster [DBG] osdmap e1224: 8 total, 8 up, 8 in 2021-07-31T02:30:06.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:06 smithi045 conmon[39718]: cluster 2021-07-31T02:30:04.410632 2021-07-31T02:30:06.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:06 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 1835 : cluster [DBG] pgmap v2355: 41 pgs: 41 active+clean; 809 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 6.3 KiB/s rd, 1.3 KiB/s wr, 1.62k op/s 2021-07-31T02:30:06.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:06 smithi045 conmon[39718]: cluster 2021- 2021-07-31T02:30:06.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:06 smithi045 conmon[39718]: 07-31T02:30:05.311125+0000 mon.a (mon. 2021-07-31T02:30:06.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:06 smithi045 conmon[39718]: 0) 2154 : cluster [DBG] osdmap e1224: 8 total, 8 up, 8 in 2021-07-31T02:30:06.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:06 smithi045 conmon[33655]: cluster 2021-07-31T02:30:06.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:06 smithi045 conmon[33655]: 2021-07-31T02:30:06.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:06 smithi045 conmon[33655]: 2021 2021-07-31T02:30:06.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:06 smithi045 conmon[33655]: - 2021-07-31T02:30:06.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:06 smithi045 conmon[33655]: 07-31T02:30:04.410632+0000 2021-07-31T02:30:06.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:06 smithi045 conmon[33655]: mgr.x (mgr.24542) 1835 : cluster [DBG] pgmap v2355: 41 pgs: 41 active+clean; 809 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 6.3 KiB/s rd, 1.3 KiB/s wr, 1.62k op/s 2021-07-31T02:30:06.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:06 smithi045 conmon[33655]: cluster 2021-07-31T02:30:05.311125+0000 mon.a (mon.0) 2154 : 2021-07-31T02:30:06.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:06 smithi045 conmon[33655]: cluster [DBG] osdmap e1224: 8 total, 8 up, 8 in 2021-07-31T02:30:07.304 INFO:tasks.workunit.client.0.smithi045.stdout:from 0 to HEAD diff [0~163577856,167772160~46137344,218103808~29360128,251658240~67108864,322961408~54525952,381681664~33554432] 2021-07-31T02:30:07.304 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~163577856,167772160~46137344,218103808~29360128,251658240~67108864,322961408~54525952,381681664~33554432] 2021-07-31T02:30:07.371 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~247463936,251658240~163577856] 2021-07-31T02:30:07.372 INFO:tasks.workunit.client.0.smithi045.stdout:from 1 to HEAD diff [0~163577856,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~33554432] 2021-07-31T02:30:07.372 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~163577856,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~33554432] 2021-07-31T02:30:07.393 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~247463936,251658240~37748736,290354177~124881919] 2021-07-31T02:30:07.393 INFO:tasks.workunit.client.0.smithi045.stdout:from 2 to HEAD diff [0~41943040,46137344~117440512,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~33554432] 2021-07-31T02:30:07.393 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~41943040,46137344~117440512,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~33554432] 2021-07-31T02:30:07.399 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~247463936,251658240~37748736,290354177~124881919] 2021-07-31T02:30:07.399 INFO:tasks.workunit.client.0.smithi045.stdout:from 3 to HEAD diff [0~41943040,46137344~117440512,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:07.400 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~41943040,46137344~117440512,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:07.405 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~247463936,251658240~37748736,290354177~124881919] 2021-07-31T02:30:07.405 INFO:tasks.workunit.client.0.smithi045.stdout:from 4 to HEAD diff [0~20971520,25165824~16777216,46137344~100663296,150994944~12582912,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:07.405 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~20971520,25165824~16777216,46137344~100663296,150994944~12582912,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:07.411 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~41943040,46137344~100663296,150994944~96468992,251658240~37748736,290354177~124881919] 2021-07-31T02:30:07.412 INFO:tasks.workunit.client.0.smithi045.stdout:from 5 to HEAD diff [0~20971520,25165824~16777216,46137344~100663296,150994944~12582912,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:07.412 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~20971520,25165824~16777216,46137344~100663296,150994944~12582912,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:07.417 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~41943040,46137344~100663296,150994944~96468992,251658240~37748736,290354177~124881919] 2021-07-31T02:30:07.417 INFO:tasks.workunit.client.0.smithi045.stdout:from 6 to HEAD diff [0~20971520,29360128~12582912,46137344~100663296,150994944~12582912,167772160~41943040,218103808~29360128,251658240~8388608,264241152~20971520,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:07.418 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~20971520,29360128~12582912,46137344~100663296,150994944~12582912,167772160~41943040,218103808~29360128,251658240~8388608,264241152~20971520,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:07.423 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~25165824,29360128~12582912,46137344~100663296,150994944~62914560,215110864~32353072,251658240~37748736,290354177~124881919] 2021-07-31T02:30:07.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:07 smithi154 conmon[34553]: cluster 2021-07-31T02:30:06.313469+0000 mon.a (mon.0) 2155 : cluster [DBG] osdmap e1225: 8 total, 8 up, 8 in 2021-07-31T02:30:07.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:07 smithi154 conmon[34553]: cluster 2021-07-31T02:30:07.288594+0000 mon.a (mon.0) 2156 : cluster [DBG] osdmap e1226: 8 total, 8 up, 8 in 2021-07-31T02:30:07.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:07 smithi045 conmon[33655]: cluster 2021-07-31T02:30:07.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:07 smithi045 conmon[33655]: 2021-07-31T02:30:06.313469+0000 2021-07-31T02:30:07.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:07 smithi045 conmon[33655]: mon.a (mon.0) 2155 : cluster [DBG] osdmap e1225: 8 total, 8 up, 8 in 2021-07-31T02:30:07.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:07 smithi045 conmon[33655]: cluster 2021-07-31T02:30:07. 2021-07-31T02:30:07.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:07 smithi045 conmon[33655]: 288594+0000 mon.a (mon.0) 2156 : cluster [DBG] 2021-07-31T02:30:07.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:07 smithi045 conmon[33655]: osdmap e1226: 8 total, 8 up, 8 in 2021-07-31T02:30:07.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:07 smithi045 conmon[39718]: cluster 2021-07-31T02:30:07.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:07 smithi045 conmon[39718]: 2021-07-31T02:30:07.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:07 smithi045 conmon[39718]: 2021 2021-07-31T02:30:07.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:07 smithi045 conmon[39718]: - 2021-07-31T02:30:07.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:07 smithi045 conmon[39718]: 07 2021-07-31T02:30:07.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:07 smithi045 conmon[39718]: - 2021-07-31T02:30:07.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:07 smithi045 conmon[39718]: 31 2021-07-31T02:30:07.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:07 smithi045 conmon[39718]: T 2021-07-31T02:30:07.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:07 smithi045 conmon[39718]: 02 2021-07-31T02:30:07.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:07 smithi045 conmon[39718]: : 2021-07-31T02:30:07.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:07 smithi045 conmon[39718]: 30 2021-07-31T02:30:07.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:07 smithi045 conmon[39718]: : 2021-07-31T02:30:07.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:07 smithi045 conmon[39718]: 06 2021-07-31T02:30:07.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:07 smithi045 conmon[39718]: . 2021-07-31T02:30:07.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:07 smithi045 conmon[39718]: 313469 2021-07-31T02:30:07.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:07 smithi045 conmon[39718]: +0000 2021-07-31T02:30:07.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:07 smithi045 conmon[39718]: 2021-07-31T02:30:07.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:07 smithi045 conmon[39718]: mon.a 2021-07-31T02:30:07.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:07 smithi045 conmon[39718]: ( 2021-07-31T02:30:07.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:07 smithi045 conmon[39718]: mon 2021-07-31T02:30:07.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:07 smithi045 conmon[39718]: . 2021-07-31T02:30:07.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:07 smithi045 conmon[39718]: 0 2021-07-31T02:30:07.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:07 smithi045 conmon[39718]: ) 2021-07-31T02:30:07.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:07 smithi045 conmon[39718]: 2155 2021-07-31T02:30:07.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:07 smithi045 conmon[39718]: : 2021-07-31T02:30:07.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:07 smithi045 conmon[39718]: cluster 2021-07-31T02:30:07.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:07 smithi045 conmon[39718]: 2021-07-31T02:30:07.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:07 smithi045 conmon[39718]: [DBG] 2021-07-31T02:30:07.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:07 smithi045 conmon[39718]: 2021-07-31T02:30:07.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:07 smithi045 conmon[39718]: osdmap e1225: 8 total, 8 up, 8 in 2021-07-31T02:30:07.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:07 smithi045 conmon[39718]: 2021-07-31T02:30:07.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:07 smithi045 conmon[39718]: cluster 2021-07-31T02:30:07.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:07 smithi045 conmon[39718]: 2021-07-31T02:30:07.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:07 smithi045 conmon[39718]: 2021 2021-07-31T02:30:07.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:07 smithi045 conmon[39718]: - 2021-07-31T02:30:07.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:07 smithi045 conmon[39718]: 07 2021-07-31T02:30:07.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:07 smithi045 conmon[39718]: - 2021-07-31T02:30:07.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:07 smithi045 conmon[39718]: 31 2021-07-31T02:30:07.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:07 smithi045 conmon[39718]: T 2021-07-31T02:30:07.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:07 smithi045 conmon[39718]: 02 2021-07-31T02:30:07.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:07 smithi045 conmon[39718]: : 2021-07-31T02:30:07.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:07 smithi045 conmon[39718]: 30 2021-07-31T02:30:07.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:07 smithi045 conmon[39718]: : 2021-07-31T02:30:07.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:07 smithi045 conmon[39718]: 07 2021-07-31T02:30:07.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:07 smithi045 conmon[39718]: . 2021-07-31T02:30:07.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:07 smithi045 conmon[39718]: 288594 2021-07-31T02:30:07.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:07 smithi045 conmon[39718]: +0000 2021-07-31T02:30:07.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:07 smithi045 conmon[39718]: 2021-07-31T02:30:07.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:07 smithi045 conmon[39718]: mon.a 2021-07-31T02:30:07.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:07 smithi045 conmon[39718]: ( 2021-07-31T02:30:07.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:07 smithi045 conmon[39718]: mon 2021-07-31T02:30:07.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:07 smithi045 conmon[39718]: . 2021-07-31T02:30:07.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:07 smithi045 conmon[39718]: 0 2021-07-31T02:30:07.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:07 smithi045 conmon[39718]: ) 2021-07-31T02:30:07.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:07 smithi045 conmon[39718]: 2156 2021-07-31T02:30:07.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:07 smithi045 conmon[39718]: : 2021-07-31T02:30:07.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:07 smithi045 conmon[39718]: cluster 2021-07-31T02:30:07.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:07 smithi045 conmon[39718]: [DBG] osdmap e1226: 8 total, 8 up, 8 in 2021-07-31T02:30:08.346 INFO:tasks.workunit.client.0.smithi045.stdout:from 0 to HEAD diff [0~163577856,167772160~46137344,218103808~29360128,251658240~67108864,322961408~54525952,381681664~33554432] 2021-07-31T02:30:08.346 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~163577856,167772160~46137344,218103808~29360128,251658240~67108864,322961408~54525952,381681664~33554432] 2021-07-31T02:30:08.402 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~247463936,251658240~163577856] 2021-07-31T02:30:08.402 INFO:tasks.workunit.client.0.smithi045.stdout:from 1 to HEAD diff [0~163577856,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~33554432] 2021-07-31T02:30:08.402 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~163577856,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~33554432] 2021-07-31T02:30:08.439 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~247463936,251658240~37748736,290354177~124881919] 2021-07-31T02:30:08.440 INFO:tasks.workunit.client.0.smithi045.stdout:from 2 to HEAD diff [0~41943040,46137344~117440512,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~33554432] 2021-07-31T02:30:08.440 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~41943040,46137344~117440512,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~33554432] 2021-07-31T02:30:08.446 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~247463936,251658240~37748736,290354177~124881919] 2021-07-31T02:30:08.447 INFO:tasks.workunit.client.0.smithi045.stdout:from 3 to HEAD diff [0~41943040,46137344~117440512,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:08.447 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~41943040,46137344~117440512,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:08.453 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~247463936,251658240~37748736,290354177~124881919] 2021-07-31T02:30:08.453 INFO:tasks.workunit.client.0.smithi045.stdout:from 4 to HEAD diff [0~20971520,25165824~16777216,46137344~100663296,150994944~12582912,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:08.453 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~20971520,25165824~16777216,46137344~100663296,150994944~12582912,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:08.458 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~41943040,46137344~100663296,150994944~96468992,251658240~37748736,290354177~124881919] 2021-07-31T02:30:08.459 INFO:tasks.workunit.client.0.smithi045.stdout:from 5 to HEAD diff [0~20971520,25165824~16777216,46137344~100663296,150994944~12582912,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:08.459 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~20971520,25165824~16777216,46137344~100663296,150994944~12582912,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:08.464 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~41943040,46137344~100663296,150994944~96468992,251658240~37748736,290354177~124881919] 2021-07-31T02:30:08.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:08 smithi154 conmon[34553]: cluster 2021-07-31T02:30:08.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:08 smithi154 conmon[34553]: 2021-07-31T02:30:08.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:08 smithi154 conmon[34553]: 2021 2021-07-31T02:30:08.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:08 smithi154 conmon[34553]: - 2021-07-31T02:30:08.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:08 smithi154 conmon[34553]: 07-31T02:30:06.411126+0000 mgr.x (mgr.24542 2021-07-31T02:30:08.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:08 smithi154 conmon[34553]: ) 1836 : cluster [DBG] pgmap v2358: 41 pgs: 41 active+clean; 809 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 6.2 KiB/s rd, 1.2 KiB/s wr, 1.60k op/s 2021-07-31T02:30:08.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:08 smithi045 conmon[33655]: cluster 2021-07-31T02:30:08.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:08 smithi045 conmon[33655]: 2021-07-31T02:30: 2021-07-31T02:30:08.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:08 smithi045 conmon[33655]: 06.411126+0000 mgr.x (mgr.24542) 1836 : cluster [DBG] pgmap v2358: 41 pgs: 41 active+clean; 809 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 6.2 KiB/s rd, 1.2 KiB/s wr, 1.60k op/s 2021-07-31T02:30:08.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:08 smithi045 conmon[39718]: cluster 2021-07-31T02:30:06.411126+0000 mgr.x (mgr.24542) 1836 : cluster 2021-07-31T02:30:08.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:08 smithi045 conmon[39718]: [DBG] pgmap v2358: 41 pgs: 41 active+clean; 809 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 6.2 KiB/s rd, 1.2 KiB/s wr, 1.60k op/s 2021-07-31T02:30:09.350 INFO:tasks.workunit.client.0.smithi045.stdout:from 0 to HEAD diff [0~163577856,167772160~46137344,218103808~29360128,251658240~67108864,322961408~54525952,381681664~33554432] 2021-07-31T02:30:09.351 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~163577856,167772160~46137344,218103808~29360128,251658240~67108864,322961408~54525952,381681664~33554432] 2021-07-31T02:30:09.409 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~247463936,251658240~163577856] 2021-07-31T02:30:09.409 INFO:tasks.workunit.client.0.smithi045.stdout:from 1 to HEAD diff [0~163577856,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~33554432] 2021-07-31T02:30:09.409 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~163577856,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~33554432] 2021-07-31T02:30:09.436 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~247463936,251658240~37748736,290354177~124881919] 2021-07-31T02:30:09.436 INFO:tasks.workunit.client.0.smithi045.stdout:from 2 to HEAD diff [0~41943040,46137344~117440512,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~33554432] 2021-07-31T02:30:09.437 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~41943040,46137344~117440512,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~33554432] 2021-07-31T02:30:09.442 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~247463936,251658240~37748736,290354177~124881919] 2021-07-31T02:30:09.442 INFO:tasks.workunit.client.0.smithi045.stdout:from 3 to HEAD diff [0~41943040,46137344~117440512,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:09.443 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~41943040,46137344~117440512,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:09.451 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~247463936,251658240~37748736,290354177~124881919] 2021-07-31T02:30:09.451 INFO:tasks.workunit.client.0.smithi045.stdout:from 4 to HEAD diff [0~20971520,25165824~16777216,46137344~100663296,150994944~12582912,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:09.452 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~20971520,25165824~16777216,46137344~100663296,150994944~12582912,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:09.457 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~41943040,46137344~100663296,150994944~96468992,251658240~37748736,290354177~124881919] 2021-07-31T02:30:09.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:09 smithi154 conmon[34553]: cluster 2021-07-31T02:30:08.329226+0000 mon.a (mon.0) 2157 : cluster [DBG] osdmap e1227: 8 total, 8 up, 8 in 2021-07-31T02:30:09.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:09 smithi045 conmon[33655]: cluster 2021-07-31T02:30:08.329226+0000 mon.a (mon.0) 2157 : 2021-07-31T02:30:09.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:09 smithi045 conmon[33655]: cluster [DBG] osdmap e1227: 8 total, 8 up, 8 in 2021-07-31T02:30:09.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:09 smithi045 conmon[39718]: cluster 2021-07-31T02:30:08.329226+0000 mon.a (mon 2021-07-31T02:30:09.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:09 smithi045 conmon[39718]: .0) 2157 : cluster [DBG] osdmap e1227: 8 total, 8 up, 8 in 2021-07-31T02:30:10.356 INFO:tasks.workunit.client.0.smithi045.stdout:from 0 to HEAD diff [0~163577856,167772160~46137344,218103808~29360128,251658240~67108864,322961408~54525952,381681664~33554432] 2021-07-31T02:30:10.357 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~163577856,167772160~46137344,218103808~29360128,251658240~67108864,322961408~54525952,381681664~33554432] 2021-07-31T02:30:10.420 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~247463936,251658240~163577856] 2021-07-31T02:30:10.420 INFO:tasks.workunit.client.0.smithi045.stdout:from 1 to HEAD diff [0~163577856,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~33554432] 2021-07-31T02:30:10.420 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~163577856,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~33554432] 2021-07-31T02:30:10.449 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~247463936,251658240~37748736,290354177~124881919] 2021-07-31T02:30:10.450 INFO:tasks.workunit.client.0.smithi045.stdout:from 2 to HEAD diff [0~41943040,46137344~117440512,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~33554432] 2021-07-31T02:30:10.450 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~41943040,46137344~117440512,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~33554432] 2021-07-31T02:30:10.456 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~247463936,251658240~37748736,290354177~124881919] 2021-07-31T02:30:10.456 INFO:tasks.workunit.client.0.smithi045.stdout:from 3 to HEAD diff [0~41943040,46137344~117440512,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:10.456 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~41943040,46137344~117440512,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~12582912,398458880~16777216] 2021-07-31T02:30:10.462 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~247463936,251658240~37748736,290354177~124881919] 2021-07-31T02:30:10.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:10 smithi154 conmon[34553]: cluster 2021-07-31T02:30:10.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:10 smithi154 conmon[34553]: 2021-07-31T02:30:10.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:10 smithi154 conmon[34553]: 2021 2021-07-31T02:30:10.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:10 smithi154 conmon[34553]: - 2021-07-31T02:30:10.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:10 smithi154 conmon[34553]: 07 2021-07-31T02:30:10.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:10 smithi154 conmon[34553]: - 2021-07-31T02:30:10.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:10 smithi154 conmon[34553]: 31 2021-07-31T02:30:10.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:10 smithi154 conmon[34553]: T 2021-07-31T02:30:10.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:10 smithi154 conmon[34553]: 02 2021-07-31T02:30:10.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:10 smithi154 conmon[34553]: : 2021-07-31T02:30:10.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:10 smithi154 conmon[34553]: 30 2021-07-31T02:30:10.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:10 smithi154 conmon[34553]: : 2021-07-31T02:30:10.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:10 smithi154 conmon[34553]: 08 2021-07-31T02:30:10.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:10 smithi154 conmon[34553]: . 2021-07-31T02:30:10.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:10 smithi154 conmon[34553]: 411520 2021-07-31T02:30:10.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:10 smithi154 conmon[34553]: +0000 2021-07-31T02:30:10.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:10 smithi154 conmon[34553]: 2021-07-31T02:30:10.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:10 smithi154 conmon[34553]: mgr.x 2021-07-31T02:30:10.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:10 smithi154 conmon[34553]: ( 2021-07-31T02:30:10.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:10 smithi154 conmon[34553]: mgr 2021-07-31T02:30:10.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:10 smithi154 conmon[34553]: . 2021-07-31T02:30:10.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:10 smithi154 conmon[34553]: 24542 2021-07-31T02:30:10.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:10 smithi154 conmon[34553]: ) 2021-07-31T02:30:10.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:10 smithi154 conmon[34553]: 1837 2021-07-31T02:30:10.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:10 smithi154 conmon[34553]: : 2021-07-31T02:30:10.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:10 smithi154 conmon[34553]: cluster 2021-07-31T02:30:10.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:10 smithi154 conmon[34553]: 2021-07-31T02:30:10.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:10 smithi154 conmon[34553]: [DBG] 2021-07-31T02:30:10.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:10 smithi154 conmon[34553]: 2021-07-31T02:30:10.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:10 smithi154 conmon[34553]: pgmap v2361: 41 pgs: 41 active+clean; 809 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail 2021-07-31T02:30:10.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:10 smithi154 conmon[34553]: 2021-07-31T02:30:10.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:10 smithi154 conmon[34553]: cluster 2021-07-31T02:30:10.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:10 smithi154 conmon[34553]: 2021-07-31T02:30:10.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:10 smithi154 conmon[34553]: 2021 2021-07-31T02:30:10.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:10 smithi154 conmon[34553]: - 2021-07-31T02:30:10.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:10 smithi154 conmon[34553]: 07 2021-07-31T02:30:10.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:10 smithi154 conmon[34553]: - 2021-07-31T02:30:10.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:10 smithi154 conmon[34553]: 31 2021-07-31T02:30:10.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:10 smithi154 conmon[34553]: T 2021-07-31T02:30:10.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:10 smithi154 conmon[34553]: 02 2021-07-31T02:30:10.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:10 smithi154 conmon[34553]: : 2021-07-31T02:30:10.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:10 smithi154 conmon[34553]: 30 2021-07-31T02:30:10.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:10 smithi154 conmon[34553]: : 2021-07-31T02:30:10.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:10 smithi154 conmon[34553]: 09 2021-07-31T02:30:10.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:10 smithi154 conmon[34553]: . 2021-07-31T02:30:10.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:10 smithi154 conmon[34553]: 340980 2021-07-31T02:30:10.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:10 smithi154 conmon[34553]: +0000 2021-07-31T02:30:10.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:10 smithi154 conmon[34553]: 2021-07-31T02:30:10.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:10 smithi154 conmon[34553]: mon.a 2021-07-31T02:30:10.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:10 smithi154 conmon[34553]: ( 2021-07-31T02:30:10.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:10 smithi154 conmon[34553]: mon 2021-07-31T02:30:10.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:10 smithi154 conmon[34553]: . 2021-07-31T02:30:10.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:10 smithi154 conmon[34553]: 0 2021-07-31T02:30:10.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:10 smithi154 conmon[34553]: ) 2021-07-31T02:30:10.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:10 smithi154 conmon[34553]: 2158 2021-07-31T02:30:10.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:10 smithi154 conmon[34553]: : 2021-07-31T02:30:10.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:10 smithi154 conmon[34553]: cluster 2021-07-31T02:30:10.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:10 smithi154 conmon[34553]: 2021-07-31T02:30:10.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:10 smithi154 conmon[34553]: [DBG] 2021-07-31T02:30:10.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:10 smithi154 conmon[34553]: 2021-07-31T02:30:10.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:10 smithi154 conmon[34553]: osdmap e1228: 8 total, 8 up, 8 in 2021-07-31T02:30:10.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:10 smithi154 conmon[34553]: 2021-07-31T02:30:10.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:10 smithi045 conmon[33655]: cluster 2021-07-31T02:30:10.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:10 smithi045 conmon[33655]: 2021-07-31 2021-07-31T02:30:10.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:10 smithi045 conmon[33655]: T02:30:08.411520+0000 mgr.x (mgr.24542) 1837 : cluster [DBG] pgmap v2361: 41 pgs: 41 active+clean; 809 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail 2021-07-31T02:30:10.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:10 smithi045 conmon[33655]: cluster 2021-07-31T02:30:09.340980+0000 mon.a (mon.0) 2158 : cluster [DBG] osdmap e1228: 8 total, 8 up, 8 in 2021-07-31T02:30:10.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:10 smithi045 conmon[39718]: cluster 2021-07-31T02:30:08.411520+0000 mgr.x (mgr 2021-07-31T02:30:10.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:10 smithi045 conmon[39718]: .24542) 1837 : cluster [DBG] pgmap v2361: 41 pgs: 41 active+clean; 809 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail 2021-07-31T02:30:10.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:10 smithi045 conmon[39718]: cluster 2021-07-31T02:30:09.340980+0000 mon.a (mon.0) 2158 : cluster [DBG] osdmap e1228: 8 total, 8 up, 8 in 2021-07-31T02:30:11.366 INFO:tasks.workunit.client.0.smithi045.stdout:from 0 to HEAD diff [0~163577856,167772160~46137344,218103808~29360128,251658240~67108864,322961408~54525952,381681664~33554432] 2021-07-31T02:30:11.366 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~163577856,167772160~46137344,218103808~29360128,251658240~67108864,322961408~54525952,381681664~33554432] 2021-07-31T02:30:11.425 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~247463936,251658240~163577856] 2021-07-31T02:30:11.426 INFO:tasks.workunit.client.0.smithi045.stdout:from 1 to HEAD diff [0~163577856,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~33554432] 2021-07-31T02:30:11.426 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~163577856,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~33554432] 2021-07-31T02:30:11.441 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~247463936,251658240~37748736,290354177~124881919] 2021-07-31T02:30:11.441 INFO:tasks.workunit.client.0.smithi045.stdout:from 2 to HEAD diff [0~41943040,46137344~117440512,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~33554432] 2021-07-31T02:30:11.442 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~41943040,46137344~117440512,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~33554432] 2021-07-31T02:30:11.447 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~247463936,251658240~37748736,290354177~124881919] 2021-07-31T02:30:11.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:11 smithi154 conmon[34553]: cluster 2021-07-31T02:30:11.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:11 smithi154 conmon[34553]: 2021-07-31T02:30:10.342567+0000 2021-07-31T02:30:11.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:11 smithi154 conmon[34553]: mon.a (mon.0) 2159 : cluster [DBG] osdmap e1229: 8 total, 8 up, 8 in 2021-07-31T02:30:11.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:11 smithi045 conmon[33655]: cluster 2021-07-31T02:30:10.342567+0000 mon.a (mon.0 2021-07-31T02:30:11.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:11 smithi045 conmon[33655]: ) 2159 : cluster [DBG] osdmap e1229: 8 total, 8 up, 8 in 2021-07-31T02:30:11.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:11 smithi045 conmon[39718]: cluster 2021-07-31T02:30:10.342567+0000 mon.a (mon 2021-07-31T02:30:11.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:11 smithi045 conmon[39718]: .0) 2159 : cluster [DBG] osdmap e1229: 8 total, 8 up, 8 in 2021-07-31T02:30:12.301 INFO:tasks.workunit.client.0.smithi045.stdout:from 0 to HEAD diff [0~163577856,167772160~46137344,218103808~29360128,251658240~67108864,322961408~54525952,381681664~33554432] 2021-07-31T02:30:12.302 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~163577856,167772160~46137344,218103808~29360128,251658240~67108864,322961408~54525952,381681664~33554432] 2021-07-31T02:30:12.369 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~247463936,251658240~163577856] 2021-07-31T02:30:12.370 INFO:tasks.workunit.client.0.smithi045.stdout:from 1 to HEAD diff [0~163577856,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~33554432] 2021-07-31T02:30:12.370 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~163577856,167772160~46137344,218103808~29360128,251658240~33554432,293601280~25165824,322961408~54525952,381681664~33554432] 2021-07-31T02:30:12.379 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~247463936,251658240~37748736,290354177~124881919] 2021-07-31T02:30:12.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:12 smithi154 conmon[34553]: cluster 2021-07-31T02:30:10.412059+0000 mgr.x (mgr.24542) 1838 2021-07-31T02:30:12.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:12 smithi154 conmon[34553]: : cluster [DBG] pgmap v2364: 41 pgs: 41 active+clean; 692 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 6.2 KiB/s rd, 1.2 KiB/s wr, 988 op/s 2021-07-31T02:30:12.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:12 smithi154 conmon[34553]: cluster 2021-07-31T02:30:11.353452+0000 mon.a (mon.0) 2160 : cluster [DBG] osdmap e1230: 8 total, 8 up, 8 in 2021-07-31T02:30:12.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:12 smithi154 conmon[34553]: cluster 2021-07-31T02:30:12.289550+0000 mon.a (mon.0) 2161 : cluster [DBG] osdmap e1231: 8 total, 8 up, 8 in 2021-07-31T02:30:12.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:12 smithi045 conmon[39718]: cluster 2021-07-31T02:30:12.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:12 smithi045 conmon[39718]: 2021-07-31T02:30:12.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:12 smithi045 conmon[39718]: 2021 2021-07-31T02:30:12.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:12 smithi045 conmon[39718]: - 2021-07-31T02:30:12.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:12 smithi045 conmon[39718]: 07 2021-07-31T02:30:12.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:12 smithi045 conmon[39718]: - 2021-07-31T02:30:12.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:12 smithi045 conmon[39718]: 31T02:30:10.412059+0000 mgr.x 2021-07-31T02:30:12.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:12 smithi045 conmon[39718]: (mgr.24542) 1838 : cluster [DBG] pgmap v2364: 41 pgs: 41 active+clean; 692 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 6.2 KiB/s rd, 1.2 KiB/s wr, 988 op/s 2021-07-31T02:30:12.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:12 smithi045 conmon[39718]: cluster 2021-07-31T02:30: 2021-07-31T02:30:12.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:12 smithi045 conmon[39718]: 11.353452+0000 mon.a (mon.0) 2160 : cluster 2021-07-31T02:30:12.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:12 smithi045 conmon[39718]: [DBG] osdmap e1230: 8 total, 8 up, 8 in 2021-07-31T02:30:12.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:12 smithi045 conmon[39718]: cluster 2021-07-31T02: 2021-07-31T02:30:12.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:12 smithi045 conmon[39718]: 30:12.289550+0000 mon.a (mon.0) 2161 : 2021-07-31T02:30:12.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:12 smithi045 conmon[39718]: cluster [DBG] osdmap e1231: 8 total, 8 up, 8 in 2021-07-31T02:30:12.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:12 smithi045 conmon[33655]: cluster 2021-07-31T02:30:12.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:12 smithi045 conmon[33655]: 2021-07-31T02:30:12.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:12 smithi045 conmon[33655]: 2021 2021-07-31T02:30:12.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:12 smithi045 conmon[33655]: - 2021-07-31T02:30:12.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:12 smithi045 conmon[33655]: 07 2021-07-31T02:30:12.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:12 smithi045 conmon[33655]: - 2021-07-31T02:30:12.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:12 smithi045 conmon[33655]: 31T02:30:10.412059+0000 2021-07-31T02:30:12.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:12 smithi045 conmon[33655]: mgr.x (mgr.24542) 1838 : cluster [DBG] pgmap v2364: 41 pgs: 41 active+clean; 692 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 6.2 KiB/s rd, 1.2 KiB/s wr, 988 op/s 2021-07-31T02:30:12.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:12 smithi045 conmon[33655]: cluster 2021-07 2021-07-31T02:30:12.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:12 smithi045 conmon[33655]: -31T02:30:11.353452+0000 2021-07-31T02:30:12.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:12 smithi045 conmon[33655]: mon.a (mon.0) 2160 : cluster [DBG] 2021-07-31T02:30:12.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:12 smithi045 conmon[33655]: osdmap e1230: 8 total, 8 up, 8 in 2021-07-31T02:30:12.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:12 smithi045 conmon[33655]: cluster 2021-07-31T02:30:12.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:12 smithi045 conmon[33655]: 2021-07-31T02:30:12. 2021-07-31T02:30:12.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:12 smithi045 conmon[33655]: 289550+0000 mon.a (mon.0) 2161 : cluster 2021-07-31T02:30:12.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:12 smithi045 conmon[33655]: [DBG] osdmap e1231: 8 total, 8 up, 8 in 2021-07-31T02:30:13.372 INFO:tasks.workunit.client.0.smithi045.stdout:from 0 to HEAD diff [0~163577856,167772160~46137344,218103808~29360128,251658240~67108864,322961408~54525952,381681664~33554432] 2021-07-31T02:30:13.372 INFO:tasks.workunit.client.0.smithi045.stdout: limited diff [0~163577856,167772160~46137344,218103808~29360128,251658240~67108864,322961408~54525952,381681664~33554432] 2021-07-31T02:30:13.431 INFO:tasks.workunit.client.0.smithi045.stdout: actual was [0~247463936,251658240~163577856] 2021-07-31T02:30:13.832 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:30:14.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:30:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:30:13.583Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:30:14.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:30:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:30:13.583Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:30:14.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:30:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:30:13.583Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:30:14.109 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:30:14.110 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 45m ago 45m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:30:14.110 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 42m ago 45m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:30:14.110 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 42m ago 49m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:30:14.110 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 45m ago 51m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:30:14.110 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 45m ago 51m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:30:14.111 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 42m ago 49m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:30:14.111 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 45m ago 49m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:30:14.111 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 45m ago 46m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:30:14.111 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 42m ago 46m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:30:14.112 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 45m ago 48m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:30:14.112 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 45m ago 48m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:30:14.112 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 45m ago 48m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:30:14.112 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 45m ago 47m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:30:14.112 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 42m ago 47m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:30:14.113 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 42m ago 47m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:30:14.113 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 42m ago 46m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:30:14.113 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 42m ago 46m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:30:14.113 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 42m ago 46m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:30:14.374 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] DiffIterateTest/1.DiffIterateStress (39943 ms) 2021-07-31T02:30:14.375 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] DiffIterateTest/1.DiffIterateRegression6926 2021-07-31T02:30:14.375 INFO:tasks.workunit.client.0.smithi045.stdout:using old format 2021-07-31T02:30:14.468 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:30:14.469 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:30:14.469 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:30:14.469 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:30:14.469 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:30:14.470 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:30:14.470 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:30:14.470 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:30:14.470 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:30:14.471 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:30:14.471 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:30:14.471 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:30:14.471 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:30:14.471 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:30:14.472 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:30:14.472 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:30:14.472 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:30:14.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:14 smithi154 conmon[34553]: cluster 2021-07-31T02:30:12.412640+0000 mgr.x (mgr.24542) 1839 : cluster [DBG] pgmap v2367: 41 pgs: 41 active+clean; 692 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 6.2 KiB/s rd, 1.2 KiB/s wr, 988 op/s 2021-07-31T02:30:14.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:14 smithi154 conmon[34553]: cluster 2021-07-31T02:30:13.359645+0000 mon.a (mon.0) 2162 : cluster [DBG] osdmap e1232: 8 total, 8 up, 8 in 2021-07-31T02:30:14.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:14 smithi045 conmon[33655]: cluster 2021-07-31T02:30:12.412640+0000 mgr.x (mgr.24542 2021-07-31T02:30:14.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:14 smithi045 conmon[33655]: ) 1839 : cluster [DBG] pgmap v2367: 41 pgs: 41 active+clean; 692 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 6.2 KiB/s rd, 1.2 KiB/s wr, 988 op/s 2021-07-31T02:30:14.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:14 smithi045 conmon[33655]: cluster 2021-07-31T02:30:13.359645+0000 mon.a (mon.0) 2162 : cluster [DBG] osdmap e1232: 8 total, 8 up, 8 in 2021-07-31T02:30:14.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:14 smithi045 conmon[39718]: cluster 2021-07-31T02:30:12 2021-07-31T02:30:14.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:14 smithi045 conmon[39718]: .412640+0000 mgr.x (mgr.24542) 1839 : cluster [DBG] pgmap v2367: 41 pgs: 41 active+clean; 692 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 6.2 KiB/s rd, 1.2 KiB/s wr, 988 op/s 2021-07-31T02:30:14.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:14 smithi045 conmon[39718]: cluster 2021-07-31T02:30:14.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:14 smithi045 conmon[39718]: 2021-07-31T02:30:13.359645 2021-07-31T02:30:14.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:14 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2162 : cluster [DBG] 2021-07-31T02:30:14.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:14 smithi045 conmon[39718]: osdmap e1232: 8 total, 8 up, 8 in 2021-07-31T02:30:15.372 INFO:tasks.workunit.client.0.smithi045.stdout:iterate_cb 0~4194304 2021-07-31T02:30:15.377 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] DiffIterateTest/1.DiffIterateRegression6926 (1002 ms) 2021-07-31T02:30:15.377 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] DiffIterateTest/1.DiffIterateIgnoreParent 2021-07-31T02:30:15.377 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:30:15.378 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] DiffIterateTest/1.DiffIterateIgnoreParent (0 ms) 2021-07-31T02:30:15.378 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] DiffIterateTest/1.DiffIterateCallbackError 2021-07-31T02:30:15.378 INFO:tasks.workunit.client.0.smithi045.stdout:using old format 2021-07-31T02:30:15.387 INFO:tasks.workunit.client.0.smithi045.stdout: wrote [1489764~101475,2111845~91974,3347652~29993,4624065~71590,5599268~40999,9208067~24919,13882442~7028,16938625~57736,18552989~54212,20416531~42455] 2021-07-31T02:30:15.395 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] DiffIterateTest/1.DiffIterateCallbackError (19 ms) 2021-07-31T02:30:15.395 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] DiffIterateTest/1.DiffIterateParentDiscard 2021-07-31T02:30:15.396 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:30:15.396 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] DiffIterateTest/1.DiffIterateParentDiscard (0 ms) 2021-07-31T02:30:15.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:15 smithi154 conmon[34553]: audit 2021-07-31T02:30:13.820994+0000 mgr.x (mgr.24542) 1840 : audit [DBG] 2021-07-31T02:30:15.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:15 smithi154 conmon[34553]: from='client.27178 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:30:15.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:15 smithi154 conmon[34553]: audit 2021-07-31T02:30:14.101909+0000 mgr.x (mgr.24542) 1841 : audit [DBG] from='client.17247 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:30:15.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:15 smithi154 conmon[34553]: cluster 2021-07-31T02:30:14.361496+0000 mon.a (mon.0) 2163 : cluster 2021-07-31T02:30:15.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:15 smithi154 conmon[34553]: [DBG] osdmap e1233: 8 total, 8 up, 8 in 2021-07-31T02:30:15.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:15 smithi154 conmon[34553]: cluster 2021-07-31T02:30:15.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:15 smithi154 conmon[34553]: 2021-07-31T02:30:14.413077+0000 mgr.x (mgr.24542 2021-07-31T02:30:15.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:15 smithi154 conmon[34553]: ) 1842 : cluster [DBG] pgmap v2370: 41 pgs: 41 active+clean; 475 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 6.2 KiB/s rd, 1.2 KiB/s wr, 376 op/s 2021-07-31T02:30:15.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:15 smithi154 conmon[34553]: audit 2021-07-31T02:30:14.469055+0000 2021-07-31T02:30:15.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:15 smithi154 conmon[34553]: mon.c (mon.1) 104 : audit [DBG] from='client.? 172.21.15.45:0/2848500294' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:30:15.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:15 smithi045 conmon[33655]: audit 2021-07-31T02:30:13.820994+0000 mgr.x (mgr.24542) 1840 : audit [DBG] from='client.27178 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:30:15.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:15 smithi045 conmon[33655]: audit 2021-07-31T02:30:14.101909+0000 mgr.x (mgr.24542) 1841 : audit [DBG] from='client.17247 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:30:15.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:15 smithi045 conmon[33655]: cluster 2021-07-31T02:30:14.361496+0000 mon.a (mon.0) 2163 : cluster [DBG] osdmap e1233: 8 total, 8 up, 8 in 2021-07-31T02:30:15.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:15 smithi045 conmon[33655]: cluster 2021-07-31T02:30:14.413077+0000 mgr.x (mgr.24542) 1842 : cluster [DBG] pgmap v2370: 41 pgs: 41 active+clean; 475 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 6.2 KiB/s rd, 1.2 KiB/s wr, 376 op/s 2021-07-31T02:30:15.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:15 smithi045 conmon[33655]: audit 2021-07-31T02:30:14.469055+0000 mon.c (mon.1) 104 : audit [DBG] from='client.? 172.21.15.45:0/2848500294' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:30:15.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:15 smithi045 conmon[39718]: audit 2021-07-31T02:30:13.820994+0000 mgr.x (mgr.24542) 1840 : audit 2021-07-31T02:30:15.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:15 smithi045 conmon[39718]: [DBG] from='client.27178 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:30:15.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:15 smithi045 conmon[39718]: audit 2021-07-31T02:30:14.101909+0000 mgr.x (mgr.24542 2021-07-31T02:30:15.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:15 smithi045 conmon[39718]: ) 1841 : audit [DBG] from='client.17247 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:30:15.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:15 smithi045 conmon[39718]: cluster 2021-07-31T02:30:15.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:15 smithi045 conmon[39718]: 2021-07-31T02:30:14.361496+0000 mon.a (mon.0 2021-07-31T02:30:15.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:15 smithi045 conmon[39718]: ) 2163 : cluster [DBG] osdmap e1233: 8 total, 8 up, 8 in 2021-07-31T02:30:15.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:15 smithi045 conmon[39718]: cluster 2021-07-31T02 2021-07-31T02:30:15.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:15 smithi045 conmon[39718]: :30:14.413077+0000 mgr.x (mgr.24542) 1842 : 2021-07-31T02:30:15.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:15 smithi045 conmon[39718]: cluster [DBG] pgmap v2370: 41 pgs: 41 active+clean; 475 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 6.2 KiB/s rd, 1.2 KiB/s wr, 376 op/s 2021-07-31T02:30:15.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:15 smithi045 conmon[39718]: audit 2021-07-31T02:30:14.469055+0000 mon.c (mon.1 2021-07-31T02:30:15.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:15 smithi045 conmon[39718]: ) 104 : audit [DBG] from='client.? 172.21.15.45:0/2848500294' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:30:16.377 INFO:tasks.workunit.client.0.smithi045.stdout:[----------] 7 tests from DiffIterateTest/1 (49000 ms total) 2021-07-31T02:30:16.422 INFO:tasks.workunit.client.0.smithi045.stdout: 2021-07-31T02:30:16.422 INFO:tasks.workunit.client.0.smithi045.stdout:[----------] 16 tests from TestImageWatcher 2021-07-31T02:30:16.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:16 smithi154 conmon[34553]: cluster 2021-07-31T02:30:15.366019+0000 mon.a (mon.0) 2164 : cluster [DBG] osdmap e1234: 8 total, 8 up, 8 in 2021-07-31T02:30:16.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:16 smithi045 conmon[39718]: cluster 2021-07-31T02:30:15.366019+0000 mon.a (mon.0) 2164 : cluster [DBG] osdmap e1234: 8 total, 8 up, 8 in 2021-07-31T02:30:16.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:16 smithi045 conmon[33655]: cluster 2021-07-31T02:30:15.366019+0000 mon.a (mon.0) 2164 : cluster 2021-07-31T02:30:16.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:16 smithi045 conmon[33655]: [DBG] osdmap e1234: 8 total, 8 up, 8 in 2021-07-31T02:30:17.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:17 smithi154 conmon[34553]: cluster 2021-07-31T02:30:17.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:17 smithi154 conmon[34553]: 2021-07-31T02:30 2021-07-31T02:30:17.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:17 smithi154 conmon[34553]: :16.372316+0000 mon.a (mon.0) 2165 : cluster [DBG] osdmap e1235: 8 total, 8 up, 8 in 2021-07-31T02:30:17.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:17 smithi154 conmon[34553]: cluster 2021-07-31T02:30:17.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:17 smithi154 conmon[34553]: 2021-07-31T02:30:16.413530+0000 mgr.x (mgr.24542) 1843 2021-07-31T02:30:17.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:17 smithi154 conmon[34553]: : cluster [DBG] pgmap v2373: 9 pgs: 9 active+clean; 4.6 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail 2021-07-31T02:30:17.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:17 smithi045 conmon[39718]: cluster 2021-07-31T02:30:16. 2021-07-31T02:30:17.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:17 smithi045 conmon[39718]: 372316+0000 mon.a (mon.0) 2165 : cluster [DBG] osdmap e1235: 8 total, 8 up, 8 in 2021-07-31T02:30:17.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:17 smithi045 conmon[39718]: cluster 2021-07-31T02:30:17.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:17 smithi045 conmon[39718]: 2021-07-31T02:30:16.413530+0000 mgr.x (mgr.24542) 2021-07-31T02:30:17.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:17 smithi045 conmon[39718]: 1843 : cluster [DBG] pgmap v2373: 9 pgs: 9 active+clean; 4.6 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail 2021-07-31T02:30:17.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:17 smithi045 conmon[33655]: cluster 2021-07-31T02:30:16.372316+0000 mon.a (mon.0) 2021-07-31T02:30:17.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:17 smithi045 conmon[33655]: 2165 : cluster [DBG] osdmap e1235: 8 total, 8 up, 8 in 2021-07-31T02:30:17.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:17 smithi045 conmon[33655]: cluster 2021-07-31T02:30:16.413530+0000 mgr.x (mgr.24542) 1843 : cluster [DBG] pgmap v2373: 9 pgs: 9 active+clean; 4.6 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail 2021-07-31T02:30:18.381 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestImageWatcher.NotifyHeaderUpdate 2021-07-31T02:30:18.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:18 smithi154 conmon[34553]: cluster 2021-07-31T02:30:17.377714+0000 mon.a (mon.0) 2166 : cluster [DBG] osdmap e1236: 8 total, 8 up, 8 in 2021-07-31T02:30:18.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:18 smithi154 conmon[34553]: audit 2021-07-31T02:30:17.380564+0000 mon.c (mon.1) 105 : audit [INF] from='client.? 172.21.15.45:0/1419107993' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:30:18.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:18 smithi154 conmon[34553]: audit 2021-07-31T02:30:17.384925+0000 mon.a (mon.0) 2167 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:30:18.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:18 smithi045 conmon[33655]: cluster 2021-07-31T02:30:17.377714+0000 mon.a ( 2021-07-31T02:30:18.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:18 smithi045 conmon[33655]: mon.0) 2166 : cluster [DBG] osdmap e1236: 8 total, 8 up, 8 in 2021-07-31T02:30:18.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:18 smithi045 conmon[33655]: audit 2021-07-31T02:30:17.380564+0000 mon.c (mon.1) 105 : audit [INF] from='client.? 172.21.15.45:0/1419107993' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:30:18.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:18 smithi045 conmon[33655]: audit 2021-07-31T02:30:17.384925+0000 mon.a (mon.0) 2167 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:30:18.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:18 smithi045 conmon[39718]: cluster 2021-07-31T02:30:17.377714+0000 mon.a (mon.0) 2166 : cluster [DBG] osdmap e1236: 8 total, 8 up, 8 in 2021-07-31T02:30:18.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:18 smithi045 conmon[39718]: audit 2021-07-31T02:30:17.380564+0000 mon.c (mon.1) 105 : audit [INF] from='client.? 172.21.15.45:0/1419107993' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:30:18.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:18 smithi045 conmon[39718]: audit 2021-07-31T02:30:17.384925+0000 mon.a (mon.0) 2167 : audit [INF] 2021-07-31T02:30:18.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:18 smithi045 conmon[39718]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:30:19.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:19 smithi154 conmon[34553]: audit 2021-07-31T02:30:18.379835+0000 mon.a (mon.0) 2168 : audit [INF] 2021-07-31T02:30:19.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:19 smithi154 conmon[34553]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-8","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:30:19.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:19 smithi154 conmon[34553]: cluster 2021-07-31T02:30:18.380071+0000 mon.a (mon.0) 2169 : cluster [DBG] osdmap e1237: 8 total, 8 up, 8 in 2021-07-31T02:30:19.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:19 smithi154 conmon[34553]: cluster 2021-07-31T02: 2021-07-31T02:30:19.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:19 smithi154 conmon[34553]: 30:18.414119+0000 mgr.x (mgr.24542) 1844 : cluster [DBG] pgmap v2376: 41 pgs: 32 unknown, 9 active+clean; 4.6 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail 2021-07-31T02:30:19.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:19 smithi045 conmon[33655]: audit 2021-07-31T02:30:18.379835+0000 mon.a (mon.0) 2168 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-8","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:30:19.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:19 smithi045 conmon[33655]: cluster 2021-07-31T02:30:18.380071+0000 mon.a (mon.0) 2169 : cluster [DBG] osdmap e1237: 8 total, 8 up, 8 in 2021-07-31T02:30:19.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:19 smithi045 conmon[33655]: cluster 2021-07-31T02:30:18.414119+0000 mgr.x (mgr.24542) 1844 : cluster [DBG] pgmap v2376: 41 pgs: 32 unknown, 9 active+clean; 4.6 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail 2021-07-31T02:30:19.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:19 smithi045 conmon[39718]: audit 2021-07-31T02:30:18.379835+0000 mon.a (mon 2021-07-31T02:30:19.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:19 smithi045 conmon[39718]: .0) 2168 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-8","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:30:19.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:19 smithi045 conmon[39718]: cluster 2021-07-31T02:30:18.380071 2021-07-31T02:30:19.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:19 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2169 : cluster [DBG] osdmap e1237: 8 total, 8 up, 8 in 2021-07-31T02:30:19.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:19 smithi045 conmon[39718]: cluster 2021-07 2021-07-31T02:30:19.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:19 smithi045 conmon[39718]: -31T02:30:18.414119+0000 mgr.x (mgr. 2021-07-31T02:30:19.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:19 smithi045 conmon[39718]: 24542) 1844 : cluster [DBG] pgmap v2376: 41 pgs: 32 unknown, 9 active+clean; 4.6 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail 2021-07-31T02:30:20.406 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:30:20.406 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestImageWatcher.NotifyHeaderUpdate (2025 ms) 2021-07-31T02:30:20.406 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestImageWatcher.NotifyFlatten 2021-07-31T02:30:20.413 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:30:20.414 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestImageWatcher.NotifyFlatten (8 ms) 2021-07-31T02:30:20.414 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestImageWatcher.NotifyResize 2021-07-31T02:30:20.420 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:30:20.421 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestImageWatcher.NotifyResize (7 ms) 2021-07-31T02:30:20.421 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestImageWatcher.NotifyRebuildObjectMap 2021-07-31T02:30:20.424 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:30:20.425 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestImageWatcher.NotifyRebuildObjectMap (4 ms) 2021-07-31T02:30:20.425 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestImageWatcher.NotifySnapCreate 2021-07-31T02:30:20.428 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:30:20.428 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestImageWatcher.NotifySnapCreate (4 ms) 2021-07-31T02:30:20.428 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestImageWatcher.NotifySnapCreateError 2021-07-31T02:30:20.432 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:30:20.432 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestImageWatcher.NotifySnapCreateError (3 ms) 2021-07-31T02:30:20.432 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestImageWatcher.NotifySnapRename 2021-07-31T02:30:20.436 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:30:20.436 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestImageWatcher.NotifySnapRename (4 ms) 2021-07-31T02:30:20.436 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestImageWatcher.NotifySnapRenameError 2021-07-31T02:30:20.439 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:30:20.440 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestImageWatcher.NotifySnapRenameError (4 ms) 2021-07-31T02:30:20.440 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestImageWatcher.NotifySnapRemove 2021-07-31T02:30:20.443 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:30:20.443 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestImageWatcher.NotifySnapRemove (4 ms) 2021-07-31T02:30:20.444 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestImageWatcher.NotifySnapProtect 2021-07-31T02:30:20.446 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:30:20.447 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestImageWatcher.NotifySnapProtect (3 ms) 2021-07-31T02:30:20.447 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestImageWatcher.NotifySnapUnprotect 2021-07-31T02:30:20.450 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:30:20.450 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestImageWatcher.NotifySnapUnprotect (4 ms) 2021-07-31T02:30:20.451 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestImageWatcher.NotifyRename 2021-07-31T02:30:20.453 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:30:20.454 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestImageWatcher.NotifyRename (3 ms) 2021-07-31T02:30:20.454 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestImageWatcher.NotifyAsyncTimedOut 2021-07-31T02:30:20.457 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:30:20.457 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestImageWatcher.NotifyAsyncTimedOut (3 ms) 2021-07-31T02:30:20.457 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestImageWatcher.NotifyAsyncError 2021-07-31T02:30:20.460 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:30:20.460 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestImageWatcher.NotifyAsyncError (4 ms) 2021-07-31T02:30:20.461 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestImageWatcher.NotifyAsyncCompleteError 2021-07-31T02:30:20.463 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:30:20.464 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestImageWatcher.NotifyAsyncCompleteError (3 ms) 2021-07-31T02:30:20.464 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestImageWatcher.NotifyAsyncRequestTimedOut 2021-07-31T02:30:20.466 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:30:20.467 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestImageWatcher.NotifyAsyncRequestTimedOut (3 ms) 2021-07-31T02:30:20.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:20 smithi154 conmon[34553]: cluster 2021-07-31T02:30:19.391590+0000 mon.a (mon.0) 2170 : cluster [DBG] 2021-07-31T02:30:20.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:20 smithi154 conmon[34553]: osdmap e1238: 8 total, 8 up, 8 in 2021-07-31T02:30:20.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:20 smithi045 conmon[33655]: cluster 2021-07-31T02:30:19.391590 2021-07-31T02:30:20.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:20 smithi045 conmon[33655]: +0000 mon.a (mon.0) 2170 : cluster [DBG] osdmap e1238: 8 total, 8 up, 8 in 2021-07-31T02:30:20.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:20 smithi045 conmon[39718]: cluster 2021-07-31T02:30:19.391590+0000 mon.a ( 2021-07-31T02:30:20.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:20 smithi045 conmon[39718]: mon.0) 2170 : cluster [DBG] osdmap e1238: 8 total, 8 up, 8 in 2021-07-31T02:30:21.408 INFO:tasks.workunit.client.0.smithi045.stdout:[----------] 16 tests from TestImageWatcher (2086 ms total) 2021-07-31T02:30:21.409 INFO:tasks.workunit.client.0.smithi045.stdout: 2021-07-31T02:30:21.409 INFO:tasks.workunit.client.0.smithi045.stdout:[----------] 32 tests from TestInternal 2021-07-31T02:30:21.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:21 smithi154 conmon[34553]: cluster 2021-07-31T02:30:20.397581+0000 mon.a (mon.0) 2171 : cluster [DBG] osdmap e1239: 8 total, 8 up, 8 in 2021-07-31T02:30:21.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:21 smithi154 conmon[34553]: cluster 2021-07-31T02:30:20.414492+0000 mgr.x (mgr.24542) 1845 : cluster [DBG] pgmap v2379: 41 pgs: 41 active+clean; 4.6 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:30:21.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:21 smithi045 conmon[33655]: cluster 2021-07-31T02:30:20.397581+0000 mon.a (mon.0) 2171 : cluster 2021-07-31T02:30:21.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:21 smithi045 conmon[33655]: [DBG] osdmap e1239: 8 total, 8 up, 8 in 2021-07-31T02:30:21.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:21 smithi045 conmon[33655]: cluster 2021-07-31T02:30:20.414492+0000 mgr.x (mgr.24542) 1845 : cluster [DBG] pgmap v2379: 41 pgs: 41 active+clean; 4.6 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:30:21.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:21 smithi045 conmon[39718]: cluster 2021-07-31T02:30:20. 2021-07-31T02:30:21.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:21 smithi045 conmon[39718]: 397581+0000 mon.a (mon.0) 2171 : cluster [DBG] osdmap e1239: 8 total, 8 up, 8 in 2021-07-31T02:30:21.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:21 smithi045 conmon[39718]: cluster 2021-07-31T02:30:21.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:21 smithi045 conmon[39718]: 2021-07-31 2021-07-31T02:30:21.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:21 smithi045 conmon[39718]: T02:30:20.414492+0000 mgr.x (mgr.24542) 2021-07-31T02:30:21.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:21 smithi045 conmon[39718]: 1845 : cluster [DBG] pgmap v2379: 41 pgs: 41 active+clean; 4.6 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:30:22.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:22 smithi154 conmon[34553]: cluster 2021-07-31T02:30:21.405456+0000 mon.a (mon.0) 2172 : cluster [DBG] osdmap e1240: 8 total, 8 up, 8 in 2021-07-31T02:30:22.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:22 smithi154 conmon[34553]: 2021-07-31T02:30:22.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:22 smithi045 conmon[33655]: cluster 2021-07-31T02 2021-07-31T02:30:22.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:22 smithi045 conmon[33655]: :30:21.405456+0000 mon.a (mon.0) 2172 : cluster [DBG] osdmap e1240: 8 total, 8 up, 8 in 2021-07-31T02:30:22.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:22 smithi045 conmon[39718]: cluster 2021-07-31T02:30: 2021-07-31T02:30:22.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:22 smithi045 conmon[39718]: 21.405456+0000 mon.a (mon.0) 2172 : cluster [DBG] osdmap e1240: 8 total, 8 up, 8 in 2021-07-31T02:30:23.418 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestInternal.OpenByID 2021-07-31T02:30:23.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:23 smithi154 conmon[34553]: cluster 2021-07-31T02:30:22.414904+0000 mgr.x (mgr.24542) 1846 : cluster [DBG] pgmap v2382: 41 pgs: 32 unknown, 9 active+clean; 4.6 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:30:23.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:23 smithi154 conmon[34553]: cluster 2021-07-31T02:30:22.414979+0000 mon.a (mon.0) 2173 : cluster [DBG] osdmap e1241: 8 total, 8 up, 8 in 2021-07-31T02:30:23.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:23 smithi154 conmon[34553]: audit 2021-07-31T02:30:22.416026+0000 mon.c (mon.1) 106 : audit [INF] from='client.? 172.21.15.45:0/1803396249' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:30:23.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:23 smithi154 conmon[34553]: audit 2021-07-31T02:30:22.416952+0000 mon.a (mon.0) 2174 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:30:23.775 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:30:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:30:23.584Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:30:23.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:30:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:30:23.584Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:30:23.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:30:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:30:23.584Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:30:23.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:23 smithi045 conmon[33655]: cluster 2021-07-31T02:30:22.414904+0000 mgr.x (mgr.24542) 1846 : cluster [DBG] pgmap v2382: 41 pgs: 32 unknown, 9 active+clean; 4.6 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:30:23.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:23 smithi045 conmon[33655]: cluster 2021-07-31T02:30:22.414979+0000 mon.a (mon.0) 2173 : cluster [DBG] osdmap e1241: 8 total, 8 up, 8 in 2021-07-31T02:30:23.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:23 smithi045 conmon[33655]: audit 2021-07-31T02:30:22.416026+0000 mon.c (mon.1) 106 : audit [INF] from='client.? 172.21.15.45:0/1803396249' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:30:23.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:23 smithi045 conmon[33655]: audit 2021-07-31T02:30:22.416952+0000 mon.a (mon.0) 2174 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:30:23.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:23 smithi045 conmon[39718]: cluster 2021-07-31T02:30:22. 2021-07-31T02:30:23.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:23 smithi045 conmon[39718]: 414904+0000 mgr.x (mgr.24542) 1846 : cluster [DBG] pgmap v2382: 41 pgs: 32 unknown, 9 active+clean; 4.6 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:30:23.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:23 smithi045 conmon[39718]: cluster 2021-07-31T02:30: 2021-07-31T02:30:23.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:23 smithi045 conmon[39718]: 22.414979+0000 mon.a (mon.0) 2173 : cluster [DBG] osdmap e1241: 8 total, 8 up, 8 in 2021-07-31T02:30:23.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:23 smithi045 conmon[39718]: audit 2021-07-31T02:30:23.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:23 smithi045 conmon[39718]: 2021-07-31T02:30:22. 2021-07-31T02:30:23.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:23 smithi045 conmon[39718]: 416026+0000 mon.c (mon.1) 106 : audit 2021-07-31T02:30:23.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:23 smithi045 conmon[39718]: [INF] 2021-07-31T02:30:23.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:23 smithi045 conmon[39718]: from='client.? 172.21.15.45:0/1803396249' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:30:23.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:23 smithi045 conmon[39718]: audit 2021-07-31T02:30:23.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:23 smithi045 conmon[39718]: 2021-07-31T02:30:22.416952 2021-07-31T02:30:23.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:23 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2174 : audit [INF] 2021-07-31T02:30:23.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:23 smithi045 conmon[39718]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:30:24.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:24 smithi154 conmon[34553]: audit 2021-07-31T02:30:23.417487+0000 mon.a (mon.0) 2175 2021-07-31T02:30:24.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:24 smithi154 conmon[34553]: : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-9","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:30:24.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:24 smithi154 conmon[34553]: cluster 2021-07-31T02:30:23.417781+0000 mon.a (mon.0) 2176 : cluster [DBG] osdmap e1242: 8 total, 8 up, 8 in 2021-07-31T02:30:24.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:24 smithi045 conmon[39718]: audit 2021-07-31T02:30:24.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:24 smithi045 conmon[39718]: 2021-07-31T02:30:23.417487+0000 mon.a (mon.0) 2175 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-9","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:30:24.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:24 smithi045 conmon[39718]: cluster 2021-07-31 2021-07-31T02:30:24.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:24 smithi045 conmon[39718]: T02:30:23.417781+0000 mon.a (mon.0) 2176 : cluster [DBG] 2021-07-31T02:30:24.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:24 smithi045 conmon[39718]: osdmap e1242: 8 total, 8 up, 8 in 2021-07-31T02:30:24.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:24 smithi045 conmon[33655]: audit 2021-07-31T02:30:23.417487+0000 mon.a (mon.0) 2175 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-9","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:30:24.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:24 smithi045 conmon[33655]: cluster 2021-07-31T02:30:23.417781+0000 mon.a (mon.0) 2176 : cluster [DBG] osdmap e1242: 8 total, 8 up, 8 in 2021-07-31T02:30:25.450 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:30:25.450 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestInternal.OpenByID (2032 ms) 2021-07-31T02:30:25.451 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestInternal.OpenSnapDNE 2021-07-31T02:30:25.468 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestInternal.OpenSnapDNE (19 ms) 2021-07-31T02:30:25.468 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestInternal.IsExclusiveLockOwner 2021-07-31T02:30:25.473 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:30:25.473 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestInternal.IsExclusiveLockOwner (4 ms) 2021-07-31T02:30:25.473 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestInternal.ResizeLocksImage 2021-07-31T02:30:25.477 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:30:25.477 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestInternal.ResizeLocksImage (4 ms) 2021-07-31T02:30:25.477 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestInternal.ResizeFailsToLockImage 2021-07-31T02:30:25.480 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:30:25.481 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestInternal.ResizeFailsToLockImage (4 ms) 2021-07-31T02:30:25.481 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestInternal.SnapCreateLocksImage 2021-07-31T02:30:25.483 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:30:25.484 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestInternal.SnapCreateLocksImage (3 ms) 2021-07-31T02:30:25.484 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestInternal.SnapCreateFailsToLockImage 2021-07-31T02:30:25.487 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:30:25.488 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestInternal.SnapCreateFailsToLockImage (4 ms) 2021-07-31T02:30:25.488 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestInternal.SnapRollbackLocksImage 2021-07-31T02:30:25.491 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:30:25.491 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestInternal.SnapRollbackLocksImage (3 ms) 2021-07-31T02:30:25.491 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestInternal.SnapRollbackFailsToLockImage 2021-07-31T02:30:25.494 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:30:25.494 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestInternal.SnapRollbackFailsToLockImage (4 ms) 2021-07-31T02:30:25.495 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestInternal.SnapSetReleasesLock 2021-07-31T02:30:25.498 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:30:25.498 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestInternal.SnapSetReleasesLock (4 ms) 2021-07-31T02:30:25.499 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestInternal.FlattenLocksImage 2021-07-31T02:30:25.502 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:30:25.502 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestInternal.FlattenLocksImage (3 ms) 2021-07-31T02:30:25.502 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestInternal.FlattenFailsToLockImage 2021-07-31T02:30:25.505 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:30:25.505 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestInternal.FlattenFailsToLockImage (4 ms) 2021-07-31T02:30:25.506 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestInternal.AioWriteRequestsLock 2021-07-31T02:30:25.508 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:30:25.509 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestInternal.AioWriteRequestsLock (3 ms) 2021-07-31T02:30:25.509 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestInternal.AioDiscardRequestsLock 2021-07-31T02:30:25.512 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:30:25.512 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestInternal.AioDiscardRequestsLock (4 ms) 2021-07-31T02:30:25.512 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestInternal.CancelAsyncResize 2021-07-31T02:30:25.515 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:30:25.516 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestInternal.CancelAsyncResize (3 ms) 2021-07-31T02:30:25.516 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestInternal.MultipleResize 2021-07-31T02:30:25.552 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestInternal.MultipleResize (37 ms) 2021-07-31T02:30:25.553 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestInternal.Metadata 2021-07-31T02:30:25.556 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:30:25.556 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestInternal.Metadata (4 ms) 2021-07-31T02:30:25.556 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestInternal.MetadataConfApply 2021-07-31T02:30:25.559 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:30:25.560 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestInternal.MetadataConfApply (3 ms) 2021-07-31T02:30:25.560 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestInternal.SnapshotCopyup 2021-07-31T02:30:25.563 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:30:25.563 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestInternal.SnapshotCopyup (3 ms) 2021-07-31T02:30:25.563 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestInternal.SnapshotCopyupZeros 2021-07-31T02:30:25.566 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:30:25.566 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestInternal.SnapshotCopyupZeros (3 ms) 2021-07-31T02:30:25.566 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestInternal.SnapshotCopyupZerosMigration 2021-07-31T02:30:25.569 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:30:25.569 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestInternal.SnapshotCopyupZerosMigration (3 ms) 2021-07-31T02:30:25.569 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestInternal.ResizeCopyup 2021-07-31T02:30:25.571 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:30:25.572 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestInternal.ResizeCopyup (3 ms) 2021-07-31T02:30:25.574 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestInternal.DiscardCopyup 2021-07-31T02:30:25.575 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:30:25.575 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestInternal.DiscardCopyup (3 ms) 2021-07-31T02:30:25.576 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestInternal.ImageOptions 2021-07-31T02:30:25.578 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestInternal.ImageOptions (4 ms) 2021-07-31T02:30:25.578 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestInternal.WriteFullCopyup 2021-07-31T02:30:25.581 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:30:25.582 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestInternal.WriteFullCopyup (3 ms) 2021-07-31T02:30:25.582 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestInternal.DiffIterateCloneOverwrite 2021-07-31T02:30:25.584 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:30:25.584 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestInternal.DiffIterateCloneOverwrite (2 ms) 2021-07-31T02:30:25.585 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestInternal.TestCoR 2021-07-31T02:30:25.587 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:30:25.587 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestInternal.TestCoR (3 ms) 2021-07-31T02:30:25.587 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestInternal.FlattenNoEmptyObjects 2021-07-31T02:30:25.590 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:30:25.590 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestInternal.FlattenNoEmptyObjects (3 ms) 2021-07-31T02:30:25.590 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestInternal.PoolMetadataConfApply 2021-07-31T02:30:25.592 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:30:25.593 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestInternal.PoolMetadataConfApply (3 ms) 2021-07-31T02:30:25.593 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestInternal.Sparsify 2021-07-31T02:30:25.609 INFO:tasks.workunit.client.0.smithi045.stdout:sparsify_supported=1 2021-07-31T02:30:25.609 INFO:tasks.workunit.client.0.smithi045.stdout:sparse_read_supported=1 2021-07-31T02:30:25.624 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestInternal.Sparsify (32 ms) 2021-07-31T02:30:25.624 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestInternal.SparsifyClone 2021-07-31T02:30:25.627 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:30:25.627 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestInternal.SparsifyClone (3 ms) 2021-07-31T02:30:25.628 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestInternal.MissingDataPool 2021-07-31T02:30:25.630 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:30:25.630 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestInternal.MissingDataPool (3 ms) 2021-07-31T02:30:25.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:25 smithi154 conmon[34553]: cluster 2021-07-31T02:30:24.415464+0000 mgr.x (mgr.24542) 1847 2021-07-31T02:30:25.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:25 smithi154 conmon[34553]: : cluster [DBG] pgmap v2384: 41 pgs: 41 active+clean; 4.6 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:30:25.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:25 smithi154 conmon[34553]: cluster 2021-07-31T02:30:24.433770+0000 mon.a (mon.0) 2177 : cluster [DBG] osdmap e1243: 8 total, 8 up, 8 in 2021-07-31T02:30:25.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:25 smithi045 conmon[39718]: cluster 2021-07-31T02:30:25.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:25 smithi045 conmon[39718]: 2021-07-31T02:30 2021-07-31T02:30:25.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:25 smithi045 conmon[39718]: :24.415464+0000 mgr.x (mgr.24542) 1847 : cluster [DBG] pgmap v2384: 41 pgs: 41 active+clean; 4.6 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:30:25.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:25 smithi045 conmon[39718]: cluster 2021-07-31T02:30:25.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:25 smithi045 conmon[39718]: 2021-07- 2021-07-31T02:30:25.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:25 smithi045 conmon[39718]: 31T02:30:24.433770+0000 mon.a (mon.0) 2177 2021-07-31T02:30:25.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:25 smithi045 conmon[39718]: : cluster [DBG] osdmap e1243: 8 total, 8 up, 8 in 2021-07-31T02:30:25.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:25 smithi045 conmon[33655]: cluster 2021-07-31T02:30:25.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:25 smithi045 conmon[33655]: 2021-07-31T02:30:24.415464 2021-07-31T02:30:25.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:25 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 1847 : cluster [DBG] pgmap v2384: 41 pgs: 41 active+clean; 4.6 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:30:25.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:25 smithi045 conmon[33655]: cluster 2021-07-31T02:30:25.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:25 smithi045 conmon[33655]: 2021-07-31T02:30:24.433770+0000 mon.a (mon.0) 2177 : cluster [DBG] osdmap e1243: 8 total, 8 up, 8 in 2021-07-31T02:30:26.448 INFO:tasks.workunit.client.0.smithi045.stdout:[----------] 32 tests from TestInternal (2213 ms total) 2021-07-31T02:30:26.449 INFO:tasks.workunit.client.0.smithi045.stdout: 2021-07-31T02:30:26.449 INFO:tasks.workunit.client.0.smithi045.stdout:[----------] 51 tests from TestMirroring 2021-07-31T02:30:26.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:26 smithi154 conmon[34553]: cluster 2021-07-31T02:30:25.435028+0000 mon.a (mon.0) 2178 : 2021-07-31T02:30:26.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:26 smithi154 conmon[34553]: cluster [DBG] osdmap e1244: 8 total, 8 up, 8 in 2021-07-31T02:30:26.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:26 smithi045 conmon[39718]: cluster 2021-07-31T02:30:25. 2021-07-31T02:30:26.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:26 smithi045 conmon[39718]: 435028+0000 mon.a (mon.0) 2178 : cluster [DBG] osdmap e1244: 8 total, 8 up, 8 in 2021-07-31T02:30:26.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:26 smithi045 conmon[33655]: cluster 2021-07-31T02:30:25.435028+0000 mon.a (mon.0) 2178 : cluster [DBG] osdmap e1244: 8 total, 8 up, 8 in 2021-07-31T02:30:27.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:27 smithi154 conmon[34553]: cluster 2021-07-31T02:30:26.416013+0000 mgr.x (mgr.24542) 1848 : cluster [DBG] pgmap v2387: 41 pgs: 41 active+clean; 4.6 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:30:27.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:27 smithi154 conmon[34553]: cluster 2021-07-31T02:30:26.445204+0000 mon.a (mon.0) 2179 : cluster [DBG] osdmap e1245: 8 total, 8 up, 8 in 2021-07-31T02:30:27.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:27 smithi045 conmon[33655]: cluster 2021-07-31T02:30:26.416013+0000 mgr.x 2021-07-31T02:30:27.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:27 smithi045 conmon[33655]: (mgr.24542) 1848 : cluster [DBG] pgmap v2387: 41 pgs: 41 active+clean; 4.6 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:30:27.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:27 smithi045 conmon[33655]: cluster 2021-07-31T02:30:26.445204+0000 mon.a (mon.0) 2179 : cluster 2021-07-31T02:30:27.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:27 smithi045 conmon[33655]: [DBG] osdmap e1245: 8 total, 8 up, 8 in 2021-07-31T02:30:27.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:27 smithi045 conmon[39718]: cluster 2021-07-31T02:30: 2021-07-31T02:30:27.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:27 smithi045 conmon[39718]: 26.416013+0000 mgr.x (mgr.24542) 1848 : cluster [DBG] pgmap v2387: 41 pgs: 41 active+clean; 4.6 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:30:27.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:27 smithi045 conmon[39718]: cluster 2021- 2021-07-31T02:30:27.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:27 smithi045 conmon[39718]: 07-31T02:30:26.445204+0000 mon.a 2021-07-31T02:30:27.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:27 smithi045 conmon[39718]: (mon.0) 2179 : cluster [DBG] osdmap e1245: 8 total, 8 up, 8 in 2021-07-31T02:30:28.460 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMirroring.EnableImageMirror_In_MirrorModeImage 2021-07-31T02:30:28.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:28 smithi154 conmon[34553]: cluster 2021-07-31T02:30:27.451712+0000 mon.a (mon.0) 2180 : cluster [DBG] 2021-07-31T02:30:28.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:28 smithi154 conmon[34553]: osdmap e1246: 8 total, 8 up, 8 in 2021-07-31T02:30:28.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:28 smithi154 conmon[34553]: audit 2021-07-31T02:30:27.453450+0000 mon.c (mon.1) 107 : audit [INF] from='client.? 172.21.15.45:0/3641716911' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:30:28.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:28 smithi154 conmon[34553]: audit 2021-07-31T02:30 2021-07-31T02:30:28.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:28 smithi154 conmon[34553]: :27.453953+0000 mon.a (mon.0) 2181 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:30:28.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:28 smithi154 conmon[34553]: audit 2021-07-31T02:30:28 2021-07-31T02:30:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:28 smithi154 conmon[34553]: .083361+0000 mon.b (mon.2) 288 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:30:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:28 smithi154 conmon[34553]: audit 2021-07-31T02:30:28. 2021-07-31T02:30:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:28 smithi154 conmon[34553]: 083736+0000 mon.a (mon.0) 2182 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:30:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:28 smithi154 conmon[34553]: audit 2021-07-31T02:30:28.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:28 smithi154 conmon[34553]: 2021-07-31T 2021-07-31T02:30:28.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:28 smithi154 conmon[34553]: 02:30:28.106425+0000 mon.b (mon.2) 289 : audit 2021-07-31T02:30:28.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:28 smithi154 conmon[34553]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:30:28.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:28 smithi154 conmon[34553]: audit 2021-07-31T 2021-07-31T02:30:28.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:28 smithi154 conmon[34553]: 02:30:28.106960+0000 mon.a (mon.0) 2183 : audit [INF] 2021-07-31T02:30:28.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:28 smithi154 conmon[34553]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:30:28.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:28 smithi045 conmon[33655]: cluster 2021-07-31T02:30:27.451712+0000 mon.a (mon.0) 2180 2021-07-31T02:30:28.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:28 smithi045 conmon[33655]: : cluster [DBG] osdmap e1246: 8 total, 8 up, 8 in 2021-07-31T02:30:28.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:28 smithi045 conmon[33655]: audit 2021-07-31T02:30:27.453450+0000 mon.c (mon.1) 107 : audit [INF] from='client.? 172.21.15.45:0/3641716911' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:30:28.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:28 smithi045 conmon[33655]: audit 2021-07-31T02:30:28.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:28 smithi045 conmon[33655]: 2021-07-31T02:30:27.453953+0000 mon.a 2021-07-31T02:30:28.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:28 smithi045 conmon[33655]: (mon.0) 2181 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:30:28.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:28 smithi045 conmon[33655]: audit 2021 2021-07-31T02:30:28.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:28 smithi045 conmon[33655]: -07-31T02:30:28 2021-07-31T02:30:28.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:28 smithi045 conmon[33655]: .083361+0000 mon.b (mon.2 2021-07-31T02:30:28.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:28 smithi045 conmon[33655]: ) 288 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:30:28.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:28 smithi045 conmon[33655]: audit 2021-07-31T02:30:28.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:28 smithi045 conmon[33655]: 2021-07- 2021-07-31T02:30:28.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:28 smithi045 conmon[33655]: 31T02:30:28.083736+0000 mon.a ( 2021-07-31T02:30:28.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:28 smithi045 conmon[33655]: mon.0) 2182 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:30:28.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:28 smithi045 conmon[33655]: audit 2021- 2021-07-31T02:30:28.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:28 smithi045 conmon[33655]: 07-31T02:30:28.106425+0000 2021-07-31T02:30:28.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:28 smithi045 conmon[33655]: mon.b (mon.2) 289 : 2021-07-31T02:30:28.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:28 smithi045 conmon[33655]: audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:30:28.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:28 smithi045 conmon[33655]: audit 2021- 2021-07-31T02:30:28.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:28 smithi045 conmon[33655]: 07-31T02:30:28. 2021-07-31T02:30:28.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:28 smithi045 conmon[33655]: 106960+0000 mon.a (mon.0 2021-07-31T02:30:28.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:28 smithi045 conmon[33655]: ) 2183 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:30:28.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:28 smithi045 conmon[33655]: 2021-07-31T02:30:28.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:28 smithi045 conmon[39718]: cluster 2021-07-31T02:30:27.451712+0000 2021-07-31T02:30:28.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:28 smithi045 conmon[39718]: mon.a (mon.0) 2180 : cluster [DBG] osdmap e1246: 8 total, 8 up, 8 in 2021-07-31T02:30:28.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:28 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T02:30:28.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:28 smithi045 conmon[39718]: 02:30:27.453450+0000 mon.c (mon.1) 107 : audit 2021-07-31T02:30:28.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:28 smithi045 conmon[39718]: [INF] from='client.? 172.21.15.45:0/3641716911' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:30:28.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:28 smithi045 conmon[39718]: audit 2021-07-31T02:30:27 2021-07-31T02:30:28.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:28 smithi045 conmon[39718]: .453953+0000 mon.a (mon. 2021-07-31T02:30:28.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:28 smithi045 conmon[39718]: 0) 2181 : audit [INF] 2021-07-31T02:30:28.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:28 smithi045 conmon[39718]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:30:28.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:28 smithi045 conmon[39718]: audit 2021- 2021-07-31T02:30:28.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:28 smithi045 conmon[39718]: 07-31T02:30: 2021-07-31T02:30:28.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:28 smithi045 conmon[39718]: 28.083361+0000 mon.b (mon 2021-07-31T02:30:28.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:28 smithi045 conmon[39718]: .2) 288 : audit [INF] 2021-07-31T02:30:28.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:28 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:30:28.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:28 smithi045 conmon[39718]: audit 2021 2021-07-31T02:30:28.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:28 smithi045 conmon[39718]: -07-31T02:30:28.083736+0000 mon.a (mon.0) 2182 : audit 2021-07-31T02:30:28.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:28 smithi045 conmon[39718]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:30:28.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:28 smithi045 conmon[39718]: audit 2021- 2021-07-31T02:30:28.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:28 smithi045 conmon[39718]: 07-31T02:30:28.106425+0000 mon.b ( 2021-07-31T02:30:28.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:28 smithi045 conmon[39718]: mon.2) 289 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:30:28.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:28 smithi045 conmon[39718]: 2021-07-31T02:30:28.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:28 smithi045 conmon[39718]: audit 2021-07 2021-07-31T02:30:28.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:28 smithi045 conmon[39718]: -31T02:30:28.106960+0000 mon.a (mon.0 2021-07-31T02:30:28.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:28 smithi045 conmon[39718]: ) 2183 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:30:29.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:29 smithi154 conmon[34553]: cluster 2021-07-31T02:30:28.416350+0000 mgr.x (mgr.24542) 1849 : cluster [DBG] pgmap v2390: 41 pgs: 32 unknown, 9 active+clean; 4.6 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:30:29.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:29 smithi154 conmon[34553]: 2021-07-31T02:30:29.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:29 smithi154 conmon[34553]: audit 2021-07-31T02:30:28.458178+0000 mon.a (mon.0) 2184 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-10","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:30:29.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:29 smithi154 conmon[34553]: cluster 2021-07-31T02:30:28.458438+0000 mon.a (mon.0) 2185 : cluster [DBG] osdmap e1247: 8 total, 8 up, 8 in 2021-07-31T02:30:29.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:29 smithi045 conmon[33655]: cluster 2021-07-31T02:30:28.416350+0000 mgr.x (mgr.24542 2021-07-31T02:30:29.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:29 smithi045 conmon[33655]: ) 1849 : cluster [DBG] pgmap v2390: 41 pgs: 32 unknown, 9 active+clean; 4.6 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:30:29.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:29 smithi045 conmon[33655]: audit 2021-07-31T02:30:28.458178+0000 mon.a (mon.0) 2184 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-10","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:30:29.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:29 smithi045 conmon[33655]: cluster 2021-07-31T02:30:29.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:29 smithi045 conmon[33655]: 2021-07-31T02:30:28.458438+0000 mon.a (mon.0) 2185 : cluster [DBG] osdmap e1247: 8 total, 8 up, 8 in 2021-07-31T02:30:29.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:29 smithi045 conmon[39718]: cluster 2021-07-31T02:30:28.416350+0000 mgr.x (mgr.24542 2021-07-31T02:30:29.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:29 smithi045 conmon[39718]: ) 1849 : cluster [DBG] pgmap v2390: 41 pgs: 32 unknown, 9 active+clean; 4.6 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:30:29.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:29 smithi045 conmon[39718]: audit 2021-07-31T02:30:28.458178 2021-07-31T02:30:29.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:29 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2184 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-10","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:30:29.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:29 smithi045 conmon[39718]: cluster 2021-07-31T02: 2021-07-31T02:30:29.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:29 smithi045 conmon[39718]: 30:28.458438+0000 mon.a (mon.0) 2185 : 2021-07-31T02:30:29.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:29 smithi045 conmon[39718]: cluster [DBG] osdmap e1247: 8 total, 8 up, 8 in 2021-07-31T02:30:30.597 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMirroring.EnableImageMirror_In_MirrorModeImage (2137 ms) 2021-07-31T02:30:30.597 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMirroring.EnableImageMirror_In_MirrorModePool 2021-07-31T02:30:30.705 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMirroring.EnableImageMirror_In_MirrorModePool (108 ms) 2021-07-31T02:30:30.705 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMirroring.EnableImageMirror_In_MirrorModeDisabled 2021-07-31T02:30:30.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:30 smithi154 conmon[34553]: cluster 2021-07-31T02:30:29.459638+0000 mon.a (mon.0 2021-07-31T02:30:30.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:30 smithi154 conmon[34553]: ) 2186 : cluster [DBG] osdmap e1248: 8 total, 8 up, 8 in 2021-07-31T02:30:30.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:30 smithi045 conmon[33655]: cluster 2021-07-31T02:30:29.459638+0000 mon.a (mon.0) 2186 : cluster [DBG] osdmap e1248: 8 total, 8 up, 8 in 2021-07-31T02:30:30.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:30 smithi045 conmon[33655]: 2021-07-31T02:30:30.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:30 smithi045 conmon[39718]: cluster 2021-07-31T02:30:29.459638+0000 mon.a 2021-07-31T02:30:30.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:30 smithi045 conmon[39718]: (mon.0) 2186 : cluster [DBG] osdmap e1248: 8 total, 8 up, 8 in 2021-07-31T02:30:30.789 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMirroring.EnableImageMirror_In_MirrorModeDisabled (84 ms) 2021-07-31T02:30:30.790 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMirroring.DisableImageMirror_In_MirrorModeImage 2021-07-31T02:30:30.903 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMirroring.DisableImageMirror_In_MirrorModeImage (113 ms) 2021-07-31T02:30:30.903 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMirroring.DisableImageMirror_In_MirrorModeImage_NoObjectMap 2021-07-31T02:30:30.999 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMirroring.DisableImageMirror_In_MirrorModeImage_NoObjectMap (97 ms) 2021-07-31T02:30:30.999 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMirroring.DisableImageMirror_In_MirrorModePool 2021-07-31T02:30:31.096 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMirroring.DisableImageMirror_In_MirrorModePool (97 ms) 2021-07-31T02:30:31.097 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMirroring.DisableImageMirror_In_MirrorModeDisabled 2021-07-31T02:30:31.214 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMirroring.DisableImageMirror_In_MirrorModeDisabled (118 ms) 2021-07-31T02:30:31.215 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMirroring.DisableImageMirrorWithPeer 2021-07-31T02:30:31.215 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:30:31.215 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMirroring.DisableImageMirrorWithPeer (0 ms) 2021-07-31T02:30:31.215 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMirroring.DisableJournalingWithPeer 2021-07-31T02:30:31.216 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:30:31.216 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMirroring.DisableJournalingWithPeer (0 ms) 2021-07-31T02:30:31.216 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMirroring.EnableImageMirror_In_MirrorModeDisabled_WithoutJournaling 2021-07-31T02:30:31.289 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMirroring.EnableImageMirror_In_MirrorModeDisabled_WithoutJournaling (75 ms) 2021-07-31T02:30:31.290 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMirroring.EnableImageMirror_In_MirrorModePool_WithoutJournaling 2021-07-31T02:30:31.371 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMirroring.EnableImageMirror_In_MirrorModePool_WithoutJournaling (82 ms) 2021-07-31T02:30:31.371 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMirroring.EnableImageMirror_In_MirrorModeImage_WithoutJournaling 2021-07-31T02:30:31.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:31 smithi154 conmon[34553]: cluster 2021-07-31T02:30:30.416837+0000 mgr.x (mgr.24542) 1850 : cluster [DBG] pgmap v2393: 41 pgs: 41 active+clean; 4.6 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:30:31.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:31 smithi154 conmon[34553]: cluster 2021-07-31T02:30:31.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:31 smithi154 conmon[34553]: 2021-07-31T02:30:30.472950+0000 mon.a (mon.0) 2187 : cluster [DBG] osdmap e1249: 8 total, 8 up, 8 in 2021-07-31T02:30:31.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:31 smithi045 conmon[33655]: cluster 2021-07-31T02:30:30.416837+0000 mgr.x (mgr 2021-07-31T02:30:31.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:31 smithi045 conmon[33655]: .24542) 1850 : cluster [DBG] pgmap v2393: 41 pgs: 41 active+clean; 4.6 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:30:31.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:31 smithi045 conmon[33655]: cluster 2021-07-31T02:30:30.472950+0000 mon.a (mon.0) 2187 : cluster [DBG] osdmap e1249: 8 total, 8 up, 8 in 2021-07-31T02:30:31.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:31 smithi045 conmon[39718]: cluster 2021-07-31T02:30:30 2021-07-31T02:30:31.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:31 smithi045 conmon[39718]: .416837+0000 mgr.x (mgr.24542) 1850 : cluster [DBG] pgmap v2393: 41 pgs: 41 active+clean; 4.6 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:30:31.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:31 smithi045 conmon[39718]: cluster 2021- 2021-07-31T02:30:31.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:31 smithi045 conmon[39718]: 07-31T02:30: 2021-07-31T02:30:31.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:31 smithi045 conmon[39718]: 30.472950+0000 mon.a (mon.0) 2021-07-31T02:30:31.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:31 smithi045 conmon[39718]: 2187 : cluster [DBG] osdmap e1249: 8 total, 8 up, 8 in 2021-07-31T02:30:32.536 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMirroring.EnableImageMirror_In_MirrorModeImage_WithoutJournaling (1164 ms) 2021-07-31T02:30:32.536 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMirroring.EnableImageMirror_In_MirrorModeImage_WithoutExclusiveLock 2021-07-31T02:30:32.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[34553]: cluster 2021-07-31T02:30:31.473202+0000 mon.a (mon.0) 2188 : cluster 2021-07-31T02:30:32.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[34553]: [DBG] osdmap e1250: 8 total, 8 up, 8 in 2021-07-31T02:30:32.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[34553]: audit 2021-07-31T02:30:32.271416+0000 mon.b (mon.2) 290 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:30:32.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[34553]: audit 2021-07-31T02: 2021-07-31T02:30:32.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[34553]: 30:32.272847+0000 mon.b (mon.2) 291 2021-07-31T02:30:32.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[34553]: : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:30:32.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[34553]: audit 2021-07-31T02:30 2021-07-31T02:30:32.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[34553]: :32.273829+0000 mon.b (mon.2) 292 2021-07-31T02:30:32.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[34553]: : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:30:32.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[34553]: cephadm 2021-07 2021-07-31T02:30:32.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[34553]: -31T02:30:32.274650+0000 mgr.x 2021-07-31T02:30:32.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[34553]: (mgr.24542) 1851 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:30:32.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[34553]: cephadm 2021-07-31T02: 2021-07-31T02:30:32.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[34553]: 30:32.318635+0000 mgr.x (mgr.24542) 1852 : cephadm 2021-07-31T02:30:32.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[34553]: [ERR] 2021-07-31T02:30:32.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:30:32.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:30:32.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[34553]: yield 2021-07-31T02:30:32.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:30:32.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:30:32.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:30:32.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:30:32.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:30:32.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:30:32.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:30:32.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:30:32.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:30:32.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[34553]: task.result() 2021-07-31T02:30:32.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:30:32.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[34553]: cephadm 2021-07-31T02:30:32.319119+0000 mgr.x (mgr.24542) 1853 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:30:32.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[34553]: cephadm 2021-07-31T02:30:32.330067+0000 mgr.x (mgr.24542) 1854 : cephadm 2021-07-31T02:30:32.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[34553]: [ERR] 2021-07-31T02:30:32.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:30:32.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:30:32.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[34553]: yield 2021-07-31T02:30:32.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:30:32.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:30:32.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:30:32.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: debug 2021-07-31T02:30:32.316+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:30:32.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:30:32.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:30:32.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: yield 2021-07-31T02:30:32.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:30:32.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:30:32.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:30:32.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:30:32.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:30:32.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:30:32.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:30:32.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:30:32.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:30:32.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: task.result() 2021-07-31T02:30:32.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:30:32.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: debug 2021-07-31T02:30:32.328+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:30:32.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:30:32.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:30:32.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: yield 2021-07-31T02:30:32.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:30:32.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:30:32.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:30:32.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:30:32.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:30:32.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:30:32.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:30:32.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:30:32.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:30:32.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: task.result() 2021-07-31T02:30:32.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:30:32.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: debug 2021-07-31T02:30:32.353+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:30:32.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:30:32.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:30:32.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: yield 2021-07-31T02:30:32.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:30:32.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:30:32.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:30:32.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:30:32.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:30:32.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:30:32.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:30:32.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:30:32.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:30:32.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:30:32.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:30:32.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[34553]: task.result() 2021-07-31T02:30:32.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:30:32.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[34553]: cephadm 2021-07-31T02:30:32.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[34553]: 2021-07-31T02:30:32.330434+0000 mgr.x (mgr.24542) 1855 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:30:32.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:30:32.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:30:32.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:30:32.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:30:32.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: task.result() 2021-07-31T02:30:32.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:30:32.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: debug 2021-07-31T02:30:32.354+0000 7fcb9095b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:30:32.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:30:32.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:30:32.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: yield 2021-07-31T02:30:32.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:30:32.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:30:32.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:30:32.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:30:32.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:30:32.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:30:32.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:30:32.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:30:32.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:30:32.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: task.result() 2021-07-31T02:30:32.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:30:32.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: 2021-07-31T02:30:32.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:30:32.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: 2021-07-31T02:30:32.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:30:32.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:30:32.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:30:32.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:30:32.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:30:32.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:30:32.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:30:32.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:30:32.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:30:32.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:30:32.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:30:32.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:30:32.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:30:32.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:30:32.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:30:32.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:30:32.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:30:32.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:30:32.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: raise 2021-07-31T02:30:32.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:30:32.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:30:32.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:30:32.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:30:32.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:30:32.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: debug 2021-07-31T02:30:32.362+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:30:32.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:30:32.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:30:32.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: yield 2021-07-31T02:30:32.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:30:32.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:30:32.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:30:32.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:30:32.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:30:32.786 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:30:32.786 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:30:32.786 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:30:32.786 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:30:32.786 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: task.result() 2021-07-31T02:30:32.787 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:30:32.787 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: debug 2021-07-31T02:30:32.363+0000 7fcb9115c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:30:32.787 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:30:32.787 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:30:32.788 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: yield 2021-07-31T02:30:32.788 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:30:32.788 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:30:32.788 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:30:32.789 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:30:32.789 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:30:32.789 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:30:32.789 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:30:32.789 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:30:32.790 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:30:32.790 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: task.result() 2021-07-31T02:30:32.790 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:30:32.790 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: 2021-07-31T02:30:32.790 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:30:32.791 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: 2021-07-31T02:30:32.791 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:30:32.791 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:30:32.791 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:30:32.792 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:30:32.792 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:30:32.792 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:30:32.792 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:30:32.793 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:30:32.793 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:30:32.793 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:30:32.793 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:30:32.793 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:30:32.794 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:30:32.794 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:30:32.794 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:30:32.794 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:30:32.794 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:30:32.795 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:30:32.795 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: raise 2021-07-31T02:30:32.795 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:30:32.795 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:30:32.796 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:30:32.796 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:30:32.796 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:30:32 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:30:32.797 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:32 smithi045 conmon[33655]: cluster 2021-07-31T02:30:31.473202+0000 mon.a (mon.0 2021-07-31T02:30:32.797 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:32 smithi045 conmon[33655]: ) 2188 : cluster [DBG] osdmap e1250: 8 total, 8 up, 8 in 2021-07-31T02:30:32.797 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:32 smithi045 conmon[33655]: audit 2021-07-31T02:30:32.271416+0000 mon.b (mon.2) 290 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:30:32.798 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:32 smithi045 conmon[33655]: audit 2021-07-31T02:30:32.272847+0000 mon.b (mon.2) 291 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:30:32.798 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:32 smithi045 conmon[33655]: audit 2021-07-31T02:30:32.273829+0000 mon.b (mon.2) 292 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:30:32.798 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:32 smithi045 conmon[33655]: cephadm 2021-07-31T02:30:32.274650+0000 mgr.x (mgr.24542) 1851 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:30:32.798 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:32 smithi045 conmon[33655]: cephadm 2021-07-31T02:30:32.799 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:32 smithi045 conmon[33655]: 2021-07-31T02:30:32.318635+0000 mgr.x (mgr.24542) 1852 : cephadm [ERR] 2021-07-31T02:30:32.799 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:32 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:30:32.799 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:30:32.799 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:32 smithi045 conmon[33655]: yield 2021-07-31T02:30:32.799 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:30:32.800 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:32 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:30:32.800 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:30:32.800 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:32 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:30:32.800 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:30:32.801 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:32 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:30:32.801 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:30:32.801 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:32 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:30:32.801 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:30:32.801 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:32 smithi045 conmon[33655]: task.result() 2021-07-31T02:30:32.802 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:32 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:30:32.802 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:32 smithi045 conmon[33655]: cephadm 2021-07-31T02:30:32.319119+0000 mgr.x (mgr.24542) 1853 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:30:32.802 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:32 smithi045 conmon[33655]: cephadm 2021-07 2021-07-31T02:30:32.802 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:32 smithi045 conmon[33655]: -31T02:30:32.330067+0000 mgr.x (mgr.24542) 1854 : cephadm [ERR] 2021-07-31T02:30:32.803 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:32 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:30:32.803 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:30:32.803 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:32 smithi045 conmon[33655]: yield 2021-07-31T02:30:32.803 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:30:32.803 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:32 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:30:32.804 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:30:32.804 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:32 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:30:32.804 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:30:32.804 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:32 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:30:32.805 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:30:32.805 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:32 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:30:32.805 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:30:32.805 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:32 smithi045 conmon[33655]: task.result() 2021-07-31T02:30:32.805 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:32 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:30:32.806 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:32 smithi045 conmon[33655]: cephadm 2021-07-31T02:30:32.330434+0000 mgr.x (mgr.24542) 1855 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:30:32.806 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:32 smithi045 conmon[39718]: cluster 2021-07-31T02:30:31.473202+0000 mon.a (mon.0) 2188 : cluster 2021-07-31T02:30:32.806 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:32 smithi045 conmon[39718]: [DBG] osdmap e1250: 8 total, 8 up, 8 in 2021-07-31T02:30:32.807 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:32 smithi045 conmon[39718]: audit 2021-07-31T02:30:32.271416+0000 mon.b (mon.2) 290 : audit [DBG] 2021-07-31T02:30:32.807 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:32 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:30:32.807 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:32 smithi045 conmon[39718]: audit 2021-07-31T02:30:32.272847+0000 2021-07-31T02:30:32.807 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:32 smithi045 conmon[39718]: mon.b (mon.2) 291 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:30:32.807 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:32 smithi045 conmon[39718]: audit 2021-07-31T02:30:32.808 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:32 smithi045 conmon[39718]: 2021-07-31T02:30:32.273829+0000 mon.b (mon.2) 292 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:30:32.808 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:32 smithi045 conmon[39718]: cephadm 2021-07-31T02 2021-07-31T02:30:32.808 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:32 smithi045 conmon[39718]: :30:32.274650+0000 mgr.x (mgr.24542) 1851 : cephadm 2021-07-31T02:30:32.808 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:32 smithi045 conmon[39718]: [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:30:32.809 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:32 smithi045 conmon[39718]: cephadm 2021-07-31T02:30:32.809 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:32 smithi045 conmon[39718]: 2021-07-31T02:30:32.318635+0000 mgr.x (mgr.24542) 1852 : cephadm [ERR] 2021-07-31T02:30:32.809 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:32 smithi045 conmon[39718]: 2021-07-31T02:30:32.809 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:32 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:30:32.809 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:30:32.810 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:32 smithi045 conmon[39718]: yield 2021-07-31T02:30:32.810 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:30:32.810 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:32 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:30:32.810 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:30:32.810 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:32 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:30:32.811 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:30:32.811 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:32 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:30:32.811 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:30:32.811 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:32 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:30:32.812 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:30:32.812 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:32 smithi045 conmon[39718]: task.result() 2021-07-31T02:30:32.812 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:32 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:30:32.812 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:32 smithi045 conmon[39718]: cephadm 2021-07- 2021-07-31T02:30:32.812 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:32 smithi045 conmon[39718]: 31T02:30:32.319119+0000 mgr.x (mgr.24542) 1853 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:30:32.813 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:32 smithi045 conmon[39718]: cephadm 2021-07-31T02:30:32.330067+0000 mgr.x (mgr.24542) 1854 : cephadm [ERR] 2021-07-31T02:30:32.813 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:32 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:30:32.813 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:30:32.813 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:32 smithi045 conmon[39718]: yield 2021-07-31T02:30:32.813 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:30:32.814 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:32 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:30:32.814 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:30:32.814 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:32 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:30:32.814 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:30:32.814 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:32 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:30:32.815 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:30:32.815 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:32 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:30:32.815 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:30:32.815 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:32 smithi045 conmon[39718]: task.result() 2021-07-31T02:30:32.816 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:32 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:30:32.816 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:32 smithi045 conmon[39718]: cephadm 2021-07-31T02:30:32.330434+0000 mgr.x (mgr.24542) 1855 : cephadm [INF] 2021-07-31T02:30:32.816 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:32 smithi045 conmon[39718]: Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:30:33.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:30:32.355634+0000 mgr.x (mgr.24542) 1856 : cephadm 2021-07-31T02:30:33.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: [ERR] 2021-07-31T02:30:33.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:30:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:30:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: yield 2021-07-31T02:30:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:30:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:30:33.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:30:33.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:30:33.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:30:33.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:30:33.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:30:33.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:30:33.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:30:33.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:30:33.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:30:33.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:30:32.356094+0000 mgr.x (mgr.24542) 1857 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:30:33.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:30:33.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:30:33.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: yield 2021-07-31T02:30:33.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:30:33.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:30:33.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:30:33.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:30:33.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:30:33.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:30:33.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:30:33.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:30:33.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:30:33.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:30:33.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:30:33.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: 2021-07-31T02:30:33.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:30:33.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: 2021-07-31T02:30:33.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:30:33.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:30:33.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:30:33.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:30:33.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:30:33.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:30:33.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:30:33.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:30:33.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:30:33.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:30:33.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:30:33.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:30:33.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:30:33.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:30:33.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:30:33.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:30:33.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:30:33.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:30:33.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: raise 2021-07-31T02:30:33.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:30:33.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:30:33.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:30:33.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:30:33.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:30:33.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:30:32.364467+0000 mgr.x (mgr.24542) 1858 : cephadm [ERR] 2021-07-31T02:30:33.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:30:33.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:30:33.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: yield 2021-07-31T02:30:33.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:30:33.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:30:33.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:30:33.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:30:33.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:30:33.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:30:33.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:30:33.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:30:33.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:30:33.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:30:33.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:30:33.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:30:32.365002+0000 mgr.x (mgr.24542) 1859 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:30:33.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:30:33.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:30:33.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: yield 2021-07-31T02:30:33.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:30:33.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:30:33.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:30:33.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:30:33.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:30:33.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:30:33.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:30:33.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:30:33.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:30:33.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:30:33.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:30:33.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: 2021-07-31T02:30:33.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:30:33.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: 2021-07-31T02:30:33.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:30:33.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:30:33.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:30:33.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:30:33.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:30:33.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:30:33.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:30:33.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:30:33.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:30:33.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:30:33.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:30:33.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:30:33.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:30:33.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:30:33.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:30:33.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:30:33.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:30:33.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:30:33.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: raise 2021-07-31T02:30:33.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:30:33.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:30:33.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:30:33.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:30:33.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:30:33.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: cluster 2021-07-31T02:30:32.417370+0000 mgr.x (mgr.24542) 1860 : cluster [DBG] pgmap v2396: 41 pgs: 41 active+clean; 4.6 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:30:33.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:33 smithi154 conmon[34553]: cluster 2021-07-31T02:30:32.486030+0000 mon.a (mon.0) 2189 : cluster [DBG] osdmap e1251: 8 total, 8 up, 8 in 2021-07-31T02:30:33.779 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:30:33.584Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:30:33.779 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:30:33.584Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:30:33.780 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:30:33.584Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:30:33.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:30:32.355634+0000 mgr.x (mgr.24542) 2021-07-31T02:30:33.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: 1856 : cephadm [ERR] 2021-07-31T02:30:33.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:30:33.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:30:33.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: yield 2021-07-31T02:30:33.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:30:33.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:30:33.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:30:33.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:30:33.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:30:33.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:30:33.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:30:33.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:30:33.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:30:33.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:30:33.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:30:33.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:30:32.356094+0000 mgr.x (mgr.24542) 1857 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:30:33.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:30:33.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:30:33.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: yield 2021-07-31T02:30:33.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:30:33.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:30:33.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:30:33.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:30:33.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:30:33.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:30:33.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:30:33.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:30:33.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:30:33.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:30:33.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:30:33.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: 2021-07-31T02:30:33.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:30:33.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: 2021-07-31T02:30:33.788 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:30:33.788 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:30:33.788 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:30:33.788 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:30:33.789 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:30:33.789 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:30:33.789 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:30:33.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:30:32.355634+0000 mgr.x (mgr.24542) 1856 : cephadm [ERR] 2021-07-31T02:30:33.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: 2021-07-31T02:30:33.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:30:33.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:30:33.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: yield 2021-07-31T02:30:33.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:30:33.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:30:33.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:30:33.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:30:33.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:30:33.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:30:33.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:30:33.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:30:33.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:30:33.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:30:33.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:30:33.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:30:32.356094+0000 mgr.x (mgr.24542) 1857 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:30:33.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:30:33.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:30:33.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: yield 2021-07-31T02:30:33.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:30:33.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:30:33.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:30:33.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:30:33.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:30:33.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:30:33.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:30:33.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:30:33.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:30:33.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:30:33.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:30:33.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: 2021-07-31T02:30:33.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:30:33.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: 2021-07-31T02:30:33.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:30:33.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:30:33.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:30:33.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:30:33.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:30:33.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:30:33.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:30:33.799 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:30:33.799 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:30:33.799 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:30:33.799 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:30:33.800 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:30:33.800 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:30:33.800 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:30:33.800 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:30:33.800 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:30:33.801 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:30:33.801 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:30:33.801 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: raise 2021-07-31T02:30:33.802 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:30:33.802 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:30:33.802 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:30:33.802 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:30:33.803 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:30:33.803 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:30: 2021-07-31T02:30:33.803 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: 32.364467+0000 mgr.x (mgr.24542) 1858 : cephadm [ERR] 2021-07-31T02:30:33.803 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:30:33.803 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:30:33.804 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: yield 2021-07-31T02:30:33.804 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:30:33.804 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:30:33.804 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:30:33.804 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:30:33.805 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:30:33.805 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:30:33.805 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:30:33.805 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:30:33.806 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:30:33.806 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:30:33.806 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:30:33.806 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:30:32.365002+0000 mgr.x (mgr.24542) 1859 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:30:33.806 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:30:33.807 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:30:33.807 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: yield 2021-07-31T02:30:33.807 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:30:33.807 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:30:33.808 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:30:33.808 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:30:33.808 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:30:33.808 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:30:33.809 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:30:33.809 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:30:33.809 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:30:33.809 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:30:33.810 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:30:33.810 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:30:33.810 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:30:33.810 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: raise 2021-07-31T02:30:33.810 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:30:33.811 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:30:33.811 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:30:33.811 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:30:33.811 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:30:33.812 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:30:32.364467+0000 mgr.x (mgr.24542) 1858 : cephadm [ERR] 2021-07-31T02:30:33.812 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:30:33.812 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:30:33.812 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: yield 2021-07-31T02:30:33.812 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:30:33.813 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:30:33.813 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:30:33.813 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:30:33.813 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:30:33.814 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:30:33.814 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:30:33.814 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:30:33.814 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:30:33.815 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:30:33.815 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:30:33.815 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:30:32.365002+0000 mgr.x (mgr.24542) 1859 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:30:33.815 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:30:33.816 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:30:33.816 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: yield 2021-07-31T02:30:33.816 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:30:33.816 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:30:33.817 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:30:33.817 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:30:33.817 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:30:33.817 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:30:33.818 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:30:33.818 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:30:33.818 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:30:33.818 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:30:33.818 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:30:33.819 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: 2021-07-31T02:30:33.819 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:30:33.819 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: 2021-07-31T02:30:33.819 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:30:33.820 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:30:33.820 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:30:33.820 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:30:33.820 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:30:33.820 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:30:33.821 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:30:33.821 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:30:33.821 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:30:33.821 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:30:33.821 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:30:33.822 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:30:33.822 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:30:33.822 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:30:33.822 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:30:33.823 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:30:33.823 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:30:33.823 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:30:33.824 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: raise 2021-07-31T02:30:33.824 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:30:33.824 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:30:33.824 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:30:33.824 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:30:33.825 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:30:33.825 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: cluster 2021-07-31T02:30:32.417370+0000 mgr.x (mgr.24542) 1860 : cluster [DBG] pgmap v2396: 41 pgs: 41 active+clean; 4.6 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:30:33.825 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[33655]: cluster 2021-07-31T02:30:32.486030+0000 mon.a (mon.0) 2189 : cluster [DBG] osdmap e1251: 8 total, 8 up, 8 in 2021-07-31T02:30:33.825 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:30:33.826 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:30:33.826 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:30:33.826 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:30:33.826 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:30:33.827 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:30:33.827 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:30:33.827 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:30:33.827 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:30:33.827 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: 2021-07-31T02:30:33.828 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:30:33.828 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: 2021-07-31T02:30:33.828 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:30:33.828 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:30:33.829 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:30:33.829 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:30:33.829 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:30:33.829 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:30:33.829 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:30:33.830 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:30:33.830 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:30:33.830 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:30:33.830 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:30:33.830 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:30:33.831 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:30:33.831 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:30:33.831 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:30:33.831 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:30:33.832 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:30:33.832 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:30:33.832 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: raise 2021-07-31T02:30:33.832 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:30:33.832 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:30:33.833 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:30:33.833 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:30:33.833 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:30:33.833 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: cluster 2021-07-31T02:30:32.417370+0000 mgr.x (mgr.24542) 1860 : cluster [DBG] pgmap v2396: 41 pgs: 41 active+clean; 4.6 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:30:33.833 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:33 smithi045 conmon[39718]: cluster 2021-07-31T02:30:32.486030+0000 mon.a (mon.0) 2189 : cluster [DBG] osdmap e1251: 8 total, 8 up, 8 in 2021-07-31T02:30:34.547 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMirroring.EnableImageMirror_In_MirrorModeImage_WithoutExclusiveLock (2012 ms) 2021-07-31T02:30:34.548 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMirroring.CreateImage_In_MirrorModeDisabled 2021-07-31T02:30:34.651 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMirroring.CreateImage_In_MirrorModeDisabled (104 ms) 2021-07-31T02:30:34.651 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMirroring.CreateImage_In_MirrorModeImage 2021-07-31T02:30:34.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:34 smithi154 conmon[34553]: cluster 2021-07-31T02:30:33.487829+0000 mon.a (mon.0) 2190 : 2021-07-31T02:30:34.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:34 smithi154 conmon[34553]: cluster [DBG] osdmap e1252: 8 total, 8 up, 8 in 2021-07-31T02:30:34.755 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMirroring.CreateImage_In_MirrorModeImage (104 ms) 2021-07-31T02:30:34.755 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMirroring.CreateImage_In_MirrorModePool 2021-07-31T02:30:34.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:34 smithi045 conmon[33655]: cluster 2021-07-31T02:30:33.487829+0000 mon.a (mon 2021-07-31T02:30:34.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:34 smithi045 conmon[33655]: .0) 2190 : cluster [DBG] osdmap e1252: 8 total, 8 up, 8 in 2021-07-31T02:30:34.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:34 smithi045 conmon[39718]: cluster 2021-07-31T02:30:33.487829 2021-07-31T02:30:34.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:34 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2190 : cluster [DBG] osdmap e1252: 8 total, 8 up, 8 in 2021-07-31T02:30:34.858 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMirroring.CreateImage_In_MirrorModePool (103 ms) 2021-07-31T02:30:34.859 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMirroring.CreateImage_In_MirrorModePool_WithoutJournaling 2021-07-31T02:30:34.944 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMirroring.CreateImage_In_MirrorModePool_WithoutJournaling (85 ms) 2021-07-31T02:30:34.945 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMirroring.CreateImage_In_MirrorModeImage_WithoutJournaling 2021-07-31T02:30:35.026 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMirroring.CreateImage_In_MirrorModeImage_WithoutJournaling (83 ms) 2021-07-31T02:30:35.027 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMirroring.EnableJournaling_In_MirrorModeDisabled 2021-07-31T02:30:35.134 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMirroring.EnableJournaling_In_MirrorModeDisabled (108 ms) 2021-07-31T02:30:35.135 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMirroring.EnableJournaling_In_MirrorModeImage 2021-07-31T02:30:35.242 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMirroring.EnableJournaling_In_MirrorModeImage (108 ms) 2021-07-31T02:30:35.242 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMirroring.EnableJournaling_In_MirrorModeImage_SnapshotMirroringEnabled 2021-07-31T02:30:35.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:35 smithi154 conmon[34553]: cluster 2021-07-31T02:30:35.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:35 smithi154 conmon[34553]: 2021-07-31T02:30:35.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:35 smithi154 conmon[34553]: 2021 2021-07-31T02:30:35.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:35 smithi154 conmon[34553]: -07-31T02:30:34. 2021-07-31T02:30:35.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:35 smithi154 conmon[34553]: 417982+0000 mgr.x (mgr.24542) 1861 : cluster [DBG] pgmap v2399: 41 pgs: 41 active+clean; 4.6 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 433 KiB/s rd, 58 KiB/s wr, 624 op/s 2021-07-31T02:30:35.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:35 smithi154 conmon[34553]: cluster 2021-07-31T02:30:35.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:35 smithi154 conmon[34553]: 2021-07-31T02:30: 2021-07-31T02:30:35.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:35 smithi154 conmon[34553]: 34.498545+0000 mon.a (mon 2021-07-31T02:30:35.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:35 smithi154 conmon[34553]: .0) 2191 : cluster [DBG] 2021-07-31T02:30:35.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:35 smithi154 conmon[34553]: osdmap e1253: 8 total, 8 up, 8 in 2021-07-31T02:30:35.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:35 smithi045 conmon[33655]: cluster 2021-07-31T02:30:34.417982+0000 mgr.x (mgr. 2021-07-31T02:30:35.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:35 smithi045 conmon[33655]: 24542) 1861 : cluster [DBG] pgmap v2399: 41 pgs: 41 active+clean; 4.6 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 433 KiB/s rd, 58 KiB/s wr, 624 op/s 2021-07-31T02:30:35.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:35 smithi045 conmon[33655]: cluster 2021-07-31T02:30:34.498545+0000 mon.a (mon.0) 2191 : cluster [DBG] osdmap e1253: 8 total, 8 up, 8 in 2021-07-31T02:30:35.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:35 smithi045 conmon[39718]: cluster 2021-07-31T02:30:34.417982+0000 mgr.x (mgr.24542 2021-07-31T02:30:35.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:35 smithi045 conmon[39718]: ) 1861 : cluster [DBG] pgmap v2399: 41 pgs: 41 active+clean; 4.6 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 433 KiB/s rd, 58 KiB/s wr, 624 op/s 2021-07-31T02:30:35.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:35 smithi045 conmon[39718]: cluster 2021-07-31T02:30:34.498545+0000 mon.a (mon 2021-07-31T02:30:35.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:35 smithi045 conmon[39718]: .0) 2191 : cluster [DBG] osdmap e1253: 8 total, 8 up, 8 in 2021-07-31T02:30:36.573 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMirroring.EnableJournaling_In_MirrorModeImage_SnapshotMirroringEnabled (1331 ms) 2021-07-31T02:30:36.573 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMirroring.EnableJournaling_In_MirrorModePool 2021-07-31T02:30:36.693 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMirroring.EnableJournaling_In_MirrorModePool (120 ms) 2021-07-31T02:30:36.694 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMirroring.DisableJournaling_In_MirrorModePool 2021-07-31T02:30:36.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:36 smithi045 conmon[39718]: cluster 2021-07-31T02:30:35.498620+0000 mon.a (mon.0) 2192 : 2021-07-31T02:30:36.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:36 smithi045 conmon[39718]: cluster [DBG] osdmap e1254: 8 total, 8 up, 8 in 2021-07-31T02:30:36.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:36 smithi045 conmon[33655]: cluster 2021-07-31T02:30:35.498620+0000 mon.a (mon. 2021-07-31T02:30:36.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:36 smithi045 conmon[33655]: 0) 2192 : cluster [DBG] osdmap e1254: 8 total, 8 up, 8 in 2021-07-31T02:30:36.805 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMirroring.DisableJournaling_In_MirrorModePool (112 ms) 2021-07-31T02:30:36.806 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMirroring.DisableJournaling_In_MirrorModeImage 2021-07-31T02:30:36.912 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMirroring.DisableJournaling_In_MirrorModeImage (107 ms) 2021-07-31T02:30:36.912 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMirroring.MirrorModeSet_DisabledMode_To_PoolMode 2021-07-31T02:30:37.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:36 smithi154 conmon[34553]: cluster 2021-07-31T02:30:35.498620+0000 mon.a (mon.0 2021-07-31T02:30:37.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:36 smithi154 conmon[34553]: ) 2192 : cluster [DBG] osdmap e1254: 8 total, 8 up, 8 in 2021-07-31T02:30:37.075 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMirroring.MirrorModeSet_DisabledMode_To_PoolMode (163 ms) 2021-07-31T02:30:37.076 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMirroring.MirrorModeSet_PoolMode_To_DisabledMode 2021-07-31T02:30:37.248 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMirroring.MirrorModeSet_PoolMode_To_DisabledMode (172 ms) 2021-07-31T02:30:37.248 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMirroring.MirrorModeSet_DisabledMode_To_ImageMode 2021-07-31T02:30:37.407 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMirroring.MirrorModeSet_DisabledMode_To_ImageMode (160 ms) 2021-07-31T02:30:37.408 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMirroring.MirrorModeSet_PoolMode_To_ImageMode 2021-07-31T02:30:37.564 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMirroring.MirrorModeSet_PoolMode_To_ImageMode (156 ms) 2021-07-31T02:30:37.564 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMirroring.MirrorModeSet_ImageMode_To_PoolMode 2021-07-31T02:30:37.730 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMirroring.MirrorModeSet_ImageMode_To_PoolMode (167 ms) 2021-07-31T02:30:37.730 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMirroring.MirrorModeSet_ImageMode_To_DisabledMode 2021-07-31T02:30:37.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:37 smithi045 conmon[39718]: cluster 2021-07-31T02:30:36. 2021-07-31T02:30:37.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:37 smithi045 conmon[39718]: 418466+0000 mgr.x (mgr.24542) 1862 : cluster [DBG] pgmap v2402: 41 pgs: 41 active+clean; 4.6 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 433 KiB/s rd, 58 KiB/s wr, 624 op/s 2021-07-31T02:30:37.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:37 smithi045 conmon[39718]: cluster 2021-07-31 2021-07-31T02:30:37.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:37 smithi045 conmon[39718]: T02:30:36.508846 2021-07-31T02:30:37.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:37 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2021-07-31T02:30:37.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:37 smithi045 conmon[39718]: 2193 : cluster [DBG] osdmap e1255: 8 total, 8 up, 8 in 2021-07-31T02:30:37.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:37 smithi045 conmon[39718]: cluster 2021 2021-07-31T02:30:37.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:37 smithi045 conmon[39718]: -07-31T02:30:37 2021-07-31T02:30:37.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:37 smithi045 conmon[39718]: .295400+0000 mon.a (mon.0) 2194 2021-07-31T02:30:37.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:37 smithi045 conmon[39718]: : cluster [DBG] osdmap e1256: 8 total, 8 up, 8 in 2021-07-31T02:30:37.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:37 smithi045 conmon[33655]: cluster 2021-07-31T02: 2021-07-31T02:30:37.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:37 smithi045 conmon[33655]: 30:36.418466+0000 mgr.x (mgr.24542) 1862 : cluster [DBG] pgmap v2402: 41 pgs: 41 active+clean; 4.6 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 433 KiB/s rd, 58 KiB/s wr, 624 op/s 2021-07-31T02:30:37.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:37 smithi045 conmon[33655]: cluster 2021-07-31T02:30:37.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:37 smithi045 conmon[33655]: 2021-07-31T 2021-07-31T02:30:37.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:37 smithi045 conmon[33655]: 02:30:36.508846+0000 mon.a 2021-07-31T02:30:37.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:37 smithi045 conmon[33655]: (mon.0) 2193 : cluster [DBG] osdmap e1255: 8 total, 8 up, 8 in 2021-07-31T02:30:37.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:37 smithi045 conmon[33655]: cluster 2021-07-31T02:30:37.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:37 smithi045 conmon[33655]: 2021-07-31T02:30:37. 2021-07-31T02:30:37.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:37 smithi045 conmon[33655]: 295400+0000 mon.a (mon.0) 2194 : cluster [DBG] osdmap e1256: 8 total, 8 up, 8 in 2021-07-31T02:30:37.908 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMirroring.MirrorModeSet_ImageMode_To_DisabledMode (177 ms) 2021-07-31T02:30:37.909 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMirroring.RemoveImage_With_MirrorImageEnabled 2021-07-31T02:30:37.992 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMirroring.RemoveImage_With_MirrorImageEnabled (85 ms) 2021-07-31T02:30:37.993 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMirroring.RemoveImage_With_MirrorImageDisabled 2021-07-31T02:30:38.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:37 smithi154 conmon[34553]: cluster 2021-07-31T02:30:36.418466+0000 mgr.x (mgr.24542) 1862 : 2021-07-31T02:30:38.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:37 smithi154 conmon[34553]: cluster [DBG] pgmap v2402: 41 pgs: 41 active+clean; 4.6 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 433 KiB/s rd, 58 KiB/s wr, 624 op/s 2021-07-31T02:30:38.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:37 smithi154 conmon[34553]: cluster 2021-07-31T02:30:36.508846+0000 mon.a (mon.0) 2193 : cluster [DBG] osdmap e1255: 8 total, 8 up, 8 in 2021-07-31T02:30:38.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:37 smithi154 conmon[34553]: cluster 2021-07-31T02 2021-07-31T02:30:38.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:37 smithi154 conmon[34553]: :30:37.295400+0000 mon.a (mon.0) 2194 : cluster [DBG] osdmap e1256: 8 total, 8 up, 8 in 2021-07-31T02:30:38.075 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMirroring.RemoveImage_With_MirrorImageDisabled (83 ms) 2021-07-31T02:30:38.075 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMirroring.RemoveImage_With_ImageWithoutJournal 2021-07-31T02:30:38.150 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMirroring.RemoveImage_With_ImageWithoutJournal (75 ms) 2021-07-31T02:30:38.150 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMirroring.RemoveImage_With_MirrorImageDemoted 2021-07-31T02:30:38.280 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMirroring.RemoveImage_With_MirrorImageDemoted (130 ms) 2021-07-31T02:30:38.280 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMirroring.TrashMoveRestore_PoolMode 2021-07-31T02:30:38.409 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMirroring.TrashMoveRestore_PoolMode (128 ms) 2021-07-31T02:30:38.411 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMirroring.TrashMoveRestore_ImageMode_MirroringDisabled 2021-07-31T02:30:38.520 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMirroring.TrashMoveRestore_ImageMode_MirroringDisabled (111 ms) 2021-07-31T02:30:38.520 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMirroring.TrashMoveRestore_ImageMode_MirroringEnabled 2021-07-31T02:30:38.643 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMirroring.TrashMoveRestore_ImageMode_MirroringEnabled (124 ms) 2021-07-31T02:30:38.643 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMirroring.MirrorStatusList 2021-07-31T02:30:38.722 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMirroring.MirrorStatusList (79 ms) 2021-07-31T02:30:38.723 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMirroring.RemoveBootstrapped 2021-07-31T02:30:38.828 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMirroring.RemoveBootstrapped (105 ms) 2021-07-31T02:30:38.828 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMirroring.AioPromoteDemote 2021-07-31T02:30:39.223 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMirroring.AioPromoteDemote (396 ms) 2021-07-31T02:30:39.223 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMirroring.AioGetInfo 2021-07-31T02:30:39.609 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMirroring.AioGetInfo (385 ms) 2021-07-31T02:30:39.609 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMirroring.AioGetStatus 2021-07-31T02:30:39.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:39 smithi045 conmon[39718]: cluster 2021-07-31T02:30 2021-07-31T02:30:39.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:39 smithi045 conmon[39718]: :38.418834+0000 mgr.x (mgr.24542) 1863 : cluster [DBG] pgmap v2405: 41 pgs: 41 active+clean; 4.6 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:30:39.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:39 smithi045 conmon[33655]: cluster 2021-07-31T02:30:38.418834 2021-07-31T02:30:39.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:39 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 1863 : cluster [DBG] pgmap v2405: 41 pgs: 41 active+clean; 4.6 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:30:39.847 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMirroring.AioGetStatus (238 ms) 2021-07-31T02:30:39.847 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMirroring.SiteName 2021-07-31T02:30:39.855 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMirroring.SiteName (9 ms) 2021-07-31T02:30:39.855 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMirroring.Bootstrap 2021-07-31T02:30:40.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:39 smithi154 conmon[34553]: cluster 2021-07-31T02:30:38.418834+0000 mgr.x 2021-07-31T02:30:40.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:39 smithi154 conmon[34553]: (mgr.24542) 1863 : cluster [DBG] pgmap v2405: 41 pgs: 41 active+clean; 4.6 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:30:40.757 INFO:tasks.workunit.client.0.smithi045.stderr:2021-07-31T02:30:40.757+0000 7f11989adc00 -1 auth: unable to find a keyring on /etc/ceph/.client.rbd-mirror-peer.keyring,/etc/ceph/.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory 2021-07-31T02:30:40.758 INFO:tasks.workunit.client.0.smithi045.stderr:2021-07-31T02:30:40.757+0000 7f11989adc00 -1 auth: unable to find a keyring on /etc/ceph/.client.rbd-mirror-peer.keyring,/etc/ceph/.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory 2021-07-31T02:30:40.758 INFO:tasks.workunit.client.0.smithi045.stderr:2021-07-31T02:30:40.757+0000 7f11989adc00 -1 auth: unable to find a keyring on /etc/ceph/.client.rbd-mirror-peer.keyring,/etc/ceph/.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory 2021-07-31T02:30:40.761 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMirroring.Bootstrap (906 ms) 2021-07-31T02:30:40.761 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMirroring.PeerDirection 2021-07-31T02:30:40.770 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMirroring.PeerDirection (9 ms) 2021-07-31T02:30:40.770 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMirroring.Snapshot 2021-07-31T02:30:40.771 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:30:40.771 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMirroring.Snapshot (0 ms) 2021-07-31T02:30:40.771 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMirroring.SnapshotRemoveOnDisable 2021-07-31T02:30:40.771 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:30:40.772 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMirroring.SnapshotRemoveOnDisable (0 ms) 2021-07-31T02:30:40.772 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMirroring.SnapshotUnlinkPeer 2021-07-31T02:30:40.772 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:30:40.772 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMirroring.SnapshotUnlinkPeer (0 ms) 2021-07-31T02:30:40.772 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMirroring.SnapshotImageState 2021-07-31T02:30:40.773 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:30:40.773 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMirroring.SnapshotImageState (0 ms) 2021-07-31T02:30:40.773 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMirroring.SnapshotPromoteDemote 2021-07-31T02:30:40.773 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:30:40.773 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMirroring.SnapshotPromoteDemote (0 ms) 2021-07-31T02:30:40.774 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMirroring.AioSnapshotCreate 2021-07-31T02:30:40.774 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:30:40.774 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMirroring.AioSnapshotCreate (0 ms) 2021-07-31T02:30:41.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:40 smithi154 conmon[34553]: audit 2021-07-31T02:30:39.852105+0000 mon.a (mon.0 2021-07-31T02:30:41.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:40 smithi154 conmon[34553]: ) 2195 : audit [INF] from='client.? ' entity='client.admin' 2021-07-31T02:30:41.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:40 smithi154 conmon[34553]: audit 2021-07-31T02:30:39.852529+0000 mon.c (mon.1) 108 : audit [DBG] from='client.? 172.21.15.45:0/3641716911' entity='client.admin' cmd=[{"prefix": "config-key get", "key": "rbd/mirror/site_name"}]: dispatch 2021-07-31T02:30:41.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:40 smithi154 conmon[34553]: audit 2021-07-31T02:30:39.852800+0000 2021-07-31T02:30:41.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:40 smithi154 conmon[34553]: mon.c (mon.1) 109 : audit [INF] from='client.? 172.21.15.45:0/3641716911' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/site_name"}]: dispatch 2021-07-31T02:30:41.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:40 smithi154 conmon[34553]: 2021-07-31T02:30:41.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:40 smithi154 conmon[34553]: audit 2021-07-31T02:30:39. 2021-07-31T02:30:41.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:40 smithi154 conmon[34553]: 853035+0000 mon.a (mon.0) 2196 : audit 2021-07-31T02:30:41.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:40 smithi154 conmon[34553]: [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/site_name"}]: dispatch 2021-07-31T02:30:41.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:40 smithi154 conmon[34553]: audit 2021-07-31T 2021-07-31T02:30:41.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:40 smithi154 conmon[34553]: 02:30:39.856062+0000 mon.a (mon. 2021-07-31T02:30:41.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:40 smithi154 conmon[34553]: 0) 2197 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/site_name"}]': finished 2021-07-31T02:30:41.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:40 smithi154 conmon[34553]: audit 2021- 2021-07-31T02:30:41.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:40 smithi154 conmon[34553]: 07-31T02:30:39.856417+0000 mon.c 2021-07-31T02:30:41.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:40 smithi154 conmon[34553]: (mon.1) 110 : audit [DBG] from='client.? 172.21.15.45:0/3641716911' entity='client.admin' cmd=[{"prefix": "config-key get", "key": "rbd/mirror/site_name"}]: dispatch 2021-07-31T02:30:41.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:40 smithi154 conmon[34553]: audit 2021-07-31T 2021-07-31T02:30:41.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:40 smithi154 conmon[34553]: 02:30:40.745385+0000 mon.c (mon.1) 111 2021-07-31T02:30:41.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:40 smithi154 conmon[34553]: : audit [DBG] from='client.? 172.21.15.45:0/3641716911' entity='client.admin' cmd=[{"prefix": "config-key get", "key": "rbd/mirror/peer_client_id"}]: dispatch 2021-07-31T02:30:41.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:40 smithi154 conmon[34553]: audit 2021-07 2021-07-31T02:30:41.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:40 smithi154 conmon[34553]: -31T02:30:40.749487+0000 mon.a 2021-07-31T02:30:41.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:40 smithi154 conmon[34553]: (mon.0) 2198 : audit [INF] from='client.? ' entity='client.admin' 2021-07-31T02:30:41.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:40 smithi154 conmon[34553]: audit 2021-07- 2021-07-31T02:30:41.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:40 smithi154 conmon[34553]: 31T02:30:40.749874+0000 mon.c (mon.1 2021-07-31T02:30:41.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:40 smithi154 conmon[34553]: ) 112 : audit [INF] from='client.? 172.21.15.45:0/3641716911' entity='client.admin' cmd=[{ "prefix": "auth get-or-create", "entity": "client.rbd-mirror-peer", "caps": [ "mon", "profile rbd-mirror-peer", "osd", "profile rbd"], "format": "json"}]: dispatch 2021-07-31T02:30:41.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:40 smithi154 conmon[34553]: audit 2021 2021-07-31T02:30:41.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:40 smithi154 conmon[34553]: -07-31T02:30:40.750132+0000 2021-07-31T02:30:41.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:40 smithi154 conmon[34553]: mon.a (mon.0) 2199 : audit [INF] from='client.? ' entity='client.admin' cmd=[{ "prefix": "auth get-or-create", "entity": "client.rbd-mirror-peer", "caps": [ "mon", "profile rbd-mirror-peer", "osd", "profile rbd"], "format": "json"}]: dispatch 2021-07-31T02:30:41.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:40 smithi154 conmon[34553]: 2021-07-31T02:30:41.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:40 smithi154 conmon[34553]: audit 2021-07-31T02:30: 2021-07-31T02:30:41.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:40 smithi154 conmon[34553]: 40.753583+0000 mon.a (mon.0) 2200 : 2021-07-31T02:30:41.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:40 smithi154 conmon[34553]: audit [INF] from='client.? ' entity='client.admin' cmd='[{ "prefix": "auth get-or-create", "entity": "client.rbd-mirror-peer", "caps": [ "mon", "profile rbd-mirror-peer", "osd", "profile rbd"], "format": "json"}]': finished 2021-07-31T02:30:41.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:40 smithi154 conmon[34553]: audit 2021-07-31 2021-07-31T02:30:41.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:40 smithi154 conmon[34553]: T02:30:40.754061+0000 mon.c (mon 2021-07-31T02:30:41.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:40 smithi154 conmon[34553]: .1) 113 : audit [DBG] from='client.? 172.21.15.45:0/3641716911' entity='client.admin' cmd=[{"prefix": "config-key get", "key": "rbd/mirror/site_name"}]: dispatch 2021-07-31T02:30:41.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:40 smithi154 conmon[34553]: audit 2021 2021-07-31T02:30:41.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:40 smithi154 conmon[34553]: -07-31T02:30:40.766329+0000 2021-07-31T02:30:41.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:40 smithi154 conmon[34553]: mon.c (mon.1) 114 : audit [INF] from='client.? 172.21.15.45:0/3641716911' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/72/e31a11fc-97db-4bbf-9de5-98dae0b134e0"}]: dispatch 2021-07-31T02:30:41.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:40 smithi154 conmon[34553]: 2021-07-31T02:30:41.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:40 smithi154 conmon[34553]: audit 2021-07-31T02:30:40.766610+0000 mon.a 2021-07-31T02:30:41.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:40 smithi154 conmon[34553]: (mon.0) 2201 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/72/e31a11fc-97db-4bbf-9de5-98dae0b134e0"}]: dispatch 2021-07-31T02:30:41.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:40 smithi154 conmon[34553]: audit 2021-07 2021-07-31T02:30:41.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:40 smithi154 conmon[34553]: -31T02:30:40.769217+0000 mon.a (mon.0 2021-07-31T02:30:41.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:40 smithi154 conmon[34553]: ) 2202 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/72/e31a11fc-97db-4bbf-9de5-98dae0b134e0"}]': finished 2021-07-31T02:30:41.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[39718]: audit 2021-07-31T02:30:39.852105+0000 mon.a 2021-07-31T02:30:41.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[39718]: (mon.0) 2195 : audit [INF] from='client.? ' entity='client.admin' 2021-07-31T02:30:41.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T02:30:41.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[39718]: T02:30:39.852529+0000 mon.c (mon.1) 2021-07-31T02:30:41.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[39718]: 108 : audit [DBG] from='client.? 172.21.15.45:0/3641716911' entity='client.admin' cmd=[{"prefix": "config-key get", "key": "rbd/mirror/site_name"}]: dispatch 2021-07-31T02:30:41.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[39718]: audit 2021-07-31T02:30:41.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[39718]: 2021-07-31T02: 2021-07-31T02:30:41.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[39718]: 30:39.852800+0000 mon.c (mon. 2021-07-31T02:30:41.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[39718]: 1) 109 : audit [INF] from='client.? 172.21.15.45:0/3641716911' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/site_name"}]: dispatch 2021-07-31T02:30:41.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T02:30:41.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[39718]: 31T02:30:39.853035+0000 2021-07-31T02:30:41.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[39718]: mon.a (mon.0) 2196 : audit 2021-07-31T02:30:41.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[39718]: [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/site_name"}]: dispatch 2021-07-31T02:30:41.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[39718]: audit 2021-07 2021-07-31T02:30:41.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[39718]: -31T02:30:39 2021-07-31T02:30:41.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[39718]: .856062+0000 mon.a (mon.0) 2197 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/site_name"}]': finished 2021-07-31T02:30:41.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[39718]: audit 2021-07-31T02:30:41.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[39718]: 2021-07-31T02 2021-07-31T02:30:41.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[39718]: :30:39.856417+0000 mon.c 2021-07-31T02:30:41.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[39718]: (mon.1) 110 : audit [DBG] 2021-07-31T02:30:41.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[39718]: from='client.? 172.21.15.45:0/3641716911' entity='client.admin' cmd=[{"prefix": "config-key get", "key": "rbd/mirror/site_name"}]: dispatch 2021-07-31T02:30:41.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T02:30:41.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[39718]: 31T02:30:40. 2021-07-31T02:30:41.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[39718]: 745385+0000 mon.c (mon.1) 111 : 2021-07-31T02:30:41.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[39718]: audit [DBG] from='client.? 172.21.15.45:0/3641716911' entity='client.admin' cmd=[{"prefix": "config-key get", "key": "rbd/mirror/peer_client_id"}]: dispatch 2021-07-31T02:30:41.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[39718]: audit 2021 2021-07-31T02:30:41.282 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[39718]: -07-31T02:30 2021-07-31T02:30:41.282 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[39718]: :40.749487+0000 mon.a (mon.0 2021-07-31T02:30:41.282 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[39718]: ) 2198 : audit [INF] from='client.? ' entity='client.admin' 2021-07-31T02:30:41.282 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[39718]: audit 2021-07-31T02:30:41.282 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[39718]: 2021-07-31T02: 2021-07-31T02:30:41.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[39718]: 30:40.749874+0000 mon.c (mon. 2021-07-31T02:30:41.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[39718]: 1) 112 : audit [INF] from='client.? 172.21.15.45:0/3641716911' entity='client.admin' cmd=[{ "prefix": "auth get-or-create", "entity": "client.rbd-mirror-peer", "caps": [ "mon", "profile rbd-mirror-peer", "osd", "profile rbd"], "format": "json"}]: dispatch 2021-07-31T02:30:41.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[39718]: audit 2021 2021-07-31T02:30:41.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[39718]: -07-31T02:30:40.750132+0000 mon.a ( 2021-07-31T02:30:41.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[39718]: mon.0) 2199 : audit [INF] from='client.? ' entity='client.admin' cmd=[{ "prefix": "auth get-or-create", "entity": "client.rbd-mirror-peer", "caps": [ "mon", "profile rbd-mirror-peer", "osd", "profile rbd"], "format": "json"}]: dispatch 2021-07-31T02:30:41.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[39718]: audit 2021-07-31T02:30:41.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[39718]: 2021-07-31T02: 2021-07-31T02:30:41.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[39718]: 30:40.753583+0000 mon.a 2021-07-31T02:30:41.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[39718]: (mon.0) 2200 : audit [INF] 2021-07-31T02:30:41.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[39718]: from='client.? ' entity='client.admin' cmd='[{ "prefix": "auth get-or-create", "entity": "client.rbd-mirror-peer", "caps": [ "mon", "profile rbd-mirror-peer", "osd", "profile rbd"], "format": "json"}]': finished 2021-07-31T02:30:41.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[39718]: audit 2021- 2021-07-31T02:30:41.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[39718]: 07-31T02:30: 2021-07-31T02:30:41.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[39718]: 40.754061+0000 mon.c (mon.1) 2021-07-31T02:30:41.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[39718]: 113 : audit [DBG] from='client.? 172.21.15.45:0/3641716911' entity='client.admin' cmd=[{"prefix": "config-key get", "key": "rbd/mirror/site_name"}]: dispatch 2021-07-31T02:30:41.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[39718]: audit 2021-07-31T02:30:41.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[39718]: 2021-07-31T02:30 2021-07-31T02:30:41.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[39718]: :40.766329+0000 mon.c (mon.1) 114 : audit [INF] 2021-07-31T02:30:41.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[39718]: from='client.? 172.21.15.45:0/3641716911' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/72/e31a11fc-97db-4bbf-9de5-98dae0b134e0"}]: dispatch 2021-07-31T02:30:41.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[39718]: audit 2021-07-31T02:30:41.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[39718]: 2021-07-31T02:30:40.766610+0000 mon.a (mon 2021-07-31T02:30:41.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[39718]: .0) 2201 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/72/e31a11fc-97db-4bbf-9de5-98dae0b134e0"}]: dispatch 2021-07-31T02:30:41.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[39718]: audit 2021- 2021-07-31T02:30:41.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[39718]: 07-31T02:30: 2021-07-31T02:30:41.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[39718]: 40.769217+0000 mon.a (mon.0 2021-07-31T02:30:41.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[39718]: ) 2202 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/72/e31a11fc-97db-4bbf-9de5-98dae0b134e0"}]': finished 2021-07-31T02:30:41.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[33655]: audit 2021-07-31T02:30:39.852105+0000 mon.a (mon.0) 2195 : audit [INF] from='client.? ' entity='client.admin' 2021-07-31T02:30:41.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[33655]: audit 2021-07-31T02:30:39.852529+0000 mon.c (mon.1) 108 : audit [DBG] from='client.? 172.21.15.45:0/3641716911' entity='client.admin' cmd=[{"prefix": "config-key get", "key": "rbd/mirror/site_name"}]: dispatch 2021-07-31T02:30:41.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[33655]: audit 2021-07-31T 2021-07-31T02:30:41.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[33655]: 02:30:39.852800+0000 mon.c (mon.1) 109 : audit [INF] from='client.? 172.21.15.45:0/3641716911' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/site_name"}]: dispatch 2021-07-31T02:30:41.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[33655]: audit 2021-07- 2021-07-31T02:30:41.290 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[33655]: 31T02:30:39.853035+0000 mon.a (mon.0) 2196 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/site_name"}]: dispatch 2021-07-31T02:30:41.290 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[33655]: audit 2021-07-31T02: 2021-07-31T02:30:41.290 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[33655]: 30:39.856062+0000 mon.a (mon.0) 2197 : audit 2021-07-31T02:30:41.290 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[33655]: [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/site_name"}]': finished 2021-07-31T02:30:41.290 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[33655]: audit 2021-07-31T02:30:41.291 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[33655]: 2021-07-31T02:30:39.856417+0000 mon.c (mon.1) 110 : audit 2021-07-31T02:30:41.291 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[33655]: [DBG] from='client.? 172.21.15.45:0/3641716911' entity='client.admin' cmd=[{"prefix": "config-key get", "key": "rbd/mirror/site_name"}]: dispatch 2021-07-31T02:30:41.291 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[33655]: audit 2021-07-31T02:30:41.291 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[33655]: 2021-07-31T02:30:40.745385+0000 mon.c (mon.1) 111 : 2021-07-31T02:30:41.292 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[33655]: audit [DBG] from='client.? 172.21.15.45:0/3641716911' entity='client.admin' cmd=[{"prefix": "config-key get", "key": "rbd/mirror/peer_client_id"}]: dispatch 2021-07-31T02:30:41.292 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[33655]: audit 2021-07-31T02:30:41.292 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[33655]: 2021-07-31T02:30:40.749487+0000 mon.a (mon.0) 2198 : 2021-07-31T02:30:41.292 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[33655]: audit [INF] from='client.? ' entity='client.admin' 2021-07-31T02:30:41.292 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[33655]: audit 2021-07-31T02:30:41.293 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[33655]: 2021-07-31T02:30:40.749874+0000 mon.c (mon.1) 112 : audit 2021-07-31T02:30:41.293 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[33655]: [INF] from='client.? 172.21.15.45:0/3641716911' entity='client.admin' cmd=[{ "prefix": "auth get-or-create", "entity": "client.rbd-mirror-peer", "caps": [ "mon", "profile rbd-mirror-peer", "osd", "profile rbd"], "format": "json"}]: dispatch 2021-07-31T02:30:41.293 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[33655]: audit 2021-07-31T02:30:40.750132 2021-07-31T02:30:41.293 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[33655]: +0000 mon.a (mon.0) 2199 : audit [INF] from='client.? ' entity='client.admin' cmd=[{ "prefix": "auth get-or-create", "entity": "client.rbd-mirror-peer", "caps": [ "mon", "profile rbd-mirror-peer", "osd", "profile rbd"], "format": "json"}]: dispatch 2021-07-31T02:30:41.293 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[33655]: audit 2021-07- 2021-07-31T02:30:41.294 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[33655]: 31T02:30:40.753583+0000 mon.a (mon.0) 2200 2021-07-31T02:30:41.294 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[33655]: : audit [INF] from='client.? ' entity='client.admin' cmd='[{ "prefix": "auth get-or-create", "entity": "client.rbd-mirror-peer", "caps": [ "mon", "profile rbd-mirror-peer", "osd", "profile rbd"], "format": "json"}]': finished 2021-07-31T02:30:41.294 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[33655]: audit 2021-07-31T02:30:41.294 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[33655]: 2021-07-31T02:30:40.754061+0000 mon.c (mon.1) 2021-07-31T02:30:41.295 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[33655]: 113 : audit [DBG] from='client.? 172.21.15.45:0/3641716911' entity='client.admin' cmd=[{"prefix": "config-key get", "key": "rbd/mirror/site_name"}]: dispatch 2021-07-31T02:30:41.295 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[33655]: audit 2021-07-31T02:30:40.766329+0000 2021-07-31T02:30:41.295 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[33655]: mon.c (mon.1) 114 : audit [INF] from='client.? 172.21.15.45:0/3641716911' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/72/e31a11fc-97db-4bbf-9de5-98dae0b134e0"}]: dispatch 2021-07-31T02:30:41.295 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[33655]: audit 2021-07-31T02 2021-07-31T02:30:41.295 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[33655]: :30:40.766610+0000 mon.a (mon.0) 2201 2021-07-31T02:30:41.296 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[33655]: : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/72/e31a11fc-97db-4bbf-9de5-98dae0b134e0"}]: dispatch 2021-07-31T02:30:41.296 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[33655]: audit 2021-07-31T 2021-07-31T02:30:41.296 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[33655]: 02:30:40.769217+0000 mon.a (mon.0) 2021-07-31T02:30:41.296 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:40 smithi045 conmon[33655]: 2202 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/72/e31a11fc-97db-4bbf-9de5-98dae0b134e0"}]': finished 2021-07-31T02:30:41.775 INFO:tasks.workunit.client.0.smithi045.stdout:[----------] 51 tests from TestMirroring (12310 ms total) 2021-07-31T02:30:41.776 INFO:tasks.workunit.client.0.smithi045.stdout: 2021-07-31T02:30:41.776 INFO:tasks.workunit.client.0.smithi045.stdout:[----------] 31 tests from TestDeepCopy 2021-07-31T02:30:42.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:42 smithi045 conmon[33655]: cluster 2021-07-31T02:30:40.419280+0000 mgr.x (mgr. 2021-07-31T02:30:42.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:42 smithi045 conmon[33655]: 24542) 1864 : cluster [DBG] pgmap v2406: 41 pgs: 41 active+clean; 4.6 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 863 KiB/s rd, 122 KiB/s wr, 1.23k op/s 2021-07-31T02:30:42.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:42 smithi045 conmon[33655]: cluster 2021-07-31T02:30:41.772584+0000 mon.a (mon.0) 2203 : cluster [DBG] osdmap e1257: 8 total, 8 up, 8 in 2021-07-31T02:30:42.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:42 smithi045 conmon[39718]: cluster 2021-07-31T02:30:40.419280+0000 mgr.x ( 2021-07-31T02:30:42.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:42 smithi045 conmon[39718]: mgr.24542) 1864 : cluster [DBG] pgmap v2406: 41 pgs: 41 active+clean; 4.6 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 863 KiB/s rd, 122 KiB/s wr, 1.23k op/s 2021-07-31T02:30:42.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:42 smithi045 conmon[39718]: cluster 2021-07-31T02:30:41.772584+0000 2021-07-31T02:30:42.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:42 smithi045 conmon[39718]: mon.a (mon.0) 2203 : cluster [DBG] osdmap e1257: 8 total, 8 up, 8 in 2021-07-31T02:30:42.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:42 smithi154 conmon[34553]: cluster 2021-07-31T02:30:40.419280+0000 mgr.x (mgr.24542) 1864 : cluster 2021-07-31T02:30:42.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:42 smithi154 conmon[34553]: [DBG] pgmap v2406: 41 pgs: 41 active+clean; 4.6 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 863 KiB/s rd, 122 KiB/s wr, 1.23k op/s 2021-07-31T02:30:42.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:42 smithi154 conmon[34553]: cluster 2021-07-31T02:30:41.772584+0000 mon.a (mon.0) 2203 : cluster [DBG] osdmap e1257: 8 total, 8 up, 8 in 2021-07-31T02:30:43.777 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestDeepCopy.Empty 2021-07-31T02:30:44.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:43 smithi045 conmon[33655]: cluster 2021-07-31T02:30:42.419740+0000 mgr.x (mgr.24542 2021-07-31T02:30:44.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:43 smithi045 conmon[33655]: ) 1865 : cluster [DBG] pgmap v2408: 9 pgs: 9 active+clean; 4.6 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:30:44.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:43 smithi045 conmon[33655]: cluster 2021-07-31T02:30:42.774464+0000 mon.a (mon.0) 2204 : cluster [DBG] osdmap e1258: 8 total, 8 up, 8 in 2021-07-31T02:30:44.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:43 smithi045 conmon[33655]: audit 2021-07-31T02:30:42.778966+0000 mon.a (mon.0) 2205 : audit [INF] from='client.? 172.21.15.45:0/3306901976' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:30:44.027 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:30:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:30:43.584Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:30:44.028 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:30:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:30:43.584Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:30:44.028 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:30:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:30:43.584Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:30:44.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:43 smithi045 conmon[39718]: cluster 2021-07-31T02:30:44.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:43 smithi045 conmon[39718]: 2021-07-31T02:30:44.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:43 smithi045 conmon[39718]: 2021-07-31T02:30: 2021-07-31T02:30:44.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:43 smithi045 conmon[39718]: 42.419740+0000 mgr.x (mgr.24542) 1865 : cluster [DBG] pgmap v2408: 9 pgs: 9 active+clean; 4.6 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:30:44.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:43 smithi045 conmon[39718]: cluster 2021-07-31T02:30:44.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:43 smithi045 conmon[39718]: 2021-07-31T02:30:42.774464+0000 mon.a ( 2021-07-31T02:30:44.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:43 smithi045 conmon[39718]: mon.0) 2204 : cluster [DBG] osdmap e1258: 8 total, 8 up, 8 in 2021-07-31T02:30:44.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:43 smithi045 conmon[39718]: audit 2021-07-31T02:30:44.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:43 smithi045 conmon[39718]: 2021-07-31T02:30:42.778966+0000 mon.a (mon.0) 2205 : audit [INF] from='client.? 172.21.15.45:0/3306901976' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:30:44.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:43 smithi154 conmon[34553]: cluster 2021-07-31T02:30:42.419740+0000 mgr.x (mgr. 2021-07-31T02:30:44.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:43 smithi154 conmon[34553]: 24542) 1865 : cluster [DBG] pgmap v2408: 9 pgs: 9 active+clean; 4.6 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:30:44.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:43 smithi154 conmon[34553]: cluster 2021-07-31T02:30:42.774464+0000 mon.a (mon.0) 2204 : cluster [DBG] osdmap e1258: 8 total, 8 up, 8 in 2021-07-31T02:30:44.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:43 smithi154 conmon[34553]: audit 2021-07-31T02:30:44.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:43 smithi154 conmon[34553]: 2021-07-31T02:30:42.778966+0000 mon.a (mon.0) 2205 : audit [INF] from='client.? 172.21.15.45:0/3306901976' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:30:44.780 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:30:45.059 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:30:45.060 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 45m ago 46m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:30:45.060 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 43m ago 46m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:30:45.060 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 43m ago 49m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:30:45.060 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 45m ago 51m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:30:45.061 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 45m ago 51m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:30:45.061 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 43m ago 49m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:30:45.061 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 45m ago 50m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:30:45.061 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 45m ago 46m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:30:45.061 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 43m ago 46m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:30:45.062 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 45m ago 49m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:30:45.062 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 45m ago 49m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:30:45.062 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 45m ago 48m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:30:45.062 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 45m ago 48m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:30:45.063 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 43m ago 48m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:30:45.063 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 43m ago 47m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:30:45.063 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 43m ago 47m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:30:45.063 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 43m ago 47m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:30:45.064 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 43m ago 46m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:30:45.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:44 smithi154 conmon[34553]: audit 2021-07-31T02:30:43.776232+0000 mon.a (mon.0) 2021-07-31T02:30:45.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:44 smithi154 conmon[34553]: 2206 : audit [INF] from='client.? 172.21.15.45:0/3306901976' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-11","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:30:45.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:44 smithi154 conmon[34553]: cluster 2021-07-31T02:30:43.776403+0000 mon.a (mon.0) 2207 : cluster [DBG] osdmap e1259: 8 total, 8 up, 8 in 2021-07-31T02:30:45.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:44 smithi045 conmon[33655]: audit 2021-07-31T02:30:43. 2021-07-31T02:30:45.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:44 smithi045 conmon[33655]: 776232+0000 mon.a (mon.0) 2206 : audit [INF] from='client.? 172.21.15.45:0/3306901976' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-11","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:30:45.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:44 smithi045 conmon[33655]: cluster 2021-07-31T02:30:43.776403+0000 mon.a (mon.0) 2207 : cluster [DBG] osdmap e1259: 8 total, 8 up, 8 in 2021-07-31T02:30:45.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:44 smithi045 conmon[39718]: audit 2021-07-31T02:30: 2021-07-31T02:30:45.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:44 smithi045 conmon[39718]: 43.776232+0000 mon.a (mon.0) 2206 : audit [INF] from='client.? 172.21.15.45:0/3306901976' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-11","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:30:45.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:44 smithi045 conmon[39718]: cluster 2021-07-31T02:30: 2021-07-31T02:30:45.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:44 smithi045 conmon[39718]: 43.776403+0000 mon.a (mon.0) 2207 : cluster 2021-07-31T02:30:45.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:44 smithi045 conmon[39718]: [DBG] osdmap e1259: 8 total, 8 up, 8 in 2021-07-31T02:30:45.396 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:30:45.397 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:30:45.397 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:30:45.397 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:30:45.398 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:30:45.398 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:30:45.398 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:30:45.398 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:30:45.399 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:30:45.399 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:30:45.399 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:30:45.399 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:30:45.400 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:30:45.400 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:30:45.400 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:30:45.400 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:30:45.401 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:30:46.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:45 smithi154 conmon[34553]: cluster 2021-07-31T02:30:44.420366+0000 mgr.x (mgr.24542) 1866 : cluster [DBG] pgmap v2411: 41 pgs: 41 active+clean; 4.6 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:30:46.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:45 smithi154 conmon[34553]: 2021-07-31T02:30:46.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:45 smithi154 conmon[34553]: audit 2021-07-31T02:30:44.768761+0000 mgr.x (mgr.24542) 1867 : audit [DBG] from='client.27250 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:30:46.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:45 smithi154 conmon[34553]: cluster 2021-07-31T02:30:44.787609+0000 mon.a (mon.0) 2208 : cluster [DBG] osdmap e1260: 8 total, 8 up, 8 in 2021-07-31T02:30:46.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:45 smithi154 conmon[34553]: audit 2021-07-31T02:30:45.051764+0000 mgr.x (mgr.24542) 1868 : audit [DBG] from='client.27256 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:30:46.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:45 smithi154 conmon[34553]: audit 2021-07-31T02:30:45.396894+0000 mon.a (mon 2021-07-31T02:30:46.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:45 smithi154 conmon[34553]: .0) 2209 : audit [DBG] from='client.? 172.21.15.45:0/3406304914' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:30:46.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:45 smithi045 conmon[39718]: cluster 2021-07-31T02:30:46.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:45 smithi045 conmon[39718]: 2021-07-31T02:30:46.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:45 smithi045 conmon[39718]: 2021-07-31T02:30:44. 2021-07-31T02:30:46.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:45 smithi045 conmon[39718]: 420366+0000 mgr.x (mgr.24542) 1866 : cluster [DBG] pgmap v2411: 41 pgs: 41 active+clean; 4.6 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:30:46.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:45 smithi045 conmon[39718]: audit 2021-07-31T02: 2021-07-31T02:30:46.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:45 smithi045 conmon[39718]: 30:44.768761+0000 mgr.x (mgr.24542) 1867 : audit 2021-07-31T02:30:46.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:45 smithi045 conmon[39718]: [DBG] from='client.27250 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:30:46.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:45 smithi045 conmon[39718]: cluster 2021-07-31T02:30 2021-07-31T02:30:46.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:45 smithi045 conmon[39718]: :44.787609+0000 mon.a (mon.0) 2208 : cluster [DBG] 2021-07-31T02:30:46.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:45 smithi045 conmon[39718]: osdmap e1260: 8 total, 8 up, 8 in 2021-07-31T02:30:46.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:45 smithi045 conmon[39718]: audit 2021-07-31T02:30:46.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:45 smithi045 conmon[39718]: 2021-07-31T02:30:45.051764+0000 mgr.x (mgr 2021-07-31T02:30:46.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:45 smithi045 conmon[39718]: .24542) 1868 : audit [DBG] from='client.27256 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:30:46.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:45 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T02:30:46.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:45 smithi045 conmon[39718]: T02:30:45.396894+0000 2021-07-31T02:30:46.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:45 smithi045 conmon[39718]: mon.a (mon.0) 2209 2021-07-31T02:30:46.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:45 smithi045 conmon[39718]: : audit [DBG] from='client.? 172.21.15.45:0/3406304914' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:30:46.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:45 smithi045 conmon[33655]: cluster 2021-07-31T02:30:44.420366 2021-07-31T02:30:46.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:45 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 1866 : cluster [DBG] pgmap v2411: 41 pgs: 41 active+clean; 4.6 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:30:46.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:45 smithi045 conmon[33655]: audit 2021-07-31T02:30:44.768761 2021-07-31T02:30:46.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:45 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 1867 : audit [DBG] from='client.27250 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:30:46.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:45 smithi045 conmon[33655]: cluster 2021-07-31T02:30:44.787609 2021-07-31T02:30:46.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:45 smithi045 conmon[33655]: +0000 mon.a (mon.0) 2208 : cluster [DBG] osdmap e1260: 8 total, 8 up, 8 in 2021-07-31T02:30:46.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:45 smithi045 conmon[33655]: audit 2021-07-31T02:30:45. 2021-07-31T02:30:46.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:45 smithi045 conmon[33655]: 051764+0000 mgr.x (mgr.24542) 1868 : audit 2021-07-31T02:30:46.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:45 smithi045 conmon[33655]: [DBG] from='client.27256 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:30:46.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:45 smithi045 conmon[33655]: audit 2021-07-31T02:30 2021-07-31T02:30:46.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:45 smithi045 conmon[33655]: :45.396894+0000 mon.a (mon.0) 2021-07-31T02:30:46.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:45 smithi045 conmon[33655]: 2209 : audit [DBG] from='client.? 172.21.15.45:0/3406304914' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:30:47.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:46 smithi154 conmon[34553]: cluster 2021-07-31T02:30:45.789522+0000 mon.a (mon.0) 2210 : cluster 2021-07-31T02:30:47.377 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:46 smithi154 conmon[34553]: [DBG] osdmap e1261: 8 total, 8 up, 8 in 2021-07-31T02:30:47.378 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:46 smithi045 conmon[39718]: cluster 2021-07-31T02:30:45.789522+0000 mon.a ( 2021-07-31T02:30:47.379 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:46 smithi045 conmon[39718]: mon.0) 2210 : cluster [DBG] osdmap e1261: 8 total, 8 up, 8 in 2021-07-31T02:30:47.379 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:46 smithi045 conmon[33655]: cluster 2021-07-31T02:30:45.789522+0000 mon.a (mon 2021-07-31T02:30:47.380 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:46 smithi045 conmon[33655]: .0) 2210 : cluster [DBG] osdmap e1261: 8 total, 8 up, 8 in 2021-07-31T02:30:47.896 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestDeepCopy.Empty (4119 ms) 2021-07-31T02:30:47.896 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestDeepCopy.NoSnaps 2021-07-31T02:30:48.109 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestDeepCopy.NoSnaps (214 ms) 2021-07-31T02:30:48.109 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestDeepCopy.Snaps 2021-07-31T02:30:48.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:47 smithi154 conmon[34553]: cluster 2021-07-31T02:30:46.420889+0000 mgr.x (mgr.24542) 1869 : cluster [DBG] pgmap v2414: 41 pgs: 41 active+clean; 4.6 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:30:48.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:47 smithi154 conmon[34553]: cluster 2021-07-31T02:30:46.800125+0000 mon.a (mon.0) 2211 : cluster [DBG] osdmap e1262: 8 total, 8 up, 8 in 2021-07-31T02:30:48.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:47 smithi045 conmon[39718]: cluster 2021-07-31T02:30:46.420889+0000 mgr.x 2021-07-31T02:30:48.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:47 smithi045 conmon[39718]: (mgr.24542) 1869 : cluster [DBG] pgmap v2414: 41 pgs: 41 active+clean; 4.6 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:30:48.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:47 smithi045 conmon[39718]: cluster 2021-07-31T02:30: 2021-07-31T02:30:48.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:47 smithi045 conmon[39718]: 46.800125+0000 mon.a (mon.0) 2211 : cluster [DBG] osdmap e1262: 8 total, 8 up, 8 in 2021-07-31T02:30:48.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:47 smithi045 conmon[33655]: cluster 2021-07-31T02:30:46.420889 2021-07-31T02:30:48.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:47 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 1869 : cluster [DBG] pgmap v2414: 41 pgs: 41 active+clean; 4.6 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:30:48.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:47 smithi045 conmon[33655]: cluster 2021-07-31T02:30:46.800125+0000 mon.a (mon.0) 2211 : cluster [DBG] 2021-07-31T02:30:48.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:47 smithi045 conmon[33655]: osdmap e1262: 8 total, 8 up, 8 in 2021-07-31T02:30:49.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:48 smithi154 conmon[34553]: cluster 2021-07-31T02:30:47.801235+0000 mon.a (mon. 2021-07-31T02:30:49.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:48 smithi154 conmon[34553]: 0) 2212 : cluster [DBG] osdmap e1263: 8 total, 8 up, 8 in 2021-07-31T02:30:49.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:48 smithi045 conmon[39718]: cluster 2021-07-31T02:30:47.801235 2021-07-31T02:30:49.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:48 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2212 : cluster [DBG] osdmap e1263: 8 total, 8 up, 8 in 2021-07-31T02:30:49.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:48 smithi045 conmon[33655]: cluster 2021-07-31T02:30:47.801235+0000 mon.a ( 2021-07-31T02:30:49.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:48 smithi045 conmon[33655]: mon.0) 2212 : cluster [DBG] osdmap e1263: 8 total, 8 up, 8 in 2021-07-31T02:30:50.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:49 smithi154 conmon[34553]: cluster 2021-07-31T02:30:48.421234+0000 mgr.x (mgr.24542) 1870 : cluster [DBG] pgmap v2417: 41 pgs: 41 active+clean; 4.6 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:30:50.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:49 smithi154 conmon[34553]: cluster 2021-07-31T02:30:48.815454+0000 mon.a (mon.0) 2213 : cluster [DBG] osdmap e1264: 8 total, 8 up, 8 in 2021-07-31T02:30:50.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:49 smithi045 conmon[39718]: cluster 2021-07-31T02:30:48.421234+0000 2021-07-31T02:30:50.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:49 smithi045 conmon[39718]: mgr.x (mgr.24542) 1870 : cluster [DBG] pgmap v2417: 41 pgs: 41 active+clean; 4.6 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:30:50.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:49 smithi045 conmon[39718]: cluster 2021-07-31T02:30:48 2021-07-31T02:30:50.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:49 smithi045 conmon[39718]: .815454+0000 mon.a (mon.0) 2213 : cluster [DBG] osdmap e1264: 8 total, 8 up, 8 in 2021-07-31T02:30:50.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:49 smithi045 conmon[33655]: cluster 2021-07-31T02:30:48.421234+0000 mgr.x (mgr.24542) 1870 : cluster [DBG] 2021-07-31T02:30:50.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:49 smithi045 conmon[33655]: pgmap v2417: 41 pgs: 41 active+clean; 4.6 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:30:50.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:49 smithi045 conmon[33655]: cluster 2021-07-31T02:30:48.815454+0000 mon.a (mon.0) 2213 : cluster [DBG] osdmap e1264: 8 total, 8 up, 8 in 2021-07-31T02:30:51.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:50 smithi154 conmon[34553]: cluster 2021-07-31T02:30:49.820204+0000 mon.a (mon.0) 2214 : cluster [DBG] osdmap e1265: 8 total, 8 up, 8 in 2021-07-31T02:30:51.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:50 smithi154 conmon[34553]: 2021-07-31T02:30:51.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:50 smithi045 conmon[33655]: cluster 2021-07-31T02:30:49.820204+0000 mon.a (mon.0) 2214 2021-07-31T02:30:51.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:50 smithi045 conmon[33655]: : cluster [DBG] osdmap e1265: 8 total, 8 up, 8 in 2021-07-31T02:30:51.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:50 smithi045 conmon[39718]: cluster 2021-07-31T02:30:49.820204+0000 mon.a (mon.0) 2214 : cluster 2021-07-31T02:30:51.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:50 smithi045 conmon[39718]: [DBG] osdmap e1265: 8 total, 8 up, 8 in 2021-07-31T02:30:52.111 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestDeepCopy.Snaps (4001 ms) 2021-07-31T02:30:52.111 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestDeepCopy.SnapDiscard 2021-07-31T02:30:52.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:51 smithi154 conmon[34553]: cluster 2021-07-31T02:30:50.421785+0000 mgr.x (mgr.24542) 1871 : 2021-07-31T02:30:52.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:51 smithi154 conmon[34553]: cluster [DBG] pgmap v2420: 41 pgs: 41 active+clean; 53 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 12 MiB/s rd, 12 MiB/s wr, 53 op/s 2021-07-31T02:30:52.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:51 smithi154 conmon[34553]: cluster 2021-07-31T02:30:50.825974+0000 mon.a (mon.0) 2215 : cluster [DBG] osdmap e1266: 8 total, 8 up, 8 in 2021-07-31T02:30:52.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:51 smithi045 conmon[39718]: cluster 2021-07-31T02:30:50.421785+0000 mgr.x (mgr.24542) 2021-07-31T02:30:52.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:51 smithi045 conmon[39718]: 1871 : cluster [DBG] pgmap v2420: 41 pgs: 41 active+clean; 53 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 12 MiB/s rd, 12 MiB/s wr, 53 op/s 2021-07-31T02:30:52.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:51 smithi045 conmon[39718]: cluster 2021-07-31T02:30:50.825974+0000 mon.a (mon.0) 2215 : cluster [DBG] osdmap e1266: 8 total, 8 up, 8 in 2021-07-31T02:30:52.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:51 smithi045 conmon[33655]: cluster 2021-07-31T02:30:50.421785+0000 mgr.x 2021-07-31T02:30:52.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:51 smithi045 conmon[33655]: (mgr.24542) 1871 : cluster [DBG] pgmap v2420: 41 pgs: 41 active+clean; 53 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 12 MiB/s rd, 12 MiB/s wr, 53 op/s 2021-07-31T02:30:52.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:51 smithi045 conmon[33655]: cluster 2021-07-31T02:30:50.825974+0000 mon.a (mon.0) 2215 : cluster [DBG] osdmap e1266: 8 total, 8 up, 8 in 2021-07-31T02:30:53.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:52 smithi154 conmon[34553]: cluster 2021-07-31T02:30:51.827330+0000 mon.a (mon.0 2021-07-31T02:30:53.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:52 smithi154 conmon[34553]: ) 2216 : cluster [DBG] osdmap e1267: 8 total, 8 up, 8 in 2021-07-31T02:30:53.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:52 smithi154 conmon[34553]: cluster 2021-07-31T02:30:52.298894+0000 mon.a (mon.0) 2217 : cluster [DBG] osdmap e1268: 8 total, 8 up, 8 in 2021-07-31T02:30:53.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:52 smithi045 conmon[39718]: cluster 2021-07-31T02:30:51.827330 2021-07-31T02:30:53.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:52 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2216 : cluster [DBG] osdmap e1267: 8 total, 8 up, 8 in 2021-07-31T02:30:53.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:52 smithi045 conmon[39718]: cluster 2021-07-31T02:30:52 2021-07-31T02:30:53.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:52 smithi045 conmon[39718]: .298894+0000 mon.a (mon.0 2021-07-31T02:30:53.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:52 smithi045 conmon[39718]: ) 2217 : cluster [DBG] osdmap e1268: 8 total, 8 up, 8 in 2021-07-31T02:30:53.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:52 smithi045 conmon[33655]: cluster 2021-07-31T02:30:51.827330+0000 mon.a (mon 2021-07-31T02:30:53.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:52 smithi045 conmon[33655]: .0) 2216 : cluster [DBG] osdmap e1267: 8 total, 8 up, 8 in 2021-07-31T02:30:53.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:52 smithi045 conmon[33655]: cluster 2021-07-31T02:30:53.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:52 smithi045 conmon[33655]: 2021-07-31T02:30:52.298894+0000 mon.a (mon.0 2021-07-31T02:30:53.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:52 smithi045 conmon[33655]: ) 2217 : cluster [DBG] osdmap e1268: 8 total, 8 up, 8 in 2021-07-31T02:30:53.439 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestDeepCopy.SnapDiscard (1329 ms) 2021-07-31T02:30:53.440 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestDeepCopy.CloneDiscard 2021-07-31T02:30:53.452 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:30:53.517 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestDeepCopy.CloneDiscard (78 ms) 2021-07-31T02:30:53.518 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestDeepCopy.CloneShrink 2021-07-31T02:30:53.532 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:30:53.594 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestDeepCopy.CloneShrink (77 ms) 2021-07-31T02:30:53.595 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestDeepCopy.CloneExpand 2021-07-31T02:30:53.606 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:30:53.669 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestDeepCopy.CloneExpand (75 ms) 2021-07-31T02:30:53.669 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestDeepCopy.CloneHideParent 2021-07-31T02:30:53.680 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:30:53.741 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestDeepCopy.CloneHideParent (72 ms) 2021-07-31T02:30:53.742 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestDeepCopy.Clone 2021-07-31T02:30:53.753 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:30:53.820 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestDeepCopy.Clone (79 ms) 2021-07-31T02:30:53.821 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestDeepCopy.CloneFlatten 2021-07-31T02:30:53.831 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:30:53.894 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestDeepCopy.CloneFlatten (74 ms) 2021-07-31T02:30:53.895 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestDeepCopy.Stress 2021-07-31T02:30:53.908 INFO:tasks.workunit.client.0.smithi045.stdout:write: A 35016510~5912965 2021-07-31T02:30:53.924 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 2596065~7052774 2021-07-31T02:30:53.926 INFO:tasks.workunit.client.0.smithi045.stdout:write: A 20900327~6624999 2021-07-31T02:30:53.943 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 3400641~6221521 2021-07-31T02:30:53.945 INFO:tasks.workunit.client.0.smithi045.stdout:write: A 44058891~2192327 2021-07-31T02:30:53.955 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 37473439~4893970 2021-07-31T02:30:53.957 INFO:tasks.workunit.client.0.smithi045.stdout:write: A 3378839~1576167 2021-07-31T02:30:53.962 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 12699990~2425041 2021-07-31T02:30:53.962 INFO:tasks.workunit.client.0.smithi045.stdout:snap: snap0 2021-07-31T02:30:54.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:30:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:30:53.584Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:30:54.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:30:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:30:53.585Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:30:54.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:30:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:30:53.585Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:30:54.318 INFO:tasks.workunit.client.0.smithi045.stdout:write: B 43674060~2794669 2021-07-31T02:30:54.319 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 68893112~473074 2021-07-31T02:30:54.322 INFO:tasks.workunit.client.0.smithi045.stdout:write: B 10259379~8229659 2021-07-31T02:30:54.323 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 55633848~3345797 2021-07-31T02:30:54.325 INFO:tasks.workunit.client.0.smithi045.stdout:write: B 55230582~4415786 2021-07-31T02:30:54.337 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 16496065~6599994 2021-07-31T02:30:54.355 INFO:tasks.workunit.client.0.smithi045.stdout:write: B 81410823~2475257 2021-07-31T02:30:54.355 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 71046200~5631814 2021-07-31T02:30:54.361 INFO:tasks.workunit.client.0.smithi045.stdout:snap: snap1 2021-07-31T02:30:54.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:54 smithi154 conmon[34553]: cluster 2021-07-31T02:30:52.422176+0000 mgr.x (mgr.24542) 2021-07-31T02:30:54.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:54 smithi154 conmon[34553]: 1872 : cluster [DBG] pgmap v2424: 41 pgs: 41 active+clean; 53 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 13 MiB/s rd, 13 MiB/s wr, 59 op/s 2021-07-31T02:30:54.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:54 smithi154 conmon[34553]: cluster 2021-07-31T02:30:53.308830+0000 mon.a (mon.0) 2218 : cluster [DBG] osdmap e1269: 8 total, 8 up, 8 in 2021-07-31T02:30:54.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:54 smithi045 conmon[33655]: cluster 2021-07-31T02:30:52.422176+0000 mgr.x (mgr.24542) 1872 : cluster [DBG] pgmap v2424: 41 pgs: 41 active+clean; 53 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 13 MiB/s rd, 13 MiB/s wr, 59 op/s 2021-07-31T02:30:54.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:54 smithi045 conmon[33655]: 2021-07-31T02:30:54.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:54 smithi045 conmon[33655]: cluster 2021-07-31T02:30:53.308830+0000 mon.a (mon.0) 2218 : cluster [DBG] osdmap e1269: 8 total, 8 up, 8 in 2021-07-31T02:30:54.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:54 smithi045 conmon[39718]: cluster 2021-07-31T02:30:52.422176+0000 2021-07-31T02:30:54.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:54 smithi045 conmon[39718]: mgr.x (mgr.24542) 1872 : cluster [DBG] pgmap v2424: 41 pgs: 41 active+clean; 53 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 13 MiB/s rd, 13 MiB/s wr, 59 op/s 2021-07-31T02:30:54.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:54 smithi045 conmon[39718]: cluster 2021-07-31T02:30:53.308830+0000 2021-07-31T02:30:54.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:54 smithi045 conmon[39718]: mon.a (mon.0) 2218 : cluster [DBG] osdmap e1269: 8 total, 8 up, 8 in 2021-07-31T02:30:55.325 INFO:tasks.workunit.client.0.smithi045.stdout:write: C 52505375~8227880 2021-07-31T02:30:55.327 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 6060793~448245 2021-07-31T02:30:55.330 INFO:tasks.workunit.client.0.smithi045.stdout:write: C 11029180~5574369 2021-07-31T02:30:55.330 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 44022771~8176644 2021-07-31T02:30:55.333 INFO:tasks.workunit.client.0.smithi045.stdout:write: C 18141420~6559579 2021-07-31T02:30:55.339 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 17373229~5458570 2021-07-31T02:30:55.371 INFO:tasks.workunit.client.0.smithi045.stdout:write: C 9562752~5675586 2021-07-31T02:30:55.371 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 57738254~3390682 2021-07-31T02:30:55.391 INFO:tasks.workunit.client.0.smithi045.stdout:snap: snap2 2021-07-31T02:30:55.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:55 smithi154 conmon[34553]: cluster 2021-07-31T02:30:54.313463+0000 mon.a (mon.0) 2219 : cluster 2021-07-31T02:30:55.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:55 smithi154 conmon[34553]: [DBG] osdmap e1270: 8 total, 8 up, 8 in 2021-07-31T02:30:55.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:55 smithi045 conmon[33655]: cluster 2021-07-31T02:30:54.313463+0000 mon.a (mon.0) 2219 2021-07-31T02:30:55.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:55 smithi045 conmon[33655]: : cluster [DBG] osdmap e1270: 8 total, 8 up, 8 in 2021-07-31T02:30:55.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:55 smithi045 conmon[39718]: cluster 2021-07-31T02:30:54.313463+0000 2021-07-31T02:30:55.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:55 smithi045 conmon[39718]: mon.a (mon.0) 2219 : cluster [DBG] osdmap e1270: 8 total, 8 up, 8 in 2021-07-31T02:30:56.333 INFO:tasks.workunit.client.0.smithi045.stdout:write: D 65967913~5174930 2021-07-31T02:30:56.334 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 33686551~3517989 2021-07-31T02:30:56.336 INFO:tasks.workunit.client.0.smithi045.stdout:write: D 58265423~4161249 2021-07-31T02:30:56.337 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 61092891~5031054 2021-07-31T02:30:56.360 INFO:tasks.workunit.client.0.smithi045.stdout:write: D 74615175~7652624 2021-07-31T02:30:56.360 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 21673047~33603 2021-07-31T02:30:56.361 INFO:tasks.workunit.client.0.smithi045.stdout:write: D 46690385~3055039 2021-07-31T02:30:56.361 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 63949023~4735103 2021-07-31T02:30:56.385 INFO:tasks.workunit.client.0.smithi045.stdout:snap: snap3 2021-07-31T02:30:56.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:56 smithi154 conmon[34553]: cluster 2021-07-31T02:30:54.422709+0000 mgr.x (mgr.24542) 1873 : cluster [DBG] pgmap v2427: 41 pgs: 41 active+clean; 69 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 27 MiB/s rd, 4.5 MiB/s wr, 196 op/s 2021-07-31T02:30:56.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:56 smithi154 conmon[34553]: cluster 2021-07-31T02:30:55.319994+0000 mon.a (mon.0) 2220 : cluster [DBG] osdmap e1271: 8 total, 8 up, 8 in 2021-07-31T02:30:56.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:56 smithi045 conmon[39718]: cluster 2021-07-31T02:30:54 2021-07-31T02:30:56.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:56 smithi045 conmon[39718]: .422709+0000 mgr.x (mgr.24542) 1873 : cluster [DBG] pgmap v2427: 41 pgs: 41 active+clean; 69 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 27 MiB/s rd, 4.5 MiB/s wr, 196 op/s 2021-07-31T02:30:56.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:56 smithi045 conmon[39718]: cluster 2021-07-31T02:30:56.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:56 smithi045 conmon[39718]: 2021-07-31T02:30:55.319994+0000 2021-07-31T02:30:56.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:56 smithi045 conmon[39718]: mon.a (mon.0) 2220 : 2021-07-31T02:30:56.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:56 smithi045 conmon[39718]: cluster [DBG] osdmap e1271: 8 total, 8 up, 8 in 2021-07-31T02:30:56.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:56 smithi045 conmon[33655]: cluster 2021-07-31T02: 2021-07-31T02:30:56.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:56 smithi045 conmon[33655]: 30:54.422709+0000 mgr.x (mgr.24542) 1873 : cluster [DBG] pgmap v2427: 41 pgs: 41 active+clean; 69 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 27 MiB/s rd, 4.5 MiB/s wr, 196 op/s 2021-07-31T02:30:56.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:56 smithi045 conmon[33655]: cluster 2021-07-31T02:30:56.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:56 smithi045 conmon[33655]: 2021-07-31T02:30:55.319994+0000 mon.a (mon. 2021-07-31T02:30:56.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:56 smithi045 conmon[33655]: 0) 2220 : cluster [DBG] osdmap e1271: 8 total, 8 up, 8 in 2021-07-31T02:30:57.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:57 smithi154 conmon[34553]: cluster 2021-07-31T02:30:56.326823+0000 mon.a (mon.0) 2221 : cluster [DBG] 2021-07-31T02:30:57.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:57 smithi154 conmon[34553]: osdmap e1272: 8 total, 8 up, 8 in 2021-07-31T02:30:57.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:57 smithi045 conmon[33655]: cluster 2021-07-31T02 2021-07-31T02:30:57.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:57 smithi045 conmon[33655]: :30:56.326823+0000 mon.a (mon.0) 2221 : cluster [DBG] osdmap e1272: 8 total, 8 up, 8 in 2021-07-31T02:30:57.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:57 smithi045 conmon[39718]: cluster 2021-07-31T02:30:56 2021-07-31T02:30:57.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:57 smithi045 conmon[39718]: .326823+0000 mon.a (mon.0) 2221 : cluster [DBG] osdmap e1272: 8 total, 8 up, 8 in 2021-07-31T02:30:58.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:58 smithi154 conmon[34553]: cluster 2021-07-31T02:30:58.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:58 smithi154 conmon[34553]: 2021-07-31T02:30:58.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:58 smithi154 conmon[34553]: 2021-07-31T02:30:56.423265+0000 mgr.x (mgr 2021-07-31T02:30:58.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:58 smithi154 conmon[34553]: .24542) 1874 : cluster [DBG] pgmap v2430: 41 pgs: 41 active+clean; 69 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 24 MiB/s rd, 4.0 MiB/s wr, 176 op/s 2021-07-31T02:30:58.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:58 smithi154 conmon[34553]: cluster 2021 2021-07-31T02:30:58.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:58 smithi154 conmon[34553]: -07-31T02:30:57.332996+0000 mon.a (mon.0) 2222 : cluster [DBG] osdmap e1273: 8 total, 8 up, 8 in 2021-07-31T02:30:58.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:58 smithi045 conmon[33655]: cluster 2021-07-31T02:30:56.423265+0000 mgr.x (mgr.24542) 1874 : cluster [DBG] pgmap v2430: 41 pgs: 41 active+clean; 69 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 24 MiB/s rd, 4.0 MiB/s wr, 176 op/s 2021-07-31T02:30:58.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:58 smithi045 conmon[33655]: cluster 2021-07-31T02:30:57.332996+0000 mon.a (mon.0) 2222 : cluster [DBG] osdmap e1273: 8 total, 8 up, 8 in 2021-07-31T02:30:58.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:58 smithi045 conmon[39718]: cluster 2021-07-31T02:30:56.423265+0000 mgr.x 2021-07-31T02:30:58.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:58 smithi045 conmon[39718]: (mgr.24542) 1874 : cluster [DBG] pgmap v2430: 41 pgs: 41 active+clean; 69 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 24 MiB/s rd, 4.0 MiB/s wr, 176 op/s 2021-07-31T02:30:58.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:58 smithi045 conmon[39718]: cluster 2021-07-31T02:30:57.332996+0000 2021-07-31T02:30:58.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:58 smithi045 conmon[39718]: mon.a (mon.0) 2222 : cluster [DBG] osdmap e1273: 8 total, 8 up, 8 in 2021-07-31T02:30:59.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:59 smithi154 conmon[34553]: cluster 2021-07-31T02:30:58.334637+0000 mon.a (mon.0) 2223 : cluster [DBG] 2021-07-31T02:30:59.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:30:59 smithi154 conmon[34553]: osdmap e1274: 8 total, 8 up, 8 in 2021-07-31T02:30:59.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:59 smithi045 conmon[39718]: cluster 2021-07-31T02:30:58.334637 2021-07-31T02:30:59.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:30:59 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2223 : cluster [DBG] osdmap e1274: 8 total, 8 up, 8 in 2021-07-31T02:30:59.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:59 smithi045 conmon[33655]: cluster 2021-07-31T02:30:58.334637+0000 mon.a (mon.0) 2223 : 2021-07-31T02:30:59.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:30:59 smithi045 conmon[33655]: cluster [DBG] osdmap e1274: 8 total, 8 up, 8 in 2021-07-31T02:31:00.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:00 smithi154 conmon[34553]: cluster 2021-07-31T02:30:58.423655+0000 mgr.x (mgr.24542) 1875 : cluster [DBG] 2021-07-31T02:31:00.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:00 smithi154 conmon[34553]: pgmap v2433: 41 pgs: 41 active+clean; 69 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:31:00.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:00 smithi154 conmon[34553]: cluster 2021-07-31T02:30:59.344814+0000 mon.a (mon.0) 2224 : cluster [DBG] osdmap e1275: 8 total, 8 up, 8 in 2021-07-31T02:31:00.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:00 smithi045 conmon[33655]: cluster 2021-07-31T02:30:58.423655+0000 mgr.x (mgr.24542) 1875 : cluster [DBG] 2021-07-31T02:31:00.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:00 smithi045 conmon[33655]: pgmap v2433: 41 pgs: 41 active+clean; 69 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:31:00.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:00 smithi045 conmon[33655]: cluster 2021-07-31T02:30:59.344814+0000 mon.a (mon.0) 2224 : cluster [DBG] osdmap e1275: 8 total, 8 up, 8 in 2021-07-31T02:31:00.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:00 smithi045 conmon[39718]: cluster 2021-07-31T02:30:58 2021-07-31T02:31:00.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:00 smithi045 conmon[39718]: .423655+0000 mgr.x (mgr.24542) 1875 : cluster [DBG] pgmap v2433: 41 pgs: 41 active+clean; 69 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:31:00.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:00 smithi045 conmon[39718]: cluster 2021-07-31T02:30:59.344814+0000 mon.a 2021-07-31T02:31:00.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:00 smithi045 conmon[39718]: (mon.0) 2224 : cluster [DBG] 2021-07-31T02:31:00.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:00 smithi045 conmon[39718]: osdmap e1275: 8 total, 8 up, 8 in 2021-07-31T02:31:01.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:01 smithi154 conmon[34553]: cluster 2021-07-31T02:31:00.346671+0000 mon.a (mon.0) 2225 : cluster [DBG] 2021-07-31T02:31:01.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:01 smithi154 conmon[34553]: osdmap e1276: 8 total, 8 up, 8 in 2021-07-31T02:31:01.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:01 smithi045 conmon[39718]: cluster 2021-07-31T02:31:00.346671+0000 mon.a 2021-07-31T02:31:01.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:01 smithi045 conmon[39718]: (mon.0) 2225 : cluster [DBG] osdmap e1276: 8 total, 8 up, 8 in 2021-07-31T02:31:01.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:01 smithi045 conmon[33655]: cluster 2021-07-31T02:31:00.346671+0000 mon.a (mon.0) 2225 : cluster 2021-07-31T02:31:01.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:01 smithi045 conmon[33655]: [DBG] osdmap e1276: 8 total, 8 up, 8 in 2021-07-31T02:31:01.955 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestDeepCopy.Stress (8061 ms) 2021-07-31T02:31:01.955 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestDeepCopy.NoSnaps_LargerDstObjSize 2021-07-31T02:31:02.205 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestDeepCopy.NoSnaps_LargerDstObjSize (250 ms) 2021-07-31T02:31:02.206 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestDeepCopy.Snaps_LargerDstObjSize 2021-07-31T02:31:02.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:02 smithi154 conmon[34553]: cluster 2021-07-31T02:31:00.424095+0000 mgr.x (mgr.24542) 1876 2021-07-31T02:31:02.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:02 smithi154 conmon[34553]: : cluster [DBG] pgmap v2436: 41 pgs: 41 active+clean; 156 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 14 KiB/s rd, 19 MiB/s wr, 35 op/s 2021-07-31T02:31:02.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:02 smithi154 conmon[34553]: cluster 2021-07-31T02:31:01.356967+0000 mon.a (mon.0) 2226 : cluster [DBG] osdmap e1277: 8 total, 8 up, 8 in 2021-07-31T02:31:02.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:02 smithi045 conmon[39718]: cluster 2021-07-31T02:31:00.424095 2021-07-31T02:31:02.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:02 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 1876 : cluster [DBG] pgmap v2436: 41 pgs: 41 active+clean; 156 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 14 KiB/s rd, 19 MiB/s wr, 35 op/s 2021-07-31T02:31:02.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:02 smithi045 conmon[39718]: cluster 2021-07-31T02:31:02.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:02 smithi045 conmon[39718]: 2021-07-31 2021-07-31T02:31:02.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:02 smithi045 conmon[39718]: T02:31:01.356967+0000 2021-07-31T02:31:02.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:02 smithi045 conmon[39718]: mon.a (mon.0) 2226 2021-07-31T02:31:02.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:02 smithi045 conmon[39718]: : cluster [DBG] osdmap e1277: 8 total, 8 up, 8 in 2021-07-31T02:31:02.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:02 smithi045 conmon[33655]: cluster 2021-07-31T02:31:00.424095+0000 mgr.x (mgr.24542 2021-07-31T02:31:02.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:02 smithi045 conmon[33655]: ) 1876 : cluster [DBG] pgmap v2436: 41 pgs: 41 active+clean; 156 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 14 KiB/s rd, 19 MiB/s wr, 35 op/s 2021-07-31T02:31:02.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:02 smithi045 conmon[33655]: cluster 2021-07-31T02:31:01.356967+0000 mon.a (mon.0) 2226 : cluster [DBG] osdmap e1277: 8 total, 8 up, 8 in 2021-07-31T02:31:03.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:03 smithi154 conmon[34553]: cluster 2021-07-31T02:31:02.361052+0000 mon.a (mon.0) 2227 : cluster [DBG] osdmap e1278: 8 total, 8 up, 8 in 2021-07-31T02:31:03.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:03 smithi045 conmon[33655]: cluster 2021-07-31T02:31:02.361052+0000 mon.a 2021-07-31T02:31:03.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:03 smithi045 conmon[33655]: (mon.0) 2227 : cluster [DBG] osdmap e1278: 8 total, 8 up, 8 in 2021-07-31T02:31:03.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:03 smithi045 conmon[39718]: cluster 2021-07-31T02:31:02.361052+0000 mon.a (mon.0 2021-07-31T02:31:03.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:03 smithi045 conmon[39718]: ) 2227 : cluster [DBG] osdmap e1278: 8 total, 8 up, 8 in 2021-07-31T02:31:03.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:31:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:31:03.585Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:31:03.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:31:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:31:03.585Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:31:03.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:31:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:31:03.585Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:31:04.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:04 smithi154 conmon[34553]: cluster 2021-07-31T02:31:02.424573+0000 mgr.x (mgr.24542) 1877 : cluster [DBG] pgmap v2439: 41 pgs: 41 active+clean; 156 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 14 KiB/s rd, 19 MiB/s wr, 35 op/s 2021-07-31T02:31:04.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:04 smithi154 conmon[34553]: cluster 2021-07-31T02:31:03.367725+0000 mon.a (mon.0) 2228 : cluster [DBG] osdmap e1279: 8 total, 8 up, 8 in 2021-07-31T02:31:04.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:04 smithi045 conmon[33655]: cluster 2021-07-31T02:31:02.424573+0000 mgr.x (mgr.24542) 1877 : cluster [DBG] pgmap v2439: 41 pgs: 41 active+clean; 156 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 14 KiB/s rd, 19 MiB/s wr, 35 op/s 2021-07-31T02:31:04.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:04 smithi045 conmon[33655]: 2021-07-31T02:31:04.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:04 smithi045 conmon[33655]: cluster 2021-07-31T02:31:03.367725+0000 mon.a (mon.0) 2228 : cluster [DBG] osdmap e1279: 8 total, 8 up, 8 in 2021-07-31T02:31:04.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:04 smithi045 conmon[39718]: cluster 2021-07-31T02:31:02 2021-07-31T02:31:04.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:04 smithi045 conmon[39718]: .424573+0000 mgr.x (mgr.24542) 1877 : cluster [DBG] pgmap v2439: 41 pgs: 41 active+clean; 156 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 14 KiB/s rd, 19 MiB/s wr, 35 op/s 2021-07-31T02:31:04.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:04 smithi045 conmon[39718]: cluster 2021-07-31 2021-07-31T02:31:04.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:04 smithi045 conmon[39718]: T02:31:03.367725+0000 2021-07-31T02:31:04.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:04 smithi045 conmon[39718]: mon.a (mon.0) 2228 : 2021-07-31T02:31:04.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:04 smithi045 conmon[39718]: cluster [DBG] osdmap e1279: 8 total, 8 up, 8 in 2021-07-31T02:31:05.652 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestDeepCopy.Snaps_LargerDstObjSize (3447 ms) 2021-07-31T02:31:05.652 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestDeepCopy.Clone_LargerDstObjSize 2021-07-31T02:31:05.663 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:31:05.728 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestDeepCopy.Clone_LargerDstObjSize (76 ms) 2021-07-31T02:31:05.728 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestDeepCopy.CloneFlatten_LargerDstObjSize 2021-07-31T02:31:05.738 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:31:05.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:05 smithi154 conmon[34553]: cluster 2021-07-31T02:31:04.369385+0000 mon.a (mon.0 2021-07-31T02:31:05.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:05 smithi154 conmon[34553]: ) 2229 : cluster [DBG] osdmap e1280: 8 total, 8 up, 8 in 2021-07-31T02:31:05.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:05 smithi154 conmon[34553]: cluster 2021-07-31T02:31:04.425186+0000 mgr.x (mgr.24542) 1878 : cluster [DBG] pgmap v2442: 41 pgs: 41 active+clean; 290 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 90 MiB/s rd, 26 MiB/s wr, 125 op/s 2021-07-31T02:31:05.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:05 smithi045 conmon[33655]: cluster 2021-07-31T02:31:04.369385+0000 mon.a (mon.0) 2021-07-31T02:31:05.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:05 smithi045 conmon[33655]: 2229 : cluster [DBG] osdmap e1280: 8 total, 8 up, 8 in 2021-07-31T02:31:05.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:05 smithi045 conmon[33655]: cluster 2021-07-31T02:31:04.425186+0000 mgr.x (mgr.24542) 1878 : cluster [DBG] pgmap v2442: 41 pgs: 41 active+clean; 290 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 90 MiB/s rd, 26 MiB/s wr, 125 op/s 2021-07-31T02:31:05.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:05 smithi045 conmon[39718]: cluster 2021-07-31T02:31:04.369385+0000 mon.a 2021-07-31T02:31:05.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:05 smithi045 conmon[39718]: (mon.0) 2229 : cluster [DBG] osdmap e1280: 8 total, 8 up, 8 in 2021-07-31T02:31:05.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:05 smithi045 conmon[39718]: cluster 2021-07-31T02:31:04.425186+0000 mgr.x (mgr.24542) 1878 : 2021-07-31T02:31:05.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:05 smithi045 conmon[39718]: cluster [DBG] pgmap v2442: 41 pgs: 41 active+clean; 290 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 90 MiB/s rd, 26 MiB/s wr, 125 op/s 2021-07-31T02:31:05.803 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestDeepCopy.CloneFlatten_LargerDstObjSize (75 ms) 2021-07-31T02:31:05.804 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestDeepCopy.Stress_LargerDstObjSize 2021-07-31T02:31:05.815 INFO:tasks.workunit.client.0.smithi045.stdout:write: A 66973369~1605221 2021-07-31T02:31:05.822 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 73889016~4385583 2021-07-31T02:31:05.825 INFO:tasks.workunit.client.0.smithi045.stdout:write: A 49846612~7112163 2021-07-31T02:31:05.849 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 69993668~6912271 2021-07-31T02:31:05.851 INFO:tasks.workunit.client.0.smithi045.stdout:write: A 11943325~7448739 2021-07-31T02:31:05.873 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 48655796~5257440 2021-07-31T02:31:05.875 INFO:tasks.workunit.client.0.smithi045.stdout:write: A 5291043~2672421 2021-07-31T02:31:05.883 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 47670500~3219979 2021-07-31T02:31:05.884 INFO:tasks.workunit.client.0.smithi045.stdout:snap: snap0 2021-07-31T02:31:06.389 INFO:tasks.workunit.client.0.smithi045.stdout:write: B 61287875~7955082 2021-07-31T02:31:06.390 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 52305410~6845370 2021-07-31T02:31:06.392 INFO:tasks.workunit.client.0.smithi045.stdout:write: B 39847357~4982147 2021-07-31T02:31:06.393 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 26253622~4169086 2021-07-31T02:31:06.394 INFO:tasks.workunit.client.0.smithi045.stdout:write: B 58060263~3911452 2021-07-31T02:31:06.416 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 2125991~2652019 2021-07-31T02:31:06.417 INFO:tasks.workunit.client.0.smithi045.stdout:write: B 72499248~186514 2021-07-31T02:31:06.417 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 51019~3731213 2021-07-31T02:31:06.426 INFO:tasks.workunit.client.0.smithi045.stdout:snap: snap1 2021-07-31T02:31:06.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:06 smithi154 conmon[34553]: cluster 2021-07-31T02:31:05.379774+0000 mon.a (mon.0) 2230 2021-07-31T02:31:06.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:06 smithi154 conmon[34553]: : cluster [DBG] osdmap e1281: 8 total, 8 up, 8 in 2021-07-31T02:31:06.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:06 smithi045 conmon[33655]: cluster 2021-07-31T02:31:05.379774+0000 mon.a (mon.0) 2230 : cluster 2021-07-31T02:31:06.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:06 smithi045 conmon[33655]: [DBG] osdmap e1281: 8 total, 8 up, 8 in 2021-07-31T02:31:06.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:06 smithi045 conmon[39718]: cluster 2021-07-31T02:31:05.379774+0000 mon.a ( 2021-07-31T02:31:06.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:06 smithi045 conmon[39718]: mon.0) 2230 : cluster [DBG] osdmap e1281: 8 total, 8 up, 8 in 2021-07-31T02:31:07.395 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 119563269 2021-07-31T02:31:07.403 INFO:tasks.workunit.client.0.smithi045.stdout:write: C 23638130~7163182 2021-07-31T02:31:07.403 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 108281266~646420 2021-07-31T02:31:07.405 INFO:tasks.workunit.client.0.smithi045.stdout:write: C 67064488~8351063 2021-07-31T02:31:07.405 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 71251638~6916819 2021-07-31T02:31:07.437 INFO:tasks.workunit.client.0.smithi045.stdout:write: C 70723278~6873559 2021-07-31T02:31:07.437 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 117202569~1506582 2021-07-31T02:31:07.438 INFO:tasks.workunit.client.0.smithi045.stdout:write: C 99613595~5469034 2021-07-31T02:31:07.439 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 56277781~6780031 2021-07-31T02:31:07.462 INFO:tasks.workunit.client.0.smithi045.stdout:snap: snap2 2021-07-31T02:31:07.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:07 smithi154 conmon[34553]: cluster 2021-07-31T02:31:06.384894+0000 mon.a (mon.0) 2231 : cluster 2021-07-31T02:31:07.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:07 smithi154 conmon[34553]: [DBG] osdmap e1282: 8 total, 8 up, 8 in 2021-07-31T02:31:07.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:07 smithi154 conmon[34553]: cluster 2021-07-31T02:31:06.425586+0000 mgr.x (mgr.24542) 1879 : cluster [DBG] pgmap v2445: 41 pgs: 41 active+clean; 290 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 90 MiB/s rd, 26 MiB/s wr, 125 op/s 2021-07-31T02:31:07.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:07 smithi045 conmon[39718]: cluster 2021-07-31T02:31:06.384894+0000 mon.a (mon 2021-07-31T02:31:07.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:07 smithi045 conmon[39718]: .0) 2231 : cluster [DBG] osdmap e1282: 8 total, 8 up, 8 in 2021-07-31T02:31:07.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:07 smithi045 conmon[39718]: cluster 2021-07-31T02:31:06.425586+0000 mgr.x (mgr.24542) 1879 : cluster [DBG] pgmap v2445: 41 pgs: 41 active+clean; 290 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 90 MiB/s rd, 26 MiB/s wr, 125 op/s 2021-07-31T02:31:07.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:07 smithi045 conmon[39718]: 2021-07-31T02:31:07.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:07 smithi045 conmon[33655]: cluster 2021-07-31T02:31:06.384894+0000 mon.a (mon.0) 2231 : cluster 2021-07-31T02:31:07.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:07 smithi045 conmon[33655]: [DBG] osdmap e1282: 8 total, 8 up, 8 in 2021-07-31T02:31:07.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:07 smithi045 conmon[33655]: cluster 2021-07-31T02:31:06.425586+0000 mgr.x (mgr.24542) 1879 : cluster [DBG] pgmap v2445: 41 pgs: 41 active+clean; 290 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 90 MiB/s rd, 26 MiB/s wr, 125 op/s 2021-07-31T02:31:08.400 INFO:tasks.workunit.client.0.smithi045.stdout:write: D 14156976~3373570 2021-07-31T02:31:08.402 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 112347365~7215904 2021-07-31T02:31:08.404 INFO:tasks.workunit.client.0.smithi045.stdout:write: D 102973693~5741566 2021-07-31T02:31:08.405 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 68403310~7113388 2021-07-31T02:31:08.407 INFO:tasks.workunit.client.0.smithi045.stdout:write: D 43730608~6742425 2021-07-31T02:31:08.410 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 84311254~3210162 2021-07-31T02:31:08.415 INFO:tasks.workunit.client.0.smithi045.stdout:write: D 3210162~5502524 2021-07-31T02:31:08.420 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 67043021~5568019 2021-07-31T02:31:08.445 INFO:tasks.workunit.client.0.smithi045.stdout:snap: snap3 2021-07-31T02:31:08.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:08 smithi154 conmon[34553]: cluster 2021-07-31T02:31:07.390561+0000 mon.a (mon.0) 2232 : cluster [DBG] osdmap e1283: 8 total, 8 up, 8 in 2021-07-31T02:31:08.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:08 smithi045 conmon[39718]: cluster 2021-07-31T02:31:07.390561+0000 mon.a (mon.0 2021-07-31T02:31:08.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:08 smithi045 conmon[39718]: ) 2232 : cluster [DBG] osdmap e1283: 8 total, 8 up, 8 in 2021-07-31T02:31:08.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:08 smithi045 conmon[33655]: cluster 2021-07-31T02:31:07.390561 2021-07-31T02:31:08.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:08 smithi045 conmon[33655]: +0000 mon.a (mon.0) 2232 : cluster [DBG] osdmap e1283: 8 total, 8 up, 8 in 2021-07-31T02:31:09.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:09 smithi154 conmon[34553]: cluster 2021-07-31T02:31:08.396773+0000 mon.a (mon.0) 2233 : cluster 2021-07-31T02:31:09.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:09 smithi154 conmon[34553]: [DBG] osdmap e1284: 8 total, 8 up, 8 in 2021-07-31T02:31:09.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:09 smithi154 conmon[34553]: cluster 2021-07-31T02:31:08.425999+0000 mgr.x (mgr.24542) 1880 : cluster [DBG] pgmap v2448: 41 pgs: 41 active+clean; 290 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail 2021-07-31T02:31:09.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:09 smithi045 conmon[33655]: cluster 2021-07-31T02:31:08.396773+0000 mon.a (mon.0) 2233 : cluster [DBG] 2021-07-31T02:31:09.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:09 smithi045 conmon[33655]: osdmap e1284: 8 total, 8 up, 8 in 2021-07-31T02:31:09.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:09 smithi045 conmon[33655]: cluster 2021-07-31T02:31:08.425999+0000 mgr.x (mgr.24542) 1880 : cluster [DBG] pgmap v2448: 41 pgs: 41 active+clean; 290 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail 2021-07-31T02:31:09.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:09 smithi045 conmon[39718]: cluster 2021-07-31T02:31:08.396773 2021-07-31T02:31:09.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:09 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2233 : cluster [DBG] osdmap e1284: 8 total, 8 up, 8 in 2021-07-31T02:31:09.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:09 smithi045 conmon[39718]: cluster 2021-07-31T02:31: 2021-07-31T02:31:09.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:09 smithi045 conmon[39718]: 08.425999+0000 mgr.x (mgr. 2021-07-31T02:31:09.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:09 smithi045 conmon[39718]: 24542) 1880 : cluster [DBG] pgmap v2448: 41 pgs: 41 active+clean; 290 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail 2021-07-31T02:31:09.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:09 smithi045 conmon[39718]: 2021-07-31T02:31:10.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:10 smithi154 conmon[34553]: cluster 2021-07-31T02:31:09.403324+0000 mon.a (mon.0) 2234 : cluster [DBG] osdmap e1285: 8 total, 8 up, 8 in 2021-07-31T02:31:10.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:10 smithi045 conmon[33655]: cluster 2021-07-31T02:31:09.403324+0000 mon.a (mon.0) 2234 : cluster [DBG] osdmap e1285: 8 total, 8 up, 8 in 2021-07-31T02:31:10.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:10 smithi045 conmon[39718]: cluster 2021-07-31T02:31:09.403324+0000 mon.a (mon.0) 2234 : cluster [DBG] osdmap e1285: 8 total, 8 up, 8 in 2021-07-31T02:31:11.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:11 smithi154 conmon[34553]: cluster 2021-07-31T02:31:10.408602+0000 mon.a (mon.0) 2235 : cluster [DBG] 2021-07-31T02:31:11.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:11 smithi154 conmon[34553]: osdmap e1286: 8 total, 8 up, 8 in 2021-07-31T02:31:11.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:11 smithi154 conmon[34553]: cluster 2021-07-31T02:31:10.426479+0000 mgr.x (mgr.24542) 1881 : cluster [DBG] pgmap v2451: 41 pgs: 41 active+clean; 408 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 24 MiB/s rd, 24 MiB/s wr, 90 op/s 2021-07-31T02:31:11.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:11 smithi045 conmon[33655]: cluster 2021-07-31T02:31:10.408602+0000 2021-07-31T02:31:11.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:11 smithi045 conmon[33655]: mon.a (mon.0) 2235 : cluster [DBG] osdmap e1286: 8 total, 8 up, 8 in 2021-07-31T02:31:11.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:11 smithi045 conmon[33655]: cluster 2021-07 2021-07-31T02:31:11.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:11 smithi045 conmon[33655]: -31T02:31:10.426479+0000 mgr.x (mgr.24542) 1881 : cluster 2021-07-31T02:31:11.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:11 smithi045 conmon[33655]: [DBG] pgmap v2451: 41 pgs: 41 active+clean; 408 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 24 MiB/s rd, 24 MiB/s wr, 90 op/s 2021-07-31T02:31:11.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:11 smithi045 conmon[39718]: cluster 2021-07-31T02:31:10.408602+0000 mon.a ( 2021-07-31T02:31:11.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:11 smithi045 conmon[39718]: mon.0) 2235 : cluster [DBG] osdmap e1286: 8 total, 8 up, 8 in 2021-07-31T02:31:11.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:11 smithi045 conmon[39718]: cluster 2021-07-31T02:31:10.426479+0000 mgr.x (mgr.24542) 1881 : cluster [DBG] pgmap v2451: 41 pgs: 41 active+clean; 408 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 24 MiB/s rd, 24 MiB/s wr, 90 op/s 2021-07-31T02:31:12.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:12 smithi154 conmon[34553]: cluster 2021-07-31T02:31:12.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:12 smithi154 conmon[34553]: 2021-07-31T02:31:12.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:12 smithi154 conmon[34553]: 2021 2021-07-31T02:31:12.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:12 smithi154 conmon[34553]: -07-31T02:31:11.417201+0000 mon.a (mon. 2021-07-31T02:31:12.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:12 smithi154 conmon[34553]: 0) 2236 : cluster [DBG] osdmap e1287: 8 total, 8 up, 8 in 2021-07-31T02:31:12.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:12 smithi045 conmon[39718]: cluster 2021-07-31T02:31:11.417201+0000 2021-07-31T02:31:12.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:12 smithi045 conmon[39718]: mon.a (mon.0) 2236 : cluster [DBG] osdmap e1287: 8 total, 8 up, 8 in 2021-07-31T02:31:12.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:12 smithi045 conmon[33655]: cluster 2021-07-31T02:31:11.417201+0000 mon.a (mon 2021-07-31T02:31:12.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:12 smithi045 conmon[33655]: .0) 2236 : cluster [DBG] osdmap e1287: 8 total, 8 up, 8 in 2021-07-31T02:31:13.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:13 smithi154 conmon[34553]: cluster 2021-07-31T02:31:12.417878+0000 mon.a (mon.0) 2237 : cluster [DBG] osdmap e1288: 8 total, 8 up, 8 in 2021-07-31T02:31:13.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:13 smithi154 conmon[34553]: cluster 2021-07-31T02:31:12.426906+0000 mgr.x (mgr.24542) 1882 : cluster [DBG] pgmap v2454: 41 pgs: 41 active+clean; 408 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 24 MiB/s rd, 24 MiB/s wr, 90 op/s 2021-07-31T02:31:13.775 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:31:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:31:13.585Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:31:13.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:31:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:31:13.585Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:31:13.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:31:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:31:13.585Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:31:13.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:13 smithi045 conmon[33655]: cluster 2021-07-31T02:31:12.417878+0000 mon.a (mon.0) 2237 : cluster [DBG] 2021-07-31T02:31:13.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:13 smithi045 conmon[33655]: osdmap e1288: 8 total, 8 up, 8 in 2021-07-31T02:31:13.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:13 smithi045 conmon[33655]: cluster 2021-07-31T02:31:12.426906+0000 mgr.x (mgr.24542) 1882 : cluster [DBG] pgmap v2454: 41 pgs: 41 active+clean; 408 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 24 MiB/s rd, 24 MiB/s wr, 90 op/s 2021-07-31T02:31:13.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:13 smithi045 conmon[39718]: cluster 2021-07-31T02:31:12.417878+0000 2021-07-31T02:31:13.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:13 smithi045 conmon[39718]: mon.a (mon.0) 2237 : cluster [DBG] osdmap e1288: 8 total, 8 up, 8 in 2021-07-31T02:31:13.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:13 smithi045 conmon[39718]: cluster 2021-07-31T02 2021-07-31T02:31:13.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:13 smithi045 conmon[39718]: :31:12.426906+0000 mgr.x (mgr.24542) 1882 : cluster [DBG] pgmap v2454: 41 pgs: 41 active+clean; 408 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 24 MiB/s rd, 24 MiB/s wr, 90 op/s 2021-07-31T02:31:13.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:13 smithi045 conmon[39718]: 2021-07-31T02:31:14.284 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestDeepCopy.Stress_LargerDstObjSize (8480 ms) 2021-07-31T02:31:14.284 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestDeepCopy.NoSnaps_SmallerDstObjSize 2021-07-31T02:31:14.484 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestDeepCopy.NoSnaps_SmallerDstObjSize (201 ms) 2021-07-31T02:31:14.485 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestDeepCopy.Snaps_SmallerDstObjSize 2021-07-31T02:31:14.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:14 smithi154 conmon[34553]: cluster 2021-07-31T02:31:13.431233+0000 mon.a (mon. 2021-07-31T02:31:14.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:14 smithi154 conmon[34553]: 0) 2238 : cluster [DBG] osdmap e1289: 8 total, 8 up, 8 in 2021-07-31T02:31:14.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:14 smithi045 conmon[39718]: cluster 2021-07-31T02:31:13.431233+0000 mon.a 2021-07-31T02:31:14.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:14 smithi045 conmon[39718]: (mon.0) 2238 : cluster [DBG] osdmap e1289: 8 total, 8 up, 8 in 2021-07-31T02:31:14.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:14 smithi045 conmon[33655]: cluster 2021-07-31T02:31:13.431233+0000 mon.a (mon. 2021-07-31T02:31:14.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:14 smithi045 conmon[33655]: 0) 2238 : cluster [DBG] osdmap e1289: 8 total, 8 up, 8 in 2021-07-31T02:31:15.718 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:31:15.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:15 smithi154 conmon[34553]: cluster 2021-07-31T02:31:14.427390+0000 mgr.x (mgr.24542) 1883 : cluster [DBG] pgmap v2456: 41 pgs: 41 active+clean; 554 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 46 MiB/s rd, 18 MiB/s wr, 76 op/s 2021-07-31T02:31:15.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:15 smithi045 conmon[33655]: cluster 2021-07-31T02:31:14.427390+0000 mgr.x (mgr.24542) 2021-07-31T02:31:15.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:15 smithi045 conmon[33655]: 1883 : cluster [DBG] pgmap v2456: 41 pgs: 41 active+clean; 554 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 46 MiB/s rd, 18 MiB/s wr, 76 op/s 2021-07-31T02:31:15.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:15 smithi045 conmon[39718]: cluster 2021-07-31T02:31:14.427390 2021-07-31T02:31:15.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:15 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 1883 : cluster [DBG] pgmap v2456: 41 pgs: 41 active+clean; 554 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 46 MiB/s rd, 18 MiB/s wr, 76 op/s 2021-07-31T02:31:15.996 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:31:15.997 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 46m ago 46m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:31:15.997 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 43m ago 46m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:31:15.997 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 43m ago 50m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:31:15.997 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 46m ago 52m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:31:15.998 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 46m ago 52m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:31:15.998 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 43m ago 50m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:31:15.998 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 46m ago 50m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:31:15.998 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 46m ago 47m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:31:15.998 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 43m ago 47m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:31:15.999 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 46m ago 49m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:31:15.999 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 46m ago 49m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:31:15.999 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 46m ago 49m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:31:15.999 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 46m ago 48m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:31:16.000 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 43m ago 48m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:31:16.000 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 43m ago 48m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:31:16.000 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 43m ago 47m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:31:16.000 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 43m ago 47m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:31:16.000 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 43m ago 47m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:31:16.332 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:31:16.332 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:31:16.332 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:31:16.333 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:31:16.333 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:31:16.333 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:31:16.333 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:31:16.333 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:31:16.334 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:31:16.334 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:31:16.334 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:31:16.334 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:31:16.335 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:31:16.335 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:31:16.335 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:31:16.336 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:31:16.336 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:31:16.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:16 smithi154 conmon[34553]: cluster 2021-07-31T02:31:15.483890+0000 mon.a (mon.0) 2239 : cluster 2021-07-31T02:31:16.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:16 smithi154 conmon[34553]: [DBG] osdmap e1290: 8 total, 8 up, 8 in 2021-07-31T02:31:16.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:16 smithi154 conmon[34553]: audit 2021-07-31T02:31:15.706797+0000 mgr.x (mgr.24542) 1884 : audit [DBG] from='client.27268 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:31:16.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:16 smithi154 conmon[34553]: audit 2021-07-31T02: 2021-07-31T02:31:16.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:16 smithi154 conmon[34553]: 31:15.988988+0000 mgr.x (mgr.24542) 1885 : audit [DBG] from='client.17337 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:31:16.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:16 smithi154 conmon[34553]: audit 2021-07-31T02:31:16.332397+0000 2021-07-31T02:31:16.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:16 smithi154 conmon[34553]: mon.a (mon.0) 2240 : audit [DBG] from='client.? 172.21.15.45:0/2960286020' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:31:16.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:16 smithi045 conmon[33655]: cluster 2021-07-31T02:31:15.483890+0000 mon.a (mon.0) 2021-07-31T02:31:16.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:16 smithi045 conmon[33655]: 2239 : cluster [DBG] osdmap e1290: 8 total, 8 up, 8 in 2021-07-31T02:31:16.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:16 smithi045 conmon[33655]: audit 2021-07-31T02:31:15.706797+0000 mgr.x (mgr.24542) 1884 : audit [DBG] from='client.27268 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:31:16.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:16 smithi045 conmon[33655]: audit 2021-07-31T 2021-07-31T02:31:16.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:16 smithi045 conmon[33655]: 02:31:15.988988+0000 mgr.x (mgr.24542) 1885 : audit [DBG] from='client.17337 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:31:16.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:16 smithi045 conmon[33655]: audit 2021-07-31T02:31: 2021-07-31T02:31:16.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:16 smithi045 conmon[33655]: 16.332397+0000 mon.a (mon.0) 2240 2021-07-31T02:31:16.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:16 smithi045 conmon[33655]: : audit [DBG] from='client.? 172.21.15.45:0/2960286020' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:31:16.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:16 smithi045 conmon[39718]: cluster 2021-07-31T02:31:15. 2021-07-31T02:31:16.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:16 smithi045 conmon[39718]: 483890+0000 mon.a (mon.0) 2239 : cluster [DBG] osdmap e1290: 8 total, 8 up, 8 in 2021-07-31T02:31:16.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:16 smithi045 conmon[39718]: audit 2021- 2021-07-31T02:31:16.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:16 smithi045 conmon[39718]: 07-31T02:31:15.706797+0000 mgr.x (mgr 2021-07-31T02:31:16.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:16 smithi045 conmon[39718]: .24542) 1884 : audit [DBG] from='client.27268 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:31:16.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:16 smithi045 conmon[39718]: audit 2021-07-31T02:31:16.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:16 smithi045 conmon[39718]: 2021-07-31T02:31 2021-07-31T02:31:16.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:16 smithi045 conmon[39718]: :15.988988+0000 mgr.x ( 2021-07-31T02:31:16.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:16 smithi045 conmon[39718]: mgr.24542) 1885 : audit [DBG] 2021-07-31T02:31:16.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:16 smithi045 conmon[39718]: from='client.17337 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:31:16.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:16 smithi045 conmon[39718]: audit 2021- 2021-07-31T02:31:16.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:16 smithi045 conmon[39718]: 07-31T02:31: 2021-07-31T02:31:16.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:16 smithi045 conmon[39718]: 16.332397+0000 mon.a (mon.0) 2021-07-31T02:31:16.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:16 smithi045 conmon[39718]: 2240 : audit [DBG] from='client.? 172.21.15.45:0/2960286020' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:31:17.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:17 smithi154 conmon[34553]: cluster 2021-07-31T02:31:16.427897+0000 mgr.x (mgr.24542) 1886 : cluster [DBG] pgmap v2458: 41 pgs: 41 active+clean; 554 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 37 MiB/s rd, 14 MiB/s wr, 61 op/s 2021-07-31T02:31:17.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:17 smithi154 conmon[34553]: cluster 2021-07-31T02:31:16.496212+0000 mon.a (mon.0) 2241 : cluster [DBG] osdmap e1291: 8 total, 8 up, 8 in 2021-07-31T02:31:17.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:17 smithi045 conmon[33655]: cluster 2021-07-31T02:31:16.427897+0000 mgr.x (mgr.24542 2021-07-31T02:31:17.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:17 smithi045 conmon[33655]: ) 1886 : cluster [DBG] pgmap v2458: 41 pgs: 41 active+clean; 554 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 37 MiB/s rd, 14 MiB/s wr, 61 op/s 2021-07-31T02:31:17.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:17 smithi045 conmon[33655]: cluster 2021-07-31T02:31:16.496212+0000 mon.a (mon 2021-07-31T02:31:17.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:17 smithi045 conmon[33655]: .0 2021-07-31T02:31:17.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:17 smithi045 conmon[33655]: ) 2241 : cluster [DBG] osdmap e1291: 8 total, 8 up, 8 in 2021-07-31T02:31:17.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:17 smithi045 conmon[39718]: cluster 2021-07-31T02:31:16.427897 2021-07-31T02:31:17.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:17 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 1886 : cluster [DBG] pgmap v2458: 41 pgs: 41 active+clean; 554 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 37 MiB/s rd, 14 MiB/s wr, 61 op/s 2021-07-31T02:31:17.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:17 smithi045 conmon[39718]: cluster 2021- 2021-07-31T02:31:17.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:17 smithi045 conmon[39718]: 07-31T02:31:16.496212+0000 mon.a (mon 2021-07-31T02:31:17.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:17 smithi045 conmon[39718]: .0) 2241 : cluster [DBG] 2021-07-31T02:31:17.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:17 smithi045 conmon[39718]: osdmap e1291: 8 total, 8 up, 8 in 2021-07-31T02:31:18.757 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestDeepCopy.Snaps_SmallerDstObjSize (4272 ms) 2021-07-31T02:31:18.763 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestDeepCopy.Clone_SmallerDstObjSize 2021-07-31T02:31:18.771 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:31:18.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:18 smithi045 conmon[39718]: cluster 2021-07-31T02:31:17. 2021-07-31T02:31:18.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:18 smithi045 conmon[39718]: 503856+0000 mon.a (mon.0) 2242 : cluster [DBG] osdmap e1292: 8 total, 8 up, 8 in 2021-07-31T02:31:18.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:18 smithi045 conmon[33655]: cluster 2021-07-31T02:31:17.503856+0000 mon.a (mon.0) 2242 : 2021-07-31T02:31:18.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:18 smithi045 conmon[33655]: cluster [DBG] osdmap e1292: 8 total, 8 up, 8 in 2021-07-31T02:31:18.830 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestDeepCopy.Clone_SmallerDstObjSize (74 ms) 2021-07-31T02:31:18.831 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestDeepCopy.CloneFlatten_SmallerDstObjSize 2021-07-31T02:31:18.842 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:31:18.903 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestDeepCopy.CloneFlatten_SmallerDstObjSize (73 ms) 2021-07-31T02:31:18.904 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestDeepCopy.Stress_SmallerDstObjSize 2021-07-31T02:31:18.916 INFO:tasks.workunit.client.0.smithi045.stdout:write: A 44135518~2880655 2021-07-31T02:31:18.927 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 39808658~1891754 2021-07-31T02:31:18.930 INFO:tasks.workunit.client.0.smithi045.stdout:write: A 9107658~4022592 2021-07-31T02:31:18.946 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 34929982~2326987 2021-07-31T02:31:18.946 INFO:tasks.workunit.client.0.smithi045.stdout:write: A 34606199~7274962 2021-07-31T02:31:18.966 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 39183211~854499 2021-07-31T02:31:18.968 INFO:tasks.workunit.client.0.smithi045.stdout:write: A 29230486~5166932 2021-07-31T02:31:18.988 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 61001105~4033616 2021-07-31T02:31:18.989 INFO:tasks.workunit.client.0.smithi045.stdout:snap: snap0 2021-07-31T02:31:19.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:18 smithi154 conmon[34553]: cluster 2021-07-31T02:31:19.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:18 smithi154 conmon[34553]: 2021-07-31T02:31:19.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:18 smithi154 conmon[34553]: 2021 2021-07-31T02:31:19.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:18 smithi154 conmon[34553]: -07-31T02: 2021-07-31T02:31:19.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:18 smithi154 conmon[34553]: 31:17.503856+0000 mon.a (mon.0) 2242 : cluster [DBG] osdmap e1292: 8 total, 8 up, 8 in 2021-07-31T02:31:19.516 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 118653540 2021-07-31T02:31:19.518 INFO:tasks.workunit.client.0.smithi045.stdout:write: B 112195957~297089 2021-07-31T02:31:19.519 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 77309752~51818 2021-07-31T02:31:19.520 INFO:tasks.workunit.client.0.smithi045.stdout:write: B 49971348~391575 2021-07-31T02:31:19.520 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 12490912~2929690 2021-07-31T02:31:19.523 INFO:tasks.workunit.client.0.smithi045.stdout:write: B 13634526~6993683 2021-07-31T02:31:19.524 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 41035677~5157629 2021-07-31T02:31:19.527 INFO:tasks.workunit.client.0.smithi045.stdout:write: B 58267344~7935114 2021-07-31T02:31:19.528 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 42065684~7452843 2021-07-31T02:31:19.550 INFO:tasks.workunit.client.0.smithi045.stdout:snap: snap1 2021-07-31T02:31:19.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:19 smithi045 conmon[33655]: cluster 2021-07-31T02:31:18.428332+0000 mgr.x (mgr.24542) 1887 : 2021-07-31T02:31:19.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:19 smithi045 conmon[33655]: cluster [DBG] pgmap v2461: 41 pgs: 41 active+clean; 554 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 37 MiB/s rd, 14 MiB/s wr, 61 op/s 2021-07-31T02:31:19.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:19 smithi045 conmon[33655]: cluster 2021-07-31T02:31:18.505667+0000 mon.a (mon.0) 2243 : cluster [DBG] osdmap e1293: 8 total, 8 up, 8 in 2021-07-31T02:31:19.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:19 smithi045 conmon[39718]: cluster 2021-07-31T02:31:18.428332+0000 mgr.x ( 2021-07-31T02:31:19.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:19 smithi045 conmon[39718]: mgr.24542) 1887 : cluster [DBG] pgmap v2461: 41 pgs: 41 active+clean; 554 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 37 MiB/s rd, 14 MiB/s wr, 61 op/s 2021-07-31T02:31:19.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:19 smithi045 conmon[39718]: cluster 2021-07-31T02:31:18.505667+0000 mon.a ( 2021-07-31T02:31:19.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:19 smithi045 conmon[39718]: mon.0) 2243 : cluster [DBG] osdmap e1293: 8 total, 8 up, 8 in 2021-07-31T02:31:20.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:19 smithi154 conmon[34553]: cluster 2021-07-31T02:31:18.428332+0000 mgr.x (mgr.24542) 2021-07-31T02:31:20.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:19 smithi154 conmon[34553]: 1887 : cluster [DBG] pgmap v2461: 41 pgs: 41 active+clean; 554 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 37 MiB/s rd, 14 MiB/s wr, 61 op/s 2021-07-31T02:31:20.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:19 smithi154 conmon[34553]: cluster 2021-07-31T02:31:18.505667+0000 mon.a (mon.0) 2243 : cluster [DBG] osdmap e1293: 8 total, 8 up, 8 in 2021-07-31T02:31:20.532 INFO:tasks.workunit.client.0.smithi045.stdout:write: C 65524502~3086827 2021-07-31T02:31:20.533 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 110422651~8230889 2021-07-31T02:31:20.534 INFO:tasks.workunit.client.0.smithi045.stdout:write: C 103778225~1043664 2021-07-31T02:31:20.534 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 61753616~5316876 2021-07-31T02:31:20.545 INFO:tasks.workunit.client.0.smithi045.stdout:write: C 19991026~6060981 2021-07-31T02:31:20.545 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 107586922~2317452 2021-07-31T02:31:20.546 INFO:tasks.workunit.client.0.smithi045.stdout:write: C 28491128~2001904 2021-07-31T02:31:20.546 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 66497445~250311 2021-07-31T02:31:20.563 INFO:tasks.workunit.client.0.smithi045.stdout:snap: snap2 2021-07-31T02:31:20.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:20 smithi045 conmon[33655]: cluster 2021-07-31T02:31:19.511612+0000 2021-07-31T02:31:20.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:20 smithi045 conmon[33655]: mon.a (mon.0) 2244 : cluster [DBG] osdmap e1294: 8 total, 8 up, 8 in 2021-07-31T02:31:20.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:20 smithi045 conmon[39718]: cluster 2021-07-31T02:31:19.511612+0000 mon.a 2021-07-31T02:31:20.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:20 smithi045 conmon[39718]: (mon.0) 2244 : cluster [DBG] osdmap e1294: 8 total, 8 up, 8 in 2021-07-31T02:31:21.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:20 smithi154 conmon[34553]: cluster 2021-07-31T02:31:19.511612+0000 mon.a (mon.0) 2244 : cluster [DBG] osdmap e1294: 8 total, 8 up, 8 in 2021-07-31T02:31:21.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:20 smithi154 conmon[34553]: 2021-07-31T02:31:21.531 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 173371247 2021-07-31T02:31:21.535 INFO:tasks.workunit.client.0.smithi045.stdout:write: D 28896329~993185 2021-07-31T02:31:21.536 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 171915462~1455785 2021-07-31T02:31:21.538 INFO:tasks.workunit.client.0.smithi045.stdout:write: D 21684002~7994802 2021-07-31T02:31:21.543 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 13255167~6819210 2021-07-31T02:31:21.544 INFO:tasks.workunit.client.0.smithi045.stdout:write: D 137816914~1703861 2021-07-31T02:31:21.544 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 165255059~3930794 2021-07-31T02:31:21.546 INFO:tasks.workunit.client.0.smithi045.stdout:write: D 107680917~3984181 2021-07-31T02:31:21.546 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 154801362~5299432 2021-07-31T02:31:21.571 INFO:tasks.workunit.client.0.smithi045.stdout:snap: snap3 2021-07-31T02:31:22.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:21 smithi154 conmon[34553]: cluster 2021-07-31T02:31:20.428935+0000 mgr.x (mgr.24542) 1888 : cluster [DBG] 2021-07-31T02:31:22.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:21 smithi154 conmon[34553]: pgmap v2464: 41 pgs: 41 active+clean; 618 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 92 MiB/s rd, 16 MiB/s wr, 152 op/s 2021-07-31T02:31:22.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:21 smithi154 conmon[34553]: cluster 2021-07-31T02:31:20.522691+0000 mon.a (mon.0) 2245 : cluster [DBG] osdmap e1295: 8 total, 8 up, 8 in 2021-07-31T02:31:22.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:21 smithi045 conmon[33655]: cluster 2021-07-31T02:31:20.428935+0000 mgr.x (mgr.24542) 1888 : 2021-07-31T02:31:22.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:21 smithi045 conmon[33655]: cluster [DBG] pgmap v2464: 41 pgs: 41 active+clean; 618 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 92 MiB/s rd, 16 MiB/s wr, 152 op/s 2021-07-31T02:31:22.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:21 smithi045 conmon[33655]: cluster 2021-07-31T02:31:20.522691+0000 mon.a (mon.0) 2245 : cluster [DBG] osdmap e1295: 8 total, 8 up, 8 in 2021-07-31T02:31:22.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:21 smithi045 conmon[39718]: cluster 2021-07-31T02:31:20.428935+0000 mgr.x ( 2021-07-31T02:31:22.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:21 smithi045 conmon[39718]: mgr.24542) 1888 : cluster [DBG] pgmap v2464: 41 pgs: 41 active+clean; 618 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 92 MiB/s rd, 16 MiB/s wr, 152 op/s 2021-07-31T02:31:22.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:21 smithi045 conmon[39718]: cluster 2021-07-31T02:31:20.522691+0000 2021-07-31T02:31:22.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:21 smithi045 conmon[39718]: mon.a (mon.0) 2245 : cluster [DBG] osdmap e1295: 8 total, 8 up, 8 in 2021-07-31T02:31:23.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:22 smithi154 conmon[34553]: cluster 2021-07-31T02:31:21.524745+0000 mon.a 2021-07-31T02:31:23.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:22 smithi154 conmon[34553]: (mon.0) 2246 : cluster [DBG] osdmap e1296: 8 total, 8 up, 8 in 2021-07-31T02:31:23.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:22 smithi045 conmon[39718]: cluster 2021-07-31T02:31:21. 2021-07-31T02:31:23.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:22 smithi045 conmon[39718]: 524745+0000 mon.a (mon.0) 2246 : cluster [DBG] osdmap e1296: 8 total, 8 up, 8 in 2021-07-31T02:31:23.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:22 smithi045 conmon[33655]: cluster 2021-07-31T02:31:21.524745+0000 mon.a (mon.0) 2246 : cluster 2021-07-31T02:31:23.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:22 smithi045 conmon[33655]: [DBG] osdmap e1296: 8 total, 8 up, 8 in 2021-07-31T02:31:24.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:23 smithi154 conmon[34553]: cluster 2021-07-31T02:31:22.429525+0000 mgr.x (mgr.24542) 1889 : cluster [DBG] pgmap v2467: 41 pgs: 41 active+clean; 618 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 92 MiB/s rd, 16 MiB/s wr, 152 op/s 2021-07-31T02:31:24.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:23 smithi154 conmon[34553]: cluster 2021-07-31T02:31:22.531285+0000 mon.a (mon.0) 2247 : cluster [DBG] osdmap e1297: 8 total, 8 up, 8 in 2021-07-31T02:31:24.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:23 smithi045 conmon[33655]: cluster 2021-07-31T02:31:22.429525+0000 mgr.x 2021-07-31T02:31:24.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:23 smithi045 conmon[33655]: (mgr.24542) 1889 : cluster [DBG] pgmap v2467: 41 pgs: 41 active+clean; 618 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 92 MiB/s rd, 16 MiB/s wr, 152 op/s 2021-07-31T02:31:24.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:23 smithi045 conmon[33655]: cluster 2021-07-31T02:31:22.531285+0000 mon.a (mon.0) 2247 : cluster 2021-07-31T02:31:24.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:23 smithi045 conmon[33655]: [DBG] osdmap e1297: 8 total, 8 up, 8 in 2021-07-31T02:31:24.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:23 smithi045 conmon[39718]: cluster 2021-07-31T02:31:22.429525+0000 mgr.x ( 2021-07-31T02:31:24.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:23 smithi045 conmon[39718]: mgr.24542) 1889 : cluster [DBG] pgmap v2467: 41 pgs: 41 active+clean; 618 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 92 MiB/s rd, 16 MiB/s wr, 152 op/s 2021-07-31T02:31:24.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:23 smithi045 conmon[39718]: cluster 2021-07-31T02:31:22. 2021-07-31T02:31:24.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:23 smithi045 conmon[39718]: 531285+0000 mon.a (mon.0) 2247 : cluster [DBG] 2021-07-31T02:31:24.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:23 smithi045 conmon[39718]: osdmap e1297: 8 total, 8 up, 8 in 2021-07-31T02:31:24.034 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:31:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:31:23.586Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:31:24.035 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:31:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:31:23.586Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:31:24.035 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:31:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:31:23.586Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:31:25.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:24 smithi154 conmon[34553]: cluster 2021-07-31T02:31:23.542500+0000 mon.a (mon.0) 2248 : cluster [DBG] osdmap e1298: 8 total, 8 up, 8 in 2021-07-31T02:31:25.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:24 smithi045 conmon[39718]: cluster 2021-07-31T02:31:23.542500+0000 mon.a (mon.0) 2248 : cluster [DBG] osdmap e1298: 8 total, 8 up, 8 in 2021-07-31T02:31:25.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:24 smithi045 conmon[33655]: cluster 2021-07-31T02:31:23.542500 2021-07-31T02:31:25.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:24 smithi045 conmon[33655]: +0000 mon.a (mon.0) 2248 : cluster [DBG] osdmap e1298: 8 total, 8 up, 8 in 2021-07-31T02:31:26.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:25 smithi154 conmon[34553]: cluster 2021-07-31T02:31:24.430216+0000 mgr.x (mgr. 2021-07-31T02:31:26.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:25 smithi154 conmon[34553]: 24542) 1890 : cluster [DBG] pgmap v2470: 41 pgs: 41 active+clean; 710 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 18 KiB/s rd, 15 MiB/s wr, 39 op/s 2021-07-31T02:31:26.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:25 smithi154 conmon[34553]: cluster 2021-07-31T02:31:24.548740+0000 mon.a (mon.0) 2249 : cluster [DBG] osdmap e1299: 8 total, 8 up, 8 in 2021-07-31T02:31:26.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:25 smithi045 conmon[39718]: cluster 2021-07-31T02:31:24.430216+0000 mgr.x (mgr. 2021-07-31T02:31:26.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:25 smithi045 conmon[39718]: 24542) 1890 : cluster [DBG] pgmap v2470: 41 pgs: 41 active+clean; 710 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 18 KiB/s rd, 15 MiB/s wr, 39 op/s 2021-07-31T02:31:26.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:25 smithi045 conmon[39718]: cluster 2021-07-31T02: 2021-07-31T02:31:26.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:25 smithi045 conmon[39718]: 31:24.548740+0000 mon.a (mon.0) 2249 : cluster [DBG] osdmap e1299: 8 total, 8 up, 8 in 2021-07-31T02:31:26.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:25 smithi045 conmon[33655]: cluster 2021-07-31T02:31:24.430216+0000 mgr.x (mgr.24542) 1890 : 2021-07-31T02:31:26.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:25 smithi045 conmon[33655]: cluster [DBG] pgmap v2470: 41 pgs: 41 active+clean; 710 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 18 KiB/s rd, 15 MiB/s wr, 39 op/s 2021-07-31T02:31:26.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:25 smithi045 conmon[33655]: cluster 2021-07-31T02:31:24.548740+0000 mon.a (mon.0) 2249 : cluster [DBG] osdmap e1299: 8 total, 8 up, 8 in 2021-07-31T02:31:27.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:26 smithi154 conmon[34553]: cluster 2021-07-31T02:31:25.550558+0000 mon.a (mon.0) 2250 : cluster 2021-07-31T02:31:27.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:26 smithi154 conmon[34553]: [DBG] osdmap e1300: 8 total, 8 up, 8 in 2021-07-31T02:31:27.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:26 smithi045 conmon[33655]: cluster 2021-07-31T02:31:25.550558+0000 mon.a (mon.0) 2250 : cluster [DBG] osdmap e1300: 8 total, 8 up, 8 in 2021-07-31T02:31:27.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:26 smithi045 conmon[39718]: cluster 2021-07-31T02:31:25.550558+0000 mon.a (mon. 2021-07-31T02:31:27.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:26 smithi045 conmon[39718]: 0) 2250 : cluster [DBG] osdmap e1300: 8 total, 8 up, 8 in 2021-07-31T02:31:27.337 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestDeepCopy.Stress_SmallerDstObjSize (8434 ms) 2021-07-31T02:31:27.337 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestDeepCopy.NoSnaps_StrippingLargerDstObjSize 2021-07-31T02:31:27.348 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:31:27.409 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestDeepCopy.NoSnaps_StrippingLargerDstObjSize (72 ms) 2021-07-31T02:31:27.409 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestDeepCopy.Snaps_StrippingLargerDstObjSize 2021-07-31T02:31:27.422 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:31:27.489 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestDeepCopy.Snaps_StrippingLargerDstObjSize (79 ms) 2021-07-31T02:31:27.490 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestDeepCopy.Clone_StrippingLargerDstObjSize 2021-07-31T02:31:27.503 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:31:27.568 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestDeepCopy.Clone_StrippingLargerDstObjSize (79 ms) 2021-07-31T02:31:27.568 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestDeepCopy.CloneFlatten_StrippingLargerDstObjSize 2021-07-31T02:31:27.578 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:31:27.643 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestDeepCopy.CloneFlatten_StrippingLargerDstObjSize (76 ms) 2021-07-31T02:31:27.644 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestDeepCopy.Stress_StrippingLargerDstObjSize 2021-07-31T02:31:27.655 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:31:27.714 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestDeepCopy.Stress_StrippingLargerDstObjSize (71 ms) 2021-07-31T02:31:27.715 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestDeepCopy.NoSnaps_StrippingSmallerDstObjSize 2021-07-31T02:31:27.725 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:31:27.792 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestDeepCopy.NoSnaps_StrippingSmallerDstObjSize (78 ms) 2021-07-31T02:31:27.793 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestDeepCopy.Snaps_StrippingSmallerDstObjSize 2021-07-31T02:31:27.805 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:31:27.865 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestDeepCopy.Snaps_StrippingSmallerDstObjSize (73 ms) 2021-07-31T02:31:27.866 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestDeepCopy.Clone_StrippingSmallerDstObjSize 2021-07-31T02:31:27.877 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:31:27.938 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestDeepCopy.Clone_StrippingSmallerDstObjSize (72 ms) 2021-07-31T02:31:27.938 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestDeepCopy.CloneFlatten_StrippingSmallerDstObjSize 2021-07-31T02:31:27.951 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:31:28.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:27 smithi154 conmon[34553]: cluster 2021-07-31T02:31:26.430657+0000 mgr.x (mgr.24542 2021-07-31T02:31:28.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:27 smithi154 conmon[34553]: ) 1891 : cluster [DBG] pgmap v2473: 41 pgs: 41 active+clean; 710 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 18 KiB/s rd, 15 MiB/s wr, 39 op/s 2021-07-31T02:31:28.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:27 smithi154 conmon[34553]: cluster 2021-07-31T02:31:26.562658+0000 mon.a (mon.0) 2251 : cluster [DBG] osdmap e1301: 8 total, 8 up, 8 in 2021-07-31T02:31:28.012 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestDeepCopy.CloneFlatten_StrippingSmallerDstObjSize (75 ms) 2021-07-31T02:31:28.013 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestDeepCopy.Stress_StrippingSmallerDstObjSize 2021-07-31T02:31:28.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:27 smithi045 conmon[33655]: cluster 2021-07-31T02:31:26.430657+0000 mgr.x (mgr. 2021-07-31T02:31:28.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:27 smithi045 conmon[33655]: 24542) 1891 : cluster [DBG] pgmap v2473: 41 pgs: 41 active+clean; 710 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 18 KiB/s rd, 15 MiB/s wr, 39 op/s 2021-07-31T02:31:28.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:27 smithi045 conmon[33655]: cluster 2021-07-31T02:31:26.562658+0000 mon.a (mon.0) 2251 : cluster [DBG] osdmap e1301: 8 total, 8 up, 8 in 2021-07-31T02:31:28.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:27 smithi045 conmon[39718]: cluster 2021-07-31T02:31:26. 2021-07-31T02:31:28.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:27 smithi045 conmon[39718]: 430657+0000 mgr.x (mgr.24542) 1891 : cluster [DBG] pgmap v2473: 41 pgs: 41 active+clean; 710 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 18 KiB/s rd, 15 MiB/s wr, 39 op/s 2021-07-31T02:31:28.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:27 smithi045 conmon[39718]: cluster 2021 2021-07-31T02:31:28.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:27 smithi045 conmon[39718]: -07-31T02:31:26.562658+0000 mon.a (mon.0) 2251 : cluster 2021-07-31T02:31:28.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:27 smithi045 conmon[39718]: [DBG] osdmap e1301: 8 total, 8 up, 8 in 2021-07-31T02:31:28.027 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:31:28.089 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestDeepCopy.Stress_StrippingSmallerDstObjSize (77 ms) 2021-07-31T02:31:28.575 INFO:tasks.workunit.client.0.smithi045.stdout:[----------] 31 tests from TestDeepCopy (44313 ms total) 2021-07-31T02:31:28.575 INFO:tasks.workunit.client.0.smithi045.stdout: 2021-07-31T02:31:28.575 INFO:tasks.workunit.client.0.smithi045.stdout:[----------] 6 tests from TestGroup 2021-07-31T02:31:29.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:28 smithi154 conmon[34553]: audit 2021-07-31T02:31:28.087125+0000 mon.b (mon.2 2021-07-31T02:31:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:28 smithi154 conmon[34553]: ) 293 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:31:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:28 smithi154 conmon[34553]: audit 2021-07-31T02:31:28.087510+0000 mon.a (mon.0) 2252 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:31:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:28 smithi154 conmon[34553]: audit 2021-07-31T02:31:28.109721+0000 mon.b (mon. 2021-07-31T02:31:29.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:28 smithi154 conmon[34553]: 2) 294 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:31:29.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:28 smithi154 conmon[34553]: audit 2021 2021-07-31T02:31:29.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:28 smithi154 conmon[34553]: -07-31T02:31:28.110270+0000 mon.a ( 2021-07-31T02:31:29.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:28 smithi154 conmon[34553]: mon.0) 2253 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:31:29.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:28 smithi154 conmon[34553]: audit 2021-07-31T02:31:29.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:28 smithi154 conmon[34553]: 2021-07-31T02:31:28.246333+0000 mon.b 2021-07-31T02:31:29.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:28 smithi154 conmon[34553]: (mon.2) 295 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "73.6", "id": [6, 5]}]: dispatch 2021-07-31T02:31:29.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:28 smithi154 conmon[34553]: audit 2021-07-31T 2021-07-31T02:31:29.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:28 smithi154 conmon[34553]: 02:31:28.246536+0000 mon.b (mon.2) 296 : audit 2021-07-31T02:31:29.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:28 smithi154 conmon[34553]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "73.9", "id": [6, 0]}]: dispatch 2021-07-31T02:31:29.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:28 smithi154 conmon[34553]: audit 2021-07-31T 2021-07-31T02:31:29.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:28 smithi154 conmon[34553]: 02:31:28.246917+0000 mon.a (mon.0 2021-07-31T02:31:29.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:28 smithi154 conmon[34553]: ) 2254 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "73.6", "id": [6, 5]}]: dispatch 2021-07-31T02:31:29.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:28 smithi154 conmon[34553]: audit 2021-07 2021-07-31T02:31:29.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:28 smithi154 conmon[34553]: -31T02:31:28.247410+0000 mon.a ( 2021-07-31T02:31:29.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:28 smithi154 conmon[34553]: mon.0) 2255 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "73.9", "id": [6, 0]}]: dispatch 2021-07-31T02:31:29.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:28 smithi045 conmon[33655]: audit 2021-07-31T02:31:28.087125+0000 mon.b (mon.2) 293 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:31:29.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:28 smithi045 conmon[33655]: audit 2021-07-31T02:31:28.087510+0000 mon.a (mon.0) 2252 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:31:29.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:28 smithi045 conmon[33655]: audit 2021-07-31T02:31:28. 2021-07-31T02:31:29.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:28 smithi045 conmon[33655]: 109721+0000 mon.b (mon.2) 294 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:31:29.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:28 smithi045 conmon[33655]: audit 2021-07-31T02:31:28.110270+0000 mon.a (mon.0) 2253 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:31:29.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:28 smithi045 conmon[33655]: audit 2021-07-31T02 2021-07-31T02:31:29.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:28 smithi045 conmon[33655]: :31:28.246333+0000 mon.b (mon.2) 295 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "73.6", "id": [6, 5]}]: dispatch 2021-07-31T02:31:29.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:28 smithi045 conmon[33655]: audit 2021-07-31T02:31:29.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:28 smithi045 conmon[33655]: 2021-07-31T02:31:28.246536+0000 mon.b (mon.2) 296 : 2021-07-31T02:31:29.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:28 smithi045 conmon[33655]: audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "73.9", "id": [6, 0]}]: dispatch 2021-07-31T02:31:29.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:28 smithi045 conmon[33655]: audit 2021-07-31T02 2021-07-31T02:31:29.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:28 smithi045 conmon[33655]: :31:28.246917+0000 mon.a (mon. 2021-07-31T02:31:29.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:28 smithi045 conmon[33655]: 0) 2254 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "73.6", "id": [6, 5]}]: dispatch 2021-07-31T02:31:29.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:28 smithi045 conmon[33655]: audit 2021 2021-07-31T02:31:29.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:28 smithi045 conmon[33655]: -07-31T02:31:28. 2021-07-31T02:31:29.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:28 smithi045 conmon[33655]: 247410+0000 mon.a (mon.0) 2255 : audit 2021-07-31T02:31:29.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:28 smithi045 conmon[33655]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "73.9", "id": [6, 0]}]: dispatch 2021-07-31T02:31:29.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:28 smithi045 conmon[39718]: audit 2021-07-31T02:31 2021-07-31T02:31:29.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:28 smithi045 conmon[39718]: :28.087125+0000 mon.b (mon.2) 293 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:31:29.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:28 smithi045 conmon[39718]: audit 2021-07-31T02:31:29.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:28 smithi045 conmon[39718]: 2021-07-31T02: 2021-07-31T02:31:29.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:28 smithi045 conmon[39718]: 31:28.087510+0000 mon.a 2021-07-31T02:31:29.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:28 smithi045 conmon[39718]: (mon.0) 2252 : audit 2021-07-31T02:31:29.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:28 smithi045 conmon[39718]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:31:29.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:28 smithi045 conmon[39718]: audit 2021 2021-07-31T02:31:29.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:28 smithi045 conmon[39718]: -07-31T02:31:28. 2021-07-31T02:31:29.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:28 smithi045 conmon[39718]: 109721+0000 mon.b (mon.2 2021-07-31T02:31:29.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:28 smithi045 conmon[39718]: ) 294 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:31:29.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:28 smithi045 conmon[39718]: 2021-07-31T02:31:29.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:28 smithi045 conmon[39718]: audit 2021-07-31T02:31:29.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:28 smithi045 conmon[39718]: 2021-07- 2021-07-31T02:31:29.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:28 smithi045 conmon[39718]: 31T02:31:28.110270+0000 mon.a ( 2021-07-31T02:31:29.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:28 smithi045 conmon[39718]: mon.0) 2253 : audit [INF] 2021-07-31T02:31:29.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:28 smithi045 conmon[39718]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:31:29.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:28 smithi045 conmon[39718]: audit 2021- 2021-07-31T02:31:29.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:28 smithi045 conmon[39718]: 07-31T02:31:28.246333+0000 mon.b (mon 2021-07-31T02:31:29.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:28 smithi045 conmon[39718]: .2) 295 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "73.6", "id": [6, 5]}]: dispatch 2021-07-31T02:31:29.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:28 smithi045 conmon[39718]: audit 2021-07-31T02:31:29.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:28 smithi045 conmon[39718]: 2021-07- 2021-07-31T02:31:29.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:28 smithi045 conmon[39718]: 31T02:31:28.246536+0000 mon.b 2021-07-31T02:31:29.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:28 smithi045 conmon[39718]: (mon.2) 296 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "73.9", "id": [6, 0]}]: dispatch 2021-07-31T02:31:29.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:28 smithi045 conmon[39718]: 2021-07-31T02:31:29.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:28 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T02:31:29.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:28 smithi045 conmon[39718]: 02:31:28.246917+0000 mon.a (mon. 2021-07-31T02:31:29.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:28 smithi045 conmon[39718]: 0) 2254 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "73.6", "id": [6, 5]}]: dispatch 2021-07-31T02:31:29.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:28 smithi045 conmon[39718]: audit 2021 2021-07-31T02:31:29.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:28 smithi045 conmon[39718]: -07-31T02:31:28.247410+0000 2021-07-31T02:31:29.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:28 smithi045 conmon[39718]: mon.a (mon.0) 2255 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "73.9", "id": [6, 0]}]: dispatch 2021-07-31T02:31:29.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:28 smithi045 conmon[39718]: 2021-07-31T02:31:30.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:29 smithi154 conmon[34553]: cluster 2021-07-31T02:31:28.431020+0000 mgr.x (mgr.24542) 1892 : cluster [DBG] pgmap v2475: 41 pgs: 41 active+clean; 710 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 15 KiB/s rd, 12 MiB/s wr, 32 op/s 2021-07-31T02:31:30.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:29 smithi154 conmon[34553]: audit 2021-07-31T02:31:28.569068+0000 mon.a (mon.0) 2256 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "73.6", "id": [6, 5]}]': finished 2021-07-31T02:31:30.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:29 smithi154 conmon[34553]: audit 2021-07-31T02:31:28.569187+0000 mon.a (mon.0) 2257 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "73.9", "id": [6, 0]}]': finished 2021-07-31T02:31:30.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:29 smithi154 conmon[34553]: cluster 2021-07-31T02:31:28.569282+0000 mon.a (mon.0) 2021-07-31T02:31:30.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:29 smithi154 conmon[34553]: 2258 : cluster [DBG] osdmap e1302: 8 total, 8 up, 8 in 2021-07-31T02:31:30.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:29 smithi045 conmon[39718]: cluster 2021-07-31T02:31:28.431020+0000 mgr.x (mgr.24542 2021-07-31T02:31:30.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:29 smithi045 conmon[39718]: ) 1892 : cluster [DBG] pgmap v2475: 41 pgs: 41 active+clean; 710 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 15 KiB/s rd, 12 MiB/s wr, 32 op/s 2021-07-31T02:31:30.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:29 smithi045 conmon[39718]: audit 2021-07-31T02:31:28.569068+0000 2021-07-31T02:31:30.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:29 smithi045 conmon[39718]: mon.a (mon.0) 2256 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "73.6", "id": [6, 5]}]': finished 2021-07-31T02:31:30.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:29 smithi045 conmon[39718]: audit 2021-07-31T02 2021-07-31T02:31:30.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:29 smithi045 conmon[39718]: :31:28.569187+0000 mon.a (mon.0) 2257 : 2021-07-31T02:31:30.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:29 smithi045 conmon[39718]: audit [INF] from='mgr.24542 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "73.9", "id": [6, 0]}]': finished 2021-07-31T02:31:30.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:29 smithi045 conmon[39718]: cluster 2021 2021-07-31T02:31:30.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:29 smithi045 conmon[39718]: -07-31T02:31:28.569282+0000 mon.a ( 2021-07-31T02:31:30.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:29 smithi045 conmon[39718]: mon.0) 2258 : cluster [DBG] osdmap e1302: 8 total, 8 up, 8 in 2021-07-31T02:31:30.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:29 smithi045 conmon[33655]: cluster 2021-07-31T02:31:28.431020+0000 mgr.x (mgr.24542) 1892 : cluster 2021-07-31T02:31:30.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:29 smithi045 conmon[33655]: [DBG] pgmap v2475: 41 pgs: 41 active+clean; 710 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 15 KiB/s rd, 12 MiB/s wr, 32 op/s 2021-07-31T02:31:30.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:29 smithi045 conmon[33655]: audit 2021-07-31T02:31:28.569068+0000 mon.a (mon.0) 2256 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "73.6", "id": [6, 5]}]': finished 2021-07-31T02:31:30.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:29 smithi045 conmon[33655]: audit 2021-07-31T02:31:28 2021-07-31T02:31:30.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:29 smithi045 conmon[33655]: .569187+0000 mon.a (mon.0) 2257 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "73.9", "id": [6, 0]}]': finished 2021-07-31T02:31:30.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:29 smithi045 conmon[33655]: cluster 2021-07 2021-07-31T02:31:30.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:29 smithi045 conmon[33655]: -31T02:31:28.569282+0000 mon.a 2021-07-31T02:31:30.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:29 smithi045 conmon[33655]: (mon.0) 2258 : cluster [DBG] 2021-07-31T02:31:30.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:29 smithi045 conmon[33655]: osdmap e1302: 8 total, 8 up, 8 in 2021-07-31T02:31:30.585 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestGroup.group_create 2021-07-31T02:31:31.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:30 smithi154 conmon[34553]: cluster 2021-07-31T02:31:31.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:30 smithi154 conmon[34553]: 2021-07-31T02:31:31.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:30 smithi154 conmon[34553]: 2021 2021-07-31T02:31:31.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:30 smithi154 conmon[34553]: - 2021-07-31T02:31:31.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:30 smithi154 conmon[34553]: 07 2021-07-31T02:31:31.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:30 smithi154 conmon[34553]: - 2021-07-31T02:31:31.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:30 smithi154 conmon[34553]: 31 2021-07-31T02:31:31.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:30 smithi154 conmon[34553]: T 2021-07-31T02:31:31.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:30 smithi154 conmon[34553]: 02 2021-07-31T02:31:31.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:30 smithi154 conmon[34553]: : 2021-07-31T02:31:31.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:30 smithi154 conmon[34553]: 31:29.577955+0000 mon.a (mon.0) 2259 : cluster [DBG] 2021-07-31T02:31:31.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:30 smithi154 conmon[34553]: osdmap e1303: 8 total, 8 up, 8 in 2021-07-31T02:31:31.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:30 smithi154 conmon[34553]: audit 2021-07-31T02:31:29.581299+0000 2021-07-31T02:31:31.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:30 smithi154 conmon[34553]: mon.c (mon.1) 115 : audit [INF] from='client.? 172.21.15.45:0/2360451763' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:31:31.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:30 smithi154 conmon[34553]: audit 2021-07 2021-07-31T02:31:31.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:30 smithi154 conmon[34553]: -31T02:31:29.582690+0000 mon.a (mon.0) 2260 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:31:31.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:30 smithi045 conmon[39718]: cluster 2021-07-31T02:31:29.577955+0000 mon.a (mon.0) 2259 : cluster 2021-07-31T02:31:31.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:30 smithi045 conmon[39718]: [DBG] osdmap e1303: 8 total, 8 up, 8 in 2021-07-31T02:31:31.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:30 smithi045 conmon[39718]: audit 2021-07-31T02:31:29.581299+0000 mon.c (mon.1) 115 : audit [INF] from='client.? 172.21.15.45:0/2360451763' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:31:31.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:30 smithi045 conmon[39718]: audit 2021-07-31T02:31:29.582690+0000 2021-07-31T02:31:31.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:30 smithi045 conmon[39718]: mon.a (mon.0) 2260 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:31:31.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:30 smithi045 conmon[33655]: cluster 2021-07-31T02:31:29.577955+0000 mon.a (mon.0) 2259 : cluster 2021-07-31T02:31:31.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:30 smithi045 conmon[33655]: [DBG] osdmap e1303: 8 total, 8 up, 8 in 2021-07-31T02:31:31.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:30 smithi045 conmon[33655]: audit 2021-07-31T02:31:29.581299+0000 mon.c (mon.1) 115 : audit [INF] from='client.? 172.21.15.45:0/2360451763' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:31:31.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:30 smithi045 conmon[33655]: audit 2021-07-31T02:31:31.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:30 smithi045 conmon[33655]: 2021-07-31T02: 2021-07-31T02:31:31.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:30 smithi045 conmon[33655]: 31:29.582690+0000 mon.a ( 2021-07-31T02:31:31.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:30 smithi045 conmon[33655]: mon.0) 2260 : audit [INF] 2021-07-31T02:31:31.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:30 smithi045 conmon[33655]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:31:32.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:31 smithi154 conmon[34553]: cluster 2021-07-31T02:31:30.431458+0000 mgr.x (mgr.24542) 1893 : cluster 2021-07-31T02:31:32.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:31 smithi154 conmon[34553]: [DBG] pgmap v2478: 41 pgs: 32 unknown, 9 active+clean; 4.6 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail 2021-07-31T02:31:32.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:31 smithi154 conmon[34553]: audit 2021-07-31T02:31:30.583664+0000 mon.a (mon.0) 2261 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-12","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:31:32.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:31 smithi154 conmon[34553]: cluster 2021-07-31T02:31:30.583946 2021-07-31T02:31:32.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:31 smithi154 conmon[34553]: +0000 mon.a (mon.0) 2262 : cluster [DBG] osdmap e1304: 8 total, 8 up, 8 in 2021-07-31T02:31:32.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:31 smithi045 conmon[33655]: cluster 2021-07-31T02:31:30. 2021-07-31T02:31:32.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:31 smithi045 conmon[33655]: 431458+0000 mgr.x (mgr.24542) 1893 : cluster [DBG] pgmap v2478: 41 pgs: 32 unknown, 9 active+clean; 4.6 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail 2021-07-31T02:31:32.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:31 smithi045 conmon[33655]: audit 2021-07-31T02:31:32.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:31 smithi045 conmon[33655]: 2021-07-31T02 2021-07-31T02:31:32.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:31 smithi045 conmon[33655]: :31:30.583664+0000 mon.a (mon.0) 2021-07-31T02:31:32.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:31 smithi045 conmon[33655]: 2261 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-12","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:31:32.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:31 smithi045 conmon[33655]: cluster 2021-07-31T02 2021-07-31T02:31:32.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:31 smithi045 conmon[33655]: :31:30.583946+0000 mon.a (mon.0) 2262 : cluster [DBG] osdmap e1304: 8 total, 8 up, 8 in 2021-07-31T02:31:32.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:31 smithi045 conmon[39718]: cluster 2021-07-31T02:31:30.431458+0000 mgr.x (mgr.24542) 1893 : cluster 2021-07-31T02:31:32.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:31 smithi045 conmon[39718]: [DBG] pgmap v2478: 41 pgs: 32 unknown, 9 active+clean; 4.6 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail 2021-07-31T02:31:32.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:31 smithi045 conmon[39718]: audit 2021-07-31T02:31:30.583664+0000 mon.a (mon.0) 2261 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-12","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:31:32.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:31 smithi045 conmon[39718]: cluster 2021-07-31T02:31:30.583946+0000 mon.a (mon.0) 2262 : cluster [DBG] 2021-07-31T02:31:32.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:31 smithi045 conmon[39718]: osdmap e1304: 8 total, 8 up, 8 in 2021-07-31T02:31:32.617 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestGroup.group_create (2032 ms) 2021-07-31T02:31:32.617 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestGroup.group_createPP 2021-07-31T02:31:32.633 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestGroup.group_createPP (17 ms) 2021-07-31T02:31:32.633 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestGroup.add_image 2021-07-31T02:31:32.637 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:31:32.638 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestGroup.add_image (4 ms) 2021-07-31T02:31:32.638 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestGroup.add_imagePP 2021-07-31T02:31:32.641 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:31:32.641 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestGroup.add_imagePP (4 ms) 2021-07-31T02:31:32.641 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestGroup.add_snapshot 2021-07-31T02:31:32.644 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:31:32.644 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestGroup.add_snapshot (3 ms) 2021-07-31T02:31:32.645 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestGroup.add_snapshotPP 2021-07-31T02:31:32.647 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:31:32.647 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestGroup.add_snapshotPP (3 ms) 2021-07-31T02:31:32.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[34553]: cluster 2021-07-31T02:31:31.584710+0000 mon.a (mon.0) 2263 : cluster [DBG] osdmap e1305: 8 total, 8 up, 8 in 2021-07-31T02:31:32.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[34553]: audit 2021-07-31T02:31:32.365859+0000 mon.b (mon.2) 297 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:31:32.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[34553]: audit 2021-07-31T 2021-07-31T02:31:32.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[34553]: 02:31:32.367305+0000 mon.b (mon.2) 298 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:31:32.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[34553]: audit 2021-07 2021-07-31T02:31:32.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[34553]: -31T02:31:32.368212+0000 mon.b 2021-07-31T02:31:32.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[34553]: (mon.2) 299 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:31:32.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: debug 2021-07-31T02:31:32.411+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:31:32.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:31:32.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:31:32.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: yield 2021-07-31T02:31:32.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:31:32.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:31:32.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:31:32.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:31:32.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:31:32.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:31:32.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:31:32.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:31:32.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:31:32.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: task.result() 2021-07-31T02:31:32.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:31:32.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: debug 2021-07-31T02:31:32.422+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:31:32.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:31:32.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:31:32.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: yield 2021-07-31T02:31:32.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:31:32.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:31:32.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:31:32.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:31:32.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:31:32.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:31:32.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:31:32.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:31:32.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:31:32.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: task.result() 2021-07-31T02:31:32.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:31:32.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: debug 2021-07-31T02:31:32.447+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:31:32.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:31:32.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:31:32.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: yield 2021-07-31T02:31:32.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:31:32.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:31:32.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:31:32.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:31:32.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:31:32.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:31:32.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:31:32.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:31:32.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:31:32.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: task.result() 2021-07-31T02:31:32.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:31:32.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: debug 2021-07-31T02:31:32.448+0000 7fcb97969700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:31:32.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:31:32.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:31:32.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: yield 2021-07-31T02:31:32.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:31:32.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:31:32.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:31:32.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:31:32.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:31:32.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:31:32.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:31:32.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:31:32.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:31:32.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: task.result() 2021-07-31T02:31:32.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:31:32.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: 2021-07-31T02:31:32.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:31:32.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: 2021-07-31T02:31:32.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:31:32.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:31:32.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:31:32.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:31:32.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:31:32.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:31:32.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:31:32.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:31:32.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:31:32.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:31:32.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:31:32.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:31:32.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:31:32.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:31:32.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:31:32.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:31:32.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:31:32.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:31:32.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: raise 2021-07-31T02:31:32.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:31:32.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:31:32.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:31:32.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:31:32.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:31:32.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: debug 2021-07-31T02:31:32.456+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:31:32.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:31:32.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:31:32.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: yield 2021-07-31T02:31:32.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:31:32.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:31:32.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:31:32.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:31:32.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:31:32.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:31:32.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:31:32.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:31:32.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:31:32.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: task.result() 2021-07-31T02:31:32.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:31:32.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: debug 2021-07-31T02:31:32.457+0000 7fcb9816a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:31:32.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:31:32.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:31:32.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: yield 2021-07-31T02:31:32.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:31:32.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:31:32.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:31:32.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:31:32.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:31:32.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:31:32.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:31:32.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:31:32.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:31:32.786 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: task.result() 2021-07-31T02:31:32.786 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:31:32.786 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: 2021-07-31T02:31:32.786 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:31:32.787 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: 2021-07-31T02:31:32.787 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:31:32.787 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:31:32.787 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:31:32.788 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:31:32.789 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:31:32.789 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:31:32.790 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:31:32.791 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:31:32.791 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:31:32.791 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:31:32.791 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:31:32.792 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:31:32.792 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:31:32.792 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:31:32.792 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:31:32.793 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:31:32.793 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:31:32.793 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:31:32.794 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: raise 2021-07-31T02:31:32.794 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:31:32.794 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:31:32.794 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:31:32.795 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:31:32.795 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:31:32 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:31:32.796 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:32 smithi045 conmon[33655]: cluster 2021-07-31T02:31:31. 2021-07-31T02:31:32.796 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:32 smithi045 conmon[33655]: 584710+0000 mon.a (mon.0) 2263 : cluster [DBG] osdmap e1305: 8 total, 8 up, 8 in 2021-07-31T02:31:32.796 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:32 smithi045 conmon[33655]: audit 2021-07-31T02 2021-07-31T02:31:32.797 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:32 smithi045 conmon[33655]: :31:32.365859+0000 2021-07-31T02:31:32.797 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:32 smithi045 conmon[33655]: mon.b (mon.2) 297 : 2021-07-31T02:31:32.797 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:32 smithi045 conmon[33655]: audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:31:32.797 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:32 smithi045 conmon[33655]: audit 2021-07-31T02:31:32. 2021-07-31T02:31:32.798 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:32 smithi045 conmon[33655]: 367305+0000 mon.b (mon.2) 298 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:31:32.798 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:32 smithi045 conmon[33655]: audit 2021-07-31 2021-07-31T02:31:32.798 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:32 smithi045 conmon[33655]: T02:31:32.368212+0000 mon.b (mon.2 2021-07-31T02:31:32.798 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:32 smithi045 conmon[33655]: ) 299 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:31:32.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:32 smithi045 conmon[39718]: cluster 2021-07-31T02:31:31.584710+0000 mon.a 2021-07-31T02:31:32.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:32 smithi045 conmon[39718]: (mon.0) 2263 : cluster [DBG] osdmap e1305: 8 total, 8 up, 8 in 2021-07-31T02:31:32.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:32 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T02:31:32.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:32 smithi045 conmon[39718]: T02:31:32.365859+0000 mon.b (mon.2) 297 : 2021-07-31T02:31:32.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:32 smithi045 conmon[39718]: audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:31:32.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:32 smithi045 conmon[39718]: audit 2021-07-31T02 2021-07-31T02:31:32.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:32 smithi045 conmon[39718]: :31:32.367305+0000 mon.b (mon. 2021-07-31T02:31:32.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:32 smithi045 conmon[39718]: 2) 298 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:31:32.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:32 smithi045 conmon[39718]: audit 2021 2021-07-31T02:31:32.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:32 smithi045 conmon[39718]: -07-31T02:31:32. 2021-07-31T02:31:32.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:32 smithi045 conmon[39718]: 368212+0000 mon.b (mon.2 2021-07-31T02:31:32.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:32 smithi045 conmon[39718]: ) 299 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:31:32.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:32 smithi045 conmon[39718]: 2021-07-31T02:31:33.601 INFO:tasks.workunit.client.0.smithi045.stdout:[----------] 6 tests from TestGroup (2063 ms total) 2021-07-31T02:31:33.602 INFO:tasks.workunit.client.0.smithi045.stdout: 2021-07-31T02:31:33.602 INFO:tasks.workunit.client.0.smithi045.stdout:[----------] 57 tests from TestMigration 2021-07-31T02:31:34.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:31:32.368998+0000 mgr.x (mgr. 2021-07-31T02:31:34.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: 24542) 1894 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:31:34.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:31:32.411986+0000 mgr.x (mgr.24542) 1895 : cephadm [ERR] 2021-07-31T02:31:34.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:31:34.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:31:34.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: yield 2021-07-31T02:31:34.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:31:34.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:31:34.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:31:34.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:31:34.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:31:34.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:31:34.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:31:34.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:31:34.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:31:34.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:31:34.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:31:34.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:31:32.412394+0000 mgr.x (mgr.24542) 1896 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:31:34.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:31:32.423296+0000 mgr.x (mgr.24542) 1897 : cephadm [ERR] 2021-07-31T02:31:34.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:31:34.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:31:34.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: yield 2021-07-31T02:31:34.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:31:34.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:31:34.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:31:34.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:31:34.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:31:34.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:31:34.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:31:34.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:31:34.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:31:34.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:31:34.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:31:34.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:31:34.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: 2021-07-31T02:31:32.424106+0000 mgr.x (mgr.24542) 1898 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:31:34.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: cluster 2021-07-31T02:31:32.431829+0000 mgr.x (mgr.24542) 1899 : cluster [DBG] pgmap v2481: 41 pgs: 32 unknown, 9 active+clean; 4.6 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail 2021-07-31T02:31:34.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: cephadm 2021-07-31T02: 2021-07-31T02:31:34.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: 31:32.448906+0000 mgr.x (mgr.24542) 1900 : cephadm [ERR] 2021-07-31T02:31:34.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:31:34.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:31:34.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: yield 2021-07-31T02:31:34.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:31:34.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:31:34.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:31:34.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:31:34.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:31:34.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:31:34.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:31:34.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:31:34.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:31:34.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:31:34.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:31:34.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:31:32.449347+0000 mgr.x (mgr.24542) 1901 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:31:34.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:31:34.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:31:34.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: yield 2021-07-31T02:31:34.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:31:34.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:31:34.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:31:34.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:31:34.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:31:34.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:31:34.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:31:34.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:31:34.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:31:34.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:31:34.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:31:34.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: 2021-07-31T02:31:34.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:31:34.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: 2021-07-31T02:31:34.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:31:34.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:31:34.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:31:34.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:31:34.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:31:34.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:31:34.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:31:34.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:31:34.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:31:34.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:31:34.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:31:34.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:31:34.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:31:34.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:31:34.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:31:34.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:31:34.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:31:34.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:31:34.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: raise 2021-07-31T02:31:34.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:31:34.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:31:34.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:31:34.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:31:34.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:31:34.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:31:32.457829+0000 mgr.x 2021-07-31T02:31:34.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: (mgr.24542) 1902 : cephadm [ERR] 2021-07-31T02:31:34.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:31:34.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:31:34.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: yield 2021-07-31T02:31:34.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:31:34.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:31:34.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:31:34.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:31:34.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:31:34.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:31:34.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:31:34.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:31:34.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:31:34.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:31:34.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:31:34.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:31:32.458180+0000 mgr.x (mgr.24542) 1903 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:31:34.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:31:34.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:31:34.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: yield 2021-07-31T02:31:34.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:31:34.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:31:34.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:31:34.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:31:34.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:31:34.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:31:34.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:31:34.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:31:34.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:31:34.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:31:34.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:31:34.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: 2021-07-31T02:31:34.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:31:34.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: 2021-07-31T02:31:34.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:31:34.035 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:31:34.035 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:31:34.035 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:31:34.035 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:31:34.036 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:31:34.036 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:31:34.036 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:31:34.036 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:31:34.036 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:31:34.037 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:31:34.037 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:31:34.037 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:31:34.037 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:31:34.037 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:31:34.038 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:31:34.038 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:31:34.038 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:31:34.038 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: raise 2021-07-31T02:31:34.039 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:31:34.039 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:31:34.039 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:31:34.039 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:31:34.039 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:31:34.040 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:33 smithi154 conmon[34553]: cluster 2021-07-31T02:31:32.595100+0000 mon.a (mon.0) 2264 : cluster [DBG] osdmap e1306: 8 total, 8 up, 8 in 2021-07-31T02:31:34.041 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:31:33.586Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:31:34.041 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:31:33.586Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:31:34.041 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:31:33.586Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:31:34.042 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:31:32.368998+0000 mgr.x (mgr.24542) 1894 : cephadm 2021-07-31T02:31:34.042 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:31:34.042 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:31:32.411986+0000 mgr.x (mgr.24542) 1895 : cephadm [ERR] 2021-07-31T02:31:34.042 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:31:34.043 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:31:34.043 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: yield 2021-07-31T02:31:34.043 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:31:34.044 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:31:34.044 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:31:34.044 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:31:34.044 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:31:34.044 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:31:34.045 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:31:34.045 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:31:34.045 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:31:34.045 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:31:34.046 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:31:34.046 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: cephadm 2021- 2021-07-31T02:31:34.046 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: 07-31T02:31:32.412394+0000 mgr.x (mgr.24542) 1896 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:31:34.046 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:31:32.423296+0000 mgr.x (mgr.24542) 1897 : cephadm [ERR] 2021-07-31T02:31:34.046 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:31:34.047 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:31:34.047 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: yield 2021-07-31T02:31:34.047 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:31:34.048 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:31:34.048 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:31:34.048 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:31:34.048 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:31:34.049 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:31:34.049 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:31:34.049 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:31:34.049 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:31:34.050 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:31:34.050 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:31:34.050 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:31:32.424106 2021-07-31T02:31:34.050 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 1898 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:31:34.051 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: cluster 2021-07-31T02:31:32.431829+0000 mgr.x (mgr.24542) 1899 : cluster [DBG] pgmap v2481: 41 pgs: 32 unknown, 9 active+clean; 4.6 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail 2021-07-31T02:31:34.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:31:34.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: 2021-07-31T02:31:32 2021-07-31T02:31:34.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: .368998+0000 mgr.x (mgr.24542) 1894 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:31:34.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:31 2021-07-31T02:31:34.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: :32.411986+0000 mgr.x (mgr.24542) 1895 2021-07-31T02:31:34.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: : cephadm [ERR] 2021-07-31T02:31:34.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:31:34.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:31:34.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: yield 2021-07-31T02:31:34.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:31:34.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:31:34.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:31:34.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:31:34.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:31:34.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:31:34.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:31:34.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:31:34.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:31:34.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:31:34.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:31:34.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:31:32.412394+0000 mgr.x (mgr.24542) 1896 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:31:34.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:31:32.423296+0000 mgr.x (mgr.24542 2021-07-31T02:31:34.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: ) 1897 : cephadm [ERR] 2021-07-31T02:31:34.057 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:31:34.057 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:31:34.057 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: yield 2021-07-31T02:31:34.057 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:31:34.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:31:34.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:31:34.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:31:34.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:31:34.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:31:34.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:31:34.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:31:34.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:31:34.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:31:34.060 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:31:34.060 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: cephadm 2021- 2021-07-31T02:31:34.060 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: 07-31T02:31:32.424106+0000 mgr.x (mgr.24542) 1898 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:31:34.061 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:31:32.448906+0000 mgr.x (mgr. 2021-07-31T02:31:34.061 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: 24542) 1900 : cephadm [ERR] 2021-07-31T02:31:34.061 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:31:34.062 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:31:34.062 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: yield 2021-07-31T02:31:34.062 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:31:34.062 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:31:34.063 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:31:34.063 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:31:34.063 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:31:34.063 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:31:34.064 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:31:34.064 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:31:34.064 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:31:34.065 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:31:34.065 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:31:34.065 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:31:32.449347+0000 mgr.x (mgr.24542) 1901 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:31:34.065 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:31:34.065 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:31:34.066 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: yield 2021-07-31T02:31:34.066 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:31:34.066 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:31:34.066 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:31:34.066 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:31:34.067 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:31:34.067 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:31:34.067 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:31:34.067 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:31:34.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:31:34.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:31:34.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:31:34.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: 2021-07-31T02:31:34.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:31:34.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: 2021-07-31T02:31:34.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:31:34.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:31:34.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:31:34.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:31:34.070 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:31:34.070 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:31:34.070 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:31:34.070 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:31:34.071 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: cluster 2021-07-31T02:31:32.431829+0000 mgr.x (mgr.24542) 1899 : cluster [DBG] pgmap v2481: 41 pgs: 32 unknown, 9 active+clean; 4.6 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail 2021-07-31T02:31:34.071 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:31:34.071 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: 2021-07-31T02:31:32.448906+0000 mgr.x (mgr.24542) 1900 : cephadm 2021-07-31T02:31:34.071 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: [ERR] 2021-07-31T02:31:34.072 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:31:34.072 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:31:34.072 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: yield 2021-07-31T02:31:34.072 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:31:34.073 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:31:34.073 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:31:34.073 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:31:34.073 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:31:34.073 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:31:34.074 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:31:34.074 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:31:34.074 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:31:34.075 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:31:34.075 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:31:34.075 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:31:32.449347+0000 mgr.x (mgr.24542) 1901 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:31:34.075 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:31:34.075 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:31:34.076 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: yield 2021-07-31T02:31:34.076 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:31:34.076 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:31:34.076 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:31:34.077 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:31:34.077 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:31:34.077 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:31:34.077 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:31:34.077 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:31:34.078 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:31:34.078 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:31:34.078 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:31:34.078 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: 2021-07-31T02:31:34.078 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:31:34.079 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: 2021-07-31T02:31:34.079 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:31:34.079 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:31:34.079 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:31:34.080 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:31:34.080 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:31:34.080 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:31:34.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:31:34.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:31:34.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:31:34.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:31:34.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:31:34.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:31:34.084 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:31:34.084 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:31:34.085 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:31:34.085 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:31:34.085 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: raise 2021-07-31T02:31:34.085 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:31:34.085 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:31:34.086 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:31:34.086 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:31:34.086 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:31:34.086 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:31:32.457829+0000 mgr.x (mgr.24542) 1902 : 2021-07-31T02:31:34.087 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: cephadm [ERR] 2021-07-31T02:31:34.087 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:31:34.087 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:31:34.087 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: yield 2021-07-31T02:31:34.088 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:31:34.088 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:31:34.088 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:31:34.088 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:31:34.088 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:31:34.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:31:34.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:31:34.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:31:34.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:31:34.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:31:34.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:31:34.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:31:32.458180+0000 mgr.x (mgr.24542) 1903 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:31:34.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:31:34.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:31:34.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: yield 2021-07-31T02:31:34.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:31:34.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:31:34.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:31:34.092 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:31:34.092 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:31:34.092 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:31:34.093 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:31:34.093 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:31:34.093 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:31:34.093 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:31:34.093 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:31:34.094 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:31:34.094 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:31:34.094 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:31:34.094 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:31:34.095 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: raise 2021-07-31T02:31:34.095 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:31:34.095 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:31:34.095 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:31:34.095 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:31:34.096 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:31:34.096 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: cephadm 2021 2021-07-31T02:31:34.096 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: -07-31T02:31:32.457829+0000 mgr.x (mgr.24542) 1902 : cephadm [ERR] 2021-07-31T02:31:34.096 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:31:34.097 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:31:34.097 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: yield 2021-07-31T02:31:34.097 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:31:34.097 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:31:34.097 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:31:34.098 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:31:34.098 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:31:34.098 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:31:34.098 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:31:34.099 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:31:34.099 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:31:34.099 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:31:34.099 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:31:34.099 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:31:32.458180+0000 mgr.x (mgr.24542) 1903 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:31:34.100 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:31:34.100 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:31:34.100 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: yield 2021-07-31T02:31:34.100 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:31:34.101 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:31:34.101 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:31:34.101 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:31:34.101 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:31:34.102 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:31:34.102 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:31:34.102 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:31:34.102 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:31:34.103 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: 2021-07-31T02:31:34.103 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:31:34.103 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: 2021-07-31T02:31:34.103 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:31:34.103 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:31:34.104 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:31:34.104 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:31:34.104 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:31:34.104 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:31:34.104 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:31:34.105 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:31:34.105 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:31:34.105 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:31:34.105 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:31:34.106 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:31:34.106 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:31:34.106 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:31:34.106 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:31:34.106 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:31:34.107 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:31:34.107 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:31:34.107 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: raise 2021-07-31T02:31:34.107 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:31:34.107 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:31:34.108 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:31:34.108 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:31:34.108 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:31:34.108 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[33655]: cluster 2021-07-31T02:31:32.595100+0000 mon.a (mon.0) 2264 : cluster [DBG] osdmap e1306: 8 total, 8 up, 8 in 2021-07-31T02:31:34.109 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:31:34.109 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:31:34.109 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:31:34.109 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:31:34.110 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:31:34.110 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:31:34.110 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:31:34.110 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:31:34.110 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:31:34.111 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:31:34.111 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: 2021-07-31T02:31:34.111 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:31:34.111 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: 2021-07-31T02:31:34.112 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:31:34.112 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:31:34.112 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:31:34.112 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:31:34.112 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:31:34.113 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:31:34.113 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:31:34.113 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:31:34.113 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:31:34.114 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:31:34.114 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:31:34.114 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:31:34.114 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:31:34.114 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:31:34.115 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:31:34.115 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:31:34.115 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:31:34.115 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:31:34.115 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: raise 2021-07-31T02:31:34.116 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:31:34.116 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:31:34.116 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:31:34.116 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:31:34.117 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:31:34.117 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:33 smithi045 conmon[39718]: cluster 2021-07-31T02:31:32.595100+0000 mon.a (mon.0) 2264 : cluster [DBG] osdmap e1306: 8 total, 8 up, 8 in 2021-07-31T02:31:35.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:34 smithi154 conmon[34553]: cluster 2021-07-31T02:31:33.597718+0000 mon.a (mon.0) 2021-07-31T02:31:35.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:34 smithi154 conmon[34553]: 2265 : cluster [DBG] osdmap e1307: 8 total, 8 up, 8 in 2021-07-31T02:31:35.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:34 smithi045 conmon[33655]: cluster 2021-07-31T02:31:33.597718 2021-07-31T02:31:35.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:34 smithi045 conmon[33655]: +0000 mon.a (mon.0) 2265 : cluster [DBG] osdmap e1307: 8 total, 8 up, 8 in 2021-07-31T02:31:35.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:34 smithi045 conmon[39718]: cluster 2021-07-31T02:31:33.597718+0000 2021-07-31T02:31:35.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:34 smithi045 conmon[39718]: mon.a (mon.0) 2265 : cluster [DBG] osdmap e1307: 8 total, 8 up, 8 in 2021-07-31T02:31:36.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:35 smithi154 conmon[34553]: cluster 2021-07-31T02:31:34.432333+0000 mgr.x (mgr.24542) 1904 : cluster [DBG] pgmap v2484: 9 pgs: 9 active+clean; 4.6 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:31:36.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:35 smithi154 conmon[34553]: cluster 2021-07-31T02:31:34.612718+0000 mon.a (mon.0) 2266 : cluster [DBG] osdmap e1308: 8 total, 8 up, 8 in 2021-07-31T02:31:36.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:35 smithi154 conmon[34553]: audit 2021-07-31T02:31:34.613961+0000 mon.a (mon.0) 2267 : audit [INF] from='client.? 172.21.15.45:0/3782078803' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:31:36.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:35 smithi045 conmon[39718]: cluster 2021-07-31T02:31:34.432333+0000 mgr.x ( 2021-07-31T02:31:36.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:35 smithi045 conmon[39718]: mgr.24542) 1904 : cluster [DBG] pgmap v2484: 9 pgs: 9 active+clean; 4.6 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:31:36.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:35 smithi045 conmon[39718]: cluster 2021-07-31T02:31:34 2021-07-31T02:31:36.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:35 smithi045 conmon[39718]: .612718+0000 mon.a (mon.0) 2266 : cluster [DBG] osdmap e1308: 8 total, 8 up, 8 in 2021-07-31T02:31:36.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:35 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T02:31:36.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:35 smithi045 conmon[39718]: T02:31:34.613961+0000 mon.a 2021-07-31T02:31:36.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:35 smithi045 conmon[39718]: (mon.0) 2267 : audit [INF] from='client.? 172.21.15.45:0/3782078803' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:31:36.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:35 smithi045 conmon[33655]: cluster 2021-07-31T02:31:34.432333+0000 2021-07-31T02:31:36.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:35 smithi045 conmon[33655]: mgr.x (mgr.24542) 1904 : cluster [DBG] pgmap v2484: 9 pgs: 9 active+clean; 4.6 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:31:36.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:35 smithi045 conmon[33655]: cluster 2021 2021-07-31T02:31:36.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:35 smithi045 conmon[33655]: -07-31T02:31:34.612718+0000 2021-07-31T02:31:36.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:35 smithi045 conmon[33655]: mon.a (mon.0) 2266 : cluster 2021-07-31T02:31:36.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:35 smithi045 conmon[33655]: [DBG] osdmap e1308: 8 total, 8 up, 8 in 2021-07-31T02:31:36.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:35 smithi045 conmon[33655]: audit 2021-07-31 2021-07-31T02:31:36.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:35 smithi045 conmon[33655]: T02:31:34.613961+0000 mon.a (mon.0) 2267 : audit [INF] 2021-07-31T02:31:36.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:35 smithi045 conmon[33655]: from='client.? 172.21.15.45:0/3782078803' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:31:36.625 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMigration.Empty 2021-07-31T02:31:37.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:36 smithi154 conmon[34553]: audit 2021-07-31T02:31:35.618961+0000 mon.a (mon.0) 2268 : audit 2021-07-31T02:31:37.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:36 smithi154 conmon[34553]: [INF] from='client.? 172.21.15.45:0/3782078803' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-13","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:31:37.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:36 smithi154 conmon[34553]: cluster 2021-07-31T02:31:35.619276+0000 mon.a (mon.0) 2269 : cluster [DBG] osdmap e1309: 8 total, 8 up, 8 in 2021-07-31T02:31:37.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:36 smithi045 conmon[33655]: audit 2021-07-31T02:31:35.618961+0000 mon.a (mon.0) 2268 : audit [INF] from='client.? 172.21.15.45:0/3782078803' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-13","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:31:37.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:36 smithi045 conmon[33655]: cluster 2021-07-31T02:31:35.619276+0000 mon.a (mon.0) 2269 : cluster [DBG] osdmap e1309: 8 total, 8 up, 8 in 2021-07-31T02:31:37.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:36 smithi045 conmon[39718]: audit 2021-07-31T02:31:35.618961 2021-07-31T02:31:37.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:36 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2268 : audit [INF] from='client.? 172.21.15.45:0/3782078803' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-13","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:31:37.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:36 smithi045 conmon[39718]: cluster 2021-07-31T02: 2021-07-31T02:31:37.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:36 smithi045 conmon[39718]: 31:35.619276+0000 mon.a (mon.0) 2269 : cluster [DBG] osdmap e1309: 8 total, 8 up, 8 in 2021-07-31T02:31:38.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:37 smithi154 conmon[34553]: cluster 2021-07-31T02:31:36.432746+0000 mgr.x (mgr.24542) 1905 : cluster [DBG] pgmap v2487: 41 pgs: 32 unknown, 9 active+clean; 4.6 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:31:38.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:37 smithi154 conmon[34553]: cluster 2021-07-31T02:31:36.624856+0000 mon.a (mon.0) 2270 : cluster [DBG] osdmap e1310: 8 total, 8 up, 8 in 2021-07-31T02:31:38.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:37 smithi045 conmon[39718]: cluster 2021-07-31T02:31:36.432746+0000 mgr.x (mgr.24542) 1905 : cluster [DBG] 2021-07-31T02:31:38.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:37 smithi045 conmon[39718]: pgmap v2487: 41 pgs: 32 unknown, 9 active+clean; 4.6 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:31:38.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:37 smithi045 conmon[39718]: cluster 2021-07-31T02:31:36.624856+0000 mon.a (mon.0) 2270 : cluster [DBG] osdmap e1310: 8 total, 8 up, 8 in 2021-07-31T02:31:38.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:37 smithi045 conmon[33655]: cluster 2021-07-31T02:31:36.432746+0000 mgr.x 2021-07-31T02:31:38.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:37 smithi045 conmon[33655]: (mgr.24542) 1905 : cluster [DBG] pgmap v2487: 41 pgs: 32 unknown, 9 active+clean; 4.6 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:31:38.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:37 smithi045 conmon[33655]: cluster 2021-07 2021-07-31T02:31:38.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:37 smithi045 conmon[33655]: -31T02:31:36.624856+0000 mon.a (mon.0) 2270 : cluster [DBG] osdmap e1310: 8 total, 8 up, 8 in 2021-07-31T02:31:38.663 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 83886080 2021-07-31T02:31:38.667 INFO:tasks.workunit.client.0.smithi045.stdout:migration_prepare 2021-07-31T02:31:39.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:38 smithi154 conmon[34553]: cluster 2021-07-31T02:31:37.630543+0000 mon.a (mon.0) 2271 : cluster [DBG] osdmap e1311: 8 total, 8 up, 8 in 2021-07-31T02:31:39.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:38 smithi045 conmon[33655]: cluster 2021-07-31T02:31:37.630543 2021-07-31T02:31:39.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:38 smithi045 conmon[33655]: +0000 mon.a (mon.0) 2271 : cluster [DBG] osdmap e1311: 8 total, 8 up, 8 in 2021-07-31T02:31:39.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:38 smithi045 conmon[39718]: cluster 2021-07-31T02:31:37.630543 2021-07-31T02:31:39.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:38 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2271 : cluster [DBG] osdmap e1311: 8 total, 8 up, 8 in 2021-07-31T02:31:40.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:39 smithi154 conmon[34553]: cluster 2021-07-31T02:31:38.433165+0000 mgr.x (mgr.24542) 1906 : cluster 2021-07-31T02:31:40.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:39 smithi154 conmon[34553]: [DBG] pgmap v2490: 73 pgs: 64 unknown, 9 active+clean; 4.6 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:31:40.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:39 smithi154 conmon[34553]: cluster 2021-07-31T02:31:38.637357+0000 mon.a (mon.0) 2272 : cluster [DBG] osdmap e1312: 8 total, 8 up, 8 in 2021-07-31T02:31:40.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:39 smithi045 conmon[39718]: cluster 2021-07-31T02:31:38.433165+0000 mgr.x 2021-07-31T02:31:40.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:39 smithi045 conmon[39718]: (mgr.24542) 1906 : cluster [DBG] pgmap v2490: 73 pgs: 64 unknown, 9 active+clean; 4.6 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:31:40.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:39 smithi045 conmon[39718]: cluster 2021-07-31T02:31 2021-07-31T02:31:40.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:39 smithi045 conmon[39718]: :38.637357+0000 mon.a (mon.0) 2272 : cluster [DBG] 2021-07-31T02:31:40.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:39 smithi045 conmon[39718]: osdmap e1312: 8 total, 8 up, 8 in 2021-07-31T02:31:40.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:39 smithi045 conmon[33655]: cluster 2021-07-31T02:31:38.433165+0000 mgr.x (mgr.24542) 1906 : 2021-07-31T02:31:40.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:39 smithi045 conmon[33655]: cluster [DBG] pgmap v2490: 73 pgs: 64 unknown, 9 active+clean; 4.6 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:31:40.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:39 smithi045 conmon[33655]: cluster 2021-07-31T02:31:38.637357+0000 mon.a (mon.0) 2272 : cluster [DBG] osdmap e1312: 8 total, 8 up, 8 in 2021-07-31T02:31:40.792 INFO:tasks.workunit.client.0.smithi045.stdout:migration_execute 2021-07-31T02:31:40.867 INFO:tasks.workunit.client.0.smithi045.stdout:migration_commit 2021-07-31T02:31:40.945 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMigration.Empty (4319 ms) 2021-07-31T02:31:40.946 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMigration.OtherName 2021-07-31T02:31:40.961 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 83886080 2021-07-31T02:31:40.964 INFO:tasks.workunit.client.0.smithi045.stdout:migration_prepare 2021-07-31T02:31:41.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:40 smithi154 conmon[34553]: cluster 2021-07-31T02:31:41.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:40 smithi154 conmon[34553]: 2021-07-31T02:31:39. 2021-07-31T02:31:41.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:40 smithi154 conmon[34553]: 643136+0000 mon.a (mon.0) 2273 : cluster [DBG] osdmap e1313: 8 total, 8 up, 8 in 2021-07-31T02:31:41.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:40 smithi045 conmon[39718]: cluster 2021-07-31T02:31:39.643136+0000 mon.a ( 2021-07-31T02:31:41.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:40 smithi045 conmon[39718]: mon.0) 2273 : cluster [DBG] osdmap e1313: 8 total, 8 up, 8 in 2021-07-31T02:31:41.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:40 smithi045 conmon[33655]: cluster 2021-07-31T02:31:39.643136+0000 mon.a (mon. 2021-07-31T02:31:41.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:40 smithi045 conmon[33655]: 0) 2273 : cluster [DBG] osdmap e1313: 8 total, 8 up, 8 in 2021-07-31T02:31:41.071 INFO:tasks.workunit.client.0.smithi045.stdout:migration_execute 2021-07-31T02:31:41.130 INFO:tasks.workunit.client.0.smithi045.stdout:migration_commit 2021-07-31T02:31:41.210 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMigration.OtherName (265 ms) 2021-07-31T02:31:41.210 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMigration.OtherPool 2021-07-31T02:31:41.225 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 83886080 2021-07-31T02:31:41.231 INFO:tasks.workunit.client.0.smithi045.stdout:migration_prepare 2021-07-31T02:31:42.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:41 smithi154 conmon[34553]: cluster 2021-07-31T02:31:40.433621+0000 mgr.x (mgr.24542) 1907 : cluster [DBG] pgmap v2493: 73 pgs: 73 active+clean; 4.6 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 2.7 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2021-07-31T02:31:42.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:41 smithi154 conmon[34553]: cluster 2021-07-31T02:31:40.650321+0000 mon.a (mon.0) 2274 : cluster [DBG] osdmap e1314: 8 total, 8 up, 8 in 2021-07-31T02:31:42.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:41 smithi045 conmon[39718]: cluster 2021-07-31T02:31:40.433621+0000 mgr.x ( 2021-07-31T02:31:42.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:41 smithi045 conmon[39718]: mgr.24542) 1907 : cluster [DBG] pgmap v2493: 73 pgs: 73 active+clean; 4.6 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 2.7 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2021-07-31T02:31:42.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:41 smithi045 conmon[39718]: cluster 2021-07-31T02:31:40. 2021-07-31T02:31:42.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:41 smithi045 conmon[39718]: 650321+0000 mon.a (mon.0) 2274 : cluster [DBG] osdmap e1314: 8 total, 8 up, 8 in 2021-07-31T02:31:42.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:41 smithi045 conmon[33655]: cluster 2021-07-31T02:31:40.433621+0000 mgr.x (mgr.24542) 1907 : cluster [DBG] 2021-07-31T02:31:42.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:41 smithi045 conmon[33655]: pgmap v2493: 73 pgs: 73 active+clean; 4.6 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 2.7 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2021-07-31T02:31:42.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:41 smithi045 conmon[33655]: cluster 2021-07-31T02:31:40.650321+0000 mon.a (mon.0) 2274 : cluster [DBG] osdmap e1314: 8 total, 8 up, 8 in 2021-07-31T02:31:42.429 INFO:tasks.workunit.client.0.smithi045.stdout:migration_execute 2021-07-31T02:31:42.483 INFO:tasks.workunit.client.0.smithi045.stdout:migration_commit 2021-07-31T02:31:42.556 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMigration.OtherPool (1346 ms) 2021-07-31T02:31:42.557 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMigration.OtherNamespace 2021-07-31T02:31:42.574 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 83886080 2021-07-31T02:31:42.579 INFO:tasks.workunit.client.0.smithi045.stdout:migration_prepare 2021-07-31T02:31:42.683 INFO:tasks.workunit.client.0.smithi045.stdout:migration_execute 2021-07-31T02:31:42.736 INFO:tasks.workunit.client.0.smithi045.stdout:migration_commit 2021-07-31T02:31:42.813 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMigration.OtherNamespace (256 ms) 2021-07-31T02:31:42.813 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMigration.DataPool 2021-07-31T02:31:42.835 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 83886080 2021-07-31T02:31:42.838 INFO:tasks.workunit.client.0.smithi045.stdout:migration_prepare 2021-07-31T02:31:42.961 INFO:tasks.workunit.client.0.smithi045.stdout:migration_execute 2021-07-31T02:31:43.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:42 smithi154 conmon[34553]: cluster 2021-07-31T02:31:41.655498+0000 mon.a 2021-07-31T02:31:43.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:42 smithi154 conmon[34553]: (mon.0) 2275 : cluster [DBG] osdmap e1315: 8 total, 8 up, 8 in 2021-07-31T02:31:43.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:42 smithi154 conmon[34553]: cluster 2021-07-31T02:31:42.310844+0000 mon.a (mon.0) 2276 : cluster 2021-07-31T02:31:43.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:42 smithi154 conmon[34553]: [DBG] osdmap e1316: 8 total, 8 up, 8 in 2021-07-31T02:31:43.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:42 smithi045 conmon[39718]: cluster 2021-07-31T02:31:41.655498+0000 mon.a (mon 2021-07-31T02:31:43.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:42 smithi045 conmon[39718]: .0) 2275 : cluster [DBG] osdmap e1315: 8 total, 8 up, 8 in 2021-07-31T02:31:43.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:42 smithi045 conmon[39718]: cluster 2021-07-31T02:31:42.310844+0000 mon.a (mon.0) 2276 : cluster [DBG] osdmap e1316: 8 total, 8 up, 8 in 2021-07-31T02:31:43.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:42 smithi045 conmon[33655]: cluster 2021-07-31T02:31:41.655498+0000 mon.a (mon.0) 2275 2021-07-31T02:31:43.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:42 smithi045 conmon[33655]: : cluster [DBG] osdmap e1315: 8 total, 8 up, 8 in 2021-07-31T02:31:43.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:42 smithi045 conmon[33655]: cluster 2021-07-31T02:31:42.310844+0000 mon.a (mon.0) 2276 : cluster [DBG] osdmap e1316: 8 total, 8 up, 8 in 2021-07-31T02:31:43.030 INFO:tasks.workunit.client.0.smithi045.stdout:migration_commit 2021-07-31T02:31:43.107 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMigration.DataPool (295 ms) 2021-07-31T02:31:43.108 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMigration.AbortAfterPrepare 2021-07-31T02:31:43.125 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 83886080 2021-07-31T02:31:43.129 INFO:tasks.workunit.client.0.smithi045.stdout:migration_prepare 2021-07-31T02:31:43.249 INFO:tasks.workunit.client.0.smithi045.stdout:migration_abort 2021-07-31T02:31:43.341 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMigration.AbortAfterPrepare (233 ms) 2021-07-31T02:31:43.341 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMigration.AbortAfterFailedPrepare 2021-07-31T02:31:43.358 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 83886080 2021-07-31T02:31:43.361 INFO:tasks.workunit.client.0.smithi045.stdout:migration_prepare 2021-07-31T02:31:43.427 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMigration.AbortAfterFailedPrepare (86 ms) 2021-07-31T02:31:43.428 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMigration.AbortAfterExecute 2021-07-31T02:31:43.446 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 83886080 2021-07-31T02:31:43.446 INFO:tasks.workunit.client.0.smithi045.stdout:migration_prepare 2021-07-31T02:31:43.560 INFO:tasks.workunit.client.0.smithi045.stdout:migration_execute 2021-07-31T02:31:43.625 INFO:tasks.workunit.client.0.smithi045.stdout:migration_abort 2021-07-31T02:31:43.714 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMigration.AbortAfterExecute (288 ms) 2021-07-31T02:31:43.715 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMigration.OtherPoolAbortAfterExecute 2021-07-31T02:31:43.731 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 83886080 2021-07-31T02:31:43.735 INFO:tasks.workunit.client.0.smithi045.stdout:migration_prepare 2021-07-31T02:31:43.837 INFO:tasks.workunit.client.0.smithi045.stdout:migration_execute 2021-07-31T02:31:43.891 INFO:tasks.workunit.client.0.smithi045.stdout:migration_abort 2021-07-31T02:31:43.980 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMigration.OtherPoolAbortAfterExecute (265 ms) 2021-07-31T02:31:43.980 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMigration.OtherNamespaceAbortAfterExecute 2021-07-31T02:31:43.998 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 83886080 2021-07-31T02:31:44.004 INFO:tasks.workunit.client.0.smithi045.stdout:migration_prepare 2021-07-31T02:31:44.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:31:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:31:43.586Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:31:44.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:31:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:31:43.586Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:31:44.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:31:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:31:43.586Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:31:44.105 INFO:tasks.workunit.client.0.smithi045.stdout:migration_execute 2021-07-31T02:31:44.161 INFO:tasks.workunit.client.0.smithi045.stdout:migration_abort 2021-07-31T02:31:44.252 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMigration.OtherNamespaceAbortAfterExecute (272 ms) 2021-07-31T02:31:44.252 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMigration.MirroringSamePool 2021-07-31T02:31:44.267 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 83886080 2021-07-31T02:31:44.272 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:31:44.275 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMigration.MirroringSamePool (23 ms) 2021-07-31T02:31:44.275 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMigration.MirroringAbort 2021-07-31T02:31:44.294 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 83886080 2021-07-31T02:31:44.297 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:31:44.299 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMigration.MirroringAbort (25 ms) 2021-07-31T02:31:44.299 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMigration.MirroringOtherPoolDisabled 2021-07-31T02:31:44.318 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 83886080 2021-07-31T02:31:44.322 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:31:44.325 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMigration.MirroringOtherPoolDisabled (24 ms) 2021-07-31T02:31:44.327 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMigration.MirroringOtherPoolEnabled 2021-07-31T02:31:44.347 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 83886080 2021-07-31T02:31:44.347 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:31:44.348 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMigration.MirroringOtherPoolEnabled (25 ms) 2021-07-31T02:31:44.349 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMigration.MirroringPool 2021-07-31T02:31:44.363 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 83886080 2021-07-31T02:31:44.369 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:31:44.371 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMigration.MirroringPool (23 ms) 2021-07-31T02:31:44.371 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMigration.Group 2021-07-31T02:31:44.388 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 83886080 2021-07-31T02:31:44.391 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:31:44.393 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMigration.Group (21 ms) 2021-07-31T02:31:44.394 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMigration.GroupAbort 2021-07-31T02:31:44.412 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 83886080 2021-07-31T02:31:44.415 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:31:44.417 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMigration.GroupAbort (25 ms) 2021-07-31T02:31:44.417 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMigration.NoSnaps 2021-07-31T02:31:44.432 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 83886080 2021-07-31T02:31:44.436 INFO:tasks.workunit.client.0.smithi045.stdout:write: 1 0~8388609 2021-07-31T02:31:44.495 INFO:tasks.workunit.client.0.smithi045.stdout:write: 1 16777218~8388609 2021-07-31T02:31:44.546 INFO:tasks.workunit.client.0.smithi045.stdout:migration_prepare 2021-07-31T02:31:44.719 INFO:tasks.workunit.client.0.smithi045.stdout:migration_execute 2021-07-31T02:31:44.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:44 smithi154 conmon[34553]: cluster 2021-07-31T02:31:42.433978+0000 mgr.x (mgr.24542) 2021-07-31T02:31:44.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:44 smithi154 conmon[34553]: 1908 : cluster [DBG] pgmap v2497: 73 pgs: 73 active+clean; 4.6 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 2.9 KiB/s rd, 2.4 KiB/s wr, 7 op/s 2021-07-31T02:31:44.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:44 smithi045 conmon[33655]: cluster 2021-07-31T02:31:42.433978+0000 2021-07-31T02:31:44.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:44 smithi045 conmon[33655]: mgr.x (mgr.24542) 1908 : cluster [DBG] pgmap v2497: 73 pgs: 73 active+clean; 4.6 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 2.9 KiB/s rd, 2.4 KiB/s wr, 7 op/s 2021-07-31T02:31:44.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:44 smithi045 conmon[39718]: cluster 2021-07-31T02 2021-07-31T02:31:44.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:44 smithi045 conmon[39718]: :31:42.433978+0000 mgr.x (mgr.24542) 1908 : cluster [DBG] pgmap v2497: 73 pgs: 73 active+clean; 4.6 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 2.9 KiB/s rd, 2.4 KiB/s wr, 7 op/s 2021-07-31T02:31:44.825 INFO:tasks.workunit.client.0.smithi045.stdout:migration_commit 2021-07-31T02:31:44.940 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMigration.NoSnaps (523 ms) 2021-07-31T02:31:44.941 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMigration.NoSnapsOtherPool 2021-07-31T02:31:44.956 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 83886080 2021-07-31T02:31:44.960 INFO:tasks.workunit.client.0.smithi045.stdout:write: 1 0~8388609 2021-07-31T02:31:45.011 INFO:tasks.workunit.client.0.smithi045.stdout:write: 1 16777218~8388609 2021-07-31T02:31:45.060 INFO:tasks.workunit.client.0.smithi045.stdout:write: 1 0~8388609 2021-07-31T02:31:45.065 INFO:tasks.workunit.client.0.smithi045.stdout:write: 1 16777218~8388609 2021-07-31T02:31:45.126 INFO:tasks.workunit.client.0.smithi045.stdout:migration_prepare 2021-07-31T02:31:45.267 INFO:tasks.workunit.client.0.smithi045.stdout:migration_execute 2021-07-31T02:31:45.360 INFO:tasks.workunit.client.0.smithi045.stdout:migration_commit 2021-07-31T02:31:45.470 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMigration.NoSnapsOtherPool (530 ms) 2021-07-31T02:31:45.470 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMigration.NoSnapsDataPool 2021-07-31T02:31:45.489 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 83886080 2021-07-31T02:31:45.495 INFO:tasks.workunit.client.0.smithi045.stdout:write: 1 0~8388609 2021-07-31T02:31:45.551 INFO:tasks.workunit.client.0.smithi045.stdout:write: 1 16777218~8388609 2021-07-31T02:31:45.599 INFO:tasks.workunit.client.0.smithi045.stdout:migration_prepare 2021-07-31T02:31:45.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:45 smithi154 conmon[34553]: cluster 2021-07-31T02:31:45.315176+0000 mon.a (mon.0 2021-07-31T02:31:45.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:45 smithi154 conmon[34553]: ) 2277 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2021-07-31T02:31:45.768 INFO:tasks.workunit.client.0.smithi045.stdout:migration_execute 2021-07-31T02:31:45.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:45 smithi045 conmon[33655]: cluster 2021-07-31T02:31:45.315176+0000 mon.a (mon.0) 2277 : cluster [WRN] 2021-07-31T02:31:45.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:45 smithi045 conmon[33655]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2021-07-31T02:31:45.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:45 smithi045 conmon[39718]: cluster 2021-07-31T02:31:45.315176+0000 2021-07-31T02:31:45.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:45 smithi045 conmon[39718]: mon.a (mon.0) 2277 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2021-07-31T02:31:45.867 INFO:tasks.workunit.client.0.smithi045.stdout:migration_commit 2021-07-31T02:31:45.979 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMigration.NoSnapsDataPool (509 ms) 2021-07-31T02:31:45.980 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMigration.NoSnapsShrinkAfterPrepare 2021-07-31T02:31:45.998 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 83886080 2021-07-31T02:31:46.001 INFO:tasks.workunit.client.0.smithi045.stdout:write: 1 0~8388609 2021-07-31T02:31:46.056 INFO:tasks.workunit.client.0.smithi045.stdout:write: 1 16777218~8388609 2021-07-31T02:31:46.097 INFO:tasks.workunit.client.0.smithi045.stdout:migration_prepare 2021-07-31T02:31:46.264 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 41943040 2021-07-31T02:31:46.272 INFO:tasks.workunit.client.0.smithi045.stdout:migration_execute 2021-07-31T02:31:46.379 INFO:tasks.workunit.client.0.smithi045.stdout:migration_commit 2021-07-31T02:31:46.472 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMigration.NoSnapsShrinkAfterPrepare (493 ms) 2021-07-31T02:31:46.472 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMigration.NoSnapsShrinkToZeroBeforePrepare 2021-07-31T02:31:46.487 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 83886080 2021-07-31T02:31:46.491 INFO:tasks.workunit.client.0.smithi045.stdout:write: 1 0~8388609 2021-07-31T02:31:46.543 INFO:tasks.workunit.client.0.smithi045.stdout:write: 1 16777218~8388609 2021-07-31T02:31:46.591 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 0 2021-07-31T02:31:46.603 INFO:tasks.workunit.client.0.smithi045.stdout:migration_prepare 2021-07-31T02:31:46.671 INFO:tasks.workunit.client.0.smithi045.stdout:migration_execute 2021-07-31T02:31:46.672 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:31:46.727 INFO:tasks.workunit.client.0.smithi045.stdout:migration_commit 2021-07-31T02:31:46.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:46 smithi154 conmon[34553]: cluster 2021-07-31T02:31:44.434513+0000 mgr.x (mgr.24542) 2021-07-31T02:31:46.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:46 smithi154 conmon[34553]: 1909 : cluster [DBG] pgmap v2498: 73 pgs: 73 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 509 KiB/s rd, 50 KiB/s wr, 681 op/s 2021-07-31T02:31:46.763 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMigration.NoSnapsShrinkToZeroBeforePrepare (290 ms) 2021-07-31T02:31:46.763 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMigration.NoSnapsShrinkToZeroAfterPrepare 2021-07-31T02:31:46.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:46 smithi045 conmon[39718]: cluster 2021-07-31T02:31:46.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:46 smithi045 conmon[39718]: 2021-07- 2021-07-31T02:31:46.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:46 smithi045 conmon[39718]: 31T02:31:44.434513+0000 mgr.x (mgr.24542) 1909 : cluster [DBG] pgmap v2498: 73 pgs: 73 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 509 KiB/s rd, 50 KiB/s wr, 681 op/s 2021-07-31T02:31:46.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:46 smithi045 conmon[33655]: cluster 2021-07-31T02:31:44.434513+0000 2021-07-31T02:31:46.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:46 smithi045 conmon[33655]: mgr.x (mgr.24542) 1909 : cluster [DBG] pgmap v2498: 73 pgs: 73 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 509 KiB/s rd, 50 KiB/s wr, 681 op/s 2021-07-31T02:31:46.781 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 83886080 2021-07-31T02:31:46.785 INFO:tasks.workunit.client.0.smithi045.stdout:write: 1 0~8388609 2021-07-31T02:31:46.831 INFO:tasks.workunit.client.0.smithi045.stdout:write: 1 16777218~8388609 2021-07-31T02:31:46.884 INFO:tasks.workunit.client.0.smithi045.stdout:migration_prepare 2021-07-31T02:31:46.986 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:31:46.986 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 46m ago 47m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:31:46.986 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 44m ago 47m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:31:46.987 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 44m ago 50m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:31:46.987 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 46m ago 52m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:31:46.987 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 46m ago 52m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:31:46.987 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 44m ago 50m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:31:46.988 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 46m ago 51m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:31:46.988 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 46m ago 47m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:31:46.988 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 44m ago 47m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:31:46.988 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 46m ago 50m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:31:46.989 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 46m ago 50m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:31:46.989 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 46m ago 49m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:31:46.989 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 46m ago 49m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:31:46.989 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 44m ago 49m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:31:46.989 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 44m ago 48m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:31:46.990 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 44m ago 48m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:31:46.990 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 44m ago 48m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:31:46.990 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 44m ago 47m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:31:47.056 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 0 2021-07-31T02:31:47.069 INFO:tasks.workunit.client.0.smithi045.stdout:migration_execute 2021-07-31T02:31:47.135 INFO:tasks.workunit.client.0.smithi045.stdout:migration_commit 2021-07-31T02:31:47.180 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMigration.NoSnapsShrinkToZeroAfterPrepare (418 ms) 2021-07-31T02:31:47.181 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMigration.NoSnapsExpandAfterPrepare 2021-07-31T02:31:47.196 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 83886080 2021-07-31T02:31:47.201 INFO:tasks.workunit.client.0.smithi045.stdout:write: 1 0~8388609 2021-07-31T02:31:47.259 INFO:tasks.workunit.client.0.smithi045.stdout:write: 1 16777218~8388609 2021-07-31T02:31:47.304 INFO:tasks.workunit.client.0.smithi045.stdout:migration_prepare 2021-07-31T02:31:47.387 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:31:47.387 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:31:47.387 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:31:47.387 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:31:47.388 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:31:47.388 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:31:47.388 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:31:47.389 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:31:47.390 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:31:47.390 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:31:47.390 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:31:47.390 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:31:47.391 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:31:47.391 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:31:47.391 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:31:47.391 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:31:47.391 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:31:47.474 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 167772160 2021-07-31T02:31:47.476 INFO:tasks.workunit.client.0.smithi045.stdout:migration_execute 2021-07-31T02:31:47.577 INFO:tasks.workunit.client.0.smithi045.stdout:migration_commit 2021-07-31T02:31:47.718 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMigration.NoSnapsExpandAfterPrepare (537 ms) 2021-07-31T02:31:47.718 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMigration.NoSnapsSnapAfterPrepare 2021-07-31T02:31:47.734 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 83886080 2021-07-31T02:31:47.737 INFO:tasks.workunit.client.0.smithi045.stdout:write: 1 0~8388609 2021-07-31T02:31:47.781 INFO:tasks.workunit.client.0.smithi045.stdout:write: 1 16777218~8388609 2021-07-31T02:31:47.823 INFO:tasks.workunit.client.0.smithi045.stdout:migration_prepare 2021-07-31T02:31:47.983 INFO:tasks.workunit.client.0.smithi045.stdout:snap_create: after_prepare_snap 2021-07-31T02:31:48.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:48 smithi154 conmon[34553]: cluster 2021-07-31T02:31:46.434912+0000 mgr.x (mgr.24542) 1910 : cluster [DBG] pgmap v2499: 73 pgs: 73 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 407 KiB/s rd, 40 KiB/s wr, 544 op/s 2021-07-31T02:31:48.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:48 smithi154 conmon[34553]: audit 2021-07-31T02:31:46.660483+0000 mgr.x (mgr.24542) 1911 : audit [DBG] from='client.27310 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:31:48.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:48 smithi154 conmon[34553]: audit 2021-07-31T02:31:46.977685+0000 mgr.x (mgr.24542) 1912 : audit [DBG] 2021-07-31T02:31:48.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:48 smithi154 conmon[34553]: from='client.27316 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:31:48.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:48 smithi154 conmon[34553]: cluster 2021-07-31T02: 2021-07-31T02:31:48.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:48 smithi154 conmon[34553]: 31:47.311052+0000 mon.a (mon 2021-07-31T02:31:48.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:48 smithi154 conmon[34553]: .0) 2278 : cluster [DBG] osdmap e1317: 8 total, 8 up, 8 in 2021-07-31T02:31:48.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:48 smithi154 conmon[34553]: 2021-07-31T02:31:48.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:48 smithi154 conmon[34553]: audit 2021-07-31T02:31:48.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:48 smithi154 conmon[34553]: 2021-07-31T02:31: 2021-07-31T02:31:48.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:48 smithi154 conmon[34553]: 47.387085+0000 mon.c (mon. 2021-07-31T02:31:48.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:48 smithi154 conmon[34553]: 1) 116 : audit [DBG] from='client.? 172.21.15.45:0/2529901286' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:31:48.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:48 smithi045 conmon[33655]: cluster 2021-07-31T02:31:46.434912 2021-07-31T02:31:48.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:48 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 1910 : cluster [DBG] pgmap v2499: 73 pgs: 73 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 407 KiB/s rd, 40 KiB/s wr, 544 op/s 2021-07-31T02:31:48.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:48 smithi045 conmon[33655]: audit 2021-07- 2021-07-31T02:31:48.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:48 smithi045 conmon[33655]: 31T02:31:46.660483+0000 mgr.x (mgr.24542) 1911 : audit [DBG] from='client.27310 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:31:48.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:48 smithi045 conmon[33655]: audit 2021- 2021-07-31T02:31:48.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:48 smithi045 conmon[33655]: 07-31T02:31:46.977685+0000 mgr.x (mgr.24542) 2021-07-31T02:31:48.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:48 smithi045 conmon[33655]: 1912 : audit [DBG] from='client.27316 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:31:48.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:48 smithi045 conmon[33655]: cluster 2021-07-31T02:31:47.311052 2021-07-31T02:31:48.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:48 smithi045 conmon[33655]: +0000 mon.a (mon.0) 2278 : cluster [DBG] osdmap e1317: 8 total, 8 up, 8 in 2021-07-31T02:31:48.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:48 smithi045 conmon[33655]: audit 2021-07 2021-07-31T02:31:48.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:48 smithi045 conmon[33655]: -31T02:31:47.387085+0000 mon.c (mon.1) 2021-07-31T02:31:48.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:48 smithi045 conmon[33655]: 116 : audit [DBG] from='client.? 172.21.15.45:0/2529901286' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:31:48.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:48 smithi045 conmon[39718]: cluster 2021-07-31T02:31:46.434912+0000 mgr.x (mgr.24542) 1910 2021-07-31T02:31:48.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:48 smithi045 conmon[39718]: : cluster [DBG] pgmap v2499: 73 pgs: 73 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 407 KiB/s rd, 40 KiB/s wr, 544 op/s 2021-07-31T02:31:48.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:48 smithi045 conmon[39718]: audit 2021-07-31T02:31:46.660483+0000 mgr.x (mgr.24542) 1911 : audit 2021-07-31T02:31:48.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:48 smithi045 conmon[39718]: [DBG] from='client.27310 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:31:48.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:48 smithi045 conmon[39718]: audit 2021-07-31T02:31:46.977685+0000 mgr.x (mgr. 2021-07-31T02:31:48.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:48 smithi045 conmon[39718]: 24542) 1912 : audit [DBG] from='client.27316 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:31:48.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:48 smithi045 conmon[39718]: cluster 2021-07-31T 2021-07-31T02:31:48.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:48 smithi045 conmon[39718]: 02:31:47.311052+0000 mon.a (mon.0) 2278 : cluster [DBG] osdmap e1317: 8 total, 8 up, 8 in 2021-07-31T02:31:48.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:48 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T02:31:48.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:48 smithi045 conmon[39718]: 02:31:47.387085+0000 mon.c (mon.1 2021-07-31T02:31:48.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:48 smithi045 conmon[39718]: ) 116 : audit [DBG] from='client.? 172.21.15.45:0/2529901286' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:31:49.383 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 41943040 2021-07-31T02:31:49.411 INFO:tasks.workunit.client.0.smithi045.stdout:write: * 0~1000 2021-07-31T02:31:49.415 INFO:tasks.workunit.client.0.smithi045.stdout:migration_execute 2021-07-31T02:31:49.521 INFO:tasks.workunit.client.0.smithi045.stdout:migration_commit 2021-07-31T02:31:49.691 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMigration.NoSnapsSnapAfterPrepare (1962 ms) 2021-07-31T02:31:49.691 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMigration.Snaps 2021-07-31T02:31:49.695 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 83886080 2021-07-31T02:31:49.699 INFO:tasks.workunit.client.0.smithi045.stdout:write: 1 0~8388609 2021-07-31T02:31:49.750 INFO:tasks.workunit.client.0.smithi045.stdout:snap_create: snap1 2021-07-31T02:31:49.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:49 smithi154 conmon[34553]: cluster 2021-07-31T02:31:48.377780+0000 mon.a (mon.0) 2279 : cluster [DBG] 2021-07-31T02:31:49.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:49 smithi154 conmon[34553]: osdmap e1318: 8 total, 8 up, 8 in 2021-07-31T02:31:49.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:49 smithi154 conmon[34553]: cluster 2021-07-31T02:31:48.435562+0000 mgr.x (mgr.24542) 1913 : cluster [DBG] pgmap v2502: 73 pgs: 73 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 398 KiB/s rd, 39 KiB/s wr, 533 op/s 2021-07-31T02:31:49.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:49 smithi045 conmon[39718]: cluster 2021-07-31T02:31:48.377780+0000 mon.a ( 2021-07-31T02:31:49.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:49 smithi045 conmon[39718]: mon.0) 2279 : cluster [DBG] osdmap e1318: 8 total, 8 up, 8 in 2021-07-31T02:31:49.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:49 smithi045 conmon[39718]: cluster 2021-07-31T02:31:48.435562+0000 2021-07-31T02:31:49.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:49 smithi045 conmon[39718]: mgr.x (mgr.24542) 1913 : cluster [DBG] pgmap v2502: 73 pgs: 73 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 398 KiB/s rd, 39 KiB/s wr, 533 op/s 2021-07-31T02:31:49.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:49 smithi045 conmon[33655]: cluster 2021-07-31 2021-07-31T02:31:49.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:49 smithi045 conmon[33655]: T02:31:48.377780+0000 mon.a (mon.0) 2279 : cluster [DBG] osdmap e1318: 8 total, 8 up, 8 in 2021-07-31T02:31:49.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:49 smithi045 conmon[33655]: cluster 2021-07-31T02:31:48.435562+0000 mgr.x (mgr.24542) 1913 : cluster [DBG] pgmap v2502: 73 pgs: 73 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 398 KiB/s rd, 39 KiB/s wr, 533 op/s 2021-07-31T02:31:50.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:50 smithi154 conmon[34553]: cluster 2021-07-31T02:31:49.378251+0000 mon.a (mon.0) 2280 : cluster [DBG] osdmap e1319: 8 total, 8 up, 8 in 2021-07-31T02:31:50.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:50 smithi045 conmon[33655]: cluster 2021-07-31T02:31:49.378251+0000 mon.a (mon. 2021-07-31T02:31:50.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:50 smithi045 conmon[33655]: 0) 2280 : cluster [DBG] osdmap e1319: 8 total, 8 up, 8 in 2021-07-31T02:31:50.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:50 smithi045 conmon[39718]: cluster 2021-07-31T02:31:49.378251+0000 mon.a (mon. 2021-07-31T02:31:50.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:50 smithi045 conmon[39718]: 0) 2280 : cluster [DBG] osdmap e1319: 8 total, 8 up, 8 in 2021-07-31T02:31:51.402 INFO:tasks.workunit.client.0.smithi045.stdout:write: 1 8388609~8388609 2021-07-31T02:31:51.414 INFO:tasks.workunit.client.0.smithi045.stdout:write: X 0~1000 2021-07-31T02:31:51.415 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 1010~1000 2021-07-31T02:31:51.415 INFO:tasks.workunit.client.0.smithi045.stdout:snap_create: snap2 2021-07-31T02:31:51.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:51 smithi154 conmon[34553]: cluster 2021-07-31T02:31:50.391134+0000 mon.a (mon.0) 2281 : cluster [DBG] 2021-07-31T02:31:51.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:51 smithi154 conmon[34553]: osdmap e1320: 8 total, 8 up, 8 in 2021-07-31T02:31:51.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:51 smithi154 conmon[34553]: cluster 2021-07-31T02:31:50.436030+0000 mgr.x (mgr.24542) 1914 : cluster [DBG] pgmap v2505: 73 pgs: 73 active+clean; 197 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 117 MiB/s rd, 92 MiB/s wr, 1.14k op/s 2021-07-31T02:31:51.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:51 smithi045 conmon[39718]: cluster 2021-07-31T02:31:50.391134+0000 mon.a (mon.0) 2281 2021-07-31T02:31:51.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:51 smithi045 conmon[39718]: : cluster [DBG] osdmap e1320: 8 total, 8 up, 8 in 2021-07-31T02:31:51.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:51 smithi045 conmon[39718]: cluster 2021-07-31T02:31:50.436030+0000 mgr.x (mgr.24542) 1914 : cluster [DBG] pgmap v2505: 73 pgs: 73 active+clean; 197 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 117 MiB/s rd, 92 MiB/s wr, 1.14k op/s 2021-07-31T02:31:51.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:51 smithi045 conmon[33655]: cluster 2021-07-31T02:31:50.391134 2021-07-31T02:31:51.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:51 smithi045 conmon[33655]: +0000 mon.a (mon.0) 2281 : cluster [DBG] osdmap e1320: 8 total, 8 up, 8 in 2021-07-31T02:31:51.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:51 smithi045 conmon[33655]: cluster 2021-07-31T02:31:50.436030+0000 mgr.x (mgr.24542) 1914 : cluster [DBG] pgmap v2505: 73 pgs: 73 active+clean; 197 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 117 MiB/s rd, 92 MiB/s wr, 1.14k op/s 2021-07-31T02:31:52.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:52 smithi154 conmon[34553]: cluster 2021-07-31T02:31:51.396647+0000 mon.a (mon.0) 2282 : cluster [DBG] osdmap e1321: 8 total, 8 up, 8 in 2021-07-31T02:31:52.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:52 smithi045 conmon[39718]: cluster 2021-07-31T02:31:51 2021-07-31T02:31:52.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:52 smithi045 conmon[39718]: .396647+0000 mon.a (mon.0) 2282 : cluster [DBG] osdmap e1321: 8 total, 8 up, 8 in 2021-07-31T02:31:52.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:52 smithi045 conmon[33655]: cluster 2021-07-31T02:31:51.396647+0000 mon.a (mon.0) 2282 : cluster 2021-07-31T02:31:52.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:52 smithi045 conmon[33655]: [DBG] osdmap e1321: 8 total, 8 up, 8 in 2021-07-31T02:31:53.413 INFO:tasks.workunit.client.0.smithi045.stdout:write: X 8388609~1000 2021-07-31T02:31:53.415 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 16777228~1000 2021-07-31T02:31:53.416 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 167772160 2021-07-31T02:31:53.424 INFO:tasks.workunit.client.0.smithi045.stdout:write: 2 83886079~8388609 2021-07-31T02:31:53.442 INFO:tasks.workunit.client.0.smithi045.stdout:snap_create: snap3 2021-07-31T02:31:53.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:53 smithi154 conmon[34553]: cluster 2021-07-31T02:31:52.402276+0000 mon.a (mon.0) 2283 : cluster 2021-07-31T02:31:53.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:53 smithi154 conmon[34553]: [DBG] osdmap e1322: 8 total, 8 up, 8 in 2021-07-31T02:31:53.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:53 smithi154 conmon[34553]: cluster 2021-07-31T02:31:52.436548+0000 mgr.x (mgr.24542) 1915 : cluster [DBG] pgmap v2508: 73 pgs: 73 active+clean; 197 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 117 MiB/s rd, 92 MiB/s wr, 1.14k op/s 2021-07-31T02:31:53.775 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:31:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:31:53.587Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:31:53.775 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:31:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:31:53.587Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:31:53.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:31:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:31:53.587Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:31:53.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:53 smithi045 conmon[39718]: cluster 2021-07-31T02:31:53.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:53 smithi045 conmon[39718]: 2021-07-31T02:31:52. 2021-07-31T02:31:53.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:53 smithi045 conmon[39718]: 402276+0000 mon.a (mon.0) 2283 : cluster [DBG] osdmap e1322: 8 total, 8 up, 8 in 2021-07-31T02:31:53.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:53 smithi045 conmon[39718]: cluster 2021-07-31 2021-07-31T02:31:53.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:53 smithi045 conmon[39718]: T02:31:52.436548+0000 mgr.x ( 2021-07-31T02:31:53.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:53 smithi045 conmon[39718]: mgr.24542) 1915 : cluster [DBG] pgmap v2508: 73 pgs: 73 active+clean; 197 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 117 MiB/s rd, 92 MiB/s wr, 1.14k op/s 2021-07-31T02:31:53.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:53 smithi045 conmon[33655]: cluster 2021-07-31T02:31:52.402276+0000 mon.a (mon.0) 2283 : cluster [DBG] 2021-07-31T02:31:53.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:53 smithi045 conmon[33655]: osdmap e1322: 8 total, 8 up, 8 in 2021-07-31T02:31:53.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:53 smithi045 conmon[33655]: cluster 2021-07-31T02:31:52.436548+0000 mgr.x (mgr.24542) 1915 : cluster [DBG] pgmap v2508: 73 pgs: 73 active+clean; 197 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 117 MiB/s rd, 92 MiB/s wr, 1.14k op/s 2021-07-31T02:31:54.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:54 smithi154 conmon[34553]: cluster 2021-07-31T02:31:53.408374+0000 mon.a (mon.0) 2284 : cluster [DBG] osdmap e1323: 8 total, 8 up, 8 in 2021-07-31T02:31:54.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:54 smithi045 conmon[39718]: cluster 2021-07-31T02:31:53.408374+0000 mon.a ( 2021-07-31T02:31:54.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:54 smithi045 conmon[39718]: mon.0) 2284 : cluster [DBG] osdmap e1323: 8 total, 8 up, 8 in 2021-07-31T02:31:54.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:54 smithi045 conmon[33655]: cluster 2021-07-31T02:31:53.408374+0000 mon.a (mon.0 2021-07-31T02:31:54.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:54 smithi045 conmon[33655]: ) 2284 : cluster [DBG] osdmap e1323: 8 total, 8 up, 8 in 2021-07-31T02:31:55.428 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 83886080 2021-07-31T02:31:55.450 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 83886079~1 2021-07-31T02:31:55.455 INFO:tasks.workunit.client.0.smithi045.stdout:migration_prepare 2021-07-31T02:31:55.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:55 smithi154 conmon[34553]: cluster 2021-07-31T02:31:54.413802+0000 mon.a (mon.0) 2285 : cluster 2021-07-31T02:31:55.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:55 smithi154 conmon[34553]: [DBG] osdmap e1324: 8 total, 8 up, 8 in 2021-07-31T02:31:55.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:55 smithi154 conmon[34553]: cluster 2021-07-31T02:31:54.437077+0000 mgr.x (mgr.24542) 1916 : cluster [DBG] pgmap v2511: 73 pgs: 73 active+clean; 253 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 20 MiB/s rd, 16 MiB/s wr, 138 op/s 2021-07-31T02:31:55.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:55 smithi045 conmon[39718]: cluster 2021-07-31T02:31:54.413802+0000 mon.a ( 2021-07-31T02:31:55.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:55 smithi045 conmon[39718]: mon.0) 2285 : cluster [DBG] osdmap e1324: 8 total, 8 up, 8 in 2021-07-31T02:31:55.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:55 smithi045 conmon[39718]: cluster 2021-07-31T02:31:54 2021-07-31T02:31:55.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:55 smithi045 conmon[39718]: .437077+0000 mgr.x (mgr.24542) 1916 : cluster [DBG] pgmap v2511: 73 pgs: 73 active+clean; 253 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 20 MiB/s rd, 16 MiB/s wr, 138 op/s 2021-07-31T02:31:55.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:55 smithi045 conmon[33655]: cluster 2021-07-31T02:31:54.413802+0000 mon.a (mon.0) 2285 : cluster [DBG] osdmap e1324: 8 total, 8 up, 8 in 2021-07-31T02:31:55.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:55 smithi045 conmon[33655]: 2021-07-31T02:31:55.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:55 smithi045 conmon[33655]: cluster 2021-07-31T02:31:54.437077+0000 mgr.x (mgr.24542) 1916 : cluster [DBG] pgmap v2511: 73 pgs: 73 active+clean; 253 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 20 MiB/s rd, 16 MiB/s wr, 138 op/s 2021-07-31T02:31:56.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:56 smithi154 conmon[34553]: cluster 2021-07-31T02:31:55.419678+0000 mon.a (mon.0) 2286 : cluster [DBG] osdmap e1325: 8 total, 8 up, 8 in 2021-07-31T02:31:56.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:56 smithi045 conmon[33655]: cluster 2021-07-31T02:31:55.419678+0000 mon.a (mon. 2021-07-31T02:31:56.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:56 smithi045 conmon[33655]: 0) 2286 : cluster [DBG] osdmap e1325: 8 total, 8 up, 8 in 2021-07-31T02:31:56.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:56 smithi045 conmon[39718]: cluster 2021-07-31T02:31:55.419678+0000 mon.a (mon 2021-07-31T02:31:56.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:56 smithi045 conmon[39718]: .0) 2286 : cluster [DBG] osdmap e1325: 8 total, 8 up, 8 in 2021-07-31T02:31:57.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:57 smithi154 conmon[34553]: cluster 2021-07-31T02:31:56.425390+0000 mon.a (mon.0) 2287 : cluster [DBG] osdmap e1326: 8 total, 8 up, 8 in 2021-07-31T02:31:57.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:57 smithi154 conmon[34553]: cluster 2021-07-31T02:31:56.437562+0000 mgr.x (mgr.24542) 1917 : cluster [DBG] pgmap v2514: 73 pgs: 73 active+clean; 253 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 20 MiB/s rd, 16 MiB/s wr, 138 op/s 2021-07-31T02:31:57.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:57 smithi045 conmon[33655]: cluster 2021-07-31T02:31:56.425390+0000 mon.a (mon.0) 2287 : cluster [DBG] osdmap e1326: 8 total, 8 up, 8 in 2021-07-31T02:31:57.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:57 smithi045 conmon[33655]: 2021-07-31T02:31:57.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:57 smithi045 conmon[33655]: cluster 2021-07-31T02:31:56.437562+0000 mgr.x (mgr.24542) 1917 : cluster [DBG] pgmap v2514: 73 pgs: 73 active+clean; 253 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 20 MiB/s rd, 16 MiB/s wr, 138 op/s 2021-07-31T02:31:57.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:57 smithi045 conmon[39718]: cluster 2021-07-31T02:31:56. 2021-07-31T02:31:57.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:57 smithi045 conmon[39718]: 425390+0000 mon.a (mon.0) 2287 : cluster [DBG] osdmap e1326: 8 total, 8 up, 8 in 2021-07-31T02:31:57.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:57 smithi045 conmon[39718]: cluster 2021-07-31T02:31:57.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:57 smithi045 conmon[39718]: 2021-07-31T02: 2021-07-31T02:31:57.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:57 smithi045 conmon[39718]: 31:56.437562+0000 mgr.x 2021-07-31T02:31:57.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:57 smithi045 conmon[39718]: (mgr.24542) 1917 : cluster 2021-07-31T02:31:57.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:57 smithi045 conmon[39718]: [DBG] pgmap v2514: 73 pgs: 73 active+clean; 253 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 20 MiB/s rd, 16 MiB/s wr, 138 op/s 2021-07-31T02:31:58.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:58 smithi154 conmon[34553]: cluster 2021-07-31T02:31:57.430542+0000 mon.a (mon.0) 2288 : cluster [DBG] osdmap e1327: 8 total, 8 up, 8 in 2021-07-31T02:31:58.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:58 smithi045 conmon[39718]: cluster 2021-07-31T02:31:57.430542+0000 2021-07-31T02:31:58.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:58 smithi045 conmon[39718]: mon.a (mon.0) 2288 : cluster [DBG] osdmap e1327: 8 total, 8 up, 8 in 2021-07-31T02:31:58.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:58 smithi045 conmon[33655]: cluster 2021-07-31T02:31:58.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:58 smithi045 conmon[33655]: 2021-07-31T02:31:57.430542 2021-07-31T02:31:58.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:58 smithi045 conmon[33655]: +0000 mon.a (mon.0) 2288 : cluster [DBG] osdmap e1327: 8 total, 8 up, 8 in 2021-07-31T02:31:58.907 INFO:tasks.workunit.client.0.smithi045.stdout:migration_execute 2021-07-31T02:31:59.028 INFO:tasks.workunit.client.0.smithi045.stdout:migration_commit 2021-07-31T02:31:59.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:59 smithi154 conmon[34553]: cluster 2021-07-31T02:31:58.431878+0000 mon.a (mon. 2021-07-31T02:31:59.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:59 smithi154 conmon[34553]: 0) 2289 : cluster [DBG] osdmap e1328: 8 total, 8 up, 8 in 2021-07-31T02:31:59.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:31:59 smithi154 conmon[34553]: cluster 2021-07-31T02:31:58.438508+0000 mgr.x (mgr.24542) 1918 : cluster [DBG] pgmap v2517: 73 pgs: 73 active+clean; 253 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail 2021-07-31T02:31:59.753 INFO:journalctl@ceph.osd.5.smithi154.stdout:Jul 31 02:31:59 smithi154 conmon[42050]: debug 2021-07-31T02:31:59.460+0000 7fc9b105c700 -1 osd.5 pg_epoch: 1329 pg[75.4( v 1328'49 (0'0,1328'49] local-lis/les=1308/1309 n=6 ec=1308/1308 lis/c=1308/1308 les/c/f=1309/1309/0 sis=1308) [5,0,7] r=0 lpr=1308 crt=1325'48 lcod 1325'48 mlcod 1325'48 active+clean+snaptrim trimq=[d~1]] removing snap head 2021-07-31T02:31:59.754 INFO:journalctl@ceph.osd.5.smithi154.stdout:Jul 31 02:31:59 smithi154 conmon[42050]: debug 2021-07-31T02:31:59.462+0000 7fc9b105c700 -1 osd.5 pg_epoch: 1329 pg[75.4( v 1329'51 (0'0,1329'51] local-lis/les=1308/1309 n=4 ec=1308/1308 lis/c=1308/1308 les/c/f=1309/1309/0 sis=1308) [5,0,7] r=0 lpr=1308 luod=1328'49 crt=1329'51 lcod 1325'48 mlcod 1325'48 active+clean+snaptrim trimq=[d~1]] removing snap head 2021-07-31T02:31:59.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:59 smithi045 conmon[39718]: cluster 2021-07-31T02:31:58.431878+0000 mon.a (mon.0) 2289 : cluster [DBG] 2021-07-31T02:31:59.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:59 smithi045 conmon[39718]: osdmap e1328: 8 total, 8 up, 8 in 2021-07-31T02:31:59.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:59 smithi045 conmon[39718]: cluster 2021-07-31T02: 2021-07-31T02:31:59.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:59 smithi045 conmon[39718]: 31:58.438508+0000 mgr.x (mgr.24542) 1918 : cluster [DBG] pgmap v2517: 73 pgs: 73 active+clean; 253 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail 2021-07-31T02:31:59.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:31:59 smithi045 conmon[39718]: 2021-07-31T02:31:59.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:59 smithi045 conmon[33655]: cluster 2021-07-31T02:31:58.431878 2021-07-31T02:31:59.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:59 smithi045 conmon[33655]: +0000 mon.a (mon.0) 2289 : cluster [DBG] osdmap e1328: 8 total, 8 up, 8 in 2021-07-31T02:31:59.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:31:59 smithi045 conmon[33655]: cluster 2021-07-31T02:31:58.438508+0000 mgr.x (mgr.24542) 1918 : cluster [DBG] pgmap v2517: 73 pgs: 73 active+clean; 253 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail 2021-07-31T02:32:00.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:00 smithi154 conmon[34553]: cluster 2021-07-31T02:31:59.441852+0000 mon.a (mon.0) 2021-07-31T02:32:00.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:00 smithi154 conmon[34553]: 2290 : cluster [DBG] osdmap e1329: 8 total, 8 up, 8 in 2021-07-31T02:32:00.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:00 smithi045 conmon[33655]: cluster 2021-07-31T02:31:59.441852+0000 mon.a (mon. 2021-07-31T02:32:00.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:00 smithi045 conmon[33655]: 0) 2290 : cluster [DBG] osdmap e1329: 8 total, 8 up, 8 in 2021-07-31T02:32:00.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:00 smithi045 conmon[39718]: cluster 2021-07-31T02:31:59 2021-07-31T02:32:00.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:00 smithi045 conmon[39718]: .441852+0000 mon.a (mon.0) 2290 : cluster [DBG] osdmap e1329: 8 total, 8 up, 8 in 2021-07-31T02:32:01.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:01 smithi154 conmon[34553]: cluster 2021-07-31T02:32:00.439025+0000 mgr.x (mgr.24542) 2021-07-31T02:32:01.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:01 smithi154 conmon[34553]: 1919 : cluster [DBG] pgmap v2519: 73 pgs: 73 active+clean; 257 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 36 MiB/s rd, 9.2 KiB/s wr, 91 op/s 2021-07-31T02:32:01.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:01 smithi154 conmon[34553]: cluster 2021-07-31T02:32:00.443844+0000 mon.a (mon.0) 2291 : cluster [DBG] osdmap e1330: 8 total, 8 up, 8 in 2021-07-31T02:32:01.775 INFO:journalctl@ceph.osd.3.smithi045.stdout:Jul 31 02:32:01 smithi045 conmon[54013]: debug 2021-07-31T02:32:01.521+0000 7f64fb8cb700 -1 osd.3 pg_epoch: 1331 pg[75.5( v 1331'28 (0'0,1331'28] local-lis/les=1308/1309 n=3 ec=1308/1308 lis/c=1308/1308 les/c/f=1309/1309/0 sis=1308) [3,4,1] r=0 lpr=1308 crt=1331'28 lcod 1331'27 mlcod 1331'27 active+clean+snaptrim trimq=[9~1] ps=[b~1,d~1]] removing snap head 2021-07-31T02:32:01.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:01 smithi045 conmon[39718]: cluster 2021-07-31T02:32:00.439025+0000 mgr.x (mgr.24542) 1919 : cluster 2021-07-31T02:32:01.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:01 smithi045 conmon[39718]: [DBG] pgmap v2519: 73 pgs: 73 active+clean; 257 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 36 MiB/s rd, 9.2 KiB/s wr, 91 op/s 2021-07-31T02:32:01.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:01 smithi045 conmon[39718]: cluster 2021-07-31T02:32:00.443844+0000 mon.a (mon.0) 2291 : cluster [DBG] osdmap e1330: 8 total, 8 up, 8 in 2021-07-31T02:32:01.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:01 smithi045 conmon[33655]: cluster 2021-07-31T02:32:00.439025+0000 mgr.x (mgr 2021-07-31T02:32:01.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:01 smithi045 conmon[33655]: .24542) 1919 : cluster [DBG] pgmap v2519: 73 pgs: 73 active+clean; 257 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 36 MiB/s rd, 9.2 KiB/s wr, 91 op/s 2021-07-31T02:32:01.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:01 smithi045 conmon[33655]: cluster 2021-07-31T02:32:00.443844+0000 mon.a (mon.0) 2291 : cluster [DBG] osdmap e1330: 8 total, 8 up, 8 in 2021-07-31T02:32:01.806 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMigration.Snaps (12127 ms) 2021-07-31T02:32:01.807 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMigration.SnapsOtherPool 2021-07-31T02:32:01.827 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 83886080 2021-07-31T02:32:01.832 INFO:tasks.workunit.client.0.smithi045.stdout:write: 1 0~8388609 2021-07-31T02:32:01.880 INFO:tasks.workunit.client.0.smithi045.stdout:snap_create: snap1 2021-07-31T02:32:02.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:02 smithi154 conmon[34553]: cluster 2021-07-31T02:32:01.449804+0000 2021-07-31T02:32:02.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:02 smithi154 conmon[34553]: mon.a (mon.0) 2292 : cluster [DBG] osdmap e1331: 8 total, 8 up, 8 in 2021-07-31T02:32:02.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:02 smithi045 conmon[39718]: cluster 2021-07-31T02:32:01.449804+0000 2021-07-31T02:32:02.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:02 smithi045 conmon[39718]: mon.a (mon.0) 2292 : cluster [DBG] osdmap e1331: 8 total, 8 up, 8 in 2021-07-31T02:32:02.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:02 smithi045 conmon[33655]: cluster 2021-07-31T02:32:01.449804+0000 mon.a (mon.0) 2292 : cluster [DBG] osdmap e1331: 8 total, 8 up, 8 in 2021-07-31T02:32:03.474 INFO:tasks.workunit.client.0.smithi045.stdout:write: 1 8388609~8388609 2021-07-31T02:32:03.483 INFO:tasks.workunit.client.0.smithi045.stdout:write: X 0~1000 2021-07-31T02:32:03.484 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 1010~1000 2021-07-31T02:32:03.484 INFO:tasks.workunit.client.0.smithi045.stdout:snap_create: snap2 2021-07-31T02:32:03.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:03 smithi154 conmon[34553]: cluster 2021-07-31T02:32:02.439468+0000 mgr.x (mgr.24542) 1920 : cluster [DBG] pgmap v2522: 73 pgs: 73 active+clean; 257 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 36 MiB/s rd, 9.2 KiB/s wr, 91 op/s 2021-07-31T02:32:03.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:03 smithi154 conmon[34553]: cluster 2021-07-31T02:32:02.456296+0000 mon.a (mon.0) 2293 : cluster [DBG] osdmap e1332: 8 total, 8 up, 8 in 2021-07-31T02:32:03.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:03 smithi045 conmon[33655]: cluster 2021-07-31T02:32:02.439468+0000 mgr.x (mgr.24542) 1920 : cluster [DBG] 2021-07-31T02:32:03.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:03 smithi045 conmon[33655]: pgmap v2522: 73 pgs: 73 active+clean; 257 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 36 MiB/s rd, 9.2 KiB/s wr, 91 op/s 2021-07-31T02:32:03.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:03 smithi045 conmon[33655]: cluster 2021-07-31T02:32:02.456296+0000 mon.a (mon.0) 2293 : cluster [DBG] osdmap e1332: 8 total, 8 up, 8 in 2021-07-31T02:32:03.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:03 smithi045 conmon[39718]: cluster 2021-07-31T02:32:02.439468+0000 mgr.x (mgr.24542) 1920 : cluster [DBG] pgmap v2522: 73 pgs: 73 active+clean; 257 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 36 MiB/s rd, 9.2 KiB/s wr, 91 op/s 2021-07-31T02:32:03.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:03 smithi045 conmon[39718]: cluster 2021-07-31T02:32:02.456296+0000 mon.a (mon.0) 2293 : cluster [DBG] osdmap e1332: 8 total, 8 up, 8 in 2021-07-31T02:32:03.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:32:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:32:03.587Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:32:03.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:32:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:32:03.587Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:32:03.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:32:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:32:03.587Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:32:04.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:04 smithi154 conmon[34553]: cluster 2021-07-31T02:32:03.468118+0000 mon.a (mon.0) 2294 : 2021-07-31T02:32:04.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:04 smithi154 conmon[34553]: cluster [DBG] osdmap e1333: 8 total, 8 up, 8 in 2021-07-31T02:32:04.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:04 smithi045 conmon[33655]: cluster 2021-07-31T02:32:03.468118+0000 mon.a (mon 2021-07-31T02:32:04.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:04 smithi045 conmon[33655]: .0) 2294 : cluster [DBG] osdmap e1333: 8 total, 8 up, 8 in 2021-07-31T02:32:04.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:04 smithi045 conmon[39718]: cluster 2021-07-31T02:32:03.468118+0000 mon.a (mon.0) 2021-07-31T02:32:04.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:04 smithi045 conmon[39718]: 2294 : cluster [DBG] osdmap e1333: 8 total, 8 up, 8 in 2021-07-31T02:32:05.482 INFO:tasks.workunit.client.0.smithi045.stdout:write: X 8388609~1000 2021-07-31T02:32:05.487 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 16777228~1000 2021-07-31T02:32:05.487 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 167772160 2021-07-31T02:32:05.494 INFO:tasks.workunit.client.0.smithi045.stdout:write: 2 83886079~8388609 2021-07-31T02:32:05.503 INFO:tasks.workunit.client.0.smithi045.stdout:snap_create: snap3 2021-07-31T02:32:05.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:05 smithi154 conmon[34553]: cluster 2021-07-31T02:32:04.440252+0000 mgr.x (mgr.24542) 1921 : cluster [DBG] pgmap v2525: 73 pgs: 73 active+clean; 285 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 34 MiB/s rd, 14 MiB/s wr, 164 op/s 2021-07-31T02:32:05.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:05 smithi154 conmon[34553]: cluster 2021-07-31T02:32:04.474473+0000 mon.a (mon.0) 2295 : cluster [DBG] osdmap e1334: 8 total, 8 up, 8 in 2021-07-31T02:32:05.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:05 smithi045 conmon[33655]: cluster 2021-07-31T02:32:04.440252+0000 mgr.x (mgr.24542 2021-07-31T02:32:05.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:05 smithi045 conmon[33655]: ) 1921 : cluster [DBG] pgmap v2525: 73 pgs: 73 active+clean; 285 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 34 MiB/s rd, 14 MiB/s wr, 164 op/s 2021-07-31T02:32:05.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:05 smithi045 conmon[33655]: cluster 2021-07-31T02:32:04.474473+0000 mon.a (mon.0) 2295 : cluster [DBG] osdmap e1334: 8 total, 8 up, 8 in 2021-07-31T02:32:05.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:05 smithi045 conmon[39718]: cluster 2021-07-31T02:32:04.440252+0000 mgr.x ( 2021-07-31T02:32:05.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:05 smithi045 conmon[39718]: mgr.24542) 1921 : cluster [DBG] pgmap v2525: 73 pgs: 73 active+clean; 285 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 34 MiB/s rd, 14 MiB/s wr, 164 op/s 2021-07-31T02:32:05.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:05 smithi045 conmon[39718]: cluster 2021-07-31T02:32: 2021-07-31T02:32:05.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:05 smithi045 conmon[39718]: 04.474473+0000 mon.a (mon.0) 2295 : cluster [DBG] osdmap e1334: 8 total, 8 up, 8 in 2021-07-31T02:32:06.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:06 smithi154 conmon[34553]: cluster 2021-07-31T02:32:05.476037+0000 mon.a (mon.0) 2296 : cluster 2021-07-31T02:32:06.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:06 smithi154 conmon[34553]: [DBG] osdmap e1335: 8 total, 8 up, 8 in 2021-07-31T02:32:06.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:06 smithi045 conmon[39718]: cluster 2021-07-31T02:32:05.476037+0000 mon.a (mon.0) 2296 : cluster [DBG] osdmap e1335: 8 total, 8 up, 8 in 2021-07-31T02:32:06.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:06 smithi045 conmon[33655]: cluster 2021-07-31T02:32:05.476037+0000 mon.a (mon.0) 2296 : cluster [DBG] 2021-07-31T02:32:06.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:06 smithi045 conmon[33655]: osdmap e1335: 8 total, 8 up, 8 in 2021-07-31T02:32:07.335 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 83886080 2021-07-31T02:32:07.364 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 83886079~1 2021-07-31T02:32:07.367 INFO:tasks.workunit.client.0.smithi045.stdout:write: 1 0~8388609 2021-07-31T02:32:07.371 INFO:tasks.workunit.client.0.smithi045.stdout:write: 1 16777218~8388609 2021-07-31T02:32:07.426 INFO:tasks.workunit.client.0.smithi045.stdout:migration_prepare 2021-07-31T02:32:07.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:07 smithi154 conmon[34553]: cluster 2021-07-31T02:32:06.440676+0000 mgr.x (mgr.24542) 1922 : cluster [DBG] 2021-07-31T02:32:07.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:07 smithi154 conmon[34553]: pgmap v2528: 73 pgs: 73 active+clean; 285 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 34 MiB/s rd, 14 MiB/s wr, 164 op/s 2021-07-31T02:32:07.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:07 smithi154 conmon[34553]: cluster 2021-07-31T02:32:06.486874+0000 mon.a (mon.0) 2297 : cluster [DBG] osdmap e1336: 8 total, 8 up, 8 in 2021-07-31T02:32:07.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:07 smithi154 conmon[34553]: cluster 2021-07-31T02:32: 2021-07-31T02:32:07.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:07 smithi154 conmon[34553]: 07.317978+0000 mon.a (mon.0) 2298 : cluster [DBG] osdmap e1337: 8 total, 8 up, 8 in 2021-07-31T02:32:07.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:07 smithi045 conmon[39718]: cluster 2021-07-31T02:32: 2021-07-31T02:32:07.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:07 smithi045 conmon[39718]: 06.440676+0000 mgr.x (mgr.24542) 1922 : cluster [DBG] pgmap v2528: 73 pgs: 73 active+clean; 285 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 34 MiB/s rd, 14 MiB/s wr, 164 op/s 2021-07-31T02:32:07.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:07 smithi045 conmon[39718]: cluster 2021-07-31T02:32:07.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:07 smithi045 conmon[39718]: 2021-07-31 2021-07-31T02:32:07.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:07 smithi045 conmon[39718]: T02:32:06.486874+0000 mon.a ( 2021-07-31T02:32:07.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:07 smithi045 conmon[39718]: mon.0) 2297 : cluster [DBG] osdmap e1336: 8 total, 8 up, 8 in 2021-07-31T02:32:07.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:07 smithi045 conmon[39718]: 2021-07-31T02:32:07.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:07 smithi045 conmon[39718]: cluster 2021-07-31T02:32:07.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:07 smithi045 conmon[39718]: 2021-07- 2021-07-31T02:32:07.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:07 smithi045 conmon[39718]: 31T02:32:07.317978+0000 2021-07-31T02:32:07.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:07 smithi045 conmon[39718]: mon.a (mon.0) 2298 : cluster 2021-07-31T02:32:07.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:07 smithi045 conmon[39718]: [DBG] osdmap e1337: 8 total, 8 up, 8 in 2021-07-31T02:32:07.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:07 smithi045 conmon[33655]: cluster 2021-07-31T02:32:06.440676+0000 2021-07-31T02:32:07.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:07 smithi045 conmon[33655]: mgr.x (mgr.24542) 1922 : cluster [DBG] pgmap v2528: 73 pgs: 73 active+clean; 285 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 34 MiB/s rd, 14 MiB/s wr, 164 op/s 2021-07-31T02:32:07.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:07 smithi045 conmon[33655]: cluster 2021-07-31T02:32:06.486874+0000 mon.a (mon.0) 2297 : cluster [DBG] osdmap e1336: 8 total, 8 up, 8 in 2021-07-31T02:32:07.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:07 smithi045 conmon[33655]: cluster 2021-07-31T02:32 2021-07-31T02:32:07.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:07 smithi045 conmon[33655]: :07.317978+0000 mon.a ( 2021-07-31T02:32:07.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:07 smithi045 conmon[33655]: mon.0) 2298 : cluster [DBG] 2021-07-31T02:32:07.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:07 smithi045 conmon[33655]: osdmap e1337: 8 total, 8 up, 8 in 2021-07-31T02:32:09.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:09 smithi154 conmon[34553]: cluster 2021-07-31T02:32:08.324485+0000 mon.a (mon.0) 2299 2021-07-31T02:32:09.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:09 smithi154 conmon[34553]: : cluster [DBG] osdmap e1338: 8 total, 8 up, 8 in 2021-07-31T02:32:09.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:09 smithi045 conmon[39718]: cluster 2021-07-31T02:32:08.324485+0000 2021-07-31T02:32:09.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:09 smithi045 conmon[39718]: mon.a (mon.0) 2299 : cluster [DBG] osdmap e1338: 8 total, 8 up, 8 in 2021-07-31T02:32:09.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:09 smithi045 conmon[33655]: cluster 2021-07-31T02:32:08.324485+0000 mon.a (mon.0) 2299 : cluster [DBG] 2021-07-31T02:32:09.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:09 smithi045 conmon[33655]: osdmap e1338: 8 total, 8 up, 8 in 2021-07-31T02:32:10.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:10 smithi154 conmon[34553]: cluster 2021-07-31T02:32:08.441198+0000 mgr.x (mgr.24542) 1923 : cluster 2021-07-31T02:32:10.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:10 smithi154 conmon[34553]: [DBG] pgmap v2532: 73 pgs: 73 active+clean; 285 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail 2021-07-31T02:32:10.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:10 smithi154 conmon[34553]: cluster 2021-07-31T02:32:09.329945+0000 mon.a (mon.0) 2300 : cluster [DBG] osdmap e1339: 8 total, 8 up, 8 in 2021-07-31T02:32:10.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:10 smithi045 conmon[39718]: cluster 2021-07-31T02:32:10.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:10 smithi045 conmon[39718]: 2021-07-31T02:32:08. 2021-07-31T02:32:10.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:10 smithi045 conmon[39718]: 441198+0000 mgr.x (mgr.24542) 1923 : cluster [DBG] pgmap v2532: 73 pgs: 73 active+clean; 285 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail 2021-07-31T02:32:10.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:10 smithi045 conmon[39718]: cluster 2021-07-31T02:32:09.329945+0000 2021-07-31T02:32:10.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:10 smithi045 conmon[39718]: mon.a (mon.0) 2300 : cluster [DBG] osdmap e1339: 8 total, 8 up, 8 in 2021-07-31T02:32:10.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:10 smithi045 conmon[33655]: cluster 2021-07-31T02:32:10.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:10 smithi045 conmon[33655]: 2021-07-31T02:32:10.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:10 smithi045 conmon[33655]: 2021-07-31T02:32:08.441198+0000 mgr.x (mgr. 2021-07-31T02:32:10.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:10 smithi045 conmon[33655]: 24542) 1923 : cluster [DBG] pgmap v2532: 73 pgs: 73 active+clean; 285 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail 2021-07-31T02:32:10.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:10 smithi045 conmon[33655]: cluster 2021-07-31T02:32:10.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:10 smithi045 conmon[33655]: 2021-07-31T02:32:09.329945+0000 mon.a (mon.0 2021-07-31T02:32:10.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:10 smithi045 conmon[33655]: ) 2300 : cluster [DBG] osdmap e1339: 8 total, 8 up, 8 in 2021-07-31T02:32:10.779 INFO:tasks.workunit.client.0.smithi045.stdout:migration_execute 2021-07-31T02:32:10.897 INFO:tasks.workunit.client.0.smithi045.stdout:migration_commit 2021-07-31T02:32:11.637 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:11 smithi154 conmon[34553]: cluster 2021-07-31T02:32:10.331247+0000 mon.a (mon.0) 2301 : cluster 2021-07-31T02:32:11.637 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:11 smithi154 conmon[34553]: [DBG] osdmap e1340: 8 total, 8 up, 8 in 2021-07-31T02:32:11.775 INFO:journalctl@ceph.osd.3.smithi045.stdout:Jul 31 02:32:11 smithi045 conmon[54013]: debug 2021-07-31T02:32:11.361+0000 7f64f90c6700 -1 osd.3 pg_epoch: 1341 pg[75.12( v 1337'57 (0'0,1337'57] local-lis/les=1308/1309 n=7 ec=1308/1308 lis/c=1308/1308 les/c/f=1309/1309/0 sis=1308) [3,2,0] r=0 lpr=1308 crt=1337'55 lcod 1337'56 mlcod 1337'56 active+clean+snaptrim trimq=[19~1]] removing snap head 2021-07-31T02:32:11.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:11 smithi045 conmon[33655]: cluster 2021-07-31T02:32:10.331247+0000 mon.a (mon.0 2021-07-31T02:32:11.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:11 smithi045 conmon[33655]: ) 2301 : cluster [DBG] osdmap e1340: 8 total, 8 up, 8 in 2021-07-31T02:32:11.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:11 smithi045 conmon[39718]: cluster 2021-07-31T02:32:10.331247+0000 mon.a (mon.0) 2301 2021-07-31T02:32:11.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:11 smithi045 conmon[39718]: : cluster [DBG] osdmap e1340: 8 total, 8 up, 8 in 2021-07-31T02:32:12.002 INFO:journalctl@ceph.osd.5.smithi154.stdout:Jul 31 02:32:11 smithi154 conmon[42050]: debug 2021-07-31T02:32:11.636+0000 7fc9b4062700 -1 osd.5 pg_epoch: 1341 pg[75.16( v 1337'30 (0'0,1337'30] local-lis/les=1308/1309 n=6 ec=1308/1308 lis/c=1308/1308 les/c/f=1309/1309/0 sis=1308) [5,1,6] r=0 lpr=1308 crt=1337'30 lcod 1335'28 mlcod 1335'28 active+clean+snaptrim trimq=[19~1]] removing snap head 2021-07-31T02:32:12.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:12 smithi154 conmon[34553]: cluster 2021-07-31T02:32:10.441644+0000 mgr.x (mgr.24542) 1924 : cluster [DBG] pgmap v2535: 73 pgs: 73 active+clean; 341 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 19 KiB/s rd, 12 MiB/s wr, 33 op/s 2021-07-31T02:32:12.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:12 smithi154 conmon[34553]: cluster 2021-07-31T02:32:11.344238+0000 mon.a (mon.0) 2302 : cluster [DBG] osdmap e1341: 8 total, 8 up, 8 in 2021-07-31T02:32:12.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:12 smithi045 conmon[33655]: cluster 2021-07-31T02:32:10.441644+0000 mgr.x (mgr.24542) 2021-07-31T02:32:12.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:12 smithi045 conmon[33655]: 1924 : cluster [DBG] pgmap v2535: 73 pgs: 73 active+clean; 341 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 19 KiB/s rd, 12 MiB/s wr, 33 op/s 2021-07-31T02:32:12.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:12 smithi045 conmon[33655]: cluster 2021-07-31T02:32:11.344238+0000 mon.a (mon.0) 2302 : cluster [DBG] osdmap e1341: 8 total, 8 up, 8 in 2021-07-31T02:32:12.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:12 smithi045 conmon[39718]: cluster 2021-07-31T02:32: 2021-07-31T02:32:12.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:12 smithi045 conmon[39718]: 10.441644+0000 mgr.x (mgr.24542) 1924 : cluster [DBG] pgmap v2535: 73 pgs: 73 active+clean; 341 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 19 KiB/s rd, 12 MiB/s wr, 33 op/s 2021-07-31T02:32:12.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:12 smithi045 conmon[39718]: cluster 2021- 2021-07-31T02:32:12.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:12 smithi045 conmon[39718]: 07-31T02:32:11.344238 2021-07-31T02:32:12.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:12 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2021-07-31T02:32:12.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:12 smithi045 conmon[39718]: 2302 : cluster [DBG] osdmap e1341: 8 total, 8 up, 8 in 2021-07-31T02:32:13.729 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMigration.SnapsOtherPool (11922 ms) 2021-07-31T02:32:13.729 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMigration.SnapsDataPool 2021-07-31T02:32:13.746 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 83886080 2021-07-31T02:32:13.749 INFO:tasks.workunit.client.0.smithi045.stdout:write: 1 0~8388609 2021-07-31T02:32:13.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:13 smithi154 conmon[34553]: cluster 2021-07-31T02:32:12.350450+0000 mon.a (mon.0) 2303 : cluster [DBG] osdmap e1342: 8 total, 8 up, 8 in 2021-07-31T02:32:13.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:13 smithi154 conmon[34553]: 2021-07-31T02:32:13.752 INFO:journalctl@ceph.osd.5.smithi154.stdout:Jul 31 02:32:13 smithi154 conmon[42050]: debug 2021-07-31T02:32:13.392+0000 7fc9af859700 -1 osd.5 pg_epoch: 1343 pg[75.f( v 1343'84 (0'0,1343'84] local-lis/les=1308/1309 n=8 ec=1308/1308 lis/c=1308/1308 les/c/f=1309/1309/0 sis=1308) [5,2,4] r=0 lpr=1308 crt=1343'84 lcod 1342'83 mlcod 1342'83 active+clean+snaptrim trimq=[15~1] ps=[17~1,19~1]] removing snap head 2021-07-31T02:32:13.775 INFO:journalctl@ceph.osd.3.smithi045.stdout:Jul 31 02:32:13 smithi045 conmon[54013]: debug 2021-07-31T02:32:13.392+0000 7f64f78c3700 -1 osd.3 pg_epoch: 1343 pg[75.5( v 1343'40 (0'0,1343'40] local-lis/les=1308/1309 n=3 ec=1308/1308 lis/c=1308/1308 les/c/f=1309/1309/0 sis=1308) [3,4,1] r=0 lpr=1308 luod=1342'38 crt=1343'40 lcod 1341'36 mlcod 1341'36 active+clean+snaptrim trimq=[15~1] ps=[17~1,19~1]] removing snap head 2021-07-31T02:32:13.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:13 smithi045 conmon[33655]: cluster 2021-07-31T02:32:12.350450+0000 mon.a (mon.0) 2303 : cluster 2021-07-31T02:32:13.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:13 smithi045 conmon[33655]: [DBG] osdmap e1342: 8 total, 8 up, 8 in 2021-07-31T02:32:13.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:13 smithi045 conmon[39718]: cluster 2021-07-31T02:32:12 2021-07-31T02:32:13.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:13 smithi045 conmon[39718]: .350450+0000 mon.a (mon.0) 2303 : cluster [DBG] osdmap e1342: 8 total, 8 up, 8 in 2021-07-31T02:32:13.779 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:32:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:32:13.587Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:32:13.779 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:32:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:32:13.587Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:32:13.779 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:32:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:32:13.587Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:32:13.797 INFO:tasks.workunit.client.0.smithi045.stdout:snap_create: snap1 2021-07-31T02:32:14.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:14 smithi154 conmon[34553]: cluster 2021-07-31T02:32:12.442060+0000 mgr.x (mgr.24542) 1925 2021-07-31T02:32:14.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:14 smithi154 conmon[34553]: : cluster [DBG] pgmap v2538: 73 pgs: 73 active+clean; 341 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 18 KiB/s rd, 12 MiB/s wr, 32 op/s 2021-07-31T02:32:14.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:14 smithi154 conmon[34553]: cluster 2021-07-31T02:32:13.356892+0000 mon.a (mon.0) 2304 : cluster [DBG] osdmap e1343: 8 total, 8 up, 8 in 2021-07-31T02:32:14.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:14 smithi045 conmon[39718]: cluster 2021-07-31T02:32 2021-07-31T02:32:14.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:14 smithi045 conmon[39718]: :12.442060+0000 mgr.x (mgr.24542) 1925 : cluster [DBG] pgmap v2538: 73 pgs: 73 active+clean; 341 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 18 KiB/s rd, 12 MiB/s wr, 32 op/s 2021-07-31T02:32:14.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:14 smithi045 conmon[39718]: cluster 2021-07- 2021-07-31T02:32:14.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:14 smithi045 conmon[39718]: 31T02:32:13.356892 2021-07-31T02:32:14.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:14 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2304 2021-07-31T02:32:14.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:14 smithi045 conmon[39718]: : cluster [DBG] osdmap e1343: 8 total, 8 up, 8 in 2021-07-31T02:32:14.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:14 smithi045 conmon[33655]: cluster 2021-07-31T02:32:12.442060 2021-07-31T02:32:14.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:14 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 1925 : cluster [DBG] pgmap v2538: 73 pgs: 73 active+clean; 341 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 18 KiB/s rd, 12 MiB/s wr, 32 op/s 2021-07-31T02:32:14.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:14 smithi045 conmon[33655]: cluster 2021-07-31T02:32:14.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:14 smithi045 conmon[33655]: 2021-07-31T02:32:13.356892+0000 mon.a (mon. 2021-07-31T02:32:14.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:14 smithi045 conmon[33655]: 0) 2304 : cluster [DBG] osdmap e1343: 8 total, 8 up, 8 in 2021-07-31T02:32:15.374 INFO:tasks.workunit.client.0.smithi045.stdout:write: 1 8388609~8388609 2021-07-31T02:32:15.388 INFO:tasks.workunit.client.0.smithi045.stdout:write: X 0~1000 2021-07-31T02:32:15.388 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 1010~1000 2021-07-31T02:32:15.389 INFO:tasks.workunit.client.0.smithi045.stdout:snap_create: snap2 2021-07-31T02:32:15.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:15 smithi154 conmon[34553]: cluster 2021-07-31T02:32:14.362559+0000 mon.a (mon.0) 2305 : cluster 2021-07-31T02:32:15.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:15 smithi154 conmon[34553]: [DBG] osdmap e1344: 8 total, 8 up, 8 in 2021-07-31T02:32:15.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:15 smithi045 conmon[39718]: cluster 2021-07-31T02:32:14.362559 2021-07-31T02:32:15.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:15 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2305 : cluster [DBG] osdmap e1344: 8 total, 8 up, 8 in 2021-07-31T02:32:15.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:15 smithi045 conmon[33655]: cluster 2021-07-31T02:32:14.362559+0000 mon.a (mon.0) 2305 : cluster 2021-07-31T02:32:15.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:15 smithi045 conmon[33655]: [DBG] osdmap e1344: 8 total, 8 up, 8 in 2021-07-31T02:32:16.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:16 smithi154 conmon[34553]: cluster 2021-07-31T02:32:14.442604+0000 mgr.x (mgr.24542) 1926 : cluster [DBG] pgmap v2541: 73 pgs: 73 active+clean; 357 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 82 MiB/s rd, 14 MiB/s wr, 209 op/s 2021-07-31T02:32:16.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:16 smithi154 conmon[34553]: cluster 2021-07-31T02:32:15.368921+0000 mon.a (mon.0) 2306 : cluster [DBG] osdmap e1345: 8 total, 8 up, 8 in 2021-07-31T02:32:16.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:16 smithi045 conmon[33655]: cluster 2021-07-31T02:32:14.442604+0000 2021-07-31T02:32:16.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:16 smithi045 conmon[33655]: mgr.x (mgr.24542) 1926 : cluster [DBG] pgmap v2541: 73 pgs: 73 active+clean; 357 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 82 MiB/s rd, 14 MiB/s wr, 209 op/s 2021-07-31T02:32:16.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:16 smithi045 conmon[33655]: cluster 2021-07-31T02:32:15.368921+0000 mon.a (mon.0) 2306 : cluster 2021-07-31T02:32:16.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:16 smithi045 conmon[33655]: [DBG] osdmap e1345: 8 total, 8 up, 8 in 2021-07-31T02:32:16.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:16 smithi045 conmon[39718]: cluster 2021-07-31T02:32:14.442604+0000 mgr.x (mgr 2021-07-31T02:32:16.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:16 smithi045 conmon[39718]: .24542) 1926 : cluster [DBG] pgmap v2541: 73 pgs: 73 active+clean; 357 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 82 MiB/s rd, 14 MiB/s wr, 209 op/s 2021-07-31T02:32:16.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:16 smithi045 conmon[39718]: cluster 2021-07-31T02:32 2021-07-31T02:32:16.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:16 smithi045 conmon[39718]: :15.368921+0000 mon.a (mon.0) 2306 : cluster [DBG] osdmap e1345: 8 total, 8 up, 8 in 2021-07-31T02:32:17.331 INFO:tasks.workunit.client.0.smithi045.stdout:write: X 8388609~1000 2021-07-31T02:32:17.337 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 16777228~1000 2021-07-31T02:32:17.338 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 167772160 2021-07-31T02:32:17.351 INFO:tasks.workunit.client.0.smithi045.stdout:write: 2 83886079~8388609 2021-07-31T02:32:17.359 INFO:tasks.workunit.client.0.smithi045.stdout:snap_create: snap3 2021-07-31T02:32:17.697 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:32:17.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:17 smithi154 conmon[34553]: cluster 2021-07-31T02:32:16.374577+0000 mon.a (mon.0) 2307 : cluster [DBG] osdmap e1346: 8 total, 8 up, 8 in 2021-07-31T02:32:17.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:17 smithi154 conmon[34553]: cluster 2021-07-31T02:32:17.319577+0000 mon.a (mon.0) 2308 : cluster [DBG] osdmap e1347: 8 total, 8 up, 8 in 2021-07-31T02:32:17.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:17 smithi045 conmon[33655]: cluster 2021-07-31T02:32:16.374577+0000 mon.a (mon 2021-07-31T02:32:17.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:17 smithi045 conmon[33655]: .0) 2307 : cluster [DBG] osdmap e1346: 8 total, 8 up, 8 in 2021-07-31T02:32:17.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:17 smithi045 conmon[33655]: cluster 2021-07-31T02:32:17.319577+0000 mon.a (mon.0) 2308 : cluster [DBG] osdmap e1347: 8 total, 8 up, 8 in 2021-07-31T02:32:17.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:17 smithi045 conmon[39718]: cluster 2021-07-31T02: 2021-07-31T02:32:17.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:17 smithi045 conmon[39718]: 32:16.374577+0000 mon.a (mon.0) 2307 : cluster [DBG] osdmap e1346: 8 total, 8 up, 8 in 2021-07-31T02:32:17.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:17 smithi045 conmon[39718]: cluster 2021 2021-07-31T02:32:17.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:17 smithi045 conmon[39718]: -07-31T02:32:17 2021-07-31T02:32:17.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:17 smithi045 conmon[39718]: .319577+0000 mon.a (mon. 2021-07-31T02:32:17.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:17 smithi045 conmon[39718]: 0) 2308 : cluster [DBG] 2021-07-31T02:32:17.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:17 smithi045 conmon[39718]: osdmap e1347: 8 total, 8 up, 8 in 2021-07-31T02:32:17.974 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:32:17.974 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 47m ago 47m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:32:17.975 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 44m ago 47m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:32:17.975 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 44m ago 51m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:32:17.975 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 47m ago 53m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:32:17.975 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 47m ago 53m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:32:17.975 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 44m ago 51m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:32:17.976 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 47m ago 51m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:32:17.976 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 47m ago 48m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:32:17.976 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 44m ago 48m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:32:17.976 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 47m ago 50m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:32:17.977 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 47m ago 50m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:32:17.977 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 47m ago 50m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:32:17.977 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 47m ago 49m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:32:17.977 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 44m ago 49m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:32:17.977 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 44m ago 49m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:32:17.978 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 44m ago 48m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:32:17.978 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 44m ago 48m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:32:17.978 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 44m ago 48m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:32:18.310 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:32:18.310 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:32:18.310 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:32:18.312 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:32:18.312 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:32:18.312 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:32:18.312 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:32:18.313 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:32:18.313 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:32:18.313 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:32:18.313 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:32:18.314 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:32:18.314 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:32:18.314 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:32:18.314 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:32:18.314 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:32:18.315 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:32:18.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:18 smithi154 conmon[34553]: cluster 2021-07-31T02:32:18.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:18 smithi154 conmon[34553]: 2021-07-31T02:32:16.443088+0000 mgr.x 2021-07-31T02:32:18.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:18 smithi154 conmon[34553]: (mgr.24542) 1927 : cluster [DBG] pgmap v2544: 73 pgs: 73 active+clean; 357 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 82 MiB/s rd, 14 MiB/s wr, 209 op/s 2021-07-31T02:32:18.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:18 smithi154 conmon[34553]: audit 2021-07-31T02:32:18.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:18 smithi154 conmon[34553]: 2021-07-31T02:32:17.685422+0000 mgr.x (mgr.24542) 1928 : audit [DBG] from='client.17388 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:32:18.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:18 smithi154 conmon[34553]: 2021-07-31T02:32:18.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:18 smithi154 conmon[34553]: audit 2021-07-31T02:32: 2021-07-31T02:32:18.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:18 smithi154 conmon[34553]: 17.966425+0000 mgr.x (mgr.24542) 1929 : 2021-07-31T02:32:18.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:18 smithi154 conmon[34553]: audit [DBG] from='client.17394 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:32:18.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:18 smithi154 conmon[34553]: audit 2021-07-31 2021-07-31T02:32:18.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:18 smithi154 conmon[34553]: T02:32:18.310445+0000 mon.c (mon 2021-07-31T02:32:18.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:18 smithi154 conmon[34553]: .1) 117 : audit [DBG] from='client.? 172.21.15.45:0/3647548701' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:32:18.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:18 smithi154 conmon[34553]: cluster 2021-07-31T02:32:18.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:18 smithi154 conmon[34553]: 2021-07-31T02:32:18.388641 2021-07-31T02:32:18.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:18 smithi154 conmon[34553]: +0000 mon.a (mon.0) 2309 : cluster [DBG] 2021-07-31T02:32:18.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:18 smithi154 conmon[34553]: osdmap e1348: 8 total, 8 up, 8 in 2021-07-31T02:32:18.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:18 smithi045 conmon[33655]: cluster 2021-07-31T02:32:16. 2021-07-31T02:32:18.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:18 smithi045 conmon[33655]: 443088+0000 mgr.x (mgr.24542) 1927 : cluster [DBG] pgmap v2544: 73 pgs: 73 active+clean; 357 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 82 MiB/s rd, 14 MiB/s wr, 209 op/s 2021-07-31T02:32:18.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:18 smithi045 conmon[33655]: audit 2021-07-31T02:32: 2021-07-31T02:32:18.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:18 smithi045 conmon[33655]: 17.685422+0000 mgr.x (mgr.24542) 1928 : audit [DBG] 2021-07-31T02:32:18.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:18 smithi045 conmon[33655]: from='client.17388 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:32:18.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:18 smithi045 conmon[33655]: audit 2021-07-31T02:32:18.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:18 smithi045 conmon[33655]: 2021-07-31T02:32 2021-07-31T02:32:18.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:18 smithi045 conmon[33655]: :17.966425+0000 mgr.x (mgr 2021-07-31T02:32:18.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:18 smithi045 conmon[33655]: .24542) 1929 : audit [DBG] 2021-07-31T02:32:18.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:18 smithi045 conmon[33655]: from='client.17394 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:32:18.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:18 smithi045 conmon[33655]: audit 2021-07 2021-07-31T02:32:18.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:18 smithi045 conmon[33655]: -31T02:32:18.310445 2021-07-31T02:32:18.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:18 smithi045 conmon[33655]: +0000 mon.c (mon.1) 2021-07-31T02:32:18.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:18 smithi045 conmon[33655]: 117 : audit [DBG] from='client.? 172.21.15.45:0/3647548701' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:32:18.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:18 smithi045 conmon[33655]: cluster 2021 2021-07-31T02:32:18.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:18 smithi045 conmon[33655]: -07-31T02:32: 2021-07-31T02:32:18.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:18 smithi045 conmon[33655]: 18.388641+0000 mon.a (mon 2021-07-31T02:32:18.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:18 smithi045 conmon[33655]: .0) 2309 : cluster [DBG] 2021-07-31T02:32:18.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:18 smithi045 conmon[33655]: osdmap e1348: 8 total, 8 up, 8 in 2021-07-31T02:32:18.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:18 smithi045 conmon[39718]: cluster 2021-07-31T02:32:16.443088 2021-07-31T02:32:18.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:18 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 1927 : cluster [DBG] pgmap v2544: 73 pgs: 73 active+clean; 357 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 82 MiB/s rd, 14 MiB/s wr, 209 op/s 2021-07-31T02:32:18.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:18 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T02:32:18.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:18 smithi045 conmon[39718]: 31T02:32:17.685422+0000 mgr.x (mgr.24542) 1928 : audit 2021-07-31T02:32:18.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:18 smithi045 conmon[39718]: [DBG] from='client.17388 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:32:18.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:18 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T02:32:18.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:18 smithi045 conmon[39718]: 02:32:17.966425+0000 mgr.x (mgr.24542) 1929 : audit [DBG] from='client.17394 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:32:18.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:18 smithi045 conmon[39718]: audit 2021-07-31T02:32:18.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:18 smithi045 conmon[39718]: 2021-07-31 2021-07-31T02:32:18.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:18 smithi045 conmon[39718]: T02:32:18.310445+0000 mon.c 2021-07-31T02:32:18.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:18 smithi045 conmon[39718]: (mon.1) 117 : audit [DBG] 2021-07-31T02:32:18.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:18 smithi045 conmon[39718]: from='client.? 172.21.15.45:0/3647548701' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:32:18.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:18 smithi045 conmon[39718]: cluster 2021-07 2021-07-31T02:32:18.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:18 smithi045 conmon[39718]: -31T02:32:18.388641+0000 2021-07-31T02:32:18.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:18 smithi045 conmon[39718]: mon.a (mon.0) 2309 : cluster 2021-07-31T02:32:18.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:18 smithi045 conmon[39718]: [DBG] osdmap e1348: 8 total, 8 up, 8 in 2021-07-31T02:32:19.394 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 83886080 2021-07-31T02:32:19.417 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 83886079~1 2021-07-31T02:32:19.421 INFO:tasks.workunit.client.0.smithi045.stdout:migration_prepare 2021-07-31T02:32:19.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:19 smithi154 conmon[34553]: cluster 2021-07-31T02:32:18.443655+0000 mgr.x (mgr. 2021-07-31T02:32:19.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:19 smithi154 conmon[34553]: 24542) 1930 : cluster [DBG] pgmap v2547: 73 pgs: 73 active+clean; 357 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail 2021-07-31T02:32:19.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:19 smithi154 conmon[34553]: cluster 2021-07-31T02:32:19.391030+0000 mon.a (mon.0) 2310 : cluster [DBG] osdmap e1349: 8 total, 8 up, 8 in 2021-07-31T02:32:19.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:19 smithi045 conmon[39718]: cluster 2021-07-31T02:32:18.443655+0000 mgr.x (mgr.24542) 1930 2021-07-31T02:32:19.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:19 smithi045 conmon[39718]: : cluster [DBG] pgmap v2547: 73 pgs: 73 active+clean; 357 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail 2021-07-31T02:32:19.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:19 smithi045 conmon[39718]: cluster 2021-07-31T02:32:19.391030+0000 mon.a (mon.0) 2310 : cluster [DBG] osdmap e1349: 8 total, 8 up, 8 in 2021-07-31T02:32:19.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:19 smithi045 conmon[33655]: cluster 2021-07-31T02:32:18.443655+0000 mgr.x (mgr. 2021-07-31T02:32:19.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:19 smithi045 conmon[33655]: 24542) 1930 : cluster [DBG] pgmap v2547: 73 pgs: 73 active+clean; 357 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail 2021-07-31T02:32:19.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:19 smithi045 conmon[33655]: cluster 2021-07-31T02:32:19.391030+0000 mon.a (mon.0) 2310 : cluster [DBG] osdmap e1349: 8 total, 8 up, 8 in 2021-07-31T02:32:21.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:21 smithi154 conmon[34553]: cluster 2021-07-31T02:32:20.424366+0000 mon.a (mon.0 2021-07-31T02:32:21.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:21 smithi154 conmon[34553]: ) 2311 : cluster [DBG] osdmap e1350: 8 total, 8 up, 8 in 2021-07-31T02:32:21.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:21 smithi154 conmon[34553]: cluster 2021-07-31T02:32:20.444191+0000 mgr.x (mgr.24542) 1931 : cluster [DBG] pgmap v2550: 73 pgs: 73 active+clean; 397 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 7.7 KiB/s rd, 8.0 MiB/s wr, 15 op/s 2021-07-31T02:32:21.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:21 smithi045 conmon[39718]: cluster 2021-07-31T02:32:20.424366+0000 2021-07-31T02:32:21.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:21 smithi045 conmon[39718]: mon.a (mon.0) 2311 : cluster [DBG] osdmap e1350: 8 total, 8 up, 8 in 2021-07-31T02:32:21.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:21 smithi045 conmon[39718]: cluster 2021-07-31T02 2021-07-31T02:32:21.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:21 smithi045 conmon[39718]: :32:20.444191+0000 mgr.x (mgr.24542) 1931 : cluster [DBG] pgmap v2550: 73 pgs: 73 active+clean; 397 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 7.7 KiB/s rd, 8.0 MiB/s wr, 15 op/s 2021-07-31T02:32:21.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:21 smithi045 conmon[33655]: cluster 2021-07-31T02:32:20.424366+0000 mon.a ( 2021-07-31T02:32:21.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:21 smithi045 conmon[33655]: mon.0) 2311 : cluster [DBG] osdmap e1350: 8 total, 8 up, 8 in 2021-07-31T02:32:21.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:21 smithi045 conmon[33655]: cluster 2021-07-31T02:32:20.444191+0000 mgr.x (mgr.24542) 1931 : cluster [DBG] pgmap v2550: 73 pgs: 73 active+clean; 397 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 7.7 KiB/s rd, 8.0 MiB/s wr, 15 op/s 2021-07-31T02:32:22.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:22 smithi154 conmon[34553]: cluster 2021-07-31T02:32:22.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:22 smithi154 conmon[34553]: 2021-07-31T02:32:21. 2021-07-31T02:32:22.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:22 smithi154 conmon[34553]: 432032+0000 mon.a (mon.0) 2312 : cluster [DBG] osdmap e1351: 8 total, 8 up, 8 in 2021-07-31T02:32:22.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:22 smithi045 conmon[39718]: cluster 2021-07-31T02:32:21 2021-07-31T02:32:22.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:22 smithi045 conmon[39718]: .432032+0000 mon.a (mon.0) 2312 : cluster [DBG] osdmap e1351: 8 total, 8 up, 8 in 2021-07-31T02:32:22.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:22 smithi045 conmon[33655]: cluster 2021-07-31T02:32:21.432032+0000 mon.a (mon.0) 2312 : cluster [DBG] osdmap e1351: 8 total, 8 up, 8 in 2021-07-31T02:32:22.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:22 smithi045 conmon[33655]: 2021-07-31T02:32:22.873 INFO:tasks.workunit.client.0.smithi045.stdout:migration_execute 2021-07-31T02:32:22.993 INFO:tasks.workunit.client.0.smithi045.stdout:migration_commit 2021-07-31T02:32:23.752 INFO:journalctl@ceph.osd.5.smithi154.stdout:Jul 31 02:32:23 smithi154 conmon[42050]: debug 2021-07-31T02:32:23.462+0000 7fc9b5064700 -1 osd.5 pg_epoch: 1353 pg[75.4( v 1349'63 (0'0,1349'63] local-lis/les=1308/1309 n=6 ec=1308/1308 lis/c=1308/1308 les/c/f=1309/1309/0 sis=1308) [5,0,7] r=0 lpr=1308 crt=1349'63 lcod 1347'61 mlcod 1347'61 active+clean+snaptrim trimq=[22~1]] removing snap head 2021-07-31T02:32:23.752 INFO:journalctl@ceph.osd.5.smithi154.stdout:Jul 31 02:32:23 smithi154 conmon[42050]: debug 2021-07-31T02:32:23.468+0000 7fc9b3861700 -1 osd.5 pg_epoch: 1353 pg[75.f( v 1349'94 (0'0,1349'94] local-lis/les=1308/1309 n=8 ec=1308/1308 lis/c=1308/1308 les/c/f=1309/1309/0 sis=1308) [5,2,4] r=0 lpr=1308 crt=1349'93 lcod 1349'93 mlcod 1349'93 active+clean+snaptrim trimq=[22~1]] removing snap head 2021-07-31T02:32:23.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:23 smithi154 conmon[34553]: cluster 2021-07-31T02:32:22.433836+0000 mon.a (mon. 2021-07-31T02:32:23.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:23 smithi154 conmon[34553]: 0) 2313 : cluster [DBG] osdmap e1352: 8 total, 8 up, 8 in 2021-07-31T02:32:23.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:23 smithi154 conmon[34553]: cluster 2021-07-31T02:32:22.444603+0000 mgr.x (mgr.24542) 1932 : cluster [DBG] pgmap v2553: 73 pgs: 73 active+clean; 397 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 7.7 KiB/s rd, 8.0 MiB/s wr, 15 op/s 2021-07-31T02:32:23.775 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:32:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:32:23.587Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:32:23.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:32:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:32:23.587Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:32:23.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:32:23 smithi045 conmon[61210]: level=error ts=2021-07-31T02:32:23.587Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:32:23.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:23 smithi045 conmon[33655]: cluster 2021-07-31T02:32:22.433836+0000 mon.a (mon. 2021-07-31T02:32:23.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:23 smithi045 conmon[33655]: 0) 2313 : cluster [DBG] osdmap e1352: 8 total, 8 up, 8 in 2021-07-31T02:32:23.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:23 smithi045 conmon[33655]: cluster 2021-07-31T02:32:22.444603+0000 mgr.x (mgr.24542) 1932 : cluster [DBG] pgmap v2553: 73 pgs: 73 active+clean; 397 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 7.7 KiB/s rd, 8.0 MiB/s wr, 15 op/s 2021-07-31T02:32:23.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:23 smithi045 conmon[39718]: cluster 2021-07-31T02:32:22.433836+0000 2021-07-31T02:32:23.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:23 smithi045 conmon[39718]: mon.a (mon.0) 2313 : cluster [DBG] osdmap e1352: 8 total, 8 up, 8 in 2021-07-31T02:32:23.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:23 smithi045 conmon[39718]: cluster 2021-07- 2021-07-31T02:32:23.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:23 smithi045 conmon[39718]: 31T02:32:22.444603+0000 mgr.x (mgr.24542) 1932 2021-07-31T02:32:23.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:23 smithi045 conmon[39718]: : cluster [DBG] pgmap v2553: 73 pgs: 73 active+clean; 397 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 7.7 KiB/s rd, 8.0 MiB/s wr, 15 op/s 2021-07-31T02:32:24.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:24 smithi154 conmon[34553]: cluster 2021-07-31T02:32:23.446058+0000 mon.a (mon.0) 2314 : cluster [DBG] osdmap e1353: 8 total, 8 up, 8 in 2021-07-31T02:32:24.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:24 smithi045 conmon[39718]: cluster 2021-07-31T02:32:23.446058+0000 mon.a (mon.0) 2021-07-31T02:32:24.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:24 smithi045 conmon[39718]: 2314 : cluster [DBG] osdmap e1353: 8 total, 8 up, 8 in 2021-07-31T02:32:24.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:24 smithi045 conmon[33655]: cluster 2021-07-31T02:32:23 2021-07-31T02:32:24.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:24 smithi045 conmon[33655]: .446058+0000 mon.a (mon.0) 2314 : cluster [DBG] osdmap e1353: 8 total, 8 up, 8 in 2021-07-31T02:32:25.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:25 smithi154 conmon[34553]: cluster 2021-07-31T02:32:24.445228+0000 mgr.x (mgr.24542) 1933 : cluster [DBG] pgmap v2555: 73 pgs: 4 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 67 active+clean; 417 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 38 MiB/s rd, 6.0 MiB/s wr, 217 op/s 2021-07-31T02:32:25.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:25 smithi154 conmon[34553]: cluster 2021-07-31T02:32:24.453062+0000 mon.a (mon.0) 2315 : cluster [DBG] osdmap e1354: 8 total, 8 up, 8 in 2021-07-31T02:32:25.777 INFO:journalctl@ceph.osd.3.smithi045.stdout:Jul 31 02:32:25 smithi045 conmon[54013]: debug 2021-07-31T02:32:25.485+0000 7f64f88c5700 -1 osd.3 pg_epoch: 1355 pg[75.13( v 1355'81 (0'0,1355'81] local-lis/les=1308/1309 n=12 ec=1308/1308 lis/c=1308/1308 les/c/f=1309/1309/0 sis=1308) [3,5,0] r=0 lpr=1308 luod=1354'80 crt=1355'81 lcod 1349'78 mlcod 1349'78 active+clean+snaptrim trimq=[1e~1] ps=[20~1,22~1]] removing snap head 2021-07-31T02:32:25.778 INFO:journalctl@ceph.osd.0.smithi045.stdout:Jul 31 02:32:25 smithi045 conmon[43243]: debug 2021-07-31T02:32:25.478+0000 7fdf69f8c700 -1 osd.0 pg_epoch: 1355 pg[75.15( v 1355'79 (0'0,1355'79] local-lis/les=1308/1309 n=11 ec=1308/1308 lis/c=1308/1308 les/c/f=1309/1309/0 sis=1308) [0,7,2] r=0 lpr=1308 luod=1355'78 crt=1355'79 lcod 1354'77 mlcod 1354'77 active+clean+snaptrim trimq=[1e~1] ps=[20~1,22~1]] removing snap head 2021-07-31T02:32:25.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:25 smithi045 conmon[39718]: cluster 2021-07-31T02:32:24.445228+0000 mgr.x (mgr.24542) 2021-07-31T02:32:25.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:25 smithi045 conmon[39718]: 1933 : cluster [DBG] pgmap v2555: 73 pgs: 4 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 67 active+clean; 417 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 38 MiB/s rd, 6.0 MiB/s wr, 217 op/s 2021-07-31T02:32:25.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:25 smithi045 conmon[39718]: cluster 2021-07-31T02:32: 2021-07-31T02:32:25.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:25 smithi045 conmon[39718]: 24.453062+0000 mon.a (mon.0) 2315 : cluster [DBG] osdmap e1354: 8 total, 8 up, 8 in 2021-07-31T02:32:25.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:25 smithi045 conmon[33655]: cluster 2021-07-31T02:32:24. 2021-07-31T02:32:25.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:25 smithi045 conmon[33655]: 445228+0000 mgr.x (mgr.24542) 1933 : cluster [DBG] pgmap v2555: 73 pgs: 4 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 67 active+clean; 417 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 38 MiB/s rd, 6.0 MiB/s wr, 217 op/s 2021-07-31T02:32:25.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:25 smithi045 conmon[33655]: cluster 2021-07-31T02:32:24.453062+0000 mon.a (mon.0) 2315 : cluster [DBG] osdmap e1354: 8 total, 8 up, 8 in 2021-07-31T02:32:25.831 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMigration.SnapsDataPool (12102 ms) 2021-07-31T02:32:25.832 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMigration.SnapsShrinkAfterPrepare 2021-07-31T02:32:25.848 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 83886080 2021-07-31T02:32:25.852 INFO:tasks.workunit.client.0.smithi045.stdout:write: 1 0~8388609 2021-07-31T02:32:25.899 INFO:tasks.workunit.client.0.smithi045.stdout:snap_create: snap1 2021-07-31T02:32:26.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:26 smithi154 conmon[34553]: cluster 2021-07-31T02:32:25.457883+0000 mon.a (mon.0) 2316 : cluster [DBG] osdmap e1355: 8 total, 8 up, 8 in 2021-07-31T02:32:26.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:26 smithi045 conmon[33655]: cluster 2021-07-31T02:32:25.457883+0000 mon.a (mon.0) 2316 : 2021-07-31T02:32:26.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:26 smithi045 conmon[33655]: cluster [DBG] osdmap e1355: 8 total, 8 up, 8 in 2021-07-31T02:32:26.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:26 smithi045 conmon[39718]: cluster 2021-07-31T02:32:25.457883+0000 2021-07-31T02:32:26.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:26 smithi045 conmon[39718]: mon.a (mon.0) 2316 : cluster [DBG] osdmap e1355: 8 total, 8 up, 8 in 2021-07-31T02:32:27.473 INFO:tasks.workunit.client.0.smithi045.stdout:write: 1 8388609~8388609 2021-07-31T02:32:27.484 INFO:tasks.workunit.client.0.smithi045.stdout:write: X 0~1000 2021-07-31T02:32:27.484 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 1010~1000 2021-07-31T02:32:27.484 INFO:tasks.workunit.client.0.smithi045.stdout:snap_create: snap2 2021-07-31T02:32:27.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:27 smithi154 conmon[34553]: cluster 2021-07-31T02:32:26.445670+0000 mgr.x (mgr.24542) 1934 : cluster [DBG] 2021-07-31T02:32:27.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:27 smithi154 conmon[34553]: pgmap v2558: 73 pgs: 4 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 67 active+clean; 417 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 38 MiB/s rd, 6.0 MiB/s wr, 218 op/s 2021-07-31T02:32:27.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:27 smithi154 conmon[34553]: cluster 2021-07-31T02:32:26.463522+0000 mon.a (mon.0) 2317 : cluster [DBG] osdmap e1356: 8 total, 8 up, 8 in 2021-07-31T02:32:27.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:27 smithi045 conmon[33655]: cluster 2021-07-31T02:32:26.445670+0000 2021-07-31T02:32:27.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:27 smithi045 conmon[33655]: mgr.x (mgr.24542) 1934 : cluster [DBG] pgmap v2558: 73 pgs: 4 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 67 active+clean; 417 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 38 MiB/s rd, 6.0 MiB/s wr, 218 op/s 2021-07-31T02:32:27.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:27 smithi045 conmon[33655]: cluster 2021-07-31T02:32:26.463522+0000 mon.a (mon.0) 2317 : cluster [DBG] osdmap e1356: 8 total, 8 up, 8 in 2021-07-31T02:32:27.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:27 smithi045 conmon[39718]: cluster 2021-07-31T02:32:26.445670+0000 2021-07-31T02:32:27.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:27 smithi045 conmon[39718]: mgr.x (mgr.24542) 1934 : cluster [DBG] pgmap v2558: 73 pgs: 4 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 67 active+clean; 417 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 38 MiB/s rd, 6.0 MiB/s wr, 218 op/s 2021-07-31T02:32:27.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:27 smithi045 conmon[39718]: cluster 2021-07-31T02:32: 2021-07-31T02:32:27.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:27 smithi045 conmon[39718]: 26.463522+0000 mon.a (mon.0) 2317 : cluster [DBG] osdmap e1356: 8 total, 8 up, 8 in 2021-07-31T02:32:28.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:28 smithi154 conmon[34553]: cluster 2021-07-31T02:32:27.469690+0000 mon.a (mon.0) 2021-07-31T02:32:28.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:28 smithi154 conmon[34553]: 2318 : cluster [DBG] osdmap e1357: 8 total, 8 up, 8 in 2021-07-31T02:32:28.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:28 smithi154 conmon[34553]: audit 2021-07-31T02:32:28.093405+0000 mon.b (mon.2) 300 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:32:28.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:28 smithi154 conmon[34553]: audit 2021-07-31T02:32:28. 2021-07-31T02:32:28.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:28 smithi154 conmon[34553]: 093952+0000 mon.a (mon.0) 2319 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:32:28.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:28 smithi154 conmon[34553]: 2021-07-31T02:32:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:28 smithi154 conmon[34553]: audit 2021-07-31T02:32 2021-07-31T02:32:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:28 smithi154 conmon[34553]: :28.112987+0000 mon.b (mon.2) 301 : 2021-07-31T02:32:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:28 smithi154 conmon[34553]: audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:32:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:28 smithi154 conmon[34553]: audit 2021-07-31T02:32: 2021-07-31T02:32:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:28 smithi154 conmon[34553]: 28.113451+0000 mon.a (mon.0) 2320 : audit 2021-07-31T02:32:28.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:28 smithi154 conmon[34553]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:32:28.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:28 smithi045 conmon[33655]: cluster 2021-07-31T02:32:27.469690+0000 mon.a (mon. 2021-07-31T02:32:28.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:28 smithi045 conmon[33655]: 0) 2318 : cluster [DBG] osdmap e1357: 8 total, 8 up, 8 in 2021-07-31T02:32:28.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:28 smithi045 conmon[33655]: audit 2021-07-31T02:32:28.093405+0000 mon.b (mon.2) 300 2021-07-31T02:32:28.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:28 smithi045 conmon[33655]: : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:32:28.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:28 smithi045 conmon[33655]: audit 2021-07-31T02 2021-07-31T02:32:28.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:28 smithi045 conmon[33655]: :32:28.093952+0000 mon.a (mon.0) 2319 : 2021-07-31T02:32:28.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:28 smithi045 conmon[33655]: audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:32:28.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:28 smithi045 conmon[33655]: audit 2021-07-31T 2021-07-31T02:32:28.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:28 smithi045 conmon[33655]: 02:32:28.112987+0000 mon.b (mon. 2021-07-31T02:32:28.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:28 smithi045 conmon[33655]: 2) 301 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:32:28.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:28 smithi045 conmon[33655]: audit 2021-07-31T02:32:28.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:28 smithi045 conmon[33655]: 2021-07-31T02:32: 2021-07-31T02:32:28.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:28 smithi045 conmon[33655]: 28.113451+0000 mon.a (mon.0) 2320 : 2021-07-31T02:32:28.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:28 smithi045 conmon[33655]: audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:32:28.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:28 smithi045 conmon[39718]: cluster 2021-07-31T02:32:27. 2021-07-31T02:32:28.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:28 smithi045 conmon[39718]: 469690+0000 mon.a (mon.0) 2318 : cluster [DBG] osdmap e1357: 8 total, 8 up, 8 in 2021-07-31T02:32:28.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:28 smithi045 conmon[39718]: audit 2021-07 2021-07-31T02:32:28.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:28 smithi045 conmon[39718]: -31T02:32:28. 2021-07-31T02:32:28.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:28 smithi045 conmon[39718]: 093405+0000 mon.b (mon.2) 2021-07-31T02:32:28.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:28 smithi045 conmon[39718]: 300 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:32:28.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:28 smithi045 conmon[39718]: audit 2021- 2021-07-31T02:32:28.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:28 smithi045 conmon[39718]: 07-31T02:32:28.093952 2021-07-31T02:32:28.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:28 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2021-07-31T02:32:28.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:28 smithi045 conmon[39718]: 2319 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:32:28.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:28 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T02:32:28.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:28 smithi045 conmon[39718]: T02:32:28.112987+0000 mon.b (mon.2) 301 : audit [INF] 2021-07-31T02:32:28.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:28 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:32:28.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:28 smithi045 conmon[39718]: audit 2021-07-31T02:32:28.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:28 smithi045 conmon[39718]: 2021-07-31T02: 2021-07-31T02:32:28.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:28 smithi045 conmon[39718]: 32:28.113451+0000 mon.a ( 2021-07-31T02:32:28.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:28 smithi045 conmon[39718]: mon.0) 2320 : audit 2021-07-31T02:32:28.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:28 smithi045 conmon[39718]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:32:29.484 INFO:tasks.workunit.client.0.smithi045.stdout:write: X 8388609~1000 2021-07-31T02:32:29.486 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 16777228~1000 2021-07-31T02:32:29.486 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 167772160 2021-07-31T02:32:29.492 INFO:tasks.workunit.client.0.smithi045.stdout:write: 2 83886079~8388609 2021-07-31T02:32:29.500 INFO:tasks.workunit.client.0.smithi045.stdout:snap_create: snap3 2021-07-31T02:32:29.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:29 smithi154 conmon[34553]: cluster 2021-07-31T02:32:28.446102+0000 mgr.x (mgr.24542) 1935 : cluster [DBG] pgmap v2561: 73 pgs: 4 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 67 active+clean; 417 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2021-07-31T02:32:29.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:29 smithi154 conmon[34553]: cluster 2021-07-31T02:32:29.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:29 smithi154 conmon[34553]: 2021-07-31T02:32:28.476358+0000 mon.a (mon.0) 2321 : cluster [DBG] osdmap e1358: 8 total, 8 up, 8 in 2021-07-31T02:32:29.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:29 smithi045 conmon[39718]: cluster 2021-07-31T02:32:28.446102+0000 mgr.x (mgr.24542) 2021-07-31T02:32:29.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:29 smithi045 conmon[39718]: 1935 : cluster [DBG] pgmap v2561: 73 pgs: 4 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 67 active+clean; 417 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2021-07-31T02:32:29.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:29 smithi045 conmon[39718]: cluster 2021-07-31T02:32:28.476358+0000 mon.a (mon.0) 2321 : cluster 2021-07-31T02:32:29.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:29 smithi045 conmon[39718]: [DBG] osdmap e1358: 8 total, 8 up, 8 in 2021-07-31T02:32:29.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:29 smithi045 conmon[33655]: cluster 2021-07-31T02:32:28.446102+0000 mgr.x (mgr.24542) 1935 : cluster [DBG] pgmap v2561: 73 pgs: 4 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 67 active+clean; 417 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2021-07-31T02:32:29.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:29 smithi045 conmon[33655]: 2021-07-31T02:32:29.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:29 smithi045 conmon[33655]: cluster 2021-07-31T02:32:28.476358+0000 mon.a (mon.0) 2321 : cluster [DBG] osdmap e1358: 8 total, 8 up, 8 in 2021-07-31T02:32:30.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:30 smithi154 conmon[34553]: cluster 2021-07-31T02:32:29.477190+0000 mon.a (mon.0) 2322 2021-07-31T02:32:30.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:30 smithi154 conmon[34553]: : cluster [DBG] osdmap e1359: 8 total, 8 up, 8 in 2021-07-31T02:32:30.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:30 smithi045 conmon[39718]: cluster 2021-07-31T02:32:29.477190+0000 mon.a (mon 2021-07-31T02:32:30.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:30 smithi045 conmon[39718]: .0) 2322 : cluster [DBG] osdmap e1359: 8 total, 8 up, 8 in 2021-07-31T02:32:30.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:30 smithi045 conmon[33655]: cluster 2021-07-31T02:32:29.477190+0000 mon.a (mon.0) 2322 : 2021-07-31T02:32:30.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:30 smithi045 conmon[33655]: cluster [DBG] osdmap e1359: 8 total, 8 up, 8 in 2021-07-31T02:32:31.497 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 83886080 2021-07-31T02:32:31.517 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 83886079~1 2021-07-31T02:32:31.520 INFO:tasks.workunit.client.0.smithi045.stdout:migration_prepare 2021-07-31T02:32:31.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:31 smithi154 conmon[34553]: cluster 2021-07-31T02:32:30.446637+0000 mgr.x (mgr.24542) 1936 : cluster [DBG] pgmap v2564: 73 pgs: 73 active+clean; 429 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 32 MiB/s rd, 8.0 MiB/s wr, 38 op/s 2021-07-31T02:32:31.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:31 smithi154 conmon[34553]: 2021-07-31T02:32:31.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:31 smithi154 conmon[34553]: cluster 2021-07-31T02:32:30.487956+0000 mon.a (mon.0) 2323 : cluster [DBG] osdmap e1360: 8 total, 8 up, 8 in 2021-07-31T02:32:31.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:31 smithi045 conmon[33655]: cluster 2021-07-31T02:32:30.446637+0000 mgr.x (mgr. 2021-07-31T02:32:31.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:31 smithi045 conmon[33655]: 24542) 1936 : cluster [DBG] pgmap v2564: 73 pgs: 73 active+clean; 429 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 32 MiB/s rd, 8.0 MiB/s wr, 38 op/s 2021-07-31T02:32:31.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:31 smithi045 conmon[33655]: cluster 2021-07-31T02:32:30.487956+0000 mon.a (mon.0) 2323 : cluster [DBG] osdmap e1360: 8 total, 8 up, 8 in 2021-07-31T02:32:31.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:31 smithi045 conmon[39718]: cluster 2021-07-31T02:32:30.446637+0000 mgr.x (mgr.24542) 1936 : 2021-07-31T02:32:31.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:31 smithi045 conmon[39718]: cluster [DBG] pgmap v2564: 73 pgs: 73 active+clean; 429 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 32 MiB/s rd, 8.0 MiB/s wr, 38 op/s 2021-07-31T02:32:31.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:31 smithi045 conmon[39718]: cluster 2021-07-31T02:32:30.487956 2021-07-31T02:32:31.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:31 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2323 : cluster [DBG] osdmap e1360: 8 total, 8 up, 8 in 2021-07-31T02:32:32.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[34553]: cluster 2021-07-31T02:32:31.489677+0000 mon.a (mon.0) 2324 : cluster [DBG] osdmap e1361: 8 total, 8 up, 8 in 2021-07-31T02:32:32.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[34553]: cluster 2021-07-31T02:32:32.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[34553]: 2021-07-31T02:32:32.343076+0000 mon.a (mon.0) 2325 : cluster [DBG] osdmap e1362: 8 total, 8 up, 8 in 2021-07-31T02:32:32.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[34553]: audit 2021-07-31T02:32:32.458931+0000 mon.b (mon.2) 302 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:32:32.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[34553]: audit 2021-07-31T02:32:32.460404+0000 mon.b (mon.2) 303 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:32:32.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[34553]: audit 2021-07-31T02:32:32.461236+0000 mon.b (mon.2) 304 : 2021-07-31T02:32:32.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[34553]: audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:32:32.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: debug 2021-07-31T02:32:32.506+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:32:32.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:32:32.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:32:32.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: yield 2021-07-31T02:32:32.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:32:32.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:32:32.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:32:32.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:32:32.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:32:32.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:32:32.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:32:32.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:32:32.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:32:32.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: task.result() 2021-07-31T02:32:32.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:32:32.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: debug 2021-07-31T02:32:32.518+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:32:32.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:32:32.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:32:32.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: yield 2021-07-31T02:32:32.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:32:32.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:32:32.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:32:32.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:32:32.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:32:32.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:32:32.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:32:32.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:32:32.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:32:32.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: task.result() 2021-07-31T02:32:32.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:32:32.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: debug 2021-07-31T02:32:32.543+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:32:32.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:32:32.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:32:32.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: yield 2021-07-31T02:32:32.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:32:32.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:32:32.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:32:32.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:32:32.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:32:32.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:32:32.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:32:32.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:32:32.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:32:32.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: task.result() 2021-07-31T02:32:32.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:32:32.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: debug 2021-07-31T02:32:32.544+0000 7fcb93961700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:32:32.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:32:32.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:32:32.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: yield 2021-07-31T02:32:32.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:32:32.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:32:32.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:32:32.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:32:32.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:32:32.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:32:32.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:32:32.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:32:32.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:32:32.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: task.result() 2021-07-31T02:32:32.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:32:32.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: 2021-07-31T02:32:32.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:32:32.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: 2021-07-31T02:32:32.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:32:32.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:32:32.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:32:32.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:32:32.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:32:32.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:32:32.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:32:32.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:32:32.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:32:32.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:32:32.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:32:32.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:32:32.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:32:32.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:32:32.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:32:32.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:32:32.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:32:32.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:32:32.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: raise 2021-07-31T02:32:32.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:32:32.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:32:32.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:32:32.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:32:32.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:32:32.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: debug 2021-07-31T02:32:32.552+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:32:32.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:32:32.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:32:32.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: yield 2021-07-31T02:32:32.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:32:32.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:32:32.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:32:32.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:32:32.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:32:32.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:32:32.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:32:32.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:32:32.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:32:32.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: task.result() 2021-07-31T02:32:32.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:32:32.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: debug 2021-07-31T02:32:32.553+0000 7fcb94162700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:32:32.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:32:32.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:32:32.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: yield 2021-07-31T02:32:32.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:32:32.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:32:32.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:32:32.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:32:32.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:32:32.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:32:32.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:32:32.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:32:32.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:32:32.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: task.result() 2021-07-31T02:32:32.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:32:32.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: 2021-07-31T02:32:32.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:32:32.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: 2021-07-31T02:32:32.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:32:32.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:32:32.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:32:32.786 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:32:32.786 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:32:32.786 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:32:32.786 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:32:32.787 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:32:32.787 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:32:32.787 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:32:32.787 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:32:32.788 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:32:32.788 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:32:32.788 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:32:32.789 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:32:32.789 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:32:32.789 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:32:32.789 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:32:32.790 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: raise 2021-07-31T02:32:32.790 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:32:32.790 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:32:32.790 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:32:32.791 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:32:32.791 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:32:32 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:32:32.792 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:32 smithi045 conmon[33655]: cluster 2021-07-31T02:32:31.489677+0000 mon.a ( 2021-07-31T02:32:32.792 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:32 smithi045 conmon[33655]: mon.0) 2324 : cluster [DBG] osdmap e1361: 8 total, 8 up, 8 in 2021-07-31T02:32:32.792 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:32 smithi045 conmon[33655]: cluster 2021-07-31 2021-07-31T02:32:32.793 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:32 smithi045 conmon[33655]: T02:32:32.343076+0000 mon.a ( 2021-07-31T02:32:32.793 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:32 smithi045 conmon[33655]: mon.0) 2325 : cluster [DBG] osdmap e1362: 8 total, 8 up, 8 in 2021-07-31T02:32:32.793 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:32 smithi045 conmon[33655]: 2021-07-31T02:32:32.793 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:32 smithi045 conmon[33655]: audit 2021-07 2021-07-31T02:32:32.794 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:32 smithi045 conmon[33655]: -31T02:32:32.458931+0000 mon.b (mon.2) 302 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:32:32.795 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:32 smithi045 conmon[33655]: audit 2021- 2021-07-31T02:32:32.795 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:32 smithi045 conmon[33655]: 07-31T02:32:32.460404+0000 2021-07-31T02:32:32.795 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:32 smithi045 conmon[33655]: mon.b (mon.2) 303 : audit [DBG] 2021-07-31T02:32:32.795 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:32 smithi045 conmon[33655]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:32:32.796 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:32 smithi045 conmon[33655]: audit 2021-07-31T02: 2021-07-31T02:32:32.796 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:32 smithi045 conmon[33655]: 32:32.461236+0000 mon.b (mon.2 2021-07-31T02:32:32.796 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:32 smithi045 conmon[33655]: ) 304 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:32:32.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:32 smithi045 conmon[39718]: cluster 2021-07-31T02:32:31.489677+0000 mon.a ( 2021-07-31T02:32:32.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:32 smithi045 conmon[39718]: mon.0) 2324 : cluster [DBG] osdmap e1361: 8 total, 8 up, 8 in 2021-07-31T02:32:32.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:32 smithi045 conmon[39718]: cluster 2021-07-31T02:32:32 2021-07-31T02:32:32.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:32 smithi045 conmon[39718]: .343076+0000 mon.a (mon.0) 2325 : cluster [DBG] osdmap e1362: 8 total, 8 up, 8 in 2021-07-31T02:32:32.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:32 smithi045 conmon[39718]: audit 2021-07-31T02:32:32.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:32 smithi045 conmon[39718]: 2021-07-31 2021-07-31T02:32:32.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:32 smithi045 conmon[39718]: T02:32:32.458931+0000 mon.b (mon. 2021-07-31T02:32:32.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:32 smithi045 conmon[39718]: 2) 302 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:32:32.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:32 smithi045 conmon[39718]: audit 2021-07-31T02:32:32.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:32 smithi045 conmon[39718]: 2021-07-31T02:32:32.460404+0000 mon.b (mon.2) 303 : audit 2021-07-31T02:32:32.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:32 smithi045 conmon[39718]: [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:32:32.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:32 smithi045 conmon[39718]: audit 2021-07-31T02:32:32.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:32 smithi045 conmon[39718]: 2021-07-31T02 2021-07-31T02:32:32.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:32 smithi045 conmon[39718]: :32:32.461236+0000 mon.b (mon.2 2021-07-31T02:32:32.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:32 smithi045 conmon[39718]: ) 304 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:32:34.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:32:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:32:33.588Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:32:34.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:32:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:32:33.588Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:32:34.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:32:33 smithi045 conmon[61210]: level=error ts=2021-07-31T02:32:33.588Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:32:34.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: cluster 2021-07-31T02:32:32.447229+0000 mgr.x (mgr.24542) 1937 : cluster [DBG] 2021-07-31T02:32:34.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: pgmap v2568: 73 pgs: 73 active+clean; 429 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 32 MiB/s rd, 8.1 MiB/s wr, 39 op/s 2021-07-31T02:32:34.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: cephadm 2021-07-31T02:32:32.461960+0000 mgr.x (mgr.24542) 1938 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:32:34.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: cephadm 2021-07-31 2021-07-31T02:32:34.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: T02:32:32.507039+0000 mgr.x (mgr.24542) 1939 : cephadm [ERR] 2021-07-31T02:32:34.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:32:34.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:32:34.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: yield 2021-07-31T02:32:34.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:32:34.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:32:34.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:32:34.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:32:34.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:32:34.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:32:34.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:32:34.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:32:34.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:32:34.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: task.result() 2021-07-31T02:32:34.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:32:34.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: cephadm 2021-07-31T02:32:32.507437+0000 mgr.x (mgr.24542) 1940 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:32:34.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: cephadm 2021-07-31T02:32:32.519084+0000 mgr.x (mgr.24542) 1941 : cephadm [ERR] 2021-07-31T02:32:34.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:32:34.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:32:34.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: yield 2021-07-31T02:32:34.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:32:34.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:32:34.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:32:34.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:32:34.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:32:34.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:32:34.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:32:34.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:32:34.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:32:34.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: task.result() 2021-07-31T02:32:34.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:32:34.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: cephadm 2021-07-31T02:32:34.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: 2021-07-31T02:32:32.520004+0000 mgr.x (mgr.24542) 1942 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:32:34.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: cephadm 2021-07-31T02:32:32.544697+0000 mgr.x (mgr.24542) 1943 : cephadm [ERR] 2021-07-31T02:32:34.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:32:34.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:32:34.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: yield 2021-07-31T02:32:34.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:32:34.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:32:34.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:32:34.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:32:34.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:32:34.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:32:34.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:32:34.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:32:34.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:32:34.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: task.result() 2021-07-31T02:32:34.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:32:34.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: cephadm 2021-07- 2021-07-31T02:32:34.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: 31T02:32:32.545158+0000 mgr.x (mgr.24542) 1944 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:32:34.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:32:34.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:32:34.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: yield 2021-07-31T02:32:34.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:32:34.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:32:34.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:32:34.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:32:34.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:32:34.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:32:34.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:32:34.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:32:34.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:32:34.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: task.result() 2021-07-31T02:32:34.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:32:34.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: 2021-07-31T02:32:34.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:32:34.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: 2021-07-31T02:32:34.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:32:34.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:32:34.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:32:34.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:32:34.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:32:34.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:32:34.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:32:34.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:32:34.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:32:34.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:32:34.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:32:34.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:32:34.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:32:34.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:32:34.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:32:34.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:32:34.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:32:34.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:32:34.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: raise 2021-07-31T02:32:34.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:32:34.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:32:34.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:32:34.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:32:34.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:32:34.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: cephadm 2021-07-31T02:32:32.553614+0000 mgr.x (mgr.24542) 1945 : cephadm [ERR] 2021-07-31T02:32:34.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:32:34.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:32:34.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: yield 2021-07-31T02:32:34.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:32:34.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:32:34.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:32:34.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:32:34.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:32:34.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:32:34.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:32:34.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:32:34.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:32:34.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: task.result() 2021-07-31T02:32:34.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:32:34.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: cephadm 2021-07-31T02:32:32.554150+0000 mgr.x (mgr.24542) 1946 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:32:34.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:32:34.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:32:34.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: yield 2021-07-31T02:32:34.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:32:34.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:32:34.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:32:34.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:32:34.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:32:34.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:32:34.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:32:34.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:32:34.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:32:34.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: task.result() 2021-07-31T02:32:34.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:32:34.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: 2021-07-31T02:32:34.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:32:34.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: 2021-07-31T02:32:34.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:32:34.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:32:34.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:32:34.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:32:34.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:32:34.782 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:32:34.783 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:32:34.783 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:32:34.783 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:32:34.783 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:32:34.784 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:32:34.784 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:32:34.784 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:32:34.784 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:32:34.785 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:32:34.785 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:32:34.785 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:32:34.785 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:32:34.785 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: raise 2021-07-31T02:32:34.786 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:32:34.786 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:32:34.786 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:32:34.786 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:32:34.787 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:32:34.787 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:34 smithi154 conmon[34553]: cluster 2021-07-31T02:32:33.353948+0000 mon.a (mon.0) 2326 : cluster [DBG] osdmap e1363: 8 total, 8 up, 8 in 2021-07-31T02:32:34.788 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: cluster 2021-07-31T02:32:34.788 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: 2021-07-31T02:32:34.788 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: 2021 2021-07-31T02:32:34.788 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: -07-31T02:32:32.447229+0000 2021-07-31T02:32:34.789 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: mgr.x (mgr.24542) 1937 : cluster [DBG] pgmap v2568: 73 pgs: 73 active+clean; 429 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 32 MiB/s rd, 8.1 MiB/s wr, 39 op/s 2021-07-31T02:32:34.789 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: cephadm 2021-07- 2021-07-31T02:32:34.789 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: 31T02:32:32.461960+0000 mgr.x (mgr.24542) 1938 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:32:34.789 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: cephadm 2021-07-31 2021-07-31T02:32:34.789 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: T02:32:32.507039+0000 mgr.x (mgr.24542) 1939 : cephadm [ERR] 2021-07-31T02:32:34.790 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:32:34.790 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:32:34.790 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: yield 2021-07-31T02:32:34.790 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:32:34.791 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:32:34.791 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:32:34.791 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:32:34.792 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:32:34.792 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:32:34.792 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:32:34.792 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:32:34.793 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:32:34.793 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: task.result() 2021-07-31T02:32:34.793 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:32:34.793 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: cephadm 2021-07-31T02:32:32.507437+0000 mgr.x (mgr.24542) 1940 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:32:34.793 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: cephadm 2021-07-31T02:32:32.519084+0000 mgr.x (mgr.24542) 1941 : cephadm [ERR] 2021-07-31T02:32:34.794 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:32:34.794 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:32:34.794 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: yield 2021-07-31T02:32:34.794 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:32:34.795 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:32:34.795 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:32:34.795 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:32:34.795 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:32:34.796 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:32:34.796 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:32:34.796 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:32:34.796 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:32:34.797 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: task.result() 2021-07-31T02:32:34.797 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:32:34.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: cluster 2021-07-31T02:32:34.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: 2021-07-31T02:32:34.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: 2021 2021-07-31T02:32:34.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: -07-31T02:32 2021-07-31T02:32:34.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: :32.447229+0000 mgr.x (mgr.24542) 1937 : cluster [DBG] pgmap v2568: 73 pgs: 73 active+clean; 429 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 32 MiB/s rd, 8.1 MiB/s wr, 39 op/s 2021-07-31T02:32:34.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: cephadm 2021-07- 2021-07-31T02:32:34.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: 31T02:32:32.461960+0000 mgr.x (mgr.24542) 1938 : cephadm [WRN] 2021-07-31T02:32:34.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:32:34.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: cephadm 2021-07-31T02:32:34.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: 2021-07-31T02:32:32.507039+0000 2021-07-31T02:32:34.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: mgr.x (mgr.24542) 1939 : cephadm [ERR] 2021-07-31T02:32:34.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:32:34.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:32:34.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: yield 2021-07-31T02:32:34.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:32:34.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:32:34.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:32:34.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:32:34.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:32:34.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:32:34.802 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:32:34.802 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:32:34.802 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:32:34.802 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: task.result() 2021-07-31T02:32:34.803 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:32:34.803 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: 2021-07-31T02:32:34.803 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: cephadm 2021-07-31T02:32:32.507437+0000 mgr.x (mgr.24542) 1940 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:32:34.803 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: cephadm 2021-07-31T02:32:34.803 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: 2021-07-31T02:32:32.519084+0000 mgr.x (mgr.24542) 1941 : cephadm [ERR] 2021-07-31T02:32:34.804 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: 2021-07-31T02:32:34.804 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:32:34.804 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:32:34.804 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: yield 2021-07-31T02:32:34.804 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:32:34.805 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:32:34.805 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:32:34.805 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:32:34.805 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:32:34.806 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:32:34.806 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:32:34.806 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:32:34.807 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:32:34.807 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: cephadm 2021-07-31T02:32:32.520004+0000 mgr.x (mgr.24542) 1942 : cephadm 2021-07-31T02:32:34.807 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:32:34.808 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: cephadm 2021-07-31T02:32:32.544697+0000 mgr.x (mgr.24542) 1943 : cephadm [ERR] 2021-07-31T02:32:34.808 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:32:34.808 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:32:34.808 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: yield 2021-07-31T02:32:34.809 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:32:34.809 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:32:34.809 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:32:34.809 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:32:34.810 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:32:34.810 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:32:34.810 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:32:34.810 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:32:34.811 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:32:34.811 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: task.result() 2021-07-31T02:32:34.811 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:32:34.812 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: cephadm 2021-07-31T02:32:32.545158+0000 mgr.x (mgr.24542) 1944 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:32:34.812 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:32:34.812 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:32:34.812 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: yield 2021-07-31T02:32:34.812 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:32:34.813 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:32:34.813 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:32:34.813 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:32:34.813 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:32:34.814 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:32:34.814 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:32:34.814 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:32:34.814 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:32:34.814 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: task.result() 2021-07-31T02:32:34.815 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:32:34.815 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: 2021-07-31T02:32:34.815 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:32:34.816 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: 2021-07-31T02:32:34.816 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:32:34.816 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:32:34.816 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:32:34.816 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:32:34.817 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:32:34.817 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:32:34.818 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: task.result() 2021-07-31T02:32:34.818 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:32:34.818 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: cephadm 2021-07-31T02:32:32.520004+0000 mgr.x (mgr.24542) 1942 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:32:34.818 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: cephadm 2021-07-31T02:32:32.544697+0000 mgr.x (mgr.24542) 1943 : cephadm [ERR] 2021-07-31T02:32:34.818 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: 2021-07-31T02:32:34.819 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:32:34.819 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:32:34.819 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: yield 2021-07-31T02:32:34.819 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:32:34.820 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:32:34.820 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:32:34.820 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:32:34.820 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:32:34.820 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:32:34.821 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:32:34.821 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:32:34.821 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:32:34.821 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: task.result() 2021-07-31T02:32:34.821 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:32:34.822 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: cephadm 2021-07-31T02:32:32.545158+0000 mgr.x (mgr.24542) 1944 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:32:34.822 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:32:34.822 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:32:34.822 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: yield 2021-07-31T02:32:34.823 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:32:34.823 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:32:34.823 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:32:34.823 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:32:34.823 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:32:34.824 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:32:34.824 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:32:34.824 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:32:34.824 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:32:34.824 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: task.result() 2021-07-31T02:32:34.825 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:32:34.825 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: 2021-07-31T02:32:34.825 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:32:34.825 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: 2021-07-31T02:32:34.826 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:32:34.826 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:32:34.826 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:32:34.826 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:32:34.826 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:32:34.827 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:32:34.827 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:32:34.827 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:32:34.827 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:32:34.828 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:32:34.828 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:32:34.828 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:32:34.828 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:32:34.828 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:32:34.829 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:32:34.829 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:32:34.829 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:32:34.829 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:32:34.830 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: raise 2021-07-31T02:32:34.830 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:32:34.830 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:32:34.830 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:32:34.830 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:32:34.831 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:32:34.831 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: cephadm 2021-07-31T02:32:32.553614+0000 mgr.x (mgr.24542) 1945 : cephadm 2021-07-31T02:32:34.831 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: [ERR] 2021-07-31T02:32:34.831 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:32:34.831 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:32:34.832 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: yield 2021-07-31T02:32:34.832 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:32:34.832 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:32:34.832 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:32:34.833 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:32:34.833 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:32:34.833 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:32:34.833 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:32:34.834 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:32:34.834 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:32:34.834 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: task.result() 2021-07-31T02:32:34.834 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:32:34.835 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: cephadm 2021-07-31T02:32:32.554150+0000 mgr.x (mgr.24542) 1946 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:32:34.835 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:32:34.835 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:32:34.835 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: yield 2021-07-31T02:32:34.836 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:32:34.836 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:32:34.836 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:32:34.837 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:32:34.837 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:32:34.837 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:32:34.837 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:32:34.837 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:32:34.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:32:34.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:32:34.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:32:34.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:32:34.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: raise 2021-07-31T02:32:34.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:32:34.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:32:34.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:32:34.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:32:34.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:32:34.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: cephadm 2021-07-31T02:32:32.553614+0000 mgr.x ( 2021-07-31T02:32:34.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: mgr.24542) 1945 : cephadm [ERR] 2021-07-31T02:32:34.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:32:34.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:32:34.841 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: yield 2021-07-31T02:32:34.841 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:32:34.841 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:32:34.841 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:32:34.841 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:32:34.842 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:32:34.842 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:32:34.842 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:32:34.842 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:32:34.842 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:32:34.843 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: task.result() 2021-07-31T02:32:34.843 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:32:34.843 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: cephadm 2021-07-31T02:32:32.554150+0000 mgr.x (mgr.24542) 1946 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:32:34.843 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:32:34.843 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:32:34.844 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: yield 2021-07-31T02:32:34.844 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:32:34.844 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:32:34.845 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:32:34.845 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:32:34.845 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:32:34.845 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:32:34.845 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:32:34.846 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:32:34.846 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:32:34.846 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:32:34.846 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: task.result() 2021-07-31T02:32:34.847 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:32:34.847 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: 2021-07-31T02:32:34.847 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:32:34.847 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: 2021-07-31T02:32:34.847 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:32:34.848 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:32:34.848 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:32:34.848 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:32:34.848 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:32:34.848 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:32:34.849 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:32:34.849 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:32:34.849 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:32:34.849 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:32:34.849 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:32:34.850 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:32:34.850 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:32:34.850 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:32:34.850 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:32:34.851 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:32:34.851 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:32:34.851 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:32:34.851 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: raise 2021-07-31T02:32:34.851 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:32:34.852 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:32:34.852 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:32:34.852 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:32:34.852 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:32:34.852 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[39718]: cluster 2021-07-31T02:32:33.353948+0000 mon.a (mon.0) 2326 : cluster [DBG] osdmap e1363: 8 total, 8 up, 8 in 2021-07-31T02:32:34.853 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:32:34.853 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:32:34.853 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:32:34.854 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:32:34.854 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:32:34.854 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:32:34.854 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:32:34.854 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:32:34.855 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: task.result() 2021-07-31T02:32:34.855 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:32:34.855 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: 2021-07-31T02:32:34.855 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:32:34.856 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: 2021-07-31T02:32:34.856 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:32:34.856 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:32:34.856 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:32:34.856 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:32:34.857 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:32:34.857 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:32:34.857 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:32:34.857 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:32:34.857 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:32:34.858 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:32:34.858 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:32:34.858 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:32:34.858 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:32:34.858 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:32:34.859 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:32:34.859 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:32:34.859 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:32:34.859 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:32:34.859 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: raise 2021-07-31T02:32:34.860 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:32:34.860 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:32:34.860 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:32:34.860 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:32:34.860 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:32:34.861 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:34 smithi045 conmon[33655]: cluster 2021-07-31T02:32:33.353948+0000 mon.a (mon.0) 2326 : cluster [DBG] osdmap e1363: 8 total, 8 up, 8 in 2021-07-31T02:32:34.861 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 41943040 2021-07-31T02:32:34.862 INFO:tasks.workunit.client.0.smithi045.stdout:migration_execute 2021-07-31T02:32:34.962 INFO:tasks.workunit.client.0.smithi045.stdout:migration_commit 2021-07-31T02:32:35.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:35 smithi154 conmon[34553]: cluster 2021-07-31T02:32:34.356200+0000 mon.a ( 2021-07-31T02:32:35.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:35 smithi154 conmon[34553]: mon.0) 2327 : cluster [DBG] osdmap e1364: 8 total, 8 up, 8 in 2021-07-31T02:32:35.775 INFO:journalctl@ceph.osd.0.smithi045.stdout:Jul 31 02:32:35 smithi045 conmon[43243]: debug 2021-07-31T02:32:35.587+0000 7fdf6c791700 -1 osd.0 pg_epoch: 1365 pg[75.8( v 1361'34 (0'0,1361'34] local-lis/les=1308/1309 n=9 ec=1308/1308 lis/c=1308/1308 les/c/f=1309/1309/0 sis=1308) [0,2,7] r=0 lpr=1308 crt=1361'34 lcod 1359'32 mlcod 1359'32 active+clean+snaptrim trimq=[2b~1]] removing snap head 2021-07-31T02:32:35.776 INFO:journalctl@ceph.osd.3.smithi045.stdout:Jul 31 02:32:35 smithi045 conmon[54013]: debug 2021-07-31T02:32:35.422+0000 7f64fa8c9700 -1 osd.3 pg_epoch: 1365 pg[75.7( v 1361'113 (0'0,1361'113] local-lis/les=1308/1309 n=8 ec=1308/1308 lis/c=1308/1308 les/c/f=1309/1309/0 sis=1308) [3,6,1] r=0 lpr=1308 crt=1361'113 lcod 1359'111 mlcod 1359'111 active+clean+snaptrim trimq=[2b~1]] removing snap head 2021-07-31T02:32:35.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:35 smithi045 conmon[33655]: cluster 2021-07-31T02:32:34.356200 2021-07-31T02:32:35.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:35 smithi045 conmon[33655]: +0000 mon.a (mon.0) 2327 : cluster [DBG] osdmap e1364: 8 total, 8 up, 8 in 2021-07-31T02:32:35.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:35 smithi045 conmon[39718]: cluster 2021-07-31T02:32:34.356200+0000 2021-07-31T02:32:35.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:35 smithi045 conmon[39718]: mon.a (mon.0) 2327 : cluster [DBG] osdmap e1364: 8 total, 8 up, 8 in 2021-07-31T02:32:36.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:36 smithi154 conmon[34553]: cluster 2021-07-31T02:32:34.447658+0000 mgr.x (mgr.24542) 1947 : cluster [DBG] pgmap v2571: 73 pgs: 73 active+clean; 453 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 20 KiB/s rd, 4.0 MiB/s wr, 31 op/s 2021-07-31T02:32:36.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:36 smithi154 conmon[34553]: 2021-07-31T02:32:36.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:36 smithi154 conmon[34553]: cluster 2021-07-31T02:32:35.365311+0000 mon.a (mon.0) 2328 : cluster [DBG] osdmap e1365: 8 total, 8 up, 8 in 2021-07-31T02:32:36.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:36 smithi045 conmon[39718]: cluster 2021-07-31T02:32:36.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:36 smithi045 conmon[39718]: 2021-07-31T02:32:36.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:36 smithi045 conmon[39718]: 2021 2021-07-31T02:32:36.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:36 smithi045 conmon[39718]: - 2021-07-31T02:32:36.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:36 smithi045 conmon[39718]: 07 2021-07-31T02:32:36.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:36 smithi045 conmon[39718]: - 2021-07-31T02:32:36.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:36 smithi045 conmon[39718]: 31 2021-07-31T02:32:36.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:36 smithi045 conmon[39718]: T 2021-07-31T02:32:36.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:36 smithi045 conmon[39718]: 02 2021-07-31T02:32:36.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:36 smithi045 conmon[39718]: : 2021-07-31T02:32:36.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:36 smithi045 conmon[39718]: 32 2021-07-31T02:32:36.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:36 smithi045 conmon[39718]: : 2021-07-31T02:32:36.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:36 smithi045 conmon[39718]: 34 2021-07-31T02:32:36.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:36 smithi045 conmon[39718]: . 2021-07-31T02:32:36.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:36 smithi045 conmon[39718]: 447658 2021-07-31T02:32:36.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:36 smithi045 conmon[39718]: +0000 2021-07-31T02:32:36.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:36 smithi045 conmon[39718]: 2021-07-31T02:32:36.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:36 smithi045 conmon[39718]: mgr.x 2021-07-31T02:32:36.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:36 smithi045 conmon[39718]: ( 2021-07-31T02:32:36.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:36 smithi045 conmon[39718]: mgr 2021-07-31T02:32:36.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:36 smithi045 conmon[39718]: . 2021-07-31T02:32:36.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:36 smithi045 conmon[39718]: 24542 2021-07-31T02:32:36.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:36 smithi045 conmon[39718]: ) 2021-07-31T02:32:36.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:36 smithi045 conmon[39718]: 1947 2021-07-31T02:32:36.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:36 smithi045 conmon[39718]: : 2021-07-31T02:32:36.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:36 smithi045 conmon[39718]: cluster 2021-07-31T02:32:36.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:36 smithi045 conmon[39718]: 2021-07-31T02:32:36.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:36 smithi045 conmon[39718]: [DBG] 2021-07-31T02:32:36.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:36 smithi045 conmon[39718]: 2021-07-31T02:32:36.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:36 smithi045 conmon[39718]: pgmap v2571: 73 pgs: 73 active+clean; 453 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 20 KiB/s rd, 4.0 MiB/s wr, 31 op/s 2021-07-31T02:32:36.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:36 smithi045 conmon[39718]: 2021-07-31T02:32:36.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:36 smithi045 conmon[39718]: cluster 2021-07-31T02:32:36.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:36 smithi045 conmon[39718]: 2021-07-31T02:32:36.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:36 smithi045 conmon[39718]: 2021 2021-07-31T02:32:36.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:36 smithi045 conmon[39718]: - 2021-07-31T02:32:36.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:36 smithi045 conmon[39718]: 07 2021-07-31T02:32:36.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:36 smithi045 conmon[39718]: - 2021-07-31T02:32:36.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:36 smithi045 conmon[39718]: 31 2021-07-31T02:32:36.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:36 smithi045 conmon[39718]: T 2021-07-31T02:32:36.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:36 smithi045 conmon[39718]: 02 2021-07-31T02:32:36.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:36 smithi045 conmon[39718]: : 2021-07-31T02:32:36.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:36 smithi045 conmon[39718]: 32 2021-07-31T02:32:36.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:36 smithi045 conmon[39718]: : 2021-07-31T02:32:36.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:36 smithi045 conmon[39718]: 35 2021-07-31T02:32:36.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:36 smithi045 conmon[39718]: . 2021-07-31T02:32:36.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:36 smithi045 conmon[39718]: 365311 2021-07-31T02:32:36.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:36 smithi045 conmon[39718]: +0000 2021-07-31T02:32:36.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:36 smithi045 conmon[39718]: 2021-07-31T02:32:36.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:36 smithi045 conmon[39718]: mon.a 2021-07-31T02:32:36.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:36 smithi045 conmon[39718]: ( 2021-07-31T02:32:36.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:36 smithi045 conmon[39718]: mon 2021-07-31T02:32:36.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:36 smithi045 conmon[39718]: . 2021-07-31T02:32:36.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:36 smithi045 conmon[39718]: 0 2021-07-31T02:32:36.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:36 smithi045 conmon[39718]: ) 2021-07-31T02:32:36.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:36 smithi045 conmon[39718]: 2328 2021-07-31T02:32:36.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:36 smithi045 conmon[39718]: : 2021-07-31T02:32:36.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:36 smithi045 conmon[39718]: cluster 2021-07-31T02:32:36.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:36 smithi045 conmon[39718]: 2021-07-31T02:32:36.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:36 smithi045 conmon[39718]: [DBG] 2021-07-31T02:32:36.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:36 smithi045 conmon[39718]: 2021-07-31T02:32:36.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:36 smithi045 conmon[39718]: osdmap e1365: 8 total, 8 up, 8 in 2021-07-31T02:32:36.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:36 smithi045 conmon[39718]: 2021-07-31T02:32:36.790 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:36 smithi045 conmon[33655]: cluster 2021-07-31T02:32:34.447658+0000 mgr.x (mgr.24542 2021-07-31T02:32:36.791 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:36 smithi045 conmon[33655]: ) 1947 : cluster [DBG] pgmap v2571: 73 pgs: 73 active+clean; 453 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 20 KiB/s rd, 4.0 MiB/s wr, 31 op/s 2021-07-31T02:32:36.791 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:36 smithi045 conmon[33655]: cluster 2021-07-31T02:32:35.365311+0000 mon.a (mon.0) 2328 : cluster [DBG] osdmap e1365: 8 total, 8 up, 8 in 2021-07-31T02:32:37.745 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMigration.SnapsShrinkAfterPrepare (11914 ms) 2021-07-31T02:32:37.745 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMigration.SnapsShrinkToZeroBeforePrepare 2021-07-31T02:32:37.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:37 smithi154 conmon[34553]: cluster 2021-07-31T02:32:36.367451+0000 mon.a (mon.0) 2329 : cluster [DBG] 2021-07-31T02:32:37.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:37 smithi154 conmon[34553]: osdmap e1366: 8 total, 8 up, 8 in 2021-07-31T02:32:37.762 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 83886080 2021-07-31T02:32:37.766 INFO:tasks.workunit.client.0.smithi045.stdout:write: 1 0~8388609 2021-07-31T02:32:37.775 INFO:journalctl@ceph.osd.3.smithi045.stdout:Jul 31 02:32:37 smithi045 conmon[54013]: debug 2021-07-31T02:32:37.418+0000 7f64f78c3700 -1 osd.3 pg_epoch: 1367 pg[75.5( v 1367'54 (0'0,1367'54] local-lis/les=1308/1309 n=5 ec=1308/1308 lis/c=1308/1308 les/c/f=1309/1309/0 sis=1308) [3,4,1] r=0 lpr=1308 crt=1367'54 lcod 1364'53 mlcod 1364'53 active+clean+snaptrim trimq=[27~1] ps=[29~1,2b~1]] removing snap head 2021-07-31T02:32:37.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:37 smithi045 conmon[33655]: cluster 2021-07-31T02:32:36.367451+0000 mon.a (mon.0) 2329 2021-07-31T02:32:37.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:37 smithi045 conmon[33655]: : cluster [DBG] osdmap e1366: 8 total, 8 up, 8 in 2021-07-31T02:32:37.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:37 smithi045 conmon[39718]: cluster 2021-07-31T02:32:36.367451+0000 2021-07-31T02:32:37.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:37 smithi045 conmon[39718]: mon.a (mon.0) 2329 : cluster [DBG] osdmap e1366: 8 total, 8 up, 8 in 2021-07-31T02:32:37.810 INFO:tasks.workunit.client.0.smithi045.stdout:snap_create: snap1 2021-07-31T02:32:38.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:38 smithi154 conmon[34553]: cluster 2021-07-31T02:32:36.448149+0000 mgr.x (mgr.24542) 1948 : cluster [DBG] pgmap v2574: 73 pgs: 73 active+clean; 453 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 19 KiB/s rd, 4.0 MiB/s wr, 31 op/s 2021-07-31T02:32:38.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:38 smithi154 conmon[34553]: cluster 2021-07-31T02:32:37.376625+0000 mon.a (mon.0) 2330 : cluster [DBG] osdmap e1367: 8 total, 8 up, 8 in 2021-07-31T02:32:38.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:38 smithi045 conmon[33655]: cluster 2021-07-31T02:32:36.448149+0000 mgr.x (mgr.24542) 1948 2021-07-31T02:32:38.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:38 smithi045 conmon[33655]: : cluster [DBG] pgmap v2574: 73 pgs: 73 active+clean; 453 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 19 KiB/s rd, 4.0 MiB/s wr, 31 op/s 2021-07-31T02:32:38.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:38 smithi045 conmon[33655]: cluster 2021-07-31T02:32:37.376625+0000 mon.a (mon.0) 2330 : cluster [DBG] osdmap e1367: 8 total, 8 up, 8 in 2021-07-31T02:32:38.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:38 smithi045 conmon[39718]: cluster 2021-07-31T02:32:36.448149+0000 2021-07-31T02:32:38.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:38 smithi045 conmon[39718]: mgr.x (mgr.24542) 1948 : cluster [DBG] pgmap v2574: 73 pgs: 73 active+clean; 453 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 19 KiB/s rd, 4.0 MiB/s wr, 31 op/s 2021-07-31T02:32:38.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:38 smithi045 conmon[39718]: cluster 2021-07-31T02 2021-07-31T02:32:38.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:38 smithi045 conmon[39718]: :32:37.376625+0000 mon.a (mon.0) 2330 : cluster [DBG] osdmap e1367: 8 total, 8 up, 8 in 2021-07-31T02:32:39.393 INFO:tasks.workunit.client.0.smithi045.stdout:write: 1 8388609~8388609 2021-07-31T02:32:39.404 INFO:tasks.workunit.client.0.smithi045.stdout:write: X 0~1000 2021-07-31T02:32:39.406 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 1010~1000 2021-07-31T02:32:39.407 INFO:tasks.workunit.client.0.smithi045.stdout:snap_create: snap2 2021-07-31T02:32:39.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:39 smithi154 conmon[34553]: cluster 2021-07-31T02:32:38.382018+0000 mon.a (mon.0) 2021-07-31T02:32:39.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:39 smithi154 conmon[34553]: 2331 : cluster [DBG] osdmap e1368: 8 total, 8 up, 8 in 2021-07-31T02:32:39.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:39 smithi154 conmon[34553]: cluster 2021-07-31T02:32:38.448599+0000 mgr.x (mgr.24542) 1949 : cluster [DBG] pgmap v2577: 73 pgs: 73 active+clean; 453 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail 2021-07-31T02:32:39.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:39 smithi045 conmon[33655]: cluster 2021-07-31T02:32:38.382018+0000 mon.a (mon.0) 2331 : cluster [DBG] osdmap e1368: 8 total, 8 up, 8 in 2021-07-31T02:32:39.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:39 smithi045 conmon[33655]: cluster 2021-07-31T02:32:38.448599+0000 mgr.x (mgr.24542) 1949 : cluster [DBG] pgmap v2577: 73 pgs: 73 active+clean; 453 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail 2021-07-31T02:32:39.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:39 smithi045 conmon[39718]: cluster 2021-07-31T02:32:38.382018+0000 mon.a (mon.0) 2331 : cluster 2021-07-31T02:32:39.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:39 smithi045 conmon[39718]: [DBG] osdmap e1368: 8 total, 8 up, 8 in 2021-07-31T02:32:39.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:39 smithi045 conmon[39718]: cluster 2021-07-31T02:32:38.448599+0000 2021-07-31T02:32:39.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:39 smithi045 conmon[39718]: mgr.x (mgr.24542) 1949 : cluster [DBG] pgmap v2577: 73 pgs: 73 active+clean; 453 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail 2021-07-31T02:32:40.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:40 smithi154 conmon[34553]: cluster 2021-07-31T02:32:39.388742+0000 mon.a (mon.0) 2332 : cluster [DBG] osdmap e1369: 8 total, 8 up, 8 in 2021-07-31T02:32:40.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:40 smithi045 conmon[33655]: cluster 2021-07-31T02:32:39.388742+0000 mon.a (mon.0) 2332 : cluster [DBG] 2021-07-31T02:32:40.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:40 smithi045 conmon[33655]: osdmap e1369: 8 total, 8 up, 8 in 2021-07-31T02:32:40.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:40 smithi045 conmon[39718]: cluster 2021-07-31T02:32:39.388742 2021-07-31T02:32:40.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:40 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2332 : cluster [DBG] osdmap e1369: 8 total, 8 up, 8 in 2021-07-31T02:32:41.408 INFO:tasks.workunit.client.0.smithi045.stdout:write: X 8388609~1000 2021-07-31T02:32:41.410 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 16777228~1000 2021-07-31T02:32:41.410 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 167772160 2021-07-31T02:32:41.418 INFO:tasks.workunit.client.0.smithi045.stdout:write: 2 83886079~8388609 2021-07-31T02:32:41.427 INFO:tasks.workunit.client.0.smithi045.stdout:snap_create: snap3 2021-07-31T02:32:41.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:41 smithi154 conmon[34553]: cluster 2021-07-31T02:32:40.394402+0000 mon.a (mon.0) 2333 : cluster 2021-07-31T02:32:41.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:41 smithi154 conmon[34553]: [DBG] osdmap e1370: 8 total, 8 up, 8 in 2021-07-31T02:32:41.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:41 smithi154 conmon[34553]: cluster 2021-07-31T02:32:40.449063+0000 mgr.x (mgr.24542) 1950 : cluster [DBG] pgmap v2580: 73 pgs: 73 active+clean; 469 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 70 MiB/s rd, 10 MiB/s wr, 230 op/s 2021-07-31T02:32:41.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:41 smithi045 conmon[33655]: cluster 2021-07-31T02:32:40.394402+0000 mon.a (mon.0) 2333 2021-07-31T02:32:41.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:41 smithi045 conmon[33655]: : cluster [DBG] osdmap e1370: 8 total, 8 up, 8 in 2021-07-31T02:32:41.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:41 smithi045 conmon[33655]: cluster 2021-07-31T02:32:40.449063+0000 mgr.x (mgr.24542) 1950 : cluster [DBG] pgmap v2580: 73 pgs: 73 active+clean; 469 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 70 MiB/s rd, 10 MiB/s wr, 230 op/s 2021-07-31T02:32:41.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:41 smithi045 conmon[39718]: cluster 2021-07-31T02:32:40.394402+0000 mon.a (mon.0) 2333 : cluster 2021-07-31T02:32:41.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:41 smithi045 conmon[39718]: [DBG] osdmap e1370: 8 total, 8 up, 8 in 2021-07-31T02:32:41.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:41 smithi045 conmon[39718]: cluster 2021-07-31T02:32:40.449063+0000 mgr.x (mgr.24542) 1950 : cluster [DBG] pgmap v2580: 73 pgs: 73 active+clean; 469 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 70 MiB/s rd, 10 MiB/s wr, 230 op/s 2021-07-31T02:32:42.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:42 smithi154 conmon[34553]: cluster 2021-07-31T02:32:41.400316+0000 mon.a (mon.0) 2334 : cluster [DBG] osdmap e1371: 8 total, 8 up, 8 in 2021-07-31T02:32:42.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:42 smithi154 conmon[34553]: cluster 2021-07-31T02:32:42.344586+0000 mon.a (mon.0) 2335 : cluster [DBG] osdmap e1372: 8 total, 8 up, 8 in 2021-07-31T02:32:42.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:42 smithi045 conmon[39718]: cluster 2021-07-31T02:32:41.400316+0000 mon.a 2021-07-31T02:32:42.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:42 smithi045 conmon[39718]: (mon.0) 2334 : cluster [DBG] osdmap e1371: 8 total, 8 up, 8 in 2021-07-31T02:32:42.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:42 smithi045 conmon[39718]: cluster 2021-07-31T02:32:42.344586+0000 mon.a (mon 2021-07-31T02:32:42.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:42 smithi045 conmon[39718]: .0) 2335 : cluster [DBG] osdmap e1372: 8 total, 8 up, 8 in 2021-07-31T02:32:42.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:42 smithi045 conmon[33655]: cluster 2021-07-31T02:32:41.400316+0000 mon.a ( 2021-07-31T02:32:42.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:42 smithi045 conmon[33655]: mon.0) 2334 : cluster [DBG] osdmap e1371: 8 total, 8 up, 8 in 2021-07-31T02:32:42.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:42 smithi045 conmon[33655]: cluster 2021-07-31T02:32:42.344586+0000 mon.a (mon.0) 2335 : cluster [DBG] osdmap e1372: 8 total, 8 up, 8 in 2021-07-31T02:32:43.355 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 83886080 2021-07-31T02:32:43.378 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 83886079~1 2021-07-31T02:32:43.382 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 0 2021-07-31T02:32:43.400 INFO:tasks.workunit.client.0.smithi045.stdout:migration_prepare 2021-07-31T02:32:44.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:32:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:32:43.588Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:32:44.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:32:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:32:43.588Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:32:44.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:32:43 smithi045 conmon[61210]: level=error ts=2021-07-31T02:32:43.588Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:32:44.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:44 smithi154 conmon[34553]: cluster 2021-07-31T02:32:42.449568+0000 mgr.x (mgr.24542) 1951 : cluster [DBG] 2021-07-31T02:32:44.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:44 smithi154 conmon[34553]: pgmap v2583: 73 pgs: 73 active+clean; 469 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 70 MiB/s rd, 10 MiB/s wr, 230 op/s 2021-07-31T02:32:44.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:44 smithi154 conmon[34553]: cluster 2021-07-31T02:32:43.351503+0000 mon.a (mon.0) 2336 : cluster [DBG] osdmap e1373: 8 total, 8 up, 8 in 2021-07-31T02:32:44.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:44 smithi045 conmon[39718]: cluster 2021-07-31T02:32:42.449568+0000 2021-07-31T02:32:44.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:44 smithi045 conmon[39718]: mgr.x (mgr.24542) 1951 : cluster [DBG] pgmap v2583: 73 pgs: 73 active+clean; 469 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 70 MiB/s rd, 10 MiB/s wr, 230 op/s 2021-07-31T02:32:44.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:44 smithi045 conmon[39718]: cluster 2021-07-31T02:32:44.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:44 smithi045 conmon[39718]: 2021-07-31T02:32:43.351503+0000 mon.a (mon.0) 2336 : 2021-07-31T02:32:44.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:44 smithi045 conmon[39718]: cluster [DBG] osdmap e1373: 8 total, 8 up, 8 in 2021-07-31T02:32:44.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:44 smithi045 conmon[33655]: cluster 2021-07-31T02:32:42.449568+0000 2021-07-31T02:32:44.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:44 smithi045 conmon[33655]: mgr.x (mgr.24542) 1951 : cluster [DBG] pgmap v2583: 73 pgs: 73 active+clean; 469 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 70 MiB/s rd, 10 MiB/s wr, 230 op/s 2021-07-31T02:32:44.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:44 smithi045 conmon[33655]: cluster 2021-07-31 2021-07-31T02:32:44.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:44 smithi045 conmon[33655]: T02:32:43.351503+0000 mon.a (mon. 2021-07-31T02:32:44.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:44 smithi045 conmon[33655]: 0) 2336 : cluster [DBG] 2021-07-31T02:32:44.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:44 smithi045 conmon[33655]: osdmap e1373: 8 total, 8 up, 8 in 2021-07-31T02:32:45.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:45 smithi154 conmon[34553]: cluster 2021-07-31T02:32:44.404516+0000 mon.a (mon.0) 2337 : cluster 2021-07-31T02:32:45.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:45 smithi154 conmon[34553]: [DBG] osdmap e1374: 8 total, 8 up, 8 in 2021-07-31T02:32:45.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:45 smithi154 conmon[34553]: cluster 2021-07-31T02:32:44.450206+0000 mgr.x (mgr.24542) 1952 : cluster [DBG] pgmap v2586: 73 pgs: 73 active+clean; 509 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 19 KiB/s rd, 8.0 MiB/s wr, 37 op/s 2021-07-31T02:32:45.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:45 smithi045 conmon[33655]: cluster 2021-07-31T02:32:44.404516+0000 mon.a (mon.0) 2337 : 2021-07-31T02:32:45.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:45 smithi045 conmon[33655]: cluster [DBG] osdmap e1374: 8 total, 8 up, 8 in 2021-07-31T02:32:45.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:45 smithi045 conmon[33655]: cluster 2021-07-31T02:32:44.450206+0000 mgr.x (mgr.24542) 1952 : cluster [DBG] pgmap v2586: 73 pgs: 73 active+clean; 509 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 19 KiB/s rd, 8.0 MiB/s wr, 37 op/s 2021-07-31T02:32:45.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:45 smithi045 conmon[39718]: cluster 2021-07-31T02:32:44.404516+0000 mon.a (mon.0) 2337 2021-07-31T02:32:45.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:45 smithi045 conmon[39718]: : cluster [DBG] osdmap e1374: 8 total, 8 up, 8 in 2021-07-31T02:32:45.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:45 smithi045 conmon[39718]: cluster 2021-07-31T02:32:44.450206+0000 mgr.x (mgr.24542) 1952 : cluster [DBG] pgmap v2586: 73 pgs: 73 active+clean; 509 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 19 KiB/s rd, 8.0 MiB/s wr, 37 op/s 2021-07-31T02:32:46.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:46 smithi154 conmon[34553]: cluster 2021-07-31T02:32:45.409959+0000 mon.a (mon.0) 2338 : cluster [DBG] osdmap e1375: 8 total, 8 up, 8 in 2021-07-31T02:32:46.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:46 smithi154 conmon[34553]: 2021-07-31T02:32:46.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:46 smithi045 conmon[39718]: cluster 2021-07-31T02:32:45.409959+0000 mon.a (mon.0) 2338 2021-07-31T02:32:46.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:46 smithi045 conmon[39718]: : cluster [DBG] osdmap e1375: 8 total, 8 up, 8 in 2021-07-31T02:32:46.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:46 smithi045 conmon[33655]: cluster 2021-07-31T02:32:45.409959+0000 mon.a (mon.0) 2338 : cluster 2021-07-31T02:32:46.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:46 smithi045 conmon[33655]: [DBG] osdmap e1375: 8 total, 8 up, 8 in 2021-07-31T02:32:46.792 INFO:tasks.workunit.client.0.smithi045.stdout:migration_execute 2021-07-31T02:32:46.919 INFO:tasks.workunit.client.0.smithi045.stdout:migration_commit 2021-07-31T02:32:47.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:47 smithi154 conmon[34553]: cluster 2021-07-31T02:32:46.416138+0000 mon.a (mon.0 2021-07-31T02:32:47.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:47 smithi154 conmon[34553]: ) 2339 : cluster [DBG] osdmap e1376: 8 total, 8 up, 8 in 2021-07-31T02:32:47.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:47 smithi154 conmon[34553]: cluster 2021-07-31T02:32:46.450869+0000 mgr.x (mgr.24542) 1953 : cluster [DBG] pgmap v2589: 73 pgs: 73 active+clean; 509 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 19 KiB/s rd, 8.0 MiB/s wr, 37 op/s 2021-07-31T02:32:47.753 INFO:journalctl@ceph.osd.7.smithi154.stdout:Jul 31 02:32:47 smithi154 conmon[49149]: debug 2021-07-31T02:32:47.436+0000 7f8a75203700 -1 osd.7 pg_epoch: 1377 pg[75.e( v 1376'48 (0'0,1376'48] local-lis/les=1308/1309 n=9 ec=1308/1308 lis/c=1308/1308 les/c/f=1309/1309/0 sis=1308) [7,0,5] r=0 lpr=1308 crt=1376'48 lcod 1373'47 mlcod 1373'47 active+clean+snaptrim trimq=[37~1]] removing snap head 2021-07-31T02:32:47.754 INFO:journalctl@ceph.osd.5.smithi154.stdout:Jul 31 02:32:47 smithi154 conmon[42050]: debug 2021-07-31T02:32:47.427+0000 7fc9b7068700 -1 osd.5 pg_epoch: 1377 pg[75.18( v 1376'123 (0'0,1376'123] local-lis/les=1308/1309 n=10 ec=1308/1308 lis/c=1308/1308 les/c/f=1309/1309/0 sis=1308) [5,7,6] r=0 lpr=1308 crt=1376'123 lcod 1376'122 mlcod 1376'122 active+clean+snaptrim trimq=[37~1]] removing snap head 2021-07-31T02:32:47.754 INFO:journalctl@ceph.osd.5.smithi154.stdout:Jul 31 02:32:47 smithi154 conmon[42050]: debug 2021-07-31T02:32:47.435+0000 7fc9b005a700 -1 osd.5 pg_epoch: 1377 pg[75.16( v 1376'52 (0'0,1376'52] local-lis/les=1308/1309 n=8 ec=1308/1308 lis/c=1308/1308 les/c/f=1309/1309/0 sis=1308) [5,1,6] r=0 lpr=1308 crt=1376'52 lcod 1376'50 mlcod 1376'50 active+clean+snaptrim trimq=[37~1]] removing snap head 2021-07-31T02:32:47.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:47 smithi045 conmon[39718]: cluster 2021-07-31T02:32:46.416138+0000 2021-07-31T02:32:47.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:47 smithi045 conmon[39718]: mon.a (mon.0) 2339 : cluster [DBG] osdmap e1376: 8 total, 8 up, 8 in 2021-07-31T02:32:47.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:47 smithi045 conmon[39718]: cluster 2021 2021-07-31T02:32:47.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:47 smithi045 conmon[39718]: -07-31T02:32:46.450869+0000 mgr.x ( 2021-07-31T02:32:47.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:47 smithi045 conmon[39718]: mgr.24542) 1953 : cluster [DBG] pgmap v2589: 73 pgs: 73 active+clean; 509 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 19 KiB/s rd, 8.0 MiB/s wr, 37 op/s 2021-07-31T02:32:47.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:47 smithi045 conmon[33655]: cluster 2021-07-31T02:32:46.416138+0000 2021-07-31T02:32:47.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:47 smithi045 conmon[33655]: mon.a (mon.0) 2339 : cluster [DBG] osdmap e1376: 8 total, 8 up, 8 in 2021-07-31T02:32:47.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:47 smithi045 conmon[33655]: cluster 2021-07-31T02:32:46.450869+0000 mgr.x (mgr. 2021-07-31T02:32:47.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:47 smithi045 conmon[33655]: 24542) 1953 : cluster [DBG] pgmap v2589: 73 pgs: 73 active+clean; 509 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 19 KiB/s rd, 8.0 MiB/s wr, 37 op/s 2021-07-31T02:32:48.644 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:32:48.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:48 smithi154 conmon[34553]: cluster 2021-07-31T02:32:47.420613+0000 mon.a (mon.0) 2340 : cluster [DBG] osdmap e1377: 8 total, 8 up, 8 in 2021-07-31T02:32:48.775 INFO:journalctl@ceph.osd.3.smithi045.stdout:Jul 31 02:32:48 smithi045 conmon[54013]: debug 2021-07-31T02:32:48.475+0000 7f64fa8c9700 -1 osd.3 pg_epoch: 1378 pg[75.7( v 1377'125 (0'0,1377'125] local-lis/les=1308/1309 n=8 ec=1308/1308 lis/c=1308/1308 les/c/f=1309/1309/0 sis=1308) [3,6,1] r=0 lpr=1308 crt=1377'125 lcod 1373'123 mlcod 1373'123 active+clean+snaptrim trimq=[35~1] ps=[37~1]] removing snap head 2021-07-31T02:32:48.776 INFO:journalctl@ceph.osd.1.smithi045.stdout:Jul 31 02:32:48 smithi045 conmon[46771]: debug 2021-07-31T02:32:48.776 INFO:journalctl@ceph.osd.1.smithi045.stdout:Jul 31 02:32:48 smithi045 conmon[46771]: 2021-07-31T02:32:48.437+0000 7f419199e700 -1 osd.1 pg_epoch: 1378 pg[75.1c( v 1378'261 (0'0,1378'261] local-lis/les=1308/1309 n=9 ec=1308/1308 lis/c=1308/1308 les/c/f=1309/1309/0 sis=1308) [1,5,3] r=0 lpr=1308 luod=1377'260 crt=1378'261 lcod 1377'258 mlcod 1377'258 active+clean+snaptrim trimq=[35~1] ps=[37~1]] removing snap head 2021-07-31T02:32:48.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:48 smithi045 conmon[39718]: cluster 2021-07-31T02:32:47.420613 2021-07-31T02:32:48.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:48 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2340 : cluster [DBG] osdmap e1377: 8 total, 8 up, 8 in 2021-07-31T02:32:48.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:48 smithi045 conmon[33655]: cluster 2021-07-31T02:32:47.420613+0000 mon.a (mon.0) 2340 : 2021-07-31T02:32:48.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:48 smithi045 conmon[33655]: cluster [DBG] osdmap e1377: 8 total, 8 up, 8 in 2021-07-31T02:32:48.924 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:32:48.924 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 47m ago 48m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:32:48.924 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 45m ago 48m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:32:48.924 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 45m ago 51m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:32:48.925 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 47m ago 53m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:32:48.925 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 47m ago 53m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:32:48.925 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 45m ago 51m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:32:48.925 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 47m ago 52m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:32:48.926 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 47m ago 48m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:32:48.926 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 45m ago 48m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:32:48.926 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 47m ago 51m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:32:48.926 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 47m ago 51m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:32:48.927 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 47m ago 50m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:32:48.927 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 47m ago 50m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:32:48.927 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 45m ago 50m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:32:48.927 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 45m ago 49m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:32:48.928 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 45m ago 49m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:32:48.928 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 45m ago 49m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:32:48.928 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 45m ago 48m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:32:49.258 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:32:49.258 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:32:49.259 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:32:49.259 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:32:49.259 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:32:49.259 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:32:49.259 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:32:49.260 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:32:49.260 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:32:49.260 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:32:49.260 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:32:49.261 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:32:49.261 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:32:49.261 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:32:49.261 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:32:49.262 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:32:49.262 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:32:49.703 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMigration.SnapsShrinkToZeroBeforePrepare (11957 ms) 2021-07-31T02:32:49.703 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMigration.SnapsShrinkToZeroAfterPrepare 2021-07-31T02:32:49.720 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 83886080 2021-07-31T02:32:49.724 INFO:tasks.workunit.client.0.smithi045.stdout:write: 1 0~8388609 2021-07-31T02:32:49.753 INFO:journalctl@ceph.osd.7.smithi154.stdout:Jul 31 02:32:49 smithi154 conmon[49149]: debug 2021-07-31T02:32:49.456+0000 7f8a73a00700 -1 osd.7 pg_epoch: 1379 pg[75.1( v 1378'136 (0'0,1378'136] local-lis/les=1308/1309 n=11 ec=1308/1308 lis/c=1308/1308 les/c/f=1309/1309/0 sis=1308) [7,0,2] r=0 lpr=1308 crt=1378'136 lcod 1377'134 mlcod 1377'134 active+clean+snaptrim trimq=[33~1] ps=[35~1,37~1]] removing snap head 2021-07-31T02:32:49.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:49 smithi154 conmon[34553]: cluster 2021-07-31T02:32:48.425794+0000 mon.a (mon. 2021-07-31T02:32:49.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:49 smithi154 conmon[34553]: 0) 2341 : cluster [DBG] osdmap e1378: 8 total, 8 up, 8 in 2021-07-31T02:32:49.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:49 smithi154 conmon[34553]: cluster 2021-07-31T02:32:48.451293+0000 mgr.x (mgr.24542) 1954 : cluster [DBG] pgmap v2592: 73 pgs: 73 active+clean; 509 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail 2021-07-31T02:32:49.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:49 smithi154 conmon[34553]: audit 2021-07-31T02:32:49.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:49 smithi154 conmon[34553]: 2021-07-31T02:32:48.632109+0000 mgr.x (mgr.24542) 1955 : audit [DBG] from='client.17406 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:32:49.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:49 smithi154 conmon[34553]: audit 2021-07 2021-07-31T02:32:49.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:49 smithi154 conmon[34553]: -31T02:32:48.916348+0000 mgr.x (mgr.24542) 1956 : audit [DBG] from='client.17412 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:32:49.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:49 smithi154 conmon[34553]: audit 2021-07-31T 2021-07-31T02:32:49.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:49 smithi154 conmon[34553]: 02:32:49.257913+0000 mon.a (mon.0) 2342 : audit [DBG] from='client.? 172.21.15.45:0/1502631127' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:32:49.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:49 smithi154 conmon[34553]: 2021-07-31T02:32:49.775 INFO:tasks.workunit.client.0.smithi045.stdout:snap_create: snap1 2021-07-31T02:32:49.775 INFO:journalctl@ceph.osd.0.smithi045.stdout:Jul 31 02:32:49 smithi045 conmon[43243]: debug 2021-07-31T02:32:49.465+0000 7fdf6df94700 -1 osd.0 pg_epoch: 1379 pg[75.15( v 1378'98 (0'0,1378'98] local-lis/les=1308/1309 n=12 ec=1308/1308 lis/c=1308/1308 les/c/f=1309/1309/0 sis=1308) [0,7,2] r=0 lpr=1308 crt=1378'98 lcod 1377'96 mlcod 1377'96 active+clean+snaptrim trimq=[33~1] ps=[35~1,37~1]] removing snap head 2021-07-31T02:32:49.776 INFO:journalctl@ceph.osd.3.smithi045.stdout:Jul 31 02:32:49 smithi045 conmon[54013]: debug 2021-07-31T02:32:49.487+0000 7f64fa8c9700 -1 osd.3 pg_epoch: 1379 pg[75.17( v 1378'80 (0'0,1378'80] local-lis/les=1308/1309 n=12 ec=1308/1308 lis/c=1308/1308 les/c/f=1309/1309/0 sis=1308) [3,6,1] r=0 lpr=1308 crt=1378'80 lcod 1377'78 mlcod 1377'78 active+clean+snaptrim trimq=[33~1] ps=[35~1,37~1]] removing snap head 2021-07-31T02:32:49.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:49 smithi045 conmon[33655]: cluster 2021-07-31T02:32:48.425794+0000 mon.a (mon.0) 2341 : cluster [DBG] osdmap e1378: 8 total, 8 up, 8 in 2021-07-31T02:32:49.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:49 smithi045 conmon[33655]: cluster 2021-07-31T02:32:48.451293+0000 mgr.x (mgr.24542) 1954 : cluster [DBG] pgmap v2592: 73 pgs: 73 active+clean; 509 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail 2021-07-31T02:32:49.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:49 smithi045 conmon[33655]: audit 2021-07-31T02:32:48.632109+0000 mgr.x (mgr.24542) 1955 2021-07-31T02:32:49.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:49 smithi045 conmon[33655]: : audit [DBG] from='client.17406 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:32:49.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:49 smithi045 conmon[33655]: audit 2021-07-31T02:32:48.916348+0000 mgr.x (mgr.24542) 1956 : audit [DBG] from='client.17412 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:32:49.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:49 smithi045 conmon[33655]: audit 2021-07-31 2021-07-31T02:32:49.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:49 smithi045 conmon[33655]: T02:32:49.257913+0000 mon.a (mon.0) 2342 : audit [DBG] from='client.? 172.21.15.45:0/1502631127' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:32:49.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:49 smithi045 conmon[39718]: cluster 2021-07-31T02:32 2021-07-31T02:32:49.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:49 smithi045 conmon[39718]: :48.425794+0000 mon.a (mon.0) 2341 : cluster [DBG] osdmap e1378: 8 total, 8 up, 8 in 2021-07-31T02:32:49.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:49 smithi045 conmon[39718]: cluster 2021 2021-07-31T02:32:49.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:49 smithi045 conmon[39718]: -07-31T02:32:48 2021-07-31T02:32:49.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:49 smithi045 conmon[39718]: .451293+0000 mgr.x (mgr. 2021-07-31T02:32:49.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:49 smithi045 conmon[39718]: 24542) 1954 : cluster [DBG] 2021-07-31T02:32:49.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:49 smithi045 conmon[39718]: pgmap v2592: 73 pgs: 73 active+clean; 509 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail 2021-07-31T02:32:49.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:49 smithi045 conmon[39718]: audit 2021 2021-07-31T02:32:49.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:49 smithi045 conmon[39718]: -07-31T02:32:48.632109+0000 2021-07-31T02:32:49.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:49 smithi045 conmon[39718]: mgr.x (mgr.24542) 1955 : audit [DBG] from='client.17406 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:32:49.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:49 smithi045 conmon[39718]: audit 2021-07-31T02:32:49.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:49 smithi045 conmon[39718]: 2021-07-31T02:32:48 2021-07-31T02:32:49.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:49 smithi045 conmon[39718]: .916348+0000 mgr.x (mgr.24542) 2021-07-31T02:32:49.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:49 smithi045 conmon[39718]: 1956 : audit [DBG] from='client.17412 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:32:49.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:49 smithi045 conmon[39718]: audit 2021-07-31T02:32:49.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:49 smithi045 conmon[39718]: 2021-07-31T02:32: 2021-07-31T02:32:49.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:49 smithi045 conmon[39718]: 49.257913+0000 mon.a (mon 2021-07-31T02:32:49.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:49 smithi045 conmon[39718]: .0) 2342 : audit [DBG] 2021-07-31T02:32:49.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:49 smithi045 conmon[39718]: from='client.? 172.21.15.45:0/1502631127' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:32:50.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:50 smithi154 conmon[34553]: cluster 2021-07-31T02:32:49.431044+0000 mon.a (mon.0) 2343 : cluster [DBG] osdmap e1379: 8 total, 8 up, 8 in 2021-07-31T02:32:50.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:50 smithi045 conmon[33655]: cluster 2021-07-31T02:32:49.431044+0000 mon.a (mon.0) 2343 2021-07-31T02:32:50.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:50 smithi045 conmon[33655]: : cluster [DBG] osdmap e1379: 8 total, 8 up, 8 in 2021-07-31T02:32:50.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:50 smithi045 conmon[39718]: cluster 2021-07-31T02:32:49.431044+0000 mon.a (mon. 2021-07-31T02:32:50.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:50 smithi045 conmon[39718]: 0) 2343 : cluster [DBG] osdmap e1379: 8 total, 8 up, 8 in 2021-07-31T02:32:51.449 INFO:tasks.workunit.client.0.smithi045.stdout:write: 1 8388609~8388609 2021-07-31T02:32:51.462 INFO:tasks.workunit.client.0.smithi045.stdout:write: X 0~1000 2021-07-31T02:32:51.463 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 1010~1000 2021-07-31T02:32:51.463 INFO:tasks.workunit.client.0.smithi045.stdout:snap_create: snap2 2021-07-31T02:32:51.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:51 smithi154 conmon[34553]: cluster 2021-07-31T02:32:50.437388+0000 mon.a (mon.0) 2344 : cluster [DBG] osdmap e1380: 8 total, 8 up, 8 in 2021-07-31T02:32:51.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:51 smithi154 conmon[34553]: cluster 2021-07-31T02:32:50.451669+0000 mgr.x (mgr.24542) 1957 : cluster [DBG] pgmap v2595: 73 pgs: 73 active+clean; 517 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 30 MiB/s rd, 6.0 MiB/s wr, 198 op/s 2021-07-31T02:32:51.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:51 smithi045 conmon[33655]: cluster 2021-07-31T02:32:50.437388+0000 2021-07-31T02:32:51.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:51 smithi045 conmon[33655]: mon.a (mon.0) 2344 : cluster [DBG] osdmap e1380: 8 total, 8 up, 8 in 2021-07-31T02:32:51.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:51 smithi045 conmon[33655]: cluster 2021-07-31T02:32:50.451669+0000 mgr.x (mgr.24542) 1957 : cluster 2021-07-31T02:32:51.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:51 smithi045 conmon[33655]: [DBG] pgmap v2595: 73 pgs: 73 active+clean; 517 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 30 MiB/s rd, 6.0 MiB/s wr, 198 op/s 2021-07-31T02:32:51.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:51 smithi045 conmon[39718]: cluster 2021-07-31T02:32:50.437388+0000 2021-07-31T02:32:51.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:51 smithi045 conmon[39718]: mon.a (mon.0) 2344 : cluster [DBG] osdmap e1380: 8 total, 8 up, 8 in 2021-07-31T02:32:51.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:51 smithi045 conmon[39718]: cluster 2021-07-31 2021-07-31T02:32:51.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:51 smithi045 conmon[39718]: T02:32:50.451669+0000 mgr.x (mgr.24542) 1957 : cluster [DBG] pgmap v2595: 73 pgs: 73 active+clean; 517 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 30 MiB/s rd, 6.0 MiB/s wr, 198 op/s 2021-07-31T02:32:52.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:52 smithi154 conmon[34553]: cluster 2021-07-31T02:32:51.443314+0000 mon.a (mon.0) 2345 : cluster [DBG] osdmap e1381: 8 total, 8 up, 8 in 2021-07-31T02:32:52.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:52 smithi154 conmon[34553]: 2021-07-31T02:32:52.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:52 smithi154 conmon[34553]: cluster 2021-07-31T02:32:52.347146+0000 mon.a (mon.0) 2346 : cluster [DBG] osdmap e1382: 8 total, 8 up, 8 in 2021-07-31T02:32:52.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:52 smithi045 conmon[33655]: cluster 2021-07-31T02:32:51.443314+0000 mon.a (mon.0) 2345 2021-07-31T02:32:52.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:52 smithi045 conmon[33655]: : cluster [DBG] osdmap e1381: 8 total, 8 up, 8 in 2021-07-31T02:32:52.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:52 smithi045 conmon[33655]: cluster 2021-07-31T02:32:52.347146+0000 mon.a (mon.0) 2346 : cluster [DBG] osdmap e1382: 8 total, 8 up, 8 in 2021-07-31T02:32:52.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:52 smithi045 conmon[39718]: cluster 2021-07-31T02:32:51.443314+0000 2021-07-31T02:32:52.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:52 smithi045 conmon[39718]: mon.a (mon.0) 2345 : cluster [DBG] osdmap e1381: 8 total, 8 up, 8 in 2021-07-31T02:32:52.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:52 smithi045 conmon[39718]: cluster 2021 2021-07-31T02:32:52.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:52 smithi045 conmon[39718]: -07-31T02:32:52.347146+0000 mon.a (mon.0) 2346 : cluster [DBG] osdmap e1382: 8 total, 8 up, 8 in 2021-07-31T02:32:53.360 INFO:tasks.workunit.client.0.smithi045.stdout:write: X 8388609~1000 2021-07-31T02:32:53.365 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 16777228~1000 2021-07-31T02:32:53.365 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 167772160 2021-07-31T02:32:53.372 INFO:tasks.workunit.client.0.smithi045.stdout:write: 2 83886079~8388609 2021-07-31T02:32:53.380 INFO:tasks.workunit.client.0.smithi045.stdout:snap_create: snap3 2021-07-31T02:32:54.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:32:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:32:53.588Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:32:54.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:32:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:32:53.588Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:32:54.026 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:32:53 smithi045 conmon[61210]: level=error ts=2021-07-31T02:32:53.588Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs; Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" 2021-07-31T02:32:54.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:54 smithi154 conmon[34553]: cluster 2021-07-31T02:32:52.452240+0000 mgr.x (mgr.24542) 1958 : cluster [DBG] pgmap v2598: 73 pgs: 73 active+clean; 517 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 30 MiB/s rd, 6.0 MiB/s wr, 198 op/s 2021-07-31T02:32:54.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:54 smithi154 conmon[34553]: cluster 2021-07-31T02:32:53. 2021-07-31T02:32:54.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:54 smithi154 conmon[34553]: 354977+0000 mon.a (mon.0) 2347 : cluster [DBG] osdmap e1383: 8 total, 8 up, 8 in 2021-07-31T02:32:54.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:54 smithi045 conmon[33655]: cluster 2021-07-31T02:32:52.452240+0000 mgr.x (mgr.24542) 1958 2021-07-31T02:32:54.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:54 smithi045 conmon[33655]: : cluster [DBG] pgmap v2598: 73 pgs: 73 active+clean; 517 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 30 MiB/s rd, 6.0 MiB/s wr, 198 op/s 2021-07-31T02:32:54.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:54 smithi045 conmon[33655]: cluster 2021-07-31T02:32:53.354977+0000 mon.a (mon.0) 2347 : cluster [DBG] osdmap e1383: 8 total, 8 up, 8 in 2021-07-31T02:32:54.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:54 smithi045 conmon[39718]: cluster 2021-07-31T02:32:52.452240+0000 mgr.x (mgr. 2021-07-31T02:32:54.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:54 smithi045 conmon[39718]: 24542) 1958 : cluster [DBG] pgmap v2598: 73 pgs: 73 active+clean; 517 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 30 MiB/s rd, 6.0 MiB/s wr, 198 op/s 2021-07-31T02:32:54.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:54 smithi045 conmon[39718]: cluster 2021-07-31T02:32:53.354977+0000 mon.a (mon 2021-07-31T02:32:54.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:54 smithi045 conmon[39718]: .0) 2347 : cluster [DBG] osdmap e1383: 8 total, 8 up, 8 in 2021-07-31T02:32:55.372 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 83886080 2021-07-31T02:32:55.393 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 83886079~1 2021-07-31T02:32:55.396 INFO:tasks.workunit.client.0.smithi045.stdout:migration_prepare 2021-07-31T02:32:55.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:55 smithi154 conmon[34553]: cluster 2021-07-31T02:32:54.360887+0000 mon.a (mon.0) 2348 : cluster [DBG] osdmap e1384: 8 total, 8 up, 8 in 2021-07-31T02:32:55.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:55 smithi045 conmon[39718]: cluster 2021-07-31T02:32:54. 2021-07-31T02:32:55.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:55 smithi045 conmon[39718]: 360887+0000 mon.a (mon.0) 2348 : cluster [DBG] osdmap e1384: 8 total, 8 up, 8 in 2021-07-31T02:32:55.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:55 smithi045 conmon[33655]: cluster 2021-07-31T02:32:54.360887+0000 mon.a (mon.0) 2348 : cluster 2021-07-31T02:32:55.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:55 smithi045 conmon[33655]: [DBG] osdmap e1384: 8 total, 8 up, 8 in 2021-07-31T02:32:56.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:56 smithi154 conmon[34553]: cluster 2021-07-31T02:32:54.452702+0000 mgr.x (mgr.24542) 1959 : cluster [DBG] 2021-07-31T02:32:56.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:56 smithi154 conmon[34553]: pgmap v2601: 73 pgs: 73 active+clean; 565 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 24 MiB/s rd, 12 MiB/s wr, 40 op/s 2021-07-31T02:32:56.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:56 smithi154 conmon[34553]: cluster 2021-07-31T02:32:55.366624+0000 mon.a (mon.0) 2349 : cluster [DBG] osdmap e1385: 8 total, 8 up, 8 in 2021-07-31T02:32:56.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:56 smithi045 conmon[39718]: cluster 2021-07-31T02:32:54.452702+0000 mgr.x ( 2021-07-31T02:32:56.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:56 smithi045 conmon[39718]: mgr.24542) 1959 : cluster [DBG] pgmap v2601: 73 pgs: 73 active+clean; 565 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 24 MiB/s rd, 12 MiB/s wr, 40 op/s 2021-07-31T02:32:56.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:56 smithi045 conmon[39718]: cluster 2021-07-31T02:32:55.366624+0000 2021-07-31T02:32:56.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:56 smithi045 conmon[39718]: mon.a (mon.0) 2349 : cluster [DBG] osdmap e1385: 8 total, 8 up, 8 in 2021-07-31T02:32:56.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:56 smithi045 conmon[33655]: cluster 2021-07-31T02:32: 2021-07-31T02:32:56.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:56 smithi045 conmon[33655]: 54.452702+0000 mgr.x (mgr.24542) 1959 : cluster [DBG] pgmap v2601: 73 pgs: 73 active+clean; 565 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 24 MiB/s rd, 12 MiB/s wr, 40 op/s 2021-07-31T02:32:56.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:56 smithi045 conmon[33655]: cluster 2021-07-31T02:32:55.366624+0000 mon.a (mon.0) 2349 : 2021-07-31T02:32:56.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:56 smithi045 conmon[33655]: cluster [DBG] osdmap e1385: 8 total, 8 up, 8 in 2021-07-31T02:32:57.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:57 smithi154 conmon[34553]: cluster 2021-07-31T02:32:56.371923+0000 mon.a (mon.0) 2350 : cluster [DBG] 2021-07-31T02:32:57.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:57 smithi154 conmon[34553]: osdmap e1386: 8 total, 8 up, 8 in 2021-07-31T02:32:57.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:57 smithi154 conmon[34553]: cluster 2021-07-31T02:32:57.349030+0000 mon.a (mon.0) 2351 : cluster [DBG] osdmap e1387: 8 total, 8 up, 8 in 2021-07-31T02:32:57.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:57 smithi045 conmon[39718]: cluster 2021-07-31T02:32:56.371923+0000 mon.a (mon 2021-07-31T02:32:57.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:57 smithi045 conmon[39718]: .0) 2350 : cluster [DBG] osdmap e1386: 8 total, 8 up, 8 in 2021-07-31T02:32:57.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:57 smithi045 conmon[39718]: cluster 2021-07-31T02:32:57.349030+0000 mon.a (mon.0) 2351 : cluster 2021-07-31T02:32:57.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:57 smithi045 conmon[39718]: [DBG] osdmap e1387: 8 total, 8 up, 8 in 2021-07-31T02:32:57.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:57 smithi045 conmon[33655]: cluster 2021-07-31T02:32:56.371923+0000 mon.a (mon.0) 2350 : cluster 2021-07-31T02:32:57.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:57 smithi045 conmon[33655]: [DBG] osdmap e1386: 8 total, 8 up, 8 in 2021-07-31T02:32:57.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:57 smithi045 conmon[33655]: cluster 2021-07-31T02:32:57.349030+0000 mon.a (mon.0) 2351 : cluster [DBG] osdmap e1387: 8 total, 8 up, 8 in 2021-07-31T02:32:58.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:58 smithi154 conmon[34553]: cluster 2021-07-31T02:32:56.453187+0000 mgr.x (mgr.24542) 1960 : cluster 2021-07-31T02:32:58.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:58 smithi154 conmon[34553]: [DBG] pgmap v2604: 73 pgs: 73 active+clean; 565 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 24 MiB/s rd, 12 MiB/s wr, 40 op/s 2021-07-31T02:32:58.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:32:58 smithi154 conmon[34553]: cluster 2021-07-31T02:32:58.355739+0000 mon.a (mon.0) 2352 : cluster [DBG] osdmap e1388: 8 total, 8 up, 8 in 2021-07-31T02:32:58.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:58 smithi045 conmon[39718]: cluster 2021-07-31T02:32:56.453187+0000 mgr.x (mgr.24542) 2021-07-31T02:32:58.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:58 smithi045 conmon[39718]: 1960 : cluster [DBG] pgmap v2604: 73 pgs: 73 active+clean; 565 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 24 MiB/s rd, 12 MiB/s wr, 40 op/s 2021-07-31T02:32:58.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:58 smithi045 conmon[39718]: cluster 2021-07-31T02:32:58.355739+0000 mon.a (mon.0) 2352 : cluster 2021-07-31T02:32:58.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:32:58 smithi045 conmon[39718]: [DBG] osdmap e1388: 8 total, 8 up, 8 in 2021-07-31T02:32:58.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:58 smithi045 conmon[33655]: cluster 2021-07-31T02:32:56.453187+0000 2021-07-31T02:32:58.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:58 smithi045 conmon[33655]: mgr.x (mgr.24542) 1960 : cluster [DBG] pgmap v2604: 73 pgs: 73 active+clean; 565 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 24 MiB/s rd, 12 MiB/s wr, 40 op/s 2021-07-31T02:32:58.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:58 smithi045 conmon[33655]: cluster 2021-07-31T02:32:58.355739+0000 mon.a 2021-07-31T02:32:58.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:32:58 smithi045 conmon[33655]: (mon.0) 2352 : cluster [DBG] osdmap e1388: 8 total, 8 up, 8 in 2021-07-31T02:32:58.851 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 0 2021-07-31T02:32:58.913 INFO:tasks.workunit.client.0.smithi045.stdout:migration_execute 2021-07-31T02:32:59.015 INFO:tasks.workunit.client.0.smithi045.stdout:migration_commit 2021-07-31T02:32:59.752 INFO:journalctl@ceph.osd.6.smithi154.stdout:Jul 31 02:32:59 smithi154 conmon[45576]: debug 2021-07-31T02:32:59.390+0000 7f47c10a5700 -1 osd.6 pg_epoch: 1389 pg[75.a( v 1388'102 (0'0,1388'102] local-lis/les=1308/1309 n=9 ec=1308/1308 lis/c=1308/1308 les/c/f=1309/1309/0 sis=1308) [6,7,4] r=0 lpr=1308 crt=1388'100 lcod 1388'101 mlcod 1388'101 active+clean+snaptrim trimq=[43~1]] removing snap head 2021-07-31T02:32:59.775 INFO:journalctl@ceph.osd.1.smithi045.stdout:Jul 31 02:32:59 smithi045 conmon[46771]: debug 2021-07-31T02:32:59.435+0000 7f41961a7700 -1 osd.1 pg_epoch: 1389 pg[75.b( v 1388'88 (0'0,1388'88] local-lis/les=1308/1309 n=11 ec=1308/1308 lis/c=1308/1308 les/c/f=1309/1309/0 sis=1308) [1,4,2] r=0 lpr=1308 crt=1388'88 lcod 1388'87 mlcod 1388'87 active+clean+snaptrim trimq=[43~1]] removing snap head 2021-07-31T02:33:00.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:00 smithi154 conmon[34553]: cluster 2021-07-31T02:32:58.453621+0000 mgr.x (mgr.24542) 1961 : cluster [DBG] pgmap v2607: 73 pgs: 73 active+clean; 565 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail 2021-07-31T02:33:00.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:00 smithi154 conmon[34553]: cluster 2021-07-31T02:32:59.383280+0000 mon.a (mon.0) 2353 : cluster [DBG] osdmap e1389: 8 total, 8 up, 8 in 2021-07-31T02:33:00.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:00 smithi045 conmon[33655]: cluster 2021-07-31T02:32:58.453621+0000 mgr.x (mgr.24542) 1961 : cluster 2021-07-31T02:33:00.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:00 smithi045 conmon[33655]: [DBG] pgmap v2607: 73 pgs: 73 active+clean; 565 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail 2021-07-31T02:33:00.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:00 smithi045 conmon[33655]: cluster 2021-07-31T02:32:59.383280+0000 mon.a (mon.0) 2353 : cluster [DBG] osdmap e1389: 8 total, 8 up, 8 in 2021-07-31T02:33:00.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:00 smithi045 conmon[39718]: cluster 2021-07-31T02:32:58.453621+0000 mgr.x (mgr. 2021-07-31T02:33:00.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:00 smithi045 conmon[39718]: 24542) 1961 : cluster [DBG] pgmap v2607: 73 pgs: 73 active+clean; 565 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail 2021-07-31T02:33:00.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:00 smithi045 conmon[39718]: cluster 2021-07-31T02:32:59.383280+0000 mon.a (mon.0) 2021-07-31T02:33:00.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:00 smithi045 conmon[39718]: 2353 : cluster [DBG] osdmap e1389: 8 total, 8 up, 8 in 2021-07-31T02:33:01.704 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMigration.SnapsShrinkToZeroAfterPrepare (12001 ms) 2021-07-31T02:33:01.704 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMigration.SnapsExpandAfterPrepare 2021-07-31T02:33:01.720 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 83886080 2021-07-31T02:33:01.723 INFO:tasks.workunit.client.0.smithi045.stdout:write: 1 0~8388609 2021-07-31T02:33:01.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:01 smithi154 conmon[34553]: cluster 2021-07-31T02:33:01.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:01 smithi154 conmon[34553]: 2021-07- 2021-07-31T02:33:01.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:01 smithi154 conmon[34553]: 31T02:33:00.390021+0000 mon.a (mon.0) 2354 : cluster [DBG] osdmap e1390: 8 total, 8 up, 8 in 2021-07-31T02:33:01.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:01 smithi154 conmon[34553]: cluster 2021-07-31T02:33:01.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:01 smithi154 conmon[34553]: 2021-07-31T02:33:00.454161+0000 mgr.x (mgr 2021-07-31T02:33:01.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:01 smithi154 conmon[34553]: .24542) 1962 : cluster [DBG] pgmap v2610: 73 pgs: 73 active+clean; 565 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 32 MiB/s rd, 6.7 KiB/s wr, 72 op/s 2021-07-31T02:33:01.766 INFO:tasks.workunit.client.0.smithi045.stdout:snap_create: snap1 2021-07-31T02:33:01.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:01 smithi045 conmon[33655]: cluster 2021-07-31T02:33:00.390021+0000 mon.a (mon.0) 2354 : 2021-07-31T02:33:01.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:01 smithi045 conmon[33655]: cluster [DBG] osdmap e1390: 8 total, 8 up, 8 in 2021-07-31T02:33:01.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:01 smithi045 conmon[33655]: cluster 2021-07-31T02:33:00.454161+0000 mgr.x (mgr.24542) 1962 : cluster [DBG] pgmap v2610: 73 pgs: 73 active+clean; 565 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 32 MiB/s rd, 6.7 KiB/s wr, 72 op/s 2021-07-31T02:33:01.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:01 smithi045 conmon[39718]: cluster 2021-07-31T02:33:00.390021 2021-07-31T02:33:01.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:01 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2354 : cluster [DBG] osdmap e1390: 8 total, 8 up, 8 in 2021-07-31T02:33:01.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:01 smithi045 conmon[39718]: cluster 2021-07-31T02:33:00.454161+0000 mgr.x 2021-07-31T02:33:01.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:01 smithi045 conmon[39718]: (mgr.24542) 1962 : cluster [DBG] pgmap v2610: 73 pgs: 73 active+clean; 565 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 32 MiB/s rd, 6.7 KiB/s wr, 72 op/s 2021-07-31T02:33:02.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:02 smithi154 conmon[34553]: cluster 2021-07-31T02:33:01.396040+0000 mon.a (mon.0) 2355 : cluster [DBG] osdmap e1391: 8 total, 8 up, 8 in 2021-07-31T02:33:02.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:02 smithi154 conmon[34553]: 2021-07-31T02:33:02.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:02 smithi045 conmon[39718]: cluster 2021-07-31T02:33:01.396040+0000 mon.a (mon. 2021-07-31T02:33:02.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:02 smithi045 conmon[39718]: 0) 2355 : cluster [DBG] osdmap e1391: 8 total, 8 up, 8 in 2021-07-31T02:33:02.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:02 smithi045 conmon[33655]: cluster 2021-07-31T02:33:01.396040+0000 mon.a (mon.0) 2355 : cluster [DBG] osdmap e1391: 8 total, 8 up, 8 in 2021-07-31T02:33:03.413 INFO:tasks.workunit.client.0.smithi045.stdout:write: 1 8388609~8388609 2021-07-31T02:33:03.424 INFO:tasks.workunit.client.0.smithi045.stdout:write: X 0~1000 2021-07-31T02:33:03.424 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 1010~1000 2021-07-31T02:33:03.425 INFO:tasks.workunit.client.0.smithi045.stdout:snap_create: snap2 2021-07-31T02:33:03.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:03 smithi154 conmon[34553]: cluster 2021-07-31T02:33:02.401135+0000 mon.a (mon.0) 2356 : cluster 2021-07-31T02:33:03.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:03 smithi154 conmon[34553]: [DBG] osdmap e1392: 8 total, 8 up, 8 in 2021-07-31T02:33:03.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:03 smithi154 conmon[34553]: cluster 2021-07-31T02:33:02.454625+0000 mgr.x (mgr.24542) 1963 : cluster [DBG] pgmap v2613: 73 pgs: 73 active+clean; 565 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 32 MiB/s rd, 6.7 KiB/s wr, 72 op/s 2021-07-31T02:33:03.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:03 smithi045 conmon[33655]: cluster 2021-07-31T02:33:02.401135+0000 mon.a (mon.0 2021-07-31T02:33:03.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:03 smithi045 conmon[33655]: ) 2356 : cluster [DBG] osdmap e1392: 8 total, 8 up, 8 in 2021-07-31T02:33:03.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:03 smithi045 conmon[33655]: cluster 2021-07-31T02:33:02.454625+0000 mgr.x (mgr.24542) 1963 : cluster [DBG] pgmap v2613: 73 pgs: 73 active+clean; 565 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 32 MiB/s rd, 6.7 KiB/s wr, 72 op/s 2021-07-31T02:33:03.776 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:33:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:33:03.589Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:33:03.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:33:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:33:03.589Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:33:03.777 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:33:03 smithi045 conmon[61210]: level=error ts=2021-07-31T02:33:03.589Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:33:03.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:03 smithi045 conmon[39718]: cluster 2021-07-31T02:33:02.401135+0000 2021-07-31T02:33:03.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:03 smithi045 conmon[39718]: mon.a (mon.0) 2356 : cluster [DBG] osdmap e1392: 8 total, 8 up, 8 in 2021-07-31T02:33:03.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:03 smithi045 conmon[39718]: cluster 2021-07-31T02:33:02.454625+0000 mgr.x (mgr. 2021-07-31T02:33:03.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:03 smithi045 conmon[39718]: 24542) 1963 : cluster [DBG] pgmap v2613: 73 pgs: 73 active+clean; 565 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 32 MiB/s rd, 6.7 KiB/s wr, 72 op/s 2021-07-31T02:33:04.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:04 smithi154 conmon[34553]: cluster 2021-07-31T02:33:03.407972+0000 mon.a (mon.0) 2357 : cluster [DBG] osdmap e1393: 8 total, 8 up, 8 in 2021-07-31T02:33:04.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:04 smithi154 conmon[34553]: 2021-07-31T02:33:04.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:04 smithi045 conmon[33655]: cluster 2021-07-31T02:33:03.407972+0000 mon.a (mon.0) 2357 : cluster [DBG] osdmap e1393: 8 total, 8 up, 8 in 2021-07-31T02:33:04.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:04 smithi045 conmon[39718]: cluster 2021-07-31T02:33:03.407972+0000 mon.a (mon.0) 2021-07-31T02:33:04.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:04 smithi045 conmon[39718]: 2357 : cluster [DBG] osdmap e1393: 8 total, 8 up, 8 in 2021-07-31T02:33:05.428 INFO:tasks.workunit.client.0.smithi045.stdout:write: X 8388609~1000 2021-07-31T02:33:05.431 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 16777228~1000 2021-07-31T02:33:05.431 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 167772160 2021-07-31T02:33:05.438 INFO:tasks.workunit.client.0.smithi045.stdout:write: 2 83886079~8388609 2021-07-31T02:33:05.446 INFO:tasks.workunit.client.0.smithi045.stdout:snap_create: snap3 2021-07-31T02:33:05.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:05 smithi154 conmon[34553]: cluster 2021-07-31T02:33:05.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:05 smithi154 conmon[34553]: 2021-07-31T02:33:04.409147+0000 2021-07-31T02:33:05.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:05 smithi154 conmon[34553]: mon.a (mon.0) 2358 : cluster [DBG] osdmap e1394: 8 total, 8 up, 8 in 2021-07-31T02:33:05.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:05 smithi154 conmon[34553]: cluster 2021-07-31T02:33:05.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:05 smithi154 conmon[34553]: 2021-07-31T02:33:04.455329+0000 mgr.x (mgr. 2021-07-31T02:33:05.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:05 smithi154 conmon[34553]: 24542) 1964 : cluster [DBG] pgmap v2616: 73 pgs: 73 active+clean; 597 MiB data, 4.7 GiB used, 710 GiB / 715 GiB avail; 34 MiB/s rd, 14 MiB/s wr, 194 op/s 2021-07-31T02:33:05.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:05 smithi045 conmon[33655]: cluster 2021-07-31T02:33:04.409147+0000 mon.a (mon.0) 2358 : cluster 2021-07-31T02:33:05.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:05 smithi045 conmon[33655]: [DBG] osdmap e1394: 8 total, 8 up, 8 in 2021-07-31T02:33:05.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:05 smithi045 conmon[33655]: cluster 2021-07-31T02:33:04.455329+0000 mgr.x (mgr.24542) 1964 : cluster [DBG] pgmap v2616: 73 pgs: 73 active+clean; 597 MiB data, 4.7 GiB used, 710 GiB / 715 GiB avail; 34 MiB/s rd, 14 MiB/s wr, 194 op/s 2021-07-31T02:33:05.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:05 smithi045 conmon[39718]: cluster 2021-07-31T02:33:04.409147+0000 mon.a (mon. 2021-07-31T02:33:05.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:05 smithi045 conmon[39718]: 0) 2358 : cluster [DBG] osdmap e1394: 8 total, 8 up, 8 in 2021-07-31T02:33:05.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:05 smithi045 conmon[39718]: cluster 2021-07-31T02:33:04.455329+0000 mgr.x (mgr 2021-07-31T02:33:05.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:05 smithi045 conmon[39718]: .24542) 1964 : cluster [DBG] pgmap v2616: 73 pgs: 73 active+clean; 597 MiB data, 4.7 GiB used, 710 GiB / 715 GiB avail; 34 MiB/s rd, 14 MiB/s wr, 194 op/s 2021-07-31T02:33:06.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:06 smithi154 conmon[34553]: cluster 2021-07-31T02:33:05.421644+0000 mon.a (mon.0) 2359 : cluster [DBG] osdmap e1395: 8 total, 8 up, 8 in 2021-07-31T02:33:06.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:06 smithi154 conmon[34553]: 2021-07-31T02:33:06.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:06 smithi045 conmon[33655]: cluster 2021-07-31T02:33:05.421644+0000 mon.a (mon.0) 2021-07-31T02:33:06.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:06 smithi045 conmon[33655]: 2359 : cluster [DBG] osdmap e1395: 8 total, 8 up, 8 in 2021-07-31T02:33:06.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:06 smithi045 conmon[39718]: cluster 2021-07-31T02:33:05.421644+0000 mon.a ( 2021-07-31T02:33:06.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:06 smithi045 conmon[39718]: mon.0) 2359 : cluster [DBG] osdmap e1395: 8 total, 8 up, 8 in 2021-07-31T02:33:07.369 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 83886080 2021-07-31T02:33:07.399 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 83886079~1 2021-07-31T02:33:07.402 INFO:tasks.workunit.client.0.smithi045.stdout:migration_prepare 2021-07-31T02:33:07.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:07 smithi154 conmon[34553]: cluster 2021-07-31T02:33:06.428351+0000 mon.a (mon.0) 2360 : cluster [DBG] osdmap e1396: 8 total, 8 up, 8 in 2021-07-31T02:33:07.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:07 smithi154 conmon[34553]: cluster 2021-07-31T02:33:07.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:07 smithi154 conmon[34553]: 2021-07-31T02:33:06.456005+0000 mgr.x (mgr.24542) 1965 : cluster [DBG] pgmap v2619: 73 pgs: 73 active+clean; 597 MiB data, 4.7 GiB used, 710 GiB / 715 GiB avail; 34 MiB/s rd, 14 MiB/s wr, 194 op/s 2021-07-31T02:33:07.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:07 smithi154 conmon[34553]: cluster 2021-07-31T02:33:07.351329+0000 mon.a (mon.0) 2361 : cluster [DBG] osdmap e1397: 8 total, 8 up, 8 in 2021-07-31T02:33:07.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:07 smithi045 conmon[39718]: cluster 2021-07-31T02:33:06.428351+0000 mon.a (mon.0) 2360 : cluster [DBG] osdmap e1396: 8 total, 8 up, 8 in 2021-07-31T02:33:07.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:07 smithi045 conmon[39718]: cluster 2021-07-31T02:33:06.456005+0000 mgr.x (mgr.24542) 1965 : cluster [DBG] pgmap v2619: 73 pgs: 73 active+clean; 597 MiB data, 4.7 GiB used, 710 GiB / 715 GiB avail; 34 MiB/s rd, 14 MiB/s wr, 194 op/s 2021-07-31T02:33:07.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:07 smithi045 conmon[39718]: cluster 2021-07-31T02:33: 2021-07-31T02:33:07.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:07 smithi045 conmon[39718]: 07.351329+0000 mon.a (mon.0) 2361 2021-07-31T02:33:07.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:07 smithi045 conmon[39718]: : cluster [DBG] osdmap e1397: 8 total, 8 up, 8 in 2021-07-31T02:33:07.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:07 smithi045 conmon[33655]: cluster 2021-07-31T02:33:06.428351+0000 mon.a (mon 2021-07-31T02:33:07.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:07 smithi045 conmon[33655]: .0) 2360 : cluster [DBG] osdmap e1396: 8 total, 8 up, 8 in 2021-07-31T02:33:07.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:07 smithi045 conmon[33655]: cluster 2021-07-31T02:33:06.456005+0000 mgr.x (mgr.24542) 1965 : cluster [DBG] pgmap v2619: 73 pgs: 73 active+clean; 597 MiB data, 4.7 GiB used, 710 GiB / 715 GiB avail; 34 MiB/s rd, 14 MiB/s wr, 194 op/s 2021-07-31T02:33:07.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:07 smithi045 conmon[33655]: cluster 2021-07-31T 2021-07-31T02:33:07.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:07 smithi045 conmon[33655]: 02:33:07.351329+0000 2021-07-31T02:33:07.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:07 smithi045 conmon[33655]: mon.a (mon.0) 2361 : cluster 2021-07-31T02:33:07.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:07 smithi045 conmon[33655]: [DBG] osdmap e1397: 8 total, 8 up, 8 in 2021-07-31T02:33:09.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:09 smithi154 conmon[34553]: cluster 2021-07-31T02:33:08.431146+0000 mon.a (mon.0) 2362 2021-07-31T02:33:09.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:09 smithi154 conmon[34553]: : cluster [DBG] osdmap e1398: 8 total, 8 up, 8 in 2021-07-31T02:33:09.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:09 smithi154 conmon[34553]: cluster 2021-07-31T02:33:08.456444+0000 mgr.x (mgr.24542) 1966 : cluster [DBG] pgmap v2622: 73 pgs: 73 active+clean; 597 MiB data, 4.7 GiB used, 710 GiB / 715 GiB avail 2021-07-31T02:33:09.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:09 smithi045 conmon[39718]: cluster 2021-07-31T02:33:08.431146+0000 2021-07-31T02:33:09.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:09 smithi045 conmon[39718]: mon.a (mon.0) 2362 : cluster [DBG] osdmap e1398: 8 total, 8 up, 8 in 2021-07-31T02:33:09.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:09 smithi045 conmon[39718]: cluster 2021-07-31T02:33:08.456444+0000 mgr.x (mgr.24542) 2021-07-31T02:33:09.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:09 smithi045 conmon[39718]: 1966 : cluster [DBG] pgmap v2622: 73 pgs: 73 active+clean; 597 MiB data, 4.7 GiB used, 710 GiB / 715 GiB avail 2021-07-31T02:33:09.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:09 smithi045 conmon[33655]: cluster 2021-07-31T02:33:08.431146+0000 mon.a (mon.0) 2362 2021-07-31T02:33:09.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:09 smithi045 conmon[33655]: : cluster [DBG] osdmap e1398: 8 total, 8 up, 8 in 2021-07-31T02:33:09.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:09 smithi045 conmon[33655]: cluster 2021-07-31T02:33:08.456444+0000 mgr.x (mgr.24542) 1966 : cluster [DBG] pgmap v2622: 73 pgs: 73 active+clean; 597 MiB data, 4.7 GiB used, 710 GiB / 715 GiB avail 2021-07-31T02:33:10.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:10 smithi154 conmon[34553]: cluster 2021-07-31T02:33:09.437039+0000 mon.a (mon.0) 2363 : cluster [DBG] osdmap e1399: 8 total, 8 up, 8 in 2021-07-31T02:33:10.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:10 smithi154 conmon[34553]: 2021-07-31T02:33:10.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:10 smithi045 conmon[33655]: cluster 2021-07-31T02:33:09.437039+0000 mon.a (mon.0) 2363 : cluster [DBG] osdmap e1399: 8 total, 8 up, 8 in 2021-07-31T02:33:10.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:10 smithi045 conmon[39718]: cluster 2021-07-31T02:33:09.437039+0000 mon.a (mon.0) 2363 : cluster [DBG] osdmap e1399: 8 total, 8 up, 8 in 2021-07-31T02:33:10.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:10 smithi045 conmon[39718]: 2021-07-31T02:33:10.872 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 167772160 2021-07-31T02:33:10.878 INFO:tasks.workunit.client.0.smithi045.stdout:write: * 83886080~1000 2021-07-31T02:33:10.902 INFO:tasks.workunit.client.0.smithi045.stdout:migration_execute 2021-07-31T02:33:11.018 INFO:tasks.workunit.client.0.smithi045.stdout:migration_commit 2021-07-31T02:33:11.752 INFO:journalctl@ceph.osd.5.smithi154.stdout:Jul 31 02:33:11 smithi154 conmon[42050]: debug 2021-07-31T02:33:11.468+0000 7fc9b5064700 -1 osd.5 pg_epoch: 1401 pg[75.4( v 1397'88 (0'0,1397'88] local-lis/les=1308/1309 n=10 ec=1308/1308 lis/c=1308/1308 les/c/f=1309/1309/0 sis=1308) [5,0,7] r=0 lpr=1308 crt=1397'87 lcod 1397'87 mlcod 1397'87 active+clean+snaptrim trimq=[4f~1]] removing snap head 2021-07-31T02:33:11.752 INFO:journalctl@ceph.osd.7.smithi154.stdout:Jul 31 02:33:11 smithi154 conmon[49149]: debug 2021-07-31T02:33:11.454+0000 7f8a75203700 -1 osd.7 pg_epoch: 1401 pg[75.e( v 1400'107 (0'0,1400'107] local-lis/les=1308/1309 n=13 ec=1308/1308 lis/c=1308/1308 les/c/f=1309/1309/0 sis=1308) [7,0,5] r=0 lpr=1308 crt=1400'107 lcod 1400'106 mlcod 1400'106 active+clean+snaptrim trimq=[4f~1]] removing snap head 2021-07-31T02:33:11.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:11 smithi154 conmon[34553]: cluster 2021-07-31T02:33:11.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:11 smithi154 conmon[34553]: 2021-07-31T02:33:10.438705+0000 mon.a 2021-07-31T02:33:11.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:11 smithi154 conmon[34553]: (mon.0) 2364 : cluster [DBG] osdmap e1400: 8 total, 8 up, 8 in 2021-07-31T02:33:11.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:11 smithi154 conmon[34553]: cluster 2021-07-31T02:33:11.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:11 smithi154 conmon[34553]: 2021-07-31T02 2021-07-31T02:33:11.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:11 smithi154 conmon[34553]: :33:10.456813+0000 mgr.x (mgr.24542) 1967 : cluster [DBG] 2021-07-31T02:33:11.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:11 smithi154 conmon[34553]: pgmap v2625: 73 pgs: 73 active+clean; 621 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 18 KiB/s rd, 4.0 MiB/s wr, 29 op/s 2021-07-31T02:33:11.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:11 smithi045 conmon[33655]: cluster 2021-07-31T02:33:10.438705+0000 mon.a (mon.0 2021-07-31T02:33:11.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:11 smithi045 conmon[33655]: ) 2364 : cluster [DBG] osdmap e1400: 8 total, 8 up, 8 in 2021-07-31T02:33:11.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:11 smithi045 conmon[33655]: cluster 2021-07-31T02:33:10.456813+0000 mgr.x (mgr.24542) 1967 : cluster [DBG] pgmap v2625: 73 pgs: 73 active+clean; 621 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 18 KiB/s rd, 4.0 MiB/s wr, 29 op/s 2021-07-31T02:33:11.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:11 smithi045 conmon[39718]: cluster 2021-07-31T02:33:11.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:11 smithi045 conmon[39718]: 2021-07-31T02:33:10.438705 2021-07-31T02:33:11.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:11 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2364 : cluster [DBG] osdmap e1400: 8 total, 8 up, 8 in 2021-07-31T02:33:11.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:11 smithi045 conmon[39718]: cluster 2021-07-31T02:33:11.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:11 smithi045 conmon[39718]: 2021-07-31T02:33:10.456813+0000 mgr.x (mgr.24542) 1967 : cluster [DBG] pgmap v2625: 73 pgs: 73 active+clean; 621 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 18 KiB/s rd, 4.0 MiB/s wr, 29 op/s 2021-07-31T02:33:11.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:11 smithi045 conmon[39718]: 2021-07-31T02:33:12.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:12 smithi154 conmon[34553]: cluster 2021-07-31T02:33:11.450903+0000 mon.a (mon.0) 2365 : cluster [DBG] osdmap e1401: 8 total, 8 up, 8 in 2021-07-31T02:33:12.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:12 smithi045 conmon[33655]: cluster 2021-07-31T02:33:11.450903+0000 mon.a (mon.0 2021-07-31T02:33:12.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:12 smithi045 conmon[33655]: ) 2365 : cluster [DBG] osdmap e1401: 8 total, 8 up, 8 in 2021-07-31T02:33:12.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:12 smithi045 conmon[39718]: cluster 2021-07-31T02:33:11.450903+0000 mon.a (mon.0) 2365 : cluster 2021-07-31T02:33:12.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:12 smithi045 conmon[39718]: [DBG] osdmap e1401: 8 total, 8 up, 8 in 2021-07-31T02:33:13.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:13 smithi154 conmon[34553]: cluster 2021-07-31T02:33:12.457226+0000 mgr.x (mgr.24542) 1968 : cluster 2021-07-31T02:33:13.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:13 smithi154 conmon[34553]: [DBG] pgmap v2627: 73 pgs: 73 active+clean; 621 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 18 KiB/s rd, 4.0 MiB/s wr, 29 op/s 2021-07-31T02:33:13.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:13 smithi154 conmon[34553]: cluster 2021-07-31T02:33:12.457643+0000 mon.a (mon.0) 2366 : cluster [DBG] osdmap e1402: 8 total, 8 up, 8 in 2021-07-31T02:33:13.753 INFO:journalctl@ceph.osd.7.smithi154.stdout:Jul 31 02:33:13 smithi154 conmon[49149]: debug 2021-07-31T02:33:13.483+0000 7f8a729fe700 -1 osd.7 pg_epoch: 1403 pg[75.1b( v 1403'113 (0'0,1403'113] local-lis/les=1308/1309 n=9 ec=1308/1308 lis/c=1308/1308 les/c/f=1309/1309/0 sis=1308) [7,5,6] r=0 lpr=1308 luod=1402'111 crt=1403'112 lcod 1400'109 mlcod 1400'109 active+clean+snaptrim trimq=[4b~1] ps=[4d~1,4f~1]] removing snap head 2021-07-31T02:33:13.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:13 smithi045 conmon[33655]: cluster 2021-07-31T02:33:12.457226 2021-07-31T02:33:13.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:13 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 1968 : cluster [DBG] pgmap v2627: 73 pgs: 73 active+clean; 621 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 18 KiB/s rd, 4.0 MiB/s wr, 29 op/s 2021-07-31T02:33:13.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:13 smithi045 conmon[33655]: cluster 2021-07-31T02:33:12.457643+0000 mon.a (mon.0) 2366 : 2021-07-31T02:33:13.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:13 smithi045 conmon[33655]: cluster [DBG] osdmap e1402: 8 total, 8 up, 8 in 2021-07-31T02:33:13.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:13 smithi045 conmon[39718]: cluster 2021-07-31T02:33:12.457226+0000 mgr.x (mgr.24542 2021-07-31T02:33:13.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:13 smithi045 conmon[39718]: ) 1968 : cluster [DBG] pgmap v2627: 73 pgs: 73 active+clean; 621 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 18 KiB/s rd, 4.0 MiB/s wr, 29 op/s 2021-07-31T02:33:13.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:13 smithi045 conmon[39718]: cluster 2021-07-31T02:33:12.457643+0000 mon.a (mon.0) 2366 : 2021-07-31T02:33:13.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:13 smithi045 conmon[39718]: cluster [DBG] osdmap e1402: 8 total, 8 up, 8 in 2021-07-31T02:33:13.778 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:33:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:33:13.589Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:33:13.778 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:33:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:33:13.589Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-07-31T02:33:13.778 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Jul 31 02:33:13 smithi045 conmon[61210]: level=error ts=2021-07-31T02:33:13.589Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Post https://172.21.15.45:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.45 because it doesn't contain any IP SANs; Post https://172.21.15.154:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.154 because it doesn't contain any IP SANs" 2021-07-31T02:33:13.830 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMigration.SnapsExpandAfterPrepare (12127 ms) 2021-07-31T02:33:13.831 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMigration.SnapsExpandAfterPrepare2 2021-07-31T02:33:13.849 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 83886080 2021-07-31T02:33:13.853 INFO:tasks.workunit.client.0.smithi045.stdout:write: X 41943040~10 2021-07-31T02:33:13.857 INFO:tasks.workunit.client.0.smithi045.stdout:snap_create: snap1 2021-07-31T02:33:14.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:14 smithi154 conmon[34553]: cluster 2021-07-31T02:33:13.463495+0000 mon.a (mon.0) 2367 2021-07-31T02:33:14.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:14 smithi154 conmon[34553]: : cluster [DBG] osdmap e1403: 8 total, 8 up, 8 in 2021-07-31T02:33:14.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:14 smithi045 conmon[33655]: cluster 2021-07-31T02:33:13.463495+0000 mon.a (mon.0) 2367 : cluster 2021-07-31T02:33:14.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:14 smithi045 conmon[33655]: [DBG] osdmap e1403: 8 total, 8 up, 8 in 2021-07-31T02:33:14.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:14 smithi045 conmon[39718]: cluster 2021-07-31T02:33:13.463495 2021-07-31T02:33:14.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:14 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2367 : cluster [DBG] osdmap e1403: 8 total, 8 up, 8 in 2021-07-31T02:33:15.476 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 41943040 2021-07-31T02:33:15.492 INFO:tasks.workunit.client.0.smithi045.stdout:migration_prepare 2021-07-31T02:33:15.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:15 smithi154 conmon[34553]: cluster 2021-07-31T02:33:14.458128+0000 mgr.x (mgr.24542) 2021-07-31T02:33:15.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:15 smithi154 conmon[34553]: 1969 : cluster [DBG] pgmap v2630: 73 pgs: 73 active+clean; 621 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 70 MiB/s rd, 6.0 MiB/s wr, 235 op/s 2021-07-31T02:33:15.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:15 smithi154 conmon[34553]: cluster 2021-07-31T02:33:14.468817+0000 mon.a (mon.0) 2368 : cluster [DBG] osdmap e1404: 8 total, 8 up, 8 in 2021-07-31T02:33:15.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:15 smithi045 conmon[33655]: cluster 2021-07-31T02:33:14.458128+0000 mgr.x (mgr 2021-07-31T02:33:15.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:15 smithi045 conmon[33655]: .24542) 1969 : cluster [DBG] pgmap v2630: 73 pgs: 73 active+clean; 621 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 70 MiB/s rd, 6.0 MiB/s wr, 235 op/s 2021-07-31T02:33:15.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:15 smithi045 conmon[33655]: cluster 2021-07-31T02:33:14.468817+0000 2021-07-31T02:33:15.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:15 smithi045 conmon[33655]: mon.a (mon.0) 2368 : cluster [DBG] osdmap e1404: 8 total, 8 up, 8 in 2021-07-31T02:33:15.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:15 smithi045 conmon[39718]: cluster 2021-07-31T02:33:14.458128+0000 2021-07-31T02:33:15.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:15 smithi045 conmon[39718]: mgr.x (mgr.24542) 1969 : cluster [DBG] pgmap v2630: 73 pgs: 73 active+clean; 621 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 70 MiB/s rd, 6.0 MiB/s wr, 235 op/s 2021-07-31T02:33:15.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:15 smithi045 conmon[39718]: cluster 2021-07-31 2021-07-31T02:33:15.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:15 smithi045 conmon[39718]: T02:33:14.468817+0000 2021-07-31T02:33:15.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:15 smithi045 conmon[39718]: mon.a (mon.0) 2368 : cluster 2021-07-31T02:33:15.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:15 smithi045 conmon[39718]: [DBG] osdmap e1404: 8 total, 8 up, 8 in 2021-07-31T02:33:16.657 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 83886080 2021-07-31T02:33:16.660 INFO:tasks.workunit.client.0.smithi045.stdout:write: Y 41943040~5 2021-07-31T02:33:16.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:16 smithi154 conmon[34553]: cluster 2021-07-31T02:33:15.470500+0000 mon.a (mon.0 2021-07-31T02:33:16.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:16 smithi154 conmon[34553]: ) 2369 : cluster [DBG] osdmap e1405: 8 total, 8 up, 8 in 2021-07-31T02:33:16.770 INFO:tasks.workunit.client.0.smithi045.stdout:migration_execute 2021-07-31T02:33:16.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:16 smithi045 conmon[33655]: cluster 2021-07-31T02:33:15.470500+0000 mon.a (mon.0) 2369 : 2021-07-31T02:33:16.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:16 smithi045 conmon[33655]: cluster [DBG] osdmap e1405: 8 total, 8 up, 8 in 2021-07-31T02:33:16.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:16 smithi045 conmon[39718]: cluster 2021-07-31T02:33:15.470500+0000 mon.a (mon. 2021-07-31T02:33:16.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:16 smithi045 conmon[39718]: 0) 2369 : cluster [DBG] osdmap e1405: 8 total, 8 up, 8 in 2021-07-31T02:33:16.843 INFO:tasks.workunit.client.0.smithi045.stdout:migration_commit 2021-07-31T02:33:17.499 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMigration.SnapsExpandAfterPrepare2 (3668 ms) 2021-07-31T02:33:17.499 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMigration.SnapsSnapAfterPrepare 2021-07-31T02:33:17.519 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 83886080 2021-07-31T02:33:17.523 INFO:tasks.workunit.client.0.smithi045.stdout:write: 1 0~8388609 2021-07-31T02:33:17.575 INFO:tasks.workunit.client.0.smithi045.stdout:snap_create: snap1 2021-07-31T02:33:17.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:17 smithi154 conmon[34553]: cluster 2021-07-31T02:33:16.458689+0000 mgr.x (mgr.24542) 2021-07-31T02:33:17.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:17 smithi154 conmon[34553]: 1970 : cluster [DBG] pgmap v2633: 73 pgs: 73 active+clean; 621 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 70 MiB/s rd, 6.0 MiB/s wr, 236 op/s 2021-07-31T02:33:17.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:17 smithi154 conmon[34553]: cluster 2021-07-31T02:33:16.480932+0000 mon.a (mon.0) 2370 : cluster [DBG] osdmap e1406: 8 total, 8 up, 8 in 2021-07-31T02:33:17.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:17 smithi154 conmon[34553]: cluster 2021-07-31T02:33 2021-07-31T02:33:17.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:17 smithi154 conmon[34553]: :17.354952+0000 mon.a (mon.0) 2371 : cluster [DBG] osdmap e1407: 8 total, 8 up, 8 in 2021-07-31T02:33:17.780 INFO:journalctl@ceph.osd.0.smithi045.stdout:Jul 31 02:33:17 smithi045 conmon[43243]: debug 2021-07-31T02:33:17.378+0000 7fdf6df94700 -1 osd.0 pg_epoch: 1407 pg[75.15( v 1405'107 (0'0,1405'107] local-lis/les=1308/1309 n=12 ec=1308/1308 lis/c=1308/1308 les/c/f=1309/1309/0 sis=1308) [0,7,2] r=0 lpr=1308 crt=1405'107 lcod 1403'105 mlcod 1403'105 active+clean+snaptrim trimq=[57~1]] removing snap head 2021-07-31T02:33:17.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:17 smithi045 conmon[33655]: cluster 2021-07-31T02:33:16.458689+0000 mgr.x (mgr.24542) 1970 : cluster 2021-07-31T02:33:17.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:17 smithi045 conmon[33655]: [DBG] pgmap v2633: 73 pgs: 73 active+clean; 621 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 70 MiB/s rd, 6.0 MiB/s wr, 236 op/s 2021-07-31T02:33:17.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:17 smithi045 conmon[33655]: cluster 2021-07-31T02:33:16.480932+0000 mon.a (mon.0) 2370 : cluster [DBG] osdmap e1406: 8 total, 8 up, 8 in 2021-07-31T02:33:17.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:17 smithi045 conmon[33655]: cluster 2021-07-31T02:33:17.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:17 smithi045 conmon[33655]: 2021-07-31T02:33:17.354952+0000 mon.a (mon.0) 2371 : cluster [DBG] osdmap e1407: 8 total, 8 up, 8 in 2021-07-31T02:33:17.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:17 smithi045 conmon[39718]: cluster 2021-07-31T02:33:16 2021-07-31T02:33:17.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:17 smithi045 conmon[39718]: .458689+0000 mgr.x (mgr.24542) 1970 : cluster [DBG] pgmap v2633: 73 pgs: 73 active+clean; 621 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 70 MiB/s rd, 6.0 MiB/s wr, 236 op/s 2021-07-31T02:33:17.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:17 smithi045 conmon[39718]: cluster 2021 2021-07-31T02:33:17.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:17 smithi045 conmon[39718]: -07-31T02:33:16.480932+0000 mon.a 2021-07-31T02:33:17.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:17 smithi045 conmon[39718]: (mon.0) 2370 : cluster 2021-07-31T02:33:17.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:17 smithi045 conmon[39718]: [DBG] osdmap e1406: 8 total, 8 up, 8 in 2021-07-31T02:33:17.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:17 smithi045 conmon[39718]: cluster 2021-07 2021-07-31T02:33:17.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:17 smithi045 conmon[39718]: -31T02:33:17.354952 2021-07-31T02:33:17.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:17 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2021-07-31T02:33:17.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:17 smithi045 conmon[39718]: 2371 : cluster [DBG] osdmap e1407: 8 total, 8 up, 8 in 2021-07-31T02:33:19.497 INFO:tasks.workunit.client.0.smithi045.stdout:write: 1 8388609~8388609 2021-07-31T02:33:19.505 INFO:tasks.workunit.client.0.smithi045.stdout:write: X 0~1000 2021-07-31T02:33:19.505 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 1010~1000 2021-07-31T02:33:19.506 INFO:tasks.workunit.client.0.smithi045.stdout:snap_create: snap2 2021-07-31T02:33:19.586 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:33:19.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:19 smithi154 conmon[34553]: cluster 2021-07-31T02:33:18.459240+0000 mgr.x (mgr.24542) 2021-07-31T02:33:19.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:19 smithi154 conmon[34553]: 1971 : cluster [DBG] pgmap v2636: 73 pgs: 73 active+clean; 621 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail 2021-07-31T02:33:19.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:19 smithi154 conmon[34553]: cluster 2021-07-31T02:33:18.487171+0000 mon.a (mon.0) 2372 : cluster [DBG] osdmap e1408: 8 total, 8 up, 8 in 2021-07-31T02:33:19.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:19 smithi045 conmon[39718]: cluster 2021-07-31T02:33:18.459240+0000 mgr.x (mgr 2021-07-31T02:33:19.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:19 smithi045 conmon[39718]: .24542) 1971 : cluster [DBG] pgmap v2636: 73 pgs: 73 active+clean; 621 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail 2021-07-31T02:33:19.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:19 smithi045 conmon[39718]: cluster 2021-07-31T02:33:18.487171+0000 2021-07-31T02:33:19.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:19 smithi045 conmon[39718]: mon.a (mon.0) 2372 : cluster [DBG] osdmap e1408: 8 total, 8 up, 8 in 2021-07-31T02:33:19.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:19 smithi045 conmon[33655]: cluster 2021-07-31T02:33: 2021-07-31T02:33:19.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:19 smithi045 conmon[33655]: 18.459240+0000 mgr.x (mgr.24542) 1971 : cluster [DBG] pgmap v2636: 73 pgs: 73 active+clean; 621 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail 2021-07-31T02:33:19.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:19 smithi045 conmon[33655]: cluster 2021-07-31T02:33:19.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:19 smithi045 conmon[33655]: 2021-07-31T02:33:18 2021-07-31T02:33:19.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:19 smithi045 conmon[33655]: .487171+0000 mon.a (mon.0) 2372 2021-07-31T02:33:19.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:19 smithi045 conmon[33655]: : cluster [DBG] osdmap e1408: 8 total, 8 up, 8 in 2021-07-31T02:33:19.866 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:33:19.866 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 48m ago 48m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:33:19.867 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 45m ago 48m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:33:19.867 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 45m ago 52m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:33:19.867 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 48m ago 54m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:33:19.868 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 48m ago 54m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:33:19.868 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 45m ago 52m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:33:19.868 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 48m ago 52m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:33:19.869 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 48m ago 49m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:33:19.869 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 45m ago 49m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:33:19.869 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 48m ago 51m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:33:19.869 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 48m ago 51m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:33:19.870 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 48m ago 51m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:33:19.870 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 48m ago 50m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:33:19.870 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 45m ago 50m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:33:19.870 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 45m ago 50m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:33:19.870 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 45m ago 49m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:33:19.871 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 45m ago 49m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:33:19.871 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 45m ago 49m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:33:20.198 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:33:20.198 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:33:20.199 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:33:20.199 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:33:20.199 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:33:20.199 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:33:20.199 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:33:20.200 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:33:20.200 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:33:20.200 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:33:20.200 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:33:20.201 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:33:20.201 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:33:20.201 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:33:20.201 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:33:20.201 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:33:20.202 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:33:20.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:20 smithi154 conmon[34553]: cluster 2021-07-31T02:33:19.488899+0000 mon.a (mon.0) 2373 : 2021-07-31T02:33:20.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:20 smithi154 conmon[34553]: cluster [DBG] osdmap e1409: 8 total, 8 up, 8 in 2021-07-31T02:33:20.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:20 smithi154 conmon[34553]: audit 2021-07-31T02:33:19.574106+0000 mgr.x (mgr.24542) 1972 : audit [DBG] from='client.27364 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:33:20.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:20 smithi154 conmon[34553]: audit 2021-07-31T02:33:19.859048+0000 2021-07-31T02:33:20.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:20 smithi154 conmon[34553]: mgr.x (mgr.24542) 1973 : audit [DBG] from='client.17430 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:33:20.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:20 smithi154 conmon[34553]: 2021-07-31T02:33:20.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:20 smithi154 conmon[34553]: audit 2021-07-31T02:33:20.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:20 smithi154 conmon[34553]: 2021-07-31T02:33:20.198552+0000 2021-07-31T02:33:20.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:20 smithi154 conmon[34553]: mon.a (mon.0) 2374 : audit [DBG] from='client.? 172.21.15.45:0/678583277' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:33:20.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:20 smithi045 conmon[33655]: cluster 2021-07-31T02:33:19.488899+0000 mon.a (mon.0) 2373 : 2021-07-31T02:33:20.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:20 smithi045 conmon[33655]: cluster [DBG] osdmap e1409: 8 total, 8 up, 8 in 2021-07-31T02:33:20.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:20 smithi045 conmon[33655]: audit 2021-07-31T02:33:19.574106+0000 mgr.x (mgr.24542) 1972 : audit [DBG] from='client.27364 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:33:20.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:20 smithi045 conmon[33655]: audit 2021-07-31T02:33:19.859048+0000 mgr.x (mgr.24542) 2021-07-31T02:33:20.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:20 smithi045 conmon[33655]: 1973 : audit [DBG] from='client.17430 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:33:20.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:20 smithi045 conmon[33655]: audit 2021-07-31T02:33:20.198552+0000 mon.a (mon.0) 2021-07-31T02:33:20.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:20 smithi045 conmon[33655]: 2374 : audit [DBG] from='client.? 172.21.15.45:0/678583277' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:33:20.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:20 smithi045 conmon[39718]: cluster 2021-07-31T02:33 2021-07-31T02:33:20.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:20 smithi045 conmon[39718]: :19.488899+0000 mon.a (mon.0) 2373 : cluster [DBG] osdmap e1409: 8 total, 8 up, 8 in 2021-07-31T02:33:20.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:20 smithi045 conmon[39718]: audit 2021- 2021-07-31T02:33:20.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:20 smithi045 conmon[39718]: 07-31T02:33:19.574106+0000 2021-07-31T02:33:20.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:20 smithi045 conmon[39718]: mgr.x (mgr.24542) 1972 : audit [DBG] from='client.27364 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:33:20.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:20 smithi045 conmon[39718]: 2021-07-31T02:33:20.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:20 smithi045 conmon[39718]: audit 2021-07-31T02:33:20.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:20 smithi045 conmon[39718]: 2021-07-31T02:33:19 2021-07-31T02:33:20.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:20 smithi045 conmon[39718]: .859048+0000 mgr.x (mgr.24542) 1973 : audit 2021-07-31T02:33:20.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:20 smithi045 conmon[39718]: [DBG] from='client.17430 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:33:20.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:20 smithi045 conmon[39718]: audit 2021-07-31T02:33:20.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:20 smithi045 conmon[39718]: 2021-07-31T02: 2021-07-31T02:33:20.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:20 smithi045 conmon[39718]: 33:20.198552+0000 mon.a (mon.0 2021-07-31T02:33:20.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:20 smithi045 conmon[39718]: ) 2374 : audit [DBG] from='client.? 172.21.15.45:0/678583277' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:33:21.512 INFO:tasks.workunit.client.0.smithi045.stdout:write: X 8388609~1000 2021-07-31T02:33:21.517 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 16777228~1000 2021-07-31T02:33:21.517 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 167772160 2021-07-31T02:33:21.523 INFO:tasks.workunit.client.0.smithi045.stdout:write: 2 83886079~8388609 2021-07-31T02:33:21.541 INFO:tasks.workunit.client.0.smithi045.stdout:snap_create: snap3 2021-07-31T02:33:21.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:21 smithi154 conmon[34553]: cluster 2021-07-31T02:33:20.459883+0000 mgr.x (mgr.24542) 1974 : cluster [DBG] 2021-07-31T02:33:21.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:21 smithi154 conmon[34553]: pgmap v2639: 73 pgs: 73 active+clean; 637 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 128 KiB/s rd, 4.0 MiB/s wr, 181 op/s 2021-07-31T02:33:21.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:21 smithi154 conmon[34553]: cluster 2021-07-31T02:33:20.500906+0000 mon.a (mon.0) 2375 : cluster [DBG] osdmap e1410: 8 total, 8 up, 8 in 2021-07-31T02:33:21.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:21 smithi045 conmon[39718]: cluster 2021-07-31T02:33:20.459883+0000 mgr.x ( 2021-07-31T02:33:21.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:21 smithi045 conmon[39718]: mgr.24542) 1974 : cluster [DBG] pgmap v2639: 73 pgs: 73 active+clean; 637 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 128 KiB/s rd, 4.0 MiB/s wr, 181 op/s 2021-07-31T02:33:21.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:21 smithi045 conmon[39718]: cluster 2021-07-31T02:33:20.500906+0000 mon.a (mon.0) 2375 : 2021-07-31T02:33:21.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:21 smithi045 conmon[39718]: cluster [DBG] osdmap e1410: 8 total, 8 up, 8 in 2021-07-31T02:33:21.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:21 smithi045 conmon[33655]: cluster 2021-07-31T02:33:20.459883+0000 mgr.x (mgr.24542) 2021-07-31T02:33:21.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:21 smithi045 conmon[33655]: 1974 : cluster [DBG] pgmap v2639: 73 pgs: 73 active+clean; 637 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 128 KiB/s rd, 4.0 MiB/s wr, 181 op/s 2021-07-31T02:33:21.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:21 smithi045 conmon[33655]: cluster 2021-07-31T02:33:20.500906+0000 mon.a (mon.0) 2021-07-31T02:33:21.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:21 smithi045 conmon[33655]: 2375 : cluster [DBG] osdmap e1410: 8 total, 8 up, 8 in 2021-07-31T02:33:22.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:22 smithi045 conmon[39718]: cluster 2021-07-31T02:33:21.506900+0000 mon.a (mon.0) 2376 2021-07-31T02:33:22.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:22 smithi045 conmon[39718]: : cluster [DBG] osdmap e1411: 8 total, 8 up, 8 in 2021-07-31T02:33:22.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:22 smithi045 conmon[39718]: cluster 2021-07-31T 2021-07-31T02:33:22.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:22 smithi045 conmon[39718]: 02:33:22.355516+0000 2021-07-31T02:33:22.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:22 smithi045 conmon[39718]: mon.a (mon.0) 2377 : cluster 2021-07-31T02:33:22.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:22 smithi045 conmon[39718]: [DBG] osdmap e1412: 8 total, 8 up, 8 in 2021-07-31T02:33:22.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:22 smithi045 conmon[33655]: cluster 2021-07-31T02:33:21.506900+0000 mon.a ( 2021-07-31T02:33:22.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:22 smithi045 conmon[33655]: mon.0) 2376 : cluster [DBG] osdmap e1411: 8 total, 8 up, 8 in 2021-07-31T02:33:22.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:22 smithi045 conmon[33655]: cluster 2021-07-31T02:33:22.355516+0000 mon.a (mon.0) 2377 : cluster 2021-07-31T02:33:22.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:22 smithi045 conmon[33655]: [DBG] osdmap e1412: 8 total, 8 up, 8 in 2021-07-31T02:33:23.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:22 smithi154 conmon[34553]: cluster 2021-07-31T02:33:21.506900+0000 mon.a (mon.0) 2376 : cluster [DBG] osdmap e1411: 8 total, 8 up, 8 in 2021-07-31T02:33:23.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:22 smithi154 conmon[34553]: cluster 2021-07-31T02:33:22.355516+0000 mon.a (mon.0) 2377 : cluster [DBG] osdmap e1412: 8 total, 8 up, 8 in 2021-07-31T02:33:23.372 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 83886080 2021-07-31T02:33:23.392 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 83886079~1 2021-07-31T02:33:23.396 INFO:tasks.workunit.client.0.smithi045.stdout:migration_prepare 2021-07-31T02:33:24.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:24 smithi154 conmon[34553]: cluster 2021-07-31T02:33:22.460387+0000 mgr.x (mgr.24542) 1975 : cluster [DBG] pgmap v2643: 73 pgs: 73 active+clean; 637 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 129 KiB/s rd, 4.0 MiB/s wr, 182 op/s 2021-07-31T02:33:24.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:24 smithi154 conmon[34553]: cluster 2021-07-31T02:33:23.366619+0000 mon.a (mon.0) 2378 : cluster [DBG] osdmap e1413: 8 total, 8 up, 8 in 2021-07-31T02:33:24.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:24 smithi045 conmon[39718]: cluster 2021-07-31T02:33:22.460387+0000 mgr.x (mgr.24542) 1975 : cluster [DBG] 2021-07-31T02:33:24.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:24 smithi045 conmon[39718]: pgmap v2643: 73 pgs: 73 active+clean; 637 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 129 KiB/s rd, 4.0 MiB/s wr, 182 op/s 2021-07-31T02:33:24.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:24 smithi045 conmon[39718]: cluster 2021-07-31T02:33:23.366619+0000 mon.a (mon.0) 2378 : cluster [DBG] osdmap e1413: 8 total, 8 up, 8 in 2021-07-31T02:33:24.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:24 smithi045 conmon[33655]: cluster 2021-07-31T02:33:22.460387+0000 2021-07-31T02:33:24.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:24 smithi045 conmon[33655]: mgr.x (mgr.24542) 1975 : cluster [DBG] pgmap v2643: 73 pgs: 73 active+clean; 637 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 129 KiB/s rd, 4.0 MiB/s wr, 182 op/s 2021-07-31T02:33:24.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:24 smithi045 conmon[33655]: cluster 2021-07-31T02:33:23.366619+0000 mon.a (mon.0) 2378 : cluster [DBG] osdmap e1413: 8 total, 8 up, 8 in 2021-07-31T02:33:25.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:25 smithi154 conmon[34553]: cluster 2021-07-31T02:33:24.372696+0000 mon.a (mon.0) 2379 : cluster [DBG] 2021-07-31T02:33:25.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:25 smithi154 conmon[34553]: osdmap e1414: 8 total, 8 up, 8 in 2021-07-31T02:33:25.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:25 smithi045 conmon[39718]: cluster 2021-07-31T02:33:24.372696+0000 mon.a (mon.0) 2021-07-31T02:33:25.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:25 smithi045 conmon[39718]: 2379 : cluster [DBG] osdmap e1414: 8 total, 8 up, 8 in 2021-07-31T02:33:25.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:25 smithi045 conmon[33655]: cluster 2021-07-31T02:33:24.372696+0000 mon.a (mon.0 2021-07-31T02:33:25.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:25 smithi045 conmon[33655]: ) 2379 : cluster [DBG] osdmap e1414: 8 total, 8 up, 8 in 2021-07-31T02:33:26.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:26 smithi154 conmon[34553]: cluster 2021-07-31T02:33:24.460874+0000 mgr.x (mgr.24542) 1976 : cluster 2021-07-31T02:33:26.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:26 smithi154 conmon[34553]: [DBG] pgmap v2646: 73 pgs: 73 active+clean; 677 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 19 KiB/s rd, 8.1 MiB/s wr, 33 op/s 2021-07-31T02:33:26.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:26 smithi154 conmon[34553]: cluster 2021-07-31T02:33:25.378601+0000 mon.a (mon.0) 2380 : cluster [DBG] osdmap e1415: 8 total, 8 up, 8 in 2021-07-31T02:33:26.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:26 smithi045 conmon[33655]: cluster 2021-07-31T02:33:24.460874+0000 2021-07-31T02:33:26.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:26 smithi045 conmon[33655]: mgr.x (mgr.24542) 1976 : cluster [DBG] pgmap v2646: 73 pgs: 73 active+clean; 677 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 19 KiB/s rd, 8.1 MiB/s wr, 33 op/s 2021-07-31T02:33:26.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:26 smithi045 conmon[33655]: cluster 2021-07-31T02:33:25.378601+0000 mon.a (mon.0) 2380 : cluster [DBG] 2021-07-31T02:33:26.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:26 smithi045 conmon[33655]: osdmap e1415: 8 total, 8 up, 8 in 2021-07-31T02:33:26.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:26 smithi045 conmon[39718]: cluster 2021-07-31T02:33:24.460874+0000 mgr.x (mgr 2021-07-31T02:33:26.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:26 smithi045 conmon[39718]: .24542) 1976 : cluster [DBG] pgmap v2646: 73 pgs: 73 active+clean; 677 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 19 KiB/s rd, 8.1 MiB/s wr, 33 op/s 2021-07-31T02:33:26.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:26 smithi045 conmon[39718]: cluster 2021-07-31T02:33:25.378601+0000 2021-07-31T02:33:26.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:26 smithi045 conmon[39718]: mon.a (mon.0) 2380 : cluster [DBG] osdmap e1415: 8 total, 8 up, 8 in 2021-07-31T02:33:26.978 INFO:tasks.workunit.client.0.smithi045.stdout:snap_create: after_prepare_snap 2021-07-31T02:33:27.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:27 smithi154 conmon[34553]: cluster 2021-07-31T02:33:26.384182+0000 mon.a (mon.0) 2381 : cluster [DBG] osdmap e1416: 8 total, 8 up, 8 in 2021-07-31T02:33:27.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:27 smithi154 conmon[34553]: 2021-07-31T02:33:27.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:27 smithi045 conmon[33655]: cluster 2021-07-31T02:33:26.384182+0000 mon.a 2021-07-31T02:33:27.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:27 smithi045 conmon[33655]: (mon.0) 2381 : cluster [DBG] osdmap e1416: 8 total, 8 up, 8 in 2021-07-31T02:33:27.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:27 smithi045 conmon[39718]: cluster 2021-07-31T02:33:26.384182+0000 mon.a ( 2021-07-31T02:33:27.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:27 smithi045 conmon[39718]: mon.0) 2381 : cluster [DBG] osdmap e1416: 8 total, 8 up, 8 in 2021-07-31T02:33:28.399 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 41943040 2021-07-31T02:33:28.425 INFO:tasks.workunit.client.0.smithi045.stdout:write: * 0~1000 2021-07-31T02:33:28.430 INFO:tasks.workunit.client.0.smithi045.stdout:migration_execute 2021-07-31T02:33:28.551 INFO:tasks.workunit.client.0.smithi045.stdout:migration_commit 2021-07-31T02:33:28.790 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:28 smithi154 conmon[34553]: cluster 2021-07-31T02:33:26.461582+0000 mgr.x (mgr.24542) 1977 : cluster [DBG] pgmap v2649: 73 pgs: 73 active+clean; 677 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 19 KiB/s rd, 8.0 MiB/s wr, 33 op/s 2021-07-31T02:33:28.791 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:28 smithi154 conmon[34553]: cluster 2021-07-31T02:33:27.388383+0000 mon.a (mon.0) 2382 : cluster [DBG] osdmap e1417: 8 total, 8 up, 8 in 2021-07-31T02:33:28.791 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:28 smithi154 conmon[34553]: audit 2021-07-31T02:33:28.096749+0000 mon.b (mon.2) 305 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:33:28.792 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:28 smithi154 conmon[34553]: audit 2021-07 2021-07-31T02:33:28.792 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:28 smithi154 conmon[34553]: -31T02:33:28.097503+0000 2021-07-31T02:33:28.793 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:28 smithi154 conmon[34553]: mon.a (mon.0) 2383 : audit [INF] 2021-07-31T02:33:28.793 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:28 smithi154 conmon[34553]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:33:28.793 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:28 smithi154 conmon[34553]: audit 2021-07-31T 2021-07-31T02:33:28.794 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:28 smithi154 conmon[34553]: 02:33:28.116187+0000 mon.b (mon 2021-07-31T02:33:28.794 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:28 smithi154 conmon[34553]: .2) 306 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:33:28.794 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:28 smithi154 conmon[34553]: 2021-07-31T02:33:28.795 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:28 smithi154 conmon[34553]: audit 2021- 2021-07-31T02:33:28.796 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:28 smithi154 conmon[34553]: 07-31T02:33:28.116518+0000 2021-07-31T02:33:28.796 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:28 smithi154 conmon[34553]: mon.a (mon.0) 2384 : audit 2021-07-31T02:33:28.796 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:28 smithi154 conmon[34553]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:33:28.797 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:28 smithi045 conmon[33655]: cluster 2021-07-31T02:33:26.461582+0000 mgr.x (mgr.24542) 2021-07-31T02:33:28.797 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:28 smithi045 conmon[33655]: 1977 : cluster [DBG] pgmap v2649: 73 pgs: 73 active+clean; 677 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 19 KiB/s rd, 8.0 MiB/s wr, 33 op/s 2021-07-31T02:33:28.798 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:28 smithi045 conmon[33655]: cluster 2021-07-31T02:33:27.388383+0000 mon.a (mon.0) 2382 : cluster [DBG] osdmap e1417: 8 total, 8 up, 8 in 2021-07-31T02:33:28.798 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:28 smithi045 conmon[33655]: audit 2021 2021-07-31T02:33:28.798 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:28 smithi045 conmon[33655]: -07-31T02:33:28.096749+0000 mon.b (mon.2) 305 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:33:28.799 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:28 smithi045 conmon[33655]: audit 2021-07-31T 2021-07-31T02:33:28.799 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:28 smithi045 conmon[33655]: 02:33:28.097503+0000 mon.a (mon.0) 2021-07-31T02:33:28.799 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:28 smithi045 conmon[33655]: 2383 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:33:28.800 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:28 smithi045 conmon[33655]: audit 2021-07-31T02:33:28.800 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:28 smithi045 conmon[33655]: 2021-07-31T02:33 2021-07-31T02:33:28.800 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:28 smithi045 conmon[33655]: :28.116187+0000 mon.b (mon. 2021-07-31T02:33:28.801 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:28 smithi045 conmon[33655]: 2) 306 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:33:28.801 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:28 smithi045 conmon[33655]: audit 2021-07-31T02:33:28.801 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:28 smithi045 conmon[33655]: 2021-07-31T02: 2021-07-31T02:33:28.802 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:28 smithi045 conmon[33655]: 33:28.116518+0000 mon.a (mon 2021-07-31T02:33:28.802 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:28 smithi045 conmon[33655]: .0) 2384 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:33:28.803 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:28 smithi045 conmon[39718]: cluster 2021-07-31T02:33:26.461582+0000 mgr.x (mgr.24542 2021-07-31T02:33:28.803 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:28 smithi045 conmon[39718]: ) 1977 : cluster [DBG] pgmap v2649: 73 pgs: 73 active+clean; 677 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 19 KiB/s rd, 8.0 MiB/s wr, 33 op/s 2021-07-31T02:33:28.804 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:28 smithi045 conmon[39718]: cluster 2021-07-31T02:33:27.388383+0000 mon.a ( 2021-07-31T02:33:28.804 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:28 smithi045 conmon[39718]: mon.0) 2382 : cluster [DBG] osdmap e1417: 8 total, 8 up, 8 in 2021-07-31T02:33:28.804 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:28 smithi045 conmon[39718]: audit 2021-07-31T02:33: 2021-07-31T02:33:28.805 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:28 smithi045 conmon[39718]: 28.096749+0000 mon.b (mon.2) 305 : audit 2021-07-31T02:33:28.805 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:28 smithi045 conmon[39718]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:33:28.805 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:28 smithi045 conmon[39718]: audit 2021-07 2021-07-31T02:33:28.806 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:28 smithi045 conmon[39718]: -31T02:33:28.097503+0000 mon.a (mon. 2021-07-31T02:33:28.806 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:28 smithi045 conmon[39718]: 0) 2383 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:33:28.806 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:28 smithi045 conmon[39718]: audit 2021-07-31T02:33:28 2021-07-31T02:33:28.807 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:28 smithi045 conmon[39718]: .116187+0000 mon.b (mon.2) 306 : audit [INF] 2021-07-31T02:33:28.807 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:28 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:33:28.807 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:28 smithi045 conmon[39718]: audit 2021-07-31T02:33 2021-07-31T02:33:28.808 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:28 smithi045 conmon[39718]: :28.116518+0000 mon.a (mon.0) 2384 : audit 2021-07-31T02:33:28.808 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:28 smithi045 conmon[39718]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:33:29.752 INFO:journalctl@ceph.osd.5.smithi154.stdout:Jul 31 02:33:29 smithi154 conmon[42050]: debug 2021-07-31T02:33:29.415+0000 7fc9b105c700 -1 osd.5 pg_epoch: 1419 pg[75.14( v 1418'111 (0'0,1418'111] local-lis/les=1308/1309 n=14 ec=1308/1308 lis/c=1308/1308 les/c/f=1309/1309/0 sis=1308) [5,3,1] r=0 lpr=1308 crt=1413'110 lcod 1413'110 mlcod 1413'110 active+clean+snaptrim trimq=[5f~1]] removing snap head 2021-07-31T02:33:29.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:29 smithi154 conmon[34553]: cluster 2021-07-31T02:33:28.395178+0000 mon.a (mon.0) 2385 : cluster [DBG] 2021-07-31T02:33:29.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:29 smithi154 conmon[34553]: osdmap e1418: 8 total, 8 up, 8 in 2021-07-31T02:33:29.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:29 smithi154 conmon[34553]: cluster 2021-07-31T02:33:28.461994+0000 mgr.x (mgr.24542) 1978 : cluster [DBG] pgmap v2652: 73 pgs: 73 active+clean; 677 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail 2021-07-31T02:33:29.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:29 smithi045 conmon[33655]: cluster 2021-07-31T02:33:28.395178+0000 mon.a 2021-07-31T02:33:29.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:29 smithi045 conmon[33655]: (mon.0) 2385 : cluster [DBG] osdmap e1418: 8 total, 8 up, 8 in 2021-07-31T02:33:29.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:29 smithi045 conmon[33655]: cluster 2021-07-31T02:33:28.461994+0000 mgr.x (mgr.24542) 1978 : cluster [DBG] pgmap v2652: 73 pgs: 73 active+clean; 677 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail 2021-07-31T02:33:29.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:29 smithi045 conmon[39718]: cluster 2021-07-31T02:33:28.395178+0000 mon.a (mon. 2021-07-31T02:33:29.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:29 smithi045 conmon[39718]: 0) 2385 : cluster [DBG] osdmap e1418: 8 total, 8 up, 8 in 2021-07-31T02:33:29.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:29 smithi045 conmon[39718]: cluster 2021-07-31T02:33:28.461994+0000 mgr.x (mgr.24542) 1978 : 2021-07-31T02:33:29.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:29 smithi045 conmon[39718]: cluster [DBG] pgmap v2652: 73 pgs: 73 active+clean; 677 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail 2021-07-31T02:33:30.502 INFO:journalctl@ceph.osd.6.smithi154.stdout:Jul 31 02:33:30 smithi154 conmon[45576]: debug 2021-07-31T02:33:30.204+0000 7f47c58ae700 -1 osd.6 pg_epoch: 1419 pg[75.19( v 1413'72 (0'0,1413'72] local-lis/les=1308/1309 n=10 ec=1308/1308 lis/c=1308/1308 les/c/f=1309/1309/0 sis=1308) [6,2,3] r=0 lpr=1308 crt=1413'72 lcod 1411'70 mlcod 1411'70 active+clean+snaptrim trimq=[5f~1]] removing snap head 2021-07-31T02:33:30.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:30 smithi154 conmon[34553]: cluster 2021-07-31T02:33:29.401037+0000 mon.a (mon.0) 2386 : cluster 2021-07-31T02:33:30.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:30 smithi154 conmon[34553]: [DBG] osdmap e1419: 8 total, 8 up, 8 in 2021-07-31T02:33:30.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:30 smithi045 conmon[33655]: cluster 2021-07-31T02:33:29.401037+0000 mon.a (mon.0) 2386 : cluster 2021-07-31T02:33:30.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:30 smithi045 conmon[33655]: [DBG] osdmap e1419: 8 total, 8 up, 8 in 2021-07-31T02:33:30.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:30 smithi045 conmon[39718]: cluster 2021-07-31T02:33:29.401037+0000 mon.a (mon. 2021-07-31T02:33:30.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:30 smithi045 conmon[39718]: 0) 2386 : cluster [DBG] osdmap e1419: 8 total, 8 up, 8 in 2021-07-31T02:33:31.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:31 smithi154 conmon[34553]: cluster 2021-07-31T02:33:30.407272+0000 mon.a (mon.0) 2387 : cluster [DBG] osdmap e1420: 8 total, 8 up, 8 in 2021-07-31T02:33:31.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:31 smithi154 conmon[34553]: cluster 2021-07-31T02:33:30.462420+0000 mgr.x (mgr.24542) 1979 : cluster [DBG] pgmap v2655: 73 pgs: 73 active+clean; 705 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 50 MiB/s rd, 6.0 MiB/s wr, 258 op/s 2021-07-31T02:33:31.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:31 smithi045 conmon[33655]: cluster 2021-07-31T02:33:30.407272+0000 mon.a (mon.0) 2387 : 2021-07-31T02:33:31.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:31 smithi045 conmon[33655]: cluster [DBG] osdmap e1420: 8 total, 8 up, 8 in 2021-07-31T02:33:31.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:31 smithi045 conmon[33655]: cluster 2021-07-31T02:33:30.462420+0000 mgr.x (mgr.24542) 1979 : cluster [DBG] pgmap v2655: 73 pgs: 73 active+clean; 705 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 50 MiB/s rd, 6.0 MiB/s wr, 258 op/s 2021-07-31T02:33:31.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:31 smithi045 conmon[39718]: cluster 2021-07-31T02:33:30.407272+0000 mon.a (mon 2021-07-31T02:33:31.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:31 smithi045 conmon[39718]: .0) 2387 : cluster [DBG] osdmap e1420: 8 total, 8 up, 8 in 2021-07-31T02:33:31.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:31 smithi045 conmon[39718]: cluster 2021-07-31T02:33:30.462420+0000 mgr.x (mgr.24542) 1979 : cluster [DBG] pgmap v2655: 73 pgs: 73 active+clean; 705 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 50 MiB/s rd, 6.0 MiB/s wr, 258 op/s 2021-07-31T02:33:31.807 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMigration.SnapsSnapAfterPrepare (14309 ms) 2021-07-31T02:33:31.808 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMigration.SnapsSnapExpandAfterPrepare 2021-07-31T02:33:31.824 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 83886080 2021-07-31T02:33:31.830 INFO:tasks.workunit.client.0.smithi045.stdout:write: 1 0~8388609 2021-07-31T02:33:31.873 INFO:tasks.workunit.client.0.smithi045.stdout:snap_create: snap1 2021-07-31T02:33:32.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[34553]: cluster 2021-07-31T02:33:32.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[34553]: 2021-07-31T02:33:32.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[34553]: 2021 2021-07-31T02:33:32.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[34553]: - 2021-07-31T02:33:32.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[34553]: 07-31T02:33:31.413120+0000 mon.a ( 2021-07-31T02:33:32.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[34553]: mon.0) 2388 : cluster [DBG] osdmap e1421: 8 total, 8 up, 8 in 2021-07-31T02:33:32.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: debug 2021-07-31T02:33:32.598+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:33:32.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:33:32.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:33:32.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: yield 2021-07-31T02:33:32.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:33:32.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:33:32.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:33:32.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:33:32.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:33:32.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:33:32.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:33:32.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:33:32.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:33:32.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: task.result() 2021-07-31T02:33:32.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:33:32.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: debug 2021-07-31T02:33:32.600+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:33:32.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:33:32.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:33:32.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: yield 2021-07-31T02:33:32.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:33:32.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:33:32.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:33:32.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:33:32.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:33:32.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:33:32.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:33:32.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:33:32.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:33:32.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: task.result() 2021-07-31T02:33:32.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:33:32.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: debug 2021-07-31T02:33:32.635+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:33:32.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:33:32.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:33:32.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: yield 2021-07-31T02:33:32.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:33:32.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:33:32.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:33:32.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:33:32.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:33:32.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:33:32.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:33:32.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:33:32.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:33:32.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: task.result() 2021-07-31T02:33:32.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:33:32.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: debug 2021-07-31T02:33:32.635+0000 7fcb9295f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:33:32.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:33:32.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:33:32.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: yield 2021-07-31T02:33:32.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:33:32.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:33:32.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:33:32.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:33:32.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:33:32.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:33:32.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:33:32.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:33:32.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:33:32.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: task.result() 2021-07-31T02:33:32.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:33:32.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: 2021-07-31T02:33:32.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:33:32.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: 2021-07-31T02:33:32.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:33:32.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:33:32.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:33:32.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:33:32.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:33:32.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:33:32.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:33:32.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:33:32.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:33:32.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:33:32.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:33:32.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:33:32.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:33:32.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:33:32.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:33:32.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:33:32.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:33:32.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:33:32.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: raise 2021-07-31T02:33:32.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:33:32.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:33:32.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:33:32.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:33:32.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:33:32.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: debug 2021-07-31T02:33:32.642+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:33:32.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:33:32.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:33:32.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: yield 2021-07-31T02:33:32.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:33:32.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:33:32.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:33:32.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:33:32.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:33:32.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:33:32.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:33:32.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:33:32.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:33:32.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: task.result() 2021-07-31T02:33:32.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:33:32.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: debug 2021-07-31T02:33:32.642+0000 7fcb93160700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:33:32.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:33:32.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:33:32.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: yield 2021-07-31T02:33:32.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:33:32.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:33:32.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:33:32.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:33:32.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:33:32.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:33:32.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:33:32.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:33:32.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:33:32.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: task.result() 2021-07-31T02:33:32.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:33:32.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: 2021-07-31T02:33:32.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:33:32.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: 2021-07-31T02:33:32.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:33:32.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:33:32.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:33:32.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:33:32.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:33:32.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:33:32.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:33:32.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:33:32.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:33:32.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:33:32.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:33:32.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:33:32.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:33:32.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:33:32.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:33:32.786 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:33:32.786 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:33:32.786 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:33:32.787 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: raise 2021-07-31T02:33:32.787 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:33:32.787 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:33:32.787 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:33:32.788 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:33:32.788 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:33:32 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:33:32.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:32 smithi045 conmon[39718]: cluster 2021-07-31T02:33:31.413120 2021-07-31T02:33:32.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:32 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2388 : cluster [DBG] osdmap e1421: 8 total, 8 up, 8 in 2021-07-31T02:33:32.789 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:32 smithi045 conmon[33655]: cluster 2021-07-31T02:33:31.413120+0000 mon.a (mon.0) 2388 : 2021-07-31T02:33:32.789 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:32 smithi045 conmon[33655]: cluster [DBG] osdmap e1421: 8 total, 8 up, 8 in 2021-07-31T02:33:33.433 INFO:tasks.workunit.client.0.smithi045.stdout:write: 1 8388609~8388609 2021-07-31T02:33:33.446 INFO:tasks.workunit.client.0.smithi045.stdout:write: X 0~1000 2021-07-31T02:33:33.446 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 1010~1000 2021-07-31T02:33:33.446 INFO:tasks.workunit.client.0.smithi045.stdout:snap_create: snap2 2021-07-31T02:33:33.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: cluster 2021-07-31T02:33:32.414103+0000 mon.a (mon.0) 2389 : cluster 2021-07-31T02:33:33.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: [DBG] osdmap e1422: 8 total, 8 up, 8 in 2021-07-31T02:33:33.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: cluster 2021-07-31T02:33:32.462995+0000 mgr.x (mgr.24542) 1980 : cluster [DBG] pgmap v2658: 73 pgs: 73 active+clean; 705 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 50 MiB/s rd, 6.0 MiB/s wr, 258 op/s 2021-07-31T02:33:33.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: audit 2021- 2021-07-31T02:33:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: 07-31T02:33:32.554837+0000 mon.b (mon.2) 307 2021-07-31T02:33:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:33:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: audit 2021-07-31 2021-07-31T02:33:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: T02:33:32.556201+0000 mon.b (mon.2) 308 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:33:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: audit 2021-07 2021-07-31T02:33:33.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: -31T02:33:32.556984+0000 mon.b (mon.2) 309 : audit 2021-07-31T02:33:33.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:33:33.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: cephadm 2021-07-31T02: 2021-07-31T02:33:33.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: 33:32.557622+0000 mgr.x (mgr.24542) 1981 : cephadm [WRN] 2021-07-31T02:33:33.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:33:33.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:33:32.599268 2021-07-31T02:33:33.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: +0000 mgr.x (mgr.24542) 1982 : cephadm [ERR] 2021-07-31T02:33:33.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:33:33.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:33:33.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: yield 2021-07-31T02:33:33.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:33:33.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:33:33.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:33:33.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:33:33.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:33:33.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:33:33.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:33:33.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:33:33.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:33:33.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:33:33.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:33:33.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:33:32.599980+0000 mgr.x (mgr.24542) 1983 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:33:33.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:33:32. 2021-07-31T02:33:33.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: 601057+0000 mgr.x (mgr.24542) 1984 : cephadm [ERR] 2021-07-31T02:33:33.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:33:33.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:33:33.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: yield 2021-07-31T02:33:33.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:33:33.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:33:33.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:33:33.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:33:33.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:33:33.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:33:33.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:33:33.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:33:33.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:33:33.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:33:33.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:33:33.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:33:32.610703+0000 mgr.x (mgr.24542) 1985 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:33:33.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:33:32.635929+0000 mgr.x 2021-07-31T02:33:33.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: (mgr.24542) 1986 : cephadm [ERR] 2021-07-31T02:33:33.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:33:33.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:33:33.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: yield 2021-07-31T02:33:33.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:33:33.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:33:33.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:33:33.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:33:33.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:33:33.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:33:33.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:33:33.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:33:33.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:33:33.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:33:33.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:33:33.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:33:32.636343+0000 mgr.x (mgr.24542) 1987 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:33:33.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:33:33.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:33:33.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: yield 2021-07-31T02:33:33.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:33:33.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:33:33.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:33:33.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:33:33.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:33:33.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:33:33.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:33:33.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:33:33.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:33:33.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:33:33.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:33:33.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: 2021-07-31T02:33:33.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:33:33.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: 2021-07-31T02:33:33.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:33:33.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:33:33.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:33:33.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:33:33.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:33:33.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:33:33.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:33:33.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:33:33.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:33:33.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:33:33.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:33:33.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:33:33.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:33:33.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:33:33.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:33:33.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:33:33.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:33:33.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:33:33.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: raise 2021-07-31T02:33:33.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:33:33.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:33:33.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:33:33.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:33:33.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:33:33.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:33:32 2021-07-31T02:33:33.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: .643544+0000 mgr.x (mgr.24542) 1988 : cephadm [ERR] 2021-07-31T02:33:33.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:33:33.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:33:33.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: yield 2021-07-31T02:33:33.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:33:33.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:33:33.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:33:33.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:33:33.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:33:33.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:33:33.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:33:33.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:33:33.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:33:33.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:33:33.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:33:33.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:33:32.643876+0000 mgr.x (mgr.24542) 1989 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:33:33.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:33:33.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:33:33.782 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: yield 2021-07-31T02:33:33.782 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:33:33.782 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:33:33.782 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:33:33.782 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:33:33.783 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:33:33.783 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:33:33.783 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:33:33.783 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:33:33.784 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:33:33.784 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:33:33.784 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:33:33.784 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: 2021-07-31T02:33:33.784 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:33:33.785 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: 2021-07-31T02:33:33.785 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:33:33.785 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:33:33.785 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:33:33.785 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:33:33.786 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:33:33.786 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:33:33.786 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:33:33.786 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:33:33.787 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:33:33.787 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:33:33.787 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:33:33.787 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:33:33.787 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:33:33.788 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:33:33.788 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:33:33.788 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:33:33.788 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:33:33.788 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:33:33.789 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: raise 2021-07-31T02:33:33.789 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:33:33.789 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:33:33.789 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:33:33.790 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:33:33.790 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:33 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:33:33.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: cluster 2021-07-31T02:33:33.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: 2021-07- 2021-07-31T02:33:33.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: 31T02:33:32.414103+0000 mon.a (mon.0) 2389 : cluster [DBG] osdmap e1422: 8 total, 8 up, 8 in 2021-07-31T02:33:33.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: cluster 2021-07- 2021-07-31T02:33:33.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: 31T02:33:32.462995+0000 mgr.x (mgr.24542) 1980 : cluster [DBG] 2021-07-31T02:33:33.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: pgmap v2658: 73 pgs: 73 active+clean; 705 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 50 MiB/s rd, 6.0 MiB/s wr, 258 op/s 2021-07-31T02:33:33.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: audit 2021-07-31T02:33:33.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: 2021-07-31T 2021-07-31T02:33:33.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: 02:33:32.554837+0000 mon.b (mon. 2021-07-31T02:33:33.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: 2) 307 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:33:33.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: audit 2021-07-31T02:33:33.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: 2021-07-31T02:33:32.556201+0000 mon.b (mon.2) 308 : audit 2021-07-31T02:33:33.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:33:33.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: audit 2021 2021-07-31T02:33:33.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: -07-31T02:33:32.556984+0000 mon.b 2021-07-31T02:33:33.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: (mon.2) 309 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:33:33.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: cephadm 2021-07-31 2021-07-31T02:33:33.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: T02:33:32.557622+0000 mgr.x (mgr.24542) 1981 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:33:33.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: 2021-07-31T02:33:33.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: cephadm 2021-07 2021-07-31T02:33:33.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: -31T02:33:32.599268+0000 mgr.x (mgr.24542) 1982 : cephadm [ERR] 2021-07-31T02:33:33.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:33:33.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:33:33.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: yield 2021-07-31T02:33:33.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:33:33.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:33:33.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:33:33.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:33:33.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:33:33.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:33:33.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:33:33.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:33:33.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:33:33.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:33:33.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:33:33.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:33:32.599980+0000 mgr.x (mgr.24542) 1983 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:33:33.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:33:32.601057+0000 mgr.x (mgr.24542) 1984 : cephadm [ERR] 2021-07-31T02:33:33.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:33:33.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:33:33.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: yield 2021-07-31T02:33:33.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:33:33.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:33:33.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:33:33.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:33:33.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:33:33.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:33:33.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:33:33.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:33:33.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:33:33.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:33:33.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:33:33.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:33:32.610703+0000 mgr.x (mgr.24542) 1985 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:33:33.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: cephadm 2021-07 2021-07-31T02:33:33.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: -31T02: 2021-07-31T02:33:33.802 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: 33:32.635929+0000 mgr.x (mgr.24542) 1986 : cephadm [ERR] 2021-07-31T02:33:33.802 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:33:33.802 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:33:33.802 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: yield 2021-07-31T02:33:33.802 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:33:33.803 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:33:33.803 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:33:33.803 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:33:33.803 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:33:33.803 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:33:33.804 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:33:33.804 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:33:33.804 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:33:33.804 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:33:33.805 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:33:33.805 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:33:32.636343+0000 mgr.x (mgr.24542) 1987 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:33:33.805 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:33:33.805 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:33:33.805 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: yield 2021-07-31T02:33:33.806 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:33:33.806 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:33:33.806 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:33:33.806 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:33:33.806 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:33:33.807 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:33:33.807 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:33:33.807 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:33:33.807 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:33:33.808 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:33:33.808 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:33:33.808 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: 2021-07-31T02:33:33.808 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:33:33.808 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: 2021-07-31T02:33:33.809 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:33:33.809 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:33:33.809 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:33:33.809 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:33:33.810 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:33:33.810 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:33:33.810 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:33:33.810 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:33:33.810 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:33:33.811 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:33:33.811 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:33:33.811 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:33:33.811 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:33:33.811 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:33:33.812 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:33:33.812 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:33:33.812 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:33:33.812 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:33:33.813 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: raise 2021-07-31T02:33:33.813 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:33:33.813 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:33:33.813 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:33:33.813 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:33:33.814 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:33:33.814 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: cephadm 2021- 2021-07-31T02:33:33.814 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: 07-31T02:33:32.643544+0000 mgr.x (mgr.24542) 1988 : cephadm [ERR] 2021-07-31T02:33:33.814 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:33:33.815 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:33:33.815 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: yield 2021-07-31T02:33:33.815 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:33:33.815 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:33:33.815 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:33:33.816 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:33:33.816 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:33:33.816 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: cluster 2021-07-31T02:33:32.414103+0000 mon.a ( 2021-07-31T02:33:33.817 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: mon.0) 2389 : cluster [DBG] osdmap e1422: 8 total, 8 up, 8 in 2021-07-31T02:33:33.817 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: cluster 2021-07-31T02:33:32.462995+0000 mgr.x (mgr.24542) 1980 : 2021-07-31T02:33:33.817 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: cluster [DBG] pgmap v2658: 73 pgs: 73 active+clean; 705 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 50 MiB/s rd, 6.0 MiB/s wr, 258 op/s 2021-07-31T02:33:33.817 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: audit 2021-07-31T02:33:33.817 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: 2021-07-31T02:33 2021-07-31T02:33:33.818 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: :32.554837+0000 mon.b (mon 2021-07-31T02:33:33.818 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: .2) 307 : audit [DBG] 2021-07-31T02:33:33.818 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:33:33.818 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: audit 2021-07-31T02:33:33.819 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: 2021-07-31T02:33:32.556201+0000 mon.b ( 2021-07-31T02:33:33.819 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: mon.2) 308 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:33:33.819 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: audit 2021-07-31T02:33:33.819 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: 2021-07-31T02:33:32.556984+0000 2021-07-31T02:33:33.819 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: mon.b (mon.2) 309 : 2021-07-31T02:33:33.820 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:33:33.820 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:33:33.820 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: 2021-07 2021-07-31T02:33:33.820 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: -31T02:33:32. 2021-07-31T02:33:33.821 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: 557622+0000 mgr.x (mgr.24542 2021-07-31T02:33:33.821 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: ) 1981 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:33:33.821 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: cephadm 2021- 2021-07-31T02:33:33.821 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: 07-31T02:33:32.599268 2021-07-31T02:33:33.821 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 1982 2021-07-31T02:33:33.822 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: : cephadm [ERR] 2021-07-31T02:33:33.822 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:33:33.822 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:33:33.822 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: yield 2021-07-31T02:33:33.822 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:33:33.823 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:33:33.823 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:33:33.823 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:33:33.823 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:33:33.823 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:33:33.824 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:33:33.824 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:33:33.824 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:33:33.824 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:33:33.825 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:33:33.825 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:33:32.599980 2021-07-31T02:33:33.825 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 1983 2021-07-31T02:33:33.825 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:33:33.825 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: cephadm 2021-07 2021-07-31T02:33:33.826 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: -31T02:33:32.601057+0000 mgr.x ( 2021-07-31T02:33:33.826 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:33:33.827 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:33:33.827 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:33:33.827 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:33:33.827 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:33:33.827 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:33:33.828 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:33:32.643876+0000 mgr.x (mgr.24542) 1989 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:33:33.828 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:33:33.828 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:33:33.828 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: yield 2021-07-31T02:33:33.828 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:33:33.829 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:33:33.829 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:33:33.829 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:33:33.829 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:33:33.830 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:33:33.830 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:33:33.830 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:33:33.830 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:33:33.830 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:33:33.831 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:33:33.831 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: 2021-07-31T02:33:33.831 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:33:33.831 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: 2021-07-31T02:33:33.831 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:33:33.832 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:33:33.832 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:33:33.832 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:33:33.832 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:33:33.832 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:33:33.833 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:33:33.833 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:33:33.833 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:33:33.833 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:33:33.834 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:33:33.834 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:33:33.834 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:33:33.834 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:33:33.834 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:33:33.835 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:33:33.835 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:33:33.835 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:33:33.835 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: raise 2021-07-31T02:33:33.835 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:33:33.836 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:33:33.836 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:33:33.836 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:33:33.836 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:33:33.837 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: mgr.24542) 1984 : cephadm [ERR] 2021-07-31T02:33:33.837 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: 2021-07-31T02:33:33.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:33:33.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:33:33.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: yield 2021-07-31T02:33:33.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:33:33.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:33:33.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:33:33.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:33:33.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:33:33.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:33:33.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:33:33.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:33:33.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:33:33.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:33:33.841 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:33:33.841 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:33:32.610703+0000 mgr.x (mgr.24542) 1985 : cephadm [INF] 2021-07-31T02:33:33.841 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:33:33.841 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: cephadm 2021-07-31 2021-07-31T02:33:33.841 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: T02:33:32.635929 2021-07-31T02:33:33.842 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 2021-07-31T02:33:33.842 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: 1986 : cephadm [ERR] 2021-07-31T02:33:33.842 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:33:33.842 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:33:33.842 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: yield 2021-07-31T02:33:33.843 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:33:33.843 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:33:33.843 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:33:33.843 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:33:33.844 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:33:33.844 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:33:33.844 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:33:33.844 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:33:33.844 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:33:33.845 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:33:33.845 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:33:33.845 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:33:32.636343+0000 mgr.x (mgr.24542) 1987 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:33:33.845 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:33:33.845 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:33:33.846 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: yield 2021-07-31T02:33:33.846 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:33:33.846 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:33:33.846 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:33:33.847 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:33:33.847 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:33:33.847 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:33:33.847 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:33:33.847 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:33:33.848 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:33:33.848 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:33:33.848 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:33:33.848 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: 2021-07-31T02:33:33.849 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:33:33.849 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: 2021-07-31T02:33:33.849 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:33:33.849 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:33:33.849 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:33:33.850 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:33:33.850 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:33:33.850 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:33:33.850 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:33:33.850 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:33:33.851 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:33:33.851 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:33:33.851 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:33:33.851 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:33:33.851 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:33:33.852 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:33:33.852 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:33:33.852 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:33:33.852 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:33:33.853 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:33:33.853 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: raise 2021-07-31T02:33:33.853 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:33:33.853 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:33:33.853 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:33:33.854 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:33:33.854 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:33:33.854 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:33:32.643544+0000 mgr.x (mgr.24542) 1988 : cephadm [ERR] 2021-07-31T02:33:33.854 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:33:33.854 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:33:33.855 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: yield 2021-07-31T02:33:33.855 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:33:33.855 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:33:33.855 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:33:33.856 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:33:33.856 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:33:33.856 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:33:33.856 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:33:33.856 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:33:33.857 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:33:33.857 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:33:33.857 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:33:33.857 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:33:32.643876+0000 mgr.x (mgr.24542) 1989 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:33:33.857 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:33:33.858 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:33:33.858 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: yield 2021-07-31T02:33:33.858 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:33:33.858 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:33:33.859 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:33:33.859 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:33:33.859 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:33:33.859 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:33:33.859 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:33:33.860 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:33:33.860 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:33:33.860 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:33:33.860 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:33:33.860 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: 2021-07-31T02:33:33.861 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:33:33.861 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: 2021-07-31T02:33:33.861 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:33:33.861 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:33:33.862 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:33:33.862 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:33:33.862 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:33:33.862 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:33:33.862 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:33:33.863 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:33:33.863 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:33:33.863 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:33:33.863 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:33:33.863 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:33:33.864 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:33:33.864 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:33:33.864 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:33:33.864 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:33:33.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:33:33.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:33:33.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: raise 2021-07-31T02:33:33.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:33:33.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:33:33.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:33:33.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:33:33.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:33 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:33:34.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:34 smithi154 conmon[34553]: cluster 2021-07-31T02:33:33.426148+0000 mon.a (mon.0) 2390 : cluster [DBG] osdmap e1423: 8 total, 8 up, 8 in 2021-07-31T02:33:34.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:34 smithi045 conmon[33655]: cluster 2021-07-31T02:33:33.426148+0000 mon.a (mon.0) 2390 2021-07-31T02:33:34.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:34 smithi045 conmon[33655]: : cluster [DBG] osdmap e1423: 8 total, 8 up, 8 in 2021-07-31T02:33:34.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:34 smithi045 conmon[39718]: cluster 2021-07-31T02:33:33.426148+0000 mon.a (mon.0 2021-07-31T02:33:34.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:34 smithi045 conmon[39718]: ) 2390 : cluster [DBG] osdmap e1423: 8 total, 8 up, 8 in 2021-07-31T02:33:35.440 INFO:tasks.workunit.client.0.smithi045.stdout:write: X 8388609~1000 2021-07-31T02:33:35.443 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 16777228~1000 2021-07-31T02:33:35.443 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 167772160 2021-07-31T02:33:35.450 INFO:tasks.workunit.client.0.smithi045.stdout:write: 2 83886079~8388609 2021-07-31T02:33:35.458 INFO:tasks.workunit.client.0.smithi045.stdout:snap_create: snap3 2021-07-31T02:33:35.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:35 smithi154 conmon[34553]: cluster 2021-07-31T02:33:34.430112+0000 mon.a (mon.0) 2021-07-31T02:33:35.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:35 smithi154 conmon[34553]: 2391 : cluster [DBG] osdmap e1424: 8 total, 8 up, 8 in 2021-07-31T02:33:35.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:35 smithi154 conmon[34553]: cluster 2021-07-31T02:33:34.463428+0000 mgr.x (mgr.24542) 1990 : cluster [DBG] pgmap v2661: 73 pgs: 73 active+clean; 709 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 40 MiB/s rd, 8.0 MiB/s wr, 43 op/s 2021-07-31T02:33:35.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:35 smithi045 conmon[39718]: cluster 2021-07-31T02:33:34.430112+0000 mon.a ( 2021-07-31T02:33:35.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:35 smithi045 conmon[39718]: mon.0) 2391 : cluster [DBG] osdmap e1424: 8 total, 8 up, 8 in 2021-07-31T02:33:35.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:35 smithi045 conmon[39718]: cluster 2021-07-31T02:33:34.463428+0000 mgr.x (mgr.24542) 2021-07-31T02:33:35.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:35 smithi045 conmon[39718]: 1990 : cluster [DBG] pgmap v2661: 73 pgs: 73 active+clean; 709 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 40 MiB/s rd, 8.0 MiB/s wr, 43 op/s 2021-07-31T02:33:35.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:35 smithi045 conmon[33655]: cluster 2021-07-31T02:33:34.430112+0000 mon.a (mon.0 2021-07-31T02:33:35.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:35 smithi045 conmon[33655]: ) 2391 : cluster [DBG] osdmap e1424: 8 total, 8 up, 8 in 2021-07-31T02:33:35.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:35 smithi045 conmon[33655]: cluster 2021-07-31T02:33:34.463428+0000 mgr.x (mgr.24542) 1990 : cluster [DBG] 2021-07-31T02:33:35.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:35 smithi045 conmon[33655]: pgmap v2661: 73 pgs: 73 active+clean; 709 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 40 MiB/s rd, 8.0 MiB/s wr, 43 op/s 2021-07-31T02:33:36.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:36 smithi154 conmon[34553]: cluster 2021-07-31T02:33:35.435991+0000 mon.a (mon.0) 2392 : cluster [DBG] 2021-07-31T02:33:36.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:36 smithi154 conmon[34553]: osdmap e1425: 8 total, 8 up, 8 in 2021-07-31T02:33:36.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:36 smithi045 conmon[33655]: cluster 2021-07-31T02:33:35.435991+0000 mon.a (mon 2021-07-31T02:33:36.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:36 smithi045 conmon[33655]: .0) 2392 : cluster [DBG] osdmap e1425: 8 total, 8 up, 8 in 2021-07-31T02:33:36.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:36 smithi045 conmon[39718]: cluster 2021-07-31T02:33:35.435991+0000 2021-07-31T02:33:36.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:36 smithi045 conmon[39718]: mon.a (mon.0) 2392 : cluster [DBG] osdmap e1425: 8 total, 8 up, 8 in 2021-07-31T02:33:37.376 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 83886080 2021-07-31T02:33:37.400 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 83886079~1 2021-07-31T02:33:37.404 INFO:tasks.workunit.client.0.smithi045.stdout:migration_prepare 2021-07-31T02:33:37.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:37 smithi154 conmon[34553]: cluster 2021-07-31T02:33:36.441877+0000 mon.a (mon.0) 2393 : cluster [DBG] osdmap e1426: 8 total, 8 up, 8 in 2021-07-31T02:33:37.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:37 smithi154 conmon[34553]: 2021-07-31T02:33:37.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:37 smithi154 conmon[34553]: cluster 2021-07-31T02:33:36.463886+0000 mgr.x (mgr.24542) 1991 : cluster [DBG] pgmap v2664: 73 pgs: 73 active+clean; 709 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 40 MiB/s rd, 8.0 MiB/s wr, 43 op/s 2021-07-31T02:33:37.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:37 smithi154 conmon[34553]: cluster 2021-07-31T02:33:37.359667+0000 mon.a (mon.0) 2394 : cluster [DBG] osdmap e1427: 8 total, 8 up, 8 in 2021-07-31T02:33:37.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:37 smithi045 conmon[33655]: cluster 2021-07-31T02:33:36.441877+0000 mon.a 2021-07-31T02:33:37.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:37 smithi045 conmon[33655]: (mon.0) 2393 : cluster [DBG] osdmap e1426: 8 total, 8 up, 8 in 2021-07-31T02:33:37.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:37 smithi045 conmon[33655]: cluster 2021-07-31T02:33:36.463886+0000 2021-07-31T02:33:37.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:37 smithi045 conmon[33655]: mgr.x (mgr.24542) 1991 : cluster [DBG] pgmap v2664: 73 pgs: 73 active+clean; 709 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 40 MiB/s rd, 8.0 MiB/s wr, 43 op/s 2021-07-31T02:33:37.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:37 smithi045 conmon[33655]: cluster 2021-07-31T 2021-07-31T02:33:37.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:37 smithi045 conmon[33655]: 02:33:37.359667+0000 mon.a (mon.0) 2394 : cluster 2021-07-31T02:33:37.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:37 smithi045 conmon[33655]: [DBG] osdmap e1427: 8 total, 8 up, 8 in 2021-07-31T02:33:37.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:37 smithi045 conmon[39718]: cluster 2021-07-31T02:33:36.441877+0000 mon.a (mon 2021-07-31T02:33:37.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:37 smithi045 conmon[39718]: .0) 2393 : cluster [DBG] osdmap e1426: 8 total, 8 up, 8 in 2021-07-31T02:33:37.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:37 smithi045 conmon[39718]: cluster 2021-07-31T02: 2021-07-31T02:33:37.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:37 smithi045 conmon[39718]: 33:36.463886+0000 mgr.x (mgr.24542) 1991 : cluster [DBG] pgmap v2664: 73 pgs: 73 active+clean; 709 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 40 MiB/s rd, 8.0 MiB/s wr, 43 op/s 2021-07-31T02:33:37.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:37 smithi045 conmon[39718]: cluster 2021- 2021-07-31T02:33:37.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:37 smithi045 conmon[39718]: 07-31T02:33:37.359667+0000 mon.a (mon.0) 2021-07-31T02:33:37.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:37 smithi045 conmon[39718]: 2394 : cluster [DBG] osdmap e1427: 8 total, 8 up, 8 in 2021-07-31T02:33:39.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:39 smithi154 conmon[34553]: cluster 2021-07-31T02:33:38.367264+0000 mon.a (mon.0) 2395 : cluster 2021-07-31T02:33:39.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:39 smithi154 conmon[34553]: [DBG] osdmap e1428: 8 total, 8 up, 8 in 2021-07-31T02:33:39.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:39 smithi045 conmon[33655]: cluster 2021-07-31T02:33:38.367264+0000 mon.a ( 2021-07-31T02:33:39.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:39 smithi045 conmon[33655]: mon.0) 2395 : cluster [DBG] osdmap e1428: 8 total, 8 up, 8 in 2021-07-31T02:33:39.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:39 smithi045 conmon[39718]: cluster 2021-07-31T02:33:38.367264+0000 mon.a (mon. 2021-07-31T02:33:39.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:39 smithi045 conmon[39718]: 0) 2395 : cluster [DBG] osdmap e1428: 8 total, 8 up, 8 in 2021-07-31T02:33:40.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:40 smithi154 conmon[34553]: cluster 2021-07-31T02:33:38.464512+0000 mgr.x (mgr.24542) 1992 : cluster [DBG] pgmap v2667: 73 pgs: 73 active+clean; 709 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail 2021-07-31T02:33:40.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:40 smithi154 conmon[34553]: cluster 2021-07-31T02:33:39.373222+0000 mon.a (mon.0) 2396 : cluster [DBG] osdmap e1429: 8 total, 8 up, 8 in 2021-07-31T02:33:40.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:40 smithi045 conmon[33655]: cluster 2021-07-31T02:33:38.464512+0000 mgr.x (mgr.24542) 1992 2021-07-31T02:33:40.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:40 smithi045 conmon[33655]: : cluster [DBG] pgmap v2667: 73 pgs: 73 active+clean; 709 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail 2021-07-31T02:33:40.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:40 smithi045 conmon[33655]: cluster 2021-07-31T02:33:39.373222+0000 mon.a (mon.0) 2396 : cluster [DBG] osdmap e1429: 8 total, 8 up, 8 in 2021-07-31T02:33:40.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:40 smithi045 conmon[39718]: cluster 2021-07-31T02:33:38.464512+0000 mgr.x (mgr.24542 2021-07-31T02:33:40.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:40 smithi045 conmon[39718]: ) 1992 : cluster [DBG] pgmap v2667: 73 pgs: 73 active+clean; 709 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail 2021-07-31T02:33:40.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:40 smithi045 conmon[39718]: cluster 2021-07-31T02:33:39.373222+0000 mon.a (mon.0) 2396 2021-07-31T02:33:40.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:40 smithi045 conmon[39718]: : cluster [DBG] osdmap e1429: 8 total, 8 up, 8 in 2021-07-31T02:33:40.799 INFO:tasks.workunit.client.0.smithi045.stdout:snap_create: after_prepare_snap 2021-07-31T02:33:41.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:41 smithi154 conmon[34553]: cluster 2021-07-31T02:33:40.381736+0000 mon.a (mon.0) 2397 : 2021-07-31T02:33:41.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:41 smithi154 conmon[34553]: cluster [DBG] osdmap e1430: 8 total, 8 up, 8 in 2021-07-31T02:33:41.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:41 smithi045 conmon[33655]: cluster 2021-07-31T02:33:40.381736+0000 mon.a (mon. 2021-07-31T02:33:41.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:41 smithi045 conmon[33655]: 0) 2397 : cluster [DBG] osdmap e1430: 8 total, 8 up, 8 in 2021-07-31T02:33:41.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:41 smithi045 conmon[39718]: cluster 2021-07-31T02:33:40.381736+0000 mon.a (mon.0) 2021-07-31T02:33:41.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:41 smithi045 conmon[39718]: 2397 : cluster [DBG] osdmap e1430: 8 total, 8 up, 8 in 2021-07-31T02:33:42.394 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 167772160 2021-07-31T02:33:42.405 INFO:tasks.workunit.client.0.smithi045.stdout:write: * 83886080~1000 2021-07-31T02:33:42.410 INFO:tasks.workunit.client.0.smithi045.stdout:migration_execute 2021-07-31T02:33:42.550 INFO:tasks.workunit.client.0.smithi045.stdout:migration_commit 2021-07-31T02:33:42.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:42 smithi154 conmon[34553]: cluster 2021-07-31T02:33:40.464974+0000 mgr.x (mgr.24542) 1993 : cluster [DBG] pgmap v2670: 73 pgs: 73 active+clean; 733 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 18 KiB/s rd, 4.0 MiB/s wr, 29 op/s 2021-07-31T02:33:42.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:42 smithi154 conmon[34553]: cluster 2021-07-31T02:33:41.388097+0000 mon.a (mon.0) 2398 : cluster [DBG] osdmap e1431: 8 total, 8 up, 8 in 2021-07-31T02:33:42.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:42 smithi045 conmon[39718]: cluster 2021-07-31T02:33:42.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:42 smithi045 conmon[39718]: 2021- 2021-07-31T02:33:42.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:42 smithi045 conmon[39718]: 07-31T02:33:40.464974+0000 mgr.x (mgr.24542) 1993 : cluster [DBG] pgmap v2670: 73 pgs: 73 active+clean; 733 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 18 KiB/s rd, 4.0 MiB/s wr, 29 op/s 2021-07-31T02:33:42.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:42 smithi045 conmon[39718]: cluster 2021-07-31T02:33:42.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:42 smithi045 conmon[39718]: 2021- 2021-07-31T02:33:42.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:42 smithi045 conmon[39718]: 07-31T02:33:41.388097+0000 mon.a (mon.0) 2398 : cluster [DBG] osdmap e1431: 8 total, 8 up, 8 in 2021-07-31T02:33:42.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:42 smithi045 conmon[33655]: cluster 2021-07-31T02:33:40.464974+0000 mgr.x ( 2021-07-31T02:33:42.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:42 smithi045 conmon[33655]: mgr.24542) 1993 : cluster [DBG] pgmap v2670: 73 pgs: 73 active+clean; 733 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 18 KiB/s rd, 4.0 MiB/s wr, 29 op/s 2021-07-31T02:33:42.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:42 smithi045 conmon[33655]: cluster 2021-07-31T02:33:41.388097+0000 mon.a (mon.0) 2398 : cluster [DBG] osdmap e1431: 8 total, 8 up, 8 in 2021-07-31T02:33:43.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:43 smithi154 conmon[34553]: cluster 2021-07-31T02:33:42.389216+0000 mon.a (mon.0) 2399 : cluster [DBG] osdmap e1432: 8 total, 8 up, 8 in 2021-07-31T02:33:43.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:43 smithi154 conmon[34553]: 2021-07-31T02:33:43.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:43 smithi154 conmon[34553]: cluster 2021-07-31T02:33:42.465520+0000 mgr.x (mgr.24542) 1994 : cluster [DBG] pgmap v2673: 73 pgs: 73 active+clean; 733 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 18 KiB/s rd, 4.0 MiB/s wr, 29 op/s 2021-07-31T02:33:43.776 INFO:journalctl@ceph.osd.0.smithi045.stdout:Jul 31 02:33:43 smithi045 conmon[43243]: debug 2021-07-31T02:33:43.416+0000 7fdf6c791700 -1 osd.0 pg_epoch: 1433 pg[75.0( v 1427'108 (0'0,1427'108] local-lis/les=1308/1309 n=16 ec=1308/1308 lis/c=1308/1308 les/c/f=1309/1309/0 sis=1308) [0,5,7] r=0 lpr=1308 crt=1427'108 lcod 1427'106 mlcod 1427'106 active+clean+snaptrim trimq=[6d~1]] removing snap head 2021-07-31T02:33:43.776 INFO:journalctl@ceph.osd.3.smithi045.stdout:Jul 31 02:33:43 smithi045 conmon[54013]: debug 2021-07-31T02:33:43.426+0000 7f64f78c3700 -1 osd.3 pg_epoch: 1433 pg[75.5( v 1432'147 (0'0,1432'147] local-lis/les=1308/1309 n=18 ec=1308/1308 lis/c=1308/1308 les/c/f=1309/1309/0 sis=1308) [3,4,1] r=0 lpr=1308 crt=1432'147 lcod 1432'146 mlcod 1432'146 active+clean+snaptrim trimq=[6d~1]] removing snap head 2021-07-31T02:33:43.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:43 smithi045 conmon[39718]: cluster 2021-07-31T02:33:42.389216+0000 mon.a ( 2021-07-31T02:33:43.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:43 smithi045 conmon[39718]: mon.0) 2399 : cluster [DBG] osdmap e1432: 8 total, 8 up, 8 in 2021-07-31T02:33:43.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:43 smithi045 conmon[39718]: cluster 2021-07-31T02:33:42.465520 2021-07-31T02:33:43.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:43 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 1994 : cluster [DBG] pgmap v2673: 73 pgs: 73 active+clean; 733 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 18 KiB/s rd, 4.0 MiB/s wr, 29 op/s 2021-07-31T02:33:43.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:43 smithi045 conmon[33655]: cluster 2021-07-31T02:33:42.389216+0000 mon.a (mon.0) 2399 : cluster [DBG] osdmap e1432: 8 total, 8 up, 8 in 2021-07-31T02:33:43.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:43 smithi045 conmon[33655]: cluster 2021-07-31T02:33:42.465520+0000 mgr.x (mgr.24542) 1994 : cluster [DBG] pgmap v2673: 73 pgs: 73 active+clean; 733 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 18 KiB/s rd, 4.0 MiB/s wr, 29 op/s 2021-07-31T02:33:44.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:44 smithi154 conmon[34553]: cluster 2021-07-31T02:33:43.399812+0000 mon.a (mon.0) 2400 : cluster [DBG] osdmap e1433: 8 total, 8 up, 8 in 2021-07-31T02:33:44.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:44 smithi045 conmon[39718]: cluster 2021-07-31T02:33:43.399812+0000 mon.a ( 2021-07-31T02:33:44.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:44 smithi045 conmon[39718]: mon.0) 2400 : cluster [DBG] osdmap e1433: 8 total, 8 up, 8 in 2021-07-31T02:33:44.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:44 smithi045 conmon[33655]: cluster 2021-07-31T02:33:43.399812+0000 mon.a (mon.0) 2400 : cluster [DBG] osdmap e1433: 8 total, 8 up, 8 in 2021-07-31T02:33:44.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:44 smithi045 conmon[33655]: 2021-07-31T02:33:45.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:45 smithi154 conmon[34553]: cluster 2021-07-31T02:33: 2021-07-31T02:33:45.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:45 smithi154 conmon[34553]: 44.406123+0000 mon.a (mon.0) 2401 : cluster [DBG] osdmap e1434: 8 total, 8 up, 8 in 2021-07-31T02:33:45.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:45 smithi154 conmon[34553]: cluster 2021-07-31T02:33:44.465987+0000 mgr.x (mgr.24542) 1995 : 2021-07-31T02:33:45.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:45 smithi154 conmon[34553]: cluster [DBG] pgmap v2676: 73 pgs: 73 active+clean; 753 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 38 MiB/s rd, 6.0 MiB/s wr, 249 op/s 2021-07-31T02:33:45.775 INFO:journalctl@ceph.osd.3.smithi045.stdout:Jul 31 02:33:45 smithi045 conmon[54013]: debug 2021-07-31T02:33:45.456+0000 7f64fa8c9700 -1 osd.3 pg_epoch: 1435 pg[75.1f( v 1435'104 (0'0,1435'104] local-lis/les=1308/1309 n=11 ec=1308/1308 lis/c=1308/1308 les/c/f=1309/1309/0 sis=1308) [3,0,6] r=0 lpr=1308 crt=1435'103 lcod 1435'103 mlcod 1435'103 active+clean+snaptrim trimq=[69~1] ps=[6b~1,6d~1]] removing snap head 2021-07-31T02:33:45.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:45 smithi045 conmon[33655]: cluster 2021-07-31T02:33:44. 2021-07-31T02:33:45.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:45 smithi045 conmon[33655]: 406123+0000 mon.a (mon.0) 2401 : cluster [DBG] osdmap e1434: 8 total, 8 up, 8 in 2021-07-31T02:33:45.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:45 smithi045 conmon[33655]: cluster 2021-07-31T02:33:44.465987+0000 mgr.x (mgr. 2021-07-31T02:33:45.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:45 smithi045 conmon[33655]: 24542) 1995 : cluster [DBG] pgmap v2676: 73 pgs: 73 active+clean; 753 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 38 MiB/s rd, 6.0 MiB/s wr, 249 op/s 2021-07-31T02:33:45.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:45 smithi045 conmon[39718]: cluster 2021-07-31T02:33: 2021-07-31T02:33:45.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:45 smithi045 conmon[39718]: 44.406123+0000 mon.a (mon.0) 2401 : cluster [DBG] osdmap e1434: 8 total, 8 up, 8 in 2021-07-31T02:33:45.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:45 smithi045 conmon[39718]: cluster 2021-07-31T02: 2021-07-31T02:33:45.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:45 smithi045 conmon[39718]: 33:44.465987+0000 mgr.x (mgr.24542) 1995 : cluster 2021-07-31T02:33:45.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:45 smithi045 conmon[39718]: [DBG] pgmap v2676: 73 pgs: 73 active+clean; 753 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 38 MiB/s rd, 6.0 MiB/s wr, 249 op/s 2021-07-31T02:33:45.862 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMigration.SnapsSnapExpandAfterPrepare (14055 ms) 2021-07-31T02:33:45.863 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMigration.Clone 2021-07-31T02:33:45.882 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 83886080 2021-07-31T02:33:45.888 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:33:45.889 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMigration.Clone (26 ms) 2021-07-31T02:33:45.889 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMigration.CloneParent 2021-07-31T02:33:45.907 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 83886080 2021-07-31T02:33:45.911 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:33:45.913 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMigration.CloneParent (25 ms) 2021-07-31T02:33:45.913 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMigration.CloneUpdateAfterPrepare 2021-07-31T02:33:45.931 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 83886080 2021-07-31T02:33:45.934 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:33:45.937 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMigration.CloneUpdateAfterPrepare (23 ms) 2021-07-31T02:33:45.937 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMigration.TriggerAssertSnapcSeq 2021-07-31T02:33:45.956 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 83886080 2021-07-31T02:33:45.960 INFO:tasks.workunit.client.0.smithi045.stdout:write: A 41943040~10 2021-07-31T02:33:45.965 INFO:tasks.workunit.client.0.smithi045.stdout:snap_create: snap1 2021-07-31T02:33:46.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:46 smithi154 conmon[34553]: cluster 2021-07-31T02:33:45.413245+0000 mon.a (mon.0) 2402 : 2021-07-31T02:33:46.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:46 smithi154 conmon[34553]: cluster [DBG] osdmap e1435: 8 total, 8 up, 8 in 2021-07-31T02:33:46.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:46 smithi045 conmon[39718]: cluster 2021-07-31T02:33:45.413245+0000 mon.a ( 2021-07-31T02:33:46.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:46 smithi045 conmon[39718]: mon.0) 2402 : cluster [DBG] osdmap e1435: 8 total, 8 up, 8 in 2021-07-31T02:33:46.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:46 smithi045 conmon[33655]: cluster 2021-07-31T02:33:45.413245+0000 mon.a (mon.0) 2021-07-31T02:33:46.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:46 smithi045 conmon[33655]: 2402 : cluster [DBG] osdmap e1435: 8 total, 8 up, 8 in 2021-07-31T02:33:47.369 INFO:tasks.workunit.client.0.smithi045.stdout:write: B 41943041~10 2021-07-31T02:33:47.374 INFO:tasks.workunit.client.0.smithi045.stdout:migration_prepare 2021-07-31T02:33:47.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:47 smithi154 conmon[34553]: cluster 2021-07-31T02:33:46.413344+0000 mon.a (mon.0) 2021-07-31T02:33:47.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:47 smithi154 conmon[34553]: 2403 : cluster [DBG] osdmap e1436: 8 total, 8 up, 8 in 2021-07-31T02:33:47.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:47 smithi154 conmon[34553]: cluster 2021-07-31T02:33:46.466406+0000 mgr.x (mgr.24542) 1996 : cluster [DBG] pgmap v2679: 73 pgs: 73 active+clean; 753 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 38 MiB/s rd, 6.0 MiB/s wr, 249 op/s 2021-07-31T02:33:47.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:47 smithi154 conmon[34553]: cluster 2021 2021-07-31T02:33:47.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:47 smithi154 conmon[34553]: -07-31T02:33:47.362547+0000 mon.a (mon.0) 2404 : cluster [DBG] osdmap e1437: 8 total, 8 up, 8 in 2021-07-31T02:33:47.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:47 smithi045 conmon[39718]: cluster 2021-07-31T02:33:46.413344+0000 mon.a ( 2021-07-31T02:33:47.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:47 smithi045 conmon[39718]: mon.0) 2403 : cluster [DBG] osdmap e1436: 8 total, 8 up, 8 in 2021-07-31T02:33:47.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:47 smithi045 conmon[39718]: cluster 2021-07-31T02:33:46 2021-07-31T02:33:47.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:47 smithi045 conmon[39718]: .466406+0000 mgr.x (mgr.24542) 1996 : cluster [DBG] pgmap v2679: 73 pgs: 73 active+clean; 753 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 38 MiB/s rd, 6.0 MiB/s wr, 249 op/s 2021-07-31T02:33:47.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:47 smithi045 conmon[39718]: cluster 2021 2021-07-31T02:33:47.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:47 smithi045 conmon[39718]: -07-31T02:33:47.362547+0000 mon.a ( 2021-07-31T02:33:47.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:47 smithi045 conmon[39718]: mon.0) 2404 : cluster [DBG] osdmap e1437: 8 total, 8 up, 8 in 2021-07-31T02:33:47.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:47 smithi045 conmon[33655]: cluster 2021-07-31T02:33:46. 2021-07-31T02:33:47.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:47 smithi045 conmon[33655]: 413344+0000 mon.a (mon.0) 2403 : cluster [DBG] osdmap e1436: 8 total, 8 up, 8 in 2021-07-31T02:33:47.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:47 smithi045 conmon[33655]: cluster 2021 2021-07-31T02:33:47.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:47 smithi045 conmon[33655]: -07-31T02:33:46.466406+0000 2021-07-31T02:33:47.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:47 smithi045 conmon[33655]: mgr.x (mgr.24542) 1996 : 2021-07-31T02:33:47.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:47 smithi045 conmon[33655]: cluster [DBG] pgmap v2679: 73 pgs: 73 active+clean; 753 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 38 MiB/s rd, 6.0 MiB/s wr, 249 op/s 2021-07-31T02:33:47.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:47 smithi045 conmon[33655]: cluster 2021-07-31 2021-07-31T02:33:47.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:47 smithi045 conmon[33655]: T02:33:47.362547+0000 mon.a ( 2021-07-31T02:33:47.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:47 smithi045 conmon[33655]: mon.0) 2404 : cluster [DBG] osdmap e1437: 8 total, 8 up, 8 in 2021-07-31T02:33:48.584 INFO:tasks.workunit.client.0.smithi045.stdout:write: C 41943042~10 2021-07-31T02:33:48.592 INFO:tasks.workunit.client.0.smithi045.stdout:snap_create: snap2 2021-07-31T02:33:49.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:49 smithi154 conmon[34553]: cluster 2021-07-31T02:33:48.420508+0000 mon.a (mon.0) 2405 : cluster [DBG] osdmap e1438: 8 total, 8 up, 8 in 2021-07-31T02:33:49.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:49 smithi154 conmon[34553]: 2021-07-31T02:33:49.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:49 smithi154 conmon[34553]: cluster 2021-07-31T02:33:48.466795+0000 mgr.x (mgr.24542) 1997 : cluster [DBG] pgmap v2682: 73 pgs: 73 active+clean; 753 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail 2021-07-31T02:33:49.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:49 smithi045 conmon[39718]: cluster 2021-07-31T02:33:48.420508+0000 mon.a (mon 2021-07-31T02:33:49.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:49 smithi045 conmon[39718]: .0) 2405 : cluster [DBG] osdmap e1438: 8 total, 8 up, 8 in 2021-07-31T02:33:49.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:49 smithi045 conmon[39718]: cluster 2021-07-31T02:33:48.466795+0000 mgr.x (mgr.24542) 1997 : cluster [DBG] pgmap v2682: 73 pgs: 73 active+clean; 753 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail 2021-07-31T02:33:49.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:49 smithi045 conmon[33655]: cluster 2021-07-31T02:33:48.420508+0000 2021-07-31T02:33:49.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:49 smithi045 conmon[33655]: mon.a (mon.0) 2405 : cluster [DBG] osdmap e1438: 8 total, 8 up, 8 in 2021-07-31T02:33:49.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:49 smithi045 conmon[33655]: cluster 2021-07-31T02:33:48.466795+0000 mgr.x (mgr 2021-07-31T02:33:49.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:49 smithi045 conmon[33655]: .24542) 1997 : cluster [DBG] pgmap v2682: 73 pgs: 73 active+clean; 753 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail 2021-07-31T02:33:50.432 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 41943040 2021-07-31T02:33:50.456 INFO:tasks.workunit.client.0.smithi045.stdout:migration_execute 2021-07-31T02:33:50.507 INFO:tasks.workunit.client.0.smithi045.stdout:migration_commit 2021-07-31T02:33:50.536 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:33:50.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:50 smithi154 conmon[34553]: cluster 2021-07-31T02:33:49.426242+0000 mon.a (mon.0) 2406 : cluster 2021-07-31T02:33:50.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:50 smithi154 conmon[34553]: [DBG] osdmap e1439: 8 total, 8 up, 8 in 2021-07-31T02:33:50.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:50 smithi045 conmon[33655]: cluster 2021-07-31T02:33 2021-07-31T02:33:50.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:50 smithi045 conmon[33655]: :49.426242+0000 mon.a (mon.0) 2406 : cluster [DBG] osdmap e1439: 8 total, 8 up, 8 in 2021-07-31T02:33:50.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:50 smithi045 conmon[39718]: cluster 2021-07-31T02:33 2021-07-31T02:33:50.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:50 smithi045 conmon[39718]: :49.426242+0000 mon.a (mon.0) 2406 : cluster [DBG] osdmap e1439: 8 total, 8 up, 8 in 2021-07-31T02:33:50.814 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:33:50.815 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 48m ago 49m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:33:50.815 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 46m ago 49m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:33:50.815 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 46m ago 52m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:33:50.816 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 48m ago 54m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:33:50.816 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 48m ago 55m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:33:50.816 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 46m ago 53m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:33:50.816 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 48m ago 53m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:33:50.816 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 48m ago 49m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:33:50.817 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 46m ago 49m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:33:50.817 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 48m ago 52m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:33:50.817 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 48m ago 52m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:33:50.817 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 48m ago 51m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:33:50.818 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 48m ago 51m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:33:50.818 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 46m ago 51m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:33:50.818 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 46m ago 50m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:33:50.818 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 46m ago 50m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:33:50.819 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 46m ago 50m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:33:50.819 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 46m ago 49m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:33:51.150 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:33:51.150 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:33:51.151 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:33:51.151 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:33:51.151 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:33:51.151 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:33:51.152 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:33:51.152 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:33:51.152 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:33:51.152 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:33:51.152 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:33:51.153 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:33:51.153 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:33:51.153 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:33:51.154 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:33:51.154 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:33:51.154 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:33:51.592 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMigration.TriggerAssertSnapcSeq (5655 ms) 2021-07-31T02:33:51.592 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMigration.SnapTrimBeforePrepare 2021-07-31T02:33:51.608 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 83886080 2021-07-31T02:33:51.612 INFO:tasks.workunit.client.0.smithi045.stdout:write: A 41943040~10 2021-07-31T02:33:51.618 INFO:tasks.workunit.client.0.smithi045.stdout:snap_create: snap1 2021-07-31T02:33:51.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:51 smithi154 conmon[34553]: cluster 2021-07-31T02:33:51.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:51 smithi154 conmon[34553]: 2021-07-31T02:33:50.426889+0000 mon.a 2021-07-31T02:33:51.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:51 smithi154 conmon[34553]: (mon.0) 2407 : cluster [DBG] osdmap e1440: 8 total, 8 up, 8 in 2021-07-31T02:33:51.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:51 smithi154 conmon[34553]: cluster 2021- 2021-07-31T02:33:51.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:51 smithi154 conmon[34553]: 07-31T02:33:50.467217+0000 mgr.x (mgr.24542) 1998 : cluster [DBG] pgmap v2685: 73 pgs: 73 active+clean; 733 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 40 MiB/s rd, 13 KiB/s wr, 94 op/s 2021-07-31T02:33:51.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:51 smithi154 conmon[34553]: audit 2021-07-31T02:33:51.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:51 smithi154 conmon[34553]: 2021-07-31T02:33:50.523657+0000 mgr.x 2021-07-31T02:33:51.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:51 smithi154 conmon[34553]: (mgr.24542) 1999 : audit [DBG] from='client.17442 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:33:51.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:51 smithi154 conmon[34553]: audit 2021 2021-07-31T02:33:51.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:51 smithi154 conmon[34553]: -07-31T02:33:50.807059+0000 2021-07-31T02:33:51.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:51 smithi154 conmon[34553]: mgr.x (mgr.24542) 2000 : audit [DBG] from='client.17448 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:33:51.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:51 smithi154 conmon[34553]: audit 2021 2021-07-31T02:33:51.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:51 smithi154 conmon[34553]: -07-31T02:33:51.150358+0000 mon.c ( 2021-07-31T02:33:51.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:51 smithi154 conmon[34553]: mon.1) 118 : audit [DBG] from='client.? 172.21.15.45:0/2599521097' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:33:51.776 INFO:journalctl@ceph.osd.3.smithi045.stdout:Jul 31 02:33:51 smithi045 conmon[54013]: debug 2021-07-31T02:33:51.486+0000 7f64fd0ce700 -1 osd.3 pg_epoch: 1441 pg[75.12( v 1441'201 (0'0,1441'201] local-lis/les=1308/1309 n=23 ec=1308/1308 lis/c=1308/1308 les/c/f=1309/1309/0 sis=1308) [3,2,0] r=0 lpr=1308 crt=1441'200 lcod 1441'200 mlcod 1441'200 active+clean+snaptrim trimq=[77~1] ps=[69~1,6b~1]] removing snap head 2021-07-31T02:33:51.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:51 smithi045 conmon[33655]: cluster 2021-07-31T02:33:50.426889+0000 mon.a (mon. 2021-07-31T02:33:51.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:51 smithi045 conmon[33655]: 0) 2407 : cluster [DBG] osdmap e1440: 8 total, 8 up, 8 in 2021-07-31T02:33:51.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:51 smithi045 conmon[33655]: cluster 2021-07-31T02:33:50.467217+0000 mgr.x (mgr.24542) 1998 : cluster [DBG] pgmap v2685: 73 pgs: 73 active+clean; 733 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 40 MiB/s rd, 13 KiB/s wr, 94 op/s 2021-07-31T02:33:51.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:51 smithi045 conmon[33655]: audit 2021- 2021-07-31T02:33:51.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:51 smithi045 conmon[33655]: 07-31T02:33:50.523657+0000 mgr.x (mgr.24542) 1999 : audit [DBG] from='client.17442 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:33:51.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:51 smithi045 conmon[33655]: audit 2021-07-31T02:33:51.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:51 smithi045 conmon[33655]: 2021-07-31T02:33:50.807059+0000 mgr.x (mgr.24542) 2000 : audit 2021-07-31T02:33:51.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:51 smithi045 conmon[33655]: [DBG] from='client.17448 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:33:51.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:51 smithi045 conmon[33655]: audit 2021-07-31T02:33:51.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:51 smithi045 conmon[33655]: 2021-07-31T02:33:51.150358+0000 mon.c (mon.1) 118 2021-07-31T02:33:51.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:51 smithi045 conmon[33655]: : audit [DBG] from='client.? 172.21.15.45:0/2599521097' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:33:51.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:51 smithi045 conmon[39718]: cluster 2021-07-31T02:33:50.426889 2021-07-31T02:33:51.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:51 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2407 : cluster [DBG] osdmap e1440: 8 total, 8 up, 8 in 2021-07-31T02:33:51.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:51 smithi045 conmon[39718]: cluster 2021-07-31T02:33:51.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:51 smithi045 conmon[39718]: 2021-07-31T02:33:50.467217+0000 mgr.x (mgr.24542) 1998 2021-07-31T02:33:51.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:51 smithi045 conmon[39718]: : cluster [DBG] pgmap v2685: 73 pgs: 73 active+clean; 733 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 40 MiB/s rd, 13 KiB/s wr, 94 op/s 2021-07-31T02:33:51.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:51 smithi045 conmon[39718]: audit 2021-07-31T02:33:50.523657+0000 mgr.x (mgr.24542 2021-07-31T02:33:51.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:51 smithi045 conmon[39718]: ) 1999 : audit [DBG] from='client.17442 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:33:51.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:51 smithi045 conmon[39718]: audit 2021-07-31T02:33:51.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:51 smithi045 conmon[39718]: 2021-07-31T02:33:50.807059+0000 mgr.x (mgr.24542) 2000 : audit [DBG] 2021-07-31T02:33:51.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:51 smithi045 conmon[39718]: from='client.17448 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:33:51.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:51 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T02:33:51.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:51 smithi045 conmon[39718]: T02:33:51.150358+0000 mon.c (mon.1) 118 : audit [DBG] from='client.? 172.21.15.45:0/2599521097' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:33:52.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:52 smithi154 conmon[34553]: cluster 2021-07-31T02:33:51.436644+0000 mon.a ( 2021-07-31T02:33:52.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:52 smithi154 conmon[34553]: mon.0) 2408 : cluster [DBG] osdmap e1441: 8 total, 8 up, 8 in 2021-07-31T02:33:52.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:52 smithi154 conmon[34553]: cluster 2021-07-31T02:33:52.364140+0000 mon.a (mon.0) 2409 : cluster [DBG] osdmap e1442: 8 total, 8 up, 8 in 2021-07-31T02:33:52.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:52 smithi045 conmon[33655]: cluster 2021-07-31T02:33:51.436644+0000 mon.a (mon.0) 2408 : cluster 2021-07-31T02:33:52.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:52 smithi045 conmon[33655]: [DBG] osdmap e1441: 8 total, 8 up, 8 in 2021-07-31T02:33:52.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:52 smithi045 conmon[33655]: cluster 2021-07-31T02:33:52.364140+0000 mon.a (mon.0) 2409 : cluster [DBG] osdmap e1442: 8 total, 8 up, 8 in 2021-07-31T02:33:52.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:52 smithi045 conmon[39718]: cluster 2021-07-31T02:33:51 2021-07-31T02:33:52.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:52 smithi045 conmon[39718]: .436644+0000 mon.a (mon.0) 2408 : cluster [DBG] osdmap e1441: 8 total, 8 up, 8 in 2021-07-31T02:33:52.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:52 smithi045 conmon[39718]: cluster 2021-07-31T02:33:52.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:52 smithi045 conmon[39718]: 2021-07-31T02:33 2021-07-31T02:33:52.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:52 smithi045 conmon[39718]: :52.364140+0000 mon.a (mon 2021-07-31T02:33:52.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:52 smithi045 conmon[39718]: .0) 2409 : cluster [DBG] 2021-07-31T02:33:52.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:52 smithi045 conmon[39718]: osdmap e1442: 8 total, 8 up, 8 in 2021-07-31T02:33:53.374 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 41943040 2021-07-31T02:33:53.392 INFO:tasks.workunit.client.0.smithi045.stdout:migration_prepare 2021-07-31T02:33:54.510 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 83886080 2021-07-31T02:33:54.514 INFO:tasks.workunit.client.0.smithi045.stdout:snap_create: snap3 2021-07-31T02:33:54.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:54 smithi154 conmon[34553]: cluster 2021-07-31T02:33:52.467749+0000 mgr.x (mgr.24542) 2001 : cluster [DBG] pgmap v2688: 73 pgs: 73 active+clean; 733 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 40 MiB/s rd, 13 KiB/s wr, 94 op/s 2021-07-31T02:33:54.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:54 smithi154 conmon[34553]: cluster 2021-07-31T 2021-07-31T02:33:54.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:54 smithi154 conmon[34553]: 02:33:53.370842+0000 mon.a (mon.0) 2410 : cluster [DBG] osdmap e1443: 8 total, 8 up, 8 in 2021-07-31T02:33:54.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:54 smithi045 conmon[33655]: cluster 2021-07-31T02:33:52.467749+0000 mgr.x (mgr.24542) 2001 : cluster [DBG] 2021-07-31T02:33:54.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:54 smithi045 conmon[33655]: pgmap v2688: 73 pgs: 73 active+clean; 733 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 40 MiB/s rd, 13 KiB/s wr, 94 op/s 2021-07-31T02:33:54.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:54 smithi045 conmon[33655]: cluster 2021-07-31T02:33:53.370842+0000 mon.a (mon.0) 2410 : cluster [DBG] osdmap e1443: 8 total, 8 up, 8 in 2021-07-31T02:33:54.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:54 smithi045 conmon[39718]: cluster 2021-07-31T02:33:52.467749+0000 mgr.x (mgr. 2021-07-31T02:33:54.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:54 smithi045 conmon[39718]: 24542) 2001 : cluster [DBG] pgmap v2688: 73 pgs: 73 active+clean; 733 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 40 MiB/s rd, 13 KiB/s wr, 94 op/s 2021-07-31T02:33:54.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:54 smithi045 conmon[39718]: cluster 2021-07-31T02: 2021-07-31T02:33:54.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:54 smithi045 conmon[39718]: 33:53.370842+0000 mon.a (mon.0) 2410 : cluster [DBG] osdmap e1443: 8 total, 8 up, 8 in 2021-07-31T02:33:55.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:55 smithi154 conmon[34553]: cluster 2021-07-31T02:33:54.376454+0000 mon.a (mon.0) 2411 : cluster [DBG] 2021-07-31T02:33:55.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:55 smithi154 conmon[34553]: osdmap e1444: 8 total, 8 up, 8 in 2021-07-31T02:33:55.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:55 smithi045 conmon[33655]: cluster 2021-07-31T02:33:54.376454+0000 mon.a (mon.0) 2411 2021-07-31T02:33:55.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:55 smithi045 conmon[33655]: : cluster [DBG] osdmap e1444: 8 total, 8 up, 8 in 2021-07-31T02:33:55.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:55 smithi045 conmon[39718]: cluster 2021-07-31T02:33:54.376454+0000 mon.a 2021-07-31T02:33:55.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:55 smithi045 conmon[39718]: (mon.0) 2411 : cluster [DBG] osdmap e1444: 8 total, 8 up, 8 in 2021-07-31T02:33:56.394 INFO:tasks.workunit.client.0.smithi045.stdout:write: B 41943040~10 2021-07-31T02:33:56.403 INFO:tasks.workunit.client.0.smithi045.stdout:snap_create: snap4 2021-07-31T02:33:56.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:56 smithi154 conmon[34553]: cluster 2021-07-31T02:33:54.468201+0000 mgr.x (mgr.24542) 2002 2021-07-31T02:33:56.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:56 smithi154 conmon[34553]: : cluster [DBG] pgmap v2691: 73 pgs: 73 active+clean; 733 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 81 KiB/s rd, 9.7 KiB/s wr, 120 op/s 2021-07-31T02:33:56.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:56 smithi154 conmon[34553]: cluster 2021-07-31T02:33:55.382543+0000 mon.a (mon.0) 2412 : cluster [DBG] osdmap e1445: 8 total, 8 up, 8 in 2021-07-31T02:33:56.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:56 smithi045 conmon[39718]: cluster 2021-07-31T02:33:54. 2021-07-31T02:33:56.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:56 smithi045 conmon[39718]: 468201+0000 mgr.x (mgr.24542) 2002 : cluster [DBG] pgmap v2691: 73 pgs: 73 active+clean; 733 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 81 KiB/s rd, 9.7 KiB/s wr, 120 op/s 2021-07-31T02:33:56.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:56 smithi045 conmon[39718]: cluster 2021-07-31T02:33:55 2021-07-31T02:33:56.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:56 smithi045 conmon[39718]: .382543+0000 mon.a (mon.0) 2412 : cluster [DBG] 2021-07-31T02:33:56.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:56 smithi045 conmon[39718]: osdmap e1445: 8 total, 8 up, 8 in 2021-07-31T02:33:56.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:56 smithi045 conmon[33655]: cluster 2021-07-31T02:33:54.468201 2021-07-31T02:33:56.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:56 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 2002 : cluster [DBG] pgmap v2691: 73 pgs: 73 active+clean; 733 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 81 KiB/s rd, 9.7 KiB/s wr, 120 op/s 2021-07-31T02:33:56.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:56 smithi045 conmon[33655]: cluster 2021-07-31 2021-07-31T02:33:56.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:56 smithi045 conmon[33655]: T02:33:55.382543+0000 mon.a (mon.0) 2412 : cluster [DBG] osdmap e1445: 8 total, 8 up, 8 in 2021-07-31T02:33:56.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:56 smithi045 conmon[33655]: 2021-07-31T02:33:57.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:57 smithi154 conmon[34553]: cluster 2021-07-31T02:33:56.388894+0000 mon.a (mon.0) 2413 : 2021-07-31T02:33:57.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:57 smithi154 conmon[34553]: cluster [DBG] osdmap e1446: 8 total, 8 up, 8 in 2021-07-31T02:33:57.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:57 smithi154 conmon[34553]: cluster 2021-07-31T02:33:57.365606+0000 mon.a (mon.0) 2414 : cluster [DBG] osdmap e1447: 8 total, 8 up, 8 in 2021-07-31T02:33:57.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:57 smithi045 conmon[33655]: cluster 2021-07-31T02:33:56.388894+0000 mon.a (mon.0) 2413 : cluster 2021-07-31T02:33:57.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:57 smithi045 conmon[33655]: [DBG] osdmap e1446: 8 total, 8 up, 8 in 2021-07-31T02:33:57.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:57 smithi045 conmon[33655]: cluster 2021-07-31T02:33:57.365606+0000 mon.a (mon.0) 2414 : cluster [DBG] osdmap e1447: 8 total, 8 up, 8 in 2021-07-31T02:33:57.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:57 smithi045 conmon[39718]: cluster 2021-07-31T02:33:56.388894+0000 mon.a ( 2021-07-31T02:33:57.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:57 smithi045 conmon[39718]: mon.0) 2413 : cluster [DBG] osdmap e1446: 8 total, 8 up, 8 in 2021-07-31T02:33:57.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:57 smithi045 conmon[39718]: cluster 2021-07-31T02:33:57.365606+0000 2021-07-31T02:33:57.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:57 smithi045 conmon[39718]: mon.a (mon.0) 2414 : cluster [DBG] osdmap e1447: 8 total, 8 up, 8 in 2021-07-31T02:33:58.377 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 41943040 2021-07-31T02:33:58.397 INFO:tasks.workunit.client.0.smithi045.stdout:migration_execute 2021-07-31T02:33:58.449 INFO:tasks.workunit.client.0.smithi045.stdout:migration_commit 2021-07-31T02:33:58.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:58 smithi154 conmon[34553]: cluster 2021-07-31T02:33:56.468844+0000 mgr.x (mgr.24542 2021-07-31T02:33:58.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:58 smithi154 conmon[34553]: ) 2003 : cluster [DBG] pgmap v2694: 73 pgs: 73 active+clean; 733 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 81 KiB/s rd, 9.7 KiB/s wr, 120 op/s 2021-07-31T02:33:58.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:58 smithi154 conmon[34553]: cluster 2021-07-31T02:33:58.371729+0000 mon.a (mon.0) 2415 : 2021-07-31T02:33:58.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:58 smithi154 conmon[34553]: cluster [DBG] osdmap e1448: 8 total, 8 up, 8 in 2021-07-31T02:33:58.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:58 smithi045 conmon[33655]: cluster 2021-07-31T02:33:56.468844+0000 2021-07-31T02:33:58.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:58 smithi045 conmon[33655]: mgr.x (mgr.24542) 2003 : cluster [DBG] pgmap v2694: 73 pgs: 73 active+clean; 733 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 81 KiB/s rd, 9.7 KiB/s wr, 120 op/s 2021-07-31T02:33:58.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:58 smithi045 conmon[33655]: cluster 2021-07-31T02:33:58.371729+0000 mon.a (mon.0) 2415 : cluster [DBG] osdmap e1448: 8 total, 8 up, 8 in 2021-07-31T02:33:58.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:58 smithi045 conmon[39718]: cluster 2021-07-31T02:33:56.468844+0000 mgr.x (mgr 2021-07-31T02:33:58.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:58 smithi045 conmon[39718]: .24542) 2003 : cluster [DBG] pgmap v2694: 73 pgs: 73 active+clean; 733 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 81 KiB/s rd, 9.7 KiB/s wr, 120 op/s 2021-07-31T02:33:58.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:58 smithi045 conmon[39718]: cluster 2021-07-31T02:33:58.371729+0000 mon.a (mon.0) 2021-07-31T02:33:58.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:58 smithi045 conmon[39718]: 2415 : cluster [DBG] osdmap e1448: 8 total, 8 up, 8 in 2021-07-31T02:33:59.600 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMigration.SnapTrimBeforePrepare (8008 ms) 2021-07-31T02:33:59.600 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMigration.AbortInUseImage 2021-07-31T02:33:59.622 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 83886080 2021-07-31T02:33:59.623 INFO:tasks.workunit.client.0.smithi045.stdout:migration_prepare 2021-07-31T02:33:59.752 INFO:journalctl@ceph.osd.6.smithi154.stdout:Jul 31 02:33:59 smithi154 conmon[45576]: debug 2021-07-31T02:33:59.427+0000 7f47c50ad700 -1 osd.6 pg_epoch: 1449 pg[75.a( v 1448'122 (0'0,1448'122] local-lis/les=1308/1309 n=10 ec=1308/1308 lis/c=1308/1308 les/c/f=1309/1309/0 sis=1308) [6,7,4] r=0 lpr=1308 crt=1443'120 lcod 1443'121 mlcod 1443'121 active+clean+snaptrim trimq=[7d~1]] removing snap head 2021-07-31T02:33:59.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:59 smithi154 conmon[34553]: cluster 2021-07-31T02:33:59.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:59 smithi154 conmon[34553]: 2021-07-31T02:33:58.469305+0000 2021-07-31T02:33:59.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:33:59 smithi154 conmon[34553]: mgr.x (mgr.24542) 2004 : cluster [DBG] pgmap v2697: 73 pgs: 73 active+clean; 733 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail 2021-07-31T02:33:59.761 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMigration.AbortInUseImage (162 ms) 2021-07-31T02:33:59.953 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMigration.AbortWithoutSnapshots 2021-07-31T02:33:59.955 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:59 smithi045 conmon[33655]: cluster 2021-07-31T02:33:58.469305+0000 mgr.x (mgr.24542) 2004 : cluster 2021-07-31T02:33:59.955 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:33:59 smithi045 conmon[33655]: [DBG] pgmap v2697: 73 pgs: 73 active+clean; 733 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail 2021-07-31T02:33:59.956 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:59 smithi045 conmon[39718]: cluster 2021-07-31T02:33:58.469305+0000 mgr.x (mgr.24542) 2004 : 2021-07-31T02:33:59.957 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:33:59 smithi045 conmon[39718]: cluster [DBG] pgmap v2697: 73 pgs: 73 active+clean; 733 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail 2021-07-31T02:33:59.958 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 83886080 2021-07-31T02:33:59.959 INFO:tasks.workunit.client.0.smithi045.stdout:write: 1 0~8388609 2021-07-31T02:33:59.959 INFO:tasks.workunit.client.0.smithi045.stdout:write: 1 16777218~8388609 2021-07-31T02:33:59.959 INFO:tasks.workunit.client.0.smithi045.stdout:migration_prepare 2021-07-31T02:34:00.055 INFO:tasks.workunit.client.0.smithi045.stdout:write: 1 0~8388609 2021-07-31T02:34:00.096 INFO:tasks.workunit.client.0.smithi045.stdout:write: 1 16777218~8388609 2021-07-31T02:34:00.161 INFO:tasks.workunit.client.0.smithi045.stdout:migration_abort 2021-07-31T02:34:00.334 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMigration.AbortWithoutSnapshots (572 ms) 2021-07-31T02:34:00.334 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMigration.AbortWithSnapshots 2021-07-31T02:34:00.352 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 83886080 2021-07-31T02:34:00.355 INFO:tasks.workunit.client.0.smithi045.stdout:write: 1 0~8388609 2021-07-31T02:34:00.401 INFO:tasks.workunit.client.0.smithi045.stdout:snap_create: snap1 2021-07-31T02:34:00.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:00 smithi154 conmon[34553]: cluster 2021-07-31T02:33:59.401056+0000 mon.a (mon.0) 2416 : cluster 2021-07-31T02:34:00.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:00 smithi154 conmon[34553]: [DBG] osdmap e1449: 8 total, 8 up, 8 in 2021-07-31T02:34:00.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:00 smithi045 conmon[33655]: cluster 2021-07-31T02:33:59.401056 2021-07-31T02:34:00.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:00 smithi045 conmon[33655]: +0000 mon.a (mon.0) 2416 : cluster [DBG] osdmap e1449: 8 total, 8 up, 8 in 2021-07-31T02:34:00.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:00 smithi045 conmon[39718]: cluster 2021-07-31T02:33:59. 2021-07-31T02:34:00.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:00 smithi045 conmon[39718]: 401056+0000 mon.a (mon.0) 2416 : cluster [DBG] osdmap e1449: 8 total, 8 up, 8 in 2021-07-31T02:34:01.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:01 smithi154 conmon[34553]: cluster 2021-07-31T02:34:00.469790+0000 mgr.x (mgr.24542 2021-07-31T02:34:01.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:01 smithi154 conmon[34553]: ) 2005 : cluster [DBG] pgmap v2699: 73 pgs: 73 active+clean; 733 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 100 KiB/s rd, 8.6 KiB/s wr, 137 op/s 2021-07-31T02:34:01.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:01 smithi045 conmon[33655]: cluster 2021-07-31T02:34:00.469790+0000 mgr.x (mgr.24542) 2021-07-31T02:34:01.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:01 smithi045 conmon[33655]: 2005 : cluster [DBG] pgmap v2699: 73 pgs: 73 active+clean; 733 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 100 KiB/s rd, 8.6 KiB/s wr, 137 op/s 2021-07-31T02:34:01.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:01 smithi045 conmon[39718]: cluster 2021-07-31T02:34:00.469790+0000 mgr.x (mgr.24542) 2005 : cluster 2021-07-31T02:34:01.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:01 smithi045 conmon[39718]: [DBG] pgmap v2699: 73 pgs: 73 active+clean; 733 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 100 KiB/s rd, 8.6 KiB/s wr, 137 op/s 2021-07-31T02:34:02.418 INFO:tasks.workunit.client.0.smithi045.stdout:write: 1 8388609~8388609 2021-07-31T02:34:02.425 INFO:tasks.workunit.client.0.smithi045.stdout:write: X 0~1000 2021-07-31T02:34:02.426 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 1010~1000 2021-07-31T02:34:02.426 INFO:tasks.workunit.client.0.smithi045.stdout:snap_create: snap2 2021-07-31T02:34:02.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:02 smithi154 conmon[34553]: cluster 2021-07-31T02:34:01.410690+0000 mon.a (mon.0) 2417 : cluster [DBG] osdmap e1450: 8 total, 8 up, 8 in 2021-07-31T02:34:02.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:02 smithi045 conmon[39718]: cluster 2021-07-31T02:34:01.410690+0000 mon.a 2021-07-31T02:34:02.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:02 smithi045 conmon[39718]: (mon.0) 2417 : cluster [DBG] osdmap e1450: 8 total, 8 up, 8 in 2021-07-31T02:34:02.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:02 smithi045 conmon[33655]: cluster 2021-07-31T02:34:01.410690+0000 mon.a ( 2021-07-31T02:34:02.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:02 smithi045 conmon[33655]: mon.0) 2417 : cluster [DBG] osdmap e1450: 8 total, 8 up, 8 in 2021-07-31T02:34:03.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:03 smithi154 conmon[34553]: cluster 2021-07-31T02:34:02.411209+0000 mon.a (mon.0) 2418 : cluster [DBG] osdmap e1451: 8 total, 8 up, 8 in 2021-07-31T02:34:03.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:03 smithi154 conmon[34553]: cluster 2021-07-31T02:34:02.470224+0000 mgr.x (mgr.24542) 2006 : cluster [DBG] pgmap v2702: 73 pgs: 73 active+clean; 733 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 100 KiB/s rd, 8.5 KiB/s wr, 137 op/s 2021-07-31T02:34:03.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:03 smithi045 conmon[39718]: cluster 2021-07-31T02:34:02.411209+0000 mon.a (mon.0) 2418 : cluster [DBG] osdmap e1451: 8 total, 8 up, 8 in 2021-07-31T02:34:03.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:03 smithi045 conmon[39718]: cluster 2021-07-31T02:34:02.470224+0000 mgr.x (mgr.24542) 2006 : cluster [DBG] pgmap v2702: 73 pgs: 73 active+clean; 733 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 100 KiB/s rd, 8.5 KiB/s wr, 137 op/s 2021-07-31T02:34:03.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:03 smithi045 conmon[33655]: cluster 2021-07-31T02:34:02.411209+0000 mon.a (mon.0) 2021-07-31T02:34:03.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:03 smithi045 conmon[33655]: 2418 : cluster [DBG] osdmap e1451: 8 total, 8 up, 8 in 2021-07-31T02:34:03.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:03 smithi045 conmon[33655]: cluster 2021-07-31T02:34:02.470224+0000 mgr.x (mgr.24542) 2006 : cluster [DBG] pgmap v2702: 73 pgs: 73 active+clean; 733 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 100 KiB/s rd, 8.5 KiB/s wr, 137 op/s 2021-07-31T02:34:04.431 INFO:tasks.workunit.client.0.smithi045.stdout:write: X 8388609~1000 2021-07-31T02:34:04.435 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 16777228~1000 2021-07-31T02:34:04.436 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 167772160 2021-07-31T02:34:04.444 INFO:tasks.workunit.client.0.smithi045.stdout:write: 2 83886079~8388609 2021-07-31T02:34:04.452 INFO:tasks.workunit.client.0.smithi045.stdout:snap_create: snap3 2021-07-31T02:34:04.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:04 smithi154 conmon[34553]: cluster 2021-07-31T02:34:03.422648+0000 mon.a (mon.0) 2419 : cluster [DBG] 2021-07-31T02:34:04.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:04 smithi154 conmon[34553]: osdmap e1452: 8 total, 8 up, 8 in 2021-07-31T02:34:04.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:04 smithi045 conmon[33655]: cluster 2021-07-31T02:34:03.422648+0000 mon.a (mon. 2021-07-31T02:34:04.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:04 smithi045 conmon[33655]: 0) 2419 : cluster [DBG] osdmap e1452: 8 total, 8 up, 8 in 2021-07-31T02:34:04.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:04 smithi045 conmon[39718]: cluster 2021-07-31T02:34:03. 2021-07-31T02:34:04.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:04 smithi045 conmon[39718]: 422648+0000 mon.a (mon.0) 2419 : cluster [DBG] osdmap e1452: 8 total, 8 up, 8 in 2021-07-31T02:34:05.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:05 smithi154 conmon[34553]: cluster 2021-07-31T02:34:04.423965+0000 mon.a (mon.0) 2420 : cluster 2021-07-31T02:34:05.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:05 smithi154 conmon[34553]: [DBG] osdmap e1453: 8 total, 8 up, 8 in 2021-07-31T02:34:05.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:05 smithi154 conmon[34553]: cluster 2021-07-31T02:34:04.470621+0000 mgr.x (mgr.24542) 2007 : cluster [DBG] pgmap v2705: 73 pgs: 73 active+clean; 797 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 21 MiB/s rd, 29 MiB/s wr, 185 op/s 2021-07-31T02:34:05.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:05 smithi045 conmon[39718]: cluster 2021-07-31T02:34:04.423965+0000 mon.a (mon.0) 2420 2021-07-31T02:34:05.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:05 smithi045 conmon[39718]: : cluster [DBG] osdmap e1453: 8 total, 8 up, 8 in 2021-07-31T02:34:05.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:05 smithi045 conmon[39718]: cluster 2021-07-31T02:34:04.470621+0000 mgr.x 2021-07-31T02:34:05.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:05 smithi045 conmon[39718]: (mgr.24542) 2007 : cluster [DBG] pgmap v2705: 73 pgs: 73 active+clean; 797 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 21 MiB/s rd, 29 MiB/s wr, 185 op/s 2021-07-31T02:34:05.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:05 smithi045 conmon[33655]: cluster 2021-07-31T02:34:04.423965 2021-07-31T02:34:05.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:05 smithi045 conmon[33655]: +0000 mon.a (mon.0) 2420 : cluster [DBG] osdmap e1453: 8 total, 8 up, 8 in 2021-07-31T02:34:05.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:05 smithi045 conmon[33655]: cluster 2021-07-31T02:34:04.470621+0000 mgr.x (mgr.24542 2021-07-31T02:34:05.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:05 smithi045 conmon[33655]: ) 2007 : cluster [DBG] pgmap v2705: 73 pgs: 73 active+clean; 797 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 21 MiB/s rd, 29 MiB/s wr, 185 op/s 2021-07-31T02:34:06.442 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 83886080 2021-07-31T02:34:06.466 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 83886079~1 2021-07-31T02:34:06.469 INFO:tasks.workunit.client.0.smithi045.stdout:migration_prepare 2021-07-31T02:34:06.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:06 smithi154 conmon[34553]: cluster 2021-07-31T02:34:05.434924+0000 mon.a (mon.0) 2421 : cluster [DBG] 2021-07-31T02:34:06.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:06 smithi154 conmon[34553]: osdmap e1454: 8 total, 8 up, 8 in 2021-07-31T02:34:06.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:06 smithi045 conmon[39718]: cluster 2021-07-31T02:34:05 2021-07-31T02:34:06.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:06 smithi045 conmon[39718]: .434924+0000 mon.a (mon.0) 2421 : cluster [DBG] osdmap e1454: 8 total, 8 up, 8 in 2021-07-31T02:34:06.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:06 smithi045 conmon[33655]: cluster 2021-07-31T02:34:05.434924+0000 mon.a (mon.0) 2421 2021-07-31T02:34:06.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:06 smithi045 conmon[33655]: : cluster [DBG] osdmap e1454: 8 total, 8 up, 8 in 2021-07-31T02:34:07.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:07 smithi154 conmon[34553]: cluster 2021-07-31T02:34:06.435787+0000 mon.a (mon.0) 2422 : cluster [DBG] 2021-07-31T02:34:07.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:07 smithi154 conmon[34553]: osdmap e1455: 8 total, 8 up, 8 in 2021-07-31T02:34:07.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:07 smithi154 conmon[34553]: cluster 2021-07-31T02:34:06.471036+0000 mgr.x (mgr.24542) 2008 : cluster [DBG] pgmap v2708: 73 pgs: 73 active+clean; 797 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 21 MiB/s rd, 29 MiB/s wr, 185 op/s 2021-07-31T02:34:07.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:07 smithi154 conmon[34553]: cluster 2021-07-31T02:34:07.367445+0000 2021-07-31T02:34:07.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:07 smithi154 conmon[34553]: mon.a (mon.0) 2423 : cluster [DBG] osdmap e1456: 8 total, 8 up, 8 in 2021-07-31T02:34:07.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:07 smithi045 conmon[39718]: cluster 2021-07-31T02:34:06.435787+0000 mon.a 2021-07-31T02:34:07.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:07 smithi045 conmon[39718]: (mon.0) 2422 : cluster [DBG] osdmap e1455: 8 total, 8 up, 8 in 2021-07-31T02:34:07.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:07 smithi045 conmon[39718]: cluster 2021-07-31T02:34:06.471036+0000 2021-07-31T02:34:07.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:07 smithi045 conmon[39718]: mgr.x (mgr.24542) 2008 : cluster [DBG] pgmap v2708: 73 pgs: 73 active+clean; 797 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 21 MiB/s rd, 29 MiB/s wr, 185 op/s 2021-07-31T02:34:07.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:07 smithi045 conmon[39718]: cluster 2021-07-31 2021-07-31T02:34:07.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:07 smithi045 conmon[39718]: T02:34:07.367445+0000 mon.a 2021-07-31T02:34:07.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:07 smithi045 conmon[39718]: (mon.0) 2423 : cluster [DBG] 2021-07-31T02:34:07.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:07 smithi045 conmon[39718]: osdmap e1456: 8 total, 8 up, 8 in 2021-07-31T02:34:07.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:07 smithi045 conmon[33655]: cluster 2021-07-31T02:34:06.435787+0000 mon.a (mon.0) 2021-07-31T02:34:07.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:07 smithi045 conmon[33655]: 2422 : cluster [DBG] osdmap e1455: 8 total, 8 up, 8 in 2021-07-31T02:34:07.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:07 smithi045 conmon[33655]: cluster 2021-07-31T02:34:06.471036+0000 mgr.x (mgr.24542) 2008 : cluster [DBG] pgmap v2708: 73 pgs: 73 active+clean; 797 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 21 MiB/s rd, 29 MiB/s wr, 185 op/s 2021-07-31T02:34:07.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:07 smithi045 conmon[33655]: cluster 2021-07-31T02:34:07.367445+0000 mon.a (mon.0) 2423 : cluster [DBG] osdmap e1456: 8 total, 8 up, 8 in 2021-07-31T02:34:09.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:09 smithi154 conmon[34553]: cluster 2021-07-31T02:34:08.373770+0000 mon.a (mon.0) 2424 : cluster [DBG] 2021-07-31T02:34:09.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:09 smithi154 conmon[34553]: osdmap e1457: 8 total, 8 up, 8 in 2021-07-31T02:34:09.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:09 smithi045 conmon[39718]: cluster 2021-07-31T02:34:08.373770+0000 mon.a (mon.0 2021-07-31T02:34:09.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:09 smithi045 conmon[39718]: ) 2424 : cluster [DBG] osdmap e1457: 8 total, 8 up, 8 in 2021-07-31T02:34:09.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:09 smithi045 conmon[33655]: cluster 2021-07-31T02:34:08.373770+0000 mon.a (mon.0) 2424 : cluster 2021-07-31T02:34:09.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:09 smithi045 conmon[33655]: [DBG] osdmap e1457: 8 total, 8 up, 8 in 2021-07-31T02:34:09.832 INFO:tasks.workunit.client.0.smithi045.stdout:write: 1 0~8388609 2021-07-31T02:34:09.912 INFO:tasks.workunit.client.0.smithi045.stdout:write: 1 16777218~8388609 2021-07-31T02:34:10.385 INFO:tasks.workunit.client.0.smithi045.stdout:write: 1 0~8388609 2021-07-31T02:34:10.394 INFO:tasks.workunit.client.0.smithi045.stdout:write: 1 16777218~8388609 2021-07-31T02:34:10.472 INFO:tasks.workunit.client.0.smithi045.stdout:migration_abort 2021-07-31T02:34:10.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:10 smithi154 conmon[34553]: cluster 2021-07-31T02:34:08.471569+0000 mgr.x (mgr.24542 2021-07-31T02:34:10.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:10 smithi154 conmon[34553]: ) 2009 : cluster [DBG] pgmap v2711: 73 pgs: 73 active+clean; 797 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail 2021-07-31T02:34:10.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:10 smithi154 conmon[34553]: cluster 2021-07-31T02:34:09.379777+0000 mon.a (mon.0) 2425 : cluster [DBG] osdmap e1458: 8 total, 8 up, 8 in 2021-07-31T02:34:10.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:10 smithi045 conmon[33655]: cluster 2021-07-31T02:34:08.471569+0000 mgr.x (mgr.24542) 2009 : cluster [DBG] pgmap v2711: 73 pgs: 73 active+clean; 797 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail 2021-07-31T02:34:10.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:10 smithi045 conmon[33655]: cluster 2021-07-31T02:34:09.379777+0000 mon.a (mon.0) 2425 : cluster [DBG] osdmap e1458: 8 total, 8 up, 8 in 2021-07-31T02:34:10.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:10 smithi045 conmon[39718]: cluster 2021-07-31T02:34:08.471569+0000 mgr.x 2021-07-31T02:34:10.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:10 smithi045 conmon[39718]: (mgr.24542) 2009 : cluster [DBG] pgmap v2711: 73 pgs: 73 active+clean; 797 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail 2021-07-31T02:34:10.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:10 smithi045 conmon[39718]: cluster 2021-07-31T02:34: 2021-07-31T02:34:10.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:10 smithi045 conmon[39718]: 09.379777+0000 mon.a (mon.0) 2425 : cluster [DBG] osdmap e1458: 8 total, 8 up, 8 in 2021-07-31T02:34:11.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:11 smithi154 conmon[34553]: cluster 2021-07-31T02:34:10.380602+0000 mon.a (mon. 2021-07-31T02:34:11.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:11 smithi154 conmon[34553]: 0) 2426 : cluster [DBG] osdmap e1459: 8 total, 8 up, 8 in 2021-07-31T02:34:11.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:11 smithi045 conmon[33655]: cluster 2021-07-31T02:34:10.380602+0000 mon.a (mon.0) 2426 : cluster 2021-07-31T02:34:11.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:11 smithi045 conmon[33655]: [DBG] osdmap e1459: 8 total, 8 up, 8 in 2021-07-31T02:34:11.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:11 smithi045 conmon[39718]: cluster 2021-07-31T02:34:10.380602+0000 2021-07-31T02:34:11.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:11 smithi045 conmon[39718]: mon.a (mon.0) 2426 : cluster [DBG] osdmap e1459: 8 total, 8 up, 8 in 2021-07-31T02:34:12.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:12 smithi154 conmon[34553]: cluster 2021-07-31T02:34:10.471976+0000 mgr.x (mgr.24542) 2010 : cluster 2021-07-31T02:34:12.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:12 smithi154 conmon[34553]: [DBG] pgmap v2714: 73 pgs: 73 active+clean; 821 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 19 KiB/s rd, 4.0 MiB/s wr, 31 op/s 2021-07-31T02:34:12.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:12 smithi154 conmon[34553]: cluster 2021-07-31T02:34:11.391040+0000 mon.a (mon.0) 2427 : cluster [DBG] osdmap e1460: 8 total, 8 up, 8 in 2021-07-31T02:34:12.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:12 smithi045 conmon[39718]: cluster 2021-07-31T02:34:10.471976+0000 mgr.x (mgr.24542) 2010 2021-07-31T02:34:12.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:12 smithi045 conmon[39718]: : cluster [DBG] pgmap v2714: 73 pgs: 73 active+clean; 821 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 19 KiB/s rd, 4.0 MiB/s wr, 31 op/s 2021-07-31T02:34:12.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:12 smithi045 conmon[39718]: cluster 2021-07-31T02:34:11.391040+0000 mon.a (mon.0) 2427 : cluster [DBG] osdmap e1460: 8 total, 8 up, 8 in 2021-07-31T02:34:12.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:12 smithi045 conmon[33655]: cluster 2021-07-31T02:34:12.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:12 smithi045 conmon[33655]: 2021-07-31T02:34:10.471976+0000 2021-07-31T02:34:12.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:12 smithi045 conmon[33655]: mgr.x (mgr.24542) 2010 : cluster [DBG] pgmap v2714: 73 pgs: 73 active+clean; 821 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 19 KiB/s rd, 4.0 MiB/s wr, 31 op/s 2021-07-31T02:34:12.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:12 smithi045 conmon[33655]: cluster 2021- 2021-07-31T02:34:12.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:12 smithi045 conmon[33655]: 07-31T02:34:11.391040+0000 mon.a (mon.0) 2427 : cluster [DBG] 2021-07-31T02:34:12.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:12 smithi045 conmon[33655]: osdmap e1460: 8 total, 8 up, 8 in 2021-07-31T02:34:13.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:13 smithi154 conmon[34553]: cluster 2021-07-31T02:34:12.396943+0000 mon.a (mon.0) 2021-07-31T02:34:13.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:13 smithi154 conmon[34553]: 2428 : cluster [DBG] osdmap e1461: 8 total, 8 up, 8 in 2021-07-31T02:34:13.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:13 smithi154 conmon[34553]: cluster 2021-07-31T02:34:12.472546+0000 mgr.x (mgr.24542) 2011 : cluster [DBG] pgmap v2717: 73 pgs: 73 active+clean; 821 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 19 KiB/s rd, 4.0 MiB/s wr, 31 op/s 2021-07-31T02:34:13.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:13 smithi045 conmon[33655]: cluster 2021-07-31T02:34:12.396943+0000 mon.a (mon.0) 2428 : cluster 2021-07-31T02:34:13.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:13 smithi045 conmon[33655]: [DBG] osdmap e1461: 8 total, 8 up, 8 in 2021-07-31T02:34:13.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:13 smithi045 conmon[33655]: cluster 2021-07-31T02:34:12.472546+0000 mgr.x (mgr.24542) 2011 : cluster [DBG] pgmap v2717: 73 pgs: 73 active+clean; 821 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 19 KiB/s rd, 4.0 MiB/s wr, 31 op/s 2021-07-31T02:34:13.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:13 smithi045 conmon[39718]: cluster 2021-07-31T02:34:12.396943 2021-07-31T02:34:13.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:13 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2428 : cluster [DBG] osdmap e1461: 8 total, 8 up, 8 in 2021-07-31T02:34:13.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:13 smithi045 conmon[39718]: cluster 2021-07-31T02:34:13.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:13 smithi045 conmon[39718]: 2021-07-31T02:34:12.472546+0000 mgr.x (mgr.24542) 2011 : 2021-07-31T02:34:13.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:13 smithi045 conmon[39718]: cluster [DBG] pgmap v2717: 73 pgs: 73 active+clean; 821 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 19 KiB/s rd, 4.0 MiB/s wr, 31 op/s 2021-07-31T02:34:14.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:14 smithi154 conmon[34553]: cluster 2021-07-31T02:34:13.402613+0000 mon.a (mon.0) 2429 : cluster [DBG] osdmap e1462: 8 total, 8 up, 8 in 2021-07-31T02:34:14.753 INFO:journalctl@ceph.osd.5.smithi154.stdout:Jul 31 02:34:14 smithi154 conmon[42050]: debug 2021-07-31T02:34:14.433+0000 7fc9b005a700 -1 osd.5 pg_epoch: 1463 pg[75.16( v 1463'110 (0'0,1463'110] local-lis/les=1308/1309 n=16 ec=1308/1308 lis/c=1308/1308 les/c/f=1309/1309/0 sis=1308) [5,1,6] r=0 lpr=1308 crt=1463'110 lcod 1462'109 mlcod 1462'109 active+clean+snaptrim trimq=[8d~1] ps=[8a~3]] removing snap head 2021-07-31T02:34:14.754 INFO:journalctl@ceph.osd.6.smithi154.stdout:Jul 31 02:34:14 smithi154 conmon[45576]: debug 2021-07-31T02:34:14.441+0000 7f47c18a6700 -1 osd.6 pg_epoch: 1463 pg[75.19( v 1463'99 (0'0,1463'99] local-lis/les=1308/1309 n=15 ec=1308/1308 lis/c=1308/1308 les/c/f=1309/1309/0 sis=1308) [6,2,3] r=0 lpr=1308 crt=1463'99 lcod 1459'98 mlcod 1459'98 active+clean+snaptrim trimq=[8d~1] ps=[8a~3]] removing snap head 2021-07-31T02:34:14.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:14 smithi045 conmon[33655]: cluster 2021-07-31T02:34:13.402613+0000 mon.a (mon.0) 2429 2021-07-31T02:34:14.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:14 smithi045 conmon[33655]: : cluster [DBG] osdmap e1462: 8 total, 8 up, 8 in 2021-07-31T02:34:14.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:14 smithi045 conmon[39718]: cluster 2021-07-31T02:34:13 2021-07-31T02:34:14.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:14 smithi045 conmon[39718]: .402613+0000 mon.a (mon.0) 2429 : cluster [DBG] osdmap e1462: 8 total, 8 up, 8 in 2021-07-31T02:34:14.777 INFO:journalctl@ceph.osd.0.smithi045.stdout:Jul 31 02:34:14 smithi045 conmon[43243]: debug 2021-07-31T02:34:14.777 INFO:journalctl@ceph.osd.0.smithi045.stdout:Jul 31 02:34:14 smithi045 conmon[43243]: 2021-07-31T02:34:14.433+0000 7fdf70799700 -1 osd.0 pg_epoch: 1463 pg[75.0( v 1463'129 (0'0,1463'129] local-lis/les=1308/1309 n=17 ec=1308/1308 lis/c=1308/1308 les/c/f=1309/1309/0 sis=1308) [0,5,7] r=0 lpr=1308 luod=1462'128 crt=1463'129 lcod 1461'126 mlcod 1461'126 active+clean+snaptrim trimq=[8d~1] ps=[8a~3]] removing snap head 2021-07-31T02:34:14.777 INFO:journalctl@ceph.osd.0.smithi045.stdout:Jul 31 02:34:14 smithi045 conmon[43243]: 2021-07-31T02:34:14.818 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMigration.AbortWithSnapshots (14485 ms) 2021-07-31T02:34:14.819 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMigration.CloneV1Parent 2021-07-31T02:34:14.835 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 83886080 2021-07-31T02:34:14.838 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:34:14.840 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMigration.CloneV1Parent (22 ms) 2021-07-31T02:34:14.840 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMigration.CloneV2Parent 2021-07-31T02:34:14.861 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 83886080 2021-07-31T02:34:14.868 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:34:14.870 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMigration.CloneV2Parent (29 ms) 2021-07-31T02:34:14.870 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMigration.CloneV1ParentAbort 2021-07-31T02:34:14.888 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 83886080 2021-07-31T02:34:14.891 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:34:14.894 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMigration.CloneV1ParentAbort (25 ms) 2021-07-31T02:34:14.894 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMigration.CloneV2ParentAbort 2021-07-31T02:34:14.910 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 83886080 2021-07-31T02:34:14.917 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:34:14.920 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMigration.CloneV2ParentAbort (25 ms) 2021-07-31T02:34:14.920 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMigration.CloneV1ParentAbortFixIncompleteChildReattach 2021-07-31T02:34:14.936 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 83886080 2021-07-31T02:34:14.939 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:34:14.941 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMigration.CloneV1ParentAbortFixIncompleteChildReattach (22 ms) 2021-07-31T02:34:14.941 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMigration.CloneV1ParentAbortFixParentReattach 2021-07-31T02:34:14.961 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 83886080 2021-07-31T02:34:14.964 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:34:14.967 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMigration.CloneV1ParentAbortFixParentReattach (26 ms) 2021-07-31T02:34:14.967 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMigration.CloneV1ParentAbortRelinkNotNeeded 2021-07-31T02:34:14.988 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 83886080 2021-07-31T02:34:14.999 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:34:14.999 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMigration.CloneV1ParentAbortRelinkNotNeeded (26 ms) 2021-07-31T02:34:14.999 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMigration.CloneV2ParentAbortFixIncompleteChildReattach 2021-07-31T02:34:15.015 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 83886080 2021-07-31T02:34:15.019 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:34:15.021 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMigration.CloneV2ParentAbortFixIncompleteChildReattach (28 ms) 2021-07-31T02:34:15.021 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMigration.CloneV2ParentAbortFixParentReattach 2021-07-31T02:34:15.038 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 83886080 2021-07-31T02:34:15.041 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:34:15.044 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMigration.CloneV2ParentAbortFixParentReattach (23 ms) 2021-07-31T02:34:15.044 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMigration.CloneV2ParentAbortRelinkNotNeeded 2021-07-31T02:34:15.062 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 83886080 2021-07-31T02:34:15.066 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:34:15.068 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMigration.CloneV2ParentAbortRelinkNotNeeded (23 ms) 2021-07-31T02:34:15.068 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMigration.StressNoMigrate 2021-07-31T02:34:15.086 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 83886080 2021-07-31T02:34:15.089 INFO:tasks.workunit.client.0.smithi045.stdout:write: A 53484687~476897 2021-07-31T02:34:15.096 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 23077732~3604789 2021-07-31T02:34:15.099 INFO:tasks.workunit.client.0.smithi045.stdout:write: A 42980382~5880044 2021-07-31T02:34:15.132 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 32770931~2974055 2021-07-31T02:34:15.135 INFO:tasks.workunit.client.0.smithi045.stdout:write: A 4804140~7841086 2021-07-31T02:34:15.182 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 50058660~3909499 2021-07-31T02:34:15.185 INFO:tasks.workunit.client.0.smithi045.stdout:write: A 58676384~7616190 2021-07-31T02:34:15.219 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 29326468~1115082 2021-07-31T02:34:15.221 INFO:tasks.workunit.client.0.smithi045.stdout:snap_create: snap0 2021-07-31T02:34:15.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:15 smithi154 conmon[34553]: cluster 2021-07-31T02:34:15.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:15 smithi154 conmon[34553]: 2021-07-31T02:34:14.409519+0000 2021-07-31T02:34:15.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:15 smithi154 conmon[34553]: mon.a (mon.0) 2430 : cluster [DBG] osdmap e1463: 8 total, 8 up, 8 in 2021-07-31T02:34:15.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:15 smithi154 conmon[34553]: cluster 2021-07-31T02:34:15.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:15 smithi154 conmon[34553]: 2021-07-31T02:34:14.473001+0000 2021-07-31T02:34:15.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:15 smithi154 conmon[34553]: mgr.x (mgr.24542) 2012 : cluster [DBG] 2021-07-31T02:34:15.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:15 smithi154 conmon[34553]: pgmap v2720: 73 pgs: 73 active+clean; 889 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 42 MiB/s rd, 23 MiB/s wr, 170 op/s 2021-07-31T02:34:15.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:15 smithi045 conmon[33655]: cluster 2021-07-31T02:34:14.409519+0000 mon.a (mon 2021-07-31T02:34:15.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:15 smithi045 conmon[33655]: .0) 2430 : cluster [DBG] osdmap e1463: 8 total, 8 up, 8 in 2021-07-31T02:34:15.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:15 smithi045 conmon[33655]: cluster 2021-07-31T02:34:14.473001+0000 mgr.x (mgr.24542) 2012 : cluster [DBG] pgmap v2720: 73 pgs: 73 active+clean; 889 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 42 MiB/s rd, 23 MiB/s wr, 170 op/s 2021-07-31T02:34:15.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:15 smithi045 conmon[39718]: cluster 2021-07-31T02:34:15.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:15 smithi045 conmon[39718]: 2021-07- 2021-07-31T02:34:15.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:15 smithi045 conmon[39718]: 31T02:34:14.409519+0000 mon.a (mon.0) 2430 : cluster [DBG] osdmap e1463: 8 total, 8 up, 8 in 2021-07-31T02:34:15.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:15 smithi045 conmon[39718]: cluster 2021-07 2021-07-31T02:34:15.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:15 smithi045 conmon[39718]: -31T02:34:14.473001+0000 mgr.x (mgr.24542) 2012 2021-07-31T02:34:15.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:15 smithi045 conmon[39718]: : cluster [DBG] pgmap v2720: 73 pgs: 73 active+clean; 889 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 42 MiB/s rd, 23 MiB/s wr, 170 op/s 2021-07-31T02:34:16.426 INFO:tasks.workunit.client.0.smithi045.stdout:write: B 78811468~5074612 2021-07-31T02:34:16.436 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 53542261~401495 2021-07-31T02:34:16.437 INFO:tasks.workunit.client.0.smithi045.stdout:write: B 54467370~8270497 2021-07-31T02:34:16.447 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 19267618~4833885 2021-07-31T02:34:16.448 INFO:tasks.workunit.client.0.smithi045.stdout:write: B 30918525~1077268 2021-07-31T02:34:16.448 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 67302351~1333236 2021-07-31T02:34:16.449 INFO:tasks.workunit.client.0.smithi045.stdout:write: B 43947427~7646435 2021-07-31T02:34:16.469 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 10799474~670385 2021-07-31T02:34:16.469 INFO:tasks.workunit.client.0.smithi045.stdout:snap_create: snap1 2021-07-31T02:34:16.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:16 smithi154 conmon[34553]: cluster 2021-07-31T02:34:15.413723+0000 mon.a (mon.0) 2431 : cluster [DBG] osdmap e1464: 8 total, 8 up, 8 in 2021-07-31T02:34:16.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:16 smithi154 conmon[34553]: 2021-07-31T02:34:16.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:16 smithi045 conmon[39718]: cluster 2021-07-31T02:34:15.413723+0000 mon.a (mon. 2021-07-31T02:34:16.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:16 smithi045 conmon[39718]: 0) 2431 : cluster [DBG] osdmap e1464: 8 total, 8 up, 8 in 2021-07-31T02:34:16.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:16 smithi045 conmon[33655]: cluster 2021-07-31T02:34:15.413723+0000 mon.a (mon.0) 2431 2021-07-31T02:34:16.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:16 smithi045 conmon[33655]: : cluster [DBG] osdmap e1464: 8 total, 8 up, 8 in 2021-07-31T02:34:17.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:17 smithi154 conmon[34553]: cluster 2021-07-31T02:34:16.420244+0000 mon.a (mon.0) 2432 : cluster [DBG] 2021-07-31T02:34:17.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:17 smithi154 conmon[34553]: osdmap e1465: 8 total, 8 up, 8 in 2021-07-31T02:34:17.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:17 smithi154 conmon[34553]: cluster 2021-07-31T02:34:16.473443+0000 mgr.x (mgr.24542) 2013 : cluster [DBG] pgmap v2723: 73 pgs: 73 active+clean; 889 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 42 MiB/s rd, 23 MiB/s wr, 170 op/s 2021-07-31T02:34:17.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:17 smithi154 conmon[34553]: cluster 2021-07-31T02:34:17.379248+0000 mon.a (mon 2021-07-31T02:34:17.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:17 smithi154 conmon[34553]: .0) 2433 : cluster [DBG] osdmap e1466: 8 total, 8 up, 8 in 2021-07-31T02:34:17.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:17 smithi045 conmon[39718]: cluster 2021-07-31T02:34:16.420244+0000 mon.a (mon.0) 2432 : cluster [DBG] osdmap e1465: 8 total, 8 up, 8 in 2021-07-31T02:34:17.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:17 smithi045 conmon[39718]: cluster 2021-07-31T02:34 2021-07-31T02:34:17.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:17 smithi045 conmon[39718]: :16.473443+0000 mgr.x (mgr. 2021-07-31T02:34:17.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:17 smithi045 conmon[39718]: 24542) 2013 : cluster [DBG] pgmap v2723: 73 pgs: 73 active+clean; 889 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 42 MiB/s rd, 23 MiB/s wr, 170 op/s 2021-07-31T02:34:17.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:17 smithi045 conmon[39718]: cluster 2021-07 2021-07-31T02:34:17.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:17 smithi045 conmon[39718]: -31T02:34:17.379248+0000 2021-07-31T02:34:17.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:17 smithi045 conmon[39718]: mon.a (mon.0) 2433 : cluster [DBG] 2021-07-31T02:34:17.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:17 smithi045 conmon[39718]: osdmap e1466: 8 total, 8 up, 8 in 2021-07-31T02:34:17.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:17 smithi045 conmon[33655]: cluster 2021-07-31T02:34:16.420244+0000 mon.a (mon.0 2021-07-31T02:34:17.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:17 smithi045 conmon[33655]: ) 2432 : cluster [DBG] osdmap e1465: 8 total, 8 up, 8 in 2021-07-31T02:34:17.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:17 smithi045 conmon[33655]: cluster 2021-07-31T02:34:16.473443+0000 mgr.x (mgr.24542) 2013 : cluster [DBG] pgmap v2723: 73 pgs: 73 active+clean; 889 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 42 MiB/s rd, 23 MiB/s wr, 170 op/s 2021-07-31T02:34:17.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:17 smithi045 conmon[33655]: cluster 2021-07-31T02: 2021-07-31T02:34:17.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:17 smithi045 conmon[33655]: 34:17.379248+0000 mon.a (mon. 2021-07-31T02:34:17.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:17 smithi045 conmon[33655]: 0) 2433 : cluster [DBG] osdmap e1466: 8 total, 8 up, 8 in 2021-07-31T02:34:18.391 INFO:tasks.workunit.client.0.smithi045.stdout:write: C 50233950~6970901 2021-07-31T02:34:18.401 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 18333564~6646518 2021-07-31T02:34:18.402 INFO:tasks.workunit.client.0.smithi045.stdout:write: C 6098997~7507410 2021-07-31T02:34:18.410 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 28194125~6360488 2021-07-31T02:34:18.416 INFO:tasks.workunit.client.0.smithi045.stdout:write: C 39142503~5826009 2021-07-31T02:34:18.450 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 40495523~2984430 2021-07-31T02:34:18.466 INFO:tasks.workunit.client.0.smithi045.stdout:write: C 35550620~1360107 2021-07-31T02:34:18.470 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 13730285~3627096 2021-07-31T02:34:18.470 INFO:tasks.workunit.client.0.smithi045.stdout:snap_create: snap2 2021-07-31T02:34:19.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:19 smithi154 conmon[34553]: cluster 2021-07-31T02:34:18.385247+0000 mon.a (mon.0) 2434 : cluster [DBG] 2021-07-31T02:34:19.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:19 smithi154 conmon[34553]: osdmap e1467: 8 total, 8 up, 8 in 2021-07-31T02:34:19.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:19 smithi045 conmon[39718]: cluster 2021-07-31T02:34:18.385247+0000 2021-07-31T02:34:19.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:19 smithi045 conmon[39718]: mon.a (mon.0) 2434 : cluster [DBG] osdmap e1467: 8 total, 8 up, 8 in 2021-07-31T02:34:19.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:19 smithi045 conmon[33655]: cluster 2021-07-31T02:34:18.385247+0000 mon.a (mon.0) 2434 2021-07-31T02:34:19.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:19 smithi045 conmon[33655]: : cluster [DBG] osdmap e1467: 8 total, 8 up, 8 in 2021-07-31T02:34:20.400 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 166221261 2021-07-31T02:34:20.406 INFO:tasks.workunit.client.0.smithi045.stdout:write: D 131728412~5223567 2021-07-31T02:34:20.418 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 124520368~3282986 2021-07-31T02:34:20.420 INFO:tasks.workunit.client.0.smithi045.stdout:write: D 64936818~7610674 2021-07-31T02:34:20.426 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 117138738~658597 2021-07-31T02:34:20.427 INFO:tasks.workunit.client.0.smithi045.stdout:write: D 96771244~8293514 2021-07-31T02:34:20.440 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 150491592~85938 2021-07-31T02:34:20.440 INFO:tasks.workunit.client.0.smithi045.stdout:write: D 83471097~8272161 2021-07-31T02:34:20.453 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 1243261~2218075 2021-07-31T02:34:20.464 INFO:tasks.workunit.client.0.smithi045.stdout:snap_create: snap3 2021-07-31T02:34:20.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:20 smithi154 conmon[34553]: cluster 2021-07-31T02:34:18.473861+0000 mgr.x (mgr.24542) 2014 : cluster 2021-07-31T02:34:20.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:20 smithi154 conmon[34553]: [DBG] pgmap v2726: 73 pgs: 73 active+clean; 889 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2021-07-31T02:34:20.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:20 smithi154 conmon[34553]: cluster 2021-07-31T02:34:19.390606+0000 mon.a (mon.0) 2435 : cluster [DBG] osdmap e1468: 8 total, 8 up, 8 in 2021-07-31T02:34:20.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:20 smithi045 conmon[39718]: cluster 2021-07-31T02:34:18.473861+0000 2021-07-31T02:34:20.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:20 smithi045 conmon[39718]: mgr.x (mgr.24542) 2014 : cluster [DBG] pgmap v2726: 73 pgs: 73 active+clean; 889 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2021-07-31T02:34:20.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:20 smithi045 conmon[39718]: cluster 2021- 2021-07-31T02:34:20.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:20 smithi045 conmon[39718]: 07-31T02:34: 2021-07-31T02:34:20.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:20 smithi045 conmon[39718]: 19.390606+0000 mon.a (mon.0 2021-07-31T02:34:20.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:20 smithi045 conmon[39718]: ) 2435 : cluster [DBG] osdmap e1468: 8 total, 8 up, 8 in 2021-07-31T02:34:20.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:20 smithi045 conmon[33655]: cluster 2021-07-31T02:34:20.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:20 smithi045 conmon[33655]: 2021-07-31T02:34:18.473861+0000 mgr.x (mgr.24542) 2014 : cluster [DBG] pgmap v2726: 73 pgs: 73 active+clean; 889 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2021-07-31T02:34:20.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:20 smithi045 conmon[33655]: cluster 2021-07-31T02:34:20.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:20 smithi045 conmon[33655]: 2021-07-31 2021-07-31T02:34:20.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:20 smithi045 conmon[33655]: T02:34:19.390606+0000 mon.a (mon.0) 2435 : cluster [DBG] osdmap e1468: 8 total, 8 up, 8 in 2021-07-31T02:34:20.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:20 smithi045 conmon[33655]: 2021-07-31T02:34:21.469 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:34:21.746 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:34:21.746 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 49m ago 50m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:34:21.747 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 46m ago 49m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:34:21.748 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 46m ago 53m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:34:21.748 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 49m ago 55m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:34:21.748 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 49m ago 55m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:34:21.749 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 46m ago 53m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:34:21.749 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 49m ago 53m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:34:21.750 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 49m ago 50m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:34:21.750 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 46m ago 50m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:34:21.751 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 49m ago 52m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:34:21.751 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 49m ago 52m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:34:21.751 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 49m ago 52m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:34:21.751 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 49m ago 51m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:34:21.752 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 46m ago 51m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:34:21.752 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 46m ago 51m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:34:21.752 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 46m ago 51m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:34:21.752 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 46m ago 50m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:34:21.752 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 46m ago 50m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:34:21.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:21 smithi154 conmon[34553]: cluster 2021-07-31T02:34:21.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:21 smithi154 conmon[34553]: 2021-07-31T02:34:20.392245+0000 2021-07-31T02:34:21.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:21 smithi154 conmon[34553]: mon.a (mon.0) 2436 : cluster [DBG] osdmap e1469: 8 total, 8 up, 8 in 2021-07-31T02:34:21.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:21 smithi154 conmon[34553]: cluster 2021-07-31T02:34:21.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:21 smithi154 conmon[34553]: 2021-07-31T02:34:20.474312+0000 mgr.x (mgr.24542) 2015 : cluster [DBG] pgmap v2729: 73 pgs: 73 active+clean; 1010 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 36 MiB/s rd, 31 MiB/s wr, 100 op/s 2021-07-31T02:34:21.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:21 smithi154 conmon[34553]: 2021-07-31T02:34:21.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:21 smithi045 conmon[39718]: cluster 2021-07-31T02:34:21.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:21 smithi045 conmon[39718]: 2021-07-31T02:34:20 2021-07-31T02:34:21.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:21 smithi045 conmon[39718]: .392245+0000 mon.a (mon.0) 2436 : cluster [DBG] osdmap e1469: 8 total, 8 up, 8 in 2021-07-31T02:34:21.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:21 smithi045 conmon[39718]: cluster 2021-07-31T02:34:21.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:21 smithi045 conmon[39718]: 2021-07-31T02:34:20.474312+0000 mgr.x (mgr 2021-07-31T02:34:21.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:21 smithi045 conmon[39718]: .24542) 2015 : cluster [DBG] pgmap v2729: 73 pgs: 73 active+clean; 1010 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 36 MiB/s rd, 31 MiB/s wr, 100 op/s 2021-07-31T02:34:21.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:21 smithi045 conmon[33655]: cluster 2021-07-31T02:34: 2021-07-31T02:34:21.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:21 smithi045 conmon[33655]: 20.392245+0000 mon.a (mon.0) 2436 : cluster [DBG] osdmap e1469: 8 total, 8 up, 8 in 2021-07-31T02:34:21.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:21 smithi045 conmon[33655]: cluster 2021-07-31T 2021-07-31T02:34:21.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:21 smithi045 conmon[33655]: 02:34:20.474312+0000 mgr.x (mgr.24542) 2015 2021-07-31T02:34:21.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:21 smithi045 conmon[33655]: : cluster [DBG] pgmap v2729: 73 pgs: 73 active+clean; 1010 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 36 MiB/s rd, 31 MiB/s wr, 100 op/s 2021-07-31T02:34:22.081 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:34:22.081 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:34:22.081 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:34:22.082 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:34:22.082 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:34:22.082 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:34:22.082 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:34:22.083 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:34:22.083 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:34:22.083 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:34:22.083 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:34:22.083 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:34:22.084 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:34:22.084 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:34:22.084 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:34:22.084 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:34:22.085 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:34:22.388 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 225644281 2021-07-31T02:34:22.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:22 smithi154 conmon[34553]: cluster 2021-07-31T02:34:21.403000+0000 mon.a (mon.0) 2437 : cluster [DBG] 2021-07-31T02:34:22.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:22 smithi154 conmon[34553]: osdmap e1470: 8 total, 8 up, 8 in 2021-07-31T02:34:22.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:22 smithi154 conmon[34553]: audit 2021-07-31T02:34:21.456567+0000 mgr.x (mgr.24542) 2016 : audit [DBG] from='client.27400 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:34:22.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:22 smithi154 conmon[34553]: audit 2021-07-31T02:34:21.738003+0000 mgr.x (mgr.24542) 2017 : audit [DBG] from='client.17466 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:34:22.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:22 smithi154 conmon[34553]: audit 2021-07-31T02:34:22.081278+0000 mon.a (mon.0) 2438 : audit [DBG] from='client.? 172.21.15.45:0/4291943876' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:34:22.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:22 smithi154 conmon[34553]: cluster 2021-07 2021-07-31T02:34:22.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:22 smithi154 conmon[34553]: -31T02:34:22.380516+0000 mon.a (mon.0) 2439 : cluster [DBG] osdmap e1471: 8 total, 8 up, 8 in 2021-07-31T02:34:22.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:22 smithi045 conmon[33655]: cluster 2021-07-31T02:34:21.403000+0000 2021-07-31T02:34:22.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:22 smithi045 conmon[33655]: mon.a (mon.0) 2437 : cluster [DBG] osdmap e1470: 8 total, 8 up, 8 in 2021-07-31T02:34:22.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:22 smithi045 conmon[33655]: audit 2021-07-31T02:34:21.456567+0000 mgr.x (mgr.24542) 2016 2021-07-31T02:34:22.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:22 smithi045 conmon[33655]: : audit [DBG] from='client.27400 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:34:22.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:22 smithi045 conmon[33655]: audit 2021-07- 2021-07-31T02:34:22.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:22 smithi045 conmon[33655]: 31T02:34:21.738003 2021-07-31T02:34:22.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:22 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 2021-07-31T02:34:22.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:22 smithi045 conmon[33655]: 2017 : audit [DBG] from='client.17466 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:34:22.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:22 smithi045 conmon[33655]: 2021-07-31T02:34:22.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:22 smithi045 conmon[33655]: audit 2021- 2021-07-31T02:34:22.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:22 smithi045 conmon[33655]: 07-31T02:34:22. 2021-07-31T02:34:22.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:22 smithi045 conmon[33655]: 081278+0000 mon.a (mon.0 2021-07-31T02:34:22.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:22 smithi045 conmon[33655]: ) 2438 : audit [DBG] 2021-07-31T02:34:22.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:22 smithi045 conmon[33655]: from='client.? 172.21.15.45:0/4291943876' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:34:22.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:22 smithi045 conmon[33655]: cluster 2021 2021-07-31T02:34:22.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:22 smithi045 conmon[33655]: -07-31T02:34: 2021-07-31T02:34:22.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:22 smithi045 conmon[33655]: 22.380516+0000 mon.a (mon 2021-07-31T02:34:22.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:22 smithi045 conmon[33655]: .0) 2439 : cluster [DBG] 2021-07-31T02:34:22.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:22 smithi045 conmon[33655]: osdmap e1471: 8 total, 8 up, 8 in 2021-07-31T02:34:22.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:22 smithi045 conmon[39718]: cluster 2021-07-31T02:34:21.403000 2021-07-31T02:34:22.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:22 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2437 : cluster [DBG] osdmap e1470: 8 total, 8 up, 8 in 2021-07-31T02:34:22.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:22 smithi045 conmon[39718]: audit 2021-07-31T02:34:22.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:22 smithi045 conmon[39718]: 2021-07-31T02:34:21.456567+0000 mgr.x (mgr.24542) 2021-07-31T02:34:22.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:22 smithi045 conmon[39718]: 2016 : audit [DBG] from='client.27400 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:34:22.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:22 smithi045 conmon[39718]: audit 2021-07 2021-07-31T02:34:22.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:22 smithi045 conmon[39718]: -31T02:34:21.738003+0000 mgr.x ( 2021-07-31T02:34:22.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:22 smithi045 conmon[39718]: mgr.24542) 2017 : audit [DBG] from='client.17466 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:34:22.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:22 smithi045 conmon[39718]: audit 2021- 2021-07-31T02:34:22.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:22 smithi045 conmon[39718]: 07-31T02:34:22.081278+0000 2021-07-31T02:34:22.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:22 smithi045 conmon[39718]: mon.a (mon.0) 2438 : audit [DBG] 2021-07-31T02:34:22.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:22 smithi045 conmon[39718]: from='client.? 172.21.15.45:0/4291943876' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:34:22.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:22 smithi045 conmon[39718]: cluster 2021-07-31T02:34: 2021-07-31T02:34:22.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:22 smithi045 conmon[39718]: 22.380516+0000 mon.a (mon.0) 2439 : cluster 2021-07-31T02:34:22.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:22 smithi045 conmon[39718]: [DBG] osdmap e1471: 8 total, 8 up, 8 in 2021-07-31T02:34:23.135 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMigration.StressNoMigrate (8068 ms) 2021-07-31T02:34:23.135 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMigration.Stress 2021-07-31T02:34:23.153 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 83886080 2021-07-31T02:34:23.156 INFO:tasks.workunit.client.0.smithi045.stdout:write: A 43620902~4365179 2021-07-31T02:34:23.181 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 32515433~3784531 2021-07-31T02:34:23.184 INFO:tasks.workunit.client.0.smithi045.stdout:write: A 72253502~230345 2021-07-31T02:34:23.187 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 62577697~957189 2021-07-31T02:34:23.189 INFO:tasks.workunit.client.0.smithi045.stdout:write: A 3354872~2097708 2021-07-31T02:34:23.203 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 15709883~2306542 2021-07-31T02:34:23.204 INFO:tasks.workunit.client.0.smithi045.stdout:write: A 30755353~1499669 2021-07-31T02:34:23.214 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 76219207~6646636 2021-07-31T02:34:23.216 INFO:tasks.workunit.client.0.smithi045.stdout:snap_create: snap0 2021-07-31T02:34:23.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:23 smithi154 conmon[34553]: cluster 2021-07-31T02:34:22.474717+0000 mgr.x 2021-07-31T02:34:23.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:23 smithi154 conmon[34553]: (mgr.24542) 2018 : cluster [DBG] pgmap v2732: 73 pgs: 73 active+clean; 1010 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 36 MiB/s rd, 31 MiB/s wr, 100 op/s 2021-07-31T02:34:23.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:23 smithi045 conmon[33655]: cluster 2021-07-31T02:34:22.474717+0000 mgr.x (mgr 2021-07-31T02:34:23.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:23 smithi045 conmon[33655]: .24542) 2018 : cluster [DBG] pgmap v2732: 73 pgs: 73 active+clean; 1010 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 36 MiB/s rd, 31 MiB/s wr, 100 op/s 2021-07-31T02:34:23.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:23 smithi045 conmon[39718]: cluster 2021-07-31T02:34 2021-07-31T02:34:23.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:23 smithi045 conmon[39718]: :22.474717+0000 mgr.x (mgr.24542) 2018 : cluster [DBG] pgmap v2732: 73 pgs: 73 active+clean; 1010 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 36 MiB/s rd, 31 MiB/s wr, 100 op/s 2021-07-31T02:34:24.419 INFO:tasks.workunit.client.0.smithi045.stdout:write: B 42080329~7305233 2021-07-31T02:34:24.431 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 60299866~6141515 2021-07-31T02:34:24.432 INFO:tasks.workunit.client.0.smithi045.stdout:write: B 42011720~223227 2021-07-31T02:34:24.455 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 57611715~1463163 2021-07-31T02:34:24.456 INFO:tasks.workunit.client.0.smithi045.stdout:write: B 7993234~2286755 2021-07-31T02:34:24.457 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 12012241~7504646 2021-07-31T02:34:24.457 INFO:tasks.workunit.client.0.smithi045.stdout:write: B 64421362~3845087 2021-07-31T02:34:24.461 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 39077381~7988812 2021-07-31T02:34:24.462 INFO:tasks.workunit.client.0.smithi045.stdout:snap_create: snap1 2021-07-31T02:34:24.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:24 smithi154 conmon[34553]: cluster 2021-07-31T02:34:23.408769+0000 mon.a (mon.0) 2440 2021-07-31T02:34:24.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:24 smithi154 conmon[34553]: : cluster [DBG] osdmap e1472: 8 total, 8 up, 8 in 2021-07-31T02:34:24.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:24 smithi045 conmon[39718]: cluster 2021-07-31T02:34:23.408769+0000 mon.a (mon. 2021-07-31T02:34:24.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:24 smithi045 conmon[39718]: 0) 2440 : cluster [DBG] osdmap e1472: 8 total, 8 up, 8 in 2021-07-31T02:34:24.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:24 smithi045 conmon[33655]: cluster 2021-07-31T02:34:23.408769+0000 mon.a (mon.0 2021-07-31T02:34:24.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:24 smithi045 conmon[33655]: ) 2440 : cluster [DBG] osdmap e1472: 8 total, 8 up, 8 in 2021-07-31T02:34:25.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:25 smithi154 conmon[34553]: cluster 2021-07-31T02:34:24.410807+0000 mon.a (mon.0) 2441 : cluster [DBG] 2021-07-31T02:34:25.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:25 smithi154 conmon[34553]: osdmap e1473: 8 total, 8 up, 8 in 2021-07-31T02:34:25.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:25 smithi154 conmon[34553]: cluster 2021-07-31T02:34:24.475124+0000 mgr.x (mgr.24542) 2019 : cluster [DBG] pgmap v2735: 73 pgs: 73 active+clean; 1.1 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 109 MiB/s rd, 18 MiB/s wr, 69 op/s 2021-07-31T02:34:25.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:25 smithi045 conmon[33655]: cluster 2021-07-31T02:34:24.410807+0000 mon.a (mon.0) 2021-07-31T02:34:25.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:25 smithi045 conmon[33655]: 2441 : cluster [DBG] osdmap e1473: 8 total, 8 up, 8 in 2021-07-31T02:34:25.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:25 smithi045 conmon[33655]: cluster 2021-07-31T02:34:24.475124+0000 mgr.x (mgr.24542) 2019 : cluster [DBG] pgmap v2735: 73 pgs: 73 active+clean; 1.1 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 109 MiB/s rd, 18 MiB/s wr, 69 op/s 2021-07-31T02:34:25.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:25 smithi045 conmon[39718]: cluster 2021-07-31T02:34:24. 2021-07-31T02:34:25.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:25 smithi045 conmon[39718]: 410807+0000 mon.a (mon.0) 2441 : cluster [DBG] osdmap e1473: 8 total, 8 up, 8 in 2021-07-31T02:34:25.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:25 smithi045 conmon[39718]: cluster 2021-07-31T02:34 2021-07-31T02:34:25.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:25 smithi045 conmon[39718]: :24.475124+0000 mgr.x (mgr 2021-07-31T02:34:25.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:25 smithi045 conmon[39718]: .24542) 2019 : cluster [DBG] 2021-07-31T02:34:25.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:25 smithi045 conmon[39718]: pgmap v2735: 73 pgs: 73 active+clean; 1.1 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 109 MiB/s rd, 18 MiB/s wr, 69 op/s 2021-07-31T02:34:26.432 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 157944757 2021-07-31T02:34:26.440 INFO:tasks.workunit.client.0.smithi045.stdout:write: C 152191463~5753294 2021-07-31T02:34:26.447 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 134245223~7907002 2021-07-31T02:34:26.447 INFO:tasks.workunit.client.0.smithi045.stdout:write: C 40744083~8339376 2021-07-31T02:34:26.462 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 77258497~3528670 2021-07-31T02:34:26.472 INFO:tasks.workunit.client.0.smithi045.stdout:write: C 131119258~4372935 2021-07-31T02:34:26.478 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 75325287~5530071 2021-07-31T02:34:26.492 INFO:tasks.workunit.client.0.smithi045.stdout:write: C 137469791~6966200 2021-07-31T02:34:26.497 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 97482119~5231960 2021-07-31T02:34:26.498 INFO:tasks.workunit.client.0.smithi045.stdout:snap_create: snap2 2021-07-31T02:34:26.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:26 smithi154 conmon[34553]: cluster 2021-07-31T02:34:25.420852+0000 mon.a (mon.0) 2442 : cluster [DBG] osdmap e1474: 8 total, 8 up, 8 in 2021-07-31T02:34:26.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:26 smithi045 conmon[33655]: cluster 2021-07-31T02:34:25.420852+0000 mon.a (mon.0) 2021-07-31T02:34:26.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:26 smithi045 conmon[33655]: 2442 : cluster [DBG] osdmap e1474: 8 total, 8 up, 8 in 2021-07-31T02:34:26.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:26 smithi045 conmon[39718]: cluster 2021-07-31T02:34:25.420852+0000 mon.a ( 2021-07-31T02:34:26.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:26 smithi045 conmon[39718]: mon.0) 2442 : cluster [DBG] osdmap e1474: 8 total, 8 up, 8 in 2021-07-31T02:34:27.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:27 smithi154 conmon[34553]: cluster 2021-07-31T02:34:26.422637+0000 mon.a 2021-07-31T02:34:27.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:27 smithi154 conmon[34553]: (mon.0) 2443 : cluster [DBG] osdmap e1475: 8 total, 8 up, 8 in 2021-07-31T02:34:27.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:27 smithi154 conmon[34553]: cluster 2021-07-31T02:34:26.475631+0000 mgr.x (mgr. 2021-07-31T02:34:27.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:27 smithi154 conmon[34553]: 24542) 2020 : cluster [DBG] pgmap v2738: 73 pgs: 73 active+clean; 1.1 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 109 MiB/s rd, 18 MiB/s wr, 69 op/s 2021-07-31T02:34:27.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:27 smithi045 conmon[33655]: cluster 2021-07-31T02:34:26.422637+0000 mon.a (mon.0) 2443 : cluster [DBG] osdmap e1475: 8 total, 8 up, 8 in 2021-07-31T02:34:27.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:27 smithi045 conmon[33655]: cluster 2021-07-31T02:34:26.475631+0000 mgr.x (mgr.24542) 2020 : cluster [DBG] pgmap v2738: 73 pgs: 73 active+clean; 1.1 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 109 MiB/s rd, 18 MiB/s wr, 69 op/s 2021-07-31T02:34:27.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:27 smithi045 conmon[39718]: cluster 2021-07-31T02:34:26.422637+0000 mon.a 2021-07-31T02:34:27.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:27 smithi045 conmon[39718]: (mon.0) 2443 : cluster [DBG] osdmap e1475: 8 total, 8 up, 8 in 2021-07-31T02:34:27.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:27 smithi045 conmon[39718]: cluster 2021-07-31T02:34:26 2021-07-31T02:34:27.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:27 smithi045 conmon[39718]: .475631+0000 mgr.x (mgr.24542) 2020 : cluster [DBG] 2021-07-31T02:34:27.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:27 smithi045 conmon[39718]: pgmap v2738: 73 pgs: 73 active+clean; 1.1 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 109 MiB/s rd, 18 MiB/s wr, 69 op/s 2021-07-31T02:34:28.445 INFO:tasks.workunit.client.0.smithi045.stdout:write: D 59679982~5455187 2021-07-31T02:34:28.453 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 102376514~7489469 2021-07-31T02:34:28.454 INFO:tasks.workunit.client.0.smithi045.stdout:write: D 17165661~7074542 2021-07-31T02:34:28.459 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 115922018~3462685 2021-07-31T02:34:28.460 INFO:tasks.workunit.client.0.smithi045.stdout:write: D 53033950~2550575 2021-07-31T02:34:28.463 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 150679860~1909771 2021-07-31T02:34:28.465 INFO:tasks.workunit.client.0.smithi045.stdout:write: D 38341793~1872213 2021-07-31T02:34:28.473 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 52934511~5438210 2021-07-31T02:34:28.500 INFO:tasks.workunit.client.0.smithi045.stdout:snap_create: snap3 2021-07-31T02:34:28.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:28 smithi154 conmon[34553]: cluster 2021-07-31T02:34:27.429023+0000 mon.a (mon.0) 2444 : 2021-07-31T02:34:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:28 smithi154 conmon[34553]: cluster [DBG] osdmap e1476: 8 total, 8 up, 8 in 2021-07-31T02:34:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:28 smithi154 conmon[34553]: audit 2021-07-31T02:34:28.100219+0000 mon.b (mon.2) 310 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:34:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:28 smithi154 conmon[34553]: audit 2021-07-31T02:34:28.100918+0000 mon.a (mon.0) 2445 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:34:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:28 smithi154 conmon[34553]: audit 2021-07-31T02:34:28 2021-07-31T02:34:28.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:28 smithi154 conmon[34553]: .120169+0000 mon.b (mon.2) 311 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:34:28.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:28 smithi154 conmon[34553]: audit 2021 2021-07-31T02:34:28.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:28 smithi154 conmon[34553]: -07-31T02:34:28.120680+0000 mon.a (mon.0) 2446 2021-07-31T02:34:28.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:28 smithi154 conmon[34553]: : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:34:28.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:28 smithi045 conmon[39718]: cluster 2021-07-31T02:34:27.429023+0000 mon.a (mon 2021-07-31T02:34:28.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:28 smithi045 conmon[39718]: .0) 2444 : cluster [DBG] osdmap e1476: 8 total, 8 up, 8 in 2021-07-31T02:34:28.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:28 smithi045 conmon[39718]: audit 2021-07-31T02:34:28.100219+0000 mon.b (mon.2) 310 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:34:28.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:28 smithi045 conmon[39718]: audit 2021-07-31T02:34:28.100918+0000 mon.a (mon.0) 2445 : audit 2021-07-31T02:34:28.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:28 smithi045 conmon[39718]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:34:28.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:28 smithi045 conmon[39718]: audit 2021-07-31T02:34:28.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:28 smithi045 conmon[39718]: 2021-07-31T02:34:28.120169+0000 mon.b (mon.2) 311 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:34:28.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:28 smithi045 conmon[39718]: audit 2021-07-31T02:34:28.120680+0000 2021-07-31T02:34:28.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:28 smithi045 conmon[39718]: mon.a (mon.0) 2446 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:34:28.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:28 smithi045 conmon[33655]: cluster 2021-07-31T02:34:27.429023 2021-07-31T02:34:28.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:28 smithi045 conmon[33655]: +0000 mon.a (mon.0) 2444 : cluster [DBG] osdmap e1476: 8 total, 8 up, 8 in 2021-07-31T02:34:28.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:28 smithi045 conmon[33655]: audit 2021-07-31T02 2021-07-31T02:34:28.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:28 smithi045 conmon[33655]: :34:28.100219+0000 mon.b (mon.2) 310 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:34:28.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:28 smithi045 conmon[33655]: audit 2021-07-31T02:34:28.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:28 smithi045 conmon[33655]: 2021-07-31T02:34:28.100918+0000 2021-07-31T02:34:28.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:28 smithi045 conmon[33655]: mon.a (mon.0) 2445 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:34:28.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:28 smithi045 conmon[33655]: audit 2021-07-31T02 2021-07-31T02:34:28.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:28 smithi045 conmon[33655]: :34:28.120169+0000 mon.b (mon.2) 311 : audit 2021-07-31T02:34:28.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:28 smithi045 conmon[33655]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:34:28.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:28 smithi045 conmon[33655]: audit 2021-07-31T02:34:28.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:28 smithi045 conmon[33655]: 2021-07-31T02:34:28.120680 2021-07-31T02:34:28.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:28 smithi045 conmon[33655]: +0000 mon.a (mon.0) 2446 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:34:29.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:29 smithi154 conmon[34553]: cluster 2021-07-31T02:34:28.435687+0000 mon.a (mon.0) 2447 : 2021-07-31T02:34:29.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:29 smithi154 conmon[34553]: cluster [DBG] osdmap e1477: 8 total, 8 up, 8 in 2021-07-31T02:34:29.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:29 smithi154 conmon[34553]: cluster 2021-07-31T02:34:28.476303+0000 mgr.x (mgr.24542) 2021 : cluster [DBG] pgmap v2741: 73 pgs: 73 active+clean; 1.1 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail 2021-07-31T02:34:29.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:29 smithi045 conmon[33655]: cluster 2021-07-31T02:34:28.435687+0000 2021-07-31T02:34:29.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:29 smithi045 conmon[33655]: mon.a (mon.0) 2447 : cluster [DBG] osdmap e1477: 8 total, 8 up, 8 in 2021-07-31T02:34:29.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:29 smithi045 conmon[33655]: cluster 2021-07-31T02:34:28.476303 2021-07-31T02:34:29.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:29 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 2021 : cluster [DBG] pgmap v2741: 73 pgs: 73 active+clean; 1.1 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail 2021-07-31T02:34:29.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:29 smithi045 conmon[39718]: cluster 2021-07-31T02:34:28.435687 2021-07-31T02:34:29.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:29 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2447 : cluster [DBG] osdmap e1477: 8 total, 8 up, 8 in 2021-07-31T02:34:29.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:29 smithi045 conmon[39718]: cluster 2021-07-31T02:34 2021-07-31T02:34:29.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:29 smithi045 conmon[39718]: :28.476303+0000 mgr.x (mgr.24542) 2021 : cluster [DBG] pgmap v2741: 73 pgs: 73 active+clean; 1.1 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail 2021-07-31T02:34:30.459 INFO:tasks.workunit.client.0.smithi045.stdout:migration_prepare 2021-07-31T02:34:30.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:30 smithi154 conmon[34553]: cluster 2021-07-31T02:34:29.446373+0000 mon.a (mon.0) 2448 : cluster [DBG] osdmap e1478: 8 total, 8 up, 8 in 2021-07-31T02:34:30.841 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:30 smithi045 conmon[33655]: cluster 2021-07-31T02:34:29.446373 2021-07-31T02:34:30.841 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:30 smithi045 conmon[33655]: +0000 mon.a (mon.0) 2448 : cluster [DBG] osdmap e1478: 8 total, 8 up, 8 in 2021-07-31T02:34:30.842 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:30 smithi045 conmon[39718]: cluster 2021-07-31T02:34:29.446373+0000 mon.a 2021-07-31T02:34:30.842 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:30 smithi045 conmon[39718]: (mon.0) 2448 : cluster [DBG] osdmap e1478: 8 total, 8 up, 8 in 2021-07-31T02:34:31.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:31 smithi154 conmon[34553]: cluster 2021-07-31T02:34:30.447899+0000 mon.a (mon.0) 2449 : cluster [DBG] 2021-07-31T02:34:31.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:31 smithi154 conmon[34553]: osdmap e1479: 8 total, 8 up, 8 in 2021-07-31T02:34:31.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:31 smithi154 conmon[34553]: cluster 2021-07-31T02:34:30.476860+0000 mgr.x (mgr.24542) 2022 : cluster [DBG] pgmap v2744: 73 pgs: 73 active+clean; 1.2 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 9.5 KiB/s rd, 27 MiB/s wr, 28 op/s 2021-07-31T02:34:31.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:31 smithi045 conmon[33655]: cluster 2021-07-31T02:34:30.447899+0000 2021-07-31T02:34:31.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:31 smithi045 conmon[33655]: mon.a (mon.0) 2449 : cluster [DBG] osdmap e1479: 8 total, 8 up, 8 in 2021-07-31T02:34:31.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:31 smithi045 conmon[33655]: cluster 2021-07-31T02:34:30. 2021-07-31T02:34:31.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:31 smithi045 conmon[33655]: 476860+0000 mgr.x (mgr.24542) 2022 : cluster [DBG] pgmap v2744: 73 pgs: 73 active+clean; 1.2 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 9.5 KiB/s rd, 27 MiB/s wr, 28 op/s 2021-07-31T02:34:31.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:31 smithi045 conmon[39718]: cluster 2021-07-31T02:34:30.447899+0000 mon.a (mon.0) 2449 : 2021-07-31T02:34:31.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:31 smithi045 conmon[39718]: cluster [DBG] osdmap e1479: 8 total, 8 up, 8 in 2021-07-31T02:34:31.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:31 smithi045 conmon[39718]: cluster 2021-07-31T02:34:30.476860+0000 mgr.x (mgr.24542) 2022 : cluster [DBG] pgmap v2744: 73 pgs: 73 active+clean; 1.2 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 9.5 KiB/s rd, 27 MiB/s wr, 28 op/s 2021-07-31T02:34:32.717 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[34553]: cluster 2021-07-31T02:34:31.462723+0000 mon.a (mon.0) 2450 : cluster [DBG] osdmap e1480: 8 total, 8 up, 8 in 2021-07-31T02:34:32.718 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: debug 2021-07-31T02:34:32.689+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:34:32.718 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:34:32.718 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:34:32.718 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: yield 2021-07-31T02:34:32.719 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:34:32.719 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:34:32.719 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:34:32.719 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:34:32.720 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:34:32.720 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:34:32.720 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:34:32.720 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:34:32.720 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:34:32.721 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: task.result() 2021-07-31T02:34:32.721 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:34:32.721 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: debug 2021-07-31T02:34:32.701+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:34:32.721 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:34:32.721 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:34:32.722 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: yield 2021-07-31T02:34:32.722 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:34:32.722 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:34:32.722 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:34:32.723 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:34:32.723 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:34:32.723 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:34:32.723 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:34:32.723 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:34:32.724 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:34:32.724 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: task.result() 2021-07-31T02:34:32.724 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:34:32.727 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:32 smithi045 conmon[39718]: cluster 2021-07-31T02:34:31.462723+0000 mon.a (mon.0) 2450 : cluster [DBG] osdmap e1480: 8 total, 8 up, 8 in 2021-07-31T02:34:32.728 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:32 smithi045 conmon[33655]: cluster 2021-07-31T02:34:31.462723+0000 mon.a (mon.0) 2450 : cluster 2021-07-31T02:34:32.728 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:32 smithi045 conmon[33655]: [DBG] osdmap e1480: 8 total, 8 up, 8 in 2021-07-31T02:34:33.003 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: debug 2021-07-31T02:34:32.727+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:34:33.003 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:34:33.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:34:33.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: yield 2021-07-31T02:34:33.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:34:33.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:34:33.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:34:33.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:34:33.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:34:33.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:34:33.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:34:33.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:34:33.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:34:33.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: task.result() 2021-07-31T02:34:33.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:34:33.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: debug 2021-07-31T02:34:32.728+0000 7fcb9215e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:34:33.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:34:33.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:34:33.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: yield 2021-07-31T02:34:33.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:34:33.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:34:33.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:34:33.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:34:33.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:34:33.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:34:33.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:34:33.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:34:33.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:34:33.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: task.result() 2021-07-31T02:34:33.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:34:33.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: 2021-07-31T02:34:33.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:34:33.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: 2021-07-31T02:34:33.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:34:33.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:34:33.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:34:33.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:34:33.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:34:33.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:34:33.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:34:33.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:34:33.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:34:33.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:34:33.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:34:33.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:34:33.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:34:33.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:34:33.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:34:33.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:34:33.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:34:33.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:34:33.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: raise 2021-07-31T02:34:33.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:34:33.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:34:33.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:34:33.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:34:33.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:34:33.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: debug 2021-07-31T02:34:32.737+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:34:33.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:34:33.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:34:33.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: yield 2021-07-31T02:34:33.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:34:33.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:34:33.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:34:33.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:34:33.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:34:33.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:34:33.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:34:33.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:34:33.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:34:33.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: task.result() 2021-07-31T02:34:33.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:34:33.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: debug 2021-07-31T02:34:32.737+0000 7fcb9195d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:34:33.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:34:33.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:34:33.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: yield 2021-07-31T02:34:33.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:34:33.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:34:33.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:34:33.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:34:33.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:34:33.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:34:33.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:34:33.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:34:33.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:34:33.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: task.result() 2021-07-31T02:34:33.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:34:33.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: 2021-07-31T02:34:33.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:34:33.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: 2021-07-31T02:34:33.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:34:33.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:34:33.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:34:33.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:34:33.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:34:33.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:34:33.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:34:33.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:34:33.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:34:33.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:34:33.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:34:33.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:34:33.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:34:33.035 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:34:33.035 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:34:33.036 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:34:33.036 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:34:33.036 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:34:33.037 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: raise 2021-07-31T02:34:33.037 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:34:33.038 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:34:33.038 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:34:33.038 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:34:33.039 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:34:32 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:34:33.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: cluster 2021-07-31T02:34:32.468046+0000 mon.a (mon.0 2021-07-31T02:34:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: ) 2451 : cluster [DBG] osdmap e1481: 8 total, 8 up, 8 in 2021-07-31T02:34:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: cluster 2021-07-31T02:34:32.477399+0000 mgr.x (mgr.24542) 2023 : cluster 2021-07-31T02:34:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: [DBG] pgmap v2747: 73 pgs: 73 active+clean; 1.2 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 9.5 KiB/s rd, 27 MiB/s wr, 28 op/s 2021-07-31T02:34:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: audit 2021-07-31T02:34:32. 2021-07-31T02:34:33.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: 644611+0000 mon.b (mon.2) 312 : audit [DBG] 2021-07-31T02:34:33.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:34:33.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: audit 2021-07-31T02:34:33.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: 2021-07-31T02 2021-07-31T02:34:33.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: :34:32.646119+0000 mon.b (mon. 2021-07-31T02:34:33.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: 2) 313 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:34:33.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: audit 2021-07 2021-07-31T02:34:33.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: -31T02:34:32.647033+0000 mon.b (mon.2) 314 : audit 2021-07-31T02:34:33.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:34:33.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: cephadm 2021-07 2021-07-31T02:34:33.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: -31T02:34:32.647823+0000 mgr.x ( 2021-07-31T02:34:33.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: mgr.24542) 2024 : cephadm [WRN] 2021-07-31T02:34:33.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:34:33.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: cephadm 2021 2021-07-31T02:34:33.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: -07-31T02:34: 2021-07-31T02:34:33.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: 32.690690+0000 mgr.x (mgr.24542 2021-07-31T02:34:33.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: ) 2025 : cephadm [ERR] 2021-07-31T02:34:33.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:34:33.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:34:33.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: yield 2021-07-31T02:34:33.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:34:33.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:34:33.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:34:33.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:34:33.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:34:33.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:34:33.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:34:33.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:34:33.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:34:33.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:34:33.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:34:33.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:34:32.691117+0000 mgr.x (mgr.24542) 2026 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:34:33.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: cephadm 2021-07 2021-07-31T02:34:33.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: -31T02:34:32.702199 2021-07-31T02:34:33.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: +0000 mgr.x (mgr.24542) 2027 : cephadm [ERR] 2021-07-31T02:34:33.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:34:33.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:34:33.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: yield 2021-07-31T02:34:33.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:34:33.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:34:33.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:34:33.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:34:33.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:34:33.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:34:33.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:34:33.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:34:33.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:34:33.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:34:33.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:34:33.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:34:32.703554+0000 mgr.x (mgr.24542) 2028 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:34:33.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:34: 2021-07-31T02:34:33.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: 32.728561+0000 mgr.x (mgr.24542) 2029 : cephadm [ERR] 2021-07-31T02:34:33.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:34:33.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:34:33.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: yield 2021-07-31T02:34:33.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:34:33.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:34:33.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:34:33.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:34:33.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:34:33.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:34:33.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:34:33.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:34:33.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:34:33.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:34:33.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:34:33.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: 2021-07-31T02:34:33.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:34:32.728980+0000 mgr.x (mgr.24542) 2030 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:34:33.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:34:33.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:34:33.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: yield 2021-07-31T02:34:33.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:34:33.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:34:33.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:34:33.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:34:33.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:34:33.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:34:33.782 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:34:33.782 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:34:33.782 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:34:33.783 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:34:33.783 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:34:33.784 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: 2021-07-31T02:34:33.784 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:34:33.784 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: 2021-07-31T02:34:33.785 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:34:33.785 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:34:33.786 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:34:33.786 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:34:33.787 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:34:33.787 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:34:33.787 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:34:33.788 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:34:33.788 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:34:33.788 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:34:33.789 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:34:33.789 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:34:33.790 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:34:33.790 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:34:33.790 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:34:33.791 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:34:33.791 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:34:33.792 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:34:33.792 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: raise 2021-07-31T02:34:33.793 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:34:33.793 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:34:33.793 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:34:33.794 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:34:33.794 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:34:33.795 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:34:32.738472+0000 mgr.x (mgr.24542) 2031 : cephadm [ERR] 2021-07-31T02:34:33.795 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:34:33.796 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:34:33.796 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: yield 2021-07-31T02:34:33.796 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:34:33.796 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:34:33.797 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:34:33.797 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:34:33.797 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:34:33.797 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:34:33.797 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:34:33.798 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:34:33.798 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:34:33.798 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:34:33.798 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:34:33.799 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:34:32.738839+0000 mgr.x (mgr.24542) 2032 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:34:33.799 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:34:33.799 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:34:33.800 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: yield 2021-07-31T02:34:33.800 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:34:33.800 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:34:33.801 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:34:33.801 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:34:33.802 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:34:33.802 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:34:33.802 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:34:33.803 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:34:33.803 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:34:33.803 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:34:33.804 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:34:33.804 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: 2021-07-31T02:34:33.804 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:34:33.805 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: 2021-07-31T02:34:33.805 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:34:33.805 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:34:33.806 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:34:33.806 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:34:33.806 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:34:33.807 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:34:33.807 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:34:33.808 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:34:33.808 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:34:33.808 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:34:33.809 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:34:33.809 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:34:33.809 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:34:33.809 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:34:33.809 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:34:33.810 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:34:33.810 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:34:33.810 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:34:33.810 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: raise 2021-07-31T02:34:33.811 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:34:33.811 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:34:33.811 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:34:33.811 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:34:33.811 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:33 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:34:33.813 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: cluster 2021-07-31T02:34:32. 2021-07-31T02:34:33.813 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: 468046+0000 mon.a (mon.0) 2451 : cluster [DBG] osdmap e1481: 8 total, 8 up, 8 in 2021-07-31T02:34:33.813 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: cluster 2021-07 2021-07-31T02:34:33.814 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: -31T02:34:32.477399 2021-07-31T02:34:33.814 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 2023 2021-07-31T02:34:33.814 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: : cluster [DBG] pgmap v2747: 73 pgs: 73 active+clean; 1.2 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 9.5 KiB/s rd, 27 MiB/s wr, 28 op/s 2021-07-31T02:34:33.814 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: audit 2021 2021-07-31T02:34:33.814 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: -07-31T02:34:32.644611+0000 2021-07-31T02:34:33.815 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: mon.b (mon.2) 312 : audit [DBG] 2021-07-31T02:34:33.815 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:34:33.815 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: audit 2021-07-31T02:34:32 2021-07-31T02:34:33.815 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: .646119+0000 mon.b (mon.2) 313 : 2021-07-31T02:34:33.816 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:34:33.816 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: audit 2021- 2021-07-31T02:34:33.816 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: 07-31T02:34:32.647033 2021-07-31T02:34:33.816 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: +0000 mon.b (mon.2) 314 : audit 2021-07-31T02:34:33.816 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:34:33.817 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: cephadm 2021 2021-07-31T02:34:33.817 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: -07-31T02:34 2021-07-31T02:34:33.817 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: :32.647823+0000 mgr.x (mgr 2021-07-31T02:34:33.817 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: .24542) 2024 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:34:33.817 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: cephadm 2021- 2021-07-31T02:34:33.818 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: 07-31T02:34:32.690690 2021-07-31T02:34:33.818 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542 2021-07-31T02:34:33.818 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: ) 2025 : cephadm [ERR] 2021-07-31T02:34:33.818 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:34:33.819 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:34:33.819 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: yield 2021-07-31T02:34:33.819 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:34:33.819 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:34:33.819 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:34:33.820 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:34:33.820 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:34:33.820 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:34:33.821 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:34:33.821 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:34:33.821 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:34:33.821 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:34:33.821 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:34:33.822 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:34:32.691117+0000 mgr.x (mgr.24542) 2026 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:34:33.822 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:34:32 2021-07-31T02:34:33.822 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: .702199+0000 mgr.x (mgr.24542) 2027 : cephadm [ERR] 2021-07-31T02:34:33.822 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:34:33.823 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:34:33.823 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: yield 2021-07-31T02:34:33.823 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:34:33.823 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:34:33.823 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:34:33.824 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:34:33.824 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:34:33.824 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:34:33.824 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:34:33.824 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:34:33.825 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:34:33.825 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:34:33.825 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:34:33.825 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:34:32.703554+0000 mgr.x (mgr.24542) 2028 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:34:33.826 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:34:32.728561+0000 2021-07-31T02:34:33.826 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: mgr.x (mgr.24542) 2029 : cephadm [ERR] 2021-07-31T02:34:33.826 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:34:33.826 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:34:33.826 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: yield 2021-07-31T02:34:33.827 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:34:33.827 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:34:33.827 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:34:33.827 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:34:33.827 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:34:33.828 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:34:33.828 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:34:33.828 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:34:33.828 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:34:33.829 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:34:33.829 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:34:33.829 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:34:32.728980+0000 mgr.x (mgr.24542) 2030 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:34:33.829 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:34:33.829 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:34:33.830 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: yield 2021-07-31T02:34:33.830 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:34:33.830 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:34:33.831 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:34:33.831 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:34:33.831 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:34:33.832 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: cluster 2021-07-31T02:34:32.468046+0000 mon.a (mon.0) 2451 : 2021-07-31T02:34:33.832 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: cluster [DBG] osdmap e1481: 8 total, 8 up, 8 in 2021-07-31T02:34:33.832 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: cluster 2021-07-31T02:34:32.477399+0000 mgr.x (mgr.24542) 2023 : cluster [DBG] pgmap v2747: 73 pgs: 73 active+clean; 1.2 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 9.5 KiB/s rd, 27 MiB/s wr, 28 op/s 2021-07-31T02:34:33.832 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: audit 2021-07-31T02:34:33.832 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: 2021-07-31T02:34:32.644611+0000 mon.b (mon. 2021-07-31T02:34:33.833 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: 2) 312 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:34:33.833 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: audit 2021 2021-07-31T02:34:33.833 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: -07-31T02:34:32.646119+0000 mon.b (mon 2021-07-31T02:34:33.833 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: .2) 313 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:34:33.834 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: audit 2021- 2021-07-31T02:34:33.834 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: 07-31T02:34:32.647033+0000 mon.b (mon.2 2021-07-31T02:34:33.834 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: ) 314 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:34:33.834 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:34:32.647823 2021-07-31T02:34:33.834 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 2024 : 2021-07-31T02:34:33.835 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:34:33.836 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: cephadm 2021- 2021-07-31T02:34:33.836 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: 07-31T02:34:32.690690+0000 2021-07-31T02:34:33.836 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: mgr.x (mgr.24542) 2025 : 2021-07-31T02:34:33.837 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: cephadm [ERR] 2021-07-31T02:34:33.837 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:34:33.837 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:34:33.837 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: yield 2021-07-31T02:34:33.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:34:33.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:34:33.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:34:33.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:34:33.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:34:33.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:34:33.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:34:33.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:34:33.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:34:33.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:34:33.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:34:33.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:34:32.691117+0000 mgr.x (mgr.24542) 2026 2021-07-31T02:34:33.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:34:33.841 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: cephadm 2021- 2021-07-31T02:34:33.841 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: 07-31T02:34:32.702199+0000 mgr.x (mgr. 2021-07-31T02:34:33.841 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: 24542) 2027 : cephadm [ERR] 2021-07-31T02:34:33.841 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:34:33.842 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:34:33.842 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: yield 2021-07-31T02:34:33.842 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:34:33.843 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:34:33.843 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:34:33.843 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:34:33.843 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:34:33.844 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:34:33.844 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: 2021-07-31T02:34:33.844 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:34:33.844 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: 2021-07-31T02:34:33.844 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:34:33.845 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:34:33.845 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:34:33.845 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:34:33.845 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:34:33.846 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:34:33.846 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:34:33.846 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:34:33.846 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:34:33.847 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:34:33.847 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:34:33.847 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:34:33.847 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:34:33.848 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:34:33.848 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:34:33.848 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:34:33.849 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:34:33.849 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:34:33.849 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: raise 2021-07-31T02:34:33.849 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:34:33.849 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:34:33.850 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:34:33.850 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:34:33.850 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:34:33.850 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:34:32.738472 2021-07-31T02:34:33.850 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 2031 : cephadm [ERR] 2021-07-31T02:34:33.851 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:34:33.851 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:34:33.851 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: yield 2021-07-31T02:34:33.851 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:34:33.852 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:34:33.852 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:34:33.852 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:34:33.853 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:34:33.853 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:34:33.853 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:34:33.853 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:34:33.854 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:34:33.854 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:34:33.854 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:34:33.854 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:34:33.854 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:34:33.855 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:34:33.855 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:34:32.703554+0000 mgr.x (mgr.24542) 2028 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:34:33.855 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: cephadm 2021 2021-07-31T02:34:33.855 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: -07-31T02:34:32.728561+0000 mgr.x ( 2021-07-31T02:34:33.856 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: mgr.24542) 2029 : cephadm [ERR] 2021-07-31T02:34:33.856 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:34:33.856 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:34:33.856 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: yield 2021-07-31T02:34:33.856 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:34:33.857 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:34:33.857 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:34:33.857 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:34:33.857 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:34:33.857 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:34:33.858 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:34:33.858 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:34:33.858 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:34:33.858 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:34:33.859 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:34:33.859 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: 2021-07-31T02:34:33.859 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:34:32.728980+0000 mgr.x (mgr. 2021-07-31T02:34:33.859 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: 24542) 2030 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:34:33.859 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:34:33.860 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:34:33.860 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: yield 2021-07-31T02:34:33.860 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:34:33.860 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:34:33.860 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:34:33.861 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:34:33.861 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:34:33.861 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:34:33.861 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:34:33.862 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:34:33.862 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:34:33.862 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:34:33.862 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:34:33.862 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: 2021-07-31T02:34:33.863 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:34:33.863 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: 2021-07-31T02:34:33.863 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:34:33.863 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:34:33.863 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:34:33.864 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:34:33.864 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:34:33.864 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:34:33.864 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:34:33.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:34:33.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:34:33.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:34:33.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:34:33.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:34:33.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:34:33.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:34:33.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:34:33.867 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:34:33.867 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:34:33.867 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:34:33.867 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: raise 2021-07-31T02:34:33.867 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:34:33.868 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:34:33.868 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:34:33.868 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:34:33.868 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:34:33.868 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:34:32.738472+0000 mgr.x (mgr.24542) 2031 : cephadm [ERR] 2021-07-31T02:34:33.869 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:34:33.869 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:34:33.869 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: yield 2021-07-31T02:34:33.869 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:34:33.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:34:33.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:34:33.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:34:33.871 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:34:33.871 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:34:33.871 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:34:33.872 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:34:33.872 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:34:33.872 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:34:33.872 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:34:33.872 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:34:33.873 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:34:32.738839+0000 mgr.x (mgr.24542) 2032 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:34:33.873 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:34:33.873 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:34:33.873 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: yield 2021-07-31T02:34:33.874 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:34:33.874 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:34:33.874 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:34:33.874 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:34:33.874 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:34:33.875 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:34:33.875 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:34:33.875 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:34:33.875 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:34:33.876 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:34:33.876 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:34:33.876 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: 2021-07-31T02:34:33.876 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:34:33.876 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: 2021-07-31T02:34:33.877 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:34:33.877 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:34:33.877 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:34:33.877 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:34:33.877 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:34:33.878 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:34:33.878 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:34:33.878 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:34:33.878 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:34:33.879 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:34:33.879 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:34:33.879 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:34:33.879 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:34:33.879 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:34:33.880 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:34:33.880 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:34:33.880 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:34:33.880 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:34:33.880 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: raise 2021-07-31T02:34:33.881 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:34:33.881 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:34:33.881 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:34:33.881 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:34:33.882 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:34:33.882 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:34:33.882 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:34:33.883 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:34:33.883 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:34:33.883 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:34:33.884 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:34:33.884 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:34:33.884 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:34:32.738839+0000 mgr.x (mgr.24542) 2032 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:34:33.884 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:34:33.885 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:34:33.885 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: yield 2021-07-31T02:34:33.885 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:34:33.885 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:34:33.885 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:34:33.886 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:34:33.886 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:34:33.886 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:34:33.886 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:34:33.887 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:34:33.887 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:34:33.887 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:34:33.887 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:34:33.888 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: 2021-07-31T02:34:33.888 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:34:33.888 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: 2021-07-31T02:34:33.888 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:34:33.888 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:34:33.889 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:34:33.889 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:34:33.889 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:34:33.889 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:34:33.889 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:34:33.890 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:34:33.890 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:34:33.890 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:34:33.890 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:34:33.891 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:34:33.891 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:34:33.891 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:34:33.892 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:34:33.892 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:34:33.892 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:34:33.892 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:34:33.893 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: raise 2021-07-31T02:34:33.893 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:34:33.893 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:34:33.893 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:34:33.893 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:34:33.894 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:33 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:34:34.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:34 smithi154 conmon[34553]: cluster 2021-07-31T02:34:33.475262+0000 mon.a (mon.0) 2452 : cluster 2021-07-31T02:34:34.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:34 smithi154 conmon[34553]: [DBG] osdmap e1482: 8 total, 8 up, 8 in 2021-07-31T02:34:34.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:34 smithi045 conmon[33655]: cluster 2021-07-31T02:34:33.475262+0000 mon.a (mon.0) 2452 : cluster [DBG] osdmap e1482: 8 total, 8 up, 8 in 2021-07-31T02:34:34.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:34 smithi045 conmon[39718]: cluster 2021-07-31T02:34:33.475262+0000 mon.a (mon 2021-07-31T02:34:34.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:34 smithi045 conmon[39718]: .0) 2452 : cluster [DBG] osdmap e1482: 8 total, 8 up, 8 in 2021-07-31T02:34:35.230 INFO:tasks.workunit.client.0.smithi045.stdout:migration_execute 2021-07-31T02:34:35.403 INFO:tasks.workunit.client.0.smithi045.stdout:migration_commit 2021-07-31T02:34:35.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:35 smithi154 conmon[34553]: cluster 2021-07-31T02:34:34.475128+0000 mon.a (mon 2021-07-31T02:34:35.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:35 smithi154 conmon[34553]: .0) 2453 : cluster [DBG] osdmap e1483: 8 total, 8 up, 8 in 2021-07-31T02:34:35.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:35 smithi154 conmon[34553]: cluster 2021-07-31T02:34:34.477791+0000 mgr.x (mgr.24542) 2033 : cluster [DBG] pgmap v2750: 73 pgs: 73 active+clean; 1.2 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 16 KiB/s rd, 4.2 KiB/s wr, 22 op/s 2021-07-31T02:34:35.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:35 smithi045 conmon[33655]: cluster 2021-07-31T02:34:34.475128+0000 mon.a ( 2021-07-31T02:34:35.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:35 smithi045 conmon[33655]: mon.0) 2453 : cluster [DBG] osdmap e1483: 8 total, 8 up, 8 in 2021-07-31T02:34:35.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:35 smithi045 conmon[33655]: cluster 2021-07-31T02:34:34.477791+0000 mgr.x (mgr.24542) 2033 : cluster [DBG] pgmap v2750: 73 pgs: 73 active+clean; 1.2 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 16 KiB/s rd, 4.2 KiB/s wr, 22 op/s 2021-07-31T02:34:35.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:35 smithi045 conmon[39718]: cluster 2021-07-31T02:34:34.475128+0000 2021-07-31T02:34:35.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:35 smithi045 conmon[39718]: mon.a (mon.0) 2453 : cluster [DBG] osdmap e1483: 8 total, 8 up, 8 in 2021-07-31T02:34:35.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:35 smithi045 conmon[39718]: cluster 2021-07-31T02 2021-07-31T02:34:35.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:35 smithi045 conmon[39718]: :34:34.477791+0000 mgr.x (mgr.24542 2021-07-31T02:34:35.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:35 smithi045 conmon[39718]: ) 2033 : cluster [DBG] pgmap v2750: 73 pgs: 73 active+clean; 1.2 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 16 KiB/s rd, 4.2 KiB/s wr, 22 op/s 2021-07-31T02:34:35.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:35 smithi045 conmon[39718]: 2021-07-31T02:34:36.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:36 smithi154 conmon[34553]: cluster 2021-07-31T02:34:35.485039+0000 mon.a (mon.0) 2454 : cluster [DBG] 2021-07-31T02:34:36.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:36 smithi154 conmon[34553]: osdmap e1484: 8 total, 8 up, 8 in 2021-07-31T02:34:36.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:36 smithi045 conmon[39718]: cluster 2021-07-31T02:34:35.485039+0000 mon.a 2021-07-31T02:34:36.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:36 smithi045 conmon[39718]: (mon.0) 2454 : cluster [DBG] osdmap e1484: 8 total, 8 up, 8 in 2021-07-31T02:34:36.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:36 smithi045 conmon[33655]: cluster 2021-07-31T02:34:35.485039+0000 mon.a (mon.0) 2454 2021-07-31T02:34:36.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:36 smithi045 conmon[33655]: : cluster [DBG] osdmap e1484: 8 total, 8 up, 8 in 2021-07-31T02:34:37.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:37 smithi154 conmon[34553]: cluster 2021-07-31T02:34:36.478290+0000 mgr.x (mgr.24542) 2034 : cluster [DBG] pgmap v2752: 73 pgs: 73 active+clean; 1.2 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 16 KiB/s rd, 4.2 KiB/s wr, 22 op/s 2021-07-31T02:34:37.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:37 smithi154 conmon[34553]: 2021-07-31T02:34:37.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:37 smithi154 conmon[34553]: cluster 2021-07-31T02:34:36.491019+0000 mon.a (mon.0) 2455 : cluster [DBG] osdmap e1485: 8 total, 8 up, 8 in 2021-07-31T02:34:37.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:37 smithi045 conmon[33655]: cluster 2021-07-31T02:34:36.478290+0000 mgr.x ( 2021-07-31T02:34:37.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:37 smithi045 conmon[33655]: mgr.24542) 2034 : cluster [DBG] pgmap v2752: 73 pgs: 73 active+clean; 1.2 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 16 KiB/s rd, 4.2 KiB/s wr, 22 op/s 2021-07-31T02:34:37.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:37 smithi045 conmon[33655]: cluster 2021-07-31T02:34:36.491019+0000 mon.a (mon.0) 2455 : cluster [DBG] osdmap e1485: 8 total, 8 up, 8 in 2021-07-31T02:34:37.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:37 smithi045 conmon[39718]: cluster 2021-07-31T02:34:36.478290+0000 mgr.x (mgr.24542) 2034 2021-07-31T02:34:37.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:37 smithi045 conmon[39718]: : cluster [DBG] pgmap v2752: 73 pgs: 73 active+clean; 1.2 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 16 KiB/s rd, 4.2 KiB/s wr, 22 op/s 2021-07-31T02:34:37.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:37 smithi045 conmon[39718]: cluster 2021-07-31T02:34:36.491019+0000 mon.a (mon.0 2021-07-31T02:34:37.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:37 smithi045 conmon[39718]: ) 2455 : cluster [DBG] osdmap e1485: 8 total, 8 up, 8 in 2021-07-31T02:34:38.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:38 smithi154 conmon[34553]: cluster 2021-07-31T02:34:37.492060+0000 mon.a (mon.0) 2456 : cluster 2021-07-31T02:34:38.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:38 smithi154 conmon[34553]: [DBG] osdmap e1486: 8 total, 8 up, 8 in 2021-07-31T02:34:38.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:38 smithi045 conmon[39718]: cluster 2021-07-31T02:34:37 2021-07-31T02:34:38.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:38 smithi045 conmon[39718]: .492060+0000 mon.a (mon.0) 2456 : cluster [DBG] osdmap e1486: 8 total, 8 up, 8 in 2021-07-31T02:34:38.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:38 smithi045 conmon[33655]: cluster 2021-07-31T02:34:37.492060+0000 mon.a (mon.0 2021-07-31T02:34:38.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:38 smithi045 conmon[33655]: ) 2456 : cluster [DBG] osdmap e1486: 8 total, 8 up, 8 in 2021-07-31T02:34:38.777 INFO:journalctl@ceph.osd.1.smithi045.stdout:Jul 31 02:34:38 smithi045 conmon[46771]: debug 2021-07-31T02:34:38.777 INFO:journalctl@ceph.osd.1.smithi045.stdout:Jul 31 02:34:38 smithi045 conmon[46771]: 2021-07-31T02:34:38.534+0000 7f41961a7700 -1 osd.1 pg_epoch: 1487 pg[75.b( v 1487'149 (0'0,1487'149] local-lis/les=1308/1309 n=31 ec=1308/1308 lis/c=1308/1308 les/c/f=1309/1309/0 sis=1308) [1,4,2] r=0 lpr=1308 luod=1485'147 crt=1487'149 lcod 1477'145 mlcod 1477'145 active+clean+snaptrim trimq=[9b~1] ps=[9d~1,9f~1,a1~1]] removing snap head 2021-07-31T02:34:38.777 INFO:journalctl@ceph.osd.1.smithi045.stdout:Jul 31 02:34:38 smithi045 conmon[46771]: 2021-07-31T02:34:39.146 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMigration.Stress (16011 ms) 2021-07-31T02:34:39.147 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMigration.Stress2 2021-07-31T02:34:39.164 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 83886080 2021-07-31T02:34:39.168 INFO:tasks.workunit.client.0.smithi045.stdout:write: A 18577617~2862636 2021-07-31T02:34:39.197 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 32401396~6948084 2021-07-31T02:34:39.200 INFO:tasks.workunit.client.0.smithi045.stdout:write: A 20866763~2529992 2021-07-31T02:34:39.221 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 34662016~3941635 2021-07-31T02:34:39.223 INFO:tasks.workunit.client.0.smithi045.stdout:write: A 51116635~147635 2021-07-31T02:34:39.227 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 39728374~5826691 2021-07-31T02:34:39.230 INFO:tasks.workunit.client.0.smithi045.stdout:write: A 19003086~6240174 2021-07-31T02:34:39.271 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 35752645~5274803 2021-07-31T02:34:39.273 INFO:tasks.workunit.client.0.smithi045.stdout:snap_create: snap0 2021-07-31T02:34:39.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:39 smithi045 conmon[33655]: cluster 2021-07-31T02:34:38.478715+0000 mgr.x (mgr.24542) 2035 : cluster [DBG] 2021-07-31T02:34:39.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:39 smithi045 conmon[33655]: pgmap v2755: 73 pgs: 73 active+clean; 1.2 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail 2021-07-31T02:34:39.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:39 smithi045 conmon[33655]: cluster 2021-07-31T02:34:38.504150+0000 mon.a (mon.0) 2457 : cluster [DBG] osdmap e1487: 8 total, 8 up, 8 in 2021-07-31T02:34:39.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:39 smithi045 conmon[39718]: cluster 2021-07-31T02:34:38.478715+0000 mgr.x 2021-07-31T02:34:39.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:39 smithi045 conmon[39718]: (mgr.24542) 2035 : cluster [DBG] pgmap v2755: 73 pgs: 73 active+clean; 1.2 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail 2021-07-31T02:34:39.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:39 smithi045 conmon[39718]: cluster 2021-07-31T02:34:38.504150+0000 2021-07-31T02:34:39.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:39 smithi045 conmon[39718]: mon.a (mon.0) 2457 : cluster [DBG] osdmap e1487: 8 total, 8 up, 8 in 2021-07-31T02:34:40.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:39 smithi154 conmon[34553]: cluster 2021-07-31T02:34:38.478715+0000 mgr.x (mgr.24542) 2035 : 2021-07-31T02:34:40.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:39 smithi154 conmon[34553]: cluster [DBG] pgmap v2755: 73 pgs: 73 active+clean; 1.2 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail 2021-07-31T02:34:40.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:39 smithi154 conmon[34553]: cluster 2021-07-31T02:34:38.504150+0000 mon.a (mon.0) 2457 : cluster [DBG] osdmap e1487: 8 total, 8 up, 8 in 2021-07-31T02:34:40.520 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 96254979 2021-07-31T02:34:40.527 INFO:tasks.workunit.client.0.smithi045.stdout:write: B 87407421~5660898 2021-07-31T02:34:40.538 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 52231355~6530866 2021-07-31T02:34:40.538 INFO:tasks.workunit.client.0.smithi045.stdout:write: B 27273258~5131144 2021-07-31T02:34:40.539 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 55805711~8308294 2021-07-31T02:34:40.579 INFO:tasks.workunit.client.0.smithi045.stdout:write: B 18134279~3265385 2021-07-31T02:34:40.580 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 27002908~5591491 2021-07-31T02:34:40.581 INFO:tasks.workunit.client.0.smithi045.stdout:write: B 54805484~7709608 2021-07-31T02:34:40.587 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 83050634~3337516 2021-07-31T02:34:40.587 INFO:tasks.workunit.client.0.smithi045.stdout:snap_create: snap1 2021-07-31T02:34:40.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:40 smithi045 conmon[33655]: cluster 2021-07-31T02:34:39.504261+0000 mon.a (mon.0) 2458 2021-07-31T02:34:40.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:40 smithi045 conmon[33655]: : cluster [DBG] osdmap e1488: 8 total, 8 up, 8 in 2021-07-31T02:34:40.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:40 smithi045 conmon[39718]: cluster 2021-07-31T02:34:39.504261+0000 2021-07-31T02:34:40.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:40 smithi045 conmon[39718]: mon.a (mon.0) 2458 : cluster [DBG] osdmap e1488: 8 total, 8 up, 8 in 2021-07-31T02:34:41.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:40 smithi154 conmon[34553]: cluster 2021-07-31T02:34:39.504261+0000 mon.a (mon.0) 2458 : cluster [DBG] osdmap e1488: 8 total, 8 up, 8 in 2021-07-31T02:34:41.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:41 smithi045 conmon[33655]: cluster 2021-07-31T02:34:40.479350+0000 2021-07-31T02:34:41.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:41 smithi045 conmon[33655]: mgr.x (mgr.24542) 2036 : cluster [DBG] pgmap v2758: 73 pgs: 73 active+clean; 1.2 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 111 MiB/s rd, 13 MiB/s wr, 291 op/s 2021-07-31T02:34:41.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:41 smithi045 conmon[33655]: cluster 2021-07-31T02:34:40.514032+0000 mon.a (mon.0) 2021-07-31T02:34:41.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:41 smithi045 conmon[33655]: 2459 : cluster [DBG] osdmap e1489: 8 total, 8 up, 8 in 2021-07-31T02:34:41.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:41 smithi045 conmon[39718]: cluster 2021-07-31T02:34:40.479350 2021-07-31T02:34:41.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:41 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 2036 : cluster [DBG] pgmap v2758: 73 pgs: 73 active+clean; 1.2 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 111 MiB/s rd, 13 MiB/s wr, 291 op/s 2021-07-31T02:34:41.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:41 smithi045 conmon[39718]: cluster 2021-07-31T02:34:40.514032+0000 mon.a ( 2021-07-31T02:34:41.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:41 smithi045 conmon[39718]: mon.0) 2459 : cluster [DBG] osdmap e1489: 8 total, 8 up, 8 in 2021-07-31T02:34:42.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:41 smithi154 conmon[34553]: cluster 2021-07-31T02:34:40.479350+0000 mgr.x (mgr.24542) 2036 : cluster [DBG] pgmap v2758: 73 pgs: 73 active+clean; 1.2 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 111 MiB/s rd, 13 MiB/s wr, 291 op/s 2021-07-31T02:34:42.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:41 smithi154 conmon[34553]: cluster 2021-07-31T02:34:40.514032+0000 mon.a (mon.0) 2459 : cluster [DBG] osdmap e1489: 8 total, 8 up, 8 in 2021-07-31T02:34:42.406 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 92749771 2021-07-31T02:34:42.421 INFO:tasks.workunit.client.0.smithi045.stdout:write: C 64737069~1970023 2021-07-31T02:34:42.425 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 60362832~6975093 2021-07-31T02:34:42.436 INFO:tasks.workunit.client.0.smithi045.stdout:write: C 12163816~3639105 2021-07-31T02:34:42.438 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 25231247~8291601 2021-07-31T02:34:42.439 INFO:tasks.workunit.client.0.smithi045.stdout:write: C 13403518~6639047 2021-07-31T02:34:42.466 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 48855923~3688885 2021-07-31T02:34:42.466 INFO:tasks.workunit.client.0.smithi045.stdout:write: C 4123870~769480 2021-07-31T02:34:42.467 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 18376566~2383617 2021-07-31T02:34:42.486 INFO:tasks.workunit.client.0.smithi045.stdout:snap_create: snap2 2021-07-31T02:34:42.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:42 smithi045 conmon[39718]: cluster 2021-07-31T02:34:41.516417+0000 2021-07-31T02:34:42.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:42 smithi045 conmon[39718]: mon.a (mon.0) 2460 : cluster [DBG] osdmap e1490: 8 total, 8 up, 8 in 2021-07-31T02:34:42.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:42 smithi045 conmon[39718]: cluster 2021-07- 2021-07-31T02:34:42.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:42 smithi045 conmon[39718]: 31T02:34:42.385600+0000 mon.a (mon.0) 2461 : 2021-07-31T02:34:42.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:42 smithi045 conmon[39718]: cluster [DBG] osdmap e1491: 8 total, 8 up, 8 in 2021-07-31T02:34:42.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:42 smithi045 conmon[33655]: cluster 2021-07-31T02:34:41.516417+0000 mon.a (mon.0) 2460 : cluster [DBG] osdmap e1490: 8 total, 8 up, 8 in 2021-07-31T02:34:42.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:42 smithi045 conmon[33655]: cluster 2021-07-31T02:34:42.385600+0000 mon.a (mon.0) 2461 : cluster [DBG] osdmap e1491: 8 total, 8 up, 8 in 2021-07-31T02:34:43.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:42 smithi154 conmon[34553]: cluster 2021-07-31T02:34:41.516417+0000 mon.a (mon.0) 2460 : cluster [DBG] osdmap e1490: 8 total, 8 up, 8 in 2021-07-31T02:34:43.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:42 smithi154 conmon[34553]: cluster 2021-07-31T02:34: 2021-07-31T02:34:43.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:42 smithi154 conmon[34553]: 42.385600+0000 mon.a (mon.0) 2461 : cluster [DBG] osdmap e1491: 8 total, 8 up, 8 in 2021-07-31T02:34:44.404 INFO:tasks.workunit.client.0.smithi045.stdout:write: D 8765299~7514761 2021-07-31T02:34:44.412 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 85288772~7316943 2021-07-31T02:34:44.412 INFO:tasks.workunit.client.0.smithi045.stdout:write: D 14237458~4298 2021-07-31T02:34:44.448 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 42534287~1931156 2021-07-31T02:34:44.448 INFO:tasks.workunit.client.0.smithi045.stdout:write: D 69417737~6282903 2021-07-31T02:34:44.451 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 75385104~4878930 2021-07-31T02:34:44.480 INFO:tasks.workunit.client.0.smithi045.stdout:write: D 84818662~909159 2021-07-31T02:34:44.481 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 29241299~6133403 2021-07-31T02:34:44.481 INFO:tasks.workunit.client.0.smithi045.stdout:snap_create: snap3 2021-07-31T02:34:44.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:44 smithi154 conmon[34553]: cluster 2021-07-31T02:34:42.479773+0000 mgr.x (mgr.24542) 2037 : cluster [DBG] pgmap v2762: 73 pgs: 73 active+clean; 1.2 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 112 MiB/s rd, 14 MiB/s wr, 293 op/s 2021-07-31T02:34:44.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:44 smithi154 conmon[34553]: cluster 2021-07-31T02:34:43. 2021-07-31T02:34:44.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:44 smithi154 conmon[34553]: 394024+0000 mon.a (mon.0) 2462 : cluster [DBG] osdmap e1492: 8 total, 8 up, 8 in 2021-07-31T02:34:44.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:44 smithi045 conmon[39718]: cluster 2021-07-31T02:34:42.479773 2021-07-31T02:34:44.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:44 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 2037 : cluster [DBG] pgmap v2762: 73 pgs: 73 active+clean; 1.2 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 112 MiB/s rd, 14 MiB/s wr, 293 op/s 2021-07-31T02:34:44.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:44 smithi045 conmon[39718]: cluster 2021-07-31T02:34:43.394024 2021-07-31T02:34:44.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:44 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2462 : cluster [DBG] osdmap e1492: 8 total, 8 up, 8 in 2021-07-31T02:34:44.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:44 smithi045 conmon[39718]: 2021-07-31T02:34:44.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:44 smithi045 conmon[33655]: cluster 2021-07-31T02:34:44.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:44 smithi045 conmon[33655]: 2021-07-31T02: 2021-07-31T02:34:44.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:44 smithi045 conmon[33655]: 34:42.479773+0000 mgr.x (mgr.24542) 2037 : cluster [DBG] pgmap v2762: 73 pgs: 73 active+clean; 1.2 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 112 MiB/s rd, 14 MiB/s wr, 293 op/s 2021-07-31T02:34:44.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:44 smithi045 conmon[33655]: cluster 2021-07-31T02:34:44.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:44 smithi045 conmon[33655]: 2021-07-31T02:34 2021-07-31T02:34:44.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:44 smithi045 conmon[33655]: :43.394024+0000 mon.a (mon.0) 2462 : cluster [DBG] 2021-07-31T02:34:44.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:44 smithi045 conmon[33655]: osdmap e1492: 8 total, 8 up, 8 in 2021-07-31T02:34:45.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:45 smithi154 conmon[34553]: cluster 2021-07-31T02:34:44.396684+0000 mon.a (mon.0) 2463 : cluster 2021-07-31T02:34:45.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:45 smithi154 conmon[34553]: [DBG] osdmap e1493: 8 total, 8 up, 8 in 2021-07-31T02:34:45.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:45 smithi154 conmon[34553]: cluster 2021-07-31T02:34:44.480150+0000 mgr.x (mgr.24542) 2038 : cluster [DBG] pgmap v2765: 73 pgs: 73 active+clean; 1.3 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 50 MiB/s rd, 22 MiB/s wr, 58 op/s 2021-07-31T02:34:45.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:45 smithi045 conmon[33655]: cluster 2021-07-31T02:34:44.396684+0000 mon.a (mon.0) 2463 2021-07-31T02:34:45.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:45 smithi045 conmon[33655]: : cluster [DBG] osdmap e1493: 8 total, 8 up, 8 in 2021-07-31T02:34:45.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:45 smithi045 conmon[33655]: cluster 2021-07-31T02:34:44.480150+0000 mgr.x (mgr.24542) 2038 : cluster [DBG] pgmap v2765: 73 pgs: 73 active+clean; 1.3 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 50 MiB/s rd, 22 MiB/s wr, 58 op/s 2021-07-31T02:34:45.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:45 smithi045 conmon[39718]: cluster 2021-07-31T02:34:44. 2021-07-31T02:34:45.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:45 smithi045 conmon[39718]: 396684+0000 mon.a (mon.0) 2463 : cluster [DBG] osdmap e1493: 8 total, 8 up, 8 in 2021-07-31T02:34:45.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:45 smithi045 conmon[39718]: cluster 2021-07-31T02:34:44.480150+0000 mgr.x (mgr.24542) 2038 : 2021-07-31T02:34:45.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:45 smithi045 conmon[39718]: cluster [DBG] pgmap v2765: 73 pgs: 73 active+clean; 1.3 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 50 MiB/s rd, 22 MiB/s wr, 58 op/s 2021-07-31T02:34:46.418 INFO:tasks.workunit.client.0.smithi045.stdout:migration_prepare 2021-07-31T02:34:46.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:46 smithi154 conmon[34553]: cluster 2021-07-31T02:34:45.407617+0000 mon.a (mon.0) 2464 : cluster [DBG] osdmap e1494: 8 total, 8 up, 8 in 2021-07-31T02:34:46.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:46 smithi045 conmon[39718]: cluster 2021-07-31T02:34:46.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:46 smithi045 conmon[39718]: 2021-07 2021-07-31T02:34:46.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:46 smithi045 conmon[39718]: -31T02:34:45.407617+0000 mon.a (mon.0) 2464 : cluster [DBG] osdmap e1494: 8 total, 8 up, 8 in 2021-07-31T02:34:46.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:46 smithi045 conmon[33655]: cluster 2021-07-31T02:34:45.407617+0000 2021-07-31T02:34:46.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:46 smithi045 conmon[33655]: mon.a (mon.0) 2464 : cluster [DBG] osdmap e1494: 8 total, 8 up, 8 in 2021-07-31T02:34:47.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:47 smithi154 conmon[34553]: cluster 2021-07-31T02:34:46.409113+0000 mon.a (mon.0) 2465 : cluster [DBG] osdmap e1495: 8 total, 8 up, 8 in 2021-07-31T02:34:47.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:47 smithi154 conmon[34553]: 2021-07-31T02:34:47.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:47 smithi154 conmon[34553]: cluster 2021-07-31T02:34:46.480573+0000 mgr.x (mgr.24542) 2039 : cluster [DBG] pgmap v2768: 73 pgs: 73 active+clean; 1.3 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 49 MiB/s rd, 22 MiB/s wr, 58 op/s 2021-07-31T02:34:47.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:47 smithi045 conmon[39718]: cluster 2021-07-31T02:34:46. 2021-07-31T02:34:47.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:47 smithi045 conmon[39718]: 409113+0000 mon.a (mon.0) 2465 : cluster [DBG] osdmap e1495: 8 total, 8 up, 8 in 2021-07-31T02:34:47.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:47 smithi045 conmon[39718]: cluster 2021-07-31T 2021-07-31T02:34:47.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:47 smithi045 conmon[39718]: 02:34:46.480573+0000 mgr.x (mgr.24542) 2039 : 2021-07-31T02:34:47.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:47 smithi045 conmon[39718]: cluster [DBG] pgmap v2768: 73 pgs: 73 active+clean; 1.3 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 49 MiB/s rd, 22 MiB/s wr, 58 op/s 2021-07-31T02:34:47.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:47 smithi045 conmon[33655]: cluster 2021-07-31T02:34:46.409113+0000 mon.a (mon.0) 2465 2021-07-31T02:34:47.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:47 smithi045 conmon[33655]: : cluster [DBG] osdmap e1495: 8 total, 8 up, 8 in 2021-07-31T02:34:47.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:47 smithi045 conmon[33655]: cluster 2021-07-31T02:34:46.480573+0000 mgr.x (mgr.24542) 2039 : cluster [DBG] pgmap v2768: 73 pgs: 73 active+clean; 1.3 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 49 MiB/s rd, 22 MiB/s wr, 58 op/s 2021-07-31T02:34:48.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:48 smithi154 conmon[34553]: cluster 2021-07-31T02:34:47.419774+0000 mon.a (mon.0) 2466 : cluster [DBG] osdmap e1496: 8 total, 8 up, 8 in 2021-07-31T02:34:48.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:48 smithi045 conmon[39718]: cluster 2021-07-31T02:34:47.419774+0000 mon.a (mon.0) 2466 : cluster [DBG] osdmap e1496: 8 total, 8 up, 8 in 2021-07-31T02:34:48.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:48 smithi045 conmon[33655]: cluster 2021-07-31T02:34:47.419774+0000 mon.a (mon. 2021-07-31T02:34:48.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:48 smithi045 conmon[33655]: 0) 2466 : cluster [DBG] osdmap e1496: 8 total, 8 up, 8 in 2021-07-31T02:34:49.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:49 smithi154 conmon[34553]: cluster 2021-07-31T02:34:48.425560+0000 mon.a (mon.0) 2467 : cluster 2021-07-31T02:34:49.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:49 smithi154 conmon[34553]: [DBG] osdmap e1497: 8 total, 8 up, 8 in 2021-07-31T02:34:49.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:49 smithi154 conmon[34553]: cluster 2021-07-31T02:34:48.481048+0000 mgr.x (mgr.24542) 2040 : cluster [DBG] pgmap v2771: 73 pgs: 73 active+clean; 1.3 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail 2021-07-31T02:34:49.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:49 smithi045 conmon[33655]: cluster 2021-07-31T02:34:48.425560+0000 mon.a (mon.0) 2467 : cluster [DBG] osdmap e1497: 8 total, 8 up, 8 in 2021-07-31T02:34:49.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:49 smithi045 conmon[33655]: cluster 2021-07-31T02:34:48.481048+0000 mgr.x (mgr.24542) 2040 : cluster [DBG] pgmap v2771: 73 pgs: 73 active+clean; 1.3 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail 2021-07-31T02:34:49.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:49 smithi045 conmon[39718]: cluster 2021-07-31T02:34:48.425560+0000 mon.a (mon.0) 2467 2021-07-31T02:34:49.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:49 smithi045 conmon[39718]: : cluster [DBG] osdmap e1497: 8 total, 8 up, 8 in 2021-07-31T02:34:49.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:49 smithi045 conmon[39718]: cluster 2021-07-31T02:34:48.481048+0000 mgr.x (mgr.24542) 2040 : 2021-07-31T02:34:49.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:49 smithi045 conmon[39718]: cluster [DBG] pgmap v2771: 73 pgs: 73 active+clean; 1.3 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail 2021-07-31T02:34:50.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:50 smithi154 conmon[34553]: cluster 2021-07-31T02:34:49.431411+0000 mon.a (mon.0) 2468 : cluster [DBG] 2021-07-31T02:34:50.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:50 smithi154 conmon[34553]: osdmap e1498: 8 total, 8 up, 8 in 2021-07-31T02:34:50.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:50 smithi045 conmon[39718]: cluster 2021-07-31T02:34:49 2021-07-31T02:34:50.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:50 smithi045 conmon[39718]: .431411+0000 mon.a (mon.0) 2468 : cluster [DBG] osdmap e1498: 8 total, 8 up, 8 in 2021-07-31T02:34:50.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:50 smithi045 conmon[33655]: cluster 2021-07-31T02:34:50.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:50 smithi045 conmon[33655]: 2021-07-31T02:34:50.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:50 smithi045 conmon[33655]: 2021 2021-07-31T02:34:50.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:50 smithi045 conmon[33655]: - 2021-07-31T02:34:50.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:50 smithi045 conmon[33655]: 07-31T02:34:49.431411 2021-07-31T02:34:50.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:50 smithi045 conmon[33655]: +0000 mon.a (mon.0) 2468 : cluster [DBG] osdmap e1498: 8 total, 8 up, 8 in 2021-07-31T02:34:51.018 INFO:tasks.workunit.client.0.smithi045.stdout:write: a 24491796~8251062 2021-07-31T02:34:51.075 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 54685039~5357493 2021-07-31T02:34:51.104 INFO:tasks.workunit.client.0.smithi045.stdout:write: a 75779824~504447 2021-07-31T02:34:51.109 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 82853499~4943243 2021-07-31T02:34:51.119 INFO:tasks.workunit.client.0.smithi045.stdout:write: a 47017794~2770177 2021-07-31T02:34:51.139 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 13542402~5154507 2021-07-31T02:34:51.193 INFO:tasks.workunit.client.0.smithi045.stdout:write: a 85198738~4624462 2021-07-31T02:34:51.231 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 59920745~4279947 2021-07-31T02:34:51.234 INFO:tasks.workunit.client.0.smithi045.stdout:snap_create: user0 2021-07-31T02:34:51.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:51 smithi154 conmon[34553]: cluster 2021-07-31T02:34:50.432999+0000 mon.a (mon.0) 2469 : cluster [DBG] osdmap e1499: 8 total, 8 up, 8 in 2021-07-31T02:34:51.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:51 smithi154 conmon[34553]: 2021-07-31T02:34:51.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:51 smithi154 conmon[34553]: cluster 2021-07-31T02:34:50.481452+0000 mgr.x (mgr.24542) 2041 : cluster [DBG] pgmap v2774: 73 pgs: 73 active+clean; 1.3 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 18 KiB/s rd, 7.0 MiB/s wr, 31 op/s 2021-07-31T02:34:51.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:51 smithi045 conmon[39718]: cluster 2021-07-31T02: 2021-07-31T02:34:51.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:51 smithi045 conmon[39718]: 34:50.432999+0000 mon.a (mon.0) 2469 : cluster [DBG] osdmap e1499: 8 total, 8 up, 8 in 2021-07-31T02:34:51.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:51 smithi045 conmon[39718]: cluster 2021-07-31T02:34:51.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:51 smithi045 conmon[39718]: 2021-07-31T02:34 2021-07-31T02:34:51.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:51 smithi045 conmon[39718]: :50.481452+0000 mgr.x ( 2021-07-31T02:34:51.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:51 smithi045 conmon[39718]: mgr.24542) 2041 : cluster 2021-07-31T02:34:51.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:51 smithi045 conmon[39718]: [DBG] pgmap v2774: 73 pgs: 73 active+clean; 1.3 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 18 KiB/s rd, 7.0 MiB/s wr, 31 op/s 2021-07-31T02:34:51.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:51 smithi045 conmon[33655]: cluster 2021-07-31T02:34:50.432999+0000 mon.a 2021-07-31T02:34:51.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:51 smithi045 conmon[33655]: (mon.0) 2469 : cluster [DBG] osdmap e1499: 8 total, 8 up, 8 in 2021-07-31T02:34:51.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:51 smithi045 conmon[33655]: cluster 2021-07-31T02:34:50.481452+0000 mgr.x (mgr.24542 2021-07-31T02:34:51.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:51 smithi045 conmon[33655]: ) 2041 : cluster [DBG] pgmap v2774: 73 pgs: 73 active+clean; 1.3 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 18 KiB/s rd, 7.0 MiB/s wr, 31 op/s 2021-07-31T02:34:52.396 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:34:52.450 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 168703970 2021-07-31T02:34:52.459 INFO:tasks.workunit.client.0.smithi045.stdout:write: b 94873601~1367673 2021-07-31T02:34:52.467 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 137126814~2550769 2021-07-31T02:34:52.468 INFO:tasks.workunit.client.0.smithi045.stdout:write: b 85638095~7947688 2021-07-31T02:34:52.471 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 119299352~7788016 2021-07-31T02:34:52.472 INFO:tasks.workunit.client.0.smithi045.stdout:write: b 135977285~5456605 2021-07-31T02:34:52.479 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 65144709~1467255 2021-07-31T02:34:52.479 INFO:tasks.workunit.client.0.smithi045.stdout:write: b 397899~2701556 2021-07-31T02:34:52.480 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 154394943~8233658 2021-07-31T02:34:52.517 INFO:tasks.workunit.client.0.smithi045.stdout:snap_create: user1 2021-07-31T02:34:52.705 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:34:52.705 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 49m ago 50m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:34:52.706 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 47m ago 50m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:34:52.706 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 47m ago 53m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:34:52.706 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 49m ago 55m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:34:52.706 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 49m ago 56m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:34:52.707 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 47m ago 54m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:34:52.707 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 49m ago 54m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:34:52.707 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 49m ago 50m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:34:52.707 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 47m ago 50m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:34:52.707 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 49m ago 53m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:34:52.708 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 49m ago 53m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:34:52.708 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 49m ago 52m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:34:52.708 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 49m ago 52m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:34:52.708 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 47m ago 52m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:34:52.709 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 47m ago 51m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:34:52.709 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 47m ago 51m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:34:52.709 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 47m ago 51m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:34:52.709 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 47m ago 50m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:34:52.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:52 smithi154 conmon[34553]: cluster 2021-07-31T02:34:51.443101+0000 mon.a (mon.0) 2470 : cluster [DBG] 2021-07-31T02:34:52.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:52 smithi154 conmon[34553]: osdmap e1500: 8 total, 8 up, 8 in 2021-07-31T02:34:52.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:52 smithi154 conmon[34553]: audit 2021-07-31T02:34:52.382499+0000 mgr.x (mgr.24542) 2042 : audit [DBG] from='client.17478 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:34:52.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:52 smithi045 conmon[33655]: cluster 2021-07-31T02:34:51.443101+0000 mon.a (mon.0) 2470 : cluster [DBG] osdmap e1500: 8 total, 8 up, 8 in 2021-07-31T02:34:52.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:52 smithi045 conmon[33655]: audit 2021-07-31T02:34:52.382499+0000 mgr.x (mgr.24542) 2042 : audit [DBG] from='client.17478 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:34:52.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:52 smithi045 conmon[39718]: cluster 2021-07-31T02:34:51.443101 2021-07-31T02:34:52.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:52 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2470 : cluster [DBG] osdmap e1500: 8 total, 8 up, 8 in 2021-07-31T02:34:52.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:52 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T02:34:52.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:52 smithi045 conmon[39718]: T02:34:52.382499+0000 mgr.x (mgr.24542) 2042 : audit [DBG] from='client.17478 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:34:52.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:52 smithi045 conmon[39718]: 2021-07-31T02:34:53.043 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:34:53.043 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:34:53.043 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:34:53.043 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:34:53.044 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:34:53.044 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:34:53.044 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:34:53.044 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:34:53.045 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:34:53.045 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:34:53.045 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:34:53.045 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:34:53.046 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:34:53.046 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:34:53.046 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:34:53.046 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:34:53.046 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:34:53.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:53 smithi154 conmon[34553]: cluster 2021-07-31T02:34:52.444929+0000 mon.a (mon.0) 2471 : cluster [DBG] osdmap e1501: 8 total, 8 up, 8 in 2021-07-31T02:34:53.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:53 smithi154 conmon[34553]: cluster 2021-07-31T02:34:52.482236+0000 mgr.x (mgr.24542) 2043 : cluster [DBG] pgmap v2777: 73 pgs: 73 active+clean; 1.3 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 18 KiB/s rd, 7.0 MiB/s wr, 31 op/s 2021-07-31T02:34:53.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:53 smithi154 conmon[34553]: audit 2021-07-31T02:34:52.698406+0000 mgr.x (mgr.24542) 2044 : audit [DBG] from='client.27424 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:34:53.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:53 smithi154 conmon[34553]: audit 2021-07-31T02:34:53.043145+0000 mon.c (mon.1) 2021-07-31T02:34:53.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:53 smithi154 conmon[34553]: 119 : audit [DBG] from='client.? 172.21.15.45:0/4017303425' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:34:53.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:53 smithi045 conmon[39718]: cluster 2021-07-31T02:34:52.444929+0000 mon.a (mon.0) 2021-07-31T02:34:53.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:53 smithi045 conmon[39718]: 2471 : cluster [DBG] osdmap e1501: 8 total, 8 up, 8 in 2021-07-31T02:34:53.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:53 smithi045 conmon[39718]: cluster 2021-07-31T02:34:52.482236+0000 mgr.x (mgr.24542) 2043 : cluster [DBG] 2021-07-31T02:34:53.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:53 smithi045 conmon[39718]: pgmap v2777: 73 pgs: 73 active+clean; 1.3 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 18 KiB/s rd, 7.0 MiB/s wr, 31 op/s 2021-07-31T02:34:53.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:53 smithi045 conmon[39718]: audit 2021-07-31T02:34:52.698406 2021-07-31T02:34:53.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:53 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 2044 : audit [DBG] from='client.27424 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:34:53.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:53 smithi045 conmon[39718]: audit 2021-07-31T02:34:53.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:53 smithi045 conmon[39718]: 2021-07-31T02:34:53.043145+0000 mon.c (mon.1) 119 : audit [DBG] from='client.? 172.21.15.45:0/4017303425' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:34:53.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:53 smithi045 conmon[33655]: cluster 2021-07-31T02:34:52.444929+0000 mon.a (mon 2021-07-31T02:34:53.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:53 smithi045 conmon[33655]: .0) 2471 : cluster [DBG] osdmap e1501: 8 total, 8 up, 8 in 2021-07-31T02:34:53.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:53 smithi045 conmon[33655]: cluster 2021-07-31T02:34:52.482236+0000 mgr.x (mgr 2021-07-31T02:34:53.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:53 smithi045 conmon[33655]: .24542) 2043 : cluster [DBG] pgmap v2777: 73 pgs: 73 active+clean; 1.3 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 18 KiB/s rd, 7.0 MiB/s wr, 31 op/s 2021-07-31T02:34:53.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:53 smithi045 conmon[33655]: 2021-07-31T02:34:53.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:53 smithi045 conmon[33655]: audit 2021-07-31T02:34:53.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:53 smithi045 conmon[33655]: 2021-07-31T02 2021-07-31T02:34:53.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:53 smithi045 conmon[33655]: :34:52.698406+0000 mgr.x 2021-07-31T02:34:53.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:53 smithi045 conmon[33655]: (mgr.24542) 2044 : audit 2021-07-31T02:34:53.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:53 smithi045 conmon[33655]: [DBG] from='client.27424 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:34:53.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:53 smithi045 conmon[33655]: audit 2021-07-31T02:34:53.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:53 smithi045 conmon[33655]: 2021-07-31 2021-07-31T02:34:53.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:53 smithi045 conmon[33655]: T 2021-07-31T02:34:53.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:53 smithi045 conmon[33655]: 02:34:53.043145+0000 2021-07-31T02:34:53.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:53 smithi045 conmon[33655]: mon.c (mon.1) 119 : audit 2021-07-31T02:34:53.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:53 smithi045 conmon[33655]: [DBG] from='client.? 172.21.15.45:0/4017303425' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:34:54.465 INFO:tasks.workunit.client.0.smithi045.stdout:write: c 35903314~349498 2021-07-31T02:34:54.472 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 41826820~6038399 2021-07-31T02:34:54.472 INFO:tasks.workunit.client.0.smithi045.stdout:write: c 116143977~6048552 2021-07-31T02:34:54.478 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 100683922~1308730 2021-07-31T02:34:54.478 INFO:tasks.workunit.client.0.smithi045.stdout:write: c 157400958~7986977 2021-07-31T02:34:54.485 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 153822626~7093686 2021-07-31T02:34:54.536 INFO:tasks.workunit.client.0.smithi045.stdout:write: c 80415341~174769 2021-07-31T02:34:54.536 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 41918974~2329417 2021-07-31T02:34:54.536 INFO:tasks.workunit.client.0.smithi045.stdout:snap_create: user2 2021-07-31T02:34:54.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:54 smithi154 conmon[34553]: cluster 2021-07-31T02:34:53.458184+0000 mon.a (mon.0) 2472 : cluster [DBG] osdmap e1502: 8 total, 8 up, 8 in 2021-07-31T02:34:54.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:54 smithi154 conmon[34553]: 2021-07-31T02:34:54.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:54 smithi045 conmon[33655]: cluster 2021-07-31T02:34:53.458184+0000 mon.a (mon.0 2021-07-31T02:34:54.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:54 smithi045 conmon[33655]: ) 2472 : cluster [DBG] osdmap e1502: 8 total, 8 up, 8 in 2021-07-31T02:34:54.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:54 smithi045 conmon[39718]: cluster 2021-07-31T02:34:54.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:54 smithi045 conmon[39718]: 2021-07-31T02:34:54.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:54 smithi045 conmon[39718]: 2021 2021-07-31T02:34:54.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:54 smithi045 conmon[39718]: - 2021-07-31T02:34:54.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:54 smithi045 conmon[39718]: 07 2021-07-31T02:34:54.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:54 smithi045 conmon[39718]: - 2021-07-31T02:34:54.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:54 smithi045 conmon[39718]: 31 2021-07-31T02:34:54.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:54 smithi045 conmon[39718]: T 2021-07-31T02:34:54.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:54 smithi045 conmon[39718]: 02 2021-07-31T02:34:54.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:54 smithi045 conmon[39718]: : 2021-07-31T02:34:54.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:54 smithi045 conmon[39718]: 34:53.458184+0000 mon.a 2021-07-31T02:34:54.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:54 smithi045 conmon[39718]: (mon.0) 2472 : cluster [DBG] osdmap e1502: 8 total, 8 up, 8 in 2021-07-31T02:34:55.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:55 smithi154 conmon[34553]: cluster 2021-07-31T02:34:54.460122+0000 mon.a (mon.0) 2473 : cluster [DBG] osdmap e1503: 8 total, 8 up, 8 in 2021-07-31T02:34:55.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:55 smithi154 conmon[34553]: cluster 2021-07-31T02:34:54.482980+0000 mgr.x (mgr.24542) 2045 : cluster [DBG] pgmap v2780: 73 pgs: 73 active+clean; 1.5 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 59 MiB/s rd, 24 MiB/s wr, 160 op/s 2021-07-31T02:34:55.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:55 smithi045 conmon[39718]: cluster 2021-07-31T02:34:54.460122+0000 mon.a (mon.0) 2473 : cluster [DBG] osdmap e1503: 8 total, 8 up, 8 in 2021-07-31T02:34:55.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:55 smithi045 conmon[39718]: cluster 2021-07-31T02:34:54.482980+0000 mgr.x (mgr.24542) 2045 : cluster [DBG] pgmap v2780: 73 pgs: 73 active+clean; 1.5 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 59 MiB/s rd, 24 MiB/s wr, 160 op/s 2021-07-31T02:34:55.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:55 smithi045 conmon[33655]: cluster 2021-07-31T02:34:54.460122+0000 mon.a (mon.0) 2473 : cluster [DBG] osdmap e1503: 8 total, 8 up, 8 in 2021-07-31T02:34:55.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:55 smithi045 conmon[33655]: cluster 2021-07-31T02:34:54.482980+0000 mgr.x (mgr.24542) 2045 : cluster [DBG] pgmap v2780: 73 pgs: 73 active+clean; 1.5 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 59 MiB/s rd, 24 MiB/s wr, 160 op/s 2021-07-31T02:34:56.482 INFO:tasks.workunit.client.0.smithi045.stdout:write: d 48937171~5238503 2021-07-31T02:34:56.496 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 18619698~6422906 2021-07-31T02:34:56.496 INFO:tasks.workunit.client.0.smithi045.stdout:write: d 55688489~5121095 2021-07-31T02:34:56.501 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 112844824~6395151 2021-07-31T02:34:56.501 INFO:tasks.workunit.client.0.smithi045.stdout:write: d 25876241~4322178 2021-07-31T02:34:56.514 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 145844072~7918059 2021-07-31T02:34:56.531 INFO:tasks.workunit.client.0.smithi045.stdout:write: d 66803979~378541 2021-07-31T02:34:56.531 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 93138491~4516726 2021-07-31T02:34:56.533 INFO:tasks.workunit.client.0.smithi045.stdout:snap_create: user3 2021-07-31T02:34:56.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:56 smithi154 conmon[34553]: cluster 2021-07-31T02:34:55.471286+0000 mon.a (mon.0) 2474 : cluster [DBG] osdmap e1504: 8 total, 8 up, 8 in 2021-07-31T02:34:56.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:56 smithi045 conmon[33655]: cluster 2021-07-31T02:34:55.471286+0000 mon.a (mon.0) 2474 : cluster 2021-07-31T02:34:56.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:56 smithi045 conmon[33655]: [DBG] osdmap e1504: 8 total, 8 up, 8 in 2021-07-31T02:34:56.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:56 smithi045 conmon[39718]: cluster 2021-07-31T02:34:55.471286+0000 mon.a (mon. 2021-07-31T02:34:56.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:56 smithi045 conmon[39718]: 0) 2474 : cluster [DBG] osdmap e1504: 8 total, 8 up, 8 in 2021-07-31T02:34:57.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:57 smithi154 conmon[34553]: cluster 2021-07-31T02:34:56.478240+0000 2021-07-31T02:34:57.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:57 smithi154 conmon[34553]: mon.a (mon.0) 2475 : cluster [DBG] osdmap e1505: 8 total, 8 up, 8 in 2021-07-31T02:34:57.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:57 smithi154 conmon[34553]: cluster 2021-07-31T02:34:56.483397+0000 mgr.x (mgr.24542) 2046 : cluster [DBG] pgmap v2783: 73 pgs: 73 active+clean; 1.5 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 59 MiB/s rd, 24 MiB/s wr, 160 op/s 2021-07-31T02:34:57.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:57 smithi045 conmon[39718]: cluster 2021-07-31T02:34:56.478240+0000 2021-07-31T02:34:57.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:57 smithi045 conmon[39718]: mon.a (mon.0) 2475 : cluster [DBG] osdmap e1505: 8 total, 8 up, 8 in 2021-07-31T02:34:57.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:57 smithi045 conmon[39718]: cluster 2021-07-31T02:34:56.483397+0000 mgr.x (mgr.24542) 2021-07-31T02:34:57.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:57 smithi045 conmon[39718]: 2046 : cluster [DBG] pgmap v2783: 73 pgs: 73 active+clean; 1.5 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 59 MiB/s rd, 24 MiB/s wr, 160 op/s 2021-07-31T02:34:57.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:57 smithi045 conmon[33655]: cluster 2021-07-31T02:34:56.478240+0000 mon.a (mon.0) 2475 2021-07-31T02:34:57.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:57 smithi045 conmon[33655]: : cluster [DBG] osdmap e1505: 8 total, 8 up, 8 in 2021-07-31T02:34:57.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:57 smithi045 conmon[33655]: cluster 2021-07-31T02:34:56.483397+0000 mgr.x (mgr.24542) 2046 : cluster [DBG] pgmap v2783: 73 pgs: 73 active+clean; 1.5 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 59 MiB/s rd, 24 MiB/s wr, 160 op/s 2021-07-31T02:34:58.500 INFO:tasks.workunit.client.0.smithi045.stdout:write: 1 16870397~4217599 2021-07-31T02:34:58.503 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 17924796~2108799 2021-07-31T02:34:58.556 INFO:tasks.workunit.client.0.smithi045.stdout:write: 2 33740794~4217599 2021-07-31T02:34:58.558 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 34795193~2108799 2021-07-31T02:34:58.580 INFO:tasks.workunit.client.0.smithi045.stdout:write: 3 50611191~4217599 2021-07-31T02:34:58.583 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 51665590~2108799 2021-07-31T02:34:58.620 INFO:tasks.workunit.client.0.smithi045.stdout:write: 4 67481588~4217599 2021-07-31T02:34:58.622 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 68535987~2108799 2021-07-31T02:34:58.654 INFO:tasks.workunit.client.0.smithi045.stdout:write: 5 84351985~4217599 2021-07-31T02:34:58.656 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 85406384~2108799 2021-07-31T02:34:58.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:58 smithi154 conmon[34553]: cluster 2021-07-31T02:34:57.485615+0000 mon.a (mon.0) 2476 : cluster [DBG] osdmap e1506: 8 total, 8 up, 8 in 2021-07-31T02:34:58.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:58 smithi154 conmon[34553]: 2021-07-31T02:34:58.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:58 smithi045 conmon[33655]: cluster 2021-07-31T02: 2021-07-31T02:34:58.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:58 smithi045 conmon[33655]: 34:57.485615+0000 mon.a (mon.0) 2476 : cluster [DBG] osdmap e1506: 8 total, 8 up, 8 in 2021-07-31T02:34:58.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:58 smithi045 conmon[39718]: cluster 2021-07-31T02:34:57.485615+0000 mon.a ( 2021-07-31T02:34:58.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:58 smithi045 conmon[39718]: mon.0) 2476 : cluster [DBG] osdmap e1506: 8 total, 8 up, 8 in 2021-07-31T02:34:59.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:59 smithi154 conmon[34553]: cluster 2021-07-31T02:34:58.483848+0000 mgr.x ( 2021-07-31T02:34:59.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:59 smithi154 conmon[34553]: mgr.24542) 2047 : cluster [DBG] pgmap v2785: 73 pgs: 73 active+clean; 1.5 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail 2021-07-31T02:34:59.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:34:59 smithi154 conmon[34553]: cluster 2021-07-31T02:34:58.487286+0000 mon.a (mon.0) 2477 : cluster [DBG] osdmap e1507: 8 total, 8 up, 8 in 2021-07-31T02:34:59.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:59 smithi045 conmon[33655]: cluster 2021-07-31T02:34:58.483848+0000 mgr.x (mgr.24542) 2021-07-31T02:34:59.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:59 smithi045 conmon[33655]: 2047 : cluster [DBG] pgmap v2785: 73 pgs: 73 active+clean; 1.5 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail 2021-07-31T02:34:59.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:34:59 smithi045 conmon[33655]: cluster 2021-07-31T02:34:58.487286+0000 mon.a (mon.0) 2477 : cluster [DBG] osdmap e1507: 8 total, 8 up, 8 in 2021-07-31T02:34:59.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:59 smithi045 conmon[39718]: cluster 2021-07-31T02:34:58.483848+0000 mgr.x ( 2021-07-31T02:34:59.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:59 smithi045 conmon[39718]: mgr.24542) 2047 : cluster [DBG] pgmap v2785: 73 pgs: 73 active+clean; 1.5 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail 2021-07-31T02:34:59.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:59 smithi045 conmon[39718]: cluster 2021-07-31T02:34:58.487286+0000 mon.a (mon.0) 2477 2021-07-31T02:34:59.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:34:59 smithi045 conmon[39718]: : cluster [DBG] osdmap e1507: 8 total, 8 up, 8 in 2021-07-31T02:34:59.950 INFO:tasks.workunit.client.0.smithi045.stdout:migration_execute 2021-07-31T02:35:00.052 INFO:tasks.workunit.client.0.smithi045.stdout:migration_commit 2021-07-31T02:35:01.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:01 smithi154 conmon[34553]: cluster 2021-07-31T02:35:00.484391+0000 mgr.x (mgr.24542) 2048 : cluster [DBG] 2021-07-31T02:35:01.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:01 smithi154 conmon[34553]: pgmap v2787: 73 pgs: 73 active+clean; 1.6 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 18 MiB/s rd, 20 MiB/s wr, 82 op/s 2021-07-31T02:35:01.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:01 smithi154 conmon[34553]: cluster 2021-07-31T02:35:00.497275+0000 mon.a (mon.0) 2478 : cluster [DBG] osdmap e1508: 8 total, 8 up, 8 in 2021-07-31T02:35:01.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:01 smithi045 conmon[39718]: cluster 2021-07-31T02:35:00.484391+0000 mgr.x ( 2021-07-31T02:35:01.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:01 smithi045 conmon[39718]: mgr.24542) 2048 : cluster [DBG] pgmap v2787: 73 pgs: 73 active+clean; 1.6 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 18 MiB/s rd, 20 MiB/s wr, 82 op/s 2021-07-31T02:35:01.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:01 smithi045 conmon[39718]: cluster 2021-07-31T02:35:00.497275+0000 mon.a 2021-07-31T02:35:01.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:01 smithi045 conmon[39718]: (mon.0) 2478 : cluster [DBG] osdmap e1508: 8 total, 8 up, 8 in 2021-07-31T02:35:01.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:01 smithi045 conmon[33655]: cluster 2021-07-31T02:35:00.484391+0000 mgr.x (mgr.24542) 2048 : cluster [DBG] pgmap v2787: 73 pgs: 73 active+clean; 1.6 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 18 MiB/s rd, 20 MiB/s wr, 82 op/s 2021-07-31T02:35:01.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:01 smithi045 conmon[33655]: cluster 2021-07-31T02:35:00.497275+0000 mon.a (mon.0) 2478 : cluster [DBG] osdmap e1508: 8 total, 8 up, 8 in 2021-07-31T02:35:02.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:02 smithi045 conmon[33655]: cluster 2021-07-31T02:35:01.503996+0000 mon.a (mon.0) 2479 : cluster 2021-07-31T02:35:02.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:02 smithi045 conmon[33655]: [DBG] osdmap e1509: 8 total, 8 up, 8 in 2021-07-31T02:35:02.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:02 smithi045 conmon[39718]: cluster 2021-07-31T02:35:01.503996+0000 mon.a (mon. 2021-07-31T02:35:02.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:02 smithi045 conmon[39718]: 0) 2479 : cluster [DBG] osdmap e1509: 8 total, 8 up, 8 in 2021-07-31T02:35:03.002 INFO:journalctl@ceph.osd.5.smithi154.stdout:Jul 31 02:35:02 smithi154 conmon[42050]: debug 2021-07-31T02:35:02.547+0000 7fc9b3060700 -1 osd.5 pg_epoch: 1510 pg[75.10( v 1509'191 (0'0,1509'191] local-lis/les=1308/1309 n=37 ec=1308/1308 lis/c=1308/1308 les/c/f=1309/1309/0 sis=1308) [5,0,3] r=0 lpr=1308 crt=1509'191 lcod 1509'189 mlcod 1509'189 active+clean+snaptrim trimq=[ad~1] ps=[af~1,b1~1]] removing snap head 2021-07-31T02:35:03.003 INFO:journalctl@ceph.osd.5.smithi154.stdout:Jul 31 02:35:02 smithi154 conmon[42050]: debug 2021-07-31T02:35:02.558+0000 7fc9b6066700 -1 osd.5 pg_epoch: 1510 pg[75.2( v 1509'185 (0'0,1509'185] local-lis/les=1308/1309 n=29 ec=1308/1308 lis/c=1308/1308 les/c/f=1309/1309/0 sis=1308) [5,0,7] r=0 lpr=1308 crt=1509'185 lcod 1507'183 mlcod 1507'183 active+clean+snaptrim trimq=[ad~1] ps=[af~1,b1~1]] removing snap head 2021-07-31T02:35:03.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:02 smithi154 conmon[34553]: cluster 2021-07-31T02:35:01.503996+0000 mon.a (mon.0) 2479 : cluster [DBG] osdmap e1509: 8 total, 8 up, 8 in 2021-07-31T02:35:03.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:02 smithi154 conmon[34553]: 2021-07-31T02:35:03.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:03 smithi045 conmon[33655]: cluster 2021-07-31T02:35:02.485032+0000 mgr.x (mgr 2021-07-31T02:35:03.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:03 smithi045 conmon[33655]: .24542) 2049 : cluster [DBG] pgmap v2790: 73 pgs: 73 active+clean; 1.6 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 18 MiB/s rd, 21 MiB/s wr, 83 op/s 2021-07-31T02:35:03.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:03 smithi045 conmon[33655]: cluster 2021-07-31T02:35:02.510076+0000 mon.a (mon.0) 2480 : cluster [DBG] osdmap e1510: 8 total, 8 up, 8 in 2021-07-31T02:35:03.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:03 smithi045 conmon[39718]: cluster 2021-07-31T02:35:02.485032+0000 2021-07-31T02:35:03.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:03 smithi045 conmon[39718]: mgr.x (mgr.24542) 2049 : cluster [DBG] pgmap v2790: 73 pgs: 73 active+clean; 1.6 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 18 MiB/s rd, 21 MiB/s wr, 83 op/s 2021-07-31T02:35:03.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:03 smithi045 conmon[39718]: cluster 2021-07-31T02:35:02.510076+0000 mon.a (mon.0) 2480 : cluster [DBG] osdmap e1510: 8 total, 8 up, 8 in 2021-07-31T02:35:04.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:03 smithi154 conmon[34553]: cluster 2021-07-31T02:35:02.485032+0000 mgr.x 2021-07-31T02:35:04.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:03 smithi154 conmon[34553]: (mgr.24542) 2049 : cluster [DBG] pgmap v2790: 73 pgs: 73 active+clean; 1.6 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 18 MiB/s rd, 21 MiB/s wr, 83 op/s 2021-07-31T02:35:04.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:03 smithi154 conmon[34553]: cluster 2021-07-31T02:35:02.510076+0000 mon.a (mon.0) 2480 : cluster 2021-07-31T02:35:04.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:03 smithi154 conmon[34553]: [DBG] osdmap e1510: 8 total, 8 up, 8 in 2021-07-31T02:35:04.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:04 smithi045 conmon[39718]: cluster 2021-07-31T02:35:03.512346+0000 2021-07-31T02:35:04.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:04 smithi045 conmon[39718]: mon.a (mon.0) 2481 : cluster [DBG] osdmap e1511: 8 total, 8 up, 8 in 2021-07-31T02:35:04.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:04 smithi045 conmon[33655]: cluster 2021-07-31T02:35:03.512346+0000 mon.a (mon. 2021-07-31T02:35:04.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:04 smithi045 conmon[33655]: 0) 2481 : cluster [DBG] osdmap e1511: 8 total, 8 up, 8 in 2021-07-31T02:35:04.796 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMigration.Stress2 (25649 ms) 2021-07-31T02:35:04.797 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMigration.StressLive 2021-07-31T02:35:04.813 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 83886080 2021-07-31T02:35:04.820 INFO:tasks.workunit.client.0.smithi045.stdout:write: A 43922719~1636913 2021-07-31T02:35:04.833 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 43178322~4130598 2021-07-31T02:35:04.835 INFO:tasks.workunit.client.0.smithi045.stdout:write: A 61555932~7451565 2021-07-31T02:35:04.878 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 66346590~2197026 2021-07-31T02:35:04.881 INFO:tasks.workunit.client.0.smithi045.stdout:write: A 54858091~2160367 2021-07-31T02:35:04.897 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 12424435~1077280 2021-07-31T02:35:04.900 INFO:tasks.workunit.client.0.smithi045.stdout:write: A 36030347~1376339 2021-07-31T02:35:04.911 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 64734507~2070125 2021-07-31T02:35:04.914 INFO:tasks.workunit.client.0.smithi045.stdout:snap_create: snap0 2021-07-31T02:35:05.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:04 smithi154 conmon[34553]: cluster 2021-07-31T02:35:03.512346 2021-07-31T02:35:05.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:04 smithi154 conmon[34553]: +0000 mon.a (mon.0) 2481 : cluster [DBG] osdmap e1511: 8 total, 8 up, 8 in 2021-07-31T02:35:05.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:05 smithi045 conmon[33655]: cluster 2021-07-31T02:35:04.485617+0000 mgr.x (mgr.24542) 2050 : 2021-07-31T02:35:05.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:05 smithi045 conmon[33655]: cluster [DBG] pgmap v2793: 73 pgs: 73 active+clean; 1.5 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 169 MiB/s rd, 1.2 MiB/s wr, 295 op/s 2021-07-31T02:35:05.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:05 smithi045 conmon[39718]: cluster 2021-07-31T02:35:04. 2021-07-31T02:35:05.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:05 smithi045 conmon[39718]: 485617+0000 mgr.x (mgr.24542) 2050 : cluster [DBG] pgmap v2793: 73 pgs: 73 active+clean; 1.5 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 169 MiB/s rd, 1.2 MiB/s wr, 295 op/s 2021-07-31T02:35:06.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:05 smithi154 conmon[34553]: cluster 2021-07-31T02:35:04.485617+0000 mgr.x (mgr.24542) 2050 : cluster [DBG] pgmap v2793: 73 pgs: 73 active+clean; 1.5 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 169 MiB/s rd, 1.2 MiB/s wr, 295 op/s 2021-07-31T02:35:06.539 INFO:tasks.workunit.client.0.smithi045.stdout:write: B 12409402~5561028 2021-07-31T02:35:06.546 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 26660422~5796752 2021-07-31T02:35:06.546 INFO:tasks.workunit.client.0.smithi045.stdout:write: B 13714811~4728894 2021-07-31T02:35:06.576 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 55439083~5661827 2021-07-31T02:35:06.576 INFO:tasks.workunit.client.0.smithi045.stdout:write: B 18567161~5950810 2021-07-31T02:35:06.579 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 21002425~375742 2021-07-31T02:35:06.617 INFO:tasks.workunit.client.0.smithi045.stdout:write: B 43635642~2460894 2021-07-31T02:35:06.618 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 29606397~5862122 2021-07-31T02:35:06.619 INFO:tasks.workunit.client.0.smithi045.stdout:snap_create: snap1 2021-07-31T02:35:07.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:06 smithi154 conmon[34553]: cluster 2021-07-31T02:35:05.529235+0000 mon.a (mon.0) 2482 : cluster 2021-07-31T02:35:07.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:06 smithi154 conmon[34553]: [DBG] osdmap e1512: 8 total, 8 up, 8 in 2021-07-31T02:35:07.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:06 smithi045 conmon[33655]: cluster 2021-07-31T02:35:05.529235+0000 mon.a (mon.0) 2482 : cluster 2021-07-31T02:35:07.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:06 smithi045 conmon[33655]: [DBG] osdmap e1512: 8 total, 8 up, 8 in 2021-07-31T02:35:07.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:06 smithi045 conmon[39718]: cluster 2021-07-31T02:35:05.529235+0000 2021-07-31T02:35:07.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:06 smithi045 conmon[39718]: mon.a (mon.0) 2482 : cluster [DBG] osdmap e1512: 8 total, 8 up, 8 in 2021-07-31T02:35:08.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:07 smithi154 conmon[34553]: cluster 2021-07-31T02:35:06.486221+0000 mgr.x (mgr.24542) 2051 : cluster [DBG] pgmap v2795: 73 pgs: 73 active+clean; 1.5 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 136 MiB/s rd, 972 KiB/s wr, 237 op/s 2021-07-31T02:35:08.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:07 smithi154 conmon[34553]: cluster 2021-07-31T02:35:06.535197+0000 mon.a (mon.0) 2483 : cluster [DBG] osdmap e1513: 8 total, 8 up, 8 in 2021-07-31T02:35:08.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:07 smithi154 conmon[34553]: cluster 2021-07-31T02:35:07.391329+0000 mon.a (mon.0) 2484 : cluster [DBG] osdmap e1514: 8 total, 8 up, 8 in 2021-07-31T02:35:08.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:07 smithi045 conmon[33655]: cluster 2021-07-31T02:35:06.486221+0000 mgr.x (mgr 2021-07-31T02:35:08.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:07 smithi045 conmon[33655]: .24542) 2051 : cluster [DBG] pgmap v2795: 73 pgs: 73 active+clean; 1.5 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 136 MiB/s rd, 972 KiB/s wr, 237 op/s 2021-07-31T02:35:08.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:07 smithi045 conmon[33655]: cluster 2021-07-31T02:35:06.535197+0000 mon.a (mon.0) 2483 : cluster [DBG] osdmap e1513: 8 total, 8 up, 8 in 2021-07-31T02:35:08.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:07 smithi045 conmon[33655]: cluster 2021-07-31T02 2021-07-31T02:35:08.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:07 smithi045 conmon[33655]: :35:07.391329+0000 mon.a (mon.0) 2484 : cluster [DBG] osdmap e1514: 8 total, 8 up, 8 in 2021-07-31T02:35:08.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:07 smithi045 conmon[39718]: cluster 2021-07-31T02:35:06.486221+0000 mgr.x (mgr 2021-07-31T02:35:08.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:07 smithi045 conmon[39718]: .24542) 2051 : cluster [DBG] pgmap v2795: 73 pgs: 73 active+clean; 1.5 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 136 MiB/s rd, 972 KiB/s wr, 237 op/s 2021-07-31T02:35:08.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:07 smithi045 conmon[39718]: cluster 2021-07-31T02:35:06.535197+0000 2021-07-31T02:35:08.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:07 smithi045 conmon[39718]: mon.a (mon.0) 2483 : cluster [DBG] osdmap e1513: 8 total, 8 up, 8 in 2021-07-31T02:35:08.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:07 smithi045 conmon[39718]: cluster 2021-07-31T02:35:07. 2021-07-31T02:35:08.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:07 smithi045 conmon[39718]: 391329+0000 mon.a (mon.0) 2484 : cluster [DBG] osdmap e1514: 8 total, 8 up, 8 in 2021-07-31T02:35:08.403 INFO:tasks.workunit.client.0.smithi045.stdout:write: C 20280746~7097405 2021-07-31T02:35:08.412 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 51237471~270268 2021-07-31T02:35:08.412 INFO:tasks.workunit.client.0.smithi045.stdout:write: C 27596459~2741256 2021-07-31T02:35:08.415 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 12207145~5432267 2021-07-31T02:35:08.415 INFO:tasks.workunit.client.0.smithi045.stdout:write: C 48751646~6466462 2021-07-31T02:35:08.426 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 42091020~6294452 2021-07-31T02:35:08.428 INFO:tasks.workunit.client.0.smithi045.stdout:write: C 30668679~4434249 2021-07-31T02:35:08.437 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 75563907~5709008 2021-07-31T02:35:08.443 INFO:tasks.workunit.client.0.smithi045.stdout:snap_create: snap2 2021-07-31T02:35:09.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:09 smithi154 conmon[34553]: cluster 2021-07-31T02:35:08.398447+0000 mon.a (mon.0) 2485 : cluster [DBG] osdmap e1515: 8 total, 8 up, 8 in 2021-07-31T02:35:09.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:09 smithi045 conmon[39718]: cluster 2021-07-31T02:35:08.398447+0000 mon.a (mon.0) 2485 : cluster 2021-07-31T02:35:09.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:09 smithi045 conmon[39718]: [DBG] osdmap e1515: 8 total, 8 up, 8 in 2021-07-31T02:35:09.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:09 smithi045 conmon[33655]: cluster 2021-07-31T02:35:08.398447+0000 mon.a (mon.0) 2485 : cluster [DBG] 2021-07-31T02:35:09.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:09 smithi045 conmon[33655]: osdmap e1515: 8 total, 8 up, 8 in 2021-07-31T02:35:10.411 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 166587159 2021-07-31T02:35:10.417 INFO:tasks.workunit.client.0.smithi045.stdout:write: D 64493856~4795329 2021-07-31T02:35:10.428 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 26679980~4476826 2021-07-31T02:35:10.428 INFO:tasks.workunit.client.0.smithi045.stdout:write: D 133280892~5799405 2021-07-31T02:35:10.437 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 82964882~1889899 2021-07-31T02:35:10.437 INFO:tasks.workunit.client.0.smithi045.stdout:write: D 50265545~339058 2021-07-31T02:35:10.438 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 148570968~7498129 2021-07-31T02:35:10.439 INFO:tasks.workunit.client.0.smithi045.stdout:write: D 111610226~1017093 2021-07-31T02:35:10.443 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 65839021~8283162 2021-07-31T02:35:10.463 INFO:tasks.workunit.client.0.smithi045.stdout:snap_create: snap3 2021-07-31T02:35:10.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:10 smithi154 conmon[34553]: cluster 2021-07-31T02:35:08.486644+0000 mgr.x (mgr.24542) 2052 : cluster [DBG] pgmap v2799: 73 pgs: 73 active+clean; 1.5 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2021-07-31T02:35:10.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:10 smithi154 conmon[34553]: cluster 2021-07-31T02:35:09.403665+0000 mon.a (mon.0) 2486 : cluster [DBG] osdmap e1516: 8 total, 8 up, 8 in 2021-07-31T02:35:10.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:10 smithi045 conmon[39718]: cluster 2021-07-31T02:35:08.486644+0000 mgr.x (mgr.24542) 2021-07-31T02:35:10.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:10 smithi045 conmon[39718]: 2052 : cluster [DBG] pgmap v2799: 73 pgs: 73 active+clean; 1.5 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2021-07-31T02:35:10.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:10 smithi045 conmon[39718]: cluster 2021-07-31T02:35:09. 2021-07-31T02:35:10.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:10 smithi045 conmon[39718]: 403665+0000 mon.a (mon.0) 2486 : cluster [DBG] osdmap e1516: 8 total, 8 up, 8 in 2021-07-31T02:35:10.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:10 smithi045 conmon[33655]: cluster 2021-07-31T02:35:08.486644 2021-07-31T02:35:10.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:10 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 2052 : cluster [DBG] pgmap v2799: 73 pgs: 73 active+clean; 1.5 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2021-07-31T02:35:10.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:10 smithi045 conmon[33655]: cluster 2021-07-31T02:35:09.403665+0000 mon.a (mon.0) 2021-07-31T02:35:10.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:10 smithi045 conmon[33655]: 2486 : cluster [DBG] osdmap e1516: 8 total, 8 up, 8 in 2021-07-31T02:35:11.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:11 smithi154 conmon[34553]: cluster 2021-07-31T02:35:10.405086+0000 mon.a (mon.0) 2487 : cluster [DBG] osdmap e1517: 8 total, 8 up, 8 in 2021-07-31T02:35:11.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:11 smithi154 conmon[34553]: cluster 2021-07-31T02:35:10.487157+0000 mgr.x (mgr.24542) 2053 : cluster [DBG] pgmap v2802: 73 pgs: 73 active+clean; 1.7 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 132 MiB/s rd, 25 MiB/s wr, 103 op/s 2021-07-31T02:35:11.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:11 smithi045 conmon[33655]: cluster 2021-07-31T02:35:10.405086+0000 mon.a (mon.0) 2487 2021-07-31T02:35:11.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:11 smithi045 conmon[33655]: : cluster [DBG] osdmap e1517: 8 total, 8 up, 8 in 2021-07-31T02:35:11.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:11 smithi045 conmon[33655]: cluster 2021-07-31T02:35:10.487157+0000 mgr.x (mgr.24542) 2053 : cluster [DBG] pgmap v2802: 73 pgs: 73 active+clean; 1.7 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 132 MiB/s rd, 25 MiB/s wr, 103 op/s 2021-07-31T02:35:11.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:11 smithi045 conmon[39718]: cluster 2021-07-31T02:35:10.405086+0000 mon.a (mon.0 2021-07-31T02:35:11.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:11 smithi045 conmon[39718]: ) 2487 : cluster [DBG] osdmap e1517: 8 total, 8 up, 8 in 2021-07-31T02:35:11.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:11 smithi045 conmon[39718]: cluster 2021-07-31T02:35:10.487157+0000 mgr.x (mgr.24542) 2053 : 2021-07-31T02:35:11.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:11 smithi045 conmon[39718]: cluster [DBG] pgmap v2802: 73 pgs: 73 active+clean; 1.7 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 132 MiB/s rd, 25 MiB/s wr, 103 op/s 2021-07-31T02:35:12.420 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 153630754 2021-07-31T02:35:12.438 INFO:tasks.workunit.client.0.smithi045.stdout:migration_prepare 2021-07-31T02:35:12.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:12 smithi154 conmon[34553]: cluster 2021-07-31T02:35:11.415441+0000 mon.a (mon.0) 2488 : cluster 2021-07-31T02:35:12.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:12 smithi154 conmon[34553]: [DBG] osdmap e1518: 8 total, 8 up, 8 in 2021-07-31T02:35:12.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:12 smithi045 conmon[39718]: cluster 2021-07-31T02:35:11.415441+0000 2021-07-31T02:35:12.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:12 smithi045 conmon[39718]: mon.a (mon.0) 2488 : cluster [DBG] osdmap e1518: 8 total, 8 up, 8 in 2021-07-31T02:35:12.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:12 smithi045 conmon[33655]: cluster 2021-07-31T02:35:11.415441+0000 mon.a (mon. 2021-07-31T02:35:12.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:12 smithi045 conmon[33655]: 0) 2488 : cluster [DBG] osdmap e1518: 8 total, 8 up, 8 in 2021-07-31T02:35:13.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:13 smithi154 conmon[34553]: cluster 2021-07-31T02:35:12.416495+0000 mon.a (mon.0) 2489 : cluster [DBG] osdmap e1519: 8 total, 8 up, 8 in 2021-07-31T02:35:13.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:13 smithi154 conmon[34553]: 2021-07-31T02:35:13.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:13 smithi154 conmon[34553]: cluster 2021-07-31T02:35:12.487617+0000 mgr.x (mgr.24542) 2054 : cluster [DBG] pgmap v2805: 73 pgs: 73 active+clean; 1.7 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 130 MiB/s rd, 25 MiB/s wr, 102 op/s 2021-07-31T02:35:13.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:13 smithi045 conmon[39718]: cluster 2021-07-31T02:35:12.416495+0000 mon.a (mon 2021-07-31T02:35:13.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:13 smithi045 conmon[39718]: .0) 2489 : cluster [DBG] osdmap e1519: 8 total, 8 up, 8 in 2021-07-31T02:35:13.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:13 smithi045 conmon[39718]: cluster 2021-07-31T02:35:12.487617+0000 2021-07-31T02:35:13.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:13 smithi045 conmon[39718]: mgr.x (mgr.24542) 2054 : cluster [DBG] pgmap v2805: 73 pgs: 73 active+clean; 1.7 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 130 MiB/s rd, 25 MiB/s wr, 102 op/s 2021-07-31T02:35:13.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:13 smithi045 conmon[33655]: cluster 2021-07-31T02:35:12.416495+0000 mon.a (mon.0) 2489 2021-07-31T02:35:13.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:13 smithi045 conmon[33655]: : cluster [DBG] osdmap e1519: 8 total, 8 up, 8 in 2021-07-31T02:35:13.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:13 smithi045 conmon[33655]: cluster 2021-07-31T02:35:12.487617+0000 mgr.x (mgr.24542) 2054 : cluster [DBG] pgmap v2805: 73 pgs: 73 active+clean; 1.7 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 130 MiB/s rd, 25 MiB/s wr, 102 op/s 2021-07-31T02:35:14.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:14 smithi154 conmon[34553]: cluster 2021-07-31T02:35:13.427420+0000 mon.a (mon.0) 2021-07-31T02:35:14.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:14 smithi154 conmon[34553]: 2490 : cluster [DBG] osdmap e1520: 8 total, 8 up, 8 in 2021-07-31T02:35:14.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:14 smithi045 conmon[33655]: cluster 2021-07-31T02:35:13.427420+0000 mon.a (mon.0) 2490 2021-07-31T02:35:14.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:14 smithi045 conmon[33655]: : cluster [DBG] osdmap e1520: 8 total, 8 up, 8 in 2021-07-31T02:35:14.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:14 smithi045 conmon[39718]: cluster 2021-07-31T02:35:13.427420+0000 mon.a (mon.0) 2490 2021-07-31T02:35:14.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:14 smithi045 conmon[39718]: : cluster [DBG] osdmap e1520: 8 total, 8 up, 8 in 2021-07-31T02:35:15.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:15 smithi154 conmon[34553]: cluster 2021-07-31T02:35:14.429586+0000 mon.a (mon.0) 2491 : cluster [DBG] osdmap e1521: 8 total, 8 up, 8 in 2021-07-31T02:35:15.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:15 smithi154 conmon[34553]: cluster 2021-07-31T02:35:14.488223+0000 mgr.x (mgr.24542) 2055 : cluster [DBG] pgmap v2808: 73 pgs: 73 active+clean; 1.7 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 19 KiB/s rd, 5.7 MiB/s wr, 33 op/s 2021-07-31T02:35:15.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:15 smithi045 conmon[39718]: cluster 2021-07-31T02:35:14. 2021-07-31T02:35:15.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:15 smithi045 conmon[39718]: 429586+0000 mon.a (mon.0) 2491 : cluster [DBG] osdmap e1521: 8 total, 8 up, 8 in 2021-07-31T02:35:15.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:15 smithi045 conmon[39718]: cluster 2021-07-31T02:35:15.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:15 smithi045 conmon[39718]: 2021-07-31T02:35:14.488223+0000 mgr.x (mgr.24542) 2055 : cluster 2021-07-31T02:35:15.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:15 smithi045 conmon[39718]: [DBG] pgmap v2808: 73 pgs: 73 active+clean; 1.7 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 19 KiB/s rd, 5.7 MiB/s wr, 33 op/s 2021-07-31T02:35:15.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:15 smithi045 conmon[33655]: cluster 2021-07-31T02:35:14.429586+0000 mon.a (mon.0) 2491 : cluster 2021-07-31T02:35:15.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:15 smithi045 conmon[33655]: [DBG] osdmap e1521: 8 total, 8 up, 8 in 2021-07-31T02:35:15.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:15 smithi045 conmon[33655]: cluster 2021-07-31T02:35:14.488223+0000 mgr.x (mgr.24542) 2055 : cluster [DBG] pgmap v2808: 73 pgs: 73 active+clean; 1.7 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 19 KiB/s rd, 5.7 MiB/s wr, 33 op/s 2021-07-31T02:35:16.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:16 smithi154 conmon[34553]: cluster 2021-07-31T02:35:15.440378+0000 mon.a (mon.0 2021-07-31T02:35:16.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:16 smithi154 conmon[34553]: ) 2492 : cluster [DBG] osdmap e1522: 8 total, 8 up, 8 in 2021-07-31T02:35:16.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:16 smithi045 conmon[39718]: cluster 2021-07-31T02:35:15.440378 2021-07-31T02:35:16.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:16 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2492 : cluster [DBG] osdmap e1522: 8 total, 8 up, 8 in 2021-07-31T02:35:16.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:16 smithi045 conmon[33655]: cluster 2021-07-31T02:35:15.440378+0000 mon.a (mon.0) 2492 : cluster [DBG] osdmap e1522: 8 total, 8 up, 8 in 2021-07-31T02:35:17.113 INFO:tasks.workunit.client.0.smithi045.stdout:migration_execute 2021-07-31T02:35:17.113 INFO:tasks.workunit.client.0.smithi045.stdout:write: a 11833918~3717996 2021-07-31T02:35:17.153 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 20497778~6454347 2021-07-31T02:35:17.238 INFO:tasks.workunit.client.0.smithi045.stdout:write: a 11075346~2576406 2021-07-31T02:35:17.268 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 15880441~1943830 2021-07-31T02:35:17.272 INFO:tasks.workunit.client.0.smithi045.stdout:write: a 137080125~1474439 2021-07-31T02:35:17.308 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 72456658~7077091 2021-07-31T02:35:17.314 INFO:tasks.workunit.client.0.smithi045.stdout:write: a 7081978~6467838 2021-07-31T02:35:17.341 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 149396721~4234033 2021-07-31T02:35:17.342 INFO:tasks.workunit.client.0.smithi045.stdout:snap_create: user0 2021-07-31T02:35:17.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:17 smithi154 conmon[34553]: cluster 2021-07-31T02:35:16.441996+0000 mon.a (mon.0) 2493 : cluster [DBG] osdmap e1523: 8 total, 8 up, 8 in 2021-07-31T02:35:17.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:17 smithi154 conmon[34553]: cluster 2021-07-31T02:35 2021-07-31T02:35:17.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:17 smithi154 conmon[34553]: :16.488771+0000 mgr.x (mgr.24542) 2056 : cluster [DBG] pgmap v2811: 73 pgs: 73 active+clean; 1.7 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 19 KiB/s rd, 5.7 MiB/s wr, 33 op/s 2021-07-31T02:35:17.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:17 smithi045 conmon[39718]: cluster 2021-07-31T02:35:16.441996 2021-07-31T02:35:17.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:17 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2493 : cluster [DBG] osdmap e1523: 8 total, 8 up, 8 in 2021-07-31T02:35:17.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:17 smithi045 conmon[39718]: cluster 2021-07-31T02:35:17.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:17 smithi045 conmon[39718]: 2021-07-31T02:35:16.488771+0000 2021-07-31T02:35:17.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:17 smithi045 conmon[39718]: mgr.x (mgr.24542) 2056 : 2021-07-31T02:35:17.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:17 smithi045 conmon[39718]: cluster [DBG] pgmap v2811: 73 pgs: 73 active+clean; 1.7 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 19 KiB/s rd, 5.7 MiB/s wr, 33 op/s 2021-07-31T02:35:17.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:17 smithi045 conmon[33655]: cluster 2021-07-31T02:35:16.441996+0000 mon.a (mon.0) 2021-07-31T02:35:17.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:17 smithi045 conmon[33655]: 2493 : cluster [DBG] osdmap e1523: 8 total, 8 up, 8 in 2021-07-31T02:35:17.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:17 smithi045 conmon[33655]: cluster 2021-07-31T02:35:16.488771+0000 mgr.x (mgr.24542) 2056 : cluster [DBG] pgmap v2811: 73 pgs: 73 active+clean; 1.7 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 19 KiB/s rd, 5.7 MiB/s wr, 33 op/s 2021-07-31T02:35:18.465 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 165588905 2021-07-31T02:35:18.479 INFO:tasks.workunit.client.0.smithi045.stdout:write: b 132716988~434219 2021-07-31T02:35:18.485 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 30994435~3280837 2021-07-31T02:35:18.485 INFO:tasks.workunit.client.0.smithi045.stdout:write: b 86965262~2081761 2021-07-31T02:35:18.487 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 53774914~5552909 2021-07-31T02:35:18.488 INFO:tasks.workunit.client.0.smithi045.stdout:write: b 39690719~208676 2021-07-31T02:35:18.488 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 154294272~4386215 2021-07-31T02:35:18.488 INFO:tasks.workunit.client.0.smithi045.stdout:write: b 101412135~8083273 2021-07-31T02:35:18.499 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 116234153~3089226 2021-07-31T02:35:18.499 INFO:tasks.workunit.client.0.smithi045.stdout:snap_create: user1 2021-07-31T02:35:18.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:18 smithi154 conmon[34553]: cluster 2021-07-31T02:35:17.453302+0000 mon.a (mon.0) 2494 : cluster [DBG] osdmap e1524: 8 total, 8 up, 8 in 2021-07-31T02:35:18.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:18 smithi154 conmon[34553]: 2021-07-31T02:35:18.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:18 smithi045 conmon[33655]: cluster 2021-07-31T02:35:17.453302+0000 mon.a (mon 2021-07-31T02:35:18.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:18 smithi045 conmon[33655]: .0) 2494 : cluster [DBG] osdmap e1524: 8 total, 8 up, 8 in 2021-07-31T02:35:18.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:18 smithi045 conmon[39718]: cluster 2021-07-31T02:35:17.453302+0000 mon.a ( 2021-07-31T02:35:18.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:18 smithi045 conmon[39718]: mon.0) 2494 : cluster [DBG] osdmap e1524: 8 total, 8 up, 8 in 2021-07-31T02:35:19.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:19 smithi154 conmon[34553]: cluster 2021-07-31T02:35:18.459987+0000 mon.a (mon.0) 2495 : cluster [DBG] osdmap e1525: 8 total, 8 up, 8 in 2021-07-31T02:35:19.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:19 smithi154 conmon[34553]: cluster 2021-07-31T02:35:18.489289+0000 mgr.x (mgr.24542) 2057 : cluster [DBG] pgmap v2814: 73 pgs: 73 active+clean; 1.7 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail 2021-07-31T02:35:19.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:19 smithi045 conmon[33655]: cluster 2021-07-31T02:35:18.459987+0000 mon.a (mon. 2021-07-31T02:35:19.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:19 smithi045 conmon[33655]: 0) 2495 : cluster [DBG] osdmap e1525: 8 total, 8 up, 8 in 2021-07-31T02:35:19.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:19 smithi045 conmon[33655]: cluster 2021-07-31T02:35:18.489289+0000 mgr.x (mgr.24542) 2057 : 2021-07-31T02:35:19.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:19 smithi045 conmon[33655]: cluster [DBG] pgmap v2814: 73 pgs: 73 active+clean; 1.7 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail 2021-07-31T02:35:19.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:19 smithi045 conmon[39718]: cluster 2021-07-31T02:35:18.459987+0000 mon.a (mon 2021-07-31T02:35:19.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:19 smithi045 conmon[39718]: .0) 2495 : cluster [DBG] osdmap e1525: 8 total, 8 up, 8 in 2021-07-31T02:35:19.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:19 smithi045 conmon[39718]: cluster 2021-07-31T02:35:18.489289+0000 mgr.x (mgr.24542) 2057 : 2021-07-31T02:35:19.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:19 smithi045 conmon[39718]: cluster [DBG] pgmap v2814: 73 pgs: 73 active+clean; 1.7 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail 2021-07-31T02:35:20.475 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 251915487 2021-07-31T02:35:20.488 INFO:tasks.workunit.client.0.smithi045.stdout:write: c 170455703~2120597 2021-07-31T02:35:20.495 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 95355230~4287680 2021-07-31T02:35:20.496 INFO:tasks.workunit.client.0.smithi045.stdout:write: c 60410184~2719515 2021-07-31T02:35:20.498 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 118239257~7980474 2021-07-31T02:35:20.499 INFO:tasks.workunit.client.0.smithi045.stdout:write: c 78037137~6459976 2021-07-31T02:35:20.506 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 245824475~3354801 2021-07-31T02:35:20.506 INFO:tasks.workunit.client.0.smithi045.stdout:write: c 136463266~5470731 2021-07-31T02:35:20.520 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 184397842~1568597 2021-07-31T02:35:20.521 INFO:tasks.workunit.client.0.smithi045.stdout:snap_create: user2 2021-07-31T02:35:20.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:20 smithi154 conmon[34553]: cluster 2021-07-31T02:35:19.466200+0000 mon.a (mon 2021-07-31T02:35:20.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:20 smithi154 conmon[34553]: .0) 2496 : cluster [DBG] osdmap e1526: 8 total, 8 up, 8 in 2021-07-31T02:35:20.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:20 smithi045 conmon[33655]: cluster 2021-07-31T02:35:19.466200 2021-07-31T02:35:20.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:20 smithi045 conmon[33655]: +0000 mon.a (mon.0) 2496 : cluster [DBG] osdmap e1526: 8 total, 8 up, 8 in 2021-07-31T02:35:20.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:20 smithi045 conmon[39718]: cluster 2021-07-31T02:35:19.466200+0000 2021-07-31T02:35:20.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:20 smithi045 conmon[39718]: mon.a (mon.0) 2496 : cluster [DBG] osdmap e1526: 8 total, 8 up, 8 in 2021-07-31T02:35:21.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:21 smithi154 conmon[34553]: cluster 2021-07-31T02:35:20.467421+0000 mon.a (mon.0) 2497 : cluster [DBG] osdmap e1527: 8 total, 8 up, 8 in 2021-07-31T02:35:21.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:21 smithi154 conmon[34553]: 2021-07-31T02:35:21.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:21 smithi154 conmon[34553]: cluster 2021-07-31T02:35:20.489745+0000 mgr.x (mgr.24542) 2058 : cluster [DBG] pgmap v2817: 73 pgs: 73 active+clean; 1.8 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 74 MiB/s rd, 24 MiB/s wr, 247 op/s 2021-07-31T02:35:21.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:21 smithi045 conmon[39718]: cluster 2021-07-31T02:35:20.467421+0000 mon.a ( 2021-07-31T02:35:21.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:21 smithi045 conmon[39718]: mon.0) 2497 : cluster [DBG] osdmap e1527: 8 total, 8 up, 8 in 2021-07-31T02:35:21.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:21 smithi045 conmon[39718]: cluster 2021-07-31T02:35:20.489745+0000 2021-07-31T02:35:21.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:21 smithi045 conmon[39718]: mgr.x (mgr.24542) 2058 : cluster [DBG] pgmap v2817: 73 pgs: 73 active+clean; 1.8 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 74 MiB/s rd, 24 MiB/s wr, 247 op/s 2021-07-31T02:35:21.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:21 smithi045 conmon[33655]: cluster 2021-07-31T02:35:20.467421+0000 mon.a (mon. 2021-07-31T02:35:21.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:21 smithi045 conmon[33655]: 0) 2497 : cluster [DBG] osdmap e1527: 8 total, 8 up, 8 in 2021-07-31T02:35:21.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:21 smithi045 conmon[33655]: cluster 2021-07-31T02:35:20.489745+0000 mgr.x (mgr.24542) 2058 : cluster [DBG] pgmap v2817: 73 pgs: 73 active+clean; 1.8 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 74 MiB/s rd, 24 MiB/s wr, 247 op/s 2021-07-31T02:35:22.492 INFO:tasks.workunit.client.0.smithi045.stdout:write: d 84581423~3650358 2021-07-31T02:35:22.505 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 48435005~1726599 2021-07-31T02:35:22.506 INFO:tasks.workunit.client.0.smithi045.stdout:write: d 6916815~2961995 2021-07-31T02:35:22.512 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 228129982~2746819 2021-07-31T02:35:22.512 INFO:tasks.workunit.client.0.smithi045.stdout:write: d 119131193~6868727 2021-07-31T02:35:22.523 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 137303378~4984040 2021-07-31T02:35:22.524 INFO:tasks.workunit.client.0.smithi045.stdout:write: d 76215055~3973875 2021-07-31T02:35:22.530 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 199080908~6749729 2021-07-31T02:35:22.539 INFO:tasks.workunit.client.0.smithi045.stdout:snap_create: user3 2021-07-31T02:35:22.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:22 smithi154 conmon[34553]: cluster 2021-07-31T02:35:21.481006+0000 mon.a (mon.0) 2498 : cluster [DBG] osdmap e1528: 8 total, 8 up, 8 in 2021-07-31T02:35:22.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:22 smithi045 conmon[39718]: cluster 2021-07-31T02:35:21.481006+0000 mon.a (mon 2021-07-31T02:35:22.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:22 smithi045 conmon[39718]: .0) 2498 : cluster [DBG] osdmap e1528: 8 total, 8 up, 8 in 2021-07-31T02:35:22.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:22 smithi045 conmon[33655]: cluster 2021-07-31T02:35:21.481006+0000 mon.a (mon.0) 2498 : cluster [DBG] osdmap e1528: 8 total, 8 up, 8 in 2021-07-31T02:35:23.357 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:35:23.637 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:35:23.637 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 50m ago 51m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:35:23.638 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 47m ago 50m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:35:23.638 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 47m ago 54m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:35:23.638 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 50m ago 56m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:35:23.638 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 50m ago 56m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:35:23.639 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 47m ago 54m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:35:23.639 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 50m ago 54m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:35:23.639 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 50m ago 51m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:35:23.640 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 47m ago 51m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:35:23.640 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 50m ago 54m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:35:23.640 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 50m ago 53m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:35:23.640 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 50m ago 53m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:35:23.641 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 50m ago 53m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:35:23.641 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 47m ago 52m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:35:23.641 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 47m ago 52m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:35:23.641 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 47m ago 52m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:35:23.642 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 47m ago 51m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:35:23.642 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 47m ago 51m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:35:23.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:23 smithi154 conmon[34553]: cluster 2021-07-31T02:35:22.486796+0000 mon.a (mon.0) 2499 : cluster [DBG] 2021-07-31T02:35:23.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:23 smithi154 conmon[34553]: osdmap e1529: 8 total, 8 up, 8 in 2021-07-31T02:35:23.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:23 smithi154 conmon[34553]: cluster 2021-07-31T02:35:22.490179+0000 mgr.x (mgr.24542) 2059 : cluster [DBG] pgmap v2820: 73 pgs: 73 active+clean; 1.8 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 74 MiB/s rd, 24 MiB/s wr, 247 op/s 2021-07-31T02:35:23.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:23 smithi154 conmon[34553]: audit 2021-07-31T02:35:23.345263+0000 mgr.x (mgr.24542) 2060 : audit [DBG] from='client.17496 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:35:23.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:23 smithi045 conmon[39718]: cluster 2021-07-31T02:35:22. 2021-07-31T02:35:23.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:23 smithi045 conmon[39718]: 486796+0000 mon.a (mon.0) 2499 : cluster [DBG] osdmap e1529: 8 total, 8 up, 8 in 2021-07-31T02:35:23.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:23 smithi045 conmon[39718]: cluster 2021-07-31T02:35:22.490179+0000 mgr.x (mgr.24542) 2059 : cluster [DBG] pgmap v2820: 73 pgs: 73 active+clean; 1.8 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 74 MiB/s rd, 24 MiB/s wr, 247 op/s 2021-07-31T02:35:23.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:23 smithi045 conmon[39718]: audit 2021-07-31T02:35:23.345263+0000 mgr.x (mgr.24542) 2060 : audit [DBG] from='client.17496 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:35:23.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:23 smithi045 conmon[33655]: cluster 2021-07-31T02:35 2021-07-31T02:35:23.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:23 smithi045 conmon[33655]: :22.486796+0000 mon.a (mon.0) 2499 : cluster [DBG] osdmap e1529: 8 total, 8 up, 8 in 2021-07-31T02:35:23.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:23 smithi045 conmon[33655]: cluster 2021-07-31T 2021-07-31T02:35:23.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:23 smithi045 conmon[33655]: 02:35:22.490179+0000 mgr.x (mgr.24542) 2021-07-31T02:35:23.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:23 smithi045 conmon[33655]: 2059 : cluster [DBG] pgmap v2820: 73 pgs: 73 active+clean; 1.8 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 74 MiB/s rd, 24 MiB/s wr, 247 op/s 2021-07-31T02:35:23.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:23 smithi045 conmon[33655]: audit 2021-07-31T02:35:23. 2021-07-31T02:35:23.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:23 smithi045 conmon[33655]: 345263+0000 mgr.x (mgr.24542) 2060 : audit 2021-07-31T02:35:23.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:23 smithi045 conmon[33655]: [DBG] from='client.17496 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:35:23.972 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:35:23.972 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:35:23.972 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:35:23.972 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:35:23.973 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:35:23.973 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:35:23.973 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:35:23.973 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:35:23.974 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:35:23.974 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:35:23.974 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:35:23.974 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:35:23.975 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:35:23.975 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:35:23.975 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:35:23.975 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:35:23.976 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:35:24.503 INFO:tasks.workunit.client.0.smithi045.stdout:resize: 288775651 2021-07-31T02:35:24.524 INFO:tasks.workunit.client.0.smithi045.stdout:write: 1 28877565~7219391 2021-07-31T02:35:24.533 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 30682412~3609695 2021-07-31T02:35:24.562 INFO:tasks.workunit.client.0.smithi045.stdout:write: 2 57755130~7219391 2021-07-31T02:35:24.567 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 59559977~3609695 2021-07-31T02:35:24.596 INFO:tasks.workunit.client.0.smithi045.stdout:write: 3 86632695~7219391 2021-07-31T02:35:24.601 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 88437542~3609695 2021-07-31T02:35:24.625 INFO:tasks.workunit.client.0.smithi045.stdout:write: 4 115510260~7219391 2021-07-31T02:35:24.629 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 117315107~3609695 2021-07-31T02:35:24.658 INFO:tasks.workunit.client.0.smithi045.stdout:write: 5 144387825~7219391 2021-07-31T02:35:24.664 INFO:tasks.workunit.client.0.smithi045.stdout:discard: 146192672~3609695 2021-07-31T02:35:24.723 INFO:tasks.workunit.client.0.smithi045.stdout:migration_commit 2021-07-31T02:35:24.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:24 smithi154 conmon[34553]: cluster 2021-07-31T02:35:23.491417+0000 mon.a (mon.0) 2500 : cluster [DBG] osdmap e1530: 8 total, 8 up, 8 in 2021-07-31T02:35:24.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:24 smithi154 conmon[34553]: audit 2021-07-31T02:35:23.629654 2021-07-31T02:35:24.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:24 smithi154 conmon[34553]: +0000 mgr.x (mgr.24542) 2061 : audit [DBG] from='client.17502 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:35:24.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:24 smithi154 conmon[34553]: audit 2021-07-31T02:35:23.972041+0000 mon.a (mon.0) 2501 : audit [DBG] from='client.? 172.21.15.45:0/1461875389' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:35:24.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:24 smithi045 conmon[33655]: cluster 2021-07-31T02:35:23.491417+0000 mon.a (mon.0) 2021-07-31T02:35:24.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:24 smithi045 conmon[33655]: 2500 : cluster [DBG] osdmap e1530: 8 total, 8 up, 8 in 2021-07-31T02:35:24.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:24 smithi045 conmon[33655]: audit 2021-07-31T02:35:23.629654+0000 mgr.x (mgr.24542) 2061 : audit [DBG] from='client.17502 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:35:24.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:24 smithi045 conmon[33655]: audit 2021-07-31T02:35:23 2021-07-31T02:35:24.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:24 smithi045 conmon[33655]: .972041+0000 mon.a (mon.0) 2501 : audit [DBG] from='client.? 172.21.15.45:0/1461875389' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:35:24.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:24 smithi045 conmon[39718]: cluster 2021-07-31T02:35:23.491417+0000 mon.a (mon.0) 2500 : cluster [DBG] osdmap e1530: 8 total, 8 up, 8 in 2021-07-31T02:35:24.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:24 smithi045 conmon[39718]: audit 2021-07-31T02:35:23.629654+0000 mgr.x (mgr.24542) 2061 : audit [DBG] 2021-07-31T02:35:24.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:24 smithi045 conmon[39718]: from='client.17502 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:35:24.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:24 smithi045 conmon[39718]: audit 2021-07-31T02:35 2021-07-31T02:35:24.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:24 smithi045 conmon[39718]: :23.972041+0000 mon.a (mon.0) 2021-07-31T02:35:24.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:24 smithi045 conmon[39718]: 2501 : audit [DBG] from='client.? 172.21.15.45:0/1461875389' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:35:25.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:25 smithi154 conmon[34553]: cluster 2021-07-31T02:35:24.490827+0000 mgr.x (mgr.24542) 2062 : cluster [DBG] pgmap v2822: 73 pgs: 73 active+clean; 1.9 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 36 KiB/s rd, 16 MiB/s wr, 61 op/s 2021-07-31T02:35:25.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:25 smithi154 conmon[34553]: cluster 2021-07-31T02:35:24.498351+0000 mon.a (mon.0) 2502 : cluster [DBG] osdmap e1531: 8 total, 8 up, 8 in 2021-07-31T02:35:25.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:25 smithi045 conmon[33655]: cluster 2021-07-31T02:35:24. 2021-07-31T02:35:25.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:25 smithi045 conmon[33655]: 490827+0000 mgr.x (mgr.24542) 2062 : cluster [DBG] pgmap v2822: 73 pgs: 73 active+clean; 1.9 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 36 KiB/s rd, 16 MiB/s wr, 61 op/s 2021-07-31T02:35:25.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:25 smithi045 conmon[33655]: cluster 2021-07-31T02:35 2021-07-31T02:35:25.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:25 smithi045 conmon[33655]: :24.498351+0000 mon.a (mon.0) 2502 : cluster [DBG] osdmap e1531: 8 total, 8 up, 8 in 2021-07-31T02:35:25.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:25 smithi045 conmon[39718]: cluster 2021-07-31T02:35:24.490827+0000 mgr.x (mgr 2021-07-31T02:35:25.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:25 smithi045 conmon[39718]: .24542) 2062 : cluster [DBG] pgmap v2822: 73 pgs: 73 active+clean; 1.9 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 36 KiB/s rd, 16 MiB/s wr, 61 op/s 2021-07-31T02:35:25.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:25 smithi045 conmon[39718]: cluster 2021-07-31T02:35:24.498351+0000 mon.a (mon.0) 2021-07-31T02:35:25.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:25 smithi045 conmon[39718]: 2502 : cluster [DBG] osdmap e1531: 8 total, 8 up, 8 in 2021-07-31T02:35:26.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:26 smithi045 conmon[39718]: cluster 2021-07-31T02:35:25.503975+0000 mon.a (mon.0 2021-07-31T02:35:26.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:26 smithi045 conmon[39718]: ) 2503 : cluster [DBG] osdmap e1532: 8 total, 8 up, 8 in 2021-07-31T02:35:26.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:26 smithi045 conmon[33655]: cluster 2021-07-31T02:35:25.503975+0000 mon.a (mon.0) 2021-07-31T02:35:26.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:26 smithi045 conmon[33655]: 2503 : cluster [DBG] osdmap e1532: 8 total, 8 up, 8 in 2021-07-31T02:35:27.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:26 smithi154 conmon[34553]: cluster 2021-07-31T02:35:25.503975+0000 mon.a (mon.0) 2503 : cluster [DBG] 2021-07-31T02:35:27.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:26 smithi154 conmon[34553]: osdmap e1532: 8 total, 8 up, 8 in 2021-07-31T02:35:27.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:27 smithi045 conmon[33655]: cluster 2021-07-31T02:35:26.491326+0000 mgr.x (mgr.24542) 2063 : cluster [DBG] pgmap v2825: 73 pgs: 73 active+clean; 1.9 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 36 KiB/s rd, 16 MiB/s wr, 61 op/s 2021-07-31T02:35:27.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:27 smithi045 conmon[33655]: cluster 2021-07-31T02:35:26.510404+0000 mon.a (mon.0) 2504 : cluster [DBG] osdmap e1533: 8 total, 8 up, 8 in 2021-07-31T02:35:27.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:27 smithi045 conmon[39718]: cluster 2021-07-31T02:35:27.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:27 smithi045 conmon[39718]: 2021-07-31T02:35:27.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:27 smithi045 conmon[39718]: 2021 2021-07-31T02:35:27.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:27 smithi045 conmon[39718]: - 2021-07-31T02:35:27.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:27 smithi045 conmon[39718]: 07 2021-07-31T02:35:27.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:27 smithi045 conmon[39718]: - 2021-07-31T02:35:27.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:27 smithi045 conmon[39718]: 31 2021-07-31T02:35:27.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:27 smithi045 conmon[39718]: T 2021-07-31T02:35:27.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:27 smithi045 conmon[39718]: 02 2021-07-31T02:35:27.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:27 smithi045 conmon[39718]: : 2021-07-31T02:35:27.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:27 smithi045 conmon[39718]: 35 2021-07-31T02:35:27.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:27 smithi045 conmon[39718]: : 2021-07-31T02:35:27.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:27 smithi045 conmon[39718]: 26 2021-07-31T02:35:27.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:27 smithi045 conmon[39718]: . 2021-07-31T02:35:27.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:27 smithi045 conmon[39718]: 491326 2021-07-31T02:35:27.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:27 smithi045 conmon[39718]: +0000 2021-07-31T02:35:27.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:27 smithi045 conmon[39718]: 2021-07-31T02:35:27.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:27 smithi045 conmon[39718]: mgr.x 2021-07-31T02:35:27.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:27 smithi045 conmon[39718]: ( 2021-07-31T02:35:27.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:27 smithi045 conmon[39718]: mgr 2021-07-31T02:35:27.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:27 smithi045 conmon[39718]: . 2021-07-31T02:35:27.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:27 smithi045 conmon[39718]: 24542 2021-07-31T02:35:27.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:27 smithi045 conmon[39718]: ) 2021-07-31T02:35:27.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:27 smithi045 conmon[39718]: 2063 2021-07-31T02:35:27.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:27 smithi045 conmon[39718]: : 2021-07-31T02:35:27.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:27 smithi045 conmon[39718]: cluster 2021-07-31T02:35:27.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:27 smithi045 conmon[39718]: 2021-07-31T02:35:27.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:27 smithi045 conmon[39718]: [DBG] 2021-07-31T02:35:27.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:27 smithi045 conmon[39718]: 2021-07-31T02:35:27.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:27 smithi045 conmon[39718]: pgmap v2825: 73 pgs: 73 active+clean; 1.9 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 36 KiB/s rd, 16 MiB/s wr, 61 op/s 2021-07-31T02:35:27.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:27 smithi045 conmon[39718]: 2021-07-31T02:35:27.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:27 smithi045 conmon[39718]: cluster 2021-07-31T02:35:27.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:27 smithi045 conmon[39718]: 2021-07-31T02:35:27.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:27 smithi045 conmon[39718]: 2021 2021-07-31T02:35:27.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:27 smithi045 conmon[39718]: - 2021-07-31T02:35:27.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:27 smithi045 conmon[39718]: 07 2021-07-31T02:35:27.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:27 smithi045 conmon[39718]: -31T02:35:26. 2021-07-31T02:35:27.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:27 smithi045 conmon[39718]: 510404+0000 mon.a (mon.0) 2504 : cluster [DBG] osdmap e1533: 8 total, 8 up, 8 in 2021-07-31T02:35:28.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:27 smithi154 conmon[34553]: cluster 2021-07-31T02:35:26.491326+0000 mgr.x ( 2021-07-31T02:35:28.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:27 smithi154 conmon[34553]: mgr.24542) 2063 : cluster [DBG] pgmap v2825: 73 pgs: 73 active+clean; 1.9 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 36 KiB/s rd, 16 MiB/s wr, 61 op/s 2021-07-31T02:35:28.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:27 smithi154 conmon[34553]: cluster 2021-07-31T02:35:26.510404+0000 mon.a (mon.0) 2504 : cluster [DBG] 2021-07-31T02:35:28.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:27 smithi154 conmon[34553]: osdmap e1533: 8 total, 8 up, 8 in 2021-07-31T02:35:28.003 INFO:journalctl@ceph.osd.6.smithi154.stdout:Jul 31 02:35:27 smithi154 conmon[45576]: debug 2021-07-31T02:35:28.004 INFO:journalctl@ceph.osd.6.smithi154.stdout:Jul 31 02:35:27 smithi154 conmon[45576]: 2021-07-31T02:35:27.528+0000 7f47c38aa700 -1 osd.6 pg_epoch: 1534 pg[75.1d( v 1531'178 (0'0,1531'178] local-lis/les=1308/1309 n=48 ec=1308/1308 lis/c=1308/1308 les/c/f=1309/1309/0 sis=1308) [6,3,0] r=0 lpr=1308 crt=1531'178 lcod 1531'177 mlcod 1531'177 active+clean+snaptrim trimq=[c5~1] ps=[c7~1,c9~1]] removing snap head 2021-07-31T02:35:28.004 INFO:journalctl@ceph.osd.6.smithi154.stdout:Jul 31 02:35:27 smithi154 conmon[45576]: 2021-07-31T02:35:28.775 INFO:journalctl@ceph.osd.1.smithi045.stdout:Jul 31 02:35:28 smithi045 conmon[46771]: debug 2021-07-31T02:35:28.549+0000 7f41959a6700 -1 osd.1 pg_epoch: 1535 pg[75.1c( v 1534'484 (0'0,1534'484] local-lis/les=1308/1309 n=35 ec=1308/1308 lis/c=1308/1308 les/c/f=1309/1309/0 sis=1308) [1,5,3] r=0 lpr=1308 crt=1534'484 lcod 1533'482 mlcod 1533'482 active+clean+snaptrim trimq=[c3~1] ps=[c5~1,c7~1,c9~1]] removing snap head 2021-07-31T02:35:28.776 INFO:journalctl@ceph.osd.3.smithi045.stdout:Jul 31 02:35:28 smithi045 conmon[54013]: debug 2021-07-31T02:35:28.590+0000 7f64fa8c9700 -1 osd.3 pg_epoch: 1535 pg[75.1f( v 1535'215 (0'0,1535'215] local-lis/les=1308/1309 n=41 ec=1308/1308 lis/c=1308/1308 les/c/f=1309/1309/0 sis=1308) [3,0,6] r=0 lpr=1308 crt=1535'215 lcod 1535'214 mlcod 1535'214 active+clean+snaptrim trimq=[c3~1] ps=[c5~1,c7~1,c9~1]] removing snap head 2021-07-31T02:35:28.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:28 smithi045 conmon[39718]: cluster 2021-07-31T02:35:27.511647+0000 mon.a (mon.0) 2505 : cluster [DBG] 2021-07-31T02:35:28.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:28 smithi045 conmon[39718]: osdmap e1534: 8 total, 8 up, 8 in 2021-07-31T02:35:28.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:28 smithi045 conmon[39718]: audit 2021-07-31T02:35:28.104181+0000 mon.b (mon.2) 315 : audit [INF] 2021-07-31T02:35:28.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:28 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:35:28.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:28 smithi045 conmon[39718]: audit 2021-07-31T02:35:28.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:28 smithi045 conmon[39718]: 2021-07-31T02:35:28.104821+0000 mon.a (mon.0) 2021-07-31T02:35:28.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:28 smithi045 conmon[39718]: 2506 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:35:28.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:28 smithi045 conmon[39718]: audit 2021-07-31T02:35:28.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:28 smithi045 conmon[39718]: 2021-07-31T02:35:28.124143+0000 mon.b ( 2021-07-31T02:35:28.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:28 smithi045 conmon[39718]: mon.2) 316 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:35:28.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:28 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T02:35:28.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:28 smithi045 conmon[39718]: 02:35:28.124652+0000 mon.a (mon.0) 2507 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:35:28.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:28 smithi045 conmon[39718]: 2021-07-31T02:35:28.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:28 smithi045 conmon[33655]: cluster 2021-07-31T02:35:27.511647+0000 mon.a (mon.0) 2505 2021-07-31T02:35:28.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:28 smithi045 conmon[33655]: : cluster [DBG] osdmap e1534: 8 total, 8 up, 8 in 2021-07-31T02:35:28.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:28 smithi045 conmon[33655]: audit 2021-07-31T02:35:28.104181+0000 mon.b (mon.2) 315 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:35:28.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:28 smithi045 conmon[33655]: audit 2021-07-31T02:35:28.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:28 smithi045 conmon[33655]: 2021-07-31T02:35: 2021-07-31T02:35:28.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:28 smithi045 conmon[33655]: 28.104821+0000 mon.a (mon. 2021-07-31T02:35:28.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:28 smithi045 conmon[33655]: 0) 2506 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:35:28.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:28 smithi045 conmon[33655]: 2021-07-31T02:35:28.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:28 smithi045 conmon[33655]: audit 2021-07-31T02:35:28.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:28 smithi045 conmon[33655]: 2021-07-31T02:35 2021-07-31T02:35:28.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:28 smithi045 conmon[33655]: :28.124143+0000 mon.b ( 2021-07-31T02:35:28.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:28 smithi045 conmon[33655]: mon.2) 316 : audit [INF] 2021-07-31T02:35:28.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:28 smithi045 conmon[33655]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:35:28.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:28 smithi045 conmon[33655]: audit 2021-07- 2021-07-31T02:35:28.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:28 smithi045 conmon[33655]: 31T02:35:28.124652+0000 2021-07-31T02:35:28.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:28 smithi045 conmon[33655]: mon.a (mon.0) 2507 : 2021-07-31T02:35:28.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:28 smithi045 conmon[33655]: audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:35:29.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:28 smithi154 conmon[34553]: cluster 2021-07-31T02:35:27.511647+0000 mon.a (mon.0) 2505 : cluster [DBG] osdmap e1534: 8 total, 8 up, 8 in 2021-07-31T02:35:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:28 smithi154 conmon[34553]: audit 2021-07-31T02:35:28.104181+0000 mon.b (mon.2) 315 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:35:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:28 smithi154 conmon[34553]: audit 2021-07-31T02:35:28.104821+0000 mon.a (mon.0) 2506 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:35:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:28 smithi154 conmon[34553]: audit 2021-07-31T02:35:28.124143+0000 mon.b (mon.2) 2021-07-31T02:35:29.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:28 smithi154 conmon[34553]: 316 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:35:29.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:28 smithi154 conmon[34553]: audit 2021-07-31T02 2021-07-31T02:35:29.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:28 smithi154 conmon[34553]: :35:28.124652+0000 mon.a (mon.0) 2507 : audit 2021-07-31T02:35:29.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:28 smithi154 conmon[34553]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:35:29.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:29 smithi045 conmon[33655]: cluster 2021-07-31T02:35:28.491914+0000 mgr.x (mgr 2021-07-31T02:35:29.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:29 smithi045 conmon[33655]: .24542) 2064 : cluster [DBG] pgmap v2828: 73 pgs: 73 active+clean; 1.9 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2021-07-31T02:35:29.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:29 smithi045 conmon[33655]: cluster 2021-07-31T02:35:28.523098+0000 mon.a (mon. 2021-07-31T02:35:29.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:29 smithi045 conmon[33655]: 0) 2508 : cluster [DBG] osdmap e1535: 8 total, 8 up, 8 in 2021-07-31T02:35:29.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:29 smithi045 conmon[39718]: cluster 2021-07-31T02:35:28.491914+0000 2021-07-31T02:35:29.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:29 smithi045 conmon[39718]: mgr.x (mgr.24542) 2064 : cluster [DBG] pgmap v2828: 73 pgs: 73 active+clean; 1.9 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2021-07-31T02:35:29.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:29 smithi045 conmon[39718]: cluster 2021- 2021-07-31T02:35:29.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:29 smithi045 conmon[39718]: 07-31T02:35:28.523098+0000 mon.a (mon.0) 2508 : 2021-07-31T02:35:29.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:29 smithi045 conmon[39718]: cluster [DBG] osdmap e1535: 8 total, 8 up, 8 in 2021-07-31T02:35:29.937 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMigration.StressLive (25141 ms) 2021-07-31T02:35:30.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:29 smithi154 conmon[34553]: cluster 2021-07-31T02:35:28.491914+0000 mgr.x (mgr.24542 2021-07-31T02:35:30.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:29 smithi154 conmon[34553]: ) 2064 : cluster [DBG] pgmap v2828: 73 pgs: 73 active+clean; 1.9 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2021-07-31T02:35:30.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:29 smithi154 conmon[34553]: cluster 2021-07-31T02:35:28.523098+0000 mon.a (mon.0) 2508 : cluster [DBG] osdmap e1535: 8 total, 8 up, 8 in 2021-07-31T02:35:31.536 INFO:tasks.workunit.client.0.smithi045.stdout:[----------] 57 tests from TestMigration (233311 ms total) 2021-07-31T02:35:31.537 INFO:tasks.workunit.client.0.smithi045.stdout: 2021-07-31T02:35:31.537 INFO:tasks.workunit.client.0.smithi045.stdout:[----------] 2 tests from TestMirroringWatcher 2021-07-31T02:35:32.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:31 smithi154 conmon[34553]: cluster 2021-07-31T02:35:32.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:31 smithi154 conmon[34553]: 2021-07-31T02:35:32.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:31 smithi154 conmon[34553]: 2021 2021-07-31T02:35:32.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:31 smithi154 conmon[34553]: - 2021-07-31T02:35:32.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:31 smithi154 conmon[34553]: 07 2021-07-31T02:35:32.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:31 smithi154 conmon[34553]: - 2021-07-31T02:35:32.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:31 smithi154 conmon[34553]: 31 2021-07-31T02:35:32.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:31 smithi154 conmon[34553]: T 2021-07-31T02:35:32.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:31 smithi154 conmon[34553]: 02 2021-07-31T02:35:32.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:31 smithi154 conmon[34553]: : 2021-07-31T02:35:32.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:31 smithi154 conmon[34553]: 35 2021-07-31T02:35:32.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:31 smithi154 conmon[34553]: : 2021-07-31T02:35:32.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:31 smithi154 conmon[34553]: 30 2021-07-31T02:35:32.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:31 smithi154 conmon[34553]: . 2021-07-31T02:35:32.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:31 smithi154 conmon[34553]: 492512 2021-07-31T02:35:32.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:31 smithi154 conmon[34553]: +0000 2021-07-31T02:35:32.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:31 smithi154 conmon[34553]: 2021-07-31T02:35:32.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:31 smithi154 conmon[34553]: mgr.x 2021-07-31T02:35:32.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:31 smithi154 conmon[34553]: ( 2021-07-31T02:35:32.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:31 smithi154 conmon[34553]: mgr 2021-07-31T02:35:32.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:31 smithi154 conmon[34553]: .24542) 2065 : cluster 2021-07-31T02:35:32.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:31 smithi154 conmon[34553]: [DBG] pgmap v2830: 73 pgs: 73 active+clean; 1.9 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 21 MiB/s rd, 14 MiB/s wr, 147 op/s 2021-07-31T02:35:32.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:31 smithi154 conmon[34553]: cluster 2021-07 2021-07-31T02:35:32.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:31 smithi154 conmon[34553]: -31T02:35:30.528862+0000 mon.a (mon 2021-07-31T02:35:32.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:31 smithi154 conmon[34553]: .0) 2509 : cluster [DBG] osdmap e1536: 8 total, 8 up, 8 in 2021-07-31T02:35:32.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:31 smithi045 conmon[39718]: cluster 2021-07-31T02:35:30.492512+0000 mgr.x (mgr.24542) 2065 2021-07-31T02:35:32.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:31 smithi045 conmon[39718]: : cluster [DBG] pgmap v2830: 73 pgs: 73 active+clean; 1.9 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 21 MiB/s rd, 14 MiB/s wr, 147 op/s 2021-07-31T02:35:32.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:31 smithi045 conmon[39718]: cluster 2021-07-31T02:35:30.528862+0000 mon.a (mon.0) 2509 : cluster [DBG] osdmap e1536: 8 total, 8 up, 8 in 2021-07-31T02:35:32.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:31 smithi045 conmon[33655]: cluster 2021-07-31T02:35:32.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:31 smithi045 conmon[33655]: 2021-07-31T02:35:32.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:31 smithi045 conmon[33655]: 2021 2021-07-31T02:35:32.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:31 smithi045 conmon[33655]: - 2021-07-31T02:35:32.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:31 smithi045 conmon[33655]: 07 2021-07-31T02:35:32.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:31 smithi045 conmon[33655]: - 2021-07-31T02:35:32.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:31 smithi045 conmon[33655]: 31 2021-07-31T02:35:32.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:31 smithi045 conmon[33655]: T 2021-07-31T02:35:32.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:31 smithi045 conmon[33655]: 02 2021-07-31T02:35:32.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:31 smithi045 conmon[33655]: : 2021-07-31T02:35:32.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:31 smithi045 conmon[33655]: 35 2021-07-31T02:35:32.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:31 smithi045 conmon[33655]: : 2021-07-31T02:35:32.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:31 smithi045 conmon[33655]: 30 2021-07-31T02:35:32.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:31 smithi045 conmon[33655]: . 2021-07-31T02:35:32.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:31 smithi045 conmon[33655]: 492512 2021-07-31T02:35:32.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:31 smithi045 conmon[33655]: +0000 2021-07-31T02:35:32.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:31 smithi045 conmon[33655]: 2021-07-31T02:35:32.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:31 smithi045 conmon[33655]: mgr.x 2021-07-31T02:35:32.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:31 smithi045 conmon[33655]: ( 2021-07-31T02:35:32.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:31 smithi045 conmon[33655]: mgr 2021-07-31T02:35:32.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:31 smithi045 conmon[33655]: . 2021-07-31T02:35:32.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:31 smithi045 conmon[33655]: 24542 2021-07-31T02:35:32.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:31 smithi045 conmon[33655]: ) 2021-07-31T02:35:32.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:31 smithi045 conmon[33655]: 2065 2021-07-31T02:35:32.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:31 smithi045 conmon[33655]: : 2021-07-31T02:35:32.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:31 smithi045 conmon[33655]: cluster 2021-07-31T02:35:32.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:31 smithi045 conmon[33655]: 2021-07-31T02:35:32.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:31 smithi045 conmon[33655]: [DBG] 2021-07-31T02:35:32.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:31 smithi045 conmon[33655]: 2021-07-31T02:35:32.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:31 smithi045 conmon[33655]: pgmap v2830: 73 pgs: 73 active+clean; 1.9 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 21 MiB/s rd, 14 MiB/s wr, 147 op/s 2021-07-31T02:35:32.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:31 smithi045 conmon[33655]: 2021-07-31T02:35:32.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:31 smithi045 conmon[33655]: cluster 2021-07-31T02:35:32.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:31 smithi045 conmon[33655]: 2021-07-31T02:35:32.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:31 smithi045 conmon[33655]: 2021 2021-07-31T02:35:32.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:31 smithi045 conmon[33655]: - 2021-07-31T02:35:32.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:31 smithi045 conmon[33655]: 07 2021-07-31T02:35:32.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:31 smithi045 conmon[33655]: - 2021-07-31T02:35:32.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:31 smithi045 conmon[33655]: 31 2021-07-31T02:35:32.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:31 smithi045 conmon[33655]: T 2021-07-31T02:35:32.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:31 smithi045 conmon[33655]: 02 2021-07-31T02:35:32.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:31 smithi045 conmon[33655]: : 2021-07-31T02:35:32.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:31 smithi045 conmon[33655]: 35 2021-07-31T02:35:32.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:31 smithi045 conmon[33655]: : 2021-07-31T02:35:32.041 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:31 smithi045 conmon[33655]: 30 2021-07-31T02:35:32.041 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:31 smithi045 conmon[33655]: . 2021-07-31T02:35:32.042 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:31 smithi045 conmon[33655]: 528862 2021-07-31T02:35:32.042 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:31 smithi045 conmon[33655]: +0000 2021-07-31T02:35:32.042 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:31 smithi045 conmon[33655]: mon.a (mon.0) 2509 : cluster [DBG] osdmap e1536: 8 total, 8 up, 8 in 2021-07-31T02:35:32.793 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[34553]: cluster 2021-07-31T02:35:31.530907+0000 mon.a (mon.0) 2510 : cluster 2021-07-31T02:35:33.202 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[34553]: [DBG] osdmap e1537: 8 total, 8 up, 8 in 2021-07-31T02:35:33.203 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: debug 2021-07-31T02:35:32.781+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:35:33.203 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:35:33.204 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:35:33.204 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: yield 2021-07-31T02:35:33.204 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:35:33.205 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:35:33.205 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:35:33.205 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:35:33.205 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:35:33.206 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:35:33.206 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:35:33.206 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:35:33.206 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:35:33.207 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: task.result() 2021-07-31T02:35:33.207 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:35:33.208 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:32 smithi045 conmon[39718]: cluster 2021-07-31T02:35:31.530907+0000 2021-07-31T02:35:33.208 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:32 smithi045 conmon[39718]: mon.a (mon.0) 2510 : cluster [DBG] osdmap e1537: 8 total, 8 up, 8 in 2021-07-31T02:35:33.208 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:32 smithi045 conmon[33655]: cluster 2021-07-31T02:35:31.530907+0000 2021-07-31T02:35:33.208 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:32 smithi045 conmon[33655]: mon.a (mon.0) 2510 : cluster [DBG] osdmap e1537: 8 total, 8 up, 8 in 2021-07-31T02:35:33.253 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: debug 2021-07-31T02:35:33.253 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: 2021-07-31T02:35:32.792+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:35:33.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:35:33.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:35:33.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: yield 2021-07-31T02:35:33.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:35:33.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:35:33.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:35:33.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:35:33.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:35:33.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:35:33.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:35:33.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:35:33.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:35:33.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: task.result() 2021-07-31T02:35:33.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:35:33.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: debug 2021-07-31T02:35:32.815+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:35:33.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:35:33.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:35:33.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: yield 2021-07-31T02:35:33.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:35:33.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:35:33.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:35:33.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:35:33.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:35:33.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:35:33.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:35:33.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:35:33.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:35:33.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: task.result() 2021-07-31T02:35:33.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:35:33.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: debug 2021-07-31T02:35:32.816+0000 7fcb9095b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:35:33.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:35:33.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:35:33.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: yield 2021-07-31T02:35:33.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:35:33.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:35:33.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:35:33.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:35:33.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:35:33.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:35:33.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:35:33.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:35:33.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:35:33.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: task.result() 2021-07-31T02:35:33.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:35:33.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: 2021-07-31T02:35:33.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:35:33.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: 2021-07-31T02:35:33.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:35:33.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:35:33.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:35:33.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:35:33.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:35:33.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:35:33.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:35:33.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:35:33.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:35:33.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:35:33.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:35:33.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:35:33.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:35:33.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:35:33.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:35:33.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:35:33.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:35:33.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:35:33.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: raise 2021-07-31T02:35:33.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:35:33.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:35:33.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:35:33.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:35:33.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:35:33.284 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: debug 2021-07-31T02:35:32.824+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:35:33.284 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:35:33.284 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:35:33.284 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: yield 2021-07-31T02:35:33.284 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:35:33.285 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:35:33.285 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:35:33.285 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:35:33.285 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:35:33.285 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:35:33.286 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:35:33.286 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:35:33.286 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:35:33.287 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: task.result() 2021-07-31T02:35:33.287 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:35:33.287 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: debug 2021-07-31T02:35:32.825+0000 7fcb9115c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:35:33.287 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:35:33.288 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:35:33.288 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: yield 2021-07-31T02:35:33.288 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:35:33.288 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:35:33.288 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:35:33.289 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:35:33.289 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:35:33.289 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:35:33.290 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:35:33.290 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:35:33.290 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:35:33.290 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: task.result() 2021-07-31T02:35:33.291 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:35:33.291 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: 2021-07-31T02:35:33.292 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:35:33.292 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: 2021-07-31T02:35:33.292 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:35:33.293 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:35:33.293 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:35:33.294 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:35:33.294 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:35:33.294 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:35:33.295 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:35:33.295 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:35:33.296 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:35:33.296 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:35:33.297 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:35:33.297 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:35:33.297 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:35:33.298 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:35:33.298 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:35:33.299 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:35:33.299 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:35:33.299 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:35:33.300 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: raise 2021-07-31T02:35:33.301 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:35:33.301 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:35:33.302 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:35:33.302 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:35:33.303 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:35:32 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:35:33.546 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMirroringWatcher.ModeUpdated 2021-07-31T02:35:34.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: cluster 2021-07-31T02:35:32.492916+0000 mgr.x (mgr.24542) 2066 : cluster [DBG] pgmap v2833: 9 pgs: 9 active+clean; 4.6 MiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2021-07-31T02:35:34.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: cluster 2021-07-31T02:35:32.554977+0000 mon.a (mon.0) 2511 : cluster [DBG] osdmap e1538: 8 total, 8 up, 8 in 2021-07-31T02:35:34.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: audit 2021-07-31T02:35:32.561134+0000 mon.c (mon.1) 120 : audit [INF] from='client.? 172.21.15.45:0/2231492010' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:35:34.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: audit 2021-07-31T02:35:32.561494+0000 mon.a (mon.0) 2512 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:35:34.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: audit 2021-07-31T02:35:32.739593+0000 mon.b (mon.2) 317 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:35:34.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: audit 2021-07-31T02:35:34.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: 2021-07-31T02:35:32.740712+0000 mon.b (mon.2) 318 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:35:34.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: audit 2021-07-31T02:35:32.741361+0000 mon.b (mon.2) 319 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:35:34.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:35:32.741979+0000 mgr.x (mgr.24542) 2067 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:35:34.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:35:34.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: 2021-07-31T02:35:32.782290+0000 mgr.x (mgr.24542) 2068 : cephadm [ERR] 2021-07-31T02:35:34.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:35:34.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:35:34.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: yield 2021-07-31T02:35:34.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:35:34.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:35:34.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:35:34.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:35:34.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:35:34.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:35:34.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:35:34.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:35:34.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:35:34.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:35:34.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:35:34.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:35:32.782636+0000 mgr.x (mgr.24542) 2069 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:35:34.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:35:32.793071+0000 mgr.x (mgr.24542) 2070 : cephadm [ERR] 2021-07-31T02:35:34.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:35:34.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:35:34.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: yield 2021-07-31T02:35:34.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:35:34.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:35:34.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:35:34.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:35:34.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:35:34.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:35:34.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:35:34.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:35:34.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:35:34.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:35:34.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:35:34.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:35:34.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: 2021-07-31T02:35:32.793857+0000 mgr.x (mgr.24542) 2071 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:35:34.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:35:32.816903+0000 mgr.x (mgr.24542) 2072 : cephadm [ERR] 2021-07-31T02:35:34.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:35:34.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:35:34.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: yield 2021-07-31T02:35:34.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:35:34.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:35:34.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:35:34.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:35:34.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:35:34.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:35:34.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:35:34.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:35:34.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:35:34.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:35:34.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:35:34.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:35: 2021-07-31T02:35:34.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: 32.817266+0000 mgr.x (mgr.24542) 2073 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:35:34.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:35:34.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:35:34.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: yield 2021-07-31T02:35:34.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:35:34.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:35:34.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:35:34.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:35:34.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:35:34.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:35:34.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:35:34.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:35:34.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:35:34.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:35:34.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:35:34.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: 2021-07-31T02:35:34.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:35:34.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: 2021-07-31T02:35:34.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:35:34.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:35:34.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:35:34.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:35:34.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:35:34.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:35:34.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:35:34.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:35:34.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:35:34.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:35:34.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:35:34.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:35:34.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:35:34.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:35:34.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:35:34.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:35:34.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:35:34.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:35:34.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: raise 2021-07-31T02:35:34.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:35:34.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:35:34.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:35:34.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:35:34.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:35:34.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:35:32.825872+0000 mgr.x (mgr.24542) 2074 : cephadm [ERR] 2021-07-31T02:35:34.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:35:34.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:35:34.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: yield 2021-07-31T02:35:34.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:35:34.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:35:34.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:35:34.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:35:34.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:35:34.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: cluster 2021-07-31T02:35:34.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: 2021-07-31T02:35:34.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: 2021 2021-07-31T02:35:34.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: - 2021-07-31T02:35:34.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: 07 2021-07-31T02:35:34.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: - 2021-07-31T02:35:34.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: 31 2021-07-31T02:35:34.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: T 2021-07-31T02:35:34.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: 02:35:32.492916+0000 mgr.x (mgr. 2021-07-31T02:35:34.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: 24542) 2066 : cluster [DBG] pgmap v2833: 9 pgs: 9 active+clean; 4.6 MiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2021-07-31T02:35:34.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: cluster 2021-07-31T02:35:32.554977+0000 mon.a (mon.0) 2511 : cluster [DBG] osdmap e1538: 8 total, 8 up, 8 in 2021-07-31T02:35:34.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: audit 2021-07-31T02:35:32.561134+0000 mon.c (mon.1) 120 : audit [INF] from='client.? 172.21.15.45:0/2231492010' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:35:34.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: audit 2021-07-31T02:35:32.561494+0000 mon.a (mon.0) 2512 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:35:34.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: audit 2021-07-31T02:35:32.739593+0000 mon.b (mon.2) 317 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:35:34.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: audit 2021-07-31T02:35:32.740712+0000 mon.b (mon.2) 318 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:35:34.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: audit 2021-07-31T02:35:32.741361+0000 mon.b (mon.2) 319 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:35:34.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:35: 2021-07-31T02:35:34.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: 32.741979+0000 mgr.x (mgr.24542) 2067 2021-07-31T02:35:34.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:35:34.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: cephadm 2021-07-31T 2021-07-31T02:35:34.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: 02:35:32.782290+0000 mgr.x (mgr.24542) 2021-07-31T02:35:34.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: 2068 : cephadm [ERR] 2021-07-31T02:35:34.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:35:34.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:35:34.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: yield 2021-07-31T02:35:34.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:35:34.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:35:34.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:35:34.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:35:34.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:35:34.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:35:34.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:35:34.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:35:34.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:35:34.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:35:34.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:35:34.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: cluster 2021-07-31T02:35:34.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: 2021-07-31T02:35:34.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: 2021 2021-07-31T02:35:34.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: - 2021-07-31T02:35:34.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: 07 2021-07-31T02:35:34.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: - 2021-07-31T02:35:34.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: 31 2021-07-31T02:35:34.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: T 2021-07-31T02:35:34.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: 02 2021-07-31T02:35:34.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: : 2021-07-31T02:35:34.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: 35 2021-07-31T02:35:34.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: : 2021-07-31T02:35:34.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: 32 2021-07-31T02:35:34.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: . 2021-07-31T02:35:34.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: 492916 2021-07-31T02:35:34.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: +0000 2021-07-31T02:35:34.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: 2021-07-31T02:35:34.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: mgr.x 2021-07-31T02:35:34.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: ( 2021-07-31T02:35:34.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: mgr 2021-07-31T02:35:34.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: . 2021-07-31T02:35:34.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: 24542 2021-07-31T02:35:34.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: ) 2021-07-31T02:35:34.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: 2066 2021-07-31T02:35:34.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: : 2021-07-31T02:35:34.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: cluster 2021-07-31T02:35:34.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: 2021-07-31T02:35:34.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: [DBG] 2021-07-31T02:35:34.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: 2021-07-31T02:35:34.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: pgmap v2833: 9 pgs: 9 active+clean; 4.6 MiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2021-07-31T02:35:34.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: 2021-07-31T02:35:34.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: cluster 2021-07-31T02:35:34.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: 2021-07-31T02:35:34.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: 2021 2021-07-31T02:35:34.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: - 2021-07-31T02:35:34.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: 07 2021-07-31T02:35:34.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: - 2021-07-31T02:35:34.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: 31 2021-07-31T02:35:34.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: T 2021-07-31T02:35:34.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: 02 2021-07-31T02:35:34.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: : 2021-07-31T02:35:34.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: 35 2021-07-31T02:35:34.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: : 2021-07-31T02:35:34.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: 32 2021-07-31T02:35:34.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: . 2021-07-31T02:35:34.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: 554977 2021-07-31T02:35:34.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: +0000 2021-07-31T02:35:34.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: 2021-07-31T02:35:34.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: mon.a 2021-07-31T02:35:34.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: ( 2021-07-31T02:35:34.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: mon 2021-07-31T02:35:34.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: . 2021-07-31T02:35:34.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: 0 2021-07-31T02:35:34.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: ) 2021-07-31T02:35:34.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: 2511 2021-07-31T02:35:34.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: : 2021-07-31T02:35:34.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: cluster 2021-07-31T02:35:34.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: 2021-07-31T02:35:34.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: [DBG] 2021-07-31T02:35:34.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: 2021-07-31T02:35:34.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: osdmap e1538: 8 total, 8 up, 8 in 2021-07-31T02:35:34.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: 2021-07-31T02:35:34.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: audit 2021-07-31T02:35:34.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: 2021-07-31T02:35:34.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: 2021 2021-07-31T02:35:34.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: - 2021-07-31T02:35:34.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: 07 2021-07-31T02:35:34.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: - 2021-07-31T02:35:34.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: 31 2021-07-31T02:35:34.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: T 2021-07-31T02:35:34.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: 02 2021-07-31T02:35:34.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: : 2021-07-31T02:35:34.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: 35 2021-07-31T02:35:34.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: : 2021-07-31T02:35:34.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: 32 2021-07-31T02:35:34.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: . 2021-07-31T02:35:34.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: 561134 2021-07-31T02:35:34.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: +0000 2021-07-31T02:35:34.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: 2021-07-31T02:35:34.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: mon.c 2021-07-31T02:35:34.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: ( 2021-07-31T02:35:34.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: mon 2021-07-31T02:35:34.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: . 2021-07-31T02:35:34.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: 1 2021-07-31T02:35:34.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: ) 2021-07-31T02:35:34.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: 120 2021-07-31T02:35:34.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: : 2021-07-31T02:35:34.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: audit 2021-07-31T02:35:34.057 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: 2021-07-31T02:35:34.057 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:35:34.057 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:35:34.057 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:35:34.058 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:35:34.058 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:35:34.058 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:35:34.058 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: cephadm 2021-07-31T 2021-07-31T02:35:34.058 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: 02:35:32.826202+0000 mgr.x (mgr.24542) 2075 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:35:34.059 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:35:34.059 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:35:34.059 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: yield 2021-07-31T02:35:34.059 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:35:34.059 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:35:34.060 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:35:34.060 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:35:34.060 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:35:34.060 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:35:34.061 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:35:34.061 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:35:34.061 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:35:34.061 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:35:34.061 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:35:34.062 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: 2021-07-31T02:35:34.062 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:35:34.062 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: 2021-07-31T02:35:34.062 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:35:34.063 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:35:34.063 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:35:34.063 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:35:34.063 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:35:34.063 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:35:34.064 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:35:34.064 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:35:34.064 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:35:34.064 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:35:34.064 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:35:34.065 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:35:34.065 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:35:34.065 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:35:34.065 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:35:34.066 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:35:34.066 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:35:34.066 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:35:34.066 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: raise 2021-07-31T02:35:34.066 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:35:34.067 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:35:34.067 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:35:34.067 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:35:34.067 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:35:34.067 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: cluster 2021-07-31T02:35:33.393868+0000 mon.a (mon.0) 2513 : cluster [INF] Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2021-07-31T02:35:34.068 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:33 smithi154 conmon[34553]: cluster 2021-07-31T02:35:33.393906+0000 mon.a (mon.0) 2514 : cluster [INF] Cluster is now healthy 2021-07-31T02:35:34.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:35:32.782636+0000 mgr.x (mgr.24542) 2069 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:35:34.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:35:32.793071+0000 mgr.x (mgr.24542) 2070 : cephadm [ERR] 2021-07-31T02:35:34.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:35:34.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:35:34.070 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: yield 2021-07-31T02:35:34.070 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:35:34.070 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:35:34.070 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:35:34.070 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:35:34.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:35:34.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:35:34.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:35:34.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:35:34.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:35:34.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:35:34.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:35:34.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:35:32.793857+0000 mgr.x (mgr.24542) 2071 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:35:34.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:35:32.816903+0000 mgr.x (mgr.24542) 2072 : cephadm [ERR] 2021-07-31T02:35:34.073 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:35:34.073 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:35:34.073 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: yield 2021-07-31T02:35:34.073 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:35:34.074 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:35:34.074 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:35:34.074 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:35:34.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:35:34.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:35:34.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:35:34.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:35:34.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:35:34.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:35:34.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:35:34.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:35:32.817266+0000 mgr.x (mgr.24542) 2073 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:35:34.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:35:34.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:35:34.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: yield 2021-07-31T02:35:34.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:35:34.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:35:34.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:35:34.078 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:35:34.078 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:35:34.078 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:35:34.078 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:35:34.078 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:35:34.079 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:35:34.079 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:35:34.079 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:35:34.079 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: 2021-07-31T02:35:34.080 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:35:34.080 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: 2021-07-31T02:35:34.080 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:35:34.080 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:35:34.080 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:35:34.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:35:34.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:35:34.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:35:34.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:35:34.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:35:34.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:35:34.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:35:34.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:35:34.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:35:34.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:35:34.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:35:34.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:35:34.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:35:34.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:35:34.084 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:35:34.084 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: raise 2021-07-31T02:35:34.084 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:35:34.084 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:35:34.085 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:35:34.085 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:35:34.085 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:35:34.085 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:35:32.825872+0000 mgr.x (mgr.24542) 2074 : cephadm [ERR] 2021-07-31T02:35:34.085 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:35:34.086 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:35:34.086 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: yield 2021-07-31T02:35:34.086 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:35:34.086 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:35:34.086 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:35:34.087 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:35:34.087 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:35:34.087 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:35:34.087 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:35:34.087 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:35:34.088 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:35:34.088 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:35:34.088 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:35:34.088 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:35:32.826202+0000 mgr.x (mgr.24542) 2075 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:35:34.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:35:34.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:35:34.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: yield 2021-07-31T02:35:34.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:35:34.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:35:34.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:35:34.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:35:34.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:35:34.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:35:34.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:35:34.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:35:34.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:35:34.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:35:34.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:35:34.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: 2021-07-31T02:35:34.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:35:34.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: 2021-07-31T02:35:34.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:35:34.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:35:34.093 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:35:34.093 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:35:34.093 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:35:34.093 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:35:34.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:35:34.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:35:34.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:35:34.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:35:34.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:35:34.095 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:35:34.095 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:35:34.095 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: [INF] 2021-07-31T02:35:34.095 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: 2021-07-31T02:35:34.096 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: from='client.? 172.21.15.45:0/2231492010' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:35:34.096 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: 2021-07-31T02:35:34.096 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: audit 2021-07-31T02:35:34.096 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: 2021-07-31T02:35:34.097 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: 2021 2021-07-31T02:35:34.097 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: - 2021-07-31T02:35:34.097 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: 07 2021-07-31T02:35:34.097 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: - 2021-07-31T02:35:34.097 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: 31 2021-07-31T02:35:34.098 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: T 2021-07-31T02:35:34.098 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: 02 2021-07-31T02:35:34.098 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: : 2021-07-31T02:35:34.098 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: 35 2021-07-31T02:35:34.099 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: : 2021-07-31T02:35:34.099 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: 32 2021-07-31T02:35:34.099 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: . 2021-07-31T02:35:34.099 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: 561494 2021-07-31T02:35:34.099 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: +0000 2021-07-31T02:35:34.100 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: 2021-07-31T02:35:34.100 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: mon.a 2021-07-31T02:35:34.100 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: ( 2021-07-31T02:35:34.100 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: mon 2021-07-31T02:35:34.100 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: . 2021-07-31T02:35:34.101 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: 0 2021-07-31T02:35:34.101 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: ) 2021-07-31T02:35:34.101 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: 2512 2021-07-31T02:35:34.101 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: : 2021-07-31T02:35:34.102 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: audit 2021-07-31T02:35:34.102 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: 2021-07-31T02:35:34.102 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: [INF] 2021-07-31T02:35:34.102 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: 2021-07-31T02:35:34.102 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:35:34.103 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: 2021-07-31T02:35:34.103 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: audit 2021-07-31T02:35:34.103 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: 2021-07-31T02:35:34.103 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: 2021 2021-07-31T02:35:34.103 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: - 2021-07-31T02:35:34.104 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: 07 2021-07-31T02:35:34.104 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: - 2021-07-31T02:35:34.104 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: 31 2021-07-31T02:35:34.104 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: T 2021-07-31T02:35:34.104 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: 02 2021-07-31T02:35:34.105 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: : 2021-07-31T02:35:34.105 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: 35 2021-07-31T02:35:34.105 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: : 2021-07-31T02:35:34.105 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: 32 2021-07-31T02:35:34.105 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: . 2021-07-31T02:35:34.106 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: 739593 2021-07-31T02:35:34.106 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: +0000 2021-07-31T02:35:34.106 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: 2021-07-31T02:35:34.106 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: mon.b (mon.2) 317 : audit 2021-07-31T02:35:34.107 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:35:34.107 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: audit 2021-07-31T02:35:34.107 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: 2021-07-31T02:35:32.740712+0000 mon.b (mon.2) 318 2021-07-31T02:35:34.107 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:35:34.107 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T02:35:34.108 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: T02:35:32.741361+0000 mon.b (mon. 2021-07-31T02:35:34.108 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: 2) 319 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:35:34.108 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: cephadm 2021-07 2021-07-31T02:35:34.108 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: -31T02:35:32.741979+0000 mgr.x ( 2021-07-31T02:35:34.108 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: mgr.24542) 2067 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:35:34.109 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: cephadm 2021- 2021-07-31T02:35:34.109 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: 07-31T02:35:32.782290+0000 mgr.x ( 2021-07-31T02:35:34.109 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: mgr.24542) 2068 : cephadm [ERR] 2021-07-31T02:35:34.109 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:35:34.110 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:35:34.110 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:35:34.110 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:35:34.110 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:35:34.111 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: raise 2021-07-31T02:35:34.111 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:35:34.111 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:35:34.111 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:35:34.111 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:35:34.112 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:35:34.112 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: cluster 2021-07-31T02:35:33.393868+0000 mon.a (mon.0) 2513 : cluster [INF] Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2021-07-31T02:35:34.112 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[33655]: cluster 2021-07-31T02:35:33.393906+0000 mon.a (mon.0) 2514 : cluster [INF] Cluster is now healthy 2021-07-31T02:35:34.113 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:35:34.113 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:35:34.113 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: yield 2021-07-31T02:35:34.113 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:35:34.114 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:35:34.114 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:35:34.114 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:35:34.114 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:35:34.115 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:35:34.115 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:35:34.115 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:35:34.115 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:35:34.116 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:35:34.116 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:35:34.116 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:35:32.782636+0000 mgr.x (mgr.24542) 2069 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:35:34.116 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:35:32.793071+0000 2021-07-31T02:35:34.116 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: mgr.x (mgr.24542) 2070 : cephadm [ERR] 2021-07-31T02:35:34.117 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:35:34.117 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:35:34.117 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: yield 2021-07-31T02:35:34.117 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:35:34.118 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:35:34.118 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:35:34.118 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:35:34.118 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:35:34.118 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:35:34.119 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:35:34.119 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:35:34.119 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:35:34.119 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:35:34.120 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:35:34.120 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:35:32.793857+0000 mgr.x (mgr.24542) 2071 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:35:34.120 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:35:32.816903+0000 mgr.x (mgr.24542) 2072 : cephadm 2021-07-31T02:35:34.120 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: [ERR] 2021-07-31T02:35:34.120 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:35:34.121 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:35:34.121 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: yield 2021-07-31T02:35:34.121 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:35:34.121 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:35:34.121 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:35:34.122 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:35:34.122 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:35:34.122 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:35:34.122 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:35:34.123 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:35:34.123 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:35:34.123 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:35:34.123 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:35:34.123 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:35:32.817266+0000 mgr.x (mgr.24542) 2073 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:35:34.124 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:35:34.124 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:35:34.124 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: yield 2021-07-31T02:35:34.124 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:35:34.124 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:35:34.125 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:35:34.125 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:35:34.125 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:35:34.125 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:35:34.125 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:35:34.126 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:35:34.126 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:35:34.126 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:35:34.126 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:35:34.127 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: 2021-07-31T02:35:34.127 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:35:34.127 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: 2021-07-31T02:35:34.127 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:35:34.127 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:35:34.128 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:35:34.128 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:35:34.128 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:35:34.128 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:35:34.129 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:35:34.129 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:35:34.129 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:35:34.129 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:35:34.129 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:35:34.130 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:35:34.130 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:35:34.130 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:35:34.130 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:35:34.131 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:35:34.131 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:35:34.131 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:35:34.131 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: raise 2021-07-31T02:35:34.131 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:35:34.132 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:35:34.132 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:35:34.132 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:35:34.132 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:35:34.132 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:35:32.825872+0000 mgr.x (mgr.24542) 2074 : 2021-07-31T02:35:34.133 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: cephadm [ERR] 2021-07-31T02:35:34.133 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:35:34.133 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:35:34.133 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: yield 2021-07-31T02:35:34.134 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:35:34.134 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:35:34.134 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:35:34.134 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:35:34.134 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:35:34.135 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:35:34.135 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:35:34.135 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:35:34.135 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:35:34.136 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:35:34.136 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:35:34.136 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:35:32.826202+0000 mgr.x (mgr.24542) 2075 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:35:34.136 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:35:34.136 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:35:34.137 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: yield 2021-07-31T02:35:34.137 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:35:34.137 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:35:34.137 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:35:34.137 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:35:34.138 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:35:34.138 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:35:34.138 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:35:34.138 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:35:34.139 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:35:34.139 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:35:34.139 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:35:34.139 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: 2021-07-31T02:35:34.139 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:35:34.140 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: 2021-07-31T02:35:34.140 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:35:34.140 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:35:34.140 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:35:34.140 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:35:34.141 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:35:34.141 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:35:34.141 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:35:34.141 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:35:34.142 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:35:34.142 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:35:34.142 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:35:34.142 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:35:34.142 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:35:34.143 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:35:34.143 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:35:34.143 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:35:34.143 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:35:34.143 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:35:34.144 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: raise 2021-07-31T02:35:34.144 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:35:34.144 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:35:34.144 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:35:34.145 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:35:34.145 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:35:34.145 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: cluster 2021-07-31T02:35:33.393868+0000 mon.a (mon.0) 2513 : cluster [INF] Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2021-07-31T02:35:34.145 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:33 smithi045 conmon[39718]: cluster 2021-07-31T02:35:33.393906+0000 mon.a (mon.0) 2514 : cluster [INF] Cluster is now healthy 2021-07-31T02:35:35.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:34 smithi154 conmon[34553]: audit 2021-07-31T02:35:33.544681+0000 mon.a (mon.0) 2515 : audit 2021-07-31T02:35:35.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:34 smithi154 conmon[34553]: [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-15","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:35:35.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:34 smithi154 conmon[34553]: cluster 2021-07-31T02:35:33.544977+0000 mon.a (mon.0) 2516 : cluster [DBG] osdmap e1539: 8 total, 8 up, 8 in 2021-07-31T02:35:35.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:34 smithi045 conmon[39718]: audit 2021-07-31T02:35:33.544681+0000 mon.a (mon 2021-07-31T02:35:35.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:34 smithi045 conmon[39718]: .0) 2515 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-15","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:35:35.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:34 smithi045 conmon[39718]: cluster 2021-07-31T02:35:33.544977+0000 mon.a (mon.0) 2021-07-31T02:35:35.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:34 smithi045 conmon[39718]: 2516 : cluster [DBG] osdmap e1539: 8 total, 8 up, 8 in 2021-07-31T02:35:35.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:34 smithi045 conmon[33655]: audit 2021-07-31T02:35:33.544681+0000 mon.a (mon.0) 2021-07-31T02:35:35.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:34 smithi045 conmon[33655]: 2515 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-15","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:35:35.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:34 smithi045 conmon[33655]: cluster 2021-07-31T02:35:33.544977+0000 mon.a (mon.0) 2516 : cluster [DBG] osdmap e1539: 8 total, 8 up, 8 in 2021-07-31T02:35:35.587 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMirroringWatcher.ModeUpdated (2042 ms) 2021-07-31T02:35:35.588 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestMirroringWatcher.ImageStatusUpdated 2021-07-31T02:35:35.605 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestMirroringWatcher.ImageStatusUpdated (17 ms) 2021-07-31T02:35:36.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:35 smithi154 conmon[34553]: cluster 2021-07-31T02:35:34.493658+0000 mgr.x (mgr.24542) 2076 : cluster [DBG] 2021-07-31T02:35:36.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:35 smithi154 conmon[34553]: pgmap v2836: 41 pgs: 41 active+clean; 4.6 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:35:36.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:35 smithi154 conmon[34553]: cluster 2021-07-31T02:35:34.557435+0000 mon.a (mon.0) 2517 : cluster [DBG] osdmap e1540: 8 total, 8 up, 8 in 2021-07-31T02:35:36.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:35 smithi045 conmon[39718]: cluster 2021-07-31T02:35:34.493658+0000 2021-07-31T02:35:36.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:35 smithi045 conmon[39718]: mgr.x (mgr.24542) 2076 : cluster [DBG] pgmap v2836: 41 pgs: 41 active+clean; 4.6 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:35:36.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:35 smithi045 conmon[39718]: cluster 2021-07-31 2021-07-31T02:35:36.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:35 smithi045 conmon[39718]: T02:35:34.557435+0000 mon.a (mon.0) 2517 2021-07-31T02:35:36.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:35 smithi045 conmon[39718]: : cluster [DBG] osdmap e1540: 8 total, 8 up, 8 in 2021-07-31T02:35:36.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:35 smithi045 conmon[33655]: cluster 2021-07-31T02:35:34.493658+0000 mgr.x (mgr.24542) 2076 : cluster [DBG] pgmap v2836: 41 pgs: 41 active+clean; 4.6 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:35:36.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:35 smithi045 conmon[33655]: cluster 2021-07-31T02:35:34.557435+0000 mon.a (mon.0) 2517 : cluster [DBG] osdmap e1540: 8 total, 8 up, 8 in 2021-07-31T02:35:36.571 INFO:tasks.workunit.client.0.smithi045.stdout:[----------] 2 tests from TestMirroringWatcher (2059 ms total) 2021-07-31T02:35:36.571 INFO:tasks.workunit.client.0.smithi045.stdout: 2021-07-31T02:35:36.572 INFO:tasks.workunit.client.0.smithi045.stdout:[----------] 4 tests from TestObjectMap 2021-07-31T02:35:37.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:36 smithi154 conmon[34553]: cluster 2021-07-31T02:35:35.562498+0000 mon.a (mon.0) 2518 : 2021-07-31T02:35:37.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:36 smithi154 conmon[34553]: cluster [DBG] osdmap e1541: 8 total, 8 up, 8 in 2021-07-31T02:35:37.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:36 smithi045 conmon[39718]: cluster 2021-07-31T02 2021-07-31T02:35:37.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:36 smithi045 conmon[39718]: :35:35.562498+0000 mon.a (mon.0) 2518 : cluster [DBG] osdmap e1541: 8 total, 8 up, 8 in 2021-07-31T02:35:37.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:36 smithi045 conmon[33655]: cluster 2021-07-31T02:35:35.562498+0000 mon.a (mon.0) 2518 : cluster [DBG] 2021-07-31T02:35:37.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:36 smithi045 conmon[33655]: osdmap e1541: 8 total, 8 up, 8 in 2021-07-31T02:35:38.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:37 smithi154 conmon[34553]: cluster 2021-07-31T02:35:36.494139+0000 mgr.x (mgr.24542) 2077 : cluster [DBG] pgmap v2839: 41 pgs: 41 active+clean; 4.6 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:35:38.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:37 smithi154 conmon[34553]: cluster 2021-07-31T02:35: 2021-07-31T02:35:38.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:37 smithi154 conmon[34553]: 36.568775+0000 mon.a (mon.0) 2519 : cluster [DBG] osdmap e1542: 8 total, 8 up, 8 in 2021-07-31T02:35:38.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:37 smithi045 conmon[39718]: cluster 2021-07-31T02:35:36.494139 2021-07-31T02:35:38.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:37 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 2077 : cluster [DBG] pgmap v2839: 41 pgs: 41 active+clean; 4.6 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:35:38.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:37 smithi045 conmon[39718]: cluster 2021-07-31T02 2021-07-31T02:35:38.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:37 smithi045 conmon[39718]: :35:36.568775+0000 mon.a (mon.0) 2519 : cluster [DBG] 2021-07-31T02:35:38.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:37 smithi045 conmon[39718]: osdmap e1542: 8 total, 8 up, 8 in 2021-07-31T02:35:38.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:37 smithi045 conmon[33655]: cluster 2021-07-31T02:35:36.494139+0000 2021-07-31T02:35:38.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:37 smithi045 conmon[33655]: mgr.x (mgr.24542) 2077 : cluster [DBG] pgmap v2839: 41 pgs: 41 active+clean; 4.6 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:35:38.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:37 smithi045 conmon[33655]: cluster 2021-07-31T02:35:36.568775+0000 mon.a (mon 2021-07-31T02:35:38.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:37 smithi045 conmon[33655]: .0) 2519 : cluster [DBG] osdmap e1542: 8 total, 8 up, 8 in 2021-07-31T02:35:38.583 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestObjectMap.RefreshInvalidatesWhenCorrupt 2021-07-31T02:35:39.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:38 smithi154 conmon[34553]: cluster 2021-07-31T02:35:37.574915+0000 mon.a (mon.0) 2021-07-31T02:35:39.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:38 smithi154 conmon[34553]: 2520 : cluster [DBG] osdmap e1543: 8 total, 8 up, 8 in 2021-07-31T02:35:39.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:38 smithi154 conmon[34553]: audit 2021-07-31T02:35:37.575955+0000 mon.a (mon.0) 2521 : audit [INF] from='client.? 172.21.15.45:0/1233898306' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:35:39.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:38 smithi045 conmon[33655]: cluster 2021-07-31T02:35:37.574915+0000 mon.a (mon.0) 2021-07-31T02:35:39.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:38 smithi045 conmon[33655]: 2520 : cluster [DBG] osdmap e1543: 8 total, 8 up, 8 in 2021-07-31T02:35:39.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:38 smithi045 conmon[33655]: audit 2021-07-31T02:35:37.575955+0000 mon.a (mon.0) 2521 : audit [INF] from='client.? 172.21.15.45:0/1233898306' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:35:39.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:38 smithi045 conmon[39718]: cluster 2021-07-31T02:35:37.574915 2021-07-31T02:35:39.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:38 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2520 : cluster [DBG] osdmap e1543: 8 total, 8 up, 8 in 2021-07-31T02:35:39.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:38 smithi045 conmon[39718]: audit 2021-07-31T02:35:37.575955+0000 mon.a (mon. 2021-07-31T02:35:39.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:38 smithi045 conmon[39718]: 0) 2521 : audit [INF] from='client.? 172.21.15.45:0/1233898306' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:35:40.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:39 smithi154 conmon[34553]: cluster 2021-07-31T02:35:38.494509+0000 mgr.x (mgr.24542) 2078 : cluster [DBG] 2021-07-31T02:35:40.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:39 smithi154 conmon[34553]: pgmap v2842: 41 pgs: 32 unknown, 9 active+clean; 4.6 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:35:40.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:39 smithi154 conmon[34553]: audit 2021-07-31T02:35:38.581591+0000 mon.a (mon.0) 2522 : audit [INF] from='client.? 172.21.15.45:0/1233898306' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-16","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:35:40.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:39 smithi154 conmon[34553]: cluster 2021-07-31T02:35:38.581801+0000 mon.a (mon 2021-07-31T02:35:40.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:39 smithi154 conmon[34553]: .0) 2523 : cluster [DBG] osdmap e1544: 8 total, 8 up, 8 in 2021-07-31T02:35:40.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:39 smithi045 conmon[33655]: cluster 2021-07-31T02:35:38.494509+0000 mgr.x (mgr.24542) 2078 : cluster [DBG] pgmap v2842: 41 pgs: 32 unknown, 9 active+clean; 4.6 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:35:40.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:39 smithi045 conmon[33655]: audit 2021-07-31T02:35:38.581591+0000 mon.a (mon.0) 2522 : audit [INF] from='client.? 172.21.15.45:0/1233898306' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-16","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:35:40.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:39 smithi045 conmon[33655]: cluster 2021-07-31T02:35:38.581801+0000 mon.a (mon.0) 2523 : cluster [DBG] osdmap e1544: 8 total, 8 up, 8 in 2021-07-31T02:35:40.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:39 smithi045 conmon[39718]: cluster 2021-07-31T02:35:38.494509+0000 mgr.x 2021-07-31T02:35:40.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:39 smithi045 conmon[39718]: (mgr.24542) 2078 : cluster [DBG] pgmap v2842: 41 pgs: 32 unknown, 9 active+clean; 4.6 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:35:40.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:39 smithi045 conmon[39718]: audit 2021 2021-07-31T02:35:40.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:39 smithi045 conmon[39718]: -07-31T02:35:38.581591+0000 mon.a (mon.0 2021-07-31T02:35:40.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:39 smithi045 conmon[39718]: ) 2522 : audit [INF] from='client.? 172.21.15.45:0/1233898306' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-16","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:35:40.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:39 smithi045 conmon[39718]: cluster 2021-07-31T02: 2021-07-31T02:35:40.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:39 smithi045 conmon[39718]: 35:38.581801+0000 mon.a (mon.0) 2523 : cluster [DBG] 2021-07-31T02:35:40.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:39 smithi045 conmon[39718]: osdmap e1544: 8 total, 8 up, 8 in 2021-07-31T02:35:40.602 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:35:40.602 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestObjectMap.RefreshInvalidatesWhenCorrupt (2018 ms) 2021-07-31T02:35:40.602 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestObjectMap.RefreshInvalidatesWhenTooSmall 2021-07-31T02:35:40.610 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:35:40.611 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestObjectMap.RefreshInvalidatesWhenTooSmall (9 ms) 2021-07-31T02:35:40.611 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestObjectMap.InvalidateFlagOnDisk 2021-07-31T02:35:40.615 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:35:40.615 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestObjectMap.InvalidateFlagOnDisk (4 ms) 2021-07-31T02:35:40.615 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestObjectMap.AcquireLockInvalidatesWhenTooSmall 2021-07-31T02:35:40.619 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:35:40.619 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestObjectMap.AcquireLockInvalidatesWhenTooSmall (4 ms) 2021-07-31T02:35:41.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:40 smithi154 conmon[34553]: cluster 2021-07-31T02:35:41.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:40 smithi154 conmon[34553]: 2021-07-31T02:35:39.582976+0000 mon.a ( 2021-07-31T02:35:41.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:40 smithi154 conmon[34553]: mon.0) 2524 : cluster [DBG] osdmap e1545: 8 total, 8 up, 8 in 2021-07-31T02:35:41.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:40 smithi045 conmon[33655]: cluster 2021-07-31T02:35:39. 2021-07-31T02:35:41.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:40 smithi045 conmon[33655]: 582976+0000 mon.a (mon.0) 2524 : cluster [DBG] osdmap e1545: 8 total, 8 up, 8 in 2021-07-31T02:35:41.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:40 smithi045 conmon[39718]: cluster 2021-07-31T02:35:39.582976+0000 mon.a ( 2021-07-31T02:35:41.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:40 smithi045 conmon[39718]: mon.0) 2524 : cluster [DBG] osdmap e1545: 8 total, 8 up, 8 in 2021-07-31T02:35:41.599 INFO:tasks.workunit.client.0.smithi045.stdout:[----------] 4 tests from TestObjectMap (2035 ms total) 2021-07-31T02:35:41.599 INFO:tasks.workunit.client.0.smithi045.stdout: 2021-07-31T02:35:41.599 INFO:tasks.workunit.client.0.smithi045.stdout:[----------] 1 test from TestOperations 2021-07-31T02:35:42.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:41 smithi154 conmon[34553]: cluster 2021-07-31T02:35:40.495095+0000 mgr.x (mgr.24542) 2079 : cluster [DBG] 2021-07-31T02:35:42.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:41 smithi154 conmon[34553]: pgmap v2845: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:35:42.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:41 smithi154 conmon[34553]: cluster 2021-07-31T02:35:40.585073+0000 mon.a (mon.0) 2525 : cluster [DBG] osdmap e1546: 8 total, 8 up, 8 in 2021-07-31T02:35:42.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:41 smithi045 conmon[39718]: cluster 2021-07-31T02:35:40.495095+0000 mgr.x ( 2021-07-31T02:35:42.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:41 smithi045 conmon[39718]: mgr.24542) 2079 : cluster [DBG] pgmap v2845: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:35:42.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:41 smithi045 conmon[39718]: cluster 2021-07-31T02:35:40.585073+0000 mon.a (mon.0) 2021-07-31T02:35:42.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:41 smithi045 conmon[39718]: 2525 : cluster [DBG] osdmap e1546: 8 total, 8 up, 8 in 2021-07-31T02:35:42.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:41 smithi045 conmon[33655]: cluster 2021-07-31T02:35:40.495095+0000 mgr.x (mgr.24542) 2079 : cluster [DBG] pgmap v2845: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:35:42.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:41 smithi045 conmon[33655]: cluster 2021-07-31T02:35:40.585073+0000 mon.a (mon.0) 2525 : cluster [DBG] osdmap e1546: 8 total, 8 up, 8 in 2021-07-31T02:35:43.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:42 smithi154 conmon[34553]: cluster 2021-07-31T02:35:41.596655+0000 mon.a (mon.0) 2526 : cluster [DBG] osdmap e1547: 8 total, 8 up, 8 in 2021-07-31T02:35:43.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:42 smithi154 conmon[34553]: 2021-07-31T02:35:43.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:42 smithi045 conmon[33655]: cluster 2021-07-31T02:35:41.596655+0000 mon.a (mon 2021-07-31T02:35:43.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:42 smithi045 conmon[33655]: .0) 2526 : cluster [DBG] osdmap e1547: 8 total, 8 up, 8 in 2021-07-31T02:35:43.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:42 smithi045 conmon[39718]: cluster 2021-07-31T02:35:41.596655+0000 mon.a (mon.0) 2526 : cluster [DBG] osdmap e1547: 8 total, 8 up, 8 in 2021-07-31T02:35:43.607 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestOperations.DisableJournalingCorrupt 2021-07-31T02:35:44.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:43 smithi154 conmon[34553]: cluster 2021-07-31T02:35:42.495516+0000 mgr.x (mgr.24542) 2080 2021-07-31T02:35:44.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:43 smithi154 conmon[34553]: : cluster [DBG] pgmap v2848: 9 pgs: 9 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:35:44.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:43 smithi154 conmon[34553]: cluster 2021-07-31T02:35:42.603294+0000 mon.a (mon.0) 2527 : cluster [DBG] osdmap e1548: 8 total, 8 up, 8 in 2021-07-31T02:35:44.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:43 smithi154 conmon[34553]: audit 2021-07-31T02:35:42.605606+0000 2021-07-31T02:35:44.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:43 smithi154 conmon[34553]: mon.c (mon.1) 121 : audit [INF] from='client.? 172.21.15.45:0/2160868905' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:35:44.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:43 smithi154 conmon[34553]: audit 2021-07-31T02:35:42.606479+0000 mon.a (mon.0) 2528 2021-07-31T02:35:44.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:43 smithi154 conmon[34553]: : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:35:44.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[39718]: cluster 2021-07-31T02:35:44.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[39718]: 2021-07-31T02:35:44.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[39718]: 2021 2021-07-31T02:35:44.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[39718]: - 2021-07-31T02:35:44.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[39718]: 07 2021-07-31T02:35:44.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[39718]: - 2021-07-31T02:35:44.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[39718]: 31 2021-07-31T02:35:44.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[39718]: T 2021-07-31T02:35:44.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[39718]: 02 2021-07-31T02:35:44.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[39718]: : 2021-07-31T02:35:44.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[39718]: 35 2021-07-31T02:35:44.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[39718]: : 2021-07-31T02:35:44.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[39718]: 42 2021-07-31T02:35:44.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[39718]: . 2021-07-31T02:35:44.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[39718]: 495516 2021-07-31T02:35:44.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[39718]: +0000 2021-07-31T02:35:44.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[39718]: 2021-07-31T02:35:44.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[39718]: mgr.x 2021-07-31T02:35:44.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[39718]: ( 2021-07-31T02:35:44.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[39718]: mgr 2021-07-31T02:35:44.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[39718]: .24542) 2080 : cluster [DBG] pgmap v2848: 9 pgs: 9 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:35:44.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[39718]: cluster 2021-07-31T02:35:42 2021-07-31T02:35:44.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[39718]: .603294+0000 mon.a (mon.0) 2527 : cluster [DBG] osdmap e1548: 8 total, 8 up, 8 in 2021-07-31T02:35:44.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[39718]: audit 2021-07-31T02:35:44.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[39718]: 2021-07-31T02:35:42.605606+0000 2021-07-31T02:35:44.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[39718]: mon.c (mon.1) 121 : audit [INF] 2021-07-31T02:35:44.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[39718]: from='client.? 172.21.15.45:0/2160868905' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:35:44.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[39718]: audit 2021-07-31T02:35 2021-07-31T02:35:44.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[39718]: :42.606479+0000 mon.a (mon.0) 2528 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:35:44.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: cluster 2021-07-31T02:35:44.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: 2021-07-31T02:35:44.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: 2021 2021-07-31T02:35:44.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: - 2021-07-31T02:35:44.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: 07 2021-07-31T02:35:44.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: - 2021-07-31T02:35:44.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: 31 2021-07-31T02:35:44.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: T 2021-07-31T02:35:44.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: 02 2021-07-31T02:35:44.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: : 2021-07-31T02:35:44.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: 35 2021-07-31T02:35:44.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: : 2021-07-31T02:35:44.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: 42 2021-07-31T02:35:44.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: . 2021-07-31T02:35:44.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: 495516 2021-07-31T02:35:44.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: +0000 2021-07-31T02:35:44.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: 2021-07-31T02:35:44.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: mgr.x 2021-07-31T02:35:44.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: ( 2021-07-31T02:35:44.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: mgr 2021-07-31T02:35:44.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: . 2021-07-31T02:35:44.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: 24542 2021-07-31T02:35:44.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: ) 2021-07-31T02:35:44.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: 2080 2021-07-31T02:35:44.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: : 2021-07-31T02:35:44.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: cluster 2021-07-31T02:35:44.040 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: 2021-07-31T02:35:44.040 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: [DBG] 2021-07-31T02:35:44.040 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: 2021-07-31T02:35:44.040 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: pgmap v2848: 9 pgs: 9 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:35:44.040 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: 2021-07-31T02:35:44.041 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: cluster 2021-07-31T02:35:44.041 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: 2021-07-31T02:35:44.041 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: 2021 2021-07-31T02:35:44.042 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: - 2021-07-31T02:35:44.042 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: 07 2021-07-31T02:35:44.042 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: - 2021-07-31T02:35:44.042 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: 31 2021-07-31T02:35:44.042 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: T 2021-07-31T02:35:44.043 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: 02 2021-07-31T02:35:44.044 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: : 2021-07-31T02:35:44.044 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: 35 2021-07-31T02:35:44.044 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: : 2021-07-31T02:35:44.045 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: 42 2021-07-31T02:35:44.045 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: . 2021-07-31T02:35:44.045 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: 603294 2021-07-31T02:35:44.046 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: +0000 2021-07-31T02:35:44.046 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: 2021-07-31T02:35:44.046 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: mon.a 2021-07-31T02:35:44.047 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: ( 2021-07-31T02:35:44.047 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: mon 2021-07-31T02:35:44.047 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: . 2021-07-31T02:35:44.047 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: 0 2021-07-31T02:35:44.047 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: ) 2021-07-31T02:35:44.048 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: 2527 2021-07-31T02:35:44.048 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: : 2021-07-31T02:35:44.048 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: cluster 2021-07-31T02:35:44.048 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: 2021-07-31T02:35:44.048 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: [DBG] 2021-07-31T02:35:44.049 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: 2021-07-31T02:35:44.049 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: osdmap e1548: 8 total, 8 up, 8 in 2021-07-31T02:35:44.049 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: 2021-07-31T02:35:44.049 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: audit 2021-07-31T02:35:44.050 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: 2021-07-31T02:35:44.050 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: 2021 2021-07-31T02:35:44.050 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: - 2021-07-31T02:35:44.050 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: 07 2021-07-31T02:35:44.050 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: - 2021-07-31T02:35:44.051 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: 31 2021-07-31T02:35:44.052 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: T 2021-07-31T02:35:44.052 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: 02 2021-07-31T02:35:44.052 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: : 2021-07-31T02:35:44.052 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: 35 2021-07-31T02:35:44.053 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: : 2021-07-31T02:35:44.053 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: 42 2021-07-31T02:35:44.053 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: . 2021-07-31T02:35:44.053 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: 605606 2021-07-31T02:35:44.054 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: +0000 2021-07-31T02:35:44.054 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: 2021-07-31T02:35:44.054 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: mon.c 2021-07-31T02:35:44.054 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: ( 2021-07-31T02:35:44.055 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: mon 2021-07-31T02:35:44.055 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: . 2021-07-31T02:35:44.055 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: 1 2021-07-31T02:35:44.055 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: ) 2021-07-31T02:35:44.055 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: 121 2021-07-31T02:35:44.056 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: : 2021-07-31T02:35:44.056 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: audit 2021-07-31T02:35:44.056 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: 2021-07-31T02:35:44.057 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: [INF] 2021-07-31T02:35:44.057 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: 2021-07-31T02:35:44.057 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: from='client.? 172.21.15.45:0/2160868905' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:35:44.057 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: 2021-07-31T02:35:44.057 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: audit 2021-07-31T02:35:44.058 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: 2021-07-31T02:35:44.058 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: 2021 2021-07-31T02:35:44.058 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: - 2021-07-31T02:35:44.058 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: 07 2021-07-31T02:35:44.059 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: - 2021-07-31T02:35:44.059 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: 31 2021-07-31T02:35:44.059 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: T 2021-07-31T02:35:44.059 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: 02 2021-07-31T02:35:44.059 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: : 2021-07-31T02:35:44.060 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: 35 2021-07-31T02:35:44.060 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: : 2021-07-31T02:35:44.060 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: 42 2021-07-31T02:35:44.060 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: . 2021-07-31T02:35:44.060 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: 606479 2021-07-31T02:35:44.061 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: +0000 2021-07-31T02:35:44.061 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: 2021-07-31T02:35:44.061 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: mon.a 2021-07-31T02:35:44.061 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: ( 2021-07-31T02:35:44.062 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: mon 2021-07-31T02:35:44.062 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: . 2021-07-31T02:35:44.062 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: 0 2021-07-31T02:35:44.062 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: ) 2021-07-31T02:35:44.062 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: 2528 2021-07-31T02:35:44.063 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: : 2021-07-31T02:35:44.063 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: audit 2021-07-31T02:35:44.063 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: 2021-07-31T02:35:44.063 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: [INF] 2021-07-31T02:35:44.063 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: 2021-07-31T02:35:44.064 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:35:44.064 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:43 smithi045 conmon[33655]: 2021-07-31T02:35:45.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:44 smithi154 conmon[34553]: audit 2021-07-31T02:35:43.605870+0000 mon.a (mon. 2021-07-31T02:35:45.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:44 smithi154 conmon[34553]: 0) 2529 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-17","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:35:45.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:44 smithi154 conmon[34553]: cluster 2021-07-31T02:35:43.606098+0000 mon.a (mon.0) 2530 : cluster [DBG] osdmap e1549: 8 total, 8 up, 8 in 2021-07-31T02:35:45.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:44 smithi045 conmon[39718]: audit 2021-07-31T02:35:43.605870+0000 2021-07-31T02:35:45.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:44 smithi045 conmon[39718]: mon.a (mon.0) 2529 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-17","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:35:45.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:44 smithi045 conmon[39718]: cluster 2021-07-31T 2021-07-31T02:35:45.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:44 smithi045 conmon[39718]: 02:35:43.606098+0000 2021-07-31T02:35:45.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:44 smithi045 conmon[39718]: mon.a (mon.0) 2530 : 2021-07-31T02:35:45.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:44 smithi045 conmon[39718]: cluster [DBG] osdmap e1549: 8 total, 8 up, 8 in 2021-07-31T02:35:45.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:44 smithi045 conmon[33655]: audit 2021-07-31T02:35:43.605870+0000 mon.a (mon.0) 2021-07-31T02:35:45.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:44 smithi045 conmon[33655]: 2529 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-17","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:35:45.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:44 smithi045 conmon[33655]: cluster 2021-07-31T02:35:43.606098+0000 mon.a (mon.0) 2530 2021-07-31T02:35:45.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:44 smithi045 conmon[33655]: : cluster [DBG] osdmap e1549: 8 total, 8 up, 8 in 2021-07-31T02:35:45.634 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:35:45.635 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestOperations.DisableJournalingCorrupt (2028 ms) 2021-07-31T02:35:46.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:45 smithi154 conmon[34553]: cluster 2021-07-31T02:35:44.496025+0000 mgr.x (mgr.24542) 2021-07-31T02:35:46.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:45 smithi154 conmon[34553]: 2081 : cluster [DBG] pgmap v2851: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:35:46.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:45 smithi154 conmon[34553]: cluster 2021-07-31T02:35:44.618245+0000 mon.a (mon.0) 2531 : cluster [DBG] osdmap e1550: 8 total, 8 up, 8 in 2021-07-31T02:35:46.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:45 smithi045 conmon[39718]: cluster 2021-07-31T02:35:44.496025+0000 mgr.x (mgr.24542) 2081 : 2021-07-31T02:35:46.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:45 smithi045 conmon[39718]: cluster [DBG] pgmap v2851: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:35:46.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:45 smithi045 conmon[39718]: cluster 2021-07-31T02:35:44.618245+0000 mon.a ( 2021-07-31T02:35:46.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:45 smithi045 conmon[39718]: mon.0) 2531 : cluster [DBG] osdmap e1550: 8 total, 8 up, 8 in 2021-07-31T02:35:46.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:45 smithi045 conmon[33655]: cluster 2021-07-31T02:35:46.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:45 smithi045 conmon[33655]: 2021-07-31T02:35:46.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:45 smithi045 conmon[33655]: 2021 2021-07-31T02:35:46.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:45 smithi045 conmon[33655]: - 2021-07-31T02:35:46.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:45 smithi045 conmon[33655]: 07 2021-07-31T02:35:46.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:45 smithi045 conmon[33655]: - 2021-07-31T02:35:46.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:45 smithi045 conmon[33655]: 31 2021-07-31T02:35:46.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:45 smithi045 conmon[33655]: T02:35:44.496025+0000 mgr.x (mgr.24542) 2081 : cluster [DBG] pgmap v2851: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:35:46.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:45 smithi045 conmon[33655]: cluster 2021-07-31T02:35:44.618245+0000 mon.a ( 2021-07-31T02:35:46.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:45 smithi045 conmon[33655]: mon.0) 2531 : cluster [DBG] osdmap e1550: 8 total, 8 up, 8 in 2021-07-31T02:35:46.639 INFO:tasks.workunit.client.0.smithi045.stdout:[----------] 1 test from TestOperations (2028 ms total) 2021-07-31T02:35:46.639 INFO:tasks.workunit.client.0.smithi045.stdout: 2021-07-31T02:35:46.640 INFO:tasks.workunit.client.0.smithi045.stdout:[----------] 2 tests from TestTrash 2021-07-31T02:35:47.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:46 smithi154 conmon[34553]: cluster 2021-07-31T02:35:45.619716+0000 mon.a (mon. 2021-07-31T02:35:47.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:46 smithi154 conmon[34553]: 0) 2532 : cluster [DBG] osdmap e1551: 8 total, 8 up, 8 in 2021-07-31T02:35:47.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:46 smithi045 conmon[33655]: cluster 2021-07-31T02:35:45.619716+0000 mon.a (mon. 2021-07-31T02:35:47.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:46 smithi045 conmon[33655]: 0) 2532 : cluster [DBG] osdmap e1551: 8 total, 8 up, 8 in 2021-07-31T02:35:47.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:46 smithi045 conmon[39718]: cluster 2021-07-31T02:35:45.619716+0000 mon.a (mon 2021-07-31T02:35:47.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:46 smithi045 conmon[39718]: .0) 2532 : cluster [DBG] osdmap e1551: 8 total, 8 up, 8 in 2021-07-31T02:35:48.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:47 smithi154 conmon[34553]: cluster 2021-07-31T02:35:46.496498+0000 mgr.x (mgr.24542) 2082 : cluster 2021-07-31T02:35:48.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:47 smithi154 conmon[34553]: [DBG] pgmap v2854: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:35:48.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:47 smithi154 conmon[34553]: cluster 2021-07-31T02:35:46.634942+0000 mon.a (mon.0) 2533 : cluster [DBG] osdmap e1552: 8 total, 8 up, 8 in 2021-07-31T02:35:48.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:47 smithi045 conmon[39718]: cluster 2021-07-31T02:35:46. 2021-07-31T02:35:48.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:47 smithi045 conmon[39718]: 496498+0000 mgr.x (mgr.24542) 2082 : cluster [DBG] pgmap v2854: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:35:48.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:47 smithi045 conmon[39718]: cluster 2021-07-31T02:35:48.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:47 smithi045 conmon[39718]: 2021-07-31T02:35:46.634942+0000 mon.a (mon.0) 2021-07-31T02:35:48.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:47 smithi045 conmon[39718]: 2533 : cluster [DBG] osdmap e1552: 8 total, 8 up, 8 in 2021-07-31T02:35:48.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:47 smithi045 conmon[33655]: cluster 2021-07-31T02:35:46.496498+0000 2021-07-31T02:35:48.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:47 smithi045 conmon[33655]: mgr.x (mgr.24542) 2082 : cluster [DBG] pgmap v2854: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:35:48.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:47 smithi045 conmon[33655]: cluster 2021-07-31T02:35:46.634942+0000 mon.a (mon.0) 2533 : cluster [DBG] osdmap e1552: 8 total, 8 up, 8 in 2021-07-31T02:35:48.649 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestTrash.UserRemovingSource 2021-07-31T02:35:49.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:48 smithi154 conmon[34553]: cluster 2021-07-31T02:35:47.638941+0000 mon.a (mon.0) 2534 : cluster 2021-07-31T02:35:49.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:48 smithi154 conmon[34553]: [DBG] osdmap e1553: 8 total, 8 up, 8 in 2021-07-31T02:35:49.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:48 smithi154 conmon[34553]: audit 2021-07-31T02:35:47.639885+0000 mon.a (mon.0) 2535 : audit [INF] from='client.? 172.21.15.45:0/70985797' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:35:49.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:48 smithi045 conmon[39718]: cluster 2021-07-31T02:35:47.638941+0000 mon.a (mon 2021-07-31T02:35:49.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:48 smithi045 conmon[39718]: .0) 2534 : cluster [DBG] osdmap e1553: 8 total, 8 up, 8 in 2021-07-31T02:35:49.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:48 smithi045 conmon[39718]: audit 2021-07-31T02:35:47.639885+0000 mon.a (mon.0) 2535 : audit 2021-07-31T02:35:49.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:48 smithi045 conmon[39718]: [INF] from='client.? 172.21.15.45:0/70985797' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:35:49.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:48 smithi045 conmon[33655]: cluster 2021-07-31T02:35:47.638941+0000 mon.a (mon. 2021-07-31T02:35:49.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:48 smithi045 conmon[33655]: 0) 2534 : cluster [DBG] osdmap e1553: 8 total, 8 up, 8 in 2021-07-31T02:35:49.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:48 smithi045 conmon[33655]: audit 2021-07-31T02:35:47.639885+0000 mon.a (mon.0) 2535 : audit [INF] from='client.? 172.21.15.45:0/70985797' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:35:50.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:49 smithi154 conmon[34553]: cluster 2021-07-31T02:35:48.496931+0000 mgr.x (mgr.24542) 2083 : cluster [DBG] pgmap v2857: 41 pgs: 32 unknown, 9 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:35:50.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:49 smithi154 conmon[34553]: audit 2021-07-31T02:35:48.647237+0000 mon.a (mon.0) 2536 : audit [INF] from='client.? 172.21.15.45:0/70985797' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-18","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:35:50.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:49 smithi154 conmon[34553]: cluster 2021-07-31T02:35:48.647524+0000 mon.a (mon.0) 2537 : cluster [DBG] osdmap e1554: 8 total, 8 up, 8 in 2021-07-31T02:35:50.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:49 smithi045 conmon[39718]: cluster 2021-07-31T02:35:48.496931+0000 mgr.x 2021-07-31T02:35:50.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:49 smithi045 conmon[39718]: (mgr.24542) 2083 : cluster [DBG] pgmap v2857: 41 pgs: 32 unknown, 9 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:35:50.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:49 smithi045 conmon[39718]: audit 2021-07-31T02:35: 2021-07-31T02:35:50.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:49 smithi045 conmon[39718]: 48.647237+0000 mon.a (mon.0) 2536 : audit [INF] from='client.? 172.21.15.45:0/70985797' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-18","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:35:50.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:49 smithi045 conmon[39718]: cluster 2021 2021-07-31T02:35:50.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:49 smithi045 conmon[39718]: -07-31T02:35:48.647524+0000 mon.a ( 2021-07-31T02:35:50.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:49 smithi045 conmon[39718]: mon.0) 2537 : cluster [DBG] osdmap e1554: 8 total, 8 up, 8 in 2021-07-31T02:35:50.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:49 smithi045 conmon[33655]: cluster 2021-07-31T02:35:48.496931+0000 mgr.x (mgr.24542) 2083 : cluster [DBG] pgmap v2857: 41 pgs: 32 unknown, 9 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:35:50.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:49 smithi045 conmon[33655]: audit 2021-07-31T02:35:48.647237+0000 mon.a (mon.0) 2536 : audit [INF] from='client.? 172.21.15.45:0/70985797' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-18","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:35:50.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:49 smithi045 conmon[33655]: cluster 2021-07-31T02:35:48. 2021-07-31T02:35:50.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:49 smithi045 conmon[33655]: 647524+0000 mon.a (mon.0) 2537 : cluster [DBG] osdmap e1554: 8 total, 8 up, 8 in 2021-07-31T02:35:50.661 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:35:50.661 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestTrash.UserRemovingSource (2012 ms) 2021-07-31T02:35:50.662 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestTrash.RestoreMirroringSource 2021-07-31T02:35:50.671 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:35:50.671 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestTrash.RestoreMirroringSource (10 ms) 2021-07-31T02:35:51.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:50 smithi154 conmon[34553]: cluster 2021-07-31T02:35:49.647869+0000 mon.a (mon.0) 2538 : 2021-07-31T02:35:51.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:50 smithi154 conmon[34553]: cluster [DBG] osdmap e1555: 8 total, 8 up, 8 in 2021-07-31T02:35:51.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:50 smithi154 conmon[34553]: cluster 2021-07-31T02:35:50.649722+0000 mon.a (mon.0) 2539 : cluster [DBG] osdmap e1556: 8 total, 8 up, 8 in 2021-07-31T02:35:51.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:50 smithi045 conmon[39718]: cluster 2021-07-31T02:35:49 2021-07-31T02:35:51.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:50 smithi045 conmon[39718]: .647869+0000 mon.a (mon.0) 2538 : cluster [DBG] osdmap e1555: 8 total, 8 up, 8 in 2021-07-31T02:35:51.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:50 smithi045 conmon[39718]: cluster 2021-07-31T02:35:51.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:50 smithi045 conmon[39718]: 2021-07-31T02:35:50.649722+0000 mon.a (mon.0) 2539 : cluster 2021-07-31T02:35:51.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:50 smithi045 conmon[39718]: [DBG] osdmap e1556: 8 total, 8 up, 8 in 2021-07-31T02:35:51.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:50 smithi045 conmon[33655]: cluster 2021-07-31T02:35:49.647869+0000 mon.a (mon.0) 2538 : cluster [DBG] osdmap e1555: 8 total, 8 up, 8 in 2021-07-31T02:35:51.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:50 smithi045 conmon[33655]: cluster 2021-07-31T02:35:50.649722+0000 mon.a (mon.0) 2539 : cluster [DBG] osdmap e1556: 8 total, 8 up, 8 in 2021-07-31T02:35:51.665 INFO:tasks.workunit.client.0.smithi045.stdout:[----------] 2 tests from TestTrash (2022 ms total) 2021-07-31T02:35:51.665 INFO:tasks.workunit.client.0.smithi045.stdout: 2021-07-31T02:35:51.665 INFO:tasks.workunit.client.0.smithi045.stdout:[----------] 3 tests from TestJournalEntries 2021-07-31T02:35:52.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:51 smithi154 conmon[34553]: cluster 2021-07-31T02:35:50.497406+0000 mgr.x (mgr.24542) 2084 : cluster [DBG] pgmap v2860: 41 pgs: 12 creating+activating, 29 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:35:52.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:51 smithi154 conmon[34553]: 2021-07-31T02:35:52.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:51 smithi045 conmon[39718]: cluster 2021-07-31T02: 2021-07-31T02:35:52.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:51 smithi045 conmon[39718]: 35:50.497406+0000 mgr.x (mgr.24542) 2084 : cluster [DBG] pgmap v2860: 41 pgs: 12 creating+activating, 29 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:35:52.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:51 smithi045 conmon[33655]: cluster 2021-07-31T02:35:52.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:51 smithi045 conmon[33655]: 2021-07-31T02:35:52.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:51 smithi045 conmon[33655]: 2021 2021-07-31T02:35:52.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:51 smithi045 conmon[33655]: - 2021-07-31T02:35:52.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:51 smithi045 conmon[33655]: 07 2021-07-31T02:35:52.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:51 smithi045 conmon[33655]: - 2021-07-31T02:35:52.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:51 smithi045 conmon[33655]: 31 2021-07-31T02:35:52.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:51 smithi045 conmon[33655]: T 2021-07-31T02:35:52.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:51 smithi045 conmon[33655]: 02 2021-07-31T02:35:52.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:51 smithi045 conmon[33655]: : 2021-07-31T02:35:52.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:51 smithi045 conmon[33655]: 35 2021-07-31T02:35:52.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:51 smithi045 conmon[33655]: : 2021-07-31T02:35:52.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:51 smithi045 conmon[33655]: 50 2021-07-31T02:35:52.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:51 smithi045 conmon[33655]: . 2021-07-31T02:35:52.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:51 smithi045 conmon[33655]: 497406 2021-07-31T02:35:52.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:51 smithi045 conmon[33655]: +0000 2021-07-31T02:35:52.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:51 smithi045 conmon[33655]: 2021-07-31T02:35:52.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:51 smithi045 conmon[33655]: mgr.x 2021-07-31T02:35:52.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:51 smithi045 conmon[33655]: ( 2021-07-31T02:35:52.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:51 smithi045 conmon[33655]: mgr 2021-07-31T02:35:52.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:51 smithi045 conmon[33655]: . 2021-07-31T02:35:52.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:51 smithi045 conmon[33655]: 24542 2021-07-31T02:35:52.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:51 smithi045 conmon[33655]: ) 2084 : cluster [DBG] pgmap v2860: 41 pgs: 12 creating+activating, 29 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:35:53.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:52 smithi154 conmon[34553]: cluster 2021-07-31T02:35:51.662519+0000 mon.a (mon.0) 2540 : cluster [DBG] osdmap e1557: 8 total, 8 up, 8 in 2021-07-31T02:35:53.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:52 smithi154 conmon[34553]: 2021-07-31T02:35:53.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:52 smithi045 conmon[33655]: cluster 2021-07-31T02:35:51.662519+0000 mon.a (mon.0) 2540 2021-07-31T02:35:53.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:52 smithi045 conmon[33655]: : cluster [DBG] osdmap e1557: 8 total, 8 up, 8 in 2021-07-31T02:35:53.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:52 smithi045 conmon[39718]: cluster 2021-07-31T02:35:51.662519+0000 mon.a (mon.0) 2540 : cluster [DBG] osdmap e1557: 8 total, 8 up, 8 in 2021-07-31T02:35:53.679 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestJournalEntries.AioWrite 2021-07-31T02:35:54.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:53 smithi154 conmon[34553]: cluster 2021-07-31T02:35:52.497822+0000 mgr.x (mgr 2021-07-31T02:35:54.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:53 smithi154 conmon[34553]: .24542) 2085 : cluster [DBG] pgmap v2863: 9 pgs: 9 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:35:54.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:53 smithi154 conmon[34553]: cluster 2021-07-31T02:35:52.677205+0000 mon.a (mon.0) 2541 : cluster [DBG] osdmap e1558: 8 total, 8 up, 8 in 2021-07-31T02:35:54.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:53 smithi154 conmon[34553]: audit 2021-07-31T02:35:52.678227+0000 mon.c (mon.1) 122 : audit [INF] from='client.? 172.21.15.45:0/4191734692' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:35:54.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:53 smithi154 conmon[34553]: audit 2021-07-31T02:35:52.678760+0000 mon.a (mon.0) 2542 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:35:54.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:53 smithi045 conmon[39718]: cluster 2021-07-31T02:35:52.497822+0000 2021-07-31T02:35:54.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:53 smithi045 conmon[39718]: mgr.x (mgr.24542) 2085 : cluster [DBG] pgmap v2863: 9 pgs: 9 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:35:54.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:53 smithi045 conmon[39718]: cluster 2021-07- 2021-07-31T02:35:54.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:53 smithi045 conmon[39718]: 31T02:35:52.677205+0000 mon.a (mon.0) 2541 : cluster 2021-07-31T02:35:54.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:53 smithi045 conmon[39718]: [DBG] osdmap e1558: 8 total, 8 up, 8 in 2021-07-31T02:35:54.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:53 smithi045 conmon[39718]: audit 2021-07 2021-07-31T02:35:54.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:53 smithi045 conmon[39718]: -31T02:35:52.678227+0000 mon.c 2021-07-31T02:35:54.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:53 smithi045 conmon[39718]: (mon.1) 122 : audit [INF] from='client.? 172.21.15.45:0/4191734692' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:35:54.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:53 smithi045 conmon[39718]: 2021-07-31T02:35:54.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:53 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T02:35:54.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:53 smithi045 conmon[39718]: 02:35:52.678760+0000 mon.a (mon.0) 2542 2021-07-31T02:35:54.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:53 smithi045 conmon[39718]: : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:35:54.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:53 smithi045 conmon[33655]: cluster 2021-07-31T02:35:54.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:53 smithi045 conmon[33655]: 2021-07-31T02:35:54.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:53 smithi045 conmon[33655]: 2021-07-31T02:35:52.497822+0000 2021-07-31T02:35:54.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:53 smithi045 conmon[33655]: mgr.x (mgr.24542) 2085 : cluster [DBG] pgmap v2863: 9 pgs: 9 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:35:54.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:53 smithi045 conmon[33655]: cluster 2021-07 2021-07-31T02:35:54.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:53 smithi045 conmon[33655]: -31T02:35:52.677205+0000 mon.a (mon 2021-07-31T02:35:54.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:53 smithi045 conmon[33655]: .0) 2541 : cluster [DBG] osdmap e1558: 8 total, 8 up, 8 in 2021-07-31T02:35:54.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:53 smithi045 conmon[33655]: audit 2021-07- 2021-07-31T02:35:54.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:53 smithi045 conmon[33655]: 31T02:35:52.678227+0000 mon.c (mon.1) 122 : 2021-07-31T02:35:54.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:53 smithi045 conmon[33655]: audit [INF] from='client.? 172.21.15.45:0/4191734692' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:35:54.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:53 smithi045 conmon[33655]: audit 2021 2021-07-31T02:35:54.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:53 smithi045 conmon[33655]: -07-31T02:35:52.678760+0000 mon.a (mon. 2021-07-31T02:35:54.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:53 smithi045 conmon[33655]: 0) 2542 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:35:54.284 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:35:54.561 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:35:54.561 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 50m ago 51m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:35:54.561 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 48m ago 51m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:35:54.562 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 48m ago 54m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:35:54.562 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 50m ago 56m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:35:54.562 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 50m ago 57m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:35:54.563 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 48m ago 55m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:35:54.563 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 50m ago 55m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:35:54.563 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 50m ago 51m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:35:54.563 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 48m ago 51m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:35:54.563 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 50m ago 54m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:35:54.564 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 50m ago 54m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:35:54.564 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 50m ago 53m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:35:54.564 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 50m ago 53m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:35:54.564 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 48m ago 53m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:35:54.565 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 48m ago 52m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:35:54.565 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 48m ago 52m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:35:54.565 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 48m ago 52m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:35:54.565 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 48m ago 51m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:35:54.898 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:35:54.899 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:35:54.899 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:35:54.899 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:35:54.900 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:35:54.900 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:35:54.900 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:35:54.900 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:35:54.901 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:35:54.901 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:35:54.901 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:35:54.901 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:35:54.901 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:35:54.902 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:35:54.902 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:35:54.902 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:35:54.902 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:35:55.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:54 smithi154 conmon[34553]: audit 2021-07-31T02:35:53.678587+0000 mon.a (mon.0) 2543 : audit 2021-07-31T02:35:55.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:54 smithi154 conmon[34553]: [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-19","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:35:55.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:54 smithi154 conmon[34553]: cluster 2021-07-31T02:35:53.678939+0000 mon.a (mon.0) 2544 : cluster [DBG] osdmap e1559: 8 total, 8 up, 8 in 2021-07-31T02:35:55.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:54 smithi154 conmon[34553]: audit 2021-07-31T02:35:54.272101+0000 mgr.x (mgr.24542) 2086 : audit [DBG] from='client.27514 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:35:55.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:54 smithi045 conmon[39718]: audit 2021-07-31T02:35:53.678587+0000 2021-07-31T02:35:55.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:54 smithi045 conmon[39718]: mon.a (mon.0) 2543 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-19","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:35:55.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:54 smithi045 conmon[39718]: cluster 2021-07-31T02: 2021-07-31T02:35:55.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:54 smithi045 conmon[39718]: 35:53.678939+0000 mon.a (mon.0) 2544 : cluster [DBG] 2021-07-31T02:35:55.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:54 smithi045 conmon[39718]: osdmap e1559: 8 total, 8 up, 8 in 2021-07-31T02:35:55.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:54 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T02:35:55.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:54 smithi045 conmon[39718]: 31T02:35:54.272101+0000 mgr.x ( 2021-07-31T02:35:55.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:54 smithi045 conmon[39718]: mgr.24542) 2086 : audit [DBG] from='client.27514 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:35:55.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:54 smithi045 conmon[33655]: audit 2021-07-31T02:35:53.678587+0000 mon.a (mon.0) 2543 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-19","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:35:55.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:54 smithi045 conmon[33655]: cluster 2021-07-31T02:35:53.678939+0000 mon.a (mon.0) 2544 : cluster [DBG] osdmap e1559: 8 total, 8 up, 8 in 2021-07-31T02:35:55.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:54 smithi045 conmon[33655]: audit 2021-07-31T02:35:55.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:54 smithi045 conmon[33655]: 2021-07-31T02:35:54.272101+0000 mgr.x (mgr. 2021-07-31T02:35:55.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:54 smithi045 conmon[33655]: 24542) 2086 : audit [DBG] from='client.27514 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:35:55.711 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:35:55.712 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestJournalEntries.AioWrite (2032 ms) 2021-07-31T02:35:55.712 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestJournalEntries.AioDiscard 2021-07-31T02:35:55.718 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:35:55.718 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestJournalEntries.AioDiscard (6 ms) 2021-07-31T02:35:55.719 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestJournalEntries.AioFlush 2021-07-31T02:35:55.722 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:35:55.723 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestJournalEntries.AioFlush (5 ms) 2021-07-31T02:35:56.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:55 smithi154 conmon[34553]: cluster 2021-07-31T02:35:54.498393+0000 mgr.x (mgr.24542) 2087 : 2021-07-31T02:35:56.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:55 smithi154 conmon[34553]: cluster [DBG] pgmap v2866: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:35:56.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:55 smithi154 conmon[34553]: audit 2021-07-31T02:35:54.553108+0000 mgr.x (mgr.24542) 2088 : audit [DBG] from='client.17580 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:35:56.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:55 smithi154 conmon[34553]: cluster 2021- 2021-07-31T02:35:56.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:55 smithi154 conmon[34553]: 07-31T02:35:54.692675+0000 mon.a (mon.0) 2021-07-31T02:35:56.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:55 smithi154 conmon[34553]: 2545 : cluster [DBG] osdmap e1560: 8 total, 8 up, 8 in 2021-07-31T02:35:56.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:55 smithi154 conmon[34553]: audit 2021 2021-07-31T02:35:56.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:55 smithi154 conmon[34553]: -07-31T02:35:54.898880+0000 mon.a (mon 2021-07-31T02:35:56.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:55 smithi154 conmon[34553]: .0) 2546 : audit [DBG] from='client.? 172.21.15.45:0/3396928066' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:35:56.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:55 smithi045 conmon[39718]: cluster 2021-07-31T02:35:54. 2021-07-31T02:35:56.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:55 smithi045 conmon[39718]: 498393+0000 mgr.x (mgr.24542) 2087 : cluster [DBG] pgmap v2866: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:35:56.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:55 smithi045 conmon[39718]: audit 2021- 2021-07-31T02:35:56.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:55 smithi045 conmon[39718]: 07-31T02:35:54. 2021-07-31T02:35:56.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:55 smithi045 conmon[39718]: 553108+0000 mgr.x (mgr.24542 2021-07-31T02:35:56.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:55 smithi045 conmon[39718]: ) 2088 : audit [DBG] from='client.17580 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:35:56.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:55 smithi045 conmon[39718]: 2021-07-31T02:35:56.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:55 smithi045 conmon[39718]: cluster 2021- 2021-07-31T02:35:56.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:55 smithi045 conmon[39718]: 07-31T02:35:54.692675 2021-07-31T02:35:56.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:55 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2021-07-31T02:35:56.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:55 smithi045 conmon[39718]: 2545 : cluster [DBG] osdmap e1560: 8 total, 8 up, 8 in 2021-07-31T02:35:56.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:55 smithi045 conmon[39718]: audit 2021-07-31T02:35:56.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:55 smithi045 conmon[39718]: 2021-07-31T02:35: 2021-07-31T02:35:56.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:55 smithi045 conmon[39718]: 54.898880+0000 mon.a (mon 2021-07-31T02:35:56.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:55 smithi045 conmon[39718]: .0) 2546 : audit [DBG] 2021-07-31T02:35:56.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:55 smithi045 conmon[39718]: from='client.? 172.21.15.45:0/3396928066' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:35:56.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:55 smithi045 conmon[33655]: cluster 2021-07-31T02:35:54.498393+0000 2021-07-31T02:35:56.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:55 smithi045 conmon[33655]: mgr.x (mgr.24542) 2087 : cluster [DBG] pgmap v2866: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:35:56.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:55 smithi045 conmon[33655]: audit 2021-07-31T02:35:54.553108+0000 2021-07-31T02:35:56.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:55 smithi045 conmon[33655]: mgr.x (mgr.24542) 2088 : audit [DBG] from='client.17580 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:35:56.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:55 smithi045 conmon[33655]: cluster 2021-07 2021-07-31T02:35:56.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:55 smithi045 conmon[33655]: -31T02:35:54.692675+0000 2021-07-31T02:35:56.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:55 smithi045 conmon[33655]: mon.a (mon.0) 2545 2021-07-31T02:35:56.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:55 smithi045 conmon[33655]: : cluster [DBG] osdmap e1560: 8 total, 8 up, 8 in 2021-07-31T02:35:56.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:55 smithi045 conmon[33655]: audit 2021-07 2021-07-31T02:35:56.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:55 smithi045 conmon[33655]: -31T02:35:54.898880+0000 mon.a (mon 2021-07-31T02:35:56.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:55 smithi045 conmon[33655]: .0) 2546 : audit [DBG] from='client.? 172.21.15.45:0/3396928066' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:35:56.709 INFO:tasks.workunit.client.0.smithi045.stdout:[----------] 3 tests from TestJournalEntries (2043 ms total) 2021-07-31T02:35:56.709 INFO:tasks.workunit.client.0.smithi045.stdout: 2021-07-31T02:35:56.709 INFO:tasks.workunit.client.0.smithi045.stdout:[----------] 16 tests from TestJournalReplay 2021-07-31T02:35:57.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:56 smithi154 conmon[34553]: cluster 2021-07-31T02:35:55.699634+0000 mon.a (mon.0) 2547 2021-07-31T02:35:57.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:56 smithi154 conmon[34553]: : cluster [DBG] osdmap e1561: 8 total, 8 up, 8 in 2021-07-31T02:35:57.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:56 smithi045 conmon[39718]: cluster 2021-07-31T02:35:55.699634+0000 mon.a (mon.0) 2021-07-31T02:35:57.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:56 smithi045 conmon[39718]: 2547 : cluster [DBG] osdmap e1561: 8 total, 8 up, 8 in 2021-07-31T02:35:57.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:56 smithi045 conmon[33655]: cluster 2021-07-31T02:35:55.699634+0000 mon.a (mon.0) 2547 : cluster 2021-07-31T02:35:57.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:56 smithi045 conmon[33655]: [DBG] osdmap e1561: 8 total, 8 up, 8 in 2021-07-31T02:35:58.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:57 smithi154 conmon[34553]: cluster 2021-07-31T02:35:56.498799+0000 mgr.x (mgr.24542) 2089 : cluster [DBG] pgmap v2869: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:35:58.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:57 smithi154 conmon[34553]: cluster 2021-07-31T02:35:56.705947+0000 mon.a 2021-07-31T02:35:58.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:57 smithi154 conmon[34553]: (mon.0) 2548 : cluster [DBG] osdmap e1562: 8 total, 8 up, 8 in 2021-07-31T02:35:58.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:57 smithi045 conmon[39718]: cluster 2021-07-31T02:35:56.498799+0000 mgr.x (mgr.24542) 2089 : cluster [DBG] pgmap v2869: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:35:58.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:57 smithi045 conmon[39718]: cluster 2021-07-31T02:35:56.705947+0000 mon.a (mon.0) 2548 : cluster [DBG] osdmap e1562: 8 total, 8 up, 8 in 2021-07-31T02:35:58.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:57 smithi045 conmon[33655]: cluster 2021-07-31T02:35:56.498799+0000 2021-07-31T02:35:58.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:57 smithi045 conmon[33655]: mgr.x (mgr.24542) 2089 : cluster [DBG] pgmap v2869: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:35:58.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:57 smithi045 conmon[33655]: cluster 2021-07-31T02:35:56.705947+0000 2021-07-31T02:35:58.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:57 smithi045 conmon[33655]: mon.a (mon.0) 2548 : cluster [DBG] osdmap e1562: 8 total, 8 up, 8 in 2021-07-31T02:35:58.721 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestJournalReplay.AioDiscardEvent 2021-07-31T02:35:59.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:58 smithi154 conmon[34553]: cluster 2021-07-31T02:35:57.711599+0000 mon.a (mon.0) 2549 : cluster [DBG] osdmap e1563: 8 total, 8 up, 8 in 2021-07-31T02:35:59.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:58 smithi154 conmon[34553]: 2021-07-31T02:35:59.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:58 smithi154 conmon[34553]: audit 2021-07-31T02:35:57.713070+0000 mon.a (mon.0) 2550 : audit [INF] from='client.? 172.21.15.45:0/3933708616' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:35:59.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:58 smithi045 conmon[39718]: cluster 2021-07-31T02:35:57.711599+0000 2021-07-31T02:35:59.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:58 smithi045 conmon[39718]: mon.a (mon.0) 2549 : cluster [DBG] osdmap e1563: 8 total, 8 up, 8 in 2021-07-31T02:35:59.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:58 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T02:35:59.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:58 smithi045 conmon[39718]: 02:35:57.713070+0000 mon.a (mon.0) 2550 : audit [INF] 2021-07-31T02:35:59.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:58 smithi045 conmon[39718]: from='client.? 172.21.15.45:0/3933708616' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:35:59.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:58 smithi045 conmon[33655]: cluster 2021-07-31T02:35:57.711599+0000 mon.a (mon 2021-07-31T02:35:59.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:58 smithi045 conmon[33655]: .0) 2549 : cluster [DBG] osdmap e1563: 8 total, 8 up, 8 in 2021-07-31T02:35:59.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:58 smithi045 conmon[33655]: audit 2021-07-31T02:35:57.713070+0000 mon.a (mon.0) 2550 : 2021-07-31T02:35:59.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:58 smithi045 conmon[33655]: audit [INF] from='client.? 172.21.15.45:0/3933708616' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:36:00.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:59 smithi154 conmon[34553]: cluster 2021-07-31T02:35:58.499285+0000 mgr.x (mgr.24542) 2090 : cluster [DBG] pgmap v2872: 41 pgs: 32 unknown, 9 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:36:00.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:59 smithi154 conmon[34553]: 2021-07-31T02:36:00.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:59 smithi154 conmon[34553]: audit 2021-07-31T02:35:58.719410+0000 mon.a (mon.0) 2551 : audit [INF] from='client.? 172.21.15.45:0/3933708616' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-20","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:36:00.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:35:59 smithi154 conmon[34553]: cluster 2021-07-31T02:35:58.719745+0000 mon.a (mon.0) 2552 : cluster [DBG] osdmap e1564: 8 total, 8 up, 8 in 2021-07-31T02:36:00.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:59 smithi045 conmon[33655]: cluster 2021-07-31T02:35:58.499285+0000 2021-07-31T02:36:00.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:59 smithi045 conmon[33655]: mgr.x (mgr.24542) 2090 : cluster [DBG] pgmap v2872: 41 pgs: 32 unknown, 9 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:36:00.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:59 smithi045 conmon[33655]: audit 2021-07-31T02:35:58.719410+0000 mon.a (mon.0) 2551 : audit [INF] from='client.? 172.21.15.45:0/3933708616' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-20","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:36:00.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:59 smithi045 conmon[33655]: cluster 2021- 2021-07-31T02:36:00.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:59 smithi045 conmon[33655]: 07-31T02:35:58.719745+0000 mon.a (mon 2021-07-31T02:36:00.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:35:59 smithi045 conmon[33655]: .0) 2552 : cluster [DBG] osdmap e1564: 8 total, 8 up, 8 in 2021-07-31T02:36:00.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:59 smithi045 conmon[39718]: cluster 2021-07-31T02:35:58 2021-07-31T02:36:00.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:59 smithi045 conmon[39718]: .499285+0000 mgr.x (mgr.24542) 2090 : cluster [DBG] pgmap v2872: 41 pgs: 32 unknown, 9 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:36:00.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:59 smithi045 conmon[39718]: audit 2021-07-31T02:35:58.719410+0000 mon.a 2021-07-31T02:36:00.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:59 smithi045 conmon[39718]: (mon.0) 2551 : audit [INF] from='client.? 172.21.15.45:0/3933708616' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi045-78694-20","app": "rados","yes_i_really_mean_it": true}]': finished 2021-07-31T02:36:00.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:59 smithi045 conmon[39718]: cluster 2021 2021-07-31T02:36:00.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:59 smithi045 conmon[39718]: -07-31T02:35:58.719745+0000 mon.a 2021-07-31T02:36:00.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:35:59 smithi045 conmon[39718]: (mon.0) 2552 : cluster [DBG] osdmap e1564: 8 total, 8 up, 8 in 2021-07-31T02:36:00.734 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:36:00.734 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestJournalReplay.AioDiscardEvent (2013 ms) 2021-07-31T02:36:00.735 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestJournalReplay.AioWriteEvent 2021-07-31T02:36:00.740 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:36:00.740 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestJournalReplay.AioWriteEvent (5 ms) 2021-07-31T02:36:00.740 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestJournalReplay.AioFlushEvent 2021-07-31T02:36:00.747 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:36:00.747 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestJournalReplay.AioFlushEvent (7 ms) 2021-07-31T02:36:00.748 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestJournalReplay.SnapCreate 2021-07-31T02:36:00.751 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:36:00.751 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestJournalReplay.SnapCreate (4 ms) 2021-07-31T02:36:00.751 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestJournalReplay.SnapProtect 2021-07-31T02:36:00.754 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:36:00.754 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestJournalReplay.SnapProtect (3 ms) 2021-07-31T02:36:00.754 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestJournalReplay.SnapUnprotect 2021-07-31T02:36:00.759 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:36:00.759 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestJournalReplay.SnapUnprotect (6 ms) 2021-07-31T02:36:00.760 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestJournalReplay.SnapRename 2021-07-31T02:36:00.762 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:36:00.763 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestJournalReplay.SnapRename (3 ms) 2021-07-31T02:36:00.763 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestJournalReplay.SnapRollback 2021-07-31T02:36:00.766 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:36:00.766 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestJournalReplay.SnapRollback (3 ms) 2021-07-31T02:36:00.766 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestJournalReplay.SnapRemove 2021-07-31T02:36:00.769 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:36:00.769 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestJournalReplay.SnapRemove (3 ms) 2021-07-31T02:36:00.770 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestJournalReplay.Rename 2021-07-31T02:36:00.773 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:36:00.773 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestJournalReplay.Rename (4 ms) 2021-07-31T02:36:00.773 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestJournalReplay.Resize 2021-07-31T02:36:00.776 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:36:00.776 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestJournalReplay.Resize (4 ms) 2021-07-31T02:36:00.776 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestJournalReplay.Flatten 2021-07-31T02:36:00.779 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:36:00.780 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestJournalReplay.Flatten (3 ms) 2021-07-31T02:36:00.780 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestJournalReplay.UpdateFeatures 2021-07-31T02:36:00.784 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:36:00.785 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestJournalReplay.UpdateFeatures (4 ms) 2021-07-31T02:36:00.785 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestJournalReplay.MetadataSet 2021-07-31T02:36:00.787 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:36:00.788 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestJournalReplay.MetadataSet (3 ms) 2021-07-31T02:36:00.788 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestJournalReplay.MetadataRemove 2021-07-31T02:36:00.790 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:36:00.790 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestJournalReplay.MetadataRemove (3 ms) 2021-07-31T02:36:00.792 INFO:tasks.workunit.client.0.smithi045.stdout:[ RUN ] TestJournalReplay.ObjectPosition 2021-07-31T02:36:00.793 INFO:tasks.workunit.client.0.smithi045.stdout:SKIPPING 2021-07-31T02:36:00.794 INFO:tasks.workunit.client.0.smithi045.stdout:[ OK ] TestJournalReplay.ObjectPosition (4 ms) 2021-07-31T02:36:01.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:00 smithi154 conmon[34553]: cluster 2021-07-31T02:35:59.720781+0000 mon.a (mon.0) 2553 : cluster [DBG] osdmap e1565: 8 total, 8 up, 8 in 2021-07-31T02:36:01.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:00 smithi154 conmon[34553]: cluster 2021-07-31T02:36:00.721971+0000 mon.a (mon.0) 2554 : cluster [DBG] osdmap e1566: 8 total, 8 up, 8 in 2021-07-31T02:36:01.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:00 smithi045 conmon[33655]: cluster 2021-07-31T02:35:59.720781+0000 mon.a (mon.0) 2021-07-31T02:36:01.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:00 smithi045 conmon[33655]: 2553 : cluster [DBG] osdmap e1565: 8 total, 8 up, 8 in 2021-07-31T02:36:01.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:00 smithi045 conmon[33655]: cluster 2021-07-31T02:36:00.721971+0000 mon.a (mon.0) 2554 : cluster [DBG] osdmap e1566: 8 total, 8 up, 8 in 2021-07-31T02:36:01.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:00 smithi045 conmon[39718]: cluster 2021-07-31T02:35:59.720781+0000 2021-07-31T02:36:01.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:00 smithi045 conmon[39718]: mon.a (mon.0) 2553 : cluster [DBG] osdmap e1565: 8 total, 8 up, 8 in 2021-07-31T02:36:01.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:00 smithi045 conmon[39718]: cluster 2021-07-31T02:36:00.721971+0000 2021-07-31T02:36:01.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:00 smithi045 conmon[39718]: mon.a (mon.0) 2554 : cluster [DBG] osdmap e1566: 8 total, 8 up, 8 in 2021-07-31T02:36:01.741 INFO:tasks.workunit.client.0.smithi045.stdout:[----------] 16 tests from TestJournalReplay (2073 ms total) 2021-07-31T02:36:01.741 INFO:tasks.workunit.client.0.smithi045.stdout: 2021-07-31T02:36:01.742 INFO:tasks.workunit.client.0.smithi045.stdout:[----------] Global test environment tear-down 2021-07-31T02:36:01.742 INFO:tasks.workunit.client.0.smithi045.stdout:[==========] 359 tests from 15 test suites ran. (601651 ms total) 2021-07-31T02:36:01.743 INFO:tasks.workunit.client.0.smithi045.stdout:[ PASSED ] 359 tests. 2021-07-31T02:36:01.743 INFO:tasks.workunit.client.0.smithi045.stdout: 2021-07-31T02:36:01.743 INFO:tasks.workunit.client.0.smithi045.stdout: YOU HAVE 2 DISABLED TESTS 2021-07-31T02:36:01.744 INFO:tasks.workunit.client.0.smithi045.stdout: 2021-07-31T02:36:01.751 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-07-31T02:36:01.751 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2021-07-31T02:36:01.796 INFO:tasks.workunit:Stopping ['rbd/test_librbd.sh'] on client.0... 2021-07-31T02:36:01.797 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2021-07-31T02:36:02.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:01 smithi154 conmon[34553]: cluster 2021-07-31T02:36:00.499729+0000 mgr.x (mgr.24542) 2091 : cluster [DBG] pgmap v2875: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:36:02.021 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:01 smithi045 conmon[33655]: cluster 2021-07-31T02:36:00.499729 2021-07-31T02:36:02.021 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:01 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 2091 : cluster [DBG] pgmap v2875: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:36:02.022 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:01 smithi045 conmon[39718]: cluster 2021-07-31T02:36:00.499729+0000 mgr.x ( 2021-07-31T02:36:02.023 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:01 smithi045 conmon[39718]: mgr.24542) 2091 : cluster [DBG] pgmap v2875: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:36:02.024 DEBUG:teuthology.parallel:result is None 2021-07-31T02:36:02.024 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2021-07-31T02:36:02.050 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2021-07-31T02:36:02.051 DEBUG:teuthology.orchestra.run.smithi045:> rmdir -- /home/ubuntu/cephtest/mnt.0 2021-07-31T02:36:02.107 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2021-07-31T02:36:02.108 INFO:teuthology.task.full_sequential:In full_sequential, running task print... 2021-07-31T02:36:02.109 INFO:teuthology.task.print:**** done end test_rbd_api.yaml 2021-07-31T02:36:02.109 INFO:teuthology.task.full_sequential:In full_sequential, running task print... 2021-07-31T02:36:02.110 INFO:teuthology.task.print:**** done start test_rbd_python.yaml 2021-07-31T02:36:02.110 INFO:teuthology.task.full_sequential:In full_sequential, running task workunit... 2021-07-31T02:36:02.111 INFO:tasks.workunit:Pulling workunits from ref pacific 2021-07-31T02:36:02.112 INFO:tasks.workunit:Making a separate scratch dir for every client... 2021-07-31T02:36:02.112 DEBUG:teuthology.orchestra.run.smithi045:> stat -- /home/ubuntu/cephtest/mnt.0 2021-07-31T02:36:02.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-07-31T02:36:02.163 INFO:teuthology.orchestra.run.smithi045.stderr:stat: cannot stat '/home/ubuntu/cephtest/mnt.0': No such file or directory 2021-07-31T02:36:02.163 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -- /home/ubuntu/cephtest/mnt.0 2021-07-31T02:36:02.218 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2021-07-31T02:36:02.219 DEBUG:teuthology.orchestra.run.smithi045:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2021-07-31T02:36:02.273 INFO:tasks.workunit:timeout=3h 2021-07-31T02:36:02.274 INFO:tasks.workunit:cleanup=True 2021-07-31T02:36:02.276 DEBUG:teuthology.orchestra.run.smithi045:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone --depth 1 --branch pacific git://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 2021-07-31T02:36:02.329 INFO:tasks.workunit.client.0.smithi045.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2021-07-31T02:36:02.353 INFO:tasks.workunit.client.0.smithi045.stderr:warning: Could not find remote branch pacific to clone. 2021-07-31T02:36:02.353 INFO:tasks.workunit.client.0.smithi045.stderr:fatal: Remote branch pacific not found in upstream origin 2021-07-31T02:36:02.354 DEBUG:teuthology.orchestra.run:got remote process result: 128 2021-07-31T02:36:02.355 INFO:tasks.workunit:failed to check out 'pacific' from git://git.ceph.com/ceph-ci.git; will also try in git://git.ceph.com/ceph.git 2021-07-31T02:36:02.355 DEBUG:teuthology.orchestra.run.smithi045:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone --depth 1 --branch pacific git://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 2021-07-31T02:36:02.413 INFO:tasks.workunit.client.0.smithi045.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2021-07-31T02:36:03.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:02 smithi154 conmon[34553]: cluster 2021-07-31T02:36:01.737982+0000 mon.a (mon.0) 2555 : cluster [DBG] osdmap e1567: 8 total, 8 up, 8 in 2021-07-31T02:36:03.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:02 smithi045 conmon[39718]: cluster 2021-07-31T02:36:01. 2021-07-31T02:36:03.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:02 smithi045 conmon[39718]: 737982+0000 mon.a (mon.0) 2555 : cluster [DBG] osdmap e1567: 8 total, 8 up, 8 in 2021-07-31T02:36:03.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:02 smithi045 conmon[33655]: cluster 2021-07-31T02:36:01.737982+0000 mon.a (mon.0) 2555 : 2021-07-31T02:36:03.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:02 smithi045 conmon[33655]: cluster [DBG] osdmap e1567: 8 total, 8 up, 8 in 2021-07-31T02:36:04.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:04 smithi154 conmon[34553]: cluster 2021-07-31T02:36:02.500223+0000 mgr.x (mgr.24542) 2092 : cluster [DBG] pgmap v2878: 9 pgs: 9 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:36:04.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:04 smithi045 conmon[39718]: cluster 2021-07-31T02:36:02.500223+0000 2021-07-31T02:36:04.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:04 smithi045 conmon[39718]: mgr.x (mgr.24542) 2092 : cluster [DBG] pgmap v2878: 9 pgs: 9 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:36:04.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:04 smithi045 conmon[33655]: cluster 2021-07-31T02:36: 2021-07-31T02:36:04.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:04 smithi045 conmon[33655]: 02.500223+0000 mgr.x (mgr.24542) 2092 : cluster [DBG] pgmap v2878: 9 pgs: 9 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:36:06.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:06 smithi154 conmon[34553]: cluster 2021-07-31T02:36:04.500700+0000 mgr.x (mgr.24542) 2093 : cluster [DBG] pgmap v2879: 9 pgs: 9 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:36:06.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:06 smithi045 conmon[33655]: cluster 2021-07-31T02:36:04.500700 2021-07-31T02:36:06.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:06 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 2093 : cluster [DBG] pgmap v2879: 9 pgs: 9 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:36:06.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:06 smithi045 conmon[39718]: cluster 2021-07-31T02: 2021-07-31T02:36:06.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:06 smithi045 conmon[39718]: 36:04.500700+0000 mgr.x (mgr.24542) 2093 : cluster [DBG] pgmap v2879: 9 pgs: 9 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:36:07.562 DEBUG:teuthology.orchestra.run.smithi045:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2021-07-31T02:36:07.616 INFO:tasks.workunit.client.0.smithi045.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2021-07-31T02:36:07.618 INFO:tasks.workunit.client.0.smithi045.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2021-07-31T02:36:07.618 INFO:tasks.workunit.client.0.smithi045.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2021-07-31T02:36:07.666 INFO:tasks.workunit.client.0.smithi045.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2021-07-31T02:36:07.714 INFO:tasks.workunit.client.0.smithi045.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2021-07-31T02:36:07.750 INFO:tasks.workunit.client.0.smithi045.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2021-07-31T02:36:07.751 INFO:tasks.workunit.client.0.smithi045.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2021-07-31T02:36:07.752 INFO:tasks.workunit.client.0.smithi045.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2021-07-31T02:36:07.789 INFO:tasks.workunit.client.0.smithi045.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2021-07-31T02:36:07.792 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2021-07-31T02:36:07.792 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2021-07-31T02:36:07.848 INFO:tasks.workunit:Running workunits matching rbd/test_librbd_python.sh on client.0... 2021-07-31T02:36:07.849 INFO:tasks.workunit:Running workunit rbd/test_librbd_python.sh... 2021-07-31T02:36:07.850 DEBUG:teuthology.orchestra.run.smithi045:workunit test rbd/test_librbd_python.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=pacific TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_librbd_python.sh 2021-07-31T02:36:07.906 INFO:tasks.workunit.client.0.smithi045.stderr:++ dirname /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_librbd_python.sh 2021-07-31T02:36:07.907 INFO:tasks.workunit.client.0.smithi045.stderr:+ relpath=/home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/../../../src/test/pybind 2021-07-31T02:36:07.907 INFO:tasks.workunit.client.0.smithi045.stderr:+ '[' -n '' ']' 2021-07-31T02:36:07.907 INFO:tasks.workunit.client.0.smithi045.stderr:+ python3 -m nose -v /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/../../../src/test/pybind/test_rbd.py 2021-07-31T02:36:09.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:08 smithi154 conmon[34553]: cluster 2021-07-31T02:36:06.501219+0000 mgr.x (mgr.24542) 2094 : cluster [DBG] pgmap v2880: 9 pgs: 9 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:36:09.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:08 smithi045 conmon[33655]: cluster 2021-07-31T02:36:06.501219+0000 mgr.x (mgr.24542 2021-07-31T02:36:09.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:08 smithi045 conmon[33655]: ) 2094 : cluster [DBG] pgmap v2880: 9 pgs: 9 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:36:09.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:08 smithi045 conmon[39718]: cluster 2021-07-31T02:36:06.501219+0000 mgr.x (mgr. 2021-07-31T02:36:09.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:08 smithi045 conmon[39718]: 24542) 2094 : cluster [DBG] pgmap v2880: 9 pgs: 9 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:36:10.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:09 smithi154 conmon[34553]: cluster 2021-07-31T02:36:08.501567+0000 mgr.x (mgr.24542) 2095 : cluster [DBG] pgmap v2881: 9 pgs: 9 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:36:10.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:09 smithi154 conmon[34553]: 2021-07-31T02:36:10.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:09 smithi154 conmon[34553]: cluster 2021-07-31T02:36:08.734586+0000 mon.a (mon.0) 2556 : cluster [DBG] osdmap e1568: 8 total, 8 up, 8 in 2021-07-31T02:36:10.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:09 smithi154 conmon[34553]: audit 2021-07-31T02:36:08.735790+0000 mon.c (mon.1) 123 : audit [INF] 2021-07-31T02:36:10.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:09 smithi154 conmon[34553]: from='client.? 172.21.15.45:0/530405139' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rbd-api-smithi045-82607-1","app": "rbd","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:36:10.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:09 smithi154 conmon[34553]: audit 2021-07-31T02:36:08.740652+0000 2021-07-31T02:36:10.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:09 smithi154 conmon[34553]: mon.a (mon.0) 2557 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rbd-api-smithi045-82607-1","app": "rbd","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:36:10.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:09 smithi045 conmon[39718]: cluster 2021-07-31T02:36:08.501567+0000 mgr.x 2021-07-31T02:36:10.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:09 smithi045 conmon[39718]: (mgr.24542) 2095 : cluster [DBG] pgmap v2881: 9 pgs: 9 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:36:10.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:09 smithi045 conmon[39718]: cluster 2021-07-31T 2021-07-31T02:36:10.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:09 smithi045 conmon[39718]: 02:36:08.734586+0000 mon.a (mon.0) 2556 : cluster 2021-07-31T02:36:10.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:09 smithi045 conmon[39718]: [DBG] osdmap e1568: 8 total, 8 up, 8 in 2021-07-31T02:36:10.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:09 smithi045 conmon[39718]: audit 2021-07 2021-07-31T02:36:10.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:09 smithi045 conmon[39718]: -31T02:36:08.735790+0000 2021-07-31T02:36:10.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:09 smithi045 conmon[39718]: mon.c (mon.1) 123 : audit [INF] 2021-07-31T02:36:10.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:09 smithi045 conmon[39718]: from='client.? 172.21.15.45:0/530405139' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rbd-api-smithi045-82607-1","app": "rbd","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:36:10.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:09 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T02:36:10.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:09 smithi045 conmon[39718]: 02:36:08.740652+0000 mon.a ( 2021-07-31T02:36:10.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:09 smithi045 conmon[39718]: mon.0) 2557 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rbd-api-smithi045-82607-1","app": "rbd","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:36:10.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:09 smithi045 conmon[39718]: 2021-07-31T02:36:10.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:09 smithi045 conmon[33655]: cluster 2021-07-31T02:36:08.501567+0000 mgr.x (mgr.24542) 2095 : cluster [DBG] 2021-07-31T02:36:10.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:09 smithi045 conmon[33655]: pgmap v2881: 9 pgs: 9 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:36:10.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:09 smithi045 conmon[33655]: cluster 2021-07-31T02:36:08.734586+0000 mon.a (mon.0) 2556 : cluster [DBG] osdmap e1568: 8 total, 8 up, 8 in 2021-07-31T02:36:10.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:09 smithi045 conmon[33655]: audit 2021-07-31T02:36:10.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:09 smithi045 conmon[33655]: 2021-07-31T02:36 2021-07-31T02:36:10.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:09 smithi045 conmon[33655]: :08.735790+0000 mon.c (mon.1) 123 2021-07-31T02:36:10.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:09 smithi045 conmon[33655]: : audit [INF] from='client.? 172.21.15.45:0/530405139' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rbd-api-smithi045-82607-1","app": "rbd","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:36:10.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:09 smithi045 conmon[33655]: audit 2021-07- 2021-07-31T02:36:10.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:09 smithi045 conmon[33655]: 31T02:36:08.740652+0000 mon.a (mon.0) 2557 : audit 2021-07-31T02:36:10.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:09 smithi045 conmon[33655]: [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rbd-api-smithi045-82607-1","app": "rbd","yes_i_really_mean_it": true}]: dispatch 2021-07-31T02:36:11.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:10 smithi154 conmon[34553]: audit 2021-07-31T02:36:09.742080+0000 mon.a (mon.0) 2558 : audit [INF] 2021-07-31T02:36:11.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:10 smithi154 conmon[34553]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rbd-api-smithi045-82607-1","app": "rbd","yes_i_really_mean_it": true}]': finished 2021-07-31T02:36:11.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:10 smithi154 conmon[34553]: cluster 2021-07-31T02:36:09.742276+0000 mon.a (mon.0) 2559 : cluster [DBG] osdmap e1569: 8 total, 8 up, 8 in 2021-07-31T02:36:11.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:10 smithi045 conmon[39718]: audit 2021-07-31T02:36:09.742080 2021-07-31T02:36:11.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:10 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2558 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rbd-api-smithi045-82607-1","app": "rbd","yes_i_really_mean_it": true}]': finished 2021-07-31T02:36:11.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:10 smithi045 conmon[39718]: cluster 2021-07-31T02:36:09.742276+0000 mon.a (mon.0) 2021-07-31T02:36:11.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:10 smithi045 conmon[39718]: 2559 : cluster [DBG] osdmap e1569: 8 total, 8 up, 8 in 2021-07-31T02:36:11.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:10 smithi045 conmon[33655]: audit 2021-07-31T02:36:09.742080+0000 mon.a (mon.0) 2558 2021-07-31T02:36:11.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:10 smithi045 conmon[33655]: : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rbd-api-smithi045-82607-1","app": "rbd","yes_i_really_mean_it": true}]': finished 2021-07-31T02:36:11.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:10 smithi045 conmon[33655]: cluster 2021-07-31T02:36:09.742276+0000 mon.a (mon.0) 2559 : cluster [DBG] osdmap e1569: 8 total, 8 up, 8 in 2021-07-31T02:36:12.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:11 smithi154 conmon[34553]: cluster 2021-07-31T02:36:10.502051+0000 mgr.x (mgr.24542) 2096 : cluster [DBG] pgmap v2884: 41 pgs: 32 creating+peering, 9 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:36:12.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:11 smithi154 conmon[34553]: cluster 2021-07-31T02:36:10.748132+0000 mon.a (mon.0) 2560 : cluster [DBG] osdmap e1570: 8 total, 8 up, 8 in 2021-07-31T02:36:12.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:11 smithi045 conmon[39718]: cluster 2021-07-31T02:36:10.502051+0000 mgr.x 2021-07-31T02:36:12.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:11 smithi045 conmon[39718]: (mgr.24542) 2096 : cluster [DBG] pgmap v2884: 41 pgs: 32 creating+peering, 9 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:36:12.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:11 smithi045 conmon[39718]: cluster 2021-07-31T02:36:10.748132+0000 2021-07-31T02:36:12.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:11 smithi045 conmon[39718]: mon.a (mon.0) 2560 : cluster [DBG] osdmap e1570: 8 total, 8 up, 8 in 2021-07-31T02:36:12.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:11 smithi045 conmon[33655]: cluster 2021-07-31T02:36:10.502051+0000 mgr.x (mgr.24542) 2096 : cluster [DBG] pgmap v2884: 41 pgs: 32 creating+peering, 9 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:36:12.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:11 smithi045 conmon[33655]: cluster 2021-07-31T02:36:10.748132+0000 mon.a (mon.0) 2560 : cluster [DBG] osdmap e1570: 8 total, 8 up, 8 in 2021-07-31T02:36:13.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:12 smithi045 conmon[33655]: cluster 2021-07-31T02:36:11.753509+0000 mon.a (mon.0) 2561 : cluster 2021-07-31T02:36:13.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:12 smithi045 conmon[33655]: [DBG] osdmap e1571: 8 total, 8 up, 8 in 2021-07-31T02:36:13.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:12 smithi045 conmon[39718]: cluster 2021-07-31T02:36:11.753509+0000 mon.a (mon.0 2021-07-31T02:36:13.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:12 smithi045 conmon[39718]: ) 2561 : cluster [DBG] osdmap e1571: 8 total, 8 up, 8 in 2021-07-31T02:36:13.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:12 smithi154 conmon[34553]: cluster 2021-07-31T02:36:11.753509+0000 mon.a (mon.0) 2561 : cluster [DBG] osdmap e1571: 8 total, 8 up, 8 in 2021-07-31T02:36:13.851 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestClone.test_flatten_basic ... ok 2021-07-31T02:36:14.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:13 smithi045 conmon[39718]: cluster 2021-07-31T02:36:12.502573+0000 mgr.x 2021-07-31T02:36:14.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:13 smithi045 conmon[39718]: (mgr.24542) 2097 : cluster [DBG] pgmap v2887: 41 pgs: 32 creating+peering, 9 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:36:14.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:13 smithi045 conmon[39718]: cluster 2021-07-31T02:36:12 2021-07-31T02:36:14.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:13 smithi045 conmon[39718]: .759607+0000 mon.a (mon.0) 2562 : cluster [DBG] osdmap e1572: 8 total, 8 up, 8 in 2021-07-31T02:36:14.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:13 smithi045 conmon[33655]: cluster 2021-07-31T02:36:12.502573+0000 mgr.x (mgr 2021-07-31T02:36:14.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:13 smithi045 conmon[33655]: .24542) 2097 : cluster [DBG] pgmap v2887: 41 pgs: 32 creating+peering, 9 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:36:14.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:13 smithi045 conmon[33655]: cluster 2021-07-31T02:36:12.759607+0000 mon.a (mon.0) 2562 : cluster [DBG] osdmap e1572: 8 total, 8 up, 8 in 2021-07-31T02:36:14.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:13 smithi154 conmon[34553]: cluster 2021-07-31T02:36:12.502573+0000 mgr.x (mgr.24542) 2097 : 2021-07-31T02:36:14.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:13 smithi154 conmon[34553]: cluster [DBG] pgmap v2887: 41 pgs: 32 creating+peering, 9 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:36:14.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:13 smithi154 conmon[34553]: cluster 2021-07-31T02:36:12.759607+0000 mon.a (mon.0) 2562 : cluster [DBG] osdmap e1572: 8 total, 8 up, 8 in 2021-07-31T02:36:15.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:14 smithi045 conmon[33655]: cluster 2021-07-31T02:36:13.765127+0000 mon.a (mon.0) 2563 : cluster [DBG] osdmap e1573: 8 total, 8 up, 8 in 2021-07-31T02:36:15.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:14 smithi045 conmon[39718]: cluster 2021-07-31T02:36:13.765127+0000 mon.a ( 2021-07-31T02:36:15.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:14 smithi045 conmon[39718]: mon.0) 2563 : cluster [DBG] osdmap e1573: 8 total, 8 up, 8 in 2021-07-31T02:36:15.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:14 smithi154 conmon[34553]: cluster 2021-07-31T02:36:13.765127+0000 mon.a (mon. 2021-07-31T02:36:15.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:14 smithi154 conmon[34553]: 0) 2563 : cluster [DBG] osdmap e1573: 8 total, 8 up, 8 in 2021-07-31T02:36:15.870 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestClone.test_flatten_drops_cache ... ok 2021-07-31T02:36:16.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:15 smithi045 conmon[33655]: cluster 2021-07-31T02:36:14.503271+0000 mgr.x (mgr.24542) 2098 : 2021-07-31T02:36:16.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:15 smithi045 conmon[33655]: cluster [DBG] pgmap v2890: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 325 KiB/s rd, 22 KiB/s wr, 434 op/s 2021-07-31T02:36:16.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:15 smithi045 conmon[33655]: cluster 2021-07-31T02:36:14.771079+0000 mon.a (mon.0) 2564 : cluster [DBG] osdmap e1574: 8 total, 8 up, 8 in 2021-07-31T02:36:16.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:15 smithi045 conmon[39718]: cluster 2021-07-31T02:36:14.503271+0000 mgr.x (mgr. 2021-07-31T02:36:16.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:15 smithi045 conmon[39718]: 24542) 2098 : cluster [DBG] pgmap v2890: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 325 KiB/s rd, 22 KiB/s wr, 434 op/s 2021-07-31T02:36:16.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:15 smithi045 conmon[39718]: cluster 2021-07-31T02:36:14.771079+0000 mon.a (mon 2021-07-31T02:36:16.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:15 smithi045 conmon[39718]: .0) 2564 : cluster [DBG] osdmap e1574: 8 total, 8 up, 8 in 2021-07-31T02:36:16.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:15 smithi154 conmon[34553]: cluster 2021-07-31T02:36:14.503271+0000 mgr.x (mgr.24542) 2098 : cluster [DBG] pgmap v2890: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 325 KiB/s rd, 22 KiB/s wr, 434 op/s 2021-07-31T02:36:16.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:15 smithi154 conmon[34553]: cluster 2021-07-31T02:36:14.771079+0000 mon.a (mon.0) 2564 : cluster [DBG] osdmap e1574: 8 total, 8 up, 8 in 2021-07-31T02:36:17.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:16 smithi154 conmon[34553]: cluster 2021-07-31T02:36:15.776821+0000 mon.a (mon.0) 2565 : cluster [DBG] osdmap e1575: 8 total, 8 up, 8 in 2021-07-31T02:36:17.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:16 smithi154 conmon[34553]: 2021-07-31T02:36:17.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:16 smithi045 conmon[39718]: cluster 2021-07-31T02:36:15. 2021-07-31T02:36:17.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:16 smithi045 conmon[39718]: 776821+0000 mon.a (mon.0) 2565 : cluster [DBG] osdmap e1575: 8 total, 8 up, 8 in 2021-07-31T02:36:17.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:16 smithi045 conmon[33655]: cluster 2021-07-31T02:36:15.776821+0000 mon.a (mon.0) 2565 2021-07-31T02:36:17.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:16 smithi045 conmon[33655]: : cluster [DBG] osdmap e1575: 8 total, 8 up, 8 in 2021-07-31T02:36:18.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:17 smithi154 conmon[34553]: cluster 2021-07-31T02:36:16.503853+0000 mgr.x (mgr.24542) 2099 : cluster [DBG] 2021-07-31T02:36:18.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:17 smithi154 conmon[34553]: pgmap v2893: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 325 KiB/s rd, 22 KiB/s wr, 434 op/s 2021-07-31T02:36:18.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:17 smithi154 conmon[34553]: cluster 2021-07-31T02:36:16.783319+0000 mon.a (mon.0) 2566 : cluster [DBG] osdmap e1576: 8 total, 8 up, 8 in 2021-07-31T02:36:18.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:17 smithi154 conmon[34553]: cluster 2021-07-31 2021-07-31T02:36:18.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:17 smithi154 conmon[34553]: T02:36:17.407994+0000 mon.a (mon.0) 2567 : cluster [DBG] osdmap e1577: 8 total, 8 up, 8 in 2021-07-31T02:36:18.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:17 smithi045 conmon[33655]: cluster 2021-07-31T02:36:16.503853+0000 mgr.x (mgr.24542 2021-07-31T02:36:18.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:17 smithi045 conmon[33655]: ) 2099 : cluster [DBG] pgmap v2893: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 325 KiB/s rd, 22 KiB/s wr, 434 op/s 2021-07-31T02:36:18.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:17 smithi045 conmon[33655]: cluster 2021-07-31T02:36:16.783319+0000 mon.a (mon.0) 2566 : cluster [DBG] osdmap e1576: 8 total, 8 up, 8 in 2021-07-31T02:36:18.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:17 smithi045 conmon[33655]: cluster 2021-07-31T02:36:17.407994+0000 mon.a (mon.0) 2567 : cluster [DBG] osdmap e1577: 8 total, 8 up, 8 in 2021-07-31T02:36:18.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:17 smithi045 conmon[39718]: cluster 2021-07-31T02:36:16.503853+0000 mgr.x (mgr. 2021-07-31T02:36:18.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:17 smithi045 conmon[39718]: 24542) 2099 : cluster [DBG] pgmap v2893: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 325 KiB/s rd, 22 KiB/s wr, 434 op/s 2021-07-31T02:36:18.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:17 smithi045 conmon[39718]: cluster 2021-07-31T02: 2021-07-31T02:36:18.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:17 smithi045 conmon[39718]: 36:16.783319+0000 mon.a (mon.0) 2566 : cluster [DBG] osdmap e1576: 8 total, 8 up, 8 in 2021-07-31T02:36:18.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:17 smithi045 conmon[39718]: cluster 2021-07-31T02:36:17 2021-07-31T02:36:18.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:17 smithi045 conmon[39718]: .407994+0000 mon.a (mon.0) 2567 : cluster [DBG] osdmap e1577: 8 total, 8 up, 8 in 2021-07-31T02:36:18.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:17 smithi045 conmon[39718]: 2021-07-31T02:36:19.506 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestClone.test_flatten_errors ... ok 2021-07-31T02:36:19.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:19 smithi154 conmon[34553]: cluster 2021-07-31T02:36:18.413793+0000 mon.a (mon 2021-07-31T02:36:19.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:19 smithi154 conmon[34553]: .0) 2568 : cluster [DBG] osdmap e1578: 8 total, 8 up, 8 in 2021-07-31T02:36:19.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:19 smithi045 conmon[39718]: cluster 2021-07-31T02:36:18.413793+0000 mon.a 2021-07-31T02:36:19.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:19 smithi045 conmon[39718]: (mon.0) 2568 : cluster [DBG] osdmap e1578: 8 total, 8 up, 8 in 2021-07-31T02:36:19.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:19 smithi045 conmon[33655]: cluster 2021-07-31T02:36:18.413793+0000 mon.a (mon.0) 2568 : cluster [DBG] osdmap e1578: 8 total, 8 up, 8 in 2021-07-31T02:36:19.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:19 smithi045 conmon[33655]: 2021-07-31T02:36:20.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:20 smithi154 conmon[34553]: cluster 2021-07-31T02:36:18.504203+0000 mgr.x (mgr.24542) 2100 : cluster [DBG] pgmap v2897: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:36:20.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:20 smithi154 conmon[34553]: cluster 2021-07-31T02:36:19.415765+0000 mon.a (mon.0) 2569 : cluster [DBG] osdmap e1579: 8 total, 8 up, 8 in 2021-07-31T02:36:20.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:20 smithi045 conmon[39718]: cluster 2021-07-31T02:36:18.504203+0000 mgr.x (mgr.24542) 2100 2021-07-31T02:36:20.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:20 smithi045 conmon[39718]: : cluster [DBG] pgmap v2897: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:36:20.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:20 smithi045 conmon[39718]: cluster 2021-07-31T02:36:19.415765+0000 mon.a (mon.0) 2569 : cluster [DBG] osdmap e1579: 8 total, 8 up, 8 in 2021-07-31T02:36:20.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:20 smithi045 conmon[33655]: cluster 2021-07-31T02:36: 2021-07-31T02:36:20.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:20 smithi045 conmon[33655]: 18.504203+0000 mgr.x (mgr.24542) 2100 : cluster [DBG] pgmap v2897: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:36:20.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:20 smithi045 conmon[33655]: cluster 2021-07-31T02 2021-07-31T02:36:20.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:20 smithi045 conmon[33655]: :36:19.415765+0000 mon.a (mon.0) 2569 : cluster [DBG] osdmap e1579: 8 total, 8 up, 8 in 2021-07-31T02:36:21.527 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestClone.test_flatten_larger_order ... ok 2021-07-31T02:36:21.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:21 smithi154 conmon[34553]: cluster 2021-07-31T02:36:20.426983+0000 mon.a (mon.0) 2570 : cluster [DBG] 2021-07-31T02:36:21.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:21 smithi154 conmon[34553]: osdmap e1580: 8 total, 8 up, 8 in 2021-07-31T02:36:21.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:21 smithi154 conmon[34553]: cluster 2021-07-31T02:36:20.504573+0000 mgr.x (mgr.24542) 2101 : cluster [DBG] pgmap v2900: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 335 KiB/s rd, 25 KiB/s wr, 452 op/s 2021-07-31T02:36:21.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:21 smithi045 conmon[33655]: cluster 2021-07-31T02:36:20.426983+0000 mon.a (mon.0) 2570 : 2021-07-31T02:36:21.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:21 smithi045 conmon[33655]: cluster [DBG] osdmap e1580: 8 total, 8 up, 8 in 2021-07-31T02:36:21.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:21 smithi045 conmon[33655]: cluster 2021-07-31T02:36:20.504573+0000 mgr.x (mgr.24542) 2101 : cluster [DBG] pgmap v2900: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 335 KiB/s rd, 25 KiB/s wr, 452 op/s 2021-07-31T02:36:21.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:21 smithi045 conmon[39718]: cluster 2021-07-31T02:36:20.426983+0000 mon.a (mon 2021-07-31T02:36:21.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:21 smithi045 conmon[39718]: .0) 2570 : cluster [DBG] osdmap e1580: 8 total, 8 up, 8 in 2021-07-31T02:36:21.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:21 smithi045 conmon[39718]: cluster 2021-07-31T02:36:20.504573+0000 mgr.x (mgr.24542) 2101 : cluster [DBG] pgmap v2900: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 335 KiB/s rd, 25 KiB/s wr, 452 op/s 2021-07-31T02:36:22.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:22 smithi154 conmon[34553]: cluster 2021-07-31T02:36:21.434182+0000 mon.a (mon.0) 2571 : cluster [DBG] osdmap e1581: 8 total, 8 up, 8 in 2021-07-31T02:36:22.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:22 smithi154 conmon[34553]: cluster 2021-07-31T02:36:22.409436+0000 mon.a (mon.0) 2572 : cluster [DBG] osdmap e1582: 8 total, 8 up, 8 in 2021-07-31T02:36:22.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:22 smithi045 conmon[33655]: cluster 2021-07-31T02:36:21.434182+0000 2021-07-31T02:36:22.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:22 smithi045 conmon[33655]: mon.a (mon.0) 2571 : cluster [DBG] osdmap e1581: 8 total, 8 up, 8 in 2021-07-31T02:36:22.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:22 smithi045 conmon[33655]: cluster 2021-07-31T02:36:22.409436+0000 mon.a (mon.0) 2021-07-31T02:36:22.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:22 smithi045 conmon[33655]: 2572 : cluster [DBG] osdmap e1582: 8 total, 8 up, 8 in 2021-07-31T02:36:22.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:22 smithi045 conmon[39718]: cluster 2021-07-31T02:36:21 2021-07-31T02:36:22.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:22 smithi045 conmon[39718]: .434182+0000 mon.a (mon.0) 2571 : cluster [DBG] osdmap e1581: 8 total, 8 up, 8 in 2021-07-31T02:36:22.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:22 smithi045 conmon[39718]: cluster 2021- 2021-07-31T02:36:22.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:22 smithi045 conmon[39718]: 07-31T02:36: 2021-07-31T02:36:22.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:22 smithi045 conmon[39718]: 22.409436+0000 mon.a (mon 2021-07-31T02:36:22.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:22 smithi045 conmon[39718]: .0) 2572 : cluster [DBG] 2021-07-31T02:36:22.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:22 smithi045 conmon[39718]: osdmap e1582: 8 total, 8 up, 8 in 2021-07-31T02:36:23.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:23 smithi154 conmon[34553]: cluster 2021-07-31T02:36:22.505166+0000 mgr.x (mgr.24542) 2102 : cluster 2021-07-31T02:36:23.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:23 smithi154 conmon[34553]: [DBG] pgmap v2903: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 312 KiB/s rd, 23 KiB/s wr, 421 op/s 2021-07-31T02:36:23.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:23 smithi045 conmon[39718]: cluster 2021-07-31T02:36:22.505166+0000 2021-07-31T02:36:23.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:23 smithi045 conmon[39718]: mgr.x (mgr.24542) 2102 : cluster [DBG] pgmap v2903: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 312 KiB/s rd, 23 KiB/s wr, 421 op/s 2021-07-31T02:36:23.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:23 smithi045 conmon[33655]: cluster 2021-07-31T02:36:22.505166 2021-07-31T02:36:23.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:23 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 2102 : cluster [DBG] pgmap v2903: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 312 KiB/s rd, 23 KiB/s wr, 421 op/s 2021-07-31T02:36:24.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:24 smithi154 conmon[34553]: cluster 2021-07-31T02:36:23.443323+0000 mon.a (mon.0) 2573 : cluster [DBG] osdmap e1583: 8 total, 8 up, 8 in 2021-07-31T02:36:24.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:24 smithi045 conmon[39718]: cluster 2021-07-31T02:36: 2021-07-31T02:36:24.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:24 smithi045 conmon[39718]: 23.443323+0000 mon.a (mon.0) 2573 : cluster [DBG] osdmap e1583: 8 total, 8 up, 8 in 2021-07-31T02:36:24.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:24 smithi045 conmon[33655]: cluster 2021-07-31T02:36:23.443323+0000 mon.a (mon.0) 2573 : cluster [DBG] osdmap e1583: 8 total, 8 up, 8 in 2021-07-31T02:36:25.209 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:36:25.493 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:36:25.493 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 51m ago 52m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:36:25.494 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 48m ago 51m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:36:25.494 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 48m ago 55m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:36:25.494 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 51m ago 57m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:36:25.494 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 51m ago 57m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:36:25.495 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 48m ago 55m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:36:25.495 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 51m ago 55m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:36:25.495 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 51m ago 52m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:36:25.495 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 48m ago 52m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:36:25.496 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 51m ago 55m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:36:25.496 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 51m ago 54m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:36:25.496 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 51m ago 54m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:36:25.496 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 51m ago 54m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:36:25.496 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 48m ago 53m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:36:25.497 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 48m ago 53m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:36:25.497 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 48m ago 53m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:36:25.497 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 48m ago 52m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:36:25.497 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 48m ago 52m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:36:25.546 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestClone.test_flatten_multi_level ... ok 2021-07-31T02:36:25.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:25 smithi154 conmon[34553]: cluster 2021-07-31T02:36:24.449897+0000 mon.a (mon.0) 2021-07-31T02:36:25.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:25 smithi154 conmon[34553]: 2574 : cluster [DBG] osdmap e1584: 8 total, 8 up, 8 in 2021-07-31T02:36:25.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:25 smithi154 conmon[34553]: cluster 2021-07-31T02:36:24.505588+0000 mgr.x (mgr.24542) 2103 : cluster [DBG] pgmap v2906: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 501 KiB/s rd, 37 KiB/s wr, 676 op/s 2021-07-31T02:36:25.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:25 smithi154 conmon[34553]: audit 2021- 2021-07-31T02:36:25.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:25 smithi154 conmon[34553]: 07-31T02:36:25.198281+0000 mgr.x (mgr.24542) 2104 : audit [DBG] from='client.27550 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:36:25.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:25 smithi045 conmon[39718]: cluster 2021-07-31T02: 2021-07-31T02:36:25.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:25 smithi045 conmon[39718]: 36:24.449897+0000 mon.a (mon.0) 2574 : cluster [DBG] osdmap e1584: 8 total, 8 up, 8 in 2021-07-31T02:36:25.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:25 smithi045 conmon[39718]: cluster 2021-07-31T02:36:24 2021-07-31T02:36:25.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:25 smithi045 conmon[39718]: .505588+0000 mgr.x (mgr.24542) 2103 : cluster [DBG] pgmap v2906: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 501 KiB/s rd, 37 KiB/s wr, 676 op/s 2021-07-31T02:36:25.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:25 smithi045 conmon[39718]: audit 2021-07-31T02:36:25 2021-07-31T02:36:25.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:25 smithi045 conmon[39718]: .198281+0000 mgr.x (mgr.24542) 2104 : audit [DBG] 2021-07-31T02:36:25.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:25 smithi045 conmon[39718]: from='client.27550 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:36:25.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:25 smithi045 conmon[33655]: cluster 2021-07-31T02:36:24.449897+0000 mon.a (mon 2021-07-31T02:36:25.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:25 smithi045 conmon[33655]: .0) 2574 : cluster [DBG] osdmap e1584: 8 total, 8 up, 8 in 2021-07-31T02:36:25.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:25 smithi045 conmon[33655]: cluster 2021-07-31T02:36:24.505588 2021-07-31T02:36:25.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:25 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 2103 : cluster [DBG] pgmap v2906: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 501 KiB/s rd, 37 KiB/s wr, 676 op/s 2021-07-31T02:36:25.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:25 smithi045 conmon[33655]: audit 2021-07 2021-07-31T02:36:25.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:25 smithi045 conmon[33655]: -31T02:36:25.198281+0000 mgr.x (mgr 2021-07-31T02:36:25.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:25 smithi045 conmon[33655]: .24542) 2104 : audit [DBG] from='client.27550 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:36:25.837 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:36:25.838 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:36:25.838 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:36:25.838 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:36:25.838 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:36:25.839 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:36:25.839 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:36:25.839 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:36:25.839 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:36:25.839 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:36:25.840 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:36:25.840 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:36:25.840 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:36:25.840 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:36:25.840 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:36:25.841 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:36:25.841 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:36:26.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:26 smithi154 conmon[34553]: cluster 2021-07-31T02:36:25.453899+0000 mon.a (mon.0) 2575 : cluster [DBG] osdmap e1585: 8 total, 8 up, 8 in 2021-07-31T02:36:26.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:26 smithi154 conmon[34553]: audit 2021-07-31T02:36:25.484889+0000 mgr.x (mgr.24542) 2105 : audit [DBG] from='client.27556 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:36:26.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:26 smithi154 conmon[34553]: audit 2021-07-31T02:36:25.837577 2021-07-31T02:36:26.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:26 smithi154 conmon[34553]: +0000 mon.c (mon.1) 124 : audit [DBG] from='client.? 172.21.15.45:0/3086790348' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:36:26.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:26 smithi045 conmon[33655]: cluster 2021-07-31T02:36:25.453899+0000 mon.a (mon.0) 2575 : cluster [DBG] osdmap e1585: 8 total, 8 up, 8 in 2021-07-31T02:36:26.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:26 smithi045 conmon[33655]: audit 2021-07-31T02:36:25.484889+0000 mgr.x (mgr.24542) 2105 : audit [DBG] from='client.27556 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:36:26.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:26 smithi045 conmon[33655]: audit 2021-07 2021-07-31T02:36:26.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:26 smithi045 conmon[33655]: -31T02:36:25.837577+0000 mon.c (mon.1) 124 : audit [DBG] from='client.? 172.21.15.45:0/3086790348' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:36:26.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:26 smithi045 conmon[39718]: cluster 2021-07-31T02:36:25.453899+0000 mon.a ( 2021-07-31T02:36:26.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:26 smithi045 conmon[39718]: mon.0) 2575 : cluster [DBG] osdmap e1585: 8 total, 8 up, 8 in 2021-07-31T02:36:26.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:26 smithi045 conmon[39718]: audit 2021 2021-07-31T02:36:26.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:26 smithi045 conmon[39718]: -07-31T02:36:25.484889+0000 mgr.x (mgr.24542) 2105 : audit [DBG] from='client.27556 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:36:26.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:26 smithi045 conmon[39718]: audit 2021-07 2021-07-31T02:36:26.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:26 smithi045 conmon[39718]: -31T02:36:25.837577+0000 mon.c (mon.1) 2021-07-31T02:36:26.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:26 smithi045 conmon[39718]: 124 : audit [DBG] from='client.? 172.21.15.45:0/3086790348' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:36:27.502 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestClone.test_flatten_smaller_order ... ok 2021-07-31T02:36:27.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:27 smithi154 conmon[34553]: cluster 2021-07-31T02:36:26.460337+0000 mon.a (mon.0) 2576 : 2021-07-31T02:36:27.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:27 smithi154 conmon[34553]: cluster [DBG] osdmap e1586: 8 total, 8 up, 8 in 2021-07-31T02:36:27.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:27 smithi154 conmon[34553]: cluster 2021-07-31T02:36:26.505978+0000 mgr.x (mgr.24542) 2106 : cluster [DBG] pgmap v2909: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 501 KiB/s rd, 37 KiB/s wr, 676 op/s 2021-07-31T02:36:27.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:27 smithi154 conmon[34553]: cluster 2021-07-31T02:36:27.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:27 smithi154 conmon[34553]: 2021-07-31T02:36:27.410052+0000 mon.a (mon.0) 2577 : cluster 2021-07-31T02:36:27.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:27 smithi154 conmon[34553]: [DBG] osdmap e1587: 8 total, 8 up, 8 in 2021-07-31T02:36:27.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:27 smithi045 conmon[33655]: cluster 2021-07-31T02:36:26.460337+0000 2021-07-31T02:36:27.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:27 smithi045 conmon[33655]: mon.a (mon.0) 2576 : cluster [DBG] osdmap e1586: 8 total, 8 up, 8 in 2021-07-31T02:36:27.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:27 smithi045 conmon[33655]: cluster 2021-07-31T02:36:26.505978+0000 mgr.x (mgr.24542) 2106 2021-07-31T02:36:27.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:27 smithi045 conmon[33655]: : cluster [DBG] pgmap v2909: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 501 KiB/s rd, 37 KiB/s wr, 676 op/s 2021-07-31T02:36:27.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:27 smithi045 conmon[33655]: cluster 2021 2021-07-31T02:36:27.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:27 smithi045 conmon[33655]: -07-31T02:36:27.410052+0000 mon.a (mon.0) 2577 : cluster [DBG] osdmap e1587: 8 total, 8 up, 8 in 2021-07-31T02:36:27.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:27 smithi045 conmon[39718]: cluster 2021-07-31T02:36:26.460337+0000 2021-07-31T02:36:27.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:27 smithi045 conmon[39718]: mon.a (mon.0) 2576 : cluster [DBG] osdmap e1586: 8 total, 8 up, 8 in 2021-07-31T02:36:27.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:27 smithi045 conmon[39718]: cluster 2021-07-31T02:36:26.505978 2021-07-31T02:36:27.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:27 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 2106 : cluster [DBG] pgmap v2909: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 501 KiB/s rd, 37 KiB/s wr, 676 op/s 2021-07-31T02:36:27.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:27 smithi045 conmon[39718]: cluster 2021-07-31T02:36:27.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:27 smithi045 conmon[39718]: 2021-07-31T02:36:27.410052+0000 mon.a (mon.0) 2577 : cluster 2021-07-31T02:36:27.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:27 smithi045 conmon[39718]: [DBG] osdmap e1587: 8 total, 8 up, 8 in 2021-07-31T02:36:28.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:28 smithi154 conmon[34553]: audit 2021-07-31T02:36:28.107954+0000 mon.b (mon.2) 320 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:36:28.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:28 smithi154 conmon[34553]: audit 2021-07-31T02:36:28.108621+0000 mon.a (mon.0) 2578 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:36:28.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:28 smithi154 conmon[34553]: audit 2021-07-31T02:36:28.128159+0000 mon.b (mon.2) 321 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:36:28.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:28 smithi154 conmon[34553]: audit 2021-07-31T02:36:28.128786+0000 mon.a (mon. 2021-07-31T02:36:28.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:28 smithi154 conmon[34553]: 0) 2579 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:36:28.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:28 smithi045 conmon[33655]: audit 2021-07-31T02:36:28.107954+0000 mon.b (mon.2) 2021-07-31T02:36:28.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:28 smithi045 conmon[33655]: 320 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:36:28.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:28 smithi045 conmon[33655]: audit 2021-07-31T02:36:28.108621+0000 mon.a (mon.0) 2578 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:36:28.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:28 smithi045 conmon[33655]: audit 2021-07 2021-07-31T02:36:28.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:28 smithi045 conmon[33655]: -31T02:36:28.128159+0000 mon.b (mon.2) 321 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:36:28.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:28 smithi045 conmon[33655]: audit 2021-07-31T02:36:28.128786+0000 mon.a (mon.0) 2579 : 2021-07-31T02:36:28.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:28 smithi045 conmon[33655]: audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:36:28.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:28 smithi045 conmon[39718]: audit 2021-07-31T02:36:28.107954+0000 mon.b (mon 2021-07-31T02:36:28.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:28 smithi045 conmon[39718]: .2) 320 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:36:28.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:28 smithi045 conmon[39718]: audit 2021-07-31T02:36:28.108621+0000 mon.a 2021-07-31T02:36:28.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:28 smithi045 conmon[39718]: (mon.0) 2578 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:36:28.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:28 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T02:36:28.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:28 smithi045 conmon[39718]: 02:36:28.128159+0000 mon.b (mon 2021-07-31T02:36:28.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:28 smithi045 conmon[39718]: .2) 321 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:36:28.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:28 smithi045 conmon[39718]: audit 2021- 2021-07-31T02:36:28.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:28 smithi045 conmon[39718]: 07-31T02:36:28.128786+0000 2021-07-31T02:36:28.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:28 smithi045 conmon[39718]: mon.a (mon.0) 2579 : audit [INF] 2021-07-31T02:36:28.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:28 smithi045 conmon[39718]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:36:29.571 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestClone.test_flatten_with_progress ... ok 2021-07-31T02:36:29.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:29 smithi154 conmon[34553]: cluster 2021-07-31T02:36:28.471413+0000 mon.a (mon.0) 2021-07-31T02:36:29.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:29 smithi154 conmon[34553]: 2580 : cluster [DBG] osdmap e1588: 8 total, 8 up, 8 in 2021-07-31T02:36:29.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:29 smithi154 conmon[34553]: cluster 2021-07-31T02:36:28.506343+0000 mgr.x (mgr.24542) 2107 : cluster [DBG] pgmap v2912: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:36:29.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:29 smithi045 conmon[33655]: cluster 2021-07-31T02:36:28.471413+0000 mon.a 2021-07-31T02:36:29.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:29 smithi045 conmon[33655]: (mon.0) 2580 : cluster [DBG] osdmap e1588: 8 total, 8 up, 8 in 2021-07-31T02:36:29.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:29 smithi045 conmon[33655]: cluster 2021-07-31T02:36:28.506343+0000 mgr.x ( 2021-07-31T02:36:29.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:29 smithi045 conmon[33655]: mgr.24542) 2107 : cluster [DBG] pgmap v2912: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:36:29.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:29 smithi045 conmon[39718]: cluster 2021-07-31T02:36:28 2021-07-31T02:36:29.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:29 smithi045 conmon[39718]: .471413+0000 mon.a (mon.0) 2580 : cluster [DBG] osdmap e1588: 8 total, 8 up, 8 in 2021-07-31T02:36:29.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:29 smithi045 conmon[39718]: cluster 2021 2021-07-31T02:36:29.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:29 smithi045 conmon[39718]: -07-31T02:36:28. 2021-07-31T02:36:29.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:29 smithi045 conmon[39718]: 506343+0000 mgr.x (mgr. 2021-07-31T02:36:29.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:29 smithi045 conmon[39718]: 24542) 2107 : cluster [DBG] pgmap v2912: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:36:29.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:29 smithi045 conmon[39718]: 2021-07-31T02:36:30.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:30 smithi154 conmon[34553]: cluster 2021-07-31T02:36:29.477305+0000 mon.a (mon.0) 2581 : cluster [DBG] osdmap e1589: 8 total, 8 up, 8 in 2021-07-31T02:36:30.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:30 smithi045 conmon[33655]: cluster 2021-07-31T02:36:29.477305+0000 mon.a (mon. 2021-07-31T02:36:30.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:30 smithi045 conmon[33655]: 0) 2581 : cluster [DBG] osdmap e1589: 8 total, 8 up, 8 in 2021-07-31T02:36:30.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:30 smithi045 conmon[39718]: cluster 2021-07-31T02:36:29.477305+0000 mon.a (mon.0 2021-07-31T02:36:30.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:30 smithi045 conmon[39718]: ) 2581 : cluster [DBG] osdmap e1589: 8 total, 8 up, 8 in 2021-07-31T02:36:31.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:31 smithi154 conmon[34553]: cluster 2021-07-31T02:36:30.478990+0000 mon.a (mon.0) 2021-07-31T02:36:31.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:31 smithi154 conmon[34553]: 2582 : cluster [DBG] osdmap e1590: 8 total, 8 up, 8 in 2021-07-31T02:36:31.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:31 smithi154 conmon[34553]: cluster 2021-07-31T02:36:30.506785+0000 mgr.x (mgr.24542) 2108 : cluster [DBG] pgmap v2915: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 586 KiB/s rd, 41 KiB/s wr, 791 op/s 2021-07-31T02:36:31.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:31 smithi045 conmon[33655]: cluster 2021-07-31T02:36:30.478990+0000 mon.a (mon.0) 2582 : cluster [DBG] osdmap e1590: 8 total, 8 up, 8 in 2021-07-31T02:36:31.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:31 smithi045 conmon[33655]: cluster 2021-07-31T02:36:30.506785+0000 mgr.x (mgr.24542) 2108 : cluster [DBG] pgmap v2915: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 586 KiB/s rd, 41 KiB/s wr, 791 op/s 2021-07-31T02:36:31.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:31 smithi045 conmon[39718]: cluster 2021-07-31T02:36:30.478990+0000 2021-07-31T02:36:31.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:31 smithi045 conmon[39718]: mon.a (mon.0) 2582 : cluster [DBG] osdmap e1590: 8 total, 8 up, 8 in 2021-07-31T02:36:31.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:31 smithi045 conmon[39718]: cluster 2021-07-31T02:36:30.506785+0000 mgr.x (mgr.24542) 2108 : cluster [DBG] pgmap v2915: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 586 KiB/s rd, 41 KiB/s wr, 791 op/s 2021-07-31T02:36:33.253 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: debug 2021-07-31T02:36:32.870+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:36:33.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:36:33.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:36:33.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: yield 2021-07-31T02:36:33.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:36:33.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:36:33.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:36:33.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:36:33.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:36:33.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:36:33.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:36:33.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:36:33.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:36:33.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: task.result() 2021-07-31T02:36:33.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:36:33.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: debug 2021-07-31T02:36:32.882+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:36:33.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:36:33.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:36:33.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: yield 2021-07-31T02:36:33.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:36:33.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:36:33.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:36:33.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:36:33.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:36:33.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:36:33.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:36:33.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:36:33.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:36:33.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: task.result() 2021-07-31T02:36:33.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:36:33.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: debug 2021-07-31T02:36:32.909+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:36:33.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:36:33.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:36:33.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: yield 2021-07-31T02:36:33.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:36:33.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:36:33.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:36:33.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:36:33.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:36:33.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:36:33.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:36:33.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:36:33.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:36:33.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: task.result() 2021-07-31T02:36:33.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:36:33.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: debug 2021-07-31T02:36:32.909+0000 7fcb97969700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:36:33.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:36:33.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:36:33.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: yield 2021-07-31T02:36:33.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:36:33.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:36:33.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:36:33.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:36:33.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:36:33.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:36:33.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:36:33.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:36:33.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:36:33.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: task.result() 2021-07-31T02:36:33.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:36:33.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: 2021-07-31T02:36:33.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:36:33.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: 2021-07-31T02:36:33.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:36:33.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:36:33.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:36:33.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:36:33.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:36:33.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:36:33.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:36:33.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:36:33.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:36:33.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:36:33.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:36:33.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:36:33.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:36:33.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:36:33.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:36:33.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:36:33.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:36:33.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:36:33.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: raise 2021-07-31T02:36:33.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:36:33.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:36:33.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:36:33.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:36:33.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:36:33.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: debug 2021-07-31T02:36:32.918+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:36:33.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:36:33.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:36:33.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: yield 2021-07-31T02:36:33.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:36:33.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:36:33.284 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:36:33.284 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:36:33.285 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:36:33.285 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:36:33.285 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:36:33.286 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:36:33.286 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:36:33.287 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: task.result() 2021-07-31T02:36:33.287 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:36:33.287 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: debug 2021-07-31T02:36:32.918+0000 7fcb9816a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:36:33.288 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:36:33.288 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:36:33.289 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: yield 2021-07-31T02:36:33.289 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:36:33.289 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:36:33.290 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:36:33.290 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:36:33.291 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:36:33.291 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:36:33.292 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:36:33.292 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:36:33.292 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:36:33.293 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: task.result() 2021-07-31T02:36:33.293 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:36:33.294 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: 2021-07-31T02:36:33.294 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:36:33.294 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: 2021-07-31T02:36:33.295 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:36:33.295 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:36:33.296 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:36:33.296 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:36:33.296 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:36:33.297 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:36:33.297 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:36:33.298 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:36:33.298 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:36:33.299 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:36:33.299 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:36:33.299 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:36:33.300 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:36:33.300 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:36:33.301 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:36:33.301 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:36:33.302 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:36:33.302 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:36:33.302 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: raise 2021-07-31T02:36:33.302 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:36:33.303 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:36:33.303 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:36:33.303 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:36:33.303 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:36:32 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:36:33.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:33 smithi154 conmon[34553]: cluster 2021-07-31T02:36:32. 2021-07-31T02:36:33.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:33 smithi154 conmon[34553]: 409929+0000 mon.a (mon.0) 2583 : cluster [DBG] osdmap e1591: 8 total, 8 up, 8 in 2021-07-31T02:36:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:33 smithi154 conmon[34553]: audit 2021-07-31T02:36:32.826828 2021-07-31T02:36:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:33 smithi154 conmon[34553]: +0000 mon.b (mon.2) 322 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:36:33.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:33 smithi154 conmon[34553]: audit 2021-07-31T02:36:33.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:33 smithi154 conmon[34553]: 2021-07-31T02:36:32.828015 2021-07-31T02:36:33.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:33 smithi154 conmon[34553]: +0000 mon.b (mon.2) 323 : audit 2021-07-31T02:36:33.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:33 smithi154 conmon[34553]: [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:36:33.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:33 smithi154 conmon[34553]: audit 2021-07-31 2021-07-31T02:36:33.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:33 smithi154 conmon[34553]: T02:36:32.828661+0000 mon.b (mon.2) 324 : 2021-07-31T02:36:33.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:33 smithi154 conmon[34553]: audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:36:33.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:33 smithi045 conmon[33655]: cluster 2021-07-31T02:36:32.409929+0000 mon.a (mon.0) 2583 : cluster [DBG] 2021-07-31T02:36:33.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:33 smithi045 conmon[33655]: osdmap e1591: 8 total, 8 up, 8 in 2021-07-31T02:36:33.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:33 smithi045 conmon[33655]: audit 2021-07-31T02:36:32.826828+0000 mon.b (mon.2) 322 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:36:33.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:33 smithi045 conmon[33655]: audit 2021-07-31T02 2021-07-31T02:36:33.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:33 smithi045 conmon[33655]: :36:32.828015+0000 mon.b (mon.2) 323 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:36:33.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:33 smithi045 conmon[33655]: audit 2021-07-31T 2021-07-31T02:36:33.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:33 smithi045 conmon[33655]: 02:36:32.828661+0000 mon.b (mon.2 2021-07-31T02:36:33.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:33 smithi045 conmon[33655]: ) 324 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:36:33.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:33 smithi045 conmon[33655]: 2021-07-31T02:36:33.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:33 smithi045 conmon[39718]: cluster 2021-07-31T02:36:32.409929 2021-07-31T02:36:33.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:33 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2583 : cluster [DBG] osdmap e1591: 8 total, 8 up, 8 in 2021-07-31T02:36:33.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:33 smithi045 conmon[39718]: audit 2021-07-31T02:36:33.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:33 smithi045 conmon[39718]: 2021-07-31T02:36:32.826828+0000 2021-07-31T02:36:33.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:33 smithi045 conmon[39718]: mon.b (mon.2) 322 : audit 2021-07-31T02:36:33.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:33 smithi045 conmon[39718]: [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:36:33.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:33 smithi045 conmon[39718]: audit 2021-07-31T02 2021-07-31T02:36:33.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:33 smithi045 conmon[39718]: :36:32.828015+0000 mon.b ( 2021-07-31T02:36:33.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:33 smithi045 conmon[39718]: mon.2) 323 : audit 2021-07-31T02:36:33.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:33 smithi045 conmon[39718]: [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:36:33.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:33 smithi045 conmon[39718]: audit 2021-07-31T02:36:33.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:33 smithi045 conmon[39718]: 2021-07-31T02:36: 2021-07-31T02:36:33.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:33 smithi045 conmon[39718]: 32.828661+0000 mon.b (mon.2) 2021-07-31T02:36:33.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:33 smithi045 conmon[39718]: 324 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:36:34.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: cluster 2021-07-31T02:36:32.507172+0000 mgr.x (mgr.24542) 2109 2021-07-31T02:36:34.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: : cluster [DBG] pgmap v2917: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 581 KiB/s rd, 41 KiB/s wr, 784 op/s 2021-07-31T02:36:34.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: cephadm 2021-07-31T02:36:32.829273+0000 mgr.x (mgr.24542) 2110 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:36:34.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: cephadm 2021-07- 2021-07-31T02:36:34.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: 31T02:36:32.871471+0000 mgr.x (mgr.24542) 2111 : cephadm [ERR] 2021-07-31T02:36:34.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:36:34.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:36:34.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: yield 2021-07-31T02:36:34.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:36:34.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:36:34.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:36:34.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:36:34.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:36:34.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:36:34.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:36:34.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:36:34.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:36:34.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: task.result() 2021-07-31T02:36:34.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:36:34.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: cephadm 2021-07-31T02:36:32.871836+0000 mgr.x (mgr.24542) 2112 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:36:34.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: cephadm 2021-07-31T02:36:32.883661+0000 mgr.x (mgr.24542) 2113 : cephadm [ERR] 2021-07-31T02:36:34.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:36:34.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:36:34.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: yield 2021-07-31T02:36:34.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:36:34.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:36:34.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:36:34.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:36:34.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:36:34.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:36:34.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:36:34.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:36:34.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:36:34.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: task.result() 2021-07-31T02:36:34.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:36:34.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: 2021-07-31T02:36:34.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: cephadm 2021-07-31T02:36:32.884475+0000 mgr.x (mgr.24542) 2114 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:36:34.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: cephadm 2021-07-31T02:36:32.909957+0000 mgr.x (mgr.24542) 2115 : cephadm 2021-07-31T02:36:34.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: [ERR] 2021-07-31T02:36:34.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:36:34.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:36:34.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: yield 2021-07-31T02:36:34.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:36:34.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:36:34.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:36:34.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:36:34.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:36:34.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:36:34.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:36:34.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:36:34.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:36:34.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: task.result() 2021-07-31T02:36:34.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:36:34.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: cephadm 2021-07-31T02:36:34.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: 2021-07-31T02:36:32.910505+0000 mgr.x (mgr.24542) 2116 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:36:34.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:36:34.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:36:34.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: yield 2021-07-31T02:36:34.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:36:34.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:36:34.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:36:34.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:36:34.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:36:34.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:36:34.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:36:34.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:36:34.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:36:34.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: task.result() 2021-07-31T02:36:34.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:36:34.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: 2021-07-31T02:36:34.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:36:34.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: 2021-07-31T02:36:34.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:36:34.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:36:34.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:36:34.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:36:34.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:36:34.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:36:34.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:36:34.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:36:34.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:36:34.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:36:34.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:36:34.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:36:34.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:36:34.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:36:34.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:36:34.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:36:34.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:36:34.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:36:34.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: raise 2021-07-31T02:36:34.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:36:34.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:36:34.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:36:34.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:36:34.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:36:34.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: cephadm 2021-07-31T02:36:32.919012+0000 mgr.x (mgr.24542) 2117 : cephadm [ERR] 2021-07-31T02:36:34.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:36:34.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:36:34.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: yield 2021-07-31T02:36:34.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:36:34.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:36:34.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:36:34.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:36:34.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:36:34.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:36:34.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:36:34.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:36:34.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:36:34.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: task.result() 2021-07-31T02:36:34.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:36:34.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: cephadm 2021-07-31T02:36:32.919377+0000 mgr.x (mgr.24542) 2118 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:36:34.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:36:34.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:36:34.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: yield 2021-07-31T02:36:34.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:36:34.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:36:34.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: cluster 2021-07-31T02:36:32 2021-07-31T02:36:34.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: .507172+0000 mgr.x (mgr.24542) 2109 : cluster [DBG] pgmap v2917: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 581 KiB/s rd, 41 KiB/s wr, 784 op/s 2021-07-31T02:36:34.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: cephadm 2021-07- 2021-07-31T02:36:34.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: 31T02:36:32.829273+0000 mgr.x (mgr.24542) 2110 : cephadm [WRN] 2021-07-31T02:36:34.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:36:34.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: cephadm 2021-07- 2021-07-31T02:36:34.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: 31T02:36:32.871471+0000 mgr.x (mgr.24542) 2111 2021-07-31T02:36:34.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: : cephadm [ERR] 2021-07-31T02:36:34.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:36:34.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:36:34.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: yield 2021-07-31T02:36:34.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:36:34.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:36:34.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:36:34.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:36:34.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:36:34.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:36:34.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:36:34.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:36:34.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:36:34.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: task.result() 2021-07-31T02:36:34.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:36:34.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: cephadm 2021-07-31T02:36:34.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: 2021-07-31T02:36:32.871836+0000 mgr.x (mgr.24542) 2112 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:36:34.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: cephadm 2021-07-31T02:36:32.883661+0000 mgr.x (mgr.24542) 2113 : cephadm [ERR] 2021-07-31T02:36:34.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:36:34.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:36:34.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: yield 2021-07-31T02:36:34.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:36:34.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:36:34.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:36:34.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:36:34.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:36:34.788 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:36:34.788 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:36:34.788 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:36:34.788 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:36:34.788 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: task.result() 2021-07-31T02:36:34.789 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:36:34.789 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: 2021-07-31T02:36:34.789 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:36:34.790 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:36:34.790 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:36:34.790 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:36:34.790 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:36:34.791 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:36:34.791 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:36:34.791 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: task.result() 2021-07-31T02:36:34.798 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:36:34.799 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: 2021-07-31T02:36:34.799 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:36:34.799 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: 2021-07-31T02:36:34.799 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:36:34.800 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:36:34.800 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:36:34.800 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:36:34.800 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:36:34.800 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:36:34.801 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:36:34.801 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:36:34.801 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:36:34.801 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:36:34.802 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:36:34.802 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:36:34.802 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:36:34.802 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:36:34.802 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:36:34.803 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:36:34.803 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:36:34.803 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:36:34.803 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: raise 2021-07-31T02:36:34.804 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:36:34.804 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:36:34.804 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:36:34.804 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:36:34.804 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:34 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:36:34.805 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: cluster 2021-07-31T02:36:32.507172 2021-07-31T02:36:34.805 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 2109 : cluster [DBG] pgmap v2917: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 581 KiB/s rd, 41 KiB/s wr, 784 op/s 2021-07-31T02:36:34.805 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: cephadm 2021-07-31T02:36: 2021-07-31T02:36:34.805 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: 32.829273+0000 mgr.x (mgr.24542) 2021-07-31T02:36:34.806 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: 2110 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:36:34.806 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: cephadm 2021-07-31T02:36:34.806 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: 2021-07-31T02:36:32. 2021-07-31T02:36:34.806 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: 871471+0000 mgr.x (mgr.24542) 2111 : cephadm 2021-07-31T02:36:34.807 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: [ERR] 2021-07-31T02:36:34.807 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:36:34.807 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:36:34.807 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: yield 2021-07-31T02:36:34.807 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:36:34.808 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:36:34.808 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:36:34.808 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:36:34.808 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:36:34.809 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:36:34.809 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:36:34.809 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:36:34.809 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:36:34.809 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: task.result() 2021-07-31T02:36:34.810 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:36:34.810 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: cephadm 2021-07-31T02:36:32.871836+0000 mgr.x (mgr.24542) 2112 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:36:34.810 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: cephadm 2021-07-31T02:36: 2021-07-31T02:36:34.810 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: 32.883661+0000 mgr.x (mgr.24542) 2113 : cephadm [ERR] 2021-07-31T02:36:34.810 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: 2021-07-31T02:36:34.811 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:36:34.811 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:36:34.811 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: yield 2021-07-31T02:36:34.811 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:36:34.812 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:36:34.812 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:36:34.812 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:36:34.812 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:36:34.812 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:36:34.813 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:36:34.813 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:36:34.813 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:36:34.813 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: task.result() 2021-07-31T02:36:34.813 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:36:34.814 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: cephadm 2021-07-31T02:36:32.884475+0000 mgr.x (mgr.24542) 2114 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:36:34.814 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: cephadm 2021-07-31T02:36:32.909957+0000 mgr.x (mgr.24542) 2115 : cephadm [ERR] 2021-07-31T02:36:34.815 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:36:34.815 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:36:34.815 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: yield 2021-07-31T02:36:34.815 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:36:34.816 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:36:34.816 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:36:34.816 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:36:34.816 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:36:34.816 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:36:34.817 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:36:34.817 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:36:34.817 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:36:34.817 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: task.result() 2021-07-31T02:36:34.818 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:36:34.818 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: cephadm 2021-07-31T02:36:32.910505+0000 mgr.x (mgr.24542) 2116 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:36:34.818 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:36:34.818 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:36:34.818 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: yield 2021-07-31T02:36:34.819 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:36:34.819 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:36:34.819 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:36:34.819 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:36:34.820 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:36:34.820 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:36:34.820 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:36:34.820 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:36:34.820 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:36:34.821 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: task.result() 2021-07-31T02:36:34.821 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:36:34.821 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: 2021-07-31T02:36:34.821 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:36:34.821 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: 2021-07-31T02:36:34.822 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:36:34.822 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:36:34.822 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:36:34.822 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:36:34.823 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:36:34.823 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:36:34.823 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: cephadm 2021-07-31T02:36:32.884475+0000 mgr.x (mgr.24542) 2114 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:36:34.824 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: cephadm 2021-07-31T02:36:32.909957+0000 mgr.x ( 2021-07-31T02:36:34.824 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: mgr.24542) 2115 : cephadm [ERR] 2021-07-31T02:36:34.824 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:36:34.824 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:36:34.824 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: yield 2021-07-31T02:36:34.825 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:36:34.825 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:36:34.825 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:36:34.825 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:36:34.825 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:36:34.826 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:36:34.826 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:36:34.826 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:36:34.826 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:36:34.827 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: task.result() 2021-07-31T02:36:34.827 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:36:34.827 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: cephadm 2021-07-31T02:36:32.910505+0000 mgr.x (mgr.24542) 2116 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:36:34.827 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:36:34.827 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:36:34.828 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: yield 2021-07-31T02:36:34.828 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:36:34.828 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:36:34.828 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:36:34.829 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:36:34.829 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:36:34.829 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:36:34.829 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:36:34.829 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:36:34.830 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:36:34.830 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: task.result() 2021-07-31T02:36:34.830 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:36:34.830 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: 2021-07-31T02:36:34.831 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:36:34.831 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: 2021-07-31T02:36:34.831 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:36:34.831 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:36:34.831 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:36:34.832 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:36:34.832 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:36:34.832 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:36:34.833 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:36:34.833 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:36:34.833 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:36:34.833 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:36:34.834 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:36:34.834 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:36:34.834 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:36:34.834 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:36:34.835 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:36:34.835 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:36:34.835 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:36:34.835 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:36:34.835 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: raise 2021-07-31T02:36:34.836 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:36:34.836 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:36:34.836 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:36:34.836 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:36:34.837 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:36:34.837 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: cephadm 2021-07-31T02:36:32.919012+0000 mgr.x (mgr.24542) 2117 : cephadm [ERR] 2021-07-31T02:36:34.837 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:36:34.837 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:36:34.837 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: yield 2021-07-31T02:36:34.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:36:34.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:36:34.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:36:34.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:36:34.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:36:34.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:36:34.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:36:34.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:36:34.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:36:34.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: task.result() 2021-07-31T02:36:34.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:36:34.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: cephadm 2021-07-31T02:36:32.919377+0000 mgr.x (mgr.24542) 2118 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:36:34.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:36:34.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:36:34.841 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: yield 2021-07-31T02:36:34.841 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:36:34.841 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:36:34.842 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:36:34.842 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:36:34.842 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:36:34.842 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:36:34.843 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:36:34.843 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:36:34.843 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:36:34.843 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:36:34.843 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:36:34.844 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:36:34.844 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:36:34.844 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:36:34.844 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: raise 2021-07-31T02:36:34.845 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:36:34.845 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:36:34.845 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:36:34.845 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:36:34.845 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:36:34.846 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: cephadm 2021-07-31T02:36:32. 2021-07-31T02:36:34.846 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: 919012+0000 mgr.x (mgr.24542) 2117 : cephadm [ERR] 2021-07-31T02:36:34.846 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:36:34.846 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:36:34.847 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: yield 2021-07-31T02:36:34.847 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:36:34.847 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:36:34.847 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:36:34.847 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:36:34.848 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:36:34.848 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:36:34.848 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:36:34.848 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:36:34.848 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:36:34.849 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: task.result() 2021-07-31T02:36:34.849 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:36:34.849 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: cephadm 2021-07-31T02:36:32.919377+0000 mgr.x (mgr.24542) 2118 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:36:34.849 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:36:34.849 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:36:34.850 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: yield 2021-07-31T02:36:34.850 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:36:34.850 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:36:34.851 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:36:34.851 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:36:34.851 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:36:34.851 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:36:34.852 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:36:34.852 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:36:34.852 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: task.result() 2021-07-31T02:36:34.852 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:36:34.853 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: 2021-07-31T02:36:34.853 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:36:34.853 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: 2021-07-31T02:36:34.853 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:36:34.853 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:36:34.854 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:36:34.854 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:36:34.854 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:36:34.854 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:36:34.855 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:36:34.855 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:36:34.855 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:36:34.855 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:36:34.855 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:36:34.856 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:36:34.856 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:36:34.856 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:36:34.856 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:36:34.856 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:36:34.857 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:36:34.857 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:36:34.857 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: raise 2021-07-31T02:36:34.857 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:36:34.858 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:36:34.858 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:36:34.858 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:36:34.858 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:36:34.859 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:36:34.859 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:36:34.859 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:36:34.859 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:36:34.860 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:36:34.860 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:36:34.860 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:36:34.860 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:36:34.860 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: task.result() 2021-07-31T02:36:34.861 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:36:34.861 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: 2021-07-31T02:36:34.861 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:36:34.861 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: 2021-07-31T02:36:34.861 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:36:34.862 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:36:34.862 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:36:34.862 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:36:34.862 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:36:34.863 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:36:34.863 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:36:34.863 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:36:34.863 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:36:34.864 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:36:34.864 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:36:34.864 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:36:34.864 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:36:34.864 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:36:34.865 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:36:34.865 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:36:34.865 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:36:34.865 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:36:34.865 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: raise 2021-07-31T02:36:34.866 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:36:34.866 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:36:34.866 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:36:34.866 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:36:34.867 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:34 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:36:35.510 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestClone.test_list_children ... ok 2021-07-31T02:36:36.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:36 smithi154 conmon[34553]: cluster 2021-07-31T02:36:34.507602+0000 mgr.x (mgr.24542) 2119 : cluster [DBG] pgmap v2918: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 2.0 MiB/s rd, 46 KiB/s wr, 2.49k op/s 2021-07-31T02:36:36.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:36 smithi154 conmon[34553]: 2021-07-31T02:36:36.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:36 smithi154 conmon[34553]: cluster 2021-07-31T02:36:35.422585+0000 mon.a (mon.0) 2584 : cluster [DBG] osdmap e1592: 8 total, 8 up, 8 in 2021-07-31T02:36:36.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:36 smithi045 conmon[39718]: cluster 2021-07-31T02:36:34.507602+0000 mgr.x (mgr.24542) 2119 : cluster 2021-07-31T02:36:36.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:36 smithi045 conmon[39718]: [DBG] pgmap v2918: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 2.0 MiB/s rd, 46 KiB/s wr, 2.49k op/s 2021-07-31T02:36:36.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:36 smithi045 conmon[39718]: cluster 2021-07-31T02:36:35.422585+0000 mon.a (mon.0) 2584 : cluster [DBG] osdmap e1592: 8 total, 8 up, 8 in 2021-07-31T02:36:36.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:36 smithi045 conmon[33655]: cluster 2021-07-31T02:36:34.507602+0000 mgr.x (mgr.24542) 2119 : cluster [DBG] pgmap v2918: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 2.0 MiB/s rd, 46 KiB/s wr, 2.49k op/s 2021-07-31T02:36:36.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:36 smithi045 conmon[33655]: cluster 2021-07-31T02:36:35.422585+0000 mon.a (mon.0) 2584 : cluster [DBG] 2021-07-31T02:36:36.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:36 smithi045 conmon[33655]: osdmap e1592: 8 total, 8 up, 8 in 2021-07-31T02:36:37.512 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestClone.test_read ... ok 2021-07-31T02:36:37.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:37 smithi154 conmon[34553]: cluster 2021-07-31T02:36:36.425867+0000 mon.a (mon.0) 2585 2021-07-31T02:36:37.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:37 smithi154 conmon[34553]: : cluster [DBG] osdmap e1593: 8 total, 8 up, 8 in 2021-07-31T02:36:37.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:37 smithi154 conmon[34553]: cluster 2021-07-31T02:36:36.508129+0000 mgr.x (mgr.24542) 2120 : cluster [DBG] pgmap v2921: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.6 MiB/s rd, 19 KiB/s wr, 1.96k op/s 2021-07-31T02:36:37.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:37 smithi154 conmon[34553]: cluster 2021- 2021-07-31T02:36:37.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:37 smithi154 conmon[34553]: 07-31T02:36:37.412395+0000 mon.a (mon.0) 2586 : cluster [DBG] osdmap e1594: 8 total, 8 up, 8 in 2021-07-31T02:36:37.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:37 smithi045 conmon[33655]: cluster 2021-07-31T02:36:36.425867+0000 mon.a (mon.0) 2021-07-31T02:36:37.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:37 smithi045 conmon[33655]: 2585 : cluster [DBG] osdmap e1593: 8 total, 8 up, 8 in 2021-07-31T02:36:37.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:37 smithi045 conmon[33655]: cluster 2021-07-31T02:36:36.508129+0000 mgr.x (mgr.24542) 2120 : cluster [DBG] pgmap v2921: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.6 MiB/s rd, 19 KiB/s wr, 1.96k op/s 2021-07-31T02:36:37.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:37 smithi045 conmon[33655]: cluster 2021 2021-07-31T02:36:37.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:37 smithi045 conmon[33655]: -07-31T02:36:37.412395+0000 mon.a (mon.0) 2586 : cluster [DBG] 2021-07-31T02:36:37.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:37 smithi045 conmon[33655]: osdmap e1594: 8 total, 8 up, 8 in 2021-07-31T02:36:37.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:37 smithi045 conmon[39718]: cluster 2021-07-31T02:36:36. 2021-07-31T02:36:37.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:37 smithi045 conmon[39718]: 425867+0000 mon.a (mon.0) 2585 : cluster [DBG] osdmap e1593: 8 total, 8 up, 8 in 2021-07-31T02:36:37.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:37 smithi045 conmon[39718]: cluster 2021-07-31T02:36:37.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:37 smithi045 conmon[39718]: 2021-07-31T02:36:36.508129+0000 mgr.x 2021-07-31T02:36:37.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:37 smithi045 conmon[39718]: (mgr.24542) 2120 : cluster [DBG] pgmap v2921: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.6 MiB/s rd, 19 KiB/s wr, 1.96k op/s 2021-07-31T02:36:37.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:37 smithi045 conmon[39718]: 2021-07-31T02:36:37.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:37 smithi045 conmon[39718]: cluster 2021-07- 2021-07-31T02:36:37.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:37 smithi045 conmon[39718]: 31T02:36:37.412395+0000 mon.a (mon. 2021-07-31T02:36:37.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:37 smithi045 conmon[39718]: 0) 2586 : cluster [DBG] osdmap e1594: 8 total, 8 up, 8 in 2021-07-31T02:36:39.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:39 smithi154 conmon[34553]: cluster 2021-07-31T02:36:38.433978+0000 mon.a (mon.0) 2587 : cluster [DBG] osdmap e1595: 8 total, 8 up, 8 in 2021-07-31T02:36:39.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:39 smithi154 conmon[34553]: cluster 2021-07-31T02:36:38.508574+0000 mgr.x (mgr.24542) 2121 : cluster [DBG] pgmap v2924: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:36:39.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:39 smithi045 conmon[33655]: cluster 2021-07-31T02:36:38.433978+0000 mon.a (mon.0) 2587 : cluster [DBG] 2021-07-31T02:36:39.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:39 smithi045 conmon[33655]: osdmap e1595: 8 total, 8 up, 8 in 2021-07-31T02:36:39.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:39 smithi045 conmon[33655]: cluster 2021-07-31T02:36:38.508574+0000 mgr.x (mgr.24542) 2121 : cluster [DBG] 2021-07-31T02:36:39.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:39 smithi045 conmon[33655]: pgmap v2924: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:36:39.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:39 smithi045 conmon[39718]: cluster 2021-07-31T02:36:38.433978+0000 2021-07-31T02:36:39.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:39 smithi045 conmon[39718]: mon.a (mon.0) 2587 : cluster [DBG] osdmap e1595: 8 total, 8 up, 8 in 2021-07-31T02:36:39.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:39 smithi045 conmon[39718]: cluster 2021-07-31T02:36:38.508574+0000 mgr.x (mgr.24542) 2121 : 2021-07-31T02:36:39.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:39 smithi045 conmon[39718]: cluster [DBG] pgmap v2924: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:36:40.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:40 smithi154 conmon[34553]: cluster 2021-07-31T02:36:39.440159+0000 mon.a (mon.0) 2588 : cluster [DBG] 2021-07-31T02:36:40.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:40 smithi154 conmon[34553]: osdmap e1596: 8 total, 8 up, 8 in 2021-07-31T02:36:40.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:40 smithi045 conmon[33655]: cluster 2021-07-31T02:36:39.440159+0000 mon.a (mon.0) 2588 : cluster [DBG] osdmap e1596: 8 total, 8 up, 8 in 2021-07-31T02:36:40.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:40 smithi045 conmon[33655]: 2021-07-31T02:36:40.777 INFO:journalctl@ceph.osd.0.smithi045.stdout:Jul 31 02:36:40 smithi045 conmon[43243]: debug 2021-07-31T02:36:40.519+0000 7fdf6af8e700 -1 osd.0 pg_epoch: 1597 pg[83.1b( v 1596'48 (0'0,1596'48] local-lis/les=1568/1569 n=2 ec=1568/1568 lis/c=1568/1568 les/c/f=1569/1569/0 sis=1568) [0,3,7] r=0 lpr=1568 crt=1596'48 lcod 1596'47 mlcod 1596'47 active+clean+snaptrim trimq=[1b~1] ps=[16~1,18~1]] removing snap head 2021-07-31T02:36:40.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:40 smithi045 conmon[39718]: cluster 2021-07-31T02:36:39.440159+0000 mon.a (mon.0) 2588 : cluster [DBG] osdmap e1596: 8 total, 8 up, 8 in 2021-07-31T02:36:41.540 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestClone.test_resize_flatten_multi_level ... ok 2021-07-31T02:36:41.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:41 smithi154 conmon[34553]: cluster 2021-07-31T02:36:40.445819+0000 mon.a (mon.0) 2589 : cluster [DBG] 2021-07-31T02:36:41.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:41 smithi154 conmon[34553]: osdmap e1597: 8 total, 8 up, 8 in 2021-07-31T02:36:41.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:41 smithi154 conmon[34553]: cluster 2021-07-31T02:36:40.509031+0000 mgr.x (mgr.24542) 2122 : cluster [DBG] pgmap v2927: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 587 KiB/s rd, 24 KiB/s wr, 751 op/s 2021-07-31T02:36:41.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:41 smithi045 conmon[33655]: cluster 2021-07-31T02:36:40.445819+0000 mon.a (mon.0) 2589 : cluster [DBG] osdmap e1597: 8 total, 8 up, 8 in 2021-07-31T02:36:41.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:41 smithi045 conmon[33655]: 2021-07-31T02:36:41.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:41 smithi045 conmon[33655]: cluster 2021-07-31T02:36:40.509031+0000 mgr.x (mgr.24542) 2122 : cluster [DBG] pgmap v2927: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 587 KiB/s rd, 24 KiB/s wr, 751 op/s 2021-07-31T02:36:41.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:41 smithi045 conmon[39718]: cluster 2021-07-31T02:36:40 2021-07-31T02:36:41.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:41 smithi045 conmon[39718]: .445819+0000 mon.a (mon.0) 2589 : cluster [DBG] osdmap e1597: 8 total, 8 up, 8 in 2021-07-31T02:36:41.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:41 smithi045 conmon[39718]: cluster 2021-07-31T02:36:41.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:41 smithi045 conmon[39718]: 2021-07-31T02: 2021-07-31T02:36:41.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:41 smithi045 conmon[39718]: 36:40.509031+0000 mgr.x ( 2021-07-31T02:36:41.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:41 smithi045 conmon[39718]: mgr.24542) 2122 : cluster 2021-07-31T02:36:41.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:41 smithi045 conmon[39718]: [DBG] pgmap v2927: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 587 KiB/s rd, 24 KiB/s wr, 751 op/s 2021-07-31T02:36:42.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:42 smithi154 conmon[34553]: cluster 2021-07-31T02:36:41.451897+0000 mon.a (mon.0) 2590 : cluster 2021-07-31T02:36:42.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:42 smithi154 conmon[34553]: [DBG] osdmap e1598: 8 total, 8 up, 8 in 2021-07-31T02:36:42.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:42 smithi154 conmon[34553]: cluster 2021-07-31T02:36:42.413558+0000 mon.a (mon.0) 2591 : cluster [DBG] osdmap e1599: 8 total, 8 up, 8 in 2021-07-31T02:36:42.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:42 smithi045 conmon[33655]: cluster 2021-07-31T02:36:42.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:42 smithi045 conmon[33655]: 2021-07-31T02:36 2021-07-31T02:36:42.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:42 smithi045 conmon[33655]: :41.451897+0000 mon.a (mon.0) 2590 : cluster [DBG] osdmap e1598: 8 total, 8 up, 8 in 2021-07-31T02:36:42.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:42 smithi045 conmon[33655]: cluster 2021-07- 2021-07-31T02:36:42.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:42 smithi045 conmon[33655]: 31T02:36:42.413558+0000 mon.a (mon.0) 2591 2021-07-31T02:36:42.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:42 smithi045 conmon[33655]: : cluster [DBG] osdmap e1599: 8 total, 8 up, 8 in 2021-07-31T02:36:42.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:42 smithi045 conmon[39718]: cluster 2021-07-31T02:36:41 2021-07-31T02:36:42.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:42 smithi045 conmon[39718]: .451897+0000 mon.a (mon.0) 2590 : cluster [DBG] osdmap e1598: 8 total, 8 up, 8 in 2021-07-31T02:36:42.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:42 smithi045 conmon[39718]: cluster 2021 2021-07-31T02:36:42.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:42 smithi045 conmon[39718]: -07-31T02:36:42 2021-07-31T02:36:42.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:42 smithi045 conmon[39718]: .413558+0000 mon.a (mon. 2021-07-31T02:36:42.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:42 smithi045 conmon[39718]: 0) 2591 : cluster [DBG] 2021-07-31T02:36:42.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:42 smithi045 conmon[39718]: osdmap e1599: 8 total, 8 up, 8 in 2021-07-31T02:36:43.551 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestClone.test_resize_io ... ok 2021-07-31T02:36:43.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:43 smithi154 conmon[34553]: cluster 2021-07-31T02:36:42.509413+0000 mgr.x (mgr 2021-07-31T02:36:43.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:43 smithi154 conmon[34553]: .24542) 2123 : cluster [DBG] pgmap v2930: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 587 KiB/s rd, 24 KiB/s wr, 751 op/s 2021-07-31T02:36:43.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:43 smithi045 conmon[39718]: cluster 2021-07-31T02:36:43.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:43 smithi045 conmon[39718]: 2021-07-31T02:36 2021-07-31T02:36:43.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:43 smithi045 conmon[39718]: :42.509413+0000 mgr.x (mgr.24542) 2123 : cluster [DBG] pgmap v2930: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 587 KiB/s rd, 24 KiB/s wr, 751 op/s 2021-07-31T02:36:43.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:43 smithi045 conmon[33655]: cluster 2021-07-31T02:36:42.509413+0000 mgr.x (mgr.24542) 2123 : cluster [DBG] pgmap v2930: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 587 KiB/s rd, 24 KiB/s wr, 751 op/s 2021-07-31T02:36:43.777 INFO:journalctl@ceph.osd.2.smithi045.stdout:Jul 31 02:36:43 smithi045 conmon[50337]: debug 2021-07-31T02:36:43.463+0000 7f95dcc7f700 -1 osd.2 pg_epoch: 1600 pg[83.f( v 1599'92 (0'0,1599'92] local-lis/les=1568/1569 n=2 ec=1568/1568 lis/c=1568/1568 les/c/f=1569/1569/0 sis=1568) [2,4,3] r=0 lpr=1568 crt=1599'92 lcod 1598'90 mlcod 1598'90 active+clean+snaptrim trimq=[1e~1] ps=[1a~2]] removing snap head 2021-07-31T02:36:44.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:44 smithi154 conmon[34553]: cluster 2021-07-31T02:36:43.460048+0000 mon.a (mon.0) 2592 : cluster [DBG] osdmap e1600: 8 total, 8 up, 8 in 2021-07-31T02:36:44.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:44 smithi045 conmon[39718]: cluster 2021-07-31T02:36:43.460048 2021-07-31T02:36:44.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:44 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2592 : cluster [DBG] osdmap e1600: 8 total, 8 up, 8 in 2021-07-31T02:36:44.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:44 smithi045 conmon[33655]: cluster 2021-07-31T02:36:43.460048+0000 mon.a (mon.0) 2592 : 2021-07-31T02:36:44.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:44 smithi045 conmon[33655]: cluster [DBG] osdmap e1600: 8 total, 8 up, 8 in 2021-07-31T02:36:45.558 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestClone.test_resize_stat ... ok 2021-07-31T02:36:45.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:45 smithi154 conmon[34553]: cluster 2021-07-31T02:36:44.465289+0000 mon.a (mon.0) 2021-07-31T02:36:45.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:45 smithi154 conmon[34553]: 2593 : cluster [DBG] osdmap e1601: 8 total, 8 up, 8 in 2021-07-31T02:36:45.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:45 smithi154 conmon[34553]: cluster 2021-07-31T02:36:44.509809+0000 mgr.x (mgr.24542) 2124 : cluster [DBG] pgmap v2933: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 352 KiB/s rd, 28 KiB/s wr, 475 op/s 2021-07-31T02:36:45.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:45 smithi045 conmon[33655]: cluster 2021-07-31T02:36:44.465289+0000 mon.a (mon.0) 2593 : cluster 2021-07-31T02:36:45.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:45 smithi045 conmon[33655]: [DBG] osdmap e1601: 8 total, 8 up, 8 in 2021-07-31T02:36:45.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:45 smithi045 conmon[33655]: cluster 2021-07-31T02:36:44.509809+0000 mgr.x (mgr.24542) 2124 : cluster [DBG] pgmap v2933: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 352 KiB/s rd, 28 KiB/s wr, 475 op/s 2021-07-31T02:36:45.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:45 smithi045 conmon[39718]: cluster 2021-07-31T02:36:44.465289+0000 mon.a (mon 2021-07-31T02:36:45.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:45 smithi045 conmon[39718]: .0) 2593 : cluster [DBG] osdmap e1601: 8 total, 8 up, 8 in 2021-07-31T02:36:45.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:45 smithi045 conmon[39718]: cluster 2021-07-31T02:36:44.509809+0000 mgr.x (mgr.24542 2021-07-31T02:36:45.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:45 smithi045 conmon[39718]: ) 2124 : cluster [DBG] pgmap v2933: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 352 KiB/s rd, 28 KiB/s wr, 475 op/s 2021-07-31T02:36:46.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:46 smithi154 conmon[34553]: cluster 2021-07-31T02:36:45.471244+0000 mon.a (mon.0) 2594 : cluster [DBG] osdmap e1602: 8 total, 8 up, 8 in 2021-07-31T02:36:46.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:46 smithi045 conmon[33655]: cluster 2021-07-31T02:36:45.471244+0000 mon.a (mon.0) 2594 : cluster [DBG] osdmap e1602: 8 total, 8 up, 8 in 2021-07-31T02:36:46.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:46 smithi045 conmon[39718]: cluster 2021-07-31T02:36:45.471244+0000 mon.a (mon.0) 2594 : cluster [DBG] osdmap e1602: 8 total, 8 up, 8 in 2021-07-31T02:36:47.523 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestClone.test_stat ... ok 2021-07-31T02:36:47.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:47 smithi154 conmon[34553]: cluster 2021-07-31T02:36:47.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:47 smithi154 conmon[34553]: 2021-07-31T02:36:47.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:47 smithi154 conmon[34553]: 2021 2021-07-31T02:36:47.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:47 smithi154 conmon[34553]: -07-31T02:36:46.477140+0000 2021-07-31T02:36:47.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:47 smithi154 conmon[34553]: mon.a (mon.0) 2595 : cluster [DBG] osdmap e1603: 8 total, 8 up, 8 in 2021-07-31T02:36:47.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:47 smithi154 conmon[34553]: cluster 2021-07-31T02:36:47.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:47 smithi154 conmon[34553]: 2021-07-31T02:36:46.510211+0000 mgr.x (mgr.24542) 2125 : 2021-07-31T02:36:47.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:47 smithi154 conmon[34553]: cluster [DBG] pgmap v2936: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 352 KiB/s rd, 28 KiB/s wr, 475 op/s 2021-07-31T02:36:47.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:47 smithi154 conmon[34553]: cluster 2021-07-31T02:36:47.415253 2021-07-31T02:36:47.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:47 smithi154 conmon[34553]: +0000 mon.a (mon.0) 2596 : cluster [DBG] osdmap e1604: 8 total, 8 up, 8 in 2021-07-31T02:36:47.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:47 smithi045 conmon[33655]: cluster 2021-07-31T02:36:46.477140+0000 mon.a (mon.0) 2595 : cluster [DBG] osdmap e1603: 8 total, 8 up, 8 in 2021-07-31T02:36:47.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:47 smithi045 conmon[33655]: cluster 2021-07-31T02:36:46.510211+0000 mgr.x (mgr.24542) 2125 : cluster [DBG] pgmap v2936: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 352 KiB/s rd, 28 KiB/s wr, 475 op/s 2021-07-31T02:36:47.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:47 smithi045 conmon[33655]: cluster 2021-07-31T02:36:47.415253+0000 mon.a (mon.0) 2596 : cluster [DBG] osdmap e1604: 8 total, 8 up, 8 in 2021-07-31T02:36:47.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:47 smithi045 conmon[39718]: cluster 2021-07-31T02:36:46.477140+0000 2021-07-31T02:36:47.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:47 smithi045 conmon[39718]: mon.a (mon.0) 2595 : cluster [DBG] osdmap e1603: 8 total, 8 up, 8 in 2021-07-31T02:36:47.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:47 smithi045 conmon[39718]: cluster 2021-07-31T02:36:46.510211 2021-07-31T02:36:47.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:47 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 2125 : cluster [DBG] pgmap v2936: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 352 KiB/s rd, 28 KiB/s wr, 475 op/s 2021-07-31T02:36:47.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:47 smithi045 conmon[39718]: cluster 2021- 2021-07-31T02:36:47.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:47 smithi045 conmon[39718]: 07-31T02:36:47.415253+0000 mon.a (mon.0) 2596 : cluster [DBG] osdmap e1604: 8 total, 8 up, 8 in 2021-07-31T02:36:49.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:49 smithi154 conmon[34553]: cluster 2021-07-31T02:36:48.484740+0000 mon.a (mon.0) 2597 2021-07-31T02:36:49.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:49 smithi154 conmon[34553]: : cluster [DBG] osdmap e1605: 8 total, 8 up, 8 in 2021-07-31T02:36:49.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:49 smithi154 conmon[34553]: cluster 2021-07-31T02:36:48.510568+0000 mgr.x (mgr.24542) 2126 : cluster [DBG] pgmap v2939: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:36:49.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:49 smithi045 conmon[33655]: cluster 2021-07-31T02:36:48.484740+0000 mon.a (mon.0) 2021-07-31T02:36:49.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:49 smithi045 conmon[33655]: 2597 : cluster [DBG] osdmap e1605: 8 total, 8 up, 8 in 2021-07-31T02:36:49.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:49 smithi045 conmon[33655]: cluster 2021-07-31T02:36:48.510568+0000 mgr.x (mgr.24542) 2126 : cluster [DBG] pgmap v2939: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:36:49.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:49 smithi045 conmon[39718]: cluster 2021-07-31T02:36:48.484740+0000 mon.a (mon.0 2021-07-31T02:36:49.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:49 smithi045 conmon[39718]: ) 2597 : cluster [DBG] osdmap e1605: 8 total, 8 up, 8 in 2021-07-31T02:36:49.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:49 smithi045 conmon[39718]: cluster 2021-07-31T02:36:48.510568+0000 mgr.x 2021-07-31T02:36:49.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:49 smithi045 conmon[39718]: (mgr.24542) 2126 : cluster [DBG] pgmap v2939: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:36:50.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:50 smithi154 conmon[34553]: cluster 2021-07-31T02:36:49.490527+0000 mon.a (mon.0) 2021-07-31T02:36:50.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:50 smithi154 conmon[34553]: 2598 : cluster [DBG] osdmap e1606: 8 total, 8 up, 8 in 2021-07-31T02:36:50.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:50 smithi045 conmon[33655]: cluster 2021-07-31T02:36:49.490527+0000 mon.a (mon. 2021-07-31T02:36:50.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:50 smithi045 conmon[33655]: 0) 2598 : cluster [DBG] osdmap e1606: 8 total, 8 up, 8 in 2021-07-31T02:36:50.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:50 smithi045 conmon[39718]: cluster 2021-07-31T02:36:50.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:50 smithi045 conmon[39718]: 2021-07-31T02:36:50.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:50 smithi045 conmon[39718]: 2021-07-31T02:36:49.490527+0000 mon.a (mon.0) 2021-07-31T02:36:50.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:50 smithi045 conmon[39718]: 2598 : cluster [DBG] osdmap e1606: 8 total, 8 up, 8 in 2021-07-31T02:36:51.600 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestClone.test_stripe_unit_and_count ... ok 2021-07-31T02:36:51.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:51 smithi154 conmon[34553]: cluster 2021-07-31T02:36:50.492400+0000 mon.a (mon.0) 2599 : cluster [DBG] osdmap e1607: 8 total, 8 up, 8 in 2021-07-31T02:36:51.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:51 smithi154 conmon[34553]: cluster 2021-07-31T02:36:50.510933+0000 mgr.x (mgr.24542) 2127 : cluster [DBG] pgmap v2942: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 294 KiB/s rd, 23 KiB/s wr, 398 op/s 2021-07-31T02:36:51.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:51 smithi045 conmon[39718]: cluster 2021-07-31T02:36:50.492400+0000 2021-07-31T02:36:51.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:51 smithi045 conmon[39718]: mon.a (mon.0) 2599 : cluster [DBG] osdmap e1607: 8 total, 8 up, 8 in 2021-07-31T02:36:51.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:51 smithi045 conmon[39718]: cluster 2021-07- 2021-07-31T02:36:51.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:51 smithi045 conmon[39718]: 31T02:36:50.510933+0000 mgr.x (mgr.24542) 2021-07-31T02:36:51.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:51 smithi045 conmon[39718]: 2127 : cluster [DBG] pgmap v2942: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 294 KiB/s rd, 23 KiB/s wr, 398 op/s 2021-07-31T02:36:51.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:51 smithi045 conmon[33655]: cluster 2021-07-31T02:36:50.492400+0000 mon.a (mon.0) 2599 : 2021-07-31T02:36:51.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:51 smithi045 conmon[33655]: cluster [DBG] osdmap e1607: 8 total, 8 up, 8 in 2021-07-31T02:36:51.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:51 smithi045 conmon[33655]: cluster 2021-07-31T02:36:50.510933+0000 mgr.x (mgr.24542) 2127 : cluster [DBG] pgmap v2942: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 294 KiB/s rd, 23 KiB/s wr, 398 op/s 2021-07-31T02:36:52.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:52 smithi154 conmon[34553]: cluster 2021-07-31T02:36:51.503401+0000 mon.a (mon.0) 2600 : cluster [DBG] osdmap e1608: 8 total, 8 up, 8 in 2021-07-31T02:36:52.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:52 smithi154 conmon[34553]: cluster 2021-07-31T02: 2021-07-31T02:36:52.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:52 smithi154 conmon[34553]: 36:52.417664+0000 mon.a (mon.0) 2601 : cluster [DBG] osdmap e1609: 8 total, 8 up, 8 in 2021-07-31T02:36:52.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:52 smithi045 conmon[33655]: cluster 2021-07-31T02:36:51.503401 2021-07-31T02:36:52.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:52 smithi045 conmon[33655]: +0000 mon.a (mon.0) 2600 : cluster [DBG] osdmap e1608: 8 total, 8 up, 8 in 2021-07-31T02:36:52.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:52 smithi045 conmon[33655]: cluster 2021-07-31T02:36:52.417664+0000 mon.a (mon.0) 2601 : cluster [DBG] osdmap e1609: 8 total, 8 up, 8 in 2021-07-31T02:36:52.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:52 smithi045 conmon[39718]: cluster 2021-07-31T02:36:51.503401+0000 mon.a 2021-07-31T02:36:52.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:52 smithi045 conmon[39718]: (mon.0) 2600 : cluster [DBG] osdmap e1608: 8 total, 8 up, 8 in 2021-07-31T02:36:52.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:52 smithi045 conmon[39718]: cluster 2021-07-31T02:36:52.417664+0000 2021-07-31T02:36:52.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:52 smithi045 conmon[39718]: mon.a (mon.0) 2601 : cluster [DBG] osdmap e1609: 8 total, 8 up, 8 in 2021-07-31T02:36:53.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:53 smithi045 conmon[39718]: cluster 2021-07-31T02:36:52.511276+0000 mgr.x (mgr.24542) 2021-07-31T02:36:53.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:53 smithi045 conmon[39718]: 2128 : cluster [DBG] pgmap v2945: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 294 KiB/s rd, 23 KiB/s wr, 398 op/s 2021-07-31T02:36:53.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:53 smithi045 conmon[33655]: cluster 2021-07-31T02:36:52.511276+0000 mgr.x (mgr.24542 2021-07-31T02:36:53.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:53 smithi045 conmon[33655]: ) 2128 : cluster [DBG] pgmap v2945: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 294 KiB/s rd, 23 KiB/s wr, 398 op/s 2021-07-31T02:36:54.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:53 smithi154 conmon[34553]: cluster 2021-07-31T02:36:52.511276+0000 mgr.x (mgr.24542) 2128 : cluster [DBG] pgmap v2945: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 294 KiB/s rd, 23 KiB/s wr, 398 op/s 2021-07-31T02:36:54.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:54 smithi045 conmon[39718]: cluster 2021-07-31T02:36:53.511420 2021-07-31T02:36:54.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:54 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2602 : cluster [DBG] osdmap e1610: 8 total, 8 up, 8 in 2021-07-31T02:36:54.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:54 smithi045 conmon[33655]: cluster 2021-07-31T02:36:53.511420+0000 2021-07-31T02:36:54.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:54 smithi045 conmon[33655]: mon.a (mon.0) 2602 : cluster [DBG] osdmap e1610: 8 total, 8 up, 8 in 2021-07-31T02:36:55.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:54 smithi154 conmon[34553]: cluster 2021-07-31T02:36:53.511420+0000 mon.a (mon.0) 2602 : cluster [DBG] osdmap e1610: 8 total, 8 up, 8 in 2021-07-31T02:36:55.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:54 smithi154 conmon[34553]: 2021-07-31T02:36:55.619 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestClone.test_trash_snapshot ... ok 2021-07-31T02:36:55.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:55 smithi045 conmon[39718]: cluster 2021-07-31T02:36:54.511860+0000 mgr.x (mgr 2021-07-31T02:36:55.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:55 smithi045 conmon[39718]: .24542) 2129 : cluster [DBG] pgmap v2947: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 304 KiB/s rd, 20 KiB/s wr, 406 op/s 2021-07-31T02:36:55.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:55 smithi045 conmon[39718]: cluster 2021-07-31T02:36:54.513069+0000 mon.a (mon.0) 2603 : cluster [DBG] osdmap e1611: 8 total, 8 up, 8 in 2021-07-31T02:36:55.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:55 smithi045 conmon[33655]: cluster 2021-07-31T02:36:54.511860+0000 2021-07-31T02:36:55.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:55 smithi045 conmon[33655]: mgr.x (mgr.24542) 2129 : cluster [DBG] pgmap v2947: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 304 KiB/s rd, 20 KiB/s wr, 406 op/s 2021-07-31T02:36:55.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:55 smithi045 conmon[33655]: cluster 2021-07-31T02:36:54.513069+0000 2021-07-31T02:36:55.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:55 smithi045 conmon[33655]: mon.a (mon.0) 2603 : cluster [DBG] osdmap e1611: 8 total, 8 up, 8 in 2021-07-31T02:36:56.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:55 smithi154 conmon[34553]: cluster 2021-07-31T02:36:54.511860+0000 mgr.x (mgr.24542) 2129 : cluster [DBG] pgmap v2947: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 304 KiB/s rd, 20 KiB/s wr, 406 op/s 2021-07-31T02:36:56.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:55 smithi154 conmon[34553]: cluster 2021-07-31T02:36:54.513069+0000 mon.a (mon.0) 2603 : cluster [DBG] osdmap e1611: 8 total, 8 up, 8 in 2021-07-31T02:36:56.150 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:36:56.425 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:36:56.426 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 51m ago 52m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:36:56.427 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 49m ago 52m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:36:56.427 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 49m ago 55m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:36:56.428 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 51m ago 57m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:36:56.428 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 51m ago 58m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:36:56.429 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 49m ago 56m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:36:56.429 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 51m ago 56m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:36:56.430 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 51m ago 52m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:36:56.430 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 49m ago 52m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:36:56.431 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 51m ago 55m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:36:56.431 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 51m ago 55m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:36:56.432 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 51m ago 54m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:36:56.432 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 51m ago 54m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:36:56.432 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 49m ago 54m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:36:56.433 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 49m ago 53m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:36:56.433 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 49m ago 53m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:36:56.434 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 49m ago 53m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:36:56.434 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 49m ago 52m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:36:57.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:56 smithi154 conmon[34553]: cluster 2021-07-31T02:36:55.523036+0000 mon.a (mon.0) 2604 : cluster [DBG] osdmap e1612: 8 total, 8 up, 8 in 2021-07-31T02:36:57.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:56 smithi154 conmon[34553]: audit 2021-07-31T02:36:56.138651+0000 mgr.x (mgr.24542) 2130 : audit [DBG] from='client.27568 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:36:57.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:56 smithi154 conmon[34553]: audit 2021-07-31T02:36: 2021-07-31T02:36:57.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:56 smithi154 conmon[34553]: 56.417330+0000 mgr.x (mgr.24542) 2131 : audit [DBG] from='client.17634 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:36:57.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:56 smithi045 conmon[33655]: cluster 2021-07-31T02:36:55.523036+0000 mon.a ( 2021-07-31T02:36:57.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:56 smithi045 conmon[33655]: mon.0) 2604 : cluster [DBG] osdmap e1612: 8 total, 8 up, 8 in 2021-07-31T02:36:57.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:56 smithi045 conmon[33655]: audit 2021-07-31T02:36:56.138651+0000 mgr.x (mgr.24542) 2021-07-31T02:36:57.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:56 smithi045 conmon[33655]: 2130 : audit [DBG] from='client.27568 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:36:57.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:56 smithi045 conmon[33655]: audit 2021-07-31T 2021-07-31T02:36:57.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:56 smithi045 conmon[33655]: 02:36:56.417330+0000 mgr.x (mgr.24542) 2131 2021-07-31T02:36:57.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:56 smithi045 conmon[33655]: : audit [DBG] from='client.17634 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:36:57.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:56 smithi045 conmon[39718]: cluster 2021-07-31T02:36:55 2021-07-31T02:36:57.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:56 smithi045 conmon[39718]: .523036+0000 mon.a (mon.0) 2604 : cluster [DBG] osdmap e1612: 8 total, 8 up, 8 in 2021-07-31T02:36:57.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:56 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T02:36:57.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:56 smithi045 conmon[39718]: 02:36:56.138651+0000 mgr.x ( 2021-07-31T02:36:57.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:56 smithi045 conmon[39718]: mgr.24542) 2130 : audit [DBG] from='client.27568 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:36:57.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:56 smithi045 conmon[39718]: audit 2021-07-31T02:36:57.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:56 smithi045 conmon[39718]: 2021-07-31T02: 2021-07-31T02:36:57.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:56 smithi045 conmon[39718]: 36:56.417330+0000 mgr.x (mgr 2021-07-31T02:36:57.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:56 smithi045 conmon[39718]: .24542) 2131 : audit [DBG] from='client.17634 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:36:57.059 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:36:57.059 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:36:57.060 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:36:57.060 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:36:57.060 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:36:57.060 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:36:57.061 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:36:57.061 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:36:57.061 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:36:57.061 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:36:57.062 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:36:57.062 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:36:57.062 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:36:57.063 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:36:57.063 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:36:57.063 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:36:57.063 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:36:58.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:57 smithi154 conmon[34553]: cluster 2021-07-31T02:36:56.512341+0000 mgr.x (mgr.24542) 2132 : cluster [DBG] pgmap v2950: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 298 KiB/s rd, 20 KiB/s wr, 399 op/s 2021-07-31T02:36:58.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:57 smithi154 conmon[34553]: cluster 2021-07-31T02:36:56.530690+0000 mon.a (mon.0) 2605 : cluster [DBG] osdmap e1613: 8 total, 8 up, 8 in 2021-07-31T02:36:58.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:57 smithi154 conmon[34553]: audit 2021-07-31T02:36:57.059582+0000 mon.a (mon.0) 2606 : audit [DBG] from='client.? 172.21.15.45:0/1302041036' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:36:58.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:57 smithi154 conmon[34553]: cluster 2021-07-31T02:36: 2021-07-31T02:36:58.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:57 smithi154 conmon[34553]: 57.418578+0000 mon.a (mon.0) 2607 2021-07-31T02:36:58.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:57 smithi154 conmon[34553]: : cluster [DBG] osdmap e1614: 8 total, 8 up, 8 in 2021-07-31T02:36:58.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:57 smithi154 conmon[34553]: audit 2021-07 2021-07-31T02:36:58.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:57 smithi154 conmon[34553]: -31T02:36:57.420003+0000 mon.c (mon 2021-07-31T02:36:58.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:57 smithi154 conmon[34553]: .1) 125 : audit [INF] from='client.? 172.21.15.45:0/530405139' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rbd-api-smithi045-82607-2","app": "rbd"}]: dispatch 2021-07-31T02:36:58.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:57 smithi154 conmon[34553]: 2021-07-31T02:36:58.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:57 smithi154 conmon[34553]: audit 2021- 2021-07-31T02:36:58.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:57 smithi154 conmon[34553]: 07-31T02:36:57.427176 2021-07-31T02:36:58.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:57 smithi154 conmon[34553]: +0000 mon.a (mon.0) 2608 2021-07-31T02:36:58.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:57 smithi154 conmon[34553]: : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rbd-api-smithi045-82607-2","app": "rbd"}]: dispatch 2021-07-31T02:36:58.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:57 smithi045 conmon[33655]: cluster 2021-07-31T02:36:56.512341+0000 mgr.x 2021-07-31T02:36:58.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:57 smithi045 conmon[33655]: (mgr.24542) 2132 : cluster [DBG] pgmap v2950: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 298 KiB/s rd, 20 KiB/s wr, 399 op/s 2021-07-31T02:36:58.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:57 smithi045 conmon[33655]: cluster 2021-07-31T02:36:56.530690+0000 mon.a (mon.0) 2605 : cluster [DBG] osdmap e1613: 8 total, 8 up, 8 in 2021-07-31T02:36:58.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:57 smithi045 conmon[33655]: audit 2021-07-31T02:36:58.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:57 smithi045 conmon[33655]: 2021-07-31T02:36:57.059582+0000 mon.a (mon.0) 2606 : audit [DBG] 2021-07-31T02:36:58.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:57 smithi045 conmon[33655]: from='client.? 172.21.15.45:0/1302041036' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:36:58.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:57 smithi045 conmon[33655]: cluster 2021-07- 2021-07-31T02:36:58.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:57 smithi045 conmon[33655]: 31T02:36:57.418578+0000 mon.a (mon.0) 2607 : 2021-07-31T02:36:58.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:57 smithi045 conmon[33655]: cluster [DBG] osdmap e1614: 8 total, 8 up, 8 in 2021-07-31T02:36:58.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:57 smithi045 conmon[33655]: audit 2021-07-31T02:36:57. 2021-07-31T02:36:58.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:57 smithi045 conmon[33655]: 420003+0000 mon.c (mon.1) 125 : audit [INF] from='client.? 172.21.15.45:0/530405139' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rbd-api-smithi045-82607-2","app": "rbd"}]: dispatch 2021-07-31T02:36:58.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:57 smithi045 conmon[33655]: audit 2021-07- 2021-07-31T02:36:58.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:57 smithi045 conmon[33655]: 31T02:36:57.427176+0000 mon.a (mon.0) 2608 : audit 2021-07-31T02:36:58.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:57 smithi045 conmon[33655]: [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rbd-api-smithi045-82607-2","app": "rbd"}]: dispatch 2021-07-31T02:36:58.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:57 smithi045 conmon[39718]: cluster 2021-07-31T02: 2021-07-31T02:36:58.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:57 smithi045 conmon[39718]: 36:56.512341+0000 mgr.x (mgr.24542) 2132 : cluster [DBG] pgmap v2950: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 298 KiB/s rd, 20 KiB/s wr, 399 op/s 2021-07-31T02:36:58.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:57 smithi045 conmon[39718]: cluster 2021 2021-07-31T02:36:58.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:57 smithi045 conmon[39718]: -07-31T02:36:56. 2021-07-31T02:36:58.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:57 smithi045 conmon[39718]: 530690+0000 mon.a (mon.0 2021-07-31T02:36:58.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:57 smithi045 conmon[39718]: ) 2605 : cluster [DBG] osdmap e1613: 8 total, 8 up, 8 in 2021-07-31T02:36:58.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:57 smithi045 conmon[39718]: 2021-07-31T02:36:58.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:57 smithi045 conmon[39718]: audit 2021 2021-07-31T02:36:58.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:57 smithi045 conmon[39718]: -07-31T02:36: 2021-07-31T02:36:58.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:57 smithi045 conmon[39718]: 57.059582+0000 mon.a (mon 2021-07-31T02:36:58.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:57 smithi045 conmon[39718]: .0) 2606 : audit [DBG] 2021-07-31T02:36:58.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:57 smithi045 conmon[39718]: from='client.? 172.21.15.45:0/1302041036' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:36:58.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:57 smithi045 conmon[39718]: cluster 2021 2021-07-31T02:36:58.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:57 smithi045 conmon[39718]: -07-31T02:36:57 2021-07-31T02:36:58.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:57 smithi045 conmon[39718]: .418578+0000 mon.a (mon 2021-07-31T02:36:58.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:57 smithi045 conmon[39718]: .0) 2607 : cluster [DBG] 2021-07-31T02:36:58.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:57 smithi045 conmon[39718]: osdmap e1614: 8 total, 8 up, 8 in 2021-07-31T02:36:58.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:57 smithi045 conmon[39718]: audit 2021 2021-07-31T02:36:58.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:57 smithi045 conmon[39718]: -07-31T02:36:57 2021-07-31T02:36:58.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:57 smithi045 conmon[39718]: .420003+0000 mon.c (mon.1) 125 : audit 2021-07-31T02:36:58.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:57 smithi045 conmon[39718]: [INF] from='client.? 172.21.15.45:0/530405139' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rbd-api-smithi045-82607-2","app": "rbd"}]: dispatch 2021-07-31T02:36:58.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:57 smithi045 conmon[39718]: audit 2021 2021-07-31T02:36:58.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:57 smithi045 conmon[39718]: -07-31T02:36: 2021-07-31T02:36:58.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:57 smithi045 conmon[39718]: 57.427176+0000 mon.a ( 2021-07-31T02:36:58.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:57 smithi045 conmon[39718]: mon.0) 2608 : audit 2021-07-31T02:36:58.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:57 smithi045 conmon[39718]: [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rbd-api-smithi045-82607-2","app": "rbd"}]: dispatch 2021-07-31T02:36:59.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:59 smithi154 conmon[34553]: audit 2021-07-31T02:36:58.421138+0000 mon.a (mon.0) 2609 : audit [INF] 2021-07-31T02:36:59.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:59 smithi154 conmon[34553]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rbd-api-smithi045-82607-2","app": "rbd"}]': finished 2021-07-31T02:36:59.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:36:59 smithi154 conmon[34553]: cluster 2021-07-31T02:36:58.421393+0000 mon.a (mon.0) 2610 : cluster [DBG] osdmap e1615: 8 total, 8 up, 8 in 2021-07-31T02:36:59.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:59 smithi045 conmon[33655]: audit 2021-07-31T02:36:58.421138+0000 mon.a (mon.0) 2609 : audit 2021-07-31T02:36:59.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:59 smithi045 conmon[33655]: [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rbd-api-smithi045-82607-2","app": "rbd"}]': finished 2021-07-31T02:36:59.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:36:59 smithi045 conmon[33655]: cluster 2021-07-31T02:36:58.421393+0000 mon.a (mon.0) 2610 : cluster [DBG] osdmap e1615: 8 total, 8 up, 8 in 2021-07-31T02:36:59.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:59 smithi045 conmon[39718]: audit 2021-07-31T02:36:58. 2021-07-31T02:36:59.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:59 smithi045 conmon[39718]: 421138+0000 mon.a (mon.0) 2609 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rbd-api-smithi045-82607-2","app": "rbd"}]': finished 2021-07-31T02:36:59.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:59 smithi045 conmon[39718]: cluster 2021-07- 2021-07-31T02:36:59.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:59 smithi045 conmon[39718]: 31T02:36:58.421393+0000 mon.a (mon.0) 2610 : 2021-07-31T02:36:59.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:36:59 smithi045 conmon[39718]: cluster [DBG] osdmap e1615: 8 total, 8 up, 8 in 2021-07-31T02:37:00.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:00 smithi154 conmon[34553]: cluster 2021-07-31T02:36:58.512760+0000 mgr.x (mgr.24542) 2133 : cluster [DBG] pgmap v2954: 73 pgs: 32 unknown, 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:37:00.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:00 smithi154 conmon[34553]: 2021-07-31T02:37:00.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:00 smithi154 conmon[34553]: cluster 2021-07-31T02:36:59.426430+0000 mon.a (mon.0) 2611 : cluster [DBG] osdmap e1616: 8 total, 8 up, 8 in 2021-07-31T02:37:00.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:00 smithi045 conmon[33655]: cluster 2021-07-31T02:36:58.512760+0000 mgr.x (mgr.24542) 2133 : cluster [DBG] pgmap v2954: 73 pgs: 32 unknown, 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:37:00.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:00 smithi045 conmon[33655]: cluster 2021-07-31T02:36:59.426430+0000 mon.a (mon.0) 2611 : cluster [DBG] osdmap e1616: 8 total, 8 up, 8 in 2021-07-31T02:37:00.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:00 smithi045 conmon[39718]: cluster 2021-07-31T02:36:58 2021-07-31T02:37:00.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:00 smithi045 conmon[39718]: .512760+0000 mgr.x (mgr.24542) 2133 : cluster [DBG] pgmap v2954: 73 pgs: 32 unknown, 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:37:00.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:00 smithi045 conmon[39718]: cluster 2021-07-31T02:37:00.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:00 smithi045 conmon[39718]: 2021-07-31T02:36:59.426430+0000 mon.a 2021-07-31T02:37:00.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:00 smithi045 conmon[39718]: (mon.0) 2611 : cluster [DBG] 2021-07-31T02:37:00.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:00 smithi045 conmon[39718]: osdmap e1616: 8 total, 8 up, 8 in 2021-07-31T02:37:01.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:01 smithi154 conmon[34553]: cluster 2021-07-31T02:37:00.432001+0000 mon.a (mon.0) 2612 : cluster [DBG] osdmap e1617: 8 total, 8 up, 8 in 2021-07-31T02:37:01.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:01 smithi154 conmon[34553]: cluster 2021-07-31T02:37:00.513181+0000 mgr.x (mgr.24542) 2134 : cluster [DBG] pgmap v2957: 73 pgs: 73 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 132 KiB/s rd, 9.0 KiB/s wr, 178 op/s 2021-07-31T02:37:01.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:01 smithi045 conmon[33655]: cluster 2021-07-31T02:37:00.432001+0000 mon.a (mon.0) 2612 : cluster [DBG] 2021-07-31T02:37:01.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:01 smithi045 conmon[33655]: osdmap e1617: 8 total, 8 up, 8 in 2021-07-31T02:37:01.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:01 smithi045 conmon[33655]: cluster 2021-07-31T02:37:00.513181+0000 mgr.x (mgr.24542) 2134 : cluster [DBG] pgmap v2957: 73 pgs: 73 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 132 KiB/s rd, 9.0 KiB/s wr, 178 op/s 2021-07-31T02:37:01.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:01 smithi045 conmon[39718]: cluster 2021-07-31T02:37:00.432001+0000 mon.a (mon 2021-07-31T02:37:01.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:01 smithi045 conmon[39718]: .0) 2612 : cluster [DBG] osdmap e1617: 8 total, 8 up, 8 in 2021-07-31T02:37:01.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:01 smithi045 conmon[39718]: cluster 2021-07-31T02:37:00. 2021-07-31T02:37:01.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:01 smithi045 conmon[39718]: 513181+0000 mgr.x (mgr.24542) 2134 : cluster [DBG] pgmap v2957: 73 pgs: 73 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 132 KiB/s rd, 9.0 KiB/s wr, 178 op/s 2021-07-31T02:37:02.638 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestClone.test_unprotect_with_children ... ok 2021-07-31T02:37:02.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:02 smithi154 conmon[34553]: cluster 2021-07-31T02:37:01.438309+0000 mon.a (mon.0) 2613 : cluster [DBG] osdmap e1618: 8 total, 8 up, 8 in 2021-07-31T02:37:02.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:02 smithi045 conmon[39718]: cluster 2021-07-31T02:37:01.438309+0000 2021-07-31T02:37:02.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:02 smithi045 conmon[39718]: mon.a (mon.0) 2613 : cluster [DBG] osdmap e1618: 8 total, 8 up, 8 in 2021-07-31T02:37:02.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:02 smithi045 conmon[33655]: cluster 2021-07-31T02:37:02.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:02 smithi045 conmon[33655]: 2021-07-31T02:37:02.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:02 smithi045 conmon[33655]: 2021 2021-07-31T02:37:02.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:02 smithi045 conmon[33655]: - 2021-07-31T02:37:02.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:02 smithi045 conmon[33655]: 07 2021-07-31T02:37:02.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:02 smithi045 conmon[33655]: - 2021-07-31T02:37:02.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:02 smithi045 conmon[33655]: 31 2021-07-31T02:37:02.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:02 smithi045 conmon[33655]: T 2021-07-31T02:37:02.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:02 smithi045 conmon[33655]: 02 2021-07-31T02:37:02.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:02 smithi045 conmon[33655]: : 2021-07-31T02:37:02.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:02 smithi045 conmon[33655]: 37 2021-07-31T02:37:02.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:02 smithi045 conmon[33655]: : 2021-07-31T02:37:02.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:02 smithi045 conmon[33655]: 01 2021-07-31T02:37:02.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:02 smithi045 conmon[33655]: . 2021-07-31T02:37:02.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:02 smithi045 conmon[33655]: 438309 2021-07-31T02:37:02.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:02 smithi045 conmon[33655]: +0000 2021-07-31T02:37:02.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:02 smithi045 conmon[33655]: 2021-07-31T02:37:02.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:02 smithi045 conmon[33655]: mon.a (mon.0) 2613 : cluster [DBG] osdmap e1618: 8 total, 8 up, 8 in 2021-07-31T02:37:02.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:02 smithi045 conmon[33655]: 2021-07-31T02:37:03.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:03 smithi154 conmon[34553]: cluster 2021-07-31T02:37:02.441805+0000 mon.a (mon.0 2021-07-31T02:37:03.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:03 smithi154 conmon[34553]: ) 2614 : cluster [DBG] osdmap e1619: 8 total, 8 up, 8 in 2021-07-31T02:37:03.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:03 smithi154 conmon[34553]: cluster 2021-07-31T02:37:02.513592+0000 mgr.x (mgr.24542) 2135 : cluster [DBG] pgmap v2960: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 131 KiB/s rd, 9.0 KiB/s wr, 177 op/s 2021-07-31T02:37:03.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:03 smithi045 conmon[33655]: cluster 2021-07-31T02:37:02.441805 2021-07-31T02:37:03.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:03 smithi045 conmon[33655]: +0000 mon.a (mon.0) 2614 : cluster [DBG] osdmap e1619: 8 total, 8 up, 8 in 2021-07-31T02:37:03.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:03 smithi045 conmon[33655]: cluster 2021-07-31T02:37:02.513592+0000 mgr.x (mgr.24542) 2135 : cluster [DBG] pgmap v2960: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 131 KiB/s rd, 9.0 KiB/s wr, 177 op/s 2021-07-31T02:37:03.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:03 smithi045 conmon[39718]: cluster 2021-07-31T02:37:02.441805+0000 mon.a (mon.0) 2614 : cluster [DBG] osdmap e1619: 8 total, 8 up, 8 in 2021-07-31T02:37:03.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:03 smithi045 conmon[39718]: cluster 2021-07-31T02:37:02.513592+0000 mgr.x (mgr 2021-07-31T02:37:03.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:03 smithi045 conmon[39718]: .24542) 2135 : cluster [DBG] pgmap v2960: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 131 KiB/s rd, 9.0 KiB/s wr, 177 op/s 2021-07-31T02:37:04.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:04 smithi154 conmon[34553]: cluster 2021-07-31T02:37:03.455632+0000 mon.a (mon.0) 2615 : cluster [DBG] osdmap e1620: 8 total, 8 up, 8 in 2021-07-31T02:37:04.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:04 smithi045 conmon[39718]: cluster 2021-07-31T02:37:03.455632+0000 mon.a (mon. 2021-07-31T02:37:04.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:04 smithi045 conmon[39718]: 0) 2615 : cluster [DBG] osdmap e1620: 8 total, 8 up, 8 in 2021-07-31T02:37:04.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:04 smithi045 conmon[33655]: cluster 2021-07-31T02:37:03.455632+0000 mon.a (mon 2021-07-31T02:37:04.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:04 smithi045 conmon[33655]: .0) 2615 : cluster [DBG] osdmap e1620: 8 total, 8 up, 8 in 2021-07-31T02:37:05.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:05 smithi154 conmon[34553]: cluster 2021-07-31T02:37:04.460551+0000 mon.a (mon.0) 2616 2021-07-31T02:37:05.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:05 smithi154 conmon[34553]: : cluster [DBG] osdmap e1621: 8 total, 8 up, 8 in 2021-07-31T02:37:05.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:05 smithi154 conmon[34553]: cluster 2021-07-31T02:37:04.513992+0000 mgr.x (mgr.24542) 2136 : cluster [DBG] pgmap v2963: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 135 KiB/s rd, 9.2 KiB/s wr, 181 op/s 2021-07-31T02:37:05.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:05 smithi045 conmon[33655]: cluster 2021-07-31T02:37:04.460551+0000 mon.a (mon.0) 2021-07-31T02:37:05.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:05 smithi045 conmon[33655]: 2616 : cluster [DBG] osdmap e1621: 8 total, 8 up, 8 in 2021-07-31T02:37:05.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:05 smithi045 conmon[33655]: cluster 2021-07-31T02:37:04.513992+0000 mgr.x (mgr.24542) 2136 : cluster [DBG] pgmap v2963: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 135 KiB/s rd, 9.2 KiB/s wr, 181 op/s 2021-07-31T02:37:05.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:05 smithi045 conmon[39718]: cluster 2021-07-31T02:37:04.460551+0000 mon.a ( 2021-07-31T02:37:05.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:05 smithi045 conmon[39718]: mon.0) 2616 : cluster [DBG] osdmap e1621: 8 total, 8 up, 8 in 2021-07-31T02:37:05.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:05 smithi045 conmon[39718]: cluster 2021-07-31T 2021-07-31T02:37:05.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:05 smithi045 conmon[39718]: 02:37:04.513992+0000 mgr.x (mgr.24542) 2136 : cluster [DBG] pgmap v2963: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 135 KiB/s rd, 9.2 KiB/s wr, 181 op/s 2021-07-31T02:37:06.565 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestClone.test_unprotected ... ok 2021-07-31T02:37:06.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:06 smithi154 conmon[34553]: cluster 2021-07-31T02:37:05.466875+0000 mon.a (mon.0) 2617 : cluster [DBG] osdmap e1622: 8 total, 8 up, 8 in 2021-07-31T02:37:06.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:06 smithi045 conmon[33655]: cluster 2021-07-31T02:37:05.466875+0000 mon.a (mon.0) 2617 : cluster [DBG] osdmap e1622: 8 total, 8 up, 8 in 2021-07-31T02:37:06.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:06 smithi045 conmon[39718]: cluster 2021-07-31T02:37:05.466875+0000 mon.a ( 2021-07-31T02:37:06.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:06 smithi045 conmon[39718]: mon.0) 2617 : cluster [DBG] osdmap e1622: 8 total, 8 up, 8 in 2021-07-31T02:37:07.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:07 smithi154 conmon[34553]: cluster 2021-07-31T02:37:06.472089+0000 mon.a (mon.0) 2021-07-31T02:37:07.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:07 smithi154 conmon[34553]: 2618 : cluster [DBG] osdmap e1623: 8 total, 8 up, 8 in 2021-07-31T02:37:07.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:07 smithi154 conmon[34553]: cluster 2021-07-31T02:37:06.514356+0000 mgr.x (mgr.24542) 2137 : cluster [DBG] pgmap v2966: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 135 KiB/s rd, 9.2 KiB/s wr, 181 op/s 2021-07-31T02:37:07.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:07 smithi154 conmon[34553]: cluster 2021-07-31T02:37:07.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:07 smithi154 conmon[34553]: 2021-07-31T02:37:07.443794+0000 mon.a (mon.0) 2619 : cluster [DBG] osdmap e1624: 8 total, 8 up, 8 in 2021-07-31T02:37:07.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:07 smithi045 conmon[33655]: cluster 2021-07-31T02:37:06.472089+0000 mon.a (mon.0) 2618 2021-07-31T02:37:07.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:07 smithi045 conmon[33655]: : cluster [DBG] osdmap e1623: 8 total, 8 up, 8 in 2021-07-31T02:37:07.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:07 smithi045 conmon[33655]: cluster 2021-07-31T02:37:06.514356+0000 mgr.x (mgr.24542) 2137 : cluster [DBG] pgmap v2966: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 135 KiB/s rd, 9.2 KiB/s wr, 181 op/s 2021-07-31T02:37:07.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:07 smithi045 conmon[33655]: cluster 2021-07-31T02:37:07.443794+0000 2021-07-31T02:37:07.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:07 smithi045 conmon[33655]: mon.a (mon.0) 2619 : cluster [DBG] osdmap e1624: 8 total, 8 up, 8 in 2021-07-31T02:37:07.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:07 smithi045 conmon[39718]: cluster 2021-07-31T02:37:06.472089+0000 mon.a (mon.0 2021-07-31T02:37:07.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:07 smithi045 conmon[39718]: ) 2618 : cluster [DBG] osdmap e1623: 8 total, 8 up, 8 in 2021-07-31T02:37:07.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:07 smithi045 conmon[39718]: cluster 2021-07-31T02:37:06.514356+0000 mgr.x (mgr. 2021-07-31T02:37:07.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:07 smithi045 conmon[39718]: 24542) 2137 : cluster [DBG] pgmap v2966: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 135 KiB/s rd, 9.2 KiB/s wr, 181 op/s 2021-07-31T02:37:07.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:07 smithi045 conmon[39718]: cluster 2021-07-31T02:37:07.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:07 smithi045 conmon[39718]: 2021-07-31T02:37:07.443794+0000 mon.a (mon 2021-07-31T02:37:07.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:07 smithi045 conmon[39718]: .0) 2619 : cluster [DBG] osdmap e1624: 8 total, 8 up, 8 in 2021-07-31T02:37:09.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:09 smithi154 conmon[34553]: cluster 2021-07-31T02:37:08.479415+0000 mon.a (mon.0) 2620 : cluster [DBG] osdmap e1625: 8 total, 8 up, 8 in 2021-07-31T02:37:09.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:09 smithi154 conmon[34553]: 2021-07-31T02:37:09.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:09 smithi154 conmon[34553]: cluster 2021-07-31T02:37:08.514725+0000 mgr.x (mgr.24542) 2138 : cluster [DBG] pgmap v2969: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:37:09.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:09 smithi045 conmon[39718]: cluster 2021-07-31T02:37:08.479415+0000 2021-07-31T02:37:09.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:09 smithi045 conmon[39718]: mon.a (mon.0) 2620 : cluster [DBG] osdmap e1625: 8 total, 8 up, 8 in 2021-07-31T02:37:09.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:09 smithi045 conmon[39718]: cluster 2021-07-31T02:37:08.514725+0000 mgr.x (mgr.24542) 2138 2021-07-31T02:37:09.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:09 smithi045 conmon[39718]: : cluster [DBG] pgmap v2969: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:37:09.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:09 smithi045 conmon[33655]: cluster 2021-07-31T02:37:08.479415+0000 mon.a ( 2021-07-31T02:37:09.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:09 smithi045 conmon[33655]: mon.0) 2620 : cluster [DBG] osdmap e1625: 8 total, 8 up, 8 in 2021-07-31T02:37:09.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:09 smithi045 conmon[33655]: cluster 2021-07-31T02:37:08.514725+0000 mgr.x (mgr.24542) 2138 : cluster [DBG] pgmap v2969: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:37:10.586 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestClone.test_with_params ... ok 2021-07-31T02:37:10.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:10 smithi154 conmon[34553]: cluster 2021-07-31T02:37:09.485331+0000 mon.a (mon.0) 2621 : cluster [DBG] osdmap e1626: 8 total, 8 up, 8 in 2021-07-31T02:37:10.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:10 smithi045 conmon[33655]: cluster 2021-07-31T02:37:09.485331+0000 mon.a (mon.0 2021-07-31T02:37:10.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:10 smithi045 conmon[33655]: ) 2621 : cluster [DBG] osdmap e1626: 8 total, 8 up, 8 in 2021-07-31T02:37:10.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:10 smithi045 conmon[39718]: cluster 2021-07-31T02:37:09.485331+0000 mon.a (mon.0) 2621 : cluster [DBG] 2021-07-31T02:37:10.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:10 smithi045 conmon[39718]: osdmap e1626: 8 total, 8 up, 8 in 2021-07-31T02:37:11.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:11 smithi154 conmon[34553]: cluster 2021-07-31T02:37:10.490798+0000 mon.a (mon.0) 2622 : cluster [DBG] osdmap e1627: 8 total, 8 up, 8 in 2021-07-31T02:37:11.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:11 smithi154 conmon[34553]: 2021-07-31T02:37:11.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:11 smithi154 conmon[34553]: cluster 2021-07-31T02:37:10.515150+0000 mgr.x (mgr.24542) 2139 : cluster [DBG] pgmap v2972: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 214 KiB/s rd, 15 KiB/s wr, 288 op/s 2021-07-31T02:37:11.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:11 smithi045 conmon[33655]: cluster 2021-07-31T02:37:10.490798+0000 mon.a (mon.0) 2622 : cluster [DBG] osdmap e1627: 8 total, 8 up, 8 in 2021-07-31T02:37:11.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:11 smithi045 conmon[33655]: 2021-07-31T02:37:11.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:11 smithi045 conmon[33655]: cluster 2021-07-31T02:37:10.515150+0000 mgr.x (mgr.24542) 2139 : cluster [DBG] pgmap v2972: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 214 KiB/s rd, 15 KiB/s wr, 288 op/s 2021-07-31T02:37:11.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:11 smithi045 conmon[39718]: cluster 2021-07-31T02:37:10.490798+0000 mon.a (mon.0 2021-07-31T02:37:11.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:11 smithi045 conmon[39718]: ) 2622 : cluster [DBG] osdmap e1627: 8 total, 8 up, 8 in 2021-07-31T02:37:11.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:11 smithi045 conmon[39718]: cluster 2021-07-31T02:37:10.515150+0000 mgr.x (mgr.24542) 2139 : 2021-07-31T02:37:11.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:11 smithi045 conmon[39718]: cluster [DBG] pgmap v2972: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 214 KiB/s rd, 15 KiB/s wr, 288 op/s 2021-07-31T02:37:12.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:12 smithi154 conmon[34553]: cluster 2021-07-31T02:37:11.496293+0000 mon.a (mon.0) 2623 : cluster [DBG] osdmap e1628: 8 total, 8 up, 8 in 2021-07-31T02:37:12.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:12 smithi154 conmon[34553]: cluster 2021-07-31T02:37:12.445483+0000 mon.a (mon.0) 2624 : cluster [DBG] osdmap e1629: 8 total, 8 up, 8 in 2021-07-31T02:37:12.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:12 smithi045 conmon[33655]: cluster 2021-07-31T02:37:11.496293+0000 2021-07-31T02:37:12.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:12 smithi045 conmon[33655]: mon.a (mon.0) 2623 : cluster [DBG] osdmap e1628: 8 total, 8 up, 8 in 2021-07-31T02:37:12.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:12 smithi045 conmon[33655]: cluster 2021-07-31T02:37:12.445483+0000 mon.a (mon.0) 2624 : cluster 2021-07-31T02:37:12.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:12 smithi045 conmon[33655]: [DBG] osdmap e1629: 8 total, 8 up, 8 in 2021-07-31T02:37:12.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:12 smithi045 conmon[39718]: cluster 2021-07-31T02:37:11.496293 2021-07-31T02:37:12.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:12 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2623 : cluster [DBG] osdmap e1628: 8 total, 8 up, 8 in 2021-07-31T02:37:12.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:12 smithi045 conmon[39718]: cluster 2021-07-31T02:37:12.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:12 smithi045 conmon[39718]: 2021-07-31T02:37:12. 2021-07-31T02:37:12.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:12 smithi045 conmon[39718]: 445483+0000 mon.a (mon.0) 2624 2021-07-31T02:37:12.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:12 smithi045 conmon[39718]: : cluster [DBG] osdmap e1629: 8 total, 8 up, 8 in 2021-07-31T02:37:13.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:13 smithi045 conmon[33655]: cluster 2021-07-31T02:37:12.515512+0000 mgr.x (mgr.24542) 2140 : cluster 2021-07-31T02:37:13.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:13 smithi045 conmon[33655]: [DBG] pgmap v2975: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 214 KiB/s rd, 15 KiB/s wr, 288 op/s 2021-07-31T02:37:13.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:13 smithi045 conmon[39718]: cluster 2021-07-31T02:37:12.515512+0000 mgr.x (mgr. 2021-07-31T02:37:13.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:13 smithi045 conmon[39718]: 24542) 2140 : cluster [DBG] pgmap v2975: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 214 KiB/s rd, 15 KiB/s wr, 288 op/s 2021-07-31T02:37:14.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:13 smithi154 conmon[34553]: cluster 2021-07-31T02:37:12.515512+0000 mgr.x (mgr.24542) 2140 : cluster [DBG] pgmap v2975: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 214 KiB/s rd, 15 KiB/s wr, 288 op/s 2021-07-31T02:37:14.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:13 smithi154 conmon[34553]: 2021-07-31T02:37:14.599 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestClone.test_with_params2 ... ok 2021-07-31T02:37:14.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:14 smithi045 conmon[33655]: cluster 2021-07-31T02:37:13.503908+0000 mon.a (mon.0) 2625 : cluster 2021-07-31T02:37:14.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:14 smithi045 conmon[33655]: [DBG] osdmap e1630: 8 total, 8 up, 8 in 2021-07-31T02:37:14.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:14 smithi045 conmon[39718]: cluster 2021-07-31T02:37:13.503908+0000 mon.a (mon.0) 2625 2021-07-31T02:37:14.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:14 smithi045 conmon[39718]: : cluster [DBG] osdmap e1630: 8 total, 8 up, 8 in 2021-07-31T02:37:15.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:14 smithi154 conmon[34553]: cluster 2021-07-31T02:37:13.503908+0000 mon.a (mon.0) 2625 : cluster [DBG] 2021-07-31T02:37:15.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:14 smithi154 conmon[34553]: osdmap e1630: 8 total, 8 up, 8 in 2021-07-31T02:37:15.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:15 smithi045 conmon[33655]: cluster 2021-07-31T02:37:14.509493+0000 mon.a (mon.0) 2626 : 2021-07-31T02:37:15.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:15 smithi045 conmon[33655]: cluster [DBG] osdmap e1631: 8 total, 8 up, 8 in 2021-07-31T02:37:15.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:15 smithi045 conmon[33655]: cluster 2021-07-31T02:37:14.517254+0000 mgr.x (mgr.24542) 2141 : cluster [DBG] pgmap v2978: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 232 KiB/s rd, 16 KiB/s wr, 315 op/s 2021-07-31T02:37:15.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:15 smithi045 conmon[39718]: cluster 2021-07-31T02:37:14.509493+0000 mon.a (mon.0) 2626 : cluster 2021-07-31T02:37:15.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:15 smithi045 conmon[39718]: [DBG] osdmap e1631: 8 total, 8 up, 8 in 2021-07-31T02:37:15.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:15 smithi045 conmon[39718]: cluster 2021-07-31T02:37:14.517254+0000 mgr.x (mgr.24542) 2141 : cluster [DBG] 2021-07-31T02:37:15.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:15 smithi045 conmon[39718]: pgmap v2978: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 232 KiB/s rd, 16 KiB/s wr, 315 op/s 2021-07-31T02:37:16.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:15 smithi154 conmon[34553]: cluster 2021-07-31T02:37:14.509493+0000 mon.a (mon.0) 2626 : cluster [DBG] osdmap e1631: 8 total, 8 up, 8 in 2021-07-31T02:37:16.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:15 smithi154 conmon[34553]: cluster 2021-07-31T02:37:14.517254+0000 mgr.x (mgr.24542) 2141 : cluster [DBG] pgmap v2978: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 232 KiB/s rd, 16 KiB/s wr, 315 op/s 2021-07-31T02:37:16.614 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestClone.test_with_params3 ... SKIP 2021-07-31T02:37:16.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:16 smithi045 conmon[33655]: cluster 2021-07-31T02:37:15.516471+0000 mon.a (mon 2021-07-31T02:37:16.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:16 smithi045 conmon[33655]: .0) 2627 : cluster [DBG] osdmap e1632: 8 total, 8 up, 8 in 2021-07-31T02:37:16.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:16 smithi045 conmon[39718]: cluster 2021-07-31T02:37:15 2021-07-31T02:37:16.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:16 smithi045 conmon[39718]: .516471+0000 mon.a (mon.0) 2627 : cluster [DBG] osdmap e1632: 8 total, 8 up, 8 in 2021-07-31T02:37:17.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:16 smithi154 conmon[34553]: cluster 2021-07-31T02:37:15.516471+0000 mon.a (mon.0) 2627 2021-07-31T02:37:17.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:16 smithi154 conmon[34553]: : cluster [DBG] osdmap e1632: 8 total, 8 up, 8 in 2021-07-31T02:37:17.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:17 smithi045 conmon[39718]: cluster 2021-07-31T02:37:16.517632 2021-07-31T02:37:17.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:17 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 2142 : cluster [DBG] pgmap v2981: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 232 KiB/s rd, 16 KiB/s wr, 315 op/s 2021-07-31T02:37:17.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:17 smithi045 conmon[39718]: cluster 2021-07-31T02:37:17.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:17 smithi045 conmon[39718]: 2021-07-31T02:37:16.519350+0000 mon.a (mon.0 2021-07-31T02:37:17.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:17 smithi045 conmon[39718]: ) 2628 : cluster [DBG] osdmap e1633: 8 total, 8 up, 8 in 2021-07-31T02:37:17.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:17 smithi045 conmon[39718]: cluster 2021-07 2021-07-31T02:37:17.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:17 smithi045 conmon[39718]: -31T02:37:17.447580+0000 mon.a ( 2021-07-31T02:37:17.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:17 smithi045 conmon[39718]: mon.0) 2629 : cluster [DBG] osdmap e1634: 8 total, 8 up, 8 in 2021-07-31T02:37:17.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:17 smithi045 conmon[33655]: cluster 2021-07-31T02: 2021-07-31T02:37:17.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:17 smithi045 conmon[33655]: 37:16.517632+0000 mgr.x (mgr.24542) 2142 : cluster [DBG] pgmap v2981: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 232 KiB/s rd, 16 KiB/s wr, 315 op/s 2021-07-31T02:37:17.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:17 smithi045 conmon[33655]: cluster 2021-07-31T02:37:17.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:17 smithi045 conmon[33655]: 2021-07-31T02:37:16.519350+0000 mon.a (mon.0) 2628 : cluster [DBG] 2021-07-31T02:37:17.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:17 smithi045 conmon[33655]: osdmap e1633: 8 total, 8 up, 8 in 2021-07-31T02:37:17.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:17 smithi045 conmon[33655]: cluster 2021-07- 2021-07-31T02:37:17.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:17 smithi045 conmon[33655]: 31T02:37:17.447580+0000 mon.a (mon.0) 2629 : cluster [DBG] 2021-07-31T02:37:17.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:17 smithi045 conmon[33655]: osdmap e1634: 8 total, 8 up, 8 in 2021-07-31T02:37:18.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:17 smithi154 conmon[34553]: cluster 2021-07-31T02:37:16.517632+0000 mgr.x (mgr.24542) 2142 : cluster 2021-07-31T02:37:18.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:17 smithi154 conmon[34553]: [DBG] pgmap v2981: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 232 KiB/s rd, 16 KiB/s wr, 315 op/s 2021-07-31T02:37:18.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:17 smithi154 conmon[34553]: cluster 2021-07-31T02:37:16.519350+0000 mon.a (mon.0) 2628 : cluster [DBG] osdmap e1633: 8 total, 8 up, 8 in 2021-07-31T02:37:18.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:17 smithi154 conmon[34553]: cluster 2021-07-31T02:37:17.447580+0000 mon.a (mon.0) 2629 : cluster [DBG] osdmap e1634: 8 total, 8 up, 8 in 2021-07-31T02:37:18.627 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestClone.test_write ... ok 2021-07-31T02:37:18.738 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestExclusiveLock.test_acquire_release_lock ... ok 2021-07-31T02:37:19.630 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestExclusiveLock.test_break_lock ... ok 2021-07-31T02:37:19.763 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestExclusiveLock.test_follower_discard ... ok 2021-07-31T02:37:20.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:19 smithi154 conmon[34553]: cluster 2021-07-31T02:37:18.518222+0000 mgr.x (mgr.24542) 2143 : cluster [DBG] 2021-07-31T02:37:20.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:19 smithi154 conmon[34553]: pgmap v2983: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:37:20.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:19 smithi154 conmon[34553]: cluster 2021-07-31T02:37:18.532076+0000 mon.a (mon.0) 2630 : cluster [DBG] osdmap e1635: 8 total, 8 up, 8 in 2021-07-31T02:37:20.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:19 smithi154 conmon[34553]: audit 2021-07-31T02:37:18.799226 2021-07-31T02:37:20.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:19 smithi154 conmon[34553]: +0000 mon.a (mon.0) 2631 : audit 2021-07-31T02:37:20.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:19 smithi154 conmon[34553]: [INF] from='client.? 172.21.15.45:0/1086976315' entity='client.admin' cmd=[{"prefix": "osd blocklist", "blocklistop": "add", "addr": "172.21.15.45:0/2129796459"}]: dispatch 2021-07-31T02:37:20.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:19 smithi045 conmon[33655]: cluster 2021-07-31T02:37:18.518222+0000 mgr.x (mgr 2021-07-31T02:37:20.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:19 smithi045 conmon[33655]: .24542) 2143 : cluster [DBG] pgmap v2983: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:37:20.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:19 smithi045 conmon[33655]: cluster 2021-07-31T02:37:18.532076+0000 mon.a (mon.0) 2630 : cluster [DBG] osdmap e1635: 8 total, 8 up, 8 in 2021-07-31T02:37:20.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:19 smithi045 conmon[33655]: audit 2021-07-31T02:37:18.799226+0000 mon.a (mon.0) 2631 : audit [INF] from='client.? 172.21.15.45:0/1086976315' entity='client.admin' cmd=[{"prefix": "osd blocklist", "blocklistop": "add", "addr": "172.21.15.45:0/2129796459"}]: dispatch 2021-07-31T02:37:20.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:19 smithi045 conmon[39718]: cluster 2021-07-31T02:37:18.518222+0000 mgr.x (mgr.24542) 2143 : cluster [DBG] pgmap v2983: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:37:20.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:19 smithi045 conmon[39718]: cluster 2021-07-31T02:37:18.532076+0000 mon.a (mon.0) 2630 : cluster [DBG] osdmap e1635: 8 total, 8 up, 8 in 2021-07-31T02:37:20.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:19 smithi045 conmon[39718]: audit 2021- 2021-07-31T02:37:20.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:19 smithi045 conmon[39718]: 07-31T02:37:18.799226+0000 mon.a (mon.0) 2631 : audit [INF] from='client.? 172.21.15.45:0/1086976315' entity='client.admin' cmd=[{"prefix": "osd blocklist", "blocklistop": "add", "addr": "172.21.15.45:0/2129796459"}]: dispatch 2021-07-31T02:37:21.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:20 smithi154 conmon[34553]: audit 2021-07-31T02:37:19.533539+0000 mon.a (mon.0) 2632 : 2021-07-31T02:37:21.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:20 smithi154 conmon[34553]: audit [INF] from='client.? 172.21.15.45:0/1086976315' entity='client.admin' cmd='[{"prefix": "osd blocklist", "blocklistop": "add", "addr": "172.21.15.45:0/2129796459"}]': finished 2021-07-31T02:37:21.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:20 smithi154 conmon[34553]: cluster 2021-07-31T02:37:19.533640+0000 mon.a (mon.0) 2633 : cluster [DBG] osdmap e1636: 8 total, 8 up, 8 in 2021-07-31T02:37:21.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:20 smithi045 conmon[39718]: audit 2021-07-31T02:37:19.533539+0000 mon.a (mon.0 2021-07-31T02:37:21.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:20 smithi045 conmon[39718]: ) 2632 : audit [INF] from='client.? 172.21.15.45:0/1086976315' entity='client.admin' cmd='[{"prefix": "osd blocklist", "blocklistop": "add", "addr": "172.21.15.45:0/2129796459"}]': finished 2021-07-31T02:37:21.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:20 smithi045 conmon[39718]: cluster 2021-07-31T02:37:19.533640+0000 mon.a ( 2021-07-31T02:37:21.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:20 smithi045 conmon[39718]: mon.0) 2633 : cluster [DBG] osdmap e1636: 8 total, 8 up, 8 in 2021-07-31T02:37:21.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:20 smithi045 conmon[33655]: audit 2021-07-31T02:37:19.533539+0000 mon.a 2021-07-31T02:37:21.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:20 smithi045 conmon[33655]: (mon.0) 2632 : audit [INF] from='client.? 172.21.15.45:0/1086976315' entity='client.admin' cmd='[{"prefix": "osd blocklist", "blocklistop": "add", "addr": "172.21.15.45:0/2129796459"}]': finished 2021-07-31T02:37:21.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:20 smithi045 conmon[33655]: cluster 2021-07-31T02:37:19.533640+0000 mon.a (mon.0 2021-07-31T02:37:21.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:20 smithi045 conmon[33655]: ) 2633 : cluster [DBG] osdmap e1636: 8 total, 8 up, 8 in 2021-07-31T02:37:21.646 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestExclusiveLock.test_follower_flatten ... ok 2021-07-31T02:37:21.781 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestExclusiveLock.test_follower_resize ... ok 2021-07-31T02:37:22.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:21 smithi154 conmon[34553]: cluster 2021-07-31T02:37:20.518798+0000 mgr.x (mgr.24542) 2144 : cluster [DBG] pgmap v2986: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 316 KiB/s rd, 24 KiB/s wr, 434 op/s 2021-07-31T02:37:22.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:21 smithi154 conmon[34553]: 2021-07-31T02:37:22.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:21 smithi154 conmon[34553]: cluster 2021-07-31T02:37:20.547854+0000 mon.a (mon.0) 2634 : cluster [DBG] osdmap e1637: 8 total, 8 up, 8 in 2021-07-31T02:37:22.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:21 smithi045 conmon[39718]: cluster 2021-07-31T02:37:20.518798+0000 mgr.x 2021-07-31T02:37:22.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:21 smithi045 conmon[39718]: (mgr.24542) 2144 : cluster [DBG] pgmap v2986: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 316 KiB/s rd, 24 KiB/s wr, 434 op/s 2021-07-31T02:37:22.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:21 smithi045 conmon[39718]: cluster 2021-07-31T02:37 2021-07-31T02:37:22.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:21 smithi045 conmon[39718]: :20.547854+0000 mon.a (mon.0) 2634 : cluster [DBG] osdmap e1637: 8 total, 8 up, 8 in 2021-07-31T02:37:22.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:21 smithi045 conmon[33655]: cluster 2021-07-31T02:37:22.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:21 smithi045 conmon[33655]: 2021-07-31T02:37:20 2021-07-31T02:37:22.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:21 smithi045 conmon[33655]: .518798+0000 mgr.x (mgr.24542) 2144 : cluster [DBG] pgmap v2986: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 316 KiB/s rd, 24 KiB/s wr, 434 op/s 2021-07-31T02:37:22.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:21 smithi045 conmon[33655]: cluster 2021-07-31 2021-07-31T02:37:22.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:21 smithi045 conmon[33655]: T02:37:20.547854+0000 mon.a (mon.0) 2634 : cluster [DBG] osdmap e1637: 8 total, 8 up, 8 in 2021-07-31T02:37:23.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:22 smithi154 conmon[34553]: cluster 2021-07-31T02:37:21.550193+0000 2021-07-31T02:37:23.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:22 smithi154 conmon[34553]: mon.a (mon.0) 2635 : cluster [DBG] osdmap e1638: 8 total, 8 up, 8 in 2021-07-31T02:37:23.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:22 smithi154 conmon[34553]: cluster 2021-07-31T02:37:22.449042+0000 mon.a (mon.0) 2636 2021-07-31T02:37:23.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:22 smithi154 conmon[34553]: : cluster [DBG] osdmap e1639: 8 total, 8 up, 8 in 2021-07-31T02:37:23.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:22 smithi045 conmon[39718]: cluster 2021-07-31T02:37:21.550193+0000 mon.a (mon.0) 2635 : 2021-07-31T02:37:23.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:22 smithi045 conmon[39718]: cluster [DBG] osdmap e1638: 8 total, 8 up, 8 in 2021-07-31T02:37:23.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:22 smithi045 conmon[39718]: cluster 2021-07-31T02:37:22.449042+0000 mon.a (mon.0) 2636 : cluster [DBG] osdmap e1639: 8 total, 8 up, 8 in 2021-07-31T02:37:23.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:22 smithi045 conmon[33655]: cluster 2021-07-31T02:37:21.550193+0000 mon.a (mon.0) 2635 : cluster 2021-07-31T02:37:23.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:22 smithi045 conmon[33655]: [DBG] osdmap e1638: 8 total, 8 up, 8 in 2021-07-31T02:37:23.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:22 smithi045 conmon[33655]: cluster 2021-07-31T02:37:22.449042+0000 mon.a (mon.0) 2636 : cluster [DBG] osdmap e1639: 8 total, 8 up, 8 in 2021-07-31T02:37:23.547 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestExclusiveLock.test_follower_snap_create ... ok 2021-07-31T02:37:24.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:23 smithi154 conmon[34553]: cluster 2021-07-31T02:37:22.519221+0000 mgr.x (mgr. 2021-07-31T02:37:24.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:23 smithi154 conmon[34553]: 24542) 2145 : cluster [DBG] pgmap v2990: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 317 KiB/s rd, 24 KiB/s wr, 436 op/s 2021-07-31T02:37:24.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:23 smithi154 conmon[34553]: cluster 2021-07-31T02:37:23.458911+0000 mon.a (mon.0) 2637 : cluster [DBG] osdmap e1640: 8 total, 8 up, 8 in 2021-07-31T02:37:24.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:23 smithi045 conmon[33655]: cluster 2021-07-31T02:37:22.519221+0000 mgr.x 2021-07-31T02:37:24.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:23 smithi045 conmon[33655]: (mgr.24542) 2145 : cluster [DBG] pgmap v2990: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 317 KiB/s rd, 24 KiB/s wr, 436 op/s 2021-07-31T02:37:24.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:23 smithi045 conmon[33655]: cluster 2021-07-31T02:37:23.458911+0000 mon.a (mon.0 2021-07-31T02:37:24.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:23 smithi045 conmon[33655]: ) 2637 : cluster [DBG] osdmap e1640: 8 total, 8 up, 8 in 2021-07-31T02:37:24.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:23 smithi045 conmon[39718]: cluster 2021-07-31T02:37:22.519221 2021-07-31T02:37:24.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:23 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 2145 : cluster [DBG] pgmap v2990: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 317 KiB/s rd, 24 KiB/s wr, 436 op/s 2021-07-31T02:37:24.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:23 smithi045 conmon[39718]: cluster 2021-07-31T02:37: 2021-07-31T02:37:24.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:23 smithi045 conmon[39718]: 23.458911+0000 mon.a (mon.0) 2637 2021-07-31T02:37:24.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:23 smithi045 conmon[39718]: : cluster [DBG] osdmap e1640: 8 total, 8 up, 8 in 2021-07-31T02:37:25.664 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestExclusiveLock.test_follower_snap_rollback ... ok 2021-07-31T02:37:25.791 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestExclusiveLock.test_follower_write ... ok 2021-07-31T02:37:25.907 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestExclusiveLock.test_ownership ... ok 2021-07-31T02:37:25.998 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestExclusiveLock.test_read_only_leadership ... ok 2021-07-31T02:37:26.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:25 smithi154 conmon[34553]: cluster 2021-07-31T02:37:24.519795+0000 mgr.x (mgr.24542) 2146 : cluster [DBG] 2021-07-31T02:37:26.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:25 smithi154 conmon[34553]: pgmap v2992: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 344 KiB/s rd, 21 KiB/s wr, 467 op/s 2021-07-31T02:37:26.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:25 smithi154 conmon[34553]: cluster 2021-07-31T02:37:24.563367+0000 mon.a (mon.0) 2638 : cluster [DBG] osdmap e1641: 8 total, 8 up, 8 in 2021-07-31T02:37:26.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:25 smithi045 conmon[39718]: cluster 2021-07-31T02:37:24.519795+0000 mgr.x (mgr.24542) 2146 : 2021-07-31T02:37:26.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:25 smithi045 conmon[39718]: cluster [DBG] pgmap v2992: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 344 KiB/s rd, 21 KiB/s wr, 467 op/s 2021-07-31T02:37:26.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:25 smithi045 conmon[39718]: cluster 2021-07-31T02:37:24.563367+0000 mon.a (mon.0) 2638 : cluster [DBG] osdmap e1641: 8 total, 8 up, 8 in 2021-07-31T02:37:26.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:25 smithi045 conmon[33655]: cluster 2021-07-31T02:37:24.519795+0000 mgr.x (mgr.24542) 2146 2021-07-31T02:37:26.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:25 smithi045 conmon[33655]: : cluster [DBG] pgmap v2992: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 344 KiB/s rd, 21 KiB/s wr, 467 op/s 2021-07-31T02:37:26.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:25 smithi045 conmon[33655]: cluster 2021-07-31T02:37:24.563367+0000 mon.a (mon.0) 2638 : cluster [DBG] osdmap e1641: 8 total, 8 up, 8 in 2021-07-31T02:37:27.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:26 smithi154 conmon[34553]: cluster 2021-07-31T02:37:25.572837+0000 mon.a (mon.0) 2639 2021-07-31T02:37:27.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:26 smithi154 conmon[34553]: : cluster [DBG] osdmap e1642: 8 total, 8 up, 8 in 2021-07-31T02:37:27.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:26 smithi045 conmon[33655]: cluster 2021-07-31T02:37:25.572837+0000 mon.a (mon.0) 2639 : cluster [DBG] osdmap e1642: 8 total, 8 up, 8 in 2021-07-31T02:37:27.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:26 smithi045 conmon[39718]: cluster 2021-07-31T02:37:25.572837+0000 mon.a (mon.0 2021-07-31T02:37:27.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:26 smithi045 conmon[39718]: ) 2639 : cluster [DBG] osdmap e1642: 8 total, 8 up, 8 in 2021-07-31T02:37:27.371 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:37:27.549 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestExclusiveLock.test_snapshot_leadership ... ok 2021-07-31T02:37:27.646 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestGroups.test_group_image_add ... ok 2021-07-31T02:37:27.688 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:37:27.689 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 52m ago 53m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:37:27.689 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 50m ago 52m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:37:27.689 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 50m ago 56m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:37:27.689 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 52m ago 58m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:37:27.690 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 52m ago 58m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:37:27.690 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 50m ago 56m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:37:27.690 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 52m ago 56m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:37:27.690 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 52m ago 53m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:37:27.690 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 50m ago 53m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:37:27.691 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 52m ago 56m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:37:27.691 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 52m ago 55m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:37:27.691 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 52m ago 55m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:37:27.691 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 52m ago 55m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:37:27.692 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 50m ago 54m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:37:27.692 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 50m ago 54m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:37:27.692 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 50m ago 54m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:37:27.692 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 50m ago 53m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:37:27.692 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 50m ago 53m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:37:27.739 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestGroups.test_group_image_list ... ok 2021-07-31T02:37:27.833 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestGroups.test_group_image_list_empty ... ok 2021-07-31T02:37:27.961 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestGroups.test_group_image_list_move_to_trash ... ok 2021-07-31T02:37:28.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:27 smithi154 conmon[34553]: cluster 2021-07-31T02:37:26.520195+0000 mgr.x (mgr 2021-07-31T02:37:28.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:27 smithi154 conmon[34553]: .24542) 2147 : cluster [DBG] pgmap v2995: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 338 KiB/s rd, 21 KiB/s wr, 459 op/s 2021-07-31T02:37:28.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:27 smithi154 conmon[34553]: cluster 2021-07-31T02:37:26.588042+0000 mon.a (mon.0) 2640 : cluster [DBG] osdmap e1643: 8 total, 8 up, 8 in 2021-07-31T02:37:28.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:27 smithi154 conmon[34553]: audit 2021-07-31T02:37:27.358996+0000 mgr.x (mgr.24542) 2148 : 2021-07-31T02:37:28.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:27 smithi154 conmon[34553]: audit [DBG] from='client.17718 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:37:28.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:27 smithi154 conmon[34553]: cluster 2021-07-31 2021-07-31T02:37:28.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:27 smithi154 conmon[34553]: T02:37:27.449688+0000 mon.a (mon 2021-07-31T02:37:28.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:27 smithi154 conmon[34553]: .0) 2641 : cluster [DBG] osdmap e1644: 8 total, 8 up, 8 in 2021-07-31T02:37:28.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:27 smithi045 conmon[39718]: cluster 2021-07-31T02:37:26 2021-07-31T02:37:28.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:27 smithi045 conmon[39718]: .520195+0000 mgr.x (mgr.24542) 2147 : cluster [DBG] pgmap v2995: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 338 KiB/s rd, 21 KiB/s wr, 459 op/s 2021-07-31T02:37:28.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:27 smithi045 conmon[39718]: cluster 2021-07-31 2021-07-31T02:37:28.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:27 smithi045 conmon[39718]: T02:37:26.588042+0000 2021-07-31T02:37:28.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:27 smithi045 conmon[39718]: mon.a (mon.0) 2640 : cluster [DBG] 2021-07-31T02:37:28.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:27 smithi045 conmon[39718]: osdmap e1643: 8 total, 8 up, 8 in 2021-07-31T02:37:28.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:27 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T02:37:28.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:27 smithi045 conmon[39718]: 31T02:37:27.358996+0000 2021-07-31T02:37:28.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:27 smithi045 conmon[39718]: mgr.x (mgr.24542) 2148 : audit 2021-07-31T02:37:28.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:27 smithi045 conmon[39718]: [DBG] from='client.17718 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:37:28.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:27 smithi045 conmon[39718]: cluster 2021-07-31T02:37:28.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:27 smithi045 conmon[39718]: 2021-07-31T02: 2021-07-31T02:37:28.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:27 smithi045 conmon[39718]: 37:27.449688+0000 mon.a (mon 2021-07-31T02:37:28.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:27 smithi045 conmon[39718]: .0) 2641 : cluster [DBG] osdmap e1644: 8 total, 8 up, 8 in 2021-07-31T02:37:28.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:27 smithi045 conmon[33655]: cluster 2021-07-31T02:37:26.520195+0000 mgr.x (mgr 2021-07-31T02:37:28.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:27 smithi045 conmon[33655]: .24542) 2147 : cluster [DBG] pgmap v2995: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 338 KiB/s rd, 21 KiB/s wr, 459 op/s 2021-07-31T02:37:28.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:27 smithi045 conmon[33655]: cluster 2021-07-31T02:37:26.588042+0000 mon.a (mon.0) 2640 : cluster 2021-07-31T02:37:28.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:27 smithi045 conmon[33655]: [DBG] osdmap e1643: 8 total, 8 up, 8 in 2021-07-31T02:37:28.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:27 smithi045 conmon[33655]: audit 2021-07-31T 2021-07-31T02:37:28.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:27 smithi045 conmon[33655]: 02:37:27.358996+0000 mgr.x (mgr.24542) 2148 : audit 2021-07-31T02:37:28.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:27 smithi045 conmon[33655]: [DBG] from='client.17718 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:37:28.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:27 smithi045 conmon[33655]: cluster 2021-07-31 2021-07-31T02:37:28.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:27 smithi045 conmon[33655]: T02:37:27.449688 2021-07-31T02:37:28.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:27 smithi045 conmon[33655]: +0000 mon.a (mon.0) 2641 : cluster 2021-07-31T02:37:28.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:27 smithi045 conmon[33655]: [DBG] osdmap e1644: 8 total, 8 up, 8 in 2021-07-31T02:37:28.076 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:37:28.076 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:37:28.076 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:37:28.077 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:37:28.077 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:37:28.077 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:37:28.077 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:37:28.078 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:37:28.078 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:37:28.078 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:37:28.078 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:37:28.079 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:37:28.079 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:37:28.079 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:37:28.079 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:37:28.079 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:37:28.080 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:37:29.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:28 smithi154 conmon[34553]: audit 2021-07-31T02:37:27.680489+0000 mgr.x 2021-07-31T02:37:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:28 smithi154 conmon[34553]: (mgr.24542) 2149 : audit [DBG] from='client.27664 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:37:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:28 smithi154 conmon[34553]: audit 2021-07-31T02:37:28.076212+0000 mon.a (mon.0) 2642 : audit [DBG] from='client.? 172.21.15.45:0/2869551356' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:37:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:28 smithi154 conmon[34553]: audit 2021-07-31T02:37:28.114270+0000 mon.b (mon.2) 325 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:37:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:28 smithi154 conmon[34553]: audit 2021-07- 2021-07-31T02:37:29.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:28 smithi154 conmon[34553]: 31T02:37:28.114675+0000 mon.a (mon.0) 2643 : audit [INF] 2021-07-31T02:37:29.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:28 smithi154 conmon[34553]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:37:29.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:28 smithi154 conmon[34553]: audit 2021-07-31T 2021-07-31T02:37:29.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:28 smithi154 conmon[34553]: 02:37:28.132510+0000 mon.b (mon.2) 326 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:37:29.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:28 smithi154 conmon[34553]: audit 2021-07-31T02 2021-07-31T02:37:29.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:28 smithi154 conmon[34553]: :37:28.132884+0000 mon.a (mon.0) 2644 : 2021-07-31T02:37:29.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:28 smithi154 conmon[34553]: audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:37:29.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:28 smithi045 conmon[39718]: audit 2021-07-31T02:37:27.680489 2021-07-31T02:37:29.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:28 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 2149 : audit [DBG] from='client.27664 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:37:29.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:28 smithi045 conmon[39718]: audit 2021-07 2021-07-31T02:37:29.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:28 smithi045 conmon[39718]: -31T02:37:28.076212+0000 mon.a (mon.0) 2642 : audit [DBG] from='client.? 172.21.15.45:0/2869551356' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:37:29.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:28 smithi045 conmon[39718]: 2021-07-31T02:37:29.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:28 smithi045 conmon[39718]: audit 2021-07-31T02:37:28.114270+0000 2021-07-31T02:37:29.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:28 smithi045 conmon[39718]: mon.b (mon.2) 325 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:37:29.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:28 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T02:37:29.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:28 smithi045 conmon[39718]: T02:37:28.114675+0000 mon.a (mon.0) 2643 2021-07-31T02:37:29.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:28 smithi045 conmon[39718]: : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:37:29.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:28 smithi045 conmon[39718]: audit 2021-07-31T02 2021-07-31T02:37:29.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:28 smithi045 conmon[39718]: :37:28.132510+0000 mon.b (mon 2021-07-31T02:37:29.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:28 smithi045 conmon[39718]: .2) 326 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:37:29.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:28 smithi045 conmon[39718]: audit 2021-07-31T02:37:29.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:28 smithi045 conmon[39718]: 2021-07-31T02 2021-07-31T02:37:29.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:28 smithi045 conmon[39718]: :37:28.132884+0000 mon.a (mon. 2021-07-31T02:37:29.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:28 smithi045 conmon[39718]: 0) 2644 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:37:29.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:28 smithi045 conmon[33655]: audit 2021-07-31T02:37:27.680489+0000 2021-07-31T02:37:29.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:28 smithi045 conmon[33655]: mgr.x (mgr.24542) 2149 : audit [DBG] from='client.27664 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:37:29.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:28 smithi045 conmon[33655]: audit 2021-07-31T02:37:28.076212+0000 mon.a (mon.0) 2642 : audit 2021-07-31T02:37:29.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:28 smithi045 conmon[33655]: [DBG] from='client.? 172.21.15.45:0/2869551356' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:37:29.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:28 smithi045 conmon[33655]: audit 2021-07-31T02:37: 2021-07-31T02:37:29.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:28 smithi045 conmon[33655]: 28.114270+0000 mon.b (mon.2) 325 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:37:29.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:28 smithi045 conmon[33655]: 2021-07-31T02:37:29.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:28 smithi045 conmon[33655]: audit 2021-07-31T02:37:28.114675+0000 mon.a ( 2021-07-31T02:37:29.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:28 smithi045 conmon[33655]: mon.0) 2643 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:37:29.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:28 smithi045 conmon[33655]: audit 2021-07-31T 2021-07-31T02:37:29.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:28 smithi045 conmon[33655]: 02:37:28.132510+0000 mon.b (mon.2 2021-07-31T02:37:29.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:28 smithi045 conmon[33655]: ) 326 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:37:29.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:28 smithi045 conmon[33655]: audit 2021-07-31 2021-07-31T02:37:29.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:28 smithi045 conmon[33655]: T02:37:28.132884+0000 mon.a (mon 2021-07-31T02:37:29.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:28 smithi045 conmon[33655]: .0) 2644 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:37:29.558 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestGroups.test_group_image_many_images ... ok 2021-07-31T02:37:29.673 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestGroups.test_group_image_remove ... ok 2021-07-31T02:37:29.767 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestGroups.test_group_snap ... ok 2021-07-31T02:37:29.865 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestGroups.test_group_snap_flags ... ok 2021-07-31T02:37:30.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:29 smithi154 conmon[34553]: cluster 2021-07-31T02:37:28.520520+0000 mgr.x (mgr.24542 2021-07-31T02:37:30.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:29 smithi154 conmon[34553]: ) 2150 : cluster [DBG] pgmap v2998: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:37:30.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:29 smithi045 conmon[39718]: cluster 2021-07-31T02:37: 2021-07-31T02:37:30.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:29 smithi045 conmon[39718]: 28.520520+0000 mgr.x (mgr.24542) 2150 : cluster [DBG] pgmap v2998: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:37:30.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:29 smithi045 conmon[33655]: cluster 2021-07-31T02:37:28. 2021-07-31T02:37:30.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:29 smithi045 conmon[33655]: 520520+0000 mgr.x (mgr.24542) 2150 : cluster [DBG] pgmap v2998: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:37:30.033 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestGroups.test_group_snap_list_many ... ok 2021-07-31T02:37:31.695 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestGroups.test_group_snap_namespace ... ok 2021-07-31T02:37:31.796 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestGroups.test_group_snap_rename ... ok 2021-07-31T02:37:32.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:31 smithi154 conmon[34553]: cluster 2021-07-31T02:37:32.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:31 smithi154 conmon[34553]: 2021-07-31T02:37:32.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:31 smithi154 conmon[34553]: 2021 2021-07-31T02:37:32.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:31 smithi154 conmon[34553]: -07-31T02:37:30.521184 2021-07-31T02:37:32.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:31 smithi154 conmon[34553]: +0000 mgr.x (mgr.24542) 2151 : cluster [DBG] pgmap v2999: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 470 KiB/s rd, 65 KiB/s wr, 689 op/s 2021-07-31T02:37:32.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:31 smithi154 conmon[34553]: cluster 2021-07-31T02:37:30.591643+0000 2021-07-31T02:37:32.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:31 smithi154 conmon[34553]: mon.a (mon.0) 2645 : cluster [DBG] osdmap e1645: 8 total, 8 up, 8 in 2021-07-31T02:37:32.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:31 smithi045 conmon[33655]: cluster 2021-07-31T02:37:30.521184+0000 mgr.x (mgr 2021-07-31T02:37:32.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:31 smithi045 conmon[33655]: .24542) 2151 : cluster [DBG] pgmap v2999: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 470 KiB/s rd, 65 KiB/s wr, 689 op/s 2021-07-31T02:37:32.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:31 smithi045 conmon[33655]: cluster 2021-07-31T02:37:30.591643+0000 mon.a (mon.0) 2645 : cluster [DBG] osdmap e1645: 8 total, 8 up, 8 in 2021-07-31T02:37:32.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:31 smithi045 conmon[39718]: cluster 2021-07-31T02:37:30.521184+0000 mgr.x (mgr 2021-07-31T02:37:32.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:31 smithi045 conmon[39718]: .24542) 2151 : cluster [DBG] pgmap v2999: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 470 KiB/s rd, 65 KiB/s wr, 689 op/s 2021-07-31T02:37:32.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:31 smithi045 conmon[39718]: cluster 2021-07-31T02:37:30.591643+0000 mon.a 2021-07-31T02:37:32.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:31 smithi045 conmon[39718]: (mon.0) 2645 : cluster [DBG] osdmap e1645: 8 total, 8 up, 8 in 2021-07-31T02:37:32.945 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:32 smithi154 conmon[34553]: cluster 2021-07-31T02:37:31.594189+0000 mon.a (mon.0) 2646 : cluster [DBG] osdmap e1646: 8 total, 8 up, 8 in 2021-07-31T02:37:32.945 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:32 smithi154 conmon[34553]: cluster 2021-07-31T02:37:32.451505+0000 mon.a (mon.0) 2647 : cluster [DBG] osdmap e1647: 8 total, 8 up, 8 in 2021-07-31T02:37:32.946 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:32 smithi045 conmon[39718]: cluster 2021-07-31T02:37:31.594189+0000 mon.a 2021-07-31T02:37:32.946 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:32 smithi045 conmon[39718]: (mon.0) 2646 : cluster [DBG] osdmap e1646: 8 total, 8 up, 8 in 2021-07-31T02:37:32.947 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:32 smithi045 conmon[39718]: cluster 2021-07-31T02:37 2021-07-31T02:37:32.947 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:32 smithi045 conmon[39718]: :32.451505+0000 mon.a (mon.0) 2647 : cluster [DBG] osdmap e1647: 8 total, 8 up, 8 in 2021-07-31T02:37:32.947 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:32 smithi045 conmon[33655]: cluster 2021-07-31T02:37:31.594189+0000 mon.a (mon.0) 2646 : cluster [DBG] osdmap e1646: 8 total, 8 up, 8 in 2021-07-31T02:37:32.948 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:32 smithi045 conmon[33655]: cluster 2021-07-31T02:37:32.451505+0000 mon.a (mon.0) 2647 : cluster [DBG] osdmap e1647: 8 total, 8 up, 8 in 2021-07-31T02:37:33.253 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:32 smithi154 conmon[57492]: debug 2021-07-31T02:37:32.963+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:37:33.253 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:37:33.253 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:37:33.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:32 smithi154 conmon[57492]: yield 2021-07-31T02:37:33.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:37:33.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:37:33.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:37:33.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:37:33.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:37:33.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:37:33.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:37:33.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:37:33.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:37:33.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:32 smithi154 conmon[57492]: task.result() 2021-07-31T02:37:33.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:37:33.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:32 smithi154 conmon[57492]: debug 2021-07-31T02:37:32.974+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:37:33.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:37:33.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:37:33.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:32 smithi154 conmon[57492]: yield 2021-07-31T02:37:33.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:37:33.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:37:33.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:37:33.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:37:33.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:37:33.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:37:33.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:37:33.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:37:33.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:37:33.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:32 smithi154 conmon[57492]: task.result() 2021-07-31T02:37:33.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:37:33.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: debug 2021-07-31T02:37:33.000+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:37:33.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:37:33.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:37:33.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: yield 2021-07-31T02:37:33.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:37:33.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:37:33.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:37:33.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:37:33.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:37:33.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:37:33.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:37:33.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:37:33.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:37:33.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: task.result() 2021-07-31T02:37:33.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:37:33.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: debug 2021-07-31T02:37:33.000+0000 7fcb94162700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:37:33.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:37:33.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:37:33.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: yield 2021-07-31T02:37:33.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:37:33.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:37:33.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:37:33.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:37:33.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:37:33.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:37:33.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:37:33.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:37:33.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:37:33.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: task.result() 2021-07-31T02:37:33.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:37:33.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: 2021-07-31T02:37:33.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:37:33.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: 2021-07-31T02:37:33.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:37:33.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:37:33.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:37:33.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:37:33.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:37:33.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:37:33.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:37:33.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:37:33.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:37:33.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:37:33.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:37:33.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:37:33.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:37:33.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:37:33.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:37:33.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:37:33.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:37:33.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:37:33.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: raise 2021-07-31T02:37:33.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:37:33.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:37:33.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:37:33.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:37:33.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:37:33.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: debug 2021-07-31T02:37:33.007+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:37:33.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:37:33.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:37:33.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: yield 2021-07-31T02:37:33.284 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:37:33.284 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:37:33.284 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:37:33.285 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:37:33.285 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:37:33.286 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:37:33.286 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:37:33.286 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:37:33.287 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:37:33.287 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: task.result() 2021-07-31T02:37:33.287 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:37:33.288 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: debug 2021-07-31T02:37:33.007+0000 7fcb93961700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:37:33.288 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:37:33.288 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:37:33.289 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: yield 2021-07-31T02:37:33.289 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:37:33.289 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:37:33.290 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:37:33.290 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:37:33.291 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:37:33.291 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:37:33.291 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:37:33.292 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:37:33.292 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:37:33.293 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: task.result() 2021-07-31T02:37:33.293 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:37:33.293 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: 2021-07-31T02:37:33.294 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:37:33.294 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: 2021-07-31T02:37:33.294 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:37:33.295 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:37:33.295 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:37:33.296 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:37:33.296 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:37:33.296 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:37:33.297 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:37:33.297 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:37:33.297 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:37:33.297 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:37:33.298 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:37:33.298 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:37:33.298 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:37:33.298 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:37:33.299 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:37:33.299 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:37:33.299 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:37:33.299 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:37:33.299 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: raise 2021-07-31T02:37:33.300 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:37:33.300 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:37:33.300 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:37:33.300 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:37:33.301 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:37:33.569 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestGroups.test_group_snap_rollback ... ok 2021-07-31T02:37:33.652 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestImage.test_access_timestamp ... ok 2021-07-31T02:37:33.743 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestImage.test_aio_discard ... ok 2021-07-31T02:37:33.828 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestImage.test_aio_flush ... ok 2021-07-31T02:37:33.907 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestImage.test_aio_read ... ok 2021-07-31T02:37:33.996 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestImage.test_aio_write ... ok 2021-07-31T02:37:34.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: cluster 2021-07-31T02:37:32.521543+0000 mgr.x (mgr 2021-07-31T02:37:34.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: .24542) 2152 : cluster [DBG] pgmap v3003: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 552 KiB/s rd, 76 KiB/s wr, 809 op/s 2021-07-31T02:37:34.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: audit 2021-07-31T02:37:32.920134+0000 mon.b (mon.2) 327 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:37:34.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: audit 2021-07-31T02:37:32.921310+0000 mon.b ( 2021-07-31T02:37:34.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: mon.2) 328 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:37:34.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: audit 2021 2021-07-31T02:37:34.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: -07-31T02:37:32.922166+0000 2021-07-31T02:37:34.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: mon.b (mon.2) 329 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:37:34.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: 2021-07-31T02:37:34.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:37: 2021-07-31T02:37:34.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: 32.922772+0000 mgr.x (mgr.24542) 2153 : 2021-07-31T02:37:34.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:37:34.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: cephadm 2021-07- 2021-07-31T02:37:34.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: 31T02:37:32.964425+0000 mgr.x ( 2021-07-31T02:37:34.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: mgr.24542) 2154 : cephadm [ERR] 2021-07-31T02:37:34.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:37:34.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:37:34.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: yield 2021-07-31T02:37:34.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:37:34.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:37:34.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:37:34.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:37:34.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:37:34.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:37:34.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:37:34.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:37:34.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:37:34.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:37:34.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:37:34.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:37:32.965144+0000 mgr.x (mgr.24542) 2155 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:37:34.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:37:32.975508+0000 mgr.x 2021-07-31T02:37:34.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: (mgr.24542) 2156 : cephadm [ERR] 2021-07-31T02:37:34.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:37:34.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:37:34.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: yield 2021-07-31T02:37:34.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:37:34.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:37:34.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:37:34.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:37:34.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:37:34.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:37:34.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:37:34.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:37:34.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:37:34.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:37:34.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:37:34.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:37:32.975929+0000 mgr.x (mgr.24542) 2157 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:37:34.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:37:33.001287+0000 mgr.x (mgr.24542 2021-07-31T02:37:34.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: ) 2158 : cephadm [ERR] 2021-07-31T02:37:34.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:37:34.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:37:34.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: yield 2021-07-31T02:37:34.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:37:34.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:37:34.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:37:34.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:37:34.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:37:34.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:37:34.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:37:34.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:37:34.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:37:34.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:37:34.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:37:34.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:37:33.001645+0000 mgr.x (mgr.24542) 2159 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:37:34.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:37:34.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:37:34.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: yield 2021-07-31T02:37:34.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:37:34.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:37:34.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:37:34.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:37:34.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:37:34.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:37:34.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:37:34.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:37:34.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:37:34.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:37:34.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:37:34.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: 2021-07-31T02:37:34.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:37:34.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: 2021-07-31T02:37:34.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:37:34.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:37:34.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:37:34.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:37:34.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:37:34.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:37:34.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:37:34.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:37:34.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:37:34.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:37:34.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:37:34.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:37:34.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:37:34.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:37:34.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:37:34.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:37:34.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:37:34.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:37:34.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: raise 2021-07-31T02:37:34.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:37:34.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:37:34.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:37:34.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:37:34.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:37:34.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:37:33.008267+0000 mgr.x (mgr.24542) 2160 : 2021-07-31T02:37:34.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: cephadm [ERR] 2021-07-31T02:37:34.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:37:34.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:37:34.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: yield 2021-07-31T02:37:34.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:37:34.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:37:34.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:37:34.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:37:34.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:37:34.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:37:34.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:37:34.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:37:34.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:37:34.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:37:34.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:37:34.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:37:33.008655+0000 mgr.x (mgr.24542) 2161 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:37:34.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:37:34.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:37:34.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: yield 2021-07-31T02:37:34.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:37:34.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:37:34.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:37:34.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:37:34.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:37:34.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:37:34.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:37:34.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:37:34.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:37:34.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:37:34.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:37:34.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: 2021-07-31T02:37:34.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:37:34.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: 2021-07-31T02:37:34.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:37:34.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:37:34.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:37:34.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:37:34.035 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:37:34.035 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:37:34.035 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:37:34.035 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:37:34.035 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:37:34.036 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:37:34.036 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:37:34.036 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:37:34.036 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:37:34.037 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:37:34.037 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:37:34.037 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:37:34.037 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:37:34.037 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:37:34.038 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: raise 2021-07-31T02:37:34.038 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:37:34.038 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:37:34.038 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:37:34.038 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:37:34.039 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:37:34.039 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:33 smithi154 conmon[34553]: cluster 2021-07-31T02:37:33.460505+0000 mon.a (mon.0) 2648 : cluster [DBG] osdmap e1648: 8 total, 8 up, 8 in 2021-07-31T02:37:34.040 INFO:journalctl@ceph.osd.0.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[43243]: debug 2021-07-31T02:37:33.559+0000 7fdf70799700 -1 osd.0 pg_epoch: 1648 pg[83.8( v 1648'124 (0'0,1648'124] local-lis/les=1568/1569 n=2 ec=1568/1568 lis/c=1568/1568 les/c/f=1569/1569/0 sis=1568) [0,1,2] r=0 lpr=1568 crt=1648'124 lcod 1647'123 mlcod 1647'123 active+clean+snaptrim trimq=[48~1] ps=[46~1]] removing snap head 2021-07-31T02:37:34.040 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: cluster 2021-07-31T02:37:32.521543 2021-07-31T02:37:34.041 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 2152 : cluster [DBG] pgmap v3003: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 552 KiB/s rd, 76 KiB/s wr, 809 op/s 2021-07-31T02:37:34.041 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: audit 2021- 2021-07-31T02:37:34.041 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: 07-31T02:37:32.920134+0000 mon.b (mon.2) 327 : audit [DBG] 2021-07-31T02:37:34.041 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:37:34.041 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: audit 2021 2021-07-31T02:37:34.042 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: -07-31T02:37:32.921310+0000 mon.b 2021-07-31T02:37:34.042 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: (mon.2) 328 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:37:34.042 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: audit 2021- 2021-07-31T02:37:34.042 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: 07-31T02:37:32.922166+0000 mon.b (mon 2021-07-31T02:37:34.043 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: .2) 329 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:37:34.043 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: cephadm 2021- 2021-07-31T02:37:34.043 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: 07-31T02:37:32.922772+0000 mgr.x (mgr. 2021-07-31T02:37:34.043 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: 24542) 2153 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:37:34.043 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: cephadm 2021 2021-07-31T02:37:34.044 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: -07-31T02:37:32.964425+0000 2021-07-31T02:37:34.044 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: mgr.x (mgr.24542) 2154 : cephadm [ERR] 2021-07-31T02:37:34.044 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:37:34.044 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:37:34.045 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: yield 2021-07-31T02:37:34.045 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:37:34.045 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:37:34.046 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:37:34.046 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:37:34.046 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:37:34.046 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:37:34.047 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:37:34.047 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:37:34.047 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:37:34.047 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:37:34.048 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:37:34.048 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: 2021-07-31T02:37:34.048 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:37:32.965144+0000 mgr.x (mgr.24542) 2155 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:37:34.048 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: cephadm 2021-07-31 2021-07-31T02:37:34.048 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: T02:37:32.975508+0000 mgr.x (mgr.24542) 2156 : cephadm [ERR] 2021-07-31T02:37:34.049 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:37:34.049 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:37:34.049 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: yield 2021-07-31T02:37:34.049 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:37:34.049 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:37:34.050 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:37:34.050 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:37:34.050 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:37:34.050 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:37:34.051 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:37:34.051 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:37:34.051 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:37:34.051 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:37:34.052 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:37:34.052 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: 2021-07-31T02:37:34.052 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:37:32.975929+0000 mgr.x (mgr.24542) 2157 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:37:34.052 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: cephadm 2021-07 2021-07-31T02:37:34.053 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: -31T02:37:33.001287+0000 mgr.x (mgr.24542) 2158 2021-07-31T02:37:34.053 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: : cephadm [ERR] 2021-07-31T02:37:34.053 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:37:34.053 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:37:34.053 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: yield 2021-07-31T02:37:34.054 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:37:34.054 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:37:34.054 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:37:34.054 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:37:34.055 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:37:34.055 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:37:34.055 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:37:34.055 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:37:34.055 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:37:34.056 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:37:34.056 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:37:34.056 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:37:33.001645+0000 mgr.x (mgr.24542) 2159 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:37:34.056 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:37:34.057 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:37:34.057 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: yield 2021-07-31T02:37:34.057 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:37:34.057 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:37:34.057 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:37:34.058 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:37:34.058 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:37:34.058 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:37:34.058 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:37:34.058 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:37:34.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: cluster 2021-07-31T02:37:32.521543 2021-07-31T02:37:34.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 2152 : cluster [DBG] pgmap v3003: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 552 KiB/s rd, 76 KiB/s wr, 809 op/s 2021-07-31T02:37:34.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: audit 2021-07-31T02:37:34.060 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: 2021-07-31T02:37:32.920134+0000 mon.b (mon.2 2021-07-31T02:37:34.060 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: ) 327 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:37:34.060 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: audit 2021 2021-07-31T02:37:34.060 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: -07-31T02:37:32.921310+0000 2021-07-31T02:37:34.061 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: mon.b (mon.2) 328 : audit [DBG] 2021-07-31T02:37:34.061 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:37:34.061 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T02:37:34.061 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: T02:37:32.922166+0000 mon.b ( 2021-07-31T02:37:34.061 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: mon.2) 329 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:37:34.062 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: cephadm 2021 2021-07-31T02:37:34.062 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: -07-31T02:37:32.922772+0000 2021-07-31T02:37:34.062 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: mgr.x (mgr.24542) 2153 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:37:34.062 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: 2021-07-31T02:37:34.063 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: cephadm 2021-07-31T02: 2021-07-31T02:37:34.063 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: 37:32.964425+0000 mgr.x (mgr.24542) 2021-07-31T02:37:34.063 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: 2154 : cephadm [ERR] 2021-07-31T02:37:34.063 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:37:34.063 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:37:34.064 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: yield 2021-07-31T02:37:34.064 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:37:34.064 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:37:34.065 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:37:34.065 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:37:34.065 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:37:34.065 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:37:34.066 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:37:34.066 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:37:34.066 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:37:34.066 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:37:34.067 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:37:34.067 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:37:32.965144+0000 mgr.x (mgr.24542) 2155 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:37:34.067 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:37:32 2021-07-31T02:37:34.067 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: .975508+0000 mgr.x (mgr.24542) 2156 : cephadm [ERR] 2021-07-31T02:37:34.067 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: 2021-07-31T02:37:34.068 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:37:34.068 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:37:34.068 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: yield 2021-07-31T02:37:34.068 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:37:34.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:37:34.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:37:34.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:37:34.070 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: 2021-07-31T02:37:34.070 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:37:34.070 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: 2021-07-31T02:37:34.070 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:37:34.070 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:37:34.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:37:34.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:37:34.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:37:34.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:37:34.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:37:34.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:37:34.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:37:34.073 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:37:34.073 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:37:34.073 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:37:34.073 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:37:34.074 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:37:34.074 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:37:34.074 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:37:34.074 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:37:34.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:37:34.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: raise 2021-07-31T02:37:34.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:37:34.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:37:34.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:37:34.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:37:34.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:37:34.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:37:34.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: 2021-07-31T02:37:33.008267+0000 mgr.x (mgr.24542) 2160 : cephadm [ERR] 2021-07-31T02:37:34.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:37:34.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:37:34.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: yield 2021-07-31T02:37:34.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:37:34.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:37:34.078 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:37:34.078 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:37:34.078 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:37:34.078 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:37:34.079 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:37:34.079 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:37:34.079 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:37:34.079 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:37:34.080 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:37:34.080 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:37:34.080 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:37:34.080 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:37:34.081 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:37:34.081 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:37:34.081 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:37:32.975929+0000 mgr.x (mgr.24542) 2157 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:37:34.081 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:37 2021-07-31T02:37:34.081 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: :33.001287+0000 mgr.x (mgr.24542) 2158 : cephadm [ERR] 2021-07-31T02:37:34.082 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:37:34.082 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:37:34.082 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: yield 2021-07-31T02:37:34.082 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:37:34.083 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:37:34.083 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:37:34.083 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:37:34.083 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:37:34.083 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:37:34.084 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:37:34.084 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:37:34.084 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:37:34.084 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:37:34.085 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:37:34.085 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: 2021-07-31T02:37:34.085 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:37:33.001645+0000 mgr.x (mgr.24542) 2159 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:37:34.085 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:37:34.085 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:37:34.086 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: yield 2021-07-31T02:37:34.086 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:37:34.086 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:37:34.086 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:37:34.087 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:37:34.087 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:37:34.087 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:37:34.087 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:37:34.087 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:37:34.088 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:37:34.088 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:37:34.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:37:34.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:37:34.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:37:34.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:37:33.008655+0000 mgr.x (mgr.24542) 2161 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:37:34.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:37:34.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:37:34.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: yield 2021-07-31T02:37:34.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:37:34.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:37:34.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:37:34.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:37:34.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:37:34.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:37:34.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:37:34.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:37:34.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:37:34.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:37:34.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:37:34.093 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: 2021-07-31T02:37:34.093 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:37:34.093 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: 2021-07-31T02:37:34.093 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:37:34.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:37:34.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:37:34.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:37:34.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:37:34.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:37:34.095 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:37:34.095 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:37:34.095 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:37:34.095 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:37:34.096 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:37:34.096 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:37:34.096 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:37:34.096 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:37:34.096 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:37:34.097 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:37:34.097 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:37:34.097 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:37:34.097 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: raise 2021-07-31T02:37:34.098 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:37:34.098 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:37:34.098 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:37:34.098 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:37:34.099 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:37:34.099 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[33655]: cluster 2021-07-31T02:37:33.460505+0000 mon.a (mon.0) 2648 : cluster [DBG] osdmap e1648: 8 total, 8 up, 8 in 2021-07-31T02:37:34.099 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:37:34.100 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:37:34.100 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:37:34.100 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: 2021-07-31T02:37:34.100 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:37:34.101 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: 2021-07-31T02:37:34.101 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:37:34.101 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:37:34.101 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:37:34.102 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:37:34.103 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:37:34.103 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:37:34.103 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:37:34.103 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:37:34.104 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:37:34.104 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:37:34.104 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:37:34.104 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:37:34.104 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:37:34.105 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:37:34.105 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:37:34.105 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:37:34.105 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:37:34.105 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:37:34.106 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: raise 2021-07-31T02:37:34.106 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:37:34.106 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:37:34.107 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:37:34.107 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:37:34.107 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:37:34.107 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:37:33.008267+0000 mgr.x (mgr.24542) 2160 : cephadm [ERR] 2021-07-31T02:37:34.107 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:37:34.108 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:37:34.108 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: yield 2021-07-31T02:37:34.108 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:37:34.108 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:37:34.109 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:37:34.109 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:37:34.109 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:37:34.109 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:37:34.109 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:37:34.110 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:37:34.110 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:37:34.110 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:37:34.110 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:37:34.111 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:37:33.008655+0000 mgr.x (mgr.24542) 2161 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:37:34.111 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:37:34.111 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:37:34.112 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: yield 2021-07-31T02:37:34.112 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:37:34.112 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:37:34.112 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:37:34.113 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:37:34.113 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:37:34.113 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:37:34.113 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:37:34.113 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:37:34.114 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:37:34.114 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:37:34.115 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:37:34.115 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: 2021-07-31T02:37:34.115 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:37:34.116 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: 2021-07-31T02:37:34.116 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:37:34.116 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:37:34.116 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:37:34.117 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:37:34.117 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:37:34.117 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:37:34.117 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:37:34.118 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:37:34.118 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:37:34.118 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:37:34.118 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:37:34.118 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:37:34.119 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:37:34.119 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:37:34.119 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:37:34.119 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:37:34.120 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:37:34.120 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:37:34.120 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: raise 2021-07-31T02:37:34.120 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:37:34.121 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:37:34.121 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:37:34.121 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:37:34.121 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:37:34.122 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:33 smithi045 conmon[39718]: cluster 2021-07-31T02:37:33.460505+0000 mon.a (mon.0) 2648 : cluster [DBG] osdmap e1648: 8 total, 8 up, 8 in 2021-07-31T02:37:34.122 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestImage.test_aio_write_zeroes ... ok 2021-07-31T02:37:34.173 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestImage.test_block_name_prefix ... ok 2021-07-31T02:37:34.288 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestImage.test_config_list ... ok 2021-07-31T02:37:34.498 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestImage.test_copy ... ok 2021-07-31T02:37:34.707 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestImage.test_copy2 ... ok 2021-07-31T02:37:34.782 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestImage.test_copy3 ... SKIP 2021-07-31T02:37:36.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:35 smithi154 conmon[34553]: cluster 2021-07-31T02:37:34.521989+0000 mgr.x (mgr.24542) 2162 : cluster [DBG] pgmap v3005: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 802 KiB/s rd, 56 KiB/s wr, 1.10k op/s 2021-07-31T02:37:36.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:35 smithi154 conmon[34553]: 2021-07-31T02:37:36.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:35 smithi045 conmon[33655]: cluster 2021-07-31T02:37:34.521989 2021-07-31T02:37:36.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:35 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 2162 : cluster [DBG] pgmap v3005: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 802 KiB/s rd, 56 KiB/s wr, 1.10k op/s 2021-07-31T02:37:36.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:35 smithi045 conmon[39718]: cluster 2021-07-31T02:37:34.521989+0000 mgr.x (mgr 2021-07-31T02:37:36.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:35 smithi045 conmon[39718]: .24542) 2162 : cluster [DBG] pgmap v3005: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 802 KiB/s rd, 56 KiB/s wr, 1.10k op/s 2021-07-31T02:37:36.715 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestImage.test_create_snap ... ok 2021-07-31T02:37:37.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:36 smithi154 conmon[34553]: cluster 2021-07-31T02:37:35.605897+0000 mon.a (mon.0) 2649 : cluster [DBG] 2021-07-31T02:37:37.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:36 smithi154 conmon[34553]: osdmap e1649: 8 total, 8 up, 8 in 2021-07-31T02:37:37.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:36 smithi045 conmon[39718]: cluster 2021-07-31T02:37:35 2021-07-31T02:37:37.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:36 smithi045 conmon[39718]: .605897+0000 mon.a (mon.0) 2649 : cluster [DBG] osdmap e1649: 8 total, 8 up, 8 in 2021-07-31T02:37:37.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:36 smithi045 conmon[33655]: cluster 2021-07-31T02:37:35.605897+0000 2021-07-31T02:37:37.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:36 smithi045 conmon[33655]: mon.a (mon.0) 2649 : cluster [DBG] osdmap e1649: 8 total, 8 up, 8 in 2021-07-31T02:37:38.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:37 smithi154 conmon[34553]: cluster 2021-07-31T02:37:36.522491+0000 mgr.x (mgr.24542) 2163 : cluster [DBG] pgmap v3007: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 651 KiB/s rd, 46 KiB/s wr, 894 op/s 2021-07-31T02:37:38.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:37 smithi154 conmon[34553]: cluster 2021-07-31T02:37:36.619277+0000 mon.a (mon.0) 2650 : cluster [DBG] osdmap e1650: 8 total, 8 up, 8 in 2021-07-31T02:37:38.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:37 smithi154 conmon[34553]: cluster 2021-07-31T02:37:37.453903+0000 mon.a (mon.0) 2651 : cluster [DBG] osdmap e1651: 8 total, 8 up, 8 in 2021-07-31T02:37:38.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:37 smithi045 conmon[39718]: cluster 2021-07-31T02:37:36.522491+0000 mgr.x (mgr.24542) 2021-07-31T02:37:38.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:37 smithi045 conmon[39718]: 2163 : cluster [DBG] pgmap v3007: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 651 KiB/s rd, 46 KiB/s wr, 894 op/s 2021-07-31T02:37:38.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:37 smithi045 conmon[39718]: cluster 2021-07-31T02:37:36.619277+0000 mon.a (mon. 2021-07-31T02:37:38.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:37 smithi045 conmon[39718]: 0) 2650 : cluster [DBG] osdmap e1650: 8 total, 8 up, 8 in 2021-07-31T02:37:38.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:37 smithi045 conmon[39718]: cluster 2021-07-31T02:37:38.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:37 smithi045 conmon[39718]: 2021-07-31T02:37:37.453903+0000 mon.a 2021-07-31T02:37:38.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:37 smithi045 conmon[39718]: (mon.0) 2651 : cluster [DBG] osdmap e1651: 8 total, 8 up, 8 in 2021-07-31T02:37:38.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:37 smithi045 conmon[33655]: cluster 2021-07-31T02:37:36.522491+0000 mgr.x (mgr.24542) 2163 : cluster [DBG] pgmap v3007: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 651 KiB/s rd, 46 KiB/s wr, 894 op/s 2021-07-31T02:37:38.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:37 smithi045 conmon[33655]: cluster 2021-07-31T02:37:36.619277+0000 mon.a (mon.0) 2650 : cluster [DBG] osdmap e1650: 8 total, 8 up, 8 in 2021-07-31T02:37:38.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:37 smithi045 conmon[33655]: cluster 2021-07-31T02:37: 2021-07-31T02:37:38.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:37 smithi045 conmon[33655]: 37.453903+0000 mon.a (mon.0) 2651 : 2021-07-31T02:37:38.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:37 smithi045 conmon[33655]: cluster [DBG] osdmap e1651: 8 total, 8 up, 8 in 2021-07-31T02:37:38.560 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestImage.test_create_snap_exists ... ok 2021-07-31T02:37:39.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:39 smithi154 conmon[34553]: cluster 2021-07-31T02:37:38.461856+0000 mon.a (mon.0) 2652 : cluster [DBG] osdmap e1652: 8 total, 8 up, 8 in 2021-07-31T02:37:39.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:39 smithi154 conmon[34553]: 2021-07-31T02:37:39.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:39 smithi154 conmon[34553]: cluster 2021-07-31T02:37:38.522890+0000 mgr.x (mgr.24542) 2164 : cluster [DBG] pgmap v3011: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:37:39.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:39 smithi045 conmon[39718]: cluster 2021-07-31T02:37:38.461856+0000 mon.a (mon. 2021-07-31T02:37:39.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:39 smithi045 conmon[39718]: 0) 2652 : cluster [DBG] osdmap e1652: 8 total, 8 up, 8 in 2021-07-31T02:37:39.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:39 smithi045 conmon[39718]: cluster 2021-07-31T02:37:38 2021-07-31T02:37:39.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:39 smithi045 conmon[39718]: .522890+0000 mgr.x (mgr.24542) 2164 : cluster [DBG] pgmap v3011: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:37:39.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:39 smithi045 conmon[33655]: cluster 2021-07-31T02:37:38.461856+0000 mon.a (mon.0) 2652 2021-07-31T02:37:39.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:39 smithi045 conmon[33655]: : cluster [DBG] osdmap e1652: 8 total, 8 up, 8 in 2021-07-31T02:37:39.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:39 smithi045 conmon[33655]: cluster 2021-07-31T02:37:38.522890+0000 mgr.x (mgr.24542) 2164 : cluster [DBG] pgmap v3011: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:37:40.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:40 smithi154 conmon[34553]: cluster 2021-07-31T02:37:39.467808+0000 2021-07-31T02:37:40.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:40 smithi154 conmon[34553]: mon.a (mon.0) 2653 : cluster [DBG] osdmap e1653: 8 total, 8 up, 8 in 2021-07-31T02:37:40.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:40 smithi045 conmon[33655]: cluster 2021-07-31T02:37:39.467808 2021-07-31T02:37:40.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:40 smithi045 conmon[33655]: +0000 mon.a (mon.0) 2653 : cluster [DBG] osdmap e1653: 8 total, 8 up, 8 in 2021-07-31T02:37:40.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:40 smithi045 conmon[39718]: cluster 2021-07-31T02:37:40.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:40 smithi045 conmon[39718]: 2021-07-31T02:37:40.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:40 smithi045 conmon[39718]: 2021 2021-07-31T02:37:40.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:40 smithi045 conmon[39718]: - 2021-07-31T02:37:40.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:40 smithi045 conmon[39718]: 07 2021-07-31T02:37:40.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:40 smithi045 conmon[39718]: - 2021-07-31T02:37:40.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:40 smithi045 conmon[39718]: 31T02:37:39. 2021-07-31T02:37:40.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:40 smithi045 conmon[39718]: 467808+0000 mon.a (mon.0) 2653 : cluster [DBG] osdmap e1653: 8 total, 8 up, 8 in 2021-07-31T02:37:41.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:41 smithi154 conmon[34553]: cluster 2021-07-31T02:37:40.469175+0000 mon.a (mon.0) 2654 : cluster 2021-07-31T02:37:41.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:41 smithi154 conmon[34553]: [DBG] osdmap e1654: 8 total, 8 up, 8 in 2021-07-31T02:37:41.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:41 smithi154 conmon[34553]: cluster 2021-07-31T02:37:40.523491+0000 mgr.x (mgr.24542) 2165 : cluster [DBG] pgmap v3014: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 494 KiB/s rd, 2.1 MiB/s wr, 689 op/s 2021-07-31T02:37:41.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:41 smithi045 conmon[39718]: cluster 2021-07-31T02:37:40.469175+0000 mon.a (mon.0 2021-07-31T02:37:41.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:41 smithi045 conmon[39718]: ) 2654 : cluster [DBG] osdmap e1654: 8 total, 8 up, 8 in 2021-07-31T02:37:41.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:41 smithi045 conmon[39718]: cluster 2021-07-31T02:37:40.523491+0000 mgr.x ( 2021-07-31T02:37:41.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:41 smithi045 conmon[39718]: mgr.24542) 2165 : cluster [DBG] pgmap v3014: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 494 KiB/s rd, 2.1 MiB/s wr, 689 op/s 2021-07-31T02:37:41.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:41 smithi045 conmon[33655]: cluster 2021-07-31T02:37:40.469175+0000 mon.a (mon.0) 2654 : cluster 2021-07-31T02:37:41.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:41 smithi045 conmon[33655]: [DBG] osdmap e1654: 8 total, 8 up, 8 in 2021-07-31T02:37:41.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:41 smithi045 conmon[33655]: cluster 2021-07-31T02:37:40.523491+0000 mgr.x (mgr.24542) 2165 : cluster [DBG] pgmap v3014: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 494 KiB/s rd, 2.1 MiB/s wr, 689 op/s 2021-07-31T02:37:42.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:42 smithi154 conmon[34553]: cluster 2021-07-31T02:37:41.480870+0000 mon.a (mon.0) 2655 : cluster 2021-07-31T02:37:42.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:42 smithi154 conmon[34553]: [DBG] osdmap e1655: 8 total, 8 up, 8 in 2021-07-31T02:37:42.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:42 smithi154 conmon[34553]: cluster 2021-07-31T02:37:42.455025+0000 mon.a (mon.0) 2656 : cluster [DBG] osdmap e1656: 8 total, 8 up, 8 in 2021-07-31T02:37:42.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:42 smithi045 conmon[39718]: cluster 2021-07-31T02:37:41.480870+0000 mon.a (mon.0) 2655 : cluster 2021-07-31T02:37:42.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:42 smithi045 conmon[39718]: [DBG] osdmap e1655: 8 total, 8 up, 8 in 2021-07-31T02:37:42.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:42 smithi045 conmon[39718]: cluster 2021-07-31T02:37:42.455025+0000 mon.a (mon.0) 2656 : cluster [DBG] osdmap e1656: 8 total, 8 up, 8 in 2021-07-31T02:37:42.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:42 smithi045 conmon[33655]: cluster 2021-07-31T02:37:41.480870+0000 mon.a (mon.0) 2655 : cluster 2021-07-31T02:37:42.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:42 smithi045 conmon[33655]: [DBG] osdmap e1655: 8 total, 8 up, 8 in 2021-07-31T02:37:42.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:42 smithi045 conmon[33655]: cluster 2021-07-31T02:37:42.455025+0000 mon.a (mon.0) 2656 : cluster [DBG] osdmap e1656: 8 total, 8 up, 8 in 2021-07-31T02:37:43.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:43 smithi154 conmon[34553]: cluster 2021-07-31T02:37:42.523888+0000 mgr.x (mgr.24542) 2166 : cluster [DBG] pgmap v3017: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 482 KiB/s rd, 2.0 MiB/s wr, 673 op/s 2021-07-31T02:37:43.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:43 smithi154 conmon[34553]: 2021-07-31T02:37:43.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:43 smithi154 conmon[34553]: cluster 2021-07-31T02:37:43.464621+0000 mon.a (mon.0) 2657 : cluster [DBG] osdmap e1657: 8 total, 8 up, 8 in 2021-07-31T02:37:43.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:43 smithi045 conmon[39718]: cluster 2021-07-31T02:37:42.523888+0000 mgr.x (mgr.24542) 2166 : cluster [DBG] pgmap v3017: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 482 KiB/s rd, 2.0 MiB/s wr, 673 op/s 2021-07-31T02:37:43.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:43 smithi045 conmon[39718]: cluster 2021-07-31T02:37:43.464621+0000 mon.a (mon.0) 2657 : cluster [DBG] osdmap e1657: 8 total, 8 up, 8 in 2021-07-31T02:37:43.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:43 smithi045 conmon[33655]: cluster 2021-07-31T02:37:42.523888+0000 mgr.x (mgr.24542) 2166 : cluster [DBG] pgmap v3017: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 482 KiB/s rd, 2.0 MiB/s wr, 673 op/s 2021-07-31T02:37:43.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:43 smithi045 conmon[33655]: cluster 2021-07-31T02:37:43.464621+0000 mon.a (mon.0) 2657 : cluster [DBG] osdmap e1657: 8 total, 8 up, 8 in 2021-07-31T02:37:44.583 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestImage.test_create_snap_flags ... ok 2021-07-31T02:37:44.674 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestImage.test_create_timestamp ... ok 2021-07-31T02:37:44.760 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestImage.test_create_with_params ... SKIP 2021-07-31T02:37:45.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:45 smithi154 conmon[34553]: cluster 2021-07-31T02:37:44.488939+0000 mon.a (mon.0) 2021-07-31T02:37:45.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:45 smithi154 conmon[34553]: 2658 : cluster [DBG] osdmap e1658: 8 total, 8 up, 8 in 2021-07-31T02:37:45.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:45 smithi154 conmon[34553]: cluster 2021-07-31T02:37:44.524413+0000 mgr.x (mgr.24542) 2167 : cluster [DBG] pgmap v3020: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 37 KiB/s rd, 5.0 KiB/s wr, 49 op/s 2021-07-31T02:37:45.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:45 smithi045 conmon[33655]: cluster 2021-07-31T02:37:44.488939+0000 mon.a (mon.0) 2658 2021-07-31T02:37:45.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:45 smithi045 conmon[33655]: : cluster [DBG] osdmap e1658: 8 total, 8 up, 8 in 2021-07-31T02:37:45.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:45 smithi045 conmon[33655]: cluster 2021-07-31T02:37:44.524413+0000 mgr.x (mgr.24542) 2167 : cluster [DBG] pgmap v3020: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 37 KiB/s rd, 5.0 KiB/s wr, 49 op/s 2021-07-31T02:37:45.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:45 smithi045 conmon[39718]: cluster 2021-07-31T02:37:44.488939+0000 mon.a (mon.0 2021-07-31T02:37:45.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:45 smithi045 conmon[39718]: ) 2658 : cluster [DBG] osdmap e1658: 8 total, 8 up, 8 in 2021-07-31T02:37:45.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:45 smithi045 conmon[39718]: cluster 2021-07-31T02:37:44.524413+0000 mgr.x (mgr. 2021-07-31T02:37:45.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:45 smithi045 conmon[39718]: 24542) 2167 : cluster [DBG] pgmap v3020: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 37 KiB/s rd, 5.0 KiB/s wr, 49 op/s 2021-07-31T02:37:46.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:46 smithi154 conmon[34553]: cluster 2021-07-31T02:37:45.494607+0000 mon.a (mon.0) 2659 : cluster [DBG] osdmap e1659: 8 total, 8 up, 8 in 2021-07-31T02:37:46.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:46 smithi154 conmon[34553]: 2021-07-31T02:37:46.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:46 smithi045 conmon[33655]: cluster 2021-07-31T02:37:45.494607+0000 mon.a (mon.0) 2659 : cluster [DBG] osdmap e1659: 8 total, 8 up, 8 in 2021-07-31T02:37:46.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:46 smithi045 conmon[39718]: cluster 2021-07-31T02:37:45.494607+0000 mon.a (mon 2021-07-31T02:37:46.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:46 smithi045 conmon[39718]: .0) 2659 : cluster [DBG] osdmap e1659: 8 total, 8 up, 8 in 2021-07-31T02:37:47.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:47 smithi045 conmon[33655]: cluster 2021-07-31T02:37:46.499937+0000 mon.a 2021-07-31T02:37:47.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:47 smithi045 conmon[33655]: (mon.0) 2660 : cluster [DBG] osdmap e1660: 8 total, 8 up, 8 in 2021-07-31T02:37:47.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:47 smithi045 conmon[33655]: cluster 2021-07-31T02:37:46.525002+0000 mgr.x (mgr.24542) 2168 : cluster [DBG] pgmap v3023: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 37 KiB/s rd, 5.0 KiB/s wr, 49 op/s 2021-07-31T02:37:47.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:47 smithi045 conmon[33655]: cluster 2021-07 2021-07-31T02:37:47.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:47 smithi045 conmon[33655]: -31T02:37:47.456394+0000 2021-07-31T02:37:47.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:47 smithi045 conmon[33655]: mon.a (mon.0) 2661 : cluster [DBG] 2021-07-31T02:37:47.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:47 smithi045 conmon[33655]: osdmap e1661: 8 total, 8 up, 8 in 2021-07-31T02:37:47.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:47 smithi045 conmon[39718]: cluster 2021-07-31T02:37:46.499937+0000 mon.a 2021-07-31T02:37:47.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:47 smithi045 conmon[39718]: (mon.0) 2660 : cluster [DBG] osdmap e1660: 8 total, 8 up, 8 in 2021-07-31T02:37:47.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:47 smithi045 conmon[39718]: cluster 2021-07 2021-07-31T02:37:47.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:47 smithi045 conmon[39718]: -31T02:37:46.525002+0000 mgr.x (mgr.24542) 2168 : cluster [DBG] pgmap v3023: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 37 KiB/s rd, 5.0 KiB/s wr, 49 op/s 2021-07-31T02:37:47.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:47 smithi045 conmon[39718]: 2021-07-31T02:37:47.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:47 smithi045 conmon[39718]: cluster 2021-07-31T02 2021-07-31T02:37:47.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:47 smithi045 conmon[39718]: :37:47.456394+0000 mon.a (mon.0 2021-07-31T02:37:47.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:47 smithi045 conmon[39718]: ) 2661 : cluster [DBG] osdmap e1661: 8 total, 8 up, 8 in 2021-07-31T02:37:48.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:47 smithi154 conmon[34553]: cluster 2021-07-31T02:37:46.499937+0000 mon.a (mon.0) 2660 : cluster [DBG] 2021-07-31T02:37:48.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:47 smithi154 conmon[34553]: osdmap e1660: 8 total, 8 up, 8 in 2021-07-31T02:37:48.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:47 smithi154 conmon[34553]: cluster 2021-07-31T02:37:46.525002+0000 mgr.x (mgr.24542) 2168 : cluster [DBG] pgmap v3023: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 37 KiB/s rd, 5.0 KiB/s wr, 49 op/s 2021-07-31T02:37:48.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:47 smithi154 conmon[34553]: cluster 2021-07-31T02:37:47.456394+0000 mon.a (mon.0) 2661 : cluster [DBG] 2021-07-31T02:37:48.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:47 smithi154 conmon[34553]: osdmap e1661: 8 total, 8 up, 8 in 2021-07-31T02:37:48.668 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestImage.test_deep_copy ... ok 2021-07-31T02:37:49.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:49 smithi045 conmon[33655]: cluster 2021-07-31T02:37:48.507212+0000 mon.a (mon.0) 2021-07-31T02:37:49.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:49 smithi045 conmon[33655]: 2662 : cluster [DBG] osdmap e1662: 8 total, 8 up, 8 in 2021-07-31T02:37:49.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:49 smithi045 conmon[33655]: cluster 2021-07-31T02:37:48.525419+0000 mgr.x (mgr.24542) 2169 : cluster [DBG] pgmap v3026: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:37:49.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:49 smithi045 conmon[39718]: cluster 2021-07-31T02:37:48.507212+0000 mon.a (mon 2021-07-31T02:37:49.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:49 smithi045 conmon[39718]: .0) 2662 : cluster [DBG] osdmap e1662: 8 total, 8 up, 8 in 2021-07-31T02:37:49.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:49 smithi045 conmon[39718]: cluster 2021-07-31T02:37:48.525419+0000 mgr.x (mgr.24542) 2169 : cluster 2021-07-31T02:37:49.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:49 smithi045 conmon[39718]: [DBG] pgmap v3026: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:37:50.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:49 smithi154 conmon[34553]: cluster 2021-07-31T02:37:48.507212+0000 mon.a (mon.0) 2662 : cluster [DBG] osdmap e1662: 8 total, 8 up, 8 in 2021-07-31T02:37:50.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:49 smithi154 conmon[34553]: cluster 2021-07-31T02:37:48.525419+0000 mgr.x (mgr.24542) 2169 : cluster [DBG] pgmap v3026: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:37:50.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:50 smithi045 conmon[39718]: cluster 2021-07-31T02:37:49.511809+0000 mon.a 2021-07-31T02:37:50.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:50 smithi045 conmon[39718]: (mon.0) 2663 : cluster [DBG] osdmap e1663: 8 total, 8 up, 8 in 2021-07-31T02:37:50.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:50 smithi045 conmon[33655]: cluster 2021-07-31T02:37:49.511809 2021-07-31T02:37:50.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:50 smithi045 conmon[33655]: +0000 mon.a (mon.0) 2663 : cluster [DBG] osdmap e1663: 8 total, 8 up, 8 in 2021-07-31T02:37:51.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:50 smithi154 conmon[34553]: cluster 2021-07-31T02:37:49.511809+0000 mon.a (mon.0) 2663 : cluster [DBG] osdmap e1663: 8 total, 8 up, 8 in 2021-07-31T02:37:51.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:50 smithi154 conmon[34553]: 2021-07-31T02:37:51.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:51 smithi045 conmon[39718]: cluster 2021-07-31T02:37:50.513849+0000 mon.a (mon.0) 2664 : cluster [DBG] osdmap e1664: 8 total, 8 up, 8 in 2021-07-31T02:37:51.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:51 smithi045 conmon[39718]: cluster 2021-07-31T02:37:50.525756+0000 mgr.x (mgr.24542) 2170 : cluster [DBG] pgmap v3029: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 207 KiB/s rd, 18 KiB/s wr, 292 op/s 2021-07-31T02:37:51.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:51 smithi045 conmon[33655]: cluster 2021-07-31T02:37:50.513849+0000 mon.a (mon.0) 2664 : cluster 2021-07-31T02:37:51.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:51 smithi045 conmon[33655]: [DBG] osdmap e1664: 8 total, 8 up, 8 in 2021-07-31T02:37:51.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:51 smithi045 conmon[33655]: cluster 2021-07-31T02:37:50.525756+0000 mgr.x (mgr.24542) 2170 : cluster [DBG] pgmap v3029: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 207 KiB/s rd, 18 KiB/s wr, 292 op/s 2021-07-31T02:37:52.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:51 smithi154 conmon[34553]: cluster 2021-07-31T02:37:50.513849+0000 mon.a (mon.0) 2664 : cluster [DBG] 2021-07-31T02:37:52.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:51 smithi154 conmon[34553]: osdmap e1664: 8 total, 8 up, 8 in 2021-07-31T02:37:52.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:51 smithi154 conmon[34553]: cluster 2021-07-31T02:37:50.525756+0000 mgr.x (mgr.24542) 2170 : cluster [DBG] pgmap v3029: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 207 KiB/s rd, 18 KiB/s wr, 292 op/s 2021-07-31T02:37:52.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:52 smithi045 conmon[39718]: cluster 2021-07-31T02:37:51.524076+0000 2021-07-31T02:37:52.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:52 smithi045 conmon[39718]: mon.a (mon.0) 2665 : cluster [DBG] osdmap e1665: 8 total, 8 up, 8 in 2021-07-31T02:37:52.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:52 smithi045 conmon[39718]: cluster 2021-07-31 2021-07-31T02:37:52.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:52 smithi045 conmon[39718]: T02:37:52.457834+0000 mon.a (mon.0) 2666 : 2021-07-31T02:37:52.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:52 smithi045 conmon[39718]: cluster [DBG] osdmap e1666: 8 total, 8 up, 8 in 2021-07-31T02:37:52.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:52 smithi045 conmon[33655]: cluster 2021-07-31T02:37:51.524076+0000 mon.a (mon. 2021-07-31T02:37:52.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:52 smithi045 conmon[33655]: 0) 2665 : cluster [DBG] osdmap e1665: 8 total, 8 up, 8 in 2021-07-31T02:37:52.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:52 smithi045 conmon[33655]: cluster 2021-07-31T02:37:52.457834+0000 mon.a (mon.0) 2666 : cluster [DBG] osdmap e1666: 8 total, 8 up, 8 in 2021-07-31T02:37:53.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:52 smithi154 conmon[34553]: cluster 2021-07-31T02:37:51.524076+0000 mon.a (mon.0) 2665 : cluster 2021-07-31T02:37:53.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:52 smithi154 conmon[34553]: [DBG] osdmap e1665: 8 total, 8 up, 8 in 2021-07-31T02:37:53.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:52 smithi154 conmon[34553]: cluster 2021-07-31T02:37:52.457834+0000 mon.a (mon.0) 2666 : cluster [DBG] osdmap e1666: 8 total, 8 up, 8 in 2021-07-31T02:37:53.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:53 smithi045 conmon[33655]: cluster 2021-07-31T02:37:52.526243+0000 mgr.x (mgr 2021-07-31T02:37:53.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:53 smithi045 conmon[33655]: .24542) 2171 : cluster [DBG] pgmap v3032: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 207 KiB/s rd, 18 KiB/s wr, 292 op/s 2021-07-31T02:37:53.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:53 smithi045 conmon[33655]: cluster 2021-07-31T02:37:53.465870+0000 mon.a (mon.0) 2667 : cluster [DBG] osdmap e1667: 8 total, 8 up, 8 in 2021-07-31T02:37:54.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:53 smithi154 conmon[34553]: cluster 2021-07-31T02:37:52.526243+0000 mgr.x (mgr.24542) 2171 2021-07-31T02:37:54.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:53 smithi154 conmon[34553]: : cluster [DBG] pgmap v3032: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 207 KiB/s rd, 18 KiB/s wr, 292 op/s 2021-07-31T02:37:54.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:53 smithi154 conmon[34553]: cluster 2021-07-31T02:37:53.465870+0000 mon.a (mon.0) 2667 : cluster [DBG] osdmap e1667: 8 total, 8 up, 8 in 2021-07-31T02:37:54.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:53 smithi045 conmon[39718]: cluster 2021-07-31T02:37:52.526243+0000 mgr.x (mgr 2021-07-31T02:37:54.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:53 smithi045 conmon[39718]: .24542) 2171 : cluster [DBG] pgmap v3032: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 207 KiB/s rd, 18 KiB/s wr, 292 op/s 2021-07-31T02:37:54.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:53 smithi045 conmon[39718]: cluster 2021-07-31T02:37:53.465870+0000 mon.a (mon.0) 2667 : cluster [DBG] osdmap e1667: 8 total, 8 up, 8 in 2021-07-31T02:37:54.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:53 smithi045 conmon[39718]: 2021-07-31T02:37:54.633 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestImage.test_deep_copy_clone ... ok 2021-07-31T02:37:56.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:55 smithi154 conmon[34553]: cluster 2021-07-31T02:37:54.526823+0000 mgr.x (mgr.24542) 2172 : cluster [DBG] pgmap v3034: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 172 KiB/s rd, 11 KiB/s wr, 230 op/s 2021-07-31T02:37:56.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:55 smithi154 conmon[34553]: cluster 2021-07-31T02:37:54.537367+0000 mon.a (mon.0) 2668 : cluster [DBG] osdmap e1668: 8 total, 8 up, 8 in 2021-07-31T02:37:56.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:55 smithi045 conmon[33655]: cluster 2021-07-31T02:37:54.526823+0000 mgr.x (mgr.24542) 2172 : 2021-07-31T02:37:56.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:55 smithi045 conmon[33655]: cluster [DBG] pgmap v3034: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 172 KiB/s rd, 11 KiB/s wr, 230 op/s 2021-07-31T02:37:56.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:55 smithi045 conmon[33655]: cluster 2021-07-31T02:37:54.537367+0000 mon.a (mon.0) 2668 : cluster [DBG] osdmap e1668: 8 total, 8 up, 8 in 2021-07-31T02:37:56.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:55 smithi045 conmon[39718]: cluster 2021-07-31T02:37:54.526823+0000 mgr.x (mgr. 2021-07-31T02:37:56.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:55 smithi045 conmon[39718]: 24542) 2172 : cluster [DBG] pgmap v3034: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 172 KiB/s rd, 11 KiB/s wr, 230 op/s 2021-07-31T02:37:56.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:55 smithi045 conmon[39718]: cluster 2021-07-31T02:37:54.537367+0000 mon.a (mon.0) 2668 : cluster [DBG] osdmap e1668: 8 total, 8 up, 8 in 2021-07-31T02:37:57.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:56 smithi154 conmon[34553]: cluster 2021-07-31T02:37:55.539951+0000 mon.a (mon.0) 2669 : cluster [DBG] osdmap e1669: 8 total, 8 up, 8 in 2021-07-31T02:37:57.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:56 smithi045 conmon[39718]: cluster 2021-07-31T02:37:55.539951+0000 mon.a (mon.0) 2669 : cluster [DBG] osdmap e1669: 8 total, 8 up, 8 in 2021-07-31T02:37:57.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:56 smithi045 conmon[33655]: cluster 2021-07-31T02:37:55.539951+0000 mon.a (mon.0) 2669 : 2021-07-31T02:37:57.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:56 smithi045 conmon[33655]: cluster [DBG] osdmap e1669: 8 total, 8 up, 8 in 2021-07-31T02:37:57.752 INFO:journalctl@ceph.osd.7.smithi154.stdout:Jul 31 02:37:57 smithi154 conmon[49149]: debug 2021-07-31T02:37:57.753 INFO:journalctl@ceph.osd.7.smithi154.stdout:Jul 31 02:37:57 smithi154 conmon[49149]: 2021-07-31T02:37:57.473+0000 7f8a75a04700 -1 osd.7 pg_epoch: 1671 pg[83.1d( v 1669'227 (0'0,1669'227] local-lis/les=1568/1569 n=2 ec=1568/1568 lis/c=1568/1568 les/c/f=1569/1569/0 sis=1568) [7,1,3] r=0 lpr=1568 crt=1669'227 lcod 1668'225 mlcod 1668'225 active+clean+snaptrim trimq=[5e~1] ps=[58~1]] removing snap head 2021-07-31T02:37:57.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:57 smithi154 conmon[34553]: cluster 2021-07-31T02:37:56.527224+0000 mgr.x (mgr.24542) 2173 : cluster [DBG] pgmap v3037: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 170 KiB/s rd, 11 KiB/s wr, 227 op/s 2021-07-31T02:37:57.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:57 smithi154 conmon[34553]: cluster 2021-07-31T02:37:56.550458+0000 mon.a (mon.0) 2670 : cluster [DBG] osdmap e1670: 8 total, 8 up, 8 in 2021-07-31T02:37:57.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:57 smithi154 conmon[34553]: cluster 2021-07-31T02:37:57.458966+0000 mon.a (mon.0) 2671 : cluster [DBG] osdmap e1671: 8 total, 8 up, 8 in 2021-07-31T02:37:58.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:57 smithi045 conmon[39718]: cluster 2021-07-31T02:37:56.527224+0000 mgr.x 2021-07-31T02:37:58.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:57 smithi045 conmon[39718]: (mgr.24542) 2173 : cluster [DBG] pgmap v3037: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 170 KiB/s rd, 11 KiB/s wr, 227 op/s 2021-07-31T02:37:58.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:57 smithi045 conmon[39718]: cluster 2021-07-31 2021-07-31T02:37:58.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:57 smithi045 conmon[39718]: T02:37:56.550458+0000 mon.a (mon.0) 2670 : cluster [DBG] 2021-07-31T02:37:58.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:57 smithi045 conmon[39718]: osdmap e1670: 8 total, 8 up, 8 in 2021-07-31T02:37:58.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:57 smithi045 conmon[39718]: cluster 2021 2021-07-31T02:37:58.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:57 smithi045 conmon[39718]: -07-31T02:37:57 2021-07-31T02:37:58.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:57 smithi045 conmon[39718]: .458966+0000 mon.a (mon.0) 2021-07-31T02:37:58.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:57 smithi045 conmon[39718]: 2671 : cluster [DBG] osdmap e1671: 8 total, 8 up, 8 in 2021-07-31T02:37:58.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:57 smithi045 conmon[33655]: cluster 2021-07-31T02:37:56.527224+0000 mgr.x (mgr.24542) 2173 : 2021-07-31T02:37:58.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:57 smithi045 conmon[33655]: cluster [DBG] pgmap v3037: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 170 KiB/s rd, 11 KiB/s wr, 227 op/s 2021-07-31T02:37:58.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:57 smithi045 conmon[33655]: cluster 2021-07-31T02:37:56.550458+0000 mon.a (mon.0) 2670 : cluster [DBG] osdmap e1670: 8 total, 8 up, 8 in 2021-07-31T02:37:58.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:57 smithi045 conmon[33655]: cluster 2021-07-31T02:37:57.458966+0000 mon.a (mon.0) 2671 : cluster [DBG] osdmap e1671: 8 total, 8 up, 8 in 2021-07-31T02:37:58.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:57 smithi045 conmon[33655]: 2021-07-31T02:37:58.389 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:37:58.553 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestImage.test_diff_iterate ... ok 2021-07-31T02:37:58.692 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:37:58.692 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 52m ago 53m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:37:58.693 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 50m ago 53m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:37:58.693 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 50m ago 56m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:37:58.693 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 52m ago 58m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:37:58.693 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 52m ago 59m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:37:58.694 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 50m ago 57m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:37:58.694 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 52m ago 57m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:37:58.694 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 52m ago 53m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:37:58.694 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 50m ago 53m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:37:58.695 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 52m ago 56m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:37:58.695 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 52m ago 56m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:37:58.695 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 52m ago 55m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:37:58.695 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 52m ago 55m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:37:58.696 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 50m ago 55m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:37:58.696 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 50m ago 54m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:37:58.696 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 50m ago 54m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:37:58.697 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 50m ago 54m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:37:58.698 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 50m ago 53m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:37:59.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:58 smithi154 conmon[34553]: audit 2021-07-31T02:37:58.377113+0000 mgr.x 2021-07-31T02:37:59.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:58 smithi154 conmon[34553]: (mgr.24542) 2174 : audit [DBG] from='client.27676 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:37:59.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:58 smithi154 conmon[34553]: cluster 2021-07-31T02:37:58.464661+0000 mon.a (mon.0 2021-07-31T02:37:59.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:58 smithi154 conmon[34553]: ) 2672 : cluster [DBG] osdmap e1672: 8 total, 8 up, 8 in 2021-07-31T02:37:59.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:58 smithi045 conmon[33655]: audit 2021-07-31T02:37:58.377113+0000 mgr.x (mgr 2021-07-31T02:37:59.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:58 smithi045 conmon[33655]: .24542) 2174 : audit [DBG] from='client.27676 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:37:59.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:58 smithi045 conmon[33655]: cluster 2021-07-31T02:37:58.464661+0000 mon.a (mon.0) 2672 : cluster 2021-07-31T02:37:59.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:58 smithi045 conmon[33655]: [DBG] osdmap e1672: 8 total, 8 up, 8 in 2021-07-31T02:37:59.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:58 smithi045 conmon[39718]: audit 2021-07-31T02:37:58.377113+0000 mgr.x ( 2021-07-31T02:37:59.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:58 smithi045 conmon[39718]: mgr.24542) 2174 : audit [DBG] from='client.27676 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:37:59.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:58 smithi045 conmon[39718]: cluster 2021-07-31T02:37:58.464661+0000 mon.a (mon 2021-07-31T02:37:59.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:58 smithi045 conmon[39718]: .0) 2672 : cluster [DBG] osdmap e1672: 8 total, 8 up, 8 in 2021-07-31T02:37:59.028 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:37:59.028 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:37:59.028 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:37:59.029 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:37:59.029 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:37:59.029 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:37:59.029 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:37:59.030 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:37:59.030 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:37:59.030 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:37:59.030 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:37:59.030 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:37:59.031 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:37:59.031 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:37:59.031 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:37:59.031 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:37:59.032 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:38:00.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:59 smithi154 conmon[34553]: cluster 2021-07-31T02:37:58.527588+0000 mgr.x (mgr.24542) 2175 : cluster [DBG] pgmap v3041: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:38:00.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:59 smithi154 conmon[34553]: 2021-07-31T02:38:00.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:59 smithi154 conmon[34553]: audit 2021-07-31T02:37:58.685402+0000 mgr.x (mgr.24542) 2176 : audit [DBG] from='client.17742 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:38:00.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:59 smithi154 conmon[34553]: audit 2021-07-31T02:37:59.028406+0000 2021-07-31T02:38:00.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:37:59 smithi154 conmon[34553]: mon.a (mon.0) 2673 : audit [DBG] from='client.? 172.21.15.45:0/513365532' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:38:00.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:59 smithi045 conmon[33655]: cluster 2021-07-31T02:37:58. 2021-07-31T02:38:00.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:59 smithi045 conmon[33655]: 527588+0000 mgr.x (mgr.24542) 2175 : cluster [DBG] pgmap v3041: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:38:00.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:59 smithi045 conmon[33655]: audit 2021-07-31T02:37:58.685402 2021-07-31T02:38:00.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:59 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 2176 : audit [DBG] from='client.17742 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:38:00.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:59 smithi045 conmon[33655]: 2021-07-31T02:38:00.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:59 smithi045 conmon[33655]: audit 2021-07 2021-07-31T02:38:00.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:59 smithi045 conmon[33655]: -31T02:37:59 2021-07-31T02:38:00.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:59 smithi045 conmon[33655]: .028406+0000 mon.a (mon.0) 2673 : audit [DBG] from='client.? 172.21.15.45:0/513365532' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:38:00.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:37:59 smithi045 conmon[33655]: 2021-07-31T02:38:00.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:59 smithi045 conmon[39718]: cluster 2021-07-31T02:37:58. 2021-07-31T02:38:00.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:59 smithi045 conmon[39718]: 527588+0000 mgr.x (mgr.24542) 2175 : cluster [DBG] pgmap v3041: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:38:00.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:59 smithi045 conmon[39718]: audit 2021-07-31T02:37:58. 2021-07-31T02:38:00.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:59 smithi045 conmon[39718]: 685402+0000 mgr.x (mgr.24542) 2176 : 2021-07-31T02:38:00.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:59 smithi045 conmon[39718]: audit [DBG] from='client.17742 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:38:00.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:59 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T02:38:00.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:59 smithi045 conmon[39718]: T02:37:59.028406+0000 mon.a 2021-07-31T02:38:00.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:59 smithi045 conmon[39718]: (mon.0) 2673 : audit [DBG] 2021-07-31T02:38:00.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:37:59 smithi045 conmon[39718]: from='client.? 172.21.15.45:0/513365532' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:38:02.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:01 smithi154 conmon[34553]: cluster 2021-07-31T02:38:00.528146+0000 mgr.x (mgr.24542) 2177 : cluster [DBG] pgmap v3042: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 89 KiB/s rd, 9.6 KiB/s wr, 126 op/s 2021-07-31T02:38:02.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:01 smithi045 conmon[33655]: cluster 2021-07-31T02:38:00.528146 2021-07-31T02:38:02.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:01 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 2177 : cluster [DBG] pgmap v3042: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 89 KiB/s rd, 9.6 KiB/s wr, 126 op/s 2021-07-31T02:38:02.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:01 smithi045 conmon[39718]: cluster 2021-07-31T02:38:00. 2021-07-31T02:38:02.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:01 smithi045 conmon[39718]: 528146+0000 mgr.x (mgr.24542) 2177 : cluster [DBG] pgmap v3042: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 89 KiB/s rd, 9.6 KiB/s wr, 126 op/s 2021-07-31T02:38:03.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:03 smithi154 conmon[34553]: cluster 2021-07-31T02:38:02.459876+0000 mon.a (mon.0) 2674 : cluster 2021-07-31T02:38:03.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:03 smithi154 conmon[34553]: [DBG] osdmap e1673: 8 total, 8 up, 8 in 2021-07-31T02:38:03.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:03 smithi154 conmon[34553]: cluster 2021-07-31T02:38:02.528734+0000 mgr.x (mgr.24542) 2178 : cluster [DBG] pgmap v3044: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 75 KiB/s rd, 8.0 KiB/s wr, 105 op/s 2021-07-31T02:38:03.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:03 smithi045 conmon[33655]: cluster 2021-07-31T02:38:02. 2021-07-31T02:38:03.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:03 smithi045 conmon[33655]: 459876+0000 mon.a (mon.0) 2674 : cluster [DBG] osdmap e1673: 8 total, 8 up, 8 in 2021-07-31T02:38:03.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:03 smithi045 conmon[33655]: cluster 2021-07-31T02:38:02 2021-07-31T02:38:03.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:03 smithi045 conmon[33655]: .528734+0000 mgr.x (mgr.24542) 2178 : cluster [DBG] pgmap v3044: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 75 KiB/s rd, 8.0 KiB/s wr, 105 op/s 2021-07-31T02:38:03.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:03 smithi045 conmon[39718]: cluster 2021-07-31T02:38:02.459876 2021-07-31T02:38:03.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:03 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2674 : cluster [DBG] osdmap e1673: 8 total, 8 up, 8 in 2021-07-31T02:38:03.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:03 smithi045 conmon[39718]: cluster 2021-07-31T02:38:02 2021-07-31T02:38:03.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:03 smithi045 conmon[39718]: .528734+0000 mgr.x (mgr.24542) 2178 : 2021-07-31T02:38:03.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:03 smithi045 conmon[39718]: cluster [DBG] pgmap v3044: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 75 KiB/s rd, 8.0 KiB/s wr, 105 op/s 2021-07-31T02:38:06.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:05 smithi154 conmon[34553]: cluster 2021-07-31T02:38:04.529337+0000 mgr.x (mgr.24542) 2179 : cluster 2021-07-31T02:38:06.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:05 smithi154 conmon[34553]: [DBG] pgmap v3045: 41 pgs: 41 active+clean; 8.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 108 KiB/s rd, 586 KiB/s wr, 95 op/s 2021-07-31T02:38:06.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:05 smithi045 conmon[39718]: cluster 2021-07-31T02:38:04.529337+0000 mgr.x 2021-07-31T02:38:06.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:05 smithi045 conmon[39718]: (mgr.24542) 2179 : cluster [DBG] pgmap v3045: 41 pgs: 41 active+clean; 8.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 108 KiB/s rd, 586 KiB/s wr, 95 op/s 2021-07-31T02:38:06.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:05 smithi045 conmon[33655]: cluster 2021-07-31T02:38:04.529337+0000 mgr.x 2021-07-31T02:38:06.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:05 smithi045 conmon[33655]: (mgr.24542) 2179 : cluster [DBG] pgmap v3045: 41 pgs: 41 active+clean; 8.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 108 KiB/s rd, 586 KiB/s wr, 95 op/s 2021-07-31T02:38:06.156 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestImage.test_encryption_luks1 ... ok 2021-07-31T02:38:08.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:07 smithi154 conmon[34553]: cluster 2021-07-31T02:38:06.529771+0000 mgr.x (mgr.24542) 2180 : cluster [DBG] pgmap v3046: 41 pgs: 41 active+clean; 8.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 95 KiB/s rd, 514 KiB/s wr, 83 op/s 2021-07-31T02:38:08.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:07 smithi154 conmon[34553]: 2021-07-31T02:38:08.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:07 smithi045 conmon[33655]: cluster 2021-07-31T02:38:06.529771+0000 mgr.x 2021-07-31T02:38:08.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:07 smithi045 conmon[33655]: (mgr.24542) 2180 : cluster [DBG] pgmap v3046: 41 pgs: 41 active+clean; 8.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 95 KiB/s rd, 514 KiB/s wr, 83 op/s 2021-07-31T02:38:08.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:07 smithi045 conmon[39718]: cluster 2021-07-31T02:38:06. 2021-07-31T02:38:08.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:07 smithi045 conmon[39718]: 529771+0000 mgr.x (mgr.24542) 2180 : cluster [DBG] pgmap v3046: 41 pgs: 41 active+clean; 8.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 95 KiB/s rd, 514 KiB/s wr, 83 op/s 2021-07-31T02:38:10.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:09 smithi154 conmon[34553]: cluster 2021-07-31T02:38:08.530319+0000 mgr.x (mgr.24542) 2181 : cluster [DBG] pgmap v3047: 41 pgs: 41 active+clean; 8.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 76 KiB/s rd, 414 KiB/s wr, 67 op/s 2021-07-31T02:38:10.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:09 smithi045 conmon[33655]: cluster 2021-07-31T02:38:08.530319+0000 2021-07-31T02:38:10.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:09 smithi045 conmon[33655]: mgr.x (mgr.24542) 2181 : cluster [DBG] pgmap v3047: 41 pgs: 41 active+clean; 8.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 76 KiB/s rd, 414 KiB/s wr, 67 op/s 2021-07-31T02:38:10.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:09 smithi045 conmon[39718]: cluster 2021-07-31T02:38:08 2021-07-31T02:38:10.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:09 smithi045 conmon[39718]: .530319+0000 mgr.x (mgr.24542) 2181 : cluster [DBG] pgmap v3047: 41 pgs: 41 active+clean; 8.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 76 KiB/s rd, 414 KiB/s wr, 67 op/s 2021-07-31T02:38:12.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:11 smithi154 conmon[34553]: cluster 2021-07-31T02:38:10.530901+0000 mgr.x (mgr.24542) 2182 : cluster [DBG] pgmap v3048: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 83 KiB/s rd, 412 KiB/s wr, 35 op/s 2021-07-31T02:38:12.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:11 smithi045 conmon[33655]: cluster 2021-07-31T02:38:10.530901+0000 mgr.x (mgr.24542) 2182 : cluster [DBG] pgmap v3048: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 83 KiB/s rd, 412 KiB/s wr, 35 op/s 2021-07-31T02:38:12.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:11 smithi045 conmon[39718]: cluster 2021-07-31T02:38:12.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:11 smithi045 conmon[39718]: 2021-07-31T02:38:12.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:11 smithi045 conmon[39718]: 2021 2021-07-31T02:38:12.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:11 smithi045 conmon[39718]: - 2021-07-31T02:38:12.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:11 smithi045 conmon[39718]: 07 2021-07-31T02:38:12.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:11 smithi045 conmon[39718]: - 2021-07-31T02:38:12.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:11 smithi045 conmon[39718]: 31 2021-07-31T02:38:12.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:11 smithi045 conmon[39718]: T 2021-07-31T02:38:12.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:11 smithi045 conmon[39718]: 02 2021-07-31T02:38:12.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:11 smithi045 conmon[39718]: : 2021-07-31T02:38:12.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:11 smithi045 conmon[39718]: 38 2021-07-31T02:38:12.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:11 smithi045 conmon[39718]: : 2021-07-31T02:38:12.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:11 smithi045 conmon[39718]: 10 2021-07-31T02:38:12.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:11 smithi045 conmon[39718]: . 2021-07-31T02:38:12.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:11 smithi045 conmon[39718]: 530901 2021-07-31T02:38:12.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:11 smithi045 conmon[39718]: +0000 2021-07-31T02:38:12.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:11 smithi045 conmon[39718]: 2021-07-31T02:38:12.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:11 smithi045 conmon[39718]: mgr.x 2021-07-31T02:38:12.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:11 smithi045 conmon[39718]: ( 2021-07-31T02:38:12.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:11 smithi045 conmon[39718]: mgr 2021-07-31T02:38:12.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:11 smithi045 conmon[39718]: . 2021-07-31T02:38:12.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:11 smithi045 conmon[39718]: 24542 2021-07-31T02:38:12.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:11 smithi045 conmon[39718]: ) 2021-07-31T02:38:12.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:11 smithi045 conmon[39718]: 2182 2021-07-31T02:38:12.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:11 smithi045 conmon[39718]: : 2021-07-31T02:38:12.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:11 smithi045 conmon[39718]: cluster 2021-07-31T02:38:12.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:11 smithi045 conmon[39718]: 2021-07-31T02:38:12.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:11 smithi045 conmon[39718]: [DBG] 2021-07-31T02:38:12.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:11 smithi045 conmon[39718]: 2021-07-31T02:38:12.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:11 smithi045 conmon[39718]: pgmap v3048: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 83 KiB/s rd, 412 KiB/s wr, 35 op/s 2021-07-31T02:38:12.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:11 smithi045 conmon[39718]: 2021-07-31T02:38:13.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:13 smithi154 conmon[34553]: cluster 2021-07-31T02:38:12.531345+0000 mgr.x (mgr.24542) 2183 2021-07-31T02:38:13.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:13 smithi154 conmon[34553]: : cluster [DBG] pgmap v3049: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 82 KiB/s rd, 409 KiB/s wr, 34 op/s 2021-07-31T02:38:13.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:13 smithi045 conmon[33655]: cluster 2021-07-31T02:38:12. 2021-07-31T02:38:13.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:13 smithi045 conmon[33655]: 531345+0000 mgr.x (mgr.24542) 2183 : cluster [DBG] pgmap v3049: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 82 KiB/s rd, 409 KiB/s wr, 34 op/s 2021-07-31T02:38:13.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:13 smithi045 conmon[39718]: cluster 2021-07-31T02:38:12.531345+0000 mgr.x (mgr. 2021-07-31T02:38:13.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:13 smithi045 conmon[39718]: 24542) 2183 : cluster [DBG] pgmap v3049: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 82 KiB/s rd, 409 KiB/s wr, 34 op/s 2021-07-31T02:38:16.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:15 smithi154 conmon[34553]: cluster 2021-07-31T02:38:14.532064+0000 mgr.x (mgr.24542) 2184 : cluster [DBG] pgmap v3050: 41 pgs: 41 active+clean; 21 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 97 KiB/s rd, 1.7 MiB/s wr, 34 op/s 2021-07-31T02:38:16.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:15 smithi045 conmon[33655]: cluster 2021-07-31T02:38:14.532064+0000 mgr.x (mgr 2021-07-31T02:38:16.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:15 smithi045 conmon[33655]: .24542) 2184 : cluster [DBG] pgmap v3050: 41 pgs: 41 active+clean; 21 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 97 KiB/s rd, 1.7 MiB/s wr, 34 op/s 2021-07-31T02:38:16.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:15 smithi045 conmon[39718]: cluster 2021-07-31T02:38:14.532064+0000 2021-07-31T02:38:16.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:15 smithi045 conmon[39718]: mgr.x (mgr.24542) 2184 : cluster [DBG] pgmap v3050: 41 pgs: 41 active+clean; 21 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 97 KiB/s rd, 1.7 MiB/s wr, 34 op/s 2021-07-31T02:38:17.573 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestImage.test_encryption_luks2 ... ok 2021-07-31T02:38:17.659 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestImage.test_flags ... ok 2021-07-31T02:38:17.734 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestImage.test_id ... ok 2021-07-31T02:38:17.825 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestImage.test_image_auto_close ... ok 2021-07-31T02:38:17.931 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestImage.test_image_config_set_and_get_and_remove ... ok 2021-07-31T02:38:18.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:17 smithi154 conmon[34553]: cluster 2021-07-31T02:38:16.532428+0000 mgr.x (mgr.24542) 2185 2021-07-31T02:38:18.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:17 smithi154 conmon[34553]: : cluster [DBG] pgmap v3051: 41 pgs: 41 active+clean; 21 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 70 KiB/s rd, 1.3 MiB/s wr, 31 op/s 2021-07-31T02:38:18.017 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestImage.test_invalidate_cache ... ok 2021-07-31T02:38:18.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:17 smithi045 conmon[33655]: cluster 2021-07-31T02:38:16.532428+0000 mgr.x ( 2021-07-31T02:38:18.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:17 smithi045 conmon[33655]: mgr.24542) 2185 : cluster [DBG] pgmap v3051: 41 pgs: 41 active+clean; 21 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 70 KiB/s rd, 1.3 MiB/s wr, 31 op/s 2021-07-31T02:38:18.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:17 smithi045 conmon[39718]: cluster 2021-07-31T02:38:16.532428 2021-07-31T02:38:18.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:17 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 2185 : cluster [DBG] pgmap v3051: 41 pgs: 41 active+clean; 21 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 70 KiB/s rd, 1.3 MiB/s wr, 31 op/s 2021-07-31T02:38:18.099 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestImage.test_large_read ... ok 2021-07-31T02:38:18.241 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestImage.test_large_write ... ok 2021-07-31T02:38:20.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:19 smithi154 conmon[34553]: cluster 2021-07-31T02:38:18.532784+0000 mgr.x (mgr.24542) 2186 : cluster [DBG] pgmap v3052: 41 pgs: 41 active+clean; 21 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 70 KiB/s rd, 1.3 MiB/s wr, 31 op/s 2021-07-31T02:38:20.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:19 smithi154 conmon[34553]: 2021-07-31T02:38:20.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:19 smithi154 conmon[34553]: cluster 2021-07-31T02:38:18.596617+0000 mon.a (mon.0) 2675 : cluster [DBG] osdmap e1674: 8 total, 8 up, 8 in 2021-07-31T02:38:20.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:19 smithi045 conmon[39718]: cluster 2021-07-31T02:38:18.532784+0000 mgr.x (mgr.24542) 2186 : cluster [DBG] pgmap v3052: 41 pgs: 41 active+clean; 21 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 70 KiB/s rd, 1.3 MiB/s wr, 31 op/s 2021-07-31T02:38:20.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:19 smithi045 conmon[39718]: cluster 2021-07-31T02:38:18.596617+0000 mon.a (mon.0) 2675 : cluster [DBG] osdmap e1674: 8 total, 8 up, 8 in 2021-07-31T02:38:20.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:19 smithi045 conmon[33655]: cluster 2021-07-31T02:38:18.532784+0000 mgr.x (mgr.24542) 2186 : cluster [DBG] pgmap v3052: 41 pgs: 41 active+clean; 21 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 70 KiB/s rd, 1.3 MiB/s wr, 31 op/s 2021-07-31T02:38:20.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:19 smithi045 conmon[33655]: 2021-07-31T02:38:20.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:19 smithi045 conmon[33655]: cluster 2021-07-31T02:38:18.596617+0000 mon.a (mon.0) 2675 : cluster [DBG] osdmap e1674: 8 total, 8 up, 8 in 2021-07-31T02:38:21.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:20 smithi154 conmon[34553]: cluster 2021-07-31T02:38:19.603948+0000 mon.a (mon.0) 2676 : cluster 2021-07-31T02:38:21.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:20 smithi154 conmon[34553]: [DBG] osdmap e1675: 8 total, 8 up, 8 in 2021-07-31T02:38:21.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:20 smithi045 conmon[33655]: cluster 2021-07-31T02:38:19.603948+0000 mon.a (mon.0) 2021-07-31T02:38:21.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:20 smithi045 conmon[33655]: 2676 : cluster [DBG] osdmap e1675: 8 total, 8 up, 8 in 2021-07-31T02:38:21.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:20 smithi045 conmon[39718]: cluster 2021-07-31T02:38:19. 2021-07-31T02:38:21.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:20 smithi045 conmon[39718]: 603948+0000 mon.a (mon.0) 2676 : cluster [DBG] osdmap e1675: 8 total, 8 up, 8 in 2021-07-31T02:38:22.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:21 smithi154 conmon[34553]: cluster 2021-07-31T02:38:20.533323+0000 mgr.x (mgr.24542) 2187 : cluster [DBG] 2021-07-31T02:38:22.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:21 smithi154 conmon[34553]: pgmap v3055: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 231 KiB/s rd, 3.0 MiB/s wr, 221 op/s 2021-07-31T02:38:22.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:21 smithi154 conmon[34553]: cluster 2021-07-31T02:38:20.609845+0000 mon.a (mon.0) 2677 : cluster [DBG] osdmap e1676: 8 total, 8 up, 8 in 2021-07-31T02:38:22.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:21 smithi045 conmon[33655]: cluster 2021-07-31T02:38:20.533323+0000 mgr.x (mgr 2021-07-31T02:38:22.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:21 smithi045 conmon[33655]: .24542) 2187 : cluster [DBG] pgmap v3055: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 231 KiB/s rd, 3.0 MiB/s wr, 221 op/s 2021-07-31T02:38:22.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:21 smithi045 conmon[33655]: cluster 2021-07-31T02:38:20.609845+0000 mon.a (mon.0) 2677 : cluster [DBG] 2021-07-31T02:38:22.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:21 smithi045 conmon[33655]: osdmap e1676: 8 total, 8 up, 8 in 2021-07-31T02:38:22.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:21 smithi045 conmon[39718]: cluster 2021-07-31T02:38:20 2021-07-31T02:38:22.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:21 smithi045 conmon[39718]: .533323+0000 mgr.x (mgr.24542) 2187 : cluster [DBG] pgmap v3055: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 231 KiB/s rd, 3.0 MiB/s wr, 221 op/s 2021-07-31T02:38:22.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:21 smithi045 conmon[39718]: cluster 2021-07-31T02:38 2021-07-31T02:38:22.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:21 smithi045 conmon[39718]: :20.609845+0000 mon.a ( 2021-07-31T02:38:22.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:21 smithi045 conmon[39718]: mon.0) 2677 : cluster [DBG] 2021-07-31T02:38:22.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:21 smithi045 conmon[39718]: osdmap e1676: 8 total, 8 up, 8 in 2021-07-31T02:38:23.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:22 smithi154 conmon[34553]: cluster 2021-07-31T02:38:21.617289+0000 mon.a (mon.0) 2678 : cluster 2021-07-31T02:38:23.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:22 smithi154 conmon[34553]: [DBG] osdmap e1677: 8 total, 8 up, 8 in 2021-07-31T02:38:23.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:22 smithi045 conmon[33655]: cluster 2021-07-31T02:38:21.617289+0000 mon.a ( 2021-07-31T02:38:23.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:22 smithi045 conmon[33655]: mon.0) 2678 : cluster [DBG] osdmap e1677: 8 total, 8 up, 8 in 2021-07-31T02:38:23.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:22 smithi045 conmon[39718]: cluster 2021-07-31T02:38:21.617289+0000 mon.a (mon.0) 2678 : 2021-07-31T02:38:23.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:22 smithi045 conmon[39718]: cluster [DBG] osdmap e1677: 8 total, 8 up, 8 in 2021-07-31T02:38:24.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:23 smithi154 conmon[34553]: cluster 2021-07-31T02:38:22.533778+0000 mgr.x (mgr.24542) 2188 : cluster 2021-07-31T02:38:24.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:23 smithi154 conmon[34553]: [DBG] pgmap v3058: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 379 KiB/s rd, 2.0 MiB/s wr, 425 op/s 2021-07-31T02:38:24.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:23 smithi154 conmon[34553]: cluster 2021-07-31T02:38:22.618276+0000 mon.a (mon.0) 2679 : cluster [DBG] osdmap e1678: 8 total, 8 up, 8 in 2021-07-31T02:38:24.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:23 smithi045 conmon[33655]: cluster 2021-07-31T02:38: 2021-07-31T02:38:24.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:23 smithi045 conmon[33655]: 22.533778+0000 mgr.x (mgr.24542) 2188 : cluster [DBG] pgmap v3058: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 379 KiB/s rd, 2.0 MiB/s wr, 425 op/s 2021-07-31T02:38:24.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:23 smithi045 conmon[33655]: cluster 2021-07-31T02:38:24.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:23 smithi045 conmon[33655]: 2021-07-31T02:38:22.618276+0000 mon.a (mon.0) 2679 : cluster [DBG] osdmap e1678: 8 total, 8 up, 8 in 2021-07-31T02:38:24.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:23 smithi045 conmon[39718]: cluster 2021-07-31T02:38:22.533778+0000 mgr.x (mgr 2021-07-31T02:38:24.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:23 smithi045 conmon[39718]: .24542) 2188 : cluster [DBG] pgmap v3058: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 379 KiB/s rd, 2.0 MiB/s wr, 425 op/s 2021-07-31T02:38:24.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:23 smithi045 conmon[39718]: cluster 2021- 2021-07-31T02:38:24.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:23 smithi045 conmon[39718]: 07-31T02:38:22.618276+0000 mon.a (mon.0) 2679 : 2021-07-31T02:38:24.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:23 smithi045 conmon[39718]: cluster [DBG] osdmap e1678: 8 total, 8 up, 8 in 2021-07-31T02:38:25.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:24 smithi154 conmon[34553]: cluster 2021-07-31T02:38:23.630143+0000 mon.a (mon.0) 2680 2021-07-31T02:38:25.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:24 smithi154 conmon[34553]: : cluster [DBG] osdmap e1679: 8 total, 8 up, 8 in 2021-07-31T02:38:25.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:24 smithi045 conmon[33655]: cluster 2021-07-31T02:38:23.630143+0000 mon.a (mon.0) 2680 : cluster [DBG] 2021-07-31T02:38:25.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:24 smithi045 conmon[33655]: osdmap e1679: 8 total, 8 up, 8 in 2021-07-31T02:38:25.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:24 smithi045 conmon[39718]: cluster 2021-07-31 2021-07-31T02:38:25.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:24 smithi045 conmon[39718]: T02:38:23.630143+0000 mon.a (mon.0) 2680 : cluster [DBG] osdmap e1679: 8 total, 8 up, 8 in 2021-07-31T02:38:25.727 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestImage.test_limit_snaps ... ok 2021-07-31T02:38:25.893 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestImage.test_list_lockers ... ok 2021-07-31T02:38:26.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:25 smithi154 conmon[34553]: cluster 2021-07-31T02:38:24.534307+0000 mgr.x (mgr.24542) 2189 : cluster [DBG] pgmap v3061: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 29 KiB/s rd, 3.2 KiB/s wr, 39 op/s 2021-07-31T02:38:26.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:25 smithi154 conmon[34553]: cluster 2021-07-31T02:38:24.636146+0000 mon.a (mon.0) 2681 : cluster [DBG] osdmap e1680: 8 total, 8 up, 8 in 2021-07-31T02:38:26.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:25 smithi045 conmon[39718]: cluster 2021-07-31T02:38:24.534307 2021-07-31T02:38:26.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:25 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 2189 : cluster [DBG] pgmap v3061: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 29 KiB/s rd, 3.2 KiB/s wr, 39 op/s 2021-07-31T02:38:26.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:25 smithi045 conmon[39718]: cluster 2021-07-31T02:38:24.636146 2021-07-31T02:38:26.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:25 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2681 : cluster [DBG] osdmap e1680: 8 total, 8 up, 8 in 2021-07-31T02:38:26.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:25 smithi045 conmon[33655]: cluster 2021-07-31T02: 2021-07-31T02:38:26.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:25 smithi045 conmon[33655]: 38:24.534307+0000 mgr.x (mgr.24542) 2189 : cluster [DBG] pgmap v3061: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 29 KiB/s rd, 3.2 KiB/s wr, 39 op/s 2021-07-31T02:38:26.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:25 smithi045 conmon[33655]: cluster 2021-07 2021-07-31T02:38:26.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:25 smithi045 conmon[33655]: -31T02:38:24.636146+0000 2021-07-31T02:38:26.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:25 smithi045 conmon[33655]: mon.a (mon.0) 2681 : cluster [DBG] 2021-07-31T02:38:26.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:25 smithi045 conmon[33655]: osdmap e1680: 8 total, 8 up, 8 in 2021-07-31T02:38:27.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:26 smithi154 conmon[34553]: cluster 2021-07-31T02:38:25.642472+0000 mon.a (mon.0) 2682 : cluster [DBG] osdmap e1681: 8 total, 8 up, 8 in 2021-07-31T02:38:27.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:26 smithi045 conmon[39718]: cluster 2021-07-31T02:38:25.642472+0000 mon.a (mon 2021-07-31T02:38:27.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:26 smithi045 conmon[39718]: .0) 2682 : cluster [DBG] osdmap e1681: 8 total, 8 up, 8 in 2021-07-31T02:38:27.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:26 smithi045 conmon[33655]: cluster 2021-07-31T02:38:25.642472+0000 mon.a (mon.0) 2682 : cluster [DBG] 2021-07-31T02:38:27.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:26 smithi045 conmon[33655]: osdmap e1681: 8 total, 8 up, 8 in 2021-07-31T02:38:28.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:27 smithi154 conmon[34553]: cluster 2021-07-31T02:38:26.534833+0000 mgr.x (mgr.24542) 2190 : cluster [DBG] pgmap v3064: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 29 KiB/s rd, 3.2 KiB/s wr, 39 op/s 2021-07-31T02:38:28.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:27 smithi154 conmon[34553]: cluster 2021-07-31T02:38:26.648666+0000 mon.a (mon.0) 2683 : cluster [DBG] osdmap e1682: 8 total, 8 up, 8 in 2021-07-31T02:38:28.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:27 smithi154 conmon[34553]: cluster 2021-07-31T02:38:27.465524+0000 mon.a (mon.0) 2684 : cluster [DBG] osdmap e1683: 8 total, 8 up, 8 in 2021-07-31T02:38:28.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:27 smithi045 conmon[39718]: cluster 2021-07-31T02:38:26.534833+0000 mgr.x (mgr.24542) 2190 : cluster [DBG] pgmap v3064: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 29 KiB/s rd, 3.2 KiB/s wr, 39 op/s 2021-07-31T02:38:28.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:27 smithi045 conmon[39718]: cluster 2021-07-31T02:38:26.648666+0000 mon.a (mon.0) 2683 : cluster [DBG] osdmap e1682: 8 total, 8 up, 8 in 2021-07-31T02:38:28.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:27 smithi045 conmon[39718]: cluster 2021-07-31 2021-07-31T02:38:28.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:27 smithi045 conmon[39718]: T02:38:27.465524+0000 mon.a (mon.0) 2684 : cluster [DBG] 2021-07-31T02:38:28.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:27 smithi045 conmon[39718]: osdmap e1683: 8 total, 8 up, 8 in 2021-07-31T02:38:28.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:27 smithi045 conmon[33655]: cluster 2021-07-31T02:38:26. 2021-07-31T02:38:28.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:27 smithi045 conmon[33655]: 534833+0000 mgr.x (mgr.24542) 2190 : cluster [DBG] pgmap v3064: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 29 KiB/s rd, 3.2 KiB/s wr, 39 op/s 2021-07-31T02:38:28.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:27 smithi045 conmon[33655]: cluster 2021-07 2021-07-31T02:38:28.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:27 smithi045 conmon[33655]: -31T02:38:26.648666+0000 mon.a (mon.0) 2683 : cluster [DBG] osdmap e1682: 8 total, 8 up, 8 in 2021-07-31T02:38:28.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:27 smithi045 conmon[33655]: cluster 2021-07-31T02:38:28.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:27 smithi045 conmon[33655]: 2021-07-31T02:38:27.465524+0000 mon.a (mon.0) 2684 : cluster [DBG] 2021-07-31T02:38:28.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:27 smithi045 conmon[33655]: osdmap e1683: 8 total, 8 up, 8 in 2021-07-31T02:38:29.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:28 smithi154 conmon[34553]: audit 2021-07-31T02:38:28.119042+0000 mon.b (mon.2) 330 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:38:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:28 smithi154 conmon[34553]: audit 2021-07-31T02:38:28.119683+0000 mon.a (mon.0) 2685 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:38:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:28 smithi154 conmon[34553]: audit 2021-07-31T02:38:28.135563+0000 mon.b (mon.2) 331 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:38:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:28 smithi154 conmon[34553]: audit 2021-07-31T02:38:28.136088+0000 2021-07-31T02:38:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:28 smithi154 conmon[34553]: mon.a (mon.0) 2686 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:38:29.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:28 smithi154 conmon[34553]: cluster 2021-07-31T02:38:28.472002+0000 mon.a (mon.0) 2021-07-31T02:38:29.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:28 smithi154 conmon[34553]: 2687 : cluster [DBG] osdmap e1684: 8 total, 8 up, 8 in 2021-07-31T02:38:29.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:28 smithi045 conmon[39718]: audit 2021-07-31T02:38:28.119042+0000 mon.b (mon.2) 330 : audit 2021-07-31T02:38:29.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:28 smithi045 conmon[39718]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:38:29.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:28 smithi045 conmon[39718]: audit 2021-07-31T02:38:28.119683+0000 mon.a (mon.0 2021-07-31T02:38:29.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:28 smithi045 conmon[39718]: ) 2685 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:38:29.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:28 smithi045 conmon[39718]: audit 2021-07-31T02:38:28.135563 2021-07-31T02:38:29.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:28 smithi045 conmon[39718]: +0000 mon.b (mon.2) 331 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:38:29.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:28 smithi045 conmon[39718]: audit 2021-07-31T02:38:29.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:28 smithi045 conmon[39718]: 2021-07-31T02:38:28.136088 2021-07-31T02:38:29.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:28 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2686 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:38:29.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:28 smithi045 conmon[39718]: 2021-07-31T02:38:29.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:28 smithi045 conmon[39718]: cluster 2021-07-31T02:38:29.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:28 smithi045 conmon[39718]: 2021-07-31T02:38:28.472002 2021-07-31T02:38:29.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:28 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2687 : cluster [DBG] osdmap e1684: 8 total, 8 up, 8 in 2021-07-31T02:38:29.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:28 smithi045 conmon[33655]: audit 2021-07-31T02:38:28.119042+0000 mon.b ( 2021-07-31T02:38:29.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:28 smithi045 conmon[33655]: mon.2) 330 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:38:29.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:28 smithi045 conmon[33655]: audit 2021-07-31T02:38:28.119683+0000 mon.a (mon.0) 2685 : 2021-07-31T02:38:29.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:28 smithi045 conmon[33655]: audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:38:29.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:28 smithi045 conmon[33655]: audit 2021-07-31T02:38:29.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:28 smithi045 conmon[33655]: 2021-07-31T02:38:28.135563+0000 mon.b (mon.2) 331 : audit [INF] 2021-07-31T02:38:29.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:28 smithi045 conmon[33655]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:38:29.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:28 smithi045 conmon[33655]: audit 2021-07-31T02 2021-07-31T02:38:29.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:28 smithi045 conmon[33655]: :38:28.136088+0000 mon.a (mon. 2021-07-31T02:38:29.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:28 smithi045 conmon[33655]: 0) 2686 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:38:29.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:28 smithi045 conmon[33655]: cluster 2021- 2021-07-31T02:38:29.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:28 smithi045 conmon[33655]: 07-31T02:38:28.472002 2021-07-31T02:38:29.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:28 smithi045 conmon[33655]: +0000 mon.a (mon.0) 2687 : cluster 2021-07-31T02:38:29.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:28 smithi045 conmon[33655]: [DBG] osdmap e1684: 8 total, 8 up, 8 in 2021-07-31T02:38:29.340 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:38:29.560 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestImage.test_list_snaps ... ok 2021-07-31T02:38:29.637 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:38:29.637 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 53m ago 54m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:38:29.638 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 51m ago 53m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:38:29.638 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 51m ago 57m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:38:29.638 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 53m ago 59m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:38:29.638 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 53m ago 59m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:38:29.639 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 51m ago 57m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:38:29.639 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 53m ago 57m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:38:29.639 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 53m ago 54m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:38:29.639 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 51m ago 54m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:38:29.640 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 53m ago 57m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:38:29.640 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 53m ago 56m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:38:29.640 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 53m ago 56m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:38:29.640 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 53m ago 56m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:38:29.640 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 51m ago 55m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:38:29.641 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 51m ago 55m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:38:29.641 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 51m ago 55m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:38:29.641 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 51m ago 54m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:38:29.641 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 51m ago 54m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:38:29.977 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:38:29.977 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:38:29.977 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:38:29.978 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:38:29.978 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:38:29.978 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:38:29.978 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:38:29.980 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:38:29.980 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:38:29.980 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:38:29.980 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:38:29.981 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:38:29.981 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:38:29.981 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:38:29.981 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:38:29.981 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:38:29.982 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:38:30.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:29 smithi154 conmon[34553]: cluster 2021-07-31T02:38:28.535393+0000 mgr.x (mgr.24542) 2191 : cluster [DBG] pgmap v3068: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:38:30.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:29 smithi154 conmon[34553]: audit 2021-07-31T02: 2021-07-31T02:38:30.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:29 smithi154 conmon[34553]: 38:29.328312+0000 mgr.x (mgr.24542) 2192 : audit [DBG] from='client.17754 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:38:30.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:29 smithi154 conmon[34553]: cluster 2021-07-31T02:38:29.476664+0000 mon.a (mon.0) 2688 : cluster [DBG] osdmap e1685: 8 total, 8 up, 8 in 2021-07-31T02:38:30.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:29 smithi045 conmon[33655]: cluster 2021-07-31T02:38:28.535393+0000 mgr.x 2021-07-31T02:38:30.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:29 smithi045 conmon[33655]: (mgr.24542) 2191 : cluster [DBG] pgmap v3068: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:38:30.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:29 smithi045 conmon[33655]: audit 2021-07-31T02:38:29.328312+0000 mgr.x 2021-07-31T02:38:30.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:29 smithi045 conmon[33655]: (mgr.24542) 2192 : audit [DBG] from='client.17754 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:38:30.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:29 smithi045 conmon[33655]: cluster 2021-07-31T 2021-07-31T02:38:30.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:29 smithi045 conmon[33655]: 02:38:29.476664+0000 mon.a (mon 2021-07-31T02:38:30.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:29 smithi045 conmon[33655]: .0) 2688 : cluster [DBG] osdmap e1685: 8 total, 8 up, 8 in 2021-07-31T02:38:30.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:29 smithi045 conmon[33655]: 2021-07-31T02:38:30.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:29 smithi045 conmon[39718]: cluster 2021-07-31T02:38:28.535393+0000 mgr.x (mgr 2021-07-31T02:38:30.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:29 smithi045 conmon[39718]: .24542) 2191 : cluster [DBG] pgmap v3068: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:38:30.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:29 smithi045 conmon[39718]: audit 2021-07-31T02:38:29.328312+0000 mgr.x ( 2021-07-31T02:38:30.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:29 smithi045 conmon[39718]: mgr.24542) 2192 : audit [DBG] from='client.17754 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:38:30.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:29 smithi045 conmon[39718]: cluster 2021-07- 2021-07-31T02:38:30.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:29 smithi045 conmon[39718]: 31T02:38:29.476664+0000 mon.a (mon.0) 2688 : cluster [DBG] osdmap e1685: 8 total, 8 up, 8 in 2021-07-31T02:38:30.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:29 smithi045 conmon[39718]: 2021-07-31T02:38:31.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:30 smithi154 conmon[34553]: audit 2021-07-31T02:38:29.630191+0000 mgr.x (mgr.24542) 2193 : audit [DBG] from='client.27700 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:38:31.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:30 smithi154 conmon[34553]: audit 2021-07-31T02 2021-07-31T02:38:31.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:30 smithi154 conmon[34553]: :38:29.977127+0000 mon.a (mon.0) 2689 : audit [DBG] from='client.? 172.21.15.45:0/564153922' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:38:31.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:30 smithi154 conmon[34553]: cluster 2021-07-31T02:38:30.478592+0000 mon.a (mon.0) 2690 : cluster [DBG] osdmap e1686: 8 total, 8 up, 8 in 2021-07-31T02:38:31.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:30 smithi045 conmon[33655]: audit 2021-07-31T02:38:29.630191+0000 mgr.x (mgr.24542) 2193 : audit [DBG] from='client.27700 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:38:31.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:30 smithi045 conmon[33655]: 2021-07-31T02:38:31.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:30 smithi045 conmon[33655]: audit 2021-07-31T02:38:29.977127+0000 mon.a (mon.0) 2689 : audit [DBG] from='client.? 172.21.15.45:0/564153922' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:38:31.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:30 smithi045 conmon[33655]: cluster 2021-07-31T02:38:30.478592+0000 2021-07-31T02:38:31.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:30 smithi045 conmon[33655]: mon.a (mon.0) 2690 : cluster [DBG] osdmap e1686: 8 total, 8 up, 8 in 2021-07-31T02:38:31.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:30 smithi045 conmon[39718]: audit 2021-07-31T02:38:29.630191+0000 mgr.x (mgr.24542) 2193 : audit [DBG] from='client.27700 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:38:31.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:30 smithi045 conmon[39718]: audit 2021-07-31T02:38:29.977127+0000 mon.a (mon.0) 2689 : audit [DBG] from='client.? 172.21.15.45:0/564153922' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:38:31.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:30 smithi045 conmon[39718]: cluster 2021-07-31 2021-07-31T02:38:31.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:30 smithi045 conmon[39718]: T02:38:30.478592+0000 mon.a (mon.0) 2690 : cluster [DBG] 2021-07-31T02:38:31.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:30 smithi045 conmon[39718]: osdmap e1686: 8 total, 8 up, 8 in 2021-07-31T02:38:31.577 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestImage.test_list_snaps_iterator_auto_close ... ok 2021-07-31T02:38:31.673 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestImage.test_lock_unlock ... ok 2021-07-31T02:38:32.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:31 smithi154 conmon[34553]: cluster 2021-07-31T02:38:30.535807+0000 mgr.x (mgr.24542) 2194 : cluster [DBG] pgmap v3071: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 193 KiB/s rd, 8.2 KiB/s wr, 248 op/s 2021-07-31T02:38:32.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:31 smithi154 conmon[34553]: cluster 2021-07-31T02:38:31.483203+0000 mon.a (mon.0) 2691 : cluster [DBG] osdmap e1687: 8 total, 8 up, 8 in 2021-07-31T02:38:32.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:31 smithi045 conmon[33655]: cluster 2021-07-31T02:38:30.535807+0000 2021-07-31T02:38:32.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:31 smithi045 conmon[33655]: mgr.x (mgr.24542) 2194 : cluster [DBG] pgmap v3071: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 193 KiB/s rd, 8.2 KiB/s wr, 248 op/s 2021-07-31T02:38:32.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:31 smithi045 conmon[33655]: cluster 2021-07-31T02:38:31.483203+0000 mon.a (mon.0) 2691 : cluster [DBG] osdmap e1687: 8 total, 8 up, 8 in 2021-07-31T02:38:32.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:31 smithi045 conmon[39718]: cluster 2021-07-31T02:38:30.535807+0000 mgr.x ( 2021-07-31T02:38:32.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:31 smithi045 conmon[39718]: mgr.24542) 2194 : cluster [DBG] pgmap v3071: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 193 KiB/s rd, 8.2 KiB/s wr, 248 op/s 2021-07-31T02:38:32.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:31 smithi045 conmon[39718]: cluster 2021-07-31T02:38:31.483203+0000 mon.a (mon.0) 2691 : cluster 2021-07-31T02:38:32.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:31 smithi045 conmon[39718]: [DBG] osdmap e1687: 8 total, 8 up, 8 in 2021-07-31T02:38:33.467 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: debug 2021-07-31T02:38:33.054+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:38:33.467 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:38:33.468 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:38:33.468 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: yield 2021-07-31T02:38:33.468 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:38:33.468 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:38:33.468 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:38:33.469 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:38:33.469 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:38:33.469 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:38:33.469 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:38:33.469 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:38:33.470 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:38:33.470 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: task.result() 2021-07-31T02:38:33.470 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:38:33.470 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: debug 2021-07-31T02:38:33.066+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:38:33.470 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:38:33.471 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:38:33.471 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: yield 2021-07-31T02:38:33.471 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:38:33.471 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:38:33.471 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:38:33.472 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:38:33.472 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:38:33.472 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:38:33.472 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:38:33.472 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:38:33.473 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:38:33.473 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: task.result() 2021-07-31T02:38:33.473 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:38:33.473 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: debug 2021-07-31T02:38:33.092+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:38:33.473 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:38:33.474 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:38:33.474 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: yield 2021-07-31T02:38:33.474 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:38:33.474 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:38:33.475 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:38:33.475 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:38:33.475 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:38:33.476 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:38:33.477 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:38:33.477 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:38:33.477 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:38:33.477 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: task.result() 2021-07-31T02:38:33.478 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:38:33.478 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: debug 2021-07-31T02:38:33.092+0000 7fcb93160700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:38:33.478 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:38:33.478 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:38:33.479 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: yield 2021-07-31T02:38:33.479 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:38:33.479 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:38:33.479 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:38:33.479 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:38:33.480 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:38:33.480 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:38:33.480 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:38:33.480 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:38:33.480 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:38:33.481 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: task.result() 2021-07-31T02:38:33.481 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:38:33.481 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: 2021-07-31T02:38:33.481 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:38:33.482 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: 2021-07-31T02:38:33.482 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:38:33.482 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:38:33.482 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:38:33.482 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:38:33.483 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:38:33.483 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:38:33.483 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:38:33.483 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:38:33.483 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:38:33.484 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:38:33.484 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:38:33.484 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:38:33.484 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:38:33.485 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:38:33.485 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:38:33.485 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:38:33.485 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:38:33.486 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:38:33.486 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: raise 2021-07-31T02:38:33.486 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:38:33.486 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:38:33.486 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:38:33.487 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:38:33.487 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:38:33.487 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: debug 2021-07-31T02:38:33.101+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:38:33.487 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:38:33.487 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:38:33.488 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: yield 2021-07-31T02:38:33.488 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:38:33.488 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:38:33.488 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:38:33.488 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:38:33.489 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:38:33.489 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:38:33.489 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:38:33.489 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:38:33.489 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:38:33.490 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: task.result() 2021-07-31T02:38:33.490 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:38:33.490 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: debug 2021-07-31T02:38:33.101+0000 7fcb9295f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:38:33.490 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:38:33.491 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:38:33.491 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: yield 2021-07-31T02:38:33.491 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:38:33.491 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:38:33.491 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:38:33.492 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:38:33.492 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:38:33.492 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:38:33.492 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:38:33.492 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:38:33.493 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:38:33.493 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: task.result() 2021-07-31T02:38:33.493 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:38:33.493 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: 2021-07-31T02:38:33.494 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:38:33.494 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: 2021-07-31T02:38:33.494 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:38:33.494 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:38:33.495 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:38:33.495 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:38:33.495 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:38:33.495 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:38:33.495 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:38:33.496 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:38:33.496 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:38:33.496 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:38:33.496 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:38:33.497 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:38:33.497 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:38:33.497 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:38:33.497 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:38:33.497 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:38:33.498 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:38:33.498 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:38:33.498 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: raise 2021-07-31T02:38:33.498 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:38:33.498 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:38:33.499 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:38:33.499 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:38:33.499 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:38:33.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: cluster 2021-07-31T02:38:32.466945+0000 mon.a (mon.0) 2692 : cluster [DBG] osdmap e1688: 8 total, 8 up, 8 in 2021-07-31T02:38:33.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: cluster 2021-07-31T02:38:32.536278+0000 mgr.x (mgr.24542) 2195 : cluster [DBG] pgmap v3074: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 187 KiB/s rd, 8.0 KiB/s wr, 241 op/s 2021-07-31T02:38:33.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: audit 2021-07-31T02:38:33.009496+0000 mon.b (mon.2) 2021-07-31T02:38:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: 332 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:38:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: audit 2021- 2021-07-31T02:38:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: 07-31T02:38:33.010998+0000 mon.b (mon.2) 333 : audit 2021-07-31T02:38:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:38:33.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: audit 2021-07-31 2021-07-31T02:38:33.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: T02:38:33.011907+0000 mon.b (mon 2021-07-31T02:38:33.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: .2) 334 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:38:33.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:38:33.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: 2021-07-31T02:38: 2021-07-31T02:38:33.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: 33.012666+0000 mgr.x (mgr.24542 2021-07-31T02:38:33.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: ) 2196 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:38:33.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:38:33.055415+0000 mgr.x (mgr 2021-07-31T02:38:33.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: .24542) 2197 : cephadm [ERR] 2021-07-31T02:38:33.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:38:33.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:38:33.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: yield 2021-07-31T02:38:33.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:38:33.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:38:33.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:38:33.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:38:33.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:38:33.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:38:33.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:38:33.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:38:33.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:38:33.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:38:33.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:38:33.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:38:33.055827+0000 mgr.x (mgr.24542) 2198 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:38:33.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: cephadm 2021 2021-07-31T02:38:33.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: -07-31T02:38:33.067029+0000 mgr.x 2021-07-31T02:38:33.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: (mgr.24542) 2199 : cephadm [ERR] 2021-07-31T02:38:33.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: 2021-07-31T02:38:33.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:38:33.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:38:33.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: yield 2021-07-31T02:38:33.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:38:33.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:38:33.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:38:33.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:38:33.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:38:33.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:38:33.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:38:33.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:38:33.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:38:33.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:38:33.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:38:33.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:38:33.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: 2021-07-31T 2021-07-31T02:38:33.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: 02:38:33.067607+0000 mgr.x (mgr.24542) 2200 : 2021-07-31T02:38:33.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:38:33.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:38:33.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: 2021-07-31 2021-07-31T02:38:33.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: T02:38:33.093051+0000 mgr.x (mgr.24542 2021-07-31T02:38:33.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: ) 2201 : cephadm [ERR] 2021-07-31T02:38:33.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:38:33.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:38:33.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: yield 2021-07-31T02:38:33.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:38:33.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:38:33.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:38:33.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:38:33.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:38:33.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:38:33.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:38:33.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:38:33.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:38:33.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:38:33.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:38:33.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:38:33.093452+0000 mgr.x (mgr.24542) 2202 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:38:33.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:38:33.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:38:33.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: yield 2021-07-31T02:38:33.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:38:33.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:38:33.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:38:33.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:38:33.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:38:33.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:38:33.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:38:33.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:38:33.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:38:33.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:38:33.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:38:33.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: 2021-07-31T02:38:33.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:38:33.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: 2021-07-31T02:38:33.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:38:33.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:38:33.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:38:33.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:38:33.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:38:33.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:38:33.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:38:33.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:38:33.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:38:33.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:38:33.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:38:33.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:38:33.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:38:33.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:38:33.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:38:33.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:38:33.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:38:33.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:38:33.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: raise 2021-07-31T02:38:33.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:38:33.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:38:33.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:38:33.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:38:33.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:38:33.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:38:33.102198+0000 mgr.x (mgr.24542) 2203 : cephadm [ERR] 2021-07-31T02:38:33.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:38:33.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:38:33.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: yield 2021-07-31T02:38:33.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:38:33.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:38:33.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:38:33.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:38:33.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: cluster 2021-07-31T02:38 2021-07-31T02:38:33.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: :32.466945+0000 mon.a (mon.0) 2692 : cluster [DBG] osdmap e1688: 8 total, 8 up, 8 in 2021-07-31T02:38:33.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: cluster 2021- 2021-07-31T02:38:33.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: 07-31T02:38:32.536278+0000 mgr.x (mgr.24542) 2195 : cluster [DBG] pgmap v3074: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 187 KiB/s rd, 8.0 KiB/s wr, 241 op/s 2021-07-31T02:38:33.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: 2021-07-31T02:38:33.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: audit 2021-07 2021-07-31T02:38:33.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: -31T02:38:33.009496+0000 mon.b (mon.2) 332 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:38:33.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: audit 2021-07-31T02 2021-07-31T02:38:33.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: :38:33.010998+0000 mon.b (mon.2) 333 : audit [DBG] 2021-07-31T02:38:33.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:38:33.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: audit 2021- 2021-07-31T02:38:33.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: 07-31T02:38:33.011907+0000 mon.b (mon.2) 334 : audit [INF] 2021-07-31T02:38:33.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:38:33.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: cephadm 2021 2021-07-31T02:38:33.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: -07-31T02:38:33.012666+0000 mgr.x (mgr.24542) 2196 : cephadm 2021-07-31T02:38:33.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:38:33.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:38:33.055415+0000 2021-07-31T02:38:33.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: mgr.x (mgr.24542) 2197 : cephadm [ERR] 2021-07-31T02:38:33.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:38:33.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:38:33.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: yield 2021-07-31T02:38:33.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:38:33.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:38:33.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:38:33.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:38:33.788 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:38:33.788 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:38:33.788 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:38:33.788 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:38:33.789 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:38:33.789 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:38:33.789 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:38:33.789 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:38:33.055827+0000 mgr.x (mgr.24542) 2198 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:38:33.789 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: cephadm 2021- 2021-07-31T02:38:33.790 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: 07-31T02:38:33.067029+0000 mgr.x (mgr 2021-07-31T02:38:33.790 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: .24542) 2199 : cephadm [ERR] 2021-07-31T02:38:33.790 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:38:33.790 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:38:33.791 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: yield 2021-07-31T02:38:33.791 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:38:33.791 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:38:33.791 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:38:33.791 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:38:33.792 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:38:33.792 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:38:33.792 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:38:33.792 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:38:33.792 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:38:33.793 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:38:33.793 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:38:33.793 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:38:33.067607+0000 mgr.x (mgr.24542) 2200 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:38:33.793 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:38:33.093051+0000 mgr.x (mgr.24542) 2201 : cephadm [ERR] 2021-07-31T02:38:33.794 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:38:33.794 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:38:33.794 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: yield 2021-07-31T02:38:33.794 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:38:33.794 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:38:33.795 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:38:33.796 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:38:33.796 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:38:33.796 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:38:33.796 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:38:33.797 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:38:33.797 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:38:33.798 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:38:33.798 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:38:33.798 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:38:33.093452+0000 mgr.x (mgr.24542) 2202 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:38:33.798 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:38:33.798 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:38:33.799 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: yield 2021-07-31T02:38:33.799 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:38:33.799 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:38:33.800 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:38:33.801 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:38:33.801 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:38:33.801 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:38:33.801 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:38:33.802 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:38:33.802 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:38:33.802 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:38:33.803 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:38:33.803 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:38:33.803 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:38:33.803 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:38:33.804 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:38:33.804 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:38:33.102594+0000 mgr.x (mgr.24542) 2204 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:38:33.804 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:38:33.804 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:38:33.805 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: yield 2021-07-31T02:38:33.805 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:38:33.805 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:38:33.805 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:38:33.805 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:38:33.806 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:38:33.806 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:38:33.806 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:38:33.806 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:38:33.807 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:38:33.807 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:38:33.807 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:38:33.808 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: 2021-07-31T02:38:33.808 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:38:33.808 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: 2021-07-31T02:38:33.808 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:38:33.809 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:38:33.809 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:38:33.809 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:38:33.809 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:38:33.810 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:38:33.810 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:38:33.811 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:38:33.811 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:38:33.811 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:38:33.812 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:38:33.812 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:38:33.812 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:38:33.812 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:38:33.812 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:38:33.814 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:38:33.814 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:38:33.814 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:38:33.814 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: raise 2021-07-31T02:38:33.814 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:38:33.815 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:38:33.816 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:38:33.816 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:38:33.816 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:33 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:38:33.817 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: cluster 2021-07-31T02:38 2021-07-31T02:38:33.817 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: :32.466945+0000 mon.a (mon.0) 2692 : cluster [DBG] osdmap e1688: 8 total, 8 up, 8 in 2021-07-31T02:38:33.817 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: cluster 2021- 2021-07-31T02:38:33.818 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: 07-31T02:38:32. 2021-07-31T02:38:33.818 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: 536278+0000 mgr.x (mgr.24542 2021-07-31T02:38:33.819 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: ) 2195 : cluster [DBG] pgmap v3074: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 187 KiB/s rd, 8.0 KiB/s wr, 241 op/s 2021-07-31T02:38:33.819 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: 2021-07-31T02:38:33.819 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: audit 2021-07 2021-07-31T02:38:33.819 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: -31T02:38:33. 2021-07-31T02:38:33.819 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: 009496+0000 mon.b (mon. 2021-07-31T02:38:33.820 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: 2) 332 : audit [DBG] 2021-07-31T02:38:33.820 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:38:33.820 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: audit 2021-07 2021-07-31T02:38:33.821 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: -31T02:38:33.010998+0000 mon.b 2021-07-31T02:38:33.821 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: (mon.2) 333 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:38:33.822 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: 2021-07-31T02:38:33.822 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: audit 2021-07-31T02 2021-07-31T02:38:33.822 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: :38:33.011907+0000 mon.b (mon 2021-07-31T02:38:33.822 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: .2) 334 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:38:33.822 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: cephadm 2021-07-31T 2021-07-31T02:38:33.823 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: 02:38:33.012666+0000 mgr.x (mgr.24542 2021-07-31T02:38:33.823 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: ) 2196 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:38:33.823 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: cephadm 2021- 2021-07-31T02:38:33.823 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: 07-31T02:38:33.055415 2021-07-31T02:38:33.824 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 2197 : cephadm 2021-07-31T02:38:33.824 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: [ERR] 2021-07-31T02:38:33.824 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:38:33.824 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:38:33.825 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: yield 2021-07-31T02:38:33.825 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:38:33.825 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:38:33.825 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:38:33.825 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:38:33.826 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:38:33.826 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:38:33.826 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:38:33.826 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:38:33.827 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:38:33.827 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:38:33.827 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:38:33.827 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:38:33.055827+0000 mgr.x (mgr.24542) 2198 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:38:33.827 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:38: 2021-07-31T02:38:33.828 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: 33.067029+0000 mgr.x (mgr.24542) 2199 : cephadm 2021-07-31T02:38:33.828 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: [ERR] 2021-07-31T02:38:33.829 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:38:33.829 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:38:33.830 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:38:33.830 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: 2021-07-31T02:38:33.830 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:38:33.831 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: 2021-07-31T02:38:33.831 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:38:33.831 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:38:33.832 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:38:33.832 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:38:33.832 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:38:33.832 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:38:33.833 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:38:33.833 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:38:33.833 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:38:33.833 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:38:33.834 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:38:33.834 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:38:33.834 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:38:33.834 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:38:33.834 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:38:33.835 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:38:33.835 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:38:33.835 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:38:33.835 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: raise 2021-07-31T02:38:33.836 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:38:33.836 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:38:33.836 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:38:33.836 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:38:33.836 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:38:33.837 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:38:33.102198+0000 mgr.x (mgr.24542) 2203 : cephadm [ERR] 2021-07-31T02:38:33.837 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:38:33.837 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:38:33.837 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: yield 2021-07-31T02:38:33.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:38:33.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:38:33.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:38:33.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:38:33.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:38:33.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:38:33.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:38:33.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:38:33.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:38:33.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:38:33.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:38:33.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:38:33.102594+0000 mgr.x (mgr.24542) 2204 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:38:33.841 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:38:33.841 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:38:33.841 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: yield 2021-07-31T02:38:33.841 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:38:33.842 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:38:33.842 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:38:33.842 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:38:33.843 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:38:33.843 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:38:33.843 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:38:33.843 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:38:33.844 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:38:33.844 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:38:33.844 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:38:33.844 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: 2021-07-31T02:38:33.844 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:38:33.845 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: 2021-07-31T02:38:33.845 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:38:33.845 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:38:33.845 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:38:33.846 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:38:33.846 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:38:33.846 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:38:33.847 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:38:33.847 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:38:33.847 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:38:33.847 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:38:33.847 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:38:33.848 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:38:33.848 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:38:33.848 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:38:33.848 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:38:33.850 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:38:33.850 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:38:33.850 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:38:33.851 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: raise 2021-07-31T02:38:33.851 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:38:33.852 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:38:33.852 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:38:33.852 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:38:33.852 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:38:33.853 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:38:33.853 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:38:33.854 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: yield 2021-07-31T02:38:33.854 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:38:33.854 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:38:33.854 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:38:33.855 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:38:33.855 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:38:33.855 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:38:33.855 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:38:33.855 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:38:33.856 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:38:33.856 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:38:33.856 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:38:33.856 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:38:33.067607+0000 mgr.x (mgr.24542) 2200 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:38:33.857 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: cephadm 2021-07-31T02 2021-07-31T02:38:33.857 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: :38:33.093051+0000 mgr.x (mgr.24542) 2201 2021-07-31T02:38:33.857 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: : cephadm [ERR] 2021-07-31T02:38:33.857 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:38:33.858 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:38:33.858 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: yield 2021-07-31T02:38:33.858 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:38:33.858 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:38:33.858 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:38:33.860 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:38:33.860 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:38:33.861 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:38:33.861 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:38:33.861 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:38:33.861 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:38:33.861 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:38:33.862 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:38:33.862 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:38:33.093452+0000 mgr.x (mgr.24542) 2202 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:38:33.863 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:38:33.863 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:38:33.863 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: yield 2021-07-31T02:38:33.863 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:38:33.864 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:38:33.864 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:38:33.864 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:38:33.864 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:38:33.864 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:38:33.865 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:38:33.865 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:38:33.865 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:38:33.865 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:38:33.865 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:38:33.866 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: 2021-07-31T02:38:33.866 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:38:33.866 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: 2021-07-31T02:38:33.866 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:38:33.866 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:38:33.867 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:38:33.867 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:38:33.867 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:38:33.867 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:38:33.868 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:38:33.868 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:38:33.868 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:38:33.868 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:38:33.868 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:38:33.869 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:38:33.869 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:38:33.869 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:38:33.869 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:38:33.869 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:38:33.870 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:38:33.870 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:38:33.870 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: raise 2021-07-31T02:38:33.870 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:38:33.871 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:38:33.871 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:38:33.871 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:38:33.871 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:38:33.871 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: cephadm 2021-07 2021-07-31T02:38:33.872 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: -31T02:38:33.102198+0000 mgr.x (mgr.24542) 2203 : cephadm [ERR] 2021-07-31T02:38:33.872 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:38:33.872 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:38:33.873 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: yield 2021-07-31T02:38:33.873 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:38:33.873 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:38:33.873 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:38:33.874 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:38:33.874 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:38:33.874 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:38:33.874 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:38:33.874 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:38:33.875 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:38:33.875 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:38:33.875 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:38:33.875 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:38:33.102594+0000 mgr.x (mgr.24542) 2204 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:38:33.876 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:38:33.876 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:38:33.876 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: yield 2021-07-31T02:38:33.876 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:38:33.876 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:38:33.877 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:38:33.877 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:38:33.877 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:38:33.877 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:38:33.877 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:38:33.878 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:38:33.878 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:38:33.878 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:38:33.878 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:38:33.879 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: 2021-07-31T02:38:33.879 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:38:33.879 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: 2021-07-31T02:38:33.879 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:38:33.879 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:38:33.880 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:38:33.880 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:38:33.880 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:38:33.880 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:38:33.881 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:38:33.881 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:38:33.881 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:38:33.881 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:38:33.881 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:38:33.882 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:38:33.882 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:38:33.882 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:38:33.882 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:38:33.882 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:38:33.883 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:38:33.883 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:38:33.883 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: raise 2021-07-31T02:38:33.883 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:38:33.884 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:38:33.884 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:38:33.884 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:38:33.884 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:33 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:38:34.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:34 smithi154 conmon[34553]: cluster 2021-07-31T02:38:33.480115+0000 mon.a (mon.0) 2693 : cluster [DBG] osdmap e1689: 8 total, 8 up, 8 in 2021-07-31T02:38:34.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:34 smithi045 conmon[33655]: cluster 2021-07-31T02:38:33.480115+0000 mon.a (mon.0) 2693 : cluster [DBG] 2021-07-31T02:38:34.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:34 smithi045 conmon[33655]: osdmap e1689: 8 total, 8 up, 8 in 2021-07-31T02:38:34.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:34 smithi045 conmon[39718]: cluster 2021-07-31T02:38:33.480115+0000 mon.a (mon.0) 2693 2021-07-31T02:38:34.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:34 smithi045 conmon[39718]: : cluster [DBG] osdmap e1689: 8 total, 8 up, 8 in 2021-07-31T02:38:35.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:35 smithi154 conmon[34553]: cluster 2021-07-31T02:38:34.486800+0000 mon.a (mon.0) 2694 : cluster [DBG] 2021-07-31T02:38:35.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:35 smithi154 conmon[34553]: osdmap e1690: 8 total, 8 up, 8 in 2021-07-31T02:38:35.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:35 smithi154 conmon[34553]: cluster 2021-07-31T02:38:34.536715+0000 mgr.x (mgr.24542) 2205 : cluster [DBG] pgmap v3077: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 129 KiB/s rd, 8.7 KiB/s wr, 177 op/s 2021-07-31T02:38:35.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:35 smithi045 conmon[39718]: cluster 2021-07-31T02:38:34.486800+0000 mon.a ( 2021-07-31T02:38:35.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:35 smithi045 conmon[39718]: mon.0) 2694 : cluster [DBG] osdmap e1690: 8 total, 8 up, 8 in 2021-07-31T02:38:35.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:35 smithi045 conmon[39718]: cluster 2021-07-31T02:38:34.536715 2021-07-31T02:38:35.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:35 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 2205 : cluster [DBG] pgmap v3077: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 129 KiB/s rd, 8.7 KiB/s wr, 177 op/s 2021-07-31T02:38:35.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:35 smithi045 conmon[33655]: cluster 2021-07-31T02:38:34.486800+0000 mon.a (mon.0) 2694 : 2021-07-31T02:38:35.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:35 smithi045 conmon[33655]: cluster [DBG] osdmap e1690: 8 total, 8 up, 8 in 2021-07-31T02:38:35.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:35 smithi045 conmon[33655]: cluster 2021-07-31T02:38:34.536715+0000 mgr.x (mgr.24542) 2205 : cluster [DBG] pgmap v3077: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 129 KiB/s rd, 8.7 KiB/s wr, 177 op/s 2021-07-31T02:38:36.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:36 smithi154 conmon[34553]: cluster 2021-07-31T02:38:36.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:36 smithi154 conmon[34553]: 2021-07-31T02:38:35.492468+0000 mon.a 2021-07-31T02:38:36.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:36 smithi154 conmon[34553]: (mon.0) 2695 : cluster [DBG] osdmap e1691: 8 total, 8 up, 8 in 2021-07-31T02:38:36.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:36 smithi045 conmon[39718]: cluster 2021-07-31T02:38:35.492468+0000 mon.a (mon.0) 2695 : 2021-07-31T02:38:36.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:36 smithi045 conmon[39718]: cluster [DBG] osdmap e1691: 8 total, 8 up, 8 in 2021-07-31T02:38:36.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:36 smithi045 conmon[33655]: cluster 2021-07-31T02:38:35.492468+0000 mon.a (mon.0) 2695 : cluster [DBG] osdmap e1691: 8 total, 8 up, 8 in 2021-07-31T02:38:37.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:37 smithi154 conmon[34553]: cluster 2021-07-31T02:38:36.494225+0000 mon.a (mon.0) 2696 : cluster 2021-07-31T02:38:37.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:37 smithi154 conmon[34553]: [DBG] osdmap e1692: 8 total, 8 up, 8 in 2021-07-31T02:38:37.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:37 smithi154 conmon[34553]: cluster 2021-07-31T02:38:36.537173+0000 mgr.x (mgr.24542) 2206 : cluster [DBG] pgmap v3080: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 129 KiB/s rd, 8.7 KiB/s wr, 177 op/s 2021-07-31T02:38:37.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:37 smithi154 conmon[34553]: cluster 2021-07-31T02:38:37 2021-07-31T02:38:37.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:37 smithi154 conmon[34553]: .468169+0000 mon.a (mon.0) 2697 : cluster [DBG] osdmap e1693: 8 total, 8 up, 8 in 2021-07-31T02:38:37.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:37 smithi045 conmon[33655]: cluster 2021-07-31T02:38:36.494225+0000 mon.a (mon. 2021-07-31T02:38:37.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:37 smithi045 conmon[33655]: 0) 2696 : cluster [DBG] osdmap e1692: 8 total, 8 up, 8 in 2021-07-31T02:38:37.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:37 smithi045 conmon[33655]: cluster 2021-07-31T02:38:36.537173+0000 mgr.x (mgr.24542) 2206 : cluster [DBG] pgmap v3080: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 129 KiB/s rd, 8.7 KiB/s wr, 177 op/s 2021-07-31T02:38:37.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:37 smithi045 conmon[33655]: cluster 2021-07-31T02: 2021-07-31T02:38:37.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:37 smithi045 conmon[33655]: 38:37.468169+0000 mon.a (mon.0) 2697 : cluster [DBG] osdmap e1693: 8 total, 8 up, 8 in 2021-07-31T02:38:37.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:37 smithi045 conmon[39718]: cluster 2021-07-31T02: 2021-07-31T02:38:37.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:37 smithi045 conmon[39718]: 38:36.494225+0000 mon.a (mon.0) 2696 : cluster [DBG] osdmap e1692: 8 total, 8 up, 8 in 2021-07-31T02:38:37.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:37 smithi045 conmon[39718]: cluster 2021 2021-07-31T02:38:37.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:37 smithi045 conmon[39718]: -07-31T02:38:36 2021-07-31T02:38:37.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:37 smithi045 conmon[39718]: .537173+0000 mgr.x (mgr. 2021-07-31T02:38:37.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:37 smithi045 conmon[39718]: 24542) 2206 : cluster [DBG] 2021-07-31T02:38:37.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:37 smithi045 conmon[39718]: pgmap v3080: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 129 KiB/s rd, 8.7 KiB/s wr, 177 op/s 2021-07-31T02:38:37.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:37 smithi045 conmon[39718]: cluster 2021-07-31T02:38:37.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:37 smithi045 conmon[39718]: 2021-07-31T02: 2021-07-31T02:38:37.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:37 smithi045 conmon[39718]: 38:37.468169+0000 mon.a 2021-07-31T02:38:37.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:37 smithi045 conmon[39718]: (mon.0) 2697 : cluster 2021-07-31T02:38:37.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:37 smithi045 conmon[39718]: [DBG] osdmap e1693: 8 total, 8 up, 8 in 2021-07-31T02:38:39.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:39 smithi154 conmon[34553]: cluster 2021-07-31T02:38:38.475240+0000 mon.a (mon.0) 2698 : cluster [DBG] osdmap e1694: 8 total, 8 up, 8 in 2021-07-31T02:38:39.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:39 smithi154 conmon[34553]: cluster 2021-07-31T02:38:38.537559+0000 mgr.x (mgr.24542) 2207 : cluster [DBG] pgmap v3083: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:38:39.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:39 smithi045 conmon[33655]: cluster 2021-07-31T02:38:38.475240+0000 mon.a ( 2021-07-31T02:38:39.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:39 smithi045 conmon[33655]: mon.0) 2698 : cluster [DBG] osdmap e1694: 8 total, 8 up, 8 in 2021-07-31T02:38:39.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:39 smithi045 conmon[33655]: cluster 2021-07-31T02:38:38.537559+0000 mgr.x (mgr.24542) 2207 : cluster [DBG] pgmap v3083: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:38:39.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:39 smithi045 conmon[39718]: cluster 2021-07-31T02:38:38.475240+0000 2021-07-31T02:38:39.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:39 smithi045 conmon[39718]: mon.a (mon.0) 2698 : cluster [DBG] osdmap e1694: 8 total, 8 up, 8 in 2021-07-31T02:38:39.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:39 smithi045 conmon[39718]: cluster 2021 2021-07-31T02:38:39.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:39 smithi045 conmon[39718]: -07-31T02:38:38.537559+0000 mgr.x (mgr.24542) 2207 : cluster [DBG] 2021-07-31T02:38:39.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:39 smithi045 conmon[39718]: pgmap v3083: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:38:40.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:40 smithi154 conmon[34553]: cluster 2021-07-31T02:38:39.482066+0000 mon.a (mon.0 2021-07-31T02:38:40.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:40 smithi154 conmon[34553]: ) 2699 : cluster [DBG] osdmap e1695: 8 total, 8 up, 8 in 2021-07-31T02:38:40.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:40 smithi045 conmon[39718]: cluster 2021-07-31T02:38:39.482066+0000 mon.a (mon.0) 2699 : cluster [DBG] osdmap e1695: 8 total, 8 up, 8 in 2021-07-31T02:38:40.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:40 smithi045 conmon[33655]: cluster 2021-07-31T02:38:39.482066+0000 mon.a (mon.0 2021-07-31T02:38:40.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:40 smithi045 conmon[33655]: ) 2699 : cluster [DBG] osdmap e1695: 8 total, 8 up, 8 in 2021-07-31T02:38:41.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:41 smithi154 conmon[34553]: cluster 2021-07-31T02:38:40.482938+0000 mon.a (mon.0) 2700 : cluster 2021-07-31T02:38:41.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:41 smithi154 conmon[34553]: [DBG] osdmap e1696: 8 total, 8 up, 8 in 2021-07-31T02:38:41.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:41 smithi154 conmon[34553]: cluster 2021-07-31T02:38:40.538086+0000 mgr.x (mgr.24542) 2208 : cluster [DBG] pgmap v3086: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 61 KiB/s rd, 3.7 KiB/s wr, 81 op/s 2021-07-31T02:38:41.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:41 smithi045 conmon[39718]: cluster 2021-07-31T02:38:40.482938+0000 mon.a (mon.0) 2700 : cluster [DBG] osdmap e1696: 8 total, 8 up, 8 in 2021-07-31T02:38:41.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:41 smithi045 conmon[39718]: cluster 2021-07-31T02:38:40.538086+0000 mgr.x (mgr.24542) 2208 : cluster [DBG] pgmap v3086: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 61 KiB/s rd, 3.7 KiB/s wr, 81 op/s 2021-07-31T02:38:41.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:41 smithi045 conmon[33655]: cluster 2021-07-31T02:38:40.482938+0000 mon.a (mon.0 2021-07-31T02:38:41.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:41 smithi045 conmon[33655]: ) 2700 : cluster [DBG] osdmap e1696: 8 total, 8 up, 8 in 2021-07-31T02:38:41.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:41 smithi045 conmon[33655]: cluster 2021-07-31T02:38:40.538086+0000 mgr.x (mgr.24542) 2208 : cluster [DBG] pgmap v3086: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 61 KiB/s rd, 3.7 KiB/s wr, 81 op/s 2021-07-31T02:38:42.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:42 smithi154 conmon[34553]: cluster 2021-07-31T02:38:41.493217+0000 mon.a (mon.0) 2701 : cluster [DBG] osdmap e1697: 8 total, 8 up, 8 in 2021-07-31T02:38:42.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:42 smithi154 conmon[34553]: 2021-07-31T02:38:42.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:42 smithi045 conmon[39718]: cluster 2021-07-31T02:38:41.493217 2021-07-31T02:38:42.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:42 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2701 : cluster [DBG] osdmap e1697: 8 total, 8 up, 8 in 2021-07-31T02:38:42.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:42 smithi045 conmon[33655]: cluster 2021-07-31T02:38:41.493217+0000 mon.a (mon.0) 2701 : cluster [DBG] osdmap e1697: 8 total, 8 up, 8 in 2021-07-31T02:38:43.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:43 smithi045 conmon[39718]: cluster 2021-07-31T02:38:42.501102+0000 mon.a (mon. 2021-07-31T02:38:43.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:43 smithi045 conmon[39718]: 0) 2702 : cluster [DBG] osdmap e1698: 8 total, 8 up, 8 in 2021-07-31T02:38:43.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:43 smithi045 conmon[39718]: cluster 2021 2021-07-31T02:38:43.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:43 smithi045 conmon[39718]: -07-31T02:38:42.538478+0000 mgr.x (mgr.24542) 2209 : cluster [DBG] pgmap v3089: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 61 KiB/s rd, 3.7 KiB/s wr, 81 op/s 2021-07-31T02:38:43.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:43 smithi045 conmon[33655]: cluster 2021-07-31T02:38:42.501102+0000 mon.a (mon.0) 2702 : 2021-07-31T02:38:43.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:43 smithi045 conmon[33655]: cluster [DBG] osdmap e1698: 8 total, 8 up, 8 in 2021-07-31T02:38:43.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:43 smithi045 conmon[33655]: cluster 2021-07-31T02:38:42.538478+0000 mgr.x (mgr.24542) 2209 : cluster [DBG] pgmap v3089: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 61 KiB/s rd, 3.7 KiB/s wr, 81 op/s 2021-07-31T02:38:44.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:43 smithi154 conmon[34553]: cluster 2021-07-31T02:38:44.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:43 smithi154 conmon[34553]: 2021-07-31T02:38:42 2021-07-31T02:38:44.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:43 smithi154 conmon[34553]: .501102+0000 mon.a (mon.0) 2702 : cluster [DBG] osdmap e1698: 8 total, 8 up, 8 in 2021-07-31T02:38:44.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:43 smithi154 conmon[34553]: cluster 2021-07-31T02:38:42. 2021-07-31T02:38:44.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:43 smithi154 conmon[34553]: 538478+0000 mgr.x (mgr.24542) 2209 : cluster [DBG] pgmap v3089: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 61 KiB/s rd, 3.7 KiB/s wr, 81 op/s 2021-07-31T02:38:44.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:44 smithi045 conmon[33655]: cluster 2021-07-31T02:38:43.507100+0000 mon.a (mon.0) 2703 2021-07-31T02:38:44.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:44 smithi045 conmon[33655]: : cluster [DBG] osdmap e1699: 8 total, 8 up, 8 in 2021-07-31T02:38:44.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:44 smithi045 conmon[39718]: cluster 2021-07-31T02:38:43. 2021-07-31T02:38:44.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:44 smithi045 conmon[39718]: 507100+0000 mon.a (mon.0) 2703 : cluster [DBG] osdmap e1699: 8 total, 8 up, 8 in 2021-07-31T02:38:45.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:44 smithi154 conmon[34553]: cluster 2021-07-31T02:38:45.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:44 smithi154 conmon[34553]: 2021-07-31T02:38:45.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:44 smithi154 conmon[34553]: 2021 2021-07-31T02:38:45.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:44 smithi154 conmon[34553]: - 2021-07-31T02:38:45.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:44 smithi154 conmon[34553]: 07 2021-07-31T02:38:45.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:44 smithi154 conmon[34553]: -31T02:38:43 2021-07-31T02:38:45.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:44 smithi154 conmon[34553]: .507100+0000 mon.a (mon.0) 2703 : cluster [DBG] osdmap e1699: 8 total, 8 up, 8 in 2021-07-31T02:38:45.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:45 smithi045 conmon[33655]: cluster 2021-07-31T02:38:44.508225+0000 mon.a (mon.0) 2704 : cluster 2021-07-31T02:38:45.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:45 smithi045 conmon[33655]: [DBG] osdmap e1700: 8 total, 8 up, 8 in 2021-07-31T02:38:45.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:45 smithi045 conmon[33655]: cluster 2021-07-31T02:38:44.538958+0000 mgr.x (mgr.24542) 2210 : cluster [DBG] pgmap v3092: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 93 KiB/s rd, 3.7 KiB/s wr, 124 op/s 2021-07-31T02:38:45.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:45 smithi045 conmon[39718]: cluster 2021-07-31T02:38:44.508225+0000 mon.a (mon 2021-07-31T02:38:45.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:45 smithi045 conmon[39718]: .0) 2704 : cluster [DBG] osdmap e1700: 8 total, 8 up, 8 in 2021-07-31T02:38:45.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:45 smithi045 conmon[39718]: cluster 2021-07-31T02:38:44.538958+0000 mgr.x (mgr.24542) 2210 : cluster [DBG] pgmap v3092: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 93 KiB/s rd, 3.7 KiB/s wr, 124 op/s 2021-07-31T02:38:46.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:45 smithi154 conmon[34553]: cluster 2021-07-31T02:38:44.508225+0000 mon.a (mon.0) 2704 : cluster [DBG] 2021-07-31T02:38:46.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:45 smithi154 conmon[34553]: osdmap e1700: 8 total, 8 up, 8 in 2021-07-31T02:38:46.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:45 smithi154 conmon[34553]: cluster 2021-07-31T02:38:44.538958+0000 mgr.x (mgr.24542) 2210 : cluster [DBG] pgmap v3092: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 93 KiB/s rd, 3.7 KiB/s wr, 124 op/s 2021-07-31T02:38:46.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:46 smithi045 conmon[33655]: cluster 2021-07-31T02:38:45.518428+0000 mon.a (mon.0) 2705 : cluster [DBG] 2021-07-31T02:38:46.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:46 smithi045 conmon[33655]: osdmap e1701: 8 total, 8 up, 8 in 2021-07-31T02:38:46.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:46 smithi045 conmon[39718]: cluster 2021-07-31T02:38:45.518428+0000 mon.a 2021-07-31T02:38:46.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:46 smithi045 conmon[39718]: (mon.0) 2705 : cluster [DBG] osdmap e1701: 8 total, 8 up, 8 in 2021-07-31T02:38:47.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:46 smithi154 conmon[34553]: cluster 2021-07-31T02:38:47.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:46 smithi154 conmon[34553]: 2021-07-31T02:38:47.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:46 smithi154 conmon[34553]: 2021 2021-07-31T02:38:47.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:46 smithi154 conmon[34553]: -07-31T02:38 2021-07-31T02:38:47.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:46 smithi154 conmon[34553]: :45.518428+0000 mon.a (mon.0) 2705 : cluster [DBG] osdmap e1701: 8 total, 8 up, 8 in 2021-07-31T02:38:48.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:47 smithi154 conmon[34553]: cluster 2021-07-31T02:38:46.520197+0000 mon.a (mon.0) 2706 : cluster [DBG] osdmap e1702: 8 total, 8 up, 8 in 2021-07-31T02:38:48.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:47 smithi154 conmon[34553]: cluster 2021-07-31T02:38:46.539342+0000 mgr.x (mgr.24542) 2211 : cluster [DBG] pgmap v3095: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 93 KiB/s rd, 3.7 KiB/s wr, 124 op/s 2021-07-31T02:38:48.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:47 smithi045 conmon[33655]: cluster 2021-07-31T02:38:46.520197+0000 mon.a (mon.0) 2021-07-31T02:38:48.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:47 smithi045 conmon[33655]: 2706 : cluster [DBG] osdmap e1702: 8 total, 8 up, 8 in 2021-07-31T02:38:48.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:47 smithi045 conmon[33655]: cluster 2021-07-31T02:38:46.539342+0000 mgr.x (mgr.24542) 2211 : cluster 2021-07-31T02:38:48.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:47 smithi045 conmon[33655]: [DBG] pgmap v3095: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 93 KiB/s rd, 3.7 KiB/s wr, 124 op/s 2021-07-31T02:38:48.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:47 smithi045 conmon[39718]: cluster 2021-07-31T02:38: 2021-07-31T02:38:48.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:47 smithi045 conmon[39718]: 46.520197+0000 mon.a (mon.0) 2706 : cluster [DBG] osdmap e1702: 8 total, 8 up, 8 in 2021-07-31T02:38:48.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:47 smithi045 conmon[39718]: cluster 2021-07- 2021-07-31T02:38:48.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:47 smithi045 conmon[39718]: 31T02:38:46.539342+0000 mgr.x (mgr.24542) 2211 : cluster [DBG] pgmap v3095: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 93 KiB/s rd, 3.7 KiB/s wr, 124 op/s 2021-07-31T02:38:48.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:47 smithi045 conmon[39718]: 2021-07-31T02:38:49.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:48 smithi154 conmon[34553]: cluster 2021-07-31T02:38:47.533082+0000 mon.a ( 2021-07-31T02:38:49.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:48 smithi154 conmon[34553]: mon.0) 2707 : cluster [DBG] osdmap e1703: 8 total, 8 up, 8 in 2021-07-31T02:38:49.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:48 smithi045 conmon[39718]: cluster 2021-07-31T02:38:47.533082+0000 mon.a (mon 2021-07-31T02:38:49.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:48 smithi045 conmon[39718]: .0) 2707 : cluster [DBG] osdmap e1703: 8 total, 8 up, 8 in 2021-07-31T02:38:49.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:48 smithi045 conmon[33655]: cluster 2021-07-31T02:38:47.533082+0000 mon.a (mon.0) 2021-07-31T02:38:49.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:48 smithi045 conmon[33655]: 2707 : cluster [DBG] osdmap e1703: 8 total, 8 up, 8 in 2021-07-31T02:38:50.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:49 smithi154 conmon[34553]: cluster 2021-07-31T02:38:48.534637+0000 mon.a (mon.0) 2708 : cluster [DBG] osdmap e1704: 8 total, 8 up, 8 in 2021-07-31T02:38:50.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:49 smithi154 conmon[34553]: cluster 2021-07-31T02:38:48.539684+0000 mgr.x (mgr.24542) 2212 : cluster [DBG] pgmap v3098: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:38:50.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:49 smithi045 conmon[33655]: cluster 2021-07-31T02:38:48.534637+0000 mon.a (mon.0) 2708 : cluster 2021-07-31T02:38:50.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:49 smithi045 conmon[33655]: [DBG] osdmap e1704: 8 total, 8 up, 8 in 2021-07-31T02:38:50.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:49 smithi045 conmon[33655]: cluster 2021-07-31T02:38:48.539684+0000 mgr.x (mgr.24542) 2212 : cluster [DBG] pgmap v3098: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:38:50.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:49 smithi045 conmon[39718]: cluster 2021-07-31T02:38:48 2021-07-31T02:38:50.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:49 smithi045 conmon[39718]: .534637+0000 mon.a (mon.0) 2708 : cluster [DBG] osdmap e1704: 8 total, 8 up, 8 in 2021-07-31T02:38:50.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:49 smithi045 conmon[39718]: cluster 2021 2021-07-31T02:38:50.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:49 smithi045 conmon[39718]: -07-31T02:38:48 2021-07-31T02:38:50.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:49 smithi045 conmon[39718]: .539684+0000 mgr.x (mgr. 2021-07-31T02:38:50.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:49 smithi045 conmon[39718]: 24542) 2212 : cluster [DBG] pgmap v3098: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:38:50.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:49 smithi045 conmon[39718]: 2021-07-31T02:38:51.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:50 smithi154 conmon[34553]: cluster 2021-07-31T02:38:49.545794+0000 mon.a (mon.0 2021-07-31T02:38:51.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:50 smithi154 conmon[34553]: ) 2709 : cluster [DBG] osdmap e1705: 8 total, 8 up, 8 in 2021-07-31T02:38:51.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:50 smithi045 conmon[33655]: cluster 2021-07-31T02:38:49.545794+0000 mon.a (mon.0) 2709 : cluster [DBG] 2021-07-31T02:38:51.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:50 smithi045 conmon[33655]: osdmap e1705: 8 total, 8 up, 8 in 2021-07-31T02:38:51.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:50 smithi045 conmon[39718]: cluster 2021-07-31T02:38:49.545794+0000 mon.a (mon.0) 2709 : cluster [DBG] 2021-07-31T02:38:51.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:50 smithi045 conmon[39718]: osdmap e1705: 8 total, 8 up, 8 in 2021-07-31T02:38:52.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:51 smithi154 conmon[34553]: cluster 2021-07-31T02:38:50.540193+0000 mgr.x (mgr.24542) 2213 : cluster 2021-07-31T02:38:52.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:51 smithi154 conmon[34553]: [DBG] pgmap v3100: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 126 KiB/s rd, 3.7 KiB/s wr, 167 op/s 2021-07-31T02:38:52.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:51 smithi154 conmon[34553]: cluster 2021-07-31T02:38:50.551675+0000 mon.a (mon.0) 2710 : cluster [DBG] osdmap e1706: 8 total, 8 up, 8 in 2021-07-31T02:38:52.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:51 smithi045 conmon[33655]: cluster 2021-07-31T02:38:50.540193+0000 mgr.x (mgr.24542) 2213 : cluster [DBG] pgmap v3100: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 126 KiB/s rd, 3.7 KiB/s wr, 167 op/s 2021-07-31T02:38:52.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:51 smithi045 conmon[33655]: cluster 2021-07-31T02:38:50.551675+0000 mon.a (mon.0) 2710 : cluster [DBG] osdmap e1706: 8 total, 8 up, 8 in 2021-07-31T02:38:52.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:51 smithi045 conmon[39718]: cluster 2021-07-31T02:38:50.540193+0000 mgr.x (mgr.24542) 2213 : cluster [DBG] 2021-07-31T02:38:52.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:51 smithi045 conmon[39718]: pgmap v3100: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 126 KiB/s rd, 3.7 KiB/s wr, 167 op/s 2021-07-31T02:38:52.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:51 smithi045 conmon[39718]: cluster 2021-07-31T02:38:50.551675+0000 mon.a (mon.0) 2710 : cluster [DBG] osdmap e1706: 8 total, 8 up, 8 in 2021-07-31T02:38:52.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:51 smithi045 conmon[39718]: 2021-07-31T02:38:53.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:52 smithi154 conmon[34553]: cluster 2021-07-31T02:38:51.553197+0000 mon.a (mon.0) 2711 2021-07-31T02:38:53.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:52 smithi154 conmon[34553]: : cluster [DBG] osdmap e1707: 8 total, 8 up, 8 in 2021-07-31T02:38:53.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:52 smithi045 conmon[33655]: cluster 2021-07-31T02:38:51.553197+0000 2021-07-31T02:38:53.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:52 smithi045 conmon[33655]: mon.a (mon.0) 2711 : cluster [DBG] osdmap e1707: 8 total, 8 up, 8 in 2021-07-31T02:38:53.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:52 smithi045 conmon[39718]: cluster 2021-07-31T02:38:51.553197+0000 mon.a (mon 2021-07-31T02:38:53.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:52 smithi045 conmon[39718]: .0) 2711 : cluster [DBG] osdmap e1707: 8 total, 8 up, 8 in 2021-07-31T02:38:54.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:53 smithi154 conmon[34553]: cluster 2021-07-31T02:38:52.540659+0000 mgr.x (mgr.24542) 2214 : cluster [DBG] 2021-07-31T02:38:54.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:53 smithi154 conmon[34553]: pgmap v3103: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 126 KiB/s rd, 3.7 KiB/s wr, 168 op/s 2021-07-31T02:38:54.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:53 smithi154 conmon[34553]: cluster 2021-07-31T02:38:52.559009+0000 mon.a (mon.0) 2712 : cluster [DBG] osdmap e1708: 8 total, 8 up, 8 in 2021-07-31T02:38:54.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:53 smithi045 conmon[39718]: cluster 2021-07-31T02:38:52.540659+0000 2021-07-31T02:38:54.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:53 smithi045 conmon[39718]: mgr.x (mgr.24542) 2214 : cluster [DBG] pgmap v3103: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 126 KiB/s rd, 3.7 KiB/s wr, 168 op/s 2021-07-31T02:38:54.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:53 smithi045 conmon[39718]: cluster 2021- 2021-07-31T02:38:54.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:53 smithi045 conmon[39718]: 07-31T02:38:52.559009+0000 mon.a (mon.0) 2712 : cluster [DBG] osdmap e1708: 8 total, 8 up, 8 in 2021-07-31T02:38:54.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:53 smithi045 conmon[33655]: cluster 2021-07-31T02:38:52.540659 2021-07-31T02:38:54.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:53 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 2214 : cluster [DBG] pgmap v3103: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 126 KiB/s rd, 3.7 KiB/s wr, 168 op/s 2021-07-31T02:38:54.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:53 smithi045 conmon[33655]: cluster 2021-07-31T02:38:54.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:53 smithi045 conmon[33655]: 2021-07-31T02:38:52.559009+0000 mon.a (mon.0) 2712 2021-07-31T02:38:54.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:53 smithi045 conmon[33655]: : cluster [DBG] osdmap e1708: 8 total, 8 up, 8 in 2021-07-31T02:38:55.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:54 smithi154 conmon[34553]: cluster 2021-07-31T02:38:53.565206+0000 mon.a (mon.0) 2713 : 2021-07-31T02:38:55.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:54 smithi154 conmon[34553]: cluster [DBG] osdmap e1709: 8 total, 8 up, 8 in 2021-07-31T02:38:55.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:54 smithi045 conmon[39718]: cluster 2021-07-31T02:38:53.565206+0000 mon.a (mon.0) 2021-07-31T02:38:55.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:54 smithi045 conmon[39718]: 2713 : cluster [DBG] osdmap e1709: 8 total, 8 up, 8 in 2021-07-31T02:38:55.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:54 smithi045 conmon[33655]: cluster 2021-07-31T02:38:53.565206+0000 mon.a (mon.0) 2713 : cluster [DBG] osdmap e1709: 8 total, 8 up, 8 in 2021-07-31T02:38:55.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:54 smithi045 conmon[33655]: 2021-07-31T02:38:56.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:55 smithi154 conmon[34553]: cluster 2021-07-31T02:38:54.541362+0000 mgr.x (mgr.24542) 2215 : cluster [DBG] pgmap v3106: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 157 KiB/s rd, 3.7 KiB/s wr, 212 op/s 2021-07-31T02:38:56.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:55 smithi154 conmon[34553]: 2021-07-31T02:38:56.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:55 smithi154 conmon[34553]: cluster 2021-07-31T02:38:54.575346+0000 mon.a (mon.0) 2714 : cluster [DBG] osdmap e1710: 8 total, 8 up, 8 in 2021-07-31T02:38:56.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:55 smithi045 conmon[33655]: cluster 2021-07-31T02:38:54.541362+0000 mgr.x (mgr.24542) 2215 : cluster 2021-07-31T02:38:56.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:55 smithi045 conmon[33655]: [DBG] pgmap v3106: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 157 KiB/s rd, 3.7 KiB/s wr, 212 op/s 2021-07-31T02:38:56.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:55 smithi045 conmon[33655]: cluster 2021-07-31T02:38:54.575346+0000 mon.a (mon.0) 2714 : cluster [DBG] osdmap e1710: 8 total, 8 up, 8 in 2021-07-31T02:38:56.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:55 smithi045 conmon[39718]: cluster 2021-07-31T02:38:54.541362+0000 mgr.x (mgr.24542) 2021-07-31T02:38:56.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:55 smithi045 conmon[39718]: 2215 : cluster [DBG] pgmap v3106: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 157 KiB/s rd, 3.7 KiB/s wr, 212 op/s 2021-07-31T02:38:56.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:55 smithi045 conmon[39718]: cluster 2021-07-31T02:38:54 2021-07-31T02:38:56.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:55 smithi045 conmon[39718]: .575346+0000 mon.a (mon.0) 2714 : cluster [DBG] osdmap e1710: 8 total, 8 up, 8 in 2021-07-31T02:38:57.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:56 smithi154 conmon[34553]: cluster 2021-07-31T02:38:55.582044+0000 mon.a (mon.0) 2715 : cluster [DBG] osdmap e1711: 8 total, 8 up, 8 in 2021-07-31T02:38:57.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:56 smithi045 conmon[33655]: cluster 2021-07-31T02:38:55.582044+0000 mon.a (mon.0) 2715 : 2021-07-31T02:38:57.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:56 smithi045 conmon[33655]: cluster [DBG] osdmap e1711: 8 total, 8 up, 8 in 2021-07-31T02:38:57.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:56 smithi045 conmon[39718]: cluster 2021-07-31T02:38:55.582044 2021-07-31T02:38:57.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:56 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2715 : cluster [DBG] osdmap e1711: 8 total, 8 up, 8 in 2021-07-31T02:38:58.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:57 smithi154 conmon[34553]: cluster 2021-07-31T02:38:58.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:57 smithi154 conmon[34553]: 2021-07-31T02:38:58.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:57 smithi154 conmon[34553]: 2021 2021-07-31T02:38:58.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:57 smithi154 conmon[34553]: - 2021-07-31T02:38:58.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:57 smithi154 conmon[34553]: 07 2021-07-31T02:38:58.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:57 smithi154 conmon[34553]: -31T02:38:56.541798+0000 mgr.x (mgr.24542) 2021-07-31T02:38:58.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:57 smithi154 conmon[34553]: 2216 : cluster [DBG] pgmap v3109: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 157 KiB/s rd, 3.7 KiB/s wr, 212 op/s 2021-07-31T02:38:58.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:57 smithi154 conmon[34553]: cluster 2021 2021-07-31T02:38:58.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:57 smithi154 conmon[34553]: -07-31T02:38:56.586711+0000 mon.a (mon.0) 2716 : cluster [DBG] osdmap e1712: 8 total, 8 up, 8 in 2021-07-31T02:38:58.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:57 smithi045 conmon[39718]: cluster 2021-07-31T02:38:56.541798+0000 mgr.x (mgr 2021-07-31T02:38:58.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:57 smithi045 conmon[39718]: .24542) 2216 : cluster [DBG] pgmap v3109: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 157 KiB/s rd, 3.7 KiB/s wr, 212 op/s 2021-07-31T02:38:58.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:57 smithi045 conmon[39718]: cluster 2021-07- 2021-07-31T02:38:58.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:57 smithi045 conmon[39718]: 31T02:38:56.586711+0000 mon.a (mon.0) 2716 : cluster [DBG] osdmap e1712: 8 total, 8 up, 8 in 2021-07-31T02:38:58.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:57 smithi045 conmon[33655]: cluster 2021-07-31T 2021-07-31T02:38:58.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:57 smithi045 conmon[33655]: 02:38:56.541798+0000 mgr.x (mgr.24542) 2216 : cluster [DBG] pgmap v3109: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 157 KiB/s rd, 3.7 KiB/s wr, 212 op/s 2021-07-31T02:38:58.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:57 smithi045 conmon[33655]: cluster 2021-07-31T02:38:56.586711+0000 2021-07-31T02:38:58.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:57 smithi045 conmon[33655]: mon.a (mon.0) 2716 : cluster [DBG] osdmap e1712: 8 total, 8 up, 8 in 2021-07-31T02:38:59.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:58 smithi154 conmon[34553]: cluster 2021-07-31T02:38:57.587782+0000 mon.a (mon.0) 2021-07-31T02:38:59.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:58 smithi154 conmon[34553]: 2717 : cluster [DBG] osdmap e1713: 8 total, 8 up, 8 in 2021-07-31T02:38:59.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:58 smithi045 conmon[33655]: cluster 2021-07-31T02:38:57.587782+0000 mon.a (mon.0 2021-07-31T02:38:59.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:58 smithi045 conmon[33655]: ) 2717 : cluster [DBG] osdmap e1713: 8 total, 8 up, 8 in 2021-07-31T02:38:59.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:58 smithi045 conmon[39718]: cluster 2021-07-31T02:38:57.587782+0000 mon.a ( 2021-07-31T02:38:59.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:58 smithi045 conmon[39718]: mon.0) 2717 : cluster [DBG] osdmap e1713: 8 total, 8 up, 8 in 2021-07-31T02:39:00.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:59 smithi154 conmon[34553]: cluster 2021-07-31T02:38:58.542239+0000 mgr.x (mgr.24542) 2217 : cluster [DBG] pgmap v3112: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:39:00.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:38:59 smithi154 conmon[34553]: cluster 2021-07-31T02:38:58.598053+0000 mon.a (mon.0) 2718 : cluster [DBG] osdmap e1714: 8 total, 8 up, 8 in 2021-07-31T02:39:00.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:59 smithi045 conmon[39718]: cluster 2021-07-31T02:38:58.542239+0000 mgr.x ( 2021-07-31T02:39:00.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:59 smithi045 conmon[39718]: mgr.24542) 2217 : cluster [DBG] pgmap v3112: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:39:00.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:59 smithi045 conmon[39718]: cluster 2021-07-31T02:39:00.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:59 smithi045 conmon[39718]: 2021-07-31T02:38:58.598053+0000 2021-07-31T02:39:00.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:59 smithi045 conmon[39718]: mon.a (mon.0) 2718 : cluster [DBG] 2021-07-31T02:39:00.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:38:59 smithi045 conmon[39718]: osdmap e1714: 8 total, 8 up, 8 in 2021-07-31T02:39:00.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:59 smithi045 conmon[33655]: cluster 2021-07-31T02:38:58.542239+0000 mgr.x (mgr.24542 2021-07-31T02:39:00.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:59 smithi045 conmon[33655]: ) 2217 : cluster [DBG] pgmap v3112: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:39:00.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:38:59 smithi045 conmon[33655]: cluster 2021-07-31T02:38:58.598053+0000 mon.a (mon.0) 2718 : cluster [DBG] osdmap e1714: 8 total, 8 up, 8 in 2021-07-31T02:39:00.289 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:39:00.567 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:39:00.567 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 53m ago 54m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:39:00.567 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 51m ago 54m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:39:00.568 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 51m ago 57m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:39:00.568 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 53m ago 60m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:39:00.568 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 53m ago 60m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:39:00.568 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 51m ago 58m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:39:00.568 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 53m ago 58m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:39:00.569 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 53m ago 54m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:39:00.569 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 51m ago 54m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:39:00.569 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 53m ago 57m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:39:00.569 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 53m ago 57m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:39:00.570 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 53m ago 56m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:39:00.570 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 53m ago 56m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:39:00.570 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 51m ago 56m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:39:00.570 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 51m ago 55m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:39:00.570 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 51m ago 55m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:39:00.571 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 51m ago 55m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:39:00.571 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 51m ago 54m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:39:00.912 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:39:00.912 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:39:00.912 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:39:00.913 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:39:00.913 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:39:00.913 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:39:00.913 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:39:00.913 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:39:00.914 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:39:00.914 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:39:00.914 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:39:00.914 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:39:00.915 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:39:00.915 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:39:00.915 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:39:00.915 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:39:00.915 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:39:01.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:00 smithi154 conmon[34553]: cluster 2021-07-31T02:38:59.603612+0000 mon.a (mon.0) 2719 : cluster 2021-07-31T02:39:01.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:00 smithi154 conmon[34553]: [DBG] osdmap e1715: 8 total, 8 up, 8 in 2021-07-31T02:39:01.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:00 smithi154 conmon[34553]: audit 2021-07-31T02:39:00.277707+0000 mgr.x (mgr.24542) 2218 : audit [DBG] from='client.27712 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:39:01.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:00 smithi045 conmon[39718]: cluster 2021-07-31T02:38 2021-07-31T02:39:01.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:00 smithi045 conmon[39718]: :59.603612+0000 mon.a (mon.0) 2719 : cluster [DBG] osdmap e1715: 8 total, 8 up, 8 in 2021-07-31T02:39:01.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:00 smithi045 conmon[39718]: audit 2021- 2021-07-31T02:39:01.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:00 smithi045 conmon[39718]: 07-31T02:39:00.277707+0000 mgr.x ( 2021-07-31T02:39:01.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:00 smithi045 conmon[39718]: mgr.24542) 2218 : audit [DBG] from='client.27712 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:39:01.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:00 smithi045 conmon[33655]: cluster 2021-07-31T02:38:59.603612+0000 mon.a (mon. 2021-07-31T02:39:01.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:00 smithi045 conmon[33655]: 0) 2719 : cluster [DBG] osdmap e1715: 8 total, 8 up, 8 in 2021-07-31T02:39:01.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:00 smithi045 conmon[33655]: audit 2021-07-31T02:39:00.277707+0000 mgr.x (mgr.24542) 2218 : audit [DBG] from='client.27712 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:39:02.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:01 smithi154 conmon[34553]: cluster 2021-07-31T02:39:00.542881+0000 mgr.x (mgr.24542) 2219 : cluster [DBG] pgmap v3115: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 191 KiB/s rd, 3.7 KiB/s wr, 256 op/s 2021-07-31T02:39:02.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:01 smithi154 conmon[34553]: audit 2021-07-31T02:39 2021-07-31T02:39:02.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:01 smithi154 conmon[34553]: :00.558734+0000 mgr.x (mgr.24542) 2220 : audit [DBG] from='client.17778 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:39:02.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:01 smithi154 conmon[34553]: cluster 2021-07-31T02:39:00.609145+0000 mon.a (mon.0) 2720 : cluster [DBG] osdmap e1716: 8 total, 8 up, 8 in 2021-07-31T02:39:02.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:01 smithi154 conmon[34553]: audit 2021-07-31T02:39:00.908087+0000 2021-07-31T02:39:02.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:01 smithi154 conmon[34553]: mon.a (mon.0) 2721 : audit [DBG] from='client.? 172.21.15.45:0/3174766608' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:39:02.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:01 smithi154 conmon[34553]: 2021-07-31T02:39:02.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:01 smithi045 conmon[33655]: cluster 2021-07-31T02:39:00.542881+0000 mgr.x (mgr.24542) 2219 : cluster [DBG] pgmap v3115: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 191 KiB/s rd, 3.7 KiB/s wr, 256 op/s 2021-07-31T02:39:02.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:01 smithi045 conmon[33655]: audit 2021-07-31T02:39:00.558734+0000 mgr.x (mgr.24542) 2220 : audit [DBG] from='client.17778 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:39:02.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:01 smithi045 conmon[33655]: cluster 2021-07-31T02:39:00.609145+0000 mon.a (mon.0) 2720 : cluster [DBG] osdmap e1716: 8 total, 8 up, 8 in 2021-07-31T02:39:02.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:01 smithi045 conmon[33655]: audit 2021-07- 2021-07-31T02:39:02.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:01 smithi045 conmon[33655]: 31T02:39:00.908087+0000 mon.a (mon.0) 2721 : audit [DBG] from='client.? 172.21.15.45:0/3174766608' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:39:02.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:01 smithi045 conmon[39718]: cluster 2021-07-31T02:39:00. 2021-07-31T02:39:02.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:01 smithi045 conmon[39718]: 542881+0000 mgr.x (mgr.24542) 2219 : cluster [DBG] pgmap v3115: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 191 KiB/s rd, 3.7 KiB/s wr, 256 op/s 2021-07-31T02:39:02.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:01 smithi045 conmon[39718]: audit 2021-07-31T02:39:02.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:01 smithi045 conmon[39718]: 2021-07-31T02:39:00.558734+0000 mgr.x (mgr.24542) 2220 : audit [DBG] 2021-07-31T02:39:02.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:01 smithi045 conmon[39718]: from='client.17778 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:39:02.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:01 smithi045 conmon[39718]: cluster 2021-07-31T02:39 2021-07-31T02:39:02.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:01 smithi045 conmon[39718]: :00.609145+0000 mon.a (mon.0) 2720 : 2021-07-31T02:39:02.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:01 smithi045 conmon[39718]: cluster [DBG] osdmap e1716: 8 total, 8 up, 8 in 2021-07-31T02:39:02.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:01 smithi045 conmon[39718]: audit 2021- 2021-07-31T02:39:02.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:01 smithi045 conmon[39718]: 07-31T02:39:00.908087+0000 mon.a ( 2021-07-31T02:39:02.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:01 smithi045 conmon[39718]: mon.0) 2721 : audit [DBG] from='client.? 172.21.15.45:0/3174766608' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:39:03.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:02 smithi154 conmon[34553]: cluster 2021-07-31T02:39:01.616511+0000 mon.a (mon 2021-07-31T02:39:03.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:02 smithi154 conmon[34553]: .0) 2722 : cluster [DBG] osdmap e1717: 8 total, 8 up, 8 in 2021-07-31T02:39:03.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:02 smithi045 conmon[39718]: cluster 2021-07-31T02:39:01.616511+0000 mon.a (mon.0) 2021-07-31T02:39:03.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:02 smithi045 conmon[39718]: 2722 : cluster [DBG] osdmap e1717: 8 total, 8 up, 8 in 2021-07-31T02:39:03.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:02 smithi045 conmon[33655]: cluster 2021-07-31T02:39:01.616511+0000 mon.a (mon.0) 2722 : cluster 2021-07-31T02:39:03.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:02 smithi045 conmon[33655]: [DBG] osdmap e1717: 8 total, 8 up, 8 in 2021-07-31T02:39:04.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:03 smithi154 conmon[34553]: cluster 2021-07-31T02:39:04.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:03 smithi154 conmon[34553]: 2021-07-31T02:39:02.543285 2021-07-31T02:39:04.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:03 smithi154 conmon[34553]: +0000 mgr.x (mgr.24542) 2221 : cluster [DBG] pgmap v3118: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 191 KiB/s rd, 3.7 KiB/s wr, 256 op/s 2021-07-31T02:39:04.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:03 smithi154 conmon[34553]: cluster 2021-07-31T02:39:02.617188+0000 mon.a (mon.0) 2021-07-31T02:39:04.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:03 smithi154 conmon[34553]: 2723 : cluster [DBG] osdmap e1718: 8 total, 8 up, 8 in 2021-07-31T02:39:04.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:03 smithi045 conmon[39718]: cluster 2021-07-31T02:39:02.543285+0000 mgr.x (mgr.24542 2021-07-31T02:39:04.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:03 smithi045 conmon[39718]: ) 2221 : cluster [DBG] pgmap v3118: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 191 KiB/s rd, 3.7 KiB/s wr, 256 op/s 2021-07-31T02:39:04.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:03 smithi045 conmon[39718]: cluster 2021-07-31T02:39:02.617188+0000 mon.a (mon 2021-07-31T02:39:04.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:03 smithi045 conmon[39718]: .0) 2723 : cluster [DBG] osdmap e1718: 8 total, 8 up, 8 in 2021-07-31T02:39:04.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:03 smithi045 conmon[33655]: cluster 2021-07-31T02:39:02.543285 2021-07-31T02:39:04.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:03 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 2221 : cluster [DBG] pgmap v3118: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 191 KiB/s rd, 3.7 KiB/s wr, 256 op/s 2021-07-31T02:39:04.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:03 smithi045 conmon[33655]: cluster 2021-07-31T02 2021-07-31T02:39:04.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:03 smithi045 conmon[33655]: :39:02.617188+0000 mon.a (mon.0) 2723 : 2021-07-31T02:39:04.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:03 smithi045 conmon[33655]: cluster [DBG] osdmap e1718: 8 total, 8 up, 8 in 2021-07-31T02:39:05.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:04 smithi154 conmon[34553]: cluster 2021-07-31T02:39:03.626962+0000 mon.a (mon.0) 2724 : cluster [DBG] osdmap e1719: 8 total, 8 up, 8 in 2021-07-31T02:39:05.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:04 smithi154 conmon[34553]: 2021-07-31T02:39:05.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:04 smithi045 conmon[33655]: cluster 2021-07-31T02:39:03.626962+0000 mon.a (mon.0) 2724 : cluster [DBG] osdmap e1719: 8 total, 8 up, 8 in 2021-07-31T02:39:05.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:04 smithi045 conmon[33655]: 2021-07-31T02:39:05.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:04 smithi045 conmon[39718]: cluster 2021-07-31T02:39:03.626962 2021-07-31T02:39:05.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:04 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2724 : cluster [DBG] osdmap e1719: 8 total, 8 up, 8 in 2021-07-31T02:39:06.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:05 smithi154 conmon[34553]: cluster 2021-07-31T02:39:04.543795+0000 mgr.x (mgr. 2021-07-31T02:39:06.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:05 smithi154 conmon[34553]: 24542) 2222 : cluster [DBG] pgmap v3121: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 222 KiB/s rd, 3.7 KiB/s wr, 299 op/s 2021-07-31T02:39:06.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:05 smithi154 conmon[34553]: cluster 2021-07-31T02:39:04.642746+0000 mon.a (mon.0) 2725 : cluster [DBG] osdmap e1720: 8 total, 8 up, 8 in 2021-07-31T02:39:06.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:05 smithi045 conmon[39718]: cluster 2021-07-31T02:39:04.543795 2021-07-31T02:39:06.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:05 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 2222 : cluster [DBG] pgmap v3121: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 222 KiB/s rd, 3.7 KiB/s wr, 299 op/s 2021-07-31T02:39:06.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:05 smithi045 conmon[39718]: cluster 2021-07-31T02:39:06.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:05 smithi045 conmon[39718]: 2021-07-31T02:39:04.642746+0000 mon.a (mon.0) 2725 : cluster [DBG] 2021-07-31T02:39:06.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:05 smithi045 conmon[39718]: osdmap e1720: 8 total, 8 up, 8 in 2021-07-31T02:39:06.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:05 smithi045 conmon[33655]: cluster 2021-07-31T02:39:04.543795+0000 mgr.x (mgr.24542) 2021-07-31T02:39:06.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:05 smithi045 conmon[33655]: 2222 : cluster [DBG] pgmap v3121: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 222 KiB/s rd, 3.7 KiB/s wr, 299 op/s 2021-07-31T02:39:06.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:05 smithi045 conmon[33655]: cluster 2021-07-31T02:39:04.642746+0000 mon.a (mon.0) 2725 : cluster [DBG] osdmap e1720: 8 total, 8 up, 8 in 2021-07-31T02:39:07.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:06 smithi154 conmon[34553]: cluster 2021-07-31T02:39:05.643598+0000 mon.a (mon.0) 2726 : cluster [DBG] osdmap e1721: 8 total, 8 up, 8 in 2021-07-31T02:39:07.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:06 smithi045 conmon[39718]: cluster 2021-07-31T02:39:05.643598+0000 mon.a 2021-07-31T02:39:07.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:06 smithi045 conmon[39718]: (mon.0) 2726 : cluster [DBG] osdmap e1721: 8 total, 8 up, 8 in 2021-07-31T02:39:07.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:06 smithi045 conmon[33655]: cluster 2021-07-31T02:39:05.643598+0000 mon.a 2021-07-31T02:39:07.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:06 smithi045 conmon[33655]: (mon.0) 2726 : cluster [DBG] osdmap e1721: 8 total, 8 up, 8 in 2021-07-31T02:39:08.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:07 smithi154 conmon[34553]: cluster 2021-07-31T02:39:06.544390+0000 mgr.x (mgr.24542) 2223 : cluster [DBG] pgmap v3124: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 222 KiB/s rd, 3.7 KiB/s wr, 299 op/s 2021-07-31T02:39:08.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:07 smithi154 conmon[34553]: cluster 2021-07-31T02:39:06.653571+0000 mon.a (mon.0) 2727 : cluster [DBG] osdmap e1722: 8 total, 8 up, 8 in 2021-07-31T02:39:08.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:07 smithi045 conmon[39718]: cluster 2021-07-31T02:39:06.544390+0000 mgr.x (mgr.24542) 2223 : cluster [DBG] pgmap v3124: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 222 KiB/s rd, 3.7 KiB/s wr, 299 op/s 2021-07-31T02:39:08.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:07 smithi045 conmon[39718]: cluster 2021-07-31T02:39:06.653571+0000 mon.a (mon.0) 2727 : cluster [DBG] osdmap e1722: 8 total, 8 up, 8 in 2021-07-31T02:39:08.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:07 smithi045 conmon[33655]: cluster 2021-07-31T02:39:06.544390+0000 mgr.x (mgr.24542) 2223 : cluster [DBG] pgmap v3124: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 222 KiB/s rd, 3.7 KiB/s wr, 299 op/s 2021-07-31T02:39:08.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:07 smithi045 conmon[33655]: cluster 2021-07-31T02:39:06.653571+0000 mon.a (mon.0) 2727 : cluster [DBG] osdmap e1722: 8 total, 8 up, 8 in 2021-07-31T02:39:09.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:08 smithi154 conmon[34553]: cluster 2021-07-31T02:39:07.655517+0000 mon.a (mon.0) 2728 2021-07-31T02:39:09.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:08 smithi154 conmon[34553]: : cluster [DBG] osdmap e1723: 8 total, 8 up, 8 in 2021-07-31T02:39:09.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:08 smithi045 conmon[39718]: cluster 2021-07-31T02:39:07.655517+0000 mon.a ( 2021-07-31T02:39:09.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:08 smithi045 conmon[39718]: mon.0) 2728 : cluster [DBG] osdmap e1723: 8 total, 8 up, 8 in 2021-07-31T02:39:09.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:08 smithi045 conmon[33655]: cluster 2021-07-31T02:39:07.655517+0000 mon.a (mon.0) 2021-07-31T02:39:09.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:08 smithi045 conmon[33655]: 2728 : cluster [DBG] osdmap e1723: 8 total, 8 up, 8 in 2021-07-31T02:39:10.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:09 smithi154 conmon[34553]: cluster 2021-07-31T02:39:08.544905+0000 mgr.x (mgr.24542) 2224 : cluster 2021-07-31T02:39:10.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:09 smithi154 conmon[34553]: [DBG] pgmap v3127: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:39:10.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:09 smithi154 conmon[34553]: cluster 2021-07-31T02:39:08.665502+0000 mon.a (mon.0) 2729 : cluster [DBG] osdmap e1724: 8 total, 8 up, 8 in 2021-07-31T02:39:10.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:09 smithi045 conmon[39718]: cluster 2021-07-31T02:39:08.544905+0000 mgr.x 2021-07-31T02:39:10.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:09 smithi045 conmon[39718]: (mgr.24542) 2224 : cluster [DBG] pgmap v3127: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:39:10.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:09 smithi045 conmon[39718]: cluster 2021-07-31T02:39:08.665502+0000 mon.a (mon.0) 2729 : cluster [DBG] osdmap e1724: 8 total, 8 up, 8 in 2021-07-31T02:39:10.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:09 smithi045 conmon[33655]: cluster 2021-07-31T02:39:08.544905+0000 mgr.x (mgr.24542) 2224 : cluster [DBG] pgmap v3127: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:39:10.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:09 smithi045 conmon[33655]: cluster 2021-07-31T02:39:10.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:09 smithi045 conmon[33655]: 2021-07-31T02:39:08.665502+0000 mon.a (mon.0) 2729 : cluster [DBG] osdmap e1724: 8 total, 8 up, 8 in 2021-07-31T02:39:11.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:10 smithi154 conmon[34553]: cluster 2021-07-31T02:39:09.667010+0000 mon.a (mon.0) 2730 : cluster 2021-07-31T02:39:11.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:10 smithi154 conmon[34553]: [DBG] osdmap e1725: 8 total, 8 up, 8 in 2021-07-31T02:39:11.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:10 smithi045 conmon[33655]: cluster 2021-07-31T02:39:09.667010+0000 mon.a (mon.0) 2730 : cluster [DBG] osdmap e1725: 8 total, 8 up, 8 in 2021-07-31T02:39:11.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:10 smithi045 conmon[39718]: cluster 2021-07-31T02:39:09. 2021-07-31T02:39:11.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:10 smithi045 conmon[39718]: 667010+0000 mon.a (mon.0) 2730 : cluster [DBG] osdmap e1725: 8 total, 8 up, 8 in 2021-07-31T02:39:12.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:11 smithi154 conmon[34553]: cluster 2021-07-31T02:39:10.545339+0000 mgr.x (mgr.24542) 2225 : cluster [DBG] 2021-07-31T02:39:12.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:11 smithi154 conmon[34553]: pgmap v3130: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 256 KiB/s rd, 3.7 KiB/s wr, 343 op/s 2021-07-31T02:39:12.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:11 smithi154 conmon[34553]: cluster 2021-07-31T02:39:10.677264+0000 mon.a (mon.0) 2731 : cluster [DBG] osdmap e1726: 8 total, 8 up, 8 in 2021-07-31T02:39:12.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:11 smithi045 conmon[39718]: cluster 2021-07-31T02:39:10.545339+0000 mgr.x (mgr.24542) 2021-07-31T02:39:12.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:11 smithi045 conmon[39718]: 2225 : cluster [DBG] pgmap v3130: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 256 KiB/s rd, 3.7 KiB/s wr, 343 op/s 2021-07-31T02:39:12.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:11 smithi045 conmon[39718]: cluster 2021-07-31T02:39:10.677264+0000 mon.a (mon.0 2021-07-31T02:39:12.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:11 smithi045 conmon[39718]: ) 2731 : cluster [DBG] osdmap e1726: 8 total, 8 up, 8 in 2021-07-31T02:39:12.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:11 smithi045 conmon[33655]: cluster 2021-07-31T02:39:10.545339+0000 mgr.x (mgr.24542) 2225 : cluster [DBG] pgmap v3130: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 256 KiB/s rd, 3.7 KiB/s wr, 343 op/s 2021-07-31T02:39:12.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:11 smithi045 conmon[33655]: cluster 2021-07-31T02:39:10.677264+0000 mon.a (mon.0) 2731 : cluster [DBG] osdmap e1726: 8 total, 8 up, 8 in 2021-07-31T02:39:13.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:12 smithi154 conmon[34553]: cluster 2021-07-31T02:39:11.680255+0000 mon.a (mon.0) 2732 : cluster [DBG] 2021-07-31T02:39:13.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:12 smithi154 conmon[34553]: osdmap e1727: 8 total, 8 up, 8 in 2021-07-31T02:39:13.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:12 smithi045 conmon[39718]: cluster 2021-07-31T02:39:11.680255+0000 mon.a (mon.0) 2732 : cluster 2021-07-31T02:39:13.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:12 smithi045 conmon[39718]: [DBG] osdmap e1727: 8 total, 8 up, 8 in 2021-07-31T02:39:13.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:12 smithi045 conmon[33655]: cluster 2021-07-31T02:39:11.680255+0000 mon.a (mon.0) 2732 : cluster [DBG] 2021-07-31T02:39:13.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:12 smithi045 conmon[33655]: osdmap e1727: 8 total, 8 up, 8 in 2021-07-31T02:39:14.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:13 smithi154 conmon[34553]: cluster 2021-07-31T02:39:12.545715+0000 mgr.x (mgr.24542) 2226 : cluster [DBG] pgmap v3133: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 256 KiB/s rd, 3.7 KiB/s wr, 343 op/s 2021-07-31T02:39:14.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:13 smithi154 conmon[34553]: cluster 2021-07-31T02:39:12.686461+0000 mon.a (mon.0) 2733 : cluster [DBG] osdmap e1728: 8 total, 8 up, 8 in 2021-07-31T02:39:14.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:13 smithi045 conmon[39718]: cluster 2021-07-31T02:39:12.545715+0000 mgr.x (mgr.24542) 2226 2021-07-31T02:39:14.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:13 smithi045 conmon[39718]: : cluster [DBG] pgmap v3133: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 256 KiB/s rd, 3.7 KiB/s wr, 343 op/s 2021-07-31T02:39:14.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:13 smithi045 conmon[39718]: cluster 2021-07-31T02:39:12.686461+0000 mon.a (mon.0) 2733 : cluster [DBG] osdmap e1728: 8 total, 8 up, 8 in 2021-07-31T02:39:14.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:13 smithi045 conmon[39718]: 2021-07-31T02:39:14.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:13 smithi045 conmon[33655]: cluster 2021-07-31T02:39:12.545715+0000 mgr.x (mgr.24542) 2226 : cluster [DBG] pgmap v3133: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 256 KiB/s rd, 3.7 KiB/s wr, 343 op/s 2021-07-31T02:39:14.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:13 smithi045 conmon[33655]: cluster 2021-07-31T02:39:12.686461+0000 mon.a (mon.0) 2733 : cluster [DBG] osdmap e1728: 8 total, 8 up, 8 in 2021-07-31T02:39:15.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:14 smithi154 conmon[34553]: cluster 2021-07-31T02:39:13.696921+0000 mon.a (mon.0) 2734 : cluster [DBG] osdmap e1729: 8 total, 8 up, 8 in 2021-07-31T02:39:15.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:14 smithi154 conmon[34553]: 2021-07-31T02:39:15.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:14 smithi045 conmon[33655]: cluster 2021-07-31T02:39:13.696921+0000 mon.a (mon.0) 2734 2021-07-31T02:39:15.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:14 smithi045 conmon[33655]: : cluster [DBG] osdmap e1729: 8 total, 8 up, 8 in 2021-07-31T02:39:15.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:14 smithi045 conmon[39718]: cluster 2021-07-31T02:39:13.696921+0000 mon.a (mon. 2021-07-31T02:39:15.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:14 smithi045 conmon[39718]: 0) 2734 : cluster [DBG] osdmap e1729: 8 total, 8 up, 8 in 2021-07-31T02:39:16.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:15 smithi154 conmon[34553]: cluster 2021-07-31T02:39:14.546253+0000 mgr.x (mgr.24542) 2227 : cluster [DBG] pgmap v3136: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 289 KiB/s rd, 3.7 KiB/s wr, 387 op/s 2021-07-31T02:39:16.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:15 smithi154 conmon[34553]: cluster 2021-07-31T02:39:14.702916+0000 mon.a (mon.0) 2735 : cluster [DBG] osdmap e1730: 8 total, 8 up, 8 in 2021-07-31T02:39:16.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:15 smithi045 conmon[33655]: cluster 2021-07-31T02:39:14.546253+0000 mgr.x (mgr.24542) 2227 : 2021-07-31T02:39:16.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:15 smithi045 conmon[33655]: cluster [DBG] pgmap v3136: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 289 KiB/s rd, 3.7 KiB/s wr, 387 op/s 2021-07-31T02:39:16.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:15 smithi045 conmon[33655]: cluster 2021-07-31T02:39:14.702916+0000 mon.a (mon.0) 2735 : cluster [DBG] osdmap e1730: 8 total, 8 up, 8 in 2021-07-31T02:39:16.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:15 smithi045 conmon[39718]: cluster 2021-07-31T02:39:14.546253+0000 mgr.x (mgr.24542) 2227 : cluster [DBG] pgmap v3136: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 289 KiB/s rd, 3.7 KiB/s wr, 387 op/s 2021-07-31T02:39:16.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:15 smithi045 conmon[39718]: cluster 2021-07-31T02:39:14.702916+0000 mon.a (mon.0) 2735 : cluster [DBG] osdmap e1730: 8 total, 8 up, 8 in 2021-07-31T02:39:17.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:16 smithi154 conmon[34553]: cluster 2021-07-31T02:39:15.704575+0000 mon.a (mon.0) 2736 : cluster [DBG] osdmap e1731: 8 total, 8 up, 8 in 2021-07-31T02:39:17.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:16 smithi045 conmon[39718]: cluster 2021-07-31T02:39:15.704575+0000 2021-07-31T02:39:17.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:16 smithi045 conmon[39718]: mon.a (mon.0) 2736 : cluster [DBG] osdmap e1731: 8 total, 8 up, 8 in 2021-07-31T02:39:17.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:16 smithi045 conmon[33655]: cluster 2021-07-31T02:39:15.704575+0000 mon.a (mon.0) 2736 : cluster [DBG] osdmap e1731: 8 total, 8 up, 8 in 2021-07-31T02:39:18.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:17 smithi154 conmon[34553]: cluster 2021-07-31T02:39:16.546730+0000 mgr.x (mgr.24542) 2228 : cluster [DBG] pgmap v3139: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 289 KiB/s rd, 3.7 KiB/s wr, 387 op/s 2021-07-31T02:39:18.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:17 smithi154 conmon[34553]: cluster 2021-07-31T02:39:18.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:17 smithi154 conmon[34553]: 2021-07-31T02:39:16.715016+0000 mon.a (mon.0) 2737 : cluster [DBG] osdmap e1732: 8 total, 8 up, 8 in 2021-07-31T02:39:18.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:17 smithi045 conmon[33655]: cluster 2021-07-31T02:39:16.546730+0000 mgr.x (mgr.24542) 2228 : cluster 2021-07-31T02:39:18.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:17 smithi045 conmon[33655]: [DBG] pgmap v3139: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 289 KiB/s rd, 3.7 KiB/s wr, 387 op/s 2021-07-31T02:39:18.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:17 smithi045 conmon[33655]: cluster 2021-07-31T02:39:16.715016+0000 mon.a (mon.0) 2737 : cluster [DBG] osdmap e1732: 8 total, 8 up, 8 in 2021-07-31T02:39:18.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:17 smithi045 conmon[39718]: cluster 2021-07-31T02:39:16.546730+0000 mgr.x (mgr. 2021-07-31T02:39:18.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:17 smithi045 conmon[39718]: 24542) 2228 : cluster [DBG] pgmap v3139: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 289 KiB/s rd, 3.7 KiB/s wr, 387 op/s 2021-07-31T02:39:18.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:17 smithi045 conmon[39718]: cluster 2021-07-31T02:39:16.715016+0000 mon.a (mon 2021-07-31T02:39:18.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:17 smithi045 conmon[39718]: .0) 2737 : cluster [DBG] osdmap e1732: 8 total, 8 up, 8 in 2021-07-31T02:39:19.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:18 smithi154 conmon[34553]: cluster 2021-07-31T02:39:17.720246+0000 mon.a (mon.0) 2738 : cluster 2021-07-31T02:39:19.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:18 smithi154 conmon[34553]: [DBG] osdmap e1733: 8 total, 8 up, 8 in 2021-07-31T02:39:19.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:18 smithi045 conmon[33655]: cluster 2021-07-31T02:39:17.720246+0000 2021-07-31T02:39:19.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:18 smithi045 conmon[33655]: mon.a (mon.0) 2738 : cluster [DBG] osdmap e1733: 8 total, 8 up, 8 in 2021-07-31T02:39:19.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:18 smithi045 conmon[39718]: cluster 2021-07-31T02:39:17.720246 2021-07-31T02:39:19.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:18 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2738 : cluster [DBG] osdmap e1733: 8 total, 8 up, 8 in 2021-07-31T02:39:20.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:19 smithi154 conmon[34553]: cluster 2021-07-31T02:39:18.547211+0000 mgr.x (mgr.24542) 2021-07-31T02:39:20.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:19 smithi154 conmon[34553]: 2229 : cluster [DBG] pgmap v3142: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:39:20.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:19 smithi154 conmon[34553]: cluster 2021-07-31T02:39:18.725812+0000 mon.a (mon.0) 2739 : cluster [DBG] osdmap e1734: 8 total, 8 up, 8 in 2021-07-31T02:39:20.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:19 smithi045 conmon[39718]: cluster 2021-07-31T02:39:18. 2021-07-31T02:39:20.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:19 smithi045 conmon[39718]: 547211+0000 mgr.x (mgr.24542) 2229 : cluster [DBG] pgmap v3142: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:39:20.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:19 smithi045 conmon[39718]: cluster 2021-07-31T02:39:20.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:19 smithi045 conmon[39718]: 2021-07-31 2021-07-31T02:39:20.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:19 smithi045 conmon[39718]: T02:39:18.725812+0000 2021-07-31T02:39:20.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:19 smithi045 conmon[39718]: mon.a (mon.0) 2739 2021-07-31T02:39:20.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:19 smithi045 conmon[39718]: : cluster [DBG] osdmap e1734: 8 total, 8 up, 8 in 2021-07-31T02:39:20.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:19 smithi045 conmon[33655]: cluster 2021-07-31T02:39:18 2021-07-31T02:39:20.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:19 smithi045 conmon[33655]: .547211+0000 mgr.x (mgr.24542) 2229 : cluster [DBG] pgmap v3142: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:39:20.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:19 smithi045 conmon[33655]: cluster 2021-07-31T02:39:20.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:19 smithi045 conmon[33655]: 2021-07-31T02:39 2021-07-31T02:39:20.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:19 smithi045 conmon[33655]: :18.725812+0000 mon.a (mon.0) 2739 : cluster [DBG] osdmap e1734: 8 total, 8 up, 8 in 2021-07-31T02:39:21.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:20 smithi154 conmon[34553]: cluster 2021-07-31T02:39:19.727450+0000 mon.a (mon.0) 2740 : cluster 2021-07-31T02:39:21.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:20 smithi154 conmon[34553]: [DBG] osdmap e1735: 8 total, 8 up, 8 in 2021-07-31T02:39:21.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:20 smithi045 conmon[33655]: cluster 2021-07-31T02:39:19.727450+0000 mon.a (mon.0) 2740 : cluster [DBG] osdmap e1735: 8 total, 8 up, 8 in 2021-07-31T02:39:21.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:20 smithi045 conmon[39718]: cluster 2021-07-31T02:39:19.727450+0000 mon.a (mon.0) 2740 : cluster 2021-07-31T02:39:21.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:20 smithi045 conmon[39718]: [DBG] osdmap e1735: 8 total, 8 up, 8 in 2021-07-31T02:39:22.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:21 smithi154 conmon[34553]: cluster 2021-07-31T02:39:20.547750+0000 mgr.x (mgr.24542) 2230 : cluster [DBG] pgmap v3145: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 321 KiB/s rd, 3.7 KiB/s wr, 431 op/s 2021-07-31T02:39:22.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:21 smithi154 conmon[34553]: cluster 2021-07-31T02:39:20.738244+0000 mon.a (mon.0) 2741 : cluster [DBG] osdmap e1736: 8 total, 8 up, 8 in 2021-07-31T02:39:22.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:21 smithi045 conmon[39718]: cluster 2021-07-31T02:39:20.547750+0000 mgr.x (mgr.24542) 2230 : cluster [DBG] pgmap v3145: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 321 KiB/s rd, 3.7 KiB/s wr, 431 op/s 2021-07-31T02:39:22.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:21 smithi045 conmon[39718]: cluster 2021-07-31T02:39:20.738244+0000 mon.a (mon.0) 2741 : cluster [DBG] osdmap e1736: 8 total, 8 up, 8 in 2021-07-31T02:39:22.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:21 smithi045 conmon[33655]: cluster 2021-07-31T02:39:20.547750+0000 mgr.x (mgr.24542) 2021-07-31T02:39:22.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:21 smithi045 conmon[33655]: 2230 : cluster [DBG] pgmap v3145: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 321 KiB/s rd, 3.7 KiB/s wr, 431 op/s 2021-07-31T02:39:22.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:21 smithi045 conmon[33655]: cluster 2021-07-31T02:39:20.738244+0000 mon.a (mon.0) 2741 : cluster [DBG] osdmap e1736: 8 total, 8 up, 8 in 2021-07-31T02:39:22.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:21 smithi045 conmon[33655]: 2021-07-31T02:39:23.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:22 smithi154 conmon[34553]: cluster 2021-07-31T02:39:21.744472+0000 mon.a (mon.0) 2742 : cluster [DBG] osdmap e1737: 8 total, 8 up, 8 in 2021-07-31T02:39:23.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:22 smithi045 conmon[39718]: cluster 2021-07-31T02:39:21.744472 2021-07-31T02:39:23.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:22 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2742 : cluster [DBG] osdmap e1737: 8 total, 8 up, 8 in 2021-07-31T02:39:23.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:22 smithi045 conmon[33655]: cluster 2021-07-31T02:39:21.744472+0000 mon.a (mon.0) 2742 : cluster [DBG] osdmap e1737: 8 total, 8 up, 8 in 2021-07-31T02:39:24.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:23 smithi154 conmon[34553]: cluster 2021-07-31T02:39:22.548163+0000 mgr.x (mgr.24542) 2231 : cluster [DBG] pgmap v3148: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 321 KiB/s rd, 3.7 KiB/s wr, 431 op/s 2021-07-31T02:39:24.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:23 smithi154 conmon[34553]: cluster 2021-07-31T02:39:22.750530+0000 mon.a (mon.0) 2743 : cluster [DBG] osdmap e1738: 8 total, 8 up, 8 in 2021-07-31T02:39:24.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:23 smithi045 conmon[39718]: cluster 2021-07-31T02:39:22 2021-07-31T02:39:24.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:23 smithi045 conmon[39718]: .548163+0000 mgr.x (mgr.24542) 2231 : cluster [DBG] pgmap v3148: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 321 KiB/s rd, 3.7 KiB/s wr, 431 op/s 2021-07-31T02:39:24.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:23 smithi045 conmon[39718]: cluster 2021-07-31T02:39:24.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:23 smithi045 conmon[39718]: 2021-07-31T02:39:22.750530+0000 mon.a (mon.0) 2743 : cluster [DBG] osdmap e1738: 8 total, 8 up, 8 in 2021-07-31T02:39:24.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:23 smithi045 conmon[33655]: cluster 2021-07-31T02:39:22.548163 2021-07-31T02:39:24.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:23 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 2231 : cluster [DBG] pgmap v3148: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 321 KiB/s rd, 3.7 KiB/s wr, 431 op/s 2021-07-31T02:39:24.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:23 smithi045 conmon[33655]: cluster 2021-07-31T02: 2021-07-31T02:39:24.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:23 smithi045 conmon[33655]: 39:22.750530+0000 mon.a (mon.0) 2743 : cluster [DBG] osdmap e1738: 8 total, 8 up, 8 in 2021-07-31T02:39:25.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:24 smithi045 conmon[39718]: cluster 2021-07-31T02:39:23.756698+0000 mon.a (mon.0) 2744 : cluster [DBG] osdmap e1739: 8 total, 8 up, 8 in 2021-07-31T02:39:25.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:24 smithi045 conmon[33655]: cluster 2021-07-31T02:39:23.756698+0000 2021-07-31T02:39:25.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:24 smithi045 conmon[33655]: mon.a (mon.0) 2744 : cluster [DBG] osdmap e1739: 8 total, 8 up, 8 in 2021-07-31T02:39:25.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:24 smithi154 conmon[34553]: cluster 2021-07-31T02:39:23.756698+0000 mon.a (mon.0) 2744 : cluster [DBG] osdmap e1739: 8 total, 8 up, 8 in 2021-07-31T02:39:26.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:25 smithi045 conmon[33655]: cluster 2021-07-31T02:39:24.548690+0000 mgr.x (mgr.24542) 2232 : cluster [DBG] 2021-07-31T02:39:26.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:25 smithi045 conmon[33655]: pgmap v3151: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 354 KiB/s rd, 3.7 KiB/s wr, 474 op/s 2021-07-31T02:39:26.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:25 smithi045 conmon[33655]: cluster 2021-07-31T02:39:24.768229+0000 mon.a (mon.0) 2745 : cluster [DBG] osdmap e1740: 8 total, 8 up, 8 in 2021-07-31T02:39:26.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:25 smithi045 conmon[39718]: cluster 2021-07-31T02:39:24.548690+0000 mgr.x 2021-07-31T02:39:26.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:25 smithi045 conmon[39718]: (mgr.24542) 2232 : cluster [DBG] pgmap v3151: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 354 KiB/s rd, 3.7 KiB/s wr, 474 op/s 2021-07-31T02:39:26.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:25 smithi045 conmon[39718]: cluster 2021-07-31T02:39: 2021-07-31T02:39:26.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:25 smithi045 conmon[39718]: 24.768229+0000 mon.a (mon.0) 2745 : cluster [DBG] osdmap e1740: 8 total, 8 up, 8 in 2021-07-31T02:39:26.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:25 smithi154 conmon[34553]: cluster 2021-07-31T02:39:24.548690+0000 mgr.x ( 2021-07-31T02:39:26.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:25 smithi154 conmon[34553]: mgr.24542) 2232 : cluster [DBG] pgmap v3151: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 354 KiB/s rd, 3.7 KiB/s wr, 474 op/s 2021-07-31T02:39:26.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:25 smithi154 conmon[34553]: cluster 2021-07-31T02:39:24.768229+0000 mon.a (mon.0) 2745 : cluster 2021-07-31T02:39:26.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:25 smithi154 conmon[34553]: [DBG] osdmap e1740: 8 total, 8 up, 8 in 2021-07-31T02:39:27.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:26 smithi154 conmon[34553]: cluster 2021-07-31T02:39:25.770065+0000 mon.a (mon.0) 2746 : cluster [DBG] osdmap e1741: 8 total, 8 up, 8 in 2021-07-31T02:39:27.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:26 smithi045 conmon[39718]: cluster 2021-07-31T02:39:27.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:26 smithi045 conmon[39718]: 2021-07-31T02:39:25. 2021-07-31T02:39:27.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:26 smithi045 conmon[39718]: 770065+0000 mon.a (mon.0) 2746 : cluster [DBG] osdmap e1741: 8 total, 8 up, 8 in 2021-07-31T02:39:27.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:26 smithi045 conmon[33655]: cluster 2021-07-31T02:39:25.770065+0000 mon.a (mon.0) 2746 : cluster 2021-07-31T02:39:27.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:26 smithi045 conmon[33655]: [DBG] osdmap e1741: 8 total, 8 up, 8 in 2021-07-31T02:39:28.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:27 smithi154 conmon[34553]: cluster 2021-07-31T02:39:26.549183+0000 mgr.x (mgr.24542) 2233 : cluster [DBG] 2021-07-31T02:39:28.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:27 smithi154 conmon[34553]: pgmap v3154: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 354 KiB/s rd, 3.7 KiB/s wr, 474 op/s 2021-07-31T02:39:28.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:27 smithi154 conmon[34553]: cluster 2021-07-31T02:39:26.782824+0000 mon.a (mon.0) 2747 : cluster [DBG] osdmap e1742: 8 total, 8 up, 8 in 2021-07-31T02:39:28.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:27 smithi045 conmon[39718]: cluster 2021-07-31T02:39:26.549183+0000 mgr.x 2021-07-31T02:39:28.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:27 smithi045 conmon[39718]: (mgr.24542) 2233 : cluster [DBG] pgmap v3154: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 354 KiB/s rd, 3.7 KiB/s wr, 474 op/s 2021-07-31T02:39:28.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:27 smithi045 conmon[39718]: cluster 2021-07-31 2021-07-31T02:39:28.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:27 smithi045 conmon[39718]: T02:39:26.782824+0000 mon.a (mon.0) 2747 : cluster 2021-07-31T02:39:28.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:27 smithi045 conmon[39718]: [DBG] osdmap e1742: 8 total, 8 up, 8 in 2021-07-31T02:39:28.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:27 smithi045 conmon[33655]: cluster 2021-07-31T02:39:26.549183+0000 mgr.x (mgr.24542) 2233 : cluster [DBG] 2021-07-31T02:39:28.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:27 smithi045 conmon[33655]: pgmap v3154: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 354 KiB/s rd, 3.7 KiB/s wr, 474 op/s 2021-07-31T02:39:28.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:27 smithi045 conmon[33655]: cluster 2021-07-31T02:39:26.782824+0000 mon.a (mon.0) 2747 : cluster [DBG] osdmap e1742: 8 total, 8 up, 8 in 2021-07-31T02:39:29.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:28 smithi154 conmon[34553]: cluster 2021-07-31T02:39:27.783578+0000 mon.a (mon.0) 2748 : cluster [DBG] osdmap e1743: 8 total, 8 up, 8 in 2021-07-31T02:39:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:28 smithi154 conmon[34553]: audit 2021-07-31T02:39:28.124935+0000 mon.b (mon.2) 335 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:39:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:28 smithi154 conmon[34553]: audit 2021-07-31T02:39:28.125548+0000 mon.a (mon.0) 2749 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:39:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:28 smithi154 conmon[34553]: audit 2021-07-31T02:39:28 2021-07-31T02:39:29.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:28 smithi154 conmon[34553]: .139900+0000 mon.b (mon.2) 336 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:39:29.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:28 smithi154 conmon[34553]: audit 2021-07-31 2021-07-31T02:39:29.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:28 smithi154 conmon[34553]: T02:39:28.140437+0000 mon.a (mon.0) 2750 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:39:29.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:28 smithi154 conmon[34553]: 2021-07-31T02:39:29.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:28 smithi045 conmon[39718]: cluster 2021-07-31T02:39:29.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:28 smithi045 conmon[39718]: 2021-07-31T02:39:27.783578+0000 mon.a (mon.0) 2748 : cluster [DBG] osdmap e1743: 8 total, 8 up, 8 in 2021-07-31T02:39:29.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:28 smithi045 conmon[39718]: audit 2021-07-31T02:39:29.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:28 smithi045 conmon[39718]: 2021-07-31T02:39:28.124935+0000 mon.b (mon.2) 335 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:39:29.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:28 smithi045 conmon[39718]: audit 2021-07 2021-07-31T02:39:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:28 smithi045 conmon[39718]: -31T02:39:28 2021-07-31T02:39:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:28 smithi045 conmon[39718]: .125548+0000 mon.a (mon.0) 2021-07-31T02:39:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:28 smithi045 conmon[39718]: 2749 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:39:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:28 smithi045 conmon[39718]: audit 2021-07-31T02:39:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:28 smithi045 conmon[39718]: 2021-07-31T02:39 2021-07-31T02:39:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:28 smithi045 conmon[39718]: :28.139900+0000 mon.b ( 2021-07-31T02:39:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:28 smithi045 conmon[39718]: mon.2) 336 : audit 2021-07-31T02:39:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:28 smithi045 conmon[39718]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:39:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:28 smithi045 conmon[39718]: audit 2021 2021-07-31T02:39:29.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:28 smithi045 conmon[39718]: -07-31T02:39 2021-07-31T02:39:29.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:28 smithi045 conmon[39718]: :28.140437+0000 mon.a (mon 2021-07-31T02:39:29.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:28 smithi045 conmon[39718]: .0) 2750 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:39:29.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:28 smithi045 conmon[33655]: cluster 2021-07-31T02:39:27.783578+0000 mon.a (mon.0) 2021-07-31T02:39:29.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:28 smithi045 conmon[33655]: 2748 : cluster [DBG] osdmap e1743: 8 total, 8 up, 8 in 2021-07-31T02:39:29.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:28 smithi045 conmon[33655]: audit 2021-07-31T02:39:28.124935+0000 mon.b (mon.2) 335 : audit 2021-07-31T02:39:29.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:28 smithi045 conmon[33655]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:39:29.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:28 smithi045 conmon[33655]: audit 2021-07-31T02:39:29.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:28 smithi045 conmon[33655]: 2021-07-31T02:39:28 2021-07-31T02:39:29.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:28 smithi045 conmon[33655]: .125548+0000 mon.a (mon.0 2021-07-31T02:39:29.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:28 smithi045 conmon[33655]: ) 2749 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:39:29.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:28 smithi045 conmon[33655]: audit 2021-07-31 2021-07-31T02:39:29.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:28 smithi045 conmon[33655]: T02:39:28.139900+0000 mon.b 2021-07-31T02:39:29.283 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:28 smithi045 conmon[33655]: (mon.2) 336 : audit 2021-07-31T02:39:29.283 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:28 smithi045 conmon[33655]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:39:29.283 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:28 smithi045 conmon[33655]: audit 2021 2021-07-31T02:39:29.283 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:28 smithi045 conmon[33655]: -07-31T02:39:28. 2021-07-31T02:39:29.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:28 smithi045 conmon[33655]: 140437+0000 mon.a (mon.0 2021-07-31T02:39:29.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:28 smithi045 conmon[33655]: ) 2750 : audit [INF] 2021-07-31T02:39:29.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:28 smithi045 conmon[33655]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:39:30.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:29 smithi154 conmon[34553]: cluster 2021-07-31T02:39:28.549588+0000 mgr.x (mgr.24542) 2234 : 2021-07-31T02:39:30.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:29 smithi154 conmon[34553]: cluster [DBG] pgmap v3157: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:39:30.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:29 smithi154 conmon[34553]: cluster 2021-07-31T02:39:28.794071+0000 mon.a (mon.0) 2751 : cluster [DBG] osdmap e1744: 8 total, 8 up, 8 in 2021-07-31T02:39:30.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:29 smithi045 conmon[39718]: cluster 2021-07-31T02:39:28.549588+0000 2021-07-31T02:39:30.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:29 smithi045 conmon[39718]: mgr.x (mgr.24542) 2234 : cluster [DBG] pgmap v3157: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:39:30.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:29 smithi045 conmon[39718]: cluster 2021-07-31T02: 2021-07-31T02:39:30.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:29 smithi045 conmon[39718]: 39:28.794071+0000 mon.a (mon.0) 2751 : cluster [DBG] osdmap e1744: 8 total, 8 up, 8 in 2021-07-31T02:39:30.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:29 smithi045 conmon[33655]: cluster 2021-07-31T02:39:28.549588+0000 mgr.x (mgr.24542) 2234 : cluster [DBG] 2021-07-31T02:39:30.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:29 smithi045 conmon[33655]: pgmap v3157: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:39:30.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:29 smithi045 conmon[33655]: cluster 2021-07-31T02:39:28.794071+0000 mon.a (mon.0) 2751 : cluster [DBG] osdmap e1744: 8 total, 8 up, 8 in 2021-07-31T02:39:31.219 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:39:31.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:30 smithi154 conmon[34553]: cluster 2021-07-31T02:39:29.795452+0000 mon.a (mon.0) 2752 : cluster [DBG] 2021-07-31T02:39:31.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:30 smithi154 conmon[34553]: osdmap e1745: 8 total, 8 up, 8 in 2021-07-31T02:39:31.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:30 smithi045 conmon[39718]: cluster 2021-07-31T02:39:29.795452+0000 mon.a (mon.0 2021-07-31T02:39:31.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:30 smithi045 conmon[39718]: ) 2752 : cluster [DBG] osdmap e1745: 8 total, 8 up, 8 in 2021-07-31T02:39:31.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:30 smithi045 conmon[33655]: cluster 2021-07-31T02:39:29.795452+0000 mon.a (mon.0) 2752 2021-07-31T02:39:31.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:30 smithi045 conmon[33655]: : cluster [DBG] osdmap e1745: 8 total, 8 up, 8 in 2021-07-31T02:39:31.496 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:39:31.497 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 54m ago 55m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:39:31.497 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 52m ago 54m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:39:31.497 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 52m ago 58m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:39:31.498 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 54m ago 60m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:39:31.498 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 54m ago 60m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:39:31.498 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 52m ago 58m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:39:31.498 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 54m ago 58m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:39:31.499 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 54m ago 55m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:39:31.499 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 52m ago 55m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:39:31.499 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 54m ago 58m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:39:31.499 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 54m ago 57m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:39:31.500 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 54m ago 57m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:39:31.500 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 54m ago 57m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:39:31.500 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 52m ago 56m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:39:31.501 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 52m ago 56m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:39:31.501 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 52m ago 56m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:39:31.501 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 52m ago 55m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:39:31.502 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 52m ago 55m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:39:31.834 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:39:31.835 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:39:31.835 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:39:31.835 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:39:31.835 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:39:31.835 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:39:31.836 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:39:31.836 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:39:31.836 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:39:31.836 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:39:31.837 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:39:31.837 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:39:31.837 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:39:31.837 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:39:31.838 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:39:31.838 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:39:31.838 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:39:32.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:31 smithi154 conmon[34553]: cluster 2021-07-31T02:39:32.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:31 smithi154 conmon[34553]: 2021- 2021-07-31T02:39:32.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:31 smithi154 conmon[34553]: 07-31T02:39:30.550029+0000 mgr.x (mgr.24542) 2235 : cluster [DBG] pgmap v3160: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 386 KiB/s rd, 3.7 KiB/s wr, 518 op/s 2021-07-31T02:39:32.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:31 smithi154 conmon[34553]: cluster 2021-07-31T02:39:30.804972+0000 mon.a (mon.0) 2753 : cluster [DBG] osdmap e1746: 8 total, 8 up, 8 in 2021-07-31T02:39:32.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:31 smithi154 conmon[34553]: audit 2021-07-31T02:39:31.207356+0000 mgr.x (mgr.24542) 2236 : audit [DBG] from='client.27730 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:39:32.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:31 smithi045 conmon[33655]: cluster 2021-07-31T02:39:30.550029+0000 mgr.x ( 2021-07-31T02:39:32.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:31 smithi045 conmon[33655]: mgr.24542) 2235 : cluster [DBG] pgmap v3160: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 386 KiB/s rd, 3.7 KiB/s wr, 518 op/s 2021-07-31T02:39:32.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:31 smithi045 conmon[33655]: cluster 2021-07-31T02:39:30.804972+0000 mon.a (mon.0) 2753 : cluster [DBG] osdmap e1746: 8 total, 8 up, 8 in 2021-07-31T02:39:32.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:31 smithi045 conmon[33655]: audit 2021-07-31T02:39:32.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:31 smithi045 conmon[33655]: 2021-07-31T02 2021-07-31T02:39:32.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:31 smithi045 conmon[33655]: :39:31.207356+0000 mgr.x 2021-07-31T02:39:32.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:31 smithi045 conmon[33655]: (mgr.24542) 2236 : audit [DBG] 2021-07-31T02:39:32.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:31 smithi045 conmon[33655]: from='client.27730 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:39:32.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:31 smithi045 conmon[39718]: cluster 2021-07-31T02:39 2021-07-31T02:39:32.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:31 smithi045 conmon[39718]: :30.550029+0000 mgr.x (mgr.24542) 2235 : cluster [DBG] pgmap v3160: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 386 KiB/s rd, 3.7 KiB/s wr, 518 op/s 2021-07-31T02:39:32.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:31 smithi045 conmon[39718]: cluster 2021-07-31T02:39:32.282 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:31 smithi045 conmon[39718]: 2021-07-31T02: 2021-07-31T02:39:32.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:31 smithi045 conmon[39718]: 39:30.804972+0000 mon.a 2021-07-31T02:39:32.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:31 smithi045 conmon[39718]: (mon.0) 2753 : cluster 2021-07-31T02:39:32.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:31 smithi045 conmon[39718]: [DBG] osdmap e1746: 8 total, 8 up, 8 in 2021-07-31T02:39:32.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:31 smithi045 conmon[39718]: audit 2021-07-31T02:39:32.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:31 smithi045 conmon[39718]: 2021-07- 2021-07-31T02:39:32.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:31 smithi045 conmon[39718]: 31T02:39:31. 2021-07-31T02:39:32.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:31 smithi045 conmon[39718]: 207356+0000 mgr.x (mgr.24542 2021-07-31T02:39:32.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:31 smithi045 conmon[39718]: ) 2236 : audit [DBG] from='client.27730 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:39:32.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:31 smithi045 conmon[39718]: 2021-07-31T02:39:33.130 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:32 smithi154 conmon[34553]: audit 2021-07-31T02:39:31.488584+0000 mgr.x (mgr.24542) 2237 2021-07-31T02:39:33.130 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:32 smithi154 conmon[34553]: : audit [DBG] from='client.17796 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:39:33.130 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:32 smithi154 conmon[34553]: cluster 2021-07-31T02:39:31.805754+0000 mon.a (mon.0) 2754 : cluster [DBG] osdmap e1747: 8 total, 8 up, 8 in 2021-07-31T02:39:33.131 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:32 smithi154 conmon[34553]: audit 2021-07-31T02:39:31.834545 2021-07-31T02:39:33.131 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:32 smithi154 conmon[34553]: +0000 mon.a (mon.0) 2755 : audit [DBG] from='client.? 172.21.15.45:0/3804275732' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:39:33.131 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:32 smithi045 conmon[33655]: audit 2021-07-31T02:39:31.488584+0000 mgr.x (mgr.24542) 2237 : audit 2021-07-31T02:39:33.132 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:32 smithi045 conmon[33655]: [DBG] from='client.17796 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:39:33.132 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:32 smithi045 conmon[33655]: cluster 2021-07-31T02:39:31.805754+0000 mon.a (mon.0) 2754 : cluster [DBG] osdmap e1747: 8 total, 8 up, 8 in 2021-07-31T02:39:33.132 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:32 smithi045 conmon[33655]: audit 2021-07-31T02:39:31.834545+0000 2021-07-31T02:39:33.132 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:32 smithi045 conmon[33655]: mon.a (mon.0) 2755 : audit [DBG] from='client.? 172.21.15.45:0/3804275732' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:39:33.133 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:32 smithi045 conmon[39718]: audit 2021-07-31T02:39:31.488584+0000 2021-07-31T02:39:33.133 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:32 smithi045 conmon[39718]: mgr.x (mgr.24542) 2237 : audit [DBG] from='client.17796 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:39:33.133 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:32 smithi045 conmon[39718]: cluster 2021-07-31T02:39:31 2021-07-31T02:39:33.134 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:32 smithi045 conmon[39718]: .805754+0000 mon.a (mon.0) 2754 : cluster [DBG] osdmap e1747: 8 total, 8 up, 8 in 2021-07-31T02:39:33.134 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:32 smithi045 conmon[39718]: audit 2021-07-31T02:39:31.834545 2021-07-31T02:39:33.134 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:32 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2755 : audit [DBG] from='client.? 172.21.15.45:0/3804275732' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:39:33.503 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: debug 2021-07-31T02:39:33.148+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:39:33.503 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:39:33.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:39:33.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: yield 2021-07-31T02:39:33.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:39:33.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:39:33.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:39:33.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:39:33.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:39:33.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:39:33.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:39:33.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:39:33.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:39:33.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: task.result() 2021-07-31T02:39:33.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:39:33.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: debug 2021-07-31T02:39:33.159+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:39:33.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:39:33.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:39:33.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: yield 2021-07-31T02:39:33.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:39:33.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:39:33.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:39:33.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:39:33.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:39:33.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:39:33.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:39:33.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:39:33.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:39:33.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: task.result() 2021-07-31T02:39:33.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:39:33.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: debug 2021-07-31T02:39:33.187+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:39:33.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:39:33.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:39:33.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: yield 2021-07-31T02:39:33.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:39:33.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:39:33.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:39:33.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:39:33.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:39:33.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:39:33.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:39:33.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:39:33.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:39:33.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: task.result() 2021-07-31T02:39:33.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:39:33.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: debug 2021-07-31T02:39:33.187+0000 7fcb9195d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:39:33.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:39:33.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:39:33.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: yield 2021-07-31T02:39:33.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:39:33.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:39:33.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:39:33.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:39:33.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:39:33.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:39:33.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:39:33.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:39:33.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:39:33.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: task.result() 2021-07-31T02:39:33.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:39:33.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: 2021-07-31T02:39:33.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:39:33.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: 2021-07-31T02:39:33.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:39:33.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:39:33.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:39:33.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:39:33.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:39:33.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:39:33.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:39:33.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:39:33.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:39:33.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:39:33.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:39:33.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:39:33.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:39:33.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:39:33.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:39:33.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:39:33.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:39:33.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:39:33.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: raise 2021-07-31T02:39:33.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:39:33.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:39:33.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:39:33.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:39:33.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:39:33.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: debug 2021-07-31T02:39:33.195+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:39:33.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:39:33.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:39:33.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: yield 2021-07-31T02:39:33.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:39:33.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:39:33.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:39:33.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:39:33.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:39:33.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:39:33.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:39:33.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:39:33.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:39:33.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: task.result() 2021-07-31T02:39:33.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:39:33.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: debug 2021-07-31T02:39:33.196+0000 7fcb9215e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:39:33.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:39:33.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:39:33.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: yield 2021-07-31T02:39:33.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:39:33.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:39:33.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:39:33.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:39:33.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:39:33.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:39:33.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:39:33.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:39:33.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:39:33.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: task.result() 2021-07-31T02:39:33.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:39:33.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: 2021-07-31T02:39:33.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:39:33.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: 2021-07-31T02:39:33.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:39:33.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:39:33.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:39:33.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:39:33.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:39:33.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:39:33.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:39:33.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:39:33.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:39:33.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:39:33.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:39:33.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:39:33.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:39:33.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:39:33.535 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:39:33.535 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:39:33.535 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:39:33.535 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:39:33.535 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: raise 2021-07-31T02:39:33.536 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:39:33.536 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:39:33.536 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:39:33.536 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:39:33.536 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:39:34.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: cluster 2021-07-31T02:39:32.550579+0000 mgr.x (mgr.24542) 2238 : cluster [DBG] pgmap v3163: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 386 KiB/s rd, 3.7 KiB/s wr, 518 op/s 2021-07-31T02:39:34.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: cluster 2021-07-31T02:39 2021-07-31T02:39:34.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: :32.812838+0000 mon.a (mon.0) 2756 : cluster [DBG] osdmap e1748: 8 total, 8 up, 8 in 2021-07-31T02:39:34.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: audit 2021-07-31T02:39:33.103543+0000 mon.b (mon.2) 337 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:39:34.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: audit 2021-07-31T02:39:34.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: 2021-07-31T02:39:33.105115+0000 mon.b (mon.2) 2021-07-31T02:39:34.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: 338 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:39:34.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: audit 2021-07-31T02:39:33.105991 2021-07-31T02:39:34.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: +0000 mon.b (mon.2) 339 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:39:34.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: cephadm 2021-07- 2021-07-31T02:39:34.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: 31T02:39:33.106741+0000 2021-07-31T02:39:34.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: mgr.x (mgr.24542) 2239 : cephadm [WRN] 2021-07-31T02:39:34.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:39:34.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:39:33 2021-07-31T02:39:34.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: .149472+0000 mgr.x (mgr.24542) 2240 : cephadm 2021-07-31T02:39:34.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: [ERR] 2021-07-31T02:39:34.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:39:34.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:39:34.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: yield 2021-07-31T02:39:34.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:39:34.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:39:34.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:39:34.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:39:34.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:39:34.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:39:34.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:39:34.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:39:34.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:39:34.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:39:34.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:39:34.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:39:33.149876+0000 mgr.x (mgr.24542) 2241 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:39:34.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:39:33 2021-07-31T02:39:34.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: .160768+0000 mgr.x (mgr.24542) 2242 : cephadm [ERR] 2021-07-31T02:39:34.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:39:34.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:39:34.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: yield 2021-07-31T02:39:34.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:39:34.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:39:34.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:39:34.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:39:34.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:39:34.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:39:34.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:39:34.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:39:34.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:39:34.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:39:34.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:39:34.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:39:33.161640+0000 mgr.x (mgr.24542) 2243 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:39:34.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: cephadm 2021-07 2021-07-31T02:39:34.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: -31T02:39:33.187956+0000 mgr.x ( 2021-07-31T02:39:34.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: mgr.24542) 2244 : cephadm [ERR] 2021-07-31T02:39:34.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:39:34.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:39:34.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: yield 2021-07-31T02:39:34.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:39:34.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:39:34.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:39:34.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:39:34.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:39:34.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:39:34.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:39:34.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:39:34.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:39:34.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:39:34.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:39:34.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: 2021-07-31T02:39:34.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:39:33.188335+0000 mgr.x (mgr.24542) 2245 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:39:34.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:39:34.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:39:34.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: yield 2021-07-31T02:39:34.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:39:34.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:39:34.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:39:34.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:39:34.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:39:34.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:39:34.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:39:34.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:39:34.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:39:34.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:39:34.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:39:34.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: 2021-07-31T02:39:34.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:39:34.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: 2021-07-31T02:39:34.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:39:34.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:39:34.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:39:34.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:39:34.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:39:34.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:39:34.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:39:34.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:39:34.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:39:34.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:39:34.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:39:34.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:39:34.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:39:34.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:39:34.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:39:34.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:39:34.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:39:34.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:39:34.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: raise 2021-07-31T02:39:34.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:39:34.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:39:34.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:39:34.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:39:34.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:39:34.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:39:33.196820+0000 mgr.x (mgr.24542) 2246 : cephadm [ERR] 2021-07-31T02:39:34.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:39:34.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:39:34.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: yield 2021-07-31T02:39:34.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:39:34.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:39:34.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:39:34.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:39:34.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:39:34.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:39:34.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: cluster 2021-07-31T02:39:32.550579+0000 mgr.x ( 2021-07-31T02:39:34.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: mgr.24542) 2238 : cluster [DBG] pgmap v3163: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 386 KiB/s rd, 3.7 KiB/s wr, 518 op/s 2021-07-31T02:39:34.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: cluster 2021-07-31T02:39:32.812838+0000 mon.a (mon.0) 2756 : cluster 2021-07-31T02:39:34.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: [DBG] osdmap e1748: 8 total, 8 up, 8 in 2021-07-31T02:39:34.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: audit 2021-07-31T02:39:33.103543+0000 mon.b 2021-07-31T02:39:34.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: (mon.2) 337 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:39:34.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: audit 2021 2021-07-31T02:39:34.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: -07-31T02:39:33.105115+0000 mon.b (mon.2) 338 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:39:34.282 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: audit 2021-07-31T02: 2021-07-31T02:39:34.282 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: 39:33.105991+0000 mon.b (mon.2) 339 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:39:34.282 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: cephadm 2021-07-31T 2021-07-31T02:39:34.282 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: 02:39:33.106741+0000 mgr.x (mgr.24542) 2239 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:39:34.282 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: 2021-07-31T02:39:34.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:39 2021-07-31T02:39:34.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: :33.149472+0000 mgr.x (mgr.24542) 2240 : cephadm [ERR] 2021-07-31T02:39:34.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:39:34.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:39:34.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: yield 2021-07-31T02:39:34.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:39:34.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:39:34.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:39:34.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:39:34.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:39:34.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:39:34.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:39:34.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:39:34.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:39:34.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:39:34.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:39:34.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:39:33.149876+0000 mgr.x (mgr.24542) 2241 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:39:34.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:39:33.160768+0000 mgr.x (mgr.24542) 2242 : cephadm [ERR] 2021-07-31T02:39:34.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:39:34.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:39:34.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: yield 2021-07-31T02:39:34.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:39:34.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:39:34.288 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:39:34.288 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:39:34.288 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:39:34.289 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:39:34.289 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:39:34.289 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:39:33.197212+0000 mgr.x (mgr.24542) 2247 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:39:34.289 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:39:34.290 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:39:34.290 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: yield 2021-07-31T02:39:34.290 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:39:34.290 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:39:34.291 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:39:34.291 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:39:34.291 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:39:34.291 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:39:34.292 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:39:34.292 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:39:34.292 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:39:34.292 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:39:34.292 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:39:34.293 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: 2021-07-31T02:39:34.293 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:39:34.293 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: 2021-07-31T02:39:34.293 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:39:34.293 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:39:34.294 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:39:34.294 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:39:34.294 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:39:34.294 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:39:34.295 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:39:34.295 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:39:34.295 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:39:34.295 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:39:34.296 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:39:34.296 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:39:34.296 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:39:34.296 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:39:34.296 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:39:34.297 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:39:34.297 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:39:34.297 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:39:34.297 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: raise 2021-07-31T02:39:34.297 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:39:34.298 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:39:34.298 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:39:34.298 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:39:34.298 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:33 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:39:34.298 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: cluster 2021-07-31T02:39:32.550579+0000 mgr.x (mgr.24542) 2238 : cluster 2021-07-31T02:39:34.299 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: [DBG] pgmap v3163: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 386 KiB/s rd, 3.7 KiB/s wr, 518 op/s 2021-07-31T02:39:34.299 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: cluster 2021-07-31T02:39:32.812838+0000 mon.a (mon.0) 2756 : cluster [DBG] osdmap e1748: 8 total, 8 up, 8 in 2021-07-31T02:39:34.299 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: audit 2021- 2021-07-31T02:39:34.299 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: 07-31T02:39:33.103543+0000 mon.b (mon.2 2021-07-31T02:39:34.300 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: ) 337 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:39:34.300 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: audit 2021-07-31T02:39:34.300 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: 2021-07-31T02:39: 2021-07-31T02:39:34.300 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: 33.105115+0000 mon.b (mon.2) 338 : audit 2021-07-31T02:39:34.300 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:39:34.301 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: audit 2021-07-31T02:39:34.301 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: 2021-07-31T02 2021-07-31T02:39:34.301 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: :39:33.105991+0000 mon.b (mon.2) 2021-07-31T02:39:34.301 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: 339 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:39:34.302 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:39:34.302 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: 2021-07-31T 2021-07-31T02:39:34.302 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: 02:39:33.106741+0000 mgr.x (mgr. 2021-07-31T02:39:34.302 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: 24542) 2239 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:39:34.302 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:39:34.303 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: 2021-07-31T02:39:33.149472+0000 mgr.x (mgr. 2021-07-31T02:39:34.303 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: 24542) 2240 : cephadm [ERR] 2021-07-31T02:39:34.303 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:39:34.303 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:39:34.304 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: yield 2021-07-31T02:39:34.304 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:39:34.304 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:39:34.304 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:39:34.305 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:39:34.305 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:39:34.305 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:39:34.305 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:39:34.305 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:39:34.306 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:39:34.306 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:39:34.306 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:39:34.306 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:39:33.149876+0000 mgr.x (mgr.24542) 2241 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:39:34.307 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: cephadm 2021 2021-07-31T02:39:34.307 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: -07-31T02:39:33.160768+0000 mgr.x (mgr.24542) 2242 : 2021-07-31T02:39:34.307 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: cephadm [ERR] 2021-07-31T02:39:34.307 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:39:34.307 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:39:34.308 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: yield 2021-07-31T02:39:34.308 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:39:34.308 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:39:34.308 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:39:34.308 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:39:34.309 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:39:34.309 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:39:34.309 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:39:34.309 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:39:34.309 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:39:34.310 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:39:34.310 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:39:34.310 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:39:33.161640+0000 mgr.x (mgr.24542) 2243 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:39:34.310 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:39:33 2021-07-31T02:39:34.311 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: .187956+0000 mgr.x (mgr.24542) 2244 : cephadm [ERR] 2021-07-31T02:39:34.311 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:39:34.311 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:39:34.311 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: yield 2021-07-31T02:39:34.311 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:39:34.312 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:39:34.312 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:39:34.312 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:39:34.312 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:39:34.312 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:39:34.313 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:39:34.313 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:39:34.313 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:39:34.313 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:39:34.314 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:39:34.314 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:39:33.188335+0000 mgr.x (mgr.24542) 2245 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:39:34.314 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:39:34.314 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:39:34.314 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: yield 2021-07-31T02:39:34.315 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:39:34.315 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:39:34.315 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:39:34.315 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:39:34.316 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:39:34.316 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:39:34.316 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:39:34.317 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:39:34.317 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:39:34.317 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:39:34.317 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:39:34.318 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:39:34.318 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:39:34.318 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:39:34.318 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:39:34.318 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: cephadm 2021-07-31T02: 2021-07-31T02:39:34.319 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: 39:33.161640+0000 mgr.x (mgr.24542) 2243 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:39:34.319 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:39:33.187956+0000 mgr.x (mgr.24542) 2244 : cephadm [ERR] 2021-07-31T02:39:34.319 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:39:34.319 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:39:34.320 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: yield 2021-07-31T02:39:34.320 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:39:34.320 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:39:34.320 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:39:34.321 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:39:34.321 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:39:34.321 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:39:34.321 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:39:34.321 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:39:34.322 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:39:34.322 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:39:34.322 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:39:34.322 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:39:33.188335+0000 mgr.x (mgr.24542 2021-07-31T02:39:34.323 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: ) 2245 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:39:34.323 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:39:34.323 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:39:34.323 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: yield 2021-07-31T02:39:34.323 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:39:34.324 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:39:34.324 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:39:34.324 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:39:34.324 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:39:34.325 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:39:34.325 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:39:34.325 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:39:34.325 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:39:34.326 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:39:34.326 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:39:34.326 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:39:34.326 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:39:34.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:39:34.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: 2021-07-31T02:39:34.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:39:34.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: 2021-07-31T02:39:34.328 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:39:34.328 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:39:34.328 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:39:34.328 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:39:34.328 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:39:34.329 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:39:34.329 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:39:34.329 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:39:34.329 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:39:34.330 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:39:34.330 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:39:34.330 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:39:34.330 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:39:34.331 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:39:34.331 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:39:34.331 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:39:34.331 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:39:34.331 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:39:34.332 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: raise 2021-07-31T02:39:34.332 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:39:34.332 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:39:34.332 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:39:34.333 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:39:34.333 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:39:34.333 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:39:33.196820 2021-07-31T02:39:34.333 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 2246 : cephadm [ERR] 2021-07-31T02:39:34.333 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:39:34.334 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:39:34.334 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: yield 2021-07-31T02:39:34.334 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:39:34.334 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:39:34.335 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:39:34.335 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:39:34.335 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:39:34.336 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:39:34.336 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: 2021-07-31T02:39:34.336 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:39:34.336 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: 2021-07-31T02:39:34.336 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:39:34.337 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:39:34.337 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:39:34.337 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:39:34.338 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:39:34.338 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:39:34.338 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:39:34.338 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:39:34.338 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:39:34.339 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:39:34.339 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:39:34.339 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:39:34.339 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:39:34.340 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:39:34.340 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:39:34.340 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:39:34.340 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:39:34.340 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:39:34.341 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: raise 2021-07-31T02:39:34.341 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:39:34.341 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:39:34.341 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:39:34.342 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:39:34.342 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:39:34.342 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:39:33.196820+0000 mgr.x (mgr.24542) 2246 : cephadm [ERR] 2021-07-31T02:39:34.342 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:39:34.343 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:39:34.343 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: yield 2021-07-31T02:39:34.343 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:39:34.343 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:39:34.343 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:39:34.344 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:39:34.344 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:39:34.344 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:39:34.344 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:39:34.345 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:39:34.345 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:39:34.346 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:39:34.346 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:39:34.346 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:39:34.346 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:39:34.347 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:39:34.347 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:39:34.347 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:39:33.197212+0000 mgr.x (mgr.24542) 2247 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:39:34.347 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:39:34.347 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:39:34.348 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: yield 2021-07-31T02:39:34.348 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:39:34.348 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:39:34.348 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:39:34.349 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:39:34.349 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:39:34.349 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:39:34.349 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:39:34.350 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:39:34.350 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:39:34.350 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:39:34.350 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:39:34.351 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: 2021-07-31T02:39:34.351 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:39:34.351 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: 2021-07-31T02:39:34.351 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:39:34.352 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:39:34.352 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:39:34.352 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:39:34.352 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:39:34.352 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:39:34.353 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:39:34.353 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:39:34.353 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:39:34.353 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:39:34.354 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:39:34.354 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:39:34.354 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:39:34.354 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:39:34.354 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:39:34.355 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:39:34.355 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:39:34.355 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:39:34.355 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: raise 2021-07-31T02:39:34.355 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:39:34.356 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:39:34.356 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:39:34.356 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:39:34.356 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:39:34.357 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:39:34.357 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:39:34.357 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:39:34.358 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:39:33.197212+0000 mgr.x (mgr.24542) 2247 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:39:34.358 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:39:34.358 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:39:34.358 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: yield 2021-07-31T02:39:34.359 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:39:34.359 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:39:34.359 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:39:34.359 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:39:34.359 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:39:34.360 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:39:34.360 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:39:34.360 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:39:34.360 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:39:34.360 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:39:34.361 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:39:34.361 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: 2021-07-31T02:39:34.361 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:39:34.361 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: 2021-07-31T02:39:34.362 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:39:34.362 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:39:34.362 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:39:34.362 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:39:34.362 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:39:34.363 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:39:34.363 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:39:34.363 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:39:34.363 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:39:34.364 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:39:34.364 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:39:34.364 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:39:34.364 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:39:34.364 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:39:34.365 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:39:34.365 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:39:34.365 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:39:34.365 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:39:34.365 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: raise 2021-07-31T02:39:34.366 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:39:34.366 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:39:34.366 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:39:34.366 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:39:34.367 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:33 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:39:35.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:34 smithi154 conmon[34553]: cluster 2021-07-31T02:39:33.824065+0000 mon.a (mon.0) 2021-07-31T02:39:35.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:34 smithi154 conmon[34553]: 2757 : cluster [DBG] osdmap e1749: 8 total, 8 up, 8 in 2021-07-31T02:39:35.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:34 smithi045 conmon[33655]: cluster 2021-07-31T02:39:33.824065+0000 mon.a (mon.0) 2757 : cluster [DBG] osdmap e1749: 8 total, 8 up, 8 in 2021-07-31T02:39:35.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:34 smithi045 conmon[33655]: 2021-07-31T02:39:35.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:34 smithi045 conmon[39718]: cluster 2021-07-31T02:39:33.824065+0000 mon.a (mon.0 2021-07-31T02:39:35.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:34 smithi045 conmon[39718]: ) 2757 : cluster [DBG] osdmap e1749: 8 total, 8 up, 8 in 2021-07-31T02:39:36.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:35 smithi154 conmon[34553]: cluster 2021-07-31T02:39:34.551093+0000 mgr.x (mgr.24542) 2248 : cluster [DBG] pgmap v3166: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 419 KiB/s rd, 3.7 KiB/s wr, 562 op/s 2021-07-31T02:39:36.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:35 smithi154 conmon[34553]: cluster 2021-07-31T02:39:34.829826+0000 mon.a (mon.0) 2758 : cluster [DBG] osdmap e1750: 8 total, 8 up, 8 in 2021-07-31T02:39:36.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:35 smithi045 conmon[39718]: cluster 2021-07-31T02:39:34.551093+0000 mgr.x ( 2021-07-31T02:39:36.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:35 smithi045 conmon[39718]: mgr.24542) 2248 : cluster [DBG] pgmap v3166: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 419 KiB/s rd, 3.7 KiB/s wr, 562 op/s 2021-07-31T02:39:36.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:35 smithi045 conmon[39718]: cluster 2021-07-31T02:39:34.829826 2021-07-31T02:39:36.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:35 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2758 : cluster [DBG] osdmap e1750: 8 total, 8 up, 8 in 2021-07-31T02:39:36.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:35 smithi045 conmon[33655]: cluster 2021-07-31T02:39:34.551093+0000 mgr.x (mgr.24542) 2248 : cluster [DBG] pgmap v3166: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 419 KiB/s rd, 3.7 KiB/s wr, 562 op/s 2021-07-31T02:39:36.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:35 smithi045 conmon[33655]: cluster 2021-07-31T02:39:34.829826+0000 mon.a (mon.0) 2758 : cluster [DBG] osdmap e1750: 8 total, 8 up, 8 in 2021-07-31T02:39:37.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:36 smithi154 conmon[34553]: cluster 2021-07-31T02:39:35.835487+0000 mon.a (mon.0) 2759 : cluster [DBG] osdmap e1751: 8 total, 8 up, 8 in 2021-07-31T02:39:37.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:36 smithi045 conmon[33655]: cluster 2021-07-31T02:39:35.835487+0000 mon.a (mon.0 2021-07-31T02:39:37.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:36 smithi045 conmon[33655]: ) 2759 : cluster [DBG] osdmap e1751: 8 total, 8 up, 8 in 2021-07-31T02:39:37.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:36 smithi045 conmon[39718]: cluster 2021-07-31T02:39:35.835487+0000 mon.a (mon.0) 2759 2021-07-31T02:39:37.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:36 smithi045 conmon[39718]: : cluster [DBG] osdmap e1751: 8 total, 8 up, 8 in 2021-07-31T02:39:38.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:37 smithi154 conmon[34553]: cluster 2021-07-31T02:39:36.551683+0000 mgr.x (mgr.24542) 2249 : cluster [DBG] pgmap v3169: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 419 KiB/s rd, 3.7 KiB/s wr, 562 op/s 2021-07-31T02:39:38.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:37 smithi154 conmon[34553]: cluster 2021-07-31T02:39:36.841820+0000 mon.a (mon.0) 2760 : cluster [DBG] osdmap e1752: 8 total, 8 up, 8 in 2021-07-31T02:39:38.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:37 smithi045 conmon[39718]: cluster 2021-07-31T02:39:36.551683+0000 mgr.x (mgr.24542) 2249 2021-07-31T02:39:38.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:37 smithi045 conmon[39718]: : cluster [DBG] pgmap v3169: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 419 KiB/s rd, 3.7 KiB/s wr, 562 op/s 2021-07-31T02:39:38.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:37 smithi045 conmon[39718]: cluster 2021-07-31T02: 2021-07-31T02:39:38.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:37 smithi045 conmon[39718]: 39:36.841820+0000 mon.a (mon.0) 2760 : cluster [DBG] osdmap e1752: 8 total, 8 up, 8 in 2021-07-31T02:39:38.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:37 smithi045 conmon[33655]: cluster 2021-07-31T02:39:36.551683+0000 mgr.x (mgr.24542) 2249 : 2021-07-31T02:39:38.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:37 smithi045 conmon[33655]: cluster [DBG] pgmap v3169: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 419 KiB/s rd, 3.7 KiB/s wr, 562 op/s 2021-07-31T02:39:38.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:37 smithi045 conmon[33655]: cluster 2021-07-31T02:39:36.841820+0000 mon.a (mon.0) 2760 : cluster [DBG] osdmap e1752: 8 total, 8 up, 8 in 2021-07-31T02:39:39.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:38 smithi154 conmon[34553]: cluster 2021-07-31T02:39:37.847284+0000 mon.a (mon.0) 2761 : cluster [DBG] 2021-07-31T02:39:39.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:38 smithi154 conmon[34553]: osdmap e1753: 8 total, 8 up, 8 in 2021-07-31T02:39:39.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:38 smithi045 conmon[33655]: cluster 2021-07-31T02:39:37.847284+0000 mon.a (mon.0) 2761 : cluster 2021-07-31T02:39:39.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:38 smithi045 conmon[33655]: [DBG] osdmap e1753: 8 total, 8 up, 8 in 2021-07-31T02:39:39.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:38 smithi045 conmon[39718]: cluster 2021-07-31T02:39:37.847284+0000 mon.a (mon.0) 2761 : cluster [DBG] osdmap e1753: 8 total, 8 up, 8 in 2021-07-31T02:39:40.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:39 smithi154 conmon[34553]: cluster 2021-07-31T02:39:38.552097+0000 mgr.x (mgr.24542) 2250 : cluster [DBG] pgmap v3172: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:39:40.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:39 smithi154 conmon[34553]: cluster 2021-07-31T02:39:38.852738+0000 mon.a (mon.0) 2762 : cluster [DBG] osdmap e1754: 8 total, 8 up, 8 in 2021-07-31T02:39:40.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:39 smithi045 conmon[33655]: cluster 2021-07-31T02:39:38.552097+0000 mgr.x (mgr.24542) 2250 : cluster 2021-07-31T02:39:40.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:39 smithi045 conmon[33655]: [DBG] pgmap v3172: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:39:40.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:39 smithi045 conmon[33655]: cluster 2021-07-31T02:39:38.852738+0000 mon.a (mon.0) 2762 : cluster [DBG] osdmap e1754: 8 total, 8 up, 8 in 2021-07-31T02:39:40.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:39 smithi045 conmon[39718]: cluster 2021-07-31T02:39:38.552097+0000 mgr.x ( 2021-07-31T02:39:40.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:39 smithi045 conmon[39718]: mgr.24542) 2250 : cluster [DBG] pgmap v3172: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:39:40.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:39 smithi045 conmon[39718]: cluster 2021-07-31T02:39:38.852738+0000 mon.a ( 2021-07-31T02:39:40.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:39 smithi045 conmon[39718]: mon.0) 2762 : cluster [DBG] osdmap e1754: 8 total, 8 up, 8 in 2021-07-31T02:39:41.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:40 smithi154 conmon[34553]: cluster 2021-07-31T02:39:39.858257+0000 mon.a (mon.0) 2763 : cluster [DBG] osdmap e1755: 8 total, 8 up, 8 in 2021-07-31T02:39:41.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:40 smithi154 conmon[34553]: 2021-07-31T02:39:41.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:40 smithi045 conmon[33655]: cluster 2021-07-31T02:39:39.858257+0000 mon.a (mon.0) 2763 : 2021-07-31T02:39:41.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:40 smithi045 conmon[33655]: cluster [DBG] osdmap e1755: 8 total, 8 up, 8 in 2021-07-31T02:39:41.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:40 smithi045 conmon[39718]: cluster 2021-07-31T02:39:39.858257+0000 2021-07-31T02:39:41.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:40 smithi045 conmon[39718]: mon.a (mon.0) 2763 : cluster [DBG] osdmap e1755: 8 total, 8 up, 8 in 2021-07-31T02:39:42.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:41 smithi154 conmon[34553]: cluster 2021-07-31T02:39:40.552670+0000 mgr.x (mgr.24542) 2251 : cluster [DBG] pgmap v3175: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 359 KiB/s rd, 3.0 KiB/s wr, 482 op/s 2021-07-31T02:39:42.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:41 smithi154 conmon[34553]: cluster 2021-07-31T02:39:40.864344+0000 mon.a (mon.0) 2764 : cluster [DBG] osdmap e1756: 8 total, 8 up, 8 in 2021-07-31T02:39:42.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:41 smithi045 conmon[33655]: cluster 2021-07-31T02:39:40.552670+0000 mgr.x (mgr.24542) 2251 : cluster 2021-07-31T02:39:42.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:41 smithi045 conmon[33655]: [DBG] pgmap v3175: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 359 KiB/s rd, 3.0 KiB/s wr, 482 op/s 2021-07-31T02:39:42.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:41 smithi045 conmon[33655]: cluster 2021-07-31T02:39:40.864344+0000 mon.a (mon.0) 2021-07-31T02:39:42.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:41 smithi045 conmon[33655]: 2764 : cluster [DBG] osdmap e1756: 8 total, 8 up, 8 in 2021-07-31T02:39:42.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:41 smithi045 conmon[39718]: cluster 2021-07-31T02:39:40.552670+0000 mgr.x ( 2021-07-31T02:39:42.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:41 smithi045 conmon[39718]: mgr.24542) 2251 : cluster [DBG] pgmap v3175: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 359 KiB/s rd, 3.0 KiB/s wr, 482 op/s 2021-07-31T02:39:42.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:41 smithi045 conmon[39718]: cluster 2021- 2021-07-31T02:39:42.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:41 smithi045 conmon[39718]: 07-31T02:39:40.864344+0000 mon.a (mon.0) 2021-07-31T02:39:42.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:41 smithi045 conmon[39718]: 2764 : cluster [DBG] osdmap e1756: 8 total, 8 up, 8 in 2021-07-31T02:39:43.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:42 smithi154 conmon[34553]: cluster 2021-07-31T02:39:41.870238+0000 mon.a (mon.0) 2765 : cluster [DBG] osdmap e1757: 8 total, 8 up, 8 in 2021-07-31T02:39:43.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:42 smithi045 conmon[33655]: cluster 2021-07-31T02:39:41.870238+0000 mon.a (mon.0) 2765 : cluster [DBG] 2021-07-31T02:39:43.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:42 smithi045 conmon[33655]: osdmap e1757: 8 total, 8 up, 8 in 2021-07-31T02:39:43.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:42 smithi045 conmon[39718]: cluster 2021-07-31T02:39:41.870238+0000 mon.a ( 2021-07-31T02:39:43.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:42 smithi045 conmon[39718]: mon.0) 2765 : cluster [DBG] osdmap e1757: 8 total, 8 up, 8 in 2021-07-31T02:39:44.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:43 smithi154 conmon[34553]: cluster 2021-07-31T02:39:42.553274+0000 mgr.x (mgr.24542) 2252 : cluster [DBG] pgmap v3178: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 359 KiB/s rd, 3.0 KiB/s wr, 482 op/s 2021-07-31T02:39:44.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:43 smithi154 conmon[34553]: cluster 2021-07-31T02:39:42.872202+0000 mon.a (mon.0) 2766 : cluster [DBG] osdmap e1758: 8 total, 8 up, 8 in 2021-07-31T02:39:44.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:43 smithi045 conmon[39718]: cluster 2021-07-31T02:39: 2021-07-31T02:39:44.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:43 smithi045 conmon[39718]: 42.553274+0000 mgr.x (mgr.24542) 2252 : cluster [DBG] pgmap v3178: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 359 KiB/s rd, 3.0 KiB/s wr, 482 op/s 2021-07-31T02:39:44.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:43 smithi045 conmon[39718]: cluster 2021-07-31T02: 2021-07-31T02:39:44.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:43 smithi045 conmon[39718]: 39:42.872202+0000 mon.a (mon 2021-07-31T02:39:44.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:43 smithi045 conmon[39718]: .0) 2766 : cluster [DBG] osdmap e1758: 8 total, 8 up, 8 in 2021-07-31T02:39:44.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:43 smithi045 conmon[39718]: 2021-07-31T02:39:44.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:43 smithi045 conmon[33655]: cluster 2021-07-31T02:39:42.553274+0000 mgr.x ( 2021-07-31T02:39:44.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:43 smithi045 conmon[33655]: mgr.24542) 2252 : cluster [DBG] pgmap v3178: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 359 KiB/s rd, 3.0 KiB/s wr, 482 op/s 2021-07-31T02:39:44.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:43 smithi045 conmon[33655]: cluster 2021-07-31T 2021-07-31T02:39:44.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:43 smithi045 conmon[33655]: 02:39:42.872202+0000 mon.a (mon.0) 2766 : cluster [DBG] 2021-07-31T02:39:44.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:43 smithi045 conmon[33655]: osdmap e1758: 8 total, 8 up, 8 in 2021-07-31T02:39:45.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:44 smithi154 conmon[34553]: cluster 2021-07-31T02:39:43.888713+0000 mon.a (mon.0) 2021-07-31T02:39:45.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:44 smithi154 conmon[34553]: 2767 : cluster [DBG] osdmap e1759: 8 total, 8 up, 8 in 2021-07-31T02:39:45.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:44 smithi045 conmon[33655]: cluster 2021-07-31T02:39:43.888713+0000 2021-07-31T02:39:45.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:44 smithi045 conmon[33655]: mon.a (mon.0) 2767 : cluster [DBG] osdmap e1759: 8 total, 8 up, 8 in 2021-07-31T02:39:45.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:44 smithi045 conmon[39718]: cluster 2021-07-31T02:39:43.888713+0000 mon.a (mon. 2021-07-31T02:39:45.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:44 smithi045 conmon[39718]: 0) 2767 : cluster [DBG] osdmap e1759: 8 total, 8 up, 8 in 2021-07-31T02:39:46.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:45 smithi154 conmon[34553]: cluster 2021-07-31T02:39:44.553677+0000 mgr.x (mgr.24542) 2253 : cluster [DBG] pgmap v3181: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 479 KiB/s rd, 3.7 KiB/s wr, 644 op/s 2021-07-31T02:39:46.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:45 smithi154 conmon[34553]: 2021-07-31T02:39:46.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:45 smithi154 conmon[34553]: cluster 2021-07-31T02:39:44.893706+0000 mon.a (mon.0) 2768 : cluster [DBG] osdmap e1760: 8 total, 8 up, 8 in 2021-07-31T02:39:46.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:45 smithi045 conmon[39718]: cluster 2021-07-31T02:39:44.553677+0000 mgr.x (mgr. 2021-07-31T02:39:46.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:45 smithi045 conmon[39718]: 24542) 2253 : cluster [DBG] pgmap v3181: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 479 KiB/s rd, 3.7 KiB/s wr, 644 op/s 2021-07-31T02:39:46.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:45 smithi045 conmon[39718]: cluster 2021-07-31T02:39:44.893706+0000 mon.a (mon.0) 2768 : 2021-07-31T02:39:46.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:45 smithi045 conmon[39718]: cluster [DBG] osdmap e1760: 8 total, 8 up, 8 in 2021-07-31T02:39:46.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:45 smithi045 conmon[33655]: cluster 2021-07-31T02:39:44 2021-07-31T02:39:46.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:45 smithi045 conmon[33655]: .553677+0000 mgr.x (mgr.24542) 2253 : cluster [DBG] pgmap v3181: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 479 KiB/s rd, 3.7 KiB/s wr, 644 op/s 2021-07-31T02:39:46.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:45 smithi045 conmon[33655]: cluster 2021-07-31T02:39:44.893706+0000 mon.a (mon.0) 2768 : cluster [DBG] osdmap e1760: 8 total, 8 up, 8 in 2021-07-31T02:39:47.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:46 smithi154 conmon[34553]: cluster 2021-07-31T02:39:45.895329+0000 mon.a (mon.0) 2769 : cluster [DBG] osdmap e1761: 8 total, 8 up, 8 in 2021-07-31T02:39:47.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:46 smithi045 conmon[33655]: cluster 2021-07-31T02:39:45.895329+0000 mon.a (mon.0) 2769 : 2021-07-31T02:39:47.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:46 smithi045 conmon[33655]: cluster [DBG] osdmap e1761: 8 total, 8 up, 8 in 2021-07-31T02:39:47.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:46 smithi045 conmon[39718]: cluster 2021-07-31T02:39:45.895329 2021-07-31T02:39:47.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:46 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2769 : cluster [DBG] osdmap e1761: 8 total, 8 up, 8 in 2021-07-31T02:39:48.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:47 smithi154 conmon[34553]: cluster 2021-07-31T02:39:48.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:47 smithi154 conmon[34553]: 2021-07-31T02:39:46.554201+0000 mgr.x (mgr.24542 2021-07-31T02:39:48.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:47 smithi154 conmon[34553]: ) 2254 : cluster [DBG] pgmap v3184: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 479 KiB/s rd, 3.7 KiB/s wr, 644 op/s 2021-07-31T02:39:48.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:47 smithi154 conmon[34553]: cluster 2021-07-31T02 2021-07-31T02:39:48.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:47 smithi154 conmon[34553]: :39:46.908095+0000 mon.a (mon.0) 2770 : cluster [DBG] osdmap e1762: 8 total, 8 up, 8 in 2021-07-31T02:39:48.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:47 smithi045 conmon[33655]: cluster 2021-07-31T02:39:46.554201+0000 mgr.x (mgr.24542 2021-07-31T02:39:48.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:47 smithi045 conmon[33655]: ) 2254 : cluster [DBG] pgmap v3184: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 479 KiB/s rd, 3.7 KiB/s wr, 644 op/s 2021-07-31T02:39:48.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:47 smithi045 conmon[33655]: cluster 2021-07-31T02:39:46.908095+0000 mon.a (mon.0) 2770 : cluster [DBG] osdmap e1762: 8 total, 8 up, 8 in 2021-07-31T02:39:48.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:47 smithi045 conmon[39718]: cluster 2021-07-31T02:39:46. 2021-07-31T02:39:48.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:47 smithi045 conmon[39718]: 554201+0000 mgr.x (mgr.24542) 2254 : cluster [DBG] pgmap v3184: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 479 KiB/s rd, 3.7 KiB/s wr, 644 op/s 2021-07-31T02:39:48.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:47 smithi045 conmon[39718]: cluster 2021-07-31T02:39:48.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:47 smithi045 conmon[39718]: 2021-07-31T02:39:46.908095+0000 mon.a (mon.0) 2770 : cluster [DBG] osdmap e1762: 8 total, 8 up, 8 in 2021-07-31T02:39:49.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:48 smithi154 conmon[34553]: cluster 2021-07-31T02:39:47.909365+0000 mon.a (mon.0) 2771 : cluster [DBG] osdmap e1763: 8 total, 8 up, 8 in 2021-07-31T02:39:49.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:48 smithi154 conmon[34553]: 2021-07-31T02:39:49.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:48 smithi045 conmon[39718]: cluster 2021-07-31T02:39:47.909365+0000 mon.a 2021-07-31T02:39:49.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:48 smithi045 conmon[39718]: (mon.0) 2771 : cluster [DBG] osdmap e1763: 8 total, 8 up, 8 in 2021-07-31T02:39:49.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:48 smithi045 conmon[33655]: cluster 2021-07-31T02:39:47.909365+0000 mon.a (mon.0) 2771 : cluster [DBG] 2021-07-31T02:39:49.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:48 smithi045 conmon[33655]: osdmap e1763: 8 total, 8 up, 8 in 2021-07-31T02:39:50.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:49 smithi154 conmon[34553]: cluster 2021-07-31T02:39:48.554707+0000 mgr.x (mgr.24542) 2255 : cluster [DBG] pgmap v3187: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:39:50.343 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:49 smithi154 conmon[34553]: 2021-07-31T02:39:50.343 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:49 smithi154 conmon[34553]: cluster 2021-07-31T02:39:48.919796+0000 mon.a (mon.0) 2772 : cluster [DBG] osdmap e1764: 8 total, 8 up, 8 in 2021-07-31T02:39:50.344 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:49 smithi045 conmon[39718]: cluster 2021-07-31T02:39:48.554707+0000 mgr.x ( 2021-07-31T02:39:50.344 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:49 smithi045 conmon[39718]: mgr.24542) 2255 : cluster [DBG] pgmap v3187: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:39:50.344 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:49 smithi045 conmon[39718]: cluster 2021-07-31T02:39:48.919796+0000 mon.a (mon.0 2021-07-31T02:39:50.345 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:49 smithi045 conmon[39718]: ) 2772 : cluster [DBG] osdmap e1764: 8 total, 8 up, 8 in 2021-07-31T02:39:50.345 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:49 smithi045 conmon[33655]: cluster 2021-07-31T02:39:48.554707+0000 mgr.x (mgr.24542) 2021-07-31T02:39:50.345 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:49 smithi045 conmon[33655]: 2255 : cluster [DBG] pgmap v3187: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:39:50.346 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:49 smithi045 conmon[33655]: cluster 2021-07-31T02:39:48.919796+0000 mon.a (mon.0) 2772 : cluster [DBG] osdmap e1764: 8 total, 8 up, 8 in 2021-07-31T02:39:51.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:50 smithi154 conmon[34553]: cluster 2021-07-31T02:39:49.925373+0000 mon.a (mon. 2021-07-31T02:39:51.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:50 smithi154 conmon[34553]: 0) 2773 : cluster [DBG] osdmap e1765: 8 total, 8 up, 8 in 2021-07-31T02:39:51.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:50 smithi045 conmon[39718]: cluster 2021-07-31T02:39:49. 2021-07-31T02:39:51.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:50 smithi045 conmon[39718]: 925373+0000 mon.a (mon.0) 2773 : cluster [DBG] osdmap e1765: 8 total, 8 up, 8 in 2021-07-31T02:39:51.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:50 smithi045 conmon[33655]: cluster 2021-07-31T02:39:49.925373+0000 mon.a (mon.0) 2773 : cluster [DBG] osdmap e1765: 8 total, 8 up, 8 in 2021-07-31T02:39:52.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:51 smithi154 conmon[34553]: cluster 2021-07-31T02:39:50.555198+0000 mgr.x (mgr.24542) 2256 : cluster [DBG] 2021-07-31T02:39:52.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:51 smithi154 conmon[34553]: pgmap v3190: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 511 KiB/s rd, 3.7 KiB/s wr, 688 op/s 2021-07-31T02:39:52.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:51 smithi154 conmon[34553]: cluster 2021-07-31T02:39:50.931211+0000 mon.a (mon.0) 2774 : cluster [DBG] osdmap e1766: 8 total, 8 up, 8 in 2021-07-31T02:39:52.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:51 smithi045 conmon[33655]: cluster 2021-07-31T02:39:50.555198+0000 mgr.x (mgr.24542) 2256 : cluster 2021-07-31T02:39:52.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:51 smithi045 conmon[33655]: [DBG] pgmap v3190: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 511 KiB/s rd, 3.7 KiB/s wr, 688 op/s 2021-07-31T02:39:52.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:51 smithi045 conmon[33655]: cluster 2021-07-31T02:39:50.931211+0000 mon.a (mon.0) 2774 : cluster [DBG] osdmap e1766: 8 total, 8 up, 8 in 2021-07-31T02:39:52.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:51 smithi045 conmon[39718]: cluster 2021-07-31T02:39:50.555198+0000 mgr.x (mgr. 2021-07-31T02:39:52.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:51 smithi045 conmon[39718]: 24542) 2256 : cluster [DBG] pgmap v3190: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 511 KiB/s rd, 3.7 KiB/s wr, 688 op/s 2021-07-31T02:39:52.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:51 smithi045 conmon[39718]: cluster 2021-07-31T02:39:50.931211+0000 mon.a (mon.0) 2774 : cluster [DBG] 2021-07-31T02:39:52.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:51 smithi045 conmon[39718]: osdmap e1766: 8 total, 8 up, 8 in 2021-07-31T02:39:53.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:52 smithi154 conmon[34553]: cluster 2021-07-31T02:39:53.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:52 smithi154 conmon[34553]: 2021-07-31T02:39:51.937062 2021-07-31T02:39:53.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:52 smithi154 conmon[34553]: +0000 mon.a (mon.0) 2775 : cluster [DBG] osdmap e1767: 8 total, 8 up, 8 in 2021-07-31T02:39:53.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:52 smithi045 conmon[33655]: cluster 2021-07-31T02:39:51.937062+0000 mon.a (mon.0) 2775 2021-07-31T02:39:53.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:52 smithi045 conmon[33655]: : cluster [DBG] osdmap e1767: 8 total, 8 up, 8 in 2021-07-31T02:39:53.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:52 smithi045 conmon[39718]: cluster 2021-07-31T02:39:51.937062+0000 mon.a ( 2021-07-31T02:39:53.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:52 smithi045 conmon[39718]: mon.0) 2775 : cluster [DBG] osdmap e1767: 8 total, 8 up, 8 in 2021-07-31T02:39:54.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:53 smithi154 conmon[34553]: cluster 2021-07-31T02:39:52.555638+0000 mgr.x (mgr.24542) 2257 : cluster [DBG] pgmap v3193: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 511 KiB/s rd, 3.7 KiB/s wr, 688 op/s 2021-07-31T02:39:54.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:53 smithi154 conmon[34553]: cluster 2021-07-31T02:39:52.938683+0000 mon.a (mon.0) 2776 : cluster [DBG] osdmap e1768: 8 total, 8 up, 8 in 2021-07-31T02:39:54.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:53 smithi045 conmon[39718]: cluster 2021-07-31T02:39:52.555638+0000 mgr.x (mgr.24542 2021-07-31T02:39:54.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:53 smithi045 conmon[39718]: ) 2257 : cluster [DBG] pgmap v3193: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 511 KiB/s rd, 3.7 KiB/s wr, 688 op/s 2021-07-31T02:39:54.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:53 smithi045 conmon[39718]: cluster 2021-07-31T02:39:52.938683+0000 mon.a (mon.0) 2776 : cluster 2021-07-31T02:39:54.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:53 smithi045 conmon[39718]: [DBG] osdmap e1768: 8 total, 8 up, 8 in 2021-07-31T02:39:54.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:53 smithi045 conmon[33655]: cluster 2021-07-31T02:39:52.555638+0000 mgr.x (mgr.24542) 2257 : cluster 2021-07-31T02:39:54.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:53 smithi045 conmon[33655]: [DBG] pgmap v3193: 41 pgs: 41 active+clean; 4.6 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 511 KiB/s rd, 3.7 KiB/s wr, 688 op/s 2021-07-31T02:39:54.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:53 smithi045 conmon[33655]: cluster 2021-07-31T02:39:52.938683+0000 mon.a (mon.0) 2776 : cluster [DBG] osdmap e1768: 8 total, 8 up, 8 in 2021-07-31T02:39:55.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:54 smithi154 conmon[34553]: cluster 2021-07-31T02:39:53.953395+0000 mon.a (mon.0) 2777 : cluster [DBG] 2021-07-31T02:39:55.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:54 smithi154 conmon[34553]: osdmap e1769: 8 total, 8 up, 8 in 2021-07-31T02:39:55.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:54 smithi045 conmon[33655]: cluster 2021-07-31T02:39:53.953395+0000 mon.a (mon.0) 2777 : cluster 2021-07-31T02:39:55.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:54 smithi045 conmon[33655]: [DBG] osdmap e1769: 8 total, 8 up, 8 in 2021-07-31T02:39:55.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:54 smithi045 conmon[39718]: cluster 2021-07-31T02:39:53.953395+0000 mon.a (mon.0 2021-07-31T02:39:55.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:54 smithi045 conmon[39718]: ) 2777 : cluster [DBG] osdmap e1769: 8 total, 8 up, 8 in 2021-07-31T02:39:56.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:55 smithi154 conmon[34553]: cluster 2021-07-31T02:39:54.556198+0000 mgr.x (mgr.24542) 2258 : cluster [DBG] pgmap v3196: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 544 KiB/s rd, 3.7 KiB/s wr, 732 op/s 2021-07-31T02:39:56.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:55 smithi154 conmon[34553]: cluster 2021-07-31T02:39:54.958467+0000 mon.a (mon.0) 2778 : cluster [DBG] osdmap e1770: 8 total, 8 up, 8 in 2021-07-31T02:39:56.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:55 smithi045 conmon[33655]: cluster 2021-07-31T02:39:54.556198+0000 mgr.x (mgr.24542) 2258 : cluster 2021-07-31T02:39:56.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:55 smithi045 conmon[33655]: [DBG] pgmap v3196: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 544 KiB/s rd, 3.7 KiB/s wr, 732 op/s 2021-07-31T02:39:56.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:55 smithi045 conmon[33655]: cluster 2021-07-31T02:39:54.958467+0000 mon.a (mon.0) 2778 : cluster [DBG] osdmap e1770: 8 total, 8 up, 8 in 2021-07-31T02:39:56.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:55 smithi045 conmon[39718]: cluster 2021-07-31T02:39:54.556198+0000 mgr.x ( 2021-07-31T02:39:56.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:55 smithi045 conmon[39718]: mgr.24542) 2258 : cluster [DBG] pgmap v3196: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 544 KiB/s rd, 3.7 KiB/s wr, 732 op/s 2021-07-31T02:39:56.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:55 smithi045 conmon[39718]: cluster 2021-07-31T02:39:54.958467+0000 mon.a (mon.0) 2021-07-31T02:39:56.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:55 smithi045 conmon[39718]: 2778 : cluster [DBG] osdmap e1770: 8 total, 8 up, 8 in 2021-07-31T02:39:57.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:56 smithi154 conmon[34553]: cluster 2021-07-31T02:39:55.960291+0000 mon.a (mon.0) 2779 : 2021-07-31T02:39:57.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:56 smithi154 conmon[34553]: cluster [DBG] osdmap e1771: 8 total, 8 up, 8 in 2021-07-31T02:39:57.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:56 smithi045 conmon[39718]: cluster 2021-07-31T02:39:55.960291 2021-07-31T02:39:57.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:56 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2779 : cluster [DBG] osdmap e1771: 8 total, 8 up, 8 in 2021-07-31T02:39:57.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:56 smithi045 conmon[33655]: cluster 2021-07-31T02:39:55.960291+0000 mon.a (mon.0) 2021-07-31T02:39:57.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:56 smithi045 conmon[33655]: 2779 : cluster [DBG] osdmap e1771: 8 total, 8 up, 8 in 2021-07-31T02:39:58.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:57 smithi154 conmon[34553]: cluster 2021-07-31T02:39:56.556764+0000 mgr.x (mgr.24542) 2259 : cluster [DBG] pgmap v3199: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 544 KiB/s rd, 3.7 KiB/s wr, 732 op/s 2021-07-31T02:39:58.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:57 smithi154 conmon[34553]: cluster 2021-07-31T02:39:56.971141+0000 mon.a (mon.0) 2780 : cluster [DBG] osdmap e1772: 8 total, 8 up, 8 in 2021-07-31T02:39:58.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:57 smithi045 conmon[39718]: cluster 2021-07-31T02:39:56.556764+0000 mgr.x (mgr.24542) 2021-07-31T02:39:58.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:57 smithi045 conmon[39718]: 2259 : cluster [DBG] pgmap v3199: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 544 KiB/s rd, 3.7 KiB/s wr, 732 op/s 2021-07-31T02:39:58.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:57 smithi045 conmon[39718]: cluster 2021-07-31T02:39:56.971141+0000 mon.a (mon.0) 2780 : cluster [DBG] osdmap e1772: 8 total, 8 up, 8 in 2021-07-31T02:39:58.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:57 smithi045 conmon[33655]: cluster 2021-07-31T02:39:56.556764+0000 mgr.x (mgr.24542 2021-07-31T02:39:58.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:57 smithi045 conmon[33655]: ) 2259 : cluster [DBG] pgmap v3199: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 544 KiB/s rd, 3.7 KiB/s wr, 732 op/s 2021-07-31T02:39:58.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:57 smithi045 conmon[33655]: cluster 2021-07-31T02:39:56.971141+0000 mon.a (mon.0) 2780 : cluster [DBG] osdmap e1772: 8 total, 8 up, 8 in 2021-07-31T02:39:59.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:58 smithi154 conmon[34553]: cluster 2021-07-31T02:39:57.976204+0000 mon.a (mon.0) 2781 : cluster [DBG] 2021-07-31T02:39:59.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:58 smithi154 conmon[34553]: osdmap e1773: 8 total, 8 up, 8 in 2021-07-31T02:39:59.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:58 smithi045 conmon[39718]: cluster 2021-07-31T02:39:57.976204+0000 mon.a (mon.0) 2781 : 2021-07-31T02:39:59.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:58 smithi045 conmon[39718]: cluster [DBG] osdmap e1773: 8 total, 8 up, 8 in 2021-07-31T02:39:59.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:58 smithi045 conmon[33655]: cluster 2021-07-31T02:39:57.976204+0000 mon.a (mon.0) 2781 : cluster [DBG] osdmap e1773: 8 total, 8 up, 8 in 2021-07-31T02:39:59.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:58 smithi045 conmon[33655]: 2021-07-31T02:40:00.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:59 smithi154 conmon[34553]: cluster 2021-07-31T02:40:00.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:59 smithi154 conmon[34553]: 2021- 2021-07-31T02:40:00.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:59 smithi154 conmon[34553]: 07-31T02:39:58.557277+0000 mgr.x (mgr.24542) 2260 : cluster [DBG] pgmap v3202: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:40:00.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:59 smithi154 conmon[34553]: cluster 2021-07-31T02:40:00.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:59 smithi154 conmon[34553]: 2021-07-31T02:39:58.982067+0000 mon.a ( 2021-07-31T02:40:00.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:39:59 smithi154 conmon[34553]: mon.0) 2782 : cluster [DBG] osdmap e1774: 8 total, 8 up, 8 in 2021-07-31T02:40:00.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:59 smithi045 conmon[33655]: cluster 2021-07-31T02:39:58.557277+0000 2021-07-31T02:40:00.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:59 smithi045 conmon[33655]: mgr.x (mgr.24542) 2260 : cluster [DBG] pgmap v3202: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:40:00.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:59 smithi045 conmon[33655]: cluster 2021-07-31T02:39:58.982067+0000 mon.a (mon.0) 2782 : cluster [DBG] 2021-07-31T02:40:00.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:39:59 smithi045 conmon[33655]: osdmap e1774: 8 total, 8 up, 8 in 2021-07-31T02:40:00.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:59 smithi045 conmon[39718]: cluster 2021-07-31T02:39:58.557277+0000 2021-07-31T02:40:00.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:59 smithi045 conmon[39718]: mgr.x (mgr.24542) 2260 : cluster [DBG] pgmap v3202: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:40:00.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:59 smithi045 conmon[39718]: cluster 2021-07-31T02:39:58.982067 2021-07-31T02:40:00.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:39:59 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2782 : cluster [DBG] osdmap e1774: 8 total, 8 up, 8 in 2021-07-31T02:40:01.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:00 smithi154 conmon[34553]: cluster 2021-07-31T02:39:59.984720+0000 mon.a (mon.0) 2783 : cluster [DBG] 2021-07-31T02:40:01.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:00 smithi154 conmon[34553]: osdmap e1775: 8 total, 8 up, 8 in 2021-07-31T02:40:01.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:00 smithi154 conmon[34553]: cluster 2021-07-31T02:40:00.000146+0000 mon.a (mon.0) 2784 : cluster [INF] overall HEALTH_OK 2021-07-31T02:40:01.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:00 smithi045 conmon[39718]: cluster 2021-07-31T02:40:01.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:00 smithi045 conmon[39718]: 2021-07- 2021-07-31T02:40:01.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:00 smithi045 conmon[39718]: 31T02:39:59.984720+0000 mon.a (mon.0) 2783 : cluster [DBG] osdmap e1775: 8 total, 8 up, 8 in 2021-07-31T02:40:01.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:00 smithi045 conmon[39718]: cluster 2021-07 2021-07-31T02:40:01.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:00 smithi045 conmon[39718]: -31T02:40:00.000146+0000 mon.a (mon.0) 2784 : cluster [INF] 2021-07-31T02:40:01.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:00 smithi045 conmon[39718]: overall HEALTH_OK 2021-07-31T02:40:01.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:00 smithi045 conmon[33655]: cluster 2021-07-31T02:39:59.984720+0000 2021-07-31T02:40:01.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:00 smithi045 conmon[33655]: mon.a (mon.0) 2783 : cluster [DBG] osdmap e1775: 8 total, 8 up, 8 in 2021-07-31T02:40:01.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:00 smithi045 conmon[33655]: cluster 2021-07-31T02:40:00. 2021-07-31T02:40:01.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:00 smithi045 conmon[33655]: 000146+0000 mon.a (mon.0) 2784 : cluster [INF] overall HEALTH_OK 2021-07-31T02:40:02.153 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:40:02.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:01 smithi154 conmon[34553]: cluster 2021-07-31T02:40:00.557851+0000 mgr.x (mgr.24542) 2261 : cluster [DBG] 2021-07-31T02:40:02.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:01 smithi154 conmon[34553]: pgmap v3205: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 575 KiB/s rd, 3.7 KiB/s wr, 776 op/s 2021-07-31T02:40:02.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:01 smithi154 conmon[34553]: cluster 2021-07-31T02:40:00.996868+0000 mon.a (mon.0) 2785 : cluster [DBG] osdmap e1776: 8 total, 8 up, 8 in 2021-07-31T02:40:02.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:01 smithi045 conmon[33655]: cluster 2021-07-31T02:40:00.557851+0000 mgr.x (mgr 2021-07-31T02:40:02.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:01 smithi045 conmon[33655]: .24542) 2261 : cluster [DBG] pgmap v3205: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 575 KiB/s rd, 3.7 KiB/s wr, 776 op/s 2021-07-31T02:40:02.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:01 smithi045 conmon[33655]: cluster 2021-07-31T02:40:00.996868+0000 mon.a (mon.0) 2785 : cluster [DBG] osdmap e1776: 8 total, 8 up, 8 in 2021-07-31T02:40:02.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:01 smithi045 conmon[39718]: cluster 2021-07-31T02:40:00.557851+0000 mgr.x ( 2021-07-31T02:40:02.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:01 smithi045 conmon[39718]: mgr.24542) 2261 : cluster [DBG] pgmap v3205: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 575 KiB/s rd, 3.7 KiB/s wr, 776 op/s 2021-07-31T02:40:02.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:01 smithi045 conmon[39718]: cluster 2021-07-31T02:40:00.996868 2021-07-31T02:40:02.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:01 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2785 : cluster [DBG] osdmap e1776: 8 total, 8 up, 8 in 2021-07-31T02:40:02.430 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:40:02.430 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 54m ago 55m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:40:02.430 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 52m ago 55m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:40:02.431 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 52m ago 58m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:40:02.431 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 54m ago 61m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:40:02.431 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 54m ago 61m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:40:02.431 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 52m ago 59m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:40:02.432 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 54m ago 59m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:40:02.432 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 54m ago 55m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:40:02.432 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 52m ago 55m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:40:02.432 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 54m ago 58m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:40:02.433 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 54m ago 58m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:40:02.433 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 54m ago 58m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:40:02.433 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 54m ago 57m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:40:02.433 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 52m ago 57m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:40:02.433 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 52m ago 57m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:40:02.434 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 52m ago 56m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:40:02.434 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 52m ago 56m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:40:02.434 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 52m ago 56m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:40:02.765 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:40:02.766 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:40:02.766 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:40:02.766 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:40:02.766 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:40:02.767 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:40:02.767 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:40:02.767 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:40:02.767 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:40:02.768 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:40:02.768 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:40:02.768 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:40:02.768 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:40:02.769 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:40:02.769 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:40:02.769 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:40:02.769 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:40:03.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:03 smithi154 conmon[34553]: cluster 2021-07-31T02:40:02.001399+0000 mon.a (mon.0) 2786 : cluster [DBG] 2021-07-31T02:40:03.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:03 smithi154 conmon[34553]: osdmap e1777: 8 total, 8 up, 8 in 2021-07-31T02:40:03.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:03 smithi154 conmon[34553]: audit 2021-07-31T02:40:02.140704+0000 mgr.x (mgr.24542) 2262 : audit [DBG] from='client.17808 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:40:03.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:03 smithi154 conmon[34553]: audit 2021-07-31T02:40:02.422322+0000 mgr.x (mgr.24542) 2021-07-31T02:40:03.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:03 smithi154 conmon[34553]: 2263 : audit [DBG] from='client.17814 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:40:03.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:03 smithi154 conmon[34553]: audit 2021-07-31T02:40:02.764640+0000 mon.a (mon.0) 2787 : audit 2021-07-31T02:40:03.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:03 smithi154 conmon[34553]: [DBG] from='client.? 172.21.15.45:0/3460179002' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:40:03.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:03 smithi045 conmon[33655]: cluster 2021-07-31T02:40:02.001399+0000 mon.a (mon.0 2021-07-31T02:40:03.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:03 smithi045 conmon[33655]: ) 2786 : cluster [DBG] osdmap e1777: 8 total, 8 up, 8 in 2021-07-31T02:40:03.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:03 smithi045 conmon[33655]: audit 2021-07-31T02:40:02.140704+0000 mgr.x (mgr.24542) 2262 : audit [DBG] from='client.17808 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:40:03.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:03 smithi045 conmon[33655]: audit 2021- 2021-07-31T02:40:03.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:03 smithi045 conmon[33655]: 07-31T02:40:02. 2021-07-31T02:40:03.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:03 smithi045 conmon[33655]: 422322+0000 mgr.x (mgr.24542 2021-07-31T02:40:03.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:03 smithi045 conmon[33655]: ) 2263 : audit [DBG] from='client.17814 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:40:03.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:03 smithi045 conmon[33655]: audit 2021-07-31T 2021-07-31T02:40:03.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:03 smithi045 conmon[33655]: 02:40:02.764640+0000 mon.a 2021-07-31T02:40:03.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:03 smithi045 conmon[33655]: (mon.0) 2787 : audit 2021-07-31T02:40:03.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:03 smithi045 conmon[33655]: [DBG] from='client.? 172.21.15.45:0/3460179002' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:40:03.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:03 smithi045 conmon[39718]: cluster 2021-07-31T02:40:02.001399+0000 mon.a ( 2021-07-31T02:40:03.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:03 smithi045 conmon[39718]: mon.0) 2786 : cluster [DBG] osdmap e1777: 8 total, 8 up, 8 in 2021-07-31T02:40:03.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:03 smithi045 conmon[39718]: audit 2021-07 2021-07-31T02:40:03.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:03 smithi045 conmon[39718]: -31T02:40:02.140704+0000 mgr.x (mgr.24542) 2262 : audit [DBG] 2021-07-31T02:40:03.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:03 smithi045 conmon[39718]: from='client.17808 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:40:03.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:03 smithi045 conmon[39718]: audit 2021-07-31T02 2021-07-31T02:40:03.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:03 smithi045 conmon[39718]: :40:02.422322+0000 mgr.x (mgr.24542 2021-07-31T02:40:03.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:03 smithi045 conmon[39718]: ) 2263 : audit [DBG] from='client.17814 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:40:03.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:03 smithi045 conmon[39718]: audit 2021 2021-07-31T02:40:03.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:03 smithi045 conmon[39718]: -07-31T02:40:02.764640+0000 2021-07-31T02:40:03.282 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:03 smithi045 conmon[39718]: mon.a (mon.0) 2787 : audit [DBG] 2021-07-31T02:40:03.282 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:03 smithi045 conmon[39718]: from='client.? 172.21.15.45:0/3460179002' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:40:04.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:04 smithi045 conmon[33655]: cluster 2021-07-31T02:40:02.558367+0000 mgr.x (mgr.24542) 2264 : cluster [DBG] pgmap v3208: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 575 KiB/s rd, 3.7 KiB/s wr, 776 op/s 2021-07-31T02:40:04.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:04 smithi045 conmon[33655]: cluster 2021-07-31T02:40:03.006946+0000 mon.a (mon.0) 2788 : cluster [DBG] osdmap e1778: 8 total, 8 up, 8 in 2021-07-31T02:40:04.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:04 smithi045 conmon[39718]: cluster 2021-07-31T02:40:02.558367+0000 mgr.x (mgr 2021-07-31T02:40:04.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:04 smithi045 conmon[39718]: .24542) 2264 : cluster [DBG] pgmap v3208: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 575 KiB/s rd, 3.7 KiB/s wr, 776 op/s 2021-07-31T02:40:04.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:04 smithi045 conmon[39718]: cluster 2021-07-31T02:40:03.006946+0000 mon.a (mon.0) 2788 : 2021-07-31T02:40:04.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:04 smithi045 conmon[39718]: cluster [DBG] osdmap e1778: 8 total, 8 up, 8 in 2021-07-31T02:40:04.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:04 smithi154 conmon[34553]: cluster 2021-07-31T02:40:02.558367+0000 mgr.x (mgr.24542) 2264 : cluster [DBG] pgmap v3208: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 575 KiB/s rd, 3.7 KiB/s wr, 776 op/s 2021-07-31T02:40:04.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:04 smithi154 conmon[34553]: cluster 2021-07-31T02:40:03.006946+0000 mon.a (mon.0) 2788 : cluster [DBG] osdmap e1778: 8 total, 8 up, 8 in 2021-07-31T02:40:05.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:05 smithi045 conmon[39718]: cluster 2021-07-31T02:40:04.009724+0000 mon.a ( 2021-07-31T02:40:05.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:05 smithi045 conmon[39718]: mon.0) 2789 : cluster [DBG] osdmap e1779: 8 total, 8 up, 8 in 2021-07-31T02:40:05.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:05 smithi045 conmon[33655]: cluster 2021-07-31T02:40:04.009724+0000 mon.a (mon.0) 2021-07-31T02:40:05.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:05 smithi045 conmon[33655]: 2789 : cluster [DBG] osdmap e1779: 8 total, 8 up, 8 in 2021-07-31T02:40:05.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:05 smithi154 conmon[34553]: cluster 2021-07-31T02:40:04.009724+0000 mon.a (mon.0) 2789 : 2021-07-31T02:40:05.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:05 smithi154 conmon[34553]: cluster [DBG] osdmap e1779: 8 total, 8 up, 8 in 2021-07-31T02:40:06.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:06 smithi045 conmon[39718]: cluster 2021-07-31T02:40:04.558863+0000 mgr.x 2021-07-31T02:40:06.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:06 smithi045 conmon[39718]: (mgr.24542) 2265 : cluster [DBG] pgmap v3211: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 609 KiB/s rd, 3.7 KiB/s wr, 819 op/s 2021-07-31T02:40:06.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:06 smithi045 conmon[39718]: cluster 2021-07 2021-07-31T02:40:06.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:06 smithi045 conmon[39718]: -31T02:40:05.020478+0000 mon.a (mon.0) 2790 : 2021-07-31T02:40:06.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:06 smithi045 conmon[39718]: cluster [DBG] osdmap e1780: 8 total, 8 up, 8 in 2021-07-31T02:40:06.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:06 smithi045 conmon[33655]: cluster 2021-07-31T02:40:04.558863+0000 mgr.x (mgr. 2021-07-31T02:40:06.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:06 smithi045 conmon[33655]: 24542) 2265 : cluster [DBG] pgmap v3211: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 609 KiB/s rd, 3.7 KiB/s wr, 819 op/s 2021-07-31T02:40:06.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:06 smithi045 conmon[33655]: cluster 2021-07-31T02:40:05.020478+0000 mon.a (mon.0) 2790 : cluster [DBG] osdmap e1780: 8 total, 8 up, 8 in 2021-07-31T02:40:06.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:06 smithi154 conmon[34553]: cluster 2021-07-31T02:40:04.558863+0000 mgr.x (mgr.24542) 2021-07-31T02:40:06.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:06 smithi154 conmon[34553]: 2265 : cluster [DBG] pgmap v3211: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 609 KiB/s rd, 3.7 KiB/s wr, 819 op/s 2021-07-31T02:40:06.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:06 smithi154 conmon[34553]: cluster 2021-07-31T02:40:05.020478+0000 mon.a (mon.0) 2790 : cluster [DBG] osdmap e1780: 8 total, 8 up, 8 in 2021-07-31T02:40:07.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:07 smithi154 conmon[34553]: cluster 2021-07-31T02:40:06.021820+0000 mon.a (mon.0) 2791 : cluster [DBG] osdmap e1781: 8 total, 8 up, 8 in 2021-07-31T02:40:07.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:07 smithi154 conmon[34553]: 2021-07-31T02:40:07.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:07 smithi045 conmon[39718]: cluster 2021-07-31T02:40:06.021820+0000 mon.a (mon.0) 2791 : cluster 2021-07-31T02:40:07.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:07 smithi045 conmon[39718]: [DBG] osdmap e1781: 8 total, 8 up, 8 in 2021-07-31T02:40:07.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:07 smithi045 conmon[33655]: cluster 2021-07-31T02:40:06.021820+0000 mon.a (mon.0) 2021-07-31T02:40:07.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:07 smithi045 conmon[33655]: 2791 : cluster [DBG] osdmap e1781: 8 total, 8 up, 8 in 2021-07-31T02:40:08.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:08 smithi154 conmon[34553]: cluster 2021-07-31T02:40:06.559295+0000 mgr.x (mgr.24542) 2266 : cluster [DBG] pgmap v3214: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 609 KiB/s rd, 3.7 KiB/s wr, 819 op/s 2021-07-31T02:40:08.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:08 smithi154 conmon[34553]: cluster 2021-07-31T02:40:07.031865+0000 mon.a (mon.0) 2792 : cluster [DBG] osdmap e1782: 8 total, 8 up, 8 in 2021-07-31T02:40:08.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:08 smithi045 conmon[33655]: cluster 2021-07-31T02:40:06.559295+0000 mgr.x (mgr. 2021-07-31T02:40:08.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:08 smithi045 conmon[33655]: 24542) 2266 : cluster [DBG] pgmap v3214: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 609 KiB/s rd, 3.7 KiB/s wr, 819 op/s 2021-07-31T02:40:08.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:08 smithi045 conmon[33655]: cluster 2021-07-31T02:40:07.031865+0000 mon.a (mon.0) 2792 : cluster [DBG] osdmap e1782: 8 total, 8 up, 8 in 2021-07-31T02:40:08.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:08 smithi045 conmon[39718]: cluster 2021-07-31T02:40:06.559295+0000 2021-07-31T02:40:08.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:08 smithi045 conmon[39718]: mgr.x (mgr.24542) 2266 : cluster [DBG] pgmap v3214: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 609 KiB/s rd, 3.7 KiB/s wr, 819 op/s 2021-07-31T02:40:08.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:08 smithi045 conmon[39718]: cluster 2021-07-31T02:40:07. 2021-07-31T02:40:08.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:08 smithi045 conmon[39718]: 031865+0000 mon.a (mon.0) 2792 : cluster [DBG] osdmap e1782: 8 total, 8 up, 8 in 2021-07-31T02:40:09.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:09 smithi154 conmon[34553]: cluster 2021-07-31T02:40:08.037754+0000 mon.a (mon.0) 2021-07-31T02:40:09.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:09 smithi154 conmon[34553]: 2793 : cluster [DBG] osdmap e1783: 8 total, 8 up, 8 in 2021-07-31T02:40:09.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:09 smithi045 conmon[33655]: cluster 2021-07-31T02:40:08.037754+0000 mon.a (mon.0) 2793 : cluster 2021-07-31T02:40:09.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:09 smithi045 conmon[33655]: [DBG] osdmap e1783: 8 total, 8 up, 8 in 2021-07-31T02:40:09.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:09 smithi045 conmon[39718]: cluster 2021-07-31T02:40:08. 2021-07-31T02:40:09.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:09 smithi045 conmon[39718]: 037754+0000 mon.a (mon.0) 2793 : cluster [DBG] osdmap e1783: 8 total, 8 up, 8 in 2021-07-31T02:40:10.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:10 smithi154 conmon[34553]: cluster 2021-07-31T02:40:08.559717+0000 mgr.x (mgr.24542) 2267 : cluster [DBG] pgmap v3217: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:40:10.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:10 smithi154 conmon[34553]: cluster 2021-07-31T02:40:09.044195+0000 mon.a (mon.0) 2794 : cluster [DBG] osdmap e1784: 8 total, 8 up, 8 in 2021-07-31T02:40:10.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:10 smithi045 conmon[39718]: cluster 2021-07-31T02:40:08.559717+0000 mgr.x 2021-07-31T02:40:10.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:10 smithi045 conmon[39718]: (mgr.24542) 2267 : cluster [DBG] pgmap v3217: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:40:10.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:10 smithi045 conmon[39718]: cluster 2021-07-31T02:40 2021-07-31T02:40:10.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:10 smithi045 conmon[39718]: :09.044195+0000 mon.a (mon.0) 2794 : cluster [DBG] osdmap e1784: 8 total, 8 up, 8 in 2021-07-31T02:40:10.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:10 smithi045 conmon[33655]: cluster 2021-07-31T02:40:08.559717+0000 mgr.x (mgr.24542 2021-07-31T02:40:10.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:10 smithi045 conmon[33655]: ) 2267 : cluster [DBG] pgmap v3217: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:40:10.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:10 smithi045 conmon[33655]: cluster 2021-07-31T02:40:09.044195+0000 mon.a (mon.0) 2794 : cluster [DBG] osdmap e1784: 8 total, 8 up, 8 in 2021-07-31T02:40:11.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:11 smithi154 conmon[34553]: cluster 2021-07-31T02:40:10.049686+0000 mon.a (mon.0) 2795 : cluster 2021-07-31T02:40:11.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:11 smithi154 conmon[34553]: [DBG] osdmap e1785: 8 total, 8 up, 8 in 2021-07-31T02:40:11.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:11 smithi045 conmon[33655]: cluster 2021-07-31T02:40:10.049686+0000 mon.a (mon. 2021-07-31T02:40:11.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:11 smithi045 conmon[33655]: 0) 2795 : cluster [DBG] osdmap e1785: 8 total, 8 up, 8 in 2021-07-31T02:40:11.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:11 smithi045 conmon[39718]: cluster 2021-07-31T02:40:10.049686+0000 mon.a (mon. 2021-07-31T02:40:11.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:11 smithi045 conmon[39718]: 0) 2795 : cluster [DBG] osdmap e1785: 8 total, 8 up, 8 in 2021-07-31T02:40:12.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:12 smithi154 conmon[34553]: cluster 2021-07-31T02:40:10.560217+0000 mgr.x (mgr.24542) 2268 : cluster 2021-07-31T02:40:12.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:12 smithi154 conmon[34553]: [DBG] pgmap v3220: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 640 KiB/s rd, 3.7 KiB/s wr, 863 op/s 2021-07-31T02:40:12.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:12 smithi154 conmon[34553]: cluster 2021-07-31T02:40:11.056244+0000 mon.a (mon.0) 2796 : cluster [DBG] osdmap e1786: 8 total, 8 up, 8 in 2021-07-31T02:40:12.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:12 smithi045 conmon[39718]: cluster 2021-07-31T02:40:10.560217+0000 mgr.x (mgr 2021-07-31T02:40:12.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:12 smithi045 conmon[39718]: .24542) 2268 : cluster [DBG] pgmap v3220: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 640 KiB/s rd, 3.7 KiB/s wr, 863 op/s 2021-07-31T02:40:12.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:12 smithi045 conmon[39718]: cluster 2021-07-31T02:40:11.056244+0000 2021-07-31T02:40:12.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:12 smithi045 conmon[39718]: mon.a (mon.0) 2796 : cluster [DBG] osdmap e1786: 8 total, 8 up, 8 in 2021-07-31T02:40:12.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:12 smithi045 conmon[33655]: cluster 2021-07-31T02:40:10.560217+0000 mgr.x (mgr.24542) 2268 : cluster [DBG] pgmap v3220: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 640 KiB/s rd, 3.7 KiB/s wr, 863 op/s 2021-07-31T02:40:12.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:12 smithi045 conmon[33655]: cluster 2021-07-31T02:40:11.056244+0000 mon.a (mon.0) 2796 : cluster [DBG] osdmap e1786: 8 total, 8 up, 8 in 2021-07-31T02:40:13.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:13 smithi154 conmon[34553]: cluster 2021-07-31T02:40:12.056951+0000 mon.a (mon 2021-07-31T02:40:13.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:13 smithi154 conmon[34553]: .0) 2797 : cluster [DBG] osdmap e1787: 8 total, 8 up, 8 in 2021-07-31T02:40:13.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:13 smithi045 conmon[39718]: cluster 2021-07-31T02:40:12.056951+0000 mon.a (mon.0 2021-07-31T02:40:13.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:13 smithi045 conmon[39718]: ) 2797 : cluster [DBG] osdmap e1787: 8 total, 8 up, 8 in 2021-07-31T02:40:13.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:13 smithi045 conmon[33655]: cluster 2021-07-31T02:40:12.056951+0000 2021-07-31T02:40:13.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:13 smithi045 conmon[33655]: mon.a (mon.0) 2797 : cluster [DBG] osdmap e1787: 8 total, 8 up, 8 in 2021-07-31T02:40:14.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:14 smithi154 conmon[34553]: cluster 2021-07-31T02:40:12.560678+0000 mgr.x (mgr.24542) 2269 : cluster [DBG] pgmap v3223: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 640 KiB/s rd, 3.7 KiB/s wr, 863 op/s 2021-07-31T02:40:14.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:14 smithi154 conmon[34553]: 2021-07-31T02:40:14.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:14 smithi154 conmon[34553]: cluster 2021-07-31T02:40:13.062952+0000 mon.a (mon.0) 2798 : cluster [DBG] osdmap e1788: 8 total, 8 up, 8 in 2021-07-31T02:40:14.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:14 smithi045 conmon[39718]: cluster 2021-07-31T02:40:12.560678+0000 mgr.x ( 2021-07-31T02:40:14.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:14 smithi045 conmon[39718]: mgr.24542) 2269 : cluster [DBG] pgmap v3223: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 640 KiB/s rd, 3.7 KiB/s wr, 863 op/s 2021-07-31T02:40:14.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:14 smithi045 conmon[39718]: cluster 2021-07-31T02:40:13.062952+0000 mon.a ( 2021-07-31T02:40:14.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:14 smithi045 conmon[39718]: mon.0) 2798 : cluster [DBG] osdmap e1788: 8 total, 8 up, 8 in 2021-07-31T02:40:14.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:14 smithi045 conmon[33655]: cluster 2021-07-31T02:40:12.560678+0000 mgr.x (mgr.24542) 2269 2021-07-31T02:40:14.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:14 smithi045 conmon[33655]: : cluster [DBG] pgmap v3223: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 640 KiB/s rd, 3.7 KiB/s wr, 863 op/s 2021-07-31T02:40:14.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:14 smithi045 conmon[33655]: cluster 2021-07-31T02:40:13.062952+0000 mon.a (mon.0) 2798 : cluster [DBG] osdmap e1788: 8 total, 8 up, 8 in 2021-07-31T02:40:15.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:15 smithi154 conmon[34553]: cluster 2021-07-31T02:40:14.074027+0000 mon.a (mon 2021-07-31T02:40:15.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:15 smithi154 conmon[34553]: .0) 2799 : cluster [DBG] osdmap e1789: 8 total, 8 up, 8 in 2021-07-31T02:40:15.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:15 smithi045 conmon[39718]: cluster 2021-07-31T02:40:14.074027+0000 mon.a (mon. 2021-07-31T02:40:15.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:15 smithi045 conmon[39718]: 0) 2799 : cluster [DBG] osdmap e1789: 8 total, 8 up, 8 in 2021-07-31T02:40:15.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:15 smithi045 conmon[33655]: cluster 2021-07-31T02:40:14.074027+0000 2021-07-31T02:40:15.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:15 smithi045 conmon[33655]: mon.a (mon.0) 2799 : cluster [DBG] osdmap e1789: 8 total, 8 up, 8 in 2021-07-31T02:40:16.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:16 smithi154 conmon[34553]: cluster 2021-07-31T02:40:14.561153+0000 mgr.x (mgr.24542) 2270 : cluster [DBG] pgmap v3226: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 674 KiB/s rd, 3.7 KiB/s wr, 907 op/s 2021-07-31T02:40:16.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:16 smithi154 conmon[34553]: 2021-07-31T02:40:16.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:16 smithi154 conmon[34553]: cluster 2021-07-31T02:40:15.076899+0000 mon.a (mon.0) 2800 : cluster [DBG] osdmap e1790: 8 total, 8 up, 8 in 2021-07-31T02:40:16.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:16 smithi045 conmon[39718]: cluster 2021-07-31T02:40:14.561153+0000 mgr.x (mgr. 2021-07-31T02:40:16.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:16 smithi045 conmon[39718]: 24542) 2270 : cluster [DBG] pgmap v3226: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 674 KiB/s rd, 3.7 KiB/s wr, 907 op/s 2021-07-31T02:40:16.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:16 smithi045 conmon[39718]: cluster 2021-07-31T02: 2021-07-31T02:40:16.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:16 smithi045 conmon[39718]: 40:15.076899+0000 mon.a (mon.0) 2800 : cluster [DBG] osdmap e1790: 8 total, 8 up, 8 in 2021-07-31T02:40:16.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:16 smithi045 conmon[33655]: cluster 2021-07-31T02:40:14.561153+0000 mgr.x (mgr.24542 2021-07-31T02:40:16.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:16 smithi045 conmon[33655]: ) 2270 : cluster [DBG] pgmap v3226: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 674 KiB/s rd, 3.7 KiB/s wr, 907 op/s 2021-07-31T02:40:16.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:16 smithi045 conmon[33655]: cluster 2021-07-31T02:40:15.076899+0000 mon.a (mon.0) 2800 : cluster [DBG] osdmap e1790: 8 total, 8 up, 8 in 2021-07-31T02:40:17.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:17 smithi154 conmon[34553]: cluster 2021-07-31T02:40:16.087891+0000 mon.a (mon.0) 2801 : cluster [DBG] osdmap e1791: 8 total, 8 up, 8 in 2021-07-31T02:40:17.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:17 smithi045 conmon[39718]: cluster 2021-07-31T02:40:16.087891+0000 mon.a (mon 2021-07-31T02:40:17.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:17 smithi045 conmon[39718]: .0) 2801 : cluster [DBG] osdmap e1791: 8 total, 8 up, 8 in 2021-07-31T02:40:17.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:17 smithi045 conmon[33655]: cluster 2021-07-31T02:40:16.087891+0000 mon.a (mon.0) 2021-07-31T02:40:17.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:17 smithi045 conmon[33655]: 2801 : cluster [DBG] osdmap e1791: 8 total, 8 up, 8 in 2021-07-31T02:40:18.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:18 smithi154 conmon[34553]: cluster 2021-07-31T02:40:16.561512+0000 mgr.x (mgr.24542) 2271 : 2021-07-31T02:40:18.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:18 smithi154 conmon[34553]: cluster [DBG] pgmap v3229: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 674 KiB/s rd, 3.7 KiB/s wr, 907 op/s 2021-07-31T02:40:18.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:18 smithi154 conmon[34553]: cluster 2021-07-31T02:40:17.094012+0000 mon.a (mon.0) 2802 : cluster [DBG] osdmap e1792: 8 total, 8 up, 8 in 2021-07-31T02:40:18.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:18 smithi045 conmon[33655]: cluster 2021-07-31T02:40:16.561512+0000 mgr.x (mgr.24542) 2271 : cluster [DBG] 2021-07-31T02:40:18.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:18 smithi045 conmon[33655]: pgmap v3229: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 674 KiB/s rd, 3.7 KiB/s wr, 907 op/s 2021-07-31T02:40:18.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:18 smithi045 conmon[33655]: cluster 2021-07-31T02:40:17.094012+0000 mon.a (mon.0) 2802 : cluster [DBG] osdmap e1792: 8 total, 8 up, 8 in 2021-07-31T02:40:18.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:18 smithi045 conmon[39718]: cluster 2021-07-31T02:40:16.561512+0000 2021-07-31T02:40:18.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:18 smithi045 conmon[39718]: mgr.x (mgr.24542) 2271 : cluster [DBG] pgmap v3229: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 674 KiB/s rd, 3.7 KiB/s wr, 907 op/s 2021-07-31T02:40:18.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:18 smithi045 conmon[39718]: cluster 2021-07-31T02:40:18.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:18 smithi045 conmon[39718]: 2021-07-31T02:40:17.094012+0000 mon.a (mon.0) 2802 : cluster [DBG] 2021-07-31T02:40:18.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:18 smithi045 conmon[39718]: osdmap e1792: 8 total, 8 up, 8 in 2021-07-31T02:40:19.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:19 smithi154 conmon[34553]: cluster 2021-07-31T02:40:18.096305+0000 mon.a (mon.0) 2803 : cluster 2021-07-31T02:40:19.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:19 smithi154 conmon[34553]: [DBG] osdmap e1793: 8 total, 8 up, 8 in 2021-07-31T02:40:19.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:19 smithi045 conmon[33655]: cluster 2021-07-31T02:40:18.096305+0000 mon.a (mon.0) 2803 : cluster [DBG] 2021-07-31T02:40:19.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:19 smithi045 conmon[33655]: osdmap e1793: 8 total, 8 up, 8 in 2021-07-31T02:40:19.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:19 smithi045 conmon[39718]: cluster 2021-07-31T02:40:18.096305+0000 mon.a (mon.0 2021-07-31T02:40:19.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:19 smithi045 conmon[39718]: ) 2803 : cluster [DBG] osdmap e1793: 8 total, 8 up, 8 in 2021-07-31T02:40:20.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:20 smithi154 conmon[34553]: cluster 2021-07-31T02:40:18.562087+0000 mgr.x (mgr.24542) 2272 : cluster 2021-07-31T02:40:20.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:20 smithi154 conmon[34553]: [DBG] pgmap v3232: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:40:20.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:20 smithi154 conmon[34553]: cluster 2021-07-31T02:40:19.106910+0000 mon.a (mon.0) 2804 : cluster [DBG] osdmap e1794: 8 total, 8 up, 8 in 2021-07-31T02:40:20.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:20 smithi045 conmon[33655]: cluster 2021-07-31T02:40: 2021-07-31T02:40:20.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:20 smithi045 conmon[33655]: 18.562087+0000 mgr.x (mgr.24542) 2272 : cluster [DBG] pgmap v3232: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:40:20.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:20 smithi045 conmon[33655]: cluster 2021-07-31T02:40:19.106910+0000 mon.a (mon.0) 2804 : cluster 2021-07-31T02:40:20.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:20 smithi045 conmon[33655]: [DBG] osdmap e1794: 8 total, 8 up, 8 in 2021-07-31T02:40:20.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:20 smithi045 conmon[39718]: cluster 2021-07-31T02:40:18.562087+0000 mgr.x (mgr.24542) 2021-07-31T02:40:20.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:20 smithi045 conmon[39718]: 2272 : cluster [DBG] pgmap v3232: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:40:20.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:20 smithi045 conmon[39718]: cluster 2021-07-31T02:40:19.106910+0000 mon.a (mon.0) 2804 : cluster [DBG] osdmap e1794: 8 total, 8 up, 8 in 2021-07-31T02:40:21.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:21 smithi154 conmon[34553]: cluster 2021-07-31T02:40:20.108949+0000 mon.a (mon.0) 2805 : cluster [DBG] osdmap e1795: 8 total, 8 up, 8 in 2021-07-31T02:40:21.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:21 smithi045 conmon[33655]: cluster 2021-07-31T02:40:20.108949+0000 mon.a (mon 2021-07-31T02:40:21.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:21 smithi045 conmon[33655]: .0) 2805 : cluster [DBG] osdmap e1795: 8 total, 8 up, 8 in 2021-07-31T02:40:21.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:21 smithi045 conmon[39718]: cluster 2021-07-31T02:40:20.108949+0000 mon.a (mon 2021-07-31T02:40:21.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:21 smithi045 conmon[39718]: .0) 2805 : cluster [DBG] osdmap e1795: 8 total, 8 up, 8 in 2021-07-31T02:40:22.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:22 smithi154 conmon[34553]: cluster 2021-07-31T02:40:20.562564+0000 mgr.x (mgr.24542) 2273 : cluster [DBG] 2021-07-31T02:40:22.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:22 smithi154 conmon[34553]: pgmap v3235: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 706 KiB/s rd, 3.7 KiB/s wr, 951 op/s 2021-07-31T02:40:22.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:22 smithi154 conmon[34553]: cluster 2021-07-31T02:40:21.119520+0000 mon.a (mon.0) 2806 : cluster [DBG] osdmap e1796: 8 total, 8 up, 8 in 2021-07-31T02:40:22.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:22 smithi045 conmon[39718]: cluster 2021-07-31T02:40:20.562564+0000 mgr.x (mgr 2021-07-31T02:40:22.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:22 smithi045 conmon[39718]: .24542) 2273 : cluster [DBG] pgmap v3235: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 706 KiB/s rd, 3.7 KiB/s wr, 951 op/s 2021-07-31T02:40:22.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:22 smithi045 conmon[39718]: cluster 2021-07-31T02:40:21.119520+0000 mon.a (mon. 2021-07-31T02:40:22.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:22 smithi045 conmon[39718]: 0) 2806 : cluster [DBG] osdmap e1796: 8 total, 8 up, 8 in 2021-07-31T02:40:22.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:22 smithi045 conmon[33655]: cluster 2021-07-31T02:40:20.562564+0000 mgr.x (mgr.24542) 2273 : cluster [DBG] pgmap v3235: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 706 KiB/s rd, 3.7 KiB/s wr, 951 op/s 2021-07-31T02:40:22.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:22 smithi045 conmon[33655]: 2021-07-31T02:40:22.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:22 smithi045 conmon[33655]: cluster 2021-07-31T02:40:21.119520+0000 mon.a (mon.0) 2806 : cluster [DBG] osdmap e1796: 8 total, 8 up, 8 in 2021-07-31T02:40:23.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:23 smithi154 conmon[34553]: cluster 2021-07-31T02:40:22.121243+0000 mon.a (mon.0) 2807 : 2021-07-31T02:40:23.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:23 smithi154 conmon[34553]: cluster [DBG] osdmap e1797: 8 total, 8 up, 8 in 2021-07-31T02:40:23.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:23 smithi045 conmon[39718]: cluster 2021-07-31T02:40:22.121243+0000 mon.a 2021-07-31T02:40:23.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:23 smithi045 conmon[39718]: (mon.0) 2807 : cluster [DBG] osdmap e1797: 8 total, 8 up, 8 in 2021-07-31T02:40:23.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:23 smithi045 conmon[33655]: cluster 2021-07-31T02:40:22.121243+0000 mon.a (mon.0) 2807 : cluster [DBG] osdmap e1797: 8 total, 8 up, 8 in 2021-07-31T02:40:23.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:23 smithi045 conmon[33655]: 2021-07-31T02:40:24.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:24 smithi154 conmon[34553]: cluster 2021-07-31T02:40:24.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:24 smithi154 conmon[34553]: 2021-07-31T02:40:24.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:24 smithi154 conmon[34553]: 2021-07-31T02:40:22.563214+0000 mgr.x ( 2021-07-31T02:40:24.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:24 smithi154 conmon[34553]: mgr.24542) 2274 : cluster [DBG] pgmap v3238: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 706 KiB/s rd, 3.7 KiB/s wr, 950 op/s 2021-07-31T02:40:24.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:24 smithi154 conmon[34553]: cluster 2021-07-31T02:40:24.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:24 smithi154 conmon[34553]: 2021-07-31T02:40:23.134838+0000 mon.a (mon.0) 2808 : cluster [DBG] osdmap e1798: 8 total, 8 up, 8 in 2021-07-31T02:40:24.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:24 smithi045 conmon[39718]: cluster 2021-07-31T02:40:22.563214+0000 mgr.x (mgr.24542) 2274 : cluster 2021-07-31T02:40:24.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:24 smithi045 conmon[39718]: [DBG] pgmap v3238: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 706 KiB/s rd, 3.7 KiB/s wr, 950 op/s 2021-07-31T02:40:24.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:24 smithi045 conmon[39718]: cluster 2021-07-31T02:40:23.134838+0000 mon.a (mon. 2021-07-31T02:40:24.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:24 smithi045 conmon[39718]: 0) 2808 : cluster [DBG] osdmap e1798: 8 total, 8 up, 8 in 2021-07-31T02:40:24.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:24 smithi045 conmon[33655]: cluster 2021-07-31T02:40:22.563214+0000 mgr.x (mgr.24542) 2274 : cluster 2021-07-31T02:40:24.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:24 smithi045 conmon[33655]: [DBG] pgmap v3238: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 706 KiB/s rd, 3.7 KiB/s wr, 950 op/s 2021-07-31T02:40:24.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:24 smithi045 conmon[33655]: cluster 2021-07-31T02:40:23.134838+0000 mon.a (mon.0) 2808 : cluster [DBG] osdmap e1798: 8 total, 8 up, 8 in 2021-07-31T02:40:25.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:25 smithi154 conmon[34553]: cluster 2021-07-31T02:40:24.136824+0000 mon.a (mon.0) 2809 : cluster [DBG] 2021-07-31T02:40:25.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:25 smithi154 conmon[34553]: osdmap e1799: 8 total, 8 up, 8 in 2021-07-31T02:40:25.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:25 smithi045 conmon[33655]: cluster 2021-07-31T02:40:24.136824+0000 mon.a (mon.0) 2809 : 2021-07-31T02:40:25.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:25 smithi045 conmon[33655]: cluster [DBG] osdmap e1799: 8 total, 8 up, 8 in 2021-07-31T02:40:25.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:25 smithi045 conmon[39718]: cluster 2021-07-31T02:40:24.136824+0000 mon.a (mon.0) 2809 2021-07-31T02:40:25.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:25 smithi045 conmon[39718]: : cluster [DBG] osdmap e1799: 8 total, 8 up, 8 in 2021-07-31T02:40:26.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:26 smithi154 conmon[34553]: cluster 2021-07-31T02:40:24.563738+0000 mgr.x (mgr.24542) 2275 2021-07-31T02:40:26.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:26 smithi154 conmon[34553]: : cluster [DBG] pgmap v3241: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 739 KiB/s rd, 3.7 KiB/s wr, 994 op/s 2021-07-31T02:40:26.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:26 smithi154 conmon[34553]: cluster 2021-07-31T02:40:25.149388+0000 mon.a (mon.0) 2810 : cluster [DBG] osdmap e1800: 8 total, 8 up, 8 in 2021-07-31T02:40:26.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:26 smithi045 conmon[33655]: cluster 2021-07-31T02:40:24.563738+0000 mgr.x (mgr.24542) 2021-07-31T02:40:26.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:26 smithi045 conmon[33655]: 2275 : cluster [DBG] pgmap v3241: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 739 KiB/s rd, 3.7 KiB/s wr, 994 op/s 2021-07-31T02:40:26.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:26 smithi045 conmon[33655]: cluster 2021-07-31T02:40:25.149388+0000 mon.a (mon.0) 2810 : cluster [DBG] osdmap e1800: 8 total, 8 up, 8 in 2021-07-31T02:40:26.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:26 smithi045 conmon[39718]: cluster 2021-07-31T02:40:24.563738+0000 mgr.x 2021-07-31T02:40:26.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:26 smithi045 conmon[39718]: (mgr.24542) 2275 : cluster [DBG] pgmap v3241: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 739 KiB/s rd, 3.7 KiB/s wr, 994 op/s 2021-07-31T02:40:26.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:26 smithi045 conmon[39718]: cluster 2021-07-31T02: 2021-07-31T02:40:26.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:26 smithi045 conmon[39718]: 40:25.149388+0000 mon.a (mon.0) 2810 : cluster [DBG] 2021-07-31T02:40:26.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:26 smithi045 conmon[39718]: osdmap e1800: 8 total, 8 up, 8 in 2021-07-31T02:40:27.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:27 smithi154 conmon[34553]: cluster 2021-07-31T02:40:26.151991+0000 mon.a (mon.0) 2811 : cluster [DBG] osdmap e1801: 8 total, 8 up, 8 in 2021-07-31T02:40:27.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:27 smithi045 conmon[39718]: cluster 2021-07-31T02:40:26.151991+0000 mon.a ( 2021-07-31T02:40:27.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:27 smithi045 conmon[39718]: mon.0) 2811 : cluster [DBG] osdmap e1801: 8 total, 8 up, 8 in 2021-07-31T02:40:27.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:27 smithi045 conmon[33655]: cluster 2021-07-31T02:40:26.151991+0000 mon.a (mon.0) 2021-07-31T02:40:27.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:27 smithi045 conmon[33655]: 2811 : cluster [DBG] osdmap e1801: 8 total, 8 up, 8 in 2021-07-31T02:40:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:28 smithi154 conmon[34553]: cluster 2021-07-31T02:40:26.564189+0000 mgr.x (mgr.24542) 2276 : cluster [DBG] pgmap v3244: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 739 KiB/s rd, 3.7 KiB/s wr, 994 op/s 2021-07-31T02:40:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:28 smithi154 conmon[34553]: cluster 2021-07-31T02:40:27.162337+0000 mon.a (mon.0) 2812 : cluster [DBG] osdmap e1802: 8 total, 8 up, 8 in 2021-07-31T02:40:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:28 smithi154 conmon[34553]: audit 2021-07-31T02:40:28.131587+0000 mon.b (mon.2) 340 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:40:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:28 smithi154 conmon[34553]: audit 2021-07-31T02:40:28.132205+0000 mon.a (mon 2021-07-31T02:40:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:28 smithi154 conmon[34553]: .0) 2813 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:40:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:28 smithi154 conmon[34553]: audit 2021 2021-07-31T02:40:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:28 smithi154 conmon[34553]: -07-31T02:40:28.143948+0000 2021-07-31T02:40:28.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:28 smithi154 conmon[34553]: mon.b (mon.2) 341 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:40:28.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:28 smithi154 conmon[34553]: audit 2021-07-31T02:40:28 2021-07-31T02:40:28.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:28 smithi154 conmon[34553]: .144374+0000 mon.a (mon.0) 2814 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:40:28.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:28 smithi154 conmon[34553]: 2021-07-31T02:40:28.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:28 smithi045 conmon[39718]: cluster 2021-07-31T02:40:26.564189+0000 mgr.x ( 2021-07-31T02:40:28.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:28 smithi045 conmon[39718]: mgr.24542) 2276 : cluster [DBG] pgmap v3244: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 739 KiB/s rd, 3.7 KiB/s wr, 994 op/s 2021-07-31T02:40:28.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:28 smithi045 conmon[39718]: cluster 2021-07-31T02:40:27.162337+0000 2021-07-31T02:40:28.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:28 smithi045 conmon[39718]: mon.a (mon.0) 2812 : cluster [DBG] osdmap e1802: 8 total, 8 up, 8 in 2021-07-31T02:40:28.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:28 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T02:40:28.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:28 smithi045 conmon[39718]: 31T02:40:28.131587+0000 2021-07-31T02:40:28.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:28 smithi045 conmon[39718]: mon.b (mon.2) 340 : audit [INF] 2021-07-31T02:40:28.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:28 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:40:28.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:28 smithi045 conmon[39718]: audit 2021-07-31T02 2021-07-31T02:40:28.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:28 smithi045 conmon[39718]: :40:28.132205+0000 mon.a (mon. 2021-07-31T02:40:28.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:28 smithi045 conmon[39718]: 0) 2813 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:40:28.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:28 smithi045 conmon[39718]: audit 2021- 2021-07-31T02:40:28.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:28 smithi045 conmon[39718]: 07-31T02:40:28.143948 2021-07-31T02:40:28.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:28 smithi045 conmon[39718]: +0000 mon.b (mon.2) 341 : audit 2021-07-31T02:40:28.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:28 smithi045 conmon[39718]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:40:28.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:28 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T02:40:28.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:28 smithi045 conmon[39718]: T02:40:28.144374+0000 mon.a 2021-07-31T02:40:28.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:28 smithi045 conmon[39718]: (mon.0) 2814 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:40:28.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:28 smithi045 conmon[39718]: 2021-07-31T02:40:28.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:28 smithi045 conmon[33655]: cluster 2021-07-31T02:40:26.564189+0000 mgr.x (mgr. 2021-07-31T02:40:28.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:28 smithi045 conmon[33655]: 24542) 2276 : cluster [DBG] pgmap v3244: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 739 KiB/s rd, 3.7 KiB/s wr, 994 op/s 2021-07-31T02:40:28.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:28 smithi045 conmon[33655]: cluster 2021-07-31T02:40:27.162337+0000 mon.a (mon.0) 2812 : cluster [DBG] osdmap e1802: 8 total, 8 up, 8 in 2021-07-31T02:40:28.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:28 smithi045 conmon[33655]: 2021-07-31T02:40:28.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:28 smithi045 conmon[33655]: audit 2021-07- 2021-07-31T02:40:28.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:28 smithi045 conmon[33655]: 31T02:40:28.131587+0000 mon.b (mon.2) 340 : audit 2021-07-31T02:40:28.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:28 smithi045 conmon[33655]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:40:28.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:28 smithi045 conmon[33655]: audit 2021-07- 2021-07-31T02:40:28.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:28 smithi045 conmon[33655]: 31T02:40:28.132205+0000 mon.a (mon.0) 2813 2021-07-31T02:40:28.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:28 smithi045 conmon[33655]: : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:40:28.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:28 smithi045 conmon[33655]: audit 2021-07-31T02:40:28.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:28 smithi045 conmon[33655]: 2021-07-31T02:40:28.143948+0000 2021-07-31T02:40:28.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:28 smithi045 conmon[33655]: mon.b (mon.2) 341 : 2021-07-31T02:40:28.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:28 smithi045 conmon[33655]: audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:40:28.535 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:28 smithi045 conmon[33655]: audit 2021-07 2021-07-31T02:40:28.535 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:28 smithi045 conmon[33655]: -31T02:40:28.144374+0000 mon.a (mon.0 2021-07-31T02:40:28.535 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:28 smithi045 conmon[33655]: ) 2814 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:40:29.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:29 smithi154 conmon[34553]: cluster 2021-07-31T02:40:28.170187+0000 mon.a (mon.0) 2021-07-31T02:40:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:29 smithi154 conmon[34553]: 2815 : cluster [DBG] osdmap e1803: 8 total, 8 up, 8 in 2021-07-31T02:40:29.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:29 smithi045 conmon[39718]: cluster 2021-07-31T02:40:28.170187+0000 mon.a (mon. 2021-07-31T02:40:29.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:29 smithi045 conmon[39718]: 0) 2815 : cluster [DBG] osdmap e1803: 8 total, 8 up, 8 in 2021-07-31T02:40:29.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:29 smithi045 conmon[33655]: cluster 2021-07-31T02:40:28.170187+0000 mon.a (mon.0) 2815 2021-07-31T02:40:29.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:29 smithi045 conmon[33655]: : cluster [DBG] osdmap e1803: 8 total, 8 up, 8 in 2021-07-31T02:40:30.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:30 smithi154 conmon[34553]: cluster 2021-07-31T02:40:28.564750+0000 mgr.x (mgr.24542) 2277 : cluster [DBG] 2021-07-31T02:40:30.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:30 smithi154 conmon[34553]: pgmap v3247: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:40:30.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:30 smithi154 conmon[34553]: cluster 2021-07-31T02:40:29.176099+0000 mon.a (mon.0) 2816 : cluster [DBG] osdmap e1804: 8 total, 8 up, 8 in 2021-07-31T02:40:30.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:30 smithi045 conmon[33655]: cluster 2021-07-31T02:40:28.564750+0000 mgr.x (mgr.24542) 2277 : cluster [DBG] pgmap v3247: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:40:30.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:30 smithi045 conmon[33655]: cluster 2021-07-31T02:40:29.176099+0000 mon.a (mon.0) 2816 : cluster [DBG] osdmap e1804: 8 total, 8 up, 8 in 2021-07-31T02:40:30.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:30 smithi045 conmon[39718]: cluster 2021-07-31T02:40:28.564750+0000 mgr.x (mgr.24542) 2021-07-31T02:40:30.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:30 smithi045 conmon[39718]: 2277 : cluster [DBG] pgmap v3247: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:40:30.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:30 smithi045 conmon[39718]: cluster 2021-07-31T02:40:29.176099+0000 mon.a (mon 2021-07-31T02:40:30.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:30 smithi045 conmon[39718]: .0) 2816 : cluster [DBG] osdmap e1804: 8 total, 8 up, 8 in 2021-07-31T02:40:31.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:31 smithi154 conmon[34553]: cluster 2021-07-31T02:40:30.181477+0000 mon.a (mon.0) 2817 : cluster [DBG] osdmap e1805: 8 total, 8 up, 8 in 2021-07-31T02:40:31.750 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:31 smithi045 conmon[39718]: cluster 2021-07-31T02:40:30.181477+0000 mon.a 2021-07-31T02:40:31.750 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:31 smithi045 conmon[39718]: (mon.0) 2817 : cluster [DBG] osdmap e1805: 8 total, 8 up, 8 in 2021-07-31T02:40:31.751 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:31 smithi045 conmon[33655]: cluster 2021-07-31T02:40:30.181477+0000 mon.a (mon.0) 2817 : cluster [DBG] osdmap e1805: 8 total, 8 up, 8 in 2021-07-31T02:40:31.751 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:31 smithi045 conmon[33655]: 2021-07-31T02:40:32.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:32 smithi154 conmon[34553]: cluster 2021-07-31T02:40:30.565205+0000 mgr.x (mgr.24542) 2278 : cluster [DBG] pgmap v3250: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 771 KiB/s rd, 3.7 KiB/s wr, 1.04k op/s 2021-07-31T02:40:32.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:32 smithi154 conmon[34553]: cluster 2021-07-31T02:40: 2021-07-31T02:40:32.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:32 smithi154 conmon[34553]: 31.188123+0000 mon.a (mon.0) 2818 : cluster [DBG] osdmap e1806: 8 total, 8 up, 8 in 2021-07-31T02:40:32.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:32 smithi045 conmon[39718]: cluster 2021-07-31T02:40:30.565205 2021-07-31T02:40:32.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:32 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 2278 : cluster [DBG] pgmap v3250: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 771 KiB/s rd, 3.7 KiB/s wr, 1.04k op/s 2021-07-31T02:40:32.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:32 smithi045 conmon[39718]: cluster 2021-07-31T02:40:31.188123+0000 mon.a (mon 2021-07-31T02:40:32.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:32 smithi045 conmon[39718]: .0) 2818 : cluster [DBG] osdmap e1806: 8 total, 8 up, 8 in 2021-07-31T02:40:32.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:32 smithi045 conmon[33655]: cluster 2021-07-31T02:40:30.565205+0000 mgr.x (mgr.24542) 2278 : cluster [DBG] pgmap v3250: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 771 KiB/s rd, 3.7 KiB/s wr, 1.04k op/s 2021-07-31T02:40:32.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:32 smithi045 conmon[33655]: cluster 2021-07-31T02:40:31.188123+0000 mon.a (mon.0) 2818 : cluster [DBG] osdmap e1806: 8 total, 8 up, 8 in 2021-07-31T02:40:33.077 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:40:33.359 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:40:33.359 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 55m ago 56m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:40:33.359 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 53m ago 55m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:40:33.360 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 53m ago 59m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:40:33.360 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 55m ago 61m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:40:33.360 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 55m ago 61m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:40:33.360 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 53m ago 59m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:40:33.360 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 55m ago 59m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:40:33.361 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 55m ago 56m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:40:33.361 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 53m ago 56m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:40:33.361 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 55m ago 59m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:40:33.361 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 55m ago 58m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:40:33.362 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 55m ago 58m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:40:33.362 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 55m ago 58m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:40:33.362 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 53m ago 57m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:40:33.362 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 53m ago 57m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:40:33.362 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 53m ago 57m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:40:33.363 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 53m ago 56m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:40:33.363 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 53m ago 56m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:40:33.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[34553]: cluster 2021-07-31T02:40:32.194619+0000 mon.a (mon. 2021-07-31T02:40:33.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[34553]: 0) 2819 : cluster [DBG] osdmap e1807: 8 total, 8 up, 8 in 2021-07-31T02:40:33.503 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: debug 2021-07-31T02:40:33.247+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:40:33.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:40:33.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:40:33.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: yield 2021-07-31T02:40:33.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:40:33.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:40:33.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:40:33.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:40:33.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:40:33.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:40:33.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:40:33.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:40:33.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:40:33.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: task.result() 2021-07-31T02:40:33.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:40:33.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: debug 2021-07-31T02:40:33.258+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:40:33.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:40:33.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:40:33.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: yield 2021-07-31T02:40:33.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:40:33.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:40:33.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:40:33.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:40:33.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:40:33.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:40:33.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:40:33.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:40:33.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:40:33.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: task.result() 2021-07-31T02:40:33.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:40:33.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: debug 2021-07-31T02:40:33.283+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:40:33.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:40:33.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:40:33.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: yield 2021-07-31T02:40:33.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:40:33.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:40:33.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:40:33.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:40:33.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:40:33.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:40:33.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:40:33.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:40:33.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:40:33.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: task.result() 2021-07-31T02:40:33.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:40:33.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: debug 2021-07-31T02:40:33.284+0000 7fcb9095b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:40:33.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:40:33.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:40:33.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: yield 2021-07-31T02:40:33.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:40:33.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:40:33.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:40:33.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:40:33.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:40:33.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:40:33.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:40:33.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:40:33.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:40:33.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: task.result() 2021-07-31T02:40:33.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:40:33.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: 2021-07-31T02:40:33.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:40:33.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: 2021-07-31T02:40:33.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:40:33.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:40:33.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:40:33.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:40:33.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:40:33.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:40:33.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:40:33.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:40:33.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:40:33.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:40:33.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:40:33.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:40:33.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:40:33.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:40:33.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:40:33.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:40:33.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:40:33.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:40:33.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: raise 2021-07-31T02:40:33.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:40:33.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:40:33.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:40:33.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:40:33.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:40:33.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: debug 2021-07-31T02:40:33.292+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:40:33.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:40:33.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:40:33.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: yield 2021-07-31T02:40:33.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:40:33.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:40:33.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:40:33.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:40:33.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:40:33.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:40:33.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:40:33.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:40:33.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:40:33.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: task.result() 2021-07-31T02:40:33.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:40:33.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: debug 2021-07-31T02:40:33.292+0000 7fcb9115c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:40:33.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:40:33.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:40:33.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: yield 2021-07-31T02:40:33.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:40:33.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:40:33.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:40:33.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:40:33.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:40:33.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:40:33.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:40:33.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:40:33.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:40:33.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: task.result() 2021-07-31T02:40:33.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:40:33.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: 2021-07-31T02:40:33.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:40:33.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: 2021-07-31T02:40:33.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:40:33.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:40:33.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:40:33.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:40:33.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:40:33.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:40:33.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:40:33.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:40:33.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:40:33.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:40:33.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:40:33.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:40:33.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:40:33.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:40:33.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:40:33.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:40:33.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:40:33.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:40:33.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: raise 2021-07-31T02:40:33.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:40:33.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:40:33.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:40:33.535 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:40:33.535 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:40:33 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:40:33.535 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:33 smithi045 conmon[33655]: cluster 2021-07-31T02:40:32.194619+0000 mon.a (mon.0) 2819 : cluster [DBG] osdmap e1807: 8 total, 8 up, 8 in 2021-07-31T02:40:33.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:33 smithi045 conmon[39718]: cluster 2021-07-31T02:40: 2021-07-31T02:40:33.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:33 smithi045 conmon[39718]: 32.194619+0000 mon.a (mon.0) 2819 : cluster [DBG] osdmap e1807: 8 total, 8 up, 8 in 2021-07-31T02:40:33.696 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:40:33.697 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:40:33.697 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:40:33.697 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:40:33.698 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:40:33.698 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:40:33.698 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:40:33.698 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:40:33.699 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:40:33.699 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:40:33.699 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:40:33.699 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:40:33.700 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:40:33.700 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:40:33.700 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:40:33.701 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:40:33.701 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:40:34.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: cluster 2021-07-31T02:40:32.565730+0000 mgr.x (mgr.24542) 2279 2021-07-31T02:40:34.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: : cluster [DBG] pgmap v3253: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 771 KiB/s rd, 3.7 KiB/s wr, 1.04k op/s 2021-07-31T02:40:34.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: audit 2021-07-31T02:40:33.065743+0000 mgr.x (mgr.24542) 2280 : audit [DBG] from='client.27766 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:40:34.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: cluster 2021-07-31T02:40:34.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: 2021-07-31T02:40:33.194936+0000 mon.a (mon.0) 2820 : cluster [DBG] osdmap e1808: 8 total, 8 up, 8 in 2021-07-31T02:40:34.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: audit 2021-07-31T02: 2021-07-31T02:40:34.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: 40:33.200150+0000 mon.b (mon.2) 342 : 2021-07-31T02:40:34.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:40:34.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: audit 2021-07 2021-07-31T02:40:34.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: -31T02:40:33.202186+0000 mon.b (mon. 2021-07-31T02:40:34.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: 2) 343 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:40:34.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: audit 2021-07-31T02:40:34.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: 2021-07-31T02:40:33.203152+0000 mon.b 2021-07-31T02:40:34.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: (mon.2) 344 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:40:34.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: cephadm 2021-07-31T02:40: 2021-07-31T02:40:34.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: 33.203932+0000 mgr.x (mgr.24542) 2281 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:40:34.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: cephadm 2021-07 2021-07-31T02:40:34.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: -31T02:40:33.248073+0000 mgr.x (mgr.24542) 2282 : cephadm [ERR] 2021-07-31T02:40:34.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: 2021-07-31T02:40:34.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:40:34.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:40:34.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: yield 2021-07-31T02:40:34.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:40:34.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:40:34.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:40:34.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:40:34.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:40:34.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:40:34.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:40:34.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:40:34.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:40:34.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: task.result() 2021-07-31T02:40:34.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:40:34.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: cephadm 2021-07-31T02:40:33.248442+0000 mgr.x (mgr.24542) 2283 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:40:34.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: cephadm 2021-07-31T02:40:33.259508+0000 mgr.x (mgr. 2021-07-31T02:40:34.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: 24542) 2284 : cephadm [ERR] 2021-07-31T02:40:34.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:40:34.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:40:34.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: yield 2021-07-31T02:40:34.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:40:34.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:40:34.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:40:34.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:40:34.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:40:34.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:40:34.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:40:34.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:40:34.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:40:34.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: task.result() 2021-07-31T02:40:34.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:40:34.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: cephadm 2021-07-31T02:40:33.260191+0000 mgr.x (mgr.24542) 2285 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:40:34.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: cephadm 2021-07-31T02:40:33.284669+0000 mgr.x (mgr.24542) 2286 : cephadm 2021-07-31T02:40:34.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: [ERR] 2021-07-31T02:40:34.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:40:34.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:40:34.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: yield 2021-07-31T02:40:34.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:40:34.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:40:34.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:40:34.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:40:34.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:40:34.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:40:34.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:40:34.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:40:34.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:40:34.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: task.result() 2021-07-31T02:40:34.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:40:34.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: cephadm 2021-07-31T02:40:33.285061+0000 mgr.x (mgr.24542) 2287 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:40:34.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:40:34.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:40:34.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: yield 2021-07-31T02:40:34.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:40:34.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:40:34.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:40:34.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:40:34.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:40:34.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:40:34.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:40:34.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:40:34.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:40:34.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: task.result() 2021-07-31T02:40:34.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:40:34.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: 2021-07-31T02:40:34.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:40:34.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: 2021-07-31T02:40:34.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:40:34.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:40:34.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:40:34.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:40:34.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:40:34.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:40:34.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:40:34.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:40:34.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:40:34.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:40:34.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:40:34.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:40:34.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:40:34.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:40:34.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:40:34.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:40:34.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:40:34.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:40:34.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: raise 2021-07-31T02:40:34.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:40:34.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:40:34.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:40:34.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:40:34.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:40:34.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: cephadm 2021-07-31T02:40:33.293491+0000 mgr.x (mgr.24542) 2288 : cephadm [ERR] 2021-07-31T02:40:34.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:40:34.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:40:34.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: yield 2021-07-31T02:40:34.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:40:34.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:40:34.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:40:34.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:40:34.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: cluster 2021-07-31T02:40:32.565730+0000 mgr.x (mgr.24542) 2021-07-31T02:40:34.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: 2279 : cluster [DBG] pgmap v3253: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 771 KiB/s rd, 3.7 KiB/s wr, 1.04k op/s 2021-07-31T02:40:34.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: audit 2021-07-31T02:40:33.065743+0000 mgr.x (mgr.24542) 2280 : audit [DBG] from='client.27766 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:40:34.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: cluster 2021-07-31T02:40:33.194936+0000 mon.a (mon.0) 2820 : cluster [DBG] osdmap e1808: 8 total, 8 up, 8 in 2021-07-31T02:40:34.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: audit 2021-07-31T02:40:33.200150+0000 mon.b (mon.2) 342 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:40:34.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: audit 2021-07-31T02:40:33.202186+0000 mon.b (mon.2) 343 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:40:34.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: audit 2021-07-31T02:40:33.203152+0000 mon.b (mon.2) 344 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:40:34.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: cephadm 2021-07-31T02:40:33.203932+0000 mgr.x (mgr.24542) 2281 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:40:34.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: cephadm 2021-07-31T02:40:33.248073+0000 mgr.x (mgr.24542) 2282 : cephadm [ERR] 2021-07-31T02:40:34.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:40:34.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:40:34.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: yield 2021-07-31T02:40:34.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:40:34.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:40:34.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:40:34.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:40:34.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:40:34.535 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:40:34.535 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:40:34.535 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:40:34.535 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:40:34.535 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: task.result() 2021-07-31T02:40:34.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:40:34.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: cephadm 2021-07-31T02:40:33.248442+0000 mgr.x (mgr.24542) 2283 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:40:34.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: cephadm 2021-07-31T02:40:33.259508+0000 mgr.x (mgr.24542) 2284 : cephadm [ERR] 2021-07-31T02:40:34.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:40:34.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:40:34.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: yield 2021-07-31T02:40:34.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:40:34.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:40:34.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:40:34.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:40:34.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:40:34.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:40:34.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:40:34.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: task.result() 2021-07-31T02:40:34.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:40:34.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: 2021-07-31T02:40:34.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: cephadm 2021-07-31T02:40:33.293875+0000 mgr.x (mgr.24542) 2289 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:40:34.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:40:34.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:40:34.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: yield 2021-07-31T02:40:34.541 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:40:34.541 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:40:34.541 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:40:34.541 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:40:34.542 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:40:34.542 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:40:34.542 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:40:34.543 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:40:34.543 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:40:34.544 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: task.result() 2021-07-31T02:40:34.544 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:40:34.544 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: 2021-07-31T02:40:34.544 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:40:34.544 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: 2021-07-31T02:40:34.545 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:40:34.545 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:40:34.545 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:40:34.545 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:40:34.546 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:40:34.546 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:40:34.546 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:40:34.546 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:40:34.546 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:40:34.547 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:40:34.547 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:40:34.547 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:40:34.547 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:40:34.548 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:40:34.548 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:40:34.548 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:40:34.548 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:40:34.548 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:40:34.549 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: raise 2021-07-31T02:40:34.549 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:40:34.549 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:40:34.549 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:40:34.549 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:40:34.550 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:40:34.550 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: audit 2021-07-31T02:40:33.351884+0000 mgr.x (mgr.24542) 2290 : audit [DBG] from='client.27772 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:40:34.550 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:34 smithi154 conmon[34553]: audit 2021-07-31T02:40:33.696490+0000 mon.c (mon.1) 126 : audit [DBG] from='client.? 172.21.15.45:0/917905293' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:40:34.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: cluster 2021-07-31T02:40:32.565730+0000 mgr.x (mgr.24542) 2279 : cluster 2021-07-31T02:40:34.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: [DBG] pgmap v3253: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 771 KiB/s rd, 3.7 KiB/s wr, 1.04k op/s 2021-07-31T02:40:34.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: audit 2021-07-31T02:40:33.065743+0000 mgr.x (mgr.24542) 2280 : audit [DBG] from='client.27766 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:40:34.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: cluster 2021-07-31T02:40:33.194936+0000 mon.a (mon.0) 2820 : cluster [DBG] 2021-07-31T02:40:34.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: osdmap e1808: 8 total, 8 up, 8 in 2021-07-31T02:40:34.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: audit 2021-07 2021-07-31T02:40:34.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: -31T02:40:33.200150+0000 mon.b (mon.2) 342 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:40:34.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: audit 2021-07-31T02:40:33. 2021-07-31T02:40:34.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: 202186+0000 mon.b (mon.2) 343 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:40:34.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: audit 2021-07-31T02:40:33. 2021-07-31T02:40:34.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: 203152+0000 mon.b (mon.2) 344 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:40:34.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: cephadm 2021-07-31T02: 2021-07-31T02:40:34.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: 40:33.203932+0000 mgr.x (mgr.24542) 2281 : cephadm [WRN] 2021-07-31T02:40:34.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:40:34.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: cephadm 2021-07- 2021-07-31T02:40:34.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: 31T02:40:33.248073+0000 mgr.x (mgr.24542) 2282 : cephadm [ERR] 2021-07-31T02:40:34.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:40:34.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:40:34.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: yield 2021-07-31T02:40:34.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:40:34.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:40:34.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:40:34.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:40:34.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:40:34.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:40:34.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:40:34.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:40:34.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:40:34.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: task.result() 2021-07-31T02:40:34.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:40:34.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: cephadm 2021-07-31T02:40:33.248442+0000 mgr.x (mgr.24542) 2283 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:40:34.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: cephadm 2021-07-31T02:40:33.259508+0000 mgr.x (mgr.24542) 2284 : cephadm [ERR] 2021-07-31T02:40:34.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:40:34.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:40:34.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: yield 2021-07-31T02:40:34.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:40:34.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:40:34.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:40:34.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:40:34.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:40:34.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:40:34.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:40:34.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:40:34.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:40:34.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: task.result() 2021-07-31T02:40:34.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:40:34.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: cephadm 2021-07-31T02:40:33.260191+0000 mgr.x (mgr.24542) 2285 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:40:34.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: cephadm 2021-07-31T02:40:33.284669+0000 mgr.x (mgr.24542) 2286 : cephadm [ERR] 2021-07-31T02:40:34.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:40:34.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:40:34.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: yield 2021-07-31T02:40:34.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:40:34.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:40:34.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:40:34.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:40:34.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:40:34.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:40:34.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:40:34.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:40:34.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:40:34.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: task.result() 2021-07-31T02:40:34.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:40:34.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: cephadm 2021-07-31T02:40:33.285061+0000 mgr.x (mgr.24542) 2287 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:40:34.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:40:34.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:40:34.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: yield 2021-07-31T02:40:34.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:40:34.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:40:34.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:40:34.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:40:34.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:40:34.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:40:34.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:40:34.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:40:34.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:40:34.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:40:34.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:40:34.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:40:34.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:40:34.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:40:34.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: task.result() 2021-07-31T02:40:34.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:40:34.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: cephadm 2021-07-31T02:40:33.260191+0000 mgr.x (mgr.24542) 2285 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:40:34.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: cephadm 2021-07-31T02:40:33.284669+0000 mgr.x (mgr.24542) 2286 : cephadm [ERR] 2021-07-31T02:40:34.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:40:34.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:40:34.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: yield 2021-07-31T02:40:34.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:40:34.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:40:34.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:40:34.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:40:34.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:40:34.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:40:34.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:40:34.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:40:34.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:40:34.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: task.result() 2021-07-31T02:40:34.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:40:34.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: cephadm 2021-07-31T02:40:33.285061+0000 mgr.x (mgr.24542) 2287 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:40:34.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:40:34.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:40:34.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: yield 2021-07-31T02:40:34.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:40:34.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:40:34.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:40:34.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:40:34.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:40:34.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:40:34.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:40:34.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:40:34.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:40:34.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: task.result() 2021-07-31T02:40:34.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:40:34.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:40:34.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:40:34.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: task.result() 2021-07-31T02:40:34.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:40:34.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: 2021-07-31T02:40:34.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:40:34.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: 2021-07-31T02:40:34.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:40:34.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:40:34.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:40:34.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:40:34.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:40:34.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:40:34.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:40:34.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:40:34.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:40:34.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:40:34.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:40:34.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:40:34.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:40:34.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:40:34.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:40:34.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:40:34.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:40:34.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:40:34.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: raise 2021-07-31T02:40:34.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:40:34.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:40:34.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:40:34.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:40:34.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:40:34.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: cephadm 2021-07-31T02:40:33.293491+0000 mgr.x (mgr.24542) 2288 : cephadm [ERR] 2021-07-31T02:40:34.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:40:34.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:40:34.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: yield 2021-07-31T02:40:34.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:40:34.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:40:34.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:40:34.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:40:34.585 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:40:34.585 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:40:34.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: 2021-07-31T02:40:34.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:40:34.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: 2021-07-31T02:40:34.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:40:34.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:40:34.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:40:34.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:40:34.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:40:34.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:40:34.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:40:34.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:40:34.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:40:34.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:40:34.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:40:34.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:40:34.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:40:34.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:40:34.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:40:34.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:40:34.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:40:34.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:40:34.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: raise 2021-07-31T02:40:34.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:40:34.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:40:34.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:40:34.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:40:34.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:40:34.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: cephadm 2021-07-31T02:40:33.293491+0000 mgr.x (mgr.24542) 2288 : cephadm [ERR] 2021-07-31T02:40:34.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:40:34.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:40:34.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: yield 2021-07-31T02:40:34.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:40:34.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:40:34.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:40:34.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:40:34.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:40:34.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:40:34.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:40:34.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:40:34.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:40:34.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: task.result() 2021-07-31T02:40:34.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:40:34.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:40:34.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:40:34.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: task.result() 2021-07-31T02:40:34.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:40:34.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: cephadm 2021-07-31T02:40:33.293875+0000 mgr.x (mgr.24542) 2289 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:40:34.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:40:34.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:40:34.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: yield 2021-07-31T02:40:34.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:40:34.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:40:34.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:40:34.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:40:34.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:40:34.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:40:34.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:40:34.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:40:34.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:40:34.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: task.result() 2021-07-31T02:40:34.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:40:34.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: 2021-07-31T02:40:34.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:40:34.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: 2021-07-31T02:40:34.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:40:34.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:40:34.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:40:34.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:40:34.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:40:34.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:40:34.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:40:34.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:40:34.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:40:34.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:40:34.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:40:34.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:40:34.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:40:34.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:40:34.603 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:40:34.603 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:40:34.603 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:40:34.603 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:40:34.603 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: raise 2021-07-31T02:40:34.604 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:40:34.604 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:40:34.604 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:40:34.604 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:40:34.605 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:40:34.605 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: audit 2021-07-31T02:40:33.351884+0000 mgr.x (mgr.24542) 2290 : audit [DBG] from='client.27772 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:40:34.605 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[39718]: audit 2021-07-31T02:40:33.696490+0000 mon.c (mon.1) 126 : audit [DBG] from='client.? 172.21.15.45:0/917905293' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:40:34.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:40:34.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: cephadm 2021-07-31T02:40:33.293875+0000 mgr.x (mgr.24542) 2289 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:40:34.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:40:34.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:40:34.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: yield 2021-07-31T02:40:34.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:40:34.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:40:34.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:40:34.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:40:34.608 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:40:34.608 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:40:34.608 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:40:34.608 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:40:34.609 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:40:34.609 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: task.result() 2021-07-31T02:40:34.609 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:40:34.609 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: 2021-07-31T02:40:34.609 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:40:34.610 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: 2021-07-31T02:40:34.610 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:40:34.610 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:40:34.610 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:40:34.611 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:40:34.611 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:40:34.611 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:40:34.611 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:40:34.611 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:40:34.612 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:40:34.612 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:40:34.612 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:40:34.612 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:40:34.613 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:40:34.613 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:40:34.613 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:40:34.613 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:40:34.613 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:40:34.614 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:40:34.614 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: raise 2021-07-31T02:40:34.614 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:40:34.614 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:40:34.614 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:40:34.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:40:34.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:40:34.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: 2021-07-31T02:40:34.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: audit 2021-07-31T02:40:33.351884+0000 mgr.x (mgr.24542) 2290 : audit [DBG] from='client.27772 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:40:34.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:34 smithi045 conmon[33655]: audit 2021-07-31T02:40:33.696490+0000 mon.c (mon.1) 126 : audit [DBG] from='client.? 172.21.15.45:0/917905293' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:40:35.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:35 smithi154 conmon[34553]: cluster 2021-07-31T02:40:34.206880+0000 mon.a (mon.0) 2821 : cluster [DBG] 2021-07-31T02:40:35.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:35 smithi154 conmon[34553]: osdmap e1809: 8 total, 8 up, 8 in 2021-07-31T02:40:35.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:35 smithi045 conmon[39718]: cluster 2021-07-31T02:40:34.206880+0000 mon.a (mon. 2021-07-31T02:40:35.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:35 smithi045 conmon[39718]: 0) 2821 : cluster [DBG] osdmap e1809: 8 total, 8 up, 8 in 2021-07-31T02:40:35.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:35 smithi045 conmon[33655]: cluster 2021-07-31T02:40:34.206880+0000 mon.a (mon.0) 2821 2021-07-31T02:40:35.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:35 smithi045 conmon[33655]: : cluster [DBG] osdmap e1809: 8 total, 8 up, 8 in 2021-07-31T02:40:36.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:36 smithi154 conmon[34553]: cluster 2021-07-31T02:40:34.566312+0000 mgr.x (mgr.24542) 2291 : cluster 2021-07-31T02:40:36.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:36 smithi154 conmon[34553]: [DBG] pgmap v3256: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 804 KiB/s rd, 3.7 KiB/s wr, 1.08k op/s 2021-07-31T02:40:36.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:36 smithi154 conmon[34553]: cluster 2021-07-31T02:40:35.213595+0000 mon.a (mon.0) 2822 : cluster [DBG] osdmap e1810: 8 total, 8 up, 8 in 2021-07-31T02:40:36.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:36 smithi045 conmon[39718]: cluster 2021-07-31T02:40: 2021-07-31T02:40:36.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:36 smithi045 conmon[39718]: 34.566312+0000 mgr.x (mgr.24542) 2291 : cluster [DBG] pgmap v3256: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 804 KiB/s rd, 3.7 KiB/s wr, 1.08k op/s 2021-07-31T02:40:36.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:36 smithi045 conmon[39718]: cluster 2021-07-31T02:40:36.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:36 smithi045 conmon[39718]: 2021-07-31T02:40: 2021-07-31T02:40:36.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:36 smithi045 conmon[39718]: 35.213595+0000 mon.a (mon 2021-07-31T02:40:36.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:36 smithi045 conmon[39718]: .0) 2822 : cluster [DBG] 2021-07-31T02:40:36.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:36 smithi045 conmon[39718]: osdmap e1810: 8 total, 8 up, 8 in 2021-07-31T02:40:36.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:36 smithi045 conmon[33655]: cluster 2021-07-31T02:40:34.566312+0000 2021-07-31T02:40:36.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:36 smithi045 conmon[33655]: mgr.x (mgr.24542) 2291 : cluster [DBG] pgmap v3256: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 804 KiB/s rd, 3.7 KiB/s wr, 1.08k op/s 2021-07-31T02:40:36.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:36 smithi045 conmon[33655]: cluster 2021-07-31T02:40:35.213595+0000 mon.a (mon.0) 2822 2021-07-31T02:40:36.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:36 smithi045 conmon[33655]: : cluster [DBG] osdmap e1810: 8 total, 8 up, 8 in 2021-07-31T02:40:37.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:37 smithi154 conmon[34553]: cluster 2021-07-31T02:40:36.215561+0000 mon.a (mon.0) 2823 : 2021-07-31T02:40:37.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:37 smithi154 conmon[34553]: cluster [DBG] osdmap e1811: 8 total, 8 up, 8 in 2021-07-31T02:40:37.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:37 smithi045 conmon[39718]: cluster 2021-07-31T02:40:36.215561+0000 mon.a (mon.0) 2823 : cluster 2021-07-31T02:40:37.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:37 smithi045 conmon[39718]: [DBG] osdmap e1811: 8 total, 8 up, 8 in 2021-07-31T02:40:37.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:37 smithi045 conmon[33655]: cluster 2021-07-31T02:40:36.215561+0000 mon.a 2021-07-31T02:40:37.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:37 smithi045 conmon[33655]: (mon.0) 2823 : cluster [DBG] osdmap e1811: 8 total, 8 up, 8 in 2021-07-31T02:40:38.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:38 smithi154 conmon[34553]: cluster 2021-07-31T02:40:38.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:38 smithi154 conmon[34553]: 2021-07-31T02:40:36.566745+0000 2021-07-31T02:40:38.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:38 smithi154 conmon[34553]: mgr.x (mgr.24542) 2292 : cluster [DBG] pgmap v3259: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 804 KiB/s rd, 3.7 KiB/s wr, 1.08k op/s 2021-07-31T02:40:38.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:38 smithi154 conmon[34553]: cluster 2021-07-31T02 2021-07-31T02:40:38.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:38 smithi154 conmon[34553]: :40:37.225533+0000 mon.a (mon.0) 2824 : cluster [DBG] osdmap e1812: 8 total, 8 up, 8 in 2021-07-31T02:40:38.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:38 smithi045 conmon[33655]: cluster 2021-07-31T02:40:36.566745+0000 mgr.x (mgr.24542 2021-07-31T02:40:38.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:38 smithi045 conmon[33655]: ) 2292 : cluster [DBG] pgmap v3259: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 804 KiB/s rd, 3.7 KiB/s wr, 1.08k op/s 2021-07-31T02:40:38.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:38 smithi045 conmon[33655]: cluster 2021-07-31T02:40:37.225533+0000 mon.a (mon.0) 2824 : cluster [DBG] osdmap e1812: 8 total, 8 up, 8 in 2021-07-31T02:40:38.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:38 smithi045 conmon[39718]: cluster 2021-07-31T02:40:36.566745+0000 mgr.x (mgr.24542) 2292 : cluster 2021-07-31T02:40:38.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:38 smithi045 conmon[39718]: [DBG] pgmap v3259: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 804 KiB/s rd, 3.7 KiB/s wr, 1.08k op/s 2021-07-31T02:40:38.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:38 smithi045 conmon[39718]: cluster 2021-07-31T02:40:37.225533+0000 mon.a (mon.0) 2824 : cluster [DBG] osdmap e1812: 8 total, 8 up, 8 in 2021-07-31T02:40:39.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:39 smithi154 conmon[34553]: cluster 2021-07-31T02:40:38.226925+0000 mon.a (mon.0) 2825 : cluster [DBG] osdmap e1813: 8 total, 8 up, 8 in 2021-07-31T02:40:39.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:39 smithi045 conmon[33655]: cluster 2021-07-31T02:40:38.226925+0000 mon.a (mon.0) 2825 : cluster [DBG] osdmap e1813: 8 total, 8 up, 8 in 2021-07-31T02:40:39.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:39 smithi045 conmon[39718]: cluster 2021-07-31T02:40:38.226925+0000 mon.a (mon.0) 2825 : 2021-07-31T02:40:39.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:39 smithi045 conmon[39718]: cluster [DBG] osdmap e1813: 8 total, 8 up, 8 in 2021-07-31T02:40:40.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:40 smithi154 conmon[34553]: cluster 2021-07-31T02:40:38.567202+0000 mgr.x (mgr.24542) 2293 : cluster [DBG] pgmap v3262: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:40:40.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:40 smithi154 conmon[34553]: cluster 2021-07-31T02:40:39.238384+0000 mon.a (mon.0) 2826 : cluster [DBG] osdmap e1814: 8 total, 8 up, 8 in 2021-07-31T02:40:40.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:40 smithi045 conmon[39718]: cluster 2021-07-31T02:40:38.567202+0000 mgr.x ( 2021-07-31T02:40:40.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:40 smithi045 conmon[39718]: mgr.24542) 2293 : cluster [DBG] pgmap v3262: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:40:40.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:40 smithi045 conmon[39718]: cluster 2021-07-31T02:40:39.238384 2021-07-31T02:40:40.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:40 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2826 : cluster [DBG] osdmap e1814: 8 total, 8 up, 8 in 2021-07-31T02:40:40.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:40 smithi045 conmon[33655]: cluster 2021-07-31T02:40:38.567202+0000 mgr.x (mgr.24542) 2293 : cluster [DBG] 2021-07-31T02:40:40.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:40 smithi045 conmon[33655]: pgmap v3262: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:40:40.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:40 smithi045 conmon[33655]: cluster 2021-07-31T02:40:39.238384+0000 mon.a (mon.0) 2826 : cluster [DBG] osdmap e1814: 8 total, 8 up, 8 in 2021-07-31T02:40:41.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:41 smithi154 conmon[34553]: cluster 2021-07-31T02:40:40.244028+0000 mon.a (mon.0) 2827 : cluster [DBG] osdmap e1815: 8 total, 8 up, 8 in 2021-07-31T02:40:41.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:41 smithi045 conmon[33655]: cluster 2021-07-31T02:40:40.244028+0000 mon.a (mon.0) 2021-07-31T02:40:41.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:41 smithi045 conmon[33655]: 2827 : cluster [DBG] osdmap e1815: 8 total, 8 up, 8 in 2021-07-31T02:40:41.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:41 smithi045 conmon[39718]: cluster 2021-07-31T02:40:40.244028+0000 mon.a (mon 2021-07-31T02:40:41.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:41 smithi045 conmon[39718]: .0) 2827 : cluster [DBG] osdmap e1815: 8 total, 8 up, 8 in 2021-07-31T02:40:42.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:42 smithi045 conmon[33655]: cluster 2021-07-31T02:40:40.567770+0000 mgr.x (mgr.24542) 2294 : cluster [DBG] pgmap v3265: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 836 KiB/s rd, 3.7 KiB/s wr, 1.12k op/s 2021-07-31T02:40:42.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:42 smithi045 conmon[33655]: cluster 2021-07-31T02:40:41.250198+0000 mon.a (mon.0) 2828 : cluster [DBG] osdmap e1816: 8 total, 8 up, 8 in 2021-07-31T02:40:42.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:42 smithi045 conmon[39718]: cluster 2021-07-31T02:40:40 2021-07-31T02:40:42.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:42 smithi045 conmon[39718]: .567770+0000 mgr.x (mgr.24542) 2294 : cluster [DBG] pgmap v3265: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 836 KiB/s rd, 3.7 KiB/s wr, 1.12k op/s 2021-07-31T02:40:42.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:42 smithi045 conmon[39718]: cluster 2021-07- 2021-07-31T02:40:42.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:42 smithi045 conmon[39718]: 31T02:40:41.250198 2021-07-31T02:40:42.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:42 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2021-07-31T02:40:42.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:42 smithi045 conmon[39718]: 2828 : cluster [DBG] osdmap e1816: 8 total, 8 up, 8 in 2021-07-31T02:40:42.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:42 smithi154 conmon[34553]: cluster 2021-07-31T02:40:40.567770+0000 mgr.x (mgr.24542) 2294 : cluster [DBG] pgmap v3265: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 836 KiB/s rd, 3.7 KiB/s wr, 1.12k op/s 2021-07-31T02:40:42.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:42 smithi154 conmon[34553]: 2021-07-31T02:40:42.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:42 smithi154 conmon[34553]: cluster 2021-07-31T02:40:41.250198+0000 mon.a (mon.0) 2828 : cluster [DBG] osdmap e1816: 8 total, 8 up, 8 in 2021-07-31T02:40:43.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:43 smithi045 conmon[33655]: cluster 2021-07-31T02:40:42.251722+0000 mon.a (mon.0) 2829 : cluster [DBG] osdmap e1817: 8 total, 8 up, 8 in 2021-07-31T02:40:43.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:43 smithi045 conmon[39718]: cluster 2021-07-31T02:40:42.251722 2021-07-31T02:40:43.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:43 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2829 : cluster [DBG] osdmap e1817: 8 total, 8 up, 8 in 2021-07-31T02:40:43.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:43 smithi154 conmon[34553]: cluster 2021-07-31T02:40:42.251722+0000 mon.a (mon.0) 2829 : cluster [DBG] osdmap e1817: 8 total, 8 up, 8 in 2021-07-31T02:40:44.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:44 smithi045 conmon[39718]: cluster 2021-07-31T02:40: 2021-07-31T02:40:44.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:44 smithi045 conmon[39718]: 42.568256+0000 mgr.x (mgr.24542) 2295 : cluster [DBG] pgmap v3268: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 836 KiB/s rd, 3.7 KiB/s wr, 1.12k op/s 2021-07-31T02:40:44.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:44 smithi045 conmon[39718]: cluster 2021 2021-07-31T02:40:44.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:44 smithi045 conmon[39718]: -07-31T02:40:43. 2021-07-31T02:40:44.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:44 smithi045 conmon[39718]: 258881+0000 mon.a (mon. 2021-07-31T02:40:44.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:44 smithi045 conmon[39718]: 0) 2830 : cluster [DBG] osdmap e1818: 8 total, 8 up, 8 in 2021-07-31T02:40:44.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:44 smithi045 conmon[39718]: 2021-07-31T02:40:44.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:44 smithi045 conmon[33655]: cluster 2021-07-31T02:40:42.568256+0000 mgr.x (mgr.24542) 2295 2021-07-31T02:40:44.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:44 smithi045 conmon[33655]: : cluster [DBG] pgmap v3268: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 836 KiB/s rd, 3.7 KiB/s wr, 1.12k op/s 2021-07-31T02:40:44.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:44 smithi045 conmon[33655]: cluster 2021-07-31T02:40:43.258881+0000 mon.a (mon.0) 2830 : cluster [DBG] osdmap e1818: 8 total, 8 up, 8 in 2021-07-31T02:40:44.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:44 smithi154 conmon[34553]: cluster 2021-07-31T02:40:42.568256+0000 mgr.x (mgr.24542) 2295 : cluster [DBG] 2021-07-31T02:40:44.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:44 smithi154 conmon[34553]: pgmap v3268: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 836 KiB/s rd, 3.7 KiB/s wr, 1.12k op/s 2021-07-31T02:40:44.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:44 smithi154 conmon[34553]: cluster 2021-07-31T02:40:43.258881+0000 mon.a (mon.0) 2830 : cluster [DBG] osdmap e1818: 8 total, 8 up, 8 in 2021-07-31T02:40:45.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:45 smithi045 conmon[39718]: cluster 2021-07-31T02:40:44.265379+0000 mon.a ( 2021-07-31T02:40:45.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:45 smithi045 conmon[39718]: mon.0) 2831 : cluster [DBG] osdmap e1819: 8 total, 8 up, 8 in 2021-07-31T02:40:45.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:45 smithi045 conmon[33655]: cluster 2021-07-31T02:40:44.265379+0000 mon.a (mon.0 2021-07-31T02:40:45.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:45 smithi045 conmon[33655]: ) 2831 : cluster [DBG] osdmap e1819: 8 total, 8 up, 8 in 2021-07-31T02:40:45.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:45 smithi154 conmon[34553]: cluster 2021-07-31T02:40: 2021-07-31T02:40:45.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:45 smithi154 conmon[34553]: 44.265379+0000 mon.a (mon.0) 2831 : cluster [DBG] osdmap e1819: 8 total, 8 up, 8 in 2021-07-31T02:40:46.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:46 smithi154 conmon[34553]: cluster 2021-07-31T02:40:44.568662+0000 mgr.x (mgr.24542) 2296 : cluster [DBG] pgmap v3271: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 869 KiB/s rd, 3.7 KiB/s wr, 1.17k op/s 2021-07-31T02:40:46.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:46 smithi154 conmon[34553]: cluster 2021-07-31T02:40:45.277226+0000 mon.a (mon.0) 2832 : cluster [DBG] osdmap e1820: 8 total, 8 up, 8 in 2021-07-31T02:40:46.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:46 smithi045 conmon[39718]: cluster 2021-07-31T02:40:44.568662+0000 mgr.x (mgr 2021-07-31T02:40:46.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:46 smithi045 conmon[39718]: .24542) 2296 : cluster [DBG] pgmap v3271: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 869 KiB/s rd, 3.7 KiB/s wr, 1.17k op/s 2021-07-31T02:40:46.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:46 smithi045 conmon[39718]: cluster 2021-07-31T02:40:45.277226+0000 2021-07-31T02:40:46.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:46 smithi045 conmon[39718]: mon.a (mon.0) 2832 : cluster [DBG] osdmap e1820: 8 total, 8 up, 8 in 2021-07-31T02:40:46.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:46 smithi045 conmon[33655]: cluster 2021-07-31T02:40:44.568662+0000 2021-07-31T02:40:46.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:46 smithi045 conmon[33655]: mgr.x (mgr.24542) 2296 : cluster [DBG] pgmap v3271: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 869 KiB/s rd, 3.7 KiB/s wr, 1.17k op/s 2021-07-31T02:40:46.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:46 smithi045 conmon[33655]: cluster 2021-07-31T02 2021-07-31T02:40:46.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:46 smithi045 conmon[33655]: :40:45.277226+0000 mon.a (mon.0) 2832 : cluster [DBG] osdmap e1820: 8 total, 8 up, 8 in 2021-07-31T02:40:47.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:47 smithi154 conmon[34553]: cluster 2021-07-31T02:40:46.283148+0000 mon.a (mon.0) 2021-07-31T02:40:47.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:47 smithi154 conmon[34553]: 2833 : cluster [DBG] osdmap e1821: 8 total, 8 up, 8 in 2021-07-31T02:40:47.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:47 smithi045 conmon[39718]: cluster 2021-07-31T02:40:46.283148+0000 mon.a (mon.0) 2833 : 2021-07-31T02:40:47.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:47 smithi045 conmon[39718]: cluster [DBG] osdmap e1821: 8 total, 8 up, 8 in 2021-07-31T02:40:47.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:47 smithi045 conmon[33655]: cluster 2021-07-31T02:40:46.283148+0000 mon.a (mon.0) 2833 2021-07-31T02:40:47.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:47 smithi045 conmon[33655]: : cluster [DBG] osdmap e1821: 8 total, 8 up, 8 in 2021-07-31T02:40:48.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:48 smithi154 conmon[34553]: cluster 2021-07-31T02:40:46.569158+0000 mgr.x (mgr.24542 2021-07-31T02:40:48.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:48 smithi154 conmon[34553]: ) 2297 : cluster [DBG] pgmap v3274: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 869 KiB/s rd, 3.7 KiB/s wr, 1.17k op/s 2021-07-31T02:40:48.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:48 smithi154 conmon[34553]: cluster 2021-07-31T02:40:47.289383+0000 mon.a (mon.0) 2834 : cluster [DBG] osdmap e1822: 8 total, 8 up, 8 in 2021-07-31T02:40:48.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:48 smithi045 conmon[33655]: cluster 2021-07-31T02:40:46.569158+0000 mgr.x (mgr.24542) 2021-07-31T02:40:48.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:48 smithi045 conmon[33655]: 2297 : cluster [DBG] pgmap v3274: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 869 KiB/s rd, 3.7 KiB/s wr, 1.17k op/s 2021-07-31T02:40:48.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:48 smithi045 conmon[33655]: cluster 2021-07-31T02:40:47.289383+0000 mon.a (mon.0) 2834 : cluster [DBG] osdmap e1822: 8 total, 8 up, 8 in 2021-07-31T02:40:48.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:48 smithi045 conmon[39718]: cluster 2021-07-31T02:40:46.569158+0000 mgr.x (mgr.24542) 2297 : cluster [DBG] pgmap v3274: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 869 KiB/s rd, 3.7 KiB/s wr, 1.17k op/s 2021-07-31T02:40:48.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:48 smithi045 conmon[39718]: cluster 2021-07-31T02:40:47.289383+0000 mon.a (mon.0) 2834 : cluster 2021-07-31T02:40:48.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:48 smithi045 conmon[39718]: [DBG] osdmap e1822: 8 total, 8 up, 8 in 2021-07-31T02:40:49.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:49 smithi154 conmon[34553]: cluster 2021-07-31T02:40:48.291059+0000 mon.a (mon.0) 2835 2021-07-31T02:40:49.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:49 smithi154 conmon[34553]: : cluster [DBG] osdmap e1823: 8 total, 8 up, 8 in 2021-07-31T02:40:49.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:49 smithi045 conmon[33655]: cluster 2021-07-31T02:40:48.291059+0000 mon.a (mon.0) 2835 : cluster [DBG] osdmap e1823: 8 total, 8 up, 8 in 2021-07-31T02:40:49.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:49 smithi045 conmon[39718]: cluster 2021-07-31T02:40:48.291059+0000 mon.a (mon 2021-07-31T02:40:49.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:49 smithi045 conmon[39718]: .0) 2835 : cluster [DBG] osdmap e1823: 8 total, 8 up, 8 in 2021-07-31T02:40:50.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:50 smithi154 conmon[34553]: cluster 2021-07-31T02:40:48.569718+0000 mgr.x (mgr.24542) 2298 : cluster [DBG] 2021-07-31T02:40:50.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:50 smithi154 conmon[34553]: pgmap v3277: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:40:50.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:50 smithi154 conmon[34553]: cluster 2021-07-31T02:40:49.302882+0000 mon.a (mon.0) 2836 : cluster [DBG] osdmap e1824: 8 total, 8 up, 8 in 2021-07-31T02:40:50.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:50 smithi045 conmon[33655]: cluster 2021-07-31T02:40:48.569718+0000 2021-07-31T02:40:50.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:50 smithi045 conmon[33655]: mgr.x (mgr.24542) 2298 : cluster [DBG] pgmap v3277: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:40:50.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:50 smithi045 conmon[33655]: cluster 2021-07-31T02:40:49.302882 2021-07-31T02:40:50.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:50 smithi045 conmon[33655]: +0000 mon.a (mon.0) 2836 : cluster [DBG] osdmap e1824: 8 total, 8 up, 8 in 2021-07-31T02:40:50.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:50 smithi045 conmon[39718]: cluster 2021-07-31T02:40:48 2021-07-31T02:40:50.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:50 smithi045 conmon[39718]: .569718+0000 mgr.x (mgr.24542) 2298 : cluster [DBG] pgmap v3277: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:40:50.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:50 smithi045 conmon[39718]: cluster 2021-07- 2021-07-31T02:40:50.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:50 smithi045 conmon[39718]: 31T02:40:49.302882 2021-07-31T02:40:50.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:50 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2836 2021-07-31T02:40:50.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:50 smithi045 conmon[39718]: : cluster [DBG] osdmap e1824: 8 total, 8 up, 8 in 2021-07-31T02:40:51.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:51 smithi154 conmon[34553]: cluster 2021-07-31T02:40:50.304636+0000 mon.a (mon.0) 2837 : cluster [DBG] osdmap e1825: 8 total, 8 up, 8 in 2021-07-31T02:40:51.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:51 smithi045 conmon[39718]: cluster 2021-07-31T02:40:50.304636+0000 mon.a (mon.0 2021-07-31T02:40:51.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:51 smithi045 conmon[39718]: ) 2837 : cluster [DBG] osdmap e1825: 8 total, 8 up, 8 in 2021-07-31T02:40:51.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:51 smithi045 conmon[33655]: cluster 2021-07-31T02:40:50.304636+0000 mon.a (mon.0) 2837 : cluster [DBG] osdmap e1825: 8 total, 8 up, 8 in 2021-07-31T02:40:51.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:51 smithi045 conmon[33655]: 2021-07-31T02:40:52.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:52 smithi154 conmon[34553]: cluster 2021-07-31T02:40:50.570355+0000 mgr.x (mgr.24542) 2299 : cluster 2021-07-31T02:40:52.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:52 smithi154 conmon[34553]: [DBG] pgmap v3280: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 902 KiB/s rd, 3.7 KiB/s wr, 1.22k op/s 2021-07-31T02:40:52.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:52 smithi154 conmon[34553]: cluster 2021-07-31T02:40:51.315183+0000 mon.a (mon.0) 2838 : cluster [DBG] osdmap e1826: 8 total, 8 up, 8 in 2021-07-31T02:40:52.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:52 smithi045 conmon[39718]: cluster 2021-07-31T02:40: 2021-07-31T02:40:52.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:52 smithi045 conmon[39718]: 50.570355+0000 mgr.x (mgr.24542) 2299 : cluster [DBG] pgmap v3280: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 902 KiB/s rd, 3.7 KiB/s wr, 1.22k op/s 2021-07-31T02:40:52.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:52 smithi045 conmon[39718]: cluster 2021-07-31T02: 2021-07-31T02:40:52.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:52 smithi045 conmon[39718]: 40:51.315183+0000 mon.a ( 2021-07-31T02:40:52.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:52 smithi045 conmon[39718]: mon.0) 2838 : cluster 2021-07-31T02:40:52.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:52 smithi045 conmon[39718]: [DBG] osdmap e1826: 8 total, 8 up, 8 in 2021-07-31T02:40:52.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:52 smithi045 conmon[33655]: cluster 2021-07-31T02:40:50.570355+0000 mgr.x (mgr.24542) 2299 : cluster [DBG] pgmap v3280: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 902 KiB/s rd, 3.7 KiB/s wr, 1.22k op/s 2021-07-31T02:40:52.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:52 smithi045 conmon[33655]: cluster 2021-07-31T02:40:51.315183+0000 mon.a ( 2021-07-31T02:40:52.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:52 smithi045 conmon[33655]: mon.0) 2838 : cluster [DBG] osdmap e1826: 8 total, 8 up, 8 in 2021-07-31T02:40:53.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:53 smithi154 conmon[34553]: cluster 2021-07-31T02:40:52.317328+0000 mon.a (mon.0) 2839 : cluster [DBG] osdmap e1827: 8 total, 8 up, 8 in 2021-07-31T02:40:53.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:53 smithi045 conmon[39718]: cluster 2021-07-31T02:40:52.317328+0000 mon.a ( 2021-07-31T02:40:53.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:53 smithi045 conmon[39718]: mon.0) 2839 : cluster [DBG] osdmap e1827: 8 total, 8 up, 8 in 2021-07-31T02:40:53.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:53 smithi045 conmon[33655]: cluster 2021-07-31T02:40:52.317328+0000 mon.a ( 2021-07-31T02:40:53.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:53 smithi045 conmon[33655]: mon.0) 2839 : cluster [DBG] osdmap e1827: 8 total, 8 up, 8 in 2021-07-31T02:40:54.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:54 smithi154 conmon[34553]: cluster 2021-07-31T02:40:54.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:54 smithi154 conmon[34553]: 2021-07-31T02:40:54.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:54 smithi154 conmon[34553]: 2021 2021-07-31T02:40:54.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:54 smithi154 conmon[34553]: -07-31T02:40:52.570765+0000 mgr.x (mgr 2021-07-31T02:40:54.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:54 smithi154 conmon[34553]: .24542) 2300 : cluster [DBG] pgmap v3283: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 902 KiB/s rd, 3.7 KiB/s wr, 1.22k op/s 2021-07-31T02:40:54.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:54 smithi154 conmon[34553]: cluster 2021-07-31T02:40:54.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:54 smithi154 conmon[34553]: 2021-07-31T02:40:53.324054+0000 mon.a (mon.0) 2840 : cluster [DBG] osdmap e1828: 8 total, 8 up, 8 in 2021-07-31T02:40:54.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:54 smithi045 conmon[33655]: cluster 2021-07-31T02:40:52.570765+0000 2021-07-31T02:40:54.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:54 smithi045 conmon[33655]: mgr.x (mgr.24542) 2300 : cluster [DBG] pgmap v3283: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 902 KiB/s rd, 3.7 KiB/s wr, 1.22k op/s 2021-07-31T02:40:54.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:54 smithi045 conmon[33655]: cluster 2021-07-31T02:40:53.324054+0000 mon.a (mon.0) 2840 : cluster [DBG] osdmap e1828: 8 total, 8 up, 8 in 2021-07-31T02:40:54.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:54 smithi045 conmon[39718]: cluster 2021-07-31T02:40:52 2021-07-31T02:40:54.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:54 smithi045 conmon[39718]: .570765+0000 mgr.x (mgr.24542) 2300 : cluster [DBG] pgmap v3283: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 902 KiB/s rd, 3.7 KiB/s wr, 1.22k op/s 2021-07-31T02:40:54.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:54 smithi045 conmon[39718]: cluster 2021-07-31T02:40:53.324054+0000 mon.a 2021-07-31T02:40:54.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:54 smithi045 conmon[39718]: (mon.0) 2840 : cluster [DBG] osdmap e1828: 8 total, 8 up, 8 in 2021-07-31T02:40:55.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:55 smithi154 conmon[34553]: cluster 2021-07-31T02:40:54.330676+0000 mon.a (mon.0) 2841 : cluster [DBG] 2021-07-31T02:40:55.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:55 smithi154 conmon[34553]: osdmap e1829: 8 total, 8 up, 8 in 2021-07-31T02:40:55.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:55 smithi045 conmon[33655]: cluster 2021-07-31T02:40:54. 2021-07-31T02:40:55.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:55 smithi045 conmon[33655]: 330676+0000 mon.a (mon.0) 2841 : cluster [DBG] osdmap e1829: 8 total, 8 up, 8 in 2021-07-31T02:40:55.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:55 smithi045 conmon[39718]: cluster 2021-07-31T02:40:54.330676+0000 2021-07-31T02:40:55.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:55 smithi045 conmon[39718]: mon.a (mon.0) 2841 : cluster [DBG] osdmap e1829: 8 total, 8 up, 8 in 2021-07-31T02:40:56.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:56 smithi154 conmon[34553]: cluster 2021-07-31T02:40:54.571180+0000 mgr.x (mgr.24542 2021-07-31T02:40:56.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:56 smithi154 conmon[34553]: ) 2301 : cluster [DBG] pgmap v3286: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 935 KiB/s rd, 3.7 KiB/s wr, 1.26k op/s 2021-07-31T02:40:56.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:56 smithi154 conmon[34553]: cluster 2021-07-31T02:40:55.344969+0000 mon.a (mon.0) 2842 : cluster [DBG] osdmap e1830: 8 total, 8 up, 8 in 2021-07-31T02:40:56.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:56 smithi045 conmon[33655]: cluster 2021-07-31T02:40:54.571180+0000 mgr.x (mgr.24542) 2301 : cluster [DBG] pgmap v3286: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 935 KiB/s rd, 3.7 KiB/s wr, 1.26k op/s 2021-07-31T02:40:56.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:56 smithi045 conmon[33655]: cluster 2021-07-31T02:40:55.344969+0000 mon.a (mon.0) 2842 : cluster [DBG] osdmap e1830: 8 total, 8 up, 8 in 2021-07-31T02:40:56.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:56 smithi045 conmon[39718]: cluster 2021-07-31T02:40:54.571180+0000 mgr.x (mgr.24542 2021-07-31T02:40:56.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:56 smithi045 conmon[39718]: ) 2301 : cluster [DBG] pgmap v3286: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 935 KiB/s rd, 3.7 KiB/s wr, 1.26k op/s 2021-07-31T02:40:56.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:56 smithi045 conmon[39718]: cluster 2021-07-31T02:40:55.344969+0000 mon.a (mon 2021-07-31T02:40:56.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:56 smithi045 conmon[39718]: .0) 2842 : cluster [DBG] osdmap e1830: 8 total, 8 up, 8 in 2021-07-31T02:40:57.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:57 smithi154 conmon[34553]: cluster 2021-07-31T02:40:56.346371+0000 mon.a (mon.0) 2843 : cluster [DBG] 2021-07-31T02:40:57.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:57 smithi154 conmon[34553]: osdmap e1831: 8 total, 8 up, 8 in 2021-07-31T02:40:57.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:57 smithi045 conmon[33655]: cluster 2021-07-31T02:40:56.346371+0000 mon.a (mon.0) 2843 : cluster [DBG] osdmap e1831: 8 total, 8 up, 8 in 2021-07-31T02:40:57.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:57 smithi045 conmon[39718]: cluster 2021-07-31T02:40:56.346371+0000 mon.a (mon. 2021-07-31T02:40:57.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:57 smithi045 conmon[39718]: 0) 2843 : cluster [DBG] osdmap e1831: 8 total, 8 up, 8 in 2021-07-31T02:40:58.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:58 smithi154 conmon[34553]: cluster 2021-07-31T02:40:58.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:58 smithi154 conmon[34553]: 2021-07-31T02:40:56.571666 2021-07-31T02:40:58.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:58 smithi154 conmon[34553]: +0000 mgr.x (mgr.24542) 2302 : cluster [DBG] pgmap v3289: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 935 KiB/s rd, 3.7 KiB/s wr, 1.26k op/s 2021-07-31T02:40:58.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:58 smithi154 conmon[34553]: cluster 2021-07-31T02:40:58.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:58 smithi154 conmon[34553]: 2021 2021-07-31T02:40:58.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:58 smithi154 conmon[34553]: -07-31T02:40:57.356805 2021-07-31T02:40:58.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:58 smithi154 conmon[34553]: +0000 mon.a (mon.0) 2844 : cluster [DBG] osdmap e1832: 8 total, 8 up, 8 in 2021-07-31T02:40:58.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:58 smithi045 conmon[33655]: cluster 2021-07-31T02:40:56.571666+0000 mgr.x (mgr.24542) 2021-07-31T02:40:58.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:58 smithi045 conmon[33655]: 2302 : cluster [DBG] pgmap v3289: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 935 KiB/s rd, 3.7 KiB/s wr, 1.26k op/s 2021-07-31T02:40:58.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:58 smithi045 conmon[33655]: cluster 2021-07-31T02:40:57.356805+0000 mon.a (mon.0) 2844 : cluster [DBG] osdmap e1832: 8 total, 8 up, 8 in 2021-07-31T02:40:58.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:58 smithi045 conmon[39718]: cluster 2021-07-31T02:40:56.571666+0000 mgr.x ( 2021-07-31T02:40:58.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:58 smithi045 conmon[39718]: mgr.24542) 2302 : cluster [DBG] pgmap v3289: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 935 KiB/s rd, 3.7 KiB/s wr, 1.26k op/s 2021-07-31T02:40:58.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:58 smithi045 conmon[39718]: cluster 2021-07-31T02:40:57.356805+0000 mon.a (mon.0) 2844 : cluster [DBG] osdmap e1832: 8 total, 8 up, 8 in 2021-07-31T02:40:59.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:59 smithi154 conmon[34553]: cluster 2021-07-31T02:40:58.358430+0000 mon.a (mon.0) 2845 : cluster [DBG] 2021-07-31T02:40:59.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:40:59 smithi154 conmon[34553]: osdmap e1833: 8 total, 8 up, 8 in 2021-07-31T02:40:59.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:40:59 smithi045 conmon[33655]: cluster 2021-07-31T02:40:58.358430+0000 mon.a (mon.0) 2845 : cluster [DBG] osdmap e1833: 8 total, 8 up, 8 in 2021-07-31T02:40:59.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:59 smithi045 conmon[39718]: cluster 2021-07-31T02:40:58.358430+0000 2021-07-31T02:40:59.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:40:59 smithi045 conmon[39718]: mon.a (mon.0) 2845 : cluster [DBG] osdmap e1833: 8 total, 8 up, 8 in 2021-07-31T02:41:00.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:00 smithi154 conmon[34553]: cluster 2021-07-31T02:40:58.572222+0000 mgr.x (mgr.24542) 2303 : cluster [DBG] pgmap v3292: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:41:00.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:00 smithi154 conmon[34553]: cluster 2021-07-31T02:40:59.369127+0000 mon.a (mon.0) 2846 : cluster [DBG] osdmap e1834: 8 total, 8 up, 8 in 2021-07-31T02:41:00.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:00 smithi045 conmon[33655]: cluster 2021-07-31T02:40:58.572222+0000 mgr.x (mgr.24542) 2303 2021-07-31T02:41:00.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:00 smithi045 conmon[33655]: : cluster [DBG] pgmap v3292: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:41:00.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:00 smithi045 conmon[33655]: cluster 2021-07-31T02:40:59.369127+0000 mon.a (mon.0) 2846 : cluster [DBG] osdmap e1834: 8 total, 8 up, 8 in 2021-07-31T02:41:00.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:00 smithi045 conmon[39718]: cluster 2021-07-31T02:40:58.572222+0000 mgr.x 2021-07-31T02:41:00.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:00 smithi045 conmon[39718]: (mgr.24542) 2303 : cluster [DBG] pgmap v3292: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:41:00.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:00 smithi045 conmon[39718]: cluster 2021-07-31T 2021-07-31T02:41:00.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:00 smithi045 conmon[39718]: 02:40:59.369127+0000 mon.a (mon.0) 2846 : cluster 2021-07-31T02:41:00.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:00 smithi045 conmon[39718]: [DBG] osdmap e1834: 8 total, 8 up, 8 in 2021-07-31T02:41:01.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:01 smithi154 conmon[34553]: cluster 2021-07-31T02:41:00.375262+0000 mon.a (mon.0) 2847 : cluster [DBG] 2021-07-31T02:41:01.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:01 smithi154 conmon[34553]: osdmap e1835: 8 total, 8 up, 8 in 2021-07-31T02:41:01.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:01 smithi045 conmon[39718]: cluster 2021-07-31T02:41:00.375262+0000 2021-07-31T02:41:01.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:01 smithi045 conmon[39718]: mon.a (mon.0) 2847 : cluster [DBG] osdmap e1835: 8 total, 8 up, 8 in 2021-07-31T02:41:01.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:01 smithi045 conmon[33655]: cluster 2021-07-31T02:41:00.375262+0000 mon.a (mon.0) 2847 2021-07-31T02:41:01.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:01 smithi045 conmon[33655]: : cluster [DBG] osdmap e1835: 8 total, 8 up, 8 in 2021-07-31T02:41:02.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:02 smithi154 conmon[34553]: cluster 2021-07-31T02:41:00.572760+0000 mgr.x (mgr.24542) 2304 : 2021-07-31T02:41:02.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:02 smithi154 conmon[34553]: cluster [DBG] pgmap v3295: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 967 KiB/s rd, 3.7 KiB/s wr, 1.30k op/s 2021-07-31T02:41:02.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:02 smithi154 conmon[34553]: cluster 2021-07-31T02:41:01.382797+0000 mon.a (mon.0) 2848 : cluster [DBG] osdmap e1836: 8 total, 8 up, 8 in 2021-07-31T02:41:02.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:02 smithi045 conmon[33655]: cluster 2021-07-31T02:41:00.572760+0000 mgr.x (mgr.24542 2021-07-31T02:41:02.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:02 smithi045 conmon[33655]: ) 2304 : cluster [DBG] pgmap v3295: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 967 KiB/s rd, 3.7 KiB/s wr, 1.30k op/s 2021-07-31T02:41:02.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:02 smithi045 conmon[33655]: cluster 2021-07-31T02:41:01.382797+0000 mon.a (mon.0) 2848 : cluster [DBG] osdmap e1836: 8 total, 8 up, 8 in 2021-07-31T02:41:02.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:02 smithi045 conmon[39718]: cluster 2021-07-31T02:41:00.572760+0000 mgr.x 2021-07-31T02:41:02.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:02 smithi045 conmon[39718]: (mgr.24542) 2304 : cluster [DBG] pgmap v3295: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 967 KiB/s rd, 3.7 KiB/s wr, 1.30k op/s 2021-07-31T02:41:02.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:02 smithi045 conmon[39718]: cluster 2021-07-31T02:41: 2021-07-31T02:41:02.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:02 smithi045 conmon[39718]: 01.382797+0000 mon.a (mon.0) 2848 : cluster [DBG] osdmap e1836: 8 total, 8 up, 8 in 2021-07-31T02:41:03.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:03 smithi154 conmon[34553]: cluster 2021-07-31T02:41:02.388580+0000 mon.a (mon.0) 2849 2021-07-31T02:41:03.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:03 smithi154 conmon[34553]: : cluster [DBG] osdmap e1837: 8 total, 8 up, 8 in 2021-07-31T02:41:03.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:03 smithi045 conmon[33655]: cluster 2021-07-31T02:41:02.388580+0000 mon.a (mon 2021-07-31T02:41:03.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:03 smithi045 conmon[33655]: .0) 2849 : cluster [DBG] osdmap e1837: 8 total, 8 up, 8 in 2021-07-31T02:41:03.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:03 smithi045 conmon[39718]: cluster 2021-07-31T02:41:02.388580+0000 2021-07-31T02:41:03.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:03 smithi045 conmon[39718]: mon.a (mon.0) 2849 : cluster [DBG] osdmap e1837: 8 total, 8 up, 8 in 2021-07-31T02:41:04.012 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:41:04.288 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:41:04.289 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 55m ago 56m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:41:04.289 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 53m ago 56m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:41:04.289 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 53m ago 59m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:41:04.289 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 55m ago 62m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:41:04.290 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 55m ago 62m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:41:04.290 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 53m ago 60m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:41:04.290 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 55m ago 60m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:41:04.290 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 55m ago 56m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:41:04.291 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 53m ago 56m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:41:04.291 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 55m ago 59m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:41:04.291 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 55m ago 59m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:41:04.291 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 55m ago 59m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:41:04.292 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 55m ago 58m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:41:04.292 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 53m ago 58m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:41:04.292 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 53m ago 58m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:41:04.292 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 53m ago 57m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:41:04.292 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 53m ago 57m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:41:04.293 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 53m ago 57m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:41:04.632 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:41:04.632 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:41:04.633 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:41:04.633 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:41:04.633 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:41:04.633 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:41:04.633 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:41:04.634 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:41:04.634 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:41:04.634 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:41:04.634 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:41:04.635 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:41:04.635 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:41:04.635 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:41:04.635 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:41:04.635 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:41:04.636 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:41:04.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:04 smithi154 conmon[34553]: cluster 2021-07-31T02:41:02.573303+0000 mgr.x (mgr.24542) 2305 : cluster [DBG] pgmap v3298: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 967 KiB/s rd, 3.7 KiB/s wr, 1.30k op/s 2021-07-31T02:41:04.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:04 smithi154 conmon[34553]: cluster 2021-07-31T02:41:03.391106+0000 mon.a (mon.0) 2850 : cluster [DBG] osdmap e1838: 8 total, 8 up, 8 in 2021-07-31T02:41:04.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:04 smithi045 conmon[39718]: cluster 2021-07-31T02 2021-07-31T02:41:04.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:04 smithi045 conmon[39718]: :41:02.573303+0000 mgr.x (mgr.24542) 2305 : cluster [DBG] pgmap v3298: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 967 KiB/s rd, 3.7 KiB/s wr, 1.30k op/s 2021-07-31T02:41:04.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:04 smithi045 conmon[39718]: cluster 2021 2021-07-31T02:41:04.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:04 smithi045 conmon[39718]: -07-31T02:41: 2021-07-31T02:41:04.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:04 smithi045 conmon[39718]: 03.391106+0000 mon.a (mon 2021-07-31T02:41:04.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:04 smithi045 conmon[39718]: .0) 2850 : cluster 2021-07-31T02:41:04.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:04 smithi045 conmon[39718]: [DBG] osdmap e1838: 8 total, 8 up, 8 in 2021-07-31T02:41:04.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:04 smithi045 conmon[33655]: cluster 2021-07-31T02:41:02. 2021-07-31T02:41:04.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:04 smithi045 conmon[33655]: 573303+0000 mgr.x (mgr.24542) 2305 : cluster [DBG] pgmap v3298: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 967 KiB/s rd, 3.7 KiB/s wr, 1.30k op/s 2021-07-31T02:41:04.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:04 smithi045 conmon[33655]: cluster 2021-07-31T02:41:04.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:04 smithi045 conmon[33655]: 2021-07-31T02:41:03.391106+0000 mon.a 2021-07-31T02:41:04.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:04 smithi045 conmon[33655]: (mon.0) 2850 : cluster [DBG] osdmap e1838: 8 total, 8 up, 8 in 2021-07-31T02:41:04.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:04 smithi045 conmon[33655]: 2021-07-31T02:41:05.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:05 smithi154 conmon[34553]: audit 2021-07-31T02:41:04.001322+0000 mgr.x (mgr.24542) 2021-07-31T02:41:05.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:05 smithi154 conmon[34553]: 2306 : audit [DBG] from='client.17844 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:41:05.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:05 smithi154 conmon[34553]: audit 2021-07-31T02:41:04.280770+0000 mgr.x (mgr.24542) 2307 : audit [DBG] from='client.17850 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:41:05.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:05 smithi154 conmon[34553]: cluster 2021-07-31T02:41:04.404836+0000 mon.a (mon.0) 2851 : cluster [DBG] osdmap e1839: 8 total, 8 up, 8 in 2021-07-31T02:41:05.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:05 smithi154 conmon[34553]: audit 2021-07-31T02:41:04.632329+0000 2021-07-31T02:41:05.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:05 smithi154 conmon[34553]: mon.a (mon.0) 2852 : audit [DBG] from='client.? 172.21.15.45:0/1816669322' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:41:05.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:05 smithi045 conmon[33655]: audit 2021-07-31T02:41:04.001322+0000 mgr.x (mgr.24542) 2306 : 2021-07-31T02:41:05.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:05 smithi045 conmon[33655]: audit [DBG] from='client.17844 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:41:05.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:05 smithi045 conmon[33655]: audit 2021-07-31T02:41:04.280770+0000 mgr.x (mgr.24542) 2307 : audit [DBG] from='client.17850 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:41:05.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:05 smithi045 conmon[33655]: cluster 2021-07-31T 2021-07-31T02:41:05.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:05 smithi045 conmon[33655]: 02:41:04.404836+0000 2021-07-31T02:41:05.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:05 smithi045 conmon[33655]: mon.a (mon.0) 2851 : cluster 2021-07-31T02:41:05.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:05 smithi045 conmon[33655]: [DBG] osdmap e1839: 8 total, 8 up, 8 in 2021-07-31T02:41:05.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:05 smithi045 conmon[33655]: audit 2021-07-31T02: 2021-07-31T02:41:05.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:05 smithi045 conmon[33655]: 41:04.632329+0000 mon.a (mon 2021-07-31T02:41:05.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:05 smithi045 conmon[33655]: .0) 2852 : audit [DBG] 2021-07-31T02:41:05.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:05 smithi045 conmon[33655]: from='client.? 172.21.15.45:0/1816669322' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:41:05.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:05 smithi045 conmon[39718]: audit 2021-07-31T02:41:04.001322 2021-07-31T02:41:05.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:05 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 2306 : audit [DBG] from='client.17844 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:41:05.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:05 smithi045 conmon[39718]: audit 2021-07-31T02:41:05.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:05 smithi045 conmon[39718]: 2021-07-31T02: 2021-07-31T02:41:05.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:05 smithi045 conmon[39718]: 41:04.280770+0000 mgr.x ( 2021-07-31T02:41:05.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:05 smithi045 conmon[39718]: mgr.24542) 2307 : audit [DBG] from='client.17850 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:41:05.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:05 smithi045 conmon[39718]: 2021-07-31T02:41:05.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:05 smithi045 conmon[39718]: cluster 2021-07-31T02:41:05.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:05 smithi045 conmon[39718]: 2021-07 2021-07-31T02:41:05.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:05 smithi045 conmon[39718]: -31T02:41:04 2021-07-31T02:41:05.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:05 smithi045 conmon[39718]: .404836+0000 mon.a (mon. 2021-07-31T02:41:05.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:05 smithi045 conmon[39718]: 0) 2851 : cluster [DBG] osdmap e1839: 8 total, 8 up, 8 in 2021-07-31T02:41:05.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:05 smithi045 conmon[39718]: 2021-07-31T02:41:05.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:05 smithi045 conmon[39718]: audit 2021-07 2021-07-31T02:41:05.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:05 smithi045 conmon[39718]: -31T02:41:04.632329+0000 2021-07-31T02:41:05.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:05 smithi045 conmon[39718]: mon.a (mon.0) 2852 : audit 2021-07-31T02:41:05.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:05 smithi045 conmon[39718]: [DBG] from='client.? 172.21.15.45:0/1816669322' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:41:06.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:06 smithi154 conmon[34553]: cluster 2021-07-31T02:41:04.573710+0000 mgr.x ( 2021-07-31T02:41:06.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:06 smithi154 conmon[34553]: mgr.24542) 2308 : cluster [DBG] pgmap v3301: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 999 KiB/s rd, 3.7 KiB/s wr, 1.35k op/s 2021-07-31T02:41:06.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:06 smithi154 conmon[34553]: cluster 2021-07-31T02:41:05.411578+0000 mon.a (mon.0) 2853 : cluster [DBG] osdmap e1840: 8 total, 8 up, 8 in 2021-07-31T02:41:06.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:06 smithi045 conmon[39718]: cluster 2021-07-31T02:41:04.573710+0000 mgr.x (mgr 2021-07-31T02:41:06.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:06 smithi045 conmon[39718]: .24542) 2308 : cluster [DBG] pgmap v3301: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 999 KiB/s rd, 3.7 KiB/s wr, 1.35k op/s 2021-07-31T02:41:06.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:06 smithi045 conmon[39718]: cluster 2021-07-31T02:41:05.411578+0000 mon.a (mon.0) 2853 2021-07-31T02:41:06.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:06 smithi045 conmon[39718]: : cluster [DBG] osdmap e1840: 8 total, 8 up, 8 in 2021-07-31T02:41:06.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:06 smithi045 conmon[33655]: cluster 2021-07-31T02:41:04.573710+0000 mgr.x (mgr.24542) 2021-07-31T02:41:06.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:06 smithi045 conmon[33655]: 2308 : cluster [DBG] pgmap v3301: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 999 KiB/s rd, 3.7 KiB/s wr, 1.35k op/s 2021-07-31T02:41:06.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:06 smithi045 conmon[33655]: cluster 2021-07-31T02:41:05.411578+0000 mon.a (mon.0) 2853 : cluster [DBG] osdmap e1840: 8 total, 8 up, 8 in 2021-07-31T02:41:07.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:07 smithi154 conmon[34553]: cluster 2021-07-31T02:41:06.413106+0000 mon.a (mon.0) 2854 : 2021-07-31T02:41:07.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:07 smithi154 conmon[34553]: cluster [DBG] osdmap e1841: 8 total, 8 up, 8 in 2021-07-31T02:41:07.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:07 smithi045 conmon[33655]: cluster 2021-07-31T02:41:06.413106+0000 2021-07-31T02:41:07.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:07 smithi045 conmon[33655]: mon.a (mon.0) 2854 : cluster [DBG] osdmap e1841: 8 total, 8 up, 8 in 2021-07-31T02:41:07.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:07 smithi045 conmon[39718]: cluster 2021-07-31T02:41:06.413106+0000 mon.a (mon.0 2021-07-31T02:41:07.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:07 smithi045 conmon[39718]: ) 2854 : cluster [DBG] osdmap e1841: 8 total, 8 up, 8 in 2021-07-31T02:41:08.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:08 smithi154 conmon[34553]: cluster 2021-07-31T02:41:06.574065+0000 mgr.x (mgr.24542) 2309 : 2021-07-31T02:41:08.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:08 smithi154 conmon[34553]: cluster [DBG] pgmap v3304: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 999 KiB/s rd, 3.7 KiB/s wr, 1.35k op/s 2021-07-31T02:41:08.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:08 smithi154 conmon[34553]: cluster 2021-07-31T02:41:07.423826+0000 mon.a (mon.0) 2855 : cluster [DBG] osdmap e1842: 8 total, 8 up, 8 in 2021-07-31T02:41:08.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:08 smithi045 conmon[33655]: cluster 2021-07-31T02:41:06.574065+0000 mgr.x (mgr.24542) 2309 : cluster [DBG] pgmap v3304: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 999 KiB/s rd, 3.7 KiB/s wr, 1.35k op/s 2021-07-31T02:41:08.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:08 smithi045 conmon[33655]: cluster 2021-07-31T02:41:07.423826+0000 mon.a (mon.0) 2855 : cluster [DBG] osdmap e1842: 8 total, 8 up, 8 in 2021-07-31T02:41:08.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:08 smithi045 conmon[39718]: cluster 2021-07-31T02:41:06. 2021-07-31T02:41:08.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:08 smithi045 conmon[39718]: 574065+0000 mgr.x (mgr.24542) 2309 : cluster [DBG] pgmap v3304: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 999 KiB/s rd, 3.7 KiB/s wr, 1.35k op/s 2021-07-31T02:41:08.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:08 smithi045 conmon[39718]: cluster 2021-07-31T02:41:07.423826+0000 mon.a 2021-07-31T02:41:08.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:08 smithi045 conmon[39718]: (mon.0) 2855 : cluster [DBG] osdmap e1842: 8 total, 8 up, 8 in 2021-07-31T02:41:09.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:09 smithi154 conmon[34553]: cluster 2021-07-31T02:41:08.425703+0000 mon.a (mon.0) 2856 : cluster [DBG] 2021-07-31T02:41:09.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:09 smithi154 conmon[34553]: osdmap e1843: 8 total, 8 up, 8 in 2021-07-31T02:41:09.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:09 smithi045 conmon[39718]: cluster 2021-07-31T02:41:09.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:09 smithi045 conmon[39718]: 2021-07-31T02:41:09.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:09 smithi045 conmon[39718]: 2021 2021-07-31T02:41:09.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:09 smithi045 conmon[39718]: - 2021-07-31T02:41:09.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:09 smithi045 conmon[39718]: 07 2021-07-31T02:41:09.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:09 smithi045 conmon[39718]: - 2021-07-31T02:41:09.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:09 smithi045 conmon[39718]: 31T02:41:08.425703+0000 mon.a (mon 2021-07-31T02:41:09.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:09 smithi045 conmon[39718]: .0) 2856 : cluster [DBG] osdmap e1843: 8 total, 8 up, 8 in 2021-07-31T02:41:09.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:09 smithi045 conmon[33655]: cluster 2021-07-31T02:41:08.425703+0000 2021-07-31T02:41:09.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:09 smithi045 conmon[33655]: mon.a (mon.0) 2856 : cluster [DBG] osdmap e1843: 8 total, 8 up, 8 in 2021-07-31T02:41:10.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:10 smithi154 conmon[34553]: cluster 2021-07-31T02:41:08.574632+0000 mgr.x (mgr.24542) 2310 : cluster [DBG] pgmap v3307: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:41:10.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:10 smithi154 conmon[34553]: cluster 2021- 2021-07-31T02:41:10.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:10 smithi154 conmon[34553]: 07-31T02:41:09.437763+0000 mon.a (mon.0) 2857 : cluster [DBG] osdmap e1844: 8 total, 8 up, 8 in 2021-07-31T02:41:10.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:10 smithi045 conmon[39718]: cluster 2021-07-31T02:41:08.574632+0000 2021-07-31T02:41:10.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:10 smithi045 conmon[39718]: mgr.x (mgr.24542) 2310 : cluster [DBG] pgmap v3307: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:41:10.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:10 smithi045 conmon[39718]: cluster 2021-07-31T02:41:10.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:10 smithi045 conmon[39718]: 2021-07-31T02:41:09.437763+0000 mon.a (mon.0) 2857 : cluster [DBG] osdmap e1844: 8 total, 8 up, 8 in 2021-07-31T02:41:10.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:10 smithi045 conmon[33655]: cluster 2021-07-31T02:41:08.574632+0000 mgr.x (mgr.24542) 2310 : cluster [DBG] 2021-07-31T02:41:10.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:10 smithi045 conmon[33655]: pgmap v3307: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:41:10.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:10 smithi045 conmon[33655]: cluster 2021-07-31T02:41:09.437763+0000 mon.a (mon.0) 2857 : cluster [DBG] osdmap e1844: 8 total, 8 up, 8 in 2021-07-31T02:41:11.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:11 smithi154 conmon[34553]: cluster 2021-07-31T02:41:11.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:11 smithi154 conmon[34553]: 2021-07-31T02:41:10.439567+0000 2021-07-31T02:41:11.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:11 smithi154 conmon[34553]: mon.a (mon.0) 2858 : cluster [DBG] osdmap e1845: 8 total, 8 up, 8 in 2021-07-31T02:41:11.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:11 smithi045 conmon[39718]: cluster 2021-07-31T02:41:10.439567+0000 mon.a 2021-07-31T02:41:11.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:11 smithi045 conmon[39718]: (mon.0) 2858 : cluster [DBG] osdmap e1845: 8 total, 8 up, 8 in 2021-07-31T02:41:11.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:11 smithi045 conmon[33655]: cluster 2021-07-31T02:41:10.439567+0000 mon.a 2021-07-31T02:41:11.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:11 smithi045 conmon[33655]: (mon.0) 2858 : cluster [DBG] osdmap e1845: 8 total, 8 up, 8 in 2021-07-31T02:41:12.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:12 smithi154 conmon[34553]: cluster 2021-07-31T02:41:10.575101+0000 mgr.x (mgr.24542 2021-07-31T02:41:12.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:12 smithi154 conmon[34553]: ) 2311 : cluster [DBG] pgmap v3310: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.0 MiB/s rd, 3.7 KiB/s wr, 1.39k op/s 2021-07-31T02:41:12.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:12 smithi154 conmon[34553]: cluster 2021-07-31T02:41:11.449501+0000 mon.a (mon.0) 2859 : cluster [DBG] osdmap e1846: 8 total, 8 up, 8 in 2021-07-31T02:41:12.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:12 smithi045 conmon[39718]: cluster 2021-07-31T02:41:10.575101+0000 mgr.x (mgr.24542) 2311 : 2021-07-31T02:41:12.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:12 smithi045 conmon[39718]: cluster [DBG] pgmap v3310: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.0 MiB/s rd, 3.7 KiB/s wr, 1.39k op/s 2021-07-31T02:41:12.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:12 smithi045 conmon[39718]: cluster 2021-07-31T02:41:11.449501+0000 mon.a (mon.0) 2859 : cluster [DBG] osdmap e1846: 8 total, 8 up, 8 in 2021-07-31T02:41:12.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:12 smithi045 conmon[33655]: cluster 2021-07-31T02:41:10.575101+0000 mgr.x (mgr.24542) 2311 : cluster 2021-07-31T02:41:12.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:12 smithi045 conmon[33655]: [DBG] pgmap v3310: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.0 MiB/s rd, 3.7 KiB/s wr, 1.39k op/s 2021-07-31T02:41:12.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:12 smithi045 conmon[33655]: cluster 2021-07-31T02:41:11.449501+0000 mon.a (mon.0) 2859 : cluster [DBG] osdmap e1846: 8 total, 8 up, 8 in 2021-07-31T02:41:13.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:13 smithi154 conmon[34553]: cluster 2021-07-31T02:41:12.451193+0000 mon.a (mon.0) 2860 : cluster [DBG] osdmap e1847: 8 total, 8 up, 8 in 2021-07-31T02:41:13.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:13 smithi045 conmon[39718]: cluster 2021-07-31T02:41:12.451193+0000 mon.a (mon.0) 2860 : cluster [DBG] osdmap e1847: 8 total, 8 up, 8 in 2021-07-31T02:41:13.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:13 smithi045 conmon[33655]: cluster 2021-07-31T02:41:12.451193+0000 mon.a (mon.0) 2860 : cluster [DBG] osdmap e1847: 8 total, 8 up, 8 in 2021-07-31T02:41:14.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:14 smithi154 conmon[34553]: cluster 2021-07-31T02:41:12.575570+0000 mgr.x (mgr.24542) 2312 : cluster [DBG] pgmap v3313: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.0 MiB/s rd, 3.7 KiB/s wr, 1.39k op/s 2021-07-31T02:41:14.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:14 smithi154 conmon[34553]: cluster 2021-07-31T02:41:13.457648+0000 mon.a (mon.0) 2861 : cluster [DBG] osdmap e1848: 8 total, 8 up, 8 in 2021-07-31T02:41:14.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:14 smithi045 conmon[33655]: cluster 2021-07-31T02:41:12.575570+0000 mgr.x (mgr.24542) 2312 : cluster [DBG] 2021-07-31T02:41:14.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:14 smithi045 conmon[33655]: pgmap v3313: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.0 MiB/s rd, 3.7 KiB/s wr, 1.39k op/s 2021-07-31T02:41:14.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:14 smithi045 conmon[33655]: cluster 2021-07-31T02:41:13.457648+0000 mon.a (mon.0) 2861 : cluster [DBG] osdmap e1848: 8 total, 8 up, 8 in 2021-07-31T02:41:14.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:14 smithi045 conmon[39718]: cluster 2021-07-31T02:41:12.575570+0000 mgr.x ( 2021-07-31T02:41:14.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:14 smithi045 conmon[39718]: mgr.24542) 2312 : cluster [DBG] pgmap v3313: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.0 MiB/s rd, 3.7 KiB/s wr, 1.39k op/s 2021-07-31T02:41:14.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:14 smithi045 conmon[39718]: cluster 2021-07-31T02:41:13.457648+0000 mon.a (mon.0) 2021-07-31T02:41:14.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:14 smithi045 conmon[39718]: 2861 : cluster [DBG] osdmap e1848: 8 total, 8 up, 8 in 2021-07-31T02:41:15.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:15 smithi154 conmon[34553]: cluster 2021-07-31T02:41:14.469961+0000 mon.a (mon.0) 2862 : cluster [DBG] 2021-07-31T02:41:15.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:15 smithi154 conmon[34553]: osdmap e1849: 8 total, 8 up, 8 in 2021-07-31T02:41:15.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:15 smithi154 conmon[34553]: cluster 2021-07-31T02:41:14.576005+0000 mgr.x (mgr.24542) 2313 : cluster [DBG] pgmap v3316: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.0 MiB/s rd, 3.7 KiB/s wr, 1.44k op/s 2021-07-31T02:41:15.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:15 smithi045 conmon[33655]: cluster 2021-07-31T02:41:14.469961+0000 2021-07-31T02:41:15.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:15 smithi045 conmon[33655]: mon.a (mon.0) 2862 : cluster [DBG] osdmap e1849: 8 total, 8 up, 8 in 2021-07-31T02:41:15.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:15 smithi045 conmon[33655]: cluster 2021-07-31T02:41:14.576005+0000 2021-07-31T02:41:15.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:15 smithi045 conmon[33655]: mgr.x (mgr.24542) 2313 : cluster [DBG] pgmap v3316: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.0 MiB/s rd, 3.7 KiB/s wr, 1.44k op/s 2021-07-31T02:41:15.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:15 smithi045 conmon[39718]: cluster 2021-07-31T02:41:14. 2021-07-31T02:41:15.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:15 smithi045 conmon[39718]: 469961+0000 mon.a (mon.0) 2862 : cluster [DBG] osdmap e1849: 8 total, 8 up, 8 in 2021-07-31T02:41:15.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:15 smithi045 conmon[39718]: cluster 2021-07-31T02:41:14 2021-07-31T02:41:15.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:15 smithi045 conmon[39718]: .576005+0000 mgr.x (mgr.24542) 2313 : cluster [DBG] pgmap v3316: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.0 MiB/s rd, 3.7 KiB/s wr, 1.44k op/s 2021-07-31T02:41:15.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:15 smithi045 conmon[39718]: 2021-07-31T02:41:16.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:16 smithi154 conmon[34553]: cluster 2021-07-31T02:41:15.474802+0000 mon.a (mon.0) 2863 : cluster [DBG] 2021-07-31T02:41:16.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:16 smithi154 conmon[34553]: osdmap e1850: 8 total, 8 up, 8 in 2021-07-31T02:41:16.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:16 smithi045 conmon[33655]: cluster 2021-07-31T02:41:15.474802+0000 mon.a (mon.0) 2021-07-31T02:41:16.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:16 smithi045 conmon[33655]: 2863 : cluster [DBG] osdmap e1850: 8 total, 8 up, 8 in 2021-07-31T02:41:16.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:16 smithi045 conmon[39718]: cluster 2021-07-31T02:41:15.474802+0000 mon.a ( 2021-07-31T02:41:16.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:16 smithi045 conmon[39718]: mon.0) 2863 : cluster [DBG] osdmap e1850: 8 total, 8 up, 8 in 2021-07-31T02:41:17.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:17 smithi154 conmon[34553]: cluster 2021-07-31T02:41:16.476366+0000 mon.a (mon.0) 2021-07-31T02:41:17.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:17 smithi154 conmon[34553]: 2864 : cluster [DBG] osdmap e1851: 8 total, 8 up, 8 in 2021-07-31T02:41:17.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:17 smithi154 conmon[34553]: cluster 2021-07-31T02:41:16.576386+0000 mgr.x (mgr.24542) 2314 : cluster [DBG] pgmap v3319: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.0 MiB/s rd, 3.7 KiB/s wr, 1.44k op/s 2021-07-31T02:41:17.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:17 smithi045 conmon[33655]: cluster 2021-07-31T02:41:16.476366+0000 mon.a (mon.0) 2021-07-31T02:41:17.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:17 smithi045 conmon[33655]: 2864 : cluster [DBG] osdmap e1851: 8 total, 8 up, 8 in 2021-07-31T02:41:17.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:17 smithi045 conmon[33655]: cluster 2021-07-31T02:41:16.576386+0000 mgr.x (mgr.24542) 2314 : cluster [DBG] pgmap v3319: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.0 MiB/s rd, 3.7 KiB/s wr, 1.44k op/s 2021-07-31T02:41:17.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:17 smithi045 conmon[39718]: cluster 2021-07-31T02:41:16 2021-07-31T02:41:17.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:17 smithi045 conmon[39718]: .476366+0000 mon.a (mon.0) 2864 : cluster [DBG] osdmap e1851: 8 total, 8 up, 8 in 2021-07-31T02:41:17.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:17 smithi045 conmon[39718]: cluster 2021-07-31T02:41: 2021-07-31T02:41:17.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:17 smithi045 conmon[39718]: 16.576386+0000 mgr.x (mgr.24542) 2314 : cluster [DBG] pgmap v3319: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.0 MiB/s rd, 3.7 KiB/s wr, 1.44k op/s 2021-07-31T02:41:18.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:18 smithi154 conmon[34553]: cluster 2021-07-31T02:41:17.486746+0000 mon.a (mon.0) 2865 : cluster [DBG] osdmap e1852: 8 total, 8 up, 8 in 2021-07-31T02:41:18.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:18 smithi154 conmon[34553]: 2021-07-31T02:41:18.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:18 smithi045 conmon[33655]: cluster 2021-07-31T02:41:17.486746+0000 mon.a (mon.0) 2865 : cluster [DBG] 2021-07-31T02:41:18.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:18 smithi045 conmon[33655]: osdmap e1852: 8 total, 8 up, 8 in 2021-07-31T02:41:18.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:18 smithi045 conmon[39718]: cluster 2021-07-31T02:41:17.486746+0000 mon.a 2021-07-31T02:41:18.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:18 smithi045 conmon[39718]: (mon.0) 2865 : cluster [DBG] osdmap e1852: 8 total, 8 up, 8 in 2021-07-31T02:41:19.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:19 smithi045 conmon[33655]: cluster 2021-07-31T02:41:18.511826+0000 mon.a (mon.0) 2866 2021-07-31T02:41:19.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:19 smithi045 conmon[33655]: : cluster [DBG] osdmap e1853: 8 total, 8 up, 8 in 2021-07-31T02:41:19.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:19 smithi045 conmon[33655]: cluster 2021-07-31T02:41:18.576909+0000 mgr.x (mgr.24542) 2315 : cluster [DBG] pgmap v3322: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:41:19.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:19 smithi045 conmon[39718]: cluster 2021-07-31T02:41:18.511826+0000 2021-07-31T02:41:19.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:19 smithi045 conmon[39718]: mon.a (mon.0) 2866 : cluster [DBG] osdmap e1853: 8 total, 8 up, 8 in 2021-07-31T02:41:19.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:19 smithi045 conmon[39718]: cluster 2021-07-31T02:41:18.576909+0000 mgr.x (mgr 2021-07-31T02:41:19.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:19 smithi045 conmon[39718]: .24542) 2315 : cluster [DBG] pgmap v3322: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:41:20.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:19 smithi154 conmon[34553]: cluster 2021-07-31T02:41:18.511826+0000 mon.a (mon.0) 2866 : cluster 2021-07-31T02:41:20.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:19 smithi154 conmon[34553]: [DBG] osdmap e1853: 8 total, 8 up, 8 in 2021-07-31T02:41:20.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:19 smithi154 conmon[34553]: cluster 2021-07-31T02:41:18.576909+0000 mgr.x (mgr.24542) 2315 : cluster [DBG] pgmap v3322: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:41:20.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:20 smithi045 conmon[33655]: cluster 2021-07-31T02:41:19.517065+0000 mon.a (mon.0) 2867 : cluster 2021-07-31T02:41:20.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:20 smithi045 conmon[33655]: [DBG] osdmap e1854: 8 total, 8 up, 8 in 2021-07-31T02:41:20.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:20 smithi045 conmon[39718]: cluster 2021-07-31T02:41:19.517065 2021-07-31T02:41:20.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:20 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2867 : cluster [DBG] osdmap e1854: 8 total, 8 up, 8 in 2021-07-31T02:41:21.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:20 smithi154 conmon[34553]: cluster 2021-07-31T02:41:19.517065+0000 mon.a (mon.0) 2867 : cluster [DBG] osdmap e1854: 8 total, 8 up, 8 in 2021-07-31T02:41:22.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:21 smithi154 conmon[34553]: cluster 2021-07-31T02:41:20.519063+0000 mon.a (mon.0) 2868 : cluster [DBG] osdmap e1855: 8 total, 8 up, 8 in 2021-07-31T02:41:22.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:21 smithi154 conmon[34553]: cluster 2021-07-31T02:41:20.577300+0000 mgr.x (mgr.24542) 2316 : cluster [DBG] pgmap v3325: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.1 MiB/s rd, 3.7 KiB/s wr, 1.48k op/s 2021-07-31T02:41:22.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:21 smithi045 conmon[39718]: cluster 2021-07-31T02:41:20.519063 2021-07-31T02:41:22.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:21 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2868 : cluster [DBG] osdmap e1855: 8 total, 8 up, 8 in 2021-07-31T02:41:22.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:21 smithi045 conmon[39718]: cluster 2021-07-31T02:41 2021-07-31T02:41:22.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:21 smithi045 conmon[39718]: :20.577300+0000 mgr.x (mgr.24542) 2316 : cluster [DBG] pgmap v3325: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.1 MiB/s rd, 3.7 KiB/s wr, 1.48k op/s 2021-07-31T02:41:22.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:21 smithi045 conmon[33655]: cluster 2021-07-31T02:41:20.519063+0000 mon.a (mon.0) 2021-07-31T02:41:22.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:21 smithi045 conmon[33655]: 2868 : cluster [DBG] osdmap e1855: 8 total, 8 up, 8 in 2021-07-31T02:41:22.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:21 smithi045 conmon[33655]: cluster 2021-07-31T02:41:20.577300+0000 mgr.x (mgr.24542) 2316 : cluster [DBG] 2021-07-31T02:41:22.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:21 smithi045 conmon[33655]: pgmap v3325: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.1 MiB/s rd, 3.7 KiB/s wr, 1.48k op/s 2021-07-31T02:41:23.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:22 smithi154 conmon[34553]: cluster 2021-07-31T02:41:21.530711+0000 mon.a (mon.0) 2869 : cluster [DBG] osdmap e1856: 8 total, 8 up, 8 in 2021-07-31T02:41:23.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:22 smithi045 conmon[39718]: cluster 2021-07-31T02:41:21.530711+0000 mon.a (mon.0) 2869 : cluster 2021-07-31T02:41:23.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:22 smithi045 conmon[39718]: [DBG] osdmap e1856: 8 total, 8 up, 8 in 2021-07-31T02:41:23.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:22 smithi045 conmon[33655]: cluster 2021-07-31T02:41:21.530711+0000 mon.a (mon.0) 2021-07-31T02:41:23.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:22 smithi045 conmon[33655]: 2869 : cluster [DBG] osdmap e1856: 8 total, 8 up, 8 in 2021-07-31T02:41:24.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:23 smithi154 conmon[34553]: cluster 2021-07-31T02:41:22.535441+0000 mon.a (mon.0) 2870 : cluster [DBG] osdmap e1857: 8 total, 8 up, 8 in 2021-07-31T02:41:24.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:23 smithi154 conmon[34553]: cluster 2021-07-31T02:41:22.577682+0000 mgr.x (mgr.24542) 2317 : cluster [DBG] pgmap v3328: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.1 MiB/s rd, 3.7 KiB/s wr, 1.48k op/s 2021-07-31T02:41:24.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:23 smithi045 conmon[39718]: cluster 2021-07-31T02:41:24.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:23 smithi045 conmon[39718]: 2021-07-31T02:41: 2021-07-31T02:41:24.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:23 smithi045 conmon[39718]: 22.535441+0000 mon.a (mon.0) 2870 : cluster [DBG] osdmap e1857: 8 total, 8 up, 8 in 2021-07-31T02:41:24.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:23 smithi045 conmon[39718]: cluster 2021-07-31T02:41:24.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:23 smithi045 conmon[39718]: 2021-07-31 2021-07-31T02:41:24.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:23 smithi045 conmon[39718]: T02:41:22.577682+0000 2021-07-31T02:41:24.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:23 smithi045 conmon[39718]: mgr.x (mgr.24542) 2317 : 2021-07-31T02:41:24.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:23 smithi045 conmon[39718]: cluster [DBG] pgmap v3328: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.1 MiB/s rd, 3.7 KiB/s wr, 1.48k op/s 2021-07-31T02:41:24.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:23 smithi045 conmon[33655]: cluster 2021-07-31T02:41:22.535441+0000 2021-07-31T02:41:24.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:23 smithi045 conmon[33655]: mon.a (mon.0) 2870 : cluster [DBG] osdmap e1857: 8 total, 8 up, 8 in 2021-07-31T02:41:24.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:23 smithi045 conmon[33655]: cluster 2021-07-31T02:41:22.577682+0000 mgr.x (mgr.24542) 2317 : cluster [DBG] pgmap v3328: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.1 MiB/s rd, 3.7 KiB/s wr, 1.48k op/s 2021-07-31T02:41:25.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:24 smithi154 conmon[34553]: cluster 2021-07-31T02:41:23.543041+0000 mon.a (mon.0) 2871 : cluster [DBG] osdmap e1858: 8 total, 8 up, 8 in 2021-07-31T02:41:25.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:24 smithi045 conmon[39718]: cluster 2021-07-31T02:41:23 2021-07-31T02:41:25.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:24 smithi045 conmon[39718]: .543041+0000 mon.a (mon.0) 2871 : cluster [DBG] osdmap e1858: 8 total, 8 up, 8 in 2021-07-31T02:41:25.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:24 smithi045 conmon[33655]: cluster 2021-07-31T02:41:23.543041+0000 mon.a (mon.0) 2871 : cluster 2021-07-31T02:41:25.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:24 smithi045 conmon[33655]: [DBG] osdmap e1858: 8 total, 8 up, 8 in 2021-07-31T02:41:26.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:25 smithi154 conmon[34553]: cluster 2021-07-31T02:41:24.544745+0000 mon.a (mon.0) 2872 : cluster 2021-07-31T02:41:26.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:25 smithi154 conmon[34553]: [DBG] osdmap e1859: 8 total, 8 up, 8 in 2021-07-31T02:41:26.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:25 smithi154 conmon[34553]: cluster 2021-07-31T02:41:24.578072+0000 mgr.x (mgr.24542) 2318 : cluster [DBG] pgmap v3331: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.1 MiB/s rd, 3.7 KiB/s wr, 1.52k op/s 2021-07-31T02:41:26.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:25 smithi045 conmon[39718]: cluster 2021-07-31T02:41:24.544745+0000 mon.a ( 2021-07-31T02:41:26.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:25 smithi045 conmon[39718]: mon.0) 2872 : cluster [DBG] osdmap e1859: 8 total, 8 up, 8 in 2021-07-31T02:41:26.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:25 smithi045 conmon[39718]: cluster 2021-07-31T02:41:24.578072+0000 mgr.x (mgr.24542) 2021-07-31T02:41:26.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:25 smithi045 conmon[39718]: 2318 : cluster [DBG] pgmap v3331: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.1 MiB/s rd, 3.7 KiB/s wr, 1.52k op/s 2021-07-31T02:41:26.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:25 smithi045 conmon[33655]: cluster 2021-07-31T02:41:24.544745+0000 2021-07-31T02:41:26.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:25 smithi045 conmon[33655]: mon.a (mon.0) 2872 : cluster [DBG] osdmap e1859: 8 total, 8 up, 8 in 2021-07-31T02:41:26.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:25 smithi045 conmon[33655]: cluster 2021-07-31T02:41:24.578072+0000 mgr.x (mgr.24542) 2318 : cluster [DBG] pgmap v3331: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.1 MiB/s rd, 3.7 KiB/s wr, 1.52k op/s 2021-07-31T02:41:27.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:26 smithi154 conmon[34553]: cluster 2021-07-31T02:41:25.555444+0000 mon.a (mon.0) 2873 : cluster [DBG] osdmap e1860: 8 total, 8 up, 8 in 2021-07-31T02:41:27.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:26 smithi154 conmon[34553]: 2021-07-31T02:41:27.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:26 smithi045 conmon[39718]: cluster 2021-07-31T02:41:25.555444+0000 mon.a ( 2021-07-31T02:41:27.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:26 smithi045 conmon[39718]: mon.0) 2873 : cluster [DBG] osdmap e1860: 8 total, 8 up, 8 in 2021-07-31T02:41:27.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:26 smithi045 conmon[33655]: cluster 2021-07-31T02:41:25.555444+0000 mon.a 2021-07-31T02:41:27.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:26 smithi045 conmon[33655]: (mon.0) 2873 : cluster [DBG] osdmap e1860: 8 total, 8 up, 8 in 2021-07-31T02:41:28.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:27 smithi154 conmon[34553]: cluster 2021-07-31T02:41:26.557244+0000 mon.a (mon.0) 2874 : cluster [DBG] osdmap e1861: 8 total, 8 up, 8 in 2021-07-31T02:41:28.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:27 smithi154 conmon[34553]: cluster 2021-07-31T02:41:26.578432+0000 mgr.x (mgr.24542) 2319 : cluster [DBG] pgmap v3334: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.1 MiB/s rd, 3.7 KiB/s wr, 1.52k op/s 2021-07-31T02:41:28.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:27 smithi045 conmon[39718]: cluster 2021-07-31T02:41:26.557244 2021-07-31T02:41:28.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:27 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2874 : cluster [DBG] osdmap e1861: 8 total, 8 up, 8 in 2021-07-31T02:41:28.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:27 smithi045 conmon[39718]: cluster 2021 2021-07-31T02:41:28.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:27 smithi045 conmon[39718]: -07-31T02:41:26.578432+0000 mgr.x (mgr.24542 2021-07-31T02:41:28.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:27 smithi045 conmon[39718]: ) 2319 : cluster [DBG] pgmap v3334: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.1 MiB/s rd, 3.7 KiB/s wr, 1.52k op/s 2021-07-31T02:41:28.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:27 smithi045 conmon[33655]: cluster 2021-07-31T02:41:26.557244+0000 mon.a (mon.0) 2874 : cluster 2021-07-31T02:41:28.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:27 smithi045 conmon[33655]: [DBG] osdmap e1861: 8 total, 8 up, 8 in 2021-07-31T02:41:28.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:27 smithi045 conmon[33655]: cluster 2021-07-31T02:41:26.578432+0000 mgr.x (mgr.24542) 2319 : cluster [DBG] pgmap v3334: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.1 MiB/s rd, 3.7 KiB/s wr, 1.52k op/s 2021-07-31T02:41:29.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:28 smithi154 conmon[34553]: cluster 2021-07-31T02:41:27.566563+0000 mon.a (mon.0) 2875 : cluster [DBG] osdmap e1862: 8 total, 8 up, 8 in 2021-07-31T02:41:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:28 smithi154 conmon[34553]: audit 2021-07-31T02:41:28.138627+0000 mon.b (mon.2) 345 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:41:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:28 smithi154 conmon[34553]: audit 2021-07-31T02:41:28.139221+0000 mon.a (mon.0) 2876 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:41:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:28 smithi154 conmon[34553]: audit 2021-07 2021-07-31T02:41:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:28 smithi154 conmon[34553]: -31T02:41:28.147750+0000 mon.b (mon.2) 346 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:41:29.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:28 smithi154 conmon[34553]: audit 2021-07-31T02:41:28.148372 2021-07-31T02:41:29.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:28 smithi154 conmon[34553]: +0000 mon.a (mon.0) 2877 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:41:29.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:28 smithi045 conmon[39718]: cluster 2021-07-31T02:41:27.566563+0000 2021-07-31T02:41:29.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:28 smithi045 conmon[39718]: mon.a (mon.0) 2875 : cluster [DBG] osdmap e1862: 8 total, 8 up, 8 in 2021-07-31T02:41:29.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:28 smithi045 conmon[39718]: audit 2021-07-31T02:41:29.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:28 smithi045 conmon[39718]: 2021-07-31T02:41:28.138627+0000 mon.b ( 2021-07-31T02:41:29.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:28 smithi045 conmon[39718]: mon.2) 345 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:41:29.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:28 smithi045 conmon[39718]: audit 2021 2021-07-31T02:41:29.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:28 smithi045 conmon[39718]: -07-31T02:41:28.139221 2021-07-31T02:41:29.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:28 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2876 : audit 2021-07-31T02:41:29.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:28 smithi045 conmon[39718]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:41:29.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:28 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T02:41:29.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:28 smithi045 conmon[39718]: 02:41:28.147750+0000 mon.b ( 2021-07-31T02:41:29.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:28 smithi045 conmon[39718]: mon.2) 346 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:41:29.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:28 smithi045 conmon[39718]: 2021-07-31T02:41:29.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:28 smithi045 conmon[39718]: audit 2021-07-31T02:41:29.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:28 smithi045 conmon[39718]: 2021-07-31T02:41:28.148372 2021-07-31T02:41:29.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:28 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2877 : audit 2021-07-31T02:41:29.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:28 smithi045 conmon[39718]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:41:29.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:28 smithi045 conmon[33655]: cluster 2021-07-31T02:41:27.566563+0000 2021-07-31T02:41:29.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:28 smithi045 conmon[33655]: mon.a (mon.0) 2875 : cluster [DBG] osdmap e1862: 8 total, 8 up, 8 in 2021-07-31T02:41:29.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:28 smithi045 conmon[33655]: audit 2021-07-31T02:41:28 2021-07-31T02:41:29.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:28 smithi045 conmon[33655]: .138627+0000 mon.b (mon.2) 345 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:41:29.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:28 smithi045 conmon[33655]: audit 2021-07-31T02:41:29.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:28 smithi045 conmon[33655]: 2021-07-31T 2021-07-31T02:41:29.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:28 smithi045 conmon[33655]: 02:41:28.139221+0000 2021-07-31T02:41:29.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:28 smithi045 conmon[33655]: mon.a (mon.0) 2876 2021-07-31T02:41:29.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:28 smithi045 conmon[33655]: : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:41:29.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:28 smithi045 conmon[33655]: audit 2021 2021-07-31T02:41:29.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:28 smithi045 conmon[33655]: -07-31T02:41:28.147750 2021-07-31T02:41:29.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:28 smithi045 conmon[33655]: +0000 mon.b (mon.2) 346 2021-07-31T02:41:29.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:28 smithi045 conmon[33655]: : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:41:29.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:28 smithi045 conmon[33655]: audit 2021- 2021-07-31T02:41:29.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:28 smithi045 conmon[33655]: 07-31T02:41:28. 2021-07-31T02:41:29.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:28 smithi045 conmon[33655]: 148372+0000 mon.a (mon.0) 2021-07-31T02:41:29.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:28 smithi045 conmon[33655]: 2877 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:41:30.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:29 smithi154 conmon[34553]: cluster 2021-07-31T02:41:28.567765+0000 mon.a (mon.0) 2878 : cluster [DBG] osdmap e1863: 8 total, 8 up, 8 in 2021-07-31T02:41:30.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:29 smithi154 conmon[34553]: cluster 2021-07-31T02:41:28.578762+0000 mgr.x (mgr.24542) 2320 : cluster [DBG] pgmap v3337: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:41:30.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:29 smithi045 conmon[33655]: cluster 2021-07-31T02:41:28.567765+0000 mon.a (mon.0) 2021-07-31T02:41:30.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:29 smithi045 conmon[33655]: 2878 : cluster [DBG] osdmap e1863: 8 total, 8 up, 8 in 2021-07-31T02:41:30.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:29 smithi045 conmon[33655]: cluster 2021-07-31T02:41:28.578762+0000 mgr.x (mgr.24542) 2320 : cluster [DBG] pgmap v3337: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:41:30.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:29 smithi045 conmon[39718]: cluster 2021-07-31T02:41:28.567765+0000 mon.a (mon.0) 2021-07-31T02:41:30.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:29 smithi045 conmon[39718]: 2878 : cluster [DBG] osdmap e1863: 8 total, 8 up, 8 in 2021-07-31T02:41:30.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:29 smithi045 conmon[39718]: cluster 2021-07-31T02:41:28.578762+0000 mgr.x (mgr.24542 2021-07-31T02:41:30.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:29 smithi045 conmon[39718]: ) 2320 : cluster [DBG] pgmap v3337: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:41:31.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:30 smithi154 conmon[34553]: cluster 2021-07-31T02:41:29.578316+0000 mon.a (mon.0) 2879 : cluster [DBG] osdmap e1864: 8 total, 8 up, 8 in 2021-07-31T02:41:31.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:30 smithi045 conmon[33655]: cluster 2021-07-31T02:41:29.578316+0000 mon.a (mon.0) 2879 : cluster 2021-07-31T02:41:31.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:30 smithi045 conmon[33655]: [DBG] osdmap e1864: 8 total, 8 up, 8 in 2021-07-31T02:41:31.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:30 smithi045 conmon[39718]: cluster 2021-07-31T02:41:29.578316+0000 mon.a (mon 2021-07-31T02:41:31.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:30 smithi045 conmon[39718]: .0) 2879 : cluster [DBG] osdmap e1864: 8 total, 8 up, 8 in 2021-07-31T02:41:32.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:31 smithi154 conmon[34553]: cluster 2021-07-31T02:41:30.579127+0000 mgr.x (mgr.24542) 2321 : cluster [DBG] pgmap v3339: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.1 MiB/s rd, 3.7 KiB/s wr, 1.56k op/s 2021-07-31T02:41:32.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:31 smithi154 conmon[34553]: cluster 2021-07-31T02:41:30.584301+0000 mon.a (mon.0) 2880 : cluster [DBG] osdmap e1865: 8 total, 8 up, 8 in 2021-07-31T02:41:32.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:31 smithi045 conmon[33655]: cluster 2021-07-31T02:41:30.579127+0000 mgr.x (mgr.24542) 2321 2021-07-31T02:41:32.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:31 smithi045 conmon[33655]: : cluster [DBG] pgmap v3339: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.1 MiB/s rd, 3.7 KiB/s wr, 1.56k op/s 2021-07-31T02:41:32.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:31 smithi045 conmon[33655]: cluster 2021-07-31T02:41:30.584301+0000 mon.a (mon.0) 2880 : cluster [DBG] osdmap e1865: 8 total, 8 up, 8 in 2021-07-31T02:41:32.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:31 smithi045 conmon[39718]: cluster 2021-07-31T02:41:30.579127+0000 mgr.x (mgr.24542 2021-07-31T02:41:32.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:31 smithi045 conmon[39718]: ) 2321 : cluster [DBG] pgmap v3339: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.1 MiB/s rd, 3.7 KiB/s wr, 1.56k op/s 2021-07-31T02:41:32.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:31 smithi045 conmon[39718]: cluster 2021-07-31T02:41:30.584301+0000 mon.a (mon.0) 2880 : cluster 2021-07-31T02:41:32.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:31 smithi045 conmon[39718]: [DBG] osdmap e1865: 8 total, 8 up, 8 in 2021-07-31T02:41:33.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:32 smithi154 conmon[34553]: cluster 2021-07-31T02:41:31.586817+0000 mon.a (mon.0) 2881 : cluster [DBG] osdmap e1866: 8 total, 8 up, 8 in 2021-07-31T02:41:33.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:32 smithi045 conmon[33655]: cluster 2021-07-31T02:41:31.586817+0000 mon.a (mon.0) 2881 : cluster [DBG] osdmap e1866: 8 total, 8 up, 8 in 2021-07-31T02:41:33.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:32 smithi045 conmon[39718]: cluster 2021-07-31T02:41:31.586817+0000 mon.a (mon. 2021-07-31T02:41:33.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:32 smithi045 conmon[39718]: 0) 2881 : cluster [DBG] osdmap e1866: 8 total, 8 up, 8 in 2021-07-31T02:41:33.611 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: debug 2021-07-31T02:41:33.341+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:41:33.611 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:41:33.612 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:41:33.613 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: yield 2021-07-31T02:41:33.613 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:41:33.613 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:41:33.613 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:41:33.614 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:41:33.614 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:41:33.614 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:41:33.614 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:41:33.615 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:41:33.615 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:41:33.615 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: task.result() 2021-07-31T02:41:33.615 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:41:33.616 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: debug 2021-07-31T02:41:33.352+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:41:33.616 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:41:33.616 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:41:33.616 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: yield 2021-07-31T02:41:33.617 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:41:33.617 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:41:33.617 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:41:33.617 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:41:33.618 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:41:33.618 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:41:33.618 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:41:33.618 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:41:33.619 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:41:33.619 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: task.result() 2021-07-31T02:41:33.619 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:41:33.619 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: debug 2021-07-31T02:41:33.378+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:41:33.619 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:41:33.620 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:41:33.620 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: yield 2021-07-31T02:41:33.620 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:41:33.620 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:41:33.621 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:41:33.621 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:41:33.621 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:41:33.621 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:41:33.622 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:41:33.622 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:41:33.622 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:41:33.622 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: task.result() 2021-07-31T02:41:33.623 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:41:33.623 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: debug 2021-07-31T02:41:33.379+0000 7fcb97969700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:41:33.623 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:41:33.623 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:41:33.623 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: yield 2021-07-31T02:41:33.624 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:41:33.624 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:41:33.624 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:41:33.624 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:41:33.625 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:41:33.625 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:41:33.625 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:41:33.625 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:41:33.625 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:41:33.626 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: task.result() 2021-07-31T02:41:33.626 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:41:33.626 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: 2021-07-31T02:41:33.626 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:41:33.627 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: 2021-07-31T02:41:33.627 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:41:33.627 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:41:33.627 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:41:33.627 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:41:33.628 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:41:33.628 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:41:33.628 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:41:33.628 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:41:33.629 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:41:33.629 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:41:33.629 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:41:33.629 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:41:33.630 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:41:33.630 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:41:33.630 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:41:33.630 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:41:33.630 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:41:33.631 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:41:33.632 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: raise 2021-07-31T02:41:33.632 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:41:33.632 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:41:33.632 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:41:33.633 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:41:33.633 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:41:33.633 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: debug 2021-07-31T02:41:33.387+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:41:33.633 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:41:33.633 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:41:33.634 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: yield 2021-07-31T02:41:33.634 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:41:33.634 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:41:33.634 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:41:33.635 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:41:33.635 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:41:33.635 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:41:33.635 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:41:33.635 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:41:33.636 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:41:33.636 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: task.result() 2021-07-31T02:41:33.636 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:41:33.636 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: debug 2021-07-31T02:41:33.388+0000 7fcb9816a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:41:33.636 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:41:33.637 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:41:33.637 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: yield 2021-07-31T02:41:33.637 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:41:33.637 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:41:33.638 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:41:33.638 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:41:33.638 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:41:33.638 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:41:33.638 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:41:33.639 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:41:33.639 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:41:33.639 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: task.result() 2021-07-31T02:41:33.639 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:41:33.639 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: 2021-07-31T02:41:33.640 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:41:33.640 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: 2021-07-31T02:41:33.640 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:41:33.640 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:41:33.641 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:41:33.641 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:41:33.641 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:41:33.641 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:41:33.641 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:41:33.642 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:41:33.642 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:41:33.642 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:41:33.643 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:41:33.643 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:41:33.643 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:41:33.643 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:41:33.644 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:41:33.644 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:41:33.644 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:41:33.644 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:41:33.644 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: raise 2021-07-31T02:41:33.645 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:41:33.645 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:41:33.645 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:41:33.645 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:41:33.646 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:41:34.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: cluster 2021-07-31T02:41:32.579536+0000 mgr.x (mgr.24542) 2322 : cluster [DBG] 2021-07-31T02:41:34.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: pgmap v3342: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.1 MiB/s rd, 3.7 KiB/s wr, 1.56k op/s 2021-07-31T02:41:34.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: cluster 2021-07-31T02:41:32.601244+0000 mon.a (mon.0) 2882 : cluster [DBG] osdmap e1867: 8 total, 8 up, 8 in 2021-07-31T02:41:34.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: audit 2021- 2021-07-31T02:41:34.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: 07-31T02:41:33.294789+0000 mon.b (mon.2) 347 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:41:34.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: audit 2021-07-31T02:41:34.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: 2021-07-31T02:41: 2021-07-31T02:41:34.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: 33.296337+0000 mon.b (mon.2) 348 : audit 2021-07-31T02:41:34.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:41:34.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: audit 2021-07 2021-07-31T02:41:34.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: -31T02:41:33.297238+0000 mon.b (mon 2021-07-31T02:41:34.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: .2) 349 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:41:34.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: cephadm 2021- 2021-07-31T02:41:34.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: 07-31T02:41:33.297998+0000 2021-07-31T02:41:34.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: mgr.x (mgr.24542) 2323 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:41:34.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: 2021-07-31T02:41:34.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: cephadm 2021-07-31 2021-07-31T02:41:34.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: T02:41:33.341992+0000 mgr.x (mgr.24542) 2021-07-31T02:41:34.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: 2324 : cephadm [ERR] 2021-07-31T02:41:34.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:41:34.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:41:34.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: yield 2021-07-31T02:41:34.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:41:34.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:41:34.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:41:34.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:41:34.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:41:34.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:41:34.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:41:34.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:41:34.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:41:34.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:41:34.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:41:34.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:41:33.342439+0000 mgr.x (mgr.24542) 2325 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:41:34.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:41:33.353484+0000 mgr.x (mgr.24542) 2326 : cephadm [ERR] 2021-07-31T02:41:34.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:41:34.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:41:34.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: yield 2021-07-31T02:41:34.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:41:34.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:41:34.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:41:34.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:41:34.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:41:34.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:41:34.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:41:34.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:41:34.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:41:34.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:41:34.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:41:34.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:41:33.354291+0000 mgr.x (mgr.24542) 2327 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:41:34.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:41:33.379643+0000 mgr.x (mgr.24542) 2328 : cephadm [ERR] 2021-07-31T02:41:34.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:41:34.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:41:34.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: yield 2021-07-31T02:41:34.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:41:34.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:41:34.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:41:34.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:41:34.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:41:34.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:41:34.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:41:34.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:41:34.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:41:34.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:41:34.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:41:34.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:41:33.380107+0000 mgr.x (mgr.24542) 2329 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:41:34.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:41:34.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:41:34.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: yield 2021-07-31T02:41:34.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:41:34.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:41:34.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:41:34.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:41:34.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:41:34.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:41:34.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:41:34.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:41:34.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:41:34.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:41:34.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:41:34.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: 2021-07-31T02:41:34.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:41:34.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: 2021-07-31T02:41:34.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:41:34.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:41:34.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:41:34.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:41:34.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:41:34.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:41:34.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:41:34.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:41:34.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:41:34.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:41:34.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:41:34.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:41:34.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:41:34.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:41:34.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:41:34.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:41:34.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:41:34.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:41:34.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: raise 2021-07-31T02:41:34.035 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:41:34.035 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:41:34.035 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:41:34.036 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:41:34.036 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:41:34.036 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: cephadm 2021-07 2021-07-31T02:41:34.036 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: -31T02:41:33.388396+0000 mgr.x (mgr.24542) 2330 : cephadm [ERR] 2021-07-31T02:41:34.037 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:41:34.037 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:41:34.037 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: yield 2021-07-31T02:41:34.038 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:41:34.039 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:41:34.039 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:41:34.039 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:41:34.040 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:41:34.040 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:41:34.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: cluster 2021-07-31T02:41:32.579536+0000 mgr.x (mgr.24542) 2322 : 2021-07-31T02:41:34.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: cluster [DBG] pgmap v3342: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.1 MiB/s rd, 3.7 KiB/s wr, 1.56k op/s 2021-07-31T02:41:34.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: cluster 2021-07-31T02:41:32.601244+0000 mon.a (mon.0) 2882 : cluster 2021-07-31T02:41:34.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: [DBG] osdmap e1867: 8 total, 8 up, 8 in 2021-07-31T02:41:34.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T02:41:34.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: 02:41:33.294789+0000 mon.b (mon.2) 347 : audit [DBG] 2021-07-31T02:41:34.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:41:34.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: audit 2021-07-31T02:41: 2021-07-31T02:41:34.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: 33.296337+0000 mon.b (mon.2) 348 : audit [DBG] 2021-07-31T02:41:34.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:41:34.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: audit 2021-07-31T02:41:34.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: 2021-07-31T02:41:33.297238+0000 mon.b ( 2021-07-31T02:41:34.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: mon.2) 349 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:41:34.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: cephadm 2021-07- 2021-07-31T02:41:34.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: 31T02:41:33.297998+0000 mgr.x (mgr 2021-07-31T02:41:34.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: .24542) 2323 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:41:34.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: cephadm 2021-07- 2021-07-31T02:41:34.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: 31T02:41:33.341992+0000 mgr.x (mgr 2021-07-31T02:41:34.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: .24542) 2324 : cephadm [ERR] 2021-07-31T02:41:34.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:41:34.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:41:34.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: yield 2021-07-31T02:41:34.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:41:34.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:41:34.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:41:34.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:41:34.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:41:34.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:41:34.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:41:34.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:41:34.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:41:34.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:41:34.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:41:34.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:41:33.342439+0000 mgr.x (mgr.24542) 2325 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:41:34.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: cephadm 2021-07-31T02 2021-07-31T02:41:34.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: :41:33.353484+0000 mgr.x (mgr.24542) 2326 : cephadm [ERR] 2021-07-31T02:41:34.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:41:34.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:41:34.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: yield 2021-07-31T02:41:34.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:41:34.051 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:41:34.051 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:41:34.052 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:41:34.052 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:41:34.052 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:41:34.052 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:41:33.388949+0000 mgr.x (mgr.24542) 2331 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:41:34.052 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:41:34.053 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:41:34.053 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: yield 2021-07-31T02:41:34.053 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:41:34.053 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:41:34.054 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:41:34.054 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:41:34.054 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:41:34.054 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:41:34.054 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:41:34.055 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:41:34.055 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:41:34.055 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:41:34.055 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:41:34.056 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: 2021-07-31T02:41:34.056 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:41:34.056 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: 2021-07-31T02:41:34.056 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:41:34.056 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:41:34.057 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:41:34.057 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:41:34.057 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:41:34.057 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:41:34.057 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:41:34.058 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:41:34.058 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:41:34.058 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:41:34.058 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:41:34.059 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:41:34.059 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:41:34.059 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:41:34.059 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:41:34.059 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:41:34.060 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:41:34.060 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:41:34.060 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: raise 2021-07-31T02:41:34.060 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:41:34.060 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:41:34.061 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:41:34.061 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:41:34.061 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:33 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:41:34.061 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: cluster 2021-07-31T02:41:32.579536+0000 mgr.x (mgr.24542 2021-07-31T02:41:34.062 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: ) 2322 : cluster [DBG] pgmap v3342: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.1 MiB/s rd, 3.7 KiB/s wr, 1.56k op/s 2021-07-31T02:41:34.062 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: cluster 2021-07-31T02:41:32.601244+0000 mon.a (mon.0) 2882 : cluster [DBG] osdmap e1867: 8 total, 8 up, 8 in 2021-07-31T02:41:34.062 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: audit 2021 2021-07-31T02:41:34.062 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: -07-31T02:41:33.294789+0000 mon.b (mon.2) 347 : audit 2021-07-31T02:41:34.062 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:41:34.063 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: audit 2021-07-31T02:41:34.063 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: 2021-07-31T02:41:33.296337 2021-07-31T02:41:34.064 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: +0000 mon.b (mon.2) 348 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:41:34.064 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: audit 2021 2021-07-31T02:41:34.065 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: -07-31T02:41:33.297238+0000 mon.b (mon 2021-07-31T02:41:34.065 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: .2) 349 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:41:34.065 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:41:34.065 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: 2021-07-31T02:41:33.297998+0000 2021-07-31T02:41:34.066 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: mgr.x (mgr.24542) 2323 : cephadm [WRN] 2021-07-31T02:41:34.066 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:41:34.066 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: cephadm 2021-07 2021-07-31T02:41:34.066 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: -31T02:41:33.341992+0000 mgr.x (mgr.24542) 2324 : cephadm 2021-07-31T02:41:34.066 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: [ERR] 2021-07-31T02:41:34.067 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:41:34.067 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:41:34.067 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: yield 2021-07-31T02:41:34.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:41:34.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:41:34.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:41:34.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:41:34.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:41:34.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:41:34.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:41:34.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:41:34.070 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:41:34.070 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:41:34.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:41:34.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:41:33.342439+0000 mgr.x (mgr.24542) 2325 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:41:34.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: cephadm 2021- 2021-07-31T02:41:34.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: 07-31T02:41:33.353484+0000 mgr.x 2021-07-31T02:41:34.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: (mgr.24542) 2326 : cephadm 2021-07-31T02:41:34.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: [ERR] 2021-07-31T02:41:34.073 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:41:34.073 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:41:34.073 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: yield 2021-07-31T02:41:34.074 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:41:34.075 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:41:34.075 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:41:34.075 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:41:34.076 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:41:34.076 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:41:34.076 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:41:34.077 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:41:34.077 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:41:34.077 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:41:34.078 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:41:33.354291+0000 mgr.x (mgr.24542) 2327 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:41:34.078 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: cephadm 2021-07-31 2021-07-31T02:41:34.078 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: T02:41:33.379643+0000 mgr.x (mgr.24542) 2328 : cephadm 2021-07-31T02:41:34.079 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: [ERR] 2021-07-31T02:41:34.079 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:41:34.079 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:41:34.080 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: yield 2021-07-31T02:41:34.080 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:41:34.080 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:41:34.081 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:41:34.081 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:41:34.081 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:41:34.082 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:41:34.082 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:41:34.082 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:41:34.082 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:41:34.083 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:41:34.083 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:41:34.083 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:41:33.380107+0000 mgr.x (mgr.24542) 2329 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:41:34.083 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:41:34.084 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:41:34.084 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: yield 2021-07-31T02:41:34.084 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:41:34.084 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:41:34.084 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:41:34.085 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:41:34.085 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:41:34.085 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:41:34.085 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:41:34.086 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:41:34.086 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:41:34.087 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:41:34.087 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:41:34.087 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:41:34.087 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:41:34.087 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:41:34.088 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:41:34.088 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:41:34.088 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:41:34.088 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:41:34.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:41:33.354291+0000 mgr.x (mgr.24542) 2327 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:41:34.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: cephadm 2021- 2021-07-31T02:41:34.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: 07-31T02:41:33.379643+0000 mgr.x (mgr.24542) 2328 : cephadm [ERR] 2021-07-31T02:41:34.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:41:34.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:41:34.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: yield 2021-07-31T02:41:34.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:41:34.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:41:34.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:41:34.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:41:34.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:41:34.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:41:34.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:41:34.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:41:34.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:41:34.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:41:34.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:41:34.093 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:41:33.380107+0000 mgr.x (mgr.24542) 2329 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:41:34.093 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:41:34.093 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:41:34.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: yield 2021-07-31T02:41:34.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:41:34.095 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:41:34.095 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:41:34.095 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:41:34.096 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:41:34.096 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:41:34.097 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:41:34.097 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:41:34.097 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:41:34.098 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:41:34.098 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:41:34.099 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: 2021-07-31T02:41:34.099 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:41:34.100 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: 2021-07-31T02:41:34.100 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:41:34.100 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:41:34.101 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:41:34.101 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:41:34.102 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:41:34.102 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:41:34.102 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:41:34.103 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:41:34.103 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:41:34.104 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:41:34.104 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:41:34.105 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:41:34.105 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:41:34.105 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:41:34.106 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:41:34.106 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:41:34.107 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:41:34.107 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:41:34.107 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: raise 2021-07-31T02:41:34.108 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:41:34.108 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:41:34.109 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:41:34.109 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:41:34.109 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:41:34.110 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:41:33.388396+0000 mgr.x (mgr.24542) 2330 : cephadm [ERR] 2021-07-31T02:41:34.110 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:41:34.111 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:41:34.111 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: yield 2021-07-31T02:41:34.112 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:41:34.112 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:41:34.112 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:41:34.113 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:41:34.113 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:41:34.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:41:34.115 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:41:34.115 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:41:34.115 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:41:34.116 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:41:34.116 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: 2021-07-31T02:41:34.117 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:41:34.117 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: 2021-07-31T02:41:34.117 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:41:34.118 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:41:34.118 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:41:34.119 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:41:34.119 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:41:34.120 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:41:34.120 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:41:34.120 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:41:34.121 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:41:34.121 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:41:34.121 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:41:34.122 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:41:34.122 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:41:34.123 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:41:34.128 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:41:34.128 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:41:34.129 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:41:34.129 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:41:34.129 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: raise 2021-07-31T02:41:34.130 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:41:34.130 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:41:34.130 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:41:34.131 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:41:34.131 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:41:34.131 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:41:33.388396 2021-07-31T02:41:34.131 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 2330 : cephadm [ERR] 2021-07-31T02:41:34.131 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:41:34.132 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:41:34.132 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: yield 2021-07-31T02:41:34.132 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:41:34.133 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:41:34.133 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:41:34.133 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:41:34.133 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:41:34.133 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:41:34.134 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:41:34.135 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:41:34.135 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:41:34.135 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:41:34.135 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:41:34.136 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:41:33.388949+0000 mgr.x (mgr.24542) 2331 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:41:34.136 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:41:34.136 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:41:34.136 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: yield 2021-07-31T02:41:34.136 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:41:34.137 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:41:34.137 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:41:34.137 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:41:34.137 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:41:34.138 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:41:34.138 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:41:34.138 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:41:34.138 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:41:34.138 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:41:34.139 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:41:34.139 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: 2021-07-31T02:41:34.139 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:41:34.140 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: 2021-07-31T02:41:34.140 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:41:34.140 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:41:34.140 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:41:34.140 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:41:34.141 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:41:34.141 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:41:34.141 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:41:34.141 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:41:34.141 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:41:34.142 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:41:34.142 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:41:34.142 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:41:34.143 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:41:34.143 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:41:34.143 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:41:34.143 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:41:34.144 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:41:34.144 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:41:34.144 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: raise 2021-07-31T02:41:34.144 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:41:34.145 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:41:34.145 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:41:34.146 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:41:34.146 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:41:34.147 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:41:34.147 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:41:34.147 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:41:34.147 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:41:34.148 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:41:34.148 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:41:33.388949+0000 mgr.x (mgr.24542) 2331 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:41:34.148 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:41:34.148 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:41:34.149 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: yield 2021-07-31T02:41:34.149 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:41:34.149 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:41:34.150 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:41:34.150 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:41:34.150 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:41:34.150 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:41:34.151 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:41:34.151 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:41:34.151 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:41:34.151 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:41:34.152 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:41:34.152 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: 2021-07-31T02:41:34.152 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:41:34.152 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: 2021-07-31T02:41:34.152 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:41:34.153 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:41:34.153 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:41:34.153 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:41:34.154 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:41:34.154 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:41:34.154 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:41:34.154 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:41:34.154 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:41:34.155 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:41:34.155 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:41:34.155 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:41:34.155 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:41:34.155 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:41:34.156 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:41:34.156 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:41:34.156 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:41:34.157 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:41:34.157 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: raise 2021-07-31T02:41:34.157 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:41:34.157 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:41:34.157 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:41:34.158 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:41:34.158 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:33 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:41:34.946 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:41:35.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:34 smithi154 conmon[34553]: cluster 2021-07-31T02:41:33.603221+0000 mon.a (mon.0) 2883 : cluster [DBG] osdmap e1868: 8 total, 8 up, 8 in 2021-07-31T02:41:35.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:34 smithi045 conmon[39718]: cluster 2021-07-31T02:41:33.603221+0000 mon.a (mon. 2021-07-31T02:41:35.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:34 smithi045 conmon[39718]: 0) 2883 : cluster [DBG] osdmap e1868: 8 total, 8 up, 8 in 2021-07-31T02:41:35.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:34 smithi045 conmon[33655]: cluster 2021-07-31T02:41:33.603221+0000 mon.a (mon.0) 2021-07-31T02:41:35.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:34 smithi045 conmon[33655]: 2883 : cluster [DBG] osdmap e1868: 8 total, 8 up, 8 in 2021-07-31T02:41:35.225 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:41:35.226 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 56m ago 57m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:41:35.226 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 54m ago 56m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:41:35.226 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 54m ago 60m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:41:35.226 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 56m ago 62m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:41:35.227 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 56m ago 62m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:41:35.227 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 54m ago 60m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:41:35.227 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 56m ago 60m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:41:35.227 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 56m ago 57m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:41:35.228 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 54m ago 57m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:41:35.228 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 56m ago 60m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:41:35.228 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 56m ago 59m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:41:35.228 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 56m ago 59m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:41:35.228 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 56m ago 59m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:41:35.229 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 54m ago 58m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:41:35.229 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 54m ago 58m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:41:35.229 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 54m ago 58m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:41:35.229 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 54m ago 57m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:41:35.229 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 54m ago 57m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:41:35.561 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:41:35.561 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:41:35.561 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:41:35.561 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:41:35.562 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:41:35.562 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:41:35.562 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:41:35.562 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:41:35.563 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:41:35.563 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:41:35.563 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:41:35.563 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:41:35.563 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:41:35.564 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:41:35.564 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:41:35.564 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:41:35.564 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:41:36.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:35 smithi154 conmon[34553]: cluster 2021-07-31T02:41:34.580231+0000 mgr.x (mgr 2021-07-31T02:41:36.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:35 smithi154 conmon[34553]: .24542) 2332 : cluster [DBG] pgmap v3345: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.2 MiB/s rd, 3.7 KiB/s wr, 1.61k op/s 2021-07-31T02:41:36.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:35 smithi154 conmon[34553]: cluster 2021-07-31T02:41:34.616636+0000 mon.a (mon.0) 2884 : cluster [DBG] osdmap e1869: 8 total, 8 up, 8 in 2021-07-31T02:41:36.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:35 smithi154 conmon[34553]: 2021-07-31T02:41:36.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:35 smithi154 conmon[34553]: audit 2021-07-31T 2021-07-31T02:41:36.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:35 smithi154 conmon[34553]: 02:41:34.934443+0000 mgr.x (mgr.24542) 2333 : audit [DBG] 2021-07-31T02:41:36.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:35 smithi154 conmon[34553]: from='client.17862 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:41:36.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:35 smithi154 conmon[34553]: audit 2021-07 2021-07-31T02:41:36.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:35 smithi154 conmon[34553]: -31T02:41:35.217558+0000 mgr.x (mgr.24542) 2021-07-31T02:41:36.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:35 smithi154 conmon[34553]: 2334 : audit [DBG] from='client.27808 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:41:36.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:35 smithi154 conmon[34553]: audit 2021-07-31T02:41:35.560884 2021-07-31T02:41:36.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:35 smithi154 conmon[34553]: +0000 mon.c (mon.1) 127 : audit [DBG] from='client.? 172.21.15.45:0/3374203785' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:41:36.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:35 smithi045 conmon[39718]: cluster 2021-07-31T02:41: 2021-07-31T02:41:36.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:35 smithi045 conmon[39718]: 34.580231+0000 mgr.x (mgr.24542) 2332 : cluster [DBG] pgmap v3345: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.2 MiB/s rd, 3.7 KiB/s wr, 1.61k op/s 2021-07-31T02:41:36.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:35 smithi045 conmon[39718]: cluster 2021-07 2021-07-31T02:41:36.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:35 smithi045 conmon[39718]: -31T02:41:34.616636+0000 mon.a ( 2021-07-31T02:41:36.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:35 smithi045 conmon[39718]: mon.0) 2884 : cluster [DBG] 2021-07-31T02:41:36.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:35 smithi045 conmon[39718]: osdmap e1869: 8 total, 8 up, 8 in 2021-07-31T02:41:36.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:35 smithi045 conmon[39718]: audit 2021-07-31T02:41:36.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:35 smithi045 conmon[39718]: 2021-07-31T 2021-07-31T02:41:36.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:35 smithi045 conmon[39718]: 02:41:34.934443+0000 mgr.x ( 2021-07-31T02:41:36.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:35 smithi045 conmon[39718]: mgr.24542) 2333 : audit 2021-07-31T02:41:36.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:35 smithi045 conmon[39718]: [DBG] from='client.17862 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:41:36.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:35 smithi045 conmon[39718]: audit 2021-07-31T02:41:36.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:35 smithi045 conmon[39718]: 2021-07-31T02:41:35. 2021-07-31T02:41:36.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:35 smithi045 conmon[39718]: 217558+0000 mgr.x (mgr.24542 2021-07-31T02:41:36.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:35 smithi045 conmon[39718]: ) 2334 : audit [DBG] from='client.27808 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:41:36.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:35 smithi045 conmon[39718]: 2021-07-31T02:41:36.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:35 smithi045 conmon[39718]: audit 2021 2021-07-31T02:41:36.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:35 smithi045 conmon[39718]: -07-31T02:41:35.560884 2021-07-31T02:41:36.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:35 smithi045 conmon[39718]: +0000 mon.c (mon.1) 127 : audit 2021-07-31T02:41:36.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:35 smithi045 conmon[39718]: [DBG] from='client.? 172.21.15.45:0/3374203785' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:41:36.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:35 smithi045 conmon[33655]: cluster 2021-07-31T02:41:34.580231+0000 mgr.x ( 2021-07-31T02:41:36.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:35 smithi045 conmon[33655]: mgr.24542) 2332 : cluster [DBG] pgmap v3345: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.2 MiB/s rd, 3.7 KiB/s wr, 1.61k op/s 2021-07-31T02:41:36.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:35 smithi045 conmon[33655]: cluster 2021-07-31T02:41:34.616636+0000 mon.a (mon.0) 2884 : cluster 2021-07-31T02:41:36.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:35 smithi045 conmon[33655]: [DBG] osdmap e1869: 8 total, 8 up, 8 in 2021-07-31T02:41:36.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:35 smithi045 conmon[33655]: audit 2021-07 2021-07-31T02:41:36.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:35 smithi045 conmon[33655]: -31T02:41:34.934443+0000 mgr.x (mgr.24542) 2333 2021-07-31T02:41:36.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:35 smithi045 conmon[33655]: : audit [DBG] from='client.17862 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:41:36.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:35 smithi045 conmon[33655]: audit 2021-07-31T 2021-07-31T02:41:36.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:35 smithi045 conmon[33655]: 02:41:35.217558+0000 mgr.x (mgr.24542 2021-07-31T02:41:36.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:35 smithi045 conmon[33655]: ) 2334 : audit [DBG] from='client.27808 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:41:36.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:35 smithi045 conmon[33655]: audit 2021-07-31T02 2021-07-31T02:41:36.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:35 smithi045 conmon[33655]: :41:35.560884+0000 mon.c (mon. 2021-07-31T02:41:36.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:35 smithi045 conmon[33655]: 1) 127 : audit [DBG] from='client.? 172.21.15.45:0/3374203785' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:41:37.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:36 smithi154 conmon[34553]: cluster 2021-07-31T02:41:35.617799+0000 mon.a (mon.0) 2021-07-31T02:41:37.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:36 smithi154 conmon[34553]: 2885 : cluster [DBG] osdmap e1870: 8 total, 8 up, 8 in 2021-07-31T02:41:37.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:36 smithi045 conmon[33655]: cluster 2021-07-31T02:41:35.617799+0000 mon.a (mon 2021-07-31T02:41:37.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:36 smithi045 conmon[33655]: .0) 2885 : cluster [DBG] osdmap e1870: 8 total, 8 up, 8 in 2021-07-31T02:41:37.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:36 smithi045 conmon[39718]: cluster 2021-07-31T02:41:35. 2021-07-31T02:41:37.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:36 smithi045 conmon[39718]: 617799+0000 mon.a (mon.0) 2885 : cluster [DBG] osdmap e1870: 8 total, 8 up, 8 in 2021-07-31T02:41:38.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:37 smithi154 conmon[34553]: cluster 2021-07-31T02:41:36.580796+0000 mgr.x ( 2021-07-31T02:41:38.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:37 smithi154 conmon[34553]: mgr.24542) 2335 : cluster [DBG] pgmap v3348: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.2 MiB/s rd, 3.7 KiB/s wr, 1.61k op/s 2021-07-31T02:41:38.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:37 smithi154 conmon[34553]: cluster 2021-07-31T02:41:36.630260+0000 mon.a (mon.0) 2886 : cluster [DBG] osdmap e1871: 8 total, 8 up, 8 in 2021-07-31T02:41:38.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:37 smithi045 conmon[33655]: cluster 2021-07-31T02:41:36.580796+0000 mgr.x (mgr.24542) 2335 : cluster [DBG] pgmap v3348: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.2 MiB/s rd, 3.7 KiB/s wr, 1.61k op/s 2021-07-31T02:41:38.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:37 smithi045 conmon[33655]: cluster 2021-07-31T02:41:36.630260+0000 mon.a (mon.0) 2886 : cluster [DBG] osdmap e1871: 8 total, 8 up, 8 in 2021-07-31T02:41:38.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:37 smithi045 conmon[39718]: cluster 2021-07-31T02:41:36 2021-07-31T02:41:38.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:37 smithi045 conmon[39718]: .580796+0000 mgr.x (mgr.24542) 2335 : cluster [DBG] pgmap v3348: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.2 MiB/s rd, 3.7 KiB/s wr, 1.61k op/s 2021-07-31T02:41:38.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:37 smithi045 conmon[39718]: cluster 2021 2021-07-31T02:41:38.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:37 smithi045 conmon[39718]: -07-31T02:41:36 2021-07-31T02:41:38.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:37 smithi045 conmon[39718]: .630260+0000 mon.a (mon. 2021-07-31T02:41:38.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:37 smithi045 conmon[39718]: 0) 2886 : cluster [DBG] osdmap e1871: 8 total, 8 up, 8 in 2021-07-31T02:41:38.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:37 smithi045 conmon[39718]: 2021-07-31T02:41:39.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:38 smithi154 conmon[34553]: cluster 2021-07-31T02:41:37.652945+0000 mon.a (mon.0) 2887 : cluster [DBG] osdmap e1872: 8 total, 8 up, 8 in 2021-07-31T02:41:39.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:38 smithi045 conmon[33655]: cluster 2021-07-31T02:41:37.652945+0000 mon.a (mon. 2021-07-31T02:41:39.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:38 smithi045 conmon[33655]: 0) 2887 : cluster [DBG] osdmap e1872: 8 total, 8 up, 8 in 2021-07-31T02:41:39.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:38 smithi045 conmon[39718]: cluster 2021-07-31T02:41:37 2021-07-31T02:41:39.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:38 smithi045 conmon[39718]: .652945+0000 mon.a (mon.0) 2887 : cluster [DBG] osdmap e1872: 8 total, 8 up, 8 in 2021-07-31T02:41:40.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:39 smithi154 conmon[34553]: cluster 2021-07-31T02:41:40.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:39 smithi154 conmon[34553]: 2021-07-31T02:41:40.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:39 smithi154 conmon[34553]: 2021-07-31T02:41 2021-07-31T02:41:40.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:39 smithi154 conmon[34553]: :38.581178+0000 mgr.x (mgr.24542) 2336 : cluster [DBG] pgmap v3351: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:41:40.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:39 smithi154 conmon[34553]: cluster 2021-07-31 2021-07-31T02:41:40.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:39 smithi154 conmon[34553]: T02:41:38.665582+0000 mon.a (mon.0) 2888 : cluster 2021-07-31T02:41:40.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:39 smithi154 conmon[34553]: [DBG] osdmap e1873: 8 total, 8 up, 8 in 2021-07-31T02:41:40.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:39 smithi045 conmon[33655]: cluster 2021-07-31T02:41:38.581178+0000 mgr.x (mgr.24542 2021-07-31T02:41:40.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:39 smithi045 conmon[33655]: ) 2336 : cluster [DBG] pgmap v3351: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:41:40.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:39 smithi045 conmon[33655]: cluster 2021-07-31T02:41:38.665582+0000 mon.a (mon.0) 2888 : cluster [DBG] osdmap e1873: 8 total, 8 up, 8 in 2021-07-31T02:41:40.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:39 smithi045 conmon[39718]: cluster 2021-07-31T02:41:38.581178+0000 mgr.x (mgr. 2021-07-31T02:41:40.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:39 smithi045 conmon[39718]: 24542) 2336 : cluster [DBG] pgmap v3351: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:41:40.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:39 smithi045 conmon[39718]: cluster 2021-07-31T02:41:38.665582+0000 2021-07-31T02:41:40.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:39 smithi045 conmon[39718]: mon.a (mon.0) 2888 : cluster [DBG] osdmap e1873: 8 total, 8 up, 8 in 2021-07-31T02:41:41.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:40 smithi154 conmon[34553]: cluster 2021-07-31T02:41:39.666358+0000 mon.a (mon.0) 2889 : cluster [DBG] 2021-07-31T02:41:41.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:40 smithi154 conmon[34553]: osdmap e1874: 8 total, 8 up, 8 in 2021-07-31T02:41:41.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:40 smithi045 conmon[33655]: cluster 2021-07-31T02:41:39.666358+0000 mon.a (mon.0) 2021-07-31T02:41:41.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:40 smithi045 conmon[33655]: 2889 : cluster [DBG] osdmap e1874: 8 total, 8 up, 8 in 2021-07-31T02:41:41.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:40 smithi045 conmon[39718]: cluster 2021-07-31T02:41:39.666358+0000 mon.a 2021-07-31T02:41:41.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:40 smithi045 conmon[39718]: (mon.0) 2889 : cluster [DBG] osdmap e1874: 8 total, 8 up, 8 in 2021-07-31T02:41:42.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:41 smithi154 conmon[34553]: cluster 2021-07-31T02:41:40.581669+0000 mgr.x (mgr.24542) 2337 : cluster [DBG] pgmap v3354: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.2 MiB/s rd, 3.7 KiB/s wr, 1.65k op/s 2021-07-31T02:41:42.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:41 smithi154 conmon[34553]: cluster 2021-07-31T02:41:40.678701+0000 mon.a (mon.0) 2890 : cluster [DBG] osdmap e1875: 8 total, 8 up, 8 in 2021-07-31T02:41:42.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:41 smithi045 conmon[33655]: cluster 2021-07-31T02:41:40.581669+0000 2021-07-31T02:41:42.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:41 smithi045 conmon[33655]: mgr.x (mgr.24542) 2337 : cluster [DBG] pgmap v3354: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.2 MiB/s rd, 3.7 KiB/s wr, 1.65k op/s 2021-07-31T02:41:42.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:41 smithi045 conmon[33655]: cluster 2021-07-31T02:41:40 2021-07-31T02:41:42.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:41 smithi045 conmon[33655]: .678701+0000 mon.a (mon.0) 2890 : cluster [DBG] osdmap e1875: 8 total, 8 up, 8 in 2021-07-31T02:41:42.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:41 smithi045 conmon[39718]: cluster 2021-07-31T02:41:40.581669 2021-07-31T02:41:42.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:41 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 2337 : cluster [DBG] pgmap v3354: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.2 MiB/s rd, 3.7 KiB/s wr, 1.65k op/s 2021-07-31T02:41:42.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:41 smithi045 conmon[39718]: cluster 2021-07-31T02:41:40.678701+0000 mon.a (mon.0) 2021-07-31T02:41:42.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:41 smithi045 conmon[39718]: 2890 : cluster [DBG] osdmap e1875: 8 total, 8 up, 8 in 2021-07-31T02:41:43.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:42 smithi154 conmon[34553]: cluster 2021-07-31T02:41:41.685924+0000 mon.a (mon.0) 2891 : cluster [DBG] osdmap e1876: 8 total, 8 up, 8 in 2021-07-31T02:41:43.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:42 smithi045 conmon[33655]: cluster 2021-07-31T02:41:41.685924+0000 mon.a 2021-07-31T02:41:43.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:42 smithi045 conmon[33655]: (mon.0) 2891 : cluster [DBG] osdmap e1876: 8 total, 8 up, 8 in 2021-07-31T02:41:43.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:42 smithi045 conmon[39718]: cluster 2021-07-31T02:41:41.685924+0000 mon.a (mon.0) 2891 : cluster [DBG] osdmap e1876: 8 total, 8 up, 8 in 2021-07-31T02:41:44.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:43 smithi154 conmon[34553]: cluster 2021-07-31T02:41:42.582192+0000 mgr.x (mgr.24542) 2338 : cluster [DBG] pgmap v3357: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.2 MiB/s rd, 3.7 KiB/s wr, 1.65k op/s 2021-07-31T02:41:44.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:43 smithi154 conmon[34553]: cluster 2021-07-31T02:41:42.687356+0000 mon.a (mon.0) 2892 : cluster [DBG] osdmap e1877: 8 total, 8 up, 8 in 2021-07-31T02:41:44.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:43 smithi045 conmon[39718]: cluster 2021-07-31T02:41:42. 2021-07-31T02:41:44.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:43 smithi045 conmon[39718]: 582192+0000 mgr.x (mgr.24542) 2338 : cluster [DBG] pgmap v3357: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.2 MiB/s rd, 3.7 KiB/s wr, 1.65k op/s 2021-07-31T02:41:44.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:43 smithi045 conmon[39718]: cluster 2021-07-31T02:41:44.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:43 smithi045 conmon[39718]: 2021-07-31T02:41:42.687356 2021-07-31T02:41:44.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:43 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2892 2021-07-31T02:41:44.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:43 smithi045 conmon[39718]: : cluster [DBG] osdmap e1877: 8 total, 8 up, 8 in 2021-07-31T02:41:44.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:43 smithi045 conmon[33655]: cluster 2021-07-31T02:41:42.582192+0000 mgr.x (mgr.24542) 2338 2021-07-31T02:41:44.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:43 smithi045 conmon[33655]: : cluster [DBG] pgmap v3357: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.2 MiB/s rd, 3.7 KiB/s wr, 1.65k op/s 2021-07-31T02:41:44.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:43 smithi045 conmon[33655]: cluster 2021-07-31T02:41:42.687356+0000 mon.a (mon.0) 2892 : cluster [DBG] osdmap e1877: 8 total, 8 up, 8 in 2021-07-31T02:41:45.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:44 smithi154 conmon[34553]: cluster 2021-07-31T02:41:43.693808+0000 mon.a (mon.0) 2893 : cluster [DBG] osdmap e1878: 8 total, 8 up, 8 in 2021-07-31T02:41:45.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:44 smithi154 conmon[34553]: 2021-07-31T02:41:45.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:44 smithi045 conmon[39718]: cluster 2021-07-31T02:41:43.693808 2021-07-31T02:41:45.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:44 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2893 : cluster [DBG] osdmap e1878: 8 total, 8 up, 8 in 2021-07-31T02:41:45.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:44 smithi045 conmon[33655]: cluster 2021-07-31T02:41:43.693808+0000 mon.a (mon.0 2021-07-31T02:41:45.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:44 smithi045 conmon[33655]: ) 2893 : cluster [DBG] osdmap e1878: 8 total, 8 up, 8 in 2021-07-31T02:41:46.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:45 smithi154 conmon[34553]: cluster 2021-07-31T02:41:44.582777+0000 mgr.x (mgr.24542) 2339 : 2021-07-31T02:41:46.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:45 smithi154 conmon[34553]: cluster [DBG] pgmap v3360: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.2 MiB/s rd, 3.7 KiB/s wr, 1.70k op/s 2021-07-31T02:41:46.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:45 smithi154 conmon[34553]: cluster 2021-07-31T02:41:44.704847+0000 mon.a (mon.0) 2894 : cluster [DBG] osdmap e1879: 8 total, 8 up, 8 in 2021-07-31T02:41:46.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:45 smithi045 conmon[39718]: cluster 2021-07-31T02:41:44.582777+0000 mgr.x (mgr 2021-07-31T02:41:46.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:45 smithi045 conmon[39718]: .24542) 2339 : cluster [DBG] pgmap v3360: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.2 MiB/s rd, 3.7 KiB/s wr, 1.70k op/s 2021-07-31T02:41:46.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:45 smithi045 conmon[39718]: cluster 2021-07-31T02:41:44.704847+0000 mon.a (mon.0) 2894 2021-07-31T02:41:46.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:45 smithi045 conmon[39718]: : cluster [DBG] osdmap e1879: 8 total, 8 up, 8 in 2021-07-31T02:41:46.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:45 smithi045 conmon[33655]: cluster 2021-07-31T02:41:44.582777+0000 mgr.x (mgr.24542) 2339 : cluster [DBG] pgmap v3360: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.2 MiB/s rd, 3.7 KiB/s wr, 1.70k op/s 2021-07-31T02:41:46.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:45 smithi045 conmon[33655]: cluster 2021-07-31T02:41:44.704847+0000 mon.a (mon.0) 2894 : cluster [DBG] osdmap e1879: 8 total, 8 up, 8 in 2021-07-31T02:41:47.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:46 smithi154 conmon[34553]: cluster 2021-07-31T02:41:45.706774+0000 mon.a (mon.0) 2895 : 2021-07-31T02:41:47.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:46 smithi154 conmon[34553]: cluster [DBG] osdmap e1880: 8 total, 8 up, 8 in 2021-07-31T02:41:47.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:46 smithi045 conmon[33655]: cluster 2021-07-31T02:41:45.706774+0000 mon.a (mon.0) 2895 2021-07-31T02:41:47.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:46 smithi045 conmon[33655]: : cluster [DBG] osdmap e1880: 8 total, 8 up, 8 in 2021-07-31T02:41:47.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:46 smithi045 conmon[39718]: cluster 2021-07-31T02:41:45. 2021-07-31T02:41:47.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:46 smithi045 conmon[39718]: 706774+0000 mon.a (mon.0) 2895 : cluster [DBG] osdmap e1880: 8 total, 8 up, 8 in 2021-07-31T02:41:48.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:47 smithi154 conmon[34553]: cluster 2021-07-31T02:41:46.583228+0000 mgr.x (mgr.24542) 2340 : cluster [DBG] pgmap v3363: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.2 MiB/s rd, 3.7 KiB/s wr, 1.70k op/s 2021-07-31T02:41:48.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:47 smithi154 conmon[34553]: cluster 2021-07-31T02:41:46.717282+0000 mon.a (mon.0) 2896 : cluster [DBG] osdmap e1881: 8 total, 8 up, 8 in 2021-07-31T02:41:48.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:47 smithi045 conmon[39718]: cluster 2021-07-31T02:41:46.583228+0000 mgr.x (mgr 2021-07-31T02:41:48.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:47 smithi045 conmon[39718]: .24542) 2340 : cluster [DBG] pgmap v3363: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.2 MiB/s rd, 3.7 KiB/s wr, 1.70k op/s 2021-07-31T02:41:48.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:47 smithi045 conmon[39718]: cluster 2021-07-31T02:41:48.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:47 smithi045 conmon[39718]: 2021-07-31T02:41:46.717282+0000 mon.a (mon.0) 2896 : cluster [DBG] osdmap e1881: 8 total, 8 up, 8 in 2021-07-31T02:41:48.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:47 smithi045 conmon[33655]: cluster 2021-07-31T02:41:46. 2021-07-31T02:41:48.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:47 smithi045 conmon[33655]: 583228+0000 mgr.x (mgr.24542) 2340 : cluster [DBG] pgmap v3363: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.2 MiB/s rd, 3.7 KiB/s wr, 1.70k op/s 2021-07-31T02:41:48.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:47 smithi045 conmon[33655]: cluster 2021-07-31T02:41:48.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:47 smithi045 conmon[33655]: 2021-07-31T02:41:46.717282+0000 mon.a (mon.0 2021-07-31T02:41:48.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:47 smithi045 conmon[33655]: ) 2896 : cluster [DBG] osdmap e1881: 8 total, 8 up, 8 in 2021-07-31T02:41:49.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:48 smithi154 conmon[34553]: cluster 2021-07-31T02:41:47.718713+0000 mon.a ( 2021-07-31T02:41:49.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:48 smithi154 conmon[34553]: mon.0) 2897 : cluster [DBG] osdmap e1882: 8 total, 8 up, 8 in 2021-07-31T02:41:49.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:48 smithi045 conmon[39718]: cluster 2021-07-31T02:41:47.718713+0000 mon.a (mon.0) 2897 : cluster 2021-07-31T02:41:49.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:48 smithi045 conmon[39718]: [DBG] osdmap e1882: 8 total, 8 up, 8 in 2021-07-31T02:41:49.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:48 smithi045 conmon[33655]: cluster 2021-07-31T02:41:47.718713 2021-07-31T02:41:49.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:48 smithi045 conmon[33655]: +0000 mon.a (mon.0) 2897 : cluster [DBG] osdmap e1882: 8 total, 8 up, 8 in 2021-07-31T02:41:50.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:49 smithi154 conmon[34553]: cluster 2021-07-31T02:41:48.583855+0000 mgr.x (mgr.24542) 2341 : cluster [DBG] pgmap v3366: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:41:50.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:49 smithi154 conmon[34553]: 2021-07-31T02:41:50.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:49 smithi154 conmon[34553]: cluster 2021-07-31T02:41:48.730226+0000 mon.a (mon.0) 2898 : cluster [DBG] osdmap e1883: 8 total, 8 up, 8 in 2021-07-31T02:41:50.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:49 smithi045 conmon[33655]: cluster 2021-07-31T02:41:48.583855+0000 mgr.x (mgr.24542) 2341 : cluster [DBG] 2021-07-31T02:41:50.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:49 smithi045 conmon[33655]: pgmap v3366: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:41:50.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:49 smithi045 conmon[33655]: cluster 2021-07-31T02:41:48.730226+0000 mon.a (mon.0) 2898 : cluster [DBG] osdmap e1883: 8 total, 8 up, 8 in 2021-07-31T02:41:50.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:49 smithi045 conmon[39718]: cluster 2021-07-31T02:41:48.583855+0000 mgr.x (mgr.24542) 2341 : cluster [DBG] pgmap v3366: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:41:50.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:49 smithi045 conmon[39718]: 2021-07-31T02:41:50.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:49 smithi045 conmon[39718]: cluster 2021-07-31T02:41:48.730226+0000 mon.a (mon.0) 2898 : cluster [DBG] osdmap e1883: 8 total, 8 up, 8 in 2021-07-31T02:41:51.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:50 smithi154 conmon[34553]: cluster 2021-07-31T02:41:49.731347+0000 mon.a (mon.0) 2899 : cluster [DBG] osdmap e1884: 8 total, 8 up, 8 in 2021-07-31T02:41:51.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:50 smithi045 conmon[33655]: cluster 2021-07-31T02:41:49.731347+0000 mon.a (mon.0) 2899 : cluster 2021-07-31T02:41:51.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:50 smithi045 conmon[33655]: [DBG] osdmap e1884: 8 total, 8 up, 8 in 2021-07-31T02:41:51.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:50 smithi045 conmon[39718]: cluster 2021-07-31T02:41:49.731347 2021-07-31T02:41:51.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:50 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2899 : cluster [DBG] osdmap e1884: 8 total, 8 up, 8 in 2021-07-31T02:41:52.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:51 smithi154 conmon[34553]: cluster 2021-07-31T02:41:50.584540+0000 mgr.x (mgr.24542) 2342 : cluster [DBG] 2021-07-31T02:41:52.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:51 smithi154 conmon[34553]: pgmap v3369: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.3 MiB/s rd, 3.7 KiB/s wr, 1.74k op/s 2021-07-31T02:41:52.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:51 smithi154 conmon[34553]: cluster 2021-07-31T02:41:50.741206+0000 mon.a (mon.0) 2900 : cluster [DBG] osdmap e1885: 8 total, 8 up, 8 in 2021-07-31T02:41:52.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:51 smithi045 conmon[33655]: cluster 2021-07-31T02:41:50.584540+0000 mgr.x (mgr.24542) 2342 : cluster [DBG] pgmap v3369: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.3 MiB/s rd, 3.7 KiB/s wr, 1.74k op/s 2021-07-31T02:41:52.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:51 smithi045 conmon[33655]: cluster 2021-07-31T02:41:52.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:51 smithi045 conmon[33655]: 2021-07-31T02:41:50.741206+0000 mon.a (mon.0) 2900 : cluster [DBG] osdmap e1885: 8 total, 8 up, 8 in 2021-07-31T02:41:52.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:51 smithi045 conmon[39718]: cluster 2021-07-31T02:41:50.584540+0000 mgr.x ( 2021-07-31T02:41:52.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:51 smithi045 conmon[39718]: mgr.24542) 2342 : cluster [DBG] pgmap v3369: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.3 MiB/s rd, 3.7 KiB/s wr, 1.74k op/s 2021-07-31T02:41:52.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:51 smithi045 conmon[39718]: cluster 2021-07-31T02:41:50.741206+0000 mon.a (mon.0) 2021-07-31T02:41:52.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:51 smithi045 conmon[39718]: 2900 : cluster [DBG] osdmap e1885: 8 total, 8 up, 8 in 2021-07-31T02:41:53.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:52 smithi045 conmon[39718]: cluster 2021-07-31T02:41:51.742598+0000 2021-07-31T02:41:53.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:52 smithi045 conmon[39718]: mon.a (mon.0) 2901 : cluster [DBG] osdmap e1886: 8 total, 8 up, 8 in 2021-07-31T02:41:53.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:52 smithi045 conmon[33655]: cluster 2021-07-31T02:41:51.742598+0000 mon.a (mon.0) 2901 : cluster [DBG] osdmap e1886: 8 total, 8 up, 8 in 2021-07-31T02:41:53.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:52 smithi154 conmon[34553]: cluster 2021-07-31T02:41:51.742598+0000 mon.a (mon.0) 2901 : cluster [DBG] osdmap e1886: 8 total, 8 up, 8 in 2021-07-31T02:41:54.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:53 smithi045 conmon[33655]: cluster 2021-07-31T02:41:52.585003+0000 mgr.x (mgr.24542) 2343 : cluster [DBG] pgmap v3372: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.3 MiB/s rd, 3.7 KiB/s wr, 1.74k op/s 2021-07-31T02:41:54.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:53 smithi045 conmon[33655]: cluster 2021-07 2021-07-31T02:41:54.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:53 smithi045 conmon[33655]: -31T02:41:52.751382+0000 mon.a (mon.0) 2902 : cluster [DBG] osdmap e1887: 8 total, 8 up, 8 in 2021-07-31T02:41:54.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:53 smithi045 conmon[39718]: cluster 2021-07-31T02:41:52.585003+0000 mgr.x (mgr.24542) 2343 : cluster [DBG] 2021-07-31T02:41:54.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:53 smithi045 conmon[39718]: pgmap v3372: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.3 MiB/s rd, 3.7 KiB/s wr, 1.74k op/s 2021-07-31T02:41:54.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:53 smithi045 conmon[39718]: cluster 2021-07-31T02:41:52.751382+0000 mon.a (mon.0) 2902 : cluster [DBG] osdmap e1887: 8 total, 8 up, 8 in 2021-07-31T02:41:54.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:53 smithi154 conmon[34553]: cluster 2021-07-31T02:41:52.585003+0000 mgr.x (mgr.24542) 2343 : cluster [DBG] pgmap v3372: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.3 MiB/s rd, 3.7 KiB/s wr, 1.74k op/s 2021-07-31T02:41:54.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:53 smithi154 conmon[34553]: 2021-07-31T02:41:54.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:53 smithi154 conmon[34553]: cluster 2021-07-31T02:41:52.751382+0000 mon.a (mon.0) 2902 : cluster [DBG] osdmap e1887: 8 total, 8 up, 8 in 2021-07-31T02:41:55.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:54 smithi045 conmon[39718]: cluster 2021-07-31T02:41:53.763015+0000 mon.a (mon.0) 2021-07-31T02:41:55.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:54 smithi045 conmon[39718]: 2903 : cluster [DBG] osdmap e1888: 8 total, 8 up, 8 in 2021-07-31T02:41:55.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:54 smithi045 conmon[33655]: cluster 2021-07-31T02:41:53.763015+0000 mon.a (mon.0) 2903 : 2021-07-31T02:41:55.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:54 smithi045 conmon[33655]: cluster [DBG] osdmap e1888: 8 total, 8 up, 8 in 2021-07-31T02:41:55.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:54 smithi154 conmon[34553]: cluster 2021-07-31T02:41:53.763015+0000 mon.a (mon. 2021-07-31T02:41:55.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:54 smithi154 conmon[34553]: 0) 2903 : cluster [DBG] osdmap e1888: 8 total, 8 up, 8 in 2021-07-31T02:41:56.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:55 smithi045 conmon[39718]: cluster 2021-07-31T02:41:54.585541+0000 mgr.x (mgr.24542) 2021-07-31T02:41:56.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:55 smithi045 conmon[39718]: 2344 : cluster [DBG] pgmap v3375: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.3 MiB/s rd, 4.7 KiB/s wr, 1.79k op/s 2021-07-31T02:41:56.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:55 smithi045 conmon[39718]: cluster 2021-07-31T02:41:54.768661+0000 mon.a (mon.0) 2904 : cluster [DBG] 2021-07-31T02:41:56.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:55 smithi045 conmon[39718]: osdmap e1889: 8 total, 8 up, 8 in 2021-07-31T02:41:56.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:55 smithi045 conmon[33655]: cluster 2021-07-31T02:41:54.585541+0000 mgr.x (mgr.24542) 2344 : cluster [DBG] pgmap v3375: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.3 MiB/s rd, 4.7 KiB/s wr, 1.79k op/s 2021-07-31T02:41:56.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:55 smithi045 conmon[33655]: cluster 2021-07-31T02:41:54.768661+0000 mon.a (mon.0) 2904 : cluster [DBG] osdmap e1889: 8 total, 8 up, 8 in 2021-07-31T02:41:56.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:55 smithi154 conmon[34553]: cluster 2021-07-31T02:41:54.585541+0000 mgr.x (mgr.24542) 2344 : cluster [DBG] pgmap v3375: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.3 MiB/s rd, 4.7 KiB/s wr, 1.79k op/s 2021-07-31T02:41:56.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:55 smithi154 conmon[34553]: 2021-07-31T02:41:56.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:55 smithi154 conmon[34553]: cluster 2021-07-31T02:41:54.768661+0000 mon.a (mon.0) 2904 : cluster [DBG] osdmap e1889: 8 total, 8 up, 8 in 2021-07-31T02:41:57.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:56 smithi154 conmon[34553]: cluster 2021-07-31T02:41:55.774297+0000 mon.a (mon.0) 2905 : cluster [DBG] osdmap e1890: 8 total, 8 up, 8 in 2021-07-31T02:41:57.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:56 smithi045 conmon[33655]: cluster 2021-07-31T02:41:55.774297+0000 mon.a 2021-07-31T02:41:57.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:56 smithi045 conmon[33655]: (mon.0) 2905 : cluster [DBG] osdmap e1890: 8 total, 8 up, 8 in 2021-07-31T02:41:57.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:56 smithi045 conmon[39718]: cluster 2021-07-31T02:41:55.774297+0000 mon.a (mon.0) 2905 : cluster 2021-07-31T02:41:57.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:56 smithi045 conmon[39718]: [DBG] osdmap e1890: 8 total, 8 up, 8 in 2021-07-31T02:41:58.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:57 smithi154 conmon[34553]: cluster 2021-07-31T02:41:56.586090+0000 mgr.x (mgr.24542) 2345 : cluster [DBG] pgmap v3378: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.3 MiB/s rd, 4.7 KiB/s wr, 1.79k op/s 2021-07-31T02:41:58.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:57 smithi154 conmon[34553]: cluster 2021-07-31T02:41:56.780131+0000 mon.a (mon.0) 2906 : cluster [DBG] osdmap e1891: 8 total, 8 up, 8 in 2021-07-31T02:41:58.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:57 smithi045 conmon[39718]: cluster 2021-07-31T02:41:56.586090+0000 mgr.x (mgr 2021-07-31T02:41:58.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:57 smithi045 conmon[39718]: .24542) 2345 : cluster [DBG] pgmap v3378: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.3 MiB/s rd, 4.7 KiB/s wr, 1.79k op/s 2021-07-31T02:41:58.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:57 smithi045 conmon[39718]: cluster 2021-07-31T02:41:58.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:57 smithi045 conmon[39718]: 2021-07-31T02:41:56.780131+0000 mon.a (mon.0) 2906 2021-07-31T02:41:58.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:57 smithi045 conmon[39718]: : cluster [DBG] osdmap e1891: 8 total, 8 up, 8 in 2021-07-31T02:41:58.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:57 smithi045 conmon[33655]: cluster 2021-07-31T02:41:56.586090 2021-07-31T02:41:58.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:57 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 2345 : cluster [DBG] pgmap v3378: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.3 MiB/s rd, 4.7 KiB/s wr, 1.79k op/s 2021-07-31T02:41:58.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:57 smithi045 conmon[33655]: cluster 2021-07-31T02:41:56.780131+0000 mon.a (mon.0) 2906 : 2021-07-31T02:41:58.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:57 smithi045 conmon[33655]: cluster [DBG] osdmap e1891: 8 total, 8 up, 8 in 2021-07-31T02:41:59.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:58 smithi154 conmon[34553]: cluster 2021-07-31T02:41:57.781190+0000 mon.a (mon.0) 2907 : cluster 2021-07-31T02:41:59.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:58 smithi154 conmon[34553]: [DBG] osdmap e1892: 8 total, 8 up, 8 in 2021-07-31T02:41:59.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:58 smithi045 conmon[39718]: cluster 2021-07-31T02:41:57. 2021-07-31T02:41:59.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:58 smithi045 conmon[39718]: 781190+0000 mon.a (mon.0) 2907 : cluster [DBG] osdmap e1892: 8 total, 8 up, 8 in 2021-07-31T02:41:59.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:58 smithi045 conmon[33655]: cluster 2021-07-31T02:41:57.781190+0000 mon.a (mon.0) 2907 : cluster 2021-07-31T02:41:59.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:58 smithi045 conmon[33655]: [DBG] osdmap e1892: 8 total, 8 up, 8 in 2021-07-31T02:42:00.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:59 smithi154 conmon[34553]: cluster 2021-07-31T02:41:58.586586+0000 mgr.x (mgr.24542) 2346 : cluster 2021-07-31T02:42:00.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:59 smithi154 conmon[34553]: [DBG] pgmap v3381: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:42:00.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:41:59 smithi154 conmon[34553]: cluster 2021-07-31T02:41:58.793942+0000 mon.a (mon.0) 2908 : cluster [DBG] osdmap e1893: 8 total, 8 up, 8 in 2021-07-31T02:42:00.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:59 smithi045 conmon[39718]: cluster 2021-07-31T02 2021-07-31T02:42:00.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:59 smithi045 conmon[39718]: :41:58.586586+0000 mgr.x (mgr.24542) 2346 : cluster [DBG] pgmap v3381: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:42:00.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:59 smithi045 conmon[39718]: cluster 2021-07-31T02: 2021-07-31T02:42:00.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:59 smithi045 conmon[39718]: 41:58.793942+0000 mon.a (mon.0) 2021-07-31T02:42:00.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:41:59 smithi045 conmon[39718]: 2908 : cluster [DBG] osdmap e1893: 8 total, 8 up, 8 in 2021-07-31T02:42:00.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:59 smithi045 conmon[33655]: cluster 2021-07-31T02:41:58.586586+0000 mgr.x (mgr.24542) 2346 : 2021-07-31T02:42:00.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:59 smithi045 conmon[33655]: cluster [DBG] pgmap v3381: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:42:00.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:41:59 smithi045 conmon[33655]: cluster 2021-07-31T02:41:58.793942+0000 mon.a (mon.0) 2908 : cluster [DBG] osdmap e1893: 8 total, 8 up, 8 in 2021-07-31T02:42:01.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:00 smithi154 conmon[34553]: cluster 2021-07-31T02:41:59.794783+0000 mon.a (mon.0) 2909 : cluster [DBG] 2021-07-31T02:42:01.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:00 smithi154 conmon[34553]: osdmap e1894: 8 total, 8 up, 8 in 2021-07-31T02:42:01.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:00 smithi045 conmon[39718]: cluster 2021-07-31T02:41:59.794783+0000 mon.a (mon.0) 2909 : cluster 2021-07-31T02:42:01.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:00 smithi045 conmon[39718]: [DBG] osdmap e1894: 8 total, 8 up, 8 in 2021-07-31T02:42:01.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:00 smithi045 conmon[33655]: cluster 2021-07-31T02:41:59.794783+0000 mon.a (mon.0) 2909 : cluster [DBG] 2021-07-31T02:42:01.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:00 smithi045 conmon[33655]: osdmap e1894: 8 total, 8 up, 8 in 2021-07-31T02:42:02.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:01 smithi154 conmon[34553]: cluster 2021-07-31T02:42:00.587012+0000 mgr.x (mgr.24542) 2347 : cluster [DBG] pgmap v3384: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.3 MiB/s rd, 5.0 KiB/s wr, 1.77k op/s 2021-07-31T02:42:02.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:01 smithi154 conmon[34553]: cluster 2021-07-31T02:42:00.807829+0000 mon.a (mon.0) 2910 : cluster [DBG] osdmap e1895: 8 total, 8 up, 8 in 2021-07-31T02:42:02.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:01 smithi045 conmon[39718]: cluster 2021-07-31T02:42:00.587012+0000 mgr.x (mgr.24542) 2347 : cluster [DBG] pgmap v3384: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.3 MiB/s rd, 5.0 KiB/s wr, 1.77k op/s 2021-07-31T02:42:02.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:01 smithi045 conmon[39718]: 2021-07-31T02:42:02.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:01 smithi045 conmon[39718]: cluster 2021-07-31T02:42:00. 2021-07-31T02:42:02.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:01 smithi045 conmon[39718]: 807829+0000 mon.a (mon.0) 2910 : cluster [DBG] 2021-07-31T02:42:02.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:01 smithi045 conmon[39718]: osdmap e1895: 8 total, 8 up, 8 in 2021-07-31T02:42:02.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:01 smithi045 conmon[33655]: cluster 2021-07-31T02:42:00.587012+0000 mgr.x (mgr.24542) 2021-07-31T02:42:02.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:01 smithi045 conmon[33655]: 2347 : cluster [DBG] pgmap v3384: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.3 MiB/s rd, 5.0 KiB/s wr, 1.77k op/s 2021-07-31T02:42:02.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:01 smithi045 conmon[33655]: cluster 2021-07-31T02:42:00.807829+0000 mon.a (mon.0) 2910 : cluster [DBG] osdmap e1895: 8 total, 8 up, 8 in 2021-07-31T02:42:03.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:02 smithi154 conmon[34553]: cluster 2021-07-31T02:42:01.814864+0000 mon.a (mon.0) 2911 : cluster [DBG] osdmap e1896: 8 total, 8 up, 8 in 2021-07-31T02:42:03.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:02 smithi154 conmon[34553]: cluster 2021-07-31T02:42:02.538045+0000 mon.a (mon.0) 2912 : cluster [DBG] osdmap e1897: 8 total, 8 up, 8 in 2021-07-31T02:42:03.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:02 smithi045 conmon[39718]: cluster 2021-07-31T02:42:01.814864+0000 mon.a ( 2021-07-31T02:42:03.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:02 smithi045 conmon[39718]: mon.0) 2911 : cluster [DBG] osdmap e1896: 8 total, 8 up, 8 in 2021-07-31T02:42:03.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:02 smithi045 conmon[39718]: cluster 2021-07-31T02:42:02.538045+0000 mon.a (mon. 2021-07-31T02:42:03.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:02 smithi045 conmon[39718]: 0) 2912 : cluster [DBG] osdmap e1897: 8 total, 8 up, 8 in 2021-07-31T02:42:03.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:02 smithi045 conmon[33655]: cluster 2021-07-31T02:42:01.814864+0000 mon.a (mon.0) 2911 : cluster [DBG] osdmap e1896: 8 total, 8 up, 8 in 2021-07-31T02:42:03.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:02 smithi045 conmon[33655]: cluster 2021-07-31T02:42:02.538045+0000 mon.a (mon.0) 2912 : cluster [DBG] osdmap e1897: 8 total, 8 up, 8 in 2021-07-31T02:42:04.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:03 smithi154 conmon[34553]: cluster 2021-07-31T02:42:02.587377+0000 mgr.x (mgr.24542) 2348 : cluster [DBG] pgmap v3388: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.3 MiB/s rd, 5.3 KiB/s wr, 1.87k op/s 2021-07-31T02:42:04.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:03 smithi154 conmon[34553]: cluster 2021-07-31T02:42:03.548716+0000 mon.a (mon.0) 2913 : cluster [DBG] osdmap e1898: 8 total, 8 up, 8 in 2021-07-31T02:42:04.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:03 smithi045 conmon[39718]: cluster 2021-07-31T02:42:02.587377+0000 mgr.x ( 2021-07-31T02:42:04.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:03 smithi045 conmon[39718]: mgr.24542) 2348 : cluster [DBG] pgmap v3388: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.3 MiB/s rd, 5.3 KiB/s wr, 1.87k op/s 2021-07-31T02:42:04.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:03 smithi045 conmon[39718]: cluster 2021-07-31T02:42:03.548716+0000 2021-07-31T02:42:04.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:03 smithi045 conmon[39718]: mon.a (mon.0) 2913 : cluster [DBG] osdmap e1898: 8 total, 8 up, 8 in 2021-07-31T02:42:04.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:03 smithi045 conmon[33655]: cluster 2021-07-31T02:42:02.587377+0000 mgr.x (mgr.24542) 2348 : cluster 2021-07-31T02:42:04.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:03 smithi045 conmon[33655]: [DBG] pgmap v3388: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.3 MiB/s rd, 5.3 KiB/s wr, 1.87k op/s 2021-07-31T02:42:04.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:03 smithi045 conmon[33655]: cluster 2021-07-31T02:42:03.548716+0000 mon.a (mon.0) 2913 : cluster [DBG] osdmap e1898: 8 total, 8 up, 8 in 2021-07-31T02:42:05.872 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:42:06.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:05 smithi154 conmon[34553]: cluster 2021-07-31T02:42:06.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:05 smithi154 conmon[34553]: 2021-07-31T02:42:04.550756+0000 mon.a 2021-07-31T02:42:06.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:05 smithi154 conmon[34553]: (mon.0) 2914 : cluster [DBG] osdmap e1899: 8 total, 8 up, 8 in 2021-07-31T02:42:06.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:05 smithi154 conmon[34553]: cluster 2021-07-31T02:42:04. 2021-07-31T02:42:06.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:05 smithi154 conmon[34553]: 587760+0000 mgr.x (mgr.24542) 2349 : cluster [DBG] pgmap v3391: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.3 MiB/s rd, 5.3 KiB/s wr, 1.82k op/s 2021-07-31T02:42:06.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:05 smithi045 conmon[33655]: cluster 2021-07-31T02:42:04.550756+0000 mon.a (mon.0) 2914 2021-07-31T02:42:06.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:05 smithi045 conmon[33655]: : cluster [DBG] osdmap e1899: 8 total, 8 up, 8 in 2021-07-31T02:42:06.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:05 smithi045 conmon[33655]: cluster 2021-07-31T02:42:04.587760+0000 mgr.x (mgr.24542) 2349 : cluster [DBG] pgmap v3391: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.3 MiB/s rd, 5.3 KiB/s wr, 1.82k op/s 2021-07-31T02:42:06.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:05 smithi045 conmon[39718]: cluster 2021-07-31T02:42:04.550756+0000 mon.a 2021-07-31T02:42:06.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:05 smithi045 conmon[39718]: (mon.0) 2914 : cluster [DBG] osdmap e1899: 8 total, 8 up, 8 in 2021-07-31T02:42:06.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:05 smithi045 conmon[39718]: cluster 2021-07-31T02:42:04.587760+0000 mgr.x (mgr.24542) 2349 : cluster [DBG] pgmap v3391: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.3 MiB/s rd, 5.3 KiB/s wr, 1.82k op/s 2021-07-31T02:42:06.149 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:42:06.149 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 56m ago 57m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:42:06.150 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 54m ago 57m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:42:06.150 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 54m ago 61m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:42:06.150 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 56m ago 63m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:42:06.150 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 56m ago 63m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:42:06.151 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 54m ago 61m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:42:06.151 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 56m ago 61m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:42:06.151 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 56m ago 57m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:42:06.151 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 54m ago 57m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:42:06.151 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 56m ago 60m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:42:06.152 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 56m ago 60m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:42:06.152 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 56m ago 60m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:42:06.152 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 56m ago 59m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:42:06.152 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 54m ago 59m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:42:06.153 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 54m ago 59m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:42:06.153 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 54m ago 58m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:42:06.153 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 54m ago 58m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:42:06.154 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 54m ago 58m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:42:06.484 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:42:06.485 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:42:06.485 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:42:06.485 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:42:06.486 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:42:06.486 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:42:06.486 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:42:06.486 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:42:06.487 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:42:06.487 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:42:06.487 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:42:06.488 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:42:06.488 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:42:06.488 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:42:06.488 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:42:06.489 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:42:06.489 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:42:07.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:06 smithi154 conmon[34553]: cluster 2021-07-31T02:42:05.556562+0000 mon.a (mon.0) 2915 : 2021-07-31T02:42:07.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:06 smithi154 conmon[34553]: cluster [DBG] osdmap e1900: 8 total, 8 up, 8 in 2021-07-31T02:42:07.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:06 smithi154 conmon[34553]: audit 2021-07-31T02:42:05.860953+0000 mgr.x (mgr.24542) 2350 : audit [DBG] from='client.17880 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:42:07.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:06 smithi154 conmon[34553]: audit 2021-07-31T02:42:06 2021-07-31T02:42:07.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:06 smithi154 conmon[34553]: .141167+0000 mgr.x (mgr.24542) 2351 : audit [DBG] from='client.17886 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:42:07.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:06 smithi154 conmon[34553]: audit 2021-07-31T02 2021-07-31T02:42:07.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:06 smithi154 conmon[34553]: :42:06.484691+0000 mon.c (mon.1) 128 : audit [DBG] from='client.? 172.21.15.45:0/1074534796' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:42:07.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:06 smithi045 conmon[33655]: cluster 2021-07-31T02: 2021-07-31T02:42:07.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:06 smithi045 conmon[33655]: 42:05.556562+0000 mon.a (mon.0) 2915 : cluster [DBG] osdmap e1900: 8 total, 8 up, 8 in 2021-07-31T02:42:07.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:06 smithi045 conmon[33655]: audit 2021-07-31T02 2021-07-31T02:42:07.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:06 smithi045 conmon[33655]: :42:05.860953+0000 2021-07-31T02:42:07.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:06 smithi045 conmon[33655]: mgr.x (mgr.24542) 2350 : 2021-07-31T02:42:07.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:06 smithi045 conmon[33655]: audit [DBG] from='client.17880 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:42:07.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:06 smithi045 conmon[33655]: audit 2021-07-31T02:42:07.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:06 smithi045 conmon[33655]: 2021-07-31T02:42:06.141167+0000 mgr.x (mgr. 2021-07-31T02:42:07.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:06 smithi045 conmon[33655]: 24542) 2351 : audit [DBG] from='client.17886 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:42:07.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:06 smithi045 conmon[33655]: audit 2021-07-31T02:42:06.484691+0000 2021-07-31T02:42:07.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:06 smithi045 conmon[33655]: mon.c (mon.1) 128 : audit [DBG] from='client.? 172.21.15.45:0/1074534796' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:42:07.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:06 smithi045 conmon[39718]: cluster 2021-07-31T02:42 2021-07-31T02:42:07.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:06 smithi045 conmon[39718]: :05.556562+0000 mon.a (mon.0) 2915 : cluster [DBG] osdmap e1900: 8 total, 8 up, 8 in 2021-07-31T02:42:07.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:06 smithi045 conmon[39718]: audit 2021- 2021-07-31T02:42:07.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:06 smithi045 conmon[39718]: 07-31T02:42:05.860953+0000 2021-07-31T02:42:07.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:06 smithi045 conmon[39718]: mgr.x (mgr.24542) 2350 : 2021-07-31T02:42:07.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:06 smithi045 conmon[39718]: audit [DBG] from='client.17880 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:42:07.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:06 smithi045 conmon[39718]: audit 2021- 2021-07-31T02:42:07.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:06 smithi045 conmon[39718]: 07-31T02:42:06.141167+0000 mgr.x 2021-07-31T02:42:07.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:06 smithi045 conmon[39718]: (mgr.24542) 2351 : audit 2021-07-31T02:42:07.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:06 smithi045 conmon[39718]: [DBG] from='client.17886 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:42:07.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:06 smithi045 conmon[39718]: audit 2021- 2021-07-31T02:42:07.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:06 smithi045 conmon[39718]: 07-31T02:42:06.484691+0000 2021-07-31T02:42:07.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:06 smithi045 conmon[39718]: mon.c (mon.1) 128 : 2021-07-31T02:42:07.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:06 smithi045 conmon[39718]: audit [DBG] from='client.? 172.21.15.45:0/1074534796' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:42:08.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:07 smithi154 conmon[34553]: cluster 2021-07-31T02:42:06.562896+0000 mon.a (mon.0) 2916 : cluster [DBG] 2021-07-31T02:42:08.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:07 smithi154 conmon[34553]: osdmap e1901: 8 total, 8 up, 8 in 2021-07-31T02:42:08.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:07 smithi154 conmon[34553]: cluster 2021-07-31T02:42:06.588198+0000 mgr.x (mgr.24542) 2352 : cluster [DBG] pgmap v3394: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.2 MiB/s rd, 5.0 KiB/s wr, 1.73k op/s 2021-07-31T02:42:08.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:07 smithi154 conmon[34553]: cluster 2021-07-31T02:42:07.540374+0000 mon.a (mon.0) 2021-07-31T02:42:08.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:07 smithi154 conmon[34553]: 2917 : cluster [DBG] osdmap e1902: 8 total, 8 up, 8 in 2021-07-31T02:42:08.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:07 smithi045 conmon[33655]: cluster 2021-07-31T02:42 2021-07-31T02:42:08.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:07 smithi045 conmon[33655]: :06.562896+0000 mon.a (mon.0) 2916 : cluster [DBG] osdmap e1901: 8 total, 8 up, 8 in 2021-07-31T02:42:08.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:07 smithi045 conmon[33655]: cluster 2021-07-31T02:42: 2021-07-31T02:42:08.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:07 smithi045 conmon[33655]: 06.588198+0000 mgr.x (mgr.24542) 2352 : cluster [DBG] pgmap v3394: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.2 MiB/s rd, 5.0 KiB/s wr, 1.73k op/s 2021-07-31T02:42:08.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:07 smithi045 conmon[33655]: cluster 2021-07-31T02:42:08.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:07 smithi045 conmon[33655]: 2021-07-31T02:42:07.540374+0000 mon.a (mon.0 2021-07-31T02:42:08.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:07 smithi045 conmon[33655]: ) 2917 : cluster [DBG] osdmap e1902: 8 total, 8 up, 8 in 2021-07-31T02:42:08.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:07 smithi045 conmon[39718]: cluster 2021-07-31T02:42:06. 2021-07-31T02:42:08.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:07 smithi045 conmon[39718]: 562896+0000 mon.a (mon.0) 2916 : cluster [DBG] osdmap e1901: 8 total, 8 up, 8 in 2021-07-31T02:42:08.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:07 smithi045 conmon[39718]: cluster 2021-07-31T02:42:06.588198+0000 mgr.x (mgr 2021-07-31T02:42:08.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:07 smithi045 conmon[39718]: .24542) 2352 : cluster [DBG] pgmap v3394: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.2 MiB/s rd, 5.0 KiB/s wr, 1.73k op/s 2021-07-31T02:42:08.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:07 smithi045 conmon[39718]: cluster 2021-07-31T02:42:07.540374+0000 mon.a 2021-07-31T02:42:08.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:07 smithi045 conmon[39718]: (mon.0) 2917 : cluster [DBG] osdmap e1902: 8 total, 8 up, 8 in 2021-07-31T02:42:10.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:09 smithi154 conmon[34553]: cluster 2021-07-31T02:42:08.571413+0000 mon.a (mon.0 2021-07-31T02:42:10.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:09 smithi154 conmon[34553]: ) 2918 : cluster [DBG] osdmap e1903: 8 total, 8 up, 8 in 2021-07-31T02:42:10.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:09 smithi154 conmon[34553]: cluster 2021-07-31T02:42:08.588623+0000 mgr.x (mgr.24542) 2353 : cluster [DBG] pgmap v3397: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:42:10.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:09 smithi045 conmon[39718]: cluster 2021-07-31T02:42:08.571413+0000 2021-07-31T02:42:10.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:09 smithi045 conmon[39718]: mon.a (mon.0) 2918 : cluster [DBG] osdmap e1903: 8 total, 8 up, 8 in 2021-07-31T02:42:10.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:09 smithi045 conmon[39718]: cluster 2021-07-31T02:42:08.588623+0000 mgr.x (mgr. 2021-07-31T02:42:10.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:09 smithi045 conmon[39718]: 24542) 2353 : cluster [DBG] pgmap v3397: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:42:10.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:09 smithi045 conmon[33655]: cluster 2021-07-31T02:42:08.571413+0000 mon.a (mon.0) 2918 : 2021-07-31T02:42:10.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:09 smithi045 conmon[33655]: cluster [DBG] osdmap e1903: 8 total, 8 up, 8 in 2021-07-31T02:42:10.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:09 smithi045 conmon[33655]: cluster 2021-07-31T02:42:08.588623+0000 mgr.x (mgr.24542) 2353 : cluster [DBG] pgmap v3397: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:42:11.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:10 smithi154 conmon[34553]: cluster 2021-07-31T02:42:09.576726+0000 mon.a (mon.0) 2919 : cluster [DBG] osdmap e1904: 8 total, 8 up, 8 in 2021-07-31T02:42:11.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:10 smithi045 conmon[33655]: cluster 2021-07-31T02:42:09.576726+0000 mon.a (mon.0) 2919 : cluster [DBG] 2021-07-31T02:42:11.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:10 smithi045 conmon[33655]: osdmap e1904: 8 total, 8 up, 8 in 2021-07-31T02:42:11.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:10 smithi045 conmon[39718]: cluster 2021-07-31T02:42:09.576726+0000 mon.a (mon.0) 2919 : cluster [DBG] osdmap e1904: 8 total, 8 up, 8 in 2021-07-31T02:42:11.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:10 smithi045 conmon[39718]: 2021-07-31T02:42:12.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:11 smithi154 conmon[34553]: cluster 2021-07-31T02:42:10.583787+0000 mon.a (mon.0) 2920 : cluster [DBG] 2021-07-31T02:42:12.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:11 smithi154 conmon[34553]: osdmap e1905: 8 total, 8 up, 8 in 2021-07-31T02:42:12.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:11 smithi154 conmon[34553]: cluster 2021-07-31T02:42:10.589164+0000 mgr.x (mgr.24542) 2354 : cluster [DBG] pgmap v3400: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.2 MiB/s rd, 5.0 KiB/s wr, 1.68k op/s 2021-07-31T02:42:12.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:11 smithi045 conmon[39718]: cluster 2021-07-31T02:42:10.583787 2021-07-31T02:42:12.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:11 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2920 : cluster [DBG] osdmap e1905: 8 total, 8 up, 8 in 2021-07-31T02:42:12.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:11 smithi045 conmon[39718]: cluster 2021-07-31T 2021-07-31T02:42:12.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:11 smithi045 conmon[39718]: 02:42:10.589164+0000 mgr.x (mgr.24542) 2354 : cluster [DBG] pgmap v3400: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.2 MiB/s rd, 5.0 KiB/s wr, 1.68k op/s 2021-07-31T02:42:12.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:11 smithi045 conmon[33655]: cluster 2021-07-31T02:42:10.583787+0000 2021-07-31T02:42:12.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:11 smithi045 conmon[33655]: mon.a (mon.0) 2920 : cluster [DBG] osdmap e1905: 8 total, 8 up, 8 in 2021-07-31T02:42:12.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:11 smithi045 conmon[33655]: cluster 2021-07- 2021-07-31T02:42:12.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:11 smithi045 conmon[33655]: 31T02:42:10.589164+0000 mgr.x (mgr.24542) 2021-07-31T02:42:12.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:11 smithi045 conmon[33655]: 2354 : cluster [DBG] pgmap v3400: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.2 MiB/s rd, 5.0 KiB/s wr, 1.68k op/s 2021-07-31T02:42:13.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:12 smithi154 conmon[34553]: cluster 2021-07-31T02:42:11.588644+0000 mon.a (mon.0) 2921 : 2021-07-31T02:42:13.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:12 smithi154 conmon[34553]: cluster [DBG] osdmap e1906: 8 total, 8 up, 8 in 2021-07-31T02:42:13.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:12 smithi154 conmon[34553]: cluster 2021-07-31T02:42:12.541035+0000 mon.a (mon.0) 2922 : cluster [DBG] osdmap e1907: 8 total, 8 up, 8 in 2021-07-31T02:42:13.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:12 smithi045 conmon[33655]: cluster 2021-07-31T02:42:11.588644+0000 mon.a (mon. 2021-07-31T02:42:13.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:12 smithi045 conmon[33655]: 0) 2921 : cluster [DBG] osdmap e1906: 8 total, 8 up, 8 in 2021-07-31T02:42:13.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:12 smithi045 conmon[33655]: cluster 2021-07-31T02:42:12.541035+0000 mon.a (mon.0) 2922 : cluster [DBG] 2021-07-31T02:42:13.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:12 smithi045 conmon[33655]: osdmap e1907: 8 total, 8 up, 8 in 2021-07-31T02:42:13.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:12 smithi045 conmon[39718]: cluster 2021-07-31T02:42 2021-07-31T02:42:13.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:12 smithi045 conmon[39718]: :11.588644+0000 mon.a (mon.0) 2921 : cluster [DBG] osdmap e1906: 8 total, 8 up, 8 in 2021-07-31T02:42:13.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:12 smithi045 conmon[39718]: cluster 2021- 2021-07-31T02:42:13.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:12 smithi045 conmon[39718]: 07-31T02:42:12.541035+0000 mon.a (mon.0 2021-07-31T02:42:13.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:12 smithi045 conmon[39718]: ) 2922 : cluster [DBG] osdmap e1907: 8 total, 8 up, 8 in 2021-07-31T02:42:14.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:13 smithi154 conmon[34553]: cluster 2021-07-31T02:42:12.589562+0000 mgr.x (mgr.24542) 2021-07-31T02:42:14.121 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:13 smithi154 conmon[34553]: 2355 : cluster [DBG] pgmap v3403: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.2 MiB/s rd, 5.0 KiB/s wr, 1.68k op/s 2021-07-31T02:42:14.123 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:13 smithi045 conmon[33655]: cluster 2021-07-31T02:42:12.589562+0000 mgr.x (mgr.24542) 2355 : cluster [DBG] pgmap v3403: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.2 MiB/s rd, 5.0 KiB/s wr, 1.68k op/s 2021-07-31T02:42:14.123 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:13 smithi045 conmon[33655]: 2021-07-31T02:42:14.123 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:13 smithi045 conmon[39718]: cluster 2021-07-31T02:42:12.589562+0000 2021-07-31T02:42:14.124 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:13 smithi045 conmon[39718]: mgr.x (mgr.24542) 2355 : cluster [DBG] pgmap v3403: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.2 MiB/s rd, 5.0 KiB/s wr, 1.68k op/s 2021-07-31T02:42:15.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:14 smithi154 conmon[34553]: cluster 2021-07-31T02:42:13.597808+0000 mon.a (mon 2021-07-31T02:42:15.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:14 smithi154 conmon[34553]: .0) 2923 : cluster [DBG] osdmap e1908: 8 total, 8 up, 8 in 2021-07-31T02:42:15.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:14 smithi045 conmon[33655]: cluster 2021-07-31T02:42:13.597808+0000 mon.a (mon.0) 2923 : cluster [DBG] osdmap e1908: 8 total, 8 up, 8 in 2021-07-31T02:42:15.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:14 smithi045 conmon[33655]: 2021-07-31T02:42:15.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:14 smithi045 conmon[39718]: cluster 2021-07-31T02:42:13.597808+0000 mon.a (mon. 2021-07-31T02:42:15.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:14 smithi045 conmon[39718]: 0) 2923 : cluster [DBG] osdmap e1908: 8 total, 8 up, 8 in 2021-07-31T02:42:16.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:15 smithi154 conmon[34553]: cluster 2021-07-31T02:42:14.590336+0000 mgr.x (mgr.24542) 2356 : cluster [DBG] 2021-07-31T02:42:16.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:15 smithi154 conmon[34553]: pgmap v3405: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.2 MiB/s rd, 5.0 KiB/s wr, 1.63k op/s 2021-07-31T02:42:16.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:15 smithi154 conmon[34553]: cluster 2021-07-31T02:42:14.603366+0000 mon.a (mon.0) 2924 : cluster [DBG] osdmap e1909: 8 total, 8 up, 8 in 2021-07-31T02:42:16.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:15 smithi045 conmon[39718]: cluster 2021-07-31T02:42:14.590336+0000 mgr.x 2021-07-31T02:42:16.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:15 smithi045 conmon[39718]: (mgr.24542) 2356 : cluster [DBG] pgmap v3405: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.2 MiB/s rd, 5.0 KiB/s wr, 1.63k op/s 2021-07-31T02:42:16.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:15 smithi045 conmon[39718]: cluster 2021-07-31T02:42:14.603366+0000 2021-07-31T02:42:16.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:15 smithi045 conmon[39718]: mon.a (mon.0) 2924 : cluster [DBG] osdmap e1909: 8 total, 8 up, 8 in 2021-07-31T02:42:16.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:15 smithi045 conmon[33655]: cluster 2021-07-31T02:42:14.590336+0000 mgr.x (mgr.24542) 2356 : 2021-07-31T02:42:16.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:15 smithi045 conmon[33655]: cluster [DBG] pgmap v3405: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.2 MiB/s rd, 5.0 KiB/s wr, 1.63k op/s 2021-07-31T02:42:16.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:15 smithi045 conmon[33655]: cluster 2021-07-31T02:42:14.603366+0000 mon.a (mon.0) 2924 : cluster [DBG] osdmap e1909: 8 total, 8 up, 8 in 2021-07-31T02:42:17.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:16 smithi154 conmon[34553]: cluster 2021-07-31T02:42:15.608769+0000 mon.a (mon.0) 2925 : cluster [DBG] osdmap e1910: 8 total, 8 up, 8 in 2021-07-31T02:42:17.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:16 smithi045 conmon[39718]: cluster 2021-07-31T02:42:15.608769+0000 mon.a (mon 2021-07-31T02:42:17.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:16 smithi045 conmon[39718]: .0) 2925 : cluster [DBG] osdmap e1910: 8 total, 8 up, 8 in 2021-07-31T02:42:17.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:16 smithi045 conmon[33655]: cluster 2021-07-31T02:42:15.608769+0000 2021-07-31T02:42:17.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:16 smithi045 conmon[33655]: mon.a (mon.0) 2925 : cluster [DBG] osdmap e1910: 8 total, 8 up, 8 in 2021-07-31T02:42:18.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:17 smithi154 conmon[34553]: cluster 2021-07-31T02:42:16.591021+0000 mgr.x (mgr.24542) 2357 : cluster [DBG] pgmap v3408: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.2 MiB/s rd, 4.9 KiB/s wr, 1.62k op/s 2021-07-31T02:42:18.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:17 smithi154 conmon[34553]: cluster 2021-07-31T02:42 2021-07-31T02:42:18.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:17 smithi154 conmon[34553]: :16.614325+0000 mon.a (mon.0) 2926 : cluster [DBG] osdmap e1911: 8 total, 8 up, 8 in 2021-07-31T02:42:18.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:17 smithi154 conmon[34553]: cluster 2021-07-31T02:42:17.542370+0000 mon.a (mon.0) 2927 : cluster [DBG] osdmap e1912: 8 total, 8 up, 8 in 2021-07-31T02:42:18.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:17 smithi045 conmon[33655]: cluster 2021-07-31T02:42:16.591021+0000 mgr.x (mgr.24542) 2357 : cluster [DBG] pgmap v3408: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.2 MiB/s rd, 4.9 KiB/s wr, 1.62k op/s 2021-07-31T02:42:18.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:17 smithi045 conmon[33655]: cluster 2021-07 2021-07-31T02:42:18.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:17 smithi045 conmon[33655]: -31T02:42:16.614325+0000 mon.a (mon.0) 2926 : cluster [DBG] osdmap e1911: 8 total, 8 up, 8 in 2021-07-31T02:42:18.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:17 smithi045 conmon[33655]: cluster 2021-07-31T02:42:17.542370+0000 mon.a (mon.0) 2927 : cluster [DBG] osdmap e1912: 8 total, 8 up, 8 in 2021-07-31T02:42:18.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:17 smithi045 conmon[39718]: cluster 2021-07-31T02:42:16.591021+0000 2021-07-31T02:42:18.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:17 smithi045 conmon[39718]: mgr.x (mgr.24542) 2357 : cluster [DBG] pgmap v3408: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.2 MiB/s rd, 4.9 KiB/s wr, 1.62k op/s 2021-07-31T02:42:18.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:17 smithi045 conmon[39718]: cluster 2021-07-31T02:42:18.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:17 smithi045 conmon[39718]: 2021-07-31T02:42:16.614325+0000 mon.a (mon. 2021-07-31T02:42:18.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:17 smithi045 conmon[39718]: 0) 2926 : cluster [DBG] osdmap e1911: 8 total, 8 up, 8 in 2021-07-31T02:42:18.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:17 smithi045 conmon[39718]: cluster 2021- 2021-07-31T02:42:18.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:17 smithi045 conmon[39718]: 07-31T02:42:17.542370 2021-07-31T02:42:18.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:17 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2927 : cluster 2021-07-31T02:42:18.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:17 smithi045 conmon[39718]: [DBG] osdmap e1912: 8 total, 8 up, 8 in 2021-07-31T02:42:20.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:19 smithi154 conmon[34553]: cluster 2021-07-31T02:42:18.549289+0000 mon.a (mon.0) 2928 : cluster [DBG] 2021-07-31T02:42:20.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:19 smithi154 conmon[34553]: osdmap e1913: 8 total, 8 up, 8 in 2021-07-31T02:42:20.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:19 smithi154 conmon[34553]: cluster 2021-07-31T02:42:18.591389+0000 mgr.x (mgr.24542) 2358 : cluster [DBG] pgmap v3412: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:42:20.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:19 smithi045 conmon[33655]: cluster 2021-07-31T02:42:18.549289 2021-07-31T02:42:20.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:19 smithi045 conmon[33655]: +0000 mon.a (mon.0) 2928 : cluster [DBG] osdmap e1913: 8 total, 8 up, 8 in 2021-07-31T02:42:20.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:19 smithi045 conmon[33655]: cluster 2021-07-31T02:42:18.591389+0000 mgr.x (mgr.24542) 2358 2021-07-31T02:42:20.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:19 smithi045 conmon[33655]: : cluster [DBG] pgmap v3412: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:42:20.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:19 smithi045 conmon[39718]: cluster 2021-07-31T02:42:18.549289+0000 mon.a 2021-07-31T02:42:20.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:19 smithi045 conmon[39718]: (mon.0) 2928 : cluster [DBG] osdmap e1913: 8 total, 8 up, 8 in 2021-07-31T02:42:20.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:19 smithi045 conmon[39718]: cluster 2021-07-31T02:42:20.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:19 smithi045 conmon[39718]: 2021-07-31T02:42:18. 2021-07-31T02:42:20.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:19 smithi045 conmon[39718]: 591389+0000 mgr.x (mgr.24542) 2021-07-31T02:42:20.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:19 smithi045 conmon[39718]: 2358 : cluster [DBG] pgmap v3412: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:42:21.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:20 smithi154 conmon[34553]: cluster 2021-07-31T02:42:21.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:20 smithi154 conmon[34553]: 2021-07-31T02:42:21.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:20 smithi154 conmon[34553]: 2021 2021-07-31T02:42:21.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:20 smithi154 conmon[34553]: - 2021-07-31T02:42:21.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:20 smithi154 conmon[34553]: 07 2021-07-31T02:42:21.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:20 smithi154 conmon[34553]: - 2021-07-31T02:42:21.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:20 smithi154 conmon[34553]: 31 2021-07-31T02:42:21.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:20 smithi154 conmon[34553]: T 2021-07-31T02:42:21.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:20 smithi154 conmon[34553]: 02 2021-07-31T02:42:21.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:20 smithi154 conmon[34553]: : 2021-07-31T02:42:21.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:20 smithi154 conmon[34553]: 42 2021-07-31T02:42:21.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:20 smithi154 conmon[34553]: : 2021-07-31T02:42:21.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:20 smithi154 conmon[34553]: 19 2021-07-31T02:42:21.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:20 smithi154 conmon[34553]: . 2021-07-31T02:42:21.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:20 smithi154 conmon[34553]: 556233+0000 mon.a (mon.0) 2929 : cluster [DBG] 2021-07-31T02:42:21.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:20 smithi154 conmon[34553]: osdmap e1914: 8 total, 8 up, 8 in 2021-07-31T02:42:21.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:20 smithi045 conmon[33655]: cluster 2021-07-31T02:42:19. 2021-07-31T02:42:21.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:20 smithi045 conmon[33655]: 556233+0000 mon.a (mon.0) 2929 : cluster [DBG] osdmap e1914: 8 total, 8 up, 8 in 2021-07-31T02:42:21.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:20 smithi045 conmon[39718]: cluster 2021-07-31T02:42:19.556233 2021-07-31T02:42:21.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:20 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2929 : cluster [DBG] osdmap e1914: 8 total, 8 up, 8 in 2021-07-31T02:42:22.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:21 smithi154 conmon[34553]: cluster 2021-07-31T02:42:22.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:21 smithi154 conmon[34553]: 2021-07-31T02:42:20.557712+0000 mon.a ( 2021-07-31T02:42:22.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:21 smithi154 conmon[34553]: mon.0) 2930 : cluster [DBG] osdmap e1915: 8 total, 8 up, 8 in 2021-07-31T02:42:22.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:21 smithi154 conmon[34553]: cluster 2021-07 2021-07-31T02:42:22.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:21 smithi154 conmon[34553]: -31T02:42:20.591753+0000 mgr.x (mgr.24542) 2359 : cluster [DBG] 2021-07-31T02:42:22.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:21 smithi154 conmon[34553]: pgmap v3415: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.2 MiB/s rd, 5.0 KiB/s wr, 1.60k op/s 2021-07-31T02:42:22.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:21 smithi045 conmon[33655]: cluster 2021-07-31T02:42:20.557712+0000 mon.a (mon.0) 2930 : cluster [DBG] osdmap e1915: 8 total, 8 up, 8 in 2021-07-31T02:42:22.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:21 smithi045 conmon[33655]: cluster 2021-07-31T02:42:20.591753+0000 mgr.x (mgr.24542) 2359 : cluster [DBG] pgmap v3415: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.2 MiB/s rd, 5.0 KiB/s wr, 1.60k op/s 2021-07-31T02:42:22.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:21 smithi045 conmon[39718]: cluster 2021-07-31T02:42:20.557712+0000 2021-07-31T02:42:22.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:21 smithi045 conmon[39718]: mon.a (mon.0) 2930 : cluster [DBG] osdmap e1915: 8 total, 8 up, 8 in 2021-07-31T02:42:22.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:21 smithi045 conmon[39718]: cluster 2021-07-31T02:42:20.591753+0000 mgr.x (mgr.24542) 2021-07-31T02:42:22.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:21 smithi045 conmon[39718]: 2359 : cluster [DBG] pgmap v3415: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.2 MiB/s rd, 5.0 KiB/s wr, 1.60k op/s 2021-07-31T02:42:23.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:22 smithi154 conmon[34553]: cluster 2021-07-31T02:42:21.570100+0000 mon.a (mon.0) 2931 : cluster [DBG] osdmap e1916: 8 total, 8 up, 8 in 2021-07-31T02:42:23.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:22 smithi154 conmon[34553]: cluster 2021-07-31T02:42:22.543214+0000 mon.a (mon.0) 2932 : cluster [DBG] osdmap e1917: 8 total, 8 up, 8 in 2021-07-31T02:42:23.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:22 smithi045 conmon[33655]: cluster 2021-07-31T02:42:21.570100+0000 mon.a (mon.0) 2931 : cluster [DBG] osdmap e1916: 8 total, 8 up, 8 in 2021-07-31T02:42:23.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:22 smithi045 conmon[33655]: cluster 2021-07-31T02:42:22.543214+0000 mon.a (mon.0) 2932 : cluster [DBG] osdmap e1917: 8 total, 8 up, 8 in 2021-07-31T02:42:23.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:22 smithi045 conmon[39718]: cluster 2021-07-31T02:42:21.570100+0000 mon.a (mon.0) 2931 : cluster 2021-07-31T02:42:23.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:22 smithi045 conmon[39718]: [DBG] osdmap e1916: 8 total, 8 up, 8 in 2021-07-31T02:42:23.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:22 smithi045 conmon[39718]: cluster 2021-07-31T02:42:22.543214+0000 mon.a (mon.0) 2021-07-31T02:42:23.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:22 smithi045 conmon[39718]: 2932 : cluster [DBG] osdmap e1917: 8 total, 8 up, 8 in 2021-07-31T02:42:24.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:23 smithi154 conmon[34553]: cluster 2021-07-31T02:42:22.592085+0000 mgr.x (mgr.24542) 2021-07-31T02:42:24.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:23 smithi154 conmon[34553]: 2360 : cluster [DBG] pgmap v3418: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.2 MiB/s rd, 5.0 KiB/s wr, 1.59k op/s 2021-07-31T02:42:24.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:23 smithi045 conmon[39718]: cluster 2021-07-31T02:42:24.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:23 smithi045 conmon[39718]: 2021-07-31T02:42:22.592085+0000 mgr.x 2021-07-31T02:42:24.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:23 smithi045 conmon[39718]: (mgr.24542) 2360 : cluster [DBG] pgmap v3418: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.2 MiB/s rd, 5.0 KiB/s wr, 1.59k op/s 2021-07-31T02:42:24.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:23 smithi045 conmon[33655]: cluster 2021-07-31T02:42:22.592085+0000 mgr.x (mgr.24542) 2360 : cluster 2021-07-31T02:42:24.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:23 smithi045 conmon[33655]: [DBG] pgmap v3418: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.2 MiB/s rd, 5.0 KiB/s wr, 1.59k op/s 2021-07-31T02:42:25.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:24 smithi154 conmon[34553]: cluster 2021-07-31T02:42:25.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:24 smithi154 conmon[34553]: 2021-07-31T02:42:23. 2021-07-31T02:42:25.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:24 smithi154 conmon[34553]: 576823+0000 mon.a (mon.0) 2933 : cluster [DBG] osdmap e1918: 8 total, 8 up, 8 in 2021-07-31T02:42:25.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:24 smithi045 conmon[39718]: cluster 2021-07-31T02:42:23.576823+0000 mon.a (mon.0) 2933 : cluster [DBG] osdmap e1918: 8 total, 8 up, 8 in 2021-07-31T02:42:25.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:24 smithi045 conmon[33655]: cluster 2021-07-31T02:42:23.576823+0000 2021-07-31T02:42:25.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:24 smithi045 conmon[33655]: mon.a (mon.0) 2933 : cluster [DBG] osdmap e1918: 8 total, 8 up, 8 in 2021-07-31T02:42:26.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:25 smithi154 conmon[34553]: cluster 2021-07-31T02:42:24.578566+0000 mon.a (mon.0) 2934 : cluster [DBG] osdmap e1919: 8 total, 8 up, 8 in 2021-07-31T02:42:26.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:25 smithi154 conmon[34553]: cluster 2021-07-31T02:42:24.592531+0000 mgr.x (mgr.24542) 2361 : cluster [DBG] pgmap v3421: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.1 MiB/s rd, 5.0 KiB/s wr, 1.55k op/s 2021-07-31T02:42:26.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:25 smithi045 conmon[33655]: cluster 2021-07-31T02:42:24.578566+0000 mon.a (mon.0) 2934 : cluster 2021-07-31T02:42:26.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:25 smithi045 conmon[33655]: [DBG] osdmap e1919: 8 total, 8 up, 8 in 2021-07-31T02:42:26.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:25 smithi045 conmon[33655]: cluster 2021-07-31T02:42:24.592531+0000 mgr.x (mgr.24542) 2361 : cluster [DBG] pgmap v3421: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.1 MiB/s rd, 5.0 KiB/s wr, 1.55k op/s 2021-07-31T02:42:26.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:25 smithi045 conmon[39718]: cluster 2021-07-31T02:42:24 2021-07-31T02:42:26.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:25 smithi045 conmon[39718]: .578566+0000 mon.a (mon.0) 2934 : cluster [DBG] osdmap e1919: 8 total, 8 up, 8 in 2021-07-31T02:42:26.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:25 smithi045 conmon[39718]: cluster 2021-07-31T02: 2021-07-31T02:42:26.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:25 smithi045 conmon[39718]: 42:24.592531+0000 mgr.x (mgr.24542) 2361 : cluster 2021-07-31T02:42:26.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:25 smithi045 conmon[39718]: [DBG] pgmap v3421: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.1 MiB/s rd, 5.0 KiB/s wr, 1.55k op/s 2021-07-31T02:42:27.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:26 smithi154 conmon[34553]: cluster 2021-07-31T02:42:25.593862+0000 mon.a (mon 2021-07-31T02:42:27.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:26 smithi154 conmon[34553]: .0) 2935 : cluster [DBG] osdmap e1920: 8 total, 8 up, 8 in 2021-07-31T02:42:27.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:26 smithi045 conmon[33655]: cluster 2021-07-31T02:42:25.593862+0000 mon.a (mon.0) 2935 2021-07-31T02:42:27.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:26 smithi045 conmon[33655]: : cluster [DBG] osdmap e1920: 8 total, 8 up, 8 in 2021-07-31T02:42:27.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:26 smithi045 conmon[39718]: cluster 2021-07-31T02:42:25.593862+0000 mon.a (mon 2021-07-31T02:42:27.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:26 smithi045 conmon[39718]: .0) 2935 : cluster [DBG] osdmap e1920: 8 total, 8 up, 8 in 2021-07-31T02:42:28.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:27 smithi154 conmon[34553]: cluster 2021-07-31T02:42:26.592952+0000 mgr.x (mgr.24542) 2021-07-31T02:42:28.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:27 smithi154 conmon[34553]: 2362 : cluster [DBG] pgmap v3423: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.1 MiB/s rd, 4.9 KiB/s wr, 1.53k op/s 2021-07-31T02:42:28.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:27 smithi154 conmon[34553]: cluster 2021-07-31T02:42:26.598818+0000 mon.a (mon.0) 2936 : cluster [DBG] osdmap e1921: 8 total, 8 up, 8 in 2021-07-31T02:42:28.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:27 smithi154 conmon[34553]: cluster 2021-07-31T02:42:28.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:27 smithi154 conmon[34553]: 2021-07-31T02:42:27.545412+0000 mon.a (mon.0) 2937 : cluster [DBG] osdmap e1922: 8 total, 8 up, 8 in 2021-07-31T02:42:28.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:27 smithi045 conmon[39718]: cluster 2021-07-31T02:42:26.592952+0000 mgr.x 2021-07-31T02:42:28.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:27 smithi045 conmon[39718]: (mgr.24542) 2362 : cluster [DBG] pgmap v3423: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.1 MiB/s rd, 4.9 KiB/s wr, 1.53k op/s 2021-07-31T02:42:28.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:27 smithi045 conmon[39718]: cluster 2021-07-31T02:42:28.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:27 smithi045 conmon[39718]: 2021-07-31T02:42:26.598818+0000 mon.a (mon.0) 2936 : cluster [DBG] osdmap e1921: 8 total, 8 up, 8 in 2021-07-31T02:42:28.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:27 smithi045 conmon[39718]: cluster 2021-07-31 2021-07-31T02:42:28.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:27 smithi045 conmon[39718]: T02:42:27.545412+0000 mon.a (mon.0) 2937 : cluster [DBG] 2021-07-31T02:42:28.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:27 smithi045 conmon[39718]: osdmap e1922: 8 total, 8 up, 8 in 2021-07-31T02:42:28.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:27 smithi045 conmon[33655]: cluster 2021-07-31T02:42:26.592952+0000 mgr.x (mgr.24542) 2362 2021-07-31T02:42:28.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:27 smithi045 conmon[33655]: : cluster [DBG] pgmap v3423: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.1 MiB/s rd, 4.9 KiB/s wr, 1.53k op/s 2021-07-31T02:42:28.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:27 smithi045 conmon[33655]: cluster 2021-07-31T02:42:26.598818+0000 mon.a (mon.0) 2936 : cluster [DBG] osdmap e1921: 8 total, 8 up, 8 in 2021-07-31T02:42:28.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:27 smithi045 conmon[33655]: cluster 2021-07-31T02:42:27.545412+0000 mon.a (mon.0 2021-07-31T02:42:28.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:27 smithi045 conmon[33655]: ) 2937 : cluster [DBG] osdmap e1922: 8 total, 8 up, 8 in 2021-07-31T02:42:29.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:28 smithi154 conmon[34553]: audit 2021-07-31T02:42:28.145409+0000 mon.b (mon.2) 350 2021-07-31T02:42:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:28 smithi154 conmon[34553]: : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:42:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:28 smithi154 conmon[34553]: audit 2021-07-31T02:42:28.146075+0000 mon.a (mon.0) 2938 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:42:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:28 smithi154 conmon[34553]: audit 2021-07-31T02:42:28.151871+0000 mon.b (mon.2) 351 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:42:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:28 smithi154 conmon[34553]: audit 2021-07-31T02:42:28.152335+0000 mon.a (mon.0) 2939 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:42:29.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:28 smithi045 conmon[39718]: audit 2021-07-31T02:42:28.145409+0000 mon.b ( 2021-07-31T02:42:29.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:28 smithi045 conmon[39718]: mon.2) 350 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:42:29.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:28 smithi045 conmon[39718]: audit 2021-07-31T02:42 2021-07-31T02:42:29.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:28 smithi045 conmon[39718]: :28.146075+0000 mon.a (mon.0) 2938 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:42:29.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:28 smithi045 conmon[39718]: audit 2021-07 2021-07-31T02:42:29.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:28 smithi045 conmon[39718]: -31T02:42:28.151871+0000 mon.b (mon.2) 351 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:42:29.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:28 smithi045 conmon[39718]: 2021-07-31T02:42:29.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:28 smithi045 conmon[39718]: audit 2021-07-31T02:42:28.152335+0000 mon.a 2021-07-31T02:42:29.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:28 smithi045 conmon[39718]: (mon.0) 2939 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:42:29.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:28 smithi045 conmon[33655]: audit 2021-07-31T02:42:28.145409+0000 mon.b (mon.2) 350 : audit [INF] 2021-07-31T02:42:29.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:28 smithi045 conmon[33655]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:42:29.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:28 smithi045 conmon[33655]: audit 2021-07-31T02:42:28.146075+0000 mon.a (mon.0) 2938 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:42:29.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:28 smithi045 conmon[33655]: audit 2021-07-31T02:42:28 2021-07-31T02:42:29.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:28 smithi045 conmon[33655]: .151871+0000 mon.b (mon.2) 351 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:42:29.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:28 smithi045 conmon[33655]: audit 2021-07-31 2021-07-31T02:42:29.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:28 smithi045 conmon[33655]: T02:42:28.152335+0000 mon.a (mon.0) 2939 : audit [INF] 2021-07-31T02:42:29.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:28 smithi045 conmon[33655]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:42:30.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:29 smithi154 conmon[34553]: cluster 2021-07-31T02:42:28.593445+0000 mgr.x (mgr.24542) 2363 : cluster [DBG] pgmap v3426: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:42:30.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:29 smithi154 conmon[34553]: cluster 2021-07-31T02:42:28.611074+0000 mon.a (mon.0) 2940 : cluster [DBG] osdmap e1923: 8 total, 8 up, 8 in 2021-07-31T02:42:30.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:29 smithi045 conmon[39718]: cluster 2021-07-31T02:42:28.593445+0000 mgr.x (mgr 2021-07-31T02:42:30.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:29 smithi045 conmon[39718]: .24542) 2363 : cluster [DBG] pgmap v3426: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:42:30.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:29 smithi045 conmon[39718]: cluster 2021-07-31T02:42:28.611074+0000 mon.a (mon.0) 2940 : cluster [DBG] osdmap e1923: 8 total, 8 up, 8 in 2021-07-31T02:42:30.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:29 smithi045 conmon[33655]: cluster 2021-07-31T02:42:28.593445+0000 mgr.x (mgr.24542) 2363 : cluster [DBG] pgmap v3426: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:42:30.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:29 smithi045 conmon[33655]: cluster 2021-07-31T02:42:28.611074+0000 mon.a (mon.0) 2940 : cluster [DBG] osdmap e1923: 8 total, 8 up, 8 in 2021-07-31T02:42:31.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:30 smithi154 conmon[34553]: cluster 2021-07-31T02:42:29.615613+0000 mon.a (mon.0) 2021-07-31T02:42:31.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:30 smithi154 conmon[34553]: 2941 : cluster [DBG] osdmap e1924: 8 total, 8 up, 8 in 2021-07-31T02:42:31.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:30 smithi045 conmon[39718]: cluster 2021-07-31T02:42:29.615613 2021-07-31T02:42:31.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:30 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2941 : cluster [DBG] osdmap e1924: 8 total, 8 up, 8 in 2021-07-31T02:42:31.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:30 smithi045 conmon[33655]: cluster 2021-07-31T02:42:29.615613+0000 mon.a (mon.0) 2941 : cluster [DBG] osdmap e1924: 8 total, 8 up, 8 in 2021-07-31T02:42:31.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:30 smithi045 conmon[33655]: 2021-07-31T02:42:32.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:31 smithi154 conmon[34553]: cluster 2021-07-31T02:42:32.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:31 smithi154 conmon[34553]: 2021-07-31T02:42:32.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:31 smithi154 conmon[34553]: 2021 2021-07-31T02:42:32.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:31 smithi154 conmon[34553]: - 2021-07-31T02:42:32.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:31 smithi154 conmon[34553]: 07 2021-07-31T02:42:32.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:31 smithi154 conmon[34553]: - 2021-07-31T02:42:32.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:31 smithi154 conmon[34553]: 31 2021-07-31T02:42:32.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:31 smithi154 conmon[34553]: T 2021-07-31T02:42:32.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:31 smithi154 conmon[34553]: 02 2021-07-31T02:42:32.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:31 smithi154 conmon[34553]: : 2021-07-31T02:42:32.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:31 smithi154 conmon[34553]: 42 2021-07-31T02:42:32.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:31 smithi154 conmon[34553]: : 2021-07-31T02:42:32.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:31 smithi154 conmon[34553]: 30 2021-07-31T02:42:32.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:31 smithi154 conmon[34553]: . 2021-07-31T02:42:32.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:31 smithi154 conmon[34553]: 593882 2021-07-31T02:42:32.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:31 smithi154 conmon[34553]: +0000 2021-07-31T02:42:32.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:31 smithi154 conmon[34553]: 2021-07-31T02:42:32.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:31 smithi154 conmon[34553]: mgr.x 2021-07-31T02:42:32.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:31 smithi154 conmon[34553]: ( 2021-07-31T02:42:32.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:31 smithi154 conmon[34553]: mgr 2021-07-31T02:42:32.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:31 smithi154 conmon[34553]: .24542) 2364 : cluster [DBG] pgmap v3429: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.1 MiB/s rd, 5.0 KiB/s wr, 1.50k op/s 2021-07-31T02:42:32.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:31 smithi154 conmon[34553]: cluster 2021-07-31T02:42 2021-07-31T02:42:32.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:31 smithi154 conmon[34553]: :30.622665+0000 mon.a (mon.0) 2942 : cluster [DBG] osdmap e1925: 8 total, 8 up, 8 in 2021-07-31T02:42:32.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:31 smithi045 conmon[39718]: cluster 2021-07-31T02:42:30.593882+0000 mgr.x (mgr 2021-07-31T02:42:32.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:31 smithi045 conmon[39718]: .24542) 2364 : cluster [DBG] pgmap v3429: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.1 MiB/s rd, 5.0 KiB/s wr, 1.50k op/s 2021-07-31T02:42:32.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:31 smithi045 conmon[39718]: cluster 2021-07-31T02:42:30. 2021-07-31T02:42:32.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:31 smithi045 conmon[39718]: 622665+0000 mon.a (mon.0) 2942 : cluster [DBG] osdmap e1925: 8 total, 8 up, 8 in 2021-07-31T02:42:32.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:31 smithi045 conmon[33655]: cluster 2021-07-31T02:42:30.593882+0000 mgr.x (mgr 2021-07-31T02:42:32.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:31 smithi045 conmon[33655]: .24542) 2364 : cluster [DBG] pgmap v3429: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.1 MiB/s rd, 5.0 KiB/s wr, 1.50k op/s 2021-07-31T02:42:32.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:31 smithi045 conmon[33655]: cluster 2021-07-31T02:42:30.622665+0000 mon.a (mon.0) 2942 : cluster [DBG] osdmap e1925: 8 total, 8 up, 8 in 2021-07-31T02:42:32.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:31 smithi045 conmon[33655]: 2021-07-31T02:42:33.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:32 smithi154 conmon[34553]: cluster 2021-07-31T02:42:31.623183+0000 mon.a (mon.0) 2943 : cluster [DBG] osdmap e1926: 8 total, 8 up, 8 in 2021-07-31T02:42:33.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:32 smithi154 conmon[34553]: cluster 2021-07-31T02:42:32.546678+0000 mon.a (mon.0) 2944 : cluster [DBG] osdmap e1927: 8 total, 8 up, 8 in 2021-07-31T02:42:33.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:32 smithi045 conmon[39718]: cluster 2021-07-31T02:42:31.623183+0000 2021-07-31T02:42:33.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:32 smithi045 conmon[39718]: mon.a (mon.0) 2943 : cluster [DBG] osdmap e1926: 8 total, 8 up, 8 in 2021-07-31T02:42:33.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:32 smithi045 conmon[39718]: cluster 2021-07- 2021-07-31T02:42:33.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:32 smithi045 conmon[39718]: 31T02:42:32.546678+0000 mon.a ( 2021-07-31T02:42:33.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:32 smithi045 conmon[39718]: mon.0) 2944 : cluster [DBG] osdmap e1927: 8 total, 8 up, 8 in 2021-07-31T02:42:33.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:32 smithi045 conmon[33655]: cluster 2021-07-31T02:42:31.623183+0000 mon.a (mon.0) 2943 : cluster [DBG] 2021-07-31T02:42:33.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:32 smithi045 conmon[33655]: osdmap e1926: 8 total, 8 up, 8 in 2021-07-31T02:42:33.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:32 smithi045 conmon[33655]: cluster 2021-07-31T02:42:32.546678+0000 mon.a (mon.0) 2944 : cluster [DBG] osdmap e1927: 8 total, 8 up, 8 in 2021-07-31T02:42:33.752 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: debug 2021-07-31T02:42:33.436+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:42:33.753 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:42:33.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:42:33.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: yield 2021-07-31T02:42:33.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:42:33.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:42:33.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:42:33.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:42:33.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:42:33.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:42:33.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:42:33.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:42:33.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:42:33.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: task.result() 2021-07-31T02:42:33.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:42:33.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: debug 2021-07-31T02:42:33.447+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:42:33.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:42:33.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:42:33.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: yield 2021-07-31T02:42:33.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:42:33.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:42:33.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:42:33.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:42:33.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:42:33.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:42:33.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:42:33.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:42:33.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:42:33.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: task.result() 2021-07-31T02:42:33.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:42:33.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: debug 2021-07-31T02:42:33.474+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:42:33.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:42:33.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:42:33.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: yield 2021-07-31T02:42:33.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:42:33.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:42:33.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:42:33.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:42:33.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:42:33.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: cluster 2021-07-31T02:42:32.594441+0000 mgr.x (mgr.24542) 2021-07-31T02:42:33.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: 2365 : cluster [DBG] pgmap v3433: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.1 MiB/s rd, 5.0 KiB/s wr, 1.51k op/s 2021-07-31T02:42:33.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: audit 2021-07-31T02:42:33.389678+0000 mon.b (mon.2) 352 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:42:33.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: audit 2021- 2021-07-31T02:42:33.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: 07-31T02:42:33.391112+0000 mon.b (mon.2) 353 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:42:33.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: audit 2021-07 2021-07-31T02:42:33.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: -31T02:42:33.392175+0000 mon.b 2021-07-31T02:42:33.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: (mon.2) 354 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:42:33.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:42:33.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: 2021-07-31T02:42:33.392949 2021-07-31T02:42:33.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: +0000 mgr.x (mgr.24542) 2366 : cephadm [WRN] 2021-07-31T02:42:33.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:42:33.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: cephadm 2021-07-31T02 2021-07-31T02:42:33.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: :42:33.437055+0000 mgr.x (mgr.24542 2021-07-31T02:42:33.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: ) 2367 : cephadm [ERR] 2021-07-31T02:42:33.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:42:33.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:42:33.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: yield 2021-07-31T02:42:33.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:42:33.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:42:33.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:42:33.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:42:33.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:42:33.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:42:33.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:42:33.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:42:33.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:42:33.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:42:33.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:42:33.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:42:33.437675+0000 mgr.x (mgr.24542) 2368 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:42:33.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:42:33.448614+0000 mgr.x (mgr.24542) 2021-07-31T02:42:33.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: 2369 : cephadm [ERR] 2021-07-31T02:42:33.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:42:33.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:42:33.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: yield 2021-07-31T02:42:33.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:42:33.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:42:33.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:42:33.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:42:33.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:42:33.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:42:33.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: task.result() 2021-07-31T02:42:33.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:42:33.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: debug 2021-07-31T02:42:33.474+0000 7fcb93961700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:42:33.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:42:33.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:42:33.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: yield 2021-07-31T02:42:33.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:42:33.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:42:33.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:42:33.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:42:33.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:42:33.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:42:33.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:42:33.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:42:33.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:42:33.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: task.result() 2021-07-31T02:42:33.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:42:33.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: 2021-07-31T02:42:33.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:42:33.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: 2021-07-31T02:42:33.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:42:33.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:42:33.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:42:33.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:42:33.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:42:33.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:42:33.786 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:42:33.786 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:42:33.786 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:42:33.787 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:42:33.787 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:42:33.787 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:42:33.787 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:42:33.788 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:42:33.788 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:42:33.788 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:42:33.788 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:42:33.788 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:42:33.789 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: raise 2021-07-31T02:42:33.789 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:42:33.789 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:42:33.790 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:42:33.790 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:42:33.790 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:42:33.790 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: debug 2021-07-31T02:42:33.482+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:42:33.790 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:42:33.791 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:42:33.791 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: yield 2021-07-31T02:42:33.791 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:42:33.791 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:42:33.792 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:42:33.792 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:42:33.792 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:42:33.792 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:42:33.792 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:42:33.793 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:42:33.793 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:42:33.793 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: task.result() 2021-07-31T02:42:33.793 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:42:33.794 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: debug 2021-07-31T02:42:33.483+0000 7fcb94162700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:42:33.794 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:42:33.794 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:42:33.794 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: yield 2021-07-31T02:42:33.794 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:42:33.795 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:42:33.795 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:42:33.795 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:42:33.795 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:42:33.795 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:42:33.796 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:42:33.796 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:42:33.796 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:42:33.797 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: task.result() 2021-07-31T02:42:33.797 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:42:33.797 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: 2021-07-31T02:42:33.797 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:42:33.798 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: 2021-07-31T02:42:33.798 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:42:33.798 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:42:33.799 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:42:33.799 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:42:33.799 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:42:33.800 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:42:33.800 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:42:33.800 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:42:33.800 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:42:33.800 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:42:33.801 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:42:33.449072+0000 mgr.x (mgr.24542) 2370 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:42:33.801 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:42:33.474935+0000 mgr.x (mgr.24542) 2371 : cephadm [ERR] 2021-07-31T02:42:33.801 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: 2021-07-31T02:42:33.801 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:42:33.801 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:42:33.802 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: yield 2021-07-31T02:42:33.802 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:42:33.802 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:42:33.802 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:42:33.803 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:42:33.803 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:42:33.803 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:42:33.803 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:42:33.803 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:42:33.804 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:42:33.804 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:42:33.804 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:42:33.804 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: cephadm 2021-07-31T02:42:33.475365+0000 mgr.x (mgr.24542) 2372 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:42:33.804 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:42:33.805 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:42:33.805 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: yield 2021-07-31T02:42:33.805 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:42:33.805 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:42:33.805 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:42:33.806 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:42:33.806 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:42:33.806 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:42:33.807 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:42:33.807 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:42:33.807 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:42:33.807 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: task.result() 2021-07-31T02:42:33.808 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:42:33.808 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:42:33.808 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:42:33.809 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:42:33.809 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:42:33.809 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:42:33.809 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:42:33.809 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:42:33.810 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:42:33.810 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:42:33.810 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:42:33.810 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:42:33.810 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:42:33.811 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:42:33.811 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:42:33.811 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:42:33.811 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:42:33.811 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: raise 2021-07-31T02:42:33.812 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:42:33.812 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:42:33.812 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:42:33.812 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:42:33.813 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:42:33.813 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:42:33.813 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: 2021-07-31T02:42:33.813 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:42:33.814 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: 2021-07-31T02:42:33.814 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:42:33.814 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:42:33.814 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:42:33.815 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:42:33.815 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:42:33.815 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:42:33.815 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:42:33.816 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:42:33.816 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:42:33.816 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:42:33.816 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:42:33.817 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:42:33.817 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:42:33.817 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:42:33.817 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:42:33.817 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:42:33.818 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:42:33.818 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:42:33.818 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: raise 2021-07-31T02:42:33.818 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:42:33.819 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:42:33.819 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:42:33.819 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:42:33.819 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:42:33.819 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: cluster 2021-07-31T02:42:33.555427+0000 mon.a ( 2021-07-31T02:42:33.820 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:33 smithi154 conmon[34553]: mon.0) 2945 : cluster [DBG] osdmap e1928: 8 total, 8 up, 8 in 2021-07-31T02:42:33.820 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: cluster 2021-07-31T02:42:32.594441+0000 2021-07-31T02:42:33.821 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: mgr.x (mgr.24542) 2365 : cluster [DBG] pgmap v3433: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.1 MiB/s rd, 5.0 KiB/s wr, 1.51k op/s 2021-07-31T02:42:33.821 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T02:42:33.821 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: 02:42:33.389678+0000 2021-07-31T02:42:33.821 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: mon.b (mon.2) 352 : audit [DBG] 2021-07-31T02:42:33.822 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:42:33.822 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: audit 2021 2021-07-31T02:42:33.822 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: -07-31T02:42: 2021-07-31T02:42:33.822 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: 33.391112+0000 mon.b (mon 2021-07-31T02:42:33.822 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: .2) 353 : audit [DBG] 2021-07-31T02:42:33.823 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:42:33.823 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: audit 2021-07-31T02:42:33.823 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: 2021-07-31T02: 2021-07-31T02:42:33.823 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: 42:33.392175+0000 mon.b ( 2021-07-31T02:42:33.824 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: mon.2) 354 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:42:33.824 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: 2021-07-31T02:42:33.824 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:42:33.824 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: 2021-07-31T02:42:33.392949+0000 mgr.x 2021-07-31T02:42:33.824 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: (mgr.24542) 2366 : cephadm [WRN] 2021-07-31T02:42:33.825 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:42:33.825 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:42:33.825 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: 2021-07 2021-07-31T02:42:33.825 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: -31T02:42:33.437055 2021-07-31T02:42:33.825 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 2367 : cephadm [ERR] 2021-07-31T02:42:33.826 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:42:33.826 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:42:33.826 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: yield 2021-07-31T02:42:33.826 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:42:33.827 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:42:33.827 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:42:33.827 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:42:33.827 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:42:33.828 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:42:33.828 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:42:33.828 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:42:33.828 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:42:33.828 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:42:33.828 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:42:33.829 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:42:33.437675+0000 mgr.x (mgr.24542) 2368 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:42:33.829 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:42:33.829 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: 2021-07- 2021-07-31T02:42:33.829 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: 31T02:42:33.448614+0000 2021-07-31T02:42:33.829 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: mgr.x (mgr.24542) 2369 : cephadm 2021-07-31T02:42:33.830 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: [ERR] 2021-07-31T02:42:33.830 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:42:33.830 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:42:33.830 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: yield 2021-07-31T02:42:33.831 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:42:33.831 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:42:33.831 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:42:33.831 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:42:33.832 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:42:33.832 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:42:33.832 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:42:33.832 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:42:33.833 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:42:33.833 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:42:33.833 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:42:33.833 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:42:33.449072+0000 mgr.x (mgr.24542) 2370 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:42:33.834 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: cephadm 2021-07-31 2021-07-31T02:42:33.834 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: T02:42:33.474935+0000 mgr.x (mgr.24542) 2021-07-31T02:42:33.834 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: 2371 : cephadm [ERR] 2021-07-31T02:42:33.834 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:42:33.835 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:42:33.835 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: yield 2021-07-31T02:42:33.835 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:42:33.835 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:42:33.836 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:42:33.836 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:42:33.836 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:42:33.836 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:42:33.837 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:42:33.837 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:42:33.837 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:42:33.837 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:42:33.838 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:42:33.838 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:42:33.475365+0000 mgr.x (mgr.24542) 2372 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:42:33.838 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:42:33.838 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:42:33.839 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: yield 2021-07-31T02:42:33.839 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:42:33.839 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:42:33.839 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:42:33.839 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:42:33.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: cluster 2021-07-31T02:42:32.594441+0000 mgr.x (mgr.24542 2021-07-31T02:42:33.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: ) 2365 : cluster [DBG] pgmap v3433: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.1 MiB/s rd, 5.0 KiB/s wr, 1.51k op/s 2021-07-31T02:42:33.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: audit 2021-07-31T02:42:33.389678+0000 mon.b (mon.2) 352 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:42:33.841 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: audit 2021-07-31T02:42 2021-07-31T02:42:33.841 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: :33.391112+0000 mon.b (mon.2) 353 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:42:33.841 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: audit 2021- 2021-07-31T02:42:33.841 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: 07-31T02:42:33.392175+0000 mon.b (mon.2) 354 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:42:33.842 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: cephadm 2021-07-31T02: 2021-07-31T02:42:33.842 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: 42:33.392949+0000 mgr.x (mgr.24542) 2366 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:42:33.842 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: cephadm 2021-07-31 2021-07-31T02:42:33.842 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: T02:42:33.437055+0000 mgr.x (mgr.24542) 2367 : cephadm [ERR] 2021-07-31T02:42:33.842 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:42:33.843 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:42:33.843 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: yield 2021-07-31T02:42:33.843 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:42:33.843 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:42:33.844 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:42:33.844 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:42:33.844 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:42:33.844 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:42:33.845 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:42:33.845 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:42:33.845 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:42:33.845 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:42:33.845 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:42:33.846 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: 2021-07-31T02:42:33.846 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:42:33.437675+0000 mgr.x (mgr.24542) 2368 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:42:33.846 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:42:33.846 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: 2021-07-31T02:42 2021-07-31T02:42:33.847 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: :33.448614+0000 mgr.x (mgr.24542) 2369 : cephadm 2021-07-31T02:42:33.847 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: [ERR] 2021-07-31T02:42:33.847 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:42:33.847 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:42:33.848 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: yield 2021-07-31T02:42:33.848 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:42:33.848 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:42:33.848 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:42:33.849 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:42:33.849 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:42:33.849 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:42:33.849 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:42:33.850 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:42:33.850 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: task.result() 2021-07-31T02:42:33.850 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:42:33.850 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: 2021-07-31T02:42:33.851 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:42:33.851 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: 2021-07-31T02:42:33.851 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:42:33.851 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:42:33.851 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:42:33.852 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:42:33.852 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:42:33.852 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:42:33.852 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:42:33.853 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:42:33.853 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:42:33.853 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:42:33.853 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:42:33.853 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:42:33.854 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:42:33.854 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:42:33.854 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:42:33.854 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:42:33.854 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:42:33.855 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:42:33.855 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: raise 2021-07-31T02:42:33.855 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:42:33.855 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:42:33.855 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:42:33.856 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:42:33.856 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:42:33.856 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: cluster 2021-07-31T02:42:33.856 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[39718]: 2021-07-31T02:42:33.555427+0000 mon.a (mon.0) 2945 : cluster [DBG] osdmap e1928: 8 total, 8 up, 8 in 2021-07-31T02:42:33.857 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:42:33.857 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:42:33.857 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:42:33.858 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:42:33.858 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:42:33.858 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:42:33.858 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:42:33.858 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:42:33.859 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:42:33.449072+0000 mgr.x (mgr.24542) 2370 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:42:33.859 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:42:33.474935+0000 mgr.x (mgr 2021-07-31T02:42:33.859 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: .24542) 2371 : cephadm [ERR] 2021-07-31T02:42:33.859 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:42:33.860 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:42:33.860 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: yield 2021-07-31T02:42:33.860 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:42:33.860 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:42:33.861 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:42:33.861 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:42:33.861 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:42:33.861 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:42:33.862 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:42:33.862 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:42:33.862 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:42:33.862 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:42:33.863 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:42:33.863 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:42:33.475365+0000 mgr.x (mgr.24542) 2372 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:42:33.863 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:42:33.863 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:42:33.863 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: yield 2021-07-31T02:42:33.864 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:42:33.864 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:42:33.864 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:42:33.864 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:42:33.864 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:42:33.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:42:33.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:42:33.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:42:33.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:42:33.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: task.result() 2021-07-31T02:42:33.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:42:33.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: 2021-07-31T02:42:33.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:42:33.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: 2021-07-31T02:42:33.867 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:42:33.867 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:42:33.867 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:42:33.867 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:42:33.867 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:42:33.868 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:42:33.868 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:42:33.868 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:42:33.868 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:42:33.869 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:42:33.869 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:42:33.869 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:42:33.869 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:42:33.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:42:33.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:42:33.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:42:33.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:42:33.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:42:33.871 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: raise 2021-07-31T02:42:33.871 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:42:33.871 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:42:33.871 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:42:33.872 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:42:33.872 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:42:33.872 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: cluster 2021-07-31T02:42:33.555427+0000 mon.a (mon.0) 2945 : 2021-07-31T02:42:33.872 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:33 smithi045 conmon[33655]: cluster [DBG] osdmap e1928: 8 total, 8 up, 8 in 2021-07-31T02:42:35.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:34 smithi154 conmon[34553]: cephadm 2021-07-31T02:42:33.483902+0000 mgr.x (mgr.24542) 2373 : cephadm [ERR] 2021-07-31T02:42:35.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:34 smithi154 conmon[34553]: 2021-07-31T02:42:35.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:42:35.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:42:35.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:34 smithi154 conmon[34553]: yield 2021-07-31T02:42:35.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:42:35.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:42:35.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:42:35.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:42:35.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:42:35.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:42:35.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:42:35.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:42:35.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:42:35.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:34 smithi154 conmon[34553]: task.result() 2021-07-31T02:42:35.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:42:35.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:34 smithi154 conmon[34553]: cephadm 2021-07-31T02:42:33.484274+0000 mgr.x (mgr.24542) 2374 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:42:35.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:42:35.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:42:35.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:34 smithi154 conmon[34553]: yield 2021-07-31T02:42:35.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:42:35.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:42:35.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:42:35.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:42:35.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:42:35.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:42:35.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:42:35.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:42:35.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:42:35.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:34 smithi154 conmon[34553]: task.result() 2021-07-31T02:42:35.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:42:35.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:34 smithi154 conmon[34553]: 2021-07-31T02:42:35.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:34 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:42:35.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:34 smithi154 conmon[34553]: 2021-07-31T02:42:35.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:42:35.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:42:35.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:34 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:42:35.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:42:35.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:34 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:42:35.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:42:35.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:34 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:42:35.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:42:35.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:34 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:42:35.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:34 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:42:35.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:34 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:42:35.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:34 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:42:35.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:34 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:42:35.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:42:35.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:34 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:42:35.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:42:35.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:34 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:42:35.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:42:35.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:34 smithi154 conmon[34553]: raise 2021-07-31T02:42:35.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:34 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:42:35.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:34 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:42:35.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:42:35.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:34 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:42:35.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:34 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:42:35.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:34 smithi154 conmon[34553]: cluster 2021-07-31T02:42:34.558188+0000 mon.a (mon.0) 2946 : cluster [DBG] osdmap e1929: 8 total, 8 up, 8 in 2021-07-31T02:42:35.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[39718]: cephadm 2021-07-31T02:42:33.483902+0000 mgr.x 2021-07-31T02:42:35.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[39718]: (mgr.24542) 2373 : cephadm [ERR] 2021-07-31T02:42:35.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:42:35.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:42:35.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[39718]: yield 2021-07-31T02:42:35.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:42:35.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:42:35.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:42:35.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:42:35.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:42:35.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:42:35.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:42:35.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:42:35.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:42:35.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[39718]: task.result() 2021-07-31T02:42:35.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:42:35.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[39718]: cephadm 2021-07-31T02:42:33.484274+0000 2021-07-31T02:42:35.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[39718]: mgr.x (mgr.24542) 2374 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:42:35.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:42:35.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:42:35.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[39718]: yield 2021-07-31T02:42:35.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:42:35.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:42:35.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:42:35.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:42:35.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:42:35.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:42:35.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:42:35.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:42:35.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:42:35.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[39718]: task.result() 2021-07-31T02:42:35.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:42:35.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[39718]: 2021-07-31T02:42:35.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:42:35.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[39718]: 2021-07-31T02:42:35.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:42:35.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:42:35.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:42:35.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:42:35.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:42:35.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:42:35.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:42:35.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:42:35.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:42:35.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:42:35.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:42:35.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:42:35.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:42:35.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:42:35.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:42:35.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:42:35.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:42:35.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:42:35.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[39718]: raise 2021-07-31T02:42:35.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:42:35.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:42:35.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:42:35.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:42:35.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:42:35.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[39718]: cluster 2021-07-31T02:42:34.558188+0000 mon.a (mon.0) 2946 : cluster [DBG] osdmap e1929: 8 total, 8 up, 8 in 2021-07-31T02:42:35.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[33655]: cephadm 2021-07-31T02:42:33.483902+0000 mgr.x (mgr. 2021-07-31T02:42:35.040 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[33655]: 24542) 2373 : cephadm [ERR] 2021-07-31T02:42:35.040 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:42:35.040 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:42:35.040 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[33655]: yield 2021-07-31T02:42:35.040 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:42:35.041 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:42:35.041 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:42:35.041 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:42:35.041 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:42:35.042 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:42:35.042 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:42:35.042 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:42:35.042 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:42:35.042 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[33655]: task.result() 2021-07-31T02:42:35.043 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:42:35.043 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[33655]: cephadm 2021-07-31T02:42:33.484274+0000 mgr.x (mgr. 2021-07-31T02:42:35.043 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[33655]: 24542) 2374 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:42:35.043 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:42:35.044 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:42:35.044 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[33655]: yield 2021-07-31T02:42:35.044 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:42:35.044 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:42:35.044 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:42:35.045 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:42:35.045 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:42:35.045 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:42:35.045 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:42:35.046 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:42:35.046 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:42:35.046 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[33655]: task.result() 2021-07-31T02:42:35.046 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:42:35.047 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[33655]: 2021-07-31T02:42:35.047 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:42:35.047 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[33655]: 2021-07-31T02:42:35.047 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:42:35.047 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:42:35.048 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:42:35.048 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:42:35.048 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:42:35.048 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:42:35.049 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:42:35.049 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:42:35.049 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:42:35.049 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:42:35.049 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:42:35.050 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:42:35.050 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:42:35.050 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:42:35.050 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:42:35.051 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:42:35.051 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:42:35.051 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:42:35.051 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[33655]: raise 2021-07-31T02:42:35.051 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:42:35.052 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:42:35.052 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:42:35.052 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:42:35.052 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:42:35.052 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[33655]: cluster 2021-07-31T02:42:34.558188+0000 mon.a ( 2021-07-31T02:42:35.053 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:34 smithi045 conmon[33655]: mon.0) 2946 : cluster [DBG] osdmap e1929: 8 total, 8 up, 8 in 2021-07-31T02:42:36.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:35 smithi154 conmon[34553]: cluster 2021-07-31T02:42:34.594865+0000 mgr.x (mgr.24542) 2375 : cluster [DBG] 2021-07-31T02:42:36.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:35 smithi154 conmon[34553]: pgmap v3436: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.1 MiB/s rd, 5.0 KiB/s wr, 1.47k op/s 2021-07-31T02:42:36.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:35 smithi154 conmon[34553]: cluster 2021-07-31T02:42:35.562619+0000 mon.a (mon.0) 2947 : cluster [DBG] osdmap e1930: 8 total, 8 up, 8 in 2021-07-31T02:42:36.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:35 smithi045 conmon[39718]: cluster 2021-07-31T02:42:34.594865+0000 mgr.x (mgr.24542 2021-07-31T02:42:36.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:35 smithi045 conmon[39718]: ) 2375 : cluster [DBG] pgmap v3436: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.1 MiB/s rd, 5.0 KiB/s wr, 1.47k op/s 2021-07-31T02:42:36.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:35 smithi045 conmon[39718]: cluster 2021-07-31T02:42:35.562619+0000 mon.a (mon.0) 2947 : cluster 2021-07-31T02:42:36.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:35 smithi045 conmon[39718]: [DBG] osdmap e1930: 8 total, 8 up, 8 in 2021-07-31T02:42:36.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:35 smithi045 conmon[33655]: cluster 2021-07-31T02:42:34.594865+0000 mgr.x (mgr. 2021-07-31T02:42:36.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:35 smithi045 conmon[33655]: 24542) 2375 : cluster [DBG] pgmap v3436: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.1 MiB/s rd, 5.0 KiB/s wr, 1.47k op/s 2021-07-31T02:42:36.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:35 smithi045 conmon[33655]: cluster 2021-07-31T02:42:35.562619+0000 mon.a (mon.0) 2947 : cluster [DBG] osdmap e1930: 8 total, 8 up, 8 in 2021-07-31T02:42:36.800 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:42:37.077 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:42:37.077 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 57m ago 58m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:42:37.078 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 55m ago 57m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:42:37.078 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 55m ago 61m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:42:37.078 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 57m ago 63m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:42:37.078 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 57m ago 63m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:42:37.079 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 55m ago 61m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:42:37.079 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 57m ago 61m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:42:37.079 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 57m ago 58m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:42:37.079 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 55m ago 58m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:42:37.080 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 57m ago 61m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:42:37.080 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 57m ago 60m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:42:37.080 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 57m ago 60m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:42:37.080 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 57m ago 60m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:42:37.080 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 55m ago 59m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:42:37.081 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 55m ago 59m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:42:37.081 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 55m ago 59m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:42:37.081 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 55m ago 58m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:42:37.081 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 55m ago 58m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:42:37.413 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:42:37.414 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:42:37.414 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:42:37.414 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:42:37.414 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:42:37.415 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:42:37.415 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:42:37.415 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:42:37.415 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:42:37.416 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:42:37.416 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:42:37.416 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:42:37.416 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:42:37.416 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:42:37.417 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:42:37.417 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:42:37.417 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:42:38.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:37 smithi154 conmon[34553]: cluster 2021-07-31T02:42:36.564068+0000 mon.a (mon.0) 2948 : cluster 2021-07-31T02:42:38.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:37 smithi154 conmon[34553]: [DBG] osdmap e1931: 8 total, 8 up, 8 in 2021-07-31T02:42:38.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:37 smithi154 conmon[34553]: cluster 2021-07-31T02:42:36.595262+0000 mgr.x (mgr.24542) 2376 : cluster [DBG] pgmap v3439: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.1 MiB/s rd, 5.0 KiB/s wr, 1.46k op/s 2021-07-31T02:42:38.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:37 smithi154 conmon[34553]: audit 2021-07-31T02:42: 2021-07-31T02:42:38.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:37 smithi154 conmon[34553]: 36.788614+0000 mgr.x (mgr.24542) 2377 : audit [DBG] from='client.27838 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:42:38.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:37 smithi154 conmon[34553]: audit 2021-07-31T02:42:38.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:37 smithi154 conmon[34553]: 2021-07-31T02 2021-07-31T02:42:38.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:37 smithi154 conmon[34553]: :42:37.069416+0000 mgr.x ( 2021-07-31T02:42:38.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:37 smithi154 conmon[34553]: mgr.24542) 2378 : audit [DBG] 2021-07-31T02:42:38.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:37 smithi154 conmon[34553]: from='client.27844 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:42:38.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:37 smithi154 conmon[34553]: audit 2021- 2021-07-31T02:42:38.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:37 smithi154 conmon[34553]: 07-31T02:42:37.413627 2021-07-31T02:42:38.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:37 smithi154 conmon[34553]: +0000 mon.a (mon.0) 2949 : 2021-07-31T02:42:38.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:37 smithi154 conmon[34553]: audit [DBG] from='client.? 172.21.15.45:0/1208236945' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:42:38.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:37 smithi154 conmon[34553]: cluster 2021- 2021-07-31T02:42:38.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:37 smithi154 conmon[34553]: 07-31T02:42:37.547386+0000 2021-07-31T02:42:38.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:37 smithi154 conmon[34553]: mon.a (mon.0) 2950 : cluster 2021-07-31T02:42:38.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:37 smithi154 conmon[34553]: [DBG] osdmap e1932: 8 total, 8 up, 8 in 2021-07-31T02:42:38.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:37 smithi045 conmon[33655]: cluster 2021-07-31T02:42:36.564068+0000 mon.a (mon.0 2021-07-31T02:42:38.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:37 smithi045 conmon[33655]: ) 2948 : cluster [DBG] osdmap e1931: 8 total, 8 up, 8 in 2021-07-31T02:42:38.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:37 smithi045 conmon[33655]: cluster 2021-07-31T02:42:36.595262+0000 mgr.x (mgr.24542) 2021-07-31T02:42:38.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:37 smithi045 conmon[33655]: 2376 : cluster [DBG] pgmap v3439: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.1 MiB/s rd, 5.0 KiB/s wr, 1.46k op/s 2021-07-31T02:42:38.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:37 smithi045 conmon[33655]: audit 2021- 2021-07-31T02:42:38.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:37 smithi045 conmon[33655]: 07-31T02:42:36.788614+0000 mgr.x (mgr.24542) 2021-07-31T02:42:38.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:37 smithi045 conmon[33655]: 2377 : audit [DBG] from='client.27838 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:42:38.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:37 smithi045 conmon[33655]: audit 2021-07-31T02:42:37.069416+0000 2021-07-31T02:42:38.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:37 smithi045 conmon[33655]: mgr.x (mgr.24542) 2378 : audit 2021-07-31T02:42:38.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:37 smithi045 conmon[33655]: [DBG] from='client.27844 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:42:38.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:37 smithi045 conmon[33655]: audit 2021-07-31T02:42:38.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:37 smithi045 conmon[33655]: 2021-07-31T02:42:37.413627+0000 mon.a (mon.0) 2949 : 2021-07-31T02:42:38.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:37 smithi045 conmon[33655]: audit [DBG] from='client.? 172.21.15.45:0/1208236945' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:42:38.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:37 smithi045 conmon[33655]: cluster 2021-07 2021-07-31T02:42:38.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:37 smithi045 conmon[33655]: -31T02:42:37.547386+0000 mon.a (mon.0) 2950 : cluster [DBG] 2021-07-31T02:42:38.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:37 smithi045 conmon[33655]: osdmap e1932: 8 total, 8 up, 8 in 2021-07-31T02:42:38.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:37 smithi045 conmon[39718]: cluster 2021-07-31T02:42:36.564068+0000 mon.a (mon.0 2021-07-31T02:42:38.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:37 smithi045 conmon[39718]: ) 2948 : cluster [DBG] osdmap e1931: 8 total, 8 up, 8 in 2021-07-31T02:42:38.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:37 smithi045 conmon[39718]: cluster 2021-07-31T02 2021-07-31T02:42:38.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:37 smithi045 conmon[39718]: :42:36.595262+0000 mgr.x (mgr.24542) 2376 : cluster [DBG] pgmap v3439: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.1 MiB/s rd, 5.0 KiB/s wr, 1.46k op/s 2021-07-31T02:42:38.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:37 smithi045 conmon[39718]: audit 2021-07-31T02:42:36.788614 2021-07-31T02:42:38.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:37 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 2377 : audit [DBG] from='client.27838 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:42:38.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:37 smithi045 conmon[39718]: audit 2021-07-31T02:42 2021-07-31T02:42:38.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:37 smithi045 conmon[39718]: :37.069416+0000 mgr.x (mgr.24542) 2378 : audit [DBG] 2021-07-31T02:42:38.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:37 smithi045 conmon[39718]: from='client.27844 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:42:38.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:37 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T02:42:38.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:37 smithi045 conmon[39718]: 31T02:42:37.413627+0000 mon.a (mon.0) 2949 : audit [DBG] from='client.? 172.21.15.45:0/1208236945' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:42:38.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:37 smithi045 conmon[39718]: cluster 2021-07-31T 2021-07-31T02:42:38.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:37 smithi045 conmon[39718]: 02:42:37.547386+0000 mon.a (mon.0) 2950 : cluster 2021-07-31T02:42:38.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:37 smithi045 conmon[39718]: [DBG] osdmap e1932: 8 total, 8 up, 8 in 2021-07-31T02:42:40.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:39 smithi154 conmon[34553]: cluster 2021-07-31T02:42:38.571468+0000 mon.a (mon.0) 2951 2021-07-31T02:42:40.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:39 smithi154 conmon[34553]: : cluster [DBG] osdmap e1933: 8 total, 8 up, 8 in 2021-07-31T02:42:40.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:39 smithi154 conmon[34553]: cluster 2021-07-31T02:42:38.595669+0000 mgr.x (mgr.24542) 2379 : cluster [DBG] pgmap v3442: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:42:40.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:39 smithi045 conmon[39718]: cluster 2021-07-31T02:42:38.571468+0000 mon.a ( 2021-07-31T02:42:40.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:39 smithi045 conmon[39718]: mon.0) 2951 : cluster [DBG] osdmap e1933: 8 total, 8 up, 8 in 2021-07-31T02:42:40.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:39 smithi045 conmon[39718]: cluster 2021- 2021-07-31T02:42:40.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:39 smithi045 conmon[39718]: 07-31T02:42:38.595669+0000 mgr.x (mgr.24542) 2379 : cluster [DBG] pgmap v3442: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:42:40.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:39 smithi045 conmon[39718]: 2021-07-31T02:42:40.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:39 smithi045 conmon[33655]: cluster 2021-07-31T02:42:38.571468+0000 mon.a (mon.0) 2951 : cluster 2021-07-31T02:42:40.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:39 smithi045 conmon[33655]: [DBG] osdmap e1933: 8 total, 8 up, 8 in 2021-07-31T02:42:40.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:39 smithi045 conmon[33655]: cluster 2021-07-31T02:42:38.595669+0000 mgr.x (mgr.24542) 2379 : cluster [DBG] pgmap v3442: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:42:41.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:40 smithi154 conmon[34553]: cluster 2021-07-31T02:42:39.576472+0000 mon.a (mon.0) 2952 : cluster [DBG] osdmap e1934: 8 total, 8 up, 8 in 2021-07-31T02:42:41.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:40 smithi045 conmon[39718]: cluster 2021-07-31T02:42:39.576472+0000 mon.a 2021-07-31T02:42:41.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:40 smithi045 conmon[39718]: (mon.0) 2952 : cluster [DBG] osdmap e1934: 8 total, 8 up, 8 in 2021-07-31T02:42:41.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:40 smithi045 conmon[33655]: cluster 2021-07-31T02:42:39.576472+0000 mon.a (mon.0) 2952 : cluster 2021-07-31T02:42:41.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:40 smithi045 conmon[33655]: [DBG] osdmap e1934: 8 total, 8 up, 8 in 2021-07-31T02:42:42.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:41 smithi154 conmon[34553]: cluster 2021-07-31T02:42:42.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:41 smithi154 conmon[34553]: 2021-07-31T02:42:40 2021-07-31T02:42:42.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:41 smithi154 conmon[34553]: .582234+0000 mon.a (mon.0) 2953 : cluster [DBG] osdmap e1935: 8 total, 8 up, 8 in 2021-07-31T02:42:42.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:41 smithi154 conmon[34553]: cluster 2021-07-31T02:42:40.596149+0000 mgr.x (mgr. 2021-07-31T02:42:42.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:41 smithi154 conmon[34553]: 24542) 2380 : cluster [DBG] pgmap v3445: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.0 MiB/s rd, 5.0 KiB/s wr, 1.42k op/s 2021-07-31T02:42:42.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:41 smithi045 conmon[39718]: cluster 2021-07-31T02:42:40.582234+0000 2021-07-31T02:42:42.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:41 smithi045 conmon[39718]: mon.a (mon.0) 2953 : cluster [DBG] osdmap e1935: 8 total, 8 up, 8 in 2021-07-31T02:42:42.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:41 smithi045 conmon[39718]: cluster 2021-07-31T02:42:40.596149+0000 mgr.x (mgr. 2021-07-31T02:42:42.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:41 smithi045 conmon[39718]: 24542) 2380 : cluster [DBG] pgmap v3445: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.0 MiB/s rd, 5.0 KiB/s wr, 1.42k op/s 2021-07-31T02:42:42.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:41 smithi045 conmon[33655]: cluster 2021-07-31T02:42:40.582234+0000 2021-07-31T02:42:42.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:41 smithi045 conmon[33655]: mon.a (mon.0) 2953 : cluster [DBG] osdmap e1935: 8 total, 8 up, 8 in 2021-07-31T02:42:42.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:41 smithi045 conmon[33655]: cluster 2021-07-31T02:42:40.596149+0000 mgr.x (mgr.24542) 2380 : cluster [DBG] pgmap v3445: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.0 MiB/s rd, 5.0 KiB/s wr, 1.42k op/s 2021-07-31T02:42:43.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:42 smithi154 conmon[34553]: cluster 2021-07-31T02:42:41.588455+0000 mon.a (mon.0) 2954 : cluster [DBG] osdmap e1936: 8 total, 8 up, 8 in 2021-07-31T02:42:43.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:42 smithi154 conmon[34553]: cluster 2021-07-31T 2021-07-31T02:42:43.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:42 smithi154 conmon[34553]: 02:42:42.549677+0000 mon.a (mon.0) 2955 : cluster [DBG] osdmap e1937: 8 total, 8 up, 8 in 2021-07-31T02:42:43.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:42 smithi045 conmon[39718]: cluster 2021-07-31T02:42:41.588455+0000 mon.a (mon. 2021-07-31T02:42:43.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:42 smithi045 conmon[39718]: 0) 2954 : cluster [DBG] osdmap e1936: 8 total, 8 up, 8 in 2021-07-31T02:42:43.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:42 smithi045 conmon[39718]: cluster 2021-07-31T02:42:42.549677+0000 mon.a (mon.0) 2955 : 2021-07-31T02:42:43.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:42 smithi045 conmon[39718]: cluster [DBG] osdmap e1937: 8 total, 8 up, 8 in 2021-07-31T02:42:43.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:42 smithi045 conmon[33655]: cluster 2021-07-31T02:42:41.588455+0000 mon.a (mon.0) 2954 : cluster 2021-07-31T02:42:43.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:42 smithi045 conmon[33655]: [DBG] osdmap e1936: 8 total, 8 up, 8 in 2021-07-31T02:42:43.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:42 smithi045 conmon[33655]: cluster 2021-07-31T02:42:42.549677+0000 mon.a (mon.0) 2955 : cluster [DBG] osdmap e1937: 8 total, 8 up, 8 in 2021-07-31T02:42:44.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:43 smithi154 conmon[34553]: cluster 2021-07-31T02:42:42.596491+0000 mgr.x (mgr.24542) 2381 : cluster [DBG] pgmap v3448: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.0 MiB/s rd, 5.0 KiB/s wr, 1.42k op/s 2021-07-31T02:42:44.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:43 smithi045 conmon[33655]: cluster 2021-07-31T02:42:42.596491+0000 mgr.x (mgr.24542) 2381 2021-07-31T02:42:44.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:43 smithi045 conmon[33655]: : cluster [DBG] pgmap v3448: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.0 MiB/s rd, 5.0 KiB/s wr, 1.42k op/s 2021-07-31T02:42:44.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:43 smithi045 conmon[39718]: cluster 2021-07-31T02:42: 2021-07-31T02:42:44.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:43 smithi045 conmon[39718]: 42.596491+0000 mgr.x (mgr.24542) 2381 : cluster [DBG] pgmap v3448: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.0 MiB/s rd, 5.0 KiB/s wr, 1.42k op/s 2021-07-31T02:42:45.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:44 smithi154 conmon[34553]: cluster 2021-07-31T02:42:43.597084+0000 mon.a (mon.0) 2956 : cluster [DBG] osdmap e1938: 8 total, 8 up, 8 in 2021-07-31T02:42:45.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:44 smithi045 conmon[39718]: cluster 2021-07-31T02:42:43 2021-07-31T02:42:45.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:44 smithi045 conmon[39718]: .597084+0000 mon.a (mon.0) 2956 : cluster [DBG] osdmap e1938: 8 total, 8 up, 8 in 2021-07-31T02:42:45.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:44 smithi045 conmon[33655]: cluster 2021-07-31T02:42:43.597084+0000 mon.a (mon.0) 2021-07-31T02:42:45.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:44 smithi045 conmon[33655]: 2956 : cluster [DBG] osdmap e1938: 8 total, 8 up, 8 in 2021-07-31T02:42:46.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:45 smithi154 conmon[34553]: cluster 2021-07-31T02:42:44.596984+0000 mgr.x (mgr.24542) 2382 : cluster [DBG] pgmap v3450: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1012 KiB/s rd, 5.0 KiB/s wr, 1.37k op/s 2021-07-31T02:42:46.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:45 smithi154 conmon[34553]: cluster 2021-07-31T02:42:44.603803+0000 mon.a (mon.0) 2957 : cluster [DBG] osdmap e1939: 8 total, 8 up, 8 in 2021-07-31T02:42:46.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:45 smithi045 conmon[39718]: cluster 2021-07-31T02:42:46.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:45 smithi045 conmon[39718]: 2021-07-31T02:42:46.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:45 smithi045 conmon[39718]: 2021-07-31T02:42 2021-07-31T02:42:46.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:45 smithi045 conmon[39718]: :44.596984+0000 mgr.x (mgr.24542) 2382 : cluster [DBG] pgmap v3450: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1012 KiB/s rd, 5.0 KiB/s wr, 1.37k op/s 2021-07-31T02:42:46.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:45 smithi045 conmon[39718]: cluster 2021-07-31T02:42:46.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:45 smithi045 conmon[39718]: 2021-07-31 2021-07-31T02:42:46.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:45 smithi045 conmon[39718]: T02:42:44.603803+0000 mon.a (mon. 2021-07-31T02:42:46.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:45 smithi045 conmon[39718]: 0) 2957 : cluster [DBG] osdmap e1939: 8 total, 8 up, 8 in 2021-07-31T02:42:46.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:45 smithi045 conmon[33655]: cluster 2021-07-31T02:42:44.596984+0000 mgr.x ( 2021-07-31T02:42:46.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:45 smithi045 conmon[33655]: mgr.24542) 2382 : cluster [DBG] pgmap v3450: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1012 KiB/s rd, 5.0 KiB/s wr, 1.37k op/s 2021-07-31T02:42:46.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:45 smithi045 conmon[33655]: cluster 2021-07-31T02:42:44.603803+0000 mon.a (mon.0) 2957 : cluster [DBG] osdmap e1939: 8 total, 8 up, 8 in 2021-07-31T02:42:47.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:46 smithi154 conmon[34553]: cluster 2021-07-31T02:42:45.604196+0000 mon.a (mon.0) 2958 : cluster [DBG] 2021-07-31T02:42:47.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:46 smithi154 conmon[34553]: osdmap e1940: 8 total, 8 up, 8 in 2021-07-31T02:42:47.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:46 smithi045 conmon[33655]: cluster 2021-07-31T02:42:45 2021-07-31T02:42:47.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:46 smithi045 conmon[33655]: .604196+0000 mon.a (mon.0) 2958 : cluster [DBG] osdmap e1940: 8 total, 8 up, 8 in 2021-07-31T02:42:47.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:46 smithi045 conmon[39718]: cluster 2021-07-31T02:42:45.604196+0000 mon.a (mon.0 2021-07-31T02:42:47.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:46 smithi045 conmon[39718]: ) 2958 : cluster [DBG] osdmap e1940: 8 total, 8 up, 8 in 2021-07-31T02:42:48.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:47 smithi154 conmon[34553]: cluster 2021-07-31T02:42:46.597359+0000 mgr.x (mgr.24542) 2383 : cluster [DBG] pgmap v3453: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1004 KiB/s rd, 4.9 KiB/s wr, 1.36k op/s 2021-07-31T02:42:48.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:47 smithi154 conmon[34553]: 2021-07-31T02:42:48.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:47 smithi154 conmon[34553]: cluster 2021-07-31T02:42:46.617248+0000 mon.a (mon.0) 2959 : cluster [DBG] osdmap e1941: 8 total, 8 up, 8 in 2021-07-31T02:42:48.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:47 smithi154 conmon[34553]: cluster 2021-07-31T02:42:47.551773+0000 mon.a (mon.0) 2960 : cluster [DBG] osdmap e1942: 8 total, 8 up, 8 in 2021-07-31T02:42:48.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:47 smithi045 conmon[33655]: cluster 2021-07-31T02:42:46.597359+0000 mgr.x (mgr.24542) 2383 : cluster 2021-07-31T02:42:48.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:47 smithi045 conmon[33655]: [DBG] pgmap v3453: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1004 KiB/s rd, 4.9 KiB/s wr, 1.36k op/s 2021-07-31T02:42:48.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:47 smithi045 conmon[33655]: cluster 2021-07-31T02:42:46.617248+0000 mon.a (mon.0) 2959 : cluster [DBG] osdmap e1941: 8 total, 8 up, 8 in 2021-07-31T02:42:48.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:47 smithi045 conmon[33655]: cluster 2021-07-31T02:42: 2021-07-31T02:42:48.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:47 smithi045 conmon[33655]: 47.551773+0000 mon.a (mon.0) 2960 : cluster [DBG] osdmap e1942: 8 total, 8 up, 8 in 2021-07-31T02:42:48.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:47 smithi045 conmon[39718]: cluster 2021-07-31T02:42:46.597359+0000 mgr.x ( 2021-07-31T02:42:48.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:47 smithi045 conmon[39718]: mgr.24542) 2383 : cluster [DBG] pgmap v3453: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1004 KiB/s rd, 4.9 KiB/s wr, 1.36k op/s 2021-07-31T02:42:48.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:47 smithi045 conmon[39718]: cluster 2021-07- 2021-07-31T02:42:48.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:47 smithi045 conmon[39718]: 31T02:42:46.617248+0000 mon.a (mon.0) 2959 : cluster 2021-07-31T02:42:48.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:47 smithi045 conmon[39718]: [DBG] osdmap e1941: 8 total, 8 up, 8 in 2021-07-31T02:42:48.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:47 smithi045 conmon[39718]: cluster 2021-07-31T 2021-07-31T02:42:48.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:47 smithi045 conmon[39718]: 02:42:47.551773+0000 mon.a ( 2021-07-31T02:42:48.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:47 smithi045 conmon[39718]: mon.0) 2960 : cluster [DBG] osdmap e1942: 8 total, 8 up, 8 in 2021-07-31T02:42:50.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:49 smithi154 conmon[34553]: cluster 2021-07-31T02:42:48.559296+0000 mon.a 2021-07-31T02:42:50.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:49 smithi154 conmon[34553]: (mon.0) 2961 : cluster [DBG] osdmap e1943: 8 total, 8 up, 8 in 2021-07-31T02:42:50.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:49 smithi154 conmon[34553]: cluster 2021-07-31T02:42:48.597783+0000 mgr.x (mgr.24542) 2384 : cluster [DBG] pgmap v3457: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:42:50.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:49 smithi045 conmon[33655]: cluster 2021-07-31T02:42:48.559296+0000 mon.a (mon.0) 2961 2021-07-31T02:42:50.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:49 smithi045 conmon[33655]: : cluster [DBG] osdmap e1943: 8 total, 8 up, 8 in 2021-07-31T02:42:50.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:49 smithi045 conmon[33655]: cluster 2021-07-31T02:42:48.597783+0000 mgr.x (mgr.24542) 2384 : cluster [DBG] pgmap v3457: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:42:50.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:49 smithi045 conmon[39718]: cluster 2021-07-31T02:42:48.559296+0000 mon.a (mon.0) 2961 : cluster 2021-07-31T02:42:50.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:49 smithi045 conmon[39718]: [DBG] osdmap e1943: 8 total, 8 up, 8 in 2021-07-31T02:42:50.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:49 smithi045 conmon[39718]: cluster 2021-07-31T02:42:48.597783+0000 mgr.x (mgr.24542) 2384 : cluster [DBG] pgmap v3457: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:42:51.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:50 smithi154 conmon[34553]: cluster 2021-07-31T02:42:49.564883+0000 mon.a (mon.0) 2962 : cluster [DBG] osdmap e1944: 8 total, 8 up, 8 in 2021-07-31T02:42:51.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:50 smithi045 conmon[39718]: cluster 2021-07-31T02:42:49.564883+0000 mon.a (mon.0) 2962 : cluster [DBG] 2021-07-31T02:42:51.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:50 smithi045 conmon[39718]: osdmap e1944: 8 total, 8 up, 8 in 2021-07-31T02:42:51.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:50 smithi045 conmon[33655]: cluster 2021-07-31T02:42:49.564883+0000 mon.a (mon.0) 2021-07-31T02:42:51.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:50 smithi045 conmon[33655]: 2962 : cluster [DBG] osdmap e1944: 8 total, 8 up, 8 in 2021-07-31T02:42:52.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:51 smithi154 conmon[34553]: cluster 2021-07-31T02:42:50.570486+0000 mon.a (mon.0) 2963 : 2021-07-31T02:42:52.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:51 smithi154 conmon[34553]: cluster [DBG] osdmap e1945: 8 total, 8 up, 8 in 2021-07-31T02:42:52.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:51 smithi154 conmon[34553]: cluster 2021-07-31T02:42:50.598182+0000 mgr.x (mgr.24542) 2385 : cluster [DBG] pgmap v3460: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 987 KiB/s rd, 5.0 KiB/s wr, 1.34k op/s 2021-07-31T02:42:52.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:51 smithi045 conmon[39718]: cluster 2021-07-31T02:42:50.570486+0000 mon.a (mon.0 2021-07-31T02:42:52.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:51 smithi045 conmon[39718]: ) 2963 : cluster [DBG] osdmap e1945: 8 total, 8 up, 8 in 2021-07-31T02:42:52.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:51 smithi045 conmon[39718]: cluster 2021-07-31T02:42:50.598182+0000 mgr.x (mgr 2021-07-31T02:42:52.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:51 smithi045 conmon[39718]: .24542) 2385 : cluster [DBG] pgmap v3460: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 987 KiB/s rd, 5.0 KiB/s wr, 1.34k op/s 2021-07-31T02:42:52.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:51 smithi045 conmon[33655]: cluster 2021-07-31T02:42:50.570486+0000 mon.a (mon.0) 2963 : cluster 2021-07-31T02:42:52.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:51 smithi045 conmon[33655]: [DBG] osdmap e1945: 8 total, 8 up, 8 in 2021-07-31T02:42:52.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:51 smithi045 conmon[33655]: cluster 2021-07-31T02:42:50.598182+0000 mgr.x (mgr.24542) 2385 : cluster [DBG] pgmap v3460: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 987 KiB/s rd, 5.0 KiB/s wr, 1.34k op/s 2021-07-31T02:42:53.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:52 smithi154 conmon[34553]: cluster 2021-07-31T02:42:51.576080+0000 mon.a (mon.0) 2964 : 2021-07-31T02:42:53.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:52 smithi154 conmon[34553]: cluster [DBG] osdmap e1946: 8 total, 8 up, 8 in 2021-07-31T02:42:53.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:52 smithi154 conmon[34553]: cluster 2021-07-31T02:42:52.552251+0000 mon.a (mon.0) 2965 : cluster [DBG] osdmap e1947: 8 total, 8 up, 8 in 2021-07-31T02:42:53.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:52 smithi045 conmon[33655]: cluster 2021-07-31T02:42:51.576080 2021-07-31T02:42:53.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:52 smithi045 conmon[33655]: +0000 mon.a (mon.0) 2964 : cluster [DBG] osdmap e1946: 8 total, 8 up, 8 in 2021-07-31T02:42:53.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:52 smithi045 conmon[33655]: cluster 2021-07-31T02:42:52.552251+0000 mon.a (mon.0) 2965 : cluster [DBG] osdmap e1947: 8 total, 8 up, 8 in 2021-07-31T02:42:53.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:52 smithi045 conmon[39718]: cluster 2021-07-31T02:42:51.576080+0000 mon.a (mon.0) 2964 2021-07-31T02:42:53.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:52 smithi045 conmon[39718]: : cluster [DBG] osdmap e1946: 8 total, 8 up, 8 in 2021-07-31T02:42:53.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:52 smithi045 conmon[39718]: cluster 2021-07-31T02:42:52.552251+0000 mon.a (mon.0 2021-07-31T02:42:53.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:52 smithi045 conmon[39718]: ) 2965 : cluster [DBG] osdmap e1947: 8 total, 8 up, 8 in 2021-07-31T02:42:54.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:53 smithi154 conmon[34553]: cluster 2021-07-31T02:42:52.598544+0000 mgr.x (mgr.24542) 2021-07-31T02:42:54.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:53 smithi154 conmon[34553]: 2386 : cluster [DBG] pgmap v3463: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 982 KiB/s rd, 5.0 KiB/s wr, 1.33k op/s 2021-07-31T02:42:54.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:53 smithi045 conmon[39718]: cluster 2021-07-31T02:42:52.598544+0000 mgr.x ( 2021-07-31T02:42:54.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:53 smithi045 conmon[39718]: mgr.24542) 2386 : cluster [DBG] pgmap v3463: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 982 KiB/s rd, 5.0 KiB/s wr, 1.33k op/s 2021-07-31T02:42:54.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:53 smithi045 conmon[33655]: cluster 2021-07-31T02:42:52.598544+0000 mgr.x (mgr.24542) 2021-07-31T02:42:54.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:53 smithi045 conmon[33655]: 2386 : cluster [DBG] pgmap v3463: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 982 KiB/s rd, 5.0 KiB/s wr, 1.33k op/s 2021-07-31T02:42:55.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:54 smithi154 conmon[34553]: cluster 2021-07-31T02:42:53.583382+0000 mon.a (mon.0) 2966 : cluster [DBG] osdmap e1948: 8 total, 8 up, 8 in 2021-07-31T02:42:55.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:54 smithi045 conmon[39718]: cluster 2021-07-31T02:42:53.583382+0000 mon.a (mon 2021-07-31T02:42:55.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:54 smithi045 conmon[39718]: .0) 2966 : cluster [DBG] osdmap e1948: 8 total, 8 up, 8 in 2021-07-31T02:42:55.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:54 smithi045 conmon[33655]: cluster 2021-07-31T02:42:53.583382+0000 mon.a (mon.0) 2966 : 2021-07-31T02:42:55.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:54 smithi045 conmon[33655]: cluster [DBG] osdmap e1948: 8 total, 8 up, 8 in 2021-07-31T02:42:56.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:55 smithi154 conmon[34553]: cluster 2021-07-31T02:42:54.589561+0000 mon.a (mon.0) 2967 : cluster [DBG] osdmap e1949: 8 total, 8 up, 8 in 2021-07-31T02:42:56.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:55 smithi154 conmon[34553]: cluster 2021-07-31T02:42:54.599052+0000 mgr.x (mgr.24542) 2387 : cluster [DBG] pgmap v3466: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 951 KiB/s rd, 5.0 KiB/s wr, 1.29k op/s 2021-07-31T02:42:56.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:55 smithi045 conmon[39718]: cluster 2021-07-31T02:42:54.589561+0000 2021-07-31T02:42:56.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:55 smithi045 conmon[39718]: mon.a (mon.0) 2967 : cluster [DBG] osdmap e1949: 8 total, 8 up, 8 in 2021-07-31T02:42:56.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:55 smithi045 conmon[39718]: cluster 2021-07-31T02:42:54.599052+0000 2021-07-31T02:42:56.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:55 smithi045 conmon[39718]: mgr.x (mgr.24542) 2387 : cluster [DBG] pgmap v3466: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 951 KiB/s rd, 5.0 KiB/s wr, 1.29k op/s 2021-07-31T02:42:56.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:55 smithi045 conmon[33655]: cluster 2021-07-31T02:42:54.589561+0000 mon.a (mon.0) 2967 2021-07-31T02:42:56.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:55 smithi045 conmon[33655]: : cluster [DBG] osdmap e1949: 8 total, 8 up, 8 in 2021-07-31T02:42:56.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:55 smithi045 conmon[33655]: cluster 2021-07-31T02:42:54.599052+0000 mgr.x (mgr.24542) 2387 : cluster [DBG] pgmap v3466: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 951 KiB/s rd, 5.0 KiB/s wr, 1.29k op/s 2021-07-31T02:42:57.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:56 smithi154 conmon[34553]: cluster 2021-07-31T02:42:55.595455+0000 mon.a (mon.0) 2968 : cluster [DBG] osdmap e1950: 8 total, 8 up, 8 in 2021-07-31T02:42:57.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:56 smithi154 conmon[34553]: 2021-07-31T02:42:57.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:56 smithi045 conmon[39718]: cluster 2021-07-31T02:42:55.595455+0000 mon.a ( 2021-07-31T02:42:57.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:56 smithi045 conmon[39718]: mon.0) 2968 : cluster [DBG] osdmap e1950: 8 total, 8 up, 8 in 2021-07-31T02:42:57.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:56 smithi045 conmon[33655]: cluster 2021-07-31T02:42:55.595455+0000 mon.a (mon.0) 2968 : cluster 2021-07-31T02:42:57.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:56 smithi045 conmon[33655]: [DBG] osdmap e1950: 8 total, 8 up, 8 in 2021-07-31T02:42:58.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:57 smithi154 conmon[34553]: cluster 2021-07-31T02:42:56.599449+0000 2021-07-31T02:42:58.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:57 smithi154 conmon[34553]: mgr.x (mgr.24542) 2388 : cluster [DBG] pgmap v3468: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 940 KiB/s rd, 4.9 KiB/s wr, 1.27k op/s 2021-07-31T02:42:58.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:57 smithi154 conmon[34553]: cluster 2021-07-31T02:42:56.601014+0000 mon.a (mon.0) 2969 : cluster [DBG] osdmap e1951: 8 total, 8 up, 8 in 2021-07-31T02:42:58.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:57 smithi154 conmon[34553]: cluster 2021-07-31T02:42:57.554066+0000 mon.a (mon.0) 2970 : cluster [DBG] osdmap e1952: 8 total, 8 up, 8 in 2021-07-31T02:42:58.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:57 smithi045 conmon[39718]: cluster 2021-07-31T02:42:58.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:57 smithi045 conmon[39718]: 2021-07-31T02:42:58.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:57 smithi045 conmon[39718]: 2021 2021-07-31T02:42:58.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:57 smithi045 conmon[39718]: - 2021-07-31T02:42:58.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:57 smithi045 conmon[39718]: 07 2021-07-31T02:42:58.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:57 smithi045 conmon[39718]: - 2021-07-31T02:42:58.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:57 smithi045 conmon[39718]: 31 2021-07-31T02:42:58.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:57 smithi045 conmon[39718]: T 2021-07-31T02:42:58.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:57 smithi045 conmon[39718]: 02 2021-07-31T02:42:58.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:57 smithi045 conmon[39718]: : 2021-07-31T02:42:58.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:57 smithi045 conmon[39718]: 42 2021-07-31T02:42:58.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:57 smithi045 conmon[39718]: : 2021-07-31T02:42:58.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:57 smithi045 conmon[39718]: 56 2021-07-31T02:42:58.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:57 smithi045 conmon[39718]: . 2021-07-31T02:42:58.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:57 smithi045 conmon[39718]: 599449 2021-07-31T02:42:58.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:57 smithi045 conmon[39718]: +0000 2021-07-31T02:42:58.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:57 smithi045 conmon[39718]: 2021-07-31T02:42:58.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:57 smithi045 conmon[39718]: mgr.x (mgr.24542) 2388 : cluster [DBG] pgmap v3468: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 940 KiB/s rd, 4.9 KiB/s wr, 1.27k op/s 2021-07-31T02:42:58.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:57 smithi045 conmon[39718]: cluster 2021-07-31T02:42:58.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:57 smithi045 conmon[39718]: 2021-07-31T02:42: 2021-07-31T02:42:58.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:57 smithi045 conmon[39718]: 56.601014+0000 mon.a (mon.0 2021-07-31T02:42:58.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:57 smithi045 conmon[39718]: ) 2969 : cluster [DBG] osdmap e1951: 8 total, 8 up, 8 in 2021-07-31T02:42:58.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:57 smithi045 conmon[39718]: cluster 2021-07-31T02:42:58.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:57 smithi045 conmon[39718]: 2021-07-31T02:42:57.554066 2021-07-31T02:42:58.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:57 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2970 : cluster 2021-07-31T02:42:58.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:57 smithi045 conmon[39718]: [DBG] osdmap e1952: 8 total, 8 up, 8 in 2021-07-31T02:42:58.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:57 smithi045 conmon[33655]: cluster 2021-07-31T02:42:56.599449+0000 mgr.x (mgr.24542) 2388 : cluster [DBG] pgmap v3468: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 940 KiB/s rd, 4.9 KiB/s wr, 1.27k op/s 2021-07-31T02:42:58.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:57 smithi045 conmon[33655]: 2021-07-31T02:42:58.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:57 smithi045 conmon[33655]: cluster 2021-07-31T02:42:56.601014+0000 mon.a (mon.0) 2969 : cluster [DBG] osdmap e1951: 8 total, 8 up, 8 in 2021-07-31T02:42:58.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:57 smithi045 conmon[33655]: cluster 2021 2021-07-31T02:42:58.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:57 smithi045 conmon[33655]: -07-31T02:42:57.554066+0000 mon.a (mon.0) 2970 : cluster [DBG] osdmap e1952: 8 total, 8 up, 8 in 2021-07-31T02:43:00.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:59 smithi154 conmon[34553]: cluster 2021-07-31T02:42:58.600020+0000 mgr.x (mgr.24542) 2389 : cluster [DBG] pgmap v3471: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:43:00.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:59 smithi154 conmon[34553]: cluster 2021-07-31T02:42:58.613726+0000 2021-07-31T02:43:00.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:42:59 smithi154 conmon[34553]: mon.a (mon.0) 2971 : cluster [DBG] osdmap e1953: 8 total, 8 up, 8 in 2021-07-31T02:43:00.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:59 smithi045 conmon[33655]: cluster 2021-07-31T02:42:58.600020+0000 mgr.x 2021-07-31T02:43:00.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:59 smithi045 conmon[33655]: (mgr.24542) 2389 : cluster [DBG] pgmap v3471: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:43:00.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:59 smithi045 conmon[33655]: cluster 2021-07-31T02:42:58.613726+0000 mon.a (mon. 2021-07-31T02:43:00.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:42:59 smithi045 conmon[33655]: 0) 2971 : cluster [DBG] osdmap e1953: 8 total, 8 up, 8 in 2021-07-31T02:43:00.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:59 smithi045 conmon[39718]: cluster 2021-07-31T02:42:58.600020+0000 mgr.x 2021-07-31T02:43:00.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:59 smithi045 conmon[39718]: (mgr.24542) 2389 : cluster [DBG] pgmap v3471: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:43:00.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:59 smithi045 conmon[39718]: cluster 2021-07- 2021-07-31T02:43:00.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:59 smithi045 conmon[39718]: 31T02:42:58.613726+0000 mon.a (mon. 2021-07-31T02:43:00.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:59 smithi045 conmon[39718]: 0) 2971 : cluster [DBG] osdmap e1953: 8 total, 8 up, 8 in 2021-07-31T02:43:00.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:42:59 smithi045 conmon[39718]: 2021-07-31T02:43:01.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:00 smithi154 conmon[34553]: cluster 2021-07-31T02:42:59.619689+0000 mon.a (mon.0) 2972 : cluster [DBG] osdmap e1954: 8 total, 8 up, 8 in 2021-07-31T02:43:01.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:00 smithi045 conmon[33655]: cluster 2021-07-31T02:42:59.619689+0000 mon.a (mon.0) 2972 : 2021-07-31T02:43:01.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:00 smithi045 conmon[33655]: cluster [DBG] osdmap e1954: 8 total, 8 up, 8 in 2021-07-31T02:43:01.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:00 smithi045 conmon[39718]: cluster 2021-07-31T02:42:59. 2021-07-31T02:43:01.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:00 smithi045 conmon[39718]: 619689+0000 mon.a (mon.0) 2972 : cluster [DBG] osdmap e1954: 8 total, 8 up, 8 in 2021-07-31T02:43:02.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:01 smithi154 conmon[34553]: cluster 2021-07-31T02:43:00.600653+0000 mgr.x (mgr.24542) 2390 : cluster [DBG] pgmap v3474: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 918 KiB/s rd, 5.0 KiB/s wr, 1.24k op/s 2021-07-31T02:43:02.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:01 smithi154 conmon[34553]: 2021-07-31T02:43:02.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:01 smithi154 conmon[34553]: cluster 2021-07-31T02:43:00.626130+0000 mon.a (mon.0) 2973 : cluster [DBG] osdmap e1955: 8 total, 8 up, 8 in 2021-07-31T02:43:02.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:01 smithi045 conmon[39718]: cluster 2021-07-31T02:43: 2021-07-31T02:43:02.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:01 smithi045 conmon[39718]: 00.600653+0000 mgr.x (mgr.24542) 2390 : cluster [DBG] pgmap v3474: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 918 KiB/s rd, 5.0 KiB/s wr, 1.24k op/s 2021-07-31T02:43:02.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:01 smithi045 conmon[39718]: cluster 2021- 2021-07-31T02:43:02.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:01 smithi045 conmon[39718]: 07-31T02:43:00 2021-07-31T02:43:02.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:01 smithi045 conmon[39718]: .626130+0000 mon.a (mon.0 2021-07-31T02:43:02.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:01 smithi045 conmon[39718]: ) 2973 : cluster [DBG] osdmap e1955: 8 total, 8 up, 8 in 2021-07-31T02:43:02.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:01 smithi045 conmon[39718]: 2021-07-31T02:43:02.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:01 smithi045 conmon[33655]: cluster 2021-07-31T02:43:00.600653+0000 mgr.x ( 2021-07-31T02:43:02.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:01 smithi045 conmon[33655]: mgr.24542) 2390 : cluster [DBG] pgmap v3474: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 918 KiB/s rd, 5.0 KiB/s wr, 1.24k op/s 2021-07-31T02:43:02.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:01 smithi045 conmon[33655]: cluster 2021-07-31T02:43:00.626130+0000 mon.a (mon. 2021-07-31T02:43:02.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:01 smithi045 conmon[33655]: 0) 2973 : cluster [DBG] osdmap e1955: 8 total, 8 up, 8 in 2021-07-31T02:43:03.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:02 smithi154 conmon[34553]: cluster 2021-07-31T02:43:01.633177+0000 mon.a (mon.0 2021-07-31T02:43:03.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:02 smithi154 conmon[34553]: ) 2974 : cluster [DBG] osdmap e1956: 8 total, 8 up, 8 in 2021-07-31T02:43:03.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:02 smithi154 conmon[34553]: cluster 2021-07-31T02:43:02.555549+0000 mon.a (mon.0) 2975 : cluster [DBG] osdmap e1957: 8 total, 8 up, 8 in 2021-07-31T02:43:03.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:02 smithi045 conmon[39718]: cluster 2021-07-31T02:43:01 2021-07-31T02:43:03.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:02 smithi045 conmon[39718]: .633177+0000 mon.a (mon.0) 2974 : cluster [DBG] osdmap e1956: 8 total, 8 up, 8 in 2021-07-31T02:43:03.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:02 smithi045 conmon[39718]: cluster 2021-07-31T02:43:02.555549+0000 2021-07-31T02:43:03.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:02 smithi045 conmon[39718]: mon.a (mon.0) 2975 : cluster [DBG] 2021-07-31T02:43:03.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:02 smithi045 conmon[39718]: osdmap e1957: 8 total, 8 up, 8 in 2021-07-31T02:43:03.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:02 smithi045 conmon[33655]: cluster 2021-07-31T02:43:01.633177+0000 mon.a (mon.0) 2974 : cluster 2021-07-31T02:43:03.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:02 smithi045 conmon[33655]: [DBG] osdmap e1956: 8 total, 8 up, 8 in 2021-07-31T02:43:03.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:02 smithi045 conmon[33655]: cluster 2021-07-31T02:43:02.555549+0000 mon.a (mon.0) 2975 : cluster [DBG] osdmap e1957: 8 total, 8 up, 8 in 2021-07-31T02:43:04.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:03 smithi154 conmon[34553]: cluster 2021-07-31T02:43:02.601060+0000 mgr.x (mgr.24542) 2391 : cluster [DBG] pgmap v3478: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 920 KiB/s rd, 5.0 KiB/s wr, 1.25k op/s 2021-07-31T02:43:04.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:03 smithi154 conmon[34553]: cluster 2021-07-31T02:43:03.566138+0000 mon.a (mon.0) 2976 : cluster [DBG] osdmap e1958: 8 total, 8 up, 8 in 2021-07-31T02:43:04.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:03 smithi045 conmon[33655]: cluster 2021-07-31T02:43:02.601060+0000 mgr.x (mgr.24542 2021-07-31T02:43:04.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:03 smithi045 conmon[33655]: ) 2391 : cluster [DBG] pgmap v3478: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 920 KiB/s rd, 5.0 KiB/s wr, 1.25k op/s 2021-07-31T02:43:04.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:03 smithi045 conmon[33655]: cluster 2021-07-31T02:43:03.566138+0000 mon.a (mon.0) 2976 : cluster [DBG] osdmap e1958: 8 total, 8 up, 8 in 2021-07-31T02:43:04.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:03 smithi045 conmon[39718]: cluster 2021-07-31T02:43:02.601060 2021-07-31T02:43:04.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:03 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 2391 : cluster [DBG] pgmap v3478: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 920 KiB/s rd, 5.0 KiB/s wr, 1.25k op/s 2021-07-31T02:43:04.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:03 smithi045 conmon[39718]: cluster 2021-07-31T02:43:03.566138+0000 mon.a 2021-07-31T02:43:04.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:03 smithi045 conmon[39718]: (mon.0) 2976 : cluster [DBG] osdmap e1958: 8 total, 8 up, 8 in 2021-07-31T02:43:06.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:05 smithi154 conmon[34553]: cluster 2021-07-31T02:43:04.573570+0000 mon.a (mon.0) 2977 : cluster [DBG] osdmap e1959: 8 total, 8 up, 8 in 2021-07-31T02:43:06.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:05 smithi154 conmon[34553]: cluster 2021-07-31T02:43:04.601469+0000 mgr.x (mgr.24542) 2392 : cluster [DBG] pgmap v3481: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 892 KiB/s rd, 5.0 KiB/s wr, 1.21k op/s 2021-07-31T02:43:06.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:05 smithi045 conmon[39718]: cluster 2021-07-31T02:43:04.573570+0000 2021-07-31T02:43:06.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:05 smithi045 conmon[39718]: mon.a (mon.0) 2977 : cluster [DBG] osdmap e1959: 8 total, 8 up, 8 in 2021-07-31T02:43:06.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:05 smithi045 conmon[39718]: cluster 2021-07-31T02:43:06.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:05 smithi045 conmon[39718]: 2021-07-31T02:43:04.601469+0000 2021-07-31T02:43:06.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:05 smithi045 conmon[39718]: mgr.x (mgr.24542) 2392 : cluster [DBG] pgmap v3481: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 892 KiB/s rd, 5.0 KiB/s wr, 1.21k op/s 2021-07-31T02:43:06.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:05 smithi045 conmon[33655]: cluster 2021-07-31T02:43:04.573570+0000 mon.a (mon.0) 2977 2021-07-31T02:43:06.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:05 smithi045 conmon[33655]: : cluster [DBG] osdmap e1959: 8 total, 8 up, 8 in 2021-07-31T02:43:06.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:05 smithi045 conmon[33655]: cluster 2021-07-31T02:43:04.601469+0000 mgr.x (mgr.24542) 2392 : cluster [DBG] pgmap v3481: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 892 KiB/s rd, 5.0 KiB/s wr, 1.21k op/s 2021-07-31T02:43:07.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:06 smithi154 conmon[34553]: cluster 2021-07-31T02:43:05.578900+0000 mon.a (mon.0) 2978 : cluster [DBG] osdmap e1960: 8 total, 8 up, 8 in 2021-07-31T02:43:07.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:06 smithi045 conmon[39718]: cluster 2021-07-31T02:43:05.578900 2021-07-31T02:43:07.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:06 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2978 : cluster [DBG] osdmap e1960: 8 total, 8 up, 8 in 2021-07-31T02:43:07.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:06 smithi045 conmon[33655]: cluster 2021-07-31T02:43:05.578900+0000 mon.a (mon.0) 2978 : cluster 2021-07-31T02:43:07.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:06 smithi045 conmon[33655]: [DBG] osdmap e1960: 8 total, 8 up, 8 in 2021-07-31T02:43:07.742 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:43:08.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:07 smithi154 conmon[34553]: cluster 2021-07-31T02:43:06.584565+0000 mon.a (mon.0) 2979 : cluster [DBG] 2021-07-31T02:43:08.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:07 smithi154 conmon[34553]: osdmap e1961: 8 total, 8 up, 8 in 2021-07-31T02:43:08.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:07 smithi154 conmon[34553]: cluster 2021-07-31T02:43:06.601892+0000 mgr.x (mgr.24542) 2393 : cluster [DBG] pgmap v3484: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 886 KiB/s rd, 5.0 KiB/s wr, 1.20k op/s 2021-07-31T02:43:08.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:07 smithi154 conmon[34553]: cluster 2021-07-31T02:43:07.556754+0000 mon.a (mon.0) 2980 : 2021-07-31T02:43:08.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:07 smithi154 conmon[34553]: cluster [DBG] osdmap e1962: 8 total, 8 up, 8 in 2021-07-31T02:43:08.020 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:43:08.020 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 58m ago 58m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:43:08.020 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 55m ago 58m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:43:08.021 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 55m ago 62m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:43:08.021 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 58m ago 64m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:43:08.021 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 58m ago 64m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:43:08.021 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 55m ago 62m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:43:08.022 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 58m ago 62m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:43:08.022 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 58m ago 59m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:43:08.022 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 55m ago 58m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:43:08.022 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 58m ago 61m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:43:08.023 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 58m ago 61m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:43:08.023 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 58m ago 61m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:43:08.023 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 58m ago 60m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:43:08.024 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 55m ago 60m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:43:08.024 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 55m ago 60m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:43:08.024 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 55m ago 59m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:43:08.024 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 55m ago 59m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:43:08.025 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 55m ago 59m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:43:08.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:07 smithi045 conmon[33655]: cluster 2021-07-31T02:43:06.584565 2021-07-31T02:43:08.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:07 smithi045 conmon[33655]: +0000 mon.a (mon.0) 2979 : cluster [DBG] osdmap e1961: 8 total, 8 up, 8 in 2021-07-31T02:43:08.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:07 smithi045 conmon[33655]: cluster 2021-07-31T02:43:06.601892 2021-07-31T02:43:08.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:07 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 2393 : cluster [DBG] pgmap v3484: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 886 KiB/s rd, 5.0 KiB/s wr, 1.20k op/s 2021-07-31T02:43:08.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:07 smithi045 conmon[33655]: cluster 2021- 2021-07-31T02:43:08.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:07 smithi045 conmon[33655]: 07-31T02:43:07.556754+0000 2021-07-31T02:43:08.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:07 smithi045 conmon[33655]: mon.a (mon.0) 2980 : cluster [DBG] osdmap e1962: 8 total, 8 up, 8 in 2021-07-31T02:43:08.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:07 smithi045 conmon[39718]: cluster 2021-07-31T02:43:08.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:07 smithi045 conmon[39718]: 2021-07-31T02:43:08.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:07 smithi045 conmon[39718]: 2021 2021-07-31T02:43:08.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:07 smithi045 conmon[39718]: - 2021-07-31T02:43:08.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:07 smithi045 conmon[39718]: 07 2021-07-31T02:43:08.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:07 smithi045 conmon[39718]: - 2021-07-31T02:43:08.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:07 smithi045 conmon[39718]: 31 2021-07-31T02:43:08.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:07 smithi045 conmon[39718]: T 2021-07-31T02:43:08.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:07 smithi045 conmon[39718]: 02 2021-07-31T02:43:08.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:07 smithi045 conmon[39718]: : 2021-07-31T02:43:08.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:07 smithi045 conmon[39718]: 43 2021-07-31T02:43:08.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:07 smithi045 conmon[39718]: : 2021-07-31T02:43:08.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:07 smithi045 conmon[39718]: 06 2021-07-31T02:43:08.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:07 smithi045 conmon[39718]: . 2021-07-31T02:43:08.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:07 smithi045 conmon[39718]: 584565 2021-07-31T02:43:08.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:07 smithi045 conmon[39718]: +0000 2021-07-31T02:43:08.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:07 smithi045 conmon[39718]: 2021-07-31T02:43:08.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:07 smithi045 conmon[39718]: mon.a 2021-07-31T02:43:08.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:07 smithi045 conmon[39718]: ( 2021-07-31T02:43:08.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:07 smithi045 conmon[39718]: mon 2021-07-31T02:43:08.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:07 smithi045 conmon[39718]: . 2021-07-31T02:43:08.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:07 smithi045 conmon[39718]: 0 2021-07-31T02:43:08.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:07 smithi045 conmon[39718]: ) 2021-07-31T02:43:08.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:07 smithi045 conmon[39718]: 2979 2021-07-31T02:43:08.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:07 smithi045 conmon[39718]: : 2021-07-31T02:43:08.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:07 smithi045 conmon[39718]: cluster 2021-07-31T02:43:08.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:07 smithi045 conmon[39718]: 2021-07-31T02:43:08.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:07 smithi045 conmon[39718]: [DBG] 2021-07-31T02:43:08.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:07 smithi045 conmon[39718]: 2021-07-31T02:43:08.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:07 smithi045 conmon[39718]: osdmap e1961: 8 total, 8 up, 8 in 2021-07-31T02:43:08.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:07 smithi045 conmon[39718]: 2021-07-31T02:43:08.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:07 smithi045 conmon[39718]: cluster 2021-07-31T02:43:08.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:07 smithi045 conmon[39718]: 2021-07-31T02:43:08.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:07 smithi045 conmon[39718]: 2021 2021-07-31T02:43:08.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:07 smithi045 conmon[39718]: - 2021-07-31T02:43:08.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:07 smithi045 conmon[39718]: 07 2021-07-31T02:43:08.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:07 smithi045 conmon[39718]: - 2021-07-31T02:43:08.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:07 smithi045 conmon[39718]: 31 2021-07-31T02:43:08.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:07 smithi045 conmon[39718]: T 2021-07-31T02:43:08.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:07 smithi045 conmon[39718]: 02 2021-07-31T02:43:08.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:07 smithi045 conmon[39718]: : 2021-07-31T02:43:08.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:07 smithi045 conmon[39718]: 43 2021-07-31T02:43:08.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:07 smithi045 conmon[39718]: : 2021-07-31T02:43:08.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:07 smithi045 conmon[39718]: 06 2021-07-31T02:43:08.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:07 smithi045 conmon[39718]: . 2021-07-31T02:43:08.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:07 smithi045 conmon[39718]: 601892 2021-07-31T02:43:08.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:07 smithi045 conmon[39718]: +0000 2021-07-31T02:43:08.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:07 smithi045 conmon[39718]: 2021-07-31T02:43:08.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:07 smithi045 conmon[39718]: mgr.x 2021-07-31T02:43:08.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:07 smithi045 conmon[39718]: ( 2021-07-31T02:43:08.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:07 smithi045 conmon[39718]: mgr 2021-07-31T02:43:08.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:07 smithi045 conmon[39718]: . 2021-07-31T02:43:08.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:07 smithi045 conmon[39718]: 24542 2021-07-31T02:43:08.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:07 smithi045 conmon[39718]: ) 2021-07-31T02:43:08.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:07 smithi045 conmon[39718]: 2393 2021-07-31T02:43:08.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:07 smithi045 conmon[39718]: : 2021-07-31T02:43:08.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:07 smithi045 conmon[39718]: cluster 2021-07-31T02:43:08.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:07 smithi045 conmon[39718]: [DBG] pgmap v3484: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 886 KiB/s rd, 5.0 KiB/s wr, 1.20k op/s 2021-07-31T02:43:08.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:07 smithi045 conmon[39718]: cluster 2021-07- 2021-07-31T02:43:08.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:07 smithi045 conmon[39718]: 31T02:43:07.556754+0000 2021-07-31T02:43:08.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:07 smithi045 conmon[39718]: mon.a (mon.0) 2980 2021-07-31T02:43:08.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:07 smithi045 conmon[39718]: : cluster [DBG] osdmap e1962: 8 total, 8 up, 8 in 2021-07-31T02:43:08.355 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:43:08.356 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:43:08.356 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:43:08.356 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:43:08.356 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:43:08.357 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:43:08.357 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:43:08.357 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:43:08.357 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:43:08.358 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:43:08.358 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:43:08.358 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:43:08.358 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:43:08.358 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:43:08.359 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:43:08.359 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:43:08.359 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:43:09.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:08 smithi154 conmon[34553]: audit 2021-07-31T02:43:07.730142+0000 mgr.x ( 2021-07-31T02:43:09.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:08 smithi154 conmon[34553]: mgr.24542) 2394 : audit [DBG] from='client.17916 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:43:09.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:08 smithi154 conmon[34553]: audit 2021-07-31T02:43:08.012033+0000 mgr.x (mgr.24542) 2395 : audit [DBG] from='client.17922 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:43:09.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:08 smithi154 conmon[34553]: audit 2021-07-31T02:43:08.355476+0000 mon.c (mon.1) 129 : audit [DBG] from='client.? 172.21.15.45:0/1529055068' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:43:09.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:08 smithi045 conmon[39718]: audit 2021-07-31T02:43:07.730142+0000 2021-07-31T02:43:09.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:08 smithi045 conmon[39718]: mgr.x (mgr.24542) 2394 : audit [DBG] from='client.17916 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:43:09.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:08 smithi045 conmon[39718]: audit 2021-07-31T02:43:08.012033+0000 2021-07-31T02:43:09.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:08 smithi045 conmon[39718]: mgr.x (mgr.24542) 2395 : audit [DBG] from='client.17922 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:43:09.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:08 smithi045 conmon[39718]: audit 2021- 2021-07-31T02:43:09.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:08 smithi045 conmon[39718]: 07-31T02:43:08.355476+0000 mon.c (mon. 2021-07-31T02:43:09.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:08 smithi045 conmon[39718]: 1) 129 : audit [DBG] from='client.? 172.21.15.45:0/1529055068' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:43:09.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:08 smithi045 conmon[33655]: audit 2021-07-31T02:43:07.730142+0000 mgr.x (mgr.24542 2021-07-31T02:43:09.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:08 smithi045 conmon[33655]: ) 2394 : audit [DBG] from='client.17916 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:43:09.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:08 smithi045 conmon[33655]: audit 2021-07-31T02:43:08.012033+0000 mgr.x (mgr.24542) 2395 : audit [DBG] from='client.17922 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:43:09.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:08 smithi045 conmon[33655]: audit 2021-07-31T02 2021-07-31T02:43:09.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:08 smithi045 conmon[33655]: :43:08.355476+0000 mon.c (mon.1) 129 : audit [DBG] from='client.? 172.21.15.45:0/1529055068' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:43:10.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:09 smithi154 conmon[34553]: cluster 2021-07-31T02:43:08.591097+0000 mon.a (mon.0) 2981 : cluster [DBG] osdmap e1963: 8 total, 8 up, 8 in 2021-07-31T02:43:10.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:09 smithi154 conmon[34553]: cluster 2021-07-31T02:43:08.602527+0000 mgr.x (mgr.24542) 2396 : cluster [DBG] pgmap v3487: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:43:10.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:09 smithi045 conmon[39718]: cluster 2021-07-31T02:43:08.591097+0000 mon.a (mon.0 2021-07-31T02:43:10.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:09 smithi045 conmon[39718]: ) 2981 : cluster [DBG] osdmap e1963: 8 total, 8 up, 8 in 2021-07-31T02:43:10.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:09 smithi045 conmon[39718]: cluster 2021-07-31T02:43:08.602527+0000 mgr.x ( 2021-07-31T02:43:10.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:09 smithi045 conmon[39718]: mgr.24542) 2396 : cluster [DBG] pgmap v3487: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:43:10.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:09 smithi045 conmon[33655]: cluster 2021-07-31T02:43:08.591097+0000 mon.a (mon.0) 2981 : cluster 2021-07-31T02:43:10.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:09 smithi045 conmon[33655]: [DBG] osdmap e1963: 8 total, 8 up, 8 in 2021-07-31T02:43:10.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:09 smithi045 conmon[33655]: cluster 2021-07-31T02:43:08.602527+0000 mgr.x (mgr.24542) 2396 : 2021-07-31T02:43:10.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:09 smithi045 conmon[33655]: cluster [DBG] pgmap v3487: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:43:11.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:10 smithi154 conmon[34553]: cluster 2021-07-31T02:43:09.602590+0000 mon.a (mon.0) 2982 : cluster 2021-07-31T02:43:11.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:10 smithi154 conmon[34553]: [DBG] osdmap e1964: 8 total, 8 up, 8 in 2021-07-31T02:43:11.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:10 smithi045 conmon[33655]: cluster 2021-07-31T02: 2021-07-31T02:43:11.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:10 smithi045 conmon[33655]: 43:09.602590+0000 mon.a (mon.0) 2982 : cluster [DBG] osdmap e1964: 8 total, 8 up, 8 in 2021-07-31T02:43:11.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:10 smithi045 conmon[39718]: cluster 2021-07-31T02:43:09.602590+0000 mon.a 2021-07-31T02:43:11.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:10 smithi045 conmon[39718]: (mon.0) 2982 : cluster [DBG] osdmap e1964: 8 total, 8 up, 8 in 2021-07-31T02:43:12.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:11 smithi154 conmon[34553]: cluster 2021-07-31T02:43:10.603107+0000 mgr.x (mgr.24542) 2397 : cluster [DBG] pgmap v3489: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 847 KiB/s rd, 5.0 KiB/s wr, 1.15k op/s 2021-07-31T02:43:12.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:11 smithi154 conmon[34553]: cluster 2021-07-31T02:43:10.604519+0000 mon.a (mon.0) 2983 : cluster [DBG] osdmap e1965: 8 total, 8 up, 8 in 2021-07-31T02:43:12.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:11 smithi045 conmon[33655]: cluster 2021-07-31T02:43:12.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:11 smithi045 conmon[33655]: 2021-07-31T02:43:10.603107+0000 mgr.x (mgr.24542) 2397 : cluster [DBG] pgmap v3489: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 847 KiB/s rd, 5.0 KiB/s wr, 1.15k op/s 2021-07-31T02:43:12.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:11 smithi045 conmon[33655]: cluster 2021-07- 2021-07-31T02:43:12.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:11 smithi045 conmon[33655]: 31T02:43:10.604519+0000 mon.a (mon.0) 2983 : cluster [DBG] 2021-07-31T02:43:12.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:11 smithi045 conmon[33655]: osdmap e1965: 8 total, 8 up, 8 in 2021-07-31T02:43:12.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:11 smithi045 conmon[39718]: cluster 2021-07-31T02:43:10.603107+0000 mgr.x (mgr.24542) 2397 : cluster [DBG] pgmap v3489: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 847 KiB/s rd, 5.0 KiB/s wr, 1.15k op/s 2021-07-31T02:43:12.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:11 smithi045 conmon[39718]: cluster 2021-07-31T02:43:12.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:11 smithi045 conmon[39718]: 2021-07-31T02:43:10.604519+0000 mon.a (mon.0) 2983 : cluster [DBG] osdmap e1965: 8 total, 8 up, 8 in 2021-07-31T02:43:13.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:12 smithi154 conmon[34553]: cluster 2021-07-31T02:43:11.615935+0000 mon.a (mon.0) 2984 : 2021-07-31T02:43:13.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:12 smithi154 conmon[34553]: cluster [DBG] osdmap e1966: 8 total, 8 up, 8 in 2021-07-31T02:43:13.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:12 smithi154 conmon[34553]: cluster 2021-07-31T02:43:12.558358+0000 mon.a (mon.0) 2985 : cluster [DBG] osdmap e1967: 8 total, 8 up, 8 in 2021-07-31T02:43:13.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:12 smithi045 conmon[33655]: cluster 2021-07-31T02:43:11.615935+0000 mon.a (mon.0) 2984 : cluster 2021-07-31T02:43:13.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:12 smithi045 conmon[33655]: [DBG] osdmap e1966: 8 total, 8 up, 8 in 2021-07-31T02:43:13.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:12 smithi045 conmon[33655]: cluster 2021-07-31T02:43:12.558358+0000 mon.a (mon.0) 2985 : cluster [DBG] osdmap e1967: 8 total, 8 up, 8 in 2021-07-31T02:43:13.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:12 smithi045 conmon[39718]: cluster 2021-07-31T02:43:11.615935+0000 2021-07-31T02:43:13.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:12 smithi045 conmon[39718]: mon.a (mon.0) 2984 : cluster [DBG] osdmap e1966: 8 total, 8 up, 8 in 2021-07-31T02:43:13.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:12 smithi045 conmon[39718]: cluster 2021-07-31T02:43:13.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:12 smithi045 conmon[39718]: 2021-07-31T02:43:12.558358+0000 mon.a (mon.0) 2021-07-31T02:43:13.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:12 smithi045 conmon[39718]: 2985 : cluster [DBG] osdmap e1967: 8 total, 8 up, 8 in 2021-07-31T02:43:14.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:13 smithi154 conmon[34553]: cluster 2021-07-31T02:43:12.603597+0000 mgr.x (mgr.24542) 2398 : cluster [DBG] pgmap v3493: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 851 KiB/s rd, 5.0 KiB/s wr, 1.15k op/s 2021-07-31T02:43:14.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:13 smithi045 conmon[33655]: cluster 2021-07-31T02:43:12.603597+0000 mgr.x (mgr.24542) 2398 : cluster 2021-07-31T02:43:14.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:13 smithi045 conmon[33655]: [DBG] pgmap v3493: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 851 KiB/s rd, 5.0 KiB/s wr, 1.15k op/s 2021-07-31T02:43:14.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:13 smithi045 conmon[39718]: cluster 2021-07-31T02:43: 2021-07-31T02:43:14.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:13 smithi045 conmon[39718]: 12.603597+0000 mgr.x (mgr.24542) 2398 : cluster [DBG] pgmap v3493: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 851 KiB/s rd, 5.0 KiB/s wr, 1.15k op/s 2021-07-31T02:43:15.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:14 smithi154 conmon[34553]: cluster 2021-07-31T02:43:13.625197+0000 mon.a (mon 2021-07-31T02:43:15.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:14 smithi154 conmon[34553]: .0) 2986 : cluster [DBG] osdmap e1968: 8 total, 8 up, 8 in 2021-07-31T02:43:15.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:14 smithi045 conmon[39718]: cluster 2021-07-31T02:43:13.625197+0000 mon.a ( 2021-07-31T02:43:15.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:14 smithi045 conmon[39718]: mon.0) 2986 : cluster [DBG] osdmap e1968: 8 total, 8 up, 8 in 2021-07-31T02:43:15.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:14 smithi045 conmon[33655]: cluster 2021-07-31T02:43:13.625197+0000 mon.a (mon.0) 2986 : cluster 2021-07-31T02:43:15.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:14 smithi045 conmon[33655]: [DBG] osdmap e1968: 8 total, 8 up, 8 in 2021-07-31T02:43:16.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:15 smithi154 conmon[34553]: cluster 2021-07-31T02:43:14.604320+0000 mgr.x (mgr.24542) 2021-07-31T02:43:16.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:15 smithi154 conmon[34553]: 2399 : cluster [DBG] pgmap v3495: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 820 KiB/s rd, 5.0 KiB/s wr, 1.11k op/s 2021-07-31T02:43:16.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:15 smithi154 conmon[34553]: cluster 2021-07-31T02:43:14.630188+0000 mon.a (mon.0) 2987 : cluster [DBG] osdmap e1969: 8 total, 8 up, 8 in 2021-07-31T02:43:16.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:15 smithi045 conmon[39718]: cluster 2021-07-31T02:43: 2021-07-31T02:43:16.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:15 smithi045 conmon[39718]: 14.604320+0000 mgr.x (mgr.24542) 2399 : cluster [DBG] pgmap v3495: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 820 KiB/s rd, 5.0 KiB/s wr, 1.11k op/s 2021-07-31T02:43:16.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:15 smithi045 conmon[39718]: cluster 2021-07-31T02:43:14.630188+0000 mon.a 2021-07-31T02:43:16.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:15 smithi045 conmon[39718]: (mon.0) 2987 : cluster [DBG] osdmap e1969: 8 total, 8 up, 8 in 2021-07-31T02:43:16.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:15 smithi045 conmon[33655]: cluster 2021-07-31T02:43:14.604320+0000 mgr.x (mgr.24542) 2399 : cluster 2021-07-31T02:43:16.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:15 smithi045 conmon[33655]: [DBG] pgmap v3495: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 820 KiB/s rd, 5.0 KiB/s wr, 1.11k op/s 2021-07-31T02:43:16.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:15 smithi045 conmon[33655]: cluster 2021-07-31T02:43:14.630188+0000 mon.a (mon.0) 2987 : cluster [DBG] osdmap e1969: 8 total, 8 up, 8 in 2021-07-31T02:43:16.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:15 smithi045 conmon[33655]: 2021-07-31T02:43:17.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:16 smithi154 conmon[34553]: cluster 2021-07-31T02:43:15.632315+0000 mon.a (mon.0) 2988 : cluster [DBG] osdmap e1970: 8 total, 8 up, 8 in 2021-07-31T02:43:17.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:16 smithi154 conmon[34553]: 2021-07-31T02:43:17.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:16 smithi045 conmon[39718]: cluster 2021-07-31T02:43: 2021-07-31T02:43:17.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:16 smithi045 conmon[39718]: 15.632315+0000 mon.a (mon.0) 2988 : cluster [DBG] osdmap e1970: 8 total, 8 up, 8 in 2021-07-31T02:43:17.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:16 smithi045 conmon[33655]: cluster 2021-07-31T02:43:15.632315 2021-07-31T02:43:17.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:16 smithi045 conmon[33655]: +0000 mon.a (mon.0) 2988 : cluster [DBG] osdmap e1970: 8 total, 8 up, 8 in 2021-07-31T02:43:18.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:17 smithi154 conmon[34553]: cluster 2021-07-31T02:43:16.604799+0000 mgr.x (mgr.24542) 2400 : cluster [DBG] pgmap v3498: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 810 KiB/s rd, 4.9 KiB/s wr, 1.09k op/s 2021-07-31T02:43:18.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:17 smithi154 conmon[34553]: cluster 2021-07-31T02:43:16.643850+0000 mon.a (mon.0) 2989 : cluster [DBG] osdmap e1971: 8 total, 8 up, 8 in 2021-07-31T02:43:18.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:17 smithi154 conmon[34553]: cluster 2021-07-31T02:43:17.559990+0000 mon.a (mon.0) 2990 : cluster [DBG] 2021-07-31T02:43:18.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:17 smithi154 conmon[34553]: osdmap e1972: 8 total, 8 up, 8 in 2021-07-31T02:43:18.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:17 smithi045 conmon[39718]: cluster 2021-07-31T02:43:16.604799+0000 mgr.x 2021-07-31T02:43:18.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:17 smithi045 conmon[39718]: (mgr.24542) 2400 : cluster [DBG] pgmap v3498: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 810 KiB/s rd, 4.9 KiB/s wr, 1.09k op/s 2021-07-31T02:43:18.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:17 smithi045 conmon[39718]: cluster 2021-07-31T02:43:16.643850+0000 mon.a 2021-07-31T02:43:18.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:17 smithi045 conmon[39718]: (mon.0) 2989 : cluster [DBG] osdmap e1971: 8 total, 8 up, 8 in 2021-07-31T02:43:18.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:17 smithi045 conmon[39718]: cluster 2021-07-31T02:43: 2021-07-31T02:43:18.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:17 smithi045 conmon[39718]: 17.559990+0000 mon.a (mon.0) 2990 : cluster [DBG] osdmap e1972: 8 total, 8 up, 8 in 2021-07-31T02:43:18.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:17 smithi045 conmon[33655]: cluster 2021-07-31T02:43:16. 2021-07-31T02:43:18.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:17 smithi045 conmon[33655]: 604799+0000 mgr.x (mgr.24542) 2400 : cluster [DBG] pgmap v3498: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 810 KiB/s rd, 4.9 KiB/s wr, 1.09k op/s 2021-07-31T02:43:18.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:17 smithi045 conmon[33655]: cluster 2021-07-31T02: 2021-07-31T02:43:18.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:17 smithi045 conmon[33655]: 43:16.643850+0000 mon.a (mon.0) 2989 : 2021-07-31T02:43:18.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:17 smithi045 conmon[33655]: cluster [DBG] osdmap e1971: 8 total, 8 up, 8 in 2021-07-31T02:43:18.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:17 smithi045 conmon[33655]: cluster 2021- 2021-07-31T02:43:18.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:17 smithi045 conmon[33655]: 07-31T02:43:17.559990+0000 mon.a (mon.0) 2990 : cluster [DBG] osdmap e1972: 8 total, 8 up, 8 in 2021-07-31T02:43:20.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:19 smithi154 conmon[34553]: cluster 2021-07-31T02:43:18.569219+0000 mon.a (mon.0) 2991 : 2021-07-31T02:43:20.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:19 smithi154 conmon[34553]: cluster [DBG] osdmap e1973: 8 total, 8 up, 8 in 2021-07-31T02:43:20.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:19 smithi154 conmon[34553]: cluster 2021-07-31T02:43:18.605158+0000 mgr.x (mgr.24542) 2401 : cluster [DBG] pgmap v3502: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:43:20.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:19 smithi045 conmon[39718]: cluster 2021-07-31T02:43:18.569219+0000 mon.a (mon.0) 2991 : 2021-07-31T02:43:20.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:19 smithi045 conmon[39718]: cluster [DBG] osdmap e1973: 8 total, 8 up, 8 in 2021-07-31T02:43:20.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:19 smithi045 conmon[39718]: cluster 2021-07-31T02:43:18.605158+0000 mgr.x (mgr.24542) 2401 : cluster 2021-07-31T02:43:20.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:19 smithi045 conmon[39718]: [DBG] pgmap v3502: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:43:20.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:19 smithi045 conmon[33655]: cluster 2021-07-31T02:43:18.569219+0000 mon.a (mon.0) 2991 2021-07-31T02:43:20.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:19 smithi045 conmon[33655]: : cluster [DBG] osdmap e1973: 8 total, 8 up, 8 in 2021-07-31T02:43:20.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:19 smithi045 conmon[33655]: cluster 2021-07-31T02:43:18.605158+0000 mgr.x (mgr.24542) 2401 : cluster [DBG] pgmap v3502: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:43:21.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:20 smithi154 conmon[34553]: cluster 2021-07-31T02:43:19.574665+0000 mon.a (mon.0) 2992 : cluster [DBG] osdmap e1974: 8 total, 8 up, 8 in 2021-07-31T02:43:21.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:20 smithi045 conmon[33655]: cluster 2021-07-31T02:43:21.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:20 smithi045 conmon[33655]: 2021-07-31T02:43:21.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:20 smithi045 conmon[33655]: 2021 2021-07-31T02:43:21.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:20 smithi045 conmon[33655]: -07-31T02:43 2021-07-31T02:43:21.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:20 smithi045 conmon[33655]: :19.574665+0000 mon.a (mon.0) 2992 : cluster [DBG] osdmap e1974: 8 total, 8 up, 8 in 2021-07-31T02:43:21.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:20 smithi045 conmon[39718]: cluster 2021-07-31T02:43:19.574665 2021-07-31T02:43:21.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:20 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2992 : cluster [DBG] osdmap e1974: 8 total, 8 up, 8 in 2021-07-31T02:43:22.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:21 smithi154 conmon[34553]: cluster 2021-07-31T02:43:20.576188+0000 mon.a (mon.0) 2993 : cluster 2021-07-31T02:43:22.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:21 smithi154 conmon[34553]: [DBG] osdmap e1975: 8 total, 8 up, 8 in 2021-07-31T02:43:22.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:21 smithi154 conmon[34553]: cluster 2021-07-31T02:43:20.605541+0000 mgr.x (mgr.24542) 2402 : cluster [DBG] pgmap v3505: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 794 KiB/s rd, 5.0 KiB/s wr, 1.07k op/s 2021-07-31T02:43:22.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:21 smithi045 conmon[33655]: cluster 2021-07-31T02:43:20.576188+0000 mon.a (mon.0) 2993 : cluster [DBG] osdmap e1975: 8 total, 8 up, 8 in 2021-07-31T02:43:22.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:21 smithi045 conmon[33655]: cluster 2021-07-31T02:43:20.605541+0000 mgr.x (mgr.24542) 2402 : cluster [DBG] pgmap v3505: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 794 KiB/s rd, 5.0 KiB/s wr, 1.07k op/s 2021-07-31T02:43:22.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:21 smithi045 conmon[39718]: cluster 2021-07-31T02:43:20.576188+0000 mon.a (mon.0) 2993 2021-07-31T02:43:22.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:21 smithi045 conmon[39718]: : cluster [DBG] osdmap e1975: 8 total, 8 up, 8 in 2021-07-31T02:43:22.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:21 smithi045 conmon[39718]: cluster 2021-07-31T02:43:20 2021-07-31T02:43:22.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:21 smithi045 conmon[39718]: .605541+0000 mgr.x (mgr.24542) 2402 : cluster [DBG] pgmap v3505: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 794 KiB/s rd, 5.0 KiB/s wr, 1.07k op/s 2021-07-31T02:43:23.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:22 smithi154 conmon[34553]: cluster 2021-07-31T02:43:21.587812+0000 mon.a (mon.0) 2994 : cluster [DBG] osdmap e1976: 8 total, 8 up, 8 in 2021-07-31T02:43:23.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:22 smithi154 conmon[34553]: cluster 2021-07-31T02:43:22.561346+0000 2021-07-31T02:43:23.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:22 smithi154 conmon[34553]: mon.a (mon.0) 2995 : cluster [DBG] osdmap e1977: 8 total, 8 up, 8 in 2021-07-31T02:43:23.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:22 smithi045 conmon[39718]: cluster 2021-07-31T02:43:21.587812+0000 mon.a ( 2021-07-31T02:43:23.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:22 smithi045 conmon[39718]: mon.0) 2994 : cluster [DBG] osdmap e1976: 8 total, 8 up, 8 in 2021-07-31T02:43:23.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:22 smithi045 conmon[39718]: cluster 2021-07-31T02:43:22.561346+0000 mon.a (mon.0 2021-07-31T02:43:23.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:22 smithi045 conmon[39718]: ) 2995 : cluster [DBG] osdmap e1977: 8 total, 8 up, 8 in 2021-07-31T02:43:23.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:22 smithi045 conmon[33655]: cluster 2021-07-31T02:43:21.587812+0000 mon.a (mon.0) 2994 2021-07-31T02:43:23.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:22 smithi045 conmon[33655]: : cluster [DBG] osdmap e1976: 8 total, 8 up, 8 in 2021-07-31T02:43:23.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:22 smithi045 conmon[33655]: cluster 2021-07-31T02:43:22.561346+0000 mon.a (mon.0) 2995 : cluster [DBG] osdmap e1977: 8 total, 8 up, 8 in 2021-07-31T02:43:24.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:23 smithi154 conmon[34553]: cluster 2021-07-31T02:43:22.605904+0000 mgr.x (mgr.24542) 2403 : cluster 2021-07-31T02:43:24.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:23 smithi154 conmon[34553]: [DBG] pgmap v3508: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 786 KiB/s rd, 5.0 KiB/s wr, 1.06k op/s 2021-07-31T02:43:24.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:23 smithi045 conmon[39718]: cluster 2021-07-31T02:43:22. 2021-07-31T02:43:24.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:23 smithi045 conmon[39718]: 605904+0000 mgr.x (mgr.24542) 2403 : cluster [DBG] pgmap v3508: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 786 KiB/s rd, 5.0 KiB/s wr, 1.06k op/s 2021-07-31T02:43:24.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:23 smithi045 conmon[33655]: cluster 2021-07-31T02:43:22.605904+0000 mgr.x (mgr.24542) 2021-07-31T02:43:24.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:23 smithi045 conmon[33655]: 2403 : cluster [DBG] pgmap v3508: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 786 KiB/s rd, 5.0 KiB/s wr, 1.06k op/s 2021-07-31T02:43:25.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:24 smithi154 conmon[34553]: cluster 2021-07-31T02:43:23.595682+0000 mon.a (mon.0 2021-07-31T02:43:25.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:24 smithi154 conmon[34553]: ) 2996 : cluster [DBG] osdmap e1978: 8 total, 8 up, 8 in 2021-07-31T02:43:25.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:24 smithi045 conmon[39718]: cluster 2021-07-31T02:43:23.595682+0000 mon.a ( 2021-07-31T02:43:25.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:24 smithi045 conmon[39718]: mon.0) 2996 : cluster [DBG] osdmap e1978: 8 total, 8 up, 8 in 2021-07-31T02:43:25.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:24 smithi045 conmon[33655]: cluster 2021-07-31T02:43:23.595682 2021-07-31T02:43:25.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:24 smithi045 conmon[33655]: +0000 mon.a (mon.0) 2996 : cluster [DBG] osdmap e1978: 8 total, 8 up, 8 in 2021-07-31T02:43:26.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:25 smithi154 conmon[34553]: cluster 2021-07-31T02:43:24.600883+0000 mon.a (mon.0) 2997 2021-07-31T02:43:26.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:25 smithi154 conmon[34553]: : cluster [DBG] osdmap e1979: 8 total, 8 up, 8 in 2021-07-31T02:43:26.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:25 smithi154 conmon[34553]: cluster 2021-07-31T02:43:24.606275+0000 mgr.x (mgr.24542) 2404 : cluster [DBG] pgmap v3511: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 755 KiB/s rd, 5.0 KiB/s wr, 1.02k op/s 2021-07-31T02:43:26.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:25 smithi045 conmon[39718]: cluster 2021-07-31T02:43:24.600883+0000 mon.a (mon.0) 2021-07-31T02:43:26.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:25 smithi045 conmon[39718]: 2997 : cluster [DBG] osdmap e1979: 8 total, 8 up, 8 in 2021-07-31T02:43:26.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:25 smithi045 conmon[39718]: cluster 2021-07-31T02:43:24.606275+0000 mgr.x (mgr. 2021-07-31T02:43:26.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:25 smithi045 conmon[39718]: 24542) 2404 : cluster [DBG] pgmap v3511: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 755 KiB/s rd, 5.0 KiB/s wr, 1.02k op/s 2021-07-31T02:43:26.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:25 smithi045 conmon[33655]: cluster 2021-07-31T02:43:24.600883+0000 2021-07-31T02:43:26.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:25 smithi045 conmon[33655]: mon.a (mon.0) 2997 : cluster [DBG] osdmap e1979: 8 total, 8 up, 8 in 2021-07-31T02:43:26.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:25 smithi045 conmon[33655]: cluster 2021-07-31T02:43:24.606275+0000 mgr.x (mgr.24542 2021-07-31T02:43:26.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:25 smithi045 conmon[33655]: ) 2404 : cluster [DBG] pgmap v3511: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 755 KiB/s rd, 5.0 KiB/s wr, 1.02k op/s 2021-07-31T02:43:27.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:26 smithi154 conmon[34553]: cluster 2021-07-31T02:43:27.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:26 smithi154 conmon[34553]: 2021-07-31T02:43 2021-07-31T02:43:27.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:26 smithi154 conmon[34553]: :25.606972+0000 mon.a (mon.0) 2998 : cluster [DBG] osdmap e1980: 8 total, 8 up, 8 in 2021-07-31T02:43:27.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:26 smithi045 conmon[39718]: cluster 2021-07-31T02:43:27.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:26 smithi045 conmon[39718]: 2021-07-31T02:43:27.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:26 smithi045 conmon[39718]: 2021-07-31T02:43:25 2021-07-31T02:43:27.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:26 smithi045 conmon[39718]: .606972+0000 mon.a (mon.0) 2998 : cluster [DBG] osdmap e1980: 8 total, 8 up, 8 in 2021-07-31T02:43:27.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:26 smithi045 conmon[33655]: cluster 2021-07-31T02:43:25.606972+0000 2021-07-31T02:43:27.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:26 smithi045 conmon[33655]: mon.a (mon.0) 2998 : cluster [DBG] osdmap e1980: 8 total, 8 up, 8 in 2021-07-31T02:43:28.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:27 smithi154 conmon[34553]: cluster 2021-07-31T02:43:26.606766+0000 mgr.x (mgr.24542) 2405 : cluster [DBG] pgmap v3513: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 746 KiB/s rd, 4.9 KiB/s wr, 1.01k op/s 2021-07-31T02:43:28.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:27 smithi154 conmon[34553]: cluster 2021-07-31T02:43:26.608730+0000 mon.a (mon.0) 2999 : cluster [DBG] osdmap e1981: 8 total, 8 up, 8 in 2021-07-31T02:43:28.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:27 smithi154 conmon[34553]: cluster 2021-07-31T02:43:27.562714+0000 mon.a (mon.0) 3000 : cluster [DBG] osdmap e1982: 8 total, 8 up, 8 in 2021-07-31T02:43:28.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:27 smithi045 conmon[39718]: cluster 2021-07-31T02:43 2021-07-31T02:43:28.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:27 smithi045 conmon[39718]: :26.606766+0000 mgr.x (mgr.24542) 2405 : cluster [DBG] pgmap v3513: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 746 KiB/s rd, 4.9 KiB/s wr, 1.01k op/s 2021-07-31T02:43:28.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:27 smithi045 conmon[39718]: cluster 2021 2021-07-31T02:43:28.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:27 smithi045 conmon[39718]: -07-31T02:43 2021-07-31T02:43:28.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:27 smithi045 conmon[39718]: :26.608730+0000 mon.a ( 2021-07-31T02:43:28.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:27 smithi045 conmon[39718]: mon.0) 2999 : cluster 2021-07-31T02:43:28.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:27 smithi045 conmon[39718]: [DBG] osdmap e1981: 8 total, 8 up, 8 in 2021-07-31T02:43:28.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:27 smithi045 conmon[39718]: cluster 2021-07-31T02:43:28.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:27 smithi045 conmon[39718]: 2021-07- 2021-07-31T02:43:28.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:27 smithi045 conmon[39718]: 31T02:43:27.562714 2021-07-31T02:43:28.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:27 smithi045 conmon[39718]: +0000 mon.a (mon.0 2021-07-31T02:43:28.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:27 smithi045 conmon[39718]: ) 3000 : cluster [DBG] osdmap e1982: 8 total, 8 up, 8 in 2021-07-31T02:43:28.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:27 smithi045 conmon[39718]: 2021-07-31T02:43:28.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:27 smithi045 conmon[33655]: cluster 2021-07-31T02:43:26.606766 2021-07-31T02:43:28.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:27 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 2405 : cluster [DBG] pgmap v3513: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 746 KiB/s rd, 4.9 KiB/s wr, 1.01k op/s 2021-07-31T02:43:28.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:27 smithi045 conmon[33655]: cluster 2021-07-31T02:43:26.608730+0000 mon.a (mon.0) 2999 2021-07-31T02:43:28.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:27 smithi045 conmon[33655]: : cluster [DBG] osdmap e1981: 8 total, 8 up, 8 in 2021-07-31T02:43:28.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:27 smithi045 conmon[33655]: cluster 2021-07- 2021-07-31T02:43:28.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:27 smithi045 conmon[33655]: 31T02:43:27.562714+0000 mon.a 2021-07-31T02:43:28.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:27 smithi045 conmon[33655]: (mon.0) 3000 : cluster [DBG] osdmap e1982: 8 total, 8 up, 8 in 2021-07-31T02:43:28.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:27 smithi045 conmon[33655]: 2021-07-31T02:43:29.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:28 smithi154 conmon[34553]: audit 2021-07-31T02:43:28.151689+0000 mon.b (mon.2) 355 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:43:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:28 smithi154 conmon[34553]: 2021-07-31T02:43:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:28 smithi154 conmon[34553]: audit 2021-07-31T02:43:28.152303+0000 mon.a (mon.0) 3001 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:43:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:28 smithi154 conmon[34553]: audit 2021-07-31T02:43:28.155425+0000 mon.b (mon.2) 356 : audit [INF] 2021-07-31T02:43:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:28 smithi154 conmon[34553]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:43:29.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:28 smithi154 conmon[34553]: audit 2021-07-31T02:43:28. 2021-07-31T02:43:29.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:28 smithi154 conmon[34553]: 155784+0000 mon.a (mon.0) 3002 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:43:29.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:28 smithi154 conmon[34553]: cluster 2021-07-31T02:43:29.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:28 smithi154 conmon[34553]: 2021-07-31T02:43:28.569296 2021-07-31T02:43:29.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:28 smithi154 conmon[34553]: +0000 mon.a (mon.0) 3003 : cluster 2021-07-31T02:43:29.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:28 smithi154 conmon[34553]: [DBG] osdmap e1983: 8 total, 8 up, 8 in 2021-07-31T02:43:29.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:28 smithi045 conmon[39718]: audit 2021-07-31T02:43:29.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:28 smithi045 conmon[39718]: 2021-07-31T02:43:29.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:28 smithi045 conmon[39718]: 2021 2021-07-31T02:43:29.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:28 smithi045 conmon[39718]: - 2021-07-31T02:43:29.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:28 smithi045 conmon[39718]: 07 2021-07-31T02:43:29.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:28 smithi045 conmon[39718]: - 2021-07-31T02:43:29.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:28 smithi045 conmon[39718]: 31 2021-07-31T02:43:29.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:28 smithi045 conmon[39718]: T 2021-07-31T02:43:29.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:28 smithi045 conmon[39718]: 02 2021-07-31T02:43:29.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:28 smithi045 conmon[39718]: : 2021-07-31T02:43:29.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:28 smithi045 conmon[39718]: 43 2021-07-31T02:43:29.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:28 smithi045 conmon[39718]: : 2021-07-31T02:43:29.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:28 smithi045 conmon[39718]: 28 2021-07-31T02:43:29.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:28 smithi045 conmon[39718]: . 2021-07-31T02:43:29.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:28 smithi045 conmon[39718]: 151689 2021-07-31T02:43:29.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:28 smithi045 conmon[39718]: +0000 2021-07-31T02:43:29.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:28 smithi045 conmon[39718]: 2021-07-31T02:43:29.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:28 smithi045 conmon[39718]: mon.b 2021-07-31T02:43:29.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:28 smithi045 conmon[39718]: ( 2021-07-31T02:43:29.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:28 smithi045 conmon[39718]: mon 2021-07-31T02:43:29.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:28 smithi045 conmon[39718]: . 2021-07-31T02:43:29.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:28 smithi045 conmon[39718]: 2 2021-07-31T02:43:29.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:28 smithi045 conmon[39718]: ) 2021-07-31T02:43:29.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:28 smithi045 conmon[39718]: 355 2021-07-31T02:43:29.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:28 smithi045 conmon[39718]: : 2021-07-31T02:43:29.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:28 smithi045 conmon[39718]: audit 2021-07-31T02:43:29.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:28 smithi045 conmon[39718]: 2021-07-31T02:43:29.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:28 smithi045 conmon[39718]: [INF] 2021-07-31T02:43:29.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:28 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:43:29.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:28 smithi045 conmon[39718]: audit 2021-07-31T02:43:28.152303+0000 mon.a 2021-07-31T02:43:29.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:28 smithi045 conmon[39718]: (mon.0) 3001 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:43:29.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:28 smithi045 conmon[39718]: audit 2021-07-31T02:43:29.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:28 smithi045 conmon[39718]: 2021-07-31T02:43:29.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:28 smithi045 conmon[39718]: 2021-07-31T02:43 2021-07-31T02:43:29.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:28 smithi045 conmon[39718]: :28.155425+0000 mon.b (mon.2) 356 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:43:29.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:28 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T02:43:29.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:28 smithi045 conmon[39718]: 31T02:43:28.155784+0000 2021-07-31T02:43:29.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:28 smithi045 conmon[39718]: mon.a (mon.0) 3002 : 2021-07-31T02:43:29.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:28 smithi045 conmon[39718]: audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:43:29.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:28 smithi045 conmon[39718]: cluster 2021-07-31T 2021-07-31T02:43:29.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:28 smithi045 conmon[39718]: 02:43:28.569296+0000 mon.a (mon.0) 3003 2021-07-31T02:43:29.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:28 smithi045 conmon[39718]: : cluster [DBG] osdmap e1983: 8 total, 8 up, 8 in 2021-07-31T02:43:29.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:28 smithi045 conmon[33655]: audit 2021-07-31T02:43:28.151689+0000 mon.b (mon.2) 355 : audit [INF] 2021-07-31T02:43:29.040 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:28 smithi045 conmon[33655]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:43:29.040 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:28 smithi045 conmon[33655]: audit 2021-07-31T02:43:28.152303+0000 mon.a (mon.0) 3001 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:43:29.041 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:28 smithi045 conmon[33655]: audit 2021-07-31T02:43:28.155425+0000 2021-07-31T02:43:29.041 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:28 smithi045 conmon[33655]: mon.b (mon.2) 356 : audit [INF] 2021-07-31T02:43:29.041 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:28 smithi045 conmon[33655]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:43:29.041 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:28 smithi045 conmon[33655]: audit 2021-07-31 2021-07-31T02:43:29.042 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:28 smithi045 conmon[33655]: T02:43:28.155784+0000 mon.a (mon 2021-07-31T02:43:29.042 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:28 smithi045 conmon[33655]: .0) 3002 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:43:29.042 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:28 smithi045 conmon[33655]: cluster 2021-07-31T 2021-07-31T02:43:29.042 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:28 smithi045 conmon[33655]: 02:43:28.569296+0000 mon.a (mon.0) 3003 : cluster 2021-07-31T02:43:29.042 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:28 smithi045 conmon[33655]: [DBG] osdmap e1983: 8 total, 8 up, 8 in 2021-07-31T02:43:30.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:29 smithi154 conmon[34553]: cluster 2021-07-31T02:43:28.607149+0000 mgr.x (mgr.24542) 2406 : cluster [DBG] pgmap v3517: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:43:30.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:29 smithi154 conmon[34553]: 2021-07-31T02:43:30.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:29 smithi045 conmon[39718]: cluster 2021-07-31T02:43:28.607149+0000 mgr.x (mgr.24542 2021-07-31T02:43:30.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:29 smithi045 conmon[39718]: ) 2406 : cluster [DBG] pgmap v3517: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:43:30.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:29 smithi045 conmon[33655]: cluster 2021-07-31T02:43:28.607149+0000 mgr.x (mgr. 2021-07-31T02:43:30.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:29 smithi045 conmon[33655]: 24542) 2406 : cluster [DBG] pgmap v3517: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:43:31.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:30 smithi154 conmon[34553]: cluster 2021-07-31T02:43:29.624799+0000 mon.a (mon.0) 3004 2021-07-31T02:43:31.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:30 smithi154 conmon[34553]: : cluster [DBG] osdmap e1984: 8 total, 8 up, 8 in 2021-07-31T02:43:31.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:30 smithi045 conmon[39718]: cluster 2021-07-31T02:43:29.624799+0000 mon.a (mon 2021-07-31T02:43:31.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:30 smithi045 conmon[39718]: .0) 3004 : cluster [DBG] osdmap e1984: 8 total, 8 up, 8 in 2021-07-31T02:43:31.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:30 smithi045 conmon[33655]: cluster 2021-07-31T02:43:29.624799+0000 mon.a (mon.0) 3004 : 2021-07-31T02:43:31.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:30 smithi045 conmon[33655]: cluster [DBG] osdmap e1984: 8 total, 8 up, 8 in 2021-07-31T02:43:32.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:31 smithi154 conmon[34553]: cluster 2021-07-31T02:43:32.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:31 smithi154 conmon[34553]: 2021-07-31T02:43:32.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:31 smithi154 conmon[34553]: 2021 2021-07-31T02:43:32.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:31 smithi154 conmon[34553]: - 2021-07-31T02:43:32.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:31 smithi154 conmon[34553]: 07 2021-07-31T02:43:32.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:31 smithi154 conmon[34553]: -31T02:43:30.607628+0000 mgr.x ( 2021-07-31T02:43:32.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:31 smithi154 conmon[34553]: mgr.24542) 2407 : cluster [DBG] pgmap v3519: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 722 KiB/s rd, 5.0 KiB/s wr, 977 op/s 2021-07-31T02:43:32.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:31 smithi154 conmon[34553]: cluster 2021-07-31T02:43:30.630697+0000 mon.a (mon.0) 3005 : cluster [DBG] osdmap e1985: 8 total, 8 up, 8 in 2021-07-31T02:43:32.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:31 smithi045 conmon[33655]: cluster 2021-07-31T02:43:30.607628+0000 mgr.x (mgr.24542) 2407 : cluster [DBG] pgmap v3519: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 722 KiB/s rd, 5.0 KiB/s wr, 977 op/s 2021-07-31T02:43:32.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:31 smithi045 conmon[33655]: cluster 2021-07-31T02:43:30.630697+0000 mon.a (mon.0) 3005 : cluster [DBG] osdmap e1985: 8 total, 8 up, 8 in 2021-07-31T02:43:32.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:31 smithi045 conmon[39718]: cluster 2021-07-31T02:43:30.607628+0000 mgr.x (mgr 2021-07-31T02:43:32.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:31 smithi045 conmon[39718]: .24542) 2407 : cluster [DBG] pgmap v3519: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 722 KiB/s rd, 5.0 KiB/s wr, 977 op/s 2021-07-31T02:43:32.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:31 smithi045 conmon[39718]: cluster 2021- 2021-07-31T02:43:32.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:31 smithi045 conmon[39718]: 07-31T02:43:30.630697+0000 mon.a (mon.0) 3005 : cluster [DBG] osdmap e1985: 8 total, 8 up, 8 in 2021-07-31T02:43:33.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:32 smithi154 conmon[34553]: cluster 2021-07-31T02:43:31.632614+0000 mon.a (mon.0) 3006 : cluster [DBG] osdmap e1986: 8 total, 8 up, 8 in 2021-07-31T02:43:33.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:32 smithi154 conmon[34553]: 2021-07-31T02:43:33.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:32 smithi154 conmon[34553]: cluster 2021-07-31T02:43:32.565210+0000 mon.a (mon.0) 3007 : cluster [DBG] osdmap e1987: 8 total, 8 up, 8 in 2021-07-31T02:43:33.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:32 smithi045 conmon[33655]: cluster 2021-07-31T02:43:31.632614+0000 mon.a (mon.0) 2021-07-31T02:43:33.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:32 smithi045 conmon[33655]: 3006 : cluster [DBG] osdmap e1986: 8 total, 8 up, 8 in 2021-07-31T02:43:33.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:32 smithi045 conmon[33655]: cluster 2021-07-31T02:43:32.565210+0000 mon.a (mon.0) 3007 : cluster [DBG] osdmap e1987: 8 total, 8 up, 8 in 2021-07-31T02:43:33.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:32 smithi045 conmon[39718]: cluster 2021-07-31T02:43:31.632614+0000 mon.a (mon.0) 2021-07-31T02:43:33.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:32 smithi045 conmon[39718]: 3006 : cluster [DBG] osdmap e1986: 8 total, 8 up, 8 in 2021-07-31T02:43:33.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:32 smithi045 conmon[39718]: cluster 2021-07-31T02:43:32.565210+0000 mon.a (mon.0) 3007 : cluster [DBG] osdmap e1987: 8 total, 8 up, 8 in 2021-07-31T02:43:33.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:32 smithi045 conmon[39718]: 2021-07-31T02:43:33.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:33 smithi045 conmon[33655]: cluster 2021-07-31T02:43:32.608078+0000 mgr.x (mgr.24542) 2021-07-31T02:43:33.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:33 smithi045 conmon[33655]: 2408 : cluster [DBG] pgmap v3523: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 722 KiB/s rd, 5.0 KiB/s wr, 977 op/s 2021-07-31T02:43:33.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:33 smithi045 conmon[33655]: audit 2021-07-31T02:43:33.485097+0000 mon.b (mon.2) 357 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:43:33.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:33 smithi045 conmon[33655]: audit 2021-07-31T02:43:33.486683+0000 mon.b (mon 2021-07-31T02:43:33.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:33 smithi045 conmon[33655]: .2) 358 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:43:33.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:33 smithi045 conmon[33655]: audit 2021-07-31T02:43:33.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:33 smithi045 conmon[33655]: 2021-07-31T02:43:33.487591+0000 2021-07-31T02:43:33.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:33 smithi045 conmon[33655]: mon.b (mon.2) 359 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:43:33.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:33 smithi045 conmon[33655]: cephadm 2021-07-31T02:43:33.488376 2021-07-31T02:43:33.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:33 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 2409 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:43:33.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:33 smithi045 conmon[33655]: cluster 2021-07-31T02:43:33.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:33 smithi045 conmon[33655]: 2021-07-31T02:43:33.576639+0000 2021-07-31T02:43:33.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:33 smithi045 conmon[33655]: mon.a (mon.0) 3008 : cluster [DBG] osdmap e1988: 8 total, 8 up, 8 in 2021-07-31T02:43:33.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:33 smithi045 conmon[33655]: 2021-07-31T02:43:33.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:33 smithi045 conmon[39718]: cluster 2021-07-31T02:43:32.608078+0000 mgr.x (mgr.24542 2021-07-31T02:43:33.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:33 smithi045 conmon[39718]: ) 2408 : cluster [DBG] pgmap v3523: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 722 KiB/s rd, 5.0 KiB/s wr, 977 op/s 2021-07-31T02:43:33.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:33 smithi045 conmon[39718]: audit 2021-07-31T02:43:33 2021-07-31T02:43:33.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:33 smithi045 conmon[39718]: .485097+0000 mon.b (mon.2) 357 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:43:33.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:33 smithi045 conmon[39718]: audit 2021-07-31T02:43:33.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:33 smithi045 conmon[39718]: 2021-07-31T02:43:33.486683 2021-07-31T02:43:33.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:33 smithi045 conmon[39718]: +0000 mon.b (mon.2) 358 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:43:33.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:33 smithi045 conmon[39718]: audit 2021-07-31T02:43:33.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:33 smithi045 conmon[39718]: 2021-07-31T02:43:33.487591+0000 mon.b (mon. 2021-07-31T02:43:33.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:33 smithi045 conmon[39718]: 2) 359 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:43:33.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:33 smithi045 conmon[39718]: cephadm 2021-07-31T02:43:33.488376+0000 2021-07-31T02:43:33.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:33 smithi045 conmon[39718]: mgr.x (mgr.24542) 2409 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:43:33.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:33 smithi045 conmon[39718]: cluster 2021-07-31T 2021-07-31T02:43:33.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:33 smithi045 conmon[39718]: 02:43:33.576639+0000 mon.a (mon.0) 3008 2021-07-31T02:43:33.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:33 smithi045 conmon[39718]: : cluster [DBG] osdmap e1988: 8 total, 8 up, 8 in 2021-07-31T02:43:34.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[34553]: cluster 2021-07-31T02:43:32.608078+0000 mgr.x (mgr.24542) 2408 : cluster [DBG] pgmap v3523: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 722 KiB/s rd, 5.0 KiB/s wr, 977 op/s 2021-07-31T02:43:34.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[34553]: audit 2021-07-31T02: 2021-07-31T02:43:34.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[34553]: 43:33.485097+0000 mon.b (mon.2) 357 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:43:34.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[34553]: audit 2021-07-31T02:43:33.486683+0000 mon.b (mon.2) 358 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:43:34.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[34553]: audit 2021-07-31T02:43 2021-07-31T02:43:34.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[34553]: :33.487591+0000 mon.b (mon.2) 359 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:43:34.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[34553]: cephadm 2021-07-31T 2021-07-31T02:43:34.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[34553]: 02:43:33.488376+0000 mgr.x (mgr. 2021-07-31T02:43:34.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[34553]: 24542) 2409 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:43:34.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[34553]: cluster 2021 2021-07-31T02:43:34.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[34553]: -07-31T02:43:33.576639+0000 2021-07-31T02:43:34.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[34553]: mon.a (mon.0) 3008 : cluster [DBG] 2021-07-31T02:43:34.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[34553]: osdmap e1988: 8 total, 8 up, 8 in 2021-07-31T02:43:34.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: debug 2021-07-31T02:43:33.532+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:43:34.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:43:34.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:43:34.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: yield 2021-07-31T02:43:34.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:43:34.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:43:34.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:43:34.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:43:34.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:43:34.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:43:34.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:43:34.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:43:34.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:43:34.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: task.result() 2021-07-31T02:43:34.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:43:34.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: debug 2021-07-31T02:43:33.534+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:43:34.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:43:34.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:43:34.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: yield 2021-07-31T02:43:34.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:43:34.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:43:34.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:43:34.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:43:34.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:43:34.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:43:34.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:43:34.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:43:34.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:43:34.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: task.result() 2021-07-31T02:43:34.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:43:34.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: debug 2021-07-31T02:43:33.569+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:43:34.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:43:34.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:43:34.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: yield 2021-07-31T02:43:34.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:43:34.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:43:34.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:43:34.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:43:34.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:43:34.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:43:34.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:43:34.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:43:34.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:43:34.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: task.result() 2021-07-31T02:43:34.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:43:34.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: debug 2021-07-31T02:43:33.570+0000 7fcb93160700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:43:34.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:43:34.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:43:34.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: yield 2021-07-31T02:43:34.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:43:34.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:43:34.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:43:34.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:43:34.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:43:34.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:43:34.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:43:34.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:43:34.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:43:34.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: task.result() 2021-07-31T02:43:34.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:43:34.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: 2021-07-31T02:43:34.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:43:34.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: 2021-07-31T02:43:34.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:43:34.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:43:34.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:43:34.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:43:34.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:43:34.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:43:34.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:43:34.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:43:34.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:43:34.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:43:34.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:43:34.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:43:34.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:43:34.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:43:34.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:43:34.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:43:34.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:43:34.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:43:34.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: raise 2021-07-31T02:43:34.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:43:34.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:43:34.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:43:34.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:43:34.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:43:34.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: debug 2021-07-31T02:43:33.578+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:43:34.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:43:34.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:43:34.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: yield 2021-07-31T02:43:34.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:43:34.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:43:34.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:43:34.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:43:34.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:43:34.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:43:34.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:43:34.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:43:34.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:43:34.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: task.result() 2021-07-31T02:43:34.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:43:34.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: debug 2021-07-31T02:43:34.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: 2021-07-31T02:43:33.579+0000 7fcb9295f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:43:34.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:43:34.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:43:34.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: yield 2021-07-31T02:43:34.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:43:34.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:43:34.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:43:34.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:43:34.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:43:34.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:43:34.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:43:34.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:43:34.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:43:34.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: task.result() 2021-07-31T02:43:34.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:43:34.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: 2021-07-31T02:43:34.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:43:34.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: 2021-07-31T02:43:34.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:43:34.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:43:34.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:43:34.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:43:34.035 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:43:34.035 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:43:34.035 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:43:34.035 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:43:34.036 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:43:34.036 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:43:34.036 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:43:34.036 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:43:34.036 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:43:34.037 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:43:34.037 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:43:34.037 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:43:34.037 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:43:34.037 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:43:34.038 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: raise 2021-07-31T02:43:34.038 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:43:34.038 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:43:34.038 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:43:34.039 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:43:34.039 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:43:33 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:43:35.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: cephadm 2021-07-31T02:43:33.533355+0000 mgr.x (mgr.24542) 2410 : cephadm [ERR] 2021-07-31T02:43:35.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:43:35.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:43:35.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: yield 2021-07-31T02:43:35.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:43:35.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:43:35.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:43:35.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:43:35.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:43:35.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:43:35.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:43:35.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:43:35.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:43:35.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: task.result() 2021-07-31T02:43:35.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:43:35.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: cephadm 2021-07-31T02:43:33.533915+0000 mgr.x (mgr.24542) 2411 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:43:35.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: cephadm 2021-07-31T02:43:33.535057+0000 mgr.x (mgr.24542) 2412 : cephadm [ERR] 2021-07-31T02:43:35.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:43:35.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:43:35.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: yield 2021-07-31T02:43:35.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:43:35.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:43:35.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:43:35.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:43:35.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:43:35.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:43:35.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:43:35.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:43:35.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:43:35.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: task.result() 2021-07-31T02:43:35.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:43:35.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: cephadm 2021-07-31T02:43:33.544934+0000 mgr.x (mgr.24542) 2413 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:43:35.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: cephadm 2021-07-31T02:43:33.570521+0000 mgr.x 2021-07-31T02:43:35.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: (mgr.24542) 2414 : cephadm [ERR] 2021-07-31T02:43:35.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:43:35.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:43:35.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: yield 2021-07-31T02:43:35.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:43:35.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:43:35.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:43:35.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:43:35.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:43:35.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:43:35.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:43:35.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:43:35.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:43:35.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: task.result() 2021-07-31T02:43:35.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:43:35.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: cephadm 2021-07-31T02:43:33.571053+0000 mgr.x (mgr.24542) 2415 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:43:35.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:43:35.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:43:35.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: yield 2021-07-31T02:43:35.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:43:35.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:43:35.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:43:35.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:43:35.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:43:35.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:43:35.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:43:35.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:43:35.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:43:35.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: task.result() 2021-07-31T02:43:35.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:43:35.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: 2021-07-31T02:43:35.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:43:35.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: 2021-07-31T02:43:35.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:43:35.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:43:35.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:43:35.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:43:35.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:43:35.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:43:35.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:43:35.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:43:35.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:43:35.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:43:35.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:43:35.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:43:35.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:43:35.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:43:35.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:43:35.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:43:35.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:43:35.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:43:35.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: raise 2021-07-31T02:43:35.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:43:35.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:43:35.035 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:43:35.035 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:43:35.036 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:43:35.036 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: cephadm 2021-07-31T02:43:33.579414+0000 mgr.x 2021-07-31T02:43:35.037 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: (mgr.24542) 2416 : cephadm [ERR] 2021-07-31T02:43:35.037 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:43:35.038 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:43:35.038 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: yield 2021-07-31T02:43:35.038 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:43:35.039 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:43:35.039 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:43:35.040 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:43:35.040 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:43:35.041 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:43:35.041 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:43:35.042 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:43:35.042 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:43:35.042 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: task.result() 2021-07-31T02:43:35.043 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:43:35.043 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: cephadm 2021-07-31T02:43:33.580604+0000 mgr.x (mgr.24542) 2417 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:43:35.043 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:43:35.044 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:43:35.044 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: yield 2021-07-31T02:43:35.044 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:43:35.045 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:43:35.045 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:43:35.045 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:43:35.046 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:43:35.046 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:43:35.046 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:43:35.047 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:43:35.047 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:43:35.048 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: task.result() 2021-07-31T02:43:35.048 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:43:35.049 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: 2021-07-31T02:43:35.049 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:43:35.049 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: 2021-07-31T02:43:35.049 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:43:35.050 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:43:35.050 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:43:35.050 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:43:35.050 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:43:35.051 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:43:35.051 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:43:35.051 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:43:35.051 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:43:35.051 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:43:35.052 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:43:35.052 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:43:35.052 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:43:35.052 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:43:35.053 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:43:35.053 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:43:35.053 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:43:35.053 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:43:35.053 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: raise 2021-07-31T02:43:35.054 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:43:35.054 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:43:35.054 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:43:35.054 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:43:35.055 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:43:35.055 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:34 smithi154 conmon[34553]: cluster 2021-07-31T02:43:34.579063+0000 mon.a (mon.0) 3009 : cluster [DBG] osdmap e1989: 8 total, 8 up, 8 in 2021-07-31T02:43:35.056 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: cephadm 2021-07-31T02:43:33.533355+0000 mgr.x (mgr.24542) 2410 : cephadm [ERR] 2021-07-31T02:43:35.057 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: 2021-07-31T02:43:35.057 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:43:35.058 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:43:35.058 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: yield 2021-07-31T02:43:35.059 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:43:35.059 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:43:35.059 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:43:35.060 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:43:35.060 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:43:35.061 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:43:35.061 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:43:35.061 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:43:35.062 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:43:35.062 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: task.result() 2021-07-31T02:43:35.063 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:43:35.063 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: cephadm 2021-07-31T02:43:33.533915+0000 mgr.x (mgr.24542) 2411 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:43:35.063 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: cephadm 2021-07-31T02:43:33.535057+0000 mgr.x (mgr.24542) 2412 : cephadm [ERR] 2021-07-31T02:43:35.064 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:43:35.064 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:43:35.064 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: yield 2021-07-31T02:43:35.064 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:43:35.065 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:43:35.065 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:43:35.065 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:43:35.065 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:43:35.065 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:43:35.066 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:43:35.066 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:43:35.066 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:43:35.067 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: task.result() 2021-07-31T02:43:35.067 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:43:35.067 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: cephadm 2021-07-31T02:43:33.544934+0000 mgr.x (mgr.24542) 2413 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:43:35.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: cephadm 2021-07-31T02:43:33.570521+0000 mgr.x (mgr.24542) 2414 : cephadm [ERR] 2021-07-31T02:43:35.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:43:35.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:43:35.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: yield 2021-07-31T02:43:35.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:43:35.069 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: cephadm 2021-07-31T02:43: 2021-07-31T02:43:35.070 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: 33.533355+0000 mgr.x (mgr.24542) 2410 : cephadm [ERR] 2021-07-31T02:43:35.070 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:43:35.070 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:43:35.070 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: yield 2021-07-31T02:43:35.071 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:43:35.071 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:43:35.071 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:43:35.071 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:43:35.072 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:43:35.072 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:43:35.072 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:43:35.072 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:43:35.073 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:43:35.073 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: task.result() 2021-07-31T02:43:35.073 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:43:35.073 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: cephadm 2021-07-31T02:43:33.533915+0000 mgr.x (mgr.24542) 2411 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:43:35.074 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: cephadm 2021-07-31T02:43:35.074 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: 2021-07-31 2021-07-31T02:43:35.075 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: T02:43:33.535057+0000 mgr.x (mgr 2021-07-31T02:43:35.075 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: .24542) 2412 : cephadm [ERR] 2021-07-31T02:43:35.075 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: 2021-07-31T02:43:35.076 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:43:35.076 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:43:35.077 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: yield 2021-07-31T02:43:35.077 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:43:35.078 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:43:35.078 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:43:35.078 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:43:35.079 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:43:35.079 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:43:35.080 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:43:35.080 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:43:35.081 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:43:35.081 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: task.result() 2021-07-31T02:43:35.081 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:43:35.082 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: cephadm 2021-07-31T02:43:33.544934+0000 mgr.x (mgr.24542) 2413 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:43:35.082 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: cephadm 2021-07-31 2021-07-31T02:43:35.083 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: T02:43:33.570521+0000 mgr.x (mgr.24542) 2021-07-31T02:43:35.083 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: 2414 : cephadm [ERR] 2021-07-31T02:43:35.087 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:43:35.087 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:43:35.088 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: yield 2021-07-31T02:43:35.088 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:43:35.088 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:43:35.089 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:43:35.089 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:43:35.089 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:43:35.090 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:43:35.090 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:43:35.090 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:43:35.090 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:43:35.090 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: task.result() 2021-07-31T02:43:35.091 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:43:35.091 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: cephadm 2021-07-31T02:43:35.091 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: 2021-07-31T02:43:33.571053+0000 2021-07-31T02:43:35.092 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: mgr.x (mgr.24542) 2415 : cephadm [ERR] 2021-07-31T02:43:35.092 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:43:35.092 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:43:35.093 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:43:35.093 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: yield 2021-07-31T02:43:35.093 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:43:35.093 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:43:35.093 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:43:35.094 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:43:35.094 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:43:35.094 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:43:35.094 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:43:35.095 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:43:35.095 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:43:35.095 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: task.result() 2021-07-31T02:43:35.095 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:43:35.096 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: 2021-07-31T02:43:35.096 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:43:35.096 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: 2021-07-31T02:43:35.096 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:43:35.096 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:43:35.097 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:43:35.097 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:43:35.097 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:43:35.097 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:43:35.098 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:43:35.098 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:43:35.098 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:43:35.098 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:43:35.099 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:43:35.099 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:43:35.100 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:43:35.100 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:43:35.100 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:43:35.100 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:43:35.101 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:43:35.101 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:43:35.101 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: task.result() 2021-07-31T02:43:35.102 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:43:35.102 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: cephadm 2021-07-31T02:43:33.571053+0000 mgr.x (mgr. 2021-07-31T02:43:35.102 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: 24542) 2415 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:43:35.103 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:43:35.103 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:43:35.104 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: yield 2021-07-31T02:43:35.104 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:43:35.104 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:43:35.105 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:43:35.105 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:43:35.106 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:43:35.106 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:43:35.106 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:43:35.107 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:43:35.107 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:43:35.108 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: task.result() 2021-07-31T02:43:35.108 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:43:35.109 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: 2021-07-31T02:43:35.109 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:43:35.110 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: 2021-07-31T02:43:35.110 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:43:35.111 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:43:35.111 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:43:35.112 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:43:35.112 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:43:35.113 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:43:35.113 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:43:35.113 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:43:35.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:43:35.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:43:35.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:43:35.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:43:35.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:43:35.117 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:43:35.117 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:43:35.118 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:43:35.118 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:43:35.119 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:43:35.119 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:43:35.119 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:43:35.120 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:43:35.120 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: raise 2021-07-31T02:43:35.121 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:43:35.121 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:43:35.122 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:43:35.122 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:43:35.122 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:43:35.123 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: cephadm 2021-07-31T02:43:33.579414+0000 mgr.x (mgr.24542) 2416 : cephadm [ERR] 2021-07-31T02:43:35.123 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:43:35.124 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:43:35.124 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: yield 2021-07-31T02:43:35.125 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:43:35.125 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:43:35.125 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:43:35.126 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:43:35.126 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:43:35.127 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:43:35.127 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:43:35.127 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:43:35.128 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:43:35.128 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: task.result() 2021-07-31T02:43:35.129 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:43:35.129 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: cephadm 2021-07-31T02:43:33.580604+0000 mgr.x (mgr.24542) 2417 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:43:35.130 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:43:35.130 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:43:35.130 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: yield 2021-07-31T02:43:35.131 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:43:35.131 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:43:35.132 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:43:35.132 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:43:35.133 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:43:35.133 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:43:35.134 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:43:35.134 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:43:35.135 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:43:35.135 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:43:35.136 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:43:35.136 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: raise 2021-07-31T02:43:35.136 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:43:35.137 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:43:35.137 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:43:35.138 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:43:35.138 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:43:35.139 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: cephadm 2021-07-31T02:43:33.579414+0000 mgr.x (mgr.24542) 2416 : cephadm [ERR] 2021-07-31T02:43:35.139 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:43:35.139 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:43:35.140 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: yield 2021-07-31T02:43:35.140 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:43:35.141 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:43:35.141 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:43:35.141 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:43:35.142 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:43:35.142 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:43:35.143 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:43:35.143 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:43:35.144 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:43:35.144 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: task.result() 2021-07-31T02:43:35.144 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:43:35.145 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: cephadm 2021-07-31T02:43 2021-07-31T02:43:35.145 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: :33.580604+0000 mgr.x (mgr.24542) 2417 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:43:35.146 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:43:35.146 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:43:35.147 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: yield 2021-07-31T02:43:35.147 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:43:35.147 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:43:35.148 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:43:35.148 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:43:35.149 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:43:35.149 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:43:35.149 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:43:35.150 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:43:35.150 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:43:35.151 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:43:35.152 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:43:35.152 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:43:35.152 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: task.result() 2021-07-31T02:43:35.153 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:43:35.153 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: 2021-07-31T02:43:35.153 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:43:35.153 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: 2021-07-31T02:43:35.154 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:43:35.154 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:43:35.154 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:43:35.154 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:43:35.154 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:43:35.155 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:43:35.155 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:43:35.155 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:43:35.155 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:43:35.156 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:43:35.156 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:43:35.156 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:43:35.156 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:43:35.156 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:43:35.157 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:43:35.157 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:43:35.157 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:43:35.157 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:43:35.157 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: raise 2021-07-31T02:43:35.158 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:43:35.158 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:43:35.158 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:43:35.158 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:43:35.159 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:43:35.159 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: cluster 2021-07-31T02: 2021-07-31T02:43:35.159 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[39718]: 43:34.579063+0000 mon.a (mon.0) 3009 : cluster [DBG] osdmap e1989: 8 total, 8 up, 8 in 2021-07-31T02:43:35.159 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: task.result() 2021-07-31T02:43:35.160 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:43:35.160 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: 2021-07-31T02:43:35.160 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:43:35.160 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: 2021-07-31T02:43:35.160 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:43:35.161 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:43:35.161 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:43:35.161 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:43:35.161 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:43:35.162 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:43:35.162 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:43:35.162 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:43:35.162 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:43:35.162 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:43:35.163 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:43:35.163 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:43:35.163 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:43:35.163 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:43:35.163 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:43:35.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:43:35.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:43:35.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:43:35.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: raise 2021-07-31T02:43:35.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:43:35.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:43:35.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:43:35.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:43:35.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:43:35.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:34 smithi045 conmon[33655]: cluster 2021-07-31T02:43:34.579063+0000 mon.a (mon.0) 3009 : cluster [DBG] osdmap e1989: 8 total, 8 up, 8 in 2021-07-31T02:43:36.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:35 smithi154 conmon[34553]: cluster 2021-07-31T02:43:34.608498+0000 mgr.x (mgr.24542) 2418 : cluster [DBG] 2021-07-31T02:43:36.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:35 smithi154 conmon[34553]: pgmap v3526: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 694 KiB/s rd, 5.0 KiB/s wr, 938 op/s 2021-07-31T02:43:36.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:35 smithi154 conmon[34553]: cluster 2021-07-31T02:43:35.583109+0000 mon.a (mon.0) 3010 : cluster [DBG] osdmap e1990: 8 total, 8 up, 8 in 2021-07-31T02:43:36.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:35 smithi045 conmon[39718]: cluster 2021-07-31T02:43:34.608498+0000 mgr.x (mgr.24542) 2418 : cluster [DBG] pgmap v3526: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 694 KiB/s rd, 5.0 KiB/s wr, 938 op/s 2021-07-31T02:43:36.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:35 smithi045 conmon[39718]: cluster 2021-07-31T02:43:35.583109+0000 mon.a (mon.0) 3010 : cluster [DBG] osdmap e1990: 8 total, 8 up, 8 in 2021-07-31T02:43:36.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:35 smithi045 conmon[33655]: cluster 2021-07-31T02:43:34.608498+0000 mgr.x (mgr.24542) 2418 : cluster 2021-07-31T02:43:36.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:35 smithi045 conmon[33655]: [DBG] pgmap v3526: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 694 KiB/s rd, 5.0 KiB/s wr, 938 op/s 2021-07-31T02:43:36.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:35 smithi045 conmon[33655]: cluster 2021-07-31T02:43:35.583109+0000 mon.a (mon.0) 3010 : cluster [DBG] osdmap e1990: 8 total, 8 up, 8 in 2021-07-31T02:43:38.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:37 smithi154 conmon[34553]: cluster 2021-07-31T02:43:36.585356+0000 mon.a (mon.0) 3011 : cluster [DBG] 2021-07-31T02:43:38.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:37 smithi154 conmon[34553]: osdmap e1991: 8 total, 8 up, 8 in 2021-07-31T02:43:38.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:37 smithi154 conmon[34553]: cluster 2021-07-31T02:43:36.608871+0000 mgr.x (mgr.24542) 2419 : cluster [DBG] pgmap v3529: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 690 KiB/s rd, 5.0 KiB/s wr, 933 op/s 2021-07-31T02:43:38.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:37 smithi154 conmon[34553]: cluster 2021-07-31T02:43:37.565556+0000 mon.a (mon.0) 3012 : cluster 2021-07-31T02:43:38.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:37 smithi154 conmon[34553]: [DBG] osdmap e1992: 8 total, 8 up, 8 in 2021-07-31T02:43:38.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:37 smithi045 conmon[39718]: cluster 2021-07-31T02:43:36.585356+0000 2021-07-31T02:43:38.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:37 smithi045 conmon[39718]: mon.a (mon.0) 3011 : cluster [DBG] osdmap e1991: 8 total, 8 up, 8 in 2021-07-31T02:43:38.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:37 smithi045 conmon[39718]: cluster 2021 2021-07-31T02:43:38.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:37 smithi045 conmon[39718]: -07-31T02:43:36.608871+0000 2021-07-31T02:43:38.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:37 smithi045 conmon[39718]: mgr.x (mgr.24542) 2419 : 2021-07-31T02:43:38.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:37 smithi045 conmon[39718]: cluster [DBG] pgmap v3529: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 690 KiB/s rd, 5.0 KiB/s wr, 933 op/s 2021-07-31T02:43:38.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:37 smithi045 conmon[39718]: cluster 2021-07 2021-07-31T02:43:38.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:37 smithi045 conmon[39718]: -31T02:43:37.565556+0000 mon.a (mon.0) 3012 : cluster 2021-07-31T02:43:38.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:37 smithi045 conmon[39718]: [DBG] osdmap e1992: 8 total, 8 up, 8 in 2021-07-31T02:43:38.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:37 smithi045 conmon[33655]: cluster 2021-07-31T02:43:36.585356+0000 mon.a 2021-07-31T02:43:38.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:37 smithi045 conmon[33655]: (mon.0) 3011 : cluster [DBG] osdmap e1991: 8 total, 8 up, 8 in 2021-07-31T02:43:38.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:37 smithi045 conmon[33655]: cluster 2021-07-31T02:43:36.608871+0000 mgr.x (mgr.24542) 2419 : cluster [DBG] pgmap v3529: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 690 KiB/s rd, 5.0 KiB/s wr, 933 op/s 2021-07-31T02:43:38.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:37 smithi045 conmon[33655]: cluster 2021- 2021-07-31T02:43:38.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:37 smithi045 conmon[33655]: 07-31T02:43:37 2021-07-31T02:43:38.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:37 smithi045 conmon[33655]: .565556+0000 mon.a (mon. 2021-07-31T02:43:38.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:37 smithi045 conmon[33655]: 0) 3012 : cluster [DBG] osdmap e1992: 8 total, 8 up, 8 in 2021-07-31T02:43:38.674 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:43:38.953 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:43:38.953 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 58m ago 59m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:43:38.954 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 56m ago 59m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:43:38.954 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 56m ago 62m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:43:38.954 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 58m ago 64m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:43:38.954 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 58m ago 64m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:43:38.955 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 56m ago 62m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:43:38.955 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 58m ago 62m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:43:38.955 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 58m ago 59m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:43:38.956 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 56m ago 59m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:43:38.956 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 58m ago 62m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:43:38.957 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 58m ago 61m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:43:38.957 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 58m ago 61m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:43:38.957 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 58m ago 61m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:43:38.958 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 56m ago 60m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:43:38.958 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 56m ago 60m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:43:38.958 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 56m ago 60m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:43:38.958 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 56m ago 59m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:43:38.958 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 56m ago 59m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:43:39.285 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:43:39.285 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:43:39.286 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:43:39.286 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:43:39.286 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:43:39.287 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:43:39.287 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:43:39.287 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:43:39.288 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:43:39.288 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:43:39.289 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:43:39.289 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:43:39.290 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:43:39.290 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:43:39.290 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:43:39.290 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:43:39.291 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:43:40.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:39 smithi154 conmon[34553]: cluster 2021-07-31T02:43:38.591156+0000 mon.a (mon.0) 3013 : cluster [DBG] osdmap e1993: 8 total, 8 up, 8 in 2021-07-31T02:43:40.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:39 smithi154 conmon[34553]: 2021-07-31T02:43:40.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:39 smithi154 conmon[34553]: cluster 2021-07-31T02:43:38.609298+0000 mgr.x (mgr.24542) 2420 : cluster [DBG] pgmap v3532: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:43:40.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:39 smithi154 conmon[34553]: audit 2021-07-31T02:43:38.662600+0000 mgr.x (mgr.24542) 2421 : audit [DBG] from='client.17934 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:43:40.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:39 smithi154 conmon[34553]: audit 2021-07 2021-07-31T02:43:40.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:39 smithi154 conmon[34553]: -31T02:43:38.945621+0000 mgr.x (mgr.24542) 2422 : audit [DBG] from='client.27880 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:43:40.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:39 smithi154 conmon[34553]: audit 2021-07-31 2021-07-31T02:43:40.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:39 smithi154 conmon[34553]: T02:43:39.284835+0000 mon.a (mon.0) 3014 : audit [DBG] from='client.? 172.21.15.45:0/871242117' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:43:40.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:39 smithi045 conmon[33655]: cluster 2021-07-31T02:43:38.591156+0000 mon.a (mon. 2021-07-31T02:43:40.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:39 smithi045 conmon[33655]: 0) 3013 : cluster [DBG] osdmap e1993: 8 total, 8 up, 8 in 2021-07-31T02:43:40.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:39 smithi045 conmon[33655]: cluster 2021-07-31T02:43:38.609298+0000 mgr.x (mgr.24542) 2420 : cluster [DBG] pgmap v3532: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:43:40.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:39 smithi045 conmon[33655]: audit 2021-07-31T02: 2021-07-31T02:43:40.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:39 smithi045 conmon[33655]: 43:38.662600+0000 mgr.x (mgr.24542) 2421 : audit [DBG] from='client.17934 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:43:40.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:39 smithi045 conmon[33655]: audit 2021-07-31T02:43:38. 2021-07-31T02:43:40.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:39 smithi045 conmon[33655]: 945621+0000 mgr.x (mgr.24542) 2422 : audit 2021-07-31T02:43:40.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:39 smithi045 conmon[33655]: [DBG] from='client.27880 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:43:40.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:39 smithi045 conmon[33655]: audit 2021-07-31T 2021-07-31T02:43:40.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:39 smithi045 conmon[33655]: 02:43:39.284835+0000 mon.a (mon.0 2021-07-31T02:43:40.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:39 smithi045 conmon[33655]: ) 3014 : audit [DBG] from='client.? 172.21.15.45:0/871242117' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:43:40.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:39 smithi045 conmon[39718]: cluster 2021-07-31T02:43: 2021-07-31T02:43:40.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:39 smithi045 conmon[39718]: 38.591156+0000 mon.a (mon.0) 3013 : cluster [DBG] osdmap e1993: 8 total, 8 up, 8 in 2021-07-31T02:43:40.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:39 smithi045 conmon[39718]: cluster 2021-07 2021-07-31T02:43:40.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:39 smithi045 conmon[39718]: -31T02:43:38.609298 2021-07-31T02:43:40.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:39 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542 2021-07-31T02:43:40.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:39 smithi045 conmon[39718]: ) 2420 : cluster [DBG] pgmap v3532: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:43:40.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:39 smithi045 conmon[39718]: audit 2021 2021-07-31T02:43:40.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:39 smithi045 conmon[39718]: -07-31T02:43:38 2021-07-31T02:43:40.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:39 smithi045 conmon[39718]: .662600+0000 mgr.x (mgr. 2021-07-31T02:43:40.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:39 smithi045 conmon[39718]: 24542) 2421 : audit [DBG] from='client.17934 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:43:40.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:39 smithi045 conmon[39718]: 2021-07-31T02:43:40.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:39 smithi045 conmon[39718]: audit 2021 2021-07-31T02:43:40.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:39 smithi045 conmon[39718]: -07-31T02:43:38 2021-07-31T02:43:40.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:39 smithi045 conmon[39718]: .945621+0000 mgr.x (mgr 2021-07-31T02:43:40.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:39 smithi045 conmon[39718]: .24542) 2422 : audit [DBG] 2021-07-31T02:43:40.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:39 smithi045 conmon[39718]: from='client.27880 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:43:40.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:39 smithi045 conmon[39718]: audit 2021 2021-07-31T02:43:40.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:39 smithi045 conmon[39718]: -07-31T02:43:39 2021-07-31T02:43:40.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:39 smithi045 conmon[39718]: .284835+0000 mon.a (mon. 2021-07-31T02:43:40.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:39 smithi045 conmon[39718]: 0) 3014 : audit [DBG] 2021-07-31T02:43:40.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:39 smithi045 conmon[39718]: from='client.? 172.21.15.45:0/871242117' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:43:41.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:40 smithi154 conmon[34553]: cluster 2021-07-31T02:43:39.597559+0000 mon.a (mon.0) 3015 : cluster [DBG] osdmap e1994: 8 total, 8 up, 8 in 2021-07-31T02:43:41.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:40 smithi045 conmon[39718]: cluster 2021-07-31T02:43:39.597559+0000 mon.a 2021-07-31T02:43:41.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:40 smithi045 conmon[39718]: (mon.0) 3015 : cluster [DBG] osdmap e1994: 8 total, 8 up, 8 in 2021-07-31T02:43:41.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:40 smithi045 conmon[33655]: cluster 2021-07-31T02:43:39.597559+0000 mon.a (mon.0) 3015 : cluster 2021-07-31T02:43:41.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:40 smithi045 conmon[33655]: [DBG] osdmap e1994: 8 total, 8 up, 8 in 2021-07-31T02:43:42.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:41 smithi154 conmon[34553]: cluster 2021-07-31T02:43:40.602875+0000 mon.a (mon.0) 3016 : cluster [DBG] 2021-07-31T02:43:42.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:41 smithi154 conmon[34553]: osdmap e1995: 8 total, 8 up, 8 in 2021-07-31T02:43:42.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:41 smithi154 conmon[34553]: cluster 2021-07-31T02:43:40.609674+0000 mgr.x (mgr.24542) 2423 : cluster [DBG] pgmap v3535: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 656 KiB/s rd, 5.0 KiB/s wr, 889 op/s 2021-07-31T02:43:42.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:41 smithi045 conmon[39718]: cluster 2021-07-31T02:43:40.602875+0000 mon.a (mon. 2021-07-31T02:43:42.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:41 smithi045 conmon[39718]: 0) 3016 : cluster [DBG] osdmap e1995: 8 total, 8 up, 8 in 2021-07-31T02:43:42.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:41 smithi045 conmon[39718]: cluster 2021-07-31T02:43:40.609674+0000 2021-07-31T02:43:42.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:41 smithi045 conmon[39718]: mgr.x (mgr.24542) 2423 : cluster [DBG] pgmap v3535: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 656 KiB/s rd, 5.0 KiB/s wr, 889 op/s 2021-07-31T02:43:42.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:41 smithi045 conmon[33655]: cluster 2021-07-31T02:43:40.602875+0000 mon.a (mon.0) 3016 : cluster [DBG] osdmap e1995: 8 total, 8 up, 8 in 2021-07-31T02:43:42.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:41 smithi045 conmon[33655]: 2021-07-31T02:43:42.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:41 smithi045 conmon[33655]: cluster 2021-07-31T02:43:40.609674+0000 mgr.x (mgr.24542) 2423 : cluster [DBG] pgmap v3535: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 656 KiB/s rd, 5.0 KiB/s wr, 889 op/s 2021-07-31T02:43:43.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:42 smithi154 conmon[34553]: cluster 2021-07-31T02:43:41.609032+0000 mon.a (mon.0) 3017 : cluster [DBG] 2021-07-31T02:43:43.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:42 smithi154 conmon[34553]: osdmap e1996: 8 total, 8 up, 8 in 2021-07-31T02:43:43.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:42 smithi154 conmon[34553]: cluster 2021-07-31T02:43:42.567451+0000 mon.a (mon.0) 3018 : cluster [DBG] osdmap e1997: 8 total, 8 up, 8 in 2021-07-31T02:43:43.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:42 smithi045 conmon[39718]: cluster 2021-07-31T02:43:41.609032+0000 mon.a (mon.0) 3017 : cluster [DBG] osdmap e1996: 8 total, 8 up, 8 in 2021-07-31T02:43:43.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:42 smithi045 conmon[39718]: cluster 2021-07-31T02:43:42.567451+0000 mon.a (mon.0) 3018 : cluster [DBG] osdmap e1997: 8 total, 8 up, 8 in 2021-07-31T02:43:43.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:42 smithi045 conmon[33655]: cluster 2021-07-31T02:43:41.609032+0000 mon.a (mon.0) 3017 : cluster [DBG] osdmap e1996: 8 total, 8 up, 8 in 2021-07-31T02:43:43.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:42 smithi045 conmon[33655]: cluster 2021-07-31T02:43:42.567451+0000 mon.a (mon.0) 3018 : cluster [DBG] osdmap e1997: 8 total, 8 up, 8 in 2021-07-31T02:43:44.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:43 smithi154 conmon[34553]: cluster 2021-07-31T02:43:42.610104+0000 mgr.x (mgr.24542) 2424 : cluster [DBG] 2021-07-31T02:43:44.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:43 smithi154 conmon[34553]: pgmap v3538: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 656 KiB/s rd, 5.0 KiB/s wr, 889 op/s 2021-07-31T02:43:44.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:43 smithi154 conmon[34553]: cluster 2021-07-31T02:43:43.574422+0000 mon.a (mon.0) 3019 : cluster [DBG] osdmap e1998: 8 total, 8 up, 8 in 2021-07-31T02:43:44.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:43 smithi045 conmon[33655]: cluster 2021-07-31T02:43:42.610104+0000 mgr.x (mgr.24542) 2424 2021-07-31T02:43:44.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:43 smithi045 conmon[33655]: : cluster [DBG] pgmap v3538: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 656 KiB/s rd, 5.0 KiB/s wr, 889 op/s 2021-07-31T02:43:44.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:43 smithi045 conmon[33655]: cluster 2021-07-31T02:43:43.574422+0000 mon.a (mon.0) 3019 : cluster [DBG] osdmap e1998: 8 total, 8 up, 8 in 2021-07-31T02:43:44.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:43 smithi045 conmon[39718]: cluster 2021-07-31T02:43:42.610104+0000 mgr.x (mgr.24542) 2021-07-31T02:43:44.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:43 smithi045 conmon[39718]: 2424 : cluster [DBG] pgmap v3538: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 656 KiB/s rd, 5.0 KiB/s wr, 889 op/s 2021-07-31T02:43:44.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:43 smithi045 conmon[39718]: cluster 2021-07-31T02:43:43.574422+0000 2021-07-31T02:43:44.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:43 smithi045 conmon[39718]: mon.a (mon.0) 3019 : 2021-07-31T02:43:44.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:43 smithi045 conmon[39718]: cluster [DBG] osdmap e1998: 8 total, 8 up, 8 in 2021-07-31T02:43:46.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:45 smithi154 conmon[34553]: cluster 2021-07-31T02:43:44.576796+0000 mon.a (mon.0) 3020 : cluster [DBG] osdmap e1999: 8 total, 8 up, 8 in 2021-07-31T02:43:46.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:45 smithi154 conmon[34553]: cluster 2021-07-31T02:43:44.610742+0000 mgr.x (mgr.24542) 2425 : cluster [DBG] pgmap v3541: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 624 KiB/s rd, 5.0 KiB/s wr, 846 op/s 2021-07-31T02:43:46.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:45 smithi045 conmon[39718]: cluster 2021-07-31T02:43:44.576796+0000 mon.a (mon.0) 3020 : cluster [DBG] osdmap e1999: 8 total, 8 up, 8 in 2021-07-31T02:43:46.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:45 smithi045 conmon[39718]: cluster 2021-07-31T02:43:44.610742+0000 2021-07-31T02:43:46.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:45 smithi045 conmon[39718]: mgr.x (mgr.24542) 2425 : cluster [DBG] 2021-07-31T02:43:46.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:45 smithi045 conmon[39718]: pgmap v3541: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 624 KiB/s rd, 5.0 KiB/s wr, 846 op/s 2021-07-31T02:43:46.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:45 smithi045 conmon[33655]: cluster 2021-07-31T02:43:44.576796+0000 mon.a (mon.0) 3020 : 2021-07-31T02:43:46.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:45 smithi045 conmon[33655]: cluster [DBG] osdmap e1999: 8 total, 8 up, 8 in 2021-07-31T02:43:46.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:45 smithi045 conmon[33655]: cluster 2021-07-31T02:43:44.610742+0000 mgr.x (mgr.24542) 2425 : cluster [DBG] pgmap v3541: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 624 KiB/s rd, 5.0 KiB/s wr, 846 op/s 2021-07-31T02:43:47.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:46 smithi154 conmon[34553]: cluster 2021-07-31T02:43:45.582595+0000 mon.a (mon.0) 3021 : cluster [DBG] osdmap e2000: 8 total, 8 up, 8 in 2021-07-31T02:43:47.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:46 smithi045 conmon[39718]: cluster 2021-07-31T02:43:45.582595+0000 mon.a (mon 2021-07-31T02:43:47.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:46 smithi045 conmon[39718]: .0) 3021 : cluster [DBG] osdmap e2000: 8 total, 8 up, 8 in 2021-07-31T02:43:47.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:46 smithi045 conmon[33655]: cluster 2021-07-31T02: 2021-07-31T02:43:47.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:46 smithi045 conmon[33655]: 43:45.582595+0000 mon.a (mon.0) 3021 : cluster [DBG] osdmap e2000: 8 total, 8 up, 8 in 2021-07-31T02:43:48.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:47 smithi154 conmon[34553]: cluster 2021-07-31T02:43:46.588504+0000 mon.a (mon.0) 3022 : cluster [DBG] osdmap e2001: 8 total, 8 up, 8 in 2021-07-31T02:43:48.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:47 smithi154 conmon[34553]: cluster 2021-07-31T02:43:46.611131+0000 mgr.x (mgr.24542) 2426 : cluster [DBG] pgmap v3544: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 624 KiB/s rd, 5.0 KiB/s wr, 846 op/s 2021-07-31T02:43:48.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:47 smithi154 conmon[34553]: cluster 2021-07-31T 2021-07-31T02:43:48.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:47 smithi154 conmon[34553]: 02:43:47.568937+0000 mon.a (mon.0) 3023 : cluster [DBG] 2021-07-31T02:43:48.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:47 smithi154 conmon[34553]: osdmap e2002: 8 total, 8 up, 8 in 2021-07-31T02:43:48.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:47 smithi045 conmon[33655]: cluster 2021-07-31T02:43:46.588504+0000 mon.a (mon.0 2021-07-31T02:43:48.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:47 smithi045 conmon[33655]: ) 3022 : cluster [DBG] osdmap e2001: 8 total, 8 up, 8 in 2021-07-31T02:43:48.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:47 smithi045 conmon[33655]: cluster 2021-07-31T02:43:46.611131+0000 mgr.x (mgr.24542) 2426 : cluster [DBG] pgmap v3544: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 624 KiB/s rd, 5.0 KiB/s wr, 846 op/s 2021-07-31T02:43:48.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:47 smithi045 conmon[33655]: cluster 2021-07-31T02:43:47.568937+0000 mon.a ( 2021-07-31T02:43:48.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:47 smithi045 conmon[33655]: mon.0) 3023 : cluster [DBG] osdmap e2002: 8 total, 8 up, 8 in 2021-07-31T02:43:48.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:47 smithi045 conmon[39718]: cluster 2021-07-31T02:43:46.588504+0000 mon.a (mon.0 2021-07-31T02:43:48.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:47 smithi045 conmon[39718]: ) 3022 : cluster [DBG] osdmap e2001: 8 total, 8 up, 8 in 2021-07-31T02:43:48.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:47 smithi045 conmon[39718]: cluster 2021-07-31T02:43:46.611131+0000 mgr.x ( 2021-07-31T02:43:48.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:47 smithi045 conmon[39718]: mgr.24542) 2426 : cluster [DBG] pgmap v3544: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 624 KiB/s rd, 5.0 KiB/s wr, 846 op/s 2021-07-31T02:43:48.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:47 smithi045 conmon[39718]: cluster 2021-07-31T02: 2021-07-31T02:43:48.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:47 smithi045 conmon[39718]: 43:47.568937+0000 mon.a (mon.0 2021-07-31T02:43:48.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:47 smithi045 conmon[39718]: ) 3023 : cluster [DBG] osdmap e2002: 8 total, 8 up, 8 in 2021-07-31T02:43:50.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:49 smithi154 conmon[34553]: cluster 2021-07-31T02:43:48.595789+0000 mon.a (mon.0) 3024 : 2021-07-31T02:43:50.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:49 smithi154 conmon[34553]: cluster [DBG] osdmap e2003: 8 total, 8 up, 8 in 2021-07-31T02:43:50.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:49 smithi154 conmon[34553]: cluster 2021-07-31T02:43:48.611556+0000 mgr.x (mgr.24542) 2427 : cluster [DBG] pgmap v3547: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:43:50.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:49 smithi045 conmon[33655]: cluster 2021-07-31T02:43:48.595789+0000 mon.a 2021-07-31T02:43:50.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:49 smithi045 conmon[33655]: (mon.0) 3024 : cluster [DBG] osdmap e2003: 8 total, 8 up, 8 in 2021-07-31T02:43:50.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:49 smithi045 conmon[33655]: cluster 2021-07-31T02:43:48.611556+0000 mgr.x (mgr.24542) 2427 : cluster [DBG] pgmap v3547: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:43:50.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:49 smithi045 conmon[39718]: cluster 2021-07-31T02:43:48.595789+0000 mon.a (mon. 2021-07-31T02:43:50.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:49 smithi045 conmon[39718]: 0) 3024 : cluster [DBG] osdmap e2003: 8 total, 8 up, 8 in 2021-07-31T02:43:50.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:49 smithi045 conmon[39718]: cluster 2021-07-31T02:43:48.611556+0000 2021-07-31T02:43:50.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:49 smithi045 conmon[39718]: mgr.x (mgr.24542) 2427 : cluster [DBG] pgmap v3547: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:43:51.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:50 smithi154 conmon[34553]: cluster 2021-07-31T02:43:49.601437+0000 mon.a (mon.0) 3025 : cluster 2021-07-31T02:43:51.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:50 smithi154 conmon[34553]: [DBG] osdmap e2004: 8 total, 8 up, 8 in 2021-07-31T02:43:51.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:50 smithi045 conmon[39718]: cluster 2021-07-31T02:43:49.601437+0000 mon.a (mon 2021-07-31T02:43:51.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:50 smithi045 conmon[39718]: .0) 3025 : cluster [DBG] osdmap e2004: 8 total, 8 up, 8 in 2021-07-31T02:43:51.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:50 smithi045 conmon[33655]: cluster 2021-07-31T02:43:49.601437+0000 mon.a (mon.0) 3025 : 2021-07-31T02:43:51.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:50 smithi045 conmon[33655]: cluster [DBG] osdmap e2004: 8 total, 8 up, 8 in 2021-07-31T02:43:52.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:51 smithi154 conmon[34553]: cluster 2021-07-31T02:43:50.607047+0000 mon.a (mon.0) 3026 : cluster 2021-07-31T02:43:52.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:51 smithi154 conmon[34553]: [DBG] osdmap e2005: 8 total, 8 up, 8 in 2021-07-31T02:43:52.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:51 smithi154 conmon[34553]: cluster 2021-07-31T02:43:50.612255+0000 mgr.x (mgr.24542) 2428 : cluster [DBG] pgmap v3550: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 592 KiB/s rd, 5.0 KiB/s wr, 802 op/s 2021-07-31T02:43:52.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:51 smithi045 conmon[33655]: cluster 2021-07-31T02:43:50.607047+0000 mon.a (mon.0) 3026 : cluster [DBG] 2021-07-31T02:43:52.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:51 smithi045 conmon[33655]: osdmap e2005: 8 total, 8 up, 8 in 2021-07-31T02:43:52.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:51 smithi045 conmon[33655]: cluster 2021-07-31T02:43:50.612255+0000 mgr.x (mgr.24542) 2428 : cluster [DBG] pgmap v3550: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 592 KiB/s rd, 5.0 KiB/s wr, 802 op/s 2021-07-31T02:43:52.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:51 smithi045 conmon[39718]: cluster 2021-07-31T02:43:50.607047+0000 mon.a (mon.0 2021-07-31T02:43:52.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:51 smithi045 conmon[39718]: ) 3026 : cluster [DBG] osdmap e2005: 8 total, 8 up, 8 in 2021-07-31T02:43:52.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:51 smithi045 conmon[39718]: cluster 2021-07-31T02:43:50.612255+0000 mgr.x (mgr.24542) 2428 : cluster [DBG] 2021-07-31T02:43:52.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:51 smithi045 conmon[39718]: pgmap v3550: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 592 KiB/s rd, 5.0 KiB/s wr, 802 op/s 2021-07-31T02:43:53.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:52 smithi154 conmon[34553]: cluster 2021-07-31T02:43:51.612639+0000 mon.a (mon.0) 3027 : cluster 2021-07-31T02:43:53.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:52 smithi154 conmon[34553]: [DBG] osdmap e2006: 8 total, 8 up, 8 in 2021-07-31T02:43:53.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:52 smithi154 conmon[34553]: cluster 2021-07-31T02:43:52.570517+0000 mon.a (mon.0) 3028 : cluster [DBG] osdmap e2007: 8 total, 8 up, 8 in 2021-07-31T02:43:53.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:52 smithi045 conmon[33655]: cluster 2021-07-31T02:43:51.612639+0000 mon.a (mon.0) 3027 2021-07-31T02:43:53.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:52 smithi045 conmon[33655]: : cluster [DBG] osdmap e2006: 8 total, 8 up, 8 in 2021-07-31T02:43:53.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:52 smithi045 conmon[33655]: cluster 2021-07-31T02:43:52.570517+0000 mon.a (mon.0) 3028 : cluster [DBG] osdmap e2007: 8 total, 8 up, 8 in 2021-07-31T02:43:53.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:52 smithi045 conmon[39718]: cluster 2021-07-31T02:43:51.612639+0000 mon.a 2021-07-31T02:43:53.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:52 smithi045 conmon[39718]: (mon.0) 3027 : cluster [DBG] osdmap e2006: 8 total, 8 up, 8 in 2021-07-31T02:43:53.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:52 smithi045 conmon[39718]: cluster 2021-07-31T02:43:52 2021-07-31T02:43:53.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:52 smithi045 conmon[39718]: .570517+0000 mon.a (mon.0) 3028 : cluster [DBG] osdmap e2007: 8 total, 8 up, 8 in 2021-07-31T02:43:54.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:53 smithi154 conmon[34553]: cluster 2021-07-31T02:43:54.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:53 smithi154 conmon[34553]: 2021-07-31T02:43:54.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:53 smithi154 conmon[34553]: 2021-07-31T02:43:52.612635 2021-07-31T02:43:54.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:53 smithi154 conmon[34553]: +0000 mgr.x (mgr.24542) 2429 : cluster [DBG] pgmap v3553: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 592 KiB/s rd, 5.0 KiB/s wr, 802 op/s 2021-07-31T02:43:54.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:53 smithi154 conmon[34553]: cluster 2021- 2021-07-31T02:43:54.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:53 smithi154 conmon[34553]: 07-31T02:43:53.577838 2021-07-31T02:43:54.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:53 smithi154 conmon[34553]: +0000 mon.a (mon.0) 3029 2021-07-31T02:43:54.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:53 smithi154 conmon[34553]: : cluster [DBG] osdmap e2008: 8 total, 8 up, 8 in 2021-07-31T02:43:54.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:53 smithi045 conmon[33655]: cluster 2021-07-31T02:43:52.612635+0000 mgr.x (mgr.24542) 2429 : cluster [DBG] pgmap v3553: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 592 KiB/s rd, 5.0 KiB/s wr, 802 op/s 2021-07-31T02:43:54.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:53 smithi045 conmon[33655]: cluster 2021-07-31T02:43:53.577838+0000 mon.a (mon.0) 3029 : cluster [DBG] osdmap e2008: 8 total, 8 up, 8 in 2021-07-31T02:43:54.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:53 smithi045 conmon[39718]: cluster 2021-07-31T02:43:52.612635+0000 mgr.x (mgr.24542) 2429 : 2021-07-31T02:43:54.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:53 smithi045 conmon[39718]: cluster [DBG] pgmap v3553: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 592 KiB/s rd, 5.0 KiB/s wr, 802 op/s 2021-07-31T02:43:54.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:53 smithi045 conmon[39718]: cluster 2021-07-31T02:43:53.577838+0000 mon.a ( 2021-07-31T02:43:54.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:53 smithi045 conmon[39718]: mon.0) 3029 : cluster [DBG] osdmap e2008: 8 total, 8 up, 8 in 2021-07-31T02:43:56.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:55 smithi154 conmon[34553]: cluster 2021-07-31T02:43:56.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:55 smithi154 conmon[34553]: 2021-07-31T02:43:56.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:55 smithi154 conmon[34553]: 2021 2021-07-31T02:43:56.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:55 smithi154 conmon[34553]: -07-31T02:43:54.613381+0000 mgr.x ( 2021-07-31T02:43:56.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:55 smithi154 conmon[34553]: mgr.24542) 2430 : cluster [DBG] pgmap v3555: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 558 KiB/s rd, 5.0 KiB/s wr, 757 op/s 2021-07-31T02:43:56.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:55 smithi154 conmon[34553]: cluster 2021-07-31T02:43:54.626423+0000 mon.a (mon.0) 3030 : cluster [DBG] osdmap e2009: 8 total, 8 up, 8 in 2021-07-31T02:43:56.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:55 smithi045 conmon[33655]: cluster 2021-07-31T02:43:54.613381+0000 mgr.x (mgr.24542 2021-07-31T02:43:56.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:55 smithi045 conmon[33655]: ) 2430 : cluster [DBG] pgmap v3555: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 558 KiB/s rd, 5.0 KiB/s wr, 757 op/s 2021-07-31T02:43:56.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:55 smithi045 conmon[33655]: cluster 2021-07-31T02:43:54.626423+0000 mon.a (mon.0) 3030 : cluster [DBG] osdmap e2009: 8 total, 8 up, 8 in 2021-07-31T02:43:56.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:55 smithi045 conmon[39718]: cluster 2021-07-31T02:43:54 2021-07-31T02:43:56.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:55 smithi045 conmon[39718]: .613381+0000 mgr.x (mgr.24542) 2430 : cluster [DBG] pgmap v3555: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 558 KiB/s rd, 5.0 KiB/s wr, 757 op/s 2021-07-31T02:43:56.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:55 smithi045 conmon[39718]: cluster 2021-07-31T02:43: 2021-07-31T02:43:56.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:55 smithi045 conmon[39718]: 54.626423+0000 mon.a (mon.0) 3030 : cluster [DBG] osdmap e2009: 8 total, 8 up, 8 in 2021-07-31T02:43:56.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:55 smithi045 conmon[39718]: 2021-07-31T02:43:57.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:56 smithi154 conmon[34553]: cluster 2021-07-31T02:43:55.627495+0000 mon.a (mon.0) 3031 2021-07-31T02:43:57.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:56 smithi154 conmon[34553]: : cluster [DBG] osdmap e2010: 8 total, 8 up, 8 in 2021-07-31T02:43:57.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:56 smithi045 conmon[33655]: cluster 2021-07-31T02:43:55.627495+0000 mon.a (mon.0) 3031 2021-07-31T02:43:57.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:56 smithi045 conmon[33655]: : cluster [DBG] osdmap e2010: 8 total, 8 up, 8 in 2021-07-31T02:43:57.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:56 smithi045 conmon[39718]: cluster 2021-07-31T02:43:55.627495+0000 mon.a (mon. 2021-07-31T02:43:57.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:56 smithi045 conmon[39718]: 0) 3031 : cluster [DBG] osdmap e2010: 8 total, 8 up, 8 in 2021-07-31T02:43:58.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:57 smithi154 conmon[34553]: cluster 2021-07-31T02:43:56.613899+0000 mgr.x (mgr.24542) 2431 : cluster [DBG] pgmap v3558: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 553 KiB/s rd, 4.9 KiB/s wr, 750 op/s 2021-07-31T02:43:58.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:57 smithi154 conmon[34553]: cluster 2021-07-31T02:43:56.640353+0000 mon.a (mon.0) 3032 : cluster [DBG] osdmap e2011: 8 total, 8 up, 8 in 2021-07-31T02:43:58.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:57 smithi154 conmon[34553]: cluster 2021-07-31T02:43:57.572729+0000 mon.a (mon.0) 2021-07-31T02:43:58.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:57 smithi154 conmon[34553]: 3033 : cluster [DBG] osdmap e2012: 8 total, 8 up, 8 in 2021-07-31T02:43:58.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:57 smithi045 conmon[39718]: cluster 2021-07-31T02:43:56.613899+0000 mgr.x (mgr.24542) 2431 : cluster [DBG] pgmap v3558: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 553 KiB/s rd, 4.9 KiB/s wr, 750 op/s 2021-07-31T02:43:58.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:57 smithi045 conmon[39718]: cluster 2021-07-31T02:43:56.640353+0000 mon.a (mon.0) 3032 : cluster [DBG] osdmap e2011: 8 total, 8 up, 8 in 2021-07-31T02:43:58.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:57 smithi045 conmon[39718]: cluster 2021- 2021-07-31T02:43:58.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:57 smithi045 conmon[39718]: 07-31T02:43:57 2021-07-31T02:43:58.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:57 smithi045 conmon[39718]: .572729+0000 mon.a (mon.0) 3033 2021-07-31T02:43:58.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:57 smithi045 conmon[39718]: : cluster [DBG] osdmap e2012: 8 total, 8 up, 8 in 2021-07-31T02:43:58.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:57 smithi045 conmon[33655]: cluster 2021-07-31T02:43:56.613899+0000 mgr.x (mgr.24542 2021-07-31T02:43:58.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:57 smithi045 conmon[33655]: ) 2431 : cluster [DBG] pgmap v3558: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 553 KiB/s rd, 4.9 KiB/s wr, 750 op/s 2021-07-31T02:43:58.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:57 smithi045 conmon[33655]: cluster 2021-07-31T02:43:56.640353+0000 mon.a (mon.0) 3032 : cluster [DBG] osdmap e2011: 8 total, 8 up, 8 in 2021-07-31T02:43:58.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:57 smithi045 conmon[33655]: cluster 2021-07-31T02:43:57.572729+0000 mon.a 2021-07-31T02:43:58.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:57 smithi045 conmon[33655]: (mon.0) 3033 : cluster [DBG] osdmap e2012: 8 total, 8 up, 8 in 2021-07-31T02:44:00.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:59 smithi154 conmon[34553]: cluster 2021-07-31T02:43:58.580230+0000 mon.a (mon.0 2021-07-31T02:44:00.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:59 smithi154 conmon[34553]: ) 3034 : cluster [DBG] osdmap e2013: 8 total, 8 up, 8 in 2021-07-31T02:44:00.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:43:59 smithi154 conmon[34553]: cluster 2021-07-31T02:43:58.614433+0000 mgr.x (mgr.24542) 2432 : cluster [DBG] pgmap v3562: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:44:00.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:59 smithi045 conmon[39718]: cluster 2021-07-31T02:43:58.580230+0000 2021-07-31T02:44:00.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:59 smithi045 conmon[39718]: mon.a (mon.0) 3034 : cluster [DBG] osdmap e2013: 8 total, 8 up, 8 in 2021-07-31T02:44:00.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:59 smithi045 conmon[39718]: cluster 2021-07-31T02:43 2021-07-31T02:44:00.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:43:59 smithi045 conmon[39718]: :58.614433+0000 mgr.x (mgr.24542) 2432 : cluster [DBG] pgmap v3562: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:44:00.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:59 smithi045 conmon[33655]: cluster 2021-07-31T02:43:58.580230 2021-07-31T02:44:00.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:59 smithi045 conmon[33655]: +0000 mon.a (mon.0) 3034 : cluster [DBG] osdmap e2013: 8 total, 8 up, 8 in 2021-07-31T02:44:00.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:59 smithi045 conmon[33655]: cluster 2021-07-31T02:43:58.614433+0000 mgr.x (mgr.24542) 2432 2021-07-31T02:44:00.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:43:59 smithi045 conmon[33655]: : cluster [DBG] pgmap v3562: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:44:01.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:00 smithi154 conmon[34553]: cluster 2021-07-31T02:43:59.582040+0000 mon.a (mon.0) 3035 : cluster [DBG] osdmap e2014: 8 total, 8 up, 8 in 2021-07-31T02:44:01.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:00 smithi045 conmon[39718]: cluster 2021-07-31T02:43:59.582040+0000 mon.a (mon 2021-07-31T02:44:01.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:00 smithi045 conmon[39718]: .0) 3035 : cluster [DBG] osdmap e2014: 8 total, 8 up, 8 in 2021-07-31T02:44:01.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:00 smithi045 conmon[33655]: cluster 2021-07-31T02:43:59.582040+0000 mon.a (mon.0) 3035 : cluster 2021-07-31T02:44:01.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:00 smithi045 conmon[33655]: [DBG] osdmap e2014: 8 total, 8 up, 8 in 2021-07-31T02:44:02.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:01 smithi154 conmon[34553]: cluster 2021-07-31T02:44:00.594120+0000 mon.a (mon.0) 3036 : cluster [DBG] osdmap e2015: 8 total, 8 up, 8 in 2021-07-31T02:44:02.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:01 smithi154 conmon[34553]: 2021-07-31T02:44:02.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:01 smithi154 conmon[34553]: cluster 2021-07-31T02:44:00.614830+0000 mgr.x (mgr.24542) 2433 : cluster [DBG] pgmap v3565: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 531 KiB/s rd, 5.0 KiB/s wr, 719 op/s 2021-07-31T02:44:02.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:01 smithi045 conmon[33655]: cluster 2021-07-31T02:44:00.594120+0000 mon.a (mon.0) 2021-07-31T02:44:02.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:01 smithi045 conmon[33655]: 3036 : cluster [DBG] osdmap e2015: 8 total, 8 up, 8 in 2021-07-31T02:44:02.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:01 smithi045 conmon[33655]: cluster 2021-07-31T02:44:00.614830+0000 mgr.x (mgr.24542) 2433 : cluster [DBG] pgmap v3565: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 531 KiB/s rd, 5.0 KiB/s wr, 719 op/s 2021-07-31T02:44:02.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:01 smithi045 conmon[39718]: cluster 2021-07-31T02:44:00.594120+0000 mon.a (mon 2021-07-31T02:44:02.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:01 smithi045 conmon[39718]: .0) 3036 : cluster [DBG] osdmap e2015: 8 total, 8 up, 8 in 2021-07-31T02:44:02.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:01 smithi045 conmon[39718]: cluster 2021-07-31T02:44:00.614830+0000 mgr.x (mgr.24542) 2433 2021-07-31T02:44:02.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:01 smithi045 conmon[39718]: : cluster [DBG] pgmap v3565: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 531 KiB/s rd, 5.0 KiB/s wr, 719 op/s 2021-07-31T02:44:03.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:02 smithi154 conmon[34553]: cluster 2021-07-31T02:44:01.600279+0000 mon.a (mon.0) 3037 2021-07-31T02:44:03.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:02 smithi154 conmon[34553]: : cluster [DBG] osdmap e2016: 8 total, 8 up, 8 in 2021-07-31T02:44:03.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:02 smithi154 conmon[34553]: cluster 2021-07-31T02:44:02.574602+0000 mon.a (mon.0) 3038 : cluster [DBG] osdmap e2017: 8 total, 8 up, 8 in 2021-07-31T02:44:03.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:02 smithi045 conmon[39718]: cluster 2021-07-31T02:44:01.600279+0000 mon.a ( 2021-07-31T02:44:03.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:02 smithi045 conmon[39718]: mon.0) 3037 : cluster [DBG] osdmap e2016: 8 total, 8 up, 8 in 2021-07-31T02:44:03.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:02 smithi045 conmon[39718]: cluster 2021-07-31T02: 2021-07-31T02:44:03.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:02 smithi045 conmon[39718]: 44:02.574602+0000 mon.a (mon.0) 3038 : 2021-07-31T02:44:03.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:02 smithi045 conmon[39718]: cluster [DBG] osdmap e2017: 8 total, 8 up, 8 in 2021-07-31T02:44:03.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:02 smithi045 conmon[33655]: cluster 2021-07-31T02:44:01.600279+0000 mon.a (mon.0) 3037 : cluster 2021-07-31T02:44:03.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:02 smithi045 conmon[33655]: [DBG] osdmap e2016: 8 total, 8 up, 8 in 2021-07-31T02:44:03.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:02 smithi045 conmon[33655]: cluster 2021-07-31T02:44:02.574602+0000 mon.a (mon.0) 3038 : cluster [DBG] osdmap e2017: 8 total, 8 up, 8 in 2021-07-31T02:44:04.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:03 smithi154 conmon[34553]: cluster 2021-07-31T02:44:02.615130+0000 mgr.x (mgr.24542) 2434 : cluster 2021-07-31T02:44:04.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:03 smithi154 conmon[34553]: [DBG] pgmap v3568: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 527 KiB/s rd, 5.0 KiB/s wr, 714 op/s 2021-07-31T02:44:04.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:03 smithi045 conmon[33655]: cluster 2021-07-31T02:44:02.615130+0000 mgr.x (mgr.24542) 2021-07-31T02:44:04.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:03 smithi045 conmon[33655]: 2434 : cluster [DBG] pgmap v3568: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 527 KiB/s rd, 5.0 KiB/s wr, 714 op/s 2021-07-31T02:44:04.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:03 smithi045 conmon[39718]: cluster 2021-07-31T02:44:02.615130+0000 mgr.x (mgr.24542) 2434 : cluster [DBG] pgmap v3568: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 527 KiB/s rd, 5.0 KiB/s wr, 714 op/s 2021-07-31T02:44:05.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:04 smithi154 conmon[34553]: cluster 2021-07-31T02:44:03.607608+0000 mon.a (mon.0) 3039 : cluster [DBG] 2021-07-31T02:44:05.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:04 smithi154 conmon[34553]: osdmap e2018: 8 total, 8 up, 8 in 2021-07-31T02:44:05.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:04 smithi045 conmon[39718]: cluster 2021-07-31T02:44:03.607608+0000 mon.a (mon.0) 2021-07-31T02:44:05.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:04 smithi045 conmon[39718]: 3039 : cluster [DBG] osdmap e2018: 8 total, 8 up, 8 in 2021-07-31T02:44:05.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:04 smithi045 conmon[33655]: cluster 2021-07-31T02:44:03.607608+0000 mon.a (mon.0 2021-07-31T02:44:05.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:04 smithi045 conmon[33655]: ) 3039 : cluster [DBG] osdmap e2018: 8 total, 8 up, 8 in 2021-07-31T02:44:06.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:05 smithi154 conmon[34553]: cluster 2021-07-31T02:44:04.613432+0000 mon.a (mon.0) 3040 2021-07-31T02:44:06.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:05 smithi154 conmon[34553]: : cluster [DBG] osdmap e2019: 8 total, 8 up, 8 in 2021-07-31T02:44:06.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:05 smithi154 conmon[34553]: cluster 2021-07-31T02:44:04.615556+0000 mgr.x (mgr 2021-07-31T02:44:06.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:05 smithi154 conmon[34553]: .24542) 2435 : cluster [DBG] pgmap v3571: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 494 KiB/s rd, 5.0 KiB/s wr, 671 op/s 2021-07-31T02:44:06.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:05 smithi045 conmon[39718]: cluster 2021-07-31T02:44:04.613432+0000 mon.a (mon. 2021-07-31T02:44:06.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:05 smithi045 conmon[39718]: 0) 3040 : cluster [DBG] osdmap e2019: 8 total, 8 up, 8 in 2021-07-31T02:44:06.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:05 smithi045 conmon[39718]: cluster 2021-07-31T02:44:04.615556+0000 mgr.x (mgr.24542 2021-07-31T02:44:06.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:05 smithi045 conmon[39718]: ) 2435 : cluster [DBG] pgmap v3571: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 494 KiB/s rd, 5.0 KiB/s wr, 671 op/s 2021-07-31T02:44:06.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:05 smithi045 conmon[33655]: cluster 2021-07-31T02:44:04.613432+0000 mon.a (mon.0) 3040 : 2021-07-31T02:44:06.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:05 smithi045 conmon[33655]: cluster [DBG] osdmap e2019: 8 total, 8 up, 8 in 2021-07-31T02:44:06.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:05 smithi045 conmon[33655]: cluster 2021-07-31T02:44:04.615556+0000 mgr.x (mgr.24542) 2435 : cluster [DBG] pgmap v3571: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 494 KiB/s rd, 5.0 KiB/s wr, 671 op/s 2021-07-31T02:44:07.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:06 smithi154 conmon[34553]: cluster 2021-07-31T02:44:05.619466+0000 mon.a (mon.0) 3041 : cluster [DBG] osdmap e2020: 8 total, 8 up, 8 in 2021-07-31T02:44:07.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:06 smithi045 conmon[39718]: cluster 2021-07-31T02:44:05.619466+0000 mon.a (mon 2021-07-31T02:44:07.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:06 smithi045 conmon[39718]: .0) 3041 : cluster [DBG] osdmap e2020: 8 total, 8 up, 8 in 2021-07-31T02:44:07.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:06 smithi045 conmon[33655]: cluster 2021-07-31T02:44:05.619466+0000 mon.a (mon.0) 2021-07-31T02:44:07.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:06 smithi045 conmon[33655]: 3041 : cluster [DBG] osdmap e2020: 8 total, 8 up, 8 in 2021-07-31T02:44:08.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:07 smithi154 conmon[34553]: cluster 2021-07-31T02:44:06.615979+0000 mgr.x (mgr.24542) 2436 : cluster [DBG] pgmap v3573: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 488 KiB/s rd, 4.9 KiB/s wr, 663 op/s 2021-07-31T02:44:08.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:07 smithi154 conmon[34553]: cluster 2021-07-31T02:44:06 2021-07-31T02:44:08.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:07 smithi154 conmon[34553]: .625001+0000 mon.a (mon.0) 3042 : cluster [DBG] osdmap e2021: 8 total, 8 up, 8 in 2021-07-31T02:44:08.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:07 smithi154 conmon[34553]: cluster 2021-07-31T02:44:07.574713+0000 mon.a (mon.0) 3043 : cluster [DBG] osdmap e2022: 8 total, 8 up, 8 in 2021-07-31T02:44:08.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:07 smithi045 conmon[33655]: cluster 2021-07-31T02:44:06.615979+0000 mgr.x (mgr. 2021-07-31T02:44:08.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:07 smithi045 conmon[33655]: 24542) 2436 : cluster [DBG] pgmap v3573: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 488 KiB/s rd, 4.9 KiB/s wr, 663 op/s 2021-07-31T02:44:08.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:07 smithi045 conmon[33655]: cluster 2021-07-31T02:44:06.625001+0000 mon.a (mon.0) 3042 : cluster [DBG] osdmap e2021: 8 total, 8 up, 8 in 2021-07-31T02:44:08.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:07 smithi045 conmon[33655]: cluster 2021-07-31T02 2021-07-31T02:44:08.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:07 smithi045 conmon[33655]: :44:07.574713+0000 mon.a (mon.0) 3043 : cluster [DBG] osdmap e2022: 8 total, 8 up, 8 in 2021-07-31T02:44:08.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:07 smithi045 conmon[39718]: cluster 2021-07-31T02:44:06.615979+0000 2021-07-31T02:44:08.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:07 smithi045 conmon[39718]: mgr.x (mgr.24542) 2436 : cluster [DBG] pgmap v3573: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 488 KiB/s rd, 4.9 KiB/s wr, 663 op/s 2021-07-31T02:44:08.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:07 smithi045 conmon[39718]: cluster 2021-07-31T02:44:08.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:07 smithi045 conmon[39718]: 2021-07-31T02:44:06.625001+0000 2021-07-31T02:44:08.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:07 smithi045 conmon[39718]: mon.a (mon.0) 3042 2021-07-31T02:44:08.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:07 smithi045 conmon[39718]: : cluster [DBG] osdmap e2021: 8 total, 8 up, 8 in 2021-07-31T02:44:08.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:07 smithi045 conmon[39718]: cluster 2021- 2021-07-31T02:44:08.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:07 smithi045 conmon[39718]: 07-31T02:44:07.574713 2021-07-31T02:44:08.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:07 smithi045 conmon[39718]: +0000 mon.a (mon.0) 3043 : cluster 2021-07-31T02:44:08.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:07 smithi045 conmon[39718]: [DBG] osdmap e2022: 8 total, 8 up, 8 in 2021-07-31T02:44:09.599 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:44:09.883 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:44:09.884 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 59m ago 59m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:44:09.884 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 56m ago 59m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:44:09.884 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 56m ago 63m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:44:09.884 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 59m ago 65m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:44:09.885 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 59m ago 65m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:44:09.885 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 56m ago 63m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:44:09.885 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 59m ago 63m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:44:09.885 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 59m ago 60m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:44:09.886 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 56m ago 59m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:44:09.886 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 59m ago 62m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:44:09.886 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 59m ago 62m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:44:09.886 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 59m ago 62m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:44:09.886 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 59m ago 61m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:44:09.887 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 56m ago 61m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:44:09.887 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 56m ago 61m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:44:09.887 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 56m ago 60m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:44:09.888 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 56m ago 60m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:44:09.888 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 56m ago 60m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:44:10.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:09 smithi154 conmon[34553]: cluster 2021-07-31T02:44:08.581880+0000 mon.a (mon.0) 3044 : cluster [DBG] 2021-07-31T02:44:10.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:09 smithi154 conmon[34553]: osdmap e2023: 8 total, 8 up, 8 in 2021-07-31T02:44:10.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:09 smithi154 conmon[34553]: cluster 2021-07-31T02:44:08.616345+0000 mgr.x (mgr.24542) 2437 : cluster [DBG] pgmap v3577: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:44:10.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:09 smithi045 conmon[33655]: cluster 2021-07-31T02:44:08.581880+0000 mon.a 2021-07-31T02:44:10.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:09 smithi045 conmon[33655]: (mon.0) 3044 : cluster [DBG] osdmap e2023: 8 total, 8 up, 8 in 2021-07-31T02:44:10.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:09 smithi045 conmon[33655]: cluster 2021-07-31T02:44:08.616345+0000 mgr.x (mgr.24542) 2437 : cluster [DBG] pgmap v3577: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:44:10.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:09 smithi045 conmon[39718]: cluster 2021-07-31T02: 2021-07-31T02:44:10.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:09 smithi045 conmon[39718]: 44:08.581880+0000 mon.a (mon.0) 3044 : cluster [DBG] osdmap e2023: 8 total, 8 up, 8 in 2021-07-31T02:44:10.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:09 smithi045 conmon[39718]: cluster 2021-07-31T02:44:10.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:09 smithi045 conmon[39718]: 2021-07-31T 2021-07-31T02:44:10.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:09 smithi045 conmon[39718]: 02:44:08.616345+0000 mgr.x (mgr. 2021-07-31T02:44:10.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:09 smithi045 conmon[39718]: 24542) 2437 : cluster [DBG] pgmap v3577: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:44:10.218 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:44:10.219 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:44:10.219 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:44:10.219 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:44:10.220 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:44:10.220 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:44:10.220 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:44:10.220 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:44:10.220 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:44:10.221 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:44:10.221 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:44:10.221 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:44:10.221 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:44:10.221 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:44:10.222 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:44:10.222 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:44:10.222 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:44:11.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:10 smithi154 conmon[34553]: audit 2021-07-31T02:44:09.585972+0000 mgr.x (mgr.24542) 2438 : audit [DBG] from='client.27892 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:44:11.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:10 smithi154 conmon[34553]: 2021-07-31T02:44:11.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:10 smithi154 conmon[34553]: cluster 2021-07-31T02:44:09.586329+0000 mon.a (mon.0) 3045 : cluster [DBG] osdmap e2024: 8 total, 8 up, 8 in 2021-07-31T02:44:11.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:10 smithi154 conmon[34553]: audit 2021-07-31T02:44:09.875807+0000 mgr.x ( 2021-07-31T02:44:11.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:10 smithi154 conmon[34553]: mgr.24542) 2439 : audit [DBG] from='client.17958 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:44:11.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:10 smithi154 conmon[34553]: audit 2021-07-31T02:44:10.218765+0000 2021-07-31T02:44:11.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:10 smithi154 conmon[34553]: mon.c (mon.1) 130 : audit [DBG] from='client.? 172.21.15.45:0/2517933844' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:44:11.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:10 smithi045 conmon[33655]: audit 2021-07-31T02:44:09.585972+0000 mgr.x (mgr.24542 2021-07-31T02:44:11.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:10 smithi045 conmon[33655]: ) 2438 : audit [DBG] from='client.27892 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:44:11.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:10 smithi045 conmon[33655]: cluster 2021-07-31T02:44:09.586329+0000 mon.a (mon. 2021-07-31T02:44:11.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:10 smithi045 conmon[33655]: 0) 3045 : cluster [DBG] osdmap e2024: 8 total, 8 up, 8 in 2021-07-31T02:44:11.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:10 smithi045 conmon[33655]: audit 2021-07-31T02:44:11.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:10 smithi045 conmon[33655]: 2021-07-31T02:44:09 2021-07-31T02:44:11.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:10 smithi045 conmon[33655]: .875807+0000 mgr.x (mgr.24542) 2439 2021-07-31T02:44:11.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:10 smithi045 conmon[33655]: : audit [DBG] from='client.17958 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:44:11.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:10 smithi045 conmon[33655]: audit 2021 2021-07-31T02:44:11.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:10 smithi045 conmon[33655]: -07-31T02:44:10. 2021-07-31T02:44:11.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:10 smithi045 conmon[33655]: 218765+0000 mon.c (mon.1) 130 : 2021-07-31T02:44:11.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:10 smithi045 conmon[33655]: audit [DBG] from='client.? 172.21.15.45:0/2517933844' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:44:11.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:10 smithi045 conmon[39718]: audit 2021-07-31T02:44:09 2021-07-31T02:44:11.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:10 smithi045 conmon[39718]: .585972+0000 mgr.x (mgr.24542) 2438 : audit [DBG] from='client.27892 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:44:11.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:10 smithi045 conmon[39718]: cluster 2021- 2021-07-31T02:44:11.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:10 smithi045 conmon[39718]: 07-31T02:44:09.586329 2021-07-31T02:44:11.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:10 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2021-07-31T02:44:11.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:10 smithi045 conmon[39718]: 3045 : cluster [DBG] osdmap e2024: 8 total, 8 up, 8 in 2021-07-31T02:44:11.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:10 smithi045 conmon[39718]: audit 2021 2021-07-31T02:44:11.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:10 smithi045 conmon[39718]: -07-31T02:44:09 2021-07-31T02:44:11.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:10 smithi045 conmon[39718]: .875807+0000 mgr.x (mgr. 2021-07-31T02:44:11.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:10 smithi045 conmon[39718]: 24542) 2439 : audit [DBG] 2021-07-31T02:44:11.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:10 smithi045 conmon[39718]: from='client.17958 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:44:11.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:10 smithi045 conmon[39718]: audit 2021 2021-07-31T02:44:11.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:10 smithi045 conmon[39718]: -07-31T02:44:10 2021-07-31T02:44:11.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:10 smithi045 conmon[39718]: .218765+0000 mon.c (mon. 2021-07-31T02:44:11.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:10 smithi045 conmon[39718]: 1) 130 : audit [DBG] 2021-07-31T02:44:11.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:10 smithi045 conmon[39718]: from='client.? 172.21.15.45:0/2517933844' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:44:12.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:11 smithi154 conmon[34553]: cluster 2021-07-31T02:44:10.591783+0000 mon.a (mon.0) 3046 : cluster [DBG] osdmap e2025: 8 total, 8 up, 8 in 2021-07-31T02:44:12.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:11 smithi154 conmon[34553]: 2021-07-31T02:44:12.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:11 smithi154 conmon[34553]: cluster 2021-07-31T02:44:10.616798+0000 mgr.x (mgr.24542) 2440 : cluster [DBG] pgmap v3580: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 463 KiB/s rd, 5.0 KiB/s wr, 628 op/s 2021-07-31T02:44:12.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:11 smithi045 conmon[39718]: cluster 2021-07-31T02:44:10.591783+0000 mon.a (mon 2021-07-31T02:44:12.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:11 smithi045 conmon[39718]: .0) 3046 : cluster [DBG] osdmap e2025: 8 total, 8 up, 8 in 2021-07-31T02:44:12.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:11 smithi045 conmon[39718]: cluster 2021-07-31T02:44:10.616798+0000 mgr.x 2021-07-31T02:44:12.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:11 smithi045 conmon[39718]: (mgr.24542) 2440 : cluster [DBG] pgmap v3580: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 463 KiB/s rd, 5.0 KiB/s wr, 628 op/s 2021-07-31T02:44:12.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:11 smithi045 conmon[33655]: cluster 2021-07-31T02:44:10.591783+0000 mon.a (mon.0) 3046 : cluster [DBG] 2021-07-31T02:44:12.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:11 smithi045 conmon[33655]: osdmap e2025: 8 total, 8 up, 8 in 2021-07-31T02:44:12.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:11 smithi045 conmon[33655]: cluster 2021-07-31T02:44:10.616798+0000 mgr.x (mgr.24542) 2440 : cluster [DBG] pgmap v3580: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 463 KiB/s rd, 5.0 KiB/s wr, 628 op/s 2021-07-31T02:44:13.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:12 smithi154 conmon[34553]: cluster 2021-07-31T02:44:11.598500+0000 mon.a 2021-07-31T02:44:13.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:12 smithi154 conmon[34553]: (mon.0) 3047 : cluster [DBG] osdmap e2026: 8 total, 8 up, 8 in 2021-07-31T02:44:13.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:12 smithi154 conmon[34553]: cluster 2021-07-31T02:44:12.576828+0000 mon.a (mon.0) 3048 : cluster [DBG] osdmap e2027: 8 total, 8 up, 8 in 2021-07-31T02:44:13.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:12 smithi045 conmon[33655]: cluster 2021-07-31T02:44:11.598500+0000 mon.a ( 2021-07-31T02:44:13.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:12 smithi045 conmon[33655]: mon.0) 3047 : cluster [DBG] osdmap e2026: 8 total, 8 up, 8 in 2021-07-31T02:44:13.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:12 smithi045 conmon[33655]: cluster 2021-07-31T02:44:12.576828+0000 mon.a (mon.0) 3048 : cluster [DBG] osdmap e2027: 8 total, 8 up, 8 in 2021-07-31T02:44:13.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:12 smithi045 conmon[39718]: cluster 2021-07-31T02:44:11.598500+0000 mon.a (mon.0) 3047 2021-07-31T02:44:13.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:12 smithi045 conmon[39718]: : cluster [DBG] osdmap e2026: 8 total, 8 up, 8 in 2021-07-31T02:44:13.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:12 smithi045 conmon[39718]: cluster 2021-07-31T02:44:12.576828+0000 mon.a (mon.0) 3048 : cluster [DBG] osdmap e2027: 8 total, 8 up, 8 in 2021-07-31T02:44:14.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:13 smithi154 conmon[34553]: cluster 2021-07-31T02:44:12.617175+0000 mgr.x (mgr.24542) 2441 : cluster [DBG] pgmap v3583: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 462 KiB/s rd, 5.0 KiB/s wr, 626 op/s 2021-07-31T02:44:14.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:13 smithi045 conmon[33655]: cluster 2021-07-31T02:44:12.617175+0000 mgr.x (mgr.24542) 2441 : 2021-07-31T02:44:14.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:13 smithi045 conmon[33655]: cluster [DBG] pgmap v3583: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 462 KiB/s rd, 5.0 KiB/s wr, 626 op/s 2021-07-31T02:44:14.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:13 smithi045 conmon[39718]: cluster 2021-07-31T02:44:12.617175+0000 mgr.x (mgr. 2021-07-31T02:44:14.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:13 smithi045 conmon[39718]: 24542) 2441 : cluster [DBG] pgmap v3583: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 462 KiB/s rd, 5.0 KiB/s wr, 626 op/s 2021-07-31T02:44:15.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:14 smithi154 conmon[34553]: cluster 2021-07-31T02:44:13.605451+0000 mon.a (mon.0) 3049 : cluster [DBG] osdmap e2028: 8 total, 8 up, 8 in 2021-07-31T02:44:15.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:14 smithi045 conmon[33655]: cluster 2021-07-31T02:44:13.605451+0000 mon.a (mon.0) 3049 : cluster 2021-07-31T02:44:15.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:14 smithi045 conmon[33655]: [DBG] osdmap e2028: 8 total, 8 up, 8 in 2021-07-31T02:44:15.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:14 smithi045 conmon[39718]: cluster 2021-07-31T02:44:13.605451+0000 mon.a (mon 2021-07-31T02:44:15.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:14 smithi045 conmon[39718]: .0) 3049 : cluster [DBG] osdmap e2028: 8 total, 8 up, 8 in 2021-07-31T02:44:16.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:15 smithi154 conmon[34553]: cluster 2021-07-31T02:44:14.610409+0000 mon.a (mon.0) 3050 : cluster [DBG] 2021-07-31T02:44:16.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:15 smithi154 conmon[34553]: osdmap e2029: 8 total, 8 up, 8 in 2021-07-31T02:44:16.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:15 smithi154 conmon[34553]: cluster 2021-07-31T02:44:14.617607+0000 mgr.x (mgr.24542) 2442 : cluster [DBG] pgmap v3586: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 427 KiB/s rd, 5.0 KiB/s wr, 579 op/s 2021-07-31T02:44:16.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:15 smithi045 conmon[33655]: cluster 2021-07-31T02:44:14.610409+0000 2021-07-31T02:44:16.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:15 smithi045 conmon[33655]: mon.a (mon.0) 3050 : cluster [DBG] osdmap e2029: 8 total, 8 up, 8 in 2021-07-31T02:44:16.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:15 smithi045 conmon[33655]: cluster 2021-07-31T02:44:14.617607+0000 mgr.x (mgr.24542) 2442 : cluster [DBG] pgmap v3586: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 427 KiB/s rd, 5.0 KiB/s wr, 579 op/s 2021-07-31T02:44:16.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:15 smithi045 conmon[39718]: cluster 2021-07-31T02:44:14.610409 2021-07-31T02:44:16.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:15 smithi045 conmon[39718]: +0000 mon.a (mon.0) 3050 : cluster [DBG] osdmap e2029: 8 total, 8 up, 8 in 2021-07-31T02:44:16.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:15 smithi045 conmon[39718]: cluster 2021-07-31T02:44:14. 2021-07-31T02:44:16.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:15 smithi045 conmon[39718]: 617607+0000 mgr.x (mgr.24542) 2442 : cluster [DBG] pgmap v3586: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 427 KiB/s rd, 5.0 KiB/s wr, 579 op/s 2021-07-31T02:44:16.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:15 smithi045 conmon[39718]: 2021-07-31T02:44:17.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:16 smithi154 conmon[34553]: cluster 2021-07-31T02:44:15.617852+0000 mon.a (mon.0) 3051 : cluster [DBG] osdmap e2030: 8 total, 8 up, 8 in 2021-07-31T02:44:17.158 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:16 smithi045 conmon[39718]: cluster 2021-07-31T02:44:15.617852+0000 mon.a ( 2021-07-31T02:44:17.159 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:16 smithi045 conmon[39718]: mon.0) 3051 : cluster [DBG] osdmap e2030: 8 total, 8 up, 8 in 2021-07-31T02:44:17.159 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:16 smithi045 conmon[33655]: cluster 2021-07-31T02:44:15.617852+0000 mon.a 2021-07-31T02:44:17.160 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:16 smithi045 conmon[33655]: (mon.0) 3051 : cluster [DBG] osdmap e2030: 8 total, 8 up, 8 in 2021-07-31T02:44:18.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:17 smithi154 conmon[34553]: cluster 2021-07-31T02:44:16.618001+0000 mgr.x (mgr.24542) 2443 : 2021-07-31T02:44:18.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:17 smithi154 conmon[34553]: cluster [DBG] pgmap v3588: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 423 KiB/s rd, 4.9 KiB/s wr, 573 op/s 2021-07-31T02:44:18.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:17 smithi154 conmon[34553]: cluster 2021-07-31T02:44:16.622811+0000 mon.a (mon.0) 3052 : cluster [DBG] osdmap e2031: 8 total, 8 up, 8 in 2021-07-31T02:44:18.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:17 smithi154 conmon[34553]: cluster 2021- 2021-07-31T02:44:18.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:17 smithi154 conmon[34553]: 07-31T02:44:17.578907+0000 mon.a (mon.0) 3053 : cluster [DBG] osdmap e2032: 8 total, 8 up, 8 in 2021-07-31T02:44:18.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:17 smithi045 conmon[33655]: cluster 2021-07-31T02:44: 2021-07-31T02:44:18.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:17 smithi045 conmon[33655]: 16.618001+0000 mgr.x (mgr.24542) 2443 : cluster [DBG] pgmap v3588: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 423 KiB/s rd, 4.9 KiB/s wr, 573 op/s 2021-07-31T02:44:18.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:17 smithi045 conmon[33655]: cluster 2021-07-31T02:44:16.622811+0000 mon.a (mon.0) 3052 : cluster [DBG] osdmap e2031: 8 total, 8 up, 8 in 2021-07-31T02:44:18.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:17 smithi045 conmon[33655]: cluster 2021-07-31T02:44:17.578907+0000 mon.a (mon.0 2021-07-31T02:44:18.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:17 smithi045 conmon[33655]: ) 3053 : cluster [DBG] osdmap e2032: 8 total, 8 up, 8 in 2021-07-31T02:44:18.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:17 smithi045 conmon[39718]: cluster 2021-07-31T02:44:16.618001+0000 2021-07-31T02:44:18.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:17 smithi045 conmon[39718]: mgr.x (mgr.24542) 2443 : cluster [DBG] pgmap v3588: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 423 KiB/s rd, 4.9 KiB/s wr, 573 op/s 2021-07-31T02:44:18.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:17 smithi045 conmon[39718]: cluster 2021-07-31T02:44:16.622811+0000 2021-07-31T02:44:18.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:17 smithi045 conmon[39718]: mon.a (mon.0) 3052 : cluster [DBG] osdmap e2031: 8 total, 8 up, 8 in 2021-07-31T02:44:18.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:17 smithi045 conmon[39718]: cluster 2021-07-31T 2021-07-31T02:44:18.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:17 smithi045 conmon[39718]: 02:44:17.578907+0000 mon.a (mon 2021-07-31T02:44:18.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:17 smithi045 conmon[39718]: .0) 3053 : cluster [DBG] osdmap e2032: 8 total, 8 up, 8 in 2021-07-31T02:44:20.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:19 smithi154 conmon[34553]: cluster 2021-07-31T02:44:20.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:19 smithi154 conmon[34553]: 2021-07-31T02:44:20.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:19 smithi154 conmon[34553]: 2021-07-31T02:44:18.586473+0000 mon.a ( 2021-07-31T02:44:20.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:19 smithi154 conmon[34553]: mon.0) 3054 : cluster [DBG] osdmap e2033: 8 total, 8 up, 8 in 2021-07-31T02:44:20.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:19 smithi154 conmon[34553]: cluster 2021-07-31T02:44:20.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:19 smithi154 conmon[34553]: 2021-07-31T02:44:18.618343 2021-07-31T02:44:20.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:19 smithi154 conmon[34553]: +0000 mgr.x (mgr.24542) 2444 : cluster [DBG] pgmap v3592: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:44:20.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:19 smithi045 conmon[33655]: cluster 2021-07-31T02:44:18.586473+0000 mon.a (mon 2021-07-31T02:44:20.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:19 smithi045 conmon[33655]: .0) 3054 : cluster [DBG] osdmap e2033: 8 total, 8 up, 8 in 2021-07-31T02:44:20.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:19 smithi045 conmon[33655]: cluster 2021-07-31T02:44:18.618343+0000 mgr.x (mgr.24542) 2444 : cluster [DBG] 2021-07-31T02:44:20.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:19 smithi045 conmon[33655]: pgmap v3592: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:44:20.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:19 smithi045 conmon[39718]: cluster 2021-07-31T02:44:18.586473+0000 mon.a ( 2021-07-31T02:44:20.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:19 smithi045 conmon[39718]: mon.0) 3054 : cluster [DBG] osdmap e2033: 8 total, 8 up, 8 in 2021-07-31T02:44:20.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:19 smithi045 conmon[39718]: cluster 2021-07-31T02:44:18.618343+0000 mgr.x (mgr.24542) 2444 : cluster [DBG] pgmap v3592: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:44:21.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:20 smithi154 conmon[34553]: cluster 2021-07-31T02:44:19.588286+0000 mon.a (mon.0) 3055 : cluster [DBG] osdmap e2034: 8 total, 8 up, 8 in 2021-07-31T02:44:21.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:20 smithi045 conmon[33655]: cluster 2021-07-31T02:44:19.588286+0000 mon.a (mon.0) 3055 : cluster [DBG] osdmap e2034: 8 total, 8 up, 8 in 2021-07-31T02:44:21.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:20 smithi045 conmon[39718]: cluster 2021-07-31T02:44:19.588286+0000 mon.a (mon.0) 2021-07-31T02:44:21.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:20 smithi045 conmon[39718]: 3055 : cluster [DBG] osdmap e2034: 8 total, 8 up, 8 in 2021-07-31T02:44:22.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:21 smithi154 conmon[34553]: cluster 2021-07-31T02:44:20.601462+0000 mon.a (mon.0) 3056 : cluster 2021-07-31T02:44:22.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:21 smithi154 conmon[34553]: [DBG] osdmap e2035: 8 total, 8 up, 8 in 2021-07-31T02:44:22.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:21 smithi154 conmon[34553]: cluster 2021-07-31T02:44:20.618770+0000 mgr.x (mgr.24542) 2445 : cluster [DBG] pgmap v3595: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 396 KiB/s rd, 5.0 KiB/s wr, 536 op/s 2021-07-31T02:44:22.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:21 smithi045 conmon[33655]: cluster 2021-07-31T02:44:20.601462+0000 mon.a (mon.0) 3056 : cluster 2021-07-31T02:44:22.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:21 smithi045 conmon[33655]: [DBG] osdmap e2035: 8 total, 8 up, 8 in 2021-07-31T02:44:22.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:21 smithi045 conmon[33655]: cluster 2021-07-31T02:44:20.618770+0000 mgr.x (mgr.24542) 2445 : cluster [DBG] pgmap v3595: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 396 KiB/s rd, 5.0 KiB/s wr, 536 op/s 2021-07-31T02:44:22.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:21 smithi045 conmon[39718]: cluster 2021-07-31T02:44:20.601462+0000 mon.a 2021-07-31T02:44:22.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:21 smithi045 conmon[39718]: (mon.0) 3056 : cluster [DBG] osdmap e2035: 8 total, 8 up, 8 in 2021-07-31T02:44:22.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:21 smithi045 conmon[39718]: cluster 2021-07- 2021-07-31T02:44:22.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:21 smithi045 conmon[39718]: 31T02:44:20.618770+0000 mgr.x (mgr.24542) 2445 : cluster [DBG] pgmap v3595: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 396 KiB/s rd, 5.0 KiB/s wr, 536 op/s 2021-07-31T02:44:23.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:22 smithi154 conmon[34553]: cluster 2021-07-31T02:44:21.608022+0000 mon.a ( 2021-07-31T02:44:23.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:22 smithi154 conmon[34553]: mon.0) 3057 : cluster [DBG] osdmap e2036: 8 total, 8 up, 8 in 2021-07-31T02:44:23.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:22 smithi154 conmon[34553]: cluster 2021-07-31T02:44:22.580152+0000 mon.a (mon.0) 3058 : cluster [DBG] osdmap e2037: 8 total, 8 up, 8 in 2021-07-31T02:44:23.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:22 smithi045 conmon[33655]: cluster 2021-07-31T02:44:21.608022+0000 mon.a (mon.0) 3057 : cluster [DBG] 2021-07-31T02:44:23.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:22 smithi045 conmon[33655]: osdmap e2036: 8 total, 8 up, 8 in 2021-07-31T02:44:23.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:22 smithi045 conmon[33655]: cluster 2021-07-31T02:44:22.580152+0000 mon.a (mon.0) 3058 : cluster [DBG] osdmap e2037: 8 total, 8 up, 8 in 2021-07-31T02:44:23.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:22 smithi045 conmon[39718]: cluster 2021-07-31T02:44:21.608022+0000 mon.a (mon 2021-07-31T02:44:23.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:22 smithi045 conmon[39718]: .0) 3057 : cluster [DBG] osdmap e2036: 8 total, 8 up, 8 in 2021-07-31T02:44:23.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:22 smithi045 conmon[39718]: cluster 2021-07-31T02:44:22.580152+0000 mon.a (mon.0) 3058 2021-07-31T02:44:23.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:22 smithi045 conmon[39718]: : cluster [DBG] osdmap e2037: 8 total, 8 up, 8 in 2021-07-31T02:44:24.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:23 smithi154 conmon[34553]: cluster 2021-07-31T02:44:24.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:23 smithi154 conmon[34553]: 2021-07-31T02:44:22.619154+0000 2021-07-31T02:44:24.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:23 smithi154 conmon[34553]: mgr.x (mgr.24542) 2446 : cluster [DBG] pgmap v3598: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 396 KiB/s rd, 5.0 KiB/s wr, 536 op/s 2021-07-31T02:44:24.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:23 smithi045 conmon[39718]: cluster 2021-07-31T02:44:22.619154 2021-07-31T02:44:24.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:23 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 2446 : cluster [DBG] pgmap v3598: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 396 KiB/s rd, 5.0 KiB/s wr, 536 op/s 2021-07-31T02:44:24.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:23 smithi045 conmon[33655]: cluster 2021-07-31T02:44:22.619154+0000 mgr.x (mgr.24542) 2446 : cluster [DBG] 2021-07-31T02:44:24.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:23 smithi045 conmon[33655]: pgmap v3598: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 396 KiB/s rd, 5.0 KiB/s wr, 536 op/s 2021-07-31T02:44:25.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:24 smithi154 conmon[34553]: cluster 2021-07-31T02:44:23.612439+0000 mon.a (mon.0) 3059 : cluster [DBG] osdmap e2038: 8 total, 8 up, 8 in 2021-07-31T02:44:25.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:24 smithi045 conmon[39718]: cluster 2021-07-31T02:44:23.612439+0000 mon.a (mon 2021-07-31T02:44:25.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:24 smithi045 conmon[39718]: .0) 3059 : cluster [DBG] osdmap e2038: 8 total, 8 up, 8 in 2021-07-31T02:44:25.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:24 smithi045 conmon[33655]: cluster 2021-07-31T02:44 2021-07-31T02:44:25.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:24 smithi045 conmon[33655]: :23.612439+0000 mon.a (mon.0) 3059 : cluster [DBG] osdmap e2038: 8 total, 8 up, 8 in 2021-07-31T02:44:26.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:25 smithi154 conmon[34553]: cluster 2021-07-31T02:44:24.619732+0000 mgr.x (mgr.24542) 2447 : cluster [DBG] 2021-07-31T02:44:26.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:25 smithi154 conmon[34553]: pgmap v3600: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 361 KiB/s rd, 5.0 KiB/s wr, 490 op/s 2021-07-31T02:44:26.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:25 smithi154 conmon[34553]: cluster 2021-07-31T02:44:24.623379+0000 mon.a (mon.0) 3060 : cluster [DBG] osdmap e2039: 8 total, 8 up, 8 in 2021-07-31T02:44:26.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:25 smithi045 conmon[39718]: cluster 2021-07-31T02:44:24.619732+0000 2021-07-31T02:44:26.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:25 smithi045 conmon[39718]: mgr.x (mgr.24542) 2447 : cluster [DBG] pgmap v3600: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 361 KiB/s rd, 5.0 KiB/s wr, 490 op/s 2021-07-31T02:44:26.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:25 smithi045 conmon[39718]: cluster 2021-07-31T 2021-07-31T02:44:26.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:25 smithi045 conmon[39718]: 02:44:24.623379+0000 mon.a (mon.0) 3060 : cluster [DBG] 2021-07-31T02:44:26.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:25 smithi045 conmon[39718]: osdmap e2039: 8 total, 8 up, 8 in 2021-07-31T02:44:26.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:25 smithi045 conmon[33655]: cluster 2021-07-31T02:44:24.619732+0000 mgr.x (mgr.24542) 2447 2021-07-31T02:44:26.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:25 smithi045 conmon[33655]: : cluster [DBG] pgmap v3600: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 361 KiB/s rd, 5.0 KiB/s wr, 490 op/s 2021-07-31T02:44:26.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:25 smithi045 conmon[33655]: cluster 2021-07-31T02:44:24.623379+0000 mon.a (mon.0) 3060 : cluster [DBG] osdmap e2039: 8 total, 8 up, 8 in 2021-07-31T02:44:27.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:26 smithi154 conmon[34553]: cluster 2021-07-31T02:44:25.629893+0000 mon.a (mon.0) 3061 2021-07-31T02:44:27.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:26 smithi154 conmon[34553]: : cluster [DBG] osdmap e2040: 8 total, 8 up, 8 in 2021-07-31T02:44:27.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:26 smithi045 conmon[39718]: cluster 2021-07-31T02:44:25.629893+0000 mon.a (mon.0) 3061 : cluster 2021-07-31T02:44:27.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:26 smithi045 conmon[39718]: [DBG] osdmap e2040: 8 total, 8 up, 8 in 2021-07-31T02:44:27.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:26 smithi045 conmon[33655]: cluster 2021-07-31T02:44:25.629893+0000 mon.a (mon.0) 3061 : 2021-07-31T02:44:27.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:26 smithi045 conmon[33655]: cluster [DBG] osdmap e2040: 8 total, 8 up, 8 in 2021-07-31T02:44:28.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:27 smithi154 conmon[34553]: cluster 2021-07-31T02:44:26.620315+0000 mgr.x (mgr.24542) 2448 : cluster 2021-07-31T02:44:28.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:27 smithi154 conmon[34553]: [DBG] pgmap v3603: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 359 KiB/s rd, 4.9 KiB/s wr, 487 op/s 2021-07-31T02:44:28.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:27 smithi154 conmon[34553]: cluster 2021-07-31T02:44:26.635091+0000 mon.a (mon.0) 3062 : cluster [DBG] osdmap e2041: 8 total, 8 up, 8 in 2021-07-31T02:44:28.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:27 smithi154 conmon[34553]: cluster 2021-07-31 2021-07-31T02:44:28.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:27 smithi154 conmon[34553]: T02:44:27.580536+0000 mon.a (mon.0) 3063 : cluster [DBG] osdmap e2042: 8 total, 8 up, 8 in 2021-07-31T02:44:28.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:27 smithi045 conmon[39718]: cluster 2021-07-31T02:44: 2021-07-31T02:44:28.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:27 smithi045 conmon[39718]: 26.620315+0000 mgr.x (mgr.24542) 2448 : cluster [DBG] pgmap v3603: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 359 KiB/s rd, 4.9 KiB/s wr, 487 op/s 2021-07-31T02:44:28.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:27 smithi045 conmon[39718]: cluster 2021 2021-07-31T02:44:28.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:27 smithi045 conmon[39718]: -07-31T02:44:26.635091 2021-07-31T02:44:28.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:27 smithi045 conmon[39718]: +0000 mon.a (mon.0 2021-07-31T02:44:28.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:27 smithi045 conmon[39718]: ) 3062 : cluster [DBG] osdmap e2041: 8 total, 8 up, 8 in 2021-07-31T02:44:28.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:27 smithi045 conmon[39718]: 2021-07-31T02:44:28.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:27 smithi045 conmon[39718]: cluster 2021 2021-07-31T02:44:28.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:27 smithi045 conmon[39718]: -07-31T02:44:27 2021-07-31T02:44:28.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:27 smithi045 conmon[39718]: .580536+0000 mon.a (mon. 2021-07-31T02:44:28.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:27 smithi045 conmon[39718]: 0) 3063 : cluster [DBG] osdmap e2042: 8 total, 8 up, 8 in 2021-07-31T02:44:28.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:27 smithi045 conmon[39718]: 2021-07-31T02:44:28.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:27 smithi045 conmon[33655]: cluster 2021-07-31T02:44:26.620315+0000 mgr.x (mgr.24542) 2021-07-31T02:44:28.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:27 smithi045 conmon[33655]: 2448 : cluster [DBG] pgmap v3603: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 359 KiB/s rd, 4.9 KiB/s wr, 487 op/s 2021-07-31T02:44:28.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:27 smithi045 conmon[33655]: cluster 2021-07-31T02:44:26.635091+0000 mon.a (mon.0) 3062 : cluster [DBG] osdmap e2041: 8 total, 8 up, 8 in 2021-07-31T02:44:28.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:27 smithi045 conmon[33655]: cluster 2021-07-31T02: 2021-07-31T02:44:28.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:27 smithi045 conmon[33655]: 44:27.580536+0000 mon.a (mon.0) 3063 : cluster [DBG] osdmap e2042: 8 total, 8 up, 8 in 2021-07-31T02:44:28.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:27 smithi045 conmon[33655]: 2021-07-31T02:44:29.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:28 smithi154 conmon[34553]: audit 2021-07-31T02:44:28.158581+0000 mon.b (mon.2) 360 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:44:29.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:28 smithi154 conmon[34553]: audit 2021-07-31T02:44:28.159261+0000 mon.a (mon.0) 3064 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:44:29.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:28 smithi154 conmon[34553]: audit 2021-07-31T02:44:28.159585+0000 mon.b (mon.2) 361 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:44:29.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:28 smithi154 conmon[34553]: audit 2021-07-31T02:44:28 2021-07-31T02:44:29.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:28 smithi154 conmon[34553]: .159848+0000 mon.a (mon.0) 3065 : audit [INF] 2021-07-31T02:44:29.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:28 smithi154 conmon[34553]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:44:29.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:28 smithi154 conmon[34553]: cluster 2021-07-31T02:44: 2021-07-31T02:44:29.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:28 smithi154 conmon[34553]: 28.587236+0000 mon.a (mon.0) 3066 : 2021-07-31T02:44:29.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:28 smithi154 conmon[34553]: cluster [DBG] osdmap e2043: 8 total, 8 up, 8 in 2021-07-31T02:44:29.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:28 smithi045 conmon[39718]: audit 2021-07-31T02:44:28.158581+0000 2021-07-31T02:44:29.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:28 smithi045 conmon[39718]: mon.b (mon.2) 360 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:44:29.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:28 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T02:44:29.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:28 smithi045 conmon[39718]: T02:44:28.159261+0000 mon.a (mon.0) 3064 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:44:29.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:28 smithi045 conmon[39718]: audit 2021-07-31T02 2021-07-31T02:44:29.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:28 smithi045 conmon[39718]: :44:28.159585+0000 mon.b (mon. 2021-07-31T02:44:29.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:28 smithi045 conmon[39718]: 2) 361 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:44:29.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:28 smithi045 conmon[39718]: audit 2021 2021-07-31T02:44:29.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:28 smithi045 conmon[39718]: -07-31T02:44:28.159848 2021-07-31T02:44:29.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:28 smithi045 conmon[39718]: +0000 mon.a (mon.0) 3065 : audit 2021-07-31T02:44:29.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:28 smithi045 conmon[39718]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:44:29.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:28 smithi045 conmon[39718]: cluster 2021-07-31T02 2021-07-31T02:44:29.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:28 smithi045 conmon[39718]: :44:28.587236+0000 mon.a (mon 2021-07-31T02:44:29.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:28 smithi045 conmon[39718]: .0) 3066 : cluster [DBG] osdmap e2043: 8 total, 8 up, 8 in 2021-07-31T02:44:29.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:28 smithi045 conmon[33655]: audit 2021-07-31T02:44:28.158581+0000 mon.b (mon.2) 360 : audit 2021-07-31T02:44:29.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:28 smithi045 conmon[33655]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:44:29.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:28 smithi045 conmon[33655]: audit 2021-07-31T02:44:28.159261+0000 mon.a (mon.0) 3064 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:44:29.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:28 smithi045 conmon[33655]: audit 2021-07-31T02:44:28.159585+0000 mon.b (mon.2) 361 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:44:29.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:28 smithi045 conmon[33655]: audit 2021-07-31T02:44: 2021-07-31T02:44:29.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:28 smithi045 conmon[33655]: 28.159848+0000 mon.a (mon.0) 3065 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:44:29.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:28 smithi045 conmon[33655]: cluster 2021-07-31T 2021-07-31T02:44:29.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:28 smithi045 conmon[33655]: 02:44:28.587236+0000 mon.a (mon.0) 3066 : cluster [DBG] 2021-07-31T02:44:29.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:28 smithi045 conmon[33655]: osdmap e2043: 8 total, 8 up, 8 in 2021-07-31T02:44:30.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:29 smithi154 conmon[34553]: cluster 2021-07-31T02:44:28.620938+0000 mgr.x (mgr.24542) 2449 : cluster [DBG] pgmap v3607: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:44:30.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:29 smithi154 conmon[34553]: cluster 2021-07-31T02:44:29.591997+0000 mon.a (mon.0) 3067 : cluster [DBG] osdmap e2044: 8 total, 8 up, 8 in 2021-07-31T02:44:30.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:29 smithi045 conmon[39718]: cluster 2021-07-31T02:44:28.620938+0000 2021-07-31T02:44:30.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:29 smithi045 conmon[39718]: mgr.x (mgr.24542) 2449 : cluster [DBG] pgmap v3607: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:44:30.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:29 smithi045 conmon[39718]: cluster 2021-07 2021-07-31T02:44:30.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:29 smithi045 conmon[39718]: -31T02:44:29.591997+0000 mon.a (mon.0) 3067 : cluster [DBG] 2021-07-31T02:44:30.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:29 smithi045 conmon[39718]: osdmap e2044: 8 total, 8 up, 8 in 2021-07-31T02:44:30.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:29 smithi045 conmon[33655]: cluster 2021-07-31T02:44:28.620938+0000 2021-07-31T02:44:30.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:29 smithi045 conmon[33655]: mgr.x (mgr.24542) 2449 : cluster [DBG] pgmap v3607: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:44:30.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:29 smithi045 conmon[33655]: cluster 2021-07-31T02:44:29.591997+0000 mon.a (mon.0) 3067 : cluster [DBG] osdmap e2044: 8 total, 8 up, 8 in 2021-07-31T02:44:32.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:31 smithi154 conmon[34553]: cluster 2021-07-31T02:44:30.594133+0000 mon.a (mon 2021-07-31T02:44:32.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:31 smithi154 conmon[34553]: .0) 3068 : cluster [DBG] osdmap e2045: 8 total, 8 up, 8 in 2021-07-31T02:44:32.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:31 smithi154 conmon[34553]: cluster 2021-07-31T02:44:30.621316+0000 mgr.x (mgr.24542) 2450 : cluster [DBG] pgmap v3610: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 332 KiB/s rd, 5.0 KiB/s wr, 450 op/s 2021-07-31T02:44:32.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:31 smithi045 conmon[33655]: cluster 2021-07-31T02:44:30.594133 2021-07-31T02:44:32.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:31 smithi045 conmon[33655]: +0000 mon.a (mon.0) 3068 : cluster [DBG] osdmap e2045: 8 total, 8 up, 8 in 2021-07-31T02:44:32.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:31 smithi045 conmon[33655]: cluster 2021-07-31T02:44:30.621316+0000 mgr.x (mgr.24542) 2450 : cluster [DBG] pgmap v3610: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 332 KiB/s rd, 5.0 KiB/s wr, 450 op/s 2021-07-31T02:44:32.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:31 smithi045 conmon[39718]: cluster 2021-07-31T02:44:32.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:31 smithi045 conmon[39718]: 2021-07-31T02:44 2021-07-31T02:44:32.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:31 smithi045 conmon[39718]: :30.594133+0000 mon.a (mon.0) 3068 : cluster [DBG] osdmap e2045: 8 total, 8 up, 8 in 2021-07-31T02:44:32.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:31 smithi045 conmon[39718]: cluster 2021 2021-07-31T02:44:32.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:31 smithi045 conmon[39718]: -07-31T02:44:30.621316+0000 2021-07-31T02:44:32.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:31 smithi045 conmon[39718]: mgr.x (mgr.24542) 2450 : 2021-07-31T02:44:32.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:31 smithi045 conmon[39718]: cluster [DBG] pgmap v3610: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 332 KiB/s rd, 5.0 KiB/s wr, 450 op/s 2021-07-31T02:44:33.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:32 smithi154 conmon[34553]: cluster 2021-07-31T02:44:31.599875+0000 mon.a (mon.0) 3069 : cluster [DBG] osdmap e2046: 8 total, 8 up, 8 in 2021-07-31T02:44:33.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:32 smithi154 conmon[34553]: 2021-07-31T02:44:33.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:32 smithi154 conmon[34553]: cluster 2021-07-31T02:44:32.582143+0000 mon.a (mon.0) 3070 : cluster [DBG] osdmap e2047: 8 total, 8 up, 8 in 2021-07-31T02:44:33.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:32 smithi045 conmon[39718]: cluster 2021-07-31T02:44:31.599875+0000 mon.a (mon.0) 3069 : cluster [DBG] 2021-07-31T02:44:33.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:32 smithi045 conmon[39718]: osdmap e2046: 8 total, 8 up, 8 in 2021-07-31T02:44:33.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:32 smithi045 conmon[39718]: cluster 2021-07-31T02:44:32.582143+0000 mon.a (mon.0) 3070 : cluster [DBG] osdmap e2047: 8 total, 8 up, 8 in 2021-07-31T02:44:33.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:32 smithi045 conmon[33655]: cluster 2021-07-31T02:44:31.599875+0000 mon.a (mon. 2021-07-31T02:44:33.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:32 smithi045 conmon[33655]: 0) 3069 : cluster [DBG] osdmap e2046: 8 total, 8 up, 8 in 2021-07-31T02:44:33.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:32 smithi045 conmon[33655]: cluster 2021-07-31T02:44:32.582143+0000 mon.a (mon.0) 2021-07-31T02:44:33.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:32 smithi045 conmon[33655]: 3070 : cluster [DBG] osdmap e2047: 8 total, 8 up, 8 in 2021-07-31T02:44:34.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[34553]: cluster 2021-07-31T02:44:32.621687+0000 mgr.x (mgr.24542) 2451 : 2021-07-31T02:44:34.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[34553]: cluster [DBG] pgmap v3613: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 331 KiB/s rd, 5.0 KiB/s wr, 448 op/s 2021-07-31T02:44:34.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[34553]: audit 2021-07-31T02:44:33.581523+0000 mon.b (mon.2) 362 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:44:34.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[34553]: audit 2021-07-31T02:44:33.582997+0000 mon.b (mon.2) 363 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:44:34.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[34553]: audit 2021-07-31T02:44:33.584011+0000 mon.b (mon.2) 364 : 2021-07-31T02:44:34.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[34553]: audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:44:34.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: debug 2021-07-31T02:44:33.632+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:44:34.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:44:34.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:44:34.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: yield 2021-07-31T02:44:34.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:44:34.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:44:34.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:44:34.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:44:34.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:44:34.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:44:34.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:44:34.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:44:34.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:44:34.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: task.result() 2021-07-31T02:44:34.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:44:34.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: debug 2021-07-31T02:44:33.644+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:44:34.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:44:34.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:44:34.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: yield 2021-07-31T02:44:34.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:44:34.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:44:34.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:44:34.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:44:34.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:44:34.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:44:34.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:44:34.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:44:34.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:44:34.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: task.result() 2021-07-31T02:44:34.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:44:34.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: debug 2021-07-31T02:44:33.669+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:44:34.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:44:34.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:44:34.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: yield 2021-07-31T02:44:34.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:44:34.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:44:34.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:44:34.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:44:34.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:44:34.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:44:34.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:44:34.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:44:34.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:44:34.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: task.result() 2021-07-31T02:44:34.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:44:34.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: debug 2021-07-31T02:44:33.670+0000 7fcb9195d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:44:34.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:44:34.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:44:34.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: yield 2021-07-31T02:44:34.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:44:34.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:44:34.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:44:34.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:44:34.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:44:34.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:44:34.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:44:34.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:44:34.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:44:34.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: task.result() 2021-07-31T02:44:34.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:44:34.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: 2021-07-31T02:44:34.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:44:34.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: 2021-07-31T02:44:34.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:44:34.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:44:34.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:44:34.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:44:34.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:44:34.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:44:34.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:44:34.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:44:34.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:44:34.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:44:34.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:44:34.035 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:44:34.035 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:44:34.036 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:44:34.036 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:44:34.036 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:44:34.037 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:44:34.037 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:44:34.038 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: raise 2021-07-31T02:44:34.039 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:44:34.039 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:44:34.039 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:44:34.040 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:44:34.040 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:44:34.040 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: debug 2021-07-31T02:44:33.678+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:44:34.041 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:44:34.041 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:44:34.041 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: yield 2021-07-31T02:44:34.041 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:44:34.042 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:44:34.042 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:44:34.042 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:44:34.042 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:44:34.043 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:44:34.043 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:44:34.043 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:44:34.043 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:44:34.043 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: task.result() 2021-07-31T02:44:34.044 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:44:34.044 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: debug 2021-07-31T02:44:33.679+0000 7fcb9215e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:44:34.044 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:44:34.044 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:44:34.045 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: yield 2021-07-31T02:44:34.045 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:44:34.045 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:44:34.045 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:44:34.045 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:44:34.046 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:44:34.046 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:44:34.046 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:44:34.046 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:44:34.047 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:44:34.047 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: task.result() 2021-07-31T02:44:34.048 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:44:34.048 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: 2021-07-31T02:44:34.048 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:44:34.048 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: 2021-07-31T02:44:34.049 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:44:34.049 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:44:34.049 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:44:34.049 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:44:34.049 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:44:34.050 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:44:34.050 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:44:34.050 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:44:34.050 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:44:34.051 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:44:34.051 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:44:34.051 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:44:34.051 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:44:34.051 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:44:34.052 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:44:34.052 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:44:34.052 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:44:34.052 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:44:34.053 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: raise 2021-07-31T02:44:34.053 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:44:34.053 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:44:34.053 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:44:34.053 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:44:34.054 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:44:33 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:44:34.055 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:33 smithi045 conmon[33655]: cluster 2021-07-31T02:44:32.621687+0000 mgr.x ( 2021-07-31T02:44:34.055 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:33 smithi045 conmon[33655]: mgr.24542) 2451 : cluster [DBG] pgmap v3613: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 331 KiB/s rd, 5.0 KiB/s wr, 448 op/s 2021-07-31T02:44:34.055 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:33 smithi045 conmon[33655]: audit 2021-07-31T02:44:33.581523+0000 mon.b (mon.2) 362 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:44:34.056 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:33 smithi045 conmon[33655]: 2021-07-31T02:44:34.056 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:33 smithi045 conmon[33655]: audit 2021-07-31T02: 2021-07-31T02:44:34.056 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:33 smithi045 conmon[33655]: 44:33.582997+0000 mon.b (mon.2) 363 : 2021-07-31T02:44:34.056 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:33 smithi045 conmon[33655]: audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:44:34.057 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:33 smithi045 conmon[33655]: audit 2021-07-31T02:44:33. 2021-07-31T02:44:34.057 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:33 smithi045 conmon[33655]: 584011+0000 mon.b (mon.2) 364 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:44:34.057 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:33 smithi045 conmon[33655]: 2021-07-31T02:44:34.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:33 smithi045 conmon[39718]: cluster 2021-07-31T02:44:32.621687+0000 mgr.x 2021-07-31T02:44:34.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:33 smithi045 conmon[39718]: (mgr.24542) 2451 : cluster [DBG] pgmap v3613: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 331 KiB/s rd, 5.0 KiB/s wr, 448 op/s 2021-07-31T02:44:34.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:33 smithi045 conmon[39718]: audit 2021-07-31T02:44 2021-07-31T02:44:34.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:33 smithi045 conmon[39718]: :33.581523+0000 mon.b (mon.2) 362 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:44:34.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:33 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T02:44:34.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:33 smithi045 conmon[39718]: T02:44:33.582997+0000 mon.b (mon.2) 363 : 2021-07-31T02:44:34.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:33 smithi045 conmon[39718]: audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:44:34.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:33 smithi045 conmon[39718]: audit 2021-07-31T02:44:33. 2021-07-31T02:44:34.060 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:33 smithi045 conmon[39718]: 584011+0000 mon.b (mon.2) 364 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:44:34.060 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:33 smithi045 conmon[39718]: 2021-07-31T02:44:35.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: cephadm 2021-07-31T02:44:35.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: 2021-07-31T02:44:35.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: 2021 2021-07-31T02:44:35.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: - 2021-07-31T02:44:35.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: 07 2021-07-31T02:44:35.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: - 2021-07-31T02:44:35.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: 31 2021-07-31T02:44:35.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: T 2021-07-31T02:44:35.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: 02 2021-07-31T02:44:35.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: : 2021-07-31T02:44:35.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: 44 2021-07-31T02:44:35.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: : 2021-07-31T02:44:35.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: 33 2021-07-31T02:44:35.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: . 2021-07-31T02:44:35.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: 584767 2021-07-31T02:44:35.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: +0000 2021-07-31T02:44:35.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: 2021-07-31T02:44:35.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: mgr.x 2021-07-31T02:44:35.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: ( 2021-07-31T02:44:35.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: mgr 2021-07-31T02:44:35.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: . 2021-07-31T02:44:35.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: 24542 2021-07-31T02:44:35.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: ) 2021-07-31T02:44:35.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: 2452 2021-07-31T02:44:35.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: : 2021-07-31T02:44:35.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: cephadm 2021-07-31T02:44:35.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: 2021-07-31T02:44:35.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: [WRN] 2021-07-31T02:44:35.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: 2021-07-31T02:44:35.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:44:35.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: 2021-07-31T02:44:35.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: cluster 2021-07-31T02:44:35.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: 2021-07-31T02:44:35.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: 2021 2021-07-31T02:44:35.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: - 2021-07-31T02:44:35.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: 07 2021-07-31T02:44:35.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: - 2021-07-31T02:44:35.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: 31 2021-07-31T02:44:35.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: T 2021-07-31T02:44:35.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: 02 2021-07-31T02:44:35.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: : 2021-07-31T02:44:35.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: 44 2021-07-31T02:44:35.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: : 2021-07-31T02:44:35.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: 33 2021-07-31T02:44:35.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: . 2021-07-31T02:44:35.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: 607894 2021-07-31T02:44:35.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: +0000 2021-07-31T02:44:35.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: 2021-07-31T02:44:35.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: mon.a 2021-07-31T02:44:35.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: ( 2021-07-31T02:44:35.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: mon 2021-07-31T02:44:35.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: . 2021-07-31T02:44:35.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: 0 2021-07-31T02:44:35.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: ) 2021-07-31T02:44:35.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: 3071 2021-07-31T02:44:35.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: : 2021-07-31T02:44:35.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: cluster 2021-07-31T02:44:35.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: 2021-07-31T02:44:35.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: [DBG] 2021-07-31T02:44:35.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: 2021-07-31T02:44:35.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: osdmap e2048: 8 total, 8 up, 8 in 2021-07-31T02:44:35.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: 2021-07-31T02:44:35.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: cephadm 2021-07-31T02:44:35.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: 2021-07-31T02:44:35.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: 2021 2021-07-31T02:44:35.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: - 2021-07-31T02:44:35.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: 07 2021-07-31T02:44:35.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: - 2021-07-31T02:44:35.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: 31 2021-07-31T02:44:35.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: T 2021-07-31T02:44:35.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: 02 2021-07-31T02:44:35.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: : 2021-07-31T02:44:35.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: 44 2021-07-31T02:44:35.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: : 2021-07-31T02:44:35.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: 33 2021-07-31T02:44:35.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: . 2021-07-31T02:44:35.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: 633263 2021-07-31T02:44:35.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: +0000 2021-07-31T02:44:35.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: 2021-07-31T02:44:35.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: mgr.x 2021-07-31T02:44:35.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: ( 2021-07-31T02:44:35.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: mgr 2021-07-31T02:44:35.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: . 2021-07-31T02:44:35.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: 24542 2021-07-31T02:44:35.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: ) 2021-07-31T02:44:35.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: 2453 2021-07-31T02:44:35.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: : 2021-07-31T02:44:35.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: cephadm 2021-07-31T02:44:35.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: 2021-07-31T02:44:35.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: [ERR] 2021-07-31T02:44:35.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: 2021-07-31T02:44:35.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: 2021-07-31T02:44:35.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:44:35.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:44:35.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: yield 2021-07-31T02:44:35.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:44:35.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:44:35.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:44:35.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:44:35.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:44:35.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:44:35.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:44:35.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:44:35.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:44:35.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: task.result() 2021-07-31T02:44:35.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:44:35.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: 2021-07-31T02:44:35.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: cephadm 2021-07-31T02:44:35.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: 2021-07-31T02:44:35.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: 2021 2021-07-31T02:44:35.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: - 2021-07-31T02:44:35.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: 07 2021-07-31T02:44:35.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: - 2021-07-31T02:44:35.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: 31 2021-07-31T02:44:35.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: T 2021-07-31T02:44:35.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: 02 2021-07-31T02:44:35.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: : 2021-07-31T02:44:35.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: 44 2021-07-31T02:44:35.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: : 2021-07-31T02:44:35.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: 33 2021-07-31T02:44:35.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: . 2021-07-31T02:44:35.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: 633680 2021-07-31T02:44:35.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: +0000 2021-07-31T02:44:35.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: 2021-07-31T02:44:35.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: mgr.x 2021-07-31T02:44:35.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: ( 2021-07-31T02:44:35.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: mgr 2021-07-31T02:44:35.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: . 2021-07-31T02:44:35.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: 24542 2021-07-31T02:44:35.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: ) 2021-07-31T02:44:35.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: 2454 2021-07-31T02:44:35.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:44:35.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: cephadm 2021-07-31T02:44:33.645286+0000 mgr.x (mgr.24542) 2455 : cephadm 2021-07-31T02:44:35.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: [ERR] 2021-07-31T02:44:35.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:44:35.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:44:35.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: yield 2021-07-31T02:44:35.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:44:35.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:44:35.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:44:35.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:44:35.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:44:35.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:44:35.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:44:35.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:44:35.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:44:35.035 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: task.result() 2021-07-31T02:44:35.035 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:44:35.035 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: cephadm 2021-07-31T02:44:35.035 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: 2021-07-31T02:44:33.645950+0000 mgr.x (mgr.24542) 2456 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:44:35.036 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: cephadm 2021-07-31T02:44:33.670551+0000 mgr.x (mgr.24542 2021-07-31T02:44:35.036 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: ) 2457 : cephadm [ERR] 2021-07-31T02:44:35.036 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:44:35.036 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:44:35.036 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: yield 2021-07-31T02:44:35.037 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:44:35.037 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:44:35.037 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:44:35.037 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:44:35.037 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:44:35.038 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:44:35.038 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:44:35.038 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:44:35.038 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:44:35.039 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: task.result() 2021-07-31T02:44:35.039 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:44:35.039 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: cephadm 2021-07-31T02:44:33.671136+0000 mgr.x (mgr.24542) 2458 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:44:35.039 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:44:35.039 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:44:35.040 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: yield 2021-07-31T02:44:35.040 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:44:35.040 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:44:35.040 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:44:35.041 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:44:35.041 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:44:35.041 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:44:35.041 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:44:35.041 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:44:35.042 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:44:35.042 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: task.result() 2021-07-31T02:44:35.042 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:44:35.042 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: 2021-07-31T02:44:35.043 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:44:35.043 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: 2021-07-31T02:44:35.043 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:44:35.043 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:44:35.044 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: cephadm 2021-07-31T02:44:33.584767+0000 mgr.x (mgr.24542) 2452 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:44:35.044 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: cluster 2021-07-31T02:44:33.607894+0000 mon.a (mon.0) 3071 : cluster [DBG] osdmap e2048: 8 total, 8 up, 8 in 2021-07-31T02:44:35.045 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: cephadm 2021-07-31T02:44:33.633263+0000 mgr.x (mgr.24542) 2453 : cephadm [ERR] 2021-07-31T02:44:35.045 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:44:35.045 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:44:35.045 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: yield 2021-07-31T02:44:35.046 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:44:35.046 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:44:35.046 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:44:35.046 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:44:35.046 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:44:35.047 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:44:35.047 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:44:35.047 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:44:35.047 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:44:35.048 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: task.result() 2021-07-31T02:44:35.048 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:44:35.048 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: cephadm 2021-07-31T02:44:33.633680+0000 mgr.x (mgr.24542) 2454 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:44:35.048 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: cephadm 2021-07-31T02:44:33.645286+0000 mgr.x (mgr.24542) 2455 : cephadm [ERR] 2021-07-31T02:44:35.048 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:44:35.049 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:44:35.049 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: yield 2021-07-31T02:44:35.049 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:44:35.049 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:44:35.050 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:44:35.050 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:44:35.050 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:44:35.050 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:44:35.050 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:44:35.051 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:44:35.051 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:44:35.051 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: task.result() 2021-07-31T02:44:35.051 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:44:35.051 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: cephadm 2021 2021-07-31T02:44:35.052 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: -07-31T02:44:33.645950+0000 mgr.x (mgr.24542) 2456 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:44:35.052 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: cephadm 2021-07-31T02:44:33.670551+0000 mgr.x (mgr.24542) 2457 : cephadm [ERR] 2021-07-31T02:44:35.052 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:44:35.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: cephadm 2021-07-31T02:44:35.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: 2021-07-31T02:44:35.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: 2021 2021-07-31T02:44:35.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: - 2021-07-31T02:44:35.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: 07 2021-07-31T02:44:35.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: - 2021-07-31T02:44:35.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: 31T02:44:33.584767+0000 mgr.x ( 2021-07-31T02:44:35.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: mgr.24542) 2452 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:44:35.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: cluster 2021-07-31T02:44:33.607894+0000 2021-07-31T02:44:35.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: mon.a (mon.0) 3071 : cluster [DBG] osdmap e2048: 8 total, 8 up, 8 in 2021-07-31T02:44:35.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: cephadm 2021 2021-07-31T02:44:35.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: -07-31T02:44:33.633263 2021-07-31T02:44:35.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 2453 : cephadm [ERR] 2021-07-31T02:44:35.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: 2021-07-31T02:44:35.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:44:35.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:44:35.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: yield 2021-07-31T02:44:35.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:44:35.057 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:44:35.057 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:44:35.057 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:44:35.057 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:44:35.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:44:35.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:44:35.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:44:35.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:44:35.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: task.result() 2021-07-31T02:44:35.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:44:35.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: cephadm 2021-07-31T02:44:33.633680+0000 mgr.x (mgr.24542) 2454 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:44:35.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: cephadm 2021-07 2021-07-31T02:44:35.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: -31T02:44:33.645286+0000 2021-07-31T02:44:35.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: mgr.x (mgr.24542) 2455 : cephadm 2021-07-31T02:44:35.060 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: [ERR] 2021-07-31T02:44:35.060 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:44:35.060 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:44:35.060 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: yield 2021-07-31T02:44:35.061 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:44:35.061 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:44:35.061 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:44:35.061 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:44:35.061 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:44:35.062 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:44:35.062 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:44:35.062 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:44:35.062 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:44:35.063 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:44:35.063 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:44:35.063 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:44:35.063 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:44:35.063 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:44:35.064 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:44:35.064 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:44:35.064 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:44:35.064 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:44:35.064 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:44:35.065 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:44:35.065 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:44:35.065 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:44:35.065 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:44:35.066 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:44:35.066 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: raise 2021-07-31T02:44:35.066 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:44:35.066 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:44:35.066 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:44:35.067 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:44:35.067 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:44:35.067 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: cephadm 2021-07-31T02 2021-07-31T02:44:35.067 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: :44:33.679721+0000 mgr.x (mgr.24542) 2459 : cephadm [ERR] 2021-07-31T02:44:35.068 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:44:35.068 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:44:35.068 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: yield 2021-07-31T02:44:35.068 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:44:35.068 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:44:35.069 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:44:35.069 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:44:35.069 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:44:35.069 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:44:35.069 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:44:35.070 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:44:35.070 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:44:35.070 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: task.result() 2021-07-31T02:44:35.070 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:44:35.071 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: cephadm 2021-07-31T02:44:33.680108+0000 mgr.x (mgr.24542) 2460 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:44:35.071 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:44:35.071 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:44:35.071 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: yield 2021-07-31T02:44:35.071 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:44:35.072 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:44:35.072 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:44:35.072 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:44:35.072 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:44:35.072 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:44:35.073 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:44:35.073 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:44:35.073 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:44:35.073 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: task.result() 2021-07-31T02:44:35.073 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:44:35.074 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: 2021-07-31T02:44:35.074 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:44:35.074 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: 2021-07-31T02:44:35.074 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:44:35.075 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:44:35.075 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:44:35.075 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:44:35.075 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:44:35.075 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:44:35.076 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:44:35.076 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:44:35.076 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:44:35.076 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:44:35.076 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:44:35.077 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:44:35.077 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:44:35.077 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:44:35.077 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:44:35.078 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:44:35.078 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:44:35.078 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:44:35.078 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: raise 2021-07-31T02:44:35.078 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:44:35.079 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:44:35.079 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:44:35.079 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:44:35.079 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:34 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:44:35.080 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:44:35.080 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: yield 2021-07-31T02:44:35.080 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:44:35.080 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:44:35.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:44:35.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:44:35.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:44:35.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:44:35.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:44:35.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:44:35.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:44:35.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: task.result() 2021-07-31T02:44:35.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:44:35.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: cephadm 2021-07-31T02:44:33.671136+0000 mgr.x (mgr.24542) 2458 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:44:35.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:44:35.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:44:35.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: yield 2021-07-31T02:44:35.084 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:44:35.084 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:44:35.084 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:44:35.084 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:44:35.084 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:44:35.085 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:44:35.085 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:44:35.085 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:44:35.085 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:44:35.085 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: task.result() 2021-07-31T02:44:35.086 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:44:35.086 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: 2021-07-31T02:44:35.086 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:44:35.086 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: 2021-07-31T02:44:35.087 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:44:35.087 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:44:35.087 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:44:35.087 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:44:35.087 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:44:35.088 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:44:35.088 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:44:35.088 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:44:35.088 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:44:35.089 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:44:35.089 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: task.result() 2021-07-31T02:44:35.089 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:44:35.090 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: cephadm 2021-07-31T02:44:33.645950+0000 mgr.x (mgr.24542) 2456 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:44:35.090 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: cephadm 2021 2021-07-31T02:44:35.090 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: -07-31T02:44 2021-07-31T02:44:35.090 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: :33.670551+0000 mgr.x ( 2021-07-31T02:44:35.090 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: mgr.24542) 2457 : cephadm 2021-07-31T02:44:35.091 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: [ERR] 2021-07-31T02:44:35.091 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:44:35.091 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:44:35.091 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: yield 2021-07-31T02:44:35.092 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:44:35.092 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:44:35.092 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:44:35.092 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:44:35.093 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:44:35.093 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:44:35.093 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:44:35.093 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:44:35.093 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:44:35.094 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: task.result() 2021-07-31T02:44:35.094 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:44:35.094 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: cephadm 2021-07-31T02:44:33.671136+0000 mgr.x (mgr.24542) 2458 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:44:35.094 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:44:35.095 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:44:35.095 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: yield 2021-07-31T02:44:35.095 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:44:35.095 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:44:35.095 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:44:35.096 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:44:35.096 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:44:35.096 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:44:35.096 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:44:35.097 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:44:35.097 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:44:35.097 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: task.result() 2021-07-31T02:44:35.097 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:44:35.097 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: 2021-07-31T02:44:35.098 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:44:35.098 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: 2021-07-31T02:44:35.098 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:44:35.098 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:44:35.098 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:44:35.099 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:44:35.099 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:44:35.099 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:44:35.099 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:44:35.100 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:44:35.100 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:44:35.100 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:44:35.100 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:44:35.100 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:44:35.101 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:44:35.101 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:44:35.101 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:44:35.101 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:44:35.101 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:44:35.102 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:44:35.102 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: raise 2021-07-31T02:44:35.102 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:44:35.102 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:44:35.103 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:44:35.103 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:44:35.103 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:44:35.103 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: cephadm 2021-07-31T02:44:33.679721+0000 mgr.x (mgr.24542) 2459 : cephadm [ERR] 2021-07-31T02:44:35.103 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:44:35.104 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:44:35.104 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: yield 2021-07-31T02:44:35.104 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:44:35.104 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:44:35.104 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:44:35.105 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:44:35.105 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:44:35.105 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:44:35.105 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:44:35.105 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:44:35.106 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:44:35.106 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: task.result() 2021-07-31T02:44:35.106 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:44:35.106 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: cephadm 2021-07-31T02:44:33.680108+0000 mgr.x (mgr.24542) 2460 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:44:35.106 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:44:35.107 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:44:35.107 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:44:35.107 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:44:35.108 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:44:35.108 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:44:35.108 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:44:35.108 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:44:35.108 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:44:35.109 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:44:35.109 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:44:35.109 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: raise 2021-07-31T02:44:35.109 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:44:35.109 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:44:35.110 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:44:35.110 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:44:35.110 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:44:35.110 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: cephadm 2021-07-31T02:44:33.679721+0000 mgr.x (mgr.24542) 2459 : cephadm [ERR] 2021-07-31T02:44:35.111 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:44:35.111 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:44:35.111 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: yield 2021-07-31T02:44:35.111 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:44:35.111 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:44:35.112 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:44:35.112 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:44:35.112 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:44:35.112 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:44:35.112 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:44:35.113 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:44:35.113 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:44:35.113 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: task.result() 2021-07-31T02:44:35.113 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:44:35.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: cephadm 2021-07-31T02:44:33.680108+0000 mgr.x (mgr.24542) 2460 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:44:35.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:44:35.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:44:35.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: yield 2021-07-31T02:44:35.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:44:35.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:44:35.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:44:35.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:44:35.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:44:35.116 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: yield 2021-07-31T02:44:35.116 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:44:35.116 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:44:35.116 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:44:35.117 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:44:35.117 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:44:35.117 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:44:35.117 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:44:35.117 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:44:35.118 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:44:35.118 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: task.result() 2021-07-31T02:44:35.118 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:44:35.118 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: 2021-07-31T02:44:35.118 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:44:35.119 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: 2021-07-31T02:44:35.119 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:44:35.119 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:44:35.119 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:44:35.120 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:44:35.120 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:44:35.120 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:44:35.120 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:44:35.120 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:44:35.121 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:44:35.121 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:44:35.121 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:44:35.121 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:44:35.121 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:44:35.122 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:44:35.122 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:44:35.122 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:44:35.122 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:44:35.122 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:44:35.123 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: raise 2021-07-31T02:44:35.123 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:44:35.123 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:44:35.123 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:44:35.124 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:44:35.124 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:44:35.124 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:44:35.124 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:44:35.125 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:44:35.125 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:44:35.125 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: task.result() 2021-07-31T02:44:35.125 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:44:35.125 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: 2021-07-31T02:44:35.126 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:44:35.126 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: 2021-07-31T02:44:35.126 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:44:35.126 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:44:35.126 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:44:35.127 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:44:35.127 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:44:35.127 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:44:35.127 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:44:35.127 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:44:35.128 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:44:35.128 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:44:35.128 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:44:35.128 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:44:35.129 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:44:35.129 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:44:35.129 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:44:35.129 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:44:35.129 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:44:35.130 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:44:35.130 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: raise 2021-07-31T02:44:35.130 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:44:35.130 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:44:35.130 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:44:35.131 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:44:35.131 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:34 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:44:36.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:35 smithi154 conmon[34553]: cluster 2021-07-31T02:44:34.609391+0000 mon.a (mon.0) 3072 : cluster [DBG] osdmap e2049: 8 total, 8 up, 8 in 2021-07-31T02:44:36.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:35 smithi154 conmon[34553]: 2021-07-31T02:44:36.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:35 smithi154 conmon[34553]: cluster 2021-07-31T02:44:34.622145+0000 mgr.x (mgr.24542) 2461 : cluster [DBG] pgmap v3616: 41 pgs: 41 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 298 KiB/s rd, 5.0 KiB/s wr, 404 op/s 2021-07-31T02:44:36.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:35 smithi045 conmon[39718]: cluster 2021-07-31T02:44:34.609391+0000 2021-07-31T02:44:36.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:35 smithi045 conmon[39718]: mon.a (mon.0) 3072 : cluster [DBG] osdmap e2049: 8 total, 8 up, 8 in 2021-07-31T02:44:36.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:35 smithi045 conmon[39718]: cluster 2021-07-31T02:44:36.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:35 smithi045 conmon[39718]: 2021-07-31T02:44:34.622145+0000 mgr.x (mgr.24542) 2461 : cluster 2021-07-31T02:44:36.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:35 smithi045 conmon[39718]: [DBG] pgmap v3616: 41 pgs: 41 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 298 KiB/s rd, 5.0 KiB/s wr, 404 op/s 2021-07-31T02:44:36.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:35 smithi045 conmon[33655]: cluster 2021-07-31T02:44:34.609391+0000 2021-07-31T02:44:36.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:35 smithi045 conmon[33655]: mon.a (mon.0) 3072 : cluster [DBG] osdmap e2049: 8 total, 8 up, 8 in 2021-07-31T02:44:36.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:35 smithi045 conmon[33655]: cluster 2021-07-31T02:44:34.622145+0000 mgr.x ( 2021-07-31T02:44:36.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:35 smithi045 conmon[33655]: mgr.24542) 2461 : cluster [DBG] pgmap v3616: 41 pgs: 41 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 298 KiB/s rd, 5.0 KiB/s wr, 404 op/s 2021-07-31T02:44:37.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:36 smithi154 conmon[34553]: cluster 2021-07-31T02:44:35.620456+0000 mon.a (mon.0) 3073 : cluster 2021-07-31T02:44:37.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:36 smithi154 conmon[34553]: [DBG] osdmap e2050: 8 total, 8 up, 8 in 2021-07-31T02:44:37.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:36 smithi045 conmon[33655]: cluster 2021-07-31T02:44:35. 2021-07-31T02:44:37.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:36 smithi045 conmon[33655]: 620456+0000 mon.a (mon.0) 3073 : cluster [DBG] osdmap e2050: 8 total, 8 up, 8 in 2021-07-31T02:44:37.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:36 smithi045 conmon[39718]: cluster 2021-07-31T02:44:35. 2021-07-31T02:44:37.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:36 smithi045 conmon[39718]: 620456+0000 mon.a (mon.0) 3073 : cluster [DBG] osdmap e2050: 8 total, 8 up, 8 in 2021-07-31T02:44:38.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:37 smithi154 conmon[34553]: cluster 2021-07-31T02:44:36.622338+0000 mon.a (mon.0) 3074 : cluster [DBG] 2021-07-31T02:44:38.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:37 smithi154 conmon[34553]: osdmap e2051: 8 total, 8 up, 8 in 2021-07-31T02:44:38.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:37 smithi154 conmon[34553]: cluster 2021-07-31T02:44:36.622530+0000 mgr.x (mgr.24542) 2462 : cluster [DBG] pgmap v3619: 41 pgs: 41 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 298 KiB/s rd, 5.0 KiB/s wr, 404 op/s 2021-07-31T02:44:38.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:37 smithi154 conmon[34553]: cluster 2021-07-31T02:44:37.584837+0000 mon.a (mon.0) 3075 : cluster [DBG] osdmap e2052: 8 total, 8 up, 8 in 2021-07-31T02:44:38.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:37 smithi045 conmon[33655]: cluster 2021-07-31T02:44:36.622338+0000 mon.a 2021-07-31T02:44:38.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:37 smithi045 conmon[33655]: (mon.0) 3074 : cluster [DBG] osdmap e2051: 8 total, 8 up, 8 in 2021-07-31T02:44:38.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:37 smithi045 conmon[33655]: cluster 2021-07-31T02:44:36.622530+0000 mgr.x (mgr.24542) 2462 : cluster [DBG] pgmap v3619: 41 pgs: 41 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 298 KiB/s rd, 5.0 KiB/s wr, 404 op/s 2021-07-31T02:44:38.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:37 smithi045 conmon[33655]: cluster 2021-07-31T02:44:37.584837+0000 2021-07-31T02:44:38.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:37 smithi045 conmon[33655]: mon.a (mon.0) 3075 : cluster [DBG] osdmap e2052: 8 total, 8 up, 8 in 2021-07-31T02:44:38.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:37 smithi045 conmon[39718]: cluster 2021-07-31T02:44:36.622338+0000 2021-07-31T02:44:38.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:37 smithi045 conmon[39718]: mon.a (mon.0) 3074 : cluster [DBG] osdmap e2051: 8 total, 8 up, 8 in 2021-07-31T02:44:38.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:37 smithi045 conmon[39718]: cluster 2021-07-31T02: 2021-07-31T02:44:38.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:37 smithi045 conmon[39718]: 44:36.622530+0000 mgr.x (mgr.24542) 2462 : cluster [DBG] 2021-07-31T02:44:38.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:37 smithi045 conmon[39718]: pgmap v3619: 41 pgs: 41 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 298 KiB/s rd, 5.0 KiB/s wr, 404 op/s 2021-07-31T02:44:38.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:37 smithi045 conmon[39718]: cluster 2021-07-31T02 2021-07-31T02:44:38.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:37 smithi045 conmon[39718]: :44:37.584837+0000 mon.a (mon 2021-07-31T02:44:38.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:37 smithi045 conmon[39718]: .0) 3075 : cluster [DBG] osdmap e2052: 8 total, 8 up, 8 in 2021-07-31T02:44:40.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:39 smithi154 conmon[34553]: cluster 2021-07-31T02:44:38.591349+0000 mon.a (mon.0) 3076 : cluster 2021-07-31T02:44:40.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:39 smithi154 conmon[34553]: [DBG] osdmap e2053: 8 total, 8 up, 8 in 2021-07-31T02:44:40.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:39 smithi154 conmon[34553]: cluster 2021-07-31T02:44:38.622940+0000 mgr.x (mgr.24542) 2463 : cluster [DBG] pgmap v3622: 41 pgs: 41 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:44:40.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:39 smithi045 conmon[39718]: cluster 2021-07-31T02:44:38.591349+0000 mon.a ( 2021-07-31T02:44:40.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:39 smithi045 conmon[39718]: mon.0) 3076 : cluster [DBG] osdmap e2053: 8 total, 8 up, 8 in 2021-07-31T02:44:40.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:39 smithi045 conmon[39718]: cluster 2021-07-31T02:44:38.622940+0000 mgr.x (mgr.24542) 2463 : cluster 2021-07-31T02:44:40.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:39 smithi045 conmon[39718]: [DBG] pgmap v3622: 41 pgs: 41 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:44:40.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:39 smithi045 conmon[33655]: cluster 2021-07-31T02:44:38.591349+0000 mon.a (mon.0) 3076 2021-07-31T02:44:40.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:39 smithi045 conmon[33655]: : cluster [DBG] osdmap e2053: 8 total, 8 up, 8 in 2021-07-31T02:44:40.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:39 smithi045 conmon[33655]: cluster 2021-07-31T02:44:38.622940+0000 mgr.x (mgr.24542) 2463 : cluster [DBG] pgmap v3622: 41 pgs: 41 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:44:40.533 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:44:40.819 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:44:40.820 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 59m ago 60m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:44:40.820 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 57m ago 60m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:44:40.820 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 57m ago 63m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:44:40.820 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 59m ago 65m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:44:40.820 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 59m ago 65m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:44:40.821 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 57m ago 63m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:44:40.821 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 59m ago 64m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:44:40.821 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 59m ago 60m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:44:40.821 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 57m ago 60m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:44:40.822 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 59m ago 63m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:44:40.822 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 59m ago 62m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:44:40.822 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 59m ago 62m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:44:40.822 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 59m ago 62m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:44:40.822 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 57m ago 61m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:44:40.823 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 57m ago 61m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:44:40.823 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 57m ago 61m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:44:40.823 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 57m ago 60m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:44:40.823 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 57m ago 60m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:44:41.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:40 smithi154 conmon[34553]: cluster 2021-07-31T02:44:39.597248+0000 mon.a (mon.0) 2021-07-31T02:44:41.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:40 smithi154 conmon[34553]: 3077 : cluster [DBG] osdmap e2054: 8 total, 8 up, 8 in 2021-07-31T02:44:41.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:40 smithi045 conmon[39718]: cluster 2021-07-31T02:44:39.597248 2021-07-31T02:44:41.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:40 smithi045 conmon[39718]: +0000 mon.a (mon.0) 3077 : cluster [DBG] osdmap e2054: 8 total, 8 up, 8 in 2021-07-31T02:44:41.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:40 smithi045 conmon[33655]: cluster 2021-07-31T02:44:39.597248+0000 2021-07-31T02:44:41.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:40 smithi045 conmon[33655]: mon.a (mon.0) 3077 : cluster [DBG] osdmap e2054: 8 total, 8 up, 8 in 2021-07-31T02:44:41.154 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:44:41.154 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:44:41.154 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:44:41.154 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:44:41.155 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:44:41.155 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:44:41.155 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:44:41.155 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:44:41.156 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:44:41.156 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:44:41.156 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:44:41.156 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:44:41.157 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:44:41.157 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:44:41.157 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:44:41.157 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:44:41.158 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:44:42.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:41 smithi154 conmon[34553]: audit 2021-07-31T02:44:40.521432+0000 mgr.x (mgr.24542) 2464 : audit 2021-07-31T02:44:42.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:41 smithi154 conmon[34553]: [DBG] from='client.27910 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:44:42.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:41 smithi154 conmon[34553]: cluster 2021-07-31T02:44:40.602009+0000 mon.a (mon.0) 3078 : cluster [DBG] osdmap e2055: 8 total, 8 up, 8 in 2021-07-31T02:44:42.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:41 smithi154 conmon[34553]: cluster 2021-07-31T02:44:40.623351+0000 mgr.x (mgr.24542) 2465 : cluster [DBG] pgmap v3625: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 266 KiB/s rd, 5.0 KiB/s wr, 361 op/s 2021-07-31T02:44:42.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:41 smithi154 conmon[34553]: audit 2021-07-31T02:44:40.811608+0000 mgr.x (mgr. 2021-07-31T02:44:42.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:41 smithi154 conmon[34553]: 24542) 2466 : audit [DBG] from='client.17976 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:44:42.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:41 smithi154 conmon[34553]: audit 2021-07-31T02:44:41.153872+0000 mon.c (mon.1) 131 : audit [DBG] from='client.? 172.21.15.45:0/4252745043' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:44:42.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:41 smithi045 conmon[33655]: audit 2021-07-31T02:44:40.521432+0000 mgr.x (mgr.24542) 2464 : audit 2021-07-31T02:44:42.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:41 smithi045 conmon[33655]: [DBG] from='client.27910 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:44:42.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:41 smithi045 conmon[33655]: cluster 2021-07-31T02:44:40.602009+0000 mon.a (mon.0) 3078 : cluster [DBG] osdmap e2055: 8 total, 8 up, 8 in 2021-07-31T02:44:42.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:41 smithi045 conmon[33655]: cluster 2021-07-31T02:44:40.623351+0000 mgr.x (mgr.24542) 2465 : cluster [DBG] pgmap v3625: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 266 KiB/s rd, 5.0 KiB/s wr, 361 op/s 2021-07-31T02:44:42.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:41 smithi045 conmon[33655]: 2021-07-31T02:44:42.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:41 smithi045 conmon[33655]: audit 2021-07-31T02:44:40 2021-07-31T02:44:42.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:41 smithi045 conmon[33655]: .811608+0000 mgr.x (mgr.24542) 2466 : audit 2021-07-31T02:44:42.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:41 smithi045 conmon[33655]: [DBG] from='client.17976 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:44:42.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:41 smithi045 conmon[33655]: audit 2021-07-31T02 2021-07-31T02:44:42.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:41 smithi045 conmon[33655]: :44:41.153872+0000 mon.c (mon.1) 2021-07-31T02:44:42.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:41 smithi045 conmon[33655]: 131 : audit [DBG] from='client.? 172.21.15.45:0/4252745043' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:44:42.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:41 smithi045 conmon[39718]: audit 2021-07-31T02:44:40.521432+0000 mgr.x 2021-07-31T02:44:42.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:41 smithi045 conmon[39718]: (mgr.24542) 2464 : audit [DBG] from='client.27910 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:44:42.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:41 smithi045 conmon[39718]: cluster 2021-07-31T02:44:40.602009+0000 mon.a (mon.0) 3078 : cluster [DBG] 2021-07-31T02:44:42.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:41 smithi045 conmon[39718]: osdmap e2055: 8 total, 8 up, 8 in 2021-07-31T02:44:42.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:41 smithi045 conmon[39718]: cluster 2021-07-31T02:44:40.623351+0000 2021-07-31T02:44:42.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:41 smithi045 conmon[39718]: mgr.x (mgr.24542) 2465 : cluster [DBG] pgmap v3625: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 266 KiB/s rd, 5.0 KiB/s wr, 361 op/s 2021-07-31T02:44:42.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:41 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T02:44:42.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:41 smithi045 conmon[39718]: T02:44:40.811608+0000 mgr.x (mgr.24542) 2021-07-31T02:44:42.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:41 smithi045 conmon[39718]: 2466 : audit [DBG] from='client.17976 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:44:42.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:41 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T02:44:42.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:41 smithi045 conmon[39718]: 31T02:44:41.153872+0000 mon.c 2021-07-31T02:44:42.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:41 smithi045 conmon[39718]: (mon.1) 131 : audit [DBG] 2021-07-31T02:44:42.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:41 smithi045 conmon[39718]: from='client.? 172.21.15.45:0/4252745043' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:44:43.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:42 smithi154 conmon[34553]: cluster 2021-07-31T02:44:41.608356+0000 mon.a (mon.0) 3079 : cluster [DBG] osdmap e2056: 8 total, 8 up, 8 in 2021-07-31T02:44:43.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:42 smithi154 conmon[34553]: cluster 2021-07-31T02:44:42. 2021-07-31T02:44:43.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:42 smithi154 conmon[34553]: 585739+0000 mon.a (mon.0) 3080 : cluster [DBG] osdmap e2057: 8 total, 8 up, 8 in 2021-07-31T02:44:43.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:42 smithi045 conmon[33655]: cluster 2021-07-31T02 2021-07-31T02:44:43.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:42 smithi045 conmon[33655]: :44:41.608356+0000 mon.a (mon.0) 3079 : cluster [DBG] osdmap e2056: 8 total, 8 up, 8 in 2021-07-31T02:44:43.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:42 smithi045 conmon[33655]: cluster 2021- 2021-07-31T02:44:43.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:42 smithi045 conmon[33655]: 07-31T02:44:42.585739+0000 mon.a (mon.0) 3080 : cluster [DBG] osdmap e2057: 8 total, 8 up, 8 in 2021-07-31T02:44:43.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:42 smithi045 conmon[39718]: cluster 2021-07-31T02:44:41.608356+0000 mon.a ( 2021-07-31T02:44:43.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:42 smithi045 conmon[39718]: mon.0) 3079 : cluster [DBG] osdmap e2056: 8 total, 8 up, 8 in 2021-07-31T02:44:43.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:42 smithi045 conmon[39718]: cluster 2021-07-31T02:44:42.585739+0000 mon.a ( 2021-07-31T02:44:43.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:42 smithi045 conmon[39718]: mon.0) 3080 : cluster [DBG] osdmap e2057: 8 total, 8 up, 8 in 2021-07-31T02:44:44.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:43 smithi154 conmon[34553]: cluster 2021-07-31T02:44:42.623692+0000 mgr.x (mgr.24542) 2467 : cluster [DBG] pgmap v3628: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 266 KiB/s rd, 5.0 KiB/s wr, 361 op/s 2021-07-31T02:44:44.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:43 smithi045 conmon[39718]: cluster 2021-07-31T02:44:42.623692+0000 mgr.x (mgr.24542) 2467 : cluster [DBG] pgmap v3628: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 266 KiB/s rd, 5.0 KiB/s wr, 361 op/s 2021-07-31T02:44:44.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:43 smithi045 conmon[33655]: cluster 2021-07-31T02:44:42.623692+0000 mgr.x ( 2021-07-31T02:44:44.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:43 smithi045 conmon[33655]: mgr.24542) 2467 : cluster [DBG] pgmap v3628: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 266 KiB/s rd, 5.0 KiB/s wr, 361 op/s 2021-07-31T02:44:45.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:44 smithi154 conmon[34553]: cluster 2021-07-31T02:44:43.615863+0000 mon.a (mon.0) 3081 : cluster [DBG] osdmap e2058: 8 total, 8 up, 8 in 2021-07-31T02:44:45.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:44 smithi045 conmon[39718]: cluster 2021-07-31T02:44:43.615863+0000 mon.a ( 2021-07-31T02:44:45.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:44 smithi045 conmon[39718]: mon.0) 3081 : cluster [DBG] osdmap e2058: 8 total, 8 up, 8 in 2021-07-31T02:44:45.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:44 smithi045 conmon[33655]: cluster 2021-07-31T02:44:43. 2021-07-31T02:44:45.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:44 smithi045 conmon[33655]: 615863+0000 mon.a (mon.0) 3081 : cluster [DBG] osdmap e2058: 8 total, 8 up, 8 in 2021-07-31T02:44:46.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:45 smithi154 conmon[34553]: cluster 2021-07-31T02:44 2021-07-31T02:44:46.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:45 smithi154 conmon[34553]: :44.621118+0000 mon.a (mon.0) 3082 : cluster [DBG] osdmap e2059: 8 total, 8 up, 8 in 2021-07-31T02:44:46.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:45 smithi154 conmon[34553]: cluster 2021-07-31T02:44:44.624144+0000 mgr.x (mgr.24542) 2468 : cluster 2021-07-31T02:44:46.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:45 smithi154 conmon[34553]: [DBG] pgmap v3631: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 232 KiB/s rd, 5.0 KiB/s wr, 317 op/s 2021-07-31T02:44:46.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:45 smithi045 conmon[39718]: cluster 2021-07-31T02:44:44. 2021-07-31T02:44:46.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:45 smithi045 conmon[39718]: 621118+0000 mon.a (mon.0) 3082 : cluster [DBG] osdmap e2059: 8 total, 8 up, 8 in 2021-07-31T02:44:46.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:45 smithi045 conmon[39718]: cluster 2021-07-31T02:44 2021-07-31T02:44:46.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:45 smithi045 conmon[39718]: :44.624144+0000 mgr.x (mgr.24542) 2468 : cluster [DBG] pgmap v3631: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 232 KiB/s rd, 5.0 KiB/s wr, 317 op/s 2021-07-31T02:44:46.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:45 smithi045 conmon[33655]: cluster 2021-07-31T02:44:44.621118+0000 mon.a (mon.0) 3082 : 2021-07-31T02:44:46.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:45 smithi045 conmon[33655]: cluster [DBG] osdmap e2059: 8 total, 8 up, 8 in 2021-07-31T02:44:46.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:45 smithi045 conmon[33655]: cluster 2021-07-31T02:44:44.624144+0000 mgr.x (mgr.24542) 2468 : cluster [DBG] 2021-07-31T02:44:46.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:45 smithi045 conmon[33655]: pgmap v3631: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 232 KiB/s rd, 5.0 KiB/s wr, 317 op/s 2021-07-31T02:44:47.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:46 smithi154 conmon[34553]: cluster 2021-07-31T02:44:45.627919+0000 mon.a (mon.0) 3083 : 2021-07-31T02:44:47.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:46 smithi154 conmon[34553]: cluster [DBG] osdmap e2060: 8 total, 8 up, 8 in 2021-07-31T02:44:47.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:46 smithi045 conmon[39718]: cluster 2021-07-31T02:44:45.627919+0000 mon.a (mon.0 2021-07-31T02:44:47.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:46 smithi045 conmon[39718]: ) 3083 : cluster [DBG] osdmap e2060: 8 total, 8 up, 8 in 2021-07-31T02:44:47.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:46 smithi045 conmon[33655]: cluster 2021-07-31T02:44:45.627919+0000 mon.a (mon.0) 2021-07-31T02:44:47.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:46 smithi045 conmon[33655]: 3083 : cluster [DBG] osdmap e2060: 8 total, 8 up, 8 in 2021-07-31T02:44:48.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:47 smithi154 conmon[34553]: cluster 2021-07-31T02:44:46.624542+0000 mgr.x (mgr.24542) 2469 : cluster [DBG] pgmap v3633: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 230 KiB/s rd, 4.9 KiB/s wr, 314 op/s 2021-07-31T02:44:48.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:47 smithi154 conmon[34553]: cluster 2021-07-31T02: 2021-07-31T02:44:48.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:47 smithi154 conmon[34553]: 44:46.629158+0000 mon.a (mon.0) 3084 : cluster [DBG] osdmap e2061: 8 total, 8 up, 8 in 2021-07-31T02:44:48.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:47 smithi154 conmon[34553]: cluster 2021-07-31T02:44:47.587540+0000 mon.a (mon.0) 3085 : cluster [DBG] osdmap e2062: 8 total, 8 up, 8 in 2021-07-31T02:44:48.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:47 smithi045 conmon[33655]: cluster 2021-07-31T02:44:46.624542+0000 mgr.x (mgr.24542 2021-07-31T02:44:48.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:47 smithi045 conmon[33655]: ) 2469 : cluster [DBG] pgmap v3633: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 230 KiB/s rd, 4.9 KiB/s wr, 314 op/s 2021-07-31T02:44:48.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:47 smithi045 conmon[33655]: cluster 2021-07-31T02:44:46.629158+0000 mon.a (mon.0) 3084 : cluster [DBG] osdmap e2061: 8 total, 8 up, 8 in 2021-07-31T02:44:48.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:47 smithi045 conmon[33655]: cluster 2021-07-31T02:44:47.587540+0000 mon.a (mon.0) 2021-07-31T02:44:48.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:47 smithi045 conmon[33655]: 3085 : cluster [DBG] osdmap e2062: 8 total, 8 up, 8 in 2021-07-31T02:44:48.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:47 smithi045 conmon[39718]: cluster 2021-07-31T02:44:46.624542 2021-07-31T02:44:48.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:47 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 2469 : cluster [DBG] pgmap v3633: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 230 KiB/s rd, 4.9 KiB/s wr, 314 op/s 2021-07-31T02:44:48.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:47 smithi045 conmon[39718]: cluster 2021-07-31T02:44:48.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:47 smithi045 conmon[39718]: 2021-07-31T02:44:46.629158+0000 mon.a (mon 2021-07-31T02:44:48.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:47 smithi045 conmon[39718]: .0) 3084 : cluster [DBG] osdmap e2061: 8 total, 8 up, 8 in 2021-07-31T02:44:48.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:47 smithi045 conmon[39718]: cluster 2021-07-31T02: 2021-07-31T02:44:48.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:47 smithi045 conmon[39718]: 44:47.587540+0000 mon.a (mon.0) 3085 2021-07-31T02:44:48.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:47 smithi045 conmon[39718]: : cluster [DBG] osdmap e2062: 8 total, 8 up, 8 in 2021-07-31T02:44:50.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:49 smithi154 conmon[34553]: cluster 2021-07-31T02:44:48.594002+0000 mon.a (mon.0) 3086 : cluster [DBG] osdmap e2063: 8 total, 8 up, 8 in 2021-07-31T02:44:50.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:49 smithi154 conmon[34553]: 2021-07-31T02:44:50.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:49 smithi154 conmon[34553]: cluster 2021-07-31T02:44:48.624961+0000 mgr.x (mgr.24542) 2470 : cluster [DBG] pgmap v3637: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:44:50.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:49 smithi045 conmon[33655]: cluster 2021-07-31T02:44:48.594002+0000 mon.a (mon.0) 3086 2021-07-31T02:44:50.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:49 smithi045 conmon[33655]: : cluster [DBG] osdmap e2063: 8 total, 8 up, 8 in 2021-07-31T02:44:50.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:49 smithi045 conmon[33655]: cluster 2021-07-31T02:44:48.624961+0000 mgr.x (mgr.24542) 2470 : cluster [DBG] pgmap v3637: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:44:50.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:49 smithi045 conmon[39718]: cluster 2021-07-31T02:44:48.594002+0000 2021-07-31T02:44:50.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:49 smithi045 conmon[39718]: mon.a (mon.0) 3086 : cluster [DBG] osdmap e2063: 8 total, 8 up, 8 in 2021-07-31T02:44:50.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:49 smithi045 conmon[39718]: cluster 2021-07-31 2021-07-31T02:44:50.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:49 smithi045 conmon[39718]: T02:44:48.624961+0000 mgr.x (mgr.24542) 2470 : cluster [DBG] pgmap v3637: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:44:51.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:50 smithi154 conmon[34553]: cluster 2021-07-31T02:44:49.599333+0000 mon.a (mon.0) 3087 2021-07-31T02:44:51.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:50 smithi154 conmon[34553]: : cluster [DBG] osdmap e2064: 8 total, 8 up, 8 in 2021-07-31T02:44:51.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:50 smithi045 conmon[33655]: cluster 2021-07-31T02:44:51.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:50 smithi045 conmon[33655]: 2021-07-31T02:44:49.599333+0000 2021-07-31T02:44:51.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:50 smithi045 conmon[33655]: mon.a (mon.0) 3087 : cluster [DBG] osdmap e2064: 8 total, 8 up, 8 in 2021-07-31T02:44:51.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:50 smithi045 conmon[39718]: cluster 2021-07-31T02:44:49.599333+0000 mon.a (mon.0) 2021-07-31T02:44:51.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:50 smithi045 conmon[39718]: 3087 : cluster [DBG] osdmap e2064: 8 total, 8 up, 8 in 2021-07-31T02:44:52.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:51 smithi154 conmon[34553]: cluster 2021-07-31T02:44:50.601442+0000 mon.a (mon.0) 3088 : cluster [DBG] 2021-07-31T02:44:52.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:51 smithi154 conmon[34553]: osdmap e2065: 8 total, 8 up, 8 in 2021-07-31T02:44:52.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:51 smithi154 conmon[34553]: cluster 2021-07-31T02:44:50.625412+0000 mgr.x (mgr.24542) 2471 : cluster [DBG] pgmap v3640: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 200 KiB/s rd, 5.0 KiB/s wr, 273 op/s 2021-07-31T02:44:52.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:51 smithi045 conmon[39718]: cluster 2021-07-31T02:44:50.601442+0000 mon.a (mon.0) 3088 : cluster 2021-07-31T02:44:52.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:51 smithi045 conmon[39718]: [DBG] osdmap e2065: 8 total, 8 up, 8 in 2021-07-31T02:44:52.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:51 smithi045 conmon[39718]: cluster 2021-07-31T02:44:50.625412+0000 mgr.x (mgr.24542) 2471 : cluster [DBG] pgmap v3640: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 200 KiB/s rd, 5.0 KiB/s wr, 273 op/s 2021-07-31T02:44:52.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:51 smithi045 conmon[33655]: cluster 2021-07-31T02:44:50.601442+0000 mon.a (mon.0) 3088 : cluster [DBG] osdmap e2065: 8 total, 8 up, 8 in 2021-07-31T02:44:52.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:51 smithi045 conmon[33655]: cluster 2021- 2021-07-31T02:44:52.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:51 smithi045 conmon[33655]: 07-31T02:44:50.625412+0000 mgr.x (mgr.24542) 2471 : cluster [DBG] pgmap v3640: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 200 KiB/s rd, 5.0 KiB/s wr, 273 op/s 2021-07-31T02:44:53.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:52 smithi154 conmon[34553]: cluster 2021-07-31T02:44:51.613277+0000 mon.a (mon.0) 3089 : cluster [DBG] osdmap e2066: 8 total, 8 up, 8 in 2021-07-31T02:44:53.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:52 smithi154 conmon[34553]: cluster 2021-07-31T02:44:52.589461+0000 mon.a (mon.0) 3090 : cluster [DBG] osdmap e2067: 8 total, 8 up, 8 in 2021-07-31T02:44:53.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:52 smithi045 conmon[33655]: cluster 2021-07-31T02:44:51.613277+0000 mon.a (mon.0) 2021-07-31T02:44:53.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:52 smithi045 conmon[33655]: 3089 : cluster [DBG] osdmap e2066: 8 total, 8 up, 8 in 2021-07-31T02:44:53.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:52 smithi045 conmon[33655]: cluster 2021-07-31T02:44:52.589461+0000 mon.a (mon.0) 3090 : cluster [DBG] osdmap e2067: 8 total, 8 up, 8 in 2021-07-31T02:44:53.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:52 smithi045 conmon[39718]: cluster 2021-07-31T02:44:51 2021-07-31T02:44:53.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:52 smithi045 conmon[39718]: .613277+0000 mon.a (mon.0) 3089 : cluster [DBG] osdmap e2066: 8 total, 8 up, 8 in 2021-07-31T02:44:53.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:52 smithi045 conmon[39718]: cluster 2021-07-31T 2021-07-31T02:44:53.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:52 smithi045 conmon[39718]: 02:44:52.589461+0000 mon.a (mon.0 2021-07-31T02:44:53.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:52 smithi045 conmon[39718]: ) 3090 : cluster [DBG] osdmap e2067: 8 total, 8 up, 8 in 2021-07-31T02:44:54.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:53 smithi154 conmon[34553]: cluster 2021-07-31T02:44:52.625792+0000 mgr.x (mgr.24542 2021-07-31T02:44:54.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:53 smithi154 conmon[34553]: ) 2472 : cluster [DBG] pgmap v3643: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 200 KiB/s rd, 5.0 KiB/s wr, 273 op/s 2021-07-31T02:44:54.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:53 smithi045 conmon[39718]: cluster 2021-07-31T02:44:52.625792+0000 mgr.x ( 2021-07-31T02:44:54.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:53 smithi045 conmon[39718]: mgr.24542) 2472 : cluster [DBG] pgmap v3643: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 200 KiB/s rd, 5.0 KiB/s wr, 273 op/s 2021-07-31T02:44:54.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:53 smithi045 conmon[33655]: cluster 2021-07-31T02:44:52.625792+0000 2021-07-31T02:44:54.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:53 smithi045 conmon[33655]: mgr.x (mgr.24542) 2472 : cluster [DBG] pgmap v3643: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 200 KiB/s rd, 5.0 KiB/s wr, 273 op/s 2021-07-31T02:44:55.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:54 smithi154 conmon[34553]: cluster 2021-07-31T02:44:53.620712+0000 mon.a (mon.0) 3091 : cluster [DBG] osdmap e2068: 8 total, 8 up, 8 in 2021-07-31T02:44:55.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:54 smithi045 conmon[39718]: cluster 2021-07-31T02:44:53.620712+0000 mon.a (mon.0) 2021-07-31T02:44:55.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:54 smithi045 conmon[39718]: 3091 : cluster [DBG] osdmap e2068: 8 total, 8 up, 8 in 2021-07-31T02:44:55.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:54 smithi045 conmon[33655]: cluster 2021-07-31T02:44:53.620712+0000 mon.a (mon.0 2021-07-31T02:44:55.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:54 smithi045 conmon[33655]: ) 3091 : cluster [DBG] osdmap e2068: 8 total, 8 up, 8 in 2021-07-31T02:44:56.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:55 smithi154 conmon[34553]: cluster 2021-07-31T02:44:54.626199+0000 mgr.x (mgr 2021-07-31T02:44:56.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:55 smithi154 conmon[34553]: .24542) 2473 : cluster [DBG] pgmap v3646: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 167 KiB/s rd, 5.0 KiB/s wr, 229 op/s 2021-07-31T02:44:56.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:55 smithi154 conmon[34553]: cluster 2021-07-31T02:44:54.626552+0000 mon.a (mon.0) 3092 : cluster [DBG] osdmap e2069: 8 total, 8 up, 8 in 2021-07-31T02:44:56.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:55 smithi045 conmon[39718]: cluster 2021-07-31T02:44:54.626199+0000 mgr.x 2021-07-31T02:44:56.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:55 smithi045 conmon[39718]: (mgr.24542) 2473 : cluster [DBG] pgmap v3646: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 167 KiB/s rd, 5.0 KiB/s wr, 229 op/s 2021-07-31T02:44:56.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:55 smithi045 conmon[39718]: cluster 2021-07-31T02:44 2021-07-31T02:44:56.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:55 smithi045 conmon[39718]: :54.626552+0000 mon.a (mon.0) 3092 : cluster [DBG] osdmap e2069: 8 total, 8 up, 8 in 2021-07-31T02:44:56.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:55 smithi045 conmon[33655]: cluster 2021-07-31T02:44:54.626199+0000 mgr.x (mgr.24542) 2473 2021-07-31T02:44:56.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:55 smithi045 conmon[33655]: : cluster [DBG] pgmap v3646: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 167 KiB/s rd, 5.0 KiB/s wr, 229 op/s 2021-07-31T02:44:56.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:55 smithi045 conmon[33655]: cluster 2021-07-31T02:44:54.626552+0000 mon.a (mon.0) 3092 : cluster [DBG] osdmap e2069: 8 total, 8 up, 8 in 2021-07-31T02:44:57.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:56 smithi154 conmon[34553]: cluster 2021-07-31T02:44:55.634603+0000 mon.a (mon.0) 3093 : cluster [DBG] osdmap e2070: 8 total, 8 up, 8 in 2021-07-31T02:44:57.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:56 smithi045 conmon[39718]: cluster 2021-07-31T02:44:55.634603+0000 mon.a 2021-07-31T02:44:57.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:56 smithi045 conmon[39718]: (mon.0) 3093 : cluster [DBG] osdmap e2070: 8 total, 8 up, 8 in 2021-07-31T02:44:57.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:56 smithi045 conmon[33655]: cluster 2021-07-31T02:44:55.634603+0000 mon.a (mon.0) 2021-07-31T02:44:57.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:56 smithi045 conmon[33655]: 3093 : cluster [DBG] osdmap e2070: 8 total, 8 up, 8 in 2021-07-31T02:44:58.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:57 smithi154 conmon[34553]: cluster 2021-07-31T02:44:56.626565+0000 mgr.x (mgr.24542) 2474 : cluster [DBG] pgmap v3648: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 166 KiB/s rd, 5.0 KiB/s wr, 227 op/s 2021-07-31T02:44:58.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:57 smithi154 conmon[34553]: cluster 2021-07-31T02:44:56. 2021-07-31T02:44:58.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:57 smithi154 conmon[34553]: 643761+0000 mon.a (mon.0) 3094 : cluster [DBG] osdmap e2071: 8 total, 8 up, 8 in 2021-07-31T02:44:58.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:57 smithi154 conmon[34553]: cluster 2021-07-31T02:44:57.591044+0000 mon.a (mon.0) 3095 : cluster [DBG] osdmap e2072: 8 total, 8 up, 8 in 2021-07-31T02:44:58.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:57 smithi045 conmon[33655]: cluster 2021-07-31T02:44:56.626565+0000 mgr.x 2021-07-31T02:44:58.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:57 smithi045 conmon[33655]: (mgr.24542) 2474 : cluster [DBG] pgmap v3648: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 166 KiB/s rd, 5.0 KiB/s wr, 227 op/s 2021-07-31T02:44:58.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:57 smithi045 conmon[33655]: cluster 2021-07-31T02:44:56.643761+0000 2021-07-31T02:44:58.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:57 smithi045 conmon[33655]: mon.a (mon.0) 3094 : cluster [DBG] osdmap e2071: 8 total, 8 up, 8 in 2021-07-31T02:44:58.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:57 smithi045 conmon[33655]: cluster 2021-07-31T02:44:58.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:57 smithi045 conmon[33655]: 2021-07-31T02:44:57.591044 2021-07-31T02:44:58.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:57 smithi045 conmon[33655]: +0000 mon.a (mon.0) 3095 : cluster 2021-07-31T02:44:58.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:57 smithi045 conmon[33655]: [DBG] osdmap e2072: 8 total, 8 up, 8 in 2021-07-31T02:44:58.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:57 smithi045 conmon[39718]: cluster 2021-07-31T02:44:56.626565+0000 mgr.x (mgr.24542) 2474 : cluster [DBG] pgmap v3648: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 166 KiB/s rd, 5.0 KiB/s wr, 227 op/s 2021-07-31T02:44:58.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:57 smithi045 conmon[39718]: cluster 2021-07 2021-07-31T02:44:58.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:57 smithi045 conmon[39718]: -31T02:44:56.643761 2021-07-31T02:44:58.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:57 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2021-07-31T02:44:58.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:57 smithi045 conmon[39718]: 3094 : cluster [DBG] osdmap e2071: 8 total, 8 up, 8 in 2021-07-31T02:44:58.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:57 smithi045 conmon[39718]: cluster 2021- 2021-07-31T02:44:58.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:57 smithi045 conmon[39718]: 07-31T02:44:57. 2021-07-31T02:44:58.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:57 smithi045 conmon[39718]: 591044+0000 mon.a (mon.0) 2021-07-31T02:44:58.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:57 smithi045 conmon[39718]: 3095 : cluster [DBG] osdmap e2072: 8 total, 8 up, 8 in 2021-07-31T02:45:00.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:59 smithi154 conmon[34553]: cluster 2021-07-31T02:44:58.598212+0000 mon.a (mon.0) 3096 : cluster [DBG] osdmap e2073: 8 total, 8 up, 8 in 2021-07-31T02:45:00.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:44:59 smithi154 conmon[34553]: cluster 2021-07-31T02:44:58.626963+0000 mgr.x (mgr.24542) 2475 : cluster [DBG] pgmap v3652: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:45:00.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:59 smithi045 conmon[33655]: cluster 2021-07-31T02:44:58.598212+0000 mon.a (mon.0) 2021-07-31T02:45:00.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:59 smithi045 conmon[33655]: 3096 : cluster [DBG] osdmap e2073: 8 total, 8 up, 8 in 2021-07-31T02:45:00.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:44:59 smithi045 conmon[33655]: cluster 2021-07-31T02:44:58.626963+0000 mgr.x (mgr.24542) 2475 : cluster [DBG] pgmap v3652: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:45:00.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:59 smithi045 conmon[39718]: cluster 2021-07-31T02:44:58.598212+0000 mon.a 2021-07-31T02:45:00.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:59 smithi045 conmon[39718]: (mon.0) 3096 : cluster [DBG] osdmap e2073: 8 total, 8 up, 8 in 2021-07-31T02:45:00.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:59 smithi045 conmon[39718]: cluster 2021-07 2021-07-31T02:45:00.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:59 smithi045 conmon[39718]: -31T02:44:58.626963+0000 mgr.x (mgr 2021-07-31T02:45:00.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:59 smithi045 conmon[39718]: .24542) 2475 : cluster [DBG] pgmap v3652: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:45:00.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:44:59 smithi045 conmon[39718]: 2021-07-31T02:45:01.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:00 smithi154 conmon[34553]: cluster 2021-07-31T02:44:59.603359+0000 mon.a (mon.0) 3097 : cluster [DBG] osdmap e2074: 8 total, 8 up, 8 in 2021-07-31T02:45:01.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:00 smithi045 conmon[39718]: cluster 2021-07-31T02:44:59.603359+0000 mon.a (mon.0) 3097 : cluster [DBG] osdmap e2074: 8 total, 8 up, 8 in 2021-07-31T02:45:01.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:00 smithi045 conmon[33655]: cluster 2021-07-31T02:44:59.603359+0000 mon.a (mon.0) 3097 : cluster [DBG] osdmap e2074: 8 total, 8 up, 8 in 2021-07-31T02:45:02.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:01 smithi154 conmon[34553]: cluster 2021-07-31T02:45:02.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:01 smithi154 conmon[34553]: 2021-07-31T02:45:00.608993 2021-07-31T02:45:02.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:01 smithi154 conmon[34553]: +0000 mon.a (mon.0) 3098 : cluster [DBG] osdmap e2075: 8 total, 8 up, 8 in 2021-07-31T02:45:02.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:01 smithi154 conmon[34553]: cluster 2021-07-31T 2021-07-31T02:45:02.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:01 smithi154 conmon[34553]: 02:45:00.627395+0000 mgr.x (mgr.24542) 2476 : cluster 2021-07-31T02:45:02.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:01 smithi154 conmon[34553]: [DBG] pgmap v3655: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 136 KiB/s rd, 5.0 KiB/s wr, 186 op/s 2021-07-31T02:45:02.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:01 smithi045 conmon[33655]: cluster 2021-07-31T02:45:00.608993+0000 mon.a (mon.0) 2021-07-31T02:45:02.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:01 smithi045 conmon[33655]: 3098 : cluster [DBG] osdmap e2075: 8 total, 8 up, 8 in 2021-07-31T02:45:02.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:01 smithi045 conmon[33655]: cluster 2021-07-31T02:45:00.627395+0000 mgr.x (mgr.24542) 2476 : cluster [DBG] pgmap v3655: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 136 KiB/s rd, 5.0 KiB/s wr, 186 op/s 2021-07-31T02:45:02.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:01 smithi045 conmon[39718]: cluster 2021-07-31T02:45: 2021-07-31T02:45:02.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:01 smithi045 conmon[39718]: 00.608993+0000 mon.a (mon.0) 3098 : cluster [DBG] osdmap e2075: 8 total, 8 up, 8 in 2021-07-31T02:45:02.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:01 smithi045 conmon[39718]: cluster 2021-07-31T02:45:02.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:01 smithi045 conmon[39718]: 2021-07-31T02:45:00.627395 2021-07-31T02:45:02.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:01 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 2476 : cluster 2021-07-31T02:45:02.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:01 smithi045 conmon[39718]: [DBG] pgmap v3655: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 136 KiB/s rd, 5.0 KiB/s wr, 186 op/s 2021-07-31T02:45:03.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:02 smithi154 conmon[34553]: cluster 2021-07-31T02:45:01.614869+0000 mon.a (mon.0) 3099 : cluster 2021-07-31T02:45:03.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:02 smithi154 conmon[34553]: [DBG] osdmap e2076: 8 total, 8 up, 8 in 2021-07-31T02:45:03.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:02 smithi154 conmon[34553]: cluster 2021-07-31T02:45:02.592427+0000 mon.a (mon.0) 3100 : cluster [DBG] osdmap e2077: 8 total, 8 up, 8 in 2021-07-31T02:45:03.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:02 smithi045 conmon[39718]: cluster 2021-07-31T02:45:03.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:02 smithi045 conmon[39718]: 2021-07-31T02:45:03.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:02 smithi045 conmon[39718]: 2021 2021-07-31T02:45:03.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:02 smithi045 conmon[39718]: - 2021-07-31T02:45:03.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:02 smithi045 conmon[39718]: 07 2021-07-31T02:45:03.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:02 smithi045 conmon[39718]: - 2021-07-31T02:45:03.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:02 smithi045 conmon[39718]: 31 2021-07-31T02:45:03.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:02 smithi045 conmon[39718]: T 2021-07-31T02:45:03.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:02 smithi045 conmon[39718]: 02 2021-07-31T02:45:03.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:02 smithi045 conmon[39718]: : 2021-07-31T02:45:03.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:02 smithi045 conmon[39718]: 45 2021-07-31T02:45:03.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:02 smithi045 conmon[39718]: : 2021-07-31T02:45:03.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:02 smithi045 conmon[39718]: 01 2021-07-31T02:45:03.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:02 smithi045 conmon[39718]: . 2021-07-31T02:45:03.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:02 smithi045 conmon[39718]: 614869+0000 mon.a (mon.0) 3099 : cluster [DBG] osdmap e2076: 8 total, 8 up, 8 in 2021-07-31T02:45:03.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:02 smithi045 conmon[39718]: cluster 2021-07-31T02:45 2021-07-31T02:45:03.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:02 smithi045 conmon[39718]: :02.592427+0000 mon.a (mon.0) 3100 : cluster [DBG] 2021-07-31T02:45:03.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:02 smithi045 conmon[39718]: osdmap e2077: 8 total, 8 up, 8 in 2021-07-31T02:45:03.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:02 smithi045 conmon[33655]: cluster 2021-07-31T02:45:01.614869+0000 mon.a ( 2021-07-31T02:45:03.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:02 smithi045 conmon[33655]: mon.0) 3099 : cluster [DBG] osdmap e2076: 8 total, 8 up, 8 in 2021-07-31T02:45:03.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:02 smithi045 conmon[33655]: cluster 2021-07-31T02:45:02.592427+0000 mon.a (mon.0) 3100 : cluster [DBG] osdmap e2077: 8 total, 8 up, 8 in 2021-07-31T02:45:04.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:03 smithi154 conmon[34553]: cluster 2021-07-31T02:45:02.627780+0000 mgr.x (mgr.24542) 2477 : cluster [DBG] 2021-07-31T02:45:04.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:03 smithi154 conmon[34553]: pgmap v3658: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 135 KiB/s rd, 5.0 KiB/s wr, 186 op/s 2021-07-31T02:45:04.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:03 smithi045 conmon[39718]: cluster 2021-07-31T02:45:02.627780+0000 2021-07-31T02:45:04.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:03 smithi045 conmon[39718]: mgr.x (mgr.24542) 2477 : cluster [DBG] pgmap v3658: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 135 KiB/s rd, 5.0 KiB/s wr, 186 op/s 2021-07-31T02:45:04.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:03 smithi045 conmon[33655]: cluster 2021-07-31T02:45:02.627780+0000 mgr.x (mgr.24542) 2477 : 2021-07-31T02:45:04.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:03 smithi045 conmon[33655]: cluster [DBG] pgmap v3658: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 135 KiB/s rd, 5.0 KiB/s wr, 186 op/s 2021-07-31T02:45:05.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:04 smithi154 conmon[34553]: cluster 2021-07-31T02:45:03.621546+0000 mon.a (mon.0) 3101 : cluster [DBG] 2021-07-31T02:45:05.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:04 smithi154 conmon[34553]: osdmap e2078: 8 total, 8 up, 8 in 2021-07-31T02:45:05.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:04 smithi045 conmon[39718]: cluster 2021-07-31T02:45:03.621546+0000 mon.a (mon.0) 3101 : cluster [DBG] osdmap e2078: 8 total, 8 up, 8 in 2021-07-31T02:45:05.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:04 smithi045 conmon[33655]: cluster 2021-07-31T02:45:03.621546+0000 mon.a (mon.0) 3101 : cluster [DBG] 2021-07-31T02:45:05.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:04 smithi045 conmon[33655]: osdmap e2078: 8 total, 8 up, 8 in 2021-07-31T02:45:06.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:05 smithi154 conmon[34553]: cluster 2021-07-31T02:45:04.627342+0000 mon.a (mon.0) 3102 : cluster [DBG] 2021-07-31T02:45:06.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:05 smithi154 conmon[34553]: osdmap e2079: 8 total, 8 up, 8 in 2021-07-31T02:45:06.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:05 smithi154 conmon[34553]: cluster 2021-07-31T02:45:04.628169+0000 mgr.x (mgr.24542) 2478 : cluster [DBG] pgmap v3661: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 103 KiB/s rd, 5.0 KiB/s wr, 142 op/s 2021-07-31T02:45:06.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:05 smithi045 conmon[39718]: cluster 2021-07-31T02:45:04.627342+0000 mon.a ( 2021-07-31T02:45:06.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:05 smithi045 conmon[39718]: mon.0) 3102 : cluster [DBG] osdmap e2079: 8 total, 8 up, 8 in 2021-07-31T02:45:06.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:05 smithi045 conmon[39718]: cluster 2021-07-31T02:45:04.628169+0000 mgr.x (mgr. 2021-07-31T02:45:06.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:05 smithi045 conmon[39718]: 24542) 2478 : cluster [DBG] pgmap v3661: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 103 KiB/s rd, 5.0 KiB/s wr, 142 op/s 2021-07-31T02:45:06.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:05 smithi045 conmon[33655]: cluster 2021-07-31T02:45:04.627342+0000 mon.a (mon.0) 3102 2021-07-31T02:45:06.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:05 smithi045 conmon[33655]: : cluster [DBG] osdmap e2079: 8 total, 8 up, 8 in 2021-07-31T02:45:06.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:05 smithi045 conmon[33655]: cluster 2021-07-31T02:45:04.628169+0000 mgr.x (mgr.24542) 2478 : cluster [DBG] pgmap v3661: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 103 KiB/s rd, 5.0 KiB/s wr, 142 op/s 2021-07-31T02:45:06.638 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:06 smithi154 conmon[34553]: cluster 2021-07-31T02:45:05.633046+0000 mon.a (mon.0) 3103 : cluster [DBG] 2021-07-31T02:45:07.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:06 smithi045 conmon[33655]: cluster 2021-07-31T02:45:05.633046+0000 2021-07-31T02:45:07.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:06 smithi045 conmon[33655]: mon.a (mon.0) 3103 : cluster [DBG] osdmap e2080: 8 total, 8 up, 8 in 2021-07-31T02:45:07.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:06 smithi045 conmon[39718]: cluster 2021-07-31T02:45:05.633046+0000 mon.a (mon. 2021-07-31T02:45:07.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:06 smithi045 conmon[39718]: 0) 3103 : cluster [DBG] osdmap e2080: 8 total, 8 up, 8 in 2021-07-31T02:45:08.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:06 smithi154 conmon[34553]: osdmap e2080: 8 total, 8 up, 8 in 2021-07-31T02:45:08.140 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:07 smithi154 conmon[34553]: cluster 2021-07-31T02:45:06.628764+0000 mgr.x (mgr.24542) 2479 : cluster [DBG] pgmap v3663: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 102 KiB/s rd, 5.0 KiB/s wr, 141 op/s 2021-07-31T02:45:08.141 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:07 smithi154 conmon[34553]: cluster 2021-07-31T02:45:06.634147+0000 mon.a (mon.0) 3104 : cluster [DBG] osdmap e2081: 8 total, 8 up, 8 in 2021-07-31T02:45:08.142 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:07 smithi154 conmon[34553]: cluster 2021-07-31T02:45:07.593840+0000 mon.a (mon.0) 3105 : cluster [DBG] osdmap e2082: 8 total, 8 up, 8 in 2021-07-31T02:45:08.143 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:07 smithi045 conmon[33655]: cluster 2021-07-31T02:45:06.628764+0000 mgr.x (mgr 2021-07-31T02:45:08.143 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:07 smithi045 conmon[33655]: .24542) 2479 : cluster [DBG] pgmap v3663: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 102 KiB/s rd, 5.0 KiB/s wr, 141 op/s 2021-07-31T02:45:08.145 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:07 smithi045 conmon[33655]: cluster 2021-07-31T02:45:06.634147+0000 mon.a (mon.0) 2021-07-31T02:45:08.145 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:07 smithi045 conmon[33655]: 3104 : cluster [DBG] osdmap e2081: 8 total, 8 up, 8 in 2021-07-31T02:45:08.146 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:07 smithi045 conmon[33655]: cluster 2021-07 2021-07-31T02:45:08.146 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:07 smithi045 conmon[33655]: -31T02:45:07.593840+0000 mon.a (mon 2021-07-31T02:45:08.146 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:07 smithi045 conmon[33655]: .0) 3105 : cluster [DBG] osdmap e2082: 8 total, 8 up, 8 in 2021-07-31T02:45:08.147 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:07 smithi045 conmon[39718]: cluster 2021-07-31T02:45: 2021-07-31T02:45:08.147 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:07 smithi045 conmon[39718]: 06.628764+0000 mgr.x (mgr.24542) 2479 : cluster [DBG] pgmap v3663: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 102 KiB/s rd, 5.0 KiB/s wr, 141 op/s 2021-07-31T02:45:08.147 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:07 smithi045 conmon[39718]: cluster 2021 2021-07-31T02:45:08.147 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:07 smithi045 conmon[39718]: -07-31T02:45: 2021-07-31T02:45:08.148 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:07 smithi045 conmon[39718]: 06.634147+0000 mon.a ( 2021-07-31T02:45:08.148 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:07 smithi045 conmon[39718]: mon.0) 3104 : cluster 2021-07-31T02:45:08.148 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:07 smithi045 conmon[39718]: [DBG] osdmap e2081: 8 total, 8 up, 8 in 2021-07-31T02:45:08.148 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:07 smithi045 conmon[39718]: cluster 2021- 2021-07-31T02:45:08.148 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:07 smithi045 conmon[39718]: 07-31T02:45:07.593840+0000 2021-07-31T02:45:08.149 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:07 smithi045 conmon[39718]: mon.a (mon.0) 3105 : cluster [DBG] 2021-07-31T02:45:08.149 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:07 smithi045 conmon[39718]: osdmap e2082: 8 total, 8 up, 8 in 2021-07-31T02:45:10.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:09 smithi154 conmon[34553]: cluster 2021-07-31T02:45:08.600401+0000 mon.a (mon.0) 3106 : 2021-07-31T02:45:10.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:09 smithi154 conmon[34553]: cluster [DBG] osdmap e2083: 8 total, 8 up, 8 in 2021-07-31T02:45:10.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:09 smithi154 conmon[34553]: cluster 2021-07-31T02:45:08.629169+0000 mgr.x (mgr.24542) 2480 : cluster [DBG] pgmap v3667: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:45:10.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:09 smithi045 conmon[33655]: cluster 2021-07-31T02:45:08.600401+0000 mon.a (mon.0) 3106 2021-07-31T02:45:10.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:09 smithi045 conmon[33655]: : cluster [DBG] osdmap e2083: 8 total, 8 up, 8 in 2021-07-31T02:45:10.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:09 smithi045 conmon[33655]: cluster 2021-07-31T02:45:08.629169+0000 mgr.x (mgr.24542) 2480 : cluster [DBG] pgmap v3667: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:45:10.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:09 smithi045 conmon[39718]: cluster 2021-07-31T02:45:08.600401+0000 mon.a 2021-07-31T02:45:10.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:09 smithi045 conmon[39718]: (mon.0) 3106 : cluster [DBG] osdmap e2083: 8 total, 8 up, 8 in 2021-07-31T02:45:10.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:09 smithi045 conmon[39718]: cluster 2021-07-31T02:45:08.629169+0000 mgr.x (mgr.24542) 2480 : cluster [DBG] 2021-07-31T02:45:10.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:09 smithi045 conmon[39718]: pgmap v3667: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:45:11.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:10 smithi154 conmon[34553]: cluster 2021-07-31T02:45:11.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:10 smithi154 conmon[34553]: 2021-07-31T02:45:11.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:10 smithi154 conmon[34553]: 2021-07-31T02: 2021-07-31T02:45:11.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:10 smithi154 conmon[34553]: 45:09.606887+0000 mon.a (mon.0) 3107 : cluster [DBG] osdmap e2084: 8 total, 8 up, 8 in 2021-07-31T02:45:11.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:10 smithi045 conmon[39718]: cluster 2021-07-31T02:45:09.606887+0000 mon.a (mon. 2021-07-31T02:45:11.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:10 smithi045 conmon[39718]: 0) 3107 : cluster [DBG] osdmap e2084: 8 total, 8 up, 8 in 2021-07-31T02:45:11.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:10 smithi045 conmon[33655]: cluster 2021-07-31T02:45:09.606887+0000 mon.a 2021-07-31T02:45:11.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:10 smithi045 conmon[33655]: (mon.0) 3107 : cluster [DBG] osdmap e2084: 8 total, 8 up, 8 in 2021-07-31T02:45:11.467 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:45:11.753 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:45:11.753 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 60m ago 60m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:45:11.753 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 57m ago 60m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:45:11.754 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 57m ago 64m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:45:11.754 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 60m ago 66m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:45:11.754 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 60m ago 66m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:45:11.754 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 57m ago 64m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:45:11.755 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 60m ago 64m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:45:11.755 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 60m ago 61m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:45:11.755 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 57m ago 61m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:45:11.755 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 60m ago 63m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:45:11.756 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 60m ago 63m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:45:11.756 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 60m ago 63m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:45:11.756 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 60m ago 62m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:45:11.756 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 57m ago 62m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:45:11.757 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 57m ago 62m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:45:11.757 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 57m ago 61m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:45:11.757 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 57m ago 61m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:45:11.757 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 57m ago 61m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:45:12.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:11 smithi154 conmon[34553]: cluster 2021-07-31T02:45:10.609272+0000 mon.a (mon.0) 3108 : cluster [DBG] osdmap e2085: 8 total, 8 up, 8 in 2021-07-31T02:45:12.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:11 smithi154 conmon[34553]: 2021-07-31T02:45:12.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:11 smithi154 conmon[34553]: cluster 2021-07-31T02:45:10.629548+0000 mgr.x (mgr.24542) 2481 : cluster [DBG] pgmap v3670: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 70 KiB/s rd, 5.0 KiB/s wr, 98 op/s 2021-07-31T02:45:12.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:11 smithi154 conmon[34553]: audit 2021-07-31T02:45:11.455571+0000 mgr.x (mgr.24542) 2482 : 2021-07-31T02:45:12.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:11 smithi154 conmon[34553]: audit [DBG] from='client.17988 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:45:12.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:11 smithi045 conmon[33655]: cluster 2021-07-31T02:45:10.609272+0000 mon.a (mon.0 2021-07-31T02:45:12.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:11 smithi045 conmon[33655]: ) 3108 : cluster [DBG] osdmap e2085: 8 total, 8 up, 8 in 2021-07-31T02:45:12.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:11 smithi045 conmon[33655]: cluster 2021-07-31T02:45:10.629548+0000 mgr.x (mgr.24542) 2481 : cluster [DBG] pgmap v3670: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 70 KiB/s rd, 5.0 KiB/s wr, 98 op/s 2021-07-31T02:45:12.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:11 smithi045 conmon[33655]: audit 2021-07-31T02:45:11.455571+0000 mgr.x (mgr.24542) 2021-07-31T02:45:12.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:11 smithi045 conmon[33655]: 2482 : audit [DBG] from='client.17988 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:45:12.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:11 smithi045 conmon[39718]: cluster 2021-07-31T02:45 2021-07-31T02:45:12.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:11 smithi045 conmon[39718]: :10.609272+0000 mon.a (mon.0) 3108 : cluster [DBG] osdmap e2085: 8 total, 8 up, 8 in 2021-07-31T02:45:12.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:11 smithi045 conmon[39718]: cluster 2021-07-31T02:45:12.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:11 smithi045 conmon[39718]: 2021-07-31T02 2021-07-31T02:45:12.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:11 smithi045 conmon[39718]: :45:10.629548+0000 mgr.x 2021-07-31T02:45:12.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:11 smithi045 conmon[39718]: (mgr.24542) 2481 : cluster 2021-07-31T02:45:12.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:11 smithi045 conmon[39718]: [DBG] pgmap v3670: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 70 KiB/s rd, 5.0 KiB/s wr, 98 op/s 2021-07-31T02:45:12.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:11 smithi045 conmon[39718]: audit 2021-07-31T02:45:12.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:11 smithi045 conmon[39718]: 2021-07-31T02:45: 2021-07-31T02:45:12.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:11 smithi045 conmon[39718]: 11.455571+0000 mgr.x (mgr.24542) 2021-07-31T02:45:12.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:11 smithi045 conmon[39718]: 2482 : audit [DBG] from='client.17988 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:45:12.087 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:45:12.088 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:45:12.088 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:45:12.088 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:45:12.089 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:45:12.089 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:45:12.089 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:45:12.090 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:45:12.090 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:45:12.090 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:45:12.090 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:45:12.090 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:45:12.091 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:45:12.091 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:45:12.091 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:45:12.091 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:45:12.092 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:45:12.690 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestImage.test_many_snaps ... ok 2021-07-31T02:45:13.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:12 smithi154 conmon[34553]: cluster 2021-07-31T02:45:11.618440+0000 mon.a (mon.0) 3109 : cluster 2021-07-31T02:45:13.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:12 smithi154 conmon[34553]: [DBG] osdmap e2086: 8 total, 8 up, 8 in 2021-07-31T02:45:13.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:12 smithi154 conmon[34553]: audit 2021-07-31T02:45:11.744857+0000 mgr.x (mgr.24542) 2483 : audit [DBG] from='client.17994 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:45:13.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:12 smithi154 conmon[34553]: audit 2021-07-31T02:45:12.087837+0000 mon.a (mon 2021-07-31T02:45:13.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:12 smithi154 conmon[34553]: .0) 3110 : audit [DBG] from='client.? 172.21.15.45:0/1205283150' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:45:13.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:12 smithi154 conmon[34553]: cluster 2021-07-31T 2021-07-31T02:45:13.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:12 smithi154 conmon[34553]: 02:45:12.594726+0000 mon.a (mon.0) 3111 : cluster [DBG] 2021-07-31T02:45:13.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:12 smithi154 conmon[34553]: osdmap e2087: 8 total, 8 up, 8 in 2021-07-31T02:45:13.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:12 smithi045 conmon[33655]: cluster 2021-07-31T02:45:11.618440+0000 mon.a (mon.0) 3109 2021-07-31T02:45:13.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:12 smithi045 conmon[33655]: : cluster [DBG] osdmap e2086: 8 total, 8 up, 8 in 2021-07-31T02:45:13.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:12 smithi045 conmon[33655]: audit 2021-07-31T02:45:11.744857+0000 mgr.x (mgr.24542) 2483 : audit [DBG] from='client.17994 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:45:13.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:12 smithi045 conmon[33655]: audit 2021-07-31T02:45:12.087837+0000 mon.a ( 2021-07-31T02:45:13.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:12 smithi045 conmon[33655]: mon.0) 3110 : audit [DBG] from='client.? 172.21.15.45:0/1205283150' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:45:13.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:12 smithi045 conmon[33655]: cluster 2021-07-31T 2021-07-31T02:45:13.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:12 smithi045 conmon[33655]: 02:45:12.594726+0000 mon.a (mon.0) 3111 : cluster 2021-07-31T02:45:13.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:12 smithi045 conmon[33655]: [DBG] osdmap e2087: 8 total, 8 up, 8 in 2021-07-31T02:45:13.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:12 smithi045 conmon[39718]: cluster 2021-07-31T02:45:11.618440+0000 mon.a ( 2021-07-31T02:45:13.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:12 smithi045 conmon[39718]: mon.0) 3109 : cluster [DBG] osdmap e2086: 8 total, 8 up, 8 in 2021-07-31T02:45:13.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:12 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T02:45:13.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:12 smithi045 conmon[39718]: 02:45:11.744857+0000 mgr.x (mgr.24542) 2483 : audit [DBG] from='client.17994 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:45:13.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:12 smithi045 conmon[39718]: audit 2021-07-31T02:45:13.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:12 smithi045 conmon[39718]: 2021-07-31T02:45:12.087837+0000 mon.a 2021-07-31T02:45:13.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:12 smithi045 conmon[39718]: (mon.0) 3110 : audit [DBG] from='client.? 172.21.15.45:0/1205283150' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:45:13.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:12 smithi045 conmon[39718]: cluster 2021-07-31T02:45:13.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:12 smithi045 conmon[39718]: 2021-07-31T02:45:12.594726+0000 mon.a (mon 2021-07-31T02:45:13.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:12 smithi045 conmon[39718]: .0) 3111 : cluster [DBG] osdmap e2087: 8 total, 8 up, 8 in 2021-07-31T02:45:13.443 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestImage.test_metadata ... ok 2021-07-31T02:45:13.521 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestImage.test_modify_timestamp ... ok 2021-07-31T02:45:14.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:13 smithi154 conmon[34553]: cluster 2021-07-31T02:45:12.629874+0000 mgr.x (mgr. 2021-07-31T02:45:14.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:13 smithi154 conmon[34553]: 24542) 2484 : cluster [DBG] pgmap v3673: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 70 KiB/s rd, 5.0 KiB/s wr, 98 op/s 2021-07-31T02:45:14.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:13 smithi045 conmon[33655]: cluster 2021-07-31T02:45:12.629874+0000 mgr.x (mgr 2021-07-31T02:45:14.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:13 smithi045 conmon[33655]: .24542) 2484 : cluster [DBG] pgmap v3673: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 70 KiB/s rd, 5.0 KiB/s wr, 98 op/s 2021-07-31T02:45:14.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:13 smithi045 conmon[39718]: cluster 2021-07-31T02:45: 2021-07-31T02:45:14.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:13 smithi045 conmon[39718]: 12.629874+0000 mgr.x (mgr.24542) 2484 : cluster [DBG] pgmap v3673: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 70 KiB/s rd, 5.0 KiB/s wr, 98 op/s 2021-07-31T02:45:14.723 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestImage.test_protect_snap ... ok 2021-07-31T02:45:14.812 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestImage.test_read ... ok 2021-07-31T02:45:14.896 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestImage.test_read_bad_offset ... ok 2021-07-31T02:45:14.981 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestImage.test_read_with_fadvise_flags ... ok 2021-07-31T02:45:15.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:14 smithi154 conmon[34553]: cluster 2021-07-31T02:45:13.624559+0000 mon.a (mon.0) 3112 : cluster 2021-07-31T02:45:15.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:14 smithi154 conmon[34553]: [DBG] osdmap e2088: 8 total, 8 up, 8 in 2021-07-31T02:45:15.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:14 smithi045 conmon[33655]: cluster 2021-07-31T02:45:13.624559+0000 mon.a (mon.0) 3112 : 2021-07-31T02:45:15.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:14 smithi045 conmon[33655]: cluster [DBG] osdmap e2088: 8 total, 8 up, 8 in 2021-07-31T02:45:15.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:14 smithi045 conmon[39718]: cluster 2021-07-31T02:45:13 2021-07-31T02:45:15.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:14 smithi045 conmon[39718]: .624559+0000 mon.a (mon.0) 3112 : cluster [DBG] osdmap e2088: 8 total, 8 up, 8 in 2021-07-31T02:45:16.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:15 smithi154 conmon[34553]: cluster 2021-07-31T02:45:14.630221+0000 mgr.x ( 2021-07-31T02:45:16.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:15 smithi154 conmon[34553]: mgr.24542) 2485 : cluster [DBG] pgmap v3676: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 2.8 MiB/s rd, 44 KiB/s wr, 873 op/s 2021-07-31T02:45:16.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:15 smithi154 conmon[34553]: cluster 2021-07-31T02:45:14.632051+0000 mon.a (mon.0) 3113 : cluster [DBG] osdmap e2089: 8 total, 8 up, 8 in 2021-07-31T02:45:16.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:15 smithi045 conmon[33655]: cluster 2021-07-31T02:45:14.630221+0000 mgr.x (mgr.24542) 2485 : cluster [DBG] pgmap v3676: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 2.8 MiB/s rd, 44 KiB/s wr, 873 op/s 2021-07-31T02:45:16.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:15 smithi045 conmon[33655]: cluster 2021-07-31T02:45:14.632051+0000 mon.a (mon.0) 3113 : cluster [DBG] osdmap e2089: 8 total, 8 up, 8 in 2021-07-31T02:45:16.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:15 smithi045 conmon[39718]: cluster 2021-07-31T02:45:14.630221+0000 mgr.x (mgr 2021-07-31T02:45:16.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:15 smithi045 conmon[39718]: .24542) 2485 : cluster [DBG] pgmap v3676: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 2.8 MiB/s rd, 44 KiB/s wr, 873 op/s 2021-07-31T02:45:16.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:15 smithi045 conmon[39718]: cluster 2021-07-31T02:45:14.632051+0000 mon.a (mon.0) 3113 : cluster [DBG] osdmap e2089: 8 total, 8 up, 8 in 2021-07-31T02:45:16.724 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestImage.test_remove_snap ... ok 2021-07-31T02:45:17.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:16 smithi154 conmon[34553]: cluster 2021-07-31T02:45:15.636332+0000 mon.a (mon.0) 3114 : cluster 2021-07-31T02:45:17.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:16 smithi154 conmon[34553]: [DBG] osdmap e2090: 8 total, 8 up, 8 in 2021-07-31T02:45:17.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:16 smithi045 conmon[39718]: cluster 2021-07-31T02:45:15.636332+0000 mon.a (mon.0) 3114 : cluster 2021-07-31T02:45:17.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:16 smithi045 conmon[39718]: [DBG] osdmap e2090: 8 total, 8 up, 8 in 2021-07-31T02:45:17.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:16 smithi045 conmon[33655]: cluster 2021-07-31T02:45:15.636332+0000 mon.a (mon. 2021-07-31T02:45:17.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:16 smithi045 conmon[33655]: 0) 3114 : cluster [DBG] osdmap e2090: 8 total, 8 up, 8 in 2021-07-31T02:45:18.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:17 smithi154 conmon[34553]: cluster 2021-07-31T02:45:16.630568+0000 mgr.x (mgr.24542) 2486 : cluster 2021-07-31T02:45:18.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:17 smithi154 conmon[34553]: [DBG] pgmap v3678: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 2.8 MiB/s rd, 43 KiB/s wr, 865 op/s 2021-07-31T02:45:18.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:17 smithi154 conmon[34553]: cluster 2021-07-31T02:45:16.641827+0000 mon.a (mon.0) 3115 : cluster [DBG] osdmap e2091: 8 total, 8 up, 8 in 2021-07-31T02:45:18.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:17 smithi154 conmon[34553]: cluster 2021-07-31T02:45:17.596613+0000 mon.a (mon 2021-07-31T02:45:18.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:17 smithi154 conmon[34553]: .0) 3116 : cluster [DBG] osdmap e2092: 8 total, 8 up, 8 in 2021-07-31T02:45:18.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:17 smithi045 conmon[33655]: cluster 2021-07-31T02:45:16.630568+0000 mgr.x ( 2021-07-31T02:45:18.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:17 smithi045 conmon[33655]: mgr.24542) 2486 : cluster [DBG] pgmap v3678: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 2.8 MiB/s rd, 43 KiB/s wr, 865 op/s 2021-07-31T02:45:18.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:17 smithi045 conmon[33655]: cluster 2021-07-31T02:45:16.641827+0000 mon.a (mon.0) 3115 : cluster [DBG] osdmap e2091: 8 total, 8 up, 8 in 2021-07-31T02:45:18.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:17 smithi045 conmon[33655]: cluster 2021-07-31T02:45:17.596613+0000 mon.a (mon.0) 3116 : cluster [DBG] osdmap e2092: 8 total, 8 up, 8 in 2021-07-31T02:45:18.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:17 smithi045 conmon[39718]: cluster 2021-07-31T02:45:18.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:17 smithi045 conmon[39718]: 2021-07-31T02:45:16 2021-07-31T02:45:18.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:17 smithi045 conmon[39718]: .630568+0000 mgr.x (mgr.24542) 2486 : cluster [DBG] pgmap v3678: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 2.8 MiB/s rd, 43 KiB/s wr, 865 op/s 2021-07-31T02:45:18.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:17 smithi045 conmon[39718]: cluster 2021-07-31T02:45:18.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:17 smithi045 conmon[39718]: 2021-07-31T02:45:16 2021-07-31T02:45:18.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:17 smithi045 conmon[39718]: .641827+0000 mon.a (mon.0) 3115 : cluster [DBG] 2021-07-31T02:45:18.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:17 smithi045 conmon[39718]: osdmap e2091: 8 total, 8 up, 8 in 2021-07-31T02:45:18.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:17 smithi045 conmon[39718]: cluster 2021-07-31T02:45: 2021-07-31T02:45:18.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:17 smithi045 conmon[39718]: 17.596613+0000 mon.a (mon.0) 3116 : cluster [DBG] 2021-07-31T02:45:18.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:17 smithi045 conmon[39718]: osdmap e2092: 8 total, 8 up, 8 in 2021-07-31T02:45:18.754 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestImage.test_remove_snap2 ... ok 2021-07-31T02:45:20.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:19 smithi154 conmon[34553]: cluster 2021-07-31T02:45:18.630961+0000 mgr.x (mgr.24542) 2487 : cluster [DBG] pgmap v3681: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:45:20.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:19 smithi154 conmon[34553]: cluster 2021 2021-07-31T02:45:20.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:19 smithi154 conmon[34553]: -07-31T02:45:18.648353+0000 mon.a (mon.0) 3117 : cluster [DBG] osdmap e2093: 8 total, 8 up, 8 in 2021-07-31T02:45:20.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:19 smithi045 conmon[39718]: cluster 2021-07-31T02:45:18.630961+0000 mgr.x (mgr.24542) 2487 : cluster [DBG] pgmap v3681: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:45:20.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:19 smithi045 conmon[39718]: cluster 2021-07-31T02:45:18.648353+0000 mon.a (mon.0) 3117 : cluster [DBG] osdmap e2093: 8 total, 8 up, 8 in 2021-07-31T02:45:20.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:19 smithi045 conmon[33655]: cluster 2021-07-31T02:45:18.630961+0000 2021-07-31T02:45:20.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:19 smithi045 conmon[33655]: mgr.x (mgr.24542) 2487 : cluster [DBG] pgmap v3681: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:45:20.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:19 smithi045 conmon[33655]: cluster 2021-07-31T02:45:18. 2021-07-31T02:45:20.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:19 smithi045 conmon[33655]: 648353+0000 mon.a (mon.0) 3117 : cluster [DBG] osdmap e2093: 8 total, 8 up, 8 in 2021-07-31T02:45:20.749 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestImage.test_remove_snap_by_id ... ok 2021-07-31T02:45:20.832 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestImage.test_remove_snap_not_found ... ok 2021-07-31T02:45:20.927 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestImage.test_remove_with_exclusive_lock ... ok 2021-07-31T02:45:21.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:20 smithi154 conmon[34553]: cluster 2021-07-31T02:45:19.654609+0000 mon.a (mon.0) 3118 : cluster [DBG] 2021-07-31T02:45:21.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:20 smithi154 conmon[34553]: osdmap e2094: 8 total, 8 up, 8 in 2021-07-31T02:45:21.011 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestImage.test_remove_with_snap ... SKIP 2021-07-31T02:45:21.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:20 smithi045 conmon[33655]: cluster 2021-07-31T02:45:19.654609+0000 mon.a (mon.0) 3118 : cluster 2021-07-31T02:45:21.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:20 smithi045 conmon[33655]: [DBG] osdmap e2094: 8 total, 8 up, 8 in 2021-07-31T02:45:21.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:20 smithi045 conmon[39718]: cluster 2021-07-31T02:45:19.654609+0000 mon.a 2021-07-31T02:45:21.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:20 smithi045 conmon[39718]: (mon.0) 3118 : cluster [DBG] osdmap e2094: 8 total, 8 up, 8 in 2021-07-31T02:45:21.095 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestImage.test_remove_with_watcher ... SKIP 2021-07-31T02:45:22.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:21 smithi154 conmon[34553]: cluster 2021-07-31T02:45:20.631439+0000 mgr.x (mgr.24542 2021-07-31T02:45:22.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:21 smithi154 conmon[34553]: ) 2488 : cluster [DBG] pgmap v3684: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 237 KiB/s rd, 15 KiB/s wr, 326 op/s 2021-07-31T02:45:22.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:21 smithi154 conmon[34553]: cluster 2021-07-31T02:45:20.660579+0000 mon.a (mon.0) 3119 : cluster [DBG] osdmap e2095: 8 total, 8 up, 8 in 2021-07-31T02:45:22.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:21 smithi045 conmon[33655]: cluster 2021-07-31T02:45:20.631439 2021-07-31T02:45:22.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:21 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 2488 : cluster [DBG] pgmap v3684: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 237 KiB/s rd, 15 KiB/s wr, 326 op/s 2021-07-31T02:45:22.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:21 smithi045 conmon[33655]: cluster 2021-07-31T02:45: 2021-07-31T02:45:22.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:21 smithi045 conmon[33655]: 20.660579+0000 mon.a (mon.0) 3119 : cluster [DBG] 2021-07-31T02:45:22.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:21 smithi045 conmon[33655]: osdmap e2095: 8 total, 8 up, 8 in 2021-07-31T02:45:22.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:21 smithi045 conmon[39718]: cluster 2021-07-31 2021-07-31T02:45:22.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:21 smithi045 conmon[39718]: T02:45:20.631439+0000 mgr.x (mgr.24542) 2488 : cluster [DBG] pgmap v3684: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 237 KiB/s rd, 15 KiB/s wr, 326 op/s 2021-07-31T02:45:22.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:21 smithi045 conmon[39718]: cluster 2021-07-31T02:45:22.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:21 smithi045 conmon[39718]: 2021-07-31T02: 2021-07-31T02:45:22.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:21 smithi045 conmon[39718]: 45:20.660579+0000 mon.a ( 2021-07-31T02:45:22.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:21 smithi045 conmon[39718]: mon.0) 3119 : cluster [DBG] osdmap e2095: 8 total, 8 up, 8 in 2021-07-31T02:45:22.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:21 smithi045 conmon[39718]: 2021-07-31T02:45:22.710 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestImage.test_rename_snap ... ok 2021-07-31T02:45:22.806 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestImage.test_resize ... ok 2021-07-31T02:45:22.903 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestImage.test_resize_allow_shrink_False ... ok 2021-07-31T02:45:23.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:22 smithi154 conmon[34553]: cluster 2021-07-31T02:45:21.661962+0000 mon.a (mon.0) 3120 : 2021-07-31T02:45:23.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:22 smithi154 conmon[34553]: cluster [DBG] osdmap e2096: 8 total, 8 up, 8 in 2021-07-31T02:45:23.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:22 smithi154 conmon[34553]: cluster 2021-07-31T02:45:22.599325+0000 mon.a (mon.0) 3121 : cluster [DBG] osdmap e2097: 8 total, 8 up, 8 in 2021-07-31T02:45:23.009 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestImage.test_resize_bytes ... ok 2021-07-31T02:45:23.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:22 smithi045 conmon[33655]: cluster 2021-07-31T02:45:21.661962+0000 mon.a 2021-07-31T02:45:23.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:22 smithi045 conmon[33655]: (mon.0) 3120 : cluster [DBG] osdmap e2096: 8 total, 8 up, 8 in 2021-07-31T02:45:23.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:22 smithi045 conmon[33655]: cluster 2021-07-31T02:45:22.599325+0000 mon.a (mon.0) 3121 : cluster [DBG] osdmap e2097: 8 total, 8 up, 8 in 2021-07-31T02:45:23.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:22 smithi045 conmon[39718]: cluster 2021-07-31T02:45:21.661962+0000 2021-07-31T02:45:23.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:22 smithi045 conmon[39718]: mon.a (mon.0) 3120 : cluster [DBG] osdmap e2096: 8 total, 8 up, 8 in 2021-07-31T02:45:23.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:22 smithi045 conmon[39718]: cluster 2021-07-31 2021-07-31T02:45:23.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:22 smithi045 conmon[39718]: T02:45:22.599325+0000 mon.a (mon.0) 3121 : cluster [DBG] osdmap e2097: 8 total, 8 up, 8 in 2021-07-31T02:45:23.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:22 smithi045 conmon[39718]: 2021-07-31T02:45:23.111 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestImage.test_resize_down ... ok 2021-07-31T02:45:24.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:23 smithi154 conmon[34553]: cluster 2021-07-31T02:45:22.631841+0000 mgr.x (mgr.24542) 2489 : cluster 2021-07-31T02:45:24.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:23 smithi154 conmon[34553]: [DBG] pgmap v3688: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 238 KiB/s rd, 15 KiB/s wr, 327 op/s 2021-07-31T02:45:24.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:23 smithi045 conmon[33655]: cluster 2021-07-31T02:45:22.631841+0000 2021-07-31T02:45:24.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:23 smithi045 conmon[33655]: mgr.x (mgr.24542) 2489 : cluster [DBG] pgmap v3688: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 238 KiB/s rd, 15 KiB/s wr, 327 op/s 2021-07-31T02:45:24.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:23 smithi045 conmon[39718]: cluster 2021-07-31T02:45:22.631841+0000 2021-07-31T02:45:24.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:23 smithi045 conmon[39718]: mgr.x (mgr.24542) 2489 : cluster [DBG] pgmap v3688: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 238 KiB/s rd, 15 KiB/s wr, 327 op/s 2021-07-31T02:45:24.779 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestImage.test_rollback_to_snap ... ok 2021-07-31T02:45:25.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:24 smithi154 conmon[34553]: cluster 2021-07-31T02:45:25.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:24 smithi154 conmon[34553]: 2021-07 2021-07-31T02:45:25.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:24 smithi154 conmon[34553]: -31T02:45:23.673471+0000 mon.a (mon.0) 3122 : cluster [DBG] osdmap e2098: 8 total, 8 up, 8 in 2021-07-31T02:45:25.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:24 smithi045 conmon[33655]: cluster 2021-07-31T02:45:23.673471+0000 mon.a ( 2021-07-31T02:45:25.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:24 smithi045 conmon[33655]: mon.0) 3122 : cluster [DBG] osdmap e2098: 8 total, 8 up, 8 in 2021-07-31T02:45:25.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:24 smithi045 conmon[39718]: cluster 2021-07-31T02:45:23.673471+0000 mon.a (mon. 2021-07-31T02:45:25.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:24 smithi045 conmon[39718]: 0) 3122 : cluster [DBG] osdmap e2098: 8 total, 8 up, 8 in 2021-07-31T02:45:26.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:25 smithi154 conmon[34553]: cluster 2021-07-31T02:45:24.632459+0000 mgr.x (mgr.24542) 2490 : cluster 2021-07-31T02:45:26.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:25 smithi154 conmon[34553]: [DBG] pgmap v3690: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 422 KiB/s rd, 33 KiB/s wr, 584 op/s 2021-07-31T02:45:26.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:25 smithi154 conmon[34553]: cluster 2021-07-31T02:45:24.679537+0000 mon.a (mon.0) 3123 : cluster [DBG] osdmap e2099: 8 total, 8 up, 8 in 2021-07-31T02:45:26.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:25 smithi045 conmon[39718]: cluster 2021-07-31T02:45:24.632459+0000 mgr.x ( 2021-07-31T02:45:26.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:25 smithi045 conmon[39718]: mgr.24542) 2490 : cluster [DBG] pgmap v3690: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 422 KiB/s rd, 33 KiB/s wr, 584 op/s 2021-07-31T02:45:26.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:25 smithi045 conmon[39718]: cluster 2021-07-31T02:45:24.679537+0000 mon.a (mon.0) 3123 : cluster [DBG] osdmap e2099: 8 total, 8 up, 8 in 2021-07-31T02:45:26.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:25 smithi045 conmon[39718]: 2021-07-31T02:45:26.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:25 smithi045 conmon[33655]: cluster 2021-07-31T02:45:24.632459 2021-07-31T02:45:26.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:25 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 2490 : cluster [DBG] pgmap v3690: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 422 KiB/s rd, 33 KiB/s wr, 584 op/s 2021-07-31T02:45:26.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:25 smithi045 conmon[33655]: cluster 2021-07-31T02:45:26.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:25 smithi045 conmon[33655]: 2021-07-31T02:45:24.679537+0000 mon.a (mon.0 2021-07-31T02:45:26.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:25 smithi045 conmon[33655]: ) 3123 : cluster [DBG] osdmap e2099: 8 total, 8 up, 8 in 2021-07-31T02:45:26.785 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestImage.test_rollback_to_snap_sparse ... ok 2021-07-31T02:45:27.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:26 smithi154 conmon[34553]: cluster 2021-07-31T02:45:25.685059+0000 mon.a (mon.0) 2021-07-31T02:45:27.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:26 smithi154 conmon[34553]: 3124 : cluster [DBG] osdmap e2100: 8 total, 8 up, 8 in 2021-07-31T02:45:27.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:26 smithi045 conmon[33655]: cluster 2021-07-31T02:45:25.685059+0000 mon.a (mon.0) 2021-07-31T02:45:27.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:26 smithi045 conmon[33655]: 3124 : cluster [DBG] osdmap e2100: 8 total, 8 up, 8 in 2021-07-31T02:45:27.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:26 smithi045 conmon[39718]: cluster 2021-07-31T02:45:25.685059+0000 mon.a ( 2021-07-31T02:45:27.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:26 smithi045 conmon[39718]: mon.0) 3124 : cluster [DBG] osdmap e2100: 8 total, 8 up, 8 in 2021-07-31T02:45:28.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:27 smithi154 conmon[34553]: cluster 2021-07-31T02:45:26.632959+0000 mgr.x (mgr.24542) 2491 : cluster [DBG] pgmap v3693: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 418 KiB/s rd, 33 KiB/s wr, 579 op/s 2021-07-31T02:45:28.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:27 smithi154 conmon[34553]: cluster 2021-07-31T02:45:26.690725 2021-07-31T02:45:28.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:27 smithi154 conmon[34553]: +0000 mon.a (mon.0) 3125 : cluster [DBG] osdmap e2101: 8 total, 8 up, 8 in 2021-07-31T02:45:28.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:27 smithi154 conmon[34553]: cluster 2021-07-31T02:45:27.599651+0000 mon.a (mon.0) 3126 : cluster [DBG] osdmap e2102: 8 total, 8 up, 8 in 2021-07-31T02:45:28.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:27 smithi045 conmon[39718]: cluster 2021-07-31T02:45 2021-07-31T02:45:28.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:27 smithi045 conmon[39718]: :26.632959+0000 mgr.x (mgr.24542) 2491 : cluster [DBG] pgmap v3693: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 418 KiB/s rd, 33 KiB/s wr, 579 op/s 2021-07-31T02:45:28.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:27 smithi045 conmon[39718]: cluster 2021- 2021-07-31T02:45:28.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:27 smithi045 conmon[39718]: 07-31T02:45:26.690725 2021-07-31T02:45:28.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:27 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2021-07-31T02:45:28.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:27 smithi045 conmon[39718]: 3125 : cluster [DBG] osdmap e2101: 8 total, 8 up, 8 in 2021-07-31T02:45:28.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:27 smithi045 conmon[39718]: cluster 2021 2021-07-31T02:45:28.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:27 smithi045 conmon[39718]: -07-31T02:45:27 2021-07-31T02:45:28.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:27 smithi045 conmon[39718]: .599651+0000 2021-07-31T02:45:28.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:27 smithi045 conmon[39718]: mon.a (mon.0) 3126 : cluster 2021-07-31T02:45:28.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:27 smithi045 conmon[39718]: [DBG] osdmap e2102: 8 total, 8 up, 8 in 2021-07-31T02:45:28.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:27 smithi045 conmon[33655]: cluster 2021-07-31T02:45:26.632959 2021-07-31T02:45:28.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:27 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 2491 : cluster [DBG] pgmap v3693: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 418 KiB/s rd, 33 KiB/s wr, 579 op/s 2021-07-31T02:45:28.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:27 smithi045 conmon[33655]: cluster 2021-07-31T02:45:26.690725+0000 mon.a (mon.0) 3125 : cluster 2021-07-31T02:45:28.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:27 smithi045 conmon[33655]: [DBG] osdmap e2101: 8 total, 8 up, 8 in 2021-07-31T02:45:28.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:27 smithi045 conmon[33655]: cluster 2021-07 2021-07-31T02:45:28.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:27 smithi045 conmon[33655]: -31T02:45:27.599651+0000 mon.a (mon 2021-07-31T02:45:28.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:27 smithi045 conmon[33655]: .0) 3126 : cluster [DBG] osdmap e2102: 8 total, 8 up, 8 in 2021-07-31T02:45:29.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:28 smithi154 conmon[34553]: audit 2021-07-31T02:45:28.166294+0000 mon.b (mon.2) 365 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:45:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:28 smithi154 conmon[34553]: audit 2021-07-31T02:45:28.166790+0000 mon.a (mon.0) 3127 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:45:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:28 smithi154 conmon[34553]: audit 2021-07-31T02:45:28.167291+0000 mon.b (mon.2) 366 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:45:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:28 smithi154 conmon[34553]: audit 2021-07-31T02:45:28. 2021-07-31T02:45:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:28 smithi154 conmon[34553]: 167659+0000 mon.a (mon.0) 3128 : audit [INF] 2021-07-31T02:45:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:28 smithi154 conmon[34553]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:45:29.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:28 smithi154 conmon[34553]: cluster 2021-07-31T02:45 2021-07-31T02:45:29.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:28 smithi154 conmon[34553]: :28.606007+0000 mon.a (mon.0) 3129 2021-07-31T02:45:29.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:28 smithi154 conmon[34553]: : cluster [DBG] osdmap e2103: 8 total, 8 up, 8 in 2021-07-31T02:45:29.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:28 smithi045 conmon[33655]: audit 2021-07-31T02:45:28.166294+0000 mon.b (mon 2021-07-31T02:45:29.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:28 smithi045 conmon[33655]: .2) 365 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:45:29.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:28 smithi045 conmon[33655]: audit 2021-07-31T02:45:28.166790+0000 mon.a (mon.0) 3127 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:45:29.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:28 smithi045 conmon[33655]: audit 2021-07-31 2021-07-31T02:45:29.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:28 smithi045 conmon[33655]: T02:45:28.167291+0000 2021-07-31T02:45:29.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:28 smithi045 conmon[33655]: mon.b (mon.2) 366 : audit 2021-07-31T02:45:29.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:28 smithi045 conmon[33655]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:45:29.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:28 smithi045 conmon[33655]: audit 2021- 2021-07-31T02:45:29.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:28 smithi045 conmon[33655]: 07-31T02:45:28.167659 2021-07-31T02:45:29.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:28 smithi045 conmon[33655]: +0000 mon.a (mon.0) 3128 2021-07-31T02:45:29.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:28 smithi045 conmon[33655]: : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:45:29.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:28 smithi045 conmon[33655]: cluster 2021- 2021-07-31T02:45:29.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:28 smithi045 conmon[33655]: 07-31T02:45:28. 2021-07-31T02:45:29.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:28 smithi045 conmon[33655]: 606007+0000 mon.a (mon.0 2021-07-31T02:45:29.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:28 smithi045 conmon[33655]: ) 3129 : cluster [DBG] osdmap e2103: 8 total, 8 up, 8 in 2021-07-31T02:45:29.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:28 smithi045 conmon[39718]: audit 2021-07-31T02:45:28.166294+0000 2021-07-31T02:45:29.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:28 smithi045 conmon[39718]: mon.b (mon.2) 365 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:45:29.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:28 smithi045 conmon[39718]: audit 2021-07-31T02:45:29.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:28 smithi045 conmon[39718]: 2021-07-31T02:45:28.166790+0000 mon.a ( 2021-07-31T02:45:29.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:28 smithi045 conmon[39718]: mon.0) 3127 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:45:29.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:28 smithi045 conmon[39718]: 2021-07-31T02:45:29.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:28 smithi045 conmon[39718]: audit 2021-07-31T02:45:29.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:28 smithi045 conmon[39718]: 2021-07-31T02:45:28.167291 2021-07-31T02:45:29.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:28 smithi045 conmon[39718]: +0000 mon.b (mon.2) 366 : audit 2021-07-31T02:45:29.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:28 smithi045 conmon[39718]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:45:29.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:28 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T02:45:29.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:28 smithi045 conmon[39718]: 02:45:28.167659+0000 mon.a ( 2021-07-31T02:45:29.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:28 smithi045 conmon[39718]: mon.0) 3128 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:45:29.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:28 smithi045 conmon[39718]: cluster 2021-07-31T02:45:29.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:28 smithi045 conmon[39718]: 2021-07-31T02:45:28 2021-07-31T02:45:29.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:28 smithi045 conmon[39718]: .606007+0000 mon.a (mon.0) 3129 2021-07-31T02:45:29.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:28 smithi045 conmon[39718]: : cluster [DBG] osdmap e2103: 8 total, 8 up, 8 in 2021-07-31T02:45:30.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:29 smithi154 conmon[34553]: cluster 2021-07-31T02:45:28.633322+0000 mgr.x (mgr.24542) 2492 : cluster [DBG] pgmap v3697: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:45:30.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:29 smithi154 conmon[34553]: cluster 2021-07-31T02:45: 2021-07-31T02:45:30.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:29 smithi154 conmon[34553]: 29.610587+0000 mon.a (mon.0) 3130 : cluster [DBG] osdmap e2104: 8 total, 8 up, 8 in 2021-07-31T02:45:30.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:29 smithi045 conmon[33655]: cluster 2021-07-31T02:45:28.633322+0000 mgr.x (mgr.24542) 2492 : cluster [DBG] pgmap v3697: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:45:30.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:29 smithi045 conmon[33655]: cluster 2021-07-31T02:45:29.610587+0000 mon.a (mon.0) 3130 : cluster [DBG] osdmap e2104: 8 total, 8 up, 8 in 2021-07-31T02:45:30.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:29 smithi045 conmon[39718]: cluster 2021-07-31T02:45:28.633322+0000 mgr.x 2021-07-31T02:45:30.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:29 smithi045 conmon[39718]: (mgr.24542) 2492 : cluster [DBG] pgmap v3697: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:45:30.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:29 smithi045 conmon[39718]: cluster 2021-07-31T02:45:29.610587+0000 2021-07-31T02:45:30.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:29 smithi045 conmon[39718]: mon.a (mon.0) 3130 : cluster [DBG] osdmap e2104: 8 total, 8 up, 8 in 2021-07-31T02:45:30.711 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestImage.test_rollback_with_resize ... ok 2021-07-31T02:45:32.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:31 smithi154 conmon[34553]: cluster 2021-07-31T02:45:30.613575+0000 mon.a (mon.0) 3131 : cluster [DBG] 2021-07-31T02:45:32.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:31 smithi154 conmon[34553]: osdmap e2105: 8 total, 8 up, 8 in 2021-07-31T02:45:32.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:31 smithi154 conmon[34553]: cluster 2021-07-31T02:45:30.633804+0000 mgr.x (mgr.24542) 2493 : cluster [DBG] pgmap v3700: 41 pgs: 41 active+clean; 13 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 134 KiB/s rd, 15 KiB/s wr, 189 op/s 2021-07-31T02:45:32.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:31 smithi045 conmon[33655]: cluster 2021-07-31T02:45:30.613575+0000 mon.a (mon.0 2021-07-31T02:45:32.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:31 smithi045 conmon[33655]: ) 3131 : cluster [DBG] osdmap e2105: 8 total, 8 up, 8 in 2021-07-31T02:45:32.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:31 smithi045 conmon[33655]: cluster 2021-07-31T02:45:30.633804+0000 mgr.x (mgr.24542) 2493 : cluster [DBG] pgmap v3700: 41 pgs: 41 active+clean; 13 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 134 KiB/s rd, 15 KiB/s wr, 189 op/s 2021-07-31T02:45:32.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:31 smithi045 conmon[39718]: cluster 2021-07-31T02:45:30.613575+0000 2021-07-31T02:45:32.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:31 smithi045 conmon[39718]: mon.a (mon.0) 3131 : cluster [DBG] osdmap e2105: 8 total, 8 up, 8 in 2021-07-31T02:45:32.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:31 smithi045 conmon[39718]: cluster 2021-07-31T02 2021-07-31T02:45:32.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:31 smithi045 conmon[39718]: :45:30.633804+0000 mgr.x ( 2021-07-31T02:45:32.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:31 smithi045 conmon[39718]: mgr.24542) 2493 : cluster [DBG] pgmap v3700: 41 pgs: 41 active+clean; 13 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 134 KiB/s rd, 15 KiB/s wr, 189 op/s 2021-07-31T02:45:32.717 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestImage.test_set_no_snap ... ok 2021-07-31T02:45:33.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:32 smithi154 conmon[34553]: cluster 2021-07-31T02:45:31.618885+0000 mon.a ( 2021-07-31T02:45:33.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:32 smithi154 conmon[34553]: mon.0) 3132 : cluster [DBG] osdmap e2106: 8 total, 8 up, 8 in 2021-07-31T02:45:33.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:32 smithi154 conmon[34553]: cluster 2021-07-31T02:45:32.602021+0000 mon.a (mon.0) 3133 : cluster [DBG] osdmap e2107: 8 total, 8 up, 8 in 2021-07-31T02:45:33.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:32 smithi045 conmon[39718]: cluster 2021-07-31T02:45:31.618885+0000 2021-07-31T02:45:33.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:32 smithi045 conmon[39718]: mon.a (mon.0) 3132 : cluster [DBG] osdmap e2106: 8 total, 8 up, 8 in 2021-07-31T02:45:33.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:32 smithi045 conmon[39718]: cluster 2021-07-31T02:45:32.602021+0000 mon.a (mon.0) 3133 : cluster [DBG] osdmap e2107: 8 total, 8 up, 8 in 2021-07-31T02:45:33.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:32 smithi045 conmon[33655]: cluster 2021-07-31T02:45:33.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:32 smithi045 conmon[33655]: 2021-07-31T02:45:33.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:32 smithi045 conmon[33655]: 2021 2021-07-31T02:45:33.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:32 smithi045 conmon[33655]: - 2021-07-31T02:45:33.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:32 smithi045 conmon[33655]: 07 2021-07-31T02:45:33.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:32 smithi045 conmon[33655]: -31T02:45:31.618885+0000 mon.a (mon.0) 3132 : cluster [DBG] osdmap e2106: 8 total, 8 up, 8 in 2021-07-31T02:45:33.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:32 smithi045 conmon[33655]: cluster 2021-07-31T02:45:32.602021+0000 mon.a (mon.0) 3133 : cluster [DBG] osdmap e2107: 8 total, 8 up, 8 in 2021-07-31T02:45:34.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[34553]: cluster 2021-07-31T02:45:32.634160+0000 mgr.x (mgr.24542) 2494 : cluster [DBG] 2021-07-31T02:45:34.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[34553]: pgmap v3703: 41 pgs: 41 active+clean; 13 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 132 KiB/s rd, 15 KiB/s wr, 186 op/s 2021-07-31T02:45:34.003 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: debug 2021-07-31T02:45:33.724+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:45:34.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:45:34.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:45:34.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: yield 2021-07-31T02:45:34.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:45:34.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:45:34.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:45:34.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:45:34.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:45:34.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:45:34.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:45:34.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:45:34.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:45:34.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: task.result() 2021-07-31T02:45:34.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:45:34.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: debug 2021-07-31T02:45:33.735+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:45:34.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:45:34.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:45:34.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: yield 2021-07-31T02:45:34.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:45:34.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:45:34.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:45:34.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:45:34.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:45:34.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:45:34.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:45:34.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:45:34.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:45:34.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: task.result() 2021-07-31T02:45:34.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:45:34.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: debug 2021-07-31T02:45:33.760+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:45:34.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:45:34.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:45:34.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: yield 2021-07-31T02:45:34.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:45:34.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:45:34.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:45:34.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:45:34.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:45:34.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:45:34.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:45:34.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:45:34.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:45:34.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: task.result() 2021-07-31T02:45:34.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:45:34.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: debug 2021-07-31T02:45:33.761+0000 7fcb9095b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:45:34.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:45:34.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:45:34.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: yield 2021-07-31T02:45:34.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:45:34.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:45:34.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:45:34.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:45:34.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:45:34.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:45:34.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:45:34.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:45:34.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:45:34.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: task.result() 2021-07-31T02:45:34.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:45:34.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: 2021-07-31T02:45:34.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:45:34.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: 2021-07-31T02:45:34.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:45:34.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:45:34.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:45:34.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:45:34.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:45:34.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:45:34.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:45:34.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:45:34.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:45:34.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:45:34.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:45:34.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:45:34.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:45:34.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:45:34.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:45:34.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:45:34.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:45:34.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:45:34.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: raise 2021-07-31T02:45:34.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:45:34.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:45:34.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:45:34.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:45:34.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:45:34.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: debug 2021-07-31T02:45:33.769+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:45:34.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:45:34.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:45:34.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: yield 2021-07-31T02:45:34.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:45:34.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:45:34.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:45:34.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:45:34.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:45:34.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:45:34.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:45:34.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:45:34.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:45:34.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: task.result() 2021-07-31T02:45:34.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:45:34.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: debug 2021-07-31T02:45:33.770+0000 7fcb9115c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:45:34.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:45:34.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:45:34.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: yield 2021-07-31T02:45:34.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:45:34.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:45:34.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:45:34.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:45:34.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:45:34.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:45:34.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:45:34.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:45:34.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:45:34.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: task.result() 2021-07-31T02:45:34.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:45:34.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: 2021-07-31T02:45:34.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:45:34.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: 2021-07-31T02:45:34.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:45:34.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:45:34.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:45:34.035 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:45:34.035 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:45:34.035 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:45:34.035 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:45:34.035 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:45:34.036 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:45:34.036 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:45:34.036 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:45:34.036 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:45:34.036 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:45:34.037 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:45:34.037 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:45:34.037 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:45:34.037 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:45:34.037 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:45:34.038 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: raise 2021-07-31T02:45:34.038 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:45:34.038 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:45:34.038 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:45:34.039 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:45:34.039 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:45:33 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:45:34.040 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:33 smithi045 conmon[33655]: cluster 2021-07-31T02:45:32.634160+0000 mgr.x (mgr.24542) 2494 : cluster 2021-07-31T02:45:34.040 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:33 smithi045 conmon[33655]: [DBG] pgmap v3703: 41 pgs: 41 active+clean; 13 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 132 KiB/s rd, 15 KiB/s wr, 186 op/s 2021-07-31T02:45:34.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:33 smithi045 conmon[39718]: cluster 2021-07-31T02:45:32.634160+0000 mgr.x (mgr.24542) 2494 : 2021-07-31T02:45:34.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:33 smithi045 conmon[39718]: cluster [DBG] pgmap v3703: 41 pgs: 41 active+clean; 13 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 132 KiB/s rd, 15 KiB/s wr, 186 op/s 2021-07-31T02:45:34.726 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestImage.test_set_snap ... ok 2021-07-31T02:45:35.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: cluster 2021-07-31T02:45:33.626794+0000 mon.a (mon.0) 2021-07-31T02:45:35.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: 3134 : cluster [DBG] osdmap e2108: 8 total, 8 up, 8 in 2021-07-31T02:45:35.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: audit 2021-07-31T02:45:33.680777+0000 mon.b (mon.2) 367 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:45:35.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: audit 2021-07-31T02: 2021-07-31T02:45:35.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: 45:33.682012+0000 mon.b (mon.2) 2021-07-31T02:45:35.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: 368 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:45:35.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: audit 2021-07-31T02:45:35.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: 2021-07-31T02:45: 2021-07-31T02:45:35.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: 33.682959+0000 mon.b (mon.2) 369 : 2021-07-31T02:45:35.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:45:35.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: cephadm 2021 2021-07-31T02:45:35.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: -07-31T02:45:33 2021-07-31T02:45:35.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: .683630+0000 mgr.x (mgr.24542) 2495 : cephadm 2021-07-31T02:45:35.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:45:35.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: cephadm 2021 2021-07-31T02:45:35.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: -07-31T02:45:33. 2021-07-31T02:45:35.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: 725437+0000 mgr.x (mgr.24542) 2496 : cephadm 2021-07-31T02:45:35.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: [ERR] 2021-07-31T02:45:35.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:45:35.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:45:35.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: yield 2021-07-31T02:45:35.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:45:35.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:45:35.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:45:35.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:45:35.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:45:35.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:45:35.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:45:35.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:45:35.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:45:35.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: task.result() 2021-07-31T02:45:35.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:45:35.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: cephadm 2021-07-31T02:45:33.725845+0000 mgr.x (mgr.24542) 2497 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:45:35.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: cephadm 2021-07-31T 2021-07-31T02:45:35.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: 02:45:33.736747+0000 mgr.x (mgr.24542) 2498 : cephadm [ERR] 2021-07-31T02:45:35.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:45:35.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:45:35.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: yield 2021-07-31T02:45:35.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:45:35.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:45:35.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:45:35.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:45:35.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:45:35.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:45:35.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:45:35.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:45:35.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:45:35.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: task.result() 2021-07-31T02:45:35.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:45:35.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: cephadm 2021-07-31T02:45:33.737622+0000 mgr.x (mgr.24542) 2499 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:45:35.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: cephadm 2021-07-31T02:45 2021-07-31T02:45:35.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: :33.761626+0000 mgr.x (mgr.24542) 2500 : cephadm [ERR] 2021-07-31T02:45:35.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: 2021-07-31T02:45:35.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:45:35.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:45:35.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: yield 2021-07-31T02:45:35.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:45:35.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:45:35.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:45:35.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:45:35.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:45:35.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:45:35.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:45:35.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:45:35.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:45:35.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: task.result() 2021-07-31T02:45:35.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:45:35.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: cephadm 2021-07-31T02:45:33.762206+0000 mgr.x (mgr.24542) 2501 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:45:35.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:45:35.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:45:35.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: yield 2021-07-31T02:45:35.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:45:35.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:45:35.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:45:35.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:45:35.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:45:35.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:45:35.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:45:35.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:45:35.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:45:35.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: task.result() 2021-07-31T02:45:35.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:45:35.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: 2021-07-31T02:45:35.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:45:35.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: 2021-07-31T02:45:35.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:45:35.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:45:35.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:45:35.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:45:35.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:45:35.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:45:35.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:45:35.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:45:35.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:45:35.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:45:35.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:45:35.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:45:35.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:45:35.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:45:35.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:45:35.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:45:35.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:45:35.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:45:35.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: raise 2021-07-31T02:45:35.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:45:35.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:45:35.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:45:35.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:45:35.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:45:35.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: cephadm 2021 2021-07-31T02:45:35.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: -07-31T02:45:33.770522+0000 mgr.x (mgr.24542) 2502 : cephadm [ERR] 2021-07-31T02:45:35.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:45:35.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:45:35.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: yield 2021-07-31T02:45:35.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:45:35.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:45:35.035 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:45:35.037 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:45:35.037 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:45:35.037 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:45:35.038 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:45:35.038 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:45:35.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: cluster 2021-07-31T02:45:33.626794+0000 mon.a ( 2021-07-31T02:45:35.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: mon.0) 3134 : cluster [DBG] osdmap e2108: 8 total, 8 up, 8 in 2021-07-31T02:45:35.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: audit 2021-07-31T02:45:33.680777+0000 mon.b (mon. 2021-07-31T02:45:35.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: 2) 367 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:45:35.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: audit 2021-07-31T02:45:33 2021-07-31T02:45:35.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: .682012+0000 mon.b (mon.2) 368 : audit 2021-07-31T02:45:35.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:45:35.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T02:45:35.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: 02:45:33.682959+0000 mon.b (mon 2021-07-31T02:45:35.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: .2) 369 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:45:35.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: cephadm 2021- 2021-07-31T02:45:35.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: 07-31T02:45:33.683630+0000 2021-07-31T02:45:35.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: mgr.x (mgr.24542) 2495 : cephadm [WRN] 2021-07-31T02:45:35.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:45:35.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: cephadm 2021-07-31T02: 2021-07-31T02:45:35.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: 45:33.725437+0000 mgr.x (mgr.24542 2021-07-31T02:45:35.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: ) 2496 : cephadm [ERR] 2021-07-31T02:45:35.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:45:35.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:45:35.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: yield 2021-07-31T02:45:35.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:45:35.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:45:35.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:45:35.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:45:35.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:45:35.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:45:35.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:45:35.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:45:35.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:45:35.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: task.result() 2021-07-31T02:45:35.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:45:35.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: cephadm 2021-07-31T02:45:33.725845+0000 mgr.x (mgr.24542) 2497 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:45:35.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: cephadm 2021-07-31T02:45: 2021-07-31T02:45:35.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: 33.736747+0000 mgr.x (mgr.24542) 2498 : cephadm [ERR] 2021-07-31T02:45:35.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:45:35.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:45:35.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: yield 2021-07-31T02:45:35.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:45:35.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:45:35.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:45:35.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:45:35.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:45:35.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:45:35.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:45:35.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:45:35.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:45:35.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: task.result() 2021-07-31T02:45:35.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:45:35.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: cephadm 2021-07-31T02:45:33.737622+0000 mgr.x (mgr.24542) 2499 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:45:35.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: cephadm 2021-07 2021-07-31T02:45:35.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: -31T02:45:33.761626+0000 mgr.x (mgr 2021-07-31T02:45:35.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: .24542) 2500 : cephadm [ERR] 2021-07-31T02:45:35.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:45:35.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:45:35.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: yield 2021-07-31T02:45:35.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:45:35.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:45:35.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:45:35.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:45:35.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:45:35.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:45:35.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:45:35.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:45:35.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:45:35.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: task.result() 2021-07-31T02:45:35.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:45:35.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: cephadm 2021-07-31T02:45:33.762206+0000 mgr.x (mgr.24542) 2501 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:45:35.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:45:35.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:45:35.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: yield 2021-07-31T02:45:35.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:45:35.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:45:35.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:45:35.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:45:35.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:45:35.057 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:45:35.057 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:45:35.057 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:45:35.057 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:45:35.057 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: task.result() 2021-07-31T02:45:35.058 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:45:35.058 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: task.result() 2021-07-31T02:45:35.059 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:45:35.059 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: cephadm 2021-07-31T02:45:33.770939+0000 mgr.x (mgr.24542) 2503 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:45:35.059 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:45:35.059 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:45:35.060 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: yield 2021-07-31T02:45:35.060 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:45:35.060 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:45:35.060 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:45:35.060 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:45:35.061 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:45:35.061 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:45:35.061 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:45:35.061 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:45:35.062 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:45:35.062 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: task.result() 2021-07-31T02:45:35.062 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:45:35.062 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: 2021-07-31T02:45:35.063 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:45:35.063 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: 2021-07-31T02:45:35.063 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:45:35.064 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:45:35.064 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:45:35.064 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:45:35.064 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:45:35.065 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:45:35.065 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:45:35.065 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:45:35.065 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:45:35.065 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:45:35.066 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:45:35.066 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:45:35.066 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:45:35.066 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:45:35.067 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:45:35.067 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:45:35.067 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:45:35.067 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:45:35.068 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: raise 2021-07-31T02:45:35.068 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:45:35.068 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:45:35.068 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:45:35.068 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:45:35.069 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:34 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:45:35.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: cluster 2021-07-31T02:45:33.626794+0000 mon.a (mon 2021-07-31T02:45:35.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: .0) 3134 : cluster [DBG] osdmap e2108: 8 total, 8 up, 8 in 2021-07-31T02:45:35.070 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: audit 2021-07-31T02:45:33.680777+0000 mon.b (mon.2) 367 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:45:35.070 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: audit 2021-07-31T02:45:33 2021-07-31T02:45:35.070 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: .682012+0000 mon.b (mon.2) 368 : audit [DBG] 2021-07-31T02:45:35.070 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:45:35.070 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: audit 2021-07 2021-07-31T02:45:35.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: -31T02:45:33.682959+0000 mon.b (mon.2) 369 2021-07-31T02:45:35.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:45:35.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: cephadm 2021-07-31T02:45:33.683630+0000 mgr.x 2021-07-31T02:45:35.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: (mgr.24542) 2495 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:45:35.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: cephadm 2021-07-31T02:45: 2021-07-31T02:45:35.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: 33.725437+0000 mgr.x (mgr.24542) 2496 : cephadm [ERR] 2021-07-31T02:45:35.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:45:35.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:45:35.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: yield 2021-07-31T02:45:35.073 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:45:35.073 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:45:35.073 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:45:35.073 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:45:35.074 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:45:35.074 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:45:35.074 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:45:35.074 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:45:35.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:45:35.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: task.result() 2021-07-31T02:45:35.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:45:35.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: cephadm 2021-07-31T02:45:33.725845+0000 mgr.x (mgr.24542) 2497 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:45:35.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: cephadm 2021-07-31T02:45:33.736747+0000 mgr.x (mgr.24542) 2498 : cephadm [ERR] 2021-07-31T02:45:35.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:45:35.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:45:35.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: yield 2021-07-31T02:45:35.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:45:35.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:45:35.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:45:35.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:45:35.078 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:45:35.078 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: 2021-07-31T02:45:35.078 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:45:35.078 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: 2021-07-31T02:45:35.079 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:45:35.079 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:45:35.079 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:45:35.079 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:45:35.080 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:45:35.080 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:45:35.080 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:45:35.080 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:45:35.081 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:45:35.081 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:45:35.081 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:45:35.081 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:45:35.082 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:45:35.082 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:45:35.082 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:45:35.083 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:45:35.083 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:45:35.083 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:45:35.084 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: raise 2021-07-31T02:45:35.084 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:45:35.084 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:45:35.085 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:45:35.085 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:45:35.085 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:45:35.085 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: cephadm 2021-07-31T02:45 2021-07-31T02:45:35.086 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: :33.770522+0000 mgr.x (mgr.24542) 2502 : cephadm [ERR] 2021-07-31T02:45:35.086 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:45:35.086 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:45:35.086 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: yield 2021-07-31T02:45:35.086 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:45:35.087 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:45:35.087 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:45:35.087 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:45:35.087 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:45:35.088 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:45:35.088 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:45:35.088 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:45:35.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:45:35.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:45:35.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:45:35.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:45:35.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:45:35.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: task.result() 2021-07-31T02:45:35.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:45:35.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: cephadm 2021-07-31T02:45:33.737622+0000 mgr.x (mgr.24542) 2499 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:45:35.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: cephadm 2021-07-31T02:45:33.761626+0000 mgr.x (mgr.24542) 2500 : cephadm [ERR] 2021-07-31T02:45:35.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:45:35.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:45:35.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: yield 2021-07-31T02:45:35.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:45:35.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:45:35.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:45:35.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:45:35.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:45:35.093 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:45:35.093 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:45:35.093 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:45:35.093 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:45:35.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: task.result() 2021-07-31T02:45:35.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:45:35.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: cephadm 2021-07-31T02:45:33.762206+0000 mgr.x ( 2021-07-31T02:45:35.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: mgr.24542) 2501 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:45:35.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:45:35.095 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:45:35.095 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: yield 2021-07-31T02:45:35.095 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:45:35.095 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:45:35.096 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:45:35.096 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:45:35.096 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:45:35.096 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:45:35.097 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:45:35.097 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:45:35.097 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:45:35.097 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: task.result() 2021-07-31T02:45:35.097 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:45:35.098 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: 2021-07-31T02:45:35.098 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:45:35.098 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: 2021-07-31T02:45:35.098 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:45:35.099 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:45:35.099 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:45:35.099 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:45:35.099 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:45:35.099 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:45:35.100 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:45:35.100 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:45:35.100 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:45:35.100 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:45:35.101 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:45:35.101 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:45:35.101 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:45:35.101 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:45:35.101 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:45:35.102 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:45:35.102 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:45:35.102 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:45:35.102 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: raise 2021-07-31T02:45:35.103 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:45:35.103 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:45:35.103 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:45:35.103 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:45:35.104 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:45:35.104 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: cephadm 2021-07-31T02:45:33.770522+0000 mgr.x (mgr.24542) 2502 : cephadm [ERR] 2021-07-31T02:45:35.104 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:45:35.104 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:45:35.105 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: yield 2021-07-31T02:45:35.105 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:45:35.105 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:45:35.106 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:45:35.106 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:45:35.106 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:45:35.107 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:45:35.107 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:45:35.107 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:45:35.107 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:45:35.108 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: task.result() 2021-07-31T02:45:35.108 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:45:35.109 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:45:35.109 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: task.result() 2021-07-31T02:45:35.109 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:45:35.110 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: cephadm 2021-07-31T02:45:33.770939+0000 mgr.x (mgr.24542) 2503 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:45:35.110 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:45:35.110 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:45:35.110 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: yield 2021-07-31T02:45:35.111 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:45:35.111 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:45:35.111 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:45:35.111 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:45:35.112 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:45:35.112 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:45:35.112 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:45:35.112 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:45:35.112 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:45:35.113 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: task.result() 2021-07-31T02:45:35.113 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:45:35.113 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: 2021-07-31T02:45:35.113 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:45:35.114 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: 2021-07-31T02:45:35.114 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:45:35.115 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:45:35.115 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:45:35.115 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:45:35.116 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:45:35.116 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:45:35.116 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:45:35.117 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:45:35.117 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:45:35.117 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:45:35.117 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:45:35.118 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:45:35.118 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:45:35.118 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:45:35.118 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:45:35.118 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:45:35.119 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:45:35.119 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:45:35.119 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: raise 2021-07-31T02:45:35.119 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:45:35.120 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:45:35.120 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:45:35.120 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:45:35.120 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:45:35.121 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: cephadm 2021-07-31T02:45:33.770939+0000 mgr.x (mgr.24542) 2503 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:45:35.121 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:45:35.121 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:45:35.121 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: yield 2021-07-31T02:45:35.122 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:45:35.122 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:45:35.122 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:45:35.122 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:45:35.122 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:45:35.123 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:45:35.123 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:45:35.123 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:45:35.123 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:45:35.124 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: task.result() 2021-07-31T02:45:35.124 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:45:35.124 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: 2021-07-31T02:45:35.124 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:45:35.124 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: 2021-07-31T02:45:35.125 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:45:35.125 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:45:35.125 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:45:35.125 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:45:35.126 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:45:35.126 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:45:35.126 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:45:35.126 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:45:35.126 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:45:35.127 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:45:35.127 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:45:35.127 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:45:35.127 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:45:35.128 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:45:35.128 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:45:35.128 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:45:35.128 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:45:35.128 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:45:35.129 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: raise 2021-07-31T02:45:35.129 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:45:35.129 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:45:35.129 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:45:35.130 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:45:35.130 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:34 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:45:36.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:35 smithi154 conmon[34553]: cluster 2021-07-31T02:45:34.633396+0000 mon.a (mon.0) 3135 : 2021-07-31T02:45:36.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:35 smithi154 conmon[34553]: cluster [DBG] osdmap e2109: 8 total, 8 up, 8 in 2021-07-31T02:45:36.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:35 smithi154 conmon[34553]: cluster 2021-07-31T02:45:34.634618+0000 mgr.x (mgr.24542) 2504 : cluster [DBG] pgmap v3705: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 108 KiB/s rd, 14 KiB/s wr, 156 op/s 2021-07-31T02:45:36.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:35 smithi045 conmon[33655]: cluster 2021-07-31T02:45:34.633396 2021-07-31T02:45:36.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:35 smithi045 conmon[33655]: +0000 mon.a (mon.0) 3135 : cluster [DBG] osdmap e2109: 8 total, 8 up, 8 in 2021-07-31T02:45:36.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:35 smithi045 conmon[33655]: cluster 2021-07-31T02:45:34.634618+0000 2021-07-31T02:45:36.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:35 smithi045 conmon[33655]: mgr.x (mgr.24542) 2504 : cluster [DBG] pgmap v3705: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 108 KiB/s rd, 14 KiB/s wr, 156 op/s 2021-07-31T02:45:36.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:35 smithi045 conmon[39718]: cluster 2021-07-31T02:45:34.633396 2021-07-31T02:45:36.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:35 smithi045 conmon[39718]: +0000 mon.a (mon.0) 3135 : cluster [DBG] osdmap e2109: 8 total, 8 up, 8 in 2021-07-31T02:45:36.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:35 smithi045 conmon[39718]: cluster 2021-07-31T02:45:36.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:35 smithi045 conmon[39718]: 2021-07-31T02:45:34.634618+0000 mgr.x (mgr. 2021-07-31T02:45:36.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:35 smithi045 conmon[39718]: 24542) 2504 : cluster [DBG] pgmap v3705: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 108 KiB/s rd, 14 KiB/s wr, 156 op/s 2021-07-31T02:45:36.739 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestImage.test_set_snap_by_id ... ok 2021-07-31T02:45:37.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:36 smithi154 conmon[34553]: cluster 2021-07-31T02:45:35.640964+0000 mon.a (mon.0) 3136 : cluster [DBG] osdmap e2110: 8 total, 8 up, 8 in 2021-07-31T02:45:37.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:36 smithi045 conmon[33655]: cluster 2021-07-31T02:45:35.640964+0000 mon.a (mon.0) 3136 : 2021-07-31T02:45:37.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:36 smithi045 conmon[33655]: cluster [DBG] osdmap e2110: 8 total, 8 up, 8 in 2021-07-31T02:45:37.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:36 smithi045 conmon[39718]: cluster 2021-07-31T02:45:35.640964+0000 mon.a (mon. 2021-07-31T02:45:37.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:36 smithi045 conmon[39718]: 0) 3136 : cluster [DBG] osdmap e2110: 8 total, 8 up, 8 in 2021-07-31T02:45:38.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:37 smithi154 conmon[34553]: cluster 2021-07-31T02:45:36.635278+0000 mgr.x (mgr.24542) 2505 : cluster [DBG] 2021-07-31T02:45:38.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:37 smithi154 conmon[34553]: pgmap v3708: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 108 KiB/s rd, 14 KiB/s wr, 155 op/s 2021-07-31T02:45:38.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:37 smithi154 conmon[34553]: cluster 2021-07-31T02:45:36.646364+0000 mon.a (mon.0) 3137 : cluster [DBG] osdmap e2111: 8 total, 8 up, 8 in 2021-07-31T02:45:38.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:37 smithi154 conmon[34553]: cluster 2021-07-31T 2021-07-31T02:45:38.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:37 smithi154 conmon[34553]: 02:45:37.602575+0000 mon.a (mon.0) 3138 : cluster [DBG] osdmap e2112: 8 total, 8 up, 8 in 2021-07-31T02:45:38.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:37 smithi045 conmon[33655]: cluster 2021-07-31T02:45:36.635278+0000 mgr.x 2021-07-31T02:45:38.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:37 smithi045 conmon[33655]: (mgr.24542) 2505 : cluster [DBG] pgmap v3708: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 108 KiB/s rd, 14 KiB/s wr, 155 op/s 2021-07-31T02:45:38.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:37 smithi045 conmon[33655]: cluster 2021-07-31T02:45:36.646364+0000 mon.a (mon.0) 3137 : cluster [DBG] osdmap e2111: 8 total, 8 up, 8 in 2021-07-31T02:45:38.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:37 smithi045 conmon[33655]: cluster 2021-07-31T02:45:37.602575+0000 mon.a (mon.0) 3138 : cluster [DBG] osdmap e2112: 8 total, 8 up, 8 in 2021-07-31T02:45:38.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:37 smithi045 conmon[39718]: cluster 2021-07-31T02:45:36.635278+0000 mgr.x (mgr. 2021-07-31T02:45:38.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:37 smithi045 conmon[39718]: 24542) 2505 : cluster [DBG] pgmap v3708: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 108 KiB/s rd, 14 KiB/s wr, 155 op/s 2021-07-31T02:45:38.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:37 smithi045 conmon[39718]: cluster 2021-07-31T02:45 2021-07-31T02:45:38.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:37 smithi045 conmon[39718]: :36.646364+0000 mon.a (mon.0) 3137 : cluster [DBG] osdmap e2111: 8 total, 8 up, 8 in 2021-07-31T02:45:38.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:37 smithi045 conmon[39718]: cluster 2021-07-31 2021-07-31T02:45:38.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:37 smithi045 conmon[39718]: T02:45:37.602575+0000 mon.a (mon.0 2021-07-31T02:45:38.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:37 smithi045 conmon[39718]: ) 3138 : cluster [DBG] osdmap e2112: 8 total, 8 up, 8 in 2021-07-31T02:45:38.751 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestImage.test_set_snap_deleted ... ok 2021-07-31T02:45:40.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:39 smithi154 conmon[34553]: cluster 2021-07-31T02:45:38.635676+0000 mgr.x (mgr.24542) 2506 : cluster [DBG] pgmap v3711: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:45:40.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:39 smithi154 conmon[34553]: 2021-07-31T02:45:40.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:39 smithi154 conmon[34553]: cluster 2021-07-31T02:45:38.652225+0000 mon.a (mon.0) 3139 : cluster [DBG] osdmap e2113: 8 total, 8 up, 8 in 2021-07-31T02:45:40.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:39 smithi045 conmon[39718]: cluster 2021-07-31T02:45:38.635676+0000 mgr.x (mgr.24542 2021-07-31T02:45:40.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:39 smithi045 conmon[39718]: ) 2506 : cluster [DBG] pgmap v3711: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:45:40.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:39 smithi045 conmon[39718]: cluster 2021-07-31T02:45:38.652225+0000 mon.a (mon.0) 3139 : cluster [DBG] osdmap e2113: 8 total, 8 up, 8 in 2021-07-31T02:45:40.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:39 smithi045 conmon[33655]: cluster 2021-07-31T02:45:38.635676+0000 mgr.x (mgr. 2021-07-31T02:45:40.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:39 smithi045 conmon[33655]: 24542) 2506 : cluster [DBG] pgmap v3711: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:45:40.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:39 smithi045 conmon[33655]: cluster 2021-07-31T02:45:38.652225+0000 mon.a (mon.0) 3139 : cluster [DBG] osdmap e2113: 8 total, 8 up, 8 in 2021-07-31T02:45:41.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:40 smithi154 conmon[34553]: cluster 2021-07-31T02:45:39.658538+0000 mon.a (mon.0) 3140 2021-07-31T02:45:41.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:40 smithi154 conmon[34553]: : cluster [DBG] osdmap e2114: 8 total, 8 up, 8 in 2021-07-31T02:45:41.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:40 smithi045 conmon[33655]: cluster 2021-07-31T02:45:39.658538+0000 mon.a (mon.0) 3140 : cluster [DBG] 2021-07-31T02:45:41.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:40 smithi045 conmon[33655]: osdmap e2114: 8 total, 8 up, 8 in 2021-07-31T02:45:41.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:40 smithi045 conmon[39718]: cluster 2021-07-31T02:45:39.658538+0000 mon.a 2021-07-31T02:45:41.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:40 smithi045 conmon[39718]: (mon.0) 3140 : cluster [DBG] osdmap e2114: 8 total, 8 up, 8 in 2021-07-31T02:45:42.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:41 smithi154 conmon[34553]: cluster 2021-07-31T02:45:42.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:41 smithi154 conmon[34553]: 2021-07-31T02:45 2021-07-31T02:45:42.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:41 smithi154 conmon[34553]: :40.636077+0000 mgr.x (mgr.24542) 2507 : cluster [DBG] pgmap v3714: 41 pgs: 41 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 141 KiB/s rd, 16 KiB/s wr, 201 op/s 2021-07-31T02:45:42.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:41 smithi154 conmon[34553]: cluster 2021-07-31T02 2021-07-31T02:45:42.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:41 smithi154 conmon[34553]: :45:40.664840+0000 mon.a (mon.0) 3141 : cluster 2021-07-31T02:45:42.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:41 smithi154 conmon[34553]: [DBG] osdmap e2115: 8 total, 8 up, 8 in 2021-07-31T02:45:42.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:41 smithi045 conmon[39718]: cluster 2021-07-31T02:45:40.636077+0000 mgr.x (mgr.24542) 2021-07-31T02:45:42.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:41 smithi045 conmon[39718]: 2507 : cluster [DBG] pgmap v3714: 41 pgs: 41 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 141 KiB/s rd, 16 KiB/s wr, 201 op/s 2021-07-31T02:45:42.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:41 smithi045 conmon[39718]: cluster 2021-07-31T02:45:40.664840+0000 mon.a (mon.0) 3141 : cluster [DBG] osdmap e2115: 8 total, 8 up, 8 in 2021-07-31T02:45:42.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:41 smithi045 conmon[39718]: 2021-07-31T02:45:42.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:41 smithi045 conmon[33655]: cluster 2021-07-31T02:45:40.636077+0000 mgr.x (mgr.24542) 2507 : cluster 2021-07-31T02:45:42.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:41 smithi045 conmon[33655]: [DBG] pgmap v3714: 41 pgs: 41 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 141 KiB/s rd, 16 KiB/s wr, 201 op/s 2021-07-31T02:45:42.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:41 smithi045 conmon[33655]: cluster 2021-07-31T02:45:40.664840+0000 mon.a (mon.0) 3141 : cluster [DBG] 2021-07-31T02:45:42.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:41 smithi045 conmon[33655]: osdmap e2115: 8 total, 8 up, 8 in 2021-07-31T02:45:42.398 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:45:42.689 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:45:42.689 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 60m ago 61m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:45:42.689 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 58m ago 61m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:45:42.690 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 58m ago 64m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:45:42.690 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 60m ago 66m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:45:42.690 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 60m ago 66m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:45:42.690 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 58m ago 64m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:45:42.691 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 60m ago 65m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:45:42.691 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 60m ago 61m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:45:42.691 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 58m ago 61m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:45:42.692 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 60m ago 64m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:45:42.692 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 60m ago 64m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:45:42.692 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 60m ago 63m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:45:42.692 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 60m ago 63m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:45:42.692 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 58m ago 63m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:45:42.693 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 58m ago 62m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:45:42.693 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 58m ago 62m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:45:42.693 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 58m ago 62m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:45:42.693 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 58m ago 61m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:45:42.706 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestImage.test_set_snap_recreated ... ok 2021-07-31T02:45:43.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:42 smithi154 conmon[34553]: cluster 2021-07-31T02:45:41.666187+0000 2021-07-31T02:45:43.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:42 smithi154 conmon[34553]: mon.a (mon.0) 3142 : cluster [DBG] osdmap e2116: 8 total, 8 up, 8 in 2021-07-31T02:45:43.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:42 smithi154 conmon[34553]: audit 2021-07-31T02:45:42.386256 2021-07-31T02:45:43.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:42 smithi154 conmon[34553]: +0000 mgr.x (mgr.24542) 2508 : audit [DBG] from='client.18006 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:45:43.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:42 smithi154 conmon[34553]: cluster 2021- 2021-07-31T02:45:43.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:42 smithi154 conmon[34553]: 07-31T02:45:42. 2021-07-31T02:45:43.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:42 smithi154 conmon[34553]: 603360+0000 mon.a (mon.0 2021-07-31T02:45:43.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:42 smithi154 conmon[34553]: ) 3143 : cluster [DBG] osdmap e2117: 8 total, 8 up, 8 in 2021-07-31T02:45:43.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:42 smithi045 conmon[33655]: cluster 2021-07-31T02:45:41.666187+0000 mon.a (mon. 2021-07-31T02:45:43.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:42 smithi045 conmon[33655]: 0) 3142 : cluster [DBG] osdmap e2116: 8 total, 8 up, 8 in 2021-07-31T02:45:43.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:42 smithi045 conmon[33655]: audit 2021-07-31T02:45:42.386256+0000 mgr.x (mgr.24542) 2508 : audit [DBG] from='client.18006 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:45:43.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:42 smithi045 conmon[33655]: cluster 2021-07-31T02:45:42.603360+0000 mon.a (mon.0) 3143 : cluster [DBG] osdmap e2117: 8 total, 8 up, 8 in 2021-07-31T02:45:43.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:42 smithi045 conmon[39718]: cluster 2021-07-31T02:45:41. 2021-07-31T02:45:43.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:42 smithi045 conmon[39718]: 666187+0000 mon.a (mon.0) 3142 : cluster [DBG] osdmap e2116: 8 total, 8 up, 8 in 2021-07-31T02:45:43.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:42 smithi045 conmon[39718]: audit 2021-07-31T02:45:43.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:42 smithi045 conmon[39718]: 2021-07-31T02:45:42.386256+0000 mgr.x (mgr.24542) 2021-07-31T02:45:43.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:42 smithi045 conmon[39718]: 2508 : audit [DBG] from='client.18006 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:45:43.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:42 smithi045 conmon[39718]: cluster 2021-07-31T02:45:43.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:42 smithi045 conmon[39718]: 2021-07-31T02:45:42.603360+0000 2021-07-31T02:45:43.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:42 smithi045 conmon[39718]: mon.a (mon.0) 3143 : cluster [DBG] 2021-07-31T02:45:43.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:42 smithi045 conmon[39718]: osdmap e2117: 8 total, 8 up, 8 in 2021-07-31T02:45:43.034 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:45:43.034 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:45:43.034 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:45:43.034 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:45:43.035 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:45:43.035 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:45:43.035 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:45:43.035 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:45:43.035 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:45:43.036 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:45:43.036 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:45:43.036 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:45:43.036 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:45:43.036 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:45:43.037 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:45:43.037 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:45:43.037 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:45:44.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:43 smithi154 conmon[34553]: cluster 2021-07-31T02:45:42.636680+0000 mgr.x (mgr.24542) 2509 : cluster [DBG] pgmap v3718: 41 pgs: 41 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 142 KiB/s rd, 17 KiB/s wr, 202 op/s 2021-07-31T02:45:44.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:43 smithi154 conmon[34553]: 2021-07-31T02:45:44.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:43 smithi154 conmon[34553]: audit 2021-07-31T02:45:42.680697+0000 mgr.x (mgr.24542) 2510 : audit [DBG] from='client.27952 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:45:44.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:43 smithi154 conmon[34553]: audit 2021-07-31T02:45:43.034232+0000 mon.c (mon.1) 132 : audit [DBG] from='client.? 172.21.15.45:0/4215722568' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:45:44.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:43 smithi045 conmon[33655]: cluster 2021-07-31T02:45:42.636680+0000 mgr.x (mgr.24542) 2509 : cluster [DBG] pgmap v3718: 41 pgs: 41 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 142 KiB/s rd, 17 KiB/s wr, 202 op/s 2021-07-31T02:45:44.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:43 smithi045 conmon[33655]: audit 2021-07-31T02:45:42.680697+0000 mgr.x (mgr.24542) 2510 : audit [DBG] from='client.27952 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:45:44.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:43 smithi045 conmon[33655]: audit 2021-07-31T02:45:43.034232+0000 mon.c (mon.1) 132 : audit [DBG] from='client.? 172.21.15.45:0/4215722568' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:45:44.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:43 smithi045 conmon[39718]: cluster 2021-07-31T02:45:42.636680+0000 mgr.x ( 2021-07-31T02:45:44.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:43 smithi045 conmon[39718]: mgr.24542) 2509 : cluster [DBG] pgmap v3718: 41 pgs: 41 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 142 KiB/s rd, 17 KiB/s wr, 202 op/s 2021-07-31T02:45:44.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:43 smithi045 conmon[39718]: audit 2021-07-31T02:45:42.680697 2021-07-31T02:45:44.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:43 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 2510 : audit [DBG] from='client.27952 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:45:44.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:43 smithi045 conmon[39718]: audit 2021-07-31T02:45:44.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:43 smithi045 conmon[39718]: 2021-07-31T02:45:43.034232+0000 mon.c (mon.1) 132 : audit [DBG] from='client.? 172.21.15.45:0/4215722568' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:45:44.792 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestImage.test_set_snap_sparse ... ok 2021-07-31T02:45:45.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:44 smithi154 conmon[34553]: cluster 2021-07-31T02:45:43.675227+0000 mon.a (mon.0) 3144 : 2021-07-31T02:45:45.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:44 smithi154 conmon[34553]: cluster [DBG] osdmap e2118: 8 total, 8 up, 8 in 2021-07-31T02:45:45.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:44 smithi045 conmon[39718]: cluster 2021-07-31T02:45:43.675227+0000 mon.a (mon.0) 3144 2021-07-31T02:45:45.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:44 smithi045 conmon[39718]: : cluster [DBG] osdmap e2118: 8 total, 8 up, 8 in 2021-07-31T02:45:45.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:44 smithi045 conmon[33655]: cluster 2021-07-31T02:45:43.675227+0000 2021-07-31T02:45:45.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:44 smithi045 conmon[33655]: mon.a (mon.0) 3144 : cluster [DBG] osdmap e2118: 8 total, 8 up, 8 in 2021-07-31T02:45:46.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:45 smithi154 conmon[34553]: cluster 2021-07-31T02:45:46.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:45 smithi154 conmon[34553]: 2021-07-31T02:45:44 2021-07-31T02:45:46.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:45 smithi154 conmon[34553]: .637353+0000 mgr.x (mgr.24542) 2511 : cluster [DBG] pgmap v3720: 41 pgs: 41 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 73 KiB/s rd, 10 KiB/s wr, 104 op/s 2021-07-31T02:45:46.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:45 smithi154 conmon[34553]: cluster 2021-07-31T02:45:44.685551+0000 mon.a (mon.0) 3145 : cluster 2021-07-31T02:45:46.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:45 smithi154 conmon[34553]: [DBG] osdmap e2119: 8 total, 8 up, 8 in 2021-07-31T02:45:46.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:45 smithi045 conmon[39718]: cluster 2021-07-31T02:45:44.637353+0000 mgr.x (mgr.24542) 2511 : cluster [DBG] 2021-07-31T02:45:46.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:45 smithi045 conmon[39718]: pgmap v3720: 41 pgs: 41 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 73 KiB/s rd, 10 KiB/s wr, 104 op/s 2021-07-31T02:45:46.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:45 smithi045 conmon[39718]: cluster 2021-07-31T02:45:44.685551+0000 mon.a (mon.0) 3145 : cluster [DBG] osdmap e2119: 8 total, 8 up, 8 in 2021-07-31T02:45:46.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:45 smithi045 conmon[33655]: cluster 2021-07-31T02:45:44.637353+0000 mgr.x (mgr.24542) 2511 : cluster 2021-07-31T02:45:46.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:45 smithi045 conmon[33655]: [DBG] pgmap v3720: 41 pgs: 41 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 73 KiB/s rd, 10 KiB/s wr, 104 op/s 2021-07-31T02:45:46.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:45 smithi045 conmon[33655]: cluster 2021-07-31T02:45:44.685551+0000 mon.a (mon 2021-07-31T02:45:46.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:45 smithi045 conmon[33655]: .0) 3145 : cluster [DBG] osdmap e2119: 8 total, 8 up, 8 in 2021-07-31T02:45:47.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:46 smithi154 conmon[34553]: cluster 2021-07-31T02:45:45.687280+0000 mon.a (mon 2021-07-31T02:45:47.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:46 smithi154 conmon[34553]: .0) 3146 : cluster [DBG] osdmap e2120: 8 total, 8 up, 8 in 2021-07-31T02:45:47.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:46 smithi045 conmon[33655]: cluster 2021-07-31T02:45:45.687280+0000 mon.a (mon.0) 3146 : 2021-07-31T02:45:47.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:46 smithi045 conmon[33655]: cluster [DBG] osdmap e2120: 8 total, 8 up, 8 in 2021-07-31T02:45:47.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:46 smithi045 conmon[39718]: cluster 2021-07-31T02:45:45.687280+0000 mon.a (mon 2021-07-31T02:45:47.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:46 smithi045 conmon[39718]: .0) 3146 : cluster [DBG] osdmap e2120: 8 total, 8 up, 8 in 2021-07-31T02:45:48.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:47 smithi154 conmon[34553]: cluster 2021-07-31T02:45:46.637803+0000 mgr.x (mgr.24542) 2512 : cluster [DBG] pgmap v3723: 41 pgs: 41 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 73 KiB/s rd, 9.9 KiB/s wr, 103 op/s 2021-07-31T02:45:48.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:47 smithi154 conmon[34553]: cluster 2021-07-31T02: 2021-07-31T02:45:48.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:47 smithi154 conmon[34553]: 45:46.697515+0000 mon.a (mon.0) 3147 : cluster [DBG] osdmap e2121: 8 total, 8 up, 8 in 2021-07-31T02:45:48.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:47 smithi154 conmon[34553]: cluster 2021-07-31T02:45:47.606190+0000 mon.a (mon.0) 3148 : cluster [DBG] osdmap e2122: 8 total, 8 up, 8 in 2021-07-31T02:45:48.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:47 smithi045 conmon[39718]: cluster 2021-07-31T02:45:46.637803+0000 mgr.x (mgr.24542) 2512 : cluster [DBG] pgmap v3723: 41 pgs: 41 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 73 KiB/s rd, 9.9 KiB/s wr, 103 op/s 2021-07-31T02:45:48.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:47 smithi045 conmon[39718]: cluster 2021-07-31T02:45:46.697515+0000 mon.a (mon.0) 3147 : cluster [DBG] osdmap e2121: 8 total, 8 up, 8 in 2021-07-31T02:45:48.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:47 smithi045 conmon[39718]: cluster 2021-07-31T02:45:48.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:47 smithi045 conmon[39718]: 2021-07-31T02:45:47.606190+0000 mon.a (mon. 2021-07-31T02:45:48.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:47 smithi045 conmon[39718]: 0) 3148 : cluster [DBG] osdmap e2122: 8 total, 8 up, 8 in 2021-07-31T02:45:48.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:47 smithi045 conmon[33655]: cluster 2021-07-31T02:45:46.637803+0000 mgr.x (mgr.24542) 2512 : cluster [DBG] pgmap v3723: 41 pgs: 41 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 73 KiB/s rd, 9.9 KiB/s wr, 103 op/s 2021-07-31T02:45:48.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:47 smithi045 conmon[33655]: cluster 2021-07-31T02:45:46.697515+0000 mon.a (mon.0) 3147 : cluster [DBG] osdmap e2121: 8 total, 8 up, 8 in 2021-07-31T02:45:48.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:47 smithi045 conmon[33655]: cluster 2021-07-31T 2021-07-31T02:45:48.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:47 smithi045 conmon[33655]: 02:45:47.606190+0000 mon.a (mon.0) 2021-07-31T02:45:48.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:47 smithi045 conmon[33655]: 3148 : cluster [DBG] osdmap e2122: 8 total, 8 up, 8 in 2021-07-31T02:45:48.710 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestImage.test_size ... ok 2021-07-31T02:45:50.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:49 smithi154 conmon[34553]: cluster 2021-07-31T02:45:48.612568+0000 mon.a (mon.0) 3149 : cluster 2021-07-31T02:45:50.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:49 smithi154 conmon[34553]: [DBG] osdmap e2123: 8 total, 8 up, 8 in 2021-07-31T02:45:50.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:49 smithi154 conmon[34553]: cluster 2021-07-31T02:45:48.638227+0000 mgr.x (mgr.24542) 2513 : cluster [DBG] pgmap v3727: 41 pgs: 41 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:45:50.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:49 smithi045 conmon[33655]: cluster 2021-07-31T02:45:48.612568+0000 mon.a (mon.0) 3149 : cluster [DBG] osdmap e2123: 8 total, 8 up, 8 in 2021-07-31T02:45:50.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:49 smithi045 conmon[33655]: cluster 2021-07-31T02:45:48.638227+0000 mgr.x (mgr.24542) 2513 : cluster [DBG] pgmap v3727: 41 pgs: 41 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:45:50.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:49 smithi045 conmon[39718]: cluster 2021-07-31T02:45:48.612568+0000 2021-07-31T02:45:50.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:49 smithi045 conmon[39718]: mon.a (mon.0) 3149 : cluster [DBG] osdmap e2123: 8 total, 8 up, 8 in 2021-07-31T02:45:50.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:49 smithi045 conmon[39718]: cluster 2021-07-31T02:45: 2021-07-31T02:45:50.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:49 smithi045 conmon[39718]: 48.638227+0000 mgr.x (mgr.24542) 2513 : cluster [DBG] pgmap v3727: 41 pgs: 41 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:45:50.723 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestImage.test_snap_exists ... ok 2021-07-31T02:45:51.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:50 smithi154 conmon[34553]: cluster 2021-07-31T02:45:49.617896+0000 mon.a (mon.0) 3150 : cluster [DBG] osdmap e2124: 8 total, 8 up, 8 in 2021-07-31T02:45:51.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:50 smithi045 conmon[33655]: cluster 2021-07-31T02:45:49.617896+0000 mon.a (mon.0) 3150 : cluster 2021-07-31T02:45:51.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:50 smithi045 conmon[33655]: [DBG] osdmap e2124: 8 total, 8 up, 8 in 2021-07-31T02:45:51.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:50 smithi045 conmon[39718]: cluster 2021-07-31T02:45:49.617896+0000 mon.a ( 2021-07-31T02:45:51.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:50 smithi045 conmon[39718]: mon.0) 3150 : cluster [DBG] osdmap e2124: 8 total, 8 up, 8 in 2021-07-31T02:45:52.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:51 smithi154 conmon[34553]: cluster 2021-07-31T02:45:50.624234+0000 mon.a (mon.0) 3151 : cluster [DBG] 2021-07-31T02:45:52.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:51 smithi154 conmon[34553]: osdmap e2125: 8 total, 8 up, 8 in 2021-07-31T02:45:52.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:51 smithi154 conmon[34553]: cluster 2021-07-31T02:45:50.638632+0000 mgr.x (mgr.24542) 2514 : cluster [DBG] pgmap v3730: 41 pgs: 41 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 105 KiB/s rd, 8.9 KiB/s wr, 147 op/s 2021-07-31T02:45:52.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:51 smithi045 conmon[39718]: cluster 2021-07-31T02:45:50. 2021-07-31T02:45:52.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:51 smithi045 conmon[39718]: 624234+0000 mon.a (mon.0) 3151 : cluster [DBG] osdmap e2125: 8 total, 8 up, 8 in 2021-07-31T02:45:52.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:51 smithi045 conmon[39718]: cluster 2021-07-31T02:45:52.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:51 smithi045 conmon[39718]: 2021-07-31T02:45:50.638632+0000 mgr.x (mgr.24542) 2514 : cluster 2021-07-31T02:45:52.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:51 smithi045 conmon[39718]: [DBG] pgmap v3730: 41 pgs: 41 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 105 KiB/s rd, 8.9 KiB/s wr, 147 op/s 2021-07-31T02:45:52.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:51 smithi045 conmon[33655]: cluster 2021-07-31T02:45:50.624234+0000 mon.a (mon.0) 3151 : cluster [DBG] osdmap e2125: 8 total, 8 up, 8 in 2021-07-31T02:45:52.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:51 smithi045 conmon[33655]: cluster 2021-07-31T02:45:50.638632+0000 mgr.x (mgr.24542) 2514 : cluster [DBG] pgmap v3730: 41 pgs: 41 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 105 KiB/s rd, 8.9 KiB/s wr, 147 op/s 2021-07-31T02:45:53.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:52 smithi154 conmon[34553]: cluster 2021-07-31T02:45:51.626962+0000 mon.a (mon.0) 3152 : cluster [DBG] osdmap e2126: 8 total, 8 up, 8 in 2021-07-31T02:45:53.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:52 smithi154 conmon[34553]: cluster 2021-07-31T02:45:52.607659+0000 mon.a (mon.0) 3153 : cluster [DBG] osdmap e2127: 8 total, 8 up, 8 in 2021-07-31T02:45:53.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:52 smithi045 conmon[39718]: cluster 2021-07-31T02:45:51.626962+0000 2021-07-31T02:45:53.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:52 smithi045 conmon[39718]: mon.a (mon.0) 3152 : cluster [DBG] osdmap e2126: 8 total, 8 up, 8 in 2021-07-31T02:45:53.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:52 smithi045 conmon[39718]: cluster 2021-07-31T02:45:52. 2021-07-31T02:45:53.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:52 smithi045 conmon[39718]: 607659+0000 mon.a (mon.0) 3153 : cluster [DBG] osdmap e2127: 8 total, 8 up, 8 in 2021-07-31T02:45:53.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:52 smithi045 conmon[33655]: cluster 2021-07-31T02:45:51.626962+0000 mon.a (mon.0) 2021-07-31T02:45:53.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:52 smithi045 conmon[33655]: 3152 : cluster [DBG] osdmap e2126: 8 total, 8 up, 8 in 2021-07-31T02:45:53.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:52 smithi045 conmon[33655]: cluster 2021-07-31T02:45:52.607659+0000 mon.a (mon.0) 3153 : cluster [DBG] osdmap e2127: 8 total, 8 up, 8 in 2021-07-31T02:45:54.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:53 smithi154 conmon[34553]: cluster 2021-07-31T02:45:52.639103+0000 mgr.x (mgr.24542 2021-07-31T02:45:54.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:53 smithi154 conmon[34553]: ) 2515 : cluster [DBG] pgmap v3733: 41 pgs: 41 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 103 KiB/s rd, 8.7 KiB/s wr, 144 op/s 2021-07-31T02:45:54.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:53 smithi045 conmon[39718]: cluster 2021-07-31T02:45:52.639103+0000 2021-07-31T02:45:54.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:53 smithi045 conmon[39718]: mgr.x (mgr.24542) 2515 : cluster [DBG] pgmap v3733: 41 pgs: 41 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 103 KiB/s rd, 8.7 KiB/s wr, 144 op/s 2021-07-31T02:45:54.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:53 smithi045 conmon[33655]: cluster 2021-07-31T02:45:52.639103+0000 2021-07-31T02:45:54.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:53 smithi045 conmon[33655]: mgr.x (mgr.24542) 2515 : cluster [DBG] pgmap v3733: 41 pgs: 41 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 103 KiB/s rd, 8.7 KiB/s wr, 144 op/s 2021-07-31T02:45:55.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:54 smithi154 conmon[34553]: cluster 2021-07-31T02:45:53.639704+0000 mon.a (mon.0) 3154 : cluster 2021-07-31T02:45:55.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:54 smithi154 conmon[34553]: [DBG] osdmap e2128: 8 total, 8 up, 8 in 2021-07-31T02:45:55.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:54 smithi045 conmon[39718]: cluster 2021-07-31T02:45:53.639704+0000 mon.a (mon. 2021-07-31T02:45:55.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:54 smithi045 conmon[39718]: 0) 3154 : cluster [DBG] osdmap e2128: 8 total, 8 up, 8 in 2021-07-31T02:45:55.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:54 smithi045 conmon[33655]: cluster 2021-07-31T02:45:53. 2021-07-31T02:45:55.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:54 smithi045 conmon[33655]: 639704+0000 mon.a (mon.0) 3154 : cluster [DBG] osdmap e2128: 8 total, 8 up, 8 in 2021-07-31T02:45:56.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:55 smithi154 conmon[34553]: cluster 2021-07-31T02:45:56.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:55 smithi154 conmon[34553]: 2021-07-31T02:45:54 2021-07-31T02:45:56.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:55 smithi154 conmon[34553]: .639618+0000 mgr.x (mgr.24542) 2516 : cluster [DBG] pgmap v3736: 41 pgs: 41 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 74 KiB/s rd, 6.5 KiB/s wr, 100 op/s 2021-07-31T02:45:56.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:55 smithi154 conmon[34553]: cluster 2021-07-31T02:45:56.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:55 smithi154 conmon[34553]: 2021-07-31T02:45:54.641211+0000 mon.a (mon.0) 3155 : cluster [DBG] osdmap e2129: 8 total, 8 up, 8 in 2021-07-31T02:45:56.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:55 smithi045 conmon[39718]: cluster 2021-07-31T02:45:54.639618+0000 2021-07-31T02:45:56.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:55 smithi045 conmon[39718]: mgr.x (mgr.24542) 2516 : cluster [DBG] pgmap v3736: 41 pgs: 41 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 74 KiB/s rd, 6.5 KiB/s wr, 100 op/s 2021-07-31T02:45:56.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:55 smithi045 conmon[39718]: cluster 2021-07-31T02:45:56.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:55 smithi045 conmon[39718]: 2021-07-31T02 2021-07-31T02:45:56.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:55 smithi045 conmon[39718]: :45:54.641211+0000 mon.a 2021-07-31T02:45:56.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:55 smithi045 conmon[39718]: (mon.0) 3155 : cluster [DBG] 2021-07-31T02:45:56.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:55 smithi045 conmon[39718]: osdmap e2129: 8 total, 8 up, 8 in 2021-07-31T02:45:56.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:55 smithi045 conmon[33655]: cluster 2021-07-31T02:45:56.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:55 smithi045 conmon[33655]: 2021-07-31T02:45:56.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:55 smithi045 conmon[33655]: 2021 2021-07-31T02:45:56.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:55 smithi045 conmon[33655]: -07-31T02:45:54.639618+0000 mgr.x (mgr 2021-07-31T02:45:56.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:55 smithi045 conmon[33655]: .24542) 2516 : cluster [DBG] pgmap v3736: 41 pgs: 41 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 74 KiB/s rd, 6.5 KiB/s wr, 100 op/s 2021-07-31T02:45:56.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:55 smithi045 conmon[33655]: cluster 2021-07-31T 2021-07-31T02:45:56.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:55 smithi045 conmon[33655]: 02:45:54.641211+0000 mon.a (mon.0) 3155 2021-07-31T02:45:56.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:55 smithi045 conmon[33655]: : cluster [DBG] osdmap e2129: 8 total, 8 up, 8 in 2021-07-31T02:45:56.763 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestImage.test_snap_get_id ... ok 2021-07-31T02:45:57.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:56 smithi154 conmon[34553]: cluster 2021-07-31T02:45:55.647603+0000 mon.a (mon.0) 3156 : cluster 2021-07-31T02:45:57.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:56 smithi154 conmon[34553]: [DBG] osdmap e2130: 8 total, 8 up, 8 in 2021-07-31T02:45:57.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:56 smithi045 conmon[33655]: cluster 2021-07-31T02:45:55.647603+0000 mon.a (mon. 2021-07-31T02:45:57.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:56 smithi045 conmon[33655]: 0) 3156 : cluster [DBG] osdmap e2130: 8 total, 8 up, 8 in 2021-07-31T02:45:57.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:56 smithi045 conmon[39718]: cluster 2021-07-31T02:45:55.647603 2021-07-31T02:45:57.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:56 smithi045 conmon[39718]: +0000 mon.a (mon.0) 3156 : cluster [DBG] osdmap e2130: 8 total, 8 up, 8 in 2021-07-31T02:45:58.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:57 smithi154 conmon[34553]: cluster 2021-07-31T02:45:56.640140+0000 mgr.x (mgr.24542) 2517 : cluster [DBG] 2021-07-31T02:45:58.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:57 smithi154 conmon[34553]: pgmap v3738: 41 pgs: 41 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 73 KiB/s rd, 6.4 KiB/s wr, 99 op/s 2021-07-31T02:45:58.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:57 smithi154 conmon[34553]: cluster 2021-07-31T02:45:56.660225+0000 mon.a (mon.0) 3157 : cluster [DBG] osdmap e2131: 8 total, 8 up, 8 in 2021-07-31T02:45:58.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:57 smithi154 conmon[34553]: cluster 2021-07-31T02:45:57 2021-07-31T02:45:58.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:57 smithi154 conmon[34553]: .608442+0000 mon.a (mon.0) 3158 : cluster [DBG] osdmap e2132: 8 total, 8 up, 8 in 2021-07-31T02:45:58.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:57 smithi045 conmon[33655]: cluster 2021-07-31T02:45:56.640140+0000 mgr.x (mgr.24542) 2021-07-31T02:45:58.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:57 smithi045 conmon[33655]: 2517 : cluster [DBG] pgmap v3738: 41 pgs: 41 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 73 KiB/s rd, 6.4 KiB/s wr, 99 op/s 2021-07-31T02:45:58.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:57 smithi045 conmon[33655]: cluster 2021-07-31T02:45:56.660225+0000 mon.a (mon.0) 3157 : cluster [DBG] osdmap e2131: 8 total, 8 up, 8 in 2021-07-31T02:45:58.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:57 smithi045 conmon[33655]: cluster 2021-07-31T02:45:57.608442+0000 mon.a (mon.0) 3158 : 2021-07-31T02:45:58.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:57 smithi045 conmon[33655]: cluster [DBG] osdmap e2132: 8 total, 8 up, 8 in 2021-07-31T02:45:58.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:57 smithi045 conmon[39718]: cluster 2021-07-31T02:45:56 2021-07-31T02:45:58.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:57 smithi045 conmon[39718]: .640140+0000 mgr.x (mgr.24542) 2517 : cluster [DBG] pgmap v3738: 41 pgs: 41 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 73 KiB/s rd, 6.4 KiB/s wr, 99 op/s 2021-07-31T02:45:58.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:57 smithi045 conmon[39718]: cluster 2021-07-31T02:45:58.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:57 smithi045 conmon[39718]: 2021-07-31T02:45 2021-07-31T02:45:58.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:57 smithi045 conmon[39718]: :56.660225+0000 mon.a 2021-07-31T02:45:58.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:57 smithi045 conmon[39718]: (mon.0) 3157 : cluster 2021-07-31T02:45:58.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:57 smithi045 conmon[39718]: [DBG] osdmap e2131: 8 total, 8 up, 8 in 2021-07-31T02:45:58.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:57 smithi045 conmon[39718]: cluster 2021- 2021-07-31T02:45:58.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:57 smithi045 conmon[39718]: 07-31T02:45:57.608442 2021-07-31T02:45:58.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:57 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2021-07-31T02:45:58.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:57 smithi045 conmon[39718]: 3158 : cluster [DBG] osdmap e2132: 8 total, 8 up, 8 in 2021-07-31T02:46:00.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:59 smithi154 conmon[34553]: cluster 2021-07-31T02:46:00.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:59 smithi154 conmon[34553]: 2021-07-31T02:45:58.615661 2021-07-31T02:46:00.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:59 smithi154 conmon[34553]: +0000 mon.a (mon.0) 3159 : cluster [DBG] osdmap e2133: 8 total, 8 up, 8 in 2021-07-31T02:46:00.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:59 smithi154 conmon[34553]: cluster 2021-07-31T02:46:00.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:59 smithi154 conmon[34553]: 2021-07-31T02:45:58. 2021-07-31T02:46:00.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:59 smithi154 conmon[34553]: 640595+0000 mgr.x (mgr.24542) 2518 : cluster 2021-07-31T02:46:00.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:45:59 smithi154 conmon[34553]: [DBG] pgmap v3742: 41 pgs: 41 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:46:00.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:59 smithi045 conmon[33655]: cluster 2021-07-31T02:45:58.615661+0000 mon.a (mon.0) 3159 2021-07-31T02:46:00.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:59 smithi045 conmon[33655]: : cluster [DBG] osdmap e2133: 8 total, 8 up, 8 in 2021-07-31T02:46:00.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:59 smithi045 conmon[33655]: cluster 2021-07-31T02:45:58.640595+0000 mgr.x (mgr.24542) 2518 2021-07-31T02:46:00.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:45:59 smithi045 conmon[33655]: : cluster [DBG] pgmap v3742: 41 pgs: 41 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:46:00.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:59 smithi045 conmon[39718]: cluster 2021-07-31T02:45:58.615661+0000 mon.a 2021-07-31T02:46:00.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:59 smithi045 conmon[39718]: (mon.0) 3159 : cluster [DBG] osdmap e2133: 8 total, 8 up, 8 in 2021-07-31T02:46:00.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:59 smithi045 conmon[39718]: cluster 2021-07-31T02:45:58.640595+0000 mgr.x (mgr.24542) 2518 2021-07-31T02:46:00.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:45:59 smithi045 conmon[39718]: : cluster [DBG] pgmap v3742: 41 pgs: 41 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:46:01.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:00 smithi154 conmon[34553]: cluster 2021-07-31T02:45:59.621489+0000 mon.a (mon.0) 3160 : cluster [DBG] osdmap e2134: 8 total, 8 up, 8 in 2021-07-31T02:46:01.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:00 smithi045 conmon[33655]: cluster 2021-07-31T02:45:59.621489 2021-07-31T02:46:01.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:00 smithi045 conmon[33655]: +0000 mon.a (mon.0) 3160 : cluster [DBG] osdmap e2134: 8 total, 8 up, 8 in 2021-07-31T02:46:01.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:00 smithi045 conmon[39718]: cluster 2021-07-31T02:45:59.621489+0000 mon.a (mon 2021-07-31T02:46:01.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:00 smithi045 conmon[39718]: .0) 3160 : cluster [DBG] osdmap e2134: 8 total, 8 up, 8 in 2021-07-31T02:46:02.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:01 smithi154 conmon[34553]: cluster 2021-07-31T02:46:02.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:01 smithi154 conmon[34553]: 2021-07-31T02:46:00.627750+0000 mon.a (mon.0) 3161 : cluster [DBG] osdmap e2135: 8 total, 8 up, 8 in 2021-07-31T02:46:02.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:01 smithi154 conmon[34553]: cluster 2021-07-31T02:46:00.640998+0000 mgr.x ( 2021-07-31T02:46:02.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:01 smithi154 conmon[34553]: mgr.24542) 2519 : cluster [DBG] pgmap v3745: 41 pgs: 41 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 71 KiB/s rd, 5.5 KiB/s wr, 96 op/s 2021-07-31T02:46:02.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:01 smithi045 conmon[33655]: cluster 2021-07-31T02:46:00.627750+0000 mon.a (mon.0 2021-07-31T02:46:02.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:01 smithi045 conmon[33655]: ) 3161 : cluster [DBG] osdmap e2135: 8 total, 8 up, 8 in 2021-07-31T02:46:02.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:01 smithi045 conmon[33655]: cluster 2021-07-31T02:46:00.640998+0000 mgr.x (mgr.24542) 2519 : cluster [DBG] pgmap v3745: 41 pgs: 41 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 71 KiB/s rd, 5.5 KiB/s wr, 96 op/s 2021-07-31T02:46:02.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:01 smithi045 conmon[39718]: cluster 2021-07-31T02:46:00. 2021-07-31T02:46:02.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:01 smithi045 conmon[39718]: 627750+0000 mon.a (mon.0) 3161 : cluster [DBG] osdmap e2135: 8 total, 8 up, 8 in 2021-07-31T02:46:02.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:01 smithi045 conmon[39718]: cluster 2021-07-31T02:46:02.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:01 smithi045 conmon[39718]: 2021-07-31T02:46:00.640998+0000 mgr.x (mgr.24542) 2519 : cluster 2021-07-31T02:46:02.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:01 smithi045 conmon[39718]: [DBG] pgmap v3745: 41 pgs: 41 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 71 KiB/s rd, 5.5 KiB/s wr, 96 op/s 2021-07-31T02:46:02.716 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestImage.test_snap_get_name ... ok 2021-07-31T02:46:03.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:02 smithi154 conmon[34553]: cluster 2021-07-31T02:46:01.634084+0000 mon.a (mon.0) 3162 : 2021-07-31T02:46:03.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:02 smithi154 conmon[34553]: cluster [DBG] osdmap e2136: 8 total, 8 up, 8 in 2021-07-31T02:46:03.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:02 smithi154 conmon[34553]: cluster 2021-07-31T02:46:02.609561+0000 mon.a (mon.0) 3163 : cluster [DBG] osdmap e2137: 8 total, 8 up, 8 in 2021-07-31T02:46:03.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:02 smithi045 conmon[39718]: cluster 2021-07-31T02:46:01.634084+0000 mon.a (mon.0) 2021-07-31T02:46:03.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:02 smithi045 conmon[39718]: 3162 : cluster [DBG] osdmap e2136: 8 total, 8 up, 8 in 2021-07-31T02:46:03.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:02 smithi045 conmon[39718]: cluster 2021-07-31T02:46:02.609561+0000 mon.a 2021-07-31T02:46:03.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:02 smithi045 conmon[39718]: (mon.0) 3163 : cluster [DBG] osdmap e2137: 8 total, 8 up, 8 in 2021-07-31T02:46:03.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:02 smithi045 conmon[33655]: cluster 2021-07-31T02:46:01.634084+0000 mon.a (mon.0) 2021-07-31T02:46:03.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:02 smithi045 conmon[33655]: 3162 : cluster [DBG] osdmap e2136: 8 total, 8 up, 8 in 2021-07-31T02:46:03.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:02 smithi045 conmon[33655]: cluster 2021-07-31T02:46:02.609561+0000 mon.a (mon.0) 3163 : cluster [DBG] osdmap e2137: 8 total, 8 up, 8 in 2021-07-31T02:46:04.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:03 smithi154 conmon[34553]: cluster 2021-07-31T02:46:02.641386+0000 mgr.x (mgr.24542) 2520 : cluster [DBG] pgmap v3748: 41 pgs: 41 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 71 KiB/s rd, 5.5 KiB/s wr, 96 op/s 2021-07-31T02:46:04.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:03 smithi045 conmon[39718]: cluster 2021-07-31T02:46:02.641386 2021-07-31T02:46:04.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:03 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 2520 : cluster [DBG] pgmap v3748: 41 pgs: 41 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 71 KiB/s rd, 5.5 KiB/s wr, 96 op/s 2021-07-31T02:46:04.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:03 smithi045 conmon[33655]: cluster 2021-07-31T02:46:02.641386+0000 mgr.x 2021-07-31T02:46:04.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:03 smithi045 conmon[33655]: (mgr.24542) 2520 : cluster [DBG] pgmap v3748: 41 pgs: 41 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 71 KiB/s rd, 5.5 KiB/s wr, 96 op/s 2021-07-31T02:46:04.736 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestImage.test_snap_timestamp ... ok 2021-07-31T02:46:04.824 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestImage.test_sparsify ... ok 2021-07-31T02:46:04.911 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestImage.test_stat ... ok 2021-07-31T02:46:04.998 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestImage.test_update_features ... SKIP 2021-07-31T02:46:05.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:04 smithi154 conmon[34553]: cluster 2021-07-31T02:46:03.642695+0000 mon.a (mon.0) 3164 : cluster [DBG] osdmap e2138: 8 total, 8 up, 8 in 2021-07-31T02:46:05.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:04 smithi045 conmon[39718]: cluster 2021-07-31T02:46:03.642695+0000 mon.a (mon.0) 2021-07-31T02:46:05.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:04 smithi045 conmon[39718]: 3164 : cluster [DBG] osdmap e2138: 8 total, 8 up, 8 in 2021-07-31T02:46:05.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:04 smithi045 conmon[33655]: cluster 2021-07-31T02:46:05.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:04 smithi045 conmon[33655]: 2021-07-31T02:46:05.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:04 smithi045 conmon[33655]: 2021 2021-07-31T02:46:05.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:04 smithi045 conmon[33655]: - 2021-07-31T02:46:05.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:04 smithi045 conmon[33655]: 07 2021-07-31T02:46:05.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:04 smithi045 conmon[33655]: - 2021-07-31T02:46:05.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:04 smithi045 conmon[33655]: 31 2021-07-31T02:46:05.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:04 smithi045 conmon[33655]: T 2021-07-31T02:46:05.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:04 smithi045 conmon[33655]: 02 2021-07-31T02:46:05.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:04 smithi045 conmon[33655]: :46:03.642695+0000 mon.a (mon.0) 3164 : cluster [DBG] osdmap e2138: 8 total, 8 up, 8 in 2021-07-31T02:46:05.074 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestImage.test_use_after_close ... ok 2021-07-31T02:46:05.156 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestImage.test_watchers_list ... ok 2021-07-31T02:46:05.252 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestImage.test_write ... ok 2021-07-31T02:46:05.346 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestImage.test_write_read ... ok 2021-07-31T02:46:05.440 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestImage.test_write_with_fadvise_flags ... ok 2021-07-31T02:46:05.532 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestImage.test_write_zeroes ... ok 2021-07-31T02:46:05.619 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestImage.test_write_zeroes_thick_provision ... ok 2021-07-31T02:46:05.713 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestImageId.test_read ... ok 2021-07-31T02:46:05.810 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestImageId.test_resize ... ok 2021-07-31T02:46:05.904 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestImageId.test_write ... ok 2021-07-31T02:46:06.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:05 smithi154 conmon[34553]: cluster 2021-07-31T02:46:04.641897+0000 mgr.x (mgr.24542) 2521 : cluster 2021-07-31T02:46:06.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:05 smithi154 conmon[34553]: [DBG] pgmap v3750: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 75 KiB/s rd, 7.0 KiB/s wr, 104 op/s 2021-07-31T02:46:06.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:05 smithi154 conmon[34553]: cluster 2021-07-31T02:46:04.644014+0000 mon.a (mon.0) 3165 : cluster [DBG] osdmap e2139: 8 total, 8 up, 8 in 2021-07-31T02:46:06.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:05 smithi045 conmon[33655]: cluster 2021-07-31T02:46:04.641897+0000 mgr.x (mgr.24542) 2521 : cluster [DBG] pgmap v3750: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 75 KiB/s rd, 7.0 KiB/s wr, 104 op/s 2021-07-31T02:46:06.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:05 smithi045 conmon[33655]: cluster 2021-07-31T02:46:04.644014+0000 mon.a (mon.0) 3165 : cluster [DBG] osdmap e2139: 8 total, 8 up, 8 in 2021-07-31T02:46:06.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:05 smithi045 conmon[39718]: cluster 2021-07-31T02:46:04.641897+0000 2021-07-31T02:46:06.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:05 smithi045 conmon[39718]: mgr.x (mgr.24542) 2521 : cluster [DBG] pgmap v3750: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 75 KiB/s rd, 7.0 KiB/s wr, 104 op/s 2021-07-31T02:46:06.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:05 smithi045 conmon[39718]: cluster 2021-07-31T 2021-07-31T02:46:06.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:05 smithi045 conmon[39718]: 02:46:04.644014+0000 mon.a (mon.0) 3165 : cluster [DBG] osdmap e2139: 8 total, 8 up, 8 in 2021-07-31T02:46:06.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:05 smithi045 conmon[39718]: 2021-07-31T02:46:06.104 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestMigration.test_migrate_abort ... ok 2021-07-31T02:46:06.290 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestMigration.test_migrate_abort_with_progress ... ok 2021-07-31T02:46:06.557 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestMigration.test_migration ... ok 2021-07-31T02:46:08.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:07 smithi154 conmon[34553]: cluster 2021-07-31T02:46:06.642288+0000 mgr.x (mgr.24542) 2522 : 2021-07-31T02:46:08.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:07 smithi154 conmon[34553]: cluster [DBG] pgmap v3752: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 60 KiB/s rd, 5.6 KiB/s wr, 83 op/s 2021-07-31T02:46:08.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:07 smithi154 conmon[34553]: cluster 2021-07-31T02:46:06.653053+0000 mon.a (mon.0) 3166 : cluster [DBG] osdmap e2140: 8 total, 8 up, 8 in 2021-07-31T02:46:08.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:07 smithi154 conmon[34553]: cluster 2021-07-31T02:46:07. 2021-07-31T02:46:08.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:07 smithi154 conmon[34553]: 611073+0000 mon.a (mon.0) 3167 : cluster [DBG] osdmap e2141: 8 total, 8 up, 8 in 2021-07-31T02:46:08.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:07 smithi045 conmon[39718]: cluster 2021-07-31T02:46:06.642288 2021-07-31T02:46:08.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:07 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 2522 : cluster [DBG] pgmap v3752: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 60 KiB/s rd, 5.6 KiB/s wr, 83 op/s 2021-07-31T02:46:08.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:07 smithi045 conmon[39718]: cluster 2021-07-31T02:46: 2021-07-31T02:46:08.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:07 smithi045 conmon[39718]: 06.653053+0000 mon.a (mon.0) 3166 : cluster [DBG] osdmap e2140: 8 total, 8 up, 8 in 2021-07-31T02:46:08.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:07 smithi045 conmon[39718]: cluster 2021-07- 2021-07-31T02:46:08.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:07 smithi045 conmon[39718]: 31T02:46:07.611073+0000 2021-07-31T02:46:08.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:07 smithi045 conmon[39718]: mon.a (mon.0) 3167 : 2021-07-31T02:46:08.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:07 smithi045 conmon[39718]: cluster [DBG] osdmap e2141: 8 total, 8 up, 8 in 2021-07-31T02:46:08.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:07 smithi045 conmon[33655]: cluster 2021-07-31T02:46:06.642288+0000 mgr.x (mgr.24542) 2522 2021-07-31T02:46:08.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:07 smithi045 conmon[33655]: : cluster [DBG] pgmap v3752: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 60 KiB/s rd, 5.6 KiB/s wr, 83 op/s 2021-07-31T02:46:08.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:07 smithi045 conmon[33655]: cluster 2021-07-31T02:46:06.653053+0000 mon.a (mon.0) 3166 : cluster [DBG] osdmap e2140: 8 total, 8 up, 8 in 2021-07-31T02:46:08.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:07 smithi045 conmon[33655]: cluster 2021-07-31T02:46:07.611073+0000 mon.a (mon.0 2021-07-31T02:46:08.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:07 smithi045 conmon[33655]: ) 3167 : cluster [DBG] osdmap e2141: 8 total, 8 up, 8 in 2021-07-31T02:46:09.821 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestMigration.test_migration_import ... ok 2021-07-31T02:46:10.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:09 smithi154 conmon[34553]: cluster 2021-07-31T02:46:08.642750+0000 mgr.x (mgr.24542) 2021-07-31T02:46:10.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:09 smithi154 conmon[34553]: 2523 : cluster [DBG] pgmap v3755: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 60 KiB/s rd, 5.6 KiB/s wr, 83 op/s 2021-07-31T02:46:10.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:09 smithi154 conmon[34553]: cluster 2021-07-31T02:46:08.660316+0000 mon.a (mon.0) 3168 : cluster [DBG] osdmap e2142: 8 total, 8 up, 8 in 2021-07-31T02:46:10.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:09 smithi045 conmon[39718]: cluster 2021-07-31T02:46:08.642750+0000 mgr.x (mgr. 2021-07-31T02:46:10.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:09 smithi045 conmon[39718]: 24542) 2523 : cluster [DBG] pgmap v3755: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 60 KiB/s rd, 5.6 KiB/s wr, 83 op/s 2021-07-31T02:46:10.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:09 smithi045 conmon[39718]: cluster 2021-07-31T02:46: 2021-07-31T02:46:10.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:09 smithi045 conmon[39718]: 08.660316+0000 mon.a (mon.0) 3168 : cluster [DBG] osdmap e2142: 8 total, 8 up, 8 in 2021-07-31T02:46:10.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:09 smithi045 conmon[33655]: cluster 2021-07-31T02:46:10.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:09 smithi045 conmon[33655]: 2021-07-31T02:46:10.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:09 smithi045 conmon[33655]: 2021 2021-07-31T02:46:10.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:09 smithi045 conmon[33655]: - 2021-07-31T02:46:10.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:09 smithi045 conmon[33655]: 07 2021-07-31T02:46:10.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:09 smithi045 conmon[33655]: - 2021-07-31T02:46:10.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:09 smithi045 conmon[33655]: 31 2021-07-31T02:46:10.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:09 smithi045 conmon[33655]: T 2021-07-31T02:46:10.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:09 smithi045 conmon[33655]: 02 2021-07-31T02:46:10.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:09 smithi045 conmon[33655]: : 2021-07-31T02:46:10.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:09 smithi045 conmon[33655]: 46:08.642750+0000 mgr.x (mgr.24542) 2523 : cluster 2021-07-31T02:46:10.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:09 smithi045 conmon[33655]: [DBG] pgmap v3755: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 60 KiB/s rd, 5.6 KiB/s wr, 83 op/s 2021-07-31T02:46:10.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:09 smithi045 conmon[33655]: cluster 2021-07-31T02:46:08. 2021-07-31T02:46:10.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:09 smithi045 conmon[33655]: 660316+0000 mon.a (mon.0) 3168 : cluster 2021-07-31T02:46:10.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:09 smithi045 conmon[33655]: [DBG] osdmap e2142: 8 total, 8 up, 8 in 2021-07-31T02:46:10.050 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestMigration.test_migration_with_progress ... ok 2021-07-31T02:46:11.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:10 smithi154 conmon[34553]: cluster 2021-07-31T02:46 2021-07-31T02:46:11.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:10 smithi154 conmon[34553]: :09.662562+0000 mon.a (mon.0) 3169 : cluster [DBG] osdmap e2143: 8 total, 8 up, 8 in 2021-07-31T02:46:11.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:10 smithi045 conmon[33655]: cluster 2021-07-31T02:46:09.662562+0000 mon.a (mon.0 2021-07-31T02:46:11.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:10 smithi045 conmon[33655]: ) 3169 : cluster [DBG] osdmap e2143: 8 total, 8 up, 8 in 2021-07-31T02:46:11.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:10 smithi045 conmon[39718]: cluster 2021-07-31T02:46:09.662562+0000 mon.a (mon 2021-07-31T02:46:11.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:10 smithi045 conmon[39718]: .0) 3169 : cluster [DBG] osdmap e2143: 8 total, 8 up, 8 in 2021-07-31T02:46:12.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:11 smithi154 conmon[34553]: cluster 2021-07-31T02:46:10.643174+0000 mgr.x (mgr.24542) 2524 : cluster [DBG] pgmap v3758: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 965 KiB/s rd, 61 KiB/s wr, 1.30k op/s 2021-07-31T02:46:12.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:11 smithi154 conmon[34553]: 2021-07-31T02:46:12.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:11 smithi154 conmon[34553]: cluster 2021-07-31T02:46:10.672731+0000 mon.a (mon.0) 3170 : cluster [DBG] osdmap e2144: 8 total, 8 up, 8 in 2021-07-31T02:46:12.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:11 smithi045 conmon[39718]: cluster 2021-07-31T02:46:10.643174+0000 mgr.x 2021-07-31T02:46:12.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:11 smithi045 conmon[39718]: (mgr.24542) 2524 : cluster [DBG] pgmap v3758: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 965 KiB/s rd, 61 KiB/s wr, 1.30k op/s 2021-07-31T02:46:12.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:11 smithi045 conmon[39718]: cluster 2021- 2021-07-31T02:46:12.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:11 smithi045 conmon[39718]: 07-31T02:46:10.672731+0000 mon.a ( 2021-07-31T02:46:12.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:11 smithi045 conmon[39718]: mon.0) 3170 : cluster 2021-07-31T02:46:12.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:11 smithi045 conmon[39718]: [DBG] osdmap e2144: 8 total, 8 up, 8 in 2021-07-31T02:46:12.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:11 smithi045 conmon[33655]: cluster 2021-07-31T02:46:10.643174+0000 2021-07-31T02:46:12.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:11 smithi045 conmon[33655]: mgr.x (mgr.24542) 2524 : cluster [DBG] pgmap v3758: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 965 KiB/s rd, 61 KiB/s wr, 1.30k op/s 2021-07-31T02:46:12.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:11 smithi045 conmon[33655]: cluster 2021-07-31T02:46:10.672731+0000 mon.a (mon.0) 3170 : cluster [DBG] osdmap e2144: 8 total, 8 up, 8 in 2021-07-31T02:46:13.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:12 smithi154 conmon[34553]: cluster 2021-07-31T02:46:11.674397+0000 mon.a (mon.0) 3171 : 2021-07-31T02:46:13.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:12 smithi154 conmon[34553]: cluster [DBG] osdmap e2145: 8 total, 8 up, 8 in 2021-07-31T02:46:13.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:12 smithi154 conmon[34553]: audit 2021-07-31T02:46:11.697510+0000 mon.c (mon.1) 133 : audit [INF] from='client.? 172.21.15.45:0/530405139' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/83/501f0719-add6-45d8-a0a3-f7b666f3c377"}]: dispatch 2021-07-31T02:46:13.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:12 smithi154 conmon[34553]: audit 2021- 2021-07-31T02:46:13.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:12 smithi154 conmon[34553]: 07-31T02:46:11.697842+0000 mon.a (mon.0) 3172 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/83/501f0719-add6-45d8-a0a3-f7b666f3c377"}]: dispatch 2021-07-31T02:46:13.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:12 smithi154 conmon[34553]: audit 2021-07-31T02: 2021-07-31T02:46:13.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:12 smithi154 conmon[34553]: 46:11.701037+0000 mon.a (mon.0) 3173 : audit [INF] 2021-07-31T02:46:13.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:12 smithi154 conmon[34553]: from='client.? ' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/83/501f0719-add6-45d8-a0a3-f7b666f3c377"}]': finished 2021-07-31T02:46:13.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:12 smithi154 conmon[34553]: cluster 2021-07-31T02 2021-07-31T02:46:13.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:12 smithi154 conmon[34553]: :46:12.612475+0000 mon.a (mon.0 2021-07-31T02:46:13.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:12 smithi154 conmon[34553]: ) 3174 : cluster [DBG] osdmap e2146: 8 total, 8 up, 8 in 2021-07-31T02:46:13.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:12 smithi045 conmon[33655]: cluster 2021-07-31T02:46:11 2021-07-31T02:46:13.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:12 smithi045 conmon[33655]: .674397+0000 mon.a (mon.0) 3171 : cluster [DBG] osdmap e2145: 8 total, 8 up, 8 in 2021-07-31T02:46:13.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:12 smithi045 conmon[33655]: audit 2021-07-31T02:46:13.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:12 smithi045 conmon[33655]: 2021-07-31T02:46:11.697510+0000 mon.c (mon.1) 133 : 2021-07-31T02:46:13.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:12 smithi045 conmon[33655]: audit [INF] from='client.? 172.21.15.45:0/530405139' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/83/501f0719-add6-45d8-a0a3-f7b666f3c377"}]: dispatch 2021-07-31T02:46:13.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:12 smithi045 conmon[33655]: audit 2021-07-31T02:46:11.697842+0000 mon.a (mon.0) 3172 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/83/501f0719-add6-45d8-a0a3-f7b666f3c377"}]: dispatch 2021-07-31T02:46:13.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:12 smithi045 conmon[33655]: audit 2021-07-31T02:46:11.701037+0000 mon.a (mon.0) 3173 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/83/501f0719-add6-45d8-a0a3-f7b666f3c377"}]': finished 2021-07-31T02:46:13.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:12 smithi045 conmon[33655]: cluster 2021-07-31T02:46:12.612475+0000 2021-07-31T02:46:13.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:12 smithi045 conmon[33655]: mon.a (mon.0) 3174 : cluster [DBG] osdmap e2146: 8 total, 8 up, 8 in 2021-07-31T02:46:13.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:12 smithi045 conmon[39718]: cluster 2021-07-31T02:46:11.674397+0000 mon.a (mon 2021-07-31T02:46:13.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:12 smithi045 conmon[39718]: .0) 3171 : cluster [DBG] osdmap e2145: 8 total, 8 up, 8 in 2021-07-31T02:46:13.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:12 smithi045 conmon[39718]: audit 2021-07-31T02:46:11.697510+0000 2021-07-31T02:46:13.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:12 smithi045 conmon[39718]: mon.c (mon.1) 133 : audit [INF] from='client.? 172.21.15.45:0/530405139' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/83/501f0719-add6-45d8-a0a3-f7b666f3c377"}]: dispatch 2021-07-31T02:46:13.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:12 smithi045 conmon[39718]: audit 2021-07-31T02:46 2021-07-31T02:46:13.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:12 smithi045 conmon[39718]: :11.697842+0000 mon.a (mon.0) 3172 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/83/501f0719-add6-45d8-a0a3-f7b666f3c377"}]: dispatch 2021-07-31T02:46:13.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:12 smithi045 conmon[39718]: audit 2021 2021-07-31T02:46:13.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:12 smithi045 conmon[39718]: -07-31T02:46:11.701037+0000 mon.a (mon.0) 3173 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/83/501f0719-add6-45d8-a0a3-f7b666f3c377"}]': finished 2021-07-31T02:46:13.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:12 smithi045 conmon[39718]: 2021-07-31T02:46:13.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:12 smithi045 conmon[39718]: cluster 2021-07-31T02:46:12. 2021-07-31T02:46:13.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:12 smithi045 conmon[39718]: 612475+0000 mon.a (mon.0) 3174 : cluster [DBG] osdmap e2146: 8 total, 8 up, 8 in 2021-07-31T02:46:13.344 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:46:13.622 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:46:13.622 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 61m ago 61m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:46:13.623 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 58m ago 61m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:46:13.623 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 58m ago 65m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:46:13.623 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 61m ago 67m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:46:13.624 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 61m ago 67m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:46:13.624 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 58m ago 65m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:46:13.625 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 61m ago 65m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:46:13.625 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 61m ago 62m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:46:13.625 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 58m ago 62m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:46:13.626 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 61m ago 64m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:46:13.626 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 61m ago 64m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:46:13.627 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 61m ago 64m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:46:13.627 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 61m ago 63m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:46:13.628 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 58m ago 63m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:46:13.628 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 58m ago 63m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:46:13.628 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 58m ago 62m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:46:13.629 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 58m ago 62m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:46:13.630 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 58m ago 62m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:46:13.742 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestMirroring.test_aio_mirror_image_create_snapshot ... ok 2021-07-31T02:46:13.833 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestMirroring.test_mirror_image ... SKIP 2021-07-31T02:46:13.987 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:46:13.987 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:46:13.988 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:46:13.988 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:46:13.988 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:46:13.988 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:46:13.989 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:46:13.989 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:46:13.989 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:46:13.989 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:46:13.990 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:46:13.990 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:46:13.990 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:46:13.990 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:46:13.990 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:46:13.991 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:46:13.991 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:46:14.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:13 smithi154 conmon[34553]: cluster 2021-07-31T02:46:12.643628+0000 mgr.x (mgr.24542) 2525 : cluster 2021-07-31T02:46:14.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:13 smithi154 conmon[34553]: [DBG] pgmap v3762: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 970 KiB/s rd, 62 KiB/s wr, 1.31k op/s 2021-07-31T02:46:14.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:13 smithi154 conmon[34553]: audit 2021-07-31T02:46:13.332915+0000 mgr.x (mgr.24542) 2526 : audit [DBG] from='client.18024 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:46:14.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:13 smithi045 conmon[39718]: cluster 2021-07-31T02:46: 2021-07-31T02:46:14.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:13 smithi045 conmon[39718]: 12.643628+0000 mgr.x (mgr.24542) 2525 : cluster [DBG] pgmap v3762: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 970 KiB/s rd, 62 KiB/s wr, 1.31k op/s 2021-07-31T02:46:14.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:13 smithi045 conmon[39718]: audit 2021 2021-07-31T02:46:14.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:13 smithi045 conmon[39718]: -07-31T02:46:13. 2021-07-31T02:46:14.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:13 smithi045 conmon[39718]: 332915+0000 mgr.x (mgr.24542) 2526 2021-07-31T02:46:14.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:13 smithi045 conmon[39718]: : audit [DBG] from='client.18024 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:46:14.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:13 smithi045 conmon[33655]: cluster 2021-07-31T02:46:14.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:13 smithi045 conmon[33655]: 2021-07-31T02:46:14.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:13 smithi045 conmon[33655]: 2021 2021-07-31T02:46:14.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:13 smithi045 conmon[33655]: - 2021-07-31T02:46:14.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:13 smithi045 conmon[33655]: 07 2021-07-31T02:46:14.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:13 smithi045 conmon[33655]: - 2021-07-31T02:46:14.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:13 smithi045 conmon[33655]: 31 2021-07-31T02:46:14.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:13 smithi045 conmon[33655]: T 2021-07-31T02:46:14.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:13 smithi045 conmon[33655]: 02 2021-07-31T02:46:14.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:13 smithi045 conmon[33655]: : 2021-07-31T02:46:14.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:13 smithi045 conmon[33655]: 46:12.643628+0000 mgr.x ( 2021-07-31T02:46:14.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:13 smithi045 conmon[33655]: mgr.24542) 2525 : cluster [DBG] pgmap v3762: 41 pgs: 41 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 970 KiB/s rd, 62 KiB/s wr, 1.31k op/s 2021-07-31T02:46:14.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:13 smithi045 conmon[33655]: audit 2021-07-31T02:46:14.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:13 smithi045 conmon[33655]: 2021-07-31T02:46:14.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:13 smithi045 conmon[33655]: 2021 2021-07-31T02:46:14.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:13 smithi045 conmon[33655]: - 2021-07-31T02:46:14.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:13 smithi045 conmon[33655]: 07 2021-07-31T02:46:14.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:13 smithi045 conmon[33655]: - 2021-07-31T02:46:14.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:13 smithi045 conmon[33655]: 31 2021-07-31T02:46:14.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:13 smithi045 conmon[33655]: T 2021-07-31T02:46:14.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:13 smithi045 conmon[33655]: 02 2021-07-31T02:46:14.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:13 smithi045 conmon[33655]: : 2021-07-31T02:46:14.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:13 smithi045 conmon[33655]: 46 2021-07-31T02:46:14.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:13 smithi045 conmon[33655]: : 2021-07-31T02:46:14.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:13 smithi045 conmon[33655]: 13 2021-07-31T02:46:14.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:13 smithi045 conmon[33655]: . 2021-07-31T02:46:14.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:13 smithi045 conmon[33655]: 332915 2021-07-31T02:46:14.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:13 smithi045 conmon[33655]: +0000 2021-07-31T02:46:14.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:13 smithi045 conmon[33655]: 2021-07-31T02:46:14.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:13 smithi045 conmon[33655]: mgr.x 2021-07-31T02:46:14.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:13 smithi045 conmon[33655]: ( 2021-07-31T02:46:14.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:13 smithi045 conmon[33655]: mgr 2021-07-31T02:46:14.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:13 smithi045 conmon[33655]: . 2021-07-31T02:46:14.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:13 smithi045 conmon[33655]: 24542 2021-07-31T02:46:14.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:13 smithi045 conmon[33655]: ) 2021-07-31T02:46:14.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:13 smithi045 conmon[33655]: 2526 2021-07-31T02:46:14.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:13 smithi045 conmon[33655]: : 2021-07-31T02:46:14.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:13 smithi045 conmon[33655]: audit 2021-07-31T02:46:14.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:13 smithi045 conmon[33655]: 2021-07-31T02:46:14.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:13 smithi045 conmon[33655]: [DBG] 2021-07-31T02:46:14.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:13 smithi045 conmon[33655]: 2021-07-31T02:46:14.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:13 smithi045 conmon[33655]: from='client.18024 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:46:14.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:13 smithi045 conmon[33655]: 2021-07-31T02:46:15.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:14 smithi154 conmon[34553]: audit 2021-07-31T02:46:13.613395+0000 mgr.x (mgr.24542) 2527 2021-07-31T02:46:15.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:14 smithi154 conmon[34553]: : audit [DBG] from='client.18030 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:46:15.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:14 smithi154 conmon[34553]: cluster 2021-07-31T02:46:13.681851+0000 mon.a (mon.0) 3175 : cluster [DBG] osdmap e2147: 8 total, 8 up, 8 in 2021-07-31T02:46:15.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:14 smithi154 conmon[34553]: audit 2021-07-31T02:46:13.987324+0000 mon.c (mon.1) 134 : audit [DBG] from='client.? 172.21.15.45:0/2077233712' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:46:15.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:14 smithi045 conmon[39718]: audit 2021-07-31T02:46:13.613395+0000 2021-07-31T02:46:15.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:14 smithi045 conmon[39718]: mgr.x (mgr.24542) 2527 : audit [DBG] from='client.18030 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:46:15.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:14 smithi045 conmon[39718]: cluster 2021-07-31T02:46:15.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:14 smithi045 conmon[39718]: 2021-07-31T02:46:13.681851 2021-07-31T02:46:15.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:14 smithi045 conmon[39718]: +0000 mon.a (mon.0) 3175 2021-07-31T02:46:15.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:14 smithi045 conmon[39718]: : cluster [DBG] osdmap e2147: 8 total, 8 up, 8 in 2021-07-31T02:46:15.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:14 smithi045 conmon[39718]: audit 2021 2021-07-31T02:46:15.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:14 smithi045 conmon[39718]: -07-31T02:46:13.987324 2021-07-31T02:46:15.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:14 smithi045 conmon[39718]: +0000 mon.c (mon.1) 2021-07-31T02:46:15.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:14 smithi045 conmon[39718]: 134 : audit [DBG] from='client.? 172.21.15.45:0/2077233712' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:46:15.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:14 smithi045 conmon[33655]: audit 2021-07-31T02:46:13.613395+0000 mgr.x (mgr.24542) 2021-07-31T02:46:15.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:14 smithi045 conmon[33655]: 2527 : audit [DBG] from='client.18030 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:46:15.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:14 smithi045 conmon[33655]: cluster 2021-07-31T02:46:13.681851+0000 mon.a (mon.0) 3175 : cluster [DBG] osdmap e2147: 8 total, 8 up, 8 in 2021-07-31T02:46:15.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:14 smithi045 conmon[33655]: audit 2021-07-31T02:46:13.987324+0000 2021-07-31T02:46:15.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:14 smithi045 conmon[33655]: mon.c (mon.1) 134 : audit [DBG] from='client.? 172.21.15.45:0/2077233712' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:46:15.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:14 smithi045 conmon[33655]: 2021-07-31T02:46:16.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:15 smithi154 conmon[34553]: cluster 2021-07-31T02:46:14.644283+0000 mgr.x (mgr.24542) 2528 : cluster [DBG] pgmap v3764: 41 pgs: 41 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 301 KiB/s rd, 22 KiB/s wr, 404 op/s 2021-07-31T02:46:16.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:15 smithi154 conmon[34553]: 2021-07-31T02:46:16.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:15 smithi154 conmon[34553]: cluster 2021-07-31T02:46:14.692470+0000 mon.a (mon.0) 3176 : cluster [DBG] osdmap e2148: 8 total, 8 up, 8 in 2021-07-31T02:46:16.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:15 smithi045 conmon[33655]: cluster 2021-07-31T02:46:14.644283+0000 mgr.x (mgr. 2021-07-31T02:46:16.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:15 smithi045 conmon[33655]: 24542) 2528 : cluster [DBG] pgmap v3764: 41 pgs: 41 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 301 KiB/s rd, 22 KiB/s wr, 404 op/s 2021-07-31T02:46:16.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:15 smithi045 conmon[33655]: cluster 2021-07-31T02:46:14.692470+0000 mon.a (mon.0) 3176 : cluster 2021-07-31T02:46:16.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:15 smithi045 conmon[33655]: [DBG] osdmap e2148: 8 total, 8 up, 8 in 2021-07-31T02:46:16.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:15 smithi045 conmon[39718]: cluster 2021-07-31T02:46:16.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:15 smithi045 conmon[39718]: 2021-07-31T02:46:16.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:15 smithi045 conmon[39718]: 2021 2021-07-31T02:46:16.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:15 smithi045 conmon[39718]: - 2021-07-31T02:46:16.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:15 smithi045 conmon[39718]: 07-31T02:46:14.644283+0000 mgr.x (mgr 2021-07-31T02:46:16.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:15 smithi045 conmon[39718]: .24542) 2528 : cluster [DBG] pgmap v3764: 41 pgs: 41 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 301 KiB/s rd, 22 KiB/s wr, 404 op/s 2021-07-31T02:46:16.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:15 smithi045 conmon[39718]: cluster 2021-07 2021-07-31T02:46:16.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:15 smithi045 conmon[39718]: -31T02:46:14.692470+0000 mon.a (mon 2021-07-31T02:46:16.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:15 smithi045 conmon[39718]: .0) 3176 : cluster [DBG] osdmap e2148: 8 total, 8 up, 8 in 2021-07-31T02:46:17.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:16 smithi154 conmon[34553]: cluster 2021-07-31T02:46:15.694150+0000 mon.a (mon.0) 3177 : cluster 2021-07-31T02:46:17.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:16 smithi154 conmon[34553]: [DBG] osdmap e2149: 8 total, 8 up, 8 in 2021-07-31T02:46:17.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:16 smithi045 conmon[39718]: cluster 2021-07-31T02:46:15.694150+0000 mon.a (mon.0) 2021-07-31T02:46:17.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:16 smithi045 conmon[39718]: 3177 : cluster [DBG] osdmap e2149: 8 total, 8 up, 8 in 2021-07-31T02:46:17.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:16 smithi045 conmon[33655]: cluster 2021-07-31T02:46:15.694150+0000 mon.a (mon.0) 3177 2021-07-31T02:46:17.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:16 smithi045 conmon[33655]: : cluster [DBG] osdmap e2149: 8 total, 8 up, 8 in 2021-07-31T02:46:18.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:17 smithi154 conmon[34553]: cluster 2021-07-31T02:46:16.644673+0000 mgr.x (mgr.24542) 2529 : cluster 2021-07-31T02:46:18.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:17 smithi154 conmon[34553]: [DBG] pgmap v3767: 41 pgs: 41 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 298 KiB/s rd, 22 KiB/s wr, 400 op/s 2021-07-31T02:46:18.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:17 smithi154 conmon[34553]: cluster 2021-07-31T02:46:16.704985+0000 mon.a (mon.0) 3178 : cluster [DBG] osdmap e2150: 8 total, 8 up, 8 in 2021-07-31T02:46:18.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:17 smithi154 conmon[34553]: audit 2021-07-31T02:46:16 2021-07-31T02:46:18.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:17 smithi154 conmon[34553]: .732407+0000 mon.c (mon.1) 135 : audit [INF] from='client.? 172.21.15.45:0/530405139' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/83/039f89bd-1df4-43d1-8eae-a25ba27cc433"}]: dispatch 2021-07-31T02:46:18.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:17 smithi154 conmon[34553]: audit 2021-07-31T02:46:18.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:17 smithi154 conmon[34553]: 2021-07-31T02:46: 2021-07-31T02:46:18.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:17 smithi154 conmon[34553]: 16.732811+0000 mon.a (mon.0 2021-07-31T02:46:18.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:17 smithi154 conmon[34553]: ) 3179 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/83/039f89bd-1df4-43d1-8eae-a25ba27cc433"}]: dispatch 2021-07-31T02:46:18.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:17 smithi154 conmon[34553]: audit 2021- 2021-07-31T02:46:18.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:17 smithi154 conmon[34553]: 07-31T02:46:16.736035+0000 2021-07-31T02:46:18.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:17 smithi154 conmon[34553]: mon.a (mon.0) 3180 : 2021-07-31T02:46:18.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:17 smithi154 conmon[34553]: audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/83/039f89bd-1df4-43d1-8eae-a25ba27cc433"}]': finished 2021-07-31T02:46:18.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:17 smithi154 conmon[34553]: audit 2021 2021-07-31T02:46:18.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:17 smithi154 conmon[34553]: -07-31T02:46:16. 2021-07-31T02:46:18.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:17 smithi154 conmon[34553]: 766976+0000 mon.c (mon.1) 136 2021-07-31T02:46:18.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:17 smithi154 conmon[34553]: : audit [INF] from='client.? 172.21.15.45:0/530405139' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/83/83389db2-9ce3-4663-be48-292f0573742d"}]: dispatch 2021-07-31T02:46:18.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:17 smithi154 conmon[34553]: audit 2021- 2021-07-31T02:46:18.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:17 smithi154 conmon[34553]: 07-31T02:46:16.767289 2021-07-31T02:46:18.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:17 smithi154 conmon[34553]: +0000 mon.a (mon.0) 3181 2021-07-31T02:46:18.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:17 smithi154 conmon[34553]: : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/83/83389db2-9ce3-4663-be48-292f0573742d"}]: dispatch 2021-07-31T02:46:18.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:17 smithi154 conmon[34553]: audit 2021 2021-07-31T02:46:18.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:17 smithi154 conmon[34553]: -07-31T02:46:16 2021-07-31T02:46:18.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:17 smithi154 conmon[34553]: .770230+0000 mon.a (mon.0 2021-07-31T02:46:18.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:17 smithi154 conmon[34553]: ) 3182 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/83/83389db2-9ce3-4663-be48-292f0573742d"}]': finished 2021-07-31T02:46:18.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:17 smithi154 conmon[34553]: cluster 2021- 2021-07-31T02:46:18.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:17 smithi154 conmon[34553]: 07-31T02:46:17.632461 2021-07-31T02:46:18.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:17 smithi154 conmon[34553]: +0000 mon.a (mon.0) 3183 2021-07-31T02:46:18.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:17 smithi154 conmon[34553]: : cluster [DBG] osdmap e2151: 8 total, 8 up, 8 in 2021-07-31T02:46:18.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:17 smithi045 conmon[39718]: cluster 2021-07-31T02:46:16.644673 2021-07-31T02:46:18.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:17 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 2529 : cluster [DBG] pgmap v3767: 41 pgs: 41 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 298 KiB/s rd, 22 KiB/s wr, 400 op/s 2021-07-31T02:46:18.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:17 smithi045 conmon[39718]: cluster 2021-07-31T 2021-07-31T02:46:18.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:17 smithi045 conmon[39718]: 02:46:16.704985+0000 mon.a (mon.0) 3178 : cluster [DBG] osdmap e2150: 8 total, 8 up, 8 in 2021-07-31T02:46:18.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:17 smithi045 conmon[39718]: audit 2021-07-31T02:46:18.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:17 smithi045 conmon[39718]: 2021-07-31T02:46:16.732407+0000 mon.c (mon. 2021-07-31T02:46:18.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:17 smithi045 conmon[39718]: 1) 135 : audit [INF] from='client.? 172.21.15.45:0/530405139' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/83/039f89bd-1df4-43d1-8eae-a25ba27cc433"}]: dispatch 2021-07-31T02:46:18.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:17 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T02:46:18.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:17 smithi045 conmon[39718]: 02:46:16.732811+0000 mon.a (mon.0) 3179 : audit [INF] 2021-07-31T02:46:18.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:17 smithi045 conmon[39718]: from='client.? ' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/83/039f89bd-1df4-43d1-8eae-a25ba27cc433"}]: dispatch 2021-07-31T02:46:18.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:17 smithi045 conmon[39718]: audit 2021-07-31T02:46:18.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:17 smithi045 conmon[39718]: 2021-07-31T02:46:16.736035+0000 mon.a (mon.0) 3180 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/83/039f89bd-1df4-43d1-8eae-a25ba27cc433"}]': finished 2021-07-31T02:46:18.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:17 smithi045 conmon[39718]: audit 2021-07-31T02:46:16.766976+0000 mon.c (mon. 2021-07-31T02:46:18.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:17 smithi045 conmon[39718]: 1) 136 : audit [INF] from='client.? 172.21.15.45:0/530405139' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/83/83389db2-9ce3-4663-be48-292f0573742d"}]: dispatch 2021-07-31T02:46:18.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:17 smithi045 conmon[39718]: audit 2021-07 2021-07-31T02:46:18.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:17 smithi045 conmon[39718]: -31T02:46:16.767289+0000 mon.a (mon.0) 3181 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/83/83389db2-9ce3-4663-be48-292f0573742d"}]: dispatch 2021-07-31T02:46:18.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:17 smithi045 conmon[39718]: audit 2021-07-31T02:46:16.770230+0000 mon.a (mon.0) 3182 2021-07-31T02:46:18.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:17 smithi045 conmon[39718]: : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/83/83389db2-9ce3-4663-be48-292f0573742d"}]': finished 2021-07-31T02:46:18.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:17 smithi045 conmon[39718]: cluster 2021-07- 2021-07-31T02:46:18.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:17 smithi045 conmon[39718]: 31T02:46:17.632461+0000 mon.a (mon.0) 3183 : cluster [DBG] osdmap e2151: 8 total, 8 up, 8 in 2021-07-31T02:46:18.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:17 smithi045 conmon[33655]: cluster 2021-07-31T02:46:16.644673+0000 mgr.x 2021-07-31T02:46:18.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:17 smithi045 conmon[33655]: (mgr.24542) 2529 : cluster [DBG] pgmap v3767: 41 pgs: 41 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 298 KiB/s rd, 22 KiB/s wr, 400 op/s 2021-07-31T02:46:18.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:17 smithi045 conmon[33655]: cluster 2021-07-31T02:46:16.704985+0000 mon.a (mon.0) 3178 : cluster [DBG] 2021-07-31T02:46:18.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:17 smithi045 conmon[33655]: osdmap e2150: 8 total, 8 up, 8 in 2021-07-31T02:46:18.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:17 smithi045 conmon[33655]: audit 2021-07 2021-07-31T02:46:18.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:17 smithi045 conmon[33655]: -31T02:46:16.732407+0000 mon.c (mon 2021-07-31T02:46:18.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:17 smithi045 conmon[33655]: .1) 135 : audit [INF] from='client.? 172.21.15.45:0/530405139' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/83/039f89bd-1df4-43d1-8eae-a25ba27cc433"}]: dispatch 2021-07-31T02:46:18.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:17 smithi045 conmon[33655]: audit 2021-07 2021-07-31T02:46:18.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:17 smithi045 conmon[33655]: -31T02:46:16.732811+0000 mon.a (mon 2021-07-31T02:46:18.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:17 smithi045 conmon[33655]: .0) 3179 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/83/039f89bd-1df4-43d1-8eae-a25ba27cc433"}]: dispatch 2021-07-31T02:46:18.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:17 smithi045 conmon[33655]: audit 2021-07-31T02:46:18.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:17 smithi045 conmon[33655]: 2021-07-31T02:46:16. 2021-07-31T02:46:18.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:17 smithi045 conmon[33655]: 736035+0000 mon.a (mon.0) 3180 : 2021-07-31T02:46:18.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:17 smithi045 conmon[33655]: audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/83/039f89bd-1df4-43d1-8eae-a25ba27cc433"}]': finished 2021-07-31T02:46:18.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:17 smithi045 conmon[33655]: audit 2021-07-31T02:46:18.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:17 smithi045 conmon[33655]: 2021-07-31T02:46: 2021-07-31T02:46:18.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:17 smithi045 conmon[33655]: 16.766976+0000 mon.c (mon.1) 136 2021-07-31T02:46:18.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:17 smithi045 conmon[33655]: : audit [INF] from='client.? 172.21.15.45:0/530405139' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/83/83389db2-9ce3-4663-be48-292f0573742d"}]: dispatch 2021-07-31T02:46:18.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:17 smithi045 conmon[33655]: audit 2021-07-31T02:46:18.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:17 smithi045 conmon[33655]: 2021-07-31T02:46: 2021-07-31T02:46:18.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:17 smithi045 conmon[33655]: 16.767289+0000 mon.a (mon.0) 2021-07-31T02:46:18.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:17 smithi045 conmon[33655]: 3181 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/83/83389db2-9ce3-4663-be48-292f0573742d"}]: dispatch 2021-07-31T02:46:18.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:17 smithi045 conmon[33655]: audit 2021-07- 2021-07-31T02:46:18.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:17 smithi045 conmon[33655]: 31T02:46:16.770230+0000 mon.a (mon 2021-07-31T02:46:18.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:17 smithi045 conmon[33655]: .0) 3182 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/83/83389db2-9ce3-4663-be48-292f0573742d"}]': finished 2021-07-31T02:46:18.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:17 smithi045 conmon[33655]: 2021-07-31T02:46:18.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:17 smithi045 conmon[33655]: cluster 2021-07-31T02:46:18.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:17 smithi045 conmon[33655]: 2021-07-31T02:46: 2021-07-31T02:46:18.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:17 smithi045 conmon[33655]: 17.632461+0000 mon.a (mon.0) 3183 2021-07-31T02:46:18.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:17 smithi045 conmon[33655]: : cluster [DBG] osdmap e2151: 8 total, 8 up, 8 in 2021-07-31T02:46:20.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:19 smithi154 conmon[34553]: cluster 2021-07-31T02:46:18.640426+0000 mon.a (mon.0) 3184 : 2021-07-31T02:46:20.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:19 smithi154 conmon[34553]: cluster [DBG] osdmap e2152: 8 total, 8 up, 8 in 2021-07-31T02:46:20.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:19 smithi154 conmon[34553]: cluster 2021-07-31T02:46:18.645120+0000 mgr.x (mgr.24542) 2530 : cluster [DBG] pgmap v3771: 41 pgs: 41 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:46:20.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:19 smithi045 conmon[33655]: cluster 2021-07-31T02:46:18.640426+0000 mon.a (mon.0) 3184 2021-07-31T02:46:20.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:19 smithi045 conmon[33655]: : cluster [DBG] osdmap e2152: 8 total, 8 up, 8 in 2021-07-31T02:46:20.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:19 smithi045 conmon[33655]: cluster 2021-07-31T02:46:18.645120+0000 mgr.x (mgr.24542) 2530 : cluster [DBG] pgmap v3771: 41 pgs: 41 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:46:20.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:19 smithi045 conmon[39718]: cluster 2021-07-31T02:46:18.640426+0000 mon.a 2021-07-31T02:46:20.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:19 smithi045 conmon[39718]: (mon.0) 3184 : cluster [DBG] osdmap e2152: 8 total, 8 up, 8 in 2021-07-31T02:46:20.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:19 smithi045 conmon[39718]: cluster 2021-07-31T02:46:18. 2021-07-31T02:46:20.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:19 smithi045 conmon[39718]: 645120+0000 mgr.x (mgr.24542) 2530 : cluster [DBG] pgmap v3771: 41 pgs: 41 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:46:21.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:20 smithi154 conmon[34553]: cluster 2021-07-31T02:46:19.644701+0000 mon.a (mon.0) 3185 : cluster [DBG] osdmap e2153: 8 total, 8 up, 8 in 2021-07-31T02:46:21.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:20 smithi045 conmon[39718]: cluster 2021-07-31T02:46:19.644701+0000 mon.a (mon.0) 3185 : cluster 2021-07-31T02:46:21.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:20 smithi045 conmon[39718]: [DBG] osdmap e2153: 8 total, 8 up, 8 in 2021-07-31T02:46:21.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:20 smithi045 conmon[33655]: cluster 2021-07-31T02:46:19.644701+0000 mon.a (mon.0) 3185 : cluster [DBG] osdmap e2153: 8 total, 8 up, 8 in 2021-07-31T02:46:21.710 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestMirroring.test_mirror_image_create_snapshot ... ok 2021-07-31T02:46:21.803 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestMirroring.test_mirror_image_status ... SKIP 2021-07-31T02:46:21.917 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestMirroring.test_mirror_peer ... ok 2021-07-31T02:46:21.953 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestMirroring.test_mirror_peer_bootstrap ... 2021-07-31T02:46:21.952+0000 7fda4604db80 -1 auth: unable to find a keyring on /etc/ceph/.client.rbd-mirror-peer.keyring,/etc/ceph/.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory 2021-07-31T02:46:21.954 INFO:tasks.workunit.client.0.smithi045.stderr:2021-07-31T02:46:21.952+0000 7fda4604db80 -1 auth: unable to find a keyring on /etc/ceph/.client.rbd-mirror-peer.keyring,/etc/ceph/.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory 2021-07-31T02:46:21.954 INFO:tasks.workunit.client.0.smithi045.stderr:2021-07-31T02:46:21.952+0000 7fda4604db80 -1 auth: unable to find a keyring on /etc/ceph/.client.rbd-mirror-peer.keyring,/etc/ceph/.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory 2021-07-31T02:46:22.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:21 smithi154 conmon[34553]: cluster 2021-07-31T02:46:20.645543+0000 mgr.x (mgr.24542) 2531 : cluster [DBG] pgmap v3773: 41 pgs: 41 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 122 KiB/s rd, 7.5 KiB/s wr, 158 op/s 2021-07-31T02:46:22.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:21 smithi154 conmon[34553]: cluster 2021-07-31T02:46:20.650944+0000 mon.a (mon.0) 3186 : cluster [DBG] osdmap e2154: 8 total, 8 up, 8 in 2021-07-31T02:46:22.024 INFO:tasks.workunit.client.0.smithi045.stderr:ok 2021-07-31T02:46:22.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:21 smithi045 conmon[39718]: cluster 2021-07-31T02:46:20.645543+0000 mgr.x (mgr.24542) 2021-07-31T02:46:22.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:21 smithi045 conmon[39718]: 2531 : cluster [DBG] pgmap v3773: 41 pgs: 41 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 122 KiB/s rd, 7.5 KiB/s wr, 158 op/s 2021-07-31T02:46:22.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:21 smithi045 conmon[39718]: cluster 2021-07-31T02:46:20.650944+0000 mon.a (mon.0) 3186 : cluster [DBG] 2021-07-31T02:46:22.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:21 smithi045 conmon[39718]: osdmap e2154: 8 total, 8 up, 8 in 2021-07-31T02:46:22.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:21 smithi045 conmon[33655]: cluster 2021-07-31T02:46:20.645543+0000 mgr.x (mgr.24542) 2021-07-31T02:46:22.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:21 smithi045 conmon[33655]: 2531 : cluster [DBG] pgmap v3773: 41 pgs: 41 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 122 KiB/s rd, 7.5 KiB/s wr, 158 op/s 2021-07-31T02:46:22.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:21 smithi045 conmon[33655]: cluster 2021-07-31T02:46:20.650944+0000 mon.a (mon.0) 3186 : cluster [DBG] osdmap e2154: 8 total, 8 up, 8 in 2021-07-31T02:46:22.127 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestMirroring.test_site_name ... ok 2021-07-31T02:46:22.215 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestMirroring.test_uuid ... ok 2021-07-31T02:46:22.289 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestTrash.test_get ... ok 2021-07-31T02:46:22.431 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestTrash.test_list ... ok 2021-07-31T02:46:22.507 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestTrash.test_move ... ok 2021-07-31T02:46:22.652 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestTrash.test_purge ... ok 2021-07-31T02:46:22.737 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestTrash.test_remove ... ok 2021-07-31T02:46:22.807 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestTrash.test_remove_denied ... ok 2021-07-31T02:46:22.880 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestTrash.test_remove_with_progress ... ok 2021-07-31T02:46:22.996 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.TestTrash.test_restore ... ok 2021-07-31T02:46:22.997 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.test_features_from_string ... ok 2021-07-31T02:46:22.997 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.test_features_to_string ... ok 2021-07-31T02:46:23.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:22 smithi154 conmon[34553]: cluster 2021-07-31T02:46:21.656487+0000 mon.a (mon.0 2021-07-31T02:46:23.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:22 smithi154 conmon[34553]: ) 3187 : cluster [DBG] osdmap e2155: 8 total, 8 up, 8 in 2021-07-31T02:46:23.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:22 smithi154 conmon[34553]: audit 2021-07-31T02:46:21.837467+0000 mon.a (mon.0) 3188 : audit [INF] from='client.? ' entity='client.admin' 2021-07-31T02:46:23.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:22 smithi154 conmon[34553]: audit 2021-07-31T02:46:23.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:22 smithi154 conmon[34553]: 2021-07-31T02:46:21.837937+0000 mon.c (mon.1) 137 : audit [DBG] from='client.? 172.21.15.45:0/530405139' entity='client.admin' cmd=[{"prefix": "config-key get", "key": "rbd/mirror/peer/83/d85674fc-9482-4454-8d92-488bc4f224b4"}]: dispatch 2021-07-31T02:46:23.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:22 smithi154 conmon[34553]: 2021-07-31T02:46:23.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:22 smithi154 conmon[34553]: audit 2021-07-31T02:46:23.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:22 smithi154 conmon[34553]: 2021-07-31T02:46:21.838283+0000 2021-07-31T02:46:23.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:22 smithi154 conmon[34553]: mon.c (mon.1) 138 : audit [INF] 2021-07-31T02:46:23.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:22 smithi154 conmon[34553]: from='client.? 172.21.15.45:0/530405139' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/83/d85674fc-9482-4454-8d92-488bc4f224b4"}]: dispatch 2021-07-31T02:46:23.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:22 smithi154 conmon[34553]: audit 2021-07-31T02: 2021-07-31T02:46:23.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:22 smithi154 conmon[34553]: 46:21.838500+0000 mon.a (mon.0) 2021-07-31T02:46:23.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:22 smithi154 conmon[34553]: 3189 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/83/d85674fc-9482-4454-8d92-488bc4f224b4"}]: dispatch 2021-07-31T02:46:23.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:22 smithi154 conmon[34553]: audit 2021-07-31T02:46:23.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:22 smithi154 conmon[34553]: 2021-07-31 2021-07-31T02:46:23.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:22 smithi154 conmon[34553]: T02:46:21.840670+0000 mon.a ( 2021-07-31T02:46:23.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:22 smithi154 conmon[34553]: mon.0) 3190 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/83/d85674fc-9482-4454-8d92-488bc4f224b4"}]': finished 2021-07-31T02:46:23.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:22 smithi154 conmon[34553]: audit 2021-07- 2021-07-31T02:46:23.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:22 smithi154 conmon[34553]: 31T02:46:21.948416+0000 mon.c (mon 2021-07-31T02:46:23.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:22 smithi154 conmon[34553]: .1) 139 : audit [DBG] from='client.? 172.21.15.45:0/530405139' entity='client.admin' cmd=[{"prefix": "config-key get", "key": "rbd/mirror/peer_client_id"}]: dispatch 2021-07-31T02:46:23.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:22 smithi154 conmon[34553]: audit 2021 2021-07-31T02:46:23.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:22 smithi154 conmon[34553]: -07-31T02:46:21.948749 2021-07-31T02:46:23.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:22 smithi154 conmon[34553]: +0000 mon.c (mon.1) 140 : audit 2021-07-31T02:46:23.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:22 smithi154 conmon[34553]: [INF] from='client.? 172.21.15.45:0/530405139' entity='client.admin' cmd=[{ "prefix": "auth get-or-create", "entity": "client.rbd-mirror-peer", "caps": [ "mon", "profile rbd-mirror-peer", "osd", "profile rbd"], "format": "json"}]: dispatch 2021-07-31T02:46:23.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:22 smithi154 conmon[34553]: audit 2021-07-31T02 2021-07-31T02:46:23.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:22 smithi154 conmon[34553]: :46:21.949022+0000 mon.a (mon.0 2021-07-31T02:46:23.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:22 smithi154 conmon[34553]: ) 3191 : audit [INF] from='client.? ' entity='client.admin' cmd=[{ "prefix": "auth get-or-create", "entity": "client.rbd-mirror-peer", "caps": [ "mon", "profile rbd-mirror-peer", "osd", "profile rbd"], "format": "json"}]: dispatch 2021-07-31T02:46:23.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:22 smithi154 conmon[34553]: audit 2021-07-31T02:46:23.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:22 smithi154 conmon[34553]: 2021-07-31T02 2021-07-31T02:46:23.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:22 smithi154 conmon[34553]: :46:21.949667+0000 mon.c (mon. 2021-07-31T02:46:23.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:22 smithi154 conmon[34553]: 1) 141 : audit [DBG] from='client.? 172.21.15.45:0/530405139' entity='client.admin' cmd=[{"prefix": "config-key get", "key": "rbd/mirror/site_name"}]: dispatch 2021-07-31T02:46:23.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:22 smithi154 conmon[34553]: audit 2021-07-31T02:46:23.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:22 smithi154 conmon[34553]: 2021-07-31T02: 2021-07-31T02:46:23.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:22 smithi154 conmon[34553]: 46:22.049420+0000 mon.a (mon.0 2021-07-31T02:46:23.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:22 smithi154 conmon[34553]: ) 3192 : audit [INF] from='client.? ' entity='client.admin' 2021-07-31T02:46:23.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:22 smithi154 conmon[34553]: audit 2021-07-31T02:46:23.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:22 smithi154 conmon[34553]: 2021-07-31T02:46:22.049879+0000 mon.c 2021-07-31T02:46:23.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:22 smithi154 conmon[34553]: (mon.1) 142 : audit [DBG] from='client.? 172.21.15.45:0/530405139' entity='client.admin' cmd=[{"prefix": "config-key get", "key": "rbd/mirror/site_name"}]: dispatch 2021-07-31T02:46:23.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:22 smithi154 conmon[34553]: audit 2021-07-31T02:46:23.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:22 smithi154 conmon[34553]: 2021-07-31T02:46:22. 2021-07-31T02:46:23.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:22 smithi154 conmon[34553]: 050184+0000 mon.c (mon.1) 143 : audit 2021-07-31T02:46:23.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:22 smithi154 conmon[34553]: [INF] from='client.? 172.21.15.45:0/530405139' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/site_name"}]: dispatch 2021-07-31T02:46:23.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:22 smithi154 conmon[34553]: audit 2021-07-31T02:46:22.050427 2021-07-31T02:46:23.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:22 smithi154 conmon[34553]: +0000 mon.a (mon.0) 3193 : audit 2021-07-31T02:46:23.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:22 smithi154 conmon[34553]: [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/site_name"}]: dispatch 2021-07-31T02:46:23.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:22 smithi154 conmon[34553]: audit 2021 2021-07-31T02:46:23.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:22 smithi154 conmon[34553]: -07-31T02:46:22.052625 2021-07-31T02:46:23.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:22 smithi154 conmon[34553]: +0000 mon.a (mon.0) 3194 : audit 2021-07-31T02:46:23.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:22 smithi154 conmon[34553]: [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/site_name"}]': finished 2021-07-31T02:46:23.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:22 smithi154 conmon[34553]: audit 2021-07 2021-07-31T02:46:23.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:22 smithi154 conmon[34553]: -31T02:46:22.053227+0000 2021-07-31T02:46:23.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:22 smithi154 conmon[34553]: mon.c (mon.1) 144 : audit [DBG] 2021-07-31T02:46:23.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:22 smithi154 conmon[34553]: from='client.? 172.21.15.45:0/530405139' entity='client.admin' cmd=[{"prefix": "config-key get", "key": "rbd/mirror/site_name"}]: dispatch 2021-07-31T02:46:23.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:22 smithi154 conmon[34553]: cluster 2021- 2021-07-31T02:46:23.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:22 smithi154 conmon[34553]: 07-31T02:46:22.632114+0000 2021-07-31T02:46:23.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:22 smithi154 conmon[34553]: mon.a (mon.0) 3195 : cluster [DBG] 2021-07-31T02:46:23.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:22 smithi154 conmon[34553]: osdmap e2156: 8 total, 8 up, 8 in 2021-07-31T02:46:23.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[39718]: cluster 2021-07-31T02:46:21.656487+0000 2021-07-31T02:46:23.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[39718]: mon.a (mon.0) 3187 : cluster [DBG] osdmap e2155: 8 total, 8 up, 8 in 2021-07-31T02:46:23.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[39718]: audit 2021- 2021-07-31T02:46:23.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[39718]: 07-31T02:46:21.837467+0000 mon.a (mon.0) 3188 : audit [INF] from='client.? ' entity='client.admin' 2021-07-31T02:46:23.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[39718]: 2021-07-31T02:46:23.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[39718]: audit 2021-07-31T02:46:21 2021-07-31T02:46:23.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[39718]: .837937+0000 mon.c (mon.1) 137 : audit [DBG] from='client.? 172.21.15.45:0/530405139' entity='client.admin' cmd=[{"prefix": "config-key get", "key": "rbd/mirror/peer/83/d85674fc-9482-4454-8d92-488bc4f224b4"}]: dispatch 2021-07-31T02:46:23.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T02:46:23.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[39718]: T02:46:21.838283+0000 mon.c (mon.1) 138 : 2021-07-31T02:46:23.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[39718]: audit [INF] from='client.? 172.21.15.45:0/530405139' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/83/d85674fc-9482-4454-8d92-488bc4f224b4"}]: dispatch 2021-07-31T02:46:23.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[39718]: audit 2021-07-31T02:46:23.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[39718]: 2021-07-31T02:46:21.838500+0000 mon.a (mon.0 2021-07-31T02:46:23.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[39718]: ) 3189 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/83/d85674fc-9482-4454-8d92-488bc4f224b4"}]: dispatch 2021-07-31T02:46:23.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[39718]: audit 2021-07-31T02:46:21. 2021-07-31T02:46:23.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[39718]: 840670+0000 mon.a (mon.0) 3190 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/83/d85674fc-9482-4454-8d92-488bc4f224b4"}]': finished 2021-07-31T02:46:23.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T02:46:23.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[39718]: 31T02:46:21.948416+0000 mon.c (mon.1 2021-07-31T02:46:23.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[39718]: ) 139 : audit [DBG] from='client.? 172.21.15.45:0/530405139' entity='client.admin' cmd=[{"prefix": "config-key get", "key": "rbd/mirror/peer_client_id"}]: dispatch 2021-07-31T02:46:23.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[39718]: audit 2021-07-31T02:46: 2021-07-31T02:46:23.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[39718]: 21.948749+0000 mon.c (mon.1) 140 : audit [INF] from='client.? 172.21.15.45:0/530405139' entity='client.admin' cmd=[{ "prefix": "auth get-or-create", "entity": "client.rbd-mirror-peer", "caps": [ "mon", "profile rbd-mirror-peer", "osd", "profile rbd"], "format": "json"}]: dispatch 2021-07-31T02:46:23.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T02:46:23.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[39718]: 31T02:46:21.949022+0000 mon.a (mon.0) 2021-07-31T02:46:23.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[39718]: 3191 : audit [INF] from='client.? ' entity='client.admin' cmd=[{ "prefix": "auth get-or-create", "entity": "client.rbd-mirror-peer", "caps": [ "mon", "profile rbd-mirror-peer", "osd", "profile rbd"], "format": "json"}]: dispatch 2021-07-31T02:46:23.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[39718]: audit 2021-07-31T02:46:21 2021-07-31T02:46:23.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[39718]: .949667+0000 mon.c (mon.1) 141 : audit [DBG] from='client.? 172.21.15.45:0/530405139' entity='client.admin' cmd=[{"prefix": "config-key get", "key": "rbd/mirror/site_name"}]: dispatch 2021-07-31T02:46:23.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[39718]: audit 2021-07-31T02:46:23.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[39718]: 2021-07 2021-07-31T02:46:23.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[39718]: -31T02:46:22.049420+0000 2021-07-31T02:46:23.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[39718]: mon.a (mon.0) 3192 : audit 2021-07-31T02:46:23.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[39718]: [INF] from='client.? ' entity='client.admin' 2021-07-31T02:46:23.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[39718]: audit 2021 2021-07-31T02:46:23.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[39718]: -07-31T02: 2021-07-31T02:46:23.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[39718]: 46:22.049879+0000 mon.c (mon. 2021-07-31T02:46:23.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[39718]: 1) 142 : audit [DBG] from='client.? 172.21.15.45:0/530405139' entity='client.admin' cmd=[{"prefix": "config-key get", "key": "rbd/mirror/site_name"}]: dispatch 2021-07-31T02:46:23.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[39718]: audit 2021-07-31T02:46:23.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[39718]: 2021-07-31T 2021-07-31T02:46:23.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[39718]: 02:46:22.050184+0000 mon.c ( 2021-07-31T02:46:23.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[33655]: cluster 2021-07-31T02:46:21 2021-07-31T02:46:23.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[33655]: .656487+0000 mon.a (mon.0) 3187 : cluster [DBG] osdmap e2155: 8 total, 8 up, 8 in 2021-07-31T02:46:23.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[33655]: audit 2021- 2021-07-31T02:46:23.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[33655]: 07-31T02:46:21.837467+0000 mon.a ( 2021-07-31T02:46:23.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[33655]: mon.0) 3188 : audit [INF] 2021-07-31T02:46:23.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[33655]: from='client.? ' entity='client.admin' 2021-07-31T02:46:23.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[33655]: audit 2021-07 2021-07-31T02:46:23.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[33655]: -31T02:46:21.837937 2021-07-31T02:46:23.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[33655]: +0000 mon.c (mon.1) 2021-07-31T02:46:23.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[33655]: 137 : audit [DBG] from='client.? 172.21.15.45:0/530405139' entity='client.admin' cmd=[{"prefix": "config-key get", "key": "rbd/mirror/peer/83/d85674fc-9482-4454-8d92-488bc4f224b4"}]: dispatch 2021-07-31T02:46:23.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[33655]: audit 2021- 2021-07-31T02:46:23.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[33655]: 07-31T02:46:21 2021-07-31T02:46:23.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[33655]: .838283+0000 mon.c (mon. 2021-07-31T02:46:23.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[33655]: 1) 138 : audit [INF] 2021-07-31T02:46:23.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[33655]: from='client.? 172.21.15.45:0/530405139' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/83/d85674fc-9482-4454-8d92-488bc4f224b4"}]: dispatch 2021-07-31T02:46:23.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[33655]: audit 2021-07-31T02:46:23.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[33655]: 2021- 2021-07-31T02:46:23.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[33655]: 07-31T02:46: 2021-07-31T02:46:23.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[33655]: 21.838500+0000 mon.a (mon 2021-07-31T02:46:23.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[33655]: .0) 3189 : audit [INF] 2021-07-31T02:46:23.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[33655]: from='client.? ' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/83/d85674fc-9482-4454-8d92-488bc4f224b4"}]: dispatch 2021-07-31T02:46:23.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[33655]: audit 2021-07-31T02:46:23.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[33655]: 2021-07-31T02 2021-07-31T02:46:23.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[33655]: :46:21.840670+0000 2021-07-31T02:46:23.040 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[33655]: mon.a (mon.0) 3190 2021-07-31T02:46:23.040 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[33655]: : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/83/d85674fc-9482-4454-8d92-488bc4f224b4"}]': finished 2021-07-31T02:46:23.040 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[33655]: audit 2021-07-31T02:46:23.040 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[33655]: 2021-07 2021-07-31T02:46:23.041 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[33655]: -31T02:46:21 2021-07-31T02:46:23.041 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[33655]: .948416+0000 mon.c (mon 2021-07-31T02:46:23.041 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[33655]: .1) 139 : audit [DBG] 2021-07-31T02:46:23.041 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[33655]: from='client.? 172.21.15.45:0/530405139' entity='client.admin' cmd=[{"prefix": "config-key get", "key": "rbd/mirror/peer_client_id"}]: dispatch 2021-07-31T02:46:23.041 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[33655]: audit 2021-07-31T02:46:23.042 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[33655]: 2021-07 2021-07-31T02:46:23.042 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[33655]: -31T02:46:21 2021-07-31T02:46:23.042 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[33655]: .948749+0000 mon.c (mon. 2021-07-31T02:46:23.042 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[33655]: 1) 140 : audit [INF] 2021-07-31T02:46:23.043 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[33655]: from='client.? 172.21.15.45:0/530405139' entity='client.admin' cmd=[{ "prefix": "auth get-or-create", "entity": "client.rbd-mirror-peer", "caps": [ "mon", "profile rbd-mirror-peer", "osd", "profile rbd"], "format": "json"}]: dispatch 2021-07-31T02:46:23.043 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[33655]: audit 2021-07-31T02:46:23.043 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[33655]: 2021-07 2021-07-31T02:46:23.043 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[33655]: -31T02:46: 2021-07-31T02:46:23.043 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[33655]: 21.949022+0000 mon.a (mon 2021-07-31T02:46:23.044 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[33655]: .0) 3191 : audit [INF] 2021-07-31T02:46:23.044 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[33655]: from='client.? ' entity='client.admin' cmd=[{ "prefix": "auth get-or-create", "entity": "client.rbd-mirror-peer", "caps": [ "mon", "profile rbd-mirror-peer", "osd", "profile rbd"], "format": "json"}]: dispatch 2021-07-31T02:46:23.044 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[33655]: audit 2021-07-31T02:46:23.044 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[33655]: 2021-07-31T02 2021-07-31T02:46:23.045 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[33655]: :46:21.949667+0000 2021-07-31T02:46:23.045 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[33655]: mon.c (mon.1) 141 : 2021-07-31T02:46:23.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[39718]: mon.1) 143 : audit [INF] from='client.? 172.21.15.45:0/530405139' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/site_name"}]: dispatch 2021-07-31T02:46:23.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[39718]: 2021-07-31T02:46:23.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[39718]: audit 2021-07-31T02:46:23.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[39718]: 2021-07-31T02 2021-07-31T02:46:23.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[39718]: :46:22.050427+0000 mon.a ( 2021-07-31T02:46:23.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[39718]: mon.0) 3193 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/site_name"}]: dispatch 2021-07-31T02:46:23.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[39718]: 2021-07-31T02:46:23.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T02:46:23.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[39718]: T02:46:22.052625 2021-07-31T02:46:23.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[39718]: +0000 mon.a (mon.0) 3194 : audit 2021-07-31T02:46:23.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[39718]: [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/site_name"}]': finished 2021-07-31T02:46:23.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[39718]: audit 2021 2021-07-31T02:46:23.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[39718]: -07-31T02: 2021-07-31T02:46:23.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[39718]: 46:22.053227+0000 mon.c (mon. 2021-07-31T02:46:23.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[39718]: 1) 144 : audit [DBG] from='client.? 172.21.15.45:0/530405139' entity='client.admin' cmd=[{"prefix": "config-key get", "key": "rbd/mirror/site_name"}]: dispatch 2021-07-31T02:46:23.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[39718]: cluster 2021-07-31T02:46:23.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[39718]: 2021-07-31T02 2021-07-31T02:46:23.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[39718]: :46:22.632114+0000 mon.a ( 2021-07-31T02:46:23.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[39718]: mon.0) 3195 : cluster [DBG] osdmap e2156: 8 total, 8 up, 8 in 2021-07-31T02:46:23.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[39718]: 2021-07-31T02:46:23.050 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[33655]: audit [DBG] from='client.? 172.21.15.45:0/530405139' entity='client.admin' cmd=[{"prefix": "config-key get", "key": "rbd/mirror/site_name"}]: dispatch 2021-07-31T02:46:23.050 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[33655]: audit 2021-07-31T02:46:23.050 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[33655]: 2021-07-31T02 2021-07-31T02:46:23.051 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[33655]: :46:22.049420+0000 mon.a (mon 2021-07-31T02:46:23.051 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[33655]: .0) 3192 : audit [INF] from='client.? ' entity='client.admin' 2021-07-31T02:46:23.051 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[33655]: audit 2021-07- 2021-07-31T02:46:23.051 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[33655]: 31T02:46:22.049879+0000 mon.c ( 2021-07-31T02:46:23.051 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[33655]: mon.1) 142 : audit [DBG] from='client.? 172.21.15.45:0/530405139' entity='client.admin' cmd=[{"prefix": "config-key get", "key": "rbd/mirror/site_name"}]: dispatch 2021-07-31T02:46:23.052 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[33655]: 2021-07-31T02:46:23.052 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[33655]: audit 2021-07-31T02:46:23.052 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[33655]: 2021-07-31T02:46:22.050184+0000 2021-07-31T02:46:23.052 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[33655]: mon.c (mon.1) 143 : audit [INF] 2021-07-31T02:46:23.053 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[33655]: from='client.? 172.21.15.45:0/530405139' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/site_name"}]: dispatch 2021-07-31T02:46:23.053 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[33655]: audit 2021-07-31T02:46:23.053 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[33655]: 2021-07-31T02:46:22.050427+0000 mon.a ( 2021-07-31T02:46:23.053 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[33655]: mon.0) 3193 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/site_name"}]: dispatch 2021-07-31T02:46:23.053 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[33655]: audit 2021-07-31T02:46:23.054 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[33655]: 2021-07 2021-07-31T02:46:23.054 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[33655]: -31T02:46:22.052625+0000 mon.a (mon. 2021-07-31T02:46:23.054 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[33655]: 0) 3194 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/site_name"}]': finished 2021-07-31T02:46:23.054 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[33655]: audit 2021-07-31T02:46:23.054 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[33655]: 2021-07-31T02:46:22.053227 2021-07-31T02:46:23.055 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[33655]: +0000 mon.c (mon.1) 144 : audit [DBG] 2021-07-31T02:46:23.055 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[33655]: from='client.? 172.21.15.45:0/530405139' entity='client.admin' cmd=[{"prefix": "config-key get", "key": "rbd/mirror/site_name"}]: dispatch 2021-07-31T02:46:23.055 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[33655]: cluster 2021-07-31T02:46:23.055 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[33655]: 2021-07-31T02:46:22.632114+0000 mon.a ( 2021-07-31T02:46:23.056 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:22 smithi045 conmon[33655]: mon.0) 3195 : cluster [DBG] osdmap e2156: 8 total, 8 up, 8 in 2021-07-31T02:46:23.098 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.test_open_by_id ... ok 2021-07-31T02:46:24.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:23 smithi154 conmon[34553]: cluster 2021-07-31T02:46:22.645866+0000 2021-07-31T02:46:24.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:23 smithi154 conmon[34553]: mgr.x (mgr.24542) 2532 : cluster [DBG] pgmap v3777: 41 pgs: 41 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 122 KiB/s rd, 7.5 KiB/s wr, 158 op/s 2021-07-31T02:46:24.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:23 smithi045 conmon[33655]: cluster 2021-07-31T02:46:22.645866+0000 mgr.x (mgr.24542 2021-07-31T02:46:24.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:23 smithi045 conmon[33655]: ) 2532 : cluster [DBG] pgmap v3777: 41 pgs: 41 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 122 KiB/s rd, 7.5 KiB/s wr, 158 op/s 2021-07-31T02:46:24.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:23 smithi045 conmon[39718]: cluster 2021-07-31T02:46:22.645866+0000 2021-07-31T02:46:24.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:23 smithi045 conmon[39718]: mgr.x (mgr.24542) 2532 : cluster [DBG] pgmap v3777: 41 pgs: 41 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 122 KiB/s rd, 7.5 KiB/s wr, 158 op/s 2021-07-31T02:46:24.768 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.test_pool_stats ... ok 2021-07-31T02:46:24.768 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.test_version ... ok 2021-07-31T02:46:24.845 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.test_create ... ok 2021-07-31T02:46:25.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:24 smithi154 conmon[34553]: cluster 2021-07-31T02:46:23.663913+0000 mon.a (mon. 2021-07-31T02:46:25.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:24 smithi154 conmon[34553]: 0) 3196 : cluster [DBG] osdmap e2157: 8 total, 8 up, 8 in 2021-07-31T02:46:25.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:24 smithi045 conmon[33655]: cluster 2021-07-31T02:46:23.663913+0000 mon.a (mon.0) 3196 : cluster [DBG] osdmap e2157: 8 total, 8 up, 8 in 2021-07-31T02:46:25.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:24 smithi045 conmon[39718]: cluster 2021-07-31T02:46:23.663913+0000 mon.a (mon.0) 3196 : 2021-07-31T02:46:25.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:24 smithi045 conmon[39718]: cluster [DBG] osdmap e2157: 8 total, 8 up, 8 in 2021-07-31T02:46:25.590 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.test_create_defaults ... ok 2021-07-31T02:46:25.707 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.test_context_manager ... ok 2021-07-31T02:46:26.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:25 smithi154 conmon[34553]: cluster 2021-07-31T02:46:24.646575+0000 mgr.x ( 2021-07-31T02:46:26.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:25 smithi154 conmon[34553]: mgr.24542) 2533 : cluster [DBG] pgmap v3779: 41 pgs: 41 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 602 KiB/s rd, 48 KiB/s wr, 856 op/s 2021-07-31T02:46:26.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:25 smithi154 conmon[34553]: cluster 2021-07-31T02:46:24.670651+0000 mon.a (mon.0) 3197 : cluster [DBG] osdmap e2158: 8 total, 8 up, 8 in 2021-07-31T02:46:26.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:25 smithi045 conmon[39718]: cluster 2021-07-31T02:46:24.646575+0000 mgr.x (mgr. 2021-07-31T02:46:26.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:25 smithi045 conmon[39718]: 24542) 2533 : cluster [DBG] pgmap v3779: 41 pgs: 41 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 602 KiB/s rd, 48 KiB/s wr, 856 op/s 2021-07-31T02:46:26.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:25 smithi045 conmon[39718]: cluster 2021-07-31T02:46:24.670651+0000 mon.a (mon.0) 3197 2021-07-31T02:46:26.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:25 smithi045 conmon[39718]: : cluster [DBG] osdmap e2158: 8 total, 8 up, 8 in 2021-07-31T02:46:26.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:25 smithi045 conmon[33655]: cluster 2021-07-31 2021-07-31T02:46:26.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:25 smithi045 conmon[33655]: T02:46:24.646575+0000 mgr.x (mgr.24542) 2533 : cluster [DBG] pgmap v3779: 41 pgs: 41 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 602 KiB/s rd, 48 KiB/s wr, 856 op/s 2021-07-31T02:46:26.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:25 smithi045 conmon[33655]: cluster 2021 2021-07-31T02:46:26.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:25 smithi045 conmon[33655]: -07-31T02:46:24.670651+0000 mon.a (mon.0) 3197 : cluster [DBG] osdmap e2158: 8 total, 8 up, 8 in 2021-07-31T02:46:27.751 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.test_open_read_only ... ok 2021-07-31T02:46:28.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:27 smithi154 conmon[34553]: cluster 2021-07-31T02:46:26.647106+0000 mgr.x (mgr.24542) 2021-07-31T02:46:28.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:27 smithi154 conmon[34553]: 2534 : cluster [DBG] pgmap v3781: 41 pgs: 41 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 483 KiB/s rd, 39 KiB/s wr, 686 op/s 2021-07-31T02:46:28.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:27 smithi154 conmon[34553]: cluster 2021-07-31T02:46:26.675813+0000 mon.a (mon.0) 3198 : cluster [DBG] osdmap e2159: 8 total, 8 up, 8 in 2021-07-31T02:46:28.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:27 smithi154 conmon[34553]: cluster 2021-07-31T02:46:27.634214+0000 2021-07-31T02:46:28.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:27 smithi154 conmon[34553]: mon.a (mon.0) 3199 : cluster [DBG] osdmap e2160: 8 total, 8 up, 8 in 2021-07-31T02:46:28.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:27 smithi045 conmon[33655]: cluster 2021-07-31T02:46:26.647106+0000 mgr.x (mgr.24542) 2534 : cluster [DBG] pgmap v3781: 41 pgs: 41 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 483 KiB/s rd, 39 KiB/s wr, 686 op/s 2021-07-31T02:46:28.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:27 smithi045 conmon[33655]: cluster 2021-07-31T02:46:26.675813+0000 mon.a (mon.0) 3198 : cluster [DBG] osdmap e2159: 8 total, 8 up, 8 in 2021-07-31T02:46:28.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:27 smithi045 conmon[33655]: cluster 2021-07-31T02:46:27. 2021-07-31T02:46:28.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:27 smithi045 conmon[33655]: 634214+0000 mon.a (mon.0) 3199 : cluster [DBG] osdmap e2160: 8 total, 8 up, 8 in 2021-07-31T02:46:28.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:27 smithi045 conmon[39718]: cluster 2021-07-31T02:46:26.647106+0000 mgr.x (mgr.24542 2021-07-31T02:46:28.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:27 smithi045 conmon[39718]: ) 2534 : cluster [DBG] pgmap v3781: 41 pgs: 41 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 483 KiB/s rd, 39 KiB/s wr, 686 op/s 2021-07-31T02:46:28.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:27 smithi045 conmon[39718]: cluster 2021-07-31T02:46:26.675813+0000 2021-07-31T02:46:28.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:27 smithi045 conmon[39718]: mon.a (mon.0) 3198 : cluster [DBG] osdmap e2159: 8 total, 8 up, 8 in 2021-07-31T02:46:28.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:27 smithi045 conmon[39718]: cluster 2021-07-31T 2021-07-31T02:46:28.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:27 smithi045 conmon[39718]: 02:46:27.634214+0000 mon.a (mon.0) 3199 : cluster [DBG] osdmap e2160: 8 total, 8 up, 8 in 2021-07-31T02:46:28.268 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.test_open_dne ... ok 2021-07-31T02:46:28.774 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.test_open_readonly_dne ... ok 2021-07-31T02:46:28.869 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.test_aio_open ... ok 2021-07-31T02:46:28.882 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.test_remove_dne ... ok 2021-07-31T02:46:28.883 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.test_list_empty ... ok 2021-07-31T02:46:28.965 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.test_list ... ok 2021-07-31T02:46:29.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:28 smithi154 conmon[34553]: audit 2021-07-31T02:46:28.171799+0000 mon.b (mon 2021-07-31T02:46:29.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:28 smithi154 conmon[34553]: .2) 370 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:46:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:28 smithi154 conmon[34553]: audit 2021-07-31T02:46:28.172122+0000 mon.a (mon.0) 3200 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:46:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:28 smithi154 conmon[34553]: audit 2021-07-31T02:46 2021-07-31T02:46:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:28 smithi154 conmon[34553]: :28.174319+0000 mon.b (mon.2) 371 : 2021-07-31T02:46:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:28 smithi154 conmon[34553]: audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:46:29.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:28 smithi154 conmon[34553]: audit 2021-07 2021-07-31T02:46:29.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:28 smithi154 conmon[34553]: -31T02:46:28.174608+0000 mon.a 2021-07-31T02:46:29.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:28 smithi154 conmon[34553]: (mon.0) 3201 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:46:29.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:28 smithi045 conmon[33655]: audit 2021-07-31T02:46:28 2021-07-31T02:46:29.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:28 smithi045 conmon[33655]: .171799+0000 mon.b (mon.2) 370 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:46:29.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:28 smithi045 conmon[33655]: audit 2021-07-31T02:46 2021-07-31T02:46:29.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:28 smithi045 conmon[33655]: :28.172122+0000 mon.a (mon.0) 3200 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:46:29.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:28 smithi045 conmon[33655]: audit 2021-07- 2021-07-31T02:46:29.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:28 smithi045 conmon[33655]: 31T02:46:28.174319+0000 mon.b (mon. 2021-07-31T02:46:29.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:28 smithi045 conmon[33655]: 2) 371 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:46:29.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:28 smithi045 conmon[33655]: audit 2021-07-31T 2021-07-31T02:46:29.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:28 smithi045 conmon[33655]: 02:46:28.174608+0000 mon.a ( 2021-07-31T02:46:29.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:28 smithi045 conmon[33655]: mon.0) 3201 : audit 2021-07-31T02:46:29.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:28 smithi045 conmon[33655]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:46:29.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:28 smithi045 conmon[39718]: audit 2021-07-31T02:46:28 2021-07-31T02:46:29.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:28 smithi045 conmon[39718]: .171799+0000 mon.b (mon.2) 370 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:46:29.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:28 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T02:46:29.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:28 smithi045 conmon[39718]: 02:46:28.172122+0000 2021-07-31T02:46:29.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:28 smithi045 conmon[39718]: mon.a (mon.0) 3200 2021-07-31T02:46:29.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:28 smithi045 conmon[39718]: : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:46:29.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:28 smithi045 conmon[39718]: audit 2021-07-31T02:46:29.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:28 smithi045 conmon[39718]: 2021-07-31T 2021-07-31T02:46:29.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:28 smithi045 conmon[39718]: 02:46:28.174319+0000 2021-07-31T02:46:29.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:28 smithi045 conmon[39718]: mon.b (mon.2) 371 : audit [INF] 2021-07-31T02:46:29.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:28 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:46:29.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:28 smithi045 conmon[39718]: audit 2021-07 2021-07-31T02:46:29.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:28 smithi045 conmon[39718]: -31T02:46:28 2021-07-31T02:46:29.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:28 smithi045 conmon[39718]: .174608+0000 mon.a (mon.0) 2021-07-31T02:46:29.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:28 smithi045 conmon[39718]: 3201 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:46:29.036 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.test_remove_with_progress ... ok 2021-07-31T02:46:29.105 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.test_remove_canceled ... ok 2021-07-31T02:46:29.389 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.test_pool_metadata ... ok 2021-07-31T02:46:29.405 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.test_config_list ... ok 2021-07-31T02:46:29.413 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.test_pool_config_set_and_get_and_remove ... ok 2021-07-31T02:46:29.444 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.test_namespaces ... ok 2021-07-31T02:46:29.451 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.test_create_group ... ok 2021-07-31T02:46:29.459 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.test_rename_group ... ok 2021-07-31T02:46:29.460 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.test_list_groups_empty ... ok 2021-07-31T02:46:29.467 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.test_list_groups ... ok 2021-07-31T02:46:29.473 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.test_list_groups_after_removed ... ok 2021-07-31T02:46:29.753 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: debug 2021-07-31T02:46:29.447+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:46:29.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:46:29.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:46:29.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: yield 2021-07-31T02:46:29.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:46:29.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:46:29.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:46:29.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:46:29.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:46:29.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:46:29.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:46:29.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:46:29.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:46:29.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: task.result() 2021-07-31T02:46:29.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:46:29.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: debug 2021-07-31T02:46:29.459+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:46:29.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:46:29.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:46:29.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: yield 2021-07-31T02:46:29.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:46:29.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:46:29.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:46:29.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:46:29.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:46:29.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:46:29.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:46:29.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:46:29.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:46:29.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: task.result() 2021-07-31T02:46:29.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:46:29.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: debug 2021-07-31T02:46:29.485+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:46:29.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:46:29.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:46:29.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: yield 2021-07-31T02:46:29.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:46:29.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:46:29.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:46:29.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:46:29.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:46:29.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:46:29.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:46:29.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:46:29.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:46:29.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: task.result() 2021-07-31T02:46:29.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:46:29.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: debug 2021-07-31T02:46:29.486+0000 7fcb9816a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:46:29.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:46:29.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:46:29.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: yield 2021-07-31T02:46:29.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:46:29.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:46:29.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:46:29.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:46:29.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:46:29.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:46:29.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:46:29.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:46:29.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:46:29.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: task.result() 2021-07-31T02:46:29.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:46:29.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: 2021-07-31T02:46:29.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:46:29.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: 2021-07-31T02:46:29.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:46:29.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:46:29.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:46:29.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:46:29.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:46:29.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:46:29.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:46:29.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:46:29.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:46:29.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:46:29.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:46:29.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:46:29.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:46:29.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:46:29.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:46:29.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:46:29.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:46:29.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:46:29.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: raise 2021-07-31T02:46:29.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:46:29.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:46:29.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:46:29.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:46:29.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:46:29.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: debug 2021-07-31T02:46:29.494+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:46:29.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:46:29.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:46:29.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: yield 2021-07-31T02:46:29.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:46:29.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:46:29.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:46:29.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:46:29.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:46:29.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:46:29.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:46:29.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:46:29.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:46:29.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: task.result() 2021-07-31T02:46:29.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:46:29.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: debug 2021-07-31T02:46:29.494+0000 7fcb97969700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:46:29.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:46:29.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:46:29.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: yield 2021-07-31T02:46:29.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:46:29.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:46:29.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:46:29.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:46:29.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:46:29.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:46:29.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:46:29.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:46:29.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:46:29.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: task.result() 2021-07-31T02:46:29.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:46:29.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: 2021-07-31T02:46:29.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:46:29.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: 2021-07-31T02:46:29.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:46:29.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:46:29.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:46:29.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:46:29.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:46:29.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:46:29.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:46:29.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:46:29.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:46:29.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:46:29.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:46:29.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:46:29.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:46:29.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:46:29.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:46:29.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:46:29.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:46:29.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:46:29.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: raise 2021-07-31T02:46:29.786 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:46:29.786 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:46:29.786 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:46:29.786 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:46:29.786 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:46:29 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:46:30.412 INFO:tasks.workunit.client.0.smithi045.stderr:test_rbd.test_rename ... ok 2021-07-31T02:46:30.413 INFO:tasks.workunit.client.0.smithi045.stderr: 2021-07-31T02:46:30.413 INFO:tasks.workunit.client.0.smithi045.stderr:---------------------------------------------------------------------- 2021-07-31T02:46:30.413 INFO:tasks.workunit.client.0.smithi045.stderr:Ran 171 tests in 622.196s 2021-07-31T02:46:30.413 INFO:tasks.workunit.client.0.smithi045.stderr: 2021-07-31T02:46:30.414 INFO:tasks.workunit.client.0.smithi045.stderr:OK (SKIP=8) 2021-07-31T02:46:30.445 INFO:tasks.workunit.client.0.smithi045.stderr:+ exit 0 2021-07-31T02:46:30.446 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-07-31T02:46:30.447 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2021-07-31T02:46:30.488 INFO:tasks.workunit:Stopping ['rbd/test_librbd_python.sh'] on client.0... 2021-07-31T02:46:30.489 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2021-07-31T02:46:30.720 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: cluster 2021-07-31T02:46:28. 2021-07-31T02:46:30.722 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: 647471+0000 mgr.x (mgr.24542) 2535 : cluster [DBG] pgmap v3784: 41 pgs: 41 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 483 KiB/s rd, 39 KiB/s wr, 687 op/s 2021-07-31T02:46:30.722 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: audit 2021-07-31T02:46:29 2021-07-31T02:46:30.722 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: .399370+0000 mon.a (mon.0) 3202 : audit 2021-07-31T02:46:30.723 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: [INF] from='client.? ' entity='client.admin' 2021-07-31T02:46:30.723 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: audit 2021-07-31T02: 2021-07-31T02:46:30.723 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: 46:29.402472+0000 mon.b (mon.2) 372 : audit 2021-07-31T02:46:30.723 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:46:30.724 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: audit 2021-07 2021-07-31T02:46:30.724 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: -31T02:46:29.403658+0000 mon.b (mon.2 2021-07-31T02:46:30.724 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: ) 373 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:46:30.724 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: audit 2021- 2021-07-31T02:46:30.724 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: 07-31T02:46:29.404379+0000 2021-07-31T02:46:30.725 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: mon.b (mon.2) 374 : audit [INF] 2021-07-31T02:46:30.725 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:46:30.725 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:46:30.725 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: 2021-07-31T02:46:29.405032+0000 mgr.x 2021-07-31T02:46:30.725 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: (mgr.24542) 2536 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:46:30.726 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:46:30.726 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: 2021-07-31T02:46 2021-07-31T02:46:30.726 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: :29.448423+0000 mgr.x (mgr.24542 2021-07-31T02:46:30.726 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: ) 2537 : cephadm [ERR] 2021-07-31T02:46:30.727 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:46:30.727 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:46:30.727 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: yield 2021-07-31T02:46:30.727 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:46:30.728 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:46:30.728 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:46:30.728 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:46:30.728 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:46:30.728 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:46:30.729 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:46:30.729 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:46:30.729 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:46:30.730 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:46:30.730 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:46:30.730 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:46:29.448744+0000 mgr.x (mgr.24542) 2538 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:46:30.730 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:46:29.460731+0000 mgr.x (mgr.24542) 2539 : cephadm [ERR] 2021-07-31T02:46:30.731 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: 2021-07-31T02:46:30.731 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:46:30.731 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:46:30.731 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: yield 2021-07-31T02:46:30.733 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: cluster 2021-07-31T02:46:28.647471+0000 2021-07-31T02:46:30.733 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: mgr.x (mgr.24542) 2535 : cluster [DBG] pgmap v3784: 41 pgs: 41 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 483 KiB/s rd, 39 KiB/s wr, 687 op/s 2021-07-31T02:46:30.733 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: audit 2021-07-31T02:46:29.399370+0000 mon.a (mon.0) 3202 : audit [INF] from='client.? ' entity='client.admin' 2021-07-31T02:46:30.733 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: audit 2021-07-31T02:46:29.402472+0000 mon.b (mon.2) 372 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:46:30.734 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: audit 2021-07-31T02:46:29.403658+0000 mon.b (mon.2) 373 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:46:30.734 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: audit 2021- 2021-07-31T02:46:30.734 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: 07-31T02:46:29.404379+0000 mon.b (mon.2) 374 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:46:30.734 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:46:29.405032+0000 mgr.x (mgr.24542) 2536 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:46:30.735 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:46:29.448423+0000 mgr.x (mgr.24542) 2537 : cephadm [ERR] 2021-07-31T02:46:30.735 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:46:30.735 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:46:30.735 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: yield 2021-07-31T02:46:30.735 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:46:30.736 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:46:30.736 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:46:30.736 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:46:30.736 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:46:30.737 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:46:30.737 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:46:30.737 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:46:30.738 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:46:30.738 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:46:30.738 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:46:30.739 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: 2021-07-31T02:46:30.739 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:46:29.448744+0000 mgr.x (mgr.24542) 2538 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:46:30.739 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:46:29.460731+0000 mgr.x (mgr.24542) 2539 : cephadm [ERR] 2021-07-31T02:46:30.739 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:46:30.739 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:46:30.740 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: yield 2021-07-31T02:46:30.740 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:46:30.740 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:46:30.740 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:46:30.741 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:46:30.741 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:46:30.742 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:46:30.742 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:46:30.742 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:46:30.742 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:46:30.743 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:46:30.743 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:46:30.743 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:46:30.743 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:46:30.743 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:46:30.744 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:46:30.744 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: cephadm 2021-07-31T 2021-07-31T02:46:30.744 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: 02:46:29.461373+0000 mgr.x (mgr.24542) 2540 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:46:30.744 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:46:29.486408+0000 mgr.x (mgr.24542) 2541 : cephadm [ERR] 2021-07-31T02:46:30.744 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:46:30.745 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:46:30.745 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: yield 2021-07-31T02:46:30.745 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:46:30.745 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:46:30.746 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:46:30.746 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:46:30.746 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:46:30.746 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:46:30.746 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:46:30.747 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:46:30.747 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:46:30.747 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:46:30.747 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:46:30.747 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:46:29.486965+0000 mgr.x (mgr.24542) 2542 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:46:30.748 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:46:30.748 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:46:30.748 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: yield 2021-07-31T02:46:30.748 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:46:30.749 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:46:30.749 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:46:30.749 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:46:30.749 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:46:30.749 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:46:30.750 DEBUG:teuthology.parallel:result is None 2021-07-31T02:46:30.751 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2021-07-31T02:46:30.752 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:46:30.752 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:46:30.752 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:46:30.752 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:46:30.753 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:46:30.753 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:46:30.753 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:46:30.753 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:46: 2021-07-31T02:46:30.753 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: 29.461373+0000 mgr.x (mgr.24542) 2540 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:46:30.754 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:46:29.486408+0000 mgr.x (mgr.24542) 2541 : cephadm [ERR] 2021-07-31T02:46:30.754 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:46:30.754 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:46:30.754 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: yield 2021-07-31T02:46:30.755 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:46:30.755 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:46:30.755 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:46:30.755 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:46:30.755 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:46:30.756 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:46:30.756 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:46:30.756 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:46:30.756 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:46:30.757 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:46:30.757 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:46:30.757 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:46:29.486965+0000 mgr.x (mgr.24542) 2542 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:46:30.757 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:46:30.758 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:46:30.758 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: yield 2021-07-31T02:46:30.758 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:46:30.759 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:46:30.759 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:46:30.759 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:46:30.759 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:46:30.759 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:46:30.760 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:46:30.760 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:46:30.760 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:46:30.760 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:46:30.760 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:46:30.761 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: 2021-07-31T02:46:30.762 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:46:30.762 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:46:30.763 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:46:30.763 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:46:30.763 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:46:30.763 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: 2021-07-31T02:46:30.764 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:46:30.764 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: 2021-07-31T02:46:30.764 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:46:30.764 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:46:30.764 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:46:30.765 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:46:30.765 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:46:30.765 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:46:30.765 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:46:30.765 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:46:30.766 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:46:30.766 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:46:30.766 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:46:30.766 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:46:30.767 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:46:30.767 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:46:30.767 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:46:30.767 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:46:30.767 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:46:30.768 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:46:30.768 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: raise 2021-07-31T02:46:30.768 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:46:30.768 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:46:30.769 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:46:30.769 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:46:30.769 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:46:30.769 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: cephadm 2021-07-31T 2021-07-31T02:46:30.769 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: 02:46:29.495216+0000 mgr.x (mgr.24542) 2543 : cephadm [ERR] 2021-07-31T02:46:30.770 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:46:30.770 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:46:30.770 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: yield 2021-07-31T02:46:30.770 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:46:30.771 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:46:30.771 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:46:30.771 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:46:30.772 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:46:30.772 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: 2021-07-31T02:46:30.772 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:46:30.772 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:46:30.772 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:46:30.773 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:46:30.773 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:46:30.773 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:46:30.773 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:46:30.774 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:46:30.774 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:46:30.774 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:46:30.774 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:46:30.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:46:30.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:46:30.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:46:30.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:46:30.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:46:30.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:46:30.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:46:30.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: raise 2021-07-31T02:46:30.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:46:30.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:46:30.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:46:30.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:46:30.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:46:30.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:46:29.495216+0000 mgr.x (mgr.24542) 2543 : cephadm [ERR] 2021-07-31T02:46:30.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:46:30.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:46:30.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: yield 2021-07-31T02:46:30.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:46:30.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:46:30.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:46:30.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:46:30.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:46:30.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:46:30.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:46:30.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:46:30.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:46:30.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:46:30.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:46:30.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:46:30.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:46:30.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:46:30.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:46:30.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:46:30.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:46:30.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:46:30.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:46:29.495629+0000 mgr.x (mgr.24542) 2544 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:46:30.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:46:30.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:46:30.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: yield 2021-07-31T02:46:30.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:46:30.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:46:30.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:46:30.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:46:30.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:46:30.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:46:30.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:46:30.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:46:30.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:46:30.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:46:30.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:46:30.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: 2021-07-31T02:46:30.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:46:30.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: 2021-07-31T02:46:30.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:46:30.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:46:30.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:46:30.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:46:30.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:46:30.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:46:30.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:46:30.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:46:30.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:46:30.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:46:30.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:46:30.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:46:30.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:46:30.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:46:30.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:46:30.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:46:30.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:46:30.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:46:30.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: raise 2021-07-31T02:46:30.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:46:30.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:46:30.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:46:30.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:46:30.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:46:30.793 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:46:29.495629+0000 mgr.x (mgr.24542) 2544 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:46:30.793 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:46:30.793 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:46:30.794 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: yield 2021-07-31T02:46:30.794 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:46:30.794 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:46:30.794 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:46:30.794 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:46:30.795 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:46:30.795 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:46:30.795 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:46:30.795 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:46:30.796 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:46:30.796 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:46:30.796 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:46:30.796 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: 2021-07-31T02:46:30.796 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:46:30.797 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: 2021-07-31T02:46:30.797 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:46:30.797 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:46:30.797 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:46:30.798 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:46:30.798 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:46:30.798 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:46:30.798 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:46:30.798 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:46:30.799 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:46:30.799 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:46:30.799 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:46:30.799 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:46:30.800 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:46:30.800 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:46:30.800 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:46:30.800 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:46:30.800 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:46:30.801 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:46:30.801 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: raise 2021-07-31T02:46:30.801 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:46:30.801 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:46:30.802 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:46:30.802 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:46:30.802 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:30 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:46:30.803 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: cluster 2021-07-31T02:46:28.647471+0000 mgr.x (mgr.24542) 2021-07-31T02:46:30.804 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: 2535 : cluster [DBG] pgmap v3784: 41 pgs: 41 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 483 KiB/s rd, 39 KiB/s wr, 687 op/s 2021-07-31T02:46:30.804 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: audit 2021-07-31T02:46:29.399370+0000 mon.a (mon.0) 3202 : audit [INF] from='client.? ' entity='client.admin' 2021-07-31T02:46:30.804 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: audit 2021-07-31T02:46:30.804 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: 2021-07-31T02:46:29.402472+0000 mon.b (mon.2) 372 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:46:30.805 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: audit 2021-07-31T02:46:29.403658+0000 mon.b (mon 2021-07-31T02:46:30.805 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: .2) 373 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:46:30.805 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: audit 2021-07-31T02:46:30.805 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: 2021-07-31T02:46:29.404379+0000 mon.b ( 2021-07-31T02:46:30.805 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: mon.2) 374 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:46:30.806 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: cephadm 2021-07-31 2021-07-31T02:46:30.806 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: T02:46:29.405032+0000 mgr.x (mgr.24542) 2536 : cephadm 2021-07-31T02:46:30.806 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:46:30.806 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: cephadm 2021-07-31T02 2021-07-31T02:46:30.807 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: :46:29.448423+0000 mgr.x (mgr.24542) 2537 2021-07-31T02:46:30.807 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: : cephadm [ERR] 2021-07-31T02:46:30.807 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:46:30.807 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:46:30.807 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: yield 2021-07-31T02:46:30.808 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:46:30.808 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:46:30.808 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:46:30.808 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:46:30.808 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:46:30.809 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:46:30.809 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:46:30.809 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:46:30.809 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:46:30.810 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:46:30.810 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:46:30.810 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:46:29.448744+0000 mgr.x (mgr.24542) 2538 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:46:30.810 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:46:29.460731+0000 mgr.x (mgr 2021-07-31T02:46:30.810 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: .24542) 2539 : cephadm [ERR] 2021-07-31T02:46:30.811 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:46:30.811 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:46:30.811 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: yield 2021-07-31T02:46:30.811 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:46:30.812 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:46:30.812 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:46:30.812 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:46:30.812 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:46:30.812 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:46:30.813 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:46:30.813 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:46:30.813 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:46:30.813 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:46:30.814 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:46:30.814 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:46:29.461373+0000 mgr.x (mgr.24542) 2540 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:46:30.814 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:46:29.486408+0000 mgr.x (mgr.24542) 2541 2021-07-31T02:46:30.814 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: : cephadm [ERR] 2021-07-31T02:46:30.814 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:46:30.815 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:46:30.815 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: yield 2021-07-31T02:46:30.815 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:46:30.815 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:46:30.816 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:46:30.816 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:46:30.816 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:46:30.816 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:46:30.816 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:46:30.817 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:46:30.817 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:46:30.817 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:46:30.817 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:46:30.817 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:46:30.818 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: 2021-07-31T02:46:29.486965+0000 mgr.x (mgr.24542) 2542 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:46:30.818 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:46:30.818 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:46:30.818 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: yield 2021-07-31T02:46:30.819 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:46:30.819 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:46:30.819 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:46:30.819 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:46:30.819 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:46:30.820 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:46:30.820 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:46:30.820 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:46:30.820 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:46:30.821 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:46:30.821 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:46:30.821 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: 2021-07-31T02:46:30.821 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:46:30.821 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: 2021-07-31T02:46:30.822 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:46:30.822 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:46:30.822 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:46:30.822 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:46:30.823 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:46:30.823 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:46:30.823 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:46:30.823 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:46:30.823 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:46:30.824 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:46:30.824 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:46:30.824 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:46:30.824 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:46:30.825 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:46:30.825 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:46:30.825 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:46:30.825 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:46:30.825 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:46:30.826 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: raise 2021-07-31T02:46:30.826 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:46:30.826 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:46:30.826 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:46:30.827 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:46:30.827 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:46:30.827 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:46:29.495216+0000 mgr.x (mgr.24542) 2543 : cephadm [ERR] 2021-07-31T02:46:30.827 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:46:30.827 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:46:30.828 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: yield 2021-07-31T02:46:30.828 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:46:30.828 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:46:30.828 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:46:30.829 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:46:30.829 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:46:30.829 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:46:30.829 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:46:30.829 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:46:30.830 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:46:30.830 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:46:30.830 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:46:30.830 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:46:29.495629+0000 mgr.x (mgr.24542) 2544 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:46:30.830 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:46:30.831 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:46:30.831 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: yield 2021-07-31T02:46:30.831 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:46:30.831 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:46:30.832 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:46:30.832 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:46:30.832 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:46:30.832 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:46:30.832 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:46:30.833 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:46:30.833 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:46:30.833 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:46:30.833 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:46:30.833 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: 2021-07-31T02:46:30.834 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:46:30.834 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: 2021-07-31T02:46:30.834 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:46:30.834 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:46:30.835 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:46:30.835 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:46:30.835 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:46:30.835 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:46:30.835 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:46:30.836 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:46:30.836 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:46:30.836 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:46:30.836 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:46:30.837 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:46:30.837 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:46:30.837 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:46:30.837 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:46:30.837 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:46:30.838 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:46:30.838 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:46:30.838 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: raise 2021-07-31T02:46:30.838 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:46:30.838 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:46:30.839 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:46:30.839 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:46:30.839 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:30 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:46:30.841 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2021-07-31T02:46:30.841 DEBUG:teuthology.orchestra.run.smithi045:> rmdir -- /home/ubuntu/cephtest/mnt.0 2021-07-31T02:46:30.896 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2021-07-31T02:46:30.897 INFO:teuthology.task.full_sequential:In full_sequential, running task print... 2021-07-31T02:46:30.897 INFO:teuthology.task.print:**** done end test_rbd_python.yaml 2021-07-31T02:46:30.898 DEBUG:teuthology.parallel:result is None 2021-07-31T02:46:31.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:31 smithi154 conmon[34553]: cluster 2021-07-31T02:46:30.408084+0000 mon.a (mon.0 2021-07-31T02:46:31.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:31 smithi154 conmon[34553]: ) 3203 : cluster [DBG] osdmap e2161: 8 total, 8 up, 8 in 2021-07-31T02:46:31.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:31 smithi045 conmon[33655]: cluster 2021-07-31T02:46:30.408084+0000 mon.a (mon. 2021-07-31T02:46:31.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:31 smithi045 conmon[33655]: 0) 3203 : cluster [DBG] osdmap e2161: 8 total, 8 up, 8 in 2021-07-31T02:46:31.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:31 smithi045 conmon[39718]: cluster 2021-07-31T02:46:30.408084 2021-07-31T02:46:31.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:31 smithi045 conmon[39718]: +0000 mon.a (mon.0) 3203 : cluster [DBG] osdmap e2161: 8 total, 8 up, 8 in 2021-07-31T02:46:32.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:32 smithi154 conmon[34553]: cluster 2021-07-31T02:46:30.647967+0000 mgr.x (mgr.24542) 2545 : cluster 2021-07-31T02:46:32.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:32 smithi154 conmon[34553]: [DBG] pgmap v3786: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:46:32.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:32 smithi045 conmon[33655]: cluster 2021-07-31T02:46:30.647967+0000 mgr.x (mgr.24542) 2545 : cluster [DBG] 2021-07-31T02:46:32.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:32 smithi045 conmon[33655]: pgmap v3786: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:46:32.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:32 smithi045 conmon[39718]: cluster 2021-07-31T02:46:30.647967+0000 mgr.x (mgr. 2021-07-31T02:46:32.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:32 smithi045 conmon[39718]: 24542) 2545 : cluster [DBG] pgmap v3786: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:46:34.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:33 smithi154 conmon[34553]: cluster 2021-07-31T02:46:32.648302+0000 mgr.x (mgr.24542) 2546 : cluster [DBG] 2021-07-31T02:46:34.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:33 smithi154 conmon[34553]: pgmap v3787: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:46:34.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:33 smithi045 conmon[33655]: cluster 2021-07-31T02:46:32.648302+0000 mgr.x (mgr.24542) 2546 : 2021-07-31T02:46:34.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:33 smithi045 conmon[33655]: cluster [DBG] pgmap v3787: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:46:34.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:33 smithi045 conmon[39718]: cluster 2021-07-31T02:46:32.648302+0000 mgr.x (mgr.24542) 2546 : cluster 2021-07-31T02:46:34.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:33 smithi045 conmon[39718]: [DBG] pgmap v3787: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:46:36.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:35 smithi154 conmon[34553]: cluster 2021-07-31T02:46:34.648722+0000 mgr.x (mgr.24542) 2547 : cluster [DBG] 2021-07-31T02:46:36.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:35 smithi154 conmon[34553]: pgmap v3788: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:46:36.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:35 smithi045 conmon[33655]: cluster 2021-07-31T02:46:34.648722+0000 mgr.x (mgr.24542) 2547 : cluster [DBG] 2021-07-31T02:46:36.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:35 smithi045 conmon[33655]: pgmap v3788: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:46:36.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:35 smithi045 conmon[39718]: cluster 2021-07-31T02:46:34.648722+0000 mgr.x (mgr.24542) 2547 : cluster [DBG] 2021-07-31T02:46:36.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:35 smithi045 conmon[39718]: pgmap v3788: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:46:38.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:37 smithi154 conmon[34553]: cluster 2021-07-31T02:46:36.649268+0000 mgr.x (mgr.24542) 2548 : cluster 2021-07-31T02:46:38.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:37 smithi154 conmon[34553]: [DBG] pgmap v3789: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:46:38.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:37 smithi045 conmon[39718]: cluster 2021-07-31T02:46:36.649268+0000 mgr.x (mgr.24542) 2548 : 2021-07-31T02:46:38.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:37 smithi045 conmon[39718]: cluster [DBG] pgmap v3789: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:46:38.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:37 smithi045 conmon[33655]: cluster 2021-07-31T02:46:36.649268+0000 mgr.x (mgr.24542) 2548 : cluster [DBG] pgmap v3789: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:46:40.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:39 smithi154 conmon[34553]: cluster 2021-07-31T02:46:38.649735+0000 mgr.x (mgr.24542) 2549 : cluster 2021-07-31T02:46:40.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:39 smithi154 conmon[34553]: [DBG] pgmap v3790: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:46:40.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:39 smithi045 conmon[33655]: cluster 2021-07-31T02:46: 2021-07-31T02:46:40.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:39 smithi045 conmon[33655]: 38.649735+0000 mgr.x (mgr.24542) 2549 : cluster [DBG] pgmap v3790: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:46:40.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:39 smithi045 conmon[39718]: cluster 2021-07-31T02:46:38.649735+0000 2021-07-31T02:46:40.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:39 smithi045 conmon[39718]: mgr.x (mgr.24542) 2549 : cluster [DBG] pgmap v3790: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:46:42.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:41 smithi154 conmon[34553]: cluster 2021-07-31T02:46:40.650457+0000 mgr.x (mgr.24542) 2550 : cluster 2021-07-31T02:46:42.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:41 smithi154 conmon[34553]: [DBG] pgmap v3791: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:46:42.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:41 smithi045 conmon[33655]: cluster 2021-07-31 2021-07-31T02:46:42.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:41 smithi045 conmon[33655]: T02:46:40.650457+0000 mgr.x (mgr.24542) 2550 : cluster [DBG] pgmap v3791: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:46:42.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:41 smithi045 conmon[39718]: cluster 2021-07-31T02:46:40.650457+0000 mgr.x (mgr.24542) 2550 : cluster [DBG] 2021-07-31T02:46:42.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:41 smithi045 conmon[39718]: pgmap v3791: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:46:44.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:43 smithi154 conmon[34553]: cluster 2021-07-31T02:46:42.650828+0000 mgr.x (mgr.24542) 2551 : cluster [DBG] 2021-07-31T02:46:44.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:43 smithi154 conmon[34553]: pgmap v3792: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:46:44.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:43 smithi045 conmon[33655]: cluster 2021-07-31T02:46:42.650828+0000 mgr.x (mgr.24542) 2551 2021-07-31T02:46:44.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:43 smithi045 conmon[33655]: : cluster [DBG] pgmap v3792: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:46:44.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:43 smithi045 conmon[39718]: cluster 2021-07-31T02:46:42.650828+0000 mgr.x (mgr.24542) 2551 : 2021-07-31T02:46:44.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:43 smithi045 conmon[39718]: cluster [DBG] pgmap v3792: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:46:44.301 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:46:44.578 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:46:44.579 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 61m ago 62m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:46:44.579 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 59m ago 62m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:46:44.579 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 59m ago 65m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:46:44.579 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 61m ago 67m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:46:44.579 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 61m ago 67m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:46:44.580 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 59m ago 65m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:46:44.580 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 61m ago 66m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:46:44.580 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 61m ago 62m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:46:44.580 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 59m ago 62m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:46:44.580 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 61m ago 65m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:46:44.581 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 61m ago 65m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:46:44.581 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 61m ago 64m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:46:44.581 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 61m ago 64m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:46:44.581 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 59m ago 64m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:46:44.581 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 59m ago 63m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:46:44.582 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 59m ago 63m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:46:44.582 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 59m ago 63m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:46:44.582 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 59m ago 62m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:46:44.917 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:46:44.917 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:46:44.917 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:46:44.918 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:46:44.918 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:46:44.918 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:46:44.918 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:46:44.919 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:46:44.919 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:46:44.919 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:46:44.919 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:46:44.920 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:46:44.920 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:46:44.920 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:46:44.920 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:46:44.920 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:46:44.921 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:46:45.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:44 smithi154 conmon[34553]: audit 2021-07-31T02:46:44.289123+0000 mgr.x (mgr.24542) 2552 : audit 2021-07-31T02:46:45.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:44 smithi154 conmon[34553]: [DBG] from='client.18120 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:46:45.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:44 smithi045 conmon[33655]: audit 2021-07-31T02:46:44.289123+0000 mgr.x (mgr 2021-07-31T02:46:45.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:44 smithi045 conmon[33655]: .24542) 2552 : audit [DBG] from='client.18120 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:46:45.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:44 smithi045 conmon[39718]: audit 2021-07-31T02:46:44.289123+0000 mgr.x (mgr. 2021-07-31T02:46:45.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:44 smithi045 conmon[39718]: 24542) 2552 : audit [DBG] from='client.18120 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:46:46.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:45 smithi154 conmon[34553]: audit 2021-07-31T02:46:44.570593+0000 mgr.x (mgr.24542) 2553 : audit 2021-07-31T02:46:46.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:45 smithi154 conmon[34553]: [DBG] from='client.28066 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:46:46.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:45 smithi154 conmon[34553]: cluster 2021-07-31T02:46:44.651346+0000 mgr.x (mgr.24542) 2554 : cluster [DBG] pgmap v3793: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:46:46.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:45 smithi154 conmon[34553]: audit 2021-07-31T02:46:44.916360+0000 mon.c (mon.1) 145 : audit [DBG] from='client.? 172.21.15.45:0/3750008396' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:46:46.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:45 smithi154 conmon[34553]: 2021-07-31T02:46:46.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:45 smithi045 conmon[39718]: audit 2021-07-31T02:46:44.570593+0000 2021-07-31T02:46:46.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:45 smithi045 conmon[39718]: mgr.x (mgr.24542) 2553 : audit [DBG] from='client.28066 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:46:46.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:45 smithi045 conmon[39718]: cluster 2021-07-31T02:46:46.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:45 smithi045 conmon[39718]: 2021-07-31T02:46:44.651346 2021-07-31T02:46:46.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:45 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 2021-07-31T02:46:46.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:45 smithi045 conmon[39718]: 2554 : cluster [DBG] pgmap v3793: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:46:46.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:45 smithi045 conmon[39718]: audit 2021 2021-07-31T02:46:46.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:45 smithi045 conmon[39718]: -07-31T02:46:44. 2021-07-31T02:46:46.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:45 smithi045 conmon[39718]: 916360+0000 mon.c (mon.1 2021-07-31T02:46:46.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:45 smithi045 conmon[39718]: ) 145 : audit [DBG] from='client.? 172.21.15.45:0/3750008396' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:46:46.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:45 smithi045 conmon[33655]: audit 2021-07-31T02:46:44.570593+0000 mgr.x (mgr.24542) 2553 : audit [DBG] from='client.28066 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:46:46.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:45 smithi045 conmon[33655]: 2021-07-31T02:46:46.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:45 smithi045 conmon[33655]: cluster 2021-07-31T02:46:44.651346+0000 mgr.x (mgr.24542) 2554 : cluster [DBG] pgmap v3793: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:46:46.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:45 smithi045 conmon[33655]: audit 2021-07-31T02:46:44.916360+0000 mon.c (mon.1) 145 : audit [DBG] 2021-07-31T02:46:46.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:45 smithi045 conmon[33655]: from='client.? 172.21.15.45:0/3750008396' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:46:48.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:47 smithi154 conmon[34553]: cluster 2021-07-31T02:46:46.651854+0000 mgr.x (mgr.24542) 2555 : cluster [DBG] 2021-07-31T02:46:48.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:47 smithi154 conmon[34553]: pgmap v3794: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:46:48.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:47 smithi045 conmon[33655]: cluster 2021-07-31T02:46:46.651854+0000 mgr.x (mgr.24542) 2555 : 2021-07-31T02:46:48.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:47 smithi045 conmon[33655]: cluster [DBG] pgmap v3794: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:46:48.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:47 smithi045 conmon[39718]: cluster 2021-07-31T02:46:46.651854+0000 mgr.x 2021-07-31T02:46:48.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:47 smithi045 conmon[39718]: (mgr.24542) 2555 : cluster [DBG] pgmap v3794: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:46:50.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:49 smithi154 conmon[34553]: cluster 2021-07-31T02:46:48.652303+0000 mgr.x (mgr.24542) 2556 : cluster [DBG] pgmap v3795: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:46:50.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:49 smithi045 conmon[39718]: cluster 2021-07-31T02:46:48.652303+0000 mgr.x ( 2021-07-31T02:46:50.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:49 smithi045 conmon[39718]: mgr.24542) 2556 : cluster [DBG] pgmap v3795: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:46:50.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:49 smithi045 conmon[33655]: cluster 2021-07-31T02:46:48.652303+0000 mgr.x (mgr.24542) 2556 : 2021-07-31T02:46:50.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:49 smithi045 conmon[33655]: cluster [DBG] pgmap v3795: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:46:52.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:51 smithi154 conmon[34553]: cluster 2021-07-31T02:46:50.652815+0000 mgr.x (mgr.24542) 2557 : cluster [DBG] pgmap v3796: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:46:52.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:51 smithi045 conmon[33655]: cluster 2021-07-31T02:46:50.652815+0000 mgr.x (mgr.24542) 2557 : 2021-07-31T02:46:52.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:51 smithi045 conmon[33655]: cluster [DBG] pgmap v3796: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:46:52.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:51 smithi045 conmon[39718]: cluster 2021-07-31T02:46:50.652815+0000 mgr.x (mgr.24542 2021-07-31T02:46:52.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:51 smithi045 conmon[39718]: ) 2557 : cluster [DBG] pgmap v3796: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:46:54.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:53 smithi154 conmon[34553]: cluster 2021-07-31T02:46:52.653162+0000 mgr.x (mgr.24542) 2558 : cluster [DBG] pgmap v3797: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:46:54.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:53 smithi154 conmon[34553]: 2021-07-31T02:46:54.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:53 smithi045 conmon[33655]: cluster 2021-07-31T02:46:52. 2021-07-31T02:46:54.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:53 smithi045 conmon[33655]: 653162+0000 mgr.x (mgr.24542) 2558 : cluster [DBG] pgmap v3797: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:46:54.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:53 smithi045 conmon[39718]: cluster 2021-07-31T02:46:52.653162+0000 mgr.x (mgr.24542) 2558 : cluster [DBG] 2021-07-31T02:46:54.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:53 smithi045 conmon[39718]: pgmap v3797: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:46:56.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:55 smithi154 conmon[34553]: cluster 2021-07-31T02:46:54.653647+0000 mgr.x (mgr.24542) 2559 : cluster [DBG] pgmap v3798: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:46:56.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:55 smithi045 conmon[33655]: cluster 2021-07-31T02:46:54.653647+0000 mgr.x (mgr 2021-07-31T02:46:56.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:55 smithi045 conmon[33655]: .24542) 2559 : cluster [DBG] pgmap v3798: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:46:56.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:55 smithi045 conmon[39718]: cluster 2021-07-31T02:46:54.653647+0000 mgr.x ( 2021-07-31T02:46:56.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:55 smithi045 conmon[39718]: mgr.24542) 2559 : cluster [DBG] pgmap v3798: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:46:58.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:57 smithi154 conmon[34553]: cluster 2021-07-31T02:46:56.653984+0000 mgr.x (mgr.24542) 2560 : cluster [DBG] pgmap v3799: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:46:58.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:57 smithi045 conmon[33655]: cluster 2021-07-31T02:46:56.653984+0000 mgr.x (mgr.24542) 2560 : cluster 2021-07-31T02:46:58.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:57 smithi045 conmon[33655]: [DBG] pgmap v3799: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:46:58.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:57 smithi045 conmon[39718]: cluster 2021-07-31T02:46:56.653984+0000 mgr.x (mgr.24542) 2560 2021-07-31T02:46:58.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:57 smithi045 conmon[39718]: : cluster [DBG] pgmap v3799: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:00.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:46:59 smithi154 conmon[34553]: cluster 2021-07-31T02:46:58.654460+0000 mgr.x (mgr.24542) 2561 : cluster [DBG] pgmap v3800: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:00.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:59 smithi045 conmon[33655]: cluster 2021-07-31T02:46:58.654460+0000 mgr.x 2021-07-31T02:47:00.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:46:59 smithi045 conmon[33655]: (mgr.24542) 2561 : cluster [DBG] pgmap v3800: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:00.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:59 smithi045 conmon[39718]: cluster 2021-07-31T02:46:58.654460+0000 mgr.x ( 2021-07-31T02:47:00.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:46:59 smithi045 conmon[39718]: mgr.24542) 2561 : cluster [DBG] pgmap v3800: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:02.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:01 smithi154 conmon[34553]: cluster 2021-07-31T02:47:00.655090+0000 mgr.x (mgr.24542) 2562 : cluster [DBG] pgmap v3801: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:02.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:01 smithi045 conmon[33655]: cluster 2021-07-31T02:47:00.655090+0000 mgr.x (mgr.24542) 2562 : cluster [DBG] 2021-07-31T02:47:02.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:01 smithi045 conmon[33655]: pgmap v3801: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:02.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:01 smithi045 conmon[39718]: cluster 2021-07-31T02:47:00.655090+0000 mgr.x ( 2021-07-31T02:47:02.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:01 smithi045 conmon[39718]: mgr.24542) 2562 : cluster [DBG] pgmap v3801: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:04.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:03 smithi154 conmon[34553]: cluster 2021-07-31T02:47:02.655449+0000 mgr.x (mgr.24542) 2563 : cluster [DBG] 2021-07-31T02:47:04.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:03 smithi154 conmon[34553]: pgmap v3802: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:04.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:03 smithi045 conmon[39718]: cluster 2021-07-31T02:47:02.655449+0000 mgr.x ( 2021-07-31T02:47:04.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:03 smithi045 conmon[39718]: mgr.24542) 2563 : cluster [DBG] pgmap v3802: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:04.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:03 smithi045 conmon[33655]: cluster 2021-07-31T02:47:02.655449+0000 mgr.x (mgr.24542 2021-07-31T02:47:04.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:03 smithi045 conmon[33655]: ) 2563 : cluster [DBG] pgmap v3802: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:06.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:05 smithi154 conmon[34553]: cluster 2021-07-31T02:47:04.655932+0000 mgr.x (mgr.24542) 2564 : cluster [DBG] pgmap v3803: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:06.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:05 smithi045 conmon[39718]: cluster 2021-07-31T02:47:04.655932+0000 mgr.x (mgr.24542) 2564 : cluster [DBG] pgmap v3803: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:06.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:05 smithi045 conmon[33655]: cluster 2021-07-31T02:47:04.655932+0000 mgr.x ( 2021-07-31T02:47:06.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:05 smithi045 conmon[33655]: mgr.24542) 2564 : cluster [DBG] pgmap v3803: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:08.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:07 smithi154 conmon[34553]: cluster 2021-07-31T02:47:06.656506+0000 mgr.x (mgr.24542) 2565 : cluster [DBG] pgmap v3804: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:08.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:07 smithi154 conmon[34553]: 2021-07-31T02:47:08.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:07 smithi045 conmon[33655]: cluster 2021-07-31T02:47:06.656506+0000 mgr.x (mgr.24542) 2565 : cluster [DBG] 2021-07-31T02:47:08.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:07 smithi045 conmon[33655]: pgmap v3804: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:08.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:07 smithi045 conmon[39718]: cluster 2021-07-31T02:47:06.656506+0000 mgr.x 2021-07-31T02:47:08.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:07 smithi045 conmon[39718]: (mgr.24542) 2565 : cluster [DBG] pgmap v3804: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:10.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:09 smithi154 conmon[34553]: cluster 2021-07-31T02:47:08.657013+0000 mgr.x (mgr.24542) 2566 : cluster [DBG] pgmap v3805: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:10.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:09 smithi154 conmon[34553]: 2021-07-31T02:47:10.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:09 smithi045 conmon[33655]: cluster 2021-07-31T02:47:08.657013+0000 mgr.x (mgr.24542 2021-07-31T02:47:10.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:09 smithi045 conmon[33655]: ) 2566 : cluster [DBG] pgmap v3805: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:10.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:09 smithi045 conmon[39718]: cluster 2021-07-31T02:47:08.657013+0000 mgr.x ( 2021-07-31T02:47:10.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:09 smithi045 conmon[39718]: mgr.24542) 2566 : cluster [DBG] pgmap v3805: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:12.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:11 smithi154 conmon[34553]: cluster 2021-07-31T02:47:10.657548+0000 mgr.x (mgr.24542) 2567 : cluster [DBG] pgmap v3806: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:12.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:11 smithi154 conmon[34553]: 2021-07-31T02:47:12.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:11 smithi045 conmon[39718]: cluster 2021-07-31T02:47:10.657548+0000 mgr.x (mgr. 2021-07-31T02:47:12.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:11 smithi045 conmon[39718]: 24542) 2567 : cluster [DBG] pgmap v3806: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:12.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:11 smithi045 conmon[33655]: cluster 2021-07-31T02:47:10.657548+0000 mgr.x (mgr.24542) 2021-07-31T02:47:12.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:11 smithi045 conmon[33655]: 2567 : cluster [DBG] pgmap v3806: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:14.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:13 smithi154 conmon[34553]: cluster 2021-07-31T02:47:12.657970+0000 mgr.x (mgr.24542) 2568 : cluster [DBG] pgmap v3807: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:14.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:13 smithi045 conmon[33655]: cluster 2021-07-31T02:47:12.657970+0000 mgr.x (mgr.24542) 2568 : cluster [DBG] pgmap v3807: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:14.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:13 smithi045 conmon[39718]: cluster 2021-07-31T02:47:12.657970+0000 mgr.x 2021-07-31T02:47:14.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:13 smithi045 conmon[39718]: (mgr.24542) 2568 : cluster [DBG] pgmap v3807: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:15.228 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:47:15.509 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:47:15.510 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 62m ago 62m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:47:15.510 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 59m ago 62m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:47:15.510 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 59m ago 66m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:47:15.510 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 62m ago 68m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:47:15.511 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 62m ago 68m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:47:15.511 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 59m ago 66m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:47:15.511 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 62m ago 66m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:47:15.511 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 62m ago 63m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:47:15.511 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 59m ago 63m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:47:15.512 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 62m ago 65m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:47:15.512 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 62m ago 65m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:47:15.512 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 62m ago 65m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:47:15.512 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 62m ago 64m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:47:15.512 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 59m ago 64m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:47:15.513 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 59m ago 64m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:47:15.513 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 59m ago 63m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:47:15.513 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 59m ago 63m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:47:15.513 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 59m ago 63m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:47:15.849 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:47:15.849 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:47:15.850 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:47:15.851 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:47:15.851 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:47:15.851 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:47:15.851 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:47:15.851 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:47:15.852 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:47:15.852 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:47:15.852 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:47:15.852 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:47:15.853 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:47:15.853 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:47:15.853 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:47:15.853 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:47:15.854 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:47:16.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:15 smithi154 conmon[34553]: cluster 2021-07-31T02:47:14.658524+0000 mgr.x (mgr.24542) 2569 : cluster [DBG] pgmap v3808: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:16.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:15 smithi154 conmon[34553]: audit 2021-07-31T02:47:15.215418+0000 mgr.x (mgr.24542) 2570 : audit [DBG] from='client.28078 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:47:16.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:15 smithi154 conmon[34553]: audit 2021-07-31T02:47:15.501364+0000 mgr.x (mgr.24542) 2571 : audit [DBG] from='client.18144 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:47:16.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:15 smithi045 conmon[33655]: cluster 2021-07-31T02:47:14.658524+0000 mgr.x 2021-07-31T02:47:16.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:15 smithi045 conmon[33655]: (mgr.24542) 2569 : cluster [DBG] pgmap v3808: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:16.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:15 smithi045 conmon[33655]: audit 2021-07-31T02:47:16.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:15 smithi045 conmon[33655]: 2021-07-31T02:47:15.215418+0000 mgr.x ( 2021-07-31T02:47:16.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:15 smithi045 conmon[33655]: mgr.24542) 2570 : audit [DBG] 2021-07-31T02:47:16.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:15 smithi045 conmon[33655]: from='client.28078 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:47:16.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:15 smithi045 conmon[33655]: audit 2021-07- 2021-07-31T02:47:16.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:15 smithi045 conmon[33655]: 31T02:47:15.501364+0000 mgr.x (mgr 2021-07-31T02:47:16.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:15 smithi045 conmon[33655]: .24542) 2571 : audit [DBG] 2021-07-31T02:47:16.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:15 smithi045 conmon[33655]: from='client.18144 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:47:16.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:15 smithi045 conmon[39718]: cluster 2021-07-31T02:47:14.658524+0000 mgr.x 2021-07-31T02:47:16.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:15 smithi045 conmon[39718]: (mgr.24542) 2569 : cluster [DBG] pgmap v3808: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:16.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:15 smithi045 conmon[39718]: audit 2021 2021-07-31T02:47:16.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:15 smithi045 conmon[39718]: -07-31T02:47:15.215418+0000 mgr.x (mgr.24542) 2570 : audit [DBG] from='client.28078 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:47:16.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:15 smithi045 conmon[39718]: audit 2021- 2021-07-31T02:47:16.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:15 smithi045 conmon[39718]: 07-31T02:47:15.501364+0000 mgr.x 2021-07-31T02:47:16.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:15 smithi045 conmon[39718]: (mgr.24542) 2571 : audit [DBG] from='client.18144 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:47:17.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:16 smithi154 conmon[34553]: audit 2021-07-31T02:47:15.849130+0000 mon.a (mon.0) 3204 : audit [DBG] from='client.? 172.21.15.45:0/1757904008' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:47:17.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:16 smithi045 conmon[33655]: audit 2021-07-31T02:47:15.849130+0000 2021-07-31T02:47:17.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:16 smithi045 conmon[33655]: mon.a (mon.0) 3204 : audit [DBG] from='client.? 172.21.15.45:0/1757904008' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:47:17.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:16 smithi045 conmon[39718]: audit 2021-07-31T02:47: 2021-07-31T02:47:17.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:16 smithi045 conmon[39718]: 15.849130+0000 mon.a (mon.0) 3204 : audit [DBG] from='client.? 172.21.15.45:0/1757904008' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:47:18.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:17 smithi154 conmon[34553]: cluster 2021-07-31T02:47:16.658941+0000 mgr.x (mgr.24542) 2572 : cluster [DBG] pgmap v3809: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:18.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:17 smithi045 conmon[33655]: cluster 2021-07-31T02:47:16.658941+0000 mgr.x (mgr.24542) 2572 : cluster [DBG] 2021-07-31T02:47:18.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:17 smithi045 conmon[33655]: pgmap v3809: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:18.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:17 smithi045 conmon[39718]: cluster 2021-07-31T02:47:16.658941+0000 mgr.x (mgr.24542) 2572 : cluster [DBG] pgmap v3809: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:20.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:19 smithi154 conmon[34553]: cluster 2021-07-31T02:47:18.659391+0000 mgr.x (mgr.24542) 2573 : cluster [DBG] pgmap v3810: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:20.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:19 smithi045 conmon[39718]: cluster 2021-07-31T02:47:18.659391+0000 mgr.x (mgr.24542) 2573 : cluster [DBG] pgmap v3810: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:20.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:19 smithi045 conmon[33655]: cluster 2021-07-31T02:47:18.659391+0000 mgr.x (mgr.24542) 2573 : 2021-07-31T02:47:20.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:19 smithi045 conmon[33655]: cluster [DBG] pgmap v3810: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:22.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:21 smithi154 conmon[34553]: cluster 2021-07-31T02:47:20.659980+0000 mgr.x (mgr.24542) 2574 : cluster [DBG] pgmap v3811: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:22.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:21 smithi045 conmon[39718]: cluster 2021-07-31T02:47:20.659980+0000 mgr.x (mgr.24542 2021-07-31T02:47:22.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:21 smithi045 conmon[39718]: ) 2574 : cluster [DBG] pgmap v3811: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:22.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:21 smithi045 conmon[33655]: cluster 2021-07-31T02:47:20.659980+0000 mgr.x (mgr.24542) 2574 : cluster [DBG] pgmap v3811: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:24.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:23 smithi154 conmon[34553]: cluster 2021-07-31T02:47:22.660342+0000 mgr.x (mgr.24542) 2575 : cluster [DBG] pgmap v3812: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:24.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:23 smithi045 conmon[39718]: cluster 2021-07-31T02:47:22.660342+0000 mgr.x (mgr 2021-07-31T02:47:24.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:23 smithi045 conmon[39718]: .24542) 2575 : cluster [DBG] pgmap v3812: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:24.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:23 smithi045 conmon[33655]: cluster 2021-07-31T02:47:22.660342 2021-07-31T02:47:24.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:23 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 2575 : cluster [DBG] pgmap v3812: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:26.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:25 smithi154 conmon[34553]: cluster 2021-07-31T02:47:24.660898+0000 mgr.x (mgr.24542) 2576 : cluster [DBG] pgmap v3813: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:26.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:25 smithi045 conmon[33655]: cluster 2021-07-31T02:47:24.660898 2021-07-31T02:47:26.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:25 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 2576 : cluster [DBG] pgmap v3813: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:26.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:25 smithi045 conmon[39718]: cluster 2021-07-31T02:47:24.660898+0000 mgr.x (mgr 2021-07-31T02:47:26.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:25 smithi045 conmon[39718]: .24542) 2576 : cluster [DBG] pgmap v3813: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:28.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:27 smithi154 conmon[34553]: cluster 2021-07-31T02:47:26.661347+0000 mgr.x (mgr.24542) 2577 : cluster [DBG] pgmap v3814: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:28.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:27 smithi154 conmon[34553]: 2021-07-31T02:47:28.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:27 smithi045 conmon[39718]: cluster 2021-07-31T02:47:26. 2021-07-31T02:47:28.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:27 smithi045 conmon[39718]: 661347+0000 mgr.x (mgr.24542) 2577 : cluster [DBG] pgmap v3814: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:28.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:27 smithi045 conmon[33655]: cluster 2021-07-31T02:47:26.661347+0000 mgr.x (mgr.24542) 2021-07-31T02:47:28.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:27 smithi045 conmon[33655]: 2577 : cluster [DBG] pgmap v3814: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:29.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:28 smithi154 conmon[34553]: audit 2021-07-31T02:47:28.177138+0000 mon.b (mon.2) 375 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:47:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:28 smithi154 conmon[34553]: 2021-07-31T02:47:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:28 smithi154 conmon[34553]: audit 2021-07-31T02:47:28.177632+0000 mon.b (mon.2) 376 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:47:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:28 smithi154 conmon[34553]: audit 2021-07-31T02:47:28.177904+0000 mon.a (mon.0) 3205 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:47:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:28 smithi154 conmon[34553]: audit 2021-07-31T02:47:28.178586+0000 mon.a (mon.0) 3206 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:47:29.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:28 smithi045 conmon[33655]: audit 2021-07-31T02:47:28.177138+0000 mon.b ( 2021-07-31T02:47:29.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:28 smithi045 conmon[33655]: mon.2) 375 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:47:29.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:28 smithi045 conmon[33655]: audit 2021-07-31T02:47:28 2021-07-31T02:47:29.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:28 smithi045 conmon[33655]: .177632+0000 mon.b (mon.2) 376 : audit [INF] 2021-07-31T02:47:29.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:28 smithi045 conmon[33655]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:47:29.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:28 smithi045 conmon[33655]: audit 2021-07-31T02:47:28.177904+0000 mon.a (mon.0) 3205 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:47:29.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:28 smithi045 conmon[33655]: audit 2021-07-31T02:47:28.178586+0000 mon.a (mon.0) 3206 : audit [INF] 2021-07-31T02:47:29.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:28 smithi045 conmon[33655]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:47:29.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:28 smithi045 conmon[39718]: audit 2021-07-31T02:47:28.177138+0000 mon.b ( 2021-07-31T02:47:29.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:28 smithi045 conmon[39718]: mon.2) 375 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:47:29.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:28 smithi045 conmon[39718]: audit 2021-07-31T02:47:28.177632+0000 2021-07-31T02:47:29.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:28 smithi045 conmon[39718]: mon.b (mon.2) 376 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:47:29.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:28 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T02:47:29.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:28 smithi045 conmon[39718]: T02:47:28.177904+0000 mon.a (mon.0) 3205 : audit 2021-07-31T02:47:29.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:28 smithi045 conmon[39718]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:47:29.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:28 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T02:47:29.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:28 smithi045 conmon[39718]: 31T02:47:28.178586+0000 mon.a (mon.0) 3206 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:47:29.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:29 smithi045 conmon[39718]: cluster 2021-07-31T02:47:28.661761+0000 mgr.x ( 2021-07-31T02:47:29.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:29 smithi045 conmon[39718]: mgr.24542) 2578 : cluster [DBG] pgmap v3815: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:29.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:29 smithi045 conmon[39718]: audit 2021-07-31T02:47:29.496369+0000 mon.b (mon.2 2021-07-31T02:47:29.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:29 smithi045 conmon[39718]: ) 377 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:47:29.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:29 smithi045 conmon[39718]: audit 2021-07 2021-07-31T02:47:29.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:29 smithi045 conmon[39718]: -31T02:47:29.497834+0000 mon.b (mon.2 2021-07-31T02:47:29.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:29 smithi045 conmon[39718]: ) 378 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:47:29.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:29 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T02:47:29.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:29 smithi045 conmon[39718]: T02:47:29.498731+0000 mon.b ( 2021-07-31T02:47:29.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:29 smithi045 conmon[39718]: mon.2) 379 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:47:29.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:29 smithi045 conmon[39718]: cephadm 2021 2021-07-31T02:47:29.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:29 smithi045 conmon[39718]: -07-31T02:47:29.499493 2021-07-31T02:47:29.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:29 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 2579 : cephadm 2021-07-31T02:47:29.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:29 smithi045 conmon[39718]: [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:47:29.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:29 smithi045 conmon[33655]: cluster 2021-07-31T02:47:28.661761+0000 mgr.x (mgr.24542 2021-07-31T02:47:29.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:29 smithi045 conmon[33655]: ) 2578 : cluster [DBG] pgmap v3815: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:29.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:29 smithi045 conmon[33655]: audit 2021-07-31T02:47 2021-07-31T02:47:29.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:29 smithi045 conmon[33655]: :29.496369+0000 mon.b (mon.2) 377 : audit [DBG] 2021-07-31T02:47:29.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:29 smithi045 conmon[33655]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:47:29.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:29 smithi045 conmon[33655]: audit 2021- 2021-07-31T02:47:29.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:29 smithi045 conmon[33655]: 07-31T02:47:29.497834+0000 2021-07-31T02:47:29.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:29 smithi045 conmon[33655]: mon.b (mon.2) 378 : audit [DBG] 2021-07-31T02:47:29.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:29 smithi045 conmon[33655]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:47:29.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:29 smithi045 conmon[33655]: audit 2021-07-31T02:47 2021-07-31T02:47:29.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:29 smithi045 conmon[33655]: :29.498731+0000 mon.b (mon.2 2021-07-31T02:47:29.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:29 smithi045 conmon[33655]: ) 379 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:47:29.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:29 smithi045 conmon[33655]: cephadm 2021- 2021-07-31T02:47:29.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:29 smithi045 conmon[33655]: 07-31T02:47:29.499493+0000 2021-07-31T02:47:29.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:29 smithi045 conmon[33655]: mgr.x (mgr.24542) 2579 : cephadm 2021-07-31T02:47:29.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:29 smithi045 conmon[33655]: [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:47:30.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[34553]: cluster 2021-07-31T02:47:28.661761+0000 mgr.x (mgr. 2021-07-31T02:47:30.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[34553]: 24542) 2578 : cluster [DBG] pgmap v3815: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:30.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[34553]: audit 2021-07-31T02:47:29.496369+0000 mon.b (mon.2) 377 : audit [DBG] 2021-07-31T02:47:30.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[34553]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:47:30.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[34553]: audit 2021-07-31T02:47:29 2021-07-31T02:47:30.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[34553]: .497834+0000 mon.b (mon.2) 378 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:47:30.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[34553]: audit 2021 2021-07-31T02:47:30.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[34553]: -07-31T02:47:29.498731+0000 2021-07-31T02:47:30.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[34553]: mon.b (mon.2) 379 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:47:30.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[34553]: 2021-07-31T02:47:30.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[34553]: cephadm 2021-07-31 2021-07-31T02:47:30.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[34553]: T02:47:29.499493+0000 mgr.x (mgr.24542) 2579 : cephadm [WRN] 2021-07-31T02:47:30.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[34553]: unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:47:30.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: debug 2021-07-31T02:47:29.542+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:47:30.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:47:30.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:47:30.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: yield 2021-07-31T02:47:30.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:47:30.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:47:30.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:47:30.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:47:30.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:47:30.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:47:30.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:47:30.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:47:30.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:47:30.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: task.result() 2021-07-31T02:47:30.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:47:30.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: debug 2021-07-31T02:47:29.554+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:47:30.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:47:30.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:47:30.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: yield 2021-07-31T02:47:30.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:47:30.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:47:30.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:47:30.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:47:30.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:47:30.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:47:30.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:47:30.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:47:30.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:47:30.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: task.result() 2021-07-31T02:47:30.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:47:30.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: debug 2021-07-31T02:47:29.580+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:47:30.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:47:30.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:47:30.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: yield 2021-07-31T02:47:30.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:47:30.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:47:30.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:47:30.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:47:30.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:47:30.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:47:30.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:47:30.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:47:30.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:47:30.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: task.result() 2021-07-31T02:47:30.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:47:30.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: debug 2021-07-31T02:47:29.580+0000 7fcb94162700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:47:30.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:47:30.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:47:30.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: yield 2021-07-31T02:47:30.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:47:30.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:47:30.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:47:30.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:47:30.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:47:30.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:47:30.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:47:30.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:47:30.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:47:30.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: task.result() 2021-07-31T02:47:30.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:47:30.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: 2021-07-31T02:47:30.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:47:30.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: 2021-07-31T02:47:30.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:47:30.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:47:30.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:47:30.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:47:30.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:47:30.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:47:30.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:47:30.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:47:30.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:47:30.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:47:30.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:47:30.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:47:30.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:47:30.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:47:30.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:47:30.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:47:30.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:47:30.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:47:30.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: raise 2021-07-31T02:47:30.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:47:30.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:47:30.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:47:30.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:47:30.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:47:30.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: debug 2021-07-31T02:47:29.588+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:47:30.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:47:30.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:47:30.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: yield 2021-07-31T02:47:30.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:47:30.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:47:30.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:47:30.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:47:30.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:47:30.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:47:30.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:47:30.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:47:30.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:47:30.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: task.result() 2021-07-31T02:47:30.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:47:30.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: debug 2021-07-31T02:47:29.589+0000 7fcb93961700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:47:30.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:47:30.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:47:30.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: yield 2021-07-31T02:47:30.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:47:30.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:47:30.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:47:30.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:47:30.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:47:30.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:47:30.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:47:30.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:47:30.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:47:30.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: task.result() 2021-07-31T02:47:30.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:47:30.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: 2021-07-31T02:47:30.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:47:30.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: 2021-07-31T02:47:30.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:47:30.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:47:30.035 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:47:30.035 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:47:30.035 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:47:30.035 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:47:30.035 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:47:30.036 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:47:30.036 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:47:30.036 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:47:30.036 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:47:30.037 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:47:30.037 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:47:30.037 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:47:30.037 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:47:30.037 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:47:30.038 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:47:30.038 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:47:30.038 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: raise 2021-07-31T02:47:30.038 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:47:30.039 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:47:30.039 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:47:30.039 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:47:30.039 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:47:29 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:47:31.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:47:29.543618+0000 mgr.x (mgr.24542) 2580 : cephadm [ERR] 2021-07-31T02:47:31.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:47:31.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:47:31.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: yield 2021-07-31T02:47:31.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:47:31.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:47:31.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:47:31.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:47:31.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:47:31.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:47:31.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:47:31.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:47:31.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:47:31.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:47:31.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:47:31.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:47:29. 2021-07-31T02:47:31.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: 544070+0000 mgr.x (mgr.24542) 2581 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:47:31.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:47:29.555329+0000 mgr.x (mgr.24542) 2582 : cephadm [ERR] 2021-07-31T02:47:31.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:47:31.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:47:31.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: yield 2021-07-31T02:47:31.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:47:31.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:47:31.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:47:31.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:47:31.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:47:31.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:47:31.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:47:31.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:47:31.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:47:31.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:47:31.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:47:31.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:47:29.555773+0000 mgr.x (mgr.24542) 2583 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:47:31.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:47:29.580953+0000 mgr.x (mgr.24542) 2584 : cephadm [ERR] 2021-07-31T02:47:31.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:47:31.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:47:31.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: yield 2021-07-31T02:47:31.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:47:31.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:47:31.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:47:31.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:47:31.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:47:31.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:47:31.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:47:31.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:47:31.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:47:31.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:47:31.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:47:31.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:47:29.581432+0000 mgr.x (mgr.24542) 2585 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:47:31.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:47:31.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:47:31.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: yield 2021-07-31T02:47:31.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:47:31.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:47:31.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:47:31.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:47:31.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:47:31.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:47:31.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:47:31.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:47:31.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:47:31.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:47:31.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:47:31.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: 2021-07-31T02:47:31.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:47:31.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: 2021-07-31T02:47:31.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:47:31.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:47:31.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:47:31.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:47:31.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:47:31.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:47:31.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:47:31.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:47:31.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:47:31.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:47:31.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:47:31.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:47:31.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:47:31.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:47:31.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:47:31.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:47:31.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:47:31.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:47:31.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: raise 2021-07-31T02:47:31.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:47:31.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:47:31.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:47:31.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:47:31.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:47:31.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:47:29.589916+0000 mgr.x (mgr.24542) 2586 : cephadm [ERR] 2021-07-31T02:47:31.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:47:31.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:47:31.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: yield 2021-07-31T02:47:31.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:47:31.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:47:31.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:47:31.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:47:31.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:47:31.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:47:31.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:47:31.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:47:31.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:47:31.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:47:31.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:47:31.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:47:29.590342+0000 mgr.x (mgr.24542) 2587 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:47:31.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:47:31.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:47:31.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: yield 2021-07-31T02:47:31.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:47:31.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:47:31.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:47:31.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:47:31.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:47:31.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:47:31.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:47:31.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:47:31.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:47:31.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:47:31.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:47:31.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: 2021-07-31T02:47:31.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:47:31.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: 2021-07-31T02:47:31.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:47:31.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:47:31.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:47:31.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:47:31.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:47:31.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:47:31.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:47:31.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:47:31.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:47:31.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:47:31.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:47:31.035 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:47:31.035 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:47:31.035 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:47:31.035 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:47:31.036 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:47:31.036 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:47:31.036 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:47:31.036 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: raise 2021-07-31T02:47:31.036 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:47:31.037 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:47:31.037 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:47:31.037 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:47:31.037 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:30 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:47:31.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:47:29. 2021-07-31T02:47:31.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: 543618+0000 mgr.x (mgr.24542) 2580 : cephadm [ERR] 2021-07-31T02:47:31.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:47:31.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:47:31.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: yield 2021-07-31T02:47:31.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:47:31.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:47:31.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:47:31.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:47:31.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:47:31.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:47:31.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:47:31.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:47:31.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:47:31.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:47:31.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:47:31.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:47:29.544070+0000 mgr.x (mgr.24542) 2021-07-31T02:47:31.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: 2581 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:47:31.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: cephadm 2021 2021-07-31T02:47:31.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: -07-31T02:47:29. 2021-07-31T02:47:31.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: 555329+0000 mgr.x (mgr. 2021-07-31T02:47:31.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: 24542) 2582 : cephadm [ERR] 2021-07-31T02:47:31.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: 2021-07-31T02:47:31.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:47:31.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:47:31.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: yield 2021-07-31T02:47:31.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:47:31.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:47:31.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:47:31.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:47:31.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:47:31.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:47:31.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:47:31.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:47:31.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:47:31.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:47:31.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:47:31.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:47:29.555773+0000 mgr.x (mgr.24542) 2583 : cephadm [INF] 2021-07-31T02:47:31.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:47:31.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: cephadm 2021 2021-07-31T02:47:31.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: -07-31T02:47:29 2021-07-31T02:47:31.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: .580953+0000 mgr.x (mgr. 2021-07-31T02:47:31.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: 24542) 2584 : cephadm [ERR] 2021-07-31T02:47:31.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: 2021-07-31T02:47:31.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:47:31.049 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:47:29.543618+0000 mgr.x (mgr.24542) 2580 : cephadm [ERR] 2021-07-31T02:47:31.049 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:47:31.049 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:47:31.050 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: yield 2021-07-31T02:47:31.050 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:47:31.050 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:47:31.050 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:47:31.050 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:47:31.051 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:47:31.051 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:47:31.051 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:47:31.051 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:47:31.052 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:47:31.052 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:47:31.052 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:47:31.052 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: 2021-07-31T02:47:31.052 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:47:29.544070+0000 mgr.x (mgr.24542) 2581 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:47:31.053 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:47:29.555329+0000 mgr.x (mgr.24542) 2582 : cephadm [ERR] 2021-07-31T02:47:31.053 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:47:31.053 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:47:31.053 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: yield 2021-07-31T02:47:31.053 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:47:31.054 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:47:31.054 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:47:31.055 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:47:31.055 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:47:31.055 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:47:31.055 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:47:31.056 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:47:31.056 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:47:31.056 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:47:31.056 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:47:31.056 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:47:29.555773+0000 mgr.x (mgr. 2021-07-31T02:47:31.057 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: 24542) 2583 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:47:31.057 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:47:29.580953+0000 mgr.x (mgr.24542) 2584 : cephadm [ERR] 2021-07-31T02:47:31.057 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:47:31.057 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:47:31.058 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: yield 2021-07-31T02:47:31.058 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:47:31.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:47:31.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: yield 2021-07-31T02:47:31.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:47:31.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:47:31.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:47:31.060 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:47:31.060 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:47:31.060 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:47:31.060 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:47:31.060 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:47:31.061 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:47:31.061 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:47:31.061 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:47:31.061 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:47:29.581432+0000 mgr.x (mgr.24542) 2021-07-31T02:47:31.062 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: 2585 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:47:31.062 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:47:31.062 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:47:31.062 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: yield 2021-07-31T02:47:31.062 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:47:31.063 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:47:31.063 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:47:31.063 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:47:31.064 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:47:31.064 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:47:31.064 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:47:31.066 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:47:31.066 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:47:31.067 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:47:31.067 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:47:31.067 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: 2021-07-31T02:47:31.067 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:47:31.067 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: 2021-07-31T02:47:31.068 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:47:31.068 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:47:31.068 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:47:31.068 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:47:31.068 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:47:31.069 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:47:31.069 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:47:31.069 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:47:31.069 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:47:31.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:47:31.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:47:31.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:47:31.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:47:31.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:47:31.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:47:31.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:47:31.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:47:31.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:47:31.074 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:47:31.074 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: cephadm 2021- 2021-07-31T02:47:31.074 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: 07-31T02:47:29.581432+0000 mgr.x (mgr.24542) 2585 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:47:31.074 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:47:31.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:47:31.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: yield 2021-07-31T02:47:31.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:47:31.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:47:31.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:47:31.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:47:31.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:47:31.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:47:31.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:47:31.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:47:31.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:47:31.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:47:31.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:47:31.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: 2021-07-31T02:47:31.078 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:47:31.078 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: 2021-07-31T02:47:31.078 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:47:31.078 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:47:31.078 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:47:31.079 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:47:31.079 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:47:31.079 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:47:31.079 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:47:31.079 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:47:31.080 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:47:31.080 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:47:31.080 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:47:31.080 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:47:31.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:47:31.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:47:31.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:47:31.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:47:31.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:47:31.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:47:31.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: raise 2021-07-31T02:47:31.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:47:31.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:47:31.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:47:31.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:47:31.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:47:31.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:47:29.589916+0000 mgr.x (mgr.24542) 2586 : cephadm [ERR] 2021-07-31T02:47:31.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: 2021-07-31T02:47:31.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:47:31.084 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:47:31.084 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: yield 2021-07-31T02:47:31.084 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:47:31.084 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:47:31.084 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:47:31.085 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:47:31.085 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:47:31.085 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:47:31.086 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:47:31.086 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:47:31.086 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:47:31.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:47:31.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:47:31.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:47:29.590342+0000 mgr.x (mgr.24542) 2587 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:47:31.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:47:31.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:47:31.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: yield 2021-07-31T02:47:31.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:47:31.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:47:31.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:47:31.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:47:31.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:47:31.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:47:31.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:47:31.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:47:31.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:47:31.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:47:31.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:47:31.093 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: 2021-07-31T02:47:31.093 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:47:31.093 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: 2021-07-31T02:47:31.093 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:47:31.093 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:47:31.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:47:31.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:47:31.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:47:31.096 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:47:31.096 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:47:31.097 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:47:31.097 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:47:31.097 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:47:31.097 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:47:31.098 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:47:31.098 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:47:31.099 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:47:31.099 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:47:31.099 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:47:31.099 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:47:31.100 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:47:31.100 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: raise 2021-07-31T02:47:31.100 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:47:31.100 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:47:31.100 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:47:31.101 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:47:31.101 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:47:31.102 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:47:31.102 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:47:31.102 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:47:31.103 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:47:31.103 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:47:31.103 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:47:31.103 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:47:31.104 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:47:31.104 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:47:31.104 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: raise 2021-07-31T02:47:31.104 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:47:31.104 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:47:31.105 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:47:31.105 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:47:31.105 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:47:31.105 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:47:29.589916+0000 mgr.x (mgr.24542) 2586 : cephadm [ERR] 2021-07-31T02:47:31.105 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:47:31.106 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:47:31.106 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: yield 2021-07-31T02:47:31.106 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:47:31.106 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:47:31.107 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:47:31.107 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:47:31.107 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:47:31.107 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:47:31.107 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:47:31.108 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:47:31.108 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:47:31.108 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:47:31.108 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:47:31.108 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:47:29.590342+0000 mgr.x (mgr.24542) 2587 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:47:31.109 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:47:31.109 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:47:31.109 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: yield 2021-07-31T02:47:31.109 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:47:31.110 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:47:31.110 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:47:31.110 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:47:31.110 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:47:31.110 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:47:31.111 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:47:31.111 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:47:31.111 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:47:31.111 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:47:31.112 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:47:31.112 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: 2021-07-31T02:47:31.112 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:47:31.112 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: 2021-07-31T02:47:31.113 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:47:31.118 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:47:31.119 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:47:31.119 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:47:31.119 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:47:31.119 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:47:31.120 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:47:31.120 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:47:31.120 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:47:31.120 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:47:31.120 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:47:31.121 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:47:31.121 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:47:31.121 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:47:31.121 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:47:31.122 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:47:31.122 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:47:31.122 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:47:31.122 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: raise 2021-07-31T02:47:31.122 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:47:31.123 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:47:31.123 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:47:31.123 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:47:31.124 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:30 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:47:32.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:31 smithi154 conmon[34553]: cluster 2021-07-31T02:47:30.662341+0000 mgr.x (mgr.24542) 2588 : cluster 2021-07-31T02:47:32.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:31 smithi154 conmon[34553]: [DBG] pgmap v3816: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:32.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:31 smithi045 conmon[33655]: cluster 2021-07-31T02:47:30.662341+0000 mgr.x (mgr. 2021-07-31T02:47:32.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:31 smithi045 conmon[33655]: 24542) 2588 : cluster [DBG] pgmap v3816: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:32.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:31 smithi045 conmon[39718]: cluster 2021-07-31T02:47:30.662341+0000 mgr.x (mgr. 2021-07-31T02:47:32.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:31 smithi045 conmon[39718]: 24542) 2588 : cluster [DBG] pgmap v3816: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:34.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:33 smithi154 conmon[34553]: cluster 2021-07-31T02:47:32.662708+0000 mgr.x (mgr.24542) 2589 : cluster [DBG] 2021-07-31T02:47:34.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:33 smithi154 conmon[34553]: pgmap v3817: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:34.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:33 smithi045 conmon[33655]: cluster 2021-07-31T02:47:32.662708+0000 2021-07-31T02:47:34.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:33 smithi045 conmon[33655]: mgr.x (mgr.24542) 2589 : cluster [DBG] pgmap v3817: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:34.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:33 smithi045 conmon[39718]: cluster 2021-07-31T02:47:32.662708+0000 mgr.x (mgr. 2021-07-31T02:47:34.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:33 smithi045 conmon[39718]: 24542) 2589 : cluster [DBG] pgmap v3817: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:36.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:35 smithi154 conmon[34553]: cluster 2021-07-31T02:47:34.663238+0000 mgr.x (mgr.24542) 2590 : cluster [DBG] pgmap v3818: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:36.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:35 smithi154 conmon[34553]: 2021-07-31T02:47:36.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:35 smithi045 conmon[33655]: cluster 2021-07-31T02:47:34.663238+0000 mgr.x (mgr.24542) 2590 2021-07-31T02:47:36.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:35 smithi045 conmon[33655]: : cluster [DBG] pgmap v3818: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:36.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:35 smithi045 conmon[39718]: cluster 2021-07-31T02:47:34.663238+0000 mgr.x ( 2021-07-31T02:47:36.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:35 smithi045 conmon[39718]: mgr.24542) 2590 : cluster [DBG] pgmap v3818: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:38.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:37 smithi154 conmon[34553]: cluster 2021-07-31T02:47:36.663754+0000 mgr.x (mgr.24542) 2591 : cluster [DBG] pgmap v3819: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:38.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:37 smithi154 conmon[34553]: 2021-07-31T02:47:38.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:37 smithi045 conmon[33655]: cluster 2021-07-31T02:47:36.663754+0000 mgr.x (mgr.24542) 2021-07-31T02:47:38.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:37 smithi045 conmon[33655]: 2591 : cluster [DBG] pgmap v3819: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:38.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:37 smithi045 conmon[39718]: cluster 2021-07-31T02:47:36 2021-07-31T02:47:38.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:37 smithi045 conmon[39718]: .663754+0000 mgr.x (mgr.24542) 2591 : cluster [DBG] pgmap v3819: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:40.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:39 smithi154 conmon[34553]: cluster 2021-07-31T02:47:38.664152+0000 mgr.x (mgr.24542) 2592 : cluster [DBG] pgmap v3820: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:40.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:39 smithi045 conmon[39718]: cluster 2021-07-31T02:47:38.664152+0000 mgr.x 2021-07-31T02:47:40.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:39 smithi045 conmon[39718]: (mgr.24542) 2592 : cluster [DBG] pgmap v3820: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:40.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:39 smithi045 conmon[33655]: cluster 2021-07-31T02:47:38.664152+0000 mgr.x (mgr.24542) 2592 : 2021-07-31T02:47:40.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:39 smithi045 conmon[33655]: cluster [DBG] pgmap v3820: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:42.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:41 smithi154 conmon[34553]: cluster 2021-07-31T02:47:40.664622+0000 mgr.x (mgr.24542) 2593 : cluster [DBG] pgmap v3821: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:42.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:41 smithi045 conmon[39718]: cluster 2021-07-31T02:47:40.664622+0000 mgr.x (mgr.24542) 2593 : cluster [DBG] pgmap v3821: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:42.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:41 smithi045 conmon[33655]: cluster 2021-07-31T02:47:40.664622 2021-07-31T02:47:42.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:41 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 2593 : cluster [DBG] pgmap v3821: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:44.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:43 smithi154 conmon[34553]: cluster 2021-07-31T02:47:42.665017+0000 mgr.x (mgr.24542) 2594 : cluster [DBG] pgmap v3822: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:44.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:43 smithi154 conmon[34553]: 2021-07-31T02:47:44.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:43 smithi045 conmon[33655]: cluster 2021-07-31T02:47:42.665017 2021-07-31T02:47:44.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:43 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 2594 : cluster [DBG] pgmap v3822: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:44.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:43 smithi045 conmon[39718]: cluster 2021-07-31T02:47:42.665017+0000 mgr.x ( 2021-07-31T02:47:44.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:43 smithi045 conmon[39718]: mgr.24542) 2594 : cluster [DBG] pgmap v3822: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:46.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:45 smithi154 conmon[34553]: cluster 2021-07-31T02:47:44.665437+0000 mgr.x (mgr.24542) 2595 : cluster [DBG] pgmap v3823: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:46.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:45 smithi045 conmon[39718]: cluster 2021-07-31T02:47:44.665437+0000 mgr.x 2021-07-31T02:47:46.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:45 smithi045 conmon[39718]: (mgr.24542) 2595 : cluster [DBG] pgmap v3823: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:46.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:45 smithi045 conmon[33655]: cluster 2021-07-31T02:47:44.665437+0000 mgr.x (mgr 2021-07-31T02:47:46.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:45 smithi045 conmon[33655]: .24542) 2595 : cluster [DBG] pgmap v3823: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:46.162 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:47:46.440 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:47:46.440 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 62m ago 63m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:47:46.440 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 60m ago 63m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:47:46.441 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 60m ago 66m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:47:46.441 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 62m ago 68m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:47:46.441 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 62m ago 68m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:47:46.441 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 60m ago 66m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:47:46.441 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 62m ago 67m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:47:46.442 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 62m ago 63m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:47:46.442 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 60m ago 63m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:47:46.442 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 62m ago 66m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:47:46.442 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 62m ago 66m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:47:46.443 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 62m ago 65m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:47:46.443 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 62m ago 65m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:47:46.443 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 60m ago 65m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:47:46.443 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 60m ago 64m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:47:46.444 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 60m ago 64m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:47:46.445 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 60m ago 64m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:47:46.445 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 60m ago 63m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:47:46.778 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:47:46.779 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:47:46.779 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:47:46.779 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:47:46.779 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:47:46.780 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:47:46.780 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:47:46.780 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:47:46.780 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:47:46.781 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:47:46.781 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:47:46.781 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:47:46.781 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:47:46.781 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:47:46.782 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:47:46.782 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:47:46.782 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:47:47.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:46 smithi154 conmon[34553]: audit 2021-07-31T02:47:46.150636+0000 mgr.x (mgr.24542) 2596 : audit [DBG] from='client.28096 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:47:47.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:46 smithi154 conmon[34553]: audit 2021-07-31T02:47:46.432059+0000 mgr.x (mgr.24542) 2597 : audit [DBG] from='client.18162 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:47:47.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:46 smithi045 conmon[39718]: audit 2021-07-31T02:47:46.150636+0000 mgr.x 2021-07-31T02:47:47.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:46 smithi045 conmon[39718]: (mgr.24542) 2596 : audit [DBG] from='client.28096 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:47:47.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:46 smithi045 conmon[39718]: audit 2021-07-31T02:47:46.432059+0000 2021-07-31T02:47:47.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:46 smithi045 conmon[39718]: mgr.x (mgr.24542) 2597 : audit [DBG] from='client.18162 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:47:47.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:46 smithi045 conmon[33655]: audit 2021-07-31T02:47:46.150636+0000 mgr.x (mgr.24542) 2596 : audit [DBG] from='client.28096 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:47:47.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:46 smithi045 conmon[33655]: 2021-07-31T02:47:47.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:46 smithi045 conmon[33655]: audit 2021-07-31T02:47:46.432059+0000 mgr.x (mgr. 2021-07-31T02:47:47.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:46 smithi045 conmon[33655]: 24542) 2597 : audit [DBG] from='client.18162 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:47:48.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:47 smithi154 conmon[34553]: cluster 2021-07-31T02:47:46.665945+0000 mgr.x (mgr.24542) 2598 : cluster [DBG] pgmap v3824: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:48.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:47 smithi154 conmon[34553]: audit 2021-07-31T02:47:46.778727+0000 mon.c (mon.1) 146 : audit [DBG] from='client.? 172.21.15.45:0/924203798' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:47:48.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:47 smithi045 conmon[33655]: cluster 2021-07-31T02:47:46.665945+0000 mgr.x (mgr. 2021-07-31T02:47:48.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:47 smithi045 conmon[33655]: 24542) 2598 : cluster [DBG] pgmap v3824: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:48.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:47 smithi045 conmon[33655]: audit 2021-07-31T02:47:46.778727+0000 mon.c (mon.1) 146 : audit [DBG] from='client.? 172.21.15.45:0/924203798' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:47:48.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:47 smithi045 conmon[39718]: cluster 2021-07-31T02:47:46.665945 2021-07-31T02:47:48.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:47 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 2598 : cluster [DBG] pgmap v3824: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:48.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:47 smithi045 conmon[39718]: audit 2021-07-31T02:47:48.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:47 smithi045 conmon[39718]: 2021-07-31T02:47:46.778727+0000 mon.c (mon.1) 146 : audit [DBG] from='client.? 172.21.15.45:0/924203798' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:47:50.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:49 smithi154 conmon[34553]: cluster 2021-07-31T02:47:48.666417+0000 mgr.x (mgr.24542) 2599 : cluster [DBG] pgmap v3825: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:50.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:49 smithi045 conmon[33655]: cluster 2021-07-31T02:47:48.666417+0000 mgr.x (mgr.24542) 2021-07-31T02:47:50.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:49 smithi045 conmon[33655]: 2599 : cluster [DBG] pgmap v3825: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:50.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:49 smithi045 conmon[39718]: cluster 2021-07-31T02:47:48.666417+0000 mgr.x (mgr 2021-07-31T02:47:50.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:49 smithi045 conmon[39718]: .24542) 2599 : cluster [DBG] pgmap v3825: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:52.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:51 smithi154 conmon[34553]: cluster 2021-07-31T02:47:50.666976+0000 mgr.x (mgr.24542) 2600 : cluster [DBG] pgmap v3826: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:52.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:51 smithi045 conmon[39718]: cluster 2021-07-31T02:47:50.666976+0000 mgr.x ( 2021-07-31T02:47:52.256 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:51 smithi045 conmon[39718]: mgr.24542) 2600 : cluster [DBG] pgmap v3826: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:52.257 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:51 smithi045 conmon[33655]: cluster 2021-07-31T02:47:50.666976+0000 2021-07-31T02:47:52.257 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:51 smithi045 conmon[33655]: mgr.x (mgr.24542) 2600 : cluster [DBG] pgmap v3826: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:54.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:53 smithi154 conmon[34553]: cluster 2021-07-31T02:47:52.667343+0000 mgr.x (mgr.24542) 2601 : cluster [DBG] pgmap v3827: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:54.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:53 smithi045 conmon[33655]: cluster 2021-07-31T02:47:52.667343+0000 mgr.x (mgr.24542) 2601 : cluster 2021-07-31T02:47:54.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:53 smithi045 conmon[33655]: [DBG] pgmap v3827: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:54.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:53 smithi045 conmon[39718]: cluster 2021-07-31T02:47:52.667343+0000 mgr.x 2021-07-31T02:47:54.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:53 smithi045 conmon[39718]: (mgr.24542) 2601 : cluster [DBG] pgmap v3827: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:56.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:55 smithi154 conmon[34553]: cluster 2021-07-31T02:47:54.667905+0000 mgr.x (mgr.24542) 2602 : cluster [DBG] pgmap v3828: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:56.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:55 smithi045 conmon[33655]: cluster 2021-07-31T02:47:54.667905+0000 mgr.x (mgr.24542) 2021-07-31T02:47:56.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:55 smithi045 conmon[33655]: 2602 : cluster [DBG] pgmap v3828: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:56.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:55 smithi045 conmon[39718]: cluster 2021-07-31T02:47:54.667905+0000 mgr.x (mgr 2021-07-31T02:47:56.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:55 smithi045 conmon[39718]: .24542) 2602 : cluster [DBG] pgmap v3828: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:58.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:57 smithi154 conmon[34553]: cluster 2021-07-31T02:47:56.668347+0000 mgr.x (mgr.24542) 2603 : cluster [DBG] pgmap v3829: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:58.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:57 smithi045 conmon[33655]: cluster 2021-07-31T02:47:56.668347+0000 mgr.x (mgr.24542 2021-07-31T02:47:58.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:57 smithi045 conmon[33655]: ) 2603 : cluster [DBG] pgmap v3829: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:47:58.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:57 smithi045 conmon[39718]: cluster 2021-07-31T02:47:56. 2021-07-31T02:47:58.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:57 smithi045 conmon[39718]: 668347+0000 mgr.x (mgr.24542) 2603 : cluster [DBG] pgmap v3829: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:00.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:47:59 smithi154 conmon[34553]: cluster 2021-07-31T02:47:58.668900+0000 mgr.x (mgr.24542) 2604 : cluster [DBG] pgmap v3830: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:00.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:59 smithi045 conmon[33655]: cluster 2021-07-31T02:47:58.668900+0000 mgr.x ( 2021-07-31T02:48:00.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:47:59 smithi045 conmon[33655]: mgr.24542) 2604 : cluster [DBG] pgmap v3830: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:00.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:59 smithi045 conmon[39718]: cluster 2021-07-31T02:47:58.668900+0000 mgr.x (mgr 2021-07-31T02:48:00.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:47:59 smithi045 conmon[39718]: .24542) 2604 : cluster [DBG] pgmap v3830: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:02.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:01 smithi154 conmon[34553]: cluster 2021-07-31T02:48:00.669361+0000 mgr.x (mgr.24542) 2605 : cluster [DBG] pgmap v3831: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:02.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:01 smithi045 conmon[33655]: cluster 2021-07-31T02:48:00.669361+0000 mgr.x (mgr.24542) 2605 : cluster [DBG] pgmap v3831: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:02.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:01 smithi045 conmon[39718]: cluster 2021-07-31T02:48:00.669361+0000 mgr.x (mgr.24542) 2605 : cluster [DBG] pgmap v3831: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:04.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:03 smithi154 conmon[34553]: cluster 2021-07-31T02:48:02.669878+0000 mgr.x (mgr.24542) 2606 : cluster [DBG] pgmap v3832: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:04.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:03 smithi045 conmon[33655]: cluster 2021-07-31T02:48:02.669878+0000 mgr.x (mgr.24542) 2606 : cluster 2021-07-31T02:48:04.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:03 smithi045 conmon[33655]: [DBG] pgmap v3832: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:04.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:03 smithi045 conmon[39718]: cluster 2021-07-31T02:48:02.669878+0000 mgr.x ( 2021-07-31T02:48:04.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:03 smithi045 conmon[39718]: mgr.24542) 2606 : cluster [DBG] pgmap v3832: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:06.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:05 smithi154 conmon[34553]: cluster 2021-07-31T02:48:04.670405+0000 mgr.x (mgr.24542) 2607 : cluster [DBG] pgmap v3833: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:06.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:05 smithi045 conmon[39718]: cluster 2021-07-31T02:48:04.670405+0000 mgr.x (mgr.24542) 2607 2021-07-31T02:48:06.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:05 smithi045 conmon[39718]: : cluster [DBG] pgmap v3833: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:06.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:05 smithi045 conmon[33655]: cluster 2021-07-31T02:48:04.670405 2021-07-31T02:48:06.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:05 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 2607 : cluster [DBG] pgmap v3833: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:08.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:07 smithi154 conmon[34553]: cluster 2021-07-31T02:48:06.670796+0000 mgr.x (mgr.24542) 2608 : cluster [DBG] pgmap v3834: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:08.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:07 smithi045 conmon[39718]: cluster 2021-07-31T02:48:06.670796+0000 mgr.x (mgr.24542) 2608 2021-07-31T02:48:08.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:07 smithi045 conmon[39718]: : cluster [DBG] pgmap v3834: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:08.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:07 smithi045 conmon[33655]: cluster 2021-07-31T02:48:06.670796+0000 mgr.x (mgr.24542 2021-07-31T02:48:08.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:07 smithi045 conmon[33655]: ) 2608 : cluster [DBG] pgmap v3834: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:10.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:09 smithi154 conmon[34553]: cluster 2021-07-31T02:48:08.671319+0000 mgr.x (mgr.24542) 2609 : cluster [DBG] pgmap v3835: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:10.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:09 smithi045 conmon[39718]: cluster 2021-07-31T02:48:08.671319+0000 mgr.x (mgr.24542) 2021-07-31T02:48:10.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:09 smithi045 conmon[39718]: 2609 : cluster [DBG] pgmap v3835: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:10.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:09 smithi045 conmon[33655]: cluster 2021-07-31T02:48:08. 2021-07-31T02:48:10.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:09 smithi045 conmon[33655]: 671319+0000 mgr.x (mgr.24542) 2609 : cluster [DBG] pgmap v3835: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:12.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:11 smithi154 conmon[34553]: cluster 2021-07-31T02:48:10.671745+0000 mgr.x (mgr.24542) 2610 : cluster [DBG] pgmap v3836: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:12.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:11 smithi045 conmon[39718]: cluster 2021-07-31T02:48:10.671745+0000 mgr.x (mgr.24542) 2610 : cluster [DBG] pgmap v3836: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:12.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:11 smithi045 conmon[33655]: cluster 2021-07-31T02:48:10.671745+0000 mgr.x (mgr.24542) 2610 : cluster [DBG] 2021-07-31T02:48:12.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:11 smithi045 conmon[33655]: pgmap v3836: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:14.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:13 smithi154 conmon[34553]: cluster 2021-07-31T02:48:12.672283+0000 mgr.x (mgr.24542) 2611 : cluster [DBG] pgmap v3837: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:14.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:13 smithi045 conmon[39718]: cluster 2021-07-31T02:48:12.672283+0000 mgr.x 2021-07-31T02:48:14.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:13 smithi045 conmon[39718]: (mgr.24542) 2611 : cluster [DBG] pgmap v3837: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:14.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:13 smithi045 conmon[33655]: cluster 2021-07-31T02:48:14.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:13 smithi045 conmon[33655]: 2021-07-31T02:48:12.672283+0000 mgr.x (mgr.24542) 2611 : cluster [DBG] pgmap v3837: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:16.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:15 smithi154 conmon[34553]: cluster 2021-07-31T02:48:14.672795+0000 mgr.x (mgr.24542) 2612 : cluster [DBG] pgmap v3838: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:16.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:15 smithi045 conmon[33655]: cluster 2021-07-31T02:48:14.672795+0000 mgr.x (mgr.24542) 2612 : cluster [DBG] pgmap v3838: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:16.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:15 smithi045 conmon[39718]: cluster 2021-07-31T02:48:14.672795+0000 mgr.x 2021-07-31T02:48:16.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:15 smithi045 conmon[39718]: (mgr.24542) 2612 : cluster [DBG] pgmap v3838: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:17.089 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:48:17.366 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:48:17.367 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 63m ago 63m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:48:17.367 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 60m ago 63m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:48:17.367 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 60m ago 67m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:48:17.367 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 63m ago 69m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:48:17.367 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 63m ago 69m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:48:17.368 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 60m ago 67m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:48:17.368 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 63m ago 67m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:48:17.368 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 63m ago 64m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:48:17.368 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 60m ago 64m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:48:17.369 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 63m ago 66m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:48:17.369 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 63m ago 66m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:48:17.369 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 63m ago 66m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:48:17.369 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 63m ago 65m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:48:17.370 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 60m ago 65m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:48:17.370 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 60m ago 65m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:48:17.370 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 60m ago 64m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:48:17.370 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 60m ago 64m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:48:17.370 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 60m ago 64m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:48:17.703 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:48:17.703 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:48:17.703 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:48:17.704 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:48:17.704 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:48:17.704 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:48:17.704 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:48:17.704 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:48:17.705 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:48:17.705 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:48:17.705 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:48:17.705 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:48:17.705 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:48:17.706 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:48:17.706 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:48:17.706 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:48:17.707 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:48:18.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:17 smithi154 conmon[34553]: cluster 2021-07-31T02:48:16.673221+0000 mgr.x (mgr.24542) 2613 : cluster [DBG] pgmap v3839: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:18.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:17 smithi154 conmon[34553]: audit 2021-07-31T02:48:17.077235+0000 mgr.x (mgr.24542) 2614 : audit [DBG] from='client.28114 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:48:18.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:17 smithi154 conmon[34553]: audit 2021-07-31T02:48:17.358529+0000 mgr.x (mgr.24542) 2615 : audit [DBG] from='client.28120 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:48:18.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:17 smithi154 conmon[34553]: audit 2021-07-31T02:48:17.702917 2021-07-31T02:48:18.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:17 smithi154 conmon[34553]: +0000 mon.c (mon.1) 147 : audit 2021-07-31T02:48:18.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:17 smithi154 conmon[34553]: [DBG] from='client.? 172.21.15.45:0/3703106139' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:48:18.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:17 smithi045 conmon[33655]: cluster 2021-07-31T02:48:16.673221+0000 mgr.x (mgr.24542 2021-07-31T02:48:18.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:17 smithi045 conmon[33655]: ) 2613 : cluster [DBG] pgmap v3839: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:18.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:17 smithi045 conmon[33655]: audit 2021-07-31T02:48:17. 2021-07-31T02:48:18.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:17 smithi045 conmon[33655]: 077235+0000 mgr.x (mgr.24542) 2614 : audit [DBG] from='client.28114 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:48:18.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:17 smithi045 conmon[33655]: audit 2021 2021-07-31T02:48:18.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:17 smithi045 conmon[33655]: -07-31T02:48:17. 2021-07-31T02:48:18.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:17 smithi045 conmon[33655]: 358529+0000 mgr.x (mgr. 2021-07-31T02:48:18.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:17 smithi045 conmon[33655]: 24542) 2615 : audit [DBG] 2021-07-31T02:48:18.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:17 smithi045 conmon[33655]: from='client.28120 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:48:18.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:17 smithi045 conmon[33655]: audit 2021-07- 2021-07-31T02:48:18.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:17 smithi045 conmon[33655]: 31T02:48:17.702917+0000 2021-07-31T02:48:18.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:17 smithi045 conmon[33655]: mon.c (mon.1) 147 2021-07-31T02:48:18.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:17 smithi045 conmon[33655]: : audit [DBG] from='client.? 172.21.15.45:0/3703106139' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:48:18.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:17 smithi045 conmon[39718]: cluster 2021-07-31T02:48:16.673221+0000 mgr.x (mgr.24542) 2613 : cluster [DBG] 2021-07-31T02:48:18.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:17 smithi045 conmon[39718]: pgmap v3839: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:18.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:17 smithi045 conmon[39718]: audit 2021-07 2021-07-31T02:48:18.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:17 smithi045 conmon[39718]: -31T02:48:17.077235+0000 2021-07-31T02:48:18.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:17 smithi045 conmon[39718]: mgr.x (mgr.24542) 2614 : 2021-07-31T02:48:18.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:17 smithi045 conmon[39718]: audit [DBG] from='client.28114 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:48:18.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:17 smithi045 conmon[39718]: audit 2021-07-31T02:48:18.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:17 smithi045 conmon[39718]: 2021-07 2021-07-31T02:48:18.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:17 smithi045 conmon[39718]: -31T02:48:17 2021-07-31T02:48:18.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:17 smithi045 conmon[39718]: .358529+0000 mgr.x (mgr.24542 2021-07-31T02:48:18.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:17 smithi045 conmon[39718]: ) 2615 : audit [DBG] from='client.28120 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:48:18.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:17 smithi045 conmon[39718]: 2021-07-31T02:48:18.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:17 smithi045 conmon[39718]: audit 2021-07-31T02:48:18.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:17 smithi045 conmon[39718]: 2021-07 2021-07-31T02:48:18.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:17 smithi045 conmon[39718]: -31T02:48:17. 2021-07-31T02:48:18.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:17 smithi045 conmon[39718]: 702917+0000 mon.c (mon.1 2021-07-31T02:48:18.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:17 smithi045 conmon[39718]: ) 147 : audit [DBG] from='client.? 172.21.15.45:0/3703106139' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:48:18.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:17 smithi045 conmon[39718]: 2021-07-31T02:48:20.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:19 smithi154 conmon[34553]: cluster 2021-07-31T02:48:18.673742+0000 mgr.x (mgr.24542) 2616 : cluster [DBG] pgmap v3840: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:20.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:19 smithi154 conmon[34553]: 2021-07-31T02:48:20.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:19 smithi045 conmon[33655]: cluster 2021-07-31T02:48:18.673742+0000 mgr.x (mgr.24542) 2616 : 2021-07-31T02:48:20.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:19 smithi045 conmon[33655]: cluster [DBG] pgmap v3840: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:20.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:19 smithi045 conmon[39718]: cluster 2021-07-31T02:48:18.673742+0000 mgr.x (mgr.24542) 2616 : 2021-07-31T02:48:20.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:19 smithi045 conmon[39718]: cluster [DBG] pgmap v3840: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:22.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:21 smithi154 conmon[34553]: cluster 2021-07-31T02:48:20.674173+0000 mgr.x (mgr.24542) 2617 : cluster [DBG] pgmap v3841: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:22.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:21 smithi154 conmon[34553]: 2021-07-31T02:48:22.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:21 smithi045 conmon[39718]: cluster 2021-07-31T02:48:20.674173+0000 mgr.x (mgr.24542) 2617 : cluster 2021-07-31T02:48:22.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:21 smithi045 conmon[39718]: [DBG] pgmap v3841: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:22.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:21 smithi045 conmon[33655]: cluster 2021-07-31T02:48:20.674173+0000 mgr.x (mgr.24542) 2617 2021-07-31T02:48:22.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:21 smithi045 conmon[33655]: : cluster [DBG] pgmap v3841: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:24.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:23 smithi154 conmon[34553]: cluster 2021-07-31T02:48:22.674555+0000 mgr.x (mgr.24542) 2618 : cluster [DBG] pgmap v3842: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:24.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:23 smithi154 conmon[34553]: 2021-07-31T02:48:24.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:23 smithi045 conmon[33655]: cluster 2021-07-31T02:48:22.674555+0000 mgr.x (mgr.24542) 2618 : cluster [DBG] pgmap v3842: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:24.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:23 smithi045 conmon[39718]: cluster 2021-07-31T02:48:22.674555+0000 2021-07-31T02:48:24.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:23 smithi045 conmon[39718]: mgr.x (mgr.24542) 2618 : cluster [DBG] pgmap v3842: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:26.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:25 smithi154 conmon[34553]: cluster 2021-07-31T02:48:24.675014+0000 mgr.x (mgr.24542) 2619 : cluster [DBG] pgmap v3843: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:26.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:25 smithi045 conmon[33655]: cluster 2021-07-31T02:48:24.675014+0000 mgr.x (mgr.24542) 2619 : cluster [DBG] pgmap v3843: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:26.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:25 smithi045 conmon[39718]: cluster 2021-07-31T02:48:24.675014+0000 mgr.x (mgr.24542 2021-07-31T02:48:26.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:25 smithi045 conmon[39718]: ) 2619 : cluster [DBG] pgmap v3843: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:28.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:27 smithi154 conmon[34553]: cluster 2021-07-31T02:48:26.675556+0000 mgr.x (mgr.24542) 2620 : cluster [DBG] pgmap v3844: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:28.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:27 smithi154 conmon[34553]: 2021-07-31T02:48:28.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:27 smithi045 conmon[33655]: cluster 2021-07-31T02:48:26.675556+0000 mgr.x (mgr.24542) 2620 : cluster [DBG] pgmap v3844: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:28.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:27 smithi045 conmon[39718]: cluster 2021-07-31T02:48:26.675556+0000 2021-07-31T02:48:28.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:27 smithi045 conmon[39718]: mgr.x (mgr.24542) 2620 : cluster [DBG] pgmap v3844: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:29.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:28 smithi154 conmon[34553]: audit 2021-07-31T02:48:28.181338+0000 mon.b (mon.2) 380 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:48:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:28 smithi154 conmon[34553]: 2021-07-31T02:48:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:28 smithi154 conmon[34553]: audit 2021-07-31T02:48:28.181643+0000 mon.b (mon.2) 381 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:48:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:28 smithi154 conmon[34553]: audit 2021-07-31T02:48:28.181809 2021-07-31T02:48:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:28 smithi154 conmon[34553]: +0000 mon.a (mon.0) 3207 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:48:29.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:28 smithi154 conmon[34553]: audit 2021-07-31T02:48:29.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:28 smithi154 conmon[34553]: 2021-07-31T02:48: 2021-07-31T02:48:29.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:28 smithi154 conmon[34553]: 28.182234+0000 mon.a (mon.0) 2021-07-31T02:48:29.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:28 smithi154 conmon[34553]: 3208 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:48:29.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:28 smithi045 conmon[39718]: audit 2021-07-31T02:48:28.181338+0000 mon.b (mon.2) 2021-07-31T02:48:29.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:28 smithi045 conmon[39718]: 380 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:48:29.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:28 smithi045 conmon[39718]: audit 2021-07-31T02:48:28.181643+0000 mon.b ( 2021-07-31T02:48:29.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:28 smithi045 conmon[39718]: mon.2) 381 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:48:29.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:28 smithi045 conmon[39718]: audit 2021-07-31T02:48:29.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:28 smithi045 conmon[39718]: 2021-07-31T02:48:28.181809+0000 2021-07-31T02:48:29.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:28 smithi045 conmon[39718]: mon.a (mon.0) 3207 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:48:29.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:28 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T02:48:29.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:28 smithi045 conmon[39718]: 02:48:28.182234+0000 mon.a (mon.0) 3208 2021-07-31T02:48:29.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:28 smithi045 conmon[39718]: : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:48:29.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:28 smithi045 conmon[33655]: audit 2021-07-31T02:48:28.181338+0000 mon.b 2021-07-31T02:48:29.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:28 smithi045 conmon[33655]: (mon.2) 380 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:48:29.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:28 smithi045 conmon[33655]: audit 2021-07-31T02:48:28.181643+0000 mon.b (mon.2) 381 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:48:29.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:28 smithi045 conmon[33655]: audit 2021-07-31T02:48:28.181809+0000 mon.a (mon.0) 2021-07-31T02:48:29.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:28 smithi045 conmon[33655]: 3207 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:48:29.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:28 smithi045 conmon[33655]: audit 2021-07-31 2021-07-31T02:48:29.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:28 smithi045 conmon[33655]: T02:48:28.182234+0000 mon.a (mon.0) 3208 : 2021-07-31T02:48:29.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:28 smithi045 conmon[33655]: audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:48:30.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[34553]: cluster 2021-07-31T02:48:28.675960+0000 mgr.x (mgr.24542) 2621 : cluster [DBG] pgmap v3845: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:30.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[34553]: audit 2021-07-31 2021-07-31T02:48:30.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[34553]: T02:48:29.591093+0000 mon.b (mon.2) 382 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:48:30.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[34553]: audit 2021-07-31T02:48:29.592520+0000 mon.b (mon.2) 383 : 2021-07-31T02:48:30.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[34553]: audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:48:30.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[34553]: audit 2021-07-31T02:48:30.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[34553]: 2021-07-31T02:48 2021-07-31T02:48:30.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[34553]: :29.593599+0000 mon.b (mon 2021-07-31T02:48:30.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[34553]: .2) 384 : audit [INF] 2021-07-31T02:48:30.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[34553]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:48:30.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: debug 2021-07-31T02:48:29.636+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:48:30.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:48:30.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:48:30.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: yield 2021-07-31T02:48:30.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:48:30.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:48:30.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:48:30.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:48:30.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:48:30.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:48:30.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:48:30.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:48:30.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:48:30.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: task.result() 2021-07-31T02:48:30.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:48:30.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: debug 2021-07-31T02:48:29.648+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:48:30.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:48:30.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:48:30.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: yield 2021-07-31T02:48:30.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:48:30.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:48:30.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:48:30.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:48:30.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:48:30.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:48:30.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:48:30.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:48:30.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:48:30.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: task.result() 2021-07-31T02:48:30.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:48:30.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: debug 2021-07-31T02:48:29.672+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:48:30.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:48:30.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:48:30.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: yield 2021-07-31T02:48:30.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:48:30.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:48:30.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:48:30.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:48:30.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:48:30.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:48:30.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:48:30.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:48:30.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:48:30.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: task.result() 2021-07-31T02:48:30.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:48:30.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: debug 2021-07-31T02:48:29.673+0000 7fcb93160700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:48:30.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:48:30.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:48:30.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: yield 2021-07-31T02:48:30.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:48:30.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:48:30.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:48:30.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:48:30.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:48:30.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:48:30.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:48:30.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:48:30.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:48:30.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: task.result() 2021-07-31T02:48:30.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:48:30.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: 2021-07-31T02:48:30.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:48:30.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: 2021-07-31T02:48:30.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:48:30.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:48:30.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:48:30.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:48:30.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:48:30.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:48:30.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:48:30.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:48:30.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:48:30.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:48:30.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:48:30.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:48:30.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:48:30.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:48:30.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:48:30.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:48:30.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:48:30.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:48:30.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: raise 2021-07-31T02:48:30.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:48:30.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:48:30.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:48:30.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:48:30.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:48:30.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: debug 2021-07-31T02:48:29.682+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:48:30.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:48:30.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:48:30.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: yield 2021-07-31T02:48:30.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:48:30.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:48:30.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:48:30.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:48:30.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:48:30.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:48:30.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:48:30.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:48:30.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:48:30.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: task.result() 2021-07-31T02:48:30.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:48:30.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: debug 2021-07-31T02:48:29.682+0000 7fcb9295f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:48:30.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:48:30.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:48:30.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: yield 2021-07-31T02:48:30.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:48:30.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:48:30.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:48:30.035 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:48:30.036 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:48:30.036 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:48:30.036 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:48:30.036 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:48:30.037 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:48:30.037 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: task.result() 2021-07-31T02:48:30.037 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:48:30.037 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: 2021-07-31T02:48:30.038 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:48:30.038 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: 2021-07-31T02:48:30.038 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:48:30.038 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:48:30.039 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:48:30.039 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:48:30.039 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:48:30.039 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:48:30.039 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:48:30.040 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:48:30.040 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:48:30.040 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:48:30.040 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:48:30.041 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:48:30.041 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:48:30.041 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:48:30.041 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:48:30.041 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:48:30.042 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:48:30.042 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:48:30.042 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: raise 2021-07-31T02:48:30.042 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:48:30.043 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:48:30.043 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:48:30.043 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:48:30.043 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:48:29 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:48:30.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:29 smithi045 conmon[39718]: cluster 2021-07-31T02:48:28.675960+0000 mgr.x (mgr.24542) 2621 : 2021-07-31T02:48:30.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:29 smithi045 conmon[39718]: cluster [DBG] pgmap v3845: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:30.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:29 smithi045 conmon[39718]: audit 2021-07-31T02:48:29.591093+0000 mon.b ( 2021-07-31T02:48:30.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:29 smithi045 conmon[39718]: mon.2) 382 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:48:30.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:29 smithi045 conmon[39718]: audit 2021-07-31T02:48:30.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:29 smithi045 conmon[39718]: 2021-07-31T02:48:29.592520+0000 mon.b (mon. 2021-07-31T02:48:30.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:29 smithi045 conmon[39718]: 2) 383 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:48:30.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:29 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T02:48:30.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:29 smithi045 conmon[39718]: T02:48:29.593599+0000 mon.b (mon. 2021-07-31T02:48:30.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:29 smithi045 conmon[39718]: 2) 384 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:48:30.047 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:29 smithi045 conmon[33655]: cluster 2021-07-31T02: 2021-07-31T02:48:30.047 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:29 smithi045 conmon[33655]: 48:28.675960+0000 mgr.x (mgr.24542) 2621 : cluster [DBG] pgmap v3845: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:30.048 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:29 smithi045 conmon[33655]: audit 2021-07-31T02:48:30.049 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:29 smithi045 conmon[33655]: 2021-07-31T02:48:29.591093+0000 mon.b (mon.2) 382 : audit [DBG] 2021-07-31T02:48:30.049 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:29 smithi045 conmon[33655]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:48:30.049 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:29 smithi045 conmon[33655]: audit 2021-07-31T02:48:29.592520+0000 mon.b ( 2021-07-31T02:48:30.049 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:29 smithi045 conmon[33655]: mon.2) 383 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:48:30.050 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:29 smithi045 conmon[33655]: audit 2021-07-31T02:48: 2021-07-31T02:48:30.050 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:29 smithi045 conmon[33655]: 29.593599+0000 mon.b (mon.2) 384 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:48:31.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:48:29.594370+0000 mgr.x (mgr.24542) 2622 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:48:31.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:48:29.637506+0000 mgr.x (mgr.24542) 2623 : cephadm [ERR] 2021-07-31T02:48:31.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:48:31.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:48:31.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: yield 2021-07-31T02:48:31.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:48:31.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:48:31.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:48:31.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:48:31.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:48:31.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:48:31.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:48:31.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:48:31.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:48:31.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:48:31.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:48:31.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:48:29.637954+0000 2021-07-31T02:48:31.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: mgr.x (mgr.24542) 2624 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:48:31.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:48:29.648931+0000 mgr.x (mgr.24542) 2625 : cephadm [ERR] 2021-07-31T02:48:31.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:48:31.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:48:31.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: yield 2021-07-31T02:48:31.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:48:31.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:48:31.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:48:31.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:48:31.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:48:31.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:48:31.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:48:31.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:48:31.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:48:31.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:48:31.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:48:31.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:48:29.649777+0000 mgr.x (mgr.24542) 2626 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:48:31.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:48:29 2021-07-31T02:48:31.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: .673909+0000 mgr.x (mgr.24542) 2627 2021-07-31T02:48:31.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: : cephadm [ERR] 2021-07-31T02:48:31.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:48:31.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:48:31.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: yield 2021-07-31T02:48:31.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:48:31.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:48:31.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:48:31.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:48:31.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:48:31.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:48:31.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:48:31.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:48:31.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:48:31.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:48:31.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:48:31.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:48:29.674378+0000 mgr.x (mgr.24542) 2628 : 2021-07-31T02:48:31.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:48:31.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:48:31.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:48:31.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: yield 2021-07-31T02:48:31.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:48:31.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:48:31.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:48:31.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:48:31.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:48:31.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:48:31.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:48:31.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:48:31.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:48:31.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:48:31.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:48:31.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: 2021-07-31T02:48:31.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:48:31.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: 2021-07-31T02:48:31.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:48:31.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:48:31.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:48:31.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:48:31.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:48:31.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:48:31.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:48:31.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:48:31.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:48:31.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:48:31.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:48:31.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:48:31.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:48:31.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:48:31.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:48:31.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:48:31.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:48:31.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:48:31.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: raise 2021-07-31T02:48:31.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:48:31.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:48:31.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:48:31.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:48:31.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:48:31.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:48:29.683006+0000 mgr.x (mgr.24542) 2629 : cephadm [ERR] 2021-07-31T02:48:31.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:48:31.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:48:31.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: yield 2021-07-31T02:48:31.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:48:31.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:48:31.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:48:31.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:48:31.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:48:31.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:48:31.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:48:31.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:48:31.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:48:31.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:48:31.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:48:31.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:48:29.683806+0000 mgr.x (mgr.24542) 2630 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:48:31.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:48:31.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:48:31.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: yield 2021-07-31T02:48:31.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:48:31.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:48:31.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:48:31.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:48:31.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:48:31.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:48:31.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:48:31.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:48:31.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:48:31.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:48:31.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:48:31.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: 2021-07-31T02:48:31.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:48:31.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: 2021-07-31T02:48:31.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:48:31.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:48:31.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:48:31.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:48:31.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:48:31.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:48:31.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:48:31.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:48:31.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:48:31.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:48:31.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:48:31.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:48:31.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:48:31.035 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:48:31.035 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:48:31.035 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:48:31.035 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:48:31.036 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:48:31.036 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: raise 2021-07-31T02:48:31.036 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:48:31.036 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:48:31.037 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:48:31.037 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:48:31.037 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:30 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:48:31.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:48:29.594370+0000 2021-07-31T02:48:31.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: mgr.x (mgr.24542) 2622 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:48:31.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:48:29.637506+0000 mgr.x (mgr.24542) 2623 : cephadm [ERR] 2021-07-31T02:48:31.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:48:31.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:48:31.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: yield 2021-07-31T02:48:31.040 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:48:31.040 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:48:31.040 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:48:31.040 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:48:31.040 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:48:31.041 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:48:31.041 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:48:31.041 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:48:31.041 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:48:31.042 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:48:31.042 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:48:31.042 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:48:29.637954+0000 mgr.x (mgr.24542) 2624 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:48:31.042 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:48:29. 2021-07-31T02:48:31.042 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: 648931+0000 mgr.x (mgr.24542) 2625 : cephadm [ERR] 2021-07-31T02:48:31.043 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:48:31.043 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:48:31.043 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: yield 2021-07-31T02:48:31.043 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:48:31.044 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:48:31.044 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:48:31.044 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:48:31.044 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:48:31.044 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:48:31.045 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:48:31.045 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:48:31.045 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:48:31.045 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:48:31.046 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:48:31.046 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:48:29.649777+0000 mgr.x (mgr.24542) 2626 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:48:31.046 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:48:29.673909+0000 mgr.x (mgr.24542) 2627 : cephadm [ERR] 2021-07-31T02:48:31.046 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:48:31.046 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:48:31.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:48:29.594370+0000 mgr.x 2021-07-31T02:48:31.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: (mgr.24542) 2622 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:48:31.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:48:31.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: 2021-07-31T02:48:29.637506+0000 mgr.x (mgr. 2021-07-31T02:48:31.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: 24542) 2623 : cephadm [ERR] 2021-07-31T02:48:31.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:48:31.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:48:31.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: yield 2021-07-31T02:48:31.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:48:31.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:48:31.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:48:31.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:48:31.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:48:31.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:48:31.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:48:31.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:48:31.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:48:31.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:48:31.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:48:31.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:48:29.637954+0000 mgr.x (mgr.24542) 2624 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:48:31.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:48:29.648931+0000 mgr.x (mgr.24542) 2625 : cephadm [ERR] 2021-07-31T02:48:31.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:48:31.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:48:31.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: yield 2021-07-31T02:48:31.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:48:31.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:48:31.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:48:31.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:48:31.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:48:31.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:48:31.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:48:31.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:48:31.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:48:31.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:48:31.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:48:31.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:48:29.649777+0000 mgr.x (mgr.24542) 2626 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:48:31.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:48:29.673909+0000 mgr.x (mgr 2021-07-31T02:48:31.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: .24542) 2627 : cephadm [ERR] 2021-07-31T02:48:31.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:48:31.056 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: yield 2021-07-31T02:48:31.056 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:48:31.056 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:48:31.057 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:48:31.057 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:48:31.057 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:48:31.057 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:48:31.057 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:48:31.058 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:48:31.058 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:48:31.058 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:48:31.058 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:48:31.059 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:48:29.674378+0000 mgr.x (mgr.24542) 2628 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:48:31.059 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:48:31.059 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:48:31.059 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: yield 2021-07-31T02:48:31.059 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:48:31.060 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:48:31.060 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:48:31.060 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:48:31.060 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:48:31.060 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:48:31.061 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:48:31.061 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:48:31.061 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:48:31.061 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:48:31.062 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:48:31.062 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: 2021-07-31T02:48:31.062 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:48:31.062 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: 2021-07-31T02:48:31.062 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:48:31.063 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:48:31.063 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:48:31.063 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:48:31.063 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:48:31.064 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:48:31.064 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:48:31.064 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:48:31.064 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:48:31.064 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:48:31.065 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:48:31.065 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:48:31.065 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: yield 2021-07-31T02:48:31.066 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:48:31.066 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:48:31.066 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:48:31.066 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:48:31.067 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:48:31.067 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:48:31.067 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:48:31.067 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:48:31.067 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:48:31.068 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:48:31.068 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:48:31.068 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:48:29.674378+0000 mgr.x (mgr.24542) 2628 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:48:31.068 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:48:31.069 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:48:31.069 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: yield 2021-07-31T02:48:31.069 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:48:31.069 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:48:31.069 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:48:31.070 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:48:31.070 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:48:31.070 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:48:31.070 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:48:31.070 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:48:31.071 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:48:31.071 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:48:31.071 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:48:31.071 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: 2021-07-31T02:48:31.072 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:48:31.072 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: 2021-07-31T02:48:31.072 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:48:31.072 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:48:31.073 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:48:31.073 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:48:31.073 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:48:31.073 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:48:31.073 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:48:31.074 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:48:31.074 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:48:31.074 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:48:31.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:48:31.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:48:31.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:48:31.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:48:31.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:48:31.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:48:31.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:48:31.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: raise 2021-07-31T02:48:31.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:48:31.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:48:31.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:48:31.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:48:31.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:48:31.078 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:48:29.683006+0000 mgr.x (mgr.24542) 2629 : cephadm [ERR] 2021-07-31T02:48:31.078 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:48:31.078 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:48:31.078 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: yield 2021-07-31T02:48:31.079 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:48:31.079 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:48:31.079 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:48:31.079 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:48:31.079 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:48:31.080 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:48:31.080 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:48:31.080 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:48:31.080 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:48:31.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:48:31.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:48:31.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:48:29.683806+0000 mgr.x (mgr.24542) 2630 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:48:31.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:48:31.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:48:31.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: yield 2021-07-31T02:48:31.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:48:31.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:48:31.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:48:31.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:48:31.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:48:31.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:48:31.083 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:48:31.084 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:48:31.084 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:48:31.084 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:48:31.084 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:48:31.085 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:48:31.085 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:48:31.085 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:48:31.085 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: raise 2021-07-31T02:48:31.086 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:48:31.086 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:48:31.086 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:48:31.086 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:48:31.086 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:48:31.087 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:48:29.683006+0000 mgr.x (mgr.24542) 2629 2021-07-31T02:48:31.087 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: : cephadm [ERR] 2021-07-31T02:48:31.087 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:48:31.087 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:48:31.088 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: yield 2021-07-31T02:48:31.088 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:48:31.088 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:48:31.088 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:48:31.088 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:48:31.089 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:48:31.089 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:48:31.089 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:48:31.089 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:48:31.089 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:48:31.090 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:48:31.090 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:48:31.090 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:48:29.683806+0000 mgr.x (mgr.24542) 2630 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:48:31.090 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:48:31.091 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:48:31.091 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: yield 2021-07-31T02:48:31.091 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:48:31.091 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:48:31.091 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:48:31.092 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:48:31.092 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:48:31.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:48:31.093 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:48:31.093 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:48:31.093 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:48:31.093 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:48:31.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: 2021-07-31T02:48:31.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:48:31.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: 2021-07-31T02:48:31.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:48:31.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:48:31.095 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:48:31.095 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:48:31.095 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:48:31.095 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:48:31.096 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:48:31.096 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:48:31.096 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:48:31.096 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:48:31.097 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:48:31.097 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:48:31.097 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:48:31.097 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:48:31.097 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:48:31.098 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:48:31.098 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:48:31.098 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:48:31.098 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: raise 2021-07-31T02:48:31.098 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:48:31.099 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:48:31.099 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:48:31.099 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:48:31.099 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:48:31.100 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:48:31.100 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:48:31.100 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:48:31.100 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:48:31.101 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:48:31.101 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:48:31.101 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: 2021-07-31T02:48:31.101 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:48:31.101 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: 2021-07-31T02:48:31.102 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:48:31.102 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:48:31.102 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:48:31.102 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:48:31.103 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:48:31.103 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:48:31.103 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:48:31.103 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:48:31.103 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:48:31.104 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:48:31.104 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:48:31.104 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:48:31.104 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:48:31.104 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:48:31.105 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:48:31.105 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:48:31.105 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:48:31.105 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:48:31.106 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: raise 2021-07-31T02:48:31.106 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:48:31.106 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:48:31.106 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:48:31.106 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:48:31.107 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:30 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:48:32.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:31 smithi154 conmon[34553]: cluster 2021-07-31T02:48:30.676416+0000 mgr.x (mgr.24542) 2631 : cluster [DBG] pgmap v3846: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:32.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:31 smithi045 conmon[39718]: cluster 2021-07-31T02:48:30.676416+0000 mgr.x ( 2021-07-31T02:48:32.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:31 smithi045 conmon[39718]: mgr.24542) 2631 : cluster [DBG] pgmap v3846: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:32.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:31 smithi045 conmon[33655]: cluster 2021-07-31T02:48:30.676416 2021-07-31T02:48:32.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:31 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 2631 : cluster [DBG] pgmap v3846: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:34.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:33 smithi154 conmon[34553]: cluster 2021-07-31T02:48:32.676881+0000 mgr.x (mgr.24542) 2632 : cluster [DBG] pgmap v3847: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:34.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:33 smithi045 conmon[39718]: cluster 2021-07-31T02:48:32.676881+0000 mgr.x (mgr.24542 2021-07-31T02:48:34.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:33 smithi045 conmon[39718]: ) 2632 : cluster [DBG] pgmap v3847: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:34.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:33 smithi045 conmon[33655]: cluster 2021-07-31T02:48:32.676881+0000 mgr.x (mgr.24542) 2632 2021-07-31T02:48:34.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:33 smithi045 conmon[33655]: : cluster [DBG] pgmap v3847: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:36.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:35 smithi154 conmon[34553]: cluster 2021-07-31T02:48:34.677295+0000 mgr.x (mgr.24542) 2633 : cluster [DBG] pgmap v3848: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:36.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:35 smithi045 conmon[33655]: cluster 2021-07-31T02:48:34.677295+0000 mgr.x (mgr.24542) 2633 : cluster 2021-07-31T02:48:36.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:35 smithi045 conmon[33655]: [DBG] pgmap v3848: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:36.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:35 smithi045 conmon[39718]: cluster 2021-07-31T02:48:34.677295+0000 mgr.x ( 2021-07-31T02:48:36.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:35 smithi045 conmon[39718]: mgr.24542) 2633 : cluster [DBG] pgmap v3848: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:38.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:37 smithi154 conmon[34553]: cluster 2021-07-31T02:48:36.677694+0000 mgr.x (mgr.24542) 2634 : cluster [DBG] pgmap v3849: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:38.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:37 smithi045 conmon[33655]: cluster 2021-07-31T02:48:36.677694+0000 mgr.x (mgr. 2021-07-31T02:48:38.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:37 smithi045 conmon[33655]: 24542) 2634 : cluster [DBG] pgmap v3849: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:38.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:37 smithi045 conmon[39718]: cluster 2021-07-31T02:48:36 2021-07-31T02:48:38.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:37 smithi045 conmon[39718]: .677694+0000 mgr.x (mgr.24542) 2634 : cluster [DBG] pgmap v3849: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:40.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:39 smithi154 conmon[34553]: cluster 2021-07-31T02:48:38.678153+0000 mgr.x (mgr.24542) 2635 : cluster [DBG] pgmap v3850: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:40.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:39 smithi045 conmon[33655]: cluster 2021-07-31T02:48:38.678153+0000 mgr.x (mgr.24542) 2021-07-31T02:48:40.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:39 smithi045 conmon[33655]: 2635 : cluster [DBG] pgmap v3850: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:40.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:39 smithi045 conmon[39718]: cluster 2021-07-31T02:48:38.678153+0000 mgr.x ( 2021-07-31T02:48:40.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:39 smithi045 conmon[39718]: mgr.24542) 2635 : cluster [DBG] pgmap v3850: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:42.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:41 smithi154 conmon[34553]: cluster 2021-07-31T02:48:40.678738+0000 mgr.x (mgr.24542) 2636 : cluster [DBG] pgmap v3851: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:42.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:41 smithi045 conmon[33655]: cluster 2021-07-31T02:48:40.678738+0000 mgr.x (mgr.24542) 2636 : cluster [DBG] 2021-07-31T02:48:42.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:41 smithi045 conmon[33655]: pgmap v3851: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:42.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:41 smithi045 conmon[39718]: cluster 2021-07-31T02:48:40.678738+0000 mgr.x (mgr.24542 2021-07-31T02:48:42.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:41 smithi045 conmon[39718]: ) 2636 : cluster [DBG] pgmap v3851: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:44.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:43 smithi154 conmon[34553]: cluster 2021-07-31T02:48:42.679231+0000 mgr.x (mgr.24542) 2637 : cluster [DBG] pgmap v3852: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:44.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:43 smithi045 conmon[33655]: cluster 2021-07-31T02:48:42.679231+0000 mgr.x (mgr. 2021-07-31T02:48:44.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:43 smithi045 conmon[33655]: 24542) 2637 : cluster [DBG] pgmap v3852: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:44.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:43 smithi045 conmon[39718]: cluster 2021-07-31T02:48:42.679231+0000 mgr.x (mgr.24542) 2637 : cluster [DBG] pgmap v3852: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:46.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:45 smithi154 conmon[34553]: cluster 2021-07-31T02:48:44.679661+0000 mgr.x (mgr.24542) 2638 : cluster [DBG] pgmap v3853: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:46.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:45 smithi045 conmon[39718]: cluster 2021-07-31T02:48:44.679661+0000 2021-07-31T02:48:46.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:45 smithi045 conmon[39718]: mgr.x (mgr.24542) 2638 : cluster [DBG] pgmap v3853: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:46.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:45 smithi045 conmon[33655]: cluster 2021-07-31T02:48:44.679661+0000 mgr.x (mgr.24542) 2638 2021-07-31T02:48:46.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:45 smithi045 conmon[33655]: : cluster [DBG] pgmap v3853: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:48.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:47 smithi154 conmon[34553]: cluster 2021-07-31T02:48:46.680173+0000 mgr.x (mgr.24542) 2639 : cluster [DBG] pgmap v3854: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:48.019 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:48:48.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:47 smithi045 conmon[33655]: cluster 2021-07-31T02:48:46.680173 2021-07-31T02:48:48.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:47 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 2639 : cluster [DBG] pgmap v3854: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:48.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:47 smithi045 conmon[39718]: cluster 2021-07-31T02:48:46.680173 2021-07-31T02:48:48.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:47 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 2639 : cluster [DBG] pgmap v3854: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:48.295 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:48:48.296 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 63m ago 64m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:48:48.296 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 61m ago 64m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:48:48.296 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 61m ago 67m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:48:48.296 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 63m ago 69m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:48:48.297 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 63m ago 69m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:48:48.297 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 61m ago 67m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:48:48.297 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 63m ago 68m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:48:48.297 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 63m ago 64m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:48:48.298 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 61m ago 64m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:48:48.298 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 63m ago 67m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:48:48.298 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 63m ago 67m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:48:48.298 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 63m ago 66m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:48:48.298 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 63m ago 66m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:48:48.299 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 61m ago 66m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:48:48.299 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 61m ago 65m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:48:48.299 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 61m ago 65m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:48:48.299 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 61m ago 65m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:48:48.300 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 61m ago 64m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:48:48.631 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:48:48.632 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:48:48.632 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:48:48.632 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:48:48.632 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:48:48.633 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:48:48.633 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:48:48.633 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:48:48.633 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:48:48.634 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:48:48.634 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:48:48.634 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:48:48.634 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:48:48.634 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:48:48.635 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:48:48.635 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:48:48.635 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:48:49.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:48 smithi154 conmon[34553]: audit 2021-07-31T02:48:48.007348+0000 mgr.x (mgr.24542) 2640 : audit [DBG] from='client.18192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:48:49.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:48 smithi154 conmon[34553]: audit 2021-07-31T02:48:48.287592+0000 mgr.x (mgr.24542) 2641 : audit [DBG] from='client.28138 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:48:49.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:48 smithi154 conmon[34553]: audit 2021-07-31T02:48:48.629654+0000 mon.c (mon.1) 148 : audit [DBG] from='client.? 172.21.15.45:0/1902792647' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:48:49.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:48 smithi045 conmon[33655]: audit 2021-07-31T02:48:48.007348+0000 mgr.x 2021-07-31T02:48:49.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:48 smithi045 conmon[33655]: (mgr.24542) 2640 : audit [DBG] from='client.18192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:48:49.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:48 smithi045 conmon[33655]: audit 2021-07-31T02:48:48.287592+0000 mgr.x (mgr. 2021-07-31T02:48:49.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:48 smithi045 conmon[33655]: 24542) 2641 : audit [DBG] from='client.28138 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:48:49.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:48 smithi045 conmon[33655]: audit 2021-07-31T02:48 2021-07-31T02:48:49.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:48 smithi045 conmon[33655]: :48.629654+0000 mon.c (mon.1) 148 : audit [DBG] from='client.? 172.21.15.45:0/1902792647' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:48:49.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:48 smithi045 conmon[33655]: 2021-07-31T02:48:49.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:48 smithi045 conmon[39718]: audit 2021-07-31T02:48:48.007348 2021-07-31T02:48:49.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:48 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 2640 : audit [DBG] from='client.18192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:48:49.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:48 smithi045 conmon[39718]: audit 2021-07-31T02:48:49.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:48 smithi045 conmon[39718]: 2021-07-31T02:48:48.287592+0000 mgr.x 2021-07-31T02:48:49.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:48 smithi045 conmon[39718]: (mgr.24542) 2641 : audit [DBG] 2021-07-31T02:48:49.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:48 smithi045 conmon[39718]: from='client.28138 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:48:49.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:48 smithi045 conmon[39718]: audit 2021-07-31T02:48: 2021-07-31T02:48:49.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:48 smithi045 conmon[39718]: 48.629654+0000 mon.c (mon.1) 148 2021-07-31T02:48:49.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:48 smithi045 conmon[39718]: : audit [DBG] from='client.? 172.21.15.45:0/1902792647' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:48:50.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:49 smithi154 conmon[34553]: cluster 2021-07-31T02:48:48.680568+0000 mgr.x (mgr.24542) 2642 : cluster [DBG] pgmap v3855: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:50.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:49 smithi045 conmon[39718]: cluster 2021-07-31T02:48: 2021-07-31T02:48:50.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:49 smithi045 conmon[39718]: 48.680568+0000 mgr.x (mgr.24542) 2642 : cluster [DBG] pgmap v3855: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:50.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:49 smithi045 conmon[33655]: cluster 2021-07-31T02:48:48.680568+0000 mgr.x (mgr.24542) 2642 : cluster 2021-07-31T02:48:50.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:49 smithi045 conmon[33655]: [DBG] pgmap v3855: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:52.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:51 smithi154 conmon[34553]: cluster 2021-07-31T02:48:50.681076+0000 mgr.x (mgr.24542) 2643 : cluster [DBG] pgmap v3856: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:52.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:51 smithi045 conmon[39718]: cluster 2021-07-31T02:48:50 2021-07-31T02:48:52.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:51 smithi045 conmon[39718]: .681076+0000 mgr.x (mgr.24542) 2643 : cluster [DBG] pgmap v3856: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:52.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:51 smithi045 conmon[33655]: cluster 2021-07-31T02:48:50.681076+0000 mgr.x (mgr.24542) 2643 : cluster [DBG] pgmap v3856: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:54.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:53 smithi154 conmon[34553]: cluster 2021-07-31T02:48:52.681614+0000 mgr.x (mgr.24542) 2644 : cluster [DBG] 2021-07-31T02:48:54.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:53 smithi154 conmon[34553]: pgmap v3857: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:54.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:53 smithi045 conmon[39718]: cluster 2021-07-31T02:48:52.681614+0000 mgr.x (mgr. 2021-07-31T02:48:54.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:53 smithi045 conmon[39718]: 24542) 2644 : cluster [DBG] pgmap v3857: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:54.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:53 smithi045 conmon[33655]: cluster 2021-07-31T02:48: 2021-07-31T02:48:54.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:53 smithi045 conmon[33655]: 52.681614+0000 mgr.x (mgr.24542) 2644 : cluster [DBG] pgmap v3857: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:56.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:55 smithi154 conmon[34553]: cluster 2021-07-31T02:48:54.682100+0000 mgr.x (mgr.24542) 2645 : cluster [DBG] pgmap v3858: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:56.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:55 smithi045 conmon[39718]: cluster 2021-07-31T02:48:54.682100+0000 mgr.x (mgr. 2021-07-31T02:48:56.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:55 smithi045 conmon[39718]: 24542) 2645 : cluster [DBG] pgmap v3858: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:56.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:55 smithi045 conmon[33655]: cluster 2021 2021-07-31T02:48:56.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:55 smithi045 conmon[33655]: -07-31T02:48:54.682100+0000 mgr.x (mgr.24542) 2645 : cluster [DBG] pgmap v3858: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:58.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:57 smithi154 conmon[34553]: cluster 2021-07-31T02:48:56.682471+0000 mgr.x (mgr.24542) 2646 : cluster [DBG] pgmap v3859: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:58.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:57 smithi045 conmon[33655]: cluster 2021-07-31T02:48:56.682471+0000 mgr.x (mgr.24542) 2646 : 2021-07-31T02:48:58.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:57 smithi045 conmon[33655]: cluster [DBG] pgmap v3859: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:48:58.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:57 smithi045 conmon[39718]: cluster 2021-07-31T02:48:56 2021-07-31T02:48:58.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:57 smithi045 conmon[39718]: .682471+0000 mgr.x (mgr.24542) 2646 : cluster [DBG] pgmap v3859: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:00.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:48:59 smithi154 conmon[34553]: cluster 2021-07-31T02:48:58.683082+0000 mgr.x (mgr.24542) 2647 : cluster [DBG] pgmap v3860: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:00.052 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:59 smithi045 conmon[33655]: cluster 2021-07-31T02:48:58.683082+0000 mgr.x (mgr.24542) 2647 : cluster [DBG] 2021-07-31T02:49:00.052 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:48:59 smithi045 conmon[33655]: pgmap v3860: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:00.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:59 smithi045 conmon[39718]: cluster 2021-07-31T02:48: 2021-07-31T02:49:00.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:48:59 smithi045 conmon[39718]: 58.683082+0000 mgr.x (mgr.24542) 2647 : cluster [DBG] pgmap v3860: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:02.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:01 smithi154 conmon[34553]: cluster 2021-07-31T02:49:00.683680+0000 mgr.x (mgr.24542) 2648 : cluster [DBG] pgmap v3861: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:02.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:01 smithi045 conmon[33655]: cluster 2021-07-31T02:49:00.683680+0000 mgr.x (mgr.24542) 2648 : cluster [DBG] pgmap v3861: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:02.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:01 smithi045 conmon[39718]: cluster 2021-07-31T02:49:00. 2021-07-31T02:49:02.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:01 smithi045 conmon[39718]: 683680+0000 mgr.x (mgr.24542) 2648 : cluster [DBG] pgmap v3861: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:04.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:03 smithi154 conmon[34553]: cluster 2021-07-31T02:49:02.684290+0000 mgr.x (mgr.24542) 2649 : cluster [DBG] pgmap v3862: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:04.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:03 smithi045 conmon[33655]: cluster 2021-07-31T02:49:02.684290+0000 mgr.x (mgr 2021-07-31T02:49:04.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:03 smithi045 conmon[33655]: .24542) 2649 : cluster [DBG] pgmap v3862: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:04.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:03 smithi045 conmon[39718]: cluster 2021-07-31T02:49:02.684290+0000 mgr.x (mgr.24542) 2021-07-31T02:49:04.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:03 smithi045 conmon[39718]: 2649 : cluster [DBG] pgmap v3862: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:06.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:05 smithi154 conmon[34553]: cluster 2021-07-31T02:49:04.684773+0000 mgr.x (mgr.24542) 2650 : cluster [DBG] 2021-07-31T02:49:06.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:05 smithi154 conmon[34553]: pgmap v3863: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:06.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:05 smithi045 conmon[33655]: cluster 2021-07-31T02:49:04.684773+0000 mgr.x (mgr.24542) 2650 : cluster 2021-07-31T02:49:06.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:05 smithi045 conmon[33655]: [DBG] pgmap v3863: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:06.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:05 smithi045 conmon[39718]: cluster 2021-07-31T02:49:04.684773+0000 2021-07-31T02:49:06.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:05 smithi045 conmon[39718]: mgr.x (mgr.24542) 2650 : cluster [DBG] pgmap v3863: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:08.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:07 smithi154 conmon[34553]: cluster 2021-07-31T02:49:06.685177+0000 mgr.x (mgr.24542) 2651 : cluster [DBG] pgmap v3864: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:08.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:07 smithi045 conmon[39718]: cluster 2021-07-31T02:49:06.685177+0000 mgr.x (mgr.24542) 2651 : cluster [DBG] pgmap v3864: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:08.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:07 smithi045 conmon[33655]: cluster 2021-07-31T02:49:06.685177+0000 mgr.x (mgr.24542) 2651 : cluster [DBG] pgmap v3864: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:10.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:09 smithi154 conmon[34553]: cluster 2021-07-31T02:49:08.685507+0000 mgr.x (mgr.24542) 2652 : cluster [DBG] pgmap v3865: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:10.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:09 smithi045 conmon[33655]: cluster 2021-07-31T02:49:08.685507+0000 mgr.x (mgr. 2021-07-31T02:49:10.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:09 smithi045 conmon[33655]: 24542) 2652 : cluster [DBG] pgmap v3865: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:10.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:09 smithi045 conmon[39718]: cluster 2021-07-31T02:49:08.685507+0000 mgr.x 2021-07-31T02:49:10.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:09 smithi045 conmon[39718]: (mgr.24542) 2652 : cluster [DBG] pgmap v3865: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:12.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:11 smithi154 conmon[34553]: cluster 2021-07-31T02:49:10.686085+0000 mgr.x (mgr.24542) 2653 : cluster [DBG] pgmap v3866: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:12.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:11 smithi045 conmon[33655]: cluster 2021-07-31T02:49:10.686085+0000 mgr.x (mgr.24542) 2653 : cluster [DBG] pgmap v3866: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:12.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:11 smithi045 conmon[39718]: cluster 2021-07-31T02:49:10.686085+0000 mgr.x 2021-07-31T02:49:12.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:11 smithi045 conmon[39718]: (mgr.24542) 2653 : cluster [DBG] pgmap v3866: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:14.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:13 smithi154 conmon[34553]: cluster 2021-07-31T02:49:12.686507+0000 mgr.x (mgr.24542) 2654 : cluster [DBG] pgmap v3867: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:14.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:13 smithi045 conmon[39718]: cluster 2021-07-31T02:49:12.686507+0000 mgr.x (mgr.24542) 2654 : cluster [DBG] pgmap v3867: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:14.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:13 smithi045 conmon[33655]: cluster 2021-07-31T02:49:12.686507+0000 mgr.x (mgr.24542) 2654 : 2021-07-31T02:49:14.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:13 smithi045 conmon[33655]: cluster [DBG] pgmap v3867: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:16.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:15 smithi154 conmon[34553]: cluster 2021-07-31T02:49:14.687041+0000 mgr.x (mgr.24542) 2655 : cluster [DBG] pgmap v3868: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:16.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:15 smithi045 conmon[39718]: cluster 2021-07-31T02:49:14.687041+0000 mgr.x ( 2021-07-31T02:49:16.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:15 smithi045 conmon[39718]: mgr.24542) 2655 : cluster [DBG] pgmap v3868: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:16.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:15 smithi045 conmon[33655]: cluster 2021-07-31T02:49:14.687041+0000 mgr.x (mgr.24542) 2655 2021-07-31T02:49:16.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:15 smithi045 conmon[33655]: : cluster [DBG] pgmap v3868: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:18.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:17 smithi154 conmon[34553]: cluster 2021-07-31T02:49:16.687490+0000 mgr.x (mgr.24542) 2656 : cluster [DBG] pgmap v3869: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:18.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:17 smithi045 conmon[39718]: cluster 2021-07-31T02:49:16.687490+0000 mgr.x ( 2021-07-31T02:49:18.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:17 smithi045 conmon[39718]: mgr.24542) 2656 : cluster [DBG] pgmap v3869: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:18.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:17 smithi045 conmon[33655]: cluster 2021-07-31T02:49:16.687490+0000 mgr.x (mgr.24542) 2656 : cluster [DBG] pgmap v3869: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:18.942 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:49:19.219 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:49:19.220 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 64m ago 64m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:49:19.220 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 61m ago 64m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:49:19.220 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 61m ago 68m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:49:19.220 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 64m ago 70m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:49:19.221 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 64m ago 70m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:49:19.221 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 61m ago 68m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:49:19.221 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 64m ago 68m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:49:19.221 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 64m ago 65m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:49:19.222 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 61m ago 65m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:49:19.222 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 64m ago 67m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:49:19.222 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 64m ago 67m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:49:19.222 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 64m ago 67m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:49:19.223 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 64m ago 66m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:49:19.223 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 61m ago 66m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:49:19.223 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 61m ago 66m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:49:19.223 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 61m ago 65m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:49:19.223 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 61m ago 65m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:49:19.224 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 61m ago 65m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:49:19.554 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:49:19.555 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:49:19.555 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:49:19.555 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:49:19.555 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:49:19.556 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:49:19.556 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:49:19.556 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:49:19.556 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:49:19.556 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:49:19.557 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:49:19.557 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:49:19.557 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:49:19.557 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:49:19.558 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:49:19.558 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:49:19.558 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:49:20.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:19 smithi154 conmon[34553]: cluster 2021-07-31T02:49:18.688022+0000 mgr.x (mgr.24542) 2657 : cluster [DBG] pgmap v3870: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:20.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:19 smithi154 conmon[34553]: audit 2021-07-31T02:49:18.930752+0000 mgr.x (mgr.24542) 2658 : audit [DBG] from='client.28150 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:49:20.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:19 smithi154 conmon[34553]: audit 2021-07-31T02:49:19.211965+0000 mgr.x (mgr.24542) 2659 : audit [DBG] from='client.18216 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:49:20.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:19 smithi154 conmon[34553]: audit 2021-07-31T02:49:19 2021-07-31T02:49:20.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:19 smithi154 conmon[34553]: .554635+0000 mon.c (mon.1) 149 : audit [DBG] 2021-07-31T02:49:20.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:19 smithi154 conmon[34553]: from='client.? 172.21.15.45:0/1533922109' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:49:20.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:19 smithi045 conmon[33655]: cluster 2021-07-31T02:49:18.688022+0000 mgr.x (mgr.24542) 2657 : cluster [DBG] pgmap v3870: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:20.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:19 smithi045 conmon[33655]: audit 2021-07-31T02:49:18.930752+0000 mgr.x (mgr.24542) 2658 : 2021-07-31T02:49:20.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:19 smithi045 conmon[33655]: audit [DBG] from='client.28150 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:49:20.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:19 smithi045 conmon[33655]: audit 2021-07-31T02:49:20.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:19 smithi045 conmon[33655]: 2021-07-31T02:49: 2021-07-31T02:49:20.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:19 smithi045 conmon[33655]: 19.211965+0000 mgr.x (mgr 2021-07-31T02:49:20.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:19 smithi045 conmon[33655]: .24542) 2659 : audit [DBG] 2021-07-31T02:49:20.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:19 smithi045 conmon[33655]: from='client.18216 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:49:20.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:19 smithi045 conmon[33655]: audit 2021-07-31T02:49:20.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:19 smithi045 conmon[33655]: 2021-07-31T02:49:19. 2021-07-31T02:49:20.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:19 smithi045 conmon[33655]: 554635+0000 mon.c (mon.1) 149 : 2021-07-31T02:49:20.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:19 smithi045 conmon[33655]: audit [DBG] from='client.? 172.21.15.45:0/1533922109' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:49:20.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:19 smithi045 conmon[39718]: cluster 2021-07-31T02:49:18. 2021-07-31T02:49:20.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:19 smithi045 conmon[39718]: 688022+0000 mgr.x (mgr.24542) 2657 : cluster [DBG] pgmap v3870: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:20.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:19 smithi045 conmon[39718]: audit 2021-07-31T02: 2021-07-31T02:49:20.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:19 smithi045 conmon[39718]: 49:18.930752+0000 mgr.x (mgr. 2021-07-31T02:49:20.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:19 smithi045 conmon[39718]: 24542) 2658 : audit [DBG] from='client.28150 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:49:20.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:19 smithi045 conmon[39718]: audit 2021- 2021-07-31T02:49:20.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:19 smithi045 conmon[39718]: 07-31T02:49:19.211965 2021-07-31T02:49:20.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:19 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 2659 : audit 2021-07-31T02:49:20.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:19 smithi045 conmon[39718]: [DBG] from='client.18216 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:49:20.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:19 smithi045 conmon[39718]: audit 2021-07-31T02 2021-07-31T02:49:20.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:19 smithi045 conmon[39718]: :49:19.554635+0000 mon.c (mon 2021-07-31T02:49:20.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:19 smithi045 conmon[39718]: .1) 149 : audit [DBG] from='client.? 172.21.15.45:0/1533922109' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:49:22.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:21 smithi154 conmon[34553]: cluster 2021-07-31T02:49:20.688605+0000 mgr.x (mgr.24542) 2660 : cluster [DBG] pgmap v3871: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:22.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:21 smithi045 conmon[39718]: cluster 2021-07-31T02:49:20.688605+0000 mgr.x (mgr.24542) 2021-07-31T02:49:22.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:21 smithi045 conmon[39718]: 2660 : cluster [DBG] pgmap v3871: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:22.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:21 smithi045 conmon[33655]: cluster 2021-07-31T02:49:20.688605+0000 mgr.x (mgr.24542) 2660 : cluster [DBG] pgmap v3871: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:22.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:21 smithi045 conmon[33655]: 2021-07-31T02:49:24.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:23 smithi154 conmon[34553]: cluster 2021-07-31T02:49:22.689172+0000 mgr.x (mgr.24542) 2661 : cluster [DBG] pgmap v3872: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:24.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:23 smithi045 conmon[33655]: cluster 2021-07-31T02:49:22.689172+0000 mgr.x (mgr.24542) 2661 : cluster 2021-07-31T02:49:24.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:23 smithi045 conmon[33655]: [DBG] pgmap v3872: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:24.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:23 smithi045 conmon[39718]: cluster 2021-07-31T02:49:22.689172+0000 mgr.x 2021-07-31T02:49:24.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:23 smithi045 conmon[39718]: (mgr.24542) 2661 : cluster [DBG] pgmap v3872: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:26.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:25 smithi154 conmon[34553]: cluster 2021-07-31T02:49:24.689729+0000 mgr.x (mgr.24542) 2662 : cluster [DBG] 2021-07-31T02:49:26.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:25 smithi154 conmon[34553]: pgmap v3873: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:26.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:25 smithi045 conmon[33655]: cluster 2021-07-31T02:49:24.689729+0000 mgr.x (mgr.24542) 2662 2021-07-31T02:49:26.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:25 smithi045 conmon[33655]: : cluster [DBG] pgmap v3873: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:26.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:25 smithi045 conmon[39718]: cluster 2021-07-31T02:49:24.689729+0000 mgr.x (mgr 2021-07-31T02:49:26.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:25 smithi045 conmon[39718]: .24542) 2662 : cluster [DBG] pgmap v3873: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:28.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:27 smithi154 conmon[34553]: cluster 2021-07-31T02:49:26.690235+0000 mgr.x (mgr.24542) 2663 : cluster [DBG] 2021-07-31T02:49:28.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:27 smithi154 conmon[34553]: pgmap v3874: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:28.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:27 smithi045 conmon[39718]: cluster 2021-07-31T02:49:26.690235+0000 mgr.x (mgr 2021-07-31T02:49:28.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:27 smithi045 conmon[39718]: .24542) 2663 : cluster [DBG] pgmap v3874: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:28.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:27 smithi045 conmon[33655]: cluster 2021-07-31T02:49:26.690235+0000 mgr.x (mgr.24542) 2663 : 2021-07-31T02:49:28.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:27 smithi045 conmon[33655]: cluster [DBG] pgmap v3874: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:29.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:28 smithi045 conmon[33655]: audit 2021-07-31T02:49:28.185053+0000 mon.b (mon.2) 385 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:49:29.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:28 smithi045 conmon[33655]: audit 2021-07-31T02:49:28.185354+0000 mon.b (mon.2) 386 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:49:29.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:28 smithi045 conmon[33655]: audit 2021-07-31T02:49:28.185581+0000 mon.a (mon.0) 3209 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:49:29.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:28 smithi045 conmon[33655]: 2021-07-31T02:49:29.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:28 smithi045 conmon[33655]: audit 2021-07-31T02:49:28.186023+0000 mon.a (mon. 2021-07-31T02:49:29.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:28 smithi045 conmon[33655]: 0) 3210 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:49:29.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:28 smithi045 conmon[39718]: audit 2021-07-31T02:49:28.185053+0000 mon.b (mon. 2021-07-31T02:49:29.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:28 smithi045 conmon[39718]: 2) 385 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:49:29.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:28 smithi045 conmon[39718]: audit 2021-07-31T02:49:28.185354 2021-07-31T02:49:29.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:28 smithi045 conmon[39718]: +0000 mon.b (mon.2) 386 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:49:29.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:28 smithi045 conmon[39718]: audit 2021-07-31T02:49:29.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:28 smithi045 conmon[39718]: 2021-07-31 2021-07-31T02:49:29.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:28 smithi045 conmon[39718]: T02:49:28. 2021-07-31T02:49:29.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:28 smithi045 conmon[39718]: 185581+0000 mon.a (mon.0) 3209 2021-07-31T02:49:29.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:28 smithi045 conmon[39718]: : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:49:29.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:28 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T02:49:29.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:28 smithi045 conmon[39718]: 31T02:49:28.186023+0000 mon.a (mon 2021-07-31T02:49:29.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:28 smithi045 conmon[39718]: .0) 3210 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:49:29.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:28 smithi154 conmon[34553]: audit 2021-07-31T02:49:28.185053+0000 mon.b (mon.2) 385 : audit 2021-07-31T02:49:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:28 smithi154 conmon[34553]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:49:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:28 smithi154 conmon[34553]: audit 2021-07-31T02:49:28.185354+0000 mon.b (mon.2) 386 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:49:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:28 smithi154 conmon[34553]: audit 2021-07 2021-07-31T02:49:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:28 smithi154 conmon[34553]: -31T02:49:28.185581+0000 mon.a (mon.0) 3209 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:49:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:28 smithi154 conmon[34553]: audit 2021-07-31T02:49:28.186023+0000 2021-07-31T02:49:29.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:28 smithi154 conmon[34553]: mon.a (mon.0) 3210 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:49:30.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[34553]: cluster 2021-07-31T02:49:28.690595+0000 mgr.x (mgr. 2021-07-31T02:49:30.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[34553]: 24542) 2664 : cluster [DBG] pgmap v3875: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:30.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[34553]: audit 2021-07-31T02:49:29.684516+0000 mon.b (mon.2 2021-07-31T02:49:30.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[34553]: ) 387 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:49:30.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[34553]: audit 2021-07-31T02:49:29 2021-07-31T02:49:30.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[34553]: .685927+0000 mon.b (mon.2) 388 : audit [DBG] 2021-07-31T02:49:30.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[34553]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:49:30.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[34553]: audit 2021-07-31T02: 2021-07-31T02:49:30.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[34553]: 49:29.686953+0000 mon.b (mon.2) 389 2021-07-31T02:49:30.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[34553]: : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:49:30.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: debug 2021-07-31T02:49:29.729+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:49:30.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:49:30.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:49:30.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: yield 2021-07-31T02:49:30.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:49:30.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:49:30.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:49:30.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:49:30.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:49:30.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:49:30.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:49:30.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:49:30.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:49:30.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: task.result() 2021-07-31T02:49:30.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:49:30.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: debug 2021-07-31T02:49:29.741+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:49:30.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:49:30.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:49:30.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: yield 2021-07-31T02:49:30.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:49:30.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:49:30.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:49:30.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:49:30.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:49:30.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:49:30.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:49:30.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:49:30.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:49:30.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: task.result() 2021-07-31T02:49:30.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:49:30.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: debug 2021-07-31T02:49:29.767+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:49:30.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:49:30.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:49:30.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: yield 2021-07-31T02:49:30.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:49:30.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:49:30.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:49:30.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:49:30.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:49:30.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:49:30.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:49:30.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:49:30.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:49:30.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: task.result() 2021-07-31T02:49:30.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:49:30.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: debug 2021-07-31T02:49:29.767+0000 7fcb9195d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:49:30.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:49:30.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:49:30.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: yield 2021-07-31T02:49:30.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:49:30.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:49:30.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:49:30.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:49:30.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:49:30.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:49:30.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:49:30.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:49:30.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:49:30.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: task.result() 2021-07-31T02:49:30.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:49:30.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: 2021-07-31T02:49:30.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:49:30.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: 2021-07-31T02:49:30.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:49:30.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:49:30.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:49:30.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:49:30.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:49:30.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:49:30.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:49:30.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:49:30.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:49:30.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:49:30.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:49:30.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:49:30.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:49:30.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:49:30.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:49:30.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:49:30.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:49:30.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:49:30.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: raise 2021-07-31T02:49:30.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:49:30.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:49:30.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:49:30.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:49:30.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:49:30.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: debug 2021-07-31T02:49:29.775+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:49:30.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:49:30.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:49:30.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: yield 2021-07-31T02:49:30.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:49:30.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:49:30.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:49:30.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:49:30.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:49:30.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:49:30.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:49:30.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:49:30.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:49:30.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: task.result() 2021-07-31T02:49:30.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:49:30.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: debug 2021-07-31T02:49:29.776+0000 7fcb9215e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:49:30.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:49:30.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:49:30.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: yield 2021-07-31T02:49:30.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:49:30.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:49:30.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:49:30.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:49:30.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:49:30.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:49:30.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:49:30.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:49:30.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:49:30.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: task.result() 2021-07-31T02:49:30.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:49:30.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: 2021-07-31T02:49:30.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:49:30.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: 2021-07-31T02:49:30.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:49:30.035 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:49:30.035 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:49:30.035 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:49:30.035 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:49:30.036 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:49:30.036 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:49:30.036 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:49:30.036 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:49:30.036 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:49:30.037 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:49:30.037 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:49:30.037 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:49:30.037 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:49:30.037 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:49:30.038 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:49:30.038 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:49:30.038 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:49:30.039 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: raise 2021-07-31T02:49:30.039 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:49:30.039 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:49:30.040 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:49:30.040 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:49:30.040 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:49:29 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:49:30.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:29 smithi045 conmon[39718]: cluster 2021-07-31T02:49: 2021-07-31T02:49:30.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:29 smithi045 conmon[39718]: 28.690595+0000 mgr.x (mgr.24542) 2664 : cluster [DBG] pgmap v3875: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:30.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:29 smithi045 conmon[39718]: audit 2021-07-31T02:49:30.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:29 smithi045 conmon[39718]: 2021-07-31T02:49 2021-07-31T02:49:30.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:29 smithi045 conmon[39718]: :29.684516+0000 mon.b ( 2021-07-31T02:49:30.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:29 smithi045 conmon[39718]: mon.2) 387 : audit [DBG] 2021-07-31T02:49:30.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:29 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:49:30.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:29 smithi045 conmon[39718]: audit 2021-07-31T02:49:30.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:29 smithi045 conmon[39718]: 2021-07-31T02: 2021-07-31T02:49:30.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:29 smithi045 conmon[39718]: 49:29.685927+0000 mon.b 2021-07-31T02:49:30.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:29 smithi045 conmon[39718]: (mon.2) 388 : audit 2021-07-31T02:49:30.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:29 smithi045 conmon[39718]: [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:49:30.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:29 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T02:49:30.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:29 smithi045 conmon[39718]: 31T02:49:29.686953+0000 2021-07-31T02:49:30.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:29 smithi045 conmon[39718]: mon.b (mon.2) 389 2021-07-31T02:49:30.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:29 smithi045 conmon[39718]: : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:49:30.045 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:29 smithi045 conmon[33655]: cluster 2021-07-31T02:49:28.690595+0000 mgr.x ( 2021-07-31T02:49:30.045 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:29 smithi045 conmon[33655]: mgr.24542) 2664 : cluster [DBG] pgmap v3875: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:30.045 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:29 smithi045 conmon[33655]: audit 2021-07-31T02:49: 2021-07-31T02:49:30.046 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:29 smithi045 conmon[33655]: 29.684516+0000 mon.b (mon.2) 387 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:49:30.046 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:29 smithi045 conmon[33655]: audit 2021-07-31T02:49:29.685927 2021-07-31T02:49:30.046 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:29 smithi045 conmon[33655]: +0000 mon.b (mon.2) 388 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:49:30.046 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:29 smithi045 conmon[33655]: audit 2021-07- 2021-07-31T02:49:30.046 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:29 smithi045 conmon[33655]: 31T02:49:29.686953+0000 mon.b 2021-07-31T02:49:30.047 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:29 smithi045 conmon[33655]: (mon.2) 389 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:49:30.047 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:29 smithi045 conmon[33655]: 2021-07-31T02:49:31.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:49:29.687719+0000 mgr.x (mgr.24542) 2021-07-31T02:49:31.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: 2665 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:49:31.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:49:29.730595+0000 mgr.x (mgr.24542) 2666 : cephadm [ERR] 2021-07-31T02:49:31.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:49:31.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:49:31.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: yield 2021-07-31T02:49:31.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:49:31.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:49:31.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:49:31.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:49:31.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:49:31.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:49:31.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:49:31.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:49:31.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:49:31.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:49:31.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:49:31.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:49:29.731049+0000 mgr.x (mgr.24542) 2667 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:49:31.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:49:29.742267+0000 mgr.x (mgr.24542) 2668 : cephadm [ERR] 2021-07-31T02:49:31.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:49:31.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:49:31.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: yield 2021-07-31T02:49:31.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:49:31.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:49:31.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:49:31.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:49:31.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:49:31.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:49:31.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:49:31.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:49:31.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:49:31.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:49:31.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:49:31.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:49:29.743145+0000 mgr.x (mgr.24542) 2669 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:49:31.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:49:31.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: 2021-07-31 2021-07-31T02:49:31.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: T02:49:29.768448+0000 2021-07-31T02:49:31.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: mgr.x (mgr.24542) 2670 : cephadm 2021-07-31T02:49:31.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: [ERR] 2021-07-31T02:49:31.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:49:31.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:49:29.687719 2021-07-31T02:49:31.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 2665 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:49:31.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: cephadm 2021-07-31T 2021-07-31T02:49:31.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: 02:49:29.730595+0000 2021-07-31T02:49:31.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: mgr.x (mgr.24542) 2666 : cephadm 2021-07-31T02:49:31.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: [ERR] 2021-07-31T02:49:31.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:49:31.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:49:31.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: yield 2021-07-31T02:49:31.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:49:31.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:49:31.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:49:31.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:49:31.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:49:31.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:49:31.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:49:31.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:49:31.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:49:31.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:49:31.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:49:31.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:49:29.731049+0000 mgr.x (mgr.24542) 2667 : cephadm [INF] 2021-07-31T02:49:31.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:49:31.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: cephadm 2021- 2021-07-31T02:49:31.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: 07-31T02:49:29.742267 2021-07-31T02:49:31.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 2021-07-31T02:49:31.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: 2668 : cephadm [ERR] 2021-07-31T02:49:31.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:49:31.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:49:31.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: yield 2021-07-31T02:49:31.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:49:31.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:49:31.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:49:31.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:49:31.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:49:31.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:49:31.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:49:31.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:49:31.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:49:31.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:49:31.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:49:31.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:49:29.743145+0000 mgr.x (mgr.24542) 2669 : cephadm [INF] 2021-07-31T02:49:31.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:49:31.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:49:31.046 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:49:31.046 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: yield 2021-07-31T02:49:31.046 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:49:31.046 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:49:31.047 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:49:31.047 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:49:31.047 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:49:31.047 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:49:31.047 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:49:31.048 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:49:31.048 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:49:31.048 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:49:31.048 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:49:31.049 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: cephadm 2021-07- 2021-07-31T02:49:31.049 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: 31T02:49:29. 2021-07-31T02:49:31.049 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: 768905+0000 mgr.x (mgr. 2021-07-31T02:49:31.049 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: 24542) 2671 : cephadm [ERR] 2021-07-31T02:49:31.050 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:49:31.050 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:49:31.050 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:49:31.050 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: yield 2021-07-31T02:49:31.050 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:49:31.051 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:49:31.051 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:49:31.051 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:49:31.051 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:49:31.052 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:49:31.052 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:49:31.052 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:49:31.052 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:49:31.052 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:49:31.053 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:49:31.053 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: 2021-07-31T02:49:31.053 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:49:31.053 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: 2021-07-31T02:49:31.053 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:49:31.054 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:49:31.054 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:49:31.054 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:49:31.054 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:49:31.055 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:49:31.055 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:49:31.055 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:49:31.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: 2021-07-31T02 2021-07-31T02:49:31.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: :49:29.768448+0000 2021-07-31T02:49:31.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: mgr.x (mgr.24542) 2670 2021-07-31T02:49:31.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: : cephadm [ERR] 2021-07-31T02:49:31.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:49:31.057 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:49:31.057 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: yield 2021-07-31T02:49:31.057 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:49:31.057 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:49:31.057 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:49:31.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:49:31.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:49:31.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:49:31.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:49:31.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:49:31.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:49:31.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:49:31.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:49:31.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:49:31.060 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: 2021-07-31T02 2021-07-31T02:49:31.060 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: :49:29.768905+0000 2021-07-31T02:49:31.060 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: mgr.x (mgr.24542) 2671 2021-07-31T02:49:31.060 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:49:31.060 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:49:31.061 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:49:31.061 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: yield 2021-07-31T02:49:31.061 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:49:31.061 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:49:31.062 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:49:31.062 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:49:31.062 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:49:31.062 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:49:31.062 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:49:31.063 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:49:31.063 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:49:31.063 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:49:31.063 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:49:31.063 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: 2021-07-31T02:49:31.064 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:49:31.064 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: 2021-07-31T02:49:31.064 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:49:31.064 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:49:31.065 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:49:31.065 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:49:31.065 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:49:31.065 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:49:31.066 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:49:31.066 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:49:31.066 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:49:31.066 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:49:31.067 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:49:31.067 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:49:31.067 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:49:31.067 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:49:31.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:49:31.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: raise 2021-07-31T02:49:31.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:49:31.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:49:31.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:49:31.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:49:31.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:49:31.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:49:29.776782+0000 mgr.x (mgr.24542) 2672 : cephadm [ERR] 2021-07-31T02:49:31.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:49:31.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:49:31.070 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: yield 2021-07-31T02:49:31.070 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:49:31.070 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:49:31.070 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:49:31.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:49:31.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:49:31.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:49:31.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:49:31.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:49:31.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:49:31.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:49:31.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:49:31.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:49:29.777195+0000 mgr.x (mgr.24542 2021-07-31T02:49:31.073 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: ) 2673 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:49:31.073 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:49:31.073 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:49:31.073 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: yield 2021-07-31T02:49:31.073 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:49:31.074 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:49:31.074 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:49:31.074 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:49:31.075 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:49:31.075 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:49:31.075 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:49:31.075 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:49:31.075 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:49:31.076 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:49:31.076 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:49:31.076 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:49:31.076 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:49:31.077 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:49:31.077 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:49:31.077 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:49:31.077 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: raise 2021-07-31T02:49:31.077 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:49:31.078 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:49:31.078 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:49:31.078 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:49:31.078 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:49:31.079 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:49:29.776782+0000 mgr.x (mgr.24542) 2672 : cephadm [ERR] 2021-07-31T02:49:31.079 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:49:31.079 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:49:31.079 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: yield 2021-07-31T02:49:31.079 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:49:31.080 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:49:31.080 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:49:31.080 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:49:31.080 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:49:31.080 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:49:31.081 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:49:31.081 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:49:31.081 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:49:31.081 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:49:31.081 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:49:31.082 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:49:29.777195+0000 mgr.x (mgr.24542) 2673 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:49:31.082 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:49:31.082 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:49:31.082 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: yield 2021-07-31T02:49:31.083 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:49:31.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:49:31.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:49:31.084 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:49:31.084 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:49:31.084 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:49:31.084 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:49:31.084 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:49:31.085 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:49:31.085 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: 2021-07-31T02:49:31.085 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:49:31.085 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: 2021-07-31T02:49:31.085 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:49:31.086 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:49:31.086 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:49:31.086 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:49:31.086 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:49:31.087 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:49:31.087 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:49:31.087 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:49:31.087 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:49:31.087 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:49:31.088 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:49:31.088 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:49:31.088 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:49:31.088 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:49:31.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:49:31.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:49:31.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:49:31.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:49:31.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: raise 2021-07-31T02:49:31.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:49:31.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:49:31.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:49:31.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:49:31.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:49:31.091 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:49:31.091 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:49:31.091 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:49:31.091 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:49:31.092 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:49:31.092 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:49:31.092 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:49:31.092 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:49:31.092 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:49:31.093 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:49:31.093 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: 2021-07-31T02:49:31.093 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:49:31.093 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: 2021-07-31T02:49:31.094 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:49:31.094 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:49:31.094 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:49:31.094 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:49:31.094 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:49:31.095 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:49:31.095 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:49:31.095 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:49:31.095 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:49:31.096 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:49:31.096 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:49:31.096 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:49:31.096 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:49:31.096 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:49:31.097 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:49:31.097 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:49:31.097 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:49:31.097 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:49:31.098 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: raise 2021-07-31T02:49:31.098 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:49:31.098 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:49:31.098 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:49:31.098 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:49:31.099 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:30 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:49:31.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:49:29.687719+0000 mgr.x (mgr.24542) 2665 : cephadm 2021-07-31T02:49:31.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:49:31.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:49:29.730595+0000 mgr.x (mgr.24542) 2666 : cephadm [ERR] 2021-07-31T02:49:31.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:49:31.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:49:31.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: yield 2021-07-31T02:49:31.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:49:31.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:49:31.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:49:31.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:49:31.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:49:31.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:49:31.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:49:31.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:49:31.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:49:31.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:49:31.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:49:31.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:49:29.731049+0000 mgr.x (mgr.24542) 2667 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:49:31.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:49:29.742267+0000 mgr.x (mgr.24542) 2668 : cephadm [ERR] 2021-07-31T02:49:31.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:49:31.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:49:31.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: yield 2021-07-31T02:49:31.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:49:31.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:49:31.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:49:31.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:49:31.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:49:31.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:49:31.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:49:31.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:49:31.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:49:31.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:49:31.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:49:31.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: cephadm 2021-07-31T02: 2021-07-31T02:49:31.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: 49:29.743145+0000 mgr.x (mgr.24542) 2669 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:49:31.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:49:29.768448+0000 mgr.x (mgr.24542) 2670 : cephadm [ERR] 2021-07-31T02:49:31.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:49:31.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:49:31.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: yield 2021-07-31T02:49:31.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:49:31.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:49:31.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:49:31.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:49:31.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:49:31.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:49:31.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:49:31.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:49:31.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:49:31.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:49:31.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:49:31.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: cephadm 2021 2021-07-31T02:49:31.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: -07-31T02:49:29.768905+0000 mgr.x (mgr.24542) 2671 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:49:31.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:49:31.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:49:31.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: yield 2021-07-31T02:49:31.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:49:31.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:49:31.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:49:31.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:49:31.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:49:31.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:49:31.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:49:31.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:49:31.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:49:31.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:49:31.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:49:31.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: 2021-07-31T02:49:31.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:49:31.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: 2021-07-31T02:49:31.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:49:31.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:49:31.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:49:31.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:49:31.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:49:31.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:49:31.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:49:31.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:49:31.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:49:31.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:49:31.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:49:31.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:49:31.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:49:31.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:49:31.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:49:31.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:49:31.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:49:31.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:49:31.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: raise 2021-07-31T02:49:31.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:49:31.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:49:31.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:49:31.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:49:31.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:49:31.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:49:29.776782+0000 mgr.x (mgr.24542) 2672 : cephadm [ERR] 2021-07-31T02:49:31.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:49:31.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:49:31.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: yield 2021-07-31T02:49:31.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:49:31.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:49:31.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:49:31.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:49:31.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:49:31.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:49:31.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:49:31.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:49:31.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:49:31.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:49:31.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:49:31.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:49:29.777195+0000 mgr.x (mgr.24542) 2673 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:49:31.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:49:31.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:49:31.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: yield 2021-07-31T02:49:31.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:49:31.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:49:31.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:49:31.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:49:31.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:49:31.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:49:31.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:49:31.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:49:31.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:49:31.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:49:31.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:49:31.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: 2021-07-31T02:49:31.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:49:31.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: 2021-07-31T02:49:31.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:49:31.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:49:31.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:49:31.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:49:31.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:49:31.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:49:31.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:49:31.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:49:31.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:49:31.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:49:31.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:49:31.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:49:31.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:49:31.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:49:31.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:49:31.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:49:31.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:49:31.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:49:31.285 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: raise 2021-07-31T02:49:31.285 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:49:31.285 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:49:31.285 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:49:31.285 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:49:31.286 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:30 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:49:32.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:31 smithi045 conmon[39718]: cluster 2021-07-31T02:49:30.691254+0000 mgr.x ( 2021-07-31T02:49:32.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:31 smithi045 conmon[39718]: mgr.24542) 2674 : cluster [DBG] pgmap v3876: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:32.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:31 smithi045 conmon[33655]: cluster 2021-07-31T02:49:30.691254+0000 2021-07-31T02:49:32.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:31 smithi045 conmon[33655]: mgr.x (mgr.24542) 2674 : cluster [DBG] pgmap v3876: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:32.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:31 smithi154 conmon[34553]: cluster 2021-07-31T02:49:30.691254+0000 mgr.x (mgr. 2021-07-31T02:49:32.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:31 smithi154 conmon[34553]: 24542) 2674 : cluster [DBG] pgmap v3876: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:34.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:33 smithi154 conmon[34553]: cluster 2021-07-31T02:49:32.691809+0000 mgr.x (mgr.24542) 2675 : cluster 2021-07-31T02:49:34.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:33 smithi154 conmon[34553]: [DBG] pgmap v3877: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:34.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:33 smithi045 conmon[33655]: cluster 2021-07-31T02:49:32.691809+0000 mgr.x (mgr.24542) 2675 : cluster [DBG] pgmap v3877: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:34.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:33 smithi045 conmon[39718]: cluster 2021-07-31T02:49:32.691809+0000 mgr.x (mgr.24542 2021-07-31T02:49:34.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:33 smithi045 conmon[39718]: ) 2675 : cluster [DBG] pgmap v3877: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:36.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:35 smithi154 conmon[34553]: cluster 2021-07-31T02:49:34.692219+0000 mgr.x (mgr.24542) 2676 : cluster [DBG] pgmap v3878: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:36.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:35 smithi045 conmon[33655]: cluster 2021-07-31T02:49:34.692219+0000 mgr.x (mgr.24542) 2676 : cluster [DBG] 2021-07-31T02:49:36.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:35 smithi045 conmon[33655]: pgmap v3878: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:36.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:35 smithi045 conmon[39718]: cluster 2021-07-31T02:49:34.692219+0000 mgr.x (mgr.24542) 2676 2021-07-31T02:49:36.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:35 smithi045 conmon[39718]: : cluster [DBG] pgmap v3878: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:38.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:37 smithi154 conmon[34553]: cluster 2021-07-31T02:49:36.692752+0000 mgr.x (mgr.24542) 2677 : cluster [DBG] pgmap v3879: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:38.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:37 smithi045 conmon[39718]: cluster 2021-07-31T02:49:36.692752+0000 mgr.x (mgr.24542) 2677 : cluster [DBG] pgmap v3879: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:38.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:37 smithi045 conmon[33655]: cluster 2021-07-31T02:49:36.692752+0000 mgr.x (mgr.24542) 2677 2021-07-31T02:49:38.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:37 smithi045 conmon[33655]: : cluster [DBG] pgmap v3879: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:40.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:39 smithi045 conmon[33655]: cluster 2021-07-31T02:49:38.693286+0000 mgr.x (mgr.24542) 2678 : 2021-07-31T02:49:40.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:39 smithi045 conmon[33655]: cluster [DBG] pgmap v3880: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:40.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:39 smithi045 conmon[39718]: cluster 2021-07-31T02:49:38.693286+0000 mgr.x ( 2021-07-31T02:49:40.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:39 smithi045 conmon[39718]: mgr.24542) 2678 : cluster [DBG] pgmap v3880: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:40.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:39 smithi154 conmon[34553]: cluster 2021-07-31T02:49:38.693286+0000 mgr.x (mgr.24542) 2678 : cluster [DBG] pgmap v3880: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:42.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:41 smithi045 conmon[33655]: cluster 2021-07-31T02:49:40.693819+0000 mgr.x (mgr.24542) 2679 : cluster [DBG] 2021-07-31T02:49:42.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:41 smithi045 conmon[33655]: pgmap v3881: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:42.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:41 smithi045 conmon[39718]: cluster 2021-07-31T02:49:40.693819+0000 2021-07-31T02:49:42.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:41 smithi045 conmon[39718]: mgr.x (mgr.24542) 2679 : cluster [DBG] pgmap v3881: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:42.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:41 smithi154 conmon[34553]: cluster 2021-07-31T02:49:40.693819+0000 mgr.x (mgr.24542) 2679 : cluster [DBG] pgmap v3881: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:44.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:43 smithi154 conmon[34553]: cluster 2021-07-31T02:49:42.694359+0000 mgr.x (mgr.24542) 2680 : cluster [DBG] pgmap v3882: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:44.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:43 smithi045 conmon[39718]: cluster 2021-07-31T02:49:42.694359+0000 mgr.x (mgr.24542) 2680 : cluster [DBG] pgmap v3882: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:44.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:43 smithi045 conmon[39718]: 2021-07-31T02:49:44.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:43 smithi045 conmon[33655]: cluster 2021-07-31T02:49:42.694359+0000 mgr.x (mgr.24542) 2680 : cluster [DBG] pgmap v3882: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:46.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:45 smithi154 conmon[34553]: cluster 2021-07-31T02:49:44.694931+0000 mgr.x (mgr.24542) 2681 : cluster [DBG] pgmap v3883: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:46.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:45 smithi045 conmon[33655]: cluster 2021-07-31T02:49:44.694931+0000 mgr.x (mgr.24542) 2681 : cluster [DBG] pgmap v3883: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:46.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:45 smithi045 conmon[39718]: cluster 2021-07-31T02:49:44.694931+0000 mgr.x (mgr.24542) 2681 : cluster 2021-07-31T02:49:46.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:45 smithi045 conmon[39718]: [DBG] pgmap v3883: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:48.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:47 smithi045 conmon[33655]: cluster 2021-07-31T02:49:46.695358+0000 mgr.x (mgr.24542) 2682 : cluster 2021-07-31T02:49:48.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:47 smithi045 conmon[33655]: [DBG] pgmap v3884: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:48.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:47 smithi045 conmon[39718]: cluster 2021-07-31T02:49:46.695358+0000 mgr.x 2021-07-31T02:49:48.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:47 smithi045 conmon[39718]: (mgr.24542) 2682 : cluster [DBG] pgmap v3884: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:48.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:47 smithi154 conmon[34553]: cluster 2021-07-31T02:49:46.695358+0000 mgr.x (mgr.24542) 2682 : cluster [DBG] pgmap v3884: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:49.873 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:49:50.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:49 smithi045 conmon[33655]: cluster 2021-07-31T02:49:48 2021-07-31T02:49:50.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:49 smithi045 conmon[33655]: .695728+0000 mgr.x (mgr.24542) 2683 : cluster [DBG] pgmap v3885: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:50.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:49 smithi045 conmon[39718]: cluster 2021-07-31T02:49:48. 2021-07-31T02:49:50.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:49 smithi045 conmon[39718]: 695728+0000 mgr.x (mgr.24542) 2683 : cluster [DBG] pgmap v3885: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:50.152 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:49:50.152 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 64m ago 65m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:49:50.153 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 62m ago 65m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:49:50.153 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 62m ago 68m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:49:50.153 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 64m ago 70m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:49:50.153 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 64m ago 70m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:49:50.154 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 62m ago 69m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:49:50.154 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 64m ago 69m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:49:50.154 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 64m ago 65m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:49:50.154 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 62m ago 65m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:49:50.154 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 64m ago 68m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:49:50.155 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 64m ago 68m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:49:50.155 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 64m ago 67m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:49:50.155 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 64m ago 67m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:49:50.155 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 62m ago 67m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:49:50.155 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 62m ago 66m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:49:50.156 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 62m ago 66m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:49:50.156 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 62m ago 66m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:49:50.156 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 62m ago 65m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:49:50.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:49 smithi154 conmon[34553]: cluster 2021-07-31T02:49:48.695728+0000 mgr.x (mgr.24542) 2683 : cluster [DBG] pgmap v3885: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:50.489 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:49:50.489 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:49:50.490 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:49:50.490 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:49:50.490 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:49:50.490 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:49:50.491 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:49:50.491 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:49:50.491 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:49:50.491 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:49:50.492 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:49:50.492 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:49:50.492 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:49:50.492 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:49:50.492 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:49:50.493 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:49:50.493 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:49:51.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:50 smithi045 conmon[33655]: audit 2021-07-31T02:49:49.861702+0000 mgr.x (mgr.24542) 2684 : audit 2021-07-31T02:49:51.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:50 smithi045 conmon[33655]: [DBG] from='client.28168 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:49:51.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:50 smithi045 conmon[33655]: audit 2021-07-31T02:49:50.144189+0000 mgr.x (mgr.24542) 2685 : audit [DBG] from='client.28174 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:49:51.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:50 smithi045 conmon[33655]: audit 2021-07-31T02:49:50.489303+0000 2021-07-31T02:49:51.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:50 smithi045 conmon[33655]: mon.c (mon.1) 150 : audit [DBG] from='client.? 172.21.15.45:0/2601541941' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:49:51.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:50 smithi045 conmon[39718]: audit 2021-07-31T02:49: 2021-07-31T02:49:51.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:50 smithi045 conmon[39718]: 49.861702+0000 mgr.x (mgr.24542) 2684 : audit [DBG] from='client.28168 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:49:51.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:50 smithi045 conmon[39718]: audit 2021 2021-07-31T02:49:51.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:50 smithi045 conmon[39718]: -07-31T02:49:50.144189+0000 mgr.x (mgr.24542 2021-07-31T02:49:51.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:50 smithi045 conmon[39718]: ) 2685 : audit [DBG] from='client.28174 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:49:51.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:50 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T02:49:51.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:50 smithi045 conmon[39718]: 02:49:50.489303+0000 mon.c ( 2021-07-31T02:49:51.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:50 smithi045 conmon[39718]: mon.1) 150 : audit [DBG] from='client.? 172.21.15.45:0/2601541941' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:49:51.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:50 smithi045 conmon[39718]: 2021-07-31T02:49:51.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:50 smithi154 conmon[34553]: audit 2021-07-31T02:49:49.861702+0000 mgr.x (mgr.24542) 2684 : audit [DBG] from='client.28168 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:49:51.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:50 smithi154 conmon[34553]: audit 2021-07-31T02:49:50.144189+0000 mgr.x (mgr.24542) 2685 : audit [DBG] from='client.28174 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:49:51.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:50 smithi154 conmon[34553]: audit 2021-07-31T02:49:50.489303+0000 mon.c (mon.1) 150 : audit [DBG] from='client.? 172.21.15.45:0/2601541941' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:49:52.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:51 smithi045 conmon[33655]: cluster 2021-07-31T02:49:50.696138+0000 mgr.x (mgr.24542) 2686 : cluster [DBG] pgmap v3886: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:52.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:51 smithi045 conmon[39718]: cluster 2021-07-31T02:49:50.696138+0000 mgr.x (mgr. 2021-07-31T02:49:52.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:51 smithi045 conmon[39718]: 24542) 2686 : cluster [DBG] pgmap v3886: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:52.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:51 smithi154 conmon[34553]: cluster 2021-07-31T02:49:50.696138+0000 mgr.x (mgr.24542) 2686 : cluster [DBG] pgmap v3886: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:54.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:53 smithi154 conmon[34553]: cluster 2021-07-31T02:49:52.696512+0000 mgr.x (mgr.24542) 2687 : cluster [DBG] pgmap v3887: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:54.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:53 smithi045 conmon[33655]: cluster 2021-07-31T02:49:52.696512+0000 mgr.x 2021-07-31T02:49:54.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:53 smithi045 conmon[33655]: (mgr.24542) 2687 : cluster [DBG] pgmap v3887: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:54.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:53 smithi045 conmon[39718]: cluster 2021-07-31T02:49:52.696512+0000 mgr.x ( 2021-07-31T02:49:54.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:53 smithi045 conmon[39718]: mgr.24542) 2687 : cluster [DBG] pgmap v3887: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:56.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:55 smithi154 conmon[34553]: cluster 2021-07-31T02:49:54.697124+0000 mgr.x (mgr.24542) 2688 : cluster [DBG] pgmap v3888: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:56.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:55 smithi045 conmon[33655]: cluster 2021-07-31T02:49:54.697124+0000 mgr.x (mgr.24542) 2688 : cluster [DBG] pgmap v3888: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:56.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:55 smithi045 conmon[39718]: cluster 2021-07-31T02:49:54.697124+0000 mgr.x (mgr 2021-07-31T02:49:56.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:55 smithi045 conmon[39718]: .24542) 2688 : cluster [DBG] pgmap v3888: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:58.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:57 smithi045 conmon[33655]: cluster 2021-07-31T02:49:56.697478+0000 mgr.x (mgr.24542) 2689 : 2021-07-31T02:49:58.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:57 smithi045 conmon[33655]: cluster [DBG] pgmap v3889: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:58.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:57 smithi045 conmon[39718]: cluster 2021-07-31T02:49:56. 2021-07-31T02:49:58.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:57 smithi045 conmon[39718]: 697478+0000 mgr.x (mgr.24542) 2689 : cluster [DBG] pgmap v3889: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:49:58.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:57 smithi154 conmon[34553]: cluster 2021-07-31T02:49:56.697478+0000 mgr.x (mgr.24542) 2689 : cluster [DBG] pgmap v3889: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:00.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:59 smithi045 conmon[33655]: cluster 2021-07-31T02:49:58.698025+0000 mgr.x (mgr.24542) 2021-07-31T02:50:00.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:49:59 smithi045 conmon[33655]: 2690 : cluster [DBG] pgmap v3890: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:00.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:59 smithi045 conmon[39718]: cluster 2021-07-31T02:49:58.698025+0000 mgr.x (mgr 2021-07-31T02:50:00.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:49:59 smithi045 conmon[39718]: .24542) 2690 : cluster [DBG] pgmap v3890: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:00.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:49:59 smithi154 conmon[34553]: cluster 2021-07-31T02:49:58.698025+0000 mgr.x (mgr.24542) 2690 : cluster [DBG] pgmap v3890: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:01.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:00 smithi045 conmon[33655]: cluster 2021-07-31T02:50:00.000155+0000 mon.a (mon.0) 3211 : 2021-07-31T02:50:01.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:00 smithi045 conmon[33655]: cluster [INF] overall HEALTH_OK 2021-07-31T02:50:01.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:00 smithi045 conmon[39718]: cluster 2021-07-31T02:50:00.000155 2021-07-31T02:50:01.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:00 smithi045 conmon[39718]: +0000 mon.a (mon.0) 3211 : cluster [INF] overall HEALTH_OK 2021-07-31T02:50:01.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:00 smithi154 conmon[34553]: cluster 2021-07-31T02:50:00.000155+0000 mon.a (mon.0) 3211 : cluster [INF] overall HEALTH_OK 2021-07-31T02:50:02.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:01 smithi045 conmon[33655]: cluster 2021-07-31T02:50:00.698494+0000 mgr.x (mgr.24542) 2021-07-31T02:50:02.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:01 smithi045 conmon[33655]: 2691 : cluster [DBG] pgmap v3891: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:02.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:01 smithi045 conmon[39718]: cluster 2021-07-31T02:50:00.698494+0000 mgr.x (mgr 2021-07-31T02:50:02.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:01 smithi045 conmon[39718]: .24542) 2691 : cluster [DBG] pgmap v3891: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:02.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:01 smithi154 conmon[34553]: cluster 2021-07-31T02:50:00.698494+0000 mgr.x (mgr.24542) 2691 : cluster [DBG] 2021-07-31T02:50:02.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:01 smithi154 conmon[34553]: pgmap v3891: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:04.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:03 smithi154 conmon[34553]: cluster 2021-07-31T02:50:02.699041+0000 mgr.x (mgr.24542) 2692 : cluster [DBG] pgmap v3892: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:04.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:03 smithi154 conmon[34553]: 2021-07-31T02:50:04.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:03 smithi045 conmon[39718]: cluster 2021-07-31T02:50:02.699041+0000 2021-07-31T02:50:04.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:03 smithi045 conmon[39718]: mgr.x (mgr.24542) 2692 : cluster [DBG] pgmap v3892: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:04.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:03 smithi045 conmon[33655]: cluster 2021-07-31T02:50:02.699041+0000 mgr.x (mgr. 2021-07-31T02:50:04.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:03 smithi045 conmon[33655]: 24542) 2692 : cluster [DBG] pgmap v3892: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:06.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:05 smithi045 conmon[33655]: cluster 2021-07-31T02:50:04.699548+0000 mgr.x (mgr.24542) 2693 : cluster [DBG] pgmap v3893: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:06.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:05 smithi045 conmon[39718]: cluster 2021-07-31T02:50:04.699548+0000 mgr.x (mgr 2021-07-31T02:50:06.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:05 smithi045 conmon[39718]: .24542) 2693 : cluster [DBG] pgmap v3893: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:06.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:05 smithi154 conmon[34553]: cluster 2021-07-31T02:50:04.699548+0000 mgr.x (mgr.24542) 2693 : cluster [DBG] pgmap v3893: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:08.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:07 smithi045 conmon[39718]: cluster 2021-07-31T02:50:06.700002+0000 mgr.x (mgr.24542) 2021-07-31T02:50:08.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:07 smithi045 conmon[39718]: 2694 : cluster [DBG] pgmap v3894: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:08.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:07 smithi045 conmon[33655]: cluster 2021-07-31T02:50:06.700002+0000 mgr.x (mgr.24542) 2694 2021-07-31T02:50:08.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:07 smithi045 conmon[33655]: : cluster [DBG] pgmap v3894: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:08.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:07 smithi154 conmon[34553]: cluster 2021-07-31T02:50:06.700002+0000 mgr.x (mgr.24542) 2694 : cluster [DBG] pgmap v3894: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:08.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:07 smithi154 conmon[34553]: 2021-07-31T02:50:10.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:09 smithi045 conmon[39718]: cluster 2021-07-31T02:50:08.700363+0000 mgr.x (mgr. 2021-07-31T02:50:10.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:09 smithi045 conmon[39718]: 24542) 2695 : cluster [DBG] pgmap v3895: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:10.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:09 smithi045 conmon[33655]: cluster 2021-07-31T02:50:08.700363+0000 mgr.x (mgr.24542) 2695 : cluster 2021-07-31T02:50:10.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:09 smithi045 conmon[33655]: [DBG] pgmap v3895: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:10.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:09 smithi154 conmon[34553]: cluster 2021-07-31T02:50:08.700363+0000 mgr.x (mgr.24542) 2695 : cluster [DBG] 2021-07-31T02:50:10.560 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:09 smithi154 conmon[34553]: pgmap v3895: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:12.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:11 smithi045 conmon[33655]: cluster 2021-07-31T02:50:10.700829+0000 mgr.x (mgr.24542) 2696 : cluster 2021-07-31T02:50:12.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:11 smithi045 conmon[33655]: [DBG] pgmap v3896: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:12.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:11 smithi045 conmon[39718]: cluster 2021-07-31T02:50:10.700829+0000 mgr.x (mgr 2021-07-31T02:50:12.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:11 smithi045 conmon[39718]: .24542) 2696 : cluster [DBG] pgmap v3896: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:12.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:11 smithi154 conmon[34553]: cluster 2021-07-31T02:50:10.700829+0000 mgr.x (mgr.24542) 2696 : cluster [DBG] pgmap v3896: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:14.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:13 smithi154 conmon[34553]: cluster 2021-07-31T02:50:12. 2021-07-31T02:50:14.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:13 smithi154 conmon[34553]: 701362+0000 mgr.x (mgr.24542) 2697 : cluster [DBG] pgmap v3897: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:14.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:13 smithi045 conmon[39718]: cluster 2021-07-31T02:50:12.701362+0000 mgr.x (mgr.24542) 2021-07-31T02:50:14.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:13 smithi045 conmon[39718]: 2697 : cluster [DBG] pgmap v3897: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:14.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:13 smithi045 conmon[33655]: cluster 2021-07-31T02:50:12.701362+0000 mgr.x (mgr.24542) 2697 : cluster 2021-07-31T02:50:14.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:13 smithi045 conmon[33655]: [DBG] pgmap v3897: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:16.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:15 smithi045 conmon[39718]: cluster 2021-07-31T02:50:14.702003+0000 mgr.x ( 2021-07-31T02:50:16.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:15 smithi045 conmon[39718]: mgr.24542) 2698 : cluster [DBG] pgmap v3898: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:16.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:15 smithi045 conmon[33655]: cluster 2021-07-31T02:50:14.702003+0000 mgr.x (mgr.24542) 2698 : cluster 2021-07-31T02:50:16.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:15 smithi045 conmon[33655]: [DBG] pgmap v3898: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:16.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:15 smithi154 conmon[34553]: cluster 2021-07-31T02:50:14.702003+0000 mgr.x (mgr.24542) 2698 : cluster [DBG] pgmap v3898: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:18.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:17 smithi045 conmon[39718]: cluster 2021-07-31T02:50:16.702415+0000 mgr.x (mgr.24542) 2699 2021-07-31T02:50:18.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:17 smithi045 conmon[39718]: : cluster [DBG] pgmap v3899: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:18.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:17 smithi045 conmon[33655]: cluster 2021-07-31T02:50:16.702415+0000 mgr.x (mgr.24542) 2699 2021-07-31T02:50:18.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:17 smithi045 conmon[33655]: : cluster [DBG] pgmap v3899: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:18.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:17 smithi154 conmon[34553]: cluster 2021-07-31T02:50:16.702415+0000 mgr.x (mgr.24542) 2699 : cluster [DBG] pgmap v3899: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:20.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:19 smithi045 conmon[33655]: cluster 2021-07-31T02:50:18.702953+0000 mgr.x (mgr.24542) 2021-07-31T02:50:20.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:19 smithi045 conmon[33655]: 2700 : cluster [DBG] pgmap v3900: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:20.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:19 smithi045 conmon[39718]: cluster 2021-07-31T02:50:18.702953+0000 mgr.x (mgr.24542) 2021-07-31T02:50:20.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:19 smithi045 conmon[39718]: 2700 : cluster [DBG] pgmap v3900: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:20.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:19 smithi154 conmon[34553]: cluster 2021-07-31T02:50:18.702953+0000 mgr.x (mgr.24542) 2700 : cluster [DBG] pgmap v3900: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:20.806 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:50:21.083 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:50:21.083 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 65m ago 66m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:50:21.084 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 62m ago 65m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:50:21.084 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 62m ago 69m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:50:21.084 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 65m ago 71m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:50:21.084 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 65m ago 71m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:50:21.084 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 62m ago 69m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:50:21.085 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 65m ago 69m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:50:21.085 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 65m ago 66m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:50:21.085 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 62m ago 66m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:50:21.086 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 65m ago 68m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:50:21.086 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 65m ago 68m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:50:21.086 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 65m ago 68m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:50:21.087 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 65m ago 67m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:50:21.087 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 62m ago 67m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:50:21.087 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 62m ago 67m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:50:21.087 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 62m ago 66m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:50:21.088 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 62m ago 66m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:50:21.088 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 62m ago 66m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:50:21.418 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:50:21.419 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:50:21.419 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:50:21.419 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:50:21.420 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:50:21.420 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:50:21.420 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:50:21.420 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:50:21.421 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:50:21.421 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:50:21.421 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:50:21.421 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:50:21.422 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:50:21.422 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:50:21.422 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:50:21.422 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:50:21.423 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:50:22.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:21 smithi045 conmon[33655]: cluster 2021-07-31T02:50:20.703519+0000 mgr.x (mgr.24542) 2701 : cluster [DBG] pgmap v3901: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:22.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:21 smithi045 conmon[33655]: audit 2021-07-31T02:50:20.793767+0000 mgr.x (mgr.24542) 2702 : audit [DBG] from='client.18246 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:50:22.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:21 smithi045 conmon[33655]: audit 2021-07-31T02:50:21.075277+0000 mgr.x (mgr.24542) 2703 : audit [DBG] from='client.18252 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:50:22.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:21 smithi045 conmon[33655]: audit 2021-07-31T02:50:21.418704+0000 mon.c (mon.1) 151 : audit [DBG] from='client.? 172.21.15.45:0/970401888' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:50:22.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:21 smithi045 conmon[39718]: cluster 2021-07-31T02:50:20.703519+0000 2021-07-31T02:50:22.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:21 smithi045 conmon[39718]: mgr.x (mgr.24542) 2701 : cluster [DBG] pgmap v3901: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:22.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:21 smithi045 conmon[39718]: audit 2021 2021-07-31T02:50:22.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:21 smithi045 conmon[39718]: -07-31T02:50:20.793767+0000 mgr.x (mgr.24542) 2702 : audit [DBG] from='client.18246 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:50:22.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:21 smithi045 conmon[39718]: audit 2021-07-31T02:50: 2021-07-31T02:50:22.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:21 smithi045 conmon[39718]: 21.075277+0000 mgr.x (mgr.24542) 2703 : audit [DBG] from='client.18252 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:50:22.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:21 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T02:50:22.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:21 smithi045 conmon[39718]: 31T02:50:21.418704+0000 mon.c (mon.1) 151 : 2021-07-31T02:50:22.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:21 smithi045 conmon[39718]: audit [DBG] from='client.? 172.21.15.45:0/970401888' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:50:22.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:21 smithi154 conmon[34553]: cluster 2021-07-31T02:50:20.703519+0000 mgr.x (mgr.24542) 2701 : cluster [DBG] pgmap v3901: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:22.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:21 smithi154 conmon[34553]: audit 2021-07-31T02:50:20.793767+0000 mgr.x (mgr.24542) 2702 : audit [DBG] from='client.18246 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:50:22.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:21 smithi154 conmon[34553]: audit 2021-07-31T02:50:21.075277+0000 mgr.x (mgr.24542) 2703 : audit [DBG] from='client.18252 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:50:22.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:21 smithi154 conmon[34553]: audit 2021-07-31T02:50:21.418704+0000 mon.c (mon.1) 151 : audit 2021-07-31T02:50:22.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:21 smithi154 conmon[34553]: [DBG] from='client.? 172.21.15.45:0/970401888' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:50:24.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:23 smithi154 conmon[34553]: cluster 2021-07-31T02:50:22.703957+0000 mgr.x (mgr.24542) 2704 : cluster [DBG] pgmap v3902: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:24.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:23 smithi045 conmon[39718]: cluster 2021-07-31T02:50:22.703957+0000 mgr.x (mgr 2021-07-31T02:50:24.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:23 smithi045 conmon[39718]: .24542) 2704 : cluster [DBG] pgmap v3902: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:24.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:23 smithi045 conmon[33655]: cluster 2021-07-31T02:50:22.703957+0000 mgr.x (mgr.24542) 2021-07-31T02:50:24.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:23 smithi045 conmon[33655]: 2704 : cluster [DBG] pgmap v3902: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:26.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:25 smithi045 conmon[33655]: cluster 2021-07-31T02:50:24.704507+0000 mgr.x (mgr.24542) 2021-07-31T02:50:26.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:25 smithi045 conmon[33655]: 2705 : cluster [DBG] pgmap v3903: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:26.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:25 smithi045 conmon[39718]: cluster 2021-07-31T02:50:24.704507+0000 mgr.x 2021-07-31T02:50:26.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:25 smithi045 conmon[39718]: (mgr.24542) 2705 : cluster [DBG] pgmap v3903: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:26.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:25 smithi154 conmon[34553]: cluster 2021-07-31T02:50:24.704507+0000 mgr.x (mgr.24542) 2705 : cluster [DBG] pgmap v3903: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:28.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:27 smithi045 conmon[39718]: cluster 2021-07-31T02:50:26.705050+0000 2021-07-31T02:50:28.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:27 smithi045 conmon[39718]: mgr.x (mgr.24542) 2706 : cluster [DBG] pgmap v3904: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:28.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:27 smithi045 conmon[33655]: cluster 2021-07-31T02:50:26.705050+0000 mgr.x (mgr.24542) 2706 : 2021-07-31T02:50:28.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:27 smithi045 conmon[33655]: cluster [DBG] pgmap v3904: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:28.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:27 smithi154 conmon[34553]: cluster 2021-07-31T02:50:26.705050+0000 mgr.x (mgr.24542) 2706 : cluster [DBG] pgmap v3904: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:29.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:28 smithi045 conmon[33655]: audit 2021-07-31T02:50:28.188715+0000 mon.b (mon.2) 390 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:50:29.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:28 smithi045 conmon[33655]: audit 2021-07-31T02:50:28.188977+0000 mon.b (mon.2) 391 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:50:29.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:28 smithi045 conmon[33655]: audit 2021-07- 2021-07-31T02:50:29.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:28 smithi045 conmon[33655]: 31T02:50:28.189340+0000 mon.a 2021-07-31T02:50:29.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:28 smithi045 conmon[33655]: (mon.0) 3212 : audit 2021-07-31T02:50:29.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:28 smithi045 conmon[33655]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:50:29.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:28 smithi045 conmon[33655]: audit 2021 2021-07-31T02:50:29.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:28 smithi045 conmon[33655]: -07-31T02:50: 2021-07-31T02:50:29.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:28 smithi045 conmon[33655]: 28.189736+0000 mon.a (mon 2021-07-31T02:50:29.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:28 smithi045 conmon[33655]: .0) 3213 : audit [INF] 2021-07-31T02:50:29.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:28 smithi045 conmon[33655]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:50:29.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:28 smithi045 conmon[39718]: audit 2021-07-31T02:50:28.188715+0000 mon.b (mon 2021-07-31T02:50:29.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:28 smithi045 conmon[39718]: .2) 390 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:50:29.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:28 smithi045 conmon[39718]: audit 2021-07-31T02:50:28. 2021-07-31T02:50:29.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:28 smithi045 conmon[39718]: 188977+0000 mon.b (mon.2) 391 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:50:29.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:28 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T02:50:29.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:28 smithi045 conmon[39718]: 02:50:28.189340+0000 mon.a (mon.0 2021-07-31T02:50:29.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:28 smithi045 conmon[39718]: ) 3212 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:50:29.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:28 smithi045 conmon[39718]: audit 2021-07-31T02:50:28 2021-07-31T02:50:29.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:28 smithi045 conmon[39718]: .189736+0000 mon.a (mon.0) 3213 : audit [INF] 2021-07-31T02:50:29.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:28 smithi045 conmon[39718]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:50:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:28 smithi154 conmon[34553]: audit 2021-07-31T02:50:28.188715+0000 mon.b (mon.2) 390 : audit [INF] 2021-07-31T02:50:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:28 smithi154 conmon[34553]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:50:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:28 smithi154 conmon[34553]: audit 2021-07-31T02:50:28.188977+0000 mon.b (mon.2) 391 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:50:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:28 smithi154 conmon[34553]: audit 2021-07-31T02:50:28. 2021-07-31T02:50:29.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:28 smithi154 conmon[34553]: 189340+0000 mon.a (mon.0) 3212 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:50:29.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:28 smithi154 conmon[34553]: audit 2021 2021-07-31T02:50:29.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:28 smithi154 conmon[34553]: -07-31T02:50:28.189736+0000 mon.a (mon.0) 3213 2021-07-31T02:50:29.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:28 smithi154 conmon[34553]: : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:50:30.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:29 smithi045 conmon[39718]: cluster 2021-07-31T02:50:28.705514+0000 mgr.x (mgr.24542) 2707 : cluster [DBG] pgmap v3905: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:30.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:29 smithi045 conmon[33655]: cluster 2021-07-31T02:50:28.705514+0000 mgr.x (mgr.24542) 2707 2021-07-31T02:50:30.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:29 smithi045 conmon[33655]: : cluster [DBG] pgmap v3905: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:30.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[34553]: cluster 2021-07-31T02:50:28.705514+0000 mgr.x (mgr.24542) 2707 : cluster [DBG] pgmap v3905: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:30.253 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: debug 2021-07-31T02:50:29.822+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:50:30.253 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:50:30.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:50:30.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: yield 2021-07-31T02:50:30.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:50:30.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:50:30.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:50:30.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:50:30.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:50:30.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:50:30.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:50:30.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:50:30.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:50:30.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: task.result() 2021-07-31T02:50:30.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:50:30.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: debug 2021-07-31T02:50:29.834+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:50:30.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:50:30.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:50:30.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: yield 2021-07-31T02:50:30.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:50:30.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:50:30.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:50:30.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:50:30.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:50:30.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:50:30.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:50:30.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:50:30.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:50:30.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: task.result() 2021-07-31T02:50:30.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:50:30.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: debug 2021-07-31T02:50:29.861+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:50:30.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:50:30.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:50:30.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: yield 2021-07-31T02:50:30.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:50:30.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:50:30.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:50:30.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:50:30.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:50:30.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:50:30.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:50:30.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:50:30.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:50:30.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: task.result() 2021-07-31T02:50:30.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:50:30.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: debug 2021-07-31T02:50:29.861+0000 7fcb9095b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:50:30.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:50:30.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:50:30.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: yield 2021-07-31T02:50:30.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:50:30.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:50:30.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:50:30.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:50:30.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:50:30.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:50:30.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:50:30.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:50:30.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:50:30.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: task.result() 2021-07-31T02:50:30.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:50:30.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: 2021-07-31T02:50:30.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:50:30.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: 2021-07-31T02:50:30.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:50:30.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:50:30.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:50:30.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:50:30.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:50:30.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:50:30.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:50:30.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:50:30.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:50:30.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:50:30.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:50:30.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:50:30.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:50:30.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:50:30.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:50:30.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:50:30.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:50:30.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:50:30.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: raise 2021-07-31T02:50:30.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:50:30.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:50:30.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:50:30.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:50:30.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:50:30.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: debug 2021-07-31T02:50:29.870+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:50:30.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:50:30.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:50:30.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: yield 2021-07-31T02:50:30.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:50:30.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:50:30.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:50:30.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:50:30.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:50:30.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:50:30.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:50:30.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:50:30.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:50:30.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: task.result() 2021-07-31T02:50:30.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:50:30.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: debug 2021-07-31T02:50:29.870+0000 7fcb9115c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:50:30.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:50:30.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:50:30.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: yield 2021-07-31T02:50:30.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:50:30.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:50:30.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:50:30.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:50:30.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:50:30.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:50:30.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:50:30.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:50:30.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:50:30.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: task.result() 2021-07-31T02:50:30.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:50:30.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: 2021-07-31T02:50:30.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:50:30.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: 2021-07-31T02:50:30.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:50:30.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:50:30.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:50:30.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:50:30.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:50:30.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:50:30.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:50:30.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:50:30.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:50:30.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:50:30.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:50:30.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:50:30.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:50:30.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:50:30.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:50:30.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:50:30.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:50:30.284 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:50:30.284 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: raise 2021-07-31T02:50:30.284 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:50:30.284 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:50:30.284 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:50:30.285 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:50:30.285 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:50:29 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:50:31.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: audit 2021-07-31T02:50:29.777876+0000 mon.b (mon.2) 392 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:50:31.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: audit 2021-07-31T02:50:29.779260 2021-07-31T02:50:31.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: +0000 mon.b (mon.2) 393 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:50:31.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: audit 2021-07-31T02:50:29.780147+0000 mon.b (mon.2) 394 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:50:31.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:50:29.780872+0000 mgr.x (mgr.24542) 2708 : 2021-07-31T02:50:31.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:50:31.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: cephadm 2021-07 2021-07-31T02:50:31.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: -31T02:50:29.823890+0000 mgr.x (mgr.24542) 2709 2021-07-31T02:50:31.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: : cephadm [ERR] 2021-07-31T02:50:31.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:50:31.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:50:31.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: yield 2021-07-31T02:50:31.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:50:31.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:50:31.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:50:31.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:50:31.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:50:31.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:50:31.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:50:31.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:50:31.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:50:31.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:50:31.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:50:31.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:50:29.824497+0000 mgr.x (mgr.24542) 2710 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:50:31.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:50:29.835533+0000 mgr.x (mgr.24542) 2021-07-31T02:50:31.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: 2711 : cephadm [ERR] 2021-07-31T02:50:31.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:50:31.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:50:31.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: yield 2021-07-31T02:50:31.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:50:31.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:50:31.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:50:31.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:50:31.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:50:31.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:50:31.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:50:31.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:50:31.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:50:31.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:50:31.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:50:31.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:50:29.835942+0000 mgr.x (mgr.24542) 2712 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:50:31.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:50:29.862242+0000 mgr.x (mgr.24542) 2713 : cephadm [ERR] 2021-07-31T02:50:31.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:50:31.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:50:31.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: yield 2021-07-31T02:50:31.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:50:31.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:50:31.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:50:31.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:50:31.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:50:31.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:50:31.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:50:31.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:50:31.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:50:31.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:50:31.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:50:31.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: 2021-07-31T02:50:31.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:50:29.862661+0000 mgr.x (mgr.24542) 2714 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:50:31.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:50:31.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:50:31.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: yield 2021-07-31T02:50:31.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:50:31.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:50:31.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:50:31.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:50:31.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:50:31.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:50:31.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:50:31.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:50:31.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:50:31.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:50:31.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:50:31.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: 2021-07-31T02:50:31.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:50:31.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: 2021-07-31T02:50:31.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:50:31.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:50:31.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:50:31.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:50:31.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:50:31.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:50:31.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:50:31.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:50:31.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:50:31.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:50:31.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:50:31.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:50:31.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:50:31.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:50:31.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:50:31.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:50:31.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:50:31.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:50:31.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: raise 2021-07-31T02:50:31.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:50:31.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:50:31.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:50:31.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:50:31.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:50:31.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:50:29.871269+0000 2021-07-31T02:50:31.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: mgr.x (mgr.24542) 2715 : cephadm [ERR] 2021-07-31T02:50:31.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:50:31.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:50:31.285 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: yield 2021-07-31T02:50:31.285 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:50:31.285 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:50:31.285 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:50:31.285 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:50:31.286 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:50:31.286 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:50:31.286 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:50:31.286 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:50:31.287 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:50:31.287 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:50:31.287 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:50:31.287 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:50:29.871654+0000 mgr.x (mgr.24542) 2716 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:50:31.287 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:50:31.288 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:50:31.288 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: yield 2021-07-31T02:50:31.288 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:50:31.288 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:50:31.288 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:50:31.289 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:50:31.289 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:50:31.289 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:50:31.289 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:50:31.290 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:50:31.290 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:50:31.290 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:50:31.290 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:50:31.290 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: 2021-07-31T02:50:31.291 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:50:31.291 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: 2021-07-31T02:50:31.291 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:50:31.291 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:50:31.292 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:50:31.292 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:50:31.292 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:50:31.292 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:50:31.292 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:50:31.293 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:50:31.293 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:50:31.293 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:50:31.293 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:50:31.294 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:50:31.294 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:50:31.294 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:50:31.294 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:50:31.294 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:50:31.295 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:50:31.295 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:50:31.295 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: raise 2021-07-31T02:50:31.295 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:50:31.295 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:50:31.296 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:50:31.296 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:50:31.296 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:30 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:50:31.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: audit 2021-07-31T02:50:29.777876+0000 mon.b ( 2021-07-31T02:50:31.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: mon.2) 392 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:50:31.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T02:50:31.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: 31T02:50:29.779260+0000 mon.b (mon.2 2021-07-31T02:50:31.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: ) 393 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:50:31.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: audit 2021-07 2021-07-31T02:50:31.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: -31T02:50:29.780147+0000 mon.b (mon 2021-07-31T02:50:31.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: .2) 394 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:50:31.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: 2021-07-31T02:50:31.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:50:31.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: 2021-07-31T02 2021-07-31T02:50:31.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: :50:29.780872+0000 mgr.x ( 2021-07-31T02:50:31.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: mgr.24542) 2708 : cephadm [WRN] 2021-07-31T02:50:31.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:50:31.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: cephadm 2021-07 2021-07-31T02:50:31.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: -31T02:50:29.823890+0000 mgr.x (mgr 2021-07-31T02:50:31.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: .24542) 2709 : cephadm [ERR] 2021-07-31T02:50:31.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: 2021-07-31T02:50:31.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:50:31.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:50:31.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: yield 2021-07-31T02:50:31.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:50:31.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:50:31.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:50:31.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:50:31.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:50:31.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:50:31.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:50:31.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:50:31.304 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:50:31.304 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:50:31.304 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:50:31.304 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:50:29.824497+0000 mgr.x (mgr.24542) 2710 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:50:31.304 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: cephadm 2021-07 2021-07-31T02:50:31.305 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: -31T02:50:29.835533 2021-07-31T02:50:31.305 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 2711 2021-07-31T02:50:31.305 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: : cephadm [ERR] 2021-07-31T02:50:31.305 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:50:31.306 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:50:31.306 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: yield 2021-07-31T02:50:31.306 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:50:31.306 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:50:31.306 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:50:31.307 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:50:31.307 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:50:31.307 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:50:31.307 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:50:31.307 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:50:31.308 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:50:31.308 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:50:31.308 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:50:31.308 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:50:29.835942+0000 mgr.x (mgr.24542) 2712 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:50:31.309 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: cephadm 2021-07-31T02: 2021-07-31T02:50:31.309 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: 50:29.862242+0000 mgr.x (mgr.24542) 2713 : cephadm 2021-07-31T02:50:31.309 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: [ERR] 2021-07-31T02:50:31.309 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:50:31.309 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:50:31.310 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: yield 2021-07-31T02:50:31.310 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:50:31.310 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:50:31.310 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:50:31.311 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:50:31.311 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:50:31.311 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:50:31.311 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:50:31.311 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:50:31.312 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:50:31.312 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:50:31.312 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:50:31.312 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:50:29.862661+0000 mgr.x (mgr.24542) 2714 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:50:31.312 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:50:31.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:50:31.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: yield 2021-07-31T02:50:31.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:50:31.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:50:31.314 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:50:31.314 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:50:31.314 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:50:31.314 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:50:31.314 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:50:31.315 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:50:31.315 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:50:31.315 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:50:31.316 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: audit 2021-07-31T02:50:29.777876+0000 mon.b 2021-07-31T02:50:31.316 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: (mon.2) 392 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:50:31.316 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: audit 2021-07-31T02:50:29.779260+0000 mon.b (mon.2) 393 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:50:31.316 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: audit 2021-07-31T02 2021-07-31T02:50:31.317 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: :50:29.780147+0000 mon.b ( 2021-07-31T02:50:31.317 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: mon.2) 394 : audit [INF] 2021-07-31T02:50:31.317 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:50:31.317 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: cephadm 2021-07- 2021-07-31T02:50:31.317 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: 31T02:50:29.780872+0000 2021-07-31T02:50:31.318 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: mgr.x (mgr.24542) 2708 : 2021-07-31T02:50:31.318 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:50:31.318 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: cephadm 2021-07- 2021-07-31T02:50:31.318 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: 31T02:50:29.823890+0000 mgr.x (mgr. 2021-07-31T02:50:31.319 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: 24542) 2709 : cephadm [ERR] 2021-07-31T02:50:31.319 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:50:31.319 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:50:31.319 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: yield 2021-07-31T02:50:31.319 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:50:31.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:50:31.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:50:31.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:50:31.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:50:31.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:50:31.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:50:31.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:50:31.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:50:31.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:50:31.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:50:31.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: 2021-07-31T02:50:31.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:50:29.824497+0000 mgr.x ( 2021-07-31T02:50:31.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: mgr.24542) 2710 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:50:31.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: 2021-07-31T02:50:31.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: cephadm 2021 2021-07-31T02:50:31.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: -07-31T02:50:29.835533+0000 mgr.x (mgr 2021-07-31T02:50:31.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: .24542) 2711 : cephadm [ERR] 2021-07-31T02:50:31.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:50:31.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:50:31.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: yield 2021-07-31T02:50:31.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:50:31.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:50:31.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:50:31.325 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:50:31.325 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: 2021-07-31T02:50:31.326 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:50:31.326 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: 2021-07-31T02:50:31.326 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:50:31.326 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:50:31.326 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:50:31.327 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:50:31.327 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:50:31.327 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:50:31.327 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:50:31.328 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:50:31.328 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:50:31.328 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:50:31.328 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:50:31.328 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:50:31.329 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:50:31.329 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:50:31.329 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:50:31.329 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:50:31.329 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:50:31.330 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:50:31.330 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: raise 2021-07-31T02:50:31.330 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:50:31.330 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:50:31.331 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:50:31.331 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:50:31.331 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:50:31.331 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:50:29.871269+0000 2021-07-31T02:50:31.331 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: mgr.x (mgr.24542) 2715 : cephadm [ERR] 2021-07-31T02:50:31.332 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:50:31.332 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:50:31.332 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: yield 2021-07-31T02:50:31.332 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:50:31.332 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:50:31.333 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:50:31.333 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:50:31.333 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:50:31.333 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:50:31.333 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:50:31.334 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:50:31.334 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:50:31.334 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:50:31.335 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:50:31.335 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:50:31.335 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:50:31.335 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:50:31.336 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:50:31.336 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:50:31.336 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:50:29.835942+0000 mgr.x (mgr.24542) 2712 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:50:31.336 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: cephadm 2021-07 2021-07-31T02:50:31.336 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: -31T02:50:29.862242+0000 mgr.x 2021-07-31T02:50:31.337 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: (mgr.24542) 2713 : cephadm [ERR] 2021-07-31T02:50:31.337 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:50:31.337 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:50:31.337 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: yield 2021-07-31T02:50:31.338 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:50:31.338 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:50:31.338 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:50:31.338 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:50:31.338 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:50:31.339 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:50:31.339 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:50:31.339 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:50:31.339 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:50:31.339 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:50:31.340 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:50:31.340 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:50:29.862661+0000 mgr.x (mgr.24542) 2714 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:50:31.340 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:50:31.340 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:50:31.341 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: yield 2021-07-31T02:50:31.341 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:50:31.341 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:50:31.341 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:50:31.341 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:50:31.342 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:50:31.342 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:50:31.342 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:50:31.342 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:50:31.343 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:50:31.343 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:50:31.343 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:50:31.344 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:50:31.344 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:50:31.344 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:50:31.344 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:50:29.871654+0000 mgr.x (mgr.24542) 2716 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:50:31.345 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:50:31.345 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:50:31.345 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: yield 2021-07-31T02:50:31.345 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:50:31.345 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:50:31.346 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:50:31.346 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:50:31.346 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:50:31.346 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:50:31.347 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:50:31.347 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:50:31.347 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:50:31.347 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:50:31.347 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:50:31.348 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: 2021-07-31T02:50:31.348 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:50:31.348 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: 2021-07-31T02:50:31.348 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:50:31.348 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:50:31.349 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:50:31.349 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:50:31.349 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:50:31.349 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:50:31.350 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:50:31.350 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:50:31.350 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:50:31.350 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:50:31.350 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:50:31.351 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:50:31.351 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:50:31.351 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:50:31.351 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:50:31.351 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:50:31.352 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:50:31.352 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:50:31.352 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: raise 2021-07-31T02:50:31.352 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:50:31.353 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:50:31.353 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:50:31.353 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:50:31.353 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:50:31.354 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: 2021-07-31T02:50:31.354 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:50:31.354 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: 2021-07-31T02:50:31.355 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:50:31.355 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:50:31.355 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:50:31.355 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:50:31.356 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:50:31.356 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:50:31.356 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:50:31.356 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:50:31.356 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:50:31.357 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:50:31.357 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:50:31.357 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:50:31.357 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:50:31.358 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:50:31.358 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:50:31.358 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:50:31.358 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:50:31.358 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:50:31.359 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: raise 2021-07-31T02:50:31.359 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:50:31.359 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:50:31.359 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:50:31.360 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:50:31.360 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:50:31.360 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:50:31.360 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: 2021-07-31T02:50:29.871269+0000 mgr.x (mgr.24542) 2715 : cephadm [ERR] 2021-07-31T02:50:31.360 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:50:31.361 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:50:31.361 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: yield 2021-07-31T02:50:31.361 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:50:31.361 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:50:31.361 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:50:31.362 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:50:31.362 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:50:31.362 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:50:31.362 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:50:31.363 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:50:31.363 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:50:31.363 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:50:31.363 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:50:31.364 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:50:29.871654+0000 mgr.x (mgr.24542) 2716 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:50:31.364 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:50:31.364 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:50:31.364 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: yield 2021-07-31T02:50:31.364 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:50:31.365 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:50:31.365 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:50:31.365 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:50:31.365 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:50:31.366 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:50:31.366 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:50:31.366 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:50:31.366 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:50:31.366 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:50:31.367 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:50:31.367 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: 2021-07-31T02:50:31.367 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:50:31.367 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: 2021-07-31T02:50:31.368 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:50:31.368 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:50:31.368 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:50:31.368 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:50:31.368 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:50:31.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:50:31.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:50:31.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:50:31.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:50:31.370 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:50:31.370 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:50:31.370 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:50:31.370 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:50:31.370 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:50:31.371 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:50:31.371 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:50:31.371 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:50:31.371 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:50:31.372 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: raise 2021-07-31T02:50:31.372 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:50:31.372 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:50:31.372 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:50:31.372 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:50:31.373 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:30 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:50:32.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:31 smithi154 conmon[34553]: cluster 2021-07-31T02:50:30.705956+0000 mgr.x (mgr.24542) 2717 : cluster [DBG] pgmap v3906: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:32.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:31 smithi045 conmon[39718]: cluster 2021-07-31T02:50:30.705956+0000 mgr.x ( 2021-07-31T02:50:32.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:31 smithi045 conmon[39718]: mgr.24542) 2717 : cluster [DBG] pgmap v3906: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:32.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:31 smithi045 conmon[33655]: cluster 2021-07-31T02:50 2021-07-31T02:50:32.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:31 smithi045 conmon[33655]: :30.705956+0000 mgr.x (mgr.24542) 2717 : cluster [DBG] pgmap v3906: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:34.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:33 smithi154 conmon[34553]: cluster 2021-07-31T02:50:32.706437+0000 mgr.x (mgr.24542) 2718 : cluster [DBG] pgmap v3907: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:34.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:33 smithi045 conmon[33655]: cluster 2021-07-31T02:50:32.706437+0000 2021-07-31T02:50:34.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:33 smithi045 conmon[33655]: mgr.x (mgr.24542) 2718 : cluster [DBG] pgmap v3907: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:34.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:33 smithi045 conmon[39718]: cluster 2021-07-31T02:50:32.706437+0000 mgr.x 2021-07-31T02:50:34.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:33 smithi045 conmon[39718]: (mgr.24542) 2718 : cluster [DBG] pgmap v3907: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:36.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:35 smithi045 conmon[33655]: cluster 2021-07-31T02:50:34.707092+0000 mgr.x (mgr.24542) 2719 : cluster [DBG] pgmap v3908: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:36.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:35 smithi045 conmon[39718]: cluster 2021-07-31T02:50:34.707092+0000 mgr.x (mgr 2021-07-31T02:50:36.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:35 smithi045 conmon[39718]: .24542) 2719 : cluster [DBG] pgmap v3908: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:36.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:35 smithi154 conmon[34553]: cluster 2021-07-31T02:50:34.707092+0000 mgr.x (mgr.24542) 2719 : cluster [DBG] pgmap v3908: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:38.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:37 smithi045 conmon[39718]: cluster 2021-07-31T02:50:36.707620+0000 mgr.x (mgr.24542) 2720 : cluster [DBG] pgmap v3909: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:38.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:37 smithi045 conmon[33655]: cluster 2021-07-31T02:50:36.707620+0000 mgr.x (mgr.24542) 2720 : cluster [DBG] pgmap v3909: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:38.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:37 smithi154 conmon[34553]: cluster 2021-07-31T02:50:36.707620+0000 mgr.x (mgr.24542) 2720 : cluster [DBG] pgmap v3909: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:40.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:39 smithi045 conmon[33655]: cluster 2021-07-31T02:50:38.708083+0000 mgr.x (mgr.24542) 2721 : cluster 2021-07-31T02:50:40.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:39 smithi045 conmon[33655]: [DBG] pgmap v3910: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:40.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:39 smithi045 conmon[39718]: cluster 2021-07-31T02:50:38.708083+0000 mgr.x ( 2021-07-31T02:50:40.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:39 smithi045 conmon[39718]: mgr.24542) 2721 : cluster [DBG] pgmap v3910: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:40.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:39 smithi154 conmon[34553]: cluster 2021-07-31T02:50:38.708083+0000 mgr.x (mgr.24542) 2721 : cluster [DBG] pgmap v3910: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:42.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:41 smithi045 conmon[39718]: cluster 2021-07-31T02:50:40.708511+0000 mgr.x (mgr.24542 2021-07-31T02:50:42.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:41 smithi045 conmon[39718]: ) 2722 : cluster [DBG] pgmap v3911: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:42.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:41 smithi045 conmon[33655]: cluster 2021-07-31T02:50:40.708511 2021-07-31T02:50:42.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:41 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 2722 : cluster [DBG] pgmap v3911: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:42.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:41 smithi154 conmon[34553]: cluster 2021-07-31T02:50:40.708511+0000 mgr.x (mgr.24542) 2722 : cluster [DBG] pgmap v3911: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:44.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:43 smithi154 conmon[34553]: cluster 2021-07-31T02:50:42.709091+0000 mgr.x (mgr.24542) 2723 : cluster [DBG] pgmap v3912: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:44.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:43 smithi045 conmon[33655]: cluster 2021-07-31T02:50:42.709091+0000 2021-07-31T02:50:44.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:43 smithi045 conmon[33655]: mgr.x (mgr.24542) 2723 : cluster [DBG] pgmap v3912: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:44.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:43 smithi045 conmon[39718]: cluster 2021-07-31T02:50:42.709091+0000 mgr.x 2021-07-31T02:50:44.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:43 smithi045 conmon[39718]: (mgr.24542) 2723 : cluster [DBG] pgmap v3912: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:46.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:45 smithi045 conmon[33655]: cluster 2021-07-31T02:50:44.709519+0000 mgr.x ( 2021-07-31T02:50:46.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:45 smithi045 conmon[33655]: mgr.24542) 2724 : cluster [DBG] pgmap v3913: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:46.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:45 smithi045 conmon[39718]: cluster 2021-07-31T02:50:44.709519+0000 mgr.x ( 2021-07-31T02:50:46.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:45 smithi045 conmon[39718]: mgr.24542) 2724 : cluster [DBG] pgmap v3913: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:46.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:45 smithi154 conmon[34553]: cluster 2021-07-31T02:50:44.709519+0000 mgr.x (mgr.24542) 2724 : cluster [DBG] pgmap v3913: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:48.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:47 smithi045 conmon[33655]: cluster 2021-07-31T02:50:46.710036+0000 mgr.x (mgr.24542) 2021-07-31T02:50:48.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:47 smithi045 conmon[33655]: 2725 : cluster [DBG] pgmap v3914: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:48.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:47 smithi045 conmon[39718]: cluster 2021-07-31T02:50:46.710036+0000 mgr.x ( 2021-07-31T02:50:48.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:47 smithi045 conmon[39718]: mgr.24542) 2725 : cluster [DBG] pgmap v3914: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:48.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:47 smithi154 conmon[34553]: cluster 2021-07-31T02:50:46.710036+0000 mgr.x (mgr.24542) 2725 : cluster [DBG] pgmap v3914: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:50.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:49 smithi045 conmon[39718]: cluster 2021-07-31T02:50:48.710508+0000 mgr.x (mgr.24542) 2726 : cluster [DBG] pgmap v3915: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:50.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:49 smithi045 conmon[33655]: cluster 2021-07-31T02:50:48.710508+0000 mgr.x ( 2021-07-31T02:50:50.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:49 smithi045 conmon[33655]: mgr.24542) 2726 : cluster [DBG] pgmap v3915: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:50.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:49 smithi154 conmon[34553]: cluster 2021-07-31T02:50:48.710508+0000 mgr.x (mgr.24542) 2726 : cluster [DBG] pgmap v3915: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:51.734 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:50:52.016 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:50:52.016 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 65m ago 66m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:50:52.017 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 63m ago 66m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:50:52.017 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 63m ago 69m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:50:52.017 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 65m ago 71m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:50:52.017 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 65m ago 72m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:50:52.018 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 63m ago 70m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:50:52.018 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 65m ago 70m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:50:52.018 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 65m ago 66m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:50:52.019 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 63m ago 66m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:50:52.019 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 65m ago 69m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:50:52.019 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 65m ago 69m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:50:52.019 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 65m ago 68m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:50:52.019 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 65m ago 68m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:50:52.020 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 63m ago 68m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:50:52.020 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 63m ago 67m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:50:52.020 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 63m ago 67m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:50:52.020 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 63m ago 67m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:50:52.020 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 63m ago 66m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:50:52.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:51 smithi045 conmon[33655]: cluster 2021-07-31T02:50:50.711012+0000 mgr.x (mgr 2021-07-31T02:50:52.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:51 smithi045 conmon[33655]: .24542) 2727 : cluster [DBG] pgmap v3916: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:52.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:51 smithi045 conmon[39718]: cluster 2021-07-31T02: 2021-07-31T02:50:52.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:51 smithi045 conmon[39718]: 50:50.711012+0000 mgr.x (mgr.24542) 2727 : cluster [DBG] pgmap v3916: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:52.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:51 smithi154 conmon[34553]: cluster 2021-07-31T02:50:50.711012+0000 mgr.x (mgr.24542) 2727 : cluster [DBG] pgmap v3916: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:52.354 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:50:52.355 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:50:52.355 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:50:52.355 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:50:52.356 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:50:52.356 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:50:52.356 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:50:52.356 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:50:52.357 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:50:52.357 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:50:52.357 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:50:52.357 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:50:52.357 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:50:52.358 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:50:52.358 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:50:52.358 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:50:52.358 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:50:53.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:52 smithi154 conmon[34553]: audit 2021-07-31T02:50:51.722552+0000 mgr.x (mgr.24542) 2728 : audit [DBG] from='client.28204 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:50:53.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:52 smithi154 conmon[34553]: audit 2021-07-31T02:50:52.008322+0000 mgr.x (mgr.24542) 2729 : audit [DBG] from='client.28210 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:50:53.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:52 smithi154 conmon[34553]: audit 2021-07-31T02:50:52.354014+0000 mon.a (mon.0) 3214 : audit [DBG] 2021-07-31T02:50:53.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:52 smithi154 conmon[34553]: from='client.? 172.21.15.45:0/1959723501' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:50:53.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:52 smithi045 conmon[33655]: audit 2021-07-31T02:50:51.722552+0000 mgr.x (mgr.24542) 2728 2021-07-31T02:50:53.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:52 smithi045 conmon[33655]: : audit [DBG] from='client.28204 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:50:53.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:52 smithi045 conmon[33655]: audit 2021-07-31T02:50:52.008322+0000 mgr.x (mgr.24542) 2729 : audit [DBG] from='client.28210 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:50:53.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:52 smithi045 conmon[33655]: audit 2021-07- 2021-07-31T02:50:53.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:52 smithi045 conmon[33655]: 31T02:50:52.354014+0000 mon.a (mon.0) 3214 : audit [DBG] from='client.? 172.21.15.45:0/1959723501' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:50:53.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:52 smithi045 conmon[39718]: audit 2021-07-31T02:50:51.722552+0000 2021-07-31T02:50:53.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:52 smithi045 conmon[39718]: mgr.x (mgr.24542) 2728 : audit [DBG] from='client.28204 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:50:53.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:52 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T02:50:53.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:52 smithi045 conmon[39718]: 02:50:52.008322+0000 mgr.x (mgr.24542) 2729 : audit [DBG] 2021-07-31T02:50:53.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:52 smithi045 conmon[39718]: from='client.28210 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:50:53.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:52 smithi045 conmon[39718]: audit 2021-07-31T02:50 2021-07-31T02:50:53.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:52 smithi045 conmon[39718]: :52.354014+0000 mon.a (mon.0) 2021-07-31T02:50:53.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:52 smithi045 conmon[39718]: 3214 : audit [DBG] from='client.? 172.21.15.45:0/1959723501' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:50:54.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:53 smithi154 conmon[34553]: cluster 2021-07-31T02:50:52.711346+0000 mgr.x (mgr.24542) 2730 : cluster [DBG] 2021-07-31T02:50:54.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:53 smithi154 conmon[34553]: pgmap v3917: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:54.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:53 smithi045 conmon[33655]: cluster 2021-07-31T02:50:52.711346+0000 mgr.x (mgr.24542) 2730 : cluster [DBG] pgmap v3917: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:54.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:53 smithi045 conmon[39718]: cluster 2021-07-31T02:50:52.711346+0000 mgr.x (mgr.24542) 2730 : cluster [DBG] pgmap v3917: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:56.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:55 smithi154 conmon[34553]: cluster 2021-07-31T02:50:54.711902+0000 mgr.x (mgr.24542) 2731 : cluster [DBG] pgmap v3918: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:56.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:55 smithi154 conmon[34553]: 2021-07-31T02:50:56.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:55 smithi045 conmon[33655]: cluster 2021-07-31T02:50:54.711902+0000 mgr.x (mgr.24542) 2731 : cluster [DBG] pgmap v3918: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:56.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:55 smithi045 conmon[39718]: cluster 2021-07-31T02:50:54 2021-07-31T02:50:56.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:55 smithi045 conmon[39718]: .711902+0000 mgr.x (mgr.24542) 2731 : cluster [DBG] pgmap v3918: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:58.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:57 smithi154 conmon[34553]: cluster 2021-07-31T02:50:56.712243+0000 mgr.x (mgr. 2021-07-31T02:50:58.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:57 smithi154 conmon[34553]: 24542) 2732 : cluster [DBG] pgmap v3919: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:58.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:57 smithi045 conmon[33655]: cluster 2021-07-31T02:50:56.712243+0000 mgr.x (mgr.24542) 2732 : cluster [DBG] pgmap v3919: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:50:58.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:57 smithi045 conmon[39718]: cluster 2021-07-31T02:50:56 2021-07-31T02:50:58.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:57 smithi045 conmon[39718]: .712243+0000 mgr.x (mgr.24542) 2732 : cluster [DBG] pgmap v3919: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:00.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:50:59 smithi154 conmon[34553]: cluster 2021-07-31T02:50:58.712618+0000 mgr.x (mgr.24542) 2733 : cluster [DBG] pgmap v3920: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:00.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:59 smithi045 conmon[33655]: cluster 2021-07-31T02:50:58.712618+0000 mgr.x (mgr.24542) 2733 : cluster 2021-07-31T02:51:00.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:50:59 smithi045 conmon[33655]: [DBG] pgmap v3920: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:00.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:59 smithi045 conmon[39718]: cluster 2021-07-31T02:50:58.712618+0000 mgr.x 2021-07-31T02:51:00.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:50:59 smithi045 conmon[39718]: (mgr.24542) 2733 : cluster [DBG] pgmap v3920: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:02.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:01 smithi154 conmon[34553]: cluster 2021-07-31T02:51:00.713191+0000 mgr.x (mgr.24542) 2734 : cluster [DBG] pgmap v3921: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:02.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:01 smithi045 conmon[39718]: cluster 2021-07-31T02:51:00.713191+0000 mgr.x (mgr.24542) 2734 : cluster [DBG] pgmap v3921: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:02.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:01 smithi045 conmon[33655]: cluster 2021-07-31T02:51:00.713191+0000 mgr.x (mgr.24542) 2734 : cluster 2021-07-31T02:51:02.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:01 smithi045 conmon[33655]: [DBG] pgmap v3921: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:04.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:03 smithi154 conmon[34553]: cluster 2021-07-31T02:51:02.713705+0000 mgr.x (mgr.24542) 2735 : cluster [DBG] pgmap v3922: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:04.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:03 smithi154 conmon[34553]: 2021-07-31T02:51:04.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:03 smithi045 conmon[39718]: cluster 2021-07-31T02:51:02.713705+0000 mgr.x (mgr.24542) 2021-07-31T02:51:04.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:03 smithi045 conmon[39718]: 2735 : cluster [DBG] pgmap v3922: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:04.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:03 smithi045 conmon[33655]: cluster 2021-07-31T02:51:02.713705+0000 mgr.x (mgr.24542) 2735 : cluster [DBG] pgmap v3922: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:06.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:05 smithi045 conmon[39718]: cluster 2021-07-31T02:51:04.714258+0000 mgr.x (mgr 2021-07-31T02:51:06.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:05 smithi045 conmon[39718]: .24542) 2736 : cluster [DBG] pgmap v3923: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:06.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:05 smithi045 conmon[33655]: cluster 2021-07-31T02:51:04.714258+0000 mgr.x (mgr.24542) 2736 : 2021-07-31T02:51:06.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:05 smithi045 conmon[33655]: cluster [DBG] pgmap v3923: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:06.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:05 smithi154 conmon[34553]: cluster 2021-07-31T02:51:04.714258+0000 mgr.x (mgr.24542) 2736 : cluster [DBG] pgmap v3923: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:08.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:07 smithi045 conmon[33655]: cluster 2021-07-31T02:51:06.714773+0000 mgr.x (mgr.24542) 2737 2021-07-31T02:51:08.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:07 smithi045 conmon[33655]: : cluster [DBG] pgmap v3924: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:08.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:07 smithi045 conmon[39718]: cluster 2021-07-31T02:51:06.714773+0000 mgr.x (mgr. 2021-07-31T02:51:08.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:07 smithi045 conmon[39718]: 24542) 2737 : cluster [DBG] pgmap v3924: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:08.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:07 smithi154 conmon[34553]: cluster 2021-07-31T02:51:06.714773+0000 mgr.x (mgr.24542) 2737 : cluster [DBG] pgmap v3924: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:10.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:09 smithi154 conmon[34553]: cluster 2021-07-31T02:51:08.715166+0000 mgr.x (mgr.24542) 2738 : cluster [DBG] pgmap v3925: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:10.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:09 smithi045 conmon[39718]: cluster 2021-07-31T02:51:08.715166+0000 mgr.x ( 2021-07-31T02:51:10.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:09 smithi045 conmon[39718]: mgr.24542) 2738 : cluster [DBG] pgmap v3925: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:10.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:09 smithi045 conmon[33655]: cluster 2021-07-31T02:51:08.715166+0000 mgr.x (mgr. 2021-07-31T02:51:10.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:09 smithi045 conmon[33655]: 24542) 2738 : cluster [DBG] pgmap v3925: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:12.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:11 smithi154 conmon[34553]: cluster 2021-07-31T02:51:10.715585+0000 mgr.x (mgr.24542) 2739 : cluster [DBG] pgmap v3926: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:12.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:11 smithi045 conmon[39718]: cluster 2021-07-31T02:51:10.715585+0000 mgr.x ( 2021-07-31T02:51:12.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:11 smithi045 conmon[39718]: mgr.24542) 2739 : cluster [DBG] pgmap v3926: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:12.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:11 smithi045 conmon[33655]: cluster 2021-07-31T02:51:10.715585+0000 mgr.x (mgr.24542) 2739 : cluster [DBG] 2021-07-31T02:51:12.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:11 smithi045 conmon[33655]: pgmap v3926: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:14.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:13 smithi154 conmon[34553]: cluster 2021-07-31T02:51:12.716097+0000 mgr.x (mgr.24542) 2740 : cluster [DBG] pgmap v3927: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:14.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:13 smithi045 conmon[33655]: cluster 2021-07-31T02:51:12.716097 2021-07-31T02:51:14.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:13 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 2740 : cluster [DBG] pgmap v3927: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:14.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:13 smithi045 conmon[39718]: cluster 2021-07-31T02:51:12.716097+0000 mgr.x 2021-07-31T02:51:14.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:13 smithi045 conmon[39718]: (mgr.24542) 2740 : cluster [DBG] pgmap v3927: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:16.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:15 smithi045 conmon[33655]: cluster 2021-07-31T02:51:14. 2021-07-31T02:51:16.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:15 smithi045 conmon[33655]: 716651+0000 mgr.x (mgr.24542) 2741 : cluster [DBG] pgmap v3928: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:16.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:15 smithi045 conmon[39718]: cluster 2021-07-31T02:51:14.716651+0000 mgr.x 2021-07-31T02:51:16.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:15 smithi045 conmon[39718]: (mgr.24542) 2741 : cluster [DBG] pgmap v3928: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:16.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:15 smithi154 conmon[34553]: cluster 2021-07-31T02:51:14.716651+0000 mgr.x (mgr.24542) 2741 : cluster [DBG] pgmap v3928: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:18.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:17 smithi045 conmon[33655]: cluster 2021-07-31T02:51:16.717120+0000 2021-07-31T02:51:18.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:17 smithi045 conmon[33655]: mgr.x (mgr.24542) 2742 : cluster [DBG] pgmap v3929: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:18.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:17 smithi045 conmon[39718]: cluster 2021-07-31T02:51:16.717120+0000 mgr.x ( 2021-07-31T02:51:18.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:17 smithi045 conmon[39718]: mgr.24542) 2742 : cluster [DBG] pgmap v3929: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:18.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:17 smithi154 conmon[34553]: cluster 2021-07-31T02:51:16.717120+0000 mgr.x (mgr.24542) 2742 : cluster [DBG] pgmap v3929: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:20.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:19 smithi154 conmon[34553]: cluster 2021-07-31T02:51:18.717512+0000 mgr.x (mgr.24542) 2743 : cluster [DBG] 2021-07-31T02:51:20.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:19 smithi154 conmon[34553]: pgmap v3930: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:20.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:19 smithi045 conmon[33655]: cluster 2021-07-31T02:51:18.717512+0000 mgr.x ( 2021-07-31T02:51:20.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:19 smithi045 conmon[33655]: mgr.24542) 2743 : cluster [DBG] pgmap v3930: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:20.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:19 smithi045 conmon[39718]: cluster 2021-07-31T02:51:18.717512+0000 mgr.x ( 2021-07-31T02:51:20.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:19 smithi045 conmon[39718]: mgr.24542) 2743 : cluster [DBG] pgmap v3930: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:22.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:21 smithi154 conmon[34553]: cluster 2021-07-31T02:51:20.718000+0000 mgr.x (mgr.24542) 2744 : cluster [DBG] pgmap v3931: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:22.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:21 smithi154 conmon[34553]: 2021-07-31T02:51:22.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:21 smithi045 conmon[39718]: cluster 2021-07-31T02:51:20.718000+0000 2021-07-31T02:51:22.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:21 smithi045 conmon[39718]: mgr.x (mgr.24542) 2744 : cluster [DBG] pgmap v3931: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:22.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:21 smithi045 conmon[33655]: cluster 2021-07-31T02:51:20.718000+0000 mgr.x (mgr.24542) 2744 : cluster [DBG] 2021-07-31T02:51:22.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:21 smithi045 conmon[33655]: pgmap v3931: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:22.668 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:51:22.947 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:51:22.947 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 66m ago 67m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:51:22.947 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 63m ago 66m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:51:22.948 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 63m ago 70m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:51:22.948 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 66m ago 72m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:51:22.948 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 66m ago 72m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:51:22.948 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 63m ago 70m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:51:22.949 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 66m ago 70m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:51:22.949 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 66m ago 67m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:51:22.949 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 63m ago 67m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:51:22.949 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 66m ago 70m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:51:22.949 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 66m ago 69m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:51:22.950 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 66m ago 69m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:51:22.950 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 66m ago 69m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:51:22.950 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 63m ago 68m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:51:22.950 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 63m ago 68m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:51:22.951 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 63m ago 68m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:51:22.951 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 63m ago 67m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:51:22.951 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 63m ago 67m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:51:23.283 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:51:23.284 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:51:23.284 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:51:23.284 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:51:23.284 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:51:23.285 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:51:23.285 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:51:23.285 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:51:23.285 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:51:23.286 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:51:23.286 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:51:23.286 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:51:23.287 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:51:23.287 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:51:23.287 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:51:23.287 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:51:23.288 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:51:24.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:23 smithi154 conmon[34553]: audit 2021-07-31T02:51:22.656078+0000 mgr.x (mgr.24542) 2745 : audit [DBG] 2021-07-31T02:51:24.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:23 smithi154 conmon[34553]: from='client.18282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:51:24.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:23 smithi154 conmon[34553]: cluster 2021-07-31T02:51:22.718420+0000 mgr.x (mgr.24542) 2746 : cluster [DBG] pgmap v3932: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:24.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:23 smithi154 conmon[34553]: audit 2021-07-31T02:51:22.939003+0000 mgr.x (mgr.24542) 2747 : audit [DBG] from='client.18288 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:51:24.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:23 smithi154 conmon[34553]: 2021-07-31T02:51:24.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:23 smithi154 conmon[34553]: audit 2021-07-31T02:51:23.283665+0000 mon.c (mon.1 2021-07-31T02:51:24.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:23 smithi154 conmon[34553]: ) 152 : audit [DBG] from='client.? 172.21.15.45:0/1446959106' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:51:24.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:23 smithi045 conmon[39718]: audit 2021-07-31T02: 2021-07-31T02:51:24.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:23 smithi045 conmon[39718]: 51:22.656078+0000 mgr.x (mgr.24542) 2745 : audit [DBG] from='client.18282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:51:24.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:23 smithi045 conmon[39718]: cluster 2021 2021-07-31T02:51:24.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:23 smithi045 conmon[39718]: -07-31T02:51:22 2021-07-31T02:51:24.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:23 smithi045 conmon[39718]: .718420+0000 mgr.x (mgr. 2021-07-31T02:51:24.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:23 smithi045 conmon[39718]: 24542) 2746 : cluster [DBG] 2021-07-31T02:51:24.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:23 smithi045 conmon[39718]: pgmap v3932: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:24.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:23 smithi045 conmon[39718]: audit 2021 2021-07-31T02:51:24.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:23 smithi045 conmon[39718]: -07-31T02:51: 2021-07-31T02:51:24.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:23 smithi045 conmon[39718]: 22.939003+0000 mgr.x (mgr 2021-07-31T02:51:24.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:23 smithi045 conmon[39718]: .24542) 2747 : audit [DBG] 2021-07-31T02:51:24.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:23 smithi045 conmon[39718]: from='client.18288 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:51:24.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:23 smithi045 conmon[39718]: audit 2021-07-31T02:51:24.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:23 smithi045 conmon[39718]: 2021-07-31T02 2021-07-31T02:51:24.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:23 smithi045 conmon[39718]: :51:23.283665+0000 2021-07-31T02:51:24.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:23 smithi045 conmon[39718]: mon.c (mon.1) 152 : 2021-07-31T02:51:24.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:23 smithi045 conmon[39718]: audit [DBG] from='client.? 172.21.15.45:0/1446959106' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:51:24.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:23 smithi045 conmon[33655]: audit 2021-07-31T02:51:22.656078+0000 mgr.x (mgr 2021-07-31T02:51:24.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:23 smithi045 conmon[33655]: .24542) 2745 : audit [DBG] from='client.18282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:51:24.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:23 smithi045 conmon[33655]: cluster 2021-07-31T02:51:22.718420+0000 mgr.x (mgr.24542) 2746 : cluster [DBG] pgmap v3932: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:24.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:23 smithi045 conmon[33655]: audit 2021-07-31T02:51:22 2021-07-31T02:51:24.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:23 smithi045 conmon[33655]: .939003+0000 mgr.x (mgr.24542) 2747 : audit [DBG] 2021-07-31T02:51:24.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:23 smithi045 conmon[33655]: from='client.18288 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:51:24.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:23 smithi045 conmon[33655]: audit 2021-07-31T02:51:24.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:23 smithi045 conmon[33655]: 2021-07-31T02:51 2021-07-31T02:51:24.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:23 smithi045 conmon[33655]: :23.283665+0000 mon.c ( 2021-07-31T02:51:24.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:23 smithi045 conmon[33655]: mon.1) 152 : audit 2021-07-31T02:51:24.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:23 smithi045 conmon[33655]: [DBG] from='client.? 172.21.15.45:0/1446959106' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:51:26.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:25 smithi045 conmon[39718]: cluster 2021-07-31T02:51:24.718877+0000 mgr.x ( 2021-07-31T02:51:26.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:25 smithi045 conmon[39718]: mgr.24542) 2748 : cluster [DBG] pgmap v3933: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:26.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:25 smithi045 conmon[33655]: cluster 2021-07-31T02:51:24.718877+0000 mgr.x (mgr.24542) 2748 : cluster [DBG] pgmap v3933: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:26.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:25 smithi154 conmon[34553]: cluster 2021-07-31T02:51:24.718877+0000 mgr.x (mgr.24542) 2748 : cluster [DBG] pgmap v3933: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:28.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:27 smithi154 conmon[34553]: cluster 2021-07-31T02:51:26.719287+0000 mgr.x (mgr.24542) 2749 : cluster [DBG] pgmap v3934: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:28.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:27 smithi045 conmon[39718]: cluster 2021-07-31T02:51:26.719287+0000 mgr.x 2021-07-31T02:51:28.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:27 smithi045 conmon[39718]: (mgr.24542) 2749 : cluster [DBG] pgmap v3934: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:28.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:27 smithi045 conmon[33655]: cluster 2021-07-31T02:51:26.719287+0000 2021-07-31T02:51:28.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:27 smithi045 conmon[33655]: mgr.x (mgr.24542) 2749 : cluster [DBG] pgmap v3934: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:29.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:28 smithi154 conmon[34553]: audit 2021-07-31T02:51:28.192624+0000 mon.b (mon.2) 395 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:51:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:28 smithi154 conmon[34553]: 2021-07-31T02:51:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:28 smithi154 conmon[34553]: audit 2021-07-31T02:51:28.193011+0000 mon.b (mon.2) 396 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:51:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:28 smithi154 conmon[34553]: audit 2021-07-31T02:51:28.193107+0000 mon.a (mon.0) 2021-07-31T02:51:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:28 smithi154 conmon[34553]: 3215 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:51:29.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:28 smithi154 conmon[34553]: audit 2021-07-31T 2021-07-31T02:51:29.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:28 smithi154 conmon[34553]: 02:51:28.193612+0000 mon.a (mon.0) 3216 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:51:29.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:28 smithi045 conmon[33655]: audit 2021-07-31T02:51:28.192624+0000 2021-07-31T02:51:29.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:28 smithi045 conmon[33655]: mon.b (mon.2) 395 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:51:29.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:28 smithi045 conmon[33655]: audit 2021-07-31T02:51:28.193011+0000 mon.b (mon.2) 396 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:51:29.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:28 smithi045 conmon[33655]: audit 2021-07-31T 2021-07-31T02:51:29.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:28 smithi045 conmon[33655]: 02:51:28.193107+0000 mon.a (mon.0) 3215 2021-07-31T02:51:29.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:28 smithi045 conmon[33655]: : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:51:29.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:28 smithi045 conmon[33655]: audit 2021-07-31T02: 2021-07-31T02:51:29.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:28 smithi045 conmon[33655]: 51:28.193612+0000 mon.a (mon.0) 2021-07-31T02:51:29.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:28 smithi045 conmon[33655]: 3216 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:51:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:28 smithi045 conmon[39718]: audit 2021-07-31T02:51:28.192624 2021-07-31T02:51:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:28 smithi045 conmon[39718]: +0000 mon.b (mon.2) 395 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:51:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:28 smithi045 conmon[39718]: audit 2021-07-31T02:51: 2021-07-31T02:51:29.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:28 smithi045 conmon[39718]: 28.193011+0000 mon.b (mon.2) 396 : 2021-07-31T02:51:29.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:28 smithi045 conmon[39718]: audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:51:29.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:28 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T02:51:29.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:28 smithi045 conmon[39718]: T02:51:28.193107+0000 mon.a ( 2021-07-31T02:51:29.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:28 smithi045 conmon[39718]: mon.0) 3215 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:51:29.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:28 smithi045 conmon[39718]: 2021-07-31T02:51:29.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:28 smithi045 conmon[39718]: audit 2021-07-31T02:51:29.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:28 smithi045 conmon[39718]: 2021-07-31T02:51:28.193612 2021-07-31T02:51:29.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:28 smithi045 conmon[39718]: +0000 mon.a (mon.0) 3216 : audit 2021-07-31T02:51:29.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:28 smithi045 conmon[39718]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:51:30.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[34553]: cluster 2021-07-31T02:51:28.719770+0000 mgr.x (mgr.24542) 2750 : cluster [DBG] pgmap v3935: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:30.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: debug 2021-07-31T02:51:29.916+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:51:30.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:51:30.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:51:30.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: yield 2021-07-31T02:51:30.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:51:30.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:51:30.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:51:30.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:51:30.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:51:30.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:51:30.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:51:30.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:51:30.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:51:30.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: task.result() 2021-07-31T02:51:30.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:51:30.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: debug 2021-07-31T02:51:29.928+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:51:30.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:51:30.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:51:30.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: yield 2021-07-31T02:51:30.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:51:30.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:51:30.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:51:30.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:51:30.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:51:30.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:51:30.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:51:30.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:51:30.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:51:30.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: task.result() 2021-07-31T02:51:30.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:51:30.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: debug 2021-07-31T02:51:29.954+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:51:30.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:51:30.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:51:30.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: yield 2021-07-31T02:51:30.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:51:30.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:51:30.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:51:30.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:51:30.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:51:30.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:51:30.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:51:30.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:51:30.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:51:30.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: task.result() 2021-07-31T02:51:30.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:51:30.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: debug 2021-07-31T02:51:29.955+0000 7fcb9816a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:51:30.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:51:30.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:51:30.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: yield 2021-07-31T02:51:30.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:51:30.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:51:30.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:51:30.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:51:30.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:51:30.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:51:30.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:51:30.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:51:30.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:51:30.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: task.result() 2021-07-31T02:51:30.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:51:30.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: 2021-07-31T02:51:30.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:51:30.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: 2021-07-31T02:51:30.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:51:30.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:51:30.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:51:30.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:51:30.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:51:30.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:51:30.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:51:30.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:51:30.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:51:30.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:51:30.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:51:30.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:51:30.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:51:30.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:51:30.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:51:30.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:51:30.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:51:30.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:51:30.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: raise 2021-07-31T02:51:30.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:51:30.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:51:30.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:51:30.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:51:30.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:51:30.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: debug 2021-07-31T02:51:29.963+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:51:30.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:51:30.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:51:30.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: yield 2021-07-31T02:51:30.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:51:30.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:51:30.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:51:30.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:51:30.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:51:30.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:51:30.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:51:30.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:51:30.284 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:51:30.284 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: task.result() 2021-07-31T02:51:30.284 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:51:30.285 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: debug 2021-07-31T02:51:29.963+0000 7fcb97969700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:51:30.285 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:51:30.285 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:51:30.286 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: yield 2021-07-31T02:51:30.286 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:51:30.286 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:51:30.286 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:51:30.286 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:51:30.287 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:51:30.287 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:51:30.287 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:51:30.287 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:51:30.288 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:51:30.288 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: task.result() 2021-07-31T02:51:30.288 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:51:30.288 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: 2021-07-31T02:51:30.289 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:51:30.289 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: 2021-07-31T02:51:30.289 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:51:30.289 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:51:30.289 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:51:30.290 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:51:30.290 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:51:30.290 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:51:30.290 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:51:30.290 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:51:30.291 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:51:30.291 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:51:30.291 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:51:30.291 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:51:30.292 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:51:30.292 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:51:30.292 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:51:30.292 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:51:30.293 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:51:30.293 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:51:30.293 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: raise 2021-07-31T02:51:30.293 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:51:30.293 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:51:30.294 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:51:30.294 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:51:30.294 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:51:29 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:51:30.295 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:29 smithi045 conmon[33655]: cluster 2021-07-31T02:51:28.719770+0000 mgr.x (mgr 2021-07-31T02:51:30.295 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:29 smithi045 conmon[33655]: .24542) 2750 : cluster [DBG] pgmap v3935: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:30.295 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:29 smithi045 conmon[39718]: cluster 2021-07-31T02:51:28.719770+0000 mgr.x ( 2021-07-31T02:51:30.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:29 smithi045 conmon[39718]: mgr.24542) 2750 : cluster [DBG] pgmap v3935: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:31.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: audit 2021-07-31T02:51:29.872403+0000 mon.b (mon.2) 397 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:51:31.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: audit 2021-07-31T02:51:29.873912+0000 mon.b (mon.2) 398 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:51:31.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: audit 2021-07-31T02:51:29.874762+0000 mon.b (mon.2) 399 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:51:31.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:51:29.875514+0000 mgr.x (mgr.24542) 2751 : 2021-07-31T02:51:31.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:51:31.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:51: 2021-07-31T02:51:31.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: 29.918636+0000 mgr.x (mgr.24542) 2752 : cephadm 2021-07-31T02:51:31.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: [ERR] 2021-07-31T02:51:31.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:51:31.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:51:31.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: yield 2021-07-31T02:51:31.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:51:31.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:51:31.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:51:31.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:51:31.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:51:31.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:51:31.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:51:31.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:51:31.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:51:31.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:51:31.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:51:31.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:51:29.919263+0000 mgr.x (mgr.24542) 2753 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:51:31.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:51:29.930500+0000 mgr.x (mgr.24542) 2754 : cephadm [ERR] 2021-07-31T02:51:31.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:51:31.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:51:31.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: yield 2021-07-31T02:51:31.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:51:31.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:51:31.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:51:31.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:51:31.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:51:31.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:51:31.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:51:31.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:51:31.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:51:31.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:51:31.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:51:31.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:51:29.931463+0000 mgr.x (mgr.24542) 2755 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:51:31.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:51:29.956448+0000 mgr.x (mgr.24542) 2756 : cephadm 2021-07-31T02:51:31.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: [ERR] 2021-07-31T02:51:31.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:51:31.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:51:31.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: yield 2021-07-31T02:51:31.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:51:31.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:51:31.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:51:31.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:51:31.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:51:31.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:51:31.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:51:31.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:51:31.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:51:31.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:51:31.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:51:31.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:51:29.957107+0000 mgr.x (mgr.24542) 2757 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:51:31.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:51:31.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:51:31.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: yield 2021-07-31T02:51:31.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:51:31.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:51:31.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:51:31.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:51:31.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:51:31.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:51:31.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:51:31.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:51:31.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:51:31.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:51:31.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:51:31.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: 2021-07-31T02:51:31.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:51:31.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: 2021-07-31T02:51:31.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:51:31.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:51:31.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:51:31.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:51:31.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:51:31.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:51:31.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:51:31.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:51:31.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:51:31.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:51:31.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:51:31.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:51:31.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:51:31.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:51:31.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:51:31.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:51:31.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:51:31.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:51:31.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: raise 2021-07-31T02:51:31.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:51:31.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:51:31.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:51:31.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:51:31.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:51:31.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:51:29.965111+0000 mgr.x (mgr.24542) 2758 2021-07-31T02:51:31.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: : cephadm [ERR] 2021-07-31T02:51:31.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:51:31.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:51:31.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: yield 2021-07-31T02:51:31.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:51:31.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:51:31.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:51:31.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:51:31.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:51:31.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:51:31.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:51:31.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:51:31.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:51:31.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:51:31.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:51:31.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:51:29.965511+0000 mgr.x (mgr.24542) 2759 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:51:31.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:51:31.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:51:31.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: yield 2021-07-31T02:51:31.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:51:31.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:51:31.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:51:31.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:51:31.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:51:31.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:51:31.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:51:31.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:51:31.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:51:31.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:51:31.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:51:31.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: 2021-07-31T02:51:31.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:51:31.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: 2021-07-31T02:51:31.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:51:31.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:51:31.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:51:31.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:51:31.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:51:31.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:51:31.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:51:31.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:51:31.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:51:31.285 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:51:31.285 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:51:31.285 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:51:31.285 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:51:31.285 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:51:31.286 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:51:31.286 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:51:31.286 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:51:31.286 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:51:31.287 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: raise 2021-07-31T02:51:31.287 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:51:31.287 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:51:31.287 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:51:31.287 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:51:31.288 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:30 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:51:31.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: audit 2021-07-31T02:51:29.872403+0000 mon.b (mon.2) 397 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:51:31.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: audit 2021- 2021-07-31T02:51:31.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: 07-31T02:51:29.873912+0000 mon.b (mon.2) 398 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:51:31.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: audit 2021-07-31T02:51:29.874762+0000 mon.b 2021-07-31T02:51:31.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: (mon.2) 399 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:51:31.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:51:31.290 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: 2021-07 2021-07-31T02:51:31.290 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: -31T02:51:29. 2021-07-31T02:51:31.290 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: 875514+0000 mgr.x (mgr. 2021-07-31T02:51:31.290 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: 24542) 2751 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:51:31.291 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: 2021-07-31T02:51:31.291 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: cephadm 2021 2021-07-31T02:51:31.291 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: -07-31T02:51:29.918636 2021-07-31T02:51:31.291 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 2021-07-31T02:51:31.291 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: 2752 : cephadm [ERR] 2021-07-31T02:51:31.292 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:51:31.292 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:51:31.292 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: yield 2021-07-31T02:51:31.292 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:51:31.293 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:51:31.293 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:51:31.293 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:51:31.293 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:51:31.293 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:51:31.294 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:51:31.294 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:51:31.294 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:51:31.294 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:51:31.295 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:51:31.295 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:51:29.919263+0000 mgr.x (mgr. 2021-07-31T02:51:31.295 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: 24542) 2753 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:51:31.295 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:51:31.295 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: 2021-07-31T02:51:29. 2021-07-31T02:51:31.296 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: 930500+0000 mgr.x (mgr.24542) 2021-07-31T02:51:31.296 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: 2754 : cephadm [ERR] 2021-07-31T02:51:31.296 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:51:31.296 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:51:31.296 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: yield 2021-07-31T02:51:31.297 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:51:31.297 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:51:31.297 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:51:31.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: audit 2021-07-31T02:51:29.872403+0000 mon.b (mon. 2021-07-31T02:51:31.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: 2) 397 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:51:31.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: audit 2021-07-31T02:51: 2021-07-31T02:51:31.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: 29.873912+0000 mon.b (mon.2) 398 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:51:31.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: audit 2021-07-31T02:51:31.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: 2021-07-31T02:51:29. 2021-07-31T02:51:31.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: 874762+0000 mon.b (mon.2) 399 : 2021-07-31T02:51:31.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:51:31.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: cephadm 2021-07-31 2021-07-31T02:51:31.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: T02:51:29.875514+0000 mgr.x 2021-07-31T02:51:31.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: (mgr.24542) 2751 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:51:31.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: 2021-07-31T02:51:31.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: cephadm 2021-07-31T02: 2021-07-31T02:51:31.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: 51:29.918636+0000 mgr.x (mgr.24542 2021-07-31T02:51:31.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: ) 2752 : cephadm [ERR] 2021-07-31T02:51:31.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:51:31.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:51:31.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: yield 2021-07-31T02:51:31.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:51:31.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:51:31.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:51:31.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:51:31.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:51:31.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:51:31.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:51:31.304 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:51:31.304 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:51:31.304 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:51:31.304 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:51:31.305 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:51:29.919263+0000 mgr.x (mgr.24542) 2753 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:51:31.305 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: cephadm 2021-07 2021-07-31T02:51:31.305 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: -31T02:51:29.930500+0000 mgr.x (mgr. 2021-07-31T02:51:31.305 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: 24542) 2754 : cephadm [ERR] 2021-07-31T02:51:31.305 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:51:31.306 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:51:31.306 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: yield 2021-07-31T02:51:31.306 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:51:31.306 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:51:31.307 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:51:31.307 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:51:31.307 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:51:31.307 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:51:31.307 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:51:31.308 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:51:31.308 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:51:31.308 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:51:31.308 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:51:31.308 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:51:29.931463+0000 mgr.x (mgr.24542) 2755 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:51:31.309 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: cephadm 2021-07- 2021-07-31T02:51:31.309 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: 31T02:51:29.956448+0000 mgr.x (mgr. 2021-07-31T02:51:31.309 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: 24542) 2756 : cephadm [ERR] 2021-07-31T02:51:31.309 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:51:31.310 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:51:31.310 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: yield 2021-07-31T02:51:31.310 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:51:31.310 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:51:31.310 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:51:31.311 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:51:31.311 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:51:31.311 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:51:31.311 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:51:31.312 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:51:31.312 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:51:31.312 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:51:31.312 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:51:31.312 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:51:29.957107+0000 mgr.x (mgr.24542) 2757 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:51:31.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:51:31.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:51:31.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: yield 2021-07-31T02:51:31.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:51:31.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:51:31.314 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:51:31.314 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:51:31.314 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:51:31.314 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:51:31.315 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:51:31.315 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:51:31.315 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:51:31.315 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:51:31.315 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:51:31.316 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: 2021-07-31T02:51:31.316 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:51:31.316 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:51:31.317 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:51:31.317 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:51:31.317 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:51:31.317 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:51:31.318 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:51:31.318 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:51:31.318 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:51:29.931463+0000 mgr.x (mgr 2021-07-31T02:51:31.318 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: .24542) 2755 : cephadm [INF] 2021-07-31T02:51:31.318 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:51:31.319 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:51:31.319 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: 2021-07- 2021-07-31T02:51:31.319 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: 31T02:51:29. 2021-07-31T02:51:31.319 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: 956448+0000 mgr.x (mgr.24542 2021-07-31T02:51:31.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: ) 2756 : cephadm [ERR] 2021-07-31T02:51:31.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:51:31.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:51:31.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: yield 2021-07-31T02:51:31.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:51:31.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:51:31.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:51:31.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:51:31.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:51:31.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:51:31.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:51:31.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:51:31.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:51:31.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:51:31.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:51:31.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: 2021-07-31T02:51:31.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:51:29.957107+0000 2021-07-31T02:51:31.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: mgr.x (mgr.24542) 2757 : cephadm [ERR] 2021-07-31T02:51:31.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:51:31.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:51:31.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:51:31.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: yield 2021-07-31T02:51:31.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:51:31.325 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:51:31.325 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:51:31.325 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:51:31.325 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:51:31.325 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:51:31.326 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:51:31.326 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:51:31.326 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:51:31.326 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:51:31.326 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:51:31.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: 2021-07-31T02:51:31.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:51:31.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: 2021-07-31T02:51:31.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:51:31.328 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:51:31.328 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:51:31.328 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:51:31.328 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:51:31.328 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:51:31.329 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:51:31.329 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:51:31.329 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:51:31.329 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:51:31.329 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:51:31.330 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:51:31.330 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:51:31.330 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:51:31.330 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:51:31.331 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:51:31.331 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:51:31.331 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:51:31.331 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: raise 2021-07-31T02:51:31.331 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:51:31.332 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:51:31.332 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:51:31.332 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:51:31.332 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:51:31.332 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:51:29.965111+0000 mgr.x (mgr.24542) 2758 : cephadm [ERR] 2021-07-31T02:51:31.333 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:51:31.333 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:51:31.333 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: yield 2021-07-31T02:51:31.333 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:51:31.333 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:51:31.334 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:51:31.334 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:51:31.334 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:51:31.334 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:51:31.335 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:51:31.335 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: 2021-07-31T02:51:31.335 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:51:31.335 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:51:31.336 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:51:31.336 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:51:31.336 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:51:31.336 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:51:31.337 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:51:31.337 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:51:31.337 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:51:31.337 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:51:31.337 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:51:31.338 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:51:31.338 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:51:31.338 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:51:31.338 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:51:31.339 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:51:31.339 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:51:31.339 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:51:31.339 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: raise 2021-07-31T02:51:31.339 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:51:31.340 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:51:31.340 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:51:31.340 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:51:31.340 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:51:31.340 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:51:31.341 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: 2021-07-31T02:51:29.965111+0000 mgr.x (mgr.24542) 2758 : cephadm [ERR] 2021-07-31T02:51:31.341 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:51:31.341 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:51:31.341 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: yield 2021-07-31T02:51:31.342 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:51:31.342 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:51:31.342 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:51:31.342 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:51:31.342 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:51:31.343 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:51:31.343 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:51:31.343 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:51:31.343 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:51:31.343 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:51:31.344 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:51:31.344 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:51:31.344 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:51:31.345 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:51:31.345 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:51:31.345 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:51:29.965511+0000 mgr.x (mgr.24542) 2759 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:51:31.345 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:51:31.346 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:51:31.346 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: yield 2021-07-31T02:51:31.346 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:51:31.346 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:51:31.346 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:51:31.347 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:51:31.347 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:51:31.347 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:51:31.348 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:51:31.348 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:51:31.348 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:51:31.348 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:51:31.348 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:51:31.349 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: 2021-07-31T02:51:31.349 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:51:31.349 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: 2021-07-31T02:51:31.349 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:51:31.350 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:51:31.350 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:51:31.350 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:51:31.350 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:51:31.350 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:51:31.351 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:51:31.351 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:51:31.351 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:51:31.351 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:51:31.352 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:51:31.352 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:51:31.352 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:51:31.352 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:51:31.352 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:51:31.353 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:51:31.353 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:51:31.353 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:51:31.353 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: raise 2021-07-31T02:51:31.354 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:51:31.354 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:51:29.965511+0000 mgr.x (mgr.24542) 2759 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:51:31.354 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:51:31.355 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:51:31.355 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: yield 2021-07-31T02:51:31.355 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:51:31.355 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:51:31.355 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:51:31.356 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:51:31.356 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:51:31.356 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:51:31.356 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:51:31.357 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:51:31.357 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:51:31.357 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:51:31.357 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:51:31.357 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: 2021-07-31T02:51:31.358 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:51:31.358 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: 2021-07-31T02:51:31.358 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:51:31.358 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:51:31.358 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:51:31.359 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:51:31.359 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:51:31.359 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:51:31.359 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:51:31.360 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:51:31.360 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:51:31.360 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:51:31.360 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:51:31.360 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:51:31.361 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:51:31.361 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:51:31.361 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:51:31.361 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:51:31.361 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:51:31.362 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:51:31.362 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: raise 2021-07-31T02:51:31.362 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:51:31.362 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:51:31.363 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:51:31.363 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:51:31.363 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:51:31.363 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:51:31.364 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:51:31.364 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:51:31.364 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:51:31.365 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:30 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:51:32.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:31 smithi154 conmon[34553]: cluster 2021-07-31T02:51:30.720356+0000 mgr.x (mgr.24542) 2760 : cluster [DBG] pgmap v3936: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:32.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:31 smithi045 conmon[39718]: cluster 2021-07-31T02:51:30.720356+0000 mgr.x (mgr. 2021-07-31T02:51:32.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:31 smithi045 conmon[39718]: 24542) 2760 : cluster [DBG] pgmap v3936: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:32.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:31 smithi045 conmon[33655]: cluster 2021-07-31T02:51:30.720356+0000 mgr.x (mgr.24542 2021-07-31T02:51:32.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:31 smithi045 conmon[33655]: ) 2760 : cluster [DBG] pgmap v3936: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:34.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:33 smithi154 conmon[34553]: cluster 2021-07-31T02:51:32.720770+0000 mgr.x (mgr.24542) 2761 : cluster [DBG] 2021-07-31T02:51:34.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:33 smithi154 conmon[34553]: pgmap v3937: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:34.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:33 smithi045 conmon[39718]: cluster 2021-07-31T02:51:32.720770+0000 mgr.x 2021-07-31T02:51:34.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:33 smithi045 conmon[39718]: (mgr.24542) 2761 : cluster [DBG] pgmap v3937: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:34.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:33 smithi045 conmon[33655]: cluster 2021-07-31T02:51:32.720770+0000 mgr.x (mgr.24542) 2761 : cluster [DBG] pgmap v3937: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:36.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:35 smithi154 conmon[34553]: cluster 2021-07-31T02:51:34.721395+0000 mgr.x (mgr.24542) 2762 : cluster [DBG] pgmap v3938: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:36.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:35 smithi045 conmon[39718]: cluster 2021-07-31T02:51:34 2021-07-31T02:51:36.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:35 smithi045 conmon[39718]: .721395+0000 mgr.x (mgr.24542) 2762 : cluster [DBG] pgmap v3938: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:36.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:35 smithi045 conmon[33655]: cluster 2021-07-31T02:51:34.721395+0000 mgr.x (mgr.24542) 2021-07-31T02:51:36.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:35 smithi045 conmon[33655]: 2762 : cluster [DBG] pgmap v3938: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:38.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:37 smithi154 conmon[34553]: cluster 2021-07-31T02:51:36.721779+0000 mgr.x (mgr.24542) 2763 : cluster [DBG] pgmap v3939: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:38.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:37 smithi154 conmon[34553]: 2021-07-31T02:51:38.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:37 smithi045 conmon[39718]: cluster 2021-07-31T02:51:36.721779+0000 mgr.x (mgr.24542) 2763 2021-07-31T02:51:38.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:37 smithi045 conmon[39718]: : cluster [DBG] pgmap v3939: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:38.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:37 smithi045 conmon[33655]: cluster 2021-07-31T02:51:36.721779+0000 mgr.x (mgr.24542) 2021-07-31T02:51:38.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:37 smithi045 conmon[33655]: 2763 : cluster [DBG] pgmap v3939: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:40.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:39 smithi154 conmon[34553]: cluster 2021-07-31T02:51:38.722221+0000 mgr.x (mgr.24542) 2764 : 2021-07-31T02:51:40.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:39 smithi154 conmon[34553]: cluster [DBG] pgmap v3940: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:40.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:39 smithi045 conmon[39718]: cluster 2021-07-31T02:51:38.722221 2021-07-31T02:51:40.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:39 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 2764 : cluster [DBG] pgmap v3940: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:40.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:39 smithi045 conmon[33655]: cluster 2021-07-31T02:51:38.722221+0000 mgr.x (mgr.24542) 2764 : cluster [DBG] pgmap v3940: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:42.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:41 smithi154 conmon[34553]: cluster 2021-07-31T02:51:40.722833+0000 mgr.x (mgr.24542) 2765 : cluster [DBG] 2021-07-31T02:51:42.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:41 smithi154 conmon[34553]: pgmap v3941: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:42.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:41 smithi045 conmon[33655]: cluster 2021-07-31T02:51:40.722833+0000 mgr.x (mgr.24542) 2765 : cluster [DBG] pgmap v3941: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:42.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:41 smithi045 conmon[39718]: cluster 2021-07-31T02:51:40.722833+0000 mgr.x (mgr.24542) 2021-07-31T02:51:42.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:41 smithi045 conmon[39718]: 2765 : cluster [DBG] pgmap v3941: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:44.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:43 smithi154 conmon[34553]: cluster 2021-07-31T02:51:42.723184+0000 mgr.x (mgr.24542) 2766 : cluster [DBG] pgmap v3942: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:44.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:43 smithi045 conmon[33655]: cluster 2021-07-31T02:51:42.723184+0000 mgr.x (mgr.24542 2021-07-31T02:51:44.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:43 smithi045 conmon[33655]: ) 2766 : cluster [DBG] pgmap v3942: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:44.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:43 smithi045 conmon[39718]: cluster 2021-07-31T02:51:42.723184+0000 mgr.x (mgr.24542 2021-07-31T02:51:44.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:43 smithi045 conmon[39718]: ) 2766 : cluster [DBG] pgmap v3942: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:46.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:45 smithi154 conmon[34553]: cluster 2021-07-31T02:51:44.723644+0000 mgr.x (mgr.24542) 2767 : cluster [DBG] pgmap v3943: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:46.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:45 smithi045 conmon[39718]: cluster 2021-07-31T02:51:44.723644+0000 mgr.x (mgr.24542) 2767 : cluster [DBG] pgmap v3943: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:46.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:45 smithi045 conmon[33655]: cluster 2021-07-31T02:51:44.723644+0000 mgr.x (mgr.24542) 2767 : cluster 2021-07-31T02:51:46.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:45 smithi045 conmon[33655]: [DBG] pgmap v3943: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:48.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:47 smithi154 conmon[34553]: cluster 2021-07-31T02:51:46.724176+0000 mgr.x (mgr.24542) 2768 : cluster [DBG] pgmap v3944: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:48.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:47 smithi045 conmon[39718]: cluster 2021-07-31T02:51:46.724176+0000 mgr.x ( 2021-07-31T02:51:48.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:47 smithi045 conmon[39718]: mgr.24542) 2768 : cluster [DBG] pgmap v3944: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:48.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:47 smithi045 conmon[33655]: cluster 2021-07-31T02:51:46.724176+0000 mgr.x (mgr.24542) 2768 : cluster [DBG] pgmap v3944: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:48.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:47 smithi045 conmon[33655]: 2021-07-31T02:51:50.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:49 smithi154 conmon[34553]: cluster 2021-07-31T02:51:48.724658+0000 mgr.x (mgr.24542) 2769 : cluster [DBG] pgmap v3945: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:50.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:49 smithi045 conmon[39718]: cluster 2021-07-31T02:51:48 2021-07-31T02:51:50.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:49 smithi045 conmon[39718]: .724658+0000 mgr.x (mgr.24542) 2769 : cluster [DBG] pgmap v3945: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:50.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:49 smithi045 conmon[33655]: cluster 2021-07-31T02:51:48.724658+0000 mgr.x (mgr.24542) 2769 : cluster [DBG] pgmap v3945: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:52.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:51 smithi154 conmon[34553]: cluster 2021-07-31T02:51:50.725271+0000 mgr.x (mgr.24542) 2770 : cluster [DBG] pgmap v3946: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:52.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:51 smithi045 conmon[33655]: cluster 2021-07-31T02:51:50.725271+0000 mgr.x (mgr.24542 2021-07-31T02:51:52.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:51 smithi045 conmon[33655]: ) 2770 : cluster [DBG] pgmap v3946: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:52.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:51 smithi045 conmon[39718]: cluster 2021-07-31T02:51:50.725271+0000 mgr.x 2021-07-31T02:51:52.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:51 smithi045 conmon[39718]: (mgr.24542) 2770 : cluster [DBG] pgmap v3946: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:53.597 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:51:53.876 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:51:53.876 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 66m ago 67m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:51:53.876 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 64m ago 67m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:51:53.876 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 64m ago 70m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:51:53.877 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 66m ago 72m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:51:53.877 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 66m ago 73m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:51:53.877 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 64m ago 71m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:51:53.877 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 66m ago 71m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:51:53.878 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 66m ago 67m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:51:53.878 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 64m ago 67m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:51:53.878 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 66m ago 70m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:51:53.878 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 66m ago 70m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:51:53.878 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 66m ago 69m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:51:53.879 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 66m ago 69m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:51:53.879 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 64m ago 69m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:51:53.879 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 64m ago 68m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:51:53.879 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 64m ago 68m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:51:53.880 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 64m ago 68m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:51:53.880 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 64m ago 67m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:51:54.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:53 smithi154 conmon[34553]: cluster 2021-07-31T02:51:52.725784+0000 mgr.x (mgr.24542) 2771 : cluster [DBG] pgmap v3947: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:54.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:53 smithi045 conmon[39718]: cluster 2021-07-31T02:51:52.725784+0000 2021-07-31T02:51:54.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:53 smithi045 conmon[39718]: mgr.x (mgr.24542) 2771 : cluster [DBG] pgmap v3947: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:54.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:53 smithi045 conmon[33655]: cluster 2021-07-31T02:51:52. 2021-07-31T02:51:54.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:53 smithi045 conmon[33655]: 725784+0000 mgr.x (mgr.24542) 2771 : cluster [DBG] pgmap v3947: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:54.213 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:51:54.214 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:51:54.214 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:51:54.214 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:51:54.214 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:51:54.214 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:51:54.215 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:51:54.215 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:51:54.215 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:51:54.215 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:51:54.216 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:51:54.216 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:51:54.216 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:51:54.216 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:51:54.216 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:51:54.217 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:51:54.217 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:51:55.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:54 smithi154 conmon[34553]: audit 2021-07-31T02:51:53.585123+0000 mgr.x (mgr.24542) 2772 : audit [DBG] from='client.28240 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:51:55.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:54 smithi154 conmon[34553]: audit 2021-07-31T02:51:53.868165+0000 mgr.x (mgr.24542) 2773 : audit [DBG] from='client.18306 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:51:55.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:54 smithi154 conmon[34553]: audit 2021-07-31T02:51:54.213461+0000 mon.c (mon 2021-07-31T02:51:55.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:54 smithi154 conmon[34553]: .1) 153 : audit [DBG] from='client.? 172.21.15.45:0/3304584125' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:51:55.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:54 smithi045 conmon[39718]: audit 2021-07-31T02:51:53.585123+0000 mgr.x (mgr 2021-07-31T02:51:55.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:54 smithi045 conmon[39718]: .24542) 2772 : audit [DBG] from='client.28240 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:51:55.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:54 smithi045 conmon[39718]: audit 2021-07-31T02:51:53.868165+0000 mgr.x (mgr. 2021-07-31T02:51:55.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:54 smithi045 conmon[39718]: 24542) 2773 : audit [DBG] from='client.18306 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:51:55.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:54 smithi045 conmon[39718]: audit 2021-07-31T02:51:54 2021-07-31T02:51:55.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:54 smithi045 conmon[39718]: .213461+0000 mon.c (mon.1) 153 : 2021-07-31T02:51:55.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:54 smithi045 conmon[39718]: audit [DBG] from='client.? 172.21.15.45:0/3304584125' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:51:55.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:54 smithi045 conmon[33655]: audit 2021-07-31T02:51:53 2021-07-31T02:51:55.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:54 smithi045 conmon[33655]: .585123+0000 mgr.x (mgr.24542) 2772 : audit [DBG] from='client.28240 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:51:55.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:54 smithi045 conmon[33655]: audit 2021-07- 2021-07-31T02:51:55.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:54 smithi045 conmon[33655]: 31T02:51:53.868165+0000 mgr.x (mgr.24542) 2773 : audit [DBG] from='client.18306 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:51:55.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:54 smithi045 conmon[33655]: audit 2021-07-31T02:51:55.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:54 smithi045 conmon[33655]: 2021-07-31T02:51:54.213461+0000 mon.c (mon.1) 153 : audit [DBG] from='client.? 172.21.15.45:0/3304584125' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:51:56.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:55 smithi154 conmon[34553]: cluster 2021-07-31T02:51:54.726354+0000 mgr.x (mgr.24542) 2774 : cluster [DBG] 2021-07-31T02:51:56.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:55 smithi154 conmon[34553]: pgmap v3948: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:56.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:55 smithi045 conmon[33655]: cluster 2021-07-31T02:51:54.726354+0000 mgr.x (mgr. 2021-07-31T02:51:56.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:55 smithi045 conmon[33655]: 24542) 2774 : cluster [DBG] pgmap v3948: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:56.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:55 smithi045 conmon[39718]: cluster 2021-07-31T02:51:54.726354+0000 mgr.x (mgr.24542 2021-07-31T02:51:56.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:55 smithi045 conmon[39718]: ) 2774 : cluster [DBG] pgmap v3948: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:58.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:57 smithi154 conmon[34553]: cluster 2021-07-31T02:51:56.726767+0000 mgr.x (mgr.24542) 2775 : cluster [DBG] pgmap v3949: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:58.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:57 smithi045 conmon[33655]: cluster 2021-07-31T02:51:56.726767+0000 mgr.x (mgr.24542) 2775 : cluster [DBG] pgmap v3949: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:51:58.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:57 smithi045 conmon[33655]: 2021-07-31T02:51:58.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:57 smithi045 conmon[39718]: cluster 2021-07-31T02:51:56.726767+0000 mgr.x 2021-07-31T02:51:58.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:57 smithi045 conmon[39718]: (mgr.24542) 2775 : cluster [DBG] pgmap v3949: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:00.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:51:59 smithi154 conmon[34553]: cluster 2021-07-31T02:51:58.727154+0000 mgr.x (mgr.24542) 2776 : cluster [DBG] pgmap v3950: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:00.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:59 smithi045 conmon[39718]: cluster 2021-07-31T02:51:58.727154+0000 2021-07-31T02:52:00.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:51:59 smithi045 conmon[39718]: mgr.x (mgr.24542) 2776 : cluster [DBG] pgmap v3950: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:00.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:59 smithi045 conmon[33655]: cluster 2021-07-31T02:51:58.727154+0000 mgr.x (mgr.24542) 2776 : cluster [DBG] pgmap v3950: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:00.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:51:59 smithi045 conmon[33655]: 2021-07-31T02:52:02.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:01 smithi154 conmon[34553]: cluster 2021-07-31T02:52:00.727669+0000 mgr.x (mgr.24542) 2777 : cluster [DBG] pgmap v3951: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:02.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:01 smithi045 conmon[39718]: cluster 2021-07-31T02:52:00.727669+0000 mgr.x ( 2021-07-31T02:52:02.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:01 smithi045 conmon[39718]: mgr.24542) 2777 : cluster [DBG] pgmap v3951: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:02.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:01 smithi045 conmon[33655]: cluster 2021-07-31T02:52:00.727669+0000 mgr.x (mgr.24542) 2777 2021-07-31T02:52:02.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:01 smithi045 conmon[33655]: : cluster [DBG] pgmap v3951: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:04.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:03 smithi154 conmon[34553]: cluster 2021-07-31T02:52:02.728032+0000 mgr.x (mgr.24542) 2778 : cluster [DBG] 2021-07-31T02:52:04.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:03 smithi154 conmon[34553]: pgmap v3952: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:04.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:03 smithi045 conmon[33655]: cluster 2021-07-31T02:52:02.728032+0000 mgr.x (mgr.24542 2021-07-31T02:52:04.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:03 smithi045 conmon[33655]: ) 2778 : cluster [DBG] pgmap v3952: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:04.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:03 smithi045 conmon[39718]: cluster 2021-07-31T02:52:02.728032+0000 mgr.x (mgr 2021-07-31T02:52:04.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:03 smithi045 conmon[39718]: .24542) 2778 : cluster [DBG] pgmap v3952: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:06.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:05 smithi154 conmon[34553]: cluster 2021-07-31T02:52:04.728530+0000 mgr.x (mgr.24542) 2779 : cluster [DBG] pgmap v3953: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:06.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:05 smithi045 conmon[39718]: cluster 2021-07-31T02:52:04.728530+0000 mgr.x (mgr.24542) 2779 : cluster [DBG] pgmap v3953: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:06.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:05 smithi045 conmon[33655]: cluster 2021-07-31T02:52:04.728530+0000 mgr.x (mgr.24542) 2779 2021-07-31T02:52:06.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:05 smithi045 conmon[33655]: : cluster [DBG] pgmap v3953: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:08.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:07 smithi154 conmon[34553]: cluster 2021-07-31T02:52:06.729059+0000 mgr.x (mgr.24542) 2780 : cluster [DBG] pgmap v3954: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:08.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:07 smithi045 conmon[39718]: cluster 2021-07-31T02:52:06.729059+0000 mgr.x (mgr.24542) 2021-07-31T02:52:08.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:07 smithi045 conmon[39718]: 2780 : cluster [DBG] pgmap v3954: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:08.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:07 smithi045 conmon[33655]: cluster 2021-07-31T02:52:06.729059+0000 mgr.x (mgr.24542) 2780 : cluster 2021-07-31T02:52:08.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:07 smithi045 conmon[33655]: [DBG] pgmap v3954: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:10.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:09 smithi154 conmon[34553]: cluster 2021-07-31T02:52:08.729543+0000 mgr.x (mgr.24542) 2781 : cluster [DBG] pgmap v3955: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:10.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:09 smithi045 conmon[39718]: cluster 2021-07-31T02:52:08.729543+0000 mgr.x 2021-07-31T02:52:10.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:09 smithi045 conmon[39718]: (mgr.24542) 2781 : cluster [DBG] pgmap v3955: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:10.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:09 smithi045 conmon[33655]: cluster 2021-07-31T02:52:08.729543+0000 mgr.x (mgr.24542) 2021-07-31T02:52:10.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:09 smithi045 conmon[33655]: 2781 : cluster [DBG] pgmap v3955: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:12.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:11 smithi154 conmon[34553]: cluster 2021-07-31T02:52:10.730058+0000 mgr.x (mgr.24542) 2782 : cluster [DBG] pgmap v3956: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:12.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:11 smithi045 conmon[39718]: cluster 2021-07-31T02:52:10.730058+0000 mgr.x (mgr.24542) 2782 : cluster [DBG] pgmap v3956: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:12.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:11 smithi045 conmon[33655]: cluster 2021-07-31T02:52:10.730058+0000 mgr.x (mgr.24542 2021-07-31T02:52:12.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:11 smithi045 conmon[33655]: ) 2782 : cluster [DBG] pgmap v3956: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:14.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:13 smithi154 conmon[34553]: cluster 2021-07-31T02:52:12.730460+0000 mgr.x (mgr.24542) 2783 : cluster [DBG] pgmap v3957: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:14.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:13 smithi154 conmon[34553]: 2021-07-31T02:52:14.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:13 smithi045 conmon[39718]: cluster 2021-07-31T02:52:12.730460+0000 mgr.x 2021-07-31T02:52:14.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:13 smithi045 conmon[39718]: (mgr.24542) 2783 : cluster [DBG] pgmap v3957: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:14.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:13 smithi045 conmon[33655]: cluster 2021-07-31T02:52:12.730460+0000 mgr.x (mgr.24542) 2783 : cluster [DBG] 2021-07-31T02:52:14.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:13 smithi045 conmon[33655]: pgmap v3957: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:16.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:15 smithi154 conmon[34553]: cluster 2021-07-31T02:52:14.730873+0000 mgr.x (mgr.24542) 2784 : cluster [DBG] pgmap v3958: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:16.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:15 smithi045 conmon[33655]: cluster 2021-07-31T02:52:14.730873+0000 mgr.x (mgr.24542) 2784 : 2021-07-31T02:52:16.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:15 smithi045 conmon[33655]: cluster [DBG] pgmap v3958: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:16.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:15 smithi045 conmon[39718]: cluster 2021-07-31T02:52: 2021-07-31T02:52:16.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:15 smithi045 conmon[39718]: 14.730873+0000 mgr.x (mgr.24542) 2784 : cluster [DBG] pgmap v3958: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:18.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:17 smithi154 conmon[34553]: cluster 2021-07-31T02:52:16.731288+0000 mgr.x (mgr.24542) 2785 : cluster [DBG] pgmap v3959: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:18.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:17 smithi045 conmon[33655]: cluster 2021-07-31T02:52:16.731288+0000 mgr.x (mgr.24542) 2785 2021-07-31T02:52:18.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:17 smithi045 conmon[33655]: : cluster [DBG] pgmap v3959: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:18.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:17 smithi045 conmon[39718]: cluster 2021-07-31T02:52:16.731288+0000 mgr.x (mgr.24542) 2785 : cluster [DBG] pgmap v3959: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:20.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:19 smithi154 conmon[34553]: cluster 2021-07-31T02:52:18.731734+0000 mgr.x (mgr.24542) 2786 : cluster [DBG] pgmap v3960: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:20.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:19 smithi045 conmon[39718]: cluster 2021-07-31T02:52:18.731734+0000 mgr.x ( 2021-07-31T02:52:20.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:19 smithi045 conmon[39718]: mgr.24542) 2786 : cluster [DBG] pgmap v3960: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:20.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:19 smithi045 conmon[33655]: cluster 2021-07-31T02:52:18.731734+0000 mgr.x (mgr.24542) 2786 : 2021-07-31T02:52:20.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:19 smithi045 conmon[33655]: cluster [DBG] pgmap v3960: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:22.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:21 smithi154 conmon[34553]: cluster 2021-07-31T02:52:20.732180+0000 mgr.x (mgr.24542) 2787 : cluster [DBG] pgmap v3961: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:22.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:21 smithi045 conmon[39718]: cluster 2021-07-31T02:52:20.732180+0000 mgr.x ( 2021-07-31T02:52:22.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:21 smithi045 conmon[39718]: mgr.24542) 2787 : cluster [DBG] pgmap v3961: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:22.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:21 smithi045 conmon[33655]: cluster 2021-07-31T02:52:20.732180 2021-07-31T02:52:22.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:21 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 2787 : cluster [DBG] pgmap v3961: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:24.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:23 smithi154 conmon[34553]: cluster 2021-07-31T02:52:22.732637+0000 mgr.x (mgr.24542) 2788 : cluster [DBG] pgmap v3962: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:24.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:23 smithi045 conmon[39718]: cluster 2021-07-31T02:52:22 2021-07-31T02:52:24.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:23 smithi045 conmon[39718]: .732637+0000 mgr.x (mgr.24542) 2788 : cluster [DBG] pgmap v3962: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:24.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:23 smithi045 conmon[33655]: cluster 2021-07-31T02:52:22.732637+0000 mgr.x (mgr.24542) 2788 : 2021-07-31T02:52:24.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:23 smithi045 conmon[33655]: cluster [DBG] pgmap v3962: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:24.528 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:52:24.805 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:52:24.805 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 67m ago 68m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:52:24.805 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 64m ago 67m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:52:24.806 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 64m ago 71m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:52:24.806 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 67m ago 73m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:52:24.806 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 67m ago 73m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:52:24.806 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 64m ago 71m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:52:24.807 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 67m ago 71m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:52:24.807 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 67m ago 68m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:52:24.807 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 64m ago 68m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:52:24.807 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 67m ago 71m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:52:24.808 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 67m ago 70m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:52:24.808 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 67m ago 70m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:52:24.808 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 67m ago 70m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:52:24.808 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 64m ago 69m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:52:24.808 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 64m ago 69m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:52:24.809 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 64m ago 69m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:52:24.809 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 64m ago 68m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:52:24.809 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 64m ago 68m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:52:25.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:24 smithi154 conmon[34553]: audit 2021-07-31T02:52:24.516007+0000 mgr.x (mgr.24542) 2789 : audit [DBG] from='client.28258 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:52:25.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:24 smithi045 conmon[33655]: audit 2021-07-31T02:52:24.516007+0000 mgr.x (mgr.24542) 2789 2021-07-31T02:52:25.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:24 smithi045 conmon[33655]: : audit [DBG] from='client.28258 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:52:25.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:24 smithi045 conmon[39718]: audit 2021-07-31T02:52:24.516007 2021-07-31T02:52:25.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:24 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 2789 : audit [DBG] from='client.28258 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:52:25.138 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:52:25.139 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:52:25.139 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:52:25.139 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:52:25.140 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:52:25.140 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:52:25.140 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:52:25.140 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:52:25.140 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:52:25.141 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:52:25.141 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:52:25.141 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:52:25.141 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:52:25.142 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:52:25.142 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:52:25.142 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:52:25.142 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:52:26.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:25 smithi154 conmon[34553]: cluster 2021-07-31T02:52:24.733096+0000 mgr.x (mgr.24542) 2790 : cluster [DBG] pgmap v3963: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:26.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:25 smithi154 conmon[34553]: 2021-07-31T02:52:26.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:25 smithi154 conmon[34553]: audit 2021-07-31T02:52:24.797391+0000 mgr.x (mgr.24542) 2791 : audit [DBG] from='client.28264 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:52:26.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:25 smithi154 conmon[34553]: audit 2021-07-31T02:52:25.138803+0000 mon.c (mon. 2021-07-31T02:52:26.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:25 smithi154 conmon[34553]: 1) 154 : audit [DBG] from='client.? 172.21.15.45:0/74214998' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:52:26.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:25 smithi045 conmon[33655]: cluster 2021-07-31T02:52:24.733096+0000 mgr.x (mgr.24542) 2021-07-31T02:52:26.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:25 smithi045 conmon[33655]: 2790 : cluster [DBG] pgmap v3963: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:26.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:25 smithi045 conmon[33655]: audit 2021-07-31T02:52:24.797391+0000 mgr.x (mgr.24542) 2791 : audit [DBG] from='client.28264 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:52:26.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:25 smithi045 conmon[33655]: audit 2021-07-31T02:52:26.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:25 smithi045 conmon[33655]: 2021-07-31T02:52:25.138803+0000 mon.c (mon.1) 154 : audit [DBG] from='client.? 172.21.15.45:0/74214998' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:52:26.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:25 smithi045 conmon[39718]: cluster 2021-07-31T02:52:24.733096+0000 mgr.x ( 2021-07-31T02:52:26.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:25 smithi045 conmon[39718]: mgr.24542) 2790 : cluster [DBG] pgmap v3963: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:26.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:25 smithi045 conmon[39718]: audit 2021-07-31T02:52:24.797391+0000 mgr.x (mgr.24542) 2791 : audit [DBG] from='client.28264 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:52:26.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:25 smithi045 conmon[39718]: 2021-07-31T02:52:26.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:25 smithi045 conmon[39718]: audit 2021-07-31T02:52:25.138803+0000 mon.c (mon.1) 154 : audit 2021-07-31T02:52:26.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:25 smithi045 conmon[39718]: [DBG] from='client.? 172.21.15.45:0/74214998' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:52:28.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:27 smithi154 conmon[34553]: cluster 2021-07-31T02:52:26.733508+0000 mgr.x (mgr.24542) 2792 : cluster [DBG] pgmap v3964: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:28.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:27 smithi045 conmon[39718]: cluster 2021-07-31T02:52:26.733508+0000 mgr.x (mgr. 2021-07-31T02:52:28.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:27 smithi045 conmon[39718]: 24542) 2792 : cluster [DBG] pgmap v3964: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:28.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:27 smithi045 conmon[33655]: cluster 2021-07-31T02:52:26.733508+0000 mgr.x (mgr.24542) 2792 : cluster [DBG] pgmap v3964: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:29.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:28 smithi154 conmon[34553]: audit 2021-07-31T02:52:28.196304+0000 mon.b (mon.2) 400 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:52:29.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:28 smithi154 conmon[34553]: audit 2021-07-31T02:52:28.196612+0000 mon.b (mon.2) 401 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:52:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:28 smithi154 conmon[34553]: audit 2021-07-31T02:52:28.196853+0000 mon.a (mon.0) 3217 : audit 2021-07-31T02:52:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:28 smithi154 conmon[34553]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:52:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:28 smithi154 conmon[34553]: audit 2021-07-31T02:52:28.197307+0000 mon.a ( 2021-07-31T02:52:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:28 smithi154 conmon[34553]: mon.0) 3218 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:52:29.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:28 smithi045 conmon[39718]: audit 2021-07-31T02:52:28.196304+0000 mon.b (mon. 2021-07-31T02:52:29.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:28 smithi045 conmon[39718]: 2) 400 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:52:29.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:28 smithi045 conmon[39718]: audit 2021-07-31T02:52:28.196612+0000 mon.b (mon.2) 401 2021-07-31T02:52:29.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:28 smithi045 conmon[39718]: : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:52:29.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:28 smithi045 conmon[39718]: audit 2021-07-31T02:52:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:28 smithi045 conmon[39718]: 2021-07-31T02:52:28.196853+0000 mon.a ( 2021-07-31T02:52:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:28 smithi045 conmon[39718]: mon.0) 3217 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:52:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:28 smithi045 conmon[39718]: audit 2021 2021-07-31T02:52:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:28 smithi045 conmon[39718]: -07-31T02:52:28.197307 2021-07-31T02:52:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:28 smithi045 conmon[39718]: +0000 mon.a (mon.0) 3218 : audit 2021-07-31T02:52:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:28 smithi045 conmon[39718]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:52:29.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:28 smithi045 conmon[33655]: audit 2021-07-31T02:52:28.196304+0000 mon.b (mon.2) 400 : audit 2021-07-31T02:52:29.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:28 smithi045 conmon[33655]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:52:29.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:28 smithi045 conmon[33655]: audit 2021-07-31T02:52:28.196612+0000 mon.b (mon.2) 401 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:52:29.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:28 smithi045 conmon[33655]: audit 2021-07-31T02:52:28.196853+0000 mon.a (mon.0) 3217 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:52:29.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:28 smithi045 conmon[33655]: audit 2021-07-31T02:52:28.197307+0000 mon.a (mon.0) 3218 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:52:30.049 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:29 smithi045 conmon[33655]: cluster 2021-07-31T02:52:28.733872+0000 mgr.x ( 2021-07-31T02:52:30.049 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:29 smithi045 conmon[33655]: mgr.24542) 2793 : cluster [DBG] pgmap v3965: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:30.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:29 smithi045 conmon[39718]: cluster 2021-07-31T02:52:28.733872+0000 mgr.x (mgr 2021-07-31T02:52:30.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:29 smithi045 conmon[39718]: .24542) 2793 : cluster [DBG] pgmap v3965: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:30.051 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:29 smithi154 conmon[34553]: cluster 2021-07-31T02:52:28.733872+0000 mgr.x (mgr.24542) 2793 : cluster [DBG] pgmap v3965: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:30.052 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: debug 2021-07-31T02:52:30.011+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:52:30.052 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:52:30.053 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:52:30.053 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: yield 2021-07-31T02:52:30.054 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:52:30.054 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:52:30.054 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:52:30.055 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:52:30.055 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:52:30.056 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:52:30.056 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:52:30.057 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:52:30.057 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:52:30.057 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:52:30.058 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:52:30.058 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: debug 2021-07-31T02:52:30.022+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:52:30.059 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:52:30.059 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:52:30.060 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: yield 2021-07-31T02:52:30.060 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:52:30.060 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:52:30.061 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:52:30.061 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:52:30.062 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:52:30.062 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:52:30.063 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:52:30.063 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:52:30.063 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:52:30.064 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:52:30.064 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:52:30.065 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: debug 2021-07-31T02:52:30.048+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:52:30.065 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:52:30.066 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:52:30.066 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: yield 2021-07-31T02:52:30.503 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:52:30.503 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:52:30.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:52:30.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:52:30.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:52:30.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:52:30.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:52:30.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:52:30.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:52:30.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:52:30.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:52:30.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: debug 2021-07-31T02:52:30.049+0000 7fcb94162700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:52:30.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:52:30.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:52:30.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: yield 2021-07-31T02:52:30.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:52:30.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:52:30.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:52:30.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:52:30.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:52:30.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:52:30.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:52:30.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:52:30.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:52:30.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:52:30.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:52:30.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: 2021-07-31T02:52:30.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:52:30.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: 2021-07-31T02:52:30.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:52:30.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:52:30.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:52:30.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:52:30.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:52:30.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:52:30.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:52:30.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:52:30.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:52:30.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:52:30.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:52:30.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:52:30.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:52:30.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:52:30.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:52:30.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:52:30.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:52:30.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:52:30.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: raise 2021-07-31T02:52:30.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:52:30.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:52:30.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:52:30.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:52:30.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:52:30.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: debug 2021-07-31T02:52:30.058+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:52:30.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:52:30.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:52:30.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: yield 2021-07-31T02:52:30.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:52:30.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:52:30.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:52:30.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:52:30.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:52:30.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:52:30.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:52:30.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:52:30.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:52:30.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:52:30.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:52:30.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: debug 2021-07-31T02:52:30.058+0000 7fcb93961700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:52:30.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:52:30.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:52:30.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: yield 2021-07-31T02:52:30.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:52:30.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:52:30.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:52:30.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:52:30.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:52:30.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:52:30.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:52:30.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:52:30.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:52:30.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:52:30.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:52:30.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: 2021-07-31T02:52:30.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:52:30.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: 2021-07-31T02:52:30.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:52:30.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:52:30.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:52:30.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:52:30.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:52:30.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:52:30.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:52:30.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:52:30.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:52:30.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:52:30.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:52:30.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:52:30.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:52:30.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:52:30.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:52:30.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:52:30.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:52:30.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:52:30.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: raise 2021-07-31T02:52:30.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:52:30.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:52:30.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:52:30.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:52:30.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:52:31.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: audit 2021-07-31T02:52:29.966219+0000 mon.b (mon.2) 402 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:52:31.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: audit 2021-07-31T02:52:29.967613+0000 mon.b (mon.2) 403 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:52:31.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: audit 2021-07-31T02:52:29.968496+0000 mon.b (mon.2) 404 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:52:31.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:52:29.969306+0000 mgr.x (mgr.24542) 2794 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:52:31.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:52:31.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: 2021-07-31T02:52:30.012260+0000 mgr.x (mgr.24542) 2795 : cephadm [ERR] 2021-07-31T02:52:31.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:52:31.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:52:31.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: yield 2021-07-31T02:52:31.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:52:31.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:52:31.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:52:31.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:52:31.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:52:31.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:52:31.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:52:31.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:52:31.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:52:31.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:52:31.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:52:31.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:52:30.012706+0000 mgr.x (mgr.24542) 2796 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:52:31.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:52:30.023728+0000 mgr.x (mgr.24542) 2797 : cephadm [ERR] 2021-07-31T02:52:31.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:52:31.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:52:31.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: yield 2021-07-31T02:52:31.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:52:31.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:52:31.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:52:31.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:52:31.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:52:31.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:52:31.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:52:31.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:52:31.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:52:31.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:52:31.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:52:31.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: cephadm 2021-07- 2021-07-31T02:52:31.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: 31T02:52:30.024533+0000 mgr.x (mgr.24542) 2798 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:52:31.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:52:30.049349+0000 mgr.x (mgr.24542) 2799 : cephadm [ERR] 2021-07-31T02:52:31.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:52:31.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:52:31.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: yield 2021-07-31T02:52:31.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:52:31.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:52:31.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:52:31.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:52:31.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:52:31.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:52:31.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:52:31.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:52:31.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:52:31.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:52:31.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:52:31.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:52:30.050519+0000 mgr.x (mgr.24542) 2800 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:52:31.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:52:31.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:52:31.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: yield 2021-07-31T02:52:31.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:52:31.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:52:31.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:52:31.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:52:31.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:52:31.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:52:31.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:52:31.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:52:31.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:52:31.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:52:31.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:52:31.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: 2021-07-31T02:52:31.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:52:31.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: 2021-07-31T02:52:31.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:52:31.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:52:31.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:52:31.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:52:31.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:52:31.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:52:31.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:52:31.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:52:31.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:52:31.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:52:31.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:52:31.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:52:31.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:52:31.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:52:31.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:52:31.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:52:31.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:52:31.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:52:31.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: raise 2021-07-31T02:52:31.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:52:31.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:52:31.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:52:31.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:52:31.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:52:31.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: 2021-07-31T02:52:31.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:52:30.059178+0000 mgr.x (mgr.24542) 2801 : cephadm [ERR] 2021-07-31T02:52:31.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:52:31.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:52:31.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: yield 2021-07-31T02:52:31.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:52:31.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:52:31.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:52:31.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:52:31.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:52:31.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:52:31.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:52:31.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:52:31.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:52:31.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:52:31.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:52:31.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:52:30.059562+0000 mgr.x (mgr.24542) 2802 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:52:31.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:52:31.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:52:31.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: yield 2021-07-31T02:52:31.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:52:31.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:52:31.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:52:31.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:52:31.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:52:31.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:52:31.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:52:31.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:52:31.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:52:31.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:52:31.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:52:31.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: 2021-07-31T02:52:31.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:52:31.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: 2021-07-31T02:52:31.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:52:31.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:52:31.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:52:31.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:52:31.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:52:31.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:52:31.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:52:31.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:52:31.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:52:31.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:52:31.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:52:31.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:52:31.285 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:52:31.285 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:52:31.285 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:52:31.285 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:52:31.285 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:52:31.286 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:52:31.286 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: raise 2021-07-31T02:52:31.286 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:52:31.286 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:52:31.287 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:52:31.287 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:52:31.287 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:30 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:52:31.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: audit 2021-07-31T02 2021-07-31T02:52:31.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: :52:29.966219+0000 mon.b (mon.2) 402 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:52:31.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: audit 2021-07-31T02: 2021-07-31T02:52:31.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: 52:29.967613+0000 mon.b (mon.2) 403 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:52:31.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: audit 2021-07-31T02:52:31.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: 2021-07-31T02:52:29.968496+0000 mon.b (mon.2) 404 : audit 2021-07-31T02:52:31.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:52:31.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:52:29.969306+0000 mgr.x (mgr.24542 2021-07-31T02:52:31.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: ) 2794 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:52:31.290 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:52:30.012260+0000 2021-07-31T02:52:31.290 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: mgr.x (mgr.24542) 2795 : cephadm [ERR] 2021-07-31T02:52:31.290 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:52:31.290 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:52:31.291 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: yield 2021-07-31T02:52:31.291 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:52:31.291 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:52:31.291 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:52:31.291 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:52:31.292 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:52:31.292 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:52:31.292 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:52:31.292 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:52:31.293 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:52:31.293 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:52:31.293 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:52:31.293 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:52:30.012706+0000 mgr.x (mgr.24542) 2796 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:52:31.293 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:52:30.023728+0000 mgr.x (mgr.24542) 2797 : cephadm [ERR] 2021-07-31T02:52:31.294 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:52:31.294 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:52:31.294 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: yield 2021-07-31T02:52:31.294 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:52:31.294 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:52:31.295 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:52:31.295 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:52:31.295 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:52:31.295 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:52:31.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: audit 2021-07-31T02:52:29.966219+0000 mon.b (mon 2021-07-31T02:52:31.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: .2) 402 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:52:31.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: audit 2021-07-31T02:52:29.967613+0000 mon.b (mon.2 2021-07-31T02:52:31.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: ) 403 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:52:31.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: audit 2021-07-31T02:52:31.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: 2021-07-31T02:52:29.968496+0000 mon.b (mon 2021-07-31T02:52:31.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: .2) 404 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:52:31.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: cephadm 2021-07 2021-07-31T02:52:31.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: -31T02:52:29.969306+0000 2021-07-31T02:52:31.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: mgr.x (mgr.24542) 2794 : cephadm [WRN] 2021-07-31T02:52:31.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:52:31.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:52 2021-07-31T02:52:31.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: :30.012260+0000 mgr.x (mgr.24542) 2795 : cephadm [ERR] 2021-07-31T02:52:31.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: 2021-07-31T02:52:31.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:52:31.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:52:31.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: yield 2021-07-31T02:52:31.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:52:31.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:52:31.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:52:31.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:52:31.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:52:31.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:52:31.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:52:31.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:52:31.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:52:31.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:52:31.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:52:31.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:52:30.012706+0000 mgr.x (mgr.24542) 2796 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:52:31.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: cephadm 2021-07- 2021-07-31T02:52:31.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: 31T02:52:30.023728+0000 2021-07-31T02:52:31.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: mgr.x (mgr.24542) 2797 : cephadm [ERR] 2021-07-31T02:52:31.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: 2021-07-31T02:52:31.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:52:31.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:52:31.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: yield 2021-07-31T02:52:31.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:52:31.304 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:52:31.304 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:52:31.304 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:52:31.305 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:52:31.305 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:52:31.305 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:52:31.305 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:52:31.306 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:52:31.306 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:52:30.024533+0000 2021-07-31T02:52:31.306 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: mgr.x (mgr.24542) 2798 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:52:31.306 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:52:30.049349+0000 mgr.x (mgr.24542) 2799 : cephadm [ERR] 2021-07-31T02:52:31.306 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:52:31.307 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:52:31.307 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: yield 2021-07-31T02:52:31.307 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:52:31.307 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:52:31.308 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:52:31.308 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:52:31.308 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:52:31.308 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:52:31.309 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:52:31.309 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:52:31.309 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:52:31.309 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:52:31.310 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:52:31.310 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:52:30.050519+0000 mgr.x (mgr.24542) 2800 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:52:31.310 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:52:31.310 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:52:31.311 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: yield 2021-07-31T02:52:31.311 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:52:31.311 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:52:31.311 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:52:31.311 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:52:31.312 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:52:31.312 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:52:31.312 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:52:31.312 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:52:31.313 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:52:31.313 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:52:31.313 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:52:31.313 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: 2021-07-31T02:52:31.313 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:52:31.314 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: 2021-07-31T02:52:31.314 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:52:31.314 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:52:31.314 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:52:31.314 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:52:31.315 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:52:31.315 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:52:31.315 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:52:31.315 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:52:31.316 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:52:31.316 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:52:31.316 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:52:31.316 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:52:31.316 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:52:31.317 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:52:31.317 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:52:31.317 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:52:31.317 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:52:31.317 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:52:31.318 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: raise 2021-07-31T02:52:31.318 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:52:31.318 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:52:31.318 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:52:31.318 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:52:31.319 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:52:31.319 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:52:30.059178+0000 mgr.x (mgr.24542) 2801 : cephadm [ERR] 2021-07-31T02:52:31.319 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:52:31.319 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:52:31.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: yield 2021-07-31T02:52:31.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:52:31.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:52:31.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:52:31.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:52:31.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:52:31.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:52:31.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:52:31.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:52:31.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:52:31.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:52:31.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:52:31.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:52:30.059562+0000 mgr.x (mgr.24542) 2802 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:52:31.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:52:31.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:52:31.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: yield 2021-07-31T02:52:31.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:52:31.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:52:31.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:52:31.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:52:31.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:52:31.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:52:31.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:52:31.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:52:31.325 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:52:31.325 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:52:31.325 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:52:31.325 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: 2021-07-31T02:52:31.325 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:52:31.326 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: 2021-07-31T02:52:31.326 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:52:31.326 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:52:31.326 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:52:31.326 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:52:31.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:52:31.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:52:31.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:52:31.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:52:31.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:52:31.328 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:52:31.328 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:52:31.328 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:52:31.328 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:52:31.328 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:52:31.329 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:52:31.329 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:52:31.329 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:52:31.329 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:52:31.329 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: raise 2021-07-31T02:52:31.330 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:52:31.330 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:52:31.330 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:52:31.330 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:52:31.331 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:52:31.331 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:52:31.332 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:52:31.332 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:52:31.332 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:52:31.332 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:52:31.333 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:52:31.333 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:52:31.333 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:52:30.024533+0000 mgr.x (mgr.24542) 2798 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:52:31.333 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:52: 2021-07-31T02:52:31.333 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: 30.049349+0000 mgr.x (mgr.24542) 2799 : cephadm 2021-07-31T02:52:31.334 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: [ERR] 2021-07-31T02:52:31.334 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:52:31.334 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:52:31.334 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: yield 2021-07-31T02:52:31.334 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:52:31.335 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:52:31.335 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:52:31.335 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:52:31.335 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:52:31.335 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:52:31.336 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:52:31.336 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:52:31.336 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:52:31.336 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:52:31.336 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:52:31.337 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:52:30.050519+0000 mgr.x (mgr.24542) 2800 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:52:31.337 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:52:31.337 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:52:31.337 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: yield 2021-07-31T02:52:31.337 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:52:31.338 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:52:31.338 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:52:31.338 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:52:31.338 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:52:31.338 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:52:31.339 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:52:31.339 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:52:31.339 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:52:31.339 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:52:31.339 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:52:31.340 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: 2021-07-31T02:52:31.340 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:52:31.340 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: 2021-07-31T02:52:31.340 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:52:31.341 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:52:31.341 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:52:31.341 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:52:31.341 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:52:31.341 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:52:31.342 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:52:31.342 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:52:31.342 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:52:31.342 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:52:31.342 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:52:31.343 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:52:31.343 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:52:31.343 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:52:31.343 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:52:31.343 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:52:31.344 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:52:31.344 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:52:31.348 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: raise 2021-07-31T02:52:31.349 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:52:31.349 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:52:31.349 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:52:31.349 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:52:31.350 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:52:31.350 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:52:30.059178+0000 mgr.x (mgr.24542) 2801 : cephadm [ERR] 2021-07-31T02:52:31.350 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: 2021-07-31T02:52:31.350 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:52:31.350 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:52:31.351 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: yield 2021-07-31T02:52:31.351 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:52:31.351 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:52:31.351 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:52:31.351 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:52:31.352 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:52:31.352 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:52:31.352 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:52:31.352 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:52:31.352 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:52:31.353 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:52:31.353 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:52:31.353 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:52:30.059562+0000 mgr.x (mgr.24542) 2802 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:52:31.353 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:52:31.353 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:52:31.354 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: yield 2021-07-31T02:52:31.354 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:52:31.354 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:52:31.354 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:52:31.354 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:52:31.355 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:52:31.355 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:52:31.355 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:52:31.355 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:52:31.356 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:52:31.356 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:52:31.356 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:52:31.356 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: 2021-07-31T02:52:31.356 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:52:31.357 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: 2021-07-31T02:52:31.357 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:52:31.357 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:52:31.357 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:52:31.357 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:52:31.358 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:52:31.358 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:52:31.358 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:52:31.358 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:52:31.358 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:52:31.359 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:52:31.359 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:52:31.359 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:52:31.359 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:52:31.359 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:52:31.360 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:52:31.360 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:52:31.360 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:52:31.360 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:52:31.360 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: raise 2021-07-31T02:52:31.361 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:52:31.361 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:52:31.361 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:52:31.361 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:52:31.361 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:30 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:52:32.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:31 smithi154 conmon[34553]: cluster 2021-07-31T02:52:30.734405+0000 mgr.x (mgr.24542) 2803 : cluster [DBG] pgmap v3966: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:32.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:31 smithi045 conmon[33655]: cluster 2021-07-31T02:52:30.734405+0000 mgr.x ( 2021-07-31T02:52:32.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:31 smithi045 conmon[33655]: mgr.24542) 2803 : cluster [DBG] pgmap v3966: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:32.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:31 smithi045 conmon[39718]: cluster 2021-07-31T02:52:30.734405+0000 mgr.x ( 2021-07-31T02:52:32.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:31 smithi045 conmon[39718]: mgr.24542) 2803 : cluster [DBG] pgmap v3966: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:34.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:33 smithi154 conmon[34553]: cluster 2021-07-31T02:52:32.734889+0000 mgr.x (mgr.24542) 2804 : cluster [DBG] pgmap v3967: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:34.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:33 smithi045 conmon[33655]: cluster 2021-07-31T02:52:32.734889+0000 mgr.x (mgr. 2021-07-31T02:52:34.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:33 smithi045 conmon[33655]: 24542) 2804 : cluster [DBG] pgmap v3967: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:34.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:33 smithi045 conmon[39718]: cluster 2021-07-31T02:52:32.734889+0000 mgr.x (mgr 2021-07-31T02:52:34.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:33 smithi045 conmon[39718]: .24542) 2804 : cluster [DBG] pgmap v3967: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:36.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:35 smithi154 conmon[34553]: cluster 2021-07-31T02:52:34.735453+0000 mgr.x (mgr.24542) 2805 : cluster [DBG] pgmap v3968: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:36.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:35 smithi154 conmon[34553]: 2021-07-31T02:52:36.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:35 smithi045 conmon[33655]: cluster 2021-07-31T02:52:34.735453+0000 mgr.x (mgr.24542 2021-07-31T02:52:36.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:35 smithi045 conmon[33655]: ) 2805 : cluster [DBG] pgmap v3968: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:36.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:35 smithi045 conmon[39718]: cluster 2021-07-31T02:52:34.735453+0000 mgr.x (mgr.24542) 2805 : cluster [DBG] pgmap v3968: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:38.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:37 smithi154 conmon[34553]: cluster 2021-07-31T02:52:36.735882+0000 mgr.x (mgr.24542) 2806 : cluster [DBG] pgmap v3969: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:38.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:37 smithi045 conmon[39718]: cluster 2021-07-31T02:52:36.735882+0000 mgr.x (mgr.24542) 2806 : cluster [DBG] pgmap v3969: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:38.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:37 smithi045 conmon[33655]: cluster 2021-07-31T02:52:36.735882+0000 mgr.x (mgr 2021-07-31T02:52:38.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:37 smithi045 conmon[33655]: .24542) 2806 : cluster [DBG] pgmap v3969: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:40.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:39 smithi154 conmon[34553]: cluster 2021-07-31T02:52:38.736328+0000 mgr.x (mgr.24542) 2807 : cluster [DBG] pgmap v3970: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:40.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:39 smithi045 conmon[39718]: cluster 2021-07-31T02:52:38.736328+0000 mgr.x (mgr.24542) 2807 : cluster [DBG] pgmap v3970: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:40.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:39 smithi045 conmon[33655]: cluster 2021-07-31T02:52:38.736328+0000 mgr.x ( 2021-07-31T02:52:40.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:39 smithi045 conmon[33655]: mgr.24542) 2807 : cluster [DBG] pgmap v3970: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:42.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:41 smithi154 conmon[34553]: cluster 2021-07-31T02:52:40.736783+0000 mgr.x (mgr.24542) 2808 : cluster [DBG] pgmap v3971: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:42.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:41 smithi154 conmon[34553]: 2021-07-31T02:52:42.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:41 smithi045 conmon[39718]: cluster 2021-07-31T02:52:40.736783+0000 mgr.x (mgr 2021-07-31T02:52:42.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:41 smithi045 conmon[39718]: .24542) 2808 : cluster [DBG] pgmap v3971: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:42.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:41 smithi045 conmon[33655]: cluster 2021-07-31T02:52:40.736783+0000 mgr.x (mgr.24542) 2808 : cluster [DBG] pgmap v3971: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:42.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:41 smithi045 conmon[33655]: 2021-07-31T02:52:44.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:43 smithi154 conmon[34553]: cluster 2021-07-31T02:52:42.737255+0000 mgr.x (mgr.24542) 2809 : cluster [DBG] 2021-07-31T02:52:44.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:43 smithi154 conmon[34553]: pgmap v3972: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:44.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:43 smithi045 conmon[39718]: cluster 2021-07-31T02:52:42.737255+0000 2021-07-31T02:52:44.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:43 smithi045 conmon[39718]: mgr.x (mgr.24542) 2809 : cluster [DBG] pgmap v3972: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:44.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:43 smithi045 conmon[33655]: cluster 2021-07-31T02:52:42. 2021-07-31T02:52:44.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:43 smithi045 conmon[33655]: 737255+0000 mgr.x (mgr.24542) 2809 : cluster [DBG] pgmap v3972: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:46.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:45 smithi154 conmon[34553]: cluster 2021-07-31T02:52:44.737669+0000 mgr.x (mgr.24542) 2810 : cluster [DBG] pgmap v3973: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:46.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:45 smithi045 conmon[39718]: cluster 2021-07-31T02:52:44.737669+0000 mgr.x (mgr. 2021-07-31T02:52:46.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:45 smithi045 conmon[39718]: 24542) 2810 : cluster [DBG] pgmap v3973: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:46.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:45 smithi045 conmon[33655]: cluster 2021-07-31T02:52:44.737669+0000 mgr.x (mgr.24542) 2021-07-31T02:52:46.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:45 smithi045 conmon[33655]: 2810 : cluster [DBG] pgmap v3973: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:48.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:47 smithi154 conmon[34553]: cluster 2021-07-31T02:52:46.738090+0000 mgr.x (mgr.24542) 2811 : cluster [DBG] pgmap v3974: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:48.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:47 smithi045 conmon[39718]: cluster 2021-07-31T02:52:46.738090+0000 mgr.x 2021-07-31T02:52:48.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:47 smithi045 conmon[39718]: (mgr.24542) 2811 : cluster [DBG] pgmap v3974: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:48.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:47 smithi045 conmon[33655]: cluster 2021-07-31T02:52:46.738090+0000 mgr.x (mgr.24542) 2811 2021-07-31T02:52:48.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:47 smithi045 conmon[33655]: : cluster [DBG] pgmap v3974: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:50.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:49 smithi154 conmon[34553]: cluster 2021-07-31T02:52:48.738500+0000 mgr.x (mgr.24542) 2812 : cluster [DBG] pgmap v3975: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:50.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:49 smithi045 conmon[39718]: cluster 2021-07-31T02:52:48.738500 2021-07-31T02:52:50.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:49 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 2812 : cluster [DBG] pgmap v3975: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:50.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:49 smithi045 conmon[33655]: cluster 2021-07-31T02:52:48.738500+0000 mgr.x (mgr.24542) 2812 : 2021-07-31T02:52:50.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:49 smithi045 conmon[33655]: cluster [DBG] pgmap v3975: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:52.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:51 smithi154 conmon[34553]: cluster 2021-07-31T02:52:50.738986+0000 mgr.x (mgr.24542) 2813 : cluster [DBG] pgmap v3976: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:52.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:51 smithi045 conmon[39718]: cluster 2021-07-31T02:52:50.738986+0000 mgr.x (mgr. 2021-07-31T02:52:52.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:51 smithi045 conmon[39718]: 24542) 2813 : cluster [DBG] pgmap v3976: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:52.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:51 smithi045 conmon[33655]: cluster 2021-07-31T02:52: 2021-07-31T02:52:52.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:51 smithi045 conmon[33655]: 50.738986+0000 mgr.x (mgr.24542) 2813 : cluster [DBG] pgmap v3976: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:54.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:53 smithi154 conmon[34553]: cluster 2021-07-31T02:52:52.739377+0000 mgr.x (mgr.24542) 2814 : cluster [DBG] pgmap v3977: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:54.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:53 smithi045 conmon[39718]: cluster 2021-07-31T02:52:52.739377+0000 mgr.x ( 2021-07-31T02:52:54.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:53 smithi045 conmon[39718]: mgr.24542) 2814 : cluster [DBG] pgmap v3977: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:54.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:53 smithi045 conmon[33655]: cluster 2021-07-31T 2021-07-31T02:52:54.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:53 smithi045 conmon[33655]: 02:52:52.739377+0000 mgr.x (mgr.24542) 2814 : cluster [DBG] pgmap v3977: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:55.453 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:52:55.733 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:52:55.733 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 67m ago 68m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:52:55.733 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 65m ago 68m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:52:55.734 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 65m ago 71m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:52:55.734 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 67m ago 73m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:52:55.734 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 67m ago 74m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:52:55.734 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 65m ago 72m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:52:55.735 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 67m ago 72m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:52:55.735 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 67m ago 68m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:52:55.735 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 65m ago 68m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:52:55.735 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 67m ago 71m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:52:55.735 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 67m ago 71m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:52:55.736 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 67m ago 70m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:52:55.736 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 67m ago 70m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:52:55.736 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 65m ago 70m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:52:55.736 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 65m ago 69m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:52:55.737 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 65m ago 69m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:52:55.737 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 65m ago 69m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:52:55.737 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 65m ago 68m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:52:56.075 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:52:56.075 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:52:56.076 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:52:56.076 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:52:56.076 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:52:56.076 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:52:56.076 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:52:56.077 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:52:56.077 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:52:56.077 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:52:56.077 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:52:56.078 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:52:56.078 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:52:56.078 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:52:56.078 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:52:56.078 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:52:56.079 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:52:56.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:55 smithi154 conmon[34553]: cluster 2021-07-31T02:52:54.739889+0000 mgr.x (mgr.24542) 2815 : cluster [DBG] pgmap v3978: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:56.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:55 smithi154 conmon[34553]: audit 2021-07-31T02:52:55.441571+0000 mgr.x (mgr.24542) 2816 : audit [DBG] from='client.28276 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:52:56.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:55 smithi045 conmon[39718]: cluster 2021-07-31T02:52:54.739889+0000 mgr.x ( 2021-07-31T02:52:56.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:55 smithi045 conmon[39718]: mgr.24542) 2815 : cluster [DBG] pgmap v3978: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:56.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:55 smithi045 conmon[39718]: audit 2021-07-31T02:52:55.441571 2021-07-31T02:52:56.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:55 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 2816 : audit [DBG] from='client.28276 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:52:56.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:55 smithi045 conmon[33655]: cluster 2021-07-31T02:52:54.739889 2021-07-31T02:52:56.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:55 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 2815 : cluster [DBG] pgmap v3978: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:56.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:55 smithi045 conmon[33655]: audit 2021-07-31T02:52:55.441571+0000 mgr.x (mgr. 2021-07-31T02:52:56.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:55 smithi045 conmon[33655]: 24542) 2816 : audit [DBG] from='client.28276 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:52:57.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:56 smithi154 conmon[34553]: audit 2021-07-31T02:52:55.725004+0000 mgr.x (mgr.24542) 2817 : audit [DBG] from='client.28282 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:52:57.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:56 smithi154 conmon[34553]: audit 2021-07-31T02:52:56 2021-07-31T02:52:57.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:56 smithi154 conmon[34553]: .075210+0000 mon.c (mon.1) 155 : audit [DBG] from='client.? 172.21.15.45:0/3173727878' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:52:57.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:56 smithi045 conmon[39718]: audit 2021-07-31T02:52:55.725004+0000 mgr.x (mgr. 2021-07-31T02:52:57.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:56 smithi045 conmon[39718]: 24542) 2817 : audit [DBG] from='client.28282 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:52:57.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:56 smithi045 conmon[39718]: audit 2021-07-31T02:52:56.075210 2021-07-31T02:52:57.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:56 smithi045 conmon[39718]: +0000 mon.c (mon.1) 155 : audit [DBG] from='client.? 172.21.15.45:0/3173727878' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:52:57.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:56 smithi045 conmon[33655]: audit 2021-07-31T02:52: 2021-07-31T02:52:57.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:56 smithi045 conmon[33655]: 55.725004+0000 mgr.x (mgr.24542) 2817 : audit [DBG] from='client.28282 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:52:57.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:56 smithi045 conmon[33655]: audit 2021-07-31T02:52:56.075210+0000 mon.c 2021-07-31T02:52:57.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:56 smithi045 conmon[33655]: (mon.1) 155 : audit [DBG] from='client.? 172.21.15.45:0/3173727878' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:52:58.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:57 smithi154 conmon[34553]: cluster 2021-07-31T02:52:56.740348+0000 mgr.x (mgr.24542) 2818 : cluster [DBG] pgmap v3979: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:58.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:57 smithi045 conmon[33655]: cluster 2021-07-31T02:52:56.740348+0000 mgr.x (mgr.24542) 2818 2021-07-31T02:52:58.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:57 smithi045 conmon[33655]: : cluster [DBG] pgmap v3979: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:52:58.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:57 smithi045 conmon[39718]: cluster 2021-07-31T02:52:56.740348+0000 mgr.x ( 2021-07-31T02:52:58.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:57 smithi045 conmon[39718]: mgr.24542) 2818 : cluster [DBG] pgmap v3979: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:00.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:52:59 smithi154 conmon[34553]: cluster 2021-07-31T02:52:58.740872+0000 mgr.x (mgr.24542) 2819 : cluster [DBG] pgmap v3980: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:00.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:59 smithi045 conmon[33655]: cluster 2021-07-31T02:52:58.740872+0000 mgr.x (mgr.24542) 2819 : cluster [DBG] 2021-07-31T02:53:00.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:52:59 smithi045 conmon[33655]: pgmap v3980: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:00.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:59 smithi045 conmon[39718]: cluster 2021-07-31T02:52:58.740872+0000 mgr.x (mgr. 2021-07-31T02:53:00.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:52:59 smithi045 conmon[39718]: 24542) 2819 : cluster [DBG] pgmap v3980: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:02.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:01 smithi154 conmon[34553]: cluster 2021-07-31T02:53:00.741446+0000 mgr.x (mgr.24542) 2820 : cluster [DBG] pgmap v3981: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:02.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:01 smithi045 conmon[39718]: cluster 2021-07-31T02:53:00.741446+0000 2021-07-31T02:53:02.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:01 smithi045 conmon[39718]: mgr.x (mgr.24542) 2820 : cluster [DBG] pgmap v3981: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:02.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:01 smithi045 conmon[33655]: cluster 2021-07-31T02:53:00. 2021-07-31T02:53:02.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:01 smithi045 conmon[33655]: 741446+0000 mgr.x (mgr.24542) 2820 : cluster [DBG] pgmap v3981: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:04.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:03 smithi154 conmon[34553]: cluster 2021-07-31T02:53:02.741985+0000 mgr.x (mgr.24542) 2821 : cluster [DBG] pgmap v3982: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:04.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:03 smithi045 conmon[39718]: cluster 2021-07-31T02:53:02.741985+0000 mgr.x (mgr.24542) 2821 : cluster [DBG] pgmap v3982: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:04.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:03 smithi045 conmon[33655]: cluster 2021-07-31T02:53:02.741985+0000 2021-07-31T02:53:04.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:03 smithi045 conmon[33655]: mgr.x (mgr.24542) 2821 : cluster [DBG] pgmap v3982: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:06.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:05 smithi154 conmon[34553]: cluster 2021-07-31T02:53:04.742519+0000 mgr.x (mgr.24542) 2822 : cluster [DBG] pgmap v3983: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:06.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:05 smithi045 conmon[39718]: cluster 2021-07-31T02:53:04.742519+0000 mgr.x ( 2021-07-31T02:53:06.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:05 smithi045 conmon[39718]: mgr.24542) 2822 : cluster [DBG] pgmap v3983: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:06.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:05 smithi045 conmon[33655]: cluster 2021-07-31T02:53:04.742519+0000 mgr.x (mgr 2021-07-31T02:53:06.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:05 smithi045 conmon[33655]: .24542) 2822 : cluster [DBG] pgmap v3983: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:08.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:07 smithi154 conmon[34553]: cluster 2021-07-31T02:53:06.743105+0000 mgr.x (mgr.24542) 2823 : cluster [DBG] pgmap v3984: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:08.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:07 smithi045 conmon[39718]: cluster 2021-07-31T02:53:06.743105+0000 mgr.x 2021-07-31T02:53:08.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:07 smithi045 conmon[39718]: (mgr.24542) 2823 : cluster [DBG] pgmap v3984: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:08.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:07 smithi045 conmon[33655]: cluster 2021-07-31T02:53:06.743105+0000 mgr.x (mgr.24542 2021-07-31T02:53:08.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:07 smithi045 conmon[33655]: ) 2823 : cluster [DBG] pgmap v3984: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:10.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:09 smithi154 conmon[34553]: cluster 2021-07-31T02:53:08.743535+0000 mgr.x (mgr.24542) 2824 : cluster [DBG] pgmap v3985: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:10.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:09 smithi045 conmon[33655]: cluster 2021-07-31T02:53:08.743535+0000 mgr.x (mgr.24542) 2824 : cluster [DBG] pgmap v3985: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:10.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:09 smithi045 conmon[39718]: cluster 2021-07-31T02:53:08.743535+0000 mgr.x (mgr 2021-07-31T02:53:10.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:09 smithi045 conmon[39718]: .24542) 2824 : cluster [DBG] pgmap v3985: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:12.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:11 smithi154 conmon[34553]: cluster 2021-07-31T02:53:10.744127+0000 mgr.x (mgr.24542) 2825 : cluster [DBG] pgmap v3986: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:12.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:11 smithi045 conmon[39718]: cluster 2021-07-31T02:53:10.744127+0000 mgr.x (mgr 2021-07-31T02:53:12.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:11 smithi045 conmon[39718]: .24542) 2825 : cluster [DBG] pgmap v3986: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:12.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:11 smithi045 conmon[33655]: cluster 2021-07-31T02:53:10.744127+0000 mgr.x (mgr.24542) 2825 : cluster [DBG] pgmap v3986: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:14.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:13 smithi154 conmon[34553]: cluster 2021-07-31T02:53:12.744543+0000 mgr.x (mgr.24542) 2826 : cluster [DBG] pgmap v3987: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:14.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:13 smithi045 conmon[39718]: cluster 2021-07-31T02:53:12.744543 2021-07-31T02:53:14.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:13 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 2826 : cluster [DBG] pgmap v3987: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:14.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:13 smithi045 conmon[33655]: cluster 2021-07-31T02:53:12.744543+0000 mgr.x 2021-07-31T02:53:14.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:13 smithi045 conmon[33655]: (mgr.24542) 2826 : cluster [DBG] pgmap v3987: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:16.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:15 smithi154 conmon[34553]: cluster 2021-07-31T02:53:14.745101+0000 mgr.x (mgr.24542) 2827 : cluster [DBG] pgmap v3988: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:16.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:15 smithi045 conmon[39718]: cluster 2021-07-31T02:53:14.745101+0000 mgr.x (mgr.24542) 2827 : cluster [DBG] pgmap v3988: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:16.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:15 smithi045 conmon[33655]: cluster 2021-07-31T02:53:14.745101+0000 mgr.x (mgr.24542) 2827 : 2021-07-31T02:53:16.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:15 smithi045 conmon[33655]: cluster [DBG] pgmap v3988: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:18.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:17 smithi154 conmon[34553]: cluster 2021-07-31T02:53:16.745637+0000 mgr.x (mgr.24542) 2828 : cluster [DBG] pgmap v3989: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:18.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:17 smithi045 conmon[39718]: cluster 2021-07-31T02:53:16.745637+0000 mgr.x ( 2021-07-31T02:53:18.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:17 smithi045 conmon[39718]: mgr.24542) 2828 : cluster [DBG] pgmap v3989: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:18.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:17 smithi045 conmon[33655]: cluster 2021-07-31T02:53:16.745637+0000 mgr.x (mgr.24542) 2828 : cluster [DBG] pgmap v3989: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:18.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:17 smithi045 conmon[33655]: 2021-07-31T02:53:20.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:19 smithi154 conmon[34553]: cluster 2021-07-31T02:53:18.746080+0000 mgr.x (mgr.24542) 2829 : cluster [DBG] pgmap v3990: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:20.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:19 smithi045 conmon[39718]: cluster 2021-07-31T02:53:18.746080+0000 mgr.x (mgr.24542) 2829 : cluster [DBG] pgmap v3990: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:20.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:19 smithi045 conmon[33655]: cluster 2021-07-31T02:53:18.746080+0000 2021-07-31T02:53:20.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:19 smithi045 conmon[33655]: mgr.x (mgr.24542) 2829 : cluster [DBG] pgmap v3990: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:22.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:21 smithi154 conmon[34553]: cluster 2021-07-31T02:53:20.746571+0000 mgr.x (mgr.24542) 2830 : cluster [DBG] pgmap v3991: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:22.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:21 smithi154 conmon[34553]: 2021-07-31T02:53:22.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:21 smithi045 conmon[39718]: cluster 2021-07-31T02:53:20.746571+0000 mgr.x (mgr.24542) 2830 : 2021-07-31T02:53:22.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:21 smithi045 conmon[39718]: cluster [DBG] pgmap v3991: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:22.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:21 smithi045 conmon[33655]: cluster 2021-07-31T02:53:20.746571+0000 mgr.x (mgr.24542) 2830 : cluster [DBG] pgmap v3991: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:24.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:23 smithi154 conmon[34553]: cluster 2021-07-31T02:53:22.747095+0000 mgr.x (mgr.24542) 2831 : cluster 2021-07-31T02:53:24.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:23 smithi154 conmon[34553]: [DBG] pgmap v3992: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:24.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:23 smithi045 conmon[39718]: cluster 2021-07-31T02:53: 2021-07-31T02:53:24.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:23 smithi045 conmon[39718]: 22.747095+0000 mgr.x (mgr.24542) 2831 : cluster [DBG] pgmap v3992: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:24.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:23 smithi045 conmon[33655]: cluster 2021-07-31T02:53:22.747095+0000 mgr.x (mgr.24542) 2831 2021-07-31T02:53:24.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:23 smithi045 conmon[33655]: : cluster [DBG] pgmap v3992: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:26.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:25 smithi154 conmon[34553]: cluster 2021-07-31T02:53:24.747509+0000 mgr.x (mgr.24542) 2832 : cluster [DBG] 2021-07-31T02:53:26.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:25 smithi154 conmon[34553]: pgmap v3993: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:26.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:25 smithi045 conmon[33655]: cluster 2021-07-31T02:53:24.747509+0000 mgr.x (mgr.24542) 2832 2021-07-31T02:53:26.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:25 smithi045 conmon[33655]: : cluster [DBG] pgmap v3993: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:26.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:25 smithi045 conmon[39718]: cluster 2021-07-31T02:53:24. 2021-07-31T02:53:26.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:25 smithi045 conmon[39718]: 747509+0000 mgr.x (mgr.24542) 2832 : cluster [DBG] pgmap v3993: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:26.390 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:53:26.667 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:53:26.667 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 68m ago 69m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:53:26.667 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 66m ago 68m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:53:26.667 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 66m ago 72m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:53:26.668 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 68m ago 74m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:53:26.668 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 68m ago 74m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:53:26.668 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 66m ago 72m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:53:26.668 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 68m ago 72m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:53:26.669 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 68m ago 69m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:53:26.669 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 66m ago 69m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:53:26.669 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 68m ago 72m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:53:26.669 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 68m ago 71m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:53:26.670 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 68m ago 71m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:53:26.670 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 68m ago 71m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:53:26.670 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 66m ago 70m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:53:26.671 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 66m ago 70m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:53:26.671 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 66m ago 70m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:53:26.671 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 66m ago 69m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:53:26.671 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 66m ago 69m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:53:27.001 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:53:27.002 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:53:27.002 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:53:27.003 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:53:27.003 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:53:27.003 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:53:27.003 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:53:27.003 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:53:27.004 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:53:27.004 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:53:27.004 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:53:27.004 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:53:27.005 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:53:27.005 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:53:27.005 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:53:27.005 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:53:27.005 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:53:27.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:26 smithi154 conmon[34553]: audit 2021-07-31T02:53:26.378378+0000 mgr.x (mgr.24542) 2833 : audit [DBG] from='client.18354 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:53:27.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:26 smithi045 conmon[39718]: audit 2021-07-31T02:53:26.378378+0000 2021-07-31T02:53:27.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:26 smithi045 conmon[39718]: mgr.x (mgr.24542) 2833 : audit [DBG] from='client.18354 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:53:27.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:26 smithi045 conmon[33655]: audit 2021-07-31T02:53:26.378378+0000 mgr.x (mgr 2021-07-31T02:53:27.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:26 smithi045 conmon[33655]: .24542) 2833 : audit [DBG] from='client.18354 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:53:28.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:27 smithi154 conmon[34553]: audit 2021-07-31T02:53:26.658721+0000 mgr.x (mgr.24542) 2834 : audit [DBG] from='client.18360 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:53:28.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:27 smithi154 conmon[34553]: cluster 2021-07-31T02:53:26.747909+0000 mgr.x (mgr.24542) 2835 : cluster [DBG] pgmap v3994: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:28.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:27 smithi154 conmon[34553]: audit 2021-07-31T02:53:27.001666+0000 mon.c (mon.1) 156 : audit [DBG] from='client.? 172.21.15.45:0/2347919532' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:53:28.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:27 smithi154 conmon[34553]: 2021-07-31T02:53:28.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:27 smithi045 conmon[39718]: audit 2021-07-31T02:53:26.658721+0000 mgr.x (mgr.24542) 2834 : 2021-07-31T02:53:28.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:27 smithi045 conmon[39718]: audit [DBG] from='client.18360 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:53:28.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:27 smithi045 conmon[39718]: cluster 2021-07-31T02:53:26.747909+0000 mgr.x (mgr.24542) 2835 : cluster [DBG] pgmap v3994: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:28.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:27 smithi045 conmon[39718]: audit 2021-07-31T02:53:27.001666+0000 mon.c (mon.1) 156 : audit [DBG] 2021-07-31T02:53:28.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:27 smithi045 conmon[39718]: from='client.? 172.21.15.45:0/2347919532' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:53:28.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:27 smithi045 conmon[33655]: audit 2021-07-31T02:53:26.658721+0000 mgr.x (mgr.24542) 2834 : audit 2021-07-31T02:53:28.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:27 smithi045 conmon[33655]: [DBG] from='client.18360 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:53:28.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:27 smithi045 conmon[33655]: cluster 2021-07-31T02:53:26.747909+0000 mgr.x (mgr.24542) 2835 : cluster [DBG] pgmap v3994: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:28.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:27 smithi045 conmon[33655]: audit 2021-07-31T02:53:27.001666+0000 mon.c (mon.1) 156 : audit [DBG] from='client.? 172.21.15.45:0/2347919532' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:53:28.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:27 smithi045 conmon[33655]: 2021-07-31T02:53:29.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:28 smithi154 conmon[34553]: audit 2021-07-31T02:53:28.200551+0000 mon.b (mon.2) 405 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:53:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:28 smithi154 conmon[34553]: 2021-07-31T02:53:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:28 smithi154 conmon[34553]: audit 2021-07-31T02:53:28.200800+0000 mon.b (mon.2) 406 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:53:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:28 smithi154 conmon[34553]: audit 2021-07-31T02:53:28.201238+0000 mon.a (mon 2021-07-31T02:53:29.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:28 smithi154 conmon[34553]: .0) 3219 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:53:29.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:28 smithi154 conmon[34553]: audit 2021- 2021-07-31T02:53:29.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:28 smithi154 conmon[34553]: 07-31T02:53:28.201699+0000 mon.a (mon.0) 3220 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:53:29.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:28 smithi045 conmon[39718]: audit 2021-07-31T02:53:28.200551+0000 mon.b 2021-07-31T02:53:29.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:28 smithi045 conmon[39718]: (mon.2) 405 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:53:29.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:28 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T02:53:29.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:28 smithi045 conmon[39718]: 31T02:53:28.200800+0000 mon.b (mon.2) 406 : 2021-07-31T02:53:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:28 smithi045 conmon[39718]: audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:53:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:28 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T02:53:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:28 smithi045 conmon[39718]: T02:53:28.201238+0000 mon.a ( 2021-07-31T02:53:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:28 smithi045 conmon[39718]: mon.0) 3219 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:53:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:28 smithi045 conmon[39718]: 2021-07-31T02:53:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:28 smithi045 conmon[39718]: audit 2021-07-31T02:53 2021-07-31T02:53:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:28 smithi045 conmon[39718]: :28.201699+0000 mon.a (mon.0 2021-07-31T02:53:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:28 smithi045 conmon[39718]: ) 3220 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:53:29.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:28 smithi045 conmon[33655]: audit 2021-07-31T02:53:28.200551+0000 mon.b 2021-07-31T02:53:29.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:28 smithi045 conmon[33655]: (mon.2) 405 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:53:29.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:28 smithi045 conmon[33655]: audit 2021-07- 2021-07-31T02:53:29.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:28 smithi045 conmon[33655]: 31T02:53:28.200800+0000 mon.b (mon.2) 2021-07-31T02:53:29.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:28 smithi045 conmon[33655]: 406 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:53:29.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:28 smithi045 conmon[33655]: audit 2021-07- 2021-07-31T02:53:29.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:28 smithi045 conmon[33655]: 31T02:53:28.201238+0000 mon.a (mon.0) 3219 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:53:29.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:28 smithi045 conmon[33655]: audit 2021 2021-07-31T02:53:29.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:28 smithi045 conmon[33655]: -07-31T02:53: 2021-07-31T02:53:29.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:28 smithi045 conmon[33655]: 28.201699+0000 mon.a (mon.0 2021-07-31T02:53:29.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:28 smithi045 conmon[33655]: ) 3220 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:53:30.087 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:29 smithi154 conmon[34553]: cluster 2021-07-31T02:53:28.748445+0000 mgr.x (mgr.24542) 2836 : cluster [DBG] pgmap v3995: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:30.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:29 smithi045 conmon[33655]: cluster 2021-07-31T02:53:28.748445+0000 mgr.x (mgr.24542) 2021-07-31T02:53:30.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:29 smithi045 conmon[33655]: 2836 : cluster [DBG] pgmap v3995: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:30.090 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:29 smithi045 conmon[39718]: cluster 2021-07-31T02:53: 2021-07-31T02:53:30.090 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:29 smithi045 conmon[39718]: 28.748445+0000 mgr.x (mgr.24542) 2836 : cluster [DBG] pgmap v3995: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:30.503 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: debug 2021-07-31T02:53:30.105+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:53:30.503 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:53:30.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:53:30.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: yield 2021-07-31T02:53:30.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:53:30.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:53:30.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:53:30.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:53:30.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:53:30.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:53:30.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:53:30.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:53:30.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:53:30.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:53:30.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:53:30.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: debug 2021-07-31T02:53:30.117+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:53:30.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:53:30.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:53:30.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: yield 2021-07-31T02:53:30.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:53:30.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:53:30.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:53:30.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:53:30.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:53:30.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:53:30.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:53:30.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:53:30.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:53:30.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:53:30.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:53:30.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: debug 2021-07-31T02:53:30.144+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:53:30.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:53:30.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:53:30.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: yield 2021-07-31T02:53:30.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:53:30.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:53:30.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:53:30.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:53:30.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:53:30.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:53:30.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:53:30.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:53:30.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:53:30.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:53:30.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:53:30.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: debug 2021-07-31T02:53:30.145+0000 7fcb93160700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:53:30.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:53:30.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:53:30.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: yield 2021-07-31T02:53:30.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:53:30.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:53:30.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:53:30.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:53:30.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:53:30.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:53:30.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:53:30.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:53:30.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:53:30.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:53:30.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:53:30.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: 2021-07-31T02:53:30.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:53:30.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: 2021-07-31T02:53:30.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:53:30.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:53:30.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:53:30.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:53:30.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:53:30.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:53:30.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:53:30.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:53:30.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:53:30.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:53:30.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:53:30.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:53:30.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:53:30.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:53:30.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:53:30.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:53:30.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:53:30.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:53:30.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: raise 2021-07-31T02:53:30.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:53:30.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:53:30.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:53:30.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:53:30.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:53:30.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: debug 2021-07-31T02:53:30.153+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:53:30.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:53:30.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:53:30.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: yield 2021-07-31T02:53:30.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:53:30.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:53:30.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:53:30.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:53:30.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:53:30.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:53:30.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:53:30.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:53:30.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:53:30.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:53:30.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:53:30.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: debug 2021-07-31T02:53:30.154+0000 7fcb9295f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:53:30.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:53:30.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:53:30.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: yield 2021-07-31T02:53:30.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:53:30.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:53:30.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:53:30.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:53:30.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:53:30.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:53:30.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:53:30.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:53:30.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:53:30.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:53:30.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:53:30.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: 2021-07-31T02:53:30.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:53:30.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: 2021-07-31T02:53:30.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:53:30.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:53:30.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:53:30.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:53:30.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:53:30.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:53:30.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:53:30.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:53:30.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:53:30.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:53:30.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:53:30.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:53:30.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:53:30.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:53:30.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:53:30.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:53:30.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:53:30.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:53:30.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: raise 2021-07-31T02:53:30.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:53:30.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:53:30.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:53:30.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:53:30.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:53:31.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: audit 2021-07-31T02:53:30.060268+0000 mon.b (mon.2) 407 : audit [DBG] 2021-07-31T02:53:31.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:53:31.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: audit 2021-07-31T02:53:30.061710+0000 mon.b (mon.2) 408 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:53:31.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: audit 2021-07-31T02:53:30.062782 2021-07-31T02:53:31.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: +0000 mon.b (mon.2) 409 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:53:31.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: cephadm 2021-07 2021-07-31T02:53:31.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: -31T02:53:30.063555+0000 mgr.x (mgr.24542) 2837 : 2021-07-31T02:53:31.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:53:31.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: cephadm 2021- 2021-07-31T02:53:31.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: 07-31T02:53:30.106617+0000 2021-07-31T02:53:31.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: mgr.x (mgr.24542) 2838 : cephadm [ERR] 2021-07-31T02:53:31.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:53:31.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:53:31.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: yield 2021-07-31T02:53:31.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:53:31.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:53:31.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:53:31.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:53:31.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:53:31.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:53:31.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:53:31.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:53:31.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:53:31.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:53:31.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:53:31.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: 2021-07-31T02:53:31.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:53:30.107073+0000 mgr.x (mgr.24542) 2839 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:53:31.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:53:30.118247+0000 2021-07-31T02:53:31.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: mgr.x (mgr.24542) 2840 : cephadm [ERR] 2021-07-31T02:53:31.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:53:31.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:53:31.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: yield 2021-07-31T02:53:31.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:53:31.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:53:31.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:53:31.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:53:31.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:53:31.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:53:31.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:53:31.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:53:31.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:53:31.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:53:31.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:53:31.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:53:30.118696+0000 mgr.x (mgr.24542) 2841 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:53:31.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:53:30.145718+0000 mgr.x (mgr.24542) 2842 : cephadm [ERR] 2021-07-31T02:53:31.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:53:31.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:53:31.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: yield 2021-07-31T02:53:31.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:53:31.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:53:31.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:53:31.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:53:31.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:53:31.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:53:31.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:53:31.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:53:31.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:53:31.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:53:31.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:53:31.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:53:30.146872+0000 mgr.x (mgr.24542) 2843 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:53:31.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:53:31.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:53:31.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: yield 2021-07-31T02:53:31.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:53:31.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:53:31.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:53:31.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:53:31.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:53:31.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:53:31.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:53:31.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:53:31.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:53:31.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:53:31.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:53:31.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: 2021-07-31T02:53:31.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:53:31.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: 2021-07-31T02:53:31.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:53:31.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:53:31.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:53:31.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:53:31.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:53:31.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:53:31.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:53:31.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:53:31.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:53:31.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:53:31.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:53:31.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:53:31.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:53:31.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:53:31.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:53:31.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:53:31.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:53:31.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:53:31.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: raise 2021-07-31T02:53:31.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:53:31.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:53:31.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:53:31.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:53:31.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:53:31.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:53:30.154436+0000 mgr.x (mgr.24542) 2844 2021-07-31T02:53:31.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: : cephadm [ERR] 2021-07-31T02:53:31.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:53:31.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:53:31.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: yield 2021-07-31T02:53:31.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:53:31.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:53:31.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:53:31.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:53:31.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:53:31.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:53:31.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:53:31.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:53:31.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:53:31.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:53:31.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:53:31.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:53:30.154995+0000 mgr.x (mgr.24542) 2845 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:53:31.283 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: audit 2021-07-31T02:53:30.060268+0000 2021-07-31T02:53:31.283 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: mon.b (mon.2) 407 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:53:31.283 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: audit 2021-07-31T02 2021-07-31T02:53:31.283 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: :53:30.061710+0000 mon.b (mon.2) 408 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:53:31.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: 2021-07-31T02:53:31.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: audit 2021-07-31T02:53:30.062782+0000 mon.b (mon.2) 409 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:53:31.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:53:30.063555+0000 mgr.x 2021-07-31T02:53:31.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: (mgr.24542) 2837 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:53:31.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: 2021-07-31T02:53:31.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: cephadm 2021-07- 2021-07-31T02:53:31.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: 31T02:53:30.106617+0000 mgr.x (mgr.24542) 2838 : cephadm 2021-07-31T02:53:31.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: [ERR] 2021-07-31T02:53:31.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:53:31.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:53:31.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: yield 2021-07-31T02:53:31.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:53:31.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:53:31.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:53:31.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:53:31.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:53:31.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:53:31.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:53:31.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:53:31.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:53:31.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:53:31.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:53:31.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:53:30.107073+0000 mgr.x (mgr.24542) 2839 : cephadm 2021-07-31T02:53:31.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:53:31.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: cephadm 2021-07-31T 2021-07-31T02:53:31.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: 02:53:30.118247+0000 mgr.x (mgr.24542) 2840 : cephadm [ERR] 2021-07-31T02:53:31.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:53:31.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:53:31.290 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: yield 2021-07-31T02:53:31.290 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:53:31.290 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:53:31.290 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:53:31.291 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:53:31.291 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:53:31.291 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:53:31.291 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:53:31.292 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: yield 2021-07-31T02:53:31.292 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:53:31.292 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:53:31.292 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:53:31.293 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:53:31.293 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:53:31.293 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:53:31.293 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:53:31.293 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:53:31.294 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:53:31.294 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:53:31.294 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:53:31.294 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: 2021-07-31T02:53:31.294 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:53:31.295 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: 2021-07-31T02:53:31.295 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:53:31.295 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:53:31.295 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:53:31.296 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:53:31.296 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:53:31.296 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:53:31.296 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:53:31.296 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:53:31.297 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:53:31.297 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:53:31.297 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:53:31.297 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:53:31.297 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:53:31.298 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:53:31.298 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:53:31.298 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:53:31.298 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:53:31.298 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:53:31.299 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: raise 2021-07-31T02:53:31.299 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:53:31.299 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:53:31.299 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:53:31.300 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:53:31.300 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:30 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:53:31.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: audit 2021-07-31T02:53: 2021-07-31T02:53:31.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: 30.060268+0000 mon.b (mon.2) 407 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:53:31.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: audit 2021- 2021-07-31T02:53:31.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: 07-31T02:53:30. 2021-07-31T02:53:31.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: 061710+0000 mon.b (mon.2 2021-07-31T02:53:31.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: ) 408 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:53:31.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: 2021-07-31T02:53:31.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: audit 2021-07-31T02:53:31.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: 2021-07-31T02: 2021-07-31T02:53:31.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: 53:30.062782+0000 mon.b 2021-07-31T02:53:31.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: (mon.2) 409 : audit 2021-07-31T02:53:31.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:53:31.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: cephadm 2021- 2021-07-31T02:53:31.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: 07-31T02:53:30. 2021-07-31T02:53:31.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: 063555+0000 mgr.x (mgr.24542 2021-07-31T02:53:31.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: ) 2837 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:53:31.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: 2021-07-31T02:53:31.304 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:53:31.304 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: 2021-07-31T02 2021-07-31T02:53:31.304 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: :53:30.106617+0000 2021-07-31T02:53:31.304 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: mgr.x (mgr.24542) 2838 : cephadm 2021-07-31T02:53:31.304 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: [ERR] 2021-07-31T02:53:31.305 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:53:31.305 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:53:31.305 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: yield 2021-07-31T02:53:31.305 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:53:31.305 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:53:31.306 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:53:31.306 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:53:31.306 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:53:31.306 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:53:31.307 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:53:31.307 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:53:31.307 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:53:31.307 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:53:31.307 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:53:31.308 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:53:30.107073+0000 mgr.x (mgr.24542) 2839 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:53:31.308 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: cephadm 2021-07 2021-07-31T02:53:31.308 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: -31T02:53:30.118247 2021-07-31T02:53:31.308 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 2021-07-31T02:53:31.309 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: 2840 : cephadm [ERR] 2021-07-31T02:53:31.309 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:53:31.309 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:53:31.309 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: yield 2021-07-31T02:53:31.309 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:53:31.310 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:53:31.310 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:53:31.310 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:53:31.311 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:53:31.311 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:53:31.311 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:53:31.311 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:53:30.118696+0000 mgr.x (mgr.24542) 2841 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:53:31.311 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:53:31.312 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: 2021-07-31T02:53:30.145718+0000 mgr.x (mgr.24542) 2842 : 2021-07-31T02:53:31.312 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: cephadm [ERR] 2021-07-31T02:53:31.312 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:53:31.312 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:53:31.312 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: yield 2021-07-31T02:53:31.313 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:53:31.313 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:53:31.313 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:53:31.313 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:53:31.314 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:53:31.314 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:53:31.314 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:53:31.314 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:53:31.314 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:53:31.315 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:53:31.315 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:53:31.315 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:53:30.146872+0000 mgr.x (mgr.24542) 2843 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:53:31.315 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:53:31.315 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:53:31.316 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: yield 2021-07-31T02:53:31.316 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:53:31.316 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:53:31.316 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:53:31.316 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:53:31.317 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:53:31.317 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:53:31.317 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:53:31.317 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:53:31.318 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:53:31.318 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:53:31.318 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:53:31.318 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: 2021-07-31T02:53:31.318 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:53:31.319 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: 2021-07-31T02:53:31.319 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:53:31.319 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:53:31.320 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:53:31.320 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:53:31.320 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:53:31.320 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:53:31.321 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:53:31.321 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:53:31.321 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:53:31.321 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:53:31.321 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:53:30.118696+0000 mgr.x (mgr.24542) 2841 : cephadm 2021-07-31T02:53:31.322 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:53:31.322 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: cephadm 2021 2021-07-31T02:53:31.322 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: -07-31T02:53:30.145718 2021-07-31T02:53:31.322 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 2021-07-31T02:53:31.323 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: 2842 : cephadm [ERR] 2021-07-31T02:53:31.323 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:53:31.323 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:53:31.323 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: yield 2021-07-31T02:53:31.324 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:53:31.324 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:53:31.324 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:53:31.324 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:53:31.324 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:53:31.325 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:53:31.325 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:53:31.325 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:53:31.325 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:53:31.326 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:53:31.326 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:53:31.326 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:53:31.326 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: 2021-07-31T02:53:30.146872 2021-07-31T02:53:31.326 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 2843 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:53:31.327 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:53:31.327 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:53:31.327 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: yield 2021-07-31T02:53:31.327 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:53:31.327 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:53:31.328 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:53:31.328 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:53:31.328 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:53:31.328 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:53:31.329 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:53:31.329 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:53:31.329 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:53:31.329 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:53:31.330 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:53:31.330 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:53:31.330 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:53:31.330 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:53:31.331 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:53:31.331 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:53:31.331 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:53:31.331 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:53:31.331 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:53:31.332 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:53:31.332 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:53:31.332 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:53:31.332 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:53:31.332 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:53:31.333 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: raise 2021-07-31T02:53:31.333 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:53:31.333 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:53:31.333 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:53:31.333 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:53:31.334 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:53:31.334 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: 2021-07-31T02:53:31.334 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:53:30.154436+0000 mgr.x (mgr.24542) 2844 : cephadm [ERR] 2021-07-31T02:53:31.334 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:53:31.335 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:53:31.335 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: yield 2021-07-31T02:53:31.335 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:53:31.335 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:53:31.335 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:53:31.336 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:53:31.336 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:53:31.336 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:53:31.336 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:53:31.336 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:53:31.337 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:53:31.337 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:53:31.337 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:53:31.337 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:53:30.154995+0000 mgr.x ( 2021-07-31T02:53:31.338 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:53:31.338 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:53:31.338 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:53:31.338 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:53:31.339 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:53:31.339 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: 2021-07-31T02:53:31.339 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:53:31.339 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: 2021-07-31T02:53:31.340 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:53:31.340 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:53:31.340 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:53:31.340 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:53:31.340 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:53:31.341 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:53:31.341 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:53:31.341 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:53:31.341 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:53:31.341 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:53:31.342 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:53:31.342 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:53:31.342 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:53:31.342 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:53:31.343 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:53:31.343 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:53:31.343 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:53:31.343 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:53:31.343 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: raise 2021-07-31T02:53:31.344 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:53:31.344 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:53:31.344 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:53:31.344 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:53:31.345 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:53:31.345 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: 2021-07-31T02:53:31.345 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:53:30.154436+0000 mgr.x (mgr.24542) 2844 : cephadm [ERR] 2021-07-31T02:53:31.345 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:53:31.345 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:53:31.346 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: yield 2021-07-31T02:53:31.346 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:53:31.346 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:53:31.346 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:53:31.346 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:53:31.347 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:53:31.347 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: mgr.24542) 2845 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:53:31.347 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:53:31.348 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:53:31.348 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: yield 2021-07-31T02:53:31.348 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:53:31.348 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:53:31.349 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:53:31.349 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:53:31.349 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:53:31.349 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:53:31.349 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:53:31.350 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:53:31.350 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:53:31.350 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:53:31.350 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:53:31.350 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: 2021-07-31T02:53:31.351 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:53:31.351 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: 2021-07-31T02:53:31.351 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:53:31.351 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:53:31.352 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:53:31.352 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:53:31.352 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:53:31.352 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:53:31.352 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:53:31.353 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:53:31.353 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:53:31.353 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:53:31.353 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:53:31.353 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:53:31.354 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:53:31.354 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:53:31.354 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:53:31.354 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:53:31.355 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:53:31.355 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:53:31.355 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: raise 2021-07-31T02:53:31.355 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:53:31.355 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:53:31.356 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:53:31.356 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:53:31.356 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:53:31.356 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:53:31.357 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:53:31.357 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:53:31.357 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:53:31.357 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:53:31.358 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:53:31.358 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:53:30.154995+0000 mgr.x (mgr.24542) 2845 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:53:31.358 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:53:31.358 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:53:31.358 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: yield 2021-07-31T02:53:31.359 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:53:31.359 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:53:31.359 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:53:31.359 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:53:31.359 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:53:31.360 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:53:31.360 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:53:31.360 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:53:31.360 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:53:31.361 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:53:31.361 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:53:31.361 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: 2021-07-31T02:53:31.361 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:53:31.361 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: 2021-07-31T02:53:31.362 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:53:31.362 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:53:31.362 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:53:31.362 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:53:31.363 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:53:31.363 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:53:31.363 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:53:31.363 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:53:31.363 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:53:31.364 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:53:31.364 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:53:31.364 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:53:31.364 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:53:31.364 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:53:31.365 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:53:31.365 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:53:31.365 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:53:31.365 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:53:31.365 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: raise 2021-07-31T02:53:31.366 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:53:31.366 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:53:31.366 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:53:31.366 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:53:31.366 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:30 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:53:32.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:31 smithi154 conmon[34553]: cluster 2021-07-31T02:53:30.748886+0000 mgr.x (mgr.24542) 2846 : cluster [DBG] pgmap v3996: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:32.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:31 smithi045 conmon[33655]: cluster 2021-07-31T02:53:30 2021-07-31T02:53:32.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:31 smithi045 conmon[33655]: .748886+0000 mgr.x (mgr.24542) 2846 : cluster [DBG] pgmap v3996: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:32.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:31 smithi045 conmon[39718]: cluster 2021-07-31T02:53:30.748886+0000 2021-07-31T02:53:32.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:31 smithi045 conmon[39718]: mgr.x (mgr.24542) 2846 : cluster [DBG] pgmap v3996: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:34.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:33 smithi154 conmon[34553]: cluster 2021-07-31T02:53:32.749355+0000 mgr.x (mgr.24542) 2847 : cluster [DBG] pgmap v3997: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:34.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:33 smithi045 conmon[33655]: cluster 2021-07-31T02:53:32.749355+0000 mgr.x (mgr.24542) 2847 : cluster [DBG] pgmap v3997: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:34.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:33 smithi045 conmon[39718]: cluster 2021-07-31T02:53:32.749355+0000 mgr.x (mgr. 2021-07-31T02:53:34.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:33 smithi045 conmon[39718]: 24542) 2847 : cluster [DBG] pgmap v3997: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:36.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:35 smithi154 conmon[34553]: cluster 2021-07-31T02:53:34.749956+0000 mgr.x (mgr.24542) 2848 : cluster [DBG] pgmap v3998: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:36.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:35 smithi154 conmon[34553]: 2021-07-31T02:53:36.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:35 smithi045 conmon[33655]: cluster 2021-07-31T02:53:34.749956+0000 mgr.x (mgr.24542) 2848 : cluster [DBG] pgmap v3998: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:36.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:35 smithi045 conmon[39718]: cluster 2021-07-31T02:53:34.749956+0000 mgr.x (mgr 2021-07-31T02:53:36.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:35 smithi045 conmon[39718]: .24542) 2848 : cluster [DBG] pgmap v3998: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:38.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:37 smithi154 conmon[34553]: cluster 2021-07-31T02:53:36.750387+0000 mgr.x (mgr.24542) 2849 : cluster [DBG] pgmap v3999: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:38.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:37 smithi045 conmon[39718]: cluster 2021-07-31T02:53:36.750387+0000 mgr.x ( 2021-07-31T02:53:38.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:37 smithi045 conmon[39718]: mgr.24542) 2849 : cluster [DBG] pgmap v3999: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:38.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:37 smithi045 conmon[33655]: cluster 2021-07-31T02:53:36.750387+0000 mgr.x (mgr.24542) 2021-07-31T02:53:38.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:37 smithi045 conmon[33655]: 2849 : cluster [DBG] pgmap v3999: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:40.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:39 smithi154 conmon[34553]: cluster 2021-07-31T02:53:38.750898+0000 mgr.x (mgr.24542) 2850 : cluster [DBG] pgmap v4000: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:40.454 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:39 smithi045 conmon[39718]: cluster 2021-07-31T02:53:38.750898+0000 mgr.x (mgr 2021-07-31T02:53:40.455 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:39 smithi045 conmon[39718]: .24542) 2850 : cluster [DBG] pgmap v4000: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:40.455 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:39 smithi045 conmon[33655]: cluster 2021-07-31T02:53:38.750898+0000 mgr.x (mgr.24542) 2021-07-31T02:53:40.455 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:39 smithi045 conmon[33655]: 2850 : cluster [DBG] pgmap v4000: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:42.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:41 smithi154 conmon[34553]: cluster 2021-07-31T02:53:40.751573+0000 mgr.x (mgr.24542) 2851 : cluster [DBG] pgmap v4001: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:42.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:41 smithi045 conmon[39718]: cluster 2021-07-31T02:53:40.751573+0000 mgr.x ( 2021-07-31T02:53:42.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:41 smithi045 conmon[39718]: mgr.24542) 2851 : cluster [DBG] pgmap v4001: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:42.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:41 smithi045 conmon[33655]: cluster 2021-07-31T02:53:40.751573+0000 mgr.x (mgr.24542) 2851 : 2021-07-31T02:53:42.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:41 smithi045 conmon[33655]: cluster [DBG] pgmap v4001: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:44.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:43 smithi045 conmon[33655]: cluster 2021-07-31T02:53:42.751985+0000 mgr.x (mgr.24542) 2852 : cluster 2021-07-31T02:53:44.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:43 smithi045 conmon[33655]: [DBG] pgmap v4002: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:44.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:43 smithi045 conmon[39718]: cluster 2021-07-31T02:53:42.751985+0000 mgr.x (mgr. 2021-07-31T02:53:44.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:43 smithi045 conmon[39718]: 24542) 2852 : cluster [DBG] pgmap v4002: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:44.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:43 smithi154 conmon[34553]: cluster 2021-07-31T02:53:42.751985+0000 mgr.x (mgr.24542) 2852 : cluster 2021-07-31T02:53:44.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:43 smithi154 conmon[34553]: [DBG] pgmap v4002: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:46.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:45 smithi045 conmon[33655]: cluster 2021-07-31T02:53:44.752402+0000 mgr.x (mgr.24542) 2853 : cluster [DBG] 2021-07-31T02:53:46.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:45 smithi045 conmon[33655]: pgmap v4003: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:46.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:45 smithi045 conmon[39718]: cluster 2021-07-31T02:53:44.752402+0000 mgr.x ( 2021-07-31T02:53:46.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:45 smithi045 conmon[39718]: mgr.24542) 2853 : cluster [DBG] pgmap v4003: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:46.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:45 smithi154 conmon[34553]: cluster 2021-07-31T02:53:44.752402+0000 mgr.x (mgr.24542) 2853 : cluster 2021-07-31T02:53:46.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:45 smithi154 conmon[34553]: [DBG] pgmap v4003: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:48.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:47 smithi045 conmon[33655]: cluster 2021-07-31T02:53:46.752800+0000 mgr.x (mgr.24542) 2854 : cluster [DBG] pgmap v4004: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:48.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:47 smithi045 conmon[39718]: cluster 2021-07-31T02:53:46.752800+0000 2021-07-31T02:53:48.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:47 smithi045 conmon[39718]: mgr.x (mgr.24542) 2854 : cluster [DBG] pgmap v4004: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:48.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:47 smithi154 conmon[34553]: cluster 2021-07-31T02:53:46.752800+0000 mgr.x (mgr.24542) 2854 : cluster [DBG] pgmap v4004: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:50.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:49 smithi045 conmon[33655]: cluster 2021-07-31T02:53:48.753221+0000 mgr.x (mgr. 2021-07-31T02:53:50.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:49 smithi045 conmon[33655]: 24542) 2855 : cluster [DBG] pgmap v4005: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:50.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:49 smithi045 conmon[39718]: cluster 2021-07-31T02:53:48.753221+0000 mgr.x ( 2021-07-31T02:53:50.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:49 smithi045 conmon[39718]: mgr.24542) 2855 : cluster [DBG] pgmap v4005: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:50.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:49 smithi154 conmon[34553]: cluster 2021-07-31T02:53:48.753221+0000 mgr.x (mgr.24542) 2855 : cluster [DBG] pgmap v4005: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:52.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:51 smithi045 conmon[33655]: cluster 2021-07-31T02:53:50.753644+0000 mgr.x (mgr.24542) 2021-07-31T02:53:52.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:51 smithi045 conmon[33655]: 2856 : cluster [DBG] pgmap v4006: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:52.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:51 smithi045 conmon[39718]: cluster 2021-07-31T02:53:50.753644+0000 mgr.x 2021-07-31T02:53:52.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:51 smithi045 conmon[39718]: (mgr.24542) 2856 : cluster [DBG] pgmap v4006: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:52.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:51 smithi154 conmon[34553]: cluster 2021-07-31T02:53:50.753644+0000 mgr.x (mgr.24542) 2856 : cluster [DBG] pgmap v4006: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:54.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:53 smithi045 conmon[33655]: cluster 2021-07-31T02:53:52.754045+0000 mgr.x ( 2021-07-31T02:53:54.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:53 smithi045 conmon[33655]: mgr.24542) 2857 : cluster [DBG] pgmap v4007: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:54.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:53 smithi045 conmon[39718]: cluster 2021-07-31T02:53:52.754045+0000 mgr.x ( 2021-07-31T02:53:54.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:53 smithi045 conmon[39718]: mgr.24542) 2857 : cluster [DBG] pgmap v4007: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:54.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:53 smithi154 conmon[34553]: cluster 2021-07-31T02:53:52.754045+0000 mgr.x (mgr.24542) 2857 : cluster [DBG] pgmap v4007: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:56.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:55 smithi045 conmon[39718]: cluster 2021-07-31T02:53:54.754650+0000 mgr.x 2021-07-31T02:53:56.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:55 smithi045 conmon[39718]: (mgr.24542) 2858 : cluster [DBG] pgmap v4008: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:56.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:55 smithi045 conmon[33655]: cluster 2021-07-31T02:53:54.754650+0000 mgr.x (mgr.24542) 2021-07-31T02:53:56.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:55 smithi045 conmon[33655]: 2858 : cluster [DBG] pgmap v4008: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:56.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:55 smithi154 conmon[34553]: cluster 2021-07-31T02:53:54.754650+0000 mgr.x (mgr.24542) 2858 : cluster [DBG] pgmap v4008: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:56.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:55 smithi154 conmon[34553]: 2021-07-31T02:53:57.312 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:53:57.591 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:53:57.591 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 68m ago 69m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:53:57.591 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 66m ago 69m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:53:57.591 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 66m ago 72m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:53:57.592 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 68m ago 74m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:53:57.592 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 68m ago 75m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:53:57.592 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 66m ago 73m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:53:57.592 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 68m ago 73m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:53:57.593 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 68m ago 69m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:53:57.593 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 66m ago 69m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:53:57.593 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 68m ago 72m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:53:57.593 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 68m ago 72m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:53:57.593 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 68m ago 71m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:53:57.594 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 68m ago 71m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:53:57.594 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 66m ago 71m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:53:57.594 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 66m ago 70m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:53:57.594 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 66m ago 70m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:53:57.594 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 66m ago 70m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:53:57.595 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 66m ago 69m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:53:57.929 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:53:57.930 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:53:57.930 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:53:57.930 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:53:57.930 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:53:57.931 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:53:57.931 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:53:57.931 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:53:57.931 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:53:57.931 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:53:57.932 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:53:57.932 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:53:57.932 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:53:57.932 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:53:57.933 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:53:57.933 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:53:57.933 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:53:58.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:57 smithi045 conmon[39718]: cluster 2021-07-31T02:53:56.755117+0000 mgr.x (mgr 2021-07-31T02:53:58.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:57 smithi045 conmon[39718]: .24542) 2859 : cluster [DBG] pgmap v4009: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:58.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:57 smithi045 conmon[39718]: audit 2021-07-31T02:53:57.300662+0000 mgr.x (mgr.24542) 2021-07-31T02:53:58.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:57 smithi045 conmon[39718]: 2860 : audit [DBG] from='client.28312 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:53:58.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:57 smithi045 conmon[39718]: audit 2021-07-31T02:53:57 2021-07-31T02:53:58.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:57 smithi045 conmon[39718]: .582585+0000 mgr.x (mgr.24542) 2861 : audit [DBG] from='client.18378 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:53:58.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:57 smithi045 conmon[33655]: cluster 2021-07-31T02:53:56.755117+0000 mgr.x (mgr.24542) 2859 : cluster [DBG] pgmap v4009: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:58.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:57 smithi045 conmon[33655]: audit 2021-07-31T02:53:57.300662+0000 mgr.x (mgr.24542) 2860 : audit [DBG] from='client.28312 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:53:58.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:57 smithi045 conmon[33655]: audit 2021-07-31T02: 2021-07-31T02:53:58.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:57 smithi045 conmon[33655]: 53:57.582585+0000 mgr.x (mgr.24542) 2861 : audit [DBG] from='client.18378 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:53:58.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:57 smithi154 conmon[34553]: cluster 2021-07-31T02:53:56.755117+0000 mgr.x (mgr.24542) 2859 : cluster [DBG] pgmap v4009: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:53:58.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:57 smithi154 conmon[34553]: 2021-07-31T02:53:58.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:57 smithi154 conmon[34553]: audit 2021-07-31T02:53:57.300662+0000 mgr.x (mgr.24542) 2860 : audit [DBG] from='client.28312 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:53:58.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:57 smithi154 conmon[34553]: audit 2021-07-31T02:53:57.582585+0000 mgr.x (mgr.24542) 2861 : audit [DBG] from='client.18378 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:53:59.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:58 smithi045 conmon[39718]: audit 2021-07-31T02:53:57.929414+0000 mon.c (mon.1) 157 : 2021-07-31T02:53:59.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:58 smithi045 conmon[39718]: audit [DBG] from='client.? 172.21.15.45:0/4173538139' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:53:59.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:58 smithi045 conmon[33655]: audit 2021-07-31T02:53:57.929414+0000 mon.c (mon.1 2021-07-31T02:53:59.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:58 smithi045 conmon[33655]: ) 157 : audit [DBG] from='client.? 172.21.15.45:0/4173538139' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:53:59.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:58 smithi154 conmon[34553]: audit 2021-07-31T02:53:57.929414+0000 mon.c (mon.1) 157 : audit 2021-07-31T02:53:59.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:58 smithi154 conmon[34553]: [DBG] from='client.? 172.21.15.45:0/4173538139' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:54:00.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:59 smithi045 conmon[33655]: cluster 2021-07-31T02:53:58.755624+0000 mgr.x (mgr.24542) 2021-07-31T02:54:00.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:53:59 smithi045 conmon[33655]: 2862 : cluster [DBG] pgmap v4010: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:00.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:59 smithi045 conmon[39718]: cluster 2021-07-31T02:53:58.755624+0000 mgr.x (mgr.24542) 2862 : 2021-07-31T02:54:00.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:53:59 smithi045 conmon[39718]: cluster [DBG] pgmap v4010: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:00.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:59 smithi154 conmon[34553]: cluster 2021-07-31T02:53:58.755624+0000 mgr.x (mgr.24542) 2862 : cluster [DBG] pgmap v4010: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:00.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:53:59 smithi154 conmon[34553]: 2021-07-31T02:54:02.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:01 smithi045 conmon[33655]: cluster 2021-07-31T02:54:00.756202+0000 mgr.x (mgr 2021-07-31T02:54:02.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:01 smithi045 conmon[33655]: .24542) 2863 : cluster [DBG] pgmap v4011: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:02.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:01 smithi045 conmon[39718]: cluster 2021-07-31T02:54:00.756202+0000 mgr.x (mgr.24542) 2863 2021-07-31T02:54:02.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:01 smithi045 conmon[39718]: : cluster [DBG] pgmap v4011: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:02.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:01 smithi154 conmon[34553]: cluster 2021-07-31T02:54:00.756202+0000 mgr.x (mgr.24542) 2863 : cluster [DBG] pgmap v4011: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:02.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:01 smithi154 conmon[34553]: 2021-07-31T02:54:04.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:03 smithi045 conmon[39718]: cluster 2021-07-31T02:54:02.756617+0000 mgr.x (mgr.24542 2021-07-31T02:54:04.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:03 smithi045 conmon[39718]: ) 2864 : cluster [DBG] pgmap v4012: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:04.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:03 smithi045 conmon[33655]: cluster 2021-07-31T02:54:02.756617+0000 mgr.x (mgr.24542) 2864 : cluster [DBG] pgmap v4012: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:04.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:03 smithi045 conmon[33655]: 2021-07-31T02:54:04.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:03 smithi154 conmon[34553]: cluster 2021-07-31T02:54:02.756617+0000 mgr.x (mgr.24542) 2864 : cluster [DBG] pgmap v4012: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:04.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:03 smithi154 conmon[34553]: 2021-07-31T02:54:06.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:05 smithi045 conmon[33655]: cluster 2021-07-31T02:54:04.757113+0000 mgr.x (mgr.24542) 2021-07-31T02:54:06.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:05 smithi045 conmon[33655]: 2865 : cluster [DBG] pgmap v4013: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:06.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:05 smithi045 conmon[39718]: cluster 2021-07-31T02:54:04.757113+0000 mgr.x (mgr.24542) 2021-07-31T02:54:06.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:05 smithi045 conmon[39718]: 2865 : cluster [DBG] pgmap v4013: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:06.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:05 smithi154 conmon[34553]: cluster 2021-07-31T02:54:04.757113+0000 mgr.x (mgr.24542) 2865 : cluster [DBG] pgmap v4013: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:06.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:05 smithi154 conmon[34553]: 2021-07-31T02:54:08.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:07 smithi045 conmon[39718]: cluster 2021-07-31T02:54:06.757636+0000 mgr.x (mgr. 2021-07-31T02:54:08.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:07 smithi045 conmon[39718]: 24542) 2866 : cluster [DBG] pgmap v4014: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:08.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:07 smithi045 conmon[33655]: cluster 2021-07-31T02:54:06.757636+0000 mgr.x ( 2021-07-31T02:54:08.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:07 smithi045 conmon[33655]: mgr.24542) 2866 : cluster [DBG] pgmap v4014: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:08.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:07 smithi154 conmon[34553]: cluster 2021-07-31T02:54:06.757636+0000 mgr.x (mgr.24542) 2866 : cluster [DBG] pgmap v4014: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:10.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:09 smithi045 conmon[39718]: cluster 2021-07-31T02:54:08.758147+0000 mgr.x (mgr.24542) 2021-07-31T02:54:10.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:09 smithi045 conmon[39718]: 2867 : cluster [DBG] pgmap v4015: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:10.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:09 smithi045 conmon[33655]: cluster 2021-07-31T02:54:08 2021-07-31T02:54:10.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:09 smithi045 conmon[33655]: .758147+0000 mgr.x (mgr.24542) 2867 : cluster [DBG] pgmap v4015: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:10.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:09 smithi154 conmon[34553]: cluster 2021-07-31T02:54:08.758147+0000 mgr.x (mgr.24542) 2867 : cluster [DBG] pgmap v4015: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:12.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:11 smithi045 conmon[39718]: cluster 2021-07-31T02:54:10.758671+0000 mgr.x (mgr. 2021-07-31T02:54:12.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:11 smithi045 conmon[39718]: 24542) 2868 : cluster [DBG] pgmap v4016: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:12.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:11 smithi045 conmon[33655]: cluster 2021-07-31T02:54:10.758671+0000 mgr.x (mgr. 2021-07-31T02:54:12.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:11 smithi045 conmon[33655]: 24542) 2868 : cluster [DBG] pgmap v4016: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:12.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:11 smithi154 conmon[34553]: cluster 2021-07-31T02:54:10.758671+0000 mgr.x (mgr.24542) 2868 : cluster [DBG] pgmap v4016: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:14.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:13 smithi045 conmon[39718]: cluster 2021-07-31T02:54:12.759073+0000 mgr.x (mgr. 2021-07-31T02:54:14.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:13 smithi045 conmon[39718]: 24542) 2869 : cluster [DBG] pgmap v4017: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:14.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:13 smithi045 conmon[33655]: cluster 2021-07-31T02:54:12.759073+0000 mgr.x (mgr.24542) 2869 : 2021-07-31T02:54:14.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:13 smithi045 conmon[33655]: cluster [DBG] pgmap v4017: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:14.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:13 smithi154 conmon[34553]: cluster 2021-07-31T02:54:12.759073+0000 mgr.x (mgr.24542) 2869 : cluster [DBG] pgmap v4017: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:16.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:15 smithi045 conmon[39718]: cluster 2021-07-31T02:54:14.759536+0000 mgr.x (mgr.24542) 2021-07-31T02:54:16.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:15 smithi045 conmon[39718]: 2870 : cluster [DBG] pgmap v4018: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:16.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:15 smithi045 conmon[33655]: cluster 2021-07-31T02:54:14.759536+0000 2021-07-31T02:54:16.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:15 smithi045 conmon[33655]: mgr.x (mgr.24542) 2870 : cluster [DBG] pgmap v4018: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:16.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:15 smithi154 conmon[34553]: cluster 2021-07-31T02:54:14.759536+0000 mgr.x (mgr.24542) 2870 : cluster [DBG] pgmap v4018: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:18.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:17 smithi045 conmon[39718]: cluster 2021-07-31T02:54:16.759931+0000 mgr.x (mgr. 2021-07-31T02:54:18.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:17 smithi045 conmon[39718]: 24542) 2871 : cluster [DBG] pgmap v4019: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:18.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:17 smithi045 conmon[33655]: cluster 2021-07-31T02:54:16.759931+0000 2021-07-31T02:54:18.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:17 smithi045 conmon[33655]: mgr.x (mgr.24542) 2871 : cluster [DBG] pgmap v4019: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:18.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:17 smithi154 conmon[34553]: cluster 2021-07-31T02:54:16.759931+0000 mgr.x (mgr.24542) 2871 : cluster [DBG] pgmap v4019: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:18.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:17 smithi154 conmon[34553]: 2021-07-31T02:54:20.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:19 smithi045 conmon[39718]: cluster 2021-07-31T02:54:18.760405+0000 mgr.x (mgr 2021-07-31T02:54:20.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:19 smithi045 conmon[39718]: .24542) 2872 : cluster [DBG] pgmap v4020: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:20.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:19 smithi045 conmon[33655]: cluster 2021-07-31T02:54:18.760405+0000 2021-07-31T02:54:20.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:19 smithi045 conmon[33655]: mgr.x (mgr.24542) 2872 : cluster [DBG] pgmap v4020: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:20.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:19 smithi154 conmon[34553]: cluster 2021-07-31T02:54:18.760405+0000 mgr.x (mgr.24542) 2872 : cluster [DBG] pgmap v4020: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:22.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:21 smithi154 conmon[34553]: cluster 2021-07-31T02:54:20.760888+0000 mgr.x (mgr.24542) 2873 : cluster [DBG] pgmap v4021: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:22.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:21 smithi045 conmon[39718]: cluster 2021-07-31T02:54:20.760888+0000 mgr.x (mgr 2021-07-31T02:54:22.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:21 smithi045 conmon[39718]: .24542) 2873 : cluster [DBG] pgmap v4021: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:22.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:21 smithi045 conmon[33655]: cluster 2021-07-31T02:54:20.760888+0000 mgr.x (mgr.24542) 2873 2021-07-31T02:54:22.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:21 smithi045 conmon[33655]: : cluster [DBG] pgmap v4021: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:24.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:23 smithi045 conmon[33655]: cluster 2021-07-31T02:54:22.761271 2021-07-31T02:54:24.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:23 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 2874 : cluster [DBG] pgmap v4022: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:24.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:23 smithi045 conmon[39718]: cluster 2021-07-31T02:54:22.761271+0000 mgr.x (mgr. 2021-07-31T02:54:24.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:23 smithi045 conmon[39718]: 24542) 2874 : cluster [DBG] pgmap v4022: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:24.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:23 smithi154 conmon[34553]: cluster 2021-07-31T02:54:22.761271+0000 mgr.x (mgr.24542) 2874 : cluster [DBG] pgmap v4022: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:26.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:25 smithi045 conmon[33655]: cluster 2021-07-31T02:54:24.761788+0000 mgr.x (mgr.24542) 2875 : 2021-07-31T02:54:26.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:25 smithi045 conmon[33655]: cluster [DBG] pgmap v4023: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:26.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:25 smithi045 conmon[39718]: cluster 2021-07-31T02:54:24.761788+0000 mgr.x (mgr. 2021-07-31T02:54:26.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:25 smithi045 conmon[39718]: 24542) 2875 : cluster [DBG] pgmap v4023: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:26.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:25 smithi154 conmon[34553]: cluster 2021-07-31T02:54:24.761788+0000 mgr.x (mgr.24542) 2875 : cluster [DBG] pgmap v4023: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:28.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:27 smithi045 conmon[39718]: cluster 2021-07-31T02:54:26.762166+0000 mgr.x (mgr. 2021-07-31T02:54:28.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:27 smithi045 conmon[39718]: 24542) 2876 : cluster [DBG] pgmap v4024: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:28.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:27 smithi045 conmon[33655]: cluster 2021-07-31T02:54:26.762166+0000 mgr.x (mgr.24542) 2021-07-31T02:54:28.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:27 smithi045 conmon[33655]: 2876 : cluster [DBG] pgmap v4024: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:28.242 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:54:28.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:27 smithi154 conmon[34553]: cluster 2021-07-31T02:54:26.762166+0000 mgr.x (mgr.24542) 2876 : cluster [DBG] pgmap v4024: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:28.521 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:54:28.521 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 69m ago 70m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:54:28.521 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 67m ago 69m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:54:28.522 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 67m ago 73m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:54:28.522 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 69m ago 75m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:54:28.522 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 69m ago 75m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:54:28.522 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 67m ago 73m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:54:28.522 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 69m ago 73m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:54:28.523 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 69m ago 70m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:54:28.523 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 67m ago 70m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:54:28.523 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 69m ago 73m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:54:28.523 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 69m ago 72m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:54:28.524 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 69m ago 72m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:54:28.524 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 69m ago 72m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:54:28.524 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 67m ago 71m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:54:28.524 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 67m ago 71m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:54:28.524 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 67m ago 71m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:54:28.525 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 67m ago 70m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:54:28.525 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 67m ago 70m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:54:28.860 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:54:28.860 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:54:28.860 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:54:28.860 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:54:28.861 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:54:28.861 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:54:28.861 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:54:28.861 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:54:28.862 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:54:28.862 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:54:28.862 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:54:28.862 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:54:28.862 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:54:28.863 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:54:28.863 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:54:28.863 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:54:28.863 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:54:29.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:28 smithi154 conmon[34553]: audit 2021-07-31T02:54:28.204039+0000 mon.b (mon.2) 410 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:54:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:28 smithi154 conmon[34553]: audit 2021-07-31T02:54:28.204421+0000 mon.b (mon.2) 411 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:54:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:28 smithi154 conmon[34553]: audit 2021-07-31T02:54:28.204495+0000 mon.a (mon.0) 3221 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:54:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:28 smithi154 conmon[34553]: audit 2021-07-31T02:54:28.204761+0000 mon.a (mon.0) 3222 : 2021-07-31T02:54:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:28 smithi154 conmon[34553]: audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:54:29.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:28 smithi154 conmon[34553]: audit 2021-07-31T02:54:28.230286+0000 mgr.x (mgr.24542) 2877 : audit 2021-07-31T02:54:29.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:28 smithi154 conmon[34553]: [DBG] from='client.28330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:54:29.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:28 smithi154 conmon[34553]: audit 2021-07-31T02:54:29.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:28 smithi154 conmon[34553]: 2021-07-31T02:54:28.513065+0000 mgr.x (mgr. 2021-07-31T02:54:29.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:28 smithi154 conmon[34553]: 24542) 2878 : audit [DBG] from='client.28336 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:54:29.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:28 smithi045 conmon[39718]: audit 2021-07-31T02:54:28.204039 2021-07-31T02:54:29.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:28 smithi045 conmon[39718]: +0000 mon.b (mon.2) 410 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:54:29.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:28 smithi045 conmon[39718]: audit 2021-07-31T02:54:28 2021-07-31T02:54:29.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:28 smithi045 conmon[39718]: .204421+0000 mon.b (mon.2) 411 2021-07-31T02:54:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:28 smithi045 conmon[39718]: : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:54:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:28 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T02:54:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:28 smithi045 conmon[39718]: 31T02:54:28.204495+0000 2021-07-31T02:54:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:28 smithi045 conmon[39718]: mon.a (mon.0) 3221 : audit [INF] 2021-07-31T02:54:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:28 smithi045 conmon[39718]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:54:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:28 smithi045 conmon[39718]: audit 2021-07-31T02:54: 2021-07-31T02:54:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:28 smithi045 conmon[39718]: 28.204761+0000 mon.a (mon.0) 2021-07-31T02:54:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:28 smithi045 conmon[39718]: 3222 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:54:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:28 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T02:54:29.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:28 smithi045 conmon[39718]: 31T02:54:28.230286+0000 2021-07-31T02:54:29.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:28 smithi045 conmon[39718]: mgr.x (mgr.24542) 2877 : audit [DBG] 2021-07-31T02:54:29.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:28 smithi045 conmon[39718]: from='client.28330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:54:29.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:28 smithi045 conmon[39718]: audit 2021-07-31T02:54: 2021-07-31T02:54:29.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:28 smithi045 conmon[39718]: 28.513065+0000 mgr.x (mgr.24542 2021-07-31T02:54:29.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:28 smithi045 conmon[39718]: ) 2878 : audit [DBG] from='client.28336 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:54:29.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:28 smithi045 conmon[33655]: audit 2021-07-31T02:54:28.204039+0000 mon.b (mon. 2021-07-31T02:54:29.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:28 smithi045 conmon[33655]: 2) 410 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:54:29.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:28 smithi045 conmon[33655]: audit 2021-07-31T02:54:28.204421+0000 mon.b (mon.2) 411 : audit 2021-07-31T02:54:29.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:28 smithi045 conmon[33655]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:54:29.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:28 smithi045 conmon[33655]: audit 2021-07-31T02:54:29.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:28 smithi045 conmon[33655]: 2021-07-31T02:54:28.204495+0000 mon.a (mon.0) 2021-07-31T02:54:29.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:28 smithi045 conmon[33655]: 3221 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:54:29.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:28 smithi045 conmon[33655]: audit 2021-07-31T 2021-07-31T02:54:29.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:28 smithi045 conmon[33655]: 02:54:28.204761+0000 mon.a (mon 2021-07-31T02:54:29.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:28 smithi045 conmon[33655]: .0) 3222 : audit [INF] 2021-07-31T02:54:29.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:28 smithi045 conmon[33655]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:54:29.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:28 smithi045 conmon[33655]: audit 2021-07-31T02:54:29.283 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:28 smithi045 conmon[33655]: 2021-07-31T 2021-07-31T02:54:29.283 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:28 smithi045 conmon[33655]: 02:54:28.230286+0000 2021-07-31T02:54:29.283 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:28 smithi045 conmon[33655]: mgr.x (mgr.24542) 2877 2021-07-31T02:54:29.283 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:28 smithi045 conmon[33655]: : audit [DBG] from='client.28330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:54:29.283 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:28 smithi045 conmon[33655]: audit 2021-07-31T02:54:28.513065 2021-07-31T02:54:29.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:28 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 2878 : audit [DBG] from='client.28336 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:54:30.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:29 smithi154 conmon[34553]: cluster 2021-07-31T02:54:28.762507+0000 mgr.x (mgr.24542) 2879 : cluster [DBG] pgmap v4025: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:30.183 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:29 smithi154 conmon[34553]: audit 2021-07-31T02:54:28.859890+0000 mon.c (mon.1) 158 : audit [DBG] from='client.? 172.21.15.45:0/1417805535' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:54:30.183 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:29 smithi045 conmon[33655]: cluster 2021-07-31T02:54:28.762507 2021-07-31T02:54:30.184 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:29 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 2879 : cluster [DBG] pgmap v4025: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:30.184 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:29 smithi045 conmon[33655]: audit 2021-07-31T02:54 2021-07-31T02:54:30.184 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:29 smithi045 conmon[33655]: :28.859890+0000 mon.c (mon.1) 158 : audit [DBG] from='client.? 172.21.15.45:0/1417805535' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:54:30.184 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:29 smithi045 conmon[33655]: 2021-07-31T02:54:30.185 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:29 smithi045 conmon[39718]: cluster 2021-07-31T02:54:28.762507+0000 mgr.x (mgr.24542) 2021-07-31T02:54:30.185 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:29 smithi045 conmon[39718]: 2879 : cluster [DBG] pgmap v4025: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:30.185 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:29 smithi045 conmon[39718]: audit 2021-07-31T02:54:28.859890+0000 mon.c (mon.1) 158 : audit 2021-07-31T02:54:30.185 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:29 smithi045 conmon[39718]: [DBG] from='client.? 172.21.15.45:0/1417805535' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:54:30.503 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: debug 2021-07-31T02:54:30.199+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:54:30.503 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:54:30.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:54:30.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: yield 2021-07-31T02:54:30.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:54:30.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:54:30.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:54:30.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:54:30.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:54:30.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:54:30.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:54:30.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:54:30.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:54:30.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:54:30.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:54:30.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: debug 2021-07-31T02:54:30.211+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:54:30.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:54:30.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:54:30.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: yield 2021-07-31T02:54:30.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:54:30.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:54:30.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:54:30.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:54:30.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:54:30.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:54:30.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:54:30.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:54:30.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:54:30.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:54:30.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:54:30.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: debug 2021-07-31T02:54:30.236+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:54:30.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:54:30.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:54:30.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: yield 2021-07-31T02:54:30.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:54:30.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:54:30.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:54:30.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:54:30.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:54:30.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:54:30.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:54:30.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:54:30.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:54:30.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:54:30.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:54:30.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: debug 2021-07-31T02:54:30.237+0000 7fcb9215e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:54:30.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:54:30.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:54:30.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: yield 2021-07-31T02:54:30.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:54:30.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:54:30.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:54:30.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:54:30.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:54:30.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:54:30.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:54:30.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:54:30.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:54:30.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:54:30.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:54:30.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: 2021-07-31T02:54:30.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:54:30.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: 2021-07-31T02:54:30.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:54:30.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:54:30.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:54:30.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:54:30.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:54:30.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:54:30.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:54:30.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:54:30.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:54:30.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:54:30.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:54:30.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:54:30.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:54:30.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:54:30.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:54:30.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:54:30.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:54:30.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:54:30.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: raise 2021-07-31T02:54:30.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:54:30.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:54:30.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:54:30.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:54:30.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:54:30.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: debug 2021-07-31T02:54:30.245+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:54:30.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:54:30.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:54:30.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: yield 2021-07-31T02:54:30.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:54:30.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:54:30.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:54:30.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:54:30.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:54:30.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:54:30.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:54:30.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:54:30.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:54:30.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:54:30.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:54:30.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: debug 2021-07-31T02:54:30.245+0000 7fcb9195d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:54:30.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:54:30.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:54:30.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: yield 2021-07-31T02:54:30.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:54:30.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:54:30.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:54:30.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:54:30.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:54:30.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:54:30.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:54:30.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:54:30.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:54:30.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:54:30.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:54:30.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: 2021-07-31T02:54:30.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:54:30.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: 2021-07-31T02:54:30.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:54:30.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:54:30.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:54:30.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:54:30.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:54:30.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:54:30.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:54:30.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:54:30.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:54:30.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:54:30.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:54:30.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:54:30.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:54:30.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:54:30.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:54:30.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:54:30.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:54:30.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:54:30.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: raise 2021-07-31T02:54:30.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:54:30.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:54:30.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:54:30.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:54:30.535 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:54:31.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: audit 2021-07-31T02:54:30.155998+0000 mon.b (mon.2) 412 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:54:31.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: audit 2021-07-31T02:54:30.157551+0000 mon.b (mon.2) 413 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:54:31.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: audit 2021-07-31T02:54:30.158385+0000 mon.b (mon.2) 414 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:54:31.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:54:30.159119+0000 mgr.x (mgr 2021-07-31T02:54:31.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: .24542) 2880 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:54:31.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: cephadm 2021 2021-07-31T02:54:31.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: -07-31T02:54:30.201602+0000 2021-07-31T02:54:31.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: mgr.x (mgr.24542) 2881 : cephadm [ERR] 2021-07-31T02:54:31.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:54:31.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:54:31.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: yield 2021-07-31T02:54:31.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:54:31.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:54:31.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:54:31.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:54:31.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:54:31.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:54:31.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:54:31.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:54:31.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:54:31.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:54:31.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:54:31.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: 2021-07-31T02:54:31.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:54:30.202063+0000 mgr.x (mgr.24542) 2882 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:54:31.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:54:30.213096+0000 mgr.x (mgr 2021-07-31T02:54:31.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: .24542) 2883 : cephadm [ERR] 2021-07-31T02:54:31.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:54:31.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:54:31.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: yield 2021-07-31T02:54:31.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:54:31.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:54:31.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:54:31.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:54:31.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:54:31.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:54:31.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:54:31.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:54:31.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:54:31.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:54:31.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:54:31.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:54:30.214007+0000 mgr.x (mgr.24542) 2884 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:54:31.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:54:30.238574+0000 mgr.x (mgr.24542) 2885 : cephadm [ERR] 2021-07-31T02:54:31.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:54:31.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:54:31.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: yield 2021-07-31T02:54:31.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:54:31.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:54:31.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:54:31.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:54:31.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:54:31.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:54:31.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:54:31.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:54:31.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:54:31.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:54:31.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:54:31.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:54:30.239459+0000 mgr.x (mgr.24542) 2886 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:54:31.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:54:31.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:54:31.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: yield 2021-07-31T02:54:31.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:54:31.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:54:31.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:54:31.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:54:31.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:54:31.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:54:31.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:54:31.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:54:31.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:54:31.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:54:31.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:54:31.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: 2021-07-31T02:54:31.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:54:31.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: 2021-07-31T02:54:31.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:54:31.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:54:31.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:54:31.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:54:31.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:54:31.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:54:31.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:54:31.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:54:31.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:54:31.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:54:31.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:54:31.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:54:31.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:54:31.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:54:31.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:54:31.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:54:31.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:54:31.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:54:31.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: raise 2021-07-31T02:54:31.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:54:31.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:54:31.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:54:31.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:54:31.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:54:31.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:54:30.247297+0000 mgr.x (mgr 2021-07-31T02:54:31.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: .24542) 2887 : cephadm [ERR] 2021-07-31T02:54:31.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:54:31.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:54:31.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: yield 2021-07-31T02:54:31.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:54:31.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:54:31.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:54:31.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:54:31.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:54:31.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:54:31.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:54:31.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:54:31.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:54:31.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:54:31.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:54:31.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:54:30.247713+0000 mgr.x (mgr.24542) 2888 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:54:31.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:54:31.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:54:31.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: yield 2021-07-31T02:54:31.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:54:31.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:54:31.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:54:31.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:54:31.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:54:31.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:54:31.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:54:31.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:54:31.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:54:31.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:54:31.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:54:31.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: 2021-07-31T02:54:31.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:54:31.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: 2021-07-31T02:54:31.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:54:31.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:54:31.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:54:31.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:54:31.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:54:31.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:54:31.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:54:31.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:54:31.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:54:31.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:54:31.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:54:31.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:54:31.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:54:31.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:54:31.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:54:31.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:54:31.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:54:31.285 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:54:31.285 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: raise 2021-07-31T02:54:31.285 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:54:31.285 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:54:31.285 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:54:31.286 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:54:31.286 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:30 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:54:31.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: audit 2021-07-31T02:54: 2021-07-31T02:54:31.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: 30.155998+0000 mon.b (mon.2) 412 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:54:31.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: audit 2021-07-31T02:54:31.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: 2021- 2021-07-31T02:54:31.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: 07-31T02:54:30 2021-07-31T02:54:31.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: .157551+0000 mon.b (mon. 2021-07-31T02:54:31.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: 2) 413 : audit [DBG] 2021-07-31T02:54:31.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:54:31.289 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: audit 2021- 2021-07-31T02:54:31.289 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: 07-31T02:54:30. 2021-07-31T02:54:31.289 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: 158385+0000 mon.b (mon.2 2021-07-31T02:54:31.289 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: ) 414 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:54:31.289 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: 2021-07-31T02:54:31.290 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:54:31.290 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: 2021-07-31T02: 2021-07-31T02:54:31.290 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: 54:30.159119+0000 mgr.x 2021-07-31T02:54:31.290 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: (mgr.24542) 2880 : cephadm 2021-07-31T02:54:31.291 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:54:31.291 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: cephadm 2021- 2021-07-31T02:54:31.291 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: 07-31T02:54:30. 2021-07-31T02:54:31.291 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: 201602+0000 mgr.x (mgr.24542 2021-07-31T02:54:31.291 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: ) 2881 : cephadm [ERR] 2021-07-31T02:54:31.292 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:54:31.292 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:54:31.292 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: yield 2021-07-31T02:54:31.292 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:54:31.292 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:54:31.293 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:54:31.293 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:54:31.293 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:54:31.293 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:54:31.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:54:31.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:54:31.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:54:31.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:54:31.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:54:31.295 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: 2021-07-31T02:54:31.295 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:54:30.202063+0000 mgr.x (mgr.24542) 2882 : cephadm 2021-07-31T02:54:31.295 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:54:31.295 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:54:31.295 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: 2021-07 2021-07-31T02:54:31.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: -31T02:54:30. 2021-07-31T02:54:31.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: 213096+0000 mgr.x (mgr.24542 2021-07-31T02:54:31.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: ) 2883 : cephadm [ERR] 2021-07-31T02:54:31.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:54:31.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:54:31.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: yield 2021-07-31T02:54:31.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:54:31.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:54:31.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:54:31.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:54:31.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:54:31.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:54:31.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:54:31.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:54:31.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:54:31.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:54:31.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:54:31.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: 2021-07-31T02:54:31.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:54: 2021-07-31T02:54:31.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: 30.214007+0000 mgr.x (mgr.24542 2021-07-31T02:54:31.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: ) 2884 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:54:31.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: 2021-07-31T02:54:31.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: cephadm 2021 2021-07-31T02:54:31.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: -07-31T02:54:30 2021-07-31T02:54:31.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: .238574+0000 mgr.x (mgr.24542 2021-07-31T02:54:31.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: ) 2885 : cephadm [ERR] 2021-07-31T02:54:31.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:54:31.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:54:31.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: yield 2021-07-31T02:54:31.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:54:31.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:54:31.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:54:31.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:54:31.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:54:31.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:54:31.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:54:31.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:54:31.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:54:31.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:54:31.304 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:54:31.304 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: 2021-07-31T02:54:31.304 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:54:30.239459+0000 mgr.x (mgr.24542) 2886 : cephadm 2021-07-31T02:54:31.304 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:54:31.304 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:54:31.305 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:54:31.305 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: yield 2021-07-31T02:54:31.305 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:54:31.305 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:54:31.306 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:54:31.306 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:54:31.306 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:54:31.306 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:54:31.306 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:54:31.307 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:54:31.307 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:54:31.307 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:54:31.307 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:54:31.308 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: 2021-07-31T02:54:31.308 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:54:31.308 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: 2021-07-31T02:54:31.308 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:54:31.308 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:54:31.309 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:54:31.309 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:54:31.309 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:54:31.309 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:54:31.309 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:54:31.310 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:54:31.310 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:54:31.310 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:54:31.310 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:54:31.311 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:54:31.311 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:54:31.311 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:54:31.311 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:54:31.311 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:54:31.312 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:54:31.312 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:54:31.312 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: raise 2021-07-31T02:54:31.312 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:54:31.312 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:54:31.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:54:31.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:54:31.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:54:31.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:54:30.247297+0000 mgr.x (mgr.24542) 2887 : cephadm [ERR] 2021-07-31T02:54:31.314 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:54:31.314 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:54:31.314 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: yield 2021-07-31T02:54:31.314 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:54:31.314 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:54:31.315 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:54:31.315 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: audit 2021-07-31T02:54:30.155998+0000 mon.b (mon.2) 412 : audit 2021-07-31T02:54:31.315 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:54:31.316 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: audit 2021-07-31T02:54:30.157551+0000 mon.b (mon.2) 413 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:54:31.316 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: audit 2021-07-31T02:54:30.158385+0000 mon.b (mon.2 2021-07-31T02:54:31.316 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: ) 414 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:54:31.316 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:54:31.316 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: 2021-07-31T02:54 2021-07-31T02:54:31.317 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: :30.159119+0000 mgr.x (mgr.24542) 2880 2021-07-31T02:54:31.317 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:54:31.317 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:54:31.317 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: 2021-07-31T02:54:30 2021-07-31T02:54:31.318 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: .201602+0000 mgr.x (mgr. 2021-07-31T02:54:31.318 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: 24542) 2881 : cephadm [ERR] 2021-07-31T02:54:31.318 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:54:31.318 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:54:31.318 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: yield 2021-07-31T02:54:31.319 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:54:31.319 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:54:31.319 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:54:31.319 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:54:31.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:54:31.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:54:31.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:54:31.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:54:31.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:54:31.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:54:31.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:54:31.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:54:30.202063+0000 mgr.x (mgr 2021-07-31T02:54:31.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: .24542) 2882 : cephadm [INF] 2021-07-31T02:54:31.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:54:31.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: cephadm 2021 2021-07-31T02:54:31.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: -07-31T02:54:30. 2021-07-31T02:54:31.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: 213096+0000 mgr.x (mgr.24542 2021-07-31T02:54:31.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: ) 2883 : cephadm [ERR] 2021-07-31T02:54:31.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:54:31.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:54:31.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: yield 2021-07-31T02:54:31.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:54:31.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:54:31.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:54:31.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:54:31.325 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:54:31.325 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:54:31.325 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:54:31.325 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:54:31.325 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:54:31.326 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:54:31.326 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:54:31.326 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:54:31.326 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:54:30.247713+0000 mgr.x (mgr.24542) 2888 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:54:31.327 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:54:31.327 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:54:31.327 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: yield 2021-07-31T02:54:31.327 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:54:31.327 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:54:31.328 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:54:31.328 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:54:31.328 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:54:31.328 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:54:31.328 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:54:31.329 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:54:31.329 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:54:31.329 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:54:31.329 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:54:31.329 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: 2021-07-31T02:54:31.330 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:54:31.330 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: 2021-07-31T02:54:31.330 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:54:31.330 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:54:31.331 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:54:31.331 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:54:31.331 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:54:31.331 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:54:31.331 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:54:31.332 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:54:31.332 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:54:31.332 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:54:31.332 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:54:31.332 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:54:31.333 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:54:31.333 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:54:31.333 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:54:31.333 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:54:31.333 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:54:31.334 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:54:31.334 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: raise 2021-07-31T02:54:31.334 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:54:31.334 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:54:31.334 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:54:31.335 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:54:31.335 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:54:31.336 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:54:31.336 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:54:31.336 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:54:31.336 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:54:31.337 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:54:31.337 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:54:31.337 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:54:31.337 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: 2021-07-31T02:54:31.337 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:54:30.214007+0000 mgr.x (mgr.24542 2021-07-31T02:54:31.338 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: ) 2884 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:54:31.338 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:54:31.338 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: 2021-07-31T02:54:30 2021-07-31T02:54:31.338 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: .238574+0000 mgr.x (mgr.24542 2021-07-31T02:54:31.338 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: ) 2885 : cephadm [ERR] 2021-07-31T02:54:31.339 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:54:31.339 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:54:31.339 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: yield 2021-07-31T02:54:31.339 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:54:31.340 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:54:31.340 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:54:31.340 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:54:31.340 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:54:31.340 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:54:31.341 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:54:31.341 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:54:31.341 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:54:31.341 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:54:31.341 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:54:31.342 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:54:30.239459+0000 mgr.x (mgr.24542) 2886 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:54:31.342 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:54:31.342 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:54:31.342 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: yield 2021-07-31T02:54:31.343 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:54:31.343 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:54:31.343 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:54:31.343 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:54:31.343 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:54:31.344 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:54:31.344 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:54:31.344 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:54:31.344 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:54:31.345 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:54:31.345 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:54:31.345 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: 2021-07-31T02:54:31.345 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:54:31.345 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: 2021-07-31T02:54:31.345 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:54:31.346 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:54:31.346 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:54:31.346 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:54:31.346 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:54:31.346 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:54:31.347 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:54:31.347 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:54:31.347 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:54:31.347 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:54:31.348 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:54:31.348 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:54:31.348 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:54:31.348 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:54:31.348 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:54:31.349 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:54:31.349 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:54:31.349 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:54:31.349 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: raise 2021-07-31T02:54:31.349 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:54:31.350 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:54:31.350 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:54:31.350 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:54:31.350 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:54:31.350 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:54:30.247297+0000 mgr.x (mgr.24542) 2887 : cephadm [ERR] 2021-07-31T02:54:31.351 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:54:31.351 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:54:31.351 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: yield 2021-07-31T02:54:31.351 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:54:31.351 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:54:31.352 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:54:31.352 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:54:31.352 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:54:31.352 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:54:31.352 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:54:31.353 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:54:31.353 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:54:31.353 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:54:31.353 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:54:31.353 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:54:30.247713+0000 mgr.x (mgr.24542) 2888 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:54:31.354 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:54:31.354 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:54:31.354 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: yield 2021-07-31T02:54:31.354 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:54:31.355 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:54:31.355 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:54:31.355 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:54:31.355 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:54:31.355 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:54:31.356 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:54:31.356 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:54:31.356 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:54:31.356 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:54:31.356 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:54:31.357 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: 2021-07-31T02:54:31.357 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:54:31.357 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: 2021-07-31T02:54:31.357 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:54:31.357 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:54:31.358 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:54:31.358 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:54:31.358 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:54:31.358 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:54:31.359 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:54:31.359 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:54:31.359 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:54:31.359 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:54:31.359 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:54:31.360 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:54:31.360 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:54:31.360 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:54:31.360 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:54:31.360 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:54:31.361 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:54:31.361 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:54:31.361 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: raise 2021-07-31T02:54:31.361 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:54:31.362 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:54:31.362 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:54:31.362 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:54:31.362 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:30 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:54:32.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:31 smithi154 conmon[34553]: cluster 2021-07-31T02:54:30.763142+0000 mgr.x (mgr.24542) 2889 : cluster 2021-07-31T02:54:32.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:31 smithi154 conmon[34553]: [DBG] pgmap v4026: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:32.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:31 smithi045 conmon[33655]: cluster 2021-07-31T02:54:30.763142+0000 mgr.x (mgr.24542) 2889 : cluster [DBG] pgmap v4026: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:32.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:31 smithi045 conmon[39718]: cluster 2021-07-31T02:54:30.763142+0000 mgr.x (mgr.24542) 2889 : cluster [DBG] pgmap v4026: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:34.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:33 smithi045 conmon[39718]: cluster 2021-07-31T02:54:32.763540+0000 mgr.x ( 2021-07-31T02:54:34.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:33 smithi045 conmon[39718]: mgr.24542) 2890 : cluster [DBG] pgmap v4027: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:34.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:33 smithi045 conmon[33655]: cluster 2021-07-31T02:54:32.763540+0000 mgr.x (mgr.24542) 2890 : cluster [DBG] pgmap v4027: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:34.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:33 smithi154 conmon[34553]: cluster 2021-07-31T02:54:32.763540+0000 mgr.x (mgr.24542) 2890 : cluster [DBG] 2021-07-31T02:54:34.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:33 smithi154 conmon[34553]: pgmap v4027: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:36.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:35 smithi045 conmon[39718]: cluster 2021-07-31T02:54:34.764179+0000 mgr.x ( 2021-07-31T02:54:36.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:35 smithi045 conmon[39718]: mgr.24542) 2891 : cluster [DBG] pgmap v4028: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:36.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:35 smithi045 conmon[33655]: cluster 2021-07-31T02:54:34.764179+0000 mgr.x (mgr.24542) 2891 : cluster [DBG] pgmap v4028: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:36.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:35 smithi154 conmon[34553]: cluster 2021-07-31T02:54:34.764179+0000 mgr.x (mgr.24542) 2891 : cluster 2021-07-31T02:54:36.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:35 smithi154 conmon[34553]: [DBG] pgmap v4028: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:38.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:37 smithi045 conmon[39718]: cluster 2021-07-31T02:54:36.764663+0000 mgr.x (mgr 2021-07-31T02:54:38.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:37 smithi045 conmon[39718]: .24542) 2892 : cluster [DBG] pgmap v4029: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:38.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:37 smithi045 conmon[33655]: cluster 2021-07-31T02:54:36.764663 2021-07-31T02:54:38.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:37 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 2892 : cluster [DBG] pgmap v4029: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:38.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:37 smithi154 conmon[34553]: cluster 2021-07-31T02:54:36.764663+0000 mgr.x (mgr.24542) 2892 : cluster [DBG] pgmap v4029: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:38.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:37 smithi154 conmon[34553]: 2021-07-31T02:54:40.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:39 smithi154 conmon[34553]: cluster 2021-07-31T02:54:38.765080+0000 mgr.x (mgr.24542) 2893 : cluster [DBG] pgmap v4030: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:40.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:39 smithi045 conmon[39718]: cluster 2021-07-31T02:54:38.765080+0000 mgr.x (mgr.24542) 2893 : cluster [DBG] pgmap v4030: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:40.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:39 smithi045 conmon[33655]: cluster 2021-07-31T02:54:38.765080 2021-07-31T02:54:40.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:39 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 2893 : cluster [DBG] pgmap v4030: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:42.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:41 smithi154 conmon[34553]: cluster 2021-07-31T02:54:40.765559+0000 mgr.x (mgr.24542) 2894 : cluster [DBG] pgmap v4031: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:42.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:41 smithi045 conmon[39718]: cluster 2021-07-31T02:54:40.765559+0000 2021-07-31T02:54:42.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:41 smithi045 conmon[39718]: mgr.x (mgr.24542) 2894 : cluster [DBG] pgmap v4031: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:42.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:41 smithi045 conmon[33655]: cluster 2021-07-31T02:54:40.765559+0000 mgr.x 2021-07-31T02:54:42.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:41 smithi045 conmon[33655]: (mgr.24542) 2894 : cluster [DBG] pgmap v4031: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:44.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:43 smithi045 conmon[39718]: cluster 2021-07-31T02:54:42.765903+0000 mgr.x (mgr 2021-07-31T02:54:44.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:43 smithi045 conmon[39718]: .24542) 2895 : cluster [DBG] pgmap v4032: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:44.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:43 smithi045 conmon[33655]: cluster 2021-07-31T02:54:42.765903+0000 mgr.x (mgr.24542) 2895 : cluster [DBG] pgmap v4032: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:44.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:43 smithi154 conmon[34553]: cluster 2021-07-31T02:54:42.765903+0000 mgr.x (mgr.24542) 2895 : cluster [DBG] 2021-07-31T02:54:44.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:43 smithi154 conmon[34553]: pgmap v4032: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:46.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:45 smithi045 conmon[39718]: cluster 2021-07-31T02:54:44.766311+0000 mgr.x ( 2021-07-31T02:54:46.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:45 smithi045 conmon[39718]: mgr.24542) 2896 : cluster [DBG] pgmap v4033: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:46.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:45 smithi045 conmon[33655]: cluster 2021-07-31T02:54:44.766311+0000 2021-07-31T02:54:46.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:45 smithi045 conmon[33655]: mgr.x (mgr.24542) 2896 : cluster [DBG] pgmap v4033: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:46.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:45 smithi154 conmon[34553]: cluster 2021-07-31T02:54:44.766311+0000 mgr.x (mgr.24542) 2896 : cluster [DBG] pgmap v4033: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:48.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:47 smithi045 conmon[33655]: cluster 2021-07-31T02:54:46.766773+0000 mgr.x (mgr. 2021-07-31T02:54:48.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:47 smithi045 conmon[33655]: 24542) 2897 : cluster [DBG] pgmap v4034: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:48.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:47 smithi045 conmon[39718]: cluster 2021-07-31T02:54: 2021-07-31T02:54:48.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:47 smithi045 conmon[39718]: 46.766773+0000 mgr.x (mgr.24542) 2897 : cluster [DBG] pgmap v4034: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:48.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:47 smithi154 conmon[34553]: cluster 2021-07-31T02:54:46.766773+0000 mgr.x (mgr.24542) 2897 : cluster 2021-07-31T02:54:48.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:47 smithi154 conmon[34553]: [DBG] pgmap v4034: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:50.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:49 smithi154 conmon[34553]: cluster 2021-07-31T02:54:48.767152+0000 mgr.x (mgr.24542) 2898 : cluster [DBG] 2021-07-31T02:54:50.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:49 smithi154 conmon[34553]: pgmap v4035: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:50.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:49 smithi045 conmon[33655]: cluster 2021-07-31T02:54:48.767152+0000 mgr.x (mgr.24542) 2898 : cluster [DBG] pgmap v4035: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:50.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:49 smithi045 conmon[39718]: cluster 2021-07-31T02:54:48.767152+0000 mgr.x (mgr.24542) 2898 : cluster [DBG] 2021-07-31T02:54:50.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:49 smithi045 conmon[39718]: pgmap v4035: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:52.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:51 smithi154 conmon[34553]: cluster 2021-07-31T02:54:50.767643+0000 mgr.x (mgr.24542) 2899 : cluster 2021-07-31T02:54:52.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:51 smithi154 conmon[34553]: [DBG] pgmap v4036: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:52.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:51 smithi045 conmon[33655]: cluster 2021-07-31T02:54:50.767643+0000 mgr.x (mgr.24542) 2899 : cluster [DBG] pgmap v4036: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:52.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:51 smithi045 conmon[39718]: cluster 2021-07-31T02:54:50.767643+0000 mgr.x (mgr.24542) 2899 : cluster 2021-07-31T02:54:52.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:51 smithi045 conmon[39718]: [DBG] pgmap v4036: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:54.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:53 smithi045 conmon[39718]: cluster 2021-07-31T02:54:52.768030+0000 mgr.x (mgr.24542) 2900 : cluster 2021-07-31T02:54:54.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:53 smithi045 conmon[39718]: [DBG] pgmap v4037: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:54.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:53 smithi045 conmon[33655]: cluster 2021-07-31T02:54:52.768030+0000 mgr.x (mgr.24542) 2900 : cluster [DBG] 2021-07-31T02:54:54.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:53 smithi045 conmon[33655]: pgmap v4037: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:54.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:53 smithi154 conmon[34553]: cluster 2021-07-31T02:54:52.768030+0000 mgr.x (mgr.24542) 2900 2021-07-31T02:54:54.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:53 smithi154 conmon[34553]: : cluster [DBG] pgmap v4037: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:56.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:55 smithi154 conmon[34553]: cluster 2021-07-31T02:54:54.768502+0000 mgr.x (mgr.24542) 2901 : cluster [DBG] 2021-07-31T02:54:56.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:55 smithi154 conmon[34553]: pgmap v4038: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:56.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:55 smithi045 conmon[33655]: cluster 2021-07-31T02:54:54.768502+0000 mgr.x (mgr.24542) 2901 : 2021-07-31T02:54:56.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:55 smithi045 conmon[33655]: cluster [DBG] pgmap v4038: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:56.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:55 smithi045 conmon[39718]: cluster 2021-07-31T02:54:54.768502+0000 mgr.x (mgr.24542) 2901 : 2021-07-31T02:54:56.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:55 smithi045 conmon[39718]: cluster [DBG] pgmap v4038: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:58.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:57 smithi154 conmon[34553]: cluster 2021-07-31T02:54:56.769006+0000 mgr.x (mgr.24542) 2021-07-31T02:54:58.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:57 smithi154 conmon[34553]: 2902 : cluster [DBG] pgmap v4039: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:58.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:57 smithi045 conmon[33655]: cluster 2021-07-31T02:54:56.769006+0000 mgr.x (mgr.24542) 2902 : cluster [DBG] pgmap v4039: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:58.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:57 smithi045 conmon[39718]: cluster 2021-07-31T02:54:56.769006+0000 mgr.x (mgr.24542) 2902 2021-07-31T02:54:58.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:57 smithi045 conmon[39718]: : cluster [DBG] pgmap v4039: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:54:59.175 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:54:59.451 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:54:59.452 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 69m ago 70m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:54:59.452 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 67m ago 70m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:54:59.452 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 67m ago 73m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:54:59.453 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 69m ago 76m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:54:59.453 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 69m ago 76m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:54:59.453 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 67m ago 74m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:54:59.453 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 69m ago 74m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:54:59.453 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 69m ago 70m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:54:59.454 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 67m ago 70m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:54:59.454 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 69m ago 73m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:54:59.454 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 69m ago 73m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:54:59.454 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 69m ago 72m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:54:59.455 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 69m ago 72m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:54:59.455 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 67m ago 72m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:54:59.455 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 67m ago 71m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:54:59.455 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 67m ago 71m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:54:59.455 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 67m ago 71m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:54:59.456 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 67m ago 70m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:54:59.789 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:54:59.789 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:54:59.790 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:54:59.790 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:54:59.790 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:54:59.790 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:54:59.791 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:54:59.791 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:54:59.791 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:54:59.791 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:54:59.791 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:54:59.792 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:54:59.792 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:54:59.792 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:54:59.792 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:54:59.792 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:54:59.793 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:55:00.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:59 smithi154 conmon[34553]: cluster 2021-07-31T02:54:58.769362+0000 mgr.x (mgr.24542) 2903 : cluster [DBG] pgmap v4040: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:55:00.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:59 smithi154 conmon[34553]: audit 2021-07-31T02:54 2021-07-31T02:55:00.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:59 smithi154 conmon[34553]: :59.163323+0000 mgr.x (mgr.24542) 2904 : audit [DBG] from='client.18408 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:55:00.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:54:59 smithi154 conmon[34553]: audit 2021-07-31T02:54:59.444010+0000 mgr.x (mgr.24542) 2905 : audit [DBG] from='client.28354 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:55:00.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:59 smithi045 conmon[39718]: cluster 2021-07-31T02:54:58. 2021-07-31T02:55:00.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:59 smithi045 conmon[39718]: 769362+0000 mgr.x (mgr.24542) 2903 : cluster [DBG] pgmap v4040: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:55:00.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:59 smithi045 conmon[39718]: audit 2021-07-31T02:54 2021-07-31T02:55:00.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:59 smithi045 conmon[39718]: :59.163323+0000 mgr.x (mgr.24542 2021-07-31T02:55:00.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:59 smithi045 conmon[39718]: ) 2904 : audit [DBG] from='client.18408 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:55:00.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:59 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T02:55:00.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:59 smithi045 conmon[39718]: T02:54:59.444010+0000 mgr.x ( 2021-07-31T02:55:00.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:59 smithi045 conmon[39718]: mgr.24542) 2905 : audit [DBG] 2021-07-31T02:55:00.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:54:59 smithi045 conmon[39718]: from='client.28354 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:55:00.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:59 smithi045 conmon[33655]: cluster 2021-07-31T02: 2021-07-31T02:55:00.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:59 smithi045 conmon[33655]: 54:58.769362+0000 mgr.x (mgr.24542) 2903 : cluster [DBG] pgmap v4040: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:55:00.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:59 smithi045 conmon[33655]: audit 2021-07-31T02:54:59.163323+0000 mgr.x (mgr.24542 2021-07-31T02:55:00.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:59 smithi045 conmon[33655]: ) 2904 : audit [DBG] from='client.18408 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:55:00.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:59 smithi045 conmon[33655]: audit 2021-07-31T02 2021-07-31T02:55:00.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:59 smithi045 conmon[33655]: :54:59.444010+0000 mgr.x (mgr.24542) 2905 : 2021-07-31T02:55:00.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:54:59 smithi045 conmon[33655]: audit [DBG] from='client.28354 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:55:01.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:00 smithi154 conmon[34553]: audit 2021-07-31T02:54:59.789287+0000 mon.a (mon.0) 3223 : audit [DBG] 2021-07-31T02:55:01.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:00 smithi154 conmon[34553]: from='client.? 172.21.15.45:0/3484753717' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:55:01.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:00 smithi045 conmon[39718]: audit 2021-07-31T02:54:59.789287+0000 mon.a (mon.0) 3223 : audit [DBG] from='client.? 172.21.15.45:0/3484753717' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:55:01.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:00 smithi045 conmon[39718]: 2021-07-31T02:55:01.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:00 smithi045 conmon[33655]: audit 2021-07-31T02:54:59.789287+0000 mon.a (mon.0) 3223 : 2021-07-31T02:55:01.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:00 smithi045 conmon[33655]: audit [DBG] from='client.? 172.21.15.45:0/3484753717' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:55:02.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:01 smithi154 conmon[34553]: cluster 2021-07-31T02:55:00.770003+0000 mgr.x (mgr.24542) 2906 : cluster [DBG] pgmap v4041: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:55:02.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:01 smithi045 conmon[39718]: cluster 2021-07-31T02:55:00.770003+0000 mgr.x 2021-07-31T02:55:02.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:01 smithi045 conmon[39718]: (mgr.24542) 2906 : cluster [DBG] pgmap v4041: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:55:02.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:01 smithi045 conmon[33655]: cluster 2021-07-31T02:55:00.770003+0000 mgr.x (mgr.24542) 2906 2021-07-31T02:55:02.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:01 smithi045 conmon[33655]: : cluster [DBG] pgmap v4041: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:55:04.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:03 smithi045 conmon[33655]: cluster 2021-07-31T02:55:02.770352+0000 mgr.x (mgr.24542) 2021-07-31T02:55:04.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:03 smithi045 conmon[33655]: 2907 : cluster [DBG] pgmap v4042: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:55:04.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:03 smithi045 conmon[39718]: cluster 2021-07-31T02:55:02.770352+0000 mgr.x (mgr. 2021-07-31T02:55:04.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:03 smithi045 conmon[39718]: 24542) 2907 : cluster [DBG] pgmap v4042: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:55:04.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:03 smithi154 conmon[34553]: cluster 2021-07-31T02:55:02.770352+0000 mgr.x (mgr.24542) 2907 : cluster [DBG] pgmap v4042: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:55:06.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:05 smithi154 conmon[34553]: cluster 2021-07-31T02:55:04.770758+0000 mgr.x (mgr.24542) 2908 : cluster [DBG] pgmap v4043: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:55:06.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:05 smithi045 conmon[33655]: cluster 2021-07-31T02:55:04.770758+0000 mgr.x (mgr.24542) 2908 2021-07-31T02:55:06.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:05 smithi045 conmon[33655]: : cluster [DBG] pgmap v4043: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:55:06.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:05 smithi045 conmon[39718]: cluster 2021-07-31T02:55:04.770758+0000 mgr.x (mgr. 2021-07-31T02:55:06.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:05 smithi045 conmon[39718]: 24542) 2908 : cluster [DBG] pgmap v4043: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:55:08.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:07 smithi154 conmon[34553]: cluster 2021-07-31T02:55:06.771273+0000 mgr.x (mgr.24542) 2909 : cluster [DBG] pgmap v4044: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:55:08.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:07 smithi045 conmon[33655]: cluster 2021-07-31T02:55:06.771273+0000 mgr.x (mgr.24542) 2909 : cluster [DBG] pgmap v4044: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:55:08.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:07 smithi045 conmon[39718]: cluster 2021-07-31T02:55:06.771273+0000 mgr.x (mgr.24542 2021-07-31T02:55:08.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:07 smithi045 conmon[39718]: ) 2909 : cluster [DBG] pgmap v4044: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:55:10.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:09 smithi154 conmon[34553]: cluster 2021-07-31T02:55:08.771644+0000 mgr.x (mgr.24542) 2910 : cluster [DBG] pgmap v4045: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:55:10.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:09 smithi045 conmon[39718]: cluster 2021-07-31T02:55:08.771644+0000 2021-07-31T02:55:10.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:09 smithi045 conmon[39718]: mgr.x (mgr.24542) 2910 : cluster [DBG] pgmap v4045: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:55:10.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:09 smithi045 conmon[33655]: cluster 2021-07-31T02:55:08.771644+0000 mgr.x (mgr.24542) 2910 : cluster 2021-07-31T02:55:10.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:09 smithi045 conmon[33655]: [DBG] pgmap v4045: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:55:12.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:11 smithi154 conmon[34553]: cluster 2021-07-31T02:55:10.772091+0000 mgr.x (mgr.24542) 2911 : cluster [DBG] pgmap v4046: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:55:12.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:11 smithi045 conmon[39718]: cluster 2021-07-31T02:55:10.772091+0000 mgr.x (mgr.24542) 2911 : cluster [DBG] 2021-07-31T02:55:12.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:11 smithi045 conmon[39718]: pgmap v4046: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:55:12.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:11 smithi045 conmon[33655]: cluster 2021-07-31T02:55:10.772091+0000 mgr.x (mgr.24542) 2911 2021-07-31T02:55:12.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:11 smithi045 conmon[33655]: : cluster [DBG] pgmap v4046: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:55:14.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:13 smithi045 conmon[33655]: cluster 2021-07-31T02:55:12.772442+0000 mgr.x ( 2021-07-31T02:55:14.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:13 smithi045 conmon[33655]: mgr.24542) 2912 : cluster [DBG] pgmap v4047: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:55:14.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:13 smithi045 conmon[39718]: cluster 2021-07-31T02:55:12.772442+0000 mgr.x (mgr.24542) 2912 : cluster 2021-07-31T02:55:14.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:13 smithi045 conmon[39718]: [DBG] pgmap v4047: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:55:14.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:13 smithi154 conmon[34553]: cluster 2021-07-31T02:55:12.772442+0000 mgr.x (mgr.24542) 2912 : cluster [DBG] 2021-07-31T02:55:14.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:13 smithi154 conmon[34553]: pgmap v4047: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:55:16.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:15 smithi154 conmon[34553]: cluster 2021-07-31T02:55:14.772963+0000 mgr.x (mgr.24542) 2913 : cluster [DBG] pgmap v4048: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:55:16.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:15 smithi045 conmon[33655]: cluster 2021-07-31T02:55:14.772963+0000 mgr.x (mgr. 2021-07-31T02:55:16.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:15 smithi045 conmon[33655]: 24542) 2913 : cluster [DBG] pgmap v4048: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:55:16.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:15 smithi045 conmon[39718]: cluster 2021-07-31T02:55:14.772963+0000 2021-07-31T02:55:16.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:15 smithi045 conmon[39718]: mgr.x (mgr.24542) 2913 : cluster [DBG] pgmap v4048: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:55:18.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:17 smithi154 conmon[34553]: cluster 2021-07-31T02:55:16.773385+0000 mgr.x (mgr.24542) 2914 : cluster [DBG] pgmap v4049: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:55:18.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:17 smithi045 conmon[33655]: cluster 2021-07-31T02:55:16.773385+0000 mgr.x (mgr.24542) 2914 : cluster [DBG] pgmap v4049: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:55:18.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:17 smithi045 conmon[39718]: cluster 2021-07-31T02:55:16.773385+0000 mgr.x (mgr. 2021-07-31T02:55:18.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:17 smithi045 conmon[39718]: 24542) 2914 : cluster [DBG] pgmap v4049: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:55:20.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:19 smithi154 conmon[34553]: cluster 2021-07-31T02:55:18.773896+0000 mgr.x (mgr.24542) 2915 : cluster [DBG] pgmap v4050: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:55:20.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:19 smithi045 conmon[33655]: cluster 2021-07-31T02:55:18.773896+0000 mgr.x (mgr.24542) 2915 : cluster [DBG] pgmap v4050: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:55:20.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:19 smithi045 conmon[39718]: cluster 2021-07-31T02:55:18 2021-07-31T02:55:20.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:19 smithi045 conmon[39718]: .773896+0000 mgr.x (mgr.24542) 2915 : cluster [DBG] pgmap v4050: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:55:22.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:21 smithi154 conmon[34553]: cluster 2021-07-31T02:55:20.774511+0000 mgr.x (mgr.24542) 2916 : cluster [DBG] pgmap v4051: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:55:22.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:21 smithi045 conmon[33655]: cluster 2021-07-31T02:55:20.774511+0000 mgr.x (mgr.24542) 2916 : cluster [DBG] pgmap v4051: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:55:22.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:21 smithi045 conmon[39718]: cluster 2021-07-31T02:55:20.774511+0000 mgr.x (mgr.24542 2021-07-31T02:55:22.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:21 smithi045 conmon[39718]: ) 2916 : cluster [DBG] pgmap v4051: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:55:24.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:23 smithi154 conmon[34553]: cluster 2021-07-31T02:55:22.774996+0000 mgr.x (mgr.24542) 2917 : cluster [DBG] pgmap v4052: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:55:24.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:23 smithi045 conmon[39718]: cluster 2021-07-31T02:55:22.774996+0000 mgr.x (mgr 2021-07-31T02:55:24.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:23 smithi045 conmon[39718]: .24542) 2917 : cluster [DBG] pgmap v4052: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:55:24.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:23 smithi045 conmon[33655]: cluster 2021-07-31T02:55:22.774996+0000 mgr.x (mgr.24542) 2917 : 2021-07-31T02:55:24.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:23 smithi045 conmon[33655]: cluster [DBG] pgmap v4052: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:55:26.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:25 smithi154 conmon[34553]: cluster 2021-07-31T02:55:24.775508+0000 mgr.x (mgr.24542) 2918 : cluster [DBG] pgmap v4053: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:55:26.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:25 smithi045 conmon[33655]: cluster 2021-07-31T02:55:24.775508+0000 mgr.x (mgr.24542) 2918 : 2021-07-31T02:55:26.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:25 smithi045 conmon[33655]: cluster [DBG] pgmap v4053: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:55:26.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:25 smithi045 conmon[39718]: cluster 2021-07-31T02:55:24.775508+0000 mgr.x 2021-07-31T02:55:26.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:25 smithi045 conmon[39718]: (mgr.24542) 2918 : cluster [DBG] pgmap v4053: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:55:28.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:27 smithi154 conmon[34553]: cluster 2021-07-31T02:55:26.776040+0000 mgr.x (mgr.24542) 2919 : cluster [DBG] pgmap v4054: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:55:28.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:27 smithi045 conmon[33655]: cluster 2021-07-31T02:55:26.776040+0000 mgr.x (mgr.24542) 2919 : cluster 2021-07-31T02:55:28.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:27 smithi045 conmon[33655]: [DBG] pgmap v4054: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:55:28.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:27 smithi045 conmon[39718]: cluster 2021-07-31T02:55:26.776040+0000 mgr.x (mgr.24542) 2919 : cluster [DBG] 2021-07-31T02:55:28.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:27 smithi045 conmon[39718]: pgmap v4054: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:55:29.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:28 smithi154 conmon[34553]: audit 2021-07-31T02:55:28.208179+0000 mon.b (mon.2) 415 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:55:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:28 smithi154 conmon[34553]: audit 2021-07-31T02:55:28.208418+0000 mon.b (mon.2) 416 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:55:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:28 smithi154 conmon[34553]: audit 2021-07-31T02:55:28.208866+0000 mon.a (mon.0) 2021-07-31T02:55:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:28 smithi154 conmon[34553]: 3224 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:55:29.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:28 smithi154 conmon[34553]: audit 2021 2021-07-31T02:55:29.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:28 smithi154 conmon[34553]: -07-31T02:55:28.209347+0000 mon.a (mon.0) 3225 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:55:29.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:28 smithi045 conmon[33655]: audit 2021-07-31T02:55:28.208179+0000 2021-07-31T02:55:29.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:28 smithi045 conmon[33655]: mon.b (mon.2) 415 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:55:29.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:28 smithi045 conmon[33655]: audit 2021-07-31T02:55:28.208418+0000 mon.b (mon.2) 416 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:55:29.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:28 smithi045 conmon[33655]: audit 2021-07-31T02:55:28.208866+0000 mon.a (mon.0) 3224 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:55:29.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:28 smithi045 conmon[33655]: audit 2021-07-31T02:55:28.209347+0000 mon.a (mon.0) 3225 : 2021-07-31T02:55:29.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:28 smithi045 conmon[33655]: audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:55:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:28 smithi045 conmon[39718]: audit 2021-07-31T02:55:28.208179+0000 mon.b (mon.2) 415 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:55:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:28 smithi045 conmon[39718]: 2021-07-31T02:55:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:28 smithi045 conmon[39718]: audit 2021-07-31T02:55:28.208418+0000 mon.b (mon.2) 416 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:55:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:28 smithi045 conmon[39718]: audit 2021-07 2021-07-31T02:55:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:28 smithi045 conmon[39718]: -31T02:55:28.208866+0000 mon.a (mon.0) 3224 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:55:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:28 smithi045 conmon[39718]: audit 2021-07 2021-07-31T02:55:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:28 smithi045 conmon[39718]: -31T02:55:28.209347+0000 mon.a (mon.0) 3225 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:55:30.101 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:55:30.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:29 smithi154 conmon[34553]: cluster 2021-07-31T02:55:28.776525+0000 mgr.x (mgr.24542) 2920 : cluster [DBG] pgmap v4055: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:55:30.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:29 smithi045 conmon[39718]: cluster 2021-07-31T02:55:28.776525+0000 mgr.x (mgr.24542) 2920 : cluster [DBG] 2021-07-31T02:55:30.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:29 smithi045 conmon[39718]: pgmap v4055: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:55:30.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:29 smithi045 conmon[33655]: cluster 2021-07-31T02:55:28.776525+0000 mgr.x (mgr.24542) 2920 : cluster [DBG] pgmap v4055: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:55:30.382 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:55:30.382 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 70m ago 71m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:55:30.382 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 68m ago 70m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:55:30.383 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 68m ago 74m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:55:30.383 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 70m ago 76m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:55:30.383 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 70m ago 76m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:55:30.383 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 68m ago 74m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:55:30.384 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 70m ago 74m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:55:30.384 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 70m ago 71m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:55:30.384 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 68m ago 71m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:55:30.384 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 70m ago 74m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:55:30.384 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 70m ago 73m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:55:30.385 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 70m ago 73m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:55:30.385 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 70m ago 73m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:55:30.385 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 68m ago 72m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:55:30.385 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 68m ago 72m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:55:30.386 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 68m ago 72m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:55:30.386 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 68m ago 71m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:55:30.386 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 68m ago 71m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:55:30.717 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:55:30.718 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:55:30.718 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:55:30.718 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:55:30.718 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:55:30.718 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:55:30.719 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:55:30.719 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:55:30.719 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:55:30.719 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:55:30.720 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:55:30.720 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:55:30.720 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:55:30.720 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:55:30.721 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:55:30.721 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:55:30.721 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:55:30.753 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: debug 2021-07-31T02:55:30.293+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:55:30.753 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:55:30.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:55:30.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: yield 2021-07-31T02:55:30.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:55:30.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:55:30.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:55:30.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:55:30.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:55:30.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:55:30.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:55:30.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:55:30.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:55:30.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:55:30.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:55:30.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: debug 2021-07-31T02:55:30.305+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:55:30.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:55:30.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:55:30.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: yield 2021-07-31T02:55:30.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:55:30.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:55:30.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:55:30.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:55:30.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:55:30.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:55:30.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:55:30.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:55:30.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:55:30.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:55:30.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:55:30.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: debug 2021-07-31T02:55:30.330+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:55:30.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:55:30.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:55:30.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: yield 2021-07-31T02:55:30.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:55:30.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:55:30.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:55:30.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:55:30.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:55:30.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:55:30.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:55:30.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:55:30.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:55:30.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:55:30.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:55:30.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: debug 2021-07-31T02:55:30.330+0000 7fcb9095b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:55:30.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:55:30.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:55:30.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: yield 2021-07-31T02:55:30.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:55:30.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:55:30.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:55:30.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:55:30.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:55:30.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:55:30.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:55:30.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:55:30.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:55:30.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:55:30.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:55:30.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: 2021-07-31T02:55:30.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:55:30.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: 2021-07-31T02:55:30.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:55:30.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:55:30.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:55:30.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:55:30.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:55:30.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:55:30.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:55:30.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:55:30.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:55:30.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:55:30.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:55:30.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:55:30.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:55:30.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:55:30.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:55:30.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:55:30.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:55:30.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:55:30.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: raise 2021-07-31T02:55:30.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:55:30.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:55:30.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:55:30.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:55:30.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:55:30.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: debug 2021-07-31T02:55:30.339+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:55:30.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:55:30.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:55:30.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: yield 2021-07-31T02:55:30.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:55:30.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:55:30.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:55:30.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:55:30.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:55:30.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:55:30.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:55:30.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:55:30.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:55:30.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:55:30.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:55:30.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: debug 2021-07-31T02:55:30.339+0000 7fcb9115c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:55:30.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:55:30.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:55:30.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: yield 2021-07-31T02:55:30.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:55:30.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:55:30.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:55:30.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:55:30.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:55:30.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:55:30.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:55:30.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:55:30.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:55:30.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:55:30.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:55:30.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: 2021-07-31T02:55:30.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:55:30.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: 2021-07-31T02:55:30.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:55:30.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:55:30.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:55:30.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:55:30.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:55:30.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:55:30.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:55:30.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:55:30.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:55:30.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:55:30.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:55:30.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:55:30.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:55:30.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:55:30.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:55:30.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:55:30.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:55:30.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:55:30.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: raise 2021-07-31T02:55:30.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:55:30.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:55:30.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:55:30.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:55:30.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:55:31.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: audit 2021-07-31T02:55:30.089129+0000 mgr.x (mgr.24542) 2921 : audit [DBG] from='client.28366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:55:31.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: 2021-07-31T02:55:31.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: audit 2021-07-31T02:55:30.248625+0000 mon.b (mon.2) 417 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:55:31.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: audit 2021-07-31T02:55:30.250234+0000 2021-07-31T02:55:31.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: mon.b (mon.2) 418 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:55:31.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: audit 2021 2021-07-31T02:55:31.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: -07-31T02:55:30.251144+0000 mon.b (mon.2) 419 : 2021-07-31T02:55:31.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:55:31.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: cephadm 2021-07 2021-07-31T02:55:31.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: -31T02:55:30.251905+0000 mgr.x 2021-07-31T02:55:31.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: (mgr.24542) 2922 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:55:31.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: cephadm 2021-07- 2021-07-31T02:55:31.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: 31T02:55:30.294191+0000 mgr.x (mgr.24542 2021-07-31T02:55:31.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: ) 2923 : cephadm [ERR] 2021-07-31T02:55:31.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:55:31.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:55:31.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: yield 2021-07-31T02:55:31.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:55:31.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:55:31.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:55:31.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:55:31.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:55:31.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:55:31.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:55:31.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:55:31.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:55:31.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:55:31.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:55:31.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:55:30.294800+0000 mgr.x (mgr.24542) 2924 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:55:31.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:55:30.305984+0000 mgr.x (mgr. 2021-07-31T02:55:31.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: 24542) 2925 : cephadm [ERR] 2021-07-31T02:55:31.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:55:31.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:55:31.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: yield 2021-07-31T02:55:31.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:55:31.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:55:31.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:55:31.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:55:31.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:55:31.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:55:31.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:55:31.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:55:31.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:55:31.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:55:31.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:55:31.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:55:30.306626+0000 mgr.x (mgr.24542) 2926 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:55:31.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:55 2021-07-31T02:55:31.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: :30.331227+0000 mgr.x (mgr.24542) 2927 : cephadm [ERR] 2021-07-31T02:55:31.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:55:31.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:55:31.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: yield 2021-07-31T02:55:31.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:55:31.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:55:31.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:55:31.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:55:31.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:55:31.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:55:31.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:55:31.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:55:31.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:55:31.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:55:31.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:55:31.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: 2021-07-31T02:55:31.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:55:30.331700+0000 mgr.x (mgr.24542) 2928 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:55:31.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:55:31.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:55:31.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: yield 2021-07-31T02:55:31.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:55:31.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:55:31.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:55:31.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:55:31.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:55:31.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:55:31.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:55:31.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:55:31.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:55:31.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:55:31.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:55:31.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: 2021-07-31T02:55:31.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:55:31.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: 2021-07-31T02:55:31.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:55:31.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:55:31.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:55:31.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:55:31.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:55:31.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:55:31.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:55:31.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:55:31.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:55:31.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:55:31.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:55:31.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:55:31.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:55:31.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:55:31.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:55:31.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:55:31.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:55:31.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:55:31.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: raise 2021-07-31T02:55:31.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:55:31.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:55:31.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:55:31.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:55:31.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:55:31.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: cephadm 2021-07-31 2021-07-31T02:55:31.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: T02:55:30.340161+0000 mgr.x (mgr.24542) 2929 : cephadm [ERR] 2021-07-31T02:55:31.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:55:31.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:55:31.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: yield 2021-07-31T02:55:31.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:55:31.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:55:31.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:55:31.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:55:31.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:55:31.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:55:31.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:55:31.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:55:31.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:55:31.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:55:31.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:55:31.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:55:30.340711+0000 mgr.x (mgr.24542) 2930 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:55:31.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:55:31.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:55:31.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: yield 2021-07-31T02:55:31.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:55:31.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:55:31.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:55:31.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:55:31.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:55:31.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:55:31.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:55:31.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:55:31.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:55:31.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:55:31.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:55:31.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: 2021-07-31T02:55:31.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:55:31.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: 2021-07-31T02:55:31.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:55:31.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:55:31.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:55:31.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:55:31.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:55:31.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:55:31.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:55:31.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:55:31.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:55:31.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:55:31.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:55:31.285 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:55:31.285 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:55:31.285 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:55:31.285 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:55:31.285 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:55:31.285 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:55:31.286 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:55:31.286 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: raise 2021-07-31T02:55:31.286 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:55:31.286 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:55:31.287 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:55:31.287 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:55:31.287 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:55:31.287 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: audit 2021-07-31T02:55:30.374726+0000 mgr.x (mgr.24542) 2931 : audit [DBG] from='client.28372 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:55:31.287 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:30 smithi154 conmon[34553]: audit 2021-07-31T02:55:30.717366+0000 mon.c (mon.1) 159 : audit [DBG] from='client.? 172.21.15.45:0/933494058' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:55:31.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: audit 2021-07-31T02:55:30.089129+0000 mgr.x (mgr.24542) 2921 : audit [DBG] from='client.28366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:55:31.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: audit 2021-07-31T02:55:30.248625+0000 mon.b (mon.2) 417 : audit [DBG] 2021-07-31T02:55:31.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:55:31.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: audit 2021-07-31T02:55:31.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: 2021-07-31T02:55 2021-07-31T02:55:31.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: :30.250234+0000 mon.b (mon.2) 2021-07-31T02:55:31.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: 418 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:55:31.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: audit 2021 2021-07-31T02:55:31.290 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: -07-31T02:55: 2021-07-31T02:55:31.290 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: 30.251144+0000 mon.b (mon 2021-07-31T02:55:31.290 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: .2) 419 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:55:31.290 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: cephadm 2021-07 2021-07-31T02:55:31.290 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: -31T02:55:30.251905+0000 mgr.x (mgr.24542) 2922 2021-07-31T02:55:31.291 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:55:31.291 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: cephadm 2021-07-31T02: 2021-07-31T02:55:31.291 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: 55:30.294191+0000 mgr.x (mgr. 2021-07-31T02:55:31.291 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: 24542) 2923 : cephadm [ERR] 2021-07-31T02:55:31.291 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:55:31.292 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:55:31.292 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: yield 2021-07-31T02:55:31.292 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:55:31.292 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:55:31.292 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:55:31.293 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:55:31.293 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:55:31.293 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:55:31.293 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:55:31.293 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:55:31.294 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:55:31.294 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:55:31.294 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:55:31.294 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:55:30.294800+0000 mgr.x (mgr.24542) 2924 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:55:31.294 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: cephadm 2021-07- 2021-07-31T02:55:31.295 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: 31T02:55:30.305984+0000 mgr.x (mgr. 2021-07-31T02:55:31.295 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: 24542) 2925 : cephadm [ERR] 2021-07-31T02:55:31.295 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:55:31.295 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:55:31.295 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: yield 2021-07-31T02:55:31.296 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:55:31.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: audit 2021-07-31T02:55:30 2021-07-31T02:55:31.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: .089129+0000 mgr.x (mgr.24542) 2921 : audit [DBG] from='client.28366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:55:31.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: audit 2021-07-31T02:55 2021-07-31T02:55:31.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: :30.248625+0000 mon.b (mon.2) 417 2021-07-31T02:55:31.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:55:31.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: audit 2021 2021-07-31T02:55:31.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: -07-31T02:55:30.250234 2021-07-31T02:55:31.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: +0000 mon.b (mon.2) 418 : audit 2021-07-31T02:55:31.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:55:31.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: audit 2021- 2021-07-31T02:55:31.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: 07-31T02:55:30.251144+0000 2021-07-31T02:55:31.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: mon.b (mon.2) 419 : audit [INF] 2021-07-31T02:55:31.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:55:31.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: cephadm 2021- 2021-07-31T02:55:31.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: 07-31T02:55:30.251905 2021-07-31T02:55:31.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 2922 : cephadm 2021-07-31T02:55:31.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:55:31.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: cephadm 2021-07 2021-07-31T02:55:31.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: -31T02:55:30.294191+0000 2021-07-31T02:55:31.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: mgr.x (mgr.24542) 2923 : cephadm 2021-07-31T02:55:31.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: [ERR] 2021-07-31T02:55:31.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:55:31.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:55:31.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: yield 2021-07-31T02:55:31.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:55:31.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:55:31.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:55:31.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:55:31.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:55:31.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:55:31.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:55:31.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:55:31.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:55:31.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:55:31.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:55:31.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:55:30.294800+0000 mgr.x (mgr.24542) 2924 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:55:31.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:55:30.305984+0000 mgr.x 2021-07-31T02:55:31.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: (mgr.24542) 2925 : cephadm [ERR] 2021-07-31T02:55:31.304 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:55:31.304 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:55:31.304 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: yield 2021-07-31T02:55:31.304 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:55:31.305 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:55:31.305 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:55:31.305 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:55:31.305 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:55:31.305 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:55:31.306 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:55:31.306 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:55:31.306 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:55:31.306 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:55:31.306 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:55:31.307 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:55:30.306626+0000 mgr.x (mgr.24542) 2926 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:55:31.307 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: cephadm 2021-07 2021-07-31T02:55:31.307 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: -31T02:55:30.331227+0000 mgr.x ( 2021-07-31T02:55:31.307 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: mgr.24542) 2927 : cephadm [ERR] 2021-07-31T02:55:31.307 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:55:31.308 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:55:31.308 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: yield 2021-07-31T02:55:31.308 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:55:31.308 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:55:31.308 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:55:31.309 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:55:31.309 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:55:31.309 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:55:31.309 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:55:31.310 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:55:31.310 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:55:31.310 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:55:31.310 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:55:31.310 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: 2021-07-31T02:55:31.311 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:55:30.331700+0000 mgr.x (mgr.24542) 2928 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:55:31.311 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:55:31.311 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:55:31.311 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: yield 2021-07-31T02:55:31.311 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:55:31.312 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:55:31.312 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:55:31.312 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:55:31.312 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:55:31.312 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:55:31.313 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:55:31.313 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:55:31.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:55:31.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:55:31.314 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:55:31.314 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:55:31.314 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:55:31.314 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:55:31.315 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:55:31.315 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:55:31.315 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:55:31.315 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:55:31.315 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:55:30.306626+0000 mgr.x (mgr.24542) 2926 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:55:31.316 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:55 2021-07-31T02:55:31.316 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: :30.331227+0000 mgr.x (mgr.24542) 2927 : cephadm [ERR] 2021-07-31T02:55:31.316 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:55:31.316 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:55:31.316 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: yield 2021-07-31T02:55:31.317 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:55:31.317 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:55:31.317 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:55:31.317 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:55:31.318 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:55:31.318 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:55:31.318 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:55:31.318 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:55:31.319 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:55:31.319 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:55:31.319 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:55:31.319 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:55:30.331700+0000 mgr.x (mgr.24542) 2928 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:55:31.319 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:55:31.320 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:55:31.320 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: yield 2021-07-31T02:55:31.320 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:55:31.320 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:55:31.320 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:55:31.321 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:55:31.321 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:55:31.321 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:55:31.321 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:55:31.321 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:55:31.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:55:31.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:55:31.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:55:31.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: 2021-07-31T02:55:31.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:55:31.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: 2021-07-31T02:55:31.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:55:31.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:55:31.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:55:31.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:55:31.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:55:31.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:55:31.325 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:55:31.325 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:55:31.325 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:55:31.325 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:55:31.325 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:55:31.326 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:55:31.326 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:55:31.326 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:55:31.326 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:55:31.326 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:55:31.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:55:31.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:55:31.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: raise 2021-07-31T02:55:31.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:55:31.328 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:55:31.328 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:55:31.328 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:55:31.328 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:55:31.328 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: cephadm 2021-07-31T 2021-07-31T02:55:31.329 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: 02:55:30.340161+0000 mgr.x (mgr.24542) 2929 : cephadm [ERR] 2021-07-31T02:55:31.329 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:55:31.329 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:55:31.329 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: yield 2021-07-31T02:55:31.329 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:55:31.330 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:55:31.330 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:55:31.330 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:55:31.330 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:55:31.330 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:55:31.331 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:55:31.331 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:55:31.331 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:55:31.331 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: 2021-07-31T02:55:31.332 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:55:31.332 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: 2021-07-31T02:55:31.332 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:55:31.332 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:55:31.333 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:55:31.333 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:55:31.333 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:55:31.333 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:55:31.333 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:55:31.334 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:55:31.334 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:55:31.334 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:55:31.334 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:55:31.334 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:55:31.335 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:55:31.335 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:55:31.335 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:55:31.335 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:55:31.335 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:55:31.336 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:55:31.336 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: raise 2021-07-31T02:55:31.336 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:55:31.336 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:55:31.337 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:55:31.337 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:55:31.337 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:55:31.337 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: cephadm 2021-07-31 2021-07-31T02:55:31.337 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: T02:55:30.340161+0000 mgr.x (mgr.24542) 2929 : cephadm [ERR] 2021-07-31T02:55:31.338 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:55:31.338 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:55:31.338 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: yield 2021-07-31T02:55:31.338 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:55:31.338 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:55:31.339 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:55:31.339 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:55:31.339 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:55:31.339 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:55:31.340 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:55:31.340 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:55:31.340 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:55:31.340 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:55:31.341 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:55:31.341 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:55:30.340711+0000 mgr.x (mgr.24542) 2930 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:55:31.341 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:55:31.342 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:55:31.342 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: yield 2021-07-31T02:55:31.342 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:55:31.342 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:55:31.342 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:55:31.343 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:55:31.343 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:55:31.343 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:55:31.343 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:55:31.343 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:55:31.344 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:55:31.344 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:55:31.344 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:55:31.344 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: 2021-07-31T02:55:31.344 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:55:31.345 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: 2021-07-31T02:55:31.345 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:55:31.345 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:55:31.345 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:55:31.345 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:55:31.346 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:55:31.346 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:55:31.346 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:55:31.346 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:55:31.346 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:55:31.347 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:55:31.347 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:55:31.347 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:55:31.347 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:55:31.347 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:55:31.348 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:55:31.348 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:55:31.348 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:55:31.348 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:55:31.349 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: raise 2021-07-31T02:55:31.349 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:55:31.349 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:55:31.349 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:55:31.349 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:55:31.350 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:55:31.350 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: audit 2021-07-31T02:55:30.374726+0000 mgr.x (mgr.24542) 2931 : audit [DBG] from='client.28372 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:55:31.350 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: audit 2021 2021-07-31T02:55:31.350 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[33655]: -07-31T02:55:30.717366+0000 mon.c (mon.1) 159 : audit [DBG] from='client.? 172.21.15.45:0/933494058' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:55:31.351 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:55:31.351 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:55:31.351 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:55:31.352 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:55:31.352 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:55:31.352 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:55:30.340711+0000 mgr.x (mgr.24542) 2930 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:55:31.352 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:55:31.352 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:55:31.353 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: yield 2021-07-31T02:55:31.353 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:55:31.353 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:55:31.353 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:55:31.354 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:55:31.354 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:55:31.354 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:55:31.354 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:55:31.354 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:55:31.355 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:55:31.355 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:55:31.355 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:55:31.355 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: 2021-07-31T02:55:31.356 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:55:31.356 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: 2021-07-31T02:55:31.356 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:55:31.356 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:55:31.356 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:55:31.356 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:55:31.357 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:55:31.357 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:55:31.357 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:55:31.357 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:55:31.358 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:55:31.358 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:55:31.358 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:55:31.358 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:55:31.358 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:55:31.359 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:55:31.359 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:55:31.359 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:55:31.359 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:55:31.359 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:55:31.360 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: raise 2021-07-31T02:55:31.360 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:55:31.360 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:55:31.360 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:55:31.360 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:55:31.361 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:55:31.361 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: audit 2021-07-31T02:55:30.374726+0000 mgr.x (mgr.24542) 2931 : audit [DBG] from='client.28372 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:55:31.361 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:30 smithi045 conmon[39718]: audit 2021-07-31T02:55:30.717366+0000 mon.c (mon.1) 159 : audit [DBG] from='client.? 172.21.15.45:0/933494058' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:55:32.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:31 smithi154 conmon[34553]: cluster 2021-07-31T02:55:30.777097+0000 mgr.x (mgr.24542) 2932 : cluster [DBG] 2021-07-31T02:55:32.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:31 smithi154 conmon[34553]: pgmap v4056: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:55:32.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:31 smithi045 conmon[33655]: cluster 2021-07-31T02:55:30.777097+0000 mgr.x (mgr.24542 2021-07-31T02:55:32.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:31 smithi045 conmon[33655]: ) 2932 : cluster [DBG] pgmap v4056: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:55:32.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:31 smithi045 conmon[39718]: cluster 2021-07-31T02:55:30.777097+0000 2021-07-31T02:55:32.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:31 smithi045 conmon[39718]: mgr.x (mgr.24542) 2932 : cluster [DBG] pgmap v4056: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:55:34.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:33 smithi154 conmon[34553]: cluster 2021-07-31T02:55:32.777475+0000 mgr.x (mgr.24542) 2933 : cluster [DBG] pgmap v4057: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:55:34.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:33 smithi045 conmon[33655]: cluster 2021-07-31T02:55:32.777475+0000 mgr.x (mgr.24542) 2933 : 2021-07-31T02:55:34.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:33 smithi045 conmon[33655]: cluster [DBG] pgmap v4057: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:55:34.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:33 smithi045 conmon[39718]: cluster 2021-07-31T02:55:32.777475+0000 mgr.x (mgr. 2021-07-31T02:55:34.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:33 smithi045 conmon[39718]: 24542) 2933 : cluster [DBG] pgmap v4057: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:55:36.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:35 smithi154 conmon[34553]: cluster 2021-07-31T02:55:34.778026+0000 mgr.x (mgr.24542) 2934 : cluster [DBG] pgmap v4058: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:55:36.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:35 smithi045 conmon[33655]: cluster 2021-07-31T02:55:34.778026+0000 mgr.x (mgr.24542) 2021-07-31T02:55:36.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:35 smithi045 conmon[33655]: 2934 : cluster [DBG] pgmap v4058: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:55:36.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:35 smithi045 conmon[39718]: cluster 2021-07-31T02:55:34.778026+0000 mgr.x (mgr. 2021-07-31T02:55:36.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:35 smithi045 conmon[39718]: 24542) 2934 : cluster [DBG] pgmap v4058: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:55:38.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:37 smithi154 conmon[34553]: cluster 2021-07-31T02:55:36.778492+0000 mgr.x (mgr.24542) 2935 : cluster [DBG] pgmap v4059: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:55:38.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:37 smithi045 conmon[39718]: cluster 2021-07-31T02:55:36.778492+0000 mgr.x ( 2021-07-31T02:55:38.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:37 smithi045 conmon[39718]: mgr.24542) 2935 : cluster [DBG] pgmap v4059: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:55:38.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:37 smithi045 conmon[33655]: cluster 2021-07-31T02:55:36.778492+0000 mgr.x (mgr.24542) 2935 : cluster [DBG] pgmap v4059: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:55:40.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:39 smithi154 conmon[34553]: cluster 2021-07-31T02:55:38.779055+0000 mgr.x (mgr.24542) 2936 : cluster [DBG] pgmap v4060: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:55:40.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:39 smithi154 conmon[34553]: 2021-07-31T02:55:40.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:39 smithi045 conmon[33655]: cluster 2021-07-31T02:55:38.779055+0000 mgr.x (mgr.24542) 2936 : cluster [DBG] pgmap v4060: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:55:40.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:39 smithi045 conmon[39718]: cluster 2021-07-31T02:55:38.779055 2021-07-31T02:55:40.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:39 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 2936 : cluster [DBG] pgmap v4060: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:55:42.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:41 smithi154 conmon[34553]: cluster 2021-07-31T02:55:40.779558+0000 mgr.x (mgr.24542) 2937 : cluster [DBG] 2021-07-31T02:55:42.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:41 smithi154 conmon[34553]: pgmap v4061: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:55:42.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:41 smithi045 conmon[33655]: cluster 2021-07-31T02:55:40.779558+0000 mgr.x (mgr 2021-07-31T02:55:42.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:41 smithi045 conmon[33655]: .24542) 2937 : cluster [DBG] pgmap v4061: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:55:42.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:41 smithi045 conmon[39718]: cluster 2021-07-31T02:55:40. 2021-07-31T02:55:42.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:41 smithi045 conmon[39718]: 779558+0000 mgr.x (mgr.24542) 2937 : cluster [DBG] pgmap v4061: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:55:44.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:43 smithi154 conmon[34553]: cluster 2021-07-31T02:55:42.779978+0000 mgr.x (mgr.24542) 2938 : cluster [DBG] 2021-07-31T02:55:44.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:43 smithi154 conmon[34553]: pgmap v4062: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:55:44.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:43 smithi045 conmon[33655]: cluster 2021-07-31T02:55:42.779978+0000 mgr.x (mgr.24542 2021-07-31T02:55:44.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:43 smithi045 conmon[33655]: ) 2938 : cluster [DBG] pgmap v4062: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:55:44.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:43 smithi045 conmon[39718]: cluster 2021-07-31T02:55:42.779978+0000 mgr.x (mgr. 2021-07-31T02:55:44.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:43 smithi045 conmon[39718]: 24542) 2938 : cluster [DBG] pgmap v4062: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:55:46.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:45 smithi154 conmon[34553]: cluster 2021-07-31T02:55:44.780423+0000 mgr.x (mgr.24542) 2939 : cluster [DBG] pgmap v4063: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:55:46.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:45 smithi045 conmon[39718]: cluster 2021-07-31T02:55:44.780423+0000 2021-07-31T02:55:46.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:45 smithi045 conmon[39718]: mgr.x (mgr.24542) 2939 : cluster [DBG] pgmap v4063: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:55:46.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:45 smithi045 conmon[33655]: cluster 2021-07-31T02:55:44.780423+0000 mgr.x (mgr 2021-07-31T02:55:46.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:45 smithi045 conmon[33655]: .24542) 2939 : cluster [DBG] pgmap v4063: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:55:48.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:47 smithi154 conmon[34553]: cluster 2021-07-31T02:55:46.780803+0000 mgr.x (mgr.24542) 2940 : cluster 2021-07-31T02:55:48.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:47 smithi154 conmon[34553]: [DBG] pgmap v4064: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:55:48.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:47 smithi045 conmon[39718]: cluster 2021-07-31T02:55: 2021-07-31T02:55:48.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:47 smithi045 conmon[39718]: 46.780803+0000 mgr.x (mgr.24542) 2940 : cluster [DBG] pgmap v4064: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:55:48.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:47 smithi045 conmon[33655]: cluster 2021-07-31T02:55:46.780803+0000 mgr.x (mgr.24542) 2940 : cluster 2021-07-31T02:55:48.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:47 smithi045 conmon[33655]: [DBG] pgmap v4064: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:55:50.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:49 smithi154 conmon[34553]: cluster 2021-07-31T02:55:48.781186+0000 mgr.x (mgr.24542) 2941 : cluster [DBG] pgmap v4065: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:55:50.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:49 smithi045 conmon[39718]: cluster 2021-07-31T02:55:48.781186 2021-07-31T02:55:50.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:49 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 2941 : cluster [DBG] pgmap v4065: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:55:50.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:49 smithi045 conmon[33655]: cluster 2021-07-31T02:55:48.781186+0000 mgr.x (mgr.24542) 2941 : cluster 2021-07-31T02:55:50.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:49 smithi045 conmon[33655]: [DBG] pgmap v4065: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:55:52.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:51 smithi154 conmon[34553]: cluster 2021-07-31T02:55:50.781767+0000 mgr.x (mgr.24542) 2942 : cluster [DBG] pgmap v4066: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:55:52.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:51 smithi045 conmon[39718]: cluster 2021-07-31T02:55: 2021-07-31T02:55:52.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:51 smithi045 conmon[39718]: 50.781767+0000 mgr.x (mgr.24542) 2942 : cluster [DBG] pgmap v4066: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:55:52.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:51 smithi045 conmon[33655]: cluster 2021-07-31T02:55:50.781767+0000 mgr.x (mgr.24542) 2942 : cluster [DBG] pgmap v4066: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:55:52.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:51 smithi045 conmon[33655]: 2021-07-31T02:55:54.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:53 smithi154 conmon[34553]: cluster 2021-07-31T02:55:52.782115+0000 mgr.x (mgr.24542) 2943 : cluster [DBG] pgmap v4067: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:55:54.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:53 smithi045 conmon[39718]: cluster 2021-07-31T02:55:52.782115+0000 mgr.x ( 2021-07-31T02:55:54.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:53 smithi045 conmon[39718]: mgr.24542) 2943 : cluster [DBG] pgmap v4067: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:55:54.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:53 smithi045 conmon[33655]: cluster 2021-07-31T02:55:52.782115+0000 mgr.x (mgr.24542) 2943 : cluster [DBG] 2021-07-31T02:55:54.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:53 smithi045 conmon[33655]: pgmap v4067: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:55:56.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:55 smithi154 conmon[34553]: cluster 2021-07-31T02:55:54.782530+0000 mgr.x (mgr.24542) 2944 : cluster [DBG] pgmap v4068: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:55:56.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:55 smithi045 conmon[39718]: cluster 2021-07-31T02:55:54.782530+0000 mgr.x (mgr.24542 2021-07-31T02:55:56.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:55 smithi045 conmon[39718]: ) 2944 : cluster [DBG] pgmap v4068: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:55:56.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:55 smithi045 conmon[33655]: cluster 2021-07-31T02:55:54.782530+0000 mgr.x (mgr.24542) 2944 2021-07-31T02:55:56.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:55 smithi045 conmon[33655]: : cluster [DBG] pgmap v4068: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:55:58.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:57 smithi154 conmon[34553]: cluster 2021-07-31T02:55:56.783074+0000 mgr.x (mgr.24542) 2945 : cluster [DBG] 2021-07-31T02:55:58.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:57 smithi154 conmon[34553]: pgmap v4069: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:55:58.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:57 smithi045 conmon[39718]: cluster 2021-07-31T02:55:56.783074+0000 mgr.x (mgr.24542) 2945 : cluster [DBG] pgmap v4069: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:55:58.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:57 smithi045 conmon[33655]: cluster 2021-07-31T02:55:56.783074+0000 mgr.x (mgr.24542) 2945 : 2021-07-31T02:55:58.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:57 smithi045 conmon[33655]: cluster [DBG] pgmap v4069: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:00.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:55:59 smithi154 conmon[34553]: cluster 2021-07-31T02:55:58.783491+0000 mgr.x (mgr.24542) 2946 : cluster [DBG] pgmap v4070: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:00.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:59 smithi045 conmon[39718]: cluster 2021-07-31T02:55:58.783491+0000 mgr.x (mgr 2021-07-31T02:56:00.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:55:59 smithi045 conmon[39718]: .24542) 2946 : cluster [DBG] pgmap v4070: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:00.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:59 smithi045 conmon[33655]: cluster 2021-07-31T02:55:58.783491+0000 mgr.x (mgr.24542) 2021-07-31T02:56:00.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:55:59 smithi045 conmon[33655]: 2946 : cluster [DBG] pgmap v4070: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:01.032 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:56:01.309 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:56:01.310 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 70m ago 71m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:56:01.310 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 68m ago 71m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:56:01.310 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 68m ago 74m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:56:01.310 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 70m ago 77m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:56:01.311 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 70m ago 77m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:56:01.311 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 68m ago 75m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:56:01.311 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 70m ago 75m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:56:01.311 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 70m ago 71m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:56:01.311 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 68m ago 71m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:56:01.312 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 70m ago 74m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:56:01.312 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 70m ago 74m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:56:01.312 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 70m ago 73m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:56:01.312 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 70m ago 73m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:56:01.313 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 68m ago 73m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:56:01.313 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 68m ago 72m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:56:01.313 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 68m ago 72m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:56:01.314 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 68m ago 72m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:56:01.314 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 68m ago 71m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:56:01.647 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:56:01.648 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:56:01.648 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:56:01.648 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:56:01.648 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:56:01.649 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:56:01.649 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:56:01.649 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:56:01.649 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:56:01.650 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:56:01.650 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:56:01.650 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:56:01.650 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:56:01.650 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:56:01.651 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:56:01.651 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:56:01.651 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:56:02.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:01 smithi154 conmon[34553]: cluster 2021-07-31T02:56:00.784040+0000 mgr.x (mgr.24542) 2947 : cluster [DBG] pgmap v4071: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:02.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:01 smithi154 conmon[34553]: audit 2021-07-31T02:56:01.020661 2021-07-31T02:56:02.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:01 smithi154 conmon[34553]: +0000 mgr.x (mgr.24542) 2948 : audit [DBG] from='client.28384 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:56:02.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:01 smithi154 conmon[34553]: audit 2021-07-31T02:56:01.301897+0000 mgr.x (mgr.24542) 2949 : audit [DBG] from='client.18450 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:56:02.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:01 smithi154 conmon[34553]: audit 2021-07-31T02:56:01.647441 2021-07-31T02:56:02.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:01 smithi154 conmon[34553]: +0000 mon.c (mon.1) 160 : audit [DBG] from='client.? 172.21.15.45:0/3481474674' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:56:02.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:01 smithi045 conmon[39718]: cluster 2021-07-31T02:56:00.784040+0000 mgr.x (mgr.24542) 2947 : cluster [DBG] pgmap v4071: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:02.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:01 smithi045 conmon[39718]: audit 2021-07-31T02:56:01.020661+0000 mgr.x (mgr.24542) 2948 : audit [DBG] from='client.28384 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:56:02.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:01 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T02:56:02.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:01 smithi045 conmon[39718]: T02:56:01.301897+0000 mgr.x ( 2021-07-31T02:56:02.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:01 smithi045 conmon[39718]: mgr.24542) 2949 : audit [DBG] from='client.18450 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:56:02.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:01 smithi045 conmon[39718]: audit 2021- 2021-07-31T02:56:02.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:01 smithi045 conmon[39718]: 07-31T02:56:01. 2021-07-31T02:56:02.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:01 smithi045 conmon[39718]: 647441+0000 mon.c (mon.1 2021-07-31T02:56:02.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:01 smithi045 conmon[39718]: ) 160 : audit [DBG] from='client.? 172.21.15.45:0/3481474674' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:56:02.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:01 smithi045 conmon[39718]: 2021-07-31T02:56:02.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:01 smithi045 conmon[33655]: cluster 2021-07-31T02:56:00.784040+0000 mgr.x (mgr.24542) 2947 : cluster [DBG] pgmap v4071: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:02.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:01 smithi045 conmon[33655]: 2021-07-31T02:56:02.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:01 smithi045 conmon[33655]: audit 2021-07-31T02:56:01.020661+0000 mgr.x (mgr.24542) 2948 : audit [DBG] from='client.28384 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:56:02.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:01 smithi045 conmon[33655]: audit 2021-07-31T02:56:01.301897+0000 mgr.x (mgr.24542) 2949 2021-07-31T02:56:02.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:01 smithi045 conmon[33655]: : audit [DBG] from='client.18450 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:56:02.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:01 smithi045 conmon[33655]: audit 2021-07-31T02:56:02.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:01 smithi045 conmon[33655]: 2021-07-31T02 2021-07-31T02:56:02.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:01 smithi045 conmon[33655]: :56:01.647441+0000 mon.c (mon. 2021-07-31T02:56:02.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:01 smithi045 conmon[33655]: 1) 160 : audit [DBG] from='client.? 172.21.15.45:0/3481474674' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:56:04.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:03 smithi154 conmon[34553]: cluster 2021-07-31T02:56:02.784401+0000 mgr.x (mgr.24542) 2950 : cluster [DBG] pgmap v4072: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:04.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:03 smithi045 conmon[39718]: cluster 2021-07-31T02:56:02.784401+0000 mgr.x ( 2021-07-31T02:56:04.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:03 smithi045 conmon[39718]: mgr.24542) 2950 : cluster [DBG] pgmap v4072: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:04.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:03 smithi045 conmon[33655]: cluster 2021-07-31T02:56:02.784401+0000 mgr.x (mgr.24542) 2950 : cluster [DBG] pgmap v4072: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:04.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:03 smithi045 conmon[33655]: 2021-07-31T02:56:06.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:05 smithi154 conmon[34553]: cluster 2021-07-31T02:56:04.784842+0000 mgr.x (mgr.24542) 2951 : cluster [DBG] pgmap v4073: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:06.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:05 smithi045 conmon[33655]: cluster 2021-07-31T02:56:04.784842+0000 mgr.x (mgr.24542) 2951 2021-07-31T02:56:06.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:05 smithi045 conmon[33655]: : cluster [DBG] pgmap v4073: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:06.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:05 smithi045 conmon[39718]: cluster 2021-07-31T02:56:04.784842+0000 mgr.x (mgr.24542) 2951 : cluster [DBG] 2021-07-31T02:56:06.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:05 smithi045 conmon[39718]: pgmap v4073: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:08.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:07 smithi154 conmon[34553]: cluster 2021-07-31T02:56:06.785336+0000 mgr.x (mgr.24542) 2952 : cluster [DBG] pgmap v4074: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:08.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:07 smithi045 conmon[39718]: cluster 2021-07-31T02:56:06.785336+0000 mgr.x (mgr 2021-07-31T02:56:08.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:07 smithi045 conmon[39718]: .24542) 2952 : cluster [DBG] pgmap v4074: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:08.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:07 smithi045 conmon[33655]: cluster 2021-07-31T02:56:06.785336+0000 mgr.x (mgr. 2021-07-31T02:56:08.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:07 smithi045 conmon[33655]: 24542) 2952 : cluster [DBG] pgmap v4074: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:10.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:09 smithi154 conmon[34553]: cluster 2021-07-31T02:56:08.785809+0000 mgr.x (mgr.24542) 2953 : cluster [DBG] pgmap v4075: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:10.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:09 smithi045 conmon[39718]: cluster 2021-07-31T02:56:08.785809+0000 mgr.x (mgr.24542) 2953 : cluster [DBG] pgmap v4075: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:10.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:09 smithi045 conmon[33655]: cluster 2021-07-31T02:56:08.785809+0000 mgr.x (mgr.24542) 2953 2021-07-31T02:56:10.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:09 smithi045 conmon[33655]: : cluster [DBG] pgmap v4075: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:12.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:11 smithi154 conmon[34553]: cluster 2021-07-31T02:56:10.786394+0000 mgr.x (mgr.24542) 2954 : cluster [DBG] pgmap v4076: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:12.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:11 smithi045 conmon[39718]: cluster 2021-07-31T02:56:10.786394+0000 mgr.x (mgr.24542) 2954 : cluster 2021-07-31T02:56:12.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:11 smithi045 conmon[39718]: [DBG] pgmap v4076: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:12.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:11 smithi045 conmon[33655]: cluster 2021-07-31T02:56:10.786394 2021-07-31T02:56:12.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:11 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 2954 : cluster [DBG] pgmap v4076: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:14.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:13 smithi154 conmon[34553]: cluster 2021-07-31T02:56:12.786747+0000 mgr.x (mgr.24542) 2955 : cluster [DBG] pgmap v4077: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:14.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:13 smithi045 conmon[33655]: cluster 2021-07-31T02:56:12.786747+0000 mgr.x (mgr.24542) 2955 : cluster [DBG] pgmap v4077: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:14.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:13 smithi045 conmon[39718]: cluster 2021-07-31T02:56:12.786747+0000 mgr.x (mgr.24542) 2955 : cluster 2021-07-31T02:56:14.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:13 smithi045 conmon[39718]: [DBG] pgmap v4077: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:16.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:15 smithi154 conmon[34553]: cluster 2021-07-31T02:56:14.787167+0000 mgr.x (mgr.24542) 2956 : cluster [DBG] pgmap v4078: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:16.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:15 smithi154 conmon[34553]: 2021-07-31T02:56:16.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:15 smithi045 conmon[33655]: cluster 2021-07-31T02:56:14.787167+0000 mgr.x (mgr 2021-07-31T02:56:16.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:15 smithi045 conmon[33655]: .24542) 2956 : cluster [DBG] pgmap v4078: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:16.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:15 smithi045 conmon[39718]: cluster 2021-07-31T02:56:14.787167+0000 mgr.x (mgr.24542) 2956 : cluster [DBG] 2021-07-31T02:56:16.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:15 smithi045 conmon[39718]: pgmap v4078: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:18.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:17 smithi154 conmon[34553]: cluster 2021-07-31T02:56:16.787587+0000 mgr.x (mgr.24542) 2957 : cluster [DBG] pgmap v4079: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:18.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:17 smithi154 conmon[34553]: 2021-07-31T02:56:18.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:17 smithi045 conmon[33655]: cluster 2021-07-31T02:56:16.787587+0000 mgr.x (mgr.24542) 2957 : cluster [DBG] pgmap v4079: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:18.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:17 smithi045 conmon[39718]: cluster 2021-07-31T02:56:16.787587+0000 mgr.x (mgr.24542) 2957 : cluster 2021-07-31T02:56:18.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:17 smithi045 conmon[39718]: [DBG] pgmap v4079: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:20.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:19 smithi154 conmon[34553]: cluster 2021-07-31T02:56:18.788142+0000 mgr.x (mgr.24542) 2958 : cluster [DBG] pgmap v4080: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:20.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:19 smithi154 conmon[34553]: 2021-07-31T02:56:20.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:19 smithi045 conmon[39718]: cluster 2021-07-31T02:56:18.788142+0000 mgr.x (mgr.24542) 2958 : cluster [DBG] 2021-07-31T02:56:20.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:19 smithi045 conmon[39718]: pgmap v4080: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:20.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:19 smithi045 conmon[33655]: cluster 2021-07-31T02:56:18. 2021-07-31T02:56:20.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:19 smithi045 conmon[33655]: 788142+0000 mgr.x (mgr.24542) 2958 : cluster [DBG] pgmap v4080: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:22.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:21 smithi154 conmon[34553]: cluster 2021-07-31T02:56:20.788581+0000 mgr.x (mgr.24542) 2959 : cluster [DBG] pgmap v4081: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:22.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:21 smithi154 conmon[34553]: 2021-07-31T02:56:22.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:21 smithi045 conmon[33655]: cluster 2021-07-31T02:56:20.788581+0000 mgr.x (mgr.24542 2021-07-31T02:56:22.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:21 smithi045 conmon[33655]: ) 2959 : cluster [DBG] pgmap v4081: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:22.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:21 smithi045 conmon[39718]: cluster 2021-07-31T02:56:20.788581+0000 mgr.x (mgr 2021-07-31T02:56:22.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:21 smithi045 conmon[39718]: .24542) 2959 : cluster [DBG] pgmap v4081: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:24.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:23 smithi154 conmon[34553]: cluster 2021-07-31T02:56:22.788907+0000 mgr.x (mgr.24542) 2960 : cluster [DBG] pgmap v4082: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:24.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:23 smithi154 conmon[34553]: 2021-07-31T02:56:24.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:23 smithi045 conmon[33655]: cluster 2021-07-31T02:56:22.788907+0000 mgr.x (mgr.24542) 2960 : cluster 2021-07-31T02:56:24.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:23 smithi045 conmon[33655]: [DBG] pgmap v4082: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:24.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:23 smithi045 conmon[39718]: cluster 2021-07-31T02:56:22.788907+0000 mgr.x ( 2021-07-31T02:56:24.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:23 smithi045 conmon[39718]: mgr.24542) 2960 : cluster [DBG] pgmap v4082: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:26.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:25 smithi154 conmon[34553]: cluster 2021-07-31T02:56:24.789433+0000 mgr.x (mgr.24542) 2961 : cluster [DBG] pgmap v4083: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:26.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:25 smithi045 conmon[33655]: cluster 2021-07-31T02:56:24.789433+0000 mgr.x (mgr. 2021-07-31T02:56:26.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:25 smithi045 conmon[33655]: 24542) 2961 : cluster [DBG] pgmap v4083: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:26.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:25 smithi045 conmon[39718]: cluster 2021-07-31T02:56:24.789433+0000 mgr.x (mgr 2021-07-31T02:56:26.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:25 smithi045 conmon[39718]: .24542) 2961 : cluster [DBG] pgmap v4083: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:28.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:27 smithi154 conmon[34553]: cluster 2021-07-31T02:56:26.789879+0000 mgr.x (mgr.24542) 2962 : cluster [DBG] pgmap v4084: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:28.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:27 smithi045 conmon[39718]: cluster 2021-07-31T02:56:26.789879+0000 mgr.x (mgr 2021-07-31T02:56:28.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:27 smithi045 conmon[39718]: .24542) 2962 : cluster [DBG] pgmap v4084: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:28.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:27 smithi045 conmon[33655]: cluster 2021-07-31T02:56:26.789879+0000 mgr.x (mgr.24542) 2962 : cluster 2021-07-31T02:56:28.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:27 smithi045 conmon[33655]: [DBG] pgmap v4084: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:29.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:28 smithi154 conmon[34553]: audit 2021-07-31T02:56:28.212126+0000 mon.b (mon.2) 420 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:56:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:28 smithi154 conmon[34553]: audit 2021-07-31T02:56:28.212398+0000 mon.b (mon.2) 421 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:56:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:28 smithi154 conmon[34553]: audit 2021-07-31T02:56:28.212797+0000 mon.a (mon.0) 3226 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:56:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:28 smithi154 conmon[34553]: 2021-07-31T02:56:29.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:28 smithi154 conmon[34553]: audit 2021-07-31T02:56:28.213129+0000 mon.a (mon.0) 3227 2021-07-31T02:56:29.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:28 smithi154 conmon[34553]: : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:56:29.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:28 smithi045 conmon[39718]: audit 2021-07-31T02:56:28.212126+0000 mon.b 2021-07-31T02:56:29.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:28 smithi045 conmon[39718]: (mon.2) 420 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:56:29.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:28 smithi045 conmon[39718]: audit 2021-07-31T02:56:28 2021-07-31T02:56:29.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:28 smithi045 conmon[39718]: .212398+0000 mon.b (mon.2) 421 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:56:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:28 smithi045 conmon[39718]: audit 2021- 2021-07-31T02:56:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:28 smithi045 conmon[39718]: 07-31T02:56:28.212797+0000 mon.a (mon.0) 3226 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:56:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:28 smithi045 conmon[39718]: audit 2021-07-31T02:56:28 2021-07-31T02:56:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:28 smithi045 conmon[39718]: .213129+0000 mon.a (mon.0) 3227 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:56:29.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:28 smithi045 conmon[33655]: audit 2021-07-31T02:56:28.212126+0000 mon.b (mon.2) 420 : 2021-07-31T02:56:29.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:28 smithi045 conmon[33655]: audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:56:29.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:28 smithi045 conmon[33655]: audit 2021-07-31T02:56:28.212398+0000 mon.b (mon.2) 421 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:56:29.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:28 smithi045 conmon[33655]: audit 2021-07-31T02:56:28.212797+0000 mon.a (mon.0) 3226 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:56:29.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:28 smithi045 conmon[33655]: 2021-07-31T02:56:29.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:28 smithi045 conmon[33655]: audit 2021-07-31T02:56:28.213129+0000 mon.a (mon 2021-07-31T02:56:29.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:28 smithi045 conmon[33655]: .0) 3227 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:56:30.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:29 smithi154 conmon[34553]: cluster 2021-07-31T02:56:28.790325+0000 mgr.x (mgr.24542) 2963 : cluster [DBG] pgmap v4085: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:30.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:29 smithi154 conmon[34553]: 2021-07-31T02:56:30.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:29 smithi045 conmon[33655]: cluster 2021-07-31T02:56:28.790325+0000 mgr.x (mgr.24542) 2963 2021-07-31T02:56:30.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:29 smithi045 conmon[33655]: : cluster [DBG] pgmap v4085: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:30.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:29 smithi045 conmon[39718]: cluster 2021-07-31T02:56:28.790325+0000 mgr.x 2021-07-31T02:56:30.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:29 smithi045 conmon[39718]: (mgr.24542) 2963 : cluster [DBG] pgmap v4085: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:30.753 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: debug 2021-07-31T02:56:30.386+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:56:30.753 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:56:30.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:56:30.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: yield 2021-07-31T02:56:30.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:56:30.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:56:30.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:56:30.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:56:30.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:56:30.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:56:30.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:56:30.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:56:30.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:56:30.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:56:30.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:56:30.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: debug 2021-07-31T02:56:30.398+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:56:30.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:56:30.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:56:30.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: yield 2021-07-31T02:56:30.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:56:30.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:56:30.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:56:30.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:56:30.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:56:30.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:56:30.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:56:30.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:56:30.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:56:30.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:56:30.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:56:30.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: debug 2021-07-31T02:56:30.424+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:56:30.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:56:30.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:56:30.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: yield 2021-07-31T02:56:30.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:56:30.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:56:30.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:56:30.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:56:30.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:56:30.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:56:30.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:56:30.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:56:30.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:56:30.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:56:30.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:56:30.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: debug 2021-07-31T02:56:30.424+0000 7fcb9816a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:56:30.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:56:30.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:56:30.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: yield 2021-07-31T02:56:30.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:56:30.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:56:30.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:56:30.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:56:30.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:56:30.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:56:30.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:56:30.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:56:30.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:56:30.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:56:30.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:56:30.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: 2021-07-31T02:56:30.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:56:30.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: 2021-07-31T02:56:30.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:56:30.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:56:30.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:56:30.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:56:30.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:56:30.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:56:30.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:56:30.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:56:30.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:56:30.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:56:30.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:56:30.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:56:30.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:56:30.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:56:30.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:56:30.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:56:30.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:56:30.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:56:30.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: raise 2021-07-31T02:56:30.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:56:30.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:56:30.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:56:30.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:56:30.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:56:30.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: debug 2021-07-31T02:56:30.432+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:56:30.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:56:30.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:56:30.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: yield 2021-07-31T02:56:30.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:56:30.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:56:30.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:56:30.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:56:30.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:56:30.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:56:30.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:56:30.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:56:30.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:56:30.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:56:30.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:56:30.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: debug 2021-07-31T02:56:30.432+0000 7fcb97969700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:56:30.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:56:30.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:56:30.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: yield 2021-07-31T02:56:30.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:56:30.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:56:30.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:56:30.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:56:30.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:56:30.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:56:30.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:56:30.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:56:30.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:56:30.786 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:56:30.786 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:56:30.786 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: 2021-07-31T02:56:30.786 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:56:30.786 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: 2021-07-31T02:56:30.787 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:56:30.787 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:56:30.787 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:56:30.787 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:56:30.788 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:56:30.788 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:56:30.788 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:56:30.788 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:56:30.788 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:56:30.789 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:56:30.789 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:56:30.789 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:56:30.789 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:56:30.789 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:56:30.790 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:56:30.790 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:56:30.790 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:56:30.790 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:56:30.791 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: raise 2021-07-31T02:56:30.791 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:56:30.791 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:56:30.791 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:56:30.791 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:56:30.792 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:56:31.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: audit 2021-07-31T02:56:30.341596+0000 mon.b (mon.2) 422 : audit [DBG] 2021-07-31T02:56:31.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:56:31.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: audit 2021-07-31T02:56:30.343048+0000 mon.b (mon.2) 423 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:56:31.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: audit 2021-07-31T02:56:30.343905+0000 mon.b (mon.2) 2021-07-31T02:56:31.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: 424 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:56:31.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:56:30.344698+0000 mgr.x (mgr.24542) 2964 : cephadm 2021-07-31T02:56:31.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:56:31.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: cephadm 2021-07-31 2021-07-31T02:56:31.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: T02:56:30.387605+0000 mgr.x (mgr.24542) 2965 : cephadm [ERR] 2021-07-31T02:56:31.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:56:31.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:56:31.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: yield 2021-07-31T02:56:31.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:56:31.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:56:31.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:56:31.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:56:31.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:56:31.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:56:31.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:56:31.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:56:31.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:56:31.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:56:31.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:56:31.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:56:30.388058+0000 mgr.x (mgr.24542) 2966 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:56:31.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:56:30.399164+0000 mgr.x (mgr.24542) 2967 : cephadm [ERR] 2021-07-31T02:56:31.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:56:31.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:56:31.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: yield 2021-07-31T02:56:31.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:56:31.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:56:31.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:56:31.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:56:31.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:56:31.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:56:31.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:56:31.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:56:31.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:56:31.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:56:31.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:56:31.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: cephadm 2021-07- 2021-07-31T02:56:31.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: 31T02:56:30.399545+0000 mgr.x (mgr.24542) 2968 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:56:31.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:56:30.425138+0000 mgr.x (mgr.24542) 2969 : cephadm [ERR] 2021-07-31T02:56:31.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:56:31.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:56:31.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: yield 2021-07-31T02:56:31.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:56:31.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:56:31.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:56:31.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:56:31.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:56:31.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:56:31.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:56:31.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:56:31.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:56:31.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:56:31.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:56:31.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:56:30.425587+0000 mgr.x (mgr.24542) 2970 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:56:31.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:56:31.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:56:31.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: yield 2021-07-31T02:56:31.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:56:31.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:56:31.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:56:31.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:56:31.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:56:31.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:56:31.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:56:31.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:56:31.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:56:31.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:56:31.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:56:31.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: 2021-07-31T02:56:31.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:56:31.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: 2021-07-31T02:56:31.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:56:31.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:56:31.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:56:31.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:56:31.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:56:31.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:56:31.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:56:31.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:56:31.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:56:31.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:56:31.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:56:31.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:56:31.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:56:31.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:56:31.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:56:31.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:56:31.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:56:31.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:56:31.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: raise 2021-07-31T02:56:31.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:56:31.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:56:31.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:56:31.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:56:31.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:56:31.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:56:30.433514+0000 mgr.x (mgr.24542) 2971 : cephadm [ERR] 2021-07-31T02:56:31.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:56:31.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:56:31.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: yield 2021-07-31T02:56:31.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:56:31.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:56:31.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:56:31.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:56:31.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:56:31.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:56:31.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:56:31.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:56:31.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:56:31.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:56:31.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:56:31.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:56:30.433854+0000 mgr.x (mgr.24542) 2972 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:56:31.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:56:31.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:56:31.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: yield 2021-07-31T02:56:31.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:56:31.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:56:31.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:56:31.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:56:31.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:56:31.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:56:31.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:56:31.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:56:31.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:56:31.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:56:31.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:56:31.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: 2021-07-31T02:56:31.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:56:31.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: 2021-07-31T02:56:31.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:56:31.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:56:31.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:56:31.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:56:31.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:56:31.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:56:31.285 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:56:31.285 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:56:31.285 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:56:31.285 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:56:31.286 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:56:31.286 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:56:31.286 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:56:31.286 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:56:31.286 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:56:31.287 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:56:31.287 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:56:31.287 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:56:31.287 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: raise 2021-07-31T02:56:31.288 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:56:31.288 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:56:31.288 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:56:31.288 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:56:31.288 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:30 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:56:31.289 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: audit 2021-07-31T02:56:30 2021-07-31T02:56:31.289 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: .341596+0000 mon.b (mon.2) 422 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:56:31.290 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: audit 2021-07-31T02 2021-07-31T02:56:31.290 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: :56:30.343048+0000 mon.b (mon.2) 423 : 2021-07-31T02:56:31.290 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:56:31.290 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T02:56:31.291 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: 02:56:30.343905+0000 mon.b (mon.2) 424 : audit 2021-07-31T02:56:31.291 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:56:31.291 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: cephadm 2021- 2021-07-31T02:56:31.291 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: 07-31T02:56:30.344698+0000 mgr.x (mgr 2021-07-31T02:56:31.292 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: .24542) 2964 : cephadm [WRN] 2021-07-31T02:56:31.292 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:56:31.292 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: cephadm 2021-07 2021-07-31T02:56:31.292 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: -31T02:56:30.387605+0000 mgr.x (mgr. 2021-07-31T02:56:31.292 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: 24542) 2965 : cephadm [ERR] 2021-07-31T02:56:31.293 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:56:31.293 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:56:31.293 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: yield 2021-07-31T02:56:31.293 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:56:31.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:56:31.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:56:31.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:56:31.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:56:31.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:56:31.295 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:56:31.295 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:56:31.295 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:56:31.295 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:56:31.295 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:56:31.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:56:30.388058+0000 mgr.x (mgr. 2021-07-31T02:56:31.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: 24542) 2966 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:56:31.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: 2021-07-31T02:56:31.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: cephadm 2021-07- 2021-07-31T02:56:31.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: 31T02:56:30.399164+0000 mgr.x (mgr.24542) 2967 : cephadm [ERR] 2021-07-31T02:56:31.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:56:31.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:56:31.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: yield 2021-07-31T02:56:31.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:56:31.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:56:31.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:56:31.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:56:31.298 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: audit 2021-07-31T02:56:30.341596+0000 mon.b (mon.2) 422 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:56:31.299 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: audit 2021-07-31T 2021-07-31T02:56:31.299 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: 02:56:30.343048+0000 mon.b (mon.2) 423 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:56:31.299 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: audit 2021-07-31T02:56:30.343905+0000 2021-07-31T02:56:31.299 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: mon.b (mon.2) 424 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:56:31.300 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: 2021-07-31T02:56:31.300 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:56:31.300 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: 2021-07-31T02:56:30. 2021-07-31T02:56:31.300 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: 344698+0000 mgr.x (mgr.24542) 2964 : 2021-07-31T02:56:31.301 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:56:31.301 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: cephadm 2021-07 2021-07-31T02:56:31.301 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: -31T02:56:30.387605+0000 2021-07-31T02:56:31.301 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: mgr.x (mgr.24542) 2965 : cephadm [ERR] 2021-07-31T02:56:31.301 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: 2021-07-31T02:56:31.302 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:56:31.302 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:56:31.302 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: yield 2021-07-31T02:56:31.302 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:56:31.303 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:56:31.303 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:56:31.303 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:56:31.303 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:56:31.304 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:56:31.304 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:56:31.304 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:56:31.304 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:56:31.304 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:56:31.305 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:56:31.305 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:56:30.388058+0000 mgr.x (mgr.24542) 2966 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:56:31.305 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: cephadm 2021-07 2021-07-31T02:56:31.305 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: -31T02:56:30.399164+0000 mgr.x (mgr. 2021-07-31T02:56:31.306 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: 24542) 2967 : cephadm [ERR] 2021-07-31T02:56:31.306 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:56:31.306 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:56:31.306 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: yield 2021-07-31T02:56:31.306 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:56:31.307 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:56:31.307 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:56:31.307 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:56:31.308 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:56:31.308 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:56:31.308 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:56:31.308 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:56:31.309 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:56:31.309 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:56:31.309 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:56:31.309 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:56:30.399545+0000 mgr.x (mgr.24542) 2968 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:56:31.309 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: cephadm 2021 2021-07-31T02:56:31.310 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: -07-31T02:56:30.425138+0000 mgr.x ( 2021-07-31T02:56:31.310 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: mgr.24542) 2969 : cephadm [ERR] 2021-07-31T02:56:31.310 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:56:31.310 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:56:31.311 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: yield 2021-07-31T02:56:31.311 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:56:31.311 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:56:31.311 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:56:31.312 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:56:31.312 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:56:31.312 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:56:31.312 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:56:31.312 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:56:31.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:56:31.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:56:31.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:56:31.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: 2021-07-31T02:56:31.314 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:56:30.425587+0000 mgr.x (mgr.24542) 2970 : 2021-07-31T02:56:31.314 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:56:31.314 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:56:31.314 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:56:31.315 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: yield 2021-07-31T02:56:31.315 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:56:31.315 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:56:31.315 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:56:31.315 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:56:31.316 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:56:31.316 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:56:31.316 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:56:31.316 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:56:31.316 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:56:31.317 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:56:31.317 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:56:31.318 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:56:31.318 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:56:31.318 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:56:31.318 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:56:31.318 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:56:31.319 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:56:31.319 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:56:30.399545+0000 mgr.x (mgr.24542) 2968 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:56:31.319 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:56:30.425138+0000 2021-07-31T02:56:31.319 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: mgr.x (mgr.24542) 2969 : cephadm [ERR] 2021-07-31T02:56:31.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:56:31.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:56:31.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: yield 2021-07-31T02:56:31.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:56:31.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:56:31.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:56:31.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:56:31.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:56:31.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:56:31.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:56:31.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:56:31.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:56:31.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:56:31.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:56:31.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:56:30.425587+0000 mgr.x (mgr.24542) 2970 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:56:31.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:56:31.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:56:31.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: yield 2021-07-31T02:56:31.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:56:31.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:56:31.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:56:31.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:56:31.325 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:56:31.325 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:56:31.325 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:56:31.325 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:56:31.326 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:56:31.326 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:56:31.326 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:56:31.326 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: 2021-07-31T02:56:31.326 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:56:31.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: 2021-07-31T02:56:31.327 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:56:31.328 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: 2021-07-31T02:56:31.328 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:56:31.328 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: 2021-07-31T02:56:31.328 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:56:31.329 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:56:31.329 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:56:31.329 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:56:31.329 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:56:31.329 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:56:31.330 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:56:31.330 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:56:31.330 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:56:31.330 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:56:31.330 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:56:31.331 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:56:31.331 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:56:31.331 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:56:31.331 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:56:31.332 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:56:31.332 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:56:31.332 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:56:31.332 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: raise 2021-07-31T02:56:31.333 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:56:31.333 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:56:31.333 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:56:31.333 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:56:31.333 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:56:31.334 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:56:30.433514+0000 mgr.x (mgr.24542) 2971 : cephadm [ERR] 2021-07-31T02:56:31.334 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:56:31.334 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:56:31.334 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: yield 2021-07-31T02:56:31.335 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:56:31.335 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:56:31.335 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:56:31.335 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:56:31.336 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:56:31.336 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:56:31.336 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:56:31.336 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:56:31.336 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:56:31.337 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:56:31.337 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:56:31.337 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:56:30.433854+0000 mgr.x (mgr.24542) 2972 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:56:31.337 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:56:31.338 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:56:31.338 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: yield 2021-07-31T02:56:31.338 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:56:31.338 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:56:31.339 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:56:31.339 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:56:31.339 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:56:31.339 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:56:31.339 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:56:31.340 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:56:31.340 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:56:31.340 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:56:31.340 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:56:31.341 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: 2021-07-31T02:56:31.341 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:56:31.341 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: 2021-07-31T02:56:31.341 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:56:31.342 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:56:31.342 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:56:31.342 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:56:31.342 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:56:31.342 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:56:31.343 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:56:31.343 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:56:31.343 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:56:31.343 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:56:31.344 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:56:31.344 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:56:31.344 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:56:31.344 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:56:31.344 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:56:31.345 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:56:31.345 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:56:31.345 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:56:31.345 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: raise 2021-07-31T02:56:31.346 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:56:31.346 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:56:31.346 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:56:31.346 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:56:31.347 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:56:31.347 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:56:31.347 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:56:31.348 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:56:31.348 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:56:31.348 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:56:31.348 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:56:31.349 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:56:31.349 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:56:31.349 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:56:31.349 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:56:31.350 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:56:31.350 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:56:31.350 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:56:31.350 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:56:31.350 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:56:31.351 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:56:31.351 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:56:31.351 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:56:31.351 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: raise 2021-07-31T02:56:31.352 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:56:31.352 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:56:31.352 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:56:31.352 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:56:31.352 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:56:31.353 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: cephadm 2021 2021-07-31T02:56:31.353 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: -07-31T02:56:30.433514+0000 mgr.x (mgr.24542) 2971 : cephadm [ERR] 2021-07-31T02:56:31.353 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:56:31.353 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:56:31.354 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: yield 2021-07-31T02:56:31.354 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:56:31.354 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:56:31.354 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:56:31.354 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:56:31.355 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:56:31.355 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:56:31.355 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:56:31.355 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:56:31.356 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:56:31.356 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:56:31.356 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:56:31.356 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:56:30.433854+0000 mgr.x (mgr.24542) 2972 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:56:31.357 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:56:31.357 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:56:31.357 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: yield 2021-07-31T02:56:31.357 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:56:31.357 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:56:31.358 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:56:31.358 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:56:31.358 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:56:31.358 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:56:31.359 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:56:31.359 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:56:31.359 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:56:31.359 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:56:31.359 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:56:31.360 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: 2021-07-31T02:56:31.360 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:56:31.360 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: 2021-07-31T02:56:31.360 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:56:31.361 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:56:31.361 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:56:31.361 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:56:31.361 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:56:31.361 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:56:31.362 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:56:31.362 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:56:31.362 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:56:31.362 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:56:31.363 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:56:31.363 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:56:31.363 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:56:31.363 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:56:31.363 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:56:31.364 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:56:31.364 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:56:31.364 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:56:31.364 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: raise 2021-07-31T02:56:31.365 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:56:31.365 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:56:31.365 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:56:31.365 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:56:31.366 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:30 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:56:31.963 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:56:32.239 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:56:32.239 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 71m ago 72m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:56:32.240 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 69m ago 71m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:56:32.240 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 69m ago 75m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:56:32.240 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 71m ago 77m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:56:32.240 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 71m ago 77m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:56:32.241 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 69m ago 75m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:56:32.241 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 71m ago 75m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:56:32.241 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 71m ago 72m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:56:32.241 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 69m ago 72m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:56:32.241 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 71m ago 75m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:56:32.242 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 71m ago 74m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:56:32.242 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 71m ago 74m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:56:32.242 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 71m ago 74m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:56:32.242 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 69m ago 73m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:56:32.243 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 69m ago 73m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:56:32.243 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 69m ago 73m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:56:32.243 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 69m ago 72m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:56:32.243 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 69m ago 72m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:56:32.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:31 smithi154 conmon[34553]: cluster 2021-07-31T02:56:30.790833+0000 mgr.x (mgr.24542) 2973 : cluster [DBG] pgmap v4086: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:32.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:31 smithi045 conmon[39718]: cluster 2021-07-31T02:56:30.790833+0000 2021-07-31T02:56:32.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:31 smithi045 conmon[39718]: mgr.x (mgr.24542) 2973 : cluster [DBG] pgmap v4086: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:32.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:31 smithi045 conmon[33655]: cluster 2021-07-31T02:56:30.790833+0000 mgr.x 2021-07-31T02:56:32.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:31 smithi045 conmon[33655]: (mgr.24542) 2973 : cluster [DBG] pgmap v4086: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:32.576 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:56:32.576 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:56:32.576 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:56:32.577 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:56:32.577 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:56:32.577 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:56:32.577 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:56:32.578 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:56:32.578 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:56:32.578 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:56:32.578 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:56:32.579 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:56:32.579 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:56:32.579 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:56:32.579 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:56:32.579 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:56:32.580 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:56:33.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:32 smithi154 conmon[34553]: audit 2021-07-31T02:56:31.950868+0000 mgr.x (mgr.24542) 2974 : audit [DBG] from='client.28402 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:56:33.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:32 smithi154 conmon[34553]: 2021-07-31T02:56:33.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:32 smithi154 conmon[34553]: audit 2021-07-31T02:56:32.231322+0000 mgr.x (mgr.24542) 2975 : audit [DBG] from='client.28408 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:56:33.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:32 smithi154 conmon[34553]: audit 2021-07- 2021-07-31T02:56:33.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:32 smithi154 conmon[34553]: 31T02:56:32.576070+0000 mon.c (mon.1) 161 : audit [DBG] from='client.? 172.21.15.45:0/2392998793' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:56:33.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:32 smithi045 conmon[33655]: audit 2021-07-31T02:56:31.950868+0000 mgr.x (mgr.24542) 2974 : audit [DBG] 2021-07-31T02:56:33.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:32 smithi045 conmon[33655]: from='client.28402 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:56:33.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:32 smithi045 conmon[33655]: audit 2021-07-31T02:56:32.231322+0000 mgr.x (mgr.24542) 2975 : audit [DBG] from='client.28408 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:56:33.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:32 smithi045 conmon[33655]: audit 2021-07-31T02: 2021-07-31T02:56:33.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:32 smithi045 conmon[33655]: 56:32.576070+0000 mon.c (mon.1) 161 : audit [DBG] from='client.? 172.21.15.45:0/2392998793' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:56:33.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:32 smithi045 conmon[39718]: audit 2021-07-31T02:56 2021-07-31T02:56:33.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:32 smithi045 conmon[39718]: :31.950868+0000 mgr.x (mgr.24542) 2974 : audit [DBG] from='client.28402 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:56:33.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:32 smithi045 conmon[39718]: audit 2021-07-31T02:56:33.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:32 smithi045 conmon[39718]: 2021-07-31T02:56:32 2021-07-31T02:56:33.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:32 smithi045 conmon[39718]: .231322+0000 mgr.x (mgr. 2021-07-31T02:56:33.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:32 smithi045 conmon[39718]: 24542) 2975 : audit [DBG] 2021-07-31T02:56:33.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:32 smithi045 conmon[39718]: from='client.28408 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:56:33.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:32 smithi045 conmon[39718]: audit 2021-07 2021-07-31T02:56:33.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:32 smithi045 conmon[39718]: -31T02:56:32.576070+0000 mon.c ( 2021-07-31T02:56:33.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:32 smithi045 conmon[39718]: mon.1) 161 : audit 2021-07-31T02:56:33.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:32 smithi045 conmon[39718]: [DBG] from='client.? 172.21.15.45:0/2392998793' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:56:34.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:33 smithi154 conmon[34553]: cluster 2021-07-31T02:56:32.791114+0000 mgr.x (mgr.24542) 2976 : cluster [DBG] pgmap v4087: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:34.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:33 smithi045 conmon[33655]: cluster 2021-07-31T02:56:32.791114+0000 mgr.x (mgr.24542 2021-07-31T02:56:34.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:33 smithi045 conmon[33655]: ) 2976 : cluster [DBG] pgmap v4087: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:34.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:33 smithi045 conmon[39718]: cluster 2021-07-31T02:56:32.791114+0000 mgr.x (mgr. 2021-07-31T02:56:34.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:33 smithi045 conmon[39718]: 24542) 2976 : cluster [DBG] pgmap v4087: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:36.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:35 smithi154 conmon[34553]: cluster 2021-07-31T02:56:34.791679+0000 mgr.x (mgr.24542) 2977 : cluster [DBG] pgmap v4088: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:36.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:35 smithi045 conmon[33655]: cluster 2021-07-31T02:56:34.791679 2021-07-31T02:56:36.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:35 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 2977 : cluster [DBG] pgmap v4088: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:36.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:35 smithi045 conmon[39718]: cluster 2021-07-31T02:56:34.791679+0000 mgr.x 2021-07-31T02:56:36.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:35 smithi045 conmon[39718]: (mgr.24542) 2977 : cluster [DBG] pgmap v4088: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:38.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:37 smithi154 conmon[34553]: cluster 2021-07-31T02:56:36.792145+0000 mgr.x (mgr.24542) 2978 : cluster [DBG] pgmap v4089: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:38.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:37 smithi045 conmon[39718]: cluster 2021-07-31T02:56:36.792145+0000 mgr.x (mgr 2021-07-31T02:56:38.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:37 smithi045 conmon[39718]: .24542) 2978 : cluster [DBG] pgmap v4089: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:38.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:37 smithi045 conmon[33655]: cluster 2021-07-31T02:56:36.792145+0000 mgr.x (mgr.24542) 2978 : cluster [DBG] 2021-07-31T02:56:38.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:37 smithi045 conmon[33655]: pgmap v4089: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:40.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:39 smithi154 conmon[34553]: cluster 2021-07-31T02:56:38.792584+0000 mgr.x (mgr.24542) 2979 : cluster [DBG] pgmap v4090: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:40.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:39 smithi045 conmon[39718]: cluster 2021-07-31T02:56:38.792584+0000 mgr.x ( 2021-07-31T02:56:40.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:39 smithi045 conmon[39718]: mgr.24542) 2979 : cluster [DBG] pgmap v4090: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:40.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:39 smithi045 conmon[33655]: cluster 2021-07-31T02:56:38.792584+0000 mgr.x (mgr.24542 2021-07-31T02:56:40.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:39 smithi045 conmon[33655]: ) 2979 : cluster [DBG] pgmap v4090: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:42.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:41 smithi154 conmon[34553]: cluster 2021-07-31T02:56:40.793101+0000 mgr.x (mgr.24542) 2980 : cluster [DBG] pgmap v4091: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:42.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:41 smithi045 conmon[33655]: cluster 2021-07-31T02:56:40.793101+0000 2021-07-31T02:56:42.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:41 smithi045 conmon[33655]: mgr.x (mgr.24542) 2980 : cluster [DBG] pgmap v4091: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:42.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:41 smithi045 conmon[39718]: cluster 2021-07-31T02:56:40.793101+0000 mgr.x (mgr.24542) 2980 : cluster 2021-07-31T02:56:42.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:41 smithi045 conmon[39718]: [DBG] pgmap v4091: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:44.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:43 smithi154 conmon[34553]: cluster 2021-07-31T02:56:42.793470+0000 mgr.x (mgr.24542) 2981 : cluster [DBG] pgmap v4092: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:44.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:43 smithi154 conmon[34553]: 2021-07-31T02:56:44.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:43 smithi045 conmon[33655]: cluster 2021-07-31T02:56:42.793470+0000 mgr.x (mgr.24542) 2981 : cluster [DBG] pgmap v4092: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:44.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:43 smithi045 conmon[39718]: cluster 2021-07-31T02:56:42.793470+0000 mgr.x ( 2021-07-31T02:56:44.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:43 smithi045 conmon[39718]: mgr.24542) 2981 : cluster [DBG] pgmap v4092: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:46.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:45 smithi154 conmon[34553]: cluster 2021-07-31T02:56:44.793884+0000 mgr.x (mgr.24542) 2982 : cluster [DBG] pgmap v4093: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:46.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:45 smithi045 conmon[33655]: cluster 2021-07-31T02:56:44.793884+0000 mgr.x (mgr.24542) 2982 : cluster [DBG] pgmap v4093: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:46.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:45 smithi045 conmon[39718]: cluster 2021-07-31T02:56:44.793884+0000 mgr.x (mgr 2021-07-31T02:56:46.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:45 smithi045 conmon[39718]: .24542) 2982 : cluster [DBG] pgmap v4093: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:48.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:47 smithi154 conmon[34553]: cluster 2021-07-31T02:56:46.794290+0000 mgr.x (mgr.24542) 2983 : cluster [DBG] pgmap v4094: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:48.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:47 smithi045 conmon[39718]: cluster 2021-07-31T02:56:46.794290+0000 mgr.x (mgr.24542) 2983 : cluster [DBG] pgmap v4094: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:48.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:47 smithi045 conmon[33655]: cluster 2021-07-31T02:56:46.794290+0000 mgr.x (mgr.24542) 2983 : cluster 2021-07-31T02:56:48.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:47 smithi045 conmon[33655]: [DBG] pgmap v4094: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:50.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:49 smithi154 conmon[34553]: cluster 2021-07-31T02:56:48.794740+0000 mgr.x (mgr.24542) 2984 : cluster [DBG] pgmap v4095: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:50.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:49 smithi045 conmon[39718]: cluster 2021-07-31T02:56: 2021-07-31T02:56:50.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:49 smithi045 conmon[39718]: 48.794740+0000 mgr.x (mgr.24542) 2984 : cluster [DBG] pgmap v4095: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:50.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:49 smithi045 conmon[33655]: cluster 2021-07-31T02:56:48.794740+0000 mgr.x (mgr.24542) 2984 : cluster [DBG] pgmap v4095: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:50.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:49 smithi045 conmon[33655]: 2021-07-31T02:56:52.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:51 smithi154 conmon[34553]: cluster 2021-07-31T02:56:50.795248+0000 mgr.x (mgr.24542) 2985 : cluster [DBG] pgmap v4096: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:52.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:51 smithi045 conmon[33655]: cluster 2021-07-31T02:56:50.795248+0000 mgr.x ( 2021-07-31T02:56:52.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:51 smithi045 conmon[33655]: mgr.24542) 2985 : cluster [DBG] pgmap v4096: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:52.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:51 smithi045 conmon[39718]: cluster 2021-07-31T02:56:50.795248+0000 mgr.x (mgr.24542) 2985 : cluster [DBG] pgmap v4096: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:54.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:53 smithi154 conmon[34553]: cluster 2021-07-31T02:56:52.795696+0000 mgr.x (mgr.24542) 2986 : cluster 2021-07-31T02:56:54.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:53 smithi154 conmon[34553]: [DBG] pgmap v4097: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:54.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:53 smithi045 conmon[39718]: cluster 2021-07-31T02:56:52.795696+0000 mgr.x 2021-07-31T02:56:54.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:53 smithi045 conmon[39718]: (mgr.24542) 2986 : cluster [DBG] pgmap v4097: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:54.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:53 smithi045 conmon[33655]: cluster 2021-07-31T02:56:52.795696+0000 mgr.x (mgr.24542) 2986 : cluster [DBG] pgmap v4097: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:56.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:55 smithi154 conmon[34553]: cluster 2021-07-31T02:56:54.796270+0000 mgr.x (mgr.24542) 2987 : cluster [DBG] pgmap v4098: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:56.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:55 smithi045 conmon[33655]: cluster 2021-07-31T02:56:54.796270+0000 mgr.x (mgr.24542 2021-07-31T02:56:56.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:55 smithi045 conmon[33655]: ) 2987 : cluster [DBG] pgmap v4098: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:56.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:55 smithi045 conmon[39718]: cluster 2021-07-31T02:56:54.796270+0000 mgr.x (mgr 2021-07-31T02:56:56.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:55 smithi045 conmon[39718]: .24542) 2987 : cluster [DBG] pgmap v4098: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:58.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:57 smithi154 conmon[34553]: cluster 2021-07-31T02:56:56.796782+0000 mgr.x (mgr.24542) 2988 : cluster [DBG] pgmap v4099: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:58.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:57 smithi045 conmon[33655]: cluster 2021-07-31T02:56:56.796782+0000 mgr.x (mgr.24542) 2988 : cluster [DBG] pgmap v4099: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:56:58.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:57 smithi045 conmon[39718]: cluster 2021-07-31T02:56:56.796782+0000 mgr.x (mgr 2021-07-31T02:56:58.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:57 smithi045 conmon[39718]: .24542) 2988 : cluster [DBG] pgmap v4099: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:00.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:56:59 smithi154 conmon[34553]: cluster 2021-07-31T02:56:58.797290+0000 mgr.x (mgr.24542) 2989 : cluster [DBG] pgmap v4100: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:00.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:56:59 smithi045 conmon[33655]: cluster 2021-07-31T02:56:58.797290+0000 mgr.x (mgr.24542) 2989 : cluster [DBG] pgmap v4100: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:00.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:59 smithi045 conmon[39718]: cluster 2021-07-31T02:56:58.797290+0000 mgr.x (mgr. 2021-07-31T02:57:00.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:56:59 smithi045 conmon[39718]: 24542) 2989 : cluster [DBG] pgmap v4100: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:02.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:01 smithi154 conmon[34553]: cluster 2021-07-31T02:57:00.797802+0000 mgr.x (mgr.24542) 2990 : cluster [DBG] 2021-07-31T02:57:02.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:01 smithi154 conmon[34553]: pgmap v4101: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:02.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:01 smithi045 conmon[39718]: cluster 2021-07-31T02:57:00.797802+0000 mgr.x (mgr.24542) 2990 : cluster [DBG] 2021-07-31T02:57:02.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:01 smithi045 conmon[39718]: pgmap v4101: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:02.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:01 smithi045 conmon[33655]: cluster 2021-07-31T02:57:00.797802+0000 mgr.x (mgr.24542) 2990 : cluster 2021-07-31T02:57:02.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:01 smithi045 conmon[33655]: [DBG] pgmap v4101: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:02.892 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:57:03.168 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:57:03.169 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 71m ago 72m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:57:03.169 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 69m ago 72m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:57:03.169 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 69m ago 75m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:57:03.170 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 71m ago 78m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:57:03.170 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 71m ago 78m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:57:03.170 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 69m ago 76m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:57:03.170 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 71m ago 76m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:57:03.170 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 71m ago 72m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:57:03.171 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 69m ago 72m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:57:03.171 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 71m ago 75m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:57:03.171 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 71m ago 75m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:57:03.171 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 71m ago 75m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:57:03.172 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 71m ago 74m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:57:03.172 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 69m ago 74m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:57:03.172 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 69m ago 74m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:57:03.172 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 69m ago 73m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:57:03.172 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 69m ago 73m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:57:03.173 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 69m ago 73m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:57:03.504 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:57:03.504 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:57:03.505 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:57:03.505 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:57:03.505 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:57:03.505 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:57:03.506 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:57:03.506 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:57:03.506 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:57:03.506 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:57:03.507 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:57:03.507 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:57:03.507 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:57:03.507 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:57:03.508 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:57:03.508 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:57:03.508 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:57:04.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:03 smithi154 conmon[34553]: cluster 2021-07-31T02:57:02.798202+0000 mgr.x (mgr.24542) 2991 : cluster [DBG] pgmap v4102: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:04.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:03 smithi154 conmon[34553]: audit 2021-07-31T02:57:02.880337+0000 mgr.x (mgr.24542) 2992 : audit [DBG] from='client.28420 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:57:04.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:03 smithi154 conmon[34553]: audit 2021-07-31T02:57:03.160736+0000 mgr.x (mgr.24542) 2993 : audit [DBG] from='client.28426 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:57:04.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:03 smithi154 conmon[34553]: audit 2021-07-31T02:57:03.503792+0000 mon.c (mon.1) 2021-07-31T02:57:04.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:03 smithi154 conmon[34553]: 162 : audit [DBG] from='client.? 172.21.15.45:0/1619620396' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:57:04.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:03 smithi045 conmon[39718]: cluster 2021-07-31T02:57:02.798202 2021-07-31T02:57:04.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:03 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 2991 : cluster [DBG] pgmap v4102: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:04.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:03 smithi045 conmon[39718]: audit 2021 2021-07-31T02:57:04.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:03 smithi045 conmon[39718]: -07-31T02:57 2021-07-31T02:57:04.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:03 smithi045 conmon[39718]: :02.880337+0000 mgr.x ( 2021-07-31T02:57:04.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:03 smithi045 conmon[39718]: mgr.24542) 2992 : audit 2021-07-31T02:57:04.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:03 smithi045 conmon[39718]: [DBG] from='client.28420 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:57:04.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:03 smithi045 conmon[39718]: audit 2021 2021-07-31T02:57:04.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:03 smithi045 conmon[39718]: -07-31T02:57:03 2021-07-31T02:57:04.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:03 smithi045 conmon[39718]: .160736+0000 mgr.x (mgr. 2021-07-31T02:57:04.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:03 smithi045 conmon[39718]: 24542) 2993 : audit [DBG] 2021-07-31T02:57:04.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:03 smithi045 conmon[39718]: from='client.28426 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:57:04.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:03 smithi045 conmon[39718]: audit 2021 2021-07-31T02:57:04.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:03 smithi045 conmon[39718]: -07-31T02:57: 2021-07-31T02:57:04.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:03 smithi045 conmon[39718]: 03.503792+0000 mon.c ( 2021-07-31T02:57:04.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:03 smithi045 conmon[39718]: mon.1) 162 : audit 2021-07-31T02:57:04.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:03 smithi045 conmon[39718]: [DBG] from='client.? 172.21.15.45:0/1619620396' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:57:04.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:03 smithi045 conmon[33655]: cluster 2021-07-31T02:57:02.798202+0000 mgr.x (mgr 2021-07-31T02:57:04.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:03 smithi045 conmon[33655]: .24542) 2991 : cluster [DBG] pgmap v4102: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:04.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:03 smithi045 conmon[33655]: audit 2021-07-31T02:57:02.880337+0000 mgr.x (mgr.24542) 2992 : audit [DBG] from='client.28420 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:57:04.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:03 smithi045 conmon[33655]: audit 2021-07-31T02:57:03.160736+0000 2021-07-31T02:57:04.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:03 smithi045 conmon[33655]: mgr.x (mgr.24542) 2993 : audit [DBG] from='client.28426 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:57:04.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:03 smithi045 conmon[33655]: 2021-07-31T02:57:04.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:03 smithi045 conmon[33655]: audit 2021-07-31T02:57:04.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:03 smithi045 conmon[33655]: 2021-07-31T02:57:03.503792+0000 2021-07-31T02:57:04.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:03 smithi045 conmon[33655]: mon.c (mon.1) 162 : audit [DBG] from='client.? 172.21.15.45:0/1619620396' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:57:06.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:05 smithi154 conmon[34553]: cluster 2021-07-31T02:57:04.798601+0000 mgr.x (mgr.24542) 2994 : cluster [DBG] pgmap v4103: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:06.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:05 smithi045 conmon[39718]: cluster 2021-07-31T02:57:04 2021-07-31T02:57:06.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:05 smithi045 conmon[39718]: .798601+0000 mgr.x (mgr.24542) 2994 : cluster [DBG] pgmap v4103: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:06.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:05 smithi045 conmon[33655]: cluster 2021-07-31T02:57:04.798601+0000 mgr.x (mgr.24542) 2994 : cluster [DBG] 2021-07-31T02:57:06.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:05 smithi045 conmon[33655]: pgmap v4103: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:08.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:07 smithi154 conmon[34553]: cluster 2021-07-31T02:57:06.798974+0000 mgr.x (mgr.24542) 2995 : cluster [DBG] pgmap v4104: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:08.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:07 smithi045 conmon[39718]: cluster 2021-07-31T02:57:06.798974+0000 mgr.x ( 2021-07-31T02:57:08.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:07 smithi045 conmon[39718]: mgr.24542) 2995 : cluster [DBG] pgmap v4104: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:08.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:07 smithi045 conmon[33655]: cluster 2021-07-31T02:57:06.798974+0000 mgr.x (mgr.24542) 2995 : cluster [DBG] 2021-07-31T02:57:08.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:07 smithi045 conmon[33655]: pgmap v4104: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:10.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:09 smithi154 conmon[34553]: cluster 2021-07-31T02:57:08.799419+0000 mgr.x (mgr.24542) 2996 : cluster [DBG] pgmap v4105: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:10.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:09 smithi045 conmon[39718]: cluster 2021-07-31T02:57:08.799419+0000 mgr.x 2021-07-31T02:57:10.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:09 smithi045 conmon[39718]: (mgr.24542) 2996 : cluster [DBG] pgmap v4105: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:10.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:09 smithi045 conmon[33655]: cluster 2021-07- 2021-07-31T02:57:10.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:09 smithi045 conmon[33655]: 31T02:57:08.799419+0000 mgr.x (mgr.24542) 2996 : cluster [DBG] pgmap v4105: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:12.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:11 smithi154 conmon[34553]: cluster 2021-07-31T02:57:10.799993+0000 mgr.x (mgr.24542) 2997 : cluster [DBG] pgmap v4106: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:12.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:11 smithi154 conmon[34553]: 2021-07-31T02:57:12.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:11 smithi045 conmon[39718]: cluster 2021-07-31T02:57:10.799993+0000 mgr.x 2021-07-31T02:57:12.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:11 smithi045 conmon[39718]: (mgr.24542) 2997 : cluster [DBG] pgmap v4106: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:12.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:11 smithi045 conmon[33655]: cluster 2021-07-31T02:57:10.799993+0000 mgr.x (mgr.24542) 2021-07-31T02:57:12.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:11 smithi045 conmon[33655]: 2997 : cluster [DBG] pgmap v4106: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:14.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:13 smithi154 conmon[34553]: cluster 2021-07-31T02:57:12.800366+0000 mgr.x (mgr.24542) 2998 : cluster [DBG] 2021-07-31T02:57:14.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:13 smithi154 conmon[34553]: pgmap v4107: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:14.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:13 smithi045 conmon[39718]: cluster 2021-07-31T02:57:12 2021-07-31T02:57:14.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:13 smithi045 conmon[39718]: .800366+0000 mgr.x (mgr.24542) 2998 : cluster [DBG] pgmap v4107: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:14.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:13 smithi045 conmon[33655]: cluster 2021-07-31T02:57:12.800366+0000 mgr.x (mgr.24542) 2998 : cluster 2021-07-31T02:57:14.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:13 smithi045 conmon[33655]: [DBG] pgmap v4107: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:16.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:15 smithi154 conmon[34553]: cluster 2021-07-31T02:57:14.800886+0000 mgr.x (mgr.24542) 2999 : cluster [DBG] pgmap v4108: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:16.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:15 smithi045 conmon[39718]: cluster 2021-07-31T02:57:14.800886 2021-07-31T02:57:16.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:15 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 2999 : cluster [DBG] pgmap v4108: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:16.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:15 smithi045 conmon[33655]: cluster 2021-07-31T02:57:14.800886+0000 mgr.x (mgr.24542) 2021-07-31T02:57:16.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:15 smithi045 conmon[33655]: 2999 : cluster [DBG] pgmap v4108: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:18.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:17 smithi154 conmon[34553]: cluster 2021-07-31T02:57:16.801276+0000 mgr.x (mgr.24542) 3000 : cluster [DBG] pgmap v4109: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:18.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:17 smithi045 conmon[33655]: cluster 2021-07-31T02:57:16.801276+0000 mgr.x (mgr.24542) 3000 : cluster [DBG] pgmap v4109: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:18.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:17 smithi045 conmon[39718]: cluster 2021-07-31T02:57:16 2021-07-31T02:57:18.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:17 smithi045 conmon[39718]: .801276+0000 mgr.x (mgr.24542) 3000 : cluster [DBG] pgmap v4109: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:20.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:19 smithi154 conmon[34553]: cluster 2021-07-31T02:57:18.801788+0000 mgr.x (mgr.24542) 3001 : cluster [DBG] 2021-07-31T02:57:20.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:19 smithi154 conmon[34553]: pgmap v4110: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:20.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:19 smithi045 conmon[39718]: cluster 2021-07-31T02:57:18.801788+0000 mgr.x (mgr. 2021-07-31T02:57:20.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:19 smithi045 conmon[39718]: 24542) 3001 : cluster [DBG] pgmap v4110: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:20.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:19 smithi045 conmon[33655]: cluster 2021-07-31T02:57:18.801788+0000 mgr.x (mgr.24542) 2021-07-31T02:57:20.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:19 smithi045 conmon[33655]: 3001 : cluster [DBG] pgmap v4110: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:22.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:21 smithi154 conmon[34553]: cluster 2021-07-31T02:57:20.802406+0000 mgr.x (mgr.24542) 3002 : cluster [DBG] pgmap v4111: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:22.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:21 smithi154 conmon[34553]: 2021-07-31T02:57:22.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:21 smithi045 conmon[39718]: cluster 2021-07-31T02:57:20.802406+0000 mgr.x ( 2021-07-31T02:57:22.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:21 smithi045 conmon[39718]: mgr.24542) 3002 : cluster [DBG] pgmap v4111: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:22.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:21 smithi045 conmon[33655]: cluster 2021-07-31T02:57:20.802406+0000 mgr.x (mgr.24542) 3002 : cluster 2021-07-31T02:57:22.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:21 smithi045 conmon[33655]: [DBG] pgmap v4111: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:24.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:23 smithi154 conmon[34553]: cluster 2021-07-31T02:57:22.802975+0000 mgr.x (mgr.24542) 3003 : cluster 2021-07-31T02:57:24.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:23 smithi154 conmon[34553]: [DBG] pgmap v4112: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:24.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:23 smithi045 conmon[39718]: cluster 2021-07-31T02:57:22.802975+0000 mgr.x (mgr.24542) 3003 : cluster [DBG] pgmap v4112: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:24.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:23 smithi045 conmon[33655]: cluster 2021-07-31T02:57:22.802975+0000 mgr.x (mgr.24542) 3003 2021-07-31T02:57:24.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:23 smithi045 conmon[33655]: : cluster [DBG] pgmap v4112: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:26.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:25 smithi154 conmon[34553]: cluster 2021-07-31T02:57:24.803566+0000 mgr.x (mgr.24542) 3004 : cluster [DBG] pgmap v4113: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:26.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:25 smithi045 conmon[39718]: cluster 2021-07-31T02:57:24.803566+0000 mgr.x ( 2021-07-31T02:57:26.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:25 smithi045 conmon[39718]: mgr.24542) 3004 : cluster [DBG] pgmap v4113: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:26.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:25 smithi045 conmon[33655]: cluster 2021-07-31T02:57:24.803566+0000 mgr.x (mgr.24542) 2021-07-31T02:57:26.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:25 smithi045 conmon[33655]: 3004 : cluster [DBG] pgmap v4113: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:28.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:27 smithi154 conmon[34553]: cluster 2021-07-31T02:57:26.804082+0000 mgr.x (mgr.24542) 3005 : cluster [DBG] pgmap v4114: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:28.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:27 smithi045 conmon[39718]: cluster 2021-07-31T02:57:26.804082+0000 mgr.x ( 2021-07-31T02:57:28.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:27 smithi045 conmon[39718]: mgr.24542) 3005 : cluster [DBG] pgmap v4114: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:28.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:27 smithi045 conmon[33655]: cluster 2021-07-31T02:57:26.804082+0000 mgr.x 2021-07-31T02:57:28.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:27 smithi045 conmon[33655]: (mgr.24542) 3005 : cluster [DBG] pgmap v4114: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:29.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:28 smithi154 conmon[34553]: audit 2021-07-31T02:57:28.215871+0000 mon.b (mon.2) 425 : audit [INF] 2021-07-31T02:57:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:28 smithi154 conmon[34553]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:57:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:28 smithi154 conmon[34553]: audit 2021-07-31T02:57:28.216171+0000 mon.b (mon.2) 426 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:57:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:28 smithi154 conmon[34553]: audit 2021-07-31T02:57:28.216625+0000 mon.a (mon.0) 3228 2021-07-31T02:57:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:28 smithi154 conmon[34553]: : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:57:29.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:28 smithi154 conmon[34553]: audit 2021-07-31T02:57:28.217040+0000 mon.a 2021-07-31T02:57:29.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:28 smithi154 conmon[34553]: (mon.0) 3229 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:57:29.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:28 smithi045 conmon[39718]: audit 2021-07-31T02:57:28.215871+0000 mon.b (mon 2021-07-31T02:57:29.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:28 smithi045 conmon[39718]: .2) 425 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:57:29.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:28 smithi045 conmon[39718]: audit 2021-07-31T02:57:28.216171+0000 mon.b (mon.2) 426 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:57:29.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:28 smithi045 conmon[39718]: audit 2021-07-31T02:57:28.216625+0000 mon.a (mon.0) 3228 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:57:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:28 smithi045 conmon[39718]: audit 2021-07-31T02:57:28.217040+0000 mon.a (mon.0) 3229 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:57:29.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:28 smithi045 conmon[33655]: audit 2021-07-31T02:57:28.215871+0000 mon.b (mon.2) 425 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:57:29.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:28 smithi045 conmon[33655]: audit 2021-07-31T02:57:28.216171+0000 mon.b (mon.2) 426 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:57:29.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:28 smithi045 conmon[33655]: audit 2021-07-31T02:57:28.216625+0000 mon.a (mon.0) 3228 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:57:29.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:28 smithi045 conmon[33655]: audit 2021-07-31T02:57 2021-07-31T02:57:29.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:28 smithi045 conmon[33655]: :28.217040+0000 mon.a (mon.0) 3229 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:57:30.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:29 smithi154 conmon[34553]: cluster 2021-07-31T02:57:28.804537+0000 mgr.x (mgr.24542) 3006 : cluster [DBG] pgmap v4115: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:30.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:29 smithi045 conmon[39718]: cluster 2021-07-31T02:57:28.804537+0000 mgr.x (mgr. 2021-07-31T02:57:30.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:29 smithi045 conmon[39718]: 24542) 3006 : cluster [DBG] pgmap v4115: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:30.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:29 smithi045 conmon[33655]: cluster 2021-07-31T02:57:28.804537+0000 mgr.x (mgr.24542 2021-07-31T02:57:30.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:29 smithi045 conmon[33655]: ) 3006 : cluster [DBG] pgmap v4115: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:30.753 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: debug 2021-07-31T02:57:30.480+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:57:30.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:57:30.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:57:30.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: yield 2021-07-31T02:57:30.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:57:30.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:57:30.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:57:30.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:57:30.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:57:30.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:57:30.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:57:30.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:57:30.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:57:30.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:57:30.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:57:30.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: debug 2021-07-31T02:57:30.492+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:57:30.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:57:30.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:57:30.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: yield 2021-07-31T02:57:30.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:57:30.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:57:30.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:57:30.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:57:30.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:57:30.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:57:30.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:57:30.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:57:30.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:57:30.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:57:30.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:57:30.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: debug 2021-07-31T02:57:30.518+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:57:30.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:57:30.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:57:30.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: yield 2021-07-31T02:57:30.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:57:30.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:57:30.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:57:30.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:57:30.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:57:30.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:57:30.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:57:30.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:57:30.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:57:30.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:57:30.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:57:30.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: debug 2021-07-31T02:57:30.518+0000 7fcb94162700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:57:30.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:57:30.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:57:30.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: yield 2021-07-31T02:57:30.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:57:30.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:57:30.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:57:30.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:57:30.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:57:30.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:57:30.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:57:30.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:57:30.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:57:30.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:57:30.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:57:30.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: 2021-07-31T02:57:30.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:57:30.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: 2021-07-31T02:57:30.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:57:30.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:57:30.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:57:30.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:57:30.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:57:30.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:57:30.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:57:30.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:57:30.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:57:30.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:57:30.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:57:30.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:57:30.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:57:30.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:57:30.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:57:30.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:57:30.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:57:30.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:57:30.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: raise 2021-07-31T02:57:30.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:57:30.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:57:30.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:57:30.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:57:30.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:57:30.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: debug 2021-07-31T02:57:30.527+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:57:30.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:57:30.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:57:30.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: yield 2021-07-31T02:57:30.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:57:30.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:57:30.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:57:30.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:57:30.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:57:30.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:57:30.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:57:30.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:57:30.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:57:30.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:57:30.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:57:30.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: debug 2021-07-31T02:57:30.527+0000 7fcb93961700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:57:30.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:57:30.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:57:30.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: yield 2021-07-31T02:57:30.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:57:30.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:57:30.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:57:30.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:57:30.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:57:30.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:57:30.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:57:30.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:57:30.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:57:30.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:57:30.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:57:30.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: 2021-07-31T02:57:30.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:57:30.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: 2021-07-31T02:57:30.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:57:30.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:57:30.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:57:30.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:57:30.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:57:30.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:57:30.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:57:30.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:57:30.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:57:30.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:57:30.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:57:30.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:57:30.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:57:30.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:57:30.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:57:30.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:57:30.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:57:30.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:57:30.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: raise 2021-07-31T02:57:30.786 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:57:30.786 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:57:30.786 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:57:30.786 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:57:30.786 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:57:31.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: audit 2021-07-31T02:57:30.434633+0000 mon.b (mon.2) 427 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:57:31.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: audit 2021-07-31T02:57:30.436272+0000 mon.b (mon.2) 428 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:57:31.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: audit 2021-07-31T02:57:30.437139+0000 mon.b (mon.2) 429 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:57:31.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:57:30 2021-07-31T02:57:31.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: .437902+0000 mgr.x (mgr.24542) 3007 : cephadm [WRN] 2021-07-31T02:57:31.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:57:31.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: cephadm 2021-07- 2021-07-31T02:57:31.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: 31T02:57:30.482328+0000 mgr.x ( 2021-07-31T02:57:31.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: mgr.24542) 3008 : cephadm [ERR] 2021-07-31T02:57:31.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:57:31.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:57:31.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: yield 2021-07-31T02:57:31.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:57:31.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:57:31.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:57:31.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:57:31.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:57:31.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:57:31.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:57:31.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:57:31.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:57:31.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:57:31.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:57:31.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:57:30.482743+0000 mgr.x (mgr.24542) 3009 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:57:31.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:57:30.494071+0000 mgr.x (mgr 2021-07-31T02:57:31.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: .24542) 3010 : cephadm [ERR] 2021-07-31T02:57:31.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:57:31.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:57:31.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: yield 2021-07-31T02:57:31.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:57:31.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:57:31.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:57:31.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:57:31.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:57:31.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:57:31.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:57:31.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:57:31.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:57:31.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:57:31.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:57:31.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:57:30.494603+0000 mgr.x (mgr.24542) 3011 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:57:31.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:57:30.520012+0000 mgr.x (mgr.24542) 3012 : cephadm [ERR] 2021-07-31T02:57:31.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: 2021-07-31T02:57:31.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:57:31.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:57:31.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: yield 2021-07-31T02:57:31.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:57:31.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:57:31.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:57:31.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:57:31.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:57:31.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:57:31.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:57:31.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:57:31.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:57:31.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:57:31.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:57:31.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:57:30.520471+0000 mgr.x (mgr.24542) 3013 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:57:31.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:57:31.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:57:31.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: yield 2021-07-31T02:57:31.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:57:31.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:57:31.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:57:31.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:57:31.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:57:31.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:57:31.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:57:31.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:57:31.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:57:31.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:57:31.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:57:31.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: 2021-07-31T02:57:31.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:57:31.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: 2021-07-31T02:57:31.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:57:31.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:57:31.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:57:31.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:57:31.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:57:31.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:57:31.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:57:31.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:57:31.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:57:31.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:57:31.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:57:31.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:57:31.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:57:31.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:57:31.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:57:31.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:57:31.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:57:31.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:57:31.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: raise 2021-07-31T02:57:31.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:57:31.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:57:31.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:57:31.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:57:31.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:57:31.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:57:30.528996+0000 mgr.x (mgr.24542) 3014 : cephadm 2021-07-31T02:57:31.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: [ERR] 2021-07-31T02:57:31.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:57:31.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:57:31.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: yield 2021-07-31T02:57:31.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:57:31.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:57:31.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:57:31.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:57:31.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:57:31.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:57:31.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:57:31.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:57:31.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:57:31.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:57:31.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:57:31.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:57:30.529430+0000 mgr.x (mgr.24542) 3015 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:57:31.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:57:31.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:57:31.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: yield 2021-07-31T02:57:31.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:57:31.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:57:31.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:57:31.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:57:31.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:57:31.285 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:57:31.285 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:57:31.285 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:57:31.285 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:57:31.286 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:57:31.286 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:57:31.286 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: 2021-07-31T02:57:31.286 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:57:31.287 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: 2021-07-31T02:57:31.287 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:57:31.287 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:57:31.287 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:57:31.288 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:57:31.288 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:57:31.288 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:57:31.289 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:57:31.289 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:57:31.289 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:57:31.289 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:57:31.290 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:57:31.290 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:57:31.290 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:57:31.290 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:57:31.291 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:57:31.291 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:57:31.291 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:57:31.291 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:57:31.291 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: raise 2021-07-31T02:57:31.292 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:57:31.292 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:57:31.292 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:57:31.292 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:57:31.293 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:30 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:57:31.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: audit 2021-07-31T02:57: 2021-07-31T02:57:31.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: 30.434633+0000 mon.b (mon.2) 427 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:57:31.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: audit 2021- 2021-07-31T02:57:31.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: 07-31T02:57:30 2021-07-31T02:57:31.295 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: .436272+0000 mon.b (mon. 2021-07-31T02:57:31.295 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: 2) 428 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:57:31.295 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: 2021-07-31T02:57:31.295 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: audit 2021- 2021-07-31T02:57:31.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: 07-31T02:57:30.437139 2021-07-31T02:57:31.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: +0000 mon.b (mon.2) 2021-07-31T02:57:31.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: 429 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:57:31.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: cephadm 2021 2021-07-31T02:57:31.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: -07-31T02:57:30. 2021-07-31T02:57:31.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: 437902+0000 mgr.x (mgr. 2021-07-31T02:57:31.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: 24542) 3007 : cephadm [WRN] 2021-07-31T02:57:31.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:57:31.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: cephadm 2021 2021-07-31T02:57:31.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: -07-31T02:57:30 2021-07-31T02:57:31.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: .482328+0000 mgr.x (mgr.24542 2021-07-31T02:57:31.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: ) 3008 : cephadm [ERR] 2021-07-31T02:57:31.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:57:31.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:57:31.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: yield 2021-07-31T02:57:31.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:57:31.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:57:31.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:57:31.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:57:31.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:57:31.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:57:31.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:57:31.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:57:31.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:57:31.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:57:31.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:57:31.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: 2021-07-31T02:57:31.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:57:30.482743+0000 mgr.x ( 2021-07-31T02:57:31.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: mgr.24542) 3009 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:57:31.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: 2021-07-31T02:57:31.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: cephadm 2021 2021-07-31T02:57:31.304 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: -07-31T02:57:30. 2021-07-31T02:57:31.304 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: 494071+0000 mgr.x ( 2021-07-31T02:57:31.304 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: mgr.24542) 3010 : cephadm [ERR] 2021-07-31T02:57:31.304 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: 2021-07-31T02:57:31.305 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:57:31.305 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:57:31.305 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: yield 2021-07-31T02:57:31.305 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:57:31.306 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:57:31.306 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:57:31.306 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:57:31.306 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:57:31.306 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:57:31.307 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:57:31.307 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:57:31.307 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:57:31.307 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:57:31.308 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:57:31.308 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:57:30.494603+0000 mgr.x ( 2021-07-31T02:57:31.308 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: mgr.24542) 3011 : cephadm [INF] 2021-07-31T02:57:31.308 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:57:31.309 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: cephadm 2021 2021-07-31T02:57:31.309 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: -07-31T02:57:30. 2021-07-31T02:57:31.309 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: 520012+0000 mgr.x (mgr.24542) 2021-07-31T02:57:31.309 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: 3012 : cephadm [ERR] 2021-07-31T02:57:31.310 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:57:31.310 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:57:31.310 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: yield 2021-07-31T02:57:31.310 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:57:31.311 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:57:31.311 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:57:31.311 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:57:31.311 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:57:31.312 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:57:31.312 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:57:31.312 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:57:31.312 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:57:31.312 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:57:31.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:57:31.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:57:31.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: 2021-07-31T02:57:30 2021-07-31T02:57:31.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: .520471+0000 mgr.x (mgr. 2021-07-31T02:57:31.314 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: 24542) 3013 : cephadm [ERR] 2021-07-31T02:57:31.314 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:57:31.314 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:57:31.314 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:57:31.315 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: yield 2021-07-31T02:57:31.315 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:57:31.315 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:57:31.315 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:57:31.316 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:57:31.316 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:57:31.316 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:57:31.316 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:57:31.316 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:57:31.317 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:57:31.317 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:57:31.317 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:57:31.317 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: 2021-07-31T02:57:31.318 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:57:31.318 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: 2021-07-31T02:57:31.318 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:57:31.318 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:57:31.319 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:57:31.319 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:57:31.319 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:57:31.319 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:57:31.320 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:57:31.320 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:57:31.320 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:57:31.320 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:57:31.320 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:57:31.321 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:57:31.321 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:57:31.321 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:57:31.321 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:57:31.322 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:57:31.322 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:57:31.322 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:57:31.322 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: raise 2021-07-31T02:57:31.322 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:57:31.323 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:57:31.323 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:57:31.323 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:57:31.323 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:57:31.324 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:57:30.528996+0000 mgr.x (mgr.24542) 3014 : cephadm [ERR] 2021-07-31T02:57:31.324 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:57:31.324 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:57:31.324 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: yield 2021-07-31T02:57:31.325 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:57:31.325 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:57:31.325 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:57:31.326 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: audit 2021-07-31T02:57:30.434633+0000 mon.b (mon.2) 427 : audit 2021-07-31T02:57:31.326 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:57:31.326 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: audit 2021-07-31T02:57:30.436272+0000 mon.b (mon.2) 428 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:57:31.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: audit 2021-07-31T02 2021-07-31T02:57:31.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: :57:30.437139+0000 mon.b (mon.2) 429 2021-07-31T02:57:31.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:57:31.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:57 2021-07-31T02:57:31.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: :30.437902+0000 mgr.x (mgr.24542) 3007 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:57:31.328 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: cephadm 2021-07- 2021-07-31T02:57:31.328 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: 31T02:57:30.482328+0000 mgr.x (mgr. 2021-07-31T02:57:31.328 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: 24542) 3008 : cephadm [ERR] 2021-07-31T02:57:31.328 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:57:31.329 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:57:31.329 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: yield 2021-07-31T02:57:31.329 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:57:31.329 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:57:31.329 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:57:31.330 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:57:31.330 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:57:31.330 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:57:31.330 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:57:31.331 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:57:31.331 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:57:31.331 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:57:31.331 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:57:31.332 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:57:30.482743+0000 mgr.x (mgr.24542) 3009 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:57:31.332 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:57: 2021-07-31T02:57:31.332 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: 30.494071+0000 mgr.x (mgr.24542) 3010 : cephadm [ERR] 2021-07-31T02:57:31.332 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: 2021-07-31T02:57:31.333 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:57:31.333 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:57:31.333 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: yield 2021-07-31T02:57:31.333 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:57:31.334 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:57:31.334 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:57:31.334 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:57:31.334 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:57:31.334 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:57:31.335 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:57:31.335 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:57:31.335 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:57:31.335 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:57:31.336 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:57:31.336 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:57:30.494603+0000 mgr.x (mgr.24542) 3011 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:57:31.336 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:57:30. 2021-07-31T02:57:31.336 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: 520012+0000 mgr.x (mgr.24542) 3012 : cephadm [ERR] 2021-07-31T02:57:31.337 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:57:31.337 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:57:31.337 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: yield 2021-07-31T02:57:31.337 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:57:31.337 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:57:31.338 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:57:31.338 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:57:31.338 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:57:31.338 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:57:31.339 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:57:31.339 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:57:31.339 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:57:31.339 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:57:31.340 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:57:31.340 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: 2021-07-31T02:57:31.340 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:57:30.520471+0000 mgr.x (mgr.24542) 3013 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:57:31.340 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:57:31.340 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:57:31.341 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: yield 2021-07-31T02:57:31.341 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:57:31.341 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:57:31.341 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:57:31.342 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:57:31.342 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:57:31.342 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:57:31.342 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:57:31.343 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:57:31.343 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:57:31.343 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:57:31.343 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:57:31.343 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: 2021-07-31T02:57:31.344 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:57:31.344 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: 2021-07-31T02:57:31.344 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:57:31.345 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:57:31.345 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:57:31.345 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:57:31.346 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:57:31.346 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:57:31.346 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:57:31.346 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:57:31.347 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:57:31.347 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:57:30.529430+0000 mgr.x (mgr.24542) 3015 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:57:31.347 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:57:31.347 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:57:31.348 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: yield 2021-07-31T02:57:31.348 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:57:31.348 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:57:31.348 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:57:31.348 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:57:31.349 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:57:31.349 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:57:31.349 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:57:31.349 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:57:31.350 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:57:31.350 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:57:31.350 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:57:31.350 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: 2021-07-31T02:57:31.350 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:57:31.351 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: 2021-07-31T02:57:31.351 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:57:31.351 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:57:31.351 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:57:31.352 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:57:31.352 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:57:31.352 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:57:31.352 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:57:31.353 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:57:31.353 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:57:31.353 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:57:31.353 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:57:31.353 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:57:31.354 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:57:31.354 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:57:31.354 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:57:31.354 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:57:31.355 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:57:31.355 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:57:31.355 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: raise 2021-07-31T02:57:31.355 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:57:31.356 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:57:31.356 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:57:31.356 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:57:31.356 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:57:31.357 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:57:31.357 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:57:31.357 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:57:31.358 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:57:31.358 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:57:31.358 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:57:31.358 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:57:31.358 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:57:31.359 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:57:31.359 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:57:31.359 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:57:31.359 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:57:31.360 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:57:31.360 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:57:31.360 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:57:31.360 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:57:31.361 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:57:31.361 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: raise 2021-07-31T02:57:31.361 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:57:31.361 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:57:31.361 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:57:31.362 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:57:31.362 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:57:31.362 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:57:30.528996+0000 mgr.x (mgr.24542) 3014 : cephadm [ERR] 2021-07-31T02:57:31.362 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:57:31.363 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:57:31.363 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: yield 2021-07-31T02:57:31.363 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:57:31.363 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:57:31.363 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:57:31.364 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:57:31.364 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:57:31.364 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:57:31.364 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:57:31.365 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:57:31.365 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:57:31.365 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:57:31.365 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:57:31.366 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:57:30.529430+0000 mgr.x (mgr.24542) 3015 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:57:31.366 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:57:31.366 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:57:31.366 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: yield 2021-07-31T02:57:31.367 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:57:31.367 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:57:31.367 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:57:31.367 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:57:31.367 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:57:31.368 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:57:31.368 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:57:31.368 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:57:31.368 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:57:31.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:57:31.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:57:31.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: 2021-07-31T02:57:31.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:57:31.370 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: 2021-07-31T02:57:31.370 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:57:31.370 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:57:31.370 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:57:31.370 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:57:31.371 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:57:31.371 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:57:31.371 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:57:31.371 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:57:31.372 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:57:31.372 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:57:31.372 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:57:31.372 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:57:31.372 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:57:31.373 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:57:31.373 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:57:31.373 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:57:31.373 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:57:31.374 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:57:31.374 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: raise 2021-07-31T02:57:31.374 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:57:31.374 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:57:31.375 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:57:31.375 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:57:31.375 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:30 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:57:32.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:31 smithi154 conmon[34553]: cluster 2021-07-31T02:57:30.804934+0000 mgr.x (mgr.24542) 3016 : cluster [DBG] pgmap v4116: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:32.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:31 smithi045 conmon[33655]: cluster 2021-07-31T02:57:30.804934+0000 mgr.x (mgr.24542) 3016 : cluster 2021-07-31T02:57:32.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:31 smithi045 conmon[33655]: [DBG] pgmap v4116: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:32.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:31 smithi045 conmon[39718]: cluster 2021-07-31T02:57:30.804934+0000 mgr.x (mgr. 2021-07-31T02:57:32.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:31 smithi045 conmon[39718]: 24542) 3016 : cluster [DBG] pgmap v4116: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:33.817 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:57:34.095 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:57:34.096 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 72m ago 73m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:57:34.096 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 70m ago 72m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:57:34.096 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 70m ago 76m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:57:34.096 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 72m ago 78m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:57:34.097 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 72m ago 78m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:57:34.097 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 70m ago 76m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:57:34.097 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 72m ago 76m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:57:34.097 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 72m ago 73m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:57:34.098 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 70m ago 73m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:57:34.098 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 72m ago 76m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:57:34.098 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 72m ago 75m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:57:34.098 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 72m ago 75m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:57:34.098 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 72m ago 75m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:57:34.099 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 70m ago 74m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:57:34.099 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 70m ago 74m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:57:34.099 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 70m ago 74m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:57:34.099 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 70m ago 73m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:57:34.100 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 70m ago 73m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:57:34.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:33 smithi154 conmon[34553]: cluster 2021-07-31T02:57:32.805313+0000 mgr.x (mgr.24542) 3017 : cluster [DBG] pgmap v4117: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:34.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:33 smithi045 conmon[39718]: cluster 2021-07-31T02: 2021-07-31T02:57:34.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:33 smithi045 conmon[39718]: 57:32.805313+0000 mgr.x (mgr.24542) 3017 : cluster [DBG] pgmap v4117: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:34.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:33 smithi045 conmon[33655]: cluster 2021-07-31T02:57:32.805313+0000 mgr.x ( 2021-07-31T02:57:34.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:33 smithi045 conmon[33655]: mgr.24542) 3017 : cluster [DBG] pgmap v4117: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:34.430 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:57:34.431 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:57:34.431 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:57:34.431 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:57:34.431 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:57:34.432 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:57:34.432 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:57:34.432 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:57:34.432 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:57:34.433 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:57:34.433 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:57:34.433 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:57:34.433 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:57:34.433 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:57:34.434 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:57:34.434 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:57:34.434 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:57:35.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:34 smithi154 conmon[34553]: audit 2021-07-31T02:57:33.805081+0000 mgr.x (mgr.24542) 3018 : 2021-07-31T02:57:35.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:34 smithi154 conmon[34553]: audit [DBG] from='client.18498 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:57:35.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:34 smithi154 conmon[34553]: audit 2021-07-31T02:57:34.087652+0000 mgr.x (mgr.24542) 3019 : audit [DBG] from='client.18504 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:57:35.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:34 smithi154 conmon[34553]: audit 2021-07-31T02:57:34.430655+0000 mon.c (mon.1) 163 : 2021-07-31T02:57:35.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:34 smithi154 conmon[34553]: audit [DBG] from='client.? 172.21.15.45:0/911602386' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:57:35.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:34 smithi045 conmon[39718]: audit 2021-07-31T02:57:33.805081+0000 2021-07-31T02:57:35.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:34 smithi045 conmon[39718]: mgr.x (mgr.24542) 3018 : audit [DBG] from='client.18498 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:57:35.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:34 smithi045 conmon[39718]: audit 2021-07 2021-07-31T02:57:35.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:34 smithi045 conmon[39718]: -31T02:57:34.087652+0000 mgr.x (mgr.24542) 3019 : audit [DBG] from='client.18504 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:57:35.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:34 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T02:57:35.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:34 smithi045 conmon[39718]: 31T02:57:34.430655+0000 mon.c (mon.1) 163 : audit 2021-07-31T02:57:35.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:34 smithi045 conmon[39718]: [DBG] from='client.? 172.21.15.45:0/911602386' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:57:35.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:34 smithi045 conmon[33655]: audit 2021-07-31T02:57:33.805081+0000 mgr.x (mgr.24542) 3018 : audit [DBG] from='client.18498 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:57:35.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:34 smithi045 conmon[33655]: audit 2021-07-31T02:57:34.087652+0000 mgr.x (mgr.24542) 3019 : audit [DBG] from='client.18504 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:57:35.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:34 smithi045 conmon[33655]: audit 2021-07 2021-07-31T02:57:35.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:34 smithi045 conmon[33655]: -31T02:57:34.430655 2021-07-31T02:57:35.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:34 smithi045 conmon[33655]: +0000 mon.c (mon.1) 163 2021-07-31T02:57:35.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:34 smithi045 conmon[33655]: : audit [DBG] from='client.? 172.21.15.45:0/911602386' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:57:36.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:35 smithi154 conmon[34553]: cluster 2021-07-31T02:57:34.805848+0000 mgr.x (mgr.24542) 3020 : 2021-07-31T02:57:36.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:35 smithi154 conmon[34553]: cluster [DBG] pgmap v4118: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:36.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:35 smithi045 conmon[39718]: cluster 2021-07-31T02:57:34.805848+0000 2021-07-31T02:57:36.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:35 smithi045 conmon[39718]: mgr.x (mgr.24542) 3020 : cluster [DBG] pgmap v4118: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:36.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:35 smithi045 conmon[33655]: cluster 2021-07-31T02:57:34.805848+0000 mgr.x (mgr.24542) 3020 : cluster [DBG] pgmap v4118: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:36.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:35 smithi045 conmon[33655]: 2021-07-31T02:57:38.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:37 smithi154 conmon[34553]: cluster 2021-07-31T02:57:36.806256+0000 mgr.x (mgr.24542) 3021 2021-07-31T02:57:38.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:37 smithi154 conmon[34553]: : cluster [DBG] pgmap v4119: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:38.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:37 smithi045 conmon[39718]: cluster 2021-07-31T02:57:36.806256+0000 mgr.x (mgr 2021-07-31T02:57:38.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:37 smithi045 conmon[39718]: .24542) 3021 : cluster [DBG] pgmap v4119: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:38.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:37 smithi045 conmon[33655]: cluster 2021-07-31T02:57:36.806256+0000 mgr.x (mgr.24542) 3021 : cluster [DBG] pgmap v4119: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:40.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:39 smithi154 conmon[34553]: cluster 2021-07-31T02:57:38.806694+0000 mgr.x (mgr.24542) 3022 : cluster 2021-07-31T02:57:40.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:39 smithi154 conmon[34553]: [DBG] pgmap v4120: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:40.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:39 smithi045 conmon[39718]: cluster 2021-07-31T02:57:38 2021-07-31T02:57:40.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:39 smithi045 conmon[39718]: .806694+0000 mgr.x (mgr.24542) 3022 : cluster [DBG] pgmap v4120: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:40.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:39 smithi045 conmon[33655]: cluster 2021-07-31T02:57:38.806694+0000 mgr.x (mgr.24542) 3022 : cluster 2021-07-31T02:57:40.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:39 smithi045 conmon[33655]: [DBG] pgmap v4120: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:42.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:41 smithi154 conmon[34553]: cluster 2021-07-31T02:57:40.807143+0000 mgr.x (mgr.24542) 3023 : cluster 2021-07-31T02:57:42.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:41 smithi154 conmon[34553]: [DBG] pgmap v4121: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:42.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:41 smithi045 conmon[33655]: cluster 2021-07-31T02:57:40.807143+0000 mgr.x (mgr.24542) 3023 : cluster [DBG] pgmap v4121: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:42.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:41 smithi045 conmon[33655]: 2021-07-31T02:57:42.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:41 smithi045 conmon[39718]: cluster 2021-07-31T02:57:40.807143+0000 mgr.x 2021-07-31T02:57:42.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:41 smithi045 conmon[39718]: (mgr.24542) 3023 : cluster [DBG] pgmap v4121: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:44.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:43 smithi154 conmon[34553]: cluster 2021-07-31T02:57:42.807568+0000 mgr.x (mgr.24542) 3024 : cluster 2021-07-31T02:57:44.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:43 smithi154 conmon[34553]: [DBG] pgmap v4122: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:44.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:43 smithi045 conmon[33655]: cluster 2021-07-31T02:57:42.807568+0000 mgr.x (mgr.24542) 3024 : cluster 2021-07-31T02:57:44.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:43 smithi045 conmon[33655]: [DBG] pgmap v4122: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:44.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:43 smithi045 conmon[39718]: cluster 2021-07-31T02:57:42.807568+0000 mgr.x ( 2021-07-31T02:57:44.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:43 smithi045 conmon[39718]: mgr.24542) 3024 : cluster [DBG] pgmap v4122: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:46.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:45 smithi154 conmon[34553]: cluster 2021-07-31T02:57:44.808056+0000 mgr.x (mgr.24542) 3025 : cluster 2021-07-31T02:57:46.458 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:45 smithi154 conmon[34553]: [DBG] pgmap v4123: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:46.459 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:45 smithi045 conmon[33655]: cluster 2021-07-31T02:57:44.808056+0000 2021-07-31T02:57:46.460 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:45 smithi045 conmon[33655]: mgr.x (mgr.24542) 3025 : cluster [DBG] pgmap v4123: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:46.460 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:45 smithi045 conmon[39718]: cluster 2021-07-31T02:57:44.808056+0000 mgr.x (mgr 2021-07-31T02:57:46.460 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:45 smithi045 conmon[39718]: .24542) 3025 : cluster [DBG] pgmap v4123: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:48.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:47 smithi154 conmon[34553]: cluster 2021-07-31T02:57:46.808566+0000 mgr.x (mgr.24542) 3026 : cluster 2021-07-31T02:57:48.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:47 smithi154 conmon[34553]: [DBG] pgmap v4124: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:48.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:47 smithi045 conmon[33655]: cluster 2021-07-31T02:57:46.808566+0000 mgr.x (mgr.24542) 3026 : cluster [DBG] pgmap v4124: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:48.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:47 smithi045 conmon[39718]: cluster 2021-07-31T02:57:46.808566+0000 mgr.x (mgr.24542) 3026 : 2021-07-31T02:57:48.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:47 smithi045 conmon[39718]: cluster [DBG] pgmap v4124: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:50.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:49 smithi154 conmon[34553]: cluster 2021-07-31T02:57:48.809015+0000 mgr.x (mgr.24542) 3027 : cluster 2021-07-31T02:57:50.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:49 smithi154 conmon[34553]: [DBG] pgmap v4125: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:50.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:49 smithi045 conmon[33655]: cluster 2021-07-31T02:57:48.809015 2021-07-31T02:57:50.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:49 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 3027 : cluster [DBG] pgmap v4125: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:50.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:49 smithi045 conmon[39718]: cluster 2021-07-31T02:57:48.809015+0000 mgr.x (mgr.24542) 2021-07-31T02:57:50.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:49 smithi045 conmon[39718]: 3027 : cluster [DBG] pgmap v4125: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:52.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:51 smithi154 conmon[34553]: cluster 2021-07-31T02:57:50.809590+0000 mgr.x (mgr.24542) 3028 : cluster 2021-07-31T02:57:52.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:51 smithi154 conmon[34553]: [DBG] pgmap v4126: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:52.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:51 smithi045 conmon[33655]: cluster 2021-07-31T02:57:50.809590+0000 mgr.x (mgr.24542) 3028 : cluster 2021-07-31T02:57:52.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:51 smithi045 conmon[33655]: [DBG] pgmap v4126: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:52.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:51 smithi045 conmon[39718]: cluster 2021-07-31T02:57:50.809590+0000 mgr.x (mgr. 2021-07-31T02:57:52.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:51 smithi045 conmon[39718]: 24542) 3028 : cluster [DBG] pgmap v4126: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:54.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:53 smithi154 conmon[34553]: cluster 2021-07-31T02:57:52.810124+0000 mgr.x (mgr.24542) 3029 : cluster 2021-07-31T02:57:54.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:53 smithi154 conmon[34553]: [DBG] pgmap v4127: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:54.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:53 smithi045 conmon[39718]: cluster 2021-07-31T02:57:52.810124+0000 mgr.x (mgr.24542) 3029 : cluster [DBG] pgmap v4127: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:54.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:53 smithi045 conmon[33655]: cluster 2021-07-31T02:57:52.810124+0000 mgr.x (mgr.24542) 2021-07-31T02:57:54.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:53 smithi045 conmon[33655]: 3029 : cluster [DBG] pgmap v4127: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:56.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:55 smithi154 conmon[34553]: cluster 2021-07-31T02:57:54.810661+0000 mgr.x (mgr. 2021-07-31T02:57:56.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:55 smithi154 conmon[34553]: 24542) 3030 : cluster [DBG] pgmap v4128: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:56.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:55 smithi045 conmon[33655]: cluster 2021-07-31T02:57:54.810661+0000 mgr.x (mgr.24542) 3030 2021-07-31T02:57:56.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:55 smithi045 conmon[33655]: : cluster [DBG] pgmap v4128: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:56.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:55 smithi045 conmon[39718]: cluster 2021-07-31T02:57:54.810661+0000 mgr.x (mgr.24542) 3030 : cluster [DBG] 2021-07-31T02:57:56.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:55 smithi045 conmon[39718]: pgmap v4128: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:58.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:57 smithi154 conmon[34553]: cluster 2021-07-31T02:57:56.811107+0000 mgr.x (mgr.24542) 3031 : cluster 2021-07-31T02:57:58.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:57 smithi154 conmon[34553]: [DBG] pgmap v4129: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:58.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:57 smithi045 conmon[39718]: cluster 2021-07-31T02:57:56.811107+0000 mgr.x (mgr.24542) 3031 : cluster [DBG] pgmap v4129: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:57:58.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:57 smithi045 conmon[33655]: cluster 2021-07-31T02:57:56.811107+0000 mgr.x (mgr.24542) 3031 : cluster [DBG] 2021-07-31T02:57:58.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:57 smithi045 conmon[33655]: pgmap v4129: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:00.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:59 smithi154 conmon[34553]: cluster 2021-07-31T02:57:58.811618+0000 mgr.x (mgr.24542) 3032 : cluster 2021-07-31T02:58:00.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:57:59 smithi154 conmon[34553]: [DBG] pgmap v4130: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:00.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:57:59 smithi045 conmon[39718]: cluster 2021-07-31T02:57:58.811618+0000 mgr.x (mgr.24542) 3032 : cluster [DBG] pgmap v4130: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:00.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:59 smithi045 conmon[33655]: cluster 2021-07-31T02:57:58.811618+0000 mgr.x (mgr.24542) 3032 : cluster 2021-07-31T02:58:00.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:57:59 smithi045 conmon[33655]: [DBG] pgmap v4130: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:02.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:01 smithi154 conmon[34553]: cluster 2021-07-31T02:58:00.812347+0000 mgr.x (mgr.24542) 3033 : cluster [DBG] 2021-07-31T02:58:02.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:01 smithi154 conmon[34553]: pgmap v4131: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:02.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:01 smithi045 conmon[33655]: cluster 2021-07-31T02:58:00.812347+0000 mgr.x (mgr.24542) 3033 : cluster [DBG] 2021-07-31T02:58:02.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:01 smithi045 conmon[33655]: pgmap v4131: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:02.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:01 smithi045 conmon[39718]: cluster 2021-07-31T02:58:00.812347+0000 mgr.x (mgr.24542) 3033 : cluster [DBG] pgmap v4131: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:04.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:03 smithi154 conmon[34553]: cluster 2021-07-31T02:58:02.812826+0000 mgr.x (mgr.24542) 3034 : cluster [DBG] 2021-07-31T02:58:04.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:03 smithi154 conmon[34553]: pgmap v4132: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:04.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:03 smithi045 conmon[39718]: cluster 2021-07-31T02:58:02.812826 2021-07-31T02:58:04.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:03 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 3034 : cluster [DBG] pgmap v4132: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:04.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:03 smithi045 conmon[33655]: cluster 2021-07-31T02:58:02.812826+0000 mgr.x (mgr.24542) 3034 : cluster [DBG] 2021-07-31T02:58:04.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:03 smithi045 conmon[33655]: pgmap v4132: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:04.741 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:58:05.017 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:58:05.018 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 72m ago 73m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:58:05.018 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 70m ago 73m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:58:05.018 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 70m ago 76m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:58:05.019 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 72m ago 79m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:58:05.019 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 72m ago 79m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:58:05.020 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 70m ago 77m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:58:05.020 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 72m ago 77m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:58:05.020 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 72m ago 73m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:58:05.021 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 70m ago 73m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:58:05.021 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 72m ago 76m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:58:05.021 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 72m ago 76m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:58:05.022 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 72m ago 76m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:58:05.022 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 72m ago 75m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:58:05.023 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 70m ago 75m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:58:05.023 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 70m ago 75m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:58:05.023 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 70m ago 74m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:58:05.024 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 70m ago 74m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:58:05.024 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 70m ago 74m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:58:05.351 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:58:05.352 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:58:05.352 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:58:05.352 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:58:05.352 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:58:05.353 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:58:05.353 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:58:05.353 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:58:05.353 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:58:05.354 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:58:05.354 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:58:05.354 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:58:05.354 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:58:05.354 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:58:05.355 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:58:05.355 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:58:05.355 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:58:06.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:05 smithi154 conmon[34553]: audit 2021-07-31T02:58:04.729486+0000 mgr.x (mgr.24542) 3035 : audit 2021-07-31T02:58:06.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:05 smithi154 conmon[34553]: [DBG] from='client.28456 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:58:06.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:05 smithi154 conmon[34553]: cluster 2021-07-31T02:58:04.813329+0000 mgr.x (mgr.24542) 3036 : cluster [DBG] pgmap v4133: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:06.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:05 smithi154 conmon[34553]: audit 2021-07-31T02:58:06.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:05 smithi154 conmon[34553]: 2021-07-31T02:58:05.009563+0000 mgr.x (mgr.24542) 3037 : audit 2021-07-31T02:58:06.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:05 smithi154 conmon[34553]: [DBG] from='client.28462 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:58:06.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:05 smithi154 conmon[34553]: audit 2021-07-31T02:58: 2021-07-31T02:58:06.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:05 smithi154 conmon[34553]: 05.351734+0000 mon.a (mon.0) 3230 : audit 2021-07-31T02:58:06.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:05 smithi154 conmon[34553]: [DBG] from='client.? 172.21.15.45:0/3995085089' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:58:06.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:05 smithi045 conmon[33655]: audit 2021-07-31T02:58:04.729486+0000 mgr.x (mgr.24542) 3035 2021-07-31T02:58:06.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:05 smithi045 conmon[33655]: : audit [DBG] from='client.28456 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:58:06.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:05 smithi045 conmon[33655]: cluster 2021-07-31T02:58:04.813329+0000 mgr.x (mgr.24542) 3036 : cluster [DBG] pgmap v4133: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:06.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:05 smithi045 conmon[33655]: audit 2021 2021-07-31T02:58:06.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:05 smithi045 conmon[33655]: -07-31T02:58:05.009563+0000 mgr.x (mgr.24542) 3037 : audit [DBG] from='client.28462 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:58:06.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:05 smithi045 conmon[33655]: 2021-07-31T02:58:06.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:05 smithi045 conmon[33655]: audit 2021-07 2021-07-31T02:58:06.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:05 smithi045 conmon[33655]: -31T02:58:05.351734 2021-07-31T02:58:06.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:05 smithi045 conmon[33655]: +0000 mon.a (mon.0) 3230 : audit 2021-07-31T02:58:06.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:05 smithi045 conmon[33655]: [DBG] from='client.? 172.21.15.45:0/3995085089' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:58:06.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:05 smithi045 conmon[39718]: audit 2021-07-31T02:58:04.729486+0000 mgr.x 2021-07-31T02:58:06.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:05 smithi045 conmon[39718]: (mgr.24542) 3035 : audit [DBG] from='client.28456 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:58:06.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:05 smithi045 conmon[39718]: cluster 2021-07-31T 2021-07-31T02:58:06.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:05 smithi045 conmon[39718]: 02:58:04.813329+0000 mgr.x (mgr.24542) 3036 : cluster [DBG] pgmap v4133: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:06.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:05 smithi045 conmon[39718]: audit 2021-07-31T02:58:05.009563+0000 mgr.x 2021-07-31T02:58:06.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:05 smithi045 conmon[39718]: (mgr.24542) 3037 : audit [DBG] from='client.28462 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:58:06.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:05 smithi045 conmon[39718]: audit 2021-07-31T02:58:05.351734+0000 mon.a (mon. 2021-07-31T02:58:06.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:05 smithi045 conmon[39718]: 0) 3230 : audit [DBG] from='client.? 172.21.15.45:0/3995085089' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:58:08.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:07 smithi154 conmon[34553]: cluster 2021-07-31T02:58:06.813767+0000 mgr.x (mgr.24542) 3038 : cluster 2021-07-31T02:58:08.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:07 smithi154 conmon[34553]: [DBG] pgmap v4134: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:08.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:07 smithi045 conmon[33655]: cluster 2021-07-31T02:58:06.813767+0000 mgr.x (mgr.24542 2021-07-31T02:58:08.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:07 smithi045 conmon[33655]: ) 3038 : cluster [DBG] pgmap v4134: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:08.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:07 smithi045 conmon[39718]: cluster 2021-07-31T02:58:06.813767+0000 mgr.x 2021-07-31T02:58:08.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:07 smithi045 conmon[39718]: (mgr.24542) 3038 : cluster [DBG] pgmap v4134: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:10.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:09 smithi154 conmon[34553]: cluster 2021-07-31T02:58:08.814233+0000 mgr.x (mgr.24542) 3039 : cluster [DBG] 2021-07-31T02:58:10.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:09 smithi154 conmon[34553]: pgmap v4135: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:10.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:09 smithi045 conmon[33655]: cluster 2021-07-31T02:58:08.814233 2021-07-31T02:58:10.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:09 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 3039 : cluster [DBG] pgmap v4135: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:10.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:09 smithi045 conmon[39718]: cluster 2021-07-31T02:58:08.814233+0000 mgr.x (mgr 2021-07-31T02:58:10.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:09 smithi045 conmon[39718]: .24542) 3039 : cluster [DBG] pgmap v4135: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:12.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:11 smithi154 conmon[34553]: cluster 2021-07-31T02:58:10.814635+0000 mgr.x (mgr.24542) 3040 : cluster [DBG] 2021-07-31T02:58:12.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:11 smithi154 conmon[34553]: pgmap v4136: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:12.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:11 smithi045 conmon[39718]: cluster 2021-07-31T02:58:10.814635+0000 mgr.x (mgr.24542) 2021-07-31T02:58:12.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:11 smithi045 conmon[39718]: 3040 : cluster [DBG] pgmap v4136: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:12.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:11 smithi045 conmon[33655]: cluster 2021-07-31T02:58:10. 2021-07-31T02:58:12.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:11 smithi045 conmon[33655]: 814635+0000 mgr.x (mgr.24542) 3040 : cluster [DBG] pgmap v4136: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:14.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:13 smithi154 conmon[34553]: cluster 2021-07-31T02:58:12.815054+0000 mgr.x (mgr.24542) 3041 : 2021-07-31T02:58:14.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:13 smithi154 conmon[34553]: cluster [DBG] pgmap v4137: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:14.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:13 smithi045 conmon[39718]: cluster 2021-07-31T02:58:12.815054+0000 mgr.x (mgr 2021-07-31T02:58:14.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:13 smithi045 conmon[39718]: .24542) 3041 : cluster [DBG] pgmap v4137: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:14.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:13 smithi045 conmon[33655]: cluster 2021-07-31T02:58:12.815054+0000 2021-07-31T02:58:14.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:13 smithi045 conmon[33655]: mgr.x (mgr.24542) 3041 : cluster [DBG] pgmap v4137: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:16.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:15 smithi154 conmon[34553]: cluster 2021-07-31T02:58:14.815440+0000 mgr.x (mgr.24542) 3042 : cluster [DBG] 2021-07-31T02:58:16.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:15 smithi154 conmon[34553]: pgmap v4138: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:16.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:15 smithi045 conmon[33655]: cluster 2021-07-31T02:58:14.815440+0000 mgr.x (mgr.24542 2021-07-31T02:58:16.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:15 smithi045 conmon[33655]: ) 3042 : cluster [DBG] pgmap v4138: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:16.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:15 smithi045 conmon[39718]: cluster 2021-07-31T02:58:14.815440+0000 mgr.x ( 2021-07-31T02:58:16.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:15 smithi045 conmon[39718]: mgr.24542) 3042 : cluster [DBG] pgmap v4138: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:18.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:17 smithi154 conmon[34553]: cluster 2021-07-31T02:58:16.815753+0000 mgr.x (mgr.24542) 3043 : cluster [DBG] 2021-07-31T02:58:18.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:17 smithi154 conmon[34553]: pgmap v4139: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:18.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:17 smithi045 conmon[33655]: cluster 2021-07-31T02:58:16.815753+0000 mgr.x (mgr.24542) 3043 : cluster [DBG] pgmap v4139: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:18.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:17 smithi045 conmon[33655]: 2021-07-31T02:58:18.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:17 smithi045 conmon[39718]: cluster 2021-07-31T02:58:18.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:17 smithi045 conmon[39718]: 2021-07-31T02:58:18.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:17 smithi045 conmon[39718]: 2021 2021-07-31T02:58:18.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:17 smithi045 conmon[39718]: -07-31T02:58:16.815753+0000 2021-07-31T02:58:18.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:17 smithi045 conmon[39718]: mgr.x (mgr.24542) 3043 : cluster [DBG] pgmap v4139: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:20.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:19 smithi154 conmon[34553]: cluster 2021-07-31T02:58:18.816292+0000 mgr.x (mgr.24542) 3044 : cluster 2021-07-31T02:58:20.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:19 smithi154 conmon[34553]: [DBG] pgmap v4140: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:20.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:19 smithi045 conmon[39718]: cluster 2021-07-31T02:58:18.816292 2021-07-31T02:58:20.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:19 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 3044 : cluster [DBG] pgmap v4140: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:20.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:19 smithi045 conmon[33655]: cluster 2021-07-31T02:58:18.816292 2021-07-31T02:58:20.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:19 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 3044 : cluster [DBG] pgmap v4140: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:22.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:21 smithi154 conmon[34553]: cluster 2021-07-31T02:58:20.816841+0000 mgr.x (mgr.24542) 3045 : cluster [DBG] pgmap v4141: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:22.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:21 smithi045 conmon[39718]: cluster 2021-07-31T02:58:20.816841+0000 mgr.x 2021-07-31T02:58:22.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:21 smithi045 conmon[39718]: (mgr.24542) 3045 : cluster [DBG] pgmap v4141: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:22.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:21 smithi045 conmon[33655]: cluster 2021-07-31T02:58:20.816841+0000 mgr.x (mgr.24542) 2021-07-31T02:58:22.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:21 smithi045 conmon[33655]: 3045 : cluster [DBG] pgmap v4141: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:24.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:23 smithi154 conmon[34553]: cluster 2021-07-31T02:58:22.817263+0000 mgr.x (mgr.24542) 3046 : cluster [DBG] pgmap v4142: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:24.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:23 smithi045 conmon[39718]: cluster 2021-07-31T02:58:22.817263+0000 mgr.x 2021-07-31T02:58:24.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:23 smithi045 conmon[39718]: (mgr.24542) 3046 : cluster [DBG] pgmap v4142: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:24.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:23 smithi045 conmon[33655]: cluster 2021-07-31T02:58:22.817263+0000 mgr.x (mgr.24542) 2021-07-31T02:58:24.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:23 smithi045 conmon[33655]: 3046 : cluster [DBG] pgmap v4142: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:26.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:25 smithi154 conmon[34553]: cluster 2021-07-31T02:58:24.817726+0000 mgr.x (mgr.24542) 3047 : cluster [DBG] pgmap v4143: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:26.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:25 smithi045 conmon[39718]: cluster 2021-07-31T02:58:24.817726+0000 mgr.x (mgr.24542 2021-07-31T02:58:26.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:25 smithi045 conmon[39718]: ) 3047 : cluster [DBG] pgmap v4143: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:26.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:25 smithi045 conmon[33655]: cluster 2021-07-31T02:58:24.817726+0000 mgr.x (mgr.24542) 3047 : cluster [DBG] pgmap v4143: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:28.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:27 smithi154 conmon[34553]: cluster 2021-07-31T02:58:26.818169+0000 mgr.x (mgr.24542) 3048 : cluster [DBG] pgmap v4144: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:28.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:27 smithi045 conmon[39718]: cluster 2021-07-31T02:58:26.818169+0000 2021-07-31T02:58:28.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:27 smithi045 conmon[39718]: mgr.x (mgr.24542) 3048 : cluster [DBG] pgmap v4144: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:28.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:27 smithi045 conmon[33655]: cluster 2021-07-31T02:58:26.818169+0000 mgr.x (mgr.24542) 3048 : 2021-07-31T02:58:28.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:27 smithi045 conmon[33655]: cluster [DBG] pgmap v4144: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:28 smithi154 conmon[34553]: audit 2021-07-31T02:58:28.219674+0000 mon.b ( 2021-07-31T02:58:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:28 smithi154 conmon[34553]: mon.2) 430 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:58:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:28 smithi154 conmon[34553]: audit 2021-07-31T02:58:28.219954+0000 mon.b (mon.2) 431 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:58:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:28 smithi154 conmon[34553]: audit 2021-07-31T02:58:28.220184+0000 mon.a (mon.0) 3231 : audit 2021-07-31T02:58:29.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:28 smithi154 conmon[34553]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:58:29.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:28 smithi154 conmon[34553]: audit 2021-07-31T 2021-07-31T02:58:29.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:28 smithi154 conmon[34553]: 02:58:28.220566+0000 mon.a (mon.0 2021-07-31T02:58:29.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:28 smithi154 conmon[34553]: ) 3232 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:58:29.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:28 smithi045 conmon[39718]: audit 2021-07-31T02:58:28.219674+0000 mon.b (mon.2) 430 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:58:29.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:28 smithi045 conmon[39718]: audit 2021-07-31T02 2021-07-31T02:58:29.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:28 smithi045 conmon[39718]: :58:28.219954+0000 mon.b (mon 2021-07-31T02:58:29.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:28 smithi045 conmon[39718]: .2) 431 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:58:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:28 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T02:58:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:28 smithi045 conmon[39718]: T02:58:28.220184+0000 mon.a 2021-07-31T02:58:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:28 smithi045 conmon[39718]: (mon.0) 3231 : audit 2021-07-31T02:58:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:28 smithi045 conmon[39718]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:58:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:28 smithi045 conmon[39718]: audit 2021-07-31T02:58:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:28 smithi045 conmon[39718]: 2021-07-31T02 2021-07-31T02:58:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:28 smithi045 conmon[39718]: :58:28.220566+0000 2021-07-31T02:58:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:28 smithi045 conmon[39718]: mon.a (mon.0) 3232 : 2021-07-31T02:58:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:28 smithi045 conmon[39718]: audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:58:29.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:28 smithi045 conmon[33655]: audit 2021-07-31T02:58:28.219674+0000 mon.b (mon.2) 2021-07-31T02:58:29.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:28 smithi045 conmon[33655]: 430 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:58:29.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:28 smithi045 conmon[33655]: audit 2021-07-31T02:58:28.219954+0000 mon.b (mon.2) 431 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:58:29.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:28 smithi045 conmon[33655]: audit 2021-07-31T02:58:29.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:28 smithi045 conmon[33655]: 2021-07-31T02:58 2021-07-31T02:58:29.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:28 smithi045 conmon[33655]: :28.220184+0000 mon.a ( 2021-07-31T02:58:29.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:28 smithi045 conmon[33655]: mon.0) 3231 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:58:29.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:28 smithi045 conmon[33655]: 2021-07-31T02:58:29.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:28 smithi045 conmon[33655]: audit 2021-07-31T02:58:29.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:28 smithi045 conmon[33655]: 2021-07-31T02: 2021-07-31T02:58:29.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:28 smithi045 conmon[33655]: 58:28.220566+0000 mon.a 2021-07-31T02:58:29.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:28 smithi045 conmon[33655]: (mon.0) 3232 : audit [INF] 2021-07-31T02:58:29.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:28 smithi045 conmon[33655]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:58:30.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:29 smithi154 conmon[34553]: cluster 2021-07-31T02:58:28.818632+0000 mgr.x (mgr.24542) 3049 : cluster [DBG] pgmap v4145: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:30.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:29 smithi045 conmon[39718]: cluster 2021-07-31T02:58:28 2021-07-31T02:58:30.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:29 smithi045 conmon[39718]: .818632+0000 mgr.x (mgr.24542) 3049 : cluster [DBG] pgmap v4145: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:30.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:29 smithi045 conmon[33655]: cluster 2021-07-31T02:58:28.818632 2021-07-31T02:58:30.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:29 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 3049 : cluster [DBG] pgmap v4145: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:30.890 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: debug 2021-07-31T02:58:30.576+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:58:30.890 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:58:30.890 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:58:30.890 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: yield 2021-07-31T02:58:30.891 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:58:30.891 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:58:30.891 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:58:30.891 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:58:30.892 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:58:30.892 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:58:30.892 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:58:30.892 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:58:30.892 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:58:30.893 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:58:30.893 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:58:30.893 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: debug 2021-07-31T02:58:30.587+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:58:30.893 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:58:30.893 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:58:30.894 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: yield 2021-07-31T02:58:30.894 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:58:30.894 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:58:30.894 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:58:30.895 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:58:30.895 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:58:30.895 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:58:30.895 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:58:30.895 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:58:30.896 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:58:30.896 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:58:30.896 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:58:30.896 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: debug 2021-07-31T02:58:30.613+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:58:30.897 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:58:30.897 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:58:30.897 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: yield 2021-07-31T02:58:30.897 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:58:30.897 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:58:30.898 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:58:30.898 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:58:30.898 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:58:30.899 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:58:30.899 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:58:30.899 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:58:30.899 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:58:30.900 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:58:30.900 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:58:30.900 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: debug 2021-07-31T02:58:30.613+0000 7fcb93160700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:58:30.900 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:58:30.901 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:58:30.901 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: yield 2021-07-31T02:58:30.901 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:58:30.901 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:58:30.901 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:58:30.902 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:58:30.902 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:58:30.902 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:58:30.902 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:58:30.903 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:58:30.903 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:58:30.903 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:58:30.903 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:58:30.903 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: 2021-07-31T02:58:30.904 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:58:30.904 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: 2021-07-31T02:58:30.904 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:58:30.904 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:58:30.904 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:58:30.905 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:58:30.905 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:58:30.905 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:58:30.905 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:58:30.906 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:58:30.906 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:58:30.906 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:58:30.906 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:58:30.906 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:58:30.907 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:58:30.907 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:58:30.907 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:58:30.907 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:58:30.908 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:58:30.908 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:58:30.908 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: raise 2021-07-31T02:58:30.908 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:58:30.908 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:58:30.909 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:58:30.909 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:58:30.909 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:58:30.909 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: debug 2021-07-31T02:58:30.622+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:58:30.909 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:58:30.910 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:58:30.910 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: yield 2021-07-31T02:58:30.910 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:58:30.910 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:58:30.911 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:58:30.911 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:58:30.911 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:58:30.911 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:58:30.911 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:58:30.912 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:58:30.912 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:58:30.912 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:58:30.912 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:58:30.912 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: debug 2021-07-31T02:58:30.622+0000 7fcb9295f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:58:30.913 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:58:30.913 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:58:30.913 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: yield 2021-07-31T02:58:30.913 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:58:30.914 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:58:30.914 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:58:30.914 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:58:30.914 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:58:30.914 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:58:30.915 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:58:30.915 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:58:30.915 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:58:30.915 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:58:30.916 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:58:30.916 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: 2021-07-31T02:58:30.916 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:58:30.916 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: 2021-07-31T02:58:30.916 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:58:30.917 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:58:30.917 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:58:30.917 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:58:30.917 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:58:30.917 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:58:30.918 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:58:30.918 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:58:30.918 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:58:30.918 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:58:30.919 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:58:30.919 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:58:30.919 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:58:30.919 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:58:30.919 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:58:30.920 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:58:30.920 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:58:30.920 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:58:30.920 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: raise 2021-07-31T02:58:30.920 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:58:30.921 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:58:30.921 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:58:30.921 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:58:30.921 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:58:31.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: audit 2021-07-31T02:58:30.530271+0000 mon.b (mon.2 2021-07-31T02:58:31.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: ) 432 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:58:31.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: audit 2021-07-31T02:58:30.531680+0000 mon.b (mon.2) 433 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:58:31.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: audit 2021-07-31T02:58:30.532589+0000 mon.b (mon.2) 434 : audit 2021-07-31T02:58:31.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:58:31.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: cephadm 2021-07-31T02: 2021-07-31T02:58:31.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: 58:30.533284+0000 mgr.x (mgr.24542) 2021-07-31T02:58:31.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: 3050 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:58:31.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:58:31.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: 2021-07-31T02:58:30.577059+0000 2021-07-31T02:58:31.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: mgr.x (mgr.24542) 3051 : cephadm [ERR] 2021-07-31T02:58:31.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:58:31.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:58:31.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: yield 2021-07-31T02:58:31.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:58:31.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:58:31.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:58:31.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:58:31.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:58:31.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:58:31.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:58:31.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:58:31.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:58:31.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:58:31.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:58:31.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: 2021-07-31T02:58:31.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:58:30.577569+0000 mgr.x (mgr.24542) 3052 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:58:31.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: cephadm 2021-07 2021-07-31T02:58:31.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: -31T02:58:30.588597+0000 mgr.x (mgr.24542) 3053 : cephadm 2021-07-31T02:58:31.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: [ERR] 2021-07-31T02:58:31.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:58:31.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:58:31.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: yield 2021-07-31T02:58:31.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:58:31.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:58:31.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:58:31.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:58:31.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:58:31.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:58:31.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:58:31.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:58:31.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:58:31.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:58:31.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:58:31.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:58:30.589068+0000 mgr.x (mgr.24542) 3054 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:58:31.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: cephadm 2021-07-31T 2021-07-31T02:58:31.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: 02:58:30.614013+0000 mgr.x (mgr. 2021-07-31T02:58:31.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: 24542) 3055 : cephadm [ERR] 2021-07-31T02:58:31.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:58:31.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:58:31.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: yield 2021-07-31T02:58:31.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:58:31.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:58:31.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:58:31.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:58:31.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:58:31.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:58:31.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:58:31.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:58:31.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:58:31.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:58:31.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:58:31.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: 2021-07-31T02:58:31.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:58:30.614429+0000 mgr.x (mgr.24542) 3056 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:58:31.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:58:31.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:58:31.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: yield 2021-07-31T02:58:31.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:58:31.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:58:31.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:58:31.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:58:31.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:58:31.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:58:31.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:58:31.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:58:31.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:58:31.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:58:31.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:58:31.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: 2021-07-31T02:58:31.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:58:31.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: 2021-07-31T02:58:31.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:58:31.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:58:31.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:58:31.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:58:31.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:58:31.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:58:31.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:58:31.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:58:31.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:58:31.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:58:31.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:58:31.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:58:31.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:58:31.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:58:31.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:58:31.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:58:31.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:58:31.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:58:31.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: raise 2021-07-31T02:58:31.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:58:31.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:58:31.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:58:31.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:58:31.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:58:31.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: 2021-07-31T02:58:31.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:58:30.623035+0000 mgr.x (mgr.24542) 3057 : cephadm [ERR] 2021-07-31T02:58:31.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:58:31.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:58:31.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: yield 2021-07-31T02:58:31.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:58:31.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:58:31.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:58:31.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:58:31.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:58:31.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:58:31.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:58:31.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:58:31.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:58:31.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:58:31.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:58:31.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: audit 2021-07-31T02:58:30.530271+0000 mon.b (mon.2) 432 : audit [DBG] 2021-07-31T02:58:31.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:58:31.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: audit 2021-07-31T02:58:30.531680+0000 mon.b (mon.2) 433 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:58:31.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: audit 2021-07-31T02:58:30.532589+0000 mon.b (mon.2) 434 : 2021-07-31T02:58:31.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:58:31.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:58:30.533284+0000 2021-07-31T02:58:31.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: mgr.x (mgr.24542) 3050 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:58:31.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:58:31.283 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: 2021-07-31T02:58:30.577059+0000 mgr.x (mgr.24542) 3051 : cephadm [ERR] 2021-07-31T02:58:31.283 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: 2021-07-31T02:58:31.283 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:58:31.283 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:58:31.283 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: yield 2021-07-31T02:58:31.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:58:31.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:58:31.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:58:31.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:58:31.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:58:31.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:58:31.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:58:31.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:58:31.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:58:31.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:58:31.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:58:31.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:58:31.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: 2021-07-31T02:58:30.577569+0000 mgr.x (mgr.24542) 3052 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:58:31.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:58:30.588597+0000 mgr.x (mgr.24542) 3053 : cephadm [ERR] 2021-07-31T02:58:31.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:58:31.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:58:31.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: yield 2021-07-31T02:58:31.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:58:31.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:58:31.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:58:31.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:58:31.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:58:31.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:58:31.289 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: audit 2021-07-31T02:58:30.530271+0000 mon.b ( 2021-07-31T02:58:31.289 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: mon.2) 432 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:58:31.290 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: audit 2021-07-31T02:58:30.531680+0000 2021-07-31T02:58:31.290 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: mon.b (mon.2) 433 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:58:31.290 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T02:58:31.290 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: 02:58:30.532589+0000 mon.b (mon 2021-07-31T02:58:31.290 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: .2) 434 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:58:31.291 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: cephadm 2021 2021-07-31T02:58:31.291 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: -07-31T02:58:30. 2021-07-31T02:58:31.291 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: 533284+0000 mgr.x (mgr.24542) 3050 : 2021-07-31T02:58:31.291 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:58:31.292 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: cephadm 2021-07 2021-07-31T02:58:31.292 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: -31T02:58:30.577059+0000 2021-07-31T02:58:31.292 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: mgr.x (mgr.24542) 3051 : cephadm [ERR] 2021-07-31T02:58:31.292 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: 2021-07-31T02:58:31.293 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:58:31.293 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:58:31.293 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: yield 2021-07-31T02:58:31.293 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:58:31.293 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:58:31.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:58:31.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:58:31.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:58:31.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:58:31.295 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:58:31.295 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:58:31.295 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:58:31.295 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:58:31.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:58:31.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:58:30.577569+0000 mgr.x (mgr.24542) 3052 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:58:31.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:58: 2021-07-31T02:58:31.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: 30.588597+0000 mgr.x (mgr.24542) 3053 : cephadm [ERR] 2021-07-31T02:58:31.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: 2021-07-31T02:58:31.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:58:31.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:58:31.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: yield 2021-07-31T02:58:31.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:58:31.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:58:31.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:58:31.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:58:31.298 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:58:30.623431+0000 mgr.x (mgr.24542) 3058 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:58:31.298 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:58:31.299 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:58:31.299 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: yield 2021-07-31T02:58:31.299 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:58:31.299 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:58:31.300 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:58:31.300 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:58:31.300 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:58:31.300 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:58:31.300 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:58:31.301 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:58:31.301 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:58:31.301 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: task.result() 2021-07-31T02:58:31.301 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:58:31.302 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: 2021-07-31T02:58:31.302 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:58:31.302 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: 2021-07-31T02:58:31.302 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:58:31.302 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:58:31.303 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:58:31.303 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:58:31.303 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:58:31.303 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:58:31.304 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:58:31.304 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:58:31.304 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:58:31.304 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:58:31.304 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:58:31.305 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:58:31.305 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:58:31.305 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:58:31.305 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:58:31.306 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:58:31.306 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:58:31.306 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:58:31.306 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: raise 2021-07-31T02:58:31.306 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:58:31.307 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:58:31.307 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:58:31.307 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:58:31.307 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:30 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:58:31.308 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:58:31.308 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:58:31.308 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:58:31.308 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:58:31.309 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:58:31.309 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:58:30.589068+0000 mgr.x (mgr.24542) 3054 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:58:31.309 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:58:30.614013+0000 mgr.x (mgr.24542) 3055 : cephadm [ERR] 2021-07-31T02:58:31.309 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:58:31.310 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:58:31.310 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: yield 2021-07-31T02:58:31.310 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:58:31.310 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:58:31.311 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:58:31.311 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:58:31.311 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:58:31.311 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:58:31.311 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:58:31.312 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:58:31.312 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:58:31.312 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:58:31.312 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:58:31.313 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:58:30.614429+0000 mgr.x (mgr.24542) 3056 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:58:31.313 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:58:31.313 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:58:31.313 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: yield 2021-07-31T02:58:31.313 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:58:31.314 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:58:31.314 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:58:31.314 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:58:31.314 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:58:31.315 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:58:31.315 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:58:31.315 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:58:31.315 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:58:31.315 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:58:31.316 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:58:31.316 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: 2021-07-31T02:58:31.316 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:58:31.316 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: 2021-07-31T02:58:31.317 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:58:31.317 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:58:31.317 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:58:31.317 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:58:31.318 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:58:31.318 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:58:31.318 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:58:31.318 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:58:31.319 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:58:31.319 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:58:31.319 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:58:30.589068+0000 mgr.x (mgr.24542) 3054 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:58:31.319 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:58:30. 2021-07-31T02:58:31.320 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: 614013+0000 mgr.x (mgr.24542) 3055 : cephadm [ERR] 2021-07-31T02:58:31.320 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: 2021-07-31T02:58:31.320 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:58:31.320 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:58:31.320 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: yield 2021-07-31T02:58:31.321 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:58:31.321 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:58:31.321 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:58:31.321 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:58:31.322 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:58:31.322 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:58:31.322 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:58:31.322 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:58:31.322 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:58:31.323 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:58:31.323 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:58:31.323 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:58:30.614429+0000 mgr.x (mgr.24542) 3056 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:58:31.323 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:58:31.323 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:58:31.324 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: yield 2021-07-31T02:58:31.324 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:58:31.324 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:58:31.324 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:58:31.325 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:58:31.325 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:58:31.325 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:58:31.325 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:58:31.325 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:58:31.326 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:58:31.326 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:58:31.326 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:58:31.326 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: 2021-07-31T02:58:31.326 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:58:31.327 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: 2021-07-31T02:58:31.327 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:58:31.327 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:58:31.327 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:58:31.327 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:58:31.328 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:58:31.328 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:58:31.328 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:58:31.328 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:58:31.329 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:58:31.329 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:58:31.329 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:58:31.329 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:58:31.329 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:58:31.330 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:58:31.330 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:58:31.330 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:58:31.330 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:58:31.331 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:58:31.331 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: raise 2021-07-31T02:58:31.331 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:58:31.331 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:58:31.331 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:58:31.332 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:58:31.332 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:58:31.332 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:58:30 2021-07-31T02:58:31.332 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: .623035+0000 mgr.x (mgr.24542) 3057 : cephadm [ERR] 2021-07-31T02:58:31.332 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:58:31.333 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:58:31.333 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: yield 2021-07-31T02:58:31.333 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:58:31.333 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:58:31.334 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:58:31.334 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:58:31.334 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:58:31.334 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:58:31.334 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:58:31.335 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:58:31.335 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:58:31.335 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:58:31.336 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:58:31.336 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:58:31.336 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:58:31.336 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:58:31.337 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:58:31.337 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:58:31.337 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:58:31.337 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:58:31.337 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:58:31.338 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:58:31.338 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:58:31.338 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:58:31.338 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:58:31.339 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:58:31.339 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:58:31.339 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: raise 2021-07-31T02:58:31.339 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:58:31.339 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:58:31.340 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:58:31.340 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:58:31.340 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:58:31.340 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: cephadm 2021-07-31T 2021-07-31T02:58:31.341 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: 02:58:30.623035+0000 mgr.x (mgr.24542) 3057 : cephadm [ERR] 2021-07-31T02:58:31.341 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:58:31.341 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:58:31.341 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: yield 2021-07-31T02:58:31.341 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:58:31.342 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:58:31.342 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:58:31.342 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:58:31.342 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:58:31.343 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:58:31.343 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:58:31.343 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:58:31.343 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:58:31.344 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:58:31.344 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:58:31.344 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: cephadm 2021-07-31T02:58:30.623431+0000 mgr.x (mgr.24542) 3058 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:58:31.344 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:58:31.345 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:58:31.345 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: cephadm 2021-07-31T02:58:30.623431+0000 mgr.x (mgr.24542) 3058 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:58:31.345 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:58:31.345 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:58:31.346 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: yield 2021-07-31T02:58:31.346 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:58:31.346 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:58:31.346 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:58:31.347 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:58:31.347 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:58:31.347 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:58:31.347 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:58:31.347 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:58:31.348 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:58:31.348 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: task.result() 2021-07-31T02:58:31.348 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:58:31.348 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: 2021-07-31T02:58:31.348 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:58:31.349 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: 2021-07-31T02:58:31.349 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:58:31.349 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:58:31.349 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:58:31.350 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:58:31.350 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:58:31.350 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:58:31.350 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:58:31.350 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:58:31.351 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:58:31.351 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:58:31.351 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:58:31.351 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:58:31.352 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:58:31.352 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:58:31.352 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:58:31.352 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:58:31.352 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:58:31.353 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:58:31.353 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: raise 2021-07-31T02:58:31.353 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:58:31.353 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:58:31.353 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:58:31.354 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:58:31.354 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:58:31.355 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:58:31.355 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: yield 2021-07-31T02:58:31.356 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:58:31.356 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:58:31.356 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:58:31.356 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:58:31.356 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:58:31.357 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:58:31.357 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:58:31.357 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:58:31.357 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:58:31.358 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: task.result() 2021-07-31T02:58:31.358 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:58:31.358 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: 2021-07-31T02:58:31.358 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:58:31.358 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: 2021-07-31T02:58:31.359 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:58:31.359 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:58:31.359 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:58:31.359 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:58:31.360 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:58:31.360 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:58:31.360 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:58:31.360 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:58:31.360 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:58:31.361 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:58:31.361 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:58:31.361 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:58:31.361 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:58:31.361 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:58:31.362 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:58:31.362 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:58:31.362 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:58:31.362 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:58:31.363 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: raise 2021-07-31T02:58:31.363 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:58:31.363 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:58:31.363 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:58:31.363 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:58:31.364 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:30 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:58:32.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:31 smithi154 conmon[34553]: cluster 2021-07-31T02:58:30.819019+0000 mgr.x (mgr.24542) 3059 : cluster [DBG] pgmap v4146: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:32.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:31 smithi045 conmon[39718]: cluster 2021-07-31T02:58:30.819019+0000 mgr.x (mgr.24542) 2021-07-31T02:58:32.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:31 smithi045 conmon[39718]: 3059 : cluster [DBG] pgmap v4146: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:32.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:31 smithi045 conmon[33655]: cluster 2021-07-31T02:58:30.819019+0000 mgr.x (mgr.24542) 2021-07-31T02:58:32.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:31 smithi045 conmon[33655]: 3059 : cluster [DBG] pgmap v4146: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:34.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:33 smithi154 conmon[34553]: cluster 2021-07-31T02:58:32.819495+0000 mgr.x (mgr.24542) 3060 : cluster [DBG] 2021-07-31T02:58:34.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:33 smithi154 conmon[34553]: pgmap v4147: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:34.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:33 smithi045 conmon[39718]: cluster 2021-07-31T02:58:32.819495+0000 mgr.x ( 2021-07-31T02:58:34.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:33 smithi045 conmon[39718]: mgr.24542) 3060 : cluster [DBG] pgmap v4147: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:34.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:33 smithi045 conmon[33655]: cluster 2021-07-31T02:58:32.819495+0000 mgr.x (mgr.24542) 3060 : 2021-07-31T02:58:34.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:33 smithi045 conmon[33655]: cluster [DBG] pgmap v4147: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:35.665 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:58:35.943 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:58:35.943 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 73m ago 74m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:58:35.943 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 71m ago 73m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:58:35.944 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 71m ago 77m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:58:35.944 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 73m ago 79m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:58:35.944 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 73m ago 79m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:58:35.945 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 71m ago 77m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:58:35.945 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 73m ago 77m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:58:35.945 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 73m ago 74m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:58:35.945 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 71m ago 74m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:58:35.946 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 73m ago 77m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:58:35.946 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 73m ago 76m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:58:35.946 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 73m ago 76m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:58:35.946 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 73m ago 76m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:58:35.947 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 71m ago 75m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:58:35.947 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 71m ago 75m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:58:35.947 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 71m ago 75m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:58:35.947 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 71m ago 74m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:58:35.948 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 71m ago 74m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:58:36.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:35 smithi154 conmon[34553]: cluster 2021-07-31T02:58:34.819958+0000 mgr.x (mgr.24542) 3061 : cluster [DBG] pgmap v4148: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:36.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:35 smithi045 conmon[39718]: cluster 2021-07-31T02:58:34.819958+0000 mgr.x (mgr.24542 2021-07-31T02:58:36.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:35 smithi045 conmon[39718]: ) 3061 : cluster [DBG] pgmap v4148: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:36.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:35 smithi045 conmon[33655]: cluster 2021-07-31T02:58:34.819958+0000 mgr.x (mgr.24542 2021-07-31T02:58:36.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:35 smithi045 conmon[33655]: ) 3061 : cluster [DBG] pgmap v4148: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:36.277 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:58:36.277 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:58:36.277 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:58:36.277 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:58:36.278 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:58:36.278 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:58:36.278 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:58:36.278 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:58:36.278 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:58:36.279 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:58:36.279 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:58:36.279 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:58:36.279 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:58:36.280 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:58:36.280 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:58:36.280 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:58:36.280 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:58:37.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:36 smithi154 conmon[34553]: audit 2021-07-31T02:58:35.653441+0000 mgr.x (mgr.24542) 3062 : audit [DBG] from='client.28474 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:58:37.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:36 smithi154 conmon[34553]: 2021-07-31T02:58:37.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:36 smithi154 conmon[34553]: audit 2021-07-31T02:58:35.935106+0000 mgr.x (mgr.24542) 3063 : audit [DBG] from='client.28480 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:58:37.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:36 smithi154 conmon[34553]: audit 2021-07-31T02:58:36.275916+0000 mon.c ( 2021-07-31T02:58:37.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:36 smithi154 conmon[34553]: mon.1) 164 : audit [DBG] from='client.? 172.21.15.45:0/1070106951' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:58:37.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:36 smithi045 conmon[39718]: audit 2021-07-31T02:58:35.653441 2021-07-31T02:58:37.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:36 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 3062 : audit [DBG] from='client.28474 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:58:37.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:36 smithi045 conmon[39718]: audit 2021-07-31T02:58:37.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:36 smithi045 conmon[39718]: 2021-07-31T02:58:35.935106+0000 mgr.x (mgr 2021-07-31T02:58:37.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:36 smithi045 conmon[39718]: .24542) 3063 : audit [DBG] from='client.28480 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:58:37.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:36 smithi045 conmon[39718]: audit 2021-07 2021-07-31T02:58:37.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:36 smithi045 conmon[39718]: -31T02:58:36.275916+0000 mon.c (mon 2021-07-31T02:58:37.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:36 smithi045 conmon[39718]: .1) 164 : audit [DBG] from='client.? 172.21.15.45:0/1070106951' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:58:37.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:36 smithi045 conmon[33655]: audit 2021-07-31T02:58:35.653441+0000 mgr.x (mgr.24542) 3062 : audit [DBG] 2021-07-31T02:58:37.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:36 smithi045 conmon[33655]: from='client.28474 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:58:37.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:36 smithi045 conmon[33655]: audit 2021-07-31T02:58:35.935106+0000 mgr.x (mgr.24542) 3063 : audit [DBG] from='client.28480 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:58:37.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:36 smithi045 conmon[33655]: audit 2021-07-31T02:58 2021-07-31T02:58:37.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:36 smithi045 conmon[33655]: :36.275916+0000 mon.c (mon.1) 164 : audit [DBG] from='client.? 172.21.15.45:0/1070106951' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:58:38.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:37 smithi154 conmon[34553]: cluster 2021-07-31T02:58:36.820487+0000 mgr.x (mgr.24542) 3064 : cluster 2021-07-31T02:58:38.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:37 smithi154 conmon[34553]: [DBG] pgmap v4149: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:38.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:37 smithi045 conmon[39718]: cluster 2021-07-31T02:58:36.820487+0000 mgr.x (mgr.24542) 3064 : cluster [DBG] pgmap v4149: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:38.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:37 smithi045 conmon[33655]: cluster 2021-07-31T02:58:36.820487+0000 mgr.x (mgr.24542) 3064 : cluster [DBG] pgmap v4149: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:40.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:39 smithi154 conmon[34553]: cluster 2021-07-31T02:58:38.821034+0000 mgr.x (mgr.24542) 3065 : cluster [DBG] pgmap v4150: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:40.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:39 smithi045 conmon[33655]: cluster 2021-07-31T02:58:38.821034 2021-07-31T02:58:40.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:39 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 3065 : cluster [DBG] pgmap v4150: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:40.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:39 smithi045 conmon[39718]: cluster 2021-07-31T02:58:38.821034+0000 mgr.x (mgr 2021-07-31T02:58:40.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:39 smithi045 conmon[39718]: .24542) 3065 : cluster [DBG] pgmap v4150: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:42.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:41 smithi154 conmon[34553]: cluster 2021-07-31T02:58:40.821671+0000 mgr.x (mgr.24542) 3066 : cluster [DBG] pgmap v4151: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:42.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:41 smithi045 conmon[33655]: cluster 2021-07-31T02:58:40.821671 2021-07-31T02:58:42.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:41 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 3066 : cluster [DBG] pgmap v4151: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:42.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:41 smithi045 conmon[39718]: cluster 2021-07-31T02:58:40.821671+0000 mgr.x ( 2021-07-31T02:58:42.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:41 smithi045 conmon[39718]: mgr.24542) 3066 : cluster [DBG] pgmap v4151: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:44.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:43 smithi154 conmon[34553]: cluster 2021-07-31T02:58:42.822266+0000 mgr.x (mgr.24542) 3067 : cluster [DBG] pgmap v4152: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:44.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:43 smithi045 conmon[33655]: cluster 2021-07-31T02:58:42.822266+0000 2021-07-31T02:58:44.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:43 smithi045 conmon[33655]: mgr.x (mgr.24542) 3067 : cluster [DBG] pgmap v4152: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:44.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:43 smithi045 conmon[39718]: cluster 2021-07-31T02:58:42.822266+0000 2021-07-31T02:58:44.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:43 smithi045 conmon[39718]: mgr.x (mgr.24542) 3067 : cluster [DBG] pgmap v4152: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:46.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:45 smithi154 conmon[34553]: cluster 2021-07-31T02:58:44.822886+0000 mgr.x (mgr.24542) 3068 : cluster [DBG] pgmap v4153: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:46.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:45 smithi045 conmon[33655]: cluster 2021-07-31T02:58:44.822886+0000 mgr.x ( 2021-07-31T02:58:46.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:45 smithi045 conmon[33655]: mgr.24542) 3068 : cluster [DBG] pgmap v4153: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:46.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:45 smithi045 conmon[39718]: cluster 2021-07-31T02:58:44.822886+0000 mgr.x ( 2021-07-31T02:58:46.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:45 smithi045 conmon[39718]: mgr.24542) 3068 : cluster [DBG] pgmap v4153: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:48.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:47 smithi154 conmon[34553]: cluster 2021-07-31T02:58:46.823347+0000 mgr.x (mgr.24542) 3069 : cluster [DBG] pgmap v4154: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:48.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:47 smithi045 conmon[33655]: cluster 2021-07-31T02:58:46.823347+0000 mgr.x (mgr.24542) 3069 : cluster [DBG] pgmap v4154: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:48.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:47 smithi045 conmon[39718]: cluster 2021-07-31T02:58:46 2021-07-31T02:58:48.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:47 smithi045 conmon[39718]: .823347+0000 mgr.x (mgr.24542) 3069 : cluster [DBG] pgmap v4154: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:50.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:49 smithi154 conmon[34553]: cluster 2021-07-31T02:58:48.823813+0000 mgr.x (mgr.24542) 3070 : cluster [DBG] pgmap v4155: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:50.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:49 smithi045 conmon[39718]: cluster 2021-07-31T02:58:48.823813+0000 mgr.x 2021-07-31T02:58:50.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:49 smithi045 conmon[39718]: (mgr.24542) 3070 : cluster [DBG] pgmap v4155: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:50.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:49 smithi045 conmon[33655]: cluster 2021-07-31T02:58:48.823813 2021-07-31T02:58:50.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:49 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 3070 : cluster [DBG] pgmap v4155: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:52.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:51 smithi154 conmon[34553]: cluster 2021-07-31T02:58:50.824218+0000 mgr.x (mgr.24542) 3071 : cluster [DBG] pgmap v4156: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:52.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:51 smithi045 conmon[39718]: cluster 2021-07-31T02:58:50.824218+0000 mgr.x 2021-07-31T02:58:52.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:51 smithi045 conmon[39718]: (mgr.24542) 3071 : cluster [DBG] pgmap v4156: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:52.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:51 smithi045 conmon[33655]: cluster 2021-07-31T02:58:50.824218 2021-07-31T02:58:52.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:51 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 3071 : cluster [DBG] pgmap v4156: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:54.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:53 smithi154 conmon[34553]: cluster 2021-07-31T02:58:52.824742+0000 mgr.x (mgr.24542) 3072 : cluster [DBG] pgmap v4157: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:54.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:53 smithi045 conmon[39718]: cluster 2021-07-31T02:58:52.824742+0000 mgr.x (mgr.24542) 3072 : 2021-07-31T02:58:54.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:53 smithi045 conmon[39718]: cluster [DBG] pgmap v4157: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:54.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:53 smithi045 conmon[33655]: cluster 2021-07-31T02:58:52.824742+0000 mgr.x 2021-07-31T02:58:54.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:53 smithi045 conmon[33655]: (mgr.24542) 3072 : cluster [DBG] pgmap v4157: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:56.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:55 smithi154 conmon[34553]: cluster 2021-07-31T02:58:54.825194+0000 mgr.x (mgr.24542) 3073 : cluster [DBG] pgmap v4158: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:56.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:55 smithi045 conmon[39718]: cluster 2021-07-31T02:58:54.825194+0000 mgr.x (mgr.24542) 3073 : cluster 2021-07-31T02:58:56.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:55 smithi045 conmon[39718]: [DBG] pgmap v4158: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:56.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:55 smithi045 conmon[33655]: cluster 2021-07-31T02:58 2021-07-31T02:58:56.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:55 smithi045 conmon[33655]: :54.825194+0000 mgr.x (mgr.24542) 3073 : cluster [DBG] pgmap v4158: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:58.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:57 smithi154 conmon[34553]: cluster 2021-07-31T02:58:56.825721+0000 mgr.x (mgr.24542) 3074 : cluster [DBG] pgmap v4159: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:58.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:57 smithi045 conmon[39718]: cluster 2021-07-31T02:58:56.825721+0000 mgr.x (mgr.24542) 3074 2021-07-31T02:58:58.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:57 smithi045 conmon[39718]: : cluster [DBG] pgmap v4159: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:58:58.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:57 smithi045 conmon[33655]: cluster 2021-07-31T02:58:56.825721+0000 mgr.x (mgr.24542) 3074 : cluster 2021-07-31T02:58:58.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:57 smithi045 conmon[33655]: [DBG] pgmap v4159: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:00.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:58:59 smithi154 conmon[34553]: cluster 2021-07-31T02:58:58.826276+0000 mgr.x (mgr.24542) 3075 : cluster [DBG] pgmap v4160: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:00.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:58:59 smithi045 conmon[39718]: cluster 2021-07-31T02:58:58.826276+0000 mgr.x (mgr.24542) 3075 : cluster [DBG] pgmap v4160: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:00.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:59 smithi045 conmon[33655]: cluster 2021-07-31T02:58:58.826276+0000 mgr.x (mgr.24542) 3075 : cluster 2021-07-31T02:59:00.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:58:59 smithi045 conmon[33655]: [DBG] pgmap v4160: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:02.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:01 smithi154 conmon[34553]: cluster 2021-07-31T02:59:00.826731+0000 mgr.x (mgr.24542) 3076 : cluster [DBG] pgmap v4161: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:02.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:01 smithi045 conmon[39718]: cluster 2021-07-31T02:59:00.826731+0000 2021-07-31T02:59:02.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:01 smithi045 conmon[39718]: mgr.x (mgr.24542) 3076 : cluster [DBG] pgmap v4161: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:02.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:01 smithi045 conmon[33655]: cluster 2021-07-31T02:59:00.826731+0000 mgr.x (mgr.24542) 3076 : 2021-07-31T02:59:02.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:01 smithi045 conmon[33655]: cluster [DBG] pgmap v4161: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:04.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:03 smithi154 conmon[34553]: cluster 2021-07-31T02:59:02.827185+0000 mgr.x (mgr.24542) 3077 : cluster [DBG] pgmap v4162: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:04.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:03 smithi045 conmon[39718]: cluster 2021-07-31T02:59:02.827185+0000 mgr.x (mgr.24542) 2021-07-31T02:59:04.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:03 smithi045 conmon[39718]: 3077 : cluster [DBG] pgmap v4162: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:04.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:03 smithi045 conmon[33655]: cluster 2021-07-31T02:59:02.827185 2021-07-31T02:59:04.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:03 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 3077 : cluster [DBG] pgmap v4162: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:06.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:05 smithi154 conmon[34553]: cluster 2021-07-31T02:59:04.827615+0000 mgr.x (mgr.24542) 3078 : cluster [DBG] pgmap v4163: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:06.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:05 smithi045 conmon[33655]: cluster 2021-07-31T02:59:04.827615+0000 mgr.x (mgr 2021-07-31T02:59:06.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:05 smithi045 conmon[33655]: .24542) 3078 : cluster [DBG] pgmap v4163: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:06.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:05 smithi045 conmon[39718]: cluster 2021-07-31T02:59:04.827615+0000 mgr.x ( 2021-07-31T02:59:06.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:05 smithi045 conmon[39718]: mgr.24542) 3078 : cluster [DBG] pgmap v4163: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:06.588 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:59:06.864 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:59:06.865 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 74m ago 74m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:59:06.865 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 71m ago 74m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:59:06.866 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 71m ago 78m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:59:06.866 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 74m ago 80m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:59:06.866 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 74m ago 80m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:59:06.867 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 71m ago 78m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:59:06.867 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 74m ago 78m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:59:06.867 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 74m ago 74m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:59:06.867 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 71m ago 74m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:59:06.868 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 74m ago 77m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:59:06.868 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 74m ago 77m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:59:06.868 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 74m ago 77m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:59:06.868 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 74m ago 76m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:59:06.869 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 71m ago 76m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:59:06.869 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 71m ago 76m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:59:06.869 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 71m ago 75m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:59:06.869 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 71m ago 75m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:59:06.869 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 71m ago 75m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:59:07.203 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:59:07.203 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:59:07.203 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:59:07.204 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:59:07.204 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:59:07.204 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:59:07.204 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:59:07.205 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:59:07.205 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:59:07.205 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:59:07.205 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:59:07.206 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:59:07.206 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:59:07.206 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:59:07.206 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:59:07.207 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:59:07.207 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:59:07.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:06 smithi154 conmon[34553]: audit 2021-07-31T02:59:06.575716+0000 mgr.x (mgr.24542) 3079 : audit [DBG] from='client.28492 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:59:07.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:06 smithi045 conmon[33655]: audit 2021-07-31T02:59:06.575716 2021-07-31T02:59:07.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:06 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 3079 : audit [DBG] from='client.28492 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:59:07.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:06 smithi045 conmon[39718]: audit 2021-07-31T02: 2021-07-31T02:59:07.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:06 smithi045 conmon[39718]: 59:06.575716+0000 mgr.x (mgr.24542) 3079 : audit [DBG] from='client.28492 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:59:08.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:07 smithi154 conmon[34553]: cluster 2021-07-31T02:59:06.827959+0000 mgr.x (mgr.24542) 3080 : cluster [DBG] pgmap v4164: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:08.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:07 smithi154 conmon[34553]: audit 2021-07- 2021-07-31T02:59:08.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:07 smithi154 conmon[34553]: 31T02:59:06.856493+0000 mgr.x (mgr.24542) 3081 : audit [DBG] from='client.18558 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:59:08.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:07 smithi154 conmon[34553]: audit 2021-07-31T02:59:07.203033+0000 mon.c (mon.1) 165 : audit [DBG] from='client.? 172.21.15.45:0/2080369935' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:59:08.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:07 smithi045 conmon[39718]: cluster 2021-07-31T02:59:06.827959+0000 2021-07-31T02:59:08.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:07 smithi045 conmon[39718]: mgr.x (mgr.24542) 3080 : cluster [DBG] pgmap v4164: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:08.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:07 smithi045 conmon[39718]: audit 2021-07 2021-07-31T02:59:08.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:07 smithi045 conmon[39718]: -31T02:59:06.856493+0000 mgr.x (mgr.24542) 3081 2021-07-31T02:59:08.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:07 smithi045 conmon[39718]: : audit [DBG] from='client.18558 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:59:08.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:07 smithi045 conmon[39718]: audit 2021- 2021-07-31T02:59:08.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:07 smithi045 conmon[39718]: 07-31T02:59:07.203033 2021-07-31T02:59:08.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:07 smithi045 conmon[39718]: +0000 mon.c (mon.1) 165 : audit 2021-07-31T02:59:08.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:07 smithi045 conmon[39718]: [DBG] from='client.? 172.21.15.45:0/2080369935' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:59:08.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:07 smithi045 conmon[33655]: cluster 2021-07-31T02:59:06.827959+0000 mgr.x (mgr. 2021-07-31T02:59:08.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:07 smithi045 conmon[33655]: 24542) 3080 : cluster [DBG] pgmap v4164: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:08.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:07 smithi045 conmon[33655]: audit 2021-07-31T02:59:06.856493+0000 mgr.x (mgr.24542) 3081 : audit [DBG] 2021-07-31T02:59:08.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:07 smithi045 conmon[33655]: from='client.18558 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:59:08.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:07 smithi045 conmon[33655]: audit 2021-07-31T02:59:08.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:07 smithi045 conmon[33655]: 2021-07-31 2021-07-31T02:59:08.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:07 smithi045 conmon[33655]: T02:59:07.203033+0000 mon.c (mon.1 2021-07-31T02:59:08.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:07 smithi045 conmon[33655]: ) 165 : audit [DBG] from='client.? 172.21.15.45:0/2080369935' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:59:10.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:09 smithi154 conmon[34553]: cluster 2021-07-31T02:59:08.828448+0000 mgr.x (mgr.24542) 3082 : cluster [DBG] pgmap v4165: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:10.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:09 smithi045 conmon[39718]: cluster 2021-07-31T02:59:08.828448+0000 mgr.x (mgr 2021-07-31T02:59:10.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:09 smithi045 conmon[39718]: .24542) 3082 : cluster [DBG] pgmap v4165: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:10.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:09 smithi045 conmon[33655]: cluster 2021-07-31T02:59:08.828448+0000 mgr.x (mgr.24542) 3082 : cluster [DBG] 2021-07-31T02:59:10.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:09 smithi045 conmon[33655]: pgmap v4165: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:12.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:11 smithi154 conmon[34553]: cluster 2021-07-31T02:59:10.828981+0000 mgr.x (mgr.24542) 3083 : cluster [DBG] pgmap v4166: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:12.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:11 smithi045 conmon[39718]: cluster 2021-07-31T02:59:10. 2021-07-31T02:59:12.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:11 smithi045 conmon[39718]: 828981+0000 mgr.x (mgr.24542) 3083 : cluster [DBG] pgmap v4166: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:12.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:11 smithi045 conmon[33655]: cluster 2021-07-31T02:59:10.828981 2021-07-31T02:59:12.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:11 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 3083 : cluster [DBG] pgmap v4166: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:14.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:13 smithi154 conmon[34553]: cluster 2021-07-31T02:59:12.829470+0000 mgr.x (mgr.24542) 3084 : cluster [DBG] pgmap v4167: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:14.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:13 smithi045 conmon[39718]: cluster 2021-07-31T02:59:12.829470+0000 mgr.x (mgr.24542) 3084 2021-07-31T02:59:14.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:13 smithi045 conmon[39718]: : cluster [DBG] pgmap v4167: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:14.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:13 smithi045 conmon[33655]: cluster 2021-07-31T02:59:12.829470+0000 mgr.x (mgr.24542) 3084 : cluster 2021-07-31T02:59:14.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:13 smithi045 conmon[33655]: [DBG] pgmap v4167: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:16.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:15 smithi154 conmon[34553]: cluster 2021-07-31T02:59:14.829985+0000 mgr.x (mgr.24542) 3085 : cluster [DBG] pgmap v4168: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:16.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:15 smithi045 conmon[39718]: cluster 2021-07-31T02:59:14.829985+0000 mgr.x 2021-07-31T02:59:16.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:15 smithi045 conmon[39718]: (mgr.24542) 3085 : cluster [DBG] pgmap v4168: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:16.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:15 smithi045 conmon[33655]: cluster 2021-07-31T02:59:14.829985+0000 mgr.x (mgr.24542 2021-07-31T02:59:16.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:15 smithi045 conmon[33655]: ) 3085 : cluster [DBG] pgmap v4168: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:18.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:17 smithi154 conmon[34553]: cluster 2021-07-31T02:59:16.830426+0000 mgr.x (mgr.24542) 3086 : cluster [DBG] pgmap v4169: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:18.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:17 smithi045 conmon[39718]: cluster 2021-07-31T02:59:16.830426+0000 mgr.x ( 2021-07-31T02:59:18.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:17 smithi045 conmon[39718]: mgr.24542) 3086 : cluster [DBG] pgmap v4169: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:18.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:17 smithi045 conmon[33655]: cluster 2021-07-31T02:59:16.830426+0000 mgr.x (mgr.24542) 3086 : 2021-07-31T02:59:18.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:17 smithi045 conmon[33655]: cluster [DBG] pgmap v4169: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:20.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:19 smithi154 conmon[34553]: cluster 2021-07-31T02:59:18.830977+0000 mgr.x (mgr.24542) 3087 : cluster [DBG] pgmap v4170: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:20.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:19 smithi045 conmon[39718]: cluster 2021-07-31T02:59:18.830977+0000 mgr.x ( 2021-07-31T02:59:20.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:19 smithi045 conmon[39718]: mgr.24542) 3087 : cluster [DBG] pgmap v4170: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:20.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:19 smithi045 conmon[33655]: cluster 2021-07-31T02:59:18.830977+0000 mgr.x ( 2021-07-31T02:59:20.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:19 smithi045 conmon[33655]: mgr.24542) 3087 : cluster [DBG] pgmap v4170: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:22.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:21 smithi154 conmon[34553]: cluster 2021-07-31T02:59:20.831536+0000 mgr.x (mgr.24542) 3088 : cluster [DBG] pgmap v4171: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:22.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:21 smithi045 conmon[33655]: cluster 2021-07-31T02:59:20.831536+0000 mgr.x (mgr 2021-07-31T02:59:22.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:21 smithi045 conmon[33655]: .24542) 3088 : cluster [DBG] pgmap v4171: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:22.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:21 smithi045 conmon[39718]: cluster 2021-07-31T02:59:20.831536+0000 2021-07-31T02:59:22.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:21 smithi045 conmon[39718]: mgr.x (mgr.24542) 3088 : cluster [DBG] pgmap v4171: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:24.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:23 smithi154 conmon[34553]: cluster 2021-07-31T02:59:22.832093+0000 mgr.x (mgr.24542) 3089 : cluster [DBG] pgmap v4172: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:24.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:23 smithi045 conmon[33655]: cluster 2021-07-31T02:59:22.832093+0000 mgr.x (mgr.24542) 3089 2021-07-31T02:59:24.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:23 smithi045 conmon[33655]: : cluster [DBG] pgmap v4172: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:24.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:23 smithi045 conmon[39718]: cluster 2021-07-31T02:59:22.832093+0000 mgr.x (mgr. 2021-07-31T02:59:24.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:23 smithi045 conmon[39718]: 24542) 3089 : cluster [DBG] pgmap v4172: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:26.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:25 smithi154 conmon[34553]: cluster 2021-07-31T02:59:24.832541+0000 mgr.x (mgr.24542) 2021-07-31T02:59:26.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:25 smithi154 conmon[34553]: 3090 : cluster [DBG] pgmap v4173: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:26.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:25 smithi045 conmon[33655]: cluster 2021-07-31T02:59:24.832541+0000 mgr.x (mgr.24542) 3090 : cluster 2021-07-31T02:59:26.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:25 smithi045 conmon[33655]: [DBG] pgmap v4173: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:26.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:25 smithi045 conmon[39718]: cluster 2021-07-31T02:59:24.832541+0000 mgr.x ( 2021-07-31T02:59:26.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:25 smithi045 conmon[39718]: mgr.24542) 3090 : cluster [DBG] pgmap v4173: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:28.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:27 smithi154 conmon[34553]: cluster 2021-07-31T02:59:26.833113+0000 mgr.x (mgr.24542) 3091 : cluster [DBG] pgmap v4174: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:28.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:27 smithi045 conmon[33655]: cluster 2021-07-31T02:59:26.833113+0000 mgr.x (mgr.24542) 2021-07-31T02:59:28.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:27 smithi045 conmon[33655]: 3091 : cluster [DBG] pgmap v4174: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:28.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:27 smithi045 conmon[39718]: cluster 2021-07-31T02:59: 2021-07-31T02:59:28.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:27 smithi045 conmon[39718]: 26.833113+0000 mgr.x (mgr.24542) 3091 : cluster [DBG] pgmap v4174: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:29.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:28 smithi154 conmon[34553]: audit 2021-07-31T02:59:28.223249+0000 mon.b (mon.2) 435 : audit [INF] 2021-07-31T02:59:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:28 smithi154 conmon[34553]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:59:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:28 smithi154 conmon[34553]: audit 2021-07-31T02:59:28.223504+0000 mon.b (mon.2) 436 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:59:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:28 smithi154 conmon[34553]: audit 2021- 2021-07-31T02:59:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:28 smithi154 conmon[34553]: 07-31T02:59:28.223880+0000 mon.a (mon.0) 3233 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:59:29.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:28 smithi154 conmon[34553]: audit 2021-07-31T02:59:28.224392+0000 mon.a (mon.0) 3234 : audit 2021-07-31T02:59:29.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:28 smithi154 conmon[34553]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:59:29.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:28 smithi045 conmon[33655]: audit 2021-07-31T02:59:28.223249+0000 mon.b (mon.2) 435 : 2021-07-31T02:59:29.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:28 smithi045 conmon[33655]: audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:59:29.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:28 smithi045 conmon[33655]: audit 2021-07-31T02:59:28.223504+0000 mon.b (mon.2) 436 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:59:29.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:28 smithi045 conmon[33655]: audit 2021-07-31T02:59:28 2021-07-31T02:59:29.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:28 smithi045 conmon[33655]: .223880+0000 mon.a (mon.0) 3233 : audit [INF] 2021-07-31T02:59:29.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:28 smithi045 conmon[33655]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:59:29.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:28 smithi045 conmon[33655]: audit 2021-07-31T02:59:29.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:28 smithi045 conmon[33655]: 2021-07-31T02:59:28 2021-07-31T02:59:29.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:28 smithi045 conmon[33655]: .224392+0000 mon.a (mon. 2021-07-31T02:59:29.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:28 smithi045 conmon[33655]: 0) 3234 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:59:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:28 smithi045 conmon[39718]: audit 2021-07-31T02:59:28.223249 2021-07-31T02:59:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:28 smithi045 conmon[39718]: +0000 mon.b (mon.2) 435 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:59:29.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:28 smithi045 conmon[39718]: audit 2021-07-31T02:59:28.223504 2021-07-31T02:59:29.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:28 smithi045 conmon[39718]: +0000 mon.b (mon.2) 436 : audit [INF] 2021-07-31T02:59:29.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:28 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:59:29.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:28 smithi045 conmon[39718]: audit 2021-07 2021-07-31T02:59:29.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:28 smithi045 conmon[39718]: -31T02:59:28.223880 2021-07-31T02:59:29.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:28 smithi045 conmon[39718]: +0000 mon.a (mon.0) 3233 2021-07-31T02:59:29.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:28 smithi045 conmon[39718]: : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T02:59:29.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:28 smithi045 conmon[39718]: audit 2021 2021-07-31T02:59:29.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:28 smithi045 conmon[39718]: -07-31T02:59:28. 2021-07-31T02:59:29.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:28 smithi045 conmon[39718]: 224392+0000 mon.a (mon.0 2021-07-31T02:59:29.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:28 smithi045 conmon[39718]: ) 3234 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T02:59:29.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:28 smithi045 conmon[39718]: 2021-07-31T02:59:30.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:29 smithi154 conmon[34553]: cluster 2021-07-31T02:59:28.833579+0000 mgr.x (mgr.24542) 3092 : cluster 2021-07-31T02:59:30.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:29 smithi154 conmon[34553]: [DBG] pgmap v4175: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:30.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:29 smithi045 conmon[33655]: cluster 2021-07-31T02:59:28.833579 2021-07-31T02:59:30.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:29 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 3092 : cluster [DBG] pgmap v4175: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:30.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:29 smithi045 conmon[39718]: cluster 2021-07-31T02:59:28.833579+0000 mgr.x (mgr.24542) 3092 : cluster 2021-07-31T02:59:30.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:29 smithi045 conmon[39718]: [DBG] pgmap v4175: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:30.908 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: debug 2021-07-31T02:59:30.669+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:59:30.909 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:59:30.909 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:59:30.909 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: yield 2021-07-31T02:59:30.909 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:59:30.910 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:59:30.910 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:59:30.910 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:59:30.910 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:59:30.910 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:59:30.911 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:59:30.911 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:59:30.911 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:59:30.911 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:59:30.912 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:59:30.912 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: debug 2021-07-31T02:59:30.680+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:59:30.912 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:59:30.912 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:59:30.912 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: yield 2021-07-31T02:59:30.913 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:59:30.913 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:59:30.913 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:59:30.913 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:59:30.914 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:59:30.914 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:59:30.914 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:59:30.914 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:59:30.915 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:59:30.915 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:59:30.915 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:59:30.915 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: debug 2021-07-31T02:59:30.706+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:59:30.915 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:59:30.916 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:59:30.916 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: yield 2021-07-31T02:59:30.916 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:59:30.916 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:59:30.916 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:59:30.917 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:59:30.917 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:59:30.917 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:59:30.917 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:59:30.918 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:59:30.918 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:59:30.918 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:59:30.918 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:59:30.919 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: debug 2021-07-31T02:59:30.706+0000 7fcb9195d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:59:30.919 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:59:30.919 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:59:30.919 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: yield 2021-07-31T02:59:30.919 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:59:30.920 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:59:30.920 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:59:30.920 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:59:30.920 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:59:30.920 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:59:30.921 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:59:30.921 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:59:30.921 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:59:30.921 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:59:30.922 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:59:30.922 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: 2021-07-31T02:59:30.922 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:59:30.922 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: 2021-07-31T02:59:30.922 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:59:30.923 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:59:30.923 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:59:30.923 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:59:30.923 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:59:30.924 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:59:30.924 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:59:30.924 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:59:30.924 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:59:30.925 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:59:30.925 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:59:30.925 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:59:30.925 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:59:30.926 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:59:30.926 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:59:30.926 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:59:30.926 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:59:30.927 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:59:30.927 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: raise 2021-07-31T02:59:30.927 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:59:30.927 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:59:30.927 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:59:30.928 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:59:30.928 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:59:30.928 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: debug 2021-07-31T02:59:30.715+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T02:59:30.928 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:59:30.929 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:59:30.929 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: yield 2021-07-31T02:59:30.929 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:59:30.929 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:59:30.929 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:59:30.930 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:59:30.930 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:59:30.930 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:59:30.930 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:59:30.931 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:59:30.931 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:59:30.931 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:59:30.931 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:59:30.931 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: debug 2021-07-31T02:59:30.715+0000 7fcb9215e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:59:30.932 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:59:30.932 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:59:30.932 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: yield 2021-07-31T02:59:30.932 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:59:30.933 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:59:30.933 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:59:30.933 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T02:59:30.933 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:59:30.933 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T02:59:30.934 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:59:30.934 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T02:59:30.934 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:59:30.934 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: task.result() 2021-07-31T02:59:30.935 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T02:59:30.935 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: 2021-07-31T02:59:30.935 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T02:59:30.935 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: 2021-07-31T02:59:30.936 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T02:59:30.936 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:59:30.936 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: return f(*arg) 2021-07-31T02:59:30.936 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:59:30.937 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:59:30.937 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:59:30.937 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:59:30.937 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:59:30.938 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:59:30.938 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:59:30.938 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T02:59:30.938 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:59:30.938 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: raise self._exception 2021-07-31T02:59:30.939 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:59:30.939 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:59:30.939 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:59:30.939 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T02:59:30.940 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:59:30.940 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: raise 2021-07-31T02:59:30.940 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:59:30.940 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T02:59:30.940 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:59:30.941 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:59:30.941 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:59:31.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[34553]: audit 2021-07-31T02:59:30.624294+0000 mon.b (mon.2) 437 : 2021-07-31T02:59:31.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[34553]: audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:59:31.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[34553]: audit 2021-07-31T02:59:30.625905+0000 mon.b (mon.2) 438 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:59:31.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[34553]: audit 2021-07-31 2021-07-31T02:59:31.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[34553]: T02:59:30.626817+0000 mon.b (mon.2) 439 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:59:31.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[34553]: cephadm 2021-07-31T02:59:31.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[34553]: 2021-07-31T02:59:30.627569+0000 mgr.x (mgr.24542) 3093 : 2021-07-31T02:59:31.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:30 smithi154 conmon[34553]: cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:59:31.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:30 smithi045 conmon[33655]: audit 2021-07-31T02:59:30.624294+0000 mon.b (mon.2) 437 : audit [DBG] 2021-07-31T02:59:31.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:30 smithi045 conmon[33655]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:59:31.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:30 smithi045 conmon[33655]: audit 2021-07-31T02:59:30.625905+0000 mon.b (mon.2) 438 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:59:31.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:30 smithi045 conmon[33655]: audit 2021-07-31 2021-07-31T02:59:31.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:30 smithi045 conmon[33655]: T02:59:30.626817+0000 mon.b ( 2021-07-31T02:59:31.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:30 smithi045 conmon[33655]: mon.2) 439 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:59:31.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:30 smithi045 conmon[33655]: cephadm 2021- 2021-07-31T02:59:31.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:30 smithi045 conmon[33655]: 07-31T02:59:30. 2021-07-31T02:59:31.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:30 smithi045 conmon[33655]: 627569+0000 mgr.x (mgr.24542) 3093 : cephadm 2021-07-31T02:59:31.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:30 smithi045 conmon[33655]: [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:59:31.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:30 smithi045 conmon[39718]: audit 2021-07-31T02:59:30.624294 2021-07-31T02:59:31.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:30 smithi045 conmon[39718]: +0000 mon.b (mon.2) 437 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T02:59:31.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:30 smithi045 conmon[39718]: audit 2021-07-31T02:59:31.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:30 smithi045 conmon[39718]: 2021-07 2021-07-31T02:59:31.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:30 smithi045 conmon[39718]: -31T02:59:30.625905+0000 mon.b (mon. 2021-07-31T02:59:31.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:30 smithi045 conmon[39718]: 2) 438 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T02:59:31.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:30 smithi045 conmon[39718]: audit 2021- 2021-07-31T02:59:31.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:30 smithi045 conmon[39718]: 07-31T02:59:30.626817+0000 2021-07-31T02:59:31.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:30 smithi045 conmon[39718]: mon.b (mon.2) 439 : 2021-07-31T02:59:31.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:30 smithi045 conmon[39718]: audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T02:59:31.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:30 smithi045 conmon[39718]: cephadm 2021 2021-07-31T02:59:31.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:30 smithi045 conmon[39718]: -07-31T02:59: 2021-07-31T02:59:31.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:30 smithi045 conmon[39718]: 30.627569+0000 mgr.x (mgr 2021-07-31T02:59:31.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:30 smithi045 conmon[39718]: .24542) 3093 : cephadm [WRN] 2021-07-31T02:59:31.282 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:30 smithi045 conmon[39718]: unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T02:59:32.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:59:30.670391+0000 mgr.x (mgr.24542) 3094 : cephadm [ERR] 2021-07-31T02:59:32.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:59:32.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:59:32.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: yield 2021-07-31T02:59:32.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:59:32.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:59:32.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:59:32.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:59:32.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:59:32.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:59:32.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:59:32.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:59:32.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:59:32.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: task.result() 2021-07-31T02:59:32.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:59:32.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:59:30.670820+0000 mgr.x (mgr.24542) 3095 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:59:32.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:59:30.681788+0000 mgr.x (mgr.24542) 3096 : cephadm [ERR] 2021-07-31T02:59:32.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:59:32.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:59:32.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: yield 2021-07-31T02:59:32.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:59:32.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:59:32.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:59:32.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:59:32.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:59:32.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:59:32.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:59:32.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:59:32.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:59:32.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: task.result() 2021-07-31T02:59:32.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:59:32.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:59:30.682745+0000 mgr.x (mgr.24542) 3097 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:59:32.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:59:30.707144+0000 mgr.x (mgr.24542) 3098 : cephadm [ERR] 2021-07-31T02:59:32.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:59:32.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:59:32.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: yield 2021-07-31T02:59:32.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:59:32.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:59:32.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:59:32.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:59:32.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:59:32.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:59:32.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:59:32.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:59:32.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:59:32.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: task.result() 2021-07-31T02:59:32.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:59:32.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:59:30 2021-07-31T02:59:32.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: .707557+0000 mgr.x (mgr.24542) 3099 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:59:32.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:59:32.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:59:32.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: yield 2021-07-31T02:59:32.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:59:32.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:59:32.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:59:32.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:59:32.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:59:32.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:59:32.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:59:32.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:59:32.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:59:32.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: task.result() 2021-07-31T02:59:32.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:59:32.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: 2021-07-31T02:59:32.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:59:32.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: 2021-07-31T02:59:32.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:59:32.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:59:32.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:59:32.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:59:32.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:59:32.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:59:32.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:59:32.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:59:32.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:59:32.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:59:32.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:59:32.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:59:32.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:59:32.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:59:32.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:59:32.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:59:32.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:59:32.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:59:32.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: raise 2021-07-31T02:59:32.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:59:32.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:59:32.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:59:32.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:59:32.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:59:32.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:59:30.716175+0000 mgr.x (mgr.24542) 3100 : cephadm [ERR] 2021-07-31T02:59:32.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:59:32.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:59:32.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: yield 2021-07-31T02:59:32.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:59:32.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:59:32.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:59:32.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:59:32.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:59:32.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:59:32.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:59:32.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:59:32.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:59:32.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: task.result() 2021-07-31T02:59:32.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:59:32.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: cephadm 2021-07-31T02:59:30.716562+0000 mgr.x (mgr.24542) 3101 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:59:32.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:59:32.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:59:32.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: yield 2021-07-31T02:59:32.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:59:32.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:59:32.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:59:32.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T02:59:32.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:59:32.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T02:59:32.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:59:32.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T02:59:32.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:59:32.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: task.result() 2021-07-31T02:59:32.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T02:59:32.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: 2021-07-31T02:59:32.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T02:59:32.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: 2021-07-31T02:59:32.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T02:59:32.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:59:32.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: return f(*arg) 2021-07-31T02:59:32.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:59:32.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:59:32.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:59:32.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:59:32.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:59:32.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:59:32.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:59:32.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T02:59:32.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:59:32.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: raise self._exception 2021-07-31T02:59:32.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:59:32.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:59:32.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:59:32.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T02:59:32.285 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:59:32.285 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: raise 2021-07-31T02:59:32.285 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:59:32.285 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T02:59:32.286 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:59:32.286 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:59:32.286 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:59:32.286 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:31 smithi154 conmon[34553]: cluster 2021-07-31T02:59:30.834054+0000 mgr.x (mgr.24542) 3102 : cluster [DBG] pgmap v4176: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:32.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:59:30.670391+0000 mgr.x (mgr.24542) 3094 : cephadm [ERR] 2021-07-31T02:59:32.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:59:32.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:59:32.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: yield 2021-07-31T02:59:32.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:59:32.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:59:32.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:59:32.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:59:32.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:59:32.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:59:32.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:59:32.290 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:59:32.290 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:59:32.290 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: task.result() 2021-07-31T02:59:32.290 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:59:32.290 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:59:30.670820+0000 mgr.x (mgr.24542) 3095 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:59:32.291 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:59:30.681788+0000 mgr.x (mgr.24542) 3096 : cephadm [ERR] 2021-07-31T02:59:32.291 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:59:32.291 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:59:32.291 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: yield 2021-07-31T02:59:32.292 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:59:32.292 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:59:32.292 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:59:32.292 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:59:32.292 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:59:32.293 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:59:32.293 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:59:32.293 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:59:32.293 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:59:32.293 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: task.result() 2021-07-31T02:59:32.294 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:59:32.294 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:59:30.682745+0000 mgr.x (mgr.24542) 3097 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:59:32.294 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:59:30.707144+0000 mgr.x (mgr.24542) 3098 : cephadm [ERR] 2021-07-31T02:59:32.294 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:59:32.295 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:59:32.295 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: yield 2021-07-31T02:59:32.295 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:59:32.295 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:59:32.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:59:30.670391+0000 mgr.x (mgr 2021-07-31T02:59:32.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: .24542) 3094 : cephadm [ERR] 2021-07-31T02:59:32.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:59:32.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:59:32.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: yield 2021-07-31T02:59:32.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:59:32.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:59:32.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:59:32.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:59:32.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:59:32.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:59:32.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:59:32.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:59:32.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:59:32.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: task.result() 2021-07-31T02:59:32.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:59:32.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: cephadm 2021-07-31T02: 2021-07-31T02:59:32.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: 59:30.670820+0000 mgr.x (mgr.24542) 3095 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T02:59:32.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:59:30.681788+0000 mgr.x (mgr.24542) 3096 : 2021-07-31T02:59:32.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: cephadm [ERR] 2021-07-31T02:59:32.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:59:32.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:59:32.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: yield 2021-07-31T02:59:32.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:59:32.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:59:32.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:59:32.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:59:32.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:59:32.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:59:32.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:59:32.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:59:32.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:59:32.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: task.result() 2021-07-31T02:59:32.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:59:32.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:59:30.682745+0000 mgr.x (mgr.24542) 3097 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T02:59:32.304 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: cephadm 2021- 2021-07-31T02:59:32.304 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: 07-31T02:59:30.707144 2021-07-31T02:59:32.304 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 3098 : cephadm 2021-07-31T02:59:32.304 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: [ERR] 2021-07-31T02:59:32.304 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:59:32.305 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:59:32.305 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: yield 2021-07-31T02:59:32.305 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:59:32.306 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:59:32.306 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:59:32.306 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:59:32.306 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:59:32.307 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:59:32.307 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:59:32.307 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: task.result() 2021-07-31T02:59:32.307 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:59:32.308 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:59:30.707557+0000 mgr.x (mgr.24542) 3099 : cephadm [ERR] 2021-07-31T02:59:32.308 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:59:32.308 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:59:32.308 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:59:32.308 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: yield 2021-07-31T02:59:32.309 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:59:32.309 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:59:32.309 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:59:32.309 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:59:32.310 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:59:32.310 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:59:32.310 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:59:32.310 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:59:32.311 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:59:32.311 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: task.result() 2021-07-31T02:59:32.311 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:59:32.311 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: 2021-07-31T02:59:32.311 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:59:32.312 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: 2021-07-31T02:59:32.312 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:59:32.312 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:59:32.312 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:59:32.312 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:59:32.313 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:59:32.313 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:59:32.313 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:59:32.313 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:59:32.314 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:59:32.314 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:59:32.314 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:59:32.314 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:59:32.315 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:59:32.315 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:59:32.315 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:59:32.315 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:59:32.316 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:59:32.316 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:59:32.316 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:59:32.316 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:59:32.317 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:59:32.317 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:59:32.317 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:59:32.317 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: task.result() 2021-07-31T02:59:32.317 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:59:32.318 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:59:30.707557+0000 mgr.x (mgr.24542) 3099 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:59:32.318 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:59:32.318 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:59:32.318 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: yield 2021-07-31T02:59:32.319 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:59:32.319 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:59:32.319 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:59:32.319 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:59:32.319 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:59:32.320 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:59:32.320 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:59:32.320 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:59:32.320 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:59:32.320 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: task.result() 2021-07-31T02:59:32.321 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:59:32.321 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: 2021-07-31T02:59:32.321 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:59:32.321 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: 2021-07-31T02:59:32.322 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:59:32.322 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:59:32.322 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:59:32.322 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:59:32.322 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:59:32.323 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:59:32.323 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:59:32.323 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:59:32.323 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:59:32.323 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:59:32.324 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:59:32.324 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:59:32.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:59:32.325 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:59:32.325 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:59:32.325 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:59:32.326 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: raise 2021-07-31T02:59:32.326 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:59:32.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:59:32.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:59:32.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:59:32.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:59:32.328 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: cephadm 2021-07-31T02:59:30.716175+0000 mgr.x (mgr.24542) 3100 : cephadm [ERR] 2021-07-31T02:59:32.328 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:59:32.328 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:59:32.328 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: yield 2021-07-31T02:59:32.328 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:59:32.329 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:59:32.329 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:59:32.329 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:59:32.329 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:59:32.330 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:59:32.330 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:59:32.330 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:59:32.330 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:59:32.330 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: task.result() 2021-07-31T02:59:32.331 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:59:32.331 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: cephadm 2021-07 2021-07-31T02:59:32.331 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: -31T02:59:30.716562+0000 mgr.x (mgr.24542) 3101 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:59:32.331 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:59:32.331 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:59:32.332 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: yield 2021-07-31T02:59:32.332 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:59:32.332 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:59:32.333 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:59:32.333 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T02:59:32.333 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:59:32.333 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T02:59:32.333 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:59:32.334 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T02:59:32.334 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:59:32.334 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:59:32.335 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:59:32.335 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:59:32.335 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:59:32.335 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:59:32.336 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:59:32.336 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: raise 2021-07-31T02:59:32.336 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:59:32.336 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:59:32.336 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:59:32.337 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:59:32.337 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T02:59:32.337 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:59:30.716175+0000 mgr.x (mgr.24542) 3100 : cephadm [ERR] 2021-07-31T02:59:32.337 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:59:32.338 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:59:32.338 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: yield 2021-07-31T02:59:32.338 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:59:32.338 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:59:32.338 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:59:32.339 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:59:32.339 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:59:32.339 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:59:32.339 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:59:32.339 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:59:32.340 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:59:32.340 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: task.result() 2021-07-31T02:59:32.340 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:59:32.340 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: cephadm 2021-07-31T02:59:30.716562+0000 mgr.x (mgr.24542) 3101 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T02:59:32.341 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:59:32.341 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T02:59:32.341 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: yield 2021-07-31T02:59:32.341 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T02:59:32.341 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T02:59:32.342 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T02:59:32.342 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T02:59:32.342 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T02:59:32.342 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T02:59:32.342 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T02:59:32.343 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T02:59:32.343 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: task.result() 2021-07-31T02:59:32.343 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T02:59:32.344 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: 2021-07-31T02:59:32.344 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T02:59:32.344 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: 2021-07-31T02:59:32.344 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T02:59:32.345 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:59:32.345 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: return f(*arg) 2021-07-31T02:59:32.345 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:59:32.345 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:59:32.345 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:59:32.346 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:59:32.346 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:59:32.346 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:59:32.346 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:59:32.346 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T02:59:32.347 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:59:32.347 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: raise self._exception 2021-07-31T02:59:32.347 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:59:32.347 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:59:32.348 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:59:32.348 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T02:59:32.348 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:59:32.348 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: raise 2021-07-31T02:59:32.349 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:59:32.349 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T02:59:32.349 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:59:32.349 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:59:32.349 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:59:32.350 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[33655]: cluster 2021-07-31T02:59:30.834054+0000 mgr.x (mgr.24542) 3102 : cluster [DBG] pgmap v4176: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:32.350 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T02:59:32.350 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: task.result() 2021-07-31T02:59:32.351 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T02:59:32.351 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: 2021-07-31T02:59:32.351 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T02:59:32.352 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: 2021-07-31T02:59:32.352 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T02:59:32.352 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T02:59:32.353 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: return f(*arg) 2021-07-31T02:59:32.353 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T02:59:32.353 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T02:59:32.353 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T02:59:32.354 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T02:59:32.354 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T02:59:32.354 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T02:59:32.354 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T02:59:32.354 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T02:59:32.355 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T02:59:32.355 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: raise self._exception 2021-07-31T02:59:32.355 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T02:59:32.355 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T02:59:32.356 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T02:59:32.356 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T02:59:32.356 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T02:59:32.356 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: raise 2021-07-31T02:59:32.356 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T02:59:32.357 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T02:59:32.357 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T02:59:32.357 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T02:59:32.357 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T02:59:32.357 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: cluster 2021-07-31T02:59:30.834054+0000 2021-07-31T02:59:32.358 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:31 smithi045 conmon[39718]: mgr.x (mgr.24542) 3102 : cluster [DBG] pgmap v4176: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:34.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:33 smithi154 conmon[34553]: cluster 2021-07-31T02:59:32.834468+0000 mgr.x (mgr.24542) 3103 : cluster [DBG] pgmap v4177: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:34.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:33 smithi045 conmon[39718]: cluster 2021-07-31T02:59:32.834468+0000 mgr.x ( 2021-07-31T02:59:34.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:33 smithi045 conmon[39718]: mgr.24542) 3103 : cluster [DBG] pgmap v4177: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:34.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:33 smithi045 conmon[33655]: cluster 2021-07-31T02:59:32.834468+0000 mgr.x (mgr. 2021-07-31T02:59:34.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:33 smithi045 conmon[33655]: 24542) 3103 : cluster [DBG] pgmap v4177: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:36.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:35 smithi154 conmon[34553]: cluster 2021-07-31T02:59:34.835055+0000 mgr.x (mgr.24542) 3104 : cluster [DBG] pgmap v4178: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:36.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:35 smithi045 conmon[39718]: cluster 2021-07-31T02:59:34.835055+0000 mgr.x (mgr.24542) 3104 : cluster [DBG] pgmap v4178: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:36.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:35 smithi045 conmon[33655]: cluster 2021-07-31T02:59:34.835055+0000 mgr.x ( 2021-07-31T02:59:36.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:35 smithi045 conmon[33655]: mgr.24542) 3104 : cluster [DBG] pgmap v4178: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:37.517 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T02:59:37.794 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T02:59:37.794 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 74m ago 75m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T02:59:37.794 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 72m ago 74m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T02:59:37.795 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 72m ago 78m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T02:59:37.795 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 74m ago 80m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T02:59:37.795 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 74m ago 80m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T02:59:37.795 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 72m ago 78m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T02:59:37.795 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 74m ago 78m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T02:59:37.796 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 74m ago 75m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T02:59:37.796 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 72m ago 75m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T02:59:37.796 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 74m ago 78m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T02:59:37.796 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 74m ago 77m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T02:59:37.796 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 74m ago 77m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T02:59:37.797 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 74m ago 77m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T02:59:37.797 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 72m ago 76m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T02:59:37.797 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 72m ago 76m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T02:59:37.797 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 72m ago 76m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T02:59:37.798 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 72m ago 75m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T02:59:37.798 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 72m ago 75m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T02:59:38.131 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T02:59:38.131 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T02:59:38.131 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T02:59:38.131 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:59:38.132 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T02:59:38.132 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T02:59:38.132 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:59:38.132 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:59:38.132 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T02:59:38.133 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T02:59:38.133 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T02:59:38.133 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T02:59:38.133 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T02:59:38.133 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T02:59:38.134 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T02:59:38.134 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T02:59:38.134 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T02:59:38.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:37 smithi154 conmon[34553]: cluster 2021-07-31T02:59:36.835572+0000 mgr.x (mgr.24542) 3105 : cluster [DBG] pgmap v4179: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:38.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:37 smithi154 conmon[34553]: audit 2021-07-31T02:59:37.504790+0000 mgr.x (mgr.24542) 3106 : audit [DBG] from='client.28510 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:59:38.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:37 smithi045 conmon[33655]: cluster 2021-07-31T02:59:36. 2021-07-31T02:59:38.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:37 smithi045 conmon[33655]: 835572+0000 mgr.x (mgr.24542) 3105 : cluster [DBG] pgmap v4179: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:38.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:37 smithi045 conmon[33655]: audit 2021-07- 2021-07-31T02:59:38.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:37 smithi045 conmon[33655]: 31T02:59:37.504790+0000 mgr.x (mgr.24542) 3106 : 2021-07-31T02:59:38.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:37 smithi045 conmon[33655]: audit [DBG] from='client.28510 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:59:38.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:37 smithi045 conmon[39718]: cluster 2021-07-31T02:59: 2021-07-31T02:59:38.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:37 smithi045 conmon[39718]: 36.835572+0000 mgr.x (mgr.24542) 3105 : cluster [DBG] pgmap v4179: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:38.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:37 smithi045 conmon[39718]: audit 2021-07-31T02:59 2021-07-31T02:59:38.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:37 smithi045 conmon[39718]: :37.504790+0000 mgr.x (mgr.24542) 3106 : 2021-07-31T02:59:38.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:37 smithi045 conmon[39718]: audit [DBG] from='client.28510 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:59:39.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:38 smithi154 conmon[34553]: audit 2021-07-31T02:59:37.786187+0000 mgr.x (mgr.24542) 3107 : audit [DBG] from='client.28516 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:59:39.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:38 smithi154 conmon[34553]: audit 2021-07-31T02:59:38.130879+0000 2021-07-31T02:59:39.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:38 smithi154 conmon[34553]: mon.c (mon.1) 166 : audit [DBG] from='client.? 172.21.15.45:0/3821319083' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:59:39.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:38 smithi045 conmon[33655]: audit 2021-07-31T02:59:37.786187+0000 mgr.x (mgr.24542) 3107 : audit 2021-07-31T02:59:39.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:38 smithi045 conmon[33655]: [DBG] from='client.28516 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:59:39.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:38 smithi045 conmon[33655]: audit 2021-07-31T02:59:38.130879+0000 mon.c (mon.1) 166 : audit [DBG] from='client.? 172.21.15.45:0/3821319083' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:59:39.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:38 smithi045 conmon[39718]: audit 2021-07-31T02:59:37.786187+0000 mgr.x ( 2021-07-31T02:59:39.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:38 smithi045 conmon[39718]: mgr.24542) 3107 : audit [DBG] from='client.28516 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T02:59:39.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:38 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T02:59:39.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:38 smithi045 conmon[39718]: T02:59:38.130879+0000 mon.c (mon.1) 166 : audit [DBG] from='client.? 172.21.15.45:0/3821319083' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T02:59:40.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:39 smithi154 conmon[34553]: cluster 2021-07-31T02:59:38.835969+0000 mgr.x (mgr.24542) 3108 : cluster [DBG] pgmap v4180: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:40.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:39 smithi045 conmon[33655]: cluster 2021-07-31T02:59:38.835969+0000 mgr.x (mgr.24542 2021-07-31T02:59:40.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:39 smithi045 conmon[33655]: ) 3108 : cluster [DBG] pgmap v4180: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:40.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:39 smithi045 conmon[39718]: cluster 2021-07-31T02:59:38.835969+0000 mgr.x ( 2021-07-31T02:59:40.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:39 smithi045 conmon[39718]: mgr.24542) 3108 : cluster [DBG] pgmap v4180: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:42.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:41 smithi154 conmon[34553]: cluster 2021-07-31T02:59:40.836436+0000 mgr.x (mgr.24542) 3109 : cluster [DBG] pgmap v4181: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:42.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:41 smithi045 conmon[39718]: cluster 2021-07-31T02:59:40.836436+0000 mgr.x ( 2021-07-31T02:59:42.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:41 smithi045 conmon[39718]: mgr.24542) 3109 : cluster [DBG] pgmap v4181: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:42.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:41 smithi045 conmon[33655]: cluster 2021-07-31T02:59:40.836436+0000 mgr.x (mgr.24542) 3109 : 2021-07-31T02:59:42.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:41 smithi045 conmon[33655]: cluster [DBG] pgmap v4181: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:44.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:43 smithi154 conmon[34553]: cluster 2021-07-31T02:59:42.836982+0000 mgr.x (mgr.24542) 3110 : cluster [DBG] pgmap v4182: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:44.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:43 smithi045 conmon[39718]: cluster 2021-07-31T02:59:42.836982+0000 mgr.x (mgr.24542 2021-07-31T02:59:44.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:43 smithi045 conmon[39718]: ) 3110 : cluster [DBG] pgmap v4182: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:44.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:43 smithi045 conmon[33655]: cluster 2021-07-31T02:59:42.836982+0000 mgr.x (mgr.24542) 2021-07-31T02:59:44.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:43 smithi045 conmon[33655]: 3110 : cluster [DBG] pgmap v4182: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:46.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:45 smithi154 conmon[34553]: cluster 2021-07-31T02:59:44.837524+0000 mgr.x (mgr.24542) 3111 : cluster [DBG] pgmap v4183: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:46.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:45 smithi045 conmon[39718]: cluster 2021-07-31T02:59:44.837524 2021-07-31T02:59:46.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:45 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 3111 : cluster [DBG] pgmap v4183: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:46.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:45 smithi045 conmon[33655]: cluster 2021-07-31T02:59:44.837524+0000 mgr.x (mgr.24542) 3111 : cluster 2021-07-31T02:59:46.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:45 smithi045 conmon[33655]: [DBG] pgmap v4183: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:48.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:47 smithi154 conmon[34553]: cluster 2021-07-31T02:59:46.838075+0000 mgr.x (mgr.24542) 3112 : cluster [DBG] pgmap v4184: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:48.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:47 smithi045 conmon[39718]: cluster 2021-07-31T02:59:46.838075+0000 mgr.x (mgr 2021-07-31T02:59:48.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:47 smithi045 conmon[39718]: .24542) 3112 : cluster [DBG] pgmap v4184: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:48.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:47 smithi045 conmon[33655]: cluster 2021-07-31T02:59:46.838075+0000 mgr.x (mgr.24542) 3112 2021-07-31T02:59:48.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:47 smithi045 conmon[33655]: : cluster [DBG] pgmap v4184: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:50.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:49 smithi154 conmon[34553]: cluster 2021-07-31T02:59:48.838610+0000 mgr.x (mgr.24542) 3113 : cluster [DBG] 2021-07-31T02:59:50.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:49 smithi154 conmon[34553]: pgmap v4185: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:50.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:49 smithi045 conmon[39718]: cluster 2021-07-31T02:59:48.838610+0000 mgr.x ( 2021-07-31T02:59:50.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:49 smithi045 conmon[39718]: mgr.24542) 3113 : cluster [DBG] pgmap v4185: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:50.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:49 smithi045 conmon[33655]: cluster 2021-07-31T02:59:48.838610+0000 mgr.x (mgr.24542) 3113 : cluster [DBG] pgmap v4185: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:52.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:51 smithi154 conmon[34553]: cluster 2021-07-31T02:59:50.839301+0000 mgr.x (mgr.24542) 3114 : cluster [DBG] pgmap v4186: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:52.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:51 smithi045 conmon[33655]: cluster 2021-07-31T02:59:50.839301+0000 mgr.x (mgr.24542) 3114 : cluster [DBG] pgmap v4186: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:52.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:51 smithi045 conmon[39718]: cluster 2021-07-31T02:59:50 2021-07-31T02:59:52.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:51 smithi045 conmon[39718]: .839301+0000 mgr.x (mgr.24542) 3114 : cluster [DBG] pgmap v4186: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:54.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:53 smithi154 conmon[34553]: cluster 2021-07-31T02:59:52.839677+0000 mgr.x (mgr.24542) 3115 : cluster [DBG] pgmap v4187: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:54.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:53 smithi045 conmon[39718]: cluster 2021-07-31T02:59:52. 2021-07-31T02:59:54.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:53 smithi045 conmon[39718]: 839677+0000 mgr.x (mgr.24542) 3115 : cluster [DBG] pgmap v4187: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:54.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:53 smithi045 conmon[33655]: cluster 2021-07-31T02:59:52.839677+0000 mgr.x (mgr.24542) 3115 : cluster [DBG] pgmap v4187: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:56.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:55 smithi154 conmon[34553]: cluster 2021-07-31T02:59:54.840149+0000 mgr.x (mgr.24542) 3116 : cluster [DBG] pgmap v4188: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:56.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:55 smithi045 conmon[33655]: cluster 2021-07-31T02:59:54.840149+0000 mgr.x (mgr.24542) 3116 : cluster [DBG] pgmap v4188: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:56.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:55 smithi045 conmon[39718]: cluster 2021-07-31T02:59:54.840149+0000 mgr.x ( 2021-07-31T02:59:56.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:55 smithi045 conmon[39718]: mgr.24542) 3116 : cluster [DBG] pgmap v4188: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:58.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:57 smithi154 conmon[34553]: cluster 2021-07-31T02:59:56.840698+0000 mgr.x (mgr.24542) 3117 : cluster [DBG] pgmap v4189: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:58.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:57 smithi045 conmon[33655]: cluster 2021-07-31T02:59:56.840698+0000 mgr.x (mgr.24542) 3117 2021-07-31T02:59:58.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:57 smithi045 conmon[33655]: : cluster [DBG] pgmap v4189: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T02:59:58.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:57 smithi045 conmon[39718]: cluster 2021-07-31T02:59:56.840698+0000 mgr.x (mgr.24542 2021-07-31T02:59:58.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:57 smithi045 conmon[39718]: ) 3117 : cluster [DBG] pgmap v4189: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:00.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 02:59:59 smithi154 conmon[34553]: cluster 2021-07-31T02:59:58.841212+0000 mgr.x (mgr.24542) 3118 : cluster [DBG] pgmap v4190: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:00.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:59 smithi045 conmon[33655]: cluster 2021-07-31T02:59:58.841212+0000 mgr.x ( 2021-07-31T03:00:00.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 02:59:59 smithi045 conmon[33655]: mgr.24542) 3118 : cluster [DBG] pgmap v4190: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:00.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:59 smithi045 conmon[39718]: cluster 2021-07-31T02:59:58.841212+0000 mgr.x (mgr 2021-07-31T03:00:00.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 02:59:59 smithi045 conmon[39718]: .24542) 3118 : cluster [DBG] pgmap v4190: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:01.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:00 smithi154 conmon[34553]: cluster 2021-07-31T03:00:00.000212+0000 mon.a (mon.0) 3235 : cluster [INF] overall HEALTH_OK 2021-07-31T03:00:01.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:00 smithi045 conmon[33655]: cluster 2021-07-31T03:00:00.000212 2021-07-31T03:00:01.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:00 smithi045 conmon[33655]: +0000 mon.a (mon.0) 3235 : cluster [INF] overall HEALTH_OK 2021-07-31T03:00:01.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:00 smithi045 conmon[39718]: cluster 2021-07-31T03:00:00.000212+0000 mon.a (mon. 2021-07-31T03:00:01.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:00 smithi045 conmon[39718]: 0) 3235 : cluster [INF] overall HEALTH_OK 2021-07-31T03:00:02.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:01 smithi154 conmon[34553]: cluster 2021-07-31T03:00:00.841670+0000 mgr.x (mgr.24542) 3119 : cluster [DBG] pgmap v4191: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:02.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:01 smithi045 conmon[39718]: cluster 2021-07-31T03:00:00.841670+0000 mgr.x (mgr.24542 2021-07-31T03:00:02.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:01 smithi045 conmon[39718]: ) 3119 : cluster [DBG] pgmap v4191: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:02.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:01 smithi045 conmon[33655]: cluster 2021-07-31T03:00:00.841670 2021-07-31T03:00:02.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:01 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 3119 : cluster [DBG] pgmap v4191: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:04.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:03 smithi154 conmon[34553]: cluster 2021-07-31T03:00:02.842200+0000 mgr.x (mgr.24542) 3120 : cluster [DBG] pgmap v4192: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:04.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:03 smithi045 conmon[39718]: cluster 2021-07-31T03:00:02.842200 2021-07-31T03:00:04.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:03 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 3120 : cluster [DBG] pgmap v4192: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:04.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:03 smithi045 conmon[33655]: cluster 2021-07-31T03:00:02.842200+0000 mgr.x ( 2021-07-31T03:00:04.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:03 smithi045 conmon[33655]: mgr.24542) 3120 : cluster [DBG] pgmap v4192: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:06.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:05 smithi154 conmon[34553]: cluster 2021-07-31T03:00:04.842799+0000 mgr.x (mgr.24542) 3121 : cluster [DBG] pgmap v4193: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:06.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:05 smithi045 conmon[39718]: cluster 2021-07-31T03:00 2021-07-31T03:00:06.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:05 smithi045 conmon[39718]: :04.842799+0000 mgr.x (mgr.24542) 3121 : cluster [DBG] pgmap v4193: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:06.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:05 smithi045 conmon[33655]: cluster 2021-07-31T03:00:04.842799+0000 mgr.x (mgr.24542) 3121 : cluster [DBG] pgmap v4193: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:08.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:07 smithi154 conmon[34553]: cluster 2021-07-31T03:00:06.843337+0000 mgr.x (mgr.24542) 3122 : cluster [DBG] pgmap v4194: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:08.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:07 smithi045 conmon[39718]: cluster 2021-07-31T03:00:06.843337+0000 mgr.x (mgr. 2021-07-31T03:00:08.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:07 smithi045 conmon[39718]: 24542) 3122 : cluster [DBG] pgmap v4194: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:08.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:07 smithi045 conmon[33655]: cluster 2021-07-31T03:00:06.843337+0000 mgr.x (mgr.24542) 2021-07-31T03:00:08.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:07 smithi045 conmon[33655]: 3122 : cluster [DBG] pgmap v4194: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:08.443 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:00:08.723 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:00:08.723 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 75m ago 75m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:00:08.723 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 72m ago 75m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:00:08.724 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 72m ago 79m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:00:08.724 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 75m ago 81m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:00:08.724 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 75m ago 81m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:00:08.724 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 72m ago 79m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:00:08.725 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 75m ago 79m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:00:08.725 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 75m ago 76m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:00:08.725 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 72m ago 75m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:00:08.725 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 75m ago 78m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:00:08.725 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 75m ago 78m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:00:08.726 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 75m ago 78m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:00:08.726 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 75m ago 77m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:00:08.726 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 72m ago 77m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:00:08.726 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 72m ago 77m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:00:08.727 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 72m ago 76m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:00:08.727 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 72m ago 76m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:00:08.727 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 72m ago 76m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:00:09.062 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:00:09.062 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:00:09.062 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:00:09.063 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:00:09.063 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:00:09.063 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:00:09.063 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:00:09.063 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:00:09.064 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:00:09.064 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:00:09.064 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:00:09.064 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:00:09.065 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:00:09.065 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:00:09.065 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:00:09.065 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:00:09.066 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:00:09.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:08 smithi154 conmon[34553]: audit 2021-07-31T03:00:08.430906+0000 mgr.x (mgr.24542) 3123 : audit [DBG] from='client.28528 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:00:09.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:08 smithi045 conmon[39718]: audit 2021-07-31T03:00:08.430906+0000 2021-07-31T03:00:09.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:08 smithi045 conmon[39718]: mgr.x (mgr.24542) 3123 : audit [DBG] from='client.28528 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:00:09.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:08 smithi045 conmon[33655]: audit 2021-07-31T03:00:08.430906+0000 mgr.x (mgr. 2021-07-31T03:00:09.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:08 smithi045 conmon[33655]: 24542) 3123 : audit [DBG] from='client.28528 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:00:10.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:09 smithi154 conmon[34553]: audit 2021-07-31T03:00:08.714827+0000 mgr.x (mgr.24542) 3124 : audit [DBG] from='client.18594 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:00:10.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:09 smithi154 conmon[34553]: cluster 2021-07-31T03:00:08.843741+0000 mgr.x (mgr.24542) 3125 : cluster [DBG] pgmap v4195: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:10.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:09 smithi154 conmon[34553]: audit 2021-07-31T03:00:09.061654+0000 mon.c (mon.1) 167 : audit [DBG] from='client.? 172.21.15.45:0/4022141142' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:00:10.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:09 smithi045 conmon[39718]: audit 2021-07-31T03:00:08.714827+0000 2021-07-31T03:00:10.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:09 smithi045 conmon[39718]: mgr.x (mgr.24542) 3124 : audit [DBG] from='client.18594 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:00:10.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:09 smithi045 conmon[39718]: cluster 2021-07-31T 2021-07-31T03:00:10.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:09 smithi045 conmon[39718]: 03:00:08.843741+0000 mgr.x (mgr.24542) 3125 : 2021-07-31T03:00:10.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:09 smithi045 conmon[39718]: cluster [DBG] pgmap v4195: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:10.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:09 smithi045 conmon[39718]: audit 2021-07-31T03:00:10.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:09 smithi045 conmon[39718]: 2021-07-31T03:00:09. 2021-07-31T03:00:10.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:09 smithi045 conmon[39718]: 061654+0000 mon.c (mon.1) 2021-07-31T03:00:10.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:09 smithi045 conmon[39718]: 167 : audit [DBG] from='client.? 172.21.15.45:0/4022141142' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:00:10.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:09 smithi045 conmon[33655]: audit 2021-07-31T03:00:08.714827+0000 2021-07-31T03:00:10.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:09 smithi045 conmon[33655]: mgr.x (mgr.24542) 3124 : audit [DBG] from='client.18594 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:00:10.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:09 smithi045 conmon[33655]: cluster 2021-07-31T03:00:08.843741+0000 mgr.x 2021-07-31T03:00:10.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:09 smithi045 conmon[33655]: (mgr. 2021-07-31T03:00:10.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:09 smithi045 conmon[33655]: 24542) 3125 : cluster [DBG] 2021-07-31T03:00:10.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:09 smithi045 conmon[33655]: pgmap v4195: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:10.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:09 smithi045 conmon[33655]: audit 2021- 2021-07-31T03:00:10.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:09 smithi045 conmon[33655]: 07-31T03:00:09.061654 2021-07-31T03:00:10.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:09 smithi045 conmon[33655]: +0000 mon.c (mon.1) 2021-07-31T03:00:10.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:09 smithi045 conmon[33655]: 167 : audit [DBG] from='client.? 172.21.15.45:0/4022141142' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:00:12.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:11 smithi154 conmon[34553]: cluster 2021-07-31T03:00:10.844138+0000 mgr.x (mgr.24542) 3126 : cluster [DBG] pgmap v4196: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:12.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:11 smithi045 conmon[39718]: cluster 2021-07-31T03:00:10.844138+0000 mgr.x (mgr.24542) 3126 : cluster [DBG] pgmap v4196: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:12.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:11 smithi045 conmon[33655]: cluster 2021-07-31T03:00:10.844138+0000 mgr.x (mgr.24542) 3126 : cluster [DBG] pgmap v4196: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:14.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:13 smithi154 conmon[34553]: cluster 2021-07-31T03:00:12.844570+0000 mgr.x (mgr.24542) 3127 : cluster [DBG] pgmap v4197: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:14.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:13 smithi045 conmon[39718]: cluster 2021-07-31T03:00:12.844570+0000 2021-07-31T03:00:14.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:13 smithi045 conmon[39718]: mgr.x (mgr.24542) 3127 : cluster [DBG] pgmap v4197: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:14.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:13 smithi045 conmon[33655]: cluster 2021-07-31T03:00:12.844570+0000 mgr.x (mgr 2021-07-31T03:00:14.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:13 smithi045 conmon[33655]: .24542) 3127 : cluster [DBG] pgmap v4197: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:16.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:15 smithi154 conmon[34553]: cluster 2021-07-31T03:00:14.845005+0000 mgr.x (mgr.24542) 3128 : cluster [DBG] pgmap v4198: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:16.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:15 smithi045 conmon[39718]: cluster 2021-07-31T03:00:14.845005+0000 mgr.x (mgr.24542) 3128 : cluster [DBG] pgmap v4198: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:16.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:15 smithi045 conmon[33655]: cluster 2021-07-31T03:00:14.845005 2021-07-31T03:00:16.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:15 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 3128 : cluster [DBG] pgmap v4198: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:18.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:17 smithi154 conmon[34553]: cluster 2021-07-31T03:00:16.845529+0000 mgr.x (mgr.24542) 3129 : cluster [DBG] pgmap v4199: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:18.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:17 smithi045 conmon[39718]: cluster 2021-07-31T03:00:16.845529+0000 mgr.x (mgr.24542) 3129 : cluster [DBG] pgmap v4199: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:18.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:17 smithi045 conmon[33655]: cluster 2021-07-31T03:00:16.845529+0000 mgr.x ( 2021-07-31T03:00:18.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:17 smithi045 conmon[33655]: mgr.24542) 3129 : cluster [DBG] pgmap v4199: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:20.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:19 smithi154 conmon[34553]: cluster 2021-07-31T03:00:18.846112+0000 mgr.x (mgr.24542) 3130 : cluster [DBG] pgmap v4200: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:20.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:19 smithi045 conmon[39718]: cluster 2021-07-31T03:00:18.846112+0000 2021-07-31T03:00:20.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:19 smithi045 conmon[39718]: mgr.x (mgr.24542) 3130 : cluster [DBG] pgmap v4200: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:20.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:19 smithi045 conmon[33655]: cluster 2021-07-31T03:00:18. 2021-07-31T03:00:20.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:19 smithi045 conmon[33655]: 846112+0000 mgr.x (mgr.24542) 3130 : cluster [DBG] pgmap v4200: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:22.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:21 smithi154 conmon[34553]: cluster 2021-07-31T03:00:20.846661+0000 mgr.x (mgr.24542) 3131 : cluster [DBG] pgmap v4201: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:22.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:21 smithi045 conmon[39718]: cluster 2021-07-31T03:00:20.846661+0000 mgr.x (mgr.24542) 3131 : cluster [DBG] pgmap v4201: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:22.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:21 smithi045 conmon[33655]: cluster 2021-07-31T03:00: 2021-07-31T03:00:22.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:21 smithi045 conmon[33655]: 20.846661+0000 mgr.x (mgr.24542) 3131 : cluster [DBG] pgmap v4201: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:24.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:23 smithi154 conmon[34553]: cluster 2021-07-31T03:00:22.847125+0000 mgr.x (mgr.24542) 3132 : cluster [DBG] pgmap v4202: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:24.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:23 smithi045 conmon[39718]: cluster 2021-07-31T03:00:22.847125+0000 mgr.x (mgr.24542) 3132 : cluster [DBG] pgmap v4202: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:24.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:23 smithi045 conmon[33655]: cluster 2021-07-31T03:00:22.847125+0000 2021-07-31T03:00:24.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:23 smithi045 conmon[33655]: mgr.x (mgr.24542) 3132 : cluster [DBG] pgmap v4202: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:26.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:25 smithi154 conmon[34553]: cluster 2021-07-31T03:00:24.847706+0000 mgr.x (mgr.24542) 3133 : cluster [DBG] 2021-07-31T03:00:26.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:25 smithi154 conmon[34553]: pgmap v4203: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:26.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:25 smithi045 conmon[39718]: cluster 2021-07-31T03:00:24 2021-07-31T03:00:26.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:25 smithi045 conmon[39718]: .847706+0000 mgr.x (mgr.24542) 3133 : cluster [DBG] pgmap v4203: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:26.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:25 smithi045 conmon[33655]: cluster 2021-07-31T03:00:24.847706+0000 mgr.x (mgr.24542) 3133 : cluster 2021-07-31T03:00:26.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:25 smithi045 conmon[33655]: [DBG] pgmap v4203: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:28.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:27 smithi154 conmon[34553]: cluster 2021-07-31T03:00:26.848098+0000 mgr.x (mgr.24542) 3134 : cluster [DBG] pgmap v4204: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:28.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:27 smithi045 conmon[39718]: cluster 2021-07-31T03:00:26.848098+0000 mgr.x (mgr.24542 2021-07-31T03:00:28.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:27 smithi045 conmon[39718]: ) 3134 : cluster [DBG] pgmap v4204: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:28.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:27 smithi045 conmon[33655]: cluster 2021-07-31T03:00:26.848098+0000 mgr.x (mgr.24542 2021-07-31T03:00:28.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:27 smithi045 conmon[33655]: ) 3134 : cluster [DBG] pgmap v4204: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:28 smithi154 conmon[34553]: audit 2021-07-31T03:00:28.226898+0000 mon.b (mon.2) 440 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:00:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:28 smithi154 conmon[34553]: audit 2021-07-31T03:00:28.227197 2021-07-31T03:00:29.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:28 smithi154 conmon[34553]: +0000 mon.b (mon.2) 441 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:00:29.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:28 smithi154 conmon[34553]: audit 2021-07-31T03:00:28.227682+0000 mon.a (mon.0) 3236 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:00:29.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:28 smithi154 conmon[34553]: audit 2021-07-31T03:00:28.228093+0000 mon.a (mon.0) 3237 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:00:29.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:28 smithi154 conmon[34553]: 2021-07-31T03:00:29.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:28 smithi045 conmon[39718]: audit 2021-07-31T03:00 2021-07-31T03:00:29.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:28 smithi045 conmon[39718]: :28.226898+0000 mon.b (mon.2) 440 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:00:29.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:28 smithi045 conmon[39718]: audit 2021-07-31T03:00:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:28 smithi045 conmon[39718]: 2021-07-31T03:00 2021-07-31T03:00:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:28 smithi045 conmon[39718]: :28.227197+0000 mon.b ( 2021-07-31T03:00:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:28 smithi045 conmon[39718]: mon.2) 441 : audit [INF] 2021-07-31T03:00:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:28 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:00:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:28 smithi045 conmon[39718]: audit 2021- 2021-07-31T03:00:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:28 smithi045 conmon[39718]: 07-31T03:00:28 2021-07-31T03:00:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:28 smithi045 conmon[39718]: .227682+0000 mon.a (mon.0 2021-07-31T03:00:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:28 smithi045 conmon[39718]: ) 3236 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:00:29.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:28 smithi045 conmon[39718]: 2021-07-31T03:00:29.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:28 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T03:00:29.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:28 smithi045 conmon[39718]: T03:00:28.228093+0000 mon.a 2021-07-31T03:00:29.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:28 smithi045 conmon[39718]: (mon.0) 3237 : audit 2021-07-31T03:00:29.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:28 smithi045 conmon[39718]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:00:29.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:28 smithi045 conmon[33655]: audit 2021-07-31T03:00:28.226898+0000 mon.b (mon.2) 2021-07-31T03:00:29.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:28 smithi045 conmon[33655]: 440 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:00:29.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:28 smithi045 conmon[33655]: audit 2021-07-31T03:00:28.227197+0000 mon.b (mon.2) 441 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:00:29.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:28 smithi045 conmon[33655]: audit 2021-07-31T03:00:28.227682+0000 mon.a (mon.0) 2021-07-31T03:00:29.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:28 smithi045 conmon[33655]: 3236 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:00:29.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:28 smithi045 conmon[33655]: audit 2021-07-31T03:00:28.228093+0000 mon.a (mon.0) 3237 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:00:30.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:29 smithi154 conmon[34553]: cluster 2021-07-31T03:00:28.848611+0000 mgr.x (mgr.24542) 3135 : cluster [DBG] pgmap v4205: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:30.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:29 smithi045 conmon[39718]: cluster 2021-07-31T03:00:28.848611+0000 2021-07-31T03:00:30.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:29 smithi045 conmon[39718]: mgr.x (mgr.24542) 3135 : cluster [DBG] pgmap v4205: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:30.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:29 smithi045 conmon[33655]: cluster 2021-07-31T03:00:28.848611+0000 mgr.x (mgr.24542 2021-07-31T03:00:30.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:29 smithi045 conmon[33655]: ) 3135 : cluster [DBG] pgmap v4205: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:31.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[34553]: audit 2021-07-31T03:00:30.717382+0000 mon.b (mon.2) 442 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:00:31.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[34553]: audit 2021-07-31T03:00:30.718954+0000 mon.b (mon.2) 443 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:00:31.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[34553]: audit 2021-07-31T03 2021-07-31T03:00:31.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[34553]: :00:30.720086+0000 mon.b (mon.2) 444 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:00:31.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: debug 2021-07-31T03:00:30.762+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:00:31.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:00:31.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:00:31.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: yield 2021-07-31T03:00:31.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:00:31.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:00:31.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:00:31.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:00:31.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:00:31.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:00:31.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:00:31.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:00:31.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:00:31.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: task.result() 2021-07-31T03:00:31.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:00:31.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: debug 2021-07-31T03:00:30.774+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:00:31.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:00:31.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:00:31.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: yield 2021-07-31T03:00:31.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:00:31.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:00:31.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:00:31.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:00:31.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:00:31.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:00:31.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:00:31.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:00:31.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:00:31.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: task.result() 2021-07-31T03:00:31.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:00:31.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: debug 2021-07-31T03:00:30.799+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:00:31.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:00:31.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:00:31.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: yield 2021-07-31T03:00:31.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:00:31.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:00:31.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:00:31.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:00:31.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:00:31.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:00:31.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:00:31.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:00:31.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:00:31.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: task.result() 2021-07-31T03:00:31.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:00:31.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: debug 2021-07-31T03:00:30.800+0000 7fcb9095b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:00:31.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:00:31.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:00:31.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: yield 2021-07-31T03:00:31.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:00:31.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:00:31.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:00:31.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:00:31.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:00:31.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:00:31.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:00:31.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:00:31.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:00:31.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: task.result() 2021-07-31T03:00:31.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:00:31.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: 2021-07-31T03:00:31.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:00:31.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: 2021-07-31T03:00:31.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:00:31.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:00:31.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:00:31.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:00:31.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:00:31.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:00:31.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:00:31.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:00:31.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:00:31.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:00:31.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:00:31.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:00:31.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:00:31.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:00:31.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:00:31.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:00:31.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:00:31.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:00:31.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: raise 2021-07-31T03:00:31.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:00:31.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:00:31.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:00:31.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:00:31.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:00:31.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: debug 2021-07-31T03:00:30.808+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:00:31.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:00:31.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:00:31.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: yield 2021-07-31T03:00:31.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:00:31.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:00:31.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:00:31.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:00:31.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:00:31.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:00:31.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:00:31.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:00:31.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:00:31.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: task.result() 2021-07-31T03:00:31.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:00:31.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: debug 2021-07-31T03:00:30.809+0000 7fcb9115c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:00:31.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:00:31.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:00:31.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: yield 2021-07-31T03:00:31.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:00:31.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:00:31.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:00:31.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:00:31.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:00:31.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:00:31.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:00:31.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:00:31.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:00:31.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: task.result() 2021-07-31T03:00:31.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:00:31.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: 2021-07-31T03:00:31.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:00:31.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: 2021-07-31T03:00:31.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:00:31.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:00:31.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:00:31.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:00:31.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:00:31.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:00:31.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:00:31.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:00:31.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:00:31.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:00:31.035 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:00:31.035 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:00:31.035 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:00:31.035 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:00:31.035 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:00:31.036 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:00:31.036 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:00:31.036 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:00:31.036 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: raise 2021-07-31T03:00:31.037 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:00:31.037 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:00:31.037 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:00:31.037 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:00:31.037 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:00:30 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:00:31.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:30 smithi045 conmon[33655]: audit 2021-07-31T03:00:30.717382+0000 mon.b (mon. 2021-07-31T03:00:31.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:30 smithi045 conmon[33655]: 2) 442 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:00:31.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:30 smithi045 conmon[33655]: audit 2021-07-31T03:00:30.718954+0000 mon.b (mon.2) 443 : audit [DBG] 2021-07-31T03:00:31.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:30 smithi045 conmon[33655]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:00:31.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:30 smithi045 conmon[33655]: audit 2021 2021-07-31T03:00:31.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:30 smithi045 conmon[33655]: -07-31T03:00:30.720086+0000 mon.b (mon.2 2021-07-31T03:00:31.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:30 smithi045 conmon[33655]: ) 444 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:00:31.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:30 smithi045 conmon[39718]: audit 2021-07-31T03:00 2021-07-31T03:00:31.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:30 smithi045 conmon[39718]: :30.717382+0000 mon.b (mon.2) 442 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:00:31.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:30 smithi045 conmon[39718]: audit 2021-07-31T03:00:31.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:30 smithi045 conmon[39718]: 2021-07-31T03:00:30.718954+0000 mon.b 2021-07-31T03:00:31.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:30 smithi045 conmon[39718]: (mon.2) 443 : audit [DBG] 2021-07-31T03:00:31.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:30 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:00:31.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:30 smithi045 conmon[39718]: audit 2021-07 2021-07-31T03:00:31.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:30 smithi045 conmon[39718]: -31T03:00:30.720086+0000 mon.b (mon.2) 2021-07-31T03:00:31.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:30 smithi045 conmon[39718]: 444 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:00:32.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: cephadm 2021-07-31T03:00:30.720854+0000 mgr.x (mgr.24542) 3136 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:00:32.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: cephadm 2021-07-31T03:00:30.763773+0000 mgr.x (mgr.24542) 3137 : cephadm [ERR] 2021-07-31T03:00:32.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:00:32.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:00:32.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: yield 2021-07-31T03:00:32.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:00:32.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:00:32.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:00:32.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:00:32.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:00:32.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:00:32.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:00:32.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:00:32.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:00:32.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: task.result() 2021-07-31T03:00:32.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:00:32.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: cephadm 2021-07-31T03:00:30.764249+0000 mgr.x (mgr.24542) 3138 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:00:32.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: cephadm 2021-07-31T03:00:30.775312+0000 mgr.x (mgr.24542) 3139 : cephadm [ERR] 2021-07-31T03:00:32.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:00:32.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:00:32.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: yield 2021-07-31T03:00:32.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:00:32.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:00:32.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:00:32.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:00:32.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:00:32.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:00:32.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:00:32.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:00:32.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:00:32.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: task.result() 2021-07-31T03:00:32.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:00:32.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: cephadm 2021-07-31T03:00:30.776323+0000 mgr.x (mgr.24542) 3140 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:00:32.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: cephadm 2021-07-31T03:00:30.800752+0000 mgr.x (mgr.24542) 3141 : cephadm [ERR] 2021-07-31T03:00:32.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: 2021-07-31T03:00:32.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:00:32.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:00:32.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: yield 2021-07-31T03:00:32.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:00:32.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:00:32.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:00:32.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:00:32.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:00:32.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:00:32.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:00:32.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:00:32.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:00:32.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: task.result() 2021-07-31T03:00:32.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:00:32.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: cephadm 2021-07-31T03:00:30.801218+0000 mgr.x (mgr.24542) 3142 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:00:32.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:00:32.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:00:32.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: yield 2021-07-31T03:00:32.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:00:32.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:00:32.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:00:32.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:00:32.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:00:32.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:00:32.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:00:32.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:00:32.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:00:32.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: task.result() 2021-07-31T03:00:32.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:00:32.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: 2021-07-31T03:00:32.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:00:32.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: 2021-07-31T03:00:32.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:00:32.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:00:32.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:00:32.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:00:32.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:00:32.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:00:32.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:00:32.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:00:32.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:00:32.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:00:32.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:00:32.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:00:32.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:00:32.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:00:32.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:00:32.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:00:32.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:00:32.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:00:32.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: raise 2021-07-31T03:00:32.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:00:32.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:00:32.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:00:32.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:00:32.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:00:32.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: cephadm 2021-07-31T03:00:30.809698+0000 mgr.x (mgr.24542) 3143 : cephadm [ERR] 2021-07-31T03:00:32.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:00:32.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:00:32.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: yield 2021-07-31T03:00:32.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:00:32.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:00:32.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:00:32.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:00:32.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:00:32.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:00:32.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:00:32.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:00:32.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:00:32.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: task.result() 2021-07-31T03:00:32.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:00:32.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: cephadm 2021-07-31T03:00:30.810272+0000 mgr.x (mgr.24542) 3144 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:00:32.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:00:32.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:00:32.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: yield 2021-07-31T03:00:32.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:00:32.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:00:32.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:00:32.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:00:32.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:00:32.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:00:32.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:00:32.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:00:32.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:00:32.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: task.result() 2021-07-31T03:00:32.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:00:32.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: 2021-07-31T03:00:32.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:00:32.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: 2021-07-31T03:00:32.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:00:32.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:00:32.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:00:32.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:00:32.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:00:32.285 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:00:32.285 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:00:32.286 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:00:32.286 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:00:32.287 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:00:32.287 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:00:32.288 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:00:32.288 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:00:32.289 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:00:32.289 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:00:32.290 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:00:32.290 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:00:32.290 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:00:32.291 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: raise 2021-07-31T03:00:32.291 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:00:32.292 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:00:32.292 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:00:32.292 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:00:32.293 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:00:32.293 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:31 smithi154 conmon[34553]: cluster 2021-07-31T03:00:30.849035+0000 mgr.x (mgr.24542) 3145 : cluster [DBG] pgmap v4206: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:32.294 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: cephadm 2021-07-31T03:00:30.720854+0000 mgr.x (mgr.24542 2021-07-31T03:00:32.294 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: ) 3136 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:00:32.294 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: cephadm 2021-07-31T03:00:30.763773+0000 mgr.x (mgr.24542) 3137 : cephadm [ERR] 2021-07-31T03:00:32.295 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:00:32.295 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:00:32.295 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: yield 2021-07-31T03:00:32.295 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:00:32.296 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:00:32.296 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:00:32.296 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:00:32.296 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:00:32.296 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:00:32.297 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:00:32.297 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:00:32.297 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:00:32.297 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: task.result() 2021-07-31T03:00:32.297 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:00:32.298 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: cephadm 2021-07-31T03:00:30.764249+0000 mgr.x (mgr.24542) 3138 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:00:32.298 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: cephadm 2021-07-31T03:00:30.775312+0000 mgr.x (mgr.24542) 3139 : cephadm [ERR] 2021-07-31T03:00:32.298 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: 2021-07-31T03:00:32.298 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:00:32.299 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:00:32.299 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: yield 2021-07-31T03:00:32.299 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:00:32.299 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:00:32.300 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:00:32.300 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:00:32.300 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:00:32.300 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:00:32.300 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:00:32.301 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:00:32.301 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:00:32.301 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: task.result() 2021-07-31T03:00:32.301 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:00:32.302 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: cephadm 2021-07-31T03:00 2021-07-31T03:00:32.302 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: :30.776323+0000 mgr.x (mgr.24542) 3140 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:00:32.302 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: cephadm 2021-07-31T03:00:30.800752+0000 mgr.x (mgr.24542) 3141 : cephadm [ERR] 2021-07-31T03:00:32.302 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:00:32.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: cephadm 2021-07-31T03:00:30.720854+0000 mgr.x 2021-07-31T03:00:32.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: (mgr.24542) 3136 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:00:32.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: cephadm 2021-07-31 2021-07-31T03:00:32.304 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: T03:00:30.763773+0000 mgr.x (mgr.24542) 3137 : cephadm [ERR] 2021-07-31T03:00:32.304 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: 2021-07-31T03:00:32.305 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:00:32.305 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:00:32.305 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: yield 2021-07-31T03:00:32.306 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:00:32.306 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:00:32.307 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:00:32.307 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:00:32.307 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:00:32.307 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:00:32.308 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:00:32.308 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:00:32.308 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:00:32.308 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: task.result() 2021-07-31T03:00:32.308 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:00:32.309 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: cephadm 2021-07-31T03:00:30.764249+0000 mgr.x (mgr.24542) 3138 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:00:32.309 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: cephadm 2021-07-31T03:00: 2021-07-31T03:00:32.309 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: 30.775312+0000 mgr.x (mgr.24542) 3139 : cephadm 2021-07-31T03:00:32.309 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: [ERR] 2021-07-31T03:00:32.310 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:00:32.310 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:00:32.310 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: yield 2021-07-31T03:00:32.310 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:00:32.311 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:00:32.311 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:00:32.311 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:00:32.311 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:00:32.311 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:00:32.312 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:00:32.312 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:00:32.312 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:00:32.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: task.result() 2021-07-31T03:00:32.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:00:32.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: cephadm 2021-07-31T03:00:30.776323+0000 mgr.x (mgr.24542) 3140 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:00:32.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: cephadm 2021-07-31T03:00 2021-07-31T03:00:32.314 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: :30.800752+0000 mgr.x (mgr.24542) 3141 : cephadm 2021-07-31T03:00:32.314 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: [ERR] 2021-07-31T03:00:32.314 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:00:32.315 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: yield 2021-07-31T03:00:32.315 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:00:32.315 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:00:32.315 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:00:32.315 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:00:32.316 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:00:32.316 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:00:32.316 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:00:32.316 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:00:32.317 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:00:32.317 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: task.result() 2021-07-31T03:00:32.317 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:00:32.317 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: cephadm 2021-07-31T03:00:30.801218+0000 2021-07-31T03:00:32.318 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: mgr.x (mgr.24542) 3142 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:00:32.318 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:00:32.318 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:00:32.318 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: yield 2021-07-31T03:00:32.319 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:00:32.319 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:00:32.319 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:00:32.319 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:00:32.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:00:32.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:00:32.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:00:32.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:00:32.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:00:32.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: task.result() 2021-07-31T03:00:32.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:00:32.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: 2021-07-31T03:00:32.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:00:32.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: 2021-07-31T03:00:32.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:00:32.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:00:32.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:00:32.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:00:32.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:00:32.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:00:32.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:00:32.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:00:32.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:00:32.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:00:32.324 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:00:32.325 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:00:32.325 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: yield 2021-07-31T03:00:32.325 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:00:32.325 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:00:32.326 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:00:32.326 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:00:32.326 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:00:32.326 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:00:32.327 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:00:32.327 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:00:32.327 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:00:32.327 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: task.result() 2021-07-31T03:00:32.328 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:00:32.328 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: cephadm 2021-07-31T03:00:30.801218+0000 mgr.x (mgr.24542) 3142 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:00:32.328 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:00:32.328 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:00:32.329 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: yield 2021-07-31T03:00:32.329 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:00:32.329 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:00:32.329 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:00:32.329 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:00:32.330 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:00:32.330 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:00:32.330 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:00:32.330 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:00:32.331 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:00:32.331 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: task.result() 2021-07-31T03:00:32.331 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:00:32.331 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: 2021-07-31T03:00:32.331 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:00:32.332 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: 2021-07-31T03:00:32.332 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:00:32.332 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:00:32.332 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:00:32.333 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:00:32.333 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:00:32.333 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:00:32.333 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:00:32.333 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:00:32.334 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:00:32.334 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:00:32.334 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:00:32.335 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:00:32.335 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:00:32.335 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:00:32.335 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:00:32.336 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:00:32.336 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: raise 2021-07-31T03:00:32.336 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:00:32.336 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:00:32.337 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:00:32.337 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:00:32.337 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:00:32.337 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: cephadm 2021-07-31T03:00:30.809698+0000 mgr.x (mgr.24542) 3143 : cephadm [ERR] 2021-07-31T03:00:32.337 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:00:32.338 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:00:32.338 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: yield 2021-07-31T03:00:32.338 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:00:32.338 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:00:32.339 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:00:32.339 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:00:32.339 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:00:32.339 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:00:32.339 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:00:32.340 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:00:32.340 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:00:32.340 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: task.result() 2021-07-31T03:00:32.340 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:00:32.341 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: cephadm 2021-07-31T03:00:30.810272+0000 mgr.x (mgr.24542 2021-07-31T03:00:32.341 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: ) 3144 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:00:32.341 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:00:32.341 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:00:32.342 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: yield 2021-07-31T03:00:32.342 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:00:32.342 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:00:32.342 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:00:32.342 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:00:32.343 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:00:32.343 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:00:32.343 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:00:32.344 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:00:32.344 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:00:32.344 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:00:32.344 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:00:32.345 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:00:32.345 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:00:32.345 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:00:32.345 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:00:32.345 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: raise 2021-07-31T03:00:32.346 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:00:32.346 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:00:32.346 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:00:32.346 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:00:32.347 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:00:32.347 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: cephadm 2021- 2021-07-31T03:00:32.347 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: 07-31T03:00:30.809698+0000 mgr.x (mgr.24542) 3143 : cephadm [ERR] 2021-07-31T03:00:32.347 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:00:32.348 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:00:32.348 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: yield 2021-07-31T03:00:32.348 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:00:32.348 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:00:32.348 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:00:32.349 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:00:32.349 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:00:32.349 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:00:32.349 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:00:32.350 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:00:32.350 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:00:32.350 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: task.result() 2021-07-31T03:00:32.350 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:00:32.350 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: cephadm 2021-07-31T03:00:30.810272+0000 mgr.x (mgr.24542) 3144 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:00:32.351 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:00:32.351 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:00:32.351 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: yield 2021-07-31T03:00:32.351 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:00:32.352 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:00:32.352 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:00:32.352 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:00:32.353 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:00:32.353 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:00:32.353 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:00:32.353 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:00:32.353 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: task.result() 2021-07-31T03:00:32.354 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:00:32.354 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: 2021-07-31T03:00:32.354 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:00:32.354 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: 2021-07-31T03:00:32.355 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:00:32.355 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:00:32.355 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:00:32.355 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:00:32.356 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:00:32.356 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:00:32.356 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:00:32.356 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:00:32.356 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:00:32.357 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:00:32.357 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:00:32.357 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:00:32.357 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:00:32.357 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:00:32.358 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:00:32.358 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:00:32.358 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:00:32.358 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:00:32.359 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: raise 2021-07-31T03:00:32.359 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:00:32.359 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:00:32.359 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:00:32.360 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:00:32.360 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:00:32.360 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[33655]: cluster 2021-07-31T03:00:30.849035+0000 mgr.x (mgr.24542) 3145 : cluster [DBG] pgmap v4206: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:32.360 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:00:32.361 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:00:32.361 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:00:32.361 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:00:32.361 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:00:32.361 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: task.result() 2021-07-31T03:00:32.362 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:00:32.362 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: 2021-07-31T03:00:32.362 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:00:32.362 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: 2021-07-31T03:00:32.363 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:00:32.363 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:00:32.363 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:00:32.363 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:00:32.364 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:00:32.364 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:00:32.364 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:00:32.364 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:00:32.364 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:00:32.365 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:00:32.365 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:00:32.365 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:00:32.365 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:00:32.366 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:00:32.366 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:00:32.366 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:00:32.366 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:00:32.367 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:00:32.367 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: raise 2021-07-31T03:00:32.367 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:00:32.367 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:00:32.367 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:00:32.368 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:00:32.368 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:00:32.368 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: cluster 2021-07-31T03:00: 2021-07-31T03:00:32.368 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:31 smithi045 conmon[39718]: 30.849035+0000 mgr.x (mgr.24542) 3145 : cluster [DBG] pgmap v4206: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:34.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:33 smithi154 conmon[34553]: cluster 2021-07-31T03:00:32.849492+0000 mgr.x (mgr.24542) 3146 : cluster [DBG] pgmap v4207: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:34.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:33 smithi045 conmon[39718]: cluster 2021-07-31T03:00:32. 2021-07-31T03:00:34.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:33 smithi045 conmon[39718]: 849492+0000 mgr.x (mgr.24542) 3146 : cluster [DBG] pgmap v4207: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:34.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:33 smithi045 conmon[33655]: cluster 2021-07-31T03:00:32.849492+0000 mgr.x 2021-07-31T03:00:34.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:33 smithi045 conmon[33655]: (mgr.24542) 3146 : cluster [DBG] pgmap v4207: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:36.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:35 smithi154 conmon[34553]: cluster 2021-07-31T03:00:34.849905+0000 mgr.x (mgr.24542) 3147 : cluster [DBG] 2021-07-31T03:00:36.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:35 smithi154 conmon[34553]: pgmap v4208: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:36.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:35 smithi045 conmon[39718]: cluster 2021-07-31T03:00:34.849905+0000 mgr.x (mgr.24542) 3147 : cluster [DBG] pgmap v4208: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:36.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:35 smithi045 conmon[39718]: 2021-07-31T03:00:36.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:35 smithi045 conmon[33655]: cluster 2021-07-31T03:00:34.849905+0000 mgr.x (mgr.24542) 3147 : 2021-07-31T03:00:36.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:35 smithi045 conmon[33655]: cluster [DBG] pgmap v4208: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:38.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:37 smithi154 conmon[34553]: cluster 2021-07-31T03:00:36.850292+0000 mgr.x (mgr.24542) 3148 : cluster [DBG] pgmap v4209: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:38.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:37 smithi045 conmon[33655]: cluster 2021-07-31T03:00:36.850292 2021-07-31T03:00:38.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:37 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 3148 : cluster [DBG] pgmap v4209: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:38.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:37 smithi045 conmon[39718]: cluster 2021-07-31T03:00:36.850292+0000 mgr.x (mgr. 2021-07-31T03:00:38.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:37 smithi045 conmon[39718]: 24542) 3148 : cluster [DBG] pgmap v4209: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:39.377 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:00:39.657 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:00:39.657 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 75m ago 76m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:00:39.657 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 73m ago 76m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:00:39.658 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 73m ago 79m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:00:39.658 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 75m ago 81m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:00:39.658 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 75m ago 81m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:00:39.658 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 73m ago 79m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:00:39.659 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 75m ago 79m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:00:39.659 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 75m ago 76m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:00:39.659 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 73m ago 76m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:00:39.659 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 75m ago 79m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:00:39.660 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 75m ago 78m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:00:39.660 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 75m ago 78m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:00:39.660 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 75m ago 78m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:00:39.660 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 73m ago 77m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:00:39.660 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 73m ago 77m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:00:39.661 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 73m ago 77m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:00:39.661 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 73m ago 76m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:00:39.661 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 73m ago 76m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:00:39.993 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:00:39.994 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:00:39.994 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:00:39.994 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:00:39.995 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:00:39.995 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:00:39.995 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:00:39.995 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:00:39.996 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:00:39.996 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:00:39.996 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:00:39.996 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:00:39.997 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:00:39.997 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:00:39.997 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:00:39.997 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:00:39.997 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:00:40.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:39 smithi154 conmon[34553]: cluster 2021-07-31T03:00:38.850817+0000 mgr.x (mgr.24542) 3149 : cluster [DBG] pgmap v4210: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:40.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:39 smithi154 conmon[34553]: audit 2021-07-31T03:00:39.364628+0000 mgr.x (mgr.24542) 3150 : audit [DBG] from='client.28546 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:00:40.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:39 smithi154 conmon[34553]: audit 2021-07-31T03:00:39.648652+0000 mgr.x (mgr.24542) 3151 : audit 2021-07-31T03:00:40.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:39 smithi154 conmon[34553]: [DBG] from='client.18612 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:00:40.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:39 smithi045 conmon[39718]: cluster 2021-07-31T03:00:38.850817 2021-07-31T03:00:40.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:39 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 3149 : cluster [DBG] pgmap v4210: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:40.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:39 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T03:00:40.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:39 smithi045 conmon[39718]: 31T03:00:39.364628+0000 mgr.x (mgr.24542) 3150 : 2021-07-31T03:00:40.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:39 smithi045 conmon[39718]: audit [DBG] from='client.28546 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:00:40.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:39 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T03:00:40.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:39 smithi045 conmon[39718]: 03:00:39.648652+0000 mgr.x ( 2021-07-31T03:00:40.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:39 smithi045 conmon[39718]: mgr.24542) 3151 : audit [DBG] from='client.18612 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:00:40.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:39 smithi045 conmon[39718]: 2021-07-31T03:00:40.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:39 smithi045 conmon[33655]: cluster 2021-07-31T03:00:38.850817+0000 mgr.x ( 2021-07-31T03:00:40.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:39 smithi045 conmon[33655]: mgr.24542) 3149 : cluster [DBG] pgmap v4210: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:40.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:39 smithi045 conmon[33655]: audit 2021-07-31T03:00:39.364628+0000 mgr.x (mgr.24542) 3150 : audit [DBG] from='client.28546 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:00:40.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:39 smithi045 conmon[33655]: audit 2021-07-31T03:00:39.648652+0000 mgr.x (mgr.24542) 3151 : audit [DBG] from='client.18612 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:00:41.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:40 smithi154 conmon[34553]: audit 2021-07-31T03:00:39.993582+0000 mon.c (mon.1) 168 : audit [DBG] from='client.? 172.21.15.45:0/4091315732' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:00:41.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:40 smithi154 conmon[34553]: 2021-07-31T03:00:41.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:40 smithi045 conmon[39718]: audit 2021-07-31T03:00:39.993582+0000 mon.c ( 2021-07-31T03:00:41.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:40 smithi045 conmon[39718]: mon.1) 168 : audit [DBG] from='client.? 172.21.15.45:0/4091315732' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:00:41.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:40 smithi045 conmon[33655]: audit 2021-07-31T03:00:39.993582+0000 mon.c (mon.1) 168 : audit 2021-07-31T03:00:41.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:40 smithi045 conmon[33655]: [DBG] from='client.? 172.21.15.45:0/4091315732' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:00:42.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:41 smithi154 conmon[34553]: cluster 2021-07-31T03:00:40.851266+0000 mgr.x (mgr.24542) 3152 : cluster [DBG] pgmap v4211: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:42.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:41 smithi045 conmon[33655]: cluster 2021-07-31T03:00:40.851266+0000 mgr.x (mgr.24542) 3152 : cluster 2021-07-31T03:00:42.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:41 smithi045 conmon[33655]: [DBG] pgmap v4211: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:42.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:41 smithi045 conmon[39718]: cluster 2021-07-31T03:00:40.851266+0000 mgr.x (mgr. 2021-07-31T03:00:42.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:41 smithi045 conmon[39718]: 24542) 3152 : cluster [DBG] pgmap v4211: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:44.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:43 smithi154 conmon[34553]: cluster 2021-07-31T03:00:42.851762+0000 mgr.x (mgr.24542) 3153 : cluster [DBG] pgmap v4212: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:44.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:43 smithi154 conmon[34553]: 2021-07-31T03:00:44.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:43 smithi045 conmon[39718]: cluster 2021-07-31T03:00:42.851762+0000 mgr.x 2021-07-31T03:00:44.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:43 smithi045 conmon[39718]: (mgr.24542) 3153 : cluster [DBG] pgmap v4212: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:44.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:43 smithi045 conmon[33655]: cluster 2021-07-31T03:00:42.851762+0000 mgr.x (mgr.24542) 3153 : cluster [DBG] pgmap v4212: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:46.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:45 smithi154 conmon[34553]: cluster 2021-07-31T03:00:44.852162+0000 mgr.x (mgr.24542) 3154 : cluster [DBG] pgmap v4213: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:46.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:45 smithi045 conmon[39718]: cluster 2021-07-31T03:00:44.852162+0000 mgr.x (mgr.24542) 3154 : cluster [DBG] pgmap v4213: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:46.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:45 smithi045 conmon[39718]: 2021-07-31T03:00:46.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:45 smithi045 conmon[33655]: cluster 2021-07-31T03:00:44.852162+0000 mgr.x (mgr.24542) 3154 : cluster 2021-07-31T03:00:46.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:45 smithi045 conmon[33655]: [DBG] pgmap v4213: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:48.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:47 smithi154 conmon[34553]: cluster 2021-07-31T03:00:46.852643+0000 mgr.x (mgr.24542) 3155 : cluster [DBG] pgmap v4214: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:48.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:47 smithi154 conmon[34553]: 2021-07-31T03:00:48.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:47 smithi045 conmon[39718]: cluster 2021-07-31T03:00:46 2021-07-31T03:00:48.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:47 smithi045 conmon[39718]: .852643+0000 mgr.x (mgr.24542) 3155 : cluster [DBG] pgmap v4214: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:48.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:47 smithi045 conmon[33655]: cluster 2021-07-31T03:00:46.852643+0000 mgr.x (mgr.24542) 3155 : cluster 2021-07-31T03:00:48.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:47 smithi045 conmon[33655]: [DBG] pgmap v4214: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:50.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:49 smithi154 conmon[34553]: cluster 2021-07-31T03:00:48.853101+0000 mgr.x (mgr.24542) 3156 : cluster [DBG] pgmap v4215: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:50.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:49 smithi045 conmon[39718]: cluster 2021-07-31T03:00:48.853101 2021-07-31T03:00:50.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:49 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 3156 : cluster [DBG] pgmap v4215: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:50.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:49 smithi045 conmon[33655]: cluster 2021-07-31T03:00:48.853101+0000 mgr.x (mgr.24542) 3156 : cluster 2021-07-31T03:00:50.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:49 smithi045 conmon[33655]: [DBG] pgmap v4215: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:52.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:51 smithi154 conmon[34553]: cluster 2021-07-31T03:00:50.853550+0000 mgr.x (mgr.24542) 3157 : cluster [DBG] pgmap v4216: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:52.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:51 smithi045 conmon[39718]: cluster 2021-07-31T03:00 2021-07-31T03:00:52.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:51 smithi045 conmon[39718]: :50.853550+0000 mgr.x (mgr.24542) 3157 : cluster [DBG] pgmap v4216: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:52.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:51 smithi045 conmon[33655]: cluster 2021-07-31T03:00:50.853550+0000 mgr.x (mgr.24542) 3157 : cluster [DBG] 2021-07-31T03:00:52.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:51 smithi045 conmon[33655]: pgmap v4216: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:54.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:53 smithi154 conmon[34553]: cluster 2021-07-31T03:00:52.854013+0000 mgr.x (mgr.24542) 3158 : cluster [DBG] pgmap v4217: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:54.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:53 smithi045 conmon[33655]: cluster 2021-07-31T03:00:52.854013+0000 mgr.x ( 2021-07-31T03:00:54.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:53 smithi045 conmon[33655]: mgr.24542) 3158 : cluster [DBG] pgmap v4217: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:54.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:53 smithi045 conmon[39718]: cluster 2021-07-31T03:00:52.854013+0000 mgr.x (mgr 2021-07-31T03:00:54.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:53 smithi045 conmon[39718]: .24542) 3158 : cluster [DBG] pgmap v4217: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:56.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:55 smithi154 conmon[34553]: cluster 2021-07-31T03:00:54.854494+0000 mgr.x (mgr.24542) 3159 : cluster [DBG] pgmap v4218: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:56.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:55 smithi045 conmon[39718]: cluster 2021-07-31T03:00:54.854494+0000 mgr.x (mgr 2021-07-31T03:00:56.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:55 smithi045 conmon[39718]: .24542) 3159 : cluster [DBG] pgmap v4218: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:56.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:55 smithi045 conmon[33655]: cluster 2021-07-31T03:00:54.854494+0000 mgr.x 2021-07-31T03:00:56.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:55 smithi045 conmon[33655]: (mgr.24542) 3159 : cluster [DBG] pgmap v4218: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:58.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:57 smithi154 conmon[34553]: cluster 2021-07-31T03:00:56.854874+0000 mgr.x (mgr.24542) 3160 : cluster [DBG] pgmap v4219: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:58.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:57 smithi045 conmon[39718]: cluster 2021-07-31T03:00:56.854874+0000 mgr.x (mgr 2021-07-31T03:00:58.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:57 smithi045 conmon[39718]: .24542) 3160 : cluster [DBG] pgmap v4219: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:00:58.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:57 smithi045 conmon[33655]: cluster 2021-07-31T03:00:56.854874+0000 mgr.x (mgr.24542) 3160 : cluster [DBG] pgmap v4219: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:00.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:00:59 smithi154 conmon[34553]: cluster 2021-07-31T03:00:58.855285+0000 mgr.x (mgr.24542) 3161 : cluster [DBG] pgmap v4220: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:00.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:59 smithi045 conmon[33655]: cluster 2021-07-31T03:00:58.855285+0000 2021-07-31T03:01:00.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:00:59 smithi045 conmon[33655]: mgr.x (mgr.24542) 3161 : cluster [DBG] pgmap v4220: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:00.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:59 smithi045 conmon[39718]: cluster 2021-07-31T03:00:58.855285+0000 mgr.x 2021-07-31T03:01:00.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:00:59 smithi045 conmon[39718]: (mgr.24542) 3161 : cluster [DBG] pgmap v4220: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:02.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:01 smithi154 conmon[34553]: cluster 2021-07-31T03:01:00.855729+0000 mgr.x (mgr.24542) 3162 : cluster [DBG] pgmap v4221: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:02.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:01 smithi154 conmon[34553]: 2021-07-31T03:01:02.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:01 smithi045 conmon[39718]: cluster 2021-07-31T03:01:00.855729+0000 2021-07-31T03:01:02.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:01 smithi045 conmon[39718]: mgr.x (mgr.24542) 3162 : cluster [DBG] pgmap v4221: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:02.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:01 smithi045 conmon[33655]: cluster 2021-07-31T03:01:00 2021-07-31T03:01:02.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:01 smithi045 conmon[33655]: .855729+0000 mgr.x (mgr.24542) 3162 : cluster [DBG] pgmap v4221: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:04.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:03 smithi154 conmon[34553]: cluster 2021-07-31T03:01:02.856161+0000 mgr.x (mgr.24542) 3163 : cluster [DBG] pgmap v4222: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:04.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:03 smithi045 conmon[39718]: cluster 2021-07-31T03:01:02 2021-07-31T03:01:04.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:03 smithi045 conmon[39718]: .856161+0000 mgr.x (mgr.24542) 3163 : cluster [DBG] pgmap v4222: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:04.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:03 smithi045 conmon[33655]: cluster 2021-07-31T03:01:02.856161+0000 mgr.x (mgr 2021-07-31T03:01:04.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:03 smithi045 conmon[33655]: .24542) 3163 : cluster [DBG] pgmap v4222: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:06.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:05 smithi154 conmon[34553]: cluster 2021-07-31T03:01:04.856750+0000 mgr.x (mgr.24542) 3164 : cluster [DBG] pgmap v4223: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:06.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:05 smithi154 conmon[34553]: 2021-07-31T03:01:06.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:05 smithi045 conmon[33655]: cluster 2021-07-31T03:01:04.856750 2021-07-31T03:01:06.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:05 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 3164 : cluster [DBG] pgmap v4223: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:06.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:05 smithi045 conmon[39718]: cluster 2021-07-31T03:01:04.856750+0000 mgr.x (mgr.24542) 3164 2021-07-31T03:01:06.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:05 smithi045 conmon[39718]: : cluster [DBG] pgmap v4223: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:08.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:07 smithi154 conmon[34553]: cluster 2021-07-31T03:01:06.857151+0000 mgr.x (mgr.24542) 3165 : cluster [DBG] pgmap v4224: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:08.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:07 smithi045 conmon[33655]: cluster 2021-07-31T03:01:06.857151+0000 mgr.x (mgr.24542) 2021-07-31T03:01:08.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:07 smithi045 conmon[33655]: 3165 : cluster [DBG] pgmap v4224: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:08.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:07 smithi045 conmon[39718]: cluster 2021-07-31T03:01:06.857151+0000 mgr.x ( 2021-07-31T03:01:08.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:07 smithi045 conmon[39718]: mgr.24542) 3165 : cluster [DBG] pgmap v4224: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:10.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:09 smithi154 conmon[34553]: cluster 2021-07-31T03:01:08.857503+0000 mgr.x (mgr.24542) 3166 : cluster [DBG] pgmap v4225: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:10.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:09 smithi045 conmon[39718]: cluster 2021-07-31T03:01:08.857503+0000 mgr.x (mgr.24542) 3166 : cluster 2021-07-31T03:01:10.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:09 smithi045 conmon[39718]: [DBG] pgmap v4225: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:10.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:09 smithi045 conmon[33655]: cluster 2021-07-31T03:01:08.857503+0000 2021-07-31T03:01:10.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:09 smithi045 conmon[33655]: mgr.x (mgr.24542) 3166 : cluster [DBG] pgmap v4225: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:10.306 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:01:10.585 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:01:10.585 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 76m ago 76m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:01:10.586 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 73m ago 76m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:01:10.586 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 73m ago 80m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:01:10.586 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 76m ago 82m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:01:10.586 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 76m ago 82m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:01:10.587 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 73m ago 80m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:01:10.587 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 76m ago 80m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:01:10.587 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 76m ago 77m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:01:10.587 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 73m ago 76m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:01:10.588 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 76m ago 79m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:01:10.588 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 76m ago 79m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:01:10.588 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 76m ago 79m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:01:10.588 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 76m ago 78m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:01:10.588 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 73m ago 78m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:01:10.589 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 73m ago 78m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:01:10.589 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 73m ago 77m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:01:10.589 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 73m ago 77m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:01:10.589 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 73m ago 77m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:01:10.924 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:01:10.924 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:01:10.924 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:01:10.924 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:01:10.924 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:01:10.925 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:01:10.925 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:01:10.925 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:01:10.925 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:01:10.926 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:01:10.926 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:01:10.926 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:01:10.926 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:01:10.926 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:01:10.927 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:01:10.927 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:01:10.927 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:01:11.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:10 smithi154 conmon[34553]: audit 2021-07-31T03:01:10.293995+0000 mgr.x (mgr.24542) 3167 : audit [DBG] from='client.18624 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:01:11.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:10 smithi154 conmon[34553]: audit 2021-07-31T03:01:10.577324+0000 mgr.x ( 2021-07-31T03:01:11.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:10 smithi154 conmon[34553]: mgr.24542) 3168 : audit [DBG] from='client.18630 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:01:11.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:10 smithi045 conmon[33655]: audit 2021-07-31T03:01:10.293995+0000 mgr.x (mgr.24542) 3167 : 2021-07-31T03:01:11.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:10 smithi045 conmon[33655]: audit [DBG] from='client.18624 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:01:11.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:10 smithi045 conmon[33655]: audit 2021-07-31T03:01:10.577324+0000 mgr.x (mgr.24542) 3168 : audit [DBG] from='client.18630 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:01:11.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:10 smithi045 conmon[39718]: audit 2021-07-31T03:01:10.293995+0000 mgr.x (mgr 2021-07-31T03:01:11.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:10 smithi045 conmon[39718]: .24542) 3167 : audit [DBG] from='client.18624 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:01:11.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:10 smithi045 conmon[39718]: audit 2021-07-31T03:01:10.577324+0000 mgr.x (mgr.24542 2021-07-31T03:01:11.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:10 smithi045 conmon[39718]: ) 3168 : audit [DBG] from='client.18630 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:01:12.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:11 smithi154 conmon[34553]: cluster 2021-07-31T03:01:10.858037+0000 mgr.x (mgr.24542) 3169 : cluster [DBG] pgmap v4226: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:12.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:11 smithi154 conmon[34553]: audit 2021 2021-07-31T03:01:12.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:11 smithi154 conmon[34553]: -07-31T03:01:10.923635+0000 mon.a (mon.0) 3238 : audit [DBG] from='client.? 172.21.15.45:0/1574416783' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:01:12.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:11 smithi045 conmon[33655]: cluster 2021-07-31T03:01:10.858037+0000 2021-07-31T03:01:12.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:11 smithi045 conmon[33655]: mgr.x (mgr.24542) 3169 : cluster [DBG] pgmap v4226: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:12.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:11 smithi045 conmon[33655]: audit 2021-07-31 2021-07-31T03:01:12.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:11 smithi045 conmon[33655]: T03:01:10.923635+0000 2021-07-31T03:01:12.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:11 smithi045 conmon[33655]: mon.a (mon.0) 3238 : audit [DBG] 2021-07-31T03:01:12.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:11 smithi045 conmon[33655]: 2021-07-31T03:01:12.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:11 smithi045 conmon[33655]: from='client.? 172.21.15.45:0/1574416783' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:01:12.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:11 smithi045 conmon[39718]: cluster 2021-07-31T03:01:10.858037+0000 mgr.x ( 2021-07-31T03:01:12.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:11 smithi045 conmon[39718]: mgr.24542) 3169 : cluster [DBG] pgmap v4226: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:12.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:11 smithi045 conmon[39718]: audit 2021-07-31T03:01:10.923635+0000 2021-07-31T03:01:12.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:11 smithi045 conmon[39718]: mon.a (mon.0) 3238 : audit [DBG] from='client.? 172.21.15.45:0/1574416783' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:01:14.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:13 smithi154 conmon[34553]: cluster 2021-07-31T03:01:12.858569+0000 mgr.x (mgr.24542) 3170 : cluster [DBG] pgmap v4227: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:14.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:13 smithi154 conmon[34553]: 2021-07-31T03:01:14.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:13 smithi045 conmon[39718]: cluster 2021-07-31T03:01:12.858569+0000 mgr.x ( 2021-07-31T03:01:14.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:13 smithi045 conmon[39718]: mgr.24542) 3170 : cluster [DBG] pgmap v4227: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:14.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:13 smithi045 conmon[33655]: cluster 2021-07-31T03:01:12.858569+0000 mgr.x (mgr.24542 2021-07-31T03:01:14.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:13 smithi045 conmon[33655]: ) 3170 : cluster [DBG] pgmap v4227: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:16.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:15 smithi154 conmon[34553]: cluster 2021-07-31T03:01:14.859212+0000 mgr.x (mgr.24542) 3171 : cluster [DBG] pgmap v4228: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:16.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:15 smithi045 conmon[39718]: cluster 2021-07-31T03:01:14.859212+0000 mgr.x (mgr. 2021-07-31T03:01:16.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:15 smithi045 conmon[39718]: 24542) 3171 : cluster [DBG] pgmap v4228: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:16.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:15 smithi045 conmon[33655]: cluster 2021-07-31T03:01:14.859212+0000 mgr.x 2021-07-31T03:01:16.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:15 smithi045 conmon[33655]: (mgr.24542) 3171 : cluster [DBG] pgmap v4228: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:18.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:17 smithi154 conmon[34553]: cluster 2021-07-31T03:01:16.859610+0000 mgr.x (mgr.24542) 3172 : cluster [DBG] pgmap v4229: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:18.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:17 smithi045 conmon[39718]: cluster 2021-07-31T03:01:16.859610+0000 mgr.x 2021-07-31T03:01:18.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:17 smithi045 conmon[39718]: (mgr.24542) 3172 : cluster [DBG] pgmap v4229: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:18.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:17 smithi045 conmon[33655]: cluster 2021-07-31T03:01:16.859610+0000 mgr.x (mgr.24542) 3172 : cluster 2021-07-31T03:01:18.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:17 smithi045 conmon[33655]: [DBG] pgmap v4229: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:20.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:19 smithi154 conmon[34553]: cluster 2021-07-31T03:01:18.860018+0000 mgr.x (mgr.24542) 3173 : cluster [DBG] 2021-07-31T03:01:20.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:19 smithi154 conmon[34553]: pgmap v4230: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:20.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:19 smithi045 conmon[39718]: cluster 2021-07-31T03:01:18.860018+0000 mgr.x (mgr.24542) 3173 : cluster [DBG] pgmap v4230: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:20.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:19 smithi045 conmon[33655]: cluster 2021-07-31T03:01:18.860018+0000 mgr.x (mgr.24542) 3173 : 2021-07-31T03:01:20.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:19 smithi045 conmon[33655]: cluster [DBG] pgmap v4230: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:22.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:21 smithi154 conmon[34553]: cluster 2021-07-31T03:01:20.860519+0000 mgr.x (mgr.24542) 3174 : cluster [DBG] pgmap v4231: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:22.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:21 smithi045 conmon[33655]: cluster 2021-07-31T03:01:20.860519+0000 mgr.x ( 2021-07-31T03:01:22.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:21 smithi045 conmon[33655]: mgr.24542) 3174 : cluster [DBG] pgmap v4231: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:22.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:21 smithi045 conmon[39718]: cluster 2021-07-31T03:01:20.860519+0000 2021-07-31T03:01:22.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:21 smithi045 conmon[39718]: mgr.x (mgr.24542) 3174 : cluster [DBG] pgmap v4231: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:24.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:23 smithi154 conmon[34553]: cluster 2021-07-31T03:01:22.861048+0000 mgr.x (mgr.24542) 3175 : cluster [DBG] 2021-07-31T03:01:24.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:23 smithi154 conmon[34553]: pgmap v4232: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:24.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:23 smithi045 conmon[39718]: cluster 2021-07-31T03:01: 2021-07-31T03:01:24.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:23 smithi045 conmon[39718]: 22.861048+0000 mgr.x (mgr.24542) 3175 : cluster [DBG] pgmap v4232: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:24.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:23 smithi045 conmon[33655]: cluster 2021-07-31T03:01:22.861048 2021-07-31T03:01:24.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:23 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 3175 : cluster [DBG] pgmap v4232: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:26.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:25 smithi154 conmon[34553]: cluster 2021-07-31T03:01:24.861703+0000 mgr.x (mgr.24542) 3176 : cluster [DBG] pgmap v4233: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:26.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:25 smithi045 conmon[33655]: cluster 2021-07-31T03:01:24.861703+0000 mgr.x (mgr.24542) 3176 : cluster [DBG] pgmap v4233: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:26.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:25 smithi045 conmon[39718]: cluster 2021-07-31T03:01:24.861703+0000 2021-07-31T03:01:26.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:25 smithi045 conmon[39718]: mgr.x (mgr.24542) 3176 : cluster [DBG] pgmap v4233: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:28.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:27 smithi154 conmon[34553]: cluster 2021-07-31T03:01:26.862033+0000 mgr.x (mgr.24542) 3177 : cluster [DBG] pgmap v4234: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:28.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:27 smithi045 conmon[33655]: cluster 2021-07-31T03:01:26.862033+0000 mgr.x (mgr.24542) 3177 : cluster [DBG] 2021-07-31T03:01:28.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:27 smithi045 conmon[33655]: pgmap v4234: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:28.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:27 smithi045 conmon[39718]: cluster 2021-07-31T03:01:26. 2021-07-31T03:01:28.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:27 smithi045 conmon[39718]: 862033+0000 mgr.x (mgr.24542) 3177 : cluster [DBG] pgmap v4234: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:29.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:29 smithi154 conmon[34553]: audit 2021-07-31T03:01:28.230685+0000 mon.b (mon.2) 445 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:01:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:29 smithi154 conmon[34553]: 2021-07-31T03:01:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:29 smithi154 conmon[34553]: audit 2021-07-31T03:01:28.230944+0000 mon.b (mon.2) 446 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:01:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:29 smithi154 conmon[34553]: audit 2021-07-31T03:01:28.231373+0000 mon.a (mon.0) 3239 : audit [INF] 2021-07-31T03:01:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:29 smithi154 conmon[34553]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:01:29.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:29 smithi154 conmon[34553]: audit 2021-07-31T03:01:28.231792+0000 mon.a (mon.0) 3240 2021-07-31T03:01:29.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:29 smithi154 conmon[34553]: : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:01:29.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:29 smithi045 conmon[33655]: audit 2021-07-31T03:01:28.230685+0000 mon.b (mon.2) 445 : audit 2021-07-31T03:01:29.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:29 smithi045 conmon[33655]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:01:29.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:29 smithi045 conmon[33655]: audit 2021-07-31T03:01:28.230944+0000 mon.b (mon.2) 446 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:01:29.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:29 smithi045 conmon[33655]: audit 2021 2021-07-31T03:01:29.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:29 smithi045 conmon[33655]: -07-31T03:01:28.231373+0000 2021-07-31T03:01:29.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:29 smithi045 conmon[33655]: mon.a (mon.0) 3239 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:01:29.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:29 smithi045 conmon[33655]: 2021-07-31T03:01:29.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:29 smithi045 conmon[33655]: audit 2021-07-31T03:01:29.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:29 smithi045 conmon[33655]: 2021-07-31T03:01:28.231792+0000 mon.a (mon.0) 3240 : audit [INF] 2021-07-31T03:01:29.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:29 smithi045 conmon[33655]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:01:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:29 smithi045 conmon[39718]: audit 2021-07-31T03:01:28.230685+0000 mon.b (mon.2) 445 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:01:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:29 smithi045 conmon[39718]: audit 2021-07-31T03:01:28 2021-07-31T03:01:29.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:29 smithi045 conmon[39718]: .230944+0000 mon.b (mon.2) 2021-07-31T03:01:29.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:29 smithi045 conmon[39718]: 446 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:01:29.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:29 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T03:01:29.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:29 smithi045 conmon[39718]: 03:01:28.231373+0000 mon.a (mon.0 2021-07-31T03:01:29.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:29 smithi045 conmon[39718]: ) 3239 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:01:29.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:29 smithi045 conmon[39718]: 2021-07-31T03:01:29.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:29 smithi045 conmon[39718]: audit 2021 2021-07-31T03:01:29.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:29 smithi045 conmon[39718]: -07-31T03:01:28 2021-07-31T03:01:29.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:29 smithi045 conmon[39718]: .231792+0000 mon.a (mon 2021-07-31T03:01:29.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:29 smithi045 conmon[39718]: .0) 3240 : audit [INF] 2021-07-31T03:01:29.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:29 smithi045 conmon[39718]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:01:30.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:30 smithi045 conmon[39718]: cluster 2021-07-31T03:01:28.862523+0000 mgr.x (mgr. 2021-07-31T03:01:30.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:30 smithi045 conmon[39718]: 24542) 3178 : cluster [DBG] pgmap v4235: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:30.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:30 smithi045 conmon[33655]: cluster 2021-07-31T03:01:28.862523+0000 mgr.x (mgr. 2021-07-31T03:01:30.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:30 smithi045 conmon[33655]: 24542) 3178 : cluster [DBG] pgmap v4235: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:30.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[34553]: cluster 2021-07-31T03:01:28.862523+0000 mgr.x (mgr.24542) 3178 : cluster [DBG] pgmap v4235: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:31.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:31 smithi154 conmon[34553]: audit 2021-07-31T03:01:30.811182+0000 mon.b (mon.2) 447 : audit 2021-07-31T03:01:31.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:31 smithi154 conmon[34553]: [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:01:31.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:31 smithi154 conmon[34553]: audit 2021-07-31T03:01:30.812777+0000 mon.b (mon.2) 448 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:01:31.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:31 smithi154 conmon[34553]: audit 2021-07-31T03:01:30.813708+0000 mon.b (mon.2) 449 : audit 2021-07-31T03:01:31.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:31 smithi154 conmon[34553]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:01:31.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: debug 2021-07-31T03:01:30.857+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:01:31.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:01:31.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:01:31.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: yield 2021-07-31T03:01:31.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:01:31.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:01:31.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:01:31.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:01:31.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:01:31.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:01:31.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:01:31.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:01:31.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:01:31.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: task.result() 2021-07-31T03:01:31.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:01:31.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: debug 2021-07-31T03:01:30.869+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:01:31.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:01:31.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:01:31.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: yield 2021-07-31T03:01:31.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:01:31.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:01:31.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:01:31.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:01:31.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:01:31.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:01:31.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:01:31.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:01:31.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:01:31.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: task.result() 2021-07-31T03:01:31.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:01:31.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: debug 2021-07-31T03:01:30.894+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:01:31.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:01:31.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:01:31.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: yield 2021-07-31T03:01:31.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:01:31.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:01:31.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:01:31.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:01:31.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:01:31.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:01:31.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:01:31.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:01:31.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:01:31.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: task.result() 2021-07-31T03:01:31.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:01:31.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: debug 2021-07-31T03:01:30.895+0000 7fcb97969700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:01:31.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:01:31.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:01:31.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: yield 2021-07-31T03:01:31.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:01:31.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:01:31.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:01:31.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:01:31.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:01:31.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:01:31.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:01:31.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:01:31.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:01:31.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: task.result() 2021-07-31T03:01:31.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:01:31.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: 2021-07-31T03:01:31.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:01:31.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: 2021-07-31T03:01:31.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:01:31.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:01:31.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:01:31.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:01:31.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:01:31.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:01:31.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:01:31.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:01:31.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:01:31.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:01:31.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:01:31.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:01:31.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:01:31.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:01:31.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:01:31.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:01:31.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:01:31.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:01:31.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: raise 2021-07-31T03:01:31.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:01:31.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:01:31.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:01:31.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:01:31.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:01:31.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: debug 2021-07-31T03:01:30.903+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:01:31.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:01:31.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:01:31.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: yield 2021-07-31T03:01:31.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:01:31.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:01:31.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:01:31.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:01:31.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:01:31.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:01:31.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:01:31.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:01:31.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:01:31.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: task.result() 2021-07-31T03:01:31.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:01:31.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: debug 2021-07-31T03:01:30.904+0000 7fcb9816a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:01:31.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:01:31.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:01:31.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: yield 2021-07-31T03:01:31.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:01:31.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:01:31.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:01:31.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:01:31.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:01:31.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:01:31.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:01:31.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:01:31.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:01:31.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: task.result() 2021-07-31T03:01:31.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:01:31.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: 2021-07-31T03:01:31.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:01:31.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: 2021-07-31T03:01:31.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:01:31.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:01:31.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:01:31.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:01:31.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:01:31.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:01:31.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:01:31.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:01:31.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:01:31.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:01:31.284 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:01:31.284 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:01:31.284 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:01:31.284 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:01:31.284 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:01:31.285 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:01:31.285 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:01:31.285 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:01:31.285 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: raise 2021-07-31T03:01:31.286 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:01:31.286 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:01:31.286 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:01:31.286 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:01:31.286 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:01:30 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:01:31.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:31 smithi045 conmon[39718]: audit 2021-07-31T03:01:30.811182+0000 mon.b ( 2021-07-31T03:01:31.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:31 smithi045 conmon[39718]: mon.2) 447 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:01:31.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:31 smithi045 conmon[39718]: audit 2021-07-31T03:01:30 2021-07-31T03:01:31.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:31 smithi045 conmon[39718]: .812777+0000 mon.b (mon.2) 448 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:01:31.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:31 smithi045 conmon[39718]: audit 2021-07-31T03:01:30.813708+0000 mon.b ( 2021-07-31T03:01:31.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:31 smithi045 conmon[39718]: mon.2) 449 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:01:31.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:31 smithi045 conmon[33655]: audit 2021-07-31T03:01:30.811182+0000 mon.b (mon.2) 447 : audit [DBG] 2021-07-31T03:01:31.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:31 smithi045 conmon[33655]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:01:31.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:31 smithi045 conmon[33655]: audit 2021-07-31T03:01:30.812777+0000 mon.b (mon.2) 448 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:01:31.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:31 smithi045 conmon[33655]: audit 2021-07-31T03:01:30.813708+0000 mon.b (mon.2) 449 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:01:32.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: cephadm 2021-07-31T03:01:30.814507+0000 mgr.x (mgr.24542) 3179 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:01:32.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: cephadm 2021-07-31T03:01:30.858417+0000 mgr.x (mgr.24542) 3180 : cephadm [ERR] 2021-07-31T03:01:32.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:01:32.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: cephadm 2021-07-31T03:01:30.814507+0000 mgr.x ( 2021-07-31T03:01:32.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: mgr.24542) 3179 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:01:32.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: cephadm 2021-07-31T03:01 2021-07-31T03:01:32.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: :30.858417+0000 mgr.x (mgr.24542) 3180 : cephadm 2021-07-31T03:01:32.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: [ERR] 2021-07-31T03:01:32.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:01:32.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:01:32.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: yield 2021-07-31T03:01:32.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:01:32.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:01:32.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:01:32.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:01:32.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:01:32.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:01:32.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:01:32.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:01:32.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:01:32.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: task.result() 2021-07-31T03:01:32.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:01:32.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: cephadm 2021-07-31T03:01:30.858823+0000 mgr.x (mgr.24542) 3181 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:01:32.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: cluster 2021 2021-07-31T03:01:32.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: -07-31T03:01: 2021-07-31T03:01:32.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: 30.862906+0000 mgr.x (mgr.24542 2021-07-31T03:01:32.282 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: ) 3182 : cluster [DBG] pgmap v4236: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:32.282 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: cephadm 2021-07-31 2021-07-31T03:01:32.282 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: T03:01:30.870155+0000 mgr.x 2021-07-31T03:01:32.282 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: (mgr.24542) 3183 : cephadm [ERR] 2021-07-31T03:01:32.282 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:01:32.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:01:32.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: yield 2021-07-31T03:01:32.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:01:32.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:01:32.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:01:32.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:01:32.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:01:32.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:01:32.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:01:32.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:01:32.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:01:32.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: task.result() 2021-07-31T03:01:32.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:01:32.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: 2021-07-31T03:01:32.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: cephadm 2021-07-31T03:01:30.870964+0000 mgr.x (mgr.24542) 3184 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:01:32.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: cephadm 2021-07-31T03:01:32.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: 2021-07-31T03: 2021-07-31T03:01:32.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: 01:30.895783+0000 mgr.x ( 2021-07-31T03:01:32.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: mgr.24542) 3185 : cephadm [ERR] 2021-07-31T03:01:32.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:01:32.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:01:32.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: yield 2021-07-31T03:01:32.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:01:32.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:01:32.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:01:32.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:01:32.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:01:32.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:01:32.289 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:01:32.289 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:01:32.289 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:01:32.289 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: task.result() 2021-07-31T03:01:32.289 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:01:32.290 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: 2021-07-31T03:01:32.290 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: cephadm 2021-07- 2021-07-31T03:01:32.290 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: 31T03:01:30.896392+0000 2021-07-31T03:01:32.290 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: mgr.x (mgr.24542) 3186 : 2021-07-31T03:01:32.291 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:01:32.291 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:01:32.291 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:01:32.291 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: yield 2021-07-31T03:01:32.291 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:01:32.292 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:01:32.292 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:01:32.292 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:01:32.292 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:01:32.292 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:01:32.293 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:01:32.293 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:01:32.293 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:01:32.293 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: task.result() 2021-07-31T03:01:32.293 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:01:32.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: 2021-07-31T03:01:32.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:01:32.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: 2021-07-31T03:01:32.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:01:32.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:01:32.295 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:01:32.295 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: cephadm 2021-07-31T03:01:30.814507+0000 mgr.x (mgr.24542) 3179 : 2021-07-31T03:01:32.295 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:01:32.296 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: cephadm 2021-07-31T03:01:30.858417+0000 mgr.x (mgr.24542) 3180 : cephadm 2021-07-31T03:01:32.296 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: [ERR] 2021-07-31T03:01:32.296 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:01:32.296 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:01:32.297 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: yield 2021-07-31T03:01:32.297 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:01:32.297 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:01:32.297 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:01:32.297 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:01:32.298 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:01:32.298 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:01:32.298 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:01:32.298 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:01:32.298 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:01:32.299 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: task.result() 2021-07-31T03:01:32.299 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:01:32.299 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: cephadm 2021-07-31T03:01:30 2021-07-31T03:01:32.299 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: .858823+0000 mgr.x (mgr.24542) 3181 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:01:32.300 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: cluster 2021-07-31T03:01:30.862906+0000 mgr.x (mgr.24542) 3182 : cluster [DBG] pgmap v4236: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:32.300 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: cephadm 2021-07-31T03:01:30.870155+0000 mgr.x (mgr.24542) 3183 : cephadm [ERR] 2021-07-31T03:01:32.300 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:01:32.300 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:01:32.300 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: yield 2021-07-31T03:01:32.301 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:01:32.301 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:01:32.301 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:01:32.301 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:01:32.301 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:01:32.302 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:01:32.302 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:01:32.302 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:01:32.302 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:01:32.303 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: task.result() 2021-07-31T03:01:32.303 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:01:32.303 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: cephadm 2021-07-31T 2021-07-31T03:01:32.303 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: 03:01:30.870964+0000 mgr.x (mgr.24542) 3184 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:01:32.304 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:01:32.304 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:01:32.304 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:01:32.304 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:01:32.305 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:01:32.305 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:01:32.305 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:01:32.305 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:01:32.306 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:01:32.306 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:01:32.306 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:01:32.306 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:01:32.307 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:01:32.307 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:01:32.307 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:01:32.307 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: raise 2021-07-31T03:01:32.308 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:01:32.308 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:01:32.308 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:01:32.308 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:01:32.308 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:01:32.309 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: cephadm 2021-07-31T03:01:30.904592+0000 mgr.x (mgr.24542) 3187 : cephadm [ERR] 2021-07-31T03:01:32.309 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:01:32.309 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:01:32.309 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: yield 2021-07-31T03:01:32.309 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:01:32.310 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:01:32.310 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:01:32.310 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:01:32.310 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:01:32.310 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:01:32.311 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:01:32.311 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:01:32.311 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:01:32.311 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: task.result() 2021-07-31T03:01:32.311 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:01:32.312 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: cephadm 2021-07-31T03:01:30.905130+0000 mgr.x (mgr.24542) 3188 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:01:32.312 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:01:32.312 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:01:32.312 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: yield 2021-07-31T03:01:32.312 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:01:32.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:01:32.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:01:32.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:01:32.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:01:32.314 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:01:32.314 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:01:32.314 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:01:32.314 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:01:32.314 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: task.result() 2021-07-31T03:01:32.315 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:01:32.315 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: 2021-07-31T03:01:32.315 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:01:32.315 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: 2021-07-31T03:01:32.315 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:01:32.316 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:01:32.316 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:01:32.316 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:01:32.316 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:01:32.316 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:01:32.317 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:01:32.317 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:01:32.317 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:01:32.317 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:01:32.318 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:01:32.318 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:01:32.318 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:01:32.318 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:01:32.319 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:01:32.319 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:01:32.319 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:01:32.319 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:01:32.320 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: raise 2021-07-31T03:01:32.320 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:01:32.320 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:01:32.320 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:01:32.320 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:01:32.321 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:01:32.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: cephadm 2021-07-31T03:01:30.895783+0000 mgr.x (mgr.24542) 3185 : cephadm [ERR] 2021-07-31T03:01:32.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:01:32.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:01:32.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: yield 2021-07-31T03:01:32.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:01:32.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:01:32.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:01:32.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:01:32.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:01:32.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:01:32.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:01:32.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:01:32.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:01:32.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: task.result() 2021-07-31T03:01:32.325 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:01:32.325 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: cephadm 2021-07-31T03:01:30.896392+0000 mgr.x (mgr.24542) 3186 : cephadm [ERR] 2021-07-31T03:01:32.325 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:01:32.325 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:01:32.325 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:01:32.326 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: yield 2021-07-31T03:01:32.326 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:01:32.326 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:01:32.326 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:01:32.326 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:01:32.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:01:32.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:01:32.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:01:32.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:01:32.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:01:32.328 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: task.result() 2021-07-31T03:01:32.328 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:01:32.328 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: 2021-07-31T03:01:32.328 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:01:32.329 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: 2021-07-31T03:01:32.329 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:01:32.329 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:01:32.329 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:01:32.329 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:01:32.330 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:01:32.330 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:01:32.330 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:01:32.330 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:01:32.330 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:01:32.331 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:01:32.331 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:01:32.331 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:01:32.331 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:01:32.331 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:01:32.332 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:01:32.332 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:01:32.332 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:01:32.332 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:01:32.332 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: raise 2021-07-31T03:01:32.333 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:01:32.333 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:01:32.333 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:01:32.333 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:01:32.333 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:01:32.334 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: cephadm 2021-07-31T03:01:30.904592+0000 mgr.x (mgr.24542) 3187 : cephadm [ERR] 2021-07-31T03:01:32.334 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:01:32.334 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:01:32.334 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: yield 2021-07-31T03:01:32.335 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:01:32.335 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:01:32.335 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:01:32.335 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:01:32.335 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:01:32.336 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:01:32.336 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:01:32.336 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:01:32.336 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:01:32.336 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: task.result() 2021-07-31T03:01:32.337 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:01:32.337 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: cephadm 2021-07-31T03:01:30.905130+0000 mgr.x (mgr.24542) 2021-07-31T03:01:32.337 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: 3188 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:01:32.337 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:01:32.337 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:01:32.338 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: yield 2021-07-31T03:01:32.338 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:01:32.338 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:01:32.338 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:01:32.338 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:01:32.339 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:01:32.339 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:01:32.339 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:01:32.339 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:01:32.339 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:01:32.340 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: task.result() 2021-07-31T03:01:32.340 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:01:32.340 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: 2021-07-31T03:01:32.340 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:01:32.340 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: 2021-07-31T03:01:32.341 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:01:32.341 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:01:32.341 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:01:32.341 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:01:32.341 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:01:32.342 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:01:32.342 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:01:32.342 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:01:32.342 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:01:32.342 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:01:32.343 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:01:32.343 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:01:32.343 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:01:32.343 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:01:32.344 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:01:32.344 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:01:32.344 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:01:32.344 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:01:32.344 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: raise 2021-07-31T03:01:32.345 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:01:32.345 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:01:32.345 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:01:32.345 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:01:32.345 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:32 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:01:32.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:01:32.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: yield 2021-07-31T03:01:32.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:01:32.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:01:32.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:01:32.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:01:32.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:01:32.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:01:32.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:01:32.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:01:32.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:01:32.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: task.result() 2021-07-31T03:01:32.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:01:32.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: cephadm 2021-07-31T03:01:30 2021-07-31T03:01:32.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: .858823+0000 mgr.x (mgr.24542) 3181 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:01:32.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: cluster 2021-07-31T03:01:30.862906+0000 mgr.x (mgr.24542) 3182 : cluster [DBG] pgmap v4236: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:32.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: cephadm 2021-07-31T03:01:30.870155+0000 2021-07-31T03:01:32.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: mgr.x (mgr.24542) 3183 : cephadm [ERR] 2021-07-31T03:01:32.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:01:32.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:01:32.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: yield 2021-07-31T03:01:32.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:01:32.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:01:32.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:01:32.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:01:32.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:01:32.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:01:32.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:01:32.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:01:32.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:01:32.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: task.result() 2021-07-31T03:01:32.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:01:32.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: cephadm 2021-07-31T03:01:30.870964+0000 mgr.x (mgr.24542) 3184 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:01:32.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: cephadm 2021-07-31T03:01:30.895783+0000 mgr.x (mgr.24542) 3185 : cephadm [ERR] 2021-07-31T03:01:32.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:01:32.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:01:32.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: yield 2021-07-31T03:01:32.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:01:32.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:01:32.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:01:32.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:01:32.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:01:32.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:01:32.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:01:32.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:01:32.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:01:32.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: task.result() 2021-07-31T03:01:32.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:01:32.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: 2021-07-31T03:01:32.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: cephadm 2021-07-31T03:01:30.896392+0000 mgr.x (mgr.24542) 3186 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:01:32.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:01:32.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:01:32.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: yield 2021-07-31T03:01:32.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:01:32.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:01:32.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:01:32.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:01:32.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:01:32.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:01:32.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:01:32.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:01:32.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:01:32.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: task.result() 2021-07-31T03:01:32.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:01:32.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: 2021-07-31T03:01:32.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:01:32.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: 2021-07-31T03:01:32.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:01:32.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:01:32.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:01:32.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:01:32.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:01:32.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:01:32.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:01:32.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:01:32.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:01:32.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:01:32.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:01:32.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:01:32.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:01:32.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:01:32.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:01:32.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:01:32.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:01:32.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:01:32.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: raise 2021-07-31T03:01:32.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:01:32.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:01:32.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:01:32.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:01:32.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:01:32.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: cephadm 2021-07-31T03:01:30.904592+0000 2021-07-31T03:01:32.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: mgr.x (mgr.24542) 3187 : cephadm [ERR] 2021-07-31T03:01:32.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:01:32.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:01:32.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: yield 2021-07-31T03:01:32.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:01:32.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:01:32.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:01:32.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:01:32.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:01:32.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:01:32.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:01:32.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:01:32.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:01:32.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: task.result() 2021-07-31T03:01:32.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:01:32.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: cephadm 2021-07-31T03:01:30.905130+0000 mgr.x (mgr.24542) 3188 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:01:32.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:01:32.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:01:32.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: yield 2021-07-31T03:01:32.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:01:32.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:01:32.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:01:32.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:01:32.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:01:32.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:01:32.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:01:32.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:01:32.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:01:32.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: task.result() 2021-07-31T03:01:32.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:01:32.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: 2021-07-31T03:01:32.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:01:32.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: 2021-07-31T03:01:32.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:01:32.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:01:32.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:01:32.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:01:32.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:01:32.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:01:32.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:01:32.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:01:32.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:01:32.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:01:32.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:01:32.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:01:32.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:01:32.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:01:32.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:01:32.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:01:32.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:01:32.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:01:32.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: raise 2021-07-31T03:01:32.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:01:32.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:01:32.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:01:32.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:01:32.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:32 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:01:34.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:34 smithi045 conmon[39718]: cluster 2021-07-31T03:01:32.863386+0000 mgr.x ( 2021-07-31T03:01:34.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:34 smithi045 conmon[39718]: mgr.24542) 3189 : cluster [DBG] pgmap v4237: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:34.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:34 smithi045 conmon[33655]: cluster 2021-07-31T03:01:32.863386+0000 mgr.x (mgr.24542) 3189 : 2021-07-31T03:01:34.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:34 smithi045 conmon[33655]: cluster [DBG] pgmap v4237: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:34.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:34 smithi154 conmon[34553]: cluster 2021-07-31T03:01:32.863386+0000 mgr.x (mgr.24542) 3189 : cluster [DBG] pgmap v4237: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:36.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:36 smithi045 conmon[39718]: cluster 2021-07-31T03:01:34.863994+0000 mgr.x 2021-07-31T03:01:36.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:36 smithi045 conmon[39718]: (mgr.24542) 3190 : cluster [DBG] pgmap v4238: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:36.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:36 smithi045 conmon[33655]: cluster 2021-07-31T03:01:34.863994+0000 mgr.x (mgr.24542) 2021-07-31T03:01:36.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:36 smithi045 conmon[33655]: 3190 : cluster [DBG] pgmap v4238: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:36.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:36 smithi154 conmon[34553]: cluster 2021-07-31T03:01:34.863994+0000 mgr.x (mgr.24542) 3190 : cluster [DBG] 2021-07-31T03:01:36.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:36 smithi154 conmon[34553]: pgmap v4238: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:38.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:38 smithi045 conmon[39718]: cluster 2021-07-31T03:01:36.864440+0000 mgr.x 2021-07-31T03:01:38.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:38 smithi045 conmon[39718]: (mgr.24542) 3191 : cluster [DBG] pgmap v4239: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:38.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:38 smithi045 conmon[33655]: cluster 2021-07-31T03:01:36.864440+0000 mgr.x (mgr.24542) 3191 : cluster [DBG] 2021-07-31T03:01:38.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:38 smithi045 conmon[33655]: pgmap v4239: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:38.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:38 smithi154 conmon[34553]: cluster 2021-07-31T03:01:36.864440+0000 mgr.x (mgr.24542) 3191 : cluster [DBG] pgmap v4239: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:40.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:40 smithi045 conmon[39718]: cluster 2021-07-31T03:01:38.864773+0000 mgr.x ( 2021-07-31T03:01:40.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:40 smithi045 conmon[39718]: mgr.24542) 3192 : cluster [DBG] pgmap v4240: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:40.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:40 smithi045 conmon[33655]: cluster 2021-07-31T03:01:38.864773+0000 mgr.x (mgr.24542) 3192 : cluster [DBG] pgmap v4240: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:40.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:40 smithi154 conmon[34553]: cluster 2021-07-31T03:01:38.864773+0000 mgr.x (mgr.24542) 3192 : cluster [DBG] pgmap v4240: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:41.240 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:01:41.519 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:01:41.519 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 76m ago 77m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:01:41.520 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 74m ago 77m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:01:41.520 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 74m ago 80m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:01:41.520 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 76m ago 82m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:01:41.520 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 76m ago 82m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:01:41.521 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 74m ago 80m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:01:41.521 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 76m ago 81m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:01:41.521 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 76m ago 77m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:01:41.521 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 74m ago 77m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:01:41.521 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 76m ago 80m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:01:41.522 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 76m ago 79m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:01:41.522 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 76m ago 79m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:01:41.522 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 76m ago 79m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:01:41.522 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 74m ago 79m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:01:41.522 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 74m ago 78m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:01:41.523 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 74m ago 78m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:01:41.523 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 74m ago 78m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:01:41.523 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 74m ago 77m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:01:41.856 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:01:41.856 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:01:41.856 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:01:41.856 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:01:41.857 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:01:41.857 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:01:41.857 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:01:41.857 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:01:41.858 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:01:41.858 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:01:41.858 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:01:41.858 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:01:41.859 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:01:41.859 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:01:41.859 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:01:41.859 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:01:41.859 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:01:42.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:42 smithi154 conmon[34553]: cluster 2021-07-31T03:01:40.865329+0000 mgr.x (mgr.24542) 3193 : cluster [DBG] pgmap v4241: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:42.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:42 smithi154 conmon[34553]: audit 2021-07-31T03:01:41.227676+0000 mgr.x (mgr.24542) 3194 : audit [DBG] from='client.18642 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:01:42.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:42 smithi154 conmon[34553]: audit 2021-07-31T03:01:41.511370+0000 mgr.x (mgr.24542) 3195 : audit [DBG] from='client.28588 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:01:42.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:42 smithi154 conmon[34553]: audit 2021-07-31T03:01:41.855878+0000 mon.a (mon.0) 3241 : audit [DBG] from='client.? 172.21.15.45:0/630785834' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:01:42.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:42 smithi045 conmon[33655]: cluster 2021-07-31T03:01:40.865329+0000 2021-07-31T03:01:42.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:42 smithi045 conmon[33655]: mgr.x (mgr.24542) 3193 : cluster [DBG] pgmap v4241: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:42.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:42 smithi045 conmon[33655]: audit 2021-07-31T03:01:41.227676+0000 mgr.x (mgr.24542) 3194 : audit [DBG] from='client.18642 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:01:42.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:42 smithi045 conmon[33655]: audit 2021- 2021-07-31T03:01:42.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:42 smithi045 conmon[33655]: 07-31T03:01:41.511370+0000 mgr.x (mgr.24542) 3195 : audit [DBG] from='client.28588 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:01:42.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:42 smithi045 conmon[33655]: audit 2021-07-31T 2021-07-31T03:01:42.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:42 smithi045 conmon[33655]: 03:01:41.855878+0000 mon.a (mon.0) 3241 : audit [DBG] from='client.? 172.21.15.45:0/630785834' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:01:42.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:42 smithi045 conmon[39718]: cluster 2021-07-31T03:01:40.865329+0000 2021-07-31T03:01:42.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:42 smithi045 conmon[39718]: mgr.x (mgr.24542) 3193 : cluster [DBG] pgmap v4241: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:42.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:42 smithi045 conmon[39718]: audit 2021-07-31T03:01:41.227676+0000 2021-07-31T03:01:42.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:42 smithi045 conmon[39718]: mgr.x (mgr.24542) 3194 : audit [DBG] 2021-07-31T03:01:42.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:42 smithi045 conmon[39718]: from='client.18642 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:01:42.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:42 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T03:01:42.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:42 smithi045 conmon[39718]: 03:01:41.511370+0000 mgr.x (mgr. 2021-07-31T03:01:42.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:42 smithi045 conmon[39718]: 24542) 3195 : audit [DBG] from='client.28588 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:01:42.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:42 smithi045 conmon[39718]: audit 2021-07-31T03:01:42.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:42 smithi045 conmon[39718]: 2021-07-31T03:01:41.855878+0000 2021-07-31T03:01:42.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:42 smithi045 conmon[39718]: mon.a (mon.0) 3241 : audit [DBG] 2021-07-31T03:01:42.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:42 smithi045 conmon[39718]: from='client.? 172.21.15.45:0/630785834' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:01:44.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:44 smithi154 conmon[34553]: cluster 2021-07-31T03:01:42.865716+0000 mgr.x (mgr.24542) 3196 : cluster [DBG] pgmap v4242: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:44.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:44 smithi045 conmon[39718]: cluster 2021-07-31T03:01:42.865716 2021-07-31T03:01:44.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:44 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 3196 : cluster [DBG] pgmap v4242: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:44.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:44 smithi045 conmon[33655]: cluster 2021-07-31T03:01:42.865716+0000 mgr.x (mgr.24542) 3196 : cluster [DBG] pgmap v4242: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:46.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:46 smithi154 conmon[34553]: cluster 2021-07-31T03:01:44.866314+0000 mgr.x (mgr.24542) 3197 : cluster [DBG] pgmap v4243: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:46.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:46 smithi045 conmon[39718]: cluster 2021-07-31T03:01:44.866314+0000 mgr.x (mgr.24542) 3197 : cluster [DBG] pgmap v4243: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:46.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:46 smithi045 conmon[33655]: cluster 2021-07-31T03:01:44.866314+0000 mgr.x ( 2021-07-31T03:01:46.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:46 smithi045 conmon[33655]: mgr.24542) 3197 : cluster [DBG] pgmap v4243: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:48.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:48 smithi154 conmon[34553]: cluster 2021-07-31T03:01:46.866709+0000 mgr.x (mgr.24542) 3198 : cluster [DBG] pgmap v4244: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:48.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:48 smithi154 conmon[34553]: 2021-07-31T03:01:48.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:48 smithi045 conmon[39718]: cluster 2021-07-31T03:01:46.866709+0000 mgr.x (mgr.24542 2021-07-31T03:01:48.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:48 smithi045 conmon[39718]: ) 3198 : cluster [DBG] pgmap v4244: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:48.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:48 smithi045 conmon[33655]: cluster 2021-07-31T03:01:46.866709+0000 mgr.x (mgr.24542) 3198 : cluster [DBG] pgmap v4244: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:48.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:48 smithi045 conmon[33655]: 2021-07-31T03:01:50.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:50 smithi154 conmon[34553]: cluster 2021-07-31T03:01:48.867119+0000 mgr.x (mgr.24542) 3199 : cluster [DBG] pgmap v4245: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:50.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:50 smithi045 conmon[39718]: cluster 2021-07-31T03:01:48.867119+0000 mgr.x (mgr. 2021-07-31T03:01:50.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:50 smithi045 conmon[39718]: 24542) 3199 : cluster [DBG] pgmap v4245: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:50.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:50 smithi045 conmon[33655]: cluster 2021-07-31T03:01:48.867119+0000 mgr.x (mgr.24542) 3199 : 2021-07-31T03:01:50.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:50 smithi045 conmon[33655]: cluster [DBG] pgmap v4245: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:52.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:52 smithi154 conmon[34553]: cluster 2021-07-31T03:01:50.867594+0000 mgr.x (mgr.24542) 3200 : cluster [DBG] pgmap v4246: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:52.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:52 smithi045 conmon[39718]: cluster 2021-07-31T03:01:50. 2021-07-31T03:01:52.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:52 smithi045 conmon[39718]: 867594+0000 mgr.x (mgr.24542) 3200 : cluster [DBG] pgmap v4246: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:52.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:52 smithi045 conmon[33655]: cluster 2021-07-31T03:01:50.867594+0000 mgr.x (mgr.24542) 3200 2021-07-31T03:01:52.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:52 smithi045 conmon[33655]: : cluster [DBG] pgmap v4246: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:54.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:54 smithi154 conmon[34553]: cluster 2021-07-31T03:01:52.868089+0000 mgr.x (mgr.24542) 3201 : cluster [DBG] pgmap v4247: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:54.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:54 smithi045 conmon[33655]: cluster 2021-07-31T03:01:52.868089+0000 mgr.x (mgr 2021-07-31T03:01:54.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:54 smithi045 conmon[33655]: .24542) 3201 : cluster [DBG] pgmap v4247: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:54.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:54 smithi045 conmon[39718]: cluster 2021-07-31T03:01:52.868089+0000 mgr.x (mgr 2021-07-31T03:01:54.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:54 smithi045 conmon[39718]: .24542) 3201 : cluster [DBG] pgmap v4247: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:56.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:56 smithi154 conmon[34553]: cluster 2021-07-31T03:01:54.868516+0000 mgr.x (mgr.24542) 3202 : cluster [DBG] pgmap v4248: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:56.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:56 smithi045 conmon[39718]: cluster 2021-07-31T03:01:54.868516+0000 mgr.x 2021-07-31T03:01:56.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:56 smithi045 conmon[39718]: (mgr.24542) 3202 : cluster [DBG] pgmap v4248: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:56.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:56 smithi045 conmon[33655]: cluster 2021-07-31T03:01:54.868516+0000 mgr.x (mgr.24542 2021-07-31T03:01:56.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:56 smithi045 conmon[33655]: ) 3202 : cluster [DBG] pgmap v4248: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:58.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:01:58 smithi154 conmon[34553]: cluster 2021-07-31T03:01:56.869004+0000 mgr.x (mgr.24542) 3203 : cluster [DBG] pgmap v4249: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:58.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:58 smithi045 conmon[33655]: cluster 2021-07-31T03:01:56.869004+0000 mgr.x (mgr.24542 2021-07-31T03:01:58.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:01:58 smithi045 conmon[33655]: ) 3203 : cluster [DBG] pgmap v4249: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:01:58.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:58 smithi045 conmon[39718]: cluster 2021-07-31T03:01:56.869004+0000 mgr.x (mgr 2021-07-31T03:01:58.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:01:58 smithi045 conmon[39718]: .24542) 3203 : cluster [DBG] pgmap v4249: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:00.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:00 smithi154 conmon[34553]: cluster 2021-07-31T03:01:58.869401+0000 mgr.x (mgr.24542) 3204 : cluster [DBG] pgmap v4250: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:00.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:00 smithi045 conmon[39718]: cluster 2021-07-31T03:01:58.869401+0000 mgr.x (mgr.24542) 3204 : cluster [DBG] pgmap v4250: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:00.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:00 smithi045 conmon[33655]: cluster 2021-07-31T03:01:58.869401+0000 mgr.x (mgr.24542) 3204 2021-07-31T03:02:00.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:00 smithi045 conmon[33655]: : cluster [DBG] pgmap v4250: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:02.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:02 smithi154 conmon[34553]: cluster 2021-07-31T03:02:00.869933+0000 mgr.x (mgr.24542) 3205 : cluster [DBG] pgmap v4251: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:02.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:02 smithi045 conmon[33655]: cluster 2021-07-31T03:02:00.869933+0000 mgr.x (mgr.24542) 3205 : 2021-07-31T03:02:02.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:02 smithi045 conmon[33655]: cluster [DBG] pgmap v4251: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:02.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:02 smithi045 conmon[39718]: cluster 2021-07-31T03:02:00 2021-07-31T03:02:02.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:02 smithi045 conmon[39718]: .869933+0000 mgr.x (mgr.24542) 3205 : cluster [DBG] pgmap v4251: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:04.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:04 smithi154 conmon[34553]: cluster 2021-07-31T03:02:02.870284+0000 mgr.x (mgr.24542) 3206 : cluster [DBG] pgmap v4252: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:04.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:04 smithi045 conmon[39718]: cluster 2021-07-31T03:02: 2021-07-31T03:02:04.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:04 smithi045 conmon[39718]: 02.870284+0000 mgr.x (mgr.24542) 3206 : cluster [DBG] pgmap v4252: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:04.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:04 smithi045 conmon[33655]: cluster 2021-07-31T03:02:02.870284+0000 mgr.x (mgr.24542) 2021-07-31T03:02:04.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:04 smithi045 conmon[33655]: 3206 : cluster [DBG] pgmap v4252: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:06.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:06 smithi154 conmon[34553]: cluster 2021-07-31T03:02:04.870707+0000 mgr.x (mgr.24542) 3207 : cluster [DBG] pgmap v4253: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:06.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:06 smithi045 conmon[39718]: cluster 2021-07-31T03:02:04.870707+0000 mgr.x (mgr.24542) 3207 : cluster 2021-07-31T03:02:06.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:06 smithi045 conmon[39718]: [DBG] pgmap v4253: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:06.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:06 smithi045 conmon[33655]: cluster 2021-07-31T03:02:04.870707+0000 mgr.x (mgr.24542) 3207 : cluster [DBG] pgmap v4253: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:08.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:08 smithi154 conmon[34553]: cluster 2021-07-31T03:02:06.871174+0000 mgr.x (mgr.24542) 3208 : cluster [DBG] pgmap v4254: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:08.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:08 smithi045 conmon[39718]: cluster 2021-07-31T03:02:06 2021-07-31T03:02:08.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:08 smithi045 conmon[39718]: .871174+0000 mgr.x (mgr.24542) 3208 : cluster [DBG] pgmap v4254: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:08.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:08 smithi045 conmon[33655]: cluster 2021-07-31T03:02:06.871174+0000 mgr.x (mgr.24542) 3208 : cluster [DBG] pgmap v4254: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:08.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:08 smithi045 conmon[33655]: 2021-07-31T03:02:10.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:10 smithi154 conmon[34553]: cluster 2021-07-31T03:02:08.871676+0000 mgr.x (mgr.24542) 3209 : cluster [DBG] pgmap v4255: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:10.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:10 smithi045 conmon[39718]: cluster 2021-07-31T03:02:08.871676+0000 mgr.x ( 2021-07-31T03:02:10.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:10 smithi045 conmon[39718]: mgr.24542) 3209 : cluster [DBG] pgmap v4255: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:10.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:10 smithi045 conmon[33655]: cluster 2021-07-31T03:02:08.871676+0000 mgr.x (mgr 2021-07-31T03:02:10.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:10 smithi045 conmon[33655]: .24542) 3209 : cluster [DBG] pgmap v4255: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:12.173 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:02:12.451 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:02:12.451 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 77m ago 77m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:02:12.452 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 74m ago 77m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:02:12.452 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 74m ago 81m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:02:12.452 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 77m ago 83m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:02:12.452 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 77m ago 83m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:02:12.453 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 74m ago 81m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:02:12.453 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 77m ago 81m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:02:12.453 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 77m ago 78m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:02:12.453 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 74m ago 78m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:02:12.454 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 77m ago 80m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:02:12.454 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 77m ago 80m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:02:12.454 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 77m ago 80m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:02:12.454 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 77m ago 79m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:02:12.455 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 74m ago 79m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:02:12.455 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 74m ago 79m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:02:12.455 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 74m ago 78m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:02:12.455 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 74m ago 78m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:02:12.455 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 74m ago 78m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:02:12.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:12 smithi154 conmon[34553]: cluster 2021-07-31T03:02:10.872220+0000 mgr.x (mgr.24542) 3210 : cluster [DBG] pgmap v4256: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:12.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:12 smithi045 conmon[33655]: cluster 2021-07-31T03:02:10.872220 2021-07-31T03:02:12.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:12 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 3210 : cluster [DBG] pgmap v4256: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:12.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:12 smithi045 conmon[39718]: cluster 2021-07-31T03:02 2021-07-31T03:02:12.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:12 smithi045 conmon[39718]: :10.872220+0000 mgr.x (mgr.24542) 3210 : cluster [DBG] pgmap v4256: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:12.785 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:02:12.786 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:02:12.786 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:02:12.786 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:02:12.786 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:02:12.787 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:02:12.787 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:02:12.787 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:02:12.787 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:02:12.788 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:02:12.788 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:02:12.788 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:02:12.788 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:02:12.788 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:02:12.789 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:02:12.789 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:02:12.789 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:02:13.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:13 smithi154 conmon[34553]: audit 2021-07-31T03:02:12.160977+0000 mgr.x (mgr.24542) 3211 : audit [DBG] from='client.18660 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:02:13.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:13 smithi154 conmon[34553]: audit 2021-07-31T03:02:12.442019+0000 mgr.x 2021-07-31T03:02:13.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:13 smithi154 conmon[34553]: (mgr.24542) 3212 : audit [DBG] from='client.28606 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:02:13.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:13 smithi154 conmon[34553]: audit 2021-07-31T03:02:12.785445+0000 mon.c (mon.1) 169 : audit [DBG] from='client.? 172.21.15.45:0/3123320358' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:02:13.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:13 smithi045 conmon[33655]: audit 2021-07-31T03:02:12.160977+0000 2021-07-31T03:02:13.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:13 smithi045 conmon[33655]: mgr.x (mgr.24542) 3211 : audit [DBG] from='client.18660 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:02:13.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:13 smithi045 conmon[33655]: audit 2021-07-31T03:02:12.442019 2021-07-31T03:02:13.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:13 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 3212 : audit [DBG] from='client.28606 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:02:13.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:13 smithi045 conmon[33655]: audit 2021-07-31T03:02:13.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:13 smithi045 conmon[33655]: 2021-07-31T03:02:12.785445+0000 mon.c (mon.1) 169 : audit 2021-07-31T03:02:13.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:13 smithi045 conmon[33655]: [DBG] from='client.? 172.21.15.45:0/3123320358' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:02:13.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:13 smithi045 conmon[39718]: audit 2021-07-31T03:02:12.160977+0000 mgr.x (mgr 2021-07-31T03:02:13.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:13 smithi045 conmon[39718]: .24542) 3211 : audit [DBG] from='client.18660 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:02:13.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:13 smithi045 conmon[39718]: audit 2021-07-31T03: 2021-07-31T03:02:13.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:13 smithi045 conmon[39718]: 02:12.442019+0000 mgr.x (mgr.24542) 3212 : audit [DBG] from='client.28606 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:02:13.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:13 smithi045 conmon[39718]: audit 2021 2021-07-31T03:02:13.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:13 smithi045 conmon[39718]: -07-31T03:02:12.785445+0000 mon.c 2021-07-31T03:02:13.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:13 smithi045 conmon[39718]: (mon.1) 169 : audit [DBG] from='client.? 172.21.15.45:0/3123320358' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:02:13.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:13 smithi045 conmon[39718]: 2021-07-31T03:02:14.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:14 smithi154 conmon[34553]: cluster 2021-07-31T03:02:12.872752+0000 mgr.x (mgr.24542) 3213 : cluster [DBG] pgmap v4257: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:14.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:14 smithi045 conmon[33655]: cluster 2021-07-31T03:02:12.872752 2021-07-31T03:02:14.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:14 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 3213 : cluster [DBG] pgmap v4257: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:14.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:14 smithi045 conmon[39718]: cluster 2021-07-31T03:02:12.872752+0000 mgr.x (mgr 2021-07-31T03:02:14.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:14 smithi045 conmon[39718]: .24542) 3213 : cluster [DBG] pgmap v4257: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:16.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:16 smithi154 conmon[34553]: cluster 2021-07-31T03:02:14.873249+0000 mgr.x (mgr.24542) 3214 : cluster [DBG] pgmap v4258: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:16.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:16 smithi045 conmon[33655]: cluster 2021-07-31T03:02:14.873249+0000 mgr.x (mgr.24542) 2021-07-31T03:02:16.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:16 smithi045 conmon[33655]: 3214 : cluster [DBG] pgmap v4258: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:16.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:16 smithi045 conmon[39718]: cluster 2021-07-31T03:02:14.873249+0000 mgr.x (mgr 2021-07-31T03:02:16.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:16 smithi045 conmon[39718]: .24542) 3214 : cluster [DBG] pgmap v4258: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:18.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:18 smithi154 conmon[34553]: cluster 2021-07-31T03:02:16.873830+0000 mgr.x (mgr.24542) 3215 : cluster [DBG] pgmap v4259: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:18.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:18 smithi045 conmon[33655]: cluster 2021-07-31T03:02:16.873830+0000 mgr.x (mgr.24542) 3215 : cluster [DBG] pgmap v4259: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:18.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:18 smithi045 conmon[39718]: cluster 2021-07-31T03:02:16.873830+0000 mgr.x ( 2021-07-31T03:02:18.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:18 smithi045 conmon[39718]: mgr.24542) 3215 : cluster [DBG] pgmap v4259: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:20.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:20 smithi154 conmon[34553]: cluster 2021-07-31T03:02:18.874269+0000 mgr.x (mgr.24542) 3216 : cluster [DBG] pgmap v4260: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:20.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:20 smithi045 conmon[33655]: cluster 2021-07-31T03:02:18.874269+0000 mgr.x (mgr.24542) 2021-07-31T03:02:20.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:20 smithi045 conmon[33655]: 3216 : cluster [DBG] pgmap v4260: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:20.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:20 smithi045 conmon[39718]: cluster 2021-07-31T03:02:18.874269+0000 mgr.x (mgr. 2021-07-31T03:02:20.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:20 smithi045 conmon[39718]: 24542) 3216 : cluster [DBG] pgmap v4260: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:22.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:22 smithi154 conmon[34553]: cluster 2021-07-31T03:02:20.874733+0000 mgr.x (mgr.24542) 3217 : cluster [DBG] pgmap v4261: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:22.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:22 smithi045 conmon[33655]: cluster 2021-07-31T03:02:20.874733+0000 mgr.x (mgr.24542) 3217 : cluster 2021-07-31T03:02:22.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:22 smithi045 conmon[33655]: [DBG] pgmap v4261: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:22.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:22 smithi045 conmon[39718]: cluster 2021-07-31T03:02:20.874733+0000 mgr.x (mgr.24542) 3217 : cluster [DBG] pgmap v4261: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:24.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:24 smithi154 conmon[34553]: cluster 2021-07-31T03:02:22.875265+0000 mgr.x (mgr.24542) 3218 : cluster [DBG] pgmap v4262: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:24.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:24 smithi045 conmon[39718]: cluster 2021-07-31T03 2021-07-31T03:02:24.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:24 smithi045 conmon[39718]: :02:22.875265+0000 mgr.x (mgr.24542) 3218 : cluster [DBG] pgmap v4262: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:24.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:24 smithi045 conmon[33655]: cluster 2021-07-31T03:02:22.875265+0000 mgr.x (mgr.24542) 3218 : 2021-07-31T03:02:24.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:24 smithi045 conmon[33655]: cluster [DBG] pgmap v4262: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:26.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:26 smithi154 conmon[34553]: cluster 2021-07-31T03:02:24.875773+0000 mgr.x (mgr.24542) 3219 : cluster [DBG] 2021-07-31T03:02:26.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:26 smithi154 conmon[34553]: pgmap v4263: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:26.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:26 smithi045 conmon[39718]: cluster 2021-07-31T03:02:24.875773+0000 mgr.x (mgr.24542) 3219 : cluster [DBG] pgmap v4263: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:26.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:26 smithi045 conmon[33655]: cluster 2021-07-31T03:02:24.875773+0000 mgr.x (mgr.24542) 3219 2021-07-31T03:02:26.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:26 smithi045 conmon[33655]: : cluster [DBG] pgmap v4263: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:28.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:28 smithi154 conmon[34553]: cluster 2021-07-31T03:02:26.876317+0000 mgr.x (mgr.24542) 3220 : cluster [DBG] pgmap v4264: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:28.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:28 smithi045 conmon[39718]: cluster 2021-07-31T03:02:26.876317+0000 mgr.x (mgr.24542) 3220 : cluster [DBG] pgmap v4264: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:28.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:28 smithi045 conmon[39718]: 2021-07-31T03:02:28.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:28 smithi045 conmon[33655]: cluster 2021-07-31T03:02:26.876317+0000 mgr.x (mgr.24542) 3220 : cluster [DBG] pgmap v4264: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:28.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:28 smithi045 conmon[33655]: 2021-07-31T03:02:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:29 smithi154 conmon[34553]: audit 2021-07-31T03:02:28.234297+0000 mon.b (mon.2) 450 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:02:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:29 smithi154 conmon[34553]: audit 2021-07-31T03:02:28.234660+0000 mon.b (mon.2) 451 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:02:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:29 smithi154 conmon[34553]: audit 2021-07-31T03:02:28.234743+0000 mon.a (mon.0) 3242 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:02:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:29 smithi154 conmon[34553]: audit 2021-07-31T 2021-07-31T03:02:29.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:29 smithi154 conmon[34553]: 03:02:28.235186+0000 mon.a (mon. 2021-07-31T03:02:29.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:29 smithi154 conmon[34553]: 0) 3243 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:02:29.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:29 smithi045 conmon[39718]: audit 2021-07-31T03:02:28.234297+0000 mon.b (mon.2) 450 2021-07-31T03:02:29.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:29 smithi045 conmon[39718]: : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:02:29.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:29 smithi045 conmon[39718]: audit 2021-07-31T03:02:28.234660+0000 2021-07-31T03:02:29.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:29 smithi045 conmon[39718]: mon.b (mon.2) 451 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:02:29.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:29 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T03:02:29.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:29 smithi045 conmon[39718]: 31T03:02:28.234743+0000 2021-07-31T03:02:29.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:29 smithi045 conmon[39718]: mon.a (mon.0) 3242 : audit [INF] 2021-07-31T03:02:29.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:29 smithi045 conmon[39718]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:02:29.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:29 smithi045 conmon[39718]: audit 2021-07-31T03 2021-07-31T03:02:29.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:29 smithi045 conmon[39718]: :02:28.235186+0000 mon.a (mon 2021-07-31T03:02:29.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:29 smithi045 conmon[39718]: .0) 3243 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:02:29.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:29 smithi045 conmon[33655]: audit 2021-07-31T03:02:28.234297+0000 mon.b (mon.2) 450 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:02:29.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:29 smithi045 conmon[33655]: audit 2021-07-31T03:02:28.234660+0000 mon.b (mon.2) 451 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:02:29.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:29 smithi045 conmon[33655]: audit 2021- 2021-07-31T03:02:29.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:29 smithi045 conmon[33655]: 07-31T03:02:28.234743+0000 mon.a (mon.0) 3242 : audit [INF] 2021-07-31T03:02:29.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:29 smithi045 conmon[33655]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:02:29.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:29 smithi045 conmon[33655]: audit 2021-07-31T03:02:29.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:29 smithi045 conmon[33655]: 2021-07-31T03:02 2021-07-31T03:02:29.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:29 smithi045 conmon[33655]: :28.235186+0000 mon.a (mon 2021-07-31T03:02:29.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:29 smithi045 conmon[33655]: .0) 3243 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:02:30.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:30 smithi154 conmon[34553]: cluster 2021-07-31T03:02:28.876765+0000 mgr.x (mgr.24542) 3221 : cluster 2021-07-31T03:02:30.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:30 smithi154 conmon[34553]: [DBG] pgmap v4265: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:30.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:30 smithi045 conmon[39718]: cluster 2021-07-31T03:02:28.876765+0000 mgr.x (mgr.24542) 3221 : cluster [DBG] pgmap v4265: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:30.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:30 smithi045 conmon[33655]: cluster 2021-07-31T03:02:28.876765+0000 mgr.x (mgr.24542) 3221 : 2021-07-31T03:02:30.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:30 smithi045 conmon[33655]: cluster [DBG] pgmap v4265: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:31.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:31 smithi154 conmon[34553]: audit 2021-07-31T03:02:30.905964+0000 mon.b (mon 2021-07-31T03:02:31.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:31 smithi154 conmon[34553]: .2) 452 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:02:31.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:31 smithi154 conmon[34553]: audit 2021-07-31T03:02:30.907456+0000 mon.b (mon.2) 453 : audit 2021-07-31T03:02:31.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:31 smithi154 conmon[34553]: [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:02:31.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:31 smithi154 conmon[34553]: audit 2021- 2021-07-31T03:02:31.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:31 smithi154 conmon[34553]: 07-31T03:02:30.908499+0000 mon.b (mon.2) 454 : audit 2021-07-31T03:02:31.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:31 smithi154 conmon[34553]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:02:31.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:30 smithi154 conmon[57492]: debug 2021-07-31T03:02:30.951+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:02:31.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:02:31.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:02:31.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:30 smithi154 conmon[57492]: yield 2021-07-31T03:02:31.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:02:31.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:02:31.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:02:31.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:02:31.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:02:31.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:02:31.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:02:31.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:02:31.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:02:31.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:30 smithi154 conmon[57492]: task.result() 2021-07-31T03:02:31.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:02:31.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:30 smithi154 conmon[57492]: debug 2021-07-31T03:02:30.962+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:02:31.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:02:31.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:02:31.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:30 smithi154 conmon[57492]: yield 2021-07-31T03:02:31.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:02:31.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:02:31.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:02:31.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:02:31.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:02:31.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:02:31.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:02:31.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:02:31.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:02:31.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:30 smithi154 conmon[57492]: task.result() 2021-07-31T03:02:31.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:02:31.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:30 smithi154 conmon[57492]: debug 2021-07-31T03:02:30.990+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:02:31.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:02:31.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:02:31.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:30 smithi154 conmon[57492]: yield 2021-07-31T03:02:31.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:02:31.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:02:31.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:02:31.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:02:31.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:02:31.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:02:31.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:02:31.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:02:31.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:02:31.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:30 smithi154 conmon[57492]: task.result() 2021-07-31T03:02:31.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:02:31.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:30 smithi154 conmon[57492]: debug 2021-07-31T03:02:30.990+0000 7fcb94162700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:02:31.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:02:31.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:02:31.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:30 smithi154 conmon[57492]: yield 2021-07-31T03:02:31.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:02:31.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:30 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:02:31.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:02:31.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:30 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:02:31.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:02:31.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:30 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:02:31.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:02:31.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:30 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:02:31.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:30 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:02:31.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:30 smithi154 conmon[57492]: task.result() 2021-07-31T03:02:31.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:30 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:02:31.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:30 smithi154 conmon[57492]: 2021-07-31T03:02:31.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:30 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:02:31.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:30 smithi154 conmon[57492]: 2021-07-31T03:02:31.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:30 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:02:31.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:02:31.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:30 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:02:31.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:02:31.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:30 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:02:31.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:02:31.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:30 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:02:31.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:02:31.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:30 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:02:31.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:30 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:02:31.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:30 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:02:31.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:30 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:02:31.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:30 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:02:31.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:02:31.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:30 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:02:31.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:02:31.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:30 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:02:31.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:02:31.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:30 smithi154 conmon[57492]: raise 2021-07-31T03:02:31.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:30 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:02:31.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:30 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:02:31.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:30 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:02:31.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:30 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:02:31.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:30 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:02:31.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:31 smithi154 conmon[57492]: debug 2021-07-31T03:02:30.999+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:02:31.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:02:31.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:02:31.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:31 smithi154 conmon[57492]: yield 2021-07-31T03:02:31.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:02:31.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:02:31.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:02:31.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:02:31.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:02:31.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:02:31.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:02:31.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:02:31.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:02:31.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:31 smithi154 conmon[57492]: task.result() 2021-07-31T03:02:31.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:02:31.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:31 smithi154 conmon[57492]: debug 2021-07-31T03:02:30.999+0000 7fcb93961700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:02:31.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:02:31.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:02:31.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:31 smithi154 conmon[57492]: yield 2021-07-31T03:02:31.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:02:31.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:02:31.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:02:31.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:02:31.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:02:31.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:02:31.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:02:31.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:02:31.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:02:31.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:31 smithi154 conmon[57492]: task.result() 2021-07-31T03:02:31.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:02:31.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:31 smithi154 conmon[57492]: 2021-07-31T03:02:31.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:31 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:02:31.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:31 smithi154 conmon[57492]: 2021-07-31T03:02:31.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:02:31.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:02:31.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:31 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:02:31.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:02:31.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:31 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:02:31.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:02:31.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:31 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:02:31.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:02:31.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:31 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:02:31.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:31 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:02:31.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:31 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:02:31.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:31 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:02:31.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:31 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:02:31.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:02:31.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:31 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:02:31.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:02:31.284 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:31 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:02:31.284 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:02:31.284 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:31 smithi154 conmon[57492]: raise 2021-07-31T03:02:31.284 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:31 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:02:31.284 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:31 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:02:31.285 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:02:31.285 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:31 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:02:31.285 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:02:31 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:02:31.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:31 smithi045 conmon[39718]: audit 2021-07-31T03:02:30. 2021-07-31T03:02:31.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:31 smithi045 conmon[39718]: 905964+0000 mon.b (mon.2) 452 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:02:31.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:31 smithi045 conmon[39718]: audit 2021-07-31T03:02:31.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:31 smithi045 conmon[39718]: 2021-07- 2021-07-31T03:02:31.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:31 smithi045 conmon[39718]: 31T03:02:30.907456+0000 2021-07-31T03:02:31.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:31 smithi045 conmon[39718]: mon.b (mon.2) 453 : audit 2021-07-31T03:02:31.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:31 smithi045 conmon[39718]: [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:02:31.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:31 smithi045 conmon[39718]: audit 2021 2021-07-31T03:02:31.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:31 smithi045 conmon[39718]: -07-31T03:02 2021-07-31T03:02:31.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:31 smithi045 conmon[39718]: :30.908499+0000 mon.b (mon 2021-07-31T03:02:31.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:31 smithi045 conmon[39718]: .2) 454 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:02:31.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:31 smithi045 conmon[33655]: audit 2021-07-31T03:02:30.905964+0000 mon.b (mon.2) 452 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:02:31.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:31 smithi045 conmon[33655]: 2021-07-31T03:02:31.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:31 smithi045 conmon[33655]: audit 2021-07-31T03:02:30.907456+0000 mon.b (mon.2) 453 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:02:31.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:31 smithi045 conmon[33655]: audit 2021-07-31T03:02:31.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:31 smithi045 conmon[33655]: 2021-07-31T03:02:30 2021-07-31T03:02:31.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:31 smithi045 conmon[33655]: .908499+0000 mon.b (mon. 2021-07-31T03:02:31.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:31 smithi045 conmon[33655]: 2) 454 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:02:32.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: cluster 2021-07-31T03:02:30.877219+0000 mgr.x (mgr.24542) 2021-07-31T03:02:32.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: 3222 : cluster [DBG] pgmap v4266: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:32.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: cephadm 2021-07-31T03:02:30.909218+0000 mgr.x (mgr.24542) 3223 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:02:32.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: cephadm 2021-07-31T03:02:30.952531+0000 mgr.x ( 2021-07-31T03:02:32.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: mgr.24542) 3224 : cephadm [ERR] 2021-07-31T03:02:32.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:02:32.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:02:32.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: yield 2021-07-31T03:02:32.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:02:32.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:02:32.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:02:32.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:02:32.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:02:32.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:02:32.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:02:32.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:02:32.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:02:32.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: task.result() 2021-07-31T03:02:32.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:02:32.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: cephadm 2021-07-31T03:02:30.952930+0000 mgr.x (mgr.24542) 3225 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:02:32.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: cephadm 2021-07-31T03:02:30.963885+0000 mgr.x (mgr. 2021-07-31T03:02:32.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: 24542) 3226 : cephadm [ERR] 2021-07-31T03:02:32.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:02:32.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:02:32.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: yield 2021-07-31T03:02:32.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:02:32.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:02:32.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:02:32.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:02:32.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:02:32.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:02:32.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:02:32.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:02:32.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:02:32.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: task.result() 2021-07-31T03:02:32.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:02:32.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: cephadm 2021-07-31T03:02:30.964712+0000 mgr.x (mgr.24542) 3227 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:02:32.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: cephadm 2021-07-31T03:02:30.991431+0000 mgr.x (mgr.24542) 3228 : cephadm [ERR] 2021-07-31T03:02:32.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: 2021-07-31T03:02:32.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:02:32.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:02:32.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: yield 2021-07-31T03:02:32.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:02:32.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:02:32.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:02:32.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:02:32.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:02:32.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:02:32.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:02:32.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:02:32.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:02:32.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: task.result() 2021-07-31T03:02:32.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:02:32.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: cephadm 2021-07-31T03:02:30.991805+0000 mgr.x (mgr.24542) 3229 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:02:32.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:02:32.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:02:32.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: yield 2021-07-31T03:02:32.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:02:32.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:02:32.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:02:32.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:02:32.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:02:32.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:02:32.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:02:32.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:02:32.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:02:32.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: task.result() 2021-07-31T03:02:32.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:02:32.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: 2021-07-31T03:02:32.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:02:32.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: 2021-07-31T03:02:32.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:02:32.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:02:32.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:02:32.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:02:32.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:02:32.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:02:32.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:02:32.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:02:32.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:02:32.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:02:32.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:02:32.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:02:32.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:02:32.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:02:32.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:02:32.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:02:32.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:02:32.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:02:32.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: raise 2021-07-31T03:02:32.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:02:32.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:02:32.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:02:32.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:02:32.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:02:32.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: cephadm 2021-07-31T03:02:31.000287+0000 mgr.x (mgr.24542) 3230 : cephadm 2021-07-31T03:02:32.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: [ERR] 2021-07-31T03:02:32.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:02:32.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:02:32.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: yield 2021-07-31T03:02:32.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:02:32.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:02:32.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:02:32.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:02:32.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:02:32.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:02:32.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:02:32.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:02:32.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:02:32.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: task.result() 2021-07-31T03:02:32.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:02:32.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: cephadm 2021-07-31T03:02:31.000677+0000 mgr.x (mgr.24542) 3231 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:02:32.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:02:32.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:02:32.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: yield 2021-07-31T03:02:32.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:02:32.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:02:32.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:02:32.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:02:32.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:02:32.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:02:32.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:02:32.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:02:32.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:02:32.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: task.result() 2021-07-31T03:02:32.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:02:32.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: 2021-07-31T03:02:32.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:02:32.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: 2021-07-31T03:02:32.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:02:32.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:02:32.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:02:32.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:02:32.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:02:32.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:02:32.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:02:32.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:02:32.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:02:32.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:02:32.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:02:32.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:02:32.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:02:32.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:02:32.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:02:32.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:02:32.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:02:32.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:02:32.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: raise 2021-07-31T03:02:32.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:02:32.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:02:32.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:02:32.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:02:32.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:32 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:02:32.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: cluster 2021-07-31T03:02:30.877219+0000 mgr.x (mgr 2021-07-31T03:02:32.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: .24542) 3222 : cluster [DBG] pgmap v4266: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:32.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: cephadm 2021-07-31T03:02:32.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: 2021-07-31T03 2021-07-31T03:02:32.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: :02:30.909218+0000 mgr.x (mgr 2021-07-31T03:02:32.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: .24542) 3223 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:02:32.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: cephadm 2021- 2021-07-31T03:02:32.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: 07-31T03:02:30.952531+0000 2021-07-31T03:02:32.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: mgr.x (mgr.24542) 3224 : cephadm [ERR] 2021-07-31T03:02:32.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: 2021-07-31T03:02:32.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:02:32.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:02:32.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: yield 2021-07-31T03:02:32.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:02:32.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:02:32.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:02:32.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:02:32.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:02:32.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:02:32.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:02:32.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:02:32.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:02:32.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: task.result() 2021-07-31T03:02:32.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:02:32.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: cephadm 2021-07-31T03:02:30.952930+0000 mgr.x (mgr.24542) 3225 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:02:32.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: cephadm 2021-07-31T03:02:32.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: 2021-07-31T 2021-07-31T03:02:32.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: 03:02:30.963885+0000 2021-07-31T03:02:32.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: mgr.x (mgr.24542) 3226 : cephadm 2021-07-31T03:02:32.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: [ERR] 2021-07-31T03:02:32.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:02:32.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:02:32.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: yield 2021-07-31T03:02:32.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:02:32.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:02:32.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:02:32.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:02:32.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:02:32.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:02:32.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:02:32.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:02:32.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:02:32.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: task.result() 2021-07-31T03:02:32.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:02:32.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: cephadm 2021-07-31T03:02:30.964712+0000 mgr.x (mgr.24542) 3227 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:02:32.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: cephadm 2021-07-31T03:02:32.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: 2021-07 2021-07-31T03:02:32.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: -31T03:02:30.991431 2021-07-31T03:02:32.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 3228 : cephadm 2021-07-31T03:02:32.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: [ERR] 2021-07-31T03:02:32.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:02:32.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:02:32.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: yield 2021-07-31T03:02:32.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:02:32.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:02:32.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:02:32.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:02:32.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:02:32.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:02:32.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:02:32.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:02:32.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:02:32.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: task.result() 2021-07-31T03:02:32.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:02:32.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: cephadm 2021-07-31T03:02:30.991805+0000 mgr.x (mgr.24542) 3229 : 2021-07-31T03:02:32.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:02:32.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:02:32.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:02:32.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: yield 2021-07-31T03:02:32.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:02:32.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:02:32.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:02:32.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:02:32.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:02:32.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:02:32.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:02:32.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:02:32.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:02:32.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: task.result() 2021-07-31T03:02:32.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:02:32.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: 2021-07-31T03:02:32.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:02:32.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: 2021-07-31T03:02:32.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:02:32.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:02:32.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:02:32.555 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: cluster 2021-07-31T03:02:30.877219+0000 mgr.x (mgr.24542) 3222 2021-07-31T03:02:32.555 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: : cluster [DBG] pgmap v4266: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:32.555 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: cephadm 2021-07-31T03:02:30.909218+0000 mgr.x (mgr.24542) 2021-07-31T03:02:32.555 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: 3223 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:02:32.556 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: cephadm 2021-07-31T03:02 2021-07-31T03:02:32.556 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: :30.952531+0000 mgr.x (mgr.24542) 2021-07-31T03:02:32.556 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: 3224 : cephadm [ERR] 2021-07-31T03:02:32.556 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:02:32.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:02:32.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: yield 2021-07-31T03:02:32.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:02:32.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:02:32.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:02:32.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:02:32.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:02:32.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:02:32.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:02:32.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:02:32.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:02:32.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: task.result() 2021-07-31T03:02:32.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:02:32.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: cephadm 2021-07-31T03:02:30.952930+0000 mgr.x (mgr.24542) 3225 : cephadm [INF] 2021-07-31T03:02:32.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:02:32.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: cephadm 2021-07-31T03:02:32.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: 2021-07-31T03: 2021-07-31T03:02:32.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: 02:30.963885+0000 mgr.x ( 2021-07-31T03:02:32.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: mgr.24542) 3226 : cephadm [ERR] 2021-07-31T03:02:32.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: 2021-07-31T03:02:32.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:02:32.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:02:32.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: yield 2021-07-31T03:02:32.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:02:32.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:02:32.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:02:32.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:02:32.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:02:32.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:02:32.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:02:32.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:02:32.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:02:32.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: task.result() 2021-07-31T03:02:32.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:02:32.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:02:32.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:02:32.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:02:32.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:02:32.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:02:32.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:02:32.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:02:32.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:02:32.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:02:32.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:02:32.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:02:32.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:02:32.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:02:32.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:02:32.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:02:32.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: raise 2021-07-31T03:02:32.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:02:32.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:02:32.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:02:32.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:02:32.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:02:32.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: cephadm 2021-07-31T03:02:31.000287+0000 mgr.x (mgr.24542) 3230 : cephadm [ERR] 2021-07-31T03:02:32.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:02:32.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:02:32.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: yield 2021-07-31T03:02:32.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:02:32.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:02:32.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:02:32.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:02:32.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:02:32.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:02:32.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:02:32.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:02:32.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:02:32.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: task.result() 2021-07-31T03:02:32.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:02:32.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: cephadm 2021-07-31T03:02:31.000677+0000 mgr.x (mgr.24542) 3231 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:02:32.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:02:32.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:02:32.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: yield 2021-07-31T03:02:32.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: cephadm 2021-07-31T03:02:30.964712+0000 mgr.x (mgr.24542) 3227 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:02:32.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: cephadm 2021-07-31T03:02:32.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: 2021-07-31T03: 2021-07-31T03:02:32.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: 02:30.991431+0000 mgr.x 2021-07-31T03:02:32.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: (mgr.24542) 3228 : cephadm 2021-07-31T03:02:32.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: [ERR] 2021-07-31T03:02:32.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:02:32.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:02:32.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: yield 2021-07-31T03:02:32.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:02:32.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:02:32.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:02:32.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:02:32.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:02:32.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:02:32.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:02:32.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:02:32.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:02:32.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: task.result() 2021-07-31T03:02:32.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:02:32.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: cephadm 2021-07-31T03:02:30.991805+0000 mgr.x (mgr.24542) 3229 2021-07-31T03:02:32.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:02:32.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:02:32.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:02:32.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: yield 2021-07-31T03:02:32.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:02:32.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:02:32.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:02:32.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:02:32.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:02:32.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:02:32.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:02:32.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:02:32.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:02:32.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: task.result() 2021-07-31T03:02:32.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:02:32.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: 2021-07-31T03:02:32.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:02:32.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: 2021-07-31T03:02:32.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:02:32.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:02:32.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:02:32.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:02:32.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:02:32.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:02:32.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:02:32.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:02:32.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:02:32.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:02:32.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:02:32.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:02:32.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:02:32.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:02:32.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:02:32.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:02:32.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:02:32.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:02:32.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: raise 2021-07-31T03:02:32.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:02:32.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:02:32.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:02:32.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:02:32.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:02:32.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: cephadm 2021-07-31T03:02:31.000287+0000 mgr.x (mgr.24542) 3230 : cephadm [ERR] 2021-07-31T03:02:32.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:02:32.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:02:32.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: yield 2021-07-31T03:02:32.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:02:32.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:02:32.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:02:32.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:02:32.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:02:32.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:02:32.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:02:32.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:02:32.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:02:32.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: task.result() 2021-07-31T03:02:32.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:02:32.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: cephadm 2021-07-31T03:02:31.000677+0000 mgr.x (mgr.24542) 3231 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:02:32.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:02:32.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:02:32.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: yield 2021-07-31T03:02:32.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:02:32.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:02:32.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:02:32.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:02:32.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:02:32.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:02:32.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:02:32.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:02:32.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:02:32.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: task.result() 2021-07-31T03:02:32.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:02:32.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: 2021-07-31T03:02:32.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:02:32.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: 2021-07-31T03:02:32.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:02:32.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:02:32.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:02:32.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:02:32.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:02:32.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:02:32.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:02:32.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:02:32.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:02:32.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:02:32.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:02:32.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:02:32.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:02:32.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:02:32.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:02:32.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:02:32.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:02:32.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:02:32.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: raise 2021-07-31T03:02:32.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:02:32.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:02:32.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:02:32.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:02:32.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:02:32.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:02:32.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:02:32.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:02:32.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:02:32.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:02:32.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:02:32.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:02:32.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:02:32.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:02:32.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: task.result() 2021-07-31T03:02:32.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:02:32.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: 2021-07-31T03:02:32.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:02:32.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: 2021-07-31T03:02:32.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:02:32.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:02:32.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:02:32.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:02:32.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:02:32.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:02:32.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:02:32.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:02:32.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:02:32.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:02:32.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:02:32.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:02:32.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:02:32.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:02:32.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:02:32.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:02:32.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:02:32.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:02:32.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: raise 2021-07-31T03:02:32.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:02:32.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:02:32.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:02:32.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:02:32.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:32 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:02:34.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:34 smithi154 conmon[34553]: cluster 2021-07-31T03:02:32.877669+0000 mgr.x (mgr.24542) 3232 : cluster [DBG] pgmap v4267: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:34.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:34 smithi045 conmon[39718]: cluster 2021-07-31T03:02:32.877669+0000 mgr.x ( 2021-07-31T03:02:34.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:34 smithi045 conmon[39718]: mgr.24542) 3232 : cluster [DBG] pgmap v4267: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:34.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:34 smithi045 conmon[33655]: cluster 2021-07-31T03:02:32.877669+0000 mgr.x (mgr.24542) 3232 : cluster [DBG] 2021-07-31T03:02:34.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:34 smithi045 conmon[33655]: pgmap v4267: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:36.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:36 smithi154 conmon[34553]: cluster 2021-07-31T03:02:34.878212+0000 mgr.x (mgr.24542) 3233 : cluster [DBG] pgmap v4268: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:36.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:36 smithi045 conmon[39718]: cluster 2021-07-31T03:02:34.878212+0000 mgr.x 2021-07-31T03:02:36.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:36 smithi045 conmon[39718]: (mgr.24542) 3233 : cluster [DBG] pgmap v4268: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:36.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:36 smithi045 conmon[33655]: cluster 2021-07-31T03:02:34.878212+0000 mgr.x (mgr.24542) 2021-07-31T03:02:36.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:36 smithi045 conmon[33655]: 3233 : cluster [DBG] pgmap v4268: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:38.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:38 smithi154 conmon[34553]: cluster 2021-07-31T03:02:36.878748+0000 mgr.x (mgr.24542) 3234 : cluster [DBG] pgmap v4269: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:38.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:38 smithi045 conmon[39718]: cluster 2021-07-31T03:02:36.878748+0000 mgr.x 2021-07-31T03:02:38.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:38 smithi045 conmon[39718]: (mgr.24542) 3234 : cluster [DBG] pgmap v4269: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:38.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:38 smithi045 conmon[33655]: cluster 2021-07-31T03:02:36.878748+0000 2021-07-31T03:02:38.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:38 smithi045 conmon[33655]: mgr.x (mgr.24542) 3234 : cluster [DBG] pgmap v4269: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:40.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:40 smithi154 conmon[34553]: cluster 2021-07-31T03:02:38.879311+0000 mgr.x (mgr.24542) 3235 : cluster [DBG] pgmap v4270: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:40.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:40 smithi045 conmon[39718]: cluster 2021-07-31T03:02:38.879311+0000 mgr.x ( 2021-07-31T03:02:40.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:40 smithi045 conmon[39718]: mgr.24542) 3235 : cluster [DBG] pgmap v4270: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:40.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:40 smithi045 conmon[33655]: cluster 2021-07-31T03:02:38.879311+0000 mgr.x 2021-07-31T03:02:40.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:40 smithi045 conmon[33655]: (mgr.24542) 3235 : cluster [DBG] pgmap v4270: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:42.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:42 smithi154 conmon[34553]: cluster 2021-07-31T03:02:40.879839+0000 mgr.x (mgr.24542) 3236 : cluster [DBG] pgmap v4271: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:42.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:42 smithi045 conmon[33655]: cluster 2021-07-31T03:02:40.879839+0000 mgr.x (mgr.24542) 3236 : cluster [DBG] pgmap v4271: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:42.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:42 smithi045 conmon[39718]: cluster 2021-07-31T03:02:40.879839+0000 mgr.x ( 2021-07-31T03:02:42.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:42 smithi045 conmon[39718]: mgr.24542) 3236 : cluster [DBG] pgmap v4271: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:43.104 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:02:43.380 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:02:43.380 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 77m ago 78m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:02:43.381 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 75m ago 78m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:02:43.381 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 75m ago 81m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:02:43.382 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 77m ago 83m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:02:43.382 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 77m ago 83m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:02:43.382 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 75m ago 81m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:02:43.383 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 77m ago 82m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:02:43.383 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 77m ago 78m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:02:43.383 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 75m ago 78m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:02:43.384 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 77m ago 81m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:02:43.384 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 77m ago 81m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:02:43.385 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 77m ago 80m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:02:43.385 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 77m ago 80m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:02:43.385 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 75m ago 80m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:02:43.386 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 75m ago 79m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:02:43.386 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 75m ago 79m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:02:43.387 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 75m ago 79m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:02:43.387 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 75m ago 78m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:02:43.717 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:02:43.718 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:02:43.718 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:02:43.718 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:02:43.718 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:02:43.719 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:02:43.719 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:02:43.719 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:02:43.719 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:02:43.719 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:02:43.720 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:02:43.720 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:02:43.720 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:02:43.720 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:02:43.721 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:02:43.721 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:02:43.721 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:02:44.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:44 smithi154 conmon[34553]: cluster 2021-07-31T03:02:42.880294+0000 mgr.x (mgr.24542) 3237 : cluster [DBG] pgmap v4272: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:44.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:44 smithi154 conmon[34553]: audit 2021-07-31T03:02:43.092220+0000 mgr.x (mgr.24542) 3238 : audit [DBG] from='client.28618 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:02:44.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:44 smithi154 conmon[34553]: audit 2021-07-31T03:02:43.372473+0000 mgr.x (mgr.24542) 3239 : audit [DBG] from='client.28624 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:02:44.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:44 smithi154 conmon[34553]: audit 2021-07-31T03:02:43.717353+0000 mon.c (mon.1) 170 : 2021-07-31T03:02:44.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:44 smithi154 conmon[34553]: audit [DBG] from='client.? 172.21.15.45:0/1184809298' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:02:44.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:44 smithi045 conmon[33655]: cluster 2021-07-31T03:02:42.880294+0000 mgr.x (mgr.24542) 3237 : cluster [DBG] pgmap v4272: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:44.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:44 smithi045 conmon[33655]: audit 2021-07-31T03:02:43.092220+0000 mgr.x (mgr.24542) 3238 : audit [DBG] from='client.28618 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:02:44.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:44 smithi045 conmon[33655]: audit 2021-07-31T03:02:43.372473+0000 mgr.x (mgr.24542) 3239 : audit [DBG] from='client.28624 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:02:44.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:44 smithi045 conmon[33655]: audit 2021-07-31T03:02:43.717353+0000 mon.c (mon.1) 170 : audit 2021-07-31T03:02:44.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:44 smithi045 conmon[33655]: [DBG] from='client.? 172.21.15.45:0/1184809298' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:02:44.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:44 smithi045 conmon[39718]: cluster 2021-07-31T03:02:42.880294+0000 mgr.x (mgr.24542) 3237 : cluster [DBG] pgmap v4272: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:44.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:44 smithi045 conmon[39718]: audit 2021-07-31T03:02:44.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:44 smithi045 conmon[39718]: 2021-07-31T03: 2021-07-31T03:02:44.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:44 smithi045 conmon[39718]: 02:43.092220+0000 2021-07-31T03:02:44.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:44 smithi045 conmon[39718]: mgr.x (mgr.24542) 3238 : 2021-07-31T03:02:44.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:44 smithi045 conmon[39718]: audit [DBG] from='client.28618 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:02:44.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:44 smithi045 conmon[39718]: audit 2021 2021-07-31T03:02:44.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:44 smithi045 conmon[39718]: -07-31T03:02:43. 2021-07-31T03:02:44.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:44 smithi045 conmon[39718]: 372473+0000 mgr.x (mgr. 2021-07-31T03:02:44.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:44 smithi045 conmon[39718]: 24542) 3239 : audit [DBG] 2021-07-31T03:02:44.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:44 smithi045 conmon[39718]: from='client.28624 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:02:44.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:44 smithi045 conmon[39718]: audit 2021- 2021-07-31T03:02:44.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:44 smithi045 conmon[39718]: 07-31T03:02:43. 2021-07-31T03:02:44.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:44 smithi045 conmon[39718]: 717353+0000 mon.c (mon. 2021-07-31T03:02:44.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:44 smithi045 conmon[39718]: 1) 170 : audit [DBG] from='client.? 172.21.15.45:0/1184809298' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:02:44.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:44 smithi045 conmon[39718]: 2021-07-31T03:02:46.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:46 smithi154 conmon[34553]: cluster 2021-07-31T03:02:44.880890+0000 mgr.x (mgr.24542) 3240 : cluster [DBG] pgmap v4273: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:46.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:46 smithi045 conmon[39718]: cluster 2021-07-31T03:02:44.880890+0000 mgr.x (mgr.24542) 3240 : cluster [DBG] pgmap v4273: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:46.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:46 smithi045 conmon[33655]: cluster 2021-07-31T03:02:44.880890+0000 mgr.x (mgr.24542) 3240 : cluster [DBG] pgmap v4273: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:46.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:46 smithi045 conmon[33655]: 2021-07-31T03:02:48.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:48 smithi154 conmon[34553]: cluster 2021-07-31T03:02:46.881399+0000 mgr.x (mgr.24542) 3241 : cluster [DBG] pgmap v4274: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:48.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:48 smithi045 conmon[39718]: cluster 2021-07-31T03:02:46.881399+0000 mgr.x (mgr.24542) 3241 : cluster 2021-07-31T03:02:48.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:48 smithi045 conmon[39718]: [DBG] pgmap v4274: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:48.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:48 smithi045 conmon[33655]: cluster 2021-07-31T03:02:46.881399+0000 mgr.x (mgr.24542) 3241 : cluster [DBG] pgmap v4274: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:50.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:50 smithi154 conmon[34553]: cluster 2021-07-31T03:02:48.881892+0000 mgr.x (mgr.24542) 3242 : cluster [DBG] pgmap v4275: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:50.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:50 smithi045 conmon[33655]: cluster 2021-07-31T03:02:48.881892+0000 mgr.x (mgr.24542) 3242 : cluster [DBG] 2021-07-31T03:02:50.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:50 smithi045 conmon[33655]: pgmap v4275: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:50.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:50 smithi045 conmon[39718]: cluster 2021-07-31T03:02:48.881892+0000 mgr.x (mgr 2021-07-31T03:02:50.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:50 smithi045 conmon[39718]: .24542) 3242 : cluster [DBG] pgmap v4275: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:52.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:52 smithi154 conmon[34553]: cluster 2021-07-31T03:02:50.882289+0000 mgr.x (mgr.24542) 3243 : cluster [DBG] pgmap v4276: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:52.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:52 smithi045 conmon[33655]: cluster 2021-07-31T03:02:50.882289+0000 mgr.x (mgr.24542) 3243 : 2021-07-31T03:02:52.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:52 smithi045 conmon[33655]: cluster [DBG] pgmap v4276: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:52.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:52 smithi045 conmon[39718]: cluster 2021-07-31T03:02:50.882289+0000 mgr.x (mgr.24542) 3243 : cluster [DBG] pgmap v4276: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:54.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:54 smithi154 conmon[34553]: cluster 2021-07-31T03:02:52.882731+0000 mgr.x (mgr.24542) 3244 : cluster [DBG] pgmap v4277: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:54.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:54 smithi045 conmon[33655]: cluster 2021-07-31T03:02:52.882731+0000 mgr.x (mgr.24542) 3244 : cluster [DBG] 2021-07-31T03:02:54.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:54 smithi045 conmon[33655]: pgmap v4277: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:54.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:54 smithi045 conmon[39718]: cluster 2021-07-31T03:02:52.882731+0000 mgr.x (mgr. 2021-07-31T03:02:54.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:54 smithi045 conmon[39718]: 24542) 3244 : cluster [DBG] pgmap v4277: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:56.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:56 smithi154 conmon[34553]: cluster 2021-07-31T03:02:54.883193+0000 mgr.x (mgr.24542) 3245 : cluster [DBG] pgmap v4278: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:56.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:56 smithi045 conmon[33655]: cluster 2021-07-31T03:02:54.883193+0000 mgr.x (mgr.24542) 2021-07-31T03:02:56.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:56 smithi045 conmon[33655]: 3245 : cluster [DBG] pgmap v4278: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:56.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:56 smithi045 conmon[39718]: cluster 2021-07-31T03:02:54.883193+0000 mgr.x ( 2021-07-31T03:02:56.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:56 smithi045 conmon[39718]: mgr.24542) 3245 : cluster [DBG] pgmap v4278: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:58.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:02:58 smithi154 conmon[34553]: cluster 2021-07-31T03:02:56.883601+0000 mgr.x (mgr.24542) 3246 : cluster [DBG] pgmap v4279: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:58.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:58 smithi045 conmon[39718]: cluster 2021-07-31T03:02:56.883601+0000 mgr.x ( 2021-07-31T03:02:58.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:02:58 smithi045 conmon[39718]: mgr.24542) 3246 : cluster [DBG] pgmap v4279: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:02:58.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:58 smithi045 conmon[33655]: cluster 2021-07-31T03:02:56.883601+0000 mgr.x (mgr.24542) 2021-07-31T03:02:58.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:02:58 smithi045 conmon[33655]: 3246 : cluster [DBG] pgmap v4279: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:00.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:00 smithi154 conmon[34553]: cluster 2021-07-31T03:02:58.884109+0000 mgr.x (mgr.24542) 3247 : cluster [DBG] pgmap v4280: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:00.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:00 smithi045 conmon[39718]: cluster 2021-07-31T03:02:58.884109+0000 mgr.x (mgr. 2021-07-31T03:03:00.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:00 smithi045 conmon[39718]: 24542) 3247 : cluster [DBG] pgmap v4280: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:00.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:00 smithi045 conmon[33655]: cluster 2021-07-31T03:02:58.884109+0000 mgr.x (mgr. 2021-07-31T03:03:00.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:00 smithi045 conmon[33655]: 24542) 3247 : cluster [DBG] pgmap v4280: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:02.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:02 smithi154 conmon[34553]: cluster 2021-07-31T03:03:00.884578+0000 mgr.x (mgr.24542) 3248 : cluster [DBG] pgmap v4281: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:02.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:02 smithi045 conmon[33655]: cluster 2021-07-31T03:03:00.884578+0000 mgr.x (mgr.24542) 3248 : cluster 2021-07-31T03:03:02.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:02 smithi045 conmon[33655]: [DBG] pgmap v4281: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:02.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:02 smithi045 conmon[39718]: cluster 2021-07-31T03:03:00.884578+0000 mgr.x (mgr.24542) 3248 : cluster [DBG] pgmap v4281: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:04.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:03 smithi154 conmon[34553]: cluster 2021-07-31T03:03:02.885069+0000 mgr.x (mgr.24542) 3249 : cluster [DBG] pgmap v4282: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:04.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:03 smithi045 conmon[33655]: cluster 2021-07-31T03:03:02.885069+0000 mgr.x (mgr.24542) 3249 : cluster [DBG] pgmap v4282: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:04.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:03 smithi045 conmon[39718]: cluster 2021-07-31T03:03:02.885069+0000 mgr.x (mgr.24542) 3249 : cluster [DBG] pgmap v4282: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:06.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:05 smithi154 conmon[34553]: cluster 2021-07-31T03:03:04.885605+0000 mgr.x (mgr.24542) 3250 : cluster [DBG] pgmap v4283: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:06.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:05 smithi045 conmon[33655]: cluster 2021-07-31T03:03:04.885605+0000 mgr.x (mgr.24542) 3250 : 2021-07-31T03:03:06.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:05 smithi045 conmon[33655]: cluster [DBG] pgmap v4283: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:06.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:05 smithi045 conmon[39718]: cluster 2021-07-31T03:03:04.885605+0000 mgr.x 2021-07-31T03:03:06.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:05 smithi045 conmon[39718]: (mgr.24542) 3250 : cluster [DBG] pgmap v4283: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:08.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:07 smithi154 conmon[34553]: cluster 2021-07-31T03:03:06.886122+0000 mgr.x (mgr.24542) 3251 : cluster [DBG] pgmap v4284: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:08.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:07 smithi045 conmon[33655]: cluster 2021-07-31T03:03:06.886122+0000 mgr.x (mgr.24542) 3251 2021-07-31T03:03:08.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:07 smithi045 conmon[33655]: : cluster [DBG] pgmap v4284: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:08.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:07 smithi045 conmon[39718]: cluster 2021-07-31T03:03:06.886122+0000 mgr.x (mgr 2021-07-31T03:03:08.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:07 smithi045 conmon[39718]: .24542) 3251 : cluster [DBG] pgmap v4284: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:10.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:09 smithi154 conmon[34553]: cluster 2021-07-31T03:03:08.886523+0000 mgr.x (mgr.24542) 3252 : cluster [DBG] pgmap v4285: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:10.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:09 smithi154 conmon[34553]: 2021-07-31T03:03:10.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:09 smithi045 conmon[39718]: cluster 2021-07-31T03:03:08.886523+0000 mgr.x (mgr.24542) 3252 : cluster [DBG] pgmap v4285: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:10.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:09 smithi045 conmon[33655]: cluster 2021-07-31T03:03:08.886523+0000 mgr.x (mgr.24542) 3252 2021-07-31T03:03:10.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:09 smithi045 conmon[33655]: : cluster [DBG] pgmap v4285: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:12.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:11 smithi154 conmon[34553]: cluster 2021-07-31T03:03:10.886966+0000 mgr.x (mgr.24542) 3253 : cluster [DBG] pgmap v4286: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:12.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:11 smithi045 conmon[33655]: cluster 2021-07-31T03:03:10.886966+0000 mgr.x (mgr.24542 2021-07-31T03:03:12.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:11 smithi045 conmon[33655]: ) 3253 : cluster [DBG] pgmap v4286: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:12.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:11 smithi045 conmon[39718]: cluster 2021-07-31T03:03:10.886966+0000 mgr.x (mgr.24542) 3253 : cluster [DBG] 2021-07-31T03:03:12.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:11 smithi045 conmon[39718]: pgmap v4286: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:14.032 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:03:14.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:13 smithi154 conmon[34553]: cluster 2021-07-31T03:03:12.887396+0000 mgr.x (mgr.24542) 3254 : cluster [DBG] pgmap v4287: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:14.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:13 smithi154 conmon[34553]: 2021-07-31T03:03:14.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:13 smithi045 conmon[33655]: cluster 2021-07-31T03:03:12.887396+0000 mgr.x (mgr.24542 2021-07-31T03:03:14.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:13 smithi045 conmon[33655]: ) 3254 : cluster [DBG] pgmap v4287: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:14.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:13 smithi045 conmon[39718]: cluster 2021-07-31T03:03:12.887396+0000 mgr.x (mgr.24542 2021-07-31T03:03:14.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:13 smithi045 conmon[39718]: ) 3254 : cluster [DBG] pgmap v4287: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:14.306 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:03:14.307 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 78m ago 78m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:03:14.307 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 75m ago 78m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:03:14.307 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 75m ago 82m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:03:14.307 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 78m ago 84m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:03:14.307 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 78m ago 84m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:03:14.308 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 75m ago 82m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:03:14.308 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 78m ago 82m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:03:14.308 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 78m ago 79m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:03:14.308 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 75m ago 79m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:03:14.308 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 78m ago 81m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:03:14.309 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 78m ago 81m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:03:14.309 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 78m ago 81m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:03:14.309 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 78m ago 80m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:03:14.309 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 75m ago 80m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:03:14.309 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 75m ago 80m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:03:14.310 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 75m ago 79m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:03:14.310 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 75m ago 79m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:03:14.310 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 75m ago 79m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:03:14.642 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:03:14.642 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:03:14.642 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:03:14.642 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:03:14.643 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:03:14.643 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:03:14.643 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:03:14.643 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:03:14.644 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:03:14.644 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:03:14.644 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:03:14.644 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:03:14.644 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:03:14.645 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:03:14.645 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:03:14.645 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:03:14.645 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:03:15.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:14 smithi154 conmon[34553]: audit 2021-07-31T03:03:14.020317+0000 mgr.x (mgr.24542) 3255 : audit [DBG] from='client.28636 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:03:15.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:14 smithi154 conmon[34553]: audit 2021-07-31T03:03:14.299013+0000 mgr.x (mgr.24542) 3256 : audit [DBG] from='client.18702 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:03:15.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:14 smithi154 conmon[34553]: audit 2021-07-31T03:03:14.641726+0000 mon.a (mon.0) 3244 : audit [DBG] from='client.? 172.21.15.45:0/721941810' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:03:15.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:14 smithi045 conmon[39718]: audit 2021-07-31T03:03:14.020317 2021-07-31T03:03:15.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:14 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 3255 : audit [DBG] from='client.28636 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:03:15.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:14 smithi045 conmon[39718]: audit 2021-07-31T03:03:15.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:14 smithi045 conmon[39718]: 2021-07-31T03:03:14.299013+0000 mgr.x ( 2021-07-31T03:03:15.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:14 smithi045 conmon[39718]: mgr.24542) 3256 : audit [DBG] from='client.18702 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:03:15.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:14 smithi045 conmon[39718]: 2021-07-31T03:03:15.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:14 smithi045 conmon[39718]: audit 2021-07-31T03:03:15.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:14 smithi045 conmon[39718]: 2021-07-31T03:03:14.641726+0000 2021-07-31T03:03:15.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:14 smithi045 conmon[39718]: mon.a (mon.0) 3244 : audit [DBG] 2021-07-31T03:03:15.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:14 smithi045 conmon[39718]: from='client.? 172.21.15.45:0/721941810' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:03:15.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:14 smithi045 conmon[33655]: audit 2021-07-31T03:03:14.020317+0000 mgr.x (mgr.24542) 3255 2021-07-31T03:03:15.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:14 smithi045 conmon[33655]: : audit [DBG] from='client.28636 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:03:15.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:14 smithi045 conmon[33655]: audit 2021-07-31T03:03:14.299013+0000 mgr.x (mgr.24542) 3256 : audit 2021-07-31T03:03:15.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:14 smithi045 conmon[33655]: [DBG] from='client.18702 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:03:15.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:14 smithi045 conmon[33655]: audit 2021- 2021-07-31T03:03:15.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:14 smithi045 conmon[33655]: 07-31T03:03:14.641726+0000 2021-07-31T03:03:15.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:14 smithi045 conmon[33655]: mon.a (mon.0) 3244 : audit [DBG] from='client.? 172.21.15.45:0/721941810' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:03:16.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:15 smithi154 conmon[34553]: cluster 2021-07-31T03:03:14.887836+0000 mgr.x (mgr.24542) 3257 : cluster [DBG] pgmap v4288: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:16.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:15 smithi045 conmon[39718]: cluster 2021-07-31T03:03:14.887836+0000 mgr.x (mgr.24542) 3257 : cluster [DBG] pgmap v4288: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:16.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:15 smithi045 conmon[33655]: cluster 2021-07-31T03:03:14.887836+0000 2021-07-31T03:03:16.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:15 smithi045 conmon[33655]: mgr.x (mgr.24542) 3257 : cluster [DBG] pgmap v4288: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:18.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:17 smithi154 conmon[34553]: cluster 2021-07-31T03:03:16.888307+0000 mgr.x (mgr.24542) 3258 : cluster 2021-07-31T03:03:18.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:17 smithi154 conmon[34553]: [DBG] pgmap v4289: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:18.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:17 smithi045 conmon[33655]: cluster 2021-07-31T03:03:16.888307+0000 mgr.x (mgr.24542) 3258 2021-07-31T03:03:18.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:17 smithi045 conmon[33655]: : cluster [DBG] pgmap v4289: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:18.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:17 smithi045 conmon[39718]: cluster 2021-07-31T03:03:16.888307+0000 mgr.x (mgr. 2021-07-31T03:03:18.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:17 smithi045 conmon[39718]: 24542) 3258 : cluster [DBG] pgmap v4289: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:20.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:19 smithi154 conmon[34553]: cluster 2021-07-31T03:03:18.888713+0000 mgr.x (mgr.24542) 3259 : cluster [DBG] pgmap v4290: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:20.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:19 smithi045 conmon[39718]: cluster 2021-07-31T03:03:18.888713 2021-07-31T03:03:20.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:19 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 3259 : cluster [DBG] pgmap v4290: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:20.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:19 smithi045 conmon[33655]: cluster 2021-07-31T03:03:18.888713+0000 mgr.x (mgr. 2021-07-31T03:03:20.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:19 smithi045 conmon[33655]: 24542) 3259 : cluster [DBG] pgmap v4290: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:22.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:21 smithi154 conmon[34553]: cluster 2021-07-31T03:03:20.889185+0000 mgr.x (mgr.24542) 3260 : cluster [DBG] pgmap v4291: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:22.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:21 smithi045 conmon[33655]: cluster 2021-07-31T03:03:20.889185+0000 mgr.x (mgr.24542) 3260 : cluster 2021-07-31T03:03:22.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:21 smithi045 conmon[33655]: [DBG] pgmap v4291: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:22.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:21 smithi045 conmon[39718]: cluster 2021-07-31T03:03:20.889185+0000 mgr.x ( 2021-07-31T03:03:22.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:21 smithi045 conmon[39718]: mgr.24542) 3260 : cluster [DBG] pgmap v4291: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:24.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:23 smithi154 conmon[34553]: cluster 2021-07-31T03:03:22.889692+0000 mgr.x (mgr.24542) 3261 : cluster [DBG] pgmap v4292: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:24.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:23 smithi045 conmon[33655]: cluster 2021-07-31T03:03:22.889692+0000 mgr.x (mgr.24542) 3261 : 2021-07-31T03:03:24.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:23 smithi045 conmon[33655]: cluster [DBG] pgmap v4292: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:24.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:23 smithi045 conmon[39718]: cluster 2021-07-31T03:03:22.889692+0000 mgr.x (mgr.24542) 3261 : cluster [DBG] pgmap v4292: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:26.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:25 smithi154 conmon[34553]: cluster 2021-07-31T03:03:24.890202+0000 mgr.x (mgr.24542) 3262 : cluster [DBG] pgmap v4293: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:26.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:25 smithi154 conmon[34553]: 2021-07-31T03:03:26.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:25 smithi045 conmon[39718]: cluster 2021-07-31T03:03:24.890202+0000 mgr.x (mgr 2021-07-31T03:03:26.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:25 smithi045 conmon[39718]: .24542) 3262 : cluster [DBG] pgmap v4293: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:26.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:25 smithi045 conmon[33655]: cluster 2021-07-31T03:03:24.890202+0000 mgr.x (mgr.24542) 3262 : cluster [DBG] 2021-07-31T03:03:26.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:25 smithi045 conmon[33655]: pgmap v4293: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:28.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:27 smithi154 conmon[34553]: cluster 2021-07-31T03:03:26.890595+0000 mgr.x (mgr.24542) 3263 : cluster [DBG] 2021-07-31T03:03:28.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:27 smithi154 conmon[34553]: pgmap v4294: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:28.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:27 smithi045 conmon[33655]: cluster 2021-07-31T03:03:26.890595+0000 mgr.x (mgr.24542) 3263 : cluster [DBG] 2021-07-31T03:03:28.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:27 smithi045 conmon[33655]: pgmap v4294: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:28.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:27 smithi045 conmon[39718]: cluster 2021-07-31T03:03:26.890595 2021-07-31T03:03:28.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:27 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 3263 : cluster [DBG] pgmap v4294: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:29.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:28 smithi154 conmon[34553]: audit 2021-07-31T03:03:28.237613+0000 mon.b (mon.2) 455 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:03:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:28 smithi154 conmon[34553]: audit 2021-07-31T03:03:28.238031+0000 mon.b (mon.2) 456 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:03:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:28 smithi154 conmon[34553]: audit 2021-07-31T03:03:28.238323+0000 mon.a (mon.0) 3245 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:03:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:28 smithi154 conmon[34553]: 2021-07-31T03:03:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:28 smithi154 conmon[34553]: audit 2021-07-31 2021-07-31T03:03:29.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:28 smithi154 conmon[34553]: T03:03:28.238931+0000 mon.a (mon.0) 2021-07-31T03:03:29.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:28 smithi154 conmon[34553]: 3246 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:03:29.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:28 smithi045 conmon[39718]: audit 2021-07-31T03:03:28.237613+0000 mon.b (mon.2) 455 : audit [INF] 2021-07-31T03:03:29.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:28 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:03:29.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:28 smithi045 conmon[39718]: audit 2021-07-31T03:03:28.238031+0000 mon.b (mon.2) 456 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:03:29.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:28 smithi045 conmon[39718]: audit 2021-07-31T03:03:28.238323+0000 mon.a (mon.0) 3245 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:03:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:28 smithi045 conmon[39718]: audit 2021-07-31T03:03: 2021-07-31T03:03:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:28 smithi045 conmon[39718]: 28.238931+0000 mon.a (mon.0) 3246 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:03:29.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:28 smithi045 conmon[33655]: audit 2021-07-31T03:03:28.237613 2021-07-31T03:03:29.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:28 smithi045 conmon[33655]: +0000 mon.b (mon.2) 455 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:03:29.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:28 smithi045 conmon[33655]: audit 2021-07-31T03:03:28.238031+0000 2021-07-31T03:03:29.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:28 smithi045 conmon[33655]: mon.b (mon.2) 456 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:03:29.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:28 smithi045 conmon[33655]: audit 2021-07-31T03:03:29.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:28 smithi045 conmon[33655]: 2021-07-31T03:03:28. 2021-07-31T03:03:29.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:28 smithi045 conmon[33655]: 238323+0000 mon.a (mon.0) 3245 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:03:29.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:28 smithi045 conmon[33655]: audit 2021-07-31T03:03:28 2021-07-31T03:03:29.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:28 smithi045 conmon[33655]: .238931+0000 mon.a (mon.0) 3246 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:03:30.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:29 smithi154 conmon[34553]: cluster 2021-07-31T03:03:28.891003+0000 mgr.x (mgr.24542) 3264 : cluster [DBG] pgmap v4295: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:30.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:29 smithi045 conmon[39718]: cluster 2021-07-31T03:03:28.891003+0000 mgr.x (mgr.24542) 3264 : cluster [DBG] 2021-07-31T03:03:30.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:29 smithi045 conmon[39718]: pgmap v4295: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:30.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:29 smithi045 conmon[33655]: cluster 2021-07-31T03:03:28.891003+0000 mgr.x (mgr 2021-07-31T03:03:30.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:29 smithi045 conmon[33655]: .24542) 3264 : cluster [DBG] pgmap v4295: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:31.503 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: debug 2021-07-31T03:03:31.047+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:03:31.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:03:31.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:03:31.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: yield 2021-07-31T03:03:31.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:03:31.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:03:31.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:03:31.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:03:31.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:03:31.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:03:31.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:03:31.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:03:31.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:03:31.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: task.result() 2021-07-31T03:03:31.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:03:31.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: debug 2021-07-31T03:03:31.059+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:03:31.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:03:31.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:03:31.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: yield 2021-07-31T03:03:31.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:03:31.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:03:31.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:03:31.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:03:31.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:03:31.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:03:31.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:03:31.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:03:31.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:03:31.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: task.result() 2021-07-31T03:03:31.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:03:31.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: debug 2021-07-31T03:03:31.084+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:03:31.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:03:31.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:03:31.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: yield 2021-07-31T03:03:31.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:03:31.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:03:31.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:03:31.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:03:31.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:03:31.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:03:31.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:03:31.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:03:31.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:03:31.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: task.result() 2021-07-31T03:03:31.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:03:31.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: debug 2021-07-31T03:03:31.085+0000 7fcb93160700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:03:31.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:03:31.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:03:31.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: yield 2021-07-31T03:03:31.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:03:31.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:03:31.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:03:31.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:03:31.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:03:31.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:03:31.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:03:31.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:03:31.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:03:31.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: task.result() 2021-07-31T03:03:31.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:03:31.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: 2021-07-31T03:03:31.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:03:31.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: 2021-07-31T03:03:31.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:03:31.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:03:31.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:03:31.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:03:31.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:03:31.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:03:31.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:03:31.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:03:31.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:03:31.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:03:31.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:03:31.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:03:31.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:03:31.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:03:31.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:03:31.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:03:31.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:03:31.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:03:31.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: raise 2021-07-31T03:03:31.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:03:31.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:03:31.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:03:31.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:03:31.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:03:31.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: debug 2021-07-31T03:03:31.094+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:03:31.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:03:31.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:03:31.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: yield 2021-07-31T03:03:31.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:03:31.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:03:31.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:03:31.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:03:31.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:03:31.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:03:31.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:03:31.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:03:31.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:03:31.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: task.result() 2021-07-31T03:03:31.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:03:31.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: debug 2021-07-31T03:03:31.094+0000 7fcb9295f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:03:31.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:03:31.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:03:31.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: yield 2021-07-31T03:03:31.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:03:31.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:03:31.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:03:31.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:03:31.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:03:31.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:03:31.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:03:31.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:03:31.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:03:31.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: task.result() 2021-07-31T03:03:31.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:03:31.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: 2021-07-31T03:03:31.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:03:31.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: 2021-07-31T03:03:31.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:03:31.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:03:31.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:03:31.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:03:31.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:03:31.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:03:31.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:03:31.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:03:31.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:03:31.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:03:31.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:03:31.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:03:31.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:03:31.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:03:31.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:03:31.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:03:31.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:03:31.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:03:31.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: raise 2021-07-31T03:03:31.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:03:31.535 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:03:31.535 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:03:31.535 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:03:31.535 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:03:32.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: cluster 2021-07-31T03:03:30.891445+0000 mgr.x (mgr.24542) 3265 2021-07-31T03:03:32.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: : cluster [DBG] pgmap v4296: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:32.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: audit 2021-07-31T03:03:31.001469+0000 mon.b (mon.2) 457 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:03:32.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: audit 2021-07-31T03: 2021-07-31T03:03:32.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: 03:31.002995+0000 mon.b (mon.2) 458 : audit 2021-07-31T03:03:32.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:03:32.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: audit 2021-07-31T03 2021-07-31T03:03:32.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: :03:31.003821+0000 mon.b (mon.2) 459 2021-07-31T03:03:32.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:03:32.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: cephadm 2021- 2021-07-31T03:03:32.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: 07-31T03:03:31.004624+0000 mgr.x (mgr 2021-07-31T03:03:32.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: .24542) 3266 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:03:32.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: cephadm 2021-07-31T03:03:32.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: 2021-07-31T03:03:31.048656+0000 mgr.x (mgr.24542) 2021-07-31T03:03:32.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: 3267 : cephadm [ERR] 2021-07-31T03:03:32.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:03:32.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:03:32.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: yield 2021-07-31T03:03:32.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:03:32.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:03:32.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:03:32.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:03:32.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:03:32.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:03:32.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:03:32.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:03:32.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:03:32.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: task.result() 2021-07-31T03:03:32.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:03:32.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: cephadm 2021-07-31T03:03:31.049110+0000 mgr.x (mgr.24542) 3268 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:03:32.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: cephadm 2021-07-31T03:03:31. 2021-07-31T03:03:32.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: 060179+0000 mgr.x (mgr.24542) 3269 : cephadm [ERR] 2021-07-31T03:03:32.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:03:32.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:03:32.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: yield 2021-07-31T03:03:32.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:03:32.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:03:32.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:03:32.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:03:32.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:03:32.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:03:32.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:03:32.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:03:32.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:03:32.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: task.result() 2021-07-31T03:03:32.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:03:32.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: cephadm 2021-07-31T03:03:31.061200+0000 mgr.x (mgr.24542) 3270 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:03:32.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: cephadm 2021 2021-07-31T03:03:32.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: -07-31T03:03:31.085942+0000 mgr.x (mgr.24542) 3271 : cephadm 2021-07-31T03:03:32.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: [ERR] 2021-07-31T03:03:32.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:03:32.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:03:32.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: yield 2021-07-31T03:03:32.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:03:32.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:03:32.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:03:32.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:03:32.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:03:32.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:03:32.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:03:32.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:03:32.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:03:32.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: task.result() 2021-07-31T03:03:32.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:03:32.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: cephadm 2021-07-31T03:03:31.086414+0000 mgr.x (mgr.24542) 3272 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:03:32.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:03:32.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:03:32.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: yield 2021-07-31T03:03:32.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:03:32.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:03:32.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:03:32.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:03:32.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:03:32.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:03:32.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:03:32.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:03:32.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:03:32.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: task.result() 2021-07-31T03:03:32.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:03:32.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: 2021-07-31T03:03:32.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:03:32.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: 2021-07-31T03:03:32.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:03:32.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:03:32.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:03:32.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:03:32.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:03:32.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:03:32.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:03:32.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:03:32.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:03:32.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:03:32.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:03:32.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:03:32.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:03:32.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:03:32.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:03:32.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:03:32.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:03:32.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:03:32.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: raise 2021-07-31T03:03:32.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:03:32.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:03:32.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:03:32.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:03:32.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:03:32.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: cephadm 2021-07-31T03:03:31.095090+0000 mgr.x ( 2021-07-31T03:03:32.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: mgr.24542) 3273 : cephadm [ERR] 2021-07-31T03:03:32.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:03:32.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:03:32.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: yield 2021-07-31T03:03:32.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:03:32.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:03:32.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:03:32.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:03:32.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:03:32.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:03:32.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:03:32.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:03:32.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:03:32.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: task.result() 2021-07-31T03:03:32.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:03:32.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: cephadm 2021-07-31T03:03:31.095494+0000 mgr.x (mgr.24542) 3274 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:03:32.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:03:32.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:03:32.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: yield 2021-07-31T03:03:32.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:03:32.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:03:32.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:03:32.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:03:32.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:03:32.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:03:32.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:03:32.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:03:32.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:03:32.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: task.result() 2021-07-31T03:03:32.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:03:32.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: 2021-07-31T03:03:32.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:03:32.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: 2021-07-31T03:03:32.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:03:32.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:03:32.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:03:32.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:03:32.285 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:03:32.285 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:03:32.285 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:03:32.285 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:03:32.286 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:03:32.286 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:03:32.286 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:03:32.286 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:03:32.286 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:03:32.287 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:03:32.287 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:03:32.287 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:03:32.287 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:03:32.287 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:03:32.288 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: raise 2021-07-31T03:03:32.288 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:03:32.288 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:03:32.288 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:03:32.288 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:03:32.289 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:31 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:03:32.289 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: cluster 2021-07-31T03:03:30.891445+0000 mgr.x 2021-07-31T03:03:32.290 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: (mgr.24542) 3265 : cluster [DBG] pgmap v4296: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:32.290 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T03:03:32.290 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: 03:03:31.001469+0000 mon.b (mon.2) 457 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:03:32.290 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: audit 2021-07-31T03:03:31.002995+0000 mon.b (mon. 2021-07-31T03:03:32.291 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: 2) 458 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:03:32.291 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: audit 2021-07-31T03:03:31.003821+0000 mon.b ( 2021-07-31T03:03:32.291 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: mon.2) 459 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:03:32.291 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: cephadm 2021-07-31T03:03:31.004624+0000 mgr.x (mgr.24542 2021-07-31T03:03:32.291 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: ) 3266 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:03:32.292 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: cephadm 2021-07-31T03:03:31.048656+0000 mgr.x 2021-07-31T03:03:32.292 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: (mgr.24542) 3267 : cephadm [ERR] 2021-07-31T03:03:32.292 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:03:32.292 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:03:32.293 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: yield 2021-07-31T03:03:32.293 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:03:32.293 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:03:32.293 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:03:32.293 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:03:32.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:03:32.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:03:32.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:03:32.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:03:32.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:03:32.295 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: task.result() 2021-07-31T03:03:32.295 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:03:32.295 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: cephadm 2021-07-31T03:03:31.049110+0000 mgr.x (mgr.24542) 3268 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:03:32.295 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: cephadm 2021-07-31T03:03:31.060179+0000 mgr.x (mgr.24542) 3269 : cephadm [ERR] 2021-07-31T03:03:32.295 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:03:32.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:03:32.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: yield 2021-07-31T03:03:32.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:03:32.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:03:32.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:03:32.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:03:32.297 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: cluster 2021-07-31T03:03:30.891445+0000 mgr.x 2021-07-31T03:03:32.297 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: (mgr.24542) 3265 : cluster [DBG] pgmap v4296: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:32.298 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: audit 2021-07-31T03:03:32.298 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: 2021-07-31T03:03: 2021-07-31T03:03:32.298 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: 31.001469+0000 mon.b (mon 2021-07-31T03:03:32.298 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: .2) 457 : audit [DBG] 2021-07-31T03:03:32.298 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:03:32.299 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: audit 2021- 2021-07-31T03:03:32.299 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: 07-31T03:03:31. 2021-07-31T03:03:32.299 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: 002995+0000 mon.b (mon.2) 2021-07-31T03:03:32.299 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: 458 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:03:32.299 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: audit 2021-07 2021-07-31T03:03:32.300 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: -31T03:03:31.003821+0000 mon.b (mon.2) 459 : audit 2021-07-31T03:03:32.300 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:03:32.300 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: cephadm 2021-07-31T03:03:31.004624+0000 mgr.x (mgr.24542) 3266 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:03:32.300 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: cephadm 2021-07-31T03:03:31.048656+0000 mgr.x (mgr.24542) 2021-07-31T03:03:32.301 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: 3267 : cephadm [ERR] 2021-07-31T03:03:32.301 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:03:32.301 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:03:32.301 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: yield 2021-07-31T03:03:32.301 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:03:32.302 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:03:32.302 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:03:32.302 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:03:32.302 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:03:32.302 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:03:32.303 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:03:32.303 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:03:32.303 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:03:32.303 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: task.result() 2021-07-31T03:03:32.303 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:03:32.304 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: cephadm 2021-07- 2021-07-31T03:03:32.304 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: 31T03:03:31.049110+0000 mgr.x (mgr.24542) 3268 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:03:32.304 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: cephadm 2021-07-31T03:03:31.060179+0000 mgr.x (mgr.24542) 3269 : cephadm [ERR] 2021-07-31T03:03:32.304 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:03:32.304 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:03:32.305 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: yield 2021-07-31T03:03:32.305 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:03:32.305 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:03:32.306 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:03:32.306 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:03:32.306 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:03:32.306 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:03:32.306 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:03:32.307 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: task.result() 2021-07-31T03:03:32.307 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:03:32.307 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: cephadm 2021-07-31T03:03:31.061200+0000 mgr.x (mgr.24542) 3270 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:03:32.307 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: cephadm 2021-07-31T03:03:31.085942+0000 mgr.x (mgr.24542) 3271 : cephadm [ERR] 2021-07-31T03:03:32.308 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:03:32.308 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:03:32.308 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: yield 2021-07-31T03:03:32.308 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:03:32.308 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:03:32.309 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:03:32.309 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:03:32.309 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:03:32.309 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:03:32.309 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:03:32.310 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:03:32.310 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:03:32.310 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: task.result() 2021-07-31T03:03:32.310 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:03:32.311 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: cephadm 2021-07-31T03:03 2021-07-31T03:03:32.311 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: :31.086414+0000 mgr.x (mgr.24542) 3272 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:03:32.311 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:03:32.311 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:03:32.311 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: yield 2021-07-31T03:03:32.312 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:03:32.312 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:03:32.312 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:03:32.312 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:03:32.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:03:32.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:03:32.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:03:32.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:03:32.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:03:32.314 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: task.result() 2021-07-31T03:03:32.314 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:03:32.314 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: 2021-07-31T03:03:32.314 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:03:32.315 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: 2021-07-31T03:03:32.315 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:03:32.315 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:03:32.316 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:03:32.316 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:03:32.316 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:03:32.316 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:03:32.317 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:03:32.317 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: task.result() 2021-07-31T03:03:32.317 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:03:32.317 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: cephadm 2021-07-31T03:03:31.061200+0000 mgr.x (mgr.24542) 3270 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:03:32.317 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: cephadm 2021 2021-07-31T03:03:32.318 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: -07-31T03:03:31.085942+0000 mgr.x (mgr.24542) 3271 : cephadm [ERR] 2021-07-31T03:03:32.318 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:03:32.318 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:03:32.318 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: yield 2021-07-31T03:03:32.319 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:03:32.319 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:03:32.319 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:03:32.319 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:03:32.319 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:03:32.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:03:32.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:03:32.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:03:32.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:03:32.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: task.result() 2021-07-31T03:03:32.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:03:32.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: cephadm 2021-07-31T03:03:31.086414+0000 mgr.x (mgr.24542) 3272 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:03:32.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:03:32.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:03:32.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: yield 2021-07-31T03:03:32.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:03:32.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:03:32.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:03:32.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:03:32.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:03:32.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:03:32.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:03:32.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:03:32.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:03:32.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: task.result() 2021-07-31T03:03:32.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:03:32.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: 2021-07-31T03:03:32.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:03:32.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: 2021-07-31T03:03:32.325 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:03:32.325 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:03:32.325 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:03:32.325 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:03:32.326 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:03:32.326 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:03:32.326 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:03:32.326 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:03:32.326 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:03:32.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:03:32.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:03:32.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:03:32.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:03:32.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:03:32.328 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:03:32.328 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:03:32.328 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:03:32.328 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:03:32.328 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: raise 2021-07-31T03:03:32.329 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:03:32.329 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:03:32.329 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:03:32.329 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:03:32.330 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:03:32.330 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: cephadm 2021-07-31T03:03:31.095090+0000 mgr.x (mgr.24542) 3273 : cephadm [ERR] 2021-07-31T03:03:32.330 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:03:32.330 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:03:32.330 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: yield 2021-07-31T03:03:32.331 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:03:32.331 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:03:32.331 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:03:32.331 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:03:32.331 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:03:32.332 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:03:32.332 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:03:32.332 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:03:32.332 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:03:32.333 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:03:32.333 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:03:32.333 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:03:32.333 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:03:32.334 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:03:32.334 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:03:32.334 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:03:32.334 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:03:32.334 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:03:32.335 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:03:32.335 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:03:32.335 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:03:32.335 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:03:32.335 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:03:32.336 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:03:32.336 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:03:32.336 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:03:32.336 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: raise 2021-07-31T03:03:32.336 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:03:32.337 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:03:32.337 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:03:32.337 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:03:32.337 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:03:32.338 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: cephadm 2021-07-31T03:03:31.095090+0000 mgr.x (mgr.24542) 3273 : cephadm [ERR] 2021-07-31T03:03:32.338 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:03:32.338 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:03:32.338 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: yield 2021-07-31T03:03:32.338 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:03:32.339 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:03:32.339 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:03:32.339 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:03:32.339 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:03:32.339 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:03:32.340 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:03:32.340 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:03:32.340 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:03:32.340 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: task.result() 2021-07-31T03:03:32.340 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:03:32.341 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:03:32.341 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: task.result() 2021-07-31T03:03:32.341 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:03:32.342 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: cephadm 2021-07-31T03:03:31.095494+0000 mgr.x (mgr.24542) 3274 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:03:32.342 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:03:32.342 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:03:32.342 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: yield 2021-07-31T03:03:32.343 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:03:32.343 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:03:32.343 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:03:32.343 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:03:32.343 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:03:32.344 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:03:32.344 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:03:32.344 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:03:32.344 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:03:32.344 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: task.result() 2021-07-31T03:03:32.345 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:03:32.345 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: 2021-07-31T03:03:32.345 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:03:32.345 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: 2021-07-31T03:03:32.346 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:03:32.346 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:03:32.346 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:03:32.346 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:03:32.346 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:03:32.347 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:03:32.347 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:03:32.347 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:03:32.347 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:03:32.347 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:03:32.348 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:03:32.348 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:03:32.348 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:03:32.348 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:03:32.348 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:03:32.349 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:03:32.349 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:03:32.349 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:03:32.349 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: raise 2021-07-31T03:03:32.349 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:03:32.350 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:03:32.350 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:03:32.350 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:03:32.350 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:03:32.351 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: cephadm 2021-07-31T03:03:31.095494+0000 mgr.x (mgr.24542) 3274 : cephadm [ERR] 2021-07-31T03:03:32.351 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:03:32.351 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:03:32.351 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:03:32.352 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: yield 2021-07-31T03:03:32.352 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:03:32.352 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:03:32.352 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:03:32.352 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:03:32.353 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:03:32.353 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:03:32.353 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:03:32.353 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:03:32.354 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:03:32.354 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: task.result() 2021-07-31T03:03:32.354 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:03:32.354 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: 2021-07-31T03:03:32.354 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:03:32.355 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: 2021-07-31T03:03:32.355 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:03:32.355 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:03:32.355 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:03:32.356 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:03:32.356 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:03:32.356 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:03:32.356 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:03:32.356 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:03:32.357 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:03:32.357 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:03:32.357 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:03:32.357 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:03:32.357 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:03:32.357 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:03:32.358 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:03:32.358 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:03:32.358 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:03:32.358 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:03:32.358 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: raise 2021-07-31T03:03:32.359 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:03:32.359 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:03:32.359 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:03:32.359 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:03:32.360 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:31 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:03:34.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:33 smithi154 conmon[34553]: cluster 2021-07-31T03:03:32.891896+0000 mgr.x (mgr.24542) 3275 : cluster [DBG] pgmap v4297: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:34.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:33 smithi154 conmon[34553]: 2021-07-31T03:03:34.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:33 smithi045 conmon[33655]: cluster 2021-07-31T03:03:32.891896+0000 mgr.x (mgr.24542 2021-07-31T03:03:34.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:33 smithi045 conmon[33655]: ) 3275 : cluster [DBG] pgmap v4297: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:34.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:33 smithi045 conmon[39718]: cluster 2021-07-31T03:03:32.891896+0000 mgr.x (mgr.24542) 3275 : cluster [DBG] 2021-07-31T03:03:34.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:33 smithi045 conmon[39718]: pgmap v4297: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:36.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:35 smithi154 conmon[34553]: cluster 2021-07-31T03:03:34.892321+0000 mgr.x (mgr.24542) 3276 : cluster [DBG] pgmap v4298: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:36.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:35 smithi045 conmon[33655]: cluster 2021-07-31T03:03:34.892321+0000 mgr.x (mgr.24542 2021-07-31T03:03:36.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:35 smithi045 conmon[33655]: ) 3276 : cluster [DBG] pgmap v4298: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:36.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:35 smithi045 conmon[39718]: cluster 2021-07-31T03:03:34.892321+0000 mgr.x (mgr.24542 2021-07-31T03:03:36.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:35 smithi045 conmon[39718]: ) 3276 : cluster [DBG] pgmap v4298: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:38.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:37 smithi154 conmon[34553]: cluster 2021-07-31T03:03:36.892830+0000 mgr.x (mgr.24542) 3277 : cluster [DBG] pgmap v4299: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:38.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:37 smithi045 conmon[33655]: cluster 2021-07-31T03:03:36.892830+0000 mgr.x (mgr.24542) 3277 : cluster [DBG] 2021-07-31T03:03:38.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:37 smithi045 conmon[33655]: pgmap v4299: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:38.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:37 smithi045 conmon[39718]: cluster 2021-07-31T03:03:36.892830+0000 mgr.x (mgr.24542 2021-07-31T03:03:38.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:37 smithi045 conmon[39718]: ) 3277 : cluster [DBG] pgmap v4299: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:40.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:39 smithi154 conmon[34553]: cluster 2021-07-31T03:03:38.893156+0000 mgr.x (mgr.24542) 3278 : cluster [DBG] pgmap v4300: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:40.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:39 smithi045 conmon[39718]: cluster 2021-07-31T03:03:38.893156+0000 mgr.x (mgr 2021-07-31T03:03:40.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:39 smithi045 conmon[39718]: .24542) 3278 : cluster [DBG] pgmap v4300: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:40.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:39 smithi045 conmon[33655]: cluster 2021-07-31T03:03:38.893156+0000 mgr.x (mgr.24542) 3278 : 2021-07-31T03:03:40.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:39 smithi045 conmon[33655]: cluster [DBG] pgmap v4300: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:42.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:41 smithi154 conmon[34553]: cluster 2021-07-31T03:03:40.893512+0000 mgr.x (mgr.24542) 3279 : cluster [DBG] pgmap v4301: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:42.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:41 smithi045 conmon[33655]: cluster 2021-07-31T03:03:40.893512+0000 mgr.x (mgr. 2021-07-31T03:03:42.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:41 smithi045 conmon[33655]: 24542) 3279 : cluster [DBG] pgmap v4301: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:42.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:41 smithi045 conmon[39718]: cluster 2021-07-31T03:03:40.893512+0000 mgr.x (mgr.24542) 2021-07-31T03:03:42.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:41 smithi045 conmon[39718]: 3279 : cluster [DBG] pgmap v4301: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:44.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:43 smithi154 conmon[34553]: cluster 2021-07-31T03:03:42.894035+0000 mgr.x (mgr.24542) 3280 : cluster [DBG] pgmap v4302: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:44.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:43 smithi045 conmon[33655]: cluster 2021-07-31T03:03:42.894035+0000 mgr.x (mgr. 2021-07-31T03:03:44.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:43 smithi045 conmon[33655]: 24542) 3280 : cluster [DBG] pgmap v4302: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:44.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:43 smithi045 conmon[39718]: cluster 2021-07-31T03:03:42.894035+0000 mgr.x (mgr.24542) 3280 : 2021-07-31T03:03:44.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:43 smithi045 conmon[39718]: cluster [DBG] pgmap v4302: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:44.957 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:03:45.237 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:03:45.237 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 78m ago 79m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:03:45.237 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 76m ago 79m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:03:45.238 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 76m ago 82m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:03:45.238 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 78m ago 84m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:03:45.238 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 78m ago 84m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:03:45.238 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 76m ago 82m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:03:45.239 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 78m ago 83m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:03:45.239 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 78m ago 79m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:03:45.239 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 76m ago 79m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:03:45.239 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 78m ago 82m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:03:45.239 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 78m ago 82m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:03:45.240 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 78m ago 81m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:03:45.240 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 78m ago 81m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:03:45.240 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 76m ago 81m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:03:45.240 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 76m ago 80m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:03:45.240 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 76m ago 80m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:03:45.241 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 76m ago 80m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:03:45.241 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 76m ago 79m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:03:45.572 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:03:45.572 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:03:45.573 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:03:45.573 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:03:45.573 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:03:45.573 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:03:45.574 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:03:45.574 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:03:45.574 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:03:45.574 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:03:45.574 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:03:45.575 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:03:45.575 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:03:45.575 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:03:45.575 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:03:45.575 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:03:45.576 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:03:46.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:45 smithi154 conmon[34553]: cluster 2021-07-31T03:03:44.894466+0000 mgr.x (mgr.24542) 3281 : cluster [DBG] pgmap v4303: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:46.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:45 smithi154 conmon[34553]: audit 2021-07-31T03:03:44.945220+0000 2021-07-31T03:03:46.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:45 smithi154 conmon[34553]: mgr.x (mgr.24542) 3282 : audit [DBG] from='client.18714 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:03:46.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:45 smithi154 conmon[34553]: audit 2021-07-31T03:03:45.229089+0000 mgr.x (mgr.24542) 3283 : audit [DBG] from='client.28660 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:03:46.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:45 smithi154 conmon[34553]: audit 2021-07-31T03:03:45.572058+0000 mon.c (mon.1) 171 : audit [DBG] 2021-07-31T03:03:46.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:45 smithi154 conmon[34553]: from='client.? 172.21.15.45:0/4136269579' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:03:46.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:45 smithi045 conmon[33655]: cluster 2021-07-31T03:03:44.894466+0000 mgr.x (mgr.24542) 2021-07-31T03:03:46.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:45 smithi045 conmon[33655]: 3281 : cluster [DBG] pgmap v4303: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:46.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:45 smithi045 conmon[33655]: audit 2021-07-31T03:03:44.945220+0000 mgr.x (mgr.24542) 3282 : audit [DBG] from='client.18714 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:03:46.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:45 smithi045 conmon[33655]: audit 2021-07-31T03:03:45.229089+0000 mgr.x (mgr. 2021-07-31T03:03:46.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:45 smithi045 conmon[33655]: 24542) 3283 : audit [DBG] from='client.28660 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:03:46.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:45 smithi045 conmon[33655]: audit 2021-07-31 2021-07-31T03:03:46.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:45 smithi045 conmon[33655]: T03:03:45.572058+0000 2021-07-31T03:03:46.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:45 smithi045 conmon[33655]: mon.c ( 2021-07-31T03:03:46.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:45 smithi045 conmon[33655]: mon.1 2021-07-31T03:03:46.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:45 smithi045 conmon[33655]: ) 171 : audit [DBG] from='client.? 172.21.15.45:0/4136269579' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:03:46.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:45 smithi045 conmon[39718]: cluster 2021-07-31T03:03:44.894466+0000 mgr.x ( 2021-07-31T03:03:46.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:45 smithi045 conmon[39718]: mgr.24542) 3281 : cluster [DBG] pgmap v4303: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:46.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:45 smithi045 conmon[39718]: audit 2021-07-31T03 2021-07-31T03:03:46.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:45 smithi045 conmon[39718]: :03:44.945220+0000 mgr.x (mgr.24542) 3282 : audit [DBG] from='client.18714 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:03:46.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:45 smithi045 conmon[39718]: audit 2021-07-31T03:03:46.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:45 smithi045 conmon[39718]: 2021-07-31T03:03:45.229089+0000 2021-07-31T03:03:46.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:45 smithi045 conmon[39718]: mgr.x (mgr.24542) 3283 : audit [DBG] from='client.28660 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:03:46.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:45 smithi045 conmon[39718]: 2021-07-31T03:03:46.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:45 smithi045 conmon[39718]: audit 2021-07-31T03:03 2021-07-31T03:03:46.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:45 smithi045 conmon[39718]: :45.572058+0000 mon.c (mon.1) 171 2021-07-31T03:03:46.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:45 smithi045 conmon[39718]: : audit [DBG] from='client.? 172.21.15.45:0/4136269579' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:03:48.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:47 smithi154 conmon[34553]: cluster 2021-07-31T03:03:46.894968+0000 mgr.x (mgr.24542) 3284 : cluster [DBG] pgmap v4304: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:48.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:47 smithi045 conmon[39718]: cluster 2021-07-31T03:03:46.894968+0000 mgr.x (mgr. 2021-07-31T03:03:48.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:47 smithi045 conmon[39718]: 24542) 3284 : cluster [DBG] pgmap v4304: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:48.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:47 smithi045 conmon[33655]: cluster 2021-07-31T03:03:46.894968+0000 mgr.x (mgr.24542) 3284 : 2021-07-31T03:03:48.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:47 smithi045 conmon[33655]: cluster [DBG] pgmap v4304: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:50.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:49 smithi154 conmon[34553]: cluster 2021-07-31T03:03:48.895517+0000 mgr.x (mgr.24542) 3285 : cluster [DBG] pgmap v4305: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:50.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:49 smithi045 conmon[39718]: cluster 2021-07-31T03:03:48.895517+0000 mgr.x (mgr. 2021-07-31T03:03:50.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:49 smithi045 conmon[39718]: 24542) 3285 : cluster [DBG] pgmap v4305: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:50.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:49 smithi045 conmon[33655]: cluster 2021-07-31T03:03:48.895517+0000 mgr.x (mgr.24542 2021-07-31T03:03:50.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:49 smithi045 conmon[33655]: ) 3285 : cluster [DBG] pgmap v4305: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:52.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:51 smithi154 conmon[34553]: cluster 2021-07-31T03:03:50.896062+0000 mgr.x (mgr.24542) 3286 : cluster [DBG] pgmap v4306: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:52.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:51 smithi045 conmon[39718]: cluster 2021-07-31T03:03:50.896062+0000 mgr.x (mgr.24542) 3286 : cluster [DBG] 2021-07-31T03:03:52.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:51 smithi045 conmon[39718]: pgmap v4306: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:52.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:51 smithi045 conmon[33655]: cluster 2021-07-31T03:03:50.896062+0000 mgr.x (mgr.24542) 3286 : cluster [DBG] pgmap v4306: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:54.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:53 smithi154 conmon[34553]: cluster 2021-07-31T03:03:52.896593+0000 mgr.x (mgr.24542) 3287 : cluster [DBG] pgmap v4307: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:54.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:53 smithi045 conmon[33655]: cluster 2021-07-31T03:03:52.896593+0000 mgr.x ( 2021-07-31T03:03:54.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:53 smithi045 conmon[33655]: mgr.24542) 3287 : cluster [DBG] pgmap v4307: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:54.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:53 smithi045 conmon[39718]: cluster 2021-07-31T03:03:52.896593+0000 mgr.x (mgr.24542 2021-07-31T03:03:54.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:53 smithi045 conmon[39718]: ) 3287 : cluster [DBG] pgmap v4307: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:56.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:55 smithi154 conmon[34553]: cluster 2021-07-31T03:03:54.897282+0000 mgr.x (mgr.24542) 3288 : cluster [DBG] pgmap v4308: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:56.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:55 smithi045 conmon[33655]: cluster 2021-07-31T03:03:54.897282+0000 mgr.x (mgr.24542) 3288 : cluster 2021-07-31T03:03:56.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:55 smithi045 conmon[33655]: [DBG] pgmap v4308: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:56.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:55 smithi045 conmon[39718]: cluster 2021-07-31T03:03:54.897282+0000 mgr.x (mgr 2021-07-31T03:03:56.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:55 smithi045 conmon[39718]: .24542) 3288 : cluster [DBG] pgmap v4308: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:58.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:57 smithi154 conmon[34553]: cluster 2021-07-31T03:03:56.897680+0000 mgr.x (mgr.24542) 3289 : cluster [DBG] pgmap v4309: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:58.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:57 smithi045 conmon[33655]: cluster 2021-07-31 2021-07-31T03:03:58.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:57 smithi045 conmon[33655]: T03:03:56.897680+0000 mgr.x (mgr.24542) 3289 : cluster [DBG] pgmap v4309: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:03:58.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:57 smithi045 conmon[39718]: cluster 2021-07-31T03:03:56.897680+0000 mgr.x (mgr 2021-07-31T03:03:58.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:57 smithi045 conmon[39718]: .24542) 3289 : cluster [DBG] pgmap v4309: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:00.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:03:59 smithi154 conmon[34553]: cluster 2021-07-31T03:03:58.898097+0000 mgr.x (mgr.24542) 3290 : cluster [DBG] pgmap v4310: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:00.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:59 smithi045 conmon[39718]: cluster 2021-07-31T03:03:58.898097+0000 mgr.x (mgr. 2021-07-31T03:04:00.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:03:59 smithi045 conmon[39718]: 24542) 3290 : cluster [DBG] pgmap v4310: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:00.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:59 smithi045 conmon[33655]: cluster 2021-07-31T03:03:58.898097+0000 mgr.x ( 2021-07-31T03:04:00.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:03:59 smithi045 conmon[33655]: mgr.24542) 3290 : cluster [DBG] pgmap v4310: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:02.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:01 smithi154 conmon[34553]: cluster 2021-07-31T03:04:00.898560+0000 mgr.x (mgr.24542) 3291 : cluster [DBG] pgmap v4311: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:02.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:01 smithi045 conmon[39718]: cluster 2021-07-31T03:04:00.898560+0000 mgr.x (mgr 2021-07-31T03:04:02.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:01 smithi045 conmon[39718]: .24542) 3291 : cluster [DBG] pgmap v4311: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:02.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:01 smithi045 conmon[33655]: cluster 2021-07-31T03:04:00.898560+0000 mgr.x (mgr.24542) 3291 : cluster 2021-07-31T03:04:02.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:01 smithi045 conmon[33655]: [DBG] pgmap v4311: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:04.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:03 smithi154 conmon[34553]: cluster 2021-07-31T03:04:02.899110+0000 mgr.x (mgr.24542) 3292 : cluster [DBG] pgmap v4312: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:04.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:03 smithi045 conmon[39718]: cluster 2021-07-31T03:04:02.899110 2021-07-31T03:04:04.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:03 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 3292 : cluster [DBG] pgmap v4312: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:04.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:03 smithi045 conmon[33655]: cluster 2021-07-31T03:04:02.899110+0000 mgr.x (mgr.24542) 3292 : cluster [DBG] pgmap v4312: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:06.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:05 smithi154 conmon[34553]: cluster 2021-07-31T03:04:04.899590+0000 mgr.x (mgr.24542) 3293 : cluster [DBG] pgmap v4313: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:06.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:05 smithi045 conmon[39718]: cluster 2021-07-31T03:04:04.899590+0000 mgr.x (mgr 2021-07-31T03:04:06.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:05 smithi045 conmon[39718]: .24542) 3293 : cluster [DBG] pgmap v4313: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:06.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:05 smithi045 conmon[33655]: cluster 2021-07-31T03:04:04.899590+0000 mgr.x (mgr.24542) 3293 : cluster [DBG] pgmap v4313: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:08.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:07 smithi154 conmon[34553]: cluster 2021-07-31T03:04:06.899961+0000 mgr.x (mgr.24542) 3294 : cluster 2021-07-31T03:04:08.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:07 smithi154 conmon[34553]: [DBG] pgmap v4314: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:08.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:07 smithi045 conmon[39718]: cluster 2021-07-31T03:04:06.899961+0000 mgr.x ( 2021-07-31T03:04:08.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:07 smithi045 conmon[39718]: mgr.24542) 3294 : cluster [DBG] pgmap v4314: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:08.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:07 smithi045 conmon[33655]: cluster 2021-07-31T03:04:06.899961+0000 mgr.x (mgr.24542) 3294 : cluster [DBG] pgmap v4314: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:10.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:09 smithi154 conmon[34553]: cluster 2021-07-31T03:04:08.900541+0000 mgr.x (mgr.24542) 3295 : cluster [DBG] pgmap v4315: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:10.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:09 smithi045 conmon[33655]: cluster 2021-07-31T03:04:08.900541+0000 mgr.x (mgr.24542) 3295 : cluster 2021-07-31T03:04:10.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:09 smithi045 conmon[33655]: [DBG] pgmap v4315: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:10.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:09 smithi045 conmon[39718]: cluster 2021-07-31T03:04:08. 2021-07-31T03:04:10.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:09 smithi045 conmon[39718]: 900541+0000 mgr.x (mgr.24542) 3295 : cluster [DBG] pgmap v4315: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:12.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:11 smithi154 conmon[34553]: cluster 2021-07-31T03:04:10.901084+0000 mgr.x (mgr.24542) 3296 : cluster [DBG] pgmap v4316: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:12.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:11 smithi045 conmon[39718]: cluster 2021-07-31T03:04:10 2021-07-31T03:04:12.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:11 smithi045 conmon[39718]: .901084+0000 mgr.x (mgr.24542) 3296 : cluster [DBG] pgmap v4316: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:12.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:11 smithi045 conmon[33655]: cluster 2021-07-31T03:04:10.901084+0000 mgr.x (mgr.24542) 3296 : cluster [DBG] pgmap v4316: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:14.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:13 smithi154 conmon[34553]: cluster 2021-07-31T03:04:12.901486+0000 mgr.x (mgr.24542) 3297 : cluster [DBG] pgmap v4317: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:14.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:13 smithi154 conmon[34553]: 2021-07-31T03:04:14.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:13 smithi045 conmon[39718]: cluster 2021-07-31T03:04:12. 2021-07-31T03:04:14.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:13 smithi045 conmon[39718]: 901486+0000 mgr.x (mgr.24542) 3297 : cluster [DBG] pgmap v4317: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:14.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:13 smithi045 conmon[33655]: cluster 2021-07-31T03:04:12.901486+0000 mgr.x (mgr.24542) 3297 : 2021-07-31T03:04:14.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:13 smithi045 conmon[33655]: cluster [DBG] pgmap v4317: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:15.886 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:04:16.167 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:04:16.167 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 79m ago 79m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:04:16.167 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 76m ago 79m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:04:16.168 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 76m ago 83m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:04:16.168 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 79m ago 85m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:04:16.168 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 79m ago 85m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:04:16.168 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 76m ago 83m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:04:16.168 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 79m ago 83m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:04:16.169 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 79m ago 80m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:04:16.169 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 76m ago 80m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:04:16.169 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 79m ago 82m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:04:16.169 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 79m ago 82m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:04:16.169 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 79m ago 82m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:04:16.170 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 79m ago 81m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:04:16.170 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 76m ago 81m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:04:16.170 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 76m ago 81m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:04:16.170 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 76m ago 80m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:04:16.170 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 76m ago 80m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:04:16.171 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 76m ago 80m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:04:16.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:15 smithi154 conmon[34553]: cluster 2021-07-31T03:04:14.901883+0000 mgr.x (mgr.24542) 3298 : cluster [DBG] pgmap v4318: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:16.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:15 smithi045 conmon[33655]: cluster 2021-07-31T03:04:14.901883+0000 mgr.x (mgr 2021-07-31T03:04:16.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:15 smithi045 conmon[33655]: .24542) 3298 : cluster [DBG] pgmap v4318: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:16.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:15 smithi045 conmon[39718]: cluster 2021-07-31T03: 2021-07-31T03:04:16.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:15 smithi045 conmon[39718]: 04:14.901883+0000 mgr.x (mgr.24542) 3298 : cluster [DBG] pgmap v4318: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:16.502 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:04:16.503 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:04:16.503 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:04:16.503 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:04:16.503 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:04:16.504 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:04:16.504 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:04:16.504 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:04:16.504 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:04:16.505 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:04:16.505 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:04:16.505 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:04:16.505 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:04:16.505 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:04:16.506 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:04:16.506 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:04:16.506 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:04:17.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:16 smithi154 conmon[34553]: audit 2021-07-31T03:04:15.874123+0000 mgr.x (mgr.24542) 3299 : audit [DBG] from='client.18732 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:04:17.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:16 smithi154 conmon[34553]: audit 2021-07-31T03:04:16.158649+0000 mgr.x (mgr.24542) 3300 : audit [DBG] from='client.18738 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:04:17.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:16 smithi154 conmon[34553]: audit 2021-07-31T03:04:16.502679+0000 mon.c (mon.1) 172 : audit [DBG] from='client.? 172.21.15.45:0/515334422' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:04:17.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:16 smithi154 conmon[34553]: 2021-07-31T03:04:17.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:16 smithi045 conmon[33655]: audit 2021-07-31T03:04:15.874123+0000 mgr.x (mgr 2021-07-31T03:04:17.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:16 smithi045 conmon[33655]: .24542) 3299 : audit [DBG] from='client.18732 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:04:17.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:16 smithi045 conmon[33655]: audit 2021-07-31T03:04:16.158649+0000 mgr.x (mgr. 2021-07-31T03:04:17.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:16 smithi045 conmon[33655]: 24542) 3300 : audit [DBG] from='client.18738 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:04:17.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:16 smithi045 conmon[33655]: audit 2021-07-31 2021-07-31T03:04:17.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:16 smithi045 conmon[33655]: T03:04:16.502679+0000 mon.c (mon.1) 172 : audit 2021-07-31T03:04:17.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:16 smithi045 conmon[33655]: [DBG] from='client.? 172.21.15.45:0/515334422' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:04:17.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:16 smithi045 conmon[39718]: audit 2021-07-31T03:04: 2021-07-31T03:04:17.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:16 smithi045 conmon[39718]: 15.874123+0000 mgr.x (mgr.24542) 3299 : audit [DBG] from='client.18732 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:04:17.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:16 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T03:04:17.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:16 smithi045 conmon[39718]: 03:04:16.158649+0000 mgr.x (mgr.24542 2021-07-31T03:04:17.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:16 smithi045 conmon[39718]: ) 3300 : audit [DBG] from='client.18738 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:04:17.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:16 smithi045 conmon[39718]: audit 2021-07 2021-07-31T03:04:17.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:16 smithi045 conmon[39718]: -31T03:04:16.502679+0000 mon.c 2021-07-31T03:04:17.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:16 smithi045 conmon[39718]: (mon.1) 172 : audit [DBG] from='client.? 172.21.15.45:0/515334422' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:04:17.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:16 smithi045 conmon[39718]: 2021-07-31T03:04:18.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:17 smithi154 conmon[34553]: cluster 2021-07-31T03:04:16.902397+0000 mgr.x (mgr.24542) 3301 : cluster [DBG] pgmap v4319: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:18.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:17 smithi045 conmon[33655]: cluster 2021-07-31T03:04:16. 2021-07-31T03:04:18.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:17 smithi045 conmon[33655]: 902397+0000 mgr.x (mgr.24542) 3301 : cluster [DBG] pgmap v4319: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:18.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:17 smithi045 conmon[39718]: cluster 2021-07-31T03:04:16.902397+0000 mgr.x (mgr 2021-07-31T03:04:18.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:17 smithi045 conmon[39718]: .24542) 3301 : cluster [DBG] pgmap v4319: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:20.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:19 smithi154 conmon[34553]: cluster 2021-07-31T03:04:18.902820+0000 mgr.x (mgr.24542) 3302 : cluster [DBG] pgmap v4320: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:20.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:19 smithi045 conmon[33655]: cluster 2021-07-31T03:04:18.902820+0000 mgr.x (mgr.24542) 3302 : cluster 2021-07-31T03:04:20.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:19 smithi045 conmon[33655]: [DBG] pgmap v4320: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:20.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:19 smithi045 conmon[39718]: cluster 2021-07-31T03:04:18.902820 2021-07-31T03:04:20.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:19 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 3302 : cluster [DBG] pgmap v4320: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:22.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:21 smithi154 conmon[34553]: cluster 2021-07-31T03:04:20.903281+0000 mgr.x (mgr.24542) 3303 : cluster [DBG] pgmap v4321: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:22.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:21 smithi045 conmon[33655]: cluster 2021-07-31T03:04:20.903281+0000 mgr.x (mgr.24542) 3303 2021-07-31T03:04:22.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:21 smithi045 conmon[33655]: : cluster [DBG] pgmap v4321: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:22.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:21 smithi045 conmon[39718]: cluster 2021-07-31T03:04: 2021-07-31T03:04:22.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:21 smithi045 conmon[39718]: 20.903281+0000 mgr.x (mgr.24542) 3303 : cluster [DBG] pgmap v4321: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:24.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:23 smithi154 conmon[34553]: cluster 2021-07-31T03:04:22.903596+0000 mgr.x (mgr.24542) 3304 : cluster [DBG] pgmap v4322: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:24.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:23 smithi045 conmon[39718]: cluster 2021-07-31T03:04:22.903596+0000 mgr.x 2021-07-31T03:04:24.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:23 smithi045 conmon[39718]: (mgr.24542) 3304 : cluster [DBG] pgmap v4322: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:24.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:23 smithi045 conmon[33655]: cluster 2021-07-31T03:04:22.903596+0000 mgr.x (mgr.24542) 3304 2021-07-31T03:04:24.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:23 smithi045 conmon[33655]: : cluster [DBG] pgmap v4322: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:26.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:25 smithi154 conmon[34553]: cluster 2021-07-31T03:04:24.904053+0000 mgr.x (mgr.24542) 3305 : cluster [DBG] pgmap v4323: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:26.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:25 smithi045 conmon[33655]: cluster 2021-07-31T03:04:24.904053+0000 mgr.x (mgr.24542) 3305 : cluster [DBG] pgmap v4323: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:26.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:25 smithi045 conmon[39718]: cluster 2021-07-31T03:04:24. 2021-07-31T03:04:26.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:25 smithi045 conmon[39718]: 904053+0000 mgr.x (mgr.24542) 3305 : cluster [DBG] pgmap v4323: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:28.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:27 smithi154 conmon[34553]: cluster 2021-07-31T03:04:26.904585+0000 mgr.x (mgr.24542) 3306 : cluster [DBG] pgmap v4324: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:28.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:27 smithi045 conmon[33655]: cluster 2021-07-31T03:04:26.904585+0000 mgr.x (mgr.24542) 3306 : cluster 2021-07-31T03:04:28.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:27 smithi045 conmon[33655]: [DBG] pgmap v4324: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:28.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:27 smithi045 conmon[39718]: cluster 2021-07-31T03:04:26.904585+0000 mgr.x (mgr 2021-07-31T03:04:28.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:27 smithi045 conmon[39718]: .24542) 3306 : cluster [DBG] pgmap v4324: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:28 smithi154 conmon[34553]: audit 2021-07-31T03:04:28.240689+0000 mon.b (mon.2) 460 : audit [INF] 2021-07-31T03:04:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:28 smithi154 conmon[34553]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:04:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:28 smithi154 conmon[34553]: audit 2021-07-31T03:04:28.241616+0000 mon.b (mon.2) 461 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:04:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:28 smithi154 conmon[34553]: audit 2021-07-31T03:04:28.241890+0000 mon.a ( 2021-07-31T03:04:29.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:28 smithi154 conmon[34553]: mon.0) 3247 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:04:29.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:28 smithi154 conmon[34553]: audit 2021-07-31T 2021-07-31T03:04:29.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:28 smithi154 conmon[34553]: 03:04:28.242356+0000 mon.a (mon.0) 3248 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:04:29.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:28 smithi154 conmon[34553]: 2021-07-31T03:04:29.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:28 smithi045 conmon[39718]: audit 2021-07-31T03:04:28.240689+0000 mon.b 2021-07-31T03:04:29.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:28 smithi045 conmon[39718]: (mon.2) 460 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:04:29.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:28 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T03:04:29.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:28 smithi045 conmon[39718]: T03:04:28.241616+0000 mon.b (mon.2) 461 : audit 2021-07-31T03:04:29.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:28 smithi045 conmon[39718]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:04:29.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:28 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T03:04:29.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:28 smithi045 conmon[39718]: 03:04:28.241890+0000 mon.a (mon 2021-07-31T03:04:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:28 smithi045 conmon[39718]: .0) 3247 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:04:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:28 smithi045 conmon[39718]: audit 2021- 2021-07-31T03:04:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:28 smithi045 conmon[39718]: 07-31T03:04:28.242356 2021-07-31T03:04:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:28 smithi045 conmon[39718]: +0000 mon.a (mon.0) 3248 : audit 2021-07-31T03:04:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:28 smithi045 conmon[39718]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:04:29.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:28 smithi045 conmon[33655]: audit 2021-07-31T03:04:28.240689+0000 mon.b (mon. 2021-07-31T03:04:29.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:28 smithi045 conmon[33655]: 2) 460 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:04:29.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:28 smithi045 conmon[33655]: audit 2021-07-31T03:04:28.241616+0000 2021-07-31T03:04:29.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:28 smithi045 conmon[33655]: mon.b (mon.2) 461 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:04:29.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:28 smithi045 conmon[33655]: audit 2021-07-31T 2021-07-31T03:04:29.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:28 smithi045 conmon[33655]: 03:04:28.241890+0000 2021-07-31T03:04:29.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:28 smithi045 conmon[33655]: mon.a (mon.0) 3247 : 2021-07-31T03:04:29.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:28 smithi045 conmon[33655]: audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:04:29.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:28 smithi045 conmon[33655]: audit 2021- 2021-07-31T03:04:29.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:28 smithi045 conmon[33655]: 07-31T03:04:28.242356+0000 2021-07-31T03:04:29.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:28 smithi045 conmon[33655]: mon.a (mon.0) 3248 : 2021-07-31T03:04:29.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:28 smithi045 conmon[33655]: audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:04:30.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:29 smithi154 conmon[34553]: cluster 2021-07-31T03:04:28.904997+0000 mgr.x (mgr.24542) 3307 : cluster [DBG] pgmap v4325: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:30.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:29 smithi045 conmon[39718]: cluster 2021-07-31T03:04:28.904997+0000 2021-07-31T03:04:30.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:29 smithi045 conmon[39718]: mgr.x (mgr.24542) 3307 : cluster [DBG] pgmap v4325: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:30.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:29 smithi045 conmon[33655]: cluster 2021-07-31T03:04:28.904997+0000 mgr.x 2021-07-31T03:04:30.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:29 smithi045 conmon[33655]: (mgr.24542) 3307 : cluster [DBG] pgmap v4325: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:31.503 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: debug 2021-07-31T03:04:31.141+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:04:31.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:04:31.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:04:31.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: yield 2021-07-31T03:04:31.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:04:31.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:04:31.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:04:31.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:04:31.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:04:31.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:04:31.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:04:31.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:04:31.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:04:31.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: task.result() 2021-07-31T03:04:31.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:04:31.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: debug 2021-07-31T03:04:31.152+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:04:31.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:04:31.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:04:31.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: yield 2021-07-31T03:04:31.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:04:31.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:04:31.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:04:31.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:04:31.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:04:31.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:04:31.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:04:31.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:04:31.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:04:31.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: task.result() 2021-07-31T03:04:31.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:04:31.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: debug 2021-07-31T03:04:31.177+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:04:31.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:04:31.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:04:31.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: yield 2021-07-31T03:04:31.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:04:31.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:04:31.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:04:31.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:04:31.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:04:31.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:04:31.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:04:31.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:04:31.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:04:31.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: task.result() 2021-07-31T03:04:31.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:04:31.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: debug 2021-07-31T03:04:31.178+0000 7fcb9195d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:04:31.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:04:31.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:04:31.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: yield 2021-07-31T03:04:31.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:04:31.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:04:31.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:04:31.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:04:31.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:04:31.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:04:31.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:04:31.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:04:31.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:04:31.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: task.result() 2021-07-31T03:04:31.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:04:31.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: 2021-07-31T03:04:31.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:04:31.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: 2021-07-31T03:04:31.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:04:31.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:04:31.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:04:31.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:04:31.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:04:31.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:04:31.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:04:31.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:04:31.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:04:31.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:04:31.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:04:31.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:04:31.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:04:31.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:04:31.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:04:31.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:04:31.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:04:31.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:04:31.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: raise 2021-07-31T03:04:31.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:04:31.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:04:31.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:04:31.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:04:31.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:04:31.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: debug 2021-07-31T03:04:31.186+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:04:31.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:04:31.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:04:31.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: yield 2021-07-31T03:04:31.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:04:31.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:04:31.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:04:31.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:04:31.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:04:31.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:04:31.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:04:31.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:04:31.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:04:31.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: task.result() 2021-07-31T03:04:31.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:04:31.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: debug 2021-07-31T03:04:31.187+0000 7fcb9215e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:04:31.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:04:31.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:04:31.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: yield 2021-07-31T03:04:31.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:04:31.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:04:31.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:04:31.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:04:31.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:04:31.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:04:31.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:04:31.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:04:31.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:04:31.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: task.result() 2021-07-31T03:04:31.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:04:31.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: 2021-07-31T03:04:31.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:04:31.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: 2021-07-31T03:04:31.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:04:31.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:04:31.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:04:31.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:04:31.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:04:31.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:04:31.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:04:31.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:04:31.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:04:31.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:04:31.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:04:31.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:04:31.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:04:31.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:04:31.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:04:31.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:04:31.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:04:31.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:04:31.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: raise 2021-07-31T03:04:31.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:04:31.535 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:04:31.535 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:04:31.535 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:04:31.535 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:04:32.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: cluster 2021-07-31T03:04:30.905378+0000 mgr.x (mgr.24542) 3308 : cluster [DBG] pgmap v4326: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:32.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: audit 2021-07-31T03:04:31.096176+0000 mon.b (mon.2) 462 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:04:32.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: audit 2021-07-31T03:04:31.097676+0000 2021-07-31T03:04:32.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: mon.b (mon.2) 463 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:04:32.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: audit 2021-07-31T03:04:31.098586+0000 mon.b (mon.2) 464 : audit [INF] 2021-07-31T03:04:32.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:04:32.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: cephadm 2021-07-31T03:04 2021-07-31T03:04:32.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: :31.099322+0000 mgr.x (mgr.24542) 3309 2021-07-31T03:04:32.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:04:32.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: cephadm 2021- 2021-07-31T03:04:32.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: 07-31T03:04:31.142149+0000 mgr.x 2021-07-31T03:04:32.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: (mgr.24542) 3310 : cephadm [ERR] 2021-07-31T03:04:32.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:04:32.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:04:32.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: yield 2021-07-31T03:04:32.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:04:32.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:04:32.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:04:32.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:04:32.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:04:32.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:04:32.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:04:32.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:04:32.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:04:32.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: task.result() 2021-07-31T03:04:32.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:04:32.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: cephadm 2021-07-31T03:04:31.142616+0000 mgr.x (mgr.24542) 3311 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:04:32.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: cephadm 2021-07-31T03:04:31.153711+0000 mgr.x (mgr. 2021-07-31T03:04:32.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: 24542) 3312 : cephadm [ERR] 2021-07-31T03:04:32.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:04:32.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:04:32.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: yield 2021-07-31T03:04:32.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:04:32.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:04:32.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:04:32.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:04:32.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:04:32.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:04:32.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:04:32.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:04:32.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:04:32.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: task.result() 2021-07-31T03:04:32.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:04:32.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: cephadm 2021-07-31T03:04:31.154727+0000 mgr.x (mgr.24542) 3313 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:04:32.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: cephadm 2021-07-31T03:04:31.178881+0000 mgr.x (mgr.24542) 3314 : cephadm [ERR] 2021-07-31T03:04:32.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:04:32.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:04:32.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: yield 2021-07-31T03:04:32.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:04:32.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:04:32.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:04:32.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:04:32.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:04:32.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:04:32.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:04:32.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:04:32.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:04:32.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: task.result() 2021-07-31T03:04:32.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:04:32.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: cephadm 2021-07-31T03:04:31.179473+0000 mgr.x (mgr.24542) 3315 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:04:32.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:04:32.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:04:32.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: yield 2021-07-31T03:04:32.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:04:32.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:04:32.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:04:32.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:04:32.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:04:32.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:04:32.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:04:32.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:04:32.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:04:32.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: task.result() 2021-07-31T03:04:32.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:04:32.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: 2021-07-31T03:04:32.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:04:32.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: 2021-07-31T03:04:32.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:04:32.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:04:32.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:04:32.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:04:32.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:04:32.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:04:32.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:04:32.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:04:32.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:04:32.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:04:32.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:04:32.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:04:32.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:04:32.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:04:32.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:04:32.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:04:32.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:04:32.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:04:32.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: raise 2021-07-31T03:04:32.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:04:32.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:04:32.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:04:32.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:04:32.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:04:32.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: cephadm 2021-07-31T03:04:31.187895+0000 mgr.x (mgr.24542 2021-07-31T03:04:32.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: ) 3316 : cephadm [ERR] 2021-07-31T03:04:32.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:04:32.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:04:32.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: yield 2021-07-31T03:04:32.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:04:32.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:04:32.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:04:32.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:04:32.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:04:32.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:04:32.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:04:32.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:04:32.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:04:32.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: task.result() 2021-07-31T03:04:32.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:04:32.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: cluster 2021-07-31T03:04:30.905378+0000 mgr.x (mgr. 2021-07-31T03:04:32.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: 24542) 3308 : cluster [DBG] pgmap v4326: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:32.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: audit 2021-07-31T03:04:31.096176+0000 mon.b (mon.2) 462 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:04:32.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: audit 2021-07-31T 2021-07-31T03:04:32.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: 03:04:31.097676+0000 mon.b (mon.2) 463 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:04:32.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: audit 2021 2021-07-31T03:04:32.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: -07-31T03:04:31 2021-07-31T03:04:32.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: .098586+0000 mon.b (mon.2) 464 2021-07-31T03:04:32.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:04:32.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: cephadm 2021-07-31T03:04:32.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: 2021-07-31T03:04 2021-07-31T03:04:32.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: :31.099322+0000 mgr.x (mgr. 2021-07-31T03:04:32.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: 24542) 3309 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:04:32.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: cephadm 2021-07-31 2021-07-31T03:04:32.283 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: T03:04:31.142149+0000 mgr.x 2021-07-31T03:04:32.283 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: (mgr.24542) 3310 : cephadm 2021-07-31T03:04:32.283 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: [ERR] 2021-07-31T03:04:32.283 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:04:32.283 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:04:32.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: yield 2021-07-31T03:04:32.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:04:32.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:04:32.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:04:32.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:04:32.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:04:32.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:04:32.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:04:32.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:04:32.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:04:32.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: task.result() 2021-07-31T03:04:32.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:04:32.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: cephadm 2021-07-31T03:04:31.142616+0000 mgr.x (mgr.24542) 3311 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:04:32.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: cephadm 2021 2021-07-31T03:04:32.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: -07-31T03:04:31.153711 2021-07-31T03:04:32.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 3312 : cephadm 2021-07-31T03:04:32.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: [ERR] 2021-07-31T03:04:32.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:04:32.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:04:32.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: yield 2021-07-31T03:04:32.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:04:32.288 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: cephadm 2021-07-31T03:04:31.188285+0000 mgr.x (mgr.24542) 3317 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:04:32.289 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:04:32.289 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:04:32.289 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: yield 2021-07-31T03:04:32.289 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:04:32.290 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:04:32.290 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:04:32.290 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:04:32.290 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:04:32.290 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:04:32.291 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:04:32.291 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:04:32.291 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:04:32.291 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: task.result() 2021-07-31T03:04:32.292 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:04:32.292 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: 2021-07-31T03:04:32.292 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:04:32.292 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: 2021-07-31T03:04:32.293 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:04:32.293 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:04:32.293 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:04:32.293 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:04:32.294 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:04:32.294 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:04:32.294 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:04:32.294 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:04:32.295 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:04:32.295 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:04:32.295 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:04:32.295 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:04:32.296 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:04:32.296 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:04:32.296 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:04:32.296 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:04:32.297 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:04:32.297 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:04:32.297 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: raise 2021-07-31T03:04:32.297 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:04:32.297 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:04:32.298 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:04:32.298 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:04:32.298 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:31 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:04:32.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: cluster 2021-07-31T03:04:30.905378+0000 mgr.x (mgr.24542) 3308 2021-07-31T03:04:32.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: : cluster [DBG] pgmap v4326: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:32.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: audit 2021-07-31T03:04 2021-07-31T03:04:32.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: :31.096176+0000 mon.b (mon.2) 462 : audit 2021-07-31T03:04:32.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:04:32.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: audit 2021-07-31T03:04:32.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: 2021-07-31T03: 2021-07-31T03:04:32.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: 04:31.097676+0000 mon.b (mon.2) 463 2021-07-31T03:04:32.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:04:32.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: audit 2021 2021-07-31T03:04:32.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: -07-31T03:04: 2021-07-31T03:04:32.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: 31.098586+0000 mon.b (mon. 2021-07-31T03:04:32.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: 2) 464 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:04:32.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: cephadm 2021- 2021-07-31T03:04:32.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: 07-31T03:04:31.099322 2021-07-31T03:04:32.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 3309 : cephadm 2021-07-31T03:04:32.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:04:32.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: cephadm 2021-07-31T03:04:32.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: 2021-07-31T03 2021-07-31T03:04:32.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: :04:31.142149+0000 mgr.x 2021-07-31T03:04:32.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: (mgr.24542) 3310 : cephadm [ERR] 2021-07-31T03:04:32.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: 2021-07-31T03:04:32.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:04:32.304 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:04:32.304 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: yield 2021-07-31T03:04:32.304 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:04:32.304 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:04:32.304 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:04:32.305 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:04:32.305 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:04:32.305 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:04:32.305 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:04:32.305 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:04:32.306 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:04:32.306 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: task.result() 2021-07-31T03:04:32.306 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:04:32.306 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: cephadm 2021-07-31T03:04:31.142616+0000 mgr.x (mgr.24542) 3311 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:04:32.307 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: cephadm 2021-07-31T03:04:32.307 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: 2021- 2021-07-31T03:04:32.307 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: 07-31T03:04:31 2021-07-31T03:04:32.307 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: .153711+0000 mgr.x (mgr.24542) 3312 : cephadm 2021-07-31T03:04:32.308 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: [ERR] 2021-07-31T03:04:32.308 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:04:32.308 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:04:32.308 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: yield 2021-07-31T03:04:32.309 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:04:32.309 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:04:32.309 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:04:32.309 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:04:32.310 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:04:32.310 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:04:32.310 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:04:32.310 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:04:32.310 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: task.result() 2021-07-31T03:04:32.311 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:04:32.311 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: cephadm 2021-07-31T03:04:31.154727+0000 mgr.x (mgr.24542) 3313 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:04:32.311 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: cephadm 2021-07-31T03:04:32.311 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: 2021-07-31T03 2021-07-31T03:04:32.311 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: :04:31.178881+0000 2021-07-31T03:04:32.312 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: mgr.x (mgr.24542) 3314 : cephadm [ERR] 2021-07-31T03:04:32.312 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: 2021-07-31T03:04:32.312 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:04:32.312 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:04:32.312 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: yield 2021-07-31T03:04:32.313 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:04:32.313 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:04:32.313 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:04:32.313 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:04:32.313 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:04:32.314 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:04:32.314 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:04:32.314 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:04:32.314 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:04:32.315 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: task.result() 2021-07-31T03:04:32.315 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:04:32.315 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: cephadm 2021-07-31T03:04:31.179473+0000 mgr.x (mgr.24542) 3315 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:04:32.315 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:04:32.316 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:04:32.316 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: yield 2021-07-31T03:04:32.316 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:04:32.316 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:04:32.316 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:04:32.316 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:04:32.317 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:04:32.317 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:04:32.317 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:04:32.317 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:04:32.318 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:04:32.318 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:04:32.318 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:04:32.318 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:04:32.319 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:04:32.319 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:04:32.319 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:04:32.319 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:04:32.319 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:04:32.320 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: task.result() 2021-07-31T03:04:32.320 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:04:32.320 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: cephadm 2021-07-31T03:04:31.154727+0000 mgr.x (mgr.24542) 3313 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:04:32.320 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: cephadm 2021 2021-07-31T03:04:32.321 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: -07-31T03:04 2021-07-31T03:04:32.321 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: :31.178881+0000 mgr.x (mgr 2021-07-31T03:04:32.321 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: .24542) 3314 : cephadm [ERR] 2021-07-31T03:04:32.321 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:04:32.321 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:04:32.322 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: yield 2021-07-31T03:04:32.322 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:04:32.322 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:04:32.322 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:04:32.323 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:04:32.323 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:04:32.323 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:04:32.323 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:04:32.323 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:04:32.324 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:04:32.324 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: task.result() 2021-07-31T03:04:32.324 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:04:32.324 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: cephadm 2021-07-31T03:04:31.179473+0000 mgr.x (mgr.24542) 3315 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:04:32.325 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:04:32.325 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:04:32.325 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: yield 2021-07-31T03:04:32.325 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:04:32.325 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:04:32.326 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:04:32.326 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:04:32.326 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:04:32.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:04:32.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: task.result() 2021-07-31T03:04:32.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:04:32.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: 2021-07-31T03:04:32.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:04:32.328 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: 2021-07-31T03:04:32.328 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:04:32.328 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:04:32.328 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:04:32.329 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:04:32.329 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:04:32.329 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:04:32.329 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:04:32.330 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:04:32.330 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:04:32.330 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:04:32.330 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:04:32.331 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:04:32.331 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:04:32.331 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:04:32.331 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:04:32.332 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:04:32.332 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:04:32.332 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:04:32.332 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: raise 2021-07-31T03:04:32.332 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:04:32.333 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:04:32.333 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:04:32.333 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:04:32.333 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:04:32.333 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: cephadm 2021-07-31T03:04:31.187895+0000 mgr.x (mgr.24542) 3316 : cephadm [ERR] 2021-07-31T03:04:32.334 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:04:32.334 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:04:32.334 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: yield 2021-07-31T03:04:32.334 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:04:32.335 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:04:32.335 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:04:32.335 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:04:32.335 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:04:32.335 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:04:32.336 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:04:32.336 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:04:32.336 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:04:32.337 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:04:32.337 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: task.result() 2021-07-31T03:04:32.337 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:04:32.337 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: 2021-07-31T03:04:32.337 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:04:32.338 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: 2021-07-31T03:04:32.338 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:04:32.338 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:04:32.338 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:04:32.338 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:04:32.339 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:04:32.339 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:04:32.339 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:04:32.339 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:04:32.340 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:04:32.340 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:04:32.340 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:04:32.340 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:04:32.340 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:04:32.341 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:04:32.341 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:04:32.341 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:04:32.341 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:04:32.341 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:04:32.342 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: raise 2021-07-31T03:04:32.342 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:04:32.342 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:04:32.342 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:04:32.342 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:04:32.343 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:04:32.343 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: cephadm 2021-07-31T03:04:31.187895+0000 mgr.x (mgr.24542) 3316 : cephadm [ERR] 2021-07-31T03:04:32.343 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:04:32.343 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:04:32.344 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: yield 2021-07-31T03:04:32.344 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:04:32.344 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:04:32.344 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:04:32.344 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:04:32.345 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:04:32.345 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:04:32.346 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:04:32.346 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: task.result() 2021-07-31T03:04:32.346 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:04:32.346 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: cephadm 2021-07-31T03:04:31.188285+0000 mgr.x (mgr.24542) 3317 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:04:32.347 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:04:32.347 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:04:32.347 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: yield 2021-07-31T03:04:32.347 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:04:32.348 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:04:32.348 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:04:32.348 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:04:32.348 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:04:32.349 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:04:32.349 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:04:32.349 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:04:32.349 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:04:32.350 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: task.result() 2021-07-31T03:04:32.350 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:04:32.350 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: 2021-07-31T03:04:32.350 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:04:32.351 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: 2021-07-31T03:04:32.351 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:04:32.351 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:04:32.351 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:04:32.351 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:04:32.352 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:04:32.352 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:04:32.352 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:04:32.352 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:04:32.353 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:04:32.353 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:04:32.353 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:04:32.353 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:04:32.353 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:04:32.354 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:04:32.354 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:04:32.354 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:04:32.354 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:04:32.354 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:04:32.355 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: raise 2021-07-31T03:04:32.355 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:04:32.355 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:04:32.355 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:04:32.355 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:04:32.356 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:04:32.356 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:04:32.357 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:04:32.357 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:04:32.357 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:04:32.357 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:04:32.357 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: task.result() 2021-07-31T03:04:32.358 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:04:32.358 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: cephadm 2021-07-31T03:04:31.188285+0000 mgr.x (mgr.24542) 3317 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:04:32.358 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:04:32.358 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:04:32.358 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: yield 2021-07-31T03:04:32.359 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:04:32.359 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:04:32.359 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:04:32.359 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:04:32.360 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:04:32.360 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:04:32.360 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:04:32.360 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:04:32.360 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:04:32.361 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: task.result() 2021-07-31T03:04:32.361 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:04:32.361 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: 2021-07-31T03:04:32.362 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:04:32.362 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: 2021-07-31T03:04:32.362 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:04:32.362 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:04:32.363 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:04:32.363 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:04:32.363 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:04:32.363 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:04:32.363 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:04:32.364 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:04:32.364 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:04:32.364 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:04:32.364 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:04:32.364 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:04:32.365 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:04:32.365 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:04:32.365 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:04:32.365 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:04:32.365 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:04:32.366 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:04:32.366 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: raise 2021-07-31T03:04:32.366 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:04:32.366 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:04:32.367 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:04:32.367 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:04:32.367 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:31 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:04:34.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:33 smithi154 conmon[34553]: cluster 2021-07-31T03:04:32.905850+0000 mgr.x (mgr.24542) 3318 : cluster [DBG] pgmap v4327: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:34.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:33 smithi045 conmon[39718]: cluster 2021-07-31T03:04:32.905850+0000 mgr.x (mgr.24542) 3318 : cluster [DBG] pgmap v4327: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:34.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:33 smithi045 conmon[33655]: cluster 2021-07-31T03:04:32.905850+0000 mgr.x (mgr.24542) 3318 : cluster [DBG] pgmap v4327: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:36.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:35 smithi154 conmon[34553]: cluster 2021-07-31T03:04:34.906411+0000 mgr.x (mgr.24542) 3319 : cluster [DBG] pgmap v4328: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:36.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:35 smithi045 conmon[39718]: cluster 2021-07-31T03:04:34.906411+0000 mgr.x (mgr.24542) 3319 : cluster 2021-07-31T03:04:36.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:35 smithi045 conmon[39718]: [DBG] pgmap v4328: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:36.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:35 smithi045 conmon[33655]: cluster 2021-07-31T03:04:34.906411+0000 mgr.x (mgr.24542 2021-07-31T03:04:36.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:35 smithi045 conmon[33655]: ) 3319 : cluster [DBG] pgmap v4328: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:38.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:37 smithi154 conmon[34553]: cluster 2021-07-31T03:04:36.906938+0000 mgr.x (mgr.24542) 3320 : cluster [DBG] pgmap v4329: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:38.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:37 smithi045 conmon[33655]: cluster 2021-07-31T03:04:36.906938+0000 mgr.x (mgr.24542) 2021-07-31T03:04:38.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:37 smithi045 conmon[33655]: 3320 : cluster [DBG] pgmap v4329: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:38.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:37 smithi045 conmon[39718]: cluster 2021-07-31T03:04:36.906938+0000 mgr.x (mgr.24542) 3320 : 2021-07-31T03:04:38.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:37 smithi045 conmon[39718]: cluster [DBG] pgmap v4329: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:40.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:39 smithi154 conmon[34553]: cluster 2021-07-31T03:04:38.907500+0000 mgr.x (mgr.24542) 3321 : cluster [DBG] pgmap v4330: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:40.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:39 smithi045 conmon[33655]: cluster 2021-07-31T03:04:38.907500+0000 mgr.x (mgr.24542) 3321 : cluster [DBG] pgmap v4330: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:40.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:39 smithi045 conmon[39718]: cluster 2021-07-31T03:04 2021-07-31T03:04:40.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:39 smithi045 conmon[39718]: :38.907500+0000 mgr.x (mgr.24542) 3321 : cluster [DBG] pgmap v4330: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:42.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:41 smithi154 conmon[34553]: cluster 2021-07-31T03:04:40.907863+0000 mgr.x (mgr.24542) 3322 : cluster [DBG] pgmap v4331: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:42.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:41 smithi045 conmon[33655]: cluster 2021-07-31T03:04:40.907863+0000 mgr.x (mgr.24542) 3322 : cluster [DBG] 2021-07-31T03:04:42.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:41 smithi045 conmon[33655]: pgmap v4331: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:42.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:41 smithi045 conmon[39718]: cluster 2021-07-31T03:04:40.907863+0000 mgr.x (mgr.24542) 3322 : cluster [DBG] pgmap v4331: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:44.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:43 smithi154 conmon[34553]: cluster 2021-07-31T03:04:42.908227+0000 mgr.x (mgr.24542) 3323 : cluster [DBG] pgmap v4332: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:44.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:43 smithi045 conmon[33655]: cluster 2021-07-31T03:04:42.908227+0000 mgr.x (mgr.24542) 2021-07-31T03:04:44.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:43 smithi045 conmon[33655]: 3323 : cluster [DBG] pgmap v4332: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:44.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:43 smithi045 conmon[39718]: cluster 2021-07-31T03:04: 2021-07-31T03:04:44.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:43 smithi045 conmon[39718]: 42.908227+0000 mgr.x (mgr.24542) 3323 : cluster [DBG] pgmap v4332: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:46.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:45 smithi154 conmon[34553]: cluster 2021-07-31T03:04:44.908726+0000 mgr.x (mgr.24542) 3324 : cluster [DBG] pgmap v4333: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:46.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:45 smithi154 conmon[34553]: 2021-07-31T03:04:46.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:45 smithi045 conmon[33655]: cluster 2021-07-31T03:04:44.908726+0000 mgr.x (mgr.24542) 3324 : cluster [DBG] pgmap v4333: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:46.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:45 smithi045 conmon[39718]: cluster 2021-07-31T03:04:44. 2021-07-31T03:04:46.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:45 smithi045 conmon[39718]: 908726+0000 mgr.x (mgr.24542) 3324 : cluster [DBG] pgmap v4333: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:46.816 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:04:47.091 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:04:47.092 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 79m ago 80m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:04:47.092 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 77m ago 80m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:04:47.092 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 77m ago 83m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:04:47.093 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 79m ago 85m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:04:47.093 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 79m ago 85m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:04:47.093 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 77m ago 83m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:04:47.093 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 79m ago 84m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:04:47.094 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 79m ago 80m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:04:47.094 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 77m ago 80m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:04:47.094 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 79m ago 83m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:04:47.095 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 79m ago 83m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:04:47.095 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 79m ago 82m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:04:47.095 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 79m ago 82m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:04:47.095 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 77m ago 82m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:04:47.096 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 77m ago 81m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:04:47.096 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 77m ago 81m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:04:47.096 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 77m ago 81m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:04:47.096 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 77m ago 80m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:04:47.425 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:04:47.425 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:04:47.425 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:04:47.425 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:04:47.426 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:04:47.426 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:04:47.426 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:04:47.426 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:04:47.427 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:04:47.427 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:04:47.427 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:04:47.427 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:04:47.428 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:04:47.428 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:04:47.428 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:04:47.428 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:04:47.429 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:04:48.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:47 smithi154 conmon[34553]: audit 2021-07-31T03:04:46.803901+0000 mgr.x (mgr.24542) 3325 : audit [DBG] from='client.18750 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:04:48.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:47 smithi154 conmon[34553]: cluster 2021-07-31T03:04:46.909082+0000 mgr.x (mgr.24542) 3326 : cluster [DBG] pgmap v4334: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:48.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:47 smithi154 conmon[34553]: audit 2021-07-31T03:04:47.083865+0000 mgr.x (mgr.24542) 3327 : audit [DBG] from='client.28696 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:04:48.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:47 smithi154 conmon[34553]: audit 2021-07-31T03:04:47.425395+0000 mon.a (mon.0) 3249 : audit [DBG] from='client.? 172.21.15.45:0/1455163924' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:04:48.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:47 smithi045 conmon[33655]: audit 2021-07-31T03:04:46.803901+0000 2021-07-31T03:04:48.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:47 smithi045 conmon[33655]: mgr.x (mgr.24542) 3325 : audit [DBG] from='client.18750 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:04:48.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:47 smithi045 conmon[33655]: cluster 2021-07-31T03:04: 2021-07-31T03:04:48.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:47 smithi045 conmon[33655]: 46.909082+0000 mgr.x (mgr.24542) 3326 : cluster [DBG] pgmap v4334: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:48.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:47 smithi045 conmon[33655]: audit 2021-07- 2021-07-31T03:04:48.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:47 smithi045 conmon[33655]: 31T03:04:47.083865+0000 mgr.x 2021-07-31T03:04:48.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:47 smithi045 conmon[33655]: (mgr.24542) 3327 : audit [DBG] from='client.28696 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:04:48.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:47 smithi045 conmon[33655]: audit 2021 2021-07-31T03:04:48.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:47 smithi045 conmon[33655]: -07-31T03:04: 2021-07-31T03:04:48.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:47 smithi045 conmon[33655]: 47.425395+0000 mon.a (mon.0) 3249 : audit [DBG] from='client.? 172.21.15.45:0/1455163924' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:04:48.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:47 smithi045 conmon[39718]: audit 2021-07-31T03:04:46.803901+0000 2021-07-31T03:04:48.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:47 smithi045 conmon[39718]: mgr.x (mgr.24542) 3325 : audit [DBG] from='client.18750 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:04:48.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:47 smithi045 conmon[39718]: cluster 2021-07- 2021-07-31T03:04:48.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:47 smithi045 conmon[39718]: 31T03:04:46.909082+0000 mgr.x (mgr.24542) 3326 : cluster 2021-07-31T03:04:48.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:47 smithi045 conmon[39718]: [DBG] pgmap v4334: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:48.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:47 smithi045 conmon[39718]: audit 2021-07-31T03 2021-07-31T03:04:48.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:47 smithi045 conmon[39718]: :04:47.083865+0000 mgr.x (mgr 2021-07-31T03:04:48.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:47 smithi045 conmon[39718]: .24542) 3327 : audit [DBG] from='client.28696 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:04:48.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:47 smithi045 conmon[39718]: audit 2021- 2021-07-31T03:04:48.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:47 smithi045 conmon[39718]: 07-31T03:04:47.425395+0000 mon.a (mon.0) 2021-07-31T03:04:48.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:47 smithi045 conmon[39718]: 3249 : audit [DBG] from='client.? 172.21.15.45:0/1455163924' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:04:50.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:49 smithi154 conmon[34553]: cluster 2021-07-31T03:04:48.909492+0000 mgr.x (mgr.24542) 3328 : cluster [DBG] 2021-07-31T03:04:50.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:49 smithi154 conmon[34553]: pgmap v4335: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:50.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:49 smithi045 conmon[33655]: cluster 2021-07-31T03:04:48.909492+0000 mgr.x (mgr.24542) 3328 : cluster [DBG] pgmap v4335: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:50.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:49 smithi045 conmon[39718]: cluster 2021-07-31T03:04:48.909492+0000 2021-07-31T03:04:50.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:49 smithi045 conmon[39718]: mgr.x (mgr.24542) 3328 : cluster [DBG] pgmap v4335: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:52.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:51 smithi154 conmon[34553]: cluster 2021-07-31T03:04:50.909895+0000 mgr.x (mgr.24542) 3329 : cluster [DBG] pgmap v4336: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:52.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:51 smithi045 conmon[39718]: cluster 2021-07-31T03:04:50.909895+0000 mgr.x (mgr.24542) 3329 : 2021-07-31T03:04:52.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:51 smithi045 conmon[39718]: cluster [DBG] pgmap v4336: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:52.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:51 smithi045 conmon[33655]: cluster 2021-07-31T03:04:50.909895+0000 mgr.x (mgr.24542) 3329 : cluster [DBG] pgmap v4336: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:54.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:53 smithi154 conmon[34553]: cluster 2021-07-31T03:04:52.910302+0000 mgr.x (mgr.24542) 3330 : cluster [DBG] pgmap v4337: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:54.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:53 smithi045 conmon[39718]: cluster 2021-07-31T03:04:52.910302+0000 mgr.x (mgr 2021-07-31T03:04:54.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:53 smithi045 conmon[39718]: .24542) 3330 : cluster [DBG] pgmap v4337: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:54.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:53 smithi045 conmon[33655]: cluster 2021-07-31T03:04:52.910302+0000 mgr.x ( 2021-07-31T03:04:54.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:53 smithi045 conmon[33655]: mgr.24542) 3330 : cluster [DBG] pgmap v4337: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:56.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:55 smithi154 conmon[34553]: cluster 2021-07-31T03:04:54.910774+0000 mgr.x (mgr.24542) 3331 : cluster [DBG] pgmap v4338: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:56.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:55 smithi045 conmon[39718]: cluster 2021-07-31T03:04:54.910774+0000 mgr.x ( 2021-07-31T03:04:56.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:55 smithi045 conmon[39718]: mgr.24542) 3331 : cluster [DBG] pgmap v4338: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:56.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:55 smithi045 conmon[33655]: cluster 2021-07-31T03:04:54.910774+0000 mgr.x (mgr.24542) 3331 : cluster [DBG] pgmap v4338: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:58.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:57 smithi154 conmon[34553]: cluster 2021-07-31T03:04:56.911180+0000 mgr.x (mgr.24542) 3332 : cluster [DBG] pgmap v4339: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:58.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:57 smithi045 conmon[39718]: cluster 2021-07-31T03:04:56. 2021-07-31T03:04:58.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:57 smithi045 conmon[39718]: 911180+0000 mgr.x (mgr.24542) 3332 : cluster [DBG] pgmap v4339: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:04:58.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:57 smithi045 conmon[33655]: cluster 2021-07-31T03:04:56.911180+0000 mgr.x (mgr.24542) 2021-07-31T03:04:58.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:57 smithi045 conmon[33655]: 3332 : cluster [DBG] pgmap v4339: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:00.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:04:59 smithi154 conmon[34553]: cluster 2021-07-31T03:04:58.911642+0000 mgr.x (mgr.24542) 3333 : cluster [DBG] pgmap v4340: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:00.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:59 smithi045 conmon[33655]: cluster 2021-07-31T03:04:58.911642+0000 mgr.x (mgr 2021-07-31T03:05:00.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:04:59 smithi045 conmon[33655]: .24542) 3333 : cluster [DBG] pgmap v4340: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:00.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:04:59 smithi045 conmon[39718]: cluster 2021-07-31T03:04:58.911642+0000 mgr.x (mgr.24542) 3333 : cluster [DBG] pgmap v4340: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:02.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:01 smithi154 conmon[34553]: cluster 2021-07-31T03:05:00.912020+0000 mgr.x (mgr.24542) 3334 : cluster [DBG] pgmap v4341: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:02.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:01 smithi045 conmon[33655]: cluster 2021-07-31T03:05:00.912020+0000 mgr.x 2021-07-31T03:05:02.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:01 smithi045 conmon[33655]: (mgr.24542) 3334 : cluster [DBG] pgmap v4341: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:02.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:01 smithi045 conmon[39718]: cluster 2021-07-31T03:05:00.912020+0000 mgr.x (mgr 2021-07-31T03:05:02.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:01 smithi045 conmon[39718]: .24542) 3334 : cluster [DBG] pgmap v4341: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:04.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:03 smithi154 conmon[34553]: cluster 2021-07-31T03:05:02.912406+0000 mgr.x (mgr.24542) 3335 : cluster [DBG] pgmap v4342: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:04.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:03 smithi154 conmon[34553]: 2021-07-31T03:05:04.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:03 smithi045 conmon[33655]: cluster 2021-07-31T03:05:02.912406+0000 2021-07-31T03:05:04.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:03 smithi045 conmon[33655]: mgr.x (mgr.24542) 3335 : cluster [DBG] pgmap v4342: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:04.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:03 smithi045 conmon[39718]: cluster 2021-07-31T03:05:02.912406+0000 mgr.x 2021-07-31T03:05:04.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:03 smithi045 conmon[39718]: (mgr.24542) 3335 : cluster [DBG] pgmap v4342: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:06.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:05 smithi154 conmon[34553]: cluster 2021-07-31T03:05:04.912891+0000 mgr.x (mgr.24542) 3336 : cluster [DBG] pgmap v4343: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:06.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:05 smithi045 conmon[39718]: cluster 2021-07-31T03:05:04.912891+0000 mgr.x (mgr. 2021-07-31T03:05:06.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:05 smithi045 conmon[39718]: 24542) 3336 : cluster [DBG] pgmap v4343: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:06.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:05 smithi045 conmon[33655]: cluster 2021-07-31T03:05:04.912891 2021-07-31T03:05:06.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:05 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 3336 : cluster [DBG] pgmap v4343: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:08.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:07 smithi154 conmon[34553]: cluster 2021-07-31T03:05:06.913482+0000 mgr.x (mgr.24542) 3337 : cluster 2021-07-31T03:05:08.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:07 smithi154 conmon[34553]: [DBG] pgmap v4344: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:08.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:07 smithi045 conmon[39718]: cluster 2021-07-31T03:05:06.913482+0000 mgr.x 2021-07-31T03:05:08.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:07 smithi045 conmon[39718]: (mgr.24542) 3337 : cluster [DBG] pgmap v4344: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:08.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:07 smithi045 conmon[33655]: cluster 2021-07-31T03:05:06.913482+0000 2021-07-31T03:05:08.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:07 smithi045 conmon[33655]: mgr.x (mgr.24542) 3337 : cluster [DBG] pgmap v4344: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:10.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:09 smithi154 conmon[34553]: cluster 2021-07-31T03:05:08.914095+0000 mgr.x (mgr.24542) 3338 : cluster [DBG] 2021-07-31T03:05:10.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:09 smithi154 conmon[34553]: pgmap v4345: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:10.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:09 smithi045 conmon[33655]: cluster 2021-07-31T03:05:08.914095+0000 mgr.x (mgr.24542) 3338 : cluster [DBG] pgmap v4345: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:10.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:09 smithi045 conmon[39718]: cluster 2021-07-31T03:05:08.914095+0000 mgr.x (mgr.24542) 3338 : 2021-07-31T03:05:10.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:09 smithi045 conmon[39718]: cluster [DBG] pgmap v4345: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:12.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:11 smithi154 conmon[34553]: cluster 2021-07-31T03:05:10.914586+0000 mgr.x (mgr.24542) 3339 : cluster [DBG] 2021-07-31T03:05:12.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:11 smithi154 conmon[34553]: pgmap v4346: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:12.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:11 smithi045 conmon[33655]: cluster 2021-07-31T03:05:10.914586+0000 mgr.x (mgr.24542) 3339 : cluster [DBG] pgmap v4346: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:12.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:11 smithi045 conmon[39718]: cluster 2021-07-31T03:05:10.914586+0000 mgr.x ( 2021-07-31T03:05:12.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:11 smithi045 conmon[39718]: mgr.24542) 3339 : cluster [DBG] pgmap v4346: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:14.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:13 smithi154 conmon[34553]: cluster 2021-07-31T03:05:12.914998+0000 mgr.x (mgr.24542) 3340 : cluster [DBG] 2021-07-31T03:05:14.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:13 smithi154 conmon[34553]: pgmap v4347: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:14.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:13 smithi045 conmon[33655]: cluster 2021-07-31T03:05:12.914998+0000 mgr.x (mgr 2021-07-31T03:05:14.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:13 smithi045 conmon[33655]: .24542) 3340 : cluster [DBG] pgmap v4347: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:14.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:13 smithi045 conmon[39718]: cluster 2021-07-31T03:05:12.914998+0000 mgr.x ( 2021-07-31T03:05:14.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:13 smithi045 conmon[39718]: mgr.24542) 3340 : cluster [DBG] pgmap v4347: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:16.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:15 smithi154 conmon[34553]: cluster 2021-07-31T03:05:14.915621+0000 mgr.x (mgr.24542) 3341 : cluster [DBG] pgmap v4348: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:16.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:15 smithi154 conmon[34553]: 2021-07-31T03:05:16.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:15 smithi045 conmon[33655]: cluster 2021-07-31T03:05:14.915621+0000 mgr.x (mgr.24542 2021-07-31T03:05:16.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:15 smithi045 conmon[33655]: ) 3341 : cluster [DBG] pgmap v4348: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:16.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:15 smithi045 conmon[39718]: cluster 2021-07-31T03:05:14.915621+0000 mgr.x ( 2021-07-31T03:05:16.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:15 smithi045 conmon[39718]: mgr.24542) 3341 : cluster [DBG] pgmap v4348: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:17.737 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:05:18.019 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:05:18.019 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 80m ago 80m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:05:18.020 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 77m ago 80m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:05:18.020 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 77m ago 84m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:05:18.020 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 80m ago 86m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:05:18.020 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 80m ago 86m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:05:18.020 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 77m ago 84m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:05:18.021 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 80m ago 84m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:05:18.021 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 80m ago 81m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:05:18.021 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 77m ago 81m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:05:18.021 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 80m ago 83m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:05:18.022 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 80m ago 83m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:05:18.022 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 80m ago 83m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:05:18.022 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 80m ago 82m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:05:18.022 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 77m ago 82m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:05:18.022 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 77m ago 82m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:05:18.023 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 77m ago 81m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:05:18.023 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 77m ago 81m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:05:18.023 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 77m ago 81m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:05:18.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:17 smithi154 conmon[34553]: cluster 2021-07-31T03:05:16.916106+0000 mgr.x (mgr.24542) 3342 : cluster [DBG] pgmap v4349: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:18.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:17 smithi154 conmon[34553]: audit 2021-07-31T03:05:17.725887+0000 mgr.x (mgr.24542) 3343 : audit [DBG] from='client.18768 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:05:18.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:17 smithi045 conmon[33655]: cluster 2021-07- 2021-07-31T03:05:18.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:17 smithi045 conmon[33655]: 31T03:05:16.916106+0000 mgr.x (mgr.24542) 3342 : cluster [DBG] pgmap v4349: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:18.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:17 smithi045 conmon[33655]: audit 2021-07-31T03:05:18.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:17 smithi045 conmon[33655]: 2021-07-31T 2021-07-31T03:05:18.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:17 smithi045 conmon[33655]: 03:05:17 2021-07-31T03:05:18.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:17 smithi045 conmon[33655]: .725887+0000 mgr.x (mgr.24542) 2021-07-31T03:05:18.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:17 smithi045 conmon[33655]: 3343 : audit [DBG] from='client.18768 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:05:18.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:17 smithi045 conmon[39718]: cluster 2021-07-31T03:05:16. 2021-07-31T03:05:18.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:17 smithi045 conmon[39718]: 916106+0000 mgr.x (mgr.24542) 3342 : cluster [DBG] pgmap v4349: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:18.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:17 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T03:05:18.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:17 smithi045 conmon[39718]: 03:05:17.725887 2021-07-31T03:05:18.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:17 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 2021-07-31T03:05:18.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:17 smithi045 conmon[39718]: 3343 : audit [DBG] from='client.18768 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:05:18.354 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:05:18.354 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:05:18.355 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:05:18.355 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:05:18.355 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:05:18.355 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:05:18.356 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:05:18.356 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:05:18.356 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:05:18.356 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:05:18.356 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:05:18.357 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:05:18.357 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:05:18.357 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:05:18.357 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:05:18.357 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:05:18.358 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:05:19.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:18 smithi154 conmon[34553]: audit 2021-07-31T03:05:18.011761+0000 mgr.x (mgr.24542) 3344 : audit [DBG] 2021-07-31T03:05:19.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:18 smithi154 conmon[34553]: from='client.28714 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:05:19.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:18 smithi154 conmon[34553]: audit 2021-07-31T03:05:18.354968+0000 mon.c (mon.1) 173 : audit [DBG] from='client.? 172.21.15.45:0/3456940433' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:05:19.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:18 smithi045 conmon[33655]: audit 2021-07-31T03:05:18.011761+0000 mgr.x (mgr.24542) 3344 : audit [DBG] from='client.28714 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:05:19.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:18 smithi045 conmon[33655]: audit 2021-07-31T03:05:18.354968+0000 mon.c (mon.1) 173 : audit [DBG] 2021-07-31T03:05:19.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:18 smithi045 conmon[33655]: from='client.? 172.21.15.45:0/3456940433' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:05:19.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:18 smithi045 conmon[39718]: audit 2021-07-31T03:05:18.011761+0000 mgr.x (mgr.24542) 3344 : audit [DBG] from='client.28714 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:05:19.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:18 smithi045 conmon[39718]: audit 2021-07-31T03:05:18.354968+0000 mon.c (mon.1) 173 : audit [DBG] from='client.? 172.21.15.45:0/3456940433' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:05:20.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:19 smithi154 conmon[34553]: cluster 2021-07-31T03:05:18.916495+0000 mgr.x (mgr.24542) 3345 : cluster [DBG] pgmap v4350: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:20.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:19 smithi045 conmon[33655]: cluster 2021-07-31T03:05:18.916495 2021-07-31T03:05:20.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:19 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 3345 : cluster [DBG] pgmap v4350: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:20.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:19 smithi045 conmon[39718]: cluster 2021-07-31T03:05:18.916495+0000 mgr.x (mgr.24542) 3345 : cluster [DBG] pgmap v4350: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:22.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:21 smithi154 conmon[34553]: cluster 2021-07-31T03:05:20.916892+0000 mgr.x (mgr.24542) 3346 : cluster [DBG] pgmap v4351: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:22.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:21 smithi045 conmon[39718]: cluster 2021-07-31T03:05:20.916892 2021-07-31T03:05:22.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:21 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 3346 : cluster [DBG] pgmap v4351: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:22.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:21 smithi045 conmon[33655]: cluster 2021-07-31T03:05:20.916892+0000 mgr.x (mgr.24542) 3346 : cluster 2021-07-31T03:05:22.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:21 smithi045 conmon[33655]: [DBG] pgmap v4351: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:24.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:23 smithi154 conmon[34553]: cluster 2021-07-31T03:05:22.917253+0000 mgr.x (mgr.24542) 3347 : cluster [DBG] pgmap v4352: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:24.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:23 smithi045 conmon[39718]: cluster 2021-07-31T03:05:22.917253+0000 mgr.x 2021-07-31T03:05:24.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:23 smithi045 conmon[39718]: (mgr.24542) 3347 : cluster [DBG] pgmap v4352: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:24.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:23 smithi045 conmon[33655]: cluster 2021-07-31T03:05:22.917253+0000 2021-07-31T03:05:24.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:23 smithi045 conmon[33655]: mgr.x (mgr.24542) 3347 : cluster [DBG] pgmap v4352: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:26.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:25 smithi154 conmon[34553]: cluster 2021-07-31T03:05:24.917666+0000 mgr.x (mgr.24542) 3348 : cluster [DBG] pgmap v4353: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:26.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:25 smithi045 conmon[39718]: cluster 2021-07-31T03:05: 2021-07-31T03:05:26.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:25 smithi045 conmon[39718]: 24.917666+0000 mgr.x (mgr.24542) 3348 : cluster [DBG] pgmap v4353: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:26.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:25 smithi045 conmon[33655]: cluster 2021-07-31T03:05:24.917666+0000 mgr.x (mgr.24542) 3348 2021-07-31T03:05:26.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:25 smithi045 conmon[33655]: : cluster [DBG] pgmap v4353: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:28.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:27 smithi154 conmon[34553]: cluster 2021-07-31T03:05:26.918095+0000 mgr.x (mgr.24542) 3349 : cluster [DBG] pgmap v4354: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:28.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:27 smithi045 conmon[39718]: cluster 2021-07-31T03:05:26.918095+0000 mgr.x ( 2021-07-31T03:05:28.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:27 smithi045 conmon[39718]: mgr.24542) 3349 : cluster [DBG] pgmap v4354: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:28.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:27 smithi045 conmon[33655]: cluster 2021-07-31T03:05:26.918095+0000 mgr.x (mgr.24542) 3349 : cluster 2021-07-31T03:05:28.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:27 smithi045 conmon[33655]: [DBG] pgmap v4354: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:29.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:28 smithi154 conmon[34553]: audit 2021-07-31T03:05:28.244686+0000 mon.b (mon.2) 465 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:05:29.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:28 smithi154 conmon[34553]: audit 2021-07-31T03:05:28.244953+0000 mon.b 2021-07-31T03:05:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:28 smithi154 conmon[34553]: (mon.2) 466 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:05:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:28 smithi154 conmon[34553]: audit 2021-07-31T03:05:28.245461+0000 mon.a (mon.0) 3250 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:05:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:28 smithi154 conmon[34553]: audit 2021-07-31T03:05:28.245859+0000 mon.a (mon.0) 3251 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:05:29.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:28 smithi045 conmon[39718]: audit 2021-07-31T03:05:28.244686+0000 mon.b (mon.2) 465 2021-07-31T03:05:29.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:28 smithi045 conmon[39718]: : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:05:29.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:28 smithi045 conmon[39718]: audit 2021-07-31T03:05:28.244953+0000 mon.b (mon.2) 466 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:05:29.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:28 smithi045 conmon[39718]: 2021-07-31T03:05:29.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:28 smithi045 conmon[39718]: audit 2021-07-31T03:05:28.245461+0000 mon.a 2021-07-31T03:05:29.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:28 smithi045 conmon[39718]: (mon.0) 3250 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:05:29.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:28 smithi045 conmon[39718]: audit 2021-07-31T03:05:28.245859+0000 2021-07-31T03:05:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:28 smithi045 conmon[39718]: mon.a (mon.0) 3251 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:05:29.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:28 smithi045 conmon[33655]: audit 2021-07-31T03:05:28.244686+0000 mon.b (mon.2) 465 : audit [INF] 2021-07-31T03:05:29.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:28 smithi045 conmon[33655]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:05:29.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:28 smithi045 conmon[33655]: audit 2021-07-31T03:05:28.244953+0000 mon.b (mon.2) 466 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:05:29.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:28 smithi045 conmon[33655]: audit 2021-07-31T03:05:29.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:28 smithi045 conmon[33655]: 2021-07-31T03:05:28.245461+0000 mon.a (mon.0) 3250 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:05:29.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:28 smithi045 conmon[33655]: audit 2021-07-31T03 2021-07-31T03:05:29.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:28 smithi045 conmon[33655]: :05:28.245859+0000 mon.a (mon.0) 3251 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:05:30.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:29 smithi154 conmon[34553]: cluster 2021-07-31T03:05:28.918506+0000 mgr.x (mgr.24542) 3350 : cluster [DBG] pgmap v4355: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:30.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:29 smithi045 conmon[39718]: cluster 2021-07-31T03:05:28.918506+0000 mgr.x (mgr. 2021-07-31T03:05:30.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:29 smithi045 conmon[39718]: 24542) 3350 : cluster [DBG] pgmap v4355: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:30.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:29 smithi045 conmon[33655]: cluster 2021-07-31T03:05:28.918506+0000 mgr.x (mgr.24542) 3350 : 2021-07-31T03:05:30.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:29 smithi045 conmon[33655]: cluster [DBG] pgmap v4355: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:31.502 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: debug 2021-07-31T03:05:31.234+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:05:31.503 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:05:31.503 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:05:31.503 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: yield 2021-07-31T03:05:31.503 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:05:31.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:05:31.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:05:31.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:05:31.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:05:31.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:05:31.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:05:31.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:05:31.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:05:31.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: task.result() 2021-07-31T03:05:31.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:05:31.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: debug 2021-07-31T03:05:31.246+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:05:31.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:05:31.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:05:31.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: yield 2021-07-31T03:05:31.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:05:31.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:05:31.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:05:31.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:05:31.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:05:31.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:05:31.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:05:31.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:05:31.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:05:31.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: task.result() 2021-07-31T03:05:31.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:05:31.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: debug 2021-07-31T03:05:31.271+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:05:31.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:05:31.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:05:31.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: yield 2021-07-31T03:05:31.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:05:31.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:05:31.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:05:31.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:05:31.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:05:31.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:05:31.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:05:31.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:05:31.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:05:31.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: task.result() 2021-07-31T03:05:31.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:05:31.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: debug 2021-07-31T03:05:31.271+0000 7fcb9095b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:05:31.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:05:31.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:05:31.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: yield 2021-07-31T03:05:31.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:05:31.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:05:31.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:05:31.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:05:31.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:05:31.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:05:31.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:05:31.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:05:31.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:05:31.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: task.result() 2021-07-31T03:05:31.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:05:31.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: 2021-07-31T03:05:31.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:05:31.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: 2021-07-31T03:05:31.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:05:31.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:05:31.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:05:31.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:05:31.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:05:31.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:05:31.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:05:31.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:05:31.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:05:31.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:05:31.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:05:31.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:05:31.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:05:31.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:05:31.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:05:31.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:05:31.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:05:31.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:05:31.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: raise 2021-07-31T03:05:31.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:05:31.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:05:31.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:05:31.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:05:31.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:05:31.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: debug 2021-07-31T03:05:31.280+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:05:31.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:05:31.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:05:31.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: yield 2021-07-31T03:05:31.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:05:31.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:05:31.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:05:31.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:05:31.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:05:31.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:05:31.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:05:31.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:05:31.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:05:31.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: task.result() 2021-07-31T03:05:31.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:05:31.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: debug 2021-07-31T03:05:31.280+0000 7fcb9115c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:05:31.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:05:31.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:05:31.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: yield 2021-07-31T03:05:31.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:05:31.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:05:31.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:05:31.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:05:31.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:05:31.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:05:31.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:05:31.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:05:31.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:05:31.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: task.result() 2021-07-31T03:05:31.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:05:31.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: 2021-07-31T03:05:31.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:05:31.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: 2021-07-31T03:05:31.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:05:31.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:05:31.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:05:31.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:05:31.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:05:31.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:05:31.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:05:31.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:05:31.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:05:31.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:05:31.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:05:31.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:05:31.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:05:31.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:05:31.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:05:31.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:05:31.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:05:31.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:05:31.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: raise 2021-07-31T03:05:31.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:05:31.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:05:31.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:05:31.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:05:31.535 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:05:32.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: cluster 2021-07-31T03:05:30.919055+0000 mgr.x (mgr.24542) 3351 : cluster [DBG] pgmap v4356: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:32.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: audit 2021-07-31T03:05:31 2021-07-31T03:05:32.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: .189112+0000 mon.b (mon.2) 467 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:05:32.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: audit 2021-07-31T03:05:31.190538+0000 mon.b (mon.2) 468 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:05:32.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: audit 2021-07-31T 2021-07-31T03:05:32.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: 03:05:31.191428+0000 mon.b (mon.2) 469 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:05:32.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: cephadm 2021-07- 2021-07-31T03:05:32.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: 31T03:05:31.192230+0000 mgr.x ( 2021-07-31T03:05:32.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: mgr.24542) 3352 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:05:32.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: cephadm 2021-07-31T03 2021-07-31T03:05:32.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: :05:31.235504+0000 mgr.x (mgr.24542) 3353 : cephadm 2021-07-31T03:05:32.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: [ERR] 2021-07-31T03:05:32.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:05:32.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:05:32.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: yield 2021-07-31T03:05:32.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:05:32.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:05:32.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:05:32.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:05:32.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:05:32.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:05:32.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:05:32.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:05:32.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:05:32.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: task.result() 2021-07-31T03:05:32.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:05:32.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: cephadm 2021-07-31T03:05:31.235892+0000 mgr.x (mgr.24542) 3354 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:05:32.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: cephadm 2021-07-31T03:05:31.246930+0000 mgr.x (mgr.24542) 3355 : cephadm [ERR] 2021-07-31T03:05:32.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:05:32.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:05:32.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: yield 2021-07-31T03:05:32.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:05:32.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:05:32.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:05:32.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:05:32.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:05:32.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:05:32.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:05:32.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:05:32.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:05:32.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: task.result() 2021-07-31T03:05:32.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:05:32.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: cephadm 2021-07-31T03:05:32.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: 2021-07-31T03:05:31.247963+0000 mgr.x (mgr.24542) 3356 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:05:32.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: cephadm 2021-07-31T03:05:31.272448+0000 mgr.x (mgr.24542) 3357 : cephadm [ERR] 2021-07-31T03:05:32.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:05:32.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:05:32.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: yield 2021-07-31T03:05:32.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:05:32.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:05:32.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:05:32.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:05:32.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:05:32.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:05:32.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:05:32.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:05:32.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:05:32.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: task.result() 2021-07-31T03:05:32.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:05:32.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: cephadm 2021-07-31T03:05:31.272854 2021-07-31T03:05:32.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: +0000 mgr.x (mgr.24542) 3358 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:05:32.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:05:32.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:05:32.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: yield 2021-07-31T03:05:32.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:05:32.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:05:32.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:05:32.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:05:32.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:05:32.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:05:32.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:05:32.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:05:32.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:05:32.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: task.result() 2021-07-31T03:05:32.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:05:32.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: 2021-07-31T03:05:32.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:05:32.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: 2021-07-31T03:05:32.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:05:32.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:05:32.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:05:32.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:05:32.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:05:32.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:05:32.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:05:32.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:05:32.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:05:32.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:05:32.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:05:32.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:05:32.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:05:32.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:05:32.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:05:32.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:05:32.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:05:32.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:05:32.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: raise 2021-07-31T03:05:32.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:05:32.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:05:32.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:05:32.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:05:32.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:05:32.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: cephadm 2021-07-31T03:05:31.281289+0000 mgr.x (mgr.24542) 3359 : cephadm [ERR] 2021-07-31T03:05:32.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:05:32.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:05:32.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: yield 2021-07-31T03:05:32.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:05:32.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:05:32.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:05:32.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:05:32.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:05:32.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:05:32.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:05:32.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:05:32.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:05:32.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: task.result() 2021-07-31T03:05:32.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:05:32.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: cephadm 2021- 2021-07-31T03:05:32.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: 07-31T03:05:31.281843+0000 mgr.x (mgr.24542) 3360 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:05:32.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:05:32.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:05:32.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: yield 2021-07-31T03:05:32.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:05:32.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:05:32.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:05:32.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:05:32.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:05:32.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:05:32.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:05:32.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:05:32.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:05:32.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: task.result() 2021-07-31T03:05:32.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:05:32.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: 2021-07-31T03:05:32.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:05:32.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: 2021-07-31T03:05:32.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:05:32.285 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:05:32.285 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:05:32.285 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:05:32.285 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:05:32.285 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:05:32.286 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:05:32.286 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:05:32.286 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:05:32.286 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:05:32.287 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:05:32.287 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:05:32.287 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:05:32.287 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:05:32.287 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:05:32.288 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:05:32.288 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:05:32.288 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:05:32.288 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: raise 2021-07-31T03:05:32.288 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:05:32.289 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:05:32.289 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:05:32.289 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:05:32.289 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:31 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:05:32.290 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: cluster 2021-07-31T03:05:30.919055+0000 mgr.x (mgr.24542) 3351 : cluster [DBG] pgmap v4356: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:32.290 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: audit 2021-07-31T03:05:31.189112+0000 mon.b (mon.2) 467 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:05:32.290 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: audit 2021-07-31T03:05:32.291 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: 2021-07-31T03:05:31.190538+0000 mon.b (mon.2) 468 : audit [DBG] 2021-07-31T03:05:32.291 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:05:32.291 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: audit 2021-07-31T03:05:31.191428+0000 mon.b ( 2021-07-31T03:05:32.291 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: mon.2) 469 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:05:32.292 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: cephadm 2021-07-31T03:05:31.192230 2021-07-31T03:05:32.292 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 3352 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:05:32.292 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: cephadm 2021-07- 2021-07-31T03:05:32.292 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: 31T03:05:31.235504+0000 mgr.x (mgr.24542) 3353 : cephadm [ERR] 2021-07-31T03:05:32.292 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:05:32.293 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:05:32.293 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: yield 2021-07-31T03:05:32.293 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:05:32.293 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:05:32.293 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:05:32.294 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:05:32.294 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:05:32.294 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:05:32.294 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:05:32.295 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:05:32.295 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:05:32.295 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: task.result() 2021-07-31T03:05:32.295 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:05:32.295 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: cephadm 2021-07-31T03:05:31.235892+0000 mgr.x (mgr.24542) 3354 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:05:32.296 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: cephadm 2021-07-31T03:05:31.246930+0000 mgr.x (mgr.24542) 3355 : cephadm [ERR] 2021-07-31T03:05:32.296 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:05:32.296 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:05:32.296 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: yield 2021-07-31T03:05:32.297 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:05:32.297 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:05:32.297 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:05:32.297 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:05:32.297 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:05:32.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: cluster 2021-07-31T03:05:30.919055+0000 mgr.x (mgr.24542) 3351 : cluster [DBG] 2021-07-31T03:05:32.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: pgmap v4356: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:32.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: audit 2021-07-31T03:05:31.189112+0000 mon.b (mon.2) 467 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:05:32.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: audit 2021-07-31T03:05:31.190538+0000 mon.b (mon.2) 468 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:05:32.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: audit 2021-07-31T03:05: 2021-07-31T03:05:32.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: 31.191428+0000 mon.b (mon.2) 469 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:05:32.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: cephadm 2021-07-31T03:05:31 2021-07-31T03:05:32.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: .192230+0000 mgr.x (mgr.24542) 3352 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:05:32.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: cephadm 2021-07-31T03:05 2021-07-31T03:05:32.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: :31.235504+0000 mgr.x (mgr.24542) 3353 : cephadm [ERR] 2021-07-31T03:05:32.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:05:32.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:05:32.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: yield 2021-07-31T03:05:32.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:05:32.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:05:32.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:05:32.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:05:32.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:05:32.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:05:32.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:05:32.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:05:32.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:05:32.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: task.result() 2021-07-31T03:05:32.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:05:32.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: cephadm 2021-07-31T03:05:31.235892+0000 mgr.x (mgr.24542) 3354 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:05:32.304 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: cephadm 2021-07-31T03:05:31.246930+0000 mgr.x (mgr.24542) 3355 : cephadm [ERR] 2021-07-31T03:05:32.304 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:05:32.304 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:05:32.304 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: yield 2021-07-31T03:05:32.304 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:05:32.305 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:05:32.305 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:05:32.305 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:05:32.305 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:05:32.306 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:05:32.306 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:05:32.306 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:05:32.306 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:05:32.307 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: task.result() 2021-07-31T03:05:32.307 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:05:32.307 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: cephadm 2021-07-31T03:05:31.247963+0000 mgr.x (mgr.24542) 3356 2021-07-31T03:05:32.307 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:05:32.308 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: cephadm 2021-07-31T03:05:31.272448+0000 mgr.x (mgr.24542) 3357 : cephadm [ERR] 2021-07-31T03:05:32.308 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:05:32.308 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:05:32.308 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: yield 2021-07-31T03:05:32.308 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:05:32.309 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:05:32.309 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:05:32.309 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:05:32.309 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:05:32.310 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:05:32.310 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:05:32.310 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:05:32.310 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:05:32.310 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: task.result() 2021-07-31T03:05:32.311 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:05:32.311 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: cephadm 2021-07-31T03:05:31.272854+0000 mgr.x (mgr.24542) 3358 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:05:32.311 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:05:32.311 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:05:32.311 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: yield 2021-07-31T03:05:32.312 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:05:32.312 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:05:32.312 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:05:32.312 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:05:32.313 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:05:32.313 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:05:32.313 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:05:32.313 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:05:32.313 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:05:32.314 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: task.result() 2021-07-31T03:05:32.314 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:05:32.314 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: 2021-07-31T03:05:32.314 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:05:32.314 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: 2021-07-31T03:05:32.315 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:05:32.315 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:05:32.315 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:05:32.316 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:05:32.316 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: task.result() 2021-07-31T03:05:32.316 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:05:32.316 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: cephadm 2021-07-31T03:05:31.247963+0000 2021-07-31T03:05:32.316 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: mgr.x (mgr.24542) 3356 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:05:32.317 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: cephadm 2021-07-31T03:05:31.272448+0000 mgr.x (mgr.24542) 3357 : cephadm [ERR] 2021-07-31T03:05:32.317 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:05:32.317 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:05:32.317 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: yield 2021-07-31T03:05:32.318 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:05:32.318 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:05:32.318 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:05:32.318 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:05:32.318 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:05:32.319 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:05:32.319 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:05:32.319 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:05:32.319 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:05:32.320 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: task.result() 2021-07-31T03:05:32.320 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:05:32.320 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: cephadm 2021-07-31T03: 2021-07-31T03:05:32.320 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: 05:31.272854+0000 mgr.x (mgr.24542) 3358 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:05:32.320 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:05:32.321 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:05:32.321 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: yield 2021-07-31T03:05:32.321 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:05:32.321 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:05:32.321 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:05:32.322 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:05:32.322 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:05:32.322 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:05:32.322 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:05:32.323 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:05:32.323 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:05:32.323 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: task.result() 2021-07-31T03:05:32.323 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:05:32.323 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: 2021-07-31T03:05:32.324 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:05:32.324 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: 2021-07-31T03:05:32.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:05:32.325 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:05:32.325 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:05:32.325 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:05:32.325 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:05:32.326 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:05:32.326 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:05:32.326 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:05:32.326 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:05:32.326 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:05:32.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:05:32.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:05:32.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:05:32.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:05:32.328 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:05:32.328 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:05:32.328 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:05:32.328 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:05:32.328 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: raise 2021-07-31T03:05:32.329 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:05:32.329 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:05:32.329 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:05:32.329 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:05:32.329 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:05:32.330 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: cephadm 2021-07-31T03:05:31.281289+0000 mgr.x (mgr.24542) 3359 : cephadm [ERR] 2021-07-31T03:05:32.330 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:05:32.330 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:05:32.330 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: yield 2021-07-31T03:05:32.330 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:05:32.331 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:05:32.331 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:05:32.331 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:05:32.331 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:05:32.332 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:05:32.332 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:05:32.332 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:05:32.332 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:05:32.332 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: task.result() 2021-07-31T03:05:32.333 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:05:32.333 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:05:32.333 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:05:32.333 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:05:32.334 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:05:32.334 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:05:32.334 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:05:32.334 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:05:32.335 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:05:32.335 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:05:32.335 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:05:32.335 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:05:32.335 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:05:32.336 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:05:32.336 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:05:32.336 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:05:32.336 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:05:32.337 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:05:32.337 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:05:32.337 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: raise 2021-07-31T03:05:32.337 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:05:32.337 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:05:32.338 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:05:32.338 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:05:32.338 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:05:32.338 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: cephadm 2021-07-31T03:05:31.281289+0000 mgr.x (mgr.24542) 3359 : cephadm [ERR] 2021-07-31T03:05:32.339 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:05:32.339 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:05:32.339 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: yield 2021-07-31T03:05:32.339 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:05:32.339 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:05:32.340 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:05:32.340 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:05:32.340 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:05:32.340 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:05:32.340 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:05:32.341 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:05:32.341 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:05:32.341 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: task.result() 2021-07-31T03:05:32.341 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:05:32.342 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: cephadm 2021-07-31T03:05:31.281843+0000 mgr.x (mgr. 2021-07-31T03:05:32.342 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: cephadm 2021-07-31T03:05:31.281843+0000 mgr.x (mgr.24542) 3360 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:05:32.342 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:05:32.343 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:05:32.343 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: yield 2021-07-31T03:05:32.343 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:05:32.343 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:05:32.343 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:05:32.344 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:05:32.344 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:05:32.344 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:05:32.344 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:05:32.345 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:05:32.345 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:05:32.345 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: task.result() 2021-07-31T03:05:32.345 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:05:32.345 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: 2021-07-31T03:05:32.346 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:05:32.346 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: 2021-07-31T03:05:32.346 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:05:32.346 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:05:32.347 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:05:32.347 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:05:32.347 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:05:32.347 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:05:32.347 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:05:32.348 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:05:32.348 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:05:32.348 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:05:32.348 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:05:32.348 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:05:32.349 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:05:32.349 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:05:32.349 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:05:32.349 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:05:32.349 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:05:32.350 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:05:32.350 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: raise 2021-07-31T03:05:32.350 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:05:32.350 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:05:32.351 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:05:32.351 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:05:32.351 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:05:32.351 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: 24542) 3360 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:05:32.352 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:05:32.352 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:05:32.352 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: yield 2021-07-31T03:05:32.352 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:05:32.352 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:05:32.353 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:05:32.353 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:05:32.353 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:05:32.353 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:05:32.354 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:05:32.354 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:05:32.354 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:05:32.354 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: task.result() 2021-07-31T03:05:32.354 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:05:32.355 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: 2021-07-31T03:05:32.355 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:05:32.355 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: 2021-07-31T03:05:32.355 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:05:32.356 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:05:32.356 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:05:32.356 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:05:32.356 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:05:32.356 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:05:32.357 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:05:32.357 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:05:32.357 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:05:32.357 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:05:32.358 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:05:32.358 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:05:32.358 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:05:32.358 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:05:32.359 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:05:32.359 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:05:32.359 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:05:32.359 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:05:32.359 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: raise 2021-07-31T03:05:32.360 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:05:32.360 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:05:32.360 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:05:32.360 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:05:32.360 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:31 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:05:34.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:33 smithi154 conmon[34553]: cluster 2021-07-31T03:05:32.919414+0000 mgr.x (mgr.24542) 3361 : cluster [DBG] pgmap v4357: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:34.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:33 smithi045 conmon[39718]: cluster 2021-07-31T03:05:32.919414+0000 2021-07-31T03:05:34.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:33 smithi045 conmon[39718]: mgr.x (mgr.24542) 3361 : cluster [DBG] pgmap v4357: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:34.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:33 smithi045 conmon[33655]: cluster 2021-07-31T03:05:32.919414+0000 mgr.x (mgr.24542) 3361 2021-07-31T03:05:34.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:33 smithi045 conmon[33655]: : cluster [DBG] pgmap v4357: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:36.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:35 smithi154 conmon[34553]: cluster 2021-07-31T03:05:34.919853+0000 mgr.x (mgr.24542) 3362 : cluster [DBG] pgmap v4358: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:36.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:35 smithi045 conmon[39718]: cluster 2021-07-31T03:05:34.919853+0000 mgr.x (mgr.24542) 3362 : cluster [DBG] pgmap v4358: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:36.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:35 smithi045 conmon[33655]: cluster 2021-07-31T03:05:34.919853+0000 mgr.x (mgr.24542) 3362 : cluster 2021-07-31T03:05:36.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:35 smithi045 conmon[33655]: [DBG] pgmap v4358: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:38.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:37 smithi154 conmon[34553]: cluster 2021-07-31T03:05:36.920405+0000 mgr.x (mgr.24542) 3363 : cluster [DBG] pgmap v4359: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:38.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:37 smithi045 conmon[33655]: cluster 2021-07-31T03:05:36.920405+0000 mgr.x (mgr.24542) 3363 : cluster [DBG] pgmap v4359: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:38.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:37 smithi045 conmon[39718]: cluster 2021-07-31T03:05:36.920405+0000 2021-07-31T03:05:38.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:37 smithi045 conmon[39718]: mgr.x (mgr.24542) 3363 : cluster [DBG] pgmap v4359: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:40.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:39 smithi154 conmon[34553]: cluster 2021-07-31T03:05:38.921057+0000 mgr.x (mgr.24542) 3364 : cluster [DBG] pgmap v4360: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:40.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:39 smithi045 conmon[33655]: cluster 2021-07-31T03:05:38.921057+0000 mgr.x (mgr.24542) 3364 2021-07-31T03:05:40.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:39 smithi045 conmon[33655]: : cluster [DBG] pgmap v4360: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:40.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:39 smithi045 conmon[39718]: cluster 2021-07-31T03:05:38.921057+0000 mgr.x (mgr.24542 2021-07-31T03:05:40.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:39 smithi045 conmon[39718]: ) 3364 : cluster [DBG] pgmap v4360: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:42.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:41 smithi154 conmon[34553]: cluster 2021-07-31T03:05:40.921445+0000 mgr.x (mgr.24542) 3365 : cluster [DBG] pgmap v4361: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:42.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:41 smithi045 conmon[39718]: cluster 2021-07-31T03:05:40.921445+0000 mgr.x 2021-07-31T03:05:42.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:41 smithi045 conmon[39718]: (mgr.24542) 3365 : cluster [DBG] pgmap v4361: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:42.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:41 smithi045 conmon[33655]: cluster 2021-07-31T03:05:40.921445+0000 mgr.x (mgr 2021-07-31T03:05:42.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:41 smithi045 conmon[33655]: .24542) 3365 : cluster [DBG] pgmap v4361: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:44.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:43 smithi154 conmon[34553]: cluster 2021-07-31T03:05:42.921816+0000 mgr.x (mgr.24542) 3366 : cluster [DBG] pgmap v4362: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:44.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:43 smithi045 conmon[39718]: cluster 2021-07-31T03:05:42.921816+0000 mgr.x 2021-07-31T03:05:44.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:43 smithi045 conmon[39718]: (mgr.24542) 3366 : cluster [DBG] pgmap v4362: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:44.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:43 smithi045 conmon[33655]: cluster 2021-07-31T03:05:42.921816+0000 mgr.x (mgr.24542) 3366 : cluster [DBG] pgmap v4362: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:46.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:45 smithi154 conmon[34553]: cluster 2021-07-31T03:05:44.922265+0000 mgr.x (mgr.24542) 3367 : cluster [DBG] pgmap v4363: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:46.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:45 smithi045 conmon[39718]: cluster 2021-07-31T03:05:44.922265+0000 mgr.x (mgr.24542) 2021-07-31T03:05:46.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:45 smithi045 conmon[39718]: 3367 : cluster [DBG] pgmap v4363: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:46.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:45 smithi045 conmon[33655]: cluster 2021-07-31T03:05:44.922265+0000 mgr.x (mgr.24542) 3367 : cluster [DBG] 2021-07-31T03:05:46.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:45 smithi045 conmon[33655]: pgmap v4363: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:48.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:47 smithi154 conmon[34553]: cluster 2021-07-31T03:05:46.922666+0000 mgr.x (mgr.24542) 3368 : cluster [DBG] pgmap v4364: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:48.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:47 smithi045 conmon[33655]: cluster 2021-07-31T03:05:46.922666+0000 mgr.x (mgr.24542) 3368 : cluster [DBG] pgmap v4364: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:48.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:47 smithi045 conmon[39718]: cluster 2021-07-31T03:05:46.922666+0000 mgr.x (mgr.24542) 3368 : cluster 2021-07-31T03:05:48.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:47 smithi045 conmon[39718]: [DBG] pgmap v4364: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:48.667 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:05:48.950 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:05:48.950 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 80m ago 81m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:05:48.951 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 78m ago 81m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:05:48.951 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 78m ago 84m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:05:48.951 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 80m ago 86m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:05:48.951 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 80m ago 86m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:05:48.951 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 78m ago 84m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:05:48.952 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 80m ago 85m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:05:48.952 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 80m ago 81m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:05:48.952 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 78m ago 81m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:05:48.952 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 80m ago 84m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:05:48.952 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 80m ago 84m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:05:48.953 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 80m ago 83m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:05:48.953 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 80m ago 83m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:05:48.953 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 78m ago 83m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:05:48.953 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 78m ago 82m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:05:48.954 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 78m ago 82m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:05:48.954 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 78m ago 82m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:05:48.954 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 78m ago 81m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:05:49.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:48 smithi154 conmon[34553]: audit 2021-07-31T03:05:48.656514+0000 mgr.x (mgr.24542) 3369 : audit [DBG] from='client.28726 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:05:49.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:48 smithi045 conmon[33655]: audit 2021-07-31T03:05:48.656514+0000 mgr.x ( 2021-07-31T03:05:49.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:48 smithi045 conmon[33655]: mgr.24542) 3369 : audit [DBG] from='client.28726 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:05:49.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:48 smithi045 conmon[39718]: audit 2021-07-31T03:05:48.656514+0000 mgr.x ( 2021-07-31T03:05:49.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:48 smithi045 conmon[39718]: mgr.24542) 3369 : audit [DBG] from='client.28726 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:05:49.282 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:05:49.282 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:05:49.283 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:05:49.283 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:05:49.283 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:05:49.283 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:05:49.284 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:05:49.284 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:05:49.284 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:05:49.284 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:05:49.284 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:05:49.285 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:05:49.285 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:05:49.285 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:05:49.285 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:05:49.285 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:05:49.286 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:05:50.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:49 smithi154 conmon[34553]: cluster 2021-07-31T03:05:48.923167+0000 mgr.x (mgr.24542) 3370 : cluster [DBG] 2021-07-31T03:05:50.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:49 smithi154 conmon[34553]: pgmap v4365: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:50.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:49 smithi154 conmon[34553]: audit 2021-07-31T03:05:48.942561+0000 mgr.x (mgr.24542) 3371 : audit [DBG] from='client.28732 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:05:50.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:49 smithi154 conmon[34553]: audit 2021-07-31 2021-07-31T03:05:50.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:49 smithi154 conmon[34553]: T03:05:49.283216+0000 mon.c (mon.1) 174 : audit [DBG] from='client.? 172.21.15.45:0/3054482508' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:05:50.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:49 smithi045 conmon[39718]: cluster 2021-07-31T03:05:48.923167+0000 mgr.x ( 2021-07-31T03:05:50.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:49 smithi045 conmon[39718]: mgr.24542) 3370 : cluster [DBG] pgmap v4365: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:50.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:49 smithi045 conmon[39718]: audit 2021-07-31T03:05:50.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:49 smithi045 conmon[39718]: 2021-07-31T03 2021-07-31T03:05:50.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:49 smithi045 conmon[39718]: :05:48.942561+0000 mgr.x (mgr 2021-07-31T03:05:50.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:49 smithi045 conmon[39718]: .24542) 3371 : audit [DBG] from='client.28732 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:05:50.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:49 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T03:05:50.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:49 smithi045 conmon[39718]: 31T03:05:49.283216+0000 2021-07-31T03:05:50.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:49 smithi045 conmon[39718]: mon.c (mon.1) 174 : audit [DBG] 2021-07-31T03:05:50.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:49 smithi045 conmon[39718]: from='client.? 172.21.15.45:0/3054482508' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:05:50.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:49 smithi045 conmon[33655]: cluster 2021-07-31T03:05:48.923167+0000 mgr.x (mgr.24542 2021-07-31T03:05:50.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:49 smithi045 conmon[33655]: ) 3370 : cluster [DBG] pgmap v4365: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:50.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:49 smithi045 conmon[33655]: audit 2021-07-31T03:05:48.942561+0000 2021-07-31T03:05:50.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:49 smithi045 conmon[33655]: mgr.x (mgr.24542) 3371 : audit [DBG] from='client.28732 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:05:50.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:49 smithi045 conmon[33655]: audit 2021-07 2021-07-31T03:05:50.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:49 smithi045 conmon[33655]: -31T03:05:49.283216+0000 mon.c (mon 2021-07-31T03:05:50.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:49 smithi045 conmon[33655]: .1) 174 : audit [DBG] from='client.? 172.21.15.45:0/3054482508' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:05:52.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:51 smithi154 conmon[34553]: cluster 2021-07-31T03:05:50.923705+0000 mgr.x (mgr.24542) 3372 : cluster [DBG] pgmap v4366: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:52.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:51 smithi154 conmon[34553]: 2021-07-31T03:05:52.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:51 smithi045 conmon[39718]: cluster 2021-07-31T03:05:50.923705+0000 mgr.x 2021-07-31T03:05:52.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:51 smithi045 conmon[39718]: (mgr.24542) 3372 : cluster [DBG] pgmap v4366: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:52.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:51 smithi045 conmon[33655]: cluster 2021-07-31T03:05:50.923705+0000 mgr.x (mgr.24542) 3372 : cluster [DBG] 2021-07-31T03:05:52.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:51 smithi045 conmon[33655]: pgmap v4366: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:54.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:53 smithi154 conmon[34553]: cluster 2021-07-31T03:05:52.924200+0000 mgr.x (mgr.24542) 3373 : cluster [DBG] pgmap v4367: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:54.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:53 smithi045 conmon[39718]: cluster 2021-07-31T03:05:52.924200+0000 2021-07-31T03:05:54.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:53 smithi045 conmon[39718]: mgr.x (mgr.24542) 3373 : cluster [DBG] pgmap v4367: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:54.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:53 smithi045 conmon[33655]: cluster 2021-07-31T03:05:52.924200+0000 mgr.x (mgr.24542) 2021-07-31T03:05:54.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:53 smithi045 conmon[33655]: 3373 : cluster [DBG] pgmap v4367: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:56.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:55 smithi154 conmon[34553]: cluster 2021-07-31T03:05:54.924815+0000 mgr.x (mgr.24542) 3374 : cluster [DBG] pgmap v4368: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:56.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:55 smithi045 conmon[39718]: cluster 2021-07-31T03:05:54. 2021-07-31T03:05:56.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:55 smithi045 conmon[39718]: 924815+0000 mgr.x (mgr.24542) 3374 : cluster [DBG] pgmap v4368: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:56.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:55 smithi045 conmon[33655]: cluster 2021-07-31T03:05:54.924815+0000 mgr.x (mgr.24542) 2021-07-31T03:05:56.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:55 smithi045 conmon[33655]: 3374 : cluster [DBG] pgmap v4368: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:58.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:57 smithi154 conmon[34553]: cluster 2021-07-31T03:05:56.925254+0000 mgr.x (mgr.24542) 3375 : cluster [DBG] pgmap v4369: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:58.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:57 smithi045 conmon[39718]: cluster 2021-07-31T03:05:56.925254+0000 mgr.x 2021-07-31T03:05:58.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:57 smithi045 conmon[39718]: (mgr.24542) 3375 : cluster [DBG] pgmap v4369: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:05:58.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:57 smithi045 conmon[33655]: cluster 2021-07-31T03:05:56.925254 2021-07-31T03:05:58.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:57 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 3375 : cluster [DBG] pgmap v4369: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:00.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:05:59 smithi154 conmon[34553]: cluster 2021-07-31T03:05:58.925708+0000 mgr.x (mgr.24542) 3376 : cluster [DBG] pgmap v4370: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:00.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:59 smithi045 conmon[39718]: cluster 2021-07-31T03:05:58.925708+0000 mgr.x (mgr 2021-07-31T03:06:00.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:05:59 smithi045 conmon[39718]: .24542) 3376 : cluster [DBG] pgmap v4370: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:00.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:59 smithi045 conmon[33655]: cluster 2021-07-31T03:05: 2021-07-31T03:06:00.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:05:59 smithi045 conmon[33655]: 58.925708+0000 mgr.x (mgr.24542) 3376 : cluster [DBG] pgmap v4370: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:02.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:01 smithi154 conmon[34553]: cluster 2021-07-31T03:06:00.926242+0000 mgr.x (mgr.24542) 3377 : cluster [DBG] pgmap v4371: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:02.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:01 smithi045 conmon[39718]: cluster 2021-07-31T03:06:00.926242+0000 mgr.x (mgr 2021-07-31T03:06:02.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:01 smithi045 conmon[39718]: .24542) 3377 : cluster [DBG] pgmap v4371: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:02.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:01 smithi045 conmon[33655]: cluster 2021-07-31T03:06:00.926242+0000 mgr.x (mgr.24542) 3377 : cluster [DBG] pgmap v4371: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:04.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:03 smithi154 conmon[34553]: cluster 2021-07-31T03:06:02.926613+0000 mgr.x (mgr.24542) 3378 : cluster [DBG] 2021-07-31T03:06:04.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:03 smithi154 conmon[34553]: pgmap v4372: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:04.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:03 smithi045 conmon[39718]: cluster 2021-07-31T03:06:02.926613 2021-07-31T03:06:04.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:03 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 3378 : cluster [DBG] pgmap v4372: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:04.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:03 smithi045 conmon[33655]: cluster 2021-07-31T03:06:02.926613+0000 mgr.x (mgr.24542) 3378 : cluster [DBG] 2021-07-31T03:06:04.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:03 smithi045 conmon[33655]: pgmap v4372: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:06.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:05 smithi154 conmon[34553]: cluster 2021-07-31T03:06:04.927090+0000 mgr.x (mgr.24542) 3379 : cluster 2021-07-31T03:06:06.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:05 smithi154 conmon[34553]: [DBG] pgmap v4373: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:06.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:05 smithi045 conmon[39718]: cluster 2021-07-31T03:06:04.927090+0000 mgr.x (mgr 2021-07-31T03:06:06.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:05 smithi045 conmon[39718]: .24542) 3379 : cluster [DBG] pgmap v4373: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:06.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:05 smithi045 conmon[33655]: cluster 2021-07-31T03:06:04.927090+0000 mgr.x (mgr 2021-07-31T03:06:06.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:05 smithi045 conmon[33655]: .24542) 3379 : cluster [DBG] pgmap v4373: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:08.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:07 smithi154 conmon[34553]: cluster 2021-07-31T03:06:06.927454+0000 mgr.x (mgr.24542) 3380 : cluster 2021-07-31T03:06:08.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:07 smithi154 conmon[34553]: [DBG] pgmap v4374: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:08.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:07 smithi045 conmon[33655]: cluster 2021-07-31T03:06:06.927454+0000 mgr.x (mgr.24542) 3380 : cluster [DBG] pgmap v4374: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:08.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:07 smithi045 conmon[39718]: cluster 2021-07-31T03:06:06.927454+0000 mgr.x 2021-07-31T03:06:08.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:07 smithi045 conmon[39718]: (mgr.24542) 3380 : cluster [DBG] pgmap v4374: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:10.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:09 smithi154 conmon[34553]: cluster 2021-07-31T03:06:08.927995+0000 mgr.x (mgr.24542) 3381 : cluster [DBG] 2021-07-31T03:06:10.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:09 smithi154 conmon[34553]: pgmap v4375: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:10.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:09 smithi045 conmon[33655]: cluster 2021-07-31T03:06:08.927995+0000 mgr.x (mgr.24542) 3381 : cluster [DBG] pgmap v4375: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:10.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:09 smithi045 conmon[39718]: cluster 2021-07-31T03:06:08 2021-07-31T03:06:10.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:09 smithi045 conmon[39718]: .927995+0000 mgr.x (mgr.24542) 3381 : cluster [DBG] pgmap v4375: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:12.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:11 smithi154 conmon[34553]: cluster 2021-07-31T03:06:10.928409+0000 mgr.x (mgr.24542) 3382 : cluster [DBG] pgmap v4376: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:12.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:11 smithi045 conmon[33655]: cluster 2021-07-31T03:06:10.928409+0000 mgr.x (mgr.24542) 3382 : cluster 2021-07-31T03:06:12.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:11 smithi045 conmon[33655]: [DBG] pgmap v4376: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:12.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:11 smithi045 conmon[39718]: cluster 2021-07-31T03:06:10. 2021-07-31T03:06:12.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:11 smithi045 conmon[39718]: 928409+0000 mgr.x (mgr.24542) 3382 : cluster [DBG] pgmap v4376: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:14.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:13 smithi154 conmon[34553]: cluster 2021-07-31T03:06:12.928778+0000 mgr.x (mgr.24542) 3383 : cluster [DBG] pgmap v4377: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:14.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:13 smithi045 conmon[39718]: cluster 2021-07-31T03:06:12 2021-07-31T03:06:14.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:13 smithi045 conmon[39718]: .928778+0000 mgr.x (mgr.24542) 3383 : cluster [DBG] pgmap v4377: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:14.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:13 smithi045 conmon[33655]: cluster 2021-07-31T03:06:12.928778+0000 mgr.x (mgr.24542) 3383 : cluster [DBG] 2021-07-31T03:06:14.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:13 smithi045 conmon[33655]: pgmap v4377: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:16.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:15 smithi154 conmon[34553]: cluster 2021-07-31T03:06:14.929188+0000 mgr.x (mgr.24542) 3384 : cluster [DBG] pgmap v4378: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:16.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:15 smithi045 conmon[39718]: cluster 2021-07-31T03:06:14.929188+0000 mgr.x (mgr.24542) 3384 : cluster [DBG] pgmap v4378: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:16.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:15 smithi045 conmon[33655]: cluster 2021-07-31T03:06:14.929188+0000 mgr.x (mgr.24542) 3384 : cluster [DBG] pgmap v4378: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:18.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:17 smithi154 conmon[34553]: cluster 2021-07-31T03:06:16.929718+0000 mgr.x (mgr.24542) 3385 : cluster [DBG] pgmap v4379: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:18.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:17 smithi045 conmon[39718]: cluster 2021-07-31T03:06:16.929718+0000 mgr.x (mgr.24542) 3385 : cluster [DBG] pgmap v4379: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:18.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:17 smithi045 conmon[33655]: cluster 2021-07-31T03:06:16.929718+0000 mgr.x (mgr.24542) 3385 : cluster [DBG] 2021-07-31T03:06:18.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:17 smithi045 conmon[33655]: pgmap v4379: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:19.596 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:06:19.874 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:06:19.874 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 81m ago 81m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:06:19.874 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 78m ago 81m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:06:19.875 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 78m ago 85m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:06:19.875 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 81m ago 87m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:06:19.875 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 81m ago 87m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:06:19.875 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 78m ago 85m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:06:19.876 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 81m ago 85m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:06:19.876 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 81m ago 82m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:06:19.876 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 78m ago 82m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:06:19.876 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 81m ago 84m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:06:19.876 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 81m ago 84m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:06:19.877 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 81m ago 84m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:06:19.877 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 81m ago 83m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:06:19.877 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 78m ago 83m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:06:19.877 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 78m ago 83m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:06:19.878 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 78m ago 82m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:06:19.878 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 78m ago 82m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:06:19.878 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 78m ago 82m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:06:20.210 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:06:20.210 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:06:20.211 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:06:20.211 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:06:20.211 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:06:20.211 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:06:20.212 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:06:20.212 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:06:20.212 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:06:20.212 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:06:20.212 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:06:20.213 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:06:20.213 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:06:20.213 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:06:20.213 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:06:20.213 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:06:20.214 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:06:20.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:19 smithi154 conmon[34553]: cluster 2021-07-31T03:06:18.930148+0000 mgr.x (mgr 2021-07-31T03:06:20.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:19 smithi154 conmon[34553]: .24542) 3386 : cluster [DBG] pgmap v4380: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:20.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:19 smithi154 conmon[34553]: audit 2021-07-31T03:06:19.584448+0000 mgr.x (mgr.24542) 3387 : audit [DBG] from='client.18804 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:06:20.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:19 smithi045 conmon[39718]: cluster 2021-07-31T03:06:18.930148+0000 mgr.x (mgr.24542) 3386 : cluster [DBG] pgmap v4380: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:20.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:19 smithi045 conmon[39718]: audit 2021-07-31T03:06:19.584448 2021-07-31T03:06:20.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:19 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 3387 : audit [DBG] from='client.18804 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:06:20.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:19 smithi045 conmon[33655]: cluster 2021-07-31T03:06:18.930148+0000 mgr.x 2021-07-31T03:06:20.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:19 smithi045 conmon[33655]: (mgr.24542) 3386 : cluster [DBG] pgmap v4380: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:20.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:19 smithi045 conmon[33655]: audit 2021-07-31T03:06:19.584448+0000 mgr.x 2021-07-31T03:06:20.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:19 smithi045 conmon[33655]: (mgr.24542) 3387 : audit [DBG] from='client.18804 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:06:21.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:20 smithi154 conmon[34553]: audit 2021-07-31T03:06:19.866455+0000 mgr.x (mgr.24542) 3388 : audit [DBG] from='client.28750 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:06:21.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:20 smithi154 conmon[34553]: 2021-07-31T03:06:21.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:20 smithi154 conmon[34553]: audit 2021-07-31T03:06:20.210696+0000 mon.c (mon.1) 175 : audit [DBG] from='client.? 172.21.15.45:0/2070777235' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:06:21.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:20 smithi045 conmon[33655]: audit 2021-07-31T03:06:19.866455+0000 2021-07-31T03:06:21.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:20 smithi045 conmon[33655]: mgr.x (mgr.24542) 3388 : audit [DBG] from='client.28750 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:06:21.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:20 smithi045 conmon[33655]: audit 2021-07-31T03: 2021-07-31T03:06:21.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:20 smithi045 conmon[33655]: 06:20.210696+0000 mon.c (mon.1) 175 : audit [DBG] from='client.? 172.21.15.45:0/2070777235' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:06:21.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:20 smithi045 conmon[33655]: 2021-07-31T03:06:21.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:20 smithi045 conmon[39718]: audit 2021-07-31T03:06:19 2021-07-31T03:06:21.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:20 smithi045 conmon[39718]: .866455+0000 mgr.x (mgr.24542) 3388 : audit [DBG] from='client.28750 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:06:21.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:20 smithi045 conmon[39718]: audit 2021 2021-07-31T03:06:21.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:20 smithi045 conmon[39718]: -07-31T03:06:20.210696+0000 mon.c ( 2021-07-31T03:06:21.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:20 smithi045 conmon[39718]: mon.1) 175 : audit [DBG] from='client.? 172.21.15.45:0/2070777235' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:06:22.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:21 smithi154 conmon[34553]: cluster 2021-07-31T03:06:20.930625+0000 mgr.x (mgr.24542) 3389 : cluster [DBG] pgmap v4381: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:22.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:21 smithi045 conmon[33655]: cluster 2021-07-31T03:06:20 2021-07-31T03:06:22.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:21 smithi045 conmon[33655]: .930625+0000 mgr.x (mgr.24542) 3389 : cluster [DBG] pgmap v4381: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:22.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:21 smithi045 conmon[39718]: cluster 2021-07-31T03:06:20.930625+0000 mgr.x (mgr.24542) 3389 : 2021-07-31T03:06:22.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:21 smithi045 conmon[39718]: cluster [DBG] pgmap v4381: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:24.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:23 smithi154 conmon[34553]: cluster 2021-07-31T03:06:22.930987+0000 mgr.x (mgr.24542) 3390 : cluster [DBG] pgmap v4382: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:24.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:23 smithi045 conmon[39718]: cluster 2021-07-31T03:06:22.930987+0000 mgr.x (mgr.24542) 3390 : cluster 2021-07-31T03:06:24.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:23 smithi045 conmon[39718]: [DBG] pgmap v4382: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:24.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:23 smithi045 conmon[33655]: cluster 2021-07-31T03:06:22.930987+0000 mgr.x (mgr.24542 2021-07-31T03:06:24.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:23 smithi045 conmon[33655]: ) 3390 : cluster [DBG] pgmap v4382: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:26.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:25 smithi154 conmon[34553]: cluster 2021-07-31T03:06:24.931488+0000 mgr.x (mgr.24542) 3391 : cluster [DBG] pgmap v4383: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:26.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:25 smithi045 conmon[33655]: cluster 2021-07-31T03:06:24.931488+0000 mgr.x (mgr.24542 2021-07-31T03:06:26.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:25 smithi045 conmon[33655]: ) 3391 : cluster [DBG] pgmap v4383: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:26.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:25 smithi045 conmon[39718]: cluster 2021-07-31T03:06: 2021-07-31T03:06:26.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:25 smithi045 conmon[39718]: 24.931488+0000 mgr.x (mgr.24542) 3391 : cluster [DBG] pgmap v4383: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:28.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:27 smithi154 conmon[34553]: cluster 2021-07-31T03:06:26.931912+0000 mgr.x (mgr.24542) 3392 : cluster [DBG] pgmap v4384: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:28.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:27 smithi045 conmon[33655]: cluster 2021-07-31T03:06:26.931912+0000 mgr.x (mgr.24542) 3392 : cluster [DBG] pgmap v4384: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:28.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:27 smithi045 conmon[33655]: 2021-07-31T03:06:28.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:27 smithi045 conmon[39718]: cluster 2021-07-31T03:06:26 2021-07-31T03:06:28.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:27 smithi045 conmon[39718]: .931912+0000 mgr.x (mgr.24542) 3392 : cluster [DBG] pgmap v4384: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:29.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:28 smithi154 conmon[34553]: audit 2021-07-31T03:06:28.248769+0000 mon.b (mon.2) 470 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:06:29.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:28 smithi154 conmon[34553]: audit 2021-07-31T03:06:28.249102+0000 mon.b (mon.2) 471 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:06:29.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:28 smithi154 conmon[34553]: audit 2021-07-31T03:06:28.249562+0000 mon.a (mon.0) 3252 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:06:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:28 smithi154 conmon[34553]: audit 2021-07-31T03:06:28.249954+0000 2021-07-31T03:06:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:28 smithi154 conmon[34553]: mon.a (mon.0) 3253 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:06:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:28 smithi154 conmon[34553]: 2021-07-31T03:06:29.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:28 smithi045 conmon[33655]: audit 2021-07-31T03:06:28.248769+0000 mon.b (mon.2) 470 : 2021-07-31T03:06:29.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:28 smithi045 conmon[33655]: audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:06:29.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:28 smithi045 conmon[33655]: audit 2021-07-31T03:06:28.249102+0000 mon.b (mon.2) 471 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:06:29.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:28 smithi045 conmon[33655]: audit 2021-07-31T03:06:28.249562+0000 mon.a (mon.0) 3252 2021-07-31T03:06:29.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:28 smithi045 conmon[33655]: : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:06:29.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:28 smithi045 conmon[33655]: audit 2021-07-31T 2021-07-31T03:06:29.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:28 smithi045 conmon[33655]: 03:06:28.249954+0000 mon.a (mon.0) 3253 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:06:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:28 smithi045 conmon[39718]: audit 2021-07-31T03:06:28.248769+0000 mon.b (mon.2 2021-07-31T03:06:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:28 smithi045 conmon[39718]: ) 470 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:06:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:28 smithi045 conmon[39718]: audit 2021-07-31T03:06:28.249102+0000 mon.b (mon.2) 471 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:06:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:28 smithi045 conmon[39718]: 2021-07-31T03:06:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:28 smithi045 conmon[39718]: audit 2021-07-31T03:06:28.249562+0000 mon.a (mon.0 2021-07-31T03:06:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:28 smithi045 conmon[39718]: ) 3252 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:06:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:28 smithi045 conmon[39718]: audit 2021 2021-07-31T03:06:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:28 smithi045 conmon[39718]: -07-31T03:06:28.249954+0000 mon.a (mon.0) 3253 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:06:30.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:29 smithi154 conmon[34553]: cluster 2021-07-31T03:06:28.932513+0000 mgr.x (mgr.24542) 3393 : cluster [DBG] pgmap v4385: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:30.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:29 smithi045 conmon[33655]: cluster 2021-07-31T03:06:28.932513+0000 mgr.x (mgr.24542) 3393 : cluster [DBG] 2021-07-31T03:06:30.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:29 smithi045 conmon[33655]: pgmap v4385: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:30.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:29 smithi045 conmon[39718]: cluster 2021-07-31T03:06:28.932513+0000 mgr.x (mgr.24542) 3393 2021-07-31T03:06:30.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:29 smithi045 conmon[39718]: : cluster [DBG] pgmap v4385: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:31.752 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: debug 2021-07-31T03:06:31.327+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:06:31.752 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:06:31.753 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:06:31.753 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: yield 2021-07-31T03:06:31.753 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:06:31.753 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:06:31.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:06:31.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:06:31.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:06:31.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:06:31.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:06:31.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:06:31.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:06:31.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: task.result() 2021-07-31T03:06:31.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:06:31.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: debug 2021-07-31T03:06:31.339+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:06:31.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:06:31.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:06:31.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: yield 2021-07-31T03:06:31.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:06:31.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:06:31.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:06:31.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:06:31.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:06:31.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:06:31.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:06:31.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:06:31.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:06:31.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: task.result() 2021-07-31T03:06:31.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:06:31.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: debug 2021-07-31T03:06:31.364+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:06:31.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:06:31.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:06:31.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: yield 2021-07-31T03:06:31.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:06:31.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:06:31.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:06:31.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:06:31.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:06:31.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:06:31.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:06:31.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:06:31.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:06:31.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: task.result() 2021-07-31T03:06:31.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:06:31.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: debug 2021-07-31T03:06:31.365+0000 7fcb97969700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:06:31.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:06:31.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:06:31.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: yield 2021-07-31T03:06:31.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:06:31.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:06:31.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:06:31.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:06:31.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:06:31.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:06:31.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:06:31.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:06:31.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:06:31.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: task.result() 2021-07-31T03:06:31.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:06:31.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: 2021-07-31T03:06:31.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:06:31.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: 2021-07-31T03:06:31.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:06:31.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:06:31.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:06:31.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:06:31.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:06:31.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:06:31.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:06:31.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:06:31.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:06:31.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:06:31.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:06:31.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:06:31.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:06:31.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:06:31.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:06:31.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:06:31.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:06:31.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:06:31.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: raise 2021-07-31T03:06:31.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:06:31.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:06:31.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:06:31.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:06:31.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:06:31.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: debug 2021-07-31T03:06:31.373+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:06:31.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:06:31.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:06:31.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: yield 2021-07-31T03:06:31.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:06:31.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:06:31.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:06:31.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:06:31.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:06:31.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:06:31.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:06:31.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:06:31.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:06:31.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: task.result() 2021-07-31T03:06:31.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:06:31.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: debug 2021-07-31T03:06:31.374+0000 7fcb9816a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:06:31.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:06:31.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:06:31.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: yield 2021-07-31T03:06:31.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:06:31.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:06:31.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:06:31.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:06:31.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:06:31.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:06:31.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:06:31.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:06:31.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:06:31.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: task.result() 2021-07-31T03:06:31.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:06:31.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: 2021-07-31T03:06:31.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:06:31.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: 2021-07-31T03:06:31.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:06:31.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:06:31.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:06:31.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:06:31.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:06:31.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:06:31.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:06:31.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:06:31.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:06:31.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:06:31.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:06:31.786 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:06:31.786 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:06:31.786 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:06:31.786 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:06:31.787 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:06:31.787 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:06:31.787 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:06:31.787 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: raise 2021-07-31T03:06:31.787 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:06:31.788 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:06:31.788 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:06:31.788 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:06:31.788 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:06:32.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: cluster 2021-07-31T03:06:30.932962+0000 mgr.x (mgr.24542) 3394 : cluster [DBG] pgmap v4386: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:32.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: audit 2021-07-31T03:06:31.282713+0000 mon.b (mon.2) 472 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:06:32.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: audit 2021-07-31T03:06:31.284208+0000 2021-07-31T03:06:32.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: mon.b (mon.2) 473 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:06:32.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: audit 2021-07-31T03: 2021-07-31T03:06:32.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: 06:31.285306+0000 mon.b (mon.2) 2021-07-31T03:06:32.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: 474 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:06:32.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: cephadm 2021 2021-07-31T03:06:32.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: -07-31T03:06:31.286083+0000 2021-07-31T03:06:32.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: mgr.x (mgr.24542) 3395 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:06:32.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: 2021-07-31T03:06:32.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: cephadm 2021-07-31T03:06: 2021-07-31T03:06:32.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: 31.328509+0000 mgr.x (mgr.24542) 3396 2021-07-31T03:06:32.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: : cephadm [ERR] 2021-07-31T03:06:32.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:06:32.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:06:32.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: yield 2021-07-31T03:06:32.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:06:32.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:06:32.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:06:32.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:06:32.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:06:32.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:06:32.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:06:32.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:06:32.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:06:32.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: task.result() 2021-07-31T03:06:32.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:06:32.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: cephadm 2021-07-31T03:06:31.328963+0000 mgr.x (mgr.24542) 3397 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:06:32.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: cephadm 2021-07-31T03:06:31.340173+0000 mgr.x (mgr.24542) 3398 : cephadm [ERR] 2021-07-31T03:06:32.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:06:32.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:06:32.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: yield 2021-07-31T03:06:32.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:06:32.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:06:32.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:06:32.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:06:32.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:06:32.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:06:32.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:06:32.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:06:32.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:06:32.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: task.result() 2021-07-31T03:06:32.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:06:32.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: cephadm 2021-07-31T03:06:31.341058+0000 mgr.x (mgr.24542) 3399 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:06:32.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: cephadm 2021-07-31T03:06:31.365816+0000 mgr.x (mgr.24542) 3400 : cephadm [ERR] 2021-07-31T03:06:32.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:06:32.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:06:32.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: yield 2021-07-31T03:06:32.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:06:32.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:06:32.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:06:32.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:06:32.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:06:32.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:06:32.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:06:32.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:06:32.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:06:32.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: task.result() 2021-07-31T03:06:32.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:06:32.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: cephadm 2021-07-31T03:06:31.366417+0000 mgr.x (mgr.24542) 3401 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:06:32.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:06:32.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:06:32.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: yield 2021-07-31T03:06:32.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:06:32.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:06:32.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:06:32.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:06:32.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:06:32.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:06:32.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:06:32.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:06:32.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:06:32.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: task.result() 2021-07-31T03:06:32.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:06:32.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: 2021-07-31T03:06:32.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:06:32.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: 2021-07-31T03:06:32.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:06:32.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:06:32.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:06:32.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:06:32.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:06:32.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:06:32.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:06:32.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:06:32.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:06:32.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:06:32.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:06:32.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:06:32.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:06:32.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:06:32.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:06:32.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:06:32.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:06:32.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:06:32.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: raise 2021-07-31T03:06:32.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:06:32.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:06:32.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:06:32.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:06:32.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:06:32.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: cephadm 2021-07-31T03:06:31. 2021-07-31T03:06:32.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: 374752+0000 mgr.x (mgr.24542) 3402 : cephadm [ERR] 2021-07-31T03:06:32.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:06:32.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:06:32.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: yield 2021-07-31T03:06:32.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:06:32.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:06:32.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:06:32.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:06:32.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:06:32.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:06:32.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:06:32.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:06:32.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:06:32.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: task.result() 2021-07-31T03:06:32.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:06:32.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: cephadm 2021-07-31T03:06:31.375161+0000 mgr.x (mgr.24542) 3403 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:06:32.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:06:32.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:06:32.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: yield 2021-07-31T03:06:32.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:06:32.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:06:32.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:06:32.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:06:32.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:06:32.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:06:32.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:06:32.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:06:32.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:06:32.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: task.result() 2021-07-31T03:06:32.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:06:32.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: 2021-07-31T03:06:32.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:06:32.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: 2021-07-31T03:06:32.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:06:32.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:06:32.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:06:32.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:06:32.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:06:32.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:06:32.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:06:32.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:06:32.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:06:32.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:06:32.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:06:32.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:06:32.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:06:32.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:06:32.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:06:32.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:06:32.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:06:32.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:06:32.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: raise 2021-07-31T03:06:32.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:06:32.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:06:32.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:06:32.285 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:06:32.285 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:31 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:06:32.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: cluster 2021-07-31T03:06:30.932962+0000 mgr.x (mgr.24542) 3394 : 2021-07-31T03:06:32.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: cluster [DBG] pgmap v4386: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:32.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: audit 2021-07-31T03:06:31.282713+0000 mon.b (mon.2) 472 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:06:32.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: audit 2021-07-31T03:06:31. 2021-07-31T03:06:32.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: 284208+0000 mon.b (mon.2) 473 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:06:32.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: audit 2021-07-31T03:06 2021-07-31T03:06:32.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: :31.285306+0000 mon.b (mon.2) 474 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:06:32.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: cephadm 2021-07-31T03:06:32.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: 2021-07-31T03:06:31.286083+0000 mgr.x (mgr.24542) 3395 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:06:32.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: cephadm 2021-07-31T03:06:31.328509+0000 mgr.x (mgr 2021-07-31T03:06:32.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: .24542) 3396 : cephadm [ERR] 2021-07-31T03:06:32.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:06:32.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:06:32.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: yield 2021-07-31T03:06:32.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:06:32.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:06:32.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:06:32.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:06:32.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:06:32.290 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:06:32.290 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:06:32.290 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:06:32.290 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:06:32.290 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: task.result() 2021-07-31T03:06:32.291 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:06:32.291 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: cephadm 2021-07-31T03:06:32.291 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: 2021-07-31T03:06:31.328963+0000 mgr.x (mgr.24542) 3397 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:06:32.291 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: cephadm 2021-07-31T03:06:31.340173+0000 mgr.x (mgr.24542) 3398 : cephadm [ERR] 2021-07-31T03:06:32.291 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:06:32.292 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:06:32.292 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: yield 2021-07-31T03:06:32.292 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:06:32.292 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:06:32.292 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:06:32.293 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:06:32.293 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:06:32.293 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:06:32.293 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:06:32.293 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:06:32.294 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:06:32.294 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: task.result() 2021-07-31T03:06:32.294 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:06:32.294 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: cephadm 2021-07-31T03:06:31.341058+0000 mgr.x (mgr.24542) 3399 2021-07-31T03:06:32.294 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:06:32.295 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: cephadm 2021-07-31T03:06:31.365816+0000 mgr.x (mgr.24542) 3400 : cephadm [ERR] 2021-07-31T03:06:32.295 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:06:32.295 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:06:32.295 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: yield 2021-07-31T03:06:32.295 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:06:32.296 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:06:32.296 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:06:32.296 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:06:32.296 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:06:32.296 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:06:32.297 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:06:32.297 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:06:32.297 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:06:32.297 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: task.result() 2021-07-31T03:06:32.297 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:06:32.297 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: cephadm 2021-07-31T03:06:31.366417+0000 mgr.x (mgr.24542) 2021-07-31T03:06:32.298 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: 3401 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:06:32.298 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:06:32.298 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:06:32.298 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: yield 2021-07-31T03:06:32.299 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:06:32.299 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:06:32.299 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:06:32.299 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:06:32.299 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:06:32.300 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:06:32.300 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:06:32.300 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:06:32.300 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:06:32.300 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: task.result() 2021-07-31T03:06:32.301 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:06:32.301 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: 2021-07-31T03:06:32.301 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:06:32.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: cluster 2021-07-31T03:06: 2021-07-31T03:06:32.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: 30.932962+0000 mgr.x (mgr.24542) 3394 : cluster [DBG] pgmap v4386: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:32.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: audit 2021-07 2021-07-31T03:06:32.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: -31T03:06:31.282713+0000 mon.b (mon.2) 472 : audit [DBG] 2021-07-31T03:06:32.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:06:32.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T03:06:32.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: T03:06:31.284208+0000 mon.b (mon.2) 473 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:06:32.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: audit 2021-07-31T03:06:31.285306+0000 mon.b (mon.2) 474 2021-07-31T03:06:32.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:06:32.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: cephadm 2021- 2021-07-31T03:06:32.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: 07-31T03:06:31.286083+0000 mgr.x (mgr.24542) 3395 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:06:32.304 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: cephadm 2021-07-31T03 2021-07-31T03:06:32.304 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: :06:31.328509+0000 mgr.x (mgr.24542) 3396 : cephadm [ERR] 2021-07-31T03:06:32.304 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:06:32.304 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:06:32.304 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: yield 2021-07-31T03:06:32.305 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:06:32.305 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:06:32.305 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:06:32.305 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:06:32.305 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:06:32.306 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:06:32.306 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:06:32.306 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:06:32.306 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:06:32.306 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: task.result() 2021-07-31T03:06:32.307 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:06:32.307 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: cephadm 2021-07-31T03:06:31.328963+0000 mgr.x (mgr.24542) 3397 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:06:32.307 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: cephadm 2021-07-31T03:06:31.340173+0000 mgr.x (mgr.24542) 3398 : cephadm [ERR] 2021-07-31T03:06:32.307 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:06:32.307 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:06:32.308 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: yield 2021-07-31T03:06:32.308 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:06:32.308 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:06:32.308 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:06:32.308 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:06:32.309 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: 2021-07-31T03:06:32.309 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:06:32.309 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:06:32.310 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:06:32.310 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:06:32.310 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:06:32.310 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:06:32.310 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:06:32.311 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:06:32.311 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:06:32.311 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:06:32.311 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:06:32.311 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:06:32.312 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:06:32.312 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:06:32.312 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:06:32.312 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:06:32.312 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:06:32.313 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:06:32.313 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: raise 2021-07-31T03:06:32.313 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:06:32.313 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:06:32.313 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:06:32.314 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:06:32.314 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:06:32.314 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: cephadm 2021-07-31T03:06:31.374752+0000 mgr.x (mgr.24542) 3402 : cephadm [ERR] 2021-07-31T03:06:32.314 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:06:32.314 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:06:32.315 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: yield 2021-07-31T03:06:32.315 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:06:32.315 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:06:32.315 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:06:32.315 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:06:32.316 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:06:32.316 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:06:32.316 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:06:32.316 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:06:32.316 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:06:32.317 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: task.result() 2021-07-31T03:06:32.317 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:06:32.317 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: cephadm 2021-07-31T03:06:31.375161+0000 mgr.x (mgr.24542) 3403 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:06:32.317 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:06:32.318 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:06:32.318 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: yield 2021-07-31T03:06:32.318 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:06:32.318 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:06:32.318 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:06:32.319 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:06:32.319 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:06:32.319 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:06:32.319 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:06:32.319 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:06:32.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:06:32.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: task.result() 2021-07-31T03:06:32.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:06:32.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: 2021-07-31T03:06:32.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:06:32.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: 2021-07-31T03:06:32.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:06:32.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:06:32.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:06:32.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:06:32.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:06:32.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:06:32.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:06:32.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:06:32.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:06:32.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:06:32.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:06:32.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:06:32.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:06:32.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:06:32.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:06:32.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:06:32.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:06:32.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:06:32.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: raise 2021-07-31T03:06:32.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:06:32.325 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:06:32.325 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:06:32.325 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:06:32.325 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:06:32.326 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:06:32.326 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:06:32.327 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:06:32.327 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:06:32.327 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:06:32.327 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: task.result() 2021-07-31T03:06:32.327 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:06:32.328 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: cephadm 2021-07-31T03:06:31.341058+0000 mgr.x (mgr.24542) 3399 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:06:32.328 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: cephadm 2021-07-31T03:06:31.365816+0000 mgr.x (mgr.24542) 3400 : cephadm [ERR] 2021-07-31T03:06:32.328 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:06:32.328 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:06:32.329 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: yield 2021-07-31T03:06:32.329 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:06:32.329 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:06:32.329 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:06:32.329 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:06:32.330 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:06:32.330 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:06:32.330 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:06:32.330 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:06:32.330 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:06:32.331 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: task.result() 2021-07-31T03:06:32.331 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:06:32.331 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: cephadm 2021-07-31T03:06:31.366417+0000 mgr.x (mgr.24542) 3401 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:06:32.331 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:06:32.331 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:06:32.332 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: yield 2021-07-31T03:06:32.332 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:06:32.332 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:06:32.332 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:06:32.332 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:06:32.333 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:06:32.333 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:06:32.333 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:06:32.333 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:06:32.333 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:06:32.334 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: task.result() 2021-07-31T03:06:32.334 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:06:32.334 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: 2021-07-31T03:06:32.334 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:06:32.334 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: 2021-07-31T03:06:32.335 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:06:32.335 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:06:32.335 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:06:32.335 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:06:32.335 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:06:32.336 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:06:32.336 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:06:32.336 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:06:32.336 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:06:32.336 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:06:32.337 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:06:32.337 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:06:32.337 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:06:32.337 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:06:32.338 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:06:32.338 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:06:32.338 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:06:32.338 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:06:32.338 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: raise 2021-07-31T03:06:32.339 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:06:32.339 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:06:32.339 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:06:32.339 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:06:32.339 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:06:32.339 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: cephadm 2021-07-31T03:06:31.374752+0000 mgr.x (mgr.24542) 3402 : cephadm [ERR] 2021-07-31T03:06:32.340 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:06:32.340 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:06:32.340 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: yield 2021-07-31T03:06:32.340 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:06:32.340 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:06:32.341 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:06:32.341 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:06:32.341 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:06:32.341 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:06:32.341 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:06:32.342 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:06:32.342 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:06:32.342 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: task.result() 2021-07-31T03:06:32.342 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:06:32.343 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: cephadm 2021-07-31T03:06:31.375161+0000 mgr.x (mgr.24542) 3403 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:06:32.343 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:06:32.343 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:06:32.343 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: yield 2021-07-31T03:06:32.343 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:06:32.344 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:06:32.344 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:06:32.344 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:06:32.344 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:06:32.344 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:06:32.344 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:06:32.345 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:06:32.345 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:06:32.345 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: task.result() 2021-07-31T03:06:32.345 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:06:32.345 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: 2021-07-31T03:06:32.346 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:06:32.346 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: 2021-07-31T03:06:32.346 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:06:32.346 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:06:32.346 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:06:32.347 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:06:32.347 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:06:32.347 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:06:32.347 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:06:32.347 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:06:32.348 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:06:32.348 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:06:32.348 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:06:32.348 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:06:32.348 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:06:32.349 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:06:32.349 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:06:32.349 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:06:32.349 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:06:32.349 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:06:32.350 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: raise 2021-07-31T03:06:32.350 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:06:32.350 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:06:32.350 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:06:32.350 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:06:32.351 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:31 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:06:34.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:33 smithi154 conmon[34553]: cluster 2021-07-31T03:06:32.933322+0000 mgr.x (mgr.24542) 3404 : cluster [DBG] pgmap v4387: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:34.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:33 smithi045 conmon[33655]: cluster 2021-07-31T03:06:32.933322+0000 mgr.x 2021-07-31T03:06:34.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:33 smithi045 conmon[33655]: (mgr.24542) 3404 : cluster [DBG] pgmap v4387: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:34.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:33 smithi045 conmon[39718]: cluster 2021-07-31T03:06:32.933322+0000 mgr.x (mgr. 2021-07-31T03:06:34.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:33 smithi045 conmon[39718]: 24542) 3404 : cluster [DBG] pgmap v4387: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:36.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:35 smithi154 conmon[34553]: cluster 2021-07-31T03:06:34.933939+0000 mgr.x (mgr.24542) 3405 : cluster [DBG] pgmap v4388: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:36.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:35 smithi045 conmon[39718]: cluster 2021-07-31T03:06:34.933939+0000 mgr.x (mgr. 2021-07-31T03:06:36.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:35 smithi045 conmon[39718]: 24542) 3405 : cluster [DBG] pgmap v4388: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:36.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:35 smithi045 conmon[33655]: cluster 2021-07-31T03:06:34.933939+0000 mgr.x ( 2021-07-31T03:06:36.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:35 smithi045 conmon[33655]: mgr.24542) 3405 : cluster [DBG] pgmap v4388: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:38.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:37 smithi154 conmon[34553]: cluster 2021-07-31T03:06:36.934366+0000 mgr.x (mgr.24542) 3406 : cluster [DBG] pgmap v4389: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:38.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:37 smithi154 conmon[34553]: 2021-07-31T03:06:38.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:37 smithi045 conmon[39718]: cluster 2021-07-31T03:06:36.934366 2021-07-31T03:06:38.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:37 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 3406 : cluster [DBG] pgmap v4389: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:38.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:37 smithi045 conmon[33655]: cluster 2021-07-31T03:06:36.934366+0000 mgr.x ( 2021-07-31T03:06:38.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:37 smithi045 conmon[33655]: mgr.24542) 3406 : cluster [DBG] pgmap v4389: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:40.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:39 smithi154 conmon[34553]: cluster 2021-07-31T03:06:38.934852+0000 mgr.x (mgr.24542) 3407 : cluster [DBG] pgmap v4390: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:40.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:39 smithi045 conmon[39718]: cluster 2021-07-31T03:06:38.934852+0000 mgr.x (mgr 2021-07-31T03:06:40.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:39 smithi045 conmon[39718]: .24542) 3407 : cluster [DBG] pgmap v4390: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:40.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:39 smithi045 conmon[33655]: cluster 2021-07-31T03:06:38.934852+0000 mgr.x (mgr.24542) 2021-07-31T03:06:40.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:39 smithi045 conmon[33655]: 3407 : cluster [DBG] pgmap v4390: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:42.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:42 smithi154 conmon[34553]: cluster 2021-07-31T03:06:40.935316+0000 mgr.x (mgr.24542) 3408 : cluster [DBG] 2021-07-31T03:06:42.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:42 smithi154 conmon[34553]: pgmap v4391: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:42.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:42 smithi045 conmon[39718]: cluster 2021-07-31T03:06:40.935316+0000 mgr.x (mgr. 2021-07-31T03:06:42.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:42 smithi045 conmon[39718]: 24542) 3408 : cluster [DBG] pgmap v4391: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:42.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:42 smithi045 conmon[33655]: cluster 2021-07-31T03:06:40.935316+0000 mgr.x 2021-07-31T03:06:42.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:42 smithi045 conmon[33655]: (mgr.24542) 3408 : cluster [DBG] pgmap v4391: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:44.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:43 smithi154 conmon[34553]: cluster 2021-07-31T03:06:42.935676+0000 mgr.x (mgr.24542) 3409 : cluster [DBG] pgmap v4392: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:44.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:43 smithi045 conmon[39718]: cluster 2021-07-31T03:06:42.935676+0000 mgr.x (mgr. 2021-07-31T03:06:44.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:43 smithi045 conmon[39718]: 24542) 3409 : cluster [DBG] pgmap v4392: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:44.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:43 smithi045 conmon[33655]: cluster 2021-07-31T03:06:42.935676 2021-07-31T03:06:44.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:43 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 3409 : cluster [DBG] pgmap v4392: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:46.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:45 smithi154 conmon[34553]: cluster 2021-07-31T03:06:44.936277+0000 mgr.x (mgr.24542) 3410 : cluster [DBG] pgmap v4393: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:46.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:45 smithi154 conmon[34553]: 2021-07-31T03:06:46.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:45 smithi045 conmon[33655]: cluster 2021-07-31T03:06:44.936277+0000 mgr.x (mgr.24542) 3410 : cluster 2021-07-31T03:06:46.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:45 smithi045 conmon[33655]: [DBG] pgmap v4393: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:46.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:45 smithi045 conmon[39718]: cluster 2021-07-31T03:06:44.936277+0000 mgr.x (mgr 2021-07-31T03:06:46.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:45 smithi045 conmon[39718]: .24542) 3410 : cluster [DBG] pgmap v4393: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:48.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:47 smithi154 conmon[34553]: cluster 2021-07-31T03:06:46.936812+0000 mgr.x (mgr.24542) 3411 : cluster [DBG] pgmap v4394: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:48.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:47 smithi045 conmon[33655]: cluster 2021-07-31T03:06:46.936812+0000 mgr.x (mgr.24542) 2021-07-31T03:06:48.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:47 smithi045 conmon[33655]: 3411 : cluster [DBG] pgmap v4394: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:48.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:47 smithi045 conmon[39718]: cluster 2021-07-31T03:06:46.936812+0000 2021-07-31T03:06:48.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:47 smithi045 conmon[39718]: mgr.x (mgr.24542) 3411 : cluster [DBG] pgmap v4394: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:50.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:50 smithi154 conmon[34553]: cluster 2021-07-31T03:06:48.937261+0000 mgr.x (mgr.24542) 3412 : cluster [DBG] pgmap v4395: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:50.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:50 smithi045 conmon[33655]: cluster 2021-07-31T03:06:48.937261+0000 mgr.x ( 2021-07-31T03:06:50.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:50 smithi045 conmon[33655]: mgr.24542) 3412 : cluster [DBG] pgmap v4395: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:50.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:50 smithi045 conmon[39718]: cluster 2021-07-31T03:06:48.937261+0000 2021-07-31T03:06:50.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:50 smithi045 conmon[39718]: mgr.x (mgr.24542) 3412 : cluster [DBG] pgmap v4395: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:50.530 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:06:50.807 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:06:50.808 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 81m ago 82m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:06:50.808 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 79m ago 82m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:06:50.808 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 79m ago 85m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:06:50.808 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 81m ago 87m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:06:50.809 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 81m ago 88m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:06:50.809 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 79m ago 86m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:06:50.809 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 81m ago 86m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:06:50.809 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 81m ago 82m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:06:50.809 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 79m ago 82m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:06:50.810 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 81m ago 85m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:06:50.810 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 81m ago 85m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:06:50.810 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 81m ago 84m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:06:50.810 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 81m ago 84m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:06:50.811 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 79m ago 84m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:06:50.811 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 79m ago 83m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:06:50.811 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 79m ago 83m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:06:50.811 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 79m ago 83m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:06:50.811 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 79m ago 82m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:06:51.147 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:06:51.147 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:06:51.147 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:06:51.147 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:06:51.148 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:06:51.148 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:06:51.148 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:06:51.148 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:06:51.148 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:06:51.149 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:06:51.149 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:06:51.149 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:06:51.149 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:06:51.149 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:06:51.150 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:06:51.150 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:06:51.150 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:06:51.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:51 smithi045 conmon[39718]: audit 2021-07-31T03: 2021-07-31T03:06:51.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:51 smithi045 conmon[39718]: 06:50.517340+0000 mgr.x (mgr.24542) 3413 : audit [DBG] from='client.28762 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:06:51.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:51 smithi045 conmon[33655]: audit 2021-07-31T03:06:50.517340+0000 mgr.x ( 2021-07-31T03:06:51.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:51 smithi045 conmon[33655]: mgr.24542) 3413 : audit [DBG] from='client.28762 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:06:51.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:51 smithi154 conmon[34553]: audit 2021-07-31T03:06:50.517340+0000 mgr.x (mgr.24542) 3413 : audit [DBG] from='client.28762 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:06:51.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:51 smithi154 conmon[34553]: 2021-07-31T03:06:52.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:52 smithi045 conmon[39718]: audit 2021-07-31T03:06:50.800565+0000 mgr.x (mgr.24542) 3414 : audit [DBG] from='client.18828 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:06:52.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:52 smithi045 conmon[39718]: cluster 2021-07-31T03:06:50.937831+0000 mgr.x (mgr.24542) 3415 : cluster [DBG] pgmap v4396: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:52.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:52 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T03:06:52.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:52 smithi045 conmon[39718]: T03:06:51.147466+0000 mon.c (mon.1) 176 : audit 2021-07-31T03:06:52.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:52 smithi045 conmon[39718]: [DBG] from='client.? 172.21.15.45:0/2873898769' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:06:52.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:52 smithi045 conmon[33655]: audit 2021-07-31T03:06:50.800565+0000 mgr.x (mgr.24542) 2021-07-31T03:06:52.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:52 smithi045 conmon[33655]: 3414 : audit [DBG] from='client.18828 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:06:52.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:52 smithi045 conmon[33655]: cluster 2021-07-31T03:06:50.937831+0000 mgr.x (mgr.24542) 3415 : cluster [DBG] pgmap v4396: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:52.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:52 smithi045 conmon[33655]: audit 2021-07-31T03:06:51.147466+0000 mon.c (mon.1) 176 : audit [DBG] 2021-07-31T03:06:52.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:52 smithi045 conmon[33655]: from='client.? 172.21.15.45:0/2873898769' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:06:52.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:52 smithi154 conmon[34553]: audit 2021-07-31T03:06:50.800565+0000 mgr.x (mgr.24542) 3414 : audit [DBG] from='client.18828 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:06:52.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:52 smithi154 conmon[34553]: cluster 2021-07-31T03:06:50.937831+0000 mgr.x (mgr.24542) 3415 : cluster [DBG] pgmap v4396: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:52.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:52 smithi154 conmon[34553]: audit 2021-07-31T03:06:51.147466+0000 mon.c ( 2021-07-31T03:06:52.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:52 smithi154 conmon[34553]: mon.1) 176 : audit [DBG] from='client.? 172.21.15.45:0/2873898769' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:06:54.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:53 smithi154 conmon[34553]: cluster 2021-07-31T03:06:52.938252+0000 mgr.x (mgr.24542) 3416 : cluster [DBG] pgmap v4397: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:54.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:53 smithi045 conmon[39718]: cluster 2021-07-31T03:06:52 2021-07-31T03:06:54.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:53 smithi045 conmon[39718]: .938252+0000 mgr.x (mgr.24542) 3416 : cluster [DBG] pgmap v4397: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:54.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:53 smithi045 conmon[33655]: cluster 2021-07-31T03:06:52.938252+0000 mgr.x (mgr.24542) 3416 2021-07-31T03:06:54.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:53 smithi045 conmon[33655]: : cluster [DBG] pgmap v4397: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:56.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:55 smithi154 conmon[34553]: cluster 2021-07-31T03:06:54.938713+0000 mgr.x (mgr.24542) 3417 : cluster [DBG] pgmap v4398: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:56.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:55 smithi045 conmon[39718]: cluster 2021-07-31T03:06:54.938713+0000 mgr.x 2021-07-31T03:06:56.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:55 smithi045 conmon[39718]: (mgr.24542) 3417 : cluster [DBG] pgmap v4398: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:56.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:55 smithi045 conmon[33655]: cluster 2021-07-31T03:06:54.938713+0000 mgr.x (mgr.24542 2021-07-31T03:06:56.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:55 smithi045 conmon[33655]: ) 3417 : cluster [DBG] pgmap v4398: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:58.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:57 smithi154 conmon[34553]: cluster 2021-07-31T03:06:56.939262+0000 mgr.x (mgr.24542) 3418 : cluster [DBG] pgmap v4399: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:58.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:57 smithi154 conmon[34553]: 2021-07-31T03:06:58.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:57 smithi045 conmon[33655]: cluster 2021-07-31T03:06:56.939262+0000 mgr.x (mgr.24542) 3418 : cluster [DBG] pgmap v4399: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:06:58.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:57 smithi045 conmon[39718]: cluster 2021-07-31T03:06:56.939262+0000 mgr.x (mgr.24542 2021-07-31T03:06:58.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:57 smithi045 conmon[39718]: ) 3418 : cluster [DBG] pgmap v4399: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:00.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:59 smithi154 conmon[34553]: cluster 2021-07-31T03:06:58.939757+0000 mgr.x (mgr.24542) 3419 : cluster [DBG] pgmap v4400: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:00.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:06:59 smithi154 conmon[34553]: 2021-07-31T03:07:00.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:59 smithi045 conmon[39718]: cluster 2021-07-31T03:06:58.939757+0000 2021-07-31T03:07:00.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:06:59 smithi045 conmon[39718]: mgr.x (mgr.24542) 3419 : cluster [DBG] pgmap v4400: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:00.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:59 smithi045 conmon[33655]: cluster 2021-07-31T03:06:58.939757+0000 mgr.x (mgr.24542) 3419 : cluster [DBG] 2021-07-31T03:07:00.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:06:59 smithi045 conmon[33655]: pgmap v4400: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:02.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:01 smithi154 conmon[34553]: cluster 2021-07-31T03:07:00.940170+0000 mgr.x (mgr.24542) 3420 : cluster [DBG] pgmap v4401: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:02.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:01 smithi045 conmon[39718]: cluster 2021-07-31T03:07:00.940170+0000 mgr.x (mgr.24542) 3420 : cluster [DBG] pgmap v4401: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:02.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:01 smithi045 conmon[33655]: cluster 2021-07-31T03:07:00.940170+0000 mgr.x (mgr.24542) 3420 : cluster [DBG] 2021-07-31T03:07:02.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:01 smithi045 conmon[33655]: pgmap v4401: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:04.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:03 smithi154 conmon[34553]: cluster 2021-07-31T03:07:02.940508+0000 mgr.x (mgr.24542) 3421 : cluster [DBG] pgmap v4402: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:04.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:03 smithi045 conmon[39718]: cluster 2021-07-31T03:07:02.940508+0000 mgr.x (mgr 2021-07-31T03:07:04.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:03 smithi045 conmon[39718]: .24542) 3421 : cluster [DBG] pgmap v4402: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:04.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:03 smithi045 conmon[33655]: cluster 2021-07-31T03:07: 2021-07-31T03:07:04.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:03 smithi045 conmon[33655]: 02.940508+0000 mgr.x (mgr.24542) 3421 : cluster [DBG] pgmap v4402: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:06.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:05 smithi154 conmon[34553]: cluster 2021-07-31T03:07:04.940987+0000 mgr.x (mgr.24542) 3422 : cluster [DBG] pgmap v4403: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:06.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:05 smithi154 conmon[34553]: 2021-07-31T03:07:06.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:05 smithi045 conmon[39718]: cluster 2021-07-31T03:07:04.940987+0000 mgr.x (mgr.24542) 2021-07-31T03:07:06.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:05 smithi045 conmon[39718]: 3422 : cluster [DBG] pgmap v4403: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:06.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:05 smithi045 conmon[33655]: cluster 2021-07 2021-07-31T03:07:06.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:05 smithi045 conmon[33655]: -31T03:07:04.940987+0000 mgr.x (mgr.24542) 3422 : cluster [DBG] pgmap v4403: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:08.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:07 smithi154 conmon[34553]: cluster 2021-07-31T03:07:06.941488+0000 mgr.x (mgr.24542) 3423 : cluster [DBG] pgmap v4404: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:08.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:07 smithi045 conmon[33655]: cluster 2021-07-31T03:07:06.941488 2021-07-31T03:07:08.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:07 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 3423 : cluster [DBG] pgmap v4404: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:08.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:07 smithi045 conmon[39718]: cluster 2021-07-31T03:07:06.941488+0000 mgr.x (mgr. 2021-07-31T03:07:08.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:07 smithi045 conmon[39718]: 24542) 3423 : cluster [DBG] pgmap v4404: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:10.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:09 smithi154 conmon[34553]: cluster 2021-07-31T03:07:08.942017+0000 mgr.x (mgr.24542) 3424 : cluster [DBG] pgmap v4405: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:10.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:09 smithi154 conmon[34553]: 2021-07-31T03:07:10.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:09 smithi045 conmon[33655]: cluster 2021-07-31T03:07:08.942017+0000 mgr.x (mgr.24542) 3424 : cluster [DBG] pgmap v4405: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:10.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:09 smithi045 conmon[39718]: cluster 2021-07-31T03:07:08.942017+0000 mgr.x (mgr.24542) 3424 : cluster [DBG] pgmap v4405: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:12.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:11 smithi154 conmon[34553]: cluster 2021-07-31T03:07:10.942489+0000 mgr.x (mgr.24542) 3425 : cluster [DBG] pgmap v4406: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:12.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:11 smithi045 conmon[33655]: cluster 2021-07-31T03:07:10.942489+0000 mgr.x (mgr.24542) 3425 : cluster [DBG] pgmap v4406: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:12.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:11 smithi045 conmon[39718]: cluster 2021-07-31T03:07:10 2021-07-31T03:07:12.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:11 smithi045 conmon[39718]: .942489+0000 mgr.x (mgr.24542) 3425 : cluster [DBG] pgmap v4406: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:14.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:13 smithi154 conmon[34553]: cluster 2021-07-31T03:07:12.942875+0000 mgr.x (mgr.24542) 3426 : cluster [DBG] 2021-07-31T03:07:14.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:13 smithi154 conmon[34553]: pgmap v4407: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:14.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:13 smithi045 conmon[33655]: cluster 2021-07-31T03:07:12.942875+0000 mgr.x (mgr.24542) 3426 : 2021-07-31T03:07:14.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:13 smithi045 conmon[33655]: cluster [DBG] pgmap v4407: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:14.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:13 smithi045 conmon[39718]: cluster 2021-07-31T03:07:12.942875+0000 mgr.x (mgr. 2021-07-31T03:07:14.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:13 smithi045 conmon[39718]: 24542) 3426 : cluster [DBG] pgmap v4407: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:16.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:15 smithi154 conmon[34553]: cluster 2021-07-31T03:07:14.943434+0000 mgr.x (mgr.24542) 3427 : cluster [DBG] pgmap v4408: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:16.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:15 smithi045 conmon[33655]: cluster 2021-07-31T03:07:14.943434 2021-07-31T03:07:16.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:15 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 3427 : cluster [DBG] pgmap v4408: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:16.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:15 smithi045 conmon[39718]: cluster 2021-07-31T03:07:14.943434+0000 mgr.x (mgr.24542) 3427 : 2021-07-31T03:07:16.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:15 smithi045 conmon[39718]: cluster [DBG] pgmap v4408: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:18.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:18 smithi045 conmon[33655]: cluster 2021-07-31T03:07:16.944015+0000 mgr.x (mgr.24542) 3428 : cluster [DBG] 2021-07-31T03:07:18.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:18 smithi045 conmon[33655]: pgmap v4409: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:18.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:18 smithi045 conmon[39718]: cluster 2021-07-31T03:07:16.944015+0000 mgr.x (mgr 2021-07-31T03:07:18.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:18 smithi045 conmon[39718]: .24542) 3428 : cluster [DBG] pgmap v4409: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:18.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:18 smithi154 conmon[34553]: cluster 2021-07-31T03:07:16.944015+0000 mgr.x (mgr.24542) 3428 : cluster [DBG] pgmap v4409: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:20.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:20 smithi045 conmon[33655]: cluster 2021-07-31T03:07:18.944444+0000 mgr.x (mgr.24542) 3429 : cluster [DBG] pgmap v4410: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:20.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:20 smithi045 conmon[39718]: cluster 2021-07-31T03:07:18.944444+0000 mgr.x (mgr.24542) 3429 : cluster [DBG] 2021-07-31T03:07:20.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:20 smithi045 conmon[39718]: pgmap v4410: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:20.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:20 smithi154 conmon[34553]: cluster 2021-07-31T03:07:18.944444+0000 mgr.x (mgr.24542) 3429 : cluster [DBG] pgmap v4410: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:21.464 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:07:21.740 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:07:21.741 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 82m ago 83m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:07:21.741 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 79m ago 82m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:07:21.741 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 79m ago 86m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:07:21.741 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 82m ago 88m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:07:21.742 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 82m ago 88m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:07:21.742 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 79m ago 86m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:07:21.742 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 82m ago 86m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:07:21.742 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 82m ago 83m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:07:21.742 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 79m ago 83m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:07:21.743 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 82m ago 85m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:07:21.743 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 82m ago 85m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:07:21.743 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 82m ago 85m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:07:21.743 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 82m ago 84m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:07:21.744 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 79m ago 84m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:07:21.744 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 79m ago 84m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:07:21.744 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 79m ago 84m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:07:21.744 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 79m ago 83m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:07:21.744 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 79m ago 83m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:07:22.079 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:07:22.079 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:07:22.079 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:07:22.079 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:07:22.080 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:07:22.080 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:07:22.080 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:07:22.080 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:07:22.081 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:07:22.081 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:07:22.081 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:07:22.081 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:07:22.081 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:07:22.082 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:07:22.082 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:07:22.082 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:07:22.082 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:07:22.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:22 smithi045 conmon[39718]: cluster 2021-07-31T03:07:20.945050+0000 2021-07-31T03:07:22.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:22 smithi045 conmon[39718]: mgr.x (mgr.24542) 3430 : cluster [DBG] pgmap v4411: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:22.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:22 smithi045 conmon[39718]: audit 2021- 2021-07-31T03:07:22.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:22 smithi045 conmon[39718]: 07-31T03:07:21.452497+0000 mgr.x (mgr.24542) 3431 : audit 2021-07-31T03:07:22.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:22 smithi045 conmon[39718]: [DBG] from='client.18840 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:07:22.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:22 smithi045 conmon[39718]: audit 2021-07-31T03 2021-07-31T03:07:22.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:22 smithi045 conmon[39718]: :07:21.733213+0000 mgr.x (mgr.24542 2021-07-31T03:07:22.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:22 smithi045 conmon[39718]: ) 3432 : audit [DBG] from='client.28786 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:07:22.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:22 smithi045 conmon[33655]: cluster 2021-07-31T03:07: 2021-07-31T03:07:22.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:22 smithi045 conmon[33655]: 20.945050+0000 mgr.x (mgr.24542) 3430 : cluster [DBG] pgmap v4411: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:22.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:22 smithi045 conmon[33655]: audit 2021-07-31T03:07:22.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:22 smithi045 conmon[33655]: 2021-07-31T03: 2021-07-31T03:07:22.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:22 smithi045 conmon[33655]: 07:21.452497+0000 mgr.x (mgr.24542) 3431 : audit 2021-07-31T03:07:22.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:22 smithi045 conmon[33655]: [DBG] from='client.18840 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:07:22.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:22 smithi045 conmon[33655]: audit 2021-07-31T03:07:22.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:22 smithi045 conmon[33655]: 2021-07-31T03 2021-07-31T03:07:22.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:22 smithi045 conmon[33655]: :07:21.733213+0000 mgr.x (mgr. 2021-07-31T03:07:22.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:22 smithi045 conmon[33655]: 24542) 3432 : audit [DBG] from='client.28786 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:07:22.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:22 smithi154 conmon[34553]: cluster 2021-07-31T03:07:20.945050+0000 mgr.x (mgr.24542) 3430 : cluster [DBG] pgmap v4411: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:22.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:22 smithi154 conmon[34553]: audit 2021-07-31T03:07:21.452497+0000 mgr.x (mgr.24542) 3431 : audit [DBG] from='client.18840 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:07:22.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:22 smithi154 conmon[34553]: audit 2021-07-31T03:07:21.733213+0000 mgr.x (mgr.24542) 3432 : audit [DBG] from='client.28786 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:07:23.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:23 smithi045 conmon[39718]: audit 2021-07-31T03:07:22.079490+0000 2021-07-31T03:07:23.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:23 smithi045 conmon[39718]: mon.c (mon.1) 177 : audit [DBG] from='client.? 172.21.15.45:0/3323870693' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:07:23.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:23 smithi045 conmon[33655]: audit 2021-07-31T03:07:22.079490+0000 mon.c 2021-07-31T03:07:23.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:23 smithi045 conmon[33655]: (mon.1) 177 : audit [DBG] from='client.? 172.21.15.45:0/3323870693' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:07:23.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:23 smithi154 conmon[34553]: audit 2021-07-31T03:07:22.079490+0000 mon.c (mon.1) 177 : audit [DBG] from='client.? 172.21.15.45:0/3323870693' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:07:24.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:24 smithi045 conmon[39718]: cluster 2021-07-31T03:07:22.945364+0000 mgr.x (mgr.24542 2021-07-31T03:07:24.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:24 smithi045 conmon[39718]: ) 3433 : cluster [DBG] pgmap v4412: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:24.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:24 smithi045 conmon[33655]: cluster 2021-07-31T03:07:22.945364+0000 mgr.x (mgr.24542 2021-07-31T03:07:24.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:24 smithi045 conmon[33655]: ) 3433 : cluster [DBG] pgmap v4412: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:24.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:24 smithi154 conmon[34553]: cluster 2021-07-31T03:07:22.945364+0000 mgr.x (mgr.24542) 3433 : cluster [DBG] pgmap v4412: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:26.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:26 smithi045 conmon[39718]: cluster 2021-07-31T03:07:24.945827+0000 mgr.x (mgr.24542) 3434 : cluster [DBG] 2021-07-31T03:07:26.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:26 smithi045 conmon[39718]: pgmap v4413: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:26.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:26 smithi045 conmon[33655]: cluster 2021-07-31T03:07:24.945827+0000 mgr.x (mgr.24542) 3434 : cluster [DBG] pgmap v4413: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:26.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:26 smithi154 conmon[34553]: cluster 2021-07-31T03:07:24.945827+0000 mgr.x (mgr.24542) 3434 : cluster [DBG] 2021-07-31T03:07:26.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:26 smithi154 conmon[34553]: pgmap v4413: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:28.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:28 smithi045 conmon[39718]: cluster 2021-07-31T03:07:26.946219+0000 mgr.x ( 2021-07-31T03:07:28.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:28 smithi045 conmon[39718]: mgr.24542) 3435 : cluster [DBG] pgmap v4414: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:28.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:28 smithi045 conmon[33655]: cluster 2021-07-31T03:07:26.946219+0000 mgr.x (mgr.24542) 3435 : cluster [DBG] pgmap v4414: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:28.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:28 smithi154 conmon[34553]: cluster 2021-07-31T03:07:26.946219+0000 mgr.x (mgr.24542) 3435 : cluster [DBG] pgmap v4414: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:29.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:29 smithi045 conmon[33655]: audit 2021-07-31T03:07:28.253284+0000 mon.b (mon.2) 475 : audit [INF] 2021-07-31T03:07:29.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:29 smithi045 conmon[33655]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:07:29.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:29 smithi045 conmon[33655]: audit 2021-07-31T03:07:28.253607+0000 mon.b (mon.2) 476 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:07:29.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:29 smithi045 conmon[33655]: audit 2021-07-31T03:07 2021-07-31T03:07:29.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:29 smithi045 conmon[33655]: :28.254031+0000 mon.a (mon.0) 3254 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:07:29.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:29 smithi045 conmon[33655]: audit 2021-07-31T03:07:28.254514+0000 mon.a (mon.0) 3255 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:07:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:29 smithi045 conmon[39718]: audit 2021-07-31T03:07:28.253284+0000 mon.b (mon.2) 475 : audit 2021-07-31T03:07:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:29 smithi045 conmon[39718]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:07:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:29 smithi045 conmon[39718]: audit 2021-07-31T03:07:28.253607+0000 mon.b (mon.2) 2021-07-31T03:07:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:29 smithi045 conmon[39718]: 476 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:07:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:29 smithi045 conmon[39718]: audit 2021-07-31T03:07:28.254031+0000 mon.a 2021-07-31T03:07:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:29 smithi045 conmon[39718]: (mon.0) 3254 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:07:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:29 smithi045 conmon[39718]: audit 2021-07-31T03:07 2021-07-31T03:07:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:29 smithi045 conmon[39718]: :28.254514+0000 mon.a (mon.0) 3255 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:07:29.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:29 smithi045 conmon[39718]: 2021-07-31T03:07:29.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:29 smithi154 conmon[34553]: audit 2021-07-31T03:07:28.253284+0000 mon.b (mon.2) 475 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:07:29.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:29 smithi154 conmon[34553]: audit 2021-07-31T03:07:28.253607+0000 mon.b (mon.2) 476 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:07:29.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:29 smithi154 conmon[34553]: audit 2021-07-31T03:07:28.254031 2021-07-31T03:07:29.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:29 smithi154 conmon[34553]: +0000 mon.a (mon.0) 3254 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:07:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:29 smithi154 conmon[34553]: audit 2021- 2021-07-31T03:07:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:29 smithi154 conmon[34553]: 07-31T03:07:28 2021-07-31T03:07:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:29 smithi154 conmon[34553]: .254514+0000 mon.a (mon.0) 3255 : 2021-07-31T03:07:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:29 smithi154 conmon[34553]: audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:07:30.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:30 smithi154 conmon[34553]: cluster 2021-07-31T03:07:28.946644+0000 mgr.x (mgr.24542) 3436 : cluster [DBG] pgmap v4415: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:30.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:30 smithi045 conmon[39718]: cluster 2021-07-31T03:07:28.946644+0000 mgr.x 2021-07-31T03:07:30.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:30 smithi045 conmon[39718]: (mgr.24542) 3436 : cluster [DBG] pgmap v4415: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:30.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:30 smithi045 conmon[33655]: cluster 2021-07-31T03:07:28.946644+0000 mgr.x (mgr.24542) 3436 : cluster [DBG] pgmap v4415: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:30.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:30 smithi045 conmon[33655]: 2021-07-31T03:07:31.753 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: debug 2021-07-31T03:07:31.421+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:07:31.753 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:07:31.753 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:07:31.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: yield 2021-07-31T03:07:31.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:07:31.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:07:31.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:07:31.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:07:31.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:07:31.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:07:31.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:07:31.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:07:31.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:07:31.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: task.result() 2021-07-31T03:07:31.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:07:31.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: debug 2021-07-31T03:07:31.432+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:07:31.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:07:31.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:07:31.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: yield 2021-07-31T03:07:31.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:07:31.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:07:31.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:07:31.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:07:31.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:07:31.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:07:31.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:07:31.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:07:31.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:07:31.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: task.result() 2021-07-31T03:07:31.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:07:31.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: debug 2021-07-31T03:07:31.458+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:07:31.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:07:31.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:07:31.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: yield 2021-07-31T03:07:31.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:07:31.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:07:31.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:07:31.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:07:31.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:07:31.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:07:31.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:07:31.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:07:31.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:07:31.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: task.result() 2021-07-31T03:07:31.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:07:31.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: debug 2021-07-31T03:07:31.459+0000 7fcb94162700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:07:31.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:07:31.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:07:31.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: yield 2021-07-31T03:07:31.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:07:31.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:07:31.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:07:31.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:07:31.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:07:31.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:07:31.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:07:31.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:07:31.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:07:31.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: task.result() 2021-07-31T03:07:31.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:07:31.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: 2021-07-31T03:07:31.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:07:31.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: 2021-07-31T03:07:31.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:07:31.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:07:31.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:07:31.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:07:31.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:07:31.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:07:31.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:07:31.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:07:31.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:07:31.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:07:31.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:07:31.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:07:31.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:07:31.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:07:31.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:07:31.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:07:31.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:07:31.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:07:31.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: raise 2021-07-31T03:07:31.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:07:31.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:07:31.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:07:31.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:07:31.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:07:31.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: debug 2021-07-31T03:07:31.468+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:07:31.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:07:31.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:07:31.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: yield 2021-07-31T03:07:31.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:07:31.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:07:31.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:07:31.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:07:31.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:07:31.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:07:31.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:07:31.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:07:31.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:07:31.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: task.result() 2021-07-31T03:07:31.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:07:31.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: debug 2021-07-31T03:07:31.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: 2021-07-31T03:07:31.469+0000 7fcb93961700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:07:31.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:07:31.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:07:31.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: yield 2021-07-31T03:07:31.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:07:31.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:07:31.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:07:31.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:07:31.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:07:31.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:07:31.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:07:31.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:07:31.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:07:31.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: task.result() 2021-07-31T03:07:31.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:07:31.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: 2021-07-31T03:07:31.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:07:31.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: 2021-07-31T03:07:31.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:07:31.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:07:31.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:07:31.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:07:31.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:07:31.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:07:31.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:07:31.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:07:31.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:07:31.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:07:31.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:07:31.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:07:31.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:07:31.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:07:31.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:07:31.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:07:31.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:07:31.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:07:31.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: raise 2021-07-31T03:07:31.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:07:31.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:07:31.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:07:31.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:07:31.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:07:31 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:07:32.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: cluster 2021-07-31T03:07:30.947108+0000 mgr.x (mgr.24542) 3437 : cluster 2021-07-31T03:07:32.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: [DBG] pgmap v4416: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:32.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: audit 2021-07-31T03:07:31.375949+0000 mon.b (mon.2) 477 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:07:32.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: audit 2021-07-31T03:07:31.377416+0000 mon.b ( 2021-07-31T03:07:32.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: mon.2) 478 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:07:32.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: audit 2021 2021-07-31T03:07:32.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: -07-31T03:07:31.378445+0000 mon.b (mon 2021-07-31T03:07:32.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: .2) 479 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:07:32.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: cephadm 2021- 2021-07-31T03:07:32.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: 07-31T03:07:31.379175+0000 mgr.x (mgr.24542) 3438 : 2021-07-31T03:07:32.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:07:32.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: cephadm 2021-07 2021-07-31T03:07:32.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: -31T03:07:31.422265+0000 mgr.x (mgr. 2021-07-31T03:07:32.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: 24542) 3439 : cephadm [ERR] 2021-07-31T03:07:32.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:07:32.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:07:32.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: yield 2021-07-31T03:07:32.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:07:32.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:07:32.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:07:32.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:07:32.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:07:32.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:07:32.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:07:32.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:07:32.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:07:32.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: task.result() 2021-07-31T03:07:32.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:07:32.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: cephadm 2021-07-31T03:07:31.422700+0000 mgr.x (mgr.24542) 3440 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:07:32.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: cephadm 2021-07-31T03:07:31.433717+0000 2021-07-31T03:07:32.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: mgr.x (mgr.24542) 3441 : cephadm [ERR] 2021-07-31T03:07:32.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:07:32.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:07:32.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: yield 2021-07-31T03:07:32.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:07:32.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:07:32.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:07:32.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:07:32.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:07:32.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:07:32.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:07:32.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:07:32.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:07:32.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: task.result() 2021-07-31T03:07:32.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:07:32.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: cephadm 2021-07-31T03:07:31.434557+0000 mgr.x (mgr.24542) 3442 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:07:32.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: cephadm 2021-07-31T03:07:31.459837+0000 mgr.x (mgr 2021-07-31T03:07:32.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: .24542) 3443 : cephadm [ERR] 2021-07-31T03:07:32.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:07:32.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:07:32.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: yield 2021-07-31T03:07:32.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:07:32.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:07:32.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:07:32.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:07:32.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:07:32.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:07:32.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:07:32.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:07:32.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:07:32.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: task.result() 2021-07-31T03:07:32.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:07:32.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: cephadm 2021-07-31T03:07:31.460311+0000 mgr.x (mgr.24542) 3444 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:07:32.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:07:32.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:07:32.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: yield 2021-07-31T03:07:32.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:07:32.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:07:32.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:07:32.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:07:32.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:07:32.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:07:32.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:07:32.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:07:32.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:07:32.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: task.result() 2021-07-31T03:07:32.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:07:32.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: 2021-07-31T03:07:32.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:07:32.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: 2021-07-31T03:07:32.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:07:32.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:07:32.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:07:32.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:07:32.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:07:32.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:07:32.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:07:32.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:07:32.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:07:32.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:07:32.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:07:32.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:07:32.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:07:32.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:07:32.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:07:32.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:07:32.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:07:32.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:07:32.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: raise 2021-07-31T03:07:32.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:07:32.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:07:32.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:07:32.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:07:32.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:07:32.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: cephadm 2021-07-31T03:07:31.469656+0000 mgr.x (mgr.24542 2021-07-31T03:07:32.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: ) 3445 : cephadm [ERR] 2021-07-31T03:07:32.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:07:32.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:07:32.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: yield 2021-07-31T03:07:32.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:07:32.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:07:32.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:07:32.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:07:32.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:07:32.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:07:32.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:07:32.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:07:32.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:07:32.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: task.result() 2021-07-31T03:07:32.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:07:32.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: cephadm 2021-07-31T03:07:31.470230+0000 mgr.x (mgr.24542) 3446 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:07:32.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:07:32.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:07:32.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: yield 2021-07-31T03:07:32.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:07:32.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:07:32.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:07:32.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:07:32.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:07:32.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:07:32.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:07:32.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:07:32.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:07:32.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: task.result() 2021-07-31T03:07:32.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:07:32.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: 2021-07-31T03:07:32.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:07:32.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: 2021-07-31T03:07:32.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:07:32.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:07:32.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:07:32.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:07:32.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:07:32.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:07:32.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:07:32.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:07:32.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:07:32.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:07:32.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:07:32.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:07:32.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:07:32.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:07:32.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:07:32.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:07:32.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:07:32.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:07:32.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: raise 2021-07-31T03:07:32.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:07:32.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:07:32.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:07:32.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:07:32.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:32 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:07:32.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: cluster 2021-07-31T03:07:30.947108+0000 mgr.x (mgr.24542 2021-07-31T03:07:32.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: ) 3437 : cluster [DBG] pgmap v4416: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:32.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: audit 2021-07-31T03:07:31 2021-07-31T03:07:32.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: .375949+0000 mon.b (mon.2) 477 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:07:32.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: audit 2021 2021-07-31T03:07:32.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: -07-31T03:07:31.377416 2021-07-31T03:07:32.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: +0000 mon.b (mon.2) 478 : audit 2021-07-31T03:07:32.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:07:32.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: audit 2021-07-31T03 2021-07-31T03:07:32.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: :07:31.378445+0000 mon.b (mon. 2021-07-31T03:07:32.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: 2) 479 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:07:32.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: cephadm 2021-07-31T03:07:31.379175+0000 2021-07-31T03:07:32.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: mgr.x (mgr.24542) 3438 : cephadm [WRN] 2021-07-31T03:07:32.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:07:32.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: cephadm 2021-07-31T03:07 2021-07-31T03:07:32.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: :31.422265+0000 mgr.x (mgr.24542) 2021-07-31T03:07:32.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: 3439 : cephadm [ERR] 2021-07-31T03:07:32.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:07:32.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:07:32.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: yield 2021-07-31T03:07:32.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:07:32.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:07:32.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:07:32.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:07:32.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:07:32.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:07:32.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:07:32.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:07:32.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:07:32.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: task.result() 2021-07-31T03:07:32.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:07:32.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: cephadm 2021-07-31T03:07:31.422700+0000 mgr.x (mgr.24542) 3440 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:07:32.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: cephadm 2021-07-31T03:07: 2021-07-31T03:07:32.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: 31.433717+0000 mgr.x (mgr.24542) 3441 : cephadm [ERR] 2021-07-31T03:07:32.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: 2021-07-31T03:07:32.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:07:32.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:07:32.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: yield 2021-07-31T03:07:32.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:07:32.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:07:32.546 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: cluster 2021-07-31T03:07:30.947108+0000 mgr.x (mgr.24542) 3437 : cluster [DBG] pgmap v4416: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:32.546 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: audit 2021-07-31T03:07:31.375949+0000 mon.b (mon.2) 2021-07-31T03:07:32.546 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: 477 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:07:32.547 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: audit 2021- 2021-07-31T03:07:32.547 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: 07-31T03:07:31.377416+0000 2021-07-31T03:07:32.547 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: mon.b (mon.2) 478 2021-07-31T03:07:32.547 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:07:32.547 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: audit 2021-07- 2021-07-31T03:07:32.548 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: 31T03:07:31.378445+0000 mon.b (mon.2) 479 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:07:32.548 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: cephadm 2021-07-31T03:07:31.379175+0000 mgr.x (mgr. 2021-07-31T03:07:32.548 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: 24542) 3438 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:07:32.548 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: cephadm 2021-07-31T03:07:32.548 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: 2021-07-31T03:07:31.422265+0000 mgr.x 2021-07-31T03:07:32.549 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: (mgr.24542) 3439 : cephadm [ERR] 2021-07-31T03:07:32.549 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:07:32.549 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:07:32.549 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: yield 2021-07-31T03:07:32.549 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:07:32.550 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:07:32.550 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:07:32.550 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:07:32.550 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:07:32.550 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:07:32.551 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:07:32.551 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:07:32.551 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:07:32.551 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: task.result() 2021-07-31T03:07:32.551 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:07:32.552 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: cephadm 2021-07-31T03:07:31.422700+0000 mgr.x (mgr.24542) 3440 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:07:32.552 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: cephadm 2021 2021-07-31T03:07:32.552 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: -07-31T03:07:31.433717+0000 mgr.x (mgr.24542) 3441 : cephadm 2021-07-31T03:07:32.552 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: [ERR] 2021-07-31T03:07:32.552 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:07:32.553 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:07:32.553 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: yield 2021-07-31T03:07:32.553 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:07:32.553 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:07:32.553 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:07:32.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:07:32.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:07:32.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:07:32.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:07:32.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:07:32.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:07:32.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:07:32.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: task.result() 2021-07-31T03:07:32.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:07:32.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: cephadm 2021-07-31T03:07:31.434557+0000 mgr.x (mgr.24542) 3442 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:07:32.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: cephadm 2021-07-31T03:07:31.459837+0000 mgr.x 2021-07-31T03:07:32.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: (mgr.24542) 3443 : cephadm [ERR] 2021-07-31T03:07:32.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:07:32.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:07:32.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: yield 2021-07-31T03:07:32.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:07:32.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:07:32.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:07:32.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:07:32.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:07:32.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:07:32.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:07:32.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:07:32.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:07:32.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: task.result() 2021-07-31T03:07:32.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:07:32.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: cephadm 2021-07-31T03:07:31.460311+0000 mgr.x (mgr.24542) 3444 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:07:32.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:07:32.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:07:32.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: yield 2021-07-31T03:07:32.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:07:32.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:07:32.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:07:32.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:07:32.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:07:32.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:07:32.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:07:32.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:07:32.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:07:32.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:07:32.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:07:32.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:07:32.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:07:32.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:07:32.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:07:32.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: task.result() 2021-07-31T03:07:32.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:07:32.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: cephadm 2021-07-31T03:07:31.434557+0000 mgr.x (mgr.24542) 3442 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:07:32.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: cephadm 2021-07-31T03:07:31.459837+0000 mgr.x (mgr.24542) 3443 : cephadm 2021-07-31T03:07:32.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: [ERR] 2021-07-31T03:07:32.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:07:32.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:07:32.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: yield 2021-07-31T03:07:32.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:07:32.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:07:32.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:07:32.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:07:32.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:07:32.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:07:32.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:07:32.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:07:32.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:07:32.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: task.result() 2021-07-31T03:07:32.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:07:32.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: cephadm 2021-07-31T03:07:31.460311+0000 mgr.x (mgr.24542) 3444 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:07:32.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:07:32.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:07:32.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: yield 2021-07-31T03:07:32.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:07:32.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:07:32.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:07:32.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:07:32.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:07:32.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:07:32.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:07:32.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:07:32.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:07:32.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: task.result() 2021-07-31T03:07:32.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:07:32.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: 2021-07-31T03:07:32.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:07:32.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: 2021-07-31T03:07:32.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:07:32.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:07:32.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:07:32.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:07:32.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:07:32.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:07:32.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:07:32.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:07:32.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:07:32.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:07:32.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:07:32.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:07:32.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:07:32.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:07:32.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:07:32.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:07:32.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:07:32.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:07:32.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: raise 2021-07-31T03:07:32.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:07:32.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:07:32.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:07:32.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:07:32.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:07:32.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: cephadm 2021-07-31T03:07:31.469656+0000 mgr.x (mgr 2021-07-31T03:07:32.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: .24542) 3445 : cephadm [ERR] 2021-07-31T03:07:32.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:07:32.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:07:32.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: yield 2021-07-31T03:07:32.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:07:32.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:07:32.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:07:32.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:07:32.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:07:32.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:07:32.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:07:32.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:07:32.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:07:32.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: task.result() 2021-07-31T03:07:32.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:07:32.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: 2021-07-31T03:07:32.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:07:32.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: 2021-07-31T03:07:32.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:07:32.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:07:32.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:07:32.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:07:32.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:07:32.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:07:32.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:07:32.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:07:32.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:07:32.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:07:32.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:07:32.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:07:32.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:07:32.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:07:32.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:07:32.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:07:32.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:07:32.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:07:32.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: raise 2021-07-31T03:07:32.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:07:32.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:07:32.585 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:07:32.585 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:07:32.585 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:07:32.585 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: cephadm 2021- 2021-07-31T03:07:32.585 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: 07-31T03:07:31.469656+0000 mgr.x (mgr.24542) 3445 : cephadm [ERR] 2021-07-31T03:07:32.586 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:07:32.586 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:07:32.586 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: yield 2021-07-31T03:07:32.586 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:07:32.586 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:07:32.587 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:07:32.587 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:07:32.587 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:07:32.587 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:07:32.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:07:32.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:07:32.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: task.result() 2021-07-31T03:07:32.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:07:32.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: cephadm 2021-07-31T03:07:31.470230+0000 mgr.x (mgr.24542) 3446 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:07:32.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:07:32.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:07:32.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: yield 2021-07-31T03:07:32.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:07:32.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:07:32.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:07:32.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:07:32.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:07:32.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:07:32.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:07:32.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:07:32.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:07:32.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: task.result() 2021-07-31T03:07:32.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:07:32.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: 2021-07-31T03:07:32.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:07:32.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: 2021-07-31T03:07:32.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:07:32.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:07:32.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:07:32.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:07:32.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:07:32.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:07:32.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:07:32.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:07:32.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:07:32.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:07:32.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:07:32.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:07:32.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:07:32.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:07:32.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:07:32.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:07:32.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:07:32.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:07:32.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: raise 2021-07-31T03:07:32.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:07:32.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:07:32.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:07:32.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:07:32.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:07:32.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:07:32.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: task.result() 2021-07-31T03:07:32.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:07:32.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: cephadm 2021-07-31T03:07:31.470230+0000 mgr.x (mgr.24542) 3446 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:07:32.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:07:32.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:07:32.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: yield 2021-07-31T03:07:32.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:07:32.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:07:32.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:07:32.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:07:32.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:07:32.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:07:32.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:07:32.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:07:32.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:07:32.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: task.result() 2021-07-31T03:07:32.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:07:32.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: 2021-07-31T03:07:32.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:07:32.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: 2021-07-31T03:07:32.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:07:32.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:07:32.603 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:07:32.603 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:07:32.603 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:07:32.603 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:07:32.603 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:07:32.603 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:07:32.604 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:07:32.604 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:07:32.604 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:07:32.604 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:07:32.604 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:07:32.605 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:07:32.605 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:07:32.605 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:07:32.605 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:07:32.605 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:07:32.606 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: raise 2021-07-31T03:07:32.606 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:07:32.606 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:07:32.606 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:07:32.607 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:07:32.607 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:32 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:07:34.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:33 smithi154 conmon[34553]: cluster 2021-07-31T03:07:32.947451+0000 mgr.x (mgr.24542) 3447 : cluster [DBG] 2021-07-31T03:07:34.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:33 smithi154 conmon[34553]: pgmap v4417: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:34.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:33 smithi045 conmon[39718]: cluster 2021-07-31T03:07:32.947451+0000 mgr.x (mgr. 2021-07-31T03:07:34.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:33 smithi045 conmon[39718]: 24542) 3447 : cluster [DBG] pgmap v4417: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:34.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:33 smithi045 conmon[33655]: cluster 2021-07-31T03:07:32.947451+0000 mgr.x (mgr. 2021-07-31T03:07:34.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:33 smithi045 conmon[33655]: 24542) 3447 : cluster [DBG] pgmap v4417: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:36.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:36 smithi045 conmon[33655]: cluster 2021-07-31T03:07:34.948046+0000 mgr.x 2021-07-31T03:07:36.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:36 smithi045 conmon[33655]: (mgr.24542) 3448 : cluster [DBG] pgmap v4418: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:36.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:36 smithi045 conmon[39718]: cluster 2021-07-31T03:07:34.948046+0000 mgr.x (mgr.24542) 3448 2021-07-31T03:07:36.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:36 smithi045 conmon[39718]: : cluster [DBG] pgmap v4418: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:36.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:36 smithi154 conmon[34553]: cluster 2021-07-31T03:07:34.948046+0000 mgr.x (mgr.24542) 3448 : cluster 2021-07-31T03:07:36.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:36 smithi154 conmon[34553]: [DBG] pgmap v4418: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:38.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:38 smithi045 conmon[39718]: cluster 2021-07-31T03:07:36.948481+0000 mgr.x ( 2021-07-31T03:07:38.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:38 smithi045 conmon[39718]: mgr.24542) 3449 : cluster [DBG] pgmap v4419: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:38.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:38 smithi045 conmon[33655]: cluster 2021-07-31T03:07:36.948481+0000 mgr.x (mgr.24542) 3449 : cluster 2021-07-31T03:07:38.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:38 smithi045 conmon[33655]: [DBG] pgmap v4419: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:38.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:38 smithi154 conmon[34553]: cluster 2021-07-31T03:07:36.948481+0000 mgr.x (mgr.24542) 3449 : cluster [DBG] pgmap v4419: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:40.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:40 smithi045 conmon[39718]: cluster 2021-07-31T03:07:38.948964+0000 2021-07-31T03:07:40.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:40 smithi045 conmon[39718]: mgr.x (mgr.24542) 3450 : cluster [DBG] pgmap v4420: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:40.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:40 smithi045 conmon[33655]: cluster 2021-07-31T03:07:38.948964+0000 mgr.x (mgr.24542) 2021-07-31T03:07:40.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:40 smithi045 conmon[33655]: 3450 : cluster [DBG] pgmap v4420: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:40.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:40 smithi154 conmon[34553]: cluster 2021-07-31T03:07:38.948964+0000 mgr.x (mgr.24542) 3450 : cluster [DBG] pgmap v4420: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:42.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:42 smithi045 conmon[39718]: cluster 2021-07-31T03:07:40.949355+0000 mgr.x ( 2021-07-31T03:07:42.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:42 smithi045 conmon[39718]: mgr.24542) 3451 : cluster [DBG] pgmap v4421: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:42.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:42 smithi045 conmon[33655]: cluster 2021-07-31T03:07:40.949355+0000 mgr.x (mgr.24542 2021-07-31T03:07:42.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:42 smithi045 conmon[33655]: ) 3451 : cluster [DBG] pgmap v4421: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:42.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:42 smithi154 conmon[34553]: cluster 2021-07-31T03:07:40.949355+0000 mgr.x (mgr.24542) 3451 : cluster [DBG] pgmap v4421: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:44.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:43 smithi154 conmon[34553]: cluster 2021-07-31T03:07:42.949734+0000 mgr.x (mgr.24542) 3452 : cluster [DBG] pgmap v4422: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:44.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:43 smithi045 conmon[39718]: cluster 2021-07-31T03:07:42. 2021-07-31T03:07:44.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:43 smithi045 conmon[39718]: 949734+0000 mgr.x (mgr.24542) 3452 : cluster [DBG] pgmap v4422: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:44.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:43 smithi045 conmon[33655]: cluster 2021-07-31T03:07:42.949734+0000 mgr.x (mgr.24542) 3452 : cluster 2021-07-31T03:07:44.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:43 smithi045 conmon[33655]: [DBG] pgmap v4422: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:46.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:46 smithi045 conmon[33655]: cluster 2021-07-31T03:07:44.950146+0000 mgr.x 2021-07-31T03:07:46.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:46 smithi045 conmon[33655]: (mgr.24542) 3453 : cluster [DBG] pgmap v4423: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:46.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:46 smithi045 conmon[39718]: cluster 2021-07-31T03:07:44.950146+0000 mgr.x (mgr.24542) 3453 : cluster [DBG] pgmap v4423: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:46.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:46 smithi154 conmon[34553]: cluster 2021-07-31T03:07:44.950146+0000 mgr.x (mgr.24542) 3453 : cluster [DBG] pgmap v4423: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:48.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:48 smithi045 conmon[33655]: cluster 2021-07-31T03:07:46.950542+0000 mgr.x (mgr.24542) 3454 2021-07-31T03:07:48.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:48 smithi045 conmon[33655]: : cluster [DBG] pgmap v4424: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:48.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:48 smithi045 conmon[39718]: cluster 2021-07-31T03:07:46.950542+0000 mgr.x (mgr.24542 2021-07-31T03:07:48.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:48 smithi045 conmon[39718]: ) 3454 : cluster [DBG] pgmap v4424: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:48.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:48 smithi154 conmon[34553]: cluster 2021-07-31T03:07:46.950542+0000 mgr.x (mgr.24542) 3454 : cluster [DBG] pgmap v4424: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:50.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:50 smithi045 conmon[39718]: cluster 2021-07-31T03:07:48.951013+0000 mgr.x (mgr.24542) 3455 : cluster [DBG] pgmap v4425: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:50.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:50 smithi045 conmon[33655]: cluster 2021-07-31T03:07:48.951013+0000 mgr.x (mgr.24542) 3455 : cluster [DBG] pgmap v4425: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:50.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:50 smithi154 conmon[34553]: cluster 2021-07-31T03:07:48.951013+0000 mgr.x (mgr.24542) 3455 : cluster [DBG] 2021-07-31T03:07:50.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:50 smithi154 conmon[34553]: pgmap v4425: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:52.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:52 smithi045 conmon[39718]: cluster 2021-07-31T03:07:50.951503+0000 mgr.x (mgr. 2021-07-31T03:07:52.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:52 smithi045 conmon[39718]: 24542) 3456 : cluster [DBG] pgmap v4426: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:52.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:52 smithi045 conmon[33655]: cluster 2021-07-31T03:07:50.951503+0000 mgr.x (mgr. 2021-07-31T03:07:52.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:52 smithi045 conmon[33655]: 24542) 3456 : cluster [DBG] pgmap v4426: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:52.393 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:07:52.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:52 smithi154 conmon[34553]: cluster 2021-07-31T03:07:50.951503+0000 mgr.x (mgr.24542) 3456 : cluster [DBG] 2021-07-31T03:07:52.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:52 smithi154 conmon[34553]: pgmap v4426: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:52.670 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:07:52.670 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 82m ago 83m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:07:52.670 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 80m ago 83m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:07:52.671 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 80m ago 86m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:07:52.671 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 82m ago 88m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:07:52.671 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 82m ago 89m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:07:52.671 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 80m ago 87m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:07:52.672 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 82m ago 87m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:07:52.672 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 82m ago 83m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:07:52.672 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 80m ago 83m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:07:52.672 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 82m ago 86m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:07:52.672 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 82m ago 86m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:07:52.673 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 82m ago 85m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:07:52.673 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 82m ago 85m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:07:52.673 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 80m ago 85m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:07:52.673 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 80m ago 84m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:07:52.673 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 80m ago 84m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:07:52.674 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 80m ago 84m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:07:52.674 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 80m ago 83m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:07:53.008 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:07:53.009 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:07:53.009 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:07:53.009 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:07:53.009 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:07:53.010 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:07:53.010 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:07:53.010 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:07:53.010 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:07:53.011 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:07:53.011 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:07:53.011 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:07:53.011 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:07:53.011 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:07:53.012 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:07:53.012 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:07:53.012 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:07:53.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:53 smithi045 conmon[33655]: audit 2021-07-31T03:07:52.381988+0000 mgr.x 2021-07-31T03:07:53.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:53 smithi045 conmon[33655]: (mgr.24542) 3457 : audit [DBG] from='client.18858 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:07:53.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:53 smithi045 conmon[33655]: audit 2021-07-31T03:07:52.662583+0000 2021-07-31T03:07:53.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:53 smithi045 conmon[33655]: mgr.x (mgr.24542) 3458 : audit [DBG] from='client.28804 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:07:53.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:53 smithi045 conmon[33655]: audit 2021-07-31T03:07:53.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:53 smithi045 conmon[33655]: 2021-07-31 2021-07-31T03:07:53.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:53 smithi045 conmon[33655]: T03:07:53. 2021-07-31T03:07:53.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:53 smithi045 conmon[33655]: 009271+0000 mon.a (mon. 2021-07-31T03:07:53.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:53 smithi045 conmon[33655]: 0) 3256 : audit [DBG] 2021-07-31T03:07:53.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:53 smithi045 conmon[33655]: from='client.? 172.21.15.45:0/4200724981' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:07:53.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:53 smithi045 conmon[39718]: audit 2021-07-31T03:07:52.381988+0000 mgr.x (mgr.24542) 2021-07-31T03:07:53.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:53 smithi045 conmon[39718]: 3457 : audit [DBG] from='client.18858 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:07:53.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:53 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T03:07:53.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:53 smithi045 conmon[39718]: T03:07:52.662583+0000 2021-07-31T03:07:53.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:53 smithi045 conmon[39718]: mgr.x (mgr.24542) 3458 2021-07-31T03:07:53.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:53 smithi045 conmon[39718]: : audit [DBG] from='client.28804 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:07:53.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:53 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T03:07:53.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:53 smithi045 conmon[39718]: T03:07:53.009271+0000 2021-07-31T03:07:53.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:53 smithi045 conmon[39718]: mon.a (mon.0) 3256 : audit 2021-07-31T03:07:53.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:53 smithi045 conmon[39718]: [DBG] from='client.? 172.21.15.45:0/4200724981' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:07:53.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:53 smithi154 conmon[34553]: audit 2021-07-31T03:07:52.381988+0000 mgr.x (mgr.24542) 2021-07-31T03:07:53.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:53 smithi154 conmon[34553]: 3457 : audit [DBG] from='client.18858 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:07:53.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:53 smithi154 conmon[34553]: audit 2021-07-31T03:07:52.662583+0000 mgr.x (mgr.24542) 3458 : audit [DBG] from='client.28804 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:07:53.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:53 smithi154 conmon[34553]: audit 2021-07-31T03:07:53.009271+0000 mon.a (mon.0) 3256 : audit [DBG] from='client.? 172.21.15.45:0/4200724981' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:07:54.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:54 smithi045 conmon[33655]: cluster 2021-07-31T03:07:52.952046+0000 mgr.x (mgr.24542) 3459 : 2021-07-31T03:07:54.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:54 smithi045 conmon[33655]: cluster [DBG] pgmap v4427: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:54.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:54 smithi045 conmon[39718]: cluster 2021-07-31T03:07:52 2021-07-31T03:07:54.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:54 smithi045 conmon[39718]: .952046+0000 mgr.x (mgr.24542) 3459 : cluster [DBG] pgmap v4427: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:54.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:54 smithi154 conmon[34553]: cluster 2021-07-31T03:07:52.952046+0000 mgr.x (mgr.24542) 3459 : cluster 2021-07-31T03:07:54.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:54 smithi154 conmon[34553]: [DBG] pgmap v4427: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:56.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:56 smithi045 conmon[33655]: cluster 2021-07-31T03:07:54.952676+0000 mgr.x (mgr.24542) 3460 : cluster [DBG] pgmap v4428: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:56.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:56 smithi045 conmon[39718]: cluster 2021-07-31T03:07:54.952676+0000 mgr.x (mgr. 2021-07-31T03:07:56.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:56 smithi045 conmon[39718]: 24542) 3460 : cluster [DBG] pgmap v4428: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:56.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:56 smithi154 conmon[34553]: cluster 2021-07-31T03:07:54.952676+0000 mgr.x (mgr.24542) 3460 : cluster 2021-07-31T03:07:56.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:56 smithi154 conmon[34553]: [DBG] pgmap v4428: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:58.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:58 smithi154 conmon[34553]: cluster 2021-07-31T03:07:56.953120+0000 mgr.x (mgr.24542) 3461 : cluster [DBG] 2021-07-31T03:07:58.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:07:58 smithi154 conmon[34553]: pgmap v4429: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:58.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:58 smithi045 conmon[33655]: cluster 2021-07-31T03:07:56.953120+0000 mgr.x (mgr.24542) 3461 : 2021-07-31T03:07:58.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:07:58 smithi045 conmon[33655]: cluster [DBG] pgmap v4429: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:07:58.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:58 smithi045 conmon[39718]: cluster 2021-07-31T03:07:56.953120+0000 mgr.x (mgr 2021-07-31T03:07:58.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:07:58 smithi045 conmon[39718]: .24542) 3461 : cluster [DBG] pgmap v4429: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:00.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:00 smithi154 conmon[34553]: cluster 2021-07-31T03:07:58.953724+0000 mgr.x (mgr.24542) 3462 : cluster [DBG] pgmap v4430: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:00.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:00 smithi045 conmon[39718]: cluster 2021-07-31T03:07:58.953724+0000 mgr.x ( 2021-07-31T03:08:00.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:00 smithi045 conmon[39718]: mgr.24542) 3462 : cluster [DBG] pgmap v4430: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:00.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:00 smithi045 conmon[33655]: cluster 2021-07-31T03:07:58.953724+0000 mgr.x (mgr.24542) 3462 : 2021-07-31T03:08:00.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:00 smithi045 conmon[33655]: cluster [DBG] pgmap v4430: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:02.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:02 smithi154 conmon[34553]: cluster 2021-07-31T03:08:00.954344+0000 mgr.x (mgr.24542) 3463 : cluster [DBG] pgmap v4431: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:02.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:02 smithi045 conmon[39718]: cluster 2021-07-31T03:08:00.954344+0000 mgr.x ( 2021-07-31T03:08:02.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:02 smithi045 conmon[39718]: mgr.24542) 3463 : cluster [DBG] pgmap v4431: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:02.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:02 smithi045 conmon[33655]: cluster 2021-07-31T03:08:00.954344+0000 mgr.x (mgr.24542) 3463 : cluster [DBG] 2021-07-31T03:08:02.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:02 smithi045 conmon[33655]: pgmap v4431: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:04.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:03 smithi154 conmon[34553]: cluster 2021-07-31T03:08:02.954722+0000 mgr.x (mgr.24542) 3464 : cluster [DBG] pgmap v4432: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:04.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:03 smithi045 conmon[33655]: cluster 2021-07-31T03:08:02.954722+0000 mgr.x (mgr.24542) 2021-07-31T03:08:04.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:03 smithi045 conmon[33655]: 3464 : cluster [DBG] pgmap v4432: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:04.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:03 smithi045 conmon[39718]: cluster 2021-07-31T03:08:02.954722+0000 mgr.x (mgr.24542) 3464 : cluster [DBG] 2021-07-31T03:08:04.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:03 smithi045 conmon[39718]: pgmap v4432: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:06.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:06 smithi045 conmon[33655]: cluster 2021-07-31T03:08:04.955208+0000 mgr.x (mgr.24542) 3465 : cluster [DBG] pgmap v4433: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:06.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:06 smithi045 conmon[39718]: cluster 2021-07-31T03:08:04.955208+0000 mgr.x ( 2021-07-31T03:08:06.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:06 smithi045 conmon[39718]: mgr.24542) 3465 : cluster [DBG] pgmap v4433: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:06.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:06 smithi154 conmon[34553]: cluster 2021-07-31T03:08:04.955208+0000 mgr.x (mgr.24542) 3465 : cluster [DBG] pgmap v4433: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:08.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:08 smithi045 conmon[33655]: cluster 2021-07-31T03:08:06.955614+0000 mgr.x (mgr.24542) 3466 : cluster [DBG] pgmap v4434: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:08.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:08 smithi045 conmon[39718]: cluster 2021-07-31T03:08:06.955614+0000 mgr.x (mgr. 2021-07-31T03:08:08.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:08 smithi045 conmon[39718]: 24542) 3466 : cluster [DBG] pgmap v4434: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:08.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:08 smithi154 conmon[34553]: cluster 2021-07-31T03:08:06.955614+0000 mgr.x (mgr.24542) 3466 : cluster [DBG] pgmap v4434: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:10.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:10 smithi045 conmon[39718]: cluster 2021-07-31T03:08:08.956090+0000 mgr.x 2021-07-31T03:08:10.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:10 smithi045 conmon[39718]: (mgr.24542) 3467 : cluster [DBG] pgmap v4435: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:10.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:10 smithi045 conmon[33655]: cluster 2021-07-31T03:08:08.956090+0000 mgr.x (mgr.24542) 3467 2021-07-31T03:08:10.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:10 smithi045 conmon[33655]: : cluster [DBG] pgmap v4435: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:10.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:10 smithi154 conmon[34553]: cluster 2021-07-31T03:08:08.956090+0000 mgr.x (mgr.24542) 3467 : cluster [DBG] pgmap v4435: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:12.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:12 smithi045 conmon[33655]: cluster 2021-07-31T03:08:10.956554+0000 mgr.x (mgr.24542) 2021-07-31T03:08:12.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:12 smithi045 conmon[33655]: 3468 : cluster [DBG] pgmap v4436: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:12.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:12 smithi045 conmon[39718]: cluster 2021-07-31T03:08:10.956554+0000 mgr.x 2021-07-31T03:08:12.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:12 smithi045 conmon[39718]: (mgr.24542) 3468 : cluster [DBG] pgmap v4436: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:12.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:12 smithi154 conmon[34553]: cluster 2021-07-31T03:08:10.956554+0000 mgr.x (mgr.24542) 3468 : cluster [DBG] pgmap v4436: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:14.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:13 smithi154 conmon[34553]: cluster 2021-07-31T03:08:12.957050+0000 mgr.x (mgr.24542) 3469 : cluster [DBG] pgmap v4437: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:14.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:13 smithi045 conmon[33655]: cluster 2021-07-31T03:08:12.957050+0000 mgr.x (mgr.24542) 3469 : cluster [DBG] pgmap v4437: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:14.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:13 smithi045 conmon[39718]: cluster 2021-07-31T03:08:12.957050+0000 mgr.x 2021-07-31T03:08:14.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:13 smithi045 conmon[39718]: (mgr.24542) 3469 : cluster [DBG] pgmap v4437: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:16.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:16 smithi045 conmon[33655]: cluster 2021-07-31T03:08:14.957577+0000 mgr.x (mgr.24542) 2021-07-31T03:08:16.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:16 smithi045 conmon[33655]: 3470 : cluster [DBG] pgmap v4438: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:16.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:16 smithi045 conmon[39718]: cluster 2021-07-31T03:08:14.957577+0000 mgr.x (mgr 2021-07-31T03:08:16.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:16 smithi045 conmon[39718]: .24542) 3470 : cluster [DBG] pgmap v4438: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:16.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:16 smithi154 conmon[34553]: cluster 2021-07-31T03:08:14.957577+0000 mgr.x (mgr.24542) 3470 : cluster [DBG] 2021-07-31T03:08:16.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:16 smithi154 conmon[34553]: pgmap v4438: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:18.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:18 smithi045 conmon[33655]: cluster 2021-07-31T03:08:16.958114+0000 mgr.x (mgr.24542) 3471 2021-07-31T03:08:18.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:18 smithi045 conmon[33655]: : cluster [DBG] pgmap v4439: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:18.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:18 smithi045 conmon[39718]: cluster 2021-07-31T03:08:16 2021-07-31T03:08:18.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:18 smithi045 conmon[39718]: .958114+0000 mgr.x (mgr.24542) 3471 : cluster [DBG] pgmap v4439: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:18.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:18 smithi154 conmon[34553]: cluster 2021-07-31T03:08:16.958114+0000 mgr.x (mgr.24542) 3471 : cluster [DBG] 2021-07-31T03:08:18.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:18 smithi154 conmon[34553]: pgmap v4439: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:20.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:20 smithi045 conmon[33655]: cluster 2021-07-31T03:08:18.958602+0000 mgr.x (mgr.24542) 2021-07-31T03:08:20.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:20 smithi045 conmon[33655]: 3472 : cluster [DBG] pgmap v4440: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:20.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:20 smithi045 conmon[39718]: cluster 2021-07-31T03:08:18 2021-07-31T03:08:20.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:20 smithi045 conmon[39718]: .958602+0000 mgr.x (mgr.24542) 3472 : cluster [DBG] pgmap v4440: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:20.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:20 smithi154 conmon[34553]: cluster 2021-07-31T03:08:18.958602+0000 mgr.x (mgr.24542) 3472 : cluster [DBG] pgmap v4440: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:20.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:20 smithi154 conmon[34553]: 2021-07-31T03:08:22.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:22 smithi045 conmon[33655]: cluster 2021-07-31T03:08:20.959153+0000 mgr.x (mgr.24542) 3473 : cluster 2021-07-31T03:08:22.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:22 smithi045 conmon[33655]: [DBG] pgmap v4441: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:22.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:22 smithi045 conmon[39718]: cluster 2021-07-31T03:08:20.959153 2021-07-31T03:08:22.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:22 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 3473 : cluster [DBG] pgmap v4441: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:22.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:22 smithi154 conmon[34553]: cluster 2021-07-31T03:08:20.959153+0000 mgr.x (mgr.24542) 3473 : cluster [DBG] pgmap v4441: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:23.324 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:08:23.601 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:08:23.601 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 83m ago 84m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:08:23.602 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 80m ago 83m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:08:23.602 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 80m ago 87m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:08:23.602 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 83m ago 89m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:08:23.602 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 83m ago 89m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:08:23.603 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 80m ago 87m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:08:23.603 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 83m ago 87m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:08:23.603 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 83m ago 84m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:08:23.603 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 80m ago 84m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:08:23.604 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 83m ago 87m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:08:23.604 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 83m ago 86m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:08:23.604 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 83m ago 86m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:08:23.604 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 83m ago 86m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:08:23.605 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 80m ago 85m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:08:23.605 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 80m ago 85m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:08:23.605 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 80m ago 85m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:08:23.605 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 80m ago 84m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:08:23.605 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 80m ago 84m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:08:23.935 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:08:23.935 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:08:23.936 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:08:23.936 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:08:23.936 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:08:23.936 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:08:23.937 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:08:23.937 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:08:23.937 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:08:23.937 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:08:23.938 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:08:23.938 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:08:23.938 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:08:23.938 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:08:23.939 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:08:23.939 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:08:23.939 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:08:24.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:23 smithi154 conmon[34553]: cluster 2021-07-31T03:08:22.959623+0000 mgr.x (mgr.24542) 3474 : cluster [DBG] pgmap v4442: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:24.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:23 smithi154 conmon[34553]: audit 2021-07-31T03:08:23.312945+0000 mgr.x (mgr.24542) 3475 : audit [DBG] from='client.18876 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:08:24.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:23 smithi154 conmon[34553]: audit 2021-07-31T03:08:23.593783+0000 mgr.x (mgr.24542) 3476 : audit [DBG] from='client.28822 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:08:24.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:23 smithi154 conmon[34553]: audit 2021-07-31T03:08:23.935927+0000 mon.c (mon.1) 178 : 2021-07-31T03:08:24.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:23 smithi154 conmon[34553]: audit [DBG] from='client.? 172.21.15.45:0/1449661190' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:08:24.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:23 smithi045 conmon[39718]: cluster 2021-07-31T03:08:22.959623+0000 mgr.x (mgr.24542 2021-07-31T03:08:24.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:23 smithi045 conmon[39718]: ) 3474 : cluster [DBG] pgmap v4442: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:24.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:23 smithi045 conmon[39718]: audit 2021-07-31T03:08:23.312945+0000 mgr.x (mgr. 2021-07-31T03:08:24.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:23 smithi045 conmon[39718]: 24542) 3475 : audit [DBG] from='client.18876 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:08:24.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:23 smithi045 conmon[39718]: audit 2021-07-31T03:08 2021-07-31T03:08:24.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:23 smithi045 conmon[39718]: :23.593783+0000 mgr.x (mgr.24542) 3476 : audit [DBG] from='client.28822 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:08:24.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:23 smithi045 conmon[39718]: 2021-07-31T03:08:24.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:23 smithi045 conmon[39718]: audit 2021-07-31T03:08:23.935927+0000 2021-07-31T03:08:24.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:23 smithi045 conmon[39718]: mon.c (mon.1) 178 : audit [DBG] from='client.? 172.21.15.45:0/1449661190' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:08:24.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:23 smithi045 conmon[33655]: cluster 2021-07-31T03:08:22.959623+0000 mgr.x (mgr.24542) 3474 2021-07-31T03:08:24.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:23 smithi045 conmon[33655]: : cluster [DBG] pgmap v4442: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:24.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:23 smithi045 conmon[33655]: audit 2021-07-31T03:08:23.312945+0000 mgr.x (mgr.24542) 3475 : audit [DBG] from='client.18876 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:08:24.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:23 smithi045 conmon[33655]: audit 2021-07-31T03:08: 2021-07-31T03:08:24.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:23 smithi045 conmon[33655]: 23.593783+0000 mgr.x (mgr.24542) 3476 : audit [DBG] from='client.28822 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:08:24.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:23 smithi045 conmon[33655]: audit 2021-07-31T03:08:23.935927 2021-07-31T03:08:24.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:23 smithi045 conmon[33655]: +0000 mon.c (mon.1) 178 : audit [DBG] from='client.? 172.21.15.45:0/1449661190' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:08:26.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:26 smithi045 conmon[33655]: cluster 2021-07-31T 2021-07-31T03:08:26.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:26 smithi045 conmon[33655]: 03:08:24.960229+0000 mgr.x (mgr.24542) 3477 : cluster [DBG] pgmap v4443: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:26.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:26 smithi045 conmon[39718]: cluster 2021-07-31T03:08:24.960229+0000 mgr.x 2021-07-31T03:08:26.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:26 smithi045 conmon[39718]: (mgr.24542) 3477 : cluster [DBG] pgmap v4443: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:26.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:26 smithi154 conmon[34553]: cluster 2021-07-31T03:08:24.960229+0000 mgr.x (mgr.24542) 3477 : cluster [DBG] pgmap v4443: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:28.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:28 smithi045 conmon[39718]: cluster 2021-07-31T03:08:26.960606+0000 mgr.x ( 2021-07-31T03:08:28.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:28 smithi045 conmon[39718]: mgr.24542) 3478 : cluster [DBG] pgmap v4444: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:28.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:28 smithi045 conmon[33655]: cluster 2021-07-31T03:08:26.960606+0000 mgr.x (mgr.24542) 3478 : 2021-07-31T03:08:28.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:28 smithi045 conmon[33655]: cluster [DBG] pgmap v4444: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:28.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:28 smithi154 conmon[34553]: cluster 2021-07-31T03:08:26.960606+0000 mgr.x (mgr.24542) 3478 : cluster [DBG] pgmap v4444: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:29.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:29 smithi045 conmon[39718]: audit 2021-07-31T03:08:28 2021-07-31T03:08:29.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:29 smithi045 conmon[39718]: .257126+0000 mon.b (mon.2) 480 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:08:29.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:29 smithi045 conmon[39718]: audit 2021-07-31T03:08:29.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:29 smithi045 conmon[39718]: 2021-07-31 2021-07-31T03:08:29.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:29 smithi045 conmon[39718]: T03:08:28.257398 2021-07-31T03:08:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:29 smithi045 conmon[39718]: +0000 mon.b (mon.2) 481 : audit 2021-07-31T03:08:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:29 smithi045 conmon[39718]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:08:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:29 smithi045 conmon[39718]: audit 2021-07-31T03:08:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:29 smithi045 conmon[39718]: 2021-07-31T03 2021-07-31T03:08:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:29 smithi045 conmon[39718]: :08:28.257805+0000 2021-07-31T03:08:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:29 smithi045 conmon[39718]: mon.a (mon.0) 3257 : audit [INF] 2021-07-31T03:08:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:29 smithi045 conmon[39718]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:08:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:29 smithi045 conmon[39718]: audit 2021- 2021-07-31T03:08:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:29 smithi045 conmon[39718]: 07-31T03:08: 2021-07-31T03:08:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:29 smithi045 conmon[39718]: 28.258125+0000 mon.a (mon.0) 2021-07-31T03:08:29.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:29 smithi045 conmon[39718]: 3258 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:08:29.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:29 smithi045 conmon[33655]: audit 2021-07-31T03:08:28.257126+0000 mon.b (mon.2) 480 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:08:29.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:29 smithi045 conmon[33655]: 2021-07-31T03:08:29.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:29 smithi045 conmon[33655]: audit 2021-07-31T03:08:28.257398+0000 mon.b (mon.2) 481 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:08:29.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:29 smithi045 conmon[33655]: audit 2021-07-31T03:08:29.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:29 smithi045 conmon[33655]: 2021-07-31T03:08:28.257805+0000 mon.a (mon.0) 3257 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:08:29.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:29 smithi045 conmon[33655]: 2021-07-31T03:08:29.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:29 smithi045 conmon[33655]: audit 2021-07- 2021-07-31T03:08:29.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:29 smithi045 conmon[33655]: 31T03:08:28.258125+0000 mon.a (mon.0) 2021-07-31T03:08:29.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:29 smithi045 conmon[33655]: 3258 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:08:29.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:29 smithi154 conmon[34553]: audit 2021-07-31T03:08:28.257126+0000 mon.b (mon.2) 480 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:08:29.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:29 smithi154 conmon[34553]: audit 2021-07-31T03:08:28.257398+0000 2021-07-31T03:08:29.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:29 smithi154 conmon[34553]: mon.b (mon.2) 481 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:08:29.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:29 smithi154 conmon[34553]: audit 2021-07-31T03:08:28.257805+0000 mon.a (mon.0) 3257 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:08:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:29 smithi154 conmon[34553]: audit 2021-07-31T03:08:28.258125+0000 mon.a (mon.0) 3258 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:08:30.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:30 smithi154 conmon[34553]: cluster 2021-07-31T03:08:28.961052+0000 mgr.x (mgr.24542) 3479 : cluster [DBG] pgmap v4445: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:30.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:30 smithi045 conmon[39718]: cluster 2021-07-31T03:08:28.961052+0000 mgr.x ( 2021-07-31T03:08:30.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:30 smithi045 conmon[39718]: mgr.24542) 3479 : cluster [DBG] pgmap v4445: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:30.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:30 smithi045 conmon[33655]: cluster 2021-07-31T03:08:28.961052+0000 2021-07-31T03:08:30.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:30 smithi045 conmon[33655]: mgr.x (mgr.24542) 3479 : cluster [DBG] pgmap v4445: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:31.753 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: debug 2021-07-31T03:08:31.516+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:08:31.753 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:08:31.753 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:08:31.753 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: yield 2021-07-31T03:08:31.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:08:31.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:08:31.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:08:31.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:08:31.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:08:31.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:08:31.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:08:31.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:08:31.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:08:31.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: task.result() 2021-07-31T03:08:31.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:08:31.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: debug 2021-07-31T03:08:31.527+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:08:31.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:08:31.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:08:31.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: yield 2021-07-31T03:08:31.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:08:31.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:08:31.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:08:31.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:08:31.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:08:31.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:08:31.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:08:31.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:08:31.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:08:31.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: task.result() 2021-07-31T03:08:31.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:08:31.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: debug 2021-07-31T03:08:31.553+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:08:31.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:08:31.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:08:31.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: yield 2021-07-31T03:08:31.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:08:31.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:08:31.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:08:31.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:08:31.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:08:31.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:08:31.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:08:31.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:08:31.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:08:31.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: task.result() 2021-07-31T03:08:31.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:08:31.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: debug 2021-07-31T03:08:31.553+0000 7fcb93160700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:08:31.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:08:31.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:08:31.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: yield 2021-07-31T03:08:31.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:08:31.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:08:31.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:08:31.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:08:31.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:08:31.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:08:31.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:08:31.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:08:31.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:08:31.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: task.result() 2021-07-31T03:08:31.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:08:31.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: 2021-07-31T03:08:31.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:08:31.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: 2021-07-31T03:08:31.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:08:31.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:08:31.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:08:31.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:08:31.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:08:31.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:08:31.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:08:31.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:08:31.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:08:31.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:08:31.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:08:31.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:08:31.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:08:31.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:08:31.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:08:31.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:08:31.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:08:31.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:08:31.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: raise 2021-07-31T03:08:31.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:08:31.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:08:31.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:08:31.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:08:31.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:08:31.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: debug 2021-07-31T03:08:31.562+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:08:31.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:08:31.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:08:31.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: yield 2021-07-31T03:08:31.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:08:31.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:08:31.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:08:31.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:08:31.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:08:31.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:08:31.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:08:31.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:08:31.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:08:31.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: task.result() 2021-07-31T03:08:31.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:08:31.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: debug 2021-07-31T03:08:31.562+0000 7fcb9295f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:08:31.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:08:31.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:08:31.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: yield 2021-07-31T03:08:31.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:08:31.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:08:31.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:08:31.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:08:31.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:08:31.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:08:31.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:08:31.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:08:31.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:08:31.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: task.result() 2021-07-31T03:08:31.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:08:31.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: 2021-07-31T03:08:31.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:08:31.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: 2021-07-31T03:08:31.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:08:31.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:08:31.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:08:31.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:08:31.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:08:31.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:08:31.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:08:31.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:08:31.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:08:31.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:08:31.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:08:31.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:08:31.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:08:31.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:08:31.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:08:31.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:08:31.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:08:31.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:08:31.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: raise 2021-07-31T03:08:31.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:08:31.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:08:31.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:08:31.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:08:31.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:08:31 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:08:32.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: cluster 2021-07-31T03:08:30.961403+0000 mgr.x (mgr.24542) 3480 : cluster [DBG] pgmap v4446: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:32.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: audit 2021-07-31T03:08:31.471140+0000 mon.b (mon.2) 482 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:08:32.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: audit 2021-07-31 2021-07-31T03:08:32.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: T03:08:31.472504+0000 mon.b (mon.2) 483 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:08:32.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: audit 2021-07-31T 2021-07-31T03:08:32.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: 03:08:31.473365+0000 mon.b (mon.2 2021-07-31T03:08:32.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: ) 484 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:08:32.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: cephadm 2021-07-31T03:08:31.474098+0000 mgr.x (mgr.24542) 3481 2021-07-31T03:08:32.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:08:32.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: cephadm 2021-07 2021-07-31T03:08:32.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: -31T03:08:31.516976+0000 mgr.x 2021-07-31T03:08:32.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: (mgr.24542) 3482 : cephadm [ERR] 2021-07-31T03:08:32.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:08:32.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:08:32.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: yield 2021-07-31T03:08:32.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:08:32.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:08:32.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:08:32.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:08:32.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:08:32.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:08:32.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:08:32.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:08:32.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:08:32.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: task.result() 2021-07-31T03:08:32.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:08:32.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: cephadm 2021-07-31T03:08:31.517393+0000 mgr.x (mgr.24542) 3483 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:08:32.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: cephadm 2021-07-31T03:08:31.528541+0000 mgr.x (mgr. 2021-07-31T03:08:32.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: 24542) 3484 : cephadm [ERR] 2021-07-31T03:08:32.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:08:32.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:08:32.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: yield 2021-07-31T03:08:32.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:08:32.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:08:32.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:08:32.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:08:32.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:08:32.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:08:32.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:08:32.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:08:32.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:08:32.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: task.result() 2021-07-31T03:08:32.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:08:32.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: cephadm 2021-07-31T03:08:31.529050+0000 mgr.x (mgr.24542) 3485 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:08:32.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: cephadm 2021-07-31T03:08:31.554006+0000 mgr.x (mgr.24542) 3486 : cephadm [ERR] 2021-07-31T03:08:32.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: 2021-07-31T03:08:32.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:08:32.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:08:32.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: yield 2021-07-31T03:08:32.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:08:32.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:08:32.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:08:32.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:08:32.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:08:32.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:08:32.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:08:32.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:08:32.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:08:32.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: task.result() 2021-07-31T03:08:32.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:08:32.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: cephadm 2021-07-31T03:08:31.554628+0000 mgr.x (mgr.24542) 3487 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:08:32.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:08:32.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:08:32.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: yield 2021-07-31T03:08:32.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:08:32.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:08:32.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:08:32.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:08:32.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:08:32.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:08:32.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:08:32.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:08:32.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:08:32.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: task.result() 2021-07-31T03:08:32.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:08:32.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: 2021-07-31T03:08:32.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:08:32.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: 2021-07-31T03:08:32.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:08:32.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:08:32.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:08:32.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:08:32.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:08:32.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:08:32.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:08:32.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:08:32.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:08:32.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:08:32.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:08:32.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:08:32.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:08:32.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:08:32.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:08:32.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:08:32.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:08:32.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:08:32.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: raise 2021-07-31T03:08:32.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:08:32.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:08:32.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:08:32.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:08:32.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:08:32.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: cephadm 2021-07-31T03:08:31.563017+0000 mgr.x 2021-07-31T03:08:32.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: (mgr.24542) 3488 : cephadm [ERR] 2021-07-31T03:08:32.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:08:32.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:08:32.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: yield 2021-07-31T03:08:32.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:08:32.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:08:32.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:08:32.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:08:32.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:08:32.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:08:32.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:08:32.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:08:32.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:08:32.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: task.result() 2021-07-31T03:08:32.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:08:32.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: cephadm 2021-07-31T03:08:31.563393+0000 mgr.x (mgr.24542) 3489 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:08:32.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:08:32.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:08:32.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: yield 2021-07-31T03:08:32.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:08:32.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:08:32.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:08:32.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:08:32.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:08:32.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:08:32.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:08:32.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:08:32.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:08:32.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: task.result() 2021-07-31T03:08:32.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:08:32.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: 2021-07-31T03:08:32.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:08:32.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: 2021-07-31T03:08:32.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:08:32.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:08:32.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:08:32.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:08:32.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:08:32.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:08:32.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:08:32.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:08:32.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:08:32.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:08:32.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:08:32.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:08:32.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:08:32.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:08:32.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:08:32.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:08:32.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:08:32.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:08:32.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: raise 2021-07-31T03:08:32.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:08:32.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:08:32.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:08:32.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:08:32.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:32 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:08:32.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: cluster 2021-07-31T03:08: 2021-07-31T03:08:32.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: 30.961403+0000 mgr.x (mgr.24542) 3480 : cluster [DBG] pgmap v4446: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:32.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: audit 2021-07-31T03:08:32.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: 2021-07-31 2021-07-31T03:08:32.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: T03:08:31.471140 2021-07-31T03:08:32.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: +0000 mon.b (mon.2) 2021-07-31T03:08:32.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: 482 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:08:32.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: 2021-07-31T03:08:32.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T03:08:32.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: T03:08:31.472504+0000 2021-07-31T03:08:32.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: mon.b (mon.2) 483 2021-07-31T03:08:32.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:08:32.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: audit 2021 2021-07-31T03:08:32.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: -07-31T03:08:31 2021-07-31T03:08:32.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: .473365+0000 mon.b (mon. 2021-07-31T03:08:32.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: 2) 484 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:08:32.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: 2021-07-31T03:08:32.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: cephadm 2021-07 2021-07-31T03:08:32.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: -31T03:08:31.474098+0000 mgr.x 2021-07-31T03:08:32.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: (mgr.24542) 3481 : cephadm [WRN] 2021-07-31T03:08:32.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:08:32.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: cephadm 2021-07-31T03 2021-07-31T03:08:32.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: :08:31.516976+0000 mgr.x (mgr 2021-07-31T03:08:32.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: .24542) 3482 : cephadm [ERR] 2021-07-31T03:08:32.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:08:32.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:08:32.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: yield 2021-07-31T03:08:32.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:08:32.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:08:32.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:08:32.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:08:32.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:08:32.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:08:32.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:08:32.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:08:32.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:08:32.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: task.result() 2021-07-31T03:08:32.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:08:32.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: cephadm 2021-07-31T03:08:31.517393+0000 mgr.x (mgr.24542) 3483 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:08:32.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: cephadm 2021-07-31T03:08: 2021-07-31T03:08:32.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: 31.528541+0000 mgr.x (mgr.24542) 3484 : cephadm [ERR] 2021-07-31T03:08:32.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:08:32.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:08:32.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: yield 2021-07-31T03:08:32.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:08:32.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:08:32.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:08:32.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:08:32.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:08:32.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:08:32.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:08:32.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:08:32.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:08:32.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: task.result() 2021-07-31T03:08:32.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:08:32.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: cephadm 2021-07-31T03:08:31.529050+0000 mgr.x (mgr.24542) 3485 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:08:32.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: cephadm 2021-07-31 2021-07-31T03:08:32.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: T03:08:31.554006+0000 mgr.x (mgr.24542 2021-07-31T03:08:32.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: ) 3486 : cephadm [ERR] 2021-07-31T03:08:32.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:08:32.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:08:32.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: yield 2021-07-31T03:08:32.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:08:32.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:08:32.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:08:32.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:08:32.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:08:32.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:08:32.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:08:32.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:08:32.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:08:32.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: task.result() 2021-07-31T03:08:32.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:08:32.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: cephadm 2021-07-31T03:08:31.554628+0000 mgr.x (mgr.24542) 3487 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:08:32.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:08:32.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:08:32.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: yield 2021-07-31T03:08:32.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:08:32.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:08:32.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:08:32.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:08:32.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:08:32.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:08:32.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:08:32.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:08:32.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:08:32.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: task.result() 2021-07-31T03:08:32.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:08:32.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: 2021-07-31T03:08:32.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:08:32.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: 2021-07-31T03:08:32.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:08:32.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:08:32.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:08:32.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:08:32.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:08:32.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:08:32.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:08:32.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:08:32.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:08:32.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:08:32.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:08:32.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:08:32.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:08:32.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:08:32.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:08:32.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:08:32.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:08:32.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:08:32.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: raise 2021-07-31T03:08:32.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:08:32.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:08:32.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:08:32.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:08:32.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:08:32.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: cephadm 2021-07-31T03:08:32.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: 2021-07-31T03:08:31.563017+0000 mgr.x (mgr.24542) 3488 : cephadm [ERR] 2021-07-31T03:08:32.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:08:32.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:08:32.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: yield 2021-07-31T03:08:32.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:08:32.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:08:32.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:08:32.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:08:32.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:08:32.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: cluster 2021-07-31T03:08:30.961403+0000 mgr.x (mgr.24542) 3480 : cluster 2021-07-31T03:08:32.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: [DBG] pgmap v4446: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:32.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: audit 2021-07-31T03:08:31.471140+0000 mon.b (mon.2) 482 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:08:32.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: audit 2021-07-31T03:08:31.472504 2021-07-31T03:08:32.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: +0000 mon.b (mon.2) 483 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:08:32.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: audit 2021-07- 2021-07-31T03:08:32.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: 31T03:08:31.473365+0000 mon.b (mon.2) 484 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:08:32.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: cephadm 2021 2021-07-31T03:08:32.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: -07-31T03:08:31.474098+0000 mgr.x (mgr 2021-07-31T03:08:32.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: .24542) 3481 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:08:32.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: cephadm 2021 2021-07-31T03:08:32.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: -07-31T03:08:31.516976+0000 mgr.x ( 2021-07-31T03:08:32.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: mgr.24542) 3482 : cephadm [ERR] 2021-07-31T03:08:32.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:08:32.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:08:32.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: yield 2021-07-31T03:08:32.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:08:32.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:08:32.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:08:32.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:08:32.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:08:32.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:08:32.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:08:32.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:08:32.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:08:32.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: task.result() 2021-07-31T03:08:32.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:08:32.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: cephadm 2021-07-31T03:08:31.517393+0000 mgr.x (mgr.24542) 3483 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:08:32.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: cephadm 2021-07-31T03 2021-07-31T03:08:32.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: :08:31.528541+0000 mgr.x (mgr.24542 2021-07-31T03:08:32.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: ) 3484 : cephadm [ERR] 2021-07-31T03:08:32.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:08:32.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:08:32.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: yield 2021-07-31T03:08:32.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:08:32.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:08:32.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:08:32.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:08:32.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:08:32.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:08:32.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:08:32.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: task.result() 2021-07-31T03:08:32.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:08:32.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: cephadm 2021-07-31T03:08:31.563393+0000 mgr.x (mgr.24542) 3489 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:08:32.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:08:32.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:08:32.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: yield 2021-07-31T03:08:32.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:08:32.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:08:32.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:08:32.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:08:32.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:08:32.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:08:32.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:08:32.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:08:32.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:08:32.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: task.result() 2021-07-31T03:08:32.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:08:32.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: 2021-07-31T03:08:32.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:08:32.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: 2021-07-31T03:08:32.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:08:32.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:08:32.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:08:32.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:08:32.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:08:32.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:08:32.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:08:32.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:08:32.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:08:32.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:08:32.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:08:32.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:08:32.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:08:32.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:08:32.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:08:32.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:08:32.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:08:32.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:08:32.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: raise 2021-07-31T03:08:32.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:08:32.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:08:32.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:08:32.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:08:32.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:08:32.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:08:32.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:08:32.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:08:32.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:08:32.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:08:32.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:08:32.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: task.result() 2021-07-31T03:08:32.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:08:32.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: cephadm 2021-07-31T03:08:31.529050+0000 mgr.x (mgr.24542) 3485 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:08:32.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: cephadm 2021-07-31T 2021-07-31T03:08:32.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: 03:08:31.554006+0000 mgr.x (mgr.24542) 3486 : cephadm [ERR] 2021-07-31T03:08:32.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:08:32.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:08:32.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: yield 2021-07-31T03:08:32.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:08:32.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:08:32.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:08:32.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:08:32.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:08:32.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:08:32.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:08:32.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:08:32.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:08:32.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: task.result() 2021-07-31T03:08:32.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:08:32.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: cephadm 2021-07-31T03:08:31.554628+0000 mgr.x (mgr.24542) 3487 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:08:32.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:08:32.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:08:32.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: yield 2021-07-31T03:08:32.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:08:32.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:08:32.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:08:32.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:08:32.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:08:32.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:08:32.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:08:32.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:08:32.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:08:32.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: task.result() 2021-07-31T03:08:32.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:08:32.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: 2021-07-31T03:08:32.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:08:32.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: 2021-07-31T03:08:32.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:08:32.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:08:32.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:08:32.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:08:32.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:08:32.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:08:32.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:08:32.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:08:32.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:08:32.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:08:32.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:08:32.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:08:32.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:08:32.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:08:32.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:08:32.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:08:32.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:08:32.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:08:32.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: raise 2021-07-31T03:08:32.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:08:32.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:08:32.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:08:32.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:08:32.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:08:32.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: cephadm 2021-07-31T03:08:31.563017+0000 mgr.x (mgr.24542 2021-07-31T03:08:32.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: ) 3488 : cephadm [ERR] 2021-07-31T03:08:32.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:08:32.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:08:32.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: yield 2021-07-31T03:08:32.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:08:32.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:08:32.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:08:32.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:08:32.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:08:32.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:08:32.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:08:32.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:08:32.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:08:32.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: task.result() 2021-07-31T03:08:32.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:08:32.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: cephadm 2021-07-31T03:08:31.563393+0000 mgr.x (mgr.24542) 3489 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:08:32.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:08:32.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:08:32.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: yield 2021-07-31T03:08:32.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:08:32.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:08:32.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:08:32.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:08:32.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:08:32.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:08:32.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:08:32.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:08:32.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:08:32.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: task.result() 2021-07-31T03:08:32.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:08:32.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: 2021-07-31T03:08:32.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:08:32.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: 2021-07-31T03:08:32.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:08:32.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:08:32.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:08:32.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:08:32.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:08:32.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:08:32.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:08:32.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:08:32.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:08:32.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:08:32.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:08:32.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:08:32.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:08:32.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:08:32.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:08:32.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:08:32.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:08:32.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:08:32.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: raise 2021-07-31T03:08:32.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:08:32.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:08:32.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:08:32.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:08:32.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:32 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:08:34.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:33 smithi154 conmon[34553]: cluster 2021-07-31T03:08:32.961868+0000 mgr.x (mgr.24542) 3490 : cluster [DBG] pgmap v4447: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:34.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:33 smithi045 conmon[33655]: cluster 2021-07-31T03:08:32.961868+0000 mgr.x (mgr.24542) 3490 : 2021-07-31T03:08:34.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:33 smithi045 conmon[33655]: cluster [DBG] pgmap v4447: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:34.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:33 smithi045 conmon[39718]: cluster 2021-07-31T03:08:32.961868+0000 mgr.x 2021-07-31T03:08:34.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:33 smithi045 conmon[39718]: (mgr.24542) 3490 : cluster [DBG] pgmap v4447: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:36.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:36 smithi045 conmon[33655]: cluster 2021-07-31T03:08:34 2021-07-31T03:08:36.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:36 smithi045 conmon[33655]: .962341+0000 mgr.x (mgr.24542) 3491 : cluster [DBG] pgmap v4448: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:36.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:36 smithi045 conmon[39718]: cluster 2021-07-31T03:08:34.962341+0000 2021-07-31T03:08:36.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:36 smithi045 conmon[39718]: mgr.x (mgr.24542) 3491 : cluster [DBG] pgmap v4448: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:36.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:36 smithi154 conmon[34553]: cluster 2021-07-31T03:08:34.962341+0000 mgr.x (mgr.24542) 3491 : cluster [DBG] pgmap v4448: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:38.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:38 smithi045 conmon[39718]: cluster 2021-07-31T03:08:36.962882+0000 mgr.x 2021-07-31T03:08:38.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:38 smithi045 conmon[39718]: (mgr.24542) 3492 : cluster [DBG] pgmap v4449: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:38.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:38 smithi045 conmon[33655]: cluster 2021-07-31T03:08:36.962882+0000 mgr.x (mgr.24542) 2021-07-31T03:08:38.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:38 smithi045 conmon[33655]: 3492 : cluster [DBG] pgmap v4449: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:38.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:38 smithi154 conmon[34553]: cluster 2021-07-31T03:08:36.962882+0000 mgr.x (mgr.24542) 3492 : cluster [DBG] pgmap v4449: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:40.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:40 smithi045 conmon[39718]: cluster 2021-07-31T03:08:38.963324+0000 mgr.x (mgr.24542) 3493 : 2021-07-31T03:08:40.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:40 smithi045 conmon[39718]: cluster [DBG] pgmap v4450: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:40.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:40 smithi045 conmon[33655]: cluster 2021-07-31T03:08:38.963324+0000 2021-07-31T03:08:40.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:40 smithi045 conmon[33655]: mgr.x (mgr.24542) 3493 : cluster [DBG] pgmap v4450: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:40.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:40 smithi154 conmon[34553]: cluster 2021-07-31T03:08:38.963324+0000 mgr.x (mgr.24542) 3493 : cluster [DBG] pgmap v4450: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:40.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:40 smithi154 conmon[34553]: 2021-07-31T03:08:42.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:42 smithi154 conmon[34553]: cluster 2021-07-31T03:08:40.963851+0000 mgr.x (mgr.24542) 3494 : cluster [DBG] pgmap v4451: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:42.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:42 smithi154 conmon[34553]: 2021-07-31T03:08:42.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:42 smithi045 conmon[39718]: cluster 2021-07-31T03:08:40.963851+0000 mgr.x ( 2021-07-31T03:08:42.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:42 smithi045 conmon[39718]: mgr.24542) 3494 : cluster [DBG] pgmap v4451: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:42.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:42 smithi045 conmon[33655]: cluster 2021-07-31T03:08:40.963851 2021-07-31T03:08:42.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:42 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 3494 : cluster [DBG] pgmap v4451: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:44.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:43 smithi154 conmon[34553]: cluster 2021-07-31T03:08:42.964269+0000 mgr.x (mgr.24542) 3495 : cluster [DBG] 2021-07-31T03:08:44.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:43 smithi154 conmon[34553]: pgmap v4452: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:44.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:43 smithi045 conmon[39718]: cluster 2021-07-31T03:08:42.964269+0000 mgr.x ( 2021-07-31T03:08:44.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:43 smithi045 conmon[39718]: mgr.24542) 3495 : cluster [DBG] pgmap v4452: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:44.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:43 smithi045 conmon[33655]: cluster 2021-07-31T03:08 2021-07-31T03:08:44.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:43 smithi045 conmon[33655]: :42.964269+0000 mgr.x (mgr.24542) 3495 : cluster [DBG] pgmap v4452: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:46.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:46 smithi045 conmon[33655]: cluster 2021-07-31T03:08:44.964686+0000 mgr.x (mgr.24542) 3496 : cluster [DBG] pgmap v4453: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:46.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:46 smithi045 conmon[39718]: cluster 2021-07-31T03:08:44.964686+0000 mgr.x (mgr. 2021-07-31T03:08:46.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:46 smithi045 conmon[39718]: 24542) 3496 : cluster [DBG] pgmap v4453: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:46.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:46 smithi154 conmon[34553]: cluster 2021-07-31T03:08:44.964686+0000 mgr.x (mgr.24542) 3496 : cluster [DBG] pgmap v4453: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:46.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:46 smithi154 conmon[34553]: 2021-07-31T03:08:48.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:48 smithi045 conmon[39718]: cluster 2021-07-31T03:08:46.965092+0000 mgr.x (mgr 2021-07-31T03:08:48.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:48 smithi045 conmon[39718]: .24542) 3497 : cluster [DBG] pgmap v4454: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:48.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:48 smithi045 conmon[33655]: cluster 2021-07-31T03:08:46.965092+0000 mgr.x (mgr.24542) 3497 : cluster 2021-07-31T03:08:48.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:48 smithi045 conmon[33655]: [DBG] pgmap v4454: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:48.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:48 smithi154 conmon[34553]: cluster 2021-07-31T03:08:46.965092+0000 mgr.x (mgr.24542) 2021-07-31T03:08:48.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:48 smithi154 conmon[34553]: 3497 : cluster [DBG] pgmap v4454: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:50.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:50 smithi045 conmon[39718]: cluster 2021-07-31T03:08:48.965650+0000 mgr.x (mgr 2021-07-31T03:08:50.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:50 smithi045 conmon[39718]: .24542) 3498 : cluster [DBG] pgmap v4455: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:50.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:50 smithi045 conmon[33655]: cluster 2021-07-31T03:08:48.965650+0000 mgr.x (mgr.24542) 3498 : 2021-07-31T03:08:50.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:50 smithi045 conmon[33655]: cluster [DBG] pgmap v4455: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:50.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:50 smithi154 conmon[34553]: cluster 2021-07-31T03:08:48.965650+0000 mgr.x (mgr.24542) 3498 : cluster [DBG] 2021-07-31T03:08:50.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:50 smithi154 conmon[34553]: pgmap v4455: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:52.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:52 smithi154 conmon[34553]: cluster 2021-07-31T03:08:50.966073+0000 mgr.x (mgr.24542) 3499 : cluster [DBG] pgmap v4456: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:52.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:52 smithi045 conmon[39718]: cluster 2021-07-31T03:08:50.966073+0000 mgr.x (mgr.24542) 3499 : cluster [DBG] pgmap v4456: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:52.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:52 smithi045 conmon[33655]: cluster 2021-07-31T03:08:50.966073+0000 mgr.x (mgr.24542) 2021-07-31T03:08:52.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:52 smithi045 conmon[33655]: 3499 : cluster [DBG] pgmap v4456: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:54.248 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:08:54.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:53 smithi154 conmon[34553]: cluster 2021-07-31T03:08:52.966585+0000 mgr.x (mgr.24542) 3500 : cluster [DBG] pgmap v4457: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:54.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:53 smithi045 conmon[33655]: cluster 2021-07-31T03:08:52.966585 2021-07-31T03:08:54.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:53 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 3500 : cluster [DBG] pgmap v4457: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:54.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:53 smithi045 conmon[39718]: cluster 2021-07-31T03:08:52.966585+0000 mgr.x (mgr.24542) 2021-07-31T03:08:54.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:53 smithi045 conmon[39718]: 3500 : cluster [DBG] pgmap v4457: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:54.526 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:08:54.527 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 83m ago 84m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:08:54.527 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 81m ago 84m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:08:54.527 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 81m ago 87m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:08:54.527 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 83m ago 89m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:08:54.527 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 83m ago 90m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:08:54.528 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 81m ago 88m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:08:54.528 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 83m ago 88m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:08:54.528 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 83m ago 84m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:08:54.528 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 81m ago 84m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:08:54.528 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 83m ago 87m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:08:54.529 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 83m ago 87m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:08:54.529 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 83m ago 86m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:08:54.529 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 83m ago 86m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:08:54.529 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 81m ago 86m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:08:54.530 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 81m ago 85m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:08:54.530 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 81m ago 85m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:08:54.530 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 81m ago 85m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:08:54.530 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 81m ago 84m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:08:54.860 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:08:54.861 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:08:54.861 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:08:54.861 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:08:54.861 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:08:54.861 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:08:54.862 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:08:54.862 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:08:54.862 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:08:54.862 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:08:54.863 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:08:54.863 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:08:54.863 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:08:54.863 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:08:54.864 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:08:54.864 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:08:54.864 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:08:55.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:54 smithi154 conmon[34553]: audit 2021-07-31T03:08:54.236335+0000 mgr.x ( 2021-07-31T03:08:55.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:54 smithi154 conmon[34553]: mgr.24542) 3501 : audit [DBG] from='client.28834 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:08:55.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:54 smithi154 conmon[34553]: audit 2021-07-31T03:08:54. 2021-07-31T03:08:55.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:54 smithi154 conmon[34553]: 518778+0000 mgr.x (mgr.24542) 3502 : audit [DBG] from='client.18900 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:08:55.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:54 smithi154 conmon[34553]: audit 2021- 2021-07-31T03:08:55.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:54 smithi154 conmon[34553]: 07-31T03:08:54.861026+0000 2021-07-31T03:08:55.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:54 smithi154 conmon[34553]: mon.a (mon.0) 3259 : audit 2021-07-31T03:08:55.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:54 smithi154 conmon[34553]: [DBG] from='client.? 172.21.15.45:0/2035982595' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:08:55.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:54 smithi045 conmon[33655]: audit 2021-07-31T03:08:54.236335+0000 mgr.x (mgr.24542) 3501 : audit [DBG] from='client.28834 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:08:55.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:54 smithi045 conmon[33655]: audit 2021-07-31T03:08:54.518778+0000 mgr.x (mgr.24542) 3502 : audit [DBG] from='client.18900 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:08:55.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:54 smithi045 conmon[33655]: audit 2021-07-31T03:08:55.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:54 smithi045 conmon[33655]: 2021-07-31T03:08:54.861026+0000 mon.a (mon.0) 2021-07-31T03:08:55.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:54 smithi045 conmon[33655]: 3259 : audit [DBG] from='client.? 172.21.15.45:0/2035982595' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:08:55.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:54 smithi045 conmon[39718]: audit 2021-07-31T03:08:54.236335+0000 mgr.x (mgr.24542 2021-07-31T03:08:55.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:54 smithi045 conmon[39718]: ) 3501 : audit [DBG] from='client.28834 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:08:55.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:54 smithi045 conmon[39718]: audit 2021-07-31T03:08:54.518778+0000 mgr.x (mgr.24542 2021-07-31T03:08:55.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:54 smithi045 conmon[39718]: ) 3502 : audit [DBG] from='client.18900 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:08:55.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:54 smithi045 conmon[39718]: audit 2021 2021-07-31T03:08:55.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:54 smithi045 conmon[39718]: -07-31T03:08:54.861026+0000 mon.a (mon.0) 2021-07-31T03:08:55.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:54 smithi045 conmon[39718]: 3259 : audit [DBG] from='client.? 172.21.15.45:0/2035982595' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:08:56.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:55 smithi154 conmon[34553]: cluster 2021-07-31T03:08:54.967118+0000 mgr.x (mgr.24542) 3503 : cluster [DBG] 2021-07-31T03:08:56.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:55 smithi154 conmon[34553]: pgmap v4458: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:56.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:55 smithi045 conmon[33655]: cluster 2021-07-31T03:08:54.967118+0000 mgr.x (mgr.24542) 3503 : 2021-07-31T03:08:56.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:55 smithi045 conmon[33655]: cluster [DBG] pgmap v4458: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:56.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:55 smithi045 conmon[39718]: cluster 2021-07-31T03:08:54.967118+0000 mgr.x ( 2021-07-31T03:08:56.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:55 smithi045 conmon[39718]: mgr.24542) 3503 : cluster [DBG] pgmap v4458: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:58.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:57 smithi154 conmon[34553]: cluster 2021-07-31T03:08:56.967597+0000 mgr.x (mgr.24542) 3504 : cluster [DBG] pgmap v4459: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:58.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:57 smithi045 conmon[39718]: cluster 2021-07-31T03:08:56.967597+0000 mgr.x (mgr. 2021-07-31T03:08:58.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:57 smithi045 conmon[39718]: 24542) 3504 : cluster [DBG] pgmap v4459: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:08:58.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:57 smithi045 conmon[33655]: cluster 2021-07-31T03:08:56.967597+0000 mgr.x (mgr 2021-07-31T03:08:58.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:57 smithi045 conmon[33655]: .24542) 3504 : cluster [DBG] pgmap v4459: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:00.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:08:59 smithi154 conmon[34553]: cluster 2021-07-31T03:08:58.968224+0000 mgr.x (mgr.24542) 3505 : cluster [DBG] pgmap v4460: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:00.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:59 smithi045 conmon[39718]: cluster 2021-07-31T03:08:58.968224 2021-07-31T03:09:00.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:08:59 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 3505 : cluster [DBG] pgmap v4460: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:00.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:59 smithi045 conmon[33655]: cluster 2021-07-31T03:08:58.968224+0000 mgr.x (mgr.24542) 3505 : cluster [DBG] 2021-07-31T03:09:00.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:08:59 smithi045 conmon[33655]: pgmap v4460: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:02.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:01 smithi154 conmon[34553]: cluster 2021-07-31T03:09:00.968772+0000 mgr.x (mgr.24542) 3506 : cluster [DBG] pgmap v4461: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:02.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:01 smithi045 conmon[33655]: cluster 2021-07-31T03:09:00.968772+0000 mgr.x (mgr 2021-07-31T03:09:02.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:01 smithi045 conmon[33655]: .24542) 3506 : cluster [DBG] pgmap v4461: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:02.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:01 smithi045 conmon[39718]: cluster 2021-07-31T03:09:00.968772+0000 mgr.x 2021-07-31T03:09:02.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:01 smithi045 conmon[39718]: (mgr.24542) 3506 : cluster [DBG] pgmap v4461: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:04.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:03 smithi154 conmon[34553]: cluster 2021-07-31T03:09:02.969182+0000 mgr.x (mgr.24542) 3507 : cluster [DBG] pgmap v4462: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:04.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:03 smithi045 conmon[39718]: cluster 2021-07-31T03:09:02.969182+0000 mgr.x (mgr. 2021-07-31T03:09:04.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:03 smithi045 conmon[39718]: 24542) 3507 : cluster [DBG] pgmap v4462: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:04.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:03 smithi045 conmon[33655]: cluster 2021-07-31T03:09:02.969182+0000 mgr.x (mgr.24542 2021-07-31T03:09:04.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:03 smithi045 conmon[33655]: ) 3507 : cluster [DBG] pgmap v4462: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:06.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:05 smithi154 conmon[34553]: cluster 2021-07-31T03:09:04.969647+0000 mgr.x (mgr.24542) 3508 : cluster [DBG] pgmap v4463: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:06.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:05 smithi045 conmon[33655]: cluster 2021-07-31T03:09:04.969647+0000 mgr.x 2021-07-31T03:09:06.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:05 smithi045 conmon[33655]: (mgr.24542) 3508 : cluster [DBG] pgmap v4463: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:06.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:05 smithi045 conmon[39718]: cluster 2021-07-31T03:09:04 2021-07-31T03:09:06.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:05 smithi045 conmon[39718]: .969647+0000 mgr.x (mgr.24542) 3508 : cluster [DBG] pgmap v4463: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:08.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:08 smithi045 conmon[39718]: cluster 2021-07-31T03:09:06. 2021-07-31T03:09:08.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:08 smithi045 conmon[39718]: 970049+0000 mgr.x (mgr.24542) 3509 : cluster [DBG] pgmap v4464: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:08.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:08 smithi045 conmon[33655]: cluster 2021-07-31T03:09:06.970049+0000 mgr.x (mgr.24542) 3509 : cluster [DBG] pgmap v4464: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:08.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:08 smithi154 conmon[34553]: cluster 2021-07-31T03:09:06.970049+0000 mgr.x (mgr.24542) 3509 : cluster [DBG] pgmap v4464: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:10.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:10 smithi045 conmon[33655]: cluster 2021-07-31T03:09:08.970556+0000 mgr.x (mgr.24542) 3510 : 2021-07-31T03:09:10.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:10 smithi045 conmon[33655]: cluster [DBG] pgmap v4465: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:10.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:10 smithi045 conmon[39718]: cluster 2021-07-31T03:09:08.970556+0000 mgr.x (mgr.24542) 3510 : cluster [DBG] pgmap v4465: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:10.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:10 smithi045 conmon[39718]: 2021-07-31T03:09:10.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:10 smithi154 conmon[34553]: cluster 2021-07-31T03:09:08.970556+0000 mgr.x (mgr.24542) 3510 : cluster [DBG] pgmap v4465: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:12.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:12 smithi045 conmon[39718]: cluster 2021-07-31T03:09:10.970959+0000 mgr.x (mgr 2021-07-31T03:09:12.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:12 smithi045 conmon[39718]: .24542) 3511 : cluster [DBG] pgmap v4466: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:12.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:12 smithi045 conmon[33655]: cluster 2021-07-31T03:09:10.970959+0000 mgr.x (mgr.24542) 3511 2021-07-31T03:09:12.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:12 smithi045 conmon[33655]: : cluster [DBG] pgmap v4466: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:12.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:12 smithi154 conmon[34553]: cluster 2021-07-31T03:09:10.970959+0000 mgr.x (mgr.24542) 3511 : cluster [DBG] pgmap v4466: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:14.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:14 smithi045 conmon[39718]: cluster 2021-07-31T03:09:12.971348+0000 mgr.x (mgr 2021-07-31T03:09:14.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:14 smithi045 conmon[39718]: .24542) 3512 : cluster [DBG] pgmap v4467: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:14.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:14 smithi045 conmon[33655]: cluster 2021-07-31T03:09:12.971348+0000 mgr.x (mgr.24542) 3512 : 2021-07-31T03:09:14.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:14 smithi045 conmon[33655]: cluster [DBG] pgmap v4467: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:14.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:14 smithi154 conmon[34553]: cluster 2021-07-31T03:09:12.971348+0000 mgr.x (mgr.24542) 3512 : cluster [DBG] pgmap v4467: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:14.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:14 smithi154 conmon[34553]: 2021-07-31T03:09:16.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:16 smithi045 conmon[39718]: cluster 2021-07-31T03:09:14.971748+0000 mgr.x ( 2021-07-31T03:09:16.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:16 smithi045 conmon[39718]: mgr.24542) 3513 : cluster [DBG] pgmap v4468: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:16.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:16 smithi045 conmon[33655]: cluster 2021-07-31T03:09:14.971748+0000 mgr.x (mgr.24542) 3513 : cluster 2021-07-31T03:09:16.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:16 smithi045 conmon[33655]: [DBG] pgmap v4468: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:16.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:16 smithi154 conmon[34553]: cluster 2021-07-31T03:09:14.971748+0000 mgr.x (mgr.24542) 3513 : cluster [DBG] pgmap v4468: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:18.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:18 smithi045 conmon[33655]: cluster 2021-07-31T03:09:16.972230+0000 mgr.x (mgr. 2021-07-31T03:09:18.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:18 smithi045 conmon[33655]: 24542) 3514 : cluster [DBG] pgmap v4469: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:18.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:18 smithi045 conmon[39718]: cluster 2021-07-31T03:09:16.972230+0000 mgr.x (mgr.24542) 3514 : cluster [DBG] pgmap v4469: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:18.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:18 smithi045 conmon[39718]: 2021-07-31T03:09:18.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:18 smithi154 conmon[34553]: cluster 2021-07-31T03:09:16.972230+0000 mgr.x (mgr.24542) 3514 : cluster [DBG] pgmap v4469: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:20.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:20 smithi045 conmon[33655]: cluster 2021-07-31T03:09:18.972676+0000 mgr.x (mgr.24542) 3515 : cluster [DBG] 2021-07-31T03:09:20.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:20 smithi045 conmon[33655]: pgmap v4470: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:20.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:20 smithi045 conmon[39718]: cluster 2021-07-31T03:09:18.972676+0000 mgr.x (mgr.24542) 3515 2021-07-31T03:09:20.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:20 smithi045 conmon[39718]: : cluster [DBG] pgmap v4470: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:20.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:20 smithi154 conmon[34553]: cluster 2021-07-31T03:09:18.972676+0000 mgr.x (mgr.24542) 3515 : cluster [DBG] pgmap v4470: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:22.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:22 smithi045 conmon[33655]: cluster 2021-07-31T03:09:20.973091+0000 mgr.x (mgr.24542) 3516 : cluster [DBG] pgmap v4471: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:22.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:22 smithi045 conmon[39718]: cluster 2021-07-31T03:09:20.973091+0000 mgr.x (mgr.24542) 3516 : cluster [DBG] pgmap v4471: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:22.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:22 smithi154 conmon[34553]: cluster 2021-07-31T03:09:20.973091+0000 mgr.x (mgr.24542) 3516 : cluster [DBG] pgmap v4471: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:24.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:24 smithi154 conmon[34553]: cluster 2021-07-31T03:09:22.973587+0000 mgr.x (mgr.24542) 3517 : cluster [DBG] pgmap v4472: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:24.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:24 smithi045 conmon[39718]: cluster 2021-07-31T03:09:22 2021-07-31T03:09:24.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:24 smithi045 conmon[39718]: .973587+0000 mgr.x (mgr.24542) 3517 : cluster [DBG] pgmap v4472: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:24.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:24 smithi045 conmon[33655]: cluster 2021-07-31T03:09:22.973587+0000 mgr.x (mgr.24542) 3517 : cluster 2021-07-31T03:09:24.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:24 smithi045 conmon[33655]: [DBG] pgmap v4472: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:25.177 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:09:25.456 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:09:25.457 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 84m ago 85m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:09:25.457 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 81m ago 84m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:09:25.457 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 81m ago 88m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:09:25.457 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 84m ago 90m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:09:25.458 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 84m ago 90m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:09:25.458 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 81m ago 88m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:09:25.458 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 84m ago 88m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:09:25.458 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 84m ago 85m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:09:25.458 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 81m ago 85m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:09:25.459 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 84m ago 88m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:09:25.459 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 84m ago 87m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:09:25.459 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 84m ago 87m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:09:25.459 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 84m ago 87m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:09:25.459 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 81m ago 86m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:09:25.460 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 81m ago 86m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:09:25.460 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 81m ago 86m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:09:25.460 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 81m ago 85m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:09:25.460 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 81m ago 85m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:09:25.793 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:09:25.793 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:09:25.793 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:09:25.793 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:09:25.793 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:09:25.794 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:09:25.794 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:09:25.794 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:09:25.794 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:09:25.795 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:09:25.795 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:09:25.795 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:09:25.795 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:09:25.795 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:09:25.796 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:09:25.796 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:09:25.796 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:09:26.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:26 smithi154 conmon[34553]: cluster 2021-07-31T03:09:24.974245+0000 mgr.x (mgr.24542) 3518 : cluster [DBG] pgmap v4473: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:26.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:26 smithi154 conmon[34553]: audit 2021-07-31T03:09:25.165814+0000 mgr.x (mgr.24542) 3519 : audit [DBG] from='client.28852 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:09:26.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:26 smithi154 conmon[34553]: audit 2021-07-31T03:09:25.448939+0000 mgr.x (mgr.24542) 3520 : audit [DBG] from='client.18918 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:09:26.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:26 smithi154 conmon[34553]: audit 2021-07-31T03:09:26.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:26 smithi154 conmon[34553]: 2021-07-31T03:09:25.793352+0000 mon.c (mon.1) 179 : audit [DBG] from='client.? 172.21.15.45:0/3524931329' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:09:26.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:26 smithi045 conmon[33655]: cluster 2021-07-31T03:09: 2021-07-31T03:09:26.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:26 smithi045 conmon[33655]: 24.974245+0000 mgr.x (mgr.24542) 3518 : cluster [DBG] pgmap v4473: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:26.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:26 smithi045 conmon[33655]: audit 2021-07- 2021-07-31T03:09:26.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:26 smithi045 conmon[33655]: 31T03:09:25.165814+0000 mgr.x (mgr.24542) 3519 : audit [DBG] from='client.28852 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:09:26.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:26 smithi045 conmon[33655]: audit 2021-07- 2021-07-31T03:09:26.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:26 smithi045 conmon[33655]: 31T03:09:25.448939+0000 mgr.x (mgr.24542) 3520 : audit 2021-07-31T03:09:26.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:26 smithi045 conmon[33655]: [DBG] from='client.18918 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:09:26.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:26 smithi045 conmon[33655]: audit 2021-07-31T03:09:26.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:26 smithi045 conmon[33655]: 2021-07 2021-07-31T03:09:26.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:26 smithi045 conmon[33655]: -31T03:09:25.793352+0000 mon.c 2021-07-31T03:09:26.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:26 smithi045 conmon[33655]: (mon.1) 179 : audit [DBG] from='client.? 172.21.15.45:0/3524931329' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:09:26.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:26 smithi045 conmon[33655]: 2021-07-31T03:09:26.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:26 smithi045 conmon[39718]: cluster 2021-07-31T03:09:24.974245 2021-07-31T03:09:26.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:26 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 3518 : cluster [DBG] pgmap v4473: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:26.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:26 smithi045 conmon[39718]: audit 2021-07-31T03:09:26.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:26 smithi045 conmon[39718]: 2021-07-31T03:09:25. 2021-07-31T03:09:26.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:26 smithi045 conmon[39718]: 165814+0000 mgr.x (mgr.24542) 3519 : 2021-07-31T03:09:26.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:26 smithi045 conmon[39718]: audit [DBG] from='client.28852 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:09:26.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:26 smithi045 conmon[39718]: audit 2021-07-31T03:09:26.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:26 smithi045 conmon[39718]: 2021-07-31T03:09:25 2021-07-31T03:09:26.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:26 smithi045 conmon[39718]: .448939+0000 mgr.x (mgr.24542) 3520 : audit [DBG] from='client.18918 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:09:26.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:26 smithi045 conmon[39718]: audit 2021-07-31T03 2021-07-31T03:09:26.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:26 smithi045 conmon[39718]: :09:25.793352+0000 mon.c ( 2021-07-31T03:09:26.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:26 smithi045 conmon[39718]: mon.1) 179 : audit [DBG] from='client.? 172.21.15.45:0/3524931329' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:09:26.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:26 smithi045 conmon[39718]: 2021-07-31T03:09:28.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:28 smithi154 conmon[34553]: cluster 2021-07-31T03:09:26.974610+0000 mgr.x (mgr.24542) 3521 : cluster [DBG] pgmap v4474: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:28.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:28 smithi045 conmon[33655]: cluster 2021-07-31T03:09:26.974610+0000 mgr.x (mgr.24542) 3521 : cluster [DBG] pgmap v4474: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:28.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:28 smithi045 conmon[39718]: cluster 2021-07-31T03:09:26.974610+0000 mgr.x ( 2021-07-31T03:09:28.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:28 smithi045 conmon[39718]: mgr.24542) 3521 : cluster [DBG] pgmap v4474: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:29.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:29 smithi154 conmon[34553]: audit 2021-07-31T03:09:28.260753+0000 mon.b (mon.2) 485 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:09:29.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:29 smithi154 conmon[34553]: audit 2021-07-31 2021-07-31T03:09:29.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:29 smithi154 conmon[34553]: T03:09:28.261065+0000 mon.b (mon.2) 486 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:09:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:29 smithi154 conmon[34553]: audit 2021-07-31T03:09:28.261505+0000 mon.a (mon.0) 3260 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:09:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:29 smithi154 conmon[34553]: audit 2021-07- 2021-07-31T03:09:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:29 smithi154 conmon[34553]: 31T03:09:28.261947+0000 mon.a (mon.0) 3261 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:09:29.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:29 smithi045 conmon[33655]: audit 2021-07-31T03:09:28.260753+0000 mon.b (mon.2) 485 2021-07-31T03:09:29.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:29 smithi045 conmon[33655]: : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:09:29.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:29 smithi045 conmon[33655]: audit 2021-07-31T03:09:28.261065+0000 mon.b (mon.2) 486 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:09:29.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:29 smithi045 conmon[33655]: audit 2021-07-31T03:09:28.261505 2021-07-31T03:09:29.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:29 smithi045 conmon[33655]: +0000 mon.a (mon.0) 3260 : audit [INF] 2021-07-31T03:09:29.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:29 smithi045 conmon[33655]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:09:29.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:29 smithi045 conmon[33655]: audit 2021-07-31T03: 2021-07-31T03:09:29.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:29 smithi045 conmon[33655]: 09:28.261947+0000 mon.a (mon.0) 3261 2021-07-31T03:09:29.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:29 smithi045 conmon[33655]: : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:09:29.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:29 smithi045 conmon[39718]: audit 2021-07-31T03:09:28.260753+0000 mon.b 2021-07-31T03:09:29.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:29 smithi045 conmon[39718]: (mon.2) 485 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:09:29.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:29 smithi045 conmon[39718]: audit 2021-07-31T03:09:28 2021-07-31T03:09:29.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:29 smithi045 conmon[39718]: .261065+0000 mon.b (mon.2) 486 : audit [INF] 2021-07-31T03:09:29.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:29 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:09:29.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:29 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T03:09:29.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:29 smithi045 conmon[39718]: 31T03:09:28.261505 2021-07-31T03:09:29.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:29 smithi045 conmon[39718]: +0000 mon.a (mon.0) 3260 2021-07-31T03:09:29.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:29 smithi045 conmon[39718]: : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:09:29.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:29 smithi045 conmon[39718]: audit 2021- 2021-07-31T03:09:29.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:29 smithi045 conmon[39718]: 07-31T03:09:28.261947 2021-07-31T03:09:29.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:29 smithi045 conmon[39718]: +0000 mon.a (mon.0) 3261 2021-07-31T03:09:29.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:29 smithi045 conmon[39718]: : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:09:30.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:30 smithi154 conmon[34553]: cluster 2021-07-31T03:09:28.975016+0000 mgr.x (mgr.24542) 3522 : cluster [DBG] pgmap v4475: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:30.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:30 smithi045 conmon[33655]: cluster 2021-07-31T03:09:28.975016+0000 mgr.x (mgr.24542) 3522 : cluster [DBG] 2021-07-31T03:09:30.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:30 smithi045 conmon[33655]: pgmap v4475: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:30.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:30 smithi045 conmon[39718]: cluster 2021-07-31T03:09:28.975016+0000 mgr.x ( 2021-07-31T03:09:30.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:30 smithi045 conmon[39718]: mgr.24542) 3522 : cluster [DBG] pgmap v4475: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:32.003 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: debug 2021-07-31T03:09:31.610+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:09:32.003 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:09:32.003 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:09:32.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: yield 2021-07-31T03:09:32.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:09:32.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:09:32.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:09:32.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:09:32.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:09:32.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:09:32.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:09:32.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:09:32.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:09:32.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: task.result() 2021-07-31T03:09:32.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:09:32.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: debug 2021-07-31T03:09:31.621+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:09:32.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:09:32.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:09:32.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: yield 2021-07-31T03:09:32.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:09:32.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:09:32.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:09:32.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:09:32.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:09:32.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:09:32.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:09:32.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:09:32.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:09:32.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: task.result() 2021-07-31T03:09:32.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:09:32.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: debug 2021-07-31T03:09:31.647+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:09:32.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:09:32.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:09:32.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: yield 2021-07-31T03:09:32.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:09:32.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:09:32.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:09:32.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:09:32.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:09:32.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:09:32.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:09:32.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:09:32.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:09:32.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: task.result() 2021-07-31T03:09:32.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:09:32.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: debug 2021-07-31T03:09:31.647+0000 7fcb9195d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:09:32.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:09:32.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:09:32.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: yield 2021-07-31T03:09:32.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:09:32.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:09:32.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:09:32.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:09:32.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:09:32.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:09:32.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:09:32.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:09:32.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:09:32.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: task.result() 2021-07-31T03:09:32.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:09:32.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: 2021-07-31T03:09:32.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:09:32.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: 2021-07-31T03:09:32.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:09:32.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:09:32.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:09:32.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:09:32.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:09:32.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:09:32.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:09:32.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:09:32.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:09:32.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:09:32.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:09:32.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:09:32.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:09:32.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:09:32.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:09:32.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:09:32.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:09:32.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:09:32.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: raise 2021-07-31T03:09:32.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:09:32.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:09:32.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:09:32.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:09:32.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:09:32.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: debug 2021-07-31T03:09:31.656+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:09:32.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:09:32.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:09:32.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: yield 2021-07-31T03:09:32.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:09:32.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:09:32.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:09:32.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:09:32.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:09:32.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:09:32.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:09:32.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:09:32.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:09:32.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: task.result() 2021-07-31T03:09:32.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:09:32.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: debug 2021-07-31T03:09:31.656+0000 7fcb9215e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:09:32.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:09:32.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:09:32.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: yield 2021-07-31T03:09:32.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:09:32.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:09:32.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:09:32.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:09:32.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:09:32.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:09:32.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:09:32.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:09:32.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:09:32.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: task.result() 2021-07-31T03:09:32.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:09:32.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: 2021-07-31T03:09:32.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:09:32.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: 2021-07-31T03:09:32.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:09:32.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:09:32.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:09:32.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:09:32.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:09:32.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:09:32.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:09:32.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:09:32.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:09:32.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:09:32.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:09:32.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:09:32.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:09:32.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:09:32.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:09:32.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:09:32.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:09:32.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:09:32.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: raise 2021-07-31T03:09:32.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:09:32.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:09:32.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:09:32.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:09:32.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:09:31 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:09:32.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: cluster 2021-07-31T03:09:30.975388+0000 mgr.x (mgr.24542) 3523 : cluster [DBG] pgmap v4476: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:32.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: 2021-07-31T03:09:32.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: audit 2021-07-31T03:09:31.564181+0000 mon.b (mon.2) 487 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:09:32.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: audit 2021-07-31T03:09:32.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: 2021-07-31T03:09:31.565814+0000 mon.b (mon.2) 488 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:09:32.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: audit 2021- 2021-07-31T03:09:32.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: 07-31T03:09:31.566754+0000 mon.b (mon.2) 489 : audit [INF] 2021-07-31T03:09:32.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:09:32.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: cephadm 2021-07-31T03:09: 2021-07-31T03:09:32.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: 31.567508+0000 mgr.x (mgr.24542) 3524 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:09:32.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: cephadm 2021-07-31T03:09:31.611051+0000 mgr.x (mgr.24542) 3525 2021-07-31T03:09:32.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: : cephadm [ERR] 2021-07-31T03:09:32.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:09:32.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:09:32.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: yield 2021-07-31T03:09:32.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:09:32.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:09:32.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:09:32.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:09:32.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:09:32.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:09:32.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:09:32.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:09:32.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:09:32.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: task.result() 2021-07-31T03:09:32.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:09:32.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: cephadm 2021-07-31T03:09:31.611477+0000 mgr.x (mgr.24542) 3526 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:09:32.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: cephadm 2021-07-31T03:09:31.622451+0000 mgr.x (mgr.24542) 3527 : cephadm [ERR] 2021-07-31T03:09:32.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:09:32.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:09:32.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: yield 2021-07-31T03:09:32.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:09:32.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:09:32.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:09:32.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:09:32.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:09:32.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:09:32.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:09:32.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:09:32.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:09:32.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: task.result() 2021-07-31T03:09:32.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:09:32.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: cephadm 2021-07-31T03:09:31.622882+0000 mgr.x (mgr.24542) 3528 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:09:32.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: cephadm 2021-07-31T03:09:31.648052+0000 mgr.x (mgr.24542) 3529 : cephadm [ERR] 2021-07-31T03:09:32.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:09:32.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:09:32.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: yield 2021-07-31T03:09:32.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:09:32.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:09:32.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:09:32.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:09:32.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:09:32.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:09:32.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:09:32.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:09:32.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:09:32.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: task.result() 2021-07-31T03:09:32.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:09:32.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: cephadm 2021-07-31T03:09:31.648518+0000 mgr.x (mgr.24542) 3530 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:09:32.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:09:32.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:09:32.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: yield 2021-07-31T03:09:32.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:09:32.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:09:32.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:09:32.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:09:32.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:09:32.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:09:32.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:09:32.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:09:32.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:09:32.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: task.result() 2021-07-31T03:09:32.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:09:32.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: 2021-07-31T03:09:32.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:09:32.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: 2021-07-31T03:09:32.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:09:32.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:09:32.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:09:32.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:09:32.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:09:32.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:09:32.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:09:32.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:09:32.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:09:32.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:09:32.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:09:32.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:09:32.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:09:32.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:09:32.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:09:32.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:09:32.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:09:32.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:09:32.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: raise 2021-07-31T03:09:32.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:09:32.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:09:32.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:09:32.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:09:32.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:09:32.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: cephadm 2021-07-31T03: 2021-07-31T03:09:32.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: 09:31.657443+0000 mgr.x (mgr.24542) 3531 : cephadm [ERR] 2021-07-31T03:09:32.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:09:32.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:09:32.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: yield 2021-07-31T03:09:32.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:09:32.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:09:32.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:09:32.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:09:32.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:09:32.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:09:32.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:09:32.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:09:32.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:09:32.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: task.result() 2021-07-31T03:09:32.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:09:32.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: cluster 2021-07-31T03:09:30.975388+0000 mgr.x (mgr.24542) 3523 : 2021-07-31T03:09:32.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: cluster [DBG] pgmap v4476: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:32.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: audit 2021-07-31T03:09:31.564181+0000 mon.b (mon.2) 487 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:09:32.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: audit 2021-07-31T03:09:31.565814+0000 mon.b (mon.2) 488 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:09:32.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: audit 2021-07-31T03:09:31.566754+0000 mon.b (mon.2) 489 2021-07-31T03:09:32.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:09:32.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: cephadm 2021 2021-07-31T03:09:32.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: -07-31T03:09:31.567508+0000 mgr.x (mgr.24542) 3524 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:09:32.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: cephadm 2021-07-31T03: 2021-07-31T03:09:32.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: 09:31.611051+0000 mgr.x (mgr.24542) 3525 : cephadm [ERR] 2021-07-31T03:09:32.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:09:32.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:09:32.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: yield 2021-07-31T03:09:32.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:09:32.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:09:32.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:09:32.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:09:32.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:09:32.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:09:32.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:09:32.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:09:32.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:09:32.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: task.result() 2021-07-31T03:09:32.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:09:32.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: cephadm 2021-07-31T03:09:31.611477+0000 mgr.x (mgr.24542) 3526 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:09:32.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: cephadm 2021-07-31T03:09:31.622451+0000 mgr.x (mgr.24542) 3527 : cephadm [ERR] 2021-07-31T03:09:32.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: 2021-07-31T03:09:32.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:09:32.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:09:32.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: yield 2021-07-31T03:09:32.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:09:32.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:09:32.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:09:32.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:09:32.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:09:32.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: cephadm 2021-07-31T03:09:31.657832+0000 mgr.x (mgr.24542) 3532 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:09:32.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:09:32.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:09:32.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: yield 2021-07-31T03:09:32.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:09:32.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:09:32.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:09:32.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:09:32.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:09:32.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:09:32.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:09:32.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:09:32.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:09:32.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: task.result() 2021-07-31T03:09:32.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:09:32.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: 2021-07-31T03:09:32.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:09:32.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: 2021-07-31T03:09:32.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:09:32.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:09:32.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:09:32.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:09:32.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:09:32.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:09:32.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:09:32.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:09:32.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:09:32.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:09:32.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:09:32.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:09:32.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:09:32.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:09:32.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:09:32.541 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:09:32.541 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:09:32.541 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:09:32.541 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: raise 2021-07-31T03:09:32.541 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:09:32.542 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:09:32.542 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:09:32.542 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:09:32.542 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:32 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:09:32.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: cluster 2021-07-31T03:09 2021-07-31T03:09:32.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: :30.975388+0000 mgr.x (mgr.24542) 3523 : cluster [DBG] pgmap v4476: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:32.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: audit 2021-07-31T03:09:32.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: 2021-07-31T03 2021-07-31T03:09:32.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: :09:31.564181+0000 2021-07-31T03:09:32.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: mon.b (mon.2) 487 : 2021-07-31T03:09:32.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:09:32.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: audit 2021-07-31T03:09 2021-07-31T03:09:32.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: :31.565814+0000 mon.b (mon.2) 488 : audit [DBG] 2021-07-31T03:09:32.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:09:32.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: audit 2021-07-31T03 2021-07-31T03:09:32.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: :09:31.566754+0000 mon.b (mon 2021-07-31T03:09:32.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: .2) 489 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:09:32.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: cephadm 2021-07-31 2021-07-31T03:09:32.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: T03:09:31.567508+0000 mgr.x (mgr. 2021-07-31T03:09:32.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: 24542) 3524 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:09:32.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: cephadm 2021- 2021-07-31T03:09:32.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: 07-31T03:09:31.611051 2021-07-31T03:09:32.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 3525 : cephadm 2021-07-31T03:09:32.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: [ERR] 2021-07-31T03:09:32.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:09:32.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:09:32.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: yield 2021-07-31T03:09:32.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:09:32.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:09:32.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:09:32.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:09:32.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:09:32.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:09:32.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:09:32.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:09:32.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:09:32.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: task.result() 2021-07-31T03:09:32.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:09:32.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: cephadm 2021-07-31T03:09:31.611477+0000 mgr.x (mgr.24542) 3526 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:09:32.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: cephadm 2021-07-31T03:09: 2021-07-31T03:09:32.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: 31.622451+0000 mgr.x (mgr.24542) 3527 : cephadm [ERR] 2021-07-31T03:09:32.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: 2021-07-31T03:09:32.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:09:32.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:09:32.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: yield 2021-07-31T03:09:32.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:09:32.551 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:09:32.552 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:09:32.552 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:09:32.552 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:09:32.552 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: task.result() 2021-07-31T03:09:32.552 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:09:32.553 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: cephadm 2021-07-31 2021-07-31T03:09:32.553 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: T03:09:31.622882+0000 mgr.x (mgr.24542) 3528 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:09:32.553 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: cephadm 2021-07-31T03:09:31 2021-07-31T03:09:32.553 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: .648052+0000 mgr.x (mgr.24542) 3529 : cephadm [ERR] 2021-07-31T03:09:32.553 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:09:32.554 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:09:32.554 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: yield 2021-07-31T03:09:32.554 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:09:32.554 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:09:32.554 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:09:32.555 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:09:32.555 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:09:32.555 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:09:32.555 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:09:32.555 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:09:32.556 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:09:32.556 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: task.result() 2021-07-31T03:09:32.556 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:09:32.556 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: cephadm 2021-07-31T03:09:31.648518+0000 mgr.x (mgr.24542) 3530 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:09:32.556 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:09:32.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:09:32.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: yield 2021-07-31T03:09:32.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:09:32.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:09:32.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:09:32.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:09:32.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:09:32.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:09:32.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:09:32.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:09:32.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:09:32.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: task.result() 2021-07-31T03:09:32.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:09:32.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: 2021-07-31T03:09:32.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:09:32.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: 2021-07-31T03:09:32.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:09:32.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:09:32.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:09:32.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:09:32.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:09:32.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:09:32.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:09:32.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:09:32.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: task.result() 2021-07-31T03:09:32.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:09:32.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: cephadm 2021-07-31T03:09:31.622882+0000 mgr.x (mgr.24542) 3528 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:09:32.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: cephadm 2021-07-31T03:09:32.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: 2021-07-31T03:09:31.648052+0000 2021-07-31T03:09:32.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: mgr.x (mgr.24542) 3529 : cephadm [ERR] 2021-07-31T03:09:32.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: 2021-07-31T03:09:32.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:09:32.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:09:32.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: yield 2021-07-31T03:09:32.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:09:32.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:09:32.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:09:32.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:09:32.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:09:32.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:09:32.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:09:32.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:09:32.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:09:32.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: task.result() 2021-07-31T03:09:32.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:09:32.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: cephadm 2021-07-31T03:09:31.648518+0000 mgr.x (mgr.24542) 3530 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:09:32.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:09:32.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:09:32.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: yield 2021-07-31T03:09:32.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:09:32.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:09:32.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:09:32.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:09:32.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:09:32.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:09:32.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:09:32.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:09:32.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:09:32.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:09:32.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:09:32.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:09:32.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:09:32.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:09:32.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:09:32.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:09:32.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:09:32.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:09:32.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:09:32.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:09:32.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:09:32.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:09:32.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:09:32.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:09:32.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:09:32.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: raise 2021-07-31T03:09:32.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:09:32.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:09:32.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:09:32.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:09:32.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:09:32.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: cephadm 2021-07-31T03:09:31.657443+0000 2021-07-31T03:09:32.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: mgr.x (mgr.24542) 3531 : cephadm [ERR] 2021-07-31T03:09:32.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:09:32.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:09:32.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: yield 2021-07-31T03:09:32.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:09:32.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:09:32.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:09:32.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:09:32.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:09:32.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:09:32.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:09:32.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:09:32.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:09:32.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: task.result() 2021-07-31T03:09:32.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:09:32.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:09:32.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:09:32.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: task.result() 2021-07-31T03:09:32.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:09:32.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: 2021-07-31T03:09:32.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:09:32.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: 2021-07-31T03:09:32.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:09:32.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:09:32.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:09:32.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:09:32.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:09:32.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:09:32.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:09:32.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:09:32.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:09:32.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:09:32.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:09:32.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:09:32.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:09:32.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:09:32.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:09:32.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:09:32.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:09:32.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:09:32.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: raise 2021-07-31T03:09:32.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:09:32.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:09:32.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:09:32.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:09:32.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:09:32.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: cephadm 2021-07 2021-07-31T03:09:32.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: -31T03:09:31.657443+0000 mgr.x (mgr.24542) 3531 : cephadm [ERR] 2021-07-31T03:09:32.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:09:32.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:09:32.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: yield 2021-07-31T03:09:32.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:09:32.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:09:32.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:09:32.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:09:32.585 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:09:32.585 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:09:32.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: cephadm 2021-07-31T03:09:31.657832+0000 mgr.x (mgr.24542) 3532 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:09:32.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:09:32.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:09:32.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: yield 2021-07-31T03:09:32.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:09:32.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:09:32.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:09:32.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:09:32.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:09:32.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:09:32.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:09:32.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:09:32.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:09:32.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: task.result() 2021-07-31T03:09:32.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:09:32.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: 2021-07-31T03:09:32.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:09:32.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: 2021-07-31T03:09:32.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:09:32.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:09:32.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:09:32.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:09:32.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:09:32.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:09:32.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:09:32.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:09:32.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:09:32.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:09:32.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:09:32.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:09:32.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:09:32.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:09:32.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:09:32.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:09:32.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:09:32.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:09:32.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: raise 2021-07-31T03:09:32.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:09:32.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:09:32.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:09:32.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:09:32.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:09:32.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:09:32.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:09:32.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:09:32.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: task.result() 2021-07-31T03:09:32.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:09:32.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: cephadm 2021-07-31T03:09:31.657832+0000 mgr.x (mgr.24542) 3532 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:09:32.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:09:32.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:09:32.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: yield 2021-07-31T03:09:32.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:09:32.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:09:32.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:09:32.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:09:32.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:09:32.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:09:32.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:09:32.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:09:32.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:09:32.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: task.result() 2021-07-31T03:09:32.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:09:32.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: 2021-07-31T03:09:32.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:09:32.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: 2021-07-31T03:09:32.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:09:32.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:09:32.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:09:32.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:09:32.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:09:32.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:09:32.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:09:32.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:09:32.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:09:32.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:09:32.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:09:32.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:09:32.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:09:32.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:09:32.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:09:32.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:09:32.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:09:32.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:09:32.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: raise 2021-07-31T03:09:32.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:09:32.603 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:09:32.603 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:09:32.603 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:09:32.603 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:32 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:09:34.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:34 smithi154 conmon[34553]: cluster 2021-07-31T03:09:32.975841+0000 mgr.x (mgr.24542) 3533 : cluster [DBG] pgmap v4477: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:34.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:34 smithi154 conmon[34553]: 2021-07-31T03:09:34.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:34 smithi045 conmon[33655]: cluster 2021-07-31T03:09:32.975841 2021-07-31T03:09:34.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:34 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 3533 : cluster [DBG] pgmap v4477: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:34.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:34 smithi045 conmon[39718]: cluster 2021-07-31T03:09:32.975841+0000 mgr.x 2021-07-31T03:09:34.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:34 smithi045 conmon[39718]: (mgr.24542) 3533 : cluster [DBG] pgmap v4477: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:36.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:36 smithi154 conmon[34553]: cluster 2021-07-31T03:09:34.976318+0000 mgr.x (mgr.24542) 3534 : cluster [DBG] pgmap v4478: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:36.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:36 smithi045 conmon[33655]: cluster 2021-07-31T03:09:34.976318 2021-07-31T03:09:36.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:36 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 3534 : cluster [DBG] pgmap v4478: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:36.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:36 smithi045 conmon[39718]: cluster 2021-07-31T03:09:34.976318+0000 mgr.x ( 2021-07-31T03:09:36.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:36 smithi045 conmon[39718]: mgr.24542) 3534 : cluster [DBG] pgmap v4478: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:38.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:38 smithi154 conmon[34553]: cluster 2021-07-31T03:09:36.976783+0000 mgr.x (mgr.24542) 3535 : cluster [DBG] pgmap v4479: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:38.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:38 smithi045 conmon[33655]: cluster 2021-07-31T03:09:36.976783+0000 mgr.x (mgr.24542 2021-07-31T03:09:38.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:38 smithi045 conmon[33655]: ) 3535 : cluster [DBG] pgmap v4479: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:38.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:38 smithi045 conmon[39718]: cluster 2021-07-31T03:09:36.976783+0000 mgr.x (mgr 2021-07-31T03:09:38.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:38 smithi045 conmon[39718]: .24542) 3535 : cluster [DBG] pgmap v4479: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:40.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:40 smithi154 conmon[34553]: cluster 2021-07-31T03:09:38.977209+0000 mgr.x (mgr.24542) 3536 : cluster [DBG] 2021-07-31T03:09:40.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:40 smithi154 conmon[34553]: pgmap v4480: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:40.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:40 smithi045 conmon[33655]: cluster 2021-07-31T03:09:38.977209+0000 mgr.x (mgr.24542) 3536 : cluster [DBG] 2021-07-31T03:09:40.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:40 smithi045 conmon[33655]: pgmap v4480: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:40.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:40 smithi045 conmon[39718]: cluster 2021-07-31T03:09:38.977209 2021-07-31T03:09:40.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:40 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 3536 : cluster [DBG] pgmap v4480: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:42.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:42 smithi154 conmon[34553]: cluster 2021-07-31T03:09:40.977720+0000 mgr.x (mgr.24542) 3537 : cluster [DBG] pgmap v4481: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:42.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:42 smithi045 conmon[39718]: cluster 2021-07-31T03:09:40.977720+0000 2021-07-31T03:09:42.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:42 smithi045 conmon[39718]: mgr.x (mgr.24542) 3537 : cluster [DBG] pgmap v4481: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:42.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:42 smithi045 conmon[33655]: cluster 2021-07-31T03:09:40.977720+0000 mgr.x (mgr.24542) 3537 : cluster 2021-07-31T03:09:42.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:42 smithi045 conmon[33655]: [DBG] pgmap v4481: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:44.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:44 smithi154 conmon[34553]: cluster 2021-07-31T03:09:42.978128+0000 mgr.x (mgr.24542) 3538 : cluster [DBG] pgmap v4482: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:44.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:44 smithi045 conmon[33655]: cluster 2021-07-31T03:09:42.978128+0000 mgr.x (mgr.24542) 3538 : cluster 2021-07-31T03:09:44.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:44 smithi045 conmon[33655]: [DBG] pgmap v4482: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:44.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:44 smithi045 conmon[39718]: cluster 2021-07-31T03:09:42. 2021-07-31T03:09:44.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:44 smithi045 conmon[39718]: 978128+0000 mgr.x (mgr.24542) 3538 : cluster [DBG] pgmap v4482: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:46.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:46 smithi154 conmon[34553]: cluster 2021-07-31T03:09:44.978726+0000 mgr.x (mgr.24542) 3539 : cluster [DBG] pgmap v4483: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:46.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:46 smithi045 conmon[39718]: cluster 2021-07-31T03:09:44.978726+0000 mgr.x (mgr.24542) 3539 : cluster [DBG] pgmap v4483: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:46.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:46 smithi045 conmon[33655]: cluster 2021-07-31T03:09:44.978726+0000 mgr.x (mgr.24542 2021-07-31T03:09:46.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:46 smithi045 conmon[33655]: ) 3539 : cluster [DBG] pgmap v4483: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:48.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:48 smithi154 conmon[34553]: cluster 2021-07-31T03:09:46.979193+0000 mgr.x (mgr.24542) 3540 : cluster [DBG] pgmap v4484: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:48.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:48 smithi045 conmon[39718]: cluster 2021-07-31T03:09:46.979193+0000 mgr.x 2021-07-31T03:09:48.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:48 smithi045 conmon[39718]: (mgr.24542) 3540 : cluster [DBG] pgmap v4484: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:48.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:48 smithi045 conmon[33655]: cluster 2021-07-31T03:09:46.979193 2021-07-31T03:09:48.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:48 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 3540 : cluster [DBG] pgmap v4484: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:50.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:50 smithi154 conmon[34553]: cluster 2021-07-31T03:09:48.979561+0000 mgr.x (mgr.24542) 3541 : cluster [DBG] pgmap v4485: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:50.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:50 smithi045 conmon[33655]: cluster 2021-07-31T03:09:48.979561+0000 mgr.x (mgr.24542) 3541 2021-07-31T03:09:50.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:50 smithi045 conmon[33655]: : cluster [DBG] pgmap v4485: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:50.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:50 smithi045 conmon[39718]: cluster 2021-07-31T03:09:48.979561+0000 mgr.x (mgr.24542) 3541 : cluster [DBG] pgmap v4485: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:52.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:52 smithi154 conmon[34553]: cluster 2021-07-31T03:09:50.980096+0000 mgr.x (mgr.24542) 3542 : cluster [DBG] pgmap v4486: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:52.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:52 smithi045 conmon[33655]: cluster 2021-07-31T03:09:50.980096+0000 mgr.x (mgr 2021-07-31T03:09:52.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:52 smithi045 conmon[33655]: .24542) 3542 : cluster [DBG] pgmap v4486: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:52.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:52 smithi045 conmon[39718]: cluster 2021-07-31T03:09:50.980096+0000 mgr.x (mgr.24542) 3542 : cluster [DBG] pgmap v4486: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:54.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:54 smithi154 conmon[34553]: cluster 2021-07-31T03:09:52.980604+0000 mgr.x (mgr.24542) 3543 : cluster [DBG] pgmap v4487: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:54.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:54 smithi045 conmon[33655]: cluster 2021-07-31T03:09:52.980604+0000 mgr.x (mgr.24542) 3543 : cluster 2021-07-31T03:09:54.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:54 smithi045 conmon[33655]: [DBG] pgmap v4487: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:54.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:54 smithi045 conmon[39718]: cluster 2021-07-31T03:09:52.980604+0000 mgr.x 2021-07-31T03:09:54.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:54 smithi045 conmon[39718]: (mgr.24542) 3543 : cluster [DBG] pgmap v4487: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:56.107 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:09:56.384 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:09:56.384 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 84m ago 85m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:09:56.384 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 82m ago 85m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:09:56.385 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 82m ago 88m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:09:56.385 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 84m ago 90m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:09:56.385 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 84m ago 91m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:09:56.385 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 82m ago 89m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:09:56.385 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 84m ago 89m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:09:56.386 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 84m ago 85m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:09:56.386 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 82m ago 85m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:09:56.386 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 84m ago 88m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:09:56.386 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 84m ago 88m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:09:56.387 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 84m ago 87m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:09:56.387 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 84m ago 87m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:09:56.387 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 82m ago 87m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:09:56.387 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 82m ago 86m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:09:56.387 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 82m ago 86m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:09:56.388 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 82m ago 86m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:09:56.388 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 82m ago 85m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:09:56.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:56 smithi154 conmon[34553]: cluster 2021-07-31T03:09:54.981105+0000 mgr.x (mgr.24542) 3544 : cluster [DBG] pgmap v4488: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:56.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:56 smithi045 conmon[33655]: cluster 2021-07-31T03:09:54.981105+0000 mgr.x (mgr. 2021-07-31T03:09:56.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:56 smithi045 conmon[33655]: 24542) 3544 : cluster [DBG] pgmap v4488: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:56.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:56 smithi045 conmon[39718]: cluster 2021-07-31T03:09:54.981105+0000 2021-07-31T03:09:56.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:56 smithi045 conmon[39718]: mgr.x (mgr.24542) 3544 : cluster [DBG] pgmap v4488: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:56.719 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:09:56.719 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:09:56.719 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:09:56.719 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:09:56.720 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:09:56.720 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:09:56.720 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:09:56.720 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:09:56.721 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:09:56.721 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:09:56.721 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:09:56.721 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:09:56.721 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:09:56.722 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:09:56.722 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:09:56.722 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:09:56.722 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:09:57.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:57 smithi154 conmon[34553]: audit 2021-07-31T03:09:56.095568+0000 mgr.x (mgr.24542) 3545 : audit [DBG] 2021-07-31T03:09:57.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:57 smithi154 conmon[34553]: from='client.28870 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:09:57.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:57 smithi154 conmon[34553]: audit 2021-07-31T03:09:56.376351+0000 mgr.x (mgr.24542) 3546 : audit [DBG] from='client.18936 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:09:57.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:57 smithi154 conmon[34553]: audit 2021-07-31T03:09:56.719834+0000 mon.c (mon.1) 180 : audit [DBG] from='client.? 172.21.15.45:0/1666637275' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:09:57.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:57 smithi045 conmon[39718]: audit 2021-07-31T03:09:56.095568+0000 mgr.x ( 2021-07-31T03:09:57.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:57 smithi045 conmon[39718]: mgr.24542) 3545 : audit [DBG] from='client.28870 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:09:57.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:57 smithi045 conmon[39718]: audit 2021-07-31T03:09:56.376351+0000 mgr.x ( 2021-07-31T03:09:57.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:57 smithi045 conmon[39718]: mgr.24542) 3546 : audit [DBG] from='client.18936 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:09:57.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:57 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T03:09:57.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:57 smithi045 conmon[39718]: 31T03:09:56.719834+0000 mon.c (mon.1) 180 : audit [DBG] from='client.? 172.21.15.45:0/1666637275' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:09:57.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:57 smithi045 conmon[33655]: audit 2021-07-31T03:09:56.095568+0000 mgr.x (mgr.24542) 3545 : audit 2021-07-31T03:09:57.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:57 smithi045 conmon[33655]: [DBG] from='client.28870 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:09:57.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:57 smithi045 conmon[33655]: audit 2021-07-31T03:09:56.376351+0000 mgr.x (mgr.24542) 3546 : audit [DBG] from='client.18936 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:09:57.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:57 smithi045 conmon[33655]: audit 2021-07-31T03:09:56 2021-07-31T03:09:57.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:57 smithi045 conmon[33655]: .719834+0000 mon.c (mon.1 2021-07-31T03:09:57.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:57 smithi045 conmon[33655]: ) 180 : audit [DBG] from='client.? 172.21.15.45:0/1666637275' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:09:58.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:09:58 smithi154 conmon[34553]: cluster 2021-07-31T03:09:56.981457+0000 mgr.x (mgr.24542) 3547 : cluster [DBG] pgmap v4489: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:58.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:58 smithi045 conmon[39718]: cluster 2021-07-31T03:09:56.981457+0000 mgr.x (mgr.24542 2021-07-31T03:09:58.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:09:58 smithi045 conmon[39718]: ) 3547 : cluster [DBG] pgmap v4489: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:09:58.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:58 smithi045 conmon[33655]: cluster 2021-07-31T03:09:56.981457+0000 mgr.x (mgr.24542) 2021-07-31T03:09:58.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:09:58 smithi045 conmon[33655]: 3547 : cluster [DBG] pgmap v4489: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:00.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:00 smithi154 conmon[34553]: cluster 2021-07-31T03:09:58.982023+0000 mgr.x (mgr.24542) 3548 : cluster [DBG] pgmap v4490: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:00.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:00 smithi154 conmon[34553]: cluster 2021-07-31T03:10:00.000160+0000 mon.a (mon.0) 3262 : cluster [INF] overall HEALTH_OK 2021-07-31T03:10:00.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:00 smithi045 conmon[39718]: cluster 2021-07-31T03:09:58 2021-07-31T03:10:00.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:00 smithi045 conmon[39718]: .982023+0000 mgr.x (mgr.24542) 3548 : cluster [DBG] pgmap v4490: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:00.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:00 smithi045 conmon[39718]: cluster 2021-07-31T03:10:00.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:00 smithi045 conmon[39718]: 2021- 2021-07-31T03:10:00.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:00 smithi045 conmon[39718]: 07-31T03:10: 2021-07-31T03:10:00.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:00 smithi045 conmon[39718]: 00.000160+0000 mon.a (mon 2021-07-31T03:10:00.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:00 smithi045 conmon[39718]: .0) 3262 : cluster [INF] 2021-07-31T03:10:00.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:00 smithi045 conmon[39718]: overall HEALTH_OK 2021-07-31T03:10:00.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:00 smithi045 conmon[33655]: cluster 2021-07- 2021-07-31T03:10:00.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:00 smithi045 conmon[33655]: 31T03:09:58.982023+0000 mgr.x (mgr.24542) 3548 : cluster [DBG] pgmap v4490: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:00.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:00 smithi045 conmon[33655]: cluster 2021-07-31T03:10:00.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:00 smithi045 conmon[33655]: 2021-07-31T03:10:00.000160+0000 mon.a (mon 2021-07-31T03:10:00.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:00 smithi045 conmon[33655]: .0) 3262 : cluster [INF] overall HEALTH_OK 2021-07-31T03:10:02.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:02 smithi154 conmon[34553]: cluster 2021-07-31T03:10:00.982372+0000 mgr.x (mgr.24542) 3549 : cluster [DBG] pgmap v4491: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:02.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:02 smithi045 conmon[39718]: cluster 2021-07-31T03:10:00 2021-07-31T03:10:02.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:02 smithi045 conmon[39718]: .982372+0000 mgr.x (mgr.24542) 3549 : cluster [DBG] pgmap v4491: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:02.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:02 smithi045 conmon[33655]: cluster 2021-07-31T03:10:00.982372+0000 mgr.x (mgr.24542) 3549 : cluster 2021-07-31T03:10:02.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:02 smithi045 conmon[33655]: [DBG] pgmap v4491: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:04.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:04 smithi154 conmon[34553]: cluster 2021-07-31T03:10:02.982741+0000 mgr.x (mgr.24542) 3550 : cluster [DBG] pgmap v4492: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:04.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:04 smithi045 conmon[39718]: cluster 2021-07-31T03:10:02.982741+0000 2021-07-31T03:10:04.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:04 smithi045 conmon[39718]: mgr.x (mgr.24542) 3550 : cluster [DBG] pgmap v4492: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:04.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:04 smithi045 conmon[33655]: cluster 2021-07-31T03:10:02.982741+0000 mgr.x (mgr.24542) 3550 : 2021-07-31T03:10:04.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:04 smithi045 conmon[33655]: cluster [DBG] pgmap v4492: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:06.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:06 smithi154 conmon[34553]: cluster 2021-07-31T03:10:04.983243+0000 mgr.x (mgr.24542) 3551 : cluster [DBG] pgmap v4493: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:06.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:06 smithi045 conmon[39718]: cluster 2021-07-31T03:10: 2021-07-31T03:10:06.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:06 smithi045 conmon[39718]: 04.983243+0000 mgr.x (mgr.24542) 3551 : cluster [DBG] pgmap v4493: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:06.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:06 smithi045 conmon[33655]: cluster 2021-07-31T03:10:04.983243+0000 mgr.x (mgr.24542) 3551 : cluster 2021-07-31T03:10:06.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:06 smithi045 conmon[33655]: [DBG] pgmap v4493: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:08.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:08 smithi154 conmon[34553]: cluster 2021-07-31T03:10:06.983597+0000 mgr.x (mgr.24542) 3552 : cluster [DBG] pgmap v4494: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:08.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:08 smithi154 conmon[34553]: 2021-07-31T03:10:08.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:08 smithi045 conmon[39718]: cluster 2021-07-31T03:10:06.983597 2021-07-31T03:10:08.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:08 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 3552 : cluster [DBG] pgmap v4494: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:08.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:08 smithi045 conmon[33655]: cluster 2021-07-31T03:10:06.983597+0000 mgr.x 2021-07-31T03:10:08.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:08 smithi045 conmon[33655]: (mgr.24542) 3552 : cluster [DBG] pgmap v4494: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:10.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:10 smithi154 conmon[34553]: cluster 2021-07-31T03:10:08.984057+0000 mgr.x (mgr.24542) 3553 : cluster [DBG] pgmap v4495: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:10.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:10 smithi045 conmon[39718]: cluster 2021-07-31T03:10:08.984057 2021-07-31T03:10:10.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:10 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 3553 : cluster [DBG] pgmap v4495: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:10.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:10 smithi045 conmon[33655]: cluster 2021-07-31T03:10:08.984057+0000 2021-07-31T03:10:10.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:10 smithi045 conmon[33655]: mgr.x (mgr.24542) 3553 : cluster [DBG] pgmap v4495: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:12.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:12 smithi154 conmon[34553]: cluster 2021-07-31T03:10:10.984430+0000 mgr.x (mgr.24542) 3554 : cluster [DBG] pgmap v4496: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:12.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:12 smithi045 conmon[39718]: cluster 2021-07-31T03:10:10.984430+0000 mgr.x (mgr. 2021-07-31T03:10:12.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:12 smithi045 conmon[39718]: 24542) 3554 : cluster [DBG] pgmap v4496: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:12.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:12 smithi045 conmon[33655]: cluster 2021-07-31T03: 2021-07-31T03:10:12.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:12 smithi045 conmon[33655]: 10:10.984430+0000 mgr.x (mgr.24542) 3554 : cluster [DBG] pgmap v4496: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:14.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:14 smithi154 conmon[34553]: cluster 2021-07-31T03:10:12.984824+0000 mgr.x (mgr.24542) 3555 : cluster [DBG] 2021-07-31T03:10:14.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:14 smithi154 conmon[34553]: pgmap v4497: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:14.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:14 smithi045 conmon[39718]: cluster 2021-07-31T03:10:12.984824+0000 mgr.x 2021-07-31T03:10:14.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:14 smithi045 conmon[39718]: (mgr.24542) 3555 : cluster [DBG] pgmap v4497: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:14.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:14 smithi045 conmon[33655]: cluster 2021-07-31T03:10:12.984824+0000 mgr.x (mgr.24542) 3555 : cluster [DBG] 2021-07-31T03:10:14.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:14 smithi045 conmon[33655]: pgmap v4497: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:16.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:16 smithi154 conmon[34553]: cluster 2021-07-31T03:10:14.985376+0000 mgr.x (mgr.24542) 3556 : cluster [DBG] 2021-07-31T03:10:16.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:16 smithi154 conmon[34553]: pgmap v4498: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:16.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:16 smithi045 conmon[39718]: cluster 2021-07-31T03:10:14.985376+0000 2021-07-31T03:10:16.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:16 smithi045 conmon[39718]: mgr.x (mgr.24542) 3556 : cluster [DBG] pgmap v4498: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:16.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:16 smithi045 conmon[33655]: cluster 2021-07-31T03:10:14.985376+0000 mgr.x (mgr.24542) 3556 : cluster [DBG] pgmap v4498: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:16.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:16 smithi045 conmon[33655]: 2021-07-31T03:10:18.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:18 smithi154 conmon[34553]: cluster 2021-07-31T03:10:16.985867+0000 mgr.x (mgr.24542) 3557 : cluster [DBG] pgmap v4499: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:18.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:18 smithi045 conmon[39718]: cluster 2021-07-31T03:10:16.985867+0000 mgr.x (mgr. 2021-07-31T03:10:18.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:18 smithi045 conmon[39718]: 24542) 3557 : cluster [DBG] pgmap v4499: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:18.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:18 smithi045 conmon[33655]: cluster 2021-07-31T03:10:16.985867+0000 mgr.x (mgr.24542) 3557 : 2021-07-31T03:10:18.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:18 smithi045 conmon[33655]: cluster [DBG] pgmap v4499: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:20.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:20 smithi154 conmon[34553]: cluster 2021-07-31T03:10:18.986424+0000 mgr.x (mgr.24542) 3558 : cluster [DBG] pgmap v4500: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:20.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:20 smithi045 conmon[39718]: cluster 2021-07-31T03:10:18.986424+0000 mgr.x (mgr 2021-07-31T03:10:20.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:20 smithi045 conmon[39718]: .24542) 3558 : cluster [DBG] pgmap v4500: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:20.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:20 smithi045 conmon[33655]: cluster 2021-07-31T03:10:18.986424+0000 mgr.x (mgr.24542) 3558 : cluster 2021-07-31T03:10:20.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:20 smithi045 conmon[33655]: [DBG] pgmap v4500: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:22.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:22 smithi154 conmon[34553]: cluster 2021-07-31T03:10:20.987011+0000 mgr.x (mgr.24542) 3559 : cluster [DBG] pgmap v4501: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:22.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:22 smithi045 conmon[39718]: cluster 2021-07-31T03:10:20.987011+0000 mgr.x ( 2021-07-31T03:10:22.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:22 smithi045 conmon[39718]: mgr.24542) 3559 : cluster [DBG] pgmap v4501: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:22.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:22 smithi045 conmon[33655]: cluster 2021-07-31T03:10:20.987011+0000 mgr.x (mgr.24542) 3559 : cluster [DBG] pgmap v4501: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:22.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:22 smithi045 conmon[33655]: 2021-07-31T03:10:24.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:24 smithi154 conmon[34553]: cluster 2021-07-31T03:10:22.987415+0000 mgr.x (mgr.24542) 3560 : cluster [DBG] pgmap v4502: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:24.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:24 smithi045 conmon[33655]: cluster 2021-07-31T03:10:22.987415+0000 mgr.x (mgr.24542) 2021-07-31T03:10:24.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:24 smithi045 conmon[33655]: 3560 : cluster [DBG] pgmap v4502: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:24.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:24 smithi045 conmon[39718]: cluster 2021-07-31T03:10:22. 2021-07-31T03:10:24.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:24 smithi045 conmon[39718]: 987415+0000 mgr.x (mgr.24542) 3560 : cluster [DBG] pgmap v4502: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:26.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:26 smithi154 conmon[34553]: cluster 2021-07-31T03:10:24.988014+0000 mgr.x (mgr.24542) 3561 : cluster [DBG] pgmap v4503: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:26.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:26 smithi045 conmon[39718]: cluster 2021-07-31T03:10:24.988014+0000 2021-07-31T03:10:26.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:26 smithi045 conmon[39718]: mgr.x (mgr.24542) 3561 : cluster [DBG] pgmap v4503: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:26.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:26 smithi045 conmon[33655]: cluster 2021-07-31T03:10:24.988014+0000 mgr.x (mgr.24542) 2021-07-31T03:10:26.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:26 smithi045 conmon[33655]: 3561 : cluster [DBG] pgmap v4503: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:27.034 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:10:27.311 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:10:27.312 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 85m ago 86m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:10:27.312 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 83m ago 85m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:10:27.312 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 83m ago 89m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:10:27.312 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 85m ago 91m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:10:27.313 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 85m ago 91m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:10:27.313 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 83m ago 89m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:10:27.313 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 85m ago 89m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:10:27.313 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 85m ago 86m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:10:27.314 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 83m ago 86m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:10:27.314 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 85m ago 89m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:10:27.314 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 85m ago 88m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:10:27.314 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 85m ago 88m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:10:27.315 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 85m ago 88m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:10:27.315 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 83m ago 87m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:10:27.315 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 83m ago 87m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:10:27.315 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 83m ago 87m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:10:27.315 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 83m ago 86m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:10:27.316 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 83m ago 86m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:10:27.656 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:10:27.656 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:10:27.656 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:10:27.656 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:10:27.657 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:10:27.657 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:10:27.657 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:10:27.657 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:10:27.657 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:10:27.658 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:10:27.658 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:10:27.658 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:10:27.658 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:10:27.658 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:10:27.659 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:10:27.659 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:10:27.659 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:10:28.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:28 smithi154 conmon[34553]: cluster 2021-07-31T03:10:26.988440+0000 mgr.x (mgr.24542) 3562 : cluster [DBG] pgmap v4504: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:28.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:28 smithi154 conmon[34553]: audit 2021-07-31T03:10:27.021989+0000 mgr.x (mgr.24542) 3563 : audit [DBG] from='client.28888 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:10:28.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:28 smithi154 conmon[34553]: audit 2021-07-31T03:10:27.304544+0000 mgr.x (mgr.24542) 3564 : audit [DBG] from='client.28894 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:10:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:28 smithi154 conmon[34553]: audit 2021-07-31T03:10:27. 2021-07-31T03:10:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:28 smithi154 conmon[34553]: 656451+0000 mon.c (mon.1) 181 : audit [DBG] from='client.? 172.21.15.45:0/1015046818' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:10:28.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:28 smithi045 conmon[33655]: cluster 2021-07-31T03:10:26.988440 2021-07-31T03:10:28.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:28 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 3562 : cluster [DBG] pgmap v4504: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:28.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:28 smithi045 conmon[33655]: audit 2021-07- 2021-07-31T03:10:28.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:28 smithi045 conmon[33655]: 31T03:10:27.021989+0000 mgr.x (mgr. 2021-07-31T03:10:28.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:28 smithi045 conmon[33655]: 24542) 3563 : audit [DBG] 2021-07-31T03:10:28.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:28 smithi045 conmon[33655]: from='client.28888 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:10:28.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:28 smithi045 conmon[33655]: audit 2021 2021-07-31T03:10:28.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:28 smithi045 conmon[33655]: -07-31T03:10:27. 2021-07-31T03:10:28.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:28 smithi045 conmon[33655]: 304544+0000 mgr.x (mgr.24542 2021-07-31T03:10:28.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:28 smithi045 conmon[33655]: ) 3564 : audit [DBG] from='client.28894 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:10:28.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:28 smithi045 conmon[33655]: audit 2021-07-31T03 2021-07-31T03:10:28.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:28 smithi045 conmon[33655]: :10:27.656451+0000 mon.c (mon.1) 181 : audit [DBG] 2021-07-31T03:10:28.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:28 smithi045 conmon[33655]: from='client.? 172.21.15.45:0/1015046818' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:10:28.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:28 smithi045 conmon[39718]: cluster 2021-07-31T03:10:26 2021-07-31T03:10:28.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:28 smithi045 conmon[39718]: .988440+0000 mgr.x (mgr.24542) 3562 : cluster [DBG] pgmap v4504: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:28.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:28 smithi045 conmon[39718]: audit 2021-07 2021-07-31T03:10:28.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:28 smithi045 conmon[39718]: -31T03:10:27. 2021-07-31T03:10:28.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:28 smithi045 conmon[39718]: 021989+0000 mgr.x (mgr. 2021-07-31T03:10:28.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:28 smithi045 conmon[39718]: 24542) 3563 : audit [DBG] 2021-07-31T03:10:28.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:28 smithi045 conmon[39718]: from='client.28888 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:10:28.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:28 smithi045 conmon[39718]: audit 2021- 2021-07-31T03:10:28.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:28 smithi045 conmon[39718]: 07-31T03:10:27. 2021-07-31T03:10:28.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:28 smithi045 conmon[39718]: 304544+0000 mgr.x (mgr.24542) 2021-07-31T03:10:28.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:28 smithi045 conmon[39718]: 3564 : audit [DBG] from='client.28894 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:10:28.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:28 smithi045 conmon[39718]: audit 2021 2021-07-31T03:10:28.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:28 smithi045 conmon[39718]: -07-31T03:10:27 2021-07-31T03:10:28.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:28 smithi045 conmon[39718]: .656451+0000 mon.c (mon.1) 2021-07-31T03:10:28.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:28 smithi045 conmon[39718]: 181 : audit [DBG] from='client.? 172.21.15.45:0/1015046818' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:10:29.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:29 smithi154 conmon[34553]: audit 2021-07-31T03:10:28.264529+0000 mon.b (mon.2) 490 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:10:29.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:29 smithi154 conmon[34553]: audit 2021-07-31T03:10: 2021-07-31T03:10:29.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:29 smithi154 conmon[34553]: 28.264791+0000 mon.b (mon.2) 491 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:10:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:29 smithi154 conmon[34553]: audit 2021-07-31T03:10:28.265213+0000 mon.a (mon.0) 3263 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:10:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:29 smithi154 conmon[34553]: audit 2021-07-31T03:10:28.265672 2021-07-31T03:10:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:29 smithi154 conmon[34553]: +0000 mon.a (mon.0) 3264 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:10:29.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:29 smithi045 conmon[33655]: audit 2021-07-31T03:10:28.264529+0000 mon.b (mon.2) 490 : audit [INF] 2021-07-31T03:10:29.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:29 smithi045 conmon[33655]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:10:29.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:29 smithi045 conmon[33655]: audit 2021-07-31T03:10:28.264791+0000 mon.b (mon.2) 491 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:10:29.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:29 smithi045 conmon[33655]: audit 2021- 2021-07-31T03:10:29.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:29 smithi045 conmon[33655]: 07-31T03:10:28.265213+0000 mon.a (mon.0) 3263 2021-07-31T03:10:29.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:29 smithi045 conmon[33655]: : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:10:29.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:29 smithi045 conmon[33655]: audit 2021-07-31T03:10:29.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:29 smithi045 conmon[33655]: 2021-07-31T03:10:28 2021-07-31T03:10:29.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:29 smithi045 conmon[33655]: .265672+0000 mon.a (mon.0) 3264 : audit 2021-07-31T03:10:29.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:29 smithi045 conmon[33655]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:10:29.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:29 smithi045 conmon[39718]: audit 2021-07-31T03:10:28.264529 2021-07-31T03:10:29.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:29 smithi045 conmon[39718]: +0000 mon.b (mon.2) 490 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:10:29.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:29 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T03:10:29.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:29 smithi045 conmon[39718]: 03:10:28.264791+0000 2021-07-31T03:10:29.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:29 smithi045 conmon[39718]: mon.b (mon.2) 491 : 2021-07-31T03:10:29.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:29 smithi045 conmon[39718]: audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:10:29.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:29 smithi045 conmon[39718]: audit 2021- 2021-07-31T03:10:29.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:29 smithi045 conmon[39718]: 07-31T03:10:28.265213 2021-07-31T03:10:29.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:29 smithi045 conmon[39718]: +0000 mon.a (mon.0) 3263 2021-07-31T03:10:29.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:29 smithi045 conmon[39718]: : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:10:29.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:29 smithi045 conmon[39718]: audit 2021- 2021-07-31T03:10:29.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:29 smithi045 conmon[39718]: 07-31T03:10:28. 2021-07-31T03:10:29.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:29 smithi045 conmon[39718]: 265672+0000 mon.a (mon.0 2021-07-31T03:10:29.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:29 smithi045 conmon[39718]: ) 3264 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:10:29.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:29 smithi045 conmon[39718]: 2021-07-31T03:10:30.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:30 smithi154 conmon[34553]: cluster 2021-07-31T03:10:28.988827+0000 mgr.x (mgr.24542) 3565 : cluster [DBG] pgmap v4505: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:30.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:30 smithi045 conmon[33655]: cluster 2021-07-31T03:10:28.988827+0000 mgr.x (mgr.24542) 2021-07-31T03:10:30.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:30 smithi045 conmon[33655]: 3565 : cluster [DBG] pgmap v4505: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:30.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:30 smithi045 conmon[39718]: cluster 2021-07-31T03:10:28. 2021-07-31T03:10:30.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:30 smithi045 conmon[39718]: 988827+0000 mgr.x (mgr.24542) 3565 : cluster [DBG] pgmap v4505: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:32.002 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: debug 2021-07-31T03:10:31.704+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:10:32.003 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:10:32.003 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:10:32.003 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: yield 2021-07-31T03:10:32.003 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:10:32.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:10:32.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:10:32.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:10:32.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:10:32.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:10:32.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:10:32.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:10:32.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:10:32.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: task.result() 2021-07-31T03:10:32.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:10:32.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: debug 2021-07-31T03:10:31.715+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:10:32.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:10:32.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:10:32.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: yield 2021-07-31T03:10:32.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:10:32.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:10:32.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:10:32.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:10:32.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:10:32.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:10:32.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:10:32.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:10:32.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:10:32.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: task.result() 2021-07-31T03:10:32.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:10:32.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: debug 2021-07-31T03:10:31.740+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:10:32.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:10:32.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:10:32.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: yield 2021-07-31T03:10:32.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:10:32.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:10:32.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:10:32.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:10:32.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:10:32.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:10:32.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:10:32.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:10:32.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:10:32.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: task.result() 2021-07-31T03:10:32.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:10:32.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: debug 2021-07-31T03:10:31.741+0000 7fcb9115c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:10:32.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:10:32.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:10:32.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: yield 2021-07-31T03:10:32.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:10:32.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:10:32.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:10:32.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:10:32.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:10:32.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:10:32.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:10:32.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:10:32.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:10:32.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: task.result() 2021-07-31T03:10:32.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:10:32.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: 2021-07-31T03:10:32.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:10:32.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: 2021-07-31T03:10:32.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:10:32.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:10:32.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:10:32.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:10:32.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:10:32.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:10:32.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:10:32.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:10:32.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:10:32.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:10:32.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:10:32.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:10:32.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:10:32.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:10:32.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:10:32.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:10:32.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:10:32.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:10:32.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: raise 2021-07-31T03:10:32.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:10:32.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:10:32.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:10:32.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:10:32.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:10:32.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: debug 2021-07-31T03:10:31.750+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:10:32.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:10:32.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:10:32.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: yield 2021-07-31T03:10:32.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:10:32.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:10:32.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:10:32.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:10:32.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:10:32.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:10:32.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:10:32.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:10:32.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:10:32.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: task.result() 2021-07-31T03:10:32.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:10:32.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: debug 2021-07-31T03:10:31.750+0000 7fcb9095b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:10:32.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:10:32.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:10:32.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: yield 2021-07-31T03:10:32.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:10:32.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:10:32.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:10:32.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:10:32.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:10:32.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:10:32.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:10:32.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:10:32.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:10:32.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: task.result() 2021-07-31T03:10:32.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:10:32.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: 2021-07-31T03:10:32.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:10:32.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: 2021-07-31T03:10:32.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:10:32.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:10:32.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:10:32.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:10:32.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:10:32.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:10:32.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:10:32.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:10:32.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:10:32.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:10:32.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:10:32.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:10:32.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:10:32.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:10:32.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:10:32.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:10:32.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:10:32.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:10:32.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: raise 2021-07-31T03:10:32.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:10:32.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:10:32.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:10:32.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:10:32.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:10:31 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:10:32.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: cluster 2021-07-31T03:10:30.989258+0000 mgr.x (mgr.24542) 3566 : cluster [DBG] pgmap v4506: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:32.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: audit 2021-07-31T03:10:31.658584+0000 mon.b (mon.2) 492 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:10:32.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: audit 2021- 2021-07-31T03:10:32.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: 07-31T03:10:31.660065+0000 mon.b (mon.2) 493 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:10:32.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: 2021-07-31T03:10:32.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: audit 2021-07-31T03:10:31 2021-07-31T03:10:32.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: .660952+0000 mon.b (mon.2) 494 : audit 2021-07-31T03:10:32.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:10:32.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: cephadm 2021-07-31 2021-07-31T03:10:32.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: T03:10:31.661686+0000 mgr.x (mgr. 2021-07-31T03:10:32.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: 24542) 3567 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:10:32.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: cephadm 2021 2021-07-31T03:10:32.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: -07-31T03:10:31.705127+0000 2021-07-31T03:10:32.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: mgr.x (mgr.24542) 3568 : cephadm [ERR] 2021-07-31T03:10:32.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: 2021-07-31T03:10:32.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:10:32.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:10:32.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: yield 2021-07-31T03:10:32.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:10:32.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:10:32.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:10:32.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:10:32.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:10:32.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:10:32.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:10:32.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:10:32.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:10:32.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: task.result() 2021-07-31T03:10:32.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:10:32.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: cephadm 2021-07-31T03:10:31.705555+0000 mgr.x (mgr.24542) 3569 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:10:32.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: cephadm 2021-07-31T03:10:31.716725+0000 mgr.x (mgr.24542) 3570 : cephadm [ERR] 2021-07-31T03:10:32.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: 2021-07-31T03:10:32.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:10:32.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:10:32.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: yield 2021-07-31T03:10:32.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:10:32.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:10:32.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:10:32.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:10:32.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:10:32.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:10:32.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:10:32.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:10:32.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:10:32.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: task.result() 2021-07-31T03:10:32.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:10:32.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: cephadm 2021-07-31T03:10:32.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: 2021-07-31T03:10:31.717772+0000 mgr.x (mgr.24542) 3571 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:10:32.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: cephadm 2021-07-31T03:10:31.741913+0000 mgr.x (mgr.24542) 3572 : cephadm [ERR] 2021-07-31T03:10:32.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:10:32.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:10:32.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: yield 2021-07-31T03:10:32.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:10:32.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:10:32.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:10:32.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:10:32.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:10:32.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:10:32.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:10:32.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:10:32.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:10:32.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: task.result() 2021-07-31T03:10:32.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:10:32.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: cephadm 2021-07-31T03:10:31.742357+0000 mgr.x (mgr.24542) 3573 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:10:32.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:10:32.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:10:32.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: yield 2021-07-31T03:10:32.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:10:32.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:10:32.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:10:32.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:10:32.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:10:32.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:10:32.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:10:32.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:10:32.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:10:32.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: task.result() 2021-07-31T03:10:32.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:10:32.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: 2021-07-31T03:10:32.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:10:32.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: 2021-07-31T03:10:32.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:10:32.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:10:32.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:10:32.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:10:32.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:10:32.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:10:32.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:10:32.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:10:32.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:10:32.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:10:32.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:10:32.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:10:32.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:10:32.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:10:32.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:10:32.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:10:32.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:10:32.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:10:32.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: raise 2021-07-31T03:10:32.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:10:32.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:10:32.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:10:32.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:10:32.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:10:32.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: cephadm 2021-07-31T03:10:31.750994+0000 mgr.x (mgr.24542) 2021-07-31T03:10:32.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: 3574 : cephadm [ERR] 2021-07-31T03:10:32.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:10:32.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:10:32.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: yield 2021-07-31T03:10:32.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:10:32.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:10:32.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:10:32.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:10:32.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:10:32.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:10:32.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:10:32.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:10:32.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:10:32.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: task.result() 2021-07-31T03:10:32.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:10:32.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: cephadm 2021-07-31T03:10:31.751552+0000 mgr.x (mgr.24542) 3575 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:10:32.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:10:32.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:10:32.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: yield 2021-07-31T03:10:32.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:10:32.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:10:32.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:10:32.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:10:32.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:10:32.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:10:32.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:10:32.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:10:32.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:10:32.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: task.result() 2021-07-31T03:10:32.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:10:32.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: 2021-07-31T03:10:32.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:10:32.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: 2021-07-31T03:10:32.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:10:32.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:10:32.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:10:32.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:10:32.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:10:32.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:10:32.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:10:32.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:10:32.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:10:32.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:10:32.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:10:32.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:10:32.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:10:32.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:10:32.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:10:32.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:10:32.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:10:32.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:10:32.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: raise 2021-07-31T03:10:32.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:10:32.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:10:32.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:10:32.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:10:32.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:32 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:10:32.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: cluster 2021-07-31T03:10:30.989258+0000 mgr.x (mgr.24542) 2021-07-31T03:10:32.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: 3566 : cluster [DBG] pgmap v4506: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:32.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: audit 2021-07-31T03:10:31.658584+0000 mon.b (mon.2) 492 : audit 2021-07-31T03:10:32.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:10:32.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T03:10:32.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: T03:10:31.660065+0000 mon.b (mon.2) 493 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:10:32.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: audit 2021-07-31T03:10:31.660952+0000 2021-07-31T03:10:32.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: mon.b (mon.2) 494 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:10:32.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: cephadm 2021-07-31T03:10:31.661686+0000 2021-07-31T03:10:32.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: mgr.x (mgr.24542) 3567 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:10:32.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: cephadm 2021-07-31T03:10:31.705127+0000 2021-07-31T03:10:32.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: mgr.x (mgr.24542) 3568 : cephadm [ERR] 2021-07-31T03:10:32.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:10:32.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:10:32.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: yield 2021-07-31T03:10:32.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:10:32.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:10:32.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:10:32.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:10:32.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:10:32.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:10:32.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:10:32.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:10:32.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:10:32.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: task.result() 2021-07-31T03:10:32.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:10:32.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: cephadm 2021-07-31T03:10:31.705555+0000 mgr.x (mgr.24542) 3569 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:10:32.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: cephadm 2021-07-31T03:10:31.716725+0000 mgr.x (mgr.24542) 3570 : cephadm [ERR] 2021-07-31T03:10:32.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:10:32.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:10:32.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: yield 2021-07-31T03:10:32.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:10:32.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:10:32.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:10:32.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:10:32.543 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: cluster 2021-07-31T 2021-07-31T03:10:32.544 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: 03:10:30.989258+0000 mgr.x (mgr.24542) 3566 : cluster [DBG] pgmap v4506: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:32.544 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: audit 2021-07-31T03: 2021-07-31T03:10:32.544 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: 10:31.658584+0000 mon.b (mon.2) 492 : audit 2021-07-31T03:10:32.544 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:10:32.545 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: audit 2021-07 2021-07-31T03:10:32.545 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: -31T03:10:31.660065+0000 mon.b (mon.2) 493 2021-07-31T03:10:32.545 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:10:32.545 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: audit 2021-07- 2021-07-31T03:10:32.545 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: 31T03:10:31.660952+0000 mon.b (mon 2021-07-31T03:10:32.546 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: .2) 494 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:10:32.546 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: 2021-07-31T03:10:32.546 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: cephadm 2021 2021-07-31T03:10:32.546 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: -07-31T03:10: 2021-07-31T03:10:32.546 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: 31.661686+0000 mgr.x (mgr. 2021-07-31T03:10:32.547 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: 24542) 3567 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:10:32.547 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: 2021-07-31T03:10:32.547 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: cephadm 2021-07-31T03:10:32.547 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: 2021-07-31T03:10:31.705127+0000 2021-07-31T03:10:32.547 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: mgr.x (mgr.24542) 3568 : cephadm [ERR] 2021-07-31T03:10:32.548 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: 2021-07-31T03:10:32.548 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:10:32.548 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:10:32.548 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: yield 2021-07-31T03:10:32.548 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:10:32.549 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:10:32.549 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:10:32.549 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:10:32.549 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:10:32.549 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:10:32.550 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:10:32.550 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:10:32.550 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:10:32.550 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: task.result() 2021-07-31T03:10:32.550 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:10:32.551 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: cephadm 2021-07-31T03:10:31.705555+0000 mgr.x (mgr.24542) 3569 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:10:32.551 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: cephadm 2021-07-31T03:10:31.716725+0000 2021-07-31T03:10:32.551 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: mgr.x (mgr.24542) 3570 : cephadm [ERR] 2021-07-31T03:10:32.551 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:10:32.551 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:10:32.552 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: yield 2021-07-31T03:10:32.552 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:10:32.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:10:32.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:10:32.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:10:32.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:10:32.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:10:32.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: task.result() 2021-07-31T03:10:32.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:10:32.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: 2021-07-31T03:10:32.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: cephadm 2021-07-31T03:10:31.717772+0000 mgr.x (mgr.24542) 3571 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:10:32.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: cephadm 2021-07-31T03:10:31.741913+0000 mgr.x (mgr.24542) 3572 2021-07-31T03:10:32.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: : cephadm [ERR] 2021-07-31T03:10:32.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:10:32.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:10:32.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: yield 2021-07-31T03:10:32.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:10:32.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:10:32.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:10:32.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:10:32.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:10:32.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:10:32.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:10:32.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:10:32.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:10:32.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: task.result() 2021-07-31T03:10:32.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:10:32.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: cephadm 2021-07-31T03:10:32.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: 2021-07-31T03:10:31.742357+0000 mgr.x (mgr.24542) 3573 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:10:32.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:10:32.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:10:32.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: yield 2021-07-31T03:10:32.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:10:32.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:10:32.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:10:32.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:10:32.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:10:32.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:10:32.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:10:32.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:10:32.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:10:32.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: task.result() 2021-07-31T03:10:32.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:10:32.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: 2021-07-31T03:10:32.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:10:32.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:10:32.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:10:32.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:10:32.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:10:32.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:10:32.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:10:32.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:10:32.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: task.result() 2021-07-31T03:10:32.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:10:32.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: cephadm 2021-07-31T03:10:31.717772+0000 mgr.x (mgr.24542) 3571 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:10:32.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: cephadm 2021-07-31T03:10:32.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: 2021-07-31T03:10:31.741913+0000 mgr.x 2021-07-31T03:10:32.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: (mgr.24542) 3572 : cephadm [ERR] 2021-07-31T03:10:32.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: 2021-07-31T03:10:32.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:10:32.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:10:32.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: yield 2021-07-31T03:10:32.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:10:32.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:10:32.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:10:32.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:10:32.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:10:32.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:10:32.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:10:32.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:10:32.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:10:32.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: task.result() 2021-07-31T03:10:32.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:10:32.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: cephadm 2021-07-31T03:10:31.742357+0000 mgr.x (mgr.24542) 3573 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:10:32.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:10:32.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:10:32.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: yield 2021-07-31T03:10:32.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:10:32.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:10:32.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:10:32.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:10:32.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:10:32.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:10:32.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:10:32.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:10:32.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:10:32.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: task.result() 2021-07-31T03:10:32.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:10:32.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: 2021-07-31T03:10:32.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:10:32.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: 2021-07-31T03:10:32.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:10:32.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:10:32.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:10:32.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:10:32.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:10:32.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:10:32.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:10:32.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:10:32.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:10:32.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:10:32.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:10:32.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:10:32.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:10:32.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:10:32.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:10:32.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:10:32.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:10:32.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:10:32.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: raise 2021-07-31T03:10:32.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:10:32.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:10:32.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:10:32.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:10:32.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:10:32.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: cephadm 2021-07-31T03:10:31.750994+0000 mgr.x (mgr.24542) 3574 : cephadm [ERR] 2021-07-31T03:10:32.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:10:32.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:10:32.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: yield 2021-07-31T03:10:32.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:10:32.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:10:32.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:10:32.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:10:32.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:10:32.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:10:32.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:10:32.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: 2021-07-31T03:10:32.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:10:32.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:10:32.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:10:32.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:10:32.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:10:32.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:10:32.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:10:32.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:10:32.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:10:32.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:10:32.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:10:32.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:10:32.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:10:32.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:10:32.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:10:32.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:10:32.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:10:32.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:10:32.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: raise 2021-07-31T03:10:32.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:10:32.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:10:32.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:10:32.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:10:32.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:10:32.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: cephadm 2021-07-31T03:10:31.750994+0000 mgr.x (mgr.24542 2021-07-31T03:10:32.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: ) 3574 : cephadm [ERR] 2021-07-31T03:10:32.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:10:32.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:10:32.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: yield 2021-07-31T03:10:32.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:10:32.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:10:32.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:10:32.585 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:10:32.585 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:10:32.585 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:10:32.585 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:10:32.585 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:10:32.586 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:10:32.586 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: task.result() 2021-07-31T03:10:32.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:10:32.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:10:32.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:10:32.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: task.result() 2021-07-31T03:10:32.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:10:32.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: cephadm 2021-07-31T03:10:31.751552+0000 mgr.x (mgr.24542) 3575 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:10:32.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:10:32.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:10:32.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: yield 2021-07-31T03:10:32.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:10:32.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:10:32.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:10:32.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:10:32.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:10:32.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:10:32.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:10:32.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:10:32.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:10:32.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: task.result() 2021-07-31T03:10:32.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:10:32.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: 2021-07-31T03:10:32.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:10:32.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: 2021-07-31T03:10:32.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:10:32.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:10:32.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:10:32.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:10:32.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:10:32.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:10:32.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:10:32.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:10:32.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:10:32.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:10:32.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:10:32.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:10:32.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:10:32.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:10:32.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:10:32.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:10:32.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:10:32.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:10:32.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: raise 2021-07-31T03:10:32.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:10:32.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: cephadm 2021-07-31T03:10:31.751552+0000 mgr.x (mgr.24542) 3575 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:10:32.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:10:32.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:10:32.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: yield 2021-07-31T03:10:32.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:10:32.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:10:32.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:10:32.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:10:32.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:10:32.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:10:32.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:10:32.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:10:32.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:10:32.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: task.result() 2021-07-31T03:10:32.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:10:32.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: 2021-07-31T03:10:32.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:10:32.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: 2021-07-31T03:10:32.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:10:32.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:10:32.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:10:32.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:10:32.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:10:32.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:10:32.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:10:32.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:10:32.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:10:32.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:10:32.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:10:32.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:10:32.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:10:32.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:10:32.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:10:32.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:10:32.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:10:32.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:10:32.603 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: raise 2021-07-31T03:10:32.603 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:10:32.603 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:10:32.603 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:10:32.604 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:10:32.604 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:10:32.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:10:32.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:10:32.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:10:32.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:10:32.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:32 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:10:34.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:34 smithi154 conmon[34553]: cluster 2021-07-31T03:10:32.989740+0000 mgr.x (mgr.24542) 3576 : cluster [DBG] pgmap v4507: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:34.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:34 smithi045 conmon[33655]: cluster 2021-07-31T03:10:32.989740+0000 mgr.x (mgr.24542) 2021-07-31T03:10:34.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:34 smithi045 conmon[33655]: 3576 : cluster [DBG] pgmap v4507: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:34.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:34 smithi045 conmon[39718]: cluster 2021-07-31T03:10:32.989740+0000 mgr.x (mgr.24542) 2021-07-31T03:10:34.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:34 smithi045 conmon[39718]: 3576 : cluster [DBG] pgmap v4507: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:36.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:36 smithi154 conmon[34553]: cluster 2021-07-31T03:10:34.990121+0000 mgr.x (mgr.24542) 3577 : cluster [DBG] pgmap v4508: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:36.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:36 smithi045 conmon[33655]: cluster 2021-07-31T03:10:34.990121+0000 mgr.x (mgr.24542) 3577 2021-07-31T03:10:36.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:36 smithi045 conmon[33655]: : cluster [DBG] pgmap v4508: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:36.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:36 smithi045 conmon[39718]: cluster 2021-07-31T03:10:34.990121+0000 mgr.x (mgr. 2021-07-31T03:10:36.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:36 smithi045 conmon[39718]: 24542) 3577 : cluster [DBG] pgmap v4508: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:38.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:38 smithi154 conmon[34553]: cluster 2021-07-31T03:10:36.990461+0000 mgr.x (mgr.24542) 3578 : cluster [DBG] pgmap v4509: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:38.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:38 smithi154 conmon[34553]: 2021-07-31T03:10:38.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:38 smithi045 conmon[39718]: cluster 2021-07-31T03:10:36.990461+0000 mgr.x (mgr 2021-07-31T03:10:38.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:38 smithi045 conmon[39718]: .24542) 3578 : cluster [DBG] pgmap v4509: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:38.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:38 smithi045 conmon[33655]: cluster 2021-07-31T03:10: 2021-07-31T03:10:38.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:38 smithi045 conmon[33655]: 36.990461+0000 mgr.x (mgr.24542) 3578 : cluster [DBG] pgmap v4509: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:40.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:40 smithi154 conmon[34553]: cluster 2021-07-31T03:10:38.990968+0000 mgr.x (mgr.24542) 3579 : cluster [DBG] pgmap v4510: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:40.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:40 smithi045 conmon[39718]: cluster 2021-07-31T03:10:38.990968 2021-07-31T03:10:40.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:40 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 3579 : cluster [DBG] pgmap v4510: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:40.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:40 smithi045 conmon[33655]: cluster 2021-07-31T03:10:38.990968+0000 mgr.x (mgr.24542) 3579 : cluster [DBG] 2021-07-31T03:10:40.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:40 smithi045 conmon[33655]: pgmap v4510: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:42.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:42 smithi154 conmon[34553]: cluster 2021-07-31T03:10:40.991489+0000 mgr.x (mgr.24542) 3580 : cluster [DBG] pgmap v4511: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:42.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:42 smithi045 conmon[39718]: cluster 2021-07-31T03:10:40.991489+0000 mgr.x (mgr 2021-07-31T03:10:42.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:42 smithi045 conmon[39718]: .24542) 3580 : cluster [DBG] pgmap v4511: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:42.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:42 smithi045 conmon[33655]: cluster 2021-07-31T03:10:40.991489+0000 mgr.x (mgr.24542) 3580 : cluster [DBG] pgmap v4511: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:44.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:44 smithi154 conmon[34553]: cluster 2021-07-31T03:10:42.991851+0000 mgr.x (mgr.24542) 3581 : cluster [DBG] 2021-07-31T03:10:44.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:44 smithi154 conmon[34553]: pgmap v4512: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:44.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:44 smithi045 conmon[39718]: cluster 2021-07-31T03:10:42.991851+0000 mgr.x (mgr.24542) 3581 : cluster [DBG] pgmap v4512: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:44.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:44 smithi045 conmon[33655]: cluster 2021-07-31T03:10:42.991851+0000 mgr.x (mgr.24542) 3581 2021-07-31T03:10:44.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:44 smithi045 conmon[33655]: : cluster [DBG] pgmap v4512: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:46.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:46 smithi154 conmon[34553]: cluster 2021-07-31T03:10:44.992491+0000 mgr.x (mgr.24542) 3582 : cluster 2021-07-31T03:10:46.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:46 smithi154 conmon[34553]: [DBG] pgmap v4513: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:46.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:46 smithi045 conmon[39718]: cluster 2021-07-31T03:10:44.992491+0000 mgr.x (mgr.24542) 3582 : cluster [DBG] pgmap v4513: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:46.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:46 smithi045 conmon[33655]: cluster 2021-07-31T03:10:44.992491+0000 mgr.x (mgr.24542) 2021-07-31T03:10:46.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:46 smithi045 conmon[33655]: 3582 : cluster [DBG] pgmap v4513: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:48.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:48 smithi154 conmon[34553]: cluster 2021-07-31T03:10:46.992956+0000 mgr.x (mgr.24542) 3583 : cluster [DBG] 2021-07-31T03:10:48.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:48 smithi154 conmon[34553]: pgmap v4514: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:48.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:48 smithi045 conmon[33655]: cluster 2021-07-31T03:10:46.992956+0000 mgr.x (mgr.24542) 2021-07-31T03:10:48.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:48 smithi045 conmon[33655]: 3583 : cluster [DBG] pgmap v4514: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:48.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:48 smithi045 conmon[39718]: cluster 2021-07-31T03:10:46.992956+0000 mgr.x (mgr.24542) 3583 : cluster [DBG] pgmap v4514: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:50.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:50 smithi154 conmon[34553]: cluster 2021-07-31T03:10:48.993412+0000 mgr.x (mgr.24542) 3584 : cluster [DBG] pgmap v4515: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:50.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:50 smithi154 conmon[34553]: 2021-07-31T03:10:50.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:50 smithi045 conmon[39718]: cluster 2021-07-31T03:10:48.993412+0000 mgr.x (mgr.24542) 3584 : cluster [DBG] pgmap v4515: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:50.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:50 smithi045 conmon[33655]: cluster 2021-07-31T03:10:48.993412+0000 mgr.x (mgr.24542) 3584 : cluster [DBG] pgmap v4515: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:50.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:50 smithi045 conmon[33655]: 2021-07-31T03:10:52.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:52 smithi154 conmon[34553]: cluster 2021-07-31T03:10:50.993933+0000 mgr.x (mgr.24542) 3585 : cluster [DBG] 2021-07-31T03:10:52.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:52 smithi154 conmon[34553]: pgmap v4516: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:52.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:52 smithi045 conmon[39718]: cluster 2021-07-31T03:10:50.993933+0000 mgr.x (mgr.24542) 3585 : cluster [DBG] pgmap v4516: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:52.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:52 smithi045 conmon[33655]: cluster 2021-07-31T03:10:50.993933+0000 mgr.x (mgr.24542) 2021-07-31T03:10:52.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:52 smithi045 conmon[33655]: 3585 : cluster [DBG] pgmap v4516: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:54.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:54 smithi154 conmon[34553]: cluster 2021-07-31T03:10:52.994358+0000 mgr.x (mgr.24542) 3586 : cluster [DBG] 2021-07-31T03:10:54.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:54 smithi154 conmon[34553]: pgmap v4517: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:54.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:54 smithi045 conmon[33655]: cluster 2021-07-31T03:10:52.994358+0000 mgr.x (mgr.24542) 3586 : cluster [DBG] pgmap v4517: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:54.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:54 smithi045 conmon[39718]: cluster 2021-07-31T03:10:52.994358+0000 mgr.x (mgr. 2021-07-31T03:10:54.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:54 smithi045 conmon[39718]: 24542) 3586 : cluster [DBG] pgmap v4517: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:56.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:56 smithi154 conmon[34553]: cluster 2021-07-31T03:10:54.994737+0000 mgr.x (mgr.24542) 3587 : cluster [DBG] pgmap v4518: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:56.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:56 smithi045 conmon[39718]: cluster 2021-07-31T03:10:54.994737+0000 mgr.x (mgr.24542) 2021-07-31T03:10:56.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:56 smithi045 conmon[39718]: 3587 : cluster [DBG] pgmap v4518: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:56.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:56 smithi045 conmon[33655]: cluster 2021-07-31T03:10:54.994737+0000 mgr.x (mgr.24542) 3587 : cluster 2021-07-31T03:10:56.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:56 smithi045 conmon[33655]: [DBG] pgmap v4518: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:57.970 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:10:58.250 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:10:58.250 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 85m ago 86m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:10:58.250 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 83m ago 86m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:10:58.250 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 83m ago 89m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:10:58.251 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 85m ago 91m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:10:58.251 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 85m ago 92m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:10:58.251 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 83m ago 90m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:10:58.251 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 85m ago 90m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:10:58.252 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 85m ago 86m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:10:58.252 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 83m ago 86m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:10:58.252 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 85m ago 89m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:10:58.252 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 85m ago 89m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:10:58.252 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 85m ago 88m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:10:58.253 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 85m ago 88m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:10:58.253 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 83m ago 88m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:10:58.253 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 83m ago 87m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:10:58.253 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 83m ago 87m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:10:58.253 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 83m ago 87m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:10:58.253 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 83m ago 86m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:10:58.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:58 smithi154 conmon[34553]: cluster 2021-07-31T03:10:56.995263+0000 mgr.x (mgr.24542) 3588 : cluster [DBG] 2021-07-31T03:10:58.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:58 smithi154 conmon[34553]: pgmap v4519: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:58.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:58 smithi045 conmon[39718]: cluster 2021-07-31T03:10:56.995263 2021-07-31T03:10:58.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:58 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 3588 : cluster [DBG] pgmap v4519: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:58.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:58 smithi045 conmon[33655]: cluster 2021-07-31T03:10:56.995263+0000 mgr.x (mgr.24542 2021-07-31T03:10:58.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:58 smithi045 conmon[33655]: ) 3588 : cluster [DBG] pgmap v4519: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:10:58.588 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:10:58.588 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:10:58.588 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:10:58.589 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:10:58.589 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:10:58.589 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:10:58.589 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:10:58.589 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:10:58.590 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:10:58.590 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:10:58.590 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:10:58.590 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:10:58.590 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:10:58.591 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:10:58.591 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:10:58.591 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:10:58.591 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:10:59.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:59 smithi154 conmon[34553]: audit 2021-07-31T03:10:57.958702+0000 mgr.x (mgr.24542) 3589 : audit [DBG] from='client.18966 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:10:59.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:59 smithi154 conmon[34553]: 2021-07-31T03:10:59.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:59 smithi154 conmon[34553]: audit 2021-07-31T03:10:58.242556+0000 mgr.x (mgr.24542) 3590 : audit [DBG] from='client.28912 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:10:59.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:10:59 smithi154 conmon[34553]: audit 2021-07-31T03:10:58.588846+0000 mon.c (mon.1) 182 : audit [DBG] from='client.? 172.21.15.45:0/3268688960' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:10:59.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:59 smithi045 conmon[33655]: audit 2021-07-31T03:10:57.958702+0000 mgr.x (mgr.24542) 3589 : audit [DBG] from='client.18966 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:10:59.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:59 smithi045 conmon[33655]: audit 2021-07-31T03:10:58.242556+0000 mgr.x (mgr.24542) 3590 : audit [DBG] from='client.28912 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:10:59.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:10:59 smithi045 conmon[33655]: audit 2021-07-31T03:10:58.588846+0000 mon.c (mon.1) 182 : audit [DBG] from='client.? 172.21.15.45:0/3268688960' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:10:59.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:59 smithi045 conmon[39718]: audit 2021-07-31T03:10:57.958702+0000 mgr.x (mgr 2021-07-31T03:10:59.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:59 smithi045 conmon[39718]: .24542) 3589 : audit [DBG] from='client.18966 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:10:59.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:59 smithi045 conmon[39718]: audit 2021-07-31T03:10:58. 2021-07-31T03:10:59.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:59 smithi045 conmon[39718]: 242556+0000 mgr.x (mgr.24542) 3590 : audit [DBG] from='client.28912 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:10:59.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:59 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T03:10:59.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:59 smithi045 conmon[39718]: 31T03:10:58.588846+0000 mon.c (mon.1 2021-07-31T03:10:59.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:10:59 smithi045 conmon[39718]: ) 182 : audit [DBG] from='client.? 172.21.15.45:0/3268688960' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:11:00.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:00 smithi154 conmon[34553]: cluster 2021-07-31T03:10:58.995720+0000 mgr.x (mgr.24542) 3591 : cluster [DBG] pgmap v4520: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:00.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:00 smithi154 conmon[34553]: 2021-07-31T03:11:00.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:00 smithi045 conmon[33655]: cluster 2021-07-31T03:10:58.995720+0000 mgr.x (mgr.24542) 3591 : cluster 2021-07-31T03:11:00.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:00 smithi045 conmon[33655]: [DBG] pgmap v4520: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:00.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:00 smithi045 conmon[39718]: cluster 2021-07-31T03:10:58.995720+0000 mgr.x (mgr. 2021-07-31T03:11:00.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:00 smithi045 conmon[39718]: 24542) 3591 : cluster [DBG] pgmap v4520: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:02.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:02 smithi154 conmon[34553]: cluster 2021-07-31T03:11:00.996072+0000 mgr.x (mgr.24542) 3592 : cluster [DBG] pgmap v4521: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:02.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:02 smithi045 conmon[33655]: cluster 2021-07-31T03:11:00.996072+0000 mgr.x (mgr.24542) 2021-07-31T03:11:02.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:02 smithi045 conmon[33655]: 3592 : cluster [DBG] pgmap v4521: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:02.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:02 smithi045 conmon[39718]: cluster 2021-07-31T03:11:00.996072+0000 mgr.x (mgr.24542) 2021-07-31T03:11:02.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:02 smithi045 conmon[39718]: 3592 : cluster [DBG] pgmap v4521: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:04.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:04 smithi154 conmon[34553]: cluster 2021-07-31T03:11:02.996526+0000 mgr.x (mgr.24542) 3593 : cluster [DBG] pgmap v4522: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:04.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:04 smithi045 conmon[33655]: cluster 2021-07-31T03:11:02.996526+0000 mgr.x (mgr.24542) 3593 2021-07-31T03:11:04.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:04 smithi045 conmon[33655]: : cluster [DBG] pgmap v4522: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:04.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:04 smithi045 conmon[39718]: cluster 2021-07-31T03:11:02.996526+0000 mgr.x (mgr.24542) 3593 : cluster [DBG] 2021-07-31T03:11:04.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:04 smithi045 conmon[39718]: pgmap v4522: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:06.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:06 smithi154 conmon[34553]: cluster 2021-07-31T03:11:04.997134+0000 mgr.x (mgr.24542) 3594 : cluster 2021-07-31T03:11:06.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:06 smithi154 conmon[34553]: [DBG] pgmap v4523: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:06.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:06 smithi045 conmon[33655]: cluster 2021-07-31T03 2021-07-31T03:11:06.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:06 smithi045 conmon[33655]: :11:04.997134+0000 mgr.x (mgr.24542) 3594 : cluster [DBG] pgmap v4523: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:06.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:06 smithi045 conmon[39718]: cluster 2021-07-31T03:11:04.997134+0000 mgr.x (mgr.24542) 3594 : cluster [DBG] 2021-07-31T03:11:06.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:06 smithi045 conmon[39718]: pgmap v4523: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:08.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:08 smithi154 conmon[34553]: cluster 2021-07-31T03:11:06.997528+0000 mgr.x (mgr.24542) 3595 : cluster [DBG] pgmap v4524: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:08.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:08 smithi045 conmon[39718]: cluster 2021-07-31T03:11:06.997528+0000 mgr.x 2021-07-31T03:11:08.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:08 smithi045 conmon[39718]: (mgr.24542) 3595 : cluster [DBG] pgmap v4524: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:08.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:08 smithi045 conmon[33655]: cluster 2021-07-31T03:11:06.997528+0000 mgr.x (mgr. 2021-07-31T03:11:08.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:08 smithi045 conmon[33655]: 24542) 3595 : cluster [DBG] pgmap v4524: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:10.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:10 smithi154 conmon[34553]: cluster 2021-07-31T03:11:08.998065+0000 mgr.x (mgr.24542) 3596 : cluster [DBG] pgmap v4525: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:10.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:10 smithi045 conmon[33655]: cluster 2021-07-31T03:11:08.998065+0000 mgr.x (mgr.24542) 3596 : cluster [DBG] pgmap v4525: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:10.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:10 smithi045 conmon[33655]: 2021-07-31T03:11:10.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:10 smithi045 conmon[39718]: cluster 2021-07-31T03:11:08.998065+0000 mgr.x ( 2021-07-31T03:11:10.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:10 smithi045 conmon[39718]: mgr.24542) 3596 : cluster [DBG] pgmap v4525: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:12.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:12 smithi154 conmon[34553]: cluster 2021-07-31T03:11:10.998579+0000 mgr.x (mgr.24542) 3597 : cluster [DBG] pgmap v4526: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:12.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:12 smithi045 conmon[33655]: cluster 2021-07-31T03:11:10.998579+0000 mgr.x (mgr.24542) 3597 : 2021-07-31T03:11:12.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:12 smithi045 conmon[33655]: cluster [DBG] pgmap v4526: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:12.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:12 smithi045 conmon[39718]: cluster 2021-07-31T03:11:10.998579+0000 mgr.x 2021-07-31T03:11:12.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:12 smithi045 conmon[39718]: (mgr.24542) 3597 : cluster [DBG] pgmap v4526: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:14.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:14 smithi154 conmon[34553]: cluster 2021-07-31T03:11:12.999019+0000 mgr.x (mgr.24542) 3598 : cluster [DBG] pgmap v4527: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:14.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:14 smithi154 conmon[34553]: 2021-07-31T03:11:14.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:14 smithi045 conmon[39718]: cluster 2021-07-31T03:11: 2021-07-31T03:11:14.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:14 smithi045 conmon[39718]: 12.999019+0000 mgr.x (mgr.24542) 3598 : cluster [DBG] pgmap v4527: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:14.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:14 smithi045 conmon[33655]: cluster 2021-07-31T03:11:12.999019+0000 mgr.x (mgr.24542) 3598 : cluster [DBG] 2021-07-31T03:11:14.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:14 smithi045 conmon[33655]: pgmap v4527: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:16.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:16 smithi154 conmon[34553]: cluster 2021-07-31T03:11:14.999411+0000 mgr.x (mgr.24542) 3599 : cluster [DBG] pgmap v4528: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:16.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:16 smithi154 conmon[34553]: 2021-07-31T03:11:16.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:16 smithi045 conmon[39718]: cluster 2021-07-31T03:11:14.999411+0000 mgr.x ( 2021-07-31T03:11:16.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:16 smithi045 conmon[39718]: mgr.24542) 3599 : cluster [DBG] pgmap v4528: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:16.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:16 smithi045 conmon[33655]: cluster 2021-07-31T03:11:14.999411+0000 mgr.x (mgr.24542) 3599 : cluster [DBG] pgmap v4528: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:18.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:18 smithi154 conmon[34553]: cluster 2021-07-31T03:11:16.999962+0000 mgr.x (mgr.24542) 3600 : cluster [DBG] pgmap v4529: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:18.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:18 smithi045 conmon[39718]: cluster 2021-07-31T03:11:16.999962+0000 mgr.x ( 2021-07-31T03:11:18.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:18 smithi045 conmon[39718]: mgr.24542) 3600 : cluster [DBG] pgmap v4529: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:18.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:18 smithi045 conmon[33655]: cluster 2021-07-31T03:11:16.999962+0000 mgr.x (mgr.24542) 3600 : cluster [DBG] 2021-07-31T03:11:18.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:18 smithi045 conmon[33655]: pgmap v4529: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:20.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:20 smithi045 conmon[33655]: cluster 2021-07-31T03:11:19.000524+0000 mgr.x (mgr.24542) 3601 : cluster [DBG] pgmap v4530: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:20.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:20 smithi045 conmon[39718]: cluster 2021-07-31T03:11:19.000524+0000 mgr.x (mgr.24542) 3601 : cluster [DBG] pgmap v4530: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:20.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:20 smithi154 conmon[34553]: cluster 2021-07-31T03:11:19.000524+0000 mgr.x (mgr.24542) 3601 : cluster [DBG] pgmap v4530: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:22.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:22 smithi045 conmon[33655]: cluster 2021-07-31T03:11:21.001041+0000 mgr.x (mgr. 2021-07-31T03:11:22.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:22 smithi045 conmon[33655]: 24542) 3602 : cluster [DBG] pgmap v4531: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:22.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:22 smithi045 conmon[39718]: cluster 2021-07-31T03:11:21.001041+0000 mgr.x (mgr.24542) 3602 : cluster [DBG] pgmap v4531: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:22.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:22 smithi154 conmon[34553]: cluster 2021-07-31T03:11:21.001041+0000 mgr.x (mgr.24542) 3602 : cluster [DBG] pgmap v4531: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:24.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:24 smithi045 conmon[33655]: cluster 2021-07-31T03:11:23.001452+0000 mgr.x (mgr.24542) 3603 : cluster 2021-07-31T03:11:24.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:24 smithi045 conmon[33655]: [DBG] pgmap v4532: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:24.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:24 smithi045 conmon[39718]: cluster 2021-07-31T03:11:23.001452+0000 mgr.x (mgr. 2021-07-31T03:11:24.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:24 smithi045 conmon[39718]: 24542) 3603 : cluster [DBG] pgmap v4532: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:24.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:24 smithi154 conmon[34553]: cluster 2021-07-31T03:11:23.001452+0000 mgr.x (mgr.24542) 3603 : cluster [DBG] pgmap v4532: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:26.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:26 smithi045 conmon[33655]: cluster 2021-07-31T03:11:25.002044+0000 mgr.x (mgr.24542) 3604 : cluster 2021-07-31T03:11:26.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:26 smithi045 conmon[33655]: [DBG] pgmap v4533: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:26.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:26 smithi045 conmon[39718]: cluster 2021-07-31T03:11:25.002044+0000 mgr.x ( 2021-07-31T03:11:26.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:26 smithi045 conmon[39718]: mgr.24542) 3604 : cluster [DBG] pgmap v4533: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:26.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:26 smithi154 conmon[34553]: cluster 2021-07-31T03:11:25.002044+0000 mgr.x (mgr.24542) 3604 : cluster [DBG] pgmap v4533: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:28.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:28 smithi045 conmon[39718]: cluster 2021-07-31T03:11:27.002437+0000 mgr.x (mgr 2021-07-31T03:11:28.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:28 smithi045 conmon[39718]: .24542) 3605 : cluster [DBG] pgmap v4534: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:28.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:28 smithi045 conmon[33655]: cluster 2021-07-31T03:11:27.002437+0000 mgr.x (mgr.24542) 3605 : 2021-07-31T03:11:28.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:28 smithi045 conmon[33655]: cluster [DBG] pgmap v4534: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:28.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:28 smithi154 conmon[34553]: cluster 2021-07-31T03:11:27.002437+0000 mgr.x (mgr.24542) 3605 : cluster [DBG] pgmap v4534: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:28.900 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:11:29.174 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:11:29.174 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 86m ago 87m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:11:29.175 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 84m ago 86m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:11:29.175 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 84m ago 90m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:11:29.175 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 86m ago 92m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:11:29.175 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 86m ago 92m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:11:29.175 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 84m ago 90m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:11:29.176 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 86m ago 90m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:11:29.176 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 86m ago 87m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:11:29.176 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 84m ago 87m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:11:29.176 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 86m ago 90m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:11:29.176 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 86m ago 89m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:11:29.177 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 86m ago 89m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:11:29.177 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 86m ago 89m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:11:29.177 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 84m ago 88m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:11:29.177 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 84m ago 88m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:11:29.177 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 84m ago 88m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:11:29.178 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 84m ago 87m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:11:29.178 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 84m ago 87m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:11:29.508 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:11:29.508 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:11:29.509 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:11:29.509 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:11:29.509 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:11:29.509 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:11:29.510 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:11:29.510 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:11:29.510 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:11:29.510 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:11:29.511 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:11:29.511 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:11:29.511 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:11:29.511 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:11:29.512 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:11:29.512 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:11:29.512 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:11:29.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:29 smithi045 conmon[33655]: audit 2021-07-31T03:11:28.269118+0000 2021-07-31T03:11:29.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:29 smithi045 conmon[33655]: mon.b (mon.2) 495 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:11:29.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:29 smithi045 conmon[33655]: audit 2021-07-31T03:11:28.269428+0000 mon.b (mon.2) 496 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:11:29.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:29 smithi045 conmon[33655]: 2021-07-31T03:11:29.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:29 smithi045 conmon[33655]: audit 2021-07-31T03:11:28. 2021-07-31T03:11:29.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:29 smithi045 conmon[33655]: 269669+0000 mon.a (mon.0) 3265 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:11:29.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:29 smithi045 conmon[33655]: audit 2021-07-31T03:11:28. 2021-07-31T03:11:29.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:29 smithi045 conmon[33655]: 270282+0000 mon.a (mon.0) 3266 : audit [INF] 2021-07-31T03:11:29.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:29 smithi045 conmon[33655]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:11:29.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:29 smithi045 conmon[39718]: audit 2021-07-31T03:11:28.269118+0000 mon.b (mon. 2021-07-31T03:11:29.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:29 smithi045 conmon[39718]: 2) 495 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:11:29.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:29 smithi045 conmon[39718]: audit 2021-07-31T03:11:28.269428+0000 mon.b (mon.2) 2021-07-31T03:11:29.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:29 smithi045 conmon[39718]: 496 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:11:29.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:29 smithi045 conmon[39718]: audit 2021-07-31T03:11:29.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:29 smithi045 conmon[39718]: 2021-07-31T03:11:28.269669+0000 2021-07-31T03:11:29.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:29 smithi045 conmon[39718]: mon.a (mon.0) 3265 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:11:29.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:29 smithi045 conmon[39718]: audit 2021-07-31T03:11:28.270282+0000 mon.a (mon.0) 3266 : audit 2021-07-31T03:11:29.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:29 smithi045 conmon[39718]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:11:29.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:29 smithi154 conmon[34553]: audit 2021-07-31T03:11:28.269118+0000 mon.b (mon.2) 495 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:11:29.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:29 smithi154 conmon[34553]: audit 2021-07-31T03:11:28.269428+0000 mon.b (mon.2) 496 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:11:29.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:29 smithi154 conmon[34553]: audit 2021-07-31T03:11:28.269669+0000 mon.a 2021-07-31T03:11:29.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:29 smithi154 conmon[34553]: (mon.0) 3265 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:11:29.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:29 smithi154 conmon[34553]: audit 2021-07-31 2021-07-31T03:11:29.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:29 smithi154 conmon[34553]: T03:11:28.270282+0000 2021-07-31T03:11:29.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:29 smithi154 conmon[34553]: mon.a (mon.0) 3266 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:11:29.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:29 smithi154 conmon[34553]: 2021-07-31T03:11:30.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:30 smithi045 conmon[33655]: audit 2021-07-31T03:11:28.888353+0000 mgr.x (mgr.24542 2021-07-31T03:11:30.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:30 smithi045 conmon[33655]: ) 3606 : audit [DBG] from='client.18984 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:11:30.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:30 smithi045 conmon[33655]: cluster 2021-07-31T03:11:29.002936+0000 mgr.x (mgr.24542) 3607 : cluster [DBG] pgmap v4535: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:30.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:30 smithi045 conmon[33655]: audit 2021-07-31T03:11:29.167186+0000 2021-07-31T03:11:30.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:30 smithi045 conmon[33655]: mgr.x (mgr.24542) 3608 : audit [DBG] from='client.28930 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:11:30.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:30 smithi045 conmon[33655]: audit 2021-07 2021-07-31T03:11:30.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:30 smithi045 conmon[33655]: -31T03:11:29.509117+0000 mon.c (mon 2021-07-31T03:11:30.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:30 smithi045 conmon[33655]: .1) 183 : audit [DBG] 2021-07-31T03:11:30.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:30 smithi045 conmon[33655]: from='client.? 172.21.15.45:0/2478482963' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:11:30.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:30 smithi045 conmon[39718]: audit 2021-07-31T03:11:28.888353+0000 mgr.x (mgr.24542) 3606 : audit [DBG] 2021-07-31T03:11:30.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:30 smithi045 conmon[39718]: from='client.18984 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:11:30.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:30 smithi045 conmon[39718]: cluster 2021-07-31T03:11:30.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:30 smithi045 conmon[39718]: 2021-07-31T03 2021-07-31T03:11:30.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:30 smithi045 conmon[39718]: :11:29.002936+0000 mgr.x 2021-07-31T03:11:30.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:30 smithi045 conmon[39718]: (mgr.24542) 3607 : cluster 2021-07-31T03:11:30.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:30 smithi045 conmon[39718]: [DBG] pgmap v4535: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:30.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:30 smithi045 conmon[39718]: audit 2021-07-31T03:11:30.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:30 smithi045 conmon[39718]: 2021-07-31T03:11:29 2021-07-31T03:11:30.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:30 smithi045 conmon[39718]: .167186+0000 mgr.x (mgr. 2021-07-31T03:11:30.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:30 smithi045 conmon[39718]: 24542) 3608 : audit [DBG] from='client.28930 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:11:30.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:30 smithi045 conmon[39718]: 2021-07-31T03:11:30.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:30 smithi045 conmon[39718]: audit 2021 2021-07-31T03:11:30.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:30 smithi045 conmon[39718]: -07-31T03:11:29. 2021-07-31T03:11:30.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:30 smithi045 conmon[39718]: 509117+0000 mon.c (mon.1 2021-07-31T03:11:30.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:30 smithi045 conmon[39718]: ) 183 : audit [DBG] from='client.? 172.21.15.45:0/2478482963' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:11:30.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:30 smithi045 conmon[39718]: 2021-07-31T03:11:30.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:30 smithi154 conmon[34553]: audit 2021-07-31T03:11:28.888353+0000 mgr.x (mgr.24542) 3606 : audit [DBG] from='client.18984 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:11:30.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:30 smithi154 conmon[34553]: cluster 2021-07-31T03:11:29.002936+0000 mgr.x (mgr.24542) 3607 : cluster [DBG] pgmap v4535: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:30.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:30 smithi154 conmon[34553]: audit 2021-07-31T03:11:29.167186+0000 2021-07-31T03:11:30.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:30 smithi154 conmon[34553]: mgr.x (mgr.24542) 3608 : audit [DBG] from='client.28930 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:11:30.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:30 smithi154 conmon[34553]: audit 2021-07-31T03:11:29.509117+0000 mon.c (mon.1 2021-07-31T03:11:30.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:30 smithi154 conmon[34553]: ) 183 : audit [DBG] from='client.? 172.21.15.45:0/2478482963' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:11:32.253 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: debug 2021-07-31T03:11:31.797+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:11:32.253 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:11:32.253 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:11:32.253 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: yield 2021-07-31T03:11:32.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:11:32.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:11:32.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:11:32.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:11:32.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:11:32.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:11:32.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:11:32.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:11:32.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:11:32.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: task.result() 2021-07-31T03:11:32.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:11:32.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: debug 2021-07-31T03:11:31.808+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:11:32.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:11:32.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:11:32.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: yield 2021-07-31T03:11:32.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:11:32.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:11:32.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:11:32.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:11:32.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:11:32.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:11:32.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:11:32.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:11:32.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:11:32.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: task.result() 2021-07-31T03:11:32.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:11:32.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: debug 2021-07-31T03:11:31.833+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:11:32.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:11:32.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:11:32.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: yield 2021-07-31T03:11:32.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:11:32.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:11:32.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:11:32.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:11:32.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:11:32.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:11:32.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:11:32.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:11:32.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:11:32.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: task.result() 2021-07-31T03:11:32.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:11:32.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: debug 2021-07-31T03:11:31.834+0000 7fcb97969700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:11:32.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:11:32.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:11:32.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: yield 2021-07-31T03:11:32.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:11:32.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:11:32.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:11:32.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:11:32.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:11:32.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:11:32.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:11:32.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:11:32.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:11:32.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: task.result() 2021-07-31T03:11:32.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:11:32.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: 2021-07-31T03:11:32.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:11:32.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: 2021-07-31T03:11:32.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:11:32.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:11:32.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:11:32.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:11:32.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:11:32.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:11:32.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:11:32.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:11:32.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:11:32.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:11:32.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:11:32.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:11:32.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:11:32.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:11:32.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:11:32.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:11:32.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:11:32.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:11:32.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: raise 2021-07-31T03:11:32.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:11:32.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:11:32.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:11:32.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:11:32.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:11:32.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: debug 2021-07-31T03:11:31.843+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:11:32.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:11:32.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:11:32.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: yield 2021-07-31T03:11:32.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:11:32.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:11:32.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:11:32.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:11:32.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:11:32.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:11:32.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:11:32.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:11:32.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:11:32.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: task.result() 2021-07-31T03:11:32.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:11:32.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: debug 2021-07-31T03:11:31.843+0000 7fcb9816a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:11:32.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:11:32.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:11:32.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: yield 2021-07-31T03:11:32.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:11:32.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:11:32.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:11:32.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:11:32.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:11:32.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:11:32.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:11:32.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:11:32.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:11:32.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: task.result() 2021-07-31T03:11:32.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:11:32.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: 2021-07-31T03:11:32.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:11:32.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: 2021-07-31T03:11:32.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:11:32.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:11:32.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:11:32.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:11:32.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:11:32.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:11:32.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:11:32.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:11:32.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:11:32.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:11:32.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:11:32.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:11:32.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:11:32.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:11:32.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:11:32.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:11:32.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:11:32.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:11:32.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: raise 2021-07-31T03:11:32.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:11:32.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:11:32.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:11:32.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:11:32.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:11:31 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:11:32.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:32 smithi154 conmon[34553]: cluster 2021-07-31T03:11:31.003267+0000 mgr.x (mgr.24542) 3609 : cluster [DBG] pgmap v4536: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:32.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:32 smithi154 conmon[34553]: audit 2021-07-31T03:11:31.752504+0000 mon.b (mon.2) 497 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:11:32.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:32 smithi154 conmon[34553]: audit 2021-07-31T03:11:31.753879 2021-07-31T03:11:32.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:32 smithi154 conmon[34553]: +0000 mon.b (mon.2) 498 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:11:32.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:32 smithi154 conmon[34553]: audit 2021-07-31T03:11 2021-07-31T03:11:32.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:32 smithi154 conmon[34553]: :31.754949+0000 mon.b (mon.2) 499 2021-07-31T03:11:32.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:32 smithi154 conmon[34553]: : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:11:32.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:32 smithi154 conmon[34553]: cephadm 2021- 2021-07-31T03:11:32.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:32 smithi154 conmon[34553]: 07-31T03:11:31.755672+0000 2021-07-31T03:11:32.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:32 smithi154 conmon[34553]: mgr.x (mgr.24542) 3610 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:11:32.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:32 smithi045 conmon[33655]: cluster 2021-07-31T03:11:31.003267+0000 mgr.x (mgr.24542) 2021-07-31T03:11:32.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:32 smithi045 conmon[33655]: 3609 : cluster [DBG] pgmap v4536: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:32.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:32 smithi045 conmon[33655]: audit 2021-07-31T03:11:31.752504+0000 mon.b (mon. 2021-07-31T03:11:32.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:32 smithi045 conmon[33655]: 2) 497 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:11:32.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:32 smithi045 conmon[33655]: audit 2021-07-31T 2021-07-31T03:11:32.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:32 smithi045 conmon[33655]: 03:11:31.753879+0000 2021-07-31T03:11:32.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:32 smithi045 conmon[33655]: mon.b (mon.2) 2021-07-31T03:11:32.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:32 smithi045 conmon[33655]: 498 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:11:32.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:32 smithi045 conmon[33655]: audit 2021-07-31T03:11:32.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:32 smithi045 conmon[33655]: 2021-07-31T03:11:31 2021-07-31T03:11:32.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:32 smithi045 conmon[33655]: .754949+0000 mon.b (mon.2 2021-07-31T03:11:32.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:32 smithi045 conmon[33655]: ) 499 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:11:32.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:32 smithi045 conmon[33655]: 2021-07-31T03:11:32.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:32 smithi045 conmon[33655]: cephadm 2021-07-31T03:11:32.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:32 smithi045 conmon[33655]: 2021-07-31T03:11: 2021-07-31T03:11:32.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:32 smithi045 conmon[33655]: 31.755672+0000 mgr.x ( 2021-07-31T03:11:32.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:32 smithi045 conmon[33655]: mgr.24542) 3610 : cephadm 2021-07-31T03:11:32.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:32 smithi045 conmon[33655]: [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:11:32.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:32 smithi045 conmon[39718]: cluster 2021-07-31T03:11:31.003267+0000 mgr.x (mgr.24542) 3609 : cluster [DBG] pgmap v4536: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:32.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:32 smithi045 conmon[39718]: audit 2021-07-31T03:11:31. 2021-07-31T03:11:32.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:32 smithi045 conmon[39718]: 752504+0000 mon.b (mon.2) 497 : 2021-07-31T03:11:32.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:32 smithi045 conmon[39718]: audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:11:32.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:32 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T03:11:32.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:32 smithi045 conmon[39718]: 31T03:11:31.753879+0000 mon.b 2021-07-31T03:11:32.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:32 smithi045 conmon[39718]: (mon.2) 498 : audit 2021-07-31T03:11:32.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:32 smithi045 conmon[39718]: [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:11:32.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:32 smithi045 conmon[39718]: audit 2021- 2021-07-31T03:11:32.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:32 smithi045 conmon[39718]: 07-31T03:11:31. 2021-07-31T03:11:32.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:32 smithi045 conmon[39718]: 754949+0000 mon.b (mon.2 2021-07-31T03:11:32.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:32 smithi045 conmon[39718]: ) 499 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:11:32.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:32 smithi045 conmon[39718]: 2021-07-31T03:11:32.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:32 smithi045 conmon[39718]: cephadm 2021-07-31T03:11:32.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:32 smithi045 conmon[39718]: 2021-07-31T03:11:31. 2021-07-31T03:11:32.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:32 smithi045 conmon[39718]: 755672+0000 mgr.x (mgr.24542) 2021-07-31T03:11:32.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:32 smithi045 conmon[39718]: 3610 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:11:33.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: cephadm 2021-07-31T03:11:31.798323+0000 mgr.x (mgr.24542) 3611 : cephadm [ERR] 2021-07-31T03:11:33.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:11:33.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:11:33.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: yield 2021-07-31T03:11:33.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:11:33.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:11:33.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:11:33.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:11:33.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:11:33.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:11:33.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:11:33.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:11:33.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:11:33.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: task.result() 2021-07-31T03:11:33.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:11:33.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: cephadm 2021-07-31T03:11:31.798740+0000 mgr.x (mgr.24542) 3612 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:11:33.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: cephadm 2021-07-31T03:11:31.809850+0000 mgr.x (mgr.24542) 3613 : cephadm [ERR] 2021-07-31T03:11:33.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:11:33.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:11:33.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: yield 2021-07-31T03:11:33.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:11:33.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:11:33.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:11:33.290 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:11:33.290 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:11:33.290 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:11:33.290 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:11:33.290 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:11:33.291 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:11:33.291 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: task.result() 2021-07-31T03:11:33.291 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:11:33.291 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: cephadm 2021-07-31T03:11:31.810618+0000 mgr.x (mgr.24542) 3614 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:11:33.291 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: cephadm 2021-07-31T03:11:31.834871+0000 mgr.x (mgr.24542) 3615 : cephadm [ERR] 2021-07-31T03:11:33.292 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:11:33.292 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:11:33.292 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: yield 2021-07-31T03:11:33.292 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:11:33.292 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:11:33.293 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:11:33.293 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:11:33.293 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:11:33.293 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:11:33.293 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:11:33.294 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:11:33.294 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:11:33.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:11:31.798323+0000 mgr.x (mgr.24542) 3611 : cephadm [ERR] 2021-07-31T03:11:33.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:11:33.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:11:33.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: yield 2021-07-31T03:11:33.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:11:33.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:11:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:11:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:11:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:11:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:11:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:11:33.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:11:33.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:11:33.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: task.result() 2021-07-31T03:11:33.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:11:33.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:11:31.798740+0000 mgr.x (mgr.24542) 3612 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:11:33.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:11:31.809850+0000 mgr.x (mgr.24542) 3613 : cephadm [ERR] 2021-07-31T03:11:33.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:11:33.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:11:33.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: yield 2021-07-31T03:11:33.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:11:33.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:11:33.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:11:33.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:11:33.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:11:33.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:11:33.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:11:33.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:11:33.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:11:33.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: task.result() 2021-07-31T03:11:33.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:11:33.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:11:31.810618+0000 mgr.x (mgr.24542) 3614 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:11:33.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:11:31.834871+0000 mgr.x (mgr.24542) 3615 : cephadm [ERR] 2021-07-31T03:11:33.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:11:33.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:11:33.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: yield 2021-07-31T03:11:33.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:11:33.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:11:33.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:11:33.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:11:33.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:11:33.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:11:33.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:11:33.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:11:33.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:11:33.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: task.result() 2021-07-31T03:11:33.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:11:33.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:11:31.835308+0000 mgr.x (mgr.24542) 3616 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:11:33.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:11:33.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:11:33.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: yield 2021-07-31T03:11:33.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:11:33.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:11:33.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:11:33.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:11:33.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:11:33.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:11:33.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:11:33.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:11:33.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:11:33.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: task.result() 2021-07-31T03:11:33.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:11:33.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: 2021-07-31T03:11:33.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:11:33.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: 2021-07-31T03:11:33.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:11:33.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:11:33.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:11:33.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:11:33.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:11:33.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:11:33.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:11:33.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:11:33.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:11:33.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:11:33.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:11:33.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:11:33.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:11:33.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:11:33.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:11:33.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:11:33.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:11:33.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:11:33.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: raise 2021-07-31T03:11:33.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:11:33.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:11:33.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:11:33.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:11:33.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:11:33.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:11:31.843997+0000 mgr.x (mgr.24542) 3617 : cephadm [ERR] 2021-07-31T03:11:33.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:11:33.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:11:33.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: yield 2021-07-31T03:11:33.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:11:33.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:11:33.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:11:33.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:11:33.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:11:33.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:11:33.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:11:33.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:11:33.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:11:33.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: task.result() 2021-07-31T03:11:33.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:11:33.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:11:31.844390+0000 mgr.x (mgr.24542) 3618 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:11:33.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:11:33.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:11:33.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: yield 2021-07-31T03:11:33.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:11:33.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:11:33.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:11:33.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:11:33.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:11:33.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:11:33.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:11:33.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:11:33.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:11:33.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: task.result() 2021-07-31T03:11:33.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:11:33.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: 2021-07-31T03:11:33.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:11:33.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: 2021-07-31T03:11:33.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:11:33.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:11:33.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:11:33.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:11:33.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:11:33.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:11:33.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:11:33.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:11:33.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:11:33.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:11:33.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:11:33.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:11:33.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:11:33.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:11:33.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:11:33.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:11:33.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:11:33.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:11:33.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: raise 2021-07-31T03:11:33.782 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:11:33.782 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:11:33.782 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:11:33.782 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:11:33.782 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:33 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:11:33.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: task.result() 2021-07-31T03:11:33.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:11:33.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: cephadm 2021-07-31T03:11:31.835308+0000 mgr.x (mgr.24542) 3616 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:11:33.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:11:33.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:11:33.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: yield 2021-07-31T03:11:33.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:11:33.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:11:33.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:11:33.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:11:33.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:11:33.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:11:33.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:11:33.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:11:33.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:11:33.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: task.result() 2021-07-31T03:11:33.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:11:33.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: 2021-07-31T03:11:33.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:11:33.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: 2021-07-31T03:11:33.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:11:33.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:11:33.788 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:11:33.788 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:11:33.788 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:11:33.788 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:11:33.788 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:11:33.789 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:11:33.789 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:11:33.789 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:11:33.789 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:11:33.789 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:11:33.790 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:11:33.790 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:11:33.790 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:11:33.790 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:11:33.790 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:11:33.791 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:11:33.791 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: raise 2021-07-31T03:11:33.791 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:11:33.791 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:11:33.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: cephadm 2021-07-31T03:11:31.798323+0000 mgr.x (mgr.24542) 3611 : cephadm [ERR] 2021-07-31T03:11:33.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: 2021-07-31T03:11:33.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:11:33.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:11:33.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: yield 2021-07-31T03:11:33.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:11:33.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:11:33.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:11:33.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:11:33.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:11:33.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:11:33.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:11:33.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:11:33.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:11:33.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: task.result() 2021-07-31T03:11:33.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:11:33.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: cephadm 2021-07-31T03:11:31.798740+0000 mgr.x (mgr.24542) 3612 : cephadm [INF] 2021-07-31T03:11:33.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:11:33.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: cephadm 2021-07-31T03:11:31.809850+0000 mgr.x (mgr.24542) 3613 : cephadm [ERR] 2021-07-31T03:11:33.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:11:33.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:11:33.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: yield 2021-07-31T03:11:33.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:11:33.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:11:33.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:11:33.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:11:33.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:11:33.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:11:33.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:11:33.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:11:33.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:11:33.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: task.result() 2021-07-31T03:11:33.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:11:33.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: cephadm 2021-07-31T03:11:31.810618+0000 mgr.x (mgr.24542) 3614 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:11:33.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: cephadm 2021-07-31T03:11 2021-07-31T03:11:33.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: :31.834871+0000 mgr.x (mgr.24542) 3615 : cephadm [ERR] 2021-07-31T03:11:33.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:11:33.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:11:33.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: yield 2021-07-31T03:11:33.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:11:33.800 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:11:33.800 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:11:33.801 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:11:33.801 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: cephadm 2021-07-31T03:11:31.843997+0000 mgr.x (mgr.24542) 3617 : cephadm [ERR] 2021-07-31T03:11:33.801 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:11:33.801 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:11:33.801 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: yield 2021-07-31T03:11:33.802 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:11:33.802 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:11:33.802 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:11:33.802 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:11:33.802 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:11:33.803 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:11:33.803 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:11:33.803 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:11:33.803 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:11:33.803 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: task.result() 2021-07-31T03:11:33.804 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:11:33.804 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: cephadm 2021-07-31T03:11:31.844390+0000 mgr.x (mgr.24542) 3618 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:11:33.804 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:11:33.804 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:11:33.804 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: yield 2021-07-31T03:11:33.805 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:11:33.805 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:11:33.805 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:11:33.805 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:11:33.805 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:11:33.806 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:11:33.806 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:11:33.806 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:11:33.806 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:11:33.806 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: task.result() 2021-07-31T03:11:33.807 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:11:33.807 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: 2021-07-31T03:11:33.807 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:11:33.807 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: 2021-07-31T03:11:33.807 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:11:33.808 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:11:33.808 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:11:33.808 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:11:33.808 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:11:33.808 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:11:33.809 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:11:33.809 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:11:33.809 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:11:33.809 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:11:33.809 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:11:33.810 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:11:33.810 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:11:33.810 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:11:33.810 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:11:33.810 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:11:33.811 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:11:33.811 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:11:33.811 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: raise 2021-07-31T03:11:33.811 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:11:33.812 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:11:33.812 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:11:33.812 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:11:33.812 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:11:33.813 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:11:33.813 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:11:33.813 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:11:33.813 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:11:33.814 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:11:33.814 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:11:33.814 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:11:33.814 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:11:33.814 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: task.result() 2021-07-31T03:11:33.815 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:11:33.815 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: cephadm 2021-07-31T03:11:31.835308+0000 mgr.x (mgr.24542) 3616 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:11:33.815 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:11:33.815 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:11:33.815 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: yield 2021-07-31T03:11:33.816 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:11:33.816 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:11:33.816 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:11:33.816 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:11:33.816 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:11:33.816 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:11:33.817 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:11:33.817 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:11:33.817 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:11:33.817 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: task.result() 2021-07-31T03:11:33.817 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:11:33.818 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: 2021-07-31T03:11:33.818 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:11:33.818 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: 2021-07-31T03:11:33.818 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:11:33.818 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:11:33.819 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:11:33.819 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:11:33.819 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:11:33.819 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:11:33.820 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:11:33.820 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:11:33.820 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:11:33.820 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:11:33.820 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:11:33.821 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:11:33.821 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:11:33.821 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:11:33.821 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:11:33.821 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:11:33.821 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:11:33.822 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:11:33.822 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: raise 2021-07-31T03:11:33.822 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:11:33.822 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:11:33.822 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:11:33.823 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:11:33.823 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:11:33.823 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: cephadm 2021-07-31T03:11:31.843997+0000 mgr.x (mgr.24542) 3617 : cephadm [ERR] 2021-07-31T03:11:33.823 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:11:33.823 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:11:33.824 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: yield 2021-07-31T03:11:33.824 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:11:33.824 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:11:33.824 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:11:33.824 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:11:33.825 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:11:33.825 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:11:33.825 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:11:33.825 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:11:33.825 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:11:33.826 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: task.result() 2021-07-31T03:11:33.826 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:11:33.826 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: cephadm 2021-07-31T03:11:31.844390+0000 mgr.x (mgr.24542) 3618 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:11:33.826 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:11:33.826 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:11:33.827 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: yield 2021-07-31T03:11:33.827 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:11:33.827 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:11:33.827 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:11:33.827 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:11:33.828 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:11:33.828 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:11:33.828 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:11:33.828 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:11:33.828 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:11:33.829 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: task.result() 2021-07-31T03:11:33.829 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:11:33.829 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: 2021-07-31T03:11:33.829 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:11:33.829 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: 2021-07-31T03:11:33.830 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:11:33.830 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:11:33.830 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:11:33.830 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:11:33.830 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:11:33.831 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:11:33.831 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:11:33.831 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:11:33.831 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:11:33.831 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:11:33.832 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:11:33.832 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:11:33.832 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:11:33.832 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:11:33.832 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:11:33.833 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:11:33.833 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:11:33.833 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:11:33.833 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: raise 2021-07-31T03:11:33.833 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:11:33.833 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:11:33.834 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:11:33.834 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:11:33.834 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:33 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:11:34.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:34 smithi154 conmon[34553]: cluster 2021-07-31T03:11:33.003710+0000 mgr.x (mgr.24542) 3619 : cluster [DBG] pgmap v4537: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:34.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:34 smithi045 conmon[39718]: cluster 2021-07-31T03:11:33.003710+0000 2021-07-31T03:11:34.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:34 smithi045 conmon[39718]: mgr.x (mgr.24542) 3619 : cluster [DBG] pgmap v4537: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:34.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:34 smithi045 conmon[33655]: cluster 2021-07-31T03:11:33.003710+0000 mgr.x (mgr.24542) 3619 2021-07-31T03:11:34.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:34 smithi045 conmon[33655]: : cluster [DBG] pgmap v4537: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:36.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:36 smithi154 conmon[34553]: cluster 2021-07-31T03:11:35.004227+0000 mgr.x (mgr.24542) 3620 : cluster [DBG] pgmap v4538: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:36.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:36 smithi045 conmon[39718]: cluster 2021-07-31T03:11:35.004227+0000 mgr.x ( 2021-07-31T03:11:36.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:36 smithi045 conmon[39718]: mgr.24542) 3620 : cluster [DBG] pgmap v4538: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:36.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:36 smithi045 conmon[33655]: cluster 2021-07-31T03:11:35.004227+0000 mgr.x (mgr.24542) 2021-07-31T03:11:36.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:36 smithi045 conmon[33655]: 3620 : cluster [DBG] pgmap v4538: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:38.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:38 smithi154 conmon[34553]: cluster 2021-07-31T03:11:37.004668+0000 mgr.x (mgr.24542) 3621 : cluster [DBG] pgmap v4539: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:38.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:38 smithi045 conmon[33655]: cluster 2021-07-31T03:11:37.004668+0000 mgr.x (mgr.24542) 3621 : 2021-07-31T03:11:38.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:38 smithi045 conmon[33655]: cluster [DBG] pgmap v4539: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:38.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:38 smithi045 conmon[39718]: cluster 2021-07-31T03:11:37.004668+0000 mgr.x (mgr 2021-07-31T03:11:38.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:38 smithi045 conmon[39718]: .24542) 3621 : cluster [DBG] pgmap v4539: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:40.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:40 smithi154 conmon[34553]: cluster 2021-07-31T03:11:39.005243+0000 mgr.x (mgr.24542) 3622 : cluster [DBG] pgmap v4540: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:40.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:40 smithi045 conmon[39718]: cluster 2021-07-31T03:11:39 2021-07-31T03:11:40.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:40 smithi045 conmon[39718]: .005243+0000 mgr.x (mgr.24542) 3622 : cluster [DBG] pgmap v4540: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:40.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:40 smithi045 conmon[33655]: cluster 2021-07-31T03:11: 2021-07-31T03:11:40.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:40 smithi045 conmon[33655]: 39.005243+0000 mgr.x (mgr.24542) 3622 : cluster [DBG] pgmap v4540: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:42.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:42 smithi154 conmon[34553]: cluster 2021-07-31T03:11:41.005622+0000 mgr.x (mgr.24542) 3623 : cluster [DBG] pgmap v4541: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:42.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:42 smithi045 conmon[33655]: cluster 2021-07-31T03:11:41.005622+0000 mgr.x (mgr.24542) 2021-07-31T03:11:42.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:42 smithi045 conmon[33655]: 3623 : cluster [DBG] pgmap v4541: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:42.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:42 smithi045 conmon[39718]: cluster 2021-07-31T03:11:41.005622+0000 mgr.x (mgr 2021-07-31T03:11:42.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:42 smithi045 conmon[39718]: .24542) 3623 : cluster [DBG] pgmap v4541: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:44.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:44 smithi154 conmon[34553]: cluster 2021-07-31T03:11:43.006219+0000 mgr.x (mgr.24542) 3624 : cluster [DBG] pgmap v4542: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:44.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:44 smithi045 conmon[33655]: cluster 2021-07-31T03:11:43.006219 2021-07-31T03:11:44.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:44 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 3624 : cluster [DBG] pgmap v4542: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:44.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:44 smithi045 conmon[39718]: cluster 2021-07-31T03:11:43.006219+0000 mgr.x (mgr 2021-07-31T03:11:44.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:44 smithi045 conmon[39718]: .24542) 3624 : cluster [DBG] pgmap v4542: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:46.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:46 smithi154 conmon[34553]: cluster 2021-07-31T03:11:45.006621+0000 mgr.x (mgr.24542) 3625 : cluster [DBG] pgmap v4543: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:46.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:46 smithi154 conmon[34553]: 2021-07-31T03:11:46.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:46 smithi045 conmon[33655]: cluster 2021-07-31T03:11:45.006621+0000 mgr.x (mgr.24542) 3625 2021-07-31T03:11:46.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:46 smithi045 conmon[33655]: : cluster [DBG] pgmap v4543: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:46.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:46 smithi045 conmon[39718]: cluster 2021-07-31T03:11:45.006621+0000 mgr.x ( 2021-07-31T03:11:46.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:46 smithi045 conmon[39718]: mgr.24542) 3625 : cluster [DBG] pgmap v4543: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:48.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:48 smithi154 conmon[34553]: cluster 2021-07-31T03:11:47.007080+0000 mgr.x (mgr.24542) 3626 : cluster [DBG] pgmap v4544: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:48.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:48 smithi045 conmon[39718]: cluster 2021-07-31T03:11:47.007080+0000 mgr.x ( 2021-07-31T03:11:48.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:48 smithi045 conmon[39718]: mgr.24542) 3626 : cluster [DBG] pgmap v4544: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:48.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:48 smithi045 conmon[33655]: cluster 2021-07-31T03:11:47.007080 2021-07-31T03:11:48.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:48 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 3626 : cluster [DBG] pgmap v4544: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:50.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:50 smithi154 conmon[34553]: cluster 2021-07-31T03:11:49.007661+0000 mgr.x (mgr.24542) 3627 : cluster [DBG] pgmap v4545: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:50.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:50 smithi045 conmon[39718]: cluster 2021-07-31T03:11:49.007661+0000 mgr.x (mgr. 2021-07-31T03:11:50.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:50 smithi045 conmon[39718]: 24542) 3627 : cluster [DBG] pgmap v4545: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:50.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:50 smithi045 conmon[33655]: cluster 2021-07-31T03:11:49.007661+0000 mgr.x (mgr.24542) 3627 : cluster [DBG] pgmap v4545: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:52.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:52 smithi154 conmon[34553]: cluster 2021-07-31T03:11:51.008100+0000 mgr.x (mgr.24542) 3628 : cluster [DBG] pgmap v4546: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:52.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:52 smithi045 conmon[39718]: cluster 2021-07-31T03:11:51. 2021-07-31T03:11:52.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:52 smithi045 conmon[39718]: 008100+0000 mgr.x (mgr.24542) 3628 : cluster [DBG] pgmap v4546: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:52.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:52 smithi045 conmon[33655]: cluster 2021-07-31T03:11:51.008100+0000 mgr.x (mgr.24542) 3628 : cluster [DBG] pgmap v4546: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:52.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:52 smithi045 conmon[33655]: 2021-07-31T03:11:54.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:54 smithi154 conmon[34553]: cluster 2021-07-31T03:11:53.008442+0000 mgr.x (mgr.24542) 3629 : cluster [DBG] 2021-07-31T03:11:54.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:54 smithi154 conmon[34553]: pgmap v4547: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:54.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:54 smithi045 conmon[39718]: cluster 2021-07-31T03:11:53.008442+0000 mgr.x (mgr.24542) 3629 : cluster [DBG] pgmap v4547: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:54.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:54 smithi045 conmon[33655]: cluster 2021-07-31T03:11:53.008442+0000 mgr.x (mgr.24542) 3629 : cluster [DBG] pgmap v4547: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:56.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:56 smithi154 conmon[34553]: cluster 2021-07-31T03:11:55.009037+0000 mgr.x (mgr.24542) 3630 : cluster [DBG] 2021-07-31T03:11:56.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:56 smithi154 conmon[34553]: pgmap v4548: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:56.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:56 smithi045 conmon[39718]: cluster 2021-07-31T03:11:55.009037+0000 mgr.x (mgr. 2021-07-31T03:11:56.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:56 smithi045 conmon[39718]: 24542) 3630 : cluster [DBG] pgmap v4548: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:56.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:56 smithi045 conmon[33655]: cluster 2021-07-31T03:11:55.009037+0000 mgr.x (mgr.24542) 2021-07-31T03:11:56.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:56 smithi045 conmon[33655]: 3630 : cluster [DBG] pgmap v4548: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:58.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:58 smithi154 conmon[34553]: cluster 2021-07-31T03:11:57.009399+0000 mgr.x (mgr.24542) 3631 : cluster [DBG] 2021-07-31T03:11:58.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:11:58 smithi154 conmon[34553]: pgmap v4549: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:58.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:58 smithi045 conmon[39718]: cluster 2021-07-31T03:11:57.009399+0000 mgr.x ( 2021-07-31T03:11:58.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:11:58 smithi045 conmon[39718]: mgr.24542) 3631 : cluster [DBG] pgmap v4549: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:58.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:58 smithi045 conmon[33655]: cluster 2021-07-31T03:11:57.009399+0000 2021-07-31T03:11:58.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:11:58 smithi045 conmon[33655]: mgr.x (mgr.24542) 3631 : cluster [DBG] pgmap v4549: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:11:59.820 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:12:00.095 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:12:00.095 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 86m ago 87m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:12:00.096 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 84m ago 87m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:12:00.096 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 84m ago 90m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:12:00.096 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 86m ago 93m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:12:00.096 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 86m ago 93m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:12:00.096 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 84m ago 91m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:12:00.097 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 86m ago 91m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:12:00.097 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 86m ago 87m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:12:00.097 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 84m ago 87m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:12:00.097 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 86m ago 90m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:12:00.098 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 86m ago 90m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:12:00.098 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 86m ago 89m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:12:00.098 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 86m ago 89m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:12:00.098 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 84m ago 89m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:12:00.098 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 84m ago 88m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:12:00.099 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 84m ago 88m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:12:00.099 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 84m ago 88m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:12:00.099 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 84m ago 87m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:12:00.434 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:12:00.435 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:12:00.435 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:12:00.435 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:12:00.435 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:12:00.436 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:12:00.436 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:12:00.436 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:12:00.436 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:12:00.436 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:12:00.437 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:12:00.437 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:12:00.437 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:12:00.437 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:12:00.438 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:12:00.438 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:12:00.438 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:12:00.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:00 smithi154 conmon[34553]: cluster 2021-07-31T03:11:59.009900+0000 mgr.x (mgr.24542) 3632 : 2021-07-31T03:12:00.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:00 smithi154 conmon[34553]: cluster [DBG] pgmap v4550: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:12:00.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:00 smithi045 conmon[33655]: cluster 2021-07-31T03:11:59.009900+0000 mgr.x 2021-07-31T03:12:00.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:00 smithi045 conmon[33655]: (mgr.24542) 3632 : cluster [DBG] pgmap v4550: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:12:00.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:00 smithi045 conmon[39718]: cluster 2021-07-31T03:11:59.009900 2021-07-31T03:12:00.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:00 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 3632 : cluster [DBG] pgmap v4550: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:12:01.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:01 smithi154 conmon[34553]: audit 2021-07-31T03:11:59.808594+0000 mgr.x (mgr.24542) 3633 : audit [DBG] from='client.19002 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:12:01.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:01 smithi154 conmon[34553]: audit 2021-07-31T03:12:00.087804+0000 mgr.x (mgr.24542) 3634 : audit [DBG] from='client.28948 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:12:01.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:01 smithi154 conmon[34553]: audit 2021-07-31T03:12:00.435238+0000 mon.c (mon.1) 184 : audit [DBG] from='client.? 172.21.15.45:0/1723143046' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:12:01.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:01 smithi045 conmon[33655]: audit 2021-07-31T03:11:59.808594+0000 mgr.x (mgr.24542) 3633 : 2021-07-31T03:12:01.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:01 smithi045 conmon[33655]: audit [DBG] from='client.19002 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:12:01.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:01 smithi045 conmon[33655]: audit 2021-07-31T03:12:00.087804+0000 mgr.x (mgr.24542) 3634 : audit [DBG] from='client.28948 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:12:01.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:01 smithi045 conmon[33655]: audit 2021 2021-07-31T03:12:01.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:01 smithi045 conmon[33655]: -07-31T03:12:00.435238+0000 mon.c (mon.1) 184 : audit [DBG] from='client.? 172.21.15.45:0/1723143046' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:12:01.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:01 smithi045 conmon[39718]: audit 2021-07-31T03:11:59.808594+0000 mgr.x (mgr.24542) 3633 : audit 2021-07-31T03:12:01.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:01 smithi045 conmon[39718]: [DBG] from='client.19002 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:12:01.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:01 smithi045 conmon[39718]: audit 2021-07-31T03:12:00.087804+0000 mgr.x (mgr.24542) 3634 : audit [DBG] from='client.28948 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:12:01.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:01 smithi045 conmon[39718]: audit 2021-07-31T03:12:01.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:01 smithi045 conmon[39718]: 2021-07-31T03:12:00.435238+0000 mon.c (mon.1) 184 : audit [DBG] from='client.? 172.21.15.45:0/1723143046' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:12:02.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:02 smithi154 conmon[34553]: cluster 2021-07-31T03:12:01.010424+0000 mgr.x (mgr.24542) 3635 : cluster [DBG] pgmap v4551: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:12:02.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:02 smithi045 conmon[39718]: cluster 2021-07-31T03:12:01.010424+0000 mgr.x (mgr 2021-07-31T03:12:02.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:02 smithi045 conmon[39718]: .24542) 3635 : cluster [DBG] pgmap v4551: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:12:02.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:02 smithi045 conmon[33655]: cluster 2021-07-31T03:12:01.010424+0000 mgr.x (mgr.24542) 2021-07-31T03:12:02.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:02 smithi045 conmon[33655]: 3635 : cluster [DBG] pgmap v4551: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:12:04.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:04 smithi154 conmon[34553]: cluster 2021-07-31T03:12:03.010943+0000 mgr.x (mgr.24542) 3636 : cluster [DBG] pgmap v4552: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:12:04.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:04 smithi045 conmon[39718]: cluster 2021-07-31T03:12:03.010943+0000 mgr.x (mgr.24542 2021-07-31T03:12:04.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:04 smithi045 conmon[39718]: ) 3636 : cluster [DBG] pgmap v4552: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:12:04.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:04 smithi045 conmon[33655]: cluster 2021-07-31T03:12:03.010943+0000 mgr.x (mgr.24542) 3636 : cluster [DBG] pgmap v4552: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:12:06.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:06 smithi154 conmon[34553]: cluster 2021-07-31T03:12:05.011358+0000 mgr.x (mgr.24542) 3637 : cluster [DBG] pgmap v4553: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:12:06.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:06 smithi045 conmon[39718]: cluster 2021-07-31T03:12:05.011358+0000 mgr.x (mgr.24542) 3637 : cluster [DBG] pgmap v4553: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:12:06.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:06 smithi045 conmon[33655]: cluster 2021-07-31T03:12:05.011358+0000 mgr.x (mgr.24542) 3637 : cluster 2021-07-31T03:12:06.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:06 smithi045 conmon[33655]: [DBG] pgmap v4553: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:12:08.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:08 smithi154 conmon[34553]: cluster 2021-07-31T03:12:07.011835+0000 mgr.x (mgr.24542) 3638 : cluster [DBG] pgmap v4554: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:12:08.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:08 smithi045 conmon[33655]: cluster 2021-07-31T03:12:07.011835+0000 mgr.x (mgr.24542) 3638 : cluster [DBG] pgmap v4554: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:12:08.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:08 smithi045 conmon[39718]: cluster 2021-07-31T03:12:07.011835+0000 mgr.x (mgr.24542) 2021-07-31T03:12:08.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:08 smithi045 conmon[39718]: 3638 : cluster [DBG] pgmap v4554: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:12:10.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:10 smithi154 conmon[34553]: cluster 2021-07-31T03:12:09.012339+0000 mgr.x (mgr.24542) 3639 : cluster [DBG] pgmap v4555: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:12:10.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:10 smithi154 conmon[34553]: 2021-07-31T03:12:10.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:10 smithi045 conmon[33655]: cluster 2021-07-31T03:12:09.012339+0000 2021-07-31T03:12:10.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:10 smithi045 conmon[33655]: mgr.x (mgr.24542) 3639 : cluster [DBG] pgmap v4555: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:12:10.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:10 smithi045 conmon[39718]: cluster 2021-07-31T03:12: 2021-07-31T03:12:10.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:10 smithi045 conmon[39718]: 09.012339+0000 mgr.x (mgr.24542) 3639 : cluster [DBG] pgmap v4555: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:12:12.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:12 smithi154 conmon[34553]: cluster 2021-07-31T03:12:11.012724+0000 mgr.x (mgr.24542) 3640 : cluster [DBG] 2021-07-31T03:12:12.871 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:12 smithi154 conmon[34553]: pgmap v4556: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:12:12.872 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:12 smithi045 conmon[39718]: cluster 2021-07-31T03:12:11.012724+0000 mgr.x ( 2021-07-31T03:12:12.872 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:12 smithi045 conmon[39718]: mgr.24542) 3640 : cluster [DBG] pgmap v4556: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:12:12.872 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:12 smithi045 conmon[33655]: cluster 2021-07-31T03:12:11.012724+0000 mgr.x 2021-07-31T03:12:12.873 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:12 smithi045 conmon[33655]: (mgr.24542) 3640 : cluster [DBG] pgmap v4556: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:12:14.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:14 smithi154 conmon[34553]: cluster 2021-07-31T03:12:13.013276+0000 mgr.x (mgr.24542) 3641 : cluster [DBG] pgmap v4557: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:12:14.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:14 smithi045 conmon[39718]: cluster 2021-07-31T03:12:13.013276+0000 mgr.x 2021-07-31T03:12:14.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:14 smithi045 conmon[39718]: (mgr.24542) 3641 : cluster [DBG] pgmap v4557: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:12:14.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:14 smithi045 conmon[33655]: cluster 2021-07-31T03:12:13.013276+0000 mgr.x 2021-07-31T03:12:14.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:14 smithi045 conmon[33655]: (mgr.24542) 3641 : cluster [DBG] pgmap v4557: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:12:16.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:16 smithi154 conmon[34553]: cluster 2021-07-31T03:12:15.013789+0000 mgr.x (mgr.24542) 3642 : cluster [DBG] pgmap v4558: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:12:16.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:16 smithi045 conmon[39718]: cluster 2021-07-31T03:12:15.013789+0000 mgr.x (mgr 2021-07-31T03:12:16.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:16 smithi045 conmon[39718]: .24542) 3642 : cluster [DBG] pgmap v4558: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:12:16.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:16 smithi045 conmon[33655]: cluster 2021-07-31T03:12:15.013789+0000 mgr.x (mgr.24542) 3642 : cluster 2021-07-31T03:12:16.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:16 smithi045 conmon[33655]: [DBG] pgmap v4558: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:12:18.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:18 smithi154 conmon[34553]: cluster 2021-07-31T03:12:17.014153+0000 mgr.x (mgr.24542) 3643 : cluster [DBG] pgmap v4559: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:12:18.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:18 smithi045 conmon[33655]: cluster 2021-07-31T03:12:17.014153+0000 mgr.x (mgr.24542) 3643 : cluster [DBG] pgmap v4559: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:12:18.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:18 smithi045 conmon[39718]: cluster 2021-07-31T03:12:17.014153+0000 mgr.x (mgr.24542) 3643 : 2021-07-31T03:12:18.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:18 smithi045 conmon[39718]: cluster [DBG] pgmap v4559: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:12:20.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:20 smithi154 conmon[34553]: cluster 2021-07-31T03:12:19.014564+0000 mgr.x (mgr.24542) 3644 : cluster [DBG] pgmap v4560: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:12:20.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:20 smithi045 conmon[39718]: cluster 2021-07-31T03:12:19.014564+0000 mgr.x (mgr.24542) 3644 : cluster 2021-07-31T03:12:20.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:20 smithi045 conmon[39718]: [DBG] pgmap v4560: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:12:20.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:20 smithi045 conmon[33655]: cluster 2021-07-31T03:12:19.014564+0000 mgr.x (mgr.24542) 2021-07-31T03:12:20.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:20 smithi045 conmon[33655]: 3644 : cluster [DBG] pgmap v4560: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:12:22.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:22 smithi154 conmon[34553]: cluster 2021-07-31T03:12:21.014957+0000 mgr.x (mgr.24542) 3645 : cluster [DBG] pgmap v4561: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:12:22.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:22 smithi045 conmon[39718]: cluster 2021-07-31T03:12:21.014957+0000 mgr.x (mgr.24542) 3645 2021-07-31T03:12:22.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:22 smithi045 conmon[39718]: : cluster [DBG] pgmap v4561: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:12:22.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:22 smithi045 conmon[33655]: cluster 2021-07-31T03:12:21.014957+0000 mgr.x (mgr.24542 2021-07-31T03:12:22.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:22 smithi045 conmon[33655]: ) 3645 : cluster [DBG] pgmap v4561: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:12:24.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:24 smithi154 conmon[34553]: cluster 2021-07-31T03:12:23.015468+0000 mgr.x (mgr.24542) 3646 : cluster [DBG] pgmap v4562: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:12:24.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:24 smithi154 conmon[34553]: 2021-07-31T03:12:24.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:24 smithi045 conmon[39718]: cluster 2021-07-31T03:12:23.015468+0000 mgr.x (mgr.24542) 3646 : cluster [DBG] 2021-07-31T03:12:24.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:24 smithi045 conmon[39718]: pgmap v4562: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:12:24.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:24 smithi045 conmon[33655]: cluster 2021-07-31T03:12:23.015468+0000 mgr.x 2021-07-31T03:12:24.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:24 smithi045 conmon[33655]: (mgr.24542) 3646 : cluster [DBG] pgmap v4562: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:12:26.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:26 smithi154 conmon[34553]: cluster 2021-07-31T03:12:25.015971+0000 mgr.x (mgr.24542) 3647 : cluster [DBG] pgmap v4563: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:12:26.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:26 smithi154 conmon[34553]: 2021-07-31T03:12:26.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:26 smithi045 conmon[39718]: cluster 2021-07-31T03:12:25.015971+0000 mgr.x 2021-07-31T03:12:26.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:26 smithi045 conmon[39718]: (mgr.24542) 3647 : cluster [DBG] pgmap v4563: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:12:26.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:26 smithi045 conmon[33655]: cluster 2021-07-31T03:12:25.015971+0000 mgr.x (mgr.24542) 3647 : 2021-07-31T03:12:26.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:26 smithi045 conmon[33655]: cluster [DBG] pgmap v4563: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:12:28.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:28 smithi154 conmon[34553]: cluster 2021-07-31T03:12:27.016491+0000 mgr.x (mgr.24542) 3648 : cluster [DBG] 2021-07-31T03:12:28.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:28 smithi154 conmon[34553]: pgmap v4564: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:12:28.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:28 smithi154 conmon[34553]: audit 2021-07-31T03:12:28.272879+0000 mon.b (mon.2) 500 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:12:28.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:28 smithi154 conmon[34553]: audit 2021-07-31T03:12:28.273152+0000 mon.b (mon.2) 2021-07-31T03:12:28.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:28 smithi154 conmon[34553]: 501 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:12:28.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:28 smithi154 conmon[34553]: audit 2021-07-31T03:12: 2021-07-31T03:12:28.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:28 smithi154 conmon[34553]: 28.273611+0000 mon.a (mon.0) 3267 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:12:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:28 smithi154 conmon[34553]: audit 2021-07-31T 2021-07-31T03:12:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:28 smithi154 conmon[34553]: 03:12:28.274093+0000 mon.a (mon.0) 3268 : 2021-07-31T03:12:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:28 smithi154 conmon[34553]: audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:12:28.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:28 smithi045 conmon[33655]: cluster 2021-07-31T03:12:27.016491+0000 mgr.x (mgr.24542) 3648 : cluster [DBG] 2021-07-31T03:12:28.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:28 smithi045 conmon[33655]: pgmap v4564: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:12:28.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:28 smithi045 conmon[33655]: audit 2021-07-31T03:12:28.272879+0000 mon.b (mon.2) 500 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:12:28.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:28 smithi045 conmon[33655]: audit 2021-07-31T03:12:28.273152+0000 mon.b (mon.2 2021-07-31T03:12:28.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:28 smithi045 conmon[33655]: ) 501 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:12:28.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:28 smithi045 conmon[33655]: audit 2021-07-31T 2021-07-31T03:12:28.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:28 smithi045 conmon[33655]: 03:12:28.273611+0000 mon.a (mon.0) 3267 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:12:28.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:28 smithi045 conmon[33655]: audit 2021-07-31T03:12:28.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:28 smithi045 conmon[33655]: 2021-07-31T03:12:28.274093+0000 mon.a 2021-07-31T03:12:28.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:28 smithi045 conmon[33655]: (mon.0) 3268 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:12:28.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:28 smithi045 conmon[39718]: cluster 2021-07-31T03:12:27.016491+0000 mgr.x 2021-07-31T03:12:28.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:28 smithi045 conmon[39718]: (mgr.24542) 3648 : cluster [DBG] pgmap v4564: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:12:28.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:28 smithi045 conmon[39718]: audit 2021-07-31T03:12:28.272879+0000 2021-07-31T03:12:28.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:28 smithi045 conmon[39718]: mon.b (mon.2) 500 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:12:28.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:28 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T03:12:28.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:28 smithi045 conmon[39718]: T03:12:28.273152+0000 mon.b ( 2021-07-31T03:12:28.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:28 smithi045 conmon[39718]: mon.2) 501 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:12:28.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:28 smithi045 conmon[39718]: audit 2021- 2021-07-31T03:12:28.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:28 smithi045 conmon[39718]: 07-31T03:12:28.273611+0000 2021-07-31T03:12:28.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:28 smithi045 conmon[39718]: mon.a (mon.0) 3267 : audit [INF] 2021-07-31T03:12:28.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:28 smithi045 conmon[39718]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:12:28.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:28 smithi045 conmon[39718]: audit 2021-07-31T03 2021-07-31T03:12:28.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:28 smithi045 conmon[39718]: :12:28.274093+0000 mon.a (mon 2021-07-31T03:12:28.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:28 smithi045 conmon[39718]: .0) 3268 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:12:30.749 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:12:30.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:30 smithi154 conmon[34553]: cluster 2021-07-31T03:12:29.016997+0000 mgr.x (mgr.24542) 3649 : cluster [DBG] pgmap v4565: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:12:30.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:30 smithi154 conmon[34553]: 2021-07-31T03:12:30.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:30 smithi045 conmon[33655]: cluster 2021-07-31T03:12:29.016997+0000 mgr.x (mgr.24542) 3649 : cluster [DBG] 2021-07-31T03:12:30.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:30 smithi045 conmon[33655]: pgmap v4565: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:12:30.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:30 smithi045 conmon[39718]: cluster 2021-07-31T03:12:29.016997+0000 mgr.x (mgr.24542) 3649 : cluster 2021-07-31T03:12:30.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:30 smithi045 conmon[39718]: [DBG] pgmap v4565: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:12:31.027 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:12:31.028 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 87m ago 88m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:12:31.028 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 85m ago 87m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:12:31.028 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 85m ago 91m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:12:31.028 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 87m ago 93m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:12:31.029 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 87m ago 93m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:12:31.029 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 85m ago 91m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:12:31.029 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 87m ago 91m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:12:31.029 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 87m ago 88m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:12:31.029 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 85m ago 88m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:12:31.030 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 87m ago 91m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:12:31.030 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 87m ago 90m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:12:31.030 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 87m ago 90m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:12:31.030 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 87m ago 90m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:12:31.030 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 85m ago 89m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:12:31.031 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 85m ago 89m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:12:31.031 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 85m ago 89m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:12:31.031 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 85m ago 88m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:12:31.031 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 85m ago 88m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:12:31.363 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:12:31.364 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:12:31.364 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:12:31.364 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:12:31.365 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:12:31.365 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:12:31.365 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:12:31.365 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:12:31.365 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:12:31.366 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:12:31.366 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:12:31.366 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:12:31.366 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:12:31.367 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:12:31.367 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:12:31.367 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:12:31.367 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:12:31.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[34553]: audit 2021-07-31T03:12:30.738365+0000 mgr.x (mgr.24542) 3650 2021-07-31T03:12:31.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[34553]: : audit [DBG] from='client.28960 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:12:31.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[34553]: audit 2021-07-31T03:12:31.364317+0000 mon.c (mon.1) 185 : audit [DBG] from='client.? 172.21.15.45:0/2763335745' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:12:31.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:31 smithi045 conmon[33655]: audit 2021-07-31T03:12:30.738365+0000 mgr.x (mgr 2021-07-31T03:12:31.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:31 smithi045 conmon[33655]: .24542) 3650 : audit [DBG] from='client.28960 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:12:31.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:31 smithi045 conmon[33655]: audit 2021-07-31T03:12:31.364317+0000 mon.c (mon.1) 2021-07-31T03:12:31.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:31 smithi045 conmon[33655]: 185 : audit [DBG] from='client.? 172.21.15.45:0/2763335745' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:12:31.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:31 smithi045 conmon[39718]: audit 2021-07-31T03:12 2021-07-31T03:12:31.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:31 smithi045 conmon[39718]: :30.738365+0000 mgr.x (mgr.24542) 3650 : audit [DBG] from='client.28960 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:12:31.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:31 smithi045 conmon[39718]: audit 2021- 2021-07-31T03:12:31.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:31 smithi045 conmon[39718]: 07-31T03:12:31.364317+0000 mon.c ( 2021-07-31T03:12:31.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:31 smithi045 conmon[39718]: mon.1) 185 : audit [DBG] 2021-07-31T03:12:31.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:31 smithi045 conmon[39718]: from='client.? 172.21.15.45:0/2763335745' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:12:32.253 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: debug 2021-07-31T03:12:31.890+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:12:32.253 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:12:32.253 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:12:32.253 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: yield 2021-07-31T03:12:32.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:12:32.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:12:32.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:12:32.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:12:32.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:12:32.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:12:32.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:12:32.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:12:32.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:12:32.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: task.result() 2021-07-31T03:12:32.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:12:32.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: debug 2021-07-31T03:12:31.901+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:12:32.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:12:32.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:12:32.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: yield 2021-07-31T03:12:32.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:12:32.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:12:32.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:12:32.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:12:32.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:12:32.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:12:32.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:12:32.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:12:32.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:12:32.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: task.result() 2021-07-31T03:12:32.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:12:32.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: debug 2021-07-31T03:12:31.926+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:12:32.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:12:32.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:12:32.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: yield 2021-07-31T03:12:32.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:12:32.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:12:32.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:12:32.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:12:32.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:12:32.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:12:32.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:12:32.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:12:32.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:12:32.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: task.result() 2021-07-31T03:12:32.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:12:32.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: debug 2021-07-31T03:12:31.927+0000 7fcb94162700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:12:32.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:12:32.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:12:32.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: yield 2021-07-31T03:12:32.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:12:32.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:12:32.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:12:32.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:12:32.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:12:32.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:12:32.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:12:32.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:12:32.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:12:32.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: task.result() 2021-07-31T03:12:32.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:12:32.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: 2021-07-31T03:12:32.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:12:32.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: 2021-07-31T03:12:32.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:12:32.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:12:32.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:12:32.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:12:32.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:12:32.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:12:32.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:12:32.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:12:32.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:12:32.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:12:32.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:12:32.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:12:32.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:12:32.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:12:32.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:12:32.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:12:32.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:12:32.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:12:32.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: raise 2021-07-31T03:12:32.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:12:32.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:12:32.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:12:32.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:12:32.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:12:32.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: debug 2021-07-31T03:12:31.935+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:12:32.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:12:32.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:12:32.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: yield 2021-07-31T03:12:32.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:12:32.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:12:32.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:12:32.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:12:32.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:12:32.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:12:32.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:12:32.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:12:32.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:12:32.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: task.result() 2021-07-31T03:12:32.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:12:32.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: debug 2021-07-31T03:12:31.936+0000 7fcb93961700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:12:32.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:12:32.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:12:32.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: yield 2021-07-31T03:12:32.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:12:32.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:12:32.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:12:32.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:12:32.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:12:32.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:12:32.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:12:32.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:12:32.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:12:32.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: task.result() 2021-07-31T03:12:32.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:12:32.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: 2021-07-31T03:12:32.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:12:32.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: 2021-07-31T03:12:32.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:12:32.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:12:32.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:12:32.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:12:32.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:12:32.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:12:32.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:12:32.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:12:32.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:12:32.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:12:32.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:12:32.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:12:32.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:12:32.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:12:32.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:12:32.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:12:32.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:12:32.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:12:32.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: raise 2021-07-31T03:12:32.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:12:32.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:12:32.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:12:32.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:12:32.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:12:31 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:12:32.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:32 smithi154 conmon[34553]: cluster 2021-07-31T03:12:31.017321+0000 mgr.x (mgr.24542) 2021-07-31T03:12:32.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:32 smithi154 conmon[34553]: 3651 : cluster [DBG] pgmap v4566: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:12:32.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:32 smithi154 conmon[34553]: audit 2021-07-31T03:12:31.020396+0000 mgr.x (mgr.24542) 3652 : audit [DBG] from='client.28966 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:12:32.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:32 smithi154 conmon[34553]: audit 2021-07-31T03:12:31.845205+0000 mon.b (mon.2) 502 : audit [DBG] 2021-07-31T03:12:32.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:32 smithi154 conmon[34553]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:12:32.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:32 smithi154 conmon[34553]: audit 2021-07-31T03:12:31 2021-07-31T03:12:32.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:32 smithi154 conmon[34553]: .846611+0000 mon.b (mon.2) 503 : audit 2021-07-31T03:12:32.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:32 smithi154 conmon[34553]: [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:12:32.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:32 smithi154 conmon[34553]: audit 2021-07-31 2021-07-31T03:12:32.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:32 smithi154 conmon[34553]: T03:12:31.847619+0000 mon.b (mon 2021-07-31T03:12:32.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:32 smithi154 conmon[34553]: .2) 504 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:12:32.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:32 smithi045 conmon[39718]: cluster 2021-07-31T03:12:31.017321+0000 mgr.x (mgr.24542) 3651 : cluster 2021-07-31T03:12:32.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:32 smithi045 conmon[39718]: [DBG] pgmap v4566: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:12:32.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:32 smithi045 conmon[39718]: audit 2021-07-31T03:12:31.020396+0000 mgr.x (mgr.24542) 3652 : audit [DBG] from='client.28966 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:12:32.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:32 smithi045 conmon[39718]: audit 2021-07-31T03:12:31.845205+0000 mon.b (mon.2) 502 : audit [DBG] 2021-07-31T03:12:32.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:32 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:12:32.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:32 smithi045 conmon[39718]: audit 2021- 2021-07-31T03:12:32.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:32 smithi045 conmon[39718]: 07-31T03:12:31.846611+0000 mon.b (mon.2) 503 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:12:32.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:32 smithi045 conmon[39718]: audit 2021-07-31T03:12:31. 2021-07-31T03:12:32.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:32 smithi045 conmon[39718]: 847619+0000 mon.b (mon.2) 504 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:12:32.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:32 smithi045 conmon[33655]: cluster 2021-07-31T03:12:31.017321+0000 mgr.x (mgr. 2021-07-31T03:12:32.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:32 smithi045 conmon[33655]: 24542) 3651 : cluster [DBG] pgmap v4566: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:12:32.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:32 smithi045 conmon[33655]: audit 2021-07-31T03:12:31.020396+0000 mgr.x (mgr.24542) 3652 2021-07-31T03:12:32.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:32 smithi045 conmon[33655]: : audit [DBG] from='client.28966 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:12:32.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:32 smithi045 conmon[33655]: audit 2021-07-31T03:12:31.845205+0000 mon.b (mon.2) 502 : audit [DBG] 2021-07-31T03:12:32.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:32 smithi045 conmon[33655]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:12:32.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:32 smithi045 conmon[33655]: audit 2021-07-31T03:12:31.846611+0000 mon.b (mon.2) 2021-07-31T03:12:32.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:32 smithi045 conmon[33655]: 503 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:12:32.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:32 smithi045 conmon[33655]: audit 2021- 2021-07-31T03:12:32.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:32 smithi045 conmon[33655]: 07-31T03:12:31.847619+0000 mon.b (mon.2) 504 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:12:33.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:12:31.848344+0000 mgr.x (mgr.24542) 3653 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:12:33.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:12:31.891160+0000 mgr.x (mgr.24542) 3654 : cephadm [ERR] 2021-07-31T03:12:33.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:12:33.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:12:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: yield 2021-07-31T03:12:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:12:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:12:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:12:33.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:12:33.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:12:33.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:12:33.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:12:33.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:12:33.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:12:33.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: task.result() 2021-07-31T03:12:33.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:12:33.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:12:31.891572+0000 mgr.x (mgr.24542) 3655 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:12:33.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:12:31.902517+0000 mgr.x (mgr.24542) 3656 : cephadm [ERR] 2021-07-31T03:12:33.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:12:33.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:12:33.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: yield 2021-07-31T03:12:33.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:12:33.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:12:33.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:12:33.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:12:33.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:12:33.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:12:33.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:12:33.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:12:33.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:12:33.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: task.result() 2021-07-31T03:12:33.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:12:33.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:12:31.903468+0000 mgr.x (mgr.24542) 3657 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:12:33.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:12:31.927834+0000 mgr.x (mgr.24542) 3658 : cephadm [ERR] 2021-07-31T03:12:33.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:12:33.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:12:33.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: yield 2021-07-31T03:12:33.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:12:33.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:12:33.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:12:33.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:12:33.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:12:33.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:12:33.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:12:33.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:12:33.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:12:33.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: task.result() 2021-07-31T03:12:33.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:12:33.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: cephadm 2021- 2021-07-31T03:12:33.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: 07-31T03:12:31.928255+0000 mgr.x (mgr.24542) 3659 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:12:33.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:12:33.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:12:33.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: yield 2021-07-31T03:12:33.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:12:33.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:12:33.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:12:33.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:12:33.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:12:33.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:12:33.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:12:33.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:12:33.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:12:33.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: task.result() 2021-07-31T03:12:33.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:12:33.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: 2021-07-31T03:12:33.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:12:33.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: 2021-07-31T03:12:33.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:12:33.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:12:33.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:12:33.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:12:33.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:12:33.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:12:33.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:12:33.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:12:33.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:12:33.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:12:33.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:12:33.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:12:33.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:12:33.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:12:33.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:12:33.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:12:33.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:12:33.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:12:33.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: raise 2021-07-31T03:12:33.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:12:33.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:12:33.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:12:33.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:12:33.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:12:33.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:12:31.936674+0000 mgr.x (mgr.24542) 3660 : cephadm 2021-07-31T03:12:33.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: [ERR] 2021-07-31T03:12:33.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:12:33.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:12:33.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: yield 2021-07-31T03:12:33.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:12:33.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:12:33.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:12:33.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:12:33.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:12:33.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:12:33.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:12:33.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:12:33.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:12:33.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: task.result() 2021-07-31T03:12:33.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:12:33.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:12:31.937223+0000 mgr.x (mgr.24542) 3661 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:12:33.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:12:33.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:12:33.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: yield 2021-07-31T03:12:33.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:12:33.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:12:33.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:12:33.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:12:33.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:12:33.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:12:33.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:12:33.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:12:33.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:12:33.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: task.result() 2021-07-31T03:12:33.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:12:33.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: 2021-07-31T03:12:33.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:12:33.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: 2021-07-31T03:12:33.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:12:33.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:12:33.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:12:33.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:12:33.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:12:33.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:12:33.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:12:33.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:12:33.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:12:33.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:12:33.782 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:12:33.782 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:12:33.782 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:12:33.782 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:12:33.783 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:12:33.783 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:12:33.783 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:12:33.783 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:12:33.783 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: raise 2021-07-31T03:12:33.784 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:12:33.784 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:12:33.784 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:12:33.784 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:12:33.784 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:33 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:12:33.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: cephadm 2021-07-31T03:12:31.848344+0000 mgr.x (mgr 2021-07-31T03:12:33.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: .24542) 3653 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:12:33.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: cephadm 2021-07-31T03:12:31.891160+0000 mgr.x 2021-07-31T03:12:33.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: (mgr.24542) 3654 : cephadm [ERR] 2021-07-31T03:12:33.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:12:33.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:12:33.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: yield 2021-07-31T03:12:33.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:12:33.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:12:33.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:12:33.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:12:33.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:12:33.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:12:33.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:12:33.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:12:33.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:12:33.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: task.result() 2021-07-31T03:12:33.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:12:33.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: cephadm 2021-07-31T 2021-07-31T03:12:33.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: 03:12:31.891572+0000 mgr.x (mgr.24542) 3655 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:12:33.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: cephadm 2021-07-31T03:12:31.902517+0000 mgr.x (mgr.24542) 3656 : cephadm [ERR] 2021-07-31T03:12:33.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:12:33.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:12:33.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: yield 2021-07-31T03:12:33.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:12:33.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:12:33.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:12:33.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:12:33.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:12:33.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:12:33.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:12:33.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:12:33.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:12:33.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: task.result() 2021-07-31T03:12:33.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:12:33.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: cephadm 2021-07-31T03:12:31.903468+0000 mgr.x (mgr.24542) 3657 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:12:33.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: cephadm 2021-07-31T03:12:31.927834+0000 mgr.x (mgr.24542) 3658 : cephadm [ERR] 2021-07-31T03:12:33.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:12:33.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:12:33.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: yield 2021-07-31T03:12:33.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:12:33.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:12:33.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:12:33.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:12:33.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:12:33.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:12:33.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:12:33.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:12:33.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:12:33.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: task.result() 2021-07-31T03:12:33.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:12:33.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: cephadm 2021-07-31T03:12:31. 2021-07-31T03:12:33.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: 928255+0000 mgr.x (mgr.24542) 3659 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:12:33.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:12:33.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:12:33.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: yield 2021-07-31T03:12:33.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:12:33.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:12:33.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:12:33.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:12:33.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:12:33.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:12:33.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:12:33.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:12:33.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:12:33.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: task.result() 2021-07-31T03:12:33.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:12:33.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: 2021-07-31T03:12:33.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:12:33.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: 2021-07-31T03:12:33.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:12:33.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:12:33.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:12:33.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:12:33.802 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:12:33.802 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:12:33.802 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:12:33.802 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:12:33.803 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:12:33.803 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:12:33.803 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: cephadm 2021-07-31T03:12:31.848344+0000 mgr.x (mgr.24542) 2021-07-31T03:12:33.804 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: 3653 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:12:33.804 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: cephadm 2021-07-31T03:12:31.891160+0000 mgr.x (mgr.24542) 3654 : cephadm [ERR] 2021-07-31T03:12:33.804 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:12:33.804 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:12:33.804 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: yield 2021-07-31T03:12:33.805 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:12:33.805 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:12:33.805 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:12:33.805 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:12:33.806 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:12:33.806 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:12:33.806 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:12:33.806 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:12:33.807 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:12:33.807 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: task.result() 2021-07-31T03:12:33.807 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:12:33.807 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: cephadm 2021-07-31T03:12:31.891572+0000 mgr.x (mgr.24542) 3655 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:12:33.807 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: cephadm 2021-07-31T03:12:31.902517+0000 2021-07-31T03:12:33.808 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: mgr.x (mgr.24542) 3656 : cephadm [ERR] 2021-07-31T03:12:33.808 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:12:33.808 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:12:33.808 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: yield 2021-07-31T03:12:33.808 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:12:33.809 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:12:33.809 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:12:33.809 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:12:33.809 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:12:33.810 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:12:33.810 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:12:33.810 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:12:33.810 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:12:33.810 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: task.result() 2021-07-31T03:12:33.811 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:12:33.811 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: cephadm 2021-07-31T03:12:31.903468+0000 mgr.x (mgr.24542) 3657 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:12:33.811 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: cephadm 2021-07-31T03:12:31.927834 2021-07-31T03:12:33.811 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 3658 : cephadm [ERR] 2021-07-31T03:12:33.812 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:12:33.812 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:12:33.812 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:12:33.812 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:12:33.813 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:12:33.813 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:12:33.813 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:12:33.813 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:12:33.814 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:12:33.814 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: raise 2021-07-31T03:12:33.814 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:12:33.814 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:12:33.814 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:12:33.815 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:12:33.815 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:12:33.815 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: cephadm 2021-07-31T03:12:31.936674+0000 mgr.x (mgr.24542) 3660 : cephadm [ERR] 2021-07-31T03:12:33.815 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:12:33.816 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:12:33.816 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: yield 2021-07-31T03:12:33.816 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:12:33.816 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:12:33.816 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:12:33.817 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:12:33.817 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:12:33.817 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:12:33.817 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:12:33.817 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:12:33.818 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:12:33.818 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: task.result() 2021-07-31T03:12:33.818 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:12:33.818 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: cephadm 2021-07-31T03:12:31.937223+0000 mgr.x (mgr.24542) 3661 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:12:33.819 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:12:33.819 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:12:33.819 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: yield 2021-07-31T03:12:33.819 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:12:33.819 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:12:33.820 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:12:33.820 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:12:33.820 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:12:33.820 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:12:33.821 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:12:33.821 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: yield 2021-07-31T03:12:33.821 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:12:33.821 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:12:33.822 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:12:33.822 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:12:33.822 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:12:33.822 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:12:33.823 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:12:33.823 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:12:33.823 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:12:33.823 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: task.result() 2021-07-31T03:12:33.824 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:12:33.824 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: cephadm 2021 2021-07-31T03:12:33.824 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: -07-31T03:12:31.928255+0000 mgr.x (mgr.24542) 3659 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:12:33.824 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:12:33.824 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:12:33.825 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: yield 2021-07-31T03:12:33.825 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:12:33.825 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:12:33.825 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:12:33.826 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:12:33.826 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:12:33.826 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:12:33.826 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:12:33.826 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:12:33.827 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:12:33.827 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: task.result() 2021-07-31T03:12:33.827 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:12:33.827 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: 2021-07-31T03:12:33.827 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:12:33.828 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: 2021-07-31T03:12:33.828 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:12:33.828 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:12:33.828 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:12:33.829 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:12:33.829 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:12:33.829 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:12:33.829 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:12:33.829 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:12:33.830 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:12:33.830 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:12:33.830 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:12:33.831 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:12:33.831 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:12:33.831 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: task.result() 2021-07-31T03:12:33.831 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:12:33.831 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: 2021-07-31T03:12:33.832 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:12:33.832 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: 2021-07-31T03:12:33.832 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:12:33.832 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:12:33.832 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:12:33.833 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:12:33.833 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:12:33.833 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:12:33.833 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:12:33.834 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:12:33.834 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:12:33.834 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:12:33.834 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:12:33.834 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:12:33.835 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:12:33.835 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:12:33.835 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:12:33.835 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:12:33.836 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:12:33.836 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:12:33.836 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: raise 2021-07-31T03:12:33.836 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:12:33.836 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:12:33.837 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:12:33.837 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:12:33.837 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:12:33.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:12:33.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:12:33.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:12:33.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:12:33.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:12:33.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:12:33.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:12:33.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:12:33.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: raise 2021-07-31T03:12:33.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:12:33.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:12:33.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:12:33.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:12:33.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:12:33.841 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: cephadm 2021-07-31T03:12:31.936674+0000 mgr.x (mgr.24542) 3660 : cephadm [ERR] 2021-07-31T03:12:33.841 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:12:33.841 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:12:33.841 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: yield 2021-07-31T03:12:33.842 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:12:33.842 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:12:33.842 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:12:33.842 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:12:33.842 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:12:33.843 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:12:33.843 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:12:33.843 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:12:33.843 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:12:33.843 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: task.result() 2021-07-31T03:12:33.844 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:12:33.844 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: cephadm 2021-07-31T 2021-07-31T03:12:33.844 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: 03:12:31.937223+0000 mgr.x (mgr.24542) 3661 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:12:33.844 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:12:33.845 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:12:33.845 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: yield 2021-07-31T03:12:33.845 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:12:33.845 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:12:33.845 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:12:33.846 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:12:33.846 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:12:33.846 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:12:33.846 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:12:33.847 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:12:33.847 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:12:33.847 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: task.result() 2021-07-31T03:12:33.847 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:12:33.847 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: 2021-07-31T03:12:33.848 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:12:33.848 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: 2021-07-31T03:12:33.848 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:12:33.848 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:12:33.848 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:12:33.849 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:12:33.849 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:12:33.849 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:12:33.849 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:12:33.850 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:12:33.850 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:12:33.850 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:12:33.850 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:12:33.851 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:12:33.851 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:12:33.851 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:12:33.851 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:12:33.851 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:12:33.852 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:12:33.852 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:12:33.852 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: raise 2021-07-31T03:12:33.852 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:12:33.852 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:12:33.853 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:12:33.853 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:12:33.853 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:33 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:12:34.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:34 smithi154 conmon[34553]: cluster 2021-07-31T03:12:33.017675+0000 mgr.x (mgr.24542) 3662 : cluster [DBG] pgmap v4567: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:12:34.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:34 smithi045 conmon[39718]: cluster 2021-07-31T03:12:33.017675+0000 2021-07-31T03:12:34.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:34 smithi045 conmon[39718]: mgr.x (mgr.24542) 3662 : cluster [DBG] pgmap v4567: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:12:34.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:34 smithi045 conmon[33655]: cluster 2021-07-31T03:12:33.017675+0000 mgr.x (mgr.24542) 3662 2021-07-31T03:12:34.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:34 smithi045 conmon[33655]: : cluster [DBG] pgmap v4567: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:12:36.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:36 smithi154 conmon[34553]: cluster 2021-07-31T03:12:35.018109+0000 mgr.x (mgr.24542) 3663 : cluster [DBG] 2021-07-31T03:12:36.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:36 smithi154 conmon[34553]: pgmap v4568: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:12:36.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:36 smithi045 conmon[39718]: cluster 2021-07-31T03:12:35.018109+0000 mgr.x 2021-07-31T03:12:36.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:36 smithi045 conmon[39718]: (mgr.24542) 3663 : cluster [DBG] pgmap v4568: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:12:36.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:36 smithi045 conmon[33655]: cluster 2021-07-31T03 2021-07-31T03:12:36.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:36 smithi045 conmon[33655]: :12:35.018109+0000 mgr.x (mgr.24542) 3663 : cluster [DBG] pgmap v4568: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:12:38.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:38 smithi154 conmon[34553]: cluster 2021-07-31T03:12:37.018552+0000 mgr.x (mgr.24542) 3664 : cluster [DBG] pgmap v4569: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:12:38.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:38 smithi045 conmon[33655]: cluster 2021-07-31T03:12:37.018552+0000 mgr.x (mgr.24542) 3664 : cluster [DBG] pgmap v4569: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:12:38.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:38 smithi045 conmon[39718]: cluster 2021-07-31T03:12:37.018552+0000 mgr.x 2021-07-31T03:12:38.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:38 smithi045 conmon[39718]: (mgr.24542) 3664 : cluster [DBG] pgmap v4569: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:12:40.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:40 smithi154 conmon[34553]: cluster 2021-07-31T03:12:39.019039+0000 mgr.x (mgr.24542) 3665 : cluster [DBG] pgmap v4570: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:12:40.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:40 smithi154 conmon[34553]: 2021-07-31T03:12:40.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:40 smithi045 conmon[33655]: cluster 2021-07-31T03:12:39.019039+0000 mgr.x (mgr.24542) 3665 : cluster [DBG] 2021-07-31T03:12:40.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:40 smithi045 conmon[33655]: pgmap v4570: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:12:40.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:40 smithi045 conmon[39718]: cluster 2021-07-31T03:12:39.019039+0000 mgr.x 2021-07-31T03:12:40.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:40 smithi045 conmon[39718]: (mgr.24542) 3665 : cluster [DBG] pgmap v4570: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:12:42.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:42 smithi154 conmon[34553]: cluster 2021-07-31T03:12:41.019562+0000 mgr.x (mgr.24542) 3666 : cluster [DBG] pgmap v4571: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:12:42.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:42 smithi154 conmon[34553]: 2021-07-31T03:12:42.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:42 smithi045 conmon[33655]: cluster 2021-07-31T03:12:41.019562+0000 mgr.x (mgr.24542) 3666 : cluster 2021-07-31T03:12:42.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:42 smithi045 conmon[33655]: [DBG] pgmap v4571: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:12:42.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:42 smithi045 conmon[39718]: cluster 2021-07-31T03:12:41.019562+0000 mgr.x (mgr.24542 2021-07-31T03:12:42.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:42 smithi045 conmon[39718]: ) 3666 : cluster [DBG] pgmap v4571: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:12:44.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:44 smithi154 conmon[34553]: cluster 2021-07-31T03:12:43.019997+0000 mgr.x (mgr.24542) 3667 : cluster [DBG] pgmap v4572: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:12:44.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:44 smithi154 conmon[34553]: 2021-07-31T03:12:44.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:44 smithi045 conmon[39718]: cluster 2021-07-31T03:12:43.019997+0000 mgr.x (mgr.24542 2021-07-31T03:12:44.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:44 smithi045 conmon[39718]: ) 3667 : cluster [DBG] pgmap v4572: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:12:44.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:44 smithi045 conmon[33655]: cluster 2021-07-31T03:12:43.019997+0000 mgr.x (mgr.24542) 3667 : cluster [DBG] 2021-07-31T03:12:44.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:44 smithi045 conmon[33655]: pgmap v4572: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:12:46.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:46 smithi154 conmon[34553]: cluster 2021-07-31T03:12:45.020544+0000 mgr.x (mgr.24542) 2021-07-31T03:12:46.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:46 smithi154 conmon[34553]: 3668 : cluster [DBG] pgmap v4573: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:12:46.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:46 smithi045 conmon[39718]: cluster 2021-07-31T03:12:45.020544+0000 mgr.x (mgr. 2021-07-31T03:12:46.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:46 smithi045 conmon[39718]: 24542) 3668 : cluster [DBG] pgmap v4573: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:12:46.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:46 smithi045 conmon[33655]: cluster 2021-07-31T03:12:45.020544+0000 mgr.x (mgr.24542) 3668 : cluster [DBG] 2021-07-31T03:12:46.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:46 smithi045 conmon[33655]: pgmap v4573: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:12:48.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:48 smithi154 conmon[34553]: cluster 2021-07-31T03:12:47.020974+0000 mgr.x (mgr.24542) 3669 : cluster 2021-07-31T03:12:48.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:48 smithi154 conmon[34553]: [DBG] pgmap v4574: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:12:48.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:48 smithi045 conmon[33655]: cluster 2021-07-31T03:12: 2021-07-31T03:12:48.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:48 smithi045 conmon[33655]: 47.020974+0000 mgr.x (mgr.24542) 3669 : cluster [DBG] pgmap v4574: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:12:48.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:48 smithi045 conmon[39718]: cluster 2021-07-31T03:12:47.020974+0000 mgr.x 2021-07-31T03:12:48.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:48 smithi045 conmon[39718]: (mgr.24542) 3669 : cluster [DBG] pgmap v4574: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:12:50.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:50 smithi154 conmon[34553]: cluster 2021-07-31T03:12:49.021343+0000 mgr.x (mgr.24542) 3670 : cluster [DBG] pgmap v4575: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:12:50.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:50 smithi045 conmon[33655]: cluster 2021-07-31T03:12:49.021343+0000 mgr.x (mgr.24542) 2021-07-31T03:12:50.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:50 smithi045 conmon[33655]: 3670 : cluster [DBG] pgmap v4575: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:12:50.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:50 smithi045 conmon[39718]: cluster 2021-07-31T03:12:49.021343+0000 mgr.x ( 2021-07-31T03:12:50.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:50 smithi045 conmon[39718]: mgr.24542) 3670 : cluster [DBG] pgmap v4575: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:12:52.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:52 smithi154 conmon[34553]: cluster 2021-07-31T03:12:51.021765+0000 mgr.x (mgr.24542) 3671 : cluster [DBG] pgmap v4576: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:12:52.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:52 smithi045 conmon[39718]: cluster 2021-07-31T03:12:51.021765+0000 mgr.x (mgr.24542) 3671 : cluster [DBG] 2021-07-31T03:12:52.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:52 smithi045 conmon[39718]: pgmap v4576: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:12:52.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:52 smithi045 conmon[33655]: cluster 2021-07-31T03:12:51.021765+0000 mgr.x (mgr.24542 2021-07-31T03:12:52.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:52 smithi045 conmon[33655]: ) 3671 : cluster [DBG] pgmap v4576: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:12:54.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:54 smithi154 conmon[34553]: cluster 2021-07-31T03:12:53.022309+0000 mgr.x (mgr.24542) 3672 : cluster [DBG] pgmap v4577: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:12:54.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:54 smithi045 conmon[39718]: cluster 2021-07-31T03:12:53.022309+0000 mgr.x ( 2021-07-31T03:12:54.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:54 smithi045 conmon[39718]: mgr.24542) 3672 : cluster [DBG] pgmap v4577: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:12:54.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:54 smithi045 conmon[33655]: cluster 2021-07-31T03:12:53.022309+0000 mgr.x (mgr.24542) 3672 : cluster [DBG] pgmap v4577: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:12:56.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:56 smithi154 conmon[34553]: cluster 2021-07-31T03:12:55.022766+0000 mgr.x (mgr.24542) 3673 : cluster [DBG] pgmap v4578: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:12:56.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:56 smithi045 conmon[39718]: cluster 2021-07-31T03:12:55.022766+0000 mgr.x (mgr. 2021-07-31T03:12:56.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:56 smithi045 conmon[39718]: 24542) 3673 : cluster [DBG] pgmap v4578: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:12:56.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:56 smithi045 conmon[33655]: cluster 2021-07-31T03:12:55.022766+0000 mgr.x (mgr.24542) 3673 : cluster [DBG] pgmap v4578: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:12:58.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:12:58 smithi154 conmon[34553]: cluster 2021-07-31T03:12:57.023155+0000 mgr.x (mgr.24542) 3674 : cluster [DBG] pgmap v4579: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:12:58.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:58 smithi045 conmon[33655]: cluster 2021-07-31T03:12:57.023155+0000 mgr.x (mgr. 2021-07-31T03:12:58.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:12:58 smithi045 conmon[33655]: 24542) 3674 : cluster [DBG] pgmap v4579: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:12:58.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:58 smithi045 conmon[39718]: cluster 2021-07-31T03:12:57.023155+0000 mgr.x (mgr.24542) 3674 : cluster 2021-07-31T03:12:58.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:12:58 smithi045 conmon[39718]: [DBG] pgmap v4579: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:00.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:00 smithi154 conmon[34553]: cluster 2021-07-31T03:12:59.023567+0000 mgr.x (mgr.24542) 3675 : cluster [DBG] pgmap v4580: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:00.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:00 smithi045 conmon[39718]: cluster 2021-07-31T03:12:59. 2021-07-31T03:13:00.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:00 smithi045 conmon[39718]: 023567+0000 mgr.x (mgr.24542) 3675 : cluster [DBG] pgmap v4580: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:00.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:00 smithi045 conmon[33655]: cluster 2021-07-31T03:12:59.023567+0000 mgr.x (mgr.24542) 3675 : cluster [DBG] pgmap v4580: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:01.675 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:13:01.951 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:13:01.951 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 87m ago 88m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:13:01.951 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 85m ago 88m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:13:01.952 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 85m ago 91m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:13:01.952 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 87m ago 94m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:13:01.952 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 87m ago 94m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:13:01.952 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 85m ago 92m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:13:01.952 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 87m ago 92m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:13:01.953 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 87m ago 88m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:13:01.953 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 85m ago 88m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:13:01.953 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 87m ago 91m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:13:01.953 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 87m ago 91m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:13:01.953 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 87m ago 90m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:13:01.954 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 87m ago 90m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:13:01.954 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 85m ago 90m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:13:01.954 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 85m ago 90m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:13:01.954 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 85m ago 89m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:13:01.955 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 85m ago 89m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:13:01.955 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 85m ago 89m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:13:02.288 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:13:02.288 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:13:02.288 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:13:02.289 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:13:02.289 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:13:02.289 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:13:02.289 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:13:02.290 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:13:02.290 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:13:02.290 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:13:02.290 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:13:02.290 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:13:02.291 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:13:02.291 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:13:02.291 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:13:02.291 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:13:02.292 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:13:02.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:02 smithi154 conmon[34553]: cluster 2021-07-31T03:13:01.024065+0000 mgr.x (mgr.24542) 3676 : cluster [DBG] pgmap v4581: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:02.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:02 smithi154 conmon[34553]: audit 2021-07-31T03:13:01.664034+0000 2021-07-31T03:13:02.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:02 smithi154 conmon[34553]: mgr.x (mgr.24542) 3677 : audit [DBG] from='client.28978 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:13:02.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:02 smithi154 conmon[34553]: audit 2021-07-31T03:13:02.288655+0000 mon.c (mon.1) 186 : audit [DBG] from='client.? 172.21.15.45:0/778517922' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:13:02.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:02 smithi045 conmon[33655]: cluster 2021-07-31T03:13:01.024065 2021-07-31T03:13:02.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:02 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 3676 : cluster [DBG] pgmap v4581: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:02.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:02 smithi045 conmon[33655]: audit 2021-07-31 2021-07-31T03:13:02.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:02 smithi045 conmon[33655]: T03:13:01.664034+0000 mgr.x (mgr.24542 2021-07-31T03:13:02.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:02 smithi045 conmon[33655]: ) 3677 : audit [DBG] from='client.28978 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:13:02.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:02 smithi045 conmon[33655]: audit 2021 2021-07-31T03:13:02.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:02 smithi045 conmon[33655]: -07-31T03:13:02. 2021-07-31T03:13:02.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:02 smithi045 conmon[33655]: 288655+0000 mon.c (mon.1 2021-07-31T03:13:02.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:02 smithi045 conmon[33655]: ) 186 : audit [DBG] from='client.? 172.21.15.45:0/778517922' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:13:02.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:02 smithi045 conmon[33655]: 2021-07-31T03:13:02.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:02 smithi045 conmon[39718]: cluster 2021-07-31T03:13:01.024065 2021-07-31T03:13:02.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:02 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 3676 : cluster [DBG] pgmap v4581: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:02.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:02 smithi045 conmon[39718]: audit 2021-07-31T03:13:02.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:02 smithi045 conmon[39718]: 2021-07-31T03:13:01.664034+0000 mgr.x ( 2021-07-31T03:13:02.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:02 smithi045 conmon[39718]: mgr.24542) 3677 : audit [DBG] from='client.28978 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:13:02.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:02 smithi045 conmon[39718]: 2021-07-31T03:13:02.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:02 smithi045 conmon[39718]: audit 2021-07-31T03:13:02.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:02 smithi045 conmon[39718]: 2021-07-31T03:13:02.288655 2021-07-31T03:13:02.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:02 smithi045 conmon[39718]: +0000 mon.c (mon.1) 186 : audit 2021-07-31T03:13:02.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:02 smithi045 conmon[39718]: [DBG] from='client.? 172.21.15.45:0/778517922' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:13:03.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:03 smithi154 conmon[34553]: audit 2021-07-31T03:13:01.943998+0000 mgr.x (mgr.24542) 3678 : audit [DBG] from='client.28984 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:13:03.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:03 smithi154 conmon[34553]: 2021-07-31T03:13:03.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:03 smithi045 conmon[39718]: audit 2021-07-31T03:13:01.943998+0000 mgr.x (mgr. 2021-07-31T03:13:03.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:03 smithi045 conmon[39718]: 24542) 3678 : audit [DBG] from='client.28984 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:13:03.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:03 smithi045 conmon[33655]: audit 2021-07-31T03:13:01.943998+0000 mgr.x 2021-07-31T03:13:03.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:03 smithi045 conmon[33655]: (mgr.24542) 3678 : audit [DBG] from='client.28984 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:13:04.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:04 smithi154 conmon[34553]: cluster 2021-07-31T03:13:03.024444+0000 mgr.x (mgr.24542) 3679 : cluster [DBG] 2021-07-31T03:13:04.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:04 smithi154 conmon[34553]: pgmap v4582: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:04.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:04 smithi045 conmon[39718]: cluster 2021-07-31T03:13:03.024444+0000 mgr.x (mgr.24542) 3679 2021-07-31T03:13:04.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:04 smithi045 conmon[39718]: : cluster [DBG] pgmap v4582: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:04.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:04 smithi045 conmon[33655]: cluster 2021-07-31T03:13:03.024444+0000 mgr.x (mgr. 2021-07-31T03:13:04.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:04 smithi045 conmon[33655]: 24542) 3679 : cluster [DBG] pgmap v4582: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:06.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:06 smithi154 conmon[34553]: cluster 2021-07-31T03:13:05.024978+0000 mgr.x (mgr.24542) 3680 : cluster 2021-07-31T03:13:06.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:06 smithi154 conmon[34553]: [DBG] pgmap v4583: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:06.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:06 smithi045 conmon[33655]: cluster 2021-07-31T03:13:05.024978+0000 mgr.x (mgr.24542) 3680 2021-07-31T03:13:06.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:06 smithi045 conmon[33655]: : cluster [DBG] pgmap v4583: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:06.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:06 smithi045 conmon[39718]: cluster 2021-07-31T03:13:05.024978+0000 mgr.x (mgr.24542) 2021-07-31T03:13:06.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:06 smithi045 conmon[39718]: 3680 : cluster [DBG] pgmap v4583: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:08.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:08 smithi154 conmon[34553]: cluster 2021-07-31T03:13:07.025407+0000 mgr.x (mgr.24542) 3681 : cluster [DBG] pgmap v4584: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:08.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:08 smithi045 conmon[39718]: cluster 2021-07-31T03:13:07.025407+0000 mgr.x (mgr. 2021-07-31T03:13:08.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:08 smithi045 conmon[39718]: 24542) 3681 : cluster [DBG] pgmap v4584: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:08.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:08 smithi045 conmon[33655]: cluster 2021-07-31T03:13:07.025407+0000 mgr.x (mgr.24542 2021-07-31T03:13:08.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:08 smithi045 conmon[33655]: ) 3681 : cluster [DBG] pgmap v4584: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:10.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:10 smithi154 conmon[34553]: cluster 2021-07-31T03:13:09.025843+0000 mgr.x (mgr.24542) 3682 : cluster [DBG] pgmap v4585: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:10.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:10 smithi045 conmon[33655]: cluster 2021-07-31T03:13:09.025843+0000 mgr.x ( 2021-07-31T03:13:10.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:10 smithi045 conmon[33655]: mgr.24542) 3682 : cluster [DBG] pgmap v4585: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:10.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:10 smithi045 conmon[39718]: cluster 2021-07-31T03:13:09.025843+0000 mgr.x (mgr. 2021-07-31T03:13:10.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:10 smithi045 conmon[39718]: 24542) 3682 : cluster [DBG] pgmap v4585: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:12.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:12 smithi154 conmon[34553]: cluster 2021-07-31T03:13:11.026345+0000 mgr.x (mgr.24542) 3683 : cluster 2021-07-31T03:13:12.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:12 smithi154 conmon[34553]: [DBG] pgmap v4586: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:12.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:12 smithi045 conmon[33655]: cluster 2021-07-31T03:13:11.026345+0000 mgr.x 2021-07-31T03:13:12.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:12 smithi045 conmon[33655]: (mgr.24542) 3683 : cluster [DBG] pgmap v4586: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:12.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:12 smithi045 conmon[39718]: cluster 2021-07-31T03:13:11.026345+0000 mgr.x ( 2021-07-31T03:13:12.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:12 smithi045 conmon[39718]: mgr.24542) 3683 : cluster [DBG] pgmap v4586: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:14.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:14 smithi154 conmon[34553]: cluster 2021-07-31T03:13:13.026735+0000 mgr.x (mgr.24542) 3684 : cluster [DBG] pgmap v4587: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:14.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:14 smithi045 conmon[33655]: cluster 2021-07-31 2021-07-31T03:13:14.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:14 smithi045 conmon[33655]: T03:13:13.026735+0000 mgr.x (mgr.24542) 3684 : cluster [DBG] pgmap v4587: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:14.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:14 smithi045 conmon[39718]: cluster 2021-07-31T03:13:13.026735+0000 mgr.x (mgr. 2021-07-31T03:13:14.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:14 smithi045 conmon[39718]: 24542) 3684 : cluster [DBG] pgmap v4587: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:16.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:16 smithi154 conmon[34553]: cluster 2021-07-31T03:13:15.027163+0000 mgr.x (mgr.24542) 3685 : cluster [DBG] pgmap v4588: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:16.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:16 smithi045 conmon[33655]: cluster 2021-07-31T03:13:15.027163+0000 mgr.x (mgr. 2021-07-31T03:13:16.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:16 smithi045 conmon[33655]: 24542) 3685 : cluster [DBG] pgmap v4588: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:16.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:16 smithi045 conmon[39718]: cluster 2021-07-31T03:13:15.027163+0000 mgr.x ( 2021-07-31T03:13:16.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:16 smithi045 conmon[39718]: mgr.24542) 3685 : cluster [DBG] pgmap v4588: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:18.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:18 smithi154 conmon[34553]: cluster 2021-07-31T03:13:17.027587+0000 mgr.x (mgr.24542) 3686 : cluster [DBG] pgmap v4589: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:18.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:18 smithi045 conmon[39718]: cluster 2021-07-31T03:13:17.027587+0000 mgr.x (mgr 2021-07-31T03:13:18.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:18 smithi045 conmon[39718]: .24542) 3686 : cluster [DBG] pgmap v4589: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:18.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:18 smithi045 conmon[33655]: cluster 2021-07-31T 2021-07-31T03:13:18.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:18 smithi045 conmon[33655]: 03:13:17.027587+0000 mgr.x (mgr.24542) 3686 : cluster [DBG] pgmap v4589: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:20.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:20 smithi154 conmon[34553]: cluster 2021-07-31T03:13:19.028213+0000 mgr.x (mgr.24542) 3687 : cluster [DBG] pgmap v4590: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:20.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:20 smithi154 conmon[34553]: 2021-07-31T03:13:20.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:20 smithi045 conmon[39718]: cluster 2021-07-31T03:13:19.028213+0000 mgr.x (mgr. 2021-07-31T03:13:20.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:20 smithi045 conmon[39718]: 24542) 3687 : cluster [DBG] pgmap v4590: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:20.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:20 smithi045 conmon[33655]: cluster 2021-07-31T03:13:19.028213+0000 mgr.x ( 2021-07-31T03:13:20.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:20 smithi045 conmon[33655]: mgr.24542) 3687 : cluster [DBG] pgmap v4590: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:22.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:22 smithi154 conmon[34553]: cluster 2021-07-31T03:13:21.028659+0000 mgr.x (mgr.24542) 3688 : cluster [DBG] 2021-07-31T03:13:22.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:22 smithi154 conmon[34553]: pgmap v4591: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:22.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:22 smithi045 conmon[39718]: cluster 2021-07-31T03:13:21.028659 2021-07-31T03:13:22.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:22 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 3688 : cluster [DBG] pgmap v4591: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:22.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:22 smithi045 conmon[33655]: cluster 2021-07-31T03:13:21.028659+0000 2021-07-31T03:13:22.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:22 smithi045 conmon[33655]: mgr.x (mgr.24542) 3688 : cluster [DBG] pgmap v4591: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:24.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:24 smithi154 conmon[34553]: cluster 2021-07-31T03:13:23.029187+0000 mgr.x (mgr.24542) 3689 : cluster 2021-07-31T03:13:24.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:24 smithi154 conmon[34553]: [DBG] pgmap v4592: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:24.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:24 smithi045 conmon[39718]: cluster 2021-07-31T03:13:23 2021-07-31T03:13:24.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:24 smithi045 conmon[39718]: .029187+0000 mgr.x (mgr.24542) 3689 : cluster [DBG] pgmap v4592: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:24.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:24 smithi045 conmon[33655]: cluster 2021-07-31T03:13:23.029187+0000 mgr.x (mgr.24542) 3689 2021-07-31T03:13:24.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:24 smithi045 conmon[33655]: : cluster [DBG] pgmap v4592: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:26.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:26 smithi154 conmon[34553]: cluster 2021-07-31T03:13:25.029624+0000 mgr.x (mgr.24542) 3690 : cluster [DBG] pgmap v4593: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:26.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:26 smithi045 conmon[33655]: cluster 2021-07-31T03:13:25.029624+0000 mgr.x (mgr.24542) 2021-07-31T03:13:26.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:26 smithi045 conmon[33655]: 3690 : cluster [DBG] pgmap v4593: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:26.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:26 smithi045 conmon[39718]: cluster 2021-07-31T03:13: 2021-07-31T03:13:26.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:26 smithi045 conmon[39718]: 25.029624+0000 mgr.x (mgr.24542) 3690 : cluster [DBG] pgmap v4593: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:28.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:28 smithi045 conmon[39718]: cluster 2021-07-31T03:13:27.030189+0000 mgr.x (mgr.24542) 3691 : cluster 2021-07-31T03:13:28.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:28 smithi045 conmon[39718]: [DBG] pgmap v4594: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:28.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:28 smithi045 conmon[39718]: audit 2021-07-31T03:13:28.276733+0000 mon.b (mon.2) 505 : audit 2021-07-31T03:13:28.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:28 smithi045 conmon[39718]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:13:28.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:28 smithi045 conmon[39718]: audit 2021-07-31T03 2021-07-31T03:13:28.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:28 smithi045 conmon[39718]: :13:28.277015+0000 mon.b (mon.2) 506 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:13:28.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:28 smithi045 conmon[39718]: audit 2021-07-31T03:13:28. 2021-07-31T03:13:28.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:28 smithi045 conmon[39718]: 277354+0000 mon.a (mon.0) 3269 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:13:28.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:28 smithi045 conmon[39718]: audit 2021-07-31T03:13:28 2021-07-31T03:13:28.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:28 smithi045 conmon[39718]: .277791+0000 mon.a (mon.0) 3270 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:13:28.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:28 smithi045 conmon[33655]: cluster 2021-07-31T03:13:27.030189+0000 mgr.x (mgr. 2021-07-31T03:13:28.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:28 smithi045 conmon[33655]: 24542) 3691 : cluster [DBG] pgmap v4594: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:28.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:28 smithi045 conmon[33655]: audit 2021-07-31T03:13:28.276733+0000 mon.b (mon.2) 505 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:13:28.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:28 smithi045 conmon[33655]: audit 2021-07-31T03:13:28.277015+0000 mon.b 2021-07-31T03:13:28.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:28 smithi045 conmon[33655]: (mon.2) 506 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:13:28.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:28 smithi045 conmon[33655]: audit 2021 2021-07-31T03:13:28.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:28 smithi045 conmon[33655]: -07-31T03:13:28.277354+0000 mon.a (mon. 2021-07-31T03:13:28.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:28 smithi045 conmon[33655]: 0) 3269 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:13:28.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:28 smithi045 conmon[33655]: audit 2021-07-31T03:13:28.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:28 smithi045 conmon[33655]: 2021-07-31T03:13:28.277791 2021-07-31T03:13:28.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:28 smithi045 conmon[33655]: +0000 mon.a (mon.0) 3270 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:13:29.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:28 smithi154 conmon[34553]: cluster 2021-07-31T03:13:27.030189+0000 mgr.x (mgr.24542) 3691 : cluster [DBG] pgmap v4594: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:29.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:28 smithi154 conmon[34553]: audit 2021-07-31T03:13:28.276733+0000 mon.b (mon.2) 505 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:13:29.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:28 smithi154 conmon[34553]: audit 2021-07-31T03:13:28.277015+0000 mon.b (mon.2) 506 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:13:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:28 smithi154 conmon[34553]: audit 2021-07-31T03:13:28.277354+0000 mon.a (mon.0 2021-07-31T03:13:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:28 smithi154 conmon[34553]: ) 3269 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:13:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:28 smithi154 conmon[34553]: audit 2021- 2021-07-31T03:13:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:28 smithi154 conmon[34553]: 07-31T03:13:28.277791+0000 2021-07-31T03:13:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:28 smithi154 conmon[34553]: mon.a (mon.0) 3270 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:13:29.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:28 smithi154 conmon[34553]: 2021-07-31T03:13:30.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:30 smithi045 conmon[39718]: cluster 2021-07-31T03:13:29.030755+0000 mgr.x (mgr.24542 2021-07-31T03:13:30.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:30 smithi045 conmon[39718]: ) 3692 : cluster [DBG] pgmap v4595: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:30.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:30 smithi045 conmon[33655]: cluster 2021-07-31T03:13:29.030755+0000 mgr.x (mgr.24542) 3692 : cluster 2021-07-31T03:13:30.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:30 smithi045 conmon[33655]: [DBG] pgmap v4595: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:31.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:30 smithi154 conmon[34553]: cluster 2021-07-31T03:13:29.030755+0000 mgr.x (mgr.24542) 3692 : cluster [DBG] 2021-07-31T03:13:31.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:30 smithi154 conmon[34553]: pgmap v4595: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:32.252 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:31 smithi154 conmon[57492]: debug 2021-07-31T03:13:31.983+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:13:32.252 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:13:32.253 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:13:32.253 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:31 smithi154 conmon[57492]: yield 2021-07-31T03:13:32.253 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:13:32.253 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:13:32.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:13:32.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:13:32.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:13:32.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:13:32.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:13:32.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:13:32.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:13:32.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:31 smithi154 conmon[57492]: task.result() 2021-07-31T03:13:32.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:13:32.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:31 smithi154 conmon[57492]: debug 2021-07-31T03:13:31.994+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:13:32.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:31 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:13:32.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:13:32.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:31 smithi154 conmon[57492]: yield 2021-07-31T03:13:32.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:31 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:13:32.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:31 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:13:32.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:13:32.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:31 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:13:32.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:13:32.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:31 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:13:32.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:13:32.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:31 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:13:32.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:31 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:13:32.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:31 smithi154 conmon[57492]: task.result() 2021-07-31T03:13:32.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:31 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:13:32.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: debug 2021-07-31T03:13:32.020+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:13:32.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:13:32.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:13:32.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: yield 2021-07-31T03:13:32.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:13:32.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:13:32.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:13:32.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:13:32.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:13:32.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:13:32.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:13:32.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:13:32.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:13:32.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: task.result() 2021-07-31T03:13:32.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:13:32.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: debug 2021-07-31T03:13:32.021+0000 7fcb93160700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:13:32.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:13:32.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:13:32.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: yield 2021-07-31T03:13:32.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:13:32.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:13:32.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:13:32.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:13:32.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:13:32.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:13:32.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:13:32.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:13:32.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:13:32.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: task.result() 2021-07-31T03:13:32.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:13:32.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: 2021-07-31T03:13:32.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:13:32.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: 2021-07-31T03:13:32.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:13:32.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:13:32.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:13:32.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:13:32.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:13:32.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:13:32.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:13:32.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:13:32.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:13:32.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:13:32.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:13:32.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:13:32.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:13:32.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:13:32.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:13:32.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:13:32.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:13:32.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:13:32.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: raise 2021-07-31T03:13:32.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:13:32.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:13:32.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:13:32.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:13:32.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:13:32.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: debug 2021-07-31T03:13:32.030+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:13:32.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:13:32.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:13:32.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: yield 2021-07-31T03:13:32.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:13:32.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:13:32.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:13:32.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:13:32.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:13:32.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:13:32.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:13:32.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:13:32.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:13:32.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: task.result() 2021-07-31T03:13:32.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:13:32.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: debug 2021-07-31T03:13:32.031+0000 7fcb9295f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:13:32.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:13:32.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:13:32.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: yield 2021-07-31T03:13:32.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:13:32.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:13:32.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:13:32.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:13:32.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:13:32.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:13:32.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:13:32.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:13:32.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:13:32.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: task.result() 2021-07-31T03:13:32.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:13:32.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: 2021-07-31T03:13:32.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:13:32.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: 2021-07-31T03:13:32.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:13:32.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:13:32.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:13:32.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:13:32.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:13:32.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:13:32.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:13:32.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:13:32.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:13:32.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:13:32.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:13:32.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:13:32.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:13:32.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:13:32.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:13:32.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:13:32.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:13:32.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:13:32.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: raise 2021-07-31T03:13:32.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:13:32.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:13:32.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:13:32.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:13:32.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:13:32.604 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:13:32.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:32 smithi045 conmon[39718]: cluster 2021-07-31T03:13:31.031187+0000 mgr.x ( 2021-07-31T03:13:32.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:32 smithi045 conmon[39718]: mgr.24542) 3693 : cluster [DBG] pgmap v4596: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:32.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:32 smithi045 conmon[39718]: audit 2021-07-31T03 2021-07-31T03:13:32.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:32 smithi045 conmon[39718]: :13:31.937878+0000 mon.b (mon.2) 507 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:13:32.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:32 smithi045 conmon[39718]: audit 2021-07 2021-07-31T03:13:32.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:32 smithi045 conmon[39718]: -31T03:13:31.939329+0000 mon.b (mon 2021-07-31T03:13:32.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:32 smithi045 conmon[39718]: .2) 508 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:13:32.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:32 smithi045 conmon[39718]: audit 2021-07-31T03 2021-07-31T03:13:32.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:32 smithi045 conmon[39718]: :13:31.940257+0000 mon.b (mon.2) 509 2021-07-31T03:13:32.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:32 smithi045 conmon[39718]: : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:13:32.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:32 smithi045 conmon[33655]: cluster 2021-07-31T03:13:31.031187+0000 mgr.x (mgr. 2021-07-31T03:13:32.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:32 smithi045 conmon[33655]: 24542) 3693 : cluster [DBG] pgmap v4596: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:32.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:32 smithi045 conmon[33655]: audit 2021-07-31T03:13:31.937878+0000 mon.b (mon.2) 507 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:13:32.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:32 smithi045 conmon[33655]: audit 2021-07-31T03:13:31.939329+0000 mon.b 2021-07-31T03:13:32.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:32 smithi045 conmon[33655]: (mon.2) 508 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:13:32.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:32 smithi045 conmon[33655]: audit 2021-07-31T03:13:32.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:32 smithi045 conmon[33655]: 2021-07-31T03:13:31.940257+0000 2021-07-31T03:13:32.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:32 smithi045 conmon[33655]: mon.b (mon.2) 509 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:13:32.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:32 smithi045 conmon[33655]: 2021-07-31T03:13:32.881 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:13:32.881 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 88m ago 89m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:13:32.881 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 86m ago 88m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:13:32.881 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 86m ago 92m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:13:32.881 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 88m ago 94m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:13:32.882 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 88m ago 94m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:13:32.882 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 86m ago 92m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:13:32.882 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 88m ago 92m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:13:32.882 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 88m ago 89m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:13:32.883 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 86m ago 89m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:13:32.883 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 88m ago 92m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:13:32.883 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 88m ago 91m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:13:32.883 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 88m ago 91m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:13:32.883 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 88m ago 91m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:13:32.884 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 86m ago 90m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:13:32.884 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 86m ago 90m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:13:32.884 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 86m ago 90m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:13:32.884 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 86m ago 89m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:13:32.884 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 86m ago 89m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:13:33.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[34553]: cluster 2021-07-31T03:13:31.031187+0000 mgr.x (mgr.24542) 3693 : cluster [DBG] pgmap v4596: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:33.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[34553]: audit 2021-07-31T03:13:31. 2021-07-31T03:13:33.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[34553]: 937878+0000 mon.b (mon.2) 507 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:13:33.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[34553]: audit 2021-07-31T03:13:31.939329+0000 mon.b (mon.2) 508 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:13:33.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[34553]: audit 2021-07-31T03:13:31 2021-07-31T03:13:33.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:32 smithi154 conmon[34553]: .940257+0000 mon.b (mon.2) 509 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:13:33.217 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:13:33.218 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:13:33.218 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:13:33.218 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:13:33.218 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:13:33.218 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:13:33.219 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:13:33.219 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:13:33.219 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:13:33.219 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:13:33.220 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:13:33.220 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:13:33.220 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:13:33.220 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:13:33.220 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:13:33.221 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:13:33.221 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:13:33.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: cephadm 2021-07-31T03:13: 2021-07-31T03:13:33.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: 31.940989+0000 mgr.x (mgr.24542) 3694 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:13:33.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: cephadm 2021-07-31T03:13:33.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: 2021-07-31T03: 2021-07-31T03:13:33.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: 13:31.984529+0000 mgr.x 2021-07-31T03:13:33.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: (mgr.24542) 3695 : cephadm 2021-07-31T03:13:33.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: [ERR] 2021-07-31T03:13:33.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:13:33.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:13:33.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: yield 2021-07-31T03:13:33.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:13:33.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:13:33.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:13:33.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:13:33.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:13:33.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:13:33.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:13:33.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:13:33.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:13:33.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: task.result() 2021-07-31T03:13:33.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:13:33.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: cephadm 2021-07-31T03:13:31.985001+0000 mgr.x (mgr.24542) 3696 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:13:33.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: cephadm 2021-07-31T03:13:33.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: 2021-07-31T03 2021-07-31T03:13:33.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: :13:31.995923+0000 2021-07-31T03:13:33.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: mgr.x (mgr.24542) 3697 2021-07-31T03:13:33.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: : cephadm [ERR] 2021-07-31T03:13:33.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:13:33.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:13:33.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: yield 2021-07-31T03:13:33.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:13:33.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:13:33.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:13:33.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:13:33.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:13:33.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:13:33.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:13:33.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:13:33.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:13:33.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: task.result() 2021-07-31T03:13:33.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:13:33.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: cephadm 2021-07-31T03:13:31.996911+0000 mgr.x (mgr.24542) 3698 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:13:33.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: cephadm 2021- 2021-07-31T03:13:33.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: 07-31T03:13:32 2021-07-31T03:13:33.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: .021683+0000 mgr.x (mgr. 2021-07-31T03:13:33.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: 24542) 3699 : cephadm [ERR] 2021-07-31T03:13:33.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: 2021-07-31T03:13:33.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:13:33.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:13:33.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: yield 2021-07-31T03:13:33.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:13:33.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:13:33.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:13:33.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:13:33.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:13:33.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:13:33.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:13:33.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:13:33.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:13:33.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: task.result() 2021-07-31T03:13:33.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:13:33.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: cephadm 2021-07-31T03:13:32.022168+0000 mgr.x (mgr.24542) 3700 : cephadm [ERR] 2021-07-31T03:13:33.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:13:33.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:13:33.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:13:33.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: yield 2021-07-31T03:13:33.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:13:33.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:13:33.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:13:33.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:13:33.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:13:33.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:13:33.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:13:33.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:13:33.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:13:33.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: task.result() 2021-07-31T03:13:33.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:13:33.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: 2021-07-31T03:13:33.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:13:33.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: 2021-07-31T03:13:33.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:13:33.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:13:33.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:13:33.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:13:33.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:13:33.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:13:33.794 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: cephadm 2021-07-31T03:13:31.940989+0000 mgr.x (mgr.24542) 3694 : 2021-07-31T03:13:33.794 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:13:33.794 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: cephadm 2021-07-31T03:13:31.984529+0000 mgr.x (mgr.24542) 3695 : cephadm [ERR] 2021-07-31T03:13:33.794 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:13:33.795 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:13:33.795 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: yield 2021-07-31T03:13:33.795 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:13:33.795 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:13:33.795 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:13:33.796 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:13:33.796 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:13:33.796 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:13:33.796 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:13:33.796 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:13:33.797 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:13:33.797 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: task.result() 2021-07-31T03:13:33.797 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:13:33.797 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: cephadm 2021-07-31T03:13:31.985001+0000 mgr.x (mgr.24542) 3696 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:13:33.797 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: cephadm 2021- 2021-07-31T03:13:33.798 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: 07-31T03:13:31.995923+0000 2021-07-31T03:13:33.798 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: mgr.x (mgr.24542) 3697 2021-07-31T03:13:33.798 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: : cephadm [ERR] 2021-07-31T03:13:33.798 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:13:33.799 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:13:33.799 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: yield 2021-07-31T03:13:33.799 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:13:33.799 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:13:33.799 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:13:33.800 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:13:33.800 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:13:33.800 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:13:33.800 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:13:33.800 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:13:33.801 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:13:33.801 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: task.result() 2021-07-31T03:13:33.801 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:13:33.801 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: cephadm 2021-07-31T03:13:31.996911+0000 mgr.x (mgr.24542) 3698 : 2021-07-31T03:13:33.801 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:13:33.802 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: cephadm 2021 2021-07-31T03:13:33.802 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: -07-31T03:13:32 2021-07-31T03:13:33.802 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: .021683+0000 mgr.x (mgr 2021-07-31T03:13:33.803 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:13:33.803 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:13:33.803 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:13:33.803 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:13:33.803 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:13:33.804 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:13:33.804 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:13:33.804 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:13:33.804 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:13:33.804 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:13:33.805 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:13:33.805 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:13:33.805 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: raise 2021-07-31T03:13:33.805 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:13:33.805 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:13:33.806 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:13:33.806 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:13:33.806 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:13:33.806 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: cephadm 2021-07-31T03:13:32.031628+0000 mgr.x (mgr.24542) 3701 : cephadm [ERR] 2021-07-31T03:13:33.806 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:13:33.807 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:13:33.807 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: yield 2021-07-31T03:13:33.807 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:13:33.807 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:13:33.807 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:13:33.808 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:13:33.808 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:13:33.808 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:13:33.808 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:13:33.808 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:13:33.809 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:13:33.809 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: task.result() 2021-07-31T03:13:33.809 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:13:33.809 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: cephadm 2021-07-31T03:13:32.032204+0000 mgr.x (mgr.24542) 3702 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:13:33.809 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:13:33.810 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:13:33.810 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: yield 2021-07-31T03:13:33.810 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:13:33.810 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: .24542) 3699 : cephadm [ERR] 2021-07-31T03:13:33.811 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: 2021-07-31T03:13:33.811 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:13:33.811 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:13:33.811 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: yield 2021-07-31T03:13:33.811 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:13:33.812 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:13:33.812 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:13:33.812 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:13:33.812 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:13:33.813 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:13:33.813 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:13:33.813 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:13:33.813 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:13:33.813 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: task.result() 2021-07-31T03:13:33.814 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:13:33.814 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: cephadm 2021-07-31T03:13:32. 2021-07-31T03:13:33.814 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: 022168+0000 mgr.x (mgr.24542) 3700 : 2021-07-31T03:13:33.814 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:13:33.814 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:13:33.815 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:13:33.815 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: yield 2021-07-31T03:13:33.815 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:13:33.815 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:13:33.815 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:13:33.815 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:13:33.816 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:13:33.816 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:13:33.816 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:13:33.816 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:13:33.816 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:13:33.817 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: task.result() 2021-07-31T03:13:33.817 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:13:33.817 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: 2021-07-31T03:13:33.817 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:13:33.817 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: 2021-07-31T03:13:33.818 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:13:33.818 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:13:33.818 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:13:33.818 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:13:33.818 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:13:33.819 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:13:33.819 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:13:33.819 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:13:33.819 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:13:33.820 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:13:33.820 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:13:33.820 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:13:33.820 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:13:33.820 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:13:33.821 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:13:33.821 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: task.result() 2021-07-31T03:13:33.821 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:13:33.821 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: 2021-07-31T03:13:33.821 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:13:33.821 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: 2021-07-31T03:13:33.822 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:13:33.822 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:13:33.822 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:13:33.822 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:13:33.822 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:13:33.823 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:13:33.823 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:13:33.823 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:13:33.823 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:13:33.823 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:13:33.824 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:13:33.824 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:13:33.824 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:13:33.824 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:13:33.824 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:13:33.825 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:13:33.825 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:13:33.825 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:13:33.825 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: raise 2021-07-31T03:13:33.825 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:13:33.825 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:13:33.826 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:13:33.826 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:13:33.826 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:13:33.826 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: audit 2021- 2021-07-31T03:13:33.826 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: 07-31T03:13:32.591798+0000 mgr.x (mgr.24542) 3703 : audit [DBG] from='client.19056 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:13:33.827 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[39718]: audit 2021-07-31T03:13:33.218199+0000 mon.c (mon.1) 187 : audit [DBG] from='client.? 172.21.15.45:0/3727965459' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:13:33.827 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:13:33.827 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:13:33.827 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:13:33.828 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:13:33.828 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:13:33.828 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:13:33.828 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:13:33.828 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:13:33.829 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:13:33.829 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:13:33.829 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:13:33.829 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: raise 2021-07-31T03:13:33.829 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:13:33.830 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:13:33.830 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:13:33.830 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:13:33.830 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:13:33.830 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: cephadm 2021-07-31T03:13:32.031628+0000 mgr.x (mgr.24542) 3701 : cephadm [ERR] 2021-07-31T03:13:33.831 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:13:33.831 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:13:33.831 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: yield 2021-07-31T03:13:33.831 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:13:33.831 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:13:33.832 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:13:33.832 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:13:33.832 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:13:33.832 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:13:33.832 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:13:33.833 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:13:33.833 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:13:33.833 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: task.result() 2021-07-31T03:13:33.833 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:13:33.833 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: cephadm 2021-07-31T03:13:32.032204+0000 mgr.x (mgr.24542) 3702 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:13:33.834 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:13:33.834 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:13:33.834 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: yield 2021-07-31T03:13:33.834 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:13:33.835 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:13:33.835 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:13:33.835 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:13:33.835 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:13:33.835 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:13:33.836 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:13:33.836 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:13:33.836 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:13:33.836 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: task.result() 2021-07-31T03:13:33.836 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:13:33.836 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: 2021-07-31T03:13:33.837 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:13:33.837 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: 2021-07-31T03:13:33.837 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:13:33.837 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:13:33.837 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:13:33.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:13:33.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:13:33.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:13:33.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:13:33.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:13:33.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:13:33.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:13:33.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:13:33.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:13:33.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:13:33.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:13:33.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:13:33.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:13:33.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:13:33.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:13:33.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: raise 2021-07-31T03:13:33.841 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:13:33.841 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:13:33.841 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:13:33.841 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:13:33.841 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:13:33.842 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: audit 2021-07-31T03:13:32.591798+0000 mgr.x 2021-07-31T03:13:33.842 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: (mgr.24542) 3703 : audit [DBG] from='client.19056 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:13:33.842 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:33 smithi045 conmon[33655]: audit 2021-07-31T03:13:33.218199+0000 mon.c (mon.1) 187 : audit [DBG] from='client.? 172.21.15.45:0/3727965459' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:13:34.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:13:31.940989+0000 mgr.x (mgr.24542) 3694 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:13:34.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:13:31.984529+0000 mgr.x (mgr.24542) 3695 : cephadm [ERR] 2021-07-31T03:13:34.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:13:34.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:13:34.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: yield 2021-07-31T03:13:34.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:13:34.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:13:34.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:13:34.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:13:34.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:13:34.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:13:34.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:13:34.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:13:34.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:13:34.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: task.result() 2021-07-31T03:13:34.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:13:34.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:13:31.985001+0000 mgr.x (mgr.24542) 3696 : cephadm 2021-07-31T03:13:34.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:13:34.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:13:31.995923+0000 mgr.x (mgr.24542) 3697 : cephadm [ERR] 2021-07-31T03:13:34.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:13:34.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:13:34.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: yield 2021-07-31T03:13:34.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:13:34.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:13:34.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:13:34.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:13:34.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:13:34.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:13:34.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:13:34.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:13:34.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:13:34.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: task.result() 2021-07-31T03:13:34.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:13:34.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:13:31.996911+0000 mgr.x (mgr.24542) 3698 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:13:34.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:13:32.021683+0000 mgr.x (mgr.24542) 3699 : cephadm [ERR] 2021-07-31T03:13:34.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:13:34.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:13:34.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: yield 2021-07-31T03:13:34.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:13:34.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:13:34.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:13:34.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:13:34.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:13:34.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:13:34.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:13:34.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:13:34.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:13:34.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: task.result() 2021-07-31T03:13:34.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:13:34.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:13:32.022168+0000 mgr.x (mgr.24542) 3700 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:13:34.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:13:34.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:13:34.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: yield 2021-07-31T03:13:34.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:13:34.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:13:34.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:13:34.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:13:34.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:13:34.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:13:34.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:13:34.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:13:34.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:13:34.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: task.result() 2021-07-31T03:13:34.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:13:34.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: 2021-07-31T03:13:34.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:13:34.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: 2021-07-31T03:13:34.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:13:34.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:13:34.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:13:34.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:13:34.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:13:34.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:13:34.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:13:34.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:13:34.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:13:34.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:13:34.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:13:34.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:13:34.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:13:34.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:13:34.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:13:34.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:13:34.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:13:34.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:13:34.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: raise 2021-07-31T03:13:34.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:13:34.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:13:34.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:13:34.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:13:34.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:13:34.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:13:32.031628+0000 mgr.x (mgr.24542) 3701 : cephadm [ERR] 2021-07-31T03:13:34.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:13:34.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:13:34.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: yield 2021-07-31T03:13:34.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:13:34.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:13:34.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:13:34.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:13:34.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:13:34.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:13:34.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:13:34.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:13:34.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:13:34.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: task.result() 2021-07-31T03:13:34.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:13:34.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:13:32.032204+0000 mgr.x (mgr.24542) 3702 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:13:34.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:13:34.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:13:34.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: yield 2021-07-31T03:13:34.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:13:34.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:13:34.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:13:34.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:13:34.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:13:34.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:13:34.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:13:34.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:13:34.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:13:34.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: task.result() 2021-07-31T03:13:34.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:13:34.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: 2021-07-31T03:13:34.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:13:34.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: 2021-07-31T03:13:34.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:13:34.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:13:34.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:13:34.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:13:34.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:13:34.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:13:34.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:13:34.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:13:34.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:13:34.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:13:34.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:13:34.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:13:34.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:13:34.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:13:34.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:13:34.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:13:34.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:13:34.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:13:34.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: raise 2021-07-31T03:13:34.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:13:34.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:13:34.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:13:34.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:13:34.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:13:34.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: audit 2021-07-31T03:13:32.591798+0000 mgr.x (mgr.24542) 3703 : audit [DBG] from='client.19056 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:13:34.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:33 smithi154 conmon[34553]: audit 2021-07-31T03:13:33.218199+0000 mon.c (mon.1) 187 : audit [DBG] from='client.? 172.21.15.45:0/3727965459' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:13:34.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:34 smithi045 conmon[33655]: audit 2021-07-31T03:13: 2021-07-31T03:13:34.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:34 smithi045 conmon[33655]: 32.873699+0000 mgr.x (mgr.24542) 3704 : audit [DBG] from='client.19062 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:13:34.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:34 smithi045 conmon[33655]: cluster 2021-07-31T03:13:34.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:34 smithi045 conmon[33655]: 2021-07-31T03:13:33.031697+0000 mgr.x (mgr.24542) 3705 : cluster [DBG] pgmap v4597: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:34.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:34 smithi045 conmon[33655]: 2021-07-31T03:13:34.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:34 smithi045 conmon[39718]: audit 2021-07-31T03:13:32. 2021-07-31T03:13:34.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:34 smithi045 conmon[39718]: 873699+0000 mgr.x (mgr.24542) 3704 : audit [DBG] from='client.19062 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:13:34.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:34 smithi045 conmon[39718]: cluster 2021 2021-07-31T03:13:34.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:34 smithi045 conmon[39718]: -07-31T03:13:33 2021-07-31T03:13:34.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:34 smithi045 conmon[39718]: .031697+0000 mgr.x (mgr. 2021-07-31T03:13:34.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:34 smithi045 conmon[39718]: 24542) 3705 : cluster [DBG] pgmap v4597: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:34.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:34 smithi045 conmon[39718]: 2021-07-31T03:13:35.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:34 smithi154 conmon[34553]: audit 2021-07-31T03:13:32.873699+0000 mgr.x (mgr.24542) 3704 : audit [DBG] from='client.19062 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:13:35.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:34 smithi154 conmon[34553]: cluster 2021-07-31T03:13:33.031697+0000 mgr.x (mgr.24542) 3705 : cluster [DBG] pgmap v4597: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:36.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:36 smithi045 conmon[33655]: cluster 2021-07-31T03:13:35.032205+0000 mgr.x (mgr.24542) 3706 : cluster 2021-07-31T03:13:36.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:36 smithi045 conmon[33655]: [DBG] pgmap v4598: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:36.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:36 smithi045 conmon[39718]: cluster 2021-07-31T03:13:35.032205+0000 mgr.x 2021-07-31T03:13:36.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:36 smithi045 conmon[39718]: (mgr.24542) 3706 : cluster [DBG] pgmap v4598: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:37.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:36 smithi154 conmon[34553]: cluster 2021-07-31T03:13:35.032205+0000 mgr.x (mgr.24542) 3706 : cluster 2021-07-31T03:13:37.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:36 smithi154 conmon[34553]: [DBG] pgmap v4598: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:38.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:38 smithi045 conmon[33655]: cluster 2021-07-31T03:13:37.032644+0000 mgr.x (mgr.24542) 3707 : cluster [DBG] pgmap v4599: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:38.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:38 smithi045 conmon[39718]: cluster 2021-07-31T03:13:37.032644+0000 mgr.x (mgr. 2021-07-31T03:13:38.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:38 smithi045 conmon[39718]: 24542) 3707 : cluster [DBG] pgmap v4599: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:39.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:38 smithi154 conmon[34553]: cluster 2021-07-31T03:13:37.032644+0000 mgr.x (mgr.24542) 3707 : cluster [DBG] pgmap v4599: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:41.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:40 smithi154 conmon[34553]: cluster 2021-07-31T03:13:39.033149+0000 mgr.x (mgr.24542) 3708 : cluster [DBG] pgmap v4600: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:41.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:40 smithi045 conmon[33655]: cluster 2021-07-31T03:13:39.033149+0000 mgr.x (mgr.24542) 3708 : cluster [DBG] pgmap v4600: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:41.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:40 smithi045 conmon[39718]: cluster 2021-07-31T03:13:39.033149+0000 mgr.x (mgr.24542) 3708 : cluster [DBG] 2021-07-31T03:13:41.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:40 smithi045 conmon[39718]: pgmap v4600: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:43.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:42 smithi154 conmon[34553]: cluster 2021-07-31T03:13:41.033653+0000 mgr.x (mgr.24542) 3709 : cluster [DBG] pgmap v4601: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:43.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:42 smithi045 conmon[33655]: cluster 2021-07-31T03:13:41.033653+0000 mgr.x (mgr.24542) 3709 2021-07-31T03:13:43.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:42 smithi045 conmon[33655]: : cluster [DBG] pgmap v4601: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:43.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:42 smithi045 conmon[39718]: cluster 2021-07-31T03:13:41.033653+0000 mgr.x (mgr. 2021-07-31T03:13:43.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:42 smithi045 conmon[39718]: 24542) 3709 : cluster [DBG] pgmap v4601: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:45.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:44 smithi154 conmon[34553]: cluster 2021-07-31T03:13:43.034163+0000 mgr.x (mgr.24542) 3710 : cluster [DBG] 2021-07-31T03:13:45.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:44 smithi154 conmon[34553]: pgmap v4602: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:45.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:44 smithi045 conmon[39718]: cluster 2021-07-31T03:13:43.034163 2021-07-31T03:13:45.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:44 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 3710 : cluster [DBG] pgmap v4602: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:45.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:44 smithi045 conmon[33655]: cluster 2021-07-31T03:13:43.034163+0000 mgr.x (mgr.24542) 3710 : cluster [DBG] 2021-07-31T03:13:45.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:44 smithi045 conmon[33655]: pgmap v4602: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:47.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:46 smithi154 conmon[34553]: cluster 2021-07-31T03:13:45.034638+0000 mgr.x (mgr.24542) 3711 : cluster [DBG] pgmap v4603: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:47.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:46 smithi045 conmon[33655]: cluster 2021-07-31T03:13:45.034638+0000 mgr.x (mgr.24542) 3711 : 2021-07-31T03:13:47.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:46 smithi045 conmon[33655]: cluster [DBG] pgmap v4603: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:47.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:46 smithi045 conmon[39718]: cluster 2021-07-31T03:13:45.034638+0000 mgr.x (mgr 2021-07-31T03:13:47.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:46 smithi045 conmon[39718]: .24542) 3711 : cluster [DBG] pgmap v4603: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:49.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:48 smithi154 conmon[34553]: cluster 2021-07-31T03:13:47.035084+0000 mgr.x (mgr.24542) 3712 : cluster [DBG] pgmap v4604: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:49.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:48 smithi045 conmon[39718]: cluster 2021-07-31T03:13:47.035084+0000 mgr.x ( 2021-07-31T03:13:49.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:48 smithi045 conmon[39718]: mgr.24542) 3712 : cluster [DBG] pgmap v4604: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:49.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:48 smithi045 conmon[33655]: cluster 2021-07-31T03:13:47.035084+0000 mgr.x ( 2021-07-31T03:13:49.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:48 smithi045 conmon[33655]: mgr.24542) 3712 : cluster [DBG] pgmap v4604: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:51.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:50 smithi154 conmon[34553]: cluster 2021-07-31T03:13:49.035684+0000 mgr.x (mgr.24542) 3713 : cluster [DBG] pgmap v4605: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:51.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:50 smithi045 conmon[33655]: cluster 2021-07-31T03:13:49.035684+0000 mgr.x 2021-07-31T03:13:51.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:50 smithi045 conmon[33655]: (mgr.24542) 3713 : cluster [DBG] pgmap v4605: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:51.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:50 smithi045 conmon[39718]: cluster 2021-07-31T03:13:49.035684+0000 mgr.x 2021-07-31T03:13:51.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:50 smithi045 conmon[39718]: (mgr.24542) 3713 : cluster [DBG] pgmap v4605: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:53.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:52 smithi154 conmon[34553]: cluster 2021-07-31T03:13:51.036088+0000 mgr.x (mgr.24542) 3714 : cluster [DBG] pgmap v4606: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:53.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:52 smithi045 conmon[39718]: cluster 2021-07-31T03:13:51.036088+0000 mgr.x ( 2021-07-31T03:13:53.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:52 smithi045 conmon[39718]: mgr.24542) 3714 : cluster [DBG] pgmap v4606: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:53.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:52 smithi045 conmon[33655]: cluster 2021-07-31T03:13:51.036088+0000 mgr.x (mgr.24542) 2021-07-31T03:13:53.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:52 smithi045 conmon[33655]: 3714 : cluster [DBG] pgmap v4606: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:55.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:54 smithi154 conmon[34553]: cluster 2021-07-31T03:13:53.036505+0000 mgr.x (mgr.24542) 3715 : cluster [DBG] pgmap v4607: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:55.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:54 smithi045 conmon[39718]: cluster 2021-07-31T03:13:53.036505+0000 2021-07-31T03:13:55.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:54 smithi045 conmon[39718]: mgr.x (mgr.24542) 3715 : cluster [DBG] pgmap v4607: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:55.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:54 smithi045 conmon[33655]: cluster 2021-07-31T03:13:53.036505+0000 mgr.x (mgr.24542) 3715 : cluster [DBG] 2021-07-31T03:13:55.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:54 smithi045 conmon[33655]: pgmap v4607: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:57.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:56 smithi154 conmon[34553]: cluster 2021-07-31T03:13:55.037133+0000 mgr.x (mgr.24542) 3716 : cluster [DBG] pgmap v4608: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:57.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:56 smithi045 conmon[39718]: cluster 2021-07-31T03:13:55.037133+0000 mgr.x (mgr 2021-07-31T03:13:57.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:56 smithi045 conmon[39718]: .24542) 3716 : cluster [DBG] pgmap v4608: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:57.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:56 smithi045 conmon[33655]: cluster 2021-07-31T03:13:55.037133+0000 mgr.x (mgr.24542) 3716 : cluster [DBG] pgmap v4608: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:59.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:58 smithi154 conmon[34553]: cluster 2021-07-31T03:13:57.037585+0000 mgr.x (mgr.24542) 3717 : cluster [DBG] pgmap v4609: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:59.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:13:58 smithi154 conmon[34553]: 2021-07-31T03:13:59.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:58 smithi045 conmon[39718]: cluster 2021-07-31T03:13:57.037585+0000 mgr.x 2021-07-31T03:13:59.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:13:58 smithi045 conmon[39718]: (mgr.24542) 3717 : cluster [DBG] pgmap v4609: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:13:59.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:58 smithi045 conmon[33655]: cluster 2021-07-31T03:13:57.037585+0000 mgr.x (mgr.24542) 3717 2021-07-31T03:13:59.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:13:58 smithi045 conmon[33655]: : cluster [DBG] pgmap v4609: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:01.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:00 smithi154 conmon[34553]: cluster 2021-07-31T03:13:59.038053+0000 mgr.x (mgr.24542) 3718 : cluster [DBG] pgmap v4610: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:01.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:00 smithi045 conmon[33655]: cluster 2021-07-31T03:14:01.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:00 smithi045 conmon[33655]: 2021-07-31T03:13:59.038053+0000 mgr.x (mgr.24542) 3718 : cluster [DBG] pgmap v4610: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:01.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:00 smithi045 conmon[39718]: cluster 2021-07-31T03:13:59.038053+0000 mgr.x (mgr.24542 2021-07-31T03:14:01.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:00 smithi045 conmon[39718]: ) 3718 : cluster [DBG] pgmap v4610: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:03.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:02 smithi154 conmon[34553]: cluster 2021-07-31T03:14:01.038559+0000 mgr.x (mgr.24542) 3719 : cluster [DBG] 2021-07-31T03:14:03.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:02 smithi154 conmon[34553]: pgmap v4611: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:03.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:02 smithi045 conmon[39718]: cluster 2021-07-31T03:14:01.038559+0000 mgr.x 2021-07-31T03:14:03.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:02 smithi045 conmon[39718]: (mgr.24542) 3719 : cluster [DBG] pgmap v4611: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:03.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:02 smithi045 conmon[33655]: cluster 2021-07-31T03:14:01.038559+0000 mgr.x (mgr.24542) 3719 : cluster [DBG] pgmap v4611: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:03.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:02 smithi045 conmon[33655]: 2021-07-31T03:14:03.531 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:14:03.808 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:14:03.808 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 88m ago 89m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:14:03.808 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 86m ago 89m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:14:03.809 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 86m ago 92m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:14:03.809 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 88m ago 95m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:14:03.809 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 88m ago 95m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:14:03.809 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 86m ago 93m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:14:03.809 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 88m ago 93m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:14:03.810 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 88m ago 89m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:14:03.810 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 86m ago 89m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:14:03.810 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 88m ago 92m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:14:03.810 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 88m ago 92m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:14:03.810 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 88m ago 92m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:14:03.811 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 88m ago 91m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:14:03.811 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 86m ago 91m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:14:03.811 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 86m ago 91m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:14:03.811 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 86m ago 90m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:14:03.811 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 86m ago 90m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:14:03.812 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 86m ago 90m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:14:04.143 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:14:04.144 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:14:04.144 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:14:04.144 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:14:04.144 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:14:04.145 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:14:04.145 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:14:04.145 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:14:04.145 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:14:04.145 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:14:04.146 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:14:04.146 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:14:04.146 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:14:04.146 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:14:04.146 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:14:04.147 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:14:04.147 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:14:05.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:04 smithi154 conmon[34553]: cluster 2021-07-31T03:14:03.039033+0000 mgr.x (mgr.24542) 3720 : cluster [DBG] pgmap v4612: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:05.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:04 smithi154 conmon[34553]: audit 2021-07-31T03:14:03.519573+0000 mgr.x (mgr.24542) 3721 : audit [DBG] from='client.29014 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:14:05.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:04 smithi154 conmon[34553]: audit 2021-07-31T03:14:03.801105+0000 mgr.x (mgr.24542) 3722 : audit [DBG] from='client.19080 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:14:05.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:04 smithi154 conmon[34553]: audit 2021-07-31T03:14:05.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:04 smithi154 conmon[34553]: 2021-07-31T03:14:04.144550+0000 mon.c (mon.1) 188 : audit [DBG] from='client.? 172.21.15.45:0/369861243' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:14:05.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:04 smithi045 conmon[39718]: cluster 2021-07-31T03:14:03.039033+0000 mgr.x (mgr.24542) 3720 2021-07-31T03:14:05.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:04 smithi045 conmon[39718]: : cluster [DBG] pgmap v4612: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:05.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:04 smithi045 conmon[39718]: audit 2021-07-31T03:14:03.519573+0000 mgr.x (mgr.24542) 3721 : audit [DBG] 2021-07-31T03:14:05.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:04 smithi045 conmon[39718]: from='client.29014 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:14:05.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:04 smithi045 conmon[39718]: audit 2021-07-31T03:14:03.801105+0000 mgr.x (mgr 2021-07-31T03:14:05.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:04 smithi045 conmon[39718]: .24542) 3722 : audit [DBG] from='client.19080 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:14:05.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:04 smithi045 conmon[39718]: audit 2021-07 2021-07-31T03:14:05.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:04 smithi045 conmon[39718]: -31T03:14:04.144550+0000 mon.c (mon.1) 188 : audit [DBG] from='client.? 172.21.15.45:0/369861243' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:14:05.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:04 smithi045 conmon[33655]: cluster 2021-07-31T03:14:03 2021-07-31T03:14:05.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:04 smithi045 conmon[33655]: .039033+0000 mgr.x (mgr.24542) 3720 : cluster [DBG] pgmap v4612: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:05.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:04 smithi045 conmon[33655]: audit 2021-07-31T03 2021-07-31T03:14:05.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:04 smithi045 conmon[33655]: :14:03.519573+0000 mgr.x (mgr.24542) 3721 : audit [DBG] from='client.29014 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:14:05.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:04 smithi045 conmon[33655]: audit 2021-07-31T03:14:03. 2021-07-31T03:14:05.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:04 smithi045 conmon[33655]: 801105+0000 mgr.x (mgr.24542) 3722 : audit [DBG] from='client.19080 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:14:05.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:04 smithi045 conmon[33655]: audit 2021-07-31T03:14:04.144550+0000 mon.c (mon.1) 2021-07-31T03:14:05.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:04 smithi045 conmon[33655]: 188 : audit [DBG] from='client.? 172.21.15.45:0/369861243' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:14:07.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:06 smithi154 conmon[34553]: cluster 2021-07-31T03:14:05.039648+0000 mgr.x (mgr.24542) 3723 : cluster [DBG] 2021-07-31T03:14:07.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:06 smithi154 conmon[34553]: pgmap v4613: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:07.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:06 smithi045 conmon[39718]: cluster 2021-07-31T03:14:05.039648+0000 mgr.x 2021-07-31T03:14:07.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:06 smithi045 conmon[39718]: (mgr.24542) 3723 : cluster [DBG] pgmap v4613: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:07.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:06 smithi045 conmon[33655]: cluster 2021-07-31T03:14:05.039648+0000 mgr.x (mgr.24542) 3723 : cluster [DBG] 2021-07-31T03:14:07.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:06 smithi045 conmon[33655]: pgmap v4613: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:09.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:08 smithi154 conmon[34553]: cluster 2021-07-31T03:14:07.040149+0000 mgr.x (mgr.24542) 3724 : cluster [DBG] pgmap v4614: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:09.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:08 smithi045 conmon[33655]: cluster 2021-07-31T03:14:07. 2021-07-31T03:14:09.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:08 smithi045 conmon[33655]: 040149+0000 mgr.x (mgr.24542) 3724 : cluster [DBG] pgmap v4614: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:09.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:08 smithi045 conmon[39718]: cluster 2021-07-31T03:14:07. 2021-07-31T03:14:09.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:08 smithi045 conmon[39718]: 040149+0000 mgr.x (mgr.24542) 3724 : cluster [DBG] pgmap v4614: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:11.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:10 smithi154 conmon[34553]: cluster 2021-07-31T03:14:09.040685+0000 mgr.x (mgr.24542) 3725 : cluster [DBG] pgmap v4615: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:11.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:10 smithi045 conmon[33655]: cluster 2021-07-31T03:14:09.040685+0000 mgr.x (mgr.24542) 3725 2021-07-31T03:14:11.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:10 smithi045 conmon[33655]: : cluster [DBG] pgmap v4615: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:11.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:10 smithi045 conmon[39718]: cluster 2021-07-31T03:14:09.040685+0000 mgr.x ( 2021-07-31T03:14:11.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:10 smithi045 conmon[39718]: mgr.24542) 3725 : cluster [DBG] pgmap v4615: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:13.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:12 smithi154 conmon[34553]: cluster 2021-07-31T03:14:11.041107+0000 mgr.x (mgr.24542) 3726 : cluster [DBG] pgmap v4616: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:13.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:12 smithi045 conmon[33655]: cluster 2021-07-31T03:14:11.041107+0000 mgr.x (mgr.24542) 3726 2021-07-31T03:14:13.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:12 smithi045 conmon[33655]: : cluster [DBG] pgmap v4616: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:13.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:12 smithi045 conmon[39718]: cluster 2021-07-31T03:14:11.041107+0000 mgr.x (mgr 2021-07-31T03:14:13.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:12 smithi045 conmon[39718]: .24542) 3726 : cluster [DBG] pgmap v4616: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:15.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:14 smithi154 conmon[34553]: cluster 2021-07-31T03:14:13.041504+0000 mgr.x (mgr.24542) 3727 : cluster [DBG] pgmap v4617: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:15.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:14 smithi045 conmon[33655]: cluster 2021-07-31T03:14:13.041504+0000 mgr.x (mgr.24542) 3727 2021-07-31T03:14:15.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:14 smithi045 conmon[33655]: : cluster [DBG] pgmap v4617: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:15.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:14 smithi045 conmon[39718]: cluster 2021-07-31T03:14:13.041504+0000 mgr.x (mgr. 2021-07-31T03:14:15.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:14 smithi045 conmon[39718]: 24542) 3727 : cluster [DBG] pgmap v4617: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:17.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:16 smithi154 conmon[34553]: cluster 2021-07-31T03:14:15.042004+0000 mgr.x (mgr.24542) 3728 : cluster [DBG] pgmap v4618: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:17.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:16 smithi045 conmon[33655]: cluster 2021-07-31T03:14:15.042004+0000 mgr.x (mgr.24542) 3728 : cluster 2021-07-31T03:14:17.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:16 smithi045 conmon[33655]: [DBG] pgmap v4618: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:17.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:16 smithi045 conmon[39718]: cluster 2021-07-31T03:14:15 2021-07-31T03:14:17.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:16 smithi045 conmon[39718]: .042004+0000 mgr.x (mgr.24542) 3728 : cluster [DBG] pgmap v4618: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:19.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:18 smithi154 conmon[34553]: cluster 2021-07-31T03:14:17.042445+0000 mgr.x (mgr.24542) 3729 : cluster 2021-07-31T03:14:19.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:18 smithi154 conmon[34553]: [DBG] pgmap v4619: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:19.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:18 smithi045 conmon[33655]: cluster 2021-07-31T03:14:17.042445+0000 mgr.x (mgr.24542) 3729 : cluster [DBG] pgmap v4619: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:19.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:18 smithi045 conmon[39718]: cluster 2021-07-31T03:14:17.042445+0000 mgr.x (mgr.24542) 3729 : cluster [DBG] pgmap v4619: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:21.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:20 smithi154 conmon[34553]: cluster 2021-07-31T03:14:19.043064+0000 mgr.x (mgr.24542) 3730 : cluster [DBG] pgmap v4620: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:21.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:20 smithi154 conmon[34553]: 2021-07-31T03:14:21.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:20 smithi045 conmon[39718]: cluster 2021-07-31T03:14:19.043064+0000 mgr.x 2021-07-31T03:14:21.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:20 smithi045 conmon[39718]: (mgr.24542) 3730 : cluster [DBG] pgmap v4620: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:21.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:20 smithi045 conmon[33655]: cluster 2021-07-31T03:14:19.043064+0000 mgr.x (mgr.24542) 3730 : cluster [DBG] 2021-07-31T03:14:21.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:20 smithi045 conmon[33655]: pgmap v4620: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:23.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:22 smithi154 conmon[34553]: cluster 2021-07-31T03:14:21.043596+0000 mgr.x (mgr.24542) 3731 : cluster [DBG] pgmap v4621: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:23.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:22 smithi045 conmon[39718]: cluster 2021-07-31T03:14:21.043596+0000 mgr.x (mgr 2021-07-31T03:14:23.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:22 smithi045 conmon[39718]: .24542) 3731 : cluster [DBG] pgmap v4621: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:23.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:22 smithi045 conmon[33655]: cluster 2021-07-31T03:14:21.043596+0000 mgr.x (mgr.24542) 3731 : cluster 2021-07-31T03:14:23.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:22 smithi045 conmon[33655]: [DBG] pgmap v4621: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:25.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:24 smithi154 conmon[34553]: cluster 2021-07-31T03:14:23.043986+0000 mgr.x (mgr.24542) 3732 : cluster [DBG] pgmap v4622: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:25.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:24 smithi045 conmon[33655]: cluster 2021-07-31T03:14:23.043986+0000 mgr.x (mgr.24542) 2021-07-31T03:14:25.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:24 smithi045 conmon[33655]: 3732 : cluster [DBG] pgmap v4622: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:25.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:24 smithi045 conmon[39718]: cluster 2021-07-31T03:14:23.043986+0000 mgr.x (mgr. 2021-07-31T03:14:25.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:24 smithi045 conmon[39718]: 24542) 3732 : cluster [DBG] pgmap v4622: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:27.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:26 smithi154 conmon[34553]: cluster 2021-07-31T03:14:25.044444+0000 mgr.x (mgr.24542) 3733 : cluster 2021-07-31T03:14:27.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:26 smithi154 conmon[34553]: [DBG] pgmap v4623: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:27.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:26 smithi045 conmon[33655]: cluster 2021-07-31T03:14:25.044444+0000 mgr.x ( 2021-07-31T03:14:27.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:26 smithi045 conmon[33655]: mgr.24542) 3733 : cluster [DBG] pgmap v4623: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:27.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:26 smithi045 conmon[39718]: cluster 2021-07-31T03:14:25.044444+0000 mgr.x (mgr.24542 2021-07-31T03:14:27.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:26 smithi045 conmon[39718]: ) 3733 : cluster [DBG] pgmap v4623: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:29.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:28 smithi154 conmon[34553]: cluster 2021-07-31T03:14:27.044870+0000 mgr.x (mgr.24542) 3734 : cluster 2021-07-31T03:14:29.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:28 smithi154 conmon[34553]: [DBG] pgmap v4624: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:29.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:28 smithi154 conmon[34553]: audit 2021-07-31T03:14:28.280515+0000 mon.b (mon.2) 510 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:14:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:28 smithi154 conmon[34553]: audit 2021-07-31T03:14:28.280773+0000 mon.b (mon 2021-07-31T03:14:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:28 smithi154 conmon[34553]: .2) 511 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:14:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:28 smithi154 conmon[34553]: audit 2021-07-31T03:14:28. 2021-07-31T03:14:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:28 smithi154 conmon[34553]: 281051+0000 mon.a (mon.0) 3271 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:14:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:28 smithi154 conmon[34553]: audit 2021-07 2021-07-31T03:14:29.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:28 smithi154 conmon[34553]: -31T03:14:28.281439+0000 mon.a 2021-07-31T03:14:29.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:28 smithi154 conmon[34553]: (mon.0) 3272 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:14:29.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:28 smithi045 conmon[33655]: cluster 2021-07-31T03:14:27.044870+0000 mgr.x (mgr.24542) 3734 : cluster [DBG] 2021-07-31T03:14:29.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:28 smithi045 conmon[33655]: pgmap v4624: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:29.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:28 smithi045 conmon[33655]: audit 2021-07-31T03:14:28.280515+0000 mon.b (mon.2) 510 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:14:29.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:28 smithi045 conmon[33655]: audit 2021-07 2021-07-31T03:14:29.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:28 smithi045 conmon[33655]: -31T03:14:28.280773+0000 mon.b ( 2021-07-31T03:14:29.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:28 smithi045 conmon[33655]: mon.2) 511 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:14:29.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:28 smithi045 conmon[33655]: audit 2021-07-31 2021-07-31T03:14:29.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:28 smithi045 conmon[33655]: T03:14:28.281051+0000 mon.a (mon 2021-07-31T03:14:29.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:28 smithi045 conmon[33655]: .0) 3271 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:14:29.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:28 smithi045 conmon[33655]: audit 2021-07-31T03 2021-07-31T03:14:29.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:28 smithi045 conmon[33655]: :14:28.281439+0000 mon.a (mon.0) 3272 : audit [INF] 2021-07-31T03:14:29.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:28 smithi045 conmon[33655]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:14:29.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:28 smithi045 conmon[39718]: cluster 2021-07-31T03:14:27.044870+0000 mgr.x (mgr. 2021-07-31T03:14:29.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:28 smithi045 conmon[39718]: 24542) 3734 : cluster [DBG] pgmap v4624: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:29.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:28 smithi045 conmon[39718]: audit 2021-07-31T03:14:28.280515+0000 mon.b (mon 2021-07-31T03:14:29.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:28 smithi045 conmon[39718]: .2) 510 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:14:29.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:28 smithi045 conmon[39718]: audit 2021- 2021-07-31T03:14:29.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:28 smithi045 conmon[39718]: 07-31T03:14:28.280773+0000 mon.b (mon.2) 511 : audit 2021-07-31T03:14:29.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:28 smithi045 conmon[39718]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:14:29.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:28 smithi045 conmon[39718]: audit 2021-07-31T03:14:28.281051+0000 mon.a ( 2021-07-31T03:14:29.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:28 smithi045 conmon[39718]: mon.0) 3271 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:14:29.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:28 smithi045 conmon[39718]: audit 2021-07-31T03:14:28 2021-07-31T03:14:29.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:28 smithi045 conmon[39718]: .281439+0000 mon.a (mon.0) 3272 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:14:31.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:30 smithi154 conmon[34553]: cluster 2021-07-31T03:14:29.045320+0000 mgr.x (mgr.24542) 3735 : cluster 2021-07-31T03:14:31.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:30 smithi154 conmon[34553]: [DBG] pgmap v4625: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:31.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:30 smithi045 conmon[39718]: cluster 2021-07-31T03:14:29.045320 2021-07-31T03:14:31.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:30 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 3735 : cluster [DBG] pgmap v4625: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:31.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:30 smithi045 conmon[33655]: cluster 2021-07-31T03:14:29.045320+0000 mgr.x ( 2021-07-31T03:14:31.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:30 smithi045 conmon[33655]: mgr.24542) 3735 : cluster [DBG] pgmap v4625: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:32.502 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: debug 2021-07-31T03:14:32.079+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:14:32.503 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:14:32.503 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:14:32.503 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: yield 2021-07-31T03:14:32.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:14:32.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:14:32.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:14:32.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:14:32.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:14:32.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:14:32.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:14:32.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:14:32.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:14:32.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: task.result() 2021-07-31T03:14:32.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:14:32.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: debug 2021-07-31T03:14:32.090+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:14:32.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:14:32.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:14:32.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: yield 2021-07-31T03:14:32.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:14:32.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:14:32.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:14:32.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:14:32.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:14:32.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:14:32.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:14:32.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:14:32.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:14:32.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: task.result() 2021-07-31T03:14:32.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:14:32.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: debug 2021-07-31T03:14:32.115+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:14:32.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:14:32.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:14:32.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: yield 2021-07-31T03:14:32.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:14:32.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:14:32.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:14:32.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:14:32.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:14:32.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:14:32.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:14:32.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:14:32.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:14:32.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: task.result() 2021-07-31T03:14:32.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:14:32.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: debug 2021-07-31T03:14:32.116+0000 7fcb9195d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:14:32.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:14:32.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:14:32.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: yield 2021-07-31T03:14:32.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:14:32.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:14:32.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:14:32.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:14:32.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:14:32.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:14:32.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:14:32.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:14:32.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:14:32.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: task.result() 2021-07-31T03:14:32.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:14:32.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: 2021-07-31T03:14:32.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:14:32.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: 2021-07-31T03:14:32.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:14:32.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:14:32.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:14:32.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:14:32.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:14:32.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:14:32.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:14:32.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:14:32.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:14:32.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:14:32.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:14:32.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:14:32.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:14:32.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:14:32.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:14:32.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:14:32.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:14:32.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:14:32.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: raise 2021-07-31T03:14:32.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:14:32.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:14:32.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:14:32.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:14:32.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:14:32.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: debug 2021-07-31T03:14:32.124+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:14:32.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:14:32.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:14:32.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: yield 2021-07-31T03:14:32.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:14:32.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:14:32.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:14:32.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:14:32.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:14:32.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:14:32.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:14:32.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:14:32.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:14:32.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: task.result() 2021-07-31T03:14:32.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:14:32.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: debug 2021-07-31T03:14:32.125+0000 7fcb9215e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:14:32.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:14:32.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:14:32.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: yield 2021-07-31T03:14:32.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:14:32.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:14:32.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:14:32.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:14:32.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:14:32.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:14:32.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:14:32.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:14:32.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:14:32.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: task.result() 2021-07-31T03:14:32.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:14:32.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: 2021-07-31T03:14:32.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:14:32.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: 2021-07-31T03:14:32.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:14:32.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:14:32.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:14:32.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:14:32.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:14:32.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:14:32.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:14:32.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:14:32.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:14:32.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:14:32.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:14:32.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:14:32.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:14:32.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:14:32.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:14:32.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:14:32.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:14:32.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:14:32.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: raise 2021-07-31T03:14:32.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:14:32.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:14:32.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:14:32.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:14:32.535 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:14:33.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[34553]: cluster 2021-07-31T03:14:31.045781+0000 mgr.x (mgr.24542) 3736 : cluster [DBG] pgmap v4626: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:33.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[34553]: audit 2021-07-31T03:14:32.033067+0000 mon.b (mon.2) 512 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:14:33.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[34553]: audit 2021-07-31T03:14:32.034495 2021-07-31T03:14:33.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[34553]: +0000 mon.b (mon.2) 513 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:14:33.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[34553]: audit 2021-07-31T03:14:32.035331+0000 mon.b (mon.2) 514 2021-07-31T03:14:33.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:32 smithi154 conmon[34553]: : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:14:33.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:32 smithi045 conmon[39718]: cluster 2021-07-31T03:14:31.045781+0000 mgr.x (mgr 2021-07-31T03:14:33.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:32 smithi045 conmon[39718]: .24542) 3736 : cluster [DBG] pgmap v4626: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:33.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:32 smithi045 conmon[39718]: audit 2021-07-31T03:14:32.033067+0000 mon.b (mon.2 2021-07-31T03:14:33.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:32 smithi045 conmon[39718]: ) 512 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:14:33.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:32 smithi045 conmon[39718]: audit 2021 2021-07-31T03:14:33.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:32 smithi045 conmon[39718]: -07-31T03:14:32.034495+0000 mon.b ( 2021-07-31T03:14:33.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:32 smithi045 conmon[39718]: mon.2) 513 : audit [DBG] 2021-07-31T03:14:33.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:32 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:14:33.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:32 smithi045 conmon[39718]: audit 2021 2021-07-31T03:14:33.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:32 smithi045 conmon[39718]: -07-31T03:14 2021-07-31T03:14:33.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:32 smithi045 conmon[39718]: :32.035331+0000 mon.b (mon.2 2021-07-31T03:14:33.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:32 smithi045 conmon[39718]: ) 514 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:14:33.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:32 smithi045 conmon[33655]: cluster 2021-07-31T03:14:31.045781+0000 mgr.x (mgr.24542) 3736 : cluster [DBG] pgmap v4626: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:33.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:32 smithi045 conmon[33655]: audit 2021-07-31T03:14:33.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:32 smithi045 conmon[33655]: 2021-07-31T03:14:32.033067+0000 mon.b (mon.2) 512 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:14:33.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:32 smithi045 conmon[33655]: audit 2021-07-31T03:14:32.034495+0000 mon.b (mon.2) 513 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:14:33.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:32 smithi045 conmon[33655]: audit 2021-07-31T03:14:32.035331+0000 mon.b (mon.2) 514 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:14:34.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:14:32.036147+0000 mgr.x (mgr.24542) 3737 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:14:34.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:14:32.080112+0000 mgr.x (mgr.24542) 3738 : cephadm [ERR] 2021-07-31T03:14:34.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:14:34.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:14:34.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: yield 2021-07-31T03:14:34.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:14:34.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:14:34.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:14:34.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:14:34.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:14:34.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:14:34.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:14:34.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:14:34.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:14:34.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: task.result() 2021-07-31T03:14:34.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:14:34.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:14:32.080530+0000 mgr.x (mgr.24542) 3739 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:14:34.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:14:32.091510+0000 mgr.x (mgr.24542) 3740 : cephadm [ERR] 2021-07-31T03:14:34.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:14:34.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:14:34.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: yield 2021-07-31T03:14:34.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:14:34.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:14:34.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:14:34.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:14:34.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:14:34.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:14:34.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:14:34.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:14:34.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:14:34.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: task.result() 2021-07-31T03:14:34.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:14:34.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:14:32.092503+0000 mgr.x (mgr.24542) 3741 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:14:34.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:14:34.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: 2021-07-31T03:14:32.116784+0000 mgr.x (mgr.24542) 3742 : cephadm [ERR] 2021-07-31T03:14:34.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:14:34.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:14:34.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: yield 2021-07-31T03:14:34.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:14:34.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:14:34.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:14:34.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:14:34.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:14:34.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:14:34.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:14:34.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:14:34.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:14:34.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: task.result() 2021-07-31T03:14:34.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:14:34.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:14:32.117245+0000 mgr.x (mgr.24542) 3743 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:14:34.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:14:34.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:14:34.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: yield 2021-07-31T03:14:34.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:14:34.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:14:34.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:14:34.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:14:34.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:14:34.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:14:34.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:14:34.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:14:34.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:14:34.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: task.result() 2021-07-31T03:14:34.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:14:34.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: 2021-07-31T03:14:34.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:14:34.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: 2021-07-31T03:14:34.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:14:34.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:14:34.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:14:34.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:14:34.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:14:34.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:14:34.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:14:34.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:14:34.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:14:34.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:14:34.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:14:34.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:14:34.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:14:34.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:14:34.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:14:34.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:14:34.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:14:34.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:14:34.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: raise 2021-07-31T03:14:34.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:14:34.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:14:34.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:14:34.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:14:34.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:14:34.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:14:32.125800+0000 mgr.x (mgr.24542) 3744 : cephadm [ERR] 2021-07-31T03:14:34.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:14:34.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:14:34.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: yield 2021-07-31T03:14:34.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:14:34.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:14:34.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:14:34.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:14:34.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:14:34.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:14:34.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:14:34.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:14:34.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:14:34.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: task.result() 2021-07-31T03:14:34.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:14:34.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:14:32.126195+0000 mgr.x (mgr.24542) 3745 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:14:34.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:14:34.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:14:34.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: yield 2021-07-31T03:14:34.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:14:34.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:14:34.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:14:34.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:14:34.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:14:34.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:14:34.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:14:34.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:14:34.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:14:34.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: task.result() 2021-07-31T03:14:34.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:14:34.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: 2021-07-31T03:14:34.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:14:34.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: 2021-07-31T03:14:34.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:14:34.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:14:34.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:14:34.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:14:34.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:14:34.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:14:34.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:14:34.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:14:34.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:14:34.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:14:34.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:14:34.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:14:34.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:14:34.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:14:34.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:14:34.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:14:34.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:14:34.035 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:14:34.035 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: raise 2021-07-31T03:14:34.035 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:14:34.035 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:14:34.036 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:14:34.036 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:14:34.036 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:33 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:14:34.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: cephadm 2021-07-31T03:14:32 2021-07-31T03:14:34.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: .036147+0000 mgr.x (mgr.24542) 3737 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:14:34.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: cephadm 2021-07-31T03: 2021-07-31T03:14:34.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: 14:32.080112+0000 mgr.x ( 2021-07-31T03:14:34.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: mgr.24542) 3738 : cephadm 2021-07-31T03:14:34.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: [ERR] 2021-07-31T03:14:34.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:14:34.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:14:34.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: yield 2021-07-31T03:14:34.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:14:34.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:14:34.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:14:34.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:14:34.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:14:34.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:14:34.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:14:34.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:14:34.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:14:34.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: task.result() 2021-07-31T03:14:34.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:14:34.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: cephadm 2021-07-31T03:14:32.080530+0000 mgr.x (mgr.24542) 3739 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:14:34.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: 2021-07-31T03:14:34.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: cephadm 2021-07-31T03:14:34.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: 2021-07-31T03 2021-07-31T03:14:34.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: :14:32.091510+0000 2021-07-31T03:14:34.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: mgr.x (mgr.24542) 3740 : cephadm 2021-07-31T03:14:34.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: [ERR] 2021-07-31T03:14:34.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:14:34.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:14:34.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: yield 2021-07-31T03:14:34.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:14:34.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:14:34.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:14:34.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:14:34.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:14:34.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:14:34.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:14:34.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:14:34.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:14:34.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: task.result() 2021-07-31T03:14:34.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:14:34.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: cephadm 2021-07-31T03:14:32.092503+0000 mgr.x (mgr.24542) 3741 : cephadm [INF] 2021-07-31T03:14:34.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:14:34.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: cephadm 2021- 2021-07-31T03:14:34.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: 07-31T03:14:32. 2021-07-31T03:14:34.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: 116784+0000 mgr.x (mgr.24542) 2021-07-31T03:14:34.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: 3742 : cephadm [ERR] 2021-07-31T03:14:34.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:14:34.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:14:34.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: yield 2021-07-31T03:14:34.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:14:34.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:14:34.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:14:34.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:14:34.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:14:34.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:14:34.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:14:34.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:14:34.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:14:34.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: task.result() 2021-07-31T03:14:34.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:14:34.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: cephadm 2021-07-31T03:14:32 2021-07-31T03:14:34.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: .117245+0000 mgr.x (mgr. 2021-07-31T03:14:34.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: 24542) 3743 : cephadm [ERR] 2021-07-31T03:14:34.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:14:34.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:14:34.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:14:34.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: yield 2021-07-31T03:14:34.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:14:34.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:14:34.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:14:34.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:14:34.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:14:34.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:14:34.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:14:34.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:14:34.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:14:34.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: task.result() 2021-07-31T03:14:34.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:14:34.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: 2021-07-31T03:14:34.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:14:34.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: 2021-07-31T03:14:34.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:14:34.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:14:34.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:14:34.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:14:34.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:14:34.056 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: cephadm 2021-07-31T03:14:32.036147+0000 mgr.x ( 2021-07-31T03:14:34.056 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: mgr.24542) 3737 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:14:34.056 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: cephadm 2021-07-31T03:14:32.080112+0000 mgr.x (mgr.24542) 3738 : cephadm [ERR] 2021-07-31T03:14:34.056 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:14:34.056 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:14:34.057 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: yield 2021-07-31T03:14:34.057 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:14:34.057 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:14:34.057 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:14:34.057 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:14:34.058 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:14:34.058 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:14:34.058 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:14:34.058 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:14:34.058 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:14:34.059 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: task.result() 2021-07-31T03:14:34.059 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:14:34.059 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: cephadm 2021-07-31T03:14:32.080530+0000 mgr.x (mgr.24542) 3739 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:14:34.059 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: cephadm 2021-07- 2021-07-31T03:14:34.060 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: 31T03:14:32.091510+0000 mgr.x (mgr.24542) 3740 2021-07-31T03:14:34.060 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: : cephadm [ERR] 2021-07-31T03:14:34.060 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:14:34.060 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:14:34.061 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: yield 2021-07-31T03:14:34.061 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:14:34.061 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:14:34.061 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:14:34.062 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:14:34.062 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:14:34.062 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:14:34.062 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:14:34.063 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:14:34.063 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:14:34.063 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: task.result() 2021-07-31T03:14:34.063 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:14:34.063 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: cephadm 2021-07-31T03:14:32.092503+0000 mgr.x (mgr.24542) 3741 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:14:34.064 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: cephadm 2021-07-31T03 2021-07-31T03:14:34.064 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: :14:32.116784+0000 mgr.x (mgr.24542) 2021-07-31T03:14:34.064 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: 3742 : cephadm [ERR] 2021-07-31T03:14:34.064 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:14:34.065 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:14:34.065 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:14:34.065 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:14:34.065 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:14:34.066 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:14:34.066 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:14:34.066 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:14:34.066 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:14:34.066 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:14:34.067 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:14:34.067 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:14:34.067 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:14:34.067 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:14:34.067 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: raise 2021-07-31T03:14:34.068 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:14:34.068 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:14:34.068 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:14:34.068 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:14:34.068 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:14:34.069 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: cephadm 2021-07-31T03:14:32.125800+0000 mgr.x (mgr.24542) 3744 : cephadm [ERR] 2021-07-31T03:14:34.069 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:14:34.069 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:14:34.069 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: yield 2021-07-31T03:14:34.069 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:14:34.070 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:14:34.070 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:14:34.070 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:14:34.070 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:14:34.070 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:14:34.071 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:14:34.071 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:14:34.071 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:14:34.071 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: task.result() 2021-07-31T03:14:34.072 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:14:34.072 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: cephadm 2021-07-31T03:14:32.126195+0000 mgr.x (mgr.24542 2021-07-31T03:14:34.072 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: ) 3745 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:14:34.072 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:14:34.072 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:14:34.073 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: yield 2021-07-31T03:14:34.073 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:14:34.073 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:14:34.073 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: yield 2021-07-31T03:14:34.074 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:14:34.074 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:14:34.074 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:14:34.074 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:14:34.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:14:34.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:14:34.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:14:34.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:14:34.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:14:34.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: task.result() 2021-07-31T03:14:34.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:14:34.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: cephadm 2021-07-31T03:14:32.117245+0000 mgr.x (mgr.24542) 3743 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:14:34.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:14:34.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:14:34.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: yield 2021-07-31T03:14:34.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:14:34.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:14:34.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:14:34.078 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:14:34.078 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:14:34.078 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:14:34.078 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:14:34.078 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:14:34.079 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:14:34.079 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: task.result() 2021-07-31T03:14:34.079 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:14:34.079 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: 2021-07-31T03:14:34.079 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:14:34.080 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: 2021-07-31T03:14:34.080 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:14:34.080 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:14:34.080 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:14:34.080 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:14:34.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:14:34.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:14:34.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:14:34.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:14:34.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:14:34.082 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:14:34.083 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:14:34.083 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:14:34.083 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:14:34.083 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:14:34.083 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:14:34.084 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:14:34.084 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:14:34.084 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: task.result() 2021-07-31T03:14:34.084 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:14:34.085 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: 2021-07-31T03:14:34.085 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:14:34.085 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: 2021-07-31T03:14:34.085 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:14:34.085 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:14:34.086 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:14:34.086 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:14:34.086 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:14:34.086 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:14:34.087 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:14:34.087 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:14:34.087 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:14:34.087 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:14:34.087 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:14:34.088 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:14:34.088 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:14:34.088 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:14:34.088 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:14:34.088 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:14:34.089 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:14:34.089 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:14:34.089 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: raise 2021-07-31T03:14:34.089 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:14:34.089 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:14:34.090 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:14:34.090 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:14:34.090 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:14:34.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:14:34.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:14:34.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:14:34.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:14:34.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:14:34.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:14:34.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:14:34.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:14:34.093 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:14:34.093 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: raise 2021-07-31T03:14:34.093 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:14:34.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:14:34.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:14:34.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:14:34.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:14:34.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: cephadm 2021-07-31T03:14:32.125800+0000 mgr.x (mgr.24542) 3744 : cephadm [ERR] 2021-07-31T03:14:34.095 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:14:34.095 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:14:34.095 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: yield 2021-07-31T03:14:34.095 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:14:34.095 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:14:34.096 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:14:34.096 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:14:34.096 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:14:34.096 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:14:34.097 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:14:34.097 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:14:34.097 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:14:34.097 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: task.result() 2021-07-31T03:14:34.097 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:14:34.098 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: cephadm 2021-07-31T03:14:32.126195+0000 mgr.x (mgr.24542) 3745 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:14:34.098 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:14:34.098 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:14:34.098 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: yield 2021-07-31T03:14:34.099 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:14:34.099 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:14:34.099 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:14:34.099 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:14:34.099 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:14:34.099 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:14:34.100 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:14:34.100 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:14:34.100 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:14:34.100 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: task.result() 2021-07-31T03:14:34.101 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:14:34.101 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: 2021-07-31T03:14:34.101 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:14:34.101 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: 2021-07-31T03:14:34.101 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:14:34.101 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:14:34.102 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:14:34.102 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:14:34.102 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:14:34.102 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:14:34.103 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:14:34.103 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:14:34.103 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:14:34.103 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:14:34.104 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:14:34.104 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:14:34.104 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:14:34.104 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:14:34.105 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:14:34.105 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:14:34.105 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:14:34.105 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:14:34.105 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: raise 2021-07-31T03:14:34.106 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:14:34.106 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:14:34.106 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:14:34.106 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:14:34.107 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:33 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:14:34.456 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:14:34.738 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:14:34.738 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 89m ago 90m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:14:34.738 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 87m ago 89m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:14:34.739 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 87m ago 93m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:14:34.739 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 89m ago 95m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:14:34.739 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 89m ago 95m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:14:34.739 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 87m ago 93m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:14:34.740 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 89m ago 93m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:14:34.740 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 89m ago 90m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:14:34.740 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 87m ago 90m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:14:34.740 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 89m ago 93m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:14:34.740 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 89m ago 92m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:14:34.741 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 89m ago 92m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:14:34.741 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 89m ago 92m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:14:34.741 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 87m ago 91m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:14:34.741 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 87m ago 91m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:14:34.742 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 87m ago 91m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:14:34.742 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 87m ago 90m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:14:34.742 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 87m ago 90m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:14:35.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:34 smithi154 conmon[34553]: cluster 2021-07-31T03:14:33.046269+0000 mgr.x (mgr.24542) 3746 : cluster [DBG] pgmap v4627: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:35.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:34 smithi154 conmon[34553]: 2021-07-31T03:14:35.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:34 smithi045 conmon[33655]: cluster 2021-07-31T03: 2021-07-31T03:14:35.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:34 smithi045 conmon[33655]: 14:33.046269+0000 mgr.x (mgr.24542) 3746 : cluster [DBG] pgmap v4627: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:35.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:34 smithi045 conmon[39718]: cluster 2021-07-31T03:14:33. 2021-07-31T03:14:35.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:34 smithi045 conmon[39718]: 046269+0000 mgr.x (mgr.24542) 3746 : cluster [DBG] pgmap v4627: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:35.071 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:14:35.071 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:14:35.072 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:14:35.072 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:14:35.072 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:14:35.072 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:14:35.073 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:14:35.073 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:14:35.073 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:14:35.073 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:14:35.073 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:14:35.074 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:14:35.074 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:14:35.074 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:14:35.074 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:14:35.075 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:14:35.075 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:14:36.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:35 smithi154 conmon[34553]: audit 2021-07-31T03:14:34.445325+0000 mgr.x (mgr.24542) 3747 : audit [DBG] from='client.29032 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:14:36.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:35 smithi154 conmon[34553]: audit 2021-07-31T03:14:34.730763+0000 mgr.x (mgr.24542) 3748 : audit [DBG] from='client.29038 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:14:36.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:35 smithi154 conmon[34553]: audit 2021-07-31T03:14:35.072295+0000 mon.c (mon.1) 189 : audit [DBG] from='client.? 172.21.15.45:0/1434257666' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:14:36.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:35 smithi045 conmon[39718]: audit 2021-07-31T03:14:34.445325+0000 2021-07-31T03:14:36.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:35 smithi045 conmon[39718]: mgr.x (mgr.24542) 3747 : audit [DBG] from='client.29032 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:14:36.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:35 smithi045 conmon[39718]: audit 2021-07-31T03:14:36.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:35 smithi045 conmon[39718]: 2021-07-31T03:14: 2021-07-31T03:14:36.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:35 smithi045 conmon[39718]: 34.730763+0000 mgr.x (mgr.24542) 2021-07-31T03:14:36.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:35 smithi045 conmon[39718]: 3748 : audit [DBG] from='client.29038 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:14:36.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:35 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T03:14:36.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:35 smithi045 conmon[39718]: T03:14:35.072295+0000 mon.c (mon. 2021-07-31T03:14:36.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:35 smithi045 conmon[39718]: 1) 189 : audit [DBG] from='client.? 172.21.15.45:0/1434257666' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:14:36.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:35 smithi045 conmon[33655]: audit 2021-07-31T03:14:34.445325+0000 mgr.x (mgr.24542) 3747 : audit [DBG] from='client.29032 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:14:36.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:35 smithi045 conmon[33655]: audit 2021-07-31T03:14:34.730763+0000 mgr.x (mgr.24542) 3748 : audit [DBG] from='client.29038 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:14:36.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:35 smithi045 conmon[33655]: audit 2021-07-31T03:14:35.072295+0000 mon.c (mon.1) 189 : audit [DBG] from='client.? 172.21.15.45:0/1434257666' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:14:37.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:36 smithi154 conmon[34553]: cluster 2021-07-31T03:14:35.046723+0000 mgr.x (mgr.24542) 3749 : cluster [DBG] 2021-07-31T03:14:37.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:36 smithi154 conmon[34553]: pgmap v4628: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:37.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:36 smithi045 conmon[39718]: cluster 2021-07-31T03:14:35.046723+0000 mgr.x (mgr. 2021-07-31T03:14:37.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:36 smithi045 conmon[39718]: 24542) 3749 : cluster [DBG] pgmap v4628: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:37.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:36 smithi045 conmon[33655]: cluster 2021-07-31T03:14:35.046723+0000 mgr.x (mgr.24542) 3749 : cluster [DBG] pgmap v4628: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:39.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:38 smithi154 conmon[34553]: cluster 2021-07-31T03:14:37.047174+0000 mgr.x (mgr.24542) 3750 : cluster [DBG] pgmap v4629: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:39.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:38 smithi045 conmon[39718]: cluster 2021-07-31T03:14:37.047174+0000 mgr.x ( 2021-07-31T03:14:39.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:38 smithi045 conmon[39718]: mgr.24542) 3750 : cluster [DBG] pgmap v4629: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:39.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:38 smithi045 conmon[33655]: cluster 2021-07-31T03:14:37.047174+0000 mgr.x (mgr.24542 2021-07-31T03:14:39.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:38 smithi045 conmon[33655]: ) 3750 : cluster [DBG] pgmap v4629: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:41.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:40 smithi154 conmon[34553]: cluster 2021-07-31T03:14:39.047728+0000 mgr.x (mgr.24542) 3751 : cluster [DBG] 2021-07-31T03:14:41.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:40 smithi154 conmon[34553]: pgmap v4630: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:41.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:40 smithi045 conmon[39718]: cluster 2021-07-31T03:14:39.047728+0000 mgr.x (mgr.24542) 2021-07-31T03:14:41.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:40 smithi045 conmon[39718]: 3751 : cluster [DBG] pgmap v4630: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:41.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:40 smithi045 conmon[33655]: cluster 2021-07-31T03:14:39.047728+0000 mgr.x (mgr.24542) 2021-07-31T03:14:41.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:40 smithi045 conmon[33655]: 3751 : cluster [DBG] pgmap v4630: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:43.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:42 smithi154 conmon[34553]: cluster 2021-07-31T03:14:41.048168+0000 mgr.x (mgr.24542) 3752 : cluster 2021-07-31T03:14:43.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:42 smithi154 conmon[34553]: [DBG] pgmap v4631: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:43.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:42 smithi045 conmon[39718]: cluster 2021-07-31T03:14:41.048168+0000 mgr.x (mgr.24542 2021-07-31T03:14:43.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:42 smithi045 conmon[39718]: ) 3752 : cluster [DBG] pgmap v4631: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:43.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:42 smithi045 conmon[33655]: cluster 2021-07-31T03:14: 2021-07-31T03:14:43.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:42 smithi045 conmon[33655]: 41.048168+0000 mgr.x (mgr.24542) 3752 : cluster [DBG] pgmap v4631: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:45.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:44 smithi154 conmon[34553]: cluster 2021-07-31T03:14:43.048511+0000 mgr.x (mgr.24542) 3753 : cluster [DBG] 2021-07-31T03:14:45.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:44 smithi154 conmon[34553]: pgmap v4632: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:45.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:44 smithi045 conmon[39718]: cluster 2021-07-31T03:14:43.048511+0000 mgr.x ( 2021-07-31T03:14:45.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:44 smithi045 conmon[39718]: mgr.24542) 3753 : cluster [DBG] pgmap v4632: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:45.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:44 smithi045 conmon[33655]: cluster 2021-07-31T03:14:43.048511+0000 2021-07-31T03:14:45.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:44 smithi045 conmon[33655]: mgr.x (mgr.24542) 3753 : cluster [DBG] pgmap v4632: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:47.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:46 smithi154 conmon[34553]: cluster 2021-07-31T03:14:45.049037+0000 mgr.x (mgr.24542) 3754 : cluster [DBG] 2021-07-31T03:14:47.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:46 smithi154 conmon[34553]: pgmap v4633: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:47.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:46 smithi045 conmon[33655]: cluster 2021-07-31T03:14:45. 2021-07-31T03:14:47.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:46 smithi045 conmon[33655]: 049037+0000 mgr.x (mgr.24542) 3754 : cluster [DBG] pgmap v4633: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:47.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:46 smithi045 conmon[39718]: cluster 2021-07-31T03:14:45.049037+0000 mgr.x ( 2021-07-31T03:14:47.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:46 smithi045 conmon[39718]: mgr.24542) 3754 : cluster [DBG] pgmap v4633: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:49.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:48 smithi154 conmon[34553]: cluster 2021-07-31T03:14:47.049501+0000 mgr.x (mgr.24542) 3755 : cluster [DBG] pgmap v4634: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:49.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:48 smithi045 conmon[39718]: cluster 2021-07-31T03:14:47.049501+0000 mgr.x (mgr.24542) 2021-07-31T03:14:49.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:48 smithi045 conmon[39718]: 3755 : cluster [DBG] pgmap v4634: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:49.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:48 smithi045 conmon[33655]: cluster 2021-07-31T03:14:47.049501+0000 2021-07-31T03:14:49.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:48 smithi045 conmon[33655]: mgr.x (mgr.24542) 3755 : cluster [DBG] pgmap v4634: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:51.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:50 smithi154 conmon[34553]: cluster 2021-07-31T03:14:49.050012+0000 mgr.x (mgr.24542) 3756 : cluster [DBG] pgmap v4635: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:51.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:50 smithi154 conmon[34553]: 2021-07-31T03:14:51.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:50 smithi045 conmon[39718]: cluster 2021-07-31T03:14:49.050012+0000 mgr.x (mgr.24542) 2021-07-31T03:14:51.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:50 smithi045 conmon[39718]: 3756 : cluster [DBG] pgmap v4635: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:51.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:50 smithi045 conmon[33655]: cluster 2021-07-31T03:14:49.050012+0000 mgr.x (mgr.24542) 3756 2021-07-31T03:14:51.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:50 smithi045 conmon[33655]: : cluster [DBG] pgmap v4635: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:53.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:52 smithi154 conmon[34553]: cluster 2021-07-31T03:14:51.050463+0000 mgr.x (mgr.24542) 3757 : cluster [DBG] pgmap v4636: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:53.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:52 smithi045 conmon[39718]: cluster 2021-07-31T03:14:51.050463+0000 mgr.x (mgr.24542) 3757 : cluster [DBG] pgmap v4636: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:53.054 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:52 smithi045 conmon[33655]: cluster 2021-07-31T03:14:51.050463+0000 mgr.x (mgr.24542) 3757 : cluster [DBG] pgmap v4636: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:53.055 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:52 smithi045 conmon[33655]: 2021-07-31T03:14:54.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:54 smithi154 conmon[34553]: cluster 2021-07-31T03:14:53.051013+0000 mgr.x (mgr.24542) 3758 : cluster [DBG] pgmap v4637: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:54.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:54 smithi045 conmon[33655]: cluster 2021-07-31T03:14:53.051013+0000 mgr.x (mgr.24542) 3758 : cluster 2021-07-31T03:14:54.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:54 smithi045 conmon[33655]: [DBG] pgmap v4637: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:54.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:54 smithi045 conmon[39718]: cluster 2021-07-31T03:14:53.051013+0000 mgr.x (mgr.24542) 3758 2021-07-31T03:14:54.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:54 smithi045 conmon[39718]: : cluster [DBG] pgmap v4637: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:56.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:56 smithi154 conmon[34553]: cluster 2021-07-31T03:14:55.051600+0000 mgr.x (mgr.24542) 3759 : cluster [DBG] pgmap v4638: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:56.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:56 smithi045 conmon[33655]: cluster 2021-07-31T03:14:55.051600+0000 mgr.x (mgr.24542) 3759 : cluster [DBG] pgmap v4638: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:56.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:56 smithi045 conmon[39718]: cluster 2021-07-31T03:14:55.051600+0000 2021-07-31T03:14:56.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:56 smithi045 conmon[39718]: mgr.x (mgr.24542) 3759 : cluster [DBG] pgmap v4638: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:58.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:14:58 smithi154 conmon[34553]: cluster 2021-07-31T03:14:57.051981+0000 mgr.x (mgr.24542) 3760 : cluster [DBG] pgmap v4639: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:58.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:58 smithi045 conmon[33655]: cluster 2021-07-31T03:14:57.051981+0000 mgr.x (mgr.24542) 3760 : cluster [DBG] 2021-07-31T03:14:58.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:14:58 smithi045 conmon[33655]: pgmap v4639: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:14:58.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:58 smithi045 conmon[39718]: cluster 2021-07-31T03:14:57.051981+0000 mgr.x 2021-07-31T03:14:58.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:14:58 smithi045 conmon[39718]: (mgr.24542) 3760 : cluster [DBG] pgmap v4639: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:00.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:00 smithi154 conmon[34553]: cluster 2021-07-31T03:14:59.052586+0000 mgr.x (mgr.24542) 3761 : cluster [DBG] pgmap v4640: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:00.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:00 smithi045 conmon[33655]: cluster 2021-07-31T03:14:59 2021-07-31T03:15:00.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:00 smithi045 conmon[33655]: .052586+0000 mgr.x (mgr.24542) 3761 : cluster [DBG] pgmap v4640: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:00.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:00 smithi045 conmon[39718]: cluster 2021-07-31T03:14:59.052586+0000 mgr.x ( 2021-07-31T03:15:00.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:00 smithi045 conmon[39718]: mgr.24542) 3761 : cluster [DBG] pgmap v4640: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:02.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:02 smithi154 conmon[34553]: cluster 2021-07-31T03:15:01.052916+0000 mgr.x (mgr.24542) 3762 : cluster [DBG] pgmap v4641: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:02.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:02 smithi045 conmon[33655]: cluster 2021-07-31T03:15:01.052916+0000 mgr.x (mgr.24542) 3762 : cluster [DBG] 2021-07-31T03:15:02.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:02 smithi045 conmon[33655]: pgmap v4641: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:02.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:02 smithi045 conmon[39718]: cluster 2021-07-31T03:15:01.052916+0000 mgr.x ( 2021-07-31T03:15:02.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:02 smithi045 conmon[39718]: mgr.24542) 3762 : cluster [DBG] pgmap v4641: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:04.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:04 smithi154 conmon[34553]: cluster 2021-07-31T03:15:03.053285+0000 mgr.x (mgr.24542) 3763 : cluster 2021-07-31T03:15:04.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:04 smithi154 conmon[34553]: [DBG] pgmap v4642: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:04.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:04 smithi045 conmon[33655]: cluster 2021-07-31T03:15:03.053285+0000 mgr.x (mgr. 2021-07-31T03:15:04.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:04 smithi045 conmon[33655]: 24542) 3763 : cluster [DBG] pgmap v4642: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:04.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:04 smithi045 conmon[39718]: cluster 2021-07-31T03:15:03.053285+0000 mgr.x (mgr 2021-07-31T03:15:04.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:04 smithi045 conmon[39718]: .24542) 3763 : cluster [DBG] pgmap v4642: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:05.383 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:15:05.659 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:15:05.659 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 89m ago 90m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:15:05.660 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 87m ago 90m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:15:05.660 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 87m ago 93m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:15:05.660 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 89m ago 96m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:15:05.660 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 89m ago 96m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:15:05.661 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 87m ago 94m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:15:05.661 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 89m ago 94m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:15:05.661 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 89m ago 90m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:15:05.661 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 87m ago 90m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:15:05.661 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 89m ago 93m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:15:05.662 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 89m ago 93m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:15:05.662 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 89m ago 93m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:15:05.662 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 89m ago 92m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:15:05.662 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 87m ago 92m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:15:05.663 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 87m ago 92m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:15:05.663 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 87m ago 91m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:15:05.663 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 87m ago 91m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:15:05.663 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 87m ago 91m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:15:05.994 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:15:05.994 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:15:05.994 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:15:05.994 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:15:05.995 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:15:05.995 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:15:05.995 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:15:05.995 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:15:05.995 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:15:05.996 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:15:05.996 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:15:05.996 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:15:05.996 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:15:05.997 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:15:05.997 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:15:05.997 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:15:05.997 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:15:06.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:06 smithi154 conmon[34553]: cluster 2021-07-31T03:15:05.053774+0000 mgr.x (mgr.24542) 2021-07-31T03:15:06.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:06 smithi154 conmon[34553]: 3764 : cluster [DBG] pgmap v4643: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:06.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:06 smithi154 conmon[34553]: audit 2021-07-31T03:15:05.372052+0000 mgr.x (mgr.24542) 3765 : audit [DBG] from='client.29050 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:15:06.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:06 smithi154 conmon[34553]: audit 2021-07-31T03:15:05.651500+0000 2021-07-31T03:15:06.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:06 smithi154 conmon[34553]: mgr.x (mgr.24542) 3766 : audit [DBG] from='client.29056 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:15:06.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:06 smithi154 conmon[34553]: audit 2021-07-31T03:15:05.994584 2021-07-31T03:15:06.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:06 smithi154 conmon[34553]: +0000 mon.c (mon.1) 190 : audit [DBG] from='client.? 172.21.15.45:0/864831567' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:15:06.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:06 smithi154 conmon[34553]: 2021-07-31T03:15:06.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:06 smithi045 conmon[33655]: cluster 2021-07-31T03:15:05.053774+0000 mgr.x (mgr 2021-07-31T03:15:06.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:06 smithi045 conmon[33655]: .24542) 3764 : cluster [DBG] pgmap v4643: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:06.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:06 smithi045 conmon[33655]: audit 2021-07-31T03:15:05.372052+0000 mgr.x (mgr.24542) 3765 : audit [DBG] from='client.29050 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:15:06.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:06 smithi045 conmon[33655]: audit 2021-07-31T03:15:05. 2021-07-31T03:15:06.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:06 smithi045 conmon[33655]: 651500+0000 mgr.x (mgr.24542) 3766 : audit [DBG] from='client.29056 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:15:06.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:06 smithi045 conmon[33655]: audit 2021 2021-07-31T03:15:06.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:06 smithi045 conmon[33655]: -07-31T03:15:05.994584+0000 2021-07-31T03:15:06.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:06 smithi045 conmon[33655]: mon.c (mon.1) 190 : audit [DBG] from='client.? 172.21.15.45:0/864831567' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:15:06.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:06 smithi045 conmon[39718]: cluster 2021-07-31T03:15:05.053774+0000 mgr.x ( 2021-07-31T03:15:06.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:06 smithi045 conmon[39718]: mgr.24542) 3764 : cluster [DBG] pgmap v4643: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:06.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:06 smithi045 conmon[39718]: audit 2021-07-31T03:15:05. 2021-07-31T03:15:06.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:06 smithi045 conmon[39718]: 372052+0000 mgr.x (mgr.24542) 3765 : audit [DBG] from='client.29050 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:15:06.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:06 smithi045 conmon[39718]: audit 2021-07-31T03:15 2021-07-31T03:15:06.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:06 smithi045 conmon[39718]: :05.651500+0000 mgr.x (mgr.24542) 3766 : audit 2021-07-31T03:15:06.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:06 smithi045 conmon[39718]: [DBG] from='client.29056 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:15:06.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:06 smithi045 conmon[39718]: audit 2021-07-31T03:15:06.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:06 smithi045 conmon[39718]: 2021-07-31T03:15:05.994584+0000 2021-07-31T03:15:06.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:06 smithi045 conmon[39718]: mon.c (mon.1) 190 : audit [DBG] from='client.? 172.21.15.45:0/864831567' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:15:08.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:08 smithi154 conmon[34553]: cluster 2021-07-31T03:15:07.054215+0000 mgr.x (mgr.24542) 3767 : cluster [DBG] pgmap v4644: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:08.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:08 smithi045 conmon[33655]: cluster 2021-07-31T03: 2021-07-31T03:15:08.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:08 smithi045 conmon[33655]: 15:07.054215+0000 mgr.x (mgr.24542) 3767 : cluster [DBG] pgmap v4644: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:08.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:08 smithi045 conmon[39718]: cluster 2021-07-31T03:15:07.054215+0000 mgr.x (mgr.24542) 3767 : 2021-07-31T03:15:08.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:08 smithi045 conmon[39718]: cluster [DBG] pgmap v4644: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:10.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:10 smithi154 conmon[34553]: cluster 2021-07-31T03:15:09.054676+0000 mgr.x (mgr.24542) 3768 : cluster [DBG] pgmap v4645: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:10.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:10 smithi045 conmon[33655]: cluster 2021-07-31T03:15:09.054676+0000 mgr.x (mgr.24542) 3768 : cluster [DBG] 2021-07-31T03:15:10.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:10 smithi045 conmon[33655]: pgmap v4645: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:10.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:10 smithi045 conmon[39718]: cluster 2021-07-31T03:15: 2021-07-31T03:15:10.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:10 smithi045 conmon[39718]: 09.054676+0000 mgr.x (mgr.24542) 3768 : cluster [DBG] pgmap v4645: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:12.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:12 smithi154 conmon[34553]: cluster 2021-07-31T03:15:11.055207+0000 mgr.x (mgr.24542) 3769 : cluster [DBG] pgmap v4646: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:12.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:12 smithi045 conmon[33655]: cluster 2021-07-31T03:15:11.055207 2021-07-31T03:15:12.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:12 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 3769 : cluster [DBG] pgmap v4646: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:12.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:12 smithi045 conmon[39718]: cluster 2021-07-31T03:15:11 2021-07-31T03:15:12.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:12 smithi045 conmon[39718]: .055207+0000 mgr.x (mgr.24542) 3769 : cluster [DBG] pgmap v4646: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:14.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:14 smithi154 conmon[34553]: cluster 2021-07-31T03:15:13.055582+0000 mgr.x (mgr.24542) 3770 : cluster [DBG] pgmap v4647: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:14.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:14 smithi154 conmon[34553]: 2021-07-31T03:15:14.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:14 smithi045 conmon[39718]: cluster 2021-07-31T03:15:13.055582+0000 2021-07-31T03:15:14.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:14 smithi045 conmon[39718]: mgr.x (mgr.24542) 3770 : cluster [DBG] pgmap v4647: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:14.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:14 smithi045 conmon[33655]: cluster 2021-07-31T03:15:13.055582+0000 mgr.x ( 2021-07-31T03:15:14.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:14 smithi045 conmon[33655]: mgr.24542) 3770 : cluster [DBG] pgmap v4647: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:16.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:16 smithi154 conmon[34553]: cluster 2021-07-31T03:15:15.056179+0000 mgr.x (mgr.24542) 3771 : cluster [DBG] pgmap v4648: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:16.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:16 smithi045 conmon[39718]: cluster 2021-07-31T03:15:15.056179+0000 mgr.x (mgr.24542) 3771 : cluster [DBG] pgmap v4648: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:16.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:16 smithi045 conmon[33655]: cluster 2021-07-31T03:15:15.056179+0000 mgr.x (mgr.24542) 2021-07-31T03:15:16.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:16 smithi045 conmon[33655]: 3771 : cluster [DBG] pgmap v4648: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:18.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:18 smithi154 conmon[34553]: cluster 2021-07-31T03:15:17.056543+0000 mgr.x (mgr.24542) 3772 : cluster [DBG] pgmap v4649: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:18.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:18 smithi045 conmon[39718]: cluster 2021-07-31T03:15:17.056543+0000 mgr.x ( 2021-07-31T03:15:18.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:18 smithi045 conmon[39718]: mgr.24542) 3772 : cluster [DBG] pgmap v4649: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:18.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:18 smithi045 conmon[33655]: cluster 2021-07-31T03:15:17.056543 2021-07-31T03:15:18.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:18 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 3772 : cluster [DBG] pgmap v4649: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:20.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:20 smithi154 conmon[34553]: cluster 2021-07-31T03:15:19.057151+0000 mgr.x (mgr.24542) 3773 : cluster [DBG] pgmap v4650: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:20.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:20 smithi154 conmon[34553]: 2021-07-31T03:15:20.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:20 smithi045 conmon[39718]: cluster 2021-07-31T03:15:19.057151+0000 2021-07-31T03:15:20.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:20 smithi045 conmon[39718]: mgr.x (mgr.24542) 3773 : cluster [DBG] pgmap v4650: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:20.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:20 smithi045 conmon[33655]: cluster 2021-07-31T03:15:19.057151 2021-07-31T03:15:20.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:20 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 3773 : cluster [DBG] pgmap v4650: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:22.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:22 smithi154 conmon[34553]: cluster 2021-07-31T03:15:21.057592+0000 mgr.x (mgr.24542) 3774 : cluster [DBG] pgmap v4651: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:22.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:22 smithi045 conmon[33655]: cluster 2021-07-31T03:15:21.057592+0000 mgr.x (mgr.24542) 3774 : cluster 2021-07-31T03:15:22.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:22 smithi045 conmon[33655]: [DBG] pgmap v4651: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:22.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:22 smithi045 conmon[39718]: cluster 2021-07-31T03:15:21.057592+0000 mgr.x ( 2021-07-31T03:15:22.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:22 smithi045 conmon[39718]: mgr.24542) 3774 : cluster [DBG] pgmap v4651: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:24.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:24 smithi154 conmon[34553]: cluster 2021-07-31T03:15:23.057963+0000 mgr.x (mgr.24542) 3775 : cluster [DBG] pgmap v4652: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:24.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:24 smithi045 conmon[39718]: cluster 2021-07-31T03:15:23.057963+0000 mgr.x (mgr.24542) 3775 2021-07-31T03:15:24.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:24 smithi045 conmon[39718]: : cluster [DBG] pgmap v4652: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:24.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:24 smithi045 conmon[33655]: cluster 2021-07-31T03:15:23.057963+0000 mgr.x (mgr.24542) 3775 : cluster 2021-07-31T03:15:24.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:24 smithi045 conmon[33655]: [DBG] pgmap v4652: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:26.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:26 smithi154 conmon[34553]: cluster 2021-07-31T03:15:25.058522+0000 mgr.x (mgr.24542) 3776 : cluster [DBG] pgmap v4653: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:26.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:26 smithi045 conmon[33655]: cluster 2021-07-31T03:15:25.058522+0000 2021-07-31T03:15:26.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:26 smithi045 conmon[33655]: mgr.x (mgr.24542) 3776 : cluster [DBG] pgmap v4653: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:26.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:26 smithi045 conmon[39718]: cluster 2021-07-31T03:15:25.058522+0000 mgr.x (mgr 2021-07-31T03:15:26.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:26 smithi045 conmon[39718]: .24542) 3776 : cluster [DBG] pgmap v4653: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:28.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:28 smithi154 conmon[34553]: cluster 2021-07-31T03:15:27.058931+0000 mgr.x (mgr.24542) 3777 : cluster [DBG] pgmap v4654: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:28.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:28 smithi045 conmon[39718]: cluster 2021-07-31T03:15:27.058931+0000 mgr.x ( 2021-07-31T03:15:28.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:28 smithi045 conmon[39718]: mgr.24542) 3777 : cluster [DBG] pgmap v4654: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:28.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:28 smithi045 conmon[33655]: cluster 2021-07-31T03:15:27.058931+0000 mgr.x (mgr.24542 2021-07-31T03:15:28.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:28 smithi045 conmon[33655]: ) 3777 : cluster [DBG] pgmap v4654: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:29.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:29 smithi154 conmon[34553]: audit 2021-07-31T03:15:28.284933+0000 mon.b (mon.2) 515 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:15:29.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:29 smithi154 conmon[34553]: 2021-07-31T03:15:29.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:29 smithi154 conmon[34553]: audit 2021-07-31T03:15:28.285181+0000 mon.b (mon.2) 516 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:15:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:29 smithi154 conmon[34553]: audit 2021-07-31T03:15:28.285643+0000 mon.a (mon.0) 3273 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:15:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:29 smithi154 conmon[34553]: audit 2021-07-31T03:15:28.286232+0000 mon.a (mon.0) 2021-07-31T03:15:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:29 smithi154 conmon[34553]: 3274 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:15:29.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:29 smithi045 conmon[39718]: audit 2021-07-31T03:15:28.284933+0000 mon.b ( 2021-07-31T03:15:29.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:29 smithi045 conmon[39718]: mon.2) 515 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:15:29.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:29 smithi045 conmon[39718]: audit 2021-07-31T03:15:28.285181+0000 mon.b (mon 2021-07-31T03:15:29.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:29 smithi045 conmon[39718]: .2) 516 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:15:29.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:29 smithi045 conmon[39718]: audit 2021-07-31T03:15:29.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:29 smithi045 conmon[39718]: 2021-07-31T03:15:28.285643+0000 mon.a 2021-07-31T03:15:29.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:29 smithi045 conmon[39718]: (mon.0) 3273 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:15:29.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:29 smithi045 conmon[39718]: 2021-07-31T03:15:29.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:29 smithi045 conmon[39718]: audit 2021-07-31T03:15 2021-07-31T03:15:29.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:29 smithi045 conmon[39718]: :28.286232+0000 mon.a (mon.0) 2021-07-31T03:15:29.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:29 smithi045 conmon[39718]: 3274 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:15:29.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:29 smithi045 conmon[33655]: audit 2021-07-31T03:15:28.284933+0000 mon.b (mon.2) 2021-07-31T03:15:29.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:29 smithi045 conmon[33655]: 515 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:15:29.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:29 smithi045 conmon[33655]: audit 2021-07-31T03:15:28.285181+0000 mon.b (mon.2) 516 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:15:29.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:29 smithi045 conmon[33655]: audit 2021-07-31T03:15:28.285643+0000 mon.a (mon.0) 3273 2021-07-31T03:15:29.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:29 smithi045 conmon[33655]: : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:15:29.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:29 smithi045 conmon[33655]: audit 2021- 2021-07-31T03:15:29.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:29 smithi045 conmon[33655]: 07-31T03:15:28.286232+0000 mon.a (mon. 2021-07-31T03:15:29.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:29 smithi045 conmon[33655]: 0) 3274 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:15:30.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:30 smithi154 conmon[34553]: cluster 2021-07-31T03:15:29.059363+0000 mgr.x (mgr.24542) 3778 : cluster [DBG] 2021-07-31T03:15:30.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:30 smithi154 conmon[34553]: pgmap v4655: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:30.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:30 smithi045 conmon[39718]: cluster 2021-07-31T03:15:29.059363+0000 mgr.x 2021-07-31T03:15:30.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:30 smithi045 conmon[39718]: (mgr.24542) 3778 : cluster [DBG] pgmap v4655: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:30.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:30 smithi045 conmon[33655]: cluster 2021-07-31T03:15:29.059363+0000 mgr.x (mgr.24542 2021-07-31T03:15:30.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:30 smithi045 conmon[33655]: ) 3778 : cluster [DBG] pgmap v4655: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:32.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[34553]: cluster 2021-07-31T03:15:31.059775+0000 mgr.x (mgr.24542) 3779 : cluster [DBG] pgmap v4656: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:32.503 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: debug 2021-07-31T03:15:32.172+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:15:32.503 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:15:32.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:15:32.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: yield 2021-07-31T03:15:32.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:15:32.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:15:32.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:15:32.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:15:32.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:15:32.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:15:32.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:15:32.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:15:32.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:15:32.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: task.result() 2021-07-31T03:15:32.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:15:32.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: debug 2021-07-31T03:15:32.184+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:15:32.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:15:32.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:15:32.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: yield 2021-07-31T03:15:32.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:15:32.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:15:32.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:15:32.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:15:32.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:15:32.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:15:32.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:15:32.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:15:32.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:15:32.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: task.result() 2021-07-31T03:15:32.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:15:32.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: debug 2021-07-31T03:15:32.209+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:15:32.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:15:32.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:15:32.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: yield 2021-07-31T03:15:32.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:15:32.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:15:32.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:15:32.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:15:32.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:15:32.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:15:32.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:15:32.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:15:32.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:15:32.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: task.result() 2021-07-31T03:15:32.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:15:32.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: debug 2021-07-31T03:15:32.210+0000 7fcb9115c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:15:32.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:15:32.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:15:32.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: yield 2021-07-31T03:15:32.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:15:32.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:15:32.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:15:32.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:15:32.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:15:32.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:15:32.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:15:32.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:15:32.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:15:32.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: task.result() 2021-07-31T03:15:32.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:15:32.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: 2021-07-31T03:15:32.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:15:32.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: 2021-07-31T03:15:32.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:15:32.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:15:32.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:15:32.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:15:32.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:15:32.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:15:32.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:15:32.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:15:32.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:15:32.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:15:32.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:15:32.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:15:32.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:15:32.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:15:32.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:15:32.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:15:32.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:15:32.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:15:32.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: raise 2021-07-31T03:15:32.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:15:32.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:15:32.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:15:32.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:15:32.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:15:32.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: debug 2021-07-31T03:15:32.219+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:15:32.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:15:32.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:15:32.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: yield 2021-07-31T03:15:32.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:15:32.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:15:32.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:15:32.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:15:32.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:15:32.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:15:32.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:15:32.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:15:32.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:15:32.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: task.result() 2021-07-31T03:15:32.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:15:32.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: debug 2021-07-31T03:15:32.220+0000 7fcb9095b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:15:32.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:15:32.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:15:32.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: yield 2021-07-31T03:15:32.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:15:32.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:15:32.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:15:32.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:15:32.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:15:32.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:15:32.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:15:32.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:15:32.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:15:32.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: task.result() 2021-07-31T03:15:32.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:15:32.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: 2021-07-31T03:15:32.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:15:32.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: 2021-07-31T03:15:32.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:15:32.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:15:32.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:15:32.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:15:32.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:15:32.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:15:32.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:15:32.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:15:32.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:15:32.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:15:32.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:15:32.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:15:32.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:15:32.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:15:32.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:15:32.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:15:32.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:15:32.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:15:32.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: raise 2021-07-31T03:15:32.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:15:32.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:15:32.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:15:32.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:15:32.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:15:32 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:15:32.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:32 smithi045 conmon[39718]: cluster 2021-07-31T03:15:31.059775+0000 mgr.x ( 2021-07-31T03:15:32.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:32 smithi045 conmon[39718]: mgr.24542) 3779 : cluster [DBG] pgmap v4656: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:32.535 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:32 smithi045 conmon[33655]: cluster 2021-07-31T03:15:31.059775+0000 mgr.x (mgr.24542) 3779 : 2021-07-31T03:15:32.535 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:32 smithi045 conmon[33655]: cluster [DBG] pgmap v4656: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:33.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: audit 2021-07-31T03:15:32.126967+0000 mon.b (mon.2) 517 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:15:33.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: audit 2021-07-31T03:15:32.128401+0000 mon.b (mon.2) 518 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:15:33.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: audit 2021-07-31T03:15:32.129376+0000 mon.b (mon.2) 519 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:15:33.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:15:32.130117+0000 mgr.x (mgr.24542) 3780 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:15:33.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: cephadm 2021-07-31T03 2021-07-31T03:15:33.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: :15:32.173561+0000 mgr.x (mgr.24542) 3781 : cephadm [ERR] 2021-07-31T03:15:33.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:15:33.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:15:33.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: yield 2021-07-31T03:15:33.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:15:33.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:15:33.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:15:33.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:15:33.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:15:33.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:15:33.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:15:33.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:15:33.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:15:33.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: task.result() 2021-07-31T03:15:33.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:15:33.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: 2021-07-31T03:15:33.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:15:32.173960+0000 mgr.x (mgr.24542) 3782 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:15:33.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:15:32.185002+0000 mgr.x (mgr.24542) 3783 : cephadm [ERR] 2021-07-31T03:15:33.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:15:33.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:15:33.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: yield 2021-07-31T03:15:33.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:15:33.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:15:33.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:15:33.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:15:33.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:15:33.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:15:33.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:15:33.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:15:33.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:15:33.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: task.result() 2021-07-31T03:15:33.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:15:33.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:15:32.186072+0000 mgr.x (mgr.24542) 3784 2021-07-31T03:15:33.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:15:33.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:15:32.210899+0000 mgr.x (mgr.24542) 3785 : cephadm [ERR] 2021-07-31T03:15:33.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:15:33.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:15:33.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: yield 2021-07-31T03:15:33.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:15:33.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:15:33.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:15:33.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:15:33.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:15:33.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:15:33.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:15:33.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:15:33.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:15:33.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: task.result() 2021-07-31T03:15:33.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:15:33.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:15:32.211311+0000 mgr.x (mgr.24542) 3786 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:15:33.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:15:33.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:15:33.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: yield 2021-07-31T03:15:33.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:15:33.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:15:33.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:15:33.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:15:33.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:15:33.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:15:33.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:15:33.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:15:33.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:15:33.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: task.result() 2021-07-31T03:15:33.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:15:33.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: 2021-07-31T03:15:33.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:15:33.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: 2021-07-31T03:15:33.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:15:33.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:15:33.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:15:33.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:15:33.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:15:33.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:15:33.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:15:33.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:15:33.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:15:33.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:15:33.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:15:33.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:15:33.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:15:33.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:15:33.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:15:33.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:15:33.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:15:33.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:15:33.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: raise 2021-07-31T03:15:33.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:15:33.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:15:33.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:15:33.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:15:33.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:15:33.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:15:32.220867+0000 mgr.x (mgr.24542) 3787 : cephadm [ERR] 2021-07-31T03:15:33.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: 2021-07-31T03:15:33.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:15:33.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:15:33.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: yield 2021-07-31T03:15:33.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:15:33.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:15:33.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:15:33.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:15:33.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:15:33.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:15:33.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:15:33.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:15:33.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:15:33.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: task.result() 2021-07-31T03:15:33.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:15:33.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:15:32.221293+0000 mgr.x (mgr.24542) 3788 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:15:33.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:15:33.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:15:33.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: yield 2021-07-31T03:15:33.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:15:33.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:15:33.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:15:33.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:15:33.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:15:33.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:15:33.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:15:33.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:15:33.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:15:33.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: task.result() 2021-07-31T03:15:33.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:15:33.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: 2021-07-31T03:15:33.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:15:33.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: 2021-07-31T03:15:33.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:15:33.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:15:33.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:15:33.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:15:33.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:15:33.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:15:33.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:15:33.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:15:33.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:15:33.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:15:33.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:15:33.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:15:33.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:15:33.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:15:33.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:15:33.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:15:33.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:15:33.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:15:33.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: raise 2021-07-31T03:15:33.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:15:33.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:15:33.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:15:33.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:15:33.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:33 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:15:33.535 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: audit 2021-07-31T03:15:32.126967+0000 mon.b (mon.2) 2021-07-31T03:15:33.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: 517 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:15:33.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: audit 2021-07-31T03:15:32.128401+0000 mon.b (mon.2) 518 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:15:33.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: audit 2021-07-31T03:15:32 2021-07-31T03:15:33.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: .129376+0000 mon.b (mon.2) 519 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:15:33.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: cephadm 2021 2021-07-31T03:15:33.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: -07-31T03:15: 2021-07-31T03:15:33.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: 32.130117+0000 mgr.x (mgr 2021-07-31T03:15:33.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: .24542) 3780 : cephadm [WRN] 2021-07-31T03:15:33.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:15:33.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: cephadm 2021-07 2021-07-31T03:15:33.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: -31T03:15:32.173561+0000 mgr.x (mgr.24542) 3781 : cephadm [ERR] 2021-07-31T03:15:33.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: 2021-07-31T03:15:33.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:15:33.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:15:33.539 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: yield 2021-07-31T03:15:33.539 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:15:33.539 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:15:33.539 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:15:33.539 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:15:33.540 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:15:33.540 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:15:33.540 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:15:33.540 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:15:33.540 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:15:33.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: task.result() 2021-07-31T03:15:33.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:15:33.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: cephadm 2021-07-31T03:15:32.173960+0000 mgr.x (mgr.24542) 3782 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:15:33.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: cephadm 2021-07-31T03:15:33.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: 2021-07-31T03:15:32.185002+0000 mgr.x (mgr.24542) 3783 : 2021-07-31T03:15:33.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: cephadm [ERR] 2021-07-31T03:15:33.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:15:33.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:15:33.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: yield 2021-07-31T03:15:33.543 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:15:33.543 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:15:33.543 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:15:33.543 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:15:33.543 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:15:33.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: audit 2021-07-31T03:15:32.126967+0000 2021-07-31T03:15:33.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: mon.b (mon.2) 517 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:15:33.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T03:15:33.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: 03:15:32.128401+0000 mon.b (mon.2) 518 2021-07-31T03:15:33.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:15:33.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: audit 2021-07-31T03:15:33.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: 2021-07-31T03:15 2021-07-31T03:15:33.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: :32.129376+0000 mon.b (mon 2021-07-31T03:15:33.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: .2) 519 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:15:33.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: cephadm 2021-07- 2021-07-31T03:15:33.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: 31T03:15:32.130117+0000 2021-07-31T03:15:33.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: mgr.x (mgr.24542) 3780 : cephadm [WRN] 2021-07-31T03:15:33.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:15:33.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: cephadm 2021-07-31T03:15:33.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: 2021-07-31T03: 2021-07-31T03:15:33.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: 15:32.173561+0000 mgr.x ( 2021-07-31T03:15:33.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: mgr.24542) 3781 : cephadm [ERR] 2021-07-31T03:15:33.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:15:33.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:15:33.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: yield 2021-07-31T03:15:33.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:15:33.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:15:33.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:15:33.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:15:33.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:15:33.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:15:33.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:15:33.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:15:33.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:15:33.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: task.result() 2021-07-31T03:15:33.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:15:33.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: 2021-07-31T03:15:33.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: cephadm 2021-07-31T03:15:32.173960+0000 mgr.x (mgr.24542) 2021-07-31T03:15:33.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: 3782 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:15:33.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: cephadm 2021-07-31T03:15:33.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: 2021-07-31 2021-07-31T03:15:33.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: T03:15:32.185002 2021-07-31T03:15:33.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 3783 2021-07-31T03:15:33.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: : cephadm [ERR] 2021-07-31T03:15:33.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:15:33.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:15:33.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: yield 2021-07-31T03:15:33.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:15:33.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:15:33.553 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:15:33.553 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:15:33.554 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:15:33.554 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:15:33.554 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: task.result() 2021-07-31T03:15:33.554 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:15:33.554 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: cephadm 2021-07-31T03:15:32.186072+0000 mgr.x (mgr.24542) 3784 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:15:33.555 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: cephadm 2021-07-31T03:15:33.555 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: 2021-07-31T03:15:32.210899 2021-07-31T03:15:33.555 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 3785 : cephadm 2021-07-31T03:15:33.555 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: [ERR] 2021-07-31T03:15:33.556 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:15:33.556 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:15:33.556 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: yield 2021-07-31T03:15:33.556 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:15:33.556 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:15:33.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:15:33.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:15:33.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:15:33.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:15:33.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:15:33.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:15:33.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:15:33.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: task.result() 2021-07-31T03:15:33.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:15:33.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: cephadm 2021-07-31T03:15:32.211311+0000 mgr.x (mgr.24542) 3786 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:15:33.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:15:33.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:15:33.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: yield 2021-07-31T03:15:33.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:15:33.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:15:33.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:15:33.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:15:33.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:15:33.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:15:33.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:15:33.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:15:33.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:15:33.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: task.result() 2021-07-31T03:15:33.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:15:33.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: 2021-07-31T03:15:33.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:15:33.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:15:33.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:15:33.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:15:33.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:15:33.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:15:33.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:15:33.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:15:33.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: task.result() 2021-07-31T03:15:33.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:15:33.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: cephadm 2021-07-31T03:15:32.186072+0000 mgr.x (mgr.24542) 3784 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:15:33.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: cephadm 2021-07-31T03:15:33.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: 2021-07-31T03: 2021-07-31T03:15:33.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: 15:32.210899+0000 mgr.x ( 2021-07-31T03:15:33.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: mgr.24542) 3785 : cephadm [ERR] 2021-07-31T03:15:33.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:15:33.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:15:33.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: yield 2021-07-31T03:15:33.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:15:33.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:15:33.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:15:33.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:15:33.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:15:33.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:15:33.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:15:33.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:15:33.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:15:33.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: task.result() 2021-07-31T03:15:33.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:15:33.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: 2021-07-31T03:15:33.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: cephadm 2021-07-31T03:15:32.211311+0000 mgr.x (mgr.24542) 3786 : cephadm [ERR] 2021-07-31T03:15:33.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:15:33.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:15:33.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:15:33.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: yield 2021-07-31T03:15:33.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:15:33.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:15:33.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:15:33.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:15:33.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:15:33.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:15:33.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:15:33.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:15:33.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: 2021-07-31T03:15:33.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:15:33.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:15:33.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:15:33.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:15:33.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:15:33.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:15:33.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:15:33.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:15:33.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:15:33.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:15:33.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:15:33.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:15:33.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:15:33.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:15:33.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:15:33.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:15:33.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:15:33.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:15:33.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: raise 2021-07-31T03:15:33.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:15:33.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:15:33.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:15:33.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:15:33.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:15:33.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: cephadm 2021-07-31T03:15:32.220867+0000 mgr.x (mgr.24542) 3787 : cephadm [ERR] 2021-07-31T03:15:33.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:15:33.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:15:33.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: yield 2021-07-31T03:15:33.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:15:33.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:15:33.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:15:33.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:15:33.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:15:33.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:15:33.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:15:33.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:15:33.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:15:33.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: task.result() 2021-07-31T03:15:33.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:15:33.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:15:33.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: task.result() 2021-07-31T03:15:33.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:15:33.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: 2021-07-31T03:15:33.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:15:33.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: 2021-07-31T03:15:33.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:15:33.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:15:33.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:15:33.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:15:33.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:15:33.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:15:33.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:15:33.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:15:33.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:15:33.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:15:33.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:15:33.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:15:33.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:15:33.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:15:33.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:15:33.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:15:33.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:15:33.585 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:15:33.585 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: raise 2021-07-31T03:15:33.585 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:15:33.585 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:15:33.586 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:15:33.586 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:15:33.586 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:15:33.586 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: cephadm 2021-07-31T03:15:32.220867+0000 mgr.x (mgr.24542) 3787 : cephadm [ERR] 2021-07-31T03:15:33.587 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:15:33.587 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:15:33.587 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: yield 2021-07-31T03:15:33.587 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:15:33.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:15:33.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:15:33.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:15:33.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:15:33.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:15:33.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: cephadm 2021-07-31T03:15:32.221293+0000 mgr.x (mgr.24542) 3788 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:15:33.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:15:33.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:15:33.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: yield 2021-07-31T03:15:33.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:15:33.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:15:33.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:15:33.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:15:33.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:15:33.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:15:33.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:15:33.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:15:33.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:15:33.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: task.result() 2021-07-31T03:15:33.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:15:33.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: 2021-07-31T03:15:33.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:15:33.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: 2021-07-31T03:15:33.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:15:33.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:15:33.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:15:33.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:15:33.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:15:33.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:15:33.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:15:33.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:15:33.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:15:33.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:15:33.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:15:33.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:15:33.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:15:33.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:15:33.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:15:33.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:15:33.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:15:33.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:15:33.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: raise 2021-07-31T03:15:33.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:15:33.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:15:33.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:15:33.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:15:33.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:15:33.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:15:33.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:15:33.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:15:33.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: task.result() 2021-07-31T03:15:33.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:15:33.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: cephadm 2021-07-31T03:15:32.221293+0000 mgr.x (mgr.24542) 3788 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:15:33.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:15:33.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:15:33.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: yield 2021-07-31T03:15:33.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:15:33.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:15:33.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:15:33.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:15:33.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:15:33.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:15:33.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:15:33.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:15:33.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:15:33.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: task.result() 2021-07-31T03:15:33.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:15:33.603 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: 2021-07-31T03:15:33.603 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:15:33.603 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: 2021-07-31T03:15:33.603 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:15:33.603 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:15:33.604 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:15:33.604 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:15:33.604 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:15:33.604 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:15:33.605 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:15:33.605 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:15:33.605 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:15:33.605 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:15:33.605 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:15:33.606 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:15:33.606 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:15:33.606 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:15:33.606 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:15:33.606 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:15:33.607 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:15:33.607 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:15:33.607 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: raise 2021-07-31T03:15:33.607 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:15:33.607 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:15:33.608 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:15:33.608 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:15:33.608 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:33 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:15:34.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:34 smithi154 conmon[34553]: cluster 2021-07-31T03:15:33.060258+0000 mgr.x (mgr.24542) 3789 : cluster [DBG] pgmap v4657: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:34.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:34 smithi045 conmon[39718]: cluster 2021-07-31T03:15:33.060258+0000 mgr.x (mgr.24542) 3789 : cluster [DBG] pgmap v4657: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:34.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:34 smithi045 conmon[33655]: cluster 2021-07-31T03:15:33.060258+0000 mgr.x (mgr. 2021-07-31T03:15:34.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:34 smithi045 conmon[33655]: 24542) 3789 : cluster [DBG] pgmap v4657: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:36.309 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:15:36.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:36 smithi154 conmon[34553]: cluster 2021-07-31T03:15:35.060769+0000 mgr.x (mgr.24542) 3790 : cluster [DBG] pgmap v4658: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:36.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:36 smithi045 conmon[39718]: cluster 2021-07-31T03:15:35.060769+0000 mgr.x 2021-07-31T03:15:36.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:36 smithi045 conmon[39718]: (mgr.24542) 3790 : cluster [DBG] pgmap v4658: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:36.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:36 smithi045 conmon[33655]: cluster 2021-07-31T03:15:35.060769+0000 mgr.x (mgr.24542) 3790 : cluster [DBG] pgmap v4658: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:36.588 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:15:36.589 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 90m ago 91m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:15:36.589 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 88m ago 90m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:15:36.589 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 88m ago 94m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:15:36.589 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 90m ago 96m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:15:36.590 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 90m ago 96m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:15:36.590 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 88m ago 94m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:15:36.590 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 90m ago 94m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:15:36.590 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 90m ago 91m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:15:36.591 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 88m ago 91m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:15:36.591 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 90m ago 94m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:15:36.591 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 90m ago 93m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:15:36.591 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 90m ago 93m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:15:36.591 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 90m ago 93m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:15:36.592 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 88m ago 92m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:15:36.592 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 88m ago 92m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:15:36.592 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 88m ago 92m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:15:36.592 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 88m ago 91m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:15:36.593 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 88m ago 91m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:15:36.929 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:15:36.929 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:15:36.929 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:15:36.929 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:15:36.930 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:15:36.930 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:15:36.930 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:15:36.930 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:15:36.931 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:15:36.931 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:15:36.931 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:15:36.931 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:15:36.931 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:15:36.932 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:15:36.932 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:15:36.932 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:15:36.932 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:15:37.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:37 smithi154 conmon[34553]: audit 2021-07-31T03:15:36.298095+0000 mgr.x (mgr.24542) 3791 : audit [DBG] from='client.29068 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:15:37.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:37 smithi154 conmon[34553]: audit 2021-07-31T03:15:36.581286+0000 mgr.x (mgr.24542) 3792 : audit [DBG] from='client.29074 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:15:37.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:37 smithi154 conmon[34553]: audit 2021-07-31T03:15:36.929508+0000 mon.a (mon.0) 3275 : audit [DBG] from='client.? 172.21.15.45:0/1508151161' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:15:37.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:37 smithi045 conmon[33655]: audit 2021-07-31T03:15:36.298095+0000 mgr.x (mgr 2021-07-31T03:15:37.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:37 smithi045 conmon[33655]: .24542) 3791 : audit [DBG] from='client.29068 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:15:37.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:37 smithi045 conmon[33655]: audit 2021-07-31T03:15:36.581286+0000 mgr.x (mgr.24542 2021-07-31T03:15:37.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:37 smithi045 conmon[33655]: ) 3792 : audit [DBG] from='client.29074 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:15:37.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:37 smithi045 conmon[33655]: audit 2021-07-31T03:15:36. 2021-07-31T03:15:37.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:37 smithi045 conmon[33655]: 929508+0000 mon.a (mon.0) 3275 : audit [DBG] from='client.? 172.21.15.45:0/1508151161' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:15:37.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:37 smithi045 conmon[39718]: audit 2021-07-31T03:15:36. 2021-07-31T03:15:37.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:37 smithi045 conmon[39718]: 298095+0000 mgr.x (mgr.24542) 3791 : audit [DBG] from='client.29068 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:15:37.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:37 smithi045 conmon[39718]: audit 2021-07-31T03:15:37.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:37 smithi045 conmon[39718]: 2021-07-31T03:15:36.581286+0000 mgr.x 2021-07-31T03:15:37.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:37 smithi045 conmon[39718]: (mgr.24542) 3792 : audit [DBG] 2021-07-31T03:15:37.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:37 smithi045 conmon[39718]: from='client.29074 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:15:37.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:37 smithi045 conmon[39718]: audit 2021-07 2021-07-31T03:15:37.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:37 smithi045 conmon[39718]: -31T03:15:36. 2021-07-31T03:15:37.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:37 smithi045 conmon[39718]: 929508+0000 mon.a (mon. 2021-07-31T03:15:37.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:37 smithi045 conmon[39718]: 0) 3275 : audit [DBG] from='client.? 172.21.15.45:0/1508151161' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:15:38.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:38 smithi154 conmon[34553]: cluster 2021-07-31T03:15:37.061151+0000 mgr.x (mgr.24542) 3793 : cluster [DBG] pgmap v4659: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:38.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:38 smithi045 conmon[39718]: cluster 2021-07-31T03:15:37.061151 2021-07-31T03:15:38.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:38 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 3793 : cluster [DBG] pgmap v4659: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:38.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:38 smithi045 conmon[33655]: cluster 2021-07-31T03:15:37.061151+0000 mgr.x (mgr.24542) 3793 : cluster [DBG] pgmap v4659: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:40.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:40 smithi154 conmon[34553]: cluster 2021-07-31T03:15:39.061690+0000 mgr.x (mgr.24542) 3794 : cluster [DBG] pgmap v4660: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:40.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:40 smithi045 conmon[33655]: cluster 2021-07-31T03: 2021-07-31T03:15:40.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:40 smithi045 conmon[33655]: 15:39.061690+0000 mgr.x (mgr.24542) 3794 : cluster [DBG] pgmap v4660: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:40.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:40 smithi045 conmon[39718]: cluster 2021-07-31T03:15: 2021-07-31T03:15:40.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:40 smithi045 conmon[39718]: 39.061690+0000 mgr.x (mgr.24542) 3794 : cluster [DBG] pgmap v4660: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:42.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:42 smithi154 conmon[34553]: cluster 2021-07-31T03:15:41.062157+0000 mgr.x (mgr.24542) 3795 : cluster [DBG] pgmap v4661: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:42.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:42 smithi045 conmon[39718]: cluster 2021-07-31T03:15:41.062157 2021-07-31T03:15:42.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:42 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 3795 : cluster [DBG] pgmap v4661: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:42.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:42 smithi045 conmon[33655]: cluster 2021-07-31T03:15:41.062157+0000 mgr.x 2021-07-31T03:15:42.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:42 smithi045 conmon[33655]: (mgr.24542) 3795 : cluster [DBG] pgmap v4661: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:44.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:44 smithi154 conmon[34553]: cluster 2021-07-31T03:15:43.062540+0000 mgr.x (mgr.24542) 3796 : cluster [DBG] pgmap v4662: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:44.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:44 smithi045 conmon[39718]: cluster 2021-07-31T03:15:43 2021-07-31T03:15:44.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:44 smithi045 conmon[39718]: .062540+0000 mgr.x (mgr.24542) 3796 : cluster [DBG] pgmap v4662: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:44.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:44 smithi045 conmon[33655]: cluster 2021-07-31T03:15:43.062540+0000 mgr.x (mgr.24542) 3796 : 2021-07-31T03:15:44.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:44 smithi045 conmon[33655]: cluster [DBG] pgmap v4662: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:46.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:46 smithi154 conmon[34553]: cluster 2021-07-31T03:15:45.063000+0000 mgr.x (mgr.24542) 3797 : cluster [DBG] pgmap v4663: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:46.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:46 smithi045 conmon[39718]: cluster 2021-07-31T03:15: 2021-07-31T03:15:46.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:46 smithi045 conmon[39718]: 45.063000+0000 mgr.x (mgr.24542) 3797 : cluster [DBG] pgmap v4663: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:46.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:46 smithi045 conmon[33655]: cluster 2021-07-31T03:15:45.063000+0000 mgr.x (mgr.24542) 3797 : cluster [DBG] pgmap v4663: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:48.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:48 smithi154 conmon[34553]: cluster 2021-07-31T03:15:47.063436+0000 mgr.x (mgr.24542) 3798 : cluster [DBG] pgmap v4664: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:48.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:48 smithi045 conmon[39718]: cluster 2021-07-31T03:15:47.063436+0000 mgr.x (mgr.24542) 3798 : cluster [DBG] pgmap v4664: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:48.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:48 smithi045 conmon[33655]: cluster 2021-07-31T03:15:47.063436+0000 mgr.x (mgr.24542) 3798 : 2021-07-31T03:15:48.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:48 smithi045 conmon[33655]: cluster [DBG] pgmap v4664: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:50.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:50 smithi154 conmon[34553]: cluster 2021-07-31T03:15:49.063976+0000 mgr.x (mgr.24542) 3799 : cluster [DBG] pgmap v4665: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:50.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:50 smithi045 conmon[39718]: cluster 2021-07-31T03:15:49.063976+0000 2021-07-31T03:15:50.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:50 smithi045 conmon[39718]: mgr.x (mgr.24542) 3799 : cluster [DBG] pgmap v4665: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:50.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:50 smithi045 conmon[33655]: cluster 2021-07-31T03:15:49.063976+0000 mgr.x (mgr.24542) 3799 : cluster 2021-07-31T03:15:50.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:50 smithi045 conmon[33655]: [DBG] pgmap v4665: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:52.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:52 smithi154 conmon[34553]: cluster 2021-07-31T03:15:51.064435+0000 mgr.x (mgr.24542) 3800 : cluster [DBG] pgmap v4666: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:52.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:52 smithi045 conmon[33655]: cluster 2021-07-31T03:15:51.064435+0000 mgr.x (mgr.24542) 3800 2021-07-31T03:15:52.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:52 smithi045 conmon[33655]: : cluster [DBG] pgmap v4666: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:52.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:52 smithi045 conmon[39718]: cluster 2021-07-31T03:15:51.064435+0000 mgr.x 2021-07-31T03:15:52.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:52 smithi045 conmon[39718]: (mgr.24542) 3800 : cluster [DBG] pgmap v4666: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:54.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:54 smithi154 conmon[34553]: cluster 2021-07-31T03:15:53.064790+0000 mgr.x (mgr.24542) 3801 : cluster [DBG] pgmap v4667: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:54.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:54 smithi154 conmon[34553]: 2021-07-31T03:15:54.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:54 smithi045 conmon[33655]: cluster 2021-07-31T03:15:53.064790+0000 mgr.x (mgr.24542) 3801 : 2021-07-31T03:15:54.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:54 smithi045 conmon[33655]: cluster [DBG] pgmap v4667: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:54.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:54 smithi045 conmon[39718]: cluster 2021-07-31T03:15:53.064790+0000 mgr.x 2021-07-31T03:15:54.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:54 smithi045 conmon[39718]: (mgr.24542) 3801 : cluster [DBG] pgmap v4667: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:56.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:56 smithi154 conmon[34553]: cluster 2021-07-31T03:15:55.065396+0000 mgr.x (mgr.24542) 3802 : cluster [DBG] pgmap v4668: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:56.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:56 smithi045 conmon[39718]: cluster 2021-07-31T03:15: 2021-07-31T03:15:56.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:56 smithi045 conmon[39718]: 55.065396+0000 mgr.x (mgr.24542) 3802 : cluster [DBG] pgmap v4668: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:56.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:56 smithi045 conmon[33655]: cluster 2021-07-31T03:15:55.065396+0000 mgr.x (mgr. 2021-07-31T03:15:56.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:56 smithi045 conmon[33655]: 24542) 3802 : cluster [DBG] pgmap v4668: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:58.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:15:58 smithi154 conmon[34553]: cluster 2021-07-31T03:15:57.065764+0000 mgr.x (mgr.24542) 3803 : cluster [DBG] pgmap v4669: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:58.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:58 smithi045 conmon[39718]: cluster 2021-07-31T03:15:57.065764+0000 mgr.x ( 2021-07-31T03:15:58.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:15:58 smithi045 conmon[39718]: mgr.24542) 3803 : cluster [DBG] pgmap v4669: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:15:58.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:58 smithi045 conmon[33655]: cluster 2021-07-31T03:15:57.065764+0000 mgr.x (mgr.24542) 3803 : 2021-07-31T03:15:58.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:15:58 smithi045 conmon[33655]: cluster [DBG] pgmap v4669: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:00.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:00 smithi154 conmon[34553]: cluster 2021-07-31T03:15:59.066286+0000 mgr.x (mgr.24542) 3804 : cluster [DBG] pgmap v4670: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:00.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:00 smithi045 conmon[39718]: cluster 2021-07-31T03:15:59.066286+0000 mgr.x 2021-07-31T03:16:00.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:00 smithi045 conmon[39718]: (mgr.24542) 3804 : cluster [DBG] pgmap v4670: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:00.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:00 smithi045 conmon[33655]: cluster 2021-07-31T03:15:59.066286+0000 mgr.x (mgr.24542) 3804 : 2021-07-31T03:16:00.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:00 smithi045 conmon[33655]: cluster [DBG] pgmap v4670: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:02.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:02 smithi154 conmon[34553]: cluster 2021-07-31T03:16:01.066775+0000 mgr.x (mgr.24542) 3805 : cluster [DBG] pgmap v4671: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:02.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:02 smithi045 conmon[39718]: cluster 2021-07-31T03:16:01.066775+0000 mgr.x ( 2021-07-31T03:16:02.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:02 smithi045 conmon[39718]: mgr.24542) 3805 : cluster [DBG] pgmap v4671: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:02.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:02 smithi045 conmon[33655]: cluster 2021-07-31T03:16:01.066775+0000 mgr.x (mgr. 2021-07-31T03:16:02.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:02 smithi045 conmon[33655]: 24542) 3805 : cluster [DBG] pgmap v4671: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:04.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:04 smithi154 conmon[34553]: cluster 2021-07-31T03:16:03.067123+0000 mgr.x (mgr.24542) 3806 : cluster 2021-07-31T03:16:04.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:04 smithi154 conmon[34553]: [DBG] pgmap v4672: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:04.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:04 smithi045 conmon[33655]: cluster 2021-07-31T03:16:03.067123+0000 mgr.x (mgr.24542) 3806 : cluster 2021-07-31T03:16:04.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:04 smithi045 conmon[33655]: [DBG] pgmap v4672: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:04.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:04 smithi045 conmon[39718]: cluster 2021-07-31T03:16:03.067123+0000 2021-07-31T03:16:04.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:04 smithi045 conmon[39718]: mgr.x (mgr.24542) 3806 : cluster [DBG] pgmap v4672: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:06.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:06 smithi154 conmon[34553]: cluster 2021-07-31T03:16:05.067720+0000 mgr.x (mgr.24542) 3807 : cluster [DBG] 2021-07-31T03:16:06.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:06 smithi154 conmon[34553]: pgmap v4673: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:06.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:06 smithi045 conmon[33655]: cluster 2021-07-31T03:16:05.067720+0000 2021-07-31T03:16:06.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:06 smithi045 conmon[33655]: mgr.x (mgr.24542) 3807 : cluster [DBG] pgmap v4673: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:06.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:06 smithi045 conmon[39718]: cluster 2021-07-31T03:16:05.067720+0000 mgr.x (mgr.24542) 3807 2021-07-31T03:16:06.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:06 smithi045 conmon[39718]: : cluster [DBG] pgmap v4673: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:07.239 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:16:07.514 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:16:07.515 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 91m ago 91m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:16:07.515 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 88m ago 91m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:16:07.515 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 88m ago 95m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:16:07.515 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 91m ago 97m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:16:07.516 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 91m ago 97m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:16:07.516 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 88m ago 95m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:16:07.516 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 91m ago 95m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:16:07.516 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 91m ago 91m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:16:07.516 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 88m ago 91m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:16:07.517 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 91m ago 94m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:16:07.517 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 91m ago 94m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:16:07.517 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 91m ago 94m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:16:07.517 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 91m ago 93m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:16:07.518 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 88m ago 93m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:16:07.518 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 88m ago 93m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:16:07.518 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 88m ago 92m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:16:07.518 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 88m ago 92m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:16:07.519 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 88m ago 92m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:16:07.848 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:16:07.848 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:16:07.849 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:16:07.849 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:16:07.849 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:16:07.849 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:16:07.850 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:16:07.850 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:16:07.850 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:16:07.850 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:16:07.851 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:16:07.851 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:16:07.851 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:16:07.851 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:16:07.851 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:16:07.852 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:16:07.852 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:16:08.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:08 smithi154 conmon[34553]: cluster 2021-07-31T03:16:07.068257+0000 mgr.x (mgr.24542) 3808 : cluster 2021-07-31T03:16:08.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:08 smithi154 conmon[34553]: [DBG] pgmap v4674: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:08.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:08 smithi154 conmon[34553]: audit 2021-07-31T03:16:07.229031+0000 mgr.x (mgr.24542) 3809 : audit [DBG] from='client.19146 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:16:08.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:08 smithi154 conmon[34553]: audit 2021-07-31T03:16:07.507433+0000 mgr.x (mgr.24542) 3810 : audit [DBG] from='client.19152 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:16:08.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:08 smithi154 conmon[34553]: audit 2021-07-31T03:16:08.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:08 smithi154 conmon[34553]: 2021-07-31T03:16:07.848952+0000 mon.c (mon 2021-07-31T03:16:08.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:08 smithi154 conmon[34553]: .1) 191 : audit [DBG] from='client.? 172.21.15.45:0/3160025640' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:16:08.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:08 smithi045 conmon[39718]: cluster 2021-07-31T03:16:07.068257+0000 mgr.x (mgr.24542) 2021-07-31T03:16:08.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:08 smithi045 conmon[39718]: 3808 : cluster [DBG] pgmap v4674: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:08.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:08 smithi045 conmon[39718]: audit 2021-07-31T03:16:07.229031+0000 mgr.x (mgr.24542) 3809 : audit [DBG] from='client.19146 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:16:08.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:08 smithi045 conmon[39718]: 2021-07-31T03:16:08.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:08 smithi045 conmon[39718]: audit 2021-07-31T03:16:07. 2021-07-31T03:16:08.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:08 smithi045 conmon[39718]: 507433+0000 mgr.x (mgr.24542) 3810 : audit [DBG] from='client.19152 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:16:08.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:08 smithi045 conmon[39718]: audit 2021-07-31T03:16:07 2021-07-31T03:16:08.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:08 smithi045 conmon[39718]: .848952+0000 mon.c (mon.1) 191 : audit [DBG] from='client.? 172.21.15.45:0/3160025640' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:16:08.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:08 smithi045 conmon[33655]: cluster 2021-07-31T03:16:07.068257 2021-07-31T03:16:08.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:08 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 3808 : cluster [DBG] pgmap v4674: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:08.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:08 smithi045 conmon[33655]: audit 2021-07- 2021-07-31T03:16:08.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:08 smithi045 conmon[33655]: 31T03:16:07.229031+0000 mgr.x (mgr.24542) 3809 : audit [DBG] from='client.19146 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:16:08.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:08 smithi045 conmon[33655]: 2021-07-31T03:16:08.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:08 smithi045 conmon[33655]: audit 2021-07-31T03:16:08.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:08 smithi045 conmon[33655]: 2021-07- 2021-07-31T03:16:08.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:08 smithi045 conmon[33655]: 31T03:16:07.507433+0000 mgr.x (mgr.24542 2021-07-31T03:16:08.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:08 smithi045 conmon[33655]: ) 3810 : audit [DBG] from='client.19152 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:16:08.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:08 smithi045 conmon[33655]: audit 2021-07-31T03:16: 2021-07-31T03:16:08.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:08 smithi045 conmon[33655]: 07.848952+0000 mon.c (mon.1) 191 : audit [DBG] 2021-07-31T03:16:08.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:08 smithi045 conmon[33655]: from='client.? 172.21.15.45:0/3160025640' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:16:10.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:10 smithi154 conmon[34553]: cluster 2021-07-31T03:16:09.068865+0000 mgr.x (mgr.24542) 3811 : cluster [DBG] pgmap v4675: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:10.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:10 smithi154 conmon[34553]: 2021-07-31T03:16:10.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:10 smithi045 conmon[39718]: cluster 2021-07-31T03:16:09.068865+0000 mgr.x (mgr.24542) 3811 : cluster [DBG] pgmap v4675: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:10.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:10 smithi045 conmon[33655]: cluster 2021-07-31T03:16:09.068865+0000 mgr.x (mgr.24542) 3811 : cluster [DBG] 2021-07-31T03:16:10.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:10 smithi045 conmon[33655]: pgmap v4675: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:12.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:12 smithi154 conmon[34553]: cluster 2021-07-31T03:16:11.069287+0000 mgr.x (mgr.24542) 3812 : cluster [DBG] 2021-07-31T03:16:12.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:12 smithi154 conmon[34553]: pgmap v4676: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:12.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:12 smithi045 conmon[39718]: cluster 2021-07-31T03:16:11.069287+0000 mgr.x (mgr.24542) 3812 : cluster [DBG] pgmap v4676: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:12.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:12 smithi045 conmon[33655]: cluster 2021-07-31T03:16:11.069287+0000 mgr.x (mgr.24542) 3812 : cluster [DBG] pgmap v4676: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:14.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:14 smithi154 conmon[34553]: cluster 2021-07-31T03:16:13.069632+0000 mgr.x (mgr.24542) 3813 : cluster [DBG] pgmap v4677: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:14.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:14 smithi154 conmon[34553]: 2021-07-31T03:16:14.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:14 smithi045 conmon[39718]: cluster 2021-07-31T03:16:13.069632+0000 mgr.x (mgr.24542) 3813 : cluster [DBG] pgmap v4677: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:14.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:14 smithi045 conmon[33655]: cluster 2021-07-31T03:16:13.069632+0000 mgr.x ( 2021-07-31T03:16:14.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:14 smithi045 conmon[33655]: mgr.24542) 3813 : cluster [DBG] pgmap v4677: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:16.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:16 smithi154 conmon[34553]: cluster 2021-07-31T03:16:15.070121+0000 mgr.x (mgr.24542) 3814 : cluster [DBG] 2021-07-31T03:16:16.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:16 smithi154 conmon[34553]: pgmap v4678: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:16.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:16 smithi045 conmon[39718]: cluster 2021-07-31T03:16:15.070121+0000 mgr.x ( 2021-07-31T03:16:16.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:16 smithi045 conmon[39718]: mgr.24542) 3814 : cluster [DBG] pgmap v4678: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:16.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:16 smithi045 conmon[33655]: cluster 2021-07-31T03:16:15.070121+0000 mgr.x (mgr.24542 2021-07-31T03:16:16.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:16 smithi045 conmon[33655]: ) 3814 : cluster [DBG] pgmap v4678: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:18.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:18 smithi154 conmon[34553]: cluster 2021-07-31T03:16:17.070419+0000 mgr.x (mgr.24542) 3815 : cluster 2021-07-31T03:16:18.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:18 smithi154 conmon[34553]: [DBG] pgmap v4679: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:18.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:18 smithi045 conmon[39718]: cluster 2021-07-31T03:16:17.070419+0000 mgr.x (mgr. 2021-07-31T03:16:18.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:18 smithi045 conmon[39718]: 24542) 3815 : cluster [DBG] pgmap v4679: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:18.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:18 smithi045 conmon[33655]: cluster 2021-07-31T03:16:17.070419+0000 mgr.x (mgr. 2021-07-31T03:16:18.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:18 smithi045 conmon[33655]: 24542) 3815 : cluster [DBG] pgmap v4679: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:20.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:20 smithi154 conmon[34553]: cluster 2021-07-31T03:16:19.070836+0000 mgr.x (mgr.24542) 3816 : cluster [DBG] 2021-07-31T03:16:20.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:20 smithi154 conmon[34553]: pgmap v4680: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:20.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:20 smithi045 conmon[39718]: cluster 2021-07-31T03:16:19.070836+0000 mgr.x (mgr 2021-07-31T03:16:20.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:20 smithi045 conmon[39718]: .24542) 3816 : cluster [DBG] pgmap v4680: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:20.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:20 smithi045 conmon[33655]: cluster 2021-07-31T03:16:19.070836+0000 mgr.x (mgr.24542) 3816 : cluster [DBG] pgmap v4680: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:20.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:20 smithi045 conmon[33655]: 2021-07-31T03:16:22.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:22 smithi154 conmon[34553]: cluster 2021-07-31T03:16:21.071333+0000 mgr.x (mgr.24542) 3817 : cluster [DBG] 2021-07-31T03:16:22.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:22 smithi154 conmon[34553]: pgmap v4681: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:22.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:22 smithi045 conmon[39718]: cluster 2021-07-31T03:16:21.071333+0000 mgr.x (mgr.24542) 3817 : cluster [DBG] pgmap v4681: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:22.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:22 smithi045 conmon[33655]: cluster 2021-07-31T03:16:21.071333+0000 mgr.x (mgr.24542) 3817 : cluster [DBG] pgmap v4681: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:22.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:22 smithi045 conmon[33655]: 2021-07-31T03:16:24.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:24 smithi154 conmon[34553]: cluster 2021-07-31T03:16:23.071735+0000 mgr.x (mgr.24542) 3818 : cluster 2021-07-31T03:16:24.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:24 smithi154 conmon[34553]: [DBG] pgmap v4682: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:24.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:24 smithi045 conmon[39718]: cluster 2021-07-31T03:16:23.071735+0000 mgr.x (mgr.24542) 3818 : cluster [DBG] pgmap v4682: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:24.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:24 smithi045 conmon[33655]: cluster 2021-07-31T03:16:23.071735+0000 mgr.x (mgr.24542) 3818 : cluster [DBG] pgmap v4682: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:26.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:26 smithi154 conmon[34553]: cluster 2021-07-31T03:16:25.072152+0000 mgr.x (mgr.24542) 3819 : cluster [DBG] pgmap v4683: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:26.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:26 smithi045 conmon[39718]: cluster 2021-07-31T03:16:25.072152+0000 mgr.x (mgr.24542) 3819 : cluster [DBG] pgmap v4683: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:26.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:26 smithi045 conmon[33655]: cluster 2021-07-31T03:16:25.072152+0000 mgr.x (mgr.24542) 3819 : cluster [DBG] pgmap v4683: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:28.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:28 smithi154 conmon[34553]: cluster 2021-07-31T03:16:27.072534+0000 mgr.x (mgr.24542) 3820 : cluster [DBG] pgmap v4684: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:28.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:28 smithi045 conmon[33655]: cluster 2021-07-31T03:16:27.072534+0000 mgr.x (mgr.24542) 3820 : cluster [DBG] pgmap v4684: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:28.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:28 smithi045 conmon[39718]: cluster 2021-07-31T03:16:27.072534+0000 2021-07-31T03:16:28.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:28 smithi045 conmon[39718]: mgr.x (mgr.24542) 3820 : cluster [DBG] pgmap v4684: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:29.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:29 smithi154 conmon[34553]: audit 2021-07-31T03:16:28.289514+0000 mon.b (mon.2) 520 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:16:29.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:29 smithi154 conmon[34553]: audit 2021-07-31T03:16:28.289771+0000 mon.b (mon.2) 521 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:16:29.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:29 smithi154 conmon[34553]: audit 2021-07-31T03:16:28.290317+0000 mon.a (mon.0) 2021-07-31T03:16:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:29 smithi154 conmon[34553]: 3276 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:16:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:29 smithi154 conmon[34553]: audit 2021- 2021-07-31T03:16:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:29 smithi154 conmon[34553]: 07-31T03:16:28.290889 2021-07-31T03:16:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:29 smithi154 conmon[34553]: +0000 mon.a (mon.0) 3277 : 2021-07-31T03:16:29.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:29 smithi154 conmon[34553]: audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:16:29.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:29 smithi045 conmon[39718]: audit 2021-07-31T03:16:28.289514+0000 mon.b (mon. 2021-07-31T03:16:29.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:29 smithi045 conmon[39718]: 2) 520 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:16:29.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:29 smithi045 conmon[39718]: audit 2021-07-31T03: 2021-07-31T03:16:29.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:29 smithi045 conmon[39718]: 16:28.289771+0000 mon.b (mon.2) 521 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:16:29.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:29 smithi045 conmon[39718]: audit 2021-07 2021-07-31T03:16:29.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:29 smithi045 conmon[39718]: -31T03:16:28.290317+0000 mon.a (mon 2021-07-31T03:16:29.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:29 smithi045 conmon[39718]: .0) 3276 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:16:29.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:29 smithi045 conmon[39718]: audit 2021-07-31T03 2021-07-31T03:16:29.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:29 smithi045 conmon[39718]: :16:28.290889+0000 mon.a (mon.0) 3277 2021-07-31T03:16:29.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:29 smithi045 conmon[39718]: : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:16:29.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:29 smithi045 conmon[33655]: audit 2021-07-31T03:16:28.289514+0000 mon.b (mon.2) 520 : audit [INF] 2021-07-31T03:16:29.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:29 smithi045 conmon[33655]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:16:29.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:29 smithi045 conmon[33655]: audit 2021-07-31T03:16:28.289771+0000 mon.b (mon.2) 521 : audit 2021-07-31T03:16:29.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:29 smithi045 conmon[33655]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:16:29.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:29 smithi045 conmon[33655]: audit 2021-07-31T03:16:29.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:29 smithi045 conmon[33655]: 2021-07-31T03: 2021-07-31T03:16:29.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:29 smithi045 conmon[33655]: 16:28.290317+0000 mon.a 2021-07-31T03:16:29.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:29 smithi045 conmon[33655]: (mon.0) 3276 : audit 2021-07-31T03:16:29.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:29 smithi045 conmon[33655]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:16:29.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:29 smithi045 conmon[33655]: audit 2021- 2021-07-31T03:16:29.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:29 smithi045 conmon[33655]: 07-31T03:16:28.290889+0000 mon.a (mon 2021-07-31T03:16:29.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:29 smithi045 conmon[33655]: .0) 3277 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:16:30.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:30 smithi154 conmon[34553]: cluster 2021-07-31T03:16:29.072958+0000 mgr.x (mgr.24542) 3821 : cluster [DBG] 2021-07-31T03:16:30.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:30 smithi154 conmon[34553]: pgmap v4685: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:30.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:30 smithi045 conmon[39718]: cluster 2021-07-31T03:16:29.072958+0000 mgr.x (mgr.24542) 3821 : cluster [DBG] pgmap v4685: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:30.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:30 smithi045 conmon[33655]: cluster 2021-07-31T03:16:29.072958+0000 mgr.x (mgr.24542 2021-07-31T03:16:30.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:30 smithi045 conmon[33655]: ) 3821 : cluster [DBG] pgmap v4685: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:32.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[34553]: cluster 2021-07-31T03:16:31.073445+0000 mgr.x (mgr.24542) 3822 : cluster [DBG] pgmap v4686: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:32.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[34553]: 2021-07-31T03:16:32.503 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: debug 2021-07-31T03:16:32.267+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:16:32.503 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:16:32.503 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:16:32.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: yield 2021-07-31T03:16:32.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:16:32.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:16:32.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:16:32.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:16:32.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:16:32.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:16:32.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:16:32.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:16:32.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:16:32.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: task.result() 2021-07-31T03:16:32.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:16:32.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: debug 2021-07-31T03:16:32.269+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:16:32.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:16:32.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:16:32.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: yield 2021-07-31T03:16:32.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:16:32.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:16:32.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:16:32.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:16:32.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:16:32.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:16:32.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:16:32.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:16:32.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:16:32.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: task.result() 2021-07-31T03:16:32.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:16:32.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: debug 2021-07-31T03:16:32.304+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:16:32.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:16:32.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:16:32.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: yield 2021-07-31T03:16:32.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:16:32.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:16:32.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:16:32.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:16:32.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:16:32.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:16:32.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:16:32.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:16:32.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:16:32.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: task.result() 2021-07-31T03:16:32.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:16:32.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: debug 2021-07-31T03:16:32.305+0000 7fcb97969700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:16:32.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:16:32.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:16:32.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: yield 2021-07-31T03:16:32.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:16:32.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:16:32.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:16:32.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:16:32.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:16:32.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:16:32.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:16:32.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:16:32.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:16:32.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: task.result() 2021-07-31T03:16:32.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:16:32.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: 2021-07-31T03:16:32.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:16:32.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: 2021-07-31T03:16:32.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:16:32.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:16:32.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:16:32.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:16:32.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:16:32.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:16:32.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:16:32.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:16:32.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:16:32.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:16:32.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:16:32.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:16:32.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:16:32.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:16:32.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:16:32.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:16:32.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:16:32.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:16:32.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: raise 2021-07-31T03:16:32.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:16:32.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:16:32.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:16:32.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:16:32.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:16:32.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: debug 2021-07-31T03:16:32.312+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:16:32.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:16:32.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:16:32.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: yield 2021-07-31T03:16:32.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:16:32.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:16:32.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:16:32.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:16:32.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:16:32.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:16:32.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:16:32.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:16:32.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:16:32.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: task.result() 2021-07-31T03:16:32.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:16:32.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: debug 2021-07-31T03:16:32.312+0000 7fcb9816a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:16:32.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:16:32.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:16:32.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: yield 2021-07-31T03:16:32.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:16:32.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:16:32.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:16:32.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:16:32.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:16:32.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:16:32.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:16:32.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:16:32.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:16:32.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: task.result() 2021-07-31T03:16:32.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:16:32.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: 2021-07-31T03:16:32.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:16:32.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: 2021-07-31T03:16:32.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:16:32.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:16:32.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:16:32.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:16:32.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:16:32.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:16:32.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:16:32.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:16:32.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:16:32.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:16:32.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:16:32.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:16:32.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:16:32.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:16:32.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:16:32.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:16:32.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:16:32.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:16:32.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: raise 2021-07-31T03:16:32.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:16:32.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:16:32.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:16:32.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:16:32.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:16:32 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:16:32.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:32 smithi045 conmon[39718]: cluster 2021-07-31T03:16:31.073445+0000 2021-07-31T03:16:32.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:32 smithi045 conmon[39718]: mgr.x (mgr.24542) 3822 : cluster [DBG] pgmap v4686: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:32.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:32 smithi045 conmon[33655]: cluster 2021-07-31T03:16:31.073445+0000 mgr.x (mgr.24542) 3822 2021-07-31T03:16:32.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:32 smithi045 conmon[33655]: : cluster [DBG] pgmap v4686: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:33.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: audit 2021-07-31T03:16:32.222041+0000 mon.b (mon.2) 522 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:16:33.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: 2021-07-31T03:16:33.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: audit 2021-07-31T03:16:32.223469+0000 mon.b (mon.2) 523 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:16:33.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: audit 2021-07-31T03:16:32.224408+0000 mon.b (mon.2) 524 : audit 2021-07-31T03:16:33.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:16:33.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:16 2021-07-31T03:16:33.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: :32.225179+0000 mgr.x (mgr.24542) 3823 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:16:33.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: cephadm 2021-07-31T03 2021-07-31T03:16:33.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: :16:32.268195+0000 mgr.x (mgr.24542) 3824 : cephadm 2021-07-31T03:16:33.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: [ERR] 2021-07-31T03:16:33.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:16:33.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:16:33.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: yield 2021-07-31T03:16:33.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:16:33.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:16:33.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:16:33.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:16:33.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:16:33.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:16:33.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:16:33.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:16:33.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:16:33.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: task.result() 2021-07-31T03:16:33.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:16:33.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:16:32.269436+0000 mgr.x (mgr.24542) 3825 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:16:33.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:16:32.270243+0000 mgr.x (mgr.24542) 3826 : cephadm [ERR] 2021-07-31T03:16:33.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:16:33.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:16:33.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: yield 2021-07-31T03:16:33.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:16:33.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:16:33.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:16:33.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:16:33.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:16:33.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:16:33.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:16:33.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:16:33.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:16:33.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: task.result() 2021-07-31T03:16:33.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:16:33.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: 2021-07-31T03:16:33.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:16:32.280154+0000 mgr.x (mgr.24542) 3827 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:16:33.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:16:32.305765+0000 mgr.x (mgr.24542) 3828 : 2021-07-31T03:16:33.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: cephadm [ERR] 2021-07-31T03:16:33.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:16:33.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:16:33.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: yield 2021-07-31T03:16:33.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:16:33.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:16:33.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:16:33.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:16:33.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:16:33.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:16:33.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:16:33.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:16:33.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:16:33.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: task.result() 2021-07-31T03:16:33.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:16:33.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:16:32.306245+0000 mgr.x (mgr.24542) 3829 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:16:33.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:16:33.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:16:33.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: yield 2021-07-31T03:16:33.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:16:33.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:16:33.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:16:33.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:16:33.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:16:33.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:16:33.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:16:33.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:16:33.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:16:33.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: task.result() 2021-07-31T03:16:33.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:16:33.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: 2021-07-31T03:16:33.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:16:33.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: 2021-07-31T03:16:33.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:16:33.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:16:33.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:16:33.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:16:33.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:16:33.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:16:33.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:16:33.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:16:33.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:16:33.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:16:33.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:16:33.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:16:33.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:16:33.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:16:33.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:16:33.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:16:33.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:16:33.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:16:33.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: raise 2021-07-31T03:16:33.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:16:33.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:16:33.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:16:33.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:16:33.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:16:33.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:16:32.313486+0000 mgr.x 2021-07-31T03:16:33.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: (mgr.24542) 3830 : cephadm [ERR] 2021-07-31T03:16:33.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:16:33.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:16:33.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: yield 2021-07-31T03:16:33.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:16:33.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:16:33.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:16:33.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:16:33.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:16:33.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:16:33.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:16:33.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:16:33.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:16:33.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: task.result() 2021-07-31T03:16:33.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:16:33.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:16:32.313884+0000 mgr.x (mgr.24542) 3831 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:16:33.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:16:33.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:16:33.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: yield 2021-07-31T03:16:33.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:16:33.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:16:33.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:16:33.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:16:33.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:16:33.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:16:33.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:16:33.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:16:33.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:16:33.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: task.result() 2021-07-31T03:16:33.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:16:33.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: 2021-07-31T03:16:33.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:16:33.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: 2021-07-31T03:16:33.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:16:33.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:16:33.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:16:33.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:16:33.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:16:33.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:16:33.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:16:33.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:16:33.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:16:33.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:16:33.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:16:33.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:16:33.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:16:33.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:16:33.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:16:33.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:16:33.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:16:33.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:16:33.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: raise 2021-07-31T03:16:33.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:16:33.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:16:33.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:16:33.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:16:33.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:33 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:16:33.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: audit 2021-07-31T03:16:32.222041+0000 mon.b (mon.2) 522 : audit [DBG] 2021-07-31T03:16:33.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:16:33.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: audit 2021-07-31T03:16:32.223469+0000 mon.b (mon.2) 523 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:16:33.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: audit 2021-07-31T03:16 2021-07-31T03:16:33.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: :32.224408+0000 mon.b (mon.2) 524 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:16:33.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: cephadm 2021-07-31T03:16:33.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: 2021-07-31T03:16:32.225179+0000 mgr.x (mgr.24542) 3823 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:16:33.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: cephadm 2021-07 2021-07-31T03:16:33.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: -31T03:16:32.268195+0000 mgr.x (mgr. 2021-07-31T03:16:33.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: 24542) 3824 : cephadm [ERR] 2021-07-31T03:16:33.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:16:33.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:16:33.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: yield 2021-07-31T03:16:33.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:16:33.539 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:16:33.539 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:16:33.539 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:16:33.539 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:16:33.539 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:16:33.539 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:16:33.540 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:16:33.540 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:16:33.540 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: task.result() 2021-07-31T03:16:33.540 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:16:33.540 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: cephadm 2021-07-31T03:16:32.269436+0000 mgr.x (mgr.24542) 3825 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:16:33.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: cephadm 2021-07-31T03:16:32.270243+0000 mgr.x (mgr.24542) 3826 : cephadm [ERR] 2021-07-31T03:16:33.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:16:33.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:16:33.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: yield 2021-07-31T03:16:33.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:16:33.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:16:33.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:16:33.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:16:33.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:16:33.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:16:33.543 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:16:33.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: audit 2021-07-31T03: 2021-07-31T03:16:33.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: 16:32.222041+0000 mon.b (mon.2) 522 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:16:33.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: audit 2021-07-31T03:16: 2021-07-31T03:16:33.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: 32.223469+0000 mon.b (mon.2) 523 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:16:33.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: audit 2021-07-31T03:16:33.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: 2021-07-31 2021-07-31T03:16:33.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: T03:16:32.224408+0000 2021-07-31T03:16:33.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: mon.b (mon.2) 524 : 2021-07-31T03:16:33.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:16:33.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: cephadm 2021 2021-07-31T03:16:33.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: -07-31T03:16:32. 2021-07-31T03:16:33.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: 225179+0000 mgr.x (mgr. 2021-07-31T03:16:33.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: 24542) 3823 : cephadm [WRN] 2021-07-31T03:16:33.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:16:33.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: cephadm 2021- 2021-07-31T03:16:33.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: 07-31T03:16:32. 2021-07-31T03:16:33.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: 268195+0000 mgr.x (mgr.24542 2021-07-31T03:16:33.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: ) 3824 : cephadm [ERR] 2021-07-31T03:16:33.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:16:33.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:16:33.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: yield 2021-07-31T03:16:33.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:16:33.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:16:33.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:16:33.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:16:33.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:16:33.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:16:33.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:16:33.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:16:33.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:16:33.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: task.result() 2021-07-31T03:16:33.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:16:33.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: 2021-07-31T03:16:33.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: cephadm 2021-07-31T03:16:32.269436+0000 mgr.x 2021-07-31T03:16:33.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: (mgr.24542) 3825 : cephadm [INF] 2021-07-31T03:16:33.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:16:33.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: cephadm 2021- 2021-07-31T03:16:33.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: 07-31T03:16:32.270243 2021-07-31T03:16:33.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 2021-07-31T03:16:33.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: 3826 : cephadm [ERR] 2021-07-31T03:16:33.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:16:33.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:16:33.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: yield 2021-07-31T03:16:33.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:16:33.552 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:16:33.552 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:16:33.552 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: task.result() 2021-07-31T03:16:33.553 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:16:33.553 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: cephadm 2021-07-31T03:16:32.280154+0000 mgr.x (mgr.24542) 3827 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:16:33.553 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: cephadm 2021-07-31T03:16:32.305765+0000 mgr.x (mgr.24542) 3828 : cephadm [ERR] 2021-07-31T03:16:33.553 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:16:33.553 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:16:33.554 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: yield 2021-07-31T03:16:33.554 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:16:33.554 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:16:33.554 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:16:33.554 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:16:33.555 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:16:33.555 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:16:33.555 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:16:33.555 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:16:33.555 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:16:33.556 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: task.result() 2021-07-31T03:16:33.556 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:16:33.556 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: cephadm 2021-07-31T03:16:32.306245 2021-07-31T03:16:33.556 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 3829 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:16:33.556 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:16:33.556 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:16:33.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: yield 2021-07-31T03:16:33.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:16:33.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:16:33.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:16:33.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:16:33.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:16:33.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:16:33.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:16:33.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:16:33.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:16:33.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: task.result() 2021-07-31T03:16:33.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:16:33.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: 2021-07-31T03:16:33.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:16:33.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: 2021-07-31T03:16:33.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:16:33.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:16:33.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:16:33.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:16:33.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:16:33.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:16:33.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:16:33.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:16:33.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:16:33.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:16:33.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:16:33.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: task.result() 2021-07-31T03:16:33.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:16:33.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: cephadm 2021-07-31T03:16:32.280154+0000 mgr.x (mgr 2021-07-31T03:16:33.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: .24542) 3827 : cephadm [INF] 2021-07-31T03:16:33.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:16:33.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: cephadm 2021 2021-07-31T03:16:33.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: -07-31T03:16:32 2021-07-31T03:16:33.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: .305765+0000 mgr.x (mgr.24542 2021-07-31T03:16:33.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: ) 3828 : cephadm [ERR] 2021-07-31T03:16:33.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:16:33.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:16:33.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: yield 2021-07-31T03:16:33.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:16:33.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:16:33.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:16:33.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:16:33.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:16:33.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:16:33.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:16:33.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:16:33.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:16:33.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: task.result() 2021-07-31T03:16:33.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:16:33.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: 2021-07-31T03:16:33.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: cephadm 2021-07-31T03:16:32.306245+0000 mgr.x (mgr.24542) 3829 : 2021-07-31T03:16:33.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:16:33.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:16:33.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:16:33.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: yield 2021-07-31T03:16:33.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:16:33.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:16:33.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:16:33.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:16:33.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:16:33.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:16:33.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:16:33.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:16:33.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:16:33.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:16:33.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:16:33.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:16:33.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:16:33.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:16:33.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:16:33.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:16:33.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:16:33.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:16:33.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:16:33.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:16:33.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:16:33.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: raise 2021-07-31T03:16:33.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:16:33.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:16:33.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:16:33.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:16:33.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:16:33.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: cephadm 2021-07-31T03:16:32.313486+0000 mgr.x (mgr.24542) 3830 : cephadm [ERR] 2021-07-31T03:16:33.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:16:33.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:16:33.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: yield 2021-07-31T03:16:33.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:16:33.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:16:33.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:16:33.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:16:33.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:16:33.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:16:33.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:16:33.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:16:33.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:16:33.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: task.result() 2021-07-31T03:16:33.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:16:33.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: cephadm 2021-07-31T03:16:32. 2021-07-31T03:16:33.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: 313884+0000 mgr.x (mgr.24542) 3831 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:16:33.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:16:33.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:16:33.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:16:33.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:16:33.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: task.result() 2021-07-31T03:16:33.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:16:33.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: 2021-07-31T03:16:33.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:16:33.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: 2021-07-31T03:16:33.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:16:33.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:16:33.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:16:33.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:16:33.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:16:33.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:16:33.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:16:33.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:16:33.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:16:33.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:16:33.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:16:33.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:16:33.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:16:33.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:16:33.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:16:33.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:16:33.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:16:33.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:16:33.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: raise 2021-07-31T03:16:33.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:16:33.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:16:33.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:16:33.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:16:33.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:16:33.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: cephadm 2021-07-31T03:16:32.313486+0000 mgr.x (mgr.24542) 3830 : cephadm [ERR] 2021-07-31T03:16:33.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:16:33.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:16:33.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: yield 2021-07-31T03:16:33.585 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:16:33.585 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:16:33.585 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:16:33.585 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:16:33.586 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:16:33.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:16:33.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: yield 2021-07-31T03:16:33.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:16:33.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:16:33.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:16:33.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:16:33.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:16:33.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:16:33.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:16:33.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:16:33.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:16:33.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: task.result() 2021-07-31T03:16:33.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:16:33.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: 2021-07-31T03:16:33.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:16:33.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: 2021-07-31T03:16:33.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:16:33.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:16:33.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:16:33.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:16:33.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:16:33.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:16:33.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:16:33.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:16:33.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:16:33.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:16:33.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:16:33.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:16:33.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:16:33.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:16:33.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:16:33.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:16:33.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:16:33.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:16:33.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: raise 2021-07-31T03:16:33.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:16:33.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:16:33.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:16:33.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:16:33.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:16:33.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:16:33.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:16:33.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:16:33.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:16:33.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: task.result() 2021-07-31T03:16:33.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:16:33.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: cephadm 2021-07-31T03:16:32.313884+0000 mgr.x (mgr.24542) 3831 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:16:33.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:16:33.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:16:33.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: yield 2021-07-31T03:16:33.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:16:33.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:16:33.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:16:33.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:16:33.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:16:33.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:16:33.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:16:33.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:16:33.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:16:33.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: task.result() 2021-07-31T03:16:33.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:16:33.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: 2021-07-31T03:16:33.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:16:33.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: 2021-07-31T03:16:33.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:16:33.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:16:33.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:16:33.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:16:33.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:16:33.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:16:33.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:16:33.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:16:33.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:16:33.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:16:33.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:16:33.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:16:33.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:16:33.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:16:33.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:16:33.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:16:33.603 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:16:33.603 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:16:33.603 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: raise 2021-07-31T03:16:33.603 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:16:33.603 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:16:33.604 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:16:33.604 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:16:33.604 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:33 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:16:34.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:34 smithi154 conmon[34553]: cluster 2021-07-31T03:16:33.073830+0000 mgr.x (mgr.24542) 3832 : cluster [DBG] pgmap v4687: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:34.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:34 smithi045 conmon[39718]: cluster 2021-07-31T03:16:33.073830+0000 mgr.x (mgr. 2021-07-31T03:16:34.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:34 smithi045 conmon[39718]: 24542) 3832 : cluster [DBG] pgmap v4687: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:34.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:34 smithi045 conmon[33655]: cluster 2021-07-31T03:16:33.073830+0000 mgr.x (mgr.24542) 3832 2021-07-31T03:16:34.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:34 smithi045 conmon[33655]: : cluster [DBG] pgmap v4687: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:36.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:36 smithi154 conmon[34553]: cluster 2021-07-31T03:16:35.074349+0000 mgr.x (mgr.24542) 3833 : cluster [DBG] pgmap v4688: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:36.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:36 smithi045 conmon[39718]: cluster 2021-07-31T03:16:35.074349+0000 mgr.x (mgr 2021-07-31T03:16:36.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:36 smithi045 conmon[39718]: .24542) 3833 : cluster [DBG] pgmap v4688: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:36.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:36 smithi045 conmon[33655]: cluster 2021-07-31T03:16:35.074349+0000 mgr.x (mgr.24542) 3833 : 2021-07-31T03:16:36.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:36 smithi045 conmon[33655]: cluster [DBG] pgmap v4688: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:38.167 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:16:38.447 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:16:38.447 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 91m ago 92m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:16:38.447 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 89m ago 92m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:16:38.448 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 89m ago 95m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:16:38.448 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 91m ago 97m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:16:38.448 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 91m ago 97m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:16:38.448 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 89m ago 95m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:16:38.449 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 91m ago 95m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:16:38.449 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 91m ago 92m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:16:38.449 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 89m ago 92m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:16:38.449 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 91m ago 95m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:16:38.449 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 91m ago 94m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:16:38.450 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 91m ago 94m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:16:38.450 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 91m ago 94m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:16:38.450 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 89m ago 93m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:16:38.451 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 89m ago 93m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:16:38.451 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 89m ago 93m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:16:38.451 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 89m ago 92m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:16:38.451 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 89m ago 92m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:16:38.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:38 smithi154 conmon[34553]: cluster 2021-07-31T03:16:37.074786+0000 mgr.x (mgr.24542) 3834 : cluster [DBG] 2021-07-31T03:16:38.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:38 smithi154 conmon[34553]: pgmap v4689: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:38.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:38 smithi045 conmon[33655]: cluster 2021-07-31T03:16:37.074786+0000 mgr.x (mgr.24542) 3834 2021-07-31T03:16:38.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:38 smithi045 conmon[33655]: : cluster [DBG] pgmap v4689: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:38.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:38 smithi045 conmon[39718]: cluster 2021-07-31T 2021-07-31T03:16:38.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:38 smithi045 conmon[39718]: 03:16:37.074786+0000 mgr.x (mgr.24542) 3834 : cluster [DBG] pgmap v4689: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:38.783 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:16:38.783 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:16:38.783 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:16:38.783 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:16:38.784 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:16:38.784 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:16:38.784 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:16:38.784 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:16:38.785 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:16:38.785 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:16:38.785 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:16:38.785 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:16:38.786 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:16:38.786 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:16:38.786 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:16:38.786 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:16:38.787 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:16:39.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:39 smithi154 conmon[34553]: audit 2021-07-31T03:16:38.156528+0000 mgr.x (mgr.24542) 3835 : audit [DBG] from='client.29104 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:16:39.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:39 smithi154 conmon[34553]: audit 2021-07-31T03:16:38.439343+0000 mgr.x (mgr.24542) 3836 : audit [DBG] from='client.19170 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:16:39.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:39 smithi154 conmon[34553]: audit 2021-07-31T03:16:38.783591+0000 mon.a (mon.0) 3278 : audit [DBG] from='client.? 172.21.15.45:0/3555366240' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:16:39.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:39 smithi045 conmon[39718]: audit 2021-07-31T03:16:38.156528 2021-07-31T03:16:39.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:39 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 3835 : audit [DBG] from='client.29104 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:16:39.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:39 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T03:16:39.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:39 smithi045 conmon[39718]: 31T03:16:38. 2021-07-31T03:16:39.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:39 smithi045 conmon[39718]: 439343+0000 mgr.x (mgr.24542) 2021-07-31T03:16:39.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:39 smithi045 conmon[39718]: 3836 : audit [DBG] from='client.19170 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:16:39.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:39 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T03:16:39.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:39 smithi045 conmon[39718]: T03:16:38.783591+0000 mon.a (mon 2021-07-31T03:16:39.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:39 smithi045 conmon[39718]: .0) 3278 : audit [DBG] from='client.? 172.21.15.45:0/3555366240' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:16:39.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:39 smithi045 conmon[33655]: audit 2021-07-31T03:16:38.156528+0000 mgr.x ( 2021-07-31T03:16:39.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:39 smithi045 conmon[33655]: mgr.24542) 3835 : audit [DBG] from='client.29104 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:16:39.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:39 smithi045 conmon[33655]: audit 2021-07-31T03:16:38.439343+0000 mgr.x (mgr.24542) 3836 : audit 2021-07-31T03:16:39.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:39 smithi045 conmon[33655]: [DBG] from='client.19170 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:16:39.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:39 smithi045 conmon[33655]: audit 2021-07-31T 2021-07-31T03:16:39.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:39 smithi045 conmon[33655]: 03:16:38.783591+0000 mon.a (mon.0) 3278 2021-07-31T03:16:39.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:39 smithi045 conmon[33655]: : audit [DBG] from='client.? 172.21.15.45:0/3555366240' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:16:40.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:40 smithi154 conmon[34553]: cluster 2021-07-31T03:16:39.075228+0000 mgr.x (mgr.24542) 3837 : cluster [DBG] pgmap v4690: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:40.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:40 smithi045 conmon[39718]: cluster 2021-07-31T03:16:39.075228+0000 mgr.x (mgr.24542) 3837 : cluster [DBG] pgmap v4690: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:40.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:40 smithi045 conmon[33655]: cluster 2021-07-31T03:16:39.075228+0000 mgr.x (mgr 2021-07-31T03:16:40.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:40 smithi045 conmon[33655]: .24542) 3837 : cluster [DBG] pgmap v4690: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:42.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:42 smithi154 conmon[34553]: cluster 2021-07-31T03:16:41.075581+0000 mgr.x (mgr.24542) 3838 : cluster [DBG] pgmap v4691: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:42.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:42 smithi045 conmon[39718]: cluster 2021-07-31T03:16:41.075581+0000 mgr.x (mgr.24542) 3838 : cluster [DBG] pgmap v4691: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:42.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:42 smithi045 conmon[33655]: cluster 2021-07-31T03:16:41.075581+0000 2021-07-31T03:16:42.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:42 smithi045 conmon[33655]: mgr.x (mgr.24542) 3838 : cluster [DBG] pgmap v4691: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:44.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:44 smithi154 conmon[34553]: cluster 2021-07-31T03:16:43.076041+0000 mgr.x (mgr.24542) 3839 : cluster [DBG] 2021-07-31T03:16:44.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:44 smithi154 conmon[34553]: pgmap v4692: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:44.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:44 smithi045 conmon[33655]: cluster 2021-07-31T03:16:43.076041+0000 2021-07-31T03:16:44.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:44 smithi045 conmon[33655]: mgr.x (mgr.24542) 3839 : cluster [DBG] pgmap v4692: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:44.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:44 smithi045 conmon[39718]: cluster 2021-07-31T03:16:43.076041+0000 mgr.x (mgr.24542 2021-07-31T03:16:44.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:44 smithi045 conmon[39718]: ) 3839 : cluster [DBG] pgmap v4692: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:46.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:46 smithi154 conmon[34553]: cluster 2021-07-31T03:16:45.076571+0000 mgr.x (mgr.24542) 3840 : cluster [DBG] pgmap v4693: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:46.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:46 smithi154 conmon[34553]: 2021-07-31T03:16:46.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:46 smithi045 conmon[33655]: cluster 2021-07-31T03:16:45.076571+0000 mgr.x (mgr.24542) 3840 : 2021-07-31T03:16:46.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:46 smithi045 conmon[33655]: cluster [DBG] pgmap v4693: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:46.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:46 smithi045 conmon[39718]: cluster 2021-07-31T03:16:45.076571+0000 mgr.x ( 2021-07-31T03:16:46.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:46 smithi045 conmon[39718]: mgr.24542) 3840 : cluster [DBG] pgmap v4693: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:48.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:48 smithi154 conmon[34553]: cluster 2021-07-31T03:16:47.077103+0000 mgr.x (mgr.24542) 3841 : cluster 2021-07-31T03:16:48.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:48 smithi154 conmon[34553]: [DBG] pgmap v4694: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:48.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:48 smithi045 conmon[33655]: cluster 2021-07-31T03:16:47.077103 2021-07-31T03:16:48.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:48 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 3841 : cluster [DBG] pgmap v4694: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:48.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:48 smithi045 conmon[39718]: cluster 2021-07-31T03:16:47.077103+0000 mgr.x 2021-07-31T03:16:48.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:48 smithi045 conmon[39718]: (mgr.24542) 3841 : cluster [DBG] pgmap v4694: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:50.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:50 smithi154 conmon[34553]: cluster 2021-07-31T03:16:49.077682+0000 mgr.x (mgr.24542) 3842 : cluster 2021-07-31T03:16:50.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:50 smithi154 conmon[34553]: [DBG] pgmap v4695: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:50.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:50 smithi045 conmon[33655]: cluster 2021-07-31T03:16:49.077682+0000 mgr.x (mgr.24542) 3842 2021-07-31T03:16:50.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:50 smithi045 conmon[33655]: : cluster [DBG] pgmap v4695: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:50.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:50 smithi045 conmon[39718]: cluster 2021-07-31T03:16:49. 2021-07-31T03:16:50.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:50 smithi045 conmon[39718]: 077682+0000 mgr.x (mgr.24542) 3842 : cluster [DBG] pgmap v4695: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:52.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:52 smithi154 conmon[34553]: cluster 2021-07-31T03:16:51.078205+0000 mgr.x (mgr.24542) 3843 : cluster [DBG] pgmap v4696: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:52.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:52 smithi045 conmon[39718]: cluster 2021-07-31T03:16:51.078205 2021-07-31T03:16:52.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:52 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 3843 : cluster [DBG] pgmap v4696: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:52.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:52 smithi045 conmon[33655]: cluster 2021-07-31T03:16:51.078205+0000 mgr.x (mgr.24542) 3843 : cluster 2021-07-31T03:16:52.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:52 smithi045 conmon[33655]: [DBG] pgmap v4696: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:54.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:54 smithi154 conmon[34553]: cluster 2021-07-31T03:16:53.078679+0000 mgr.x (mgr.24542) 3844 : cluster [DBG] pgmap v4697: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:54.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:54 smithi154 conmon[34553]: 2021-07-31T03:16:54.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:54 smithi045 conmon[33655]: cluster 2021-07-31T03:16:53. 2021-07-31T03:16:54.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:54 smithi045 conmon[33655]: 078679+0000 mgr.x (mgr.24542) 3844 : cluster [DBG] pgmap v4697: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:54.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:54 smithi045 conmon[39718]: cluster 2021-07-31T03:16:53.078679+0000 mgr.x (mgr.24542) 3844 : cluster [DBG] 2021-07-31T03:16:54.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:54 smithi045 conmon[39718]: pgmap v4697: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:56.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:56 smithi154 conmon[34553]: cluster 2021-07-31T03:16:55.079276+0000 mgr.x (mgr.24542) 3845 : cluster [DBG] 2021-07-31T03:16:56.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:56 smithi154 conmon[34553]: pgmap v4698: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:56.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:56 smithi045 conmon[39718]: cluster 2021-07-31T03:16:55.079276+0000 mgr.x 2021-07-31T03:16:56.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:56 smithi045 conmon[39718]: (mgr.24542) 3845 : cluster [DBG] pgmap v4698: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:56.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:56 smithi045 conmon[33655]: cluster 2021-07-31T03:16:55.079276+0000 2021-07-31T03:16:56.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:56 smithi045 conmon[33655]: mgr.x (mgr.24542) 3845 : cluster [DBG] pgmap v4698: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:58.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:58 smithi154 conmon[34553]: cluster 2021-07-31T03:16:57.079711+0000 mgr.x (mgr.24542) 3846 : cluster [DBG] 2021-07-31T03:16:58.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:16:58 smithi154 conmon[34553]: pgmap v4699: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:58.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:58 smithi045 conmon[33655]: cluster 2021-07-31T03:16:57.079711+0000 mgr.x (mgr.24542) 3846 : cluster [DBG] pgmap v4699: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:16:58.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:16:58 smithi045 conmon[33655]: 2021-07-31T03:16:58.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:16:58 smithi045 conmon[39718]: cluster 2021-07-31T03:16:57.079711+0000 mgr.x (mgr.24542) 3846 : cluster [DBG] pgmap v4699: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:00.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:00 smithi154 conmon[34553]: cluster 2021-07-31T03:16:59.080202+0000 mgr.x (mgr.24542) 3847 : cluster [DBG] pgmap v4700: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:00.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:00 smithi154 conmon[34553]: 2021-07-31T03:17:00.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:00 smithi045 conmon[33655]: cluster 2021-07-31T03:16:59.080202+0000 mgr.x (mgr 2021-07-31T03:17:00.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:00 smithi045 conmon[33655]: .24542) 3847 : cluster [DBG] pgmap v4700: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:00.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:00 smithi045 conmon[39718]: cluster 2021-07-31T03:16:59.080202+0000 mgr.x (mgr. 2021-07-31T03:17:00.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:00 smithi045 conmon[39718]: 24542) 3847 : cluster [DBG] pgmap v4700: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:02.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:02 smithi154 conmon[34553]: cluster 2021-07-31T03:17:01.080550+0000 mgr.x (mgr.24542) 3848 : cluster [DBG] 2021-07-31T03:17:02.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:02 smithi154 conmon[34553]: pgmap v4701: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:02.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:02 smithi045 conmon[39718]: cluster 2021-07-31T03:17:01.080550+0000 mgr.x (mgr. 2021-07-31T03:17:02.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:02 smithi045 conmon[39718]: 24542) 3848 : cluster [DBG] pgmap v4701: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:02.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:02 smithi045 conmon[33655]: cluster 2021-07-31T03:17:01.080550+0000 mgr.x (mgr.24542) 3848 : cluster [DBG] pgmap v4701: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:04.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:04 smithi154 conmon[34553]: cluster 2021-07-31T03:17:03.080990+0000 mgr.x (mgr.24542) 3849 : cluster 2021-07-31T03:17:04.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:04 smithi154 conmon[34553]: [DBG] pgmap v4702: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:04.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:04 smithi045 conmon[33655]: cluster 2021-07-31T03:17:03.080990+0000 mgr.x (mgr.24542) 3849 : cluster [DBG] pgmap v4702: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:04.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:04 smithi045 conmon[39718]: cluster 2021-07-31T03:17:03.080990+0000 mgr.x ( 2021-07-31T03:17:04.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:04 smithi045 conmon[39718]: mgr.24542) 3849 : cluster [DBG] pgmap v4702: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:06.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:06 smithi154 conmon[34553]: cluster 2021-07-31T03:17:05.081471+0000 mgr.x (mgr.24542) 3850 : cluster [DBG] pgmap v4703: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:06.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:06 smithi045 conmon[39718]: cluster 2021-07-31T03:17:05.081471+0000 mgr.x ( 2021-07-31T03:17:06.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:06 smithi045 conmon[39718]: mgr.24542) 3850 : cluster [DBG] pgmap v4703: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:06.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:06 smithi045 conmon[33655]: cluster 2021-07-31T03:17:05.081471+0000 mgr.x (mgr.24542) 3850 : cluster [DBG] pgmap v4703: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:06.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:06 smithi045 conmon[33655]: 2021-07-31T03:17:08.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:08 smithi154 conmon[34553]: cluster 2021-07-31T03:17:07.081870+0000 mgr.x (mgr.24542) 3851 : cluster [DBG] pgmap v4704: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:08.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:08 smithi045 conmon[39718]: cluster 2021-07-31T03:17:07.081870+0000 mgr.x 2021-07-31T03:17:08.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:08 smithi045 conmon[39718]: (mgr.24542) 3851 : cluster [DBG] pgmap v4704: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:08.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:08 smithi045 conmon[33655]: cluster 2021-07-31T03:17:07.081870+0000 mgr.x (mgr.24542) 3851 : cluster [DBG] pgmap v4704: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:09.096 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:17:09.374 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:17:09.374 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 92m ago 92m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:17:09.375 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 89m ago 92m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:17:09.375 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 89m ago 96m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:17:09.375 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 92m ago 98m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:17:09.375 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 92m ago 98m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:17:09.376 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 89m ago 96m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:17:09.376 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 92m ago 96m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:17:09.376 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 92m ago 93m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:17:09.376 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 89m ago 92m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:17:09.376 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 92m ago 95m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:17:09.377 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 92m ago 95m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:17:09.377 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 92m ago 95m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:17:09.377 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 92m ago 94m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:17:09.377 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 89m ago 94m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:17:09.377 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 89m ago 94m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:17:09.378 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 89m ago 93m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:17:09.378 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 89m ago 93m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:17:09.378 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 89m ago 93m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:17:09.710 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:17:09.710 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:17:09.710 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:17:09.711 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:17:09.711 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:17:09.711 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:17:09.711 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:17:09.711 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:17:09.712 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:17:09.712 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:17:09.712 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:17:09.712 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:17:09.712 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:17:09.713 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:17:09.713 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:17:09.713 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:17:09.713 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:17:10.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:10 smithi154 conmon[34553]: cluster 2021-07-31T03:17:09.082559+0000 mgr.x (mgr.24542) 3852 : cluster [DBG] pgmap v4705: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:10.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:10 smithi154 conmon[34553]: audit 2021-07-31T03:17:09.084390+0000 mgr.x (mgr.24542) 3853 : audit [DBG] from='client.19182 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:17:10.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:10 smithi154 conmon[34553]: audit 2021-07-31T03:17:09.366982+0000 mgr.x (mgr.24542) 3854 : audit [DBG] from='client.19188 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:17:10.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:10 smithi154 conmon[34553]: audit 2021-07-31T03:17:10.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:10 smithi154 conmon[34553]: 2021-07-31T03:17:09.710739+0000 mon.a (mon.0) 3279 : audit [DBG] from='client.? 172.21.15.45:0/2558175264' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:17:10.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:10 smithi045 conmon[39718]: cluster 2021-07-31T03:17:09.082559+0000 mgr.x (mgr.24542) 3852 : cluster 2021-07-31T03:17:10.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:10 smithi045 conmon[39718]: [DBG] pgmap v4705: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:10.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:10 smithi045 conmon[39718]: audit 2021-07-31T03:17:09.084390+0000 mgr.x (mgr.24542) 3853 : audit [DBG] from='client.19182 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:17:10.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:10 smithi045 conmon[39718]: audit 2021-07-31T03:17:09.366982+0000 mgr.x (mgr.24542 2021-07-31T03:17:10.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:10 smithi045 conmon[39718]: ) 3854 : audit [DBG] from='client.19188 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:17:10.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:10 smithi045 conmon[39718]: audit 2021 2021-07-31T03:17:10.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:10 smithi045 conmon[39718]: -07-31T03:17:09.710739+0000 mon.a (mon.0) 3279 : audit [DBG] from='client.? 172.21.15.45:0/2558175264' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:17:10.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:10 smithi045 conmon[33655]: cluster 2021-07-31T03:17:09.082559+0000 mgr.x (mgr.24542 2021-07-31T03:17:10.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:10 smithi045 conmon[33655]: ) 3852 : cluster [DBG] pgmap v4705: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:10.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:10 smithi045 conmon[33655]: audit 2021-07-31T03:17:09.084390+0000 mgr.x (mgr.24542) 3853 : audit [DBG] from='client.19182 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:17:10.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:10 smithi045 conmon[33655]: audit 2021-07-31T03:17:09.366982+0000 mgr.x (mgr.24542) 3854 : audit [DBG] from='client.19188 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:17:10.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:10 smithi045 conmon[33655]: audit 2021-07-31T03:17:09.710739+0000 mon.a (mon. 2021-07-31T03:17:10.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:10 smithi045 conmon[33655]: 0) 3279 : audit [DBG] from='client.? 172.21.15.45:0/2558175264' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:17:12.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:12 smithi154 conmon[34553]: cluster 2021-07-31T03:17:11.082915+0000 mgr.x (mgr.24542) 3855 : cluster [DBG] pgmap v4706: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:12.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:12 smithi045 conmon[39718]: cluster 2021-07-31T03:17:11.082915+0000 mgr.x (mgr. 2021-07-31T03:17:12.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:12 smithi045 conmon[39718]: 24542) 3855 : cluster [DBG] pgmap v4706: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:12.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:12 smithi045 conmon[33655]: cluster 2021-07-31T03:17:11.082915+0000 mgr.x (mgr.24542) 3855 : cluster [DBG] pgmap v4706: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:14.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:14 smithi154 conmon[34553]: cluster 2021-07-31T03:17:13.083423+0000 mgr.x (mgr.24542) 3856 : cluster [DBG] pgmap v4707: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:14.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:14 smithi154 conmon[34553]: 2021-07-31T03:17:14.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:14 smithi045 conmon[39718]: cluster 2021-07-31T03:17:13.083423 2021-07-31T03:17:14.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:14 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 3856 : cluster [DBG] pgmap v4707: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:14.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:14 smithi045 conmon[33655]: cluster 2021-07-31T03:17:13.083423+0000 mgr.x (mgr.24542) 2021-07-31T03:17:14.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:14 smithi045 conmon[33655]: 3856 : cluster [DBG] pgmap v4707: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:16.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:16 smithi154 conmon[34553]: cluster 2021-07-31T03:17:15.084017+0000 mgr.x (mgr.24542) 3857 : cluster [DBG] pgmap v4708: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:16.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:16 smithi045 conmon[39718]: cluster 2021-07-31T03:17:15 2021-07-31T03:17:16.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:16 smithi045 conmon[39718]: .084017+0000 mgr.x (mgr.24542) 3857 : cluster [DBG] pgmap v4708: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:16.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:16 smithi045 conmon[33655]: cluster 2021-07-31T03:17:15.084017+0000 mgr.x (mgr.24542) 2021-07-31T03:17:16.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:16 smithi045 conmon[33655]: 3857 : cluster [DBG] pgmap v4708: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:18.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:18 smithi154 conmon[34553]: cluster 2021-07-31T03:17:17.084409+0000 mgr.x (mgr.24542) 3858 : cluster [DBG] pgmap v4709: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:18.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:18 smithi045 conmon[33655]: cluster 2021-07-31T03:17:17.084409+0000 mgr.x (mgr.24542) 3858 : cluster [DBG] pgmap v4709: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:18.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:18 smithi045 conmon[33655]: 2021-07-31T03:17:18.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:18 smithi045 conmon[39718]: cluster 2021-07-31T03:17:17.084409+0000 mgr.x ( 2021-07-31T03:17:18.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:18 smithi045 conmon[39718]: mgr.24542) 3858 : cluster [DBG] pgmap v4709: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:20.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:20 smithi154 conmon[34553]: cluster 2021-07-31T03:17:19.084963+0000 mgr.x (mgr.24542) 3859 : cluster [DBG] pgmap v4710: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:20.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:20 smithi045 conmon[33655]: cluster 2021-07-31T03:17:19.084963 2021-07-31T03:17:20.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:20 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 3859 : cluster [DBG] pgmap v4710: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:20.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:20 smithi045 conmon[39718]: cluster 2021-07-31T03:17:19.084963+0000 mgr.x (mgr.24542) 3859 : cluster [DBG] pgmap v4710: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:22.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:22 smithi154 conmon[34553]: cluster 2021-07-31T03:17:21.085451+0000 mgr.x (mgr.24542) 3860 : cluster [DBG] 2021-07-31T03:17:22.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:22 smithi154 conmon[34553]: pgmap v4711: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:22.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:22 smithi045 conmon[33655]: cluster 2021-07-31T03:17:21.085451+0000 2021-07-31T03:17:22.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:22 smithi045 conmon[33655]: mgr.x (mgr.24542) 3860 : cluster [DBG] pgmap v4711: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:22.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:22 smithi045 conmon[39718]: cluster 2021-07-31T03:17:21.085451+0000 mgr.x (mgr. 2021-07-31T03:17:22.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:22 smithi045 conmon[39718]: 24542) 3860 : cluster [DBG] pgmap v4711: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:24.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:24 smithi154 conmon[34553]: cluster 2021-07-31T03:17:23.085802+0000 mgr.x (mgr.24542) 3861 : cluster [DBG] pgmap v4712: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:24.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:24 smithi154 conmon[34553]: 2021-07-31T03:17:24.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:24 smithi045 conmon[33655]: cluster 2021-07-31 2021-07-31T03:17:24.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:24 smithi045 conmon[33655]: T03:17:23.085802+0000 mgr.x (mgr.24542) 3861 : cluster [DBG] pgmap v4712: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:24.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:24 smithi045 conmon[39718]: cluster 2021-07-31T03:17:23. 2021-07-31T03:17:24.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:24 smithi045 conmon[39718]: 085802+0000 mgr.x (mgr.24542) 3861 : cluster [DBG] pgmap v4712: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:26.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:26 smithi154 conmon[34553]: cluster 2021-07-31T03:17:25.086212+0000 mgr.x (mgr.24542) 3862 : cluster [DBG] 2021-07-31T03:17:26.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:26 smithi154 conmon[34553]: pgmap v4713: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:26.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:26 smithi045 conmon[33655]: cluster 2021-07-31T03:17:25.086212+0000 mgr.x (mgr.24542) 3862 : cluster [DBG] 2021-07-31T03:17:26.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:26 smithi045 conmon[33655]: pgmap v4713: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:26.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:26 smithi045 conmon[39718]: cluster 2021-07-31T03:17:25.086212+0000 2021-07-31T03:17:26.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:26 smithi045 conmon[39718]: mgr.x (mgr.24542) 3862 : cluster [DBG] pgmap v4713: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:28.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:28 smithi154 conmon[34553]: cluster 2021-07-31T03:17:27.086728+0000 mgr.x (mgr.24542) 3863 : cluster [DBG] pgmap v4714: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:28.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:28 smithi045 conmon[33655]: cluster 2021-07-31T03:17:27.086728+0000 mgr.x (mgr.24542) 3863 2021-07-31T03:17:28.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:28 smithi045 conmon[33655]: : cluster [DBG] pgmap v4714: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:28.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:28 smithi045 conmon[39718]: cluster 2021-07-31T03:17:27.086728+0000 mgr.x 2021-07-31T03:17:28.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:28 smithi045 conmon[39718]: (mgr.24542) 3863 : cluster [DBG] pgmap v4714: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:29.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:29 smithi154 conmon[34553]: audit 2021-07-31T03:17:28.293456+0000 mon.b (mon.2) 525 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:17:29.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:29 smithi154 conmon[34553]: 2021-07-31T03:17:29.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:29 smithi154 conmon[34553]: audit 2021-07-31T03:17:28.293727+0000 mon.b (mon.2) 526 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:17:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:29 smithi154 conmon[34553]: audit 2021-07-31 2021-07-31T03:17:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:29 smithi154 conmon[34553]: T03:17:28.294088+0000 mon.a (mon.0) 3280 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:17:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:29 smithi154 conmon[34553]: audit 2021 2021-07-31T03:17:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:29 smithi154 conmon[34553]: -07-31T03:17:28 2021-07-31T03:17:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:29 smithi154 conmon[34553]: .294513+0000 mon.a (mon.0) 3281 : audit 2021-07-31T03:17:29.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:29 smithi154 conmon[34553]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:17:29.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:29 smithi045 conmon[39718]: audit 2021-07-31T03:17:28 2021-07-31T03:17:29.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:29 smithi045 conmon[39718]: .293456+0000 mon.b (mon.2) 525 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:17:29.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:29 smithi045 conmon[39718]: audit 2021-07-31T03:17:29.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:29 smithi045 conmon[39718]: 2021-07-31T03 2021-07-31T03:17:29.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:29 smithi045 conmon[39718]: :17:28.293727+0000 2021-07-31T03:17:29.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:29 smithi045 conmon[39718]: mon.b (mon.2) 526 2021-07-31T03:17:29.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:29 smithi045 conmon[39718]: : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:17:29.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:29 smithi045 conmon[39718]: audit 2021 2021-07-31T03:17:29.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:29 smithi045 conmon[39718]: -07-31T03:17:28. 2021-07-31T03:17:29.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:29 smithi045 conmon[39718]: 294088+0000 mon.a (mon.0 2021-07-31T03:17:29.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:29 smithi045 conmon[39718]: ) 3280 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:17:29.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:29 smithi045 conmon[39718]: 2021-07-31T03:17:29.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:29 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T03:17:29.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:29 smithi045 conmon[39718]: 31T03:17:28. 2021-07-31T03:17:29.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:29 smithi045 conmon[39718]: 294513+0000 mon.a (mon.0 2021-07-31T03:17:29.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:29 smithi045 conmon[39718]: ) 3281 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:17:29.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:29 smithi045 conmon[39718]: 2021-07-31T03:17:29.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:29 smithi045 conmon[33655]: audit 2021-07-31T03:17:28.293456+0000 mon.b (mon.2) 525 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:17:29.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:29 smithi045 conmon[33655]: audit 2021-07-31T03:17:28.293727+0000 mon.b (mon.2) 526 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:17:29.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:29 smithi045 conmon[33655]: audit 2021-07-31T03:17:28.294088 2021-07-31T03:17:29.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:29 smithi045 conmon[33655]: +0000 mon.a (mon.0) 3280 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:17:29.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:29 smithi045 conmon[33655]: audit 2021-07-31T03:17:29.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:29 smithi045 conmon[33655]: 2021- 2021-07-31T03:17:29.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:29 smithi045 conmon[33655]: 07-31T03:17:28 2021-07-31T03:17:29.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:29 smithi045 conmon[33655]: .294513+0000 mon.a (mon 2021-07-31T03:17:29.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:29 smithi045 conmon[33655]: .0) 3281 : audit [INF] 2021-07-31T03:17:29.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:29 smithi045 conmon[33655]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:17:30.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:30 smithi154 conmon[34553]: cluster 2021-07-31T03:17:29.087193+0000 mgr.x (mgr.24542) 3864 : cluster [DBG] pgmap v4715: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:30.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:30 smithi045 conmon[39718]: cluster 2021-07-31T03:17 2021-07-31T03:17:30.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:30 smithi045 conmon[39718]: :29.087193+0000 mgr.x (mgr.24542) 3864 : cluster [DBG] pgmap v4715: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:30.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:30 smithi045 conmon[33655]: cluster 2021-07-31T03:17:29.087193+0000 mgr.x (mgr.24542) 2021-07-31T03:17:30.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:30 smithi045 conmon[33655]: 3864 : cluster [DBG] pgmap v4715: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:32.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[34553]: cluster 2021-07-31T03:17:31.087516+0000 mgr.x (mgr.24542) 3865 : cluster [DBG] pgmap v4716: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:32.503 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: debug 2021-07-31T03:17:32.360+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:17:32.503 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:17:32.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:17:32.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: yield 2021-07-31T03:17:32.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:17:32.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:17:32.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:17:32.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:17:32.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:17:32.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:17:32.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:17:32.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:17:32.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:17:32.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: task.result() 2021-07-31T03:17:32.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:17:32.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: debug 2021-07-31T03:17:32.371+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:17:32.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:17:32.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:17:32.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: yield 2021-07-31T03:17:32.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:17:32.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:17:32.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:17:32.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:17:32.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:17:32.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:17:32.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:17:32.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:17:32.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:17:32.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: task.result() 2021-07-31T03:17:32.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:17:32.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: debug 2021-07-31T03:17:32.397+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:17:32.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:17:32.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:17:32.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: yield 2021-07-31T03:17:32.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:17:32.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:17:32.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:17:32.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:17:32.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:17:32.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:17:32.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:17:32.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:17:32.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:17:32.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: task.result() 2021-07-31T03:17:32.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:17:32.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: debug 2021-07-31T03:17:32.398+0000 7fcb94162700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:17:32.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:17:32.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:17:32.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: yield 2021-07-31T03:17:32.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:17:32.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:17:32.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:17:32.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:17:32.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:17:32.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:17:32.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:17:32.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:17:32.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:17:32.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: task.result() 2021-07-31T03:17:32.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:17:32.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: 2021-07-31T03:17:32.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:17:32.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: 2021-07-31T03:17:32.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:17:32.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:17:32.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:17:32.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:17:32.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:17:32.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:17:32.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:17:32.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:17:32.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:17:32.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:17:32.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:17:32.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:17:32.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:17:32.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:17:32.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:17:32.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:17:32.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:17:32.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:17:32.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: raise 2021-07-31T03:17:32.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:17:32.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:17:32.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:17:32.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:17:32.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:17:32.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: debug 2021-07-31T03:17:32.406+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:17:32.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:17:32.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:17:32.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: yield 2021-07-31T03:17:32.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:17:32.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:17:32.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:17:32.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:17:32.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:17:32.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:17:32.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:17:32.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:17:32.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:17:32.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: task.result() 2021-07-31T03:17:32.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:17:32.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: debug 2021-07-31T03:17:32.406+0000 7fcb93961700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:17:32.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:17:32.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:17:32.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: yield 2021-07-31T03:17:32.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:17:32.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:17:32.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:17:32.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:17:32.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:17:32.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:17:32.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:17:32.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:17:32.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:17:32.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: task.result() 2021-07-31T03:17:32.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:17:32.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: 2021-07-31T03:17:32.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:17:32.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: 2021-07-31T03:17:32.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:17:32.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:17:32.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:17:32.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:17:32.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:17:32.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:17:32.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:17:32.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:17:32.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:17:32.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:17:32.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:17:32.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:17:32.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:17:32.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:17:32.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:17:32.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:17:32.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:17:32.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:17:32.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: raise 2021-07-31T03:17:32.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:17:32.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:17:32.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:17:32.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:17:32.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:17:32 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:17:32.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:32 smithi045 conmon[33655]: cluster 2021-07 2021-07-31T03:17:32.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:32 smithi045 conmon[33655]: -31T03:17:31.087516+0000 mgr.x (mgr.24542) 3865 : cluster [DBG] pgmap v4716: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:32.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:32 smithi045 conmon[39718]: cluster 2021-07-31T03:17:31.087516+0000 mgr.x (mgr.24542) 3865 : cluster [DBG] pgmap v4716: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:33.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: audit 2021-07-31T03:17:32.314645+0000 mon.b (mon.2) 527 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:17:33.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: audit 2021-07-31T03:17:32.315990+0000 2021-07-31T03:17:33.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: mon.b (mon.2) 528 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:17:33.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: audit 2021-07-31T03:17:32.316999+0000 mon.b (mon.2) 529 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:17:33.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:17:32.317819+0000 2021-07-31T03:17:33.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: mgr.x (mgr.24542) 3866 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:17:33.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:17:32.361084+0000 mgr.x 2021-07-31T03:17:33.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: (mgr.24542) 3867 : cephadm [ERR] 2021-07-31T03:17:33.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:17:33.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:17:33.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: yield 2021-07-31T03:17:33.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:17:33.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:17:33.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:17:33.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:17:33.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:17:33.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:17:33.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:17:33.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:17:33.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:17:33.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: task.result() 2021-07-31T03:17:33.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:17:33.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:17:32.361700+0000 mgr.x (mgr.24542) 3868 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:17:33.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:17:32.372874+0000 mgr.x (mgr.24542) 3869 : 2021-07-31T03:17:33.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: cephadm [ERR] 2021-07-31T03:17:33.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:17:33.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:17:33.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: yield 2021-07-31T03:17:33.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:17:33.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:17:33.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:17:33.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:17:33.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:17:33.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:17:33.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:17:33.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:17:33.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:17:33.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: task.result() 2021-07-31T03:17:33.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:17:33.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:17:32.373710+0000 mgr.x (mgr.24542) 3870 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:17:33.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:17:32.398364+0000 mgr.x (mgr.24542) 3871 : cephadm [ERR] 2021-07-31T03:17:33.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:17:33.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:17:33.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: yield 2021-07-31T03:17:33.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:17:33.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:17:33.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:17:33.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:17:33.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:17:33.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:17:33.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:17:33.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:17:33.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:17:33.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: task.result() 2021-07-31T03:17:33.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:17:33.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: 2021-07-31T03:17:33.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:17:32.398987+0000 mgr.x (mgr.24542) 3872 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:17:33.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:17:33.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:17:33.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: yield 2021-07-31T03:17:33.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:17:33.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:17:33.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:17:33.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:17:33.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:17:33.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:17:33.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:17:33.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:17:33.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:17:33.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: task.result() 2021-07-31T03:17:33.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:17:33.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: 2021-07-31T03:17:33.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:17:33.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: 2021-07-31T03:17:33.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:17:33.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:17:33.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:17:33.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:17:33.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:17:33.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:17:33.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:17:33.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:17:33.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:17:33.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:17:33.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:17:33.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:17:33.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:17:33.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:17:33.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:17:33.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:17:33.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:17:33.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:17:33.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: raise 2021-07-31T03:17:33.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:17:33.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:17:33.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:17:33.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:17:33.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:17:33.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:17:32.406999+0000 2021-07-31T03:17:33.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: mgr.x (mgr.24542) 3873 : cephadm [ERR] 2021-07-31T03:17:33.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:17:33.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:17:33.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: yield 2021-07-31T03:17:33.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:17:33.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:17:33.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:17:33.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:17:33.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:17:33.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:17:33.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:17:33.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:17:33.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:17:33.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: task.result() 2021-07-31T03:17:33.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:17:33.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:17:32.407388+0000 mgr.x (mgr.24542) 3874 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:17:33.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:17:33.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:17:33.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: yield 2021-07-31T03:17:33.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:17:33.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:17:33.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:17:33.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:17:33.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:17:33.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:17:33.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:17:33.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:17:33.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:17:33.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: task.result() 2021-07-31T03:17:33.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:17:33.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: 2021-07-31T03:17:33.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:17:33.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: 2021-07-31T03:17:33.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:17:33.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:17:33.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:17:33.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:17:33.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:17:33.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:17:33.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:17:33.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:17:33.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:17:33.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:17:33.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:17:33.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:17:33.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:17:33.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:17:33.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:17:33.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:17:33.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:17:33.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:17:33.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: raise 2021-07-31T03:17:33.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:17:33.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:17:33.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:17:33.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:17:33.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:33 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:17:33.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: audit 2021-07-31T03:17:32 2021-07-31T03:17:33.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: .314645+0000 mon.b (mon.2) 527 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:17:33.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T03:17:33.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: 03:17:32.315990+0000 2021-07-31T03:17:33.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: mon.b (mon.2) 528 : 2021-07-31T03:17:33.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:17:33.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T03:17:33.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: 31T03:17:32.316999+0000 2021-07-31T03:17:33.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: mon.b (mon.2) 529 : 2021-07-31T03:17:33.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:17:33.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: cephadm 2021- 2021-07-31T03:17:33.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: 07-31T03:17:32.317819 2021-07-31T03:17:33.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 2021-07-31T03:17:33.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: 3866 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:17:33.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: cephadm 2021- 2021-07-31T03:17:33.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: 07-31T03:17:32. 2021-07-31T03:17:33.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: 361084+0000 mgr.x (mgr.24542 2021-07-31T03:17:33.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: ) 3867 : cephadm [ERR] 2021-07-31T03:17:33.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:17:33.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:17:33.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: yield 2021-07-31T03:17:33.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:17:33.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:17:33.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:17:33.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:17:33.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:17:33.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:17:33.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:17:33.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:17:33.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:17:33.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: task.result() 2021-07-31T03:17:33.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:17:33.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: 2021-07-31T03:17:33.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: cephadm 2021-07-31T03:17:32.361700+0000 mgr.x (mgr.24542) 3868 2021-07-31T03:17:33.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:17:33.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: cephadm 2021- 2021-07-31T03:17:33.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: 07-31T03:17:32 2021-07-31T03:17:33.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: .372874+0000 mgr.x (mgr.24542 2021-07-31T03:17:33.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: ) 3869 : cephadm [ERR] 2021-07-31T03:17:33.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:17:33.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:17:33.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: yield 2021-07-31T03:17:33.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:17:33.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:17:33.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:17:33.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:17:33.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:17:33.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:17:33.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:17:33.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:17:33.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:17:33.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: task.result() 2021-07-31T03:17:33.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:17:33.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: cephadm 2021-07-31T03:17:32.373710+0000 mgr.x (mgr.24542) 3870 : cephadm 2021-07-31T03:17:33.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:17:33.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: cephadm 2021-07- 2021-07-31T03:17:33.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: 31T03:17:32. 2021-07-31T03:17:33.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: 398364+0000 mgr.x (mgr.24542) 2021-07-31T03:17:33.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: 3871 : cephadm [ERR] 2021-07-31T03:17:33.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:17:33.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:17:33.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: yield 2021-07-31T03:17:33.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:17:33.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:17:33.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:17:33.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:17:33.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:17:33.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:17:33.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:17:33.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:17:33.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:17:33.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: task.result() 2021-07-31T03:17:33.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:17:33.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: cephadm 2021-07-31T03:17:32.398987+0000 mgr.x (mgr.24542) 3872 : cephadm 2021-07-31T03:17:33.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:17:33.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:17:33.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:17:33.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: yield 2021-07-31T03:17:33.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:17:33.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:17:33.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:17:33.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:17:33.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:17:33.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:17:33.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:17:33.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:17:33.554 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: audit 2021-07-31T03:17:32.314645+0000 mon.b (mon.2) 527 2021-07-31T03:17:33.554 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:17:33.554 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: audit 2021-07-31T03:17:32.315990+0000 mon.b (mon.2) 528 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:17:33.554 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: audit 2021-07 2021-07-31T03:17:33.555 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: -31T03:17:32.316999+0000 mon.b (mon.2) 529 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:17:33.555 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: cephadm 2021-07-31T03:17:33.555 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: 2021-07-31T03:17: 2021-07-31T03:17:33.555 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: 32.317819+0000 mgr.x (mgr.24542) 2021-07-31T03:17:33.555 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: 3866 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:17:33.556 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: cephadm 2021- 2021-07-31T03:17:33.556 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: 07-31T03:17:32.361084+0000 mgr.x (mgr.24542) 2021-07-31T03:17:33.556 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: 3867 : cephadm [ERR] 2021-07-31T03:17:33.556 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:17:33.556 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:17:33.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: yield 2021-07-31T03:17:33.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:17:33.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:17:33.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:17:33.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:17:33.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:17:33.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:17:33.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:17:33.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:17:33.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:17:33.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: task.result() 2021-07-31T03:17:33.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:17:33.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: cephadm 2021-07-31T03:17:32.361700+0000 mgr.x (mgr.24542) 3868 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:17:33.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: cephadm 2021-07-31 2021-07-31T03:17:33.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: T03:17:32.372874+0000 mgr.x (mgr.24542) 3869 : cephadm 2021-07-31T03:17:33.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: [ERR] 2021-07-31T03:17:33.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:17:33.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:17:33.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: yield 2021-07-31T03:17:33.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:17:33.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:17:33.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:17:33.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:17:33.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:17:33.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:17:33.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: task.result() 2021-07-31T03:17:33.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:17:33.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: 2021-07-31T03:17:33.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:17:33.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: 2021-07-31T03:17:33.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:17:33.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:17:33.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:17:33.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:17:33.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:17:33.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:17:33.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:17:33.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:17:33.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:17:33.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:17:33.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:17:33.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:17:33.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:17:33.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:17:33.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:17:33.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:17:33.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:17:33.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:17:33.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: raise 2021-07-31T03:17:33.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:17:33.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:17:33.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:17:33.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:17:33.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:17:33.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: cephadm 2021-07-31T03:17:32.406999+0000 mgr.x (mgr.24542) 3873 : cephadm [ERR] 2021-07-31T03:17:33.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:17:33.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:17:33.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: yield 2021-07-31T03:17:33.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:17:33.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:17:33.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:17:33.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:17:33.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:17:33.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:17:33.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:17:33.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:17:33.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:17:33.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:17:33.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: task.result() 2021-07-31T03:17:33.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:17:33.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: cephadm 2021-07-31T03:17:32.373710+0000 mgr.x (mgr.24542) 3870 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:17:33.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: cephadm 2021-07-31T03: 2021-07-31T03:17:33.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: 17:32.398364+0000 mgr.x (mgr.24542) 3871 : cephadm [ERR] 2021-07-31T03:17:33.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:17:33.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:17:33.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: yield 2021-07-31T03:17:33.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:17:33.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:17:33.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:17:33.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:17:33.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:17:33.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:17:33.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:17:33.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:17:33.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:17:33.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: task.result() 2021-07-31T03:17:33.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:17:33.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: cephadm 2021-07-31T03:17:32.398987+0000 mgr.x (mgr.24542) 3872 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:17:33.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:17:33.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:17:33.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: yield 2021-07-31T03:17:33.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:17:33.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:17:33.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:17:33.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:17:33.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:17:33.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:17:33.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:17:33.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:17:33.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:17:33.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: task.result() 2021-07-31T03:17:33.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:17:33.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: 2021-07-31T03:17:33.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:17:33.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: 2021-07-31T03:17:33.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:17:33.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:17:33.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:17:33.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:17:33.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: task.result() 2021-07-31T03:17:33.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:17:33.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: cephadm 2021-07-31T03:17:32.407388+0000 mgr.x (mgr.24542) 3874 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:17:33.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:17:33.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:17:33.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: yield 2021-07-31T03:17:33.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:17:33.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:17:33.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:17:33.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:17:33.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:17:33.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:17:33.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:17:33.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:17:33.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:17:33.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: task.result() 2021-07-31T03:17:33.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:17:33.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: 2021-07-31T03:17:33.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:17:33.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: 2021-07-31T03:17:33.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:17:33.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:17:33.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:17:33.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:17:33.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:17:33.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:17:33.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:17:33.585 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:17:33.585 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:17:33.585 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:17:33.585 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:17:33.585 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:17:33.586 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:17:33.586 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:17:33.586 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:17:33.586 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:17:33.586 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:17:33.587 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:17:33.587 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: raise 2021-07-31T03:17:33.587 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:17:33.587 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:17:33.587 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:17:33.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:17:33.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:17:33.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:17:33.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:17:33.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:17:33.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:17:33.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:17:33.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:17:33.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:17:33.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:17:33.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:17:33.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:17:33.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:17:33.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:17:33.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:17:33.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:17:33.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:17:33.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:17:33.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:17:33.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: raise 2021-07-31T03:17:33.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:17:33.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:17:33.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:17:33.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:17:33.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:17:33.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: cephadm 2021-07-31T03:17:32.406999+0000 mgr.x (mgr.24542) 3873 : cephadm 2021-07-31T03:17:33.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: [ERR] 2021-07-31T03:17:33.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:17:33.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:17:33.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: yield 2021-07-31T03:17:33.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:17:33.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:17:33.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:17:33.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:17:33.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:17:33.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:17:33.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:17:33.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:17:33.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:17:33.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: task.result() 2021-07-31T03:17:33.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:17:33.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: cephadm 2021-07-31T03:17:32.407388+0000 mgr.x (mgr.24542) 3874 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:17:33.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:17:33.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:17:33.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: yield 2021-07-31T03:17:33.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:17:33.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:17:33.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:17:33.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:17:33.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:17:33.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:17:33.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:17:33.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:17:33.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:17:33.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: task.result() 2021-07-31T03:17:33.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:17:33.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: 2021-07-31T03:17:33.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:17:33.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: 2021-07-31T03:17:33.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:17:33.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:17:33.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:17:33.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:17:33.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:17:33.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:17:33.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:17:33.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:17:33.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:17:33.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:17:33.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:17:33.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:17:33.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:17:33.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:17:33.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:17:33.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:17:33.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:17:33.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:17:33.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: raise 2021-07-31T03:17:33.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:17:33.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:17:33.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:17:33.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:17:33.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:33 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:17:34.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:34 smithi154 conmon[34553]: cluster 2021-07-31T03:17:33.087826+0000 mgr.x (mgr.24542) 3875 : cluster [DBG] pgmap v4717: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:34.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:34 smithi045 conmon[33655]: cluster 2021-07-31T03:17:33.087826+0000 mgr.x (mgr.24542) 3875 : cluster [DBG] 2021-07-31T03:17:34.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:34 smithi045 conmon[33655]: pgmap v4717: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:34.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:34 smithi045 conmon[39718]: cluster 2021-07-31T03:17: 2021-07-31T03:17:34.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:34 smithi045 conmon[39718]: 33.087826+0000 mgr.x (mgr.24542) 3875 : cluster [DBG] pgmap v4717: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:36.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:36 smithi154 conmon[34553]: cluster 2021-07-31T03:17:35.088337+0000 mgr.x (mgr.24542) 3876 : cluster [DBG] 2021-07-31T03:17:36.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:36 smithi154 conmon[34553]: pgmap v4718: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:36.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:36 smithi045 conmon[33655]: cluster 2021-07-31T03:17:35.088337+0000 mgr.x (mgr.24542) 3876 : cluster 2021-07-31T03:17:36.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:36 smithi045 conmon[33655]: [DBG] pgmap v4718: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:36.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:36 smithi045 conmon[39718]: cluster 2021-07-31T03:17: 2021-07-31T03:17:36.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:36 smithi045 conmon[39718]: 35.088337+0000 mgr.x (mgr.24542) 3876 : cluster [DBG] pgmap v4718: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:38.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:38 smithi154 conmon[34553]: cluster 2021-07-31T03:17:37.088879+0000 mgr.x (mgr.24542) 3877 : cluster [DBG] pgmap v4719: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:38.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:38 smithi045 conmon[39718]: cluster 2021-07-31T03:17:37.088879+0000 2021-07-31T03:17:38.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:38 smithi045 conmon[39718]: mgr.x (mgr.24542) 3877 : cluster [DBG] pgmap v4719: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:38.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:38 smithi045 conmon[33655]: cluster 2021-07-31T03:17:37.088879+0000 mgr.x (mgr. 2021-07-31T03:17:38.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:38 smithi045 conmon[33655]: 24542) 3877 : cluster [DBG] pgmap v4719: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:40.024 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:17:40.302 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:17:40.303 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 92m ago 93m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:17:40.303 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 90m ago 93m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:17:40.303 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 90m ago 96m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:17:40.304 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 92m ago 98m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:17:40.304 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 92m ago 98m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:17:40.304 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 90m ago 96m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:17:40.304 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 92m ago 97m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:17:40.304 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 92m ago 93m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:17:40.305 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 90m ago 93m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:17:40.305 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 92m ago 96m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:17:40.305 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 92m ago 95m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:17:40.305 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 92m ago 95m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:17:40.306 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 92m ago 95m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:17:40.306 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 90m ago 94m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:17:40.306 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 90m ago 94m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:17:40.306 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 90m ago 94m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:17:40.306 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 90m ago 93m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:17:40.307 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 90m ago 93m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:17:40.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:40 smithi154 conmon[34553]: cluster 2021-07-31T03:17:39.089342+0000 mgr.x (mgr.24542) 3878 : cluster [DBG] pgmap v4720: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:40.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:40 smithi154 conmon[34553]: 2021-07-31T03:17:40.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:40 smithi045 conmon[33655]: cluster 2021-07-31T03:17:39.089342+0000 mgr.x (mgr. 2021-07-31T03:17:40.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:40 smithi045 conmon[33655]: 24542) 3878 : cluster [DBG] pgmap v4720: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:40.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:40 smithi045 conmon[39718]: cluster 2021-07-31T03:17:39 2021-07-31T03:17:40.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:40 smithi045 conmon[39718]: .089342+0000 mgr.x (mgr.24542) 3878 : cluster [DBG] pgmap v4720: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:40.639 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:17:40.640 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:17:40.640 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:17:40.640 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:17:40.640 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:17:40.641 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:17:40.641 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:17:40.641 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:17:40.641 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:17:40.642 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:17:40.642 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:17:40.642 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:17:40.642 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:17:40.642 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:17:40.643 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:17:40.643 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:17:40.643 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:17:41.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:41 smithi154 conmon[34553]: audit 2021-07-31T03:17:40.012571+0000 mgr.x (mgr.24542) 3879 : audit [DBG] from='client.29140 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:17:41.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:41 smithi154 conmon[34553]: audit 2021-07-31T03:17:40.295184+0000 mgr.x (mgr.24542) 3880 : audit [DBG] from='client.29146 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:17:41.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:41 smithi154 conmon[34553]: audit 2021-07-31T03:17:40.640395+0000 mon.c (mon.1) 192 : audit [DBG] from='client.? 172.21.15.45:0/1712348650' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:17:41.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:41 smithi045 conmon[39718]: audit 2021-07-31T03:17:40.012571+0000 mgr.x (mgr.24542) 3879 : audit [DBG] 2021-07-31T03:17:41.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:41 smithi045 conmon[39718]: from='client.29140 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:17:41.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:41 smithi045 conmon[39718]: audit 2021-07-31T03:17:40.295184+0000 mgr.x (mgr.24542) 3880 : audit [DBG] from='client.29146 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:17:41.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:41 smithi045 conmon[39718]: audit 2021-07-31T03:17:41.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:41 smithi045 conmon[39718]: 2021-07-31T03:17:40.640395+0000 mon.c (mon.1) 192 : audit [DBG] from='client.? 172.21.15.45:0/1712348650' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:17:41.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:41 smithi045 conmon[33655]: audit 2021-07-31T03:17:40.012571+0000 mgr.x (mgr.24542) 3879 : audit [DBG] from='client.29140 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:17:41.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:41 smithi045 conmon[33655]: 2021-07-31T03:17:41.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:41 smithi045 conmon[33655]: audit 2021-07-31T03:17:40.295184+0000 mgr.x (mgr.24542) 3880 : audit [DBG] from='client.29146 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:17:41.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:41 smithi045 conmon[33655]: audit 2021-07-31T03:17:40.640395+0000 mon.c (mon.1) 192 : audit [DBG] from='client.? 172.21.15.45:0/1712348650' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:17:42.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:42 smithi154 conmon[34553]: cluster 2021-07-31T03:17:41.089859+0000 mgr.x (mgr.24542) 3881 : cluster [DBG] pgmap v4721: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:42.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:42 smithi045 conmon[33655]: cluster 2021-07-31T03:17:41.089859+0000 mgr.x (mgr.24542) 3881 : cluster [DBG] pgmap v4721: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:42.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:42 smithi045 conmon[33655]: 2021-07-31T03:17:42.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:42 smithi045 conmon[39718]: cluster 2021-07-31T03:17:41.089859+0000 mgr.x (mgr.24542) 3881 : cluster 2021-07-31T03:17:42.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:42 smithi045 conmon[39718]: [DBG] pgmap v4721: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:44.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:44 smithi154 conmon[34553]: cluster 2021-07-31T03:17:43.090348+0000 mgr.x (mgr.24542) 3882 : cluster [DBG] pgmap v4722: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:44.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:44 smithi045 conmon[33655]: cluster 2021-07-31T03:17:43.090348+0000 mgr.x (mgr.24542) 3882 : cluster [DBG] pgmap v4722: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:44.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:44 smithi045 conmon[39718]: cluster 2021-07-31T03:17:43.090348+0000 mgr.x 2021-07-31T03:17:44.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:44 smithi045 conmon[39718]: (mgr.24542) 3882 : cluster [DBG] pgmap v4722: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:46.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:46 smithi154 conmon[34553]: cluster 2021-07-31T03:17:45.090898+0000 mgr.x (mgr.24542) 3883 : cluster [DBG] 2021-07-31T03:17:46.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:46 smithi154 conmon[34553]: pgmap v4723: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:46.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:46 smithi045 conmon[33655]: cluster 2021-07-31T03:17:45.090898+0000 mgr.x (mgr.24542) 3883 : cluster [DBG] 2021-07-31T03:17:46.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:46 smithi045 conmon[33655]: pgmap v4723: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:46.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:46 smithi045 conmon[39718]: cluster 2021-07-31T03:17: 2021-07-31T03:17:46.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:46 smithi045 conmon[39718]: 45.090898+0000 mgr.x (mgr.24542) 3883 : cluster [DBG] pgmap v4723: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:48.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:48 smithi045 conmon[39718]: cluster 2021-07-31T03:17:47.091370+0000 mgr.x (mgr 2021-07-31T03:17:48.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:48 smithi045 conmon[39718]: .24542) 3884 : cluster [DBG] pgmap v4724: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:48.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:48 smithi045 conmon[33655]: cluster 2021-07-31T03:17:47.091370+0000 mgr.x (mgr.24542) 3884 : cluster [DBG] pgmap v4724: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:48.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:48 smithi045 conmon[33655]: 2021-07-31T03:17:48.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:48 smithi154 conmon[34553]: cluster 2021-07-31T03:17:47.091370+0000 mgr.x (mgr.24542) 3884 : cluster 2021-07-31T03:17:48.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:48 smithi154 conmon[34553]: [DBG] pgmap v4724: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:50.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:50 smithi045 conmon[39718]: cluster 2021-07-31T03:17:49.091763+0000 mgr.x (mgr.24542) 3885 : cluster [DBG] pgmap v4725: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:50.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:50 smithi045 conmon[33655]: cluster 2021-07-31T03:17:49.091763+0000 mgr.x (mgr.24542) 3885 : cluster [DBG] 2021-07-31T03:17:50.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:50 smithi045 conmon[33655]: pgmap v4725: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:50.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:50 smithi154 conmon[34553]: cluster 2021-07-31T03:17:49.091763+0000 mgr.x (mgr.24542) 3885 : cluster [DBG] pgmap v4725: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:52.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:52 smithi045 conmon[39718]: cluster 2021-07-31T03:17:51 2021-07-31T03:17:52.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:52 smithi045 conmon[39718]: .092286+0000 mgr.x (mgr.24542) 3886 : cluster [DBG] pgmap v4726: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:52.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:52 smithi045 conmon[33655]: cluster 2021-07-31T03:17:51.092286+0000 mgr.x (mgr.24542) 3886 : cluster [DBG] pgmap v4726: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:52.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:52 smithi045 conmon[33655]: 2021-07-31T03:17:52.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:52 smithi154 conmon[34553]: cluster 2021-07-31T03:17:51.092286+0000 mgr.x (mgr.24542) 3886 : cluster [DBG] pgmap v4726: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:54.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:54 smithi045 conmon[39718]: cluster 2021-07-31T03:17:53.092690+0000 mgr.x ( 2021-07-31T03:17:54.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:54 smithi045 conmon[39718]: mgr.24542) 3887 : cluster [DBG] pgmap v4727: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:54.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:54 smithi045 conmon[33655]: cluster 2021-07-31T03:17:53.092690+0000 mgr.x (mgr.24542) 3887 : cluster [DBG] pgmap v4727: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:54.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:54 smithi154 conmon[34553]: cluster 2021-07-31T03:17:53.092690+0000 mgr.x (mgr.24542) 3887 : cluster [DBG] pgmap v4727: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:56.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:56 smithi045 conmon[33655]: cluster 2021-07-31T03:17:55.093303 2021-07-31T03:17:56.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:56 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 3888 : cluster [DBG] pgmap v4728: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:56.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:56 smithi045 conmon[39718]: cluster 2021-07-31T03:17:55.093303+0000 mgr.x ( 2021-07-31T03:17:56.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:56 smithi045 conmon[39718]: mgr.24542) 3888 : cluster [DBG] pgmap v4728: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:56.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:56 smithi154 conmon[34553]: cluster 2021-07-31T03:17:55.093303+0000 mgr.x (mgr.24542) 3888 : cluster [DBG] pgmap v4728: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:56.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:56 smithi154 conmon[34553]: 2021-07-31T03:17:58.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:58 smithi045 conmon[39718]: cluster 2021-07-31T03:17:57.093838 2021-07-31T03:17:58.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:17:58 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 3889 : cluster [DBG] pgmap v4729: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:58.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:17:58 smithi045 conmon[33655]: cluster 2021-07-31T03:17:57.093838+0000 mgr.x (mgr.24542) 3889 : cluster [DBG] pgmap v4729: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:17:58.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:17:58 smithi154 conmon[34553]: cluster 2021-07-31T03:17:57.093838+0000 mgr.x (mgr.24542) 3889 : cluster [DBG] pgmap v4729: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:00.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:00 smithi045 conmon[33655]: cluster 2021-07-31T03:17:59.094396+0000 mgr.x (mgr.24542) 3890 : cluster [DBG] 2021-07-31T03:18:00.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:00 smithi045 conmon[33655]: pgmap v4730: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:00.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:00 smithi045 conmon[39718]: cluster 2021-07-31T03:17:59.094396+0000 mgr.x 2021-07-31T03:18:00.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:00 smithi045 conmon[39718]: (mgr.24542) 3890 : cluster [DBG] pgmap v4730: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:00.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:00 smithi154 conmon[34553]: cluster 2021-07-31T03:17:59.094396+0000 mgr.x (mgr.24542) 3890 : cluster [DBG] 2021-07-31T03:18:00.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:00 smithi154 conmon[34553]: pgmap v4730: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:02.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:02 smithi154 conmon[34553]: cluster 2021-07-31T03:18:01.094907+0000 mgr.x (mgr.24542) 3891 : cluster [DBG] pgmap v4731: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:02.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:02 smithi045 conmon[39718]: cluster 2021-07-31T03:18:01.094907+0000 mgr.x (mgr. 2021-07-31T03:18:02.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:02 smithi045 conmon[39718]: 24542) 3891 : cluster [DBG] pgmap v4731: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:02.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:02 smithi045 conmon[33655]: cluster 2021-07-31T03:18:01.094907+0000 mgr.x (mgr.24542) 3891 : cluster [DBG] pgmap v4731: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:02.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:02 smithi045 conmon[33655]: 2021-07-31T03:18:04.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:04 smithi154 conmon[34553]: cluster 2021-07-31T03:18:03.095427+0000 mgr.x (mgr.24542) 3892 : cluster [DBG] pgmap v4732: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:04.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:04 smithi154 conmon[34553]: 2021-07-31T03:18:04.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:04 smithi045 conmon[39718]: cluster 2021-07-31T03:18:03.095427+0000 mgr.x 2021-07-31T03:18:04.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:04 smithi045 conmon[39718]: (mgr.24542) 3892 : cluster [DBG] pgmap v4732: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:04.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:04 smithi045 conmon[33655]: cluster 2021-07-31T03:18:03.095427+0000 mgr.x (mgr.24542) 3892 2021-07-31T03:18:04.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:04 smithi045 conmon[33655]: : cluster [DBG] pgmap v4732: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:06.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:06 smithi154 conmon[34553]: cluster 2021-07-31T03:18:05.095866+0000 mgr.x (mgr.24542) 3893 : cluster [DBG] pgmap v4733: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:06.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:06 smithi045 conmon[39718]: cluster 2021-07-31T03:18:05.095866+0000 mgr.x (mgr.24542) 2021-07-31T03:18:06.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:06 smithi045 conmon[39718]: 3893 : cluster [DBG] pgmap v4733: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:06.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:06 smithi045 conmon[33655]: cluster 2021-07-31T03:18:05.095866 2021-07-31T03:18:06.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:06 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 3893 : cluster [DBG] pgmap v4733: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:08.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:08 smithi154 conmon[34553]: cluster 2021-07-31T03:18:07.096350+0000 mgr.x (mgr.24542) 3894 : cluster [DBG] pgmap v4734: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:08.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:08 smithi154 conmon[34553]: 2021-07-31T03:18:08.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:08 smithi045 conmon[39718]: cluster 2021-07-31T03:18:07.096350+0000 mgr.x ( 2021-07-31T03:18:08.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:08 smithi045 conmon[39718]: mgr.24542) 3894 : cluster [DBG] pgmap v4734: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:08.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:08 smithi045 conmon[33655]: cluster 2021-07-31T03:18:07.096350+0000 mgr.x (mgr.24542) 3894 : cluster 2021-07-31T03:18:08.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:08 smithi045 conmon[33655]: [DBG] pgmap v4734: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:10.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:10 smithi154 conmon[34553]: cluster 2021-07-31T03:18:09.096942+0000 mgr.x (mgr.24542) 3895 : cluster [DBG] pgmap v4735: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:10.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:10 smithi045 conmon[39718]: cluster 2021-07-31T03:18: 2021-07-31T03:18:10.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:10 smithi045 conmon[39718]: 09.096942+0000 mgr.x (mgr.24542) 3895 : cluster [DBG] pgmap v4735: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:10.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:10 smithi045 conmon[33655]: cluster 2021-07-31T03:18:09.096942+0000 mgr.x (mgr.24542) 3895 2021-07-31T03:18:10.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:10 smithi045 conmon[33655]: : cluster [DBG] pgmap v4735: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:10.952 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:18:11.233 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:18:11.233 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 93m ago 93m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:18:11.234 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 90m ago 93m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:18:11.234 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 90m ago 97m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:18:11.234 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 93m ago 99m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:18:11.234 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 93m ago 99m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:18:11.235 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 90m ago 97m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:18:11.235 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 93m ago 97m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:18:11.235 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 93m ago 94m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:18:11.235 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 90m ago 94m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:18:11.235 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 93m ago 96m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:18:11.236 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 93m ago 96m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:18:11.236 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 93m ago 96m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:18:11.236 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 93m ago 95m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:18:11.236 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 90m ago 95m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:18:11.236 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 90m ago 95m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:18:11.237 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 90m ago 94m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:18:11.237 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 90m ago 94m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:18:11.237 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 90m ago 94m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:18:11.569 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:18:11.570 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:18:11.570 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:18:11.570 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:18:11.570 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:18:11.570 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:18:11.571 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:18:11.571 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:18:11.571 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:18:11.571 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:18:11.572 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:18:11.572 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:18:11.572 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:18:11.572 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:18:11.572 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:18:11.573 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:18:11.573 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:18:12.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:12 smithi154 conmon[34553]: audit 2021-07-31T03:18:10.940914+0000 mgr.x (mgr.24542) 3896 : audit [DBG] from='client.29158 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:18:12.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:12 smithi154 conmon[34553]: cluster 2021-07-31T03:18:11.097360+0000 mgr.x (mgr.24542) 3897 : cluster [DBG] pgmap v4736: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:12.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:12 smithi154 conmon[34553]: audit 2021-07-31T03:18:11.225900+0000 mgr.x (mgr.24542) 3898 : audit [DBG] from='client.29164 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:18:12.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:12 smithi154 conmon[34553]: audit 2021-07-31T03:18:11.570251+0000 mon.c (mon.1) 2021-07-31T03:18:12.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:12 smithi154 conmon[34553]: 193 : audit [DBG] from='client.? 172.21.15.45:0/2658485741' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:18:12.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:12 smithi045 conmon[33655]: audit 2021-07-31T03:18:10.940914 2021-07-31T03:18:12.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:12 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 3896 : audit [DBG] from='client.29158 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:18:12.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:12 smithi045 conmon[33655]: cluster 2021-07-31T03:18:12.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:12 smithi045 conmon[33655]: 2021-07-31T03:18:11 2021-07-31T03:18:12.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:12 smithi045 conmon[33655]: .097360+0000 mgr.x (mgr.24542) 3897 2021-07-31T03:18:12.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:12 smithi045 conmon[33655]: : cluster [DBG] pgmap v4736: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:12.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:12 smithi045 conmon[33655]: audit 2021-07-31T03:18:11.225900+0000 mgr.x (mgr 2021-07-31T03:18:12.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:12 smithi045 conmon[33655]: .24542) 3898 : audit [DBG] from='client.29164 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:18:12.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:12 smithi045 conmon[33655]: audit 2021 2021-07-31T03:18:12.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:12 smithi045 conmon[33655]: -07-31T03:18:11.570251+0000 mon.c (mon.1) 193 : audit [DBG] from='client.? 172.21.15.45:0/2658485741' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:18:12.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:12 smithi045 conmon[39718]: audit 2021-07-31T03:18:10.940914+0000 2021-07-31T03:18:12.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:12 smithi045 conmon[39718]: mgr.x (mgr.24542) 3896 : audit [DBG] from='client.29158 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:18:12.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:12 smithi045 conmon[39718]: cluster 2021-07-31T03:18:12.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:12 smithi045 conmon[39718]: 2021-07-31T03:18:11 2021-07-31T03:18:12.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:12 smithi045 conmon[39718]: .097360+0000 mgr.x (mgr.24542) 3897 2021-07-31T03:18:12.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:12 smithi045 conmon[39718]: : cluster [DBG] pgmap v4736: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:12.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:12 smithi045 conmon[39718]: audit 2021 2021-07-31T03:18:12.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:12 smithi045 conmon[39718]: -07-31T03:18:11. 2021-07-31T03:18:12.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:12 smithi045 conmon[39718]: 225900+0000 mgr.x (mgr.24542) 3898 : audit 2021-07-31T03:18:12.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:12 smithi045 conmon[39718]: [DBG] from='client.29164 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:18:12.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:12 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T03:18:12.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:12 smithi045 conmon[39718]: T03:18:11.570251+0000 mon.c 2021-07-31T03:18:12.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:12 smithi045 conmon[39718]: (mon.1) 193 : audit [DBG] 2021-07-31T03:18:12.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:12 smithi045 conmon[39718]: from='client.? 172.21.15.45:0/2658485741' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:18:14.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:14 smithi154 conmon[34553]: cluster 2021-07-31T03:18:13.097905+0000 mgr.x (mgr.24542) 3899 : cluster [DBG] pgmap v4737: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:14.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:14 smithi045 conmon[39718]: cluster 2021-07-31T03:18:13.097905+0000 mgr.x ( 2021-07-31T03:18:14.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:14 smithi045 conmon[39718]: mgr.24542) 3899 : cluster [DBG] pgmap v4737: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:14.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:14 smithi045 conmon[33655]: cluster 2021-07-31T03:18:13.097905+0000 mgr.x (mgr.24542) 3899 : cluster [DBG] pgmap v4737: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:16.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:16 smithi154 conmon[34553]: cluster 2021-07-31T03:18:15.098542+0000 mgr.x (mgr.24542) 3900 : cluster [DBG] pgmap v4738: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:16.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:16 smithi045 conmon[33655]: cluster 2021-07-31T03:18:15.098542+0000 mgr.x (mgr.24542) 3900 : 2021-07-31T03:18:16.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:16 smithi045 conmon[33655]: cluster [DBG] pgmap v4738: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:16.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:16 smithi045 conmon[39718]: cluster 2021-07-31T03:18:15.098542+0000 mgr.x 2021-07-31T03:18:16.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:16 smithi045 conmon[39718]: (mgr.24542) 3900 : cluster [DBG] pgmap v4738: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:18.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:18 smithi154 conmon[34553]: cluster 2021-07-31T03:18:17.098979+0000 mgr.x (mgr.24542) 3901 : cluster [DBG] pgmap v4739: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:18.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:18 smithi045 conmon[39718]: cluster 2021-07-31T03:18:17.098979+0000 mgr.x (mgr.24542) 3901 : cluster [DBG] pgmap v4739: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:18.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:18 smithi045 conmon[33655]: cluster 2021-07-31T03:18:17.098979+0000 mgr.x (mgr.24542) 3901 : cluster [DBG] pgmap v4739: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:18.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:18 smithi045 conmon[33655]: 2021-07-31T03:18:20.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:20 smithi154 conmon[34553]: cluster 2021-07-31T03:18:19.099436+0000 mgr.x (mgr.24542) 3902 : cluster [DBG] pgmap v4740: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:20.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:20 smithi045 conmon[39718]: cluster 2021-07-31T03:18:19.099436+0000 mgr.x (mgr.24542 2021-07-31T03:18:20.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:20 smithi045 conmon[39718]: ) 3902 : cluster [DBG] pgmap v4740: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:20.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:20 smithi045 conmon[33655]: cluster 2021-07-31T03:18:19.099436+0000 mgr.x (mgr.24542) 3902 : cluster [DBG] 2021-07-31T03:18:20.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:20 smithi045 conmon[33655]: pgmap v4740: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:22.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:22 smithi154 conmon[34553]: cluster 2021-07-31T03:18:21.099992+0000 mgr.x (mgr.24542) 3903 : cluster [DBG] pgmap v4741: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:22.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:22 smithi045 conmon[39718]: cluster 2021-07-31T03:18:21.099992+0000 2021-07-31T03:18:22.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:22 smithi045 conmon[39718]: mgr.x (mgr.24542) 3903 : cluster [DBG] pgmap v4741: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:22.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:22 smithi045 conmon[33655]: cluster 2021-07-31T03:18:21.099992+0000 mgr.x (mgr.24542) 3903 : 2021-07-31T03:18:22.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:22 smithi045 conmon[33655]: cluster [DBG] pgmap v4741: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:24.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:24 smithi154 conmon[34553]: cluster 2021-07-31T03:18:23.100461+0000 mgr.x (mgr.24542) 3904 : cluster [DBG] pgmap v4742: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:24.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:24 smithi045 conmon[39718]: cluster 2021-07-31T03:18:23.100461+0000 mgr.x (mgr.24542 2021-07-31T03:18:24.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:24 smithi045 conmon[39718]: ) 3904 : cluster [DBG] pgmap v4742: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:24.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:24 smithi045 conmon[33655]: cluster 2021-07-31T03:18:23.100461+0000 mgr.x (mgr.24542) 3904 : 2021-07-31T03:18:24.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:24 smithi045 conmon[33655]: cluster [DBG] pgmap v4742: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:26.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:26 smithi154 conmon[34553]: cluster 2021-07-31T03:18:25.100974+0000 mgr.x (mgr.24542) 3905 : cluster [DBG] pgmap v4743: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:26.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:26 smithi045 conmon[39718]: cluster 2021-07-31T03:18:25.100974+0000 mgr.x (mgr.24542 2021-07-31T03:18:26.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:26 smithi045 conmon[39718]: ) 3905 : cluster [DBG] pgmap v4743: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:26.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:26 smithi045 conmon[33655]: cluster 2021-07-31T03:18:25.100974+0000 mgr.x (mgr.24542) 3905 2021-07-31T03:18:26.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:26 smithi045 conmon[33655]: : cluster [DBG] pgmap v4743: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:28.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:28 smithi154 conmon[34553]: cluster 2021-07-31T03:18:27.101488+0000 mgr.x (mgr.24542) 3906 : cluster [DBG] pgmap v4744: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:28.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:28 smithi154 conmon[34553]: audit 2021-07-31T03:18:28.297598+0000 mon.b (mon.2) 530 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:18:28.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:28 smithi154 conmon[34553]: audit 2021-07-31T03:18:28.297844+0000 mon.b (mon.2) 531 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:18:28.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:28 smithi154 conmon[34553]: audit 2021-07-31T03:18:28.298067+0000 mon.a (mon.0) 3282 : audit 2021-07-31T03:18:28.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:28 smithi154 conmon[34553]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:18:28.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:28 smithi154 conmon[34553]: audit 2021-07-31T 2021-07-31T03:18:28.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:28 smithi154 conmon[34553]: 03:18:28.298484+0000 mon.a (mon.0 2021-07-31T03:18:28.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:28 smithi154 conmon[34553]: ) 3283 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:18:28.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:28 smithi045 conmon[39718]: cluster 2021-07-31T03:18:27.101488 2021-07-31T03:18:28.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:28 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 3906 : cluster [DBG] pgmap v4744: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:28.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:28 smithi045 conmon[39718]: audit 2021-07-31T03 2021-07-31T03:18:28.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:28 smithi045 conmon[39718]: :18:28.297598+0000 mon.b (mon.2) 530 : 2021-07-31T03:18:28.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:28 smithi045 conmon[39718]: audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:18:28.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:28 smithi045 conmon[39718]: audit 2021-07 2021-07-31T03:18:28.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:28 smithi045 conmon[39718]: -31T03:18:28.297844+0000 mon.b (mon.2) 2021-07-31T03:18:28.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:28 smithi045 conmon[39718]: 531 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:18:28.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:28 smithi045 conmon[39718]: audit 2021- 2021-07-31T03:18:28.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:28 smithi045 conmon[39718]: 07-31T03:18:28.298067+0000 mon.a (mon. 2021-07-31T03:18:28.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:28 smithi045 conmon[39718]: 0) 3282 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:18:28.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:28 smithi045 conmon[39718]: 2021-07-31T03:18:28.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:28 smithi045 conmon[39718]: audit 2021-07 2021-07-31T03:18:28.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:28 smithi045 conmon[39718]: -31T03:18:28.298484+0000 mon.a (mon.0) 2021-07-31T03:18:28.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:28 smithi045 conmon[39718]: 3283 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:18:28.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:28 smithi045 conmon[33655]: cluster 2021-07-31T03:18:27.101488+0000 mgr.x (mgr.24542) 2021-07-31T03:18:28.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:28 smithi045 conmon[33655]: 3906 : cluster [DBG] pgmap v4744: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:28.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:28 smithi045 conmon[33655]: audit 2021-07-31T03:18:28.297598+0000 mon.b (mon.2) 530 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:18:28.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:28 smithi045 conmon[33655]: audit 2021-07-31T03:18:28.297844+0000 mon.b ( 2021-07-31T03:18:28.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:28 smithi045 conmon[33655]: mon.2) 531 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:18:28.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:28 smithi045 conmon[33655]: audit 2021-07-31T03:18:28.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:28 smithi045 conmon[33655]: 2021-07-31T03: 2021-07-31T03:18:28.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:28 smithi045 conmon[33655]: 18:28.298067+0000 mon.a (mon.0 2021-07-31T03:18:28.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:28 smithi045 conmon[33655]: ) 3282 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:18:28.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:28 smithi045 conmon[33655]: audit 2021 2021-07-31T03:18:28.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:28 smithi045 conmon[33655]: -07-31T03:18: 2021-07-31T03:18:28.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:28 smithi045 conmon[33655]: 28.298484+0000 mon.a (mon.0 2021-07-31T03:18:28.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:28 smithi045 conmon[33655]: ) 3283 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:18:30.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:30 smithi154 conmon[34553]: cluster 2021-07-31T03:18:29.102063+0000 mgr.x (mgr.24542) 3907 : cluster [DBG] pgmap v4745: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:30.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:30 smithi045 conmon[39718]: cluster 2021-07-31T03:18:29.102063+0000 mgr.x ( 2021-07-31T03:18:30.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:30 smithi045 conmon[39718]: mgr.24542) 3907 : cluster [DBG] pgmap v4745: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:30.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:30 smithi045 conmon[33655]: cluster 2021-07-31T03:18:29.102063+0000 mgr.x (mgr.24542) 3907 2021-07-31T03:18:30.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:30 smithi045 conmon[33655]: : cluster [DBG] pgmap v4745: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:32.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[34553]: cluster 2021-07-31T03:18:31.102500+0000 mgr.x (mgr.24542) 3908 : cluster [DBG] pgmap v4746: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:32.753 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: debug 2021-07-31T03:18:32.453+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:18:32.753 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:18:32.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:18:32.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: yield 2021-07-31T03:18:32.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:18:32.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:18:32.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:18:32.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:18:32.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:18:32.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:18:32.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:18:32.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:18:32.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:18:32.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: task.result() 2021-07-31T03:18:32.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:18:32.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: debug 2021-07-31T03:18:32.465+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:18:32.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:18:32.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:18:32.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: yield 2021-07-31T03:18:32.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:18:32.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:18:32.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:18:32.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:18:32.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:18:32.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:18:32.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:18:32.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:18:32.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:18:32.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: task.result() 2021-07-31T03:18:32.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:18:32.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: debug 2021-07-31T03:18:32.490+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:18:32.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:18:32.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:18:32.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: yield 2021-07-31T03:18:32.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:18:32.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:18:32.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:18:32.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:18:32.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:18:32.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:18:32.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:18:32.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:18:32.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:18:32.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: task.result() 2021-07-31T03:18:32.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:18:32.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: debug 2021-07-31T03:18:32.491+0000 7fcb93160700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:18:32.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:18:32.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:18:32.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: yield 2021-07-31T03:18:32.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:18:32.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:18:32.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:18:32.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:18:32.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:18:32.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:18:32.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:18:32.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:18:32.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:18:32.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: task.result() 2021-07-31T03:18:32.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:18:32.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: 2021-07-31T03:18:32.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:18:32.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: 2021-07-31T03:18:32.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:18:32.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:18:32.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:18:32.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:18:32.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:18:32.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:18:32.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:18:32.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:18:32.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:18:32.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:18:32.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:18:32.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:18:32.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:18:32.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:18:32.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:18:32.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:18:32.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:18:32.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:18:32.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: raise 2021-07-31T03:18:32.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:18:32.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:18:32.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:18:32.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:18:32.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:18:32.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: debug 2021-07-31T03:18:32.499+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:18:32.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:18:32.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:18:32.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: yield 2021-07-31T03:18:32.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:18:32.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:18:32.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:18:32.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:18:32.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:18:32.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:18:32.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:18:32.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:18:32.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:18:32.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: task.result() 2021-07-31T03:18:32.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:18:32.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: debug 2021-07-31T03:18:32.499+0000 7fcb9295f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:18:32.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:18:32.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:18:32.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: yield 2021-07-31T03:18:32.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:18:32.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:18:32.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:18:32.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:18:32.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:18:32.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:18:32.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:18:32.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:18:32.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:18:32.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: task.result() 2021-07-31T03:18:32.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:18:32.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: 2021-07-31T03:18:32.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:18:32.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: 2021-07-31T03:18:32.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:18:32.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:18:32.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:18:32.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:18:32.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:18:32.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:18:32.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:18:32.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:18:32.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:18:32.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:18:32.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:18:32.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:18:32.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:18:32.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:18:32.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:18:32.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:18:32.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:18:32.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:18:32.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: raise 2021-07-31T03:18:32.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:18:32.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:18:32.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:18:32.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:18:32.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:18:32 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:18:32.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:32 smithi045 conmon[33655]: cluster 2021-07-31T03:18:31.102500+0000 mgr.x (mgr.24542) 2021-07-31T03:18:32.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:32 smithi045 conmon[33655]: 3908 : cluster [DBG] pgmap v4746: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:32.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:32 smithi045 conmon[39718]: cluster 2021-07-31T03:18:31.102500+0000 mgr.x (mgr.24542) 3908 : cluster [DBG] pgmap v4746: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:33.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: audit 2021-07-31T03:18:32.408088+0000 mon.b (mon.2) 532 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:18:33.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: audit 2021-07-31T03:18:32.409504+0000 mon.b (mon.2) 533 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:18:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: audit 2021-07-31T03:18:32.410615+0000 mon.b (mon.2) 534 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:18:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:18:32.411397+0000 mgr.x 2021-07-31T03:18:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: (mgr.24542) 3909 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:18:33.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: cephadm 2021-07- 2021-07-31T03:18:33.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: 31T03:18:32.454451+0000 mgr.x ( 2021-07-31T03:18:33.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: mgr.24542) 3910 : cephadm [ERR] 2021-07-31T03:18:33.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:18:33.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:18:33.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: yield 2021-07-31T03:18:33.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:18:33.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:18:33.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:18:33.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:18:33.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:18:33.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:18:33.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:18:33.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:18:33.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:18:33.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: task.result() 2021-07-31T03:18:33.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:18:33.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:18:32.454888+0000 mgr.x (mgr.24542) 3911 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:18:33.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:18:32.465986+0000 mgr.x (mgr.24542) 3912 : cephadm [ERR] 2021-07-31T03:18:33.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: 2021-07-31T03:18:33.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:18:33.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:18:33.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: yield 2021-07-31T03:18:33.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:18:33.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:18:33.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:18:33.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:18:33.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:18:33.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:18:33.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:18:33.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:18:33.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:18:33.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: task.result() 2021-07-31T03:18:33.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:18:33.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:18:33.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: 2021-07-31T03:18:32.466784+0000 mgr.x (mgr.24542) 3913 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:18:33.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:18:32.491375+0000 mgr.x (mgr.24542) 3914 : cephadm [ERR] 2021-07-31T03:18:33.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:18:33.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:18:33.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: yield 2021-07-31T03:18:33.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:18:33.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:18:33.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:18:33.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:18:33.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:18:33.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:18:33.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:18:33.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:18:33.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:18:33.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: task.result() 2021-07-31T03:18:33.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:18:33.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: cephadm 2021-07-31 2021-07-31T03:18:33.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: T03:18:32.492306+0000 mgr.x (mgr.24542) 3915 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:18:33.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:18:33.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:18:33.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: yield 2021-07-31T03:18:33.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:18:33.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:18:33.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:18:33.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:18:33.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:18:33.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:18:33.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:18:33.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:18:33.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:18:33.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: task.result() 2021-07-31T03:18:33.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:18:33.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: 2021-07-31T03:18:33.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:18:33.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: 2021-07-31T03:18:33.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:18:33.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:18:33.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:18:33.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:18:33.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:18:33.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:18:33.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:18:33.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:18:33.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:18:33.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:18:33.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:18:33.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:18:33.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:18:33.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:18:33.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:18:33.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:18:33.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:18:33.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:18:33.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: raise 2021-07-31T03:18:33.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:18:33.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:18:33.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:18:33.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:18:33.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:18:33.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:18:32.500322+0000 mgr.x (mgr.24542) 3916 : cephadm [ERR] 2021-07-31T03:18:33.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:18:33.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:18:33.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: yield 2021-07-31T03:18:33.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:18:33.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:18:33.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:18:33.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:18:33.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:18:33.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:18:33.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:18:33.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:18:33.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:18:33.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: task.result() 2021-07-31T03:18:33.782 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:18:33.782 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:18:32.500675+0000 mgr.x (mgr.24542) 3917 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:18:33.782 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:18:33.782 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:18:33.783 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: yield 2021-07-31T03:18:33.783 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:18:33.783 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:18:33.783 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:18:33.783 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:18:33.784 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:18:33.784 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:18:33.784 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:18:33.784 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:18:33.784 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:18:33.785 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: task.result() 2021-07-31T03:18:33.785 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:18:33.785 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: 2021-07-31T03:18:33.785 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:18:33.785 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: 2021-07-31T03:18:33.786 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:18:33.786 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:18:33.786 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:18:33.786 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:18:33.786 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:18:33.787 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:18:33.787 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:18:33.787 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:18:33.787 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:18:33.787 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:18:33.788 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:18:33.788 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:18:33.788 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:18:33.788 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:18:33.789 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:18:33.789 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:18:33.789 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:18:33.789 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:18:33.789 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: raise 2021-07-31T03:18:33.790 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:18:33.790 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:18:33.790 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:18:33.790 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:18:33.791 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:33 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:18:33.791 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: audit 2021-07-31T03:18:32.408088+0000 mon.b (mon.2) 532 2021-07-31T03:18:33.791 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:18:33.792 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: audit 2021-07-31T03:18:32.409504+0000 mon.b (mon.2) 533 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:18:33.792 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: audit 2021-07-31T03:18:32.410615+0000 mon.b (mon.2) 534 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:18:33.792 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: cephadm 2021-07-31T03:18:32.411397+0000 mgr.x (mgr.24542) 3909 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:18:33.792 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: cephadm 2021-07-31T03 2021-07-31T03:18:33.793 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: :18:32.454451+0000 mgr.x (mgr.24542) 3910 : cephadm [ERR] 2021-07-31T03:18:33.793 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:18:33.793 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:18:33.793 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: yield 2021-07-31T03:18:33.793 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:18:33.794 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:18:33.794 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:18:33.794 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:18:33.794 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:18:33.795 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:18:33.795 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:18:33.795 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:18:33.795 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:18:33.795 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: task.result() 2021-07-31T03:18:33.796 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:18:33.796 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: cephadm 2021-07-31T03:18:32.454888+0000 mgr.x (mgr.24542) 3911 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:18:33.796 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: cephadm 2021-07-31T03:18:32.465986+0000 mgr.x (mgr.24542) 3912 : cephadm [ERR] 2021-07-31T03:18:33.796 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:18:33.796 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:18:33.797 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: yield 2021-07-31T03:18:33.797 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:18:33.797 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:18:33.797 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:18:33.797 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:18:33.798 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:18:33.798 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:18:33.798 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:18:33.798 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:18:33.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: audit 2021-07-31T03:18:32 2021-07-31T03:18:33.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: .408088+0000 mon.b (mon.2) 532 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:18:33.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: audit 2021-07-31T03:18 2021-07-31T03:18:33.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: :32.409504+0000 mon.b (mon.2) 533 : audit 2021-07-31T03:18:33.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:18:33.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: audit 2021-07 2021-07-31T03:18:33.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: -31T03:18:32.410615+0000 mon.b (mon. 2021-07-31T03:18:33.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: 2) 534 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:18:33.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: 2021-07-31T03:18:33.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: cephadm 2021-07-31T03:18 2021-07-31T03:18:33.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: :32.411397+0000 mgr.x (mgr.24542) 3909 : cephadm [WRN] 2021-07-31T03:18:33.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:18:33.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: cephadm 2021-07-31T03:18:32.454451+0000 mgr.x (mgr 2021-07-31T03:18:33.802 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: .24542) 3910 : cephadm [ERR] 2021-07-31T03:18:33.802 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:18:33.802 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:18:33.802 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: yield 2021-07-31T03:18:33.802 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:18:33.803 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:18:33.803 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:18:33.803 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:18:33.803 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:18:33.803 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:18:33.804 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:18:33.804 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:18:33.804 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:18:33.804 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: task.result() 2021-07-31T03:18:33.804 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:18:33.805 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: cephadm 2021-07-31T03:18:32.454888+0000 mgr.x (mgr.24542) 3911 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:18:33.805 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: cephadm 2021-07-31T03:18:32.465986+0000 mgr.x (mgr.24542) 3912 : cephadm [ERR] 2021-07-31T03:18:33.805 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:18:33.805 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:18:33.805 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: yield 2021-07-31T03:18:33.806 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:18:33.806 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:18:33.806 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:18:33.806 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:18:33.806 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:18:33.807 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:18:33.807 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: task.result() 2021-07-31T03:18:33.807 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:18:33.808 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: cephadm 2021-07-31T03:18:33.808 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: 2021-07-31T03:18:32.466784+0000 mgr.x (mgr.24542) 3913 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:18:33.808 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: cephadm 2021-07-31T03:18:32.491375+0000 mgr.x (mgr.24542) 3914 : cephadm 2021-07-31T03:18:33.808 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: [ERR] 2021-07-31T03:18:33.808 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:18:33.809 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:18:33.809 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: yield 2021-07-31T03:18:33.809 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:18:33.809 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:18:33.809 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:18:33.810 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:18:33.810 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:18:33.810 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:18:33.810 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:18:33.811 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:18:33.811 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:18:33.811 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: task.result() 2021-07-31T03:18:33.811 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:18:33.811 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: cephadm 2021-07-31T03:18:33.812 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: 2021-07-31T03:18:32.492306+0000 mgr.x (mgr.24542) 3915 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:18:33.812 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:18:33.812 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:18:33.812 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: yield 2021-07-31T03:18:33.812 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:18:33.813 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:18:33.813 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:18:33.813 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:18:33.813 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:18:33.813 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:18:33.814 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:18:33.814 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:18:33.814 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:18:33.814 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: task.result() 2021-07-31T03:18:33.814 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:18:33.815 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: 2021-07-31T03:18:33.815 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:18:33.815 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: 2021-07-31T03:18:33.815 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:18:33.815 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:18:33.816 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:18:33.816 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:18:33.816 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:18:33.817 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:18:33.817 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:18:33.817 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: task.result() 2021-07-31T03:18:33.817 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:18:33.817 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: cephadm 2021-07-31T03:18:32.466784+0000 mgr.x (mgr.24542) 3913 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:18:33.818 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: cephadm 2021-07-31T03:18:32.491375+0000 mgr.x (mgr.24542) 3914 : cephadm [ERR] 2021-07-31T03:18:33.818 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:18:33.818 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:18:33.818 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: yield 2021-07-31T03:18:33.819 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:18:33.819 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:18:33.819 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:18:33.819 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:18:33.819 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:18:33.820 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:18:33.820 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:18:33.820 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:18:33.820 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:18:33.820 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: task.result() 2021-07-31T03:18:33.821 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:18:33.821 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: cephadm 2021-07-31T03:18:32.492306+0000 mgr.x (mgr.24542) 3915 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:18:33.821 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:18:33.821 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:18:33.821 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: yield 2021-07-31T03:18:33.822 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:18:33.822 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:18:33.822 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:18:33.822 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:18:33.823 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:18:33.823 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:18:33.823 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:18:33.823 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:18:33.823 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:18:33.824 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: task.result() 2021-07-31T03:18:33.824 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:18:33.824 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: 2021-07-31T03:18:33.824 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:18:33.824 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: 2021-07-31T03:18:33.825 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:18:33.825 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:18:33.825 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:18:33.826 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:18:33.826 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:18:33.826 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:18:33.826 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:18:33.826 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:18:33.827 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:18:33.827 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:18:33.827 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:18:33.827 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:18:33.827 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:18:33.828 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:18:33.828 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:18:33.828 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:18:33.828 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: raise 2021-07-31T03:18:33.828 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:18:33.829 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:18:33.829 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:18:33.829 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:18:33.829 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:18:33.829 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: cephadm 2021-07-31T03:18:32.500322+0000 mgr.x (mgr.24542) 3916 : cephadm 2021-07-31T03:18:33.830 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: [ERR] 2021-07-31T03:18:33.830 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:18:33.830 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:18:33.830 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: yield 2021-07-31T03:18:33.830 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:18:33.831 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:18:33.831 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:18:33.831 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:18:33.831 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:18:33.831 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:18:33.832 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:18:33.832 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:18:33.832 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:18:33.832 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: task.result() 2021-07-31T03:18:33.832 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:18:33.833 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: cephadm 2021-07-31T03:18:32.500675+0000 mgr.x (mgr.24542) 3917 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:18:33.833 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:18:33.833 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:18:33.833 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:18:33.834 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:18:33.834 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:18:33.834 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:18:33.834 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:18:33.834 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:18:33.835 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:18:33.835 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:18:33.835 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:18:33.835 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:18:33.835 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:18:33.836 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:18:33.836 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:18:33.836 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:18:33.836 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:18:33.836 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:18:33.837 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: raise 2021-07-31T03:18:33.837 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:18:33.837 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:18:33.837 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:18:33.837 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:18:33.838 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:18:33.838 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: cephadm 2021-07-31T03:18:32.500322+0000 mgr.x (mgr.24542) 3916 : cephadm [ERR] 2021-07-31T03:18:33.838 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:18:33.838 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:18:33.838 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: yield 2021-07-31T03:18:33.839 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:18:33.839 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:18:33.839 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:18:33.839 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:18:33.839 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:18:33.840 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:18:33.840 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:18:33.840 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:18:33.840 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:18:33.840 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: task.result() 2021-07-31T03:18:33.840 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:18:33.841 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: cephadm 2021-07-31T03:18:32.500675+0000 mgr.x (mgr.24542) 3917 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:18:33.841 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:18:33.842 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: yield 2021-07-31T03:18:33.842 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:18:33.842 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:18:33.842 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:18:33.842 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:18:33.843 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:18:33.843 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:18:33.843 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:18:33.843 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:18:33.843 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:18:33.844 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: task.result() 2021-07-31T03:18:33.844 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:18:33.844 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: 2021-07-31T03:18:33.844 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:18:33.844 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: 2021-07-31T03:18:33.845 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:18:33.845 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:18:33.845 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:18:33.845 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:18:33.845 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:18:33.846 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:18:33.846 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:18:33.846 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:18:33.846 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:18:33.846 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:18:33.847 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:18:33.847 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:18:33.847 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:18:33.847 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:18:33.847 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:18:33.848 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:18:33.848 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:18:33.848 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:18:33.848 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: raise 2021-07-31T03:18:33.848 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:18:33.849 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:18:33.849 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:18:33.849 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:18:33.849 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:18:33.849 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:18:33.850 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:18:33.850 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: yield 2021-07-31T03:18:33.850 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:18:33.850 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:18:33.851 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:18:33.851 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:18:33.851 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:18:33.851 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:18:33.851 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:18:33.852 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:18:33.852 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:18:33.852 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: task.result() 2021-07-31T03:18:33.852 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:18:33.852 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: 2021-07-31T03:18:33.852 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:18:33.853 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: 2021-07-31T03:18:33.853 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:18:33.853 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:18:33.853 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:18:33.853 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:18:33.854 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:18:33.854 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:18:33.854 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:18:33.854 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:18:33.854 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:18:33.855 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:18:33.855 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:18:33.855 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:18:33.855 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:18:33.855 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:18:33.856 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:18:33.856 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:18:33.856 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:18:33.856 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:18:33.856 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: raise 2021-07-31T03:18:33.857 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:18:33.857 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:18:33.857 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:18:33.857 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:18:33.857 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:33 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:18:34.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:34 smithi154 conmon[34553]: cluster 2021-07-31T03:18:33.102963+0000 mgr.x (mgr.24542) 3918 : cluster [DBG] pgmap v4747: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:34.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:34 smithi045 conmon[33655]: cluster 2021-07-31T03:18:33.102963+0000 mgr.x (mgr.24542) 3918 : cluster 2021-07-31T03:18:34.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:34 smithi045 conmon[33655]: [DBG] pgmap v4747: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:34.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:34 smithi045 conmon[39718]: cluster 2021-07-31T03:18:33.102963+0000 mgr.x (mgr. 2021-07-31T03:18:34.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:34 smithi045 conmon[39718]: 24542) 3918 : cluster [DBG] pgmap v4747: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:36.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:36 smithi154 conmon[34553]: cluster 2021-07-31T03:18:35.103568+0000 mgr.x (mgr.24542) 3919 : cluster [DBG] pgmap v4748: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:36.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:36 smithi045 conmon[33655]: cluster 2021-07-31T03:18:35.103568 2021-07-31T03:18:36.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:36 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 3919 : cluster [DBG] pgmap v4748: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:36.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:36 smithi045 conmon[39718]: cluster 2021-07-31T03:18:35.103568+0000 mgr.x ( 2021-07-31T03:18:36.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:36 smithi045 conmon[39718]: mgr.24542) 3919 : cluster [DBG] pgmap v4748: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:38.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:38 smithi154 conmon[34553]: cluster 2021-07-31T03:18:37.103982+0000 mgr.x (mgr.24542) 3920 : cluster [DBG] pgmap v4749: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:38.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:38 smithi045 conmon[33655]: cluster 2021-07-31T03:18:37.103982+0000 mgr.x (mgr.24542) 3920 : cluster [DBG] pgmap v4749: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:38.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:38 smithi045 conmon[39718]: cluster 2021-07-31T03:18:37.103982+0000 mgr.x 2021-07-31T03:18:38.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:38 smithi045 conmon[39718]: (mgr.24542) 3920 : cluster [DBG] pgmap v4749: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:40.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:40 smithi154 conmon[34553]: cluster 2021-07-31T03:18:39.104557+0000 mgr.x (mgr.24542) 3921 : cluster [DBG] pgmap v4750: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:40.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:40 smithi154 conmon[34553]: 2021-07-31T03:18:40.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:40 smithi045 conmon[39718]: cluster 2021-07-31T03:18:39.104557+0000 mgr.x ( 2021-07-31T03:18:40.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:40 smithi045 conmon[39718]: mgr.24542) 3921 : cluster [DBG] pgmap v4750: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:40.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:40 smithi045 conmon[33655]: cluster 2021-07-31T03:18:39.104557+0000 2021-07-31T03:18:40.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:40 smithi045 conmon[33655]: mgr.x (mgr.24542) 3921 : cluster [DBG] pgmap v4750: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:41.884 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:18:42.160 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:18:42.160 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 93m ago 94m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:18:42.160 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 91m ago 94m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:18:42.161 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 91m ago 97m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:18:42.161 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 93m ago 99m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:18:42.161 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 93m ago 99m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:18:42.161 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 91m ago 97m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:18:42.161 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 93m ago 98m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:18:42.162 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 93m ago 94m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:18:42.162 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 91m ago 94m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:18:42.162 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 93m ago 97m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:18:42.162 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 93m ago 97m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:18:42.162 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 93m ago 96m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:18:42.163 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 93m ago 96m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:18:42.163 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 91m ago 96m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:18:42.163 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 91m ago 95m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:18:42.163 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 91m ago 95m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:18:42.163 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 91m ago 95m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:18:42.164 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 91m ago 94m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:18:42.497 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:18:42.497 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:18:42.497 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:18:42.497 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:18:42.497 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:18:42.498 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:18:42.498 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:18:42.498 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:18:42.498 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:18:42.498 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:18:42.499 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:18:42.499 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:18:42.499 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:18:42.499 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:18:42.499 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:18:42.500 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:18:42.500 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:18:42.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:42 smithi154 conmon[34553]: cluster 2021-07-31T03:18:41.104989+0000 mgr.x (mgr.24542) 3922 : cluster [DBG] pgmap v4751: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:42.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:42 smithi045 conmon[33655]: cluster 2021-07-31T03:18:41 2021-07-31T03:18:42.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:42 smithi045 conmon[33655]: .104989+0000 mgr.x (mgr.24542) 3922 : cluster [DBG] pgmap v4751: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:42.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:42 smithi045 conmon[39718]: cluster 2021-07-31T03:18:41.104989+0000 mgr.x ( 2021-07-31T03:18:42.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:42 smithi045 conmon[39718]: mgr.24542) 3922 : cluster [DBG] pgmap v4751: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:43.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:43 smithi154 conmon[34553]: audit 2021-07-31T03:18:41.872609+0000 mgr.x (mgr.24542) 3923 : audit [DBG] from='client.29176 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:18:43.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:43 smithi154 conmon[34553]: audit 2021-07-31T03:18:42.152611+0000 mgr.x (mgr.24542) 3924 : audit [DBG] from='client.29182 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:18:43.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:43 smithi154 conmon[34553]: audit 2021-07-31T03:18:42.497336+0000 mon.c (mon.1) 194 : audit [DBG] from='client.? 172.21.15.45:0/1201263969' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:18:43.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:43 smithi045 conmon[33655]: audit 2021-07-31T03:18:41.872609+0000 mgr.x (mgr.24542) 3923 : audit [DBG] from='client.29176 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:18:43.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:43 smithi045 conmon[33655]: 2021-07-31T03:18:43.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:43 smithi045 conmon[33655]: audit 2021-07-31T03:18:42.152611+0000 mgr.x (mgr.24542) 3924 : audit [DBG] from='client.29182 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:18:43.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:43 smithi045 conmon[33655]: audit 2021-07-31T03:18:42.497336+0000 mon.c (mon.1) 194 : audit [DBG] from='client.? 172.21.15.45:0/1201263969' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:18:43.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:43 smithi045 conmon[39718]: audit 2021-07-31T03:18 2021-07-31T03:18:43.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:43 smithi045 conmon[39718]: :41.872609+0000 mgr.x (mgr.24542) 3923 : audit [DBG] from='client.29176 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:18:43.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:43 smithi045 conmon[39718]: audit 2021-07-31T03 2021-07-31T03:18:43.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:43 smithi045 conmon[39718]: :18:42.152611+0000 mgr.x (mgr. 2021-07-31T03:18:43.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:43 smithi045 conmon[39718]: 24542) 3924 : audit [DBG] from='client.29182 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:18:43.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:43 smithi045 conmon[39718]: audit 2021- 2021-07-31T03:18:43.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:43 smithi045 conmon[39718]: 07-31T03:18:42.497336+0000 2021-07-31T03:18:43.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:43 smithi045 conmon[39718]: mon.c (mon.1) 194 : audit [DBG] 2021-07-31T03:18:43.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:43 smithi045 conmon[39718]: from='client.? 172.21.15.45:0/1201263969' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:18:44.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:44 smithi154 conmon[34553]: cluster 2021-07-31T03:18:43.105437+0000 mgr.x (mgr.24542) 3925 : cluster [DBG] pgmap v4752: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:44.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:44 smithi045 conmon[39718]: cluster 2021-07-31T03:18:43.105437+0000 mgr.x (mgr.24542) 3925 : cluster [DBG] pgmap v4752: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:44.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:44 smithi045 conmon[33655]: cluster 2021-07-31T03:18:43.105437+0000 mgr.x (mgr.24542) 2021-07-31T03:18:44.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:44 smithi045 conmon[33655]: 3925 : cluster [DBG] pgmap v4752: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:46.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:46 smithi154 conmon[34553]: cluster 2021-07-31T03:18:45.106051+0000 mgr.x (mgr.24542) 3926 : cluster [DBG] pgmap v4753: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:46.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:46 smithi045 conmon[39718]: cluster 2021-07-31T03:18:45.106051+0000 mgr.x ( 2021-07-31T03:18:46.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:46 smithi045 conmon[39718]: mgr.24542) 3926 : cluster [DBG] pgmap v4753: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:46.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:46 smithi045 conmon[33655]: cluster 2021-07-31T03:18:45.106051+0000 mgr.x 2021-07-31T03:18:46.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:46 smithi045 conmon[33655]: (mgr.24542) 3926 : cluster [DBG] pgmap v4753: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:48.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:48 smithi154 conmon[34553]: cluster 2021-07-31T03:18:47.106475+0000 mgr.x (mgr.24542) 3927 : cluster [DBG] pgmap v4754: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:48.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:48 smithi045 conmon[39718]: cluster 2021-07-31T03:18:47.106475+0000 mgr.x (mgr 2021-07-31T03:18:48.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:48 smithi045 conmon[39718]: .24542) 3927 : cluster [DBG] pgmap v4754: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:48.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:48 smithi045 conmon[33655]: cluster 2021-07-31T03:18:47.106475 2021-07-31T03:18:48.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:48 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 3927 : cluster [DBG] pgmap v4754: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:50.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:50 smithi154 conmon[34553]: cluster 2021-07-31T03:18:49.107075+0000 mgr.x (mgr.24542) 3928 : cluster [DBG] pgmap v4755: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:50.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:50 smithi045 conmon[39718]: cluster 2021-07-31T03:18:49.107075+0000 mgr.x (mgr.24542) 3928 : cluster [DBG] pgmap v4755: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:50.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:50 smithi045 conmon[33655]: cluster 2021-07-31T03:18:49.107075+0000 mgr.x (mgr.24542) 3928 : 2021-07-31T03:18:50.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:50 smithi045 conmon[33655]: cluster [DBG] pgmap v4755: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:52.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:52 smithi154 conmon[34553]: cluster 2021-07-31T03:18:51.107601+0000 mgr.x (mgr.24542) 3929 : cluster [DBG] pgmap v4756: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:52.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:52 smithi154 conmon[34553]: 2021-07-31T03:18:52.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:52 smithi045 conmon[33655]: cluster 2021-07-31T03:18:51.107601+0000 mgr.x ( 2021-07-31T03:18:52.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:52 smithi045 conmon[33655]: mgr.24542) 3929 : cluster [DBG] pgmap v4756: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:52.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:52 smithi045 conmon[39718]: cluster 2021-07-31T03:18:51.107601+0000 mgr.x (mgr. 2021-07-31T03:18:52.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:52 smithi045 conmon[39718]: 24542) 3929 : cluster [DBG] pgmap v4756: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:54.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:54 smithi154 conmon[34553]: cluster 2021-07-31T03:18:53.108034+0000 mgr.x (mgr.24542) 3930 : cluster [DBG] pgmap v4757: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:54.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:54 smithi045 conmon[39718]: cluster 2021-07-31T03:18:53.108034+0000 mgr.x (mgr. 2021-07-31T03:18:54.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:54 smithi045 conmon[39718]: 24542) 3930 : cluster [DBG] pgmap v4757: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:54.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:54 smithi045 conmon[33655]: cluster 2021-07-31T03:18:53.108034+0000 mgr.x (mgr.24542) 3930 2021-07-31T03:18:54.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:54 smithi045 conmon[33655]: : cluster [DBG] pgmap v4757: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:56.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:56 smithi154 conmon[34553]: cluster 2021-07-31T03:18:55.108568+0000 mgr.x (mgr.24542) 3931 : cluster [DBG] pgmap v4758: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:56.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:56 smithi045 conmon[33655]: cluster 2021-07-31T03:18:55.108568+0000 mgr.x (mgr.24542) 3931 : 2021-07-31T03:18:56.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:56 smithi045 conmon[33655]: cluster [DBG] pgmap v4758: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:56.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:56 smithi045 conmon[39718]: cluster 2021-07-31T03:18:55.108568+0000 mgr.x ( 2021-07-31T03:18:56.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:56 smithi045 conmon[39718]: mgr.24542) 3931 : cluster [DBG] pgmap v4758: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:58.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:18:58 smithi154 conmon[34553]: cluster 2021-07-31T03:18:57.109089+0000 mgr.x (mgr.24542) 3932 : cluster [DBG] pgmap v4759: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:58.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:58 smithi045 conmon[39718]: cluster 2021-07-31T03:18:57.109089+0000 mgr.x (mgr 2021-07-31T03:18:58.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:18:58 smithi045 conmon[39718]: .24542) 3932 : cluster [DBG] pgmap v4759: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:18:58.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:58 smithi045 conmon[33655]: cluster 2021-07-31T03:18:57.109089+0000 mgr.x (mgr.24542) 3932 : cluster 2021-07-31T03:18:58.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:18:58 smithi045 conmon[33655]: [DBG] pgmap v4759: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:00.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:00 smithi154 conmon[34553]: cluster 2021-07-31T03:18:59.109483+0000 mgr.x (mgr.24542) 3933 : cluster [DBG] pgmap v4760: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:00.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:00 smithi045 conmon[39718]: cluster 2021-07-31T03:18:59.109483+0000 mgr.x ( 2021-07-31T03:19:00.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:00 smithi045 conmon[39718]: mgr.24542) 3933 : cluster [DBG] pgmap v4760: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:00.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:00 smithi045 conmon[33655]: cluster 2021-07-31T03:18:59.109483+0000 mgr.x ( 2021-07-31T03:19:00.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:00 smithi045 conmon[33655]: mgr.24542) 3933 : cluster [DBG] pgmap v4760: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:02.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:02 smithi154 conmon[34553]: cluster 2021-07-31T03:19:01.109840+0000 mgr.x (mgr.24542) 3934 : cluster [DBG] pgmap v4761: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:02.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:02 smithi045 conmon[39718]: cluster 2021-07-31T03:19:01.109840+0000 mgr.x (mgr. 2021-07-31T03:19:02.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:02 smithi045 conmon[39718]: 24542) 3934 : cluster [DBG] pgmap v4761: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:02.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:02 smithi045 conmon[33655]: cluster 2021-07-31T03:19:01.109840+0000 mgr.x (mgr.24542) 3934 : cluster 2021-07-31T03:19:02.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:02 smithi045 conmon[33655]: [DBG] pgmap v4761: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:04.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:04 smithi154 conmon[34553]: cluster 2021-07-31T03:19:03.110285+0000 mgr.x (mgr.24542) 3935 : cluster [DBG] pgmap v4762: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:04.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:04 smithi045 conmon[39718]: cluster 2021-07-31T03:19:03.110285+0000 mgr.x (mgr.24542) 3935 : cluster 2021-07-31T03:19:04.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:04 smithi045 conmon[39718]: [DBG] pgmap v4762: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:04.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:04 smithi045 conmon[33655]: cluster 2021-07-31T03:19:03.110285+0000 mgr.x (mgr. 2021-07-31T03:19:04.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:04 smithi045 conmon[33655]: 24542) 3935 : cluster [DBG] pgmap v4762: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:06.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:06 smithi154 conmon[34553]: cluster 2021-07-31T03:19:05.110774+0000 mgr.x (mgr.24542) 3936 : cluster [DBG] pgmap v4763: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:06.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:06 smithi154 conmon[34553]: 2021-07-31T03:19:06.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:06 smithi045 conmon[39718]: cluster 2021-07-31T03:19:05 2021-07-31T03:19:06.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:06 smithi045 conmon[39718]: .110774+0000 mgr.x (mgr.24542) 3936 : cluster [DBG] pgmap v4763: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:06.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:06 smithi045 conmon[33655]: cluster 2021-07-31T03:19:05.110774+0000 mgr.x (mgr.24542) 2021-07-31T03:19:06.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:06 smithi045 conmon[33655]: 3936 : cluster [DBG] pgmap v4763: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:08.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:08 smithi154 conmon[34553]: cluster 2021-07-31T03:19:07.111170+0000 mgr.x (mgr.24542) 3937 : cluster [DBG] pgmap v4764: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:08.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:08 smithi045 conmon[33655]: cluster 2021-07-31T03:19:07.111170 2021-07-31T03:19:08.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:08 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 3937 : cluster [DBG] pgmap v4764: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:08.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:08 smithi045 conmon[39718]: cluster 2021-07-31T03:19:07.111170+0000 mgr.x ( 2021-07-31T03:19:08.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:08 smithi045 conmon[39718]: mgr.24542) 3937 : cluster [DBG] pgmap v4764: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:10.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:10 smithi154 conmon[34553]: cluster 2021-07-31T03:19:09.111607+0000 mgr.x (mgr.24542) 3938 : cluster [DBG] pgmap v4765: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:10.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:10 smithi045 conmon[39718]: cluster 2021-07-31T03:19:09.111607+0000 mgr.x (mgr 2021-07-31T03:19:10.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:10 smithi045 conmon[39718]: .24542) 3938 : cluster [DBG] pgmap v4765: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:10.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:10 smithi045 conmon[33655]: cluster 2021-07-31T03:19:09.111607+0000 mgr.x (mgr.24542) 3938 : 2021-07-31T03:19:10.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:10 smithi045 conmon[33655]: cluster [DBG] pgmap v4765: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:12.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:12 smithi154 conmon[34553]: cluster 2021-07-31T03:19:11.112171+0000 mgr.x (mgr.24542) 3939 : cluster [DBG] pgmap v4766: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:12.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:12 smithi045 conmon[33655]: cluster 2021-07-31T03:19:11.112171+0000 mgr.x ( 2021-07-31T03:19:12.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:12 smithi045 conmon[33655]: mgr.24542) 3939 : cluster [DBG] pgmap v4766: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:12.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:12 smithi045 conmon[39718]: cluster 2021-07-31T03:19:11.112171+0000 mgr.x (mgr.24542) 3939 : cluster [DBG] pgmap v4766: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:12.809 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:19:13.085 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:19:13.085 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 94m ago 94m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:19:13.085 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 91m ago 94m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:19:13.086 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 91m ago 98m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:19:13.086 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 94m ago 100m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:19:13.086 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 94m ago 100m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:19:13.086 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 91m ago 98m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:19:13.087 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 94m ago 98m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:19:13.087 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 94m ago 95m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:19:13.087 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 91m ago 95m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:19:13.087 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 94m ago 97m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:19:13.088 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 94m ago 97m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:19:13.088 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 94m ago 97m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:19:13.088 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 94m ago 96m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:19:13.088 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 91m ago 96m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:19:13.088 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 91m ago 96m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:19:13.089 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 91m ago 95m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:19:13.089 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 91m ago 95m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:19:13.089 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 91m ago 95m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:19:13.425 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:19:13.425 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:19:13.426 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:19:13.426 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:19:13.426 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:19:13.426 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:19:13.427 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:19:13.427 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:19:13.427 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:19:13.427 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:19:13.428 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:19:13.428 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:19:13.428 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:19:13.428 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:19:13.428 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:19:13.429 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:19:13.429 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:19:13.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:13 smithi154 conmon[34553]: audit 2021-07-31T03:19:12.798375+0000 mgr.x (mgr.24542) 3940 : audit [DBG] from='client.19254 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:19:13.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:13 smithi045 conmon[33655]: audit 2021 2021-07-31T03:19:13.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:13 smithi045 conmon[33655]: -07-31T03:19:12.798375+0000 mgr.x (mgr.24542) 3940 : audit [DBG] from='client.19254 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:19:13.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:13 smithi045 conmon[39718]: audit 2021-07-31T03 2021-07-31T03:19:13.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:13 smithi045 conmon[39718]: :19:12.798375+0000 mgr.x (mgr.24542) 3940 : audit [DBG] from='client.19254 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:19:14.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:14 smithi154 conmon[34553]: audit 2021-07-31T03:19:13.078236+0000 mgr.x ( 2021-07-31T03:19:14.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:14 smithi154 conmon[34553]: mgr.24542) 3941 : audit [DBG] from='client.29200 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:19:14.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:14 smithi154 conmon[34553]: cluster 2021-07-31T 2021-07-31T03:19:14.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:14 smithi154 conmon[34553]: 03:19:13.112729+0000 mgr.x (mgr.24542) 3942 : cluster [DBG] pgmap v4767: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:14.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:14 smithi154 conmon[34553]: audit 2021-07-31T03:19:14.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:14 smithi154 conmon[34553]: 2021-07-31T03 2021-07-31T03:19:14.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:14 smithi154 conmon[34553]: :19:13.426050+0000 mon.a (mon. 2021-07-31T03:19:14.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:14 smithi154 conmon[34553]: 0) 3284 : audit [DBG] from='client.? 172.21.15.45:0/2519356424' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:19:14.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:14 smithi154 conmon[34553]: 2021-07-31T03:19:14.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:14 smithi045 conmon[39718]: audit 2021-07-31T03:19:13. 2021-07-31T03:19:14.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:14 smithi045 conmon[39718]: 078236+0000 mgr.x (mgr.24542) 3941 : audit [DBG] from='client.29200 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:19:14.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:14 smithi045 conmon[39718]: cluster 2021- 2021-07-31T03:19:14.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:14 smithi045 conmon[39718]: 07-31T03:19:13.112729+0000 mgr.x (mgr.24542) 3942 : cluster 2021-07-31T03:19:14.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:14 smithi045 conmon[39718]: [DBG] pgmap v4767: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:14.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:14 smithi045 conmon[39718]: audit 2021-07-31T03:19:14.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:14 smithi045 conmon[39718]: 2021-07 2021-07-31T03:19:14.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:14 smithi045 conmon[39718]: -31T03:19:13.426050 2021-07-31T03:19:14.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:14 smithi045 conmon[39718]: +0000 mon.a (mon.0) 3284 : audit 2021-07-31T03:19:14.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:14 smithi045 conmon[39718]: [DBG] from='client.? 172.21.15.45:0/2519356424' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:19:14.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:14 smithi045 conmon[33655]: audit 2021-07-31T03:19:13.078236+0000 2021-07-31T03:19:14.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:14 smithi045 conmon[33655]: mgr.x (mgr.24542) 3941 : audit [DBG] from='client.29200 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:19:14.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:14 smithi045 conmon[33655]: cluster 2021-07-31T03:19:14.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:14 smithi045 conmon[33655]: 2021-07-31T03:19:13. 2021-07-31T03:19:14.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:14 smithi045 conmon[33655]: 112729+0000 mgr.x (mgr.24542) 3942 : cluster [DBG] pgmap v4767: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:14.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:14 smithi045 conmon[33655]: audit 2021-07 2021-07-31T03:19:14.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:14 smithi045 conmon[33655]: -31T03:19:13.426050+0000 mon.a (mon.0) 3284 : audit [DBG] 2021-07-31T03:19:14.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:14 smithi045 conmon[33655]: from='client.? 172.21.15.45:0/2519356424' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:19:16.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:16 smithi154 conmon[34553]: cluster 2021-07-31T03:19:15.113224+0000 mgr.x (mgr.24542) 3943 : cluster [DBG] pgmap v4768: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:16.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:16 smithi154 conmon[34553]: 2021-07-31T03:19:16.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:16 smithi045 conmon[39718]: cluster 2021-07-31T03:19:15.113224+0000 mgr.x (mgr 2021-07-31T03:19:16.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:16 smithi045 conmon[39718]: .24542) 3943 : cluster [DBG] pgmap v4768: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:16.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:16 smithi045 conmon[33655]: cluster 2021-07-31T03:19:15.113224+0000 mgr.x (mgr.24542) 3943 : cluster [DBG] 2021-07-31T03:19:16.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:16 smithi045 conmon[33655]: pgmap v4768: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:18.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:18 smithi154 conmon[34553]: cluster 2021-07-31T03:19:17.113726+0000 mgr.x (mgr.24542) 3944 : cluster [DBG] pgmap v4769: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:18.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:18 smithi045 conmon[33655]: cluster 2021-07-31T03:19:17.113726+0000 mgr.x (mgr.24542) 2021-07-31T03:19:18.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:18 smithi045 conmon[33655]: 3944 : cluster [DBG] pgmap v4769: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:18.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:18 smithi045 conmon[39718]: cluster 2021-07-31T03:19:17.113726+0000 mgr.x 2021-07-31T03:19:18.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:18 smithi045 conmon[39718]: (mgr.24542) 3944 : cluster [DBG] pgmap v4769: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:20.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:20 smithi154 conmon[34553]: cluster 2021-07-31T03:19:19.114319+0000 mgr.x (mgr.24542) 3945 : cluster [DBG] pgmap v4770: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:20.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:20 smithi045 conmon[39718]: cluster 2021-07-31T03:19:19.114319+0000 mgr.x (mgr.24542) 3945 : cluster [DBG] pgmap v4770: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:20.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:20 smithi045 conmon[33655]: cluster 2021-07-31T03:19:19.114319+0000 mgr.x (mgr.24542) 3945 : cluster [DBG] pgmap v4770: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:22.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:22 smithi154 conmon[34553]: cluster 2021-07-31T03:19:21.114754+0000 mgr.x (mgr.24542) 3946 : cluster [DBG] pgmap v4771: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:22.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:22 smithi154 conmon[34553]: 2021-07-31T03:19:22.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:22 smithi045 conmon[39718]: cluster 2021-07-31T03:19: 2021-07-31T03:19:22.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:22 smithi045 conmon[39718]: 21.114754+0000 mgr.x (mgr.24542) 3946 : cluster [DBG] pgmap v4771: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:22.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:22 smithi045 conmon[33655]: cluster 2021-07-31T03:19:21.114754+0000 mgr.x (mgr.24542) 2021-07-31T03:19:22.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:22 smithi045 conmon[33655]: 3946 : cluster [DBG] pgmap v4771: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:24.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:24 smithi154 conmon[34553]: cluster 2021-07-31T03:19:23.115153+0000 mgr.x (mgr.24542) 3947 : cluster [DBG] pgmap v4772: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:24.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:24 smithi045 conmon[33655]: cluster 2021-07-31T03:19:23.115153+0000 mgr.x (mgr.24542) 3947 2021-07-31T03:19:24.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:24 smithi045 conmon[33655]: : cluster [DBG] pgmap v4772: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:24.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:24 smithi045 conmon[39718]: cluster 2021-07-31T03:19:23.115153 2021-07-31T03:19:24.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:24 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 3947 : cluster [DBG] pgmap v4772: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:26.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:26 smithi154 conmon[34553]: cluster 2021-07-31T03:19:25.115602+0000 mgr.x (mgr.24542) 3948 : cluster [DBG] pgmap v4773: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:26.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:26 smithi045 conmon[39718]: cluster 2021-07-31T03:19:25.115602+0000 mgr.x ( 2021-07-31T03:19:26.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:26 smithi045 conmon[39718]: mgr.24542) 3948 : cluster [DBG] pgmap v4773: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:26.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:26 smithi045 conmon[33655]: cluster 2021-07-31T03:19:25.115602+0000 mgr.x (mgr.24542) 3948 2021-07-31T03:19:26.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:26 smithi045 conmon[33655]: : cluster [DBG] pgmap v4773: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:28.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:28 smithi154 conmon[34553]: cluster 2021-07-31T03:19:27.116025+0000 mgr.x (mgr.24542) 3949 : cluster [DBG] pgmap v4774: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:28.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:28 smithi154 conmon[34553]: 2021-07-31T03:19:28.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:28 smithi154 conmon[34553]: audit 2021-07-31T03:19:28.300864+0000 mon.b (mon.2) 535 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:19:28.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:28 smithi154 conmon[34553]: audit 2021-07-31T03:19:28.301167+0000 mon.b (mon.2) 536 : audit 2021-07-31T03:19:28.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:28 smithi154 conmon[34553]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:19:28.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:28 smithi154 conmon[34553]: audit 2021-07-31T03:19:28. 2021-07-31T03:19:28.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:28 smithi154 conmon[34553]: 301540+0000 mon.a (mon.0) 3285 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:19:28.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:28 smithi154 conmon[34553]: audit 2021- 2021-07-31T03:19:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:28 smithi154 conmon[34553]: 07-31T03:19:28.301856+0000 mon.a 2021-07-31T03:19:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:28 smithi154 conmon[34553]: (mon.0) 3286 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:19:28.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:28 smithi045 conmon[33655]: cluster 2021-07-31T03:19:27.116025+0000 mgr.x (mgr.24542) 2021-07-31T03:19:28.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:28 smithi045 conmon[33655]: 3949 : cluster [DBG] pgmap v4774: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:28.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:28 smithi045 conmon[33655]: audit 2021-07-31T03:19:28.300864+0000 mon.b (mon.2) 535 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:19:28.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:28 smithi045 conmon[33655]: audit 2021-07-31T03: 2021-07-31T03:19:28.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:28 smithi045 conmon[33655]: 19:28.301167+0000 mon.b (mon.2) 536 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:19:28.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:28 smithi045 conmon[33655]: audit 2021-07-31T03: 2021-07-31T03:19:28.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:28 smithi045 conmon[33655]: 19:28.301540+0000 mon.a (mon.0 2021-07-31T03:19:28.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:28 smithi045 conmon[33655]: ) 3285 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:19:28.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:28 smithi045 conmon[33655]: audit 2021-07-31T03:19:28.301856+0000 mon.a (mon.0) 3286 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:19:28.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:28 smithi045 conmon[39718]: cluster 2021-07-31T03:19:27.116025+0000 mgr.x ( 2021-07-31T03:19:28.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:28 smithi045 conmon[39718]: mgr.24542) 3949 : cluster [DBG] pgmap v4774: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:28.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:28 smithi045 conmon[39718]: audit 2021-07-31T03:19:28.300864+0000 2021-07-31T03:19:28.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:28 smithi045 conmon[39718]: mon.b (mon.2) 535 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:19:28.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:28 smithi045 conmon[39718]: audit 2021-07-31T03:19:28.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:28 smithi045 conmon[39718]: 2021-07-31T03:19:28.301167+0000 mon.b (mon.2) 536 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:19:28.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:28 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T03:19:28.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:28 smithi045 conmon[39718]: 31T03:19:28.301540+0000 mon.a (mon.0) 3285 : audit 2021-07-31T03:19:28.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:28 smithi045 conmon[39718]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:19:28.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:28 smithi045 conmon[39718]: audit 2021-07-31T03:19:28.301856+0000 mon.a (mon.0) 3286 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:19:30.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:30 smithi154 conmon[34553]: cluster 2021-07-31T03:19:29.116456+0000 mgr.x (mgr.24542) 3950 : cluster [DBG] pgmap v4775: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:30.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:30 smithi154 conmon[34553]: 2021-07-31T03:19:30.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:30 smithi045 conmon[33655]: cluster 2021-07-31T03:19:29.116456+0000 mgr.x (mgr.24542) 3950 : cluster [DBG] pgmap v4775: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:30.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:30 smithi045 conmon[39718]: cluster 2021-07-31T03:19:29.116456+0000 mgr.x (mgr.24542) 3950 : cluster [DBG] pgmap v4775: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:32.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[34553]: cluster 2021-07-31T03:19:31.116847+0000 mgr.x (mgr.24542) 3951 : cluster [DBG] pgmap v4776: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:32.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[34553]: 2021-07-31T03:19:32.753 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: debug 2021-07-31T03:19:32.545+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:19:32.753 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:19:32.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:19:32.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: yield 2021-07-31T03:19:32.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:19:32.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:19:32.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:19:32.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:19:32.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:19:32.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:19:32.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:19:32.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:19:32.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:19:32.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: task.result() 2021-07-31T03:19:32.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:19:32.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: debug 2021-07-31T03:19:32.556+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:19:32.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:19:32.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:19:32.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: yield 2021-07-31T03:19:32.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:19:32.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:19:32.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:19:32.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:19:32.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:19:32.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:19:32.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:19:32.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:19:32.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:19:32.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: task.result() 2021-07-31T03:19:32.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:19:32.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: debug 2021-07-31T03:19:32.582+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:19:32.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:19:32.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:19:32.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: yield 2021-07-31T03:19:32.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:19:32.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:19:32.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:19:32.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:19:32.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:19:32.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:19:32.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:19:32.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:19:32.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:19:32.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: task.result() 2021-07-31T03:19:32.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:19:32.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: debug 2021-07-31T03:19:32.582+0000 7fcb9215e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:19:32.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:19:32.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:19:32.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: yield 2021-07-31T03:19:32.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:19:32.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:19:32.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:19:32.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:19:32.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:19:32.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:19:32.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:19:32.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:19:32.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:19:32.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: task.result() 2021-07-31T03:19:32.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:19:32.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: 2021-07-31T03:19:32.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:19:32.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: 2021-07-31T03:19:32.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:19:32.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:19:32.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:19:32.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:19:32.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:19:32.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:19:32.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:19:32.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:19:32.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:19:32.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:19:32.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:19:32.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:19:32.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:19:32.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:19:32.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:19:32.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:19:32.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:19:32.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:19:32.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: raise 2021-07-31T03:19:32.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:19:32.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:19:32.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:19:32.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:19:32.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:19:32.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: debug 2021-07-31T03:19:32.590+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:19:32.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:19:32.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:19:32.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: yield 2021-07-31T03:19:32.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:19:32.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:19:32.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:19:32.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:19:32.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:19:32.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:19:32.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:19:32.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:19:32.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:19:32.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: task.result() 2021-07-31T03:19:32.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:19:32.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: debug 2021-07-31T03:19:32.591+0000 7fcb9195d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:19:32.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:19:32.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:19:32.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: yield 2021-07-31T03:19:32.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:19:32.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:19:32.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:19:32.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:19:32.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:19:32.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:19:32.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:19:32.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:19:32.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:19:32.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: task.result() 2021-07-31T03:19:32.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:19:32.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: 2021-07-31T03:19:32.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:19:32.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: 2021-07-31T03:19:32.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:19:32.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:19:32.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:19:32.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:19:32.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:19:32.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:19:32.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:19:32.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:19:32.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:19:32.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:19:32.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:19:32.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:19:32.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:19:32.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:19:32.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:19:32.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:19:32.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:19:32.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:19:32.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: raise 2021-07-31T03:19:32.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:19:32.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:19:32.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:19:32.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:19:32.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:19:32 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:19:32.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:32 smithi045 conmon[33655]: cluster 2021-07-31T03:19:31.116847+0000 mgr.x (mgr.24542) 3951 : cluster 2021-07-31T03:19:32.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:32 smithi045 conmon[33655]: [DBG] pgmap v4776: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:32.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:32 smithi045 conmon[39718]: cluster 2021-07-31T03:19:31.116847+0000 mgr.x ( 2021-07-31T03:19:32.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:32 smithi045 conmon[39718]: mgr.24542) 3951 : cluster [DBG] pgmap v4776: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:33.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: audit 2021-07-31T03:19:32.501373+0000 mon.b (mon.2) 537 : audit [DBG] 2021-07-31T03:19:33.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:19:33.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: audit 2021-07-31T03:19:32.502833+0000 mon.b (mon.2) 538 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:19:33.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: audit 2021-07-31T03:19:32.503667+0000 mon.b 2021-07-31T03:19:33.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: (mon.2) 539 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:19:33.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: cephadm 2021-07-31 2021-07-31T03:19:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: T03:19:32.504449+0000 mgr.x (mgr.24542) 3952 : cephadm [WRN] 2021-07-31T03:19:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:19:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: cephadm 2021-07-31T 2021-07-31T03:19:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: 03:19:32.547019+0000 mgr.x (mgr. 2021-07-31T03:19:33.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: 24542) 3953 : cephadm [ERR] 2021-07-31T03:19:33.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:19:33.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:19:33.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: yield 2021-07-31T03:19:33.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:19:33.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:19:33.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:19:33.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:19:33.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:19:33.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:19:33.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:19:33.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:19:33.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:19:33.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: task.result() 2021-07-31T03:19:33.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:19:33.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:19:32.547421+0000 mgr.x (mgr.24542) 3954 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:19:33.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:19:32.558570+0000 mgr.x (mgr.24542 2021-07-31T03:19:33.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: ) 3955 : cephadm [ERR] 2021-07-31T03:19:33.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:19:33.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:19:33.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: yield 2021-07-31T03:19:33.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:19:33.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:19:33.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:19:33.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:19:33.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:19:33.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:19:33.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:19:33.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:19:33.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:19:33.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: task.result() 2021-07-31T03:19:33.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:19:33.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:19:32.559406+0000 mgr.x (mgr.24542) 3956 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:19:33.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:19:32.584189+0000 mgr.x (mgr.24542) 3957 : cephadm [ERR] 2021-07-31T03:19:33.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:19:33.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:19:33.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: yield 2021-07-31T03:19:33.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:19:33.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:19:33.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:19:33.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:19:33.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:19:33.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:19:33.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:19:33.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:19:33.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:19:33.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: task.result() 2021-07-31T03:19:33.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:19:33.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:19:32.584651+0000 mgr.x (mgr. 2021-07-31T03:19:33.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: 24542) 3958 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:19:33.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:19:33.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:19:33.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: yield 2021-07-31T03:19:33.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:19:33.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:19:33.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:19:33.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:19:33.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:19:33.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:19:33.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:19:33.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:19:33.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:19:33.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: task.result() 2021-07-31T03:19:33.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:19:33.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: 2021-07-31T03:19:33.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:19:33.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: 2021-07-31T03:19:33.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:19:33.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:19:33.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:19:33.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:19:33.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:19:33.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:19:33.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:19:33.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:19:33.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:19:33.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:19:33.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:19:33.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:19:33.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:19:33.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:19:33.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:19:33.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:19:33.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:19:33.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:19:33.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: raise 2021-07-31T03:19:33.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:19:33.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:19:33.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:19:33.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:19:33.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:19:33.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:19:32.592899+0000 mgr.x (mgr.24542) 3959 : cephadm [ERR] 2021-07-31T03:19:33.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:19:33.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:19:33.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: yield 2021-07-31T03:19:33.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:19:33.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:19:33.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:19:33.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:19:33.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:19:33.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:19:33.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:19:33.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:19:33.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:19:33.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: task.result() 2021-07-31T03:19:33.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:19:33.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: cephadm 2021-07-31T03: 2021-07-31T03:19:33.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: 19:32.593328+0000 mgr.x (mgr.24542) 3960 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:19:33.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:19:33.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:19:33.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: yield 2021-07-31T03:19:33.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:19:33.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:19:33.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:19:33.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:19:33.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:19:33.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:19:33.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:19:33.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:19:33.782 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:19:33.782 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: task.result() 2021-07-31T03:19:33.782 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:19:33.782 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: 2021-07-31T03:19:33.783 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:19:33.783 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: 2021-07-31T03:19:33.783 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:19:33.783 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:19:33.783 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:19:33.784 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:19:33.784 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:19:33.784 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:19:33.784 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:19:33.785 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:19:33.785 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:19:33.785 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:19:33.785 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:19:33.785 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:19:33.786 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:19:33.786 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:19:33.786 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:19:33.786 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:19:33.786 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:19:33.787 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:19:33.787 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: raise 2021-07-31T03:19:33.787 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:19:33.787 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:19:33.788 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:19:33.788 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:19:33.788 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:33 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:19:33.789 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: audit 2021-07-31T03:19:32.501373+0000 mon.b (mon.2) 2021-07-31T03:19:33.789 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: 537 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:19:33.789 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: audit 2021-07-31T03:19:32.502833+0000 mon.b (mon.2) 538 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:19:33.789 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: audit 2021-07-31T03:19:32. 2021-07-31T03:19:33.789 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: 503667+0000 mon.b (mon.2) 539 : audit 2021-07-31T03:19:33.790 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:19:33.790 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: cephadm 2021-07-31T 2021-07-31T03:19:33.790 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: 03:19:32.504449+0000 mgr.x ( 2021-07-31T03:19:33.790 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: mgr.24542) 3952 : cephadm [WRN] 2021-07-31T03:19:33.791 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:19:33.791 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: cephadm 2021-07-31T03:19:33.791 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: 2021-07- 2021-07-31T03:19:33.791 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: 31T03:19:32. 2021-07-31T03:19:33.791 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: 547019+0000 mgr.x (mgr.24542 2021-07-31T03:19:33.792 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: ) 3953 : cephadm [ERR] 2021-07-31T03:19:33.792 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:19:33.792 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:19:33.792 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: yield 2021-07-31T03:19:33.793 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:19:33.793 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:19:33.793 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:19:33.793 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:19:33.793 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:19:33.794 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:19:33.794 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:19:33.794 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:19:33.794 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:19:33.794 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: task.result() 2021-07-31T03:19:33.795 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:19:33.795 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: 2021-07-31T03:19:33.795 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: cephadm 2021-07-31T03:19:32.547421+0000 mgr.x (mgr. 2021-07-31T03:19:33.795 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: 24542) 3954 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:19:33.796 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: cephadm 2021 2021-07-31T03:19:33.796 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: -07-31T03:19:32 2021-07-31T03:19:33.796 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: .558570+0000 mgr.x (mgr.24542 2021-07-31T03:19:33.796 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: ) 3955 : cephadm [ERR] 2021-07-31T03:19:33.797 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:19:33.797 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:19:33.797 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: yield 2021-07-31T03:19:33.797 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:19:33.798 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:19:33.798 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:19:33.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: audit 2021-07-31T03:19:32.501373 2021-07-31T03:19:33.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: +0000 mon.b (mon.2) 537 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:19:33.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: audit 2021-07-31T03 2021-07-31T03:19:33.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: :19:32.502833+0000 mon.b (mon.2) 538 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:19:33.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: 2021-07-31T03:19:33.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: audit 2021-07 2021-07-31T03:19:33.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: -31T03:19:32.503667+0000 mon.b (mon.2) 2021-07-31T03:19:33.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: 539 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:19:33.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: cephadm 2021-07- 2021-07-31T03:19:33.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: 31T03:19:32.504449+0000 mgr.x 2021-07-31T03:19:33.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: (mgr.24542) 3952 : cephadm [WRN] 2021-07-31T03:19:33.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:19:33.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: cephadm 2021-07-31T03: 2021-07-31T03:19:33.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: 19:32.547019+0000 mgr.x (mgr. 2021-07-31T03:19:33.802 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: 24542) 3953 : cephadm [ERR] 2021-07-31T03:19:33.802 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:19:33.802 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:19:33.802 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: yield 2021-07-31T03:19:33.803 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:19:33.803 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:19:33.803 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:19:33.803 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:19:33.803 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:19:33.804 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:19:33.804 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:19:33.804 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:19:33.804 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:19:33.804 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: task.result() 2021-07-31T03:19:33.805 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:19:33.805 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: cephadm 2021-07-31T03:19:32.547421+0000 mgr.x (mgr.24542) 3954 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:19:33.805 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: cephadm 2021- 2021-07-31T03:19:33.805 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: 07-31T03:19:32.558570+0000 mgr.x (mgr 2021-07-31T03:19:33.805 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: .24542) 3955 : cephadm [ERR] 2021-07-31T03:19:33.806 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:19:33.806 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:19:33.806 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: yield 2021-07-31T03:19:33.806 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:19:33.807 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:19:33.807 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:19:33.807 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:19:33.807 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:19:33.807 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:19:33.808 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:19:33.808 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:19:33.808 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:19:33.808 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: task.result() 2021-07-31T03:19:33.808 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:19:33.809 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: cephadm 2021-07-31T03:19:32.559406+0000 mgr.x (mgr.24542) 3956 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:19:33.809 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: cephadm 2021-07 2021-07-31T03:19:33.809 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: -31T03:19:32.584189+0000 mgr.x (mgr 2021-07-31T03:19:33.809 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: .24542) 3957 : cephadm [ERR] 2021-07-31T03:19:33.809 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:19:33.810 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:19:33.810 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: yield 2021-07-31T03:19:33.810 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:19:33.810 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:19:33.810 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:19:33.811 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:19:33.811 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:19:33.811 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:19:33.811 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:19:33.811 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:19:33.812 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:19:33.812 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: task.result() 2021-07-31T03:19:33.812 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:19:33.812 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: cephadm 2021-07-31T03:19:32.584651+0000 mgr.x (mgr.24542) 3958 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:19:33.812 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:19:33.813 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:19:33.813 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: yield 2021-07-31T03:19:33.813 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:19:33.813 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:19:33.813 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:19:33.814 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:19:33.814 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:19:33.814 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:19:33.814 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:19:33.815 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:19:33.815 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:19:33.815 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: task.result() 2021-07-31T03:19:33.815 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:19:33.815 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: 2021-07-31T03:19:33.816 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:19:33.816 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:19:33.816 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:19:33.817 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:19:33.817 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:19:33.817 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:19:33.817 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: task.result() 2021-07-31T03:19:33.817 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:19:33.818 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: 2021-07-31T03:19:33.818 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: cephadm 2021-07-31T03:19:32.559406+0000 mgr.x (mgr.24542) 2021-07-31T03:19:33.818 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: 3956 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:19:33.818 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: cephadm 2021 2021-07-31T03:19:33.818 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: -07-31T03:19:32. 2021-07-31T03:19:33.819 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: 584189+0000 mgr.x (mgr. 2021-07-31T03:19:33.819 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: 24542) 3957 : cephadm [ERR] 2021-07-31T03:19:33.819 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:19:33.819 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:19:33.819 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: yield 2021-07-31T03:19:33.820 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:19:33.820 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:19:33.820 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:19:33.820 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:19:33.820 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:19:33.821 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:19:33.821 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:19:33.821 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:19:33.821 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:19:33.821 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: task.result() 2021-07-31T03:19:33.822 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:19:33.822 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: 2021-07-31T03:19:33.822 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: cephadm 2021-07-31T03:19:32.584651+0000 mgr.x (mgr.24542) 3958 : cephadm [ERR] 2021-07-31T03:19:33.822 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:19:33.822 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:19:33.823 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:19:33.823 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: yield 2021-07-31T03:19:33.823 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:19:33.823 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:19:33.824 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:19:33.824 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:19:33.824 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:19:33.824 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:19:33.824 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:19:33.825 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:19:33.825 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:19:33.825 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: 2021-07-31T03:19:33.825 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:19:33.826 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:19:33.826 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:19:33.826 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:19:33.826 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:19:33.826 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:19:33.827 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:19:33.827 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:19:33.827 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:19:33.827 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:19:33.828 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:19:33.828 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:19:33.828 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:19:33.828 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:19:33.828 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:19:33.829 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:19:33.829 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:19:33.829 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:19:33.829 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: raise 2021-07-31T03:19:33.829 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:19:33.830 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:19:33.830 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:19:33.830 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:19:33.830 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:19:33.831 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: cephadm 2021-07-31T03:19:32.592899+0000 mgr.x (mgr.24542) 3959 : cephadm [ERR] 2021-07-31T03:19:33.831 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:19:33.831 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:19:33.831 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: yield 2021-07-31T03:19:33.831 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:19:33.832 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:19:33.832 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:19:33.832 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:19:33.832 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:19:33.832 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:19:33.833 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:19:33.833 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:19:33.833 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:19:33.833 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: task.result() 2021-07-31T03:19:33.833 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:19:33.834 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:19:33.834 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: task.result() 2021-07-31T03:19:33.834 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:19:33.835 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: 2021-07-31T03:19:33.835 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:19:33.835 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: 2021-07-31T03:19:33.835 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:19:33.835 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:19:33.836 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:19:33.836 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:19:33.836 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:19:33.836 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:19:33.836 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:19:33.837 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:19:33.837 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:19:33.837 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:19:33.837 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:19:33.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:19:33.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:19:33.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:19:33.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:19:33.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:19:33.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:19:33.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:19:33.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: raise 2021-07-31T03:19:33.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:19:33.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:19:33.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:19:33.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:19:33.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:19:33.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: cephadm 2021-07-31T03:19:32.592899+0000 mgr.x (mgr.24542) 3959 : cephadm [ERR] 2021-07-31T03:19:33.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:19:33.841 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:19:33.841 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: yield 2021-07-31T03:19:33.841 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:19:33.841 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:19:33.841 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:19:33.842 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:19:33.842 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:19:33.842 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:19:33.842 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:19:33.843 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: cephadm 2021-07-31T03:19:32.593328+0000 mgr.x (mgr.24542) 3960 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:19:33.843 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:19:33.843 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:19:33.843 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: yield 2021-07-31T03:19:33.843 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:19:33.844 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:19:33.844 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:19:33.844 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:19:33.844 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:19:33.845 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:19:33.845 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:19:33.845 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:19:33.845 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:19:33.846 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: task.result() 2021-07-31T03:19:33.846 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:19:33.846 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: 2021-07-31T03:19:33.846 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:19:33.847 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: 2021-07-31T03:19:33.847 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:19:33.847 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:19:33.847 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:19:33.848 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:19:33.848 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:19:33.848 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:19:33.848 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:19:33.848 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:19:33.849 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:19:33.849 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:19:33.849 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:19:33.849 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:19:33.849 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:19:33.850 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:19:33.850 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:19:33.850 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:19:33.850 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:19:33.850 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:19:33.851 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: raise 2021-07-31T03:19:33.851 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:19:33.851 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:19:33.851 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:19:33.851 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:19:33.852 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:19:33.852 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:19:33.852 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:19:33.853 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: task.result() 2021-07-31T03:19:33.853 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:19:33.853 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: cephadm 2021-07-31T03:19:32.593328+0000 mgr.x (mgr.24542) 3960 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:19:33.853 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:19:33.853 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:19:33.854 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: yield 2021-07-31T03:19:33.854 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:19:33.854 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:19:33.854 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:19:33.854 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:19:33.855 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:19:33.855 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:19:33.855 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:19:33.855 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:19:33.856 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:19:33.856 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: task.result() 2021-07-31T03:19:33.856 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:19:33.856 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: 2021-07-31T03:19:33.857 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:19:33.857 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: 2021-07-31T03:19:33.857 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:19:33.857 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:19:33.857 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:19:33.858 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:19:33.858 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:19:33.858 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:19:33.858 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:19:33.858 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:19:33.859 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:19:33.859 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:19:33.859 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:19:33.859 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:19:33.859 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:19:33.860 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:19:33.860 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:19:33.860 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:19:33.860 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:19:33.860 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:19:33.861 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: raise 2021-07-31T03:19:33.861 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:19:33.861 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:19:33.861 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:19:33.861 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:19:33.862 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:33 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:19:34.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:34 smithi154 conmon[34553]: cluster 2021-07-31T03:19:33.117336+0000 mgr.x (mgr.24542) 3961 : cluster [DBG] pgmap v4777: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:34.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:34 smithi045 conmon[39718]: cluster 2021-07-31T03:19:33.117336+0000 mgr.x (mgr 2021-07-31T03:19:34.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:34 smithi045 conmon[39718]: .24542) 3961 : cluster [DBG] pgmap v4777: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:34.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:34 smithi045 conmon[33655]: cluster 2021-07-31T03:19:33.117336+0000 mgr.x (mgr.24542) 3961 : cluster [DBG] pgmap v4777: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:36.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:36 smithi154 conmon[34553]: cluster 2021-07-31T03:19:35.117982+0000 mgr.x (mgr.24542) 3962 : cluster [DBG] pgmap v4778: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:36.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:36 smithi045 conmon[33655]: cluster 2021-07-31T03:19:35.117982+0000 2021-07-31T03:19:36.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:36 smithi045 conmon[33655]: mgr.x (mgr.24542) 3962 : cluster [DBG] pgmap v4778: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:36.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:36 smithi045 conmon[39718]: cluster 2021-07-31T03:19:35.117982+0000 mgr.x (mgr.24542) 3962 : cluster [DBG] pgmap v4778: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:38.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:38 smithi154 conmon[34553]: cluster 2021-07-31T03:19:37.118510+0000 mgr.x (mgr.24542) 3963 : cluster [DBG] pgmap v4779: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:38.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:38 smithi045 conmon[33655]: cluster 2021-07-31T03:19:37.118510+0000 mgr.x (mgr.24542 2021-07-31T03:19:38.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:38 smithi045 conmon[33655]: ) 3963 : cluster [DBG] pgmap v4779: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:38.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:38 smithi045 conmon[39718]: cluster 2021-07-31T03:19:37.118510+0000 2021-07-31T03:19:38.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:38 smithi045 conmon[39718]: mgr.x (mgr.24542) 3963 : cluster [DBG] pgmap v4779: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:40.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:40 smithi154 conmon[34553]: cluster 2021-07-31T03:19:39.118973+0000 mgr.x (mgr.24542) 3964 : cluster [DBG] pgmap v4780: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:40.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:40 smithi045 conmon[33655]: cluster 2021-07-31T03:19:39.118973+0000 mgr.x (mgr.24542) 3964 2021-07-31T03:19:40.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:40 smithi045 conmon[33655]: : cluster [DBG] pgmap v4780: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:40.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:40 smithi045 conmon[39718]: cluster 2021-07-31T03:19:39.118973+0000 mgr.x (mgr.24542) 3964 : cluster [DBG] pgmap v4780: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:42.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:42 smithi154 conmon[34553]: cluster 2021-07-31T03:19:41.119397+0000 mgr.x (mgr.24542) 3965 : cluster [DBG] pgmap v4781: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:42.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:42 smithi045 conmon[33655]: cluster 2021-07-31T03:19:41.119397+0000 mgr.x (mgr.24542) 3965 : cluster 2021-07-31T03:19:42.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:42 smithi045 conmon[33655]: [DBG] pgmap v4781: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:42.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:42 smithi045 conmon[39718]: cluster 2021-07-31T03:19:41.119397+0000 mgr.x 2021-07-31T03:19:42.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:42 smithi045 conmon[39718]: (mgr.24542) 3965 : cluster [DBG] pgmap v4781: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:43.741 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:19:44.016 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:19:44.016 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 94m ago 95m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:19:44.016 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 92m ago 95m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:19:44.016 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 92m ago 98m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:19:44.017 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 94m ago 100m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:19:44.017 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 94m ago 100m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:19:44.017 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 92m ago 98m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:19:44.017 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 94m ago 99m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:19:44.018 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 94m ago 95m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:19:44.018 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 92m ago 95m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:19:44.018 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 94m ago 98m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:19:44.018 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 94m ago 98m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:19:44.019 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 94m ago 97m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:19:44.019 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 94m ago 97m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:19:44.019 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 92m ago 97m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:19:44.019 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 92m ago 96m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:19:44.020 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 92m ago 96m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:19:44.020 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 92m ago 96m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:19:44.020 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 92m ago 95m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:19:44.351 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:19:44.351 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:19:44.352 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:19:44.352 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:19:44.352 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:19:44.353 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:19:44.353 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:19:44.353 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:19:44.354 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:19:44.354 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:19:44.354 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:19:44.355 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:19:44.355 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:19:44.355 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:19:44.356 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:19:44.356 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:19:44.356 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:19:44.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:44 smithi154 conmon[34553]: cluster 2021-07-31T03:19:43.119945+0000 mgr.x (mgr.24542) 3966 : cluster [DBG] pgmap v4782: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:44.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:44 smithi154 conmon[34553]: audit 2021-07-31T03:19:43 2021-07-31T03:19:44.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:44 smithi154 conmon[34553]: .729867+0000 mgr.x (mgr.24542) 3967 : audit [DBG] from='client.19272 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:19:44.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:44 smithi154 conmon[34553]: audit 2021-07-31T03:19:44.351525+0000 mon.c (mon.1) 195 : audit [DBG] from='client.? 172.21.15.45:0/215598631' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:19:44.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:44 smithi045 conmon[33655]: cluster 2021-07-31T03:19:43.119945+0000 mgr.x (mgr.24542) 3966 : cluster [DBG] pgmap v4782: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:44.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:44 smithi045 conmon[33655]: audit 2021-07-31T03:19:43.729867+0000 mgr.x (mgr.24542) 3967 : audit [DBG] from='client.19272 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:19:44.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:44 smithi045 conmon[33655]: audit 2021-07-31T03:19:44.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:44 smithi045 conmon[33655]: 2021-07-31T03:19:44.351525+0000 mon.c (mon.1) 195 : audit [DBG] 2021-07-31T03:19:44.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:44 smithi045 conmon[33655]: from='client.? 172.21.15.45:0/215598631' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:19:44.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:44 smithi045 conmon[39718]: cluster 2021-07-31T03:19:43. 2021-07-31T03:19:44.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:44 smithi045 conmon[39718]: 119945+0000 mgr.x (mgr.24542) 3966 : cluster [DBG] pgmap v4782: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:44.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:44 smithi045 conmon[39718]: audit 2021-07-31T03:19:44.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:44 smithi045 conmon[39718]: 2021-07-31T03:19 2021-07-31T03:19:44.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:44 smithi045 conmon[39718]: :43.729867+0000 mgr.x (mgr.24542 2021-07-31T03:19:44.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:44 smithi045 conmon[39718]: ) 3967 : audit [DBG] from='client.19272 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:19:44.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:44 smithi045 conmon[39718]: audit 2021 2021-07-31T03:19:44.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:44 smithi045 conmon[39718]: -07-31T03:19:44.351525+0000 2021-07-31T03:19:44.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:44 smithi045 conmon[39718]: mon.c (mon.1) 195 : audit [DBG] from='client.? 172.21.15.45:0/215598631' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:19:45.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:45 smithi154 conmon[34553]: audit 2021-07-31T03:19:44.008911+0000 mgr.x (mgr.24542) 3968 : audit [DBG] from='client.19278 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:19:45.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:45 smithi045 conmon[33655]: audit 2021-07-31T03:19:44.008911+0000 2021-07-31T03:19:45.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:45 smithi045 conmon[33655]: mgr.x (mgr.24542) 3968 : audit [DBG] from='client.19278 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:19:45.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:45 smithi045 conmon[39718]: audit 2021-07-31T03:19:44.008911+0000 2021-07-31T03:19:45.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:45 smithi045 conmon[39718]: mgr.x (mgr.24542) 3968 : audit [DBG] from='client.19278 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:19:46.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:46 smithi154 conmon[34553]: cluster 2021-07-31T03:19:45.120487+0000 mgr.x (mgr.24542) 3969 : cluster [DBG] pgmap v4783: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:46.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:46 smithi045 conmon[33655]: cluster 2021-07-31T03:19:45.120487+0000 mgr.x (mgr 2021-07-31T03:19:46.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:46 smithi045 conmon[33655]: .24542) 3969 : cluster [DBG] pgmap v4783: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:46.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:46 smithi045 conmon[39718]: cluster 2021-07-31T03:19:45.120487 2021-07-31T03:19:46.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:46 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 3969 : cluster [DBG] pgmap v4783: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:48.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:48 smithi154 conmon[34553]: cluster 2021-07-31T03:19:47.120865+0000 mgr.x (mgr.24542) 3970 : cluster [DBG] pgmap v4784: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:48.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:48 smithi045 conmon[33655]: cluster 2021-07-31T03:19:47.120865+0000 mgr.x (mgr 2021-07-31T03:19:48.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:48 smithi045 conmon[33655]: .24542) 3970 : cluster [DBG] pgmap v4784: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:48.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:48 smithi045 conmon[39718]: cluster 2021-07-31T03:19:47.120865+0000 2021-07-31T03:19:48.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:48 smithi045 conmon[39718]: mgr.x (mgr.24542) 3970 : cluster [DBG] pgmap v4784: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:50.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:50 smithi154 conmon[34553]: cluster 2021-07-31T03:19:49.121339+0000 mgr.x (mgr.24542) 3971 : cluster [DBG] pgmap v4785: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:50.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:50 smithi045 conmon[33655]: cluster 2021-07-31T03:19:49.121339+0000 2021-07-31T03:19:50.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:50 smithi045 conmon[33655]: mgr.x (mgr.24542) 3971 : cluster [DBG] pgmap v4785: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:50.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:50 smithi045 conmon[39718]: cluster 2021-07-31T03:19:49.121339+0000 mgr.x ( 2021-07-31T03:19:50.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:50 smithi045 conmon[39718]: mgr.24542) 3971 : cluster [DBG] pgmap v4785: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:52.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:52 smithi154 conmon[34553]: cluster 2021-07-31T03:19:51.121764+0000 mgr.x (mgr.24542) 3972 : cluster [DBG] pgmap v4786: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:52.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:52 smithi045 conmon[33655]: cluster 2021-07-31T03:19:51.121764+0000 mgr.x (mgr.24542) 3972 : cluster [DBG] 2021-07-31T03:19:52.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:52 smithi045 conmon[33655]: pgmap v4786: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:52.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:52 smithi045 conmon[39718]: cluster 2021-07-31T03:19:51.121764+0000 mgr.x (mgr.24542) 3972 : cluster [DBG] pgmap v4786: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:54.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:54 smithi154 conmon[34553]: cluster 2021-07-31T03:19:53.122309+0000 mgr.x (mgr.24542) 3973 : cluster [DBG] pgmap v4787: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:54.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:54 smithi045 conmon[33655]: cluster 2021-07-31T03:19:53.122309+0000 mgr.x (mgr.24542) 3973 : 2021-07-31T03:19:54.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:54 smithi045 conmon[33655]: cluster [DBG] pgmap v4787: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:54.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:54 smithi045 conmon[39718]: cluster 2021-07-31T03:19:53.122309 2021-07-31T03:19:54.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:54 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 3973 : cluster [DBG] pgmap v4787: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:56.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:56 smithi154 conmon[34553]: cluster 2021-07-31T03:19:55.122784+0000 mgr.x (mgr.24542) 3974 : cluster [DBG] pgmap v4788: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:56.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:56 smithi045 conmon[39718]: cluster 2021-07-31T03:19:55.122784+0000 mgr.x (mgr.24542) 3974 : cluster 2021-07-31T03:19:56.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:56 smithi045 conmon[39718]: [DBG] pgmap v4788: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:56.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:56 smithi045 conmon[33655]: cluster 2021-07-31T03:19:55.122784+0000 mgr.x (mgr.24542) 3974 : cluster [DBG] pgmap v4788: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:58.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:19:58 smithi154 conmon[34553]: cluster 2021-07-31T03:19:57.123312+0000 mgr.x (mgr.24542) 3975 : cluster [DBG] pgmap v4789: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:58.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:58 smithi045 conmon[33655]: cluster 2021-07-31T03:19:57.123312+0000 mgr.x (mgr.24542 2021-07-31T03:19:58.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:19:58 smithi045 conmon[33655]: ) 3975 : cluster [DBG] pgmap v4789: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:19:58.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:58 smithi045 conmon[39718]: cluster 2021-07-31T03:19:57.123312+0000 mgr.x 2021-07-31T03:19:58.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:19:58 smithi045 conmon[39718]: (mgr.24542) 3975 : cluster [DBG] pgmap v4789: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:00.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:00 smithi045 conmon[39718]: cluster 2021-07-31T03:19:59.123760+0000 mgr.x (mgr.24542) 3976 : cluster [DBG] pgmap v4790: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:00.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:00 smithi045 conmon[39718]: cluster 2021-07-31T03:20:00.000109+0000 mon.a (mon.0 2021-07-31T03:20:00.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:00 smithi045 conmon[39718]: ) 3287 : cluster [INF] overall HEALTH_OK 2021-07-31T03:20:00.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:00 smithi045 conmon[33655]: cluster 2021-07-31T03:19:59.123760+0000 mgr.x (mgr.24542) 2021-07-31T03:20:00.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:00 smithi045 conmon[33655]: 3976 : cluster [DBG] pgmap v4790: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:00.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:00 smithi045 conmon[33655]: cluster 2021-07-31T03:20:00.000109+0000 2021-07-31T03:20:00.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:00 smithi045 conmon[33655]: mon.a (mon.0) 3287 : cluster [INF] overall HEALTH_OK 2021-07-31T03:20:01.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:00 smithi154 conmon[34553]: cluster 2021-07-31T03:19:59.123760+0000 mgr.x (mgr.24542) 3976 : cluster [DBG] pgmap v4790: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:01.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:00 smithi154 conmon[34553]: cluster 2021-07-31T03:20:00.000109+0000 mon.a (mon.0) 3287 : cluster [INF] overall HEALTH_OK 2021-07-31T03:20:02.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:02 smithi045 conmon[33655]: cluster 2021-07-31T03:20:01.124173+0000 mgr.x (mgr.24542) 3977 : cluster 2021-07-31T03:20:02.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:02 smithi045 conmon[33655]: [DBG] pgmap v4791: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:02.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:02 smithi045 conmon[39718]: cluster 2021-07-31T03:20:01. 2021-07-31T03:20:02.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:02 smithi045 conmon[39718]: 124173+0000 mgr.x (mgr.24542) 3977 : cluster [DBG] pgmap v4791: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:03.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:02 smithi154 conmon[34553]: cluster 2021-07-31T03:20:01.124173+0000 mgr.x (mgr.24542) 3977 : cluster [DBG] pgmap v4791: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:04.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:04 smithi045 conmon[33655]: cluster 2021-07-31T03:20:03.124642+0000 mgr.x (mgr.24542) 3978 : cluster 2021-07-31T03:20:04.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:04 smithi045 conmon[33655]: [DBG] pgmap v4792: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:04.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:04 smithi045 conmon[39718]: cluster 2021-07-31T03:20: 2021-07-31T03:20:04.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:04 smithi045 conmon[39718]: 03.124642+0000 mgr.x (mgr.24542) 3978 : cluster [DBG] pgmap v4792: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:05.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:04 smithi154 conmon[34553]: cluster 2021-07-31T03:20:03.124642+0000 mgr.x (mgr.24542) 3978 : cluster [DBG] pgmap v4792: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:06.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:06 smithi045 conmon[33655]: cluster 2021-07-31T03:20:05.125289+0000 mgr.x (mgr.24542) 3979 : cluster [DBG] 2021-07-31T03:20:06.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:06 smithi045 conmon[33655]: pgmap v4793: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:06.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:06 smithi045 conmon[39718]: cluster 2021-07-31T03:20:05.125289+0000 mgr.x ( 2021-07-31T03:20:06.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:06 smithi045 conmon[39718]: mgr.24542) 3979 : cluster [DBG] pgmap v4793: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:07.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:06 smithi154 conmon[34553]: cluster 2021-07-31T03:20:05.125289+0000 mgr.x (mgr.24542) 3979 : cluster [DBG] pgmap v4793: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:08.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:08 smithi045 conmon[39718]: cluster 2021-07-31T03:20:07.125680+0000 mgr.x (mgr.24542) 3980 : cluster [DBG] pgmap v4794: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:08.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:08 smithi045 conmon[33655]: cluster 2021-07-31T03:20:07.125680+0000 mgr.x (mgr.24542) 3980 : cluster [DBG] pgmap v4794: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:09.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:08 smithi154 conmon[34553]: cluster 2021-07-31T03:20:07.125680+0000 mgr.x (mgr.24542) 3980 : cluster [DBG] pgmap v4794: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:11.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:10 smithi154 conmon[34553]: cluster 2021-07-31T03:20:09.126172+0000 mgr.x (mgr.24542) 3981 : cluster [DBG] pgmap v4795: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:11.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:10 smithi045 conmon[39718]: cluster 2021-07-31T03:20:09.126172+0000 mgr.x (mgr.24542) 3981 : cluster [DBG] pgmap v4795: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:11.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:10 smithi045 conmon[33655]: cluster 2021-07-31T03:20:09.126172+0000 mgr.x (mgr.24542) 3981 : cluster [DBG] pgmap v4795: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:13.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:12 smithi154 conmon[34553]: cluster 2021-07-31T03:20:11.126705+0000 mgr.x (mgr.24542) 3982 : cluster [DBG] pgmap v4796: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:13.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:12 smithi045 conmon[39718]: cluster 2021-07-31T03:20:11.126705+0000 mgr.x ( 2021-07-31T03:20:13.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:12 smithi045 conmon[39718]: mgr.24542) 3982 : cluster [DBG] pgmap v4796: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:13.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:12 smithi045 conmon[33655]: cluster 2021-07-31T03:20:11.126705+0000 mgr.x (mgr.24542) 3982 : cluster [DBG] pgmap v4796: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:14.667 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:20:14.944 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:20:14.945 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 95m ago 95m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:20:14.945 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 92m ago 95m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:20:14.945 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 92m ago 99m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:20:14.946 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 95m ago 101m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:20:14.946 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 95m ago 101m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:20:14.946 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 92m ago 99m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:20:14.946 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 95m ago 99m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:20:14.947 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 95m ago 96m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:20:14.947 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 92m ago 96m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:20:14.947 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 95m ago 98m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:20:14.947 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 95m ago 98m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:20:14.948 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 95m ago 98m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:20:14.948 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 95m ago 97m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:20:14.948 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 92m ago 97m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:20:14.948 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 92m ago 97m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:20:14.948 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 92m ago 96m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:20:14.949 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 92m ago 96m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:20:14.949 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 92m ago 96m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:20:15.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:14 smithi154 conmon[34553]: cluster 2021-07-31T03:20:13.127138+0000 mgr.x (mgr.24542) 3983 : cluster [DBG] pgmap v4797: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:15.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:14 smithi154 conmon[34553]: 2021-07-31T03:20:15.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:14 smithi045 conmon[33655]: cluster 2021-07-31T03:20:13.127138+0000 mgr.x ( 2021-07-31T03:20:15.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:14 smithi045 conmon[33655]: mgr.24542) 3983 : cluster [DBG] pgmap v4797: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:15.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:14 smithi045 conmon[39718]: cluster 2021-07-31T03:20:13.127138+0000 mgr.x 2021-07-31T03:20:15.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:14 smithi045 conmon[39718]: (mgr.24542) 3983 : cluster [DBG] pgmap v4797: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:15.282 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:20:15.282 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:20:15.282 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:20:15.283 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:20:15.283 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:20:15.283 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:20:15.283 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:20:15.284 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:20:15.284 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:20:15.284 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:20:15.284 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:20:15.284 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:20:15.285 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:20:15.285 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:20:15.285 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:20:15.285 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:20:15.286 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:20:16.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:15 smithi154 conmon[34553]: audit 2021-07-31T03:20:14.655358+0000 mgr.x (mgr.24542) 3984 : audit [DBG] from='client.19290 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:20:16.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:15 smithi154 conmon[34553]: audit 2021-07-31T03:20:15.282833+0000 mon.a (mon.0) 3288 : audit [DBG] from='client.? 172.21.15.45:0/724352412' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:20:16.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:15 smithi045 conmon[33655]: audit 2021-07-31T03:20:14.655358+0000 mgr.x (mgr.24542) 3984 : audit [DBG] from='client.19290 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:20:16.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:15 smithi045 conmon[33655]: audit 2021-07-31T03:20:15.282833+0000 mon.a (mon.0) 3288 : audit [DBG] from='client.? 172.21.15.45:0/724352412' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:20:16.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:15 smithi045 conmon[39718]: audit 2021-07-31T03:20:14.655358+0000 2021-07-31T03:20:16.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:15 smithi045 conmon[39718]: mgr.x (mgr.24542) 3984 : audit [DBG] from='client.19290 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:20:16.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:15 smithi045 conmon[39718]: audit 2021-07-31T03:20: 2021-07-31T03:20:16.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:15 smithi045 conmon[39718]: 15.282833+0000 mon.a (mon.0) 3288 : audit [DBG] from='client.? 172.21.15.45:0/724352412' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:20:17.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:16 smithi154 conmon[34553]: audit 2021-07-31T03:20:14.937702+0000 mgr.x (mgr.24542) 3985 : 2021-07-31T03:20:17.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:16 smithi154 conmon[34553]: audit [DBG] from='client.19296 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:20:17.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:16 smithi154 conmon[34553]: cluster 2021-07-31T03:20:15.127576+0000 mgr.x (mgr.24542) 3986 : cluster [DBG] pgmap v4798: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:17.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:16 smithi045 conmon[33655]: audit 2021-07-31T03:20:14.937702+0000 mgr.x (mgr.24542) 2021-07-31T03:20:17.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:16 smithi045 conmon[33655]: 3985 : audit [DBG] from='client.19296 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:20:17.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:16 smithi045 conmon[33655]: cluster 2021-07-31T03:20:15.127576+0000 mgr.x (mgr.24542) 3986 : cluster [DBG] pgmap v4798: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:17.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:16 smithi045 conmon[39718]: audit 2021-07-31T03:20:14.937702+0000 mgr.x ( 2021-07-31T03:20:17.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:16 smithi045 conmon[39718]: mgr.24542) 3985 : audit [DBG] from='client.19296 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:20:17.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:16 smithi045 conmon[39718]: cluster 2021-07-31T03:20:17.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:16 smithi045 conmon[39718]: 2021-07-31T03:20:15.127576 2021-07-31T03:20:17.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:16 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 3986 : cluster 2021-07-31T03:20:17.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:16 smithi045 conmon[39718]: [DBG] pgmap v4798: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:19.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:18 smithi154 conmon[34553]: cluster 2021-07-31T03:20:17.128061+0000 mgr.x (mgr.24542) 3987 : cluster [DBG] pgmap v4799: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:19.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:18 smithi045 conmon[39718]: cluster 2021-07-31T03:20:17.128061+0000 mgr.x (mgr.24542) 3987 : cluster [DBG] pgmap v4799: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:19.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:18 smithi045 conmon[33655]: cluster 2021-07-31T03:20:17.128061+0000 mgr.x (mgr.24542 2021-07-31T03:20:19.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:18 smithi045 conmon[33655]: ) 3987 : cluster [DBG] pgmap v4799: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:21.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:20 smithi154 conmon[34553]: cluster 2021-07-31T03:20:19.128545+0000 mgr.x (mgr.24542) 3988 : cluster [DBG] pgmap v4800: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:21.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:20 smithi045 conmon[39718]: cluster 2021-07-31T03:20:19.128545+0000 mgr.x (mgr.24542) 3988 : cluster [DBG] pgmap v4800: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:21.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:20 smithi045 conmon[33655]: cluster 2021-07-31T03:20:19.128545+0000 mgr.x (mgr 2021-07-31T03:20:21.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:20 smithi045 conmon[33655]: .24542) 3988 : cluster [DBG] pgmap v4800: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:23.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:22 smithi154 conmon[34553]: cluster 2021-07-31T03:20:21.129023+0000 mgr.x (mgr.24542) 3989 : cluster [DBG] pgmap v4801: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:23.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:22 smithi045 conmon[39718]: cluster 2021-07-31T03:20:21.129023+0000 mgr.x ( 2021-07-31T03:20:23.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:22 smithi045 conmon[39718]: mgr.24542) 3989 : cluster [DBG] pgmap v4801: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:23.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:22 smithi045 conmon[33655]: cluster 2021-07-31T03:20:21.129023+0000 mgr.x (mgr.24542) 3989 : cluster [DBG] pgmap v4801: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:25.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:24 smithi154 conmon[34553]: cluster 2021-07-31T03:20:23.129553+0000 mgr.x (mgr.24542) 3990 : cluster [DBG] pgmap v4802: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:25.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:24 smithi045 conmon[33655]: cluster 2021-07-31T03:20:23.129553+0000 mgr.x (mgr.24542) 2021-07-31T03:20:25.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:24 smithi045 conmon[33655]: 3990 : cluster [DBG] pgmap v4802: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:25.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:24 smithi045 conmon[39718]: cluster 2021-07-31T03:20:23. 2021-07-31T03:20:25.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:24 smithi045 conmon[39718]: 129553+0000 mgr.x (mgr.24542) 3990 : cluster [DBG] pgmap v4802: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:27.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:26 smithi154 conmon[34553]: cluster 2021-07-31T03:20:25.130206+0000 mgr.x (mgr.24542) 3991 : cluster [DBG] pgmap v4803: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:27.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:26 smithi154 conmon[34553]: 2021-07-31T03:20:27.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:26 smithi045 conmon[39718]: cluster 2021-07-31T03:20:25. 2021-07-31T03:20:27.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:26 smithi045 conmon[39718]: 130206+0000 mgr.x (mgr.24542) 3991 : cluster [DBG] pgmap v4803: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:27.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:26 smithi045 conmon[33655]: cluster 2021-07-31T03:20:25.130206+0000 mgr.x (mgr.24542) 3991 : cluster [DBG] pgmap v4803: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:29.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:28 smithi154 conmon[34553]: cluster 2021-07-31T03:20:27.130734+0000 mgr.x (mgr.24542) 3992 : cluster [DBG] pgmap v4804: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:29.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:28 smithi154 conmon[34553]: audit 2021-07-31T03:20:28.304566+0000 mon.b (mon.2) 540 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:20:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:28 smithi154 conmon[34553]: audit 2021-07-31T03:20:28.304814+0000 mon.b (mon.2) 541 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:20:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:28 smithi154 conmon[34553]: audit 2021-07-31T03:20:28.305136+0000 mon.a (mon.0) 3289 : 2021-07-31T03:20:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:28 smithi154 conmon[34553]: audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:20:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:28 smithi154 conmon[34553]: audit 2021-07-31T03:20:28.305472+0000 mon.a (mon.0) 3290 : 2021-07-31T03:20:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:28 smithi154 conmon[34553]: audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:20:29.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:28 smithi045 conmon[39718]: cluster 2021-07-31T03:20:27.130734+0000 mgr.x ( 2021-07-31T03:20:29.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:28 smithi045 conmon[39718]: mgr.24542) 3992 : cluster [DBG] pgmap v4804: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:29.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:28 smithi045 conmon[39718]: audit 2021-07-31T03:20:28.304566+0000 mon.b ( 2021-07-31T03:20:29.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:28 smithi045 conmon[39718]: mon.2) 540 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:20:29.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:28 smithi045 conmon[39718]: audit 2021-07-31T03 2021-07-31T03:20:29.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:28 smithi045 conmon[39718]: :20:28.304814+0000 mon.b (mon. 2021-07-31T03:20:29.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:28 smithi045 conmon[39718]: 2) 541 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:20:29.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:28 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T03:20:29.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:28 smithi045 conmon[39718]: 31T03:20:28.305136+0000 mon.a (mon.0) 3289 : 2021-07-31T03:20:29.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:28 smithi045 conmon[39718]: audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:20:29.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:28 smithi045 conmon[39718]: audit 2021- 2021-07-31T03:20:29.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:28 smithi045 conmon[39718]: 07-31T03:20:28.305472+0000 mon.a (mon.0) 3290 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:20:29.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:28 smithi045 conmon[33655]: cluster 2021-07-31T03:20:27.130734+0000 mgr.x (mgr.24542) 3992 : cluster [DBG] pgmap v4804: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:29.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:28 smithi045 conmon[33655]: audit 2021-07-31T03:20:28.304566+0000 mon.b (mon.2) 540 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:20:29.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:28 smithi045 conmon[33655]: audit 2021-07-31T03:20:28.304814+0000 mon.b 2021-07-31T03:20:29.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:28 smithi045 conmon[33655]: (mon.2) 541 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:20:29.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:28 smithi045 conmon[33655]: audit 2021-07- 2021-07-31T03:20:29.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:28 smithi045 conmon[33655]: 31T03:20:28.305136+0000 mon.a ( 2021-07-31T03:20:29.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:28 smithi045 conmon[33655]: mon.0) 3289 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:20:29.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:28 smithi045 conmon[33655]: audit 2021- 2021-07-31T03:20:29.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:28 smithi045 conmon[33655]: 07-31T03:20:28.305472+0000 mon.a (mon.0) 3290 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:20:31.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:30 smithi154 conmon[34553]: cluster 2021-07-31T03:20:29.131249+0000 mgr.x (mgr.24542) 3993 : cluster [DBG] pgmap v4805: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:31.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:30 smithi045 conmon[39718]: cluster 2021-07-31T03:20:29.131249+0000 mgr.x 2021-07-31T03:20:31.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:30 smithi045 conmon[39718]: (mgr.24542) 3993 : cluster [DBG] pgmap v4805: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:31.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:30 smithi045 conmon[33655]: cluster 2021-07-31T03:20:29.131249+0000 mgr.x (mgr.24542) 2021-07-31T03:20:31.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:30 smithi045 conmon[33655]: 3993 : cluster [DBG] pgmap v4805: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:33.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[34553]: cluster 2021-07-31T03:20:31.131754+0000 mgr.x (mgr.24542) 2021-07-31T03:20:33.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[34553]: 3994 : cluster [DBG] pgmap v4806: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:33.003 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: debug 2021-07-31T03:20:32.640+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:20:33.003 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:20:33.003 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:20:33.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: yield 2021-07-31T03:20:33.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:20:33.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:20:33.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:20:33.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:20:33.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:20:33.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:20:33.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:20:33.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:20:33.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:20:33.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: task.result() 2021-07-31T03:20:33.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:20:33.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: debug 2021-07-31T03:20:32.651+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:20:33.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:20:33.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:20:33.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: yield 2021-07-31T03:20:33.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:20:33.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:20:33.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:20:33.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:20:33.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:20:33.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:20:33.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:20:33.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:20:33.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:20:33.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: task.result() 2021-07-31T03:20:33.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:20:33.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: debug 2021-07-31T03:20:32.677+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:20:33.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:20:33.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:20:33.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: yield 2021-07-31T03:20:33.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:20:33.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:20:33.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:20:33.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:20:33.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:20:33.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:20:33.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:20:33.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:20:33.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:20:33.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: task.result() 2021-07-31T03:20:33.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:20:33.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: debug 2021-07-31T03:20:32.677+0000 7fcb9095b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:20:33.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:20:33.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:20:33.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: yield 2021-07-31T03:20:33.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:20:33.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:20:33.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:20:33.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:20:33.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:20:33.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:20:33.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:20:33.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:20:33.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:20:33.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: task.result() 2021-07-31T03:20:33.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:20:33.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: 2021-07-31T03:20:33.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:20:33.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: 2021-07-31T03:20:33.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:20:33.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:20:33.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:20:33.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:20:33.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:20:33.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:20:33.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:20:33.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:20:33.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:20:33.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:20:33.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:20:33.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:20:33.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:20:33.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:20:33.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:20:33.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:20:33.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:20:33.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:20:33.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: raise 2021-07-31T03:20:33.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:20:33.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:20:33.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:20:33.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:20:33.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:20:33.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: debug 2021-07-31T03:20:32.686+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:20:33.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:20:33.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:20:33.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: yield 2021-07-31T03:20:33.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:20:33.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:20:33.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:20:33.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:20:33.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:20:33.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:20:33.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:20:33.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:20:33.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:20:33.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: task.result() 2021-07-31T03:20:33.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:20:33.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: debug 2021-07-31T03:20:32.687+0000 7fcb9115c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:20:33.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:20:33.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:20:33.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: yield 2021-07-31T03:20:33.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:20:33.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:20:33.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:20:33.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:20:33.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:20:33.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:20:33.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:20:33.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:20:33.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:20:33.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: task.result() 2021-07-31T03:20:33.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:20:33.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: 2021-07-31T03:20:33.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:20:33.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: 2021-07-31T03:20:33.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:20:33.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:20:33.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:20:33.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:20:33.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:20:33.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:20:33.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:20:33.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:20:33.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:20:33.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:20:33.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:20:33.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:20:33.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:20:33.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:20:33.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:20:33.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:20:33.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:20:33.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:20:33.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: raise 2021-07-31T03:20:33.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:20:33.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:20:33.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:20:33.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:20:33.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:20:32 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:20:33.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:32 smithi045 conmon[33655]: cluster 2021-07-31T03:20:33.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:32 smithi045 conmon[33655]: 2021-07-31T03:20:31.131754+0000 mgr.x (mgr. 2021-07-31T03:20:33.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:32 smithi045 conmon[33655]: 24542) 3994 : cluster [DBG] pgmap v4806: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:33.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:32 smithi045 conmon[33655]: 2021-07-31T03:20:33.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:32 smithi045 conmon[39718]: cluster 2021-07-31T03:20:31 2021-07-31T03:20:33.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:32 smithi045 conmon[39718]: .131754+0000 mgr.x (mgr.24542) 3994 : cluster [DBG] pgmap v4806: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:34.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: audit 2021-07-31T03:20:32.594077+0000 mon.b (mon.2) 542 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:20:34.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: 2021-07-31T03:20:34.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: audit 2021-07-31T03:20:32.595542+0000 mon.b (mon.2) 543 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:20:34.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: audit 2021-07-31 2021-07-31T03:20:34.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: T03:20:32.596490+0000 mon.b (mon.2) 544 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:20:34.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:20:32. 2021-07-31T03:20:34.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: 597248+0000 mgr.x (mgr.24542) 3995 : cephadm [WRN] 2021-07-31T03:20:34.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:20:34.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: cephadm 2021-07-31T03 2021-07-31T03:20:34.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: :20:32.641256+0000 mgr.x (mgr.24542) 2021-07-31T03:20:34.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: 3996 : cephadm [ERR] 2021-07-31T03:20:34.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:20:34.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:20:34.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: yield 2021-07-31T03:20:34.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:20:34.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:20:34.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:20:34.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:20:34.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:20:34.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:20:34.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:20:34.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:20:34.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:20:34.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: task.result() 2021-07-31T03:20:34.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:20:34.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:20:32.641880+0000 mgr.x (mgr.24542) 3997 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:20:34.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:20:32.652902+0000 mgr.x ( 2021-07-31T03:20:34.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: mgr.24542) 3998 : cephadm [ERR] 2021-07-31T03:20:34.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:20:34.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:20:34.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: yield 2021-07-31T03:20:34.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:20:34.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:20:34.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:20:34.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:20:34.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:20:34.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:20:34.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:20:34.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:20:34.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:20:34.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: task.result() 2021-07-31T03:20:34.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:20:34.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:20:32.653899+0000 mgr.x (mgr.24542) 3999 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:20:34.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:20:32.678173+0000 mgr.x (mgr.24542 2021-07-31T03:20:34.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: ) 4000 : cephadm [ERR] 2021-07-31T03:20:34.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:20:34.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:20:34.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: yield 2021-07-31T03:20:34.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:20:34.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:20:34.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:20:34.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:20:34.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:20:34.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:20:34.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:20:34.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:20:34.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:20:34.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: task.result() 2021-07-31T03:20:34.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:20:34.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:20:32.678628+0000 mgr.x (mgr.24542) 4001 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:20:34.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:20:34.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:20:34.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: yield 2021-07-31T03:20:34.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:20:34.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:20:34.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:20:34.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:20:34.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:20:34.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:20:34.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:20:34.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:20:34.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:20:34.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: task.result() 2021-07-31T03:20:34.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:20:34.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: 2021-07-31T03:20:34.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:20:34.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: 2021-07-31T03:20:34.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:20:34.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:20:34.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:20:34.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:20:34.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:20:34.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:20:34.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:20:34.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:20:34.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:20:34.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:20:34.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:20:34.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:20:34.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:20:34.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:20:34.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:20:34.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:20:34.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:20:34.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:20:34.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: raise 2021-07-31T03:20:34.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:20:34.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:20:34.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:20:34.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:20:34.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:20:34.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: cephadm 2021-07-31 2021-07-31T03:20:34.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: T03:20:32.687391+0000 mgr.x (mgr.24542) 4002 : cephadm [ERR] 2021-07-31T03:20:34.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:20:34.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:20:34.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: yield 2021-07-31T03:20:34.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:20:34.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:20:34.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:20:34.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:20:34.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:20:34.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:20:34.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:20:34.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:20:34.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:20:34.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: task.result() 2021-07-31T03:20:34.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:20:34.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:20:32.687948+0000 mgr.x (mgr.24542) 4003 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:20:34.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:20:34.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:20:34.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: yield 2021-07-31T03:20:34.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:20:34.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:20:34.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:20:34.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:20:34.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:20:34.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:20:34.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:20:34.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:20:34.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:20:34.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: task.result() 2021-07-31T03:20:34.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:20:34.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: 2021-07-31T03:20:34.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:20:34.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: 2021-07-31T03:20:34.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:20:34.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:20:34.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:20:34.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:20:34.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:20:34.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:20:34.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:20:34.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:20:34.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:20:34.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:20:34.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:20:34.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:20:34.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:20:34.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:20:34.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:20:34.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:20:34.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:20:34.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:20:34.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: raise 2021-07-31T03:20:34.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:20:34.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:20:34.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:20:34.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:20:34.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:33 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:20:34.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: audit 2021-07-31T03:20:32.594077+0000 mon.b (mon.2) 542 2021-07-31T03:20:34.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:20:34.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: audit 2021-07-31T03:20:32.595542+0000 mon.b (mon. 2021-07-31T03:20:34.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: 2) 543 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:20:34.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: audit 2021-07-31T03:20:34.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: 2021-07-31T03:20:32.596490+0000 mon.b (mon.2 2021-07-31T03:20:34.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: ) 544 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:20:34.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: cephadm 2021-07-31T03 2021-07-31T03:20:34.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: :20:32.597248+0000 mgr.x (mgr.24542) 3995 2021-07-31T03:20:34.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:20:34.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: cephadm 2021-07-31 2021-07-31T03:20:34.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: T03:20:32.641256+0000 mgr.x (mgr.24542) 3996 : cephadm 2021-07-31T03:20:34.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: [ERR] 2021-07-31T03:20:34.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:20:34.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:20:34.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: yield 2021-07-31T03:20:34.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:20:34.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:20:34.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:20:34.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:20:34.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:20:34.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:20:34.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:20:34.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:20:34.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:20:34.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: task.result() 2021-07-31T03:20:34.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:20:34.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: cephadm 2021-07-31T03:20:32.641880+0000 mgr.x (mgr.24542) 3997 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:20:34.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: cephadm 2021-07-31T03:20:32.652902+0000 mgr.x (mgr.24542) 3998 2021-07-31T03:20:34.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: : cephadm [ERR] 2021-07-31T03:20:34.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:20:34.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:20:34.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: yield 2021-07-31T03:20:34.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:20:34.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:20:34.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:20:34.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:20:34.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:20:34.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:20:34.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:20:34.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:20:34.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:20:34.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: task.result() 2021-07-31T03:20:34.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:20:34.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: cephadm 2021-07-31T03:20:34.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: 2021-07-31T03:20:32.653899+0000 mgr.x (mgr.24542) 3999 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:20:34.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: cephadm 2021-07-31T03:20:32.678173+0000 mgr.x (mgr.24542) 4000 : cephadm [ERR] 2021-07-31T03:20:34.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:20:34.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:20:34.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: yield 2021-07-31T03:20:34.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:20:34.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:20:34.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:20:34.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:20:34.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:20:34.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:20:34.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:20:34.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:20:34.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:20:34.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: task.result() 2021-07-31T03:20:34.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:20:34.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: cephadm 2021-07-31T03:20:32.678628+0000 mgr.x (mgr.24542) 4001 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:20:34.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:20:34.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:20:34.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: yield 2021-07-31T03:20:34.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:20:34.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:20:34.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:20:34.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:20:34.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:20:34.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:20:34.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:20:34.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:20:34.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:20:34.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: task.result() 2021-07-31T03:20:34.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:20:34.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: 2021-07-31T03:20:34.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:20:34.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: 2021-07-31T03:20:34.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:20:34.050 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: audit 2021-07-31T03:20:32.594077+0000 mon.b 2021-07-31T03:20:34.051 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: (mon.2) 542 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:20:34.051 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: audit 2021-07-31T03:20:32.595542+0000 mon.b (mon.2) 543 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:20:34.051 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: audit 2021-07-31T03: 2021-07-31T03:20:34.051 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: 20:32.596490+0000 mon.b ( 2021-07-31T03:20:34.051 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: mon.2) 544 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:20:34.052 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: 2021-07-31T03:20:34.052 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: cephadm 2021- 2021-07-31T03:20:34.052 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: 07-31T03:20:32.597248 2021-07-31T03:20:34.052 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 3995 : cephadm 2021-07-31T03:20:34.052 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:20:34.053 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: cephadm 2021-07-31T03 2021-07-31T03:20:34.053 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: :20:32.641256+0000 mgr.x (mgr.24542) 3996 : cephadm [ERR] 2021-07-31T03:20:34.053 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:20:34.053 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:20:34.053 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: yield 2021-07-31T03:20:34.054 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:20:34.054 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:20:34.054 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:20:34.054 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:20:34.054 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:20:34.055 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:20:34.055 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:20:34.055 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:20:34.055 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:20:34.056 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: task.result() 2021-07-31T03:20:34.056 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:20:34.056 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: cephadm 2021-07-31T03:20:32.641880+0000 mgr.x (mgr.24542) 3997 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:20:34.056 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: cephadm 2021-07-31T03 2021-07-31T03:20:34.056 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: :20:32.652902+0000 mgr.x (mgr.24542) 3998 : 2021-07-31T03:20:34.057 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: cephadm [ERR] 2021-07-31T03:20:34.057 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:20:34.057 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:20:34.057 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: yield 2021-07-31T03:20:34.057 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:20:34.058 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:20:34.058 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:20:34.058 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:20:34.058 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:20:34.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:20:34.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:20:34.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:20:34.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:20:34.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:20:34.060 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:20:34.060 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:20:34.060 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:20:34.060 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:20:34.060 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:20:34.061 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:20:34.061 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:20:34.061 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:20:34.061 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:20:34.061 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:20:34.062 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:20:34.062 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:20:34.062 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: raise 2021-07-31T03:20:34.062 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:20:34.062 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:20:34.063 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:20:34.063 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:20:34.063 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:20:34.063 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: cephadm 2021-07-31T03:20:32.687391+0000 mgr.x (mgr 2021-07-31T03:20:34.063 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: .24542) 4002 : cephadm [ERR] 2021-07-31T03:20:34.064 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:20:34.064 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:20:34.064 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: yield 2021-07-31T03:20:34.064 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:20:34.064 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:20:34.065 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:20:34.065 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:20:34.065 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:20:34.065 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:20:34.065 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:20:34.066 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:20:34.066 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:20:34.066 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: task.result() 2021-07-31T03:20:34.066 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:20:34.066 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:20:34.067 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:20:34.067 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:20:34.067 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:20:34.067 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: task.result() 2021-07-31T03:20:34.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:20:34.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: cephadm 2021-07-31T03:20:32.653899+0000 mgr.x (mgr.24542) 3999 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:20:34.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: cephadm 2021-07-31T03:20:32 2021-07-31T03:20:34.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: .678173+0000 mgr.x (mgr.24542) 4000 : cephadm [ERR] 2021-07-31T03:20:34.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: 2021-07-31T03:20:34.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:20:34.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:20:34.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: yield 2021-07-31T03:20:34.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:20:34.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:20:34.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:20:34.070 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:20:34.070 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:20:34.070 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:20:34.070 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:20:34.070 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:20:34.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:20:34.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: task.result() 2021-07-31T03:20:34.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:20:34.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: cephadm 2021-07-31T03:20:32.678628+0000 mgr.x (mgr.24542) 2021-07-31T03:20:34.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: 4001 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:20:34.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:20:34.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:20:34.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: yield 2021-07-31T03:20:34.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:20:34.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:20:34.073 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:20:34.073 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:20:34.073 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:20:34.073 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:20:34.073 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:20:34.074 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:20:34.074 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:20:34.074 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: task.result() 2021-07-31T03:20:34.074 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:20:34.074 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: 2021-07-31T03:20:34.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:20:34.075 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: cephadm 2021-07-31T03:20:32.687948+0000 mgr.x (mgr.24542) 4003 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:20:34.075 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:20:34.075 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:20:34.076 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: yield 2021-07-31T03:20:34.076 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:20:34.076 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:20:34.076 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:20:34.076 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:20:34.077 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:20:34.077 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:20:34.077 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:20:34.077 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:20:34.077 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:20:34.078 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: task.result() 2021-07-31T03:20:34.078 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:20:34.078 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: 2021-07-31T03:20:34.078 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:20:34.078 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: 2021-07-31T03:20:34.079 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:20:34.079 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:20:34.079 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:20:34.079 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:20:34.079 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:20:34.079 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:20:34.080 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:20:34.080 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:20:34.080 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:20:34.080 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:20:34.080 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:20:34.081 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:20:34.081 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:20:34.081 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:20:34.081 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:20:34.081 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:20:34.082 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:20:34.082 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:20:34.082 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: raise 2021-07-31T03:20:34.082 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:20:34.082 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:20:34.083 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:20:34.083 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:20:34.083 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:20:34.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: 2021-07-31T03:20:34.084 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:20:34.084 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:20:34.084 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:20:34.084 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:20:34.084 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:20:34.085 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:20:34.085 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:20:34.085 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:20:34.085 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:20:34.085 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:20:34.086 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:20:34.086 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:20:34.086 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:20:34.086 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:20:34.086 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:20:34.087 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:20:34.087 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:20:34.087 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:20:34.087 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: raise 2021-07-31T03:20:34.087 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:20:34.088 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:20:34.088 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:20:34.088 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:20:34.088 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:20:34.088 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: cephadm 2021-07-31T03:20:32.687391+0000 mgr.x (mgr.24542) 4002 : cephadm [ERR] 2021-07-31T03:20:34.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:20:34.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:20:34.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: yield 2021-07-31T03:20:34.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:20:34.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:20:34.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:20:34.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:20:34.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:20:34.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:20:34.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:20:34.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:20:34.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:20:34.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: task.result() 2021-07-31T03:20:34.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:20:34.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: cephadm 2021-07-31T03:20:32.687948+0000 mgr.x (mgr.24542) 4003 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:20:34.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:20:34.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:20:34.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: yield 2021-07-31T03:20:34.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:20:34.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:20:34.093 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:20:34.093 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:20:34.093 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:20:34.093 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:20:34.093 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:20:34.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:20:34.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:20:34.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: task.result() 2021-07-31T03:20:34.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:20:34.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: 2021-07-31T03:20:34.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:20:34.095 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: 2021-07-31T03:20:34.095 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:20:34.095 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:20:34.095 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:20:34.095 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:20:34.096 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:20:34.096 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:20:34.096 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:20:34.096 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:20:34.096 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:20:34.097 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:20:34.097 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:20:34.097 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:20:34.097 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:20:34.097 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:20:34.098 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:20:34.098 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:20:34.098 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:20:34.098 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:20:34.099 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: raise 2021-07-31T03:20:34.099 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:20:34.099 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:20:34.099 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:20:34.099 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:20:34.100 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:33 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:20:35.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:34 smithi154 conmon[34553]: cluster 2021-07-31T03:20:33.132243+0000 mgr.x (mgr. 2021-07-31T03:20:35.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:34 smithi154 conmon[34553]: 24542) 4004 : cluster [DBG] pgmap v4807: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:35.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:34 smithi045 conmon[33655]: cluster 2021-07-31T03:20:33.132243+0000 mgr.x (mgr.24542) 4004 : 2021-07-31T03:20:35.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:34 smithi045 conmon[33655]: cluster [DBG] pgmap v4807: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:35.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:34 smithi045 conmon[39718]: cluster 2021-07-31T03:20:33. 2021-07-31T03:20:35.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:34 smithi045 conmon[39718]: 132243+0000 mgr.x (mgr.24542) 4004 : cluster [DBG] pgmap v4807: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:37.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:36 smithi154 conmon[34553]: cluster 2021-07-31T03:20:35.132711+0000 mgr.x (mgr.24542) 4005 : cluster [DBG] pgmap v4808: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:37.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:36 smithi045 conmon[33655]: cluster 2021-07-31T03:20:35.132711+0000 mgr.x (mgr 2021-07-31T03:20:37.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:36 smithi045 conmon[33655]: .24542) 4005 : cluster [DBG] pgmap v4808: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:37.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:36 smithi045 conmon[39718]: cluster 2021-07-31T03:20:35.132711+0000 mgr.x (mgr 2021-07-31T03:20:37.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:36 smithi045 conmon[39718]: .24542) 4005 : cluster [DBG] pgmap v4808: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:39.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:38 smithi154 conmon[34553]: cluster 2021-07-31T03:20:37.133233+0000 mgr.x (mgr.24542) 4006 : cluster [DBG] pgmap v4809: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:39.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:38 smithi045 conmon[39718]: cluster 2021-07-31T03:20:37.133233+0000 mgr.x (mgr.24542) 4006 : cluster [DBG] 2021-07-31T03:20:39.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:38 smithi045 conmon[39718]: pgmap v4809: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:39.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:38 smithi045 conmon[33655]: cluster 2021-07-31T03:20:37.133233+0000 mgr.x (mgr.24542 2021-07-31T03:20:39.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:38 smithi045 conmon[33655]: ) 4006 : cluster [DBG] pgmap v4809: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:41.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:40 smithi154 conmon[34553]: cluster 2021-07-31T03:20:39.133639+0000 mgr.x (mgr.24542) 4007 : cluster [DBG] pgmap v4810: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:41.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:40 smithi045 conmon[33655]: cluster 2021-07-31T03:20:39.133639+0000 2021-07-31T03:20:41.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:40 smithi045 conmon[33655]: mgr.x (mgr.24542) 4007 : cluster [DBG] pgmap v4810: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:41.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:40 smithi045 conmon[39718]: cluster 2021-07-31T03:20:39.133639+0000 mgr.x (mgr. 2021-07-31T03:20:41.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:40 smithi045 conmon[39718]: 24542) 4007 : cluster [DBG] pgmap v4810: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:43.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:42 smithi154 conmon[34553]: cluster 2021-07-31T03:20:41.134188+0000 mgr.x (mgr.24542) 4008 : cluster [DBG] pgmap v4811: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:43.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:42 smithi045 conmon[39718]: cluster 2021-07-31T03:20:41.134188+0000 mgr.x ( 2021-07-31T03:20:43.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:42 smithi045 conmon[39718]: mgr.24542) 4008 : cluster [DBG] pgmap v4811: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:43.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:42 smithi045 conmon[33655]: cluster 2021-07-31T03:20:41.134188+0000 mgr.x (mgr.24542) 4008 : cluster [DBG] pgmap v4811: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:45.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:44 smithi154 conmon[34553]: cluster 2021-07-31T03:20:43.134694+0000 mgr.x (mgr.24542) 4009 : cluster [DBG] pgmap v4812: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:45.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:44 smithi045 conmon[39718]: cluster 2021-07-31T03:20:43.134694+0000 mgr.x ( 2021-07-31T03:20:45.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:44 smithi045 conmon[39718]: mgr.24542) 4009 : cluster [DBG] pgmap v4812: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:45.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:44 smithi045 conmon[33655]: cluster 2021-07-31T03:20:43.134694+0000 2021-07-31T03:20:45.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:44 smithi045 conmon[33655]: mgr.x (mgr.24542) 4009 : cluster [DBG] pgmap v4812: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:45.599 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:20:45.877 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:20:45.878 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 95m ago 96m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:20:45.878 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 93m ago 96m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:20:45.878 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 93m ago 99m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:20:45.879 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 95m ago 101m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:20:45.879 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 95m ago 101m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:20:45.879 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 93m ago 99m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:20:45.879 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 95m ago 100m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:20:45.880 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 95m ago 96m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:20:45.880 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 93m ago 96m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:20:45.880 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 95m ago 99m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:20:45.881 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 95m ago 99m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:20:45.881 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 95m ago 98m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:20:45.881 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 95m ago 98m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:20:45.881 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 93m ago 98m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:20:45.882 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 93m ago 97m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:20:45.882 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 93m ago 97m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:20:45.882 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 93m ago 97m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:20:45.883 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 93m ago 96m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:20:46.214 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:20:46.214 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:20:46.215 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:20:46.215 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:20:46.215 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:20:46.216 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:20:46.216 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:20:46.216 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:20:46.217 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:20:46.217 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:20:46.217 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:20:46.217 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:20:46.218 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:20:46.218 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:20:46.218 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:20:46.219 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:20:46.219 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:20:47.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:46 smithi154 conmon[34553]: cluster 2021-07-31T03:20:45.135069+0000 mgr.x (mgr.24542) 4010 : cluster [DBG] pgmap v4813: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:47.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:46 smithi154 conmon[34553]: audit 2021-07-31T03:20:45.586973+0000 mgr.x (mgr.24542) 4011 : audit [DBG] from='client.19308 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:20:47.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:46 smithi154 conmon[34553]: audit 2021-07-31T03:20:45.870434+0000 mgr.x (mgr.24542) 4012 : audit [DBG] from='client.19314 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:20:47.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:46 smithi154 conmon[34553]: audit 2021-07-31 2021-07-31T03:20:47.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:46 smithi154 conmon[34553]: T03:20:46.214487+0000 mon.c ( 2021-07-31T03:20:47.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:46 smithi154 conmon[34553]: mon.1) 196 : audit [DBG] 2021-07-31T03:20:47.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:46 smithi154 conmon[34553]: from='client.? 172.21.15.45:0/1487649321' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:20:47.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:46 smithi045 conmon[39718]: cluster 2021-07-31T03:20:45.135069+0000 mgr.x (mgr.24542) 4010 : 2021-07-31T03:20:47.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:46 smithi045 conmon[39718]: cluster [DBG] pgmap v4813: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:47.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:46 smithi045 conmon[39718]: audit 2021-07-31T03:20:45.586973+0000 mgr.x (mgr.24542) 4011 : audit [DBG] from='client.19308 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:20:47.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:46 smithi045 conmon[39718]: audit 2021-07-31T03:20:45.870434+0000 mgr.x (mgr. 2021-07-31T03:20:47.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:46 smithi045 conmon[39718]: 24542) 4012 : audit [DBG] from='client.19314 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:20:47.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:46 smithi045 conmon[39718]: audit 2021-07-31T03:20:46.214487+0000 mon.c (mon.1 2021-07-31T03:20:47.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:46 smithi045 conmon[39718]: ) 196 : audit [DBG] from='client.? 172.21.15.45:0/1487649321' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:20:47.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:46 smithi045 conmon[33655]: cluster 2021-07-31T03:20:45.135069+0000 mgr.x (mgr.24542) 4010 : cluster [DBG] pgmap v4813: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:47.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:46 smithi045 conmon[33655]: audit 2021-07-31T03:20:45.586973+0000 mgr.x (mgr.24542) 4011 : audit [DBG] from='client.19308 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:20:47.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:46 smithi045 conmon[33655]: audit 2021-07-31T03:20:45.870434+0000 mgr.x ( 2021-07-31T03:20:47.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:46 smithi045 conmon[33655]: mgr.24542) 4012 : audit [DBG] from='client.19314 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:20:47.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:46 smithi045 conmon[33655]: audit 2021-07-31T 2021-07-31T03:20:47.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:46 smithi045 conmon[33655]: 03:20:46.214487+0000 mon.c (mon.1) 196 : audit 2021-07-31T03:20:47.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:46 smithi045 conmon[33655]: [DBG] from='client.? 172.21.15.45:0/1487649321' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:20:49.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:48 smithi154 conmon[34553]: cluster 2021-07-31T03:20:47.135616+0000 mgr.x (mgr.24542) 4013 : cluster [DBG] pgmap v4814: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:49.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:48 smithi045 conmon[39718]: cluster 2021-07-31T03:20:47 2021-07-31T03:20:49.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:48 smithi045 conmon[39718]: .135616+0000 mgr.x (mgr.24542) 4013 : cluster [DBG] pgmap v4814: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:49.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:48 smithi045 conmon[33655]: cluster 2021-07-31T03:20:47.135616+0000 mgr.x (mgr.24542) 4013 : cluster [DBG] pgmap v4814: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:51.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:50 smithi154 conmon[34553]: cluster 2021-07-31T03:20:49.136226+0000 mgr.x (mgr.24542) 4014 : cluster 2021-07-31T03:20:51.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:50 smithi154 conmon[34553]: [DBG] pgmap v4815: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:51.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:50 smithi045 conmon[39718]: cluster 2021-07-31T03:20:49 2021-07-31T03:20:51.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:50 smithi045 conmon[39718]: .136226+0000 mgr.x (mgr.24542) 4014 : cluster [DBG] pgmap v4815: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:51.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:50 smithi045 conmon[33655]: cluster 2021-07-31T03:20:49.136226+0000 mgr.x (mgr.24542) 4014 : cluster 2021-07-31T03:20:51.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:50 smithi045 conmon[33655]: [DBG] pgmap v4815: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:53.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:52 smithi154 conmon[34553]: cluster 2021-07-31T03:20:51.136755+0000 mgr.x (mgr.24542) 4015 : cluster [DBG] pgmap v4816: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:53.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:52 smithi045 conmon[33655]: cluster 2021-07-31T03:20:51.136755+0000 mgr.x (mgr.24542) 4015 : cluster [DBG] 2021-07-31T03:20:53.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:52 smithi045 conmon[33655]: pgmap v4816: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:53.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:52 smithi045 conmon[39718]: cluster 2021-07-31T03:20:51.136755+0000 2021-07-31T03:20:53.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:52 smithi045 conmon[39718]: mgr.x (mgr.24542) 4015 : cluster [DBG] pgmap v4816: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:55.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:54 smithi154 conmon[34553]: cluster 2021-07-31T03:20:53.137294+0000 mgr.x (mgr.24542) 4016 : cluster 2021-07-31T03:20:55.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:54 smithi154 conmon[34553]: [DBG] pgmap v4817: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:55.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:54 smithi045 conmon[33655]: cluster 2021-07-31T03:20:53.137294+0000 2021-07-31T03:20:55.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:54 smithi045 conmon[33655]: mgr.x (mgr.24542) 4016 : cluster [DBG] pgmap v4817: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:55.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:54 smithi045 conmon[39718]: cluster 2021-07-31T03:20:53.137294+0000 mgr.x ( 2021-07-31T03:20:55.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:54 smithi045 conmon[39718]: mgr.24542) 4016 : cluster [DBG] pgmap v4817: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:57.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:56 smithi154 conmon[34553]: cluster 2021-07-31T03:20:55.137878+0000 mgr.x (mgr.24542) 4017 : cluster [DBG] 2021-07-31T03:20:57.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:56 smithi154 conmon[34553]: pgmap v4818: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:57.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:56 smithi045 conmon[33655]: cluster 2021-07-31T03:20:55.137878+0000 mgr.x (mgr.24542) 4017 : cluster 2021-07-31T03:20:57.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:56 smithi045 conmon[33655]: [DBG] pgmap v4818: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:57.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:56 smithi045 conmon[39718]: cluster 2021-07-31T03:20:55.137878+0000 mgr.x ( 2021-07-31T03:20:57.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:56 smithi045 conmon[39718]: mgr.24542) 4017 : cluster [DBG] pgmap v4818: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:59.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:58 smithi154 conmon[34553]: cluster 2021-07-31T03:20:57.138388+0000 mgr.x (mgr.24542) 4018 : 2021-07-31T03:20:59.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:20:58 smithi154 conmon[34553]: cluster [DBG] pgmap v4819: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:59.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:58 smithi045 conmon[39718]: cluster 2021-07-31T03:20:57.138388+0000 mgr.x (mgr 2021-07-31T03:20:59.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:20:58 smithi045 conmon[39718]: .24542) 4018 : cluster [DBG] pgmap v4819: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:20:59.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:58 smithi045 conmon[33655]: cluster 2021-07-31T03:20:57.138388+0000 mgr.x (mgr.24542) 4018 : cluster [DBG] 2021-07-31T03:20:59.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:20:58 smithi045 conmon[33655]: pgmap v4819: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:01.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:00 smithi154 conmon[34553]: cluster 2021-07-31T03:20:59.138987+0000 mgr.x (mgr.24542) 4019 : cluster [DBG] pgmap v4820: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:01.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:00 smithi045 conmon[39718]: cluster 2021-07-31T03:20:59.138987+0000 mgr.x 2021-07-31T03:21:01.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:00 smithi045 conmon[39718]: (mgr.24542) 4019 : cluster [DBG] pgmap v4820: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:01.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:00 smithi045 conmon[33655]: cluster 2021-07-31T03:20:59.138987+0000 mgr.x (mgr.24542) 2021-07-31T03:21:01.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:00 smithi045 conmon[33655]: 4019 : cluster [DBG] pgmap v4820: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:03.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:02 smithi154 conmon[34553]: cluster 2021-07-31T03:21:01.139388+0000 mgr.x (mgr.24542) 4020 : cluster [DBG] pgmap v4821: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:03.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:02 smithi045 conmon[39718]: cluster 2021-07-31T03:21:01.139388+0000 mgr.x (mgr 2021-07-31T03:21:03.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:02 smithi045 conmon[39718]: .24542) 4020 : cluster [DBG] pgmap v4821: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:03.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:02 smithi045 conmon[33655]: cluster 2021-07-31T03:21:01.139388+0000 mgr.x (mgr 2021-07-31T03:21:03.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:02 smithi045 conmon[33655]: .24542) 4020 : cluster [DBG] pgmap v4821: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:05.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:04 smithi154 conmon[34553]: cluster 2021-07-31T03:21:03.139775+0000 mgr.x (mgr.24542) 4021 : cluster [DBG] pgmap v4822: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:05.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:04 smithi045 conmon[33655]: cluster 2021-07-31T03:21:03.139775+0000 mgr.x (mgr.24542) 4021 : cluster [DBG] pgmap v4822: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:05.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:04 smithi045 conmon[39718]: cluster 2021-07-31T03:21:03.139775+0000 mgr.x (mgr 2021-07-31T03:21:05.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:04 smithi045 conmon[39718]: .24542) 4021 : cluster [DBG] pgmap v4822: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:07.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:06 smithi154 conmon[34553]: cluster 2021-07-31T03:21:05.140241+0000 mgr.x (mgr.24542) 4022 : cluster [DBG] pgmap v4823: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:07.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:06 smithi045 conmon[39718]: cluster 2021-07-31T03:21:05.140241+0000 mgr.x (mgr.24542) 4022 : cluster [DBG] pgmap v4823: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:07.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:06 smithi045 conmon[33655]: cluster 2021-07-31T03:21:05.140241+0000 mgr.x (mgr.24542) 4022 : cluster [DBG] pgmap v4823: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:09.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:08 smithi154 conmon[34553]: cluster 2021-07-31T03:21:07.140752+0000 mgr.x (mgr.24542) 4023 : cluster [DBG] pgmap v4824: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:09.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:08 smithi045 conmon[39718]: cluster 2021-07-31T03:21:07.140752+0000 mgr.x (mgr.24542) 4023 : cluster [DBG] pgmap v4824: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:09.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:08 smithi045 conmon[33655]: cluster 2021-07-31T03:21:07.140752+0000 mgr.x (mgr.24542) 4023 : cluster 2021-07-31T03:21:09.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:08 smithi045 conmon[33655]: [DBG] pgmap v4824: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:11.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:10 smithi154 conmon[34553]: cluster 2021-07-31T03:21:09.141166+0000 mgr.x (mgr.24542) 4024 : cluster 2021-07-31T03:21:11.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:10 smithi154 conmon[34553]: [DBG] pgmap v4825: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:11.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:10 smithi045 conmon[33655]: cluster 2021-07-31T03:21:09.141166+0000 mgr.x (mgr.24542) 4024 : cluster [DBG] pgmap v4825: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:11.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:10 smithi045 conmon[39718]: cluster 2021-07-31T03:21:09.141166+0000 mgr.x (mgr.24542) 4024 : 2021-07-31T03:21:11.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:10 smithi045 conmon[39718]: cluster [DBG] pgmap v4825: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:13.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:12 smithi154 conmon[34553]: cluster 2021-07-31T03:21:11.141746+0000 mgr.x (mgr.24542) 4025 : cluster [DBG] pgmap v4826: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:13.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:12 smithi045 conmon[33655]: cluster 2021-07-31T03:21:11.141746+0000 mgr.x (mgr.24542 2021-07-31T03:21:13.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:12 smithi045 conmon[33655]: ) 4025 : cluster [DBG] pgmap v4826: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:13.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:12 smithi045 conmon[39718]: cluster 2021-07-31T03:21:11.141746+0000 mgr.x (mgr.24542) 4025 2021-07-31T03:21:13.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:12 smithi045 conmon[39718]: : cluster [DBG] pgmap v4826: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:15.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:14 smithi154 conmon[34553]: cluster 2021-07-31T03:21:13.142154+0000 mgr.x (mgr.24542) 4026 : cluster [DBG] pgmap v4827: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:15.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:14 smithi045 conmon[39718]: cluster 2021-07-31T03:21:13. 2021-07-31T03:21:15.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:14 smithi045 conmon[39718]: 142154+0000 mgr.x (mgr.24542) 4026 : cluster [DBG] pgmap v4827: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:15.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:14 smithi045 conmon[33655]: cluster 2021-07-31T03:21:13.142154+0000 2021-07-31T03:21:15.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:14 smithi045 conmon[33655]: mgr.x (mgr.24542) 4026 : cluster [DBG] pgmap v4827: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:16.526 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:21:16.804 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:21:16.804 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 96m ago 96m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:21:16.805 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 93m ago 96m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:21:16.805 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 93m ago 100m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:21:16.805 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 96m ago 102m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:21:16.805 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 96m ago 102m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:21:16.806 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 93m ago 100m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:21:16.806 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 96m ago 100m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:21:16.806 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 96m ago 97m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:21:16.806 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 93m ago 97m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:21:16.806 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 96m ago 99m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:21:16.807 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 96m ago 99m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:21:16.807 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 96m ago 99m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:21:16.807 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 96m ago 98m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:21:16.807 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 93m ago 98m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:21:16.808 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 93m ago 98m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:21:16.808 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 93m ago 97m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:21:16.808 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 93m ago 97m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:21:16.808 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 93m ago 97m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:21:17.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:16 smithi154 conmon[34553]: cluster 2021-07-31T03:21:15.142643+0000 mgr.x (mgr.24542) 4027 : cluster [DBG] pgmap v4828: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:17.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:16 smithi045 conmon[33655]: cluster 2021-07-31T03:21:15.142643+0000 2021-07-31T03:21:17.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:16 smithi045 conmon[33655]: mgr.x (mgr.24542) 4027 : cluster [DBG] pgmap v4828: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:17.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:16 smithi045 conmon[39718]: cluster 2021-07-31T03:21:15.142643+0000 mgr.x (mgr.24542) 4027 : cluster [DBG] pgmap v4828: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:17.137 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:21:17.138 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:21:17.138 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:21:17.138 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:21:17.139 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:21:17.139 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:21:17.139 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:21:17.139 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:21:17.140 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:21:17.140 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:21:17.140 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:21:17.140 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:21:17.140 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:21:17.141 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:21:17.141 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:21:17.141 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:21:17.141 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:21:18.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:17 smithi154 conmon[34553]: audit 2021-07-31T03:21:16.515087+0000 mgr.x (mgr.24542) 4028 : audit [DBG] 2021-07-31T03:21:18.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:17 smithi154 conmon[34553]: from='client.29266 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:21:18.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:17 smithi154 conmon[34553]: audit 2021-07-31T03:21:16.797123+0000 mgr.x (mgr.24542) 4029 : audit [DBG] from='client.19332 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:21:18.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:17 smithi154 conmon[34553]: audit 2021-07-31T03:21:18.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:17 smithi154 conmon[34553]: 2021-07-31T03:21:17.138463+0000 mon.a (mon.0) 3291 : audit [DBG] from='client.? 172.21.15.45:0/155229791' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:21:18.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:17 smithi154 conmon[34553]: 2021-07-31T03:21:18.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:17 smithi045 conmon[33655]: audit 2021-07-31T03:21:16.515087+0000 mgr.x (mgr.24542) 2021-07-31T03:21:18.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:17 smithi045 conmon[33655]: 4028 : audit [DBG] from='client.29266 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:21:18.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:17 smithi045 conmon[33655]: audit 2021-07-31T03:21:16.797123+0000 mgr.x (mgr.24542) 4029 : audit [DBG] from='client.19332 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:21:18.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:17 smithi045 conmon[33655]: audit 2021-07-31T 2021-07-31T03:21:18.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:17 smithi045 conmon[33655]: 03:21:17.138463+0000 mon.a (mon.0) 3291 : audit [DBG] from='client.? 172.21.15.45:0/155229791' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:21:18.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:17 smithi045 conmon[39718]: audit 2021-07-31T03:21:16.515087+0000 mgr.x (mgr.24542) 4028 : 2021-07-31T03:21:18.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:17 smithi045 conmon[39718]: audit [DBG] from='client.29266 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:21:18.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:17 smithi045 conmon[39718]: audit 2021-07-31T03:21:16.797123+0000 mgr.x (mgr.24542) 4029 : audit [DBG] from='client.19332 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:21:18.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:17 smithi045 conmon[39718]: audit 2021-07-31T03:21:17.138463+0000 mon.a (mon.0) 3291 : audit [DBG] from='client.? 172.21.15.45:0/155229791' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:21:19.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:18 smithi154 conmon[34553]: cluster 2021-07-31T03:21:17.143089+0000 mgr.x (mgr.24542) 4030 : cluster [DBG] pgmap v4829: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:19.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:18 smithi045 conmon[33655]: cluster 2021-07-31T03:21:17.143089+0000 mgr.x (mgr. 2021-07-31T03:21:19.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:18 smithi045 conmon[33655]: 24542) 4030 : cluster [DBG] pgmap v4829: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:19.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:18 smithi045 conmon[39718]: cluster 2021-07-31T03:21:17 2021-07-31T03:21:19.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:18 smithi045 conmon[39718]: .143089+0000 mgr.x (mgr.24542) 4030 : cluster [DBG] pgmap v4829: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:21.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:20 smithi154 conmon[34553]: cluster 2021-07-31T03:21:19.143547+0000 mgr.x (mgr.24542) 4031 : cluster [DBG] pgmap v4830: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:21.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:20 smithi045 conmon[39718]: cluster 2021-07-31T03:21:19.143547+0000 mgr.x ( 2021-07-31T03:21:21.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:20 smithi045 conmon[39718]: mgr.24542) 4031 : cluster [DBG] pgmap v4830: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:21.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:20 smithi045 conmon[33655]: cluster 2021-07-31T03:21:19.143547+0000 mgr.x (mgr.24542) 4031 : cluster 2021-07-31T03:21:21.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:20 smithi045 conmon[33655]: [DBG] pgmap v4830: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:23.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:22 smithi154 conmon[34553]: cluster 2021-07-31T03:21:21.143979+0000 mgr.x (mgr.24542) 4032 : cluster [DBG] pgmap v4831: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:23.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:22 smithi045 conmon[33655]: cluster 2021-07-31T03:21:21.143979+0000 mgr.x (mgr.24542) 4032 : 2021-07-31T03:21:23.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:22 smithi045 conmon[33655]: cluster [DBG] pgmap v4831: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:23.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:22 smithi045 conmon[39718]: cluster 2021-07-31T03:21:21.143979+0000 mgr.x 2021-07-31T03:21:23.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:22 smithi045 conmon[39718]: (mgr.24542) 4032 : cluster [DBG] pgmap v4831: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:25.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:24 smithi154 conmon[34553]: cluster 2021-07-31T03:21:23.144434+0000 mgr.x (mgr.24542) 4033 : cluster [DBG] 2021-07-31T03:21:25.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:24 smithi154 conmon[34553]: pgmap v4832: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:25.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:24 smithi045 conmon[39718]: cluster 2021-07-31T03:21:23.144434+0000 mgr.x ( 2021-07-31T03:21:25.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:24 smithi045 conmon[39718]: mgr.24542) 4033 : cluster [DBG] pgmap v4832: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:25.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:24 smithi045 conmon[33655]: cluster 2021-07-31T03:21:23.144434+0000 mgr.x (mgr.24542) 4033 : 2021-07-31T03:21:25.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:24 smithi045 conmon[33655]: cluster [DBG] pgmap v4832: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:27.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:26 smithi154 conmon[34553]: cluster 2021-07-31T03:21:25.144965+0000 mgr.x (mgr.24542) 4034 2021-07-31T03:21:27.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:26 smithi154 conmon[34553]: : cluster [DBG] pgmap v4833: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:27.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:26 smithi045 conmon[39718]: cluster 2021-07-31T03:21:25.144965+0000 mgr.x 2021-07-31T03:21:27.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:26 smithi045 conmon[39718]: (mgr.24542) 4034 : cluster [DBG] pgmap v4833: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:27.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:26 smithi045 conmon[33655]: cluster 2021-07-31T03:21:25.144965+0000 mgr.x (mgr.24542) 4034 2021-07-31T03:21:27.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:26 smithi045 conmon[33655]: : cluster [DBG] pgmap v4833: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:29.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:28 smithi154 conmon[34553]: cluster 2021-07-31T03:21:27.145406+0000 mgr.x (mgr.24542) 4035 : cluster [DBG] pgmap v4834: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:29.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:28 smithi154 conmon[34553]: audit 2021-07-31T03:21:28.308028+0000 mon.b (mon.2) 545 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:21:29.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:28 smithi154 conmon[34553]: audit 2021-07-31T03:21:28.308338+0000 mon.b (mon. 2021-07-31T03:21:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:28 smithi154 conmon[34553]: 2) 546 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:21:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:28 smithi154 conmon[34553]: audit 2021-07-31T03:21:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:28 smithi154 conmon[34553]: 2021-07-31T03:21:28 2021-07-31T03:21:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:28 smithi154 conmon[34553]: .308588+0000 mon.a (mon.0) 2021-07-31T03:21:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:28 smithi154 conmon[34553]: 3292 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:21:29.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:28 smithi154 conmon[34553]: audit 2021-07-31T03:21:29.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:28 smithi154 conmon[34553]: 2021-07-31T03:21:28 2021-07-31T03:21:29.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:28 smithi154 conmon[34553]: .308984+0000 mon.a (mon.0) 2021-07-31T03:21:29.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:28 smithi154 conmon[34553]: 3293 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:21:29.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:28 smithi045 conmon[33655]: cluster 2021-07-31T03:21:27.145406+0000 mgr.x (mgr 2021-07-31T03:21:29.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:28 smithi045 conmon[33655]: .24542) 4035 : cluster [DBG] pgmap v4834: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:29.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:28 smithi045 conmon[33655]: audit 2021-07-31T03:21:28.308028+0000 mon.b (mon.2) 545 : audit 2021-07-31T03:21:29.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:28 smithi045 conmon[33655]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:21:29.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:28 smithi045 conmon[33655]: audit 2021-07-31T03:21:29.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:28 smithi045 conmon[33655]: 2021-07-31T 2021-07-31T03:21:29.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:28 smithi045 conmon[33655]: 03:21:28.308338+0000 mon.b (mon. 2021-07-31T03:21:29.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:28 smithi045 conmon[33655]: 2) 546 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:21:29.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:28 smithi045 conmon[33655]: 2021-07-31T03:21:29.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:28 smithi045 conmon[33655]: audit 2021- 2021-07-31T03:21:29.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:28 smithi045 conmon[33655]: 07-31T03:21:28.308588+0000 mon.a (mon.0) 2021-07-31T03:21:29.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:28 smithi045 conmon[33655]: 3292 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:21:29.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:28 smithi045 conmon[33655]: audit 2021-07 2021-07-31T03:21:29.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:28 smithi045 conmon[33655]: -31T03:21:28.308984+0000 mon.a (mon.0) 2021-07-31T03:21:29.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:28 smithi045 conmon[33655]: 3293 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:21:29.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:28 smithi045 conmon[39718]: cluster 2021-07-31T03:21:27 2021-07-31T03:21:29.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:28 smithi045 conmon[39718]: .145406+0000 mgr.x (mgr.24542) 4035 : cluster [DBG] pgmap v4834: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:29.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:28 smithi045 conmon[39718]: audit 2021-07-31T03:21:29.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:28 smithi045 conmon[39718]: 2021-07-31T03: 2021-07-31T03:21:29.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:28 smithi045 conmon[39718]: 21:28.308028+0000 mon.b 2021-07-31T03:21:29.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:28 smithi045 conmon[39718]: (mon.2) 545 : audit 2021-07-31T03:21:29.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:28 smithi045 conmon[39718]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:21:29.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:28 smithi045 conmon[39718]: audit 2021 2021-07-31T03:21:29.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:28 smithi045 conmon[39718]: -07-31T03:21:28. 2021-07-31T03:21:29.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:28 smithi045 conmon[39718]: 308338+0000 mon.b (mon.2 2021-07-31T03:21:29.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:28 smithi045 conmon[39718]: ) 546 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:21:29.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:28 smithi045 conmon[39718]: audit 2021- 2021-07-31T03:21:29.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:28 smithi045 conmon[39718]: 07-31T03:21:28.308588 2021-07-31T03:21:29.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:28 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2021-07-31T03:21:29.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:28 smithi045 conmon[39718]: 3292 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:21:29.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:28 smithi045 conmon[39718]: audit 2021 2021-07-31T03:21:29.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:28 smithi045 conmon[39718]: -07-31T03:21:28 2021-07-31T03:21:29.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:28 smithi045 conmon[39718]: .308984+0000 mon.a (mon. 2021-07-31T03:21:29.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:28 smithi045 conmon[39718]: 0) 3293 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:21:31.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:30 smithi154 conmon[34553]: cluster 2021-07-31T03:21:29.145820+0000 mgr.x (mgr.24542) 4036 : cluster [DBG] pgmap v4835: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:31.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:30 smithi045 conmon[33655]: cluster 2021-07-31T03:21:29.145820+0000 mgr.x (mgr. 2021-07-31T03:21:31.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:30 smithi045 conmon[33655]: 24542) 4036 : cluster [DBG] pgmap v4835: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:31.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:30 smithi045 conmon[39718]: cluster 2021-07-31T03:21:29.145820+0000 mgr.x (mgr 2021-07-31T03:21:31.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:30 smithi045 conmon[39718]: .24542) 4036 : cluster [DBG] pgmap v4835: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:33.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[34553]: cluster 2021-07-31T03:21:31.146295+0000 mgr.x (mgr.24542) 4037 : cluster 2021-07-31T03:21:33.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[34553]: [DBG] pgmap v4836: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:33.003 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: debug 2021-07-31T03:21:32.733+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:21:33.003 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:21:33.003 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:21:33.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: yield 2021-07-31T03:21:33.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:21:33.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:21:33.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:21:33.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:21:33.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:21:33.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:21:33.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:21:33.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:21:33.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:21:33.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: task.result() 2021-07-31T03:21:33.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:21:33.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: debug 2021-07-31T03:21:32.744+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:21:33.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:21:33.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:21:33.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: yield 2021-07-31T03:21:33.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:21:33.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:21:33.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:21:33.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:21:33.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:21:33.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:21:33.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:21:33.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:21:33.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:21:33.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: task.result() 2021-07-31T03:21:33.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:21:33.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: debug 2021-07-31T03:21:32.770+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:21:33.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:21:33.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:21:33.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: yield 2021-07-31T03:21:33.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:21:33.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:21:33.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:21:33.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:21:33.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:21:33.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:21:33.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:21:33.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:21:33.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:21:33.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: task.result() 2021-07-31T03:21:33.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:21:33.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: debug 2021-07-31T03:21:32.770+0000 7fcb97969700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:21:33.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:21:33.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:21:33.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: yield 2021-07-31T03:21:33.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:21:33.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:21:33.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:21:33.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:21:33.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:21:33.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:21:33.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:21:33.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:21:33.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:21:33.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: task.result() 2021-07-31T03:21:33.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:21:33.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: 2021-07-31T03:21:33.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:21:33.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: 2021-07-31T03:21:33.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:21:33.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:21:33.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:21:33.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:21:33.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:21:33.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:21:33.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:21:33.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:21:33.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:21:33.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:21:33.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:21:33.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:21:33.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:21:33.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:21:33.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:21:33.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:21:33.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:21:33.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:21:33.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: raise 2021-07-31T03:21:33.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:21:33.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:21:33.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:21:33.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:21:33.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:21:33.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: debug 2021-07-31T03:21:32.779+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:21:33.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:21:33.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:21:33.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: yield 2021-07-31T03:21:33.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:21:33.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:21:33.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:21:33.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:21:33.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:21:33.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:21:33.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:21:33.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:21:33.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:21:33.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: task.result() 2021-07-31T03:21:33.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:21:33.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: debug 2021-07-31T03:21:32.779+0000 7fcb9816a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:21:33.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:21:33.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:21:33.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: yield 2021-07-31T03:21:33.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:21:33.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:21:33.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:21:33.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:21:33.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:21:33.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:21:33.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:21:33.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:21:33.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:21:33.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: task.result() 2021-07-31T03:21:33.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:21:33.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: 2021-07-31T03:21:33.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:21:33.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: 2021-07-31T03:21:33.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:21:33.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:21:33.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:21:33.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:21:33.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:21:33.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:21:33.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:21:33.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:21:33.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:21:33.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:21:33.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:21:33.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:21:33.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:21:33.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:21:33.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:21:33.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:21:33.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:21:33.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:21:33.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: raise 2021-07-31T03:21:33.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:21:33.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:21:33.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:21:33.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:21:33.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:21:32 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:21:33.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:32 smithi045 conmon[33655]: cluster 2021-07-31T03:21:31.146295+0000 mgr.x (mgr.24542) 2021-07-31T03:21:33.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:32 smithi045 conmon[33655]: 4037 : cluster [DBG] pgmap v4836: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:33.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:32 smithi045 conmon[39718]: cluster 2021-07-31T03:21:31.146295+0000 mgr.x (mgr. 2021-07-31T03:21:33.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:32 smithi045 conmon[39718]: 24542) 4037 : cluster [DBG] pgmap v4836: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:34.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: audit 2021-07-31T03:21:32.688626+0000 mon.b (mon.2) 547 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:21:34.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: audit 2021-07-31T03:21:32.689868+0000 mon.b (mon.2) 548 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:21:34.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: audit 2021-07-31T03:21:32.690836+0000 mon.b (mon 2021-07-31T03:21:34.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: .2) 549 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:21:34.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: cephadm 2021 2021-07-31T03:21:34.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: -07-31T03:21:32 2021-07-31T03:21:34.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: .691581+0000 2021-07-31T03:21:34.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: mgr.x (mgr.24542) 4038 : cephadm 2021-07-31T03:21:34.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:21:34.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:21:34.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: 2021-07-31T 2021-07-31T03:21:34.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: 03:21:32.734417+0000 2021-07-31T03:21:34.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: mgr.x (mgr.24542) 4039 : cephadm [ERR] 2021-07-31T03:21:34.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: 2021-07-31T03:21:34.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:21:34.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:21:34.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: yield 2021-07-31T03:21:34.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:21:34.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:21:34.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:21:34.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:21:34.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:21:34.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:21:34.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:21:34.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:21:34.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:21:34.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: task.result() 2021-07-31T03:21:34.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:21:34.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:21:32.734843+0000 mgr.x (mgr.24542) 4040 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:21:34.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: cephadm 2021- 2021-07-31T03:21:34.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: 07-31T03:21:32.745793 2021-07-31T03:21:34.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: +0000 mgr.x (mgr.24542) 4041 2021-07-31T03:21:34.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: : cephadm [ERR] 2021-07-31T03:21:34.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:21:34.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:21:34.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: yield 2021-07-31T03:21:34.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:21:34.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:21:34.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:21:34.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:21:34.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:21:34.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:21:34.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:21:34.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:21:34.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:21:34.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: task.result() 2021-07-31T03:21:34.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:21:34.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:21:32.746644+0000 mgr.x (mgr.24542) 4042 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:21:34.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: cephadm 2021- 2021-07-31T03:21:34.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: 07-31T03:21:32.771063+0000 2021-07-31T03:21:34.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: mgr.x (mgr.24542) 4043 : 2021-07-31T03:21:34.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: cephadm [ERR] 2021-07-31T03:21:34.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:21:34.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:21:34.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: yield 2021-07-31T03:21:34.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:21:34.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:21:34.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:21:34.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:21:34.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:21:34.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:21:34.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:21:34.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:21:34.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:21:34.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: task.result() 2021-07-31T03:21:34.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:21:34.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:21:32.771484+0000 mgr.x (mgr.24542) 4044 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:21:34.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:21:34.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:21:34.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: yield 2021-07-31T03:21:34.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:21:34.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:21:34.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:21:34.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:21:34.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:21:34.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:21:34.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:21:34.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:21:34.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:21:34.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: task.result() 2021-07-31T03:21:34.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:21:34.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: 2021-07-31T03:21:34.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:21:34.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: 2021-07-31T03:21:34.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:21:34.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:21:34.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:21:34.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:21:34.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:21:34.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:21:34.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:21:34.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:21:34.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:21:34.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:21:34.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:21:34.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:21:34.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:21:34.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:21:34.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:21:34.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:21:34.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:21:34.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:21:34.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: raise 2021-07-31T03:21:34.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:21:34.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:21:34.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:21:34.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:21:34.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:21:34.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:21:32.780019+0000 mgr.x (mgr.24542) 4045 : cephadm [ERR] 2021-07-31T03:21:34.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:21:34.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:21:34.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: yield 2021-07-31T03:21:34.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:21:34.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:21:34.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:21:34.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:21:34.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:21:34.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:21:34.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:21:34.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:21:34.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:21:34.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: task.result() 2021-07-31T03:21:34.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:21:34.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:21:32.780377+0000 mgr.x (mgr.24542) 4046 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:21:34.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:21:34.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:21:34.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: yield 2021-07-31T03:21:34.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:21:34.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:21:34.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:21:34.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:21:34.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:21:34.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:21:34.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:21:34.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:21:34.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:21:34.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: task.result() 2021-07-31T03:21:34.035 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:21:34.035 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: 2021-07-31T03:21:34.035 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:21:34.035 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: 2021-07-31T03:21:34.035 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:21:34.036 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:21:34.036 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:21:34.036 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:21:34.036 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:21:34.036 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:21:34.037 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:21:34.037 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:21:34.037 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:21:34.037 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:21:34.038 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:21:34.038 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:21:34.038 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:21:34.038 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:21:34.038 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:21:34.039 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:21:34.039 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:21:34.039 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:21:34.039 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: raise 2021-07-31T03:21:34.040 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:21:34.040 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:21:34.040 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:21:34.040 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:21:34.040 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:33 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:21:34.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: audit 2021-07-31T03:21:32.688626+0000 2021-07-31T03:21:34.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: mon.b (mon.2) 547 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:21:34.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: audit 2021-07-31T03 2021-07-31T03:21:34.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: :21:32.689868+0000 mon.b (mon.2) 548 : audit [DBG] 2021-07-31T03:21:34.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:21:34.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: audit 2021-07-31T03:21:34.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: 2021-07-31T03:21:32.690836+0000 2021-07-31T03:21:34.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: mon.b (mon.2) 549 : audit [INF] 2021-07-31T03:21:34.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:21:34.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: cephadm 2021-07-31T03 2021-07-31T03:21:34.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: :21:32.691581+0000 mgr.x (mgr 2021-07-31T03:21:34.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: .24542) 4038 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:21:34.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: cephadm 2021-07 2021-07-31T03:21:34.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: -31T03:21:32.734417+0000 mgr.x 2021-07-31T03:21:34.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: (mgr.24542) 4039 : cephadm [ERR] 2021-07-31T03:21:34.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: 2021-07-31T03:21:34.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:21:34.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:21:34.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: yield 2021-07-31T03:21:34.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:21:34.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:21:34.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:21:34.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:21:34.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:21:34.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:21:34.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:21:34.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:21:34.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:21:34.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: task.result() 2021-07-31T03:21:34.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:21:34.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: cephadm 2021-07-31T03:21:32.734843+0000 mgr.x (mgr.24542) 4040 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:21:34.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: cephadm 2021-07-31T03: 2021-07-31T03:21:34.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: 21:32.745793+0000 mgr.x (mgr.24542 2021-07-31T03:21:34.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: ) 4041 : cephadm [ERR] 2021-07-31T03:21:34.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:21:34.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:21:34.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: yield 2021-07-31T03:21:34.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:21:34.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:21:34.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:21:34.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:21:34.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:21:34.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:21:34.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:21:34.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:21:34.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:21:34.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: task.result() 2021-07-31T03:21:34.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:21:34.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: cephadm 2021- 2021-07-31T03:21:34.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: 07-31T03:21:32.746644+0000 mgr.x (mgr.24542) 4042 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:21:34.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: cephadm 2021-07-31T03:21:32.771063+0000 mgr.x (mgr.24542) 4043 : 2021-07-31T03:21:34.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: cephadm [ERR] 2021-07-31T03:21:34.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:21:34.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:21:34.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: yield 2021-07-31T03:21:34.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:21:34.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:21:34.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:21:34.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:21:34.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:21:34.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:21:34.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:21:34.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:21:34.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:21:34.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: task.result() 2021-07-31T03:21:34.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:21:34.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: cephadm 2021-07-31T03:21:32.771484+0000 mgr.x (mgr.24542) 4044 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:21:34.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:21:34.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:21:34.057 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: yield 2021-07-31T03:21:34.057 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:21:34.057 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:21:34.057 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:21:34.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:21:34.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:21:34.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:21:34.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:21:34.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:21:34.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:21:34.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: task.result() 2021-07-31T03:21:34.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:21:34.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: 2021-07-31T03:21:34.060 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: audit 2021-07-31T03:21:32.688626+0000 mon.b (mon.2) 547 : 2021-07-31T03:21:34.060 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:21:34.060 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: audit 2021-07-31T03:21:32.689868+0000 mon.b (mon.2) 548 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:21:34.060 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: audit 2021- 2021-07-31T03:21:34.061 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: 07-31T03:21:32.690836+0000 mon.b (mon.2) 549 2021-07-31T03:21:34.061 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:21:34.061 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: cephadm 2021-07- 2021-07-31T03:21:34.061 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: 31T03:21:32.691581+0000 mgr.x (mgr.24542) 4038 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:21:34.061 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: cephadm 2021-07-31 2021-07-31T03:21:34.062 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: T03:21:32.734417+0000 mgr.x (mgr.24542) 4039 2021-07-31T03:21:34.062 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: : cephadm [ERR] 2021-07-31T03:21:34.062 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:21:34.062 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:21:34.063 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: yield 2021-07-31T03:21:34.063 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:21:34.063 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:21:34.063 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:21:34.063 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:21:34.064 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:21:34.064 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:21:34.064 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:21:34.064 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:21:34.064 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:21:34.065 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: task.result() 2021-07-31T03:21:34.065 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:21:34.065 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: cephadm 2021-07-31T03:21:32.734843+0000 mgr.x (mgr.24542) 4040 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:21:34.065 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: cephadm 2021-07-31T03:21:34.066 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: 2021-07-31T 2021-07-31T03:21:34.066 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: 03:21:32.745793+0000 mgr.x (mgr.24542 2021-07-31T03:21:34.066 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: ) 4041 : cephadm [ERR] 2021-07-31T03:21:34.066 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:21:34.067 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:21:34.067 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: yield 2021-07-31T03:21:34.067 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:21:34.067 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:21:34.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:21:34.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:21:34.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:21:34.069 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:21:34.069 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: 2021-07-31T03:21:34.069 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:21:34.069 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:21:34.070 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:21:34.070 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:21:34.070 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:21:34.070 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:21:34.070 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:21:34.071 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:21:34.071 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:21:34.071 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:21:34.071 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:21:34.071 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:21:34.072 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:21:34.072 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:21:34.072 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:21:34.072 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:21:34.072 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:21:34.073 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:21:34.073 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: raise 2021-07-31T03:21:34.073 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:21:34.073 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:21:34.073 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:21:34.074 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:21:34.074 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:21:34.074 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: cephadm 2021-07-31T03:21:32.780019+0000 mgr.x (mgr.24542) 4045 : cephadm [ERR] 2021-07-31T03:21:34.074 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:21:34.075 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:21:34.075 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: yield 2021-07-31T03:21:34.075 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:21:34.075 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:21:34.075 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:21:34.076 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:21:34.076 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:21:34.076 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:21:34.076 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:21:34.076 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:21:34.077 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:21:34.077 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: task.result() 2021-07-31T03:21:34.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:21:34.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:21:34.078 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:21:34.078 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:21:34.078 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: task.result() 2021-07-31T03:21:34.078 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:21:34.079 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: cephadm 2021-07-31T03:21:32.746644+0000 mgr.x (mgr.24542) 4042 : cephadm [INF] 2021-07-31T03:21:34.079 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:21:34.079 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: cephadm 2021-07-31T03:21:34.079 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: 2021- 2021-07-31T03:21:34.079 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: 07-31T03:21:32 2021-07-31T03:21:34.080 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: .771063+0000 mgr.x (mgr.24542) 2021-07-31T03:21:34.080 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: 4043 : cephadm [ERR] 2021-07-31T03:21:34.080 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:21:34.080 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:21:34.080 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: yield 2021-07-31T03:21:34.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:21:34.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:21:34.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:21:34.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:21:34.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:21:34.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:21:34.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:21:34.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:21:34.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:21:34.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: task.result() 2021-07-31T03:21:34.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:21:34.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: cephadm 2021-07-31T03:21:32.771484+0000 mgr.x (mgr.24542) 4044 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:21:34.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:21:34.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:21:34.084 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: yield 2021-07-31T03:21:34.084 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:21:34.084 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:21:34.084 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:21:34.085 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:21:34.085 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:21:34.085 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:21:34.085 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:21:34.085 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:21:34.086 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:21:34.086 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: task.result() 2021-07-31T03:21:34.086 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:21:34.086 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: 2021-07-31T03:21:34.087 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:21:34.087 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: cephadm 2021-07-31T03:21:32.780377+0000 mgr.x (mgr.24542) 4046 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:21:34.087 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:21:34.087 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:21:34.088 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: yield 2021-07-31T03:21:34.088 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:21:34.088 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:21:34.088 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:21:34.088 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:21:34.089 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:21:34.089 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:21:34.089 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:21:34.089 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:21:34.089 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:21:34.090 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: task.result() 2021-07-31T03:21:34.090 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:21:34.090 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: 2021-07-31T03:21:34.090 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:21:34.091 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: 2021-07-31T03:21:34.091 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:21:34.091 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:21:34.091 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:21:34.091 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:21:34.092 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:21:34.092 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:21:34.092 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:21:34.092 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:21:34.093 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:21:34.093 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:21:34.093 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:21:34.093 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:21:34.093 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:21:34.094 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:21:34.094 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:21:34.094 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:21:34.094 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:21:34.095 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:21:34.095 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: raise 2021-07-31T03:21:34.095 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:21:34.095 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:21:34.095 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:21:34.096 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:21:34.096 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:21:34.096 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:21:34.097 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: 2021-07-31T03:21:34.097 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:21:34.097 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:21:34.097 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:21:34.098 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:21:34.098 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:21:34.098 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:21:34.098 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:21:34.098 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:21:34.099 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:21:34.099 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:21:34.099 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:21:34.099 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:21:34.100 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:21:34.100 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:21:34.100 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:21:34.100 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:21:34.100 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:21:34.101 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:21:34.101 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: raise 2021-07-31T03:21:34.101 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:21:34.101 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:21:34.101 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:21:34.102 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:21:34.102 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:21:34.102 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: 2021-07-31T03:21:34.102 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: cephadm 2021-07-31T03:21:32.780019+0000 mgr.x (mgr.24542) 4045 : cephadm [ERR] 2021-07-31T03:21:34.103 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:21:34.103 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:21:34.103 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: yield 2021-07-31T03:21:34.103 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:21:34.103 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:21:34.104 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:21:34.104 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:21:34.104 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:21:34.104 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:21:34.105 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:21:34.105 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:21:34.105 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:21:34.105 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: task.result() 2021-07-31T03:21:34.106 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:21:34.106 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: cephadm 2021-07-31T03:21:32.780377+0000 mgr.x (mgr.24542) 4046 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:21:34.106 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:21:34.106 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:21:34.106 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: yield 2021-07-31T03:21:34.107 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:21:34.107 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:21:34.107 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:21:34.107 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:21:34.108 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:21:34.108 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:21:34.108 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:21:34.108 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:21:34.108 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:21:34.109 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: task.result() 2021-07-31T03:21:34.109 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:21:34.109 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: 2021-07-31T03:21:34.109 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:21:34.110 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: 2021-07-31T03:21:34.110 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:21:34.110 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:21:34.110 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:21:34.110 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:21:34.111 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:21:34.111 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:21:34.111 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:21:34.111 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:21:34.112 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:21:34.112 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:21:34.112 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:21:34.112 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:21:34.112 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:21:34.113 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:21:34.113 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:21:34.113 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:21:34.113 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:21:34.113 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:21:34.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: raise 2021-07-31T03:21:34.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:21:34.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:21:34.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:21:34.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:21:34.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:33 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:21:35.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:34 smithi154 conmon[34553]: cluster 2021-07-31T03:21:33.146718+0000 mgr.x (mgr.24542) 4047 : cluster 2021-07-31T03:21:35.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:34 smithi154 conmon[34553]: [DBG] pgmap v4837: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:35.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:34 smithi045 conmon[33655]: cluster 2021-07-31T03:21:33.146718+0000 mgr.x (mgr.24542) 4047 : cluster [DBG] pgmap v4837: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:35.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:34 smithi045 conmon[39718]: cluster 2021-07-31T03:21:33.146718+0000 mgr.x (mgr 2021-07-31T03:21:35.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:34 smithi045 conmon[39718]: .24542) 4047 : cluster [DBG] pgmap v4837: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:37.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:36 smithi154 conmon[34553]: cluster 2021-07-31T03:21:35.147281+0000 mgr.x (mgr.24542) 4048 : cluster [DBG] pgmap v4838: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:37.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:36 smithi045 conmon[33655]: cluster 2021-07-31T03:21:35.147281+0000 mgr.x (mgr.24542) 4048 : cluster [DBG] pgmap v4838: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:37.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:36 smithi045 conmon[39718]: cluster 2021-07-31T03:21:35. 2021-07-31T03:21:37.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:36 smithi045 conmon[39718]: 147281+0000 mgr.x (mgr.24542) 4048 : cluster [DBG] pgmap v4838: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:39.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:38 smithi154 conmon[34553]: cluster 2021-07-31T03:21:37.147657+0000 mgr.x (mgr.24542) 4049 : cluster 2021-07-31T03:21:39.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:38 smithi154 conmon[34553]: [DBG] pgmap v4839: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:39.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:38 smithi045 conmon[39718]: cluster 2021-07-31T03:21:37.147657+0000 mgr.x (mgr.24542) 4049 : cluster [DBG] pgmap v4839: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:39.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:38 smithi045 conmon[33655]: cluster 2021-07-31T03:21:37.147657+0000 mgr.x 2021-07-31T03:21:39.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:38 smithi045 conmon[33655]: (mgr.24542) 4049 : cluster [DBG] pgmap v4839: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:41.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:40 smithi154 conmon[34553]: cluster 2021-07-31T03:21:39.148299+0000 mgr.x (mgr.24542) 4050 : cluster [DBG] pgmap v4840: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:41.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:40 smithi154 conmon[34553]: 2021-07-31T03:21:41.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:40 smithi045 conmon[39718]: cluster 2021-07-31T03:21:39.148299+0000 mgr.x (mgr. 2021-07-31T03:21:41.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:40 smithi045 conmon[39718]: 24542) 4050 : cluster [DBG] pgmap v4840: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:41.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:40 smithi045 conmon[33655]: cluster 2021-07-31T03:21:39.148299+0000 mgr.x (mgr.24542 2021-07-31T03:21:41.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:40 smithi045 conmon[33655]: ) 4050 : cluster [DBG] pgmap v4840: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:43.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:42 smithi154 conmon[34553]: cluster 2021-07-31T03:21:41.148646+0000 mgr.x (mgr.24542 2021-07-31T03:21:43.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:42 smithi154 conmon[34553]: ) 4051 : cluster [DBG] pgmap v4841: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:43.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:42 smithi045 conmon[39718]: cluster 2021-07-31T03:21:41.148646+0000 mgr.x 2021-07-31T03:21:43.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:42 smithi045 conmon[39718]: (mgr.24542) 4051 : cluster [DBG] pgmap v4841: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:43.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:42 smithi045 conmon[33655]: cluster 2021-07-31T03:21:41.148646+0000 mgr.x (mgr.24542) 4051 : cluster [DBG] pgmap v4841: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:45.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:44 smithi154 conmon[34553]: cluster 2021-07-31T03:21:43.149053+0000 mgr.x (mgr.24542) 4052 : cluster [DBG] 2021-07-31T03:21:45.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:44 smithi154 conmon[34553]: pgmap v4842: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:45.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:44 smithi045 conmon[39718]: cluster 2021-07-31T03:21:43.149053+0000 mgr.x (mgr 2021-07-31T03:21:45.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:44 smithi045 conmon[39718]: .24542) 4052 : cluster [DBG] pgmap v4842: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:45.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:44 smithi045 conmon[33655]: cluster 2021-07-31T03:21:43.149053+0000 mgr.x (mgr.24542) 4052 : 2021-07-31T03:21:45.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:44 smithi045 conmon[33655]: cluster [DBG] pgmap v4842: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:47.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:46 smithi154 conmon[34553]: cluster 2021-07-31T03:21:45.149439+0000 mgr.x (mgr.24542) 4053 : cluster [DBG] 2021-07-31T03:21:47.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:46 smithi154 conmon[34553]: pgmap v4843: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:47.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:46 smithi045 conmon[39718]: cluster 2021-07-31T03:21:45.149439+0000 mgr.x (mgr.24542) 4053 : cluster [DBG] pgmap v4843: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:47.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:46 smithi045 conmon[33655]: cluster 2021-07-31T03:21:45.149439+0000 mgr.x (mgr.24542 2021-07-31T03:21:47.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:46 smithi045 conmon[33655]: ) 4053 : cluster [DBG] pgmap v4843: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:47.449 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:21:47.726 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:21:47.726 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 96m ago 97m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:21:47.727 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 94m ago 97m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:21:47.727 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 94m ago 100m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:21:47.727 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 96m ago 102m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:21:47.728 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 96m ago 102m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:21:47.728 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 94m ago 100m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:21:47.728 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 96m ago 101m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:21:47.728 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 96m ago 97m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:21:47.729 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 94m ago 97m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:21:47.729 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 96m ago 100m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:21:47.729 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 96m ago 100m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:21:47.729 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 96m ago 99m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:21:47.730 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 96m ago 99m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:21:47.730 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 94m ago 99m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:21:47.730 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 94m ago 98m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:21:47.730 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 94m ago 98m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:21:47.731 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 94m ago 98m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:21:47.731 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 94m ago 97m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:21:48.064 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:21:48.065 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:21:48.065 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:21:48.065 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:21:48.066 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:21:48.066 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:21:48.066 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:21:48.066 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:21:48.067 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:21:48.067 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:21:48.067 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:21:48.068 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:21:48.068 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:21:48.068 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:21:48.068 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:21:48.069 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:21:48.069 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:21:49.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:48 smithi154 conmon[34553]: cluster 2021-07-31T03:21:47.149971+0000 mgr.x (mgr.24542) 4054 : cluster 2021-07-31T03:21:49.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:48 smithi154 conmon[34553]: [DBG] pgmap v4844: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:49.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:48 smithi154 conmon[34553]: audit 2021-07-31T03:21:47.438624+0000 mgr.x (mgr.24542) 4055 : audit [DBG] from='client.19344 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:21:49.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:48 smithi154 conmon[34553]: audit 2021-07-31T03:21:49.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:48 smithi154 conmon[34553]: 2021-07-31T03:21:47.718604+0000 mgr.x (mgr.24542) 4056 : audit [DBG] from='client.19350 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:21:49.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:48 smithi154 conmon[34553]: audit 2021-07-31T 2021-07-31T03:21:49.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:48 smithi154 conmon[34553]: 03:21:48.065452+0000 mon.a (mon.0) 3294 2021-07-31T03:21:49.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:48 smithi154 conmon[34553]: : audit [DBG] from='client.? 172.21.15.45:0/1795371557' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:21:49.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:48 smithi045 conmon[33655]: cluster 2021-07-31T03:21:47.149971+0000 mgr.x (mgr.24542) 4054 : cluster [DBG] pgmap v4844: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:49.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:48 smithi045 conmon[33655]: audit 2021-07-31T03:21:47 2021-07-31T03:21:49.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:48 smithi045 conmon[33655]: .438624+0000 mgr.x (mgr.24542) 4055 : audit [DBG] from='client.19344 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:21:49.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:48 smithi045 conmon[33655]: audit 2021-07-31T03:21:47.718604+0000 mgr.x (mgr.24542) 4056 : audit 2021-07-31T03:21:49.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:48 smithi045 conmon[33655]: [DBG] from='client.19350 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:21:49.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:48 smithi045 conmon[33655]: audit 2021-07-31T03:21:49.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:48 smithi045 conmon[33655]: 2021-07-31T03:21:48.065452 2021-07-31T03:21:49.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:48 smithi045 conmon[33655]: +0000 mon.a (mon.0) 3294 : audit [DBG] from='client.? 172.21.15.45:0/1795371557' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:21:49.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:48 smithi045 conmon[39718]: cluster 2021-07-31T03:21:47.149971+0000 2021-07-31T03:21:49.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:48 smithi045 conmon[39718]: mgr.x (mgr.24542) 4054 : cluster [DBG] pgmap v4844: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:49.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:48 smithi045 conmon[39718]: audit 2021-07-31T03 2021-07-31T03:21:49.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:48 smithi045 conmon[39718]: :21:47.438624+0000 mgr.x (mgr.24542) 4055 2021-07-31T03:21:49.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:48 smithi045 conmon[39718]: : audit [DBG] from='client.19344 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:21:49.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:48 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T03:21:49.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:48 smithi045 conmon[39718]: T03:21:47.718604+0000 mgr.x 2021-07-31T03:21:49.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:48 smithi045 conmon[39718]: (mgr.24542) 4056 : audit [DBG] from='client.19350 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:21:49.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:48 smithi045 conmon[39718]: 2021-07-31T03:21:49.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:48 smithi045 conmon[39718]: audit 2021 2021-07-31T03:21:49.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:48 smithi045 conmon[39718]: -07-31T03:21 2021-07-31T03:21:49.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:48 smithi045 conmon[39718]: :48.065452+0000 mon.a (mon 2021-07-31T03:21:49.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:48 smithi045 conmon[39718]: .0) 3294 : audit [DBG] from='client.? 172.21.15.45:0/1795371557' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:21:51.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:50 smithi154 conmon[34553]: cluster 2021-07-31T03:21:49.150448+0000 mgr.x (mgr.24542) 4057 : cluster 2021-07-31T03:21:51.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:50 smithi154 conmon[34553]: [DBG] pgmap v4845: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:51.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:50 smithi045 conmon[39718]: cluster 2021-07-31T03:21:49.150448+0000 mgr.x 2021-07-31T03:21:51.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:50 smithi045 conmon[39718]: (mgr.24542) 4057 : cluster [DBG] pgmap v4845: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:51.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:50 smithi045 conmon[33655]: cluster 2021-07-31T03:21:49.150448+0000 mgr.x (mgr.24542 2021-07-31T03:21:51.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:50 smithi045 conmon[33655]: ) 4057 : cluster [DBG] pgmap v4845: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:53.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:52 smithi154 conmon[34553]: cluster 2021-07-31T03:21:51.150806+0000 mgr.x (mgr.24542) 4058 2021-07-31T03:21:53.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:52 smithi154 conmon[34553]: : cluster [DBG] pgmap v4846: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:53.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:52 smithi045 conmon[39718]: cluster 2021-07-31T03:21:51. 2021-07-31T03:21:53.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:52 smithi045 conmon[39718]: 150806+0000 mgr.x (mgr.24542) 4058 : cluster [DBG] pgmap v4846: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:53.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:52 smithi045 conmon[33655]: cluster 2021-07-31T03:21:51.150806+0000 mgr.x (mgr.24542) 4058 : 2021-07-31T03:21:53.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:52 smithi045 conmon[33655]: cluster [DBG] pgmap v4846: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:55.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:54 smithi154 conmon[34553]: cluster 2021-07-31T03:21:53.151184+0000 mgr.x (mgr.24542) 2021-07-31T03:21:55.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:54 smithi154 conmon[34553]: 4059 : cluster [DBG] pgmap v4847: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:55.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:54 smithi045 conmon[39718]: cluster 2021-07-31T03:21:53.151184+0000 mgr.x (mgr 2021-07-31T03:21:55.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:54 smithi045 conmon[39718]: .24542) 4059 : cluster [DBG] pgmap v4847: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:55.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:54 smithi045 conmon[33655]: cluster 2021-07-31T03:21:53.151184+0000 mgr.x (mgr.24542) 4059 : cluster [DBG] 2021-07-31T03:21:55.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:54 smithi045 conmon[33655]: pgmap v4847: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:57.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:56 smithi154 conmon[34553]: cluster 2021-07-31T03:21:55.151610+0000 mgr.x (mgr.24542) 4060 : cluster [DBG] pgmap v4848: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:57.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:56 smithi045 conmon[39718]: cluster 2021-07-31T03:21:55.151610+0000 mgr.x ( 2021-07-31T03:21:57.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:56 smithi045 conmon[39718]: mgr.24542) 4060 : cluster [DBG] pgmap v4848: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:57.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:56 smithi045 conmon[33655]: cluster 2021-07-31T03:21:55.151610+0000 mgr.x (mgr.24542) 4060 : 2021-07-31T03:21:57.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:56 smithi045 conmon[33655]: cluster [DBG] pgmap v4848: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:59.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:58 smithi154 conmon[34553]: cluster 2021-07-31T03:21:57.152000+0000 mgr.x (mgr.24542) 4061 : cluster [DBG] 2021-07-31T03:21:59.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:21:58 smithi154 conmon[34553]: pgmap v4849: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:59.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:58 smithi045 conmon[39718]: cluster 2021-07-31T03:21:57.152000+0000 mgr.x 2021-07-31T03:21:59.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:21:58 smithi045 conmon[39718]: (mgr.24542) 4061 : cluster [DBG] pgmap v4849: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:21:59.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:21:58 smithi045 conmon[33655]: cluster 2021-07-31T03:21:57.152000+0000 mgr.x (mgr.24542) 4061 : cluster [DBG] pgmap v4849: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:01.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:00 smithi154 conmon[34553]: cluster 2021-07-31T03:21:59.152383+0000 mgr.x (mgr.24542) 4062 : cluster [DBG] pgmap v4850: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:01.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:00 smithi154 conmon[34553]: 2021-07-31T03:22:01.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:00 smithi045 conmon[33655]: cluster 2021-07-31T03:21:59.152383+0000 mgr.x (mgr.24542) 4062 : cluster [DBG] pgmap v4850: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:01.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:00 smithi045 conmon[39718]: cluster 2021-07-31T03:21:59.152383+0000 mgr.x (mgr 2021-07-31T03:22:01.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:00 smithi045 conmon[39718]: .24542) 4062 : cluster [DBG] pgmap v4850: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:03.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:02 smithi154 conmon[34553]: cluster 2021-07-31T03:22:01.152817+0000 mgr.x (mgr.24542) 4063 : cluster [DBG] pgmap v4851: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:03.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:02 smithi045 conmon[33655]: cluster 2021-07-31T03:22:01.152817+0000 mgr.x (mgr.24542) 4063 : cluster [DBG] pgmap v4851: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:03.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:02 smithi045 conmon[39718]: cluster 2021-07-31T03:22:01.152817+0000 mgr.x (mgr. 2021-07-31T03:22:03.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:02 smithi045 conmon[39718]: 24542) 4063 : cluster [DBG] pgmap v4851: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:05.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:04 smithi154 conmon[34553]: cluster 2021-07-31T03:22:03.153378+0000 mgr.x (mgr.24542) 4064 : cluster [DBG] 2021-07-31T03:22:05.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:04 smithi154 conmon[34553]: pgmap v4852: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:05.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:04 smithi045 conmon[33655]: cluster 2021-07-31T03:22:03.153378+0000 mgr.x (mgr.24542) 4064 : cluster [DBG] 2021-07-31T03:22:05.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:04 smithi045 conmon[33655]: pgmap v4852: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:05.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:04 smithi045 conmon[39718]: cluster 2021-07-31T03:22:03.153378+0000 mgr.x (mgr 2021-07-31T03:22:05.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:04 smithi045 conmon[39718]: .24542) 4064 : cluster [DBG] pgmap v4852: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:07.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:06 smithi154 conmon[34553]: cluster 2021-07-31T03:22:05.153963+0000 mgr.x (mgr.24542) 4065 : cluster [DBG] pgmap v4853: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:07.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:06 smithi045 conmon[33655]: cluster 2021-07-31T03:22:05.153963+0000 mgr.x (mgr.24542) 4065 : cluster [DBG] 2021-07-31T03:22:07.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:06 smithi045 conmon[33655]: pgmap v4853: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:07.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:06 smithi045 conmon[39718]: cluster 2021-07-31T03:22:05.153963+0000 mgr.x (mgr 2021-07-31T03:22:07.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:06 smithi045 conmon[39718]: .24542) 4065 : cluster [DBG] pgmap v4853: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:09.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:08 smithi045 conmon[33655]: cluster 2021-07-31T03:22:07.154506+0000 mgr.x (mgr.24542 2021-07-31T03:22:09.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:08 smithi045 conmon[33655]: ) 4066 : cluster [DBG] pgmap v4854: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:09.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:08 smithi045 conmon[39718]: cluster 2021-07-31T03:22:07.154506+0000 mgr.x (mgr.24542) 4066 : cluster [DBG] pgmap v4854: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:09.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:08 smithi154 conmon[34553]: cluster 2021-07-31T03:22:07.154506+0000 mgr.x (mgr.24542) 4066 : cluster [DBG] pgmap v4854: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:11.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:10 smithi045 conmon[39718]: cluster 2021-07-31T03:22:09.154962+0000 mgr.x (mgr.24542) 4067 : cluster [DBG] pgmap v4855: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:11.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:10 smithi045 conmon[39718]: 2021-07-31T03:22:11.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:10 smithi045 conmon[33655]: cluster 2021-07-31T03:22:09.154962+0000 mgr.x (mgr.24542) 4067 : cluster 2021-07-31T03:22:11.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:10 smithi045 conmon[33655]: [DBG] pgmap v4855: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:11.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:10 smithi154 conmon[34553]: cluster 2021-07-31T03:22:09.154962+0000 mgr.x (mgr.24542) 4067 : cluster 2021-07-31T03:22:11.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:10 smithi154 conmon[34553]: [DBG] pgmap v4855: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:13.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:12 smithi045 conmon[39718]: cluster 2021-07-31T03:22:11.155399+0000 mgr.x ( 2021-07-31T03:22:13.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:12 smithi045 conmon[39718]: mgr.24542) 4068 : cluster [DBG] pgmap v4856: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:13.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:12 smithi045 conmon[33655]: cluster 2021-07-31T03:22:11.155399+0000 mgr.x 2021-07-31T03:22:13.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:12 smithi045 conmon[33655]: (mgr.24542) 4068 : cluster [DBG] pgmap v4856: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:13.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:12 smithi154 conmon[34553]: cluster 2021-07-31T03:22:11.155399+0000 mgr.x (mgr.24542) 4068 2021-07-31T03:22:13.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:12 smithi154 conmon[34553]: : cluster [DBG] pgmap v4856: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:15.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:14 smithi045 conmon[39718]: cluster 2021-07-31T03:22:13.155790+0000 mgr.x (mgr.24542) 4069 : cluster [DBG] 2021-07-31T03:22:15.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:14 smithi045 conmon[39718]: pgmap v4857: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:15.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:14 smithi045 conmon[33655]: cluster 2021-07-31T03:22:13. 2021-07-31T03:22:15.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:14 smithi045 conmon[33655]: 155790+0000 mgr.x (mgr.24542) 4069 : cluster [DBG] pgmap v4857: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:15.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:14 smithi154 conmon[34553]: cluster 2021-07-31T03:22:13.155790+0000 mgr.x (mgr.24542) 4069 : cluster [DBG] pgmap v4857: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:17.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:16 smithi045 conmon[39718]: cluster 2021-07-31T03:22:15.156394+0000 mgr.x 2021-07-31T03:22:17.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:16 smithi045 conmon[39718]: (mgr.24542) 4070 : cluster [DBG] pgmap v4858: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:17.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:16 smithi045 conmon[33655]: cluster 2021-07-31T03:22:15.156394+0000 mgr.x (mgr.24542 2021-07-31T03:22:17.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:16 smithi045 conmon[33655]: ) 4070 : cluster [DBG] pgmap v4858: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:17.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:16 smithi154 conmon[34553]: cluster 2021-07-31T03:22:15.156394+0000 mgr.x (mgr.24542) 4070 : cluster [DBG] pgmap v4858: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:18.376 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:22:18.651 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:22:18.651 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 97m ago 97m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:22:18.651 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 94m ago 97m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:22:18.651 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 94m ago 101m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:22:18.652 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 97m ago 103m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:22:18.652 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 97m ago 103m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:22:18.652 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 94m ago 101m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:22:18.652 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 97m ago 101m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:22:18.652 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 97m ago 98m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:22:18.653 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 94m ago 98m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:22:18.653 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 97m ago 100m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:22:18.653 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 97m ago 100m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:22:18.653 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 97m ago 100m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:22:18.653 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 97m ago 99m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:22:18.654 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 94m ago 99m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:22:18.654 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 94m ago 99m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:22:18.654 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 94m ago 98m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:22:18.654 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 94m ago 98m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:22:18.654 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 94m ago 98m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:22:18.987 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:22:18.987 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:22:18.988 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:22:18.988 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:22:18.988 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:22:18.988 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:22:18.988 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:22:18.989 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:22:18.989 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:22:18.989 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:22:18.989 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:22:18.990 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:22:18.990 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:22:18.990 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:22:18.990 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:22:18.991 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:22:18.991 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:22:19.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:18 smithi045 conmon[39718]: cluster 2021-07-31T03:22:17.156895+0000 mgr.x (mgr.24542) 4071 : cluster [DBG] 2021-07-31T03:22:19.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:18 smithi045 conmon[39718]: pgmap v4859: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:19.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:18 smithi045 conmon[33655]: cluster 2021-07-31T03:22:17 2021-07-31T03:22:19.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:18 smithi045 conmon[33655]: .156895+0000 mgr.x (mgr.24542) 4071 : cluster [DBG] pgmap v4859: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:19.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:18 smithi154 conmon[34553]: cluster 2021-07-31T03:22:17.156895+0000 mgr.x (mgr.24542) 4071 : cluster 2021-07-31T03:22:19.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:18 smithi154 conmon[34553]: [DBG] pgmap v4859: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:20.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:19 smithi154 conmon[34553]: audit 2021-07-31T03:22:18.365846+0000 mgr.x (mgr.24542) 4072 : audit [DBG] from='client.29302 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:22:20.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:19 smithi154 conmon[34553]: audit 2021-07-31T03:22:18.643759+0000 mgr.x (mgr.24542) 4073 : audit [DBG] from='client.29308 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:22:20.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:19 smithi154 conmon[34553]: audit 2021-07-31T03:22:18.988110+0000 mon.c (mon. 2021-07-31T03:22:20.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:19 smithi154 conmon[34553]: 1) 197 : audit [DBG] from='client.? 172.21.15.45:0/86861285' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:22:20.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:19 smithi045 conmon[33655]: audit 2021-07-31T03:22: 2021-07-31T03:22:20.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:19 smithi045 conmon[33655]: 18.365846+0000 mgr.x (mgr.24542) 4072 : audit [DBG] from='client.29302 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:22:20.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:19 smithi045 conmon[33655]: audit 2021-07-31T03:22:20.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:19 smithi045 conmon[33655]: 2021-07-31T03:22:18.643759 2021-07-31T03:22:20.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:19 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 4073 : audit [DBG] from='client.29308 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:22:20.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:19 smithi045 conmon[33655]: audit 2021-07-31T03 2021-07-31T03:22:20.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:19 smithi045 conmon[33655]: :22:18.988110+0000 mon.c (mon.1) 197 : audit [DBG] from='client.? 172.21.15.45:0/86861285' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:22:20.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:19 smithi045 conmon[39718]: audit 2021-07-31T03:22:18.365846+0000 mgr.x (mgr.24542) 4072 : audit [DBG] from='client.29302 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:22:20.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:19 smithi045 conmon[39718]: audit 2021-07-31T03:22: 2021-07-31T03:22:20.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:19 smithi045 conmon[39718]: 18.643759+0000 mgr.x (mgr.24542 2021-07-31T03:22:20.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:19 smithi045 conmon[39718]: ) 4073 : audit [DBG] from='client.29308 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:22:20.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:19 smithi045 conmon[39718]: 2021-07-31T03:22:20.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:19 smithi045 conmon[39718]: audit 2021- 2021-07-31T03:22:20.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:19 smithi045 conmon[39718]: 07-31T03:22: 2021-07-31T03:22:20.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:19 smithi045 conmon[39718]: 18.988110+0000 mon.c (mon 2021-07-31T03:22:20.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:19 smithi045 conmon[39718]: .1) 197 : audit [DBG] 2021-07-31T03:22:20.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:19 smithi045 conmon[39718]: from='client.? 172.21.15.45:0/86861285' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:22:21.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:20 smithi154 conmon[34553]: cluster 2021-07-31T03:22:19.157372+0000 mgr.x (mgr.24542) 4074 : cluster [DBG] pgmap v4860: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:21.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:20 smithi045 conmon[33655]: cluster 2021-07-31T03:22:21.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:20 smithi045 conmon[33655]: 2021-07-31T03:22:19.157372+0000 mgr.x (mgr.24542) 4074 : cluster [DBG] 2021-07-31T03:22:21.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:20 smithi045 conmon[33655]: pgmap v4860: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:21.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:20 smithi045 conmon[39718]: cluster 2021-07-31T03:22:19.157372+0000 mgr.x (mgr.24542) 4074 : 2021-07-31T03:22:21.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:20 smithi045 conmon[39718]: cluster [DBG] pgmap v4860: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:23.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:22 smithi154 conmon[34553]: cluster 2021-07-31T03:22:21.157865+0000 mgr.x (mgr.24542) 4075 : cluster 2021-07-31T03:22:23.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:22 smithi154 conmon[34553]: [DBG] pgmap v4861: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:23.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:22 smithi045 conmon[33655]: cluster 2021-07-31T03: 2021-07-31T03:22:23.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:22 smithi045 conmon[33655]: 22:21.157865+0000 mgr.x (mgr.24542) 4075 : cluster [DBG] pgmap v4861: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:23.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:22 smithi045 conmon[39718]: cluster 2021-07-31T03:22:21.157865+0000 mgr.x (mgr.24542) 4075 : cluster [DBG] pgmap v4861: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:25.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:24 smithi154 conmon[34553]: cluster 2021-07-31T03:22:23.158363+0000 mgr.x (mgr.24542) 4076 : cluster [DBG] 2021-07-31T03:22:25.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:24 smithi154 conmon[34553]: pgmap v4862: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:25.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:24 smithi045 conmon[39718]: cluster 2021-07-31T03:22:23 2021-07-31T03:22:25.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:24 smithi045 conmon[39718]: .158363+0000 mgr.x (mgr.24542) 4076 : cluster [DBG] pgmap v4862: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:25.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:24 smithi045 conmon[33655]: cluster 2021-07-31T03:22:23.158363+0000 mgr.x (mgr.24542) 4076 : cluster [DBG] pgmap v4862: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:27.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:26 smithi154 conmon[34553]: cluster 2021-07-31T03:22:25.158907+0000 mgr.x (mgr.24542) 4077 : cluster [DBG] 2021-07-31T03:22:27.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:26 smithi154 conmon[34553]: pgmap v4863: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:27.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:26 smithi045 conmon[39718]: cluster 2021-07-31T03:22: 2021-07-31T03:22:27.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:26 smithi045 conmon[39718]: 25.158907+0000 mgr.x (mgr.24542) 4077 : cluster [DBG] pgmap v4863: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:27.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:26 smithi045 conmon[33655]: cluster 2021-07-31T03:22:25.158907+0000 mgr.x (mgr.24542) 4077 2021-07-31T03:22:27.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:26 smithi045 conmon[33655]: : cluster [DBG] pgmap v4863: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:29.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:28 smithi154 conmon[34553]: cluster 2021-07-31T03:22:27.159352+0000 mgr.x (mgr.24542) 4078 : cluster [DBG] pgmap v4864: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:29.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:28 smithi154 conmon[34553]: audit 2021-07-31T03:22:28.311669 2021-07-31T03:22:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:28 smithi154 conmon[34553]: +0000 mon.b (mon.2) 550 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:22:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:28 smithi154 conmon[34553]: audit 2021-07-31T03:22:28.311930+0000 mon.b (mon.2) 551 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:22:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:28 smithi154 conmon[34553]: audit 2021-07-31T03:22:28.312434+0000 mon.a (mon.0) 3295 : 2021-07-31T03:22:29.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:28 smithi154 conmon[34553]: audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:22:29.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:28 smithi154 conmon[34553]: audit 2021-07-31T03:22 2021-07-31T03:22:29.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:28 smithi154 conmon[34553]: :28.312810+0000 mon.a (mon.0) 3296 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:22:29.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:28 smithi045 conmon[39718]: cluster 2021-07-31T03:22:27.159352+0000 mgr.x (mgr.24542) 4078 : cluster [DBG] pgmap v4864: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:29.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:28 smithi045 conmon[39718]: audit 2021-07-31T03:22:28.311669+0000 mon.b (mon.2) 550 : audit 2021-07-31T03:22:29.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:28 smithi045 conmon[39718]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:22:29.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:28 smithi045 conmon[39718]: audit 2021-07-31T03 2021-07-31T03:22:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:28 smithi045 conmon[39718]: :22:28.311930+0000 mon.b (mon. 2021-07-31T03:22:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:28 smithi045 conmon[39718]: 2) 551 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:22:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:28 smithi045 conmon[39718]: audit 2021- 2021-07-31T03:22:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:28 smithi045 conmon[39718]: 07-31T03:22:28.312434+0000 2021-07-31T03:22:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:28 smithi045 conmon[39718]: mon.a (mon.0) 3295 : audit [INF] 2021-07-31T03:22:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:28 smithi045 conmon[39718]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:22:29.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:28 smithi045 conmon[39718]: audit 2021 2021-07-31T03:22:29.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:28 smithi045 conmon[39718]: -07-31T03:22:28. 2021-07-31T03:22:29.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:28 smithi045 conmon[39718]: 312810+0000 mon.a (mon.0) 3296 : 2021-07-31T03:22:29.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:28 smithi045 conmon[39718]: audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:22:29.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:28 smithi045 conmon[33655]: cluster 2021-07-31T03:22:27.159352+0000 mgr.x (mgr.24542) 4078 : cluster 2021-07-31T03:22:29.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:28 smithi045 conmon[33655]: [DBG] pgmap v4864: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:29.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:28 smithi045 conmon[33655]: audit 2021-07-31T03:22:28.311669+0000 mon.b (mon.2) 550 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:22:29.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:28 smithi045 conmon[33655]: 2021-07-31T03:22:29.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:28 smithi045 conmon[33655]: audit 2021-07-31T03:22:29.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:28 smithi045 conmon[33655]: 2021-07-31T03:22:28. 2021-07-31T03:22:29.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:28 smithi045 conmon[33655]: 311930+0000 mon.b (mon.2) 551 2021-07-31T03:22:29.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:28 smithi045 conmon[33655]: : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:22:29.283 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:28 smithi045 conmon[33655]: audit 2021 2021-07-31T03:22:29.283 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:28 smithi045 conmon[33655]: -07-31T03:22: 2021-07-31T03:22:29.283 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:28 smithi045 conmon[33655]: 28.312434+0000 mon.a (mon 2021-07-31T03:22:29.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:28 smithi045 conmon[33655]: .0) 3295 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:22:29.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:28 smithi045 conmon[33655]: 2021-07-31T03:22:29.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:28 smithi045 conmon[33655]: audit 2021-07 2021-07-31T03:22:29.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:28 smithi045 conmon[33655]: -31T03:22:28. 2021-07-31T03:22:29.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:28 smithi045 conmon[33655]: 312810+0000 mon.a (mon.0) 3296 2021-07-31T03:22:29.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:28 smithi045 conmon[33655]: : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:22:31.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:30 smithi154 conmon[34553]: cluster 2021-07-31T03:22:29.159775+0000 mgr.x (mgr.24542) 4079 : cluster [DBG] pgmap v4865: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:31.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:30 smithi154 conmon[34553]: 2021-07-31T03:22:31.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:30 smithi045 conmon[39718]: cluster 2021-07-31T03:22:29.159775+0000 mgr.x (mgr.24542 2021-07-31T03:22:31.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:30 smithi045 conmon[39718]: ) 4079 : cluster [DBG] pgmap v4865: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:31.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:30 smithi045 conmon[33655]: cluster 2021-07-31T03:22:29.159775+0000 mgr.x (mgr.24542 2021-07-31T03:22:31.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:30 smithi045 conmon[33655]: ) 4079 : cluster [DBG] pgmap v4865: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:33.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[34553]: cluster 2021-07-31T03:22:31.160204+0000 mgr.x (mgr.24542) 4080 : cluster [DBG] pgmap v4866: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:33.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[34553]: 2021-07-31T03:22:33.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[34553]: audit 2021-07-31T03:22:32.781064+0000 mon.b (mon.2) 552 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:22:33.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[34553]: audit 2021-07-31T03:22:32.782604+0000 mon.b (mon.2) 553 : audit [DBG] 2021-07-31T03:22:33.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[34553]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:22:33.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[34553]: audit 2021-07-31T03:22:32.783716+0000 mon.b (mon.2) 554 : 2021-07-31T03:22:33.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[34553]: audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:22:33.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: debug 2021-07-31T03:22:32.827+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:22:33.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:22:33.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:22:33.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: yield 2021-07-31T03:22:33.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:22:33.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:22:33.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:22:33.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:22:33.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:22:33.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:22:33.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:22:33.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:22:33.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:22:33.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: task.result() 2021-07-31T03:22:33.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:22:33.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: debug 2021-07-31T03:22:32.839+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:22:33.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:22:33.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:22:33.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: yield 2021-07-31T03:22:33.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:22:33.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:22:33.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:22:33.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:22:33.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:22:33.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:22:33.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:22:33.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:22:33.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:22:33.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: task.result() 2021-07-31T03:22:33.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:22:33.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: debug 2021-07-31T03:22:32.864+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:22:33.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:22:33.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:22:33.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: yield 2021-07-31T03:22:33.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:22:33.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:22:33.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:22:33.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:22:33.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:22:33.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:22:33.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:22:33.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:22:33.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:22:33.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: task.result() 2021-07-31T03:22:33.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:22:33.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: debug 2021-07-31T03:22:32.864+0000 7fcb94162700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:22:33.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:22:33.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:22:33.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: yield 2021-07-31T03:22:33.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:22:33.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:22:33.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:22:33.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:22:33.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:22:33.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:22:33.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:22:33.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:22:33.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:22:33.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: task.result() 2021-07-31T03:22:33.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:22:33.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: 2021-07-31T03:22:33.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:22:33.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: 2021-07-31T03:22:33.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:22:33.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:22:33.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:22:33.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:22:33.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:22:33.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:22:33.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:22:33.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:22:33.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:22:33.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:22:33.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:22:33.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:22:33.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:22:33.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:22:33.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:22:33.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:22:33.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:22:33.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:22:33.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: raise 2021-07-31T03:22:33.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:22:33.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:22:33.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:22:33.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:22:33.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:22:33.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: debug 2021-07-31T03:22:32.873+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:22:33.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:22:33.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:22:33.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: yield 2021-07-31T03:22:33.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:22:33.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:22:33.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:22:33.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:22:33.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:22:33.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:22:33.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:22:33.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:22:33.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:22:33.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: task.result() 2021-07-31T03:22:33.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:22:33.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: debug 2021-07-31T03:22:32.873+0000 7fcb93961700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:22:33.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:22:33.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:22:33.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: yield 2021-07-31T03:22:33.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:22:33.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:22:33.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:22:33.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:22:33.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:22:33.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:22:33.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:22:33.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:22:33.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:22:33.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: task.result() 2021-07-31T03:22:33.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:22:33.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: 2021-07-31T03:22:33.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:22:33.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: 2021-07-31T03:22:33.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:22:33.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:22:33.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:22:33.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:22:33.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:22:33.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:22:33.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:22:33.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:22:33.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:22:33.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:22:33.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:22:33.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:22:33.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:22:33.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:22:33.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:22:33.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:22:33.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:22:33.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:22:33.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: raise 2021-07-31T03:22:33.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:22:33.284 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:22:33.284 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:22:33.284 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:22:33.284 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:22:32 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:22:33.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:32 smithi045 conmon[33655]: cluster 2021-07-31T03:22:31.160204+0000 mgr.x (mgr.24542) 4080 : cluster 2021-07-31T03:22:33.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:32 smithi045 conmon[33655]: [DBG] pgmap v4866: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:33.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:32 smithi045 conmon[33655]: audit 2021-07-31T03:22:32.781064+0000 mon.b (mon.2) 552 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:22:33.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:32 smithi045 conmon[33655]: audit 2021-07-31T03:22:33.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:32 smithi045 conmon[33655]: 2021-07-31T03:22:32.782604+0000 mon.b (mon.2) 553 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:22:33.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:32 smithi045 conmon[33655]: audit 2021-07 2021-07-31T03:22:33.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:32 smithi045 conmon[33655]: -31T03:22:32.783716+0000 mon.b (mon 2021-07-31T03:22:33.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:32 smithi045 conmon[33655]: .2) 554 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:22:33.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:32 smithi045 conmon[39718]: cluster 2021-07-31T03:22:31.160204+0000 mgr.x (mgr. 2021-07-31T03:22:33.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:32 smithi045 conmon[39718]: 24542) 4080 : cluster [DBG] pgmap v4866: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:33.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:32 smithi045 conmon[39718]: audit 2021-07-31T03: 2021-07-31T03:22:33.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:32 smithi045 conmon[39718]: 22:32.781064+0000 mon.b (mon.2) 552 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:22:33.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:32 smithi045 conmon[39718]: audit 2021 2021-07-31T03:22:33.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:32 smithi045 conmon[39718]: -07-31T03:22:32.782604+0000 mon.b ( 2021-07-31T03:22:33.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:32 smithi045 conmon[39718]: mon.2) 553 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:22:33.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:32 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T03:22:33.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:32 smithi045 conmon[39718]: 03:22:32.783716+0000 mon.b (mon.2) 554 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:22:33.289 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:32 smithi045 conmon[39718]: 2021-07-31T03:22:34.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:22:32.784484+0000 mgr.x (mgr.24542) 4081 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:22:34.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:22:32.828583+0000 mgr.x (mgr.24542) 4082 : cephadm [ERR] 2021-07-31T03:22:34.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:22:34.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:22:34.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: yield 2021-07-31T03:22:34.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:22:34.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:22:34.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:22:34.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:22:34.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:22:34.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:22:34.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:22:34.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:22:34.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:22:34.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: task.result() 2021-07-31T03:22:34.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:22:34.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:22:32.829004+0000 mgr.x ( 2021-07-31T03:22:34.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: mgr.24542) 4083 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:22:34.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:22:32.840033+0000 mgr.x (mgr.24542) 4084 : cephadm [ERR] 2021-07-31T03:22:34.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:22:34.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:22:34.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: yield 2021-07-31T03:22:34.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:22:34.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:22:34.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:22:34.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:22:34.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:22:34.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:22:34.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:22:34.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:22:34.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:22:34.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: task.result() 2021-07-31T03:22:34.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:22:34.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:22:32.840990+0000 mgr.x ( 2021-07-31T03:22:34.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: mgr.24542) 4085 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:22:34.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:22:32.865057+0000 mgr.x (mgr.24542) 4086 : cephadm [ERR] 2021-07-31T03:22:34.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:22:34.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:22:34.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: yield 2021-07-31T03:22:34.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:22:34.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:22:34.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:22:34.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:22:34.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:22:34.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:22:34.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:22:34.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:22:34.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:22:34.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: task.result() 2021-07-31T03:22:34.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:22:34.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:22:32.865446+0000 mgr.x (mgr. 2021-07-31T03:22:34.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: 24542) 4087 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:22:34.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:22:34.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:22:34.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: yield 2021-07-31T03:22:34.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:22:34.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:22:34.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:22:34.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:22:34.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:22:34.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:22:34.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:22:34.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:22:34.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:22:34.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: task.result() 2021-07-31T03:22:34.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:22:34.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: 2021-07-31T03:22:34.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:22:34.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: 2021-07-31T03:22:34.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:22:34.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:22:34.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:22:34.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:22:34.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:22:34.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:22:34.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:22:34.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:22:34.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:22:34.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:22:34.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:22:34.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:22:34.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:22:34.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:22:34.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:22:34.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:22:34.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:22:34.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:22:34.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: raise 2021-07-31T03:22:34.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:22:34.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:22:34.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:22:34.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:22:34.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:22:34.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:22:32.873989+0000 mgr.x (mgr.24542) 4088 : cephadm [ERR] 2021-07-31T03:22:34.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:22:34.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:22:34.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: yield 2021-07-31T03:22:34.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:22:34.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:22:34.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:22:34.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:22:34.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:22:34.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:22:34.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:22:34.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:22:34.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:22:34.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: task.result() 2021-07-31T03:22:34.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:22:34.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: cephadm 2021-07- 2021-07-31T03:22:34.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: 31T03:22:32.874376+0000 mgr.x (mgr.24542) 4089 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:22:34.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:22:34.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:22:34.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: yield 2021-07-31T03:22:34.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:22:34.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:22:34.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:22:34.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:22:34.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:22:34.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:22:34.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:22:34.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:22:34.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:22:34.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: task.result() 2021-07-31T03:22:34.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:22:34.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: 2021-07-31T03:22:34.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:22:34.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: 2021-07-31T03:22:34.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:22:34.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:22:34.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:22:34.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:22:34.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:22:34.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:22:34.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:22:34.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:22:34.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:22:34.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:22:34.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:22:34.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:22:34.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:22:34.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:22:34.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:22:34.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:22:34.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:22:34.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:22:34.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: raise 2021-07-31T03:22:34.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:22:34.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:22:34.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:22:34.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:22:34.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:33 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:22:34.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: cephadm 2021-07-31T03:22:32.784484+0000 mgr.x (mgr.24542) 4081 2021-07-31T03:22:34.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:22:34.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: cephadm 2021-07-31T03:22:32.828583+0000 mgr.x (mgr.24542) 4082 : cephadm [ERR] 2021-07-31T03:22:34.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:22:34.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:22:34.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: yield 2021-07-31T03:22:34.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:22:34.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:22:34.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:22:34.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:22:34.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:22:34.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:22:34.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:22:34.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:22:34.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:22:34.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: task.result() 2021-07-31T03:22:34.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:22:34.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: cephadm 2021-07-31 2021-07-31T03:22:34.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: T03:22:32.829004+0000 mgr.x (mgr.24542) 4083 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:22:34.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: cephadm 2021-07-31T03:22:32.840033+0000 mgr.x (mgr.24542) 4084 : 2021-07-31T03:22:34.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: cephadm [ERR] 2021-07-31T03:22:34.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:22:34.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:22:34.290 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: yield 2021-07-31T03:22:34.290 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:22:34.290 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:22:34.290 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:22:34.290 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:22:34.291 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:22:34.291 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:22:34.291 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:22:34.291 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:22:34.291 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:22:34.292 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: task.result() 2021-07-31T03:22:34.292 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:22:34.292 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: cephadm 2021-07-31T03:22:32.840990+0000 mgr.x (mgr.24542) 4085 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:22:34.292 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: cephadm 2021-07-31T03:22:34.292 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: 2021-07-31T03:22 2021-07-31T03:22:34.293 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: :32.865057+0000 mgr.x ( 2021-07-31T03:22:34.293 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: mgr.24542) 4086 : cephadm [ERR] 2021-07-31T03:22:34.293 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: 2021-07-31T03:22:34.293 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: cephadm 2021-07-31T03:22: 2021-07-31T03:22:34.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: 32.784484+0000 mgr.x (mgr.24542) 4081 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:22:34.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: cephadm 2021 2021-07-31T03:22:34.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: -07-31T03:22:32. 2021-07-31T03:22:34.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: 828583+0000 mgr.x (mgr.24542 2021-07-31T03:22:34.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: ) 4082 : cephadm [ERR] 2021-07-31T03:22:34.295 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:22:34.295 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:22:34.295 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: yield 2021-07-31T03:22:34.295 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:22:34.295 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:22:34.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:22:34.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:22:34.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:22:34.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:22:34.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:22:34.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:22:34.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:22:34.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: task.result() 2021-07-31T03:22:34.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:22:34.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: 2021-07-31T03:22:34.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: cephadm 2021-07-31T03:22:32.829004+0000 mgr.x (mgr.24542) 4083 : cephadm [INF] 2021-07-31T03:22:34.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:22:34.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: cephadm 2021-07 2021-07-31T03:22:34.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: -31T03:22:32.840033+0000 mgr.x (mgr.24542) 2021-07-31T03:22:34.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: 4084 : cephadm [ERR] 2021-07-31T03:22:34.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:22:34.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:22:34.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: yield 2021-07-31T03:22:34.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:22:34.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:22:34.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:22:34.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:22:34.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:22:34.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:22:34.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:22:34.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:22:34.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:22:34.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: task.result() 2021-07-31T03:22:34.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:22:34.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: cephadm 2021-07-31T03:22:32.840990+0000 mgr.x (mgr.24542) 4085 : cephadm 2021-07-31T03:22:34.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:22:34.302 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:22:34.302 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:22:34.303 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: yield 2021-07-31T03:22:34.303 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:22:34.303 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:22:34.303 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:22:34.303 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:22:34.304 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:22:34.304 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:22:34.304 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:22:34.304 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:22:34.304 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:22:34.305 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: task.result() 2021-07-31T03:22:34.305 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:22:34.305 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: cephadm 2021-07-31T03:22:32.865446+0000 mgr.x 2021-07-31T03:22:34.305 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: (mgr.24542) 4087 : cephadm 2021-07-31T03:22:34.305 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:22:34.306 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:22:34.306 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:22:34.306 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: yield 2021-07-31T03:22:34.306 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:22:34.306 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:22:34.307 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:22:34.307 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:22:34.307 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:22:34.307 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:22:34.307 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:22:34.308 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:22:34.308 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:22:34.308 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: task.result() 2021-07-31T03:22:34.308 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:22:34.308 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: 2021-07-31T03:22:34.309 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:22:34.309 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: 2021-07-31T03:22:34.309 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:22:34.309 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:22:34.309 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:22:34.310 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:22:34.310 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:22:34.310 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:22:34.310 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:22:34.310 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:22:34.311 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: cephadm 2021- 2021-07-31T03:22:34.311 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: 07-31T03:22:32.865057 2021-07-31T03:22:34.311 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 2021-07-31T03:22:34.311 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: 4086 : cephadm [ERR] 2021-07-31T03:22:34.312 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:22:34.312 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:22:34.312 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: yield 2021-07-31T03:22:34.312 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:22:34.312 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:22:34.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:22:34.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:22:34.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:22:34.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:22:34.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:22:34.314 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:22:34.314 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:22:34.314 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: task.result() 2021-07-31T03:22:34.314 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:22:34.314 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: cephadm 2021-07-31T03:22:32.865446+0000 mgr.x (mgr.24542 2021-07-31T03:22:34.315 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: ) 4087 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:22:34.315 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:22:34.315 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:22:34.315 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: yield 2021-07-31T03:22:34.315 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:22:34.316 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:22:34.316 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:22:34.316 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:22:34.316 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:22:34.316 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:22:34.317 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:22:34.317 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:22:34.317 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:22:34.317 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: task.result() 2021-07-31T03:22:34.317 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:22:34.318 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: 2021-07-31T03:22:34.318 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:22:34.318 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: 2021-07-31T03:22:34.318 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:22:34.318 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:22:34.319 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:22:34.319 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:22:34.319 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:22:34.319 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:22:34.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:22:34.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:22:34.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:22:34.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:22:34.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:22:34.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:22:34.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:22:34.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:22:34.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:22:34.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:22:34.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: raise 2021-07-31T03:22:34.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:22:34.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:22:34.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:22:34.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:22:34.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:22:34.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: cephadm 2021-07-31T03:22:32.873989+0000 mgr.x (mgr.24542) 4088 : cephadm [ERR] 2021-07-31T03:22:34.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:22:34.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:22:34.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: yield 2021-07-31T03:22:34.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:22:34.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:22:34.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:22:34.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:22:34.325 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:22:34.325 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:22:34.325 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:22:34.325 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:22:34.325 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:22:34.326 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: task.result() 2021-07-31T03:22:34.326 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:22:34.326 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: cephadm 2021-07-31T03:22:32.874376+0000 mgr.x (mgr.24542) 4089 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:22:34.326 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:22:34.326 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:22:34.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: yield 2021-07-31T03:22:34.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:22:34.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:22:34.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:22:34.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:22:34.328 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:22:34.328 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:22:34.329 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:22:34.329 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:22:34.329 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:22:34.329 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:22:34.329 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:22:34.330 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:22:34.330 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:22:34.330 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:22:34.330 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:22:34.330 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:22:34.331 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: raise 2021-07-31T03:22:34.331 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:22:34.331 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:22:34.331 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:22:34.331 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:22:34.332 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:22:34.332 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: 2021-07-31T03:22:34.332 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: cephadm 2021-07-31T03:22:32.873989+0000 mgr.x (mgr.24542) 4088 : cephadm [ERR] 2021-07-31T03:22:34.332 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:22:34.332 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:22:34.333 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: yield 2021-07-31T03:22:34.333 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:22:34.333 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:22:34.333 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:22:34.333 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:22:34.334 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:22:34.334 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:22:34.334 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:22:34.334 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:22:34.334 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:22:34.335 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: task.result() 2021-07-31T03:22:34.335 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:22:34.335 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: cephadm 2021-07-31T03:22:32.874376+0000 mgr.x (mgr.24542) 4089 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:22:34.335 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:22:34.335 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:22:34.336 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: yield 2021-07-31T03:22:34.336 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:22:34.336 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:22:34.337 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:22:34.337 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:22:34.337 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:22:34.337 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:22:34.337 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: task.result() 2021-07-31T03:22:34.338 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:22:34.338 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: 2021-07-31T03:22:34.338 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:22:34.338 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: 2021-07-31T03:22:34.339 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:22:34.339 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:22:34.339 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:22:34.339 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:22:34.340 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:22:34.340 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:22:34.340 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:22:34.340 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:22:34.340 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:22:34.341 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:22:34.341 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:22:34.341 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:22:34.341 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:22:34.341 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:22:34.342 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:22:34.342 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:22:34.342 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:22:34.342 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:22:34.343 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: raise 2021-07-31T03:22:34.343 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:22:34.343 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:22:34.343 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:22:34.344 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:22:34.344 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:22:34.344 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:22:34.344 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:22:34.345 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:22:34.345 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:22:34.345 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:22:34.345 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:22:34.346 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:22:34.346 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:22:34.346 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: task.result() 2021-07-31T03:22:34.346 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:22:34.347 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: 2021-07-31T03:22:34.347 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:22:34.347 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: 2021-07-31T03:22:34.347 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:22:34.347 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:22:34.348 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:22:34.348 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:22:34.348 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:22:34.348 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:22:34.349 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:22:34.349 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:22:34.349 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:22:34.349 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:22:34.349 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:22:34.350 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:22:34.350 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:22:34.350 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:22:34.350 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:22:34.350 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:22:34.351 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:22:34.351 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:22:34.351 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: raise 2021-07-31T03:22:34.351 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:22:34.351 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:22:34.352 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:22:34.352 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:22:34.352 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:33 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:22:35.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:34 smithi154 conmon[34553]: cluster 2021-07-31T03:22:33.160587+0000 mgr.x (mgr.24542) 4090 : cluster [DBG] pgmap v4867: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:35.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:34 smithi045 conmon[39718]: cluster 2021-07-31T03:22:33.160587+0000 mgr.x (mgr.24542) 4090 : cluster [DBG] pgmap v4867: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:35.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:34 smithi045 conmon[33655]: cluster 2021-07-31T03:22:33.160587+0000 mgr.x (mgr.24542) 4090 : 2021-07-31T03:22:35.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:34 smithi045 conmon[33655]: cluster [DBG] pgmap v4867: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:37.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:36 smithi154 conmon[34553]: cluster 2021-07-31T03:22:35.161027+0000 mgr.x (mgr.24542) 4091 : cluster [DBG] pgmap v4868: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:37.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:36 smithi045 conmon[33655]: cluster 2021-07-31T03:22:35.161027+0000 mgr.x (mgr.24542) 4091 : 2021-07-31T03:22:37.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:36 smithi045 conmon[33655]: cluster [DBG] pgmap v4868: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:37.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:36 smithi045 conmon[39718]: cluster 2021-07-31T03:22:35.161027+0000 mgr.x (mgr.24542) 4091 : cluster [DBG] pgmap v4868: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:39.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:38 smithi154 conmon[34553]: cluster 2021-07-31T03:22:37.161541+0000 mgr.x (mgr.24542) 4092 : cluster [DBG] pgmap v4869: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:39.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:38 smithi045 conmon[33655]: cluster 2021-07-31T03:22:37.161541+0000 mgr.x (mgr.24542) 2021-07-31T03:22:39.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:38 smithi045 conmon[33655]: 4092 : cluster [DBG] pgmap v4869: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:39.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:38 smithi045 conmon[39718]: cluster 2021-07-31T03:22:37.161541+0000 mgr.x (mgr.24542 2021-07-31T03:22:39.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:38 smithi045 conmon[39718]: ) 4092 : cluster [DBG] pgmap v4869: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:41.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:40 smithi154 conmon[34553]: cluster 2021-07-31T03:22:39.162047+0000 mgr.x (mgr.24542) 4093 : cluster [DBG] pgmap v4870: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:41.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:40 smithi045 conmon[39718]: cluster 2021-07-31T03:22:39.162047+0000 mgr.x 2021-07-31T03:22:41.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:40 smithi045 conmon[39718]: (mgr.24542) 4093 : cluster [DBG] pgmap v4870: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:41.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:40 smithi045 conmon[33655]: cluster 2021-07-31T03:22:39.162047+0000 mgr.x (mgr.24542) 4093 : cluster [DBG] pgmap v4870: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:43.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:42 smithi154 conmon[34553]: cluster 2021-07-31T03:22:41.162517+0000 mgr.x (mgr.24542) 4094 : cluster [DBG] pgmap v4871: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:43.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:42 smithi045 conmon[33655]: cluster 2021-07-31T03:22:41.162517+0000 mgr.x (mgr.24542) 4094 : cluster 2021-07-31T03:22:43.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:42 smithi045 conmon[33655]: [DBG] pgmap v4871: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:43.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:42 smithi045 conmon[39718]: cluster 2021-07-31T03:22:41.162517+0000 mgr.x (mgr.24542 2021-07-31T03:22:43.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:42 smithi045 conmon[39718]: ) 4094 : cluster [DBG] pgmap v4871: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:45.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:44 smithi154 conmon[34553]: cluster 2021-07-31T03:22:43.163014+0000 mgr.x (mgr.24542) 4095 : cluster [DBG] pgmap v4872: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:45.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:44 smithi045 conmon[39718]: cluster 2021-07-31T03:22:43.163014+0000 mgr.x (mgr 2021-07-31T03:22:45.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:44 smithi045 conmon[39718]: .24542) 4095 : cluster [DBG] pgmap v4872: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:45.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:44 smithi045 conmon[33655]: cluster 2021-07-31T03:22:43.163014+0000 mgr.x (mgr.24542) 4095 : cluster [DBG] pgmap v4872: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:47.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:46 smithi154 conmon[34553]: cluster 2021-07-31T03:22:45.163434+0000 mgr.x (mgr.24542) 4096 : cluster [DBG] pgmap v4873: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:47.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:46 smithi045 conmon[39718]: cluster 2021-07-31T03:22:45.163434+0000 mgr.x (mgr 2021-07-31T03:22:47.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:46 smithi045 conmon[39718]: .24542) 4096 : cluster [DBG] pgmap v4873: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:47.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:46 smithi045 conmon[33655]: cluster 2021-07-31T03:22:45.163434+0000 mgr.x (mgr. 2021-07-31T03:22:47.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:46 smithi045 conmon[33655]: 24542) 4096 : cluster [DBG] pgmap v4873: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:49.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:48 smithi154 conmon[34553]: cluster 2021-07-31T03:22:47.163846+0000 mgr.x (mgr.24542) 4097 : cluster [DBG] pgmap v4874: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:49.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:48 smithi045 conmon[33655]: cluster 2021-07-31T03:22:47.163846+0000 mgr.x (mgr.24542 2021-07-31T03:22:49.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:48 smithi045 conmon[33655]: ) 4097 : cluster [DBG] pgmap v4874: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:49.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:48 smithi045 conmon[39718]: cluster 2021-07-31T03:22:47.163846+0000 mgr.x (mgr.24542) 4097 : cluster [DBG] 2021-07-31T03:22:49.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:48 smithi045 conmon[39718]: pgmap v4874: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:49.299 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:22:49.577 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:22:49.577 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 97m ago 98m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:22:49.578 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 95m ago 98m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:22:49.578 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 95m ago 101m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:22:49.578 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 97m ago 103m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:22:49.578 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 97m ago 103m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:22:49.579 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 95m ago 102m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:22:49.579 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 97m ago 102m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:22:49.579 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 97m ago 98m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:22:49.579 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 95m ago 98m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:22:49.579 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 97m ago 101m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:22:49.580 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 97m ago 101m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:22:49.580 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 97m ago 100m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:22:49.580 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 97m ago 100m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:22:49.580 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 95m ago 100m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:22:49.580 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 95m ago 99m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:22:49.580 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 95m ago 99m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:22:49.581 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 95m ago 99m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:22:49.581 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 95m ago 98m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:22:49.911 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:22:49.911 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:22:49.912 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:22:49.912 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:22:49.912 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:22:49.912 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:22:49.913 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:22:49.913 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:22:49.913 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:22:49.913 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:22:49.913 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:22:49.914 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:22:49.914 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:22:49.914 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:22:49.914 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:22:49.914 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:22:49.915 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:22:51.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:50 smithi154 conmon[34553]: cluster 2021-07-31T03:22:49.164318+0000 mgr.x (mgr.24542) 4098 : cluster 2021-07-31T03:22:51.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:50 smithi154 conmon[34553]: [DBG] pgmap v4875: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:51.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:50 smithi154 conmon[34553]: audit 2021-07-31T03:22:49.288245+0000 mgr.x (mgr.24542) 4099 : audit [DBG] from='client.29320 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:22:51.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:50 smithi154 conmon[34553]: audit 2021-07-31T03:22:49.570248 2021-07-31T03:22:51.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:50 smithi154 conmon[34553]: +0000 mgr.x (mgr.24542) 4100 : audit [DBG] from='client.19386 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:22:51.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:50 smithi154 conmon[34553]: audit 2021-07-31T03: 2021-07-31T03:22:51.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:50 smithi154 conmon[34553]: 22:49.912497+0000 mon.c (mon.1) 198 : audit [DBG] from='client.? 172.21.15.45:0/3950871329' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:22:51.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:50 smithi045 conmon[33655]: cluster 2021-07-31T03:22:49.164318+0000 mgr.x (mgr.24542) 4098 2021-07-31T03:22:51.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:50 smithi045 conmon[33655]: : cluster [DBG] pgmap v4875: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:51.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:50 smithi045 conmon[33655]: audit 2021-07-31T03:22:49.288245+0000 mgr.x (mgr.24542) 4099 : audit [DBG] from='client.29320 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:22:51.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:50 smithi045 conmon[33655]: audit 2021-07-31T03:22:49. 2021-07-31T03:22:51.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:50 smithi045 conmon[33655]: 570248+0000 mgr.x (mgr.24542) 4100 : audit [DBG] from='client.19386 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:22:51.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:50 smithi045 conmon[33655]: audit 2021-07-31T03:22:51.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:50 smithi045 conmon[33655]: 2021-07-31T03:22:49.912497+0000 mon.c (mon.1) 198 : 2021-07-31T03:22:51.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:50 smithi045 conmon[33655]: audit [DBG] from='client.? 172.21.15.45:0/3950871329' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:22:51.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:50 smithi045 conmon[39718]: cluster 2021-07-31T03:22:49.164318+0000 mgr.x ( 2021-07-31T03:22:51.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:50 smithi045 conmon[39718]: mgr.24542) 4098 : cluster [DBG] pgmap v4875: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:51.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:50 smithi045 conmon[39718]: audit 2021-07-31T03:22:49.288245+0000 mgr.x (mgr. 2021-07-31T03:22:51.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:50 smithi045 conmon[39718]: 24542) 4099 : audit [DBG] from='client.29320 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:22:51.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:50 smithi045 conmon[39718]: audit 2021-07-31T03:22:51.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:50 smithi045 conmon[39718]: 2021-07-31T03:22:49.570248+0000 2021-07-31T03:22:51.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:50 smithi045 conmon[39718]: mgr.x (mgr.24542) 4100 : audit [DBG] 2021-07-31T03:22:51.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:50 smithi045 conmon[39718]: from='client.19386 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:22:51.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:50 smithi045 conmon[39718]: audit 2021-07-31T03: 2021-07-31T03:22:51.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:50 smithi045 conmon[39718]: 22:49.912497+0000 mon.c (mon.1 2021-07-31T03:22:51.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:50 smithi045 conmon[39718]: ) 198 : audit [DBG] from='client.? 172.21.15.45:0/3950871329' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:22:53.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:52 smithi154 conmon[34553]: cluster 2021-07-31T03:22:51.164817+0000 mgr.x (mgr.24542) 4101 : cluster 2021-07-31T03:22:53.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:52 smithi154 conmon[34553]: [DBG] pgmap v4876: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:53.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:52 smithi045 conmon[33655]: cluster 2021-07-31T03:22:51.164817+0000 mgr.x (mgr.24542) 4101 : cluster 2021-07-31T03:22:53.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:52 smithi045 conmon[33655]: [DBG] pgmap v4876: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:53.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:52 smithi045 conmon[39718]: cluster 2021-07-31T03:22:51.164817+0000 mgr.x (mgr 2021-07-31T03:22:53.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:52 smithi045 conmon[39718]: .24542) 4101 : cluster [DBG] pgmap v4876: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:55.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:54 smithi154 conmon[34553]: cluster 2021-07-31T03:22:53.165187+0000 mgr.x (mgr.24542) 4102 : cluster [DBG] pgmap v4877: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:55.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:54 smithi045 conmon[33655]: cluster 2021-07-31T03:22:53.165187+0000 mgr.x ( 2021-07-31T03:22:55.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:54 smithi045 conmon[33655]: mgr.24542) 4102 : cluster [DBG] pgmap v4877: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:55.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:54 smithi045 conmon[39718]: cluster 2021-07-31T03:22:53.165187+0000 mgr.x (mgr.24542) 4102 2021-07-31T03:22:55.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:54 smithi045 conmon[39718]: : cluster [DBG] pgmap v4877: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:57.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:56 smithi154 conmon[34553]: cluster 2021-07-31T03:22:55.165583+0000 mgr.x (mgr.24542) 4103 : cluster [DBG] pgmap v4878: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:57.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:56 smithi045 conmon[39718]: cluster 2021-07-31T03:22:55.165583+0000 mgr.x (mgr.24542) 4103 : cluster 2021-07-31T03:22:57.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:56 smithi045 conmon[39718]: [DBG] pgmap v4878: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:57.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:56 smithi045 conmon[33655]: cluster 2021-07-31T03:22:55.165583+0000 mgr.x (mgr.24542) 4103 2021-07-31T03:22:57.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:56 smithi045 conmon[33655]: : cluster [DBG] pgmap v4878: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:59.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:22:58 smithi154 conmon[34553]: cluster 2021-07-31T03:22:57.166036+0000 mgr.x (mgr.24542) 4104 : cluster [DBG] pgmap v4879: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:59.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:58 smithi045 conmon[33655]: cluster 2021-07-31T03:22:57.166036 2021-07-31T03:22:59.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:22:58 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 4104 : cluster [DBG] pgmap v4879: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:22:59.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:58 smithi045 conmon[39718]: cluster 2021-07-31T03:22:57.166036+0000 mgr.x (mgr 2021-07-31T03:22:59.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:22:58 smithi045 conmon[39718]: .24542) 4104 : cluster [DBG] pgmap v4879: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:01.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:00 smithi154 conmon[34553]: cluster 2021-07-31T03:22:59.166443+0000 mgr.x (mgr.24542) 4105 : cluster [DBG] pgmap v4880: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:01.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:00 smithi045 conmon[39718]: cluster 2021-07-31T03:22:59.166443+0000 2021-07-31T03:23:01.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:00 smithi045 conmon[39718]: mgr.x (mgr.24542) 4105 : cluster [DBG] pgmap v4880: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:01.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:00 smithi045 conmon[33655]: cluster 2021-07-31T03:22:59.166443+0000 2021-07-31T03:23:01.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:00 smithi045 conmon[33655]: mgr.x (mgr.24542) 4105 : cluster [DBG] pgmap v4880: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:03.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:02 smithi154 conmon[34553]: cluster 2021-07-31T03:23:01.166943+0000 mgr.x (mgr.24542) 4106 : cluster [DBG] pgmap v4881: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:03.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:02 smithi045 conmon[33655]: cluster 2021-07-31T03:23:01.166943 2021-07-31T03:23:03.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:02 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 4106 : cluster [DBG] pgmap v4881: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:03.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:02 smithi045 conmon[39718]: cluster 2021-07-31T03:23:01.166943+0000 mgr.x (mgr 2021-07-31T03:23:03.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:02 smithi045 conmon[39718]: .24542) 4106 : cluster [DBG] pgmap v4881: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:05.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:04 smithi154 conmon[34553]: cluster 2021-07-31T03:23:03.167483+0000 mgr.x (mgr.24542) 4107 : cluster 2021-07-31T03:23:05.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:04 smithi154 conmon[34553]: [DBG] pgmap v4882: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:05.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:04 smithi045 conmon[39718]: cluster 2021-07-31T03:23:03.167483+0000 mgr.x (mgr 2021-07-31T03:23:05.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:04 smithi045 conmon[39718]: .24542) 4107 : cluster [DBG] pgmap v4882: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:05.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:04 smithi045 conmon[33655]: cluster 2021-07-31T03:23:03.167483+0000 mgr.x (mgr.24542) 4107 : cluster 2021-07-31T03:23:05.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:04 smithi045 conmon[33655]: [DBG] pgmap v4882: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:07.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:06 smithi154 conmon[34553]: cluster 2021-07-31T03:23:05.167978+0000 mgr.x (mgr.24542) 4108 2021-07-31T03:23:07.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:06 smithi154 conmon[34553]: : cluster [DBG] pgmap v4883: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:07.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:06 smithi045 conmon[39718]: cluster 2021-07-31T03:23:05.167978+0000 mgr.x ( 2021-07-31T03:23:07.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:06 smithi045 conmon[39718]: mgr.24542) 4108 : cluster [DBG] pgmap v4883: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:07.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:06 smithi045 conmon[33655]: cluster 2021-07-31T03:23:05.167978+0000 mgr.x (mgr.24542) 4108 : cluster [DBG] 2021-07-31T03:23:07.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:06 smithi045 conmon[33655]: pgmap v4883: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:09.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:08 smithi154 conmon[34553]: cluster 2021-07-31T03:23:07.168335+0000 mgr.x (mgr.24542) 4109 : cluster [DBG] pgmap v4884: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:09.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:08 smithi045 conmon[33655]: cluster 2021-07-31T03:23:07.168335+0000 mgr.x (mgr.24542) 4109 2021-07-31T03:23:09.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:08 smithi045 conmon[33655]: : cluster [DBG] pgmap v4884: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:09.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:08 smithi045 conmon[39718]: cluster 2021-07-31T03:23:07.168335+0000 mgr.x (mgr. 2021-07-31T03:23:09.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:08 smithi045 conmon[39718]: 24542) 4109 : cluster [DBG] pgmap v4884: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:11.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:10 smithi154 conmon[34553]: cluster 2021-07-31T03:23:09.168908+0000 mgr.x (mgr.24542) 4110 : cluster [DBG] 2021-07-31T03:23:11.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:10 smithi154 conmon[34553]: pgmap v4885: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:11.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:10 smithi045 conmon[33655]: cluster 2021-07-31T03:23:09.168908+0000 mgr.x (mgr.24542) 4110 2021-07-31T03:23:11.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:10 smithi045 conmon[33655]: : cluster [DBG] pgmap v4885: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:11.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:10 smithi045 conmon[39718]: cluster 2021-07-31T03:23:09.168908+0000 mgr.x (mgr 2021-07-31T03:23:11.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:10 smithi045 conmon[39718]: .24542) 4110 : cluster [DBG] pgmap v4885: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:13.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:12 smithi154 conmon[34553]: cluster 2021-07-31T03:23:11.169323+0000 mgr.x (mgr.24542) 4111 : cluster [DBG] pgmap v4886: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:13.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:12 smithi045 conmon[33655]: cluster 2021-07-31T03:23:11.169323+0000 mgr.x 2021-07-31T03:23:13.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:12 smithi045 conmon[33655]: (mgr.24542) 4111 : cluster [DBG] pgmap v4886: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:13.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:12 smithi045 conmon[39718]: cluster 2021-07-31T03:23:11. 2021-07-31T03:23:13.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:12 smithi045 conmon[39718]: 169323+0000 mgr.x (mgr.24542) 4111 : cluster [DBG] pgmap v4886: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:15.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:14 smithi154 conmon[34553]: cluster 2021-07-31T03:23:13.169695+0000 mgr.x (mgr.24542) 4112 : cluster 2021-07-31T03:23:15.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:14 smithi154 conmon[34553]: [DBG] pgmap v4887: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:15.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:14 smithi045 conmon[39718]: cluster 2021-07-31T03:23:13.169695+0000 mgr.x (mgr.24542) 4112 : cluster [DBG] pgmap v4887: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:15.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:14 smithi045 conmon[39718]: 2021-07-31T03:23:15.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:14 smithi045 conmon[33655]: cluster 2021-07-31T03:23:13.169695+0000 mgr.x (mgr.24542) 4112 2021-07-31T03:23:15.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:14 smithi045 conmon[33655]: : cluster [DBG] pgmap v4887: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:17.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:16 smithi154 conmon[34553]: cluster 2021-07-31T03:23:15.170214+0000 mgr.x (mgr.24542) 4113 : cluster [DBG] pgmap v4888: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:17.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:16 smithi154 conmon[34553]: 2021-07-31T03:23:17.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:16 smithi045 conmon[39718]: cluster 2021-07-31T03:23:15.170214+0000 mgr.x (mgr.24542) 4113 : cluster [DBG] pgmap v4888: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:17.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:16 smithi045 conmon[39718]: 2021-07-31T03:23:17.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:16 smithi045 conmon[33655]: cluster 2021-07-31T03:23:15.170214+0000 mgr.x (mgr.24542) 4113 : cluster [DBG] 2021-07-31T03:23:17.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:16 smithi045 conmon[33655]: pgmap v4888: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:19.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:18 smithi154 conmon[34553]: cluster 2021-07-31T03:23:17.170663+0000 mgr.x (mgr.24542) 4114 : cluster [DBG] pgmap v4889: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:19.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:18 smithi045 conmon[33655]: cluster 2021-07-31T03:23:17.170663+0000 mgr.x (mgr.24542) 4114 : cluster [DBG] pgmap v4889: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:19.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:18 smithi045 conmon[33655]: 2021-07-31T03:23:19.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:18 smithi045 conmon[39718]: cluster 2021-07-31T03:23:17 2021-07-31T03:23:19.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:18 smithi045 conmon[39718]: .170663+0000 mgr.x (mgr.24542) 4114 : cluster [DBG] pgmap v4889: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:20.221 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:23:20.499 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:23:20.499 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 98m ago 98m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:23:20.499 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 95m ago 98m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:23:20.500 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 95m ago 102m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:23:20.500 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 98m ago 104m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:23:20.500 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 98m ago 104m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:23:20.501 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 95m ago 102m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:23:20.501 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 98m ago 102m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:23:20.501 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 98m ago 99m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:23:20.501 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 95m ago 99m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:23:20.502 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 98m ago 101m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:23:20.502 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 98m ago 101m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:23:20.502 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 98m ago 101m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:23:20.502 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 98m ago 100m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:23:20.503 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 95m ago 100m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:23:20.503 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 95m ago 100m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:23:20.503 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 95m ago 99m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:23:20.503 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 95m ago 99m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:23:20.504 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 95m ago 99m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:23:20.832 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:23:20.833 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:23:20.833 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:23:20.833 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:23:20.834 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:23:20.834 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:23:20.834 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:23:20.835 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:23:20.835 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:23:20.835 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:23:20.836 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:23:20.836 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:23:20.836 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:23:20.836 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:23:20.837 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:23:20.837 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:23:20.837 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:23:21.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:20 smithi154 conmon[34553]: cluster 2021-07-31T03:23:19.171091+0000 mgr.x (mgr.24542) 4115 : cluster [DBG] pgmap v4890: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:21.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:20 smithi154 conmon[34553]: audit 2021-07-31T03:23:20.833335+0000 mon.c (mon.1) 199 : audit [DBG] from='client.? 172.21.15.45:0/4271505798' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:23:21.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:20 smithi045 conmon[39718]: cluster 2021-07-31T03:23:19.171091+0000 mgr.x 2021-07-31T03:23:21.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:20 smithi045 conmon[39718]: (mgr.24542) 4115 : cluster [DBG] pgmap v4890: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:21.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:20 smithi045 conmon[39718]: audit 2021-07-31T03:23 2021-07-31T03:23:21.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:20 smithi045 conmon[39718]: :20.833335+0000 mon.c (mon.1) 199 : audit [DBG] from='client.? 172.21.15.45:0/4271505798' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:23:21.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:20 smithi045 conmon[33655]: cluster 2021-07-31T03:23:19.171091+0000 mgr.x (mgr. 2021-07-31T03:23:21.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:20 smithi045 conmon[33655]: 24542) 4115 : cluster [DBG] pgmap v4890: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:21.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:20 smithi045 conmon[33655]: audit 2021-07-31T03:23:20 2021-07-31T03:23:21.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:20 smithi045 conmon[33655]: .833335+0000 mon.c (mon.1) 199 : audit [DBG] from='client.? 172.21.15.45:0/4271505798' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:23:22.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:21 smithi154 conmon[34553]: audit 2021-07-31T03:23:20.210246+0000 mgr.x (mgr.24542) 4116 : audit [DBG] from='client.29338 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:23:22.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:21 smithi154 conmon[34553]: audit 2021-07-31T03:23:20.492212+0000 mgr.x (mgr.24542) 4117 : audit [DBG] from='client.19404 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:23:22.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:21 smithi045 conmon[39718]: audit 2021-07-31T03:23:20.210246+0000 mgr.x ( 2021-07-31T03:23:22.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:21 smithi045 conmon[39718]: mgr.24542) 4116 : audit [DBG] from='client.29338 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:23:22.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:21 smithi045 conmon[39718]: audit 2021-07-31T03:23: 2021-07-31T03:23:22.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:21 smithi045 conmon[39718]: 20.492212+0000 mgr.x (mgr.24542) 4117 : audit 2021-07-31T03:23:22.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:21 smithi045 conmon[39718]: [DBG] from='client.19404 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:23:22.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:21 smithi045 conmon[33655]: audit 2021-07-31T03:23:20.210246+0000 mgr.x (mgr.24542) 4116 : audit [DBG] from='client.29338 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:23:22.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:21 smithi045 conmon[33655]: audit 2021-07-31T03:23:20.492212+0000 mgr.x (mgr.24542) 4117 : audit [DBG] from='client.19404 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:23:23.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:22 smithi154 conmon[34553]: cluster 2021-07-31T03:23:21.171547+0000 mgr.x (mgr.24542) 4118 : cluster [DBG] pgmap v4891: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:23.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:22 smithi045 conmon[39718]: cluster 2021-07-31T03:23:21.171547+0000 mgr.x (mgr.24542) 4118 : cluster [DBG] pgmap v4891: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:23.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:22 smithi045 conmon[33655]: cluster 2021-07-31T03:23:21.171547+0000 mgr.x (mgr.24542) 4118 : cluster [DBG] 2021-07-31T03:23:23.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:22 smithi045 conmon[33655]: pgmap v4891: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:25.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:24 smithi154 conmon[34553]: cluster 2021-07-31T03:23:23.171978+0000 mgr.x (mgr.24542) 4119 : cluster [DBG] pgmap v4892: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:25.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:24 smithi045 conmon[33655]: cluster 2021-07-31T03:23:23.171978+0000 mgr.x (mgr.24542) 4119 2021-07-31T03:23:25.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:24 smithi045 conmon[33655]: : cluster [DBG] pgmap v4892: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:25.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:24 smithi045 conmon[39718]: cluster 2021-07-31T03:23:23.171978+0000 mgr.x ( 2021-07-31T03:23:25.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:24 smithi045 conmon[39718]: mgr.24542) 4119 : cluster [DBG] pgmap v4892: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:27.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:26 smithi154 conmon[34553]: cluster 2021-07-31T03:23:25.172395+0000 mgr.x (mgr.24542) 4120 : cluster [DBG] pgmap v4893: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:27.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:26 smithi045 conmon[39718]: cluster 2021-07-31T03:23:25.172395+0000 mgr.x (mgr.24542) 4120 : cluster [DBG] pgmap v4893: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:27.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:26 smithi045 conmon[33655]: cluster 2021-07-31T03:23:25 2021-07-31T03:23:27.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:26 smithi045 conmon[33655]: .172395+0000 mgr.x (mgr.24542) 4120 : cluster [DBG] pgmap v4893: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:29.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:28 smithi154 conmon[34553]: cluster 2021-07-31T03:23:27.172855+0000 mgr.x (mgr.24542) 4121 : cluster [DBG] pgmap v4894: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:29.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:28 smithi154 conmon[34553]: audit 2021-07-31T03:23:28.315529+0000 mon.b (mon.2) 555 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:23:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:28 smithi154 conmon[34553]: audit 2021-07-31T03:23:28.315898+0000 mon.b (mon.2) 556 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:23:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:28 smithi154 conmon[34553]: audit 2021-07-31T03:23:28.316149+0000 mon.a (mon.0 2021-07-31T03:23:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:28 smithi154 conmon[34553]: ) 3297 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:23:29.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:28 smithi154 conmon[34553]: audit 2021-07-31T03: 2021-07-31T03:23:29.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:28 smithi154 conmon[34553]: 23:28.316632+0000 mon.a (mon.0) 3298 : 2021-07-31T03:23:29.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:28 smithi154 conmon[34553]: audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:23:29.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:28 smithi045 conmon[39718]: cluster 2021-07-31T03:23: 2021-07-31T03:23:29.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:28 smithi045 conmon[39718]: 27.172855+0000 mgr.x (mgr.24542) 4121 : cluster [DBG] pgmap v4894: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:29.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:28 smithi045 conmon[39718]: audit 2021-07-31T03 2021-07-31T03:23:29.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:28 smithi045 conmon[39718]: :23:28.315529+0000 mon.b (mon. 2021-07-31T03:23:29.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:28 smithi045 conmon[39718]: 2) 555 : audit [INF] 2021-07-31T03:23:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:28 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:23:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:28 smithi045 conmon[39718]: audit 2021- 2021-07-31T03:23:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:28 smithi045 conmon[39718]: 07-31T03:23:28. 2021-07-31T03:23:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:28 smithi045 conmon[39718]: 315898+0000 mon.b (mon.2 2021-07-31T03:23:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:28 smithi045 conmon[39718]: ) 556 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:23:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:28 smithi045 conmon[39718]: 2021-07-31T03:23:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:28 smithi045 conmon[39718]: audit 2021-07-31T03:23:29.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:28 smithi045 conmon[39718]: 2021-07-31T03:23 2021-07-31T03:23:29.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:28 smithi045 conmon[39718]: :28.316149+0000 mon.a ( 2021-07-31T03:23:29.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:28 smithi045 conmon[39718]: mon.0) 3297 : audit [INF] 2021-07-31T03:23:29.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:28 smithi045 conmon[39718]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:23:29.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:28 smithi045 conmon[39718]: audit 2021 2021-07-31T03:23:29.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:28 smithi045 conmon[39718]: -07-31T03:23:28. 2021-07-31T03:23:29.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:28 smithi045 conmon[39718]: 316632+0000 mon.a (mon.0 2021-07-31T03:23:29.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:28 smithi045 conmon[39718]: ) 3298 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:23:29.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:28 smithi045 conmon[33655]: cluster 2021-07-31T03:23:27.172855+0000 mgr.x (mgr.24542) 2021-07-31T03:23:29.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:28 smithi045 conmon[33655]: 4121 : cluster [DBG] pgmap v4894: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:29.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:28 smithi045 conmon[33655]: audit 2021-07-31T03:23:28.315529+0000 mon.b (mon.2) 555 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:23:29.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:28 smithi045 conmon[33655]: audit 2021-07-31T03 2021-07-31T03:23:29.283 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:28 smithi045 conmon[33655]: :23:28.315898+0000 mon.b (mon.2) 2021-07-31T03:23:29.283 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:28 smithi045 conmon[33655]: 556 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:23:29.283 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:28 smithi045 conmon[33655]: audit 2021-07 2021-07-31T03:23:29.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:28 smithi045 conmon[33655]: -31T03:23:28.316149+0000 mon.a (mon.0 2021-07-31T03:23:29.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:28 smithi045 conmon[33655]: ) 3297 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:23:29.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:28 smithi045 conmon[33655]: 2021-07-31T03:23:29.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:28 smithi045 conmon[33655]: audit 2021-07-31T 2021-07-31T03:23:29.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:28 smithi045 conmon[33655]: 03:23:28.316632+0000 mon.a (mon.0) 3298 : audit 2021-07-31T03:23:29.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:28 smithi045 conmon[33655]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:23:31.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:30 smithi154 conmon[34553]: cluster 2021-07-31T03:23:29.173217+0000 mgr.x (mgr.24542) 4122 : cluster [DBG] pgmap v4895: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:31.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:30 smithi045 conmon[39718]: cluster 2021-07-31T03:23: 2021-07-31T03:23:31.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:30 smithi045 conmon[39718]: 29.173217+0000 mgr.x (mgr.24542) 4122 : cluster [DBG] pgmap v4895: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:31.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:30 smithi045 conmon[33655]: cluster 2021-07-31T03:23:29.173217+0000 mgr.x 2021-07-31T03:23:31.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:30 smithi045 conmon[33655]: (mgr.24542) 4122 : cluster [DBG] pgmap v4895: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:33.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[34553]: cluster 2021-07-31T03:23:31.173680+0000 mgr.x (mgr.24542) 4123 : cluster [DBG] pgmap v4896: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:33.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[34553]: audit 2021-07-31T03:23:32.875175+0000 mon.b (mon.2) 557 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:23:33.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[34553]: audit 2021-07-31T03:23:32.876618+0000 mon.b (mon.2) 558 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:23:33.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[34553]: audit 2021-07-31T03:23:32.877584+0000 mon.b (mon.2) 559 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:23:33.253 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: debug 2021-07-31T03:23:32.920+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:23:33.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:23:33.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:23:33.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: yield 2021-07-31T03:23:33.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:23:33.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:23:33.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:23:33.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:23:33.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:23:33.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:23:33.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:23:33.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:23:33.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:23:33.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: task.result() 2021-07-31T03:23:33.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:23:33.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: debug 2021-07-31T03:23:32.932+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:23:33.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:23:33.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:23:33.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: yield 2021-07-31T03:23:33.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:23:33.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:23:33.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:23:33.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:23:33.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:23:33.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:23:33.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:23:33.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:23:33.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:23:33.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: task.result() 2021-07-31T03:23:33.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:23:33.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: debug 2021-07-31T03:23:32.958+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:23:33.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:23:33.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:23:33.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: yield 2021-07-31T03:23:33.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:23:33.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:23:33.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:23:33.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:23:33.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:23:33.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:23:33.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:23:33.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:23:33.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:23:33.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: task.result() 2021-07-31T03:23:33.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:23:33.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: debug 2021-07-31T03:23:32.959+0000 7fcb93160700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:23:33.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:23:33.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:23:33.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: yield 2021-07-31T03:23:33.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:23:33.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:23:33.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:23:33.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:23:33.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:23:33.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:23:33.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:23:33.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:23:33.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:23:33.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: task.result() 2021-07-31T03:23:33.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:23:33.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: 2021-07-31T03:23:33.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:23:33.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: 2021-07-31T03:23:33.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:23:33.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:23:33.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:23:33.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:23:33.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:23:33.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:23:33.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:23:33.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:23:33.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:23:33.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:23:33.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:23:33.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:23:33.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:23:33.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:23:33.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:23:33.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:23:33.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:23:33.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:23:33.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: raise 2021-07-31T03:23:33.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:23:33.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:23:33.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:23:33.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:23:33.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:23:33.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: debug 2021-07-31T03:23:32.967+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:23:33.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:23:33.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:23:33.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: yield 2021-07-31T03:23:33.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:23:33.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:23:33.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:23:33.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:23:33.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:23:33.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:23:33.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:23:33.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:23:33.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:23:33.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: task.result() 2021-07-31T03:23:33.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:23:33.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: debug 2021-07-31T03:23:32.968+0000 7fcb9295f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:23:33.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:23:33.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:23:33.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: yield 2021-07-31T03:23:33.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:23:33.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:23:33.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:23:33.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:23:33.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:23:33.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:23:33.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:23:33.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:23:33.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:23:33.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: task.result() 2021-07-31T03:23:33.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:23:33.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: 2021-07-31T03:23:33.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:23:33.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: 2021-07-31T03:23:33.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:23:33.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:23:33.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:23:33.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:23:33.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:23:33.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:23:33.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:23:33.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:23:33.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:23:33.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:23:33.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:23:33.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:23:33.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:23:33.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:23:33.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:23:33.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:23:33.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:23:33.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:23:33.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: raise 2021-07-31T03:23:33.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:23:33.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:23:33.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:23:33.284 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:23:33.284 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:23:32 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:23:33.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:32 smithi045 conmon[39718]: cluster 2021-07-31T03:23:31.173680+0000 mgr.x 2021-07-31T03:23:33.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:32 smithi045 conmon[39718]: (mgr.24542) 4123 : cluster [DBG] pgmap v4896: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:33.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:32 smithi045 conmon[39718]: audit 2021-07-31T03 2021-07-31T03:23:33.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:32 smithi045 conmon[39718]: :23:32.875175+0000 mon.b (mon.2) 557 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:23:33.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:32 smithi045 conmon[39718]: audit 2021- 2021-07-31T03:23:33.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:32 smithi045 conmon[39718]: 07-31T03:23:32.876618+0000 2021-07-31T03:23:33.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:32 smithi045 conmon[39718]: mon.b (mon.2) 558 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:23:33.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:32 smithi045 conmon[39718]: audit 2021 2021-07-31T03:23:33.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:32 smithi045 conmon[39718]: -07-31T03:23:32.877584+0000 mon.b 2021-07-31T03:23:33.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:32 smithi045 conmon[39718]: (mon.2) 559 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:23:33.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:32 smithi045 conmon[33655]: cluster 2021-07-31T03:23:31.173680+0000 mgr.x 2021-07-31T03:23:33.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:32 smithi045 conmon[33655]: (mgr.24542) 4123 : cluster [DBG] pgmap v4896: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:33.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:32 smithi045 conmon[33655]: audit 2021-07-31T 2021-07-31T03:23:33.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:32 smithi045 conmon[33655]: 03:23:32.875175+0000 mon.b (mon.2) 557 : 2021-07-31T03:23:33.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:32 smithi045 conmon[33655]: audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:23:33.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:32 smithi045 conmon[33655]: audit 2021-07-31T03:23:33.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:32 smithi045 conmon[33655]: 2021-07-31T03:23 2021-07-31T03:23:33.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:32 smithi045 conmon[33655]: :32.876618+0000 mon.b 2021-07-31T03:23:33.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:32 smithi045 conmon[33655]: (mon.2) 558 : audit 2021-07-31T03:23:33.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:32 smithi045 conmon[33655]: [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:23:33.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:32 smithi045 conmon[33655]: audit 2021 2021-07-31T03:23:33.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:32 smithi045 conmon[33655]: -07-31T03:23: 2021-07-31T03:23:33.290 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:32 smithi045 conmon[33655]: 32.877584+0000 mon.b (mon 2021-07-31T03:23:33.290 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:32 smithi045 conmon[33655]: .2) 559 : audit [INF] 2021-07-31T03:23:33.290 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:32 smithi045 conmon[33655]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:23:34.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:33 smithi154 conmon[34553]: cephadm 2021-07-31T03:23:32.878340+0000 mgr.x (mgr.24542) 4124 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:23:34.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:33 smithi045 conmon[39718]: cephadm 2021-07-31T03:23:32.878340+0000 2021-07-31T03:23:34.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:33 smithi045 conmon[39718]: mgr.x (mgr.24542) 4124 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:23:34.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:33 smithi045 conmon[33655]: cephadm 2021-07-31T03:23:32.878340+0000 mgr.x (mgr.24542) 4124 : cephadm [WRN] 2021-07-31T03:23:34.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:33 smithi045 conmon[33655]: unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:23:35.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: cephadm 2021-07-31T03:23:32.921672+0000 mgr.x (mgr.24542) 4125 : cephadm [ERR] 2021-07-31T03:23:35.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:23:35.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:23:35.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: yield 2021-07-31T03:23:35.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:23:35.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:23:35.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:23:35.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:23:35.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:23:35.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:23:35.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:23:35.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:23:35.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:23:35.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: task.result() 2021-07-31T03:23:35.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:23:35.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: 2021-07-31T03:23:35.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: cephadm 2021-07-31T03:23:32.922096+0000 mgr.x (mgr.24542) 4126 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:23:35.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: cephadm 2021-07-31T03:23:32.933858+0000 mgr.x (mgr.24542) 4127 : cephadm [ERR] 2021-07-31T03:23:35.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:23:35.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:23:35.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: yield 2021-07-31T03:23:35.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:23:35.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:23:35.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:23:35.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:23:35.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:23:35.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:23:35.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:23:35.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:23:35.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:23:35.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: task.result() 2021-07-31T03:23:35.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:23:35.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: cephadm 2021-07-31T03:23:32.935016+0000 mgr.x (mgr.24542) 4128 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:23:35.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: cephadm 2021-07-31T03:23:32.959563+0000 mgr.x (mgr.24542) 4129 : cephadm [ERR] 2021-07-31T03:23:35.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:23:35.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:23:35.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: yield 2021-07-31T03:23:35.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:23:35.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:23:35.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:23:35.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:23:35.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:23:35.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:23:35.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:23:35.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:23:35.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:23:35.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: task.result() 2021-07-31T03:23:35.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:23:35.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: cephadm 2021-07-31T03:23:32.960015+0000 mgr.x (mgr.24542) 4130 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:23:35.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:23:35.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:23:35.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: yield 2021-07-31T03:23:35.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:23:35.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:23:35.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:23:35.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:23:35.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:23:35.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:23:35.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:23:35.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:23:35.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:23:35.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: task.result() 2021-07-31T03:23:35.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:23:35.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: 2021-07-31T03:23:35.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:23:35.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: 2021-07-31T03:23:35.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:23:35.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:23:35.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:23:35.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:23:35.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:23:35.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:23:35.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:23:35.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:23:35.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:23:35.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:23:35.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:23:35.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:23:35.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:23:35.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:23:35.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:23:35.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:23:35.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:23:35.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:23:35.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: raise 2021-07-31T03:23:35.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:23:35.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:23:35.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:23:35.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:23:35.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:23:35.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: cephadm 2021-07-31T03:23:32.968604+0000 mgr.x (mgr.24542) 4131 : cephadm [ERR] 2021-07-31T03:23:35.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:23:35.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:23:35.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: yield 2021-07-31T03:23:35.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:23:35.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:23:35.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:23:35.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:23:35.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:23:35.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:23:35.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:23:35.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:23:35.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:23:35.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: task.result() 2021-07-31T03:23:35.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:23:35.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: cephadm 2021-07-31T03:23:32.969024+0000 mgr.x (mgr.24542) 4132 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:23:35.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:23:35.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:23:35.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: yield 2021-07-31T03:23:35.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:23:35.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:23:35.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:23:35.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:23:35.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:23:35.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:23:35.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:23:35.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:23:35.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:23:35.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: task.result() 2021-07-31T03:23:35.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:23:35.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: 2021-07-31T03:23:35.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:23:35.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: 2021-07-31T03:23:35.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:23:35.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:23:35.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:23:35.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:23:35.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:23:35.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:23:35.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:23:35.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:23:35.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:23:35.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:23:35.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:23:35.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:23:35.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:23:35.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:23:35.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:23:35.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:23:35.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:23:35.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:23:35.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: raise 2021-07-31T03:23:35.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:23:35.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:23:35.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:23:35.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:23:35.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:23:35.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:34 smithi154 conmon[34553]: cluster 2021-07-31T03:23:33.174051+0000 mgr.x (mgr.24542) 4133 : cluster [DBG] pgmap v4897: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:35.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: cephadm 2021-07-31T03:23:32.921672+0000 mgr.x (mgr. 2021-07-31T03:23:35.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: 24542) 4125 : cephadm [ERR] 2021-07-31T03:23:35.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:23:35.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:23:35.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: yield 2021-07-31T03:23:35.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:23:35.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:23:35.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:23:35.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:23:35.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:23:35.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:23:35.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:23:35.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:23:35.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:23:35.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: task.result() 2021-07-31T03:23:35.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:23:35.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: cephadm 2021-07-31T03:23:32 2021-07-31T03:23:35.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: .922096+0000 mgr.x (mgr.24542) 4126 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:23:35.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: cephadm 2021-07-31T03:23: 2021-07-31T03:23:35.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: 32.933858+0000 mgr.x (mgr.24542) 4127 2021-07-31T03:23:35.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: : cephadm [ERR] 2021-07-31T03:23:35.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:23:35.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:23:35.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: yield 2021-07-31T03:23:35.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:23:35.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:23:35.289 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:23:35.289 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:23:35.289 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:23:35.289 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:23:35.289 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:23:35.290 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:23:35.290 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:23:35.290 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: task.result() 2021-07-31T03:23:35.290 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:23:35.290 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: cephadm 2021-07-31T03:23:32.935016+0000 mgr.x (mgr.24542) 4128 : cephadm [INF] 2021-07-31T03:23:35.291 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:23:35.291 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: cephadm 2021-07-31T03:23:35.291 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: 2021-07-31T03:23 2021-07-31T03:23:35.291 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: :32.959563+0000 mgr.x (mgr 2021-07-31T03:23:35.291 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: .24542) 4129 : cephadm [ERR] 2021-07-31T03:23:35.292 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:23:35.292 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:23:35.292 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: yield 2021-07-31T03:23:35.292 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:23:35.292 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:23:35.293 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:23:35.293 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:23:35.293 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:23:35.293 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:23:35.293 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:23:35.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:23:35.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:23:35.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: task.result() 2021-07-31T03:23:35.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:23:35.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: cephadm 2021-07-31T03:23:32.960015+0000 mgr.x (mgr.24542) 4130 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:23:35.295 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:23:35.295 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:23:35.295 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: yield 2021-07-31T03:23:35.295 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:23:35.295 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:23:35.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:23:35.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:23:35.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:23:35.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:23:35.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:23:35.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:23:35.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:23:35.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: task.result() 2021-07-31T03:23:35.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:23:35.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: 2021-07-31T03:23:35.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:23:35.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: 2021-07-31T03:23:35.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:23:35.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:23:35.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:23:35.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:23:35.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:23:35.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:23:35.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:23:35.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:23:35.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:23:35.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:23:35.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:23:35.300 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: cephadm 2021-07-31T03:23:32.921672+0000 2021-07-31T03:23:35.301 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: mgr.x (mgr.24542) 4125 : cephadm [ERR] 2021-07-31T03:23:35.301 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:23:35.301 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:23:35.301 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: yield 2021-07-31T03:23:35.302 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:23:35.302 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:23:35.302 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:23:35.302 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:23:35.302 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:23:35.303 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:23:35.303 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:23:35.303 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:23:35.303 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:23:35.303 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: task.result() 2021-07-31T03:23:35.303 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:23:35.304 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: cephadm 2021-07- 2021-07-31T03:23:35.304 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: 31T03:23:32.922096+0000 mgr.x (mgr.24542) 4126 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:23:35.304 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: cephadm 2021-07-31T03:23:32.933858+0000 mgr.x (mgr.24542) 4127 : cephadm [ERR] 2021-07-31T03:23:35.304 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:23:35.305 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:23:35.305 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: yield 2021-07-31T03:23:35.305 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:23:35.305 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:23:35.305 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:23:35.305 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:23:35.306 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:23:35.306 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:23:35.306 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:23:35.306 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:23:35.306 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:23:35.307 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: task.result() 2021-07-31T03:23:35.307 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:23:35.307 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: cephadm 2021-07-31T03:23:32.935016+0000 mgr.x (mgr.24542) 4128 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:23:35.307 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: cephadm 2021-07-31T03: 2021-07-31T03:23:35.307 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: 23:32.959563+0000 mgr.x (mgr.24542) 4129 : cephadm [ERR] 2021-07-31T03:23:35.308 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: 2021-07-31T03:23:35.308 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:23:35.308 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:23:35.308 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: yield 2021-07-31T03:23:35.308 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:23:35.309 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:23:35.309 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:23:35.309 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:23:35.310 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:23:35.310 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:23:35.310 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:23:35.310 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:23:35.310 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: raise 2021-07-31T03:23:35.311 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:23:35.311 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:23:35.311 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:23:35.311 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:23:35.311 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:23:35.311 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: 2021-07-31T03:23:35.312 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: cephadm 2021-07-31T03:23:32.968604+0000 mgr.x (mgr.24542) 4131 : cephadm [ERR] 2021-07-31T03:23:35.312 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:23:35.312 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:23:35.312 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: yield 2021-07-31T03:23:35.312 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:23:35.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:23:35.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:23:35.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:23:35.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:23:35.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:23:35.314 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:23:35.314 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:23:35.314 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:23:35.314 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: task.result() 2021-07-31T03:23:35.314 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:23:35.315 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: cephadm 2021-07-31T03:23:32.969024+0000 mgr.x (mgr.24542) 4132 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:23:35.315 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:23:35.315 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:23:35.315 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: yield 2021-07-31T03:23:35.315 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:23:35.316 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:23:35.316 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:23:35.316 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:23:35.316 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:23:35.316 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:23:35.317 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:23:35.317 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:23:35.317 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:23:35.317 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:23:35.318 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:23:35.318 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:23:35.318 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:23:35.318 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:23:35.318 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: task.result() 2021-07-31T03:23:35.319 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:23:35.319 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: cephadm 2021-07-31T03:23:32.960015+0000 mgr.x (mgr.24542) 4130 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:23:35.319 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:23:35.319 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:23:35.319 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: yield 2021-07-31T03:23:35.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:23:35.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:23:35.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:23:35.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:23:35.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:23:35.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:23:35.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:23:35.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:23:35.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:23:35.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: task.result() 2021-07-31T03:23:35.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:23:35.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: 2021-07-31T03:23:35.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:23:35.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: 2021-07-31T03:23:35.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:23:35.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:23:35.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:23:35.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:23:35.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:23:35.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:23:35.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:23:35.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:23:35.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:23:35.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:23:35.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:23:35.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:23:35.325 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:23:35.325 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:23:35.325 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:23:35.325 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:23:35.326 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: task.result() 2021-07-31T03:23:35.326 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:23:35.326 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: 2021-07-31T03:23:35.326 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:23:35.326 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: 2021-07-31T03:23:35.327 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:23:35.327 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:23:35.327 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:23:35.327 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:23:35.327 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:23:35.328 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:23:35.328 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:23:35.328 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:23:35.328 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:23:35.328 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:23:35.329 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:23:35.329 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:23:35.329 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:23:35.329 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:23:35.329 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:23:35.330 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:23:35.330 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:23:35.330 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:23:35.330 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: raise 2021-07-31T03:23:35.330 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:23:35.331 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:23:35.331 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:23:35.331 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:23:35.331 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:23:35.331 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: cluster 2021-07-31T03:23: 2021-07-31T03:23:35.332 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[39718]: 33.174051+0000 mgr.x (mgr.24542) 4133 : cluster [DBG] pgmap v4897: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:35.332 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:23:35.332 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:23:35.332 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:23:35.333 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:23:35.333 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:23:35.333 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: raise 2021-07-31T03:23:35.333 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:23:35.334 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:23:35.334 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:23:35.334 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:23:35.334 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:23:35.334 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: cephadm 2021-07-31T03:23:32.968604+0000 mgr.x (mgr.24542) 4131 : cephadm [ERR] 2021-07-31T03:23:35.335 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:23:35.335 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:23:35.335 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: yield 2021-07-31T03:23:35.335 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:23:35.335 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:23:35.336 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:23:35.336 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:23:35.336 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:23:35.336 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:23:35.336 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:23:35.337 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:23:35.337 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:23:35.337 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: task.result() 2021-07-31T03:23:35.337 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:23:35.337 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: cephadm 2021-07-31T03:23:32.969024+0000 mgr.x (mgr.24542) 4132 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:23:35.338 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:23:35.338 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:23:35.338 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: yield 2021-07-31T03:23:35.338 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:23:35.339 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:23:35.339 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:23:35.339 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:23:35.339 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:23:35.339 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:23:35.340 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:23:35.340 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:23:35.340 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:23:35.340 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: task.result() 2021-07-31T03:23:35.340 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:23:35.341 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: 2021-07-31T03:23:35.341 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:23:35.341 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: 2021-07-31T03:23:35.341 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:23:35.341 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:23:35.342 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:23:35.342 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:23:35.342 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:23:35.342 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:23:35.342 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:23:35.343 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:23:35.343 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:23:35.343 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:23:35.343 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:23:35.343 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:23:35.343 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:23:35.344 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:23:35.344 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:23:35.344 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:23:35.344 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:23:35.344 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:23:35.345 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: raise 2021-07-31T03:23:35.345 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:23:35.345 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:23:35.345 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:23:35.345 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:23:35.346 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:23:35.346 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:34 smithi045 conmon[33655]: cluster 2021-07-31T03:23:33.174051+0000 mgr.x (mgr.24542) 4133 : cluster [DBG] pgmap v4897: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:36.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:35 smithi154 conmon[34553]: cluster 2021-07-31T03:23:35.174459+0000 mgr.x (mgr.24542) 4134 : cluster [DBG] pgmap v4898: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:36.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:35 smithi045 conmon[33655]: cluster 2021-07-31T03:23:35.174459+0000 mgr.x (mgr.24542 2021-07-31T03:23:36.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:35 smithi045 conmon[33655]: ) 4134 : cluster [DBG] pgmap v4898: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:36.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:35 smithi045 conmon[39718]: cluster 2021-07-31T03:23:35.174459+0000 mgr.x (mgr 2021-07-31T03:23:36.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:35 smithi045 conmon[39718]: .24542) 4134 : cluster [DBG] pgmap v4898: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:38.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:38 smithi154 conmon[34553]: cluster 2021-07-31T03:23:37.174982+0000 mgr.x (mgr.24542) 4135 : cluster [DBG] pgmap v4899: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:38.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:38 smithi045 conmon[33655]: cluster 2021-07-31T03:23:37.174982+0000 2021-07-31T03:23:38.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:38 smithi045 conmon[33655]: mgr.x (mgr.24542) 4135 : cluster [DBG] pgmap v4899: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:38.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:38 smithi045 conmon[39718]: cluster 2021-07-31T03:23:37.174982+0000 2021-07-31T03:23:38.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:38 smithi045 conmon[39718]: mgr.x (mgr.24542) 4135 : cluster [DBG] pgmap v4899: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:40.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:40 smithi154 conmon[34553]: cluster 2021-07-31T03:23:39.175395+0000 mgr.x (mgr.24542) 4136 : cluster [DBG] pgmap v4900: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:40.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:40 smithi045 conmon[33655]: cluster 2021-07-31T03:23:39.175395+0000 mgr.x 2021-07-31T03:23:40.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:40 smithi045 conmon[33655]: (mgr.24542) 4136 : cluster [DBG] pgmap v4900: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:40.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:40 smithi045 conmon[39718]: cluster 2021-07-31T03:23:39.175395+0000 mgr.x (mgr.24542) 2021-07-31T03:23:40.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:40 smithi045 conmon[39718]: 4136 : cluster [DBG] pgmap v4900: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:42.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:42 smithi154 conmon[34553]: cluster 2021-07-31T03:23:41.175873+0000 mgr.x (mgr.24542) 4137 : cluster [DBG] pgmap v4901: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:42.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:42 smithi045 conmon[33655]: cluster 2021-07-31T03:23:41.175873+0000 mgr.x 2021-07-31T03:23:42.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:42 smithi045 conmon[33655]: (mgr.24542) 4137 : cluster [DBG] pgmap v4901: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:42.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:42 smithi045 conmon[39718]: cluster 2021-07-31T03:23:41. 2021-07-31T03:23:42.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:42 smithi045 conmon[39718]: 175873+0000 mgr.x (mgr.24542) 4137 : cluster [DBG] pgmap v4901: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:44.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:44 smithi154 conmon[34553]: cluster 2021-07-31T03:23:43.176388+0000 mgr.x (mgr.24542) 4138 : cluster [DBG] pgmap v4902: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:44.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:44 smithi045 conmon[33655]: cluster 2021-07-31T03:23:43.176388+0000 mgr.x (mgr.24542) 4138 2021-07-31T03:23:44.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:44 smithi045 conmon[33655]: : cluster [DBG] pgmap v4902: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:44.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:44 smithi045 conmon[39718]: cluster 2021-07-31T03:23:43.176388+0000 mgr.x 2021-07-31T03:23:44.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:44 smithi045 conmon[39718]: (mgr.24542) 4138 : cluster [DBG] pgmap v4902: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:46.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:46 smithi154 conmon[34553]: cluster 2021-07-31T03:23:45.176764+0000 mgr.x (mgr.24542) 4139 : cluster [DBG] pgmap v4903: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:46.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:46 smithi045 conmon[39718]: cluster 2021-07-31T03:23:45.176764+0000 mgr.x (mgr 2021-07-31T03:23:46.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:46 smithi045 conmon[39718]: .24542) 4139 : cluster [DBG] pgmap v4903: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:46.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:46 smithi045 conmon[33655]: cluster 2021-07-31T03:23:45.176764+0000 mgr.x (mgr.24542) 2021-07-31T03:23:46.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:46 smithi045 conmon[33655]: 4139 : cluster [DBG] pgmap v4903: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:48.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:48 smithi154 conmon[34553]: cluster 2021-07-31T03:23:47.177270+0000 mgr.x (mgr.24542) 4140 : cluster 2021-07-31T03:23:48.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:48 smithi154 conmon[34553]: [DBG] pgmap v4904: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:48.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:48 smithi045 conmon[39718]: cluster 2021-07-31T03:23:47.177270+0000 mgr.x (mgr. 2021-07-31T03:23:48.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:48 smithi045 conmon[39718]: 24542) 4140 : cluster [DBG] pgmap v4904: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:48.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:48 smithi045 conmon[33655]: cluster 2021-07-31T03:23:47.177270+0000 mgr.x (mgr.24542) 4140 : 2021-07-31T03:23:48.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:48 smithi045 conmon[33655]: cluster [DBG] pgmap v4904: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:50.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:50 smithi154 conmon[34553]: cluster 2021-07-31T03:23:49.177804+0000 mgr.x (mgr.24542) 4141 : cluster [DBG] pgmap v4905: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:50.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:50 smithi045 conmon[39718]: cluster 2021-07-31T03:23:49.177804+0000 mgr.x (mgr.24542) 4141 : cluster [DBG] pgmap v4905: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:50.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:50 smithi045 conmon[33655]: cluster 2021-07-31T03:23:49.177804+0000 2021-07-31T03:23:50.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:50 smithi045 conmon[33655]: mgr.x (mgr.24542) 4141 : cluster [DBG] pgmap v4905: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:51.147 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:23:51.426 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:23:51.426 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 98m ago 99m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:23:51.426 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 96m ago 99m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:23:51.427 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 96m ago 102m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:23:51.427 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 98m ago 104m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:23:51.427 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 98m ago 105m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:23:51.427 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 96m ago 103m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:23:51.428 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 98m ago 103m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:23:51.428 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 98m ago 99m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:23:51.428 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 96m ago 99m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:23:51.428 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 98m ago 102m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:23:51.429 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 98m ago 102m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:23:51.429 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 98m ago 101m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:23:51.429 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 98m ago 101m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:23:51.429 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 96m ago 101m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:23:51.430 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 96m ago 100m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:23:51.430 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 96m ago 100m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:23:51.430 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 96m ago 100m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:23:51.430 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 96m ago 99m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:23:51.762 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:23:51.763 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:23:51.763 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:23:51.763 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:23:51.763 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:23:51.764 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:23:51.764 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:23:51.764 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:23:51.765 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:23:51.765 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:23:51.765 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:23:51.766 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:23:51.766 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:23:51.766 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:23:51.767 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:23:51.767 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:23:51.767 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:23:52.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:52 smithi045 conmon[33655]: audit 2021-07-31T03:23:51.135859+0000 mgr.x (mgr.24542) 4142 2021-07-31T03:23:52.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:52 smithi045 conmon[33655]: : audit [DBG] from='client.19416 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:23:52.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:52 smithi045 conmon[33655]: cluster 2021-07-31T03:23:51.178206+0000 mgr.x (mgr.24542) 4143 : cluster [DBG] pgmap v4906: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:52.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:52 smithi045 conmon[33655]: audit 2021-07-31T03:23:51.419131 2021-07-31T03:23:52.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:52 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 4144 : audit [DBG] from='client.19422 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:23:52.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:52 smithi045 conmon[33655]: audit 2021-07-31T03:23:52.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:52 smithi045 conmon[33655]: 2021-07-31 2021-07-31T03:23:52.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:52 smithi045 conmon[33655]: T03:23:51.762728 2021-07-31T03:23:52.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:52 smithi045 conmon[33655]: +0000 mon.c (mon.1 2021-07-31T03:23:52.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:52 smithi045 conmon[33655]: ) 200 : audit [DBG] from='client.? 172.21.15.45:0/4105784504' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:23:52.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:52 smithi045 conmon[33655]: 2021-07-31T03:23:52.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:52 smithi045 conmon[39718]: audit 2021-07-31T03:23:51. 2021-07-31T03:23:52.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:52 smithi045 conmon[39718]: 135859+0000 mgr.x (mgr.24542) 4142 : audit [DBG] from='client.19416 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:23:52.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:52 smithi045 conmon[39718]: cluster 2021 2021-07-31T03:23:52.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:52 smithi045 conmon[39718]: -07-31T03:23: 2021-07-31T03:23:52.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:52 smithi045 conmon[39718]: 51.178206+0000 mgr.x (mgr. 2021-07-31T03:23:52.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:52 smithi045 conmon[39718]: 24542) 4143 : cluster [DBG] pgmap v4906: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:52.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:52 smithi045 conmon[39718]: 2021-07-31T03:23:52.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:52 smithi045 conmon[39718]: audit 2021-07-31T03:23:52.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:52 smithi045 conmon[39718]: 2021-07-31T03:23 2021-07-31T03:23:52.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:52 smithi045 conmon[39718]: :51.419131+0000 mgr.x ( 2021-07-31T03:23:52.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:52 smithi045 conmon[39718]: mgr.24542) 4144 : audit [DBG] from='client.19422 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:23:52.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:52 smithi045 conmon[39718]: audit 2021-07-31T03:23:52.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:52 smithi045 conmon[39718]: 2021-07- 2021-07-31T03:23:52.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:52 smithi045 conmon[39718]: 31T03:23:51. 2021-07-31T03:23:52.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:52 smithi045 conmon[39718]: 762728+0000 mon.c (mon. 2021-07-31T03:23:52.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:52 smithi045 conmon[39718]: 1) 200 : audit [DBG] 2021-07-31T03:23:52.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:52 smithi045 conmon[39718]: from='client.? 172.21.15.45:0/4105784504' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:23:52.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:52 smithi154 conmon[34553]: audit 2021-07-31T03:23:51.135859+0000 mgr.x (mgr.24542) 4142 : audit [DBG] from='client.19416 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:23:52.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:52 smithi154 conmon[34553]: cluster 2021-07-31T03:23:51.178206+0000 mgr.x (mgr.24542) 4143 : cluster [DBG] pgmap v4906: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:52.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:52 smithi154 conmon[34553]: audit 2021-07-31T03:23:51.419131+0000 mgr.x (mgr.24542) 4144 : audit [DBG] from='client.19422 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:23:52.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:52 smithi154 conmon[34553]: audit 2021-07-31T03:23:51.762728+0000 mon.c (mon.1) 200 2021-07-31T03:23:52.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:52 smithi154 conmon[34553]: : audit [DBG] from='client.? 172.21.15.45:0/4105784504' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:23:54.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:54 smithi045 conmon[39718]: cluster 2021-07-31T03:23:53.178677+0000 mgr.x ( 2021-07-31T03:23:54.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:54 smithi045 conmon[39718]: mgr.24542) 4145 : cluster [DBG] pgmap v4907: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:54.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:54 smithi045 conmon[33655]: cluster 2021-07-31T03:23:53.178677+0000 mgr.x (mgr.24542) 4145 2021-07-31T03:23:54.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:54 smithi045 conmon[33655]: : cluster [DBG] pgmap v4907: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:54.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:54 smithi154 conmon[34553]: cluster 2021-07-31T03:23:53.178677+0000 mgr.x (mgr.24542) 4145 : cluster [DBG] pgmap v4907: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:56.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:56 smithi045 conmon[33655]: cluster 2021-07-31T03:23:55.179197+0000 mgr.x (mgr.24542) 2021-07-31T03:23:56.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:56 smithi045 conmon[33655]: 4146 : cluster [DBG] pgmap v4908: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:56.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:56 smithi045 conmon[39718]: cluster 2021-07-31T03:23:55.179197+0000 mgr.x 2021-07-31T03:23:56.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:56 smithi045 conmon[39718]: (mgr.24542) 4146 : cluster [DBG] pgmap v4908: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:56.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:56 smithi154 conmon[34553]: cluster 2021-07-31T03:23:55.179197+0000 mgr.x (mgr.24542) 4146 : cluster [DBG] pgmap v4908: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:58.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:58 smithi045 conmon[39718]: cluster 2021-07-31T03:23:57.179588+0000 mgr.x ( 2021-07-31T03:23:58.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:23:58 smithi045 conmon[39718]: mgr.24542) 4147 : cluster [DBG] pgmap v4909: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:58.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:58 smithi045 conmon[33655]: cluster 2021-07-31T03:23:57.179588 2021-07-31T03:23:58.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:23:58 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 4147 : cluster [DBG] pgmap v4909: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:23:58.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:23:58 smithi154 conmon[34553]: cluster 2021-07-31T03:23:57.179588+0000 mgr.x (mgr.24542) 4147 : cluster [DBG] pgmap v4909: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:00.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:00 smithi045 conmon[39718]: cluster 2021-07-31T03:23:59.180115+0000 mgr.x (mgr 2021-07-31T03:24:00.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:00 smithi045 conmon[39718]: .24542) 4148 : cluster [DBG] pgmap v4910: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:00.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:00 smithi045 conmon[33655]: cluster 2021-07-31T03:23:59.180115+0000 mgr.x (mgr.24542) 4148 : cluster [DBG] pgmap v4910: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:00.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:00 smithi045 conmon[33655]: 2021-07-31T03:24:00.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:00 smithi154 conmon[34553]: cluster 2021-07-31T03:23:59.180115+0000 mgr.x (mgr.24542) 4148 : cluster [DBG] pgmap v4910: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:02.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:02 smithi045 conmon[33655]: cluster 2021-07-31T03:24:01.180509+0000 2021-07-31T03:24:02.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:02 smithi045 conmon[33655]: mgr.x (mgr.24542) 4149 : cluster [DBG] pgmap v4911: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:02.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:02 smithi045 conmon[39718]: cluster 2021-07-31T03:24:01.180509+0000 mgr.x ( 2021-07-31T03:24:02.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:02 smithi045 conmon[39718]: mgr.24542) 4149 : cluster [DBG] pgmap v4911: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:02.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:02 smithi154 conmon[34553]: cluster 2021-07-31T03:24:01.180509+0000 mgr.x (mgr.24542) 4149 : cluster [DBG] pgmap v4911: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:04.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:04 smithi045 conmon[39718]: cluster 2021-07-31T03:24:03.181049+0000 mgr.x (mgr.24542) 4150 2021-07-31T03:24:04.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:04 smithi045 conmon[39718]: : cluster [DBG] pgmap v4912: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:04.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:04 smithi045 conmon[33655]: cluster 2021-07- 2021-07-31T03:24:04.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:04 smithi045 conmon[33655]: 31T03:24:03.181049+0000 mgr.x (mgr.24542) 4150 : cluster [DBG] pgmap v4912: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:04.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:04 smithi154 conmon[34553]: cluster 2021-07-31T03:24:03.181049+0000 mgr.x (mgr.24542) 4150 : cluster [DBG] pgmap v4912: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:06.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:06 smithi154 conmon[34553]: cluster 2021-07-31T03:24:05.181565+0000 mgr.x (mgr.24542) 4151 : cluster [DBG] pgmap v4913: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:06.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:06 smithi045 conmon[39718]: cluster 2021-07-31T03:24:05.181565+0000 mgr.x (mgr 2021-07-31T03:24:06.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:06 smithi045 conmon[39718]: .24542) 4151 : cluster [DBG] pgmap v4913: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:06.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:06 smithi045 conmon[33655]: cluster 2021-07-31T03:24:05.181565+0000 mgr.x (mgr.24542) 4151 : cluster [DBG] pgmap v4913: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:08.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:08 smithi154 conmon[34553]: cluster 2021-07-31T03:24:07.182028+0000 mgr.x (mgr.24542) 4152 : cluster [DBG] pgmap v4914: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:08.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:08 smithi045 conmon[39718]: cluster 2021-07-31T03:24:07.182028+0000 2021-07-31T03:24:08.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:08 smithi045 conmon[39718]: mgr.x (mgr.24542) 4152 : cluster [DBG] pgmap v4914: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:08.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:08 smithi045 conmon[33655]: cluster 2021-07-31T03:24:07.182028+0000 mgr.x (mgr.24542 2021-07-31T03:24:08.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:08 smithi045 conmon[33655]: ) 4152 : cluster [DBG] pgmap v4914: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:10.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:10 smithi154 conmon[34553]: cluster 2021-07-31T03:24:09.182524+0000 mgr.x (mgr.24542) 4153 : cluster [DBG] pgmap v4915: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:10.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:10 smithi045 conmon[39718]: cluster 2021-07-31T03:24:09.182524+0000 mgr.x (mgr.24542) 4153 : cluster [DBG] pgmap v4915: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:10.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:10 smithi045 conmon[33655]: cluster 2021-07-31T03:24:09.182524+0000 mgr.x (mgr.24542) 4153 : 2021-07-31T03:24:10.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:10 smithi045 conmon[33655]: cluster [DBG] pgmap v4915: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:12.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:12 smithi154 conmon[34553]: cluster 2021-07-31T03:24:11.183012+0000 mgr.x (mgr.24542) 4154 : cluster [DBG] pgmap v4916: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:12.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:12 smithi154 conmon[34553]: 2021-07-31T03:24:12.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:12 smithi045 conmon[39718]: cluster 2021-07-31T03:24 2021-07-31T03:24:12.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:12 smithi045 conmon[39718]: :11.183012+0000 mgr.x (mgr.24542) 4154 : cluster [DBG] pgmap v4916: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:12.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:12 smithi045 conmon[33655]: cluster 2021-07-31T03:24:11.183012+0000 mgr.x (mgr.24542) 4154 2021-07-31T03:24:12.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:12 smithi045 conmon[33655]: : cluster [DBG] pgmap v4916: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:14.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:14 smithi154 conmon[34553]: cluster 2021-07-31T03:24:13.183409+0000 mgr.x (mgr.24542) 4155 : cluster [DBG] pgmap v4917: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:14.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:14 smithi045 conmon[39718]: cluster 2021-07-31T03:24:13.183409+0000 mgr.x (mgr 2021-07-31T03:24:14.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:14 smithi045 conmon[39718]: .24542) 4155 : cluster [DBG] pgmap v4917: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:14.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:14 smithi045 conmon[33655]: cluster 2021-07-31T03:24:13.183409+0000 mgr.x (mgr.24542) 4155 : cluster 2021-07-31T03:24:14.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:14 smithi045 conmon[33655]: [DBG] pgmap v4917: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:16.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:16 smithi154 conmon[34553]: cluster 2021-07-31T03:24:15.183821+0000 mgr.x (mgr.24542) 4156 : cluster [DBG] pgmap v4918: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:16.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:16 smithi045 conmon[39718]: cluster 2021-07-31T03:24:15.183821+0000 mgr.x (mgr 2021-07-31T03:24:16.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:16 smithi045 conmon[39718]: .24542) 4156 : cluster [DBG] pgmap v4918: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:16.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:16 smithi045 conmon[33655]: cluster 2021-07-31T03:24:15.183821+0000 mgr.x (mgr.24542) 4156 : cluster [DBG] pgmap v4918: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:18.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:18 smithi154 conmon[34553]: cluster 2021-07-31T03:24:17.184347+0000 mgr.x (mgr.24542) 4157 : cluster [DBG] pgmap v4919: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:18.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:18 smithi045 conmon[39718]: cluster 2021-07-31T03:24:17.184347+0000 mgr.x ( 2021-07-31T03:24:18.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:18 smithi045 conmon[39718]: mgr.24542) 4157 : cluster [DBG] pgmap v4919: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:18.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:18 smithi045 conmon[33655]: cluster 2021-07-31T03:24:17.184347+0000 mgr.x (mgr.24542) 4157 : cluster [DBG] pgmap v4919: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:20.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:20 smithi154 conmon[34553]: cluster 2021-07-31T03:24:19.184809+0000 mgr.x (mgr.24542) 4158 : cluster [DBG] pgmap v4920: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:20.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:20 smithi045 conmon[33655]: cluster 2021-07-31T03:24:19.184809+0000 mgr.x (mgr.24542) 4158 : 2021-07-31T03:24:20.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:20 smithi045 conmon[33655]: cluster [DBG] pgmap v4920: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:20.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:20 smithi045 conmon[39718]: cluster 2021-07-31T03:24:19.184809+0000 mgr.x (mgr. 2021-07-31T03:24:20.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:20 smithi045 conmon[39718]: 24542) 4158 : cluster [DBG] pgmap v4920: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:22.074 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:24:22.352 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:24:22.352 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 99m ago 100m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:24:22.353 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 96m ago 99m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:24:22.353 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 96m ago 103m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:24:22.353 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 99m ago 105m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:24:22.353 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 99m ago 105m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:24:22.354 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 96m ago 103m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:24:22.354 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 99m ago 103m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:24:22.354 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 99m ago 100m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:24:22.354 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 96m ago 100m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:24:22.355 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 99m ago 102m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:24:22.355 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 99m ago 102m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:24:22.355 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 99m ago 102m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:24:22.355 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 99m ago 101m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:24:22.356 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 96m ago 101m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:24:22.356 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 96m ago 101m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:24:22.356 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 96m ago 101m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:24:22.356 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 96m ago 100m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:24:22.356 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 96m ago 100m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:24:22.686 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:24:22.687 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:24:22.687 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:24:22.687 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:24:22.687 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:24:22.688 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:24:22.688 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:24:22.688 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:24:22.688 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:24:22.688 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:24:22.689 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:24:22.689 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:24:22.689 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:24:22.689 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:24:22.689 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:24:22.690 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:24:22.690 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:24:22.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:22 smithi154 conmon[34553]: cluster 2021-07-31T03:24:21.185135+0000 mgr.x (mgr.24542) 4159 : cluster [DBG] pgmap v4921: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:22.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:22 smithi045 conmon[33655]: cluster 2021-07-31T03:24:21.185135+0000 mgr.x 2021-07-31T03:24:22.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:22 smithi045 conmon[33655]: (mgr.24542) 4159 : cluster [DBG] pgmap v4921: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:22.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:22 smithi045 conmon[39718]: cluster 2021-07-31T03:24:21. 2021-07-31T03:24:22.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:22 smithi045 conmon[39718]: 185135+0000 mgr.x (mgr.24542) 4159 : cluster [DBG] pgmap v4921: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:23.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:23 smithi154 conmon[34553]: audit 2021-07-31T03:24:22.063307+0000 mgr.x (mgr.24542) 4160 : audit [DBG] from='client.19434 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:24:23.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:23 smithi154 conmon[34553]: audit 2021-07-31T03:24:23.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:23 smithi154 conmon[34553]: 2021-07-31T03:24:22.345178+0000 mgr.x (mgr.24542) 4161 : audit [DBG] from='client.19440 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:24:23.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:23 smithi154 conmon[34553]: audit 2021-07-31T03:24:22.687428+0000 mon.c (mon.1) 201 : audit [DBG] from='client.? 172.21.15.45:0/1708345740' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:24:23.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:23 smithi045 conmon[33655]: audit 2021-07-31T03:24:22.063307+0000 mgr.x (mgr 2021-07-31T03:24:23.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:23 smithi045 conmon[33655]: .24542) 4160 : audit [DBG] from='client.19434 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:24:23.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:23 smithi045 conmon[33655]: audit 2021-07-31T03:24:22.345178+0000 mgr.x ( 2021-07-31T03:24:23.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:23 smithi045 conmon[33655]: mgr.24542) 4161 : audit [DBG] from='client.19440 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:24:23.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:23 smithi045 conmon[33655]: audit 2021- 2021-07-31T03:24:23.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:23 smithi045 conmon[33655]: 07-31T03:24:22.687428+0000 mon.c (mon 2021-07-31T03:24:23.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:23 smithi045 conmon[33655]: .1) 201 : audit [DBG] from='client.? 172.21.15.45:0/1708345740' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:24:23.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:23 smithi045 conmon[39718]: audit 2021-07-31T03: 2021-07-31T03:24:23.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:23 smithi045 conmon[39718]: 24:22.063307+0000 mgr.x (mgr.24542) 4160 : audit [DBG] from='client.19434 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:24:23.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:23 smithi045 conmon[39718]: audit 2021 2021-07-31T03:24:23.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:23 smithi045 conmon[39718]: -07-31T03:24:22.345178 2021-07-31T03:24:23.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:23 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542 2021-07-31T03:24:23.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:23 smithi045 conmon[39718]: ) 4161 : audit [DBG] from='client.19440 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:24:23.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:23 smithi045 conmon[39718]: 2021-07-31T03:24:23.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:23 smithi045 conmon[39718]: audit 2021-07-31T03:24:23.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:23 smithi045 conmon[39718]: 2021-07-31T03:24: 2021-07-31T03:24:23.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:23 smithi045 conmon[39718]: 22.687428+0000 mon.c (mon 2021-07-31T03:24:23.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:23 smithi045 conmon[39718]: .1) 201 : audit [DBG] 2021-07-31T03:24:23.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:23 smithi045 conmon[39718]: from='client.? 172.21.15.45:0/1708345740' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:24:24.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:24 smithi154 conmon[34553]: cluster 2021-07-31T03:24:23.185631+0000 mgr.x (mgr.24542) 4162 : cluster [DBG] pgmap v4922: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:24.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:24 smithi154 conmon[34553]: 2021-07-31T03:24:24.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:24 smithi045 conmon[39718]: cluster 2021-07-31T03:24:23.185631+0000 mgr.x (mgr.24542) 4162 : cluster [DBG] pgmap v4922: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:24.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:24 smithi045 conmon[33655]: cluster 2021-07-31T03:24:23.185631+0000 mgr.x (mgr 2021-07-31T03:24:24.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:24 smithi045 conmon[33655]: .24542) 4162 : cluster [DBG] pgmap v4922: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:26.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:26 smithi154 conmon[34553]: cluster 2021-07-31T03:24:25.186247+0000 mgr.x (mgr.24542) 4163 : cluster [DBG] pgmap v4923: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:26.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:26 smithi045 conmon[39718]: cluster 2021-07-31T03:24:25.186247+0000 mgr.x ( 2021-07-31T03:24:26.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:26 smithi045 conmon[39718]: mgr.24542) 4163 : cluster [DBG] pgmap v4923: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:26.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:26 smithi045 conmon[33655]: cluster 2021-07-31T03:24:25.186247+0000 mgr.x ( 2021-07-31T03:24:26.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:26 smithi045 conmon[33655]: mgr.24542) 4163 : cluster [DBG] pgmap v4923: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:28.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:28 smithi154 conmon[34553]: cluster 2021-07-31T03:24:27.186650+0000 mgr.x (mgr.24542) 4164 : cluster [DBG] pgmap v4924: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:28.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:28 smithi154 conmon[34553]: 2021-07-31T03:24:28.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:28 smithi045 conmon[39718]: cluster 2021-07-31T03:24:27.186650+0000 mgr.x ( 2021-07-31T03:24:28.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:28 smithi045 conmon[39718]: mgr.24542) 4164 : cluster [DBG] pgmap v4924: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:28.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:28 smithi045 conmon[33655]: cluster 2021-07-31T03:24:27.186650 2021-07-31T03:24:28.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:28 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 4164 : cluster [DBG] pgmap v4924: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:29.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:29 smithi154 conmon[34553]: audit 2021-07-31T03:24:28.322099+0000 mon.b (mon.2) 560 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:24:29.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:29 smithi154 conmon[34553]: audit 2021-07-31T03:24:28.322334+0000 mon.b (mon.2) 561 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:24:29.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:29 smithi154 conmon[34553]: audit 2021-07-31T03:24:28.322486+0000 mon.a 2021-07-31T03:24:29.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:29 smithi154 conmon[34553]: (mon.0) 3299 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:24:29.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:29 smithi154 conmon[34553]: audit 2021-07-31 2021-07-31T03:24:29.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:29 smithi154 conmon[34553]: T03:24:28.322886+0000 mon.a (mon.0) 3300 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:24:29.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:29 smithi045 conmon[39718]: audit 2021-07-31T03:24:28.322099+0000 mon.b (mon.2) 560 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:24:29.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:29 smithi045 conmon[39718]: audit 2021-07-31T03:24:28 2021-07-31T03:24:29.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:29 smithi045 conmon[39718]: .322334+0000 mon.b (mon.2) 561 2021-07-31T03:24:29.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:29 smithi045 conmon[39718]: : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:24:29.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:29 smithi045 conmon[39718]: audit 2021-07-31T03:24:29.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:29 smithi045 conmon[39718]: 2021-07-31T03 2021-07-31T03:24:29.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:29 smithi045 conmon[39718]: :24:28.322486+0000 mon.a 2021-07-31T03:24:29.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:29 smithi045 conmon[39718]: (mon.0) 3299 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:24:29.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:29 smithi045 conmon[39718]: 2021-07-31T03:24:29.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:29 smithi045 conmon[39718]: audit 2021-07-31T03:24:29.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:29 smithi045 conmon[39718]: 2021-07 2021-07-31T03:24:29.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:29 smithi045 conmon[39718]: -31T03:24:28.322886 2021-07-31T03:24:29.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:29 smithi045 conmon[39718]: +0000 mon.a (mon.0) 3300 : audit 2021-07-31T03:24:29.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:29 smithi045 conmon[39718]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:24:29.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:29 smithi045 conmon[33655]: audit 2021-07-31T03:24:28.322099+0000 mon.b (mon.2) 2021-07-31T03:24:29.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:29 smithi045 conmon[33655]: 560 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:24:29.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:29 smithi045 conmon[33655]: audit 2021-07-31T03:24:28.322334+0000 mon.b (mon.2) 561 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:24:29.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:29 smithi045 conmon[33655]: audit 2021-07-31T03:24:29.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:29 smithi045 conmon[33655]: 2021-07-31T03:24:28 2021-07-31T03:24:29.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:29 smithi045 conmon[33655]: .322486+0000 mon.a (mon.0) 3299 : 2021-07-31T03:24:29.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:29 smithi045 conmon[33655]: audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:24:29.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:29 smithi045 conmon[33655]: audit 2021-07-31T03:24:29.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:29 smithi045 conmon[33655]: 2021-07-31T03:24:28.322886 2021-07-31T03:24:29.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:29 smithi045 conmon[33655]: +0000 mon.a (mon.0) 3300 : audit 2021-07-31T03:24:29.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:29 smithi045 conmon[33655]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:24:30.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:30 smithi154 conmon[34553]: cluster 2021-07-31T03:24:29.187085+0000 mgr.x (mgr.24542) 4165 : cluster [DBG] pgmap v4925: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:30.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:30 smithi045 conmon[39718]: cluster 2021-07-31T03:24:29.187085+0000 mgr.x (mgr 2021-07-31T03:24:30.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:30 smithi045 conmon[39718]: .24542) 4165 : cluster [DBG] pgmap v4925: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:30.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:30 smithi045 conmon[33655]: cluster 2021-07-31T03:24:29.187085+0000 mgr.x (mgr.24542) 4165 2021-07-31T03:24:30.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:30 smithi045 conmon[33655]: : cluster [DBG] pgmap v4925: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:32.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:32 smithi154 conmon[34553]: cluster 2021-07-31T03:24:31.187554+0000 mgr.x (mgr.24542) 4166 : cluster [DBG] pgmap v4926: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:32.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:32 smithi045 conmon[39718]: cluster 2021-07-31T03:24:31.187554+0000 mgr.x (mgr. 2021-07-31T03:24:32.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:32 smithi045 conmon[39718]: 24542) 4166 : cluster [DBG] pgmap v4926: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:32.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:32 smithi045 conmon[33655]: cluster 2021-07-31T03:24:31.187554+0000 mgr.x (mgr.24542) 4166 : cluster [DBG] pgmap v4926: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:32.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:32 smithi045 conmon[33655]: 2021-07-31T03:24:33.252 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: debug 2021-07-31T03:24:33.015+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:24:33.253 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:24:33.253 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:24:33.253 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: yield 2021-07-31T03:24:33.253 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:24:33.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:24:33.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:24:33.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:24:33.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:24:33.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:24:33.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:24:33.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:24:33.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:24:33.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: task.result() 2021-07-31T03:24:33.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:24:33.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: debug 2021-07-31T03:24:33.026+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:24:33.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:24:33.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:24:33.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: yield 2021-07-31T03:24:33.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:24:33.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:24:33.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:24:33.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:24:33.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:24:33.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:24:33.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:24:33.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:24:33.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:24:33.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: task.result() 2021-07-31T03:24:33.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:24:33.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: debug 2021-07-31T03:24:33.052+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:24:33.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:24:33.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:24:33.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: yield 2021-07-31T03:24:33.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:24:33.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:24:33.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:24:33.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:24:33.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:24:33.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:24:33.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:24:33.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:24:33.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:24:33.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: task.result() 2021-07-31T03:24:33.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:24:33.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: debug 2021-07-31T03:24:33.052+0000 7fcb9215e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:24:33.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:24:33.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:24:33.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: yield 2021-07-31T03:24:33.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:24:33.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:24:33.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:24:33.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:24:33.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:24:33.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:24:33.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:24:33.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:24:33.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:24:33.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: task.result() 2021-07-31T03:24:33.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:24:33.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: 2021-07-31T03:24:33.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:24:33.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: 2021-07-31T03:24:33.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:24:33.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:24:33.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:24:33.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:24:33.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:24:33.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:24:33.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:24:33.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:24:33.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:24:33.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:24:33.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:24:33.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:24:33.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:24:33.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:24:33.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:24:33.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:24:33.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:24:33.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:24:33.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: raise 2021-07-31T03:24:33.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:24:33.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:24:33.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:24:33.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:24:33.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:24:33.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: debug 2021-07-31T03:24:33.061+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:24:33.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:24:33.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:24:33.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: yield 2021-07-31T03:24:33.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:24:33.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:24:33.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:24:33.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:24:33.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:24:33.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:24:33.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:24:33.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:24:33.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:24:33.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: task.result() 2021-07-31T03:24:33.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:24:33.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: debug 2021-07-31T03:24:33.061+0000 7fcb9195d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:24:33.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:24:33.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:24:33.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: yield 2021-07-31T03:24:33.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:24:33.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:24:33.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:24:33.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:24:33.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:24:33.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:24:33.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:24:33.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:24:33.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:24:33.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: task.result() 2021-07-31T03:24:33.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:24:33.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: 2021-07-31T03:24:33.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:24:33.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: 2021-07-31T03:24:33.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:24:33.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:24:33.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:24:33.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:24:33.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:24:33.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:24:33.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:24:33.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:24:33.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:24:33.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:24:33.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:24:33.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:24:33.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:24:33.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:24:33.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:24:33.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:24:33.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:24:33.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:24:33.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: raise 2021-07-31T03:24:33.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:24:33.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:24:33.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:24:33.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:24:33.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:24:33.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[34553]: audit 2021-07-31T03:24:32.969754+0000 mon.b (mon.2) 562 : 2021-07-31T03:24:33.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[34553]: audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:24:33.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[34553]: audit 2021-07-31T03:24:32.971213+0000 mon.b (mon.2) 563 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:24:33.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[34553]: audit 2021-07- 2021-07-31T03:24:33.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:33 smithi154 conmon[34553]: 31T03:24:32.972260+0000 mon.b (mon.2) 564 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:24:33.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:33 smithi045 conmon[33655]: audit 2021-07-31T03:24:32.969754+0000 mon.b (mon. 2021-07-31T03:24:33.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:33 smithi045 conmon[33655]: 2) 562 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:24:33.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:33 smithi045 conmon[33655]: audit 2021-07-31T03:24:32.971213+0000 mon.b (mon.2) 563 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:24:33.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:33 smithi045 conmon[33655]: audit 2021-07-31T03:24:32.972260+0000 mon.b (mon. 2021-07-31T03:24:33.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:33 smithi045 conmon[33655]: 2) 564 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:24:33.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:33 smithi045 conmon[39718]: audit 2021-07-31T03:24:32.969754+0000 2021-07-31T03:24:33.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:33 smithi045 conmon[39718]: mon.b (mon.2) 562 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:24:33.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:33 smithi045 conmon[39718]: audit 2021-07 2021-07-31T03:24:33.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:33 smithi045 conmon[39718]: -31T03:24:32.971213+0000 mon.b (mon.2) 563 : audit [DBG] 2021-07-31T03:24:33.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:33 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:24:33.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:33 smithi045 conmon[39718]: audit 2021-07-31T03: 2021-07-31T03:24:33.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:33 smithi045 conmon[39718]: 24:32.972260+0000 mon.b (mon.2) 2021-07-31T03:24:33.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:33 smithi045 conmon[39718]: 564 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:24:34.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: cephadm 2021-07-31T03:24:32.973175+0000 mgr.x (mgr.24542) 4167 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:24:34.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: cephadm 2021-07-31T03:24:33.016270+0000 mgr.x (mgr.24542) 4168 : cephadm [ERR] 2021-07-31T03:24:34.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:24:34.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:24:34.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: yield 2021-07-31T03:24:34.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:24:34.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:24:34.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:24:34.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:24:34.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:24:34.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:24:34.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:24:34.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:24:34.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:24:34.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: task.result() 2021-07-31T03:24:34.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:24:34.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: cephadm 2021-07-31T03:24:33.016767+0000 2021-07-31T03:24:34.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: mgr.x (mgr.24542) 4169 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:24:34.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: cephadm 2021-07-31T03:24:33.027814+0000 mgr.x (mgr.24542) 4170 : cephadm [ERR] 2021-07-31T03:24:34.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:24:34.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:24:34.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: yield 2021-07-31T03:24:34.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:24:34.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:24:34.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:24:34.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:24:34.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:24:34.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:24:34.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:24:34.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:24:34.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:24:34.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: task.result() 2021-07-31T03:24:34.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:24:34.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: cephadm 2021-07-31T03:24:33.028669+0000 mgr.x (mgr.24542) 4171 : cephadm [INF] 2021-07-31T03:24:34.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:24:34.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: cephadm 2021-07-31T03:24:33.053287+0000 mgr.x (mgr.24542) 4172 : cephadm [ERR] 2021-07-31T03:24:34.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:24:34.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:24:34.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: yield 2021-07-31T03:24:34.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:24:34.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:24:34.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:24:34.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:24:34.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:24:34.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:24:34.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:24:34.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:24:34.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:24:34.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: task.result() 2021-07-31T03:24:34.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:24:34.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: cephadm 2021-07-31T03:24:33.053673+0000 mgr.x (mgr.24542) 4173 : cephadm 2021-07-31T03:24:34.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:24:34.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:24:34.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:24:34.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: yield 2021-07-31T03:24:34.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:24:34.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:24:34.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:24:34.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:24:34.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:24:34.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:24:34.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:24:34.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:24:34.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:24:34.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: task.result() 2021-07-31T03:24:34.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:24:34.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: 2021-07-31T03:24:34.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:24:34.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: 2021-07-31T03:24:34.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:24:34.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:24:34.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:24:34.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:24:34.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:24:34.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:24:34.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:24:34.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:24:34.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:24:34.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:24:34.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:24:34.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:24:34.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:24:34.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:24:34.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:24:34.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:24:34.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:24:34.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:24:34.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: raise 2021-07-31T03:24:34.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:24:34.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:24:34.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:24:34.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:24:34.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:24:34.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: cephadm 2021-07-31T03:24:33.062259+0000 mgr.x (mgr.24542) 4174 : cephadm [ERR] 2021-07-31T03:24:34.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:24:34.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:24:34.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: yield 2021-07-31T03:24:34.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:24:34.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:24:34.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:24:34.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:24:34.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:24:34.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:24:34.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:24:34.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:24:34.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:24:34.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: task.result() 2021-07-31T03:24:34.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:24:34.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: cephadm 2021-07-31T03:24:33.062671+0000 mgr.x 2021-07-31T03:24:34.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: (mgr.24542) 4175 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:24:34.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:24:34.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:24:34.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: yield 2021-07-31T03:24:34.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:24:34.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:24:34.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:24:34.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:24:34.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:24:34.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:24:34.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:24:34.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:24:34.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:24:34.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: task.result() 2021-07-31T03:24:34.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:24:34.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: 2021-07-31T03:24:34.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:24:34.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: 2021-07-31T03:24:34.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:24:34.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:24:34.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:24:34.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:24:34.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:24:34.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:24:34.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:24:34.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:24:34.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:24:34.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:24:34.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:24:34.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:24:34.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:24:34.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:24:34.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:24:34.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:24:34.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:24:34.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:24:34.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: raise 2021-07-31T03:24:34.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:24:34.782 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:24:34.782 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:24:34.782 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:24:34.782 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:24:34.782 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:34 smithi154 conmon[34553]: cluster 2021-07-31T03:24:33.187969+0000 mgr.x (mgr.24542) 4176 : cluster [DBG] pgmap v4927: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:34.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: cephadm 2021-07-31T03:24:32.973175+0000 mgr.x (mgr. 2021-07-31T03:24:34.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: 24542) 4167 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:24:34.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: cephadm 2021-07-31T03:24:33.016270+0000 mgr.x (mgr.24542) 4168 : cephadm [ERR] 2021-07-31T03:24:34.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:24:34.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:24:34.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: yield 2021-07-31T03:24:34.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:24:34.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:24:34.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:24:34.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:24:34.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:24:34.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:24:34.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:24:34.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:24:34.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:24:34.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: task.result() 2021-07-31T03:24:34.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:24:34.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: cephadm 2021-07-31T03:24:33.016767+0000 mgr.x (mgr.24542) 4169 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:24:34.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: cephadm 2021 2021-07-31T03:24:34.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: -07-31T03:24:33 2021-07-31T03:24:34.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: .027814+0000 mgr.x (mgr.24542 2021-07-31T03:24:34.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: ) 4170 : cephadm [ERR] 2021-07-31T03:24:34.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:24:34.788 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:24:34.788 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: yield 2021-07-31T03:24:34.788 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:24:34.788 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:24:34.788 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:24:34.789 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:24:34.789 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:24:34.789 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:24:34.789 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:24:34.790 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:24:34.790 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:24:34.790 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: task.result() 2021-07-31T03:24:34.790 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:24:34.790 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: cephadm 2021-07-31T03:24:33.028669+0000 mgr.x (mgr.24542) 4171 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:24:34.791 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: cephadm 2021-07-31T03:24: 2021-07-31T03:24:34.791 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: 33.053287+0000 mgr.x (mgr.24542) 4172 : 2021-07-31T03:24:34.791 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: cephadm [ERR] 2021-07-31T03:24:34.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: cephadm 2021-07-31T03:24:32.973175+0000 mgr.x (mgr.24542) 2021-07-31T03:24:34.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: 4167 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:24:34.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: cephadm 2021-07-31T03:24:33.016270+0000 2021-07-31T03:24:34.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: mgr.x (mgr.24542) 4168 : cephadm [ERR] 2021-07-31T03:24:34.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:24:34.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:24:34.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: yield 2021-07-31T03:24:34.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:24:34.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:24:34.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:24:34.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:24:34.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:24:34.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:24:34.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:24:34.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:24:34.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:24:34.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: task.result() 2021-07-31T03:24:34.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:24:34.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: cephadm 2021-07-31T03:24:33.016767+0000 mgr.x (mgr.24542) 4169 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:24:34.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: cephadm 2021-07-31T03:24:34.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: 2021-07-31T03:24:33.027814+0000 mgr.x (mgr. 2021-07-31T03:24:34.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: 24542) 4170 : cephadm [ERR] 2021-07-31T03:24:34.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:24:34.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:24:34.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: yield 2021-07-31T03:24:34.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:24:34.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:24:34.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:24:34.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:24:34.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:24:34.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:24:34.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:24:34.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:24:34.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:24:34.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: task.result() 2021-07-31T03:24:34.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:24:34.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: cephadm 2021-07-31T03:24:33.028669+0000 mgr.x (mgr.24542) 4171 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:24:34.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: cephadm 2021-07-31T03:24:33. 2021-07-31T03:24:34.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: 053287+0000 mgr.x (mgr.24542) 4172 : cephadm [ERR] 2021-07-31T03:24:34.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:24:34.800 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:24:34.800 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:24:34.800 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: yield 2021-07-31T03:24:34.800 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:24:34.801 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:24:34.801 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:24:34.801 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:24:34.801 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:24:34.801 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:24:34.802 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:24:34.802 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:24:34.802 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:24:34.802 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: task.result() 2021-07-31T03:24:34.802 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:24:34.803 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: cephadm 2021- 2021-07-31T03:24:34.803 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: 07-31T03:24:33 2021-07-31T03:24:34.803 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: .053673+0000 mgr.x (mgr 2021-07-31T03:24:34.803 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: .24542) 4173 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:24:34.803 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:24:34.804 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:24:34.804 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: yield 2021-07-31T03:24:34.804 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:24:34.804 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:24:34.804 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:24:34.804 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:24:34.805 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:24:34.805 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:24:34.805 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:24:34.805 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:24:34.805 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:24:34.806 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: task.result() 2021-07-31T03:24:34.806 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:24:34.806 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: 2021-07-31T03:24:34.806 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:24:34.806 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: 2021-07-31T03:24:34.807 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:24:34.807 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:24:34.807 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:24:34.807 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:24:34.807 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:24:34.808 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:24:34.808 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:24:34.808 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:24:34.808 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:24:34.809 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: yield 2021-07-31T03:24:34.809 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:24:34.809 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:24:34.809 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:24:34.809 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:24:34.810 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:24:34.810 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:24:34.810 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:24:34.810 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:24:34.810 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:24:34.811 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: task.result() 2021-07-31T03:24:34.811 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:24:34.811 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: cephadm 2021-07-31T03:24:33.053673+0000 mgr.x (mgr.24542) 4173 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:24:34.811 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:24:34.811 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:24:34.812 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: yield 2021-07-31T03:24:34.812 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:24:34.812 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:24:34.812 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:24:34.812 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:24:34.813 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:24:34.813 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:24:34.813 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:24:34.813 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:24:34.814 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:24:34.814 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: task.result() 2021-07-31T03:24:34.814 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:24:34.814 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: 2021-07-31T03:24:34.814 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:24:34.815 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: 2021-07-31T03:24:34.815 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:24:34.815 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:24:34.815 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:24:34.815 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:24:34.816 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:24:34.816 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:24:34.816 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:24:34.816 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:24:34.816 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:24:34.817 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:24:34.817 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:24:34.817 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:24:34.818 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:24:34.818 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:24:34.818 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:24:34.818 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:24:34.819 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:24:34.819 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:24:34.819 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:24:34.819 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: raise 2021-07-31T03:24:34.819 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:24:34.820 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:24:34.820 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:24:34.820 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:24:34.820 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:24:34.821 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: 2021-07-31T03:24:34.821 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: cephadm 2021-07-31T03:24:33.062259+0000 mgr.x (mgr.24542) 4174 : cephadm [ERR] 2021-07-31T03:24:34.821 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:24:34.821 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:24:34.821 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: yield 2021-07-31T03:24:34.822 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:24:34.822 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:24:34.822 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:24:34.822 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:24:34.822 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:24:34.823 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:24:34.823 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:24:34.823 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:24:34.823 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:24:34.823 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: task.result() 2021-07-31T03:24:34.824 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:24:34.824 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: cephadm 2021-07-31T03:24:33.062671+0000 mgr.x (mgr.24542) 4175 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:24:34.824 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:24:34.824 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:24:34.824 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: yield 2021-07-31T03:24:34.825 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:24:34.825 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:24:34.825 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:24:34.825 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:24:34.826 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:24:34.826 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:24:34.826 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:24:34.826 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:24:34.827 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:24:34.827 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:24:34.827 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:24:34.827 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:24:34.827 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: raise 2021-07-31T03:24:34.828 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:24:34.828 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:24:34.828 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:24:34.828 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:24:34.828 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:24:34.829 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: cephadm 2021-07-31 2021-07-31T03:24:34.829 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: T03:24:33.062259+0000 mgr.x (mgr.24542) 4174 : cephadm [ERR] 2021-07-31T03:24:34.829 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:24:34.829 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:24:34.829 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: yield 2021-07-31T03:24:34.830 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:24:34.830 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:24:34.830 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:24:34.830 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:24:34.831 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:24:34.831 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:24:34.831 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:24:34.831 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:24:34.831 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:24:34.832 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: task.result() 2021-07-31T03:24:34.832 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:24:34.832 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: cephadm 2021-07-31T03:24:33.062671+0000 mgr.x (mgr.24542) 4175 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:24:34.832 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:24:34.832 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:24:34.833 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: yield 2021-07-31T03:24:34.833 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:24:34.833 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:24:34.833 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:24:34.833 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:24:34.833 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:24:34.834 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:24:34.834 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:24:34.834 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:24:34.835 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:24:34.835 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:24:34.835 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:24:34.835 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: task.result() 2021-07-31T03:24:34.836 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:24:34.836 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: 2021-07-31T03:24:34.836 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:24:34.836 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: 2021-07-31T03:24:34.836 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:24:34.837 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:24:34.837 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:24:34.837 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:24:34.837 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:24:34.837 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:24:34.837 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:24:34.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:24:34.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:24:34.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:24:34.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:24:34.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:24:34.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:24:34.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:24:34.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:24:34.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:24:34.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:24:34.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:24:34.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: raise 2021-07-31T03:24:34.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:24:34.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:24:34.841 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:24:34.841 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:24:34.841 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:24:34.841 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: cluster 2021-07- 2021-07-31T03:24:34.841 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[33655]: 31T03:24:33.187969+0000 mgr.x (mgr.24542) 4176 : cluster [DBG] pgmap v4927: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:34.842 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:24:34.842 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:24:34.842 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:24:34.842 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:24:34.842 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: task.result() 2021-07-31T03:24:34.843 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:24:34.843 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: 2021-07-31T03:24:34.843 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:24:34.843 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: 2021-07-31T03:24:34.843 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:24:34.844 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:24:34.844 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:24:34.844 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:24:34.844 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:24:34.844 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:24:34.845 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:24:34.845 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:24:34.845 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:24:34.845 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:24:34.845 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:24:34.846 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:24:34.846 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:24:34.846 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:24:34.846 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:24:34.847 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:24:34.847 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:24:34.847 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:24:34.847 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: raise 2021-07-31T03:24:34.847 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:24:34.848 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:24:34.848 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:24:34.848 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:24:34.848 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:24:34.848 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:34 smithi045 conmon[39718]: cluster 2021-07-31T03:24:33.187969+0000 mgr.x (mgr.24542) 4176 : cluster [DBG] pgmap v4927: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:36.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:36 smithi154 conmon[34553]: cluster 2021-07-31T03:24:35.188373+0000 mgr.x (mgr.24542) 4177 : cluster [DBG] pgmap v4928: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:36.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:36 smithi045 conmon[39718]: cluster 2021-07-31T03:24:35.188373+0000 mgr.x 2021-07-31T03:24:36.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:36 smithi045 conmon[39718]: (mgr.24542) 4177 : cluster [DBG] pgmap v4928: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:36.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:36 smithi045 conmon[33655]: cluster 2021-07-31T03:24:35.188373+0000 2021-07-31T03:24:36.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:36 smithi045 conmon[33655]: mgr.x (mgr.24542) 4177 : cluster [DBG] pgmap v4928: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:38.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:38 smithi154 conmon[34553]: cluster 2021-07-31T03:24:37.188771+0000 mgr.x (mgr.24542) 4178 : cluster [DBG] pgmap v4929: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:38.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:38 smithi045 conmon[33655]: cluster 2021-07-31T03:24:37.188771+0000 mgr.x (mgr.24542) 4178 : cluster [DBG] pgmap v4929: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:38.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:38 smithi045 conmon[39718]: cluster 2021-07-31T03:24:37.188771+0000 mgr.x (mgr 2021-07-31T03:24:38.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:38 smithi045 conmon[39718]: .24542) 4178 : cluster [DBG] pgmap v4929: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:40.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:40 smithi154 conmon[34553]: cluster 2021-07-31T03:24:39.189218+0000 mgr.x (mgr.24542) 4179 : cluster [DBG] pgmap v4930: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:40.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:40 smithi045 conmon[33655]: cluster 2021-07-31T03:24:39.189218+0000 mgr.x (mgr.24542) 4179 : cluster [DBG] 2021-07-31T03:24:40.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:40 smithi045 conmon[33655]: pgmap v4930: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:40.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:40 smithi045 conmon[39718]: cluster 2021-07-31T03:24:40.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:40 smithi045 conmon[39718]: 2021-07-31T03:24:40.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:40 smithi045 conmon[39718]: 2021 2021-07-31T03:24:40.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:40 smithi045 conmon[39718]: - 2021-07-31T03:24:40.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:40 smithi045 conmon[39718]: 07 2021-07-31T03:24:40.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:40 smithi045 conmon[39718]: - 2021-07-31T03:24:40.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:40 smithi045 conmon[39718]: 31T03:24:39.189218+0000 mgr.x ( 2021-07-31T03:24:40.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:40 smithi045 conmon[39718]: mgr.24542) 4179 : cluster [DBG] pgmap v4930: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:42.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:42 smithi154 conmon[34553]: cluster 2021-07-31T03:24:41.189728+0000 mgr.x (mgr.24542) 4180 : cluster [DBG] 2021-07-31T03:24:42.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:42 smithi154 conmon[34553]: pgmap v4931: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:42.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:42 smithi045 conmon[33655]: cluster 2021-07-31T03:24:41.189728+0000 mgr.x (mgr.24542) 4180 : cluster [DBG] pgmap v4931: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:42.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:42 smithi045 conmon[39718]: cluster 2021-07-31T03:24:41.189728+0000 mgr.x (mgr. 2021-07-31T03:24:42.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:42 smithi045 conmon[39718]: 24542) 4180 : cluster [DBG] pgmap v4931: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:44.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:44 smithi154 conmon[34553]: cluster 2021-07-31T03:24:43.190317+0000 mgr.x (mgr.24542) 4181 : cluster [DBG] 2021-07-31T03:24:44.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:44 smithi154 conmon[34553]: pgmap v4932: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:44.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:44 smithi045 conmon[39718]: cluster 2021-07-31T03:24:43.190317+0000 2021-07-31T03:24:44.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:44 smithi045 conmon[39718]: mgr.x (mgr.24542) 4181 : cluster [DBG] pgmap v4932: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:44.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:44 smithi045 conmon[33655]: cluster 2021-07-31T03:24:43.190317+0000 mgr.x 2021-07-31T03:24:44.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:44 smithi045 conmon[33655]: (mgr.24542) 4181 : cluster [DBG] pgmap v4932: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:46.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:46 smithi154 conmon[34553]: cluster 2021-07-31T03:24:45.190787+0000 mgr.x (mgr.24542) 4182 : cluster [DBG] pgmap v4933: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:46.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:46 smithi045 conmon[39718]: cluster 2021-07-31T03:24:45.190787+0000 mgr.x (mgr 2021-07-31T03:24:46.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:46 smithi045 conmon[39718]: .24542) 4182 : cluster [DBG] pgmap v4933: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:46.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:46 smithi045 conmon[33655]: cluster 2021-07-31T03:24:45.190787+0000 mgr.x (mgr.24542) 4182 2021-07-31T03:24:46.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:46 smithi045 conmon[33655]: : cluster [DBG] pgmap v4933: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:48.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:48 smithi154 conmon[34553]: cluster 2021-07-31T03:24:47.191249+0000 mgr.x (mgr.24542) 4183 : cluster [DBG] pgmap v4934: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:48.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:48 smithi045 conmon[39718]: cluster 2021-07-31T03:24:47 2021-07-31T03:24:48.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:48 smithi045 conmon[39718]: .191249+0000 mgr.x (mgr.24542) 4183 : cluster [DBG] pgmap v4934: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:48.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:48 smithi045 conmon[33655]: cluster 2021-07-31T03:24:47.191249+0000 mgr.x (mgr.24542) 4183 2021-07-31T03:24:48.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:48 smithi045 conmon[33655]: : cluster [DBG] pgmap v4934: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:50.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:50 smithi154 conmon[34553]: cluster 2021-07-31T03:24:49.191626+0000 mgr.x (mgr.24542) 4184 : cluster [DBG] pgmap v4935: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:50.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:50 smithi045 conmon[39718]: cluster 2021-07-31T03:24:49. 2021-07-31T03:24:50.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:50 smithi045 conmon[39718]: 191626+0000 mgr.x (mgr.24542) 4184 : cluster [DBG] pgmap v4935: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:50.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:50 smithi045 conmon[33655]: cluster 2021-07-31T03:24:49.191626+0000 mgr.x (mgr.24542) 4184 : cluster [DBG] pgmap v4935: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:52.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:52 smithi154 conmon[34553]: cluster 2021-07-31T03:24:51.192037+0000 mgr.x (mgr.24542) 4185 : cluster [DBG] pgmap v4936: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:52.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:52 smithi045 conmon[39718]: cluster 2021-07-31T03:24:51.192037+0000 mgr.x (mgr.24542) 4185 : cluster [DBG] 2021-07-31T03:24:52.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:52 smithi045 conmon[39718]: pgmap v4936: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:52.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:52 smithi045 conmon[33655]: cluster 2021-07-31T03:24:51.192037+0000 mgr.x (mgr.24542) 2021-07-31T03:24:52.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:52 smithi045 conmon[33655]: 4185 : cluster [DBG] pgmap v4936: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:53.000 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:24:53.279 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:24:53.279 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 99m ago 100m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:24:53.280 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 97m ago 100m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:24:53.280 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 97m ago 103m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:24:53.280 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 99m ago 105m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:24:53.281 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 99m ago 106m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:24:53.281 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 97m ago 104m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:24:53.281 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 99m ago 104m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:24:53.282 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 99m ago 100m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:24:53.282 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 97m ago 100m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:24:53.282 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 99m ago 103m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:24:53.282 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 99m ago 103m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:24:53.283 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 99m ago 102m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:24:53.283 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 99m ago 102m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:24:53.283 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 97m ago 102m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:24:53.284 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 97m ago 101m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:24:53.284 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 97m ago 101m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:24:53.284 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 97m ago 101m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:24:53.284 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 97m ago 100m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:24:53.612 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:24:53.613 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:24:53.613 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:24:53.613 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:24:53.614 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:24:53.614 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:24:53.614 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:24:53.614 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:24:53.615 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:24:53.615 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:24:53.615 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:24:53.616 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:24:53.616 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:24:53.616 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:24:53.616 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:24:53.617 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:24:53.617 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:24:54.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:54 smithi154 conmon[34553]: audit 2021-07-31T03:24:52.989394+0000 mgr.x (mgr.24542) 4186 : audit [DBG] from='client.19452 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:24:54.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:54 smithi154 conmon[34553]: cluster 2021-07-31T03:24:53.192381+0000 mgr.x (mgr. 2021-07-31T03:24:54.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:54 smithi154 conmon[34553]: 24542) 4187 : cluster [DBG] pgmap v4937: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:54.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:54 smithi154 conmon[34553]: audit 2021-07-31T03:24:53.272163+0000 mgr.x (mgr.24542) 4188 : audit [DBG] from='client.29398 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:24:54.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:54 smithi154 conmon[34553]: audit 2021-07-31T03:24:53.613497+0000 mon.c (mon.1) 202 : audit [DBG] from='client.? 172.21.15.45:0/3149798777' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:24:54.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:54 smithi045 conmon[39718]: audit 2021-07-31T03:24:52.989394+0000 mgr.x (mgr.24542) 4186 : audit 2021-07-31T03:24:54.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:54 smithi045 conmon[39718]: [DBG] from='client.19452 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:24:54.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:54 smithi045 conmon[39718]: cluster 2021-07-31T03:24:53. 2021-07-31T03:24:54.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:54 smithi045 conmon[39718]: 192381+0000 mgr.x (mgr.24542) 4187 : cluster [DBG] pgmap v4937: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:54.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:54 smithi045 conmon[39718]: audit 2021-07-31T03:24:53.272163+0000 mgr.x 2021-07-31T03:24:54.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:54 smithi045 conmon[39718]: (mgr.24542) 4188 : audit [DBG] from='client.29398 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:24:54.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:54 smithi045 conmon[39718]: audit 2021-07-31T03:24:53.613497 2021-07-31T03:24:54.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:54 smithi045 conmon[39718]: +0000 mon.c (mon.1) 202 : audit [DBG] from='client.? 172.21.15.45:0/3149798777' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:24:54.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:54 smithi045 conmon[33655]: audit 2021-07-31T03:24:52.989394+0000 mgr.x (mgr.24542) 4186 : audit [DBG] 2021-07-31T03:24:54.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:54 smithi045 conmon[33655]: from='client.19452 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:24:54.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:54 smithi045 conmon[33655]: cluster 2021-07-31T03:24:53.192381+0000 mgr.x (mgr.24542) 4187 : cluster [DBG] pgmap v4937: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:54.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:54 smithi045 conmon[33655]: audit 2021-07-31T03:24 2021-07-31T03:24:54.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:54 smithi045 conmon[33655]: :53.272163+0000 mgr.x (mgr.24542) 4188 : audit [DBG] from='client.29398 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:24:54.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:54 smithi045 conmon[33655]: audit 2021- 2021-07-31T03:24:54.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:54 smithi045 conmon[33655]: 07-31T03:24:53.613497+0000 mon.c (mon 2021-07-31T03:24:54.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:54 smithi045 conmon[33655]: .1) 202 : audit [DBG] from='client.? 172.21.15.45:0/3149798777' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:24:56.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:56 smithi154 conmon[34553]: cluster 2021-07-31T03:24:55.192911+0000 mgr.x (mgr.24542) 4189 : cluster [DBG] 2021-07-31T03:24:56.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:56 smithi154 conmon[34553]: pgmap v4938: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:56.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:56 smithi045 conmon[39718]: cluster 2021-07-31T03:24:55.192911+0000 mgr.x ( 2021-07-31T03:24:56.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:56 smithi045 conmon[39718]: mgr.24542) 4189 : cluster [DBG] pgmap v4938: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:56.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:56 smithi045 conmon[33655]: cluster 2021-07-31T03:24:55.192911+0000 mgr.x (mgr.24542) 2021-07-31T03:24:56.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:56 smithi045 conmon[33655]: 4189 : cluster [DBG] pgmap v4938: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:58.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:24:58 smithi154 conmon[34553]: cluster 2021-07-31T03:24:57.193330+0000 mgr.x (mgr.24542) 4190 : cluster [DBG] pgmap v4939: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:58.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:58 smithi045 conmon[39718]: cluster 2021-07-31T03:24:57.193330+0000 mgr.x (mgr.24542) 4190 : 2021-07-31T03:24:58.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:24:58 smithi045 conmon[39718]: cluster [DBG] pgmap v4939: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:24:58.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:58 smithi045 conmon[33655]: cluster 2021-07-31T03:24:57.193330+0000 mgr.x (mgr.24542) 4190 : cluster 2021-07-31T03:24:58.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:24:58 smithi045 conmon[33655]: [DBG] pgmap v4939: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:00.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:00 smithi154 conmon[34553]: cluster 2021-07-31T03:24:59.193857+0000 mgr.x (mgr.24542) 4191 : cluster [DBG] pgmap v4940: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:00.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:00 smithi045 conmon[39718]: cluster 2021-07-31T03:24:59.193857+0000 mgr.x (mgr.24542) 4191 : cluster 2021-07-31T03:25:00.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:00 smithi045 conmon[39718]: [DBG] pgmap v4940: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:00.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:00 smithi045 conmon[33655]: cluster 2021-07-31T03:24:59.193857+0000 mgr.x ( 2021-07-31T03:25:00.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:00 smithi045 conmon[33655]: mgr.24542) 4191 : cluster [DBG] pgmap v4940: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:02.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:02 smithi154 conmon[34553]: cluster 2021-07-31T03:25:01.194296+0000 mgr.x (mgr.24542) 4192 : cluster [DBG] pgmap v4941: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:02.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:02 smithi045 conmon[39718]: cluster 2021-07-31T03:25:01.194296+0000 2021-07-31T03:25:02.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:02 smithi045 conmon[39718]: mgr.x (mgr.24542) 4192 : cluster [DBG] pgmap v4941: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:02.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:02 smithi045 conmon[33655]: cluster 2021-07-31T03:25:01.194296+0000 mgr.x (mgr.24542) 4192 : cluster 2021-07-31T03:25:02.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:02 smithi045 conmon[33655]: [DBG] pgmap v4941: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:04.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:04 smithi154 conmon[34553]: cluster 2021-07-31T03:25:03.194796+0000 mgr.x (mgr.24542) 4193 : cluster [DBG] pgmap v4942: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:04.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:04 smithi045 conmon[39718]: cluster 2021-07-31T03:25:03.194796+0000 2021-07-31T03:25:04.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:04 smithi045 conmon[39718]: mgr.x (mgr.24542) 4193 : cluster [DBG] pgmap v4942: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:04.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:04 smithi045 conmon[33655]: cluster 2021-07-31T03:25:03.194796+0000 mgr.x (mgr.24542 2021-07-31T03:25:04.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:04 smithi045 conmon[33655]: ) 4193 : cluster [DBG] pgmap v4942: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:06.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:06 smithi154 conmon[34553]: cluster 2021-07-31T03:25:05.195432+0000 mgr.x (mgr.24542) 4194 : cluster [DBG] pgmap v4943: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:06.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:06 smithi045 conmon[39718]: cluster 2021-07-31T03:25:05.195432+0000 mgr.x 2021-07-31T03:25:06.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:06 smithi045 conmon[39718]: (mgr.24542) 4194 : cluster [DBG] pgmap v4943: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:06.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:06 smithi045 conmon[33655]: cluster 2021-07-31T03:25:05.195432+0000 mgr.x (mgr.24542) 4194 : 2021-07-31T03:25:06.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:06 smithi045 conmon[33655]: cluster [DBG] pgmap v4943: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:08.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:08 smithi154 conmon[34553]: cluster 2021-07-31T03:25:07.195885+0000 mgr.x (mgr.24542) 4195 : cluster [DBG] pgmap v4944: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:08.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:08 smithi154 conmon[34553]: 2021-07-31T03:25:08.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:08 smithi045 conmon[33655]: cluster 2021-07-31T03:25:07.195885+0000 mgr.x (mgr 2021-07-31T03:25:08.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:08 smithi045 conmon[33655]: .24542) 4195 : cluster [DBG] pgmap v4944: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:08.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:08 smithi045 conmon[39718]: cluster 2021-07-31T03:25:07.195885+0000 2021-07-31T03:25:08.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:08 smithi045 conmon[39718]: mgr.x (mgr.24542) 4195 : cluster [DBG] pgmap v4944: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:10.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:10 smithi154 conmon[34553]: cluster 2021-07-31T03:25:09.196418+0000 mgr.x (mgr.24542) 4196 : cluster [DBG] 2021-07-31T03:25:10.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:10 smithi154 conmon[34553]: pgmap v4945: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:10.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:10 smithi045 conmon[39718]: cluster 2021-07-31T03:25:09.196418+0000 2021-07-31T03:25:10.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:10 smithi045 conmon[39718]: mgr.x (mgr.24542) 4196 : cluster [DBG] pgmap v4945: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:10.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:10 smithi045 conmon[33655]: cluster 2021-07-31T03:25:09.196418+0000 mgr.x (mgr.24542) 4196 : cluster [DBG] 2021-07-31T03:25:10.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:10 smithi045 conmon[33655]: pgmap v4945: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:12.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:12 smithi154 conmon[34553]: cluster 2021-07-31T03:25:11.196847+0000 mgr.x (mgr.24542) 2021-07-31T03:25:12.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:12 smithi154 conmon[34553]: 4197 : cluster [DBG] pgmap v4946: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:12.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:12 smithi045 conmon[39718]: cluster 2021-07-31T03:25:11.196847+0000 mgr.x (mgr.24542) 4197 : cluster [DBG] pgmap v4946: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:12.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:12 smithi045 conmon[33655]: cluster 2021-07-31T03:25:11.196847+0000 mgr.x (mgr.24542) 4197 : cluster [DBG] pgmap v4946: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:14.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:14 smithi154 conmon[34553]: cluster 2021-07-31T03:25:13.197307+0000 mgr.x (mgr.24542) 4198 : cluster [DBG] pgmap v4947: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:14.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:14 smithi045 conmon[33655]: cluster 2021-07-31T03:25:13.197307+0000 mgr.x (mgr.24542) 4198 : cluster [DBG] pgmap v4947: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:14.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:14 smithi045 conmon[39718]: cluster 2021-07-31T03:25:13.197307+0000 mgr.x (mgr. 2021-07-31T03:25:14.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:14 smithi045 conmon[39718]: 24542) 4198 : cluster [DBG] pgmap v4947: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:16.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:16 smithi154 conmon[34553]: cluster 2021-07-31T03:25:15.197676+0000 mgr.x (mgr.24542) 4199 : cluster [DBG] pgmap v4948: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:16.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:16 smithi154 conmon[34553]: 2021-07-31T03:25:16.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:16 smithi045 conmon[33655]: cluster 2021-07-31T03:25:15.197676+0000 mgr.x (mgr.24542) 4199 : cluster [DBG] pgmap v4948: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:16.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:16 smithi045 conmon[39718]: cluster 2021-07-31T03:25:15.197676+0000 mgr.x (mgr.24542) 4199 : 2021-07-31T03:25:16.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:16 smithi045 conmon[39718]: cluster [DBG] pgmap v4948: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:18.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:18 smithi154 conmon[34553]: cluster 2021-07-31T03:25:17.198141+0000 mgr.x (mgr.24542) 4200 : cluster [DBG] pgmap v4949: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:18.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:18 smithi045 conmon[39718]: cluster 2021-07-31T03:25:17.198141+0000 mgr.x ( 2021-07-31T03:25:18.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:18 smithi045 conmon[39718]: mgr.24542) 4200 : cluster [DBG] pgmap v4949: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:18.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:18 smithi045 conmon[33655]: cluster 2021-07-31T03:25:17.198141+0000 mgr.x (mgr.24542 2021-07-31T03:25:18.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:18 smithi045 conmon[33655]: ) 4200 : cluster [DBG] pgmap v4949: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:20.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:20 smithi154 conmon[34553]: cluster 2021-07-31T03:25:19.198529+0000 mgr.x (mgr.24542) 4201 : cluster [DBG] pgmap v4950: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:20.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:20 smithi045 conmon[33655]: cluster 2021-07-31T03:25:19. 2021-07-31T03:25:20.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:20 smithi045 conmon[33655]: 198529+0000 mgr.x (mgr.24542) 4201 : cluster [DBG] pgmap v4950: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:20.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:20 smithi045 conmon[39718]: cluster 2021-07-31T03:25:19.198529+0000 mgr.x ( 2021-07-31T03:25:20.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:20 smithi045 conmon[39718]: mgr.24542) 4201 : cluster [DBG] pgmap v4950: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:22.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:22 smithi154 conmon[34553]: cluster 2021-07-31T03:25:21.199070+0000 mgr.x (mgr.24542) 4202 : cluster [DBG] pgmap v4951: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:22.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:22 smithi045 conmon[39718]: cluster 2021-07-31T03:25:21.199070+0000 mgr.x (mgr. 2021-07-31T03:25:22.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:22 smithi045 conmon[39718]: 24542) 4202 : cluster [DBG] pgmap v4951: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:22.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:22 smithi045 conmon[33655]: cluster 2021-07-31T03:25:21.199070+0000 mgr.x (mgr.24542) 4202 : cluster [DBG] pgmap v4951: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:23.926 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:25:24.203 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:25:24.203 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 100m ago 101m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:25:24.204 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 97m ago 100m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:25:24.204 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 97m ago 104m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:25:24.204 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 100m ago 106m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:25:24.205 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 100m ago 106m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:25:24.205 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 97m ago 104m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:25:24.205 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 100m ago 104m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:25:24.205 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 100m ago 101m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:25:24.206 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 97m ago 101m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:25:24.206 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 100m ago 104m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:25:24.206 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 100m ago 103m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:25:24.206 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 100m ago 103m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:25:24.207 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 100m ago 103m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:25:24.207 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 97m ago 102m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:25:24.207 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 97m ago 102m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:25:24.207 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 97m ago 102m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:25:24.208 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 97m ago 101m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:25:24.208 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 97m ago 101m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:25:24.538 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:25:24.538 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:25:24.538 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:25:24.539 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:25:24.539 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:25:24.539 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:25:24.540 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:25:24.540 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:25:24.540 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:25:24.540 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:25:24.541 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:25:24.541 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:25:24.541 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:25:24.542 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:25:24.542 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:25:24.542 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:25:24.542 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:25:24.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:24 smithi154 conmon[34553]: cluster 2021-07-31T03:25:23.199624+0000 mgr.x (mgr.24542) 4203 : cluster [DBG] pgmap v4952: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:24.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:24 smithi154 conmon[34553]: audit 2021-07-31T03: 2021-07-31T03:25:24.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:24 smithi154 conmon[34553]: 25:23.914513+0000 mgr.x (mgr.24542) 4204 : audit [DBG] from='client.29410 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:25:24.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:24 smithi045 conmon[33655]: cluster 2021-07-31T03:25:23.199624+0000 mgr.x (mgr 2021-07-31T03:25:24.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:24 smithi045 conmon[33655]: .24542) 4203 : cluster [DBG] pgmap v4952: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:24.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:24 smithi045 conmon[33655]: audit 2021-07-31T03:25:23.914513+0000 mgr.x (mgr.24542) 4204 : audit 2021-07-31T03:25:24.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:24 smithi045 conmon[33655]: [DBG] from='client.29410 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:25:24.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:24 smithi045 conmon[39718]: cluster 2021-07-31T03 2021-07-31T03:25:24.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:24 smithi045 conmon[39718]: :25:23.199624+0000 mgr.x (mgr.24542) 4203 : cluster [DBG] pgmap v4952: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:24.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:24 smithi045 conmon[39718]: audit 2021- 2021-07-31T03:25:24.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:24 smithi045 conmon[39718]: 07-31T03:25:23.914513+0000 mgr.x (mgr 2021-07-31T03:25:24.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:24 smithi045 conmon[39718]: .24542) 4204 : audit [DBG] from='client.29410 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:25:25.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:25 smithi154 conmon[34553]: audit 2021-07-31T03:25:24.195788+0000 mgr.x (mgr.24542) 4205 : audit [DBG] from='client.29416 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:25:25.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:25 smithi154 conmon[34553]: audit 2021-07-31T03:25:25.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:25 smithi154 conmon[34553]: 2021-07-31T03:25:24.538845+0000 mon.c (mon.1) 203 : audit [DBG] from='client.? 172.21.15.45:0/389939077' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:25:25.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:25 smithi045 conmon[39718]: audit 2021-07-31T03:25:24.195788+0000 mgr.x (mgr.24542) 4205 : audit [DBG] from='client.29416 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:25:25.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:25 smithi045 conmon[39718]: audit 2021-07-31T03:25:24.538845+0000 mon.c (mon.1) 203 : audit [DBG] from='client.? 172.21.15.45:0/389939077' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:25:25.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:25 smithi045 conmon[33655]: audit 2021-07-31T03:25: 2021-07-31T03:25:25.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:25 smithi045 conmon[33655]: 24.195788+0000 mgr.x (mgr.24542) 4205 : audit [DBG] from='client.29416 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:25:25.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:25 smithi045 conmon[33655]: audit 2021- 2021-07-31T03:25:25.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:25 smithi045 conmon[33655]: 07-31T03:25:24.538845+0000 mon.c (mon.1 2021-07-31T03:25:25.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:25 smithi045 conmon[33655]: ) 203 : audit [DBG] from='client.? 172.21.15.45:0/389939077' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:25:26.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:26 smithi154 conmon[34553]: cluster 2021-07-31T03:25:25.200147+0000 mgr.x (mgr.24542) 4206 : cluster [DBG] pgmap v4953: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:26.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:26 smithi045 conmon[39718]: cluster 2021-07-31T03:25:25.200147+0000 mgr.x (mgr.24542) 4206 : cluster [DBG] pgmap v4953: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:26.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:26 smithi045 conmon[33655]: cluster 2021-07-31T03:25:25.200147+0000 2021-07-31T03:25:26.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:26 smithi045 conmon[33655]: mgr.x (mgr.24542) 4206 : cluster [DBG] pgmap v4953: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:28.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:28 smithi154 conmon[34553]: cluster 2021-07-31T03:25:27.200642+0000 mgr.x (mgr.24542) 4207 : cluster [DBG] pgmap v4954: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:28.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:28 smithi154 conmon[34553]: audit 2021-07-31T03:25:28.325612+0000 2021-07-31T03:25:28.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:28 smithi154 conmon[34553]: mon.b (mon.2) 565 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:25:28.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:28 smithi154 conmon[34553]: audit 2021-07-31T03:25:28.325860+0000 mon.b (mon.2) 566 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:25:28.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:28 smithi154 conmon[34553]: audit 2021-07-31T03:25:28.326247+0000 mon.a (mon.0) 3301 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:25:28.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:28 smithi154 conmon[34553]: 2021-07-31T03:25:28.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:28 smithi154 conmon[34553]: audit 2021-07-31T03:25:28.326625 2021-07-31T03:25:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:28 smithi154 conmon[34553]: +0000 mon.a (mon.0) 3302 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:25:28.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:28 smithi045 conmon[33655]: cluster 2021-07-31T03:25:27.200642+0000 2021-07-31T03:25:28.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:28 smithi045 conmon[33655]: mgr.x (mgr.24542) 4207 : cluster [DBG] pgmap v4954: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:28.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:28 smithi045 conmon[33655]: audit 2021-07-31T03:25:28.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:28 smithi045 conmon[33655]: 2021-07-31T03:25:28.325612+0000 mon.b (mon 2021-07-31T03:25:28.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:28 smithi045 conmon[33655]: .2) 565 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:25:28.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:28 smithi045 conmon[33655]: audit 2021-07- 2021-07-31T03:25:28.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:28 smithi045 conmon[33655]: 31T03:25:28.325860+0000 mon.b 2021-07-31T03:25:28.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:28 smithi045 conmon[33655]: (mon.2) 566 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:25:28.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:28 smithi045 conmon[33655]: 2021-07-31T03:25:28.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:28 smithi045 conmon[33655]: audit 2021-07 2021-07-31T03:25:28.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:28 smithi045 conmon[33655]: -31T03:25:28.326247+0000 mon.a (mon.0) 3301 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:25:28.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:28 smithi045 conmon[33655]: audit 2021-07 2021-07-31T03:25:28.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:28 smithi045 conmon[33655]: -31T03:25:28.326625+0000 2021-07-31T03:25:28.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:28 smithi045 conmon[33655]: mon.a (mon.0) 3302 : audit [INF] 2021-07-31T03:25:28.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:28 smithi045 conmon[33655]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:25:28.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:28 smithi045 conmon[39718]: cluster 2021-07-31T03:25:27.200642+0000 mgr.x (mgr.24542) 4207 : cluster [DBG] pgmap v4954: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:28.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:28 smithi045 conmon[39718]: audit 2021-07-31T03:25:28.325612+0000 mon.b (mon 2021-07-31T03:25:28.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:28 smithi045 conmon[39718]: .2) 565 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:25:28.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:28 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T03:25:28.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:28 smithi045 conmon[39718]: 31T03:25:28.325860+0000 2021-07-31T03:25:28.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:28 smithi045 conmon[39718]: mon.b (mon.2) 566 : audit [INF] 2021-07-31T03:25:28.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:28 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:25:28.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:28 smithi045 conmon[39718]: audit 2021 2021-07-31T03:25:28.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:28 smithi045 conmon[39718]: -07-31T03:25 2021-07-31T03:25:28.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:28 smithi045 conmon[39718]: :28.326247+0000 mon.a (mon.0) 3301 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:25:28.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:28 smithi045 conmon[39718]: audit 2021- 2021-07-31T03:25:28.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:28 smithi045 conmon[39718]: 07-31T03:25:28.326625 2021-07-31T03:25:28.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:28 smithi045 conmon[39718]: +0000 mon.a (mon.0) 3302 : 2021-07-31T03:25:28.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:28 smithi045 conmon[39718]: audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:25:30.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:30 smithi154 conmon[34553]: cluster 2021-07-31T03:25:29.201190+0000 mgr.x (mgr.24542) 4208 : cluster [DBG] pgmap v4955: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:30.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:30 smithi045 conmon[33655]: cluster 2021-07-31T03:25:29.201190+0000 mgr.x (mgr. 2021-07-31T03:25:30.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:30 smithi045 conmon[33655]: 24542) 4208 : cluster [DBG] pgmap v4955: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:30.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:30 smithi045 conmon[39718]: cluster 2021-07-31T03:25:29.201190+0000 mgr.x (mgr 2021-07-31T03:25:30.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:30 smithi045 conmon[39718]: .24542) 4208 : cluster [DBG] pgmap v4955: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:32.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:32 smithi154 conmon[34553]: cluster 2021-07-31T03:25:31.201712+0000 mgr.x (mgr.24542) 4209 : cluster [DBG] pgmap v4956: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:32.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:32 smithi045 conmon[39718]: cluster 2021-07-31T03:25:31.201712+0000 mgr.x (mgr.24542) 4209 : cluster [DBG] pgmap v4956: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:32.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:32 smithi045 conmon[33655]: cluster 2021-07-31T03:25:31.201712+0000 mgr.x (mgr.24542) 4209 : 2021-07-31T03:25:32.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:32 smithi045 conmon[33655]: cluster [DBG] pgmap v4956: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:33.447 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: debug 2021-07-31T03:25:33.109+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:25:33.447 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:25:33.447 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:25:33.448 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: yield 2021-07-31T03:25:33.448 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:25:33.448 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:25:33.448 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:25:33.448 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:25:33.449 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:25:33.449 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:25:33.449 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:25:33.449 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:25:33.450 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:25:33.450 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: task.result() 2021-07-31T03:25:33.450 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:25:33.450 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: debug 2021-07-31T03:25:33.120+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:25:33.450 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:25:33.451 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:25:33.451 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: yield 2021-07-31T03:25:33.451 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:25:33.451 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:25:33.452 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:25:33.452 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:25:33.452 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:25:33.452 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:25:33.452 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:25:33.453 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:25:33.453 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:25:33.453 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: task.result() 2021-07-31T03:25:33.453 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:25:33.453 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: debug 2021-07-31T03:25:33.146+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:25:33.454 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:25:33.454 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:25:33.454 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: yield 2021-07-31T03:25:33.454 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:25:33.455 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:25:33.455 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:25:33.455 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:25:33.455 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:25:33.455 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:25:33.456 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:25:33.456 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:25:33.456 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:25:33.456 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: task.result() 2021-07-31T03:25:33.457 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:25:33.457 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: debug 2021-07-31T03:25:33.146+0000 7fcb9095b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:25:33.457 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:25:33.457 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:25:33.457 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: yield 2021-07-31T03:25:33.458 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:25:33.458 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:25:33.458 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:25:33.458 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:25:33.458 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:25:33.459 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:25:33.459 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:25:33.459 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:25:33.459 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:25:33.459 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: task.result() 2021-07-31T03:25:33.460 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:25:33.460 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: 2021-07-31T03:25:33.460 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:25:33.460 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: 2021-07-31T03:25:33.461 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:25:33.461 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:25:33.461 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:25:33.461 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:25:33.461 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:25:33.462 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:25:33.462 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:25:33.462 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:25:33.462 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:25:33.462 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:25:33.463 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:25:33.463 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:25:33.463 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:25:33.463 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:25:33.463 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:25:33.464 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:25:33.464 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:25:33.464 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:25:33.465 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: raise 2021-07-31T03:25:33.465 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:25:33.465 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:25:33.465 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:25:33.466 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:25:33.466 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:25:33.466 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: debug 2021-07-31T03:25:33.155+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:25:33.466 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:25:33.466 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:25:33.467 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: yield 2021-07-31T03:25:33.467 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:25:33.467 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:25:33.467 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:25:33.468 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:25:33.468 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:25:33.468 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:25:33.468 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:25:33.468 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:25:33.469 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:25:33.469 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: task.result() 2021-07-31T03:25:33.469 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:25:33.469 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: debug 2021-07-31T03:25:33.155+0000 7fcb9115c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:25:33.469 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:25:33.470 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:25:33.470 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: yield 2021-07-31T03:25:33.470 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:25:33.470 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:25:33.470 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:25:33.471 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:25:33.471 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:25:33.471 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:25:33.471 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:25:33.472 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:25:33.472 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:25:33.472 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: task.result() 2021-07-31T03:25:33.472 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:25:33.472 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: 2021-07-31T03:25:33.473 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:25:33.473 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: 2021-07-31T03:25:33.473 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:25:33.473 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:25:33.473 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:25:33.474 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:25:33.474 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:25:33.474 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:25:33.474 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:25:33.474 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:25:33.475 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:25:33.475 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:25:33.475 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:25:33.475 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:25:33.475 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:25:33.476 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:25:33.476 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:25:33.476 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:25:33.476 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:25:33.476 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:25:33.477 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: raise 2021-07-31T03:25:33.477 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:25:33.477 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:25:33.477 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:25:33.477 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:25:33.478 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:25:33.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[34553]: audit 2021-07-31T03:25:33.063371+0000 mon.b (mon.2) 567 2021-07-31T03:25:33.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[34553]: : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:25:33.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[34553]: audit 2021-07-31T03:25:33.064894+0000 mon.b (mon.2) 568 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:25:33.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[34553]: audit 2021-07-31T03:25:33.065781+0000 mon.b (mon.2) 569 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:25:33.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:33 smithi154 conmon[34553]: 2021-07-31T03:25:33.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:33 smithi045 conmon[33655]: audit 2021-07-31T03:25:33. 2021-07-31T03:25:33.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:33 smithi045 conmon[33655]: 063371+0000 mon.b (mon.2) 567 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:25:33.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:33 smithi045 conmon[33655]: audit 2021-07-31T03:25:33.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:33 smithi045 conmon[33655]: 2021-07-31T03:25:33.064894+0000 2021-07-31T03:25:33.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:33 smithi045 conmon[33655]: mon.b (mon.2) 568 : audit [DBG] 2021-07-31T03:25:33.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:33 smithi045 conmon[33655]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:25:33.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:33 smithi045 conmon[33655]: audit 2021-07-31T 2021-07-31T03:25:33.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:33 smithi045 conmon[33655]: 03:25:33.065781+0000 mon.b (mon. 2021-07-31T03:25:33.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:33 smithi045 conmon[33655]: 2) 569 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:25:33.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:33 smithi045 conmon[39718]: audit 2021-07-31T03:25:33. 2021-07-31T03:25:33.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:33 smithi045 conmon[39718]: 063371+0000 mon.b (mon.2) 567 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:25:33.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:33 smithi045 conmon[39718]: audit 2021-07 2021-07-31T03:25:33.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:33 smithi045 conmon[39718]: -31T03:25:33.064894+0000 2021-07-31T03:25:33.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:33 smithi045 conmon[39718]: mon.b (mon.2) 568 2021-07-31T03:25:33.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:33 smithi045 conmon[39718]: : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:25:33.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:33 smithi045 conmon[39718]: audit 2021 2021-07-31T03:25:33.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:33 smithi045 conmon[39718]: -07-31T03:25:33. 2021-07-31T03:25:33.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:33 smithi045 conmon[39718]: 065781+0000 mon.b (mon. 2021-07-31T03:25:33.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:33 smithi045 conmon[39718]: 2) 569 : audit [INF] 2021-07-31T03:25:33.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:33 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:25:34.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: cephadm 2021-07-31T03:25:33.066577+0000 mgr.x (mgr.24542) 4210 : cephadm 2021-07-31T03:25:34.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:25:34.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: cephadm 2021-07-31T03:25:33.110493+0000 mgr.x (mgr.24542) 4211 : cephadm [ERR] 2021-07-31T03:25:34.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:25:34.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:25:34.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: yield 2021-07-31T03:25:34.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:25:34.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:25:34.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:25:34.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:25:34.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:25:34.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:25:34.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:25:34.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:25:34.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:25:34.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: task.result() 2021-07-31T03:25:34.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:25:34.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: cephadm 2021-07-31T03:25:33.110911+0000 mgr.x (mgr.24542) 4212 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:25:34.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: cephadm 2021-07-31T03:25:33.121790+0000 mgr.x (mgr.24542) 4213 : cephadm [ERR] 2021-07-31T03:25:34.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:25:34.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:25:34.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: yield 2021-07-31T03:25:34.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:25:34.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:25:34.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:25:34.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:25:34.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:25:34.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:25:34.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:25:34.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:25:34.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:25:34.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: task.result() 2021-07-31T03:25:34.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:25:34.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: cephadm 2021-07-31T03:25:33.122661+0000 mgr.x (mgr.24542) 4214 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:25:34.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: cephadm 2021-07-31T03:25:33.147450+0000 mgr.x (mgr.24542) 4215 : cephadm [ERR] 2021-07-31T03:25:34.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:25:34.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:25:34.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: yield 2021-07-31T03:25:34.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:25:34.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:25:34.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:25:34.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:25:34.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:25:34.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:25:34.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:25:34.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:25:34.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:25:34.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: task.result() 2021-07-31T03:25:34.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:25:34.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: cephadm 2021-07-31T03:25:33.147839+0000 mgr.x (mgr.24542) 4216 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:25:34.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:25:34.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:25:34.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: yield 2021-07-31T03:25:34.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:25:34.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:25:34.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:25:34.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:25:34.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:25:34.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:25:34.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:25:34.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:25:34.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:25:34.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: task.result() 2021-07-31T03:25:34.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:25:34.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: 2021-07-31T03:25:34.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:25:34.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: 2021-07-31T03:25:34.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:25:34.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:25:34.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:25:34.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:25:34.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:25:34.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:25:34.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:25:34.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:25:34.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:25:34.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:25:34.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:25:34.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:25:34.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:25:34.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:25:34.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:25:34.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:25:34.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:25:34.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:25:34.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: raise 2021-07-31T03:25:34.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:25:34.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:25:34.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:25:34.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:25:34.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:25:34.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: cephadm 2021-07-31T03:25:33.156381+0000 mgr.x (mgr.24542 2021-07-31T03:25:34.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: ) 4217 : cephadm [ERR] 2021-07-31T03:25:34.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:25:34.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:25:34.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: yield 2021-07-31T03:25:34.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:25:34.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:25:34.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:25:34.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:25:34.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:25:34.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:25:34.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:25:34.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:25:34.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:25:34.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: task.result() 2021-07-31T03:25:34.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:25:34.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: cephadm 2021-07-31T03:25:33.156788+0000 mgr.x (mgr.24542) 4218 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:25:34.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:25:34.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:25:34.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: yield 2021-07-31T03:25:34.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:25:34.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:25:34.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:25:34.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:25:34.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:25:34.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:25:34.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:25:34.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:25:34.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:25:34.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: task.result() 2021-07-31T03:25:34.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:25:34.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: 2021-07-31T03:25:34.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:25:34.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: 2021-07-31T03:25:34.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:25:34.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:25:34.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:25:34.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:25:34.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:25:34.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:25:34.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:25:34.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:25:34.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:25:34.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:25:34.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:25:34.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:25:34.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:25:34.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:25:34.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:25:34.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:25:34.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:25:34.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:25:34.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: raise 2021-07-31T03:25:34.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:25:34.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:25:34.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:25:34.782 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:25:34.782 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:25:34.782 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:34 smithi154 conmon[34553]: cluster 2021-07-31T03:25:33.202107+0000 mgr.x (mgr.24542) 4219 : cluster [DBG] pgmap v4957: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:34.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: cephadm 2021-07-31T03:25:33.066577+0000 mgr.x (mgr.24542) 4210 : cephadm 2021-07-31T03:25:34.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:25:34.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: cephadm 2021-07-31T03:25:33.110493+0000 2021-07-31T03:25:34.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: mgr.x (mgr.24542) 4211 : 2021-07-31T03:25:34.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: cephadm [ERR] 2021-07-31T03:25:34.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:25:34.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:25:34.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: yield 2021-07-31T03:25:34.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:25:34.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:25:34.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:25:34.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:25:34.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:25:34.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:25:34.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:25:34.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:25:34.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:25:34.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: task.result() 2021-07-31T03:25:34.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:25:34.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: cephadm 2021-07-31T03:25:33.110911+0000 mgr.x (mgr.24542) 4212 : 2021-07-31T03:25:34.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:25:34.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: cephadm 2021 2021-07-31T03:25:34.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: -07-31T03:25:33. 2021-07-31T03:25:34.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: 121790+0000 mgr.x (mgr.24542 2021-07-31T03:25:34.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: ) 4213 : cephadm [ERR] 2021-07-31T03:25:34.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:25:34.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:25:34.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: yield 2021-07-31T03:25:34.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:25:34.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:25:34.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:25:34.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:25:34.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:25:34.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:25:34.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:25:34.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:25:34.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:25:34.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: task.result() 2021-07-31T03:25:34.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:25:34.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: 2021-07-31T03:25:34.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: cephadm 2021-07-31T03:25:33.122661+0000 mgr.x (mgr.24542) 2021-07-31T03:25:34.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: 4214 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:25:34.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: cephadm 2021 2021-07-31T03:25:34.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: -07-31T03:25:33 2021-07-31T03:25:34.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: .147450+0000 mgr.x (mgr. 2021-07-31T03:25:34.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: 24542) 4215 : cephadm [ERR] 2021-07-31T03:25:34.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: 2021-07-31T03:25:34.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:25:34.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:25:34.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: yield 2021-07-31T03:25:34.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:25:34.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:25:34.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:25:34.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:25:34.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:25:34.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:25:34.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:25:34.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:25:34.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:25:34.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: task.result() 2021-07-31T03:25:34.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:25:34.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: cephadm 2021-07-31T03:25:33.147839+0000 mgr.x ( 2021-07-31T03:25:34.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: mgr.24542) 4216 : cephadm [ERR] 2021-07-31T03:25:34.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:25:34.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:25:34.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:25:34.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: yield 2021-07-31T03:25:34.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:25:34.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:25:34.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:25:34.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:25:34.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:25:34.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:25:34.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:25:34.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:25:34.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:25:34.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: task.result() 2021-07-31T03:25:34.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:25:34.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: 2021-07-31T03:25:34.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:25:34.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: 2021-07-31T03:25:34.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:25:34.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:25:34.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:25:34.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:25:34.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:25:34.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:25:34.800 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: cephadm 2021-07-31T03:25:33.066577+0000 mgr.x (mgr.24542) 4210 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:25:34.801 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: cephadm 2021-07-31T03:25:33.110493+0000 mgr.x (mgr.24542) 4211 : cephadm [ERR] 2021-07-31T03:25:34.801 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:25:34.801 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:25:34.801 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: yield 2021-07-31T03:25:34.801 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:25:34.802 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:25:34.802 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:25:34.802 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:25:34.802 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:25:34.802 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:25:34.803 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:25:34.803 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:25:34.803 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:25:34.803 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: task.result() 2021-07-31T03:25:34.803 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:25:34.804 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: cephadm 2021-07-31T03:25:33.110911+0000 mgr.x (mgr.24542 2021-07-31T03:25:34.804 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: ) 4212 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:25:34.804 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: cephadm 2021-07-31T03:25:33.121790+0000 mgr.x (mgr.24542) 4213 : cephadm [ERR] 2021-07-31T03:25:34.804 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:25:34.804 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:25:34.805 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: yield 2021-07-31T03:25:34.805 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:25:34.805 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:25:34.805 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:25:34.805 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:25:34.806 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:25:34.806 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:25:34.806 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:25:34.806 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:25:34.806 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:25:34.807 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: task.result() 2021-07-31T03:25:34.807 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:25:34.807 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: cephadm 2021-07-31T03:25:34.807 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: 2021-07-31T03:25:33.122661+0000 mgr.x (mgr.24542) 4214 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:25:34.807 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: cephadm 2021-07-31 2021-07-31T03:25:34.808 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: T03:25:33.147450+0000 mgr.x (mgr 2021-07-31T03:25:34.808 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: .24542) 4215 : cephadm [ERR] 2021-07-31T03:25:34.808 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: 2021-07-31T03:25:34.808 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:25:34.809 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:25:34.809 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:25:34.809 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:25:34.809 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:25:34.809 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:25:34.810 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:25:34.810 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:25:34.810 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:25:34.810 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:25:34.810 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:25:34.811 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:25:34.811 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:25:34.811 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: raise 2021-07-31T03:25:34.811 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:25:34.811 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:25:34.812 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:25:34.812 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:25:34.812 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:25:34.812 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: cephadm 2021-07-31T03:25:33.156381+0000 mgr.x (mgr.24542) 4217 : cephadm [ERR] 2021-07-31T03:25:34.812 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:25:34.813 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:25:34.813 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: yield 2021-07-31T03:25:34.813 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:25:34.813 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:25:34.813 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:25:34.814 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:25:34.814 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:25:34.814 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:25:34.814 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:25:34.814 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:25:34.815 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:25:34.815 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: task.result() 2021-07-31T03:25:34.815 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:25:34.815 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: cephadm 2021-07-31T03:25:33.156788 2021-07-31T03:25:34.815 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 4218 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:25:34.816 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:25:34.816 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:25:34.816 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: yield 2021-07-31T03:25:34.816 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:25:34.817 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:25:34.817 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: yield 2021-07-31T03:25:34.817 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:25:34.817 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:25:34.817 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:25:34.818 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:25:34.818 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:25:34.818 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:25:34.818 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:25:34.818 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:25:34.819 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:25:34.819 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: task.result() 2021-07-31T03:25:34.819 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:25:34.819 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: cephadm 2021-07-31T03:25:33.147839+0000 mgr.x (mgr.24542) 4216 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:25:34.819 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:25:34.820 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:25:34.820 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: yield 2021-07-31T03:25:34.820 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:25:34.820 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:25:34.820 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:25:34.821 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:25:34.821 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:25:34.821 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:25:34.821 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:25:34.821 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:25:34.822 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:25:34.822 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: task.result() 2021-07-31T03:25:34.822 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:25:34.822 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: 2021-07-31T03:25:34.822 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:25:34.823 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: 2021-07-31T03:25:34.823 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:25:34.823 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:25:34.823 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:25:34.823 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:25:34.824 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:25:34.824 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:25:34.824 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:25:34.824 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:25:34.824 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:25:34.825 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:25:34.825 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:25:34.825 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:25:34.825 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:25:34.826 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:25:34.826 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:25:34.826 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:25:34.826 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:25:34.826 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: task.result() 2021-07-31T03:25:34.827 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:25:34.827 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: 2021-07-31T03:25:34.827 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:25:34.827 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: 2021-07-31T03:25:34.827 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:25:34.828 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:25:34.828 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:25:34.828 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:25:34.828 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:25:34.828 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:25:34.829 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:25:34.829 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:25:34.829 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:25:34.829 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:25:34.829 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:25:34.830 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:25:34.830 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:25:34.830 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:25:34.830 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:25:34.830 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:25:34.831 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:25:34.831 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:25:34.831 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: raise 2021-07-31T03:25:34.831 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:25:34.831 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:25:34.832 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:25:34.832 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:25:34.832 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:25:34.832 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: cluster 2021-07 2021-07-31T03:25:34.832 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[39718]: -31T03:25:33.202107+0000 mgr.x (mgr.24542) 4219 : cluster [DBG] pgmap v4957: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:34.833 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:25:34.833 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:25:34.833 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:25:34.834 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:25:34.834 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:25:34.834 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:25:34.834 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:25:34.834 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:25:34.835 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:25:34.835 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: raise 2021-07-31T03:25:34.835 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:25:34.835 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:25:34.835 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:25:34.836 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:25:34.836 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:25:34.836 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: cephadm 2021-07-31T03:25:33.156381+0000 mgr.x (mgr.24542) 4217 : cephadm [ERR] 2021-07-31T03:25:34.836 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:25:34.836 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:25:34.837 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: yield 2021-07-31T03:25:34.837 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:25:34.837 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:25:34.837 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:25:34.837 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:25:34.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:25:34.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:25:34.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:25:34.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:25:34.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:25:34.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: task.result() 2021-07-31T03:25:34.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:25:34.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: cephadm 2021-07-31T03:25:33.156788+0000 mgr.x (mgr.24542) 4218 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:25:34.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:25:34.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:25:34.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: yield 2021-07-31T03:25:34.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:25:34.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:25:34.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:25:34.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:25:34.841 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:25:34.841 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:25:34.841 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:25:34.841 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:25:34.841 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:25:34.842 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: task.result() 2021-07-31T03:25:34.842 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:25:34.842 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: 2021-07-31T03:25:34.842 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:25:34.842 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: 2021-07-31T03:25:34.843 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:25:34.843 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:25:34.843 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:25:34.843 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:25:34.843 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:25:34.844 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:25:34.844 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:25:34.844 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:25:34.844 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:25:34.844 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:25:34.845 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:25:34.845 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:25:34.845 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:25:34.845 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:25:34.845 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:25:34.846 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:25:34.846 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:25:34.846 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:25:34.846 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: raise 2021-07-31T03:25:34.846 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:25:34.847 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:25:34.847 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:25:34.847 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:25:34.847 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:25:34.847 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:34 smithi045 conmon[33655]: cluster 2021-07-31T03:25:33.202107+0000 mgr.x (mgr.24542) 4219 : cluster [DBG] pgmap v4957: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:36.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:36 smithi154 conmon[34553]: cluster 2021-07-31T03:25:35.202696+0000 mgr.x (mgr.24542) 4220 : cluster [DBG] pgmap v4958: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:36.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:36 smithi045 conmon[39718]: cluster 2021-07-31T03:25:35.202696+0000 mgr.x (mgr.24542) 4220 : cluster [DBG] pgmap v4958: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:36.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:36 smithi045 conmon[33655]: cluster 2021-07-31T03: 2021-07-31T03:25:36.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:36 smithi045 conmon[33655]: 25:35.202696+0000 mgr.x (mgr.24542) 4220 : cluster [DBG] pgmap v4958: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:38.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:38 smithi154 conmon[34553]: cluster 2021-07-31T03:25:37.203252+0000 mgr.x (mgr.24542) 4221 : cluster [DBG] pgmap v4959: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:38.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:38 smithi045 conmon[39718]: cluster 2021-07-31T03:25:37.203252+0000 mgr.x 2021-07-31T03:25:38.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:38 smithi045 conmon[39718]: (mgr.24542) 4221 : cluster [DBG] pgmap v4959: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:38.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:38 smithi045 conmon[33655]: cluster 2021-07-31T03:25:37.203252 2021-07-31T03:25:38.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:38 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 4221 : cluster [DBG] pgmap v4959: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:40.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:40 smithi154 conmon[34553]: cluster 2021-07-31T03:25:39.203674+0000 mgr.x (mgr.24542) 4222 : cluster [DBG] pgmap v4960: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:40.774 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:40 smithi045 conmon[39718]: cluster 2021-07-31T03:25:39.203674+0000 mgr.x (mgr 2021-07-31T03:25:40.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:40 smithi045 conmon[39718]: .24542) 4222 : cluster [DBG] pgmap v4960: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:40.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:40 smithi045 conmon[33655]: cluster 2021-07-31T03:25:39.203674+0000 mgr.x (mgr.24542 2021-07-31T03:25:40.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:40 smithi045 conmon[33655]: ) 4222 : cluster [DBG] pgmap v4960: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:42.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:42 smithi154 conmon[34553]: cluster 2021-07-31T03:25:41.204090+0000 mgr.x (mgr.24542) 4223 : cluster [DBG] pgmap v4961: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:42.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:42 smithi045 conmon[39718]: cluster 2021-07-31T03:25:41.204090+0000 mgr.x ( 2021-07-31T03:25:42.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:42 smithi045 conmon[39718]: mgr.24542) 4223 : cluster [DBG] pgmap v4961: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:42.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:42 smithi045 conmon[33655]: cluster 2021-07-31T03:25:41.204090+0000 mgr.x (mgr.24542) 4223 : cluster [DBG] 2021-07-31T03:25:42.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:42 smithi045 conmon[33655]: pgmap v4961: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:44.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:44 smithi154 conmon[34553]: cluster 2021-07-31T03:25:43.204473+0000 mgr.x (mgr.24542) 4224 : cluster [DBG] pgmap v4962: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:44.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:44 smithi045 conmon[33655]: cluster 2021-07-31T03:25:43.204473+0000 mgr.x (mgr.24542) 4224 : cluster [DBG] 2021-07-31T03:25:44.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:44 smithi045 conmon[33655]: pgmap v4962: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:44.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:44 smithi045 conmon[39718]: cluster 2021-07-31T03:25:43.204473+0000 mgr.x (mgr. 2021-07-31T03:25:44.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:44 smithi045 conmon[39718]: 24542) 4224 : cluster [DBG] pgmap v4962: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:46.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:46 smithi154 conmon[34553]: cluster 2021-07-31T03:25:45.205069+0000 mgr.x (mgr.24542) 4225 : cluster [DBG] pgmap v4963: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:46.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:46 smithi154 conmon[34553]: 2021-07-31T03:25:46.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:46 smithi045 conmon[33655]: cluster 2021-07-31T03:25:45.205069+0000 mgr.x (mgr.24542) 4225 : cluster 2021-07-31T03:25:46.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:46 smithi045 conmon[33655]: [DBG] pgmap v4963: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:46.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:46 smithi045 conmon[39718]: cluster 2021-07-31T03:25:45.205069+0000 mgr.x (mgr 2021-07-31T03:25:46.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:46 smithi045 conmon[39718]: .24542) 4225 : cluster [DBG] pgmap v4963: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:48.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:48 smithi154 conmon[34553]: cluster 2021-07-31T03:25:47.205469+0000 mgr.x (mgr.24542) 4226 : cluster [DBG] pgmap v4964: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:48.774 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:48 smithi045 conmon[39718]: cluster 2021-07-31T03:25:47.205469+0000 mgr.x (mgr. 2021-07-31T03:25:48.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:48 smithi045 conmon[39718]: 24542) 4226 : cluster [DBG] pgmap v4964: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:48.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:48 smithi045 conmon[33655]: cluster 2021-07-31T03:25:47.205469+0000 mgr.x (mgr.24542) 4226 2021-07-31T03:25:48.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:48 smithi045 conmon[33655]: : cluster [DBG] pgmap v4964: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:50.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:50 smithi154 conmon[34553]: cluster 2021-07-31T03:25:49.206066+0000 mgr.x (mgr.24542) 4227 : cluster [DBG] pgmap v4965: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:50.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:50 smithi045 conmon[33655]: cluster 2021-07-31T03:25:49.206066+0000 2021-07-31T03:25:50.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:50 smithi045 conmon[33655]: mgr.x (mgr.24542) 4227 : cluster [DBG] pgmap v4965: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:50.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:50 smithi045 conmon[39718]: cluster 2021-07-31T03:25:49.206066+0000 mgr.x (mgr. 2021-07-31T03:25:50.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:50 smithi045 conmon[39718]: 24542) 4227 : cluster [DBG] pgmap v4965: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:52.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:52 smithi154 conmon[34553]: cluster 2021-07-31T03:25:51.206587+0000 mgr.x (mgr.24542) 4228 : cluster [DBG] pgmap v4966: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:52.774 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:52 smithi045 conmon[39718]: cluster 2021-07-31T03:25:51.206587+0000 mgr.x (mgr.24542) 4228 : cluster [DBG] pgmap v4966: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:52.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:52 smithi045 conmon[33655]: cluster 2021-07-31T03:25:51.206587+0000 mgr.x ( 2021-07-31T03:25:52.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:52 smithi045 conmon[33655]: mgr.24542) 4228 : cluster [DBG] pgmap v4966: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:54.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:54 smithi154 conmon[34553]: cluster 2021-07-31T03:25:53.207172+0000 mgr.x (mgr.24542) 4229 : cluster [DBG] 2021-07-31T03:25:54.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:54 smithi154 conmon[34553]: pgmap v4967: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:54.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:54 smithi045 conmon[33655]: cluster 2021-07-31T03:25:53.207172+0000 mgr.x (mgr.24542) 4229 : 2021-07-31T03:25:54.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:54 smithi045 conmon[33655]: cluster [DBG] pgmap v4967: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:54.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:54 smithi045 conmon[39718]: cluster 2021-07-31T03:25:53.207172+0000 mgr.x 2021-07-31T03:25:54.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:54 smithi045 conmon[39718]: (mgr.24542) 4229 : cluster [DBG] pgmap v4967: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:54.851 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:25:55.127 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:25:55.127 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 100m ago 101m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:25:55.128 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 98m ago 101m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:25:55.128 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 98m ago 104m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:25:55.128 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 100m ago 106m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:25:55.128 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 100m ago 107m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:25:55.128 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 98m ago 105m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:25:55.129 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 100m ago 105m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:25:55.129 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 100m ago 101m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:25:55.129 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 98m ago 101m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:25:55.129 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 100m ago 104m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:25:55.130 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 100m ago 104m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:25:55.130 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 100m ago 103m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:25:55.130 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 100m ago 103m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:25:55.130 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 98m ago 103m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:25:55.130 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 98m ago 102m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:25:55.131 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 98m ago 102m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:25:55.131 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 98m ago 102m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:25:55.131 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 98m ago 101m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:25:55.463 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:25:55.464 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:25:55.464 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:25:55.464 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:25:55.464 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:25:55.465 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:25:55.465 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:25:55.465 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:25:55.465 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:25:55.465 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:25:55.466 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:25:55.466 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:25:55.466 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:25:55.466 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:25:55.467 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:25:55.467 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:25:55.467 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:25:55.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:55 smithi154 conmon[34553]: audit 2021-07-31T03:25:54.840442+0000 mgr.x (mgr.24542) 4230 : audit 2021-07-31T03:25:55.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:55 smithi154 conmon[34553]: [DBG] from='client.29428 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:25:55.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:55 smithi154 conmon[34553]: audit 2021-07-31T03:25:55.464501+0000 mon.c (mon.1) 204 : audit [DBG] from='client.? 172.21.15.45:0/2181646153' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:25:55.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:55 smithi045 conmon[33655]: audit 2021-07-31T03:25:54.840442+0000 mgr.x ( 2021-07-31T03:25:55.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:55 smithi045 conmon[33655]: mgr.24542) 4230 : audit [DBG] from='client.29428 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:25:55.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:55 smithi045 conmon[33655]: audit 2021-07-31T03:25:55.464501+0000 mon.c (mon.1) 204 : audit [DBG] from='client.? 172.21.15.45:0/2181646153' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:25:55.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:55 smithi045 conmon[39718]: audit 2021-07-31T03:25 2021-07-31T03:25:55.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:55 smithi045 conmon[39718]: :54.840442+0000 mgr.x (mgr.24542) 4230 : audit [DBG] from='client.29428 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:25:55.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:55 smithi045 conmon[39718]: audit 2021-07 2021-07-31T03:25:55.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:55 smithi045 conmon[39718]: -31T03:25:55 2021-07-31T03:25:55.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:55 smithi045 conmon[39718]: .464501+0000 mon.c (mon.1) 204 2021-07-31T03:25:55.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:55 smithi045 conmon[39718]: : audit [DBG] from='client.? 172.21.15.45:0/2181646153' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:25:56.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:56 smithi154 conmon[34553]: audit 2021-07-31T03:25:55.120249+0000 mgr.x (mgr.24542) 4231 2021-07-31T03:25:56.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:56 smithi154 conmon[34553]: : audit [DBG] from='client.29434 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:25:56.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:56 smithi154 conmon[34553]: cluster 2021-07-31T03:25:55.207735+0000 mgr.x (mgr.24542) 4232 : cluster [DBG] pgmap v4968: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:56.774 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:56 smithi045 conmon[39718]: audit 2021-07-31T03:25:55.120249+0000 mgr.x (mgr.24542) 4231 : audit [DBG] from='client.29434 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:25:56.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:56 smithi045 conmon[39718]: cluster 2021-07-31T03:25:55.207735+0000 2021-07-31T03:25:56.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:56 smithi045 conmon[39718]: mgr.x (mgr.24542) 4232 : cluster 2021-07-31T03:25:56.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:56 smithi045 conmon[39718]: [DBG] pgmap v4968: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:56.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:56 smithi045 conmon[33655]: audit 2021-07-31T03:25:55.120249+0000 mgr.x (mgr 2021-07-31T03:25:56.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:56 smithi045 conmon[33655]: .24542) 4231 : audit [DBG] from='client.29434 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:25:56.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:56 smithi045 conmon[33655]: cluster 2021-07-31T03:25:55.207735+0000 mgr.x (mgr.24542) 4232 2021-07-31T03:25:56.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:56 smithi045 conmon[33655]: : cluster [DBG] pgmap v4968: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:58.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:58 smithi154 conmon[34553]: cluster 2021-07-31T03:25:57.208254+0000 mgr.x (mgr.24542) 4233 : cluster [DBG] 2021-07-31T03:25:58.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:25:58 smithi154 conmon[34553]: pgmap v4969: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:58.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:58 smithi045 conmon[33655]: cluster 2021-07-31T03:25:57. 2021-07-31T03:25:58.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:25:58 smithi045 conmon[33655]: 208254+0000 mgr.x (mgr.24542) 4233 : cluster [DBG] pgmap v4969: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:25:58.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:58 smithi045 conmon[39718]: cluster 2021-07-31T03:25:57. 2021-07-31T03:25:58.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:25:58 smithi045 conmon[39718]: 208254+0000 mgr.x (mgr.24542) 4233 : cluster [DBG] pgmap v4969: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:00.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:00 smithi154 conmon[34553]: cluster 2021-07-31T03:25:59.208671+0000 mgr.x (mgr.24542) 4234 : cluster [DBG] 2021-07-31T03:26:00.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:00 smithi154 conmon[34553]: pgmap v4970: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:00.774 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:00 smithi045 conmon[33655]: cluster 2021-07-31T03:25:59.208671+0000 mgr.x (mgr.24542) 4234 : cluster [DBG] pgmap v4970: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:00.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:00 smithi045 conmon[39718]: cluster 2021-07-31T03:25:59.208671+0000 mgr.x ( 2021-07-31T03:26:00.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:00 smithi045 conmon[39718]: mgr.24542) 4234 : cluster [DBG] pgmap v4970: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:02.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:02 smithi045 conmon[33655]: cluster 2021-07-31T03:26:01.209073+0000 mgr.x (mgr.24542) 2021-07-31T03:26:02.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:02 smithi045 conmon[33655]: 4235 : cluster [DBG] pgmap v4971: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:02.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:02 smithi045 conmon[39718]: cluster 2021-07-31T03:26:01 2021-07-31T03:26:02.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:02 smithi045 conmon[39718]: .209073+0000 mgr.x (mgr.24542) 4235 : cluster [DBG] pgmap v4971: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:03.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:02 smithi154 conmon[34553]: cluster 2021-07-31T03:26:01.209073+0000 mgr.x (mgr.24542) 4235 : cluster [DBG] pgmap v4971: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:04.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:04 smithi045 conmon[39718]: cluster 2021-07-31T03:26:03.209548+0000 mgr.x (mgr.24542 2021-07-31T03:26:04.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:04 smithi045 conmon[39718]: ) 4236 : cluster [DBG] pgmap v4972: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:04.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:04 smithi045 conmon[33655]: cluster 2021-07-31T03:26:03.209548+0000 mgr.x (mgr.24542) 4236 : cluster [DBG] pgmap v4972: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:05.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:04 smithi154 conmon[34553]: cluster 2021-07-31T03:26:03.209548+0000 mgr.x (mgr.24542) 4236 : cluster [DBG] pgmap v4972: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:06.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:06 smithi045 conmon[39718]: cluster 2021-07-31T03:26:05.210135+0000 2021-07-31T03:26:06.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:06 smithi045 conmon[39718]: mgr.x (mgr.24542) 4237 : cluster [DBG] pgmap v4973: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:06.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:06 smithi045 conmon[33655]: cluster 2021-07-31T03:26:05.210135+0000 mgr.x (mgr.24542) 4237 : cluster [DBG] 2021-07-31T03:26:06.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:06 smithi045 conmon[33655]: pgmap v4973: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:07.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:06 smithi154 conmon[34553]: cluster 2021-07-31T03:26:05.210135+0000 mgr.x (mgr.24542) 4237 : cluster [DBG] pgmap v4973: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:08.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:08 smithi045 conmon[39718]: cluster 2021-07-31T03:26:07.210551+0000 mgr.x 2021-07-31T03:26:08.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:08 smithi045 conmon[39718]: (mgr.24542) 4238 : cluster [DBG] pgmap v4974: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:08.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:08 smithi045 conmon[33655]: cluster 2021-07-31T03:26:07.210551+0000 mgr.x (mgr.24542) 4238 : cluster [DBG] pgmap v4974: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:08.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:08 smithi045 conmon[33655]: 2021-07-31T03:26:09.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:08 smithi154 conmon[34553]: cluster 2021-07-31T03:26:07.210551+0000 mgr.x (mgr.24542) 4238 : cluster [DBG] 2021-07-31T03:26:09.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:08 smithi154 conmon[34553]: pgmap v4974: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:10.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:10 smithi045 conmon[39718]: cluster 2021-07-31T03:26:09.211129+0000 mgr.x (mgr 2021-07-31T03:26:10.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:10 smithi045 conmon[39718]: .24542) 4239 : cluster [DBG] pgmap v4975: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:10.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:10 smithi045 conmon[33655]: cluster 2021-07-31T03:26:09.211129+0000 mgr.x ( 2021-07-31T03:26:10.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:10 smithi045 conmon[33655]: mgr.24542) 4239 : cluster [DBG] pgmap v4975: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:11.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:10 smithi154 conmon[34553]: cluster 2021-07-31T03:26:09.211129+0000 mgr.x (mgr.24542) 4239 : cluster [DBG] 2021-07-31T03:26:11.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:10 smithi154 conmon[34553]: pgmap v4975: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:12.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:12 smithi045 conmon[39718]: cluster 2021-07-31T03:26:11.211591+0000 mgr.x ( 2021-07-31T03:26:12.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:12 smithi045 conmon[39718]: mgr.24542) 4240 : cluster [DBG] pgmap v4976: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:12.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:12 smithi045 conmon[33655]: cluster 2021-07-31T03:26:11.211591+0000 mgr.x (mgr.24542) 4240 2021-07-31T03:26:12.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:12 smithi045 conmon[33655]: : cluster [DBG] pgmap v4976: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:13.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:12 smithi154 conmon[34553]: cluster 2021-07-31T03:26:11.211591+0000 mgr.x (mgr.24542) 4240 : cluster [DBG] 2021-07-31T03:26:13.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:12 smithi154 conmon[34553]: pgmap v4976: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:14.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:14 smithi045 conmon[39718]: cluster 2021-07-31T03:26:13.212153+0000 mgr.x (mgr 2021-07-31T03:26:14.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:14 smithi045 conmon[39718]: .24542) 4241 : cluster [DBG] pgmap v4977: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:14.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:14 smithi045 conmon[33655]: cluster 2021-07-31T03:26:13.212153+0000 mgr.x (mgr.24542) 2021-07-31T03:26:14.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:14 smithi045 conmon[33655]: 4241 : cluster [DBG] pgmap v4977: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:15.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:14 smithi154 conmon[34553]: cluster 2021-07-31T03:26:13.212153+0000 mgr.x (mgr.24542) 4241 : cluster [DBG] 2021-07-31T03:26:15.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:14 smithi154 conmon[34553]: pgmap v4977: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:17.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:16 smithi154 conmon[34553]: cluster 2021-07-31T03:26:15.212643+0000 mgr.x (mgr.24542) 4242 : cluster [DBG] 2021-07-31T03:26:17.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:16 smithi154 conmon[34553]: pgmap v4978: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:17.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:16 smithi045 conmon[39718]: cluster 2021-07-31T03:26:15.212643+0000 2021-07-31T03:26:17.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:16 smithi045 conmon[39718]: mgr.x (mgr.24542) 4242 : cluster [DBG] pgmap v4978: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:17.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:16 smithi045 conmon[33655]: cluster 2021-07-31T03:26:15.212643+0000 mgr.x (mgr.24542) 4242 2021-07-31T03:26:17.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:16 smithi045 conmon[33655]: : cluster [DBG] pgmap v4978: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:19.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:18 smithi154 conmon[34553]: cluster 2021-07-31T03:26:17.213074+0000 mgr.x (mgr.24542) 4243 : 2021-07-31T03:26:19.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:18 smithi154 conmon[34553]: cluster [DBG] pgmap v4979: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:19.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:18 smithi045 conmon[39718]: cluster 2021-07-31T03:26:17.213074+0000 mgr.x (mgr 2021-07-31T03:26:19.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:18 smithi045 conmon[39718]: .24542) 4243 : cluster [DBG] pgmap v4979: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:19.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:18 smithi045 conmon[33655]: cluster 2021-07-31T03:26:17.213074+0000 mgr.x (mgr. 2021-07-31T03:26:19.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:18 smithi045 conmon[33655]: 24542) 4243 : cluster [DBG] pgmap v4979: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:21.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:20 smithi154 conmon[34553]: cluster 2021-07-31T03:26:19.213435+0000 mgr.x (mgr.24542) 4244 : cluster [DBG] pgmap v4980: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:21.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:20 smithi154 conmon[34553]: 2021-07-31T03:26:21.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:20 smithi045 conmon[39718]: cluster 2021-07-31T03:26:19.213435+0000 mgr.x ( 2021-07-31T03:26:21.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:20 smithi045 conmon[39718]: mgr.24542) 4244 : cluster [DBG] pgmap v4980: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:21.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:20 smithi045 conmon[33655]: cluster 2021-07-31T03:26:19.213435+0000 mgr.x (mgr.24542) 4244 : 2021-07-31T03:26:21.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:20 smithi045 conmon[33655]: cluster [DBG] pgmap v4980: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:23.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:22 smithi154 conmon[34553]: cluster 2021-07-31T03:26:21.213768+0000 mgr.x (mgr.24542) 4245 : 2021-07-31T03:26:23.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:22 smithi154 conmon[34553]: cluster [DBG] pgmap v4981: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:23.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:22 smithi045 conmon[39718]: cluster 2021-07-31T03:26:21.213768+0000 mgr.x (mgr. 2021-07-31T03:26:23.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:22 smithi045 conmon[39718]: 24542) 4245 : cluster [DBG] pgmap v4981: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:23.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:22 smithi045 conmon[33655]: cluster 2021-07-31T03:26:21.213768+0000 mgr.x (mgr.24542) 4245 : 2021-07-31T03:26:23.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:22 smithi045 conmon[33655]: cluster [DBG] pgmap v4981: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:25.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:24 smithi154 conmon[34553]: cluster 2021-07-31T03:26:23.214243+0000 mgr.x (mgr.24542) 4246 : cluster 2021-07-31T03:26:25.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:24 smithi154 conmon[34553]: [DBG] pgmap v4982: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:25.024 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:24 smithi045 conmon[39718]: cluster 2021-07-31T03:26:23.214243+0000 mgr.x ( 2021-07-31T03:26:25.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:24 smithi045 conmon[39718]: mgr.24542) 4246 : cluster [DBG] pgmap v4982: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:25.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:24 smithi045 conmon[33655]: cluster 2021-07-31T03:26:23.214243+0000 mgr.x (mgr.24542) 4246 2021-07-31T03:26:25.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:24 smithi045 conmon[33655]: : cluster [DBG] pgmap v4982: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:25.780 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:26:26.056 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:26:26.056 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 101m ago 102m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:26:26.056 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 98m ago 101m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:26:26.057 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 98m ago 105m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:26:26.057 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 101m ago 107m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:26:26.057 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 101m ago 107m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:26:26.057 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 98m ago 105m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:26:26.057 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 101m ago 105m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:26:26.058 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 101m ago 102m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:26:26.058 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 98m ago 102m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:26:26.058 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 101m ago 105m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:26:26.058 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 101m ago 104m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:26:26.058 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 101m ago 104m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:26:26.059 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 101m ago 104m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:26:26.059 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 98m ago 103m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:26:26.059 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 98m ago 103m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:26:26.059 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 98m ago 103m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:26:26.060 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 98m ago 102m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:26:26.060 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 98m ago 102m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:26:26.395 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:26:26.395 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:26:26.396 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:26:26.396 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:26:26.396 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:26:26.396 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:26:26.396 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:26:26.397 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:26:26.397 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:26:26.397 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:26:26.397 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:26:26.397 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:26:26.398 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:26:26.398 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:26:26.398 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:26:26.398 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:26:26.398 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:26:27.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:26 smithi154 conmon[34553]: cluster 2021-07-31T03:26:25.214751+0000 mgr.x (mgr.24542) 4247 : cluster 2021-07-31T03:26:27.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:26 smithi154 conmon[34553]: [DBG] pgmap v4983: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:27.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:26 smithi154 conmon[34553]: audit 2021-07-31T03:26:25.769736+0000 mgr.x (mgr.24542) 4248 : audit [DBG] from='client.19506 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:26:27.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:26 smithi154 conmon[34553]: audit 2021-07-31T03:26:26.396219+0000 mon.a (mon.0) 3303 : audit [DBG] 2021-07-31T03:26:27.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:26 smithi154 conmon[34553]: from='client.? 172.21.15.45:0/1070265427' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:26:27.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:26 smithi045 conmon[39718]: cluster 2021-07-31T03 2021-07-31T03:26:27.075 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:26 smithi045 conmon[39718]: :26:25.214751+0000 mgr.x (mgr.24542) 4247 : cluster [DBG] 2021-07-31T03:26:27.076 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:26 smithi045 conmon[39718]: pgmap v4983: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:27.076 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:26 smithi045 conmon[39718]: audit 2021 2021-07-31T03:26:27.076 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:26 smithi045 conmon[39718]: -07-31T03:26:25 2021-07-31T03:26:27.076 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:26 smithi045 conmon[39718]: .769736+0000 mgr.x (mgr. 2021-07-31T03:26:27.077 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:26 smithi045 conmon[39718]: 24542) 4248 : audit [DBG] from='client.19506 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:26:27.077 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:26 smithi045 conmon[39718]: 2021-07-31T03:26:27.077 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:26 smithi045 conmon[39718]: audit 2021 2021-07-31T03:26:27.077 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:26 smithi045 conmon[39718]: -07-31T03:26 2021-07-31T03:26:27.078 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:26 smithi045 conmon[39718]: :26.396219+0000 mon.a 2021-07-31T03:26:27.078 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:26 smithi045 conmon[39718]: (mon.0) 3303 : audit 2021-07-31T03:26:27.078 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:26 smithi045 conmon[39718]: [DBG] from='client.? 172.21.15.45:0/1070265427' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:26:27.079 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:26 smithi045 conmon[33655]: cluster 2021-07-31T03:26:25.214751+0000 mgr.x (mgr. 2021-07-31T03:26:27.079 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:26 smithi045 conmon[33655]: 24542) 4247 : cluster [DBG] pgmap v4983: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:27.079 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:26 smithi045 conmon[33655]: audit 2021-07-31T03:26:25.769736+0000 2021-07-31T03:26:27.079 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:26 smithi045 conmon[33655]: mgr.x (mgr.24542) 4248 : audit [DBG] from='client.19506 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:26:27.080 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:26 smithi045 conmon[33655]: audit 2021 2021-07-31T03:26:27.080 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:26 smithi045 conmon[33655]: -07-31T03:26 2021-07-31T03:26:27.080 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:26 smithi045 conmon[33655]: :26.396219+0000 2021-07-31T03:26:27.080 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:26 smithi045 conmon[33655]: mon.a (mon.0) 3303 2021-07-31T03:26:27.080 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:26 smithi045 conmon[33655]: : audit [DBG] from='client.? 172.21.15.45:0/1070265427' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:26:28.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:27 smithi154 conmon[34553]: audit 2021-07-31T03:26:26.048969+0000 mgr.x (mgr.24542) 4249 : audit 2021-07-31T03:26:28.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:27 smithi154 conmon[34553]: [DBG] from='client.19512 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:26:28.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:27 smithi045 conmon[39718]: audit 2021-07-31T03:26:26.048969+0000 mgr.x 2021-07-31T03:26:28.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:27 smithi045 conmon[39718]: (mgr.24542) 4249 : audit [DBG] from='client.19512 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:26:28.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:27 smithi045 conmon[33655]: audit 2021-07-31T03:26:26.048969+0000 mgr.x (mgr.24542) 2021-07-31T03:26:28.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:27 smithi045 conmon[33655]: 4249 : audit [DBG] from='client.19512 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:26:29.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:28 smithi154 conmon[34553]: cluster 2021-07-31T03:26:27.215199+0000 mgr.x (mgr.24542) 4250 : cluster [DBG] pgmap v4984: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:29.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:28 smithi154 conmon[34553]: audit 2021-07-31T03:26:28.329481+0000 mon.b (mon.2) 570 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:26:29.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:28 smithi154 conmon[34553]: audit 2021-07-31T03:26:28.329719+0000 mon.b (mon.2) 571 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:26:29.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:28 smithi154 conmon[34553]: audit 2021-07-31T03:26 2021-07-31T03:26:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:28 smithi154 conmon[34553]: :28.330220+0000 mon.a (mon.0 2021-07-31T03:26:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:28 smithi154 conmon[34553]: ) 3304 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:26:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:28 smithi154 conmon[34553]: audit 2021 2021-07-31T03:26:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:28 smithi154 conmon[34553]: -07-31T03:26:28. 2021-07-31T03:26:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:28 smithi154 conmon[34553]: 330484+0000 mon.a (mon.0) 2021-07-31T03:26:29.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:28 smithi154 conmon[34553]: 3305 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:26:29.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:28 smithi045 conmon[39718]: cluster 2021-07-31T03:26:27.215199+0000 mgr.x ( 2021-07-31T03:26:29.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:28 smithi045 conmon[39718]: mgr.24542) 4250 : cluster [DBG] pgmap v4984: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:29.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:28 smithi045 conmon[39718]: audit 2021-07-31T03 2021-07-31T03:26:29.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:28 smithi045 conmon[39718]: :26:28.329481+0000 mon.b (mon.2) 570 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:26:29.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:28 smithi045 conmon[39718]: audit 2021-07-31T03:26:28.329719+0000 mon.b 2021-07-31T03:26:29.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:28 smithi045 conmon[39718]: (mon.2) 571 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:26:29.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:28 smithi045 conmon[39718]: audit 2021-07-31T03:26:28 2021-07-31T03:26:29.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:28 smithi045 conmon[39718]: .330220+0000 mon.a (mon.0) 3304 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:26:29.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:28 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T03:26:29.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:28 smithi045 conmon[39718]: 03:26:28.330484+0000 mon.a (mon.0) 3305 : audit [INF] 2021-07-31T03:26:29.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:28 smithi045 conmon[39718]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:26:29.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:28 smithi045 conmon[33655]: cluster 2021-07-31T03:26:27. 2021-07-31T03:26:29.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:28 smithi045 conmon[33655]: 215199+0000 mgr.x (mgr.24542) 4250 : cluster [DBG] pgmap v4984: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:29.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:28 smithi045 conmon[33655]: audit 2021- 2021-07-31T03:26:29.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:28 smithi045 conmon[33655]: 07-31T03:26:28.329481+0000 mon.b (mon.2) 570 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:26:29.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:28 smithi045 conmon[33655]: 2021-07-31T03:26:29.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:28 smithi045 conmon[33655]: audit 2021-07-31 2021-07-31T03:26:29.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:28 smithi045 conmon[33655]: T03:26:28.329719+0000 mon.b (mon.2) 571 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:26:29.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:28 smithi045 conmon[33655]: 2021-07-31T03:26:29.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:28 smithi045 conmon[33655]: audit 2021-07-31T03:26: 2021-07-31T03:26:29.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:28 smithi045 conmon[33655]: 28.330220+0000 mon.a (mon.0) 3304 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:26:29.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:28 smithi045 conmon[33655]: audit 2021-07-31T 2021-07-31T03:26:29.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:28 smithi045 conmon[33655]: 03:26:28.330484+0000 mon.a ( 2021-07-31T03:26:29.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:28 smithi045 conmon[33655]: mon.0) 3305 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:26:29.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:28 smithi045 conmon[33655]: 2021-07-31T03:26:31.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:30 smithi154 conmon[34553]: cluster 2021-07-31T03:26:29.215677+0000 mgr.x (mgr.24542) 4251 : cluster [DBG] pgmap v4985: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:31.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:30 smithi045 conmon[33655]: cluster 2021-07-31T03:26:29.215677+0000 mgr.x (mgr.24542) 4251 : cluster [DBG] pgmap v4985: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:31.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:30 smithi045 conmon[39718]: cluster 2021-07-31T03:26:29.215677+0000 mgr.x ( 2021-07-31T03:26:31.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:30 smithi045 conmon[39718]: mgr.24542) 4251 : cluster [DBG] pgmap v4985: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:33.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:32 smithi154 conmon[34553]: cluster 2021-07-31T03:26:31.216111+0000 mgr.x (mgr.24542) 4252 : cluster [DBG] pgmap v4986: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:33.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:32 smithi045 conmon[33655]: cluster 2021-07-31T03:26:31.216111+0000 mgr.x (mgr.24542 2021-07-31T03:26:33.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:32 smithi045 conmon[33655]: ) 4252 : cluster [DBG] pgmap v4986: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:33.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:32 smithi045 conmon[39718]: cluster 2021-07-31T03:26:31.216111+0000 mgr.x (mgr.24542 2021-07-31T03:26:33.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:32 smithi045 conmon[39718]: ) 4252 : cluster [DBG] pgmap v4986: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:33.502 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: debug 2021-07-31T03:26:33.202+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:26:33.502 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:26:33.503 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:26:33.503 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: yield 2021-07-31T03:26:33.503 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:26:33.503 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:26:33.503 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:26:33.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:26:33.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:26:33.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:26:33.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:26:33.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:26:33.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:26:33.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: task.result() 2021-07-31T03:26:33.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:26:33.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: debug 2021-07-31T03:26:33.213+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:26:33.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:26:33.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:26:33.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: yield 2021-07-31T03:26:33.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:26:33.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:26:33.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:26:33.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:26:33.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:26:33.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:26:33.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:26:33.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:26:33.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:26:33.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: task.result() 2021-07-31T03:26:33.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:26:33.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: debug 2021-07-31T03:26:33.239+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:26:33.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:26:33.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:26:33.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: yield 2021-07-31T03:26:33.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:26:33.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:26:33.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:26:33.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:26:33.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:26:33.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:26:33.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:26:33.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:26:33.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:26:33.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: task.result() 2021-07-31T03:26:33.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:26:33.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: debug 2021-07-31T03:26:33.240+0000 7fcb97969700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:26:33.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:26:33.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:26:33.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: yield 2021-07-31T03:26:33.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:26:33.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:26:33.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:26:33.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:26:33.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:26:33.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:26:33.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:26:33.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:26:33.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:26:33.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: task.result() 2021-07-31T03:26:33.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:26:33.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: 2021-07-31T03:26:33.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:26:33.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: 2021-07-31T03:26:33.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:26:33.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:26:33.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:26:33.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:26:33.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:26:33.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:26:33.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:26:33.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:26:33.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:26:33.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:26:33.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:26:33.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:26:33.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:26:33.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:26:33.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:26:33.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:26:33.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:26:33.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:26:33.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: raise 2021-07-31T03:26:33.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:26:33.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:26:33.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:26:33.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:26:33.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:26:33.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: debug 2021-07-31T03:26:33.248+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:26:33.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:26:33.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:26:33.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: yield 2021-07-31T03:26:33.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:26:33.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:26:33.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:26:33.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:26:33.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:26:33.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:26:33.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:26:33.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:26:33.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:26:33.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: task.result() 2021-07-31T03:26:33.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:26:33.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: debug 2021-07-31T03:26:33.248+0000 7fcb9816a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:26:33.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:26:33.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:26:33.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: yield 2021-07-31T03:26:33.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:26:33.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:26:33.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:26:33.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:26:33.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:26:33.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:26:33.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:26:33.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:26:33.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:26:33.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: task.result() 2021-07-31T03:26:33.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:26:33.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: 2021-07-31T03:26:33.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:26:33.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: 2021-07-31T03:26:33.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:26:33.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:26:33.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:26:33.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:26:33.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:26:33.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:26:33.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:26:33.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:26:33.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:26:33.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:26:33.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:26:33.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:26:33.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:26:33.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:26:33.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:26:33.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:26:33.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:26:33.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:26:33.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: raise 2021-07-31T03:26:33.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:26:33.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:26:33.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:26:33.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:26:33.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:26:34.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[34553]: audit 2021-07-31T03:26:33.157552+0000 mon.b (mon.2) 572 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:26:34.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[34553]: audit 2021-07-31T03:26:33.158954+0000 mon.b (mon.2) 573 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:26:34.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:33 smithi154 conmon[34553]: audit 2021-07-31T03:26:33.159991+0000 mon.b (mon.2) 574 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:26:34.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:33 smithi045 conmon[39718]: audit 2021-07-31T03:26:33.157552+0000 mon.b (mon. 2021-07-31T03:26:34.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:33 smithi045 conmon[39718]: 2) 572 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:26:34.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:33 smithi045 conmon[39718]: audit 2021-07-31T03:26:33.158954+0000 2021-07-31T03:26:34.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:33 smithi045 conmon[39718]: mon.b (mon.2) 573 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:26:34.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:33 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T03:26:34.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:33 smithi045 conmon[39718]: T03:26:33.159991+0000 mon.b (mon.2 2021-07-31T03:26:34.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:33 smithi045 conmon[39718]: ) 574 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:26:34.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:33 smithi045 conmon[33655]: audit 2021-07-31T03:26:33.157552+0000 mon.b (mon.2) 572 2021-07-31T03:26:34.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:33 smithi045 conmon[33655]: : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:26:34.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:33 smithi045 conmon[33655]: audit 2021-07-31T03:26:33.158954+0000 mon.b (mon.2) 573 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:26:34.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:33 smithi045 conmon[33655]: audit 2021-07-31T 2021-07-31T03:26:34.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:33 smithi045 conmon[33655]: 03:26:33.159991+0000 mon.b (mon.2) 574 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:26:34.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: cephadm 2021-07-31T03:26:33.160778+0000 mgr.x (mgr.24542) 4253 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:26:34.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: cephadm 2021-07-31T03:26:33.203582+0000 mgr.x (mgr.24542) 4254 : cephadm [ERR] 2021-07-31T03:26:34.570 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: cephadm 2021-07-31T03:26:33.160778+0000 mgr.x (mgr.24542) 4253 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:26:34.571 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: cephadm 2021-07-31T 2021-07-31T03:26:34.571 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: 03:26:33.203582+0000 mgr.x (mgr.24542) 4254 : cephadm [ERR] 2021-07-31T03:26:34.571 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:26:34.571 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:26:34.571 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: yield 2021-07-31T03:26:34.572 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:26:34.572 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:26:34.572 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:26:34.572 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:26:34.572 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:26:34.573 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:26:34.573 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:26:34.573 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:26:34.573 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:26:34.573 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: task.result() 2021-07-31T03:26:34.574 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:26:34.574 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: cephadm 2021-07-31T03:26:33.204054+0000 mgr.x (mgr.24542) 4255 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:26:34.574 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: cephadm 2021-07-31T03:26 2021-07-31T03:26:34.574 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: :33.214887+0000 mgr.x (mgr.24542) 4256 : cephadm [ERR] 2021-07-31T03:26:34.574 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:26:34.575 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:26:34.575 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: yield 2021-07-31T03:26:34.575 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:26:34.575 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:26:34.575 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:26:34.576 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:26:34.576 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:26:34.576 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:26:34.576 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:26:34.576 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:26:35.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:26:35.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: task.result() 2021-07-31T03:26:35.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:26:35.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: cephadm 2021-07-31T03:26:33.215337+0000 mgr.x (mgr.24542) 4257 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:26:35.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: cluster 2021-07-31T03:26:33.216417 2021-07-31T03:26:35.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: +0000 mgr.x (mgr.24542) 4258 : cluster [DBG] pgmap v4987: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:35.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: cephadm 2021-07-31T03:26:33.240503+0000 mgr.x (mgr.24542) 4259 : cephadm [ERR] 2021-07-31T03:26:35.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:26:35.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:26:35.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: yield 2021-07-31T03:26:35.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:26:35.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:26:35.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:26:35.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:26:35.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:26:35.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:26:35.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:26:35.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:26:35.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:26:35.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: task.result() 2021-07-31T03:26:35.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:26:35.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: cephadm 2021-07-31T03:26 2021-07-31T03:26:35.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: :33.241134+0000 mgr.x (mgr.24542) 4260 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:26:35.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:26:35.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:26:35.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: yield 2021-07-31T03:26:35.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:26:35.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:26:35.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:26:35.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:26:35.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:26:35.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:26:35.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:26:35.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:26:35.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:26:35.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: task.result() 2021-07-31T03:26:35.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:26:35.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: 2021-07-31T03:26:35.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:26:35.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: 2021-07-31T03:26:35.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:26:35.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:26:35.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:26:35.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:26:35.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:26:35.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:26:35.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:26:35.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:26:35.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:26:35.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:26:35.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:26:35.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:26:35.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:26:35.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:26:35.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:26:35.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:26:35.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:26:35.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:26:35.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: raise 2021-07-31T03:26:35.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:26:35.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:26:35.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:26:35.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:26:35.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:26:35.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: cephadm 2021-07-31T03:26:33.249452+0000 mgr.x (mgr.24542) 4261 : cephadm [ERR] 2021-07-31T03:26:35.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:26:35.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:26:35.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: yield 2021-07-31T03:26:35.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:26:35.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:26:35.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:26:35.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:26:35.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:26:35.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:26:35.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:26:35.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:26:35.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:26:35.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: task.result() 2021-07-31T03:26:35.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:26:35.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: cephadm 2021 2021-07-31T03:26:35.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: -07-31T03:26:33.249799+0000 mgr.x (mgr.24542) 4262 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:26:35.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:26:35.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:26:35.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: yield 2021-07-31T03:26:35.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:26:35.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:26:35.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:26:35.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:26:35.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:26:35.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:26:35.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:26:35.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:26:35.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:26:35.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: task.result() 2021-07-31T03:26:35.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:26:35.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: 2021-07-31T03:26:35.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:26:35.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: 2021-07-31T03:26:35.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:26:35.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:26:35.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:26:35.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:26:35.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:26:35.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:26:35.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:26:35.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:26:35.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:26:35.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:26:35.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:26:35.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:26:35.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:26:35.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:26:35.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:26:35.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:26:35.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:26:35.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:26:35.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: raise 2021-07-31T03:26:35.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:26:35.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:26:35.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:26:35.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:26:35.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:34 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:26:35.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: cephadm 2021-07-31T03:26:33.160778+0000 mgr.x (mgr.24542) 4253 : cephadm 2021-07-31T03:26:35.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:26:35.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: cephadm 2021-07-31T03:26:33.203582+0000 mgr.x (mgr.24542) 4254 : cephadm [ERR] 2021-07-31T03:26:35.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:26:35.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:26:35.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: yield 2021-07-31T03:26:35.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:26:35.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:26:35.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:26:35.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:26:35.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:26:35.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:26:35.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:26:35.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:26:35.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:26:35.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: task.result() 2021-07-31T03:26:35.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:26:35.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: cephadm 2021-07-31T03:26:33.204054+0000 mgr.x (mgr.24542) 4255 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:26:35.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: cephadm 2021-07-31T03:26:33.214887+0000 mgr.x (mgr.24542) 4256 : cephadm [ERR] 2021-07-31T03:26:35.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:26:35.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:26:35.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: yield 2021-07-31T03:26:35.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:26:35.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:26:35.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:26:35.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:26:35.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:26:35.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:26:35.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:26:35.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:26:35.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:26:35.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: task.result() 2021-07-31T03:26:35.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:26:35.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: cephadm 2021- 2021-07-31T03:26:35.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: 07-31T03:26:33.215337+0000 mgr.x (mgr.24542) 4257 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:26:35.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: cluster 2021-07-31T03:26:33.216417+0000 mgr.x (mgr.24542) 4258 : cluster [DBG] pgmap v4987: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:35.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: cephadm 2021-07-31T03:26:33.240503+0000 mgr.x (mgr.24542 2021-07-31T03:26:35.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: ) 4259 : cephadm [ERR] 2021-07-31T03:26:35.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:26:35.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:26:35.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: yield 2021-07-31T03:26:35.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:26:35.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:26:35.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:26:35.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:26:35.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:26:35.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:26:35.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:26:35.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:26:35.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:26:35.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: task.result() 2021-07-31T03:26:35.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:26:35.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: cephadm 2021-07-31T03:26: 2021-07-31T03:26:35.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: 33.241134+0000 mgr.x (mgr.24542) 4260 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:26:35.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:26:35.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:26:35.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: yield 2021-07-31T03:26:35.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:26:35.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:26:35.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:26:35.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:26:35.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:26:35.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:26:35.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:26:35.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:26:35.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:26:35.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: task.result() 2021-07-31T03:26:35.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:26:35.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: 2021-07-31T03:26:35.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:26:35.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: 2021-07-31T03:26:35.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:26:35.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:26:35.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:26:35.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:26:35.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:26:35.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:26:35.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:26:35.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:26:35.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:26:35.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:26:35.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:26:35.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:26:35.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:26:35.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:26:35.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:26:35.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:26:35.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:26:35.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:26:35.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: raise 2021-07-31T03:26:35.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:26:35.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:26:35.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:26:35.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:26:35.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:26:35.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: cephadm 2021-07-31T03:26:33.249452+0000 mgr.x (mgr.24542) 4261 : cephadm [ERR] 2021-07-31T03:26:35.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:26:35.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:26:35.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: yield 2021-07-31T03:26:35.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:26:35.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:26:35.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:26:35.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:26:35.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:26:35.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:26:35.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:26:35.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:26:35.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:26:35.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: task.result() 2021-07-31T03:26:35.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:26:35.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: cephadm 2021-07-31T03:26:33.249799+0000 mgr.x (mgr.24542) 4262 : cephadm [ERR] 2021-07-31T03:26:35.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:26:35.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:26:35.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:26:35.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: yield 2021-07-31T03:26:35.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:26:35.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:26:35.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:26:35.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:26:35.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:26:35.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:26:35.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:26:35.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:26:35.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:26:35.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: task.result() 2021-07-31T03:26:35.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:26:35.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: 2021-07-31T03:26:35.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:26:35.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: 2021-07-31T03:26:35.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:26:35.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:26:35.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:26:35.057 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:26:35.057 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:26:35.057 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:26:35.057 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:26:35.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:26:35.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:26:35.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:26:35.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:26:35.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:26:35.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:26:35.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:26:35.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:26:35.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:26:35.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:26:35.060 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:26:35.060 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: raise 2021-07-31T03:26:35.060 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:26:35.060 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:26:35.060 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:26:35.061 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:26:35.061 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:26:35.062 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:26:35.062 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:26:35.062 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: yield 2021-07-31T03:26:35.062 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:26:35.062 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:26:35.063 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:26:35.063 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:26:35.063 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:26:35.063 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:26:35.064 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:26:35.064 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:26:35.064 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:26:35.064 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: task.result() 2021-07-31T03:26:35.064 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:26:35.065 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: cephadm 2021-07-31T03:26 2021-07-31T03:26:35.065 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: :33.204054+0000 mgr.x (mgr.24542) 4255 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:26:35.065 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: cephadm 2021-07-31T03:26:33.214887+0000 mgr.x (mgr.24542) 4256 : cephadm [ERR] 2021-07-31T03:26:35.065 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:26:35.065 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:26:35.066 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: yield 2021-07-31T03:26:35.066 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:26:35.066 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:26:35.066 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:26:35.066 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:26:35.067 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:26:35.067 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:26:35.067 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:26:35.067 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:26:35.067 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:26:35.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: task.result() 2021-07-31T03:26:35.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:26:35.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: cephadm 2021- 2021-07-31T03:26:35.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: 07-31T03:26:33.215337+0000 mgr.x (mgr.24542) 4257 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:26:35.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: cluster 2021-07-31T03:26:33.216417+0000 mgr.x (mgr.24542) 4258 : cluster [DBG] pgmap v4987: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:35.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: cephadm 2021-07-31T03:26:33. 2021-07-31T03:26:35.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: 240503+0000 mgr.x (mgr.24542) 4259 : cephadm 2021-07-31T03:26:35.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: [ERR] 2021-07-31T03:26:35.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:26:35.070 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:26:35.070 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: yield 2021-07-31T03:26:35.070 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:26:35.070 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:26:35.070 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:26:35.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:26:35.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:26:35.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:26:35.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:26:35.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:26:35.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:26:35.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: task.result() 2021-07-31T03:26:35.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:26:35.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: cephadm 2021-07-31T03:26:33.241134+0000 mgr.x (mgr.24542) 4260 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:26:35.073 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:26:35.073 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:26:35.073 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: yield 2021-07-31T03:26:35.073 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:26:35.073 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:26:35.074 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:26:35.074 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:26:35.074 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:26:35.074 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:26:35.074 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:26:35.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:26:35.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:26:35.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: task.result() 2021-07-31T03:26:35.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:26:35.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: 2021-07-31T03:26:35.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:26:35.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: 2021-07-31T03:26:35.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:26:35.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:26:35.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:26:35.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:26:35.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:26:35.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:26:35.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:26:35.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:26:35.078 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:26:35.078 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:26:35.078 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:26:35.078 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:26:35.079 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:26:35.079 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:26:35.079 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:26:35.079 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:26:35.079 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:26:35.080 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:26:35.080 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: raise 2021-07-31T03:26:35.080 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:26:35.080 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:26:35.080 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:26:35.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:26:35.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:26:35.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: cephadm 2021-07-31T03:26:33.249452+0000 mgr.x ( 2021-07-31T03:26:35.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: mgr.24542) 4261 : cephadm [ERR] 2021-07-31T03:26:35.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:26:35.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:26:35.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: yield 2021-07-31T03:26:35.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:26:35.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:26:35.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:26:35.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:26:35.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:26:35.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:26:35.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:26:35.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:26:35.084 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:26:35.084 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: task.result() 2021-07-31T03:26:35.084 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:26:35.084 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: cephadm 2021-07-31T03:26:33.249799+0000 mgr.x (mgr.24542) 4262 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:26:35.084 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:26:35.085 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:26:35.085 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: yield 2021-07-31T03:26:35.085 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:26:35.085 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:26:35.085 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:26:35.085 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:26:35.086 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:26:35.086 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:26:35.086 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:26:35.086 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:26:35.086 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:26:35.087 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: task.result() 2021-07-31T03:26:35.087 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:26:35.087 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: 2021-07-31T03:26:35.087 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:26:35.087 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: 2021-07-31T03:26:35.088 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:26:35.088 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:26:35.088 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:26:35.088 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:26:35.088 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:26:35.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:26:35.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:26:35.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:26:35.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:26:35.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:26:35.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:26:35.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:26:35.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:26:35.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:26:35.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:26:35.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:26:35.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:26:35.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:26:35.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: raise 2021-07-31T03:26:35.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:26:35.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:26:35.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:26:35.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:26:35.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:34 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:26:37.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:36 smithi154 conmon[34553]: cluster 2021-07-31T03:26:35.216785+0000 mgr.x (mgr.24542) 4263 : cluster [DBG] pgmap v4988: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:37.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:36 smithi045 conmon[33655]: cluster 2021-07-31T03:26:35.216785+0000 mgr.x (mgr.24542) 4263 2021-07-31T03:26:37.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:36 smithi045 conmon[33655]: : cluster [DBG] pgmap v4988: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:37.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:36 smithi045 conmon[39718]: cluster 2021-07-31T03:26:35.216785+0000 mgr.x 2021-07-31T03:26:37.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:36 smithi045 conmon[39718]: (mgr.24542) 4263 : cluster [DBG] pgmap v4988: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:39.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:38 smithi154 conmon[34553]: cluster 2021-07-31T03:26:37.217181+0000 mgr.x (mgr.24542) 4264 : cluster [DBG] pgmap v4989: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:39.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:38 smithi045 conmon[33655]: cluster 2021-07-31T03:26:37.217181+0000 mgr.x (mgr. 2021-07-31T03:26:39.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:38 smithi045 conmon[33655]: 24542) 4264 : cluster [DBG] pgmap v4989: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:39.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:38 smithi045 conmon[39718]: cluster 2021-07-31T03:26:37.217181+0000 2021-07-31T03:26:39.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:38 smithi045 conmon[39718]: mgr.x (mgr.24542) 4264 : cluster [DBG] pgmap v4989: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:41.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:40 smithi154 conmon[34553]: cluster 2021-07-31T03:26:39.217578+0000 mgr.x (mgr.24542) 4265 : cluster [DBG] pgmap v4990: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:41.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:40 smithi045 conmon[33655]: cluster 2021-07-31T03:26:39.217578+0000 mgr.x (mgr.24542) 4265 : cluster [DBG] pgmap v4990: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:41.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:40 smithi045 conmon[39718]: cluster 2021-07-31T03:26:39.217578+0000 mgr.x (mgr.24542) 2021-07-31T03:26:41.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:40 smithi045 conmon[39718]: 4265 : cluster [DBG] pgmap v4990: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:43.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:42 smithi154 conmon[34553]: cluster 2021-07-31T03:26:41.218016+0000 mgr.x (mgr.24542) 4266 : cluster [DBG] pgmap v4991: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:43.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:42 smithi045 conmon[33655]: cluster 2021-07-31T03:26:41.218016+0000 mgr.x (mgr.24542) 4266 : cluster 2021-07-31T03:26:43.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:42 smithi045 conmon[33655]: [DBG] pgmap v4991: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:43.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:42 smithi045 conmon[39718]: cluster 2021-07-31T03:26:41.218016+0000 mgr.x ( 2021-07-31T03:26:43.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:42 smithi045 conmon[39718]: mgr.24542) 4266 : cluster [DBG] pgmap v4991: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:45.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:44 smithi154 conmon[34553]: cluster 2021-07-31T03:26:43.218506+0000 mgr.x (mgr.24542) 4267 : cluster [DBG] pgmap v4992: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:45.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:44 smithi045 conmon[39718]: cluster 2021-07-31T03:26:43.218506+0000 mgr.x (mgr 2021-07-31T03:26:45.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:44 smithi045 conmon[39718]: .24542) 4267 : cluster [DBG] pgmap v4992: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:45.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:44 smithi045 conmon[33655]: cluster 2021-07-31T03:26:43.218506+0000 mgr.x (mgr.24542) 4267 2021-07-31T03:26:45.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:44 smithi045 conmon[33655]: : cluster [DBG] pgmap v4992: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:47.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:46 smithi154 conmon[34553]: cluster 2021-07-31T03:26:45.218987+0000 mgr.x (mgr.24542) 4268 : cluster [DBG] pgmap v4993: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:47.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:46 smithi045 conmon[39718]: cluster 2021-07-31T03:26:45.218987+0000 mgr.x 2021-07-31T03:26:47.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:46 smithi045 conmon[39718]: (mgr.24542) 4268 : cluster [DBG] pgmap v4993: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:47.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:46 smithi045 conmon[33655]: cluster 2021-07-31T03:26:45.218987+0000 mgr.x (mgr 2021-07-31T03:26:47.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:46 smithi045 conmon[33655]: .24542) 4268 : cluster [DBG] pgmap v4993: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:49.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:48 smithi154 conmon[34553]: cluster 2021-07-31T03:26:47.219382+0000 mgr.x (mgr.24542) 4269 : cluster [DBG] pgmap v4994: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:49.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:48 smithi045 conmon[39718]: cluster 2021-07-31T03:26:47.219382+0000 mgr.x (mgr.24542) 4269 : cluster [DBG] 2021-07-31T03:26:49.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:48 smithi045 conmon[39718]: pgmap v4994: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:49.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:48 smithi045 conmon[33655]: cluster 2021-07-31T03:26:47.219382+0000 mgr.x (mgr. 2021-07-31T03:26:49.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:48 smithi045 conmon[33655]: 24542) 4269 : cluster [DBG] pgmap v4994: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:51.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:50 smithi154 conmon[34553]: cluster 2021-07-31T03:26:49.219912+0000 mgr.x (mgr.24542) 4270 : cluster [DBG] 2021-07-31T03:26:51.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:50 smithi154 conmon[34553]: pgmap v4995: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:51.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:50 smithi045 conmon[39718]: cluster 2021-07-31T03:26:49.219912+0000 mgr.x (mgr.24542) 4270 : cluster [DBG] 2021-07-31T03:26:51.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:50 smithi045 conmon[39718]: pgmap v4995: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:51.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:50 smithi045 conmon[33655]: cluster 2021-07-31T03:26:49.219912+0000 mgr.x (mgr.24542) 4270 : cluster [DBG] pgmap v4995: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:53.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:52 smithi154 conmon[34553]: cluster 2021-07-31T03:26:51.220364+0000 mgr.x (mgr.24542) 4271 : cluster [DBG] pgmap v4996: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:53.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:52 smithi045 conmon[39718]: cluster 2021-07-31T03:26:51.220364+0000 mgr.x (mgr.24542) 4271 : cluster [DBG] 2021-07-31T03:26:53.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:52 smithi045 conmon[39718]: pgmap v4996: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:53.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:52 smithi045 conmon[33655]: cluster 2021-07-31T03:26:51.220364+0000 mgr.x (mgr.24542) 4271 : cluster [DBG] pgmap v4996: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:55.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:54 smithi154 conmon[34553]: cluster 2021-07-31T03:26:53.220886+0000 mgr.x (mgr.24542) 4272 : cluster [DBG] pgmap v4997: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:55.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:54 smithi045 conmon[39718]: cluster 2021-07-31T03:26:53.220886+0000 mgr.x (mgr.24542) 4272 : cluster [DBG] 2021-07-31T03:26:55.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:54 smithi045 conmon[39718]: pgmap v4997: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:55.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:54 smithi045 conmon[33655]: cluster 2021-07-31T03:26:53.220886+0000 mgr.x ( 2021-07-31T03:26:55.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:54 smithi045 conmon[33655]: mgr.24542) 4272 : cluster [DBG] pgmap v4997: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:56.709 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:26:56.985 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:26:56.985 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 101m ago 102m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:26:56.986 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 99m ago 102m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:26:56.986 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 99m ago 105m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:26:56.986 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 101m ago 107m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:26:56.986 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 101m ago 108m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:26:56.987 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 99m ago 106m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:26:56.987 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 101m ago 106m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:26:56.987 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 101m ago 102m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:26:56.987 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 99m ago 102m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:26:56.987 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 101m ago 105m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:26:56.988 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 101m ago 105m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:26:56.988 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 101m ago 104m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:26:56.988 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 101m ago 104m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:26:56.988 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 99m ago 104m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:26:56.988 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 99m ago 103m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:26:56.989 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 99m ago 103m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:26:56.989 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 99m ago 103m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:26:56.989 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 99m ago 102m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:26:57.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:56 smithi154 conmon[34553]: cluster 2021-07-31T03:26:55.221477+0000 mgr.x (mgr.24542) 4273 : cluster [DBG] pgmap v4998: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:57.024 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:56 smithi045 conmon[33655]: cluster 2021-07-31T03:26:55.221477+0000 mgr.x (mgr.24542 2021-07-31T03:26:57.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:56 smithi045 conmon[33655]: ) 4273 : cluster [DBG] pgmap v4998: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:57.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:56 smithi045 conmon[39718]: cluster 2021-07-31T03:26:55. 2021-07-31T03:26:57.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:56 smithi045 conmon[39718]: 221477+0000 mgr.x (mgr.24542) 4273 : cluster [DBG] pgmap v4998: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:57.320 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:26:57.320 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:26:57.320 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:26:57.320 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:26:57.321 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:26:57.321 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:26:57.321 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:26:57.321 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:26:57.321 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:26:57.322 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:26:57.322 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:26:57.322 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:26:57.322 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:26:57.322 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:26:57.323 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:26:57.323 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:26:57.323 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:26:58.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:57 smithi154 conmon[34553]: audit 2021-07-31T03:26:56.698687+0000 mgr.x (mgr.24542) 4274 : audit [DBG] from='client.29464 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:26:58.079 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:57 smithi154 conmon[34553]: audit 2021-07-31T03:26:57.320899+0000 mon.c (mon.1) 205 : audit [DBG] from='client.? 172.21.15.45:0/1153035053' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:26:58.080 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:57 smithi045 conmon[33655]: audit 2021-07-31T03:26:56.698687+0000 mgr.x (mgr.24542) 4274 : 2021-07-31T03:26:58.080 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:57 smithi045 conmon[33655]: audit [DBG] from='client.29464 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:26:58.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:57 smithi045 conmon[33655]: audit 2021-07-31T03:26:57.320899+0000 mon.c (mon.1) 205 : audit [DBG] from='client.? 172.21.15.45:0/1153035053' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:26:58.081 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:57 smithi045 conmon[39718]: audit 2021-07-31T03:26:56.698687 2021-07-31T03:26:58.081 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:57 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 4274 : audit [DBG] from='client.29464 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:26:58.081 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:57 smithi045 conmon[39718]: audit 2021-07-31T03:26:58.082 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:57 smithi045 conmon[39718]: 2021-07-31T03:26:57.320899+0000 mon.c (mon.1) 205 : audit [DBG] from='client.? 172.21.15.45:0/1153035053' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:26:58.082 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:57 smithi045 conmon[39718]: 2021-07-31T03:26:59.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:58 smithi154 conmon[34553]: audit 2021-07-31T03:26:56.978401+0000 mgr.x (mgr.24542) 4275 : audit [DBG] from='client.29470 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:26:59.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:58 smithi154 conmon[34553]: cluster 2021-07-31T03:26: 2021-07-31T03:26:59.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:26:58 smithi154 conmon[34553]: 57.221911+0000 mgr.x (mgr.24542) 4276 : cluster [DBG] pgmap v4999: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:59.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:58 smithi045 conmon[33655]: audit 2021-07-31T03:26:56.978401+0000 mgr.x (mgr.24542) 4275 : audit [DBG] from='client.29470 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:26:59.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:58 smithi045 conmon[33655]: 2021-07-31T03:26:59.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:26:58 smithi045 conmon[33655]: cluster 2021-07-31T03:26:57.221911+0000 mgr.x (mgr.24542) 4276 : cluster [DBG] pgmap v4999: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:26:59.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:58 smithi045 conmon[39718]: audit 2021-07-31T03:26:56.978401+0000 mgr.x (mgr. 2021-07-31T03:26:59.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:58 smithi045 conmon[39718]: 24542) 4275 : audit [DBG] from='client.29470 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:26:59.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:58 smithi045 conmon[39718]: cluster 2021-07-31T03:26:57.221911+0000 mgr.x ( 2021-07-31T03:26:59.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:26:58 smithi045 conmon[39718]: mgr.24542) 4276 : cluster [DBG] pgmap v4999: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:01.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:00 smithi154 conmon[34553]: cluster 2021-07-31T03:26:59.222361+0000 mgr.x (mgr.24542) 4277 : cluster [DBG] pgmap v5000: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:01.024 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:00 smithi045 conmon[33655]: cluster 2021-07-31T03:26:59.222361+0000 mgr.x (mgr.24542) 4277 : cluster 2021-07-31T03:27:01.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:00 smithi045 conmon[33655]: [DBG] pgmap v5000: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:01.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:00 smithi045 conmon[39718]: cluster 2021-07-31T03:26:59.222361+0000 mgr.x ( 2021-07-31T03:27:01.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:00 smithi045 conmon[39718]: mgr.24542) 4277 : cluster [DBG] pgmap v5000: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:03.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:02 smithi154 conmon[34553]: cluster 2021-07-31T03:27:01.222842+0000 mgr.x (mgr.24542) 4278 : cluster [DBG] 2021-07-31T03:27:03.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:02 smithi154 conmon[34553]: pgmap v5001: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:03.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:02 smithi045 conmon[33655]: cluster 2021-07-31T03:27:01. 2021-07-31T03:27:03.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:02 smithi045 conmon[33655]: 222842+0000 mgr.x (mgr.24542) 4278 : cluster [DBG] pgmap v5001: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:03.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:02 smithi045 conmon[39718]: cluster 2021-07-31T03: 2021-07-31T03:27:03.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:02 smithi045 conmon[39718]: 27:01.222842+0000 mgr.x (mgr.24542) 4278 : cluster [DBG] pgmap v5001: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:05.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:04 smithi154 conmon[34553]: cluster 2021-07-31T03:27:03.223300+0000 mgr.x (mgr.24542) 4279 : cluster [DBG] pgmap v5002: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:05.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:04 smithi045 conmon[33655]: cluster 2021-07-31T03:27:03.223300+0000 mgr.x (mgr.24542) 2021-07-31T03:27:05.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:04 smithi045 conmon[33655]: 4279 : cluster [DBG] pgmap v5002: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:05.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:04 smithi045 conmon[39718]: cluster 2021-07-31T03:27:03.223300+0000 mgr.x ( 2021-07-31T03:27:05.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:04 smithi045 conmon[39718]: mgr.24542) 4279 : cluster [DBG] pgmap v5002: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:07.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:06 smithi154 conmon[34553]: cluster 2021-07-31T03:27:05.223841+0000 mgr.x (mgr.24542) 4280 : cluster [DBG] pgmap v5003: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:07.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:06 smithi154 conmon[34553]: 2021-07-31T03:27:07.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:06 smithi045 conmon[33655]: cluster 2021-07-31T03:27:05.223841+0000 mgr.x (mgr.24542) 4280 : cluster [DBG] pgmap v5003: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:07.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:06 smithi045 conmon[39718]: cluster 2021-07-31T03:27:05.223841+0000 mgr.x (mgr.24542) 4280 : cluster [DBG] pgmap v5003: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:09.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:08 smithi154 conmon[34553]: cluster 2021-07-31T03:27:07.224350+0000 mgr.x (mgr.24542) 4281 : cluster [DBG] pgmap v5004: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:09.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:08 smithi045 conmon[33655]: cluster 2021-07-31T03: 2021-07-31T03:27:09.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:08 smithi045 conmon[33655]: 27:07.224350+0000 mgr.x (mgr.24542) 4281 : cluster [DBG] pgmap v5004: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:09.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:08 smithi045 conmon[39718]: cluster 2021-07-31T03:27: 2021-07-31T03:27:09.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:08 smithi045 conmon[39718]: 07.224350+0000 mgr.x (mgr.24542) 4281 : cluster [DBG] pgmap v5004: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:11.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:10 smithi154 conmon[34553]: cluster 2021-07-31T03:27:09.224816+0000 mgr.x (mgr.24542) 4282 : cluster [DBG] pgmap v5005: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:11.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:10 smithi045 conmon[33655]: cluster 2021-07-31T03:27:09 2021-07-31T03:27:11.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:10 smithi045 conmon[33655]: .224816+0000 mgr.x (mgr.24542) 4282 : cluster [DBG] pgmap v5005: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:11.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:10 smithi045 conmon[39718]: cluster 2021-07-31T03:27:09. 2021-07-31T03:27:11.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:10 smithi045 conmon[39718]: 224816+0000 mgr.x (mgr.24542) 4282 : cluster [DBG] pgmap v5005: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:13.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:12 smithi154 conmon[34553]: cluster 2021-07-31T03:27:11.225228+0000 mgr.x (mgr.24542) 4283 : cluster [DBG] pgmap v5006: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:13.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:12 smithi154 conmon[34553]: 2021-07-31T03:27:13.024 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:12 smithi045 conmon[33655]: cluster 2021-07-31T03:27:11.225228+0000 mgr.x (mgr.24542) 4283 : cluster 2021-07-31T03:27:13.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:12 smithi045 conmon[33655]: [DBG] pgmap v5006: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:13.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:12 smithi045 conmon[39718]: cluster 2021-07-31T03:27:11.225228 2021-07-31T03:27:13.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:12 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 4283 : cluster [DBG] pgmap v5006: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:15.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:14 smithi154 conmon[34553]: cluster 2021-07-31T03:27:13.225747+0000 mgr.x (mgr.24542) 4284 : cluster [DBG] pgmap v5007: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:15.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:14 smithi154 conmon[34553]: 2021-07-31T03:27:15.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:14 smithi045 conmon[33655]: cluster 2021-07-31T03:27:13.225747+0000 mgr.x (mgr.24542) 4284 : cluster [DBG] pgmap v5007: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:15.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:14 smithi045 conmon[39718]: cluster 2021-07-31T03:27:13.225747+0000 mgr.x 2021-07-31T03:27:15.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:14 smithi045 conmon[39718]: (mgr.24542) 4284 : cluster [DBG] pgmap v5007: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:17.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:16 smithi154 conmon[34553]: cluster 2021-07-31T03:27:15.226230+0000 mgr.x (mgr.24542) 4285 : cluster [DBG] pgmap v5008: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:17.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:16 smithi045 conmon[33655]: cluster 2021-07-31T03:27:15.226230+0000 mgr.x (mgr.24542) 2021-07-31T03:27:17.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:16 smithi045 conmon[33655]: 4285 : cluster [DBG] pgmap v5008: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:17.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:16 smithi045 conmon[39718]: cluster 2021-07-31T03:27:15.226230+0000 mgr.x (mgr. 2021-07-31T03:27:17.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:16 smithi045 conmon[39718]: 24542) 4285 : cluster [DBG] pgmap v5008: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:19.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:18 smithi154 conmon[34553]: cluster 2021-07-31T03:27:17.226754+0000 mgr.x (mgr.24542) 4286 : cluster 2021-07-31T03:27:19.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:18 smithi154 conmon[34553]: [DBG] pgmap v5009: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:19.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:18 smithi045 conmon[33655]: cluster 2021-07-31T03:27:17.226754+0000 mgr.x (mgr.24542) 4286 : cluster [DBG] pgmap v5009: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:19.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:18 smithi045 conmon[33655]: 2021-07-31T03:27:19.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:18 smithi045 conmon[39718]: cluster 2021-07-31T03:27:17.226754+0000 mgr.x (mgr 2021-07-31T03:27:19.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:18 smithi045 conmon[39718]: .24542) 4286 : cluster [DBG] pgmap v5009: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:21.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:20 smithi154 conmon[34553]: cluster 2021-07-31T03:27:19.227234+0000 mgr.x (mgr.24542) 4287 : cluster [DBG] pgmap v5010: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:21.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:20 smithi045 conmon[33655]: cluster 2021-07-31T03:27:19 2021-07-31T03:27:21.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:20 smithi045 conmon[33655]: .227234+0000 mgr.x (mgr.24542) 4287 : cluster [DBG] pgmap v5010: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:21.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:20 smithi045 conmon[39718]: cluster 2021-07-31T03:27:19.227234 2021-07-31T03:27:21.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:20 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 4287 : cluster [DBG] pgmap v5010: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:23.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:22 smithi154 conmon[34553]: cluster 2021-07-31T03:27:21.227664+0000 mgr.x (mgr.24542) 4288 : cluster [DBG] pgmap v5011: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:23.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:22 smithi045 conmon[39718]: cluster 2021-07-31T03:27:21.227664+0000 2021-07-31T03:27:23.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:22 smithi045 conmon[39718]: mgr.x (mgr.24542) 4288 : cluster [DBG] pgmap v5011: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:23.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:22 smithi045 conmon[33655]: cluster 2021-07-31T03:27:21. 2021-07-31T03:27:23.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:22 smithi045 conmon[33655]: 227664+0000 mgr.x (mgr.24542) 4288 : cluster [DBG] pgmap v5011: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:25.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:24 smithi154 conmon[34553]: cluster 2021-07-31T03:27:23.228088+0000 mgr.x (mgr.24542) 4289 : cluster [DBG] pgmap v5012: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:25.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:24 smithi154 conmon[34553]: 2021-07-31T03:27:25.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:24 smithi045 conmon[39718]: cluster 2021-07-31T03:27:23.228088+0000 mgr.x (mgr.24542) 4289 : cluster [DBG] pgmap v5012: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:25.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:24 smithi045 conmon[33655]: cluster 2021-07-31T03 2021-07-31T03:27:25.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:24 smithi045 conmon[33655]: :27:23.228088+0000 mgr.x (mgr.24542) 4289 : cluster [DBG] pgmap v5012: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:27.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:26 smithi154 conmon[34553]: cluster 2021-07-31T03:27:25.228477+0000 mgr.x (mgr.24542) 4290 : cluster [DBG] pgmap v5013: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:27.024 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:26 smithi045 conmon[39718]: cluster 2021-07-31T03:27:25.228477+0000 mgr.x (mgr.24542) 4290 : cluster [DBG] pgmap v5013: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:27.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:26 smithi045 conmon[33655]: cluster 2021-07-31T03:27:25.228477+0000 mgr.x (mgr.24542) 2021-07-31T03:27:27.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:26 smithi045 conmon[33655]: 4290 : cluster [DBG] pgmap v5013: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:27.632 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:27:27.907 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:27:27.907 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 102m ago 103m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:27:27.907 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 100m ago 102m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:27:27.908 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 100m ago 106m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:27:27.908 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 102m ago 108m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:27:27.908 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 102m ago 108m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:27:27.908 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 100m ago 106m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:27:27.908 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 102m ago 106m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:27:27.909 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 102m ago 103m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:27:27.909 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 100m ago 103m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:27:27.909 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 102m ago 106m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:27:27.909 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 102m ago 105m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:27:27.909 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 102m ago 105m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:27:27.910 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 102m ago 105m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:27:27.910 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 100m ago 104m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:27:27.910 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 100m ago 104m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:27:27.910 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 100m ago 104m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:27:27.911 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 100m ago 103m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:27:27.911 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 100m ago 103m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:27:28.243 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:27:28.244 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:27:28.244 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:27:28.244 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:27:28.244 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:27:28.245 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:27:28.245 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:27:28.245 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:27:28.245 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:27:28.245 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:27:28.246 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:27:28.246 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:27:28.246 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:27:28.246 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:27:28.246 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:27:28.247 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:27:28.247 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:27:29.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:28 smithi154 conmon[34553]: cluster 2021-07-31T03:27:27.228857+0000 mgr.x (mgr.24542) 4291 : cluster [DBG] pgmap v5014: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:29.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:28 smithi154 conmon[34553]: audit 2021-07-31T03:27:27.620890+0000 mgr.x (mgr.24542) 4292 : audit [DBG] from='client.19542 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:27:29.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:28 smithi154 conmon[34553]: audit 2021-07-31T03:27:27.900343+0000 mgr.x (mgr.24542) 4293 : audit [DBG] from='client.29488 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:27:29.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:28 smithi154 conmon[34553]: audit 2021-07-31T03:27 2021-07-31T03:27:29.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:28 smithi154 conmon[34553]: :28.244763+0000 mon.c (mon.1) 206 2021-07-31T03:27:29.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:28 smithi154 conmon[34553]: : audit [DBG] from='client.? 172.21.15.45:0/3881532387' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:27:29.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:28 smithi154 conmon[34553]: audit 2021-07-31T03:27:29.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:28 smithi154 conmon[34553]: 2021-07-31T03:27: 2021-07-31T03:27:29.035 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:28 smithi154 conmon[34553]: 28.333334+0000 mon.b (mon.2) 575 2021-07-31T03:27:29.035 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:28 smithi154 conmon[34553]: : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:27:29.035 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:28 smithi154 conmon[34553]: audit 2021 2021-07-31T03:27:29.035 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:28 smithi154 conmon[34553]: -07-31T03:27: 2021-07-31T03:27:29.036 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:28 smithi154 conmon[34553]: 28.333603+0000 mon.b (mon. 2021-07-31T03:27:29.036 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:28 smithi154 conmon[34553]: 2) 576 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:27:29.036 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:28 smithi154 conmon[34553]: audit 2021-07 2021-07-31T03:27:29.036 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:28 smithi154 conmon[34553]: -31T03:27:28.333825+0000 mon.a (mon 2021-07-31T03:27:29.036 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:28 smithi154 conmon[34553]: .0) 3306 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:27:29.037 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:28 smithi154 conmon[34553]: audit 2021-07-31 2021-07-31T03:27:29.037 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:28 smithi154 conmon[34553]: T03:27:28.334132+0000 2021-07-31T03:27:29.037 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:28 smithi154 conmon[34553]: mon.a (mon.0) 3307 : audit [INF] 2021-07-31T03:27:29.037 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:28 smithi154 conmon[34553]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:27:29.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:28 smithi045 conmon[39718]: cluster 2021-07-31T03:27:27.228857+0000 mgr.x 2021-07-31T03:27:29.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:28 smithi045 conmon[39718]: (mgr.24542) 4291 : cluster [DBG] pgmap v5014: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:29.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:28 smithi045 conmon[39718]: audit 2021-07-31T03:27:27 2021-07-31T03:27:29.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:28 smithi045 conmon[39718]: .620890+0000 mgr.x (mgr.24542) 4292 : audit [DBG] from='client.19542 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:27:29.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:28 smithi045 conmon[39718]: audit 2021 2021-07-31T03:27:29.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:28 smithi045 conmon[39718]: -07-31T03:27:27.900343 2021-07-31T03:27:29.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:28 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 4293 : audit 2021-07-31T03:27:29.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:28 smithi045 conmon[39718]: [DBG] from='client.29488 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:27:29.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:28 smithi045 conmon[39718]: audit 2021-07-31T03 2021-07-31T03:27:29.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:28 smithi045 conmon[39718]: :27:28.244763+0000 mon.c (mon. 2021-07-31T03:27:29.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:28 smithi045 conmon[39718]: 1) 206 : audit [DBG] from='client.? 172.21.15.45:0/3881532387' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:27:29.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:28 smithi045 conmon[39718]: audit 2021 2021-07-31T03:27:29.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:28 smithi045 conmon[39718]: -07-31T03:27:28.333334 2021-07-31T03:27:29.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:28 smithi045 conmon[39718]: +0000 mon.b (mon.2) 575 : audit 2021-07-31T03:27:29.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:28 smithi045 conmon[39718]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:27:29.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:28 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T03:27:29.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:28 smithi045 conmon[39718]: T03:27:28.333603+0000 mon.b ( 2021-07-31T03:27:29.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:28 smithi045 conmon[39718]: mon.2) 576 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:27:29.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:28 smithi045 conmon[39718]: 2021-07-31T03:27:29.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:28 smithi045 conmon[39718]: audit 2021-07-31T03:27:29.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:28 smithi045 conmon[39718]: 2021-07-31T03:27:28.333825+0000 2021-07-31T03:27:29.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:28 smithi045 conmon[39718]: mon.a (mon.0) 3306 : audit [INF] 2021-07-31T03:27:29.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:28 smithi045 conmon[39718]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:27:29.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:28 smithi045 conmon[39718]: audit 2021-07-31T03: 2021-07-31T03:27:29.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:28 smithi045 conmon[39718]: 27:28.334132+0000 mon.a (mon. 2021-07-31T03:27:29.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:28 smithi045 conmon[39718]: 0) 3307 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:27:29.044 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:28 smithi045 conmon[33655]: cluster 2021-07-31T03:27:27.228857+0000 mgr.x (mgr.24542) 2021-07-31T03:27:29.044 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:28 smithi045 conmon[33655]: 4291 : cluster [DBG] pgmap v5014: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:29.045 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:28 smithi045 conmon[33655]: audit 2021-07-31T03:27:27.620890+0000 mgr.x (mgr.24542) 4292 : audit [DBG] from='client.19542 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:27:29.045 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:28 smithi045 conmon[33655]: audit 2021-07-31T03: 2021-07-31T03:27:29.045 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:28 smithi045 conmon[33655]: 27:27.900343+0000 mgr.x (mgr.24542) 4293 : audit 2021-07-31T03:27:29.045 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:28 smithi045 conmon[33655]: [DBG] from='client.29488 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:27:29.045 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:28 smithi045 conmon[33655]: audit 2021-07 2021-07-31T03:27:29.046 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:28 smithi045 conmon[33655]: -31T03:27:28.244763+0000 mon.c ( 2021-07-31T03:27:29.046 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:28 smithi045 conmon[33655]: mon.1) 206 : audit [DBG] from='client.? 172.21.15.45:0/3881532387' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:27:29.046 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:28 smithi045 conmon[33655]: audit 2021-07-31T 2021-07-31T03:27:29.046 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:28 smithi045 conmon[33655]: 03:27:28.333334+0000 mon.b (mon.2) 575 : audit 2021-07-31T03:27:29.047 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:28 smithi045 conmon[33655]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:27:29.047 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:28 smithi045 conmon[33655]: audit 2021- 2021-07-31T03:27:29.047 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:28 smithi045 conmon[33655]: 07-31T03:27:28.333603+0000 mon.b 2021-07-31T03:27:29.047 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:28 smithi045 conmon[33655]: (mon.2) 576 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:27:29.047 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:28 smithi045 conmon[33655]: 2021-07-31T03:27:29.048 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:28 smithi045 conmon[33655]: audit 2021-07-31T03 2021-07-31T03:27:29.048 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:28 smithi045 conmon[33655]: :27:28.333825+0000 mon.a (mon.0 2021-07-31T03:27:29.048 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:28 smithi045 conmon[33655]: ) 3306 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:27:29.048 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:28 smithi045 conmon[33655]: audit 2021-07-31T03:27:29.048 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:28 smithi045 conmon[33655]: 2021-07-31T03:27:28.334132+0000 2021-07-31T03:27:29.049 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:28 smithi045 conmon[33655]: mon.a (mon.0) 3307 : audit [INF] 2021-07-31T03:27:29.049 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:28 smithi045 conmon[33655]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:27:31.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:30 smithi154 conmon[34553]: cluster 2021-07-31T03:27:29.229467+0000 mgr.x (mgr.24542) 4294 : cluster [DBG] pgmap v5015: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:31.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:30 smithi045 conmon[39718]: cluster 2021-07-31T03:27:29.229467+0000 mgr.x 2021-07-31T03:27:31.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:30 smithi045 conmon[39718]: (mgr.24542) 4294 : cluster [DBG] pgmap v5015: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:31.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:30 smithi045 conmon[33655]: cluster 2021-07-31T03:27:29.229467+0000 mgr.x (mgr.24542) 4294 : cluster 2021-07-31T03:27:31.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:30 smithi045 conmon[33655]: [DBG] pgmap v5015: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:33.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:32 smithi154 conmon[34553]: cluster 2021-07-31T03:27:31.229839+0000 mgr.x (mgr.24542) 4295 : cluster [DBG] pgmap v5016: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:33.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:32 smithi154 conmon[34553]: 2021-07-31T03:27:33.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:32 smithi045 conmon[39718]: cluster 2021-07-31T03:27:31.229839+0000 mgr.x 2021-07-31T03:27:33.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:32 smithi045 conmon[39718]: (mgr.24542) 4295 : cluster [DBG] pgmap v5016: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:33.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:32 smithi045 conmon[33655]: cluster 2021-07-31T03:27:31.229839+0000 mgr.x (mgr.24542) 4295 : cluster 2021-07-31T03:27:33.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:32 smithi045 conmon[33655]: [DBG] pgmap v5016: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:33.752 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: debug 2021-07-31T03:27:33.294+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:27:33.753 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:27:33.753 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:27:33.753 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: yield 2021-07-31T03:27:33.753 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:27:33.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:27:33.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:27:33.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:27:33.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:27:33.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:27:33.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:27:33.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:27:33.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:27:33.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: task.result() 2021-07-31T03:27:33.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:27:33.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: debug 2021-07-31T03:27:33.305+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:27:33.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:27:33.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:27:33.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: yield 2021-07-31T03:27:33.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:27:33.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:27:33.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:27:33.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:27:33.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:27:33.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:27:33.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:27:33.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:27:33.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:27:33.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: task.result() 2021-07-31T03:27:33.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:27:33.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: debug 2021-07-31T03:27:33.330+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:27:33.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:27:33.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:27:33.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: yield 2021-07-31T03:27:33.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:27:33.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:27:33.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:27:33.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:27:33.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:27:33.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:27:33.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:27:33.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:27:33.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:27:33.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: task.result() 2021-07-31T03:27:33.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:27:33.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: debug 2021-07-31T03:27:33.331+0000 7fcb94162700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:27:33.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:27:33.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:27:33.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: yield 2021-07-31T03:27:33.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:27:33.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:27:33.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:27:33.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:27:33.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:27:33.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:27:33.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:27:33.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:27:33.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:27:33.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: task.result() 2021-07-31T03:27:33.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:27:33.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: 2021-07-31T03:27:33.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:27:33.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: 2021-07-31T03:27:33.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:27:33.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:27:33.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:27:33.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:27:33.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:27:33.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:27:33.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:27:33.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:27:33.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:27:33.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:27:33.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:27:33.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:27:33.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:27:33.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:27:33.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:27:33.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:27:33.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:27:33.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:27:33.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: raise 2021-07-31T03:27:33.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:27:33.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:27:33.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:27:33.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:27:33.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:27:33.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: debug 2021-07-31T03:27:33.339+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:27:33.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:27:33.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:27:33.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: yield 2021-07-31T03:27:33.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:27:33.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:27:33.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:27:33.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:27:33.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:27:33.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:27:33.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:27:33.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:27:33.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:27:33.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: task.result() 2021-07-31T03:27:33.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:27:33.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: debug 2021-07-31T03:27:33.340+0000 7fcb93961700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:27:33.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:27:33.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:27:33.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: yield 2021-07-31T03:27:33.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:27:33.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:27:33.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:27:33.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:27:33.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:27:33.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:27:33.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:27:33.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:27:33.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:27:33.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: task.result() 2021-07-31T03:27:33.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:27:33.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: 2021-07-31T03:27:33.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:27:33.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: 2021-07-31T03:27:33.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:27:33.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:27:33.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:27:33.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:27:33.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:27:33.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:27:33.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:27:33.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:27:33.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:27:33.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:27:33.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:27:33.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:27:33.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:27:33.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:27:33.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:27:33.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:27:33.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:27:33.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:27:33.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: raise 2021-07-31T03:27:33.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:27:33.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:27:33.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:27:33.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:27:33.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:27:33 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:27:34.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: cluster 2021-07-31T03:27:33.230173+0000 mgr.x (mgr.24542) 4296 : cluster [DBG] pgmap v5017: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:34.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: audit 2021-07-31T03:27:33.250482+0000 mon.b (mon.2) 577 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:27:34.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: audit 2021-07-31T03:27:33.251816+0000 mon.b (mon.2) 578 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:27:34.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: audit 2021-07-31T03:27:33.252741 2021-07-31T03:27:34.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: +0000 mon.b (mon.2) 579 : audit [INF] 2021-07-31T03:27:34.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:27:34.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: cephadm 2021-07-31T03 2021-07-31T03:27:34.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: :27:33.253525+0000 mgr.x (mgr.24542 2021-07-31T03:27:34.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: ) 4297 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:27:34.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: cephadm 2021 2021-07-31T03:27:34.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: -07-31T03:27:33.296446+0000 mgr.x 2021-07-31T03:27:34.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: (mgr.24542) 4298 : cephadm [ERR] 2021-07-31T03:27:34.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:27:34.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:27:34.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: yield 2021-07-31T03:27:34.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:27:34.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:27:34.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:27:34.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:27:34.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:27:34.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:27:34.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:27:34.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:27:34.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:27:34.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: task.result() 2021-07-31T03:27:34.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:27:34.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: 2021-07-31T03:27:34.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: cephadm 2021-07-31T03:27:33.296885+0000 mgr.x (mgr.24542) 4299 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:27:34.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: cephadm 2021-07-31T03:27:33.307717+0000 mgr.x 2021-07-31T03:27:34.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: (mgr.24542) 4300 : cephadm [ERR] 2021-07-31T03:27:34.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:27:34.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:27:34.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: yield 2021-07-31T03:27:34.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:27:34.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:27:34.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:27:34.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:27:34.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:27:34.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:27:34.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:27:34.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:27:34.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:27:34.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: task.result() 2021-07-31T03:27:34.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:27:34.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: cephadm 2021-07-31T03:27:33.308572+0000 mgr.x (mgr.24542) 4301 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:27:34.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: cephadm 2021-07-31T03:27:33.332787+0000 mgr.x (mgr.24542) 4302 : cephadm [ERR] 2021-07-31T03:27:34.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:27:34.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:27:34.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: yield 2021-07-31T03:27:34.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:27:34.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:27:34.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:27:34.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:27:34.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:27:34.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:27:34.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:27:34.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:27:34.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:27:34.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: task.result() 2021-07-31T03:27:34.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:27:34.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: cephadm 2021-07-31T03:27:33.333237+0000 mgr.x (mgr.24542) 4303 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:27:34.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:27:34.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:27:34.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: yield 2021-07-31T03:27:34.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:27:34.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:27:34.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:27:34.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:27:34.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:27:34.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:27:34.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:27:34.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:27:34.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:27:34.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: task.result() 2021-07-31T03:27:34.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:27:34.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: 2021-07-31T03:27:34.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:27:34.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: 2021-07-31T03:27:34.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:27:34.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:27:34.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:27:34.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:27:34.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:27:34.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:27:34.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:27:34.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:27:34.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:27:34.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:27:34.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:27:34.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:27:34.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:27:34.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:27:34.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:27:34.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:27:34.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:27:34.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:27:34.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: raise 2021-07-31T03:27:34.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:27:34.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:27:34.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:27:34.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:27:34.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:27:34.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: cephadm 2021-07-31T03:27:33.341607+0000 mgr.x (mgr.24542 2021-07-31T03:27:34.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: ) 4304 : cephadm [ERR] 2021-07-31T03:27:34.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:27:34.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:27:34.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: yield 2021-07-31T03:27:34.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:27:34.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:27:34.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:27:34.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:27:34.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:27:34.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:27:34.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:27:34.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:27:34.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:27:34.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: task.result() 2021-07-31T03:27:34.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:27:34.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: cephadm 2021-07-31T03:27:33.342168+0000 mgr.x (mgr.24542) 4305 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:27:34.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:27:34.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:27:34.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: yield 2021-07-31T03:27:34.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:27:34.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:27:34.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:27:34.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:27:34.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:27:34.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:27:34.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:27:34.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:27:34.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:27:34.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: task.result() 2021-07-31T03:27:34.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:27:34.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: 2021-07-31T03:27:34.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:27:34.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: 2021-07-31T03:27:34.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:27:34.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:27:34.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:27:34.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:27:34.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:27:34.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:27:34.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:27:34.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:27:34.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:27:34.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:27:34.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:27:34.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:27:34.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:27:34.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:27:34.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:27:34.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:27:34.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:27:34.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:27:34.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: raise 2021-07-31T03:27:34.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:27:34.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:27:34.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:27:34.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:27:34.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:34 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:27:34.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: cluster 2021-07-31T03:27:33.230173+0000 mgr.x (mgr.24542 2021-07-31T03:27:34.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: ) 4296 : cluster [DBG] pgmap v5017: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:34.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: audit 2021-07-31T03:27:33.250482+0000 mon.b (mon.2) 577 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:27:34.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: audit 2021-07-31T03:27:33.251816+0000 mon.b (mon.2) 578 : audit 2021-07-31T03:27:34.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:27:34.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T03:27:34.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: T03:27:33.252741+0000 mon.b (mon.2) 579 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:27:34.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: cephadm 2021-07-31T03:27:33.253525+0000 mgr.x 2021-07-31T03:27:34.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: (mgr.24542) 4297 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:27:34.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: cephadm 2021-07-31T03:27:33.296446+0000 2021-07-31T03:27:34.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: mgr.x (mgr.24542) 4298 : cephadm [ERR] 2021-07-31T03:27:34.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:27:34.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:27:34.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: yield 2021-07-31T03:27:34.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:27:34.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:27:34.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:27:34.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:27:34.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:27:34.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:27:34.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:27:34.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:27:34.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:27:34.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: task.result() 2021-07-31T03:27:34.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:27:34.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: cephadm 2021-07-31T03:27:33.296885+0000 mgr.x (mgr.24542) 4299 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:27:34.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: cephadm 2021-07-31T03:27:33.307717+0000 mgr.x (mgr.24542) 4300 : cephadm [ERR] 2021-07-31T03:27:34.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:27:34.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:27:34.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: yield 2021-07-31T03:27:34.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:27:34.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:27:34.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:27:34.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:27:34.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:27:34.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:27:34.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:27:34.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:27:34.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:27:34.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: task.result() 2021-07-31T03:27:34.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:27:34.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: cephadm 2021-07-31T03:27:33.308572+0000 mgr.x (mgr.24542) 4301 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:27:34.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: cephadm 2021-07-31T03:27:33.332787+0000 mgr.x (mgr.24542) 4302 : cephadm [ERR] 2021-07-31T03:27:34.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:27:34.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:27:34.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: yield 2021-07-31T03:27:34.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:27:34.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:27:34.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:27:34.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:27:34.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:27:34.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:27:34.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:27:34.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:27:34.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:27:34.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: task.result() 2021-07-31T03:27:34.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:27:34.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: cephadm 2021-07-31T03: 2021-07-31T03:27:34.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: 27:33.333237+0000 mgr.x (mgr.24542) 4303 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:27:34.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:27:34.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:27:34.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: yield 2021-07-31T03:27:34.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:27:34.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:27:34.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:27:34.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:27:34.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:27:34.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:27:34.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:27:34.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:27:34.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:27:34.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: task.result() 2021-07-31T03:27:34.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:27:34.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: 2021-07-31T03:27:34.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:27:34.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: 2021-07-31T03:27:34.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:27:34.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:27:34.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:27:34.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:27:34.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:27:34.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:27:34.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:27:34.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:27:34.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:27:34.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:27:34.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:27:34.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:27:34.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:27:34.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:27:34.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:27:34.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:27:34.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:27:34.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:27:34.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: raise 2021-07-31T03:27:34.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:27:34.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:27:34.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:27:34.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:27:34.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:27:34.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: cephadm 2021-07-31T03:27:33.341607+0000 mgr.x (mgr.24542) 4304 : cephadm [ERR] 2021-07-31T03:27:34.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:27:34.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:27:34.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: yield 2021-07-31T03:27:34.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:27:34.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:27:34.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:27:34.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:27:34.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:27:34.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:27:34.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:27:34.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:27:34.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:27:34.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: task.result() 2021-07-31T03:27:34.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:27:34.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: cluster 2021-07-31T03:27:33.230173+0000 mgr.x (mgr.24542) 4296 : cluster 2021-07-31T03:27:34.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: [DBG] pgmap v5017: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:34.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: audit 2021-07-31T03:27:33.250482+0000 mon.b (mon.2) 577 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:27:34.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: audit 2021-07-31T03:27:33.251816+0000 mon.b (mon.2) 2021-07-31T03:27:34.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: 578 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:27:34.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: audit 2021 2021-07-31T03:27:34.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: -07-31T03:27:33.252741+0000 mon.b ( 2021-07-31T03:27:34.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: mon.2) 579 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:27:34.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: cephadm 2021-07-31T03:27:34.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: 2021-07-31T03:27:33.253525+0000 mgr.x (mgr.24542) 4297 : 2021-07-31T03:27:34.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:27:34.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: cephadm 2021-07-31T03:27:34.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: 2021-07 2021-07-31T03:27:34.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: -31T03:27:33.296446+0000 mgr.x (mgr 2021-07-31T03:27:34.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: .24542) 4298 : cephadm [ERR] 2021-07-31T03:27:34.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:27:34.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:27:34.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: yield 2021-07-31T03:27:34.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:27:34.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:27:34.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:27:34.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:27:34.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:27:34.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:27:34.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:27:34.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:27:34.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:27:34.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: task.result() 2021-07-31T03:27:34.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:27:34.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: cephadm 2021-07-31T03:27:33.296885+0000 mgr.x (mgr.24542) 4299 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:27:34.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: cephadm 2021-07- 2021-07-31T03:27:34.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: 31T03:27:33.307717+0000 mgr.x (mgr. 2021-07-31T03:27:34.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: 24542) 4300 : cephadm [ERR] 2021-07-31T03:27:34.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:27:34.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:27:34.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: yield 2021-07-31T03:27:34.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:27:34.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:27:34.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: cephadm 2021-07-31T03:27:33.342168+0000 mgr.x (mgr.24542) 4305 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:27:34.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:27:34.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:27:34.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: yield 2021-07-31T03:27:34.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:27:34.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:27:34.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:27:34.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:27:34.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:27:34.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:27:34.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:27:34.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:27:34.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:27:34.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: task.result() 2021-07-31T03:27:34.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:27:34.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: 2021-07-31T03:27:34.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:27:34.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: 2021-07-31T03:27:34.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:27:34.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:27:34.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:27:34.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:27:34.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:27:34.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:27:34.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:27:34.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:27:34.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:27:34.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:27:34.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:27:34.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:27:34.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:27:34.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:27:34.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:27:34.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:27:34.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:27:34.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:27:34.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: raise 2021-07-31T03:27:34.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:27:34.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:27:34.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:27:34.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:27:34.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:27:34.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[39718]: 2021-07-31T03:27:34.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:27:34.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:27:34.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:27:34.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:27:34.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:27:34.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:27:34.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:27:34.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: task.result() 2021-07-31T03:27:34.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:27:34.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: cephadm 2021-07-31T03:27:33.308572+0000 mgr.x (mgr.24542) 4301 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:27:34.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: cephadm 2021-07-31T03 2021-07-31T03:27:34.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: :27:33.332787+0000 mgr.x (mgr.24542) 4302 : cephadm [ERR] 2021-07-31T03:27:34.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:27:34.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:27:34.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: yield 2021-07-31T03:27:34.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:27:34.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:27:34.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:27:34.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:27:34.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:27:34.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:27:34.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:27:34.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:27:34.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:27:34.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: task.result() 2021-07-31T03:27:34.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:27:34.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: 2021-07-31T03:27:34.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: cephadm 2021-07-31T03:27:33.333237+0000 mgr.x (mgr.24542) 4303 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:27:34.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:27:34.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:27:34.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: yield 2021-07-31T03:27:34.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:27:34.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:27:34.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:27:34.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:27:34.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:27:34.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:27:34.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:27:34.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:27:34.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:27:34.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: task.result() 2021-07-31T03:27:34.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:27:34.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: 2021-07-31T03:27:34.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:27:34.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: 2021-07-31T03:27:34.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:27:34.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:27:34.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:27:34.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:27:34.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:27:34.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:27:34.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:27:34.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:27:34.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:27:34.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:27:34.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:27:34.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:27:34.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:27:34.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:27:34.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:27:34.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:27:34.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:27:34.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:27:34.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: raise 2021-07-31T03:27:34.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:27:34.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:27:34.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:27:34.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:27:34.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:27:34.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: cephadm 2021-07-31T03:27:34.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: 2021-07-31T03:27:33.341607+0000 mgr.x (mgr.24542) 4304 : cephadm [ERR] 2021-07-31T03:27:34.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:27:34.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:27:34.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: yield 2021-07-31T03:27:34.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:27:34.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:27:34.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:27:34.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:27:34.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:27:34.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:27:34.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:27:34.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:27:34.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:27:34.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: task.result() 2021-07-31T03:27:34.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:27:34.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: cephadm 2021-07-31T03:27:33.342168+0000 mgr.x (mgr.24542) 4305 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:27:34.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:27:34.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:27:34.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: yield 2021-07-31T03:27:34.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:27:34.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:27:34.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:27:34.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:27:34.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:27:34.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:27:34.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:27:34.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:27:34.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:27:34.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: task.result() 2021-07-31T03:27:34.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:27:34.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: 2021-07-31T03:27:34.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:27:34.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: 2021-07-31T03:27:34.608 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:27:34.608 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:27:34.608 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:27:34.608 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:27:34.608 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:27:34.609 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:27:34.609 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:27:34.609 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:27:34.609 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:27:34.610 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:27:34.610 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:27:34.610 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:27:34.610 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:27:34.610 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:27:34.611 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:27:34.611 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:27:34.611 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:27:34.611 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:27:34.611 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: raise 2021-07-31T03:27:34.612 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:27:34.612 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:27:34.612 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:27:34.612 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:27:34.612 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:34 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:27:36.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:36 smithi154 conmon[34553]: cluster 2021-07-31T03:27:35.230776+0000 mgr.x (mgr.24542) 4306 : cluster [DBG] 2021-07-31T03:27:36.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:36 smithi154 conmon[34553]: pgmap v5018: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:36.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:36 smithi045 conmon[39718]: cluster 2021-07-31T03:27:35.230776+0000 mgr.x 2021-07-31T03:27:36.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:36 smithi045 conmon[39718]: (mgr.24542) 4306 : cluster [DBG] pgmap v5018: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:36.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:36 smithi045 conmon[33655]: cluster 2021-07-31T03:27:35.230776+0000 mgr.x (mgr.24542 2021-07-31T03:27:36.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:36 smithi045 conmon[33655]: ) 4306 : cluster [DBG] pgmap v5018: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:38.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:38 smithi154 conmon[34553]: cluster 2021-07-31T03:27:37.231198+0000 mgr.x (mgr.24542) 4307 : cluster [DBG] pgmap v5019: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:38.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:38 smithi154 conmon[34553]: 2021-07-31T03:27:38.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:38 smithi045 conmon[39718]: cluster 2021-07-31T03:27:37.231198+0000 mgr.x (mgr.24542) 4307 : cluster [DBG] pgmap v5019: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:38.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:38 smithi045 conmon[33655]: cluster 2021-07-31T03:27:37.231198+0000 mgr.x (mgr.24542) 4307 : cluster [DBG] pgmap v5019: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:40.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:40 smithi045 conmon[39718]: cluster 2021-07-31T03:27:39.231587+0000 2021-07-31T03:27:40.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:40 smithi045 conmon[39718]: mgr.x (mgr.24542) 4308 : cluster [DBG] pgmap v5020: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:40.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:40 smithi045 conmon[33655]: cluster 2021-07-31T03:27:39.231587+0000 mgr.x ( 2021-07-31T03:27:40.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:40 smithi045 conmon[33655]: mgr.24542) 4308 : cluster [DBG] pgmap v5020: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:40.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:40 smithi154 conmon[34553]: cluster 2021-07-31T03:27:39.231587+0000 mgr.x (mgr.24542) 4308 : cluster [DBG] pgmap v5020: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:42.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:42 smithi045 conmon[39718]: cluster 2021-07-31T03:27:41.232052+0000 mgr.x 2021-07-31T03:27:42.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:42 smithi045 conmon[39718]: (mgr.24542) 4309 : cluster [DBG] pgmap v5021: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:42.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:42 smithi045 conmon[33655]: cluster 2021-07-31T03:27:41.232052+0000 mgr.x (mgr.24542) 4309 2021-07-31T03:27:42.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:42 smithi045 conmon[33655]: : cluster [DBG] pgmap v5021: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:42.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:42 smithi154 conmon[34553]: cluster 2021-07-31T03:27:41.232052+0000 mgr.x (mgr.24542) 4309 : cluster [DBG] pgmap v5021: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:44.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:44 smithi045 conmon[39718]: cluster 2021-07-31T03:27:43.232532+0000 mgr.x (mgr 2021-07-31T03:27:44.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:44 smithi045 conmon[39718]: .24542) 4310 : cluster [DBG] pgmap v5022: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:44.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:44 smithi045 conmon[33655]: cluster 2021-07-31T03:27:43.232532+0000 mgr.x (mgr.24542 2021-07-31T03:27:44.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:44 smithi045 conmon[33655]: ) 4310 : cluster [DBG] pgmap v5022: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:44.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:44 smithi154 conmon[34553]: cluster 2021-07-31T03:27:43.232532+0000 mgr.x (mgr.24542) 4310 : cluster [DBG] pgmap v5022: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:46.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:46 smithi045 conmon[39718]: cluster 2021-07-31T03:27:45.232939+0000 mgr.x (mgr 2021-07-31T03:27:46.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:46 smithi045 conmon[39718]: .24542) 4311 : cluster [DBG] pgmap v5023: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:46.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:46 smithi045 conmon[33655]: cluster 2021-07-31T03:27:45.232939+0000 mgr.x (mgr.24542) 4311 : cluster [DBG] pgmap v5023: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:46.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:46 smithi154 conmon[34553]: cluster 2021-07-31T03:27:45.232939+0000 mgr.x (mgr.24542) 4311 : cluster [DBG] pgmap v5023: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:48.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:48 smithi045 conmon[39718]: cluster 2021-07-31T03:27:47.233463+0000 mgr.x (mgr.24542) 4312 : cluster [DBG] pgmap v5024: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:48.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:48 smithi045 conmon[33655]: cluster 2021-07-31T03:27:47.233463+0000 mgr.x (mgr 2021-07-31T03:27:48.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:48 smithi045 conmon[33655]: .24542) 4312 : cluster [DBG] pgmap v5024: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:48.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:48 smithi154 conmon[34553]: cluster 2021-07-31T03:27:47.233463+0000 mgr.x (mgr.24542) 4312 : cluster [DBG] pgmap v5024: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:50.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:50 smithi045 conmon[33655]: cluster 2021-07-31T03:27:49.234009+0000 mgr.x ( 2021-07-31T03:27:50.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:50 smithi045 conmon[33655]: mgr.24542) 4313 : cluster [DBG] pgmap v5025: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:50.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:50 smithi045 conmon[39718]: cluster 2021-07-31T03:27:49.234009+0000 mgr.x (mgr.24542) 4313 : cluster [DBG] pgmap v5025: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:50.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:50 smithi154 conmon[34553]: cluster 2021-07-31T03:27:49.234009+0000 mgr.x (mgr.24542) 4313 : cluster [DBG] pgmap v5025: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:52.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:52 smithi154 conmon[34553]: cluster 2021-07-31T03:27:51.234429+0000 mgr.x (mgr.24542) 4314 : cluster [DBG] pgmap v5026: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:52.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:52 smithi045 conmon[39718]: cluster 2021-07-31T03:27:51.234429+0000 mgr.x (mgr.24542) 4314 : cluster [DBG] pgmap v5026: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:52.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:52 smithi045 conmon[33655]: cluster 2021-07-31T03:27:51.234429+0000 mgr.x ( 2021-07-31T03:27:52.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:52 smithi045 conmon[33655]: mgr.24542) 4314 : cluster [DBG] pgmap v5026: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:54.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:54 smithi154 conmon[34553]: cluster 2021-07-31T03:27:53.234806+0000 mgr.x (mgr.24542) 4315 : cluster [DBG] pgmap v5027: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:54.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:54 smithi154 conmon[34553]: 2021-07-31T03:27:54.774 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:54 smithi045 conmon[39718]: cluster 2021-07-31T03:27:53.234806+0000 mgr.x (mgr.24542) 4315 : 2021-07-31T03:27:54.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:54 smithi045 conmon[39718]: cluster [DBG] pgmap v5027: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:54.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:54 smithi045 conmon[33655]: cluster 2021-07-31T03:27:53.234806+0000 mgr.x (mgr.24542) 4315 : cluster 2021-07-31T03:27:54.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:54 smithi045 conmon[33655]: [DBG] pgmap v5027: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:56.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:56 smithi154 conmon[34553]: cluster 2021-07-31T03:27:55.235159+0000 mgr.x (mgr.24542) 4316 : cluster [DBG] pgmap v5028: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:56.774 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:56 smithi045 conmon[33655]: cluster 2021-07-31T03:27:55.235159+0000 mgr.x (mgr.24542) 4316 : cluster [DBG] pgmap v5028: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:56.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:56 smithi045 conmon[39718]: cluster 2021-07-31T03:27:55.235159+0000 mgr.x (mgr.24542 2021-07-31T03:27:56.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:56 smithi045 conmon[39718]: ) 4316 : cluster [DBG] pgmap v5028: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:58.556 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:27:58.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:58 smithi154 conmon[34553]: cluster 2021-07-31T03:27:57.235588+0000 mgr.x (mgr.24542) 4317 : cluster [DBG] pgmap v5029: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:58.774 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:58 smithi045 conmon[39718]: cluster 2021-07-31T03:27 2021-07-31T03:27:58.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:58 smithi045 conmon[39718]: :57.235588+0000 mgr.x (mgr.24542) 4317 : cluster [DBG] pgmap v5029: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:58.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:58 smithi045 conmon[33655]: cluster 2021-07-31T03:27:57.235588 2021-07-31T03:27:58.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:58 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 4317 : cluster [DBG] pgmap v5029: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:27:58.832 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:27:58.832 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 102m ago 103m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:27:58.832 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 100m ago 103m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:27:58.833 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 100m ago 106m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:27:58.833 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 102m ago 108m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:27:58.833 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 102m ago 109m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:27:58.833 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 100m ago 107m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:27:58.833 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 102m ago 107m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:27:58.834 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 102m ago 103m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:27:58.834 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 100m ago 103m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:27:58.834 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 102m ago 106m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:27:58.834 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 102m ago 106m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:27:58.834 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 102m ago 105m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:27:58.835 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 102m ago 105m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:27:58.835 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 100m ago 105m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:27:58.835 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 100m ago 104m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:27:58.835 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 100m ago 104m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:27:58.835 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 100m ago 104m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:27:58.836 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 100m ago 103m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:27:59.167 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:27:59.168 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:27:59.168 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:27:59.168 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:27:59.168 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:27:59.168 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:27:59.169 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:27:59.169 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:27:59.169 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:27:59.169 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:27:59.169 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:27:59.170 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:27:59.170 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:27:59.170 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:27:59.170 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:27:59.170 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:27:59.171 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:27:59.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:59 smithi154 conmon[34553]: audit 2021-07-31T03:27:58.546084+0000 mgr.x (mgr.24542) 4318 : audit [DBG] 2021-07-31T03:27:59.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:59 smithi154 conmon[34553]: from='client.29500 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:27:59.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:59 smithi154 conmon[34553]: audit 2021-07-31T03:27:58.825486+0000 mgr.x (mgr.24542) 4319 : audit [DBG] from='client.29506 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:27:59.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:59 smithi154 conmon[34553]: audit 2021-07-31T03:27:59.168603+0000 mon.c (mon.1) 207 : audit 2021-07-31T03:27:59.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:27:59 smithi154 conmon[34553]: [DBG] from='client.? 172.21.15.45:0/1125823246' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:27:59.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:59 smithi045 conmon[33655]: audit 2021-07-31T03:27:58.546084+0000 mgr.x (mgr.24542) 4318 : audit [DBG] from='client.29500 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:27:59.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:59 smithi045 conmon[33655]: audit 2021-07-31T03:27:58.825486+0000 mgr.x (mgr.24542) 4319 : audit [DBG] from='client.29506 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:27:59.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:27:59 smithi045 conmon[33655]: audit 2021-07-31T03:27:59.168603+0000 mon.c (mon.1) 207 : audit [DBG] from='client.? 172.21.15.45:0/1125823246' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:27:59.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:59 smithi045 conmon[39718]: audit 2021-07-31T03:27:58.546084+0000 2021-07-31T03:27:59.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:59 smithi045 conmon[39718]: mgr.x (mgr.24542) 4318 : audit [DBG] from='client.29500 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:27:59.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:59 smithi045 conmon[39718]: audit 2021-07-31T03: 2021-07-31T03:27:59.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:59 smithi045 conmon[39718]: 27:58.825486+0000 mgr.x (mgr.24542) 4319 : audit [DBG] from='client.29506 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:27:59.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:59 smithi045 conmon[39718]: audit 2021-07-31T03:27:59.168603+0000 mon.c ( 2021-07-31T03:27:59.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:27:59 smithi045 conmon[39718]: mon.1) 207 : audit [DBG] from='client.? 172.21.15.45:0/1125823246' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:28:00.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:00 smithi154 conmon[34553]: cluster 2021-07-31T03:27:59.236144+0000 mgr.x (mgr.24542) 4320 : cluster [DBG] pgmap v5030: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:28:00.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:00 smithi045 conmon[33655]: cluster 2021-07-31T03:27:59.236144+0000 mgr.x (mgr.24542) 2021-07-31T03:28:00.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:00 smithi045 conmon[33655]: 4320 : cluster [DBG] pgmap v5030: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:28:00.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:00 smithi045 conmon[39718]: cluster 2021-07-31T03:27:59.236144+0000 mgr.x ( 2021-07-31T03:28:00.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:00 smithi045 conmon[39718]: mgr.24542) 4320 : cluster [DBG] pgmap v5030: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:28:02.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:02 smithi154 conmon[34553]: cluster 2021-07-31T03:28:01.236539+0000 mgr.x (mgr.24542) 4321 : cluster [DBG] pgmap v5031: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:28:02.774 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:02 smithi045 conmon[39718]: cluster 2021-07-31T03:28:01.236539+0000 mgr.x (mgr. 2021-07-31T03:28:02.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:02 smithi045 conmon[39718]: 24542) 4321 : cluster [DBG] pgmap v5031: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:28:02.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:02 smithi045 conmon[33655]: cluster 2021-07-31T03:28:01.236539+0000 mgr.x (mgr.24542) 4321 2021-07-31T03:28:02.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:02 smithi045 conmon[33655]: : cluster [DBG] pgmap v5031: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:28:04.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:04 smithi154 conmon[34553]: cluster 2021-07-31T03:28:03.236964+0000 mgr.x (mgr.24542) 4322 : cluster [DBG] pgmap v5032: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:28:04.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:04 smithi045 conmon[39718]: cluster 2021-07-31T03:28:03.236964+0000 mgr.x ( 2021-07-31T03:28:04.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:04 smithi045 conmon[39718]: mgr.24542) 4322 : cluster [DBG] pgmap v5032: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:28:04.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:04 smithi045 conmon[33655]: cluster 2021-07-31T03:28:03.236964+0000 mgr.x (mgr.24542) 2021-07-31T03:28:04.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:04 smithi045 conmon[33655]: 4322 : cluster [DBG] pgmap v5032: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:28:06.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:06 smithi154 conmon[34553]: cluster 2021-07-31T03:28:05.237369+0000 mgr.x (mgr.24542) 4323 : cluster [DBG] pgmap v5033: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:28:06.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:06 smithi045 conmon[39718]: cluster 2021-07-31T03:28:05.237369+0000 2021-07-31T03:28:06.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:06 smithi045 conmon[39718]: mgr.x (mgr.24542) 4323 : cluster [DBG] pgmap v5033: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:28:06.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:06 smithi045 conmon[33655]: cluster 2021-07-31T03:28:05.237369+0000 mgr.x (mgr. 2021-07-31T03:28:06.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:06 smithi045 conmon[33655]: 24542) 4323 : cluster [DBG] pgmap v5033: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:28:08.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:08 smithi154 conmon[34553]: cluster 2021-07-31T03:28:07.237745+0000 mgr.x (mgr.24542) 4324 : cluster [DBG] pgmap v5034: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:28:08.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:08 smithi045 conmon[39718]: cluster 2021-07-31T03:28:07.237745 2021-07-31T03:28:08.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:08 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 4324 : cluster [DBG] pgmap v5034: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:28:08.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:08 smithi045 conmon[33655]: cluster 2021-07-31T03:28:07.237745+0000 mgr.x (mgr.24542) 4324 2021-07-31T03:28:08.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:08 smithi045 conmon[33655]: : cluster [DBG] pgmap v5034: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:28:10.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:10 smithi154 conmon[34553]: cluster 2021-07-31T03:28:09.238325+0000 mgr.x (mgr.24542) 4325 : cluster [DBG] pgmap v5035: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:28:10.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:10 smithi045 conmon[39718]: cluster 2021-07-31T03:28:09.238325+0000 mgr.x (mgr.24542 2021-07-31T03:28:10.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:10 smithi045 conmon[39718]: ) 4325 : cluster [DBG] pgmap v5035: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:28:10.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:10 smithi045 conmon[33655]: cluster 2021-07-31T03:28:09.238325+0000 mgr.x (mgr.24542 2021-07-31T03:28:10.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:10 smithi045 conmon[33655]: ) 4325 : cluster [DBG] pgmap v5035: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:28:12.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:12 smithi154 conmon[34553]: cluster 2021-07-31T03:28:11.238818+0000 mgr.x (mgr.24542) 4326 : cluster [DBG] pgmap v5036: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:28:12.774 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:12 smithi045 conmon[39718]: cluster 2021-07-31T03:28:11.238818+0000 mgr.x (mgr.24542 2021-07-31T03:28:12.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:12 smithi045 conmon[39718]: ) 4326 : cluster [DBG] pgmap v5036: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:28:12.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:12 smithi045 conmon[33655]: cluster 2021-07-31T03:28:11.238818+0000 mgr.x (mgr.24542) 4326 2021-07-31T03:28:12.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:12 smithi045 conmon[33655]: : cluster [DBG] pgmap v5036: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:28:14.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:14 smithi154 conmon[34553]: cluster 2021-07-31T03:28:13.239375+0000 mgr.x (mgr.24542) 4327 : cluster [DBG] pgmap v5037: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:28:14.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:14 smithi045 conmon[39718]: cluster 2021-07-31T03:28:13.239375+0000 mgr.x (mgr. 2021-07-31T03:28:14.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:14 smithi045 conmon[39718]: 24542) 4327 : cluster [DBG] pgmap v5037: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:28:14.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:14 smithi045 conmon[33655]: cluster 2021-07-31T03:28:13.239375+0000 mgr.x (mgr.24542) 2021-07-31T03:28:14.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:14 smithi045 conmon[33655]: 4327 : cluster [DBG] pgmap v5037: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:28:16.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:16 smithi154 conmon[34553]: cluster 2021-07-31T03:28:15.239886+0000 mgr.x (mgr.24542) 4328 : cluster [DBG] pgmap v5038: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:28:16.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:16 smithi045 conmon[39718]: cluster 2021-07-31T03:28:15.239886+0000 mgr.x (mgr.24542 2021-07-31T03:28:16.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:16 smithi045 conmon[39718]: ) 4328 : cluster [DBG] pgmap v5038: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:28:16.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:16 smithi045 conmon[33655]: cluster 2021-07-31T03:28:15.239886+0000 mgr.x (mgr.24542) 4328 : 2021-07-31T03:28:16.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:16 smithi045 conmon[33655]: cluster [DBG] pgmap v5038: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:28:18.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:18 smithi154 conmon[34553]: cluster 2021-07-31T03:28:17.240376+0000 mgr.x (mgr.24542) 4329 : cluster [DBG] pgmap v5039: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:28:18.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:18 smithi045 conmon[39718]: cluster 2021-07-31T03:28:17.240376+0000 mgr.x (mgr.24542) 2021-07-31T03:28:18.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:18 smithi045 conmon[39718]: 4329 : cluster [DBG] pgmap v5039: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:28:18.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:18 smithi045 conmon[33655]: cluster 2021-07-31T03:28:17.240376+0000 mgr.x (mgr.24542) 4329 : cluster 2021-07-31T03:28:18.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:18 smithi045 conmon[33655]: [DBG] pgmap v5039: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:28:20.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:20 smithi154 conmon[34553]: cluster 2021-07-31T03:28:19.240740+0000 mgr.x (mgr.24542) 4330 : cluster [DBG] pgmap v5040: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:28:20.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:20 smithi045 conmon[39718]: cluster 2021-07-31T03:28:19.240740+0000 mgr.x (mgr.24542) 4330 : cluster [DBG] pgmap v5040: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:28:20.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:20 smithi045 conmon[33655]: cluster 2021-07-31T03:28:19.240740+0000 mgr.x (mgr. 2021-07-31T03:28:20.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:20 smithi045 conmon[33655]: 24542) 4330 : cluster [DBG] pgmap v5040: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:28:22.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:22 smithi154 conmon[34553]: cluster 2021-07-31T03:28:21.241156+0000 mgr.x (mgr.24542) 4331 : cluster [DBG] pgmap v5041: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:28:22.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:22 smithi045 conmon[33655]: cluster 2021-07-31T03:28:21.241156+0000 mgr.x (mgr.24542) 4331 : cluster [DBG] pgmap v5041: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:28:22.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:22 smithi045 conmon[33655]: 2021-07-31T03:28:22.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:22 smithi045 conmon[39718]: cluster 2021-07-31T03:28:21.241156+0000 mgr.x (mgr 2021-07-31T03:28:22.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:22 smithi045 conmon[39718]: .24542) 4331 : cluster [DBG] pgmap v5041: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:28:24.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:24 smithi154 conmon[34553]: cluster 2021-07-31T03:28:23.241471+0000 mgr.x (mgr.24542) 4332 : cluster [DBG] pgmap v5042: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:28:24.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:24 smithi045 conmon[33655]: cluster 2021-07-31T03:28:23.241471+0000 mgr.x (mgr.24542) 4332 : 2021-07-31T03:28:24.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:24 smithi045 conmon[33655]: cluster [DBG] pgmap v5042: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:28:24.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:24 smithi045 conmon[39718]: cluster 2021-07-31T03:28:23.241471+0000 mgr.x ( 2021-07-31T03:28:24.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:24 smithi045 conmon[39718]: mgr.24542) 4332 : cluster [DBG] pgmap v5042: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:28:26.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:26 smithi154 conmon[34553]: cluster 2021-07-31T03:28:25.241910+0000 mgr.x (mgr.24542) 4333 : cluster [DBG] pgmap v5043: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:28:26.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:26 smithi045 conmon[39718]: cluster 2021-07-31T03:28:25.241910+0000 mgr.x (mgr.24542 2021-07-31T03:28:26.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:26 smithi045 conmon[39718]: ) 4333 : cluster [DBG] pgmap v5043: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:28:26.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:26 smithi045 conmon[33655]: cluster 2021-07-31T03:28:25.241910+0000 mgr.x (mgr. 2021-07-31T03:28:26.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:26 smithi045 conmon[33655]: 24542) 4333 : cluster [DBG] pgmap v5043: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:28:28.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:28 smithi154 conmon[34553]: cluster 2021-07-31T03:28:27.242285+0000 mgr.x (mgr.24542) 4334 : cluster [DBG] pgmap v5044: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:28:28.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:28 smithi154 conmon[34553]: audit 2021-07-31T03:28:28.336876+0000 mon.b (mon.2) 580 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:28:28.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:28 smithi154 conmon[34553]: audit 2021-07-31T03:28:28.337156+0000 mon.b (mon.2) 581 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:28:28.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:28 smithi154 conmon[34553]: audit 2021-07-31T03:28:28.337493+0000 mon.a (mon.0) 2021-07-31T03:28:28.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:28 smithi154 conmon[34553]: 3308 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:28:28.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:28 smithi154 conmon[34553]: audit 2021-07-31T03:28:28.337930+0000 mon.a ( 2021-07-31T03:28:28.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:28 smithi154 conmon[34553]: mon.0) 3309 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:28:28.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:28 smithi045 conmon[39718]: cluster 2021-07-31T03:28:27.242285+0000 2021-07-31T03:28:28.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:28 smithi045 conmon[39718]: mgr.x (mgr.24542) 4334 : cluster [DBG] pgmap v5044: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:28:28.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:28 smithi045 conmon[39718]: audit 2021-07-31T03 2021-07-31T03:28:28.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:28 smithi045 conmon[39718]: :28:28.336876+0000 mon.b (mon.2) 580 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:28:28.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:28 smithi045 conmon[39718]: audit 2021-07-31T03:28:28.337156 2021-07-31T03:28:28.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:28 smithi045 conmon[39718]: +0000 mon.b (mon.2) 581 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:28:28.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:28 smithi045 conmon[39718]: audit 2021-07-31T03 2021-07-31T03:28:28.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:28 smithi045 conmon[39718]: :28:28.337493+0000 mon.a (mon.0) 3308 : audit [INF] 2021-07-31T03:28:28.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:28 smithi045 conmon[39718]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:28:28.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:28 smithi045 conmon[39718]: audit 2021-07 2021-07-31T03:28:28.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:28 smithi045 conmon[39718]: -31T03:28:28.337930+0000 mon.a (mon.0) 3309 : audit 2021-07-31T03:28:28.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:28 smithi045 conmon[39718]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:28:28.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:28 smithi045 conmon[33655]: cluster 2021-07-31T03:28:27.242285+0000 2021-07-31T03:28:28.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:28 smithi045 conmon[33655]: mgr.x (mgr.24542) 4334 : cluster [DBG] pgmap v5044: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:28:28.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:28 smithi045 conmon[33655]: audit 2021-07-31T03:28:28.336876+0000 mon.b (mon.2) 580 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:28:28.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:28 smithi045 conmon[33655]: audit 2021-07-31T 2021-07-31T03:28:28.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:28 smithi045 conmon[33655]: 03:28:28.337156+0000 mon.b (mon.2) 581 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:28:28.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:28 smithi045 conmon[33655]: audit 2021-07-31T03:28:28.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:28 smithi045 conmon[33655]: 2021-07-31T03:28:28.337493+0000 mon.a (mon 2021-07-31T03:28:28.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:28 smithi045 conmon[33655]: .0) 3308 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:28:28.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:28 smithi045 conmon[33655]: audit 2021-07-31T03:28: 2021-07-31T03:28:28.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:28 smithi045 conmon[33655]: 28.337930+0000 mon.a (mon.0) 3309 : audit 2021-07-31T03:28:28.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:28 smithi045 conmon[33655]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:28:29.481 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:28:29.762 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:28:29.762 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 103m ago 104m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:28:29.762 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 101m ago 103m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:28:29.762 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 101m ago 107m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:28:29.763 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 103m ago 109m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:28:29.763 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 103m ago 109m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:28:29.763 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 101m ago 107m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:28:29.763 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 103m ago 107m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:28:29.763 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 103m ago 104m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:28:29.764 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 101m ago 104m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:28:29.764 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 103m ago 107m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:28:29.764 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 103m ago 106m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:28:29.764 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 103m ago 106m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:28:29.765 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 103m ago 106m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:28:29.765 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 101m ago 105m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:28:29.765 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 101m ago 105m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:28:29.765 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 101m ago 105m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:28:29.765 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 101m ago 104m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:28:29.766 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 101m ago 104m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:28:30.095 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:28:30.096 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:28:30.096 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:28:30.096 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:28:30.096 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:28:30.097 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:28:30.097 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:28:30.097 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:28:30.097 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:28:30.098 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:28:30.098 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:28:30.098 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:28:30.098 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:28:30.098 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:28:30.099 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:28:30.099 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:28:30.099 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:28:30.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:30 smithi154 conmon[34553]: cluster 2021-07-31T03:28:29.242714+0000 mgr.x (mgr.24542) 4335 : cluster [DBG] pgmap v5045: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:28:30.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:30 smithi154 conmon[34553]: audit 2021-07-31T03:28:29.470647 2021-07-31T03:28:30.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:30 smithi154 conmon[34553]: +0000 mgr.x (mgr.24542) 4336 : audit [DBG] from='client.29518 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:28:30.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:30 smithi154 conmon[34553]: audit 2021-07-31T03:28:29.754893+0000 mgr.x (mgr.24542) 4337 : audit [DBG] from='client.29524 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:28:30.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:30 smithi154 conmon[34553]: audit 2021-07-31T03:28:30.096767+0000 mon.c (mon.1) 208 : audit [DBG] 2021-07-31T03:28:30.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:30 smithi154 conmon[34553]: from='client.? 172.21.15.45:0/1359995489' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:28:30.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:30 smithi045 conmon[39718]: cluster 2021-07-31T03:28:29.242714 2021-07-31T03:28:30.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:30 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 4335 : cluster [DBG] pgmap v5045: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:28:30.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:30 smithi045 conmon[39718]: audit 2021-07-31T03:28:30.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:30 smithi045 conmon[39718]: 2021-07-31T03:28: 2021-07-31T03:28:30.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:30 smithi045 conmon[39718]: 29.470647+0000 mgr.x (mgr.24542 2021-07-31T03:28:30.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:30 smithi045 conmon[39718]: ) 4336 : audit [DBG] from='client.29518 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:28:30.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:30 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T03:28:30.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:30 smithi045 conmon[39718]: 03:28:29.754893+0000 mgr.x (mgr 2021-07-31T03:28:30.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:30 smithi045 conmon[39718]: .24542) 4337 : audit [DBG] from='client.29524 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:28:30.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:30 smithi045 conmon[39718]: audit 2021- 2021-07-31T03:28:30.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:30 smithi045 conmon[39718]: 07-31T03:28:30.096767 2021-07-31T03:28:30.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:30 smithi045 conmon[39718]: +0000 mon.c (mon.1) 208 : audit 2021-07-31T03:28:30.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:30 smithi045 conmon[39718]: [DBG] from='client.? 172.21.15.45:0/1359995489' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:28:30.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:30 smithi045 conmon[33655]: cluster 2021-07-31T03:28:29.242714+0000 mgr.x (mgr.24542) 4335 : cluster 2021-07-31T03:28:30.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:30 smithi045 conmon[33655]: [DBG] pgmap v5045: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:28:30.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:30 smithi045 conmon[33655]: audit 2021-07-31T03:28:29.470647+0000 mgr.x (mgr.24542) 4336 : audit [DBG] from='client.29518 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:28:30.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:30 smithi045 conmon[33655]: audit 2021-07-31 2021-07-31T03:28:30.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:30 smithi045 conmon[33655]: T03:28:29.754893+0000 mgr.x (mgr.24542) 4337 : audit [DBG] from='client.29524 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:28:30.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:30 smithi045 conmon[33655]: audit 2021-07-31T03:28:30. 2021-07-31T03:28:30.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:30 smithi045 conmon[33655]: 096767+0000 mon.c (mon.1) 208 : audit [DBG] 2021-07-31T03:28:30.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:30 smithi045 conmon[33655]: from='client.? 172.21.15.45:0/1359995489' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:28:32.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:32 smithi154 conmon[34553]: cluster 2021-07-31T03:28:31.243232+0000 mgr.x (mgr.24542) 4338 : cluster [DBG] pgmap v5046: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:28:32.774 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:32 smithi045 conmon[39718]: cluster 2021-07-31T03:28:31.243232+0000 mgr.x 2021-07-31T03:28:32.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:32 smithi045 conmon[39718]: (mgr.24542) 4338 : cluster [DBG] pgmap v5046: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:28:32.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:32 smithi045 conmon[33655]: cluster 2021-07-31T03:28:31.243232+0000 mgr.x (mgr 2021-07-31T03:28:32.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:32 smithi045 conmon[33655]: .24542) 4338 : cluster [DBG] pgmap v5046: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:28:33.752 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: debug 2021-07-31T03:28:33.390+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:28:33.753 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:28:33.753 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:28:33.753 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: yield 2021-07-31T03:28:33.753 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:28:33.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:28:33.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:28:33.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:28:33.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:28:33.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:28:33.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:28:33.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:28:33.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:28:33.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: task.result() 2021-07-31T03:28:33.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:28:33.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: debug 2021-07-31T03:28:33.402+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:28:33.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:28:33.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:28:33.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: yield 2021-07-31T03:28:33.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:28:33.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:28:33.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:28:33.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:28:33.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:28:33.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:28:33.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:28:33.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:28:33.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:28:33.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: task.result() 2021-07-31T03:28:33.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:28:33.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: debug 2021-07-31T03:28:33.428+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:28:33.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:28:33.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:28:33.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: yield 2021-07-31T03:28:33.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:28:33.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:28:33.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:28:33.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:28:33.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:28:33.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:28:33.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:28:33.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:28:33.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:28:33.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: task.result() 2021-07-31T03:28:33.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:28:33.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: debug 2021-07-31T03:28:33.428+0000 7fcb9295f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:28:33.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:28:33.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:28:33.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: yield 2021-07-31T03:28:33.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:28:33.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:28:33.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:28:33.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:28:33.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:28:33.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:28:33.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:28:33.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:28:33.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:28:33.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: task.result() 2021-07-31T03:28:33.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:28:33.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: 2021-07-31T03:28:33.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:28:33.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: 2021-07-31T03:28:33.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:28:33.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:28:33.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:28:33.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:28:33.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:28:33.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:28:33.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:28:33.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:28:33.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:28:33.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:28:33.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:28:33.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:28:33.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:28:33.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:28:33.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:28:33.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:28:33.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:28:33.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:28:33.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: raise 2021-07-31T03:28:33.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:28:33.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:28:33.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:28:33.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:28:33.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:28:33.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: debug 2021-07-31T03:28:33.437+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:28:33.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:28:33.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:28:33.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: yield 2021-07-31T03:28:33.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:28:33.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:28:33.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:28:33.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:28:33.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:28:33.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:28:33.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:28:33.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:28:33.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:28:33.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: task.result() 2021-07-31T03:28:33.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:28:33.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: debug 2021-07-31T03:28:33.437+0000 7fcb93160700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:28:33.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:28:33.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:28:33.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: yield 2021-07-31T03:28:33.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:28:33.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:28:33.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:28:33.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:28:33.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:28:33.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:28:33.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:28:33.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:28:33.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:28:33.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: task.result() 2021-07-31T03:28:33.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:28:33.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: 2021-07-31T03:28:33.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:28:33.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: 2021-07-31T03:28:33.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:28:33.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:28:33.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:28:33.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:28:33.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:28:33.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:28:33.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:28:33.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:28:33.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:28:33.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:28:33.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:28:33.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:28:33.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:28:33.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:28:33.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:28:33.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:28:33.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:28:33.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:28:33.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: raise 2021-07-31T03:28:33.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:28:33.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:28:33.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:28:33.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:28:33.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:28:33 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:28:34.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: cluster 2021-07-31T03:28:33.243760+0000 2021-07-31T03:28:34.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: mgr.x (mgr.24542) 4339 : cluster [DBG] pgmap v5047: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:28:34.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: audit 2021-07-31T03:28:34.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: 2021-07-31T03:28:33. 2021-07-31T03:28:34.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: 343030+0000 mon.b (mon.2) 582 2021-07-31T03:28:34.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:28:34.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T03:28:34.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: T03:28:33.344428+0000 mon.b ( 2021-07-31T03:28:34.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: mon.2) 583 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:28:34.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: 2021-07-31T03:28:34.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: audit 2021-07 2021-07-31T03:28:34.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: -31T03:28:33 2021-07-31T03:28:34.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: .345309+0000 mon.b (mon.2) 584 2021-07-31T03:28:34.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:28:34.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: cephadm 2021-07-31T03:28:34.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: 2021-07-31T03 2021-07-31T03:28:34.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: :28:33.346085+0000 mgr.x 2021-07-31T03:28:34.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: (mgr.24542) 4340 : cephadm [WRN] 2021-07-31T03:28:34.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:28:34.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: cephadm 2021 2021-07-31T03:28:34.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: -07-31T03:28 2021-07-31T03:28:34.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: :33.391791+0000 mgr.x (mgr 2021-07-31T03:28:34.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: .24542) 4341 : cephadm [ERR] 2021-07-31T03:28:34.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:28:34.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:28:34.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: yield 2021-07-31T03:28:34.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:28:34.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:28:34.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:28:34.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:28:34.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:28:34.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:28:34.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:28:34.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:28:34.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:28:34.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: task.result() 2021-07-31T03:28:34.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:28:34.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: cephadm 2021-07-31T03:28:33.392354+0000 mgr.x (mgr.24542) 4342 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:28:34.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: cephadm 2021-07 2021-07-31T03:28:34.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: -31T03:28:33.403438+0000 2021-07-31T03:28:34.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: mgr.x (mgr.24542) 4343 2021-07-31T03:28:34.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: : cephadm [ERR] 2021-07-31T03:28:34.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:28:34.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:28:34.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: yield 2021-07-31T03:28:34.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:28:34.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:28:34.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:28:34.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:28:34.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:28:34.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:28:34.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:28:34.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:28:34.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:28:34.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: task.result() 2021-07-31T03:28:34.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:28:34.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: cephadm 2021-07-31T03:28:33.404380+0000 mgr.x (mgr.24542) 4344 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:28:34.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: cephadm 2021-07-31T03:28:34.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: 2021-07-31T03: 2021-07-31T03:28:34.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: 28:33.429105+0000 mgr.x (mgr 2021-07-31T03:28:34.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: .24542) 4345 : cephadm [ERR] 2021-07-31T03:28:34.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:28:34.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:28:34.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: yield 2021-07-31T03:28:34.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:28:34.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:28:34.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:28:34.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:28:34.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:28:34.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:28:34.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:28:34.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:28:34.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:28:34.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: task.result() 2021-07-31T03:28:34.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:28:34.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: cephadm 2021-07-31T03:28:33.429516+0000 mgr.x (mgr.24542) 4346 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:28:34.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:28:34.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:28:34.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: yield 2021-07-31T03:28:34.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:28:34.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:28:34.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:28:34.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:28:34.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:28:34.545 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: cluster 2021-07-31T03:28:33.243760+0000 mgr.x (mgr.24542) 4339 : 2021-07-31T03:28:34.546 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: cluster [DBG] pgmap v5047: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:28:34.546 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: audit 2021-07-31T03:28:33.343030+0000 mon.b (mon.2) 582 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:28:34.546 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: audit 2021-07-31T03:28:34.546 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: 2021-07-31T03:28:33.344428+0000 mon.b (mon.2) 583 : audit [DBG] 2021-07-31T03:28:34.546 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:28:34.547 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: audit 2021-07-31T03:28: 2021-07-31T03:28:34.547 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: 33.345309+0000 mon.b (mon.2) 584 : 2021-07-31T03:28:34.547 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:28:34.548 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: cephadm 2021-07-31T03 2021-07-31T03:28:34.548 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: :28:33.346085+0000 mgr.x (mgr.24542) 4340 : cephadm 2021-07-31T03:28:34.548 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:28:34.548 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: cephadm 2021-07-31 2021-07-31T03:28:34.549 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: T03:28:33.391791+0000 mgr.x (mgr.24542 2021-07-31T03:28:34.549 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: ) 4341 : cephadm [ERR] 2021-07-31T03:28:34.549 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:28:34.549 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:28:34.549 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: yield 2021-07-31T03:28:34.550 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:28:34.550 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:28:34.550 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:28:34.550 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:28:34.551 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:28:34.551 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:28:34.551 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:28:34.551 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:28:34.552 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:28:34.552 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: task.result() 2021-07-31T03:28:34.552 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:28:34.552 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: cephadm 2021-07-31T03:28:33.392354+0000 mgr.x (mgr.24542) 4342 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:28:34.553 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: cephadm 2021 2021-07-31T03:28:34.553 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: -07-31T03:28:33.403438+0000 mgr.x ( 2021-07-31T03:28:34.553 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: mgr.24542) 4343 : cephadm [ERR] 2021-07-31T03:28:34.553 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:28:34.553 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:28:34.554 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: yield 2021-07-31T03:28:34.554 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:28:34.554 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:28:34.554 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:28:34.555 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:28:34.555 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:28:34.555 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:28:34.555 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:28:34.555 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:28:34.556 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:28:34.556 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: task.result() 2021-07-31T03:28:34.556 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:28:34.556 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: 2021-07-31T03:28:34.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: cephadm 2021-07-31T03:28:33.404380+0000 mgr.x (mgr.24542) 4344 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:28:34.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: cephadm 2021-07-31T03:28:33.429105+0000 mgr.x (mgr 2021-07-31T03:28:34.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: .24542) 4345 : cephadm [ERR] 2021-07-31T03:28:34.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:28:34.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:28:34.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: yield 2021-07-31T03:28:34.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:28:34.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:28:34.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:28:34.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:28:34.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:28:34.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:28:34.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:28:34.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:28:34.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:28:34.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: task.result() 2021-07-31T03:28:34.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:28:34.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: cephadm 2021-07-31T03:28:33.429516+0000 mgr.x (mgr.24542) 4346 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:28:34.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:28:34.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:28:34.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: yield 2021-07-31T03:28:34.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:28:34.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:28:34.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:28:34.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:28:34.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:28:34.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:28:34.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:28:34.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:28:34.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:28:34.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: task.result() 2021-07-31T03:28:34.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:28:34.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:28:34.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:28:34.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:28:34.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: task.result() 2021-07-31T03:28:34.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:28:34.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: 2021-07-31T03:28:34.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:28:34.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: 2021-07-31T03:28:34.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:28:34.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:28:34.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:28:34.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:28:34.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:28:34.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:28:34.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:28:34.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:28:34.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:28:34.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:28:34.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:28:34.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:28:34.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:28:34.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:28:34.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:28:34.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:28:34.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:28:34.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:28:34.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: raise 2021-07-31T03:28:34.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:28:34.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:28:34.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:28:34.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:28:34.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:28:34.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: cephadm 2021-07-31T03:28:33.438053+0000 mgr.x (mgr.24542) 4347 : cephadm [ERR] 2021-07-31T03:28:34.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:28:34.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:28:34.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: yield 2021-07-31T03:28:34.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:28:34.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:28:34.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:28:34.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:28:34.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:28:34.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: 2021-07-31T03:28:34.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:28:34.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: 2021-07-31T03:28:34.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:28:34.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:28:34.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:28:34.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:28:34.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:28:34.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:28:34.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:28:34.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:28:34.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:28:34.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:28:34.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:28:34.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:28:34.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:28:34.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:28:34.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:28:34.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:28:34.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:28:34.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:28:34.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: raise 2021-07-31T03:28:34.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:28:34.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:28:34.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:28:34.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:28:34.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:28:34.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: cephadm 2021-07-31T03:28:34.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: 2021-07-31T03:28:33.438053+0000 mgr.x (mgr.24542) 4347 : cephadm [ERR] 2021-07-31T03:28:34.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:28:34.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:28:34.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: yield 2021-07-31T03:28:34.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:28:34.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:28:34.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:28:34.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:28:34.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:28:34.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:28:34.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:28:34.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:28:34.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:28:34.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: task.result() 2021-07-31T03:28:34.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:28:34.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: cephadm 2021-07-31T03:28:33.438432+0000 mgr.x (mgr.24542) 4348 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:28:34.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:28:34.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:28:34.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: yield 2021-07-31T03:28:34.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:28:34.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:28:34.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:28:34.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:28:34.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:28:34.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:28:34.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:28:34.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:28:34.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:28:34.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: task.result() 2021-07-31T03:28:34.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:28:34.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: 2021-07-31T03:28:34.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:28:34.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: 2021-07-31T03:28:34.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:28:34.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:28:34.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:28:34.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:28:34.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:28:34.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:28:34.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:28:34.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:28:34.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:28:34.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:28:34.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:28:34.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:28:34.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:28:34.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:28:34.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:28:34.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:28:34.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:28:34.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:28:34.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: raise 2021-07-31T03:28:34.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:28:34.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:28:34.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:28:34.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:28:34.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:28:34.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:28:34.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:28:34.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:28:34.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:28:34.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:28:34.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: task.result() 2021-07-31T03:28:34.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:28:34.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: cephadm 2021-07-31T03:28:33.438432+0000 mgr.x (mgr.24542) 4348 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:28:34.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:28:34.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:28:34.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: yield 2021-07-31T03:28:34.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:28:34.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:28:34.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:28:34.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:28:34.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:28:34.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:28:34.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:28:34.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:28:34.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:28:34.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: task.result() 2021-07-31T03:28:34.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:28:34.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: 2021-07-31T03:28:34.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:28:34.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: 2021-07-31T03:28:34.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:28:34.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:28:34.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:28:34.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:28:34.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:28:34.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:28:34.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:28:34.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:28:34.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:28:34.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:28:34.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:28:34.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:28:34.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:28:34.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:28:34.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:28:34.603 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:28:34.603 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:28:34.603 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:28:34.604 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: raise 2021-07-31T03:28:34.604 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:28:34.604 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:28:34.604 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:28:34.605 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:28:34.605 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:34 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:28:34.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: cluster 2021-07-31T03:28:33.243760+0000 mgr.x (mgr.24542) 4339 : cluster [DBG] pgmap v5047: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:28:34.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: audit 2021-07-31T03:28:33.343030+0000 mon.b (mon.2) 582 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:28:34.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: audit 2021-07-31T03:28:33.344428+0000 mon.b (mon.2) 583 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:28:34.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: audit 2021-07-31T03:28:33.345309+0000 mon.b (mon.2) 584 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:28:34.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: cephadm 2021-07-31T03 2021-07-31T03:28:34.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: :28:33.346085+0000 mgr.x (mgr.24542) 4340 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:28:34.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: cephadm 2021-07-31 2021-07-31T03:28:34.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: T03:28:33.391791+0000 mgr.x (mgr.24542) 4341 : cephadm 2021-07-31T03:28:34.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: [ERR] 2021-07-31T03:28:34.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:28:34.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:28:34.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: yield 2021-07-31T03:28:34.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:28:34.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:28:34.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:28:34.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:28:34.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:28:34.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:28:34.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:28:34.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:28:34.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:28:34.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: task.result() 2021-07-31T03:28:34.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:28:34.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: cephadm 2021-07-31T03:28:34.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: 2021-07-31T03:28:33.392354+0000 mgr.x (mgr.24542) 4342 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:28:34.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: cephadm 2021-07-31T03:28:33.403438+0000 mgr.x (mgr.24542) 4343 : cephadm [ERR] 2021-07-31T03:28:34.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:28:34.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:28:34.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: yield 2021-07-31T03:28:34.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:28:34.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:28:34.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:28:34.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:28:34.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:28:34.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:28:34.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:28:34.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:28:34.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:28:34.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: task.result() 2021-07-31T03:28:34.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:28:34.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: cephadm 2021-07-31T03:28:33.404380+0000 mgr.x (mgr.24542) 4344 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:28:34.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: cephadm 2021-07-31T03:28:33.429105+0000 mgr.x (mgr.24542) 4345 : cephadm [ERR] 2021-07-31T03:28:34.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:28:34.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:28:34.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: yield 2021-07-31T03:28:34.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:28:34.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:28:34.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:28:34.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:28:34.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:28:34.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:28:34.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:28:34.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:28:34.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:28:34.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: task.result() 2021-07-31T03:28:34.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:28:34.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: cephadm 2021-07-31T03:28:33.429516+0000 mgr.x (mgr.24542) 4346 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:28:34.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:28:34.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:28:34.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: yield 2021-07-31T03:28:34.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:28:34.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:28:34.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:28:34.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:28:34.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:28:34.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:28:34.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:28:34.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:28:34.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:28:34.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: task.result() 2021-07-31T03:28:34.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:28:34.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: 2021-07-31T03:28:34.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:28:34.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: 2021-07-31T03:28:34.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:28:34.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:28:34.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:28:34.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:28:34.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:28:34.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:28:34.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:28:34.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:28:34.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:28:34.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:28:34.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:28:34.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:28:34.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:28:34.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:28:34.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:28:34.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:28:34.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:28:34.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:28:34.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: raise 2021-07-31T03:28:34.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:28:34.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:28:34.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:28:34.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:28:34.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:28:34.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: cephadm 2021-07-31T03:28:33.438053+0000 mgr.x (mgr.24542) 4347 : cephadm [ERR] 2021-07-31T03:28:34.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:28:34.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:28:34.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: yield 2021-07-31T03:28:34.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:28:34.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:28:34.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:28:34.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:28:34.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:28:34.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:28:34.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:28:34.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:28:34.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:28:34.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: task.result() 2021-07-31T03:28:34.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:28:34.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: cephadm 2021-07-31T03:28:33.438432+0000 mgr.x (mgr.24542) 4348 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:28:34.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:28:34.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:28:34.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: yield 2021-07-31T03:28:34.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:28:34.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:28:34.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:28:34.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:28:34.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:28:34.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:28:34.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:28:34.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:28:34.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:28:34.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: task.result() 2021-07-31T03:28:34.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:28:34.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: 2021-07-31T03:28:34.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:28:34.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: 2021-07-31T03:28:34.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:28:34.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:28:34.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:28:34.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:28:34.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:28:34.782 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:28:34.782 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:28:34.782 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:28:34.782 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:28:34.782 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:28:34.783 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:28:34.783 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:28:34.783 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:28:34.783 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:28:34.783 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:28:34.784 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:28:34.784 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:28:34.784 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:28:34.784 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: raise 2021-07-31T03:28:34.784 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:28:34.785 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:28:34.785 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:28:34.785 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:28:34.785 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:34 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:28:36.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:36 smithi045 conmon[33655]: cluster 2021-07-31T03:28:35.244188+0000 mgr.x (mgr.24542) 4349 : cluster [DBG] 2021-07-31T03:28:36.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:36 smithi045 conmon[33655]: pgmap v5048: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:28:36.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:36 smithi045 conmon[39718]: cluster 2021-07-31T03:28:35.244188+0000 2021-07-31T03:28:36.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:36 smithi045 conmon[39718]: mgr.x (mgr.24542) 4349 : cluster [DBG] pgmap v5048: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:28:36.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:36 smithi154 conmon[34553]: cluster 2021-07-31T03:28:35.244188+0000 mgr.x (mgr.24542) 4349 : cluster [DBG] pgmap v5048: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:28:36.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:36 smithi154 conmon[34553]: 2021-07-31T03:28:38.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:38 smithi045 conmon[33655]: cluster 2021-07-31T03:28:37.244553+0000 mgr.x (mgr.24542) 4350 : cluster [DBG] pgmap v5049: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:28:38.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:38 smithi045 conmon[39718]: cluster 2021-07-31T03:28:37.244553+0000 mgr.x (mgr 2021-07-31T03:28:38.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:38 smithi045 conmon[39718]: .24542) 4350 : cluster [DBG] pgmap v5049: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:28:38.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:38 smithi154 conmon[34553]: cluster 2021-07-31T03:28:37.244553+0000 mgr.x (mgr.24542) 4350 : cluster [DBG] pgmap v5049: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:28:40.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:40 smithi045 conmon[39718]: cluster 2021-07-31T03:28:39.245059+0000 mgr.x ( 2021-07-31T03:28:40.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:40 smithi045 conmon[39718]: mgr.24542) 4351 : cluster [DBG] pgmap v5050: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:28:40.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:40 smithi045 conmon[33655]: cluster 2021-07-31T03:28:39.245059+0000 mgr.x (mgr 2021-07-31T03:28:40.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:40 smithi045 conmon[33655]: .24542) 4351 : cluster [DBG] pgmap v5050: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:28:40.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:40 smithi154 conmon[34553]: cluster 2021-07-31T03:28:39.245059+0000 mgr.x (mgr.24542) 4351 : cluster [DBG] pgmap v5050: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:28:42.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:42 smithi045 conmon[33655]: cluster 2021-07-31T03:28:41.245499+0000 mgr.x (mgr.24542) 4352 : cluster [DBG] 2021-07-31T03:28:42.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:42 smithi045 conmon[33655]: pgmap v5051: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:28:42.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:42 smithi045 conmon[39718]: cluster 2021-07-31T03:28:41.245499+0000 mgr.x ( 2021-07-31T03:28:42.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:42 smithi045 conmon[39718]: mgr.24542) 4352 : cluster [DBG] pgmap v5051: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:28:42.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:42 smithi154 conmon[34553]: cluster 2021-07-31T03:28:41.245499+0000 mgr.x (mgr.24542) 4352 : cluster [DBG] pgmap v5051: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:28:44.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:44 smithi154 conmon[34553]: cluster 2021-07-31T03:28:43.245815+0000 mgr.x (mgr.24542) 4353 : cluster [DBG] pgmap v5052: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:28:44.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:44 smithi045 conmon[33655]: cluster 2021-07-31T03:28:43.245815+0000 mgr.x (mgr 2021-07-31T03:28:44.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:44 smithi045 conmon[33655]: .24542) 4353 : cluster [DBG] pgmap v5052: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:28:44.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:44 smithi045 conmon[39718]: cluster 2021-07-31T03:28:43.245815+0000 mgr.x ( 2021-07-31T03:28:44.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:44 smithi045 conmon[39718]: mgr.24542) 4353 : cluster [DBG] pgmap v5052: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:28:46.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:46 smithi154 conmon[34553]: cluster 2021-07-31T03:28:45.246328+0000 mgr.x (mgr.24542) 4354 : cluster [DBG] pgmap v5053: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:28:46.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:46 smithi154 conmon[34553]: 2021-07-31T03:28:46.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:46 smithi045 conmon[33655]: cluster 2021-07-31T03:28:45.246328+0000 mgr.x (mgr.24542) 2021-07-31T03:28:46.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:46 smithi045 conmon[33655]: 4354 : cluster [DBG] pgmap v5053: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:28:46.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:46 smithi045 conmon[39718]: cluster 2021-07-31T03:28:45.246328+0000 mgr.x (mgr.24542) 4354 : cluster [DBG] pgmap v5053: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:28:48.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:48 smithi154 conmon[34553]: cluster 2021-07-31T03:28:47.246724+0000 mgr.x (mgr.24542) 4355 : cluster [DBG] pgmap v5054: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:28:48.774 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:48 smithi045 conmon[33655]: cluster 2021-07-31T03:28:47.246724+0000 mgr.x (mgr.24542) 4355 : 2021-07-31T03:28:48.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:48 smithi045 conmon[33655]: cluster [DBG] pgmap v5054: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:28:48.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:48 smithi045 conmon[39718]: cluster 2021-07-31T03:28:47.246724+0000 2021-07-31T03:28:48.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:48 smithi045 conmon[39718]: mgr.x (mgr.24542) 4355 : cluster [DBG] pgmap v5054: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:28:50.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:50 smithi154 conmon[34553]: cluster 2021-07-31T03:28:49.247242+0000 mgr.x (mgr.24542) 4356 : cluster [DBG] pgmap v5055: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:28:50.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:50 smithi045 conmon[39718]: cluster 2021-07-31T03:28:49. 2021-07-31T03:28:50.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:50 smithi045 conmon[39718]: 247242+0000 mgr.x (mgr.24542) 4356 : cluster [DBG] pgmap v5055: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:28:50.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:50 smithi045 conmon[33655]: cluster 2021-07-31T03:28:49.247242+0000 mgr.x (mgr.24542) 4356 : cluster [DBG] pgmap v5055: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:28:52.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:52 smithi154 conmon[34553]: cluster 2021-07-31T03:28:51.247581+0000 mgr.x (mgr.24542) 4357 : cluster [DBG] 2021-07-31T03:28:52.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:52 smithi154 conmon[34553]: pgmap v5056: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:28:52.774 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:52 smithi045 conmon[33655]: cluster 2021-07-31T03:28:51.247581+0000 mgr.x (mgr. 2021-07-31T03:28:52.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:52 smithi045 conmon[33655]: 24542) 4357 : cluster [DBG] pgmap v5056: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:28:52.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:52 smithi045 conmon[39718]: cluster 2021-07-31T03:28:51. 2021-07-31T03:28:52.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:52 smithi045 conmon[39718]: 247581+0000 mgr.x (mgr.24542) 4357 : cluster [DBG] pgmap v5056: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:28:54.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:54 smithi154 conmon[34553]: cluster 2021-07-31T03:28:53.248050+0000 mgr.x (mgr.24542) 4358 : cluster [DBG] pgmap v5057: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:28:54.774 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:54 smithi045 conmon[33655]: cluster 2021-07-31T03:28:53.248050+0000 mgr.x (mgr.24542) 4358 : cluster [DBG] 2021-07-31T03:28:54.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:54 smithi045 conmon[33655]: pgmap v5057: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:28:54.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:54 smithi045 conmon[39718]: cluster 2021-07-31T03:28:53.248050+0000 mgr.x (mgr.24542) 4358 : cluster 2021-07-31T03:28:54.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:54 smithi045 conmon[39718]: [DBG] pgmap v5057: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:28:56.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:56 smithi154 conmon[34553]: cluster 2021-07-31T03:28:55.248515+0000 mgr.x (mgr.24542) 4359 : cluster [DBG] pgmap v5058: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:28:56.774 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:56 smithi045 conmon[39718]: cluster 2021-07-31T03:28:55.248515+0000 mgr.x ( 2021-07-31T03:28:56.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:56 smithi045 conmon[39718]: mgr.24542) 4359 : cluster [DBG] pgmap v5058: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:28:56.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:56 smithi045 conmon[33655]: cluster 2021-07-31T03:28:55.248515+0000 mgr.x (mgr.24542) 2021-07-31T03:28:56.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:56 smithi045 conmon[33655]: 4359 : cluster [DBG] pgmap v5058: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:28:58.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:28:58 smithi154 conmon[34553]: cluster 2021-07-31T03:28:57.248910+0000 mgr.x (mgr.24542) 4360 : cluster [DBG] pgmap v5059: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:28:58.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:58 smithi045 conmon[33655]: cluster 2021-07-31T03:28:57.248910+0000 2021-07-31T03:28:58.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:28:58 smithi045 conmon[33655]: mgr.x (mgr.24542) 4360 : cluster [DBG] pgmap v5059: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:28:58.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:58 smithi045 conmon[39718]: cluster 2021-07-31T03:28:57.248910+0000 mgr.x (mgr. 2021-07-31T03:28:58.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:28:58 smithi045 conmon[39718]: 24542) 4360 : cluster [DBG] pgmap v5059: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:00.412 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:29:00.688 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:29:00.689 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 103m ago 104m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:29:00.689 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 101m ago 104m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:29:00.689 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 101m ago 107m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:29:00.689 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 103m ago 110m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:29:00.689 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 103m ago 110m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:29:00.690 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 101m ago 108m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:29:00.690 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 103m ago 108m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:29:00.690 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 103m ago 104m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:29:00.690 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 101m ago 104m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:29:00.691 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 103m ago 107m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:29:00.691 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 103m ago 107m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:29:00.691 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 103m ago 106m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:29:00.691 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 103m ago 106m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:29:00.691 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 101m ago 106m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:29:00.692 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 101m ago 105m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:29:00.692 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 101m ago 105m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:29:00.692 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 101m ago 105m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:29:00.692 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 101m ago 104m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:29:00.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:00 smithi154 conmon[34553]: cluster 2021-07-31T03:28:59.249421+0000 mgr.x (mgr.24542) 4361 : cluster [DBG] pgmap v5060: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:00.774 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:00 smithi045 conmon[39718]: cluster 2021-07-31T03:28:59.249421+0000 mgr.x ( 2021-07-31T03:29:00.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:00 smithi045 conmon[39718]: mgr.24542) 4361 : cluster [DBG] pgmap v5060: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:00.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:00 smithi045 conmon[33655]: cluster 2021-07-31T03:28:59.249421+0000 2021-07-31T03:29:00.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:00 smithi045 conmon[33655]: mgr.x (mgr.24542) 4361 : cluster [DBG] pgmap v5060: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:01.025 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:29:01.026 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:29:01.026 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:29:01.026 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:29:01.026 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:29:01.027 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:29:01.027 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:29:01.027 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:29:01.027 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:29:01.027 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:29:01.028 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:29:01.028 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:29:01.028 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:29:01.028 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:29:01.028 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:29:01.029 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:29:01.029 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:29:01.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:01 smithi154 conmon[34553]: audit 2021-07-31T03:29:00.401370+0000 mgr.x (mgr.24542) 4362 : audit [DBG] from='client.19596 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:29:01.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:01 smithi154 conmon[34553]: audit 2021-07-31T03:29:00.681640 2021-07-31T03:29:01.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:01 smithi154 conmon[34553]: +0000 mgr.x (mgr.24542) 4363 : audit [DBG] from='client.29542 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:29:01.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:01 smithi154 conmon[34553]: audit 2021-07-31T03:29:01.026595+0000 mon.a (mon.0) 3310 : audit [DBG] from='client.? 172.21.15.45:0/4275587305' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:29:01.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:01 smithi045 conmon[39718]: audit 2021-07-31T03:29:00.401370+0000 mgr.x 2021-07-31T03:29:01.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:01 smithi045 conmon[39718]: (mgr.24542) 4362 : audit [DBG] from='client.19596 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:29:01.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:01 smithi045 conmon[39718]: audit 2021-07 2021-07-31T03:29:01.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:01 smithi045 conmon[39718]: -31T03:29:00.681640+0000 mgr.x (mgr.24542) 4363 : audit [DBG] from='client.29542 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:29:01.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:01 smithi045 conmon[39718]: 2021-07-31T03:29:01.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:01 smithi045 conmon[39718]: audit 2021-07-31T03:29:01.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:01 smithi045 conmon[39718]: 2021-07-31T03:29:01.026595+0000 2021-07-31T03:29:01.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:01 smithi045 conmon[39718]: mon.a (mon.0) 3310 : audit [DBG] 2021-07-31T03:29:01.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:01 smithi045 conmon[39718]: from='client.? 172.21.15.45:0/4275587305' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:29:01.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:01 smithi045 conmon[33655]: audit 2021-07-31T03:29:00.401370+0000 mgr.x (mgr.24542) 2021-07-31T03:29:01.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:01 smithi045 conmon[33655]: 4362 : audit [DBG] from='client.19596 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:29:01.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:01 smithi045 conmon[33655]: audit 2021-07-31T03:29:00.681640+0000 mgr.x (mgr.24542) 4363 : audit [DBG] from='client.29542 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:29:01.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:01 smithi045 conmon[33655]: audit 2021-07-31T03:29:01.026595+0000 mon.a ( 2021-07-31T03:29:01.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:01 smithi045 conmon[33655]: mon.0) 3310 : audit [DBG] from='client.? 172.21.15.45:0/4275587305' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:29:02.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:02 smithi154 conmon[34553]: cluster 2021-07-31T03:29:01.249800+0000 mgr.x (mgr.24542) 4364 : cluster [DBG] pgmap v5061: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:02.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:02 smithi045 conmon[39718]: cluster 2021-07-31T03:29:01.249800+0000 mgr.x (mgr. 2021-07-31T03:29:02.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:02 smithi045 conmon[39718]: 24542) 4364 : cluster [DBG] pgmap v5061: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:02.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:02 smithi045 conmon[33655]: cluster 2021-07-31T03:29:01.249800+0000 mgr.x (mgr.24542) 4364 2021-07-31T03:29:02.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:02 smithi045 conmon[33655]: : cluster [DBG] pgmap v5061: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:04.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:04 smithi154 conmon[34553]: cluster 2021-07-31T03:29:03.250195+0000 mgr.x (mgr.24542) 4365 : cluster [DBG] pgmap v5062: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:04.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:04 smithi045 conmon[33655]: cluster 2021-07-31T03:29:03.250195+0000 mgr.x ( 2021-07-31T03:29:04.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:04 smithi045 conmon[33655]: mgr.24542) 4365 : cluster [DBG] pgmap v5062: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:04.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:04 smithi045 conmon[39718]: cluster 2021-07-31T03:29:03.250195+0000 mgr.x (mgr 2021-07-31T03:29:04.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:04 smithi045 conmon[39718]: .24542) 4365 : cluster [DBG] pgmap v5062: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:06.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:06 smithi154 conmon[34553]: cluster 2021-07-31T03:29:05.250615+0000 mgr.x (mgr.24542) 4366 : cluster [DBG] pgmap v5063: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:06.774 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:06 smithi045 conmon[39718]: cluster 2021-07-31T03:29:05.250615+0000 mgr.x (mgr.24542) 4366 2021-07-31T03:29:06.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:06 smithi045 conmon[39718]: : cluster [DBG] pgmap v5063: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:06.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:06 smithi045 conmon[33655]: cluster 2021-07-31T03:29:05.250615+0000 mgr.x (mgr.24542) 4366 2021-07-31T03:29:06.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:06 smithi045 conmon[33655]: : cluster [DBG] pgmap v5063: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:08.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:08 smithi154 conmon[34553]: cluster 2021-07-31T03:29:07.251069+0000 mgr.x (mgr.24542) 4367 : cluster [DBG] pgmap v5064: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:08.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:08 smithi045 conmon[39718]: cluster 2021-07-31T03:29:07.251069+0000 mgr.x (mgr. 2021-07-31T03:29:08.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:08 smithi045 conmon[39718]: 24542) 4367 : cluster [DBG] pgmap v5064: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:08.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:08 smithi045 conmon[33655]: cluster 2021-07-31T03:29:07.251069+0000 mgr.x (mgr.24542) 4367 : cluster [DBG] pgmap v5064: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:10.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:10 smithi154 conmon[34553]: cluster 2021-07-31T03:29:09.251499+0000 mgr.x (mgr.24542) 4368 : cluster [DBG] pgmap v5065: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:10.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:10 smithi045 conmon[39718]: cluster 2021-07-31T03:29:09.251499+0000 mgr.x (mgr.24542) 2021-07-31T03:29:10.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:10 smithi045 conmon[39718]: 4368 : cluster [DBG] pgmap v5065: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:10.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:10 smithi045 conmon[33655]: cluster 2021-07-31T03:29:09.251499+0000 mgr.x (mgr.24542) 4368 : cluster 2021-07-31T03:29:10.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:10 smithi045 conmon[33655]: [DBG] pgmap v5065: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:12.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:12 smithi154 conmon[34553]: cluster 2021-07-31T03:29:11.251948+0000 mgr.x (mgr.24542) 4369 : cluster [DBG] pgmap v5066: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:12.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:12 smithi045 conmon[39718]: cluster 2021-07-31T03:29:11.251948+0000 mgr.x (mgr 2021-07-31T03:29:12.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:12 smithi045 conmon[39718]: .24542) 4369 : cluster [DBG] pgmap v5066: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:12.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:12 smithi045 conmon[33655]: cluster 2021-07-31T03:29:11.251948+0000 mgr.x (mgr.24542) 4369 : cluster [DBG] pgmap v5066: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:14.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:14 smithi154 conmon[34553]: cluster 2021-07-31T03:29:13.252353+0000 mgr.x (mgr.24542) 4370 : cluster [DBG] pgmap v5067: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:14.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:14 smithi045 conmon[39718]: cluster 2021-07-31T03:29:13.252353+0000 mgr.x (mgr. 2021-07-31T03:29:14.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:14 smithi045 conmon[39718]: 24542) 4370 : cluster [DBG] pgmap v5067: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:14.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:14 smithi045 conmon[33655]: cluster 2021-07-31T03 2021-07-31T03:29:14.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:14 smithi045 conmon[33655]: :29:13.252353+0000 mgr.x (mgr.24542) 4370 : cluster [DBG] pgmap v5067: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:16.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:16 smithi154 conmon[34553]: cluster 2021-07-31T03:29:15.252914+0000 mgr.x (mgr.24542) 4371 : cluster [DBG] pgmap v5068: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:16.774 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:16 smithi045 conmon[39718]: cluster 2021-07-31T03:29:15.252914+0000 mgr.x 2021-07-31T03:29:16.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:16 smithi045 conmon[39718]: (mgr.24542) 4371 : cluster [DBG] pgmap v5068: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:16.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:16 smithi045 conmon[33655]: cluster 2021-07-31T03:29:15.252914+0000 mgr.x (mgr.24542) 4371 : cluster [DBG] pgmap v5068: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:16.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:16 smithi045 conmon[33655]: 2021-07-31T03:29:18.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:18 smithi154 conmon[34553]: cluster 2021-07-31T03:29:17.253328+0000 mgr.x (mgr.24542) 4372 : cluster [DBG] 2021-07-31T03:29:18.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:18 smithi154 conmon[34553]: pgmap v5069: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:18.774 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:18 smithi045 conmon[39718]: cluster 2021-07-31T03:29:17.253328+0000 mgr.x ( 2021-07-31T03:29:18.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:18 smithi045 conmon[39718]: mgr.24542) 4372 : cluster [DBG] pgmap v5069: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:18.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:18 smithi045 conmon[33655]: cluster 2021-07-31T03:29:17.253328+0000 mgr.x (mgr.24542) 4372 : cluster [DBG] 2021-07-31T03:29:18.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:18 smithi045 conmon[33655]: pgmap v5069: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:20.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:20 smithi154 conmon[34553]: cluster 2021-07-31T03:29:19.253735+0000 mgr.x (mgr.24542) 4373 : cluster [DBG] pgmap v5070: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:20.774 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:20 smithi045 conmon[39718]: cluster 2021-07-31T03:29:19.253735+0000 mgr.x (mgr.24542 2021-07-31T03:29:20.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:20 smithi045 conmon[39718]: ) 4373 : cluster [DBG] pgmap v5070: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:20.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:20 smithi045 conmon[33655]: cluster 2021-07-31T03:29:19.253735+0000 mgr.x (mgr.24542) 4373 : cluster 2021-07-31T03:29:20.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:20 smithi045 conmon[33655]: [DBG] pgmap v5070: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:22.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:22 smithi154 conmon[34553]: cluster 2021-07-31T03:29:21.254193+0000 mgr.x (mgr.24542) 4374 : cluster [DBG] pgmap v5071: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:22.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:22 smithi045 conmon[39718]: cluster 2021-07-31T03:29:21.254193+0000 mgr.x (mgr.24542 2021-07-31T03:29:22.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:22 smithi045 conmon[39718]: ) 4374 : cluster [DBG] pgmap v5071: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:22.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:22 smithi045 conmon[33655]: cluster 2021-07-31T03:29:21.254193+0000 2021-07-31T03:29:22.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:22 smithi045 conmon[33655]: mgr.x (mgr.24542) 4374 : cluster [DBG] pgmap v5071: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:24.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:24 smithi154 conmon[34553]: cluster 2021-07-31T03:29:23.254631+0000 mgr.x (mgr.24542) 4375 : cluster [DBG] pgmap v5072: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:24.774 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:24 smithi045 conmon[33655]: cluster 2021-07-31T03:29:23.254631+0000 mgr.x (mgr.24542) 2021-07-31T03:29:24.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:24 smithi045 conmon[33655]: 4375 : cluster [DBG] pgmap v5072: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:24.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:24 smithi045 conmon[39718]: cluster 2021-07-31T03:29:23.254631+0000 mgr.x (mgr.24542) 4375 : cluster 2021-07-31T03:29:24.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:24 smithi045 conmon[39718]: [DBG] pgmap v5072: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:26.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:26 smithi154 conmon[34553]: cluster 2021-07-31T03:29:25.255130+0000 mgr.x (mgr.24542) 4376 : cluster [DBG] pgmap v5073: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:26.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:26 smithi045 conmon[39718]: cluster 2021-07-31T03:29:25.255130+0000 mgr.x (mgr.24542) 4376 : 2021-07-31T03:29:26.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:26 smithi045 conmon[39718]: cluster [DBG] pgmap v5073: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:26.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:26 smithi045 conmon[33655]: cluster 2021-07-31T03:29:25.255130+0000 mgr.x (mgr.24542) 4376 : cluster [DBG] pgmap v5073: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:28.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:28 smithi154 conmon[34553]: cluster 2021-07-31T03:29:27.255512+0000 mgr.x (mgr.24542) 4377 : cluster [DBG] pgmap v5074: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:28.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:28 smithi154 conmon[34553]: 2021-07-31T03:29:28.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:28 smithi154 conmon[34553]: audit 2021-07-31T03:29:28.340880+0000 mon.b (mon.2) 585 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:29:28.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:28 smithi154 conmon[34553]: audit 2021-07-31T03:29:28.341208 2021-07-31T03:29:28.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:28 smithi154 conmon[34553]: +0000 mon.b (mon.2) 586 : audit [INF] 2021-07-31T03:29:28.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:28 smithi154 conmon[34553]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:29:28.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:28 smithi154 conmon[34553]: audit 2021-07 2021-07-31T03:29:28.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:28 smithi154 conmon[34553]: -31T03:29:28.341383 2021-07-31T03:29:28.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:28 smithi154 conmon[34553]: +0000 mon.a (mon.0) 3311 : audit [INF] 2021-07-31T03:29:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:28 smithi154 conmon[34553]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:29:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:28 smithi154 conmon[34553]: audit 2021-07 2021-07-31T03:29:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:28 smithi154 conmon[34553]: -31T03:29:28.341742 2021-07-31T03:29:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:28 smithi154 conmon[34553]: +0000 mon.a (mon.0) 3312 : 2021-07-31T03:29:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:28 smithi154 conmon[34553]: audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:29:28.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:28 smithi045 conmon[33655]: cluster 2021-07-31T03:29:27.255512+0000 mgr.x (mgr.24542) 4377 : cluster [DBG] 2021-07-31T03:29:28.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:28 smithi045 conmon[33655]: pgmap v5074: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:28.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:28 smithi045 conmon[33655]: audit 2021-07-31T03:29:28.340880+0000 mon.b (mon.2) 585 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:29:28.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:28 smithi045 conmon[33655]: audit 2021-07-31T03:29 2021-07-31T03:29:28.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:28 smithi045 conmon[33655]: :28.341208+0000 mon.b (mon.2) 586 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:29:28.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:28 smithi045 conmon[33655]: 2021-07-31T03:29:28.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:28 smithi045 conmon[33655]: audit 2021-07 2021-07-31T03:29:28.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:28 smithi045 conmon[33655]: -31T03:29:28.341383+0000 mon.a (mon.0) 3311 : audit [INF] 2021-07-31T03:29:28.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:28 smithi045 conmon[33655]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:29:28.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:28 smithi045 conmon[33655]: audit 2021 2021-07-31T03:29:28.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:28 smithi045 conmon[33655]: -07-31T03:29:28.341742+0000 mon.a (mon.0) 3312 : audit [INF] 2021-07-31T03:29:28.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:28 smithi045 conmon[33655]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:29:28.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:28 smithi045 conmon[39718]: cluster 2021-07-31T03:29:27.255512+0000 mgr.x ( 2021-07-31T03:29:28.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:28 smithi045 conmon[39718]: mgr.24542) 4377 : cluster [DBG] pgmap v5074: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:28.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:28 smithi045 conmon[39718]: audit 2021-07-31T03:29:28.340880+0000 mon.b ( 2021-07-31T03:29:28.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:28 smithi045 conmon[39718]: mon.2) 585 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:29:28.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:28 smithi045 conmon[39718]: audit 2021-07-31T03 2021-07-31T03:29:28.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:28 smithi045 conmon[39718]: :29:28.341208+0000 mon.b (mon.2 2021-07-31T03:29:28.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:28 smithi045 conmon[39718]: ) 586 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:29:28.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:28 smithi045 conmon[39718]: audit 2021 2021-07-31T03:29:28.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:28 smithi045 conmon[39718]: -07-31T03:29:28.341383+0000 mon.a ( 2021-07-31T03:29:28.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:28 smithi045 conmon[39718]: mon.0) 3311 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:29:28.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:28 smithi045 conmon[39718]: audit 2021-07-31T03:29:28.341742+0000 mon.a (mon.0 2021-07-31T03:29:28.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:28 smithi045 conmon[39718]: ) 3312 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:29:30.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:30 smithi154 conmon[34553]: cluster 2021-07-31T03:29:29.256065+0000 mgr.x (mgr.24542) 4378 : cluster [DBG] pgmap v5075: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:30.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:30 smithi154 conmon[34553]: 2021-07-31T03:29:30.774 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:30 smithi045 conmon[33655]: cluster 2021-07-31T03:29:29.256065+0000 mgr.x (mgr.24542) 4378 : cluster 2021-07-31T03:29:30.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:30 smithi045 conmon[33655]: [DBG] pgmap v5075: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:30.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:30 smithi045 conmon[39718]: cluster 2021-07-31T03:29:29.256065+0000 mgr.x ( 2021-07-31T03:29:30.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:30 smithi045 conmon[39718]: mgr.24542) 4378 : cluster [DBG] pgmap v5075: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:31.341 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:29:31.619 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:29:31.619 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 104m ago 105m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:29:31.619 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 102m ago 104m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:29:31.620 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 102m ago 108m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:29:31.620 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 104m ago 110m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:29:31.620 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 104m ago 110m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:29:31.620 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 102m ago 108m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:29:31.620 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 104m ago 108m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:29:31.621 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 104m ago 105m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:29:31.621 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 102m ago 105m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:29:31.621 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 104m ago 108m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:29:31.621 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 104m ago 107m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:29:31.621 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 104m ago 107m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:29:31.622 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 104m ago 107m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:29:31.622 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 102m ago 106m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:29:31.622 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 102m ago 106m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:29:31.622 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 102m ago 106m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:29:31.622 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 102m ago 105m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:29:31.623 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 102m ago 105m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:29:31.952 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:29:31.953 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:29:31.953 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:29:31.953 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:29:31.953 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:29:31.954 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:29:31.954 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:29:31.954 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:29:31.954 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:29:31.954 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:29:31.955 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:29:31.955 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:29:31.955 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:29:31.955 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:29:31.955 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:29:31.956 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:29:31.956 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:29:32.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:32 smithi154 conmon[34553]: cluster 2021-07-31T03:29:31.256402+0000 mgr.x (mgr.24542) 4379 : cluster [DBG] pgmap v5076: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:32.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:32 smithi154 conmon[34553]: audit 2021-07-31T03:29:31.330135+0000 mgr.x (mgr.24542) 4380 : audit [DBG] from='client.29554 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:29:32.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:32 smithi154 conmon[34553]: audit 2021-07-31T03:29:31.611956+0000 mgr.x (mgr.24542) 4381 : audit [DBG] from='client.19620 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:29:32.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:32 smithi154 conmon[34553]: audit 2021-07-31T03:29:31.953457+0000 mon.c 2021-07-31T03:29:32.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:32 smithi154 conmon[34553]: (mon.1) 209 : audit [DBG] from='client.? 172.21.15.45:0/2596724954' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:29:32.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:32 smithi045 conmon[39718]: cluster 2021-07-31T03:29:31.256402+0000 mgr.x 2021-07-31T03:29:32.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:32 smithi045 conmon[39718]: (mgr.24542) 4379 : cluster [DBG] pgmap v5076: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:32.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:32 smithi045 conmon[39718]: audit 2021-07-31T03:29:31.330135+0000 mgr.x ( 2021-07-31T03:29:32.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:32 smithi045 conmon[39718]: mgr.24542) 4380 : audit [DBG] from='client.29554 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:29:32.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:32 smithi045 conmon[39718]: audit 2021-07 2021-07-31T03:29:32.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:32 smithi045 conmon[39718]: -31T03:29:31.611956+0000 mgr.x (mgr.24542) 4381 : audit [DBG] from='client.19620 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:29:32.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:32 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T03:29:32.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:32 smithi045 conmon[39718]: 31T03:29:31.953457+0000 mon.c (mon.1) 209 : 2021-07-31T03:29:32.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:32 smithi045 conmon[39718]: audit [DBG] from='client.? 172.21.15.45:0/2596724954' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:29:32.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:32 smithi045 conmon[33655]: cluster 2021-07-31T03:29:31.256402+0000 mgr.x (mgr 2021-07-31T03:29:32.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:32 smithi045 conmon[33655]: .24542) 4379 : cluster [DBG] pgmap v5076: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:32.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:32 smithi045 conmon[33655]: audit 2021-07-31T03:29:31.330135+0000 mgr.x (mgr.24542) 4380 : 2021-07-31T03:29:32.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:32 smithi045 conmon[33655]: audit [DBG] from='client.29554 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:29:32.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:32 smithi045 conmon[33655]: audit 2021- 2021-07-31T03:29:32.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:32 smithi045 conmon[33655]: 07-31T03:29:31.611956+0000 mgr.x (mgr.24542) 4381 2021-07-31T03:29:32.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:32 smithi045 conmon[33655]: : audit [DBG] from='client.19620 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:29:32.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:32 smithi045 conmon[33655]: audit 2021-07-31T 2021-07-31T03:29:32.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:32 smithi045 conmon[33655]: 03:29:31.953457+0000 mon.c (mon.1) 209 2021-07-31T03:29:32.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:32 smithi045 conmon[33655]: : audit [DBG] from='client.? 172.21.15.45:0/2596724954' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:29:33.752 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: debug 2021-07-31T03:29:33.485+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:29:33.752 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:29:33.753 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:29:33.753 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: yield 2021-07-31T03:29:33.753 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:29:33.753 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:29:33.753 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:29:33.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:29:33.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:29:33.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:29:33.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:29:33.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:29:33.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:29:33.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: task.result() 2021-07-31T03:29:33.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:29:33.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: debug 2021-07-31T03:29:33.496+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:29:33.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:29:33.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:29:33.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: yield 2021-07-31T03:29:33.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:29:33.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:29:33.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:29:33.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:29:33.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:29:33.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:29:33.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:29:33.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:29:33.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:29:33.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: task.result() 2021-07-31T03:29:33.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:29:33.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: debug 2021-07-31T03:29:33.522+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:29:33.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:29:33.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:29:33.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: yield 2021-07-31T03:29:33.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:29:33.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:29:33.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:29:33.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:29:33.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:29:33.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:29:33.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:29:33.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:29:33.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:29:33.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: task.result() 2021-07-31T03:29:33.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:29:33.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: debug 2021-07-31T03:29:33.522+0000 7fcb9215e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:29:33.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:29:33.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:29:33.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: yield 2021-07-31T03:29:33.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:29:33.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:29:33.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:29:33.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:29:33.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:29:33.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:29:33.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:29:33.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:29:33.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:29:33.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: task.result() 2021-07-31T03:29:33.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:29:33.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: 2021-07-31T03:29:33.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:29:33.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: 2021-07-31T03:29:33.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:29:33.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:29:33.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:29:33.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:29:33.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:29:33.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:29:33.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:29:33.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:29:33.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:29:33.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:29:33.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:29:33.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:29:33.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:29:33.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:29:33.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:29:33.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:29:33.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:29:33.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:29:33.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: raise 2021-07-31T03:29:33.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:29:33.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:29:33.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:29:33.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:29:33.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:29:33.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: debug 2021-07-31T03:29:33.530+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:29:33.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:29:33.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:29:33.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: yield 2021-07-31T03:29:33.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:29:33.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:29:33.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:29:33.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:29:33.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:29:33.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:29:33.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:29:33.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:29:33.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:29:33.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: task.result() 2021-07-31T03:29:33.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:29:33.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: debug 2021-07-31T03:29:33.530+0000 7fcb9195d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:29:33.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:29:33.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:29:33.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: yield 2021-07-31T03:29:33.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:29:33.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:29:33.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:29:33.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:29:33.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:29:33.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:29:33.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:29:33.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:29:33.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:29:33.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: task.result() 2021-07-31T03:29:33.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:29:33.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: 2021-07-31T03:29:33.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:29:33.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: 2021-07-31T03:29:33.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:29:33.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:29:33.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:29:33.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:29:33.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:29:33.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:29:33.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:29:33.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:29:33.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:29:33.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:29:33.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:29:33.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:29:33.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:29:33.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:29:33.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:29:33.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:29:33.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:29:33.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:29:33.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: raise 2021-07-31T03:29:33.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:29:33.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:29:33.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:29:33.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:29:33.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:29:33 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:29:34.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: cluster 2021-07-31T03:29:33.256781+0000 mgr.x (mgr.24542) 4382 : cluster [DBG] pgmap v5077: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:34.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: audit 2021-07-31T03:29:33.439304+0000 mon.b (mon.2) 587 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:29:34.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: audit 2021-07-31T03:29:33.440816+0000 mon.b (mon.2) 588 2021-07-31T03:29:34.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:29:34.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: audit 2021-07 2021-07-31T03:29:34.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: -31T03:29:33.441733 2021-07-31T03:29:34.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: +0000 mon.b (mon.2) 589 : 2021-07-31T03:29:34.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:29:34.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: cephadm 2021-07 2021-07-31T03:29:34.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: -31T03:29:33.442639+0000 mgr.x 2021-07-31T03:29:34.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: (mgr.24542) 4383 : cephadm [WRN] 2021-07-31T03:29:34.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:29:34.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: cephadm 2021- 2021-07-31T03:29:34.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: 07-31T03:29:33. 2021-07-31T03:29:34.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: 486295+0000 mgr.x (mgr.24542) 2021-07-31T03:29:34.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: 4384 : cephadm [ERR] 2021-07-31T03:29:34.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:29:34.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:29:34.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: yield 2021-07-31T03:29:34.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:29:34.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:29:34.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:29:34.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:29:34.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:29:34.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:29:34.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:29:34.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:29:34.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:29:34.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: task.result() 2021-07-31T03:29:34.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:29:34.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: cephadm 2021-07-31T03:29:33.486688+0000 mgr.x (mgr.24542) 4385 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:29:34.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: cephadm 2021- 2021-07-31T03:29:34.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: 07-31T03:29:33.497646 2021-07-31T03:29:34.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: +0000 mgr.x (mgr.24542) 4386 : 2021-07-31T03:29:34.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: cephadm [ERR] 2021-07-31T03:29:34.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:29:34.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:29:34.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: yield 2021-07-31T03:29:34.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:29:34.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:29:34.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:29:34.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:29:34.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:29:34.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:29:34.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:29:34.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:29:34.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:29:34.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: task.result() 2021-07-31T03:29:34.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:29:34.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: cephadm 2021-07-31T03:29:33.498568+0000 mgr.x (mgr.24542) 4387 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:29:34.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: cephadm 2021- 2021-07-31T03:29:34.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: 07-31T03:29:33.522947 2021-07-31T03:29:34.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: +0000 mgr.x (mgr.24542) 4388 : 2021-07-31T03:29:34.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: cephadm [ERR] 2021-07-31T03:29:34.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:29:34.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:29:34.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: yield 2021-07-31T03:29:34.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:29:34.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:29:34.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:29:34.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:29:34.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:29:34.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:29:34.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:29:34.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:29:34.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:29:34.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: task.result() 2021-07-31T03:29:34.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:29:34.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: cephadm 2021-07-31T03:29:33.523538+0000 mgr.x (mgr.24542) 4389 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:29:34.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:29:34.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:29:34.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: yield 2021-07-31T03:29:34.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:29:34.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:29:34.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:29:34.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:29:34.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:29:34.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:29:34.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:29:34.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:29:34.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:29:34.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: task.result() 2021-07-31T03:29:34.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:29:34.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: 2021-07-31T03:29:34.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:29:34.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: 2021-07-31T03:29:34.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:29:34.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:29:34.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:29:34.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:29:34.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:29:34.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:29:34.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:29:34.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:29:34.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:29:34.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:29:34.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:29:34.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:29:34.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:29:34.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:29:34.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:29:34.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:29:34.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:29:34.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:29:34.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: raise 2021-07-31T03:29:34.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:29:34.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:29:34.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:29:34.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:29:34.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:29:34.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: cephadm 2021-07-31T03:29:33.531331+0000 mgr.x (mgr.24542) 4390 : cephadm [ERR] 2021-07-31T03:29:34.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:29:34.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:29:34.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: yield 2021-07-31T03:29:34.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:29:34.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:29:34.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:29:34.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:29:34.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:29:34.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:29:34.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:29:34.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: cluster 2021-07-31T03:29:33.256781+0000 mgr.x (mgr.24542) 4382 : cluster [DBG] 2021-07-31T03:29:34.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: pgmap v5077: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:34.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: audit 2021-07-31T03:29:33.439304+0000 mon.b (mon.2) 587 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:29:34.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: audit 2021-07-31T03:29:33.440816+0000 mon.b (mon.2) 588 : audit 2021-07-31T03:29:34.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:29:34.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: audit 2021-07-31T03:29:33.441733+0000 mon.b (mon. 2021-07-31T03:29:34.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: 2) 589 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:29:34.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: cephadm 2021-07-31T03:29:33. 2021-07-31T03:29:34.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: 442639+0000 mgr.x (mgr.24542) 4383 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:29:34.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: cephadm 2021-07-31T03:29:33.486295+0000 2021-07-31T03:29:34.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: mgr.x (mgr.24542) 4384 : cephadm [ERR] 2021-07-31T03:29:34.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:29:34.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:29:34.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: yield 2021-07-31T03:29:34.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:29:34.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:29:34.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:29:34.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:29:34.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:29:34.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:29:34.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:29:34.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:29:34.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:29:34.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: task.result() 2021-07-31T03:29:34.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:29:34.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: cephadm 2021-07-31T03: 2021-07-31T03:29:34.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: 29:33.486688+0000 mgr.x (mgr.24542) 4385 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:29:34.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: cephadm 2021-07-31T03:29:33.497646+0000 mgr.x (mgr.24542) 4386 : cephadm [ERR] 2021-07-31T03:29:34.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:29:34.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:29:34.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: yield 2021-07-31T03:29:34.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:29:34.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:29:34.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:29:34.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:29:34.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: cluster 2021-07-31T03:29:33 2021-07-31T03:29:34.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: .256781+0000 mgr.x (mgr.24542) 4382 : cluster [DBG] pgmap v5077: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:34.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: audit 2021 2021-07-31T03:29:34.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: -07-31T03:29:33.439304 2021-07-31T03:29:34.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: +0000 mon.b (mon.2) 587 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:29:34.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: audit 2021-07-31T03:29 2021-07-31T03:29:34.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: :33.440816+0000 mon.b (mon.2) 588 : audit 2021-07-31T03:29:34.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:29:34.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: audit 2021-07-31T03 2021-07-31T03:29:34.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: :29:33.441733+0000 mon.b (mon. 2021-07-31T03:29:34.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: 2) 589 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:29:34.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: cephadm 2021- 2021-07-31T03:29:34.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: 07-31T03:29:33.442639 2021-07-31T03:29:34.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 4383 : cephadm 2021-07-31T03:29:34.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:29:34.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: cephadm 2021-07-31T 2021-07-31T03:29:34.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: 03:29:33.486295+0000 mgr.x (mgr 2021-07-31T03:29:34.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: .24542) 4384 : cephadm [ERR] 2021-07-31T03:29:34.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:29:34.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:29:34.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: yield 2021-07-31T03:29:34.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:29:34.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:29:34.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:29:34.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:29:34.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:29:34.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:29:34.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:29:34.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:29:34.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:29:34.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: task.result() 2021-07-31T03:29:34.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:29:34.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: cephadm 2021-07-31T03:29:33.486688+0000 mgr.x (mgr.24542) 4385 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:29:34.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: cephadm 2021-07-31T03:29:33.497646+0000 mgr.x (mgr.24542) 4386 : cephadm [ERR] 2021-07-31T03:29:34.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:29:34.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:29:34.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: yield 2021-07-31T03:29:34.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:29:34.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:29:34.795 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:29:34.795 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:29:34.796 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: task.result() 2021-07-31T03:29:34.796 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:29:34.796 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: cephadm 2021-07-31T03:29:33.531839+0000 mgr.x (mgr.24542) 4391 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:29:34.796 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:29:34.796 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:29:34.797 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: yield 2021-07-31T03:29:34.797 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:29:34.797 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:29:34.797 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:29:34.797 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:29:34.798 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:29:34.798 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:29:34.798 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:29:34.798 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:29:34.799 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:29:34.799 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: task.result() 2021-07-31T03:29:34.799 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:29:34.799 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: 2021-07-31T03:29:34.799 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:29:34.800 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: 2021-07-31T03:29:34.800 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:29:34.800 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:29:34.800 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:29:34.800 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:29:34.801 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:29:34.801 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:29:34.801 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:29:34.801 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:29:34.802 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:29:34.802 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:29:34.802 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:29:34.803 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:29:34.803 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:29:34.804 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:29:34.804 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:29:34.804 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:29:34.804 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:29:34.805 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:29:34.805 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: raise 2021-07-31T03:29:34.805 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:29:34.805 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:29:34.805 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:29:34.806 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:29:34.806 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:34 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:29:34.806 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:29:34.806 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:29:34.807 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:29:34.807 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:29:34.807 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:29:34.807 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: task.result() 2021-07-31T03:29:34.808 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:29:34.808 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: cephadm 2021-07-31T03:29:33.498568+0000 2021-07-31T03:29:34.808 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: mgr.x (mgr.24542) 4387 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:29:34.808 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: cephadm 2021-07-31T03:29:33.522947+0000 mgr.x (mgr.24542) 4388 : cephadm [ERR] 2021-07-31T03:29:34.808 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:29:34.809 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:29:34.809 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: yield 2021-07-31T03:29:34.809 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:29:34.809 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:29:34.809 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:29:34.810 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:29:34.810 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:29:34.810 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:29:34.810 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:29:34.811 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:29:34.811 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:29:34.811 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: task.result() 2021-07-31T03:29:34.811 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:29:34.811 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: cephadm 2021-07-31T03:29:33.523538+0000 mgr.x (mgr. 2021-07-31T03:29:34.812 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: 24542) 4389 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:29:34.812 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:29:34.812 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:29:34.812 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: yield 2021-07-31T03:29:34.812 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:29:34.813 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:29:34.813 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:29:34.813 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:29:34.813 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:29:34.814 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:29:34.814 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:29:34.814 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:29:34.814 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:29:34.814 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: task.result() 2021-07-31T03:29:34.815 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:29:34.815 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: 2021-07-31T03:29:34.815 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:29:34.816 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:29:34.816 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:29:34.816 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:29:34.816 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:29:34.816 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:29:34.817 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:29:34.817 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:29:34.817 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: task.result() 2021-07-31T03:29:34.817 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:29:34.818 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: cephadm 2021-07-31T03:29:33.498568+0000 mgr.x (mgr.24542) 4387 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:29:34.818 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: cephadm 2021-07-31T03:29:33.522947+0000 mgr.x (mgr.24542) 4388 : cephadm [ERR] 2021-07-31T03:29:34.818 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:29:34.818 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:29:34.818 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: yield 2021-07-31T03:29:34.819 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:29:34.819 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:29:34.819 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:29:34.819 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:29:34.820 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:29:34.820 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:29:34.820 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:29:34.820 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:29:34.820 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:29:34.821 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: task.result() 2021-07-31T03:29:34.821 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:29:34.821 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: 2021-07-31T03:29:34.821 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: cephadm 2021-07-31T03:29:33.523538+0000 mgr.x (mgr.24542) 4389 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:29:34.821 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:29:34.822 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:29:34.822 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: yield 2021-07-31T03:29:34.822 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:29:34.822 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:29:34.823 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:29:34.823 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:29:34.823 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:29:34.823 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:29:34.823 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:29:34.824 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:29:34.824 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:29:34.824 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: task.result() 2021-07-31T03:29:34.824 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: 2021-07-31T03:29:34.825 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:29:34.825 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:29:34.825 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:29:34.826 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:29:34.826 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:29:34.826 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:29:34.826 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:29:34.826 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:29:34.827 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:29:34.827 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:29:34.827 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:29:34.827 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:29:34.827 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:29:34.828 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:29:34.828 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:29:34.828 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:29:34.828 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:29:34.829 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:29:34.829 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: raise 2021-07-31T03:29:34.829 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:29:34.829 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:29:34.829 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:29:34.830 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:29:34.830 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:29:34.830 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: cephadm 2021-07-31T03:29:33.531331+0000 mgr.x (mgr.24542) 4390 : cephadm [ERR] 2021-07-31T03:29:34.830 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:29:34.831 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:29:34.831 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: yield 2021-07-31T03:29:34.831 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:29:34.831 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:29:34.831 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:29:34.832 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:29:34.832 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:29:34.832 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:29:34.832 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:29:34.832 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:29:34.833 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:29:34.833 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: task.result() 2021-07-31T03:29:34.833 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:29:34.833 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:29:34.834 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: 2021-07-31T03:29:34.834 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:29:34.834 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: 2021-07-31T03:29:34.834 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:29:34.834 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:29:34.835 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:29:34.835 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:29:34.835 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:29:34.835 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:29:34.836 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:29:34.836 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:29:34.836 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:29:34.836 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:29:34.836 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:29:34.837 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:29:34.837 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:29:34.837 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:29:34.837 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:29:34.837 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:29:34.838 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:29:34.838 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:29:34.838 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: raise 2021-07-31T03:29:34.838 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:29:34.839 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:29:34.839 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:29:34.839 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:29:34.839 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:29:34.839 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: cephadm 2021-07-31T03:29:33.531331+0000 mgr.x (mgr.24542) 4390 : cephadm [ERR] 2021-07-31T03:29:34.840 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:29:34.840 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:29:34.840 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: yield 2021-07-31T03:29:34.840 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:29:34.840 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:29:34.841 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:29:34.841 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:29:34.841 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:29:34.841 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:29:34.841 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:29:34.842 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:29:34.842 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: cephadm 2021-07-31T03:29:33.531839+0000 mgr.x (mgr.24542) 4391 : cephadm 2021-07-31T03:29:34.842 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:29:34.843 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:29:34.843 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:29:34.843 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: yield 2021-07-31T03:29:34.843 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:29:34.844 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:29:34.844 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:29:34.845 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:29:34.845 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:29:34.845 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:29:34.846 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:29:34.846 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:29:34.847 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:29:34.847 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: task.result() 2021-07-31T03:29:34.848 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:29:34.848 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: 2021-07-31T03:29:34.848 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:29:34.849 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: 2021-07-31T03:29:34.849 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:29:34.850 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:29:34.850 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:29:34.850 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:29:34.850 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:29:34.850 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:29:34.851 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:29:34.851 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:29:34.851 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:29:34.851 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:29:34.852 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:29:34.852 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:29:34.852 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:29:34.852 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:29:34.852 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:29:34.853 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:29:34.853 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:29:34.853 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:29:34.853 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: raise 2021-07-31T03:29:34.853 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:29:34.854 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:29:34.854 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:29:34.854 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:29:34.854 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:29:34.855 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:29:34.855 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: task.result() 2021-07-31T03:29:34.855 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:29:34.855 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: cephadm 2021-07-31T03:29:33.531839+0000 mgr.x (mgr.24542) 4391 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:29:34.856 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:29:34.856 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:29:34.856 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: yield 2021-07-31T03:29:34.856 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:29:34.857 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:29:34.857 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:29:34.857 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:29:34.857 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:29:34.857 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:29:34.858 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:29:34.858 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:29:34.858 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:29:34.858 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: task.result() 2021-07-31T03:29:34.858 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:29:34.859 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: 2021-07-31T03:29:34.859 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:29:34.859 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: 2021-07-31T03:29:34.859 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:29:34.859 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:29:34.860 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:29:34.860 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:29:34.860 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:29:34.860 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:29:34.861 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:29:34.861 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:29:34.861 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:29:34.861 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:29:34.861 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:29:34.862 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:29:34.862 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:29:34.862 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:29:34.862 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:29:34.863 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:29:34.863 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:29:34.863 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:29:34.863 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: raise 2021-07-31T03:29:34.863 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:29:34.864 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:29:34.864 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:29:34.864 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:29:34.864 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:34 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:29:36.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:36 smithi154 conmon[34553]: cluster 2021-07-31T03:29:35.257255+0000 mgr.x (mgr.24542) 4392 : cluster [DBG] pgmap v5078: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:36.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:36 smithi045 conmon[39718]: cluster 2021-07-31T03:29:35.257255+0000 mgr.x ( 2021-07-31T03:29:36.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:36 smithi045 conmon[39718]: mgr.24542) 4392 : cluster [DBG] pgmap v5078: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:36.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:36 smithi045 conmon[33655]: cluster 2021-07-31T03:29:35.257255+0000 mgr.x (mgr.24542) 4392 : cluster 2021-07-31T03:29:36.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:36 smithi045 conmon[33655]: [DBG] pgmap v5078: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:38.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:38 smithi154 conmon[34553]: cluster 2021-07-31T03:29:37.257644+0000 mgr.x (mgr.24542) 4393 : cluster [DBG] pgmap v5079: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:38.774 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:38 smithi045 conmon[39718]: cluster 2021-07-31T03:29:37.257644+0000 mgr.x (mgr.24542) 4393 : 2021-07-31T03:29:38.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:38 smithi045 conmon[39718]: cluster [DBG] pgmap v5079: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:38.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:38 smithi045 conmon[33655]: cluster 2021-07-31T03:29:37.257644 2021-07-31T03:29:38.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:38 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 4393 : cluster [DBG] pgmap v5079: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:40.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:40 smithi154 conmon[34553]: cluster 2021-07-31T03:29:39.258052+0000 mgr.x (mgr.24542) 4394 : cluster [DBG] pgmap v5080: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:40.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:40 smithi045 conmon[33655]: cluster 2021-07-31T03:29:39.258052+0000 mgr.x (mgr.24542) 4394 : cluster [DBG] 2021-07-31T03:29:40.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:40 smithi045 conmon[33655]: pgmap v5080: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:40.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:40 smithi045 conmon[39718]: cluster 2021-07-31T03:29:39.258052+0000 mgr.x ( 2021-07-31T03:29:40.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:40 smithi045 conmon[39718]: mgr.24542) 4394 : cluster [DBG] pgmap v5080: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:42.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:42 smithi154 conmon[34553]: cluster 2021-07-31T03:29:41.258459+0000 mgr.x (mgr.24542) 4395 : cluster [DBG] pgmap v5081: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:42.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:42 smithi045 conmon[39718]: cluster 2021-07-31T03:29:41.258459+0000 mgr.x (mgr.24542) 4395 : cluster [DBG] pgmap v5081: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:42.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:42 smithi045 conmon[33655]: cluster 2021-07-31T03:29:41. 2021-07-31T03:29:42.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:42 smithi045 conmon[33655]: 258459+0000 mgr.x (mgr.24542) 4395 : cluster [DBG] pgmap v5081: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:44.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:44 smithi154 conmon[34553]: cluster 2021-07-31T03:29:43.258802+0000 mgr.x (mgr.24542) 4396 : cluster [DBG] pgmap v5082: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:44.774 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:44 smithi045 conmon[39718]: cluster 2021-07-31T03:29:43.258802+0000 mgr.x (mgr.24542) 4396 : cluster [DBG] pgmap v5082: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:44.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:44 smithi045 conmon[33655]: cluster 2021-07-31T03:29:43.258802+0000 mgr.x (mgr.24542) 4396 : cluster 2021-07-31T03:29:44.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:44 smithi045 conmon[33655]: [DBG] pgmap v5082: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:46.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:46 smithi154 conmon[34553]: cluster 2021-07-31T03:29:45.259336+0000 mgr.x (mgr.24542) 4397 : cluster [DBG] pgmap v5083: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:46.774 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:46 smithi045 conmon[33655]: cluster 2021-07-31T03:29:45.259336+0000 mgr.x (mgr.24542) 4397 : cluster [DBG] pgmap v5083: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:46.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:46 smithi045 conmon[39718]: cluster 2021-07-31T03:29:45.259336+0000 mgr.x (mgr.24542) 4397 : cluster 2021-07-31T03:29:46.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:46 smithi045 conmon[39718]: [DBG] pgmap v5083: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:48.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:48 smithi154 conmon[34553]: cluster 2021-07-31T03:29:47.259693+0000 mgr.x (mgr.24542) 4398 : cluster [DBG] 2021-07-31T03:29:48.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:48 smithi154 conmon[34553]: pgmap v5084: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:48.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:48 smithi045 conmon[33655]: cluster 2021-07-31T03:29:47.259693+0000 mgr.x (mgr.24542) 4398 : cluster [DBG] 2021-07-31T03:29:48.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:48 smithi045 conmon[33655]: pgmap v5084: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:48.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:48 smithi045 conmon[39718]: cluster 2021-07-31T03:29:47.259693+0000 mgr.x (mgr.24542) 4398 : cluster [DBG] 2021-07-31T03:29:48.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:48 smithi045 conmon[39718]: pgmap v5084: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:50.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:50 smithi154 conmon[34553]: cluster 2021-07-31T03:29:49.260160+0000 mgr.x (mgr.24542) 4399 : cluster [DBG] pgmap v5085: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:50.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:50 smithi045 conmon[33655]: cluster 2021-07-31T03:29:49.260160+0000 mgr.x (mgr.24542) 4399 : cluster 2021-07-31T03:29:50.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:50 smithi045 conmon[33655]: [DBG] pgmap v5085: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:50.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:50 smithi045 conmon[39718]: cluster 2021-07-31T03:29:49.260160+0000 mgr.x (mgr.24542) 4399 : cluster [DBG] pgmap v5085: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:50.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:50 smithi045 conmon[39718]: 2021-07-31T03:29:52.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:52 smithi154 conmon[34553]: cluster 2021-07-31T03:29:51.260498+0000 mgr.x (mgr.24542) 4400 : cluster [DBG] pgmap v5086: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:52.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:52 smithi045 conmon[33655]: cluster 2021-07-31T03:29:51.260498+0000 mgr.x (mgr.24542) 4400 2021-07-31T03:29:52.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:52 smithi045 conmon[33655]: : cluster [DBG] pgmap v5086: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:52.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:52 smithi045 conmon[39718]: cluster 2021-07-31T03:29:51.260498+0000 mgr.x (mgr.24542) 4400 : 2021-07-31T03:29:52.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:52 smithi045 conmon[39718]: cluster [DBG] pgmap v5086: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:54.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:54 smithi154 conmon[34553]: cluster 2021-07-31T03:29:53.261051+0000 mgr.x (mgr.24542) 4401 : cluster [DBG] pgmap v5087: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:54.774 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:54 smithi045 conmon[33655]: cluster 2021-07-31T03:29:53.261051+0000 mgr.x (mgr.24542 2021-07-31T03:29:54.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:54 smithi045 conmon[33655]: ) 4401 : cluster [DBG] pgmap v5087: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:54.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:54 smithi045 conmon[39718]: cluster 2021-07-31T03:29:53 2021-07-31T03:29:54.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:54 smithi045 conmon[39718]: .261051+0000 mgr.x (mgr.24542) 4401 : cluster [DBG] pgmap v5087: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:56.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:56 smithi154 conmon[34553]: cluster 2021-07-31T03:29:55.261560+0000 mgr.x (mgr.24542) 4402 : cluster [DBG] pgmap v5088: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:56.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:56 smithi045 conmon[39718]: cluster 2021-07-31T03:29:55.261560+0000 mgr.x ( 2021-07-31T03:29:56.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:56 smithi045 conmon[39718]: mgr.24542) 4402 : cluster [DBG] pgmap v5088: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:56.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:56 smithi045 conmon[33655]: cluster 2021-07-31T03:29:55.261560+0000 mgr.x (mgr.24542) 4402 : 2021-07-31T03:29:56.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:56 smithi045 conmon[33655]: cluster [DBG] pgmap v5088: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:58.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:29:58 smithi154 conmon[34553]: cluster 2021-07-31T03:29:57.262097+0000 mgr.x (mgr.24542) 4403 : cluster [DBG] pgmap v5089: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:58.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:58 smithi045 conmon[39718]: cluster 2021-07-31T03:29:57.262097+0000 mgr.x (mgr 2021-07-31T03:29:58.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:29:58 smithi045 conmon[39718]: .24542) 4403 : cluster [DBG] pgmap v5089: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:29:58.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:58 smithi045 conmon[33655]: cluster 2021-07-31T03:29:57.262097+0000 mgr.x (mgr.24542) 4403 : cluster 2021-07-31T03:29:58.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:29:58 smithi045 conmon[33655]: [DBG] pgmap v5089: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:00.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:00 smithi154 conmon[34553]: cluster 2021-07-31T03:29:59.262570+0000 mgr.x (mgr.24542) 4404 : cluster [DBG] pgmap v5090: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:00.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:00 smithi154 conmon[34553]: cluster 2021-07-31T03:30:00.000182+0000 mon.a (mon.0) 3313 : cluster [INF] overall HEALTH_OK 2021-07-31T03:30:00.774 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:00 smithi045 conmon[39718]: cluster 2021-07-31T03:29:59.262570+0000 mgr.x ( 2021-07-31T03:30:00.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:00 smithi045 conmon[39718]: mgr.24542) 4404 : cluster [DBG] pgmap v5090: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:00.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:00 smithi045 conmon[39718]: cluster 2021-07-31 2021-07-31T03:30:00.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:00 smithi045 conmon[39718]: T03:30:00.000182+0000 mon.a (mon.0) 3313 : cluster [INF] overall HEALTH_OK 2021-07-31T03:30:00.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:00 smithi045 conmon[33655]: cluster 2021-07-31T03:29:59.262570+0000 2021-07-31T03:30:00.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:00 smithi045 conmon[33655]: mgr.x (mgr.24542) 4404 : cluster [DBG] pgmap v5090: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:00.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:00 smithi045 conmon[33655]: cluster 2021-07-31T03:30:00.000182+0000 2021-07-31T03:30:00.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:00 smithi045 conmon[33655]: mon.a (mon.0) 3313 : cluster [INF] overall HEALTH_OK 2021-07-31T03:30:02.268 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:30:02.547 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:30:02.547 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 104m ago 105m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:30:02.548 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 102m ago 105m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:30:02.548 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 102m ago 108m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:30:02.548 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 104m ago 111m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:30:02.548 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 104m ago 111m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:30:02.549 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 102m ago 109m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:30:02.549 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 104m ago 109m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:30:02.549 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 104m ago 105m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:30:02.549 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 102m ago 105m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:30:02.549 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 104m ago 108m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:30:02.550 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 104m ago 108m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:30:02.550 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 104m ago 108m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:30:02.550 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 104m ago 107m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:30:02.550 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 102m ago 107m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:30:02.550 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 102m ago 107m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:30:02.551 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 102m ago 106m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:30:02.551 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 102m ago 106m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:30:02.551 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 102m ago 106m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:30:02.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:02 smithi154 conmon[34553]: cluster 2021-07-31T03:30:01.263025+0000 mgr.x (mgr.24542) 4405 : cluster [DBG] pgmap v5091: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:02.774 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:02 smithi045 conmon[33655]: cluster 2021-07-31T03:30:01.263025+0000 mgr.x (mgr.24542) 4405 2021-07-31T03:30:02.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:02 smithi045 conmon[33655]: : cluster [DBG] pgmap v5091: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:02.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:02 smithi045 conmon[39718]: cluster 2021-07-31T03:30:01.263025+0000 2021-07-31T03:30:02.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:02 smithi045 conmon[39718]: mgr.x (mgr.24542) 4405 : cluster [DBG] pgmap v5091: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:02.883 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:30:02.883 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:30:02.883 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:30:02.883 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:30:02.884 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:30:02.884 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:30:02.884 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:30:02.884 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:30:02.885 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:30:02.885 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:30:02.885 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:30:02.885 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:30:02.886 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:30:02.886 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:30:02.886 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:30:02.886 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:30:02.886 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:30:03.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:03 smithi154 conmon[34553]: audit 2021-07-31T03:30:02.256699+0000 mgr.x (mgr.24542) 4406 : audit [DBG] from='client.29572 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:30:03.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:03 smithi154 conmon[34553]: audit 2021-07-31T03:30:02.540448+0000 mgr.x (mgr.24542) 4407 : audit [DBG] from='client.19638 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:30:03.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:03 smithi154 conmon[34553]: audit 2021-07-31T03:30:02.883975+0000 mon.a (mon.0) 3314 : audit [DBG] from='client.? 172.21.15.45:0/3396558053' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:30:03.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:03 smithi045 conmon[39718]: audit 2021-07-31T03:30:02.256699+0000 mgr.x 2021-07-31T03:30:03.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:03 smithi045 conmon[39718]: (mgr.24542) 4406 : audit [DBG] from='client.29572 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:30:03.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:03 smithi045 conmon[39718]: audit 2021-07-31T03:30:03.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:03 smithi045 conmon[39718]: 2021-07-31T03:30:02.540448+0000 mgr.x (mgr.24542 2021-07-31T03:30:03.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:03 smithi045 conmon[39718]: ) 4407 : audit [DBG] from='client.19638 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:30:03.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:03 smithi045 conmon[39718]: audit 2021- 2021-07-31T03:30:03.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:03 smithi045 conmon[39718]: 07-31T03:30:02.883975+0000 mon.a (mon.0) 2021-07-31T03:30:03.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:03 smithi045 conmon[39718]: 3314 : audit [DBG] from='client.? 172.21.15.45:0/3396558053' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:30:03.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:03 smithi045 conmon[33655]: audit 2021-07-31T03:30:02.256699+0000 mgr.x (mgr.24542 2021-07-31T03:30:03.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:03 smithi045 conmon[33655]: ) 4406 : audit [DBG] from='client.29572 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:30:03.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:03 smithi045 conmon[33655]: audit 2021-07-31T03:30:02.540448+0000 mgr.x (mgr.24542) 4407 : audit [DBG] from='client.19638 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:30:03.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:03 smithi045 conmon[33655]: audit 2021-07-31T03 2021-07-31T03:30:03.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:03 smithi045 conmon[33655]: :30:02.883975+0000 mon.a (mon.0) 3314 : audit [DBG] from='client.? 172.21.15.45:0/3396558053' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:30:04.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:04 smithi154 conmon[34553]: cluster 2021-07-31T03:30:03.263355+0000 mgr.x (mgr.24542) 4408 : cluster [DBG] pgmap v5092: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:04.774 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:04 smithi045 conmon[39718]: cluster 2021-07-31T03:30:03.263355+0000 mgr.x ( 2021-07-31T03:30:04.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:04 smithi045 conmon[39718]: mgr.24542) 4408 : cluster [DBG] pgmap v5092: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:04.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:04 smithi045 conmon[33655]: cluster 2021-07-31T03:30:03.263355 2021-07-31T03:30:04.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:04 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 4408 : cluster [DBG] pgmap v5092: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:06.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:06 smithi154 conmon[34553]: cluster 2021-07-31T03:30:05.263852+0000 mgr.x (mgr.24542) 4409 : cluster [DBG] pgmap v5093: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:06.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:06 smithi045 conmon[39718]: cluster 2021-07-31T03:30:05.263852+0000 mgr.x (mgr.24542) 4409 : cluster [DBG] pgmap v5093: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:06.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:06 smithi045 conmon[33655]: cluster 2021-07-31T03:30:05.263852+0000 mgr.x (mgr.24542) 4409 : 2021-07-31T03:30:06.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:06 smithi045 conmon[33655]: cluster [DBG] pgmap v5093: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:08.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:08 smithi154 conmon[34553]: cluster 2021-07-31T03:30:07.264333+0000 mgr.x (mgr.24542) 4410 : cluster 2021-07-31T03:30:08.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:08 smithi154 conmon[34553]: [DBG] pgmap v5094: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:08.774 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:08 smithi045 conmon[39718]: cluster 2021-07-31T03:30:07.264333 2021-07-31T03:30:08.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:08 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 4410 : cluster [DBG] pgmap v5094: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:08.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:08 smithi045 conmon[33655]: cluster 2021-07-31T03:30:07.264333+0000 mgr.x ( 2021-07-31T03:30:08.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:08 smithi045 conmon[33655]: mgr.24542) 4410 : cluster [DBG] pgmap v5094: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:10.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:10 smithi154 conmon[34553]: cluster 2021-07-31T03:30:09.264841+0000 mgr.x (mgr.24542) 4411 : cluster [DBG] pgmap v5095: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:10.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:10 smithi045 conmon[33655]: cluster 2021-07-31T03:30:09.264841+0000 mgr.x (mgr.24542) 4411 : cluster [DBG] 2021-07-31T03:30:10.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:10 smithi045 conmon[33655]: pgmap v5095: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:10.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:10 smithi045 conmon[39718]: cluster 2021-07-31T03:30:09.264841 2021-07-31T03:30:10.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:10 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 4411 : cluster [DBG] pgmap v5095: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:12.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:12 smithi154 conmon[34553]: cluster 2021-07-31T03:30:11.265156+0000 mgr.x (mgr.24542) 4412 : cluster [DBG] pgmap v5096: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:12.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:12 smithi154 conmon[34553]: 2021-07-31T03:30:12.774 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:12 smithi045 conmon[39718]: cluster 2021-07-31T03:30:11.265156 2021-07-31T03:30:12.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:12 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 4412 : cluster [DBG] pgmap v5096: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:12.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:12 smithi045 conmon[33655]: cluster 2021-07-31T03:30:11.265156+0000 2021-07-31T03:30:12.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:12 smithi045 conmon[33655]: mgr.x (mgr.24542) 4412 : cluster [DBG] pgmap v5096: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:14.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:14 smithi154 conmon[34553]: cluster 2021-07-31T03:30:13.265518+0000 mgr.x (mgr.24542) 4413 : cluster [DBG] pgmap v5097: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:14.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:14 smithi045 conmon[39718]: cluster 2021-07-31T03:30:13.265518+0000 mgr.x (mgr.24542 2021-07-31T03:30:14.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:14 smithi045 conmon[39718]: ) 4413 : cluster [DBG] pgmap v5097: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:14.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:14 smithi045 conmon[33655]: cluster 2021-07-31T03:30:13.265518 2021-07-31T03:30:14.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:14 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 4413 : cluster [DBG] pgmap v5097: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:16.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:16 smithi154 conmon[34553]: cluster 2021-07-31T03:30:15.266191+0000 mgr.x (mgr.24542) 4414 : cluster [DBG] pgmap v5098: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:16.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:16 smithi154 conmon[34553]: 2021-07-31T03:30:16.774 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:16 smithi045 conmon[39718]: cluster 2021-07-31T03:30:15.266191+0000 mgr.x (mgr.24542) 4414 : cluster [DBG] pgmap v5098: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:16.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:16 smithi045 conmon[33655]: cluster 2021-07-31T03:30:15.266191+0000 mgr.x 2021-07-31T03:30:16.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:16 smithi045 conmon[33655]: (mgr.24542) 4414 : cluster [DBG] pgmap v5098: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:18.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:18 smithi154 conmon[34553]: cluster 2021-07-31T03:30:17.266573+0000 mgr.x (mgr.24542) 4415 : cluster [DBG] pgmap v5099: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:18.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:18 smithi154 conmon[34553]: 2021-07-31T03:30:18.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:18 smithi045 conmon[33655]: cluster 2021-07-31T03:30:17.266573+0000 mgr.x (mgr 2021-07-31T03:30:18.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:18 smithi045 conmon[33655]: .24542) 4415 : cluster [DBG] pgmap v5099: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:18.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:18 smithi045 conmon[39718]: cluster 2021-07-31T03:30:17.266573+0000 2021-07-31T03:30:18.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:18 smithi045 conmon[39718]: mgr.x (mgr.24542) 4415 : cluster [DBG] pgmap v5099: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:20.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:20 smithi154 conmon[34553]: cluster 2021-07-31T03:30:19.267192+0000 mgr.x (mgr.24542) 4416 : cluster [DBG] pgmap v5100: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:20.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:20 smithi045 conmon[39718]: cluster 2021-07-31T03:30:19.267192+0000 mgr.x (mgr 2021-07-31T03:30:20.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:20 smithi045 conmon[39718]: .24542) 4416 : cluster [DBG] pgmap v5100: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:20.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:20 smithi045 conmon[33655]: cluster 2021-07-31T03:30:19.267192+0000 mgr.x (mgr.24542) 4416 : 2021-07-31T03:30:20.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:20 smithi045 conmon[33655]: cluster [DBG] pgmap v5100: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:22.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:22 smithi154 conmon[34553]: cluster 2021-07-31T03:30:21.267599+0000 mgr.x (mgr.24542) 4417 : cluster [DBG] pgmap v5101: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:22.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:22 smithi045 conmon[39718]: cluster 2021-07-31T03:30:21.267599+0000 mgr.x ( 2021-07-31T03:30:22.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:22 smithi045 conmon[39718]: mgr.24542) 4417 : cluster [DBG] pgmap v5101: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:22.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:22 smithi045 conmon[33655]: cluster 2021-07-31T03:30:21.267599+0000 mgr.x (mgr.24542) 4417 : 2021-07-31T03:30:22.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:22 smithi045 conmon[33655]: cluster [DBG] pgmap v5101: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:24.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:24 smithi154 conmon[34553]: cluster 2021-07-31T03:30:23.268107+0000 mgr.x (mgr.24542) 4418 : cluster [DBG] pgmap v5102: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:24.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:24 smithi154 conmon[34553]: 2021-07-31T03:30:24.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:24 smithi045 conmon[39718]: cluster 2021-07-31T03:30:23.268107+0000 mgr.x (mgr.24542) 4418 : cluster [DBG] pgmap v5102: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:24.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:24 smithi045 conmon[33655]: cluster 2021-07-31T03:30:23.268107+0000 mgr.x (mgr.24542) 4418 : cluster [DBG] pgmap v5102: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:24.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:24 smithi045 conmon[33655]: 2021-07-31T03:30:26.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:26 smithi154 conmon[34553]: cluster 2021-07-31T03:30:25.268521+0000 mgr.x (mgr.24542) 4419 : cluster [DBG] pgmap v5103: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:26.774 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:26 smithi045 conmon[39718]: cluster 2021-07-31T03:30:25.268521+0000 mgr.x (mgr.24542) 4419 : cluster [DBG] pgmap v5103: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:26.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:26 smithi045 conmon[33655]: cluster 2021-07-31T03:30:25.268521+0000 mgr.x (mgr.24542 2021-07-31T03:30:26.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:26 smithi045 conmon[33655]: ) 4419 : cluster [DBG] pgmap v5103: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:28.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:28 smithi154 conmon[34553]: cluster 2021-07-31T03:30:27.269017+0000 mgr.x (mgr.24542) 4420 : cluster [DBG] pgmap v5104: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:28.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:28 smithi154 conmon[34553]: audit 2021-07-31T03:30:28.344392+0000 2021-07-31T03:30:28.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:28 smithi154 conmon[34553]: mon.b (mon.2) 590 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:30:28.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:28 smithi154 conmon[34553]: audit 2021-07-31T03:30:28.344693+0000 mon.b (mon.2) 591 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:30:28.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:28 smithi154 conmon[34553]: audit 2021-07-31T03:30:28.345213+0000 mon.a (mon.0) 3315 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:30:28.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:28 smithi154 conmon[34553]: audit 2021-07-31T03:30:28.345824+0000 2021-07-31T03:30:28.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:28 smithi154 conmon[34553]: mon.a (mon.0) 3316 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:30:28.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:28 smithi154 conmon[34553]: 2021-07-31T03:30:28.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:28 smithi045 conmon[33655]: cluster 2021-07-31T03:30:27.269017+0000 2021-07-31T03:30:28.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:28 smithi045 conmon[33655]: mgr.x (mgr.24542) 4420 : cluster [DBG] pgmap v5104: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:28.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:28 smithi045 conmon[33655]: audit 2021-07-31T03:30:28.344392+0000 mon.b (mon.2) 590 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:30:28.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:28 smithi045 conmon[33655]: audit 2021-07-31T03:30 2021-07-31T03:30:28.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:28 smithi045 conmon[33655]: :28.344693+0000 mon.b (mon 2021-07-31T03:30:28.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:28 smithi045 conmon[33655]: .2) 591 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:30:28.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:28 smithi045 conmon[33655]: audit 2021-07-31T03:30:28.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:28 smithi045 conmon[33655]: 2021-07-31T03:30:28. 2021-07-31T03:30:28.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:28 smithi045 conmon[33655]: 345213+0000 mon.a (mon.0) 3315 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:30:28.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:28 smithi045 conmon[33655]: audit 2021- 2021-07-31T03:30:28.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:28 smithi045 conmon[33655]: 07-31T03:30:28.345824+0000 mon.a (mon.0) 3316 : audit 2021-07-31T03:30:28.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:28 smithi045 conmon[33655]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:30:28.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:28 smithi045 conmon[39718]: cluster 2021-07-31T03:30 2021-07-31T03:30:28.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:28 smithi045 conmon[39718]: :27.269017+0000 mgr.x (mgr.24542) 4420 : cluster [DBG] pgmap v5104: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:28.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:28 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T03:30:28.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:28 smithi045 conmon[39718]: 03:30:28.344392+0000 2021-07-31T03:30:28.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:28 smithi045 conmon[39718]: mon.b (mon.2) 590 : 2021-07-31T03:30:28.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:28 smithi045 conmon[39718]: audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:30:28.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:28 smithi045 conmon[39718]: audit 2021- 2021-07-31T03:30:28.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:28 smithi045 conmon[39718]: 07-31T03:30:28.344693 2021-07-31T03:30:28.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:28 smithi045 conmon[39718]: +0000 mon.b (mon.2) 591 2021-07-31T03:30:28.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:28 smithi045 conmon[39718]: : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:30:28.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:28 smithi045 conmon[39718]: audit 2021 2021-07-31T03:30:28.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:28 smithi045 conmon[39718]: -07-31T03:30: 2021-07-31T03:30:28.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:28 smithi045 conmon[39718]: 28.345213+0000 mon.a (mon 2021-07-31T03:30:28.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:28 smithi045 conmon[39718]: .0) 3315 : audit [INF] 2021-07-31T03:30:28.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:28 smithi045 conmon[39718]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:30:28.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:28 smithi045 conmon[39718]: audit 2021 2021-07-31T03:30:28.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:28 smithi045 conmon[39718]: -07-31T03:30:28. 2021-07-31T03:30:28.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:28 smithi045 conmon[39718]: 345824+0000 mon.a (mon.0 2021-07-31T03:30:28.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:28 smithi045 conmon[39718]: ) 3316 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:30:30.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:30 smithi154 conmon[34553]: cluster 2021-07-31T03:30:29.269553+0000 mgr.x (mgr.24542) 4421 : cluster [DBG] pgmap v5105: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:30.774 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:30 smithi045 conmon[39718]: cluster 2021-07-31T03:30:29.269553+0000 mgr.x (mgr.24542) 4421 : cluster [DBG] pgmap v5105: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:30.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:30 smithi045 conmon[33655]: cluster 2021-07-31T03:30:29.269553+0000 mgr.x (mgr.24542) 4421 2021-07-31T03:30:30.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:30 smithi045 conmon[33655]: : cluster [DBG] pgmap v5105: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:32.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:32 smithi154 conmon[34553]: cluster 2021-07-31T03:30:31.270022+0000 mgr.x (mgr.24542) 4422 : cluster [DBG] pgmap v5106: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:32.774 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:32 smithi045 conmon[39718]: cluster 2021-07-31T03:30:31.270022+0000 mgr.x (mgr 2021-07-31T03:30:32.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:32 smithi045 conmon[39718]: .24542) 4422 : cluster [DBG] pgmap v5106: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:32.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:32 smithi045 conmon[33655]: cluster 2021-07-31T03:30:31.270022+0000 2021-07-31T03:30:32.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:32 smithi045 conmon[33655]: mgr.x (mgr.24542) 4422 : cluster [DBG] pgmap v5106: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:33.198 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:30:33.479 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:30:33.480 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 105m ago 106m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:30:33.480 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 103m ago 105m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:30:33.480 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 103m ago 109m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:30:33.481 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 105m ago 111m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:30:33.481 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 105m ago 111m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:30:33.481 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 103m ago 109m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:30:33.481 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 105m ago 109m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:30:33.482 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 105m ago 106m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:30:33.482 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 103m ago 106m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:30:33.482 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 105m ago 109m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:30:33.482 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 105m ago 108m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:30:33.482 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 105m ago 108m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:30:33.483 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 105m ago 108m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:30:33.483 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 103m ago 107m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:30:33.483 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 103m ago 107m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:30:33.483 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 103m ago 107m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:30:33.483 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 103m ago 106m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:30:33.484 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 103m ago 106m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:30:33.815 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:30:33.816 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:30:33.816 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:30:33.816 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:30:33.816 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:30:33.817 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:30:33.817 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:30:33.817 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:30:33.817 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:30:33.818 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:30:33.818 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:30:33.818 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:30:33.818 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:30:33.819 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:30:33.819 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:30:33.819 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:30:33.819 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:30:34.002 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: debug 2021-07-31T03:30:33.576+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:30:34.003 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:30:34.003 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:30:34.003 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: yield 2021-07-31T03:30:34.003 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:30:34.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:30:34.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:30:34.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:30:34.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:30:34.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:30:34.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:30:34.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:30:34.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:30:34.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: task.result() 2021-07-31T03:30:34.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:30:34.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: debug 2021-07-31T03:30:33.588+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:30:34.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:30:34.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:30:34.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: yield 2021-07-31T03:30:34.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:30:34.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:30:34.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:30:34.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:30:34.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:30:34.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:30:34.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:30:34.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:30:34.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:30:34.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: task.result() 2021-07-31T03:30:34.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:30:34.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: debug 2021-07-31T03:30:33.613+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:30:34.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:30:34.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:30:34.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: yield 2021-07-31T03:30:34.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:30:34.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:30:34.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:30:34.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:30:34.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:30:34.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:30:34.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:30:34.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:30:34.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:30:34.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: task.result() 2021-07-31T03:30:34.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:30:34.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: debug 2021-07-31T03:30:33.614+0000 7fcb9115c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:30:34.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:30:34.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:30:34.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: yield 2021-07-31T03:30:34.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:30:34.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:30:34.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:30:34.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:30:34.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:30:34.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:30:34.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:30:34.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:30:34.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:30:34.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: task.result() 2021-07-31T03:30:34.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:30:34.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: 2021-07-31T03:30:34.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:30:34.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: 2021-07-31T03:30:34.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:30:34.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:30:34.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:30:34.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:30:34.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:30:34.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:30:34.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:30:34.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:30:34.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:30:34.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:30:34.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:30:34.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:30:34.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:30:34.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:30:34.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:30:34.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:30:34.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:30:34.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:30:34.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: raise 2021-07-31T03:30:34.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:30:34.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:30:34.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:30:34.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:30:34.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:30:34.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: debug 2021-07-31T03:30:33.622+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:30:34.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:30:34.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:30:34.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: yield 2021-07-31T03:30:34.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:30:34.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:30:34.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:30:34.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:30:34.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:30:34.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:30:34.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:30:34.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:30:34.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:30:34.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: task.result() 2021-07-31T03:30:34.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:30:34.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: debug 2021-07-31T03:30:33.622+0000 7fcb9095b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:30:34.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:30:34.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:30:34.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: yield 2021-07-31T03:30:34.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:30:34.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:30:34.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:30:34.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:30:34.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:30:34.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:30:34.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:30:34.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:30:34.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:30:34.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: task.result() 2021-07-31T03:30:34.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:30:34.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: 2021-07-31T03:30:34.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:30:34.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: 2021-07-31T03:30:34.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:30:34.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:30:34.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:30:34.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:30:34.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:30:34.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:30:34.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:30:34.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:30:34.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:30:34.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:30:34.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:30:34.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:30:34.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:30:34.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:30:34.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:30:34.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:30:34.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:30:34.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:30:34.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: raise 2021-07-31T03:30:34.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:30:34.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:30:34.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:30:34.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:30:34.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:30:33 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:30:34.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: audit 2021-07-31T03:30:33.186863+0000 mgr.x (mgr.24542) 4423 : audit 2021-07-31T03:30:34.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: [DBG] from='client.29590 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:30:34.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: cluster 2021-07-31T03:30:33.270428+0000 mgr.x (mgr.24542) 4424 : cluster [DBG] pgmap v5107: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:34.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: audit 2021-07-31T03:30:33.472142+0000 mgr.x (mgr.24542) 4425 2021-07-31T03:30:34.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: : audit [DBG] from='client.19656 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:30:34.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: audit 2021-07- 2021-07-31T03:30:34.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: 31T03:30:33.532561+0000 mon.b (mon. 2021-07-31T03:30:34.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: 2) 592 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:30:34.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: audit 2021-07-31T03:30:34.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: 2021-07-31T03:30:33.533991+0000 mon.b ( 2021-07-31T03:30:34.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: mon.2) 593 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:30:34.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: audit 2021-07-31T03:30:33.534846 2021-07-31T03:30:34.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: +0000 mon.b (mon.2) 594 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:30:34.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: cephadm 2021-07-31T 2021-07-31T03:30:34.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: 03:30:33.535597+0000 mgr.x (mgr.24542) 4426 : cephadm [WRN] 2021-07-31T03:30:34.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:30:34.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: cephadm 2021-07-31T 2021-07-31T03:30:34.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: 03:30:33.577350+0000 mgr.x (mgr.24542) 2021-07-31T03:30:34.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: 4427 : cephadm [ERR] 2021-07-31T03:30:34.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:30:34.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:30:34.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: yield 2021-07-31T03:30:34.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:30:34.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:30:34.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:30:34.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:30:34.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:30:34.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:30:34.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:30:34.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:30:34.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:30:34.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: task.result() 2021-07-31T03:30:34.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:30:34.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: cephadm 2021-07-31T03:30:33.577784+0000 mgr.x (mgr.24542) 4428 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:30:34.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: cephadm 2021-07-31T03:30: 2021-07-31T03:30:34.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: 33.589372+0000 mgr.x (mgr.24542) 4429 : cephadm [ERR] 2021-07-31T03:30:34.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:30:34.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:30:34.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: yield 2021-07-31T03:30:34.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:30:34.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:30:34.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:30:34.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:30:34.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:30:34.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:30:34.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:30:34.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:30:34.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:30:34.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: task.result() 2021-07-31T03:30:34.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:30:34.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: cephadm 2021-07-31T03:30:33.590052+0000 mgr.x (mgr.24542) 4430 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:30:34.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: cephadm 2021-07-31T03:30:33.614235+0000 mgr.x (mgr 2021-07-31T03:30:34.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: .24542) 4431 : cephadm [ERR] 2021-07-31T03:30:34.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:30:34.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:30:34.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: yield 2021-07-31T03:30:34.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:30:34.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:30:34.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:30:34.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:30:34.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:30:34.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:30:34.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:30:34.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:30:34.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:30:34.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: task.result() 2021-07-31T03:30:34.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:30:34.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: cephadm 2021-07-31T03:30:33.615441+0000 mgr.x (mgr.24542) 4432 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:30:34.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:30:34.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:30:34.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: yield 2021-07-31T03:30:34.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:30:34.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:30:34.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:30:34.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:30:34.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:30:34.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:30:34.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:30:34.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:30:34.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:30:34.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: task.result() 2021-07-31T03:30:34.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:30:34.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: 2021-07-31T03:30:34.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:30:34.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: 2021-07-31T03:30:34.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:30:34.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:30:34.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:30:34.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:30:34.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:30:34.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:30:34.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:30:34.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:30:34.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:30:34.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:30:34.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:30:34.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:30:34.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:30:34.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:30:34.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:30:34.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:30:34.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:30:34.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:30:34.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: raise 2021-07-31T03:30:34.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:30:34.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:30:34.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:30:34.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:30:34.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:30:34.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: cephadm 2021-07-31T03:30:33.623281+0000 mgr.x (mgr.24542) 2021-07-31T03:30:34.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: 4433 : cephadm [ERR] 2021-07-31T03:30:34.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:30:34.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:30:34.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: yield 2021-07-31T03:30:34.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:30:34.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:30:34.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:30:34.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:30:34.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: audit 2021-07-31T03:30:33.186863+0000 mgr.x (mgr.24542) 4423 : audit 2021-07-31T03:30:34.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: [DBG] from='client.29590 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:30:34.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: cluster 2021-07-31T03:30:33.270428+0000 mgr.x (mgr.24542) 4424 : cluster [DBG] pgmap v5107: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:34.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: audit 2021-07-31T03:30:33.472142+0000 mgr.x (mgr.24542) 2021-07-31T03:30:34.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: 4425 : audit [DBG] from='client.19656 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:30:34.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: audit 2021 2021-07-31T03:30:34.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: -07-31T03:30 2021-07-31T03:30:34.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: :33.532561+0000 mon.b 2021-07-31T03:30:34.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: (mon.2) 592 : audit [DBG] 2021-07-31T03:30:34.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:30:34.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: audit 2021-07 2021-07-31T03:30:34.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: -31T03:30:33.533991 2021-07-31T03:30:34.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: +0000 mon.b (mon.2) 593 2021-07-31T03:30:34.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:30:34.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: audit 2021- 2021-07-31T03:30:34.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: 07-31T03:30:33.534846 2021-07-31T03:30:34.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: +0000 mon.b (mon.2 2021-07-31T03:30:34.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: ) 594 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:30:34.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: 2021-07-31T03:30:34.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: cephadm 2021 2021-07-31T03:30:34.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: -07-31T03:30:33. 2021-07-31T03:30:34.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: 535597+0000 mgr.x (mgr.24542 2021-07-31T03:30:34.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: ) 4426 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:30:34.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: 2021-07-31T03:30:34.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: cephadm 2021-07-31T03:30: 2021-07-31T03:30:34.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: 33.577350+0000 mgr.x (mgr.24542) 4427 : cephadm [ERR] 2021-07-31T03:30:34.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: 2021-07-31T03:30:34.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:30:34.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:30:34.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: yield 2021-07-31T03:30:34.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:30:34.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:30:34.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:30:34.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:30:34.788 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:30:34.788 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:30:34.788 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:30:34.788 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:30:34.788 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:30:34.789 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: task.result() 2021-07-31T03:30:34.789 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:30:34.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: audit 2021-07-31T03:30:33.186863+0000 2021-07-31T03:30:34.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: mgr.x (mgr.24542) 4423 : audit [DBG] from='client.29590 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:30:34.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: cluster 2021-07-31T03:30: 2021-07-31T03:30:34.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: 33.270428+0000 mgr.x (mgr.24542) 4424 : cluster 2021-07-31T03:30:34.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: [DBG] pgmap v5107: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:34.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: audit 2021-07-31T03:30:34.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: 2021-07-31T03:30:33.472142+0000 2021-07-31T03:30:34.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: mgr.x (mgr.24542) 4425 : audit [DBG] from='client.19656 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:30:34.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: 2021-07-31T03:30:34.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T03:30:34.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: 03:30:33.532561+0000 mon.b (mon. 2021-07-31T03:30:34.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: 2) 592 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:30:34.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: audit 2021-07-31T03:30:34.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: 2021-07-31T03:30:33.533991+0000 2021-07-31T03:30:34.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: mon.b (mon.2) 593 : audit 2021-07-31T03:30:34.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:30:34.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: audit 2021- 2021-07-31T03:30:34.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: 07-31T03:30:33.534846 2021-07-31T03:30:34.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: +0000 mon.b (mon.2) 2021-07-31T03:30:34.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: 594 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:30:34.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: cephadm 2021 2021-07-31T03:30:34.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: -07-31T03:30:33 2021-07-31T03:30:34.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: .535597+0000 mgr.x (mgr. 2021-07-31T03:30:34.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: 24542) 4426 : cephadm [WRN] 2021-07-31T03:30:34.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:30:34.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: cephadm 2021-07 2021-07-31T03:30:34.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: -31T03:30:33.577350 2021-07-31T03:30:34.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 2021-07-31T03:30:34.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: 4427 : cephadm [ERR] 2021-07-31T03:30:34.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:30:34.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:30:34.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: yield 2021-07-31T03:30:34.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:30:34.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:30:34.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:30:34.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:30:34.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:30:34.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:30:34.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:30:34.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:30:34.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:30:34.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: task.result() 2021-07-31T03:30:34.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:30:34.799 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:30:34.799 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:30:34.799 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:30:34.799 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:30:34.800 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:30:34.800 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: task.result() 2021-07-31T03:30:34.800 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:30:34.800 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: cephadm 2021-07-31T03:30:33.623711+0000 mgr.x (mgr.24542) 4434 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:30:34.800 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:30:34.801 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:30:34.801 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: yield 2021-07-31T03:30:34.801 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:30:34.801 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:30:34.802 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:30:34.802 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:30:34.802 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:30:34.802 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:30:34.802 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:30:34.803 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:30:34.803 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:30:34.803 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: task.result() 2021-07-31T03:30:34.803 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:30:34.804 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: 2021-07-31T03:30:34.804 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:30:34.804 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: 2021-07-31T03:30:34.804 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:30:34.804 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:30:34.805 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:30:34.805 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:30:34.805 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:30:34.805 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:30:34.805 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:30:34.806 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:30:34.806 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:30:34.806 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:30:34.806 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:30:34.807 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:30:34.807 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:30:34.807 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:30:34.807 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:30:34.807 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:30:34.808 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:30:34.808 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:30:34.808 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: raise 2021-07-31T03:30:34.808 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:30:34.808 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:30:34.809 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:30:34.809 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:30:34.809 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:30:34.809 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:34 smithi154 conmon[34553]: audit 2021-07-31T03:30:33.816589+0000 mon.c (mon.1) 210 : audit [DBG] from='client.? 172.21.15.45:0/3460258465' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:30:34.810 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: cephadm 2021-07-31T03:30:33.577784+0000 mgr.x (mgr.24542) 4428 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:30:34.810 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: cephadm 2021-07-31T 2021-07-31T03:30:34.810 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: 03:30:33.589372+0000 mgr.x ( 2021-07-31T03:30:34.811 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: mgr.24542) 4429 : cephadm 2021-07-31T03:30:34.811 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: [ERR] 2021-07-31T03:30:34.811 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:30:34.811 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:30:34.812 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: yield 2021-07-31T03:30:34.812 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:30:34.812 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:30:34.812 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:30:34.812 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:30:34.813 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:30:34.813 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:30:34.813 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:30:34.813 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:30:34.814 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:30:34.814 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: task.result() 2021-07-31T03:30:34.814 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:30:34.814 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: cephadm 2021-07-31T03:30:33.590052+0000 mgr.x (mgr.24542) 4430 : cephadm 2021-07-31T03:30:34.815 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:30:34.815 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: cephadm 2021-07-31T03:30:34.815 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: 2021-07-31T03:30: 2021-07-31T03:30:34.815 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: 33.614235+0000 mgr.x (mgr 2021-07-31T03:30:34.815 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: .24542) 4431 : cephadm [ERR] 2021-07-31T03:30:34.816 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: 2021-07-31T03:30:34.816 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:30:34.816 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:30:34.816 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: yield 2021-07-31T03:30:34.816 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:30:34.817 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:30:34.817 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:30:34.817 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:30:34.817 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:30:34.818 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:30:34.818 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:30:34.818 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:30:34.818 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:30:34.818 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: task.result() 2021-07-31T03:30:34.819 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:30:34.819 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: cephadm 2021-07-31T03:30:33.615441+0000 mgr.x (mgr.24542) 4432 : cephadm 2021-07-31T03:30:34.819 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:30:34.819 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:30:34.819 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:30:34.820 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: yield 2021-07-31T03:30:34.820 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: cephadm 2021-07-31T03:30:33.577784+0000 mgr.x (mgr.24542) 4428 : cephadm 2021-07-31T03:30:34.821 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:30:34.821 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: cephadm 2021-07- 2021-07-31T03:30:34.821 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: 31T03:30:33.589372+0000 2021-07-31T03:30:34.821 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: mgr.x (mgr.24542) 2021-07-31T03:30:34.821 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: 4429 : cephadm [ERR] 2021-07-31T03:30:34.822 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:30:34.822 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:30:34.822 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: yield 2021-07-31T03:30:34.822 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:30:34.823 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:30:34.823 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:30:34.823 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:30:34.823 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:30:34.823 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:30:34.824 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:30:34.824 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:30:34.824 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:30:34.824 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: task.result() 2021-07-31T03:30:34.825 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:30:34.825 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: cephadm 2021-07-31T03:30:33.590052+0000 mgr.x (mgr.24542) 4430 : cephadm [INF] 2021-07-31T03:30:34.825 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:30:34.825 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: cephadm 2021-07 2021-07-31T03:30:34.825 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: -31T03:30:33. 2021-07-31T03:30:34.826 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: 614235+0000 mgr.x (mgr.24542 2021-07-31T03:30:34.826 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: ) 4431 : cephadm [ERR] 2021-07-31T03:30:34.826 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:30:34.826 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:30:34.827 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: yield 2021-07-31T03:30:34.827 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:30:34.827 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:30:34.827 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:30:34.827 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:30:34.828 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:30:34.828 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:30:34.828 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:30:34.828 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:30:34.829 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:30:34.829 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: task.result() 2021-07-31T03:30:34.829 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:30:34.829 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: 2021-07-31T03:30:34.829 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: cephadm 2021-07-31T03:30:33.615441+0000 mgr.x (mgr.24542 2021-07-31T03:30:34.830 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: ) 4432 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:30:34.830 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:30:34.830 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:30:34.831 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:30:34.831 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:30:34.831 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:30:34.831 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:30:34.831 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:30:34.832 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:30:34.832 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:30:34.832 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:30:34.832 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: task.result() 2021-07-31T03:30:34.833 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:30:34.833 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: 2021-07-31T03:30:34.833 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:30:34.833 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: 2021-07-31T03:30:34.833 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:30:34.834 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:30:34.834 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:30:34.834 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:30:34.834 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:30:34.835 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:30:34.835 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:30:34.835 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:30:34.835 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:30:34.835 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:30:34.836 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:30:34.836 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:30:34.836 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:30:34.836 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:30:34.836 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:30:34.837 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:30:34.837 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:30:34.837 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:30:34.837 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: raise 2021-07-31T03:30:34.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:30:34.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:30:34.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:30:34.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:30:34.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:30:34.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: cephadm 2021-07-31T03:30:33.623281+0000 mgr.x (mgr.24542) 4433 : cephadm [ERR] 2021-07-31T03:30:34.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:30:34.839 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:30:34.840 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: yield 2021-07-31T03:30:34.840 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:30:34.840 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:30:34.840 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:30:34.841 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:30:34.841 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:30:34.841 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:30:34.841 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:30:34.841 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:30:34.842 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:30:34.842 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: task.result() 2021-07-31T03:30:34.842 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:30:34.842 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: 2021-07-31T03:30:34.843 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:30:34.843 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: 2021-07-31T03:30:34.843 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:30:34.843 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:30:34.843 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:30:34.844 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:30:34.844 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:30:34.844 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:30:34.844 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:30:34.844 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:30:34.845 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:30:34.845 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:30:34.845 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:30:34.845 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:30:34.845 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:30:34.846 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:30:34.846 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:30:34.846 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:30:34.846 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:30:34.846 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:30:34.847 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: raise 2021-07-31T03:30:34.847 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:30:34.847 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:30:34.847 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:30:34.848 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:30:34.848 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:30:34.848 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: cephadm 2021-07-31T03:30:33.623281+0000 mgr.x (mgr.24542) 4433 : cephadm [ERR] 2021-07-31T03:30:34.848 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:30:34.849 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: yield 2021-07-31T03:30:34.849 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:30:34.849 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:30:34.849 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:30:34.849 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:30:34.850 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:30:34.850 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:30:34.850 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:30:34.850 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:30:34.851 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:30:34.851 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: task.result() 2021-07-31T03:30:34.851 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:30:34.851 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: cephadm 2021-07-31T03:30:33.623711+0000 mgr.x 2021-07-31T03:30:34.851 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: (mgr.24542) 4434 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:30:34.852 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:30:34.852 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:30:34.852 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: yield 2021-07-31T03:30:34.852 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:30:34.853 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:30:34.853 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:30:34.853 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:30:34.853 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:30:34.853 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:30:34.854 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:30:34.854 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:30:34.854 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:30:34.854 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: task.result() 2021-07-31T03:30:34.854 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:30:34.855 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: 2021-07-31T03:30:34.855 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:30:34.855 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: 2021-07-31T03:30:34.855 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:30:34.855 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:30:34.856 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:30:34.856 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:30:34.856 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:30:34.856 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:30:34.856 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:30:34.857 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:30:34.857 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:30:34.857 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:30:34.857 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:30:34.858 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:30:34.858 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: yield 2021-07-31T03:30:34.858 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:30:34.858 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:30:34.859 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:30:34.859 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:30:34.859 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:30:34.859 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:30:34.859 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:30:34.860 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:30:34.860 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:30:34.860 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: task.result() 2021-07-31T03:30:34.860 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:30:34.861 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: cephadm 2021-07-31T03:30:33.623711+0000 mgr.x (mgr.24542) 4434 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:30:34.861 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:30:34.861 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:30:34.861 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: yield 2021-07-31T03:30:34.861 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:30:34.862 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:30:34.862 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:30:34.862 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:30:34.862 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:30:34.863 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:30:34.863 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:30:34.863 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:30:34.863 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:30:34.863 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: task.result() 2021-07-31T03:30:34.864 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:30:34.864 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: 2021-07-31T03:30:34.864 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:30:34.864 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: 2021-07-31T03:30:34.864 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:30:34.864 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:30:34.865 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:30:34.865 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:30:34.865 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:30:34.865 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:30:34.865 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:30:34.866 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:30:34.866 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:30:34.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:30:34.867 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:30:34.867 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:30:34.867 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:30:34.867 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:30:34.867 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:30:34.868 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:30:34.868 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:30:34.868 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: raise 2021-07-31T03:30:34.868 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:30:34.868 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:30:34.869 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:30:34.869 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:30:34.869 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:30:34.869 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: audit 2021-07-31T03:30:33.816589+0000 mon.c ( 2021-07-31T03:30:34.869 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[33655]: mon.1) 210 : audit [DBG] from='client.? 172.21.15.45:0/3460258465' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:30:34.870 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:30:34.870 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:30:34.870 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:30:34.871 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:30:34.871 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:30:34.871 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:30:34.871 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:30:34.871 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:30:34.872 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:30:34.872 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: raise 2021-07-31T03:30:34.872 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:30:34.872 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:30:34.872 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:30:34.873 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:30:34.873 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:30:34.873 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:34 smithi045 conmon[39718]: audit 2021-07-31T03:30:33.816589+0000 mon.c (mon.1) 210 : audit [DBG] from='client.? 172.21.15.45:0/3460258465' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:30:36.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:36 smithi154 conmon[34553]: cluster 2021-07-31T03:30:35.271032+0000 mgr.x (mgr.24542) 4435 : cluster 2021-07-31T03:30:36.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:36 smithi154 conmon[34553]: [DBG] pgmap v5108: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:36.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:36 smithi045 conmon[33655]: cluster 2021-07-31T03:30:35.271032+0000 mgr.x (mgr.24542) 4435 : cluster 2021-07-31T03:30:36.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:36 smithi045 conmon[33655]: [DBG] pgmap v5108: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:36.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:36 smithi045 conmon[39718]: cluster 2021-07-31T03:30:35.271032+0000 mgr.x (mgr.24542) 4435 : cluster [DBG] pgmap v5108: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:38.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:38 smithi154 conmon[34553]: cluster 2021-07-31T03:30:37.271404+0000 mgr.x (mgr.24542) 4436 : cluster [DBG] 2021-07-31T03:30:38.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:38 smithi154 conmon[34553]: pgmap v5109: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:38.774 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:38 smithi045 conmon[33655]: cluster 2021-07-31T03:30:37.271404+0000 mgr.x (mgr.24542) 4436 : cluster 2021-07-31T03:30:38.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:38 smithi045 conmon[33655]: [DBG] pgmap v5109: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:38.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:38 smithi045 conmon[39718]: cluster 2021-07-31T03:30: 2021-07-31T03:30:38.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:38 smithi045 conmon[39718]: 37.271404+0000 mgr.x (mgr.24542) 4436 : cluster [DBG] pgmap v5109: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:40.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:40 smithi154 conmon[34553]: cluster 2021-07-31T03:30:39.271895+0000 mgr.x (mgr.24542) 4437 : cluster [DBG] 2021-07-31T03:30:40.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:40 smithi154 conmon[34553]: pgmap v5110: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:40.774 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:40 smithi045 conmon[33655]: cluster 2021-07-31T03:30:39.271895+0000 mgr.x (mgr.24542) 4437 : cluster 2021-07-31T03:30:40.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:40 smithi045 conmon[33655]: [DBG] pgmap v5110: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:40.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:40 smithi045 conmon[39718]: cluster 2021-07-31T03:30:39.271895+0000 mgr.x 2021-07-31T03:30:40.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:40 smithi045 conmon[39718]: (mgr.24542) 4437 : cluster [DBG] pgmap v5110: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:42.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:42 smithi154 conmon[34553]: cluster 2021-07-31T03:30:41.272401+0000 mgr.x (mgr.24542) 4438 : cluster 2021-07-31T03:30:42.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:42 smithi154 conmon[34553]: [DBG] pgmap v5111: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:42.774 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:42 smithi045 conmon[39718]: cluster 2021-07-31T03:30:41.272401+0000 mgr.x (mgr.24542) 2021-07-31T03:30:42.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:42 smithi045 conmon[39718]: 4438 : cluster [DBG] pgmap v5111: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:42.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:42 smithi045 conmon[33655]: cluster 2021-07-31T03:30:41.272401+0000 mgr.x (mgr.24542) 4438 : 2021-07-31T03:30:42.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:42 smithi045 conmon[33655]: cluster [DBG] pgmap v5111: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:44.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:44 smithi154 conmon[34553]: cluster 2021-07-31T03:30:43.272785+0000 mgr.x (mgr.24542) 4439 : cluster [DBG] pgmap v5112: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:44.774 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:44 smithi045 conmon[33655]: cluster 2021-07-31T03:30:43.272785+0000 mgr.x (mgr.24542) 4439 2021-07-31T03:30:44.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:44 smithi045 conmon[33655]: : cluster [DBG] pgmap v5112: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:44.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:44 smithi045 conmon[39718]: cluster 2021-07-31T03:30:43.272785+0000 mgr.x ( 2021-07-31T03:30:44.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:44 smithi045 conmon[39718]: mgr.24542) 4439 : cluster [DBG] pgmap v5112: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:46.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:46 smithi154 conmon[34553]: cluster 2021-07-31T03:30:45.273307+0000 mgr.x (mgr.24542) 4440 : cluster [DBG] pgmap v5113: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:46.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:46 smithi045 conmon[33655]: cluster 2021-07-31T03:30:45.273307+0000 mgr.x 2021-07-31T03:30:46.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:46 smithi045 conmon[33655]: (mgr.24542) 4440 : cluster [DBG] pgmap v5113: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:46.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:46 smithi045 conmon[39718]: cluster 2021-07-31T03:30:45.273307+0000 mgr.x ( 2021-07-31T03:30:46.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:46 smithi045 conmon[39718]: mgr.24542) 4440 : cluster [DBG] pgmap v5113: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:48.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:48 smithi154 conmon[34553]: cluster 2021-07-31T03:30:47.273676+0000 mgr.x (mgr.24542) 4441 : cluster [DBG] pgmap v5114: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:48.774 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:48 smithi045 conmon[33655]: cluster 2021-07-31T03:30:47.273676+0000 mgr.x (mgr.24542) 4441 : cluster [DBG] pgmap v5114: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:48.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:48 smithi045 conmon[39718]: cluster 2021-07-31T03:30:47.273676+0000 mgr.x ( 2021-07-31T03:30:48.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:48 smithi045 conmon[39718]: mgr.24542) 4441 : cluster [DBG] pgmap v5114: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:50.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:50 smithi154 conmon[34553]: cluster 2021-07-31T03:30:49.274154+0000 mgr.x (mgr.24542) 4442 : cluster [DBG] pgmap v5115: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:51.397 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:50 smithi045 conmon[33655]: cluster 2021-07-31T03:30:49.274154 2021-07-31T03:30:51.398 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:50 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 4442 : cluster [DBG] pgmap v5115: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:51.398 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:50 smithi045 conmon[39718]: cluster 2021-07-31T03:30:49.274154+0000 mgr.x (mgr 2021-07-31T03:30:51.398 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:50 smithi045 conmon[39718]: .24542) 4442 : cluster [DBG] pgmap v5115: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:52.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:52 smithi045 conmon[33655]: cluster 2021-07-31T03:30:51.274615+0000 mgr.x (mgr 2021-07-31T03:30:52.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:52 smithi045 conmon[33655]: .24542) 4443 : cluster [DBG] pgmap v5116: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:52.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:52 smithi045 conmon[39718]: cluster 2021-07-31T03:30:51.274615+0000 mgr.x (mgr.24542) 4443 : cluster [DBG] pgmap v5116: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:52.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:52 smithi045 conmon[39718]: 2021-07-31T03:30:53.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:52 smithi154 conmon[34553]: cluster 2021-07-31T03:30:51.274615+0000 mgr.x (mgr.24542) 4443 : cluster [DBG] pgmap v5116: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:54.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:54 smithi045 conmon[33655]: cluster 2021-07-31T03:30:53.274998 2021-07-31T03:30:54.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:54 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 4444 : cluster [DBG] pgmap v5117: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:54.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:54 smithi045 conmon[39718]: cluster 2021-07-31T03:30:53.274998+0000 mgr.x (mgr.24542) 2021-07-31T03:30:54.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:54 smithi045 conmon[39718]: 4444 : cluster [DBG] pgmap v5117: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:55.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:54 smithi154 conmon[34553]: cluster 2021-07-31T03:30:53.274998+0000 mgr.x (mgr.24542) 4444 : cluster [DBG] pgmap v5117: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:56.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:56 smithi045 conmon[33655]: cluster 2021-07-31T03:30:55.275417+0000 mgr.x 2021-07-31T03:30:56.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:56 smithi045 conmon[33655]: (mgr.24542) 4445 : cluster [DBG] pgmap v5118: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:56.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:56 smithi045 conmon[39718]: cluster 2021-07-31T03:30:55.275417+0000 mgr.x (mgr.24542) 4445 : cluster [DBG] pgmap v5118: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:57.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:56 smithi154 conmon[34553]: cluster 2021-07-31T03:30:55.275417+0000 mgr.x (mgr.24542) 4445 : cluster [DBG] pgmap v5118: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:58.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:58 smithi045 conmon[33655]: cluster 2021-07-31T03:30:57.275878+0000 mgr.x (mgr.24542) 4446 : cluster [DBG] 2021-07-31T03:30:58.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:30:58 smithi045 conmon[33655]: pgmap v5119: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:58.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:58 smithi045 conmon[39718]: cluster 2021-07-31T03:30:57 2021-07-31T03:30:58.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:30:58 smithi045 conmon[39718]: .275878+0000 mgr.x (mgr.24542) 4446 : cluster [DBG] pgmap v5119: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:30:59.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:30:58 smithi154 conmon[34553]: cluster 2021-07-31T03:30:57.275878+0000 mgr.x (mgr.24542) 4446 : cluster [DBG] pgmap v5119: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:00.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:00 smithi045 conmon[39718]: cluster 2021-07-31T03:30:59.276320+0000 mgr.x ( 2021-07-31T03:31:00.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:00 smithi045 conmon[39718]: mgr.24542) 4447 : cluster [DBG] pgmap v5120: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:00.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:00 smithi045 conmon[33655]: cluster 2021-07-31T03:30:59.276320 2021-07-31T03:31:00.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:00 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 4447 : cluster [DBG] pgmap v5120: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:01.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:00 smithi154 conmon[34553]: cluster 2021-07-31T03:30:59.276320+0000 mgr.x (mgr.24542) 4447 : cluster [DBG] pgmap v5120: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:02.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:02 smithi045 conmon[33655]: cluster 2021-07-31T03:31:01.276731+0000 mgr.x (mgr.24542) 4448 : cluster 2021-07-31T03:31:02.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:02 smithi045 conmon[33655]: [DBG] pgmap v5121: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:02.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:02 smithi045 conmon[39718]: cluster 2021-07-31T03:31:01.276731+0000 mgr.x (mgr.24542) 4448 : cluster [DBG] pgmap v5121: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:03.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:02 smithi154 conmon[34553]: cluster 2021-07-31T03:31:01.276731+0000 mgr.x (mgr.24542) 4448 : cluster [DBG] pgmap v5121: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:04.131 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:31:04.407 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:31:04.408 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 105m ago 106m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:31:04.408 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 103m ago 106m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:31:04.408 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 103m ago 109m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:31:04.408 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 105m ago 112m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:31:04.409 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 105m ago 112m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:31:04.409 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 103m ago 110m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:31:04.409 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 105m ago 110m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:31:04.409 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 105m ago 106m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:31:04.409 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 103m ago 106m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:31:04.410 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 105m ago 109m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:31:04.410 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 105m ago 109m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:31:04.410 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 105m ago 109m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:31:04.410 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 105m ago 108m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:31:04.411 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 103m ago 108m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:31:04.411 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 103m ago 108m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:31:04.411 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 103m ago 107m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:31:04.411 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 103m ago 107m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:31:04.411 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 103m ago 107m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:31:04.750 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:31:04.751 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:31:04.751 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:31:04.751 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:31:04.751 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:31:04.752 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:31:04.752 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:31:04.752 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:31:04.752 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:31:04.752 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:31:04.753 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:31:04.753 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:31:04.753 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:31:04.753 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:31:04.753 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:31:04.754 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:31:04.754 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:31:05.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:04 smithi154 conmon[34553]: cluster 2021-07-31T03:31:03.277094+0000 mgr.x (mgr.24542) 4449 : cluster [DBG] pgmap v5122: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:05.024 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:04 smithi045 conmon[39718]: cluster 2021-07-31T03:31:03.277094 2021-07-31T03:31:05.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:04 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 4449 : cluster [DBG] pgmap v5122: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:05.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:04 smithi045 conmon[33655]: cluster 2021-07-31T03:31: 2021-07-31T03:31:05.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:04 smithi045 conmon[33655]: 03.277094+0000 mgr.x (mgr.24542) 4449 : cluster [DBG] pgmap v5122: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:06.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:05 smithi154 conmon[34553]: audit 2021-07-31T03:31:04.120062+0000 mgr.x (mgr.24542) 4450 : audit [DBG] from='client.29608 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:31:06.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:05 smithi154 conmon[34553]: 2021-07-31T03:31:06.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:05 smithi154 conmon[34553]: audit 2021-07-31T03:31:04.400645+0000 mgr.x (mgr.24542) 4451 : audit [DBG] from='client.19674 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:31:06.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:05 smithi154 conmon[34553]: audit 2021-07-31T03:31:04. 2021-07-31T03:31:06.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:05 smithi154 conmon[34553]: 751505+0000 mon.a (mon.0) 3317 : audit [DBG] from='client.? 172.21.15.45:0/2201996411' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:31:06.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:05 smithi045 conmon[39718]: audit 2021-07-31T03:31:04.120062+0000 2021-07-31T03:31:06.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:05 smithi045 conmon[39718]: mgr.x (mgr.24542) 4450 : audit [DBG] from='client.29608 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:31:06.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:05 smithi045 conmon[39718]: audit 2021-07-31T03:31 2021-07-31T03:31:06.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:05 smithi045 conmon[39718]: :04.400645+0000 mgr.x (mgr.24542) 4451 : audit [DBG] from='client.19674 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:31:06.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:05 smithi045 conmon[39718]: 2021-07-31T03:31:06.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:05 smithi045 conmon[39718]: audit 2021-07-31T03:31:04. 2021-07-31T03:31:06.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:05 smithi045 conmon[39718]: 751505+0000 mon.a (mon.0) 3317 : audit 2021-07-31T03:31:06.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:05 smithi045 conmon[39718]: [DBG] from='client.? 172.21.15.45:0/2201996411' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:31:06.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:05 smithi045 conmon[33655]: audit 2021-07-31T03:31:04.120062+0000 mgr.x (mgr.24542) 4450 : audit [DBG] from='client.29608 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:31:06.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:05 smithi045 conmon[33655]: audit 2021-07-31T03:31:04.400645+0000 mgr.x (mgr.24542) 4451 : audit [DBG] from='client.19674 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:31:06.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:05 smithi045 conmon[33655]: audit 2021-07-31T03:31:04.751505+0000 mon.a (mon.0) 3317 : 2021-07-31T03:31:06.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:05 smithi045 conmon[33655]: audit [DBG] from='client.? 172.21.15.45:0/2201996411' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:31:07.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:06 smithi154 conmon[34553]: cluster 2021-07-31T03:31:05.277493+0000 mgr.x (mgr.24542) 4452 : cluster [DBG] pgmap v5123: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:07.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:06 smithi045 conmon[39718]: cluster 2021-07-31T03:31:05.277493+0000 2021-07-31T03:31:07.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:06 smithi045 conmon[39718]: mgr.x (mgr.24542) 4452 : cluster [DBG] pgmap v5123: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:07.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:06 smithi045 conmon[33655]: cluster 2021-07-31T03:31:05.277493+0000 2021-07-31T03:31:07.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:06 smithi045 conmon[33655]: mgr.x (mgr.24542) 4452 : cluster [DBG] pgmap v5123: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:09.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:08 smithi154 conmon[34553]: cluster 2021-07-31T03:31:07.278032+0000 mgr.x (mgr.24542) 4453 : cluster [DBG] pgmap v5124: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:09.024 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:08 smithi045 conmon[39718]: cluster 2021-07-31T03:31:07.278032+0000 mgr.x (mgr 2021-07-31T03:31:09.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:08 smithi045 conmon[39718]: .24542) 4453 : cluster [DBG] pgmap v5124: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:09.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:08 smithi045 conmon[33655]: cluster 2021-07-31T03:31:07.278032+0000 mgr.x (mgr.24542) 2021-07-31T03:31:09.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:08 smithi045 conmon[33655]: 4453 : cluster [DBG] pgmap v5124: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:11.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:10 smithi154 conmon[34553]: cluster 2021-07-31T03:31:09.278417+0000 mgr.x (mgr.24542) 4454 : cluster [DBG] 2021-07-31T03:31:11.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:10 smithi154 conmon[34553]: pgmap v5125: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:11.024 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:10 smithi045 conmon[39718]: cluster 2021-07-31T03:31:09.278417+0000 mgr.x (mgr. 2021-07-31T03:31:11.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:10 smithi045 conmon[39718]: 24542) 4454 : cluster [DBG] pgmap v5125: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:11.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:10 smithi045 conmon[33655]: cluster 2021-07-31T03:31:09.278417 2021-07-31T03:31:11.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:10 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 4454 : cluster [DBG] pgmap v5125: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:13.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:12 smithi154 conmon[34553]: cluster 2021-07-31T03:31:11.278847+0000 mgr.x (mgr.24542) 4455 : cluster [DBG] pgmap v5126: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:13.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:12 smithi045 conmon[33655]: cluster 2021-07-31T03:31:11.278847+0000 mgr.x (mgr.24542) 4455 : cluster [DBG] pgmap v5126: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:13.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:12 smithi045 conmon[39718]: cluster 2021-07-31T03:31:11.278847+0000 mgr.x (mgr.24542) 4455 : cluster [DBG] pgmap v5126: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:15.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:14 smithi154 conmon[34553]: cluster 2021-07-31T03:31:13.279358+0000 mgr.x (mgr.24542) 4456 : cluster [DBG] pgmap v5127: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:15.024 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:14 smithi045 conmon[33655]: cluster 2021-07-31T03:31:13.279358+0000 mgr.x (mgr.24542) 4456 : 2021-07-31T03:31:15.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:14 smithi045 conmon[33655]: cluster [DBG] pgmap v5127: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:15.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:14 smithi045 conmon[39718]: cluster 2021-07-31T03:31:13.279358+0000 mgr.x 2021-07-31T03:31:15.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:14 smithi045 conmon[39718]: (mgr.24542) 4456 : cluster [DBG] pgmap v5127: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:17.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:16 smithi154 conmon[34553]: cluster 2021-07-31T03:31:15.279868+0000 mgr.x (mgr.24542) 4457 : cluster [DBG] pgmap v5128: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:17.024 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:16 smithi045 conmon[39718]: cluster 2021-07-31T03:31:15.279868+0000 mgr.x (mgr. 2021-07-31T03:31:17.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:16 smithi045 conmon[39718]: 24542) 4457 : cluster [DBG] pgmap v5128: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:17.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:16 smithi045 conmon[33655]: cluster 2021-07-31T03:31:15.279868+0000 mgr.x (mgr. 2021-07-31T03:31:17.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:16 smithi045 conmon[33655]: 24542) 4457 : cluster [DBG] pgmap v5128: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:19.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:18 smithi154 conmon[34553]: cluster 2021-07-31T03:31:17.280320+0000 mgr.x (mgr.24542) 4458 : cluster [DBG] pgmap v5129: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:19.024 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:18 smithi045 conmon[39718]: cluster 2021-07-31T03:31:17.280320+0000 mgr.x (mgr 2021-07-31T03:31:19.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:18 smithi045 conmon[39718]: .24542) 4458 : cluster [DBG] pgmap v5129: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:19.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:18 smithi045 conmon[33655]: cluster 2021-07-31T03:31:17.280320+0000 mgr.x (mgr. 2021-07-31T03:31:19.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:18 smithi045 conmon[33655]: 24542) 4458 : cluster [DBG] pgmap v5129: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:21.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:20 smithi154 conmon[34553]: cluster 2021-07-31T03:31:19.280772+0000 mgr.x (mgr.24542) 4459 : cluster [DBG] pgmap v5130: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:21.024 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:20 smithi045 conmon[39718]: cluster 2021-07-31T03:31:19.280772+0000 mgr.x (mgr.24542) 2021-07-31T03:31:21.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:20 smithi045 conmon[39718]: 4459 : cluster [DBG] pgmap v5130: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:21.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:20 smithi045 conmon[33655]: cluster 2021-07-31T03:31:19.280772+0000 mgr.x (mgr.24542) 4459 : cluster 2021-07-31T03:31:21.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:20 smithi045 conmon[33655]: [DBG] pgmap v5130: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:23.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:22 smithi154 conmon[34553]: cluster 2021-07-31T03:31:21.281163+0000 mgr.x (mgr.24542) 4460 : cluster [DBG] pgmap v5131: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:23.024 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:22 smithi045 conmon[39718]: cluster 2021-07-31T03:31:21 2021-07-31T03:31:23.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:22 smithi045 conmon[39718]: .281163+0000 mgr.x (mgr.24542) 4460 : cluster [DBG] pgmap v5131: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:23.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:22 smithi045 conmon[33655]: cluster 2021-07-31T03:31:21.281163+0000 mgr.x (mgr.24542) 4460 2021-07-31T03:31:23.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:22 smithi045 conmon[33655]: : cluster [DBG] pgmap v5131: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:25.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:24 smithi154 conmon[34553]: cluster 2021-07-31T03:31:23.281620+0000 mgr.x (mgr.24542) 4461 : cluster [DBG] pgmap v5132: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:25.024 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:24 smithi045 conmon[39718]: cluster 2021-07-31T03:31:23.281620+0000 mgr.x 2021-07-31T03:31:25.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:24 smithi045 conmon[39718]: (mgr.24542) 4461 : cluster [DBG] pgmap v5132: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:25.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:24 smithi045 conmon[33655]: cluster 2021-07-31T03:31:23.281620+0000 mgr.x (mgr.24542) 4461 : cluster [DBG] pgmap v5132: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:27.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:26 smithi154 conmon[34553]: cluster 2021-07-31T03:31:25.282147+0000 mgr.x (mgr.24542) 4462 : cluster [DBG] pgmap v5133: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:27.024 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:26 smithi045 conmon[39718]: cluster 2021-07-31T03:31:25.282147+0000 mgr.x (mgr.24542) 4462 : cluster [DBG] pgmap v5133: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:27.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:26 smithi045 conmon[33655]: cluster 2021-07-31T03:31:25.282147+0000 mgr.x (mgr.24542) 4462 : cluster [DBG] pgmap v5133: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:29.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:28 smithi154 conmon[34553]: cluster 2021-07-31T03:31:27.282500+0000 mgr.x (mgr.24542) 4463 : cluster [DBG] pgmap v5134: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:29.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:28 smithi154 conmon[34553]: audit 2021-07-31T03:31:28.348428+0000 mon.b (mon.2) 595 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:31:29.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:28 smithi154 conmon[34553]: audit 2021-07-31T03:31:28.348695+0000 mon.b (mon.2) 596 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:31:29.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:28 smithi154 conmon[34553]: audit 2021-07-31T03:31:28.349187+0000 mon.a (mon.0) 3318 : 2021-07-31T03:31:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:28 smithi154 conmon[34553]: audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:31:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:28 smithi154 conmon[34553]: audit 2021-07-31T03:31:28.349567+0000 mon.a (mon 2021-07-31T03:31:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:28 smithi154 conmon[34553]: .0) 3319 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:31:29.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:28 smithi045 conmon[39718]: cluster 2021-07-31T03:31:27.282500 2021-07-31T03:31:29.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:28 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 4463 : cluster [DBG] pgmap v5134: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:29.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:28 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T03:31:29.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:28 smithi045 conmon[39718]: 03:31:28.348428+0000 mon.b 2021-07-31T03:31:29.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:28 smithi045 conmon[39718]: (mon.2) 595 : audit 2021-07-31T03:31:29.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:28 smithi045 conmon[39718]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:31:29.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:28 smithi045 conmon[39718]: audit 2021-07-31T03:31 2021-07-31T03:31:29.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:28 smithi045 conmon[39718]: :28.348695+0000 mon.b (mon.2) 2021-07-31T03:31:29.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:28 smithi045 conmon[39718]: 596 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:31:29.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:28 smithi045 conmon[39718]: audit 2021 2021-07-31T03:31:29.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:28 smithi045 conmon[39718]: -07-31T03:31:28.349187+0000 mon.a 2021-07-31T03:31:29.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:28 smithi045 conmon[39718]: (mon.0) 3318 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:31:29.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:28 smithi045 conmon[39718]: audit 2021 2021-07-31T03:31:29.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:28 smithi045 conmon[39718]: -07-31T03:31:28.349567+0000 2021-07-31T03:31:29.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:28 smithi045 conmon[39718]: mon.a (mon.0) 3319 : 2021-07-31T03:31:29.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:28 smithi045 conmon[39718]: audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:31:29.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:28 smithi045 conmon[33655]: cluster 2021-07-31T03:31:27.282500+0000 mgr.x (mgr.24542) 4463 : cluster [DBG] pgmap v5134: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:29.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:28 smithi045 conmon[33655]: audit 2021-07-31T03:31:28.348428+0000 mon.b (mon.2) 595 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:31:29.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:28 smithi045 conmon[33655]: audit 2021-07-31T03:31:28 2021-07-31T03:31:29.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:28 smithi045 conmon[33655]: .348695+0000 mon.b (mon.2) 596 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:31:29.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:28 smithi045 conmon[33655]: audit 2021-07-31T03:31:29.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:28 smithi045 conmon[33655]: 2021-07-31T03:31:28.349187 2021-07-31T03:31:29.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:28 smithi045 conmon[33655]: +0000 mon.a (mon.0) 3318 : audit [INF] 2021-07-31T03:31:29.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:28 smithi045 conmon[33655]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:31:29.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:28 smithi045 conmon[33655]: audit 2021- 2021-07-31T03:31:29.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:28 smithi045 conmon[33655]: 07-31T03:31:28.349567+0000 2021-07-31T03:31:29.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:28 smithi045 conmon[33655]: mon.a (mon.0) 3319 2021-07-31T03:31:29.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:28 smithi045 conmon[33655]: : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:31:31.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:30 smithi154 conmon[34553]: cluster 2021-07-31T03:31:29.282914+0000 mgr.x (mgr.24542) 4464 : cluster [DBG] pgmap v5135: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:31.024 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:30 smithi045 conmon[39718]: cluster 2021-07-31T03:31:29.282914 2021-07-31T03:31:31.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:30 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 4464 : cluster [DBG] pgmap v5135: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:31.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:30 smithi045 conmon[33655]: cluster 2021-07-31T03:31:29.282914+0000 mgr.x (mgr.24542) 4464 : cluster [DBG] pgmap v5135: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:31.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:30 smithi045 conmon[33655]: 2021-07-31T03:31:33.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:32 smithi154 conmon[34553]: cluster 2021-07-31T03:31:31.283416+0000 mgr.x (mgr.24542) 4465 : cluster [DBG] pgmap v5136: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:33.024 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:32 smithi045 conmon[39718]: cluster 2021-07-31T03:31:31. 2021-07-31T03:31:33.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:32 smithi045 conmon[39718]: 283416+0000 mgr.x (mgr.24542) 4465 : cluster [DBG] pgmap v5136: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:33.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:32 smithi045 conmon[33655]: cluster 2021-07-31T03:31:31.283416+0000 mgr.x (mgr.24542) 4465 : cluster 2021-07-31T03:31:33.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:32 smithi045 conmon[33655]: [DBG] pgmap v5136: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:34.002 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: debug 2021-07-31T03:31:33.669+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:31:34.002 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:31:34.003 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:31:34.003 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: yield 2021-07-31T03:31:34.003 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:31:34.003 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:31:34.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:31:34.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:31:34.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:31:34.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:31:34.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:31:34.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:31:34.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:31:34.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: task.result() 2021-07-31T03:31:34.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:31:34.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: debug 2021-07-31T03:31:33.681+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:31:34.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:31:34.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:31:34.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: yield 2021-07-31T03:31:34.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:31:34.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:31:34.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:31:34.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:31:34.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:31:34.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:31:34.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:31:34.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:31:34.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:31:34.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: task.result() 2021-07-31T03:31:34.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:31:34.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: debug 2021-07-31T03:31:33.706+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:31:34.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:31:34.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:31:34.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: yield 2021-07-31T03:31:34.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:31:34.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:31:34.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:31:34.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:31:34.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:31:34.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:31:34.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:31:34.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:31:34.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:31:34.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: task.result() 2021-07-31T03:31:34.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:31:34.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: debug 2021-07-31T03:31:33.707+0000 7fcb97969700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:31:34.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:31:34.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:31:34.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: yield 2021-07-31T03:31:34.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:31:34.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:31:34.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:31:34.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:31:34.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:31:34.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:31:34.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:31:34.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:31:34.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:31:34.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: task.result() 2021-07-31T03:31:34.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:31:34.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: 2021-07-31T03:31:34.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:31:34.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: 2021-07-31T03:31:34.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:31:34.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:31:34.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:31:34.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:31:34.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:31:34.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:31:34.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:31:34.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:31:34.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:31:34.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:31:34.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:31:34.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:31:34.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:31:34.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:31:34.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:31:34.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:31:34.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:31:34.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:31:34.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: raise 2021-07-31T03:31:34.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:31:34.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:31:34.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:31:34.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:31:34.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:31:34.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: debug 2021-07-31T03:31:33.715+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:31:34.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:31:34.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:31:34.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: yield 2021-07-31T03:31:34.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:31:34.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:31:34.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:31:34.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:31:34.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:31:34.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:31:34.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:31:34.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:31:34.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:31:34.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: task.result() 2021-07-31T03:31:34.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:31:34.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: debug 2021-07-31T03:31:33.715+0000 7fcb9816a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:31:34.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:31:34.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:31:34.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: yield 2021-07-31T03:31:34.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:31:34.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:31:34.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:31:34.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:31:34.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:31:34.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:31:34.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:31:34.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:31:34.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:31:34.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: task.result() 2021-07-31T03:31:34.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:31:34.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: 2021-07-31T03:31:34.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:31:34.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: 2021-07-31T03:31:34.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:31:34.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:31:34.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:31:34.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:31:34.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:31:34.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:31:34.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:31:34.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:31:34.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:31:34.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:31:34.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:31:34.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:31:34.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:31:34.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:31:34.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:31:34.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:31:34.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:31:34.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:31:34.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: raise 2021-07-31T03:31:34.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:31:34.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:31:34.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:31:34.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:31:34.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:31:33 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:31:35.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: cluster 2021-07-31T03:31:33.283894+0000 mgr.x (mgr.24542) 4466 : cluster [DBG] pgmap v5137: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:35.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: 2021-07-31T03:31:35.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: audit 2021-07-31T03:31:33.624453+0000 mon.b (mon.2) 597 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:31:35.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: audit 2021-07-31T03 2021-07-31T03:31:35.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: :31:33.625782+0000 mon.b (mon.2) 598 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:31:35.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: audit 2021-07-31T03 2021-07-31T03:31:35.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: :31:33.626690+0000 mon.b (mon.2 2021-07-31T03:31:35.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: ) 599 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:31:35.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: cephadm 2021 2021-07-31T03:31:35.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: -07-31T03:31:33.627499 2021-07-31T03:31:35.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: +0000 mgr.x (mgr.24542) 4467 : cephadm [WRN] 2021-07-31T03:31:35.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:31:35.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: cephadm 2021-07-31T03 2021-07-31T03:31:35.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: :31:33.670696+0000 mgr.x (mgr.24542 2021-07-31T03:31:35.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: ) 4468 : cephadm [ERR] 2021-07-31T03:31:35.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:31:35.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:31:35.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: yield 2021-07-31T03:31:35.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:31:35.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:31:35.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:31:35.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:31:35.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:31:35.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:31:35.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:31:35.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:31:35.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:31:35.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: task.result() 2021-07-31T03:31:35.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:31:35.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: cephadm 2021-07-31T03:31:33.671124+0000 mgr.x (mgr.24542) 4469 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:31:35.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: cephadm 2021-07-31T03:31:33.682020+0000 mgr.x (mgr.24542) 4470 : 2021-07-31T03:31:35.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: cephadm [ERR] 2021-07-31T03:31:35.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:31:35.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:31:35.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: yield 2021-07-31T03:31:35.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:31:35.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:31:35.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:31:35.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:31:35.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:31:35.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:31:35.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:31:35.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:31:35.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:31:35.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: task.result() 2021-07-31T03:31:35.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:31:35.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: cephadm 2021-07-31T03:31:33.682458+0000 mgr.x (mgr.24542) 4471 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:31:35.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: cephadm 2021-07-31T03:31:33.707162+0000 mgr.x (mgr.24542) 4472 : cephadm [ERR] 2021-07-31T03:31:35.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:31:35.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:31:35.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: yield 2021-07-31T03:31:35.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:31:35.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:31:35.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:31:35.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:31:35.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:31:35.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:31:35.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:31:35.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:31:35.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:31:35.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: task.result() 2021-07-31T03:31:35.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:31:35.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: cephadm 2021-07-31T03:31:33.708361+0000 mgr.x (mgr.24542) 4473 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:31:35.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:31:35.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:31:35.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: yield 2021-07-31T03:31:35.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:31:35.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:31:35.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:31:35.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:31:35.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:31:35.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:31:35.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:31:35.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:31:35.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:31:35.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: task.result() 2021-07-31T03:31:35.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:31:35.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: 2021-07-31T03:31:35.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:31:35.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: 2021-07-31T03:31:35.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:31:35.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:31:35.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:31:35.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:31:35.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:31:35.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:31:35.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:31:35.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:31:35.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:31:35.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:31:35.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:31:35.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:31:35.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:31:35.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:31:35.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:31:35.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:31:35.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:31:35.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:31:35.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: raise 2021-07-31T03:31:35.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:31:35.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:31:35.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:31:35.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:31:35.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:31:35.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: cephadm 2021-07- 2021-07-31T03:31:35.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: 31T03:31:33.716238+0000 mgr.x (mgr.24542) 4474 : cephadm [ERR] 2021-07-31T03:31:35.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:31:35.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:31:35.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: yield 2021-07-31T03:31:35.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:31:35.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:31:35.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:31:35.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:31:35.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:31:35.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:31:35.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:31:35.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:31:35.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:31:35.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: task.result() 2021-07-31T03:31:35.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: cluster 2021-07-31T03:31:33.283894+0000 2021-07-31T03:31:35.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: mgr.x (mgr.24542) 4466 : cluster [DBG] pgmap v5137: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:35.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: audit 2021-07-31T03:31: 2021-07-31T03:31:35.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: 33.624453+0000 mon.b (mon 2021-07-31T03:31:35.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: .2) 597 : audit [DBG] 2021-07-31T03:31:35.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:31:35.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: audit 2021-07-31T03:31:35.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: 2021-07-31T03:31:33 2021-07-31T03:31:35.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: .625782+0000 mon.b (mon. 2021-07-31T03:31:35.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: 2) 598 : audit [DBG] 2021-07-31T03:31:35.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:31:35.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: audit 2021-07 2021-07-31T03:31:35.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: -31T03:31:33.626690+0000 2021-07-31T03:31:35.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: mon.b (mon.2) 599 2021-07-31T03:31:35.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:31:35.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: cephadm 2021- 2021-07-31T03:31:35.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: 07-31T03:31:33 2021-07-31T03:31:35.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: .627499+0000 mgr.x (mgr. 2021-07-31T03:31:35.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: 24542) 4467 : cephadm [WRN] 2021-07-31T03:31:35.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:31:35.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: cephadm 2021-07-31T03:31:35.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: 2021-07-31T03: 2021-07-31T03:31:35.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: 31:33.670696+0000 mgr.x 2021-07-31T03:31:35.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: (mgr.24542) 4468 : cephadm 2021-07-31T03:31:35.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: [ERR] 2021-07-31T03:31:35.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:31:35.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:31:35.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: yield 2021-07-31T03:31:35.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:31:35.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:31:35.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:31:35.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:31:35.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:31:35.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:31:35.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:31:35.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:31:35.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:31:35.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: task.result() 2021-07-31T03:31:35.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:31:35.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: cephadm 2021-07-31T03:31:33.671124+0000 mgr.x (mgr.24542) 4469 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:31:35.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: cephadm 2021- 2021-07-31T03:31:35.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: 07-31T03:31:33. 2021-07-31T03:31:35.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: 682020+0000 mgr.x (mgr.24542 2021-07-31T03:31:35.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: ) 4470 : cephadm [ERR] 2021-07-31T03:31:35.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:31:35.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: cluster 2021-07-31T03:31:33.283894+0000 mgr.x (mgr. 2021-07-31T03:31:35.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: 24542) 4466 : cluster [DBG] pgmap v5137: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:35.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: audit 2021-07-31T03:31:33.624453+0000 mon.b (mon.2) 597 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:31:35.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: audit 2021-07-31T03:31:33.625782+0000 mon.b (mon.2) 2021-07-31T03:31:35.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: 598 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:31:35.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: audit 2021 2021-07-31T03:31:35.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: -07-31T03:31:33.626690+0000 mon.b ( 2021-07-31T03:31:35.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: mon.2) 599 : audit 2021-07-31T03:31:35.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:31:35.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: cephadm 2021- 2021-07-31T03:31:35.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: 07-31T03:31:33.627499+0000 mgr.x ( 2021-07-31T03:31:35.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: mgr.24542) 4467 : cephadm [WRN] 2021-07-31T03:31:35.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:31:35.040 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: cephadm 2021-07 2021-07-31T03:31:35.040 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: -31T03:31:33.670696+0000 mgr.x (mgr 2021-07-31T03:31:35.040 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: .24542) 4468 : cephadm [ERR] 2021-07-31T03:31:35.040 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: 2021-07-31T03:31:35.040 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:31:35.041 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:31:35.041 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: yield 2021-07-31T03:31:35.041 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:31:35.041 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:31:35.041 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:31:35.042 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:31:35.042 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:31:35.042 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:31:35.042 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:31:35.042 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:31:35.043 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:31:35.043 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: task.result() 2021-07-31T03:31:35.043 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:31:35.043 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: cephadm 2021-07-31T03:31:33.671124+0000 mgr.x (mgr.24542) 4469 : 2021-07-31T03:31:35.044 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:31:35.044 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: cephadm 2021-07- 2021-07-31T03:31:35.044 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: 31T03:31:33.682020+0000 mgr.x (mgr.24542) 4470 : cephadm [ERR] 2021-07-31T03:31:35.044 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:31:35.044 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:31:35.045 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: yield 2021-07-31T03:31:35.045 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:31:35.045 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:31:35.045 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:31:35.045 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: cephadm 2021-07-31T03:31:33.716808+0000 mgr.x (mgr.24542) 4475 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:31:35.046 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:31:35.046 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:31:35.046 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: yield 2021-07-31T03:31:35.046 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:31:35.047 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:31:35.047 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:31:35.047 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:31:35.047 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:31:35.047 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:31:35.048 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:31:35.048 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:31:35.048 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:31:35.048 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: task.result() 2021-07-31T03:31:35.048 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:31:35.049 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: 2021-07-31T03:31:35.049 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:31:35.049 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: 2021-07-31T03:31:35.049 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:31:35.049 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:31:35.050 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:31:35.050 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:31:35.050 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:31:35.050 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:31:35.050 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:31:35.051 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:31:35.051 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:31:35.051 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:31:35.051 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:31:35.051 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:31:35.052 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:31:35.052 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:31:35.052 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:31:35.052 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:31:35.052 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:31:35.053 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:31:35.053 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: raise 2021-07-31T03:31:35.053 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:31:35.053 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:31:35.053 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:31:35.054 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:31:35.054 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:34 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:31:35.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:31:35.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: yield 2021-07-31T03:31:35.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:31:35.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:31:35.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:31:35.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:31:35.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:31:35.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:31:35.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:31:35.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:31:35.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:31:35.057 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: task.result() 2021-07-31T03:31:35.057 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:31:35.057 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: 2021-07-31T03:31:35.057 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: cephadm 2021-07-31T03:31:33.682458+0000 mgr.x (mgr.24542) 4471 : 2021-07-31T03:31:35.057 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:31:35.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: cephadm 2021-07-31T03:31:35.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: 2021-07-31T03:31:33 2021-07-31T03:31:35.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: .707162+0000 mgr.x (mgr.24542 2021-07-31T03:31:35.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: ) 4472 : cephadm [ERR] 2021-07-31T03:31:35.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:31:35.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:31:35.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: yield 2021-07-31T03:31:35.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:31:35.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:31:35.060 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:31:35.060 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:31:35.060 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:31:35.060 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:31:35.060 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:31:35.061 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:31:35.061 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:31:35.061 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: task.result() 2021-07-31T03:31:35.061 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:31:35.061 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: 2021-07-31T03:31:35.062 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: cephadm 2021-07-31 2021-07-31T03:31:35.062 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: T03:31:33.708361+0000 mgr.x 2021-07-31T03:31:35.062 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: (mgr.24542) 4473 : cephadm 2021-07-31T03:31:35.062 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:31:35.062 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:31:35.063 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:31:35.063 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: yield 2021-07-31T03:31:35.063 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:31:35.063 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:31:35.064 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:31:35.064 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:31:35.064 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:31:35.065 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:31:35.065 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:31:35.065 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:31:35.065 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:31:35.065 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: task.result() 2021-07-31T03:31:35.066 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:31:35.066 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: 2021-07-31T03:31:35.066 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: cephadm 2021-07-31T03:31:33.682458+0000 mgr.x (mgr.24542) 4471 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:31:35.066 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: cephadm 2021-07-31T03:31 2021-07-31T03:31:35.067 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: :33.707162+0000 mgr.x (mgr.24542) 4472 : cephadm [ERR] 2021-07-31T03:31:35.067 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: 2021-07-31T03:31:35.067 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:31:35.067 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:31:35.067 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: yield 2021-07-31T03:31:35.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:31:35.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:31:35.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:31:35.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:31:35.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:31:35.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:31:35.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:31:35.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:31:35.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:31:35.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: task.result() 2021-07-31T03:31:35.070 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:31:35.070 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: cephadm 2021-07-31T03:31:33.708361+0000 mgr.x (mgr.24542) 2021-07-31T03:31:35.070 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: 4473 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:31:35.070 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:31:35.070 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:31:35.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: yield 2021-07-31T03:31:35.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:31:35.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:31:35.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:31:35.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:31:35.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:31:35.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:31:35.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:31:35.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:31:35.073 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:31:35.073 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: task.result() 2021-07-31T03:31:35.073 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:31:35.073 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: 2021-07-31T03:31:35.073 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:31:35.074 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: 2021-07-31T03:31:35.074 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:31:35.074 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:31:35.074 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:31:35.074 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:31:35.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:31:35.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:31:35.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:31:35.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:31:35.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:31:35.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:31:35.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:31:35.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:31:35.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:31:35.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:31:35.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:31:35.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:31:35.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:31:35.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:31:35.078 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: raise 2021-07-31T03:31:35.078 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:31:35.078 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:31:35.078 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:31:35.078 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:31:35.079 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:31:35.079 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: cephadm 2021-07-31T03:31:33.716238+0000 mgr.x (mgr.24542) 4474 : cephadm [ERR] 2021-07-31T03:31:35.079 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:31:35.079 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:31:35.079 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: yield 2021-07-31T03:31:35.080 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:31:35.080 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:31:35.080 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:31:35.080 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:31:35.080 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:31:35.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:31:35.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:31:35.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:31:35.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:31:35.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: task.result() 2021-07-31T03:31:35.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:31:35.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: cephadm 2021-07-31T03:31:33.716808+0000 mgr.x (mgr.24542) 4475 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:31:35.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:31:35.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:31:35.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: yield 2021-07-31T03:31:35.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:31:35.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:31:35.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:31:35.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:31:35.084 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:31:35.084 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:31:35.084 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:31:35.084 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:31:35.084 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:31:35.085 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: task.result() 2021-07-31T03:31:35.085 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:31:35.085 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: 2021-07-31T03:31:35.085 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:31:35.085 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: 2021-07-31T03:31:35.086 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:31:35.086 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:31:35.086 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:31:35.086 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:31:35.086 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:31:35.087 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:31:35.087 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:31:35.087 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:31:35.087 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:31:35.087 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:31:35.088 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:31:35.088 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:31:35.088 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:31:35.088 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:31:35.088 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:31:35.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:31:35.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:31:35.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:31:35.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: raise 2021-07-31T03:31:35.090 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:31:35.090 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:31:35.090 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:31:35.090 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:31:35.091 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:31:35.091 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:31:35.091 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:31:35.091 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: task.result() 2021-07-31T03:31:35.091 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:31:35.092 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: 2021-07-31T03:31:35.092 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:31:35.092 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: 2021-07-31T03:31:35.092 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:31:35.092 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:31:35.093 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:31:35.093 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:31:35.093 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:31:35.093 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:31:35.094 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:31:35.094 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:31:35.094 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:31:35.094 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:31:35.094 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:31:35.095 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:31:35.095 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:31:35.095 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:31:35.095 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:31:35.095 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:31:35.096 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:31:35.096 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:31:35.096 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: raise 2021-07-31T03:31:35.096 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:31:35.096 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:31:35.097 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:31:35.097 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:31:35.097 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:31:35.097 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: cephadm 2021-07-31T03:31:33.716238+0000 mgr.x (mgr.24542) 4474 : cephadm [ERR] 2021-07-31T03:31:35.097 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:31:35.098 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:31:35.098 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: yield 2021-07-31T03:31:35.098 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:31:35.099 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:31:35.099 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:31:35.099 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:31:35.099 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:31:35.099 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:31:35.100 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:31:35.100 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:31:35.100 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:31:35.101 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:31:35.101 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:31:35.101 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:31:35.101 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:31:35.101 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:31:35.102 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: task.result() 2021-07-31T03:31:35.102 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:31:35.102 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: cephadm 2021-07-31T03:31:33.716808+0000 mgr.x (mgr.24542) 4475 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:31:35.102 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:31:35.103 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:31:35.103 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: yield 2021-07-31T03:31:35.103 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:31:35.103 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:31:35.103 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:31:35.104 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:31:35.104 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:31:35.104 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:31:35.104 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:31:35.104 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:31:35.105 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:31:35.105 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: task.result() 2021-07-31T03:31:35.105 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:31:35.105 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: 2021-07-31T03:31:35.105 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:31:35.106 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: 2021-07-31T03:31:35.106 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:31:35.106 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:31:35.106 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:31:35.107 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:31:35.107 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:31:35.107 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:31:35.107 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:31:35.107 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:31:35.108 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:31:35.108 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:31:35.108 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:31:35.108 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:31:35.108 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:31:35.109 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:31:35.109 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:31:35.109 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:31:35.109 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:31:35.109 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:31:35.109 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: raise 2021-07-31T03:31:35.110 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:31:35.110 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:31:35.110 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:31:35.110 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:31:35.111 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:34 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:31:35.111 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:31:35.342 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:31:35.342 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 106m ago 107m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:31:35.342 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 104m ago 106m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:31:35.342 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 104m ago 110m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:31:35.343 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 106m ago 112m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:31:35.343 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 106m ago 112m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:31:35.343 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 104m ago 110m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:31:35.343 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 106m ago 110m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:31:35.343 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 106m ago 107m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:31:35.344 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 104m ago 107m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:31:35.344 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 106m ago 110m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:31:35.344 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 106m ago 109m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:31:35.344 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 106m ago 109m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:31:35.344 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 106m ago 109m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:31:35.345 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 104m ago 108m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:31:35.345 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 104m ago 108m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:31:35.345 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 104m ago 108m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:31:35.345 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 104m ago 107m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:31:35.346 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 104m ago 107m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:31:35.679 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:31:35.679 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:31:35.679 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:31:35.680 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:31:35.680 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:31:35.680 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:31:35.680 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:31:35.681 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:31:35.681 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:31:35.681 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:31:35.681 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:31:35.681 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:31:35.682 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:31:35.682 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:31:35.682 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:31:35.682 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:31:35.683 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:31:37.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:36 smithi154 conmon[34553]: audit 2021-07-31T03:31:35.053435+0000 mgr.x (mgr.24542) 4476 : audit [DBG] from='client.19686 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:31:37.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:36 smithi154 conmon[34553]: cluster 2021-07-31T03:31:35.284476+0000 mgr.x (mgr.24542) 4477 : cluster [DBG] pgmap v5138: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:37.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:36 smithi154 conmon[34553]: audit 2021-07-31T03:31:35.335004+0000 mgr.x (mgr.24542) 4478 : audit [DBG] from='client.19692 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:31:37.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:36 smithi154 conmon[34553]: audit 2021-07-31T03:31:35.680128+0000 mon.c (mon.1) 211 : 2021-07-31T03:31:37.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:36 smithi154 conmon[34553]: audit [DBG] from='client.? 172.21.15.45:0/2823665' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:31:37.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:36 smithi045 conmon[33655]: audit 2021-07-31T03:31:35.053435+0000 mgr.x (mgr.24542) 4476 : audit 2021-07-31T03:31:37.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:36 smithi045 conmon[33655]: [DBG] from='client.19686 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:31:37.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:36 smithi045 conmon[33655]: cluster 2021-07-31T03:31:35.284476+0000 mgr.x (mgr.24542) 4477 : cluster [DBG] 2021-07-31T03:31:37.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:36 smithi045 conmon[33655]: pgmap v5138: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:37.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:36 smithi045 conmon[33655]: audit 2021-07-31T03:31:37.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:36 smithi045 conmon[33655]: 2021-07-31T03:31:35.335004+0000 mgr.x (mgr.24542) 4478 : 2021-07-31T03:31:37.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:36 smithi045 conmon[33655]: audit [DBG] from='client.19692 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:31:37.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:36 smithi045 conmon[33655]: audit 2021-07 2021-07-31T03:31:37.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:36 smithi045 conmon[33655]: -31T03:31:35.680128+0000 mon.c (mon. 2021-07-31T03:31:37.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:36 smithi045 conmon[33655]: 1) 211 : audit [DBG] from='client.? 172.21.15.45:0/2823665' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:31:37.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:36 smithi045 conmon[39718]: audit 2021-07-31T03:31:35.053435+0000 mgr.x (mgr 2021-07-31T03:31:37.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:36 smithi045 conmon[39718]: .24542) 4476 : audit [DBG] from='client.19686 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:31:37.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:36 smithi045 conmon[39718]: cluster 2021-07-31T03:31:35.284476+0000 mgr.x (mgr.24542) 4477 : cluster 2021-07-31T03:31:37.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:36 smithi045 conmon[39718]: [DBG] pgmap v5138: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:37.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:36 smithi045 conmon[39718]: audit 2021-07-31T03:31:35.335004+0000 mgr.x ( 2021-07-31T03:31:37.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:36 smithi045 conmon[39718]: mgr.24542) 4478 : audit [DBG] from='client.19692 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:31:37.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:36 smithi045 conmon[39718]: audit 2021-07-31T03:31:37.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:36 smithi045 conmon[39718]: 2021-07-31T03:31:35.680128+0000 mon.c (mon.1) 211 : audit [DBG] from='client.? 172.21.15.45:0/2823665' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:31:39.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:38 smithi154 conmon[34553]: cluster 2021-07-31T03:31:37.284856+0000 mgr.x (mgr.24542) 4479 : cluster [DBG] pgmap v5139: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:39.024 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:38 smithi045 conmon[39718]: cluster 2021-07-31T03:31:37.284856+0000 2021-07-31T03:31:39.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:38 smithi045 conmon[39718]: mgr.x (mgr.24542) 4479 : cluster [DBG] pgmap v5139: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:39.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:38 smithi045 conmon[33655]: cluster 2021-07-31T03:31:37.284856+0000 mgr.x (mgr. 2021-07-31T03:31:39.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:38 smithi045 conmon[33655]: 24542) 4479 : cluster [DBG] pgmap v5139: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:41.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:40 smithi154 conmon[34553]: cluster 2021-07-31T03:31:39.285331+0000 mgr.x (mgr.24542) 4480 : cluster [DBG] pgmap v5140: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:41.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:40 smithi045 conmon[33655]: cluster 2021-07-31T03:31:39.285331+0000 mgr.x (mgr.24542 2021-07-31T03:31:41.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:40 smithi045 conmon[33655]: ) 4480 : cluster [DBG] pgmap v5140: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:41.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:40 smithi045 conmon[39718]: cluster 2021-07-31T03:31:39.285331+0000 mgr.x ( 2021-07-31T03:31:41.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:40 smithi045 conmon[39718]: mgr.24542) 4480 : cluster [DBG] pgmap v5140: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:43.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:42 smithi154 conmon[34553]: cluster 2021-07-31T03:31:41.285871+0000 mgr.x (mgr.24542) 4481 : cluster [DBG] pgmap v5141: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:43.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:42 smithi045 conmon[39718]: cluster 2021-07-31T03:31:41.285871+0000 mgr.x (mgr.24542) 4481 : cluster [DBG] pgmap v5141: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:43.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:42 smithi045 conmon[33655]: cluster 2021-07-31T03:31:41.285871+0000 mgr.x (mgr.24542) 4481 : 2021-07-31T03:31:43.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:42 smithi045 conmon[33655]: cluster [DBG] pgmap v5141: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:45.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:44 smithi154 conmon[34553]: cluster 2021-07-31T03:31:43.286275+0000 mgr.x (mgr.24542) 4482 : cluster [DBG] pgmap v5142: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:45.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:44 smithi045 conmon[33655]: cluster 2021-07-31T03:31:43.286275 2021-07-31T03:31:45.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:44 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 4482 : cluster [DBG] pgmap v5142: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:45.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:44 smithi045 conmon[39718]: cluster 2021-07-31T03:31:43.286275+0000 mgr.x (mgr.24542 2021-07-31T03:31:45.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:44 smithi045 conmon[39718]: ) 4482 : cluster [DBG] pgmap v5142: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:47.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:46 smithi154 conmon[34553]: cluster 2021-07-31T03:31:45.286807+0000 mgr.x (mgr.24542) 4483 : cluster [DBG] pgmap v5143: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:47.024 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:46 smithi045 conmon[33655]: cluster 2021-07-31T03:31:45.286807+0000 mgr.x (mgr.24542) 4483 : cluster [DBG] pgmap v5143: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:47.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:46 smithi045 conmon[39718]: cluster 2021-07-31T03:31:45 2021-07-31T03:31:47.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:46 smithi045 conmon[39718]: .286807+0000 mgr.x (mgr.24542) 4483 : cluster [DBG] pgmap v5143: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:49.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:48 smithi154 conmon[34553]: cluster 2021-07-31T03:31:47.287339+0000 mgr.x (mgr.24542) 4484 : cluster [DBG] 2021-07-31T03:31:49.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:48 smithi154 conmon[34553]: pgmap v5144: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:49.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:48 smithi045 conmon[39718]: cluster 2021-07-31T03:31:47.287339+0000 mgr.x (mgr.24542) 4484 : cluster [DBG] pgmap v5144: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:49.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:48 smithi045 conmon[33655]: cluster 2021-07-31T03:31:47.287339+0000 mgr.x (mgr 2021-07-31T03:31:49.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:48 smithi045 conmon[33655]: .24542) 4484 : cluster [DBG] pgmap v5144: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:51.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:50 smithi154 conmon[34553]: cluster 2021-07-31T03:31:49.287773+0000 mgr.x (mgr.24542) 4485 : cluster 2021-07-31T03:31:51.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:50 smithi154 conmon[34553]: [DBG] pgmap v5145: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:51.024 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:50 smithi045 conmon[39718]: cluster 2021-07-31T03:31:49.287773+0000 mgr.x 2021-07-31T03:31:51.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:50 smithi045 conmon[39718]: (mgr.24542) 4485 : cluster [DBG] pgmap v5145: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:51.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:50 smithi045 conmon[33655]: cluster 2021-07-31T03:31:49.287773+0000 mgr.x (mgr.24542) 2021-07-31T03:31:51.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:50 smithi045 conmon[33655]: 4485 : cluster [DBG] pgmap v5145: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:53.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:52 smithi154 conmon[34553]: cluster 2021-07-31T03:31:51.288274+0000 mgr.x (mgr.24542) 4486 : cluster 2021-07-31T03:31:53.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:52 smithi154 conmon[34553]: [DBG] pgmap v5146: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:53.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:52 smithi045 conmon[39718]: cluster 2021-07-31T03:31:51.288274+0000 mgr.x ( 2021-07-31T03:31:53.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:52 smithi045 conmon[39718]: mgr.24542) 4486 : cluster [DBG] pgmap v5146: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:53.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:52 smithi045 conmon[33655]: cluster 2021-07-31T03:31:51.288274+0000 mgr.x 2021-07-31T03:31:53.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:52 smithi045 conmon[33655]: (mgr.24542) 4486 : cluster [DBG] pgmap v5146: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:55.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:54 smithi154 conmon[34553]: cluster 2021-07-31T03:31:53.288643+0000 mgr.x (mgr.24542) 4487 2021-07-31T03:31:55.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:54 smithi154 conmon[34553]: : cluster [DBG] pgmap v5147: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:55.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:54 smithi045 conmon[39718]: cluster 2021-07-31T03:31:53.288643+0000 mgr.x 2021-07-31T03:31:55.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:54 smithi045 conmon[39718]: (mgr.24542) 4487 : cluster [DBG] pgmap v5147: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:55.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:54 smithi045 conmon[33655]: cluster 2021-07-31T03:31:53.288643+0000 mgr.x (mgr.24542) 4487 : cluster [DBG] pgmap v5147: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:55.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:54 smithi045 conmon[33655]: 2021-07-31T03:31:57.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:56 smithi154 conmon[34553]: cluster 2021-07-31T03:31:55.289206+0000 mgr.x (mgr.24542) 4488 : cluster [DBG] 2021-07-31T03:31:57.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:56 smithi154 conmon[34553]: pgmap v5148: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:57.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:56 smithi045 conmon[39718]: cluster 2021-07-31T03:31:55.289206+0000 mgr.x (mgr 2021-07-31T03:31:57.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:56 smithi045 conmon[39718]: .24542) 4488 : cluster [DBG] pgmap v5148: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:57.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:56 smithi045 conmon[33655]: cluster 2021-07-31T03:31:55.289206+0000 mgr.x (mgr.24542) 4488 2021-07-31T03:31:57.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:56 smithi045 conmon[33655]: : cluster [DBG] pgmap v5148: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:59.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:58 smithi154 conmon[34553]: cluster 2021-07-31T03:31:57.289655+0000 mgr.x (mgr.24542) 4489 : cluster [DBG] 2021-07-31T03:31:59.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:31:58 smithi154 conmon[34553]: pgmap v5149: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:59.024 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:58 smithi045 conmon[39718]: cluster 2021-07-31T03:31:57.289655+0000 mgr.x ( 2021-07-31T03:31:59.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:31:58 smithi045 conmon[39718]: mgr.24542) 4489 : cluster [DBG] pgmap v5149: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:31:59.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:31:58 smithi045 conmon[33655]: cluster 2021-07-31T03:31:57.289655+0000 mgr.x (mgr.24542) 4489 : cluster [DBG] pgmap v5149: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:01.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:00 smithi154 conmon[34553]: cluster 2021-07-31T03:31:59.290259+0000 mgr.x (mgr. 2021-07-31T03:32:01.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:00 smithi154 conmon[34553]: 24542) 4490 : cluster [DBG] pgmap v5150: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:01.024 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:00 smithi045 conmon[33655]: cluster 2021-07-31T03:31:59.290259+0000 mgr.x (mgr.24542) 4490 : cluster 2021-07-31T03:32:01.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:00 smithi045 conmon[33655]: [DBG] pgmap v5150: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:01.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:00 smithi045 conmon[39718]: cluster 2021-07-31T03:31:59.290259+0000 mgr.x ( 2021-07-31T03:32:01.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:00 smithi045 conmon[39718]: mgr.24542) 4490 : cluster [DBG] pgmap v5150: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:03.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:02 smithi154 conmon[34553]: cluster 2021-07-31T03:32:01.290655+0000 mgr.x (mgr.24542) 4491 : 2021-07-31T03:32:03.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:02 smithi154 conmon[34553]: cluster [DBG] pgmap v5151: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:03.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:02 smithi045 conmon[39718]: cluster 2021-07-31T03:32:01.290655+0000 mgr.x (mgr.24542) 4491 : cluster [DBG] pgmap v5151: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:03.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:02 smithi045 conmon[33655]: cluster 2021-07-31T03:32:01.290655+0000 mgr.x (mgr.24542) 4491 : cluster 2021-07-31T03:32:03.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:02 smithi045 conmon[33655]: [DBG] pgmap v5151: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:05.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:04 smithi154 conmon[34553]: cluster 2021-07-31T03:32:03.291033+0000 mgr.x (mgr.24542 2021-07-31T03:32:05.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:04 smithi154 conmon[34553]: ) 4492 : cluster [DBG] pgmap v5152: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:05.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:04 smithi045 conmon[39718]: cluster 2021-07-31T03:32:03. 2021-07-31T03:32:05.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:04 smithi045 conmon[39718]: 291033+0000 mgr.x (mgr.24542) 4492 : cluster [DBG] pgmap v5152: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:05.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:04 smithi045 conmon[33655]: cluster 2021-07-31T03:32:03.291033 2021-07-31T03:32:05.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:04 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 4492 : cluster [DBG] pgmap v5152: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:05.994 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:32:06.270 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:32:06.271 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 106m ago 107m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:32:06.271 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 104m ago 107m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:32:06.271 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 104m ago 111m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:32:06.272 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 106m ago 113m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:32:06.272 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 106m ago 113m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:32:06.272 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 104m ago 111m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:32:06.272 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 106m ago 111m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:32:06.272 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 106m ago 107m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:32:06.273 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 104m ago 107m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:32:06.273 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 106m ago 110m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:32:06.273 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 106m ago 110m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:32:06.273 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 106m ago 110m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:32:06.274 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 106m ago 109m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:32:06.274 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 104m ago 109m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:32:06.274 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 104m ago 109m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:32:06.274 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 104m ago 108m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:32:06.275 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 104m ago 108m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:32:06.275 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 104m ago 108m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:32:06.604 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:32:06.605 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:32:06.605 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:32:06.605 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:32:06.605 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:32:06.605 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:32:06.606 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:32:06.606 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:32:06.606 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:32:06.606 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:32:06.607 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:32:06.607 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:32:06.607 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:32:06.607 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:32:06.607 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:32:06.608 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:32:06.608 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:32:07.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:06 smithi154 conmon[34553]: cluster 2021-07-31T03:32:05.291545+0000 mgr.x (mgr.24542) 4493 : cluster [DBG] 2021-07-31T03:32:07.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:06 smithi154 conmon[34553]: pgmap v5153: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:07.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:06 smithi154 conmon[34553]: audit 2021-07-31T03:32:05.983744+0000 mgr.x (mgr.24542) 4494 : audit [DBG] from='client.19704 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:32:07.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:06 smithi154 conmon[34553]: audit 2021-07-31T03:32:06. 2021-07-31T03:32:07.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:06 smithi154 conmon[34553]: 605372+0000 mon.c (mon.1) 212 : audit [DBG] from='client.? 172.21.15.45:0/3398852375' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:32:07.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:06 smithi045 conmon[33655]: cluster 2021-07-31T03:32:05.291545+0000 mgr.x (mgr. 2021-07-31T03:32:07.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:06 smithi045 conmon[33655]: 24542) 4493 : cluster [DBG] pgmap v5153: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:07.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:06 smithi045 conmon[33655]: audit 2021-07-31T03:32:05. 2021-07-31T03:32:07.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:06 smithi045 conmon[33655]: 983744+0000 mgr.x (mgr.24542) 4494 : audit [DBG] from='client.19704 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:32:07.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:06 smithi045 conmon[33655]: 2021-07-31T03:32:07.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:06 smithi045 conmon[33655]: audit 2021-07-31T03:32:06. 2021-07-31T03:32:07.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:06 smithi045 conmon[33655]: 605372+0000 mon.c (mon.1) 212 : audit 2021-07-31T03:32:07.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:06 smithi045 conmon[33655]: [DBG] from='client.? 172.21.15.45:0/3398852375' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:32:07.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:06 smithi045 conmon[39718]: cluster 2021-07-31T03:32:05 2021-07-31T03:32:07.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:06 smithi045 conmon[39718]: .291545+0000 mgr.x (mgr.24542) 4493 : cluster [DBG] pgmap v5153: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:07.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:06 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T03:32:07.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:06 smithi045 conmon[39718]: 31T03:32:05. 2021-07-31T03:32:07.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:06 smithi045 conmon[39718]: 983744+0000 mgr.x (mgr.24542) 4494 : 2021-07-31T03:32:07.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:06 smithi045 conmon[39718]: audit [DBG] from='client.19704 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:32:07.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:06 smithi045 conmon[39718]: audit 2021 2021-07-31T03:32:07.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:06 smithi045 conmon[39718]: -07-31T03:32 2021-07-31T03:32:07.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:06 smithi045 conmon[39718]: :06.605372+0000 mon.c (mon 2021-07-31T03:32:07.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:06 smithi045 conmon[39718]: .1) 212 : audit [DBG] from='client.? 172.21.15.45:0/3398852375' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:32:08.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:07 smithi154 conmon[34553]: audit 2021-07-31T03:32:06.263447+0000 mgr.x (mgr.24542) 2021-07-31T03:32:08.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:07 smithi154 conmon[34553]: 4495 : audit [DBG] from='client.29650 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:32:08.024 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:07 smithi045 conmon[33655]: audit 2021-07-31T03:32:06.263447+0000 mgr.x (mgr.24542 2021-07-31T03:32:08.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:07 smithi045 conmon[33655]: ) 4495 : audit [DBG] from='client.29650 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:32:08.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:07 smithi045 conmon[39718]: audit 2021-07-31T03:32:06.263447+0000 mgr.x (mgr.24542) 4495 : audit 2021-07-31T03:32:08.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:07 smithi045 conmon[39718]: [DBG] from='client.29650 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:32:09.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:08 smithi154 conmon[34553]: cluster 2021-07-31T03:32:07.291911+0000 mgr.x (mgr.24542) 2021-07-31T03:32:09.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:08 smithi154 conmon[34553]: 4496 : cluster [DBG] pgmap v5154: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:09.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:08 smithi045 conmon[33655]: cluster 2021-07-31T03:32:07.291911+0000 mgr.x (mgr.24542 2021-07-31T03:32:09.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:08 smithi045 conmon[33655]: ) 4496 : cluster [DBG] pgmap v5154: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:09.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:08 smithi045 conmon[39718]: cluster 2021-07-31T03:32:07.291911+0000 mgr.x (mgr 2021-07-31T03:32:09.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:08 smithi045 conmon[39718]: .24542) 4496 : cluster [DBG] pgmap v5154: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:11.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:10 smithi154 conmon[34553]: cluster 2021-07-31T03:32:09.292400+0000 mgr.x (mgr.24542) 4497 : cluster [DBG] pgmap v5155: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:11.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:10 smithi045 conmon[33655]: cluster 2021-07-31T03:32:09.292400+0000 mgr.x ( 2021-07-31T03:32:11.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:10 smithi045 conmon[33655]: mgr.24542) 4497 : cluster [DBG] pgmap v5155: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:11.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:10 smithi045 conmon[39718]: cluster 2021-07-31T03:32:09.292400+0000 mgr.x 2021-07-31T03:32:11.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:10 smithi045 conmon[39718]: (mgr.24542) 4497 : cluster [DBG] pgmap v5155: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:13.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:12 smithi154 conmon[34553]: cluster 2021-07-31T03:32:11.292811+0000 mgr.x (mgr.24542) 4498 : cluster [DBG] 2021-07-31T03:32:13.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:12 smithi154 conmon[34553]: pgmap v5156: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:13.024 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:12 smithi045 conmon[33655]: cluster 2021-07-31T03:32:11.292811+0000 mgr.x (mgr 2021-07-31T03:32:13.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:12 smithi045 conmon[33655]: .24542) 4498 : cluster [DBG] pgmap v5156: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:13.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:12 smithi045 conmon[39718]: cluster 2021-07-31T03:32:11.292811+0000 mgr.x (mgr.24542) 4498 : cluster [DBG] pgmap v5156: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:15.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:14 smithi154 conmon[34553]: cluster 2021-07-31T03:32:13.293333+0000 mgr.x (mgr.24542) 4499 : cluster 2021-07-31T03:32:15.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:14 smithi154 conmon[34553]: [DBG] pgmap v5157: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:15.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:14 smithi045 conmon[33655]: cluster 2021-07-31T03:32:13.293333+0000 mgr.x (mgr.24542) 4499 2021-07-31T03:32:15.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:14 smithi045 conmon[33655]: : cluster [DBG] pgmap v5157: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:15.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:14 smithi045 conmon[39718]: cluster 2021-07-31T03:32:13.293333+0000 2021-07-31T03:32:15.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:14 smithi045 conmon[39718]: mgr.x (mgr.24542) 4499 : cluster [DBG] pgmap v5157: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:17.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:16 smithi154 conmon[34553]: cluster 2021-07-31T03:32:15.293811+0000 mgr.x (mgr.24542) 4500 : cluster 2021-07-31T03:32:17.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:16 smithi154 conmon[34553]: [DBG] pgmap v5158: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:17.024 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:16 smithi045 conmon[33655]: cluster 2021-07-31T03:32:15.293811+0000 2021-07-31T03:32:17.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:16 smithi045 conmon[33655]: mgr.x (mgr.24542) 4500 : cluster [DBG] pgmap v5158: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:17.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:16 smithi045 conmon[39718]: cluster 2021-07-31T03:32:15. 2021-07-31T03:32:17.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:16 smithi045 conmon[39718]: 293811+0000 mgr.x (mgr.24542) 4500 : cluster [DBG] pgmap v5158: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:19.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:18 smithi154 conmon[34553]: cluster 2021-07-31T03:32:17.294348+0000 mgr.x (mgr.24542) 4501 : cluster [DBG] pgmap v5159: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:19.024 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:18 smithi045 conmon[39718]: cluster 2021-07-31T03:32:17.294348+0000 mgr.x (mgr.24542) 2021-07-31T03:32:19.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:18 smithi045 conmon[39718]: 4501 : cluster [DBG] pgmap v5159: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:19.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:18 smithi045 conmon[33655]: cluster 2021-07-31T03:32:17.294348+0000 mgr.x (mgr.24542) 4501 : cluster 2021-07-31T03:32:19.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:18 smithi045 conmon[33655]: [DBG] pgmap v5159: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:21.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:20 smithi154 conmon[34553]: cluster 2021-07-31T03:32:19.294881+0000 mgr.x (mgr.24542) 4502 : cluster [DBG] pgmap v5160: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:21.024 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:20 smithi045 conmon[39718]: cluster 2021-07-31T03:32:19.294881+0000 mgr.x (mgr.24542) 4502 2021-07-31T03:32:21.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:20 smithi045 conmon[39718]: : cluster [DBG] pgmap v5160: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:21.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:20 smithi045 conmon[33655]: cluster 2021-07-31T03:32:19.294881+0000 mgr.x (mgr.24542) 2021-07-31T03:32:21.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:20 smithi045 conmon[33655]: 4502 : cluster [DBG] pgmap v5160: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:23.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:22 smithi154 conmon[34553]: cluster 2021-07-31T03:32:21.295325+0000 mgr.x (mgr.24542) 4503 : cluster [DBG] pgmap v5161: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:23.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:22 smithi045 conmon[33655]: cluster 2021-07-31T03:32:21.295325+0000 mgr.x (mgr.24542) 2021-07-31T03:32:23.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:22 smithi045 conmon[33655]: 4503 : cluster [DBG] pgmap v5161: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:23.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:22 smithi045 conmon[39718]: cluster 2021-07-31T03:32:21.295325+0000 mgr.x (mgr.24542 2021-07-31T03:32:23.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:22 smithi045 conmon[39718]: ) 4503 : cluster [DBG] pgmap v5161: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:25.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:24 smithi154 conmon[34553]: cluster 2021-07-31T03:32:23.295704+0000 mgr.x (mgr.24542) 4504 : cluster [DBG] pgmap v5162: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:25.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:24 smithi045 conmon[33655]: cluster 2021-07-31T03:32:23.295704+0000 mgr.x (mgr.24542) 4504 : cluster [DBG] pgmap v5162: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:25.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:24 smithi045 conmon[39718]: cluster 2021-07-31T03:32:23.295704+0000 mgr.x (mgr.24542) 4504 : 2021-07-31T03:32:25.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:24 smithi045 conmon[39718]: cluster [DBG] pgmap v5162: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:27.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:26 smithi154 conmon[34553]: cluster 2021-07-31T03:32:25.296279+0000 mgr.x (mgr.24542) 4505 : cluster [DBG] pgmap v5163: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:27.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:26 smithi045 conmon[39718]: cluster 2021-07-31T03:32:25.296279+0000 mgr.x (mgr.24542) 4505 2021-07-31T03:32:27.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:26 smithi045 conmon[39718]: : cluster [DBG] pgmap v5163: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:27.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:26 smithi045 conmon[33655]: cluster 2021-07-31T03:32:25.296279+0000 mgr.x (mgr.24542) 4505 : cluster 2021-07-31T03:32:27.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:26 smithi045 conmon[33655]: [DBG] pgmap v5163: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:29.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:28 smithi154 conmon[34553]: cluster 2021-07-31T03:32:27.296677+0000 mgr.x (mgr.24542) 4506 : cluster [DBG] pgmap v5164: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:29.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:28 smithi154 conmon[34553]: audit 2021-07-31T03:32:28.352193+0000 mon.b (mon.2) 600 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:32:29.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:28 smithi154 conmon[34553]: audit 2021-07-31T03:32:28.352434+0000 mon.b (mon.2) 601 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:32:29.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:28 smithi154 conmon[34553]: audit 2021-07-31T03:32:28.352918 2021-07-31T03:32:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:28 smithi154 conmon[34553]: +0000 mon.a (mon.0) 3320 : audit 2021-07-31T03:32:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:28 smithi154 conmon[34553]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:32:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:28 smithi154 conmon[34553]: audit 2021-07-31 2021-07-31T03:32:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:28 smithi154 conmon[34553]: T03:32:28.353225+0000 mon.a 2021-07-31T03:32:29.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:28 smithi154 conmon[34553]: (mon.0) 3321 : audit [INF] 2021-07-31T03:32:29.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:28 smithi154 conmon[34553]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:32:29.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:28 smithi045 conmon[33655]: cluster 2021-07-31T03:32:27.296677+0000 mgr.x (mgr.24542) 4506 : cluster [DBG] pgmap v5164: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:29.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:28 smithi045 conmon[33655]: audit 2021-07-31T03:32:28.352193+0000 mon.b (mon.2) 600 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:32:29.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:28 smithi045 conmon[33655]: audit 2021-07-31T03:32:28.352434+0000 mon.b (mon.2) 601 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:32:29.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:28 smithi045 conmon[33655]: audit 2021-07-31T03:32:28.352918+0000 mon.a (mon.0) 3320 : audit [INF] 2021-07-31T03:32:29.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:28 smithi045 conmon[33655]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:32:29.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:28 smithi045 conmon[33655]: audit 2021-07-31T03:32:28. 2021-07-31T03:32:29.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:28 smithi045 conmon[33655]: 353225+0000 mon.a (mon.0) 3321 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:32:29.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:28 smithi045 conmon[39718]: cluster 2021-07-31T03:32:27.296677+0000 2021-07-31T03:32:29.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:28 smithi045 conmon[39718]: mgr.x (mgr.24542) 4506 : cluster [DBG] pgmap v5164: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:29.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:28 smithi045 conmon[39718]: audit 2021 2021-07-31T03:32:29.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:28 smithi045 conmon[39718]: -07-31T03:32:28. 2021-07-31T03:32:29.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:28 smithi045 conmon[39718]: 352193+0000 mon.b (mon.2) 600 : 2021-07-31T03:32:29.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:28 smithi045 conmon[39718]: audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:32:29.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:28 smithi045 conmon[39718]: audit 2021- 2021-07-31T03:32:29.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:28 smithi045 conmon[39718]: 07-31T03:32:28.352434+0000 2021-07-31T03:32:29.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:28 smithi045 conmon[39718]: mon.b (mon.2) 601 : audit 2021-07-31T03:32:29.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:28 smithi045 conmon[39718]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:32:29.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:28 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T03:32:29.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:28 smithi045 conmon[39718]: 03:32:28.352918+0000 mon.a (mon 2021-07-31T03:32:29.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:28 smithi045 conmon[39718]: .0) 3320 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:32:29.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:28 smithi045 conmon[39718]: audit 2021-07-31T03:32: 2021-07-31T03:32:29.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:28 smithi045 conmon[39718]: 28.353225+0000 mon.a (mon.0) 3321 : 2021-07-31T03:32:29.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:28 smithi045 conmon[39718]: audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:32:31.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:30 smithi154 conmon[34553]: cluster 2021-07-31T03:32:29.297076+0000 mgr.x (mgr.24542) 4507 : cluster [DBG] pgmap v5165: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:31.024 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:30 smithi045 conmon[33655]: cluster 2021- 2021-07-31T03:32:31.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:30 smithi045 conmon[33655]: 07-31T03:32:29.297076+0000 mgr.x (mgr.24542) 4507 : cluster [DBG] pgmap v5165: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:31.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:30 smithi045 conmon[39718]: cluster 2021-07-31T03:32:29.297076+0000 mgr.x 2021-07-31T03:32:31.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:30 smithi045 conmon[39718]: (mgr.24542) 4507 : cluster [DBG] pgmap v5165: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:33.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:32 smithi154 conmon[34553]: cluster 2021-07-31T03:32:31.297551+0000 mgr.x (mgr.24542) 4508 : cluster [DBG] pgmap v5166: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:33.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:32 smithi154 conmon[34553]: 2021-07-31T03:32:33.024 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:32 smithi045 conmon[39718]: cluster 2021-07-31T03:32:31.297551+0000 mgr.x (mgr 2021-07-31T03:32:33.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:32 smithi045 conmon[39718]: .24542) 4508 : cluster [DBG] pgmap v5166: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:33.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:32 smithi045 conmon[33655]: cluster 2021-07-31T03:32:31.297551+0000 mgr.x (mgr.24542) 4508 : cluster 2021-07-31T03:32:33.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:32 smithi045 conmon[33655]: [DBG] pgmap v5166: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:34.002 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: debug 2021-07-31T03:32:33.761+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:32:34.002 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:32:34.003 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:32:34.003 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: yield 2021-07-31T03:32:34.003 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:32:34.003 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:32:34.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:32:34.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:32:34.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:32:34.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:32:34.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:32:34.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:32:34.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:32:34.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: task.result() 2021-07-31T03:32:34.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:32:34.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: debug 2021-07-31T03:32:33.772+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:32:34.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:32:34.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:32:34.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: yield 2021-07-31T03:32:34.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:32:34.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:32:34.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:32:34.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:32:34.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:32:34.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:32:34.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:32:34.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:32:34.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:32:34.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: task.result() 2021-07-31T03:32:34.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:32:34.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: debug 2021-07-31T03:32:33.797+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:32:34.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:32:34.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:32:34.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: yield 2021-07-31T03:32:34.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:32:34.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:32:34.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:32:34.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:32:34.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:32:34.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:32:34.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:32:34.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:32:34.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:32:34.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: task.result() 2021-07-31T03:32:34.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:32:34.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: debug 2021-07-31T03:32:33.797+0000 7fcb94162700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:32:34.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:32:34.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:32:34.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: yield 2021-07-31T03:32:34.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:32:34.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:32:34.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:32:34.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:32:34.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:32:34.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:32:34.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:32:34.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:32:34.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:32:34.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: task.result() 2021-07-31T03:32:34.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:32:34.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: 2021-07-31T03:32:34.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:32:34.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: 2021-07-31T03:32:34.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:32:34.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:32:34.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:32:34.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:32:34.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:32:34.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:32:34.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:32:34.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:32:34.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:32:34.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:32:34.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:32:34.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:32:34.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:32:34.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:32:34.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:32:34.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:32:34.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:32:34.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:32:34.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: raise 2021-07-31T03:32:34.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:32:34.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:32:34.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:32:34.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:32:34.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:32:34.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: debug 2021-07-31T03:32:33.806+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:32:34.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:32:34.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:32:34.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: yield 2021-07-31T03:32:34.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:32:34.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:32:34.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:32:34.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:32:34.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:32:34.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:32:34.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:32:34.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:32:34.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:32:34.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: task.result() 2021-07-31T03:32:34.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:32:34.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: debug 2021-07-31T03:32:33.806+0000 7fcb93961700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:32:34.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:32:34.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:32:34.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: yield 2021-07-31T03:32:34.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:32:34.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:32:34.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:32:34.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:32:34.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:32:34.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:32:34.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:32:34.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:32:34.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:32:34.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: task.result() 2021-07-31T03:32:34.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:32:34.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: 2021-07-31T03:32:34.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:32:34.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: 2021-07-31T03:32:34.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:32:34.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:32:34.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:32:34.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:32:34.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:32:34.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:32:34.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:32:34.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:32:34.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:32:34.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:32:34.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:32:34.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:32:34.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:32:34.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:32:34.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:32:34.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:32:34.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:32:34.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:32:34.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: raise 2021-07-31T03:32:34.035 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:32:34.035 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:32:34.035 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:32:34.035 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:32:34.036 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:32:33 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:32:35.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: cluster 2021-07-31T03:32:33.298141+0000 mgr.x (mgr.24542) 4509 : cluster [DBG] pgmap v5167: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:35.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: audit 2021-07-31T03:32:33.717568+0000 mon.b (mon.2) 602 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:32:35.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: audit 2021-07-31T03:32:33.718815+0000 mon.b (mon.2) 603 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:32:35.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: audit 2021-07-31T03:32:33.719565+0000 mon.b ( 2021-07-31T03:32:35.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: mon.2) 604 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:32:35.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: cephadm 2021-07 2021-07-31T03:32:35.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: -31T03:32:33.720238+0000 mgr.x ( 2021-07-31T03:32:35.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: mgr.24542) 4510 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:32:35.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: cephadm 2021 2021-07-31T03:32:35.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: -07-31T03:32:33.762190+0000 2021-07-31T03:32:35.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: mgr.x (mgr.24542) 4511 : cephadm [ERR] 2021-07-31T03:32:35.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:32:35.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:32:35.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: yield 2021-07-31T03:32:35.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:32:35.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:32:35.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:32:35.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:32:35.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:32:35.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:32:35.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:32:35.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:32:35.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:32:35.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: task.result() 2021-07-31T03:32:35.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:32:35.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: 2021-07-31T03:32:35.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: cephadm 2021-07-31T03:32:33.762569+0000 mgr.x (mgr.24542) 4512 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:32:35.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: cephadm 2021-07-31T03:32:33.773513+0000 mgr.x (mgr 2021-07-31T03:32:35.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: .24542) 4513 : cephadm [ERR] 2021-07-31T03:32:35.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:32:35.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:32:35.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: yield 2021-07-31T03:32:35.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:32:35.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:32:35.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:32:35.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:32:35.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:32:35.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:32:35.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:32:35.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:32:35.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:32:35.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: task.result() 2021-07-31T03:32:35.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:32:35.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: cephadm 2021-07-31T03:32:33.774414+0000 mgr.x (mgr.24542) 4514 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:32:35.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: cephadm 2021-07-31T03:32:33.798505+0000 mgr.x (mgr.24542) 4515 : cephadm [ERR] 2021-07-31T03:32:35.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:32:35.035 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:32:35.037 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: yield 2021-07-31T03:32:35.037 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:32:35.038 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:32:35.038 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:32:35.039 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:32:35.039 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:32:35.040 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:32:35.040 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:32:35.041 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:32:35.041 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:32:35.042 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: task.result() 2021-07-31T03:32:35.042 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:32:35.043 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: cephadm 2021-07-31T03:32:33.798875+0000 mgr.x (mgr.24542) 4516 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:32:35.043 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:32:35.044 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:32:35.044 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: yield 2021-07-31T03:32:35.045 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:32:35.045 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:32:35.047 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:32:35.047 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:32:35.047 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:32:35.048 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:32:35.048 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:32:35.048 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:32:35.049 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:32:35.049 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: task.result() 2021-07-31T03:32:35.049 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:32:35.050 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: 2021-07-31T03:32:35.050 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:32:35.050 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: 2021-07-31T03:32:35.051 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:32:35.051 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:32:35.051 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:32:35.052 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:32:35.052 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:32:35.052 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:32:35.053 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:32:35.053 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:32:35.053 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:32:35.054 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:32:35.054 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:32:35.054 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:32:35.055 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:32:35.055 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:32:35.055 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:32:35.056 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:32:35.056 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:32:35.056 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:32:35.057 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: raise 2021-07-31T03:32:35.057 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:32:35.057 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:32:35.058 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:32:35.058 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:32:35.058 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:32:35.059 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: cephadm 2021-07-31T03:32:33.807525+0000 mgr.x ( 2021-07-31T03:32:35.059 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: mgr.24542) 4517 : cephadm [ERR] 2021-07-31T03:32:35.059 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:32:35.060 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:32:35.060 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: yield 2021-07-31T03:32:35.060 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:32:35.061 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:32:35.061 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:32:35.061 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:32:35.062 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:32:35.062 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:32:35.062 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:32:35.062 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:32:35.063 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:32:35.063 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: task.result() 2021-07-31T03:32:35.063 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:32:35.064 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: cephadm 2021-07-31T03:32:33.807900+0000 mgr.x (mgr.24542) 4518 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:32:35.064 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:32:35.064 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:32:35.065 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: yield 2021-07-31T03:32:35.065 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:32:35.065 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:32:35.066 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:32:35.066 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:32:35.066 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:32:35.066 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:32:35.067 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:32:35.067 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:32:35.068 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:32:35.068 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: task.result() 2021-07-31T03:32:35.068 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:32:35.069 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: 2021-07-31T03:32:35.069 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:32:35.069 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: 2021-07-31T03:32:35.070 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:32:35.070 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:32:35.070 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:32:35.071 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:32:35.071 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:32:35.071 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:32:35.072 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:32:35.072 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:32:35.072 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:32:35.072 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:32:35.073 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:32:35.073 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:32:35.073 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:32:35.074 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:32:35.074 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:32:35.074 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:32:35.075 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:32:35.075 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:32:35.075 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: raise 2021-07-31T03:32:35.076 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:32:35.076 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:32:35.077 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:32:35.077 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:32:35.078 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:34 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:32:35.079 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: cluster 2021-07-31T03:32:33.298141 2021-07-31T03:32:35.079 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 4509 : cluster [DBG] pgmap v5167: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:35.080 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: audit 2021-07-31T03:32 2021-07-31T03:32:35.080 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: :33.717568+0000 mon.b (mon.2) 602 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:32:35.081 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: audit 2021-07-31T03:32:35.081 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: 2021-07-31T03:32:33.718815+0000 mon.b (mon.2) 603 : audit [DBG] 2021-07-31T03:32:35.081 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:32:35.082 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: audit 2021- 2021-07-31T03:32:35.082 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: 07-31T03:32:33.719565+0000 mon.b (mon.2) 604 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:32:35.082 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: 2021-07-31T03:32:35.083 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: cephadm 2021-07- 2021-07-31T03:32:35.083 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: 31T03:32:33.720238+0000 mgr.x (mgr.24542) 4510 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:32:35.083 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: 2021-07-31T03:32:35.084 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: cephadm 2021-07-31 2021-07-31T03:32:35.084 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: T03:32:33.762190+0000 mgr.x (mgr.24542) 4511 : cephadm [ERR] 2021-07-31T03:32:35.084 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:32:35.085 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:32:35.085 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: yield 2021-07-31T03:32:35.086 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:32:35.086 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:32:35.086 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:32:35.087 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:32:35.087 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:32:35.087 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:32:35.088 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:32:35.088 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:32:35.089 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:32:35.089 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: task.result() 2021-07-31T03:32:35.089 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:32:35.089 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: cephadm 2021-07-31T03:32:33.762569+0000 mgr.x (mgr.24542) 4512 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:32:35.090 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: cephadm 2021-07-31T03:32:33.773513+0000 mgr.x (mgr.24542) 4513 : cephadm [ERR] 2021-07-31T03:32:35.090 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:32:35.090 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:32:35.091 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: yield 2021-07-31T03:32:35.091 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:32:35.091 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:32:35.092 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:32:35.092 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:32:35.092 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:32:35.093 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:32:35.093 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:32:35.093 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:32:35.094 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:32:35.094 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: task.result() 2021-07-31T03:32:35.094 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:32:35.095 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: cephadm 2021-07-31T03:32:35.095 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: 2021-07-31T03:32:33.774414+0000 mgr.x (mgr.24542) 4514 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:32:35.095 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: cephadm 2021-07-31T03:32:33.798505+0000 mgr.x (mgr.24542) 4515 : cephadm [ERR] 2021-07-31T03:32:35.096 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:32:35.096 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:32:35.097 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: yield 2021-07-31T03:32:35.097 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:32:35.097 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:32:35.098 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:32:35.098 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:32:35.098 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:32:35.099 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:32:35.099 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:32:35.099 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:32:35.100 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:32:35.100 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: task.result() 2021-07-31T03:32:35.100 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:32:35.101 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: cephadm 2021-07-31T03:32:33. 2021-07-31T03:32:35.101 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: 798875+0000 mgr.x (mgr.24542) 4516 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:32:35.101 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:32:35.102 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:32:35.102 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: yield 2021-07-31T03:32:35.102 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:32:35.103 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:32:35.103 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:32:35.103 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:32:35.104 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:32:35.104 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:32:35.104 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:32:35.105 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:32:35.105 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:32:35.105 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: task.result() 2021-07-31T03:32:35.106 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:32:35.106 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: cluster 2021-07-31T03:32:33.298141+0000 mgr.x (mgr.24542) 4509 : cluster [DBG] pgmap v5167: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:35.107 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: audit 2021-07-31T03:32:33.717568+0000 mon.b (mon.2) 602 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:32:35.107 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: audit 2021-07-31T03:32:33.718815+0000 mon.b (mon.2) 603 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:32:35.107 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: audit 2021-07- 2021-07-31T03:32:35.108 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: 31T03:32:33.719565 2021-07-31T03:32:35.108 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: +0000 mon.b (mon.2) 604 2021-07-31T03:32:35.109 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:32:35.109 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: cephadm 2021- 2021-07-31T03:32:35.109 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: 07-31T03:32:33.720238+0000 2021-07-31T03:32:35.109 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: mgr.x (mgr.24542) 4510 : cephadm 2021-07-31T03:32:35.110 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:32:35.110 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: cephadm 2021 2021-07-31T03:32:35.110 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: -07-31T03:32:33. 2021-07-31T03:32:35.111 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: 762190+0000 mgr.x (mgr.24542 2021-07-31T03:32:35.111 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: ) 4511 : cephadm [ERR] 2021-07-31T03:32:35.111 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:32:35.112 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:32:35.112 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: yield 2021-07-31T03:32:35.112 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:32:35.113 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:32:35.113 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:32:35.113 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:32:35.113 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:32:35.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:32:35.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:32:35.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:32:35.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:32:35.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: task.result() 2021-07-31T03:32:35.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:32:35.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: 2021-07-31T03:32:35.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: cephadm 2021-07-31T03:32:33.762569+0000 mgr.x (mgr.24542) 2021-07-31T03:32:35.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: 4512 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:32:35.117 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: cephadm 2021 2021-07-31T03:32:35.117 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: -07-31T03:32:33. 2021-07-31T03:32:35.117 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: 773513+0000 mgr.x (mgr.24542) 2021-07-31T03:32:35.118 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: 4513 : cephadm [ERR] 2021-07-31T03:32:35.118 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:32:35.118 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:32:35.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: yield 2021-07-31T03:32:35.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:32:35.120 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: 2021-07-31T03:32:35.120 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:32:35.120 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: 2021-07-31T03:32:35.121 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:32:35.121 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:32:35.121 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:32:35.122 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:32:35.122 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:32:35.122 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:32:35.123 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:32:35.123 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:32:35.123 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:32:35.124 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:32:35.124 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:32:35.124 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:32:35.125 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:32:35.125 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:32:35.125 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:32:35.126 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:32:35.126 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:32:35.126 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:32:35.127 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: raise 2021-07-31T03:32:35.127 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:32:35.127 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:32:35.128 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:32:35.128 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:32:35.128 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:32:35.129 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: cephadm 2021-07-31T03:32:33.807525+0000 mgr.x (mgr.24542) 4517 : cephadm [ERR] 2021-07-31T03:32:35.129 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:32:35.129 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:32:35.130 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: yield 2021-07-31T03:32:35.130 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:32:35.130 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:32:35.130 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:32:35.131 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:32:35.131 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:32:35.131 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:32:35.132 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:32:35.132 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:32:35.132 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:32:35.133 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: task.result() 2021-07-31T03:32:35.133 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:32:35.134 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:32:35.134 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:32:35.134 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:32:35.135 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:32:35.135 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:32:35.135 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:32:35.136 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:32:35.136 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: task.result() 2021-07-31T03:32:35.136 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:32:35.137 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: cephadm 2021-07-31T03:32:33.774414+0000 mgr.x (mgr.24542) 2021-07-31T03:32:35.137 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: 4514 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:32:35.137 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: cephadm 2021 2021-07-31T03:32:35.137 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: -07-31T03:32:33. 2021-07-31T03:32:35.138 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: 798505+0000 mgr.x (mgr.24542 2021-07-31T03:32:35.138 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: ) 4515 : cephadm [ERR] 2021-07-31T03:32:35.138 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:32:35.139 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:32:35.139 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: yield 2021-07-31T03:32:35.139 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:32:35.140 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:32:35.140 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:32:35.140 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:32:35.141 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:32:35.141 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:32:35.141 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:32:35.141 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:32:35.142 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:32:35.142 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: task.result() 2021-07-31T03:32:35.142 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:32:35.143 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: 2021-07-31T03:32:35.143 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: cephadm 2021-07-31T03:32:33.798875 2021-07-31T03:32:35.143 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 4516 : cephadm 2021-07-31T03:32:35.144 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:32:35.144 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:32:35.144 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:32:35.145 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: yield 2021-07-31T03:32:35.145 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:32:35.145 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:32:35.146 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:32:35.146 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:32:35.146 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:32:35.147 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:32:35.148 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:32:35.148 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: cephadm 2021-07-31T03:32:33.807900+0000 mgr.x (mgr.24542) 4518 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:32:35.148 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:32:35.149 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:32:35.149 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: yield 2021-07-31T03:32:35.149 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:32:35.150 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:32:35.150 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:32:35.150 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:32:35.151 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:32:35.151 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:32:35.151 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:32:35.152 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:32:35.152 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:32:35.152 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: task.result() 2021-07-31T03:32:35.153 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:32:35.153 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: 2021-07-31T03:32:35.153 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:32:35.154 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: 2021-07-31T03:32:35.154 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:32:35.154 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:32:35.155 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:32:35.155 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:32:35.155 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:32:35.156 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:32:35.156 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:32:35.156 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:32:35.157 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:32:35.157 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:32:35.157 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:32:35.157 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:32:35.158 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:32:35.158 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:32:35.158 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:32:35.159 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:32:35.159 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:32:35.159 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:32:35.160 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: raise 2021-07-31T03:32:35.160 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:32:35.160 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:32:35.161 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:32:35.161 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:32:35.161 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:32:35.163 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:32:35.163 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:32:35.163 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:32:35.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: task.result() 2021-07-31T03:32:35.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:32:35.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: 2021-07-31T03:32:35.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:32:35.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: 2021-07-31T03:32:35.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:32:35.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:32:35.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:32:35.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:32:35.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:32:35.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:32:35.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:32:35.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:32:35.168 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:32:35.168 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:32:35.168 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:32:35.169 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:32:35.169 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:32:35.169 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:32:35.170 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:32:35.170 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:32:35.171 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:32:35.171 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:32:35.171 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: raise 2021-07-31T03:32:35.171 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:32:35.172 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:32:35.172 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:32:35.172 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:32:35.173 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:32:35.173 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: cephadm 2021-07-31T03:32:33.807525+0000 mgr.x (mgr.24542) 4517 : cephadm [ERR] 2021-07-31T03:32:35.173 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:32:35.174 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:32:35.174 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: yield 2021-07-31T03:32:35.174 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:32:35.175 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:32:35.175 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:32:35.175 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:32:35.176 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:32:35.176 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:32:35.176 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:32:35.177 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:32:35.177 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:32:35.177 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: task.result() 2021-07-31T03:32:35.177 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:32:35.178 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: cephadm 2021-07-31T03:32:33.807900+0000 mgr.x (mgr.24542) 4518 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:32:35.178 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:32:35.178 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:32:35.179 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: yield 2021-07-31T03:32:35.179 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:32:35.179 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:32:35.180 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:32:35.180 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:32:35.180 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:32:35.181 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:32:35.181 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:32:35.181 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:32:35.182 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:32:35.182 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: task.result() 2021-07-31T03:32:35.182 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:32:35.183 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: 2021-07-31T03:32:35.183 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:32:35.183 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: 2021-07-31T03:32:35.184 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:32:35.184 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:32:35.184 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:32:35.185 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:32:35.185 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:32:35.185 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:32:35.186 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:32:35.186 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:32:35.186 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:32:35.186 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:32:35.187 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:32:35.187 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:32:35.187 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:32:35.188 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:32:35.188 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:32:35.188 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:32:35.189 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:32:35.189 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:32:35.189 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: raise 2021-07-31T03:32:35.190 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:32:35.190 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:32:35.190 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:32:35.190 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:32:35.191 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:34 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:32:36.920 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:32:37.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:36 smithi154 conmon[34553]: cluster 2021-07-31T03:32:35.298716+0000 mgr.x (mgr.24542) 4519 : cluster [DBG] pgmap v5168: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:37.024 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:36 smithi045 conmon[33655]: cluster 2021-07-31T03:32:35.298716+0000 2021-07-31T03:32:37.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:36 smithi045 conmon[33655]: mgr.x (mgr.24542) 4519 : cluster [DBG] pgmap v5168: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:37.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:36 smithi045 conmon[39718]: cluster 2021-07-31T03:32:35 2021-07-31T03:32:37.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:36 smithi045 conmon[39718]: .298716+0000 mgr.x (mgr.24542) 4519 : cluster [DBG] pgmap v5168: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:37.197 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:32:37.197 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 107m ago 108m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:32:37.198 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 105m ago 107m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:32:37.198 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 105m ago 111m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:32:37.198 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 107m ago 113m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:32:37.198 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 107m ago 113m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:32:37.199 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 105m ago 111m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:32:37.199 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 107m ago 111m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:32:37.199 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 107m ago 108m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:32:37.199 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 105m ago 108m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:32:37.199 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 107m ago 111m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:32:37.200 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 107m ago 110m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:32:37.200 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 107m ago 110m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:32:37.200 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 107m ago 110m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:32:37.200 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 105m ago 109m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:32:37.200 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 105m ago 109m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:32:37.201 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 105m ago 109m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:32:37.201 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 105m ago 108m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:32:37.201 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 105m ago 108m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:32:37.533 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:32:37.533 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:32:37.533 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:32:37.534 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:32:37.534 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:32:37.534 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:32:37.534 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:32:37.535 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:32:37.535 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:32:37.535 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:32:37.535 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:32:37.535 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:32:37.536 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:32:37.536 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:32:37.536 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:32:37.536 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:32:37.537 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:32:38.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:37 smithi154 conmon[34553]: audit 2021-07-31T03:32:36.909269+0000 mgr.x (mgr.24542) 4520 : audit [DBG] from='client.19722 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:32:38.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:37 smithi154 conmon[34553]: audit 2021-07-31T03:32:37.534183 2021-07-31T03:32:38.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:37 smithi154 conmon[34553]: +0000 mon.a (mon.0) 3322 : audit [DBG] from='client.? 172.21.15.45:0/2820522792' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:32:38.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:37 smithi045 conmon[33655]: audit 2021-07-31T03:32:36 2021-07-31T03:32:38.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:37 smithi045 conmon[33655]: .909269+0000 mgr.x (mgr.24542) 4520 : audit [DBG] from='client.19722 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:32:38.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:37 smithi045 conmon[33655]: audit 2021-07-31T03:32 2021-07-31T03:32:38.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:37 smithi045 conmon[33655]: :37.534183+0000 mon.a (mon.0) 3322 : audit [DBG] from='client.? 172.21.15.45:0/2820522792' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:32:38.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:37 smithi045 conmon[39718]: audit 2021-07-31T03:32:36.909269+0000 mgr.x (mgr 2021-07-31T03:32:38.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:37 smithi045 conmon[39718]: .24542) 4520 : audit [DBG] from='client.19722 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:32:38.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:37 smithi045 conmon[39718]: audit 2021-07-31T03:32:37.534183 2021-07-31T03:32:38.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:37 smithi045 conmon[39718]: +0000 mon.a (mon.0) 3322 : audit [DBG] from='client.? 172.21.15.45:0/2820522792' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:32:39.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:38 smithi154 conmon[34553]: audit 2021-07-31T03:32:37.190814+0000 mgr.x (mgr.24542) 4521 : audit [DBG] from='client.29668 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:32:39.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:38 smithi154 conmon[34553]: cluster 2021-07-31T03:32:37 2021-07-31T03:32:39.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:38 smithi154 conmon[34553]: .299075+0000 mgr.x (mgr.24542) 4522 : cluster [DBG] pgmap v5169: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:39.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:38 smithi045 conmon[39718]: audit 2021-07-31T03:32:37 2021-07-31T03:32:39.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:38 smithi045 conmon[39718]: .190814+0000 mgr.x (mgr.24542) 4521 : audit [DBG] from='client.29668 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:32:39.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:38 smithi045 conmon[39718]: cluster 2021- 2021-07-31T03:32:39.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:38 smithi045 conmon[39718]: 07-31T03:32:37.299075+0000 mgr.x (mgr.24542) 4522 : cluster 2021-07-31T03:32:39.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:38 smithi045 conmon[39718]: [DBG] pgmap v5169: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:39.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:38 smithi045 conmon[33655]: audit 2021-07-31T03:32:37.190814+0000 mgr.x (mgr.24542) 4521 : 2021-07-31T03:32:39.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:38 smithi045 conmon[33655]: audit [DBG] from='client.29668 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:32:39.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:38 smithi045 conmon[33655]: cluster 2021-07-31T03:32:37.299075+0000 mgr.x (mgr 2021-07-31T03:32:39.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:38 smithi045 conmon[33655]: .24542) 4522 : cluster [DBG] pgmap v5169: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:41.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:40 smithi154 conmon[34553]: cluster 2021-07-31T03:32:39.299624+0000 mgr.x (mgr.24542) 4523 : cluster [DBG] pgmap v5170: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:41.024 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:40 smithi045 conmon[33655]: cluster 2021-07-31T03:32:39.299624+0000 mgr.x (mgr.24542) 4523 : cluster 2021-07-31T03:32:41.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:40 smithi045 conmon[33655]: [DBG] pgmap v5170: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:41.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:40 smithi045 conmon[39718]: cluster 2021-07-31T03:32:39. 2021-07-31T03:32:41.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:40 smithi045 conmon[39718]: 299624+0000 mgr.x (mgr.24542) 4523 : cluster [DBG] pgmap v5170: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:43.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:42 smithi154 conmon[34553]: cluster 2021-07-31T03:32:41.300038+0000 mgr.x (mgr.24542) 4524 : cluster 2021-07-31T03:32:43.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:42 smithi154 conmon[34553]: [DBG] pgmap v5171: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:43.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:42 smithi045 conmon[33655]: cluster 2021-07-31T03:32:41.300038+0000 mgr.x (mgr.24542) 4524 2021-07-31T03:32:43.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:42 smithi045 conmon[33655]: : cluster [DBG] pgmap v5171: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:43.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:42 smithi045 conmon[39718]: cluster 2021-07-31T03:32:41. 2021-07-31T03:32:43.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:42 smithi045 conmon[39718]: 300038+0000 mgr.x (mgr.24542) 4524 : cluster [DBG] pgmap v5171: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:45.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:44 smithi154 conmon[34553]: cluster 2021-07-31T03:32:43.300435+0000 mgr.x (mgr.24542) 4525 : cluster [DBG] pgmap v5172: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:45.024 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:44 smithi045 conmon[33655]: cluster 2021-07-31T03:32:43.300435+0000 mgr.x (mgr.24542) 4525 : cluster 2021-07-31T03:32:45.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:44 smithi045 conmon[33655]: [DBG] pgmap v5172: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:45.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:44 smithi045 conmon[39718]: cluster 2021-07-31T03:32:43.300435+0000 mgr.x (mgr.24542) 4525 : cluster [DBG] pgmap v5172: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:47.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:46 smithi154 conmon[34553]: cluster 2021-07-31T03:32:45.300892+0000 mgr.x (mgr.24542) 4526 : cluster [DBG] pgmap v5173: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:47.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:46 smithi154 conmon[34553]: 2021-07-31T03:32:47.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:46 smithi045 conmon[39718]: cluster 2021-07-31T03:32:45.300892+0000 mgr.x (mgr.24542) 4526 : cluster [DBG] 2021-07-31T03:32:47.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:46 smithi045 conmon[39718]: pgmap v5173: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:47.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:46 smithi045 conmon[33655]: cluster 2021-07-31T03:32:45.300892+0000 mgr.x (mgr.24542) 4526 : cluster [DBG] pgmap v5173: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:49.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:48 smithi154 conmon[34553]: cluster 2021-07-31T03:32:47.301406+0000 mgr.x (mgr.24542) 4527 : cluster [DBG] 2021-07-31T03:32:49.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:48 smithi154 conmon[34553]: pgmap v5174: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:49.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:48 smithi045 conmon[39718]: cluster 2021-07-31T03:32:47.301406+0000 mgr.x ( 2021-07-31T03:32:49.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:48 smithi045 conmon[39718]: mgr.24542) 4527 : cluster [DBG] pgmap v5174: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:49.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:48 smithi045 conmon[33655]: cluster 2021-07-31T03:32:47.301406+0000 mgr.x (mgr.24542 2021-07-31T03:32:49.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:48 smithi045 conmon[33655]: ) 4527 : cluster [DBG] pgmap v5174: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:51.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:50 smithi154 conmon[34553]: cluster 2021-07-31T03:32:49.301795+0000 mgr.x (mgr.24542) 4528 : cluster [DBG] pgmap v5175: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:51.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:50 smithi045 conmon[39718]: cluster 2021-07-31T03:32:49. 2021-07-31T03:32:51.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:50 smithi045 conmon[39718]: 301795+0000 mgr.x (mgr.24542) 4528 : cluster [DBG] pgmap v5175: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:51.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:50 smithi045 conmon[33655]: cluster 2021-07-31T03:32:49.301795+0000 mgr.x (mgr. 2021-07-31T03:32:51.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:50 smithi045 conmon[33655]: 24542) 4528 : cluster [DBG] pgmap v5175: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:53.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:52 smithi154 conmon[34553]: cluster 2021-07-31T03:32:51.302178+0000 mgr.x (mgr.24542) 4529 : cluster [DBG] 2021-07-31T03:32:53.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:52 smithi154 conmon[34553]: pgmap v5176: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:53.024 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:52 smithi045 conmon[33655]: cluster 2021-07-31T03:32:51.302178+0000 mgr.x (mgr.24542) 4529 2021-07-31T03:32:53.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:52 smithi045 conmon[33655]: : cluster [DBG] pgmap v5176: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:53.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:52 smithi045 conmon[39718]: cluster 2021-07-31T03:32:51. 2021-07-31T03:32:53.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:52 smithi045 conmon[39718]: 302178+0000 mgr.x (mgr.24542) 4529 : cluster [DBG] pgmap v5176: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:55.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:54 smithi154 conmon[34553]: cluster 2021-07-31T03:32:53.302674+0000 mgr.x (mgr.24542) 4530 : cluster [DBG] pgmap v5177: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:55.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:54 smithi045 conmon[33655]: cluster 2021-07-31T03:32:53.302674+0000 mgr.x (mgr.24542) 2021-07-31T03:32:55.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:54 smithi045 conmon[33655]: 4530 : cluster [DBG] pgmap v5177: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:55.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:54 smithi045 conmon[39718]: cluster 2021-07-31T03:32:53. 2021-07-31T03:32:55.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:54 smithi045 conmon[39718]: 302674+0000 mgr.x (mgr.24542) 4530 : cluster [DBG] pgmap v5177: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:57.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:56 smithi154 conmon[34553]: cluster 2021-07-31T03:32:55.303128+0000 mgr.x (mgr.24542) 4531 : cluster [DBG] pgmap v5178: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:57.024 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:56 smithi045 conmon[33655]: cluster 2021-07-31T03:32:55.303128+0000 mgr.x (mgr.24542) 4531 : cluster [DBG] pgmap v5178: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:57.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:56 smithi045 conmon[39718]: cluster 2021-07-31T03:32:55. 2021-07-31T03:32:57.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:56 smithi045 conmon[39718]: 303128+0000 mgr.x (mgr.24542) 4531 : cluster [DBG] pgmap v5178: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:59.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:32:58 smithi154 conmon[34553]: cluster 2021-07-31T03:32:57.303629+0000 mgr.x (mgr.24542) 4532 : cluster [DBG] pgmap v5179: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:59.024 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:58 smithi045 conmon[33655]: cluster 2021-07-31T03:32:57.303629+0000 2021-07-31T03:32:59.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:32:58 smithi045 conmon[33655]: mgr.x (mgr.24542) 4532 : cluster [DBG] pgmap v5179: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:32:59.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:32:58 smithi045 conmon[39718]: cluster 2021-07-31T03:32:57.303629+0000 mgr.x (mgr.24542) 4532 : cluster [DBG] pgmap v5179: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:01.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:00 smithi154 conmon[34553]: cluster 2021-07-31T03:32:59.304082+0000 mgr.x (mgr.24542) 4533 : cluster [DBG] pgmap v5180: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:01.024 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:00 smithi045 conmon[33655]: cluster 2021-07-31T03:32:59.304082+0000 mgr.x (mgr.24542) 4533 : cluster [DBG] pgmap v5180: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:01.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:00 smithi045 conmon[33655]: 2021-07-31T03:33:01.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:00 smithi045 conmon[39718]: cluster 2021-07-31T03:32:59.304082+0000 mgr.x ( 2021-07-31T03:33:01.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:00 smithi045 conmon[39718]: mgr.24542) 4533 : cluster [DBG] pgmap v5180: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:03.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:02 smithi154 conmon[34553]: cluster 2021-07-31T03:33:01.304543+0000 mgr.x (mgr.24542) 4534 : cluster [DBG] pgmap v5181: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:03.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:02 smithi045 conmon[39718]: cluster 2021-07-31T03:33:01.304543+0000 2021-07-31T03:33:03.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:02 smithi045 conmon[39718]: mgr.x (mgr.24542) 4534 : cluster [DBG] pgmap v5181: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:03.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:02 smithi045 conmon[33655]: cluster 2021-07-31T03:33:01.304543+0000 mgr.x (mgr.24542) 4534 2021-07-31T03:33:03.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:02 smithi045 conmon[33655]: : cluster [DBG] pgmap v5181: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:05.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:04 smithi045 conmon[33655]: cluster 2021-07-31T03:33:03.305119+0000 mgr.x (mgr.24542) 4535 : cluster [DBG] pgmap v5182: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:05.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:04 smithi045 conmon[39718]: cluster 2021-07-31T03:33:03.305119+0000 mgr.x (mgr 2021-07-31T03:33:05.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:04 smithi045 conmon[39718]: .24542) 4535 : cluster [DBG] pgmap v5182: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:05.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:04 smithi154 conmon[34553]: cluster 2021-07-31T03:33:03.305119+0000 mgr.x (mgr.24542) 4535 : cluster [DBG] pgmap v5182: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:07.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:06 smithi045 conmon[39718]: cluster 2021-07-31T03:33:05.305553+0000 mgr.x (mgr.24542) 4536 : 2021-07-31T03:33:07.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:06 smithi045 conmon[39718]: cluster [DBG] pgmap v5183: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:07.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:06 smithi045 conmon[33655]: cluster 2021-07-31T03:33:05.305553+0000 mgr.x (mgr.24542) 4536 : cluster [DBG] pgmap v5183: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:07.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:06 smithi154 conmon[34553]: cluster 2021-07-31T03:33:05.305553+0000 mgr.x (mgr.24542) 4536 : cluster [DBG] pgmap v5183: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:07.845 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:33:08.121 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:33:08.121 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 108m ago 108m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:33:08.121 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 105m ago 108m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:33:08.122 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 105m ago 112m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:33:08.122 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 108m ago 114m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:33:08.122 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 108m ago 114m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:33:08.122 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 105m ago 112m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:33:08.122 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 108m ago 112m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:33:08.123 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 108m ago 109m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:33:08.123 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 105m ago 108m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:33:08.123 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 108m ago 111m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:33:08.123 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 108m ago 111m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:33:08.124 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 108m ago 111m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:33:08.124 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 108m ago 110m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:33:08.124 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 105m ago 110m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:33:08.124 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 105m ago 110m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:33:08.124 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 105m ago 109m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:33:08.125 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 105m ago 109m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:33:08.125 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 105m ago 109m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:33:08.457 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:33:08.457 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:33:08.458 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:33:08.458 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:33:08.458 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:33:08.458 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:33:08.458 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:33:08.459 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:33:08.459 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:33:08.459 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:33:08.459 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:33:08.459 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:33:08.460 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:33:08.460 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:33:08.460 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:33:08.460 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:33:08.460 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:33:09.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:08 smithi045 conmon[33655]: cluster 2021-07-31T03:33:07.306091 2021-07-31T03:33:09.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:08 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 4537 : cluster [DBG] pgmap v5184: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:09.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:08 smithi045 conmon[33655]: audit 2021-07-31T03:33:07 2021-07-31T03:33:09.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:08 smithi045 conmon[33655]: .834165+0000 mgr.x (mgr.24542) 4538 : audit [DBG] from='client.19740 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:33:09.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:08 smithi045 conmon[33655]: audit 2021 2021-07-31T03:33:09.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:08 smithi045 conmon[33655]: -07-31T03:33:08.458139+0000 mon.c (mon.1) 213 : audit 2021-07-31T03:33:09.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:08 smithi045 conmon[33655]: [DBG] from='client.? 172.21.15.45:0/3663975848' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:33:09.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:08 smithi045 conmon[39718]: cluster 2021-07-31T03:33:07 2021-07-31T03:33:09.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:08 smithi045 conmon[39718]: .306091+0000 mgr.x (mgr.24542) 4537 : cluster [DBG] pgmap v5184: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:09.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:08 smithi045 conmon[39718]: audit 2021-07 2021-07-31T03:33:09.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:08 smithi045 conmon[39718]: -31T03:33:07. 2021-07-31T03:33:09.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:08 smithi045 conmon[39718]: 834165+0000 mgr.x (mgr.24542) 4538 2021-07-31T03:33:09.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:08 smithi045 conmon[39718]: : audit [DBG] from='client.19740 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:33:09.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:08 smithi045 conmon[39718]: audit 2021-07-31T03:33:09.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:08 smithi045 conmon[39718]: 2021- 2021-07-31T03:33:09.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:08 smithi045 conmon[39718]: 07-31T03:33:08.458139 2021-07-31T03:33:09.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:08 smithi045 conmon[39718]: +0000 mon.c (mon.1) 213 : audit 2021-07-31T03:33:09.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:08 smithi045 conmon[39718]: [DBG] from='client.? 172.21.15.45:0/3663975848' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:33:09.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:08 smithi154 conmon[34553]: cluster 2021-07-31T03:33:07.306091+0000 mgr.x (mgr.24542) 4537 : cluster [DBG] pgmap v5184: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:09.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:08 smithi154 conmon[34553]: audit 2021-07-31T03:33:07.834165 2021-07-31T03:33:09.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:08 smithi154 conmon[34553]: +0000 mgr.x (mgr.24542) 4538 : audit [DBG] from='client.19740 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:33:09.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:08 smithi154 conmon[34553]: audit 2021-07-31T03:33:08.458139+0000 mon.c (mon.1) 213 : audit [DBG] from='client.? 172.21.15.45:0/3663975848' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:33:10.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:09 smithi045 conmon[33655]: audit 2021-07-31T03:33:08.114172 2021-07-31T03:33:10.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:09 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 4539 : audit [DBG] from='client.29686 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:33:10.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:09 smithi045 conmon[39718]: audit 2021-07-31T03:33:08.114172+0000 mgr.x (mgr 2021-07-31T03:33:10.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:09 smithi045 conmon[39718]: .24542) 4539 : audit [DBG] from='client.29686 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:33:10.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:09 smithi154 conmon[34553]: audit 2021-07-31T03:33:08.114172+0000 mgr.x (mgr.24542) 4539 : audit [DBG] from='client.29686 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:33:10.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:09 smithi154 conmon[34553]: 2021-07-31T03:33:11.024 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:10 smithi045 conmon[39718]: cluster 2021-07-31T03:33:09.306501+0000 mgr.x (mgr 2021-07-31T03:33:11.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:10 smithi045 conmon[39718]: .24542) 4540 : cluster [DBG] pgmap v5185: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:11.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:10 smithi045 conmon[33655]: cluster 2021-07-31T03:33:09.306501+0000 2021-07-31T03:33:11.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:10 smithi045 conmon[33655]: mgr.x (mgr.24542) 4540 : cluster [DBG] pgmap v5185: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:11.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:10 smithi154 conmon[34553]: cluster 2021-07-31T03:33:09.306501+0000 mgr.x (mgr.24542) 4540 : cluster [DBG] pgmap v5185: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:13.024 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:12 smithi045 conmon[33655]: cluster 2021-07-31T03:33:11.306821+0000 mgr.x (mgr.24542) 4541 : cluster [DBG] pgmap v5186: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:13.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:12 smithi045 conmon[39718]: cluster 2021-07-31T03:33:11.306821+0000 mgr.x (mgr.24542) 4541 : cluster [DBG] pgmap v5186: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:13.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:12 smithi154 conmon[34553]: cluster 2021-07-31T03:33:11.306821+0000 mgr.x (mgr.24542) 4541 : cluster [DBG] pgmap v5186: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:15.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:14 smithi045 conmon[33655]: cluster 2021-07-31T03: 2021-07-31T03:33:15.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:14 smithi045 conmon[33655]: 33:13.307318+0000 mgr.x (mgr.24542) 4542 : cluster [DBG] pgmap v5187: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:15.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:14 smithi045 conmon[39718]: cluster 2021-07-31T03:33:13 2021-07-31T03:33:15.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:14 smithi045 conmon[39718]: .307318+0000 mgr.x (mgr.24542) 4542 : cluster [DBG] pgmap v5187: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:15.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:14 smithi154 conmon[34553]: cluster 2021-07-31T03:33:13.307318+0000 mgr.x (mgr.24542) 4542 : cluster [DBG] pgmap v5187: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:17.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:16 smithi154 conmon[34553]: cluster 2021-07-31T03:33:15.307727+0000 mgr.x (mgr.24542) 4543 : cluster [DBG] pgmap v5188: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:17.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:16 smithi154 conmon[34553]: 2021-07-31T03:33:17.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:16 smithi045 conmon[39718]: cluster 2021-07-31T03:33:15.307727+0000 mgr.x 2021-07-31T03:33:17.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:16 smithi045 conmon[39718]: (mgr.24542) 4543 : cluster [DBG] pgmap v5188: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:17.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:16 smithi045 conmon[33655]: cluster 2021-07-31T03:33:15.307727+0000 mgr.x (mgr.24542 2021-07-31T03:33:17.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:16 smithi045 conmon[33655]: ) 4543 : cluster [DBG] pgmap v5188: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:19.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:18 smithi154 conmon[34553]: cluster 2021-07-31T03:33:17.308151+0000 mgr.x (mgr.24542) 2021-07-31T03:33:19.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:18 smithi154 conmon[34553]: 4544 : cluster [DBG] pgmap v5189: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:19.274 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:18 smithi045 conmon[39718]: cluster 2021-07-31T03:33:17.308151+0000 mgr.x ( 2021-07-31T03:33:19.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:18 smithi045 conmon[39718]: mgr.24542) 4544 : cluster [DBG] pgmap v5189: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:19.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:18 smithi045 conmon[33655]: cluster 2021-07-31T03:33:17.308151+0000 mgr.x (mgr.24542 2021-07-31T03:33:19.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:18 smithi045 conmon[33655]: ) 4544 : cluster [DBG] pgmap v5189: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:21.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:20 smithi154 conmon[34553]: cluster 2021-07-31T03:33:19.308616+0000 mgr.x (mgr.24542) 2021-07-31T03:33:21.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:20 smithi154 conmon[34553]: 4545 : cluster [DBG] pgmap v5190: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:21.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:20 smithi045 conmon[39718]: cluster 2021-07-31T03:33:19.308616+0000 mgr.x (mgr.24542) 4545 : cluster [DBG] pgmap v5190: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:21.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:20 smithi045 conmon[33655]: cluster 2021-07-31T03:33:19.308616+0000 mgr.x (mgr.24542) 4545 : cluster 2021-07-31T03:33:21.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:20 smithi045 conmon[33655]: [DBG] pgmap v5190: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:23.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:22 smithi154 conmon[34553]: cluster 2021-07-31T03:33:21.309197+0000 mgr.x (mgr.24542) 4546 : cluster 2021-07-31T03:33:23.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:22 smithi154 conmon[34553]: [DBG] pgmap v5191: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:23.274 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:22 smithi045 conmon[39718]: cluster 2021-07-31T03:33:21.309197+0000 mgr.x ( 2021-07-31T03:33:23.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:22 smithi045 conmon[39718]: mgr.24542) 4546 : cluster [DBG] pgmap v5191: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:23.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:22 smithi045 conmon[33655]: cluster 2021-07-31T03:33:21.309197+0000 mgr.x (mgr.24542) 4546 : cluster [DBG] pgmap v5191: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:25.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:24 smithi154 conmon[34553]: cluster 2021-07-31T03:33:23.309781+0000 mgr.x (mgr.24542) 4547 : cluster [DBG] pgmap v5192: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:25.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:24 smithi045 conmon[33655]: cluster 2021-07-31T03:33:23.309781+0000 mgr.x (mgr.24542) 4547 : cluster 2021-07-31T03:33:25.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:24 smithi045 conmon[33655]: [DBG] pgmap v5192: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:25.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:24 smithi045 conmon[39718]: cluster 2021-07-31T03:33:23.309781+0000 mgr.x (mgr 2021-07-31T03:33:25.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:24 smithi045 conmon[39718]: .24542) 4547 : cluster [DBG] pgmap v5192: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:27.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:26 smithi154 conmon[34553]: cluster 2021-07-31T03:33:25.310266+0000 mgr.x (mgr.24542) 4548 : cluster [DBG] pgmap v5193: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:27.274 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:26 smithi045 conmon[33655]: cluster 2021-07-31T03:33:25.310266+0000 mgr.x (mgr.24542) 4548 : cluster 2021-07-31T03:33:27.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:26 smithi045 conmon[33655]: [DBG] pgmap v5193: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:27.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:26 smithi045 conmon[39718]: cluster 2021-07-31T03:33:25.310266+0000 mgr.x (mgr.24542) 4548 : cluster 2021-07-31T03:33:27.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:26 smithi045 conmon[39718]: [DBG] pgmap v5193: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:29.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:28 smithi154 conmon[34553]: cluster 2021-07-31T03:33:27.310715+0000 mgr.x (mgr.24542) 4549 : cluster [DBG] pgmap v5194: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:29.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:28 smithi154 conmon[34553]: audit 2021-07-31T03:33:28.356062+0000 mon.b (mon.2) 605 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:33:29.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:28 smithi154 conmon[34553]: audit 2021-07-31T03:33:28.356297+0000 mon.b (mon 2021-07-31T03:33:29.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:28 smithi154 conmon[34553]: .2) 606 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:33:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:28 smithi154 conmon[34553]: audit 2021-07-31T03:33:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:28 smithi154 conmon[34553]: 2021-07-31T03:33:28 2021-07-31T03:33:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:28 smithi154 conmon[34553]: .356776+0000 mon.a (mon.0) 2021-07-31T03:33:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:28 smithi154 conmon[34553]: 3323 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:33:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:28 smithi154 conmon[34553]: audit 2021-07-31T03:33:29.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:28 smithi154 conmon[34553]: 2021-07-31T03:33: 2021-07-31T03:33:29.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:28 smithi154 conmon[34553]: 28.357187+0000 mon.a (mon.0 2021-07-31T03:33:29.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:28 smithi154 conmon[34553]: ) 3324 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:33:29.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:28 smithi045 conmon[39718]: cluster 2021-07-31T03:33:27.310715+0000 mgr.x 2021-07-31T03:33:29.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:28 smithi045 conmon[39718]: (mgr.24542) 4549 : cluster [DBG] pgmap v5194: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:29.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:28 smithi045 conmon[39718]: audit 2021 2021-07-31T03:33:29.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:28 smithi045 conmon[39718]: -07-31T03:33:28.356062+0000 mon.b (mon.2) 605 2021-07-31T03:33:29.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:28 smithi045 conmon[39718]: : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:33:29.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:28 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T03:33:29.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:28 smithi045 conmon[39718]: T03:33:28.356297+0000 mon.b (mon.2 2021-07-31T03:33:29.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:28 smithi045 conmon[39718]: ) 606 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:33:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:28 smithi045 conmon[39718]: audit 2021- 2021-07-31T03:33:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:28 smithi045 conmon[39718]: 07-31T03:33:28.356776+0000 mon.a (mon. 2021-07-31T03:33:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:28 smithi045 conmon[39718]: 0) 3323 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:33:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:28 smithi045 conmon[39718]: audit 2021-07 2021-07-31T03:33:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:28 smithi045 conmon[39718]: -31T03:33:28.357187+0000 mon.a (mon 2021-07-31T03:33:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:28 smithi045 conmon[39718]: .0) 3324 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:33:29.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:28 smithi045 conmon[33655]: cluster 2021-07-31T03:33:27.310715+0000 mgr.x (mgr.24542) 4549 : cluster [DBG] 2021-07-31T03:33:29.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:28 smithi045 conmon[33655]: pgmap v5194: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:29.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:28 smithi045 conmon[33655]: audit 2021-07-31T03:33:28.356062+0000 mon.b (mon.2) 605 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:33:29.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:28 smithi045 conmon[33655]: audit 2021-07-31T03:33:28.356297+0000 mon.b (mon.2) 606 : audit [INF] 2021-07-31T03:33:29.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:28 smithi045 conmon[33655]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:33:29.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:28 smithi045 conmon[33655]: audit 2021-07-31T03:33:28.356776+0000 mon.a (mon.0) 3323 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:33:29.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:28 smithi045 conmon[33655]: 2021-07-31T03:33:29.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:28 smithi045 conmon[33655]: audit 2021-07-31T03:33:28.357187+0000 mon.a (mon. 2021-07-31T03:33:29.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:28 smithi045 conmon[33655]: 0) 3324 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:33:31.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:30 smithi154 conmon[34553]: cluster 2021-07-31T03:33:29.311255+0000 mgr.x (mgr.24542) 4550 : cluster [DBG] pgmap v5195: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:31.274 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:30 smithi045 conmon[39718]: cluster 2021-07-31T03:33:29.311255+0000 mgr.x (mgr.24542) 4550 : cluster [DBG] pgmap v5195: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:31.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:30 smithi045 conmon[39718]: 2021-07-31T03:33:31.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:30 smithi045 conmon[33655]: cluster 2021-07-31T03:33:29.311255+0000 mgr.x (mgr.24542) 4550 2021-07-31T03:33:31.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:30 smithi045 conmon[33655]: : cluster [DBG] pgmap v5195: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:33.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:32 smithi154 conmon[34553]: cluster 2021-07-31T03:33:31.311641+0000 mgr.x (mgr.24542) 4551 : cluster [DBG] pgmap v5196: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:33.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:32 smithi045 conmon[39718]: cluster 2021-07-31T03:33:31.311641+0000 2021-07-31T03:33:33.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:32 smithi045 conmon[39718]: mgr.x (mgr.24542) 4551 : cluster [DBG] pgmap v5196: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:33.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:32 smithi045 conmon[33655]: cluster 2021-07-31T03:33:31.311641+0000 mgr.x (mgr.24542 2021-07-31T03:33:33.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:32 smithi045 conmon[33655]: ) 4551 : cluster [DBG] pgmap v5196: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:34.252 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: debug 2021-07-31T03:33:33.855+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:33:34.253 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:33:34.253 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:33:34.253 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: yield 2021-07-31T03:33:34.253 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:33:34.253 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:33:34.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:33:34.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:33:34.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:33:34.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:33:34.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:33:34.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:33:34.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:33:34.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: task.result() 2021-07-31T03:33:34.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:33:34.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: debug 2021-07-31T03:33:33.866+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:33:34.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:33:34.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:33:34.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: yield 2021-07-31T03:33:34.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:33:34.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:33:34.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:33:34.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:33:34.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:33:34.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:33:34.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:33:34.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:33:34.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:33:34.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: task.result() 2021-07-31T03:33:34.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:33:34.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: debug 2021-07-31T03:33:33.892+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:33:34.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:33:34.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:33:34.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: yield 2021-07-31T03:33:34.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:33:34.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:33:34.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:33:34.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:33:34.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:33:34.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:33:34.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:33:34.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:33:34.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:33:34.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: task.result() 2021-07-31T03:33:34.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:33:34.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: debug 2021-07-31T03:33:33.892+0000 7fcb9295f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:33:34.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:33:34.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:33:34.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: yield 2021-07-31T03:33:34.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:33:34.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:33:34.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:33:34.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:33:34.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:33:34.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:33:34.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:33:34.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:33:34.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:33:34.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: task.result() 2021-07-31T03:33:34.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:33:34.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: 2021-07-31T03:33:34.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:33:34.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: 2021-07-31T03:33:34.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:33:34.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:33:34.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:33:34.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:33:34.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:33:34.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:33:34.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:33:34.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:33:34.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:33:34.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:33:34.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:33:34.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:33:34.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:33:34.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:33:34.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:33:34.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:33:34.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:33:34.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:33:34.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: raise 2021-07-31T03:33:34.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:33:34.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:33:34.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:33:34.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:33:34.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:33:34.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: debug 2021-07-31T03:33:33.901+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:33:34.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:33:34.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:33:34.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: yield 2021-07-31T03:33:34.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:33:34.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:33:34.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:33:34.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:33:34.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:33:34.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:33:34.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:33:34.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:33:34.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:33:34.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: task.result() 2021-07-31T03:33:34.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:33:34.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: debug 2021-07-31T03:33:33.901+0000 7fcb93160700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:33:34.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:33:34.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:33:34.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: yield 2021-07-31T03:33:34.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:33:34.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:33:34.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:33:34.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:33:34.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:33:34.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:33:34.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:33:34.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:33:34.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:33:34.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: task.result() 2021-07-31T03:33:34.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:33:34.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: 2021-07-31T03:33:34.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:33:34.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: 2021-07-31T03:33:34.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:33:34.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:33:34.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:33:34.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:33:34.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:33:34.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:33:34.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:33:34.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:33:34.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:33:34.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:33:34.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:33:34.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:33:34.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:33:34.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:33:34.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:33:34.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:33:34.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:33:34.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:33:34.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: raise 2021-07-31T03:33:34.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:33:34.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:33:34.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:33:34.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:33:34.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:33:33 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:33:34.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: cluster 2021-07-31T03:33:33.312050+0000 mgr.x (mgr.24542) 4552 : cluster [DBG] pgmap v5197: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:34.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: audit 2021-07-31T03:33:33.808770+0000 mon.b (mon.2) 607 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:33:34.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: audit 2021-07-31T03:33 2021-07-31T03:33:34.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: :33.810308+0000 mon.b (mon. 2021-07-31T03:33:34.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: 2) 608 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:33:34.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: audit 2021- 2021-07-31T03:33:34.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: 07-31T03:33:33.811206+0000 mon.b 2021-07-31T03:33:34.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: (mon.2) 609 : audit [INF] 2021-07-31T03:33:34.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:33:34.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: cephadm 2021- 2021-07-31T03:33:34.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: 07-31T03:33:33.811995 2021-07-31T03:33:34.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: +0000 mgr.x (mgr.24542) 4553 2021-07-31T03:33:34.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:33:34.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: cephadm 2021- 2021-07-31T03:33:34.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: 07-31T03:33:33.856198+0000 2021-07-31T03:33:34.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: mgr.x (mgr.24542) 4554 : 2021-07-31T03:33:34.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: cephadm [ERR] 2021-07-31T03:33:34.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:33:34.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:33:34.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: yield 2021-07-31T03:33:34.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:33:34.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:33:34.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:33:34.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:33:34.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:33:34.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:33:34.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:33:34.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:33:34.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:33:34.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: task.result() 2021-07-31T03:33:34.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:33:34.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: cephadm 2021-07-31T03:33:33.856617+0000 mgr.x (mgr.24542) 4555 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:33:34.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: cephadm 2021- 2021-07-31T03:33:34.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: 07-31T03:33:33.867785 2021-07-31T03:33:34.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: +0000 mgr.x (mgr.24542) 4556 : cephadm 2021-07-31T03:33:34.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: [ERR] 2021-07-31T03:33:34.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:33:34.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:33:34.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: yield 2021-07-31T03:33:34.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:33:34.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:33:34.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:33:34.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:33:34.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:33:34.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:33:34.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:33:34.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:33:34.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:33:34.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: task.result() 2021-07-31T03:33:34.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:33:34.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: cephadm 2021-07-31T03:33:33.868625+0000 mgr.x (mgr.24542) 4557 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:33:34.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: cephadm 2021- 2021-07-31T03:33:34.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: 07-31T03:33:33.893453+0000 2021-07-31T03:33:34.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: mgr.x (mgr.24542) 4558 : 2021-07-31T03:33:34.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: cephadm [ERR] 2021-07-31T03:33:34.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:33:34.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:33:34.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: yield 2021-07-31T03:33:34.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:33:34.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:33:34.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:33:34.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:33:34.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:33:34.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:33:34.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:33:34.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:33:34.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:33:34.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: task.result() 2021-07-31T03:33:34.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:33:34.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: cephadm 2021-07-31T03:33:33.893870+0000 mgr.x (mgr.24542) 4559 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:33:34.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:33:34.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:33:34.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: yield 2021-07-31T03:33:34.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:33:34.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:33:34.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:33:34.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:33:34.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:33:34.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:33:34.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:33:34.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:33:34.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:33:34.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: task.result() 2021-07-31T03:33:34.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:33:34.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: 2021-07-31T03:33:34.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:33:34.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: 2021-07-31T03:33:34.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:33:34.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:33:34.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:33:34.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:33:34.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:33:34.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:33:34.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:33:34.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:33:34.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:33:34.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:33:34.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:33:34.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:33:34.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:33:34.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:33:34.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:33:34.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:33:34.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:33:34.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:33:34.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: raise 2021-07-31T03:33:34.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:33:34.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:33:34.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:33:34.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:33:34.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:33:34.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: cephadm 2021-07-31T03:33:33.902260+0000 mgr.x (mgr.24542) 4560 : cephadm [ERR] 2021-07-31T03:33:34.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:33:34.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:33:34.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: yield 2021-07-31T03:33:34.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:33:34.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:33:34.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:33:34.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:33:34.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:33:34.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:33:34.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:33:34.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: cluster 2021-07-31T03:33:33.312050+0000 mgr.x (mgr.24542) 4552 : cluster 2021-07-31T03:33:34.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: [DBG] pgmap v5197: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:34.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: audit 2021-07-31T03:33:33.808770+0000 mon.b (mon. 2021-07-31T03:33:34.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: 2) 607 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:33:34.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: audit 2021- 2021-07-31T03:33:34.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: 07-31T03:33:33.810308+0000 mon.b (mon.2) 608 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:33:34.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: audit 2021-07-31T03:33:33 2021-07-31T03:33:34.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: .811206+0000 mon.b (mon.2) 609 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:33:34.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: cephadm 2021-07-31T03: 2021-07-31T03:33:34.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: 33:33.811995+0000 mgr.x (mgr.24542) 4553 : cephadm [WRN] 2021-07-31T03:33:34.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:33:34.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: cephadm 2021-07 2021-07-31T03:33:34.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: -31T03:33:33.856198+0000 mgr.x (mgr.24542) 4554 : cephadm [ERR] 2021-07-31T03:33:34.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:33:34.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:33:34.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: yield 2021-07-31T03:33:34.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:33:34.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:33:34.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:33:34.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:33:34.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:33:34.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:33:34.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:33:34.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:33:34.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:33:34.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: task.result() 2021-07-31T03:33:34.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:33:34.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: cephadm 2021-07-31T03:33:33.856617+0000 mgr.x (mgr.24542) 4555 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:33:34.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: cephadm 2021-07-31T03:33:33.867785+0000 mgr.x (mgr.24542) 4556 : cephadm [ERR] 2021-07-31T03:33:34.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:33:34.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:33:34.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: yield 2021-07-31T03:33:34.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:33:34.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:33:34.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:33:34.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:33:34.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:33:34.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:33:34.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:33:34.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:33:34.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:33:34.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: task.result() 2021-07-31T03:33:34.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:33:34.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: cephadm 2021-07-31T03:33:33. 2021-07-31T03:33:34.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: 868625+0000 mgr.x (mgr.24542) 4557 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:33:34.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: cephadm 2021-07-31T03:33:33.893453+0000 mgr.x (mgr.24542) 4558 : cephadm [ERR] 2021-07-31T03:33:34.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:33:34.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:33:34.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: yield 2021-07-31T03:33:34.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:33:34.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:33:34.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:33:34.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:33:34.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:33:34.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:33:34.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:33:34.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:33:34.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:33:34.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: task.result() 2021-07-31T03:33:34.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:33:34.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: cephadm 2021-07-31T03:33:33.893870+0000 mgr.x (mgr.24542) 4559 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:33:34.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:33:34.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:33:34.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: yield 2021-07-31T03:33:34.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:33:34.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:33:34.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:33:34.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:33:34.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:33:34.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:33:34.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:33:34.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:33:34.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:33:34.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: task.result() 2021-07-31T03:33:34.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:33:34.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: 2021-07-31T03:33:34.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:33:34.797 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:33:34.797 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:33:34.797 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: task.result() 2021-07-31T03:33:34.797 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:33:34.798 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: cephadm 2021-07-31T03:33:33.902643+0000 mgr.x (mgr.24542) 4561 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:33:34.798 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:33:34.798 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:33:34.798 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: yield 2021-07-31T03:33:34.799 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:33:34.799 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:33:34.799 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:33:34.799 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:33:34.799 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:33:34.800 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:33:34.800 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:33:34.800 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:33:34.800 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:33:34.801 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: task.result() 2021-07-31T03:33:34.801 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:33:34.801 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: 2021-07-31T03:33:34.801 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:33:34.801 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: 2021-07-31T03:33:34.802 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:33:34.802 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:33:34.802 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:33:34.802 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:33:34.802 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:33:34.803 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:33:34.803 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:33:34.803 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:33:34.803 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:33:34.804 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:33:34.804 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:33:34.804 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:33:34.804 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:33:34.804 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:33:34.805 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:33:34.805 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:33:34.805 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:33:34.805 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:33:34.805 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: raise 2021-07-31T03:33:34.806 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:33:34.806 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:33:34.806 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:33:34.806 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:33:34.807 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:34 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:33:34.807 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: cluster 2021-07-31T03:33:33.312050+0000 mgr.x (mgr.24542) 4552 : cluster [DBG] pgmap v5197: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:34.807 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: audit 2021-07-31T03:33:34.807 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: 2021-07-31T03:33:33.808770+0000 mon.b (mon.2) 607 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:33:34.808 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: audit 2021-07-31T03:33:33.810308+0000 mon.b (mon.2 2021-07-31T03:33:34.808 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: ) 608 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:33:34.808 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: audit 2021-07-31T03:33:34.808 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: 2021-07-31T03:33:33.811206+0000 mon.b 2021-07-31T03:33:34.808 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: (mon.2) 609 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:33:34.809 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: cephadm 2021-07-31T03:33:34.809 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: 2021-07-31T03:33:33 2021-07-31T03:33:34.809 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: .811995+0000 mgr.x (mgr. 2021-07-31T03:33:34.809 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: 24542) 4553 : cephadm [WRN] 2021-07-31T03:33:34.810 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:33:34.810 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: cephadm 2021- 2021-07-31T03:33:34.810 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: 07-31T03:33:33.856198 2021-07-31T03:33:34.810 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 2021-07-31T03:33:34.810 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: 4554 : cephadm [ERR] 2021-07-31T03:33:34.811 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:33:34.811 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:33:34.811 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: yield 2021-07-31T03:33:34.811 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:33:34.811 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:33:34.812 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:33:34.812 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:33:34.812 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:33:34.812 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:33:34.813 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:33:34.813 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:33:34.813 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:33:34.813 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: task.result() 2021-07-31T03:33:34.813 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:33:34.814 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: cephadm 2021-07-31T03:33:33.856617+0000 mgr.x (mgr.24542 2021-07-31T03:33:34.814 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: ) 4555 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:33:34.814 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: 2021-07-31T03:33:34.814 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: cephadm 2021-07- 2021-07-31T03:33:34.814 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: 31T03:33:33.867785+0000 2021-07-31T03:33:34.815 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: mgr.x (mgr.24542) 4556 2021-07-31T03:33:34.815 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: : cephadm [ERR] 2021-07-31T03:33:34.815 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:33:34.815 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:33:34.816 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: yield 2021-07-31T03:33:34.816 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:33:34.816 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: 2021-07-31T03:33:34.816 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:33:34.817 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:33:34.817 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:33:34.817 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:33:34.817 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:33:34.818 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:33:34.818 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:33:34.818 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:33:34.818 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:33:34.819 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:33:34.819 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:33:34.819 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:33:34.819 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:33:34.819 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:33:34.820 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:33:34.820 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:33:34.820 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:33:34.820 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:33:34.820 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: raise 2021-07-31T03:33:34.821 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:33:34.821 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:33:34.821 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:33:34.821 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:33:34.822 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:33:34.822 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: cephadm 2021-07-31T03:33: 2021-07-31T03:33:34.822 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: 33.902260+0000 mgr.x (mgr.24542) 4560 : cephadm [ERR] 2021-07-31T03:33:34.822 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:33:34.822 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:33:34.823 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: yield 2021-07-31T03:33:34.823 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:33:34.823 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:33:34.823 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:33:34.824 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:33:34.824 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:33:34.824 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:33:34.824 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:33:34.824 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:33:34.825 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:33:34.825 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: task.result() 2021-07-31T03:33:34.825 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:33:34.825 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:33:34.826 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:33:34.826 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:33:34.826 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:33:34.826 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:33:34.827 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:33:34.827 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:33:34.827 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:33:34.827 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: task.result() 2021-07-31T03:33:34.828 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:33:34.828 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: cephadm 2021-07-31T03:33:33.868625+0000 mgr.x (mgr.24542) 4557 2021-07-31T03:33:34.828 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:33:34.828 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: cephadm 2021- 2021-07-31T03:33:34.828 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: 07-31T03:33:33.893453+0000 2021-07-31T03:33:34.829 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: mgr.x (mgr.24542) 4558 2021-07-31T03:33:34.829 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: : cephadm [ERR] 2021-07-31T03:33:34.829 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:33:34.829 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:33:34.830 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: yield 2021-07-31T03:33:34.830 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:33:34.830 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:33:34.830 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:33:34.830 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:33:34.831 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:33:34.831 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:33:34.831 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:33:34.831 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:33:34.832 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:33:34.832 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: task.result() 2021-07-31T03:33:34.832 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:33:34.832 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: cephadm 2021-07-31T03:33:33.893870+0000 mgr.x (mgr.24542) 4559 : 2021-07-31T03:33:34.832 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:33:34.833 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:33:34.833 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:33:34.833 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: yield 2021-07-31T03:33:34.833 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:33:34.834 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:33:34.834 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:33:34.834 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:33:34.834 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:33:34.834 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:33:34.835 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: cephadm 2021-07-31T03:33:33.902643+0000 mgr.x (mgr.24542) 4561 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:33:34.835 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:33:34.835 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:33:34.836 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: yield 2021-07-31T03:33:34.836 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:33:34.836 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:33:34.836 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:33:34.836 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:33:34.837 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:33:34.837 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:33:34.837 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:33:34.837 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:33:34.838 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:33:34.838 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: task.result() 2021-07-31T03:33:34.838 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:33:34.838 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: 2021-07-31T03:33:34.838 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:33:34.839 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: 2021-07-31T03:33:34.839 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:33:34.839 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:33:34.839 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:33:34.839 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:33:34.840 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:33:34.840 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:33:34.840 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:33:34.840 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:33:34.841 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:33:34.841 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:33:34.841 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:33:34.841 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:33:34.841 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:33:34.842 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:33:34.842 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:33:34.842 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:33:34.842 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:33:34.842 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:33:34.843 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: raise 2021-07-31T03:33:34.843 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:33:34.843 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:33:34.843 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:33:34.844 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:33:34.844 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:33:34.844 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:33:34.845 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:33:34.845 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:33:34.845 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: task.result() 2021-07-31T03:33:34.845 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:33:34.846 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: 2021-07-31T03:33:34.846 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:33:34.846 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: 2021-07-31T03:33:34.846 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:33:34.847 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:33:34.847 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:33:34.847 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:33:34.847 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:33:34.847 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:33:34.848 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:33:34.848 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:33:34.848 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:33:34.848 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:33:34.849 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:33:34.849 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:33:34.849 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:33:34.849 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:33:34.849 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:33:34.850 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:33:34.850 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:33:34.850 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:33:34.850 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: raise 2021-07-31T03:33:34.851 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:33:34.851 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:33:34.851 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:33:34.851 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:33:34.851 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:33:34.852 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: cephadm 2021-07-31T03:33:33.902260+0000 mgr.x (mgr.24542) 4560 : cephadm [ERR] 2021-07-31T03:33:34.852 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:33:34.852 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:33:34.852 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: yield 2021-07-31T03:33:34.852 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:33:34.853 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:33:34.853 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:33:34.853 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:33:34.853 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:33:34.854 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:33:34.854 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:33:34.854 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:33:34.854 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:33:34.854 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: task.result() 2021-07-31T03:33:34.855 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:33:34.855 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: cephadm 2021-07-31T03:33:33.902643+0000 mgr.x (mgr.24542) 4561 : cephadm 2021-07-31T03:33:34.855 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:33:34.855 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:33:34.856 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:33:34.856 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: yield 2021-07-31T03:33:34.856 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:33:34.856 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:33:34.856 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:33:34.857 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:33:34.857 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:33:34.857 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:33:34.857 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:33:34.858 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:33:34.858 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:33:34.858 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: task.result() 2021-07-31T03:33:34.858 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:33:34.859 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: 2021-07-31T03:33:34.859 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:33:34.859 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: 2021-07-31T03:33:34.859 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:33:34.860 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:33:34.860 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:33:34.860 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:33:34.860 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:33:34.860 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:33:34.861 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:33:34.861 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:33:34.861 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:33:34.861 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:33:34.861 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:33:34.862 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:33:34.862 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:33:34.862 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:33:34.862 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:33:34.863 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:33:34.863 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:33:34.863 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:33:34.863 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: raise 2021-07-31T03:33:34.863 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:33:34.864 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:33:34.864 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:33:34.864 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:33:34.864 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:34 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:33:36.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:36 smithi154 conmon[34553]: cluster 2021-07-31T03:33:35.312516+0000 mgr.x (mgr.24542) 4562 : cluster [DBG] pgmap v5198: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:36.774 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:36 smithi045 conmon[39718]: cluster 2021-07-31T03:33:35.312516+0000 mgr.x (mgr.24542) 4562 : cluster [DBG] pgmap v5198: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:36.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:36 smithi045 conmon[33655]: cluster 2021-07-31T03:33:35.312516+0000 mgr.x (mgr.24542) 4562 : cluster [DBG] pgmap v5198: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:36.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:36 smithi045 conmon[33655]: 2021-07-31T03:33:38.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:38 smithi154 conmon[34553]: cluster 2021-07-31T03:33:37.313023+0000 mgr.x (mgr.24542) 4563 : cluster [DBG] pgmap v5199: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:38.769 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:33:38.774 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:38 smithi045 conmon[33655]: cluster 2021-07-31T03:33:37.313023+0000 mgr.x (mgr.24542) 4563 : cluster [DBG] pgmap v5199: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:38.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:38 smithi045 conmon[39718]: cluster 2021-07-31T03:33:37.313023+0000 mgr.x (mgr. 2021-07-31T03:33:38.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:38 smithi045 conmon[39718]: 24542) 4563 : cluster [DBG] pgmap v5199: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:39.048 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:33:39.048 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 108m ago 109m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:33:39.048 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 106m ago 109m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:33:39.049 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 106m ago 112m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:33:39.049 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 108m ago 114m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:33:39.049 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 108m ago 114m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:33:39.049 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 106m ago 112m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:33:39.049 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 108m ago 112m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:33:39.050 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 108m ago 109m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:33:39.050 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 106m ago 109m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:33:39.050 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 108m ago 112m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:33:39.050 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 108m ago 111m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:33:39.051 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 108m ago 111m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:33:39.051 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 108m ago 111m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:33:39.051 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 106m ago 110m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:33:39.051 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 106m ago 110m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:33:39.052 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 106m ago 110m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:33:39.052 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 106m ago 109m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:33:39.052 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 106m ago 109m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:33:39.385 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:33:39.386 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:33:39.386 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:33:39.386 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:33:39.386 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:33:39.387 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:33:39.387 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:33:39.387 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:33:39.387 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:33:39.388 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:33:39.388 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:33:39.388 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:33:39.389 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:33:39.389 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:33:39.389 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:33:39.389 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:33:39.390 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:33:39.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:39 smithi154 conmon[34553]: audit 2021-07-31T03:33:38.757837+0000 mgr.x (mgr.24542) 4564 : audit [DBG] 2021-07-31T03:33:39.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:39 smithi154 conmon[34553]: from='client.19758 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:33:39.774 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:39 smithi045 conmon[33655]: audit 2021-07-31T03:33:38.757837+0000 mgr.x (mgr.24542 2021-07-31T03:33:39.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:39 smithi045 conmon[33655]: ) 4564 : audit [DBG] from='client.19758 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:33:39.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:39 smithi045 conmon[39718]: audit 2021-07-31T03:33:38.757837 2021-07-31T03:33:39.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:39 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 4564 : audit [DBG] from='client.19758 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:33:40.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:40 smithi154 conmon[34553]: audit 2021-07-31T03:33:39.040400+0000 mgr.x (mgr.24542) 4565 : audit [DBG] from='client.29704 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:33:40.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:40 smithi154 conmon[34553]: cluster 2021-07-31T03:33:39.313440+0000 mgr.x (mgr.24542) 4566 : cluster [DBG] pgmap v5200: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:40.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:40 smithi154 conmon[34553]: audit 2021-07-31T03:33:39.386704+0000 mon.a (mon.0) 3325 : audit [DBG] from='client.? 172.21.15.45:0/1077129280' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:33:40.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:40 smithi045 conmon[39718]: audit 2021-07-31T03:33:39.040400+0000 mgr.x (mgr. 2021-07-31T03:33:40.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:40 smithi045 conmon[39718]: 24542) 4565 : audit [DBG] from='client.29704 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:33:40.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:40 smithi045 conmon[39718]: cluster 2021-07- 2021-07-31T03:33:40.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:40 smithi045 conmon[39718]: 31T03:33:39.313440+0000 mgr.x (mgr.24542) 4566 : cluster [DBG] pgmap v5200: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:40.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:40 smithi045 conmon[39718]: audit 2021-07-31T03:33:40.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:40 smithi045 conmon[39718]: 2021-07-31T03:33:39.386704+0000 mon.a (mon.0) 2021-07-31T03:33:40.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:40 smithi045 conmon[39718]: 3325 : audit [DBG] from='client.? 172.21.15.45:0/1077129280' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:33:40.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:40 smithi045 conmon[33655]: audit 2021-07-31T03:33:39.040400 2021-07-31T03:33:40.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:40 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 4565 : audit [DBG] from='client.29704 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:33:40.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:40 smithi045 conmon[33655]: cluster 2021-07-31T03:33:39.313440 2021-07-31T03:33:40.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:40 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 4566 : cluster [DBG] pgmap v5200: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:40.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:40 smithi045 conmon[33655]: audit 2021-07-31T03:33:40.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:40 smithi045 conmon[33655]: 2021-07-31T03:33:39.386704+0000 mon.a (mon.0) 3325 : audit 2021-07-31T03:33:40.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:40 smithi045 conmon[33655]: [DBG] from='client.? 172.21.15.45:0/1077129280' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:33:42.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:42 smithi154 conmon[34553]: cluster 2021-07-31T03:33:41.313830+0000 mgr.x (mgr.24542) 4567 : cluster [DBG] pgmap v5201: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:42.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:42 smithi154 conmon[34553]: 2021-07-31T03:33:42.774 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:42 smithi045 conmon[39718]: cluster 2021-07-31T03:33:41.313830+0000 mgr.x (mgr 2021-07-31T03:33:42.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:42 smithi045 conmon[39718]: .24542) 4567 : cluster [DBG] pgmap v5201: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:42.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:42 smithi045 conmon[33655]: cluster 2021-07-31T03:33:41. 2021-07-31T03:33:42.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:42 smithi045 conmon[33655]: 313830+0000 mgr.x (mgr.24542) 4567 : cluster [DBG] pgmap v5201: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:44.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:44 smithi154 conmon[34553]: cluster 2021-07-31T03:33:43.314249+0000 mgr.x (mgr.24542) 4568 : cluster [DBG] pgmap v5202: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:44.774 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:44 smithi045 conmon[33655]: cluster 2021-07-31T03: 2021-07-31T03:33:44.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:44 smithi045 conmon[33655]: 33:43.314249+0000 mgr.x (mgr.24542) 4568 : cluster [DBG] pgmap v5202: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:44.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:44 smithi045 conmon[39718]: cluster 2021-07-31T03:33:43.314249+0000 mgr.x 2021-07-31T03:33:44.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:44 smithi045 conmon[39718]: (mgr.24542) 4568 : cluster [DBG] pgmap v5202: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:46.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:46 smithi154 conmon[34553]: cluster 2021-07-31T03:33:45.314934+0000 mgr.x (mgr.24542) 4569 : cluster [DBG] pgmap v5203: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:46.774 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:46 smithi045 conmon[33655]: cluster 2021-07-31T03:33:45.314934+0000 2021-07-31T03:33:46.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:46 smithi045 conmon[33655]: mgr.x (mgr.24542) 4569 : cluster [DBG] pgmap v5203: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:46.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:46 smithi045 conmon[39718]: cluster 2021-07-31T03:33:45.314934+0000 mgr.x ( 2021-07-31T03:33:46.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:46 smithi045 conmon[39718]: mgr.24542) 4569 : cluster [DBG] pgmap v5203: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:48.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:48 smithi154 conmon[34553]: cluster 2021-07-31T03:33:47.315317+0000 mgr.x (mgr.24542) 4570 : cluster [DBG] pgmap v5204: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:48.774 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:48 smithi045 conmon[33655]: cluster 2021-07-31T03:33:47.315317+0000 mgr.x 2021-07-31T03:33:48.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:48 smithi045 conmon[33655]: (mgr.24542) 4570 : cluster [DBG] pgmap v5204: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:48.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:48 smithi045 conmon[39718]: cluster 2021-07-31T03:33:47.315317+0000 2021-07-31T03:33:48.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:48 smithi045 conmon[39718]: mgr.x (mgr.24542) 4570 : cluster [DBG] pgmap v5204: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:50.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:50 smithi154 conmon[34553]: cluster 2021-07-31T03:33:49.315913+0000 mgr.x (mgr.24542) 4571 : cluster [DBG] pgmap v5205: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:50.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:50 smithi045 conmon[33655]: cluster 2021-07-31T03:33:49.315913+0000 mgr.x (mgr.24542) 4571 : 2021-07-31T03:33:50.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:50 smithi045 conmon[33655]: cluster [DBG] pgmap v5205: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:50.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:50 smithi045 conmon[39718]: cluster 2021-07-31T03:33:49.315913+0000 2021-07-31T03:33:50.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:50 smithi045 conmon[39718]: mgr.x (mgr.24542) 4571 : cluster [DBG] pgmap v5205: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:52.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:52 smithi154 conmon[34553]: cluster 2021-07-31T03:33:51.316346+0000 mgr.x (mgr.24542) 4572 : cluster [DBG] pgmap v5206: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:52.774 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:52 smithi045 conmon[39718]: cluster 2021-07-31T03:33:51. 2021-07-31T03:33:52.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:52 smithi045 conmon[39718]: 316346+0000 mgr.x (mgr.24542) 4572 : cluster [DBG] pgmap v5206: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:52.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:52 smithi045 conmon[33655]: cluster 2021-07-31T03:33:51.316346+0000 mgr.x (mgr 2021-07-31T03:33:52.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:52 smithi045 conmon[33655]: .24542) 4572 : cluster [DBG] pgmap v5206: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:54.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:54 smithi154 conmon[34553]: cluster 2021-07-31T03:33:53.316913+0000 mgr.x (mgr.24542) 4573 : cluster [DBG] pgmap v5207: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:54.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:54 smithi154 conmon[34553]: 2021-07-31T03:33:54.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:54 smithi045 conmon[39718]: cluster 2021-07-31T03:33:53.316913+0000 mgr.x ( 2021-07-31T03:33:54.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:54 smithi045 conmon[39718]: mgr.24542) 4573 : cluster [DBG] pgmap v5207: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:54.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:54 smithi045 conmon[33655]: cluster 2021-07-31T03:33:53.316913+0000 mgr.x (mgr.24542) 4573 : cluster [DBG] pgmap v5207: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:56.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:56 smithi154 conmon[34553]: cluster 2021-07-31T03:33:55.317385+0000 mgr.x (mgr.24542) 4574 : cluster [DBG] 2021-07-31T03:33:56.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:56 smithi154 conmon[34553]: pgmap v5208: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:56.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:56 smithi045 conmon[39718]: cluster 2021-07-31T03:33:55.317385+0000 mgr.x ( 2021-07-31T03:33:56.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:56 smithi045 conmon[39718]: mgr.24542) 4574 : cluster [DBG] pgmap v5208: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:56.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:56 smithi045 conmon[33655]: cluster 2021-07-31T03:33:55.317385+0000 mgr.x (mgr.24542) 4574 : cluster [DBG] pgmap v5208: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:56.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:56 smithi045 conmon[33655]: 2021-07-31T03:33:58.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:33:58 smithi154 conmon[34553]: cluster 2021-07-31T03:33:57.317790+0000 mgr.x (mgr.24542) 4575 : cluster [DBG] pgmap v5209: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:58.774 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:58 smithi045 conmon[39718]: cluster 2021-07-31T03:33:57.317790+0000 mgr.x (mgr 2021-07-31T03:33:58.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:33:58 smithi045 conmon[39718]: .24542) 4575 : cluster [DBG] pgmap v5209: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:33:58.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:58 smithi045 conmon[33655]: cluster 2021-07-31T03:33:57.317790+0000 2021-07-31T03:33:58.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:33:58 smithi045 conmon[33655]: mgr.x (mgr.24542) 4575 : cluster [DBG] pgmap v5209: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:00.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:00 smithi154 conmon[34553]: cluster 2021-07-31T03:33:59.318410+0000 mgr.x (mgr.24542) 4576 : cluster [DBG] pgmap v5210: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:00.774 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:00 smithi045 conmon[39718]: cluster 2021-07-31T03:33:59.318410+0000 mgr.x ( 2021-07-31T03:34:00.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:00 smithi045 conmon[39718]: mgr.24542) 4576 : cluster [DBG] pgmap v5210: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:00.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:00 smithi045 conmon[33655]: cluster 2021-07-31T03:33:59 2021-07-31T03:34:00.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:00 smithi045 conmon[33655]: .318410+0000 mgr.x (mgr.24542) 4576 : cluster [DBG] pgmap v5210: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:02.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:02 smithi154 conmon[34553]: cluster 2021-07-31T03:34:01.318771+0000 mgr.x (mgr.24542) 4577 : cluster [DBG] pgmap v5211: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:02.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:02 smithi154 conmon[34553]: 2021-07-31T03:34:02.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:02 smithi045 conmon[39718]: cluster 2021-07-31T03:34:01.318771+0000 mgr.x (mgr 2021-07-31T03:34:02.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:02 smithi045 conmon[39718]: .24542) 4577 : cluster [DBG] pgmap v5211: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:02.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:02 smithi045 conmon[33655]: cluster 2021-07-31T03:34:01.318771+0000 mgr.x (mgr.24542 2021-07-31T03:34:02.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:02 smithi045 conmon[33655]: ) 4577 : cluster [DBG] pgmap v5211: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:04.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:04 smithi154 conmon[34553]: cluster 2021-07-31T03:34:03.319344+0000 mgr.x (mgr.24542) 4578 : cluster [DBG] pgmap v5212: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:04.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:04 smithi045 conmon[33655]: cluster 2021-07-31T03:34:03.319344+0000 mgr.x ( 2021-07-31T03:34:04.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:04 smithi045 conmon[33655]: mgr.24542) 4578 : cluster [DBG] pgmap v5212: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:04.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:04 smithi045 conmon[39718]: cluster 2021-07-31T03:34:03.319344+0000 2021-07-31T03:34:04.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:04 smithi045 conmon[39718]: mgr.x (mgr.24542) 4578 : cluster [DBG] pgmap v5212: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:06.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:06 smithi154 conmon[34553]: cluster 2021-07-31T03:34:05.319852+0000 mgr.x (mgr.24542) 4579 : cluster [DBG] pgmap v5213: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:06.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:06 smithi045 conmon[33655]: cluster 2021-07-31T03:34:05.319852+0000 mgr.x (mgr 2021-07-31T03:34:06.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:06 smithi045 conmon[33655]: .24542) 4579 : cluster [DBG] pgmap v5213: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:06.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:06 smithi045 conmon[39718]: cluster 2021-07-31T03:34:05.319852+0000 mgr.x (mgr.24542) 2021-07-31T03:34:06.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:06 smithi045 conmon[39718]: 4579 : cluster [DBG] pgmap v5213: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:08.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:08 smithi154 conmon[34553]: cluster 2021-07-31T03:34:07.320283+0000 mgr.x (mgr.24542) 4580 : cluster [DBG] pgmap v5214: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:08.774 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:08 smithi045 conmon[39718]: cluster 2021-07-31T03:34: 2021-07-31T03:34:08.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:08 smithi045 conmon[39718]: 07.320283+0000 mgr.x (mgr.24542) 4580 : cluster [DBG] pgmap v5214: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:08.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:08 smithi045 conmon[33655]: cluster 2021-07-31T03:34:07.320283+0000 mgr.x (mgr.24542) 2021-07-31T03:34:08.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:08 smithi045 conmon[33655]: 4580 : cluster [DBG] pgmap v5214: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:09.697 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:34:09.973 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:34:09.973 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 109m ago 109m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:34:09.974 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 106m ago 109m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:34:09.974 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 106m ago 113m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:34:09.974 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 109m ago 115m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:34:09.974 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 109m ago 115m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:34:09.974 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 106m ago 113m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:34:09.975 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 109m ago 113m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:34:09.975 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 109m ago 110m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:34:09.975 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 106m ago 109m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:34:09.975 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 109m ago 112m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:34:09.976 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 109m ago 112m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:34:09.976 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 109m ago 112m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:34:09.976 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 109m ago 111m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:34:09.976 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 106m ago 111m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:34:09.976 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 106m ago 111m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:34:09.977 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 106m ago 110m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:34:09.977 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 106m ago 110m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:34:09.977 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 106m ago 110m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:34:10.310 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:34:10.310 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:34:10.310 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:34:10.310 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:34:10.311 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:34:10.311 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:34:10.311 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:34:10.311 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:34:10.312 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:34:10.312 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:34:10.312 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:34:10.312 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:34:10.312 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:34:10.312 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:34:10.313 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:34:10.313 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:34:10.313 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:34:10.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:10 smithi154 conmon[34553]: cluster 2021-07-31T03:34:09.320749+0000 mgr.x (mgr.24542) 4581 : cluster [DBG] pgmap v5215: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:10.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:10 smithi154 conmon[34553]: audit 2021-07-31T03:34:09.686323+0000 mgr.x (mgr.24542) 4582 : audit [DBG] from='client.19776 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:34:10.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:10 smithi154 conmon[34553]: audit 2021-07-31T03:34:09.966624+0000 mgr.x (mgr.24542) 4583 : audit [DBG] from='client.19782 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:34:10.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:10 smithi154 conmon[34553]: audit 2021-07-31T03:34:10.310931+0000 mon.a (mon.0) 3326 : audit [DBG] from='client.? 172.21.15.45:0/4119694744' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:34:10.774 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:10 smithi045 conmon[39718]: cluster 2021-07-31T03:34:09.320749+0000 mgr.x ( 2021-07-31T03:34:10.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:10 smithi045 conmon[39718]: mgr.24542) 4581 : cluster [DBG] pgmap v5215: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:10.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:10 smithi045 conmon[39718]: audit 2021-07-31T03:34 2021-07-31T03:34:10.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:10 smithi045 conmon[39718]: :09.686323+0000 mgr.x (mgr.24542) 4582 : audit [DBG] from='client.19776 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:34:10.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:10 smithi045 conmon[39718]: audit 2021-07-31T03:34 2021-07-31T03:34:10.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:10 smithi045 conmon[39718]: :09.966624+0000 mgr.x (mgr.24542) 4583 : audit [DBG] 2021-07-31T03:34:10.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:10 smithi045 conmon[39718]: from='client.19782 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:34:10.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:10 smithi045 conmon[39718]: audit 2021-07-31T03:34:10.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:10 smithi045 conmon[39718]: 2021-07-31T03:34:10.310931+0000 mon.a (mon 2021-07-31T03:34:10.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:10 smithi045 conmon[39718]: .0) 3326 : audit [DBG] from='client.? 172.21.15.45:0/4119694744' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:34:10.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:10 smithi045 conmon[33655]: cluster 2021-07-31T03:34:09 2021-07-31T03:34:10.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:10 smithi045 conmon[33655]: .320749+0000 mgr.x (mgr.24542) 4581 : cluster [DBG] pgmap v5215: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:10.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:10 smithi045 conmon[33655]: audit 2021-07-31T03:34:10.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:10 smithi045 conmon[33655]: 2021-07-31T03:34:09.686323+0000 mgr.x (mgr.24542) 4582 : 2021-07-31T03:34:10.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:10 smithi045 conmon[33655]: audit [DBG] from='client.19776 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:34:10.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:10 smithi045 conmon[33655]: audit 2021-07-31T03: 2021-07-31T03:34:10.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:10 smithi045 conmon[33655]: 34:09.966624+0000 mgr.x (mgr. 2021-07-31T03:34:10.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:10 smithi045 conmon[33655]: 24542) 4583 : audit [DBG] from='client.19782 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:34:10.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:10 smithi045 conmon[33655]: audit 2021-07-31T03:34:10 2021-07-31T03:34:10.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:10 smithi045 conmon[33655]: .310931+0000 mon.a (mon.0) 3326 : audit [DBG] 2021-07-31T03:34:10.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:10 smithi045 conmon[33655]: from='client.? 172.21.15.45:0/4119694744' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:34:12.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:12 smithi154 conmon[34553]: cluster 2021-07-31T03:34:11.321253+0000 mgr.x (mgr.24542) 4584 : cluster [DBG] pgmap v5216: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:12.774 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:12 smithi045 conmon[39718]: cluster 2021-07-31T03:34:11.321253+0000 mgr.x (mgr.24542) 4584 : cluster [DBG] 2021-07-31T03:34:12.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:12 smithi045 conmon[39718]: pgmap v5216: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:12.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:12 smithi045 conmon[33655]: cluster 2021-07-31T03:34:11.321253+0000 mgr.x (mgr.24542) 4584 : cluster [DBG] pgmap v5216: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:14.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:14 smithi154 conmon[34553]: cluster 2021-07-31T03:34:13.321725+0000 mgr.x (mgr.24542) 4585 : cluster [DBG] pgmap v5217: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:14.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:14 smithi045 conmon[33655]: cluster 2021-07-31T03:34:13.321725+0000 2021-07-31T03:34:14.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:14 smithi045 conmon[33655]: mgr.x (mgr.24542) 4585 : cluster [DBG] pgmap v5217: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:14.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:14 smithi045 conmon[39718]: cluster 2021-07-31T03:34:13.321725+0000 mgr.x (mgr.24542) 4585 : cluster 2021-07-31T03:34:14.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:14 smithi045 conmon[39718]: [DBG] pgmap v5217: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:16.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:16 smithi154 conmon[34553]: cluster 2021-07-31T03:34:15.322240+0000 mgr.x (mgr.24542) 4586 : cluster [DBG] pgmap v5218: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:16.774 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:16 smithi045 conmon[33655]: cluster 2021-07-31T03:34:15.322240+0000 mgr.x (mgr.24542) 4586 : cluster 2021-07-31T03:34:16.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:16 smithi045 conmon[33655]: [DBG] pgmap v5218: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:16.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:16 smithi045 conmon[39718]: cluster 2021-07-31T03:34:15.322240+0000 mgr.x (mgr.24542) 4586 : cluster 2021-07-31T03:34:16.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:16 smithi045 conmon[39718]: [DBG] pgmap v5218: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:18.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:18 smithi154 conmon[34553]: cluster 2021-07-31T03:34:17.322797+0000 mgr.x (mgr.24542) 4587 : cluster [DBG] pgmap v5219: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:18.774 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:18 smithi045 conmon[33655]: cluster 2021-07-31T03:34:17.322797+0000 mgr.x (mgr. 2021-07-31T03:34:18.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:18 smithi045 conmon[33655]: 24542) 4587 : cluster [DBG] pgmap v5219: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:18.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:18 smithi045 conmon[39718]: cluster 2021-07-31T03:34:17.322797+0000 mgr.x (mgr.24542) 4587 : cluster [DBG] pgmap v5219: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:20.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:20 smithi154 conmon[34553]: cluster 2021-07-31T03:34:19.323218+0000 mgr.x (mgr.24542) 4588 : cluster [DBG] pgmap v5220: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:20.774 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:20 smithi045 conmon[39718]: cluster 2021-07-31T03:34:19.323218+0000 mgr.x 2021-07-31T03:34:20.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:20 smithi045 conmon[39718]: (mgr.24542) 4588 : cluster [DBG] pgmap v5220: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:20.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:20 smithi045 conmon[33655]: cluster 2021-07-31T03:34:19.323218+0000 mgr.x (mgr.24542) 4588 : cluster [DBG] pgmap v5220: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:22.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:22 smithi154 conmon[34553]: cluster 2021-07-31T03:34:21.323674+0000 mgr.x (mgr.24542) 4589 : cluster [DBG] pgmap v5221: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:22.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:22 smithi045 conmon[33655]: cluster 2021-07-31T03:34:21.323674+0000 mgr.x (mgr.24542) 4589 : 2021-07-31T03:34:22.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:22 smithi045 conmon[33655]: cluster [DBG] pgmap v5221: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:22.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:22 smithi045 conmon[39718]: cluster 2021-07-31T03:34:21.323674+0000 mgr.x (mgr. 2021-07-31T03:34:22.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:22 smithi045 conmon[39718]: 24542) 4589 : cluster [DBG] pgmap v5221: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:24.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:24 smithi154 conmon[34553]: cluster 2021-07-31T03:34:23.324223+0000 mgr.x (mgr.24542) 4590 : cluster [DBG] pgmap v5222: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:24.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:24 smithi045 conmon[33655]: cluster 2021-07-31T03:34:23.324223+0000 mgr.x (mgr.24542) 4590 : cluster [DBG] pgmap v5222: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:24.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:24 smithi045 conmon[39718]: cluster 2021-07-31T03:34:23. 2021-07-31T03:34:24.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:24 smithi045 conmon[39718]: 324223+0000 mgr.x (mgr.24542) 4590 : cluster [DBG] pgmap v5222: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:26.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:26 smithi154 conmon[34553]: cluster 2021-07-31T03:34:25.324791+0000 mgr.x (mgr.24542) 4591 : cluster [DBG] pgmap v5223: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:26.774 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:26 smithi045 conmon[33655]: cluster 2021-07-31T03:34:25.324791+0000 mgr.x (mgr.24542) 4591 2021-07-31T03:34:26.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:26 smithi045 conmon[33655]: : cluster [DBG] pgmap v5223: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:26.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:26 smithi045 conmon[39718]: cluster 2021-07-31T03:34:25.324791 2021-07-31T03:34:26.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:26 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 4591 : cluster [DBG] pgmap v5223: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:28.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:28 smithi154 conmon[34553]: cluster 2021-07-31T03:34:27.325233+0000 mgr.x (mgr.24542) 4592 : cluster [DBG] pgmap v5224: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:28.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:28 smithi154 conmon[34553]: audit 2021-07-31T03:34:28.359860+0000 mon.b (mon.2) 610 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:34:28.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:28 smithi154 conmon[34553]: audit 2021-07-31T03:34:28.360175+0000 mon.b (mon.2) 611 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:34:28.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:28 smithi154 conmon[34553]: audit 2021-07-31T03:34:28.360528+0000 2021-07-31T03:34:28.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:28 smithi154 conmon[34553]: mon.a (mon.0) 3327 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:34:28.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:28 smithi154 conmon[34553]: audit 2021-07-31T 2021-07-31T03:34:28.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:28 smithi154 conmon[34553]: 03:34:28.360980+0000 mon.a (mon.0) 3328 : audit 2021-07-31T03:34:28.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:28 smithi154 conmon[34553]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:34:28.774 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:28 smithi045 conmon[39718]: cluster 2021-07-31T03:34:27.325233+0000 mgr.x ( 2021-07-31T03:34:28.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:28 smithi045 conmon[39718]: mgr.24542) 4592 : cluster [DBG] pgmap v5224: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:28.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:28 smithi045 conmon[39718]: audit 2021-07-31T03:34:28.359860+0000 mon.b (mon. 2021-07-31T03:34:28.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:28 smithi045 conmon[39718]: 2) 610 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:34:28.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:28 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T03:34:28.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:28 smithi045 conmon[39718]: 03:34:28.360175+0000 mon.b (mon.2) 611 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:34:28.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:28 smithi045 conmon[39718]: audit 2021-07-31T03:34:28.360528+0000 mon.a (mon.0) 3327 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:34:28.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:28 smithi045 conmon[39718]: audit 2021-07-31T03:34 2021-07-31T03:34:28.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:28 smithi045 conmon[39718]: :28.360980+0000 mon.a (mon.0) 3328 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:34:28.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:28 smithi045 conmon[33655]: cluster 2021-07-31T03:34:27.325233+0000 mgr.x (mgr.24542) 4592 : cluster [DBG] pgmap v5224: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:28.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:28 smithi045 conmon[33655]: audit 2021-07-31T03:34:28.359860+0000 mon.b (mon.2) 610 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:34:28.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:28 smithi045 conmon[33655]: audit 2021-07-31T03:34:28.360175+0000 mon.b (mon.2) 611 : audit 2021-07-31T03:34:28.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:28 smithi045 conmon[33655]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:34:28.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:28 smithi045 conmon[33655]: audit 2021-07-31T03:34:28.360528+0000 mon.a (mon.0) 3327 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:34:28.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:28 smithi045 conmon[33655]: audit 2021-07-31T03:34:28.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:28 smithi045 conmon[33655]: 2021-07-31T03:34:28.360980+0000 mon.a (mon.0) 3328 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:34:30.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:30 smithi154 conmon[34553]: cluster 2021-07-31T03:34:29.325663+0000 mgr.x (mgr.24542) 4593 : cluster [DBG] pgmap v5225: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:30.774 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:30 smithi045 conmon[33655]: cluster 2021-07-31T03:34:29.325663+0000 mgr.x (mgr.24542 2021-07-31T03:34:30.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:30 smithi045 conmon[33655]: ) 4593 : cluster [DBG] pgmap v5225: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:30.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:30 smithi045 conmon[39718]: cluster 2021-07-31T03:34:29.325663 2021-07-31T03:34:30.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:30 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 4593 : cluster [DBG] pgmap v5225: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:32.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:32 smithi154 conmon[34553]: cluster 2021-07-31T03:34:31.326189+0000 mgr.x (mgr.24542) 4594 : cluster [DBG] pgmap v5226: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:32.774 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:32 smithi045 conmon[33655]: cluster 2021-07-31T03:34:31.326189+0000 mgr.x (mgr.24542) 4594 : cluster 2021-07-31T03:34:32.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:32 smithi045 conmon[33655]: [DBG] pgmap v5226: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:32.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:32 smithi045 conmon[39718]: cluster 2021-07-31T03:34:31.326189+0000 mgr.x (mgr.24542) 4594 : cluster [DBG] pgmap v5226: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:34.252 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: debug 2021-07-31T03:34:33.948+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:34:34.252 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:34:34.252 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:34:34.252 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: yield 2021-07-31T03:34:34.253 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:34:34.253 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:34:34.253 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:34:34.253 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:34:34.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:34:34.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:34:34.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:34:34.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:34:34.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:34:34.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: task.result() 2021-07-31T03:34:34.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:34:34.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: debug 2021-07-31T03:34:33.959+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:34:34.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:34:34.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:34:34.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: yield 2021-07-31T03:34:34.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:34:34.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:34:34.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:34:34.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:34:34.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:34:34.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:34:34.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:34:34.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:34:34.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:34:34.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: task.result() 2021-07-31T03:34:34.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:34:34.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: debug 2021-07-31T03:34:33.985+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:34:34.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:34:34.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:34:34.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: yield 2021-07-31T03:34:34.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:34:34.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:34:34.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:34:34.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:34:34.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:34:34.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:34:34.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:34:34.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:34:34.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:34:34.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: task.result() 2021-07-31T03:34:34.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:34:34.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: debug 2021-07-31T03:34:33.985+0000 7fcb9215e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:34:34.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:34:34.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:34:34.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: yield 2021-07-31T03:34:34.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:34:34.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:34:34.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:34:34.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:34:34.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:34:34.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:34:34.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:34:34.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:34:34.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:34:34.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: task.result() 2021-07-31T03:34:34.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:34:34.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: 2021-07-31T03:34:34.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:34:34.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: 2021-07-31T03:34:34.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:34:34.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:34:34.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:34:34.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:34:34.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:34:34.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:34:34.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:34:34.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:34:34.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:34:34.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:34:34.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:34:34.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:34:34.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:34:34.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:34:34.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:34:34.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:34:34.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:34:34.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:34:34.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: raise 2021-07-31T03:34:34.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:34:34.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:34:34.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:34:34.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:34:34.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:34:34.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: debug 2021-07-31T03:34:33.994+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:34:34.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:34:34.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:34:34.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: yield 2021-07-31T03:34:34.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:34:34.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:34:34.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:34:34.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:34:34.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:34:34.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:34:34.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:34:34.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:34:34.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:34:34.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: task.result() 2021-07-31T03:34:34.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:34:34.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: debug 2021-07-31T03:34:33.994+0000 7fcb9195d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:34:34.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:34:34.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:34:34.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: yield 2021-07-31T03:34:34.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:34:34.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:34:34.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:34:34.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:34:34.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:34:34.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:34:34.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:34:34.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:34:34.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:34:34.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: task.result() 2021-07-31T03:34:34.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:34:34.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: 2021-07-31T03:34:34.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:34:34.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: 2021-07-31T03:34:34.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:34:34.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:34:34.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:34:34.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:34:34.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:34:34.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:34:34.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:34:34.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:34:34.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:34:34.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:34:34.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:34:34.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:34:34.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:34:34.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:34:34.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:34:34.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:34:34.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:34:34.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:34:34.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: raise 2021-07-31T03:34:34.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:34:34.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:34:34.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:34:34.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:34:34.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:34:33 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:34:34.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: cluster 2021-07-31T03:34:33.326651+0000 mgr.x (mgr.24542) 4595 : cluster 2021-07-31T03:34:34.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: [DBG] pgmap v5227: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:34.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: audit 2021-07-31T03:34:33.903447+0000 mon.b (mon.2) 612 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:34:34.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: audit 2021-07-31T03:34:33.904890+0000 mon.b (mon.2) 613 : audit [DBG] 2021-07-31T03:34:34.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:34:34.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: audit 2021-07-31T03 2021-07-31T03:34:34.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: :34:33.906008+0000 mon.b (mon.2) 2021-07-31T03:34:34.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: 614 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:34:34.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: cephadm 2021-07-31T03:34:33.906775+0000 2021-07-31T03:34:34.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: mgr.x (mgr.24542) 4596 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:34:34.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: 2021-07-31T03:34:34.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: cephadm 2021-07-31T03:34:33 2021-07-31T03:34:34.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: .949311+0000 mgr.x (mgr.24542) 4597 : cephadm 2021-07-31T03:34:34.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: [ERR] 2021-07-31T03:34:34.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:34:34.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:34:34.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: yield 2021-07-31T03:34:34.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:34:34.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:34:34.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:34:34.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:34:34.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:34:34.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:34:34.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:34:34.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:34:34.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:34:34.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: task.result() 2021-07-31T03:34:34.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:34:34.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: cephadm 2021-07-31T03:34:33.949819+0000 mgr.x (mgr.24542) 4598 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:34:34.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: cephadm 2021-07-31T03:34:33.960632+0000 mgr.x (mgr.24542) 4599 : cephadm [ERR] 2021-07-31T03:34:34.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: 2021-07-31T03:34:34.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:34:34.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:34:34.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: yield 2021-07-31T03:34:34.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:34:34.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:34:34.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:34:34.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:34:34.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:34:34.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:34:34.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:34:34.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:34:34.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:34:34.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: task.result() 2021-07-31T03:34:34.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:34:34.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: cephadm 2021-07-31T03:34:34.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: 2021-07-31T03:34:33.961477+0000 mgr.x (mgr.24542) 4600 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:34:34.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: cephadm 2021-07-31T03:34:33.985957+0000 mgr.x (mgr.24542) 4601 : cephadm [ERR] 2021-07-31T03:34:34.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:34:34.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:34:34.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: yield 2021-07-31T03:34:34.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:34:34.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:34:34.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:34:34.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:34:34.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:34:34.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:34:34.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:34:34.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:34:34.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:34:34.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: task.result() 2021-07-31T03:34:34.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:34:34.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: cephadm 2021-07-31T03:34:33.986561+0000 mgr.x (mgr.24542) 4602 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:34:34.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:34:34.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:34:34.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: yield 2021-07-31T03:34:34.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:34:34.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:34:34.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:34:34.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:34:34.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:34:34.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:34:34.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:34:34.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:34:34.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:34:34.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: task.result() 2021-07-31T03:34:34.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:34:34.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: 2021-07-31T03:34:34.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:34:34.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: 2021-07-31T03:34:34.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:34:34.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:34:34.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:34:34.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:34:34.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:34:34.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:34:34.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:34:34.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:34:34.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:34:34.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:34:34.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:34:34.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:34:34.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:34:34.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:34:34.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:34:34.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:34:34.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:34:34.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:34:34.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: raise 2021-07-31T03:34:34.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:34:34.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:34:34.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:34:34.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:34:34.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:34:34.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: cephadm 2021-07-31T03:34:33.995167+0000 mgr.x (mgr.24542 2021-07-31T03:34:34.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: ) 4603 : cephadm [ERR] 2021-07-31T03:34:34.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:34:34.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:34:34.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: yield 2021-07-31T03:34:34.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:34:34.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:34:34.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:34:34.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:34:34.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:34:34.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:34:34.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:34:34.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:34:34.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:34:34.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: task.result() 2021-07-31T03:34:34.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: cluster 2021-07-31T03:34:33.326651+0000 mgr.x (mgr.24542) 4595 : cluster [DBG] pgmap v5227: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:34.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: audit 2021-07-31T03:34:33.903447+0000 mon.b 2021-07-31T03:34:34.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: (mon.2) 612 : audit [DBG] 2021-07-31T03:34:34.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:34:34.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: audit 2021-07-31T03:34:34.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: 2021-07-31T03:34 2021-07-31T03:34:34.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: :33.904890+0000 mon.b ( 2021-07-31T03:34:34.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: mon.2) 613 : audit 2021-07-31T03:34:34.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:34:34.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: audit 2021-07 2021-07-31T03:34:34.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: -31T03:34:33.906008+0000 2021-07-31T03:34:34.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: mon.b (mon.2) 614 2021-07-31T03:34:34.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:34:34.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: cephadm 2021- 2021-07-31T03:34:34.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: 07-31T03:34:33. 2021-07-31T03:34:34.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: 906775+0000 mgr.x (mgr.24542 2021-07-31T03:34:34.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: ) 4596 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:34:34.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: 2021-07-31T03:34:34.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: cephadm 2021-07-31T03:34:34.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: 2021-07-31T03: 2021-07-31T03:34:34.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: 34:33.949311+0000 2021-07-31T03:34:34.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: mgr.x (mgr.24542) 4597 : 2021-07-31T03:34:34.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: cephadm [ERR] 2021-07-31T03:34:34.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:34:34.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:34:34.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: yield 2021-07-31T03:34:34.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:34:34.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:34:34.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:34:34.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:34:34.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:34:34.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:34:34.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:34:34.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:34:34.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:34:34.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: task.result() 2021-07-31T03:34:34.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:34:34.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: cephadm 2021-07-31T03:34:33.949819+0000 mgr.x (mgr.24542) 4598 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:34:34.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: cephadm 2021-07-31T03:34:34.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: 2021-07-31T03: 2021-07-31T03:34:34.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: 34:33.960632+0000 mgr.x 2021-07-31T03:34:34.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: (mgr.24542) 4599 : cephadm 2021-07-31T03:34:34.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: [ERR] 2021-07-31T03:34:34.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:34:34.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:34:34.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: cluster 2021-07-31T03:34:33.326651+0000 mgr.x (mgr.24542) 4595 : cluster 2021-07-31T03:34:34.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: [DBG] pgmap v5227: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:34.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: audit 2021-07-31T03:34:33.903447+0000 mon.b (mon 2021-07-31T03:34:34.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: .2) 612 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:34:34.788 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: audit 2021-07-31T03:34:33. 2021-07-31T03:34:34.788 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: 904890+0000 mon.b (mon.2) 613 : audit 2021-07-31T03:34:34.788 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:34:34.788 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: audit 2021-07-31T03:34:34.788 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: 2021-07-31T03:34:33.906008+0000 2021-07-31T03:34:34.789 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: mon.b (mon.2) 614 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:34:34.789 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: cephadm 2021-07- 2021-07-31T03:34:34.789 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: 31T03:34:33.906775+0000 mgr.x (mgr.24542) 4596 : 2021-07-31T03:34:34.789 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:34:34.789 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: cephadm 2021-07-31T03:34:34.790 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: 2021-07-31 2021-07-31T03:34:34.790 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: T03:34:33.949311+0000 mgr.x (mgr. 2021-07-31T03:34:34.790 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: 24542) 4597 : cephadm [ERR] 2021-07-31T03:34:34.790 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:34:34.791 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:34:34.791 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: yield 2021-07-31T03:34:34.791 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:34:34.791 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:34:34.791 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:34:34.791 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:34:34.792 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:34:34.792 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:34:34.792 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:34:34.792 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:34:34.793 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:34:34.793 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: task.result() 2021-07-31T03:34:34.793 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:34:34.793 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: cephadm 2021-07-31T03:34:33.949819+0000 mgr.x (mgr.24542) 4598 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:34:34.793 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: cephadm 2021-07-31T03:34:33.960632+0000 2021-07-31T03:34:34.793 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: mgr.x (mgr.24542) 4599 : cephadm [ERR] 2021-07-31T03:34:34.794 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:34:34.794 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:34:34.794 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: yield 2021-07-31T03:34:34.794 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:34:34.795 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:34:34.795 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:34:34.795 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: cephadm 2021-07-31T03:34:33.995579+0000 mgr.x (mgr.24542) 4604 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:34:34.795 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:34:34.796 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:34:34.796 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: yield 2021-07-31T03:34:34.796 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:34:34.796 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:34:34.796 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:34:34.797 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:34:34.797 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:34:34.797 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:34:34.797 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:34:34.797 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:34:34.797 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:34:34.798 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: task.result() 2021-07-31T03:34:34.798 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:34:34.798 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: 2021-07-31T03:34:34.798 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:34:34.799 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: 2021-07-31T03:34:34.799 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:34:34.799 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:34:34.799 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:34:34.799 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:34:34.800 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:34:34.800 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:34:34.800 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:34:34.800 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:34:34.800 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:34:34.801 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:34:34.801 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:34:34.801 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:34:34.801 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:34:34.801 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:34:34.802 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:34:34.802 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:34:34.802 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:34:34.802 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:34:34.802 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: raise 2021-07-31T03:34:34.803 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:34:34.803 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:34:34.803 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:34:34.803 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:34:34.803 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:34 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:34:34.804 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: yield 2021-07-31T03:34:34.804 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:34:34.804 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:34:34.805 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:34:34.805 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:34:34.805 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:34:34.805 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:34:34.805 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:34:34.806 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:34:34.806 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:34:34.806 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: task.result() 2021-07-31T03:34:34.806 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:34:34.807 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: cephadm 2021-07-31T03:34:33.961477+0000 mgr.x (mgr.24542) 4600 : cephadm [INF] 2021-07-31T03:34:34.807 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:34:34.807 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: cephadm 2021-07-31T03:34:34.807 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: 2021-07-31T03: 2021-07-31T03:34:34.807 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: 34:33.985957+0000 mgr.x ( 2021-07-31T03:34:34.808 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: mgr.24542) 4601 : cephadm [ERR] 2021-07-31T03:34:34.808 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: 2021-07-31T03:34:34.808 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:34:34.808 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:34:34.808 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: yield 2021-07-31T03:34:34.809 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:34:34.809 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:34:34.809 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:34:34.809 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:34:34.809 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:34:34.810 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:34:34.810 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:34:34.810 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:34:34.810 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:34:34.810 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: task.result() 2021-07-31T03:34:34.811 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:34:34.811 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: cephadm 2021-07-31T03:34:33.986561+0000 mgr.x (mgr.24542) 4602 : cephadm [ERR] 2021-07-31T03:34:34.811 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:34:34.811 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:34:34.812 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:34:34.812 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: yield 2021-07-31T03:34:34.812 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:34:34.812 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:34:34.812 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:34:34.812 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:34:34.813 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:34:34.813 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:34:34.813 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:34:34.813 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:34:34.813 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:34:34.814 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: task.result() 2021-07-31T03:34:34.814 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:34:34.814 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: 2021-07-31T03:34:34.814 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:34:34.814 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: 2021-07-31T03:34:34.815 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:34:34.815 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:34:34.815 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:34:34.815 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:34:34.815 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:34:34.816 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:34:34.816 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:34:34.816 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:34:34.816 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:34:34.816 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:34:34.817 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:34:34.817 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:34:34.817 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:34:34.817 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:34:34.817 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:34:34.818 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:34:34.818 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:34:34.818 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:34:34.818 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: raise 2021-07-31T03:34:34.818 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:34:34.819 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:34:34.819 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:34:34.819 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:34:34.819 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:34:34.819 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: cephadm 2021-07-31T03:34:33.995167+0000 mgr.x (mgr.24542) 4603 : cephadm [ERR] 2021-07-31T03:34:34.820 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:34:34.820 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:34:34.820 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: yield 2021-07-31T03:34:34.820 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:34:34.821 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:34:34.821 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:34:34.821 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:34:34.821 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:34:34.822 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:34:34.822 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:34:34.822 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:34:34.822 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:34:34.822 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:34:34.823 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: task.result() 2021-07-31T03:34:34.823 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:34:34.823 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: cephadm 2021-07-31T03:34:33.961477+0000 mgr.x (mgr.24542) 4600 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:34:34.823 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: cephadm 2021-07-31T03:34:33.985957 2021-07-31T03:34:34.823 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 4601 : cephadm [ERR] 2021-07-31T03:34:34.824 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:34:34.824 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:34:34.824 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: yield 2021-07-31T03:34:34.824 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:34:34.824 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:34:34.825 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:34:34.825 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:34:34.825 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:34:34.825 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:34:34.825 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:34:34.825 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:34:34.826 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:34:34.826 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: task.result() 2021-07-31T03:34:34.826 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:34:34.826 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: cephadm 2021-07-31T03:34:33.986561+0000 mgr.x (mgr.24542) 4602 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:34:34.827 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:34:34.827 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:34:34.827 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: yield 2021-07-31T03:34:34.827 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:34:34.827 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:34:34.828 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:34:34.828 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:34:34.828 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:34:34.828 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:34:34.828 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:34:34.828 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:34:34.829 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:34:34.829 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: task.result() 2021-07-31T03:34:34.829 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:34:34.830 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:34:34.830 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:34:34.830 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:34:34.830 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:34:34.831 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:34:34.831 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: task.result() 2021-07-31T03:34:34.831 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:34:34.831 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: cephadm 2021-07-31T03:34:33.995579+0000 mgr.x (mgr.24542) 4604 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:34:34.831 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:34:34.832 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:34:34.832 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: yield 2021-07-31T03:34:34.832 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:34:34.832 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:34:34.832 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:34:34.833 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:34:34.833 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:34:34.833 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:34:34.833 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:34:34.833 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:34:34.834 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:34:34.834 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: task.result() 2021-07-31T03:34:34.834 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:34:34.834 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: 2021-07-31T03:34:34.834 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:34:34.835 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: 2021-07-31T03:34:34.835 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:34:34.835 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:34:34.835 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:34:34.835 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:34:34.836 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:34:34.836 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:34:34.836 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:34:34.836 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:34:34.836 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:34:34.837 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:34:34.837 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:34:34.837 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:34:34.837 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:34:34.837 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:34:34.838 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:34:34.838 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:34:34.838 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:34:34.838 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:34:34.838 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: raise 2021-07-31T03:34:34.839 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:34:34.839 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:34:34.839 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:34:34.839 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:34:34.839 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:34:34.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:34:34.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: 2021-07-31T03:34:34.841 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:34:34.841 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: 2021-07-31T03:34:34.841 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:34:34.841 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:34:34.841 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:34:34.842 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:34:34.842 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:34:34.842 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:34:34.842 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:34:34.842 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:34:34.843 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:34:34.843 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:34:34.843 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:34:34.843 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:34:34.843 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:34:34.844 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:34:34.844 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:34:34.844 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:34:34.844 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:34:34.844 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:34:34.845 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: raise 2021-07-31T03:34:34.845 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:34:34.845 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:34:34.845 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:34:34.845 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:34:34.846 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:34:34.846 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: cephadm 2021-07-31T03:34:33.995167+0000 mgr.x (mgr.24542) 4603 : cephadm [ERR] 2021-07-31T03:34:34.846 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:34:34.846 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:34:34.846 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: yield 2021-07-31T03:34:34.847 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:34:34.847 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:34:34.847 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:34:34.847 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:34:34.848 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:34:34.848 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:34:34.848 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:34:34.848 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:34:34.848 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:34:34.848 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: task.result() 2021-07-31T03:34:34.849 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:34:34.849 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: cephadm 2021-07-31T03:34:33.995579+0000 mgr.x (mgr.24542) 4604 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:34:34.849 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:34:34.849 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:34:34.849 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: yield 2021-07-31T03:34:34.850 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:34:34.850 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:34:34.850 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:34:34.850 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:34:34.850 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:34:34.851 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:34:34.851 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:34:34.851 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:34:34.851 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:34:34.851 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: task.result() 2021-07-31T03:34:34.852 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:34:34.852 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: 2021-07-31T03:34:34.852 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:34:34.852 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: 2021-07-31T03:34:34.852 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:34:34.853 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:34:34.853 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:34:34.853 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:34:34.853 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:34:34.853 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:34:34.854 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:34:34.854 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:34:34.854 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:34:34.854 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:34:34.854 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:34:34.855 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:34:34.855 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:34:34.855 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:34:34.855 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:34:34.855 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:34:34.856 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:34:34.856 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:34:34.856 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: raise 2021-07-31T03:34:34.856 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:34:34.856 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:34:34.857 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:34:34.857 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:34:34.857 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:34 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:34:36.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:36 smithi154 conmon[34553]: cluster 2021-07-31T03:34:35.327129+0000 mgr.x (mgr.24542) 4605 : cluster [DBG] pgmap v5228: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:36.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:36 smithi154 conmon[34553]: 2021-07-31T03:34:36.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:36 smithi045 conmon[33655]: cluster 2021-07-31T03:34:35.327129+0000 mgr.x (mgr.24542) 4605 : 2021-07-31T03:34:36.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:36 smithi045 conmon[33655]: cluster [DBG] pgmap v5228: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:36.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:36 smithi045 conmon[39718]: cluster 2021-07-31T03:34:35.327129+0000 mgr.x (mgr.24542) 4605 : cluster [DBG] 2021-07-31T03:34:36.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:36 smithi045 conmon[39718]: pgmap v5228: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:38.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:38 smithi154 conmon[34553]: cluster 2021-07-31T03:34:37.327592+0000 mgr.x (mgr.24542) 4606 : cluster [DBG] 2021-07-31T03:34:38.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:38 smithi154 conmon[34553]: pgmap v5229: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:38.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:38 smithi045 conmon[39718]: cluster 2021-07-31T03:34:37 2021-07-31T03:34:38.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:38 smithi045 conmon[39718]: .327592+0000 mgr.x (mgr.24542) 4606 : cluster [DBG] pgmap v5229: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:38.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:38 smithi045 conmon[33655]: cluster 2021-07-31T03:34:37.327592+0000 mgr.x (mgr.24542) 4606 : cluster [DBG] 2021-07-31T03:34:38.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:38 smithi045 conmon[33655]: pgmap v5229: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:40.628 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:34:40.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:40 smithi154 conmon[34553]: cluster 2021-07-31T03:34:39.328196+0000 mgr.x (mgr.24542) 4607 : cluster 2021-07-31T03:34:40.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:40 smithi154 conmon[34553]: [DBG] pgmap v5230: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:40.774 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:40 smithi045 conmon[39718]: cluster 2021-07-31T03:34:39.328196+0000 2021-07-31T03:34:40.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:40 smithi045 conmon[39718]: mgr.x (mgr.24542) 4607 : cluster [DBG] pgmap v5230: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:40.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:40 smithi045 conmon[33655]: cluster 2021-07-31T03:34:39 2021-07-31T03:34:40.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:40 smithi045 conmon[33655]: .328196+0000 mgr.x (mgr.24542) 4607 : cluster [DBG] pgmap v5230: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:40.909 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:34:40.909 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 109m ago 110m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:34:40.909 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 107m ago 110m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:34:40.909 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 107m ago 113m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:34:40.909 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 109m ago 115m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:34:40.910 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 109m ago 115m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:34:40.910 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 107m ago 113m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:34:40.910 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 109m ago 114m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:34:40.910 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 109m ago 110m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:34:40.910 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 107m ago 110m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:34:40.911 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 109m ago 113m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:34:40.911 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 109m ago 112m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:34:40.911 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 109m ago 112m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:34:40.911 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 109m ago 112m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:34:40.911 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 107m ago 111m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:34:40.912 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 107m ago 111m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:34:40.912 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 107m ago 111m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:34:40.912 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 107m ago 110m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:34:40.912 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 107m ago 110m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:34:41.243 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:34:41.243 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:34:41.244 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:34:41.244 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:34:41.244 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:34:41.244 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:34:41.244 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:34:41.245 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:34:41.245 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:34:41.245 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:34:41.245 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:34:41.245 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:34:41.246 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:34:41.246 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:34:41.246 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:34:41.246 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:34:41.247 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:34:41.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:41 smithi154 conmon[34553]: audit 2021-07-31T03:34:40.617632+0000 mgr.x (mgr.24542) 4608 : audit [DBG] from='client.19794 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:34:41.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:41 smithi154 conmon[34553]: 2021-07-31T03:34:41.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:41 smithi154 conmon[34553]: audit 2021-07-31T03:34:40.902069+0000 mgr.x (mgr.24542) 4609 : audit [DBG] from='client.29740 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:34:41.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:41 smithi154 conmon[34553]: audit 2021-07-31T03:34:41.244267+0000 mon.c (mon.1) 214 : audit [DBG] from='client.? 172.21.15.45:0/202757519' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:34:41.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:41 smithi045 conmon[33655]: audit 2021-07-31T03:34:40.617632+0000 mgr.x (mgr.24542) 4608 : audit [DBG] from='client.19794 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:34:41.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:41 smithi045 conmon[33655]: audit 2021-07-31T03:34:40.902069+0000 mgr.x (mgr.24542) 4609 : audit [DBG] 2021-07-31T03:34:41.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:41 smithi045 conmon[33655]: from='client.29740 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:34:41.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:41 smithi045 conmon[33655]: audit 2021-07-31T 2021-07-31T03:34:41.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:41 smithi045 conmon[33655]: 03:34:41.244267+0000 2021-07-31T03:34:41.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:41 smithi045 conmon[33655]: mon.c (mon.1) 214 : audit 2021-07-31T03:34:41.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:41 smithi045 conmon[33655]: [DBG] from='client.? 172.21.15.45:0/202757519' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:34:41.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:41 smithi045 conmon[39718]: audit 2021-07-31T03:34:40.617632+0000 2021-07-31T03:34:41.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:41 smithi045 conmon[39718]: mgr.x (mgr.24542) 4608 : audit [DBG] from='client.19794 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:34:41.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:41 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T03:34:41.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:41 smithi045 conmon[39718]: T03:34:40.902069+0000 2021-07-31T03:34:41.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:41 smithi045 conmon[39718]: mgr.x (mgr.24542) 4609 2021-07-31T03:34:41.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:41 smithi045 conmon[39718]: : audit [DBG] from='client.29740 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:34:41.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:41 smithi045 conmon[39718]: audit 2021-07-31T03:34:41.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:41 smithi045 conmon[39718]: 2021-07-31T03:34: 2021-07-31T03:34:41.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:41 smithi045 conmon[39718]: 41.244267+0000 mon.c 2021-07-31T03:34:41.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:41 smithi045 conmon[39718]: (mon.1) 214 : audit 2021-07-31T03:34:41.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:41 smithi045 conmon[39718]: [DBG] from='client.? 172.21.15.45:0/202757519' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:34:42.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:42 smithi154 conmon[34553]: cluster 2021-07-31T03:34:41.328606+0000 mgr.x (mgr.24542) 4610 : cluster [DBG] 2021-07-31T03:34:42.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:42 smithi154 conmon[34553]: pgmap v5231: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:42.774 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:42 smithi045 conmon[39718]: cluster 2021-07-31T03:34:41.328606+0000 mgr.x (mgr. 2021-07-31T03:34:42.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:42 smithi045 conmon[39718]: 24542) 4610 : cluster [DBG] pgmap v5231: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:42.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:42 smithi045 conmon[33655]: cluster 2021-07-31T03:34:41.328606+0000 mgr.x (mgr.24542) 4610 : 2021-07-31T03:34:42.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:42 smithi045 conmon[33655]: cluster [DBG] pgmap v5231: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:44.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:44 smithi154 conmon[34553]: cluster 2021-07-31T03:34:43.329056+0000 mgr.x (mgr.24542) 4611 : cluster [DBG] pgmap v5232: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:44.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:44 smithi154 conmon[34553]: 2021-07-31T03:34:44.774 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:44 smithi045 conmon[39718]: cluster 2021-07-31T03:34:43.329056+0000 mgr.x (mgr.24542) 2021-07-31T03:34:44.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:44 smithi045 conmon[39718]: 4611 : cluster [DBG] pgmap v5232: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:44.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:44 smithi045 conmon[33655]: cluster 2021-07-31T03:34:43.329056+0000 mgr.x (mgr.24542) 4611 : cluster [DBG] pgmap v5232: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:46.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:46 smithi154 conmon[34553]: cluster 2021-07-31T03:34:45.329631+0000 mgr.x (mgr.24542) 4612 : cluster [DBG] pgmap v5233: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:46.774 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:46 smithi045 conmon[39718]: cluster 2021-07-31T03:34:45.329631+0000 mgr.x (mgr 2021-07-31T03:34:46.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:46 smithi045 conmon[39718]: .24542) 4612 : cluster [DBG] pgmap v5233: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:46.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:46 smithi045 conmon[33655]: cluster 2021-07-31T03:34:45.329631+0000 mgr.x (mgr.24542) 4612 : cluster 2021-07-31T03:34:46.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:46 smithi045 conmon[33655]: [DBG] pgmap v5233: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:48.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:48 smithi154 conmon[34553]: cluster 2021-07-31T03:34:47.330036+0000 mgr.x (mgr.24542) 4613 : cluster [DBG] pgmap v5234: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:48.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:48 smithi045 conmon[33655]: cluster 2021-07-31T03:34:47.330036 2021-07-31T03:34:48.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:48 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 4613 : cluster [DBG] pgmap v5234: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:48.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:48 smithi045 conmon[39718]: cluster 2021-07-31T03:34:47.330036+0000 mgr.x (mgr.24542) 2021-07-31T03:34:48.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:48 smithi045 conmon[39718]: 4613 : cluster [DBG] pgmap v5234: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:50.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:50 smithi154 conmon[34553]: cluster 2021-07-31T03:34:49.330489+0000 mgr.x (mgr.24542) 4614 : cluster [DBG] pgmap v5235: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:50.774 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:50 smithi045 conmon[39718]: cluster 2021-07-31T03:34:49.330489+0000 mgr.x ( 2021-07-31T03:34:50.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:50 smithi045 conmon[39718]: mgr.24542) 4614 : cluster [DBG] pgmap v5235: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:50.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:50 smithi045 conmon[33655]: cluster 2021-07-31T03:34:49.330489+0000 mgr.x (mgr.24542) 4614 : cluster 2021-07-31T03:34:50.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:50 smithi045 conmon[33655]: [DBG] pgmap v5235: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:52.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:52 smithi154 conmon[34553]: cluster 2021-07-31T03:34:51.330906+0000 mgr.x (mgr.24542) 4615 : cluster [DBG] pgmap v5236: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:52.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:52 smithi045 conmon[33655]: cluster 2021-07-31T03:34:51.330906+0000 mgr.x (mgr.24542) 4615 : cluster [DBG] 2021-07-31T03:34:52.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:52 smithi045 conmon[33655]: pgmap v5236: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:52.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:52 smithi045 conmon[39718]: cluster 2021-07-31T03:34:51.330906+0000 2021-07-31T03:34:52.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:52 smithi045 conmon[39718]: mgr.x (mgr.24542) 4615 : cluster [DBG] pgmap v5236: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:54.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:54 smithi154 conmon[34553]: cluster 2021-07-31T03:34:53.331436+0000 mgr.x (mgr.24542) 4616 : cluster [DBG] pgmap v5237: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:54.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:54 smithi045 conmon[33655]: cluster 2021-07-31T03:34:53.331436+0000 mgr.x (mgr.24542) 4616 : cluster [DBG] pgmap v5237: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:54.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:54 smithi045 conmon[33655]: 2021-07-31T03:34:54.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:54 smithi045 conmon[39718]: cluster 2021-07-31T03:34:53.331436+0000 mgr.x (mgr.24542) 4616 : cluster [DBG] pgmap v5237: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:56.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:56 smithi154 conmon[34553]: cluster 2021-07-31T03:34:55.331888+0000 mgr.x (mgr.24542) 4617 : cluster [DBG] pgmap v5238: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:56.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:56 smithi045 conmon[33655]: cluster 2021-07-31T03:34:55.331888+0000 mgr.x (mgr.24542) 4617 : 2021-07-31T03:34:56.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:56 smithi045 conmon[33655]: cluster [DBG] pgmap v5238: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:56.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:56 smithi045 conmon[39718]: cluster 2021-07-31T03:34:55.331888+0000 mgr.x (mgr.24542) 4617 : cluster [DBG] pgmap v5238: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:58.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:34:58 smithi154 conmon[34553]: cluster 2021-07-31T03:34:57.332211+0000 mgr.x (mgr.24542) 4618 : cluster [DBG] pgmap v5239: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:58.774 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:34:58 smithi045 conmon[39718]: cluster 2021-07-31T03:34:57.332211+0000 mgr.x (mgr.24542) 4618 : cluster [DBG] pgmap v5239: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:34:58.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:58 smithi045 conmon[33655]: cluster 2021-07-31T03:34:57.332211+0000 mgr.x 2021-07-31T03:34:58.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:34:58 smithi045 conmon[33655]: (mgr.24542) 4618 : cluster [DBG] pgmap v5239: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:00.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:00 smithi154 conmon[34553]: cluster 2021-07-31T03:34:59.332821+0000 mgr.x (mgr.24542) 4619 : cluster [DBG] pgmap v5240: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:00.774 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:00 smithi045 conmon[33655]: cluster 2021-07-31T03:34:59.332821+0000 mgr.x (mgr.24542) 4619 : cluster [DBG] pgmap v5240: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:00.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:00 smithi045 conmon[39718]: cluster 2021-07-31T03:34:59.332821+0000 mgr.x (mgr.24542) 4619 : cluster [DBG] pgmap v5240: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:02.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:02 smithi154 conmon[34553]: cluster 2021-07-31T03:35:01.333212+0000 mgr.x (mgr.24542) 4620 : cluster [DBG] pgmap v5241: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:02.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:02 smithi045 conmon[33655]: cluster 2021-07-31T03:35:01.333212+0000 2021-07-31T03:35:02.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:02 smithi045 conmon[33655]: mgr.x (mgr.24542) 4620 : cluster [DBG] pgmap v5241: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:02.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:02 smithi045 conmon[39718]: cluster 2021-07-31T03:35:01.333212+0000 2021-07-31T03:35:02.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:02 smithi045 conmon[39718]: mgr.x (mgr.24542) 4620 : cluster [DBG] pgmap v5241: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:04.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:04 smithi154 conmon[34553]: cluster 2021-07-31T03:35:03.333595+0000 mgr.x (mgr.24542) 4621 : cluster [DBG] pgmap v5242: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:04.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:04 smithi045 conmon[33655]: cluster 2021-07-31T03:35:03.333595+0000 mgr.x (mgr 2021-07-31T03:35:04.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:04 smithi045 conmon[33655]: .24542) 4621 : cluster [DBG] pgmap v5242: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:04.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:04 smithi045 conmon[39718]: cluster 2021-07-31T03:35:03.333595+0000 mgr.x (mgr.24542) 4621 : 2021-07-31T03:35:04.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:04 smithi045 conmon[39718]: cluster [DBG] pgmap v5242: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:06.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:06 smithi045 conmon[33655]: cluster 2021-07-31T03:35:05.334111+0000 mgr.x (mgr.24542) 4622 : cluster 2021-07-31T03:35:06.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:06 smithi045 conmon[33655]: [DBG] pgmap v5243: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:06.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:06 smithi045 conmon[39718]: cluster 2021-07-31T03:35:05.334111+0000 mgr.x (mgr.24542 2021-07-31T03:35:06.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:06 smithi045 conmon[39718]: ) 4622 : cluster [DBG] pgmap v5243: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:07.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:06 smithi154 conmon[34553]: cluster 2021-07-31T03:35:05.334111+0000 mgr.x (mgr.24542) 4622 : cluster [DBG] pgmap v5243: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:08.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:08 smithi045 conmon[33655]: cluster 2021-07-31T03:35:07.334467+0000 mgr.x (mgr.24542) 4623 : cluster [DBG] pgmap v5244: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:08.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:08 smithi045 conmon[39718]: cluster 2021-07-31T03:35:07.334467+0000 mgr.x ( 2021-07-31T03:35:08.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:08 smithi045 conmon[39718]: mgr.24542) 4623 : cluster [DBG] pgmap v5244: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:09.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:08 smithi154 conmon[34553]: cluster 2021-07-31T03:35:07.334467+0000 mgr.x (mgr.24542) 4623 : cluster [DBG] pgmap v5244: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:10.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:10 smithi045 conmon[39718]: cluster 2021-07-31T03:35:09.334958+0000 mgr.x ( 2021-07-31T03:35:10.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:10 smithi045 conmon[39718]: mgr.24542) 4624 : cluster [DBG] pgmap v5245: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:10.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:10 smithi045 conmon[33655]: cluster 2021-07-31T03:35:09.334958+0000 mgr.x (mgr.24542) 4624 : cluster [DBG] pgmap v5245: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:11.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:10 smithi154 conmon[34553]: cluster 2021-07-31T03:35:09.334958+0000 mgr.x (mgr.24542) 4624 : cluster [DBG] pgmap v5245: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:11.556 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:35:11.833 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:35:11.834 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 110m ago 110m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:35:11.834 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 107m ago 110m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:35:11.834 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 107m ago 114m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:35:11.834 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 110m ago 116m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:35:11.835 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 110m ago 116m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:35:11.835 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 107m ago 114m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:35:11.835 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 110m ago 114m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:35:11.835 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 110m ago 111m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:35:11.835 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 107m ago 111m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:35:11.836 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 110m ago 113m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:35:11.836 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 110m ago 113m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:35:11.836 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 110m ago 113m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:35:11.836 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 110m ago 112m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:35:11.837 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 107m ago 112m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:35:11.837 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 107m ago 112m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:35:11.837 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 107m ago 111m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:35:11.837 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 107m ago 111m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:35:11.837 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 107m ago 111m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:35:12.168 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:35:12.169 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:35:12.169 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:35:12.169 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:35:12.169 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:35:12.170 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:35:12.170 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:35:12.170 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:35:12.170 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:35:12.170 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:35:12.171 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:35:12.171 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:35:12.171 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:35:12.171 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:35:12.172 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:35:12.172 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:35:12.172 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:35:12.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:12 smithi045 conmon[33655]: cluster 2021-07-31T03:35:11.335432+0000 mgr.x (mgr.24542 2021-07-31T03:35:12.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:12 smithi045 conmon[33655]: ) 4625 : cluster [DBG] pgmap v5246: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:12.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:12 smithi045 conmon[33655]: audit 2021-07-31T03:35:11.545583+0000 mgr.x (mgr.24542 2021-07-31T03:35:12.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:12 smithi045 conmon[33655]: ) 4626 : audit [DBG] from='client.29752 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:35:12.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:12 smithi045 conmon[33655]: audit 2021 2021-07-31T03:35:12.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:12 smithi045 conmon[33655]: -07-31T03:35:11.826196 2021-07-31T03:35:12.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:12 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 4627 2021-07-31T03:35:12.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:12 smithi045 conmon[33655]: : audit [DBG] from='client.29758 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:35:12.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:12 smithi045 conmon[33655]: audit 2021- 2021-07-31T03:35:12.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:12 smithi045 conmon[33655]: 07-31T03:35:12.169776+0000 mon.a 2021-07-31T03:35:12.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:12 smithi045 conmon[33655]: (mon.0) 3329 : audit 2021-07-31T03:35:12.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:12 smithi045 conmon[33655]: [DBG] from='client.? 172.21.15.45:0/3732376131' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:35:12.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:12 smithi045 conmon[39718]: cluster 2021-07-31T03:35 2021-07-31T03:35:12.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:12 smithi045 conmon[39718]: :11.335432+0000 mgr.x (mgr.24542) 4625 : cluster [DBG] pgmap v5246: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:12.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:12 smithi045 conmon[39718]: audit 2021- 2021-07-31T03:35:12.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:12 smithi045 conmon[39718]: 07-31T03:35:11. 2021-07-31T03:35:12.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:12 smithi045 conmon[39718]: 545583+0000 mgr.x (mgr.24542 2021-07-31T03:35:12.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:12 smithi045 conmon[39718]: ) 4626 : audit [DBG] from='client.29752 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:35:12.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:12 smithi045 conmon[39718]: 2021-07-31T03:35:12.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:12 smithi045 conmon[39718]: audit 2021-07 2021-07-31T03:35:12.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:12 smithi045 conmon[39718]: -31T03:35:11.826196 2021-07-31T03:35:12.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:12 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 4627 2021-07-31T03:35:12.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:12 smithi045 conmon[39718]: : audit [DBG] from='client.29758 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:35:12.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:12 smithi045 conmon[39718]: audit 2021 2021-07-31T03:35:12.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:12 smithi045 conmon[39718]: -07-31T03:35:12 2021-07-31T03:35:12.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:12 smithi045 conmon[39718]: .169776+0000 mon.a (mon 2021-07-31T03:35:12.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:12 smithi045 conmon[39718]: .0) 3329 : audit [DBG] 2021-07-31T03:35:12.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:12 smithi045 conmon[39718]: from='client.? 172.21.15.45:0/3732376131' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:35:13.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:12 smithi154 conmon[34553]: cluster 2021-07-31T03:35:11.335432+0000 mgr.x (mgr.24542) 4625 : cluster [DBG] pgmap v5246: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:13.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:12 smithi154 conmon[34553]: audit 2021-07-31T03:35:11.545583+0000 mgr.x (mgr.24542) 4626 : audit [DBG] from='client.29752 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:35:13.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:12 smithi154 conmon[34553]: audit 2021-07-31T03:35:11.826196+0000 mgr.x (mgr.24542) 4627 : audit [DBG] from='client.29758 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:35:13.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:12 smithi154 conmon[34553]: audit 2021-07-31T03:35:12.169776+0000 mon.a (mon.0) 3329 : audit [DBG] from='client.? 172.21.15.45:0/3732376131' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:35:14.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:14 smithi045 conmon[33655]: cluster 2021-07-31T03:35:13.335815+0000 mgr.x (mgr.24542) 4628 : cluster [DBG] pgmap v5247: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:14.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:14 smithi045 conmon[39718]: cluster 2021-07-31T03:35:13.335815 2021-07-31T03:35:14.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:14 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 4628 : cluster [DBG] pgmap v5247: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:15.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:14 smithi154 conmon[34553]: cluster 2021-07-31T03:35:13.335815+0000 mgr.x (mgr.24542) 4628 : cluster [DBG] pgmap v5247: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:16.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:16 smithi045 conmon[39718]: cluster 2021-07-31T03:35:15. 2021-07-31T03:35:16.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:16 smithi045 conmon[39718]: 336252+0000 mgr.x (mgr.24542) 4629 : cluster [DBG] pgmap v5248: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:16.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:16 smithi045 conmon[33655]: cluster 2021-07-31T03:35:15.336252+0000 mgr.x (mgr.24542) 4629 : cluster [DBG] 2021-07-31T03:35:16.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:16 smithi045 conmon[33655]: pgmap v5248: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:17.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:16 smithi154 conmon[34553]: cluster 2021-07-31T03:35:15.336252+0000 mgr.x (mgr.24542) 4629 : cluster 2021-07-31T03:35:17.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:16 smithi154 conmon[34553]: [DBG] pgmap v5248: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:18.774 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:18 smithi045 conmon[39718]: cluster 2021-07-31T03:35:17.336640+0000 mgr.x (mgr.24542 2021-07-31T03:35:18.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:18 smithi045 conmon[39718]: ) 4630 : cluster [DBG] pgmap v5249: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:18.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:18 smithi045 conmon[33655]: cluster 2021-07-31T03:35:17.336640+0000 mgr.x (mgr.24542) 4630 : cluster 2021-07-31T03:35:18.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:18 smithi045 conmon[33655]: [DBG] pgmap v5249: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:19.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:18 smithi154 conmon[34553]: cluster 2021-07-31T03:35:17.336640+0000 mgr.x (mgr.24542) 4630 : cluster [DBG] 2021-07-31T03:35:19.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:18 smithi154 conmon[34553]: pgmap v5249: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:21.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:20 smithi154 conmon[34553]: cluster 2021-07-31T03:35:19.337243+0000 mgr.x (mgr.24542) 4631 : cluster [DBG] 2021-07-31T03:35:21.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:20 smithi154 conmon[34553]: pgmap v5250: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:21.024 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:20 smithi045 conmon[39718]: cluster 2021-07-31T03:35:19.337243+0000 mgr.x (mgr.24542) 4631 2021-07-31T03:35:21.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:20 smithi045 conmon[39718]: : cluster [DBG] pgmap v5250: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:21.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:20 smithi045 conmon[33655]: cluster 2021-07-31T03:35:19.337243+0000 mgr.x (mgr.24542) 4631 : cluster [DBG] pgmap v5250: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:23.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:22 smithi154 conmon[34553]: cluster 2021-07-31T03:35:21.337602+0000 mgr.x (mgr.24542) 4632 : cluster 2021-07-31T03:35:23.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:22 smithi154 conmon[34553]: [DBG] pgmap v5251: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:23.024 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:22 smithi045 conmon[39718]: cluster 2021-07-31T03:35:21.337602+0000 mgr.x (mgr.24542 2021-07-31T03:35:23.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:22 smithi045 conmon[39718]: ) 4632 : cluster [DBG] pgmap v5251: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:23.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:22 smithi045 conmon[33655]: cluster 2021-07-31T03:35:21.337602+0000 mgr.x (mgr.24542) 4632 : cluster [DBG] pgmap v5251: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:25.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:24 smithi154 conmon[34553]: cluster 2021-07-31T03:35:23.338135+0000 mgr.x (mgr.24542) 4633 : cluster [DBG] pgmap v5252: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:25.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:24 smithi045 conmon[33655]: cluster 2021-07-31T03:35:23.338135+0000 mgr.x (mgr 2021-07-31T03:35:25.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:24 smithi045 conmon[33655]: .24542) 4633 : cluster [DBG] pgmap v5252: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:25.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:24 smithi045 conmon[39718]: cluster 2021-07-31T03:35:23.338135+0000 mgr.x (mgr 2021-07-31T03:35:25.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:24 smithi045 conmon[39718]: .24542) 4633 : cluster [DBG] pgmap v5252: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:27.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:26 smithi154 conmon[34553]: cluster 2021-07-31T03:35:25.338717+0000 mgr.x (mgr.24542) 4634 : cluster [DBG] pgmap v5253: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:27.024 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:26 smithi045 conmon[39718]: cluster 2021-07-31T03:35:25.338717+0000 mgr.x (mgr.24542 2021-07-31T03:35:27.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:26 smithi045 conmon[39718]: ) 4634 : cluster [DBG] pgmap v5253: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:27.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:26 smithi045 conmon[33655]: cluster 2021-07-31T03:35:25.338717+0000 mgr.x 2021-07-31T03:35:27.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:26 smithi045 conmon[33655]: (mgr.24542) 4634 : cluster [DBG] pgmap v5253: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:29.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:28 smithi154 conmon[34553]: cluster 2021-07-31T03:35:27.339104+0000 mgr.x (mgr.24542) 4635 : cluster [DBG] pgmap v5254: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:29.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:28 smithi154 conmon[34553]: audit 2021-07-31T03:35:28.363498+0000 mon.b (mon.2) 615 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:35:29.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:28 smithi154 conmon[34553]: audit 2021-07-31T03:35:28.363796+0000 mon.b (mon.2) 616 : audit [INF] 2021-07-31T03:35:29.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:28 smithi154 conmon[34553]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:35:29.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:28 smithi154 conmon[34553]: audit 2021 2021-07-31T03:35:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:28 smithi154 conmon[34553]: -07-31T03:35:28.364121+0000 mon.a (mon 2021-07-31T03:35:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:28 smithi154 conmon[34553]: .0) 3330 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:35:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:28 smithi154 conmon[34553]: audit 2021-07-31T03 2021-07-31T03:35:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:28 smithi154 conmon[34553]: :35:28.364543+0000 mon.a (mon.0) 3331 : audit 2021-07-31T03:35:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:28 smithi154 conmon[34553]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:35:29.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:28 smithi045 conmon[39718]: cluster 2021-07-31T03:35 2021-07-31T03:35:29.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:28 smithi045 conmon[39718]: :27.339104+0000 mgr.x (mgr.24542) 4635 : cluster [DBG] pgmap v5254: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:29.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:28 smithi045 conmon[39718]: audit 2021 2021-07-31T03:35:29.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:28 smithi045 conmon[39718]: -07-31T03:35:28. 2021-07-31T03:35:29.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:28 smithi045 conmon[39718]: 363498+0000 mon.b (mon.2 2021-07-31T03:35:29.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:28 smithi045 conmon[39718]: ) 615 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:35:29.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:28 smithi045 conmon[39718]: 2021-07-31T03:35:29.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:28 smithi045 conmon[39718]: audit 2021 2021-07-31T03:35:29.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:28 smithi045 conmon[39718]: -07-31T03:35: 2021-07-31T03:35:29.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:28 smithi045 conmon[39718]: 28.363796+0000 mon.b ( 2021-07-31T03:35:29.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:28 smithi045 conmon[39718]: mon.2) 616 : audit 2021-07-31T03:35:29.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:28 smithi045 conmon[39718]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:35:29.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:28 smithi045 conmon[39718]: audit 2021- 2021-07-31T03:35:29.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:28 smithi045 conmon[39718]: 07-31T03:35:28. 2021-07-31T03:35:29.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:28 smithi045 conmon[39718]: 364121+0000 mon.a (mon.0 2021-07-31T03:35:29.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:28 smithi045 conmon[39718]: ) 3330 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:35:29.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:28 smithi045 conmon[39718]: 2021-07-31T03:35:29.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:28 smithi045 conmon[39718]: audit 2021 2021-07-31T03:35:29.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:28 smithi045 conmon[39718]: -07-31T03:35:28 2021-07-31T03:35:29.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:28 smithi045 conmon[39718]: .364543+0000 mon.a (mon 2021-07-31T03:35:29.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:28 smithi045 conmon[39718]: .0) 3331 : audit [INF] 2021-07-31T03:35:29.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:28 smithi045 conmon[39718]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:35:29.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:28 smithi045 conmon[33655]: cluster 2021-07-31T03:35:27.339104+0000 mgr.x (mgr 2021-07-31T03:35:29.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:28 smithi045 conmon[33655]: .24542) 4635 : cluster [DBG] pgmap v5254: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:29.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:28 smithi045 conmon[33655]: audit 2021-07-31T03:35:28.363498+0000 mon.b (mon.2) 615 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:35:29.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:28 smithi045 conmon[33655]: audit 2021-07-31T03:35: 2021-07-31T03:35:29.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:28 smithi045 conmon[33655]: 28.363796+0000 mon.b (mon.2) 616 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:35:29.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:28 smithi045 conmon[33655]: audit 2021-07-31 2021-07-31T03:35:29.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:28 smithi045 conmon[33655]: T03:35:28.364121 2021-07-31T03:35:29.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:28 smithi045 conmon[33655]: +0000 mon.a (mon.0) 3330 : audit 2021-07-31T03:35:29.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:28 smithi045 conmon[33655]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:35:29.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:28 smithi045 conmon[33655]: audit 2021-07- 2021-07-31T03:35:29.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:28 smithi045 conmon[33655]: 31T03:35:28. 2021-07-31T03:35:29.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:28 smithi045 conmon[33655]: 364543+0000 mon.a (mon.0) 3331 : 2021-07-31T03:35:29.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:28 smithi045 conmon[33655]: audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:35:31.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:30 smithi154 conmon[34553]: cluster 2021-07-31T03:35:29.339505+0000 mgr.x (mgr.24542) 4636 : cluster [DBG] pgmap v5255: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:31.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:30 smithi045 conmon[33655]: cluster 2021-07-31T03:35:29.339505+0000 mgr.x (mgr.24542) 4636 : cluster [DBG] 2021-07-31T03:35:31.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:30 smithi045 conmon[33655]: pgmap v5255: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:31.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:30 smithi045 conmon[39718]: cluster 2021-07-31T03:35:29.339505+0000 mgr.x (mgr.24542) 4636 : 2021-07-31T03:35:31.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:30 smithi045 conmon[39718]: cluster [DBG] pgmap v5255: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:33.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:32 smithi154 conmon[34553]: cluster 2021-07-31T03:35:31.339963+0000 mgr.x (mgr.24542) 4637 : cluster [DBG] pgmap v5256: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:33.024 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:32 smithi045 conmon[33655]: cluster 2021-07-31T03:35:31.339963+0000 mgr.x (mgr.24542) 4637 : cluster [DBG] pgmap v5256: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:33.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:32 smithi045 conmon[39718]: cluster 2021-07-31T03:35:31.339963+0000 mgr.x 2021-07-31T03:35:33.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:32 smithi045 conmon[39718]: (mgr.24542) 4637 : cluster [DBG] pgmap v5256: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:34.502 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: debug 2021-07-31T03:35:34.041+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:35:34.502 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:35:34.502 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:35:34.503 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: yield 2021-07-31T03:35:34.503 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:35:34.503 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:35:34.503 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:35:34.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:35:34.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:35:34.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:35:34.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:35:34.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:35:34.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:35:34.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: task.result() 2021-07-31T03:35:34.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:35:34.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: debug 2021-07-31T03:35:34.053+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:35:34.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:35:34.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:35:34.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: yield 2021-07-31T03:35:34.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:35:34.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:35:34.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:35:34.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:35:34.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:35:34.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:35:34.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:35:34.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:35:34.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:35:34.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: task.result() 2021-07-31T03:35:34.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:35:34.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: debug 2021-07-31T03:35:34.078+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:35:34.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:35:34.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:35:34.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: yield 2021-07-31T03:35:34.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:35:34.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:35:34.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:35:34.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:35:34.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:35:34.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:35:34.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:35:34.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:35:34.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:35:34.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: task.result() 2021-07-31T03:35:34.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:35:34.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: debug 2021-07-31T03:35:34.079+0000 7fcb9115c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:35:34.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:35:34.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:35:34.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: yield 2021-07-31T03:35:34.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:35:34.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:35:34.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:35:34.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:35:34.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:35:34.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:35:34.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:35:34.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:35:34.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:35:34.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: task.result() 2021-07-31T03:35:34.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:35:34.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: 2021-07-31T03:35:34.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:35:34.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: 2021-07-31T03:35:34.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:35:34.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:35:34.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:35:34.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:35:34.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:35:34.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:35:34.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:35:34.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:35:34.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:35:34.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:35:34.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:35:34.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:35:34.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:35:34.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:35:34.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:35:34.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:35:34.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:35:34.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:35:34.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: raise 2021-07-31T03:35:34.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:35:34.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:35:34.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:35:34.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:35:34.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:35:34.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: debug 2021-07-31T03:35:34.087+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:35:34.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:35:34.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:35:34.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: yield 2021-07-31T03:35:34.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:35:34.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:35:34.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:35:34.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:35:34.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:35:34.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:35:34.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:35:34.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:35:34.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:35:34.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: task.result() 2021-07-31T03:35:34.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:35:34.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: debug 2021-07-31T03:35:34.087+0000 7fcb9095b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:35:34.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:35:34.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:35:34.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: yield 2021-07-31T03:35:34.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:35:34.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:35:34.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:35:34.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:35:34.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:35:34.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:35:34.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:35:34.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:35:34.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:35:34.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: task.result() 2021-07-31T03:35:34.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:35:34.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: 2021-07-31T03:35:34.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:35:34.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: 2021-07-31T03:35:34.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:35:34.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:35:34.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:35:34.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:35:34.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:35:34.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:35:34.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:35:34.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:35:34.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:35:34.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:35:34.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:35:34.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:35:34.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:35:34.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:35:34.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:35:34.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:35:34.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:35:34.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:35:34.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: raise 2021-07-31T03:35:34.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:35:34.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:35:34.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:35:34.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:35:34.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:35:35.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[34553]: cluster 2021-07-31T03:35:33.340427+0000 mgr.x (mgr.24542) 4638 : cluster [DBG] pgmap v5257: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:35.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[34553]: audit 2021-07-31T03:35:33.996274+0000 mon.b (mon.2) 617 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:35:35.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[34553]: audit 2021-07- 2021-07-31T03:35:35.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[34553]: 31T03:35:33.997768+0000 mon.b (mon.2) 618 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:35:35.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[34553]: audit 2021-07-31T03:35 2021-07-31T03:35:35.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[34553]: :33.998886+0000 mon.b (mon.2) 619 : audit [INF] 2021-07-31T03:35:35.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[34553]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:35:35.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[34553]: cephadm 2021-07-31T03: 2021-07-31T03:35:35.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[34553]: 35:33.999731+0000 mgr.x (mgr.24542) 2021-07-31T03:35:35.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[34553]: 4639 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:35:35.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[34553]: cephadm 2021-07 2021-07-31T03:35:35.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[34553]: -31T03:35:34.042630+0000 mgr.x 2021-07-31T03:35:35.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[34553]: (mgr.24542) 4640 : cephadm [ERR] 2021-07-31T03:35:35.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:35:35.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:35:35.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[34553]: yield 2021-07-31T03:35:35.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:35:35.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:35:35.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:35:35.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:35:35.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:35:35.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:35:35.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:35:35.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:35:35.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:35:35.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[34553]: task.result() 2021-07-31T03:35:35.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:35:35.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:34 smithi154 conmon[34553]: cephadm 2021-07-31T03:35:34.043097+0000 mgr.x (mgr.24542) 4641 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:35:35.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:34 smithi045 conmon[39718]: cluster 2021-07-31T03:35:33.340427+0000 mgr.x (mgr.24542) 4638 : 2021-07-31T03:35:35.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:34 smithi045 conmon[39718]: cluster [DBG] pgmap v5257: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:35.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:34 smithi045 conmon[39718]: audit 2021-07-31T03:35:33.996274+0000 mon.b (mon.2) 2021-07-31T03:35:35.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:34 smithi045 conmon[39718]: 617 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:35:35.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:34 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T03:35:35.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:34 smithi045 conmon[39718]: 31T03:35:33.997768+0000 mon.b (mon.2) 618 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:35:35.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:34 smithi045 conmon[39718]: 2021-07-31T03:35:35.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:34 smithi045 conmon[39718]: audit 2021-07 2021-07-31T03:35:35.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:34 smithi045 conmon[39718]: -31T03:35:33.998886+0000 mon.b (mon.2) 619 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:35:35.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:34 smithi045 conmon[39718]: cephadm 2021-07-31T03 2021-07-31T03:35:35.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:34 smithi045 conmon[39718]: :35:33.999731+0000 mgr.x (mgr.24542) 4639 : cephadm [WRN] 2021-07-31T03:35:35.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:34 smithi045 conmon[39718]: unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:35:35.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:34 smithi045 conmon[39718]: cephadm 2021-07-31T03:35:35.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:34 smithi045 conmon[39718]: 2021-07-31T03:35:34.042630+0000 mgr.x (mgr.24542) 4640 : cephadm 2021-07-31T03:35:35.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:34 smithi045 conmon[39718]: [ERR] 2021-07-31T03:35:35.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:34 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:35:35.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:35:35.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:34 smithi045 conmon[39718]: yield 2021-07-31T03:35:35.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:34 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:35:35.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:34 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:35:35.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:35:35.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:34 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:35:35.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:35:35.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:34 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:35:35.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:35:35.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:34 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:35:35.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:34 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:35:35.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:34 smithi045 conmon[39718]: task.result() 2021-07-31T03:35:35.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:34 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:35:35.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:34 smithi045 conmon[39718]: cephadm 2021-07-31T03:35:35.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:34 smithi045 conmon[39718]: 2021-07-31T03:35:34.043097+0000 mgr.x (mgr.24542) 4641 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:35:35.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:34 smithi045 conmon[33655]: cluster 2021-07-31T03:35:33.340427+0000 mgr.x (mgr.24542) 4638 : 2021-07-31T03:35:35.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:34 smithi045 conmon[33655]: cluster [DBG] pgmap v5257: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:35.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:34 smithi045 conmon[33655]: audit 2021-07-31T03:35:33.996274+0000 mon.b (mon.2) 617 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:35:35.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:34 smithi045 conmon[33655]: audit 2021-07-31T03:35:33.997768+0000 2021-07-31T03:35:35.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:34 smithi045 conmon[33655]: mon.b (mon.2) 618 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:35:35.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:34 smithi045 conmon[33655]: audit 2021-07-31T03 2021-07-31T03:35:35.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:34 smithi045 conmon[33655]: :35:33.998886+0000 mon.b (mon.2) 619 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:35:35.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:34 smithi045 conmon[33655]: 2021-07-31T03:35:35.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:34 smithi045 conmon[33655]: cephadm 2021-07-31T03 2021-07-31T03:35:35.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:34 smithi045 conmon[33655]: :35:33.999731+0000 mgr.x (mgr.24542) 4639 2021-07-31T03:35:35.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:34 smithi045 conmon[33655]: : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:35:35.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:34 smithi045 conmon[33655]: cephadm 2021-07-31T03:35:35.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:34 smithi045 conmon[33655]: 2021-07-31T03:35:34.042630+0000 mgr.x 2021-07-31T03:35:35.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:34 smithi045 conmon[33655]: (mgr.24542) 4640 : cephadm [ERR] 2021-07-31T03:35:35.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:34 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:35:35.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:35:35.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:34 smithi045 conmon[33655]: yield 2021-07-31T03:35:35.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:34 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:35:35.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:34 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:35:35.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:35:35.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:34 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:35:35.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:35:35.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:34 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:35:35.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:35:35.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:34 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:35:35.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:34 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:35:35.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:34 smithi045 conmon[33655]: task.result() 2021-07-31T03:35:35.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:34 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:35:35.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:34 smithi045 conmon[33655]: cephadm 2021-07-31T03:35:34.043097+0000 mgr.x (mgr.24542) 4641 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:35:36.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:35:34.054143+0000 mgr.x (mgr.24542) 4642 : cephadm [ERR] 2021-07-31T03:35:36.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:35:36.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:35:36.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: yield 2021-07-31T03:35:36.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:35:36.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:35:36.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:35:36.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:35:36.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:35:36.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:35:36.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:35:36.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:35:36.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:35:36.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: task.result() 2021-07-31T03:35:36.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:35:36.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:35:34.055167+0000 mgr.x (mgr.24542) 4643 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:35:36.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:35:34.079531+0000 mgr.x (mgr.24542) 4644 : cephadm [ERR] 2021-07-31T03:35:36.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:35:36.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:35:36.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: yield 2021-07-31T03:35:36.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:35:36.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:35:36.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:35:36.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:35:36.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:35:36.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:35:36.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:35:36.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:35:36.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:35:36.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: task.result() 2021-07-31T03:35:36.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:35:36.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:35:34.079952+0000 mgr.x (mgr.24542) 4645 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:35:36.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:35:36.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:35:36.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: yield 2021-07-31T03:35:36.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:35:36.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:35:36.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:35:36.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:35:36.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:35:36.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:35:36.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:35:36.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:35:36.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:35:36.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: task.result() 2021-07-31T03:35:36.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:35:36.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: 2021-07-31T03:35:36.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:35:36.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: 2021-07-31T03:35:36.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:35:36.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:35:36.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:35:36.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:35:36.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:35:36.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:35:36.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:35:36.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:35:36.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:35:36.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:35:36.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:35:36.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:35:36.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:35:36.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:35:36.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:35:36.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:35:36.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:35:36.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:35:36.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: raise 2021-07-31T03:35:36.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:35:36.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:35:36.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:35:36.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:35:36.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:35:36.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: cephadm 2021-07-31T 2021-07-31T03:35:36.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: 03:35:34.088294+0000 mgr.x (mgr.24542) 4646 : cephadm [ERR] 2021-07-31T03:35:36.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:35:36.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:35:36.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: yield 2021-07-31T03:35:36.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:35:36.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:35:36.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:35:36.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:35:36.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:35:36.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:35:36.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:35:36.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:35:36.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:35:36.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: task.result() 2021-07-31T03:35:36.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:35:36.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:35:34.088687+0000 mgr.x (mgr.24542) 4647 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:35:36.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:35:36.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:35:36.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: yield 2021-07-31T03:35:36.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:35:36.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:35:36.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:35:36.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:35:36.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:35:36.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:35:36.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:35:36.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:35:36.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:35:36.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: task.result() 2021-07-31T03:35:36.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:35:36.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: 2021-07-31T03:35:36.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:35:36.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: 2021-07-31T03:35:36.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:35:36.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:35:36.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:35:36.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:35:36.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:35:36.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:35:36.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:35:36.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:35:36.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:35:36.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:35:36.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:35:36.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:35:36.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:35:36.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:35:36.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:35:36.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:35:36.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:35:36.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:35:36.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: raise 2021-07-31T03:35:36.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:35:36.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:35:36.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:35:36.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:35:36.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:35 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:35:36.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:35:34.054143+0000 2021-07-31T03:35:36.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: mgr.x (mgr.24542) 4642 : cephadm [ERR] 2021-07-31T03:35:36.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:35:36.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:35:36.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: yield 2021-07-31T03:35:36.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:35:36.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:35:36.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:35:36.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:35:36.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:35:36.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:35:36.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:35:36.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:35:36.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:35:36.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: task.result() 2021-07-31T03:35:36.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:35:36.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:35:34.055167+0000 mgr.x (mgr.24542) 4643 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:35:36.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:35:34.079531+0000 mgr.x (mgr. 2021-07-31T03:35:36.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: 24542) 4644 : cephadm [ERR] 2021-07-31T03:35:36.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:35:36.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:35:36.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: yield 2021-07-31T03:35:36.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:35:36.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:35:36.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:35:36.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:35:36.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:35:36.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:35:36.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:35:36.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:35:36.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:35:36.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: task.result() 2021-07-31T03:35:36.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:35:36.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:35:34.079952+0000 mgr.x (mgr.24542) 4645 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:35:36.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:35:36.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:35:36.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: yield 2021-07-31T03:35:36.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:35:36.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:35:36.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:35:36.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:35:36.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:35:36.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:35:36.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:35:36.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:35:36.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:35:36.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: task.result() 2021-07-31T03:35:36.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:35:36.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: 2021-07-31T03:35:36.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:35:36.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: 2021-07-31T03:35:36.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:35:36.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:35:36.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:35:36.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:35:36.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:35:36.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:35:36.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:35:36.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:35:36.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:35:36.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:35:36.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:35:36.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:35:36.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:35:36.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:35:36.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:35:36.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:35:36.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:35:36.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:35:36.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: raise 2021-07-31T03:35:36.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:35:36.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:35:36.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:35:36.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:35:36.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:35:36.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:35:34.088294+0000 2021-07-31T03:35:36.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: mgr.x (mgr.24542) 4646 : cephadm [ERR] 2021-07-31T03:35:36.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:35:36.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:35:36.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: yield 2021-07-31T03:35:36.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:35:36.045 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:35:34.054143+0000 mgr.x (mgr.24542) 4642 : 2021-07-31T03:35:36.045 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: cephadm [ERR] 2021-07-31T03:35:36.045 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:35:36.046 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:35:36.046 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: yield 2021-07-31T03:35:36.046 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:35:36.046 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:35:36.046 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:35:36.047 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:35:36.047 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:35:36.047 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:35:36.047 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:35:36.047 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:35:36.048 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:35:36.048 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: task.result() 2021-07-31T03:35:36.048 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:35:36.048 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:35:34.055167+0000 mgr.x (mgr.24542) 4643 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:35:36.048 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:35:34.079531+0000 mgr.x (mgr.24542) 4644 : cephadm [ERR] 2021-07-31T03:35:36.049 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:35:36.049 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:35:36.049 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: yield 2021-07-31T03:35:36.049 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:35:36.049 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:35:36.050 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:35:36.050 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:35:36.050 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:35:36.050 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:35:36.050 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:35:36.051 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:35:36.051 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:35:36.051 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: task.result() 2021-07-31T03:35:36.051 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:35:36.051 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:35:34. 2021-07-31T03:35:36.052 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: 079952+0000 mgr.x (mgr.24542) 4645 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:35:36.052 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:35:36.052 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:35:36.052 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: yield 2021-07-31T03:35:36.052 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:35:36.053 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:35:36.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:35:36.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:35:36.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:35:36.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:35:36.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:35:36.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:35:36.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:35:36.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:35:36.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: task.result() 2021-07-31T03:35:36.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:35:36.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:35:34.088687+0000 mgr.x (mgr.24542) 4647 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:35:36.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:35:36.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:35:36.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: yield 2021-07-31T03:35:36.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:35:36.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:35:36.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:35:36.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:35:36.057 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:35:36.057 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:35:36.057 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:35:36.057 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:35:36.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:35:36.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: task.result() 2021-07-31T03:35:36.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:35:36.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: 2021-07-31T03:35:36.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:35:36.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: 2021-07-31T03:35:36.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:35:36.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:35:36.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:35:36.060 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:35:36.060 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:35:36.060 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:35:36.060 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:35:36.060 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:35:36.061 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:35:36.061 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:35:36.061 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:35:36.061 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:35:36.061 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:35:36.062 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:35:36.062 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:35:36.062 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:35:36.062 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:35:36.062 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:35:36.063 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: raise 2021-07-31T03:35:36.063 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:35:36.063 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:35:36.063 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:35:36.063 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:35:36.064 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:35:36.064 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:35:36.064 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:35:36.065 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:35:36.065 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:35:36.065 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:35:36.065 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:35:36.065 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:35:36.066 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: task.result() 2021-07-31T03:35:36.066 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:35:36.066 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: 2021-07-31T03:35:36.066 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:35:36.067 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: 2021-07-31T03:35:36.067 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:35:36.067 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:35:36.067 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:35:36.067 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:35:36.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:35:36.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:35:36.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:35:36.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:35:36.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:35:36.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:35:36.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:35:36.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:35:36.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:35:36.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:35:36.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:35:36.070 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:35:36.070 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:35:36.070 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:35:36.070 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: raise 2021-07-31T03:35:36.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:35:36.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:35:36.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:35:36.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:35:36.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:35:36.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:35:34.088294+0000 mgr.x (mgr.24542) 4646 : cephadm [ERR] 2021-07-31T03:35:36.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:35:36.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:35:36.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: yield 2021-07-31T03:35:36.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:35:36.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:35:36.073 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:35:36.073 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:35:36.073 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:35:36.073 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:35:36.073 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:35:36.074 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:35:36.074 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:35:36.074 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: task.result() 2021-07-31T03:35:36.074 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:35:36.074 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:35:34.088687+0000 mgr.x (mgr.24542) 4647 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:35:36.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:35:36.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:35:36.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: yield 2021-07-31T03:35:36.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:35:36.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:35:36.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:35:36.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:35:36.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:35:36.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:35:36.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:35:36.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:35:36.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:35:36.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: task.result() 2021-07-31T03:35:36.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:35:36.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: 2021-07-31T03:35:36.078 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:35:36.078 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: 2021-07-31T03:35:36.078 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:35:36.078 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:35:36.079 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:35:36.079 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:35:36.079 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:35:36.079 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:35:36.080 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:35:36.080 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:35:36.080 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:35:36.080 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:35:36.080 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:35:36.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:35:36.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:35:36.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:35:36.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:35:36.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:35:36.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:35:36.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:35:36.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: raise 2021-07-31T03:35:36.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:35:36.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:35:36.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:35:36.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:35:36.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:35 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:35:37.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:36 smithi154 conmon[34553]: cluster 2021-07-31T03:35:35.340911+0000 mgr.x (mgr.24542) 4648 : cluster [DBG] pgmap v5258: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:37.024 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:36 smithi045 conmon[33655]: cluster 2021-07-31T03:35:35.340911+0000 mgr.x (mgr.24542) 4648 : cluster 2021-07-31T03:35:37.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:36 smithi045 conmon[33655]: [DBG] pgmap v5258: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:37.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:36 smithi045 conmon[39718]: cluster 2021-07-31T03:35:35.340911+0000 mgr.x (mgr.24542) 4648 : cluster 2021-07-31T03:35:37.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:36 smithi045 conmon[39718]: [DBG] pgmap v5258: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:39.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:38 smithi154 conmon[34553]: cluster 2021-07-31T03:35:37.341345+0000 mgr.x (mgr.24542) 4649 : cluster [DBG] 2021-07-31T03:35:39.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:38 smithi154 conmon[34553]: pgmap v5259: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:39.024 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:38 smithi045 conmon[33655]: cluster 2021-07-31T03:35:37.341345+0000 mgr.x (mgr.24542) 2021-07-31T03:35:39.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:38 smithi045 conmon[33655]: 4649 : cluster [DBG] pgmap v5259: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:39.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:38 smithi045 conmon[39718]: cluster 2021-07-31T03:35: 2021-07-31T03:35:39.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:38 smithi045 conmon[39718]: 37.341345+0000 mgr.x (mgr.24542) 4649 : cluster [DBG] pgmap v5259: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:41.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:40 smithi154 conmon[34553]: cluster 2021-07-31T03:35:39.341834+0000 mgr.x (mgr.24542) 4650 : cluster [DBG] pgmap v5260: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:41.024 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:40 smithi045 conmon[39718]: cluster 2021-07-31T03:35:39.341834+0000 mgr.x (mgr.24542) 4650 : cluster [DBG] pgmap v5260: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:41.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:40 smithi045 conmon[33655]: cluster 2021-07-31T03:35:39.341834+0000 mgr.x (mgr 2021-07-31T03:35:41.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:40 smithi045 conmon[33655]: .24542) 4650 : cluster [DBG] pgmap v5260: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:42.483 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:35:42.760 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:35:42.760 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 110m ago 111m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:35:42.760 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 108m ago 111m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:35:42.761 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 108m ago 114m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:35:42.761 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 110m ago 116m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:35:42.761 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 110m ago 116m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:35:42.761 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 108m ago 114m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:35:42.761 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 110m ago 115m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:35:42.762 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 110m ago 111m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:35:42.762 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 108m ago 111m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:35:42.762 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 110m ago 114m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:35:42.762 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 110m ago 114m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:35:42.763 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 110m ago 113m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:35:42.763 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 110m ago 113m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:35:42.763 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 108m ago 113m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:35:42.763 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 108m ago 112m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:35:42.763 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 108m ago 112m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:35:42.764 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 108m ago 112m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:35:42.764 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 108m ago 111m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:35:43.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:42 smithi154 conmon[34553]: cluster 2021-07-31T03:35:41.342230+0000 mgr.x (mgr.24542) 4651 : cluster [DBG] pgmap v5261: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:43.024 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:42 smithi045 conmon[33655]: cluster 2021-07-31T03:35:41.342230+0000 mgr.x (mgr.24542) 4651 : cluster [DBG] pgmap v5261: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:43.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:42 smithi045 conmon[39718]: cluster 2021-07-31T03:35:41.342230 2021-07-31T03:35:43.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:42 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 4651 : cluster [DBG] pgmap v5261: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:43.095 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:35:43.095 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:35:43.095 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:35:43.095 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:35:43.096 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:35:43.096 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:35:43.096 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:35:43.096 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:35:43.096 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:35:43.097 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:35:43.097 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:35:43.097 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:35:43.097 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:35:43.097 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:35:43.098 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:35:43.098 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:35:43.098 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:35:44.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:43 smithi154 conmon[34553]: audit 2021-07-31T03:35:42.472318+0000 mgr.x (mgr.24542) 4652 : audit [DBG] from='client.29770 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:35:44.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:43 smithi154 conmon[34553]: audit 2021-07-31T03:35:42.753347+0000 mgr.x (mgr.24542) 4653 : audit [DBG] from='client.29776 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:35:44.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:43 smithi154 conmon[34553]: audit 2021-07-31T03:35:43.096168+0000 mon.c (mon.1) 215 : audit 2021-07-31T03:35:44.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:43 smithi154 conmon[34553]: [DBG] from='client.? 172.21.15.45:0/2320617114' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:35:44.024 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:43 smithi045 conmon[33655]: audit 2021-07-31T03:35:42.472318+0000 mgr.x (mgr.24542) 2021-07-31T03:35:44.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:43 smithi045 conmon[33655]: 4652 : audit [DBG] from='client.29770 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:35:44.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:43 smithi045 conmon[33655]: audit 2021-07-31T03:35:42.753347+0000 mgr.x (mgr.24542) 4653 : audit [DBG] from='client.29776 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:35:44.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:43 smithi045 conmon[33655]: audit 2021-07-31T03:35:43.096168+0000 mon.c ( 2021-07-31T03:35:44.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:43 smithi045 conmon[33655]: mon.1) 215 : audit [DBG] from='client.? 172.21.15.45:0/2320617114' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:35:44.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:43 smithi045 conmon[39718]: audit 2021-07-31T03:35: 2021-07-31T03:35:44.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:43 smithi045 conmon[39718]: 42.472318+0000 mgr.x (mgr.24542) 4652 : audit [DBG] from='client.29770 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:35:44.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:43 smithi045 conmon[39718]: audit 2021- 2021-07-31T03:35:44.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:43 smithi045 conmon[39718]: 07-31T03:35:42. 2021-07-31T03:35:44.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:43 smithi045 conmon[39718]: 753347+0000 mgr.x (mgr.24542 2021-07-31T03:35:44.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:43 smithi045 conmon[39718]: ) 4653 : audit [DBG] from='client.29776 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:35:44.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:43 smithi045 conmon[39718]: 2021-07-31T03:35:44.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:43 smithi045 conmon[39718]: audit 2021 2021-07-31T03:35:44.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:43 smithi045 conmon[39718]: -07-31T03:35: 2021-07-31T03:35:44.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:43 smithi045 conmon[39718]: 43.096168+0000 mon.c (mon 2021-07-31T03:35:44.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:43 smithi045 conmon[39718]: .1) 215 : audit [DBG] 2021-07-31T03:35:44.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:43 smithi045 conmon[39718]: from='client.? 172.21.15.45:0/2320617114' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:35:45.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:44 smithi154 conmon[34553]: cluster 2021-07-31T03:35:43.342599+0000 mgr.x (mgr.24542) 4654 : cluster [DBG] pgmap v5262: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:45.024 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:44 smithi045 conmon[39718]: cluster 2021-07-31T03:35:43.342599+0000 mgr.x ( 2021-07-31T03:35:45.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:44 smithi045 conmon[39718]: mgr.24542) 4654 : cluster [DBG] pgmap v5262: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:45.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:44 smithi045 conmon[33655]: cluster 2021-07-31T03:35:43.342599+0000 mgr.x (mgr.24542) 4654 : cluster [DBG] 2021-07-31T03:35:45.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:44 smithi045 conmon[33655]: pgmap v5262: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:47.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:46 smithi154 conmon[34553]: cluster 2021-07-31T03:35:45.343154+0000 mgr.x (mgr.24542) 4655 : cluster [DBG] pgmap v5263: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:47.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:46 smithi045 conmon[39718]: cluster 2021-07-31T03:35:45.343154+0000 mgr.x (mgr. 2021-07-31T03:35:47.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:46 smithi045 conmon[39718]: 24542) 4655 : cluster [DBG] pgmap v5263: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:47.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:46 smithi045 conmon[33655]: cluster 2021-07-31T03:35:45.343154+0000 mgr.x (mgr.24542) 4655 : 2021-07-31T03:35:47.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:46 smithi045 conmon[33655]: cluster [DBG] pgmap v5263: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:49.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:48 smithi154 conmon[34553]: cluster 2021-07-31T03:35:47.343674+0000 mgr.x (mgr.24542) 4656 : cluster [DBG] pgmap v5264: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:49.024 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:48 smithi045 conmon[33655]: cluster 2021-07-31T03:35:47.343674+0000 mgr.x (mgr.24542) 4656 : 2021-07-31T03:35:49.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:48 smithi045 conmon[33655]: cluster [DBG] pgmap v5264: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:49.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:48 smithi045 conmon[39718]: cluster 2021-07-31T03:35:47.343674+0000 mgr.x ( 2021-07-31T03:35:49.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:48 smithi045 conmon[39718]: mgr.24542) 4656 : cluster [DBG] pgmap v5264: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:51.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:50 smithi154 conmon[34553]: cluster 2021-07-31T03:35:49.344217+0000 mgr.x (mgr.24542) 4657 : cluster [DBG] pgmap v5265: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:51.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:50 smithi045 conmon[33655]: cluster 2021-07-31T03:35:49.344217+0000 mgr.x (mgr. 2021-07-31T03:35:51.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:50 smithi045 conmon[33655]: 24542) 4657 : cluster [DBG] pgmap v5265: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:51.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:50 smithi045 conmon[39718]: cluster 2021-07-31T03:35:49.344217+0000 mgr.x 2021-07-31T03:35:51.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:50 smithi045 conmon[39718]: (mgr.24542) 4657 : cluster [DBG] pgmap v5265: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:53.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:52 smithi154 conmon[34553]: cluster 2021-07-31T03:35:51.344616+0000 mgr.x (mgr.24542) 4658 : cluster [DBG] pgmap v5266: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:53.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:52 smithi045 conmon[39718]: cluster 2021-07-31T03:35:51.344616+0000 2021-07-31T03:35:53.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:52 smithi045 conmon[39718]: mgr.x (mgr.24542) 4658 : cluster [DBG] pgmap v5266: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:53.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:52 smithi045 conmon[33655]: cluster 2021-07-31T03:35:51.344616+0000 2021-07-31T03:35:53.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:52 smithi045 conmon[33655]: mgr.x (mgr.24542) 4658 : cluster [DBG] pgmap v5266: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:55.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:54 smithi154 conmon[34553]: cluster 2021-07-31T03:35:53.345117+0000 mgr.x (mgr.24542) 4659 : cluster [DBG] pgmap v5267: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:55.024 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:54 smithi045 conmon[33655]: cluster 2021-07-31T03 2021-07-31T03:35:55.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:54 smithi045 conmon[33655]: :35:53.345117+0000 mgr.x (mgr.24542) 4659 : cluster [DBG] pgmap v5267: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:55.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:54 smithi045 conmon[39718]: cluster 2021-07-31T03:35:53.345117+0000 2021-07-31T03:35:55.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:54 smithi045 conmon[39718]: mgr.x (mgr.24542) 4659 : cluster [DBG] pgmap v5267: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:57.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:56 smithi154 conmon[34553]: cluster 2021-07-31T03:35:55.345561+0000 mgr.x (mgr.24542) 4660 : cluster [DBG] pgmap v5268: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:57.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:56 smithi045 conmon[33655]: cluster 2021-07-31T03:35:55.345561+0000 mgr.x (mgr.24542) 4660 : cluster 2021-07-31T03:35:57.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:56 smithi045 conmon[33655]: [DBG] pgmap v5268: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:57.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:56 smithi045 conmon[39718]: cluster 2021-07-31T03:35:55.345561+0000 mgr.x (mgr 2021-07-31T03:35:57.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:56 smithi045 conmon[39718]: .24542) 4660 : cluster [DBG] pgmap v5268: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:59.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:35:58 smithi154 conmon[34553]: cluster 2021-07-31T03:35:57.345985+0000 mgr.x (mgr.24542) 4661 : cluster [DBG] pgmap v5269: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:59.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:58 smithi045 conmon[39718]: cluster 2021-07-31T03:35:57.345985+0000 mgr.x (mgr. 2021-07-31T03:35:59.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:35:58 smithi045 conmon[39718]: 24542) 4661 : cluster [DBG] pgmap v5269: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:35:59.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:35:58 smithi045 conmon[33655]: cluster 2021-07-31T03:35:57.345985+0000 mgr.x (mgr.24542) 4661 : cluster [DBG] pgmap v5269: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:01.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:00 smithi154 conmon[34553]: cluster 2021-07-31T03:35:59.346491+0000 mgr.x (mgr.24542) 4662 : cluster [DBG] pgmap v5270: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:01.024 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:00 smithi045 conmon[33655]: cluster 2021-07-31T03:35:59.346491+0000 mgr.x (mgr.24542) 4662 : cluster [DBG] pgmap v5270: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:01.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:00 smithi045 conmon[39718]: cluster 2021-07-31T03:35:59.346491+0000 mgr.x (mgr.24542) 4662 : cluster [DBG] pgmap v5270: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:03.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:02 smithi154 conmon[34553]: cluster 2021-07-31T03:36:01.347041+0000 mgr.x (mgr.24542) 4663 : cluster [DBG] pgmap v5271: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:03.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:02 smithi045 conmon[33655]: cluster 2021-07-31T03:36:01.347041+0000 mgr.x (mgr.24542) 4663 : cluster 2021-07-31T03:36:03.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:02 smithi045 conmon[33655]: [DBG] pgmap v5271: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:03.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:02 smithi045 conmon[39718]: cluster 2021-07-31T03:36:01.347041+0000 mgr.x 2021-07-31T03:36:03.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:02 smithi045 conmon[39718]: (mgr.24542) 4663 : cluster [DBG] pgmap v5271: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:05.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:04 smithi154 conmon[34553]: cluster 2021-07-31T03:36:03.347425+0000 mgr.x (mgr.24542) 4664 : cluster 2021-07-31T03:36:05.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:04 smithi154 conmon[34553]: [DBG] pgmap v5272: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:05.024 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:04 smithi045 conmon[39718]: cluster 2021-07-31T03:36:03.347425+0000 mgr.x (mgr.24542) 4664 : cluster [DBG] pgmap v5272: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:05.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:04 smithi045 conmon[33655]: cluster 2021-07-31T03:36:03.347425+0000 mgr.x (mgr.24542) 4664 : cluster 2021-07-31T03:36:05.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:04 smithi045 conmon[33655]: [DBG] pgmap v5272: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:07.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:06 smithi154 conmon[34553]: cluster 2021-07-31T03:36:05.347878+0000 mgr.x (mgr.24542) 4665 2021-07-31T03:36:07.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:06 smithi154 conmon[34553]: : cluster [DBG] pgmap v5273: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:07.024 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:06 smithi045 conmon[39718]: cluster 2021-07-31T03:36:05.347878+0000 2021-07-31T03:36:07.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:06 smithi045 conmon[39718]: mgr.x (mgr.24542) 4665 : cluster [DBG] pgmap v5273: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:07.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:06 smithi045 conmon[33655]: cluster 2021-07-31T03:36:05.347878+0000 mgr.x (mgr 2021-07-31T03:36:07.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:06 smithi045 conmon[33655]: .24542) 4665 : cluster [DBG] pgmap v5273: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:09.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:08 smithi154 conmon[34553]: cluster 2021-07-31T03:36:07.348290+0000 mgr.x (mgr.24542) 4666 : cluster 2021-07-31T03:36:09.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:08 smithi154 conmon[34553]: [DBG] pgmap v5274: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:09.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:08 smithi045 conmon[39718]: cluster 2021-07-31T03:36:07.348290+0000 mgr.x (mgr.24542) 4666 2021-07-31T03:36:09.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:08 smithi045 conmon[39718]: : cluster [DBG] pgmap v5274: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:09.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:08 smithi045 conmon[33655]: cluster 2021-07-31T03:36:07.348290+0000 mgr.x 2021-07-31T03:36:09.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:08 smithi045 conmon[33655]: (mgr.24542) 4666 : cluster [DBG] pgmap v5274: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:11.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:10 smithi154 conmon[34553]: cluster 2021-07-31T03:36:09.348823+0000 mgr.x ( 2021-07-31T03:36:11.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:10 smithi154 conmon[34553]: mgr.24542) 4667 : cluster [DBG] pgmap v5275: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:11.024 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:10 smithi045 conmon[39718]: cluster 2021-07-31T03:36:09.348823+0000 mgr.x (mgr.24542) 2021-07-31T03:36:11.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:10 smithi045 conmon[39718]: 4667 : cluster [DBG] pgmap v5275: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:11.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:10 smithi045 conmon[33655]: cluster 2021-07-31T03:36:09.348823+0000 mgr.x (mgr.24542) 4667 : cluster 2021-07-31T03:36:11.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:10 smithi045 conmon[33655]: [DBG] pgmap v5275: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:13.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:12 smithi154 conmon[34553]: cluster 2021-07-31T03:36:11.349224+0000 mgr.x (mgr.24542) 4668 : cluster [DBG] pgmap v5276: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:13.024 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:12 smithi045 conmon[39718]: cluster 2021-07-31T03:36:11.349224+0000 mgr.x (mgr.24542) 4668 : cluster [DBG] pgmap v5276: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:13.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:12 smithi045 conmon[33655]: cluster 2021-07-31T03:36:11.349224+0000 mgr.x (mgr.24542) 4668 : cluster [DBG] pgmap v5276: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:13.406 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:36:13.682 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:36:13.682 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 111m ago 111m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:36:13.682 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 108m ago 111m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:36:13.683 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 108m ago 115m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:36:13.683 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 111m ago 117m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:36:13.683 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 111m ago 117m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:36:13.683 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 108m ago 115m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:36:13.684 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 111m ago 115m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:36:13.684 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 111m ago 112m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:36:13.684 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 108m ago 112m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:36:13.684 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 111m ago 114m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:36:13.684 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 111m ago 114m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:36:13.685 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 111m ago 114m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:36:13.685 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 111m ago 113m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:36:13.685 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 108m ago 113m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:36:13.685 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 108m ago 113m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:36:13.685 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 108m ago 112m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:36:13.686 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 108m ago 112m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:36:13.686 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 108m ago 112m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:36:14.018 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:36:14.018 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:36:14.019 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:36:14.019 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:36:14.019 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:36:14.019 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:36:14.019 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:36:14.020 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:36:14.020 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:36:14.020 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:36:14.020 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:36:14.020 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:36:14.021 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:36:14.021 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:36:14.021 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:36:14.021 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:36:14.021 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:36:15.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:14 smithi154 conmon[34553]: cluster 2021-07-31T03:36:13.349663+0000 mgr.x (mgr.24542) 4669 : cluster 2021-07-31T03:36:15.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:14 smithi154 conmon[34553]: [DBG] pgmap v5277: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:15.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:14 smithi154 conmon[34553]: audit 2021-07-31T03:36:13.395628+0000 mgr.x (mgr.24542) 4670 : audit [DBG] from='client.19848 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:36:15.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:14 smithi154 conmon[34553]: audit 2021-07-31T03:36:15.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:14 smithi154 conmon[34553]: 2021-07-31T03:36:13.675228+0000 mgr.x (mgr.24542) 4671 : audit [DBG] from='client.19854 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:36:15.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:14 smithi154 conmon[34553]: audit 2021-07-31T03:36:15.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:14 smithi154 conmon[34553]: 2021-07-31T03:36: 2021-07-31T03:36:15.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:14 smithi154 conmon[34553]: 14.019540+0000 mon.a (mon.0) 3332 : 2021-07-31T03:36:15.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:14 smithi154 conmon[34553]: audit [DBG] from='client.? 172.21.15.45:0/3085964805' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:36:15.024 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:14 smithi045 conmon[39718]: cluster 2021-07-31T03:36:13.349663+0000 mgr.x (mgr. 2021-07-31T03:36:15.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:14 smithi045 conmon[39718]: 24542) 4669 : cluster [DBG] pgmap v5277: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:15.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:14 smithi045 conmon[39718]: audit 2021-07-31T03:36:13.395628+0000 2021-07-31T03:36:15.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:14 smithi045 conmon[39718]: mgr.x (mgr.24542) 4670 : audit [DBG] from='client.19848 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:36:15.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:14 smithi045 conmon[39718]: audit 2021-07-31T03:36:13.675228+0000 mgr.x (mgr 2021-07-31T03:36:15.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:14 smithi045 conmon[39718]: .24542) 4671 : audit [DBG] from='client.19854 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:36:15.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:14 smithi045 conmon[39718]: audit 2021-07-31T03:36:15.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:14 smithi045 conmon[39718]: 2021-07-31T03:36:14.019540+0000 mon.a ( 2021-07-31T03:36:15.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:14 smithi045 conmon[39718]: mon.0) 3332 : audit [DBG] from='client.? 172.21.15.45:0/3085964805' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:36:15.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:14 smithi045 conmon[33655]: cluster 2021-07-31T03:36:13. 2021-07-31T03:36:15.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:14 smithi045 conmon[33655]: 349663+0000 mgr.x (mgr.24542) 4669 : cluster [DBG] pgmap v5277: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:15.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:14 smithi045 conmon[33655]: audit 2021-07-31T03:36:15.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:14 smithi045 conmon[33655]: 2021-07-31T03:36:13.395628 2021-07-31T03:36:15.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:14 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 4670 : audit [DBG] 2021-07-31T03:36:15.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:14 smithi045 conmon[33655]: from='client.19848 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:36:15.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:14 smithi045 conmon[33655]: audit 2021 2021-07-31T03:36:15.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:14 smithi045 conmon[33655]: -07-31T03:36:13.675228+0000 mgr.x (mgr. 2021-07-31T03:36:15.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:14 smithi045 conmon[33655]: 24542) 4671 : audit [DBG] from='client.19854 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:36:15.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:14 smithi045 conmon[33655]: audit 2021-07-31T03:36:14.019540 2021-07-31T03:36:15.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:14 smithi045 conmon[33655]: +0000 mon.a (mon.0) 3332 : audit [DBG] from='client.? 172.21.15.45:0/3085964805' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:36:17.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:16 smithi154 conmon[34553]: cluster 2021-07-31T03:36:15.350269+0000 mgr.x (mgr.24542) 4672 : cluster [DBG] pgmap v5278: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:17.024 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:16 smithi045 conmon[39718]: cluster 2021-07-31T03:36:15.350269+0000 mgr.x (mgr.24542 2021-07-31T03:36:17.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:16 smithi045 conmon[39718]: ) 4672 : cluster [DBG] pgmap v5278: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:17.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:16 smithi045 conmon[33655]: cluster 2021-07-31T03:36:15.350269+0000 2021-07-31T03:36:17.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:16 smithi045 conmon[33655]: mgr.x (mgr.24542) 4672 : cluster [DBG] pgmap v5278: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:19.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:18 smithi154 conmon[34553]: cluster 2021-07-31T03:36:17.350728+0000 mgr.x (mgr.24542) 4673 : cluster 2021-07-31T03:36:19.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:18 smithi154 conmon[34553]: [DBG] pgmap v5279: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:19.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:18 smithi045 conmon[33655]: cluster 2021-07-31T03:36:17.350728+0000 mgr.x (mgr.24542) 4673 : cluster [DBG] pgmap v5279: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:19.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:18 smithi045 conmon[39718]: cluster 2021-07-31T03:36:17.350728+0000 mgr.x (mgr. 2021-07-31T03:36:19.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:18 smithi045 conmon[39718]: 24542) 4673 : cluster [DBG] pgmap v5279: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:21.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:20 smithi154 conmon[34553]: cluster 2021-07-31T03:36:19.351247+0000 mgr.x (mgr.24542) 4674 : cluster [DBG] 2021-07-31T03:36:21.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:20 smithi154 conmon[34553]: pgmap v5280: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:21.024 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:20 smithi045 conmon[33655]: cluster 2021-07-31T03:36:19.351247+0000 mgr.x (mgr.24542) 4674 : cluster [DBG] pgmap v5280: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:21.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:20 smithi045 conmon[39718]: cluster 2021-07-31T03:36:19.351247+0000 mgr.x 2021-07-31T03:36:21.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:20 smithi045 conmon[39718]: (mgr.24542) 4674 : cluster [DBG] pgmap v5280: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:23.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:22 smithi154 conmon[34553]: cluster 2021-07-31T03:36:21.351763+0000 mgr.x (mgr.24542) 2021-07-31T03:36:23.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:22 smithi154 conmon[34553]: 4675 : cluster [DBG] pgmap v5281: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:23.024 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:22 smithi045 conmon[39718]: cluster 2021-07-31T03:36:21.351763+0000 mgr.x (mgr.24542) 4675 : cluster 2021-07-31T03:36:23.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:22 smithi045 conmon[39718]: [DBG] pgmap v5281: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:23.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:22 smithi045 conmon[33655]: cluster 2021-07-31T03:36:21.351763 2021-07-31T03:36:23.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:22 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 4675 : cluster [DBG] pgmap v5281: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:25.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:24 smithi154 conmon[34553]: cluster 2021-07-31T03:36:23.352303+0000 mgr.x (mgr.24542) 4676 : cluster [DBG] pgmap v5282: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:25.024 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:24 smithi045 conmon[33655]: cluster 2021-07-31T03:36:23.352303+0000 mgr.x (mgr.24542) 2021-07-31T03:36:25.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:24 smithi045 conmon[33655]: 4676 : cluster [DBG] pgmap v5282: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:25.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:24 smithi045 conmon[39718]: cluster 2021-07-31T03:36:23.352303+0000 mgr.x 2021-07-31T03:36:25.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:24 smithi045 conmon[39718]: (mgr.24542) 4676 : cluster [DBG] pgmap v5282: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:27.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:26 smithi154 conmon[34553]: cluster 2021-07-31T03:36:25.352743+0000 mgr.x (mgr.24542) 4677 : cluster [DBG] 2021-07-31T03:36:27.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:26 smithi154 conmon[34553]: pgmap v5283: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:27.024 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:26 smithi045 conmon[33655]: cluster 2021-07-31T03:36:25.352743+0000 mgr.x (mgr.24542) 4677 : cluster 2021-07-31T03:36:27.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:26 smithi045 conmon[33655]: [DBG] pgmap v5283: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:27.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:26 smithi045 conmon[39718]: cluster 2021-07-31T03:36:25 2021-07-31T03:36:27.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:26 smithi045 conmon[39718]: .352743+0000 mgr.x (mgr.24542) 4677 : cluster [DBG] pgmap v5283: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:29.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:28 smithi154 conmon[34553]: cluster 2021-07-31T03:36:27.353098+0000 mgr.x (mgr.24542) 4678 : cluster [DBG] pgmap v5284: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:29.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:28 smithi154 conmon[34553]: audit 2021-07-31 2021-07-31T03:36:29.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:28 smithi154 conmon[34553]: T03:36:28.369658+0000 mon.b (mon.2) 620 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:36:29.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:28 smithi154 conmon[34553]: audit 2021-07-31T03:36:28.369887+0000 mon.b (mon.2) 621 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:36:29.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:28 smithi154 conmon[34553]: audit 2021-07-31T03:36:28.370094+0000 mon.a (mon.0) 3333 : audit 2021-07-31T03:36:29.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:28 smithi154 conmon[34553]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:36:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:28 smithi154 conmon[34553]: audit 2021-07-31T03:36:28.370565+0000 mon.a ( 2021-07-31T03:36:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:28 smithi154 conmon[34553]: mon.0) 3334 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:36:29.024 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:28 smithi045 conmon[39718]: cluster 2021-07-31T03:36:27.353098+0000 mgr.x (mgr 2021-07-31T03:36:29.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:28 smithi045 conmon[39718]: .24542) 4678 : cluster [DBG] pgmap v5284: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:29.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:28 smithi045 conmon[39718]: audit 2021-07-31T03:36:28.369658+0000 2021-07-31T03:36:29.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:28 smithi045 conmon[39718]: mon.b (mon.2) 620 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:36:29.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:28 smithi045 conmon[39718]: audit 2021-07-31T03:36 2021-07-31T03:36:29.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:28 smithi045 conmon[39718]: :28.369887+0000 mon.b (mon.2) 621 : 2021-07-31T03:36:29.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:28 smithi045 conmon[39718]: audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:36:29.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:28 smithi045 conmon[39718]: audit 2021-07-31T03:36:29.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:28 smithi045 conmon[39718]: 2021-07-31T03:36:28.370094+0000 mon.a 2021-07-31T03:36:29.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:28 smithi045 conmon[39718]: (mon.0) 3333 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:36:29.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:28 smithi045 conmon[39718]: audit 2021-07-31T03 2021-07-31T03:36:29.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:28 smithi045 conmon[39718]: :36:28.370565+0000 mon.a (mon.0) 3334 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:36:29.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:28 smithi045 conmon[33655]: cluster 2021-07-31T03:36:27.353098+0000 mgr.x ( 2021-07-31T03:36:29.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:28 smithi045 conmon[33655]: mgr.24542) 4678 : cluster [DBG] pgmap v5284: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:29.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:28 smithi045 conmon[33655]: audit 2021-07-31T03:36:28.369658+0000 mon.b (mon.2) 620 : audit [INF] 2021-07-31T03:36:29.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:28 smithi045 conmon[33655]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:36:29.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:28 smithi045 conmon[33655]: audit 2021-07-31T03:36:28. 2021-07-31T03:36:29.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:28 smithi045 conmon[33655]: 369887+0000 mon.b (mon.2) 621 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:36:29.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:28 smithi045 conmon[33655]: audit 2021-07-31T03 2021-07-31T03:36:29.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:28 smithi045 conmon[33655]: :36:28.370094+0000 mon.a (mon.0) 3333 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:36:29.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:28 smithi045 conmon[33655]: audit 2021-07-31T03:36:29.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:28 smithi045 conmon[33655]: 2021-07-31T03:36:28.370565+0000 mon.a (mon.0) 3334 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:36:31.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:30 smithi154 conmon[34553]: cluster 2021-07-31T03:36:29.353630+0000 mgr.x (mgr.24542) 4679 : cluster [DBG] pgmap v5285: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:31.024 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:30 smithi045 conmon[39718]: cluster 2021-07-31T03:36:29.353630+0000 mgr.x 2021-07-31T03:36:31.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:30 smithi045 conmon[39718]: (mgr.24542) 4679 : cluster [DBG] pgmap v5285: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:31.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:30 smithi045 conmon[33655]: cluster 2021-07-31T03:36:29.353630+0000 mgr.x (mgr 2021-07-31T03:36:31.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:30 smithi045 conmon[33655]: .24542) 4679 : cluster [DBG] pgmap v5285: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:33.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:32 smithi154 conmon[34553]: cluster 2021-07-31T03:36:31.354133+0000 mgr.x (mgr.24542) 4680 : cluster [DBG] pgmap v5286: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:33.024 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:32 smithi045 conmon[39718]: cluster 2021-07-31T03:36:31.354133+0000 mgr.x (mgr. 2021-07-31T03:36:33.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:32 smithi045 conmon[39718]: 24542) 4680 : cluster [DBG] pgmap v5286: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:33.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:32 smithi045 conmon[33655]: cluster 2021-07-31T03:36:31.354133+0000 mgr.x (mgr.24542) 4680 : 2021-07-31T03:36:33.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:32 smithi045 conmon[33655]: cluster [DBG] pgmap v5286: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:34.502 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: debug 2021-07-31T03:36:34.135+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:36:34.502 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:36:34.503 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:36:34.503 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: yield 2021-07-31T03:36:34.503 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:36:34.503 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:36:34.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:36:34.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:36:34.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:36:34.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:36:34.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:36:34.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:36:34.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:36:34.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: task.result() 2021-07-31T03:36:34.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:36:34.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: debug 2021-07-31T03:36:34.147+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:36:34.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:36:34.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:36:34.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: yield 2021-07-31T03:36:34.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:36:34.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:36:34.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:36:34.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:36:34.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:36:34.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:36:34.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:36:34.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:36:34.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:36:34.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: task.result() 2021-07-31T03:36:34.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:36:34.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: debug 2021-07-31T03:36:34.172+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:36:34.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:36:34.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:36:34.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: yield 2021-07-31T03:36:34.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:36:34.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:36:34.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:36:34.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:36:34.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:36:34.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:36:34.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:36:34.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:36:34.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:36:34.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: task.result() 2021-07-31T03:36:34.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:36:34.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: debug 2021-07-31T03:36:34.172+0000 7fcb9816a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:36:34.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:36:34.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:36:34.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: yield 2021-07-31T03:36:34.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:36:34.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:36:34.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:36:34.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:36:34.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:36:34.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:36:34.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:36:34.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:36:34.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:36:34.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: task.result() 2021-07-31T03:36:34.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:36:34.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: 2021-07-31T03:36:34.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:36:34.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: 2021-07-31T03:36:34.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:36:34.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:36:34.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:36:34.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:36:34.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:36:34.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:36:34.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:36:34.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:36:34.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:36:34.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:36:34.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:36:34.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:36:34.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:36:34.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:36:34.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:36:34.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:36:34.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:36:34.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:36:34.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: raise 2021-07-31T03:36:34.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:36:34.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:36:34.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:36:34.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:36:34.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:36:34.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: debug 2021-07-31T03:36:34.181+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:36:34.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:36:34.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:36:34.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: yield 2021-07-31T03:36:34.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:36:34.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:36:34.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:36:34.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:36:34.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:36:34.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:36:34.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:36:34.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:36:34.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:36:34.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: task.result() 2021-07-31T03:36:34.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:36:34.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: debug 2021-07-31T03:36:34.182+0000 7fcb97969700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:36:34.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:36:34.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:36:34.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: yield 2021-07-31T03:36:34.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:36:34.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:36:34.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:36:34.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:36:34.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:36:34.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:36:34.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:36:34.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:36:34.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:36:34.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: task.result() 2021-07-31T03:36:34.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:36:34.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: 2021-07-31T03:36:34.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:36:34.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: 2021-07-31T03:36:34.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:36:34.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:36:34.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:36:34.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:36:34.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:36:34.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:36:34.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:36:34.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:36:34.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:36:34.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:36:34.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:36:34.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:36:34.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:36:34.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:36:34.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:36:34.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:36:34.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:36:34.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:36:34.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: raise 2021-07-31T03:36:34.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:36:34.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:36:34.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:36:34.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:36:34.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:36:35.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[34553]: cluster 2021-07-31T03:36:33.354649+0000 mgr.x (mgr.24542) 4681 : cluster [DBG] pgmap v5287: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:35.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[34553]: audit 2021-07-31T03:36:34.089499+0000 mon.b (mon.2) 622 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:36:35.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[34553]: audit 2021-07-31T03 2021-07-31T03:36:35.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[34553]: :36:34.091004+0000 mon.b (mon.2) 623 : audit [DBG] 2021-07-31T03:36:35.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[34553]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:36:35.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[34553]: audit 2021-07 2021-07-31T03:36:35.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[34553]: -31T03:36:34.092068+0000 mon.b 2021-07-31T03:36:35.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[34553]: (mon.2) 624 : audit [INF] 2021-07-31T03:36:35.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:34 smithi154 conmon[34553]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:36:35.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:34 smithi045 conmon[33655]: cluster 2021-07-31T03:36:33.354649+0000 mgr.x (mgr.24542) 4681 2021-07-31T03:36:35.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:34 smithi045 conmon[33655]: : cluster [DBG] pgmap v5287: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:35.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:34 smithi045 conmon[33655]: audit 2021-07-31T03:36:34.089499+0000 mon.b (mon.2) 622 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:36:35.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:34 smithi045 conmon[33655]: audit 2021-07-31T03:36:34.091004+0000 mon.b (mon.2) 623 : audit [DBG] 2021-07-31T03:36:35.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:34 smithi045 conmon[33655]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:36:35.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:34 smithi045 conmon[33655]: audit 2021-07-31T03:36:34.092068+0000 mon.b (mon.2) 624 : audit 2021-07-31T03:36:35.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:34 smithi045 conmon[33655]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:36:35.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:34 smithi045 conmon[39718]: cluster 2021-07-31T03:36:33.354649+0000 mgr.x (mgr.24542) 4681 : 2021-07-31T03:36:35.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:34 smithi045 conmon[39718]: cluster [DBG] pgmap v5287: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:35.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:34 smithi045 conmon[39718]: audit 2021-07-31T03: 2021-07-31T03:36:35.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:34 smithi045 conmon[39718]: 36:34.089499+0000 mon.b (mon.2) 622 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:36:35.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:34 smithi045 conmon[39718]: audit 2021-07-31T03:36:35.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:34 smithi045 conmon[39718]: 2021-07-31T03:36:34.091004+0000 mon.b (mon. 2021-07-31T03:36:35.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:34 smithi045 conmon[39718]: 2) 623 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:36:35.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:34 smithi045 conmon[39718]: audit 2021-07-31T03:36:35.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:34 smithi045 conmon[39718]: 2021-07-31T03:36:34.092068+0000 mon.b (mon.2) 2021-07-31T03:36:35.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:34 smithi045 conmon[39718]: 624 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:36:36.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:36:34.092822+0000 mgr.x (mgr.24542) 4682 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:36:36.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:36:34.136649+0000 mgr.x (mgr.24542) 4683 : cephadm [ERR] 2021-07-31T03:36:36.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:36:36.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:36:36.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: yield 2021-07-31T03:36:36.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:36:36.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:36:36.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:36:36.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:36:36.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:36:36.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:36:36.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:36:36.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:36:36.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:36:36.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: task.result() 2021-07-31T03:36:36.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:36:36.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:36:34.137204+0000 mgr.x (mgr.24542) 4684 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:36:36.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:36:34.148111+0000 mgr.x (mgr.24542) 4685 : cephadm [ERR] 2021-07-31T03:36:36.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:36:36.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:36:36.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: yield 2021-07-31T03:36:36.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:36:36.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:36:36.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:36:36.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:36:36.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:36:36.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:36:36.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:36:36.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:36:36.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:36:36.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: task.result() 2021-07-31T03:36:36.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:36:36.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: cephadm 2021- 2021-07-31T03:36:36.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: 07-31T03:36:34.149031+0000 mgr.x (mgr.24542) 4686 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:36:36.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:36:34. 2021-07-31T03:36:36.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: 173336+0000 mgr.x (mgr.24542) 4687 : cephadm [ERR] 2021-07-31T03:36:36.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:36:36.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:36:36.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: yield 2021-07-31T03:36:36.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:36:36.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:36:36.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:36:36.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:36:36.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:36:36.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:36:36.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:36:36.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:36:36.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:36:36.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: task.result() 2021-07-31T03:36:36.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:36:36.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:36:34.173768+0000 mgr.x (mgr.24542) 4688 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:36:36.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:36:36.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:36:36.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: yield 2021-07-31T03:36:36.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:36:36.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:36:36.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:36:36.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:36:36.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:36:36.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:36:36.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:36:36.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:36:36.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:36:36.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: task.result() 2021-07-31T03:36:36.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:36:36.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: 2021-07-31T03:36:36.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:36:36.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: 2021-07-31T03:36:36.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:36:36.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:36:36.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:36:36.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:36:36.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:36:36.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:36:36.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:36:36.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:36:36.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:36:36.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:36:36.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:36:36.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:36:36.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:36:36.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:36:36.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:36:36.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:36:36.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:36:36.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:36:36.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: raise 2021-07-31T03:36:36.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:36:36.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:36:36.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:36:36.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:36:36.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:36:36.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: 2021-07-31T03:36:36.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:36:34.182638+0000 mgr.x (mgr.24542) 4689 : cephadm [ERR] 2021-07-31T03:36:36.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:36:36.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:36:36.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: yield 2021-07-31T03:36:36.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:36:36.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:36:36.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:36:36.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:36:36.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:36:36.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:36:36.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:36:36.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:36:36.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:36:36.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: task.result() 2021-07-31T03:36:36.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:36:36.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:36:34.183184+0000 mgr.x (mgr.24542) 4690 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:36:36.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:36:36.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:36:36.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: yield 2021-07-31T03:36:36.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:36:36.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:36:36.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:36:36.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:36:36.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:36:36.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:36:36.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:36:36.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:36:36.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:36:36.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: task.result() 2021-07-31T03:36:36.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:36:36.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: 2021-07-31T03:36:36.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:36:36.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: 2021-07-31T03:36:36.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:36:36.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:36:36.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:36:36.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:36:36.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:36:36.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:36:36.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:36:36.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:36:36.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:36:36.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:36:36.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:36:36.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:36:36.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:36:36.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:36:36.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:36:36.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:36:36.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:36:36.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:36:36.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: raise 2021-07-31T03:36:36.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:36:36.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:36:36.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:36:36.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:36:36.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:35 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:36:36.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:36:34 2021-07-31T03:36:36.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: .092822+0000 mgr.x (mgr.24542) 4682 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:36:36.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: cephadm 2021-07-31T03: 2021-07-31T03:36:36.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: 36:34.136649+0000 mgr.x (mgr.24542) 4683 : cephadm [ERR] 2021-07-31T03:36:36.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:36:36.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:36:36.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: yield 2021-07-31T03:36:36.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:36:36.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:36:36.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:36:36.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:36:36.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:36:36.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:36:36.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:36:36.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:36:36.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:36:36.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: task.result() 2021-07-31T03:36:36.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:36:36.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:36:34.137204+0000 mgr.x (mgr.24542) 4684 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:36:36.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:36:34.148111+0000 mgr.x (mgr.24542) 4685 : cephadm [ERR] 2021-07-31T03:36:36.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:36:36.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:36:36.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: yield 2021-07-31T03:36:36.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:36:36.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:36:36.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:36:36.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:36:36.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:36:36.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:36:36.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:36:36.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:36:36.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:36:36.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: task.result() 2021-07-31T03:36:36.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:36:36.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:36:34.149031+0000 mgr.x (mgr.24542) 4686 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:36:36.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:36:34.173336+0000 2021-07-31T03:36:36.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: mgr.x (mgr.24542) 4687 : cephadm [ERR] 2021-07-31T03:36:36.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:36:36.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:36:36.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: yield 2021-07-31T03:36:36.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:36:36.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:36:36.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:36:36.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:36:36.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:36:36.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:36:36.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:36:36.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:36:36.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:36:36.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: task.result() 2021-07-31T03:36:36.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:36:36.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:36:34.173768+0000 mgr.x (mgr.24542) 4688 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:36:36.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:36:36.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:36:36.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: yield 2021-07-31T03:36:36.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:36:36.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:36:36.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:36:36.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:36:36.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:36:36.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:36:36.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:36:36.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:36:36.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:36:36.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: task.result() 2021-07-31T03:36:36.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:36:36.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: 2021-07-31T03:36:36.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:36:36.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: 2021-07-31T03:36:36.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:36:36.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:36:36.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:36:36.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:36:36.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:36:36.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:36:36.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:36:36.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:36:36.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:36:36.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:36:36.050 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:36:34.092822+0000 mgr.x (mgr. 2021-07-31T03:36:36.050 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: 24542) 4682 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:36:36.050 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:36:34.136649+0000 mgr.x (mgr.24542) 4683 : cephadm [ERR] 2021-07-31T03:36:36.051 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:36:36.051 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:36:36.051 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: yield 2021-07-31T03:36:36.051 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:36:36.051 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:36:36.052 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:36:36.052 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:36:36.052 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:36:36.052 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:36:36.052 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:36:36.053 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:36:36.053 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:36:36.053 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: task.result() 2021-07-31T03:36:36.053 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:36:36.053 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:36:34.137204+0000 mgr.x (mgr.24542) 4684 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:36:36.054 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:36:34.148111+0000 mgr.x (mgr.24542) 4685 : cephadm [ERR] 2021-07-31T03:36:36.054 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:36:36.054 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:36:36.054 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: yield 2021-07-31T03:36:36.054 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:36:36.055 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:36:36.055 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:36:36.055 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:36:36.055 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:36:36.055 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:36:36.056 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:36:36.056 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:36:36.056 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:36:36.056 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: task.result() 2021-07-31T03:36:36.056 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:36:36.057 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:36:34. 2021-07-31T03:36:36.057 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: 149031+0000 mgr.x (mgr.24542) 4686 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:36:36.057 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:36:34.173336+0000 mgr.x (mgr.24542) 4687 : cephadm [ERR] 2021-07-31T03:36:36.057 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:36:36.057 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:36:36.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:36:36.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:36:36.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:36:36.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:36:36.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:36:36.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:36:36.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:36:36.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:36:36.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: raise 2021-07-31T03:36:36.060 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:36:36.060 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:36:36.060 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:36:36.060 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:36:36.060 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:36:36.061 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:36:34.182638+0000 mgr.x (mgr.24542) 4689 : cephadm [ERR] 2021-07-31T03:36:36.061 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:36:36.061 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:36:36.061 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: yield 2021-07-31T03:36:36.061 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:36:36.062 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:36:36.062 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:36:36.062 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:36:36.062 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:36:36.062 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:36:36.063 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:36:36.063 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:36:36.063 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:36:36.063 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: task.result() 2021-07-31T03:36:36.064 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:36:36.064 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:36:34.183184+0000 mgr.x (mgr.24542) 4690 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:36:36.064 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:36:36.064 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:36:36.064 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: yield 2021-07-31T03:36:36.065 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:36:36.065 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:36:36.065 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:36:36.065 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:36:36.065 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:36:36.066 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:36:36.066 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: yield 2021-07-31T03:36:36.066 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:36:36.067 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:36:36.067 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:36:36.067 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:36:36.067 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:36:36.067 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:36:36.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:36:36.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:36:36.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:36:36.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: task.result() 2021-07-31T03:36:36.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:36:36.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: cephadm 2021- 2021-07-31T03:36:36.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: 07-31T03:36:34.173768+0000 mgr.x (mgr.24542) 4688 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:36:36.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:36:36.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:36:36.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: yield 2021-07-31T03:36:36.070 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:36:36.070 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:36:36.070 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:36:36.070 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:36:36.070 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:36:36.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:36:36.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:36:36.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:36:36.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:36:36.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: task.result() 2021-07-31T03:36:36.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:36:36.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: 2021-07-31T03:36:36.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:36:36.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: 2021-07-31T03:36:36.073 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:36:36.073 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:36:36.073 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:36:36.073 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:36:36.073 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:36:36.074 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:36:36.074 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:36:36.074 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:36:36.074 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:36:36.074 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:36:36.075 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:36:36.075 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:36:36.075 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:36:36.075 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: task.result() 2021-07-31T03:36:36.076 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:36:36.076 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: 2021-07-31T03:36:36.076 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:36:36.076 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: 2021-07-31T03:36:36.076 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:36:36.077 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:36:36.077 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:36:36.077 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:36:36.077 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:36:36.077 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:36:36.078 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:36:36.078 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:36:36.078 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:36:36.078 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:36:36.079 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:36:36.079 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:36:36.079 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:36:36.079 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:36:36.079 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:36:36.080 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:36:36.080 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:36:36.080 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:36:36.080 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: raise 2021-07-31T03:36:36.080 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:36:36.081 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:36:36.081 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:36:36.081 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:36:36.081 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:36:36.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:36:36.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:36:36.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:36:36.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:36:36.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:36:36.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:36:36.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:36:36.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:36:36.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: raise 2021-07-31T03:36:36.084 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:36:36.084 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:36:36.084 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:36:36.084 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:36:36.084 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:36:36.085 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:36:34.182638+0000 mgr.x (mgr.24542) 4689 : cephadm [ERR] 2021-07-31T03:36:36.085 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:36:36.085 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:36:36.085 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: yield 2021-07-31T03:36:36.085 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:36:36.086 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:36:36.086 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:36:36.086 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:36:36.086 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:36:36.086 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:36:36.087 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:36:36.087 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:36:36.087 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:36:36.087 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: task.result() 2021-07-31T03:36:36.087 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:36:36.088 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: 2021-07-31T03:36:36.088 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:36:34.183184+0000 mgr.x (mgr.24542) 4690 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:36:36.088 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:36:36.088 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:36:36.088 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: yield 2021-07-31T03:36:36.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:36:36.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:36:36.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:36:36.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:36:36.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:36:36.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:36:36.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:36:36.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:36:36.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:36:36.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: task.result() 2021-07-31T03:36:36.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:36:36.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: 2021-07-31T03:36:36.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:36:36.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: 2021-07-31T03:36:36.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:36:36.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:36:36.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:36:36.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:36:36.093 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:36:36.093 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:36:36.093 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:36:36.093 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:36:36.093 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:36:36.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:36:36.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:36:36.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:36:36.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:36:36.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:36:36.095 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:36:36.095 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:36:36.095 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:36:36.095 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:36:36.096 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: raise 2021-07-31T03:36:36.096 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:36:36.096 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:36:36.096 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:36:36.096 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:36:36.097 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:35 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:36:37.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:36 smithi154 conmon[34553]: cluster 2021-07-31T03:36:35.355185+0000 mgr.x (mgr.24542) 4691 : cluster [DBG] 2021-07-31T03:36:37.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:36 smithi154 conmon[34553]: pgmap v5288: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:37.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:36 smithi045 conmon[33655]: cluster 2021-07-31T03:36:35.355185+0000 mgr.x (mgr.24542) 4691 : cluster [DBG] pgmap v5288: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:37.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:36 smithi045 conmon[33655]: 2021-07-31T03:36:37.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:36 smithi045 conmon[39718]: cluster 2021-07-31T03:36:35.355185+0000 mgr.x (mgr.24542) 4691 : cluster [DBG] pgmap v5288: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:39.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:38 smithi154 conmon[34553]: cluster 2021-07-31T03:36:37.355756+0000 mgr.x (mgr.24542) 4692 : cluster [DBG] 2021-07-31T03:36:39.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:38 smithi154 conmon[34553]: pgmap v5289: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:39.024 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:38 smithi045 conmon[33655]: cluster 2021-07-31T03:36:37.355756+0000 mgr.x (mgr.24542) 4692 2021-07-31T03:36:39.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:38 smithi045 conmon[33655]: : cluster [DBG] pgmap v5289: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:39.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:38 smithi045 conmon[39718]: cluster 2021-07-31T03:36:37.355756+0000 2021-07-31T03:36:39.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:38 smithi045 conmon[39718]: mgr.x (mgr.24542) 4692 : cluster [DBG] pgmap v5289: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:41.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:40 smithi154 conmon[34553]: cluster 2021-07-31T03:36:39.356253+0000 mgr.x (mgr.24542) 4693 : cluster [DBG] pgmap v5290: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:41.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:40 smithi045 conmon[39718]: cluster 2021-07-31T03:36:39.356253+0000 2021-07-31T03:36:41.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:40 smithi045 conmon[39718]: mgr.x (mgr.24542) 4693 : cluster [DBG] pgmap v5290: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:41.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:40 smithi045 conmon[33655]: cluster 2021-07-31T03:36:39.356253+0000 mgr.x ( 2021-07-31T03:36:41.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:40 smithi045 conmon[33655]: mgr.24542) 4693 : cluster [DBG] pgmap v5290: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:43.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:42 smithi154 conmon[34553]: cluster 2021-07-31T03:36:41.356790+0000 mgr.x (mgr.24542) 4694 : cluster [DBG] pgmap v5291: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:43.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:42 smithi045 conmon[39718]: cluster 2021-07-31T03:36:41.356790+0000 mgr.x (mgr.24542) 4694 : cluster [DBG] pgmap v5291: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:43.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:42 smithi045 conmon[33655]: cluster 2021-07-31T03:36:41.356790+0000 mgr.x (mgr.24542) 4694 : cluster 2021-07-31T03:36:43.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:42 smithi045 conmon[33655]: [DBG] pgmap v5291: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:44.330 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:36:44.608 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:36:44.608 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 111m ago 112m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:36:44.608 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 109m ago 112m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:36:44.609 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 109m ago 115m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:36:44.609 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 111m ago 117m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:36:44.609 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 111m ago 117m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:36:44.610 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 109m ago 115m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:36:44.610 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 111m ago 116m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:36:44.611 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 111m ago 112m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:36:44.611 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 109m ago 112m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:36:44.611 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 111m ago 115m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:36:44.612 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 111m ago 115m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:36:44.612 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 111m ago 114m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:36:44.612 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 111m ago 114m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:36:44.613 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 109m ago 114m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:36:44.613 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 109m ago 113m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:36:44.613 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 109m ago 113m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:36:44.614 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 109m ago 113m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:36:44.614 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 109m ago 112m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:36:44.943 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:36:44.943 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:36:44.944 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:36:44.944 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:36:44.944 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:36:44.945 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:36:44.945 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:36:44.945 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:36:44.946 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:36:44.946 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:36:44.947 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:36:44.947 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:36:44.947 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:36:44.948 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:36:44.948 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:36:44.948 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:36:44.949 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:36:45.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:44 smithi154 conmon[34553]: cluster 2021-07-31T03:36:43.357273+0000 mgr.x (mgr.24542) 4695 : cluster [DBG] pgmap v5292: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:45.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:44 smithi045 conmon[33655]: cluster 2021-07-31T03:36:43.357273+0000 mgr.x (mgr. 2021-07-31T03:36:45.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:44 smithi045 conmon[33655]: 24542) 4695 : cluster [DBG] pgmap v5292: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:45.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:44 smithi045 conmon[39718]: cluster 2021-07-31T03:36:43.357273+0000 2021-07-31T03:36:45.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:44 smithi045 conmon[39718]: mgr.x (mgr.24542) 4695 : cluster [DBG] pgmap v5292: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:46.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:45 smithi154 conmon[34553]: audit 2021-07-31T03:36:44.319511+0000 mgr.x (mgr.24542) 4696 : audit [DBG] from='client.29806 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:36:46.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:45 smithi154 conmon[34553]: 2021-07-31T03:36:46.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:45 smithi154 conmon[34553]: audit 2021-07-31T03:36:44.600905+0000 mgr.x (mgr.24542) 4697 : audit [DBG] from='client.29812 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:36:46.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:45 smithi154 conmon[34553]: audit 2021-07-31T03:36:44.943966+0000 mon.c 2021-07-31T03:36:46.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:45 smithi154 conmon[34553]: (mon.1) 216 : audit [DBG] 2021-07-31T03:36:46.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:45 smithi154 conmon[34553]: from='client.? 172.21.15.45:0/2666573603' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:36:46.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:45 smithi045 conmon[33655]: audit 2021-07-31T03:36:44.319511+0000 mgr.x (mgr.24542) 4696 2021-07-31T03:36:46.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:45 smithi045 conmon[33655]: : audit [DBG] from='client.29806 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:36:46.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:45 smithi045 conmon[33655]: audit 2021-07-31T03:36:44.600905+0000 mgr.x (mgr.24542) 4697 : audit [DBG] from='client.29812 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:36:46.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:45 smithi045 conmon[33655]: audit 2021-07-31T03:36:44 2021-07-31T03:36:46.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:45 smithi045 conmon[33655]: .943966+0000 mon.c (mon.1) 2021-07-31T03:36:46.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:45 smithi045 conmon[33655]: 216 : audit [DBG] from='client.? 172.21.15.45:0/2666573603' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:36:46.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:45 smithi045 conmon[39718]: audit 2021-07-31T03:36:44.319511+0000 mgr.x (mgr.24542) 4696 : audit [DBG] 2021-07-31T03:36:46.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:45 smithi045 conmon[39718]: from='client.29806 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:36:46.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:45 smithi045 conmon[39718]: audit 2021-07-31T03:36:44.600905+0000 mgr.x (mgr.24542) 4697 : audit 2021-07-31T03:36:46.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:45 smithi045 conmon[39718]: [DBG] from='client.29812 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:36:46.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:45 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T03:36:46.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:45 smithi045 conmon[39718]: 31T03:36:44.943966+0000 mon.c (mon.1) 2021-07-31T03:36:46.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:45 smithi045 conmon[39718]: 216 : audit [DBG] from='client.? 172.21.15.45:0/2666573603' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:36:47.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:46 smithi154 conmon[34553]: cluster 2021-07-31T03:36:45.357866+0000 mgr.x (mgr.24542) 4698 : cluster [DBG] pgmap v5293: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:47.024 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:46 smithi045 conmon[39718]: cluster 2021-07-31T03:36:45.357866+0000 mgr.x ( 2021-07-31T03:36:47.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:46 smithi045 conmon[39718]: mgr.24542) 4698 : cluster [DBG] pgmap v5293: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:47.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:46 smithi045 conmon[33655]: cluster 2021-07-31T03:36:45.357866+0000 mgr.x (mgr.24542) 2021-07-31T03:36:47.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:46 smithi045 conmon[33655]: 4698 : cluster [DBG] pgmap v5293: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:49.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:48 smithi154 conmon[34553]: cluster 2021-07-31T03:36:47.358399+0000 mgr.x (mgr.24542) 4699 : cluster [DBG] pgmap v5294: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:49.024 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:48 smithi045 conmon[39718]: cluster 2021-07-31T03:36:47.358399+0000 mgr.x ( 2021-07-31T03:36:49.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:48 smithi045 conmon[39718]: mgr.24542) 4699 : cluster [DBG] pgmap v5294: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:49.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:48 smithi045 conmon[33655]: cluster 2021-07-31T03:36:47.358399+0000 mgr.x (mgr.24542 2021-07-31T03:36:49.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:48 smithi045 conmon[33655]: ) 4699 : cluster [DBG] pgmap v5294: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:51.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:50 smithi154 conmon[34553]: cluster 2021-07-31T03:36:49.358838+0000 mgr.x (mgr.24542) 4700 : cluster [DBG] pgmap v5295: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:51.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:50 smithi045 conmon[33655]: cluster 2021-07-31T03:36:49.358838 2021-07-31T03:36:51.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:50 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 4700 : cluster [DBG] pgmap v5295: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:51.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:50 smithi045 conmon[39718]: cluster 2021-07-31T03:36:49.358838+0000 mgr.x (mgr 2021-07-31T03:36:51.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:50 smithi045 conmon[39718]: .24542) 4700 : cluster [DBG] pgmap v5295: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:53.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:52 smithi154 conmon[34553]: cluster 2021-07-31T03:36:51.359415+0000 mgr.x (mgr.24542) 4701 : cluster [DBG] pgmap v5296: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:53.024 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:52 smithi045 conmon[39718]: cluster 2021-07-31T03:36:51.359415+0000 mgr.x ( 2021-07-31T03:36:53.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:52 smithi045 conmon[39718]: mgr.24542) 4701 : cluster [DBG] pgmap v5296: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:53.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:52 smithi045 conmon[33655]: cluster 2021-07-31T03:36:51.359415+0000 mgr.x (mgr.24542) 4701 : cluster [DBG] 2021-07-31T03:36:53.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:52 smithi045 conmon[33655]: pgmap v5296: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:55.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:54 smithi154 conmon[34553]: cluster 2021-07-31T03:36:53.359931+0000 mgr.x (mgr.24542) 4702 : cluster [DBG] pgmap v5297: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:55.024 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:54 smithi045 conmon[33655]: cluster 2021-07-31T03:36:53.359931+0000 2021-07-31T03:36:55.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:54 smithi045 conmon[33655]: mgr.x (mgr.24542) 4702 : cluster [DBG] pgmap v5297: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:55.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:54 smithi045 conmon[39718]: cluster 2021-07-31T03:36:53.359931+0000 mgr.x (mgr. 2021-07-31T03:36:55.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:54 smithi045 conmon[39718]: 24542) 4702 : cluster [DBG] pgmap v5297: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:57.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:56 smithi154 conmon[34553]: cluster 2021-07-31T03:36:55.360609+0000 mgr.x (mgr.24542) 4703 : cluster [DBG] pgmap v5298: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:57.024 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:56 smithi045 conmon[39718]: cluster 2021-07-31T03:36:55.360609+0000 mgr.x (mgr.24542 2021-07-31T03:36:57.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:56 smithi045 conmon[39718]: ) 4703 : cluster [DBG] pgmap v5298: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:57.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:56 smithi045 conmon[33655]: cluster 2021-07-31T03:36:55.360609+0000 mgr.x (mgr.24542) 4703 : 2021-07-31T03:36:57.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:56 smithi045 conmon[33655]: cluster [DBG] pgmap v5298: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:59.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:58 smithi154 conmon[34553]: cluster 2021-07-31T03:36:57.361153+0000 mgr.x (mgr.24542) 4704 : 2021-07-31T03:36:59.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:36:58 smithi154 conmon[34553]: cluster [DBG] pgmap v5299: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:59.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:58 smithi045 conmon[39718]: cluster 2021-07-31T03:36:57.361153+0000 mgr.x (mgr 2021-07-31T03:36:59.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:36:58 smithi045 conmon[39718]: .24542) 4704 : cluster [DBG] pgmap v5299: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:36:59.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:58 smithi045 conmon[33655]: cluster 2021-07-31T03:36:57.361153+0000 mgr.x (mgr.24542) 4704 : 2021-07-31T03:36:59.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:36:58 smithi045 conmon[33655]: cluster [DBG] pgmap v5299: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:01.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:00 smithi154 conmon[34553]: cluster 2021-07-31T03:36:59.361728+0000 mgr.x (mgr.24542) 4705 : cluster [DBG] 2021-07-31T03:37:01.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:00 smithi154 conmon[34553]: pgmap v5300: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:01.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:00 smithi045 conmon[39718]: cluster 2021-07-31T03:36:59.361728+0000 mgr.x (mgr.24542) 4705 2021-07-31T03:37:01.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:00 smithi045 conmon[39718]: : cluster [DBG] pgmap v5300: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:01.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:00 smithi045 conmon[33655]: cluster 2021-07-31T03:36:59.361728+0000 mgr.x (mgr.24542) 4705 : cluster 2021-07-31T03:37:01.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:00 smithi045 conmon[33655]: [DBG] pgmap v5300: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:03.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:02 smithi154 conmon[34553]: cluster 2021-07-31T03:37:01.362341+0000 mgr.x (mgr.24542 2021-07-31T03:37:03.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:02 smithi154 conmon[34553]: ) 4706 : cluster [DBG] pgmap v5301: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:03.024 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:02 smithi045 conmon[39718]: cluster 2021-07-31T03:37:01.362341+0000 mgr.x (mgr.24542) 2021-07-31T03:37:03.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:02 smithi045 conmon[39718]: 4706 : cluster [DBG] pgmap v5301: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:03.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:02 smithi045 conmon[33655]: cluster 2021-07-31T03:37:01.362341+0000 mgr.x (mgr.24542 2021-07-31T03:37:03.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:02 smithi045 conmon[33655]: ) 4706 : cluster [DBG] pgmap v5301: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:05.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:04 smithi154 conmon[34553]: cluster 2021-07-31T03:37:03.362778+0000 mgr.x (mgr.24542) 4707 : cluster [DBG] 2021-07-31T03:37:05.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:04 smithi154 conmon[34553]: pgmap v5302: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:05.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:04 smithi045 conmon[39718]: cluster 2021-07-31T03:37:03 2021-07-31T03:37:05.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:04 smithi045 conmon[39718]: .362778+0000 mgr.x (mgr.24542) 4707 : cluster [DBG] pgmap v5302: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:05.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:04 smithi045 conmon[33655]: cluster 2021-07-31T03:37:03.362778+0000 mgr.x (mgr.24542) 4707 : cluster [DBG] pgmap v5302: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:07.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:06 smithi154 conmon[34553]: cluster 2021-07-31T03:37:05.363288+0000 mgr.x (mgr.24542) 4708 : cluster [DBG] pgmap v5303: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:07.024 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:06 smithi045 conmon[33655]: cluster 2021-07-31T03:37:05.363288+0000 mgr.x (mgr.24542) 4708 : cluster 2021-07-31T03:37:07.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:06 smithi045 conmon[33655]: [DBG] pgmap v5303: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:07.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:06 smithi045 conmon[39718]: cluster 2021-07-31T03:37:05.363288+0000 2021-07-31T03:37:07.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:06 smithi045 conmon[39718]: mgr.x (mgr.24542) 4708 : cluster [DBG] pgmap v5303: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:09.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:08 smithi154 conmon[34553]: cluster 2021-07-31T03:37:07.363679+0000 mgr.x (mgr.24542) 4709 : cluster 2021-07-31T03:37:09.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:08 smithi154 conmon[34553]: [DBG] pgmap v5304: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:09.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:08 smithi045 conmon[33655]: cluster 2021-07-31T03:37:07.363679+0000 mgr.x (mgr.24542) 4709 : cluster 2021-07-31T03:37:09.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:08 smithi045 conmon[33655]: [DBG] pgmap v5304: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:09.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:08 smithi045 conmon[39718]: cluster 2021-07-31T03:37:07.363679+0000 mgr.x ( 2021-07-31T03:37:09.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:08 smithi045 conmon[39718]: mgr.24542) 4709 : cluster [DBG] pgmap v5304: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:11.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:10 smithi154 conmon[34553]: cluster 2021-07-31T03:37:09.364179+0000 mgr.x (mgr.24542) 4710 2021-07-31T03:37:11.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:10 smithi154 conmon[34553]: : cluster [DBG] pgmap v5305: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:11.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:10 smithi045 conmon[39718]: cluster 2021-07-31T03:37:09.364179+0000 mgr.x (mgr.24542) 2021-07-31T03:37:11.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:10 smithi045 conmon[39718]: 4710 : cluster [DBG] pgmap v5305: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:11.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:10 smithi045 conmon[33655]: cluster 2021-07-31T03:37:09.364179+0000 2021-07-31T03:37:11.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:10 smithi045 conmon[33655]: mgr.x (mgr.24542) 4710 : cluster [DBG] pgmap v5305: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:13.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:12 smithi154 conmon[34553]: cluster 2021-07-31T03:37:11.364711+0000 mgr.x (mgr.24542) 4711 2021-07-31T03:37:13.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:12 smithi154 conmon[34553]: : cluster [DBG] pgmap v5306: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:13.024 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:12 smithi045 conmon[33655]: cluster 2021-07-31T03:37:11.364711+0000 mgr.x (mgr.24542) 4711 : cluster 2021-07-31T03:37:13.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:12 smithi045 conmon[33655]: [DBG] pgmap v5306: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:13.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:12 smithi045 conmon[39718]: cluster 2021-07-31T03:37:11.364711+0000 mgr.x 2021-07-31T03:37:13.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:12 smithi045 conmon[39718]: (mgr.24542) 4711 : cluster [DBG] pgmap v5306: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:15.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:14 smithi045 conmon[33655]: cluster 2021-07-31T03:37:13.365260+0000 mgr.x (mgr.24542) 4712 : cluster [DBG] 2021-07-31T03:37:15.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:14 smithi045 conmon[33655]: pgmap v5307: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:15.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:14 smithi045 conmon[39718]: cluster 2021-07-31T03:37:13.365260+0000 2021-07-31T03:37:15.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:14 smithi045 conmon[39718]: mgr.x (mgr.24542) 4712 : cluster [DBG] pgmap v5307: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:15.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:14 smithi154 conmon[34553]: cluster 2021-07-31T03:37:13.365260+0000 mgr.x (mgr.24542) 4712 : cluster 2021-07-31T03:37:15.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:14 smithi154 conmon[34553]: [DBG] pgmap v5307: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:15.255 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:37:15.533 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:37:15.533 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 112m ago 112m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:37:15.534 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 109m ago 112m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:37:15.534 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 109m ago 116m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:37:15.534 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 112m ago 118m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:37:15.534 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 112m ago 118m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:37:15.535 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 109m ago 116m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:37:15.535 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 112m ago 116m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:37:15.535 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 112m ago 113m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:37:15.535 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 109m ago 113m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:37:15.535 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 112m ago 115m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:37:15.536 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 112m ago 115m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:37:15.536 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 112m ago 115m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:37:15.536 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 112m ago 114m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:37:15.536 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 109m ago 114m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:37:15.537 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 109m ago 114m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:37:15.537 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 109m ago 113m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:37:15.537 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 109m ago 113m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:37:15.537 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 109m ago 113m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:37:15.867 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:37:15.868 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:37:15.868 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:37:15.869 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:37:15.869 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:37:15.870 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:37:15.870 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:37:15.870 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:37:15.871 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:37:15.871 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:37:15.872 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:37:15.872 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:37:15.873 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:37:15.873 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:37:15.873 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:37:15.874 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:37:15.874 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:37:17.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:16 smithi045 conmon[33655]: audit 2021-07-31T03:37:15.244374+0000 mgr.x (mgr.24542) 2021-07-31T03:37:17.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:16 smithi045 conmon[33655]: 4713 : audit [DBG] from='client.29824 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:37:17.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:16 smithi045 conmon[33655]: cluster 2021-07-31T03:37:15.365835+0000 mgr.x (mgr.24542) 4714 : cluster [DBG] pgmap v5308: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:17.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:16 smithi045 conmon[33655]: audit 2021-07-31T03:37:15.526785+0000 2021-07-31T03:37:17.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:16 smithi045 conmon[33655]: mgr.x (mgr.24542) 4715 : audit [DBG] from='client.29830 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:37:17.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:16 smithi045 conmon[33655]: audit 2021 2021-07-31T03:37:17.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:16 smithi045 conmon[33655]: -07-31T03:37:15.868749+0000 2021-07-31T03:37:17.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:16 smithi045 conmon[33655]: mon.c (mon.1) 217 : audit [DBG] from='client.? 172.21.15.45:0/1230872201' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:37:17.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:16 smithi045 conmon[39718]: audit 2021-07-31T03:37:15.244374+0000 mgr.x (mgr.24542) 4713 : 2021-07-31T03:37:17.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:16 smithi045 conmon[39718]: audit [DBG] from='client.29824 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:37:17.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:16 smithi045 conmon[39718]: cluster 2021-07-31T03:37:15.365835+0000 mgr.x (mgr.24542) 4714 : cluster [DBG] pgmap v5308: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:17.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:16 smithi045 conmon[39718]: audit 2021-07-31T03:37:15.526785+0000 2021-07-31T03:37:17.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:16 smithi045 conmon[39718]: mgr.x (mgr.24542) 4715 : audit [DBG] from='client.29830 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:37:17.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:16 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T03:37:17.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:16 smithi045 conmon[39718]: T03:37:15.868749+0000 mon.c (mon.1) 217 : audit 2021-07-31T03:37:17.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:16 smithi045 conmon[39718]: [DBG] from='client.? 172.21.15.45:0/1230872201' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:37:17.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:16 smithi154 conmon[34553]: audit 2021-07-31T03:37:15.244374+0000 mgr.x (mgr.24542) 4713 : audit [DBG] 2021-07-31T03:37:17.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:16 smithi154 conmon[34553]: from='client.29824 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:37:17.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:16 smithi154 conmon[34553]: cluster 2021-07-31T03:37:15.365835+0000 mgr.x (mgr.24542) 4714 : cluster [DBG] pgmap v5308: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:17.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:16 smithi154 conmon[34553]: audit 2021-07-31T03:37:15.526785+0000 mgr.x (mgr.24542) 4715 : audit [DBG] from='client.29830 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:37:17.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:16 smithi154 conmon[34553]: audit 2021- 2021-07-31T03:37:17.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:16 smithi154 conmon[34553]: 07-31T03:37:15.868749+0000 mon.c (mon.1) 2021-07-31T03:37:17.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:16 smithi154 conmon[34553]: 217 : audit [DBG] from='client.? 172.21.15.45:0/1230872201' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:37:19.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:18 smithi045 conmon[33655]: cluster 2021-07-31T03:37:17.366378+0000 mgr.x (mgr.24542) 4716 : cluster 2021-07-31T03:37:19.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:18 smithi045 conmon[33655]: [DBG] pgmap v5309: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:19.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:18 smithi045 conmon[39718]: cluster 2021-07-31T03:37:17.366378+0000 mgr.x ( 2021-07-31T03:37:19.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:18 smithi045 conmon[39718]: mgr.24542) 4716 : cluster [DBG] pgmap v5309: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:19.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:18 smithi154 conmon[34553]: cluster 2021-07-31T03:37:17.366378+0000 mgr.x (mgr.24542) 4716 : cluster [DBG] 2021-07-31T03:37:19.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:18 smithi154 conmon[34553]: pgmap v5309: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:21.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:20 smithi045 conmon[39718]: cluster 2021-07-31T03:37:19.367053+0000 mgr.x (mgr 2021-07-31T03:37:21.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:20 smithi045 conmon[39718]: .24542) 4717 : cluster [DBG] pgmap v5310: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:21.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:20 smithi045 conmon[33655]: cluster 2021-07-31T03:37:19.367053+0000 2021-07-31T03:37:21.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:20 smithi045 conmon[33655]: mgr.x (mgr.24542) 4717 : cluster [DBG] pgmap v5310: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:21.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:20 smithi154 conmon[34553]: cluster 2021-07-31T03:37:19.367053+0000 mgr.x (mgr.24542) 4717 : cluster [DBG] pgmap v5310: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:23.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:22 smithi045 conmon[33655]: cluster 2021-07-31T03:37:21.367574+0000 mgr.x (mgr. 2021-07-31T03:37:23.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:22 smithi045 conmon[33655]: 24542) 4718 : cluster [DBG] pgmap v5311: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:23.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:22 smithi045 conmon[39718]: cluster 2021-07-31T03:37:21.367574+0000 mgr.x (mgr 2021-07-31T03:37:23.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:22 smithi045 conmon[39718]: .24542) 4718 : cluster [DBG] pgmap v5311: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:23.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:22 smithi154 conmon[34553]: cluster 2021-07-31T03:37:21.367574+0000 mgr.x (mgr.24542) 4718 : cluster [DBG] pgmap v5311: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:24.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:24 smithi154 conmon[34553]: cluster 2021-07-31T03:37:23.367942+0000 mgr.x (mgr.24542) 4719 : cluster [DBG] pgmap v5312: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:24.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:24 smithi154 conmon[34553]: 2021-07-31T03:37:24.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:24 smithi154 conmon[34553]: cluster 2021-07-31T03:37:23.383185+0000 mon.a (mon.0) 3335 : cluster [DBG] osdmap e2162: 8 total, 8 up, 8 in 2021-07-31T03:37:24.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:24 smithi045 conmon[33655]: cluster 2021-07-31T03:37:23.367942+0000 mgr.x (mgr.24542) 4719 : cluster 2021-07-31T03:37:24.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:24 smithi045 conmon[33655]: [DBG] pgmap v5312: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:24.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:24 smithi045 conmon[33655]: cluster 2021-07-31T03:37:23.383185+0000 mon.a (mon.0) 3335 : cluster [DBG] osdmap e2162: 8 total, 8 up, 8 in 2021-07-31T03:37:24.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:24 smithi045 conmon[39718]: cluster 2021-07-31T03:37:23.367942+0000 2021-07-31T03:37:24.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:24 smithi045 conmon[39718]: mgr.x (mgr.24542) 4719 : cluster [DBG] pgmap v5312: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:24.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:24 smithi045 conmon[39718]: cluster 2021-07-31T03:37:24.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:24 smithi045 conmon[39718]: 2021-07-31T03:37:23.383185+0000 mon.a ( 2021-07-31T03:37:24.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:24 smithi045 conmon[39718]: mon.0) 3335 : cluster [DBG] osdmap e2162: 8 total, 8 up, 8 in 2021-07-31T03:37:26.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:26 smithi154 conmon[34553]: cluster 2021-07-31T03:37:25.368478+0000 mgr.x (mgr.24542) 4720 : cluster [DBG] pgmap v5314: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:26.774 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:26 smithi045 conmon[33655]: cluster 2021-07-31T03: 2021-07-31T03:37:26.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:26 smithi045 conmon[33655]: 37:25.368478+0000 mgr.x (mgr.24542) 4720 : cluster [DBG] pgmap v5314: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:26.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:26 smithi045 conmon[39718]: cluster 2021-07-31T03:37:25.368478+0000 mgr.x (mgr. 2021-07-31T03:37:26.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:26 smithi045 conmon[39718]: 24542) 4720 : cluster [DBG] pgmap v5314: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:28.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:28 smithi154 conmon[34553]: cluster 2021-07-31T03:37:27.368876+0000 mgr.x (mgr.24542) 4721 : cluster [DBG] 2021-07-31T03:37:28.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:28 smithi154 conmon[34553]: pgmap v5315: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:28.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:28 smithi154 conmon[34553]: audit 2021-07-31T03:37:28.373205+0000 mon.b (mon.2) 625 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:37:28.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:28 smithi154 conmon[34553]: audit 2021-07-31T03:37:28.373491+0000 mon.b (mon 2021-07-31T03:37:28.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:28 smithi154 conmon[34553]: .2) 626 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:37:28.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:28 smithi154 conmon[34553]: audit 2021-07-31T 2021-07-31T03:37:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:28 smithi154 conmon[34553]: 03:37:28.373956+0000 mon.a (mon.0) 3336 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:37:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:28 smithi154 conmon[34553]: audit 2021-07-31T03:37: 2021-07-31T03:37:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:28 smithi154 conmon[34553]: 28.374545+0000 mon.a (mon.0) 3337 : audit [INF] 2021-07-31T03:37:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:28 smithi154 conmon[34553]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:37:28.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:28 smithi045 conmon[39718]: cluster 2021-07-31T03: 2021-07-31T03:37:28.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:28 smithi045 conmon[39718]: 37:27.368876+0000 mgr.x (mgr.24542) 4721 : cluster [DBG] pgmap v5315: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:28.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:28 smithi045 conmon[39718]: audit 2021 2021-07-31T03:37:28.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:28 smithi045 conmon[39718]: -07-31T03:37:28. 2021-07-31T03:37:28.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:28 smithi045 conmon[39718]: 373205+0000 mon.b (mon.2 2021-07-31T03:37:28.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:28 smithi045 conmon[39718]: ) 625 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:37:28.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:28 smithi045 conmon[39718]: 2021-07-31T03:37:28.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:28 smithi045 conmon[39718]: audit 2021 2021-07-31T03:37:28.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:28 smithi045 conmon[39718]: -07-31T03:37:28 2021-07-31T03:37:28.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:28 smithi045 conmon[39718]: .373491+0000 mon.b (mon. 2021-07-31T03:37:28.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:28 smithi045 conmon[39718]: 2) 626 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:37:28.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:28 smithi045 conmon[39718]: 2021-07-31T03:37:28.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:28 smithi045 conmon[39718]: audit 2021 2021-07-31T03:37:28.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:28 smithi045 conmon[39718]: -07-31T03:37:28 2021-07-31T03:37:28.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:28 smithi045 conmon[39718]: .373956+0000 mon.a (mon 2021-07-31T03:37:28.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:28 smithi045 conmon[39718]: .0) 3336 : audit [INF] 2021-07-31T03:37:28.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:28 smithi045 conmon[39718]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:37:28.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:28 smithi045 conmon[39718]: audit 2021 2021-07-31T03:37:28.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:28 smithi045 conmon[39718]: -07-31T03:37:28 2021-07-31T03:37:28.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:28 smithi045 conmon[39718]: .374545+0000 mon.a (mon. 2021-07-31T03:37:28.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:28 smithi045 conmon[39718]: 0) 3337 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:37:28.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:28 smithi045 conmon[39718]: 2021-07-31T03:37:28.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:28 smithi045 conmon[33655]: cluster 2021-07-31T03:37:27.368876+0000 2021-07-31T03:37:28.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:28 smithi045 conmon[33655]: mgr.x (mgr.24542) 4721 : cluster [DBG] pgmap v5315: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:28.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:28 smithi045 conmon[33655]: audit 2021-07-31T03: 2021-07-31T03:37:28.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:28 smithi045 conmon[33655]: 37:28.373205+0000 mon.b (mon.2) 625 : audit 2021-07-31T03:37:28.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:28 smithi045 conmon[33655]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:37:28.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:28 smithi045 conmon[33655]: audit 2021-07-31T03:37:28.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:28 smithi045 conmon[33655]: 2021-07-31T03:37 2021-07-31T03:37:28.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:28 smithi045 conmon[33655]: :28.373491+0000 mon.b (mon.2) 626 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:37:28.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:28 smithi045 conmon[33655]: audit 2021-07-31T 2021-07-31T03:37:28.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:28 smithi045 conmon[33655]: 03:37:28.373956+0000 mon.a (mon.0 2021-07-31T03:37:28.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:28 smithi045 conmon[33655]: ) 3336 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:37:28.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:28 smithi045 conmon[33655]: audit 2021 2021-07-31T03:37:28.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:28 smithi045 conmon[33655]: -07-31T03:37:28.374545 2021-07-31T03:37:28.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:28 smithi045 conmon[33655]: +0000 mon.a (mon.0) 3337 : audit 2021-07-31T03:37:28.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:28 smithi045 conmon[33655]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:37:30.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:30 smithi154 conmon[34553]: cluster 2021-07-31T03:37:29.369476+0000 mgr.x (mgr.24542) 4722 : cluster [DBG] pgmap v5316: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:30.774 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:30 smithi045 conmon[33655]: cluster 2021-07-31T03:37:29.369476+0000 mgr.x (mgr.24542) 4722 : cluster [DBG] pgmap v5316: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:30.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:30 smithi045 conmon[39718]: cluster 2021-07-31T03:37:29.369476+0000 2021-07-31T03:37:30.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:30 smithi045 conmon[39718]: mgr.x (mgr.24542) 4722 : cluster [DBG] pgmap v5316: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:32.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:32 smithi154 conmon[34553]: cluster 2021-07-31T03:37:31.370049+0000 mgr.x (mgr.24542) 4723 : cluster [DBG] pgmap v5317: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:32.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:32 smithi045 conmon[33655]: cluster 2021-07-31T03:37:31.370049+0000 mgr.x (mgr.24542) 4723 : cluster [DBG] pgmap v5317: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:32.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:32 smithi045 conmon[39718]: cluster 2021-07-31T03:37:31.370049+0000 2021-07-31T03:37:32.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:32 smithi045 conmon[39718]: mgr.x (mgr.24542) 4723 : cluster [DBG] pgmap v5317: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:34.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[34553]: cluster 2021-07-31T03:37:33.370535+0000 mgr.x (mgr.24542) 4724 : cluster [DBG] pgmap v5318: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:34.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[34553]: audit 2021-07-31T03:37:34.183894+0000 mon.b (mon.2) 627 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:37:34.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[34553]: audit 2021-07-31T03:37:34.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[34553]: 2021-07-31T03:37:34.185404+0000 mon.b (mon.2) 628 : audit 2021-07-31T03:37:34.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[34553]: [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:37:34.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[34553]: audit 2021-07-31T03:37:34. 2021-07-31T03:37:34.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[34553]: 186349+0000 mon.b (mon.2) 629 : audit 2021-07-31T03:37:34.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[34553]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:37:34.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: debug 2021-07-31T03:37:34.229+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:37:34.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:37:34.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:37:34.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: yield 2021-07-31T03:37:34.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:37:34.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:37:34.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:37:34.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:37:34.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:37:34.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:37:34.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:37:34.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:37:34.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:37:34.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: task.result() 2021-07-31T03:37:34.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:37:34.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: debug 2021-07-31T03:37:34.240+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:37:34.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:37:34.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:37:34.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: yield 2021-07-31T03:37:34.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:37:34.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:37:34.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:37:34.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:37:34.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:37:34.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:37:34.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:37:34.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:37:34.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:37:34.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: task.result() 2021-07-31T03:37:34.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:37:34.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: debug 2021-07-31T03:37:34.265+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:37:34.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:37:34.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:37:34.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: yield 2021-07-31T03:37:34.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:37:34.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:37:34.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:37:34.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:37:34.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:37:34.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:37:34.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:37:34.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:37:34.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:37:34.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: task.result() 2021-07-31T03:37:34.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:37:34.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: debug 2021-07-31T03:37:34.266+0000 7fcb94162700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:37:34.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:37:34.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:37:34.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: yield 2021-07-31T03:37:34.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:37:34.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:37:34.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:37:34.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:37:34.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:37:34.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:37:34.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:37:34.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:37:34.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:37:34.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: task.result() 2021-07-31T03:37:34.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:37:34.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: 2021-07-31T03:37:34.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:37:34.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: 2021-07-31T03:37:34.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:37:34.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:37:34.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:37:34.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:37:34.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:37:34.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:37:34.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:37:34.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:37:34.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:37:34.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:37:34.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:37:34.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:37:34.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:37:34.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:37:34.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:37:34.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:37:34.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:37:34.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:37:34.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: raise 2021-07-31T03:37:34.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:37:34.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:37:34.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:37:34.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:37:34.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:37:34.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: debug 2021-07-31T03:37:34.274+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:37:34.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:37:34.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:37:34.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: yield 2021-07-31T03:37:34.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:37:34.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:37:34.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:37:34.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:37:34.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:37:34.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:37:34.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:37:34.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:37:34.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:37:34.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: task.result() 2021-07-31T03:37:34.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:37:34.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: debug 2021-07-31T03:37:34.275+0000 7fcb93961700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:37:34.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:37:34.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:37:34.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: yield 2021-07-31T03:37:34.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:37:34.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:37:34.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:37:34.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:37:34.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:37:34.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:37:34.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:37:34.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:37:34.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:37:34.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: task.result() 2021-07-31T03:37:34.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:37:34.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: 2021-07-31T03:37:34.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:37:34.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: 2021-07-31T03:37:34.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:37:34.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:37:34.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:37:34.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:37:34.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:37:34.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:37:34.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:37:34.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:37:34.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:37:34.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:37:34.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:37:34.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:37:34.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:37:34.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:37:34.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:37:34.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:37:34.535 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:37:34.535 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:37:34.535 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: raise 2021-07-31T03:37:34.535 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:37:34.536 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:37:34.536 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:37:34.536 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:37:34.536 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:37:34 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:37:34.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:34 smithi045 conmon[33655]: cluster 2021-07-31T03:37:33.370535+0000 mgr.x (mgr.24542) 4724 2021-07-31T03:37:34.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:34 smithi045 conmon[33655]: : cluster [DBG] pgmap v5318: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:34.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:34 smithi045 conmon[33655]: audit 2021-07-31T03:37:34.183894+0000 mon.b (mon.2) 627 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:37:34.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:34 smithi045 conmon[33655]: audit 2021-07-31T03:37:34.185404+0000 mon.b (mon.2) 628 : audit [DBG] 2021-07-31T03:37:34.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:34 smithi045 conmon[33655]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:37:34.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:34 smithi045 conmon[33655]: audit 2021-07-31T03:37:34 2021-07-31T03:37:34.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:34 smithi045 conmon[33655]: .186349+0000 mon.b (mon.2) 629 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:37:34.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:34 smithi045 conmon[39718]: cluster 2021-07-31T03:37:33.370535+0000 2021-07-31T03:37:34.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:34 smithi045 conmon[39718]: mgr.x (mgr.24542) 4724 : cluster [DBG] pgmap v5318: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:34.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:34 smithi045 conmon[39718]: audit 2021-07 2021-07-31T03:37:34.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:34 smithi045 conmon[39718]: -31T03:37:34.183894+0000 mon.b (mon.2) 2021-07-31T03:37:34.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:34 smithi045 conmon[39718]: 627 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:37:34.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:34 smithi045 conmon[39718]: audit 2021- 2021-07-31T03:37:34.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:34 smithi045 conmon[39718]: 07-31T03:37:34.185404+0000 2021-07-31T03:37:34.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:34 smithi045 conmon[39718]: mon.b (mon.2) 628 : audit [DBG] 2021-07-31T03:37:34.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:34 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:37:34.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:34 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T03:37:34.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:34 smithi045 conmon[39718]: 03:37:34.186349+0000 mon.b (mon 2021-07-31T03:37:34.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:34 smithi045 conmon[39718]: .2) 629 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:37:34.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:34 smithi045 conmon[39718]: 2021-07-31T03:37:35.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:37:34.187157+0000 mgr.x (mgr.24542) 4725 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:37:35.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:37:34 2021-07-31T03:37:35.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: .229950+0000 mgr.x (mgr.24542) 4726 : cephadm [ERR] 2021-07-31T03:37:35.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:37:35.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:37:35.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: yield 2021-07-31T03:37:35.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:37:35.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:37:35.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:37:35.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:37:35.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:37:35.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:37:35.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:37:35.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:37:35.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:37:35.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: task.result() 2021-07-31T03:37:35.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:37:35.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:37:34.230365+0000 mgr.x (mgr.24542) 4727 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:37:35.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:37:34. 2021-07-31T03:37:35.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: 241450+0000 mgr.x (mgr.24542) 4728 : cephadm [ERR] 2021-07-31T03:37:35.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:37:35.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:37:35.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: yield 2021-07-31T03:37:35.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:37:35.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:37:35.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:37:35.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:37:35.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:37:35.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:37:35.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:37:35.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:37:35.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:37:35.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: task.result() 2021-07-31T03:37:35.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:37:35.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:37:34.242312+0000 mgr.x (mgr.24542) 4729 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:37:35.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:37:34.266862+0000 mgr.x (mgr.24542) 4730 : cephadm [ERR] 2021-07-31T03:37:35.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:37:35.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:37:35.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: yield 2021-07-31T03:37:35.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:37:35.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:37:35.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:37:35.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:37:35.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:37:35.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:37:35.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:37:35.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:37:35.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:37:35.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: task.result() 2021-07-31T03:37:35.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:37:35.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:37:34.267483+0000 mgr.x (mgr.24542) 4731 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:37:35.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:37:35.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:37:35.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: yield 2021-07-31T03:37:35.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:37:35.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:37:35.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:37:35.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:37:35.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:37:35.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:37:35.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:37:35.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:37:35.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:37:35.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: task.result() 2021-07-31T03:37:35.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:37:35.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: 2021-07-31T03:37:35.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:37:35.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: 2021-07-31T03:37:35.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:37:35.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:37:35.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:37:35.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:37:35.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:37:35.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:37:35.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:37:35.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:37:35.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:37:35.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:37:35.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:37:35.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:37:35.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:37:35.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:37:35.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:37:35.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:37:35.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:37:35.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:37:35.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: raise 2021-07-31T03:37:35.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:37:35.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:37:35.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:37:35.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:37:35.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:37:35.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:37:34.275767+0000 mgr.x (mgr.24542) 4732 : cephadm [ERR] 2021-07-31T03:37:35.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:37:35.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:37:35.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: yield 2021-07-31T03:37:35.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:37:35.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:37:35.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:37:35.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:37:35.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:37:35.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:37:35.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:37:35.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:37:35.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:37:35.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: task.result() 2021-07-31T03:37:35.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:37:35.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:37:34.276186+0000 mgr.x (mgr.24542) 4733 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:37:35.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:37:35.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:37:35.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: yield 2021-07-31T03:37:35.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:37:35.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:37:35.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:37:35.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:37:35.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:37:35.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:37:35.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:37:34.187157+0000 mgr.x (mgr.24542) 4725 : cephadm 2021-07-31T03:37:35.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:37:35.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:37:34.229950+0000 mgr.x (mgr.24542) 4726 : cephadm [ERR] 2021-07-31T03:37:35.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:37:35.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:37:35.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: yield 2021-07-31T03:37:35.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:37:35.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:37:35.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:37:35.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:37:35.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:37:35.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:37:35.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:37:35.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:37:35.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:37:35.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: task.result() 2021-07-31T03:37:35.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:37:35.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:37:34.230365+0000 mgr.x (mgr.24542) 4727 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:37:35.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:37:34.241450+0000 mgr.x (mgr.24542) 4728 : cephadm [ERR] 2021-07-31T03:37:35.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:37:35.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:37:35.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: yield 2021-07-31T03:37:35.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:37:35.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:37:35.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:37:35.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:37:35.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:37:35.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:37:35.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:37:35.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:37:35.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:37:35.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: task.result() 2021-07-31T03:37:35.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:37:35.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:37:34.242312+0000 2021-07-31T03:37:35.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: mgr.x (mgr.24542) 4729 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:37:35.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:37:34.266862+0000 mgr.x (mgr.24542) 4730 : cephadm [ERR] 2021-07-31T03:37:35.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:37:35.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:37:35.785 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:37:35.785 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:37:35.785 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:37:35.785 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: task.result() 2021-07-31T03:37:35.785 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:37:35.786 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: 2021-07-31T03:37:35.786 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:37:35.786 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: 2021-07-31T03:37:35.786 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:37:35.787 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:37:35.787 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:37:35.787 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:37:35.787 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:37:35.787 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:37:35.788 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:37:35.788 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:37:35.788 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:37:35.788 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:37:35.788 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:37:35.789 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:37:35.789 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:37:35.789 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:37:35.789 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:37:35.789 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:37:35.790 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:37:35.790 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:37:35.790 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: raise 2021-07-31T03:37:35.790 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:37:35.790 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:37:35.791 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:37:35.791 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:37:35.791 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:35 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:37:35.791 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:37:34.187157+0000 mgr.x (mgr. 2021-07-31T03:37:35.792 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: 24542) 4725 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:37:35.792 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:37:34.229950+0000 mgr.x (mgr.24542) 4726 : cephadm [ERR] 2021-07-31T03:37:35.792 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:37:35.792 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:37:35.792 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: yield 2021-07-31T03:37:35.792 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:37:35.793 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:37:35.793 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:37:35.793 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:37:35.793 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:37:35.793 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:37:35.794 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:37:35.794 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:37:35.794 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:37:35.794 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: task.result() 2021-07-31T03:37:35.795 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:37:35.795 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:37:34.230365+0000 mgr.x (mgr.24542) 4727 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:37:35.795 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:37:34.241450+0000 mgr.x (mgr.24542) 4728 : cephadm [ERR] 2021-07-31T03:37:35.795 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: 2021-07-31T03:37:35.795 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:37:35.796 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:37:35.796 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: yield 2021-07-31T03:37:35.796 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:37:35.796 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:37:35.796 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:37:35.797 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:37:35.797 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:37:35.797 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:37:35.797 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:37:35.797 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:37:35.798 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:37:35.798 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: task.result() 2021-07-31T03:37:35.798 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:37:35.798 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:37:35.798 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: 2021-07-31T03:37:34.242312+0000 mgr.x (mgr.24542) 4729 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:37:35.799 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:37:34.266862+0000 mgr.x (mgr.24542) 4730 : cephadm [ERR] 2021-07-31T03:37:35.799 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:37:35.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: yield 2021-07-31T03:37:35.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:37:35.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:37:35.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:37:35.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:37:35.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:37:35.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:37:35.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:37:35.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:37:35.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:37:35.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: task.result() 2021-07-31T03:37:35.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:37:35.802 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:37:34.267483+0000 mgr.x (mgr.24542) 4731 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:37:35.802 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:37:35.802 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:37:35.802 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: yield 2021-07-31T03:37:35.802 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:37:35.803 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:37:35.803 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:37:35.803 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:37:35.803 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:37:35.803 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:37:35.804 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:37:35.804 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:37:35.804 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:37:35.804 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: task.result() 2021-07-31T03:37:35.804 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:37:35.805 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: 2021-07-31T03:37:35.805 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:37:35.805 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: 2021-07-31T03:37:35.805 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:37:35.805 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:37:35.806 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:37:35.806 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:37:35.806 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:37:35.806 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:37:35.806 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:37:35.807 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:37:35.807 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:37:35.807 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:37:35.807 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:37:35.808 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:37:35.808 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: yield 2021-07-31T03:37:35.808 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:37:35.808 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:37:35.809 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:37:35.809 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:37:35.809 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:37:35.809 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:37:35.810 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:37:35.810 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:37:35.810 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:37:35.810 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: task.result() 2021-07-31T03:37:35.811 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:37:35.811 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:37:34.267483+0000 mgr.x (mgr.24542) 4731 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:37:35.811 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:37:35.811 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:37:35.812 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: yield 2021-07-31T03:37:35.812 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:37:35.812 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:37:35.812 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:37:35.813 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:37:35.813 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:37:35.813 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:37:35.813 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:37:35.814 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:37:35.814 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:37:35.814 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: task.result() 2021-07-31T03:37:35.814 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:37:35.815 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: 2021-07-31T03:37:35.815 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:37:35.815 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: 2021-07-31T03:37:35.815 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:37:35.815 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:37:35.816 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:37:35.816 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:37:35.816 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:37:35.816 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:37:35.816 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:37:35.817 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:37:35.817 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:37:35.817 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:37:35.817 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:37:35.817 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:37:35.818 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:37:35.818 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:37:35.818 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:37:35.818 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:37:35.818 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:37:35.819 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:37:35.819 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: raise 2021-07-31T03:37:35.819 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:37:35.819 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:37:35.819 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:37:35.820 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:37:35.820 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:37:35.820 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: cephadm 2021-07-31 2021-07-31T03:37:35.820 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: T03:37:34.275767+0000 mgr.x (mgr.24542) 4732 : cephadm [ERR] 2021-07-31T03:37:35.820 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:37:35.821 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:37:35.821 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: yield 2021-07-31T03:37:35.821 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:37:35.821 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:37:35.821 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:37:35.822 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:37:35.822 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:37:35.822 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:37:35.822 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:37:35.822 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:37:35.823 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:37:35.823 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: task.result() 2021-07-31T03:37:35.823 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:37:35.823 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:37:34.276186+0000 mgr.x (mgr.24542) 4733 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:37:35.823 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:37:35.824 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:37:35.824 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: yield 2021-07-31T03:37:35.824 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:37:35.824 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:37:35.824 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:37:35.825 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:37:35.825 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:37:35.825 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:37:35.825 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:37:35.826 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:37:35.826 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:37:35.826 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:37:35.826 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:37:35.827 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:37:35.827 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:37:35.827 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: raise 2021-07-31T03:37:35.827 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:37:35.827 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:37:35.828 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:37:35.828 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:37:35.828 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:37:35.828 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:37:34.275767+0000 mgr.x (mgr.24542) 4732 : cephadm [ERR] 2021-07-31T03:37:35.828 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:37:35.829 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:37:35.829 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: yield 2021-07-31T03:37:35.829 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:37:35.829 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:37:35.830 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:37:35.830 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:37:35.830 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:37:35.830 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:37:35.830 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:37:35.831 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:37:35.831 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:37:35.831 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: task.result() 2021-07-31T03:37:35.831 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:37:35.831 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:37:34.276186+0000 mgr.x (mgr.24542) 4733 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:37:35.832 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:37:35.832 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:37:35.832 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: yield 2021-07-31T03:37:35.832 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:37:35.832 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:37:35.833 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:37:35.833 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:37:35.833 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:37:35.833 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:37:35.834 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:37:35.834 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:37:35.834 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:37:35.834 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: task.result() 2021-07-31T03:37:35.835 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:37:35.835 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: 2021-07-31T03:37:35.835 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:37:35.835 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: 2021-07-31T03:37:35.835 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:37:35.836 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:37:35.836 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:37:35.836 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:37:35.836 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:37:35.836 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:37:35.837 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:37:35.837 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:37:35.837 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:37:35.837 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:37:35.837 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:37:35.837 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:37:35.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:37:35.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:37:35.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:37:35.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:37:35.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:37:35.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:37:35.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: raise 2021-07-31T03:37:35.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:37:35.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:37:35.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:37:35.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:37:35.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:37:35.840 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:37:35.840 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:37:35.841 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:37:35.841 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: task.result() 2021-07-31T03:37:35.841 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:37:35.841 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: 2021-07-31T03:37:35.841 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:37:35.842 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: 2021-07-31T03:37:35.842 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:37:35.842 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:37:35.842 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:37:35.842 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:37:35.843 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:37:35.843 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:37:35.843 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:37:35.843 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:37:35.843 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:37:35.844 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:37:35.844 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:37:35.844 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:37:35.844 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:37:35.844 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:37:35.845 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:37:35.845 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:37:35.845 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:37:35.845 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:37:35.845 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: raise 2021-07-31T03:37:35.846 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:37:35.846 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:37:35.846 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:37:35.846 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:37:35.846 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:35 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:37:36.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:36 smithi154 conmon[34553]: cluster 2021-07-31T03:37:35.371012+0000 mgr.x (mgr.24542) 4734 : cluster [DBG] pgmap v5319: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:36.774 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:36 smithi045 conmon[39718]: cluster 2021-07-31T03:37:35.371012+0000 mgr.x (mgr.24542) 4734 : 2021-07-31T03:37:36.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:36 smithi045 conmon[39718]: cluster [DBG] pgmap v5319: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:36.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:36 smithi045 conmon[33655]: cluster 2021-07-31T03:37:35.371012+0000 mgr.x (mgr.24542) 4734 : cluster 2021-07-31T03:37:36.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:36 smithi045 conmon[33655]: [DBG] pgmap v5319: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:38.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:38 smithi154 conmon[34553]: cluster 2021-07-31T03:37:37.371529+0000 mgr.x (mgr.24542) 4735 : cluster [DBG] pgmap v5320: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:38.774 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:38 smithi045 conmon[33655]: cluster 2021-07-31T03:37:37.371529 2021-07-31T03:37:38.774 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:38 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 4735 : cluster [DBG] pgmap v5320: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:38.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:38 smithi045 conmon[39718]: cluster 2021-07-31T03:37:37.371529+0000 2021-07-31T03:37:38.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:38 smithi045 conmon[39718]: mgr.x (mgr.24542) 4735 : cluster [DBG] pgmap v5320: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:40.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:40 smithi154 conmon[34553]: cluster 2021-07-31T03:37:39.372164+0000 mgr.x (mgr.24542) 4736 : cluster [DBG] pgmap v5321: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:40.774 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:40 smithi045 conmon[33655]: cluster 2021-07-31T03:37:39.372164+0000 mgr.x (mgr.24542) 4736 2021-07-31T03:37:40.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:40 smithi045 conmon[33655]: : cluster [DBG] pgmap v5321: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:40.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:40 smithi045 conmon[39718]: cluster 2021-07-31T03:37:39.372164+0000 mgr.x (mgr. 2021-07-31T03:37:40.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:40 smithi045 conmon[39718]: 24542) 4736 : cluster [DBG] pgmap v5321: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:42.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:42 smithi154 conmon[34553]: cluster 2021-07-31T03:37:41.372598+0000 mgr.x (mgr.24542) 4737 : cluster [DBG] pgmap v5322: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:42.774 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:42 smithi045 conmon[33655]: cluster 2021-07-31T03:37:41.372598+0000 mgr.x (mgr.24542) 4737 : 2021-07-31T03:37:42.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:42 smithi045 conmon[33655]: cluster [DBG] pgmap v5322: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:42.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:42 smithi045 conmon[39718]: cluster 2021-07-31T03:37:41.372598+0000 mgr.x (mgr. 2021-07-31T03:37:42.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:42 smithi045 conmon[39718]: 24542) 4737 : cluster [DBG] pgmap v5322: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:44.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:44 smithi154 conmon[34553]: cluster 2021-07-31T03:37:43.372989+0000 mgr.x (mgr.24542) 4738 : cluster [DBG] pgmap v5323: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:44.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:44 smithi045 conmon[33655]: cluster 2021-07-31T03:37:43.372989+0000 mgr.x (mgr.24542) 4738 : cluster 2021-07-31T03:37:44.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:44 smithi045 conmon[33655]: [DBG] pgmap v5323: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:44.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:44 smithi045 conmon[39718]: cluster 2021-07-31T03 2021-07-31T03:37:44.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:44 smithi045 conmon[39718]: :37:43.372989+0000 mgr.x (mgr.24542) 4738 : cluster [DBG] pgmap v5323: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:46.182 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:37:46.463 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:37:46.463 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 112m ago 113m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:37:46.463 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 110m ago 113m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:37:46.464 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 110m ago 116m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:37:46.464 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 112m ago 118m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:37:46.464 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 112m ago 118m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:37:46.464 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 110m ago 116m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:37:46.464 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 112m ago 117m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:37:46.465 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 112m ago 113m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:37:46.465 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 110m ago 113m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:37:46.465 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 112m ago 116m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:37:46.465 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 112m ago 116m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:37:46.465 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 112m ago 115m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:37:46.466 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 112m ago 115m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:37:46.466 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 110m ago 115m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:37:46.466 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 110m ago 114m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:37:46.466 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 110m ago 114m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:37:46.466 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 110m ago 114m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:37:46.467 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 110m ago 113m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:37:46.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:46 smithi154 conmon[34553]: cluster 2021-07-31T03:37:45.373493+0000 mgr.x (mgr.24542) 4739 : cluster [DBG] pgmap v5324: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:46.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:46 smithi045 conmon[39718]: cluster 2021-07-31T03:37:45.373493+0000 2021-07-31T03:37:46.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:46 smithi045 conmon[39718]: mgr.x (mgr.24542) 4739 : cluster [DBG] pgmap v5324: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:46.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:46 smithi045 conmon[33655]: cluster 2021-07-31T03:37:45 2021-07-31T03:37:46.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:46 smithi045 conmon[33655]: .373493+0000 mgr.x (mgr.24542) 4739 : cluster [DBG] pgmap v5324: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:46.800 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:37:46.801 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:37:46.801 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:37:46.801 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:37:46.801 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:37:46.801 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:37:46.802 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:37:46.802 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:37:46.802 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:37:46.802 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:37:46.803 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:37:46.803 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:37:46.803 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:37:46.803 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:37:46.803 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:37:46.804 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:37:46.804 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:37:47.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:47 smithi154 conmon[34553]: audit 2021-07-31T03:37:46.171124+0000 mgr.x (mgr.24542) 4740 : audit [DBG] from='client.19902 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:37:47.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:47 smithi154 conmon[34553]: audit 2021-07-31T03:37:46.455881+0000 mgr.x (mgr.24542) 4741 : audit [DBG] from='client.29848 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:37:47.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:47 smithi154 conmon[34553]: audit 2021-07-31T03:37:46.801530+0000 mon.c (mon.1) 218 : audit [DBG] from='client.? 172.21.15.45:0/3316210447' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:37:47.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:47 smithi045 conmon[39718]: audit 2021-07-31T03:37:46.171124+0000 mgr.x ( 2021-07-31T03:37:47.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:47 smithi045 conmon[39718]: mgr.24542) 4740 : audit [DBG] from='client.19902 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:37:47.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:47 smithi045 conmon[39718]: audit 2021-07-31T03:37:46.455881+0000 mgr.x (mgr.24542) 4741 : audit 2021-07-31T03:37:47.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:47 smithi045 conmon[39718]: [DBG] from='client.29848 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:37:47.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:47 smithi045 conmon[39718]: audit 2021-07-31T03:37:46.801530+0000 mon.c 2021-07-31T03:37:47.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:47 smithi045 conmon[39718]: (mon.1) 218 : audit [DBG] from='client.? 172.21.15.45:0/3316210447' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:37:47.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:47 smithi045 conmon[33655]: audit 2021-07-31T03:37:46.171124+0000 mgr.x 2021-07-31T03:37:47.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:47 smithi045 conmon[33655]: (mgr.24542) 4740 : audit [DBG] from='client.19902 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:37:47.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:47 smithi045 conmon[33655]: audit 2021-07-31T03:37:46.455881+0000 mgr.x (mgr.24542) 4741 : audit [DBG] from='client.29848 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:37:47.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:47 smithi045 conmon[33655]: audit 2021 2021-07-31T03:37:47.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:47 smithi045 conmon[33655]: -07-31T03:37:46.801530+0000 mon.c (mon.1) 218 : audit [DBG] from='client.? 172.21.15.45:0/3316210447' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:37:48.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:48 smithi154 conmon[34553]: cluster 2021-07-31T03:37:47.374030+0000 mgr.x (mgr.24542) 4742 : cluster [DBG] pgmap v5325: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:48.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:48 smithi154 conmon[34553]: 2021-07-31T03:37:48.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:48 smithi045 conmon[33655]: cluster 2021-07-31T03:37:47.374030+0000 2021-07-31T03:37:48.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:48 smithi045 conmon[33655]: mgr.x (mgr.24542) 4742 : cluster [DBG] pgmap v5325: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:48.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:48 smithi045 conmon[39718]: cluster 2021-07-31T03:37:47.374030+0000 mgr.x (mgr 2021-07-31T03:37:48.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:48 smithi045 conmon[39718]: .24542) 4742 : cluster [DBG] pgmap v5325: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:50.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:50 smithi154 conmon[34553]: cluster 2021-07-31T03:37:49.374574+0000 mgr.x (mgr.24542) 4743 : cluster [DBG] pgmap v5326: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:50.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:50 smithi045 conmon[39718]: cluster 2021-07-31T03:37:49.374574+0000 mgr.x (mgr.24542) 4743 : cluster [DBG] pgmap v5326: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:50.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:50 smithi045 conmon[33655]: cluster 2021-07-31T03:37:49.374574+0000 mgr.x (mgr.24542) 4743 : cluster 2021-07-31T03:37:50.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:50 smithi045 conmon[33655]: [DBG] pgmap v5326: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:52.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:52 smithi154 conmon[34553]: cluster 2021-07-31T03:37:51.374957+0000 mgr.x (mgr.24542) 4744 : cluster [DBG] pgmap v5327: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:52.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:52 smithi045 conmon[39718]: cluster 2021-07-31T03:37:51.374957+0000 2021-07-31T03:37:52.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:52 smithi045 conmon[39718]: mgr.x (mgr.24542) 4744 : cluster [DBG] pgmap v5327: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:52.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:52 smithi045 conmon[33655]: cluster 2021-07-31T03:37:51.374957+0000 mgr.x (mgr. 2021-07-31T03:37:52.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:52 smithi045 conmon[33655]: 24542) 4744 : cluster [DBG] pgmap v5327: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:54.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:54 smithi154 conmon[34553]: cluster 2021-07-31T03:37:53.375351+0000 mgr.x (mgr.24542) 4745 : cluster [DBG] pgmap v5328: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:54.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:54 smithi045 conmon[33655]: cluster 2021-07-31T03:37:53.375351+0000 mgr.x (mgr.24542) 4745 : cluster 2021-07-31T03:37:54.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:54 smithi045 conmon[33655]: [DBG] pgmap v5328: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:54.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:54 smithi045 conmon[39718]: cluster 2021-07-31T03:37:53.375351+0000 mgr.x (mgr 2021-07-31T03:37:54.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:54 smithi045 conmon[39718]: .24542) 4745 : cluster [DBG] pgmap v5328: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:56.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:56 smithi154 conmon[34553]: cluster 2021-07-31T03:37:55.376006+0000 mgr.x (mgr.24542) 4746 : cluster [DBG] pgmap v5329: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:56.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:56 smithi045 conmon[33655]: cluster 2021-07-31T03:37:55.376006+0000 mgr.x (mgr.24542) 4746 : cluster [DBG] pgmap v5329: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:56.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:56 smithi045 conmon[39718]: cluster 2021-07-31T03:37:55.376006+0000 mgr.x (mgr 2021-07-31T03:37:56.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:56 smithi045 conmon[39718]: .24542) 4746 : cluster [DBG] pgmap v5329: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:58.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:37:58 smithi154 conmon[34553]: cluster 2021-07-31T03:37:57.376376+0000 mgr.x (mgr.24542) 4747 : cluster [DBG] pgmap v5330: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:58.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:58 smithi045 conmon[33655]: cluster 2021-07-31T03:37:57.376376+0000 mgr.x (mgr.24542) 2021-07-31T03:37:58.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:37:58 smithi045 conmon[33655]: 4747 : cluster [DBG] pgmap v5330: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:37:58.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:58 smithi045 conmon[39718]: cluster 2021-07-31T03:37:57 2021-07-31T03:37:58.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:37:58 smithi045 conmon[39718]: .376376+0000 mgr.x (mgr.24542) 4747 : cluster [DBG] pgmap v5330: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:00.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:00 smithi154 conmon[34553]: cluster 2021-07-31T03:37:59.376867+0000 mgr.x (mgr.24542) 4748 : cluster 2021-07-31T03:38:00.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:00 smithi154 conmon[34553]: [DBG] pgmap v5331: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:00.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:00 smithi045 conmon[39718]: cluster 2021-07-31T03:37:59.376867+0000 mgr.x (mgr.24542) 2021-07-31T03:38:00.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:00 smithi045 conmon[39718]: 4748 : cluster [DBG] pgmap v5331: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:00.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:00 smithi045 conmon[33655]: cluster 2021-07-31T03:37:59.376867+0000 mgr.x (mgr.24542) 4748 : cluster [DBG] 2021-07-31T03:38:00.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:00 smithi045 conmon[33655]: pgmap v5331: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:02.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:02 smithi154 conmon[34553]: cluster 2021-07-31T03:38:01.377244+0000 mgr.x (mgr.24542) 4749 : cluster [DBG] pgmap v5332: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:02.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:02 smithi045 conmon[33655]: cluster 2021-07-31T03:38:01.377244+0000 mgr.x (mgr.24542) 4749 : 2021-07-31T03:38:02.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:02 smithi045 conmon[33655]: cluster [DBG] pgmap v5332: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:02.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:02 smithi045 conmon[39718]: cluster 2021-07-31T03:38:01.377244+0000 mgr.x (mgr.24542) 2021-07-31T03:38:02.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:02 smithi045 conmon[39718]: 4749 : cluster [DBG] pgmap v5332: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:04.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:04 smithi154 conmon[34553]: cluster 2021-07-31T03:38:03.377771+0000 mgr.x (mgr.24542) 4750 : cluster [DBG] pgmap v5333: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:04.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:04 smithi045 conmon[39718]: cluster 2021-07-31T03:38:03.377771+0000 mgr.x (mgr 2021-07-31T03:38:04.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:04 smithi045 conmon[39718]: .24542) 4750 : cluster [DBG] pgmap v5333: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:04.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:04 smithi045 conmon[33655]: cluster 2021-07-31T03:38:03.377771+0000 mgr.x (mgr.24542) 4750 : cluster 2021-07-31T03:38:04.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:04 smithi045 conmon[33655]: [DBG] pgmap v5333: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:06.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:06 smithi154 conmon[34553]: cluster 2021-07-31T03:38:05.378241+0000 mgr.x (mgr.24542) 4751 : cluster [DBG] pgmap v5334: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:06.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:06 smithi045 conmon[33655]: cluster 2021-07-31T03:38:05.378241+0000 mgr.x (mgr.24542) 4751 : cluster [DBG] pgmap v5334: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:06.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:06 smithi045 conmon[39718]: cluster 2021-07-31T03:38:05.378241+0000 mgr.x (mgr.24542) 4751 : cluster [DBG] pgmap v5334: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:08.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:08 smithi154 conmon[34553]: cluster 2021-07-31T03:38:07.378792+0000 mgr.x (mgr.24542) 4752 : cluster [DBG] 2021-07-31T03:38:08.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:08 smithi154 conmon[34553]: pgmap v5335: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:08.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:08 smithi045 conmon[33655]: cluster 2021-07-31T03:38:07.378792+0000 mgr.x (mgr. 2021-07-31T03:38:08.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:08 smithi045 conmon[33655]: 24542) 4752 : cluster [DBG] pgmap v5335: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:08.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:08 smithi045 conmon[39718]: cluster 2021-07-31T03:38:07.378792+0000 mgr.x ( 2021-07-31T03:38:08.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:08 smithi045 conmon[39718]: mgr.24542) 4752 : cluster [DBG] pgmap v5335: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:10.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:10 smithi154 conmon[34553]: cluster 2021-07-31T03:38:09.379318+0000 mgr.x (mgr.24542) 4753 : cluster [DBG] pgmap v5336: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:10.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:10 smithi045 conmon[39718]: cluster 2021-07-31T03:38:09.379318+0000 mgr.x ( 2021-07-31T03:38:10.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:10 smithi045 conmon[39718]: mgr.24542) 4753 : cluster [DBG] pgmap v5336: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:10.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:10 smithi045 conmon[33655]: cluster 2021-07-31T03:38:09.379318+0000 mgr.x (mgr.24542) 4753 : 2021-07-31T03:38:10.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:10 smithi045 conmon[33655]: cluster [DBG] pgmap v5336: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:12.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:12 smithi154 conmon[34553]: cluster 2021-07-31T03:38:11.379709+0000 mgr.x (mgr.24542) 4754 : cluster [DBG] pgmap v5337: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:12.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:12 smithi045 conmon[33655]: cluster 2021-07-31T03:38:11.379709+0000 mgr.x (mgr.24542) 4754 2021-07-31T03:38:12.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:12 smithi045 conmon[33655]: : cluster [DBG] pgmap v5337: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:12.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:12 smithi045 conmon[39718]: cluster 2021-07-31T03:38:11.379709+0000 mgr.x (mgr.24542) 2021-07-31T03:38:12.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:12 smithi045 conmon[39718]: 4754 : cluster [DBG] pgmap v5337: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:14.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:14 smithi154 conmon[34553]: cluster 2021-07-31T03:38:13.380188+0000 mgr.x (mgr.24542) 4755 : cluster [DBG] pgmap v5338: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:14.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:14 smithi045 conmon[39718]: cluster 2021-07-31T03:38:13.380188+0000 mgr.x 2021-07-31T03:38:14.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:14 smithi045 conmon[39718]: (mgr.24542) 4755 : cluster [DBG] pgmap v5338: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:14.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:14 smithi045 conmon[33655]: cluster 2021-07-31T03:38:13.380188+0000 mgr.x (mgr.24542) 2021-07-31T03:38:14.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:14 smithi045 conmon[33655]: 4755 : cluster [DBG] pgmap v5338: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:16.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:16 smithi154 conmon[34553]: cluster 2021-07-31T03:38:15.380635+0000 mgr.x (mgr.24542) 4756 : cluster [DBG] pgmap v5339: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:16.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:16 smithi045 conmon[39718]: cluster 2021-07-31T03:38:15.380635+0000 mgr.x (mgr.24542) 4756 : cluster [DBG] pgmap v5339: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:16.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:16 smithi045 conmon[33655]: cluster 2021-07-31T03:38:15.380635+0000 2021-07-31T03:38:16.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:16 smithi045 conmon[33655]: mgr.x (mgr.24542) 4756 : cluster [DBG] pgmap v5339: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:17.114 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:38:17.391 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:38:17.391 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 113m ago 113m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:38:17.392 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 110m ago 113m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:38:17.392 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 110m ago 117m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:38:17.392 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 113m ago 119m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:38:17.392 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 113m ago 119m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:38:17.392 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 110m ago 117m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:38:17.393 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 113m ago 117m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:38:17.393 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 113m ago 114m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:38:17.393 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 110m ago 114m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:38:17.393 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 113m ago 116m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:38:17.393 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 113m ago 116m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:38:17.394 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 113m ago 116m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:38:17.394 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 113m ago 115m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:38:17.394 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 110m ago 115m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:38:17.394 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 110m ago 115m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:38:17.395 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 110m ago 114m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:38:17.395 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 110m ago 114m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:38:17.395 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 110m ago 114m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:38:17.726 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:38:17.727 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:38:17.727 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:38:17.727 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:38:17.727 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:38:17.728 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:38:17.728 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:38:17.728 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:38:17.728 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:38:17.728 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:38:17.729 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:38:17.729 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:38:17.729 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:38:17.729 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:38:17.730 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:38:17.730 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:38:17.730 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:38:18.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:18 smithi154 conmon[34553]: audit 2021-07-31T03:38:17.103536+0000 mgr.x (mgr.24542) 4757 : audit [DBG] from='client.19920 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:38:18.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:18 smithi154 conmon[34553]: cluster 2021-07-31T03:38:17.381168+0000 mgr.x (mgr.24542) 4758 : cluster [DBG] pgmap v5340: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:18.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:18 smithi154 conmon[34553]: audit 2021-07-31T03:38:17.384260+0000 mgr.x (mgr.24542) 4759 : audit [DBG] from='client.19926 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:38:18.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:18 smithi154 conmon[34553]: audit 2021-07-31T03:38:17.727273+0000 mon.c (mon.1) 219 2021-07-31T03:38:18.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:18 smithi154 conmon[34553]: : audit [DBG] from='client.? 172.21.15.45:0/346594382' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:38:18.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:18 smithi045 conmon[39718]: audit 2021-07-31T03:38:17.103536+0000 2021-07-31T03:38:18.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:18 smithi045 conmon[39718]: mgr.x (mgr.24542) 4757 : audit [DBG] from='client.19920 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:38:18.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:18 smithi045 conmon[39718]: cluster 2021-07-31T03:38:18.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:18 smithi045 conmon[39718]: 2021-07-31T03 2021-07-31T03:38:18.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:18 smithi045 conmon[39718]: :38:17.381168+0000 mgr.x ( 2021-07-31T03:38:18.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:18 smithi045 conmon[39718]: mgr.24542) 4758 : cluster [DBG] pgmap v5340: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:18.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:18 smithi045 conmon[39718]: audit 2021-07-31T03 2021-07-31T03:38:18.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:18 smithi045 conmon[39718]: :38:17.384260+0000 mgr.x (mgr.24542) 2021-07-31T03:38:18.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:18 smithi045 conmon[39718]: 4759 : audit [DBG] from='client.19926 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:38:18.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:18 smithi045 conmon[39718]: audit 2021-07-31T03:38:18.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:18 smithi045 conmon[39718]: 2021-07-31T03:38 2021-07-31T03:38:18.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:18 smithi045 conmon[39718]: :17.727273+0000 mon.c (mon. 2021-07-31T03:38:18.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:18 smithi045 conmon[39718]: 1) 219 : audit [DBG] from='client.? 172.21.15.45:0/346594382' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:38:18.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:18 smithi045 conmon[33655]: audit 2021-07-31T03:38:17.103536+0000 mgr.x (mgr.24542) 4757 2021-07-31T03:38:18.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:18 smithi045 conmon[33655]: : audit [DBG] from='client.19920 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:38:18.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:18 smithi045 conmon[33655]: cluster 2021-07-31T03:38:17.381168+0000 mgr.x (mgr.24542) 4758 : cluster [DBG] pgmap v5340: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:18.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:18 smithi045 conmon[33655]: audit 2021-07-31T03 2021-07-31T03:38:18.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:18 smithi045 conmon[33655]: :38:17.384260+0000 mgr.x (mgr.24542) 4759 : audit [DBG] from='client.19926 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:38:18.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:18 smithi045 conmon[33655]: audit 2021-07-31T03:38: 2021-07-31T03:38:18.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:18 smithi045 conmon[33655]: 17.727273+0000 mon.c (mon.1) 219 : audit [DBG] 2021-07-31T03:38:18.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:18 smithi045 conmon[33655]: from='client.? 172.21.15.45:0/346594382' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:38:20.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:20 smithi154 conmon[34553]: cluster 2021-07-31T03:38:19.381740+0000 mgr.x (mgr.24542) 4760 : cluster [DBG] pgmap v5341: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:20.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:20 smithi045 conmon[33655]: cluster 2021-07-31T03:38:19.381740+0000 mgr.x (mgr.24542) 4760 : cluster [DBG] 2021-07-31T03:38:20.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:20 smithi045 conmon[33655]: pgmap v5341: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:20.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:20 smithi045 conmon[39718]: cluster 2021-07-31T03:38:19.381740+0000 2021-07-31T03:38:20.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:20 smithi045 conmon[39718]: mgr.x (mgr.24542) 4760 : cluster [DBG] pgmap v5341: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:22.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:22 smithi154 conmon[34553]: cluster 2021-07-31T03:38:21.382148+0000 mgr.x (mgr.24542) 4761 : cluster [DBG] pgmap v5342: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:22.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:22 smithi045 conmon[33655]: cluster 2021-07-31T03:38:21.382148+0000 2021-07-31T03:38:22.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:22 smithi045 conmon[33655]: mgr.x (mgr.24542) 4761 : cluster [DBG] pgmap v5342: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:22.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:22 smithi045 conmon[39718]: cluster 2021-07-31T03:38:21.382148+0000 mgr.x 2021-07-31T03:38:22.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:22 smithi045 conmon[39718]: (mgr.24542) 4761 : cluster [DBG] pgmap v5342: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:24.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:24 smithi154 conmon[34553]: cluster 2021-07-31T03:38:23.382523+0000 mgr.x (mgr.24542) 4762 : cluster [DBG] pgmap v5343: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:24.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:24 smithi045 conmon[33655]: cluster 2021-07-31T03:38:23.382523+0000 mgr.x (mgr.24542) 4762 : cluster [DBG] pgmap v5343: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:24.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:24 smithi045 conmon[33655]: 2021-07-31T03:38:24.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:24 smithi045 conmon[39718]: cluster 2021-07-31T03:38:23.382523+0000 mgr.x (mgr. 2021-07-31T03:38:24.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:24 smithi045 conmon[39718]: 24542) 4762 : cluster [DBG] pgmap v5343: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:26.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:26 smithi154 conmon[34553]: cluster 2021-07-31T03:38:25.382928+0000 mgr.x (mgr.24542) 4763 : cluster [DBG] pgmap v5344: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:26.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:26 smithi045 conmon[33655]: cluster 2021-07-31T03:38:25.382928+0000 mgr.x (mgr.24542) 4763 : 2021-07-31T03:38:26.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:26 smithi045 conmon[33655]: cluster [DBG] pgmap v5344: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:26.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:26 smithi045 conmon[39718]: cluster 2021-07-31T03:38:25.382928+0000 mgr.x ( 2021-07-31T03:38:26.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:26 smithi045 conmon[39718]: mgr.24542) 4763 : cluster [DBG] pgmap v5344: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:28.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:28 smithi154 conmon[34553]: cluster 2021-07-31T03:38:27.383341+0000 mgr.x (mgr.24542) 4764 : cluster [DBG] 2021-07-31T03:38:28.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:28 smithi154 conmon[34553]: pgmap v5345: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:28.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:28 smithi154 conmon[34553]: audit 2021-07-31T03:38:28.377129+0000 mon.b (mon.2) 630 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:38:28.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:28 smithi154 conmon[34553]: audit 2021-07-31T03:38:28.377370+0000 mon.b (mon.2) 631 : audit 2021-07-31T03:38:28.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:28 smithi154 conmon[34553]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:38:28.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:28 smithi154 conmon[34553]: audit 2021-07-31T03:38:28.377383+0000 mon.a (mon. 2021-07-31T03:38:28.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:28 smithi154 conmon[34553]: 0) 3338 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:38:28.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:28 smithi154 conmon[34553]: audit 2021-07-31T03:38:28.377783+0000 mon.a (mon.0) 3339 : audit [INF] 2021-07-31T03:38:28.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:28 smithi154 conmon[34553]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:38:28.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:28 smithi045 conmon[33655]: cluster 2021-07-31T03:38:27.383341+0000 mgr.x (mgr.24542 2021-07-31T03:38:28.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:28 smithi045 conmon[33655]: ) 4764 : cluster [DBG] pgmap v5345: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:28.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:28 smithi045 conmon[33655]: audit 2021-07-31T03:38:28.377129+0000 mon.b (mon.2) 630 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:38:28.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:28 smithi045 conmon[33655]: audit 2021 2021-07-31T03:38:28.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:28 smithi045 conmon[33655]: -07-31T03:38:28.377370+0000 mon.b (mon 2021-07-31T03:38:28.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:28 smithi045 conmon[33655]: .2) 631 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:38:28.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:28 smithi045 conmon[33655]: 2021-07-31T03:38:28.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:28 smithi045 conmon[33655]: audit 2021-07-31T03:38 2021-07-31T03:38:28.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:28 smithi045 conmon[33655]: :28.377383+0000 mon.a (mon.0) 3338 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:38:28.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:28 smithi045 conmon[33655]: audit 2021-07-31T03:38:28.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:28 smithi045 conmon[33655]: 2021-07-31T03: 2021-07-31T03:38:28.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:28 smithi045 conmon[33655]: 38:28.377783+0000 mon.a 2021-07-31T03:38:28.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:28 smithi045 conmon[33655]: (mon.0) 3339 : audit 2021-07-31T03:38:28.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:28 smithi045 conmon[33655]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:38:28.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:28 smithi045 conmon[39718]: cluster 2021-07-31T03:38:27 2021-07-31T03:38:28.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:28 smithi045 conmon[39718]: .383341+0000 mgr.x (mgr.24542) 4764 : cluster [DBG] pgmap v5345: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:28.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:28 smithi045 conmon[39718]: audit 2021-07-31T03:38:28.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:28 smithi045 conmon[39718]: 2021-07-31T03: 2021-07-31T03:38:28.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:28 smithi045 conmon[39718]: 38:28.377129+0000 mon.b 2021-07-31T03:38:28.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:28 smithi045 conmon[39718]: (mon.2) 630 : audit 2021-07-31T03:38:28.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:28 smithi045 conmon[39718]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:38:28.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:28 smithi045 conmon[39718]: audit 2021- 2021-07-31T03:38:28.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:28 smithi045 conmon[39718]: 07-31T03:38:28.377370 2021-07-31T03:38:28.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:28 smithi045 conmon[39718]: +0000 mon.b (mon.2) 631 2021-07-31T03:38:28.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:28 smithi045 conmon[39718]: : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:38:28.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:28 smithi045 conmon[39718]: audit 2021 2021-07-31T03:38:28.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:28 smithi045 conmon[39718]: -07-31T03:38:28. 2021-07-31T03:38:28.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:28 smithi045 conmon[39718]: 377383+0000 mon.a (mon. 2021-07-31T03:38:28.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:28 smithi045 conmon[39718]: 0) 3338 : audit [INF] 2021-07-31T03:38:28.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:28 smithi045 conmon[39718]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:38:28.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:28 smithi045 conmon[39718]: audit 2021 2021-07-31T03:38:28.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:28 smithi045 conmon[39718]: -07-31T03:38: 2021-07-31T03:38:28.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:28 smithi045 conmon[39718]: 28.377783+0000 mon.a (mon 2021-07-31T03:38:28.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:28 smithi045 conmon[39718]: .0) 3339 : audit [INF] 2021-07-31T03:38:28.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:28 smithi045 conmon[39718]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:38:30.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:30 smithi045 conmon[39718]: cluster 2021-07-31T03:38:29.383853+0000 mgr.x (mgr.24542) 4765 : cluster 2021-07-31T03:38:30.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:30 smithi045 conmon[39718]: [DBG] pgmap v5346: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:30.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:30 smithi045 conmon[33655]: cluster 2021-07-31T03:38:29.383853+0000 mgr.x (mgr 2021-07-31T03:38:30.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:30 smithi045 conmon[33655]: .24542) 4765 : cluster [DBG] pgmap v5346: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:31.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:30 smithi154 conmon[34553]: cluster 2021-07-31T03:38:29.383853+0000 mgr.x (mgr.24542) 4765 : cluster 2021-07-31T03:38:31.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:30 smithi154 conmon[34553]: [DBG] pgmap v5346: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:32.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:32 smithi045 conmon[39718]: cluster 2021-07-31T03:38:31.384235+0000 mgr.x (mgr 2021-07-31T03:38:32.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:32 smithi045 conmon[39718]: .24542) 4766 : cluster [DBG] pgmap v5347: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:32.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:32 smithi045 conmon[33655]: cluster 2021-07-31T03:38:31.384235+0000 mgr.x (mgr.24542 2021-07-31T03:38:32.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:32 smithi045 conmon[33655]: ) 4766 : cluster [DBG] pgmap v5347: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:33.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:32 smithi154 conmon[34553]: cluster 2021-07-31T03:38:31.384235+0000 mgr.x (mgr.24542) 4766 : cluster [DBG] 2021-07-31T03:38:33.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:32 smithi154 conmon[34553]: pgmap v5347: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:34.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[34553]: cluster 2021-07-31T03:38:33.384679+0000 mgr.x (mgr.24542) 4767 : cluster 2021-07-31T03:38:34.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[34553]: [DBG] pgmap v5348: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:34.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[34553]: audit 2021-07-31T03:38:34.276864+0000 mon.b (mon.2) 632 : audit 2021-07-31T03:38:34.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[34553]: [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:38:34.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[34553]: audit 2021-07 2021-07-31T03:38:34.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[34553]: -31T03:38:34.278343+0000 mon.b (mon.2) 633 2021-07-31T03:38:34.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[34553]: : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:38:34.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[34553]: audit 2021-07- 2021-07-31T03:38:34.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[34553]: 31T03:38:34.279468+0000 mon.b ( 2021-07-31T03:38:34.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[34553]: mon.2) 634 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:38:34.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: debug 2021-07-31T03:38:34.322+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:38:34.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:38:34.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:38:34.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: yield 2021-07-31T03:38:34.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:38:34.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:38:34.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:38:34.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:38:34.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:38:34.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:38:34.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:38:34.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:38:34.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:38:34.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: task.result() 2021-07-31T03:38:34.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:38:34.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: debug 2021-07-31T03:38:34.334+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:38:34.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:38:34.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:38:34.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: yield 2021-07-31T03:38:34.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:38:34.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:38:34.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:38:34.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:38:34.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:38:34.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:38:34.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:38:34.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:38:34.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:38:34.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: task.result() 2021-07-31T03:38:34.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:38:34.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: debug 2021-07-31T03:38:34.359+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:38:34.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:38:34.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:38:34.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: yield 2021-07-31T03:38:34.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:38:34.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:38:34.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:38:34.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:38:34.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:38:34.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:38:34.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:38:34.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:38:34.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:38:34.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: task.result() 2021-07-31T03:38:34.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:38:34.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: debug 2021-07-31T03:38:34.360+0000 7fcb9295f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:38:34.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:38:34.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:38:34.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: yield 2021-07-31T03:38:34.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:38:34.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:38:34.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:38:34.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:38:34.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:38:34.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:38:34.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:38:34.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:38:34.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:38:34.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: task.result() 2021-07-31T03:38:34.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:38:34.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: 2021-07-31T03:38:34.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:38:34.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: 2021-07-31T03:38:34.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:38:34.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:38:34.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:38:34.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:38:34.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:38:34.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:38:34.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:38:34.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:38:34.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:38:34.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:38:34.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:38:34.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:38:34.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:38:34.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:38:34.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:38:34.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:38:34.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:38:34.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:38:34.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: raise 2021-07-31T03:38:34.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:38:34.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:38:34.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:38:34.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:38:34.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:38:34.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: debug 2021-07-31T03:38:34.369+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:38:34.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:38:34.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:38:34.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: yield 2021-07-31T03:38:34.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:38:34.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:38:34.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:38:34.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:38:34.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:38:34.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:38:34.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:38:34.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:38:34.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:38:34.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: task.result() 2021-07-31T03:38:34.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:38:34.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: debug 2021-07-31T03:38:34.369+0000 7fcb93160700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:38:34.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:38:34.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:38:34.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: yield 2021-07-31T03:38:34.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:38:34.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:38:34.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:38:34.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:38:34.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:38:34.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:38:34.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:38:34.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:38:34.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:38:34.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: task.result() 2021-07-31T03:38:34.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:38:34.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: 2021-07-31T03:38:34.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:38:34.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: 2021-07-31T03:38:34.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:38:34.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:38:34.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:38:34.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:38:34.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:38:34.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:38:34.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:38:34.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:38:34.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:38:34.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:38:34.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:38:34.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:38:34.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:38:34.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:38:34.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:38:34.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:38:34.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:38:34.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:38:34.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: raise 2021-07-31T03:38:34.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:38:34.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:38:34.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:38:34.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:38:34.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:38:34 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:38:34.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:34 smithi045 conmon[39718]: cluster 2021-07-31T03:38:33.384679+0000 mgr.x (mgr.24542) 2021-07-31T03:38:34.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:34 smithi045 conmon[39718]: 4767 : cluster [DBG] pgmap v5348: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:34.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:34 smithi045 conmon[39718]: audit 2021-07-31T03:38:34.276864+0000 mon.b (mon.2 2021-07-31T03:38:34.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:34 smithi045 conmon[39718]: ) 632 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:38:34.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:34 smithi045 conmon[39718]: audit 2021-07-31T03:38:34.278343 2021-07-31T03:38:34.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:34 smithi045 conmon[39718]: +0000 mon.b (mon.2) 633 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:38:34.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:34 smithi045 conmon[39718]: audit 2021-07-31T03:38:34.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:34 smithi045 conmon[39718]: 2021-07-31T03:38:34.279468+0000 mon.b (mon.2) 634 : audit 2021-07-31T03:38:34.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:34 smithi045 conmon[39718]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:38:34.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:34 smithi045 conmon[33655]: cluster 2021-07-31T03:38:33.384679+0000 mgr.x (mgr.24542 2021-07-31T03:38:34.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:34 smithi045 conmon[33655]: ) 4767 : cluster [DBG] pgmap v5348: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:34.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:34 smithi045 conmon[33655]: audit 2021-07-31T03:38:34.276864+0000 2021-07-31T03:38:34.788 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:34 smithi045 conmon[33655]: mon.b (mon.2) 632 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:38:34.788 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:34 smithi045 conmon[33655]: audit 2021-07-31T03:38:34.788 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:34 smithi045 conmon[33655]: 2021-07-31T03:38:34.278343+0000 mon.b (mon.2) 633 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:38:34.788 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:34 smithi045 conmon[33655]: audit 2021-07-31T03:38:34.279468+0000 mon.b (mon.2) 2021-07-31T03:38:34.789 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:34 smithi045 conmon[33655]: 634 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:38:35.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:38:34.280234+0000 mgr.x (mgr.24542) 4768 : cephadm 2021-07-31T03:38:35.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:38:35.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:38:34.323596+0000 mgr.x (mgr.24542) 4769 : cephadm [ERR] 2021-07-31T03:38:35.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:38:35.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:38:35.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: yield 2021-07-31T03:38:35.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:38:35.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:38:35.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:38:35.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:38:35.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:38:35.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:38:35.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:38:35.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:38:35.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:38:35.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: task.result() 2021-07-31T03:38:35.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:38:35.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:38:34.324083+0000 mgr.x (mgr.24542) 4770 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:38:35.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:38:34.335062+0000 mgr.x (mgr.24542) 4771 : cephadm [ERR] 2021-07-31T03:38:35.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:38:35.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:38:35.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: yield 2021-07-31T03:38:35.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:38:35.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:38:35.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:38:35.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:38:35.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:38:35.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:38:35.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:38:35.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:38:35.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:38:35.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: task.result() 2021-07-31T03:38:35.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:38:35.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:38 2021-07-31T03:38:35.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: :34.335962+0000 mgr.x (mgr.24542) 4772 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:38:35.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:38:34.360550+0000 mgr.x (mgr.24542) 4773 : cephadm [ERR] 2021-07-31T03:38:35.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:38:35.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:38:35.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: yield 2021-07-31T03:38:35.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:38:35.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:38:35.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:38:35.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:38:35.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:38:35.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:38:35.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:38:35.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:38:35.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:38:35.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: task.result() 2021-07-31T03:38:35.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:38:35.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:38:34.361030+0000 mgr.x (mgr.24542) 4774 2021-07-31T03:38:35.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:38:35.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:38:35.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:38:35.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: yield 2021-07-31T03:38:35.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:38:35.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:38:35.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:38:35.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:38:35.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:38:35.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:38:35.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:38:35.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:38:35.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:38:35.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: task.result() 2021-07-31T03:38:35.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:38:35.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: 2021-07-31T03:38:35.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:38:35.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: 2021-07-31T03:38:35.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:38:35.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:38:35.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:38:35.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:38:35.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:38:35.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:38:35.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:38:35.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:38:35.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:38:35.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:38:35.795 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:38:34.280234+0000 mgr.x (mgr.24542) 4768 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:38:35.795 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:38:35.795 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: 2021-07-31T03:38:34.323596+0000 mgr.x (mgr.24542) 4769 : cephadm [ERR] 2021-07-31T03:38:35.796 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:38:35.796 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:38:35.796 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: yield 2021-07-31T03:38:35.796 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:38:35.797 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:38:35.797 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:38:35.797 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:38:35.797 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:38:35.798 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:38:35.798 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:38:35.798 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:38:35.798 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:38:35.799 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: task.result() 2021-07-31T03:38:35.799 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:38:35.800 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:38:34.324083+0000 mgr.x (mgr.24542) 4770 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:38:35.800 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:38:35.800 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: 2021-07-31T03:38:34.335062+0000 mgr.x (mgr.24542) 4771 : cephadm [ERR] 2021-07-31T03:38:35.800 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:38:35.801 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:38:35.801 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: yield 2021-07-31T03:38:35.802 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:38:35.802 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:38:35.803 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:38:35.803 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:38:35.803 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:38:35.803 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:38:35.804 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:38:35.804 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:38:35.804 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:38:35.804 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: task.result() 2021-07-31T03:38:35.805 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:38:35.805 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:38:34.335962+0000 mgr.x (mgr.24542) 4772 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:38:35.805 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:38:34.360550+0000 mgr.x (mgr.24542) 4773 : cephadm [ERR] 2021-07-31T03:38:35.805 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:38:35.806 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:38:35.806 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:38:35.806 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:38:35.807 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:38:35.807 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:38:35.808 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:38:35.808 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:38:35.808 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:38:35.808 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:38:35.809 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: raise 2021-07-31T03:38:35.809 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:38:35.809 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:38:35.809 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:38:35.810 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:38:35.810 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:38:35.810 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:38:34.370374+0000 mgr.x (mgr.24542) 4775 : cephadm [ERR] 2021-07-31T03:38:35.810 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:38:35.811 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:38:35.811 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: yield 2021-07-31T03:38:35.811 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:38:35.811 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:38:35.812 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:38:35.812 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:38:35.812 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:38:35.812 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:38:35.812 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:38:35.813 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:38:35.813 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:38:35.813 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: task.result() 2021-07-31T03:38:35.813 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:38:35.814 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:38:34.370775+0000 mgr.x (mgr.24542) 4776 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:38:35.814 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:38:35.814 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:38:35.814 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: yield 2021-07-31T03:38:35.814 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:38:35.815 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:38:35.815 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:38:35.815 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:38:35.815 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:38:35.816 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:38:35.816 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: yield 2021-07-31T03:38:35.816 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:38:35.817 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:38:35.817 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:38:35.818 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:38:35.818 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:38:35.818 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:38:35.818 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:38:35.818 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:38:35.819 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:38:35.820 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: task.result() 2021-07-31T03:38:35.820 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:38:35.820 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:38:34.361030+0000 2021-07-31T03:38:35.820 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: mgr.x (mgr.24542) 4774 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:38:35.820 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:38:35.821 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:38:35.821 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: yield 2021-07-31T03:38:35.821 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:38:35.821 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:38:35.822 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:38:35.822 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:38:35.822 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:38:35.822 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:38:35.822 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:38:35.823 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:38:35.823 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:38:35.823 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: task.result() 2021-07-31T03:38:35.823 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:38:35.824 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: 2021-07-31T03:38:35.824 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:38:35.824 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: 2021-07-31T03:38:35.824 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:38:35.825 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:38:35.825 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:38:35.825 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:38:35.825 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:38:35.825 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:38:35.826 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:38:35.826 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:38:35.826 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:38:35.826 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:38:35.827 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:38:35.827 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:38:35.827 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:38:35.828 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: task.result() 2021-07-31T03:38:35.828 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:38:35.828 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: 2021-07-31T03:38:35.828 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:38:35.829 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: 2021-07-31T03:38:35.829 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:38:35.829 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:38:35.829 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:38:35.830 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:38:35.830 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:38:35.830 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:38:35.830 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:38:35.830 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:38:35.831 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:38:35.831 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:38:35.831 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:38:35.831 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:38:35.832 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:38:35.832 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:38:35.833 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:38:35.833 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:38:35.833 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:38:35.833 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:38:35.833 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: raise 2021-07-31T03:38:35.834 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:38:35.834 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:38:35.834 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:38:35.834 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:38:35.835 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:38:35.835 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:38:35.835 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:38:35.836 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:38:35.836 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:38:35.836 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:38:35.836 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:38:35.837 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:38:35.837 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:38:35.837 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: raise 2021-07-31T03:38:35.837 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:38:35.837 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:38:35.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:38:35.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:38:35.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:38:35.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:38:34.370374+0000 mgr.x (mgr.24542) 4775 : cephadm [ERR] 2021-07-31T03:38:35.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:38:35.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:38:35.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: yield 2021-07-31T03:38:35.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:38:35.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:38:35.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:38:35.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:38:35.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:38:35.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:38:35.841 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:38:35.841 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:38:35.841 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:38:35.841 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: task.result() 2021-07-31T03:38:35.842 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:38:35.842 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:38:34.370775+0000 mgr.x (mgr.24542) 4776 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:38:35.842 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:38:35.842 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:38:35.842 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: yield 2021-07-31T03:38:35.843 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:38:35.843 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:38:35.843 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:38:35.843 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:38:35.844 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:38:35.844 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:38:35.844 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:38:35.845 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:38:35.845 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:38:35.845 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: task.result() 2021-07-31T03:38:35.845 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:38:35.846 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: 2021-07-31T03:38:35.846 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:38:35.846 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: 2021-07-31T03:38:35.846 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:38:35.847 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:38:35.847 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:38:35.847 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:38:35.847 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:38:35.848 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:38:35.848 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:38:35.848 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:38:35.848 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:38:35.848 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:38:35.849 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:38:35.849 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:38:35.849 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:38:35.849 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:38:35.849 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:38:35.850 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:38:35.850 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:38:35.850 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:38:35.851 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: raise 2021-07-31T03:38:35.851 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:38:35.851 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:38:35.851 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:38:35.851 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:38:35.852 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:35 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:38:36.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:38:34.280234+0000 mgr.x (mgr.24542) 4768 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:38:36.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: 2021-07-31T03:38:36.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:38:34.323596+0000 mgr.x (mgr.24542) 4769 : cephadm [ERR] 2021-07-31T03:38:36.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:38:36.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:38:36.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: yield 2021-07-31T03:38:36.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:38:36.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:38:36.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:38:36.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:38:36.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:38:36.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:38:36.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:38:36.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:38:36.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:38:36.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: task.result() 2021-07-31T03:38:36.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:38:36.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:38:34.324083+0000 mgr.x (mgr.24542) 2021-07-31T03:38:36.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: 4770 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:38:36.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:38:34.335062+0000 mgr.x (mgr.24542) 4771 : cephadm [ERR] 2021-07-31T03:38:36.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:38:36.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:38:36.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: yield 2021-07-31T03:38:36.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:38:36.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:38:36.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:38:36.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:38:36.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:38:36.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:38:36.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:38:36.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:38:36.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:38:36.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: task.result() 2021-07-31T03:38:36.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:38:36.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:38:34.335962+0000 mgr.x (mgr.24542) 4772 : cephadm 2021-07-31T03:38:36.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:38:36.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:38:34.360550+0000 mgr.x (mgr.24542) 4773 : cephadm [ERR] 2021-07-31T03:38:36.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:38:36.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:38:36.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: yield 2021-07-31T03:38:36.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:38:36.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:38:36.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:38:36.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:38:36.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:38:36.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:38:36.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:38:36.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:38:36.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:38:36.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: task.result() 2021-07-31T03:38:36.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:38:36.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:38:34.361030+0000 mgr.x (mgr.24542) 4774 : cephadm 2021-07-31T03:38:36.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:38:36.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:38:36.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:38:36.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: yield 2021-07-31T03:38:36.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:38:36.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:38:36.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:38:36.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:38:36.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:38:36.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:38:36.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:38:36.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:38:36.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:38:36.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: task.result() 2021-07-31T03:38:36.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:38:36.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: 2021-07-31T03:38:36.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:38:36.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: 2021-07-31T03:38:36.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:38:36.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:38:36.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:38:36.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:38:36.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:38:36.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:38:36.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:38:36.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:38:36.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:38:36.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:38:36.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:38:36.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:38:36.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:38:36.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:38:36.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:38:36.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:38:36.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:38:36.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:38:36.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: raise 2021-07-31T03:38:36.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:38:36.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:38:36.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:38:36.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:38:36.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:38:36.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:38:34.370374+0000 mgr.x (mgr.24542) 4775 : cephadm [ERR] 2021-07-31T03:38:36.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:38:36.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:38:36.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: yield 2021-07-31T03:38:36.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:38:36.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:38:36.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:38:36.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:38:36.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:38:36.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:38:36.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:38:36.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:38:36.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:38:36.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: task.result() 2021-07-31T03:38:36.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:38:36.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:38:34.370775+0000 mgr.x ( 2021-07-31T03:38:36.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: mgr.24542) 4776 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:38:36.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:38:36.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:38:36.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: yield 2021-07-31T03:38:36.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:38:36.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:38:36.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:38:36.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:38:36.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:38:36.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:38:36.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:38:36.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:38:36.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:38:36.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: task.result() 2021-07-31T03:38:36.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:38:36.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: 2021-07-31T03:38:36.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:38:36.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: 2021-07-31T03:38:36.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:38:36.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:38:36.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:38:36.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:38:36.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:38:36.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:38:36.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:38:36.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:38:36.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:38:36.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:38:36.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:38:36.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:38:36.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:38:36.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:38:36.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:38:36.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:38:36.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:38:36.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:38:36.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: raise 2021-07-31T03:38:36.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:38:36.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:38:36.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:38:36.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:38:36.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:35 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:38:36.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:36 smithi045 conmon[33655]: cluster 2021-07- 2021-07-31T03:38:36.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:36 smithi045 conmon[33655]: 31T03:38:35.385136+0000 mgr.x (mgr.24542) 4777 : cluster [DBG] pgmap v5349: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:36.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:36 smithi045 conmon[39718]: cluster 2021-07-31T03:38:35.385136+0000 mgr.x (mgr.24542) 4777 2021-07-31T03:38:36.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:36 smithi045 conmon[39718]: : cluster [DBG] pgmap v5349: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:37.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:36 smithi154 conmon[34553]: cluster 2021-07-31T03:38:35.385136+0000 mgr.x (mgr.24542) 4777 : cluster [DBG] 2021-07-31T03:38:37.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:36 smithi154 conmon[34553]: pgmap v5349: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:38.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:38 smithi045 conmon[33655]: cluster 2021-07-31T03:38:37.385656+0000 mgr.x (mgr.24542) 4778 : cluster [DBG] pgmap v5350: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:38.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:38 smithi045 conmon[39718]: cluster 2021-07-31T03:38:37.385656+0000 mgr.x (mgr.24542) 4778 : cluster 2021-07-31T03:38:38.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:38 smithi045 conmon[39718]: [DBG] pgmap v5350: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:39.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:38 smithi154 conmon[34553]: cluster 2021-07-31T03:38:37.385656+0000 mgr.x (mgr.24542) 4778 : cluster [DBG] pgmap v5350: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:39.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:38 smithi154 conmon[34553]: 2021-07-31T03:38:41.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:40 smithi154 conmon[34553]: cluster 2021-07-31T03:38:39.386113+0000 mgr.x (mgr.24542) 4779 : cluster 2021-07-31T03:38:41.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:40 smithi154 conmon[34553]: [DBG] pgmap v5351: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:41.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:40 smithi045 conmon[33655]: cluster 2021-07-31T03:38:39.386113+0000 mgr.x (mgr.24542) 2021-07-31T03:38:41.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:40 smithi045 conmon[33655]: 4779 : cluster [DBG] pgmap v5351: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:41.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:40 smithi045 conmon[39718]: cluster 2021-07-31T03:38:39.386113+0000 mgr.x (mgr.24542) 4779 : 2021-07-31T03:38:41.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:40 smithi045 conmon[39718]: cluster [DBG] pgmap v5351: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:43.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:42 smithi154 conmon[34553]: cluster 2021-07-31T03:38:41.386491+0000 mgr.x (mgr.24542) 4780 : cluster [DBG] pgmap v5352: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:43.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:42 smithi045 conmon[33655]: cluster 2021-07-31T03:38:41.386491+0000 mgr.x (mgr.24542) 4780 : 2021-07-31T03:38:43.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:42 smithi045 conmon[33655]: cluster [DBG] pgmap v5352: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:43.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:42 smithi045 conmon[39718]: cluster 2021-07-31T03:38:41.386491+0000 mgr.x ( 2021-07-31T03:38:43.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:42 smithi045 conmon[39718]: mgr.24542) 4780 : cluster [DBG] pgmap v5352: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:45.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:44 smithi154 conmon[34553]: cluster 2021-07-31T03:38:43.386903+0000 mgr.x (mgr.24542) 4781 : cluster [DBG] pgmap v5353: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:45.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:44 smithi045 conmon[39718]: cluster 2021-07-31T03:38:43.386903+0000 mgr.x (mgr.24542) 4781 : cluster [DBG] pgmap v5353: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:45.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:44 smithi045 conmon[33655]: cluster 2021-07-31T03:38:43.386903+0000 mgr.x (mgr.24542) 4781 : 2021-07-31T03:38:45.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:44 smithi045 conmon[33655]: cluster [DBG] pgmap v5353: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:47.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:46 smithi154 conmon[34553]: cluster 2021-07-31T03:38:45.387458+0000 mgr.x (mgr.24542) 4782 : cluster [DBG] pgmap v5354: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:47.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:46 smithi045 conmon[39718]: cluster 2021-07-31T03:38:45.387458+0000 mgr.x ( 2021-07-31T03:38:47.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:46 smithi045 conmon[39718]: mgr.24542) 4782 : cluster [DBG] pgmap v5354: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:47.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:46 smithi045 conmon[33655]: cluster 2021-07-31T03:38:45.387458+0000 mgr.x (mgr.24542) 4782 : cluster [DBG] pgmap v5354: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:47.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:46 smithi045 conmon[33655]: 2021-07-31T03:38:48.036 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:38:48.314 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:38:48.315 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 113m ago 114m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:38:48.315 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 111m ago 114m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:38:48.315 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 111m ago 117m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:38:48.315 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 113m ago 119m - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:38:48.316 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 113m ago 119m - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:38:48.316 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 111m ago 117m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:38:48.316 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 113m ago 118m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:38:48.316 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 113m ago 114m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:38:48.316 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 111m ago 114m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:38:48.317 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 113m ago 117m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:38:48.317 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 113m ago 117m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:38:48.317 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 113m ago 116m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:38:48.317 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 113m ago 116m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:38:48.317 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 111m ago 116m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:38:48.318 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 111m ago 115m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:38:48.318 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 111m ago 115m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:38:48.318 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 111m ago 115m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:38:48.318 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 111m ago 114m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:38:48.653 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:38:48.653 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:38:48.653 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:38:48.654 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:38:48.654 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:38:48.654 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:38:48.654 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:38:48.654 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:38:48.655 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:38:48.655 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:38:48.655 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:38:48.655 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:38:48.655 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:38:48.656 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:38:48.656 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:38:48.656 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:38:48.656 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:38:49.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:48 smithi154 conmon[34553]: cluster 2021-07-31T03:38:47.387958+0000 mgr.x (mgr.24542) 4783 : cluster [DBG] pgmap v5355: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:49.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:48 smithi154 conmon[34553]: audit 2021-07-31T03:38:48.026047+0000 mgr.x (mgr.24542) 4784 : audit [DBG] from='client.19938 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:38:49.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:48 smithi045 conmon[33655]: cluster 2021-07-31T03:38:47.387958+0000 2021-07-31T03:38:49.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:48 smithi045 conmon[33655]: mgr.x (mgr.24542) 4783 : cluster [DBG] pgmap v5355: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:49.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:48 smithi045 conmon[33655]: audit 2021-07-31T03:38:48.026047+0000 mgr.x (mgr.24542) 4784 : audit [DBG] from='client.19938 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:38:49.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:48 smithi045 conmon[39718]: cluster 2021-07-31T03:38:47.387958 2021-07-31T03:38:49.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:48 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 4783 : cluster [DBG] pgmap v5355: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:49.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:48 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T03:38:49.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:48 smithi045 conmon[39718]: T03:38:48.026047+0000 mgr.x (mgr.24542) 4784 : 2021-07-31T03:38:49.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:48 smithi045 conmon[39718]: audit [DBG] from='client.19938 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:38:50.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:49 smithi154 conmon[34553]: audit 2021-07-31T03:38:48.307773+0000 mgr.x (mgr.24542) 4785 : audit [DBG] 2021-07-31T03:38:50.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:49 smithi154 conmon[34553]: from='client.19944 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:38:50.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:49 smithi154 conmon[34553]: audit 2021-07-31T03:38:48.653980+0000 mon.c (mon.1) 220 : audit [DBG] from='client.? 172.21.15.45:0/972480275' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:38:50.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:49 smithi045 conmon[33655]: audit 2021-07-31T03:38:48.307773+0000 mgr.x (mgr.24542) 4785 : audit [DBG] from='client.19944 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:38:50.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:49 smithi045 conmon[33655]: audit 2021-07-31T03:38:48.653980+0000 mon.c (mon.1) 220 : audit [DBG] from='client.? 172.21.15.45:0/972480275' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:38:50.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:49 smithi045 conmon[39718]: audit 2021-07-31T03:38:48.307773+0000 mgr.x (mgr.24542) 2021-07-31T03:38:50.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:49 smithi045 conmon[39718]: 4785 : audit [DBG] from='client.19944 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:38:50.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:49 smithi045 conmon[39718]: audit 2021-07-31T03:38:48.653980+0000 mon.c (mon.1) 220 : audit [DBG] from='client.? 172.21.15.45:0/972480275' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:38:51.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:50 smithi154 conmon[34553]: cluster 2021-07-31T03:38:49.388539+0000 mgr.x (mgr.24542) 4786 : cluster [DBG] pgmap v5356: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:51.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:50 smithi045 conmon[39718]: cluster 2021-07-31T03:38:49.388539+0000 mgr.x (mgr.24542 2021-07-31T03:38:51.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:50 smithi045 conmon[39718]: ) 4786 : cluster [DBG] pgmap v5356: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:51.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:50 smithi045 conmon[33655]: cluster 2021-07-31T03:38:49.388539+0000 mgr.x (mgr.24542) 4786 : cluster 2021-07-31T03:38:51.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:50 smithi045 conmon[33655]: [DBG] pgmap v5356: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:53.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:52 smithi154 conmon[34553]: cluster 2021-07-31T03:38:51.389017+0000 mgr.x (mgr.24542) 4787 : cluster 2021-07-31T03:38:53.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:52 smithi154 conmon[34553]: [DBG] pgmap v5357: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:53.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:52 smithi045 conmon[39718]: cluster 2021-07-31T03:38:51.389017+0000 mgr.x (mgr.24542) 4787 : cluster [DBG] 2021-07-31T03:38:53.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:52 smithi045 conmon[39718]: pgmap v5357: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:53.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:52 smithi045 conmon[33655]: cluster 2021-07-31T03:38:51.389017+0000 2021-07-31T03:38:53.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:52 smithi045 conmon[33655]: mgr.x (mgr.24542) 4787 : cluster [DBG] pgmap v5357: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:55.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:54 smithi154 conmon[34553]: cluster 2021-07-31T03:38:53.389473+0000 mgr.x (mgr.24542) 4788 : cluster [DBG] pgmap v5358: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:55.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:54 smithi045 conmon[33655]: cluster 2021-07-31T03:38:53.389473+0000 mgr.x (mgr.24542) 4788 : cluster 2021-07-31T03:38:55.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:54 smithi045 conmon[33655]: [DBG] pgmap v5358: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:55.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:54 smithi045 conmon[39718]: cluster 2021-07-31T03:38:53.389473 2021-07-31T03:38:55.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:54 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 4788 : cluster [DBG] pgmap v5358: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:57.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:56 smithi154 conmon[34553]: cluster 2021-07-31T03:38:55.389982+0000 mgr.x (mgr.24542) 4789 : cluster [DBG] pgmap v5359: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:57.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:56 smithi045 conmon[33655]: cluster 2021-07-31T03:38:55.389982 2021-07-31T03:38:57.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:56 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 4789 : cluster [DBG] pgmap v5359: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:57.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:56 smithi045 conmon[39718]: cluster 2021-07-31T03:38:55.389982+0000 2021-07-31T03:38:57.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:56 smithi045 conmon[39718]: mgr.x (mgr.24542) 4789 : cluster [DBG] pgmap v5359: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:59.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:38:58 smithi154 conmon[34553]: cluster 2021-07-31T03:38:57.390463+0000 mgr.x (mgr.24542) 4790 : cluster [DBG] pgmap v5360: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:59.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:58 smithi045 conmon[33655]: cluster 2021-07-31T03:38:57.390463 2021-07-31T03:38:59.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:38:58 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 4790 : cluster [DBG] pgmap v5360: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:38:59.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:58 smithi045 conmon[39718]: cluster 2021-07-31T03:38:57.390463+0000 mgr.x (mgr. 2021-07-31T03:38:59.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:38:58 smithi045 conmon[39718]: 24542) 4790 : cluster [DBG] pgmap v5360: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:01.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:00 smithi154 conmon[34553]: cluster 2021-07-31T03:38:59.391015+0000 mgr.x (mgr.24542) 4791 : cluster [DBG] pgmap v5361: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:01.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:00 smithi154 conmon[34553]: 2021-07-31T03:39:01.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:00 smithi045 conmon[39718]: cluster 2021-07-31T03:38:59.391015+0000 mgr.x (mgr.24542) 4791 2021-07-31T03:39:01.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:00 smithi045 conmon[39718]: : cluster [DBG] pgmap v5361: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:01.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:00 smithi045 conmon[33655]: cluster 2021-07-31T03:38:59.391015+0000 mgr.x (mgr.24542) 4791 : cluster [DBG] pgmap v5361: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:03.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:02 smithi154 conmon[34553]: cluster 2021-07-31T03:39:01.391439+0000 mgr.x (mgr.24542) 4792 : 2021-07-31T03:39:03.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:02 smithi154 conmon[34553]: cluster [DBG] pgmap v5362: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:03.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:02 smithi045 conmon[33655]: cluster 2021-07-31T03:39:01.391439+0000 mgr.x (mgr.24542) 4792 : cluster [DBG] pgmap v5362: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:03.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:02 smithi045 conmon[39718]: cluster 2021-07-31T03:39:01.391439+0000 mgr.x (mgr.24542 2021-07-31T03:39:03.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:02 smithi045 conmon[39718]: ) 4792 : cluster [DBG] pgmap v5362: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:05.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:04 smithi154 conmon[34553]: cluster 2021-07-31T03:39:03.392007+0000 mgr.x (mgr.24542) 4793 : cluster [DBG] pgmap v5363: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:05.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:04 smithi045 conmon[39718]: cluster 2021-07-31T03:39:03.392007+0000 mgr.x (mgr 2021-07-31T03:39:05.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:04 smithi045 conmon[39718]: .24542) 4793 : cluster [DBG] pgmap v5363: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:05.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:04 smithi045 conmon[33655]: cluster 2021-07-31T03:39:03.392007+0000 mgr.x (mgr.24542) 4793 : 2021-07-31T03:39:05.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:04 smithi045 conmon[33655]: cluster [DBG] pgmap v5363: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:07.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:06 smithi154 conmon[34553]: cluster 2021-07-31T03:39:05.392501+0000 mgr.x (mgr.24542) 4794 : cluster [DBG] pgmap v5364: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:07.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:06 smithi154 conmon[34553]: 2021-07-31T03:39:07.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:06 smithi045 conmon[33655]: cluster 2021-07-31T03:39:05.392501+0000 mgr.x (mgr 2021-07-31T03:39:07.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:06 smithi045 conmon[33655]: .24542) 4794 : cluster [DBG] pgmap v5364: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:07.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:06 smithi045 conmon[39718]: cluster 2021-07-31T03:39:05. 2021-07-31T03:39:07.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:06 smithi045 conmon[39718]: 392501+0000 mgr.x (mgr.24542) 4794 : cluster [DBG] pgmap v5364: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:09.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:08 smithi154 conmon[34553]: cluster 2021-07-31T03:39:07.392878+0000 mgr.x (mgr.24542) 4795 : cluster [DBG] pgmap v5365: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:09.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:08 smithi045 conmon[39718]: cluster 2021-07-31T03:39:07.392878+0000 mgr.x (mgr.24542) 4795 : cluster [DBG] pgmap v5365: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:09.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:08 smithi045 conmon[33655]: cluster 2021-07-31T03:39:07.392878+0000 mgr.x (mgr.24542) 4795 2021-07-31T03:39:09.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:08 smithi045 conmon[33655]: : cluster [DBG] pgmap v5365: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:11.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:10 smithi154 conmon[34553]: cluster 2021-07-31T03:39:09.393312+0000 mgr.x (mgr.24542) 4796 : cluster [DBG] pgmap v5366: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:11.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:10 smithi154 conmon[34553]: 2021-07-31T03:39:11.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:10 smithi045 conmon[39718]: cluster 2021-07-31T03:39:09.393312+0000 mgr.x (mgr.24542) 4796 : cluster [DBG] pgmap v5366: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:11.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:10 smithi045 conmon[33655]: cluster 2021-07-31T03:39:09.393312+0000 mgr.x (mgr.24542) 4796 : 2021-07-31T03:39:11.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:10 smithi045 conmon[33655]: cluster [DBG] pgmap v5366: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:13.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:12 smithi154 conmon[34553]: cluster 2021-07-31T03:39:11.393705+0000 mgr.x (mgr.24542) 4797 : cluster [DBG] pgmap v5367: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:13.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:12 smithi045 conmon[39718]: cluster 2021-07-31T03:39:11.393705+0000 mgr.x (mgr 2021-07-31T03:39:13.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:12 smithi045 conmon[39718]: .24542) 4797 : cluster [DBG] pgmap v5367: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:13.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:12 smithi045 conmon[33655]: cluster 2021-07-31T03:39:11.393705+0000 mgr.x (mgr.24542 2021-07-31T03:39:13.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:12 smithi045 conmon[33655]: ) 4797 : cluster [DBG] pgmap v5367: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:15.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:14 smithi154 conmon[34553]: cluster 2021-07-31T03:39:13.394262+0000 mgr.x (mgr.24542) 4798 : cluster [DBG] pgmap v5368: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:15.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:14 smithi045 conmon[39718]: cluster 2021-07-31T03:39:13.394262+0000 mgr.x (mgr.24542) 4798 : cluster [DBG] pgmap v5368: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:15.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:14 smithi045 conmon[33655]: cluster 2021-07-31T03:39:13.394262+0000 mgr.x (mgr.24542) 2021-07-31T03:39:15.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:14 smithi045 conmon[33655]: 4798 : cluster [DBG] pgmap v5368: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:17.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:16 smithi154 conmon[34553]: cluster 2021-07-31T03:39:15.394726+0000 mgr.x (mgr.24542) 4799 : cluster [DBG] pgmap v5369: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:17.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:16 smithi045 conmon[39718]: cluster 2021-07-31T03:39:15.394726+0000 mgr.x (mgr.24542 2021-07-31T03:39:17.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:16 smithi045 conmon[39718]: ) 4799 : cluster [DBG] pgmap v5369: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:17.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:16 smithi045 conmon[33655]: cluster 2021-07-31T03:39:15.394726+0000 2021-07-31T03:39:17.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:16 smithi045 conmon[33655]: mgr.x (mgr.24542) 4799 : cluster [DBG] pgmap v5369: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:18.963 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:39:19.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:18 smithi154 conmon[34553]: cluster 2021-07-31T03:39:17.395169+0000 mgr.x (mgr.24542) 4800 : cluster [DBG] pgmap v5370: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:19.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:18 smithi154 conmon[34553]: 2021-07-31T03:39:19.024 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:18 smithi045 conmon[39718]: cluster 2021-07-31T03:39:17.395169+0000 mgr.x ( 2021-07-31T03:39:19.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:18 smithi045 conmon[39718]: mgr.24542) 4800 : cluster [DBG] pgmap v5370: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:19.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:18 smithi045 conmon[33655]: cluster 2021-07-31T03:39:17.395169+0000 mgr.x 2021-07-31T03:39:19.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:18 smithi045 conmon[33655]: (mgr.24542) 4800 : cluster [DBG] pgmap v5370: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:19.239 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:39:19.240 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 114m ago 114m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:39:19.240 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 111m ago 114m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:39:19.240 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 111m ago 118m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:39:19.240 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 114m ago 2h - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:39:19.240 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 114m ago 2h - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:39:19.241 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 111m ago 118m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:39:19.241 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 114m ago 118m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:39:19.241 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 114m ago 115m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:39:19.241 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 111m ago 115m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:39:19.241 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 114m ago 117m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:39:19.241 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 114m ago 117m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:39:19.242 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 114m ago 117m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:39:19.242 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 114m ago 116m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:39:19.242 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 111m ago 116m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:39:19.242 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 111m ago 116m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:39:19.242 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 111m ago 115m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:39:19.243 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 111m ago 115m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:39:19.243 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 111m ago 115m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:39:19.575 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:39:19.576 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:39:19.576 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:39:19.576 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:39:19.576 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:39:19.576 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:39:19.577 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:39:19.577 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:39:19.577 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:39:19.577 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:39:19.577 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:39:19.578 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:39:19.578 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:39:19.578 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:39:19.578 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:39:19.579 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:39:19.579 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:39:20.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:19 smithi154 conmon[34553]: audit 2021-07-31T03:39:18.952377+0000 mgr.x (mgr.24542) 4801 : audit [DBG] from='client.29896 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:39:20.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:19 smithi154 conmon[34553]: audit 2021-07-31T03:39 2021-07-31T03:39:20.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:19 smithi154 conmon[34553]: :19.576221+0000 mon.a (mon.0) 3340 : audit [DBG] from='client.? 172.21.15.45:0/2981850662' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:39:20.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:19 smithi045 conmon[39718]: audit 2021-07-31T03:39:18. 2021-07-31T03:39:20.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:19 smithi045 conmon[39718]: 952377+0000 mgr.x (mgr.24542) 4801 : audit [DBG] from='client.29896 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:39:20.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:19 smithi045 conmon[39718]: audit 2021-07-31T03:39:19.576221+0000 mon.a ( 2021-07-31T03:39:20.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:19 smithi045 conmon[39718]: mon.0) 3340 : audit [DBG] from='client.? 172.21.15.45:0/2981850662' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:39:20.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:19 smithi045 conmon[33655]: audit 2021-07-31T03:39:18.952377+0000 mgr.x ( 2021-07-31T03:39:20.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:19 smithi045 conmon[33655]: mgr.24542) 4801 : audit [DBG] from='client.29896 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:39:20.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:19 smithi045 conmon[33655]: audit 2021-07-31T03:39:19.576221+0000 mon.a (mon.0) 3340 : audit [DBG] from='client.? 172.21.15.45:0/2981850662' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:39:21.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:20 smithi154 conmon[34553]: audit 2021-07-31T03:39:19.232382+0000 mgr.x (mgr.24542) 4802 : audit [DBG] from='client.19962 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:39:21.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:20 smithi154 conmon[34553]: cluster 2021-07-31T03:39:19.395676+0000 mgr.x (mgr.24542) 4803 : cluster [DBG] pgmap v5371: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:21.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:20 smithi045 conmon[33655]: audit 2021-07-31T03:39:19.232382+0000 2021-07-31T03:39:21.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:20 smithi045 conmon[33655]: mgr.x (mgr.24542) 4802 : audit [DBG] from='client.19962 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:39:21.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:20 smithi045 conmon[33655]: cluster 2021-07-31T03:39:19.395676+0000 mgr.x (mgr.24542) 4803 : cluster 2021-07-31T03:39:21.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:20 smithi045 conmon[33655]: [DBG] pgmap v5371: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:21.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:20 smithi045 conmon[39718]: audit 2021-07-31T03:39:19.232382+0000 mgr.x 2021-07-31T03:39:21.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:20 smithi045 conmon[39718]: (mgr.24542) 4802 : audit [DBG] from='client.19962 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:39:21.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:20 smithi045 conmon[39718]: cluster 2021 2021-07-31T03:39:21.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:20 smithi045 conmon[39718]: -07-31T03:39:19.395676+0000 mgr.x (mgr. 2021-07-31T03:39:21.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:20 smithi045 conmon[39718]: 24542) 4803 : cluster [DBG] pgmap v5371: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:23.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:22 smithi154 conmon[34553]: cluster 2021-07-31T03:39:21.396095+0000 mgr.x (mgr.24542) 4804 : cluster [DBG] pgmap v5372: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:23.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:22 smithi045 conmon[39718]: cluster 2021-07-31T03:39:21.396095+0000 mgr.x (mgr.24542) 2021-07-31T03:39:23.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:22 smithi045 conmon[39718]: 4804 : cluster [DBG] pgmap v5372: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:23.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:22 smithi045 conmon[33655]: cluster 2021-07-31T03:39:21.396095+0000 mgr.x (mgr.24542) 4804 : cluster 2021-07-31T03:39:23.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:22 smithi045 conmon[33655]: [DBG] pgmap v5372: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:25.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:24 smithi154 conmon[34553]: cluster 2021-07-31T03:39:23.396601+0000 mgr.x (mgr.24542) 4805 : cluster [DBG] pgmap v5373: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:25.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:24 smithi045 conmon[33655]: cluster 2021-07-31T03:39:23.396601+0000 mgr.x (mgr.24542) 4805 : 2021-07-31T03:39:25.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:24 smithi045 conmon[33655]: cluster [DBG] pgmap v5373: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:25.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:24 smithi045 conmon[39718]: cluster 2021-07-31T03:39:23.396601+0000 mgr.x (mgr 2021-07-31T03:39:25.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:24 smithi045 conmon[39718]: .24542) 4805 : cluster [DBG] pgmap v5373: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:27.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:26 smithi154 conmon[34553]: cluster 2021-07-31T03:39:25.397247+0000 mgr.x (mgr.24542) 4806 : cluster [DBG] pgmap v5374: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:27.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:26 smithi045 conmon[33655]: cluster 2021-07-31T03:39:25.397247 2021-07-31T03:39:27.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:26 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 4806 : cluster [DBG] pgmap v5374: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:27.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:26 smithi045 conmon[39718]: cluster 2021-07-31T03:39:25.397247+0000 mgr.x (mgr. 2021-07-31T03:39:27.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:26 smithi045 conmon[39718]: 24542) 4806 : cluster [DBG] pgmap v5374: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:29.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:28 smithi154 conmon[34553]: cluster 2021-07-31T03:39:27.397640+0000 mgr.x (mgr.24542) 4807 : cluster [DBG] pgmap v5375: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:29.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:28 smithi154 conmon[34553]: 2021-07-31T03:39:29.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:28 smithi154 conmon[34553]: audit 2021-07-31T03:39:28.381614+0000 mon.b (mon.2) 635 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:39:29.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:28 smithi154 conmon[34553]: audit 2021-07-31T03:39:28.381905+0000 mon.b (mon.2) 636 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:39:29.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:28 smithi154 conmon[34553]: 2021-07-31T03:39:29.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:28 smithi154 conmon[34553]: audit 2021-07-31T03:39:28.382134+0000 2021-07-31T03:39:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:28 smithi154 conmon[34553]: mon.a (mon.0) 3341 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:39:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:28 smithi154 conmon[34553]: audit 2021-07- 2021-07-31T03:39:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:28 smithi154 conmon[34553]: 31T03:39:28.382605+0000 mon.a ( 2021-07-31T03:39:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:28 smithi154 conmon[34553]: mon.0) 3342 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:39:29.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:28 smithi045 conmon[33655]: cluster 2021-07-31T03:39:27.397640+0000 mgr.x (mgr.24542) 4807 : cluster [DBG] pgmap v5375: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:29.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:28 smithi045 conmon[33655]: 2021-07-31T03:39:29.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:28 smithi045 conmon[33655]: audit 2021-07-31T03:39:28.381614+0000 mon.b (mon.2) 635 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:39:29.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:28 smithi045 conmon[33655]: audit 2021-07-31T03:39:28.381905+0000 2021-07-31T03:39:29.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:28 smithi045 conmon[33655]: mon.b (mon.2) 636 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:39:29.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:28 smithi045 conmon[33655]: audit 2021 2021-07-31T03:39:29.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:28 smithi045 conmon[33655]: -07-31T03:39:28.382134 2021-07-31T03:39:29.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:28 smithi045 conmon[33655]: +0000 mon.a (mon.0) 3341 : audit 2021-07-31T03:39:29.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:28 smithi045 conmon[33655]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:39:29.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:28 smithi045 conmon[33655]: audit 2021-07-31T 2021-07-31T03:39:29.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:28 smithi045 conmon[33655]: 03:39:28.382605+0000 mon.a ( 2021-07-31T03:39:29.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:28 smithi045 conmon[33655]: mon.0) 3342 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:39:29.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:28 smithi045 conmon[39718]: cluster 2021-07-31T03:39:27.397640+0000 mgr.x (mgr.24542) 2021-07-31T03:39:29.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:28 smithi045 conmon[39718]: 4807 : cluster [DBG] pgmap v5375: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:29.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:28 smithi045 conmon[39718]: audit 2021-07-31T03:39:28.381614+0000 mon.b 2021-07-31T03:39:29.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:28 smithi045 conmon[39718]: (mon.2) 635 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:39:29.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:28 smithi045 conmon[39718]: audit 2021-07-31T03: 2021-07-31T03:39:29.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:28 smithi045 conmon[39718]: 39:28.381905+0000 mon.b (mon.2) 636 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:39:29.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:28 smithi045 conmon[39718]: audit 2021-07-31T03:39:28.382134+0000 2021-07-31T03:39:29.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:28 smithi045 conmon[39718]: mon.a (mon.0) 3341 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:39:29.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:28 smithi045 conmon[39718]: audit 2021-07-31T03:39:28. 2021-07-31T03:39:29.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:28 smithi045 conmon[39718]: 382605+0000 mon.a (mon.0) 3342 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:39:31.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:30 smithi154 conmon[34553]: cluster 2021-07-31T03:39:29.398234+0000 mgr.x (mgr.24542) 4808 : cluster [DBG] pgmap v5376: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:31.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:30 smithi045 conmon[39718]: cluster 2021-07-31T03:39:29.398234+0000 2021-07-31T03:39:31.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:30 smithi045 conmon[39718]: mgr.x (mgr.24542) 4808 : cluster [DBG] pgmap v5376: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:31.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:30 smithi045 conmon[33655]: cluster 2021-07-31T03:39:29.398234+0000 mgr.x ( 2021-07-31T03:39:31.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:30 smithi045 conmon[33655]: mgr.24542) 4808 : cluster [DBG] pgmap v5376: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:33.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:32 smithi154 conmon[34553]: cluster 2021-07-31T03:39:31.398692+0000 mgr.x (mgr.24542) 4809 : cluster [DBG] pgmap v5377: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:33.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:32 smithi045 conmon[39718]: cluster 2021-07-31T03:39:31.398692+0000 mgr.x (mgr.24542) 4809 : cluster [DBG] pgmap v5377: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:33.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:32 smithi045 conmon[33655]: cluster 2021-07-31T03:39:31.398692+0000 mgr.x (mgr.24542) 2021-07-31T03:39:33.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:32 smithi045 conmon[33655]: 4809 : cluster [DBG] pgmap v5377: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:34.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[34553]: cluster 2021-07-31T03:39:33.399119+0000 mgr.x (mgr. 2021-07-31T03:39:34.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[34553]: 24542) 4810 : cluster [DBG] pgmap v5378: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:34.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[34553]: audit 2021-07-31T03:39:34.371409+0000 mon.b (mon.2) 637 : audit [DBG] 2021-07-31T03:39:34.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[34553]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:39:34.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[34553]: audit 2021-07-31T03:39:34 2021-07-31T03:39:34.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[34553]: .372775+0000 mon.b (mon.2) 638 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:39:34.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[34553]: audit 2021 2021-07-31T03:39:34.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[34553]: -07-31T03:39:34.373818+0000 2021-07-31T03:39:34.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[34553]: mon.b (mon.2) 639 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:39:34.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[34553]: 2021-07-31T03:39:34.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: debug 2021-07-31T03:39:34.416+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:39:34.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:39:34.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:39:34.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: yield 2021-07-31T03:39:34.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:39:34.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:39:34.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:39:34.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:39:34.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:39:34.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:39:34.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:39:34.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:39:34.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:39:34.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: task.result() 2021-07-31T03:39:34.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:39:34.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: debug 2021-07-31T03:39:34.428+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:39:34.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:39:34.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:39:34.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: yield 2021-07-31T03:39:34.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:39:34.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:39:34.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:39:34.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:39:34.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:39:34.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:39:34.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:39:34.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:39:34.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:39:34.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: task.result() 2021-07-31T03:39:34.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:39:34.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: debug 2021-07-31T03:39:34.453+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:39:34.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:39:34.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:39:34.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: yield 2021-07-31T03:39:34.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:39:34.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:39:34.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:39:34.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:39:34.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:39:34.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:39:34.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:39:34.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:39:34.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:39:34.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: task.result() 2021-07-31T03:39:34.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:39:34.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: debug 2021-07-31T03:39:34.454+0000 7fcb9215e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:39:34.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:39:34.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:39:34.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: yield 2021-07-31T03:39:34.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:39:34.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:39:34.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:39:34.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:39:34.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:39:34.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:39:34.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:39:34.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:39:34.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:39:34.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: task.result() 2021-07-31T03:39:34.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:39:34.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: 2021-07-31T03:39:34.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:39:34.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: 2021-07-31T03:39:34.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:39:34.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:39:34.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:39:34.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:39:34.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:39:34.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:39:34.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:39:34.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:39:34.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:39:34.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:39:34.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:39:34.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:39:34.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:39:34.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:39:34.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:39:34.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:39:34.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:39:34.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:39:34.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: raise 2021-07-31T03:39:34.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:39:34.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:39:34.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:39:34.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:39:34.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:39:34.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: debug 2021-07-31T03:39:34.462+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:39:34.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:39:34.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:39:34.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: yield 2021-07-31T03:39:34.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:39:34.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:39:34.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:39:34.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:39:34.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:39:34.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:39:34.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:39:34.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:39:34.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:39:34.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: task.result() 2021-07-31T03:39:34.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:39:34.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: debug 2021-07-31T03:39:34.463+0000 7fcb9195d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:39:34.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:39:34.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:39:34.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: yield 2021-07-31T03:39:34.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:39:34.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:39:34.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:39:34.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:39:34.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:39:34.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:39:34.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:39:34.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:39:34.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:39:34.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: task.result() 2021-07-31T03:39:34.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:39:34.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: 2021-07-31T03:39:34.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:39:34.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: 2021-07-31T03:39:34.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:39:34.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:39:34.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:39:34.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:39:34.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:39:34.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:39:34.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:39:34.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:39:34.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:39:34.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:39:34.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:39:34.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:39:34.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:39:34.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:39:34.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:39:34.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:39:34.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:39:34.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:39:34.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: raise 2021-07-31T03:39:34.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:39:34.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:39:34.786 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:39:34.786 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:39:34.786 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:39:34 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:39:34.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:34 smithi045 conmon[33655]: cluster 2021-07-31T03:39:33.399119+0000 mgr.x (mgr.24542) 4810 2021-07-31T03:39:34.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:34 smithi045 conmon[33655]: : cluster [DBG] pgmap v5378: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:34.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:34 smithi045 conmon[33655]: audit 2021-07-31T03:39:34.371409+0000 2021-07-31T03:39:34.788 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:34 smithi045 conmon[33655]: mon.b (mon.2) 637 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:39:34.788 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:34 smithi045 conmon[33655]: audit 2021-07-31T03:39:34.788 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:34 smithi045 conmon[33655]: 2021-07-31T03:39 2021-07-31T03:39:34.788 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:34 smithi045 conmon[33655]: :34.372775+0000 mon.b 2021-07-31T03:39:34.789 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:34 smithi045 conmon[33655]: (mon.2) 638 : audit 2021-07-31T03:39:34.789 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:34 smithi045 conmon[33655]: [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:39:34.789 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:34 smithi045 conmon[33655]: audit 2021 2021-07-31T03:39:34.789 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:34 smithi045 conmon[33655]: -07-31T03:39:34. 2021-07-31T03:39:34.789 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:34 smithi045 conmon[33655]: 373818+0000 mon.b (mon. 2021-07-31T03:39:34.790 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:34 smithi045 conmon[33655]: 2) 639 : audit [INF] 2021-07-31T03:39:34.790 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:34 smithi045 conmon[33655]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:39:34.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:34 smithi045 conmon[39718]: cluster 2021-07-31T03:39:33.399119+0000 mgr.x (mgr.24542) 4810 : cluster [DBG] pgmap v5378: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:34.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:34 smithi045 conmon[39718]: audit 2021-07-31T03:39:34.371409+0000 mon.b ( 2021-07-31T03:39:34.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:34 smithi045 conmon[39718]: mon.2) 637 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:39:34.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:34 smithi045 conmon[39718]: 2021-07-31T03:39:34.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:34 smithi045 conmon[39718]: audit 2021-07-31T03:39:34.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:34 smithi045 conmon[39718]: 2021-07-31T03:39: 2021-07-31T03:39:34.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:34 smithi045 conmon[39718]: 34.372775+0000 mon.b (mon. 2021-07-31T03:39:34.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:34 smithi045 conmon[39718]: 2) 638 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:39:34.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:34 smithi045 conmon[39718]: 2021-07-31T03:39:34.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:34 smithi045 conmon[39718]: audit 2021 2021-07-31T03:39:34.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:34 smithi045 conmon[39718]: -07-31T03:39: 2021-07-31T03:39:34.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:34 smithi045 conmon[39718]: 34.373818+0000 mon.b (mon. 2021-07-31T03:39:34.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:34 smithi045 conmon[39718]: 2) 639 : audit [INF] 2021-07-31T03:39:34.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:34 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:39:36.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:39:34.374569+0000 mgr.x (mgr.24542) 4811 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:39:36.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:39:34.417593+0000 mgr.x (mgr.24542) 4812 : cephadm [ERR] 2021-07-31T03:39:36.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:39:36.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:39:36.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: yield 2021-07-31T03:39:36.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:39:36.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:39:36.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:39:36.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:39:36.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:39:36.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:39:36.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:39:36.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:39:36.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:39:36.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: task.result() 2021-07-31T03:39:36.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:39:36.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:39:34.418045+0000 mgr.x (mgr.24542) 4813 : cephadm 2021-07-31T03:39:36.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:39:36.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:39:34.429292+0000 mgr.x (mgr.24542) 4814 : cephadm [ERR] 2021-07-31T03:39:36.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:39:36.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:39:36.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: yield 2021-07-31T03:39:36.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:39:36.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:39:36.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:39:36.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:39:36.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:39:36.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:39:36.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:39:36.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:39:36.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:39:36.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: task.result() 2021-07-31T03:39:36.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:39:36.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:39:34.429747+0000 mgr.x (mgr.24542) 4815 : 2021-07-31T03:39:36.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:39:36.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:39:34.454896+0000 mgr.x (mgr.24542) 4816 : cephadm [ERR] 2021-07-31T03:39:36.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:39:36.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:39:36.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: yield 2021-07-31T03:39:36.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:39:36.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:39:36.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:39:36.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:39:36.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:39:36.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:39:36.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:39:36.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:39:36.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:39:36.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: task.result() 2021-07-31T03:39:36.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:39:36.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:39:34.455376+0000 mgr.x (mgr.24542 2021-07-31T03:39:36.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: ) 4817 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:39:36.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:39:36.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:39:36.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: yield 2021-07-31T03:39:36.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:39:36.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:39:36.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:39:36.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:39:36.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:39:36.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:39:36.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:39:36.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:39:36.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:39:36.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: task.result() 2021-07-31T03:39:36.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:39:36.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: 2021-07-31T03:39:36.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:39:36.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: 2021-07-31T03:39:36.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:39:36.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:39:36.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:39:36.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:39:36.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:39:36.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:39:36.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:39:36.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:39:36.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:39:36.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:39:36.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:39:36.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:39:36.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:39:36.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:39:36.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:39:36.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:39:36.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:39:36.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:39:36.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: raise 2021-07-31T03:39:36.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:39:36.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:39:36.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:39:36.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:39:36.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:39:36.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:39:34.463900+0000 mgr.x (mgr.24542) 4818 : cephadm [ERR] 2021-07-31T03:39:36.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:39:36.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:39:36.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: yield 2021-07-31T03:39:36.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:39:36.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:39:36.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:39:36.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:39:36.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:39:36.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:39:36.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:39:36.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:39:36.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:39:36.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: task.result() 2021-07-31T03:39:36.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:39:36.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:39:34.464330+0000 2021-07-31T03:39:36.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: mgr.x (mgr.24542) 4819 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:39:36.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:39:36.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:39:36.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: yield 2021-07-31T03:39:36.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:39:36.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:39:36.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:39:36.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:39:36.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:39:36.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:39:36.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:39:36.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:39:36.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:39:36.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: task.result() 2021-07-31T03:39:36.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:39:36.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: 2021-07-31T03:39:36.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:39:36.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: 2021-07-31T03:39:36.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:39:36.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:39:36.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:39:36.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:39:36.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:39:36.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:39:36.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:39:36.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:39:36.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:39:36.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:39:36.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:39:36.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:39:36.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:39:36.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:39:36.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:39:36.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:39:36.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:39:36.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:39:36.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: raise 2021-07-31T03:39:36.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:39:36.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:39:36.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:39:36.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:39:36.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:35 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:39:36.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:39:34.374569+0000 mgr.x (mgr.24542) 4811 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:39:36.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: 2021-07-31T03:39:36.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:39:34.417593+0000 mgr.x (mgr.24542) 4812 : cephadm [ERR] 2021-07-31T03:39:36.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:39:36.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:39:36.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: yield 2021-07-31T03:39:36.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:39:36.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:39:36.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:39:36.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:39:36.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:39:36.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:39:36.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:39:36.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:39:36.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:39:36.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: task.result() 2021-07-31T03:39:36.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:39:36.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:39:34.418045+0000 mgr.x (mgr.24542) 4813 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:39:36.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:39:34.429292+0000 mgr.x (mgr.24542) 4814 : cephadm [ERR] 2021-07-31T03:39:36.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:39:36.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:39:36.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: yield 2021-07-31T03:39:36.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:39:36.040 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:39:36.040 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:39:36.040 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:39:36.040 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:39:36.041 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:39:36.041 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:39:36.041 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:39:36.041 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:39:36.041 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: task.result() 2021-07-31T03:39:36.042 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:39:36.042 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:39:36.042 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: 2021-07-31T03:39:34.429747+0000 mgr.x (mgr.24542) 4815 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:39:36.042 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: cephadm 2021 2021-07-31T03:39:36.042 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: -07-31T03:39:34.454896 2021-07-31T03:39:36.043 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542 2021-07-31T03:39:36.043 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: ) 4816 : cephadm [ERR] 2021-07-31T03:39:36.043 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:39:36.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:39:34.374569+0000 mgr.x (mgr.24542) 4811 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:39:36.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:39:34.417593+0000 2021-07-31T03:39:36.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: mgr.x (mgr.24542) 4812 : cephadm [ERR] 2021-07-31T03:39:36.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: 2021-07-31T03:39:36.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:39:36.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:39:36.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: yield 2021-07-31T03:39:36.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:39:36.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:39:36.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:39:36.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:39:36.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:39:36.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:39:36.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:39:36.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:39:36.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:39:36.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: task.result() 2021-07-31T03:39:36.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:39:36.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:39:34.418045 2021-07-31T03:39:36.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 4813 : 2021-07-31T03:39:36.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:39:36.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: cephadm 2021- 2021-07-31T03:39:36.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: 07-31T03:39:34.429292 2021-07-31T03:39:36.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 2021-07-31T03:39:36.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: 4814 : cephadm [ERR] 2021-07-31T03:39:36.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:39:36.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:39:36.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: yield 2021-07-31T03:39:36.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:39:36.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:39:36.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:39:36.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:39:36.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:39:36.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:39:36.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:39:36.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:39:36.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:39:36.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: task.result() 2021-07-31T03:39:36.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:39:36.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:39:34.429747+0000 mgr.x (mgr. 2021-07-31T03:39:36.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: 24542) 4815 : cephadm [INF] 2021-07-31T03:39:36.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:39:36.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:39:36.053 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:39:36.053 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: yield 2021-07-31T03:39:36.053 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:39:36.054 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:39:36.054 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:39:36.054 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:39:36.054 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:39:36.054 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:39:36.055 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:39:36.055 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:39:36.055 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:39:36.055 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: task.result() 2021-07-31T03:39:36.055 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:39:36.056 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: 2021-07-31T03:39:36.056 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:39:34.455376 2021-07-31T03:39:36.056 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 4817 : cephadm 2021-07-31T03:39:36.056 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:39:36.056 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:39:36.057 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:39:36.057 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: yield 2021-07-31T03:39:36.057 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:39:36.057 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:39:36.057 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:39:36.058 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:39:36.058 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:39:36.058 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:39:36.058 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:39:36.058 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:39:36.059 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:39:36.059 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: task.result() 2021-07-31T03:39:36.059 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:39:36.059 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: 2021-07-31T03:39:36.060 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:39:36.060 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: 2021-07-31T03:39:36.060 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:39:36.060 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:39:36.060 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:39:36.061 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:39:36.061 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:39:36.061 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:39:36.061 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:39:36.061 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:39:36.062 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: 2021-07-31T03: 2021-07-31T03:39:36.062 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: 39:34.454896+0000 mgr.x 2021-07-31T03:39:36.062 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: (mgr.24542) 4816 : 2021-07-31T03:39:36.062 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: cephadm [ERR] 2021-07-31T03:39:36.063 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:39:36.063 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:39:36.063 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: yield 2021-07-31T03:39:36.063 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:39:36.064 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:39:36.064 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:39:36.064 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:39:36.064 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:39:36.064 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:39:36.065 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:39:36.065 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:39:36.065 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:39:36.065 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: task.result() 2021-07-31T03:39:36.065 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:39:36.066 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:39:34.455376+0000 mgr.x (mgr.24542 2021-07-31T03:39:36.066 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: ) 4817 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:39:36.066 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:39:36.066 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:39:36.066 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: yield 2021-07-31T03:39:36.067 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:39:36.067 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:39:36.067 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:39:36.067 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:39:36.068 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:39:36.068 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:39:36.068 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:39:36.068 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:39:36.068 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:39:36.069 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: task.result() 2021-07-31T03:39:36.069 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:39:36.069 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: 2021-07-31T03:39:36.069 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:39:36.069 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: 2021-07-31T03:39:36.070 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:39:36.070 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:39:36.070 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:39:36.070 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:39:36.070 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:39:36.071 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:39:36.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:39:36.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:39:36.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:39:36.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:39:36.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:39:36.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:39:36.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:39:36.073 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:39:36.073 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:39:36.073 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:39:36.073 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: raise 2021-07-31T03:39:36.073 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:39:36.074 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:39:36.074 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:39:36.074 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:39:36.074 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:39:36.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:39:34.463900+0000 mgr.x (mgr.24542) 4818 : cephadm [ERR] 2021-07-31T03:39:36.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:39:36.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:39:36.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: yield 2021-07-31T03:39:36.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:39:36.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:39:36.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:39:36.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:39:36.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:39:36.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:39:36.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:39:36.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:39:36.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:39:36.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: task.result() 2021-07-31T03:39:36.078 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:39:36.078 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:39:34.464330+0000 mgr.x (mgr.24542) 4819 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:39:36.078 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:39:36.078 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:39:36.078 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: yield 2021-07-31T03:39:36.079 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:39:36.079 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:39:36.079 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:39:36.079 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:39:36.080 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:39:36.080 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:39:36.080 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:39:36.080 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:39:36.080 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:39:36.081 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:39:36.081 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:39:36.081 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:39:36.081 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:39:36.081 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:39:36.082 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:39:36.082 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:39:36.082 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: raise 2021-07-31T03:39:36.082 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:39:36.083 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:39:36.083 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:39:36.083 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:39:36.083 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:39:36.083 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: 2021-07-31T03:39:36.084 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:39:34.463900+0000 mgr.x (mgr.24542) 4818 : cephadm [ERR] 2021-07-31T03:39:36.084 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:39:36.084 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:39:36.084 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: yield 2021-07-31T03:39:36.084 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:39:36.085 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:39:36.085 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:39:36.085 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:39:36.085 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:39:36.085 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:39:36.086 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:39:36.086 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:39:36.086 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:39:36.086 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: task.result() 2021-07-31T03:39:36.086 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:39:36.087 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:39:34.464330+0000 mgr.x (mgr.24542) 4819 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:39:36.087 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:39:36.087 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:39:36.087 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: yield 2021-07-31T03:39:36.087 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:39:36.088 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:39:36.088 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:39:36.088 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:39:36.088 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:39:36.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:39:36.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: task.result() 2021-07-31T03:39:36.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:39:36.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: 2021-07-31T03:39:36.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:39:36.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: 2021-07-31T03:39:36.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:39:36.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:39:36.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:39:36.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:39:36.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:39:36.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:39:36.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:39:36.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:39:36.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:39:36.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:39:36.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:39:36.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:39:36.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:39:36.093 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:39:36.093 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:39:36.093 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:39:36.093 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:39:36.093 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:39:36.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: raise 2021-07-31T03:39:36.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:39:36.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:39:36.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:39:36.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:39:36.095 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:39:36.095 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:39:36.095 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:39:36.095 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:39:36.096 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:39:36.096 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:39:36.096 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:39:36.096 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:39:36.096 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:39:36.097 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: task.result() 2021-07-31T03:39:36.097 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:39:36.097 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: 2021-07-31T03:39:36.097 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:39:36.098 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: 2021-07-31T03:39:36.098 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:39:36.098 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:39:36.098 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:39:36.098 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:39:36.099 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:39:36.099 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:39:36.099 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:39:36.099 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:39:36.099 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:39:36.100 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:39:36.100 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:39:36.100 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:39:36.100 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:39:36.100 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:39:36.101 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:39:36.101 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:39:36.101 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:39:36.101 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:39:36.102 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: raise 2021-07-31T03:39:36.102 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:39:36.102 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:39:36.102 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:39:36.102 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:39:36.103 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:35 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:39:37.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:36 smithi154 conmon[34553]: cluster 2021-07-31T03:39:35.399548+0000 mgr.x (mgr.24542) 4820 : cluster [DBG] pgmap v5379: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:37.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:36 smithi045 conmon[39718]: cluster 2021-07-31T03:39:35.399548+0000 mgr.x (mgr.24542) 4820 : cluster [DBG] pgmap v5379: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:37.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:36 smithi045 conmon[33655]: cluster 2021-07-31T03:39:35.399548+0000 mgr.x 2021-07-31T03:39:37.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:36 smithi045 conmon[33655]: (mgr.24542) 4820 : cluster [DBG] pgmap v5379: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:39.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:38 smithi154 conmon[34553]: cluster 2021-07-31T03:39:37.400089+0000 mgr.x (mgr.24542) 4821 : cluster [DBG] pgmap v5380: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:39.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:38 smithi045 conmon[39718]: cluster 2021-07-31T03:39:37.400089+0000 2021-07-31T03:39:39.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:38 smithi045 conmon[39718]: mgr.x (mgr.24542) 4821 : cluster [DBG] pgmap v5380: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:39.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:38 smithi045 conmon[33655]: cluster 2021-07-31T03:39:37.400089+0000 mgr.x (mgr.24542) 4821 : cluster 2021-07-31T03:39:39.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:38 smithi045 conmon[33655]: [DBG] pgmap v5380: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:41.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:40 smithi154 conmon[34553]: cluster 2021-07-31T03:39:39.400530+0000 mgr.x (mgr.24542) 4822 : cluster [DBG] pgmap v5381: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:41.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:40 smithi045 conmon[39718]: cluster 2021-07-31T03:39:39.400530+0000 mgr.x 2021-07-31T03:39:41.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:40 smithi045 conmon[39718]: (mgr.24542) 4822 : cluster [DBG] pgmap v5381: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:41.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:40 smithi045 conmon[33655]: cluster 2021-07-31T03:39:39.400530+0000 mgr.x 2021-07-31T03:39:41.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:40 smithi045 conmon[33655]: (mgr.24542) 4822 : cluster [DBG] pgmap v5381: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:43.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:42 smithi154 conmon[34553]: cluster 2021-07-31T03:39:41.400864+0000 mgr.x (mgr.24542) 4823 : cluster 2021-07-31T03:39:43.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:42 smithi154 conmon[34553]: [DBG] pgmap v5382: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:43.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:42 smithi045 conmon[39718]: cluster 2021-07-31T03:39:41.400864+0000 2021-07-31T03:39:43.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:42 smithi045 conmon[39718]: mgr.x (mgr.24542) 4823 : cluster [DBG] pgmap v5382: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:43.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:42 smithi045 conmon[33655]: cluster 2021-07-31T03:39:41.400864+0000 mgr.x (mgr.24542) 4823 : cluster 2021-07-31T03:39:43.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:42 smithi045 conmon[33655]: [DBG] pgmap v5382: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:45.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:44 smithi154 conmon[34553]: cluster 2021-07-31T03:39:43.401381+0000 mgr.x (mgr.24542) 4824 : cluster [DBG] pgmap v5383: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:45.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:44 smithi045 conmon[39718]: cluster 2021-07-31T03:39:43.401381+0000 mgr.x (mgr.24542) 2021-07-31T03:39:45.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:44 smithi045 conmon[39718]: 4824 : cluster [DBG] pgmap v5383: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:45.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:44 smithi045 conmon[33655]: cluster 2021-07-31T03:39:43.401381+0000 mgr.x (mgr.24542) 4824 : cluster [DBG] pgmap v5383: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:47.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:46 smithi154 conmon[34553]: cluster 2021-07-31T03:39:45.401897+0000 mgr.x (mgr.24542) 4825 : cluster [DBG] pgmap v5384: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:47.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:46 smithi045 conmon[39718]: cluster 2021-07-31T03:39:45.401897+0000 mgr.x (mgr 2021-07-31T03:39:47.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:46 smithi045 conmon[39718]: .24542) 4825 : cluster [DBG] pgmap v5384: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:47.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:46 smithi045 conmon[33655]: cluster 2021-07-31T03:39:45.401897+0000 mgr.x (mgr.24542) 4825 : cluster [DBG] pgmap v5384: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:49.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:48 smithi154 conmon[34553]: cluster 2021-07-31T03:39:47.402487+0000 mgr.x (mgr.24542) 4826 : cluster [DBG] pgmap v5385: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:49.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:48 smithi045 conmon[33655]: cluster 2021-07-31T03:39:47.402487+0000 mgr.x (mgr 2021-07-31T03:39:49.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:48 smithi045 conmon[33655]: .24542) 4826 : cluster [DBG] pgmap v5385: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:49.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:48 smithi045 conmon[39718]: cluster 2021-07-31T03:39:47.402487+0000 mgr.x (mgr.24542 2021-07-31T03:39:49.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:48 smithi045 conmon[39718]: ) 4826 : cluster [DBG] pgmap v5385: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:49.891 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:39:50.169 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:39:50.169 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 114m ago 115m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:39:50.169 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 112m ago 115m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:39:50.169 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 112m ago 118m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:39:50.170 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 114m ago 2h - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:39:50.170 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 114m ago 2h - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:39:50.170 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 112m ago 119m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:39:50.170 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 114m ago 119m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:39:50.171 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 114m ago 115m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:39:50.171 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 112m ago 115m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:39:50.171 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 114m ago 118m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:39:50.171 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 114m ago 118m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:39:50.171 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 114m ago 117m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:39:50.172 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 114m ago 117m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:39:50.172 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 112m ago 117m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:39:50.172 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 112m ago 116m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:39:50.172 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 112m ago 116m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:39:50.172 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 112m ago 116m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:39:50.173 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 112m ago 115m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:39:50.509 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:39:50.510 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:39:50.510 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:39:50.510 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:39:50.510 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:39:50.511 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:39:50.511 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:39:50.511 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:39:50.511 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:39:50.512 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:39:50.512 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:39:50.512 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:39:50.512 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:39:50.512 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:39:50.513 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:39:50.513 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:39:50.513 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:39:51.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:50 smithi154 conmon[34553]: cluster 2021-07-31T03:39:49.402910+0000 mgr.x (mgr.24542) 4827 : cluster [DBG] pgmap v5386: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:51.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:50 smithi154 conmon[34553]: audit 2021-07-31T03:39:49.880761+0000 mgr.x (mgr.24542) 4828 : audit [DBG] from='client.19974 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:39:51.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:50 smithi154 conmon[34553]: audit 2021-07-31T03:39:50.510533+0000 mon.c (mon.1) 221 : audit [DBG] from='client.? 172.21.15.45:0/1775674824' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:39:51.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:50 smithi045 conmon[39718]: cluster 2021-07-31T03:39:49.402910+0000 mgr.x ( 2021-07-31T03:39:51.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:50 smithi045 conmon[39718]: mgr.24542) 4827 : cluster [DBG] pgmap v5386: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:51.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:50 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T03:39:51.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:50 smithi045 conmon[39718]: T03:39:49.880761+0000 mgr.x (mgr.24542) 4828 : audit [DBG] from='client.19974 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:39:51.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:50 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T03:39:51.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:50 smithi045 conmon[39718]: 31T03:39:50.510533+0000 mon.c (mon.1) 2021-07-31T03:39:51.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:50 smithi045 conmon[39718]: 221 : audit [DBG] from='client.? 172.21.15.45:0/1775674824' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:39:51.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:50 smithi045 conmon[33655]: cluster 2021-07-31T03:39:49.402910+0000 mgr.x ( 2021-07-31T03:39:51.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:50 smithi045 conmon[33655]: mgr.24542) 4827 : cluster [DBG] pgmap v5386: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:51.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:50 smithi045 conmon[33655]: audit 2021-07-31T03:39:49.880761+0000 mgr.x (mgr.24542) 4828 : audit [DBG] from='client.19974 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:39:51.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:50 smithi045 conmon[33655]: audit 2021-07-31T03:39: 2021-07-31T03:39:51.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:50 smithi045 conmon[33655]: 50.510533+0000 mon.c (mon.1) 2021-07-31T03:39:51.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:50 smithi045 conmon[33655]: 221 : audit [DBG] from='client.? 172.21.15.45:0/1775674824' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:39:52.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:51 smithi154 conmon[34553]: audit 2021-07-31T03:39:50.161629+0000 mgr.x (mgr.24542) 4829 : audit [DBG] from='client.19980 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:39:52.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:51 smithi045 conmon[39718]: audit 2021-07-31T03:39:50.161629+0000 mgr.x ( 2021-07-31T03:39:52.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:51 smithi045 conmon[39718]: mgr.24542) 4829 : audit [DBG] from='client.19980 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:39:52.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:51 smithi045 conmon[33655]: audit 2021-07-31T03:39:50.161629+0000 mgr.x 2021-07-31T03:39:52.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:51 smithi045 conmon[33655]: (mgr.24542) 4829 : audit [DBG] from='client.19980 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:39:53.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:52 smithi154 conmon[34553]: cluster 2021-07-31T03:39:51.403438+0000 mgr.x (mgr.24542) 4830 : cluster [DBG] pgmap v5387: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:53.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:52 smithi045 conmon[33655]: cluster 2021-07-31T03:39:51.403438+0000 mgr.x (mgr.24542) 4830 : cluster 2021-07-31T03:39:53.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:52 smithi045 conmon[33655]: [DBG] pgmap v5387: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:53.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:52 smithi045 conmon[39718]: cluster 2021-07-31T03:39:51.403438+0000 mgr.x (mgr.24542) 4830 2021-07-31T03:39:53.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:52 smithi045 conmon[39718]: : cluster [DBG] pgmap v5387: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:55.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:54 smithi154 conmon[34553]: cluster 2021-07-31T03:39:53.403869+0000 mgr.x (mgr.24542) 4831 : cluster [DBG] 2021-07-31T03:39:55.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:54 smithi154 conmon[34553]: pgmap v5388: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:55.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:54 smithi045 conmon[39718]: cluster 2021-07-31T03:39:53.403869+0000 2021-07-31T03:39:55.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:54 smithi045 conmon[39718]: mgr.x (mgr.24542) 4831 : cluster [DBG] pgmap v5388: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:55.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:54 smithi045 conmon[33655]: cluster 2021-07-31T03:39:53.403869+0000 mgr.x (mgr.24542) 4831 : 2021-07-31T03:39:55.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:54 smithi045 conmon[33655]: cluster [DBG] pgmap v5388: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:57.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:56 smithi154 conmon[34553]: cluster 2021-07-31T03:39:55.404305+0000 mgr.x (mgr.24542) 4832 : cluster [DBG] pgmap v5389: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:57.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:56 smithi045 conmon[39718]: cluster 2021-07-31T03:39:55.404305+0000 mgr.x (mgr.24542) 4832 : cluster [DBG] pgmap v5389: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:57.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:56 smithi045 conmon[33655]: cluster 2021-07-31T03:39:55.404305+0000 mgr.x (mgr.24542) 4832 : cluster 2021-07-31T03:39:57.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:56 smithi045 conmon[33655]: [DBG] pgmap v5389: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:59.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:39:58 smithi154 conmon[34553]: cluster 2021-07-31T03:39:57.404850+0000 mgr.x (mgr.24542) 4833 : cluster [DBG] pgmap v5390: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:59.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:58 smithi045 conmon[33655]: cluster 2021-07-31T03:39:57.404850+0000 mgr.x (mgr. 2021-07-31T03:39:59.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:39:58 smithi045 conmon[33655]: 24542) 4833 : cluster [DBG] pgmap v5390: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:39:59.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:58 smithi045 conmon[39718]: cluster 2021-07-31T03:39:57.404850+0000 mgr.x (mgr.24542) 4833 : cluster [DBG] 2021-07-31T03:39:59.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:39:58 smithi045 conmon[39718]: pgmap v5390: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:01.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:00 smithi154 conmon[34553]: cluster 2021-07-31T03:39:59.405352+0000 mgr.x (mgr.24542) 4834 : cluster [DBG] pgmap v5391: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:01.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:00 smithi154 conmon[34553]: cluster 2021-07-31T03:40:00.000172+0000 mon.a (mon.0) 3343 : cluster [INF] overall HEALTH_OK 2021-07-31T03:40:01.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:00 smithi045 conmon[39718]: cluster 2021-07-31T03:39:59.405352+0000 mgr.x (mgr.24542) 4834 : cluster [DBG] pgmap v5391: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:01.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:00 smithi045 conmon[39718]: cluster 2021-07-31T03:40:00.000172+0000 mon.a (mon.0) 3343 : cluster [INF] overall HEALTH_OK 2021-07-31T03:40:01.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:00 smithi045 conmon[33655]: cluster 2021-07-31T03:39:59.405352+0000 mgr.x (mgr 2021-07-31T03:40:01.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:00 smithi045 conmon[33655]: .24542) 4834 : cluster [DBG] pgmap v5391: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:01.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:00 smithi045 conmon[33655]: cluster 2021-07-31T03:40:00.000172+0000 mon.a (mon.0) 3343 : cluster [INF] overall HEALTH_OK 2021-07-31T03:40:03.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:02 smithi154 conmon[34553]: cluster 2021-07-31T03:40:01.405859+0000 mgr.x (mgr.24542) 4835 : cluster [DBG] 2021-07-31T03:40:03.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:02 smithi154 conmon[34553]: pgmap v5392: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:03.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:02 smithi045 conmon[33655]: cluster 2021-07-31T03:40:01.405859+0000 mgr.x 2021-07-31T03:40:03.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:02 smithi045 conmon[33655]: (mgr.24542) 4835 : cluster [DBG] pgmap v5392: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:03.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:02 smithi045 conmon[39718]: cluster 2021-07-31T03:40:01.405859+0000 2021-07-31T03:40:03.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:02 smithi045 conmon[39718]: mgr.x (mgr.24542) 4835 : cluster [DBG] pgmap v5392: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:05.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:04 smithi154 conmon[34553]: cluster 2021-07-31T03:40:03.406435+0000 mgr.x (mgr.24542) 4836 : cluster [DBG] pgmap v5393: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:05.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:04 smithi045 conmon[33655]: cluster 2021-07-31T03:40:03.406435+0000 mgr.x (mgr.24542) 4836 : cluster [DBG] pgmap v5393: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:05.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:04 smithi045 conmon[39718]: cluster 2021-07-31T03:40:03.406435+0000 mgr.x ( 2021-07-31T03:40:05.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:04 smithi045 conmon[39718]: mgr.24542) 4836 : cluster [DBG] pgmap v5393: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:07.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:06 smithi154 conmon[34553]: cluster 2021-07-31T03:40:05.406997+0000 mgr.x (mgr.24542) 4837 : cluster [DBG] pgmap v5394: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:07.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:06 smithi045 conmon[39718]: cluster 2021-07-31T03:40:05.406997+0000 mgr.x ( 2021-07-31T03:40:07.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:06 smithi045 conmon[39718]: mgr.24542) 4837 : cluster [DBG] pgmap v5394: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:07.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:06 smithi045 conmon[33655]: cluster 2021-07-31T03:40:05.406997+0000 mgr.x (mgr.24542 2021-07-31T03:40:07.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:06 smithi045 conmon[33655]: ) 4837 : cluster [DBG] pgmap v5394: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:09.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:08 smithi154 conmon[34553]: cluster 2021-07-31T03:40:07.407535+0000 mgr.x (mgr.24542) 4838 : cluster [DBG] pgmap v5395: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:09.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:08 smithi045 conmon[39718]: cluster 2021-07-31T03:40:07.407535+0000 mgr.x (mgr 2021-07-31T03:40:09.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:08 smithi045 conmon[39718]: .24542) 4838 : cluster [DBG] pgmap v5395: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:09.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:08 smithi045 conmon[33655]: cluster 2021-07-31T03:40:07.407535+0000 mgr.x (mgr.24542 2021-07-31T03:40:09.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:08 smithi045 conmon[33655]: ) 4838 : cluster [DBG] pgmap v5395: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:11.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:10 smithi154 conmon[34553]: cluster 2021-07-31T03:40:09.408034+0000 mgr.x (mgr.24542) 4839 : cluster [DBG] pgmap v5396: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:11.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:10 smithi045 conmon[39718]: cluster 2021-07-31T03:40:09.408034+0000 mgr.x (mgr.24542) 4839 : cluster 2021-07-31T03:40:11.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:10 smithi045 conmon[39718]: [DBG] pgmap v5396: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:11.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:10 smithi045 conmon[33655]: cluster 2021-07-31T03:40:09.408034+0000 2021-07-31T03:40:11.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:10 smithi045 conmon[33655]: mgr.x (mgr.24542) 4839 : cluster [DBG] pgmap v5396: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:13.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:12 smithi154 conmon[34553]: cluster 2021-07-31T03:40:11.408392+0000 mgr.x (mgr.24542) 4840 : cluster [DBG] pgmap v5397: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:13.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:12 smithi045 conmon[39718]: cluster 2021-07-31T03:40:11.408392+0000 2021-07-31T03:40:13.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:12 smithi045 conmon[39718]: mgr.x (mgr.24542) 4840 : cluster [DBG] pgmap v5397: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:13.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:12 smithi045 conmon[33655]: cluster 2021-07-31T03:40:11.408392+0000 mgr.x ( 2021-07-31T03:40:13.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:12 smithi045 conmon[33655]: mgr.24542) 4840 : cluster [DBG] pgmap v5397: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:15.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:14 smithi154 conmon[34553]: cluster 2021-07-31T03:40:13.408776+0000 mgr.x (mgr.24542) 4841 : cluster [DBG] pgmap v5398: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:15.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:14 smithi154 conmon[34553]: 2021-07-31T03:40:15.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:14 smithi045 conmon[33655]: cluster 2021-07-31T03:40:13.408776+0000 mgr.x (mgr.24542) 4841 : cluster [DBG] pgmap v5398: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:15.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:14 smithi045 conmon[33655]: 2021-07-31T03:40:15.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:14 smithi045 conmon[39718]: cluster 2021-07-31T03:40:13.408776+0000 mgr.x 2021-07-31T03:40:15.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:14 smithi045 conmon[39718]: (mgr.24542) 4841 : cluster [DBG] pgmap v5398: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:17.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:16 smithi154 conmon[34553]: cluster 2021-07-31T03:40:15.409410+0000 mgr.x (mgr.24542) 4842 : cluster [DBG] pgmap v5399: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:17.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:16 smithi045 conmon[39718]: cluster 2021-07-31T03:40:15.409410+0000 mgr.x (mgr. 2021-07-31T03:40:17.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:16 smithi045 conmon[39718]: 24542) 4842 : cluster [DBG] pgmap v5399: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:17.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:16 smithi045 conmon[33655]: cluster 2021-07-31T03:40:15.409410+0000 mgr.x (mgr.24542) 4842 : cluster [DBG] pgmap v5399: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:19.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:18 smithi154 conmon[34553]: cluster 2021-07-31T03:40:17.409913+0000 mgr.x (mgr.24542) 4843 : cluster [DBG] pgmap v5400: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:19.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:18 smithi154 conmon[34553]: 2021-07-31T03:40:19.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:18 smithi045 conmon[39718]: cluster 2021-07-31T03:40:17.409913+0000 mgr.x (mgr.24542 2021-07-31T03:40:19.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:18 smithi045 conmon[39718]: ) 4843 : cluster [DBG] pgmap v5400: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:19.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:18 smithi045 conmon[33655]: cluster 2021-07-31T03:40:17.409913+0000 mgr.x (mgr.24542) 4843 : cluster [DBG] pgmap v5400: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:20.824 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:40:21.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:20 smithi154 conmon[34553]: cluster 2021-07-31T03:40:19.410422+0000 mgr.x (mgr.24542) 4844 : cluster [DBG] pgmap v5401: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:21.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:20 smithi045 conmon[39718]: cluster 2021-07-31T03:40:19.410422+0000 mgr.x 2021-07-31T03:40:21.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:20 smithi045 conmon[39718]: (mgr.24542) 4844 : cluster [DBG] pgmap v5401: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:21.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:20 smithi045 conmon[33655]: cluster 2021-07-31T03:40:19.410422+0000 mgr.x 2021-07-31T03:40:21.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:20 smithi045 conmon[33655]: (mgr.24542) 4844 : cluster [DBG] pgmap v5401: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:21.101 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:40:21.101 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 115m ago 116m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:40:21.101 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 112m ago 115m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:40:21.102 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 112m ago 119m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:40:21.102 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 115m ago 2h - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:40:21.102 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 115m ago 2h - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:40:21.102 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 112m ago 119m 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:40:21.103 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 115m ago 119m - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:40:21.103 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 115m ago 116m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:40:21.103 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 112m ago 116m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:40:21.103 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 115m ago 118m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:40:21.103 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 115m ago 118m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:40:21.104 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 115m ago 118m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:40:21.104 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 115m ago 117m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:40:21.104 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 112m ago 117m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:40:21.104 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 112m ago 117m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:40:21.105 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 112m ago 116m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:40:21.105 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 112m ago 116m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:40:21.105 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 112m ago 116m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:40:21.440 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:40:21.440 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:40:21.440 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:40:21.440 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:40:21.441 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:40:21.441 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:40:21.441 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:40:21.441 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:40:21.442 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:40:21.442 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:40:21.442 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:40:21.442 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:40:21.442 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:40:21.443 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:40:21.443 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:40:21.443 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:40:21.443 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:40:22.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:21 smithi154 conmon[34553]: audit 2021-07-31T03:40:20.812596+0000 mgr.x (mgr.24542) 4845 : audit [DBG] from='client.29932 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:40:22.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:21 smithi154 conmon[34553]: audit 2021-07-31T03:40:21.094679+0000 mgr.x (mgr.24542) 4846 : audit [DBG] from='client.19998 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:40:22.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:21 smithi154 conmon[34553]: audit 2021-07-31T03:40:21.441000+0000 mon.c (mon.1) 222 : audit [DBG] from='client.? 172.21.15.45:0/1375748212' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:40:22.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:21 smithi045 conmon[39718]: audit 2021-07-31T03: 2021-07-31T03:40:22.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:21 smithi045 conmon[39718]: 40:20.812596+0000 mgr.x (mgr.24542) 4845 : audit [DBG] from='client.29932 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:40:22.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:21 smithi045 conmon[39718]: audit 2021-07-31T03:40:22.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:21 smithi045 conmon[39718]: 2021-07-31 2021-07-31T03:40:22.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:21 smithi045 conmon[39718]: T03:40:21.094679+0000 2021-07-31T03:40:22.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:21 smithi045 conmon[39718]: mgr.x (mgr.24542) 2021-07-31T03:40:22.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:21 smithi045 conmon[39718]: 4846 : audit [DBG] from='client.19998 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:40:22.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:21 smithi045 conmon[39718]: 2021-07-31T03:40:22.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:21 smithi045 conmon[39718]: audit 2021-07-31T03:40:22.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:21 smithi045 conmon[39718]: 2021-07-31T 2021-07-31T03:40:22.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:21 smithi045 conmon[39718]: 03:40:21.441000+0000 2021-07-31T03:40:22.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:21 smithi045 conmon[39718]: mon.c (mon.1) 222 2021-07-31T03:40:22.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:21 smithi045 conmon[39718]: : audit [DBG] from='client.? 172.21.15.45:0/1375748212' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:40:22.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:21 smithi045 conmon[33655]: audit 2021-07-31T03:40:20.812596+0000 mgr.x (mgr.24542) 4845 : audit 2021-07-31T03:40:22.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:21 smithi045 conmon[33655]: [DBG] from='client.29932 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:40:22.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:21 smithi045 conmon[33655]: audit 2021-07-31T03:40:21.094679+0000 mgr.x (mgr.24542) 4846 : audit [DBG] from='client.19998 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:40:22.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:21 smithi045 conmon[33655]: audit 2021- 2021-07-31T03:40:22.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:21 smithi045 conmon[33655]: 07-31T03:40:21.441000+0000 mon.c (mon.1) 222 2021-07-31T03:40:22.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:21 smithi045 conmon[33655]: : audit [DBG] from='client.? 172.21.15.45:0/1375748212' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:40:23.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:22 smithi154 conmon[34553]: cluster 2021-07-31T03:40:21.410780+0000 mgr.x (mgr.24542) 4847 2021-07-31T03:40:23.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:22 smithi154 conmon[34553]: : cluster [DBG] pgmap v5402: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:23.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:22 smithi045 conmon[39718]: cluster 2021-07-31T03:40:21. 2021-07-31T03:40:23.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:22 smithi045 conmon[39718]: 410780+0000 mgr.x (mgr.24542) 4847 : cluster [DBG] pgmap v5402: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:23.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:22 smithi045 conmon[33655]: cluster 2021-07-31T03:40:21.410780+0000 mgr.x (mgr.24542) 4847 : cluster [DBG] 2021-07-31T03:40:23.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:22 smithi045 conmon[33655]: pgmap v5402: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:25.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:24 smithi154 conmon[34553]: cluster 2021-07-31T03:40:23.411208+0000 mgr.x (mgr.24542) 4848 : cluster [DBG] pgmap v5403: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:25.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:24 smithi045 conmon[33655]: cluster 2021-07-31T03:40:23.411208+0000 mgr.x (mgr.24542) 4848 : cluster [DBG] 2021-07-31T03:40:25.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:24 smithi045 conmon[33655]: pgmap v5403: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:25.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:24 smithi045 conmon[39718]: cluster 2021-07-31T03:40:23.411208 2021-07-31T03:40:25.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:24 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 4848 : cluster [DBG] pgmap v5403: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:27.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:26 smithi154 conmon[34553]: cluster 2021-07-31T03:40:25.411651+0000 mgr.x (mgr.24542) 4849 : cluster 2021-07-31T03:40:27.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:26 smithi154 conmon[34553]: [DBG] pgmap v5404: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:27.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:26 smithi045 conmon[33655]: cluster 2021-07-31T03:40:25.411651+0000 mgr.x (mgr.24542) 4849 2021-07-31T03:40:27.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:26 smithi045 conmon[33655]: : cluster [DBG] pgmap v5404: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:27.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:26 smithi045 conmon[39718]: cluster 2021-07-31T03:40:25. 2021-07-31T03:40:27.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:26 smithi045 conmon[39718]: 411651+0000 mgr.x (mgr.24542) 4849 : cluster [DBG] pgmap v5404: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:29.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:28 smithi154 conmon[34553]: cluster 2021-07-31T03:40:27.412148+0000 mgr.x (mgr.24542) 4850 : cluster [DBG] pgmap v5405: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:29.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:28 smithi154 conmon[34553]: audit 2021-07-31T03:40:28.385691+0000 mon.b (mon.2) 640 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:40:29.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:28 smithi154 conmon[34553]: audit 2021-07-31T03:40:28.386022+0000 mon.b (mon.2) 641 : audit 2021-07-31T03:40:29.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:28 smithi154 conmon[34553]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:40:29.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:28 smithi154 conmon[34553]: audit 2021-07-31T03:40:29.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:28 smithi154 conmon[34553]: 2021-07-31T03:40:28.386130 2021-07-31T03:40:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:28 smithi154 conmon[34553]: +0000 mon.a (mon.0) 3344 : audit [INF] 2021-07-31T03:40:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:28 smithi154 conmon[34553]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:40:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:28 smithi154 conmon[34553]: audit 2021-07- 2021-07-31T03:40:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:28 smithi154 conmon[34553]: 31T03:40:28.386458+0000 2021-07-31T03:40:29.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:28 smithi154 conmon[34553]: mon.a (mon.0) 3345 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:40:29.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:28 smithi154 conmon[34553]: 2021-07-31T03:40:29.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:28 smithi045 conmon[39718]: cluster 2021-07-31T03:40:27. 2021-07-31T03:40:29.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:28 smithi045 conmon[39718]: 412148+0000 mgr.x (mgr.24542) 4850 : cluster [DBG] pgmap v5405: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:29.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:28 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T03:40:29.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:28 smithi045 conmon[39718]: 03:40:28.385691+0000 2021-07-31T03:40:29.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:28 smithi045 conmon[39718]: mon.b (mon.2) 640 : 2021-07-31T03:40:29.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:28 smithi045 conmon[39718]: audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:40:29.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:28 smithi045 conmon[39718]: audit 2021- 2021-07-31T03:40:29.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:28 smithi045 conmon[39718]: 07-31T03:40:28.386022 2021-07-31T03:40:29.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:28 smithi045 conmon[39718]: +0000 mon.b (mon.2) 641 2021-07-31T03:40:29.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:28 smithi045 conmon[39718]: : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:40:29.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:28 smithi045 conmon[39718]: audit 2021 2021-07-31T03:40:29.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:28 smithi045 conmon[39718]: -07-31T03:40:28. 2021-07-31T03:40:29.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:28 smithi045 conmon[39718]: 386130+0000 mon.a (mon.0 2021-07-31T03:40:29.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:28 smithi045 conmon[39718]: ) 3344 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:40:29.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:28 smithi045 conmon[39718]: 2021-07-31T03:40:29.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:28 smithi045 conmon[39718]: audit 2021-07-31T03:40:28.386458+0000 mon.a ( 2021-07-31T03:40:29.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:28 smithi045 conmon[39718]: mon.0) 3345 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:40:29.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:28 smithi045 conmon[33655]: cluster 2021-07-31T03:40:27.412148+0000 mgr.x ( 2021-07-31T03:40:29.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:28 smithi045 conmon[33655]: mgr.24542) 4850 : cluster [DBG] pgmap v5405: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:29.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:28 smithi045 conmon[33655]: audit 2021-07-31T03:40:28.385691+0000 mon.b (mon.2) 640 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:40:29.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:28 smithi045 conmon[33655]: 2021-07-31T03:40:29.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:28 smithi045 conmon[33655]: audit 2021- 2021-07-31T03:40:29.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:28 smithi045 conmon[33655]: 07-31T03:40:28.386022+0000 mon.b (mon.2) 641 : audit 2021-07-31T03:40:29.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:28 smithi045 conmon[33655]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:40:29.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:28 smithi045 conmon[33655]: audit 2021- 2021-07-31T03:40:29.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:28 smithi045 conmon[33655]: 07-31T03:40:28.386130+0000 2021-07-31T03:40:29.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:28 smithi045 conmon[33655]: mon.a (mon.0) 3344 : 2021-07-31T03:40:29.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:28 smithi045 conmon[33655]: audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:40:29.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:28 smithi045 conmon[33655]: audit 2021 2021-07-31T03:40:29.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:28 smithi045 conmon[33655]: -07-31T03:40:28 2021-07-31T03:40:29.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:28 smithi045 conmon[33655]: .386458+0000 mon.a (mon. 2021-07-31T03:40:29.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:28 smithi045 conmon[33655]: 0) 3345 : audit [INF] 2021-07-31T03:40:29.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:28 smithi045 conmon[33655]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:40:31.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:30 smithi154 conmon[34553]: cluster 2021-07-31T03:40:29.412577+0000 mgr.x (mgr.24542) 4851 : cluster [DBG] pgmap v5406: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:31.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:30 smithi045 conmon[33655]: cluster 2021-07-31T03:40:29.412577+0000 mgr.x (mgr.24542) 4851 : cluster [DBG] pgmap v5406: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:31.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:30 smithi045 conmon[33655]: 2021-07-31T03:40:31.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:30 smithi045 conmon[39718]: cluster 2021-07-31T03:40:29.412577 2021-07-31T03:40:31.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:30 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 4851 : cluster [DBG] pgmap v5406: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:33.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:32 smithi154 conmon[34553]: cluster 2021-07-31T03:40:31.413026+0000 mgr.x (mgr.24542) 4852 : cluster [DBG] pgmap v5407: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:33.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:32 smithi045 conmon[33655]: cluster 2021-07-31T03:40:31.413026+0000 mgr.x (mgr.24542) 4852 : 2021-07-31T03:40:33.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:32 smithi045 conmon[33655]: cluster [DBG] pgmap v5407: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:33.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:32 smithi045 conmon[39718]: cluster 2021-07-31T03:40:31. 2021-07-31T03:40:33.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:32 smithi045 conmon[39718]: 413026+0000 mgr.x (mgr.24542) 4852 : cluster [DBG] pgmap v5407: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:34.752 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: debug 2021-07-31T03:40:34.510+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:40:34.752 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:40:34.752 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:40:34.753 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: yield 2021-07-31T03:40:34.753 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:40:34.753 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:40:34.753 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:40:34.753 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:40:34.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:40:34.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:40:34.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:40:34.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:40:34.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:40:34.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: task.result() 2021-07-31T03:40:34.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:40:34.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: debug 2021-07-31T03:40:34.521+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:40:34.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:40:34.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:40:34.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: yield 2021-07-31T03:40:34.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:40:34.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:40:34.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:40:34.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:40:34.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:40:34.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:40:34.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:40:34.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:40:34.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:40:34.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: task.result() 2021-07-31T03:40:34.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:40:34.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: debug 2021-07-31T03:40:34.546+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:40:34.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:40:34.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:40:34.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: yield 2021-07-31T03:40:34.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:40:34.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:40:34.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:40:34.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:40:34.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:40:34.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:40:34.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:40:34.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:40:34.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:40:34.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: task.result() 2021-07-31T03:40:34.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:40:34.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: debug 2021-07-31T03:40:34.547+0000 7fcb9115c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:40:34.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:40:34.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:40:34.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: yield 2021-07-31T03:40:34.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:40:34.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:40:34.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:40:34.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:40:34.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:40:34.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:40:34.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:40:34.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:40:34.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:40:34.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: task.result() 2021-07-31T03:40:34.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:40:34.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: 2021-07-31T03:40:34.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:40:34.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: 2021-07-31T03:40:34.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:40:34.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:40:34.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:40:34.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:40:34.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:40:34.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:40:34.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:40:34.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:40:34.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:40:34.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:40:34.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:40:34.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:40:34.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:40:34.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:40:34.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:40:34.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:40:34.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:40:34.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:40:34.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: raise 2021-07-31T03:40:34.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:40:34.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:40:34.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:40:34.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:40:34.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:40:34.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: debug 2021-07-31T03:40:34.556+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:40:34.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:40:34.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:40:34.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: yield 2021-07-31T03:40:34.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:40:34.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:40:34.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:40:34.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:40:34.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:40:34.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:40:34.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:40:34.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:40:34.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:40:34.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: task.result() 2021-07-31T03:40:34.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:40:34.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: debug 2021-07-31T03:40:34.557+0000 7fcb9095b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:40:34.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:40:34.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:40:34.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: yield 2021-07-31T03:40:34.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:40:34.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:40:34.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:40:34.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:40:34.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:40:34.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:40:34.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:40:34.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:40:34.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:40:34.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: task.result() 2021-07-31T03:40:34.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:40:34.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: 2021-07-31T03:40:34.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:40:34.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: 2021-07-31T03:40:34.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:40:34.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:40:34.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:40:34.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:40:34.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:40:34.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:40:34.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:40:34.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:40:34.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:40:34.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:40:34.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:40:34.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:40:34.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:40:34.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:40:34.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:40:34.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:40:34.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:40:34.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:40:34.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: raise 2021-07-31T03:40:34.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:40:34.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:40:34.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:40:34.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:40:34.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:40:35.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:34 smithi045 conmon[33655]: cluster 2021-07-31T03:40:33.413492+0000 mgr.x (mgr.24542 2021-07-31T03:40:35.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:34 smithi045 conmon[33655]: ) 4853 : cluster [DBG] pgmap v5408: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:35.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:34 smithi045 conmon[33655]: audit 2021-07-31T03:40:34.465123+0000 mon.b (mon 2021-07-31T03:40:35.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:34 smithi045 conmon[33655]: .2) 642 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:40:35.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:34 smithi045 conmon[33655]: audit 2021-07-31T03:40:34.466685+0000 2021-07-31T03:40:35.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:34 smithi045 conmon[33655]: mon.b (mon.2) 643 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:40:35.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:34 smithi045 conmon[33655]: audit 2021-07-31T03:40:34.467519+0000 2021-07-31T03:40:35.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:34 smithi045 conmon[33655]: mon.b (mon.2) 644 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:40:35.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:34 smithi045 conmon[39718]: cluster 2021-07-31T03:40:33.413492+0000 mgr.x (mgr.24542) 4853 : cluster 2021-07-31T03:40:35.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:34 smithi045 conmon[39718]: [DBG] pgmap v5408: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:35.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:34 smithi045 conmon[39718]: audit 2021-07-31T03:40:34.465123+0000 mon.b (mon.2) 642 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:40:35.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:34 smithi045 conmon[39718]: audit 2021-07-31T03:40:34.466685+0000 mon.b (mon.2) 643 : audit 2021-07-31T03:40:35.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:34 smithi045 conmon[39718]: [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:40:35.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:34 smithi045 conmon[39718]: audit 2021-07 2021-07-31T03:40:35.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:34 smithi045 conmon[39718]: -31T03:40:34.467519+0000 mon.b (mon.2) 644 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:40:35.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[34553]: cluster 2021-07-31T03:40:33.413492+0000 2021-07-31T03:40:35.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[34553]: mgr.x (mgr.24542) 4853 : cluster [DBG] pgmap v5408: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:35.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[34553]: audit 2021-07- 2021-07-31T03:40:35.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[34553]: 31T03:40:34.465123+0000 mon.b (mon.2) 642 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:40:35.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[34553]: audit 2021-07-31T03:40:35.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[34553]: 2021-07-31T03:40:34.466685+0000 2021-07-31T03:40:35.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[34553]: mon.b (mon.2) 643 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:40:35.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[34553]: 2021-07-31T03:40:35.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[34553]: audit 2021-07-31T03:40:34 2021-07-31T03:40:35.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:34 smithi154 conmon[34553]: .467519+0000 mon.b (mon.2) 644 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:40:36.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:40:34.468279+0000 mgr.x (mgr 2021-07-31T03:40:36.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: .24542) 4854 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:40:36.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:40:34.511105+0000 mgr.x (mgr.24542) 4855 : cephadm [ERR] 2021-07-31T03:40:36.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: 2021-07-31T03:40:36.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:40:36.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:40:36.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: yield 2021-07-31T03:40:36.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:40:36.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:40:36.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:40:36.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:40:36.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:40:36.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:40:36.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:40:36.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:40:36.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:40:36.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: task.result() 2021-07-31T03:40:36.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:40:36.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:40:36.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: 2021-07-31T03:40:34.511485+0000 mgr.x (mgr.24542) 4856 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:40:36.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:40:36.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: 2021-07-31T03:40: 2021-07-31T03:40:36.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: 34.522510+0000 mgr.x (mgr. 2021-07-31T03:40:36.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: 24542) 4857 : cephadm [ERR] 2021-07-31T03:40:36.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:40:36.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:40:36.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: yield 2021-07-31T03:40:36.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:40:36.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:40:36.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:40:36.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:40:36.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:40:36.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:40:36.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:40:36.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:40:36.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:40:36.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: task.result() 2021-07-31T03:40:36.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:40:36.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: 2021-07-31T03:40:36.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:40:34.523091+0000 mgr.x (mgr.24542) 4858 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:40:36.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:40:36.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: 2021- 2021-07-31T03:40:36.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:40:34.468279+0000 mgr.x (mgr 2021-07-31T03:40:36.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: .24542) 4854 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:40:36.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:40:36.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: 2021-07-31T03:40:34.511105+0000 mgr.x (mgr.24542) 4855 : cephadm [ERR] 2021-07-31T03:40:36.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:40:36.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:40:36.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: yield 2021-07-31T03:40:36.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:40:36.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:40:36.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:40:36.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:40:36.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:40:36.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:40:36.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:40:36.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:40:36.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:40:36.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: task.result() 2021-07-31T03:40:36.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:40:36.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:40:34.511485+0000 mgr.x (mgr.24542) 4856 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:40:36.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:40:34 2021-07-31T03:40:36.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: .522510+0000 mgr.x (mgr.24542) 4857 : cephadm [ERR] 2021-07-31T03:40:36.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:40:36.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:40:36.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: yield 2021-07-31T03:40:36.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:40:36.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:40:36.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:40:36.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:40:36.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:40:36.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:40:36.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:40:36.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:40:36.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:40:36.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: task.result() 2021-07-31T03:40:36.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:40:36.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:40:34.523091+0000 mgr.x (mgr.24542) 4858 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:40:36.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:40:34.547910+0000 2021-07-31T03:40:36.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: mgr.x (mgr.24542) 4859 : cephadm [ERR] 2021-07-31T03:40:36.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:40:36.043 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: 07-31T03:40:34. 2021-07-31T03:40:36.043 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: 547910+0000 mgr.x (mgr.24542) 4859 : 2021-07-31T03:40:36.043 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: cephadm [ERR] 2021-07-31T03:40:36.044 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:40:36.044 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:40:36.044 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: yield 2021-07-31T03:40:36.044 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:40:36.044 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:40:36.045 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:40:36.045 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:40:36.045 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:40:36.045 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:40:36.045 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:40:36.046 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:40:36.046 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:40:36.046 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: task.result() 2021-07-31T03:40:36.046 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:40:36.046 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:40:34.548310+0000 2021-07-31T03:40:36.047 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: mgr.x (mgr.24542) 4860 : cephadm 2021-07-31T03:40:36.047 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:40:36.047 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:40:36.047 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:40:36.047 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: yield 2021-07-31T03:40:36.048 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:40:36.048 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:40:36.048 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:40:36.048 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:40:36.048 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:40:36.049 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:40:36.049 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:40:36.049 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:40:36.049 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:40:36.049 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: task.result() 2021-07-31T03:40:36.050 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:40:36.050 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: 2021-07-31T03:40:36.050 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:40:36.050 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: 2021-07-31T03:40:36.051 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:40:36.051 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:40:36.051 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:40:36.051 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:40:36.051 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:40:36.052 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:40:36.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:40:36.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: yield 2021-07-31T03:40:36.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:40:36.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:40:36.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:40:36.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:40:36.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:40:36.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:40:36.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:40:36.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:40:36.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:40:36.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: task.result() 2021-07-31T03:40:36.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:40:36.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:40:34.548310+0000 mgr.x (mgr.24542) 4860 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:40:36.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:40:36.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:40:36.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: yield 2021-07-31T03:40:36.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:40:36.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:40:36.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:40:36.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:40:36.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:40:36.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:40:36.057 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:40:36.057 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:40:36.057 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:40:36.057 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: task.result() 2021-07-31T03:40:36.057 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:40:36.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: 2021-07-31T03:40:36.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:40:36.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: 2021-07-31T03:40:36.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:40:36.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:40:36.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:40:36.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:40:36.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:40:36.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:40:36.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:40:36.060 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:40:36.060 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:40:36.060 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:40:36.060 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:40:36.061 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:40:36.061 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:40:36.061 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:40:36.061 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:40:36.061 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:40:36.062 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:40:36.062 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:40:36.062 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:40:36.062 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:40:36.062 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:40:36.063 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:40:36.063 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: raise 2021-07-31T03:40:36.063 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:40:36.063 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:40:36.063 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:40:36.064 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:40:36.064 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:40:36.064 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:40:34.557734+0000 mgr.x (mgr.24542) 4861 : cephadm [ERR] 2021-07-31T03:40:36.064 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:40:36.064 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:40:36.065 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: yield 2021-07-31T03:40:36.065 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:40:36.065 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:40:36.065 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:40:36.065 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:40:36.066 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:40:36.066 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:40:36.066 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:40:36.066 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:40:36.066 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:40:36.067 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: task.result() 2021-07-31T03:40:36.067 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:40:36.067 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:40:34.558276+0000 mgr.x (mgr.24542) 4862 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:40:36.067 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:40:36.067 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:40:36.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: yield 2021-07-31T03:40:36.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:40:36.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:40:36.068 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:40:36.069 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:40:36.069 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:40:36.069 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:40:36.069 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:40:36.069 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:40:36.070 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:40:36.070 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:40:36.070 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: raise 2021-07-31T03:40:36.070 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:40:36.070 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:40:36.071 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:40:36.071 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:40:36.071 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:40:36.071 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: cephadm 2021-07-31T03: 2021-07-31T03:40:36.071 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: 40:34.557734+0000 mgr.x (mgr.24542) 4861 : cephadm [ERR] 2021-07-31T03:40:36.072 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:40:36.072 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:40:36.072 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: yield 2021-07-31T03:40:36.072 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:40:36.072 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:40:36.073 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:40:36.073 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:40:36.073 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:40:36.073 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:40:36.073 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:40:36.074 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:40:36.074 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:40:36.074 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: task.result() 2021-07-31T03:40:36.074 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:40:36.074 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:40:34.558276+0000 mgr.x (mgr.24542) 4862 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:40:36.075 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:40:36.075 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:40:36.075 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: yield 2021-07-31T03:40:36.075 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:40:36.075 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:40:36.076 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:40:36.076 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:40:36.076 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:40:36.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:40:36.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:40:36.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:40:36.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:40:36.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:40:36.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:40:36.078 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:40:36.078 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: task.result() 2021-07-31T03:40:36.078 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:40:36.078 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: 2021-07-31T03:40:36.078 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:40:36.079 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: 2021-07-31T03:40:36.079 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:40:36.079 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:40:36.079 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:40:36.079 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:40:36.080 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:40:36.080 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:40:36.080 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:40:36.080 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:40:36.080 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:40:36.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:40:36.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:40:36.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:40:36.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:40:36.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:40:36.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:40:36.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:40:36.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:40:36.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:40:36.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: raise 2021-07-31T03:40:36.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:40:36.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:40:36.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:40:36.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:40:36.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:40:36.084 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:40:36.084 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:40:36.084 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:40:36.084 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:40:36.084 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: task.result() 2021-07-31T03:40:36.085 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:40:36.085 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: 2021-07-31T03:40:36.085 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:40:36.085 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: 2021-07-31T03:40:36.086 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:40:36.086 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:40:36.086 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:40:36.086 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:40:36.086 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:40:36.087 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:40:36.087 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:40:36.087 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:40:36.087 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:40:36.087 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:40:36.088 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:40:36.088 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:40:36.088 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:40:36.088 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:40:36.088 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:40:36.088 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:40:36.089 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:40:36.089 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:40:36.089 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: raise 2021-07-31T03:40:36.089 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:40:36.089 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:40:36.090 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:40:36.090 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:40:36.090 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:35 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:40:36.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:40:34.468279+0000 mgr.x (mgr.24542) 4854 : cephadm [WRN] 2021-07-31T03:40:36.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:40:36.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:40:34.511105+0000 mgr.x (mgr.24542) 4855 : cephadm [ERR] 2021-07-31T03:40:36.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:40:36.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:40:36.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: yield 2021-07-31T03:40:36.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:40:36.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:40:36.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:40:36.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:40:36.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:40:36.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:40:36.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:40:36.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:40:36.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:40:36.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: task.result() 2021-07-31T03:40:36.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:40:36.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:40:34.511485+0000 mgr.x (mgr.24542) 4856 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:40:36.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:40:34.522510+0000 mgr.x (mgr.24542) 4857 : cephadm [ERR] 2021-07-31T03:40:36.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:40:36.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:40:36.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: yield 2021-07-31T03:40:36.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:40:36.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:40:36.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:40:36.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:40:36.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:40:36.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:40:36.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:40:36.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:40:36.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:40:36.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: task.result() 2021-07-31T03:40:36.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:40:36.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:40:36.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: 2021-07-31T03:40:34.523091+0000 mgr.x (mgr.24542) 4858 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:40:36.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:40:34.547910+0000 mgr.x (mgr.24542) 4859 : cephadm [ERR] 2021-07-31T03:40:36.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:40:36.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:40:36.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: yield 2021-07-31T03:40:36.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:40:36.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:40:36.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:40:36.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:40:36.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:40:36.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:40:36.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:40:36.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:40:36.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:40:36.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: task.result() 2021-07-31T03:40:36.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:40:36.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:40:34.548310+0000 mgr.x (mgr.24542) 4860 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:40:36.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:40:36.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:40:36.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: yield 2021-07-31T03:40:36.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:40:36.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:40:36.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:40:36.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:40:36.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:40:36.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:40:36.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:40:36.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:40:36.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:40:36.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: task.result() 2021-07-31T03:40:36.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:40:36.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: 2021-07-31T03:40:36.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:40:36.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: 2021-07-31T03:40:36.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:40:36.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:40:36.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:40:36.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:40:36.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:40:36.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:40:36.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:40:36.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:40:36.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:40:36.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:40:36.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:40:36.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:40:36.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:40:36.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:40:36.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:40:36.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:40:36.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:40:36.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:40:36.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: raise 2021-07-31T03:40:36.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:40:36.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:40:36.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:40:36.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:40:36.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:40:36.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:40:34.557734+0000 mgr.x (mgr.24542) 4861 : cephadm [ERR] 2021-07-31T03:40:36.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: 2021-07-31T03:40:36.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:40:36.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:40:36.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: yield 2021-07-31T03:40:36.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:40:36.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:40:36.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:40:36.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:40:36.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:40:36.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:40:36.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:40:36.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:40:36.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:40:36.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: task.result() 2021-07-31T03:40:36.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:40:36.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:40:34.558276+0000 mgr.x (mgr.24542) 4862 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:40:36.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:40:36.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:40:36.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: yield 2021-07-31T03:40:36.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:40:36.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:40:36.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:40:36.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:40:36.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:40:36.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:40:36.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:40:36.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:40:36.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:40:36.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: task.result() 2021-07-31T03:40:36.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:40:36.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: 2021-07-31T03:40:36.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:40:36.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: 2021-07-31T03:40:36.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:40:36.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:40:36.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:40:36.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:40:36.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:40:36.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:40:36.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:40:36.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:40:36.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:40:36.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:40:36.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:40:36.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:40:36.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:40:36.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:40:36.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:40:36.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:40:36.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:40:36.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:40:36.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: raise 2021-07-31T03:40:36.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:40:36.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:40:36.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:40:36.285 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:40:36.285 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:35 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:40:37.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:36 smithi045 conmon[39718]: cluster 2021-07-31T03:40:35.414147+0000 mgr.x (mgr.24542) 4863 2021-07-31T03:40:37.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:36 smithi045 conmon[39718]: : cluster [DBG] pgmap v5409: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:37.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:36 smithi045 conmon[33655]: cluster 2021-07-31T03:40: 2021-07-31T03:40:37.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:36 smithi045 conmon[33655]: 35.414147+0000 mgr.x (mgr.24542) 4863 : cluster [DBG] pgmap v5409: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:37.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:36 smithi154 conmon[34553]: cluster 2021-07-31T03:40:35.414147+0000 mgr.x (mgr.24542) 4863 : cluster 2021-07-31T03:40:37.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:36 smithi154 conmon[34553]: [DBG] pgmap v5409: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:39.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:38 smithi045 conmon[39718]: cluster 2021-07-31T03:40:37.414534+0000 mgr.x (mgr.24542) 4864 : cluster [DBG] 2021-07-31T03:40:39.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:38 smithi045 conmon[39718]: pgmap v5410: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:39.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:38 smithi045 conmon[33655]: cluster 2021-07-31T03:40:37.414534+0000 mgr.x (mgr.24542) 4864 2021-07-31T03:40:39.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:38 smithi045 conmon[33655]: : cluster [DBG] pgmap v5410: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:39.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:38 smithi154 conmon[34553]: cluster 2021-07-31T03:40:37.414534+0000 mgr.x (mgr.24542) 4864 : 2021-07-31T03:40:39.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:38 smithi154 conmon[34553]: cluster [DBG] pgmap v5410: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:41.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:40 smithi045 conmon[33655]: cluster 2021-07-31T03:40:39.415009+0000 mgr.x (mgr.24542) 4865 : cluster [DBG] pgmap v5411: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:41.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:40 smithi045 conmon[39718]: cluster 2021-07-31T03:40:39.415009+0000 mgr.x ( 2021-07-31T03:40:41.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:40 smithi045 conmon[39718]: mgr.24542) 4865 : cluster [DBG] pgmap v5411: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:41.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:40 smithi154 conmon[34553]: cluster 2021-07-31T03:40:39.415009+0000 mgr.x (mgr.24542) 4865 : cluster 2021-07-31T03:40:41.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:40 smithi154 conmon[34553]: [DBG] pgmap v5411: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:43.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:42 smithi045 conmon[33655]: cluster 2021-07-31T03:40:41.415359+0000 mgr.x (mgr.24542) 4866 : cluster [DBG] pgmap v5412: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:43.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:42 smithi045 conmon[39718]: cluster 2021-07-31T03:40:41.415359+0000 mgr.x 2021-07-31T03:40:43.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:42 smithi045 conmon[39718]: (mgr.24542) 4866 : cluster [DBG] pgmap v5412: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:43.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:42 smithi154 conmon[34553]: cluster 2021-07-31T03:40:41.415359+0000 mgr.x (mgr.24542) 2021-07-31T03:40:43.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:42 smithi154 conmon[34553]: 4866 : cluster [DBG] pgmap v5412: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:45.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:44 smithi045 conmon[39718]: cluster 2021-07-31T03: 2021-07-31T03:40:45.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:44 smithi045 conmon[39718]: 40:43.415895+0000 mgr.x (mgr.24542) 4867 : cluster [DBG] pgmap v5413: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:45.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:44 smithi045 conmon[33655]: cluster 2021-07-31T03:40:43.415895+0000 mgr.x (mgr.24542) 4867 : cluster [DBG] 2021-07-31T03:40:45.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:44 smithi045 conmon[33655]: pgmap v5413: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:45.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:44 smithi154 conmon[34553]: cluster 2021-07-31T03:40:43.415895+0000 mgr.x (mgr.24542) 4867 2021-07-31T03:40:45.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:44 smithi154 conmon[34553]: : cluster [DBG] pgmap v5413: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:47.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:46 smithi154 conmon[34553]: cluster 2021-07-31T03:40:45.416379+0000 mgr.x (mgr.24542) 4868 : 2021-07-31T03:40:47.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:46 smithi154 conmon[34553]: cluster [DBG] pgmap v5414: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:47.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:46 smithi045 conmon[33655]: cluster 2021-07-31T03:40:45.416379+0000 mgr.x (mgr.24542) 4868 : cluster [DBG] pgmap v5414: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:47.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:46 smithi045 conmon[39718]: cluster 2021-07-31T03:40:45.416379 2021-07-31T03:40:47.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:46 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 4868 : cluster [DBG] pgmap v5414: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:49.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:48 smithi154 conmon[34553]: cluster 2021-07-31T03:40:47.416776+0000 mgr.x (mgr.24542) 4869 : cluster 2021-07-31T03:40:49.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:48 smithi154 conmon[34553]: [DBG] pgmap v5415: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:49.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:48 smithi045 conmon[33655]: cluster 2021-07-31T03:40:47. 2021-07-31T03:40:49.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:48 smithi045 conmon[33655]: 416776+0000 mgr.x (mgr.24542) 4869 : cluster [DBG] pgmap v5415: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:49.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:48 smithi045 conmon[39718]: cluster 2021-07-31T03:40:47.416776+0000 mgr.x ( 2021-07-31T03:40:49.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:48 smithi045 conmon[39718]: mgr.24542) 4869 : cluster [DBG] pgmap v5415: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:51.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:50 smithi154 conmon[34553]: cluster 2021-07-31T03:40:49.417266+0000 mgr.x (mgr.24542) 2021-07-31T03:40:51.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:50 smithi154 conmon[34553]: 4870 : cluster [DBG] pgmap v5416: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:51.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:50 smithi045 conmon[39718]: cluster 2021-07-31T03:40:49.417266 2021-07-31T03:40:51.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:50 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 4870 : cluster [DBG] pgmap v5416: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:51.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:50 smithi045 conmon[33655]: cluster 2021-07-31T03:40:49.417266+0000 mgr.x (mgr.24542) 4870 : cluster [DBG] pgmap v5416: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:51.753 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:40:52.031 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:40:52.031 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 115m ago 116m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:40:52.031 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 113m ago 116m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:40:52.032 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 113m ago 119m 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:40:52.032 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 115m ago 2h - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:40:52.032 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 115m ago 2h - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:40:52.032 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 113m ago 2h 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:40:52.033 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 115m ago 2h - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:40:52.033 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 115m ago 116m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:40:52.033 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 113m ago 116m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:40:52.033 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 115m ago 119m - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:40:52.034 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 115m ago 119m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:40:52.034 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 115m ago 118m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:40:52.034 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 115m ago 118m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:40:52.034 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 113m ago 118m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:40:52.034 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 113m ago 117m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:40:52.035 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 113m ago 117m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:40:52.035 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 113m ago 117m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:40:52.035 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 113m ago 116m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:40:52.369 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:40:52.369 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:40:52.370 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:40:52.370 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:40:52.370 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:40:52.370 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:40:52.370 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:40:52.371 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:40:52.371 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:40:52.371 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:40:52.371 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:40:52.371 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:40:52.372 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:40:52.372 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:40:52.372 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:40:52.372 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:40:52.372 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:40:53.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:52 smithi154 conmon[34553]: cluster 2021-07-31T03:40:51.417740+0000 mgr.x (mgr 2021-07-31T03:40:53.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:52 smithi154 conmon[34553]: .24542) 4871 : cluster [DBG] pgmap v5417: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:53.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:52 smithi154 conmon[34553]: audit 2021-07-31T03:40:51.742274+0000 mgr.x (mgr.24542) 4872 : audit [DBG] from='client.20010 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:40:53.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:52 smithi154 conmon[34553]: audit 2021-07-31T03:40:52.024426+0000 mgr.x (mgr 2021-07-31T03:40:53.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:52 smithi154 conmon[34553]: .24542) 4873 : audit [DBG] from='client.29956 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:40:53.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:52 smithi154 conmon[34553]: audit 2021-07 2021-07-31T03:40:53.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:52 smithi154 conmon[34553]: -31T03:40:52.369968+0000 mon.a (mon. 2021-07-31T03:40:53.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:52 smithi154 conmon[34553]: 0) 3346 : audit [DBG] from='client.? 172.21.15.45:0/472432219' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:40:53.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:52 smithi045 conmon[33655]: cluster 2021-07-31T03:40:51.417740+0000 mgr.x (mgr.24542) 4871 : 2021-07-31T03:40:53.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:52 smithi045 conmon[33655]: cluster [DBG] pgmap v5417: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:53.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:52 smithi045 conmon[33655]: audit 2021-07-31T03:40:51.742274+0000 mgr.x (mgr.24542) 4872 : audit [DBG] from='client.20010 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:40:53.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:52 smithi045 conmon[33655]: audit 2021-07-31T03:40:52.024426+0000 mgr.x (mgr.24542) 4873 : audit [DBG] from='client.29956 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:40:53.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:52 smithi045 conmon[33655]: audit 2021-07-31T03:40:52.369968+0000 mon.a 2021-07-31T03:40:53.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:52 smithi045 conmon[33655]: (mon.0) 3346 : audit [DBG] from='client.? 172.21.15.45:0/472432219' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:40:53.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:52 smithi045 conmon[39718]: cluster 2021-07-31T03:40:51.417740+0000 mgr.x 2021-07-31T03:40:53.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:52 smithi045 conmon[39718]: (mgr.24542) 4871 : cluster [DBG] pgmap v5417: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:53.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:52 smithi045 conmon[39718]: audit 2021-07-31T03:40:51. 2021-07-31T03:40:53.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:52 smithi045 conmon[39718]: 742274+0000 mgr.x (mgr.24542) 4872 : audit [DBG] from='client.20010 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:40:53.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:52 smithi045 conmon[39718]: audit 2021-07 2021-07-31T03:40:53.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:52 smithi045 conmon[39718]: -31T03:40:52.024426+0000 2021-07-31T03:40:53.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:52 smithi045 conmon[39718]: mgr.x (mgr.24542) 4873 : audit [DBG] 2021-07-31T03:40:53.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:52 smithi045 conmon[39718]: from='client.29956 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:40:53.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:52 smithi045 conmon[39718]: audit 2021-07-31T03:40:52 2021-07-31T03:40:53.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:52 smithi045 conmon[39718]: .369968+0000 mon.a (mon.0) 3346 : audit [DBG] from='client.? 172.21.15.45:0/472432219' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:40:55.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:54 smithi154 conmon[34553]: cluster 2021-07-31T03:40:53.418120+0000 mgr.x (mgr.24542) 4874 : cluster [DBG] pgmap v5418: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:55.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:54 smithi045 conmon[39718]: cluster 2021-07-31T03:40:53.418120+0000 mgr.x (mgr 2021-07-31T03:40:55.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:54 smithi045 conmon[39718]: .24542) 4874 : cluster [DBG] pgmap v5418: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:55.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:54 smithi045 conmon[33655]: cluster 2021-07-31T03:40:53.418120+0000 mgr.x (mgr.24542) 4874 2021-07-31T03:40:55.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:54 smithi045 conmon[33655]: : cluster [DBG] pgmap v5418: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:57.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:56 smithi154 conmon[34553]: cluster 2021-07-31T03:40:55.418544+0000 mgr.x (mgr.24542) 4875 : cluster [DBG] pgmap v5419: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:57.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:56 smithi045 conmon[33655]: cluster 2021-07-31T03:40:55.418544+0000 mgr.x (mgr.24542) 4875 : 2021-07-31T03:40:57.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:56 smithi045 conmon[33655]: cluster [DBG] pgmap v5419: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:57.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:56 smithi045 conmon[39718]: cluster 2021-07-31T03:40:55.418544+0000 mgr.x (mgr 2021-07-31T03:40:57.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:56 smithi045 conmon[39718]: .24542) 4875 : cluster [DBG] pgmap v5419: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:59.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:40:58 smithi154 conmon[34553]: cluster 2021-07-31T03:40:57.419075+0000 mgr.x (mgr.24542) 4876 : cluster [DBG] pgmap v5420: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:59.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:58 smithi045 conmon[33655]: cluster 2021-07-31T03:40:57.419075+0000 2021-07-31T03:40:59.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:40:58 smithi045 conmon[33655]: mgr.x (mgr.24542) 4876 : cluster [DBG] pgmap v5420: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:40:59.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:58 smithi045 conmon[39718]: cluster 2021-07-31T03:40:57.419075+0000 mgr.x (mgr.24542) 2021-07-31T03:40:59.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:40:58 smithi045 conmon[39718]: 4876 : cluster [DBG] pgmap v5420: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:01.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:00 smithi154 conmon[34553]: cluster 2021-07-31T03:40:59.419529+0000 mgr.x (mgr.24542) 4877 : cluster [DBG] pgmap v5421: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:01.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:00 smithi045 conmon[33655]: cluster 2021-07-31T03:40:59.419529+0000 mgr.x (mgr.24542) 4877 2021-07-31T03:41:01.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:00 smithi045 conmon[33655]: : cluster [DBG] pgmap v5421: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:01.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:00 smithi045 conmon[39718]: cluster 2021-07-31T03:40:59 2021-07-31T03:41:01.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:00 smithi045 conmon[39718]: .419529+0000 mgr.x (mgr.24542) 4877 : cluster [DBG] pgmap v5421: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:03.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:02 smithi154 conmon[34553]: cluster 2021-07-31T03:41:01.420030+0000 mgr.x (mgr.24542) 4878 : cluster [DBG] pgmap v5422: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:03.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:02 smithi045 conmon[33655]: cluster 2021-07-31T03:41:01.420030+0000 2021-07-31T03:41:03.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:02 smithi045 conmon[33655]: mgr.x (mgr.24542) 4878 : cluster [DBG] pgmap v5422: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:03.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:02 smithi045 conmon[39718]: cluster 2021-07-31T03:41:01.420030+0000 2021-07-31T03:41:03.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:02 smithi045 conmon[39718]: mgr.x (mgr.24542) 4878 : cluster [DBG] pgmap v5422: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:05.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:04 smithi154 conmon[34553]: cluster 2021-07-31T03:41:03.420424+0000 mgr.x (mgr.24542) 4879 : cluster [DBG] pgmap v5423: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:05.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:04 smithi045 conmon[33655]: cluster 2021-07-31T03:41:03.420424+0000 mgr.x (mgr.24542) 4879 2021-07-31T03:41:05.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:04 smithi045 conmon[33655]: : cluster [DBG] pgmap v5423: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:05.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:04 smithi045 conmon[39718]: cluster 2021-07-31T03:41:03. 2021-07-31T03:41:05.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:04 smithi045 conmon[39718]: 420424+0000 mgr.x (mgr.24542) 4879 : cluster [DBG] pgmap v5423: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:07.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:06 smithi154 conmon[34553]: cluster 2021-07-31T03:41:05.420990+0000 mgr.x (mgr.24542) 4880 : cluster [DBG] pgmap v5424: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:07.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:06 smithi045 conmon[39718]: cluster 2021-07-31T03:41: 2021-07-31T03:41:07.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:06 smithi045 conmon[39718]: 05.420990+0000 mgr.x (mgr.24542) 4880 : cluster [DBG] pgmap v5424: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:07.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:06 smithi045 conmon[33655]: cluster 2021-07-31T03:41:05.420990+0000 mgr.x (mgr.24542) 2021-07-31T03:41:07.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:06 smithi045 conmon[33655]: 4880 : cluster [DBG] pgmap v5424: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:09.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:08 smithi154 conmon[34553]: cluster 2021-07-31T03:41:07.421345+0000 mgr.x (mgr.24542) 4881 : cluster [DBG] pgmap v5425: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:09.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:08 smithi045 conmon[39718]: cluster 2021-07-31T03:41:07.421345+0000 mgr.x (mgr.24542) 4881 : cluster [DBG] pgmap v5425: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:09.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:08 smithi045 conmon[33655]: cluster 2021-07-31T03:41:07.421345+0000 mgr.x (mgr.24542) 4881 : cluster 2021-07-31T03:41:09.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:08 smithi045 conmon[33655]: [DBG] pgmap v5425: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:11.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:10 smithi154 conmon[34553]: cluster 2021-07-31T03:41:09.421743+0000 mgr.x (mgr.24542) 4882 : cluster [DBG] pgmap v5426: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:11.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:10 smithi045 conmon[33655]: cluster 2021-07-31T03:41:09.421743+0000 2021-07-31T03:41:11.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:10 smithi045 conmon[33655]: mgr.x (mgr.24542) 4882 : cluster [DBG] pgmap v5426: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:11.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:10 smithi045 conmon[39718]: cluster 2021-07-31T03:41:09.421743+0000 mgr.x (mgr 2021-07-31T03:41:11.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:10 smithi045 conmon[39718]: .24542) 4882 : cluster [DBG] pgmap v5426: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:13.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:12 smithi154 conmon[34553]: cluster 2021-07-31T03:41:11.422164+0000 mgr.x (mgr.24542) 4883 : cluster [DBG] pgmap v5427: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:13.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:12 smithi045 conmon[33655]: cluster 2021-07-31T03:41:11.422164+0000 mgr.x (mgr.24542) 4883 : cluster [DBG] pgmap v5427: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:13.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:12 smithi045 conmon[39718]: cluster 2021-07-31T03:41:11.422164+0000 mgr.x ( 2021-07-31T03:41:13.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:12 smithi045 conmon[39718]: mgr.24542) 4883 : cluster [DBG] pgmap v5427: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:15.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:14 smithi154 conmon[34553]: cluster 2021-07-31T03:41:13.422560+0000 mgr.x (mgr.24542) 4884 : cluster [DBG] pgmap v5428: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:15.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:14 smithi045 conmon[39718]: cluster 2021-07-31T03:41:13.422560+0000 mgr.x 2021-07-31T03:41:15.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:14 smithi045 conmon[39718]: (mgr.24542) 4884 : cluster [DBG] pgmap v5428: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:15.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:14 smithi045 conmon[33655]: cluster 2021-07-31T03:41:13.422560+0000 mgr.x (mgr.24542) 4884 : cluster [DBG] 2021-07-31T03:41:15.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:14 smithi045 conmon[33655]: pgmap v5428: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:17.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:16 smithi154 conmon[34553]: cluster 2021-07-31T03:41:15.423026+0000 mgr.x (mgr.24542) 4885 : cluster [DBG] pgmap v5429: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:17.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:16 smithi045 conmon[39718]: cluster 2021-07-31T03:41:15.423026+0000 mgr.x (mgr.24542) 4885 : cluster [DBG] 2021-07-31T03:41:17.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:16 smithi045 conmon[39718]: pgmap v5429: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:17.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:16 smithi045 conmon[33655]: cluster 2021-07-31T03:41:15.423026+0000 2021-07-31T03:41:17.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:16 smithi045 conmon[33655]: mgr.x (mgr.24542) 4885 : cluster [DBG] pgmap v5429: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:19.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:18 smithi154 conmon[34553]: cluster 2021-07-31T03:41:17.423380+0000 mgr.x (mgr.24542) 4886 : cluster [DBG] pgmap v5430: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:19.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:18 smithi045 conmon[39718]: cluster 2021-07-31T03:41:17.423380+0000 mgr.x (mgr.24542) 4886 : cluster [DBG] 2021-07-31T03:41:19.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:18 smithi045 conmon[39718]: pgmap v5430: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:19.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:18 smithi045 conmon[33655]: cluster 2021-07-31T03:41:17. 2021-07-31T03:41:19.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:18 smithi045 conmon[33655]: 423380+0000 mgr.x (mgr.24542) 4886 : cluster [DBG] pgmap v5430: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:21.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:20 smithi154 conmon[34553]: cluster 2021-07-31T03:41:19.424011+0000 mgr.x (mgr.24542) 4887 : cluster [DBG] pgmap v5431: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:21.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:20 smithi045 conmon[33655]: cluster 2021-07-31T03:41:19 2021-07-31T03:41:21.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:20 smithi045 conmon[33655]: .424011+0000 mgr.x (mgr.24542) 4887 : cluster [DBG] pgmap v5431: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:21.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:20 smithi045 conmon[39718]: cluster 2021-07-31T03:41:19.424011+0000 mgr.x ( 2021-07-31T03:41:21.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:20 smithi045 conmon[39718]: mgr.24542) 4887 : cluster [DBG] pgmap v5431: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:22.682 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:41:22.961 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:41:22.962 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 116m ago 117m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:41:22.962 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 113m ago 116m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:41:22.962 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 113m ago 2h 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:41:22.962 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 116m ago 2h - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:41:22.962 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 116m ago 2h - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:41:22.963 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 113m ago 2h 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:41:22.963 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 116m ago 2h - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:41:22.963 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 116m ago 117m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:41:22.963 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 113m ago 117m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:41:22.964 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 116m ago 2h - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:41:22.964 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 116m ago 119m - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:41:22.964 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 116m ago 119m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:41:22.964 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 116m ago 119m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:41:22.964 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 113m ago 118m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:41:22.965 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 113m ago 118m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:41:22.965 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 113m ago 118m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:41:22.965 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 113m ago 117m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:41:22.965 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 113m ago 117m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:41:23.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:22 smithi154 conmon[34553]: cluster 2021-07-31T03:41:21.424542+0000 mgr.x (mgr.24542) 4888 : cluster [DBG] pgmap v5432: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:23.274 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:22 smithi045 conmon[33655]: cluster 2021-07-31T03:41:21.424542+0000 mgr.x (mgr 2021-07-31T03:41:23.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:22 smithi045 conmon[33655]: .24542) 4888 : cluster [DBG] pgmap v5432: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:23.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:22 smithi045 conmon[39718]: cluster 2021-07-31T03:41:21.424542+0000 2021-07-31T03:41:23.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:22 smithi045 conmon[39718]: mgr.x (mgr.24542) 4888 : cluster [DBG] pgmap v5432: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:23.298 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:41:23.299 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:41:23.299 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:41:23.299 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:41:23.299 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:41:23.299 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:41:23.300 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:41:23.300 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:41:23.300 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:41:23.300 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:41:23.300 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:41:23.301 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:41:23.301 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:41:23.301 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:41:23.301 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:41:23.301 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:41:23.302 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:41:24.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:23 smithi154 conmon[34553]: audit 2021-07-31T03:41:22.671873+0000 mgr.x (mgr.24542) 4889 : audit [DBG] from='client.29968 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:41:24.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:23 smithi154 conmon[34553]: audit 2021-07-31T03:41:24.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:23 smithi154 conmon[34553]: 2021-07-31T03:41:22.954606+0000 mgr.x (mgr.24542) 4890 : audit [DBG] from='client.29974 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:41:24.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:23 smithi154 conmon[34553]: audit 2021-07-31T03:41:23.299601+0000 mon.c (mon.1) 223 : audit [DBG] from='client.? 172.21.15.45:0/595646186' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:41:24.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:23 smithi045 conmon[33655]: audit 2021-07-31T03:41:22.671873+0000 mgr.x (mgr 2021-07-31T03:41:24.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:23 smithi045 conmon[33655]: .24542) 4889 : audit [DBG] from='client.29968 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:41:24.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:23 smithi045 conmon[33655]: audit 2021-07-31T03:41:22.954606+0000 mgr.x (mgr.24542) 4890 : audit [DBG] from='client.29974 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:41:24.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:23 smithi045 conmon[33655]: audit 2021-07-31T03:41:24.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:23 smithi045 conmon[33655]: 2021-07-31T03:41:23.299601+0000 mon.c (mon.1) 223 : audit [DBG] from='client.? 172.21.15.45:0/595646186' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:41:24.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:23 smithi045 conmon[39718]: audit 2021-07-31T03:41:22.671873+0000 mgr.x 2021-07-31T03:41:24.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:23 smithi045 conmon[39718]: (mgr.24542) 4889 : audit [DBG] from='client.29968 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:41:24.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:23 smithi045 conmon[39718]: audit 2021-07-31T03:41:22.954606+0000 mgr.x 2021-07-31T03:41:24.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:23 smithi045 conmon[39718]: (mgr.24542) 4890 : audit [DBG] from='client.29974 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:41:24.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:23 smithi045 conmon[39718]: audit 2021- 2021-07-31T03:41:24.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:23 smithi045 conmon[39718]: 07-31T03:41:23.299601 2021-07-31T03:41:24.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:23 smithi045 conmon[39718]: +0000 mon.c (mon.1) 223 2021-07-31T03:41:24.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:23 smithi045 conmon[39718]: : audit [DBG] from='client.? 172.21.15.45:0/595646186' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:41:25.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:24 smithi154 conmon[34553]: cluster 2021-07-31T03:41:23.425102+0000 mgr.x (mgr.24542) 4891 : cluster [DBG] pgmap v5433: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:25.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:24 smithi045 conmon[39718]: cluster 2021-07-31T03:41:23.425102+0000 mgr.x (mgr.24542) 2021-07-31T03:41:25.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:24 smithi045 conmon[39718]: 4891 : cluster [DBG] pgmap v5433: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:25.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:24 smithi045 conmon[33655]: cluster 2021-07-31T03:41:23.425102+0000 mgr.x (mgr.24542) 2021-07-31T03:41:25.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:24 smithi045 conmon[33655]: 4891 : cluster [DBG] pgmap v5433: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:27.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:26 smithi154 conmon[34553]: cluster 2021-07-31T03:41:25.425566+0000 mgr.x (mgr.24542) 4892 : cluster [DBG] pgmap v5434: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:27.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:26 smithi045 conmon[39718]: cluster 2021-07-31T03:41:25.425566+0000 mgr.x (mgr.24542) 4892 : cluster [DBG] pgmap v5434: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:27.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:26 smithi045 conmon[33655]: cluster 2021-07-31T03:41:25.425566+0000 mgr.x (mgr.24542) 2021-07-31T03:41:27.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:26 smithi045 conmon[33655]: 4892 : cluster [DBG] pgmap v5434: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:29.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:28 smithi154 conmon[34553]: cluster 2021-07-31T03:41:27.425975+0000 mgr.x (mgr.24542) 4893 : cluster [DBG] 2021-07-31T03:41:29.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:28 smithi154 conmon[34553]: pgmap v5435: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:29.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:28 smithi154 conmon[34553]: audit 2021-07-31T03:41:28.389692+0000 mon.b (mon.2) 645 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:41:29.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:28 smithi154 conmon[34553]: audit 2021-07-31T03:41:28.389956+0000 2021-07-31T03:41:29.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:28 smithi154 conmon[34553]: mon.b (mon.2) 646 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:41:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:28 smithi154 conmon[34553]: audit 2021 2021-07-31T03:41:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:28 smithi154 conmon[34553]: -07-31T03:41:28.390011+0000 mon.a (mon.0) 3347 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:41:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:28 smithi154 conmon[34553]: audit 2021-07 2021-07-31T03:41:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:28 smithi154 conmon[34553]: -31T03:41:28.390413+0000 mon.a 2021-07-31T03:41:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:28 smithi154 conmon[34553]: (mon.0) 3348 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:41:29.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:28 smithi045 conmon[33655]: cluster 2021-07-31T03:41:27.425975+0000 2021-07-31T03:41:29.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:28 smithi045 conmon[33655]: mgr.x (mgr.24542) 4893 : cluster [DBG] pgmap v5435: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:29.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:28 smithi045 conmon[33655]: audit 2021-07- 2021-07-31T03:41:29.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:28 smithi045 conmon[33655]: 31T03:41:28.389692+0000 mon.b (mon.2) 645 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:41:29.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:28 smithi045 conmon[33655]: audit 2021-07- 2021-07-31T03:41:29.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:28 smithi045 conmon[33655]: 31T03:41:28.389956+0000 mon.b (mon.2) 646 : audit [INF] 2021-07-31T03:41:29.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:28 smithi045 conmon[33655]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:41:29.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:28 smithi045 conmon[33655]: audit 2021-07-31T03:41: 2021-07-31T03:41:29.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:28 smithi045 conmon[33655]: 28.390011+0000 mon.a (mon.0) 3347 : 2021-07-31T03:41:29.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:28 smithi045 conmon[33655]: audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:41:29.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:28 smithi045 conmon[33655]: audit 2021-07-31 2021-07-31T03:41:29.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:28 smithi045 conmon[33655]: T03:41:28.390413+0000 mon.a ( 2021-07-31T03:41:29.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:28 smithi045 conmon[33655]: mon.0) 3348 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:41:29.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:28 smithi045 conmon[33655]: 2021-07-31T03:41:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:28 smithi045 conmon[39718]: cluster 2021-07-31T03:41:27.425975+0000 2021-07-31T03:41:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:28 smithi045 conmon[39718]: mgr.x (mgr.24542) 4893 : cluster [DBG] pgmap v5435: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:28 smithi045 conmon[39718]: audit 2021- 2021-07-31T03:41:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:28 smithi045 conmon[39718]: 07-31T03:41:28 2021-07-31T03:41:29.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:28 smithi045 conmon[39718]: .389692+0000 mon.b (mon.2) 645 2021-07-31T03:41:29.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:28 smithi045 conmon[39718]: : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:41:29.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:28 smithi045 conmon[39718]: audit 2021 2021-07-31T03:41:29.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:28 smithi045 conmon[39718]: -07-31T03:41:28.389956 2021-07-31T03:41:29.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:28 smithi045 conmon[39718]: +0000 mon.b (mon.2) 646 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:41:29.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:28 smithi045 conmon[39718]: audit 2021-07-31T03 2021-07-31T03:41:29.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:28 smithi045 conmon[39718]: :41:28.390011+0000 mon.a (mon 2021-07-31T03:41:29.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:28 smithi045 conmon[39718]: .0) 3347 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:41:29.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:28 smithi045 conmon[39718]: audit 2021-07-31T03:41:29.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:28 smithi045 conmon[39718]: 2021-07-31T03:41 2021-07-31T03:41:29.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:28 smithi045 conmon[39718]: :28.390413+0000 mon.a ( 2021-07-31T03:41:29.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:28 smithi045 conmon[39718]: mon.0) 3348 : audit [INF] 2021-07-31T03:41:29.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:28 smithi045 conmon[39718]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:41:31.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:30 smithi154 conmon[34553]: cluster 2021-07-31T03:41:29.426397+0000 mgr.x (mgr.24542) 4894 : cluster [DBG] pgmap v5436: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:31.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:30 smithi045 conmon[39718]: cluster 2021-07-31T03:41:29.426397+0000 mgr.x 2021-07-31T03:41:31.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:30 smithi045 conmon[39718]: (mgr.24542) 4894 : cluster [DBG] pgmap v5436: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:31.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:30 smithi045 conmon[33655]: cluster 2021-07-31T03:41:29.426397 2021-07-31T03:41:31.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:30 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 4894 : cluster [DBG] pgmap v5436: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:33.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:32 smithi154 conmon[34553]: cluster 2021-07-31T03:41:31.426995+0000 mgr.x (mgr.24542) 4895 : cluster [DBG] pgmap v5437: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:33.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:32 smithi045 conmon[39718]: cluster 2021-07-31T03:41:31.426995+0000 mgr.x ( 2021-07-31T03:41:33.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:32 smithi045 conmon[39718]: mgr.24542) 4895 : cluster [DBG] pgmap v5437: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:33.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:32 smithi045 conmon[33655]: cluster 2021-07-31T03:41:31.426995+0000 mgr.x (mgr.24542) 4895 : 2021-07-31T03:41:33.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:32 smithi045 conmon[33655]: cluster [DBG] pgmap v5437: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:34.867 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: debug 2021-07-31T03:41:34.604+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:41:34.868 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:41:34.868 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:41:34.868 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: yield 2021-07-31T03:41:34.869 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:41:34.869 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:41:34.869 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:41:34.869 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:41:34.869 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:41:34.870 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:41:34.870 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:41:34.870 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:41:34.870 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:41:34.870 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: task.result() 2021-07-31T03:41:34.871 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:41:34.871 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: debug 2021-07-31T03:41:34.615+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:41:34.871 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:41:34.871 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:41:34.872 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: yield 2021-07-31T03:41:34.872 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:41:34.872 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:41:34.872 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:41:34.872 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:41:34.873 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:41:34.873 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:41:34.873 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:41:34.873 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:41:34.873 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:41:34.874 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: task.result() 2021-07-31T03:41:34.874 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:41:34.874 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: debug 2021-07-31T03:41:34.641+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:41:34.874 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:41:34.874 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:41:34.875 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: yield 2021-07-31T03:41:34.875 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:41:34.875 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:41:34.875 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:41:34.875 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:41:34.876 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:41:34.877 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:41:34.877 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:41:34.877 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:41:34.877 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:41:34.878 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: task.result() 2021-07-31T03:41:34.878 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:41:34.878 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: debug 2021-07-31T03:41:34.641+0000 7fcb97969700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:41:34.878 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:41:34.878 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:41:34.879 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: yield 2021-07-31T03:41:34.879 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:41:34.879 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:41:34.879 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:41:34.880 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:41:34.880 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:41:34.880 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:41:34.880 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:41:34.880 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:41:34.881 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:41:34.881 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: task.result() 2021-07-31T03:41:34.881 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:41:34.881 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: 2021-07-31T03:41:34.881 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:41:34.882 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: 2021-07-31T03:41:34.882 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:41:34.882 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:41:34.882 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:41:34.882 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:41:34.883 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:41:34.883 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:41:34.883 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:41:34.883 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:41:34.883 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:41:34.884 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:41:34.884 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:41:34.884 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:41:34.884 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:41:34.884 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:41:34.885 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:41:34.885 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:41:34.885 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:41:34.885 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:41:34.885 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: raise 2021-07-31T03:41:34.886 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:41:34.886 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:41:34.886 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:41:34.886 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:41:34.886 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:41:34.887 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: debug 2021-07-31T03:41:34.650+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:41:34.887 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:41:34.887 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:41:34.887 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: yield 2021-07-31T03:41:34.887 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:41:34.888 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:41:34.888 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:41:34.888 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:41:34.888 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:41:34.888 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:41:34.889 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:41:34.889 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:41:34.889 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:41:34.889 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: task.result() 2021-07-31T03:41:34.889 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:41:34.890 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: debug 2021-07-31T03:41:34.650+0000 7fcb9816a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:41:34.890 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:41:34.890 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:41:34.890 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: yield 2021-07-31T03:41:34.891 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:41:34.891 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:41:34.891 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:41:34.891 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:41:34.891 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:41:34.892 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:41:34.892 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:41:34.892 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:41:34.892 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:41:34.892 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: task.result() 2021-07-31T03:41:34.893 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:41:34.893 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: 2021-07-31T03:41:34.893 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:41:34.893 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: 2021-07-31T03:41:34.893 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:41:34.894 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:41:34.894 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:41:34.894 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:41:34.894 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:41:34.894 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:41:34.895 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:41:34.895 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:41:34.895 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:41:34.895 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:41:34.896 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:41:34.896 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:41:34.896 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:41:34.896 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:41:34.896 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:41:34.897 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:41:34.897 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:41:34.897 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:41:34.897 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: raise 2021-07-31T03:41:34.897 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:41:34.898 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:41:34.898 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:41:34.898 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:41:34.898 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:41:35.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[34553]: cluster 2021-07-31T03:41:33.427562+0000 mgr.x (mgr.24542) 4896 : cluster [DBG] pgmap v5438: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:35.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[34553]: audit 2021-07-31T03:41:34.559090+0000 mon.b (mon.2) 647 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:41:35.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[34553]: audit 2021-07-31T03:41:34.560524+0000 mon.b 2021-07-31T03:41:35.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[34553]: (mon.2) 648 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:41:35.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[34553]: audit 2021-07-31T03:41:34. 2021-07-31T03:41:35.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[34553]: 561407+0000 mon.b (mon.2) 649 : audit [INF] 2021-07-31T03:41:35.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:34 smithi154 conmon[34553]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:41:35.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:34 smithi045 conmon[33655]: cluster 2021-07-31T03:41:33.427562+0000 mgr.x (mgr.24542) 4896 2021-07-31T03:41:35.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:34 smithi045 conmon[33655]: : cluster [DBG] pgmap v5438: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:35.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:34 smithi045 conmon[33655]: audit 2021-07-31T03:41:34.559090+0000 mon.b (mon.2) 647 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:41:35.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:34 smithi045 conmon[33655]: audit 2021-07-31T03:41:34.560524+0000 mon.b (mon.2) 648 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:41:35.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:34 smithi045 conmon[33655]: audit 2021-07-31T03:41:34.561407+0000 mon.b (mon. 2021-07-31T03:41:35.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:34 smithi045 conmon[33655]: 2) 649 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:41:35.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:34 smithi045 conmon[39718]: cluster 2021-07-31T03:41:33.427562+0000 2021-07-31T03:41:35.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:34 smithi045 conmon[39718]: mgr.x (mgr.24542) 4896 : cluster [DBG] pgmap v5438: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:35.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:34 smithi045 conmon[39718]: audit 2021-07-31T03:41: 2021-07-31T03:41:35.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:34 smithi045 conmon[39718]: 34.559090+0000 mon.b (mon.2) 647 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:41:35.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:34 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T03:41:35.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:34 smithi045 conmon[39718]: T03:41:34.560524+0000 mon.b ( 2021-07-31T03:41:35.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:34 smithi045 conmon[39718]: mon.2) 648 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:41:35.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:34 smithi045 conmon[39718]: 2021-07-31T03:41:35.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:34 smithi045 conmon[39718]: audit 2021 2021-07-31T03:41:35.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:34 smithi045 conmon[39718]: -07-31T03:41:34. 2021-07-31T03:41:35.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:34 smithi045 conmon[39718]: 561407+0000 mon.b (mon.2) 649 2021-07-31T03:41:35.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:34 smithi045 conmon[39718]: : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:41:35.887 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:41:34.562183+0000 mgr.x (mgr.24542) 4897 : cephadm [WRN] 2021-07-31T03:41:35.888 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:41:35.888 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:41:34.604991+0000 mgr.x (mgr.24542) 4898 : cephadm [ERR] 2021-07-31T03:41:35.888 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:41:35.888 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:41:35.889 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: yield 2021-07-31T03:41:35.889 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:41:35.889 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:41:35.889 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:41:35.889 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:41:35.890 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:41:35.890 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:41:35.890 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:41:35.890 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:41:35.891 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:41:35.891 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: task.result() 2021-07-31T03:41:35.891 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:41:35.891 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:41:34.605466+0000 mgr.x (mgr.24542) 4899 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:41:35.891 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:41:34.616642+0000 mgr.x (mgr.24542) 4900 : cephadm [ERR] 2021-07-31T03:41:35.892 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: 2021-07-31T03:41:35.892 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:41:35.892 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:41:35.892 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: yield 2021-07-31T03:41:35.892 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:41:35.893 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:41:35.893 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:41:35.893 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:41:35.893 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:41:35.894 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:41:35.894 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:41:35.894 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:41:35.894 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:41:35.894 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: task.result() 2021-07-31T03:41:35.895 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:41:35.895 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:41 2021-07-31T03:41:35.895 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: :34.617497+0000 mgr.x (mgr.24542) 4901 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:41:35.895 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:41:34.642207+0000 mgr.x (mgr.24542) 4902 : cephadm [ERR] 2021-07-31T03:41:35.895 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:41:35.896 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:41:34.562183+0000 mgr.x (mgr.24542 2021-07-31T03:41:35.896 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: ) 4897 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:41:35.896 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:41:34.604991+0000 2021-07-31T03:41:35.897 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: mgr.x (mgr.24542) 4898 : cephadm [ERR] 2021-07-31T03:41:35.897 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:41:35.897 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:41:35.897 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: yield 2021-07-31T03:41:35.897 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:41:35.898 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:41:35.898 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:41:35.898 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:41:35.898 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:41:35.898 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:41:35.899 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:41:35.899 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:41:35.899 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:41:35.899 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: task.result() 2021-07-31T03:41:35.900 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:41:35.900 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:41:34.605466+0000 mgr.x (mgr.24542) 4899 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:41:35.900 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: cephadm 2021-07- 2021-07-31T03:41:35.900 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: 31T03:41:34.616642+0000 mgr.x (mgr.24542) 4900 : cephadm [ERR] 2021-07-31T03:41:35.900 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: 2021-07-31T03:41:35.901 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:41:35.901 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:41:35.901 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: yield 2021-07-31T03:41:35.901 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:41:35.901 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:41:35.902 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:41:35.902 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:41:35.902 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:41:35.902 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:41:35.902 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:41:35.903 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:41:35.903 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:41:35.903 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: task.result() 2021-07-31T03:41:35.903 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:41:35.903 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:41:34.617497+0000 mgr.x (mgr.24542) 4901 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:41:35.904 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:41:35.904 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: 2021-07-31T03:41:34.642207+0000 mgr.x (mgr. 2021-07-31T03:41:35.904 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: 24542) 4902 : cephadm [ERR] 2021-07-31T03:41:35.905 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:41:35.905 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: yield 2021-07-31T03:41:35.905 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:41:35.905 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:41:35.906 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:41:35.906 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:41:35.906 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:41:35.906 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:41:35.906 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:41:35.907 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:41:35.907 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:41:35.907 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: task.result() 2021-07-31T03:41:35.907 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:41:35.908 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:41:34.642636+0000 mgr.x (mgr.24542) 2021-07-31T03:41:35.908 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: 4903 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:41:35.908 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:41:35.908 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:41:35.908 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: yield 2021-07-31T03:41:35.909 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:41:35.909 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:41:35.909 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:41:35.909 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:41:35.909 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:41:35.910 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:41:35.910 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:41:35.910 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:41:35.910 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:41:35.910 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: task.result() 2021-07-31T03:41:35.911 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:41:35.911 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: 2021-07-31T03:41:35.911 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:41:35.911 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: 2021-07-31T03:41:35.911 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:41:35.912 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:41:35.912 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:41:35.912 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:41:35.912 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:41:35.912 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:41:35.913 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:41:35.913 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:41:35.913 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:41:35.913 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:41:35.913 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:41:35.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:41:35.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:41:35.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:41:35.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:41:35.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:41:35.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:41:35.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:41:35.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: raise 2021-07-31T03:41:35.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:41:35.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:41:35.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:41:35.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:41:35.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:41:35.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:41:34.651197+0000 mgr.x (mgr.24542) 4904 : cephadm [ERR] 2021-07-31T03:41:35.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:41:35.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:41:35.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: yield 2021-07-31T03:41:35.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:41:35.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:41:35.918 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:41:35.918 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:41:35.918 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:41:35.918 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:41:35.919 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:41:35.919 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:41:35.919 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:41:35.919 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: task.result() 2021-07-31T03:41:35.919 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:41:35.920 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:41:34.651605+0000 mgr.x (mgr.24542) 4905 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:41:35.920 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:41:35.920 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:41:35.920 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: yield 2021-07-31T03:41:35.920 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:41:35.921 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:41:35.921 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:41:35.921 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:41:35.921 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:41:35.921 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:41:35.922 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:41:35.922 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:41:35.922 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: yield 2021-07-31T03:41:35.923 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:41:35.923 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:41:35.923 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:41:35.923 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:41:35.923 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:41:35.924 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:41:35.924 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:41:35.924 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:41:35.924 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:41:35.924 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: task.result() 2021-07-31T03:41:35.925 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:41:35.925 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:41:34.642636+0000 mgr.x (mgr.24542) 4903 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:41:35.925 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:41:35.925 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:41:35.925 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: yield 2021-07-31T03:41:35.926 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:41:35.926 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:41:35.926 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:41:35.926 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:41:35.926 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:41:35.927 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:41:35.927 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:41:35.927 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:41:35.927 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:41:35.928 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: task.result() 2021-07-31T03:41:35.928 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:41:35.928 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: 2021-07-31T03:41:35.928 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:41:35.928 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: 2021-07-31T03:41:35.929 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:41:35.929 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:41:35.929 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:41:35.929 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:41:35.929 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:41:35.929 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:41:35.930 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:41:35.930 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:41:35.930 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:41:35.931 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:41:35.931 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:41:35.931 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:41:35.931 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: task.result() 2021-07-31T03:41:35.931 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:41:35.932 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: 2021-07-31T03:41:35.932 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:41:35.932 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: 2021-07-31T03:41:35.932 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:41:35.932 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:41:35.933 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:41:35.933 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:41:35.933 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:41:35.933 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:41:35.933 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:41:35.934 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:41:35.934 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:41:35.934 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:41:35.934 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:41:35.934 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:41:35.935 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:41:35.935 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:41:35.935 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:41:35.935 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:41:35.936 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:41:35.936 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:41:35.936 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: raise 2021-07-31T03:41:35.936 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:41:35.936 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:41:35.937 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:41:35.937 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:41:35.937 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:41:35.937 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:41:35.938 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:41:35.938 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:41:35.938 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:41:35.938 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:41:35.939 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:41:35.939 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:41:35.939 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:41:35.939 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:41:35.939 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: raise 2021-07-31T03:41:35.940 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:41:35.940 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:41:35.940 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:41:35.940 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:41:35.940 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:41:35.941 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:41:34.651197+0000 mgr.x (mgr.24542) 4904 : cephadm [ERR] 2021-07-31T03:41:35.941 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:41:35.941 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:41:35.941 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: yield 2021-07-31T03:41:35.941 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:41:35.942 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:41:35.942 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:41:35.942 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:41:35.942 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:41:35.942 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:41:35.943 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:41:35.943 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:41:35.943 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:41:35.943 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: task.result() 2021-07-31T03:41:35.943 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:41:35.944 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:41:34.651605+0000 mgr.x (mgr.24542) 4905 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:41:35.944 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:41:35.944 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:41:35.944 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: yield 2021-07-31T03:41:35.944 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:41:35.945 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:41:35.945 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:41:35.945 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:41:35.945 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:41:35.945 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:41:35.946 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:41:35.946 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:41:35.946 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:41:35.946 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: task.result() 2021-07-31T03:41:35.946 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:41:35.947 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: 2021-07-31T03:41:35.947 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:41:35.947 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: 2021-07-31T03:41:35.947 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:41:35.948 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:41:35.948 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:41:35.948 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:41:35.948 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:41:35.948 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:41:35.949 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:41:35.949 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:41:35.949 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:41:35.949 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:41:35.949 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:41:35.950 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:41:35.950 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:41:35.950 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:41:35.950 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:41:35.950 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:41:35.951 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:41:35.951 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:41:35.951 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: raise 2021-07-31T03:41:35.951 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:41:35.951 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:41:35.952 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:41:35.952 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:41:35.952 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:35 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:41:36.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:41:34.562183+0000 mgr.x (mgr.24542) 4897 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:41:36.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: 2021-07-31T03:41:36.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:41:34.604991+0000 mgr.x (mgr.24542) 4898 : cephadm [ERR] 2021-07-31T03:41:36.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:41:36.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:41:36.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: yield 2021-07-31T03:41:36.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:41:36.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:41:36.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:41:36.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:41:36.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:41:36.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:41:36.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:41:36.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:41:36.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:41:36.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: task.result() 2021-07-31T03:41:36.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:41:36.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:41:34.605466+0000 mgr.x (mgr.24542) 4899 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:41:36.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:41:34.616642+0000 mgr.x (mgr.24542) 4900 : cephadm [ERR] 2021-07-31T03:41:36.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:41:36.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:41:36.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: yield 2021-07-31T03:41:36.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:41:36.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:41:36.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:41:36.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:41:36.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:41:36.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:41:36.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:41:36.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:41:36.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:41:36.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: task.result() 2021-07-31T03:41:36.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:41:36.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:41:34.617497+0000 mgr.x (mgr.24542) 4901 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:41:36.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:41:34.642207+0000 mgr.x (mgr.24542) 4902 : cephadm [ERR] 2021-07-31T03:41:36.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:41:36.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:41:36.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: yield 2021-07-31T03:41:36.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:41:36.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:41:36.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:41:36.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:41:36.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:41:36.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:41:36.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:41:36.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:41:36.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:41:36.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: task.result() 2021-07-31T03:41:36.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:41:36.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:41:34.642636+0000 mgr.x (mgr.24542) 4903 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:41:36.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:41:36.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:41:36.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: yield 2021-07-31T03:41:36.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:41:36.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:41:36.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:41:36.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:41:36.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:41:36.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:41:36.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:41:36.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:41:36.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:41:36.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: task.result() 2021-07-31T03:41:36.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:41:36.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: 2021-07-31T03:41:36.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:41:36.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: 2021-07-31T03:41:36.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:41:36.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:41:36.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:41:36.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:41:36.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:41:36.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:41:36.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:41:36.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:41:36.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:41:36.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:41:36.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:41:36.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:41:36.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:41:36.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:41:36.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:41:36.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:41:36.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:41:36.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:41:36.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: raise 2021-07-31T03:41:36.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:41:36.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:41:36.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:41:36.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:41:36.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:41:36.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:41:34.651197+0000 mgr.x (mgr.24542) 4904 : cephadm [ERR] 2021-07-31T03:41:36.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:41:36.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:41:36.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: yield 2021-07-31T03:41:36.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:41:36.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:41:36.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:41:36.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:41:36.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:41:36.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:41:36.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:41:36.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:41:36.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:41:36.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: task.result() 2021-07-31T03:41:36.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:41:36.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:41:34.651605+0000 mgr.x (mgr.24542) 4905 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:41:36.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:41:36.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:41:36.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: yield 2021-07-31T03:41:36.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:41:36.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:41:36.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:41:36.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:41:36.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:41:36.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:41:36.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:41:36.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:41:36.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:41:36.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: task.result() 2021-07-31T03:41:36.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:41:36.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: 2021-07-31T03:41:36.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:41:36.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: 2021-07-31T03:41:36.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:41:36.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:41:36.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:41:36.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:41:36.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:41:36.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:41:36.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:41:36.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:41:36.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:41:36.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:41:36.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:41:36.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:41:36.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:41:36.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:41:36.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:41:36.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:41:36.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:41:36.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:41:36.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: raise 2021-07-31T03:41:36.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:41:36.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:41:36.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:41:36.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:41:36.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:35 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:41:37.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:36 smithi154 conmon[34553]: cluster 2021-07-31T03:41:35.428024+0000 mgr.x (mgr.24542) 4906 : cluster [DBG] pgmap v5439: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:37.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:36 smithi045 conmon[39718]: cluster 2021-07-31T03:41:35. 2021-07-31T03:41:37.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:36 smithi045 conmon[39718]: 428024+0000 mgr.x (mgr.24542) 4906 : cluster [DBG] pgmap v5439: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:37.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:36 smithi045 conmon[33655]: cluster 2021-07-31T03:41:35.428024 2021-07-31T03:41:37.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:36 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 4906 : cluster [DBG] pgmap v5439: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:39.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:38 smithi154 conmon[34553]: cluster 2021-07-31T03:41:37.428401+0000 mgr.x (mgr.24542) 4907 : cluster [DBG] pgmap v5440: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:39.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:38 smithi045 conmon[39718]: cluster 2021-07-31T03:41:37.428401+0000 mgr.x (mgr 2021-07-31T03:41:39.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:38 smithi045 conmon[39718]: .24542) 4907 : cluster [DBG] pgmap v5440: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:39.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:38 smithi045 conmon[33655]: cluster 2021-07-31T03:41:37.428401+0000 mgr.x (mgr.24542) 4907 2021-07-31T03:41:39.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:38 smithi045 conmon[33655]: : cluster [DBG] pgmap v5440: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:41.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:40 smithi154 conmon[34553]: cluster 2021-07-31T03:41:39.428804+0000 mgr.x (mgr.24542) 4908 : cluster [DBG] pgmap v5441: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:41.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:40 smithi045 conmon[39718]: cluster 2021-07-31T03:41:39.428804+0000 mgr.x ( 2021-07-31T03:41:41.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:40 smithi045 conmon[39718]: mgr.24542) 4908 : cluster [DBG] pgmap v5441: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:41.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:40 smithi045 conmon[33655]: cluster 2021-07-31T03:41:39.428804+0000 mgr.x (mgr.24542) 4908 : 2021-07-31T03:41:41.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:40 smithi045 conmon[33655]: cluster [DBG] pgmap v5441: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:43.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:42 smithi154 conmon[34553]: cluster 2021-07-31T03:41:41.429167+0000 mgr.x (mgr.24542) 4909 : cluster [DBG] 2021-07-31T03:41:43.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:42 smithi154 conmon[34553]: pgmap v5442: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:43.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:42 smithi045 conmon[39718]: cluster 2021-07-31T03:41:41.429167+0000 mgr.x ( 2021-07-31T03:41:43.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:42 smithi045 conmon[39718]: mgr.24542) 4909 : cluster [DBG] pgmap v5442: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:43.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:42 smithi045 conmon[33655]: cluster 2021-07-31T03:41:41.429167+0000 mgr.x (mgr.24542) 4909 : cluster [DBG] pgmap v5442: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:43.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:42 smithi045 conmon[33655]: 2021-07-31T03:41:45.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:44 smithi154 conmon[34553]: cluster 2021-07-31T03:41:43.429586+0000 mgr.x (mgr.24542) 4910 : cluster [DBG] pgmap v5443: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:45.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:44 smithi045 conmon[33655]: cluster 2021-07-31T03:41:43.429586+0000 mgr.x (mgr.24542) 4910 : 2021-07-31T03:41:45.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:44 smithi045 conmon[33655]: cluster [DBG] pgmap v5443: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:45.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:44 smithi045 conmon[39718]: cluster 2021-07-31T03:41:43 2021-07-31T03:41:45.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:44 smithi045 conmon[39718]: .429586+0000 mgr.x (mgr.24542) 4910 : cluster [DBG] pgmap v5443: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:47.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:46 smithi154 conmon[34553]: cluster 2021-07-31T03:41:45.430074+0000 mgr.x (mgr.24542) 4911 : cluster [DBG] pgmap v5444: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:47.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:46 smithi045 conmon[33655]: cluster 2021-07-31T03:41:45.430074+0000 mgr.x (mgr.24542) 4911 : cluster [DBG] pgmap v5444: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:47.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:46 smithi045 conmon[39718]: cluster 2021-07-31T03:41:45.430074+0000 mgr.x 2021-07-31T03:41:47.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:46 smithi045 conmon[39718]: (mgr.24542) 4911 : cluster [DBG] pgmap v5444: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:49.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:48 smithi154 conmon[34553]: cluster 2021-07-31T03:41:47.430575+0000 mgr.x (mgr.24542) 4912 : cluster [DBG] pgmap v5445: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:49.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:48 smithi045 conmon[39718]: cluster 2021-07-31T03:41:47. 2021-07-31T03:41:49.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:48 smithi045 conmon[39718]: 430575+0000 mgr.x (mgr.24542) 4912 : cluster [DBG] pgmap v5445: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:49.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:48 smithi045 conmon[33655]: cluster 2021-07-31T03:41:47. 2021-07-31T03:41:49.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:48 smithi045 conmon[33655]: 430575+0000 mgr.x (mgr.24542) 4912 : cluster [DBG] pgmap v5445: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:51.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:50 smithi154 conmon[34553]: cluster 2021-07-31T03:41:49.431116+0000 mgr.x (mgr.24542) 4913 : cluster [DBG] pgmap v5446: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:51.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:50 smithi045 conmon[39718]: cluster 2021-07-31T03:41:49.431116+0000 mgr.x (mgr.24542) 4913 : cluster [DBG] pgmap v5446: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:51.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:50 smithi045 conmon[33655]: cluster 2021-07-31T03:41:49. 2021-07-31T03:41:51.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:50 smithi045 conmon[33655]: 431116+0000 mgr.x (mgr.24542) 4913 : cluster [DBG] pgmap v5446: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:53.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:52 smithi154 conmon[34553]: cluster 2021-07-31T03:41:51.431525+0000 mgr.x (mgr.24542) 4914 : cluster [DBG] pgmap v5447: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:53.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:52 smithi045 conmon[33655]: cluster 2021-07-31T03:41:51.431525+0000 mgr.x (mgr.24542) 2021-07-31T03:41:53.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:52 smithi045 conmon[33655]: 4914 : cluster [DBG] pgmap v5447: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:53.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:52 smithi045 conmon[39718]: cluster 2021-07-31T03:41:51.431525+0000 mgr.x (mgr 2021-07-31T03:41:53.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:52 smithi045 conmon[39718]: .24542) 4914 : cluster [DBG] pgmap v5447: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:53.611 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:41:53.890 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:41:53.890 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 116m ago 117m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:41:53.890 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 114m ago 117m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:41:53.891 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 114m ago 2h 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:41:53.891 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 116m ago 2h - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:41:53.891 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 116m ago 2h - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:41:53.891 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 114m ago 2h 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:41:53.891 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 116m ago 2h - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:41:53.892 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 116m ago 117m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:41:53.892 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 114m ago 117m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:41:53.892 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 116m ago 2h - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:41:53.892 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 116m ago 2h - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:41:53.892 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 116m ago 119m - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:41:53.893 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 116m ago 119m - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:41:53.893 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 114m ago 119m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:41:53.893 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 114m ago 118m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:41:53.893 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 114m ago 118m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:41:53.893 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 114m ago 118m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:41:53.894 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 114m ago 117m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:41:54.225 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:41:54.226 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:41:54.226 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:41:54.226 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:41:54.226 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:41:54.227 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:41:54.227 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:41:54.227 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:41:54.227 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:41:54.227 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:41:54.228 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:41:54.228 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:41:54.228 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:41:54.228 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:41:54.228 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:41:54.229 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:41:54.229 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:41:55.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:54 smithi154 conmon[34553]: cluster 2021-07-31T03:41:53.431970+0000 mgr.x (mgr.24542) 4915 : cluster [DBG] pgmap v5448: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:55.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:54 smithi154 conmon[34553]: audit 2021-07-31T03:41:53.601195+0000 mgr.x (mgr.24542) 4916 : audit [DBG] from='client.29986 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:41:55.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:54 smithi154 conmon[34553]: audit 2021-07-31T03:41:53.883410+0000 mgr.x (mgr.24542) 4917 : audit [DBG] from='client.29992 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:41:55.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:54 smithi154 conmon[34553]: audit 2021-07-31T03:41:54.226731+0000 2021-07-31T03:41:55.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:54 smithi154 conmon[34553]: mon.a (mon.0) 3349 : audit [DBG] from='client.? 172.21.15.45:0/1239044541' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:41:55.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:54 smithi045 conmon[39718]: cluster 2021-07-31T03:41:53.431970+0000 mgr.x (mgr.24542) 4915 : 2021-07-31T03:41:55.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:54 smithi045 conmon[39718]: cluster [DBG] pgmap v5448: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:55.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:54 smithi045 conmon[39718]: audit 2021-07-31T03:41:53.601195+0000 mgr.x (mgr.24542) 4916 : audit [DBG] from='client.29986 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:41:55.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:54 smithi045 conmon[39718]: audit 2021-07-31T03:41:53.883410+0000 mgr.x (mgr.24542) 4917 : audit 2021-07-31T03:41:55.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:54 smithi045 conmon[39718]: [DBG] from='client.29992 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:41:55.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:54 smithi045 conmon[39718]: audit 2021-07-31T03:41:55.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:54 smithi045 conmon[39718]: 2021-07-31T03:41:54.226731+0000 mon.a (mon.0) 3349 : 2021-07-31T03:41:55.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:54 smithi045 conmon[39718]: audit [DBG] from='client.? 172.21.15.45:0/1239044541' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:41:55.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:54 smithi045 conmon[33655]: cluster 2021-07-31T03:41:53.431970+0000 mgr.x (mgr. 2021-07-31T03:41:55.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:54 smithi045 conmon[33655]: 24542) 4915 : cluster [DBG] pgmap v5448: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:55.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:54 smithi045 conmon[33655]: audit 2021-07-31T03:41:53.601195+0000 mgr.x (mgr.24542) 4916 : audit [DBG] from='client.29986 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:41:55.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:54 smithi045 conmon[33655]: audit 2021-07- 2021-07-31T03:41:55.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:54 smithi045 conmon[33655]: 31T03:41:53.883410+0000 mgr.x (mgr.24542) 4917 2021-07-31T03:41:55.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:54 smithi045 conmon[33655]: : audit [DBG] from='client.29992 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:41:55.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:54 smithi045 conmon[33655]: audit 2021-07 2021-07-31T03:41:55.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:54 smithi045 conmon[33655]: -31T03:41:54.226731+0000 mon.a (mon.0) 3349 : 2021-07-31T03:41:55.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:54 smithi045 conmon[33655]: audit [DBG] from='client.? 172.21.15.45:0/1239044541' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:41:57.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:56 smithi154 conmon[34553]: cluster 2021-07-31T03:41:55.432377+0000 mgr.x (mgr.24542) 4918 : cluster [DBG] pgmap v5449: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:57.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:56 smithi045 conmon[39718]: cluster 2021-07-31T03:41:55.432377+0000 mgr.x (mgr.24542) 4918 : cluster [DBG] 2021-07-31T03:41:57.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:56 smithi045 conmon[39718]: pgmap v5449: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:57.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:56 smithi045 conmon[33655]: cluster 2021-07-31T03:41:55.432377 2021-07-31T03:41:57.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:56 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 4918 : cluster [DBG] pgmap v5449: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:59.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:58 smithi154 conmon[34553]: cluster 2021-07-31T03:41:57.432850+0000 mgr.x ( 2021-07-31T03:41:59.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:41:58 smithi154 conmon[34553]: mgr.24542) 4919 : cluster [DBG] pgmap v5450: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:59.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:58 smithi045 conmon[39718]: cluster 2021-07-31T03:41:57.432850+0000 mgr.x (mgr.24542) 4919 : cluster [DBG] 2021-07-31T03:41:59.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:41:58 smithi045 conmon[39718]: pgmap v5450: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:41:59.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:58 smithi045 conmon[33655]: cluster 2021-07-31T03:41:57.432850 2021-07-31T03:41:59.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:41:58 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 4919 : cluster [DBG] pgmap v5450: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:01.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:00 smithi154 conmon[34553]: cluster 2021-07-31T03:41:59.433322+0000 mgr.x (mgr.24542) 4920 : cluster [DBG] pgmap v5451: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:01.274 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:00 smithi045 conmon[39718]: cluster 2021-07-31T03:41:59.433322+0000 mgr.x (mgr.24542) 4920 : cluster [DBG] 2021-07-31T03:42:01.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:00 smithi045 conmon[39718]: pgmap v5451: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:01.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:00 smithi045 conmon[33655]: cluster 2021-07-31T03:41:59.433322 2021-07-31T03:42:01.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:00 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 4920 : cluster [DBG] pgmap v5451: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:03.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:02 smithi154 conmon[34553]: cluster 2021-07-31T03:42:01.433680+0000 mgr.x (mgr.24542) 4921 : cluster [DBG] pgmap v5452: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:03.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:02 smithi045 conmon[39718]: cluster 2021-07-31T03:42:01.433680+0000 mgr.x (mgr. 2021-07-31T03:42:03.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:02 smithi045 conmon[39718]: 24542) 4921 : cluster [DBG] pgmap v5452: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:03.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:02 smithi045 conmon[33655]: cluster 2021-07-31T03:42:01.433680+0000 mgr.x (mgr.24542) 4921 : cluster [DBG] 2021-07-31T03:42:03.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:02 smithi045 conmon[33655]: pgmap v5452: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:05.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:04 smithi154 conmon[34553]: cluster 2021-07-31T03:42:03.434132+0000 mgr.x (mgr.24542) 4922 : cluster [DBG] pgmap v5453: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:05.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:04 smithi045 conmon[39718]: cluster 2021-07-31T03:42:03.434132+0000 mgr.x 2021-07-31T03:42:05.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:04 smithi045 conmon[39718]: (mgr.24542) 4922 : cluster [DBG] pgmap v5453: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:05.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:04 smithi045 conmon[33655]: cluster 2021-07-31T03:42:03.434132+0000 mgr.x (mgr. 2021-07-31T03:42:05.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:04 smithi045 conmon[33655]: 24542) 4922 : cluster [DBG] pgmap v5453: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:07.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:06 smithi154 conmon[34553]: cluster 2021-07-31T03:42:05.434579+0000 mgr.x (mgr.24542) 4923 : cluster [DBG] pgmap v5454: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:07.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:06 smithi045 conmon[33655]: cluster 2021-07-31T03:42:05.434579+0000 mgr.x (mgr.24542) 4923 : cluster [DBG] pgmap v5454: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:07.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:06 smithi045 conmon[33655]: 2021-07-31T03:42:07.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:06 smithi045 conmon[39718]: cluster 2021-07-31T03:42:05 2021-07-31T03:42:07.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:06 smithi045 conmon[39718]: .434579+0000 mgr.x (mgr.24542) 4923 : cluster [DBG] pgmap v5454: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:09.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:08 smithi154 conmon[34553]: cluster 2021-07-31T03:42:07.435038+0000 mgr.x (mgr.24542) 4924 : cluster [DBG] pgmap v5455: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:09.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:08 smithi045 conmon[39718]: cluster 2021-07-31T03:42:07. 2021-07-31T03:42:09.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:08 smithi045 conmon[39718]: 435038+0000 mgr.x (mgr.24542) 4924 : cluster [DBG] pgmap v5455: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:09.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:08 smithi045 conmon[33655]: cluster 2021-07-31T03:42:07.435038+0000 mgr.x (mgr.24542) 4924 : cluster 2021-07-31T03:42:09.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:08 smithi045 conmon[33655]: [DBG] pgmap v5455: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:11.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:10 smithi154 conmon[34553]: cluster 2021-07-31T03:42:09.435536+0000 mgr.x (mgr.24542) 4925 : cluster [DBG] pgmap v5456: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:11.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:10 smithi045 conmon[39718]: cluster 2021-07-31T03:42: 2021-07-31T03:42:11.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:10 smithi045 conmon[39718]: 09.435536+0000 mgr.x (mgr.24542) 4925 : cluster [DBG] pgmap v5456: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:11.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:10 smithi045 conmon[33655]: cluster 2021-07-31T03:42:09.435536+0000 mgr.x (mgr.24542) 2021-07-31T03:42:11.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:10 smithi045 conmon[33655]: 4925 : cluster [DBG] pgmap v5456: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:13.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:12 smithi154 conmon[34553]: cluster 2021-07-31T03:42:11.435902+0000 mgr.x (mgr.24542) 4926 : cluster [DBG] pgmap v5457: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:13.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:12 smithi045 conmon[33655]: cluster 2021-07-31T03:42:11.435902+0000 mgr.x (mgr.24542) 4926 : cluster [DBG] pgmap v5457: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:13.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:12 smithi045 conmon[39718]: cluster 2021-07-31T03:42:11.435902+0000 mgr.x (mgr. 2021-07-31T03:42:13.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:12 smithi045 conmon[39718]: 24542) 4926 : cluster [DBG] pgmap v5457: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:15.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:14 smithi154 conmon[34553]: cluster 2021-07-31T03:42:13.436434+0000 mgr.x (mgr.24542) 4927 : cluster [DBG] pgmap v5458: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:15.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:14 smithi045 conmon[39718]: cluster 2021-07-31T03:42:13.436434+0000 mgr.x (mgr.24542 2021-07-31T03:42:15.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:14 smithi045 conmon[39718]: ) 4927 : cluster [DBG] pgmap v5458: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:15.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:14 smithi045 conmon[33655]: cluster 2021-07-31T03:42:13.436434+0000 mgr.x 2021-07-31T03:42:15.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:14 smithi045 conmon[33655]: (mgr.24542) 4927 : cluster [DBG] pgmap v5458: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:17.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:16 smithi154 conmon[34553]: cluster 2021-07-31T03:42:15.436837+0000 mgr.x (mgr.24542) 4928 : cluster [DBG] pgmap v5459: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:17.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:16 smithi045 conmon[33655]: cluster 2021-07-31T03:42:15.436837+0000 mgr.x (mgr.24542) 4928 : cluster 2021-07-31T03:42:17.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:16 smithi045 conmon[33655]: [DBG] pgmap v5459: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:17.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:16 smithi045 conmon[39718]: cluster 2021-07-31T03:42:15.436837+0000 mgr.x (mgr. 2021-07-31T03:42:17.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:16 smithi045 conmon[39718]: 24542) 4928 : cluster [DBG] pgmap v5459: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:19.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:18 smithi154 conmon[34553]: cluster 2021-07-31T03:42:17.437423+0000 mgr.x (mgr.24542) 4929 : cluster [DBG] pgmap v5460: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:19.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:18 smithi045 conmon[33655]: cluster 2021-07-31T03:42:17.437423+0000 mgr.x (mgr.24542) 4929 2021-07-31T03:42:19.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:18 smithi045 conmon[33655]: : cluster [DBG] pgmap v5460: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:19.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:18 smithi045 conmon[39718]: cluster 2021-07-31T03:42:17.437423+0000 mgr.x (mgr. 2021-07-31T03:42:19.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:18 smithi045 conmon[39718]: 24542) 4929 : cluster [DBG] pgmap v5460: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:21.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:20 smithi154 conmon[34553]: cluster 2021-07-31T03:42:19.437860+0000 mgr.x (mgr.24542) 4930 : cluster [DBG] pgmap v5461: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:21.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:20 smithi045 conmon[33655]: cluster 2021-07-31T03:42:19.437860+0000 2021-07-31T03:42:21.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:20 smithi045 conmon[33655]: mgr.x (mgr.24542) 4930 : cluster [DBG] pgmap v5461: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:21.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:20 smithi045 conmon[39718]: cluster 2021-07-31T03:42:19.437860+0000 mgr.x (mgr. 2021-07-31T03:42:21.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:20 smithi045 conmon[39718]: 24542) 4930 : cluster [DBG] pgmap v5461: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:23.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:22 smithi154 conmon[34553]: cluster 2021-07-31T03:42:21.438299+0000 mgr.x (mgr.24542) 4931 : cluster [DBG] pgmap v5462: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:23.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:22 smithi045 conmon[33655]: cluster 2021-07-31T03:42:21.438299 2021-07-31T03:42:23.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:22 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 4931 : cluster [DBG] pgmap v5462: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:23.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:22 smithi045 conmon[39718]: cluster 2021-07-31T03:42:21.438299+0000 mgr.x 2021-07-31T03:42:23.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:22 smithi045 conmon[39718]: (mgr.24542) 4931 : cluster [DBG] pgmap v5462: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:24.540 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:42:24.817 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:42:24.817 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 117m ago 118m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:42:24.817 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 114m ago 117m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:42:24.818 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 114m ago 2h 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:42:24.818 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 117m ago 2h - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:42:24.818 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 117m ago 2h - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:42:24.818 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 114m ago 2h 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:42:24.818 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 117m ago 2h - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:42:24.819 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 117m ago 118m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:42:24.819 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 114m ago 118m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:42:24.819 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 117m ago 2h - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:42:24.819 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 117m ago 2h - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:42:24.820 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 117m ago 2h - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:42:24.820 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 117m ago 2h - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:42:24.820 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 114m ago 119m 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:42:24.820 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 114m ago 119m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:42:24.820 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 114m ago 119m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:42:24.821 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 114m ago 118m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:42:24.821 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 114m ago 118m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:42:25.153 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:42:25.154 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:42:25.154 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:42:25.154 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:42:25.154 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:42:25.155 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:42:25.155 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:42:25.155 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:42:25.155 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:42:25.155 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:42:25.156 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:42:25.156 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:42:25.156 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:42:25.156 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:42:25.157 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:42:25.157 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:42:25.157 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:42:25.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:24 smithi154 conmon[34553]: cluster 2021-07-31T03:42:23.438761+0000 mgr.x (mgr.24542) 4932 : cluster [DBG] pgmap v5463: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:25.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:24 smithi045 conmon[33655]: cluster 2021-07-31T03:42:23.438761+0000 mgr.x (mgr.24542) 4932 : cluster [DBG] pgmap v5463: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:25.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:24 smithi045 conmon[39718]: cluster 2021-07-31T03:42:23.438761+0000 2021-07-31T03:42:25.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:24 smithi045 conmon[39718]: mgr.x (mgr.24542) 4932 : cluster [DBG] pgmap v5463: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:26.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:25 smithi154 conmon[34553]: audit 2021-07-31T03:42:24.529691+0000 mgr.x (mgr.24542) 4933 : audit [DBG] from='client.20064 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:42:26.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:25 smithi154 conmon[34553]: 2021-07-31T03:42:26.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:25 smithi154 conmon[34553]: audit 2021-07-31T03:42:24.810287+0000 mgr.x (mgr.24542) 4934 : audit [DBG] from='client.30010 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:42:26.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:25 smithi154 conmon[34553]: audit 2021-07-31T03:42:25.154539+0000 mon.c (mon.1) 224 : audit [DBG] from='client.? 172.21.15.45:0/3056373499' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:42:26.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:25 smithi045 conmon[33655]: audit 2021-07-31T03:42:24.529691+0000 mgr.x (mgr.24542) 4933 : audit [DBG] from='client.20064 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:42:26.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:25 smithi045 conmon[33655]: audit 2021-07-31T03:42:24.810287+0000 mgr.x (mgr.24542) 4934 : audit [DBG] from='client.30010 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:42:26.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:25 smithi045 conmon[33655]: audit 2021-07-31T03:42:25.154539+0000 mon.c (mon.1) 224 : 2021-07-31T03:42:26.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:25 smithi045 conmon[33655]: audit [DBG] from='client.? 172.21.15.45:0/3056373499' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:42:26.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:25 smithi045 conmon[39718]: audit 2021-07-31T03:42: 2021-07-31T03:42:26.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:25 smithi045 conmon[39718]: 24.529691+0000 mgr.x (mgr.24542) 4933 : audit [DBG] from='client.20064 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:42:26.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:25 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T03:42:26.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:25 smithi045 conmon[39718]: T03:42:24.810287+0000 mgr.x (mgr.24542) 4934 : audit 2021-07-31T03:42:26.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:25 smithi045 conmon[39718]: [DBG] from='client.30010 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:42:26.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:25 smithi045 conmon[39718]: audit 2021-07 2021-07-31T03:42:26.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:25 smithi045 conmon[39718]: -31T03:42:25.154539+0000 mon.c (mon. 2021-07-31T03:42:26.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:25 smithi045 conmon[39718]: 1) 224 : audit [DBG] from='client.? 172.21.15.45:0/3056373499' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:42:26.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:25 smithi045 conmon[39718]: 2021-07-31T03:42:27.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:26 smithi154 conmon[34553]: cluster 2021-07-31T03:42:25.439373+0000 mgr.x (mgr.24542) 2021-07-31T03:42:27.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:26 smithi154 conmon[34553]: 4935 : cluster [DBG] pgmap v5464: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:27.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:26 smithi045 conmon[33655]: cluster 2021-07-31T03:42:25.439373+0000 mgr.x (mgr.24542) 2021-07-31T03:42:27.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:26 smithi045 conmon[33655]: 4935 : cluster [DBG] pgmap v5464: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:27.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:26 smithi045 conmon[39718]: cluster 2021-07-31T03:42:25.439373+0000 mgr.x 2021-07-31T03:42:27.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:26 smithi045 conmon[39718]: (mgr.24542) 4935 : cluster [DBG] pgmap v5464: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:29.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:28 smithi154 conmon[34553]: cluster 2021-07-31T03:42:27.439874+0000 mgr.x (mgr.24542) 4936 : cluster [DBG] pgmap v5465: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:29.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:28 smithi154 conmon[34553]: 2021-07-31T03:42:29.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:28 smithi154 conmon[34553]: audit 2021-07-31T03:42:28.393331+0000 mon.b (mon.2) 650 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:42:29.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:28 smithi154 conmon[34553]: audit 2021-07-31T03:42:28.393612+0000 mon.a (mon.0) 3350 2021-07-31T03:42:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:28 smithi154 conmon[34553]: : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:42:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:28 smithi154 conmon[34553]: audit 2021-07-31T03:42:28.393633+0000 mon.b ( 2021-07-31T03:42:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:28 smithi154 conmon[34553]: mon.2) 651 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:42:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:28 smithi154 conmon[34553]: audit 2021-07-31T03:42:28. 2021-07-31T03:42:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:28 smithi154 conmon[34553]: 393933+0000 mon.a (mon.0) 3351 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:42:29.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:28 smithi045 conmon[33655]: cluster 2021-07-31T03 2021-07-31T03:42:29.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:28 smithi045 conmon[33655]: :42:27.439874+0000 mgr.x (mgr.24542) 4936 : cluster [DBG] pgmap v5465: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:29.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:28 smithi045 conmon[33655]: audit 2021-07-31T03:42:28.393331+0000 mon.b (mon. 2021-07-31T03:42:29.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:28 smithi045 conmon[33655]: 2) 650 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:42:29.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:28 smithi045 conmon[33655]: audit 2021-07-31 2021-07-31T03:42:29.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:28 smithi045 conmon[33655]: T03:42:28.393612+0000 mon.a (mon.0) 3350 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:42:29.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:28 smithi045 conmon[33655]: audit 2021-07-31T03:42:28.393633 2021-07-31T03:42:29.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:28 smithi045 conmon[33655]: +0000 mon.b (mon.2) 651 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:42:29.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:28 smithi045 conmon[33655]: audit 2021-07-31T03:42:28.393933+0000 2021-07-31T03:42:29.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:28 smithi045 conmon[33655]: mon.a (mon.0) 3351 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:42:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:28 smithi045 conmon[39718]: cluster 2021-07-31T03:42:27. 2021-07-31T03:42:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:28 smithi045 conmon[39718]: 439874+0000 mgr.x (mgr.24542) 4936 : cluster [DBG] pgmap v5465: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:28 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T03:42:29.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:28 smithi045 conmon[39718]: 31T03:42:28.393331 2021-07-31T03:42:29.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:28 smithi045 conmon[39718]: +0000 mon.b (mon.2) 2021-07-31T03:42:29.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:28 smithi045 conmon[39718]: 650 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:42:29.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:28 smithi045 conmon[39718]: audit 2021- 2021-07-31T03:42:29.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:28 smithi045 conmon[39718]: 07-31T03:42:28.393612 2021-07-31T03:42:29.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:28 smithi045 conmon[39718]: +0000 mon.a (mon.0) 3350 2021-07-31T03:42:29.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:28 smithi045 conmon[39718]: : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:42:29.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:28 smithi045 conmon[39718]: audit 2021 2021-07-31T03:42:29.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:28 smithi045 conmon[39718]: -07-31T03:42:28 2021-07-31T03:42:29.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:28 smithi045 conmon[39718]: .393633+0000 mon.b (mon 2021-07-31T03:42:29.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:28 smithi045 conmon[39718]: .2) 651 : audit [INF] 2021-07-31T03:42:29.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:28 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:42:29.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:28 smithi045 conmon[39718]: audit 2021- 2021-07-31T03:42:29.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:28 smithi045 conmon[39718]: 07-31T03:42:28.393933 2021-07-31T03:42:29.282 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:28 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2021-07-31T03:42:29.282 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:28 smithi045 conmon[39718]: 3351 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:42:31.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:30 smithi154 conmon[34553]: cluster 2021-07-31T03:42:29.440300+0000 mgr.x (mgr.24542) 4937 : cluster [DBG] pgmap v5466: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:31.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:30 smithi045 conmon[39718]: cluster 2021-07-31T03:42:29.440300 2021-07-31T03:42:31.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:30 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 4937 : cluster [DBG] pgmap v5466: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:31.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:30 smithi045 conmon[33655]: cluster 2021-07-31T03:42:29.440300+0000 mgr.x (mgr.24542) 4937 2021-07-31T03:42:31.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:30 smithi045 conmon[33655]: : cluster [DBG] pgmap v5466: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:33.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:32 smithi154 conmon[34553]: cluster 2021-07-31T03:42:31.440913+0000 mgr.x (mgr.24542) 4938 : cluster [DBG] pgmap v5467: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:33.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:32 smithi045 conmon[39718]: cluster 2021-07-31T03:42:31.440913+0000 mgr.x 2021-07-31T03:42:33.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:32 smithi045 conmon[39718]: (mgr.24542) 4938 : cluster [DBG] pgmap v5467: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:33.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:32 smithi045 conmon[33655]: cluster 2021-07-31T03:42:31.440913+0000 mgr.x (mgr.24542) 2021-07-31T03:42:33.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:32 smithi045 conmon[33655]: 4938 : cluster [DBG] pgmap v5467: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:34.981 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: debug 2021-07-31T03:42:34.698+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:42:34.981 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:42:34.981 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:42:34.982 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: yield 2021-07-31T03:42:34.982 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:42:34.982 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:42:34.982 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:42:34.983 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:42:34.983 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:42:34.983 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:42:34.983 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:42:34.983 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:42:34.984 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:42:34.984 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: task.result() 2021-07-31T03:42:34.984 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:42:34.984 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: debug 2021-07-31T03:42:34.710+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:42:34.984 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:42:34.985 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:42:34.985 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: yield 2021-07-31T03:42:34.985 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:42:34.985 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:42:34.985 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:42:34.986 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:42:34.986 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:42:34.986 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:42:34.986 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:42:34.986 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:42:34.987 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:42:34.987 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: task.result() 2021-07-31T03:42:34.987 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:42:34.987 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: debug 2021-07-31T03:42:34.736+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:42:34.987 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:42:34.988 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:42:34.988 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: yield 2021-07-31T03:42:34.988 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:42:34.988 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:42:34.988 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:42:34.989 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:42:34.989 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:42:34.989 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:42:34.989 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:42:34.989 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:42:34.990 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:42:34.990 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: task.result() 2021-07-31T03:42:34.990 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:42:34.990 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: debug 2021-07-31T03:42:34.736+0000 7fcb94162700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:42:34.990 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:42:34.991 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:42:34.991 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: yield 2021-07-31T03:42:34.991 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:42:34.991 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:42:34.991 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:42:34.992 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:42:34.992 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:42:34.992 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:42:34.992 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:42:34.992 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:42:34.993 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:42:34.993 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: task.result() 2021-07-31T03:42:34.993 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:42:34.993 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: 2021-07-31T03:42:34.993 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:42:34.993 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: 2021-07-31T03:42:34.994 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:42:34.994 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:42:34.994 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:42:34.994 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:42:34.995 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:42:34.995 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:42:34.995 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:42:34.995 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:42:34.995 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:42:34.996 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:42:34.996 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:42:34.996 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:42:34.996 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:42:34.996 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:42:34.997 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:42:34.997 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:42:34.997 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:42:34.998 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:42:34.998 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: raise 2021-07-31T03:42:34.998 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:42:34.998 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:42:34.998 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:42:34.999 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:42:34.999 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:42:34.999 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: debug 2021-07-31T03:42:34.744+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:42:34.999 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:42:34.999 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:42:35.000 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: yield 2021-07-31T03:42:35.000 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:42:35.000 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:42:35.000 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:42:35.000 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:42:35.001 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:42:35.001 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:42:35.001 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:42:35.001 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:42:35.001 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:42:35.002 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: task.result() 2021-07-31T03:42:35.002 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:42:35.002 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: debug 2021-07-31T03:42:34.745+0000 7fcb93961700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:42:35.002 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:42:35.002 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:42:35.003 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: yield 2021-07-31T03:42:35.003 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:42:35.003 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:42:35.003 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:42:35.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:42:35.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:42:35.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:42:35.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:42:35.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:42:35.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:42:35.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: task.result() 2021-07-31T03:42:35.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:42:35.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: 2021-07-31T03:42:35.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:42:35.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: 2021-07-31T03:42:35.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:42:35.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:42:35.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:42:35.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:42:35.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:42:35.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:42:35.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:42:35.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:42:35.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:42:35.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:42:35.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:42:35.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:42:35.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:42:35.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:42:35.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:42:35.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:42:35.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:42:35.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:42:35.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: raise 2021-07-31T03:42:35.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:42:35.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:42:35.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:42:35.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:42:35.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:42:35.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[34553]: cluster 2021-07-31T03:42:33.441493+0000 mgr.x (mgr.24542) 4939 : cluster [DBG] pgmap v5468: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:35.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[34553]: audit 2021-07-31T03: 2021-07-31T03:42:35.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[34553]: 42:34.652404+0000 mon.b (mon.2) 652 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:42:35.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[34553]: audit 2021-07-31T03:42:34.653796+0000 mon.b (mon.2) 653 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:42:35.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[34553]: audit 2021-07-31T03:42:35.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[34553]: 2021-07-31T03:42:34 2021-07-31T03:42:35.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[34553]: .654719+0000 mon.b (mon. 2021-07-31T03:42:35.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[34553]: 2) 654 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:42:35.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:34 smithi154 conmon[34553]: 2021-07-31T03:42:35.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:34 smithi045 conmon[39718]: cluster 2021-07-31T03:42:33.441493+0000 2021-07-31T03:42:35.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:34 smithi045 conmon[39718]: mgr.x (mgr.24542) 4939 : cluster [DBG] pgmap v5468: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:35.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:34 smithi045 conmon[39718]: audit 2021-07-31T03:42:34.652404+0000 2021-07-31T03:42:35.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:34 smithi045 conmon[39718]: mon.b (mon.2) 652 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:42:35.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:34 smithi045 conmon[39718]: audit 2021-07-31T03:42:35.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:34 smithi045 conmon[39718]: 2021-07-31T03:42:34.653796+0000 mon.b ( 2021-07-31T03:42:35.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:34 smithi045 conmon[39718]: mon.2) 653 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:42:35.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:34 smithi045 conmon[39718]: audit 2021-07-31T03:42:34. 2021-07-31T03:42:35.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:34 smithi045 conmon[39718]: 654719+0000 mon.b (mon.2) 654 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:42:35.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:34 smithi045 conmon[33655]: cluster 2021-07-31T03:42:33.441493+0000 mgr.x ( 2021-07-31T03:42:35.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:34 smithi045 conmon[33655]: mgr.24542) 4939 : cluster [DBG] pgmap v5468: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:35.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:34 smithi045 conmon[33655]: audit 2021-07-31T03:42:34.652404+0000 mon.b (mon. 2021-07-31T03:42:35.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:34 smithi045 conmon[33655]: 2) 652 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:42:35.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:34 smithi045 conmon[33655]: audit 2021-07-31T 2021-07-31T03:42:35.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:34 smithi045 conmon[33655]: 03:42:34.653796+0000 mon.b (mon.2) 653 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:42:35.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:34 smithi045 conmon[33655]: audit 2021-07-31T03:42:34 2021-07-31T03:42:35.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:34 smithi045 conmon[33655]: .654719+0000 mon.b (mon.2) 654 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:42:36.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:42:34.655543+0000 mgr.x (mgr.24542) 4940 : cephadm [WRN] 2021-07-31T03:42:36.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:42:36.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:42:34.699620+0000 mgr.x (mgr.24542) 4941 : cephadm [ERR] 2021-07-31T03:42:36.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:42:36.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:42:36.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: yield 2021-07-31T03:42:36.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:42:36.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:42:36.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:42:36.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:42:36.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:42:36.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:42:36.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:42:36.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:42:36.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:42:36.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: task.result() 2021-07-31T03:42:36.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:42:36.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:42:34. 2021-07-31T03:42:36.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: 700045+0000 mgr.x (mgr.24542) 4942 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:42:36.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:42:34.711157+0000 mgr.x (mgr.24542) 4943 : cephadm [ERR] 2021-07-31T03:42:36.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:42:36.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:42:36.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: yield 2021-07-31T03:42:36.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:42:36.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:42:36.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:42:36.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:42:36.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:42:36.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:42:36.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:42:36.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:42:36.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:42:36.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: task.result() 2021-07-31T03:42:36.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:42:36.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:42:34.712051+0000 mgr.x (mgr.24542) 4944 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:42:36.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: cephadm 2021- 2021-07-31T03:42:36.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: 07-31T03:42:34. 2021-07-31T03:42:36.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: 737113+0000 mgr.x (mgr.24542) 2021-07-31T03:42:36.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: 4945 : cephadm [ERR] 2021-07-31T03:42:36.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:42:36.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:42:36.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: yield 2021-07-31T03:42:36.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:42:36.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:42:36.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:42:36.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:42:36.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:42:36.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:42:36.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:42:36.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:42:36.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:42:36.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: task.result() 2021-07-31T03:42:36.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:42:36.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:42:34.737586+0000 mgr.x (mgr.24542) 4946 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:42:36.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:42:36.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:42:36.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: yield 2021-07-31T03:42:36.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:42:36.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:42:36.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:42:36.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:42:36.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:42:36.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:42:36.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:42:36.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:42:36.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:42:36.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: task.result() 2021-07-31T03:42:36.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:42:36.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: 2021-07-31T03:42:36.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:42:36.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: 2021-07-31T03:42:36.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:42:36.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:42:36.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:42:36.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:42:36.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:42:36.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:42:36.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:42:36.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:42:36.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:42:36.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:42:36.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:42:36.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:42:36.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:42:36.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:42:36.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:42:36.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:42:36.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:42:36.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:42:36.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: raise 2021-07-31T03:42:36.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:42:36.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:42:36.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:42:36.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:42:36.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:42:36.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:42:34.745572+0000 mgr.x (mgr.24542) 4947 : cephadm [ERR] 2021-07-31T03:42:36.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:42:36.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:42:36.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: yield 2021-07-31T03:42:36.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:42:36.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:42:36.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:42:36.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:42:36.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:42:36.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:42:36.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:42:36.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:42:36.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:42:36.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: task.result() 2021-07-31T03:42:36.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:42:36.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:42:34.745968+0000 mgr.x (mgr.24542) 4948 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:42:36.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:42:36.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:42:36.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: yield 2021-07-31T03:42:36.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:42:36.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:42:36.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:42:36.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:42:36.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:42:36.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:42:36.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:42:36.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:42:36.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:42:36.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: task.result() 2021-07-31T03:42:36.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:42:36.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: 2021-07-31T03:42:36.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:42:36.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: 2021-07-31T03:42:36.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:42:36.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:42:36.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:42:36.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:42:36.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:42:36.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:42:36.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:42:36.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:42:36.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:42:36.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:42:36.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:42:36.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:42:36.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:42:36.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:42:36.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:42:36.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:42:36.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:42:36.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:42:36.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: raise 2021-07-31T03:42:36.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:42:36.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:42:36.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:42:36.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:42:36.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:35 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:42:36.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:42:34.655543+0000 mgr.x (mgr. 2021-07-31T03:42:36.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: 24542) 4940 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:42:36.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:42:34.699620+0000 mgr.x (mgr.24542 2021-07-31T03:42:36.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: ) 4941 : cephadm [ERR] 2021-07-31T03:42:36.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:42:36.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:42:36.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: yield 2021-07-31T03:42:36.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:42:36.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:42:36.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:42:36.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:42:36.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:42:36.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:42:36.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:42:36.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:42:36.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:42:36.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: task.result() 2021-07-31T03:42:36.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:42:36.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:42:34.700045+0000 mgr.x (mgr.24542) 4942 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:42:36.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:42:34.711157+0000 mgr.x (mgr.24542) 4943 : cephadm [ERR] 2021-07-31T03:42:36.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:42:36.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:42:36.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: yield 2021-07-31T03:42:36.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:42:36.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:42:36.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:42:36.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:42:36.289 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:42:36.289 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:42:36.289 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:42:36.289 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:42:36.289 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:42:36.290 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: task.result() 2021-07-31T03:42:36.290 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:42:36.290 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: 2021-07-31T03:42:36.290 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:42:34.712051+0000 mgr.x (mgr.24542) 4944 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:42:36.290 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:42:34.737113+0000 2021-07-31T03:42:36.291 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: mgr.x (mgr.24542) 4945 : cephadm [ERR] 2021-07-31T03:42:36.291 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:42:36.291 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:42:36.291 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: yield 2021-07-31T03:42:36.291 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:42:36.291 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:42:36.292 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:42:36.292 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:42:36.292 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:42:36.292 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:42:36.292 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:42:36.293 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:42:36.293 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:42:36.293 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: task.result() 2021-07-31T03:42:36.293 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:42:36.293 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:42:34.737586+0000 mgr.x (mgr.24542) 4946 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:42:36.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:42:36.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:42:36.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: yield 2021-07-31T03:42:36.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:42:36.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:42:36.295 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:42:36.295 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:42:36.295 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:42:36.295 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:42:36.295 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:42:36.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:42:36.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:42:36.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: task.result() 2021-07-31T03:42:36.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:42:36.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: 2021-07-31T03:42:36.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:42:36.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: 2021-07-31T03:42:36.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:42:36.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:42:36.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:42:36.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:42:36.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:42:36.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:42:36.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:42:36.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:42:36.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:42:36.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:42:36.299 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:42:34.655543 2021-07-31T03:42:36.299 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 4940 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:42:36.300 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:42:36.300 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: 2021-07-31T03:42:34.699620+0000 mgr.x (mgr.24542) 4941 : 2021-07-31T03:42:36.300 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: cephadm [ERR] 2021-07-31T03:42:36.300 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:42:36.300 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:42:36.301 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: yield 2021-07-31T03:42:36.301 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:42:36.301 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:42:36.301 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:42:36.301 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:42:36.302 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:42:36.302 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:42:36.302 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:42:36.302 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:42:36.302 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:42:36.303 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: task.result() 2021-07-31T03:42:36.303 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:42:36.303 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:42:34.700045+0000 mgr.x (mgr.24542) 4942 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:42:36.303 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: cephadm 2021 2021-07-31T03:42:36.303 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: -07-31T03:42:34.711157+0000 mgr.x (mgr.24542) 4943 : cephadm [ERR] 2021-07-31T03:42:36.303 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: 2021-07-31T03:42:36.304 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:42:36.304 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:42:36.304 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: yield 2021-07-31T03:42:36.304 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:42:36.304 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:42:36.305 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:42:36.305 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:42:36.305 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:42:36.305 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:42:36.305 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:42:36.306 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:42:36.306 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:42:36.306 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: task.result() 2021-07-31T03:42:36.306 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:42:36.306 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:42:34.712051+0000 mgr.x (mgr.24542) 4944 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:42:36.307 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:42:34.737113+0000 mgr.x (mgr.24542) 4945 : cephadm [ERR] 2021-07-31T03:42:36.307 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:42:36.307 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:42:36.307 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:42:36.308 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:42:36.308 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:42:36.308 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:42:36.308 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:42:36.308 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:42:36.309 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:42:36.309 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: raise 2021-07-31T03:42:36.309 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:42:36.309 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:42:36.309 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:42:36.310 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:42:36.310 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:42:36.310 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:42:34.745572+0000 mgr.x (mgr.24542) 4947 : 2021-07-31T03:42:36.310 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: cephadm [ERR] 2021-07-31T03:42:36.310 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:42:36.311 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:42:36.311 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: yield 2021-07-31T03:42:36.311 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:42:36.311 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:42:36.311 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:42:36.312 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:42:36.312 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:42:36.312 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:42:36.312 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:42:36.312 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:42:36.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:42:36.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: task.result() 2021-07-31T03:42:36.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:42:36.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:42:34.745968+0000 mgr.x (mgr.24542) 4948 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:42:36.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:42:36.314 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:42:36.314 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: yield 2021-07-31T03:42:36.314 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:42:36.314 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:42:36.314 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:42:36.315 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:42:36.315 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:42:36.315 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:42:36.315 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: yield 2021-07-31T03:42:36.316 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:42:36.316 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:42:36.316 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:42:36.316 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:42:36.317 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:42:36.317 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:42:36.317 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:42:36.317 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:42:36.317 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:42:36.317 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: task.result() 2021-07-31T03:42:36.318 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:42:36.318 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: cephadm 2021-07- 2021-07-31T03:42:36.318 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: 31T03:42:34.737586+0000 mgr.x (mgr.24542) 4946 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:42:36.318 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:42:36.318 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:42:36.319 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: yield 2021-07-31T03:42:36.319 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:42:36.319 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:42:36.319 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:42:36.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:42:36.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:42:36.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:42:36.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:42:36.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:42:36.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:42:36.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: task.result() 2021-07-31T03:42:36.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:42:36.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: 2021-07-31T03:42:36.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:42:36.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: 2021-07-31T03:42:36.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:42:36.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:42:36.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:42:36.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:42:36.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:42:36.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:42:36.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:42:36.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:42:36.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:42:36.324 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:42:36.324 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:42:36.325 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:42:36.325 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:42:36.325 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: task.result() 2021-07-31T03:42:36.325 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:42:36.325 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: 2021-07-31T03:42:36.326 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:42:36.326 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: 2021-07-31T03:42:36.326 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:42:36.326 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:42:36.326 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:42:36.327 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:42:36.327 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:42:36.327 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:42:36.327 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:42:36.327 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:42:36.328 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:42:36.328 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:42:36.328 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:42:36.328 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:42:36.328 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:42:36.329 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:42:36.329 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:42:36.329 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:42:36.329 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:42:36.329 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:42:36.330 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: raise 2021-07-31T03:42:36.330 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:42:36.330 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:42:36.330 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:42:36.330 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:42:36.331 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:42:36.331 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:42:36.331 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:42:36.332 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:42:36.332 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:42:36.332 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:42:36.332 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:42:36.332 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:42:36.333 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:42:36.333 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:42:36.333 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: raise 2021-07-31T03:42:36.333 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:42:36.334 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:42:36.334 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:42:36.334 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:42:36.334 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:42:36.334 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:42:34.745572+0000 mgr.x (mgr.24542) 4947 : cephadm [ERR] 2021-07-31T03:42:36.335 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:42:36.335 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:42:36.335 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: yield 2021-07-31T03:42:36.335 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:42:36.336 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:42:36.336 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:42:36.336 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:42:36.336 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:42:36.336 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:42:36.336 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:42:36.337 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:42:36.337 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:42:36.337 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: task.result() 2021-07-31T03:42:36.337 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:42:36.337 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:42:34.745968+0000 mgr.x (mgr. 2021-07-31T03:42:36.338 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: 24542) 4948 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:42:36.338 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:42:36.338 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:42:36.338 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: yield 2021-07-31T03:42:36.339 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:42:36.339 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:42:36.339 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:42:36.339 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:42:36.339 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:42:36.340 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:42:36.340 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:42:36.340 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:42:36.340 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:42:36.341 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: task.result() 2021-07-31T03:42:36.341 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:42:36.341 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: 2021-07-31T03:42:36.341 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:42:36.341 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: 2021-07-31T03:42:36.342 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:42:36.342 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:42:36.342 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:42:36.342 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:42:36.342 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:42:36.342 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:42:36.343 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:42:36.343 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:42:36.343 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:42:36.343 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:42:36.343 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:42:36.344 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:42:36.344 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:42:36.344 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:42:36.344 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:42:36.344 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:42:36.345 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:42:36.345 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:42:36.345 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: raise 2021-07-31T03:42:36.345 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:42:36.345 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:42:36.346 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:42:36.346 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:42:36.346 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:35 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:42:37.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:36 smithi154 conmon[34553]: cluster 2021-07-31T03:42:35.442104+0000 mgr.x (mgr.24542) 4949 : cluster [DBG] pgmap v5469: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:37.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:36 smithi154 conmon[34553]: 2021-07-31T03:42:37.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:36 smithi045 conmon[39718]: cluster 2021-07-31T03:42:35.442104+0000 mgr.x (mgr.24542) 4949 : cluster [DBG] pgmap v5469: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:37.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:36 smithi045 conmon[33655]: cluster 2021-07-31T03:42:35.442104+0000 mgr.x (mgr.24542) 4949 : cluster [DBG] pgmap v5469: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:39.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:38 smithi154 conmon[34553]: cluster 2021-07-31T03:42:37.442581+0000 mgr.x (mgr.24542) 4950 : cluster [DBG] pgmap v5470: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:39.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:38 smithi154 conmon[34553]: 2021-07-31T03:42:39.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:38 smithi045 conmon[39718]: cluster 2021-07-31T03:42:37.442581+0000 mgr.x ( 2021-07-31T03:42:39.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:38 smithi045 conmon[39718]: mgr.24542) 4950 : cluster [DBG] pgmap v5470: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:39.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:38 smithi045 conmon[33655]: cluster 2021-07-31T03:42:37.442581+0000 mgr.x (mgr. 2021-07-31T03:42:39.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:38 smithi045 conmon[33655]: 24542) 4950 : cluster [DBG] pgmap v5470: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:41.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:40 smithi154 conmon[34553]: cluster 2021-07-31T03:42:39.443028+0000 mgr.x (mgr.24542) 4951 : cluster [DBG] pgmap v5471: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:41.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:40 smithi045 conmon[39718]: cluster 2021-07-31T03:42:39.443028+0000 mgr.x (mgr.24542 2021-07-31T03:42:41.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:40 smithi045 conmon[39718]: ) 4951 : cluster [DBG] pgmap v5471: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:41.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:40 smithi045 conmon[33655]: cluster 2021-07-31T03:42:39.443028+0000 mgr.x (mgr.24542) 4951 : 2021-07-31T03:42:41.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:40 smithi045 conmon[33655]: cluster [DBG] pgmap v5471: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:43.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:42 smithi154 conmon[34553]: cluster 2021-07-31T03:42:41.443517+0000 mgr.x (mgr.24542) 4952 : cluster [DBG] 2021-07-31T03:42:43.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:42 smithi154 conmon[34553]: pgmap v5472: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:43.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:42 smithi045 conmon[33655]: cluster 2021-07-31T03:42:41.443517+0000 mgr.x 2021-07-31T03:42:43.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:42 smithi045 conmon[33655]: (mgr.24542) 4952 : cluster [DBG] pgmap v5472: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:43.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:42 smithi045 conmon[39718]: cluster 2021-07-31T03:42:41.443517+0000 mgr.x (mgr. 2021-07-31T03:42:43.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:42 smithi045 conmon[39718]: 24542) 4952 : cluster [DBG] pgmap v5472: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:45.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:45 smithi045 conmon[33655]: cluster 2021-07-31T03:42:43.443866+0000 mgr.x (mgr.24542) 4953 : cluster 2021-07-31T03:42:45.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:45 smithi045 conmon[33655]: [DBG] pgmap v5473: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:45.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:45 smithi045 conmon[39718]: cluster 2021-07-31T03:42:43.443866+0000 mgr.x (mgr.24542) 2021-07-31T03:42:45.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:45 smithi045 conmon[39718]: 4953 : cluster [DBG] pgmap v5473: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:45.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:45 smithi154 conmon[34553]: cluster 2021-07-31T03:42:43.443866+0000 mgr.x (mgr.24542) 4953 : cluster [DBG] pgmap v5473: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:47.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:47 smithi045 conmon[39718]: cluster 2021-07-31T03:42:45.444270+0000 mgr.x (mgr.24542) 4954 2021-07-31T03:42:47.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:47 smithi045 conmon[39718]: : cluster [DBG] pgmap v5474: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:47.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:47 smithi045 conmon[33655]: cluster 2021-07-31T03:42:45.444270+0000 mgr.x (mgr.24542) 4954 : 2021-07-31T03:42:47.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:47 smithi045 conmon[33655]: cluster [DBG] pgmap v5474: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:47.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:47 smithi154 conmon[34553]: cluster 2021-07-31T03:42:45.444270+0000 mgr.x (mgr.24542) 4954 : cluster [DBG] pgmap v5474: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:49.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:49 smithi045 conmon[39718]: cluster 2021-07-31T03:42:47.444722+0000 mgr.x 2021-07-31T03:42:49.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:49 smithi045 conmon[39718]: (mgr.24542) 4955 : cluster [DBG] pgmap v5475: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:49.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:49 smithi045 conmon[33655]: cluster 2021-07-31T03:42:47.444722+0000 mgr.x (mgr.24542) 4955 : cluster 2021-07-31T03:42:49.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:49 smithi045 conmon[33655]: [DBG] pgmap v5475: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:49.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:49 smithi154 conmon[34553]: cluster 2021-07-31T03:42:47.444722+0000 mgr.x (mgr.24542) 4955 : cluster [DBG] pgmap v5475: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:51.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:51 smithi045 conmon[39718]: cluster 2021-07-31T03:42:49.445232+0000 mgr.x (mgr.24542) 4956 : cluster [DBG] pgmap v5476: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:51.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:51 smithi045 conmon[33655]: cluster 2021-07-31T03:42:49.445232+0000 mgr.x (mgr. 2021-07-31T03:42:51.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:51 smithi045 conmon[33655]: 24542) 4956 : cluster [DBG] pgmap v5476: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:51.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:51 smithi154 conmon[34553]: cluster 2021-07-31T03:42:49.445232+0000 mgr.x (mgr.24542) 4956 : cluster [DBG] pgmap v5476: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:53.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:53 smithi045 conmon[39718]: cluster 2021-07-31T03:42:51.445622+0000 mgr.x (mgr. 2021-07-31T03:42:53.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:53 smithi045 conmon[39718]: 24542) 4957 : cluster [DBG] pgmap v5477: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:53.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:53 smithi045 conmon[33655]: cluster 2021-07-31T03:42:51. 2021-07-31T03:42:53.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:53 smithi045 conmon[33655]: 445622+0000 mgr.x (mgr.24542) 4957 : cluster [DBG] pgmap v5477: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:53.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:53 smithi154 conmon[34553]: cluster 2021-07-31T03:42:51.445622+0000 mgr.x (mgr.24542) 4957 : cluster [DBG] pgmap v5477: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:55.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:55 smithi045 conmon[33655]: cluster 2021-07-31T03:42:53.446141+0000 mgr.x (mgr.24542) 4958 : cluster [DBG] pgmap v5478: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:55.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:55 smithi045 conmon[39718]: cluster 2021-07-31T03:42:53.446141+0000 2021-07-31T03:42:55.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:55 smithi045 conmon[39718]: mgr.x (mgr.24542) 4958 : cluster [DBG] pgmap v5478: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:55.467 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:42:55.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:55 smithi154 conmon[34553]: cluster 2021-07-31T03:42:53.446141+0000 mgr.x (mgr.24542) 4958 : cluster [DBG] pgmap v5478: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:55.745 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:42:55.745 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 117m ago 118m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:42:55.745 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 115m ago 118m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:42:55.746 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 115m ago 2h 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:42:55.746 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 117m ago 2h - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:42:55.746 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 117m ago 2h - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:42:55.746 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 115m ago 2h 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:42:55.746 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 117m ago 2h - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:42:55.747 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 117m ago 118m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:42:55.747 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 115m ago 118m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:42:55.747 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 117m ago 2h - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:42:55.747 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 117m ago 2h - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:42:55.747 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 117m ago 2h - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:42:55.748 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 117m ago 2h - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:42:55.748 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 115m ago 2h 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:42:55.748 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 115m ago 119m 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:42:55.748 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 115m ago 119m 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:42:55.748 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 115m ago 119m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:42:55.749 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 115m ago 118m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:42:56.082 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:42:56.082 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:42:56.083 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:42:56.083 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:42:56.083 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:42:56.083 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:42:56.083 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:42:56.084 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:42:56.084 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:42:56.084 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:42:56.084 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:42:56.084 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:42:56.084 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:42:56.085 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:42:56.085 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:42:56.085 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:42:56.085 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:42:57.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:57 smithi045 conmon[33655]: cluster 2021-07-31T03:42:55.446725+0000 mgr.x (mgr.24542) 4959 : 2021-07-31T03:42:57.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:57 smithi045 conmon[33655]: cluster [DBG] pgmap v5479: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:57.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:57 smithi045 conmon[33655]: audit 2021-07-31T03:42:55.456343+0000 mgr.x (mgr.24542) 2021-07-31T03:42:57.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:57 smithi045 conmon[33655]: 4960 : audit [DBG] from='client.20082 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:42:57.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:57 smithi045 conmon[33655]: audit 2021-07-31T03:42:57.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:57 smithi045 conmon[33655]: 2021-07 2021-07-31T03:42:57.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:57 smithi045 conmon[33655]: -31T03:42:55.738287 2021-07-31T03:42:57.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:57 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 4961 : audit 2021-07-31T03:42:57.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:57 smithi045 conmon[33655]: [DBG] from='client.30028 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:42:57.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:57 smithi045 conmon[33655]: audit 2021-07-31T03:42:57.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:57 smithi045 conmon[33655]: 2021-07 2021-07-31T03:42:57.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:57 smithi045 conmon[33655]: -31T03:42:56.083399 2021-07-31T03:42:57.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:57 smithi045 conmon[33655]: +0000 mon.c (mon.1) 225 : 2021-07-31T03:42:57.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:57 smithi045 conmon[33655]: audit [DBG] from='client.? 172.21.15.45:0/3945107619' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:42:57.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:57 smithi045 conmon[39718]: cluster 2021-07-31T03:42:55.446725+0000 2021-07-31T03:42:57.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:57 smithi045 conmon[39718]: mgr.x (mgr.24542) 4959 : cluster [DBG] pgmap v5479: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:57.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:57 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T03:42:57.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:57 smithi045 conmon[39718]: 03:42:55.456343+0000 mgr.x (mgr.24542) 4960 : audit [DBG] from='client.20082 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:42:57.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:57 smithi045 conmon[39718]: audit 2021-07-31T03:42:57.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:57 smithi045 conmon[39718]: 2021-07-31T03:42:55. 2021-07-31T03:42:57.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:57 smithi045 conmon[39718]: 738287+0000 mgr.x (mgr.24542) 4961 : 2021-07-31T03:42:57.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:57 smithi045 conmon[39718]: audit [DBG] from='client.30028 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:42:57.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:57 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T03:42:57.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:57 smithi045 conmon[39718]: 31T03:42:56.083399+0000 mon.c 2021-07-31T03:42:57.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:57 smithi045 conmon[39718]: (mon.1) 225 : audit [DBG] 2021-07-31T03:42:57.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:57 smithi045 conmon[39718]: from='client.? 172.21.15.45:0/3945107619' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:42:57.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:57 smithi154 conmon[34553]: cluster 2021-07-31T03:42:55.446725+0000 mgr.x (mgr.24542) 4959 : cluster [DBG] pgmap v5479: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:57.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:57 smithi154 conmon[34553]: audit 2021-07-31T03:42:55.456343+0000 mgr.x (mgr.24542) 4960 : audit [DBG] from='client.20082 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:42:57.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:57 smithi154 conmon[34553]: audit 2021-07-31T03:42:55.738287+0000 mgr.x (mgr.24542) 4961 : audit [DBG] from='client.30028 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:42:57.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:57 smithi154 conmon[34553]: audit 2021-07-31T03:42:57.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:57 smithi154 conmon[34553]: 2021-07-31T03:42:56.083399+0000 mon.c (mon.1) 225 : audit [DBG] from='client.? 172.21.15.45:0/3945107619' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:42:59.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:42:59 smithi154 conmon[34553]: cluster 2021-07-31T03:42:57.447167+0000 mgr.x (mgr.24542) 4962 : cluster [DBG] pgmap v5480: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:59.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:59 smithi045 conmon[39718]: cluster 2021-07-31T03:42:57.447167+0000 mgr.x 2021-07-31T03:42:59.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:42:59 smithi045 conmon[39718]: (mgr.24542) 4962 : cluster [DBG] pgmap v5480: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:42:59.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:59 smithi045 conmon[33655]: cluster 2021-07-31T03:42:57.447167+0000 mgr.x (mgr.24542) 4962 2021-07-31T03:42:59.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:42:59 smithi045 conmon[33655]: : cluster [DBG] pgmap v5480: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:01.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:01 smithi154 conmon[34553]: cluster 2021-07-31T03:42:59.447705+0000 mgr.x (mgr.24542) 4963 : cluster [DBG] pgmap v5481: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:01.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:01 smithi045 conmon[33655]: cluster 2021-07-31T03:42:59.447705+0000 mgr.x (mgr.24542) 4963 2021-07-31T03:43:01.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:01 smithi045 conmon[33655]: : cluster [DBG] pgmap v5481: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:01.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:01 smithi045 conmon[39718]: cluster 2021-07-31T03:42:59.447705+0000 mgr.x (mgr. 2021-07-31T03:43:01.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:01 smithi045 conmon[39718]: 24542) 4963 : cluster [DBG] pgmap v5481: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:03.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:03 smithi154 conmon[34553]: cluster 2021-07-31T03:43:01.448090+0000 mgr.x (mgr.24542) 4964 : cluster [DBG] pgmap v5482: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:03.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:03 smithi045 conmon[33655]: cluster 2021-07-31T03:43:01.448090+0000 mgr.x (mgr.24542) 4964 : cluster [DBG] pgmap v5482: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:03.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:03 smithi045 conmon[33655]: 2021-07-31T03:43:03.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:03 smithi045 conmon[39718]: cluster 2021-07-31T03:43:01.448090+0000 mgr.x ( 2021-07-31T03:43:03.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:03 smithi045 conmon[39718]: mgr.24542) 4964 : cluster [DBG] pgmap v5482: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:05.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:05 smithi154 conmon[34553]: cluster 2021-07-31T03:43:03.448470+0000 mgr.x (mgr.24542) 4965 : cluster [DBG] 2021-07-31T03:43:05.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:05 smithi154 conmon[34553]: pgmap v5483: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:05.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:05 smithi045 conmon[33655]: cluster 2021-07-31T03:43:03.448470+0000 mgr.x (mgr.24542 2021-07-31T03:43:05.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:05 smithi045 conmon[33655]: ) 4965 : cluster [DBG] pgmap v5483: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:05.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:05 smithi045 conmon[39718]: cluster 2021-07-31T03:43:03. 2021-07-31T03:43:05.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:05 smithi045 conmon[39718]: 448470+0000 mgr.x (mgr.24542) 4965 : cluster [DBG] pgmap v5483: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:07.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:07 smithi154 conmon[34553]: cluster 2021-07-31T03:43:05.449002+0000 mgr.x (mgr.24542) 4966 : cluster 2021-07-31T03:43:07.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:07 smithi154 conmon[34553]: [DBG] pgmap v5484: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:07.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:07 smithi045 conmon[33655]: cluster 2021-07-31T03:43:05.449002+0000 mgr.x (mgr 2021-07-31T03:43:07.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:07 smithi045 conmon[33655]: .24542) 4966 : cluster [DBG] pgmap v5484: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:07.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:07 smithi045 conmon[39718]: cluster 2021-07-31T03:43:05.449002+0000 mgr.x (mgr 2021-07-31T03:43:07.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:07 smithi045 conmon[39718]: .24542) 4966 : cluster [DBG] pgmap v5484: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:09.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:09 smithi154 conmon[34553]: cluster 2021-07-31T03:43:07.449428+0000 mgr.x (mgr.24542) 4967 : cluster [DBG] 2021-07-31T03:43:09.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:09 smithi154 conmon[34553]: pgmap v5485: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:09.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:09 smithi045 conmon[33655]: cluster 2021-07-31T03:43:07.449428+0000 mgr.x (mgr.24542) 2021-07-31T03:43:09.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:09 smithi045 conmon[33655]: 4967 : cluster [DBG] pgmap v5485: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:09.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:09 smithi045 conmon[39718]: cluster 2021-07-31T03:43:07.449428+0000 mgr.x ( 2021-07-31T03:43:09.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:09 smithi045 conmon[39718]: mgr.24542) 4967 : cluster [DBG] pgmap v5485: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:11.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:11 smithi154 conmon[34553]: cluster 2021-07-31T03:43:09.449853+0000 mgr.x (mgr.24542) 4968 : cluster [DBG] pgmap v5486: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:11.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:11 smithi045 conmon[39718]: cluster 2021-07-31T03:43:09.449853+0000 mgr.x (mgr.24542) 4968 : cluster [DBG] pgmap v5486: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:11.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:11 smithi045 conmon[33655]: cluster 2021-07-31T03:43:09.449853 2021-07-31T03:43:11.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:11 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 4968 : cluster [DBG] pgmap v5486: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:13.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:13 smithi154 conmon[34553]: cluster 2021-07-31T03:43:11.450243+0000 mgr.x (mgr.24542) 4969 : cluster [DBG] pgmap v5487: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:13.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:13 smithi045 conmon[39718]: cluster 2021-07-31T03:43:11.450243+0000 mgr.x (mgr. 2021-07-31T03:43:13.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:13 smithi045 conmon[39718]: 24542) 4969 : cluster [DBG] pgmap v5487: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:13.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:13 smithi045 conmon[33655]: cluster 2021-07-31T03:43:11.450243+0000 mgr.x (mgr.24542) 4969 : cluster [DBG] pgmap v5487: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:15.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:15 smithi154 conmon[34553]: cluster 2021-07-31T03:43:13.450605+0000 mgr.x (mgr.24542) 4970 : cluster [DBG] pgmap v5488: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:15.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:15 smithi045 conmon[39718]: cluster 2021-07-31T03:43:13. 2021-07-31T03:43:15.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:15 smithi045 conmon[39718]: 450605+0000 mgr.x (mgr.24542) 4970 : cluster [DBG] pgmap v5488: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:15.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:15 smithi045 conmon[33655]: cluster 2021-07-31T03:43:13.450605+0000 mgr.x (mgr.24542) 4970 : cluster [DBG] pgmap v5488: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:17.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:17 smithi154 conmon[34553]: cluster 2021-07-31T03:43:15.451038+0000 mgr.x (mgr.24542) 4971 : cluster [DBG] pgmap v5489: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:17.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:17 smithi045 conmon[33655]: cluster 2021-07-31T03:43:15.451038+0000 mgr.x (mgr.24542) 4971 : cluster [DBG] pgmap v5489: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:17.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:17 smithi045 conmon[39718]: cluster 2021-07-31T03:43:15.451038+0000 mgr.x (mgr.24542) 4971 : cluster [DBG] pgmap v5489: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:19.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:19 smithi154 conmon[34553]: cluster 2021-07-31T03:43:17.451514+0000 mgr.x (mgr.24542) 4972 : cluster [DBG] pgmap v5490: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:19.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:19 smithi045 conmon[39718]: cluster 2021-07-31T03:43:17.451514+0000 mgr.x (mgr.24542) 4972 : cluster 2021-07-31T03:43:19.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:19 smithi045 conmon[39718]: [DBG] pgmap v5490: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:19.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:19 smithi045 conmon[33655]: cluster 2021-07-31T03:43:17.451514+0000 mgr.x (mgr.24542) 4972 : 2021-07-31T03:43:19.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:19 smithi045 conmon[33655]: cluster [DBG] pgmap v5490: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:21.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:21 smithi154 conmon[34553]: cluster 2021-07-31T03:43:19.451979+0000 mgr.x (mgr.24542) 4973 : cluster [DBG] pgmap v5491: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:21.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:21 smithi045 conmon[39718]: cluster 2021-07-31T03:43:19.451979+0000 mgr.x (mgr.24542) 4973 : 2021-07-31T03:43:21.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:21 smithi045 conmon[39718]: cluster [DBG] pgmap v5491: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:21.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:21 smithi045 conmon[33655]: cluster 2021-07-31T03:43:19.451979+0000 mgr.x (mgr.24542) 4973 : cluster [DBG] 2021-07-31T03:43:21.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:21 smithi045 conmon[33655]: pgmap v5491: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:23.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:23 smithi154 conmon[34553]: cluster 2021-07-31T03:43:21.452708+0000 mgr.x (mgr.24542) 4974 : cluster [DBG] pgmap v5492: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:23.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:23 smithi045 conmon[39718]: cluster 2021-07-31T03:43:21.452708+0000 mgr.x (mgr.24542) 4974 : 2021-07-31T03:43:23.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:23 smithi045 conmon[39718]: cluster [DBG] pgmap v5492: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:23.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:23 smithi045 conmon[33655]: cluster 2021-07-31T03:43:21.452708+0000 mgr.x (mgr.24542) 4974 : cluster 2021-07-31T03:43:23.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:23 smithi045 conmon[33655]: [DBG] pgmap v5492: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:25.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:25 smithi154 conmon[34553]: cluster 2021-07-31T03:43:23.453148+0000 mgr.x (mgr.24542) 4975 : cluster [DBG] pgmap v5493: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:25.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:25 smithi045 conmon[39718]: cluster 2021-07-31T03:43:23.453148+0000 mgr.x (mgr.24542) 4975 : cluster [DBG] 2021-07-31T03:43:25.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:25 smithi045 conmon[39718]: pgmap v5493: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:25.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:25 smithi045 conmon[33655]: cluster 2021-07-31T03:43:23.453148+0000 mgr.x (mgr.24542 2021-07-31T03:43:25.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:25 smithi045 conmon[33655]: ) 4975 : cluster [DBG] pgmap v5493: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:26.392 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:43:26.669 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:43:26.670 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 118m ago 119m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:43:26.670 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 116m ago 118m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:43:26.670 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 116m ago 2h 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:43:26.671 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 118m ago 2h - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:43:26.671 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 118m ago 2h - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:43:26.671 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 116m ago 2h 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:43:26.671 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 118m ago 2h - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:43:26.671 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 118m ago 119m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:43:26.672 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 116m ago 119m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:43:26.672 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 118m ago 2h - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:43:26.672 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 118m ago 2h - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:43:26.672 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 118m ago 2h - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:43:26.673 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 118m ago 2h - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:43:26.673 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 116m ago 2h 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:43:26.673 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 116m ago 2h 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:43:26.673 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 116m ago 2h 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:43:26.673 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 116m ago 119m 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:43:26.674 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 116m ago 119m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:43:27.004 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:43:27.004 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:43:27.004 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:43:27.004 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:43:27.004 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:43:27.005 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:43:27.005 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:43:27.005 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:43:27.005 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:43:27.006 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:43:27.006 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:43:27.006 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:43:27.006 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:43:27.007 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:43:27.007 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:43:27.007 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:43:27.007 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:43:27.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:27 smithi154 conmon[34553]: cluster 2021-07-31T03:43:25.453729+0000 mgr.x (mgr.24542) 4976 : cluster [DBG] pgmap v5494: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:27.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:27 smithi154 conmon[34553]: audit 2021-07-31T 2021-07-31T03:43:27.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:27 smithi154 conmon[34553]: 03:43:27.004755+0000 mon.c (mon.1) 226 : audit [DBG] from='client.? 172.21.15.45:0/3744078065' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:43:27.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:27 smithi045 conmon[33655]: cluster 2021-07-31T03:43:25.453729+0000 mgr.x (mgr. 2021-07-31T03:43:27.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:27 smithi045 conmon[33655]: 24542) 4976 : cluster [DBG] pgmap v5494: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:27.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:27 smithi045 conmon[33655]: audit 2021-07-31T03:43:27.004755+0000 mon.c 2021-07-31T03:43:27.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:27 smithi045 conmon[33655]: (mon.1) 226 : audit [DBG] from='client.? 172.21.15.45:0/3744078065' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:43:27.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:27 smithi045 conmon[39718]: cluster 2021-07-31T03:43:25.453729 2021-07-31T03:43:27.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:27 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 4976 : cluster [DBG] pgmap v5494: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:27.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:27 smithi045 conmon[39718]: audit 2021- 2021-07-31T03:43:27.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:27 smithi045 conmon[39718]: 07-31T03:43:27.004755+0000 2021-07-31T03:43:27.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:27 smithi045 conmon[39718]: mon.c (mon.1) 226 : 2021-07-31T03:43:27.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:27 smithi045 conmon[39718]: audit [DBG] from='client.? 172.21.15.45:0/3744078065' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:43:28.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:28 smithi154 conmon[34553]: audit 2021-07-31T03:43:26.382091+0000 mgr.x (mgr.24542) 4977 : audit [DBG] from='client.30040 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:43:28.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:28 smithi154 conmon[34553]: audit 2021-07-31T03:43:26.662518+0000 mgr.x (mgr.24542) 4978 : audit [DBG] from='client.20106 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:43:28.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:28 smithi045 conmon[39718]: audit 2021-07-31T03:43:26.382091+0000 mgr.x ( 2021-07-31T03:43:28.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:28 smithi045 conmon[39718]: mgr.24542) 4977 : audit [DBG] from='client.30040 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:43:28.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:28 smithi045 conmon[39718]: audit 2021-07-31T03:43:26.662518+0000 mgr.x 2021-07-31T03:43:28.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:28 smithi045 conmon[39718]: (mgr.24542) 4978 : audit [DBG] from='client.20106 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:43:28.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:28 smithi045 conmon[33655]: audit 2021-07-31T03:43:26.382091+0000 mgr.x (mgr.24542) 4977 : audit [DBG] 2021-07-31T03:43:28.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:28 smithi045 conmon[33655]: from='client.30040 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:43:28.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:28 smithi045 conmon[33655]: audit 2021-07-31T03:43:26.662518+0000 mgr.x (mgr.24542) 4978 : audit [DBG] from='client.20106 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:43:29.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:29 smithi154 conmon[34553]: cluster 2021-07-31T03:43:27.454120+0000 mgr.x (mgr.24542) 4979 : cluster [DBG] pgmap v5495: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:29.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:29 smithi154 conmon[34553]: audit 2021-07-31T03:43:28.396914+0000 mon.b (mon.2) 655 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:43:29.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:29 smithi154 conmon[34553]: audit 2021-07-31T03:43:28.397188+0000 mon.b (mon.2) 656 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:43:29.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:29 smithi154 conmon[34553]: audit 2021-07-31T03:43:28.397378+0000 2021-07-31T03:43:29.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:29 smithi154 conmon[34553]: mon.a (mon.0) 3352 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:43:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:29 smithi154 conmon[34553]: audit 2021-07-31T03:43:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:29 smithi154 conmon[34553]: 2021-07-31T03:43:28. 2021-07-31T03:43:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:29 smithi154 conmon[34553]: 397747+0000 mon.a (mon.0) 3353 : audit [INF] 2021-07-31T03:43:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:29 smithi154 conmon[34553]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:43:29.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:29 smithi045 conmon[39718]: cluster 2021-07-31T03:43:27.454120+0000 mgr.x (mgr.24542) 4979 : cluster [DBG] pgmap v5495: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:29.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:29 smithi045 conmon[39718]: audit 2021-07-31T03:43:28.396914+0000 mon.b (mon.2) 655 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:43:29.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:29 smithi045 conmon[39718]: audit 2021-07-31T03:43:28.397188 2021-07-31T03:43:29.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:29 smithi045 conmon[39718]: +0000 mon.b (mon.2) 656 : audit [INF] 2021-07-31T03:43:29.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:29 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:43:29.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:29 smithi045 conmon[39718]: audit 2021-07-31T03:43 2021-07-31T03:43:29.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:29 smithi045 conmon[39718]: :28.397378+0000 mon.a (mon.0) 2021-07-31T03:43:29.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:29 smithi045 conmon[39718]: 3352 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:43:29.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:29 smithi045 conmon[39718]: audit 2021 2021-07-31T03:43:29.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:29 smithi045 conmon[39718]: -07-31T03:43:28.397747 2021-07-31T03:43:29.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:29 smithi045 conmon[39718]: +0000 mon.a (mon.0) 3353 : 2021-07-31T03:43:29.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:29 smithi045 conmon[39718]: audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:43:29.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:29 smithi045 conmon[33655]: cluster 2021-07-31T03:43: 2021-07-31T03:43:29.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:29 smithi045 conmon[33655]: 27.454120+0000 mgr.x (mgr.24542) 4979 : cluster [DBG] pgmap v5495: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:29.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:29 smithi045 conmon[33655]: audit 2021-07-31T03:43:28.396914+0000 mon.b 2021-07-31T03:43:29.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:29 smithi045 conmon[33655]: (mon.2) 655 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:43:29.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:29 smithi045 conmon[33655]: audit 2021-07-31T03:43:29.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:29 smithi045 conmon[33655]: 2021-07-31T03:43 2021-07-31T03:43:29.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:29 smithi045 conmon[33655]: :28.397188+0000 mon.b ( 2021-07-31T03:43:29.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:29 smithi045 conmon[33655]: mon.2) 656 : audit [INF] 2021-07-31T03:43:29.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:29 smithi045 conmon[33655]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:43:29.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:29 smithi045 conmon[33655]: audit 2021-07 2021-07-31T03:43:29.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:29 smithi045 conmon[33655]: -31T03:43:28.397378 2021-07-31T03:43:29.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:29 smithi045 conmon[33655]: +0000 mon.a (mon.0) 3352 2021-07-31T03:43:29.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:29 smithi045 conmon[33655]: : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:43:29.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:29 smithi045 conmon[33655]: audit 2021 2021-07-31T03:43:29.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:29 smithi045 conmon[33655]: -07-31T03:43:28.397747 2021-07-31T03:43:29.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:29 smithi045 conmon[33655]: +0000 mon.a (mon.0) 2021-07-31T03:43:29.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:29 smithi045 conmon[33655]: 3353 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:43:29.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:29 smithi045 conmon[33655]: 2021-07-31T03:43:31.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:31 smithi154 conmon[34553]: cluster 2021-07-31T03:43:29.454714+0000 mgr.x (mgr.24542) 4980 : cluster [DBG] 2021-07-31T03:43:31.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:31 smithi154 conmon[34553]: pgmap v5496: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:31.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:31 smithi045 conmon[33655]: cluster 2021-07-31T03:43:29.454714+0000 mgr.x (mgr.24542) 4980 : 2021-07-31T03:43:31.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:31 smithi045 conmon[33655]: cluster [DBG] pgmap v5496: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:31.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:31 smithi045 conmon[39718]: cluster 2021-07-31T03:43:29.454714+0000 mgr.x (mgr.24542) 4980 2021-07-31T03:43:31.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:31 smithi045 conmon[39718]: : cluster [DBG] pgmap v5496: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:33.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:33 smithi154 conmon[34553]: cluster 2021-07-31T03:43:31.455258+0000 mgr.x (mgr.24542) 2021-07-31T03:43:33.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:33 smithi154 conmon[34553]: 4981 : cluster [DBG] pgmap v5497: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:33.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:33 smithi045 conmon[39718]: cluster 2021-07-31T03:43:31.455258+0000 mgr.x 2021-07-31T03:43:33.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:33 smithi045 conmon[39718]: (mgr.24542) 4981 : cluster [DBG] pgmap v5497: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:33.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:33 smithi045 conmon[33655]: cluster 2021-07-31T03:43:31.455258+0000 mgr.x (mgr.24542) 4981 : cluster [DBG] pgmap v5497: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:35.100 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: debug 2021-07-31T03:43:34.791+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:43:35.100 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:43:35.101 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:43:35.101 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: yield 2021-07-31T03:43:35.101 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:43:35.101 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:43:35.101 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:43:35.102 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:43:35.102 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:43:35.102 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:43:35.102 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:43:35.103 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:43:35.103 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:43:35.103 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: task.result() 2021-07-31T03:43:35.103 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:43:35.103 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: debug 2021-07-31T03:43:34.803+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:43:35.104 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:43:35.104 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:43:35.104 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: yield 2021-07-31T03:43:35.104 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:43:35.104 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:43:35.105 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:43:35.105 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:43:35.105 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:43:35.105 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:43:35.106 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:43:35.106 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:43:35.106 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:43:35.106 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: task.result() 2021-07-31T03:43:35.106 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:43:35.107 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: debug 2021-07-31T03:43:34.829+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:43:35.107 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:43:35.107 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:43:35.107 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: yield 2021-07-31T03:43:35.107 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:43:35.108 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:43:35.108 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:43:35.108 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:43:35.108 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:43:35.109 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:43:35.109 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:43:35.110 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:43:35.110 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:43:35.110 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: task.result() 2021-07-31T03:43:35.110 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:43:35.111 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: debug 2021-07-31T03:43:34.830+0000 7fcb9295f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:43:35.111 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:43:35.111 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:43:35.111 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: yield 2021-07-31T03:43:35.111 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:43:35.112 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:43:35.112 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:43:35.112 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:43:35.112 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:43:35.113 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:43:35.113 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:43:35.113 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:43:35.113 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:43:35.113 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: task.result() 2021-07-31T03:43:35.114 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:43:35.114 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: 2021-07-31T03:43:35.114 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:43:35.114 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: 2021-07-31T03:43:35.115 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:43:35.115 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:43:35.115 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:43:35.115 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:43:35.115 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:43:35.116 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:43:35.116 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:43:35.116 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:43:35.116 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:43:35.116 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:43:35.117 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:43:35.117 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:43:35.117 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:43:35.117 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:43:35.117 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:43:35.118 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:43:35.118 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:43:35.118 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:43:35.118 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: raise 2021-07-31T03:43:35.119 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:43:35.119 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:43:35.119 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:43:35.119 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:43:35.119 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:43:35.120 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: debug 2021-07-31T03:43:34.840+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:43:35.120 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:43:35.120 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:43:35.120 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: yield 2021-07-31T03:43:35.121 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:43:35.121 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:43:35.121 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:43:35.121 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:43:35.121 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:43:35.122 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:43:35.122 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:43:35.122 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:43:35.122 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:43:35.122 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: task.result() 2021-07-31T03:43:35.123 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:43:35.123 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: debug 2021-07-31T03:43:34.840+0000 7fcb93160700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:43:35.123 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:43:35.123 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:43:35.124 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: yield 2021-07-31T03:43:35.124 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:43:35.124 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:43:35.124 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:43:35.124 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:43:35.125 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:43:35.125 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:43:35.125 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:43:35.125 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:43:35.125 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:43:35.126 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: task.result() 2021-07-31T03:43:35.126 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:43:35.126 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: 2021-07-31T03:43:35.126 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:43:35.126 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: 2021-07-31T03:43:35.127 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:43:35.127 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:43:35.127 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:43:35.127 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:43:35.128 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:43:35.128 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:43:35.128 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:43:35.128 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:43:35.128 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:43:35.129 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:43:35.129 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:43:35.129 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:43:35.129 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:43:35.130 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:43:35.130 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:43:35.130 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:43:35.130 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:43:35.130 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:43:35.131 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: raise 2021-07-31T03:43:35.131 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:43:35.131 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:43:35.131 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:43:35.132 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:43:35.132 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:43:34 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:43:35.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:35 smithi154 conmon[34553]: cluster 2021-07-31T03:43:33.455802+0000 mgr.x (mgr.24542) 4982 : 2021-07-31T03:43:35.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:35 smithi154 conmon[34553]: cluster [DBG] pgmap v5498: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:35.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:35 smithi154 conmon[34553]: audit 2021-07-31T03:43:34.746764+0000 mon.b (mon.2) 657 : audit [DBG] 2021-07-31T03:43:35.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:35 smithi154 conmon[34553]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:43:35.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:35 smithi154 conmon[34553]: audit 2021-07-31T03:43:34.748314+0000 2021-07-31T03:43:35.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:35 smithi154 conmon[34553]: mon.b (mon.2) 658 : audit [DBG] 2021-07-31T03:43:35.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:35 smithi154 conmon[34553]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:43:35.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:35 smithi154 conmon[34553]: audit 2021-07-31T03 2021-07-31T03:43:35.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:35 smithi154 conmon[34553]: :43:34.749158+0000 mon.b (mon.2) 2021-07-31T03:43:35.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:35 smithi154 conmon[34553]: 659 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:43:35.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:35 smithi045 conmon[33655]: cluster 2021-07-31T03:43:33.455802+0000 mgr.x (mgr.24542) 4982 : cluster [DBG] pgmap v5498: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:35.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:35 smithi045 conmon[33655]: audit 2021-07-31T03:43:34.746764+0000 mon.b (mon.2) 657 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:43:35.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:35 smithi045 conmon[33655]: audit 2021-07-31T03:43:34.748314+0000 mon.b (mon.2) 658 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:43:35.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:35 smithi045 conmon[33655]: audit 2021-07-31T03:43:34.749158+0000 mon.b (mon.2) 659 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:43:35.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:35 smithi045 conmon[39718]: cluster 2021-07-31T03:43:33.455802+0000 mgr.x (mgr 2021-07-31T03:43:35.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:35 smithi045 conmon[39718]: .24542) 4982 : cluster [DBG] pgmap v5498: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:35.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:35 smithi045 conmon[39718]: audit 2021-07-31T03:43:34.746764+0000 mon.b (mon 2021-07-31T03:43:35.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:35 smithi045 conmon[39718]: .2) 657 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:43:35.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:35 smithi045 conmon[39718]: audit 2021- 2021-07-31T03:43:35.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:35 smithi045 conmon[39718]: 07-31T03:43:34.748314+0000 mon.b (mon.2) 658 : 2021-07-31T03:43:35.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:35 smithi045 conmon[39718]: audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:43:35.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:35 smithi045 conmon[39718]: audit 2021-07-31T03:43:35.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:35 smithi045 conmon[39718]: 2021-07-31T03:43:34.749158+0000 mon.b ( 2021-07-31T03:43:35.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:35 smithi045 conmon[39718]: mon.2) 659 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:43:36.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: cephadm 2021-07-31T03:43:34.750010+0000 mgr.x (mgr.24542) 4983 : cephadm 2021-07-31T03:43:36.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:43:36.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: cephadm 2021-07-31T03:43:34.792901+0000 mgr.x (mgr.24542) 4984 : cephadm [ERR] 2021-07-31T03:43:36.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:43:36.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:43:36.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: yield 2021-07-31T03:43:36.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:43:36.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:43:36.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:43:36.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:43:36.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:43:36.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:43:36.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:43:36.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:43:36.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:43:36.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: task.result() 2021-07-31T03:43:36.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:43:36.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: cephadm 2021-07-31T03:43:34.793400+0000 mgr.x (mgr.24542) 4985 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:43:36.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: cephadm 2021-07-31T03:43:34.804511+0000 mgr.x (mgr.24542) 4986 : cephadm [ERR] 2021-07-31T03:43:36.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:43:36.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:43:36.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: yield 2021-07-31T03:43:36.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:43:36.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:43:36.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:43:36.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:43:36.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:43:36.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:43:36.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:43:36.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:43:36.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:43:36.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: task.result() 2021-07-31T03:43:36.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:43:36.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: cephadm 2021-07-31T03:43:34.805538+0000 mgr.x (mgr.24542) 4987 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:43:36.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: cephadm 2021-07-31T03:43:34.830059+0000 mgr.x (mgr.24542) 4988 : cephadm [ERR] 2021-07-31T03:43:36.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:43:36.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:43:36.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: yield 2021-07-31T03:43:36.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:43:36.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:43:36.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:43:36.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:43:36.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:43:36.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:43:36.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:43:36.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:43:36.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:43:36.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: task.result() 2021-07-31T03:43:36.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:43:36.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: cephadm 2021-07-31T03:43:34.831316+0000 mgr.x (mgr.24542) 4989 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:43:36.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:43:36.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:43:36.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: yield 2021-07-31T03:43:36.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:43:36.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:43:36.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:43:36.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:43:36.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:43:36.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:43:36.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:43:36.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:43:36.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:43:36.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: task.result() 2021-07-31T03:43:36.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:43:36.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: 2021-07-31T03:43:36.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:43:36.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: 2021-07-31T03:43:36.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:43:36.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:43:36.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:43:36.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:43:36.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:43:36.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:43:36.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:43:36.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:43:36.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:43:36.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:43:36.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:43:36.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:43:36.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:43:36.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:43:36.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:43:36.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:43:36.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:43:36.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:43:36.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: raise 2021-07-31T03:43:36.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:43:36.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:43:36.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:43:36.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:43:36.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:43:36.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: cephadm 2021-07-31T03:43:34.841339+0000 mgr.x (mgr.24542) 4990 : cephadm [ERR] 2021-07-31T03:43:36.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:43:36.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:43:36.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: yield 2021-07-31T03:43:36.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:43:36.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:43:36.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:43:36.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:43:36.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:43:36.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:43:36.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:43:36.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:43:36.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:43:36.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: task.result() 2021-07-31T03:43:36.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:43:36.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: 2021-07-31T03:43:36.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: cephadm 2021-07-31T03:43:34.841725+0000 mgr.x (mgr.24542) 4991 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:43:36.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:43:36.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:43:36.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: yield 2021-07-31T03:43:36.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:43:36.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:43:36.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:43:36.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:43:36.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:43:36.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:43:36.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:43:36.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:43:36.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:43:36.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: task.result() 2021-07-31T03:43:36.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:43:36.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: 2021-07-31T03:43:36.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:43:36.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: 2021-07-31T03:43:36.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:43:36.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:43:36.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:43:36.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:43:36.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:43:36.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:43:36.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:43:36.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:43:36.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:43:36.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:43:36.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:43:36.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:43:36.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:43:36.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:43:36.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:43:36.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:43:36.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:43:36.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:43:36.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: raise 2021-07-31T03:43:36.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:43:36.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:43:36.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:43:36.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:43:36.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:36 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:43:36.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: cephadm 2021-07-31T03:43:34.750010+0000 mgr.x (mgr.24542) 4983 2021-07-31T03:43:36.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:43:36.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: cephadm 2021-07-31T03:43:34.792901+0000 mgr.x (mgr.24542) 4984 : cephadm [ERR] 2021-07-31T03:43:36.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:43:36.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:43:36.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: yield 2021-07-31T03:43:36.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:43:36.539 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:43:36.539 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:43:36.539 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:43:36.539 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:43:36.540 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:43:36.540 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:43:36.540 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:43:36.540 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:43:36.540 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: task.result() 2021-07-31T03:43:36.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:43:36.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: cephadm 2021-07-31T03:43:34.793400+0000 mgr.x (mgr.24542) 4985 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:43:36.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: cephadm 2021-07 2021-07-31T03:43:36.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: -31T03:43:34.804511+0000 mgr.x (mgr 2021-07-31T03:43:36.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: .24542) 4986 2021-07-31T03:43:36.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: : cephadm [ERR] 2021-07-31T03:43:36.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:43:36.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:43:36.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: yield 2021-07-31T03:43:36.543 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:43:36.543 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:43:36.543 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:43:36.543 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:43:36.544 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:43:36.544 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:43:36.544 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:43:36.544 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:43:36.544 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:43:36.545 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: task.result() 2021-07-31T03:43:36.545 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:43:36.545 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: cephadm 2021-07-31T03:43:36.545 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: 2021-07-31T03:43:34 2021-07-31T03:43:36.546 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: .805538+0000 mgr.x (mgr.24542 2021-07-31T03:43:36.546 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: ) 4987 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:43:36.546 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: 2021-07-31T03:43:36.546 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: cephadm 2021-07-31T03:43:36.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: cephadm 2021-07-31T03:43: 2021-07-31T03:43:36.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: 34.750010+0000 mgr.x (mgr.24542) 4983 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:43:36.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: cephadm 2021-07-31T03:43:36.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: 2021-07-31T03:43:34.792901+0000 mgr.x ( 2021-07-31T03:43:36.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: mgr.24542) 4984 : cephadm [ERR] 2021-07-31T03:43:36.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: 2021-07-31T03:43:36.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:43:36.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:43:36.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: yield 2021-07-31T03:43:36.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:43:36.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:43:36.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:43:36.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:43:36.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:43:36.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:43:36.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:43:36.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:43:36.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:43:36.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: task.result() 2021-07-31T03:43:36.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:43:36.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: cephadm 2021-07-31T03:43:34.793400+0000 mgr.x (mgr.24542) 4985 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:43:36.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: cephadm 2021- 2021-07-31T03:43:36.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: 07-31T03:43:34.804511 2021-07-31T03:43:36.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 2021-07-31T03:43:36.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: 4986 : cephadm [ERR] 2021-07-31T03:43:36.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:43:36.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:43:36.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: yield 2021-07-31T03:43:36.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:43:36.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:43:36.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:43:36.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:43:36.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:43:36.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:43:36.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:43:36.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:43:36.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:43:36.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: task.result() 2021-07-31T03:43:36.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:43:36.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: cephadm 2021-07-31T03:43:34.805538+0000 mgr.x (mgr.24542) 4987 : cephadm [INF] 2021-07-31T03:43:36.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:43:36.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: cephadm 2021- 2021-07-31T03:43:36.556 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: 2021-07-31T03:43: 2021-07-31T03:43:36.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: 34.830059+0000 mgr.x ( 2021-07-31T03:43:36.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: mgr.24542) 4988 : cephadm 2021-07-31T03:43:36.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: [ERR] 2021-07-31T03:43:36.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:43:36.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:43:36.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: yield 2021-07-31T03:43:36.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:43:36.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:43:36.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:43:36.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:43:36.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:43:36.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:43:36.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:43:36.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:43:36.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:43:36.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: task.result() 2021-07-31T03:43:36.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:43:36.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: cephadm 2021-07-31T03:43:34.831316+0000 mgr.x (mgr 2021-07-31T03:43:36.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: .24542) 4989 : cephadm [ERR] 2021-07-31T03:43:36.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:43:36.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:43:36.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:43:36.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: yield 2021-07-31T03:43:36.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:43:36.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:43:36.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:43:36.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:43:36.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:43:36.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:43:36.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:43:36.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:43:36.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:43:36.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: task.result() 2021-07-31T03:43:36.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:43:36.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: 2021-07-31T03:43:36.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:43:36.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: 2021-07-31T03:43:36.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:43:36.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:43:36.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:43:36.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:43:36.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:43:36.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:43:36.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: 07-31T03:43:34.830059 2021-07-31T03:43:36.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 4988 2021-07-31T03:43:36.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: : cephadm [ERR] 2021-07-31T03:43:36.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:43:36.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:43:36.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: yield 2021-07-31T03:43:36.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:43:36.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:43:36.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:43:36.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:43:36.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:43:36.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:43:36.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:43:36.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:43:36.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:43:36.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: task.result() 2021-07-31T03:43:36.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:43:36.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: cephadm 2021-07-31T03:43:34.831316+0000 mgr.x (mgr.24542 2021-07-31T03:43:36.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: ) 4989 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:43:36.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:43:36.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:43:36.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: yield 2021-07-31T03:43:36.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:43:36.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:43:36.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:43:36.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:43:36.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:43:36.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:43:36.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:43:36.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:43:36.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:43:36.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: task.result() 2021-07-31T03:43:36.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:43:36.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: 2021-07-31T03:43:36.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:43:36.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: 2021-07-31T03:43:36.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:43:36.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:43:36.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:43:36.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:43:36.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:43:36.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:43:36.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:43:36.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:43:36.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:43:36.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:43:36.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:43:36.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:43:36.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:43:36.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:43:36.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:43:36.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:43:36.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:43:36.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:43:36.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:43:36.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: raise 2021-07-31T03:43:36.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:43:36.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:43:36.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:43:36.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:43:36.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:43:36.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: cephadm 2021-07-31T03:43:34.841339+0000 mgr.x (mgr.24542) 4990 : cephadm [ERR] 2021-07-31T03:43:36.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:43:36.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:43:36.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: yield 2021-07-31T03:43:36.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:43:36.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:43:36.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:43:36.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:43:36.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:43:36.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:43:36.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:43:36.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:43:36.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:43:36.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: task.result() 2021-07-31T03:43:36.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:43:36.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: cephadm 2021-07-31T03:43:34.841725+0000 mgr.x (mgr.24542) 4991 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:43:36.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:43:36.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:43:36.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: yield 2021-07-31T03:43:36.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:43:36.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:43:36.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:43:36.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:43:36.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:43:36.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:43:36.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:43:36.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:43:36.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:43:36.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: task.result() 2021-07-31T03:43:36.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:43:36.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: 2021-07-31T03:43:36.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:43:36.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: 2021-07-31T03:43:36.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:43:36.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:43:36.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:43:36.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:43:36.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:43:36.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:43:36.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:43:36.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:43:36.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:43:36.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:43:36.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:43:36.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:43:36.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:43:36.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:43:36.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:43:36.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:43:36.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:43:36.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:43:36.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: raise 2021-07-31T03:43:36.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:43:36.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:43:36.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:43:36.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:43:36.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:43:36.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:43:36.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:43:36.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:43:36.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:43:36.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:43:36.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:43:36.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:43:36.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:43:36.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:43:36.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:43:36.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:43:36.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: raise 2021-07-31T03:43:36.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:43:36.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:43:36.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:43:36.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:43:36.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:43:36.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: cephadm 2021-07-31T03:43:34.841339+0000 mgr.x (mgr.24542) 4990 : cephadm [ERR] 2021-07-31T03:43:36.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:43:36.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:43:36.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: yield 2021-07-31T03:43:36.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:43:36.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:43:36.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:43:36.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:43:36.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:43:36.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:43:36.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:43:36.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:43:36.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:43:36.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: task.result() 2021-07-31T03:43:36.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:43:36.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: cephadm 2021-07-31T03:43:34.841725+0000 mgr.x (mgr.24542) 4991 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:43:36.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:43:36.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:43:36.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: yield 2021-07-31T03:43:36.603 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:43:36.603 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:43:36.603 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:43:36.603 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:43:36.604 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:43:36.604 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:43:36.604 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:43:36.604 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:43:36.605 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:43:36.605 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: task.result() 2021-07-31T03:43:36.605 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:43:36.605 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: 2021-07-31T03:43:36.606 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:43:36.606 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: 2021-07-31T03:43:36.606 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:43:36.606 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:43:36.607 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:43:36.607 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:43:36.607 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:43:36.607 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:43:36.607 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:43:36.608 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:43:36.608 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:43:36.608 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:43:36.608 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:43:36.609 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:43:36.609 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:43:36.609 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:43:36.609 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:43:36.610 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:43:36.610 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:43:36.610 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:43:36.610 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: raise 2021-07-31T03:43:36.611 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:43:36.611 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:43:36.611 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:43:36.611 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:43:36.612 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:36 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:43:37.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:37 smithi154 conmon[34553]: cluster 2021-07-31T03:43:35.456373+0000 mgr.x (mgr.24542) 4992 : cluster [DBG] pgmap v5499: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:37.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:37 smithi045 conmon[39718]: cluster 2021-07-31T03:43:35.456373+0000 2021-07-31T03:43:37.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:37 smithi045 conmon[39718]: mgr.x (mgr.24542) 4992 : cluster [DBG] pgmap v5499: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:37.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:37 smithi045 conmon[33655]: cluster 2021-07-31T03:43:35.456373+0000 mgr.x (mgr.24542 2021-07-31T03:43:37.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:37 smithi045 conmon[33655]: ) 4992 : cluster [DBG] pgmap v5499: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:39.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:39 smithi154 conmon[34553]: cluster 2021-07-31T03:43:37.456901+0000 mgr.x (mgr.24542) 4993 : cluster [DBG] pgmap v5500: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:39.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:39 smithi045 conmon[39718]: cluster 2021-07-31T03:43:37.456901+0000 mgr.x (mgr.24542) 4993 : cluster 2021-07-31T03:43:39.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:39 smithi045 conmon[39718]: [DBG] pgmap v5500: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:39.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:39 smithi045 conmon[33655]: cluster 2021-07-31T03:43:37 2021-07-31T03:43:39.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:39 smithi045 conmon[33655]: .456901+0000 mgr.x (mgr.24542) 4993 : cluster [DBG] pgmap v5500: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:41.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:41 smithi154 conmon[34553]: cluster 2021-07-31T03:43:39.457308+0000 mgr.x (mgr.24542) 4994 : cluster [DBG] pgmap v5501: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:41.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:41 smithi045 conmon[39718]: cluster 2021-07-31T03:43:39.457308+0000 mgr.x (mgr. 2021-07-31T03:43:41.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:41 smithi045 conmon[39718]: 24542) 4994 : cluster [DBG] pgmap v5501: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:41.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:41 smithi045 conmon[33655]: cluster 2021-07-31T03:43:39.457308+0000 mgr.x (mgr.24542) 4994 : cluster [DBG] pgmap v5501: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:43.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:43 smithi154 conmon[34553]: cluster 2021-07-31T03:43:41.457675+0000 mgr.x (mgr.24542) 4995 : cluster [DBG] pgmap v5502: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:43.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:43 smithi045 conmon[33655]: cluster 2021-07-31T03:43:41.457675+0000 mgr.x (mgr 2021-07-31T03:43:43.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:43 smithi045 conmon[33655]: .24542) 4995 : cluster [DBG] pgmap v5502: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:43.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:43 smithi045 conmon[39718]: cluster 2021-07-31T03:43:41.457675+0000 mgr.x (mgr 2021-07-31T03:43:43.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:43 smithi045 conmon[39718]: .24542) 4995 : cluster [DBG] pgmap v5502: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:45.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:45 smithi154 conmon[34553]: cluster 2021-07-31T03:43:43.458131+0000 mgr.x (mgr.24542) 4996 : cluster [DBG] pgmap v5503: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:45.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:45 smithi045 conmon[33655]: cluster 2021-07-31T03:43:43.458131+0000 mgr.x (mgr.24542) 4996 : 2021-07-31T03:43:45.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:45 smithi045 conmon[33655]: cluster [DBG] pgmap v5503: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:45.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:45 smithi045 conmon[39718]: cluster 2021-07-31T03:43:43.458131+0000 2021-07-31T03:43:45.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:45 smithi045 conmon[39718]: mgr.x (mgr.24542) 4996 : cluster [DBG] pgmap v5503: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:47.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:47 smithi154 conmon[34553]: cluster 2021-07-31T03:43:45.458609+0000 mgr.x (mgr.24542) 4997 : cluster [DBG] 2021-07-31T03:43:47.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:47 smithi154 conmon[34553]: pgmap v5504: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:47.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:47 smithi045 conmon[39718]: cluster 2021-07-31T03:43:45.458609+0000 mgr.x 2021-07-31T03:43:47.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:47 smithi045 conmon[39718]: (mgr.24542) 4997 : cluster [DBG] pgmap v5504: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:47.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:47 smithi045 conmon[33655]: cluster 2021-07-31T03:43:45.458609+0000 mgr.x (mgr.24542) 4997 : cluster [DBG] pgmap v5504: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:49.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:49 smithi154 conmon[34553]: cluster 2021-07-31T03:43:47.459065+0000 mgr.x (mgr.24542) 4998 : cluster [DBG] pgmap v5505: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:49.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:49 smithi045 conmon[33655]: cluster 2021-07-31T03:43:47 2021-07-31T03:43:49.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:49 smithi045 conmon[33655]: .459065+0000 mgr.x (mgr.24542) 4998 : cluster [DBG] pgmap v5505: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:49.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:49 smithi045 conmon[39718]: cluster 2021-07-31T03:43:47.459065+0000 mgr.x (mgr 2021-07-31T03:43:49.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:49 smithi045 conmon[39718]: .24542) 4998 : cluster [DBG] pgmap v5505: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:51.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:51 smithi154 conmon[34553]: cluster 2021-07-31T03:43:49.459703+0000 mgr.x (mgr.24542) 4999 : cluster [DBG] 2021-07-31T03:43:51.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:51 smithi154 conmon[34553]: pgmap v5506: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:51.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:51 smithi045 conmon[33655]: cluster 2021-07-31T03:43:49.459703+0000 mgr.x (mgr.24542) 4999 2021-07-31T03:43:51.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:51 smithi045 conmon[33655]: : cluster [DBG] pgmap v5506: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:51.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:51 smithi045 conmon[39718]: cluster 2021-07-31T03:43:49.459703+0000 mgr.x 2021-07-31T03:43:51.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:51 smithi045 conmon[39718]: (mgr.24542) 4999 : cluster [DBG] pgmap v5506: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:53.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:53 smithi154 conmon[34553]: cluster 2021-07-31T03:43:51.460232+0000 mgr.x (mgr.24542) 5000 : cluster [DBG] pgmap v5507: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:53.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:53 smithi045 conmon[39718]: cluster 2021-07-31T03:43:51.460232+0000 2021-07-31T03:43:53.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:53 smithi045 conmon[39718]: mgr.x (mgr.24542) 5000 : cluster [DBG] pgmap v5507: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:53.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:53 smithi045 conmon[33655]: cluster 2021-07-31T03:43:51.460232+0000 2021-07-31T03:43:53.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:53 smithi045 conmon[33655]: mgr.x (mgr.24542) 5000 : cluster [DBG] pgmap v5507: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:54.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:54 smithi154 conmon[34553]: cluster 2021-07-31T03:43:53.460728+0000 mgr.x (mgr.24542) 5001 : cluster [DBG] pgmap v5508: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:54.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:54 smithi045 conmon[33655]: cluster 2021-07-31T03:43 2021-07-31T03:43:54.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:54 smithi045 conmon[33655]: :53.460728+0000 mgr.x (mgr.24542) 5001 : cluster [DBG] pgmap v5508: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:54.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:54 smithi045 conmon[39718]: cluster 2021-07-31T03:43:53. 2021-07-31T03:43:54.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:54 smithi045 conmon[39718]: 460728+0000 mgr.x (mgr.24542) 5001 : cluster [DBG] pgmap v5508: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:56.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:56 smithi154 conmon[34553]: cluster 2021-07-31T03:43:55.461350+0000 mgr.x (mgr.24542) 5002 : cluster [DBG] pgmap v5509: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:56.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:56 smithi045 conmon[33655]: cluster 2021-07-31T03:43:55.461350 2021-07-31T03:43:56.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:56 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 5002 : cluster [DBG] pgmap v5509: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:56.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:56 smithi045 conmon[39718]: cluster 2021-07-31T03:43:55.461350+0000 mgr.x (mgr 2021-07-31T03:43:56.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:56 smithi045 conmon[39718]: .24542) 5002 : cluster [DBG] pgmap v5509: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:57.319 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:43:57.597 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:43:57.597 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 118m ago 119m - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:43:57.597 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 116m ago 119m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:43:57.597 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 116m ago 2h 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:43:57.598 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 118m ago 2h - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:43:57.598 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 118m ago 2h - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:43:57.598 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 116m ago 2h 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:43:57.598 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 118m ago 2h - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:43:57.598 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 118m ago 119m - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:43:57.599 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 116m ago 119m 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:43:57.599 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 118m ago 2h - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:43:57.599 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 118m ago 2h - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:43:57.599 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 118m ago 2h - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:43:57.599 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 118m ago 2h - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:43:57.600 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 116m ago 2h 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:43:57.600 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 116m ago 2h 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:43:57.600 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 116m ago 2h 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:43:57.600 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 116m ago 2h 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:43:57.600 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 116m ago 119m 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:43:57.931 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:43:57.931 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:43:57.932 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:43:57.932 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:43:57.932 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:43:57.932 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:43:57.933 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:43:57.933 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:43:57.933 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:43:57.934 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:43:57.934 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:43:57.934 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:43:57.935 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:43:57.935 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:43:57.935 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:43:57.936 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:43:57.936 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:43:58.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:58 smithi154 conmon[34553]: audit 2021-07-31T03:43:57.308013+0000 mgr.x (mgr.24542) 5003 : audit [DBG] from='client.20118 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:43:58.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:58 smithi154 conmon[34553]: cluster 2021-07-31T03:43:58.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:58 smithi154 conmon[34553]: 2021-07-31T03:43:57.461721+0000 mgr.x (mgr.24542) 5004 : cluster [DBG] pgmap v5510: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:58.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:58 smithi154 conmon[34553]: audit 2021-07-31T03:43:57.589666+0000 mgr.x (mgr.24542) 5005 : audit [DBG] from='client.30064 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:43:58.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:43:58 smithi154 conmon[34553]: audit 2021-07-31T03:43:57.932062+0000 mon.a (mon.0) 3354 : audit [DBG] from='client.? 172.21.15.45:0/1889989618' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:43:58.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:58 smithi045 conmon[39718]: audit 2021-07-31T03:43:57.308013+0000 mgr.x (mgr.24542) 2021-07-31T03:43:58.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:58 smithi045 conmon[39718]: 5003 : audit [DBG] from='client.20118 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:43:58.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:58 smithi045 conmon[39718]: cluster 2021-07-31T03:43:57.461721+0000 2021-07-31T03:43:58.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:58 smithi045 conmon[39718]: mgr.x (mgr.24542) 5004 : cluster [DBG] pgmap v5510: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:58.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:58 smithi045 conmon[39718]: audit 2021- 2021-07-31T03:43:58.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:58 smithi045 conmon[39718]: 07-31T03:43:57.589666+0000 mgr.x (mgr.24542) 2021-07-31T03:43:58.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:58 smithi045 conmon[39718]: 5005 : audit [DBG] from='client.30064 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:43:58.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:58 smithi045 conmon[39718]: audit 2021-07-31T03:43:57.932062+0000 mon.a (mon 2021-07-31T03:43:58.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:43:58 smithi045 conmon[39718]: .0) 3354 : audit [DBG] from='client.? 172.21.15.45:0/1889989618' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:43:58.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:58 smithi045 conmon[33655]: audit 2021-07-31T03:43:57.308013+0000 mgr.x (mgr.24542) 5003 : audit 2021-07-31T03:43:58.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:58 smithi045 conmon[33655]: [DBG] from='client.20118 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:43:58.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:58 smithi045 conmon[33655]: cluster 2021-07-31T03:43:57.461721+0000 mgr.x (mgr.24542) 5004 : cluster [DBG] pgmap v5510: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:43:58.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:58 smithi045 conmon[33655]: audit 2021-07-31T03:43:57.589666 2021-07-31T03:43:58.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:58 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 5005 : audit [DBG] from='client.30064 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:43:58.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:58 smithi045 conmon[33655]: audit 2021-07-31T03:43:57. 2021-07-31T03:43:58.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:43:58 smithi045 conmon[33655]: 932062+0000 mon.a (mon.0) 3354 : audit [DBG] from='client.? 172.21.15.45:0/1889989618' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:44:00.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:00 smithi154 conmon[34553]: cluster 2021-07-31T03:43:59.462225+0000 mgr.x (mgr.24542) 5006 : cluster [DBG] pgmap v5511: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:00.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:00 smithi045 conmon[39718]: cluster 2021-07-31T03:43:59.462225+0000 mgr.x (mgr 2021-07-31T03:44:00.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:00 smithi045 conmon[39718]: .24542) 5006 : cluster [DBG] pgmap v5511: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:00.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:00 smithi045 conmon[33655]: cluster 2021-07-31T03:43:59.462225 2021-07-31T03:44:00.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:00 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 5006 : cluster [DBG] pgmap v5511: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:02.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:02 smithi154 conmon[34553]: cluster 2021-07-31T03:44:01.462713+0000 mgr.x (mgr.24542) 5007 : cluster [DBG] pgmap v5512: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:02.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:02 smithi045 conmon[39718]: cluster 2021-07-31T03:44:01.462713+0000 mgr.x (mgr 2021-07-31T03:44:02.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:02 smithi045 conmon[39718]: .24542) 5007 : cluster [DBG] pgmap v5512: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:02.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:02 smithi045 conmon[33655]: cluster 2021-07-31T03:44:01.462713+0000 mgr.x (mgr.24542) 5007 : 2021-07-31T03:44:02.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:02 smithi045 conmon[33655]: cluster [DBG] pgmap v5512: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:04.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:04 smithi154 conmon[34553]: cluster 2021-07-31T03:44:03.463070+0000 mgr.x (mgr.24542) 5008 : cluster [DBG] pgmap v5513: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:04.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:04 smithi045 conmon[39718]: cluster 2021-07-31T03:44:03.463070+0000 mgr.x (mgr 2021-07-31T03:44:04.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:04 smithi045 conmon[39718]: .24542) 5008 : cluster [DBG] pgmap v5513: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:04.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:04 smithi045 conmon[33655]: cluster 2021-07-31T03: 2021-07-31T03:44:04.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:04 smithi045 conmon[33655]: 44:03.463070+0000 mgr.x (mgr.24542) 5008 : cluster [DBG] pgmap v5513: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:06.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:06 smithi154 conmon[34553]: cluster 2021-07-31T03:44:05.463617+0000 mgr.x (mgr.24542) 5009 : cluster [DBG] pgmap v5514: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:06.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:06 smithi045 conmon[33655]: cluster 2021-07-31T03:44:05.463617+0000 mgr.x (mgr.24542) 5009 : cluster 2021-07-31T03:44:06.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:06 smithi045 conmon[33655]: [DBG] pgmap v5514: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:06.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:06 smithi045 conmon[39718]: cluster 2021-07-31T03:44:05.463617+0000 mgr.x (mgr.24542) 5009 : cluster 2021-07-31T03:44:06.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:06 smithi045 conmon[39718]: [DBG] pgmap v5514: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:08.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:08 smithi045 conmon[33655]: cluster 2021-07-31T03:44:07.464174+0000 mgr.x (mgr.24542) 5010 : cluster 2021-07-31T03:44:08.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:08 smithi045 conmon[33655]: [DBG] pgmap v5515: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:08.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:08 smithi045 conmon[39718]: cluster 2021-07-31T03:44:07.464174+0000 mgr.x (mgr.24542) 2021-07-31T03:44:08.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:08 smithi045 conmon[39718]: 5010 : cluster [DBG] pgmap v5515: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:09.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:08 smithi154 conmon[34553]: cluster 2021-07-31T03:44:07.464174+0000 mgr.x (mgr.24542) 5010 : cluster [DBG] pgmap v5515: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:10.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:10 smithi045 conmon[33655]: cluster 2021- 2021-07-31T03:44:10.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:10 smithi045 conmon[33655]: 07-31T03:44:09.464639+0000 mgr.x (mgr.24542) 5011 : cluster [DBG] pgmap v5516: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:10.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:10 smithi045 conmon[39718]: cluster 2021-07-31T03:44:09.464639+0000 mgr.x ( 2021-07-31T03:44:10.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:10 smithi045 conmon[39718]: mgr.24542) 5011 : cluster [DBG] pgmap v5516: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:11.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:10 smithi154 conmon[34553]: cluster 2021-07-31T03:44:09.464639+0000 mgr.x (mgr.24542) 5011 : cluster [DBG] pgmap v5516: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:12.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:12 smithi045 conmon[39718]: cluster 2021-07-31T03:44:11.465053+0000 mgr.x (mgr. 2021-07-31T03:44:12.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:12 smithi045 conmon[39718]: 24542) 5012 : cluster [DBG] pgmap v5517: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:12.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:12 smithi045 conmon[33655]: cluster 2021-07-31T03:44:11.465053 2021-07-31T03:44:12.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:12 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 5012 : cluster [DBG] pgmap v5517: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:13.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:12 smithi154 conmon[34553]: cluster 2021-07-31T03:44:11.465053+0000 mgr.x (mgr.24542) 5012 : cluster [DBG] pgmap v5517: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:14.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:14 smithi045 conmon[39718]: cluster 2021-07-31T03:44:13.465512+0000 mgr.x (mgr 2021-07-31T03:44:14.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:14 smithi045 conmon[39718]: .24542) 5013 : cluster [DBG] pgmap v5518: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:14.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:14 smithi045 conmon[33655]: cluster 2021-07-31T03:44:13.465512+0000 mgr.x (mgr.24542) 5013 : 2021-07-31T03:44:14.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:14 smithi045 conmon[33655]: cluster [DBG] pgmap v5518: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:15.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:14 smithi154 conmon[34553]: cluster 2021-07-31T03:44:13.465512+0000 mgr.x (mgr.24542) 5013 : cluster [DBG] pgmap v5518: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:16.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:16 smithi045 conmon[39718]: cluster 2021-07-31T03:44:15.466097+0000 mgr.x (mgr.24542 2021-07-31T03:44:16.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:16 smithi045 conmon[39718]: ) 5014 : cluster [DBG] pgmap v5519: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:16.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:16 smithi045 conmon[33655]: cluster 2021-07-31T03 2021-07-31T03:44:16.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:16 smithi045 conmon[33655]: :44:15.466097+0000 mgr.x (mgr.24542) 5014 : cluster [DBG] pgmap v5519: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:17.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:16 smithi154 conmon[34553]: cluster 2021-07-31T03:44:15.466097+0000 mgr.x (mgr.24542) 5014 : cluster [DBG] pgmap v5519: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:18.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:18 smithi045 conmon[33655]: cluster 2021-07-31T03:44:17.466574+0000 mgr.x (mgr.24542) 5015 : cluster 2021-07-31T03:44:18.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:18 smithi045 conmon[33655]: [DBG] pgmap v5520: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:18.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:18 smithi045 conmon[39718]: cluster 2021-07-31T03:44:17.466574+0000 2021-07-31T03:44:18.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:18 smithi045 conmon[39718]: mgr.x (mgr.24542) 5015 : cluster [DBG] pgmap v5520: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:19.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:18 smithi154 conmon[34553]: cluster 2021-07-31T03:44:17.466574+0000 mgr.x (mgr.24542) 5015 : cluster [DBG] pgmap v5520: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:20.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:20 smithi045 conmon[33655]: cluster 2021-07-31T03:44:19.467142+0000 mgr.x (mgr.24542) 5016 : cluster [DBG] 2021-07-31T03:44:20.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:20 smithi045 conmon[33655]: pgmap v5521: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:20.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:20 smithi045 conmon[39718]: cluster 2021-07-31T03:44:19.467142+0000 2021-07-31T03:44:20.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:20 smithi045 conmon[39718]: mgr.x (mgr.24542) 5016 : cluster [DBG] pgmap v5521: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:21.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:20 smithi154 conmon[34553]: cluster 2021-07-31T03:44:19.467142+0000 mgr.x (mgr.24542) 5016 : cluster [DBG] pgmap v5521: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:23.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:22 smithi154 conmon[34553]: cluster 2021-07-31T03:44:21.467625+0000 mgr.x (mgr.24542) 5017 : cluster [DBG] 2021-07-31T03:44:23.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:22 smithi154 conmon[34553]: pgmap v5522: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:23.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:22 smithi045 conmon[33655]: cluster 2021-07-31T03:44:21.467625+0000 mgr.x (mgr.24542) 5017 : cluster 2021-07-31T03:44:23.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:22 smithi045 conmon[33655]: [DBG] pgmap v5522: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:23.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:22 smithi045 conmon[39718]: cluster 2021-07-31T03:44:21.467625+0000 mgr.x 2021-07-31T03:44:23.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:22 smithi045 conmon[39718]: (mgr.24542) 5017 : cluster [DBG] pgmap v5522: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:25.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:24 smithi154 conmon[34553]: cluster 2021-07-31T03:44:23.468146+0000 mgr.x (mgr.24542) 5018 : cluster [DBG] pgmap v5523: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:25.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:24 smithi045 conmon[39718]: cluster 2021-07-31T03:44:23.468146+0000 mgr.x 2021-07-31T03:44:25.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:24 smithi045 conmon[39718]: (mgr.24542) 5018 : cluster [DBG] pgmap v5523: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:25.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:24 smithi045 conmon[33655]: cluster 2021-07-31T03:44:23.468146+0000 2021-07-31T03:44:25.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:24 smithi045 conmon[33655]: mgr.x (mgr.24542) 5018 : cluster [DBG] pgmap v5523: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:27.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:26 smithi154 conmon[34553]: cluster 2021-07-31T03:44:25.468617+0000 mgr.x (mgr.24542) 5019 : 2021-07-31T03:44:27.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:26 smithi154 conmon[34553]: cluster [DBG] pgmap v5524: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:27.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:26 smithi045 conmon[33655]: cluster 2021-07-31T03:44:25.468617+0000 mgr.x (mgr. 2021-07-31T03:44:27.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:26 smithi045 conmon[33655]: 24542) 5019 : cluster [DBG] pgmap v5524: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:27.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:26 smithi045 conmon[39718]: cluster 2021-07-31T03:44:25.468617+0000 mgr.x (mgr.24542) 5019 : cluster [DBG] pgmap v5524: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:28.245 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:44:28.528 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:44:28.528 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 119m ago 2h - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:44:28.528 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 117m ago 119m 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:44:28.528 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 117m ago 2h 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:44:28.529 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 119m ago 2h - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:44:28.529 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 119m ago 2h - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:44:28.529 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 117m ago 2h 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:44:28.529 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 119m ago 2h - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:44:28.529 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 119m ago 2h - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:44:28.530 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 117m ago 2h 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:44:28.530 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 119m ago 2h - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:44:28.530 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 119m ago 2h - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:44:28.530 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 119m ago 2h - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:44:28.531 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 119m ago 2h - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:44:28.531 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 117m ago 2h 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:44:28.531 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 117m ago 2h 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:44:28.531 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 117m ago 2h 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:44:28.531 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 117m ago 2h 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:44:28.532 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 117m ago 2h 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:44:28.866 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:44:28.867 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:44:28.867 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:44:28.867 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:44:28.867 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:44:28.868 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:44:28.868 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:44:28.868 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:44:28.868 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:44:28.869 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:44:28.869 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:44:28.869 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:44:28.869 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:44:28.869 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:44:28.870 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:44:28.870 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:44:28.870 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:44:29.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:28 smithi154 conmon[34553]: cluster 2021-07-31T03:44:27.469173+0000 mgr.x (mgr.24542) 5020 : cluster [DBG] pgmap v5525: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:29.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:28 smithi154 conmon[34553]: audit 2021-07-31T03:44:28.400452+0000 mon.b (mon.2) 660 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:44:29.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:28 smithi154 conmon[34553]: audit 2021-07-31T03:44:28.400718+0000 mon.b (mon.2) 661 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:44:29.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:28 smithi154 conmon[34553]: audit 2021-07-31T03:44:29.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:28 smithi154 conmon[34553]: 2021-07-31T03:44:28.400928+0000 mon.a (mon.0) 3355 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:44:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:28 smithi154 conmon[34553]: audit 2021-07-31T03:44:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:28 smithi154 conmon[34553]: 2021-07-31T03:44:28.401251+0000 mon.a (mon.0) 3356 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:44:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:28 smithi154 conmon[34553]: 2021-07-31T03:44:29.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:28 smithi045 conmon[39718]: cluster 2021-07-31T03:44:27.469173+0000 2021-07-31T03:44:29.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:28 smithi045 conmon[39718]: mgr.x (mgr.24542) 5020 : cluster [DBG] pgmap v5525: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:29.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:28 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T03:44:29.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:28 smithi045 conmon[39718]: 31T03:44:28.400452+0000 mon.b (mon.2) 660 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:44:29.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:28 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T03:44:29.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:28 smithi045 conmon[39718]: T03:44:28.400718+0000 mon.b (mon.2) 661 2021-07-31T03:44:29.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:28 smithi045 conmon[39718]: : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:44:29.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:28 smithi045 conmon[39718]: audit 2021- 2021-07-31T03:44:29.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:28 smithi045 conmon[39718]: 07-31T03:44:28.400928+0000 2021-07-31T03:44:29.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:28 smithi045 conmon[39718]: mon.a (mon.0) 3355 : audit [INF] 2021-07-31T03:44:29.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:28 smithi045 conmon[39718]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:44:29.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:28 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T03:44:29.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:28 smithi045 conmon[39718]: 03:44:28.401251+0000 mon.a (mon 2021-07-31T03:44:29.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:28 smithi045 conmon[39718]: .0) 3356 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:44:29.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:28 smithi045 conmon[33655]: cluster 2021-07-31T03:44:27.469173+0000 mgr.x (mgr.24542) 5020 : 2021-07-31T03:44:29.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:28 smithi045 conmon[33655]: cluster [DBG] pgmap v5525: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:29.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:28 smithi045 conmon[33655]: audit 2021-07-31T03:44:28.400452+0000 mon.b (mon.2) 2021-07-31T03:44:29.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:28 smithi045 conmon[33655]: 660 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:44:29.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:28 smithi045 conmon[33655]: audit 2021-07-31T03:44:29.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:28 smithi045 conmon[33655]: 2021-07-31T03 2021-07-31T03:44:29.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:28 smithi045 conmon[33655]: :44:28.400718+0000 mon.b 2021-07-31T03:44:29.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:28 smithi045 conmon[33655]: (mon.2) 661 : audit 2021-07-31T03:44:29.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:28 smithi045 conmon[33655]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:44:29.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:28 smithi045 conmon[33655]: audit 2021- 2021-07-31T03:44:29.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:28 smithi045 conmon[33655]: 07-31T03:44:28. 2021-07-31T03:44:29.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:28 smithi045 conmon[33655]: 400928+0000 mon.a (mon.0 2021-07-31T03:44:29.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:28 smithi045 conmon[33655]: ) 3355 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:44:29.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:28 smithi045 conmon[33655]: 2021-07-31T03:44:29.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:28 smithi045 conmon[33655]: audit 2021-07-31T03:44:29.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:28 smithi045 conmon[33655]: 2021-07-31T03: 2021-07-31T03:44:29.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:28 smithi045 conmon[33655]: 44:28.401251+0000 mon.a ( 2021-07-31T03:44:29.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:28 smithi045 conmon[33655]: mon.0) 3356 : audit 2021-07-31T03:44:29.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:28 smithi045 conmon[33655]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:44:30.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:29 smithi154 conmon[34553]: audit 2021-07-31T03:44:28.234087+0000 mgr.x (mgr.24542) 5021 : audit [DBG] from='client.20136 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:44:30.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:29 smithi154 conmon[34553]: audit 2021-07-31T03:44:28.520790+0000 mgr.x (mgr.24542) 5022 : audit [DBG] from='client.30082 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:44:30.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:29 smithi154 conmon[34553]: audit 2021-07-31T03:44:28.867571+0000 mon.a (mon.0) 3357 : audit [DBG] from='client.? 172.21.15.45:0/3261909177' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:44:30.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:29 smithi154 conmon[34553]: 2021-07-31T03:44:30.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:29 smithi045 conmon[39718]: audit 2021-07-31T03:44:28.234087+0000 mgr.x (mgr.24542 2021-07-31T03:44:30.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:29 smithi045 conmon[39718]: ) 5021 : audit [DBG] from='client.20136 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:44:30.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:29 smithi045 conmon[39718]: audit 2021-07-31T03:44:28.520790+0000 mgr.x (mgr.24542) 5022 : audit [DBG] from='client.30082 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:44:30.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:29 smithi045 conmon[39718]: audit 2021-07-31T03:44:28.867571+0000 mon.a (mon.0) 3357 : audit [DBG] from='client.? 172.21.15.45:0/3261909177' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:44:30.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:29 smithi045 conmon[33655]: audit 2021-07-31T03:44:28.234087+0000 2021-07-31T03:44:30.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:29 smithi045 conmon[33655]: mgr.x (mgr.24542) 5021 : audit [DBG] from='client.20136 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:44:30.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:29 smithi045 conmon[33655]: audit 2021-07-31T03:44:28.520790+0000 mgr.x (mgr.24542) 5022 : audit [DBG] from='client.30082 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:44:30.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:29 smithi045 conmon[33655]: audit 2021-07-31T03:44:28.867571 2021-07-31T03:44:30.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:29 smithi045 conmon[33655]: +0000 mon.a (mon.0) 3357 : audit [DBG] from='client.? 172.21.15.45:0/3261909177' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:44:31.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:30 smithi154 conmon[34553]: cluster 2021-07-31T03:44:29.469638+0000 mgr.x (mgr.24542) 5023 : 2021-07-31T03:44:31.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:30 smithi154 conmon[34553]: cluster [DBG] pgmap v5526: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:31.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:30 smithi045 conmon[39718]: cluster 2021-07-31T03:44:29.469638+0000 mgr.x (mgr.24542) 5023 2021-07-31T03:44:31.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:30 smithi045 conmon[39718]: : cluster [DBG] pgmap v5526: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:31.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:30 smithi045 conmon[33655]: cluster 2021-07-31T03:44:29.469638+0000 mgr.x (mgr.24542) 2021-07-31T03:44:31.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:30 smithi045 conmon[33655]: 5023 : cluster [DBG] pgmap v5526: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:33.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:32 smithi154 conmon[34553]: cluster 2021-07-31T03:44:31.470112+0000 mgr.x (mgr.24542) 5024 2021-07-31T03:44:33.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:32 smithi154 conmon[34553]: : cluster [DBG] pgmap v5527: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:33.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:32 smithi045 conmon[39718]: cluster 2021-07-31T03:44:31.470112+0000 mgr.x (mgr. 2021-07-31T03:44:33.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:32 smithi045 conmon[39718]: 24542) 5024 : cluster [DBG] pgmap v5527: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:33.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:32 smithi045 conmon[33655]: cluster 2021-07-31T03:44:31.470112+0000 mgr.x (mgr.24542) 5024 : cluster [DBG] pgmap v5527: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:34.868 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[34553]: cluster 2021-07-31T03:44:33.470577+0000 mgr.x (mgr.24542) 5025 : cluster [DBG] pgmap v5528: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:34.869 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:34 smithi045 conmon[33655]: cluster 2021-07-31T03:44:33.470577+0000 mgr.x (mgr.24542 2021-07-31T03:44:34.869 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:34 smithi045 conmon[33655]: ) 5025 : cluster [DBG] pgmap v5528: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:34.870 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:34 smithi045 conmon[39718]: cluster 2021-07-31T03:44:33.470577+0000 mgr.x (mgr.24542) 5025 : cluster [DBG] pgmap v5528: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:35.252 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: debug 2021-07-31T03:44:34.887+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:44:35.253 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:44:35.253 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:44:35.253 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: yield 2021-07-31T03:44:35.253 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:44:35.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:44:35.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:44:35.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:44:35.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:44:35.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:44:35.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:44:35.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:44:35.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:44:35.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: task.result() 2021-07-31T03:44:35.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:44:35.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: debug 2021-07-31T03:44:34.899+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:44:35.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:44:35.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:44:35.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: yield 2021-07-31T03:44:35.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:44:35.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:44:35.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:44:35.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:44:35.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:44:35.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:44:35.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:44:35.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:44:35.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:44:35.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: task.result() 2021-07-31T03:44:35.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:44:35.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: debug 2021-07-31T03:44:34.924+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:44:35.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:44:35.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:44:35.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: yield 2021-07-31T03:44:35.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:44:35.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:44:35.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:44:35.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:44:35.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:44:35.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:44:35.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:44:35.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:44:35.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:44:35.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: task.result() 2021-07-31T03:44:35.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:44:35.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: debug 2021-07-31T03:44:34.925+0000 7fcb9195d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:44:35.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:44:35.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:44:35.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: yield 2021-07-31T03:44:35.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:44:35.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:44:35.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:44:35.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:44:35.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:44:35.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:44:35.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:44:35.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:44:35.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:44:35.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: task.result() 2021-07-31T03:44:35.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:44:35.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: 2021-07-31T03:44:35.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:44:35.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: 2021-07-31T03:44:35.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:44:35.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:44:35.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:44:35.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:44:35.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:44:35.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:44:35.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:44:35.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:44:35.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:44:35.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:44:35.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:44:35.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:44:35.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:44:35.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:44:35.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:44:35.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:44:35.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:44:35.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:44:35.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: raise 2021-07-31T03:44:35.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:44:35.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:44:35.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:44:35.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:44:35.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:44:35.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: debug 2021-07-31T03:44:34.933+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:44:35.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:44:35.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:44:35.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: yield 2021-07-31T03:44:35.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:44:35.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:44:35.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:44:35.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:44:35.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:44:35.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:44:35.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:44:35.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:44:35.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:44:35.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: task.result() 2021-07-31T03:44:35.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:44:35.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: debug 2021-07-31T03:44:34.933+0000 7fcb9215e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:44:35.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:44:35.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:44:35.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: yield 2021-07-31T03:44:35.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:44:35.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:44:35.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:44:35.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:44:35.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:44:35.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:44:35.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:44:35.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:44:35.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:44:35.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: task.result() 2021-07-31T03:44:35.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:44:35.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: 2021-07-31T03:44:35.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:44:35.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: 2021-07-31T03:44:35.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:44:35.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:44:35.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:44:35.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:44:35.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:44:35.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:44:35.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:44:35.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:44:35.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:44:35.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:44:35.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:44:35.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:44:35.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:44:35.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:44:35.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:44:35.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:44:35.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:44:35.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:44:35.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: raise 2021-07-31T03:44:35.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:44:35.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:44:35.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:44:35.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:44:35.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:44:34 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:44:36.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: audit 2021-07-31T03:44:34.842614+0000 mon.b (mon.2) 662 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:44:36.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: audit 2021-07-31T03:44:34.844039 2021-07-31T03:44:36.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: +0000 mon.b (mon.2) 663 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:44:36.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: audit 2021-07-31T03:44:34.844902+0000 mon.b (mon.2) 664 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:44:36.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:44:34.845650+0000 mgr.x (mgr.24542 2021-07-31T03:44:36.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: ) 5026 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:44:36.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:44:34.888547+0000 mgr.x (mgr.24542) 5027 : cephadm [ERR] 2021-07-31T03:44:36.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:44:36.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:44:36.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: yield 2021-07-31T03:44:36.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:44:36.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:44:36.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:44:36.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:44:36.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:44:36.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:44:36.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:44:36.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:44:36.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:44:36.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: task.result() 2021-07-31T03:44:36.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:44:36.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: 2021-07-31T03:44:36.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:44:34.889000+0000 mgr.x (mgr.24542) 5028 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:44:36.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:44:34.900098+0000 mgr.x (mgr.24542) 5029 : cephadm [ERR] 2021-07-31T03:44:36.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:44:36.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:44:36.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: yield 2021-07-31T03:44:36.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:44:36.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:44:36.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:44:36.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:44:36.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:44:36.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:44:36.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:44:36.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:44:36.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:44:36.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: task.result() 2021-07-31T03:44:36.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:44:36.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:44:34.900604+0000 mgr.x (mgr.24542) 5030 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:44:36.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:44:34.925620+0000 mgr.x (mgr.24542) 5031 : cephadm [ERR] 2021-07-31T03:44:36.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:44:36.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:44:36.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: yield 2021-07-31T03:44:36.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:44:36.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:44:36.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:44:36.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:44:36.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:44:36.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:44:36.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:44:36.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:44:36.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:44:36.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: task.result() 2021-07-31T03:44:36.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:44:36.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:44:34.926051+0000 mgr.x ( 2021-07-31T03:44:36.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: mgr.24542) 5032 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:44:36.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:44:36.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:44:36.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: yield 2021-07-31T03:44:36.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:44:36.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:44:36.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:44:36.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:44:36.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:44:36.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:44:36.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:44:36.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:44:36.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:44:36.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: task.result() 2021-07-31T03:44:36.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:44:36.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: 2021-07-31T03:44:36.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:44:36.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: 2021-07-31T03:44:36.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:44:36.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:44:36.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:44:36.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:44:36.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:44:36.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:44:36.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:44:36.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:44:36.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:44:36.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:44:36.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:44:36.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:44:36.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:44:36.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:44:36.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:44:36.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:44:36.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:44:36.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:44:36.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: raise 2021-07-31T03:44:36.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:44:36.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:44:36.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:44:36.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:44:36.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:44:36.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:44:34.934453+0000 mgr.x (mgr.24542) 5033 : cephadm [ERR] 2021-07-31T03:44:36.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:44:36.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:44:36.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: yield 2021-07-31T03:44:36.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:44:36.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:44:36.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:44:36.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:44:36.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:44:36.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:44:36.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:44:36.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:44:36.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:44:36.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: task.result() 2021-07-31T03:44:36.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:44:36.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:44:34.934835+0000 mgr.x (mgr.24542) 5034 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:44:36.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:44:36.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:44:36.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: yield 2021-07-31T03:44:36.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:44:36.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:44:36.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:44:36.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:44:36.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:44:36.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:44:36.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:44:36.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:44:36.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:44:36.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: task.result() 2021-07-31T03:44:36.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:44:36.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: 2021-07-31T03:44:36.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:44:36.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: 2021-07-31T03:44:36.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:44:36.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:44:36.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:44:36.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:44:36.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:44:36.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:44:36.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:44:36.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:44:36.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:44:36.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:44:36.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:44:36.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:44:36.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:44:36.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:44:36.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:44:36.032 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:44:36.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:44:36.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:44:36.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: raise 2021-07-31T03:44:36.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:44:36.033 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:44:36.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:44:36.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:44:36.034 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:35 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:44:36.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: audit 2021-07-31T03:44:34.842614+0000 2021-07-31T03:44:36.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: mon.b (mon.2) 662 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:44:36.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: audit 2021-07-31T03 2021-07-31T03:44:36.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: :44:34.844039+0000 mon.b (mon.2) 663 : audit 2021-07-31T03:44:36.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:44:36.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: audit 2021-07-31T03:44:36.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: 2021-07-31T03:44 2021-07-31T03:44:36.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: :34.844902+0000 mon.b (mon 2021-07-31T03:44:36.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: .2) 664 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:44:36.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: cephadm 2021-07 2021-07-31T03:44:36.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: -31T03:44:34.845650+0000 2021-07-31T03:44:36.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: mgr.x (mgr.24542) 5026 : cephadm [WRN] 2021-07-31T03:44:36.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:44:36.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:44:36.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: 2021-07-31T03: 2021-07-31T03:44:36.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: 44:34.888547+0000 mgr.x (mgr. 2021-07-31T03:44:36.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: 24542) 5027 : cephadm [ERR] 2021-07-31T03:44:36.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:44:36.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:44:36.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: yield 2021-07-31T03:44:36.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:44:36.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:44:36.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:44:36.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:44:36.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:44:36.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:44:36.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:44:36.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:44:36.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:44:36.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: task.result() 2021-07-31T03:44:36.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:44:36.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:44:34.889000+0000 mgr.x (mgr.24542) 5028 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:44:36.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:44:36.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: 2021-07-31T03:44 2021-07-31T03:44:36.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: :34.900098+0000 mgr.x ( 2021-07-31T03:44:36.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: mgr.24542) 5029 : cephadm [ERR] 2021-07-31T03:44:36.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: 2021-07-31T03:44:36.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:44:36.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:44:36.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: yield 2021-07-31T03:44:36.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:44:36.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:44:36.043 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: audit 2021-07-31T03:44:34.842614+0000 mon.b (mon.2) 662 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:44:36.044 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: audit 2021-07-31T03:44:34.844039+0000 mon.b (mon.2) 663 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:44:36.044 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: audit 2021-07-31T03:44:34.844902+0000 mon.b (mon.2) 664 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:44:36.044 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: cephadm 2021-07- 2021-07-31T03:44:36.044 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: 31T03:44:34.845650+0000 mgr.x (mgr.24542) 5026 : cephadm 2021-07-31T03:44:36.044 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:44:36.045 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:44:34.888547+0000 mgr.x 2021-07-31T03:44:36.045 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: (mgr.24542) 5027 : cephadm [ERR] 2021-07-31T03:44:36.045 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:44:36.045 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:44:36.045 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: yield 2021-07-31T03:44:36.046 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:44:36.046 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:44:36.046 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:44:36.046 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:44:36.046 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:44:36.047 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:44:36.047 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:44:36.047 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:44:36.047 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:44:36.047 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: task.result() 2021-07-31T03:44:36.048 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:44:36.048 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:44:34.889000+0000 mgr.x (mgr.24542) 5028 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:44:36.048 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:44:34.900098+0000 mgr.x (mgr.24542) 5029 : cephadm [ERR] 2021-07-31T03:44:36.048 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:44:36.049 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:44:36.049 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: yield 2021-07-31T03:44:36.049 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:44:36.049 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:44:36.049 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:44:36.049 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:44:36.050 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:44:36.050 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:44:36.050 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:44:36.050 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:44:36.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:44:36.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:44:36.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:44:36.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:44:36.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:44:36.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:44:36.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:44:36.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: task.result() 2021-07-31T03:44:36.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:44:36.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:44:34.900604+0000 mgr.x (mgr.24542) 5030 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:44:36.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:44:36.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: 2021-07-31T03: 2021-07-31T03:44:36.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: 44:34.925620+0000 mgr.x ( 2021-07-31T03:44:36.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: mgr.24542) 5031 : cephadm [ERR] 2021-07-31T03:44:36.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:44:36.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:44:36.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: yield 2021-07-31T03:44:36.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:44:36.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:44:36.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:44:36.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:44:36.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:44:36.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:44:36.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:44:36.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:44:36.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:44:36.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: task.result() 2021-07-31T03:44:36.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:44:36.057 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: 2021-07-31T03:44:36.057 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:44:34.926051+0000 mgr.x (mgr.24542) 5032 : cephadm 2021-07-31T03:44:36.057 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:44:36.057 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:44:36.057 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:44:36.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: yield 2021-07-31T03:44:36.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:44:36.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:44:36.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:44:36.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:44:36.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:44:36.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:44:36.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:44:36.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:44:36.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:44:36.060 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: task.result() 2021-07-31T03:44:36.060 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:44:36.060 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: 2021-07-31T03:44:36.060 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:44:36.060 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: 2021-07-31T03:44:36.061 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:44:36.061 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:44:36.061 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:44:36.061 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:44:36.062 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:44:36.062 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:44:36.062 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:44:36.062 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:44:36.062 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:44:36.063 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:44:36.063 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:44:36.063 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:44:36.063 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:44:36.063 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:44:36.064 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:44:36.064 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:44:36.064 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:44:36.064 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:44:36.064 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: raise 2021-07-31T03:44:36.064 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:44:36.065 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:44:36.065 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:44:36.065 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:44:36.065 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:44:36.065 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:44:34.934453+0000 mgr.x (mgr.24542) 5033 : cephadm [ERR] 2021-07-31T03:44:36.066 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:44:36.066 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:44:36.066 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: yield 2021-07-31T03:44:36.066 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:44:36.067 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:44:36.067 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:44:36.067 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:44:36.067 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:44:36.067 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:44:36.068 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:44:36.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:44:36.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: task.result() 2021-07-31T03:44:36.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:44:36.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:44: 2021-07-31T03:44:36.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: 34.900604+0000 mgr.x (mgr.24542) 5030 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:44:36.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:44:34.925620+0000 mgr.x (mgr.24542) 5031 : cephadm [ERR] 2021-07-31T03:44:36.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:44:36.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:44:36.070 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: yield 2021-07-31T03:44:36.070 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:44:36.070 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:44:36.070 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:44:36.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:44:36.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:44:36.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:44:36.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:44:36.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:44:36.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:44:36.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: task.result() 2021-07-31T03:44:36.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:44:36.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:44:34.926051+0000 mgr.x (mgr.24542) 5032 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:44:36.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:44:36.073 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:44:36.073 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: yield 2021-07-31T03:44:36.073 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:44:36.073 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:44:36.073 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:44:36.074 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:44:36.074 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:44:36.074 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:44:36.074 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:44:36.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:44:36.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:44:36.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: task.result() 2021-07-31T03:44:36.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:44:36.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: 2021-07-31T03:44:36.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:44:36.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: 2021-07-31T03:44:36.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:44:36.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:44:36.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:44:36.077 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:44:36.077 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:44:36.078 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: task.result() 2021-07-31T03:44:36.078 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:44:36.078 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:44:34.934835+0000 mgr.x (mgr.24542) 5034 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:44:36.078 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:44:36.078 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:44:36.079 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: yield 2021-07-31T03:44:36.079 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:44:36.079 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:44:36.079 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:44:36.080 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:44:36.080 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:44:36.080 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:44:36.080 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:44:36.081 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:44:36.081 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:44:36.081 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: task.result() 2021-07-31T03:44:36.081 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:44:36.081 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: 2021-07-31T03:44:36.082 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:44:36.082 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: 2021-07-31T03:44:36.082 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:44:36.082 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:44:36.082 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:44:36.083 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:44:36.083 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:44:36.083 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:44:36.083 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:44:36.084 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:44:36.084 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:44:36.084 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:44:36.084 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:44:36.084 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:44:36.085 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:44:36.085 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:44:36.085 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:44:36.085 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:44:36.085 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:44:36.086 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:44:36.086 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: raise 2021-07-31T03:44:36.086 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:44:36.086 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:44:36.086 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:44:36.087 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:44:36.087 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:44:36.087 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:44:36.088 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:44:36.088 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:44:36.088 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:44:36.088 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:44:36.088 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:44:36.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:44:36.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:44:36.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:44:36.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:44:36.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:44:36.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:44:36.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:44:36.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:44:36.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:44:36.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: raise 2021-07-31T03:44:36.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:44:36.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:44:36.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:44:36.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:44:36.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:44:36.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:44:34.934453+0000 mgr.x (mgr.24542) 5033 : cephadm [ERR] 2021-07-31T03:44:36.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:44:36.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:44:36.093 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: yield 2021-07-31T03:44:36.093 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:44:36.093 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:44:36.093 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:44:36.093 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:44:36.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:44:36.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:44:36.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:44:36.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:44:36.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:44:36.095 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: task.result() 2021-07-31T03:44:36.095 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:44:36.095 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:44:34.934835+0000 mgr.x (mgr.24542) 5034 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:44:36.095 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:44:36.095 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:44:36.096 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: yield 2021-07-31T03:44:36.096 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:44:36.096 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:44:36.096 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:44:36.097 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:44:36.097 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:44:36.097 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:44:36.097 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:44:36.097 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:44:36.098 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:44:36.098 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: task.result() 2021-07-31T03:44:36.098 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:44:36.098 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: 2021-07-31T03:44:36.098 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:44:36.099 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: 2021-07-31T03:44:36.099 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:44:36.099 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:44:36.099 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:44:36.099 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:44:36.100 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:44:36.100 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:44:36.100 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:44:36.100 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:44:36.101 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:44:36.101 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:44:36.101 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:44:36.101 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:44:36.101 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:44:36.102 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:44:36.102 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:44:36.102 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:44:36.102 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:44:36.102 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:44:36.103 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: raise 2021-07-31T03:44:36.103 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:44:36.103 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:44:36.103 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:44:36.103 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:44:36.104 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:35 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:44:37.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:36 smithi154 conmon[34553]: cluster 2021-07-31T03:44:35.471209+0000 mgr.x (mgr.24542) 5035 : cluster [DBG] pgmap v5529: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:37.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:36 smithi045 conmon[33655]: cluster 2021-07-31T03:44:35.471209+0000 mgr.x (mgr.24542 2021-07-31T03:44:37.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:36 smithi045 conmon[33655]: ) 5035 : cluster [DBG] pgmap v5529: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:37.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:36 smithi045 conmon[39718]: cluster 2021-07-31T03:44:35.471209+0000 mgr.x (mgr.24542) 5035 : cluster [DBG] pgmap v5529: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:39.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:38 smithi154 conmon[34553]: cluster 2021-07-31T03:44:37.471611+0000 mgr.x (mgr.24542) 5036 : cluster [DBG] pgmap v5530: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:39.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:38 smithi045 conmon[39718]: cluster 2021-07-31T03:44:37.471611+0000 mgr.x (mgr 2021-07-31T03:44:39.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:38 smithi045 conmon[39718]: .24542) 5036 : cluster [DBG] pgmap v5530: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:39.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:38 smithi045 conmon[33655]: cluster 2021-07-31T03:44:37.471611+0000 mgr.x (mgr.24542 2021-07-31T03:44:39.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:38 smithi045 conmon[33655]: ) 5036 : cluster [DBG] pgmap v5530: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:41.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:40 smithi154 conmon[34553]: cluster 2021-07-31T03:44:39.472073+0000 mgr.x (mgr.24542) 5037 : cluster [DBG] pgmap v5531: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:41.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:40 smithi045 conmon[39718]: cluster 2021-07-31T03:44:39.472073+0000 mgr.x ( 2021-07-31T03:44:41.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:40 smithi045 conmon[39718]: mgr.24542) 5037 : cluster [DBG] pgmap v5531: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:41.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:40 smithi045 conmon[33655]: cluster 2021-07-31T03:44:39.472073+0000 mgr.x (mgr 2021-07-31T03:44:41.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:40 smithi045 conmon[33655]: .24542) 5037 : cluster [DBG] pgmap v5531: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:43.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:42 smithi154 conmon[34553]: cluster 2021-07-31T03:44:41.472595+0000 mgr.x (mgr.24542 2021-07-31T03:44:43.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:42 smithi154 conmon[34553]: ) 5038 : cluster [DBG] pgmap v5532: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:43.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:42 smithi045 conmon[39718]: cluster 2021-07-31T03:44:41.472595+0000 mgr.x (mgr 2021-07-31T03:44:43.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:42 smithi045 conmon[39718]: .24542) 5038 : cluster [DBG] pgmap v5532: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:43.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:42 smithi045 conmon[33655]: cluster 2021-07-31T03:44:41.472595+0000 2021-07-31T03:44:43.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:42 smithi045 conmon[33655]: mgr.x (mgr.24542) 5038 : cluster [DBG] pgmap v5532: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:45.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:44 smithi154 conmon[34553]: cluster 2021-07-31T03:44:43.473148+0000 mgr.x (mgr.24542) 5039 : cluster [DBG] pgmap v5533: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:45.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:44 smithi045 conmon[33655]: cluster 2021-07-31T03:44:43.473148 2021-07-31T03:44:45.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:44 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 5039 : cluster [DBG] pgmap v5533: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:45.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:44 smithi045 conmon[39718]: cluster 2021-07-31T03:44:43.473148+0000 2021-07-31T03:44:45.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:44 smithi045 conmon[39718]: mgr.x (mgr.24542) 5039 : cluster [DBG] pgmap v5533: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:47.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:46 smithi154 conmon[34553]: cluster 2021-07-31T03:44:45.473734+0000 mgr.x (mgr.24542) 5040 : cluster [DBG] pgmap v5534: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:47.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:46 smithi045 conmon[33655]: cluster 2021-07 2021-07-31T03:44:47.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:46 smithi045 conmon[33655]: -31T03:44:45.473734+0000 mgr.x (mgr.24542) 5040 : cluster [DBG] pgmap v5534: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:47.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:46 smithi045 conmon[39718]: cluster 2021-07-31T03:44:45.473734+0000 mgr.x 2021-07-31T03:44:47.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:46 smithi045 conmon[39718]: (mgr.24542) 5040 : cluster [DBG] pgmap v5534: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:49.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:48 smithi154 conmon[34553]: cluster 2021-07-31T03:44:47.474280+0000 mgr.x (mgr.24542) 5041 : cluster [DBG] pgmap v5535: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:49.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:48 smithi154 conmon[34553]: 2021-07-31T03:44:49.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:48 smithi045 conmon[33655]: cluster 2021-07-31T03:44:47.474280+0000 mgr.x (mgr.24542) 5041 2021-07-31T03:44:49.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:48 smithi045 conmon[33655]: : cluster [DBG] pgmap v5535: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:49.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:48 smithi045 conmon[39718]: cluster 2021-07-31T03:44:47.474280+0000 mgr.x 2021-07-31T03:44:49.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:48 smithi045 conmon[39718]: (mgr.24542) 5041 : cluster [DBG] pgmap v5535: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:51.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:50 smithi154 conmon[34553]: cluster 2021-07-31T03:44:49.474953+0000 mgr.x (mgr.24542) 5042 : cluster [DBG] pgmap v5536: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:51.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:50 smithi045 conmon[33655]: cluster 2021-07-31T03:44:49.474953+0000 mgr.x (mgr.24542) 5042 2021-07-31T03:44:51.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:50 smithi045 conmon[33655]: : cluster [DBG] pgmap v5536: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:51.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:50 smithi045 conmon[39718]: cluster 2021-07-31T03:44:49.474953+0000 mgr.x (mgr 2021-07-31T03:44:51.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:50 smithi045 conmon[39718]: .24542) 5042 : cluster [DBG] pgmap v5536: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:53.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:52 smithi154 conmon[34553]: cluster 2021-07-31T03:44:51.475331+0000 mgr.x (mgr.24542) 5043 : cluster [DBG] pgmap v5537: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:53.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:52 smithi045 conmon[33655]: cluster 2021-07-31T03 2021-07-31T03:44:53.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:52 smithi045 conmon[33655]: :44:51.475331+0000 mgr.x (mgr.24542) 5043 : cluster [DBG] pgmap v5537: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:53.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:52 smithi045 conmon[39718]: cluster 2021-07-31T03:44:51.475331+0000 2021-07-31T03:44:53.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:52 smithi045 conmon[39718]: mgr.x (mgr.24542) 5043 : cluster [DBG] pgmap v5537: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:55.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:54 smithi154 conmon[34553]: cluster 2021-07-31T03:44:53.475818+0000 mgr.x (mgr.24542) 5044 : cluster [DBG] pgmap v5538: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:55.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:54 smithi045 conmon[33655]: cluster 2021-07-31T03:44:53.475818+0000 mgr.x (mgr 2021-07-31T03:44:55.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:54 smithi045 conmon[33655]: .24542) 5044 : cluster [DBG] pgmap v5538: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:55.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:54 smithi045 conmon[39718]: cluster 2021-07-31T03:44:53.475818+0000 2021-07-31T03:44:55.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:54 smithi045 conmon[39718]: mgr.x (mgr.24542) 5044 : cluster [DBG] pgmap v5538: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:57.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:56 smithi154 conmon[34553]: cluster 2021-07-31T03:44:55.476234+0000 mgr.x (mgr.24542) 5045 : cluster [DBG] pgmap v5539: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:57.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:56 smithi045 conmon[33655]: cluster 2021-07-31T03:44:55.476234+0000 mgr.x (mgr.24542) 5045 : cluster [DBG] 2021-07-31T03:44:57.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:56 smithi045 conmon[33655]: pgmap v5539: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:57.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:56 smithi045 conmon[39718]: cluster 2021-07-31T03:44:55.476234 2021-07-31T03:44:57.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:56 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 5045 : cluster [DBG] pgmap v5539: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:59.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:44:58 smithi154 conmon[34553]: cluster 2021-07-31T03:44:57.476786+0000 mgr.x (mgr.24542) 5046 : cluster [DBG] pgmap v5540: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:59.024 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:58 smithi045 conmon[33655]: cluster 2021-07-31T03:44:57.476786+0000 mgr.x (mgr. 2021-07-31T03:44:59.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:44:58 smithi045 conmon[33655]: 24542) 5046 : cluster [DBG] pgmap v5540: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:59.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:58 smithi045 conmon[39718]: cluster 2021-07-31T03:44:57.476786+0000 mgr.x 2021-07-31T03:44:59.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:44:58 smithi045 conmon[39718]: (mgr.24542) 5046 : cluster [DBG] pgmap v5540: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:44:59.178 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:44:59.456 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:44:59.456 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 119m ago 2h - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:44:59.456 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 117m ago 2h 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:44:59.457 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 117m ago 2h 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:44:59.457 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 119m ago 2h - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:44:59.457 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 119m ago 2h - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:44:59.457 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 117m ago 2h 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:44:59.457 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 119m ago 2h - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:44:59.458 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 119m ago 2h - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:44:59.458 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 117m ago 2h 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:44:59.458 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 119m ago 2h - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:44:59.458 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 119m ago 2h - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:44:59.458 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 119m ago 2h - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:44:59.459 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 119m ago 2h - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:44:59.459 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 117m ago 2h 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:44:59.459 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 117m ago 2h 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:44:59.459 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 117m ago 2h 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:44:59.459 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 117m ago 2h 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:44:59.460 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 117m ago 2h 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:44:59.794 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:44:59.794 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:44:59.795 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:44:59.795 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:44:59.795 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:44:59.795 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:44:59.795 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:44:59.796 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:44:59.796 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:44:59.796 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:44:59.796 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:44:59.796 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:44:59.797 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:44:59.797 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:44:59.797 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:44:59.797 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:44:59.797 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:45:01.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:00 smithi154 conmon[34553]: audit 2021-07-31T03:44:59.167260+0000 mgr.x (mgr.24542) 5047 : audit [DBG] from='client.30094 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:45:01.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:00 smithi154 conmon[34553]: audit 2021-07-31T03:44:59.449074+0000 mgr.x (mgr.24542) 5048 : audit [DBG] from='client.30100 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:45:01.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:00 smithi154 conmon[34553]: cluster 2021-07-31T03:44:59.477264+0000 mgr.x (mgr.24542) 5049 : cluster [DBG] pgmap v5541: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:01.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:00 smithi154 conmon[34553]: audit 2021-07-31T03:44:59.795346+0000 mon.a (mon.0 2021-07-31T03:45:01.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:00 smithi154 conmon[34553]: ) 3358 : audit [DBG] from='client.? 172.21.15.45:0/1616337270' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:45:01.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:00 smithi045 conmon[39718]: audit 2021-07-31T03:44:59.167260+0000 mgr.x (mgr.24542) 5047 : audit 2021-07-31T03:45:01.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:00 smithi045 conmon[39718]: [DBG] from='client.30094 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:45:01.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:00 smithi045 conmon[39718]: audit 2021-07-31T03:44:59.449074+0000 mgr.x (mgr.24542) 5048 : audit [DBG] 2021-07-31T03:45:01.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:00 smithi045 conmon[39718]: from='client.30100 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:45:01.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:00 smithi045 conmon[39718]: cluster 2021-07-31T03:44:59.477264+0000 2021-07-31T03:45:01.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:00 smithi045 conmon[39718]: mgr.x (mgr.24542) 5049 : cluster [DBG] pgmap v5541: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:01.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:00 smithi045 conmon[39718]: audit 2021-07-31T03:44:59.795346+0000 mon.a 2021-07-31T03:45:01.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:00 smithi045 conmon[39718]: (mon.0) 3358 : audit [DBG] from='client.? 172.21.15.45:0/1616337270' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:45:01.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:00 smithi045 conmon[33655]: audit 2021-07-31T03:44:59.167260+0000 mgr.x (mgr.24542) 5047 2021-07-31T03:45:01.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:00 smithi045 conmon[33655]: : audit [DBG] from='client.30094 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:45:01.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:00 smithi045 conmon[33655]: audit 2021-07-31T03:44:59.449074+0000 mgr.x (mgr.24542) 5048 : audit [DBG] from='client.30100 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:45:01.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:00 smithi045 conmon[33655]: cluster 2021-07 2021-07-31T03:45:01.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:00 smithi045 conmon[33655]: -31T03:44:59.477264+0000 mgr.x (mgr 2021-07-31T03:45:01.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:00 smithi045 conmon[33655]: .24542) 5049 : cluster [DBG] pgmap v5541: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:01.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:00 smithi045 conmon[33655]: audit 2021-07-31 2021-07-31T03:45:01.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:00 smithi045 conmon[33655]: T03:44:59.795346+0000 mon.a (mon.0) 3358 2021-07-31T03:45:01.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:00 smithi045 conmon[33655]: : audit [DBG] from='client.? 172.21.15.45:0/1616337270' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:45:03.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:02 smithi154 conmon[34553]: cluster 2021-07-31T03:45:01.477638+0000 mgr.x (mgr.24542) 5050 : cluster 2021-07-31T03:45:03.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:02 smithi154 conmon[34553]: [DBG] pgmap v5542: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:03.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:02 smithi045 conmon[39718]: cluster 2021-07-31T03:45:01.477638+0000 mgr.x (mgr. 2021-07-31T03:45:03.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:02 smithi045 conmon[39718]: 24542) 5050 : cluster [DBG] pgmap v5542: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:03.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:02 smithi045 conmon[33655]: cluster 2021-07-31T03:45:01.477638+0000 mgr.x (mgr.24542) 2021-07-31T03:45:03.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:02 smithi045 conmon[33655]: 5050 : cluster [DBG] pgmap v5542: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:05.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:04 smithi154 conmon[34553]: cluster 2021-07-31T03:45:03.478112+0000 mgr.x (mgr.24542) 5051 : cluster [DBG] pgmap v5543: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:05.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:04 smithi154 conmon[34553]: 2021-07-31T03:45:05.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:04 smithi045 conmon[39718]: cluster 2021-07-31T03:45:03.478112+0000 mgr.x (mgr 2021-07-31T03:45:05.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:04 smithi045 conmon[39718]: .24542) 5051 : cluster [DBG] pgmap v5543: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:05.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:04 smithi045 conmon[33655]: cluster 2021-07-31T03:45:03 2021-07-31T03:45:05.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:04 smithi045 conmon[33655]: .478112+0000 mgr.x (mgr.24542) 5051 : cluster [DBG] pgmap v5543: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:07.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:06 smithi154 conmon[34553]: cluster 2021-07-31T03:45:05.478632+0000 mgr.x (mgr.24542) 5052 : cluster [DBG] 2021-07-31T03:45:07.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:06 smithi154 conmon[34553]: pgmap v5544: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:07.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:06 smithi045 conmon[39718]: cluster 2021-07-31T03:45:05.478632+0000 mgr.x 2021-07-31T03:45:07.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:06 smithi045 conmon[39718]: (mgr.24542) 5052 : cluster [DBG] pgmap v5544: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:07.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:06 smithi045 conmon[33655]: cluster 2021-07-31T03:45:05.478632+0000 mgr.x (mgr. 2021-07-31T03:45:07.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:06 smithi045 conmon[33655]: 24542) 5052 : cluster [DBG] pgmap v5544: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:09.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:08 smithi154 conmon[34553]: cluster 2021-07-31T03:45:07.479017+0000 mgr.x (mgr.24542) 5053 : cluster [DBG] pgmap v5545: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:09.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:08 smithi045 conmon[33655]: cluster 2021- 2021-07-31T03:45:09.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:08 smithi045 conmon[33655]: 07-31T03:45:07.479017+0000 mgr.x (mgr.24542) 5053 : cluster [DBG] pgmap v5545: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:09.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:08 smithi045 conmon[39718]: cluster 2021-07-31T03:45:07.479017+0000 mgr.x (mgr.24542) 5053 2021-07-31T03:45:09.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:08 smithi045 conmon[39718]: : cluster [DBG] pgmap v5545: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:11.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:10 smithi154 conmon[34553]: cluster 2021-07-31T03:45:09.479526+0000 mgr.x (mgr.24542) 5054 : cluster [DBG] pgmap v5546: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:11.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:10 smithi045 conmon[33655]: cluster 2021-07-31T03:45:09.479526+0000 mgr.x (mgr. 2021-07-31T03:45:11.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:10 smithi045 conmon[33655]: 24542) 5054 : cluster [DBG] pgmap v5546: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:11.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:10 smithi045 conmon[39718]: cluster 2021-07-31T03:45:09.479526+0000 mgr.x (mgr 2021-07-31T03:45:11.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:10 smithi045 conmon[39718]: .24542) 5054 : cluster [DBG] pgmap v5546: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:13.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:12 smithi154 conmon[34553]: cluster 2021-07-31T03:45:11.480076+0000 mgr.x (mgr.24542) 5055 : cluster [DBG] pgmap v5547: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:13.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:12 smithi045 conmon[33655]: cluster 2021-07-31T03:45:11.480076+0000 mgr.x (mgr.24542) 2021-07-31T03:45:13.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:12 smithi045 conmon[33655]: 5055 : cluster [DBG] pgmap v5547: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:13.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:12 smithi045 conmon[39718]: cluster 2021-07-31T03:45:11.480076+0000 mgr.x (mgr 2021-07-31T03:45:13.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:12 smithi045 conmon[39718]: .24542) 5055 : cluster [DBG] pgmap v5547: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:15.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:14 smithi154 conmon[34553]: cluster 2021-07-31T03:45:13.480485+0000 mgr.x (mgr.24542) 5056 : cluster [DBG] pgmap v5548: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:15.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:14 smithi045 conmon[33655]: cluster 2021-07-31T03:45:13.480485+0000 2021-07-31T03:45:15.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:14 smithi045 conmon[33655]: mgr.x (mgr.24542) 5056 : cluster [DBG] pgmap v5548: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:15.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:14 smithi045 conmon[39718]: cluster 2021-07-31T03:45:13.480485+0000 2021-07-31T03:45:15.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:14 smithi045 conmon[39718]: mgr.x (mgr.24542) 5056 : cluster [DBG] pgmap v5548: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:17.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:16 smithi154 conmon[34553]: cluster 2021-07-31T03:45:15.481029+0000 mgr.x (mgr.24542) 5057 : cluster [DBG] pgmap v5549: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:17.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:16 smithi154 conmon[34553]: 2021-07-31T03:45:17.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:16 smithi045 conmon[33655]: cluster 2021-07-31T03:45:15.481029+0000 mgr.x (mgr. 2021-07-31T03:45:17.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:16 smithi045 conmon[33655]: 24542) 5057 : cluster [DBG] pgmap v5549: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:17.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:16 smithi045 conmon[39718]: cluster 2021-07-31T03:45:15.481029+0000 mgr.x ( 2021-07-31T03:45:17.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:16 smithi045 conmon[39718]: mgr.24542) 5057 : cluster [DBG] pgmap v5549: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:19.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:18 smithi154 conmon[34553]: cluster 2021-07-31T03:45:17.481420+0000 mgr.x (mgr.24542) 5058 : cluster [DBG] 2021-07-31T03:45:19.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:18 smithi154 conmon[34553]: pgmap v5550: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:19.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:18 smithi045 conmon[39718]: cluster 2021-07-31T03:45:17.481420+0000 mgr.x (mgr. 2021-07-31T03:45:19.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:18 smithi045 conmon[39718]: 24542) 5058 : cluster [DBG] pgmap v5550: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:19.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:18 smithi045 conmon[33655]: cluster 2021-07-31T03:45:17.481420+0000 mgr.x (mgr.24542) 5058 : cluster [DBG] pgmap v5550: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:21.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:20 smithi154 conmon[34553]: cluster 2021-07-31T03:45:19.481942+0000 mgr.x (mgr.24542) 5059 : cluster [DBG] pgmap v5551: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:21.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:20 smithi045 conmon[33655]: cluster 2021-07-31T03:45:19.481942+0000 mgr.x (mgr.24542) 5059 : 2021-07-31T03:45:21.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:20 smithi045 conmon[33655]: cluster [DBG] pgmap v5551: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:21.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:20 smithi045 conmon[39718]: cluster 2021-07-31T03:45:19.481942+0000 mgr.x 2021-07-31T03:45:21.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:20 smithi045 conmon[39718]: (mgr.24542) 5059 : cluster [DBG] pgmap v5551: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:23.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:22 smithi154 conmon[34553]: cluster 2021-07-31T03:45:21.482389+0000 mgr.x (mgr.24542) 5060 : cluster [DBG] 2021-07-31T03:45:23.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:22 smithi154 conmon[34553]: pgmap v5552: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:23.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:22 smithi045 conmon[39718]: cluster 2021-07-31T03:45:21.482389+0000 mgr.x (mgr.24542) 2021-07-31T03:45:23.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:22 smithi045 conmon[39718]: 5060 : cluster [DBG] pgmap v5552: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:23.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:22 smithi045 conmon[33655]: cluster 2021-07-31T03:45:21.482389 2021-07-31T03:45:23.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:22 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 5060 : cluster [DBG] pgmap v5552: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:25.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:24 smithi154 conmon[34553]: cluster 2021-07-31T03:45:23.482836+0000 mgr.x (mgr.24542) 5061 : cluster [DBG] pgmap v5553: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:25.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:24 smithi045 conmon[39718]: cluster 2021-07-31T03:45:23.482836+0000 mgr.x (mgr 2021-07-31T03:45:25.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:24 smithi045 conmon[39718]: .24542) 5061 : cluster [DBG] pgmap v5553: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:25.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:24 smithi045 conmon[33655]: cluster 2021-07-31T03:45:23.482836+0000 mgr.x (mgr.24542) 5061 : 2021-07-31T03:45:25.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:24 smithi045 conmon[33655]: cluster [DBG] pgmap v5553: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:27.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:26 smithi154 conmon[34553]: cluster 2021-07-31T03:45:25.483224+0000 mgr.x (mgr.24542) 5062 : cluster [DBG] 2021-07-31T03:45:27.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:26 smithi154 conmon[34553]: pgmap v5554: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:27.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:26 smithi045 conmon[39718]: cluster 2021-07-31T03:45:25.483224+0000 mgr.x (mgr 2021-07-31T03:45:27.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:26 smithi045 conmon[39718]: .24542) 5062 : cluster [DBG] pgmap v5554: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:27.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:26 smithi045 conmon[33655]: cluster 2021-07-31T03:45:25.483224+0000 mgr.x (mgr.24542) 5062 : cluster [DBG] pgmap v5554: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:29.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:28 smithi154 conmon[34553]: cluster 2021-07-31T03:45:27.483620+0000 mgr.x (mgr.24542) 5063 : cluster [DBG] pgmap v5555: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:29.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:28 smithi154 conmon[34553]: audit 2021-07-31T03:45:28.404353+0000 mon.b (mon.2) 665 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:45:29.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:28 smithi154 conmon[34553]: audit 2021-07-31T03:45:28.404597+0000 mon.b (mon.2) 666 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:45:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:28 smithi154 conmon[34553]: audit 2021-07-31T03:45: 2021-07-31T03:45:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:28 smithi154 conmon[34553]: 28.405139+0000 mon.a (mon.0) 3359 : audit 2021-07-31T03:45:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:28 smithi154 conmon[34553]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:45:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:28 smithi154 conmon[34553]: audit 2021-07-31T03: 2021-07-31T03:45:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:28 smithi154 conmon[34553]: 45:28.405711+0000 mon.a (mon.0 2021-07-31T03:45:29.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:28 smithi154 conmon[34553]: ) 3360 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:45:29.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:28 smithi045 conmon[39718]: cluster 2021-07-31T03:45:27.483620 2021-07-31T03:45:29.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:28 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 5063 : cluster [DBG] pgmap v5555: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:29.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:28 smithi045 conmon[39718]: audit 2021 2021-07-31T03:45:29.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:28 smithi045 conmon[39718]: -07-31T03:45:28 2021-07-31T03:45:29.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:28 smithi045 conmon[39718]: .404353+0000 mon.b (mon 2021-07-31T03:45:29.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:28 smithi045 conmon[39718]: .2) 665 : audit [INF] 2021-07-31T03:45:29.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:28 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:45:29.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:28 smithi045 conmon[39718]: audit 2021 2021-07-31T03:45:29.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:28 smithi045 conmon[39718]: -07-31T03:45:28.404597+0000 mon.b (mon 2021-07-31T03:45:29.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:28 smithi045 conmon[39718]: .2) 666 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:45:29.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:28 smithi045 conmon[39718]: audit 2021-07-31T03:45:29.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:28 smithi045 conmon[39718]: 2021-07-31T03:45:28.405139+0000 mon.a ( 2021-07-31T03:45:29.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:28 smithi045 conmon[39718]: mon.0) 3359 : audit [INF] 2021-07-31T03:45:29.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:28 smithi045 conmon[39718]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:45:29.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:28 smithi045 conmon[39718]: audit 2021 2021-07-31T03:45:29.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:28 smithi045 conmon[39718]: -07-31T03:45:28. 2021-07-31T03:45:29.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:28 smithi045 conmon[39718]: 405711+0000 mon.a (mon.0 2021-07-31T03:45:29.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:28 smithi045 conmon[39718]: ) 3360 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:45:29.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:28 smithi045 conmon[39718]: 2021-07-31T03:45:29.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:28 smithi045 conmon[33655]: cluster 2021-07-31T03:45:27.483620+0000 mgr.x (mgr.24542) 5063 : cluster [DBG] pgmap v5555: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:29.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:28 smithi045 conmon[33655]: 2021-07-31T03:45:29.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:28 smithi045 conmon[33655]: audit 2021-07-31T03:45:28.404353+0000 mon.b (mon.2) 665 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:45:29.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:28 smithi045 conmon[33655]: audit 2021-07-31T03: 2021-07-31T03:45:29.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:28 smithi045 conmon[33655]: 45:28.404597+0000 mon.b (mon 2021-07-31T03:45:29.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:28 smithi045 conmon[33655]: .2) 666 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:45:29.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:28 smithi045 conmon[33655]: audit 2021-07-31 2021-07-31T03:45:29.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:28 smithi045 conmon[33655]: T03:45:28.405139+0000 mon.a (mon.0 2021-07-31T03:45:29.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:28 smithi045 conmon[33655]: ) 3359 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:45:29.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:28 smithi045 conmon[33655]: audit 2021 2021-07-31T03:45:29.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:28 smithi045 conmon[33655]: -07-31T03:45:28.405711 2021-07-31T03:45:29.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:28 smithi045 conmon[33655]: +0000 mon.a (mon.0) 2021-07-31T03:45:29.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:28 smithi045 conmon[33655]: 3360 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:45:30.108 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:45:30.386 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:45:30.387 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 2h ago 2h - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:45:30.387 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 118m ago 2h 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:45:30.387 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 118m ago 2h 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:45:30.387 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 2h ago 2h - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:45:30.387 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:45:30.388 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 118m ago 2h 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:45:30.388 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:45:30.388 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 2h ago 2h - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:45:30.388 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 118m ago 2h 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:45:30.388 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:45:30.389 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:45:30.389 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:45:30.389 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:45:30.389 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 118m ago 2h 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:45:30.389 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 118m ago 2h 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:45:30.390 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 118m ago 2h 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:45:30.390 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 118m ago 2h 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:45:30.390 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 118m ago 2h 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:45:30.730 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:45:30.731 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:45:30.731 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:45:30.731 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:45:30.731 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:45:30.732 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:45:30.732 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:45:30.732 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:45:30.732 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:45:30.732 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:45:30.733 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:45:30.733 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:45:30.733 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:45:30.733 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:45:30.733 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:45:30.734 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:45:30.734 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:45:31.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:30 smithi154 conmon[34553]: cluster 2021-07-31T03:45:29.484020+0000 mgr.x (mgr.24542) 5064 : cluster [DBG] pgmap v5556: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:31.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:30 smithi154 conmon[34553]: audit 2021-07-31T03:45:31.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:30 smithi154 conmon[34553]: 2021-07-31T03:45:30.096339+0000 mgr.x (mgr.24542) 5065 : audit [DBG] from='client.20172 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:45:31.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:30 smithi045 conmon[33655]: cluster 2021-07-31T03:45:29.484020+0000 2021-07-31T03:45:31.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:30 smithi045 conmon[33655]: mgr.x (mgr.24542) 5064 : cluster [DBG] pgmap v5556: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:31.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:30 smithi045 conmon[33655]: audit 2021-07-31T03:45:30.096339+0000 mgr.x (mgr.24542) 5065 : audit [DBG] from='client.20172 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:45:31.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:30 smithi045 conmon[39718]: cluster 2021-07-31T03:45:29.484020+0000 mgr.x (mgr. 2021-07-31T03:45:31.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:30 smithi045 conmon[39718]: 24542) 5064 : cluster [DBG] pgmap v5556: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:31.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:30 smithi045 conmon[39718]: audit 2021-07-31T03:45:30.096339+0000 mgr.x (mgr.24542 2021-07-31T03:45:31.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:30 smithi045 conmon[39718]: ) 5065 : audit [DBG] from='client.20172 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:45:32.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:31 smithi154 conmon[34553]: audit 2021-07-31T03:45:30.379362+0000 mgr.x (mgr.24542) 5066 : audit [DBG] 2021-07-31T03:45:32.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:31 smithi154 conmon[34553]: from='client.30118 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:45:32.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:31 smithi154 conmon[34553]: audit 2021-07-31T03:45:30.731259+0000 mon.a (mon.0) 3361 : audit [DBG] from='client.? 172.21.15.45:0/2773381919' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:45:32.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:31 smithi045 conmon[33655]: audit 2021-07-31T03:45:30.379362+0000 mgr.x (mgr.24542) 5066 : audit [DBG] from='client.30118 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:45:32.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:31 smithi045 conmon[33655]: audit 2021-07-31T03:45:30.731259+0000 mon.a (mon.0) 3361 : audit [DBG] from='client.? 172.21.15.45:0/2773381919' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:45:32.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:31 smithi045 conmon[39718]: audit 2021-07-31T03:45:30.379362+0000 mgr.x (mgr.24542) 5066 : audit [DBG] 2021-07-31T03:45:32.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:31 smithi045 conmon[39718]: from='client.30118 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:45:32.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:31 smithi045 conmon[39718]: audit 2021-07-31T03:45:30.731259+0000 mon.a (mon.0) 3361 : audit 2021-07-31T03:45:32.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:31 smithi045 conmon[39718]: [DBG] from='client.? 172.21.15.45:0/2773381919' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:45:33.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:32 smithi154 conmon[34553]: cluster 2021-07-31T03:45:31.484380+0000 mgr.x (mgr.24542) 5067 : cluster 2021-07-31T03:45:33.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:32 smithi154 conmon[34553]: [DBG] pgmap v5557: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:33.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:32 smithi045 conmon[39718]: cluster 2021-07-31T03:45:31.484380+0000 mgr.x (mgr.24542 2021-07-31T03:45:33.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:32 smithi045 conmon[39718]: ) 5067 : cluster [DBG] pgmap v5557: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:33.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:32 smithi045 conmon[33655]: cluster 2021-07-31T03:45:31.484380 2021-07-31T03:45:33.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:32 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 5067 : cluster [DBG] pgmap v5557: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:34.961 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:34 smithi154 conmon[34553]: cluster 2021-07-31T03:45:33.484772+0000 mgr.x (mgr.24542) 5068 : cluster [DBG] pgmap v5558: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:34.962 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:34 smithi045 conmon[33655]: cluster 2021-07-31T03:45:33.484772+0000 mgr.x (mgr.24542) 5068 2021-07-31T03:45:34.962 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:34 smithi045 conmon[33655]: : cluster [DBG] pgmap v5558: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:34.963 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:34 smithi045 conmon[39718]: cluster 2021-07-31T03:45:33.484772+0000 mgr.x (mgr.24542 2021-07-31T03:45:34.963 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:34 smithi045 conmon[39718]: ) 5068 : cluster [DBG] pgmap v5558: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:35.252 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:34 smithi154 conmon[57492]: debug 2021-07-31T03:45:34.980+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:45:35.253 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:34 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:45:35.253 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:45:35.253 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:34 smithi154 conmon[57492]: yield 2021-07-31T03:45:35.253 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:45:35.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:34 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:45:35.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:45:35.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:34 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:45:35.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:45:35.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:34 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:45:35.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:45:35.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:34 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:45:35.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:45:35.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:34 smithi154 conmon[57492]: task.result() 2021-07-31T03:45:35.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:34 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:45:35.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:34 smithi154 conmon[57492]: debug 2021-07-31T03:45:34.992+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:45:35.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:34 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:45:35.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:45:35.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:34 smithi154 conmon[57492]: yield 2021-07-31T03:45:35.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:34 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:45:35.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:34 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:45:35.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:45:35.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:34 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:45:35.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:45:35.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:34 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:45:35.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:45:35.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:34 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:45:35.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:34 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:45:35.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:34 smithi154 conmon[57492]: task.result() 2021-07-31T03:45:35.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:34 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:45:35.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: debug 2021-07-31T03:45:35.017+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:45:35.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:45:35.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:45:35.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: yield 2021-07-31T03:45:35.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:45:35.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:45:35.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:45:35.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:45:35.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:45:35.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:45:35.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:45:35.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:45:35.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:45:35.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: task.result() 2021-07-31T03:45:35.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:45:35.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: debug 2021-07-31T03:45:35.018+0000 7fcb9115c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:45:35.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:45:35.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:45:35.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: yield 2021-07-31T03:45:35.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:45:35.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:45:35.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:45:35.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:45:35.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:45:35.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:45:35.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:45:35.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:45:35.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:45:35.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: task.result() 2021-07-31T03:45:35.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:45:35.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: 2021-07-31T03:45:35.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:45:35.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: 2021-07-31T03:45:35.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:45:35.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:45:35.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:45:35.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:45:35.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:45:35.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:45:35.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:45:35.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:45:35.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:45:35.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:45:35.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:45:35.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:45:35.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:45:35.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:45:35.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:45:35.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:45:35.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:45:35.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:45:35.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: raise 2021-07-31T03:45:35.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:45:35.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:45:35.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:45:35.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:45:35.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:45:35.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: debug 2021-07-31T03:45:35.026+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:45:35.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:45:35.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:45:35.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: yield 2021-07-31T03:45:35.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:45:35.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:45:35.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:45:35.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:45:35.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:45:35.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:45:35.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:45:35.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:45:35.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:45:35.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: task.result() 2021-07-31T03:45:35.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:45:35.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: debug 2021-07-31T03:45:35.027+0000 7fcb9095b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:45:35.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:45:35.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:45:35.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: yield 2021-07-31T03:45:35.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:45:35.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:45:35.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:45:35.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:45:35.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:45:35.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:45:35.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:45:35.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:45:35.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:45:35.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: task.result() 2021-07-31T03:45:35.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:45:35.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: 2021-07-31T03:45:35.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:45:35.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: 2021-07-31T03:45:35.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:45:35.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:45:35.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:45:35.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:45:35.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:45:35.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:45:35.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:45:35.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:45:35.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:45:35.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:45:35.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:45:35.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:45:35.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:45:35.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:45:35.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:45:35.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:45:35.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:45:35.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:45:35.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: raise 2021-07-31T03:45:35.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:45:35.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:45:35.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:45:35.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:45:35.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:45:36.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: audit 2021-07-31T03:45:34.935618+0000 mon.b (mon.2) 667 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:45:36.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: audit 2021-07-31T03:45:34.936961+0000 mon.b (mon.2) 668 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:45:36.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: audit 2021-07-31T03:45:34.937871+0000 mon.b (mon 2021-07-31T03:45:36.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: .2) 669 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:45:36.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:45:36.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: 2021-07-31T03:45: 2021-07-31T03:45:36.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: 34.938643+0000 mgr.x (mgr. 2021-07-31T03:45:36.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: 24542) 5069 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:45:36.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: 2021-07-31T03:45:36.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:45:34.981566+0000 2021-07-31T03:45:36.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: mgr.x (mgr.24542) 5070 : cephadm [ERR] 2021-07-31T03:45:36.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:45:36.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:45:36.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: yield 2021-07-31T03:45:36.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:45:36.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:45:36.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:45:36.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:45:36.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:45:36.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:45:36.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:45:36.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:45:36.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:45:36.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: task.result() 2021-07-31T03:45:36.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:45:36.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: 2021-07-31T03:45:36.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:45:34.982023+0000 mgr.x (mgr.24542) 5071 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:45:36.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: cephadm 2021-07 2021-07-31T03:45:36.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: -31T03:45:34.993428+0000 2021-07-31T03:45:36.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: mgr.x (mgr.24542) 5072 : cephadm 2021-07-31T03:45:36.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: [ERR] 2021-07-31T03:45:36.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:45:36.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:45:36.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: yield 2021-07-31T03:45:36.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:45:36.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:45:36.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:45:36.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:45:36.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:45:36.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:45:36.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:45:36.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:45:36.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:45:36.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: task.result() 2021-07-31T03:45:36.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:45:36.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:45:34.993984+0000 mgr.x (mgr.24542) 5073 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:45:36.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: cephadm 2021-07- 2021-07-31T03:45:36.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: 31T03:45:35.018781 2021-07-31T03:45:36.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: +0000 mgr.x (mgr.24542) 5074 : 2021-07-31T03:45:36.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: cephadm [ERR] 2021-07-31T03:45:36.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:45:36.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:45:36.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: yield 2021-07-31T03:45:36.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:45:36.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:45:36.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:45:36.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:45:36.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:45:36.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:45:36.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:45:36.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:45:36.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:45:36.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: task.result() 2021-07-31T03:45:36.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:45:36.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:45:35.019231+0000 mgr.x (mgr.24542) 5075 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:45:36.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:45:36.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:45:36.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: yield 2021-07-31T03:45:36.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:45:36.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:45:36.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:45:36.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:45:36.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:45:36.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:45:36.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:45:36.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:45:36.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:45:36.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: task.result() 2021-07-31T03:45:36.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:45:36.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: 2021-07-31T03:45:36.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:45:36.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: 2021-07-31T03:45:36.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:45:36.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:45:36.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:45:36.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:45:36.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:45:36.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:45:36.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:45:36.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:45:36.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:45:36.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:45:36.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:45:36.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:45:36.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:45:36.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:45:36.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:45:36.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:45:36.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:45:36.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:45:36.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: raise 2021-07-31T03:45:36.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:45:36.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:45:36.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:45:36.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:45:36.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:45:36.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:45:35.027696+0000 mgr.x (mgr.24542) 5076 : cephadm [ERR] 2021-07-31T03:45:36.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:45:36.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:45:36.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: yield 2021-07-31T03:45:36.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:45:36.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:45:36.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:45:36.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:45:36.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:45:36.029 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:45:36.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:45:36.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:45:36.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:45:36.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: task.result() 2021-07-31T03:45:36.031 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:45:36.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: audit 2021-07-31T03:45:34.935618+0000 mon.b (mon.2) 667 : 2021-07-31T03:45:36.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:45:36.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: audit 2021-07-31T03:45:34.936961+0000 mon.b (mon.2) 668 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:45:36.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: audit 2021-07-31T03:45:36.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: 2021-07-31T03:45:34.937871+0000 mon.b (mon.2) 669 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:45:36.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:45:36.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: 2021-07-31T03 2021-07-31T03:45:36.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: :45:34.938643+0000 mgr.x ( 2021-07-31T03:45:36.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: mgr.24542) 5069 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:45:36.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:45:36.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: 2021-07-31T03:45:34. 2021-07-31T03:45:36.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: 981566+0000 mgr.x (mgr.24542) 5070 : 2021-07-31T03:45:36.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: cephadm [ERR] 2021-07-31T03:45:36.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:45:36.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:45:36.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: yield 2021-07-31T03:45:36.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:45:36.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:45:36.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:45:36.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:45:36.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:45:36.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:45:36.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:45:36.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:45:36.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:45:36.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: task.result() 2021-07-31T03:45:36.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:45:36.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:45:34.982023+0000 mgr.x (mgr.24542) 5071 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:45:36.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: cephadm 2021 2021-07-31T03:45:36.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: -07-31T03:45:34.993428+0000 mgr.x 2021-07-31T03:45:36.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: (mgr.24542) 5072 : cephadm [ERR] 2021-07-31T03:45:36.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: 2021-07-31T03:45:36.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:45:36.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:45:36.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: yield 2021-07-31T03:45:36.040 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:45:36.040 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:45:36.040 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:45:36.040 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:45:36.041 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:45:35.028133+0000 mgr.x (mgr.24542) 5077 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:45:36.041 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:45:36.041 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:45:36.042 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: yield 2021-07-31T03:45:36.042 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:45:36.042 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:45:36.042 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:45:36.043 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:45:36.043 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:45:36.043 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:45:36.043 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:45:36.044 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:45:36.044 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:45:36.044 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: task.result() 2021-07-31T03:45:36.044 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:45:36.044 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: 2021-07-31T03:45:36.045 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:45:36.045 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: 2021-07-31T03:45:36.045 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:45:36.045 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:45:36.046 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:45:36.046 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:45:36.046 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:45:36.046 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:45:36.047 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:45:36.047 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:45:36.047 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:45:36.047 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:45:36.047 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:45:36.048 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:45:36.048 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:45:36.048 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:45:36.048 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:45:36.049 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:45:36.049 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:45:36.049 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:45:36.049 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: raise 2021-07-31T03:45:36.049 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:45:36.050 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:45:36.050 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:45:36.050 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:45:36.050 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:35 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:45:36.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: audit 2021-07-31T03:45:34.935618+0000 mon.b (mon.2 2021-07-31T03:45:36.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: ) 667 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:45:36.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: audit 2021-07-31T03:45:34.936961+0000 2021-07-31T03:45:36.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: mon.b (mon.2) 668 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:45:36.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: audit 2021-07-31T03:45 2021-07-31T03:45:36.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: :34.937871+0000 mon.b (mon.2) 669 : audit 2021-07-31T03:45:36.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:45:36.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:45:36.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: 2021-07-31T03 2021-07-31T03:45:36.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: :45:34.938643+0000 mgr.x (mgr.24542) 2021-07-31T03:45:36.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: 5069 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:45:36.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:45:36.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: 2021-07-31T03:45:34.981566+0000 mgr.x 2021-07-31T03:45:36.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: (mgr.24542) 5070 : cephadm [ERR] 2021-07-31T03:45:36.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:45:36.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:45:36.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: yield 2021-07-31T03:45:36.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:45:36.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:45:36.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:45:36.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:45:36.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:45:36.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:45:36.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:45:36.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:45:36.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:45:36.057 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: task.result() 2021-07-31T03:45:36.057 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:45:36.057 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:45:34.982023+0000 mgr.x (mgr.24542) 5071 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:45:36.057 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: cephadm 2021- 2021-07-31T03:45:36.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: 07-31T03:45:34.993428+0000 mgr.x (mgr.24542) 5072 : cephadm 2021-07-31T03:45:36.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: [ERR] 2021-07-31T03:45:36.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:45:36.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:45:36.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: yield 2021-07-31T03:45:36.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:45:36.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:45:36.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:45:36.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:45:36.060 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:45:36.060 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:45:36.060 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:45:36.061 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:45:36.061 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:45:36.061 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: task.result() 2021-07-31T03:45:36.061 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:45:36.061 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:45:34.993984+0000 mgr.x (mgr.24542) 5073 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:45:36.062 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:45:36.062 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: 2021-07-31T03:45:35 2021-07-31T03:45:36.062 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: .018781+0000 mgr.x (mgr. 2021-07-31T03:45:36.062 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: 24542) 5074 : cephadm [ERR] 2021-07-31T03:45:36.063 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:45:36.063 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:45:36.063 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: yield 2021-07-31T03:45:36.063 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:45:36.063 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:45:36.064 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:45:36.064 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:45:36.064 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:45:36.064 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:45:36.065 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:45:36.065 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:45:36.065 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:45:36.065 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: task.result() 2021-07-31T03:45:36.066 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:45:36.066 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: 2021-07-31T03:45:36.066 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:45:35.019231+0000 mgr.x (mgr.24542) 5075 : cephadm 2021-07-31T03:45:36.066 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:45:36.066 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:45:36.067 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:45:36.067 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: yield 2021-07-31T03:45:36.067 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:45:36.067 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:45:36.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:45:36.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:45:36.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:45:36.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:45:36.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:45:36.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:45:36.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:45:36.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: task.result() 2021-07-31T03:45:36.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:45:36.070 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:45:36.070 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:45:36.070 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:45:36.071 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:45:36.071 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:45:36.071 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: task.result() 2021-07-31T03:45:36.071 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:45:36.071 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:45:34.993984+0000 mgr.x (mgr.24542) 5073 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:45:36.072 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:45:35.018781+0000 2021-07-31T03:45:36.072 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: mgr.x (mgr.24542) 5074 : cephadm [ERR] 2021-07-31T03:45:36.072 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:45:36.072 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:45:36.073 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: yield 2021-07-31T03:45:36.073 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:45:36.073 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:45:36.073 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:45:36.074 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:45:36.074 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:45:36.074 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:45:36.074 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:45:36.074 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:45:36.075 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:45:36.075 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: task.result() 2021-07-31T03:45:36.075 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:45:36.075 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:45:35.019231+0000 mgr.x (mgr.24542) 5075 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:45:36.076 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:45:36.076 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:45:36.076 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: yield 2021-07-31T03:45:36.076 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:45:36.077 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:45:36.077 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:45:36.077 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:45:36.077 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:45:36.077 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:45:36.078 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:45:36.078 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:45:36.078 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:45:36.078 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: task.result() 2021-07-31T03:45:36.079 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:45:36.079 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: 2021-07-31T03:45:36.079 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:45:36.079 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: 2021-07-31T03:45:36.080 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: 2021-07-31T03:45:36.080 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:45:36.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: 2021-07-31T03:45:36.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:45:36.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:45:36.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:45:36.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:45:36.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:45:36.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:45:36.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:45:36.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:45:36.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:45:36.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:45:36.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:45:36.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:45:36.084 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:45:36.084 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:45:36.084 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:45:36.084 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:45:36.084 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:45:36.085 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:45:36.085 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: raise 2021-07-31T03:45:36.085 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:45:36.085 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:45:36.086 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:45:36.086 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:45:36.086 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:45:36.086 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:45:35.027696+0000 mgr.x (mgr.24542) 5076 : cephadm [ERR] 2021-07-31T03:45:36.087 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:45:36.087 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:45:36.087 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: yield 2021-07-31T03:45:36.087 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:45:36.087 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:45:36.088 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:45:36.088 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:45:36.088 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:45:36.088 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:45:36.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:45:36.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:45:36.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:45:36.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: task.result() 2021-07-31T03:45:36.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:45:36.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:45:35.028133+0000 mgr.x (mgr.24542) 5077 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:45:36.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:45:36.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:45:36.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: yield 2021-07-31T03:45:36.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:45:36.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:45:36.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:45:36.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:45:36.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:45:36.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:45:36.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:45:36.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:45:36.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:45:36.093 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: task.result() 2021-07-31T03:45:36.093 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:45:36.093 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: 2021-07-31T03:45:36.093 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:45:36.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: 2021-07-31T03:45:36.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:45:36.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:45:36.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:45:36.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:45:36.095 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:45:36.095 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:45:36.095 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:45:36.095 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:45:36.096 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:45:36.096 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:45:36.096 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:45:36.096 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:45:36.096 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:45:36.097 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:45:36.097 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:45:36.097 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:45:36.097 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:45:36.098 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:45:36.098 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: raise 2021-07-31T03:45:36.098 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:45:36.098 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:45:36.098 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:45:36.099 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:45:36.099 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:45:36.099 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:45:36.100 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:45:36.100 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:45:36.100 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:45:36.100 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:45:36.101 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:45:36.101 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:45:36.101 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:45:36.101 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:45:36.102 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:45:36.102 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:45:36.102 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:45:36.102 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:45:36.102 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:45:36.103 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:45:36.103 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:45:36.103 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:45:36.103 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:45:36.104 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: raise 2021-07-31T03:45:36.104 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:45:36.104 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:45:36.104 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:45:36.104 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:45:36.105 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:45:36.105 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:45:35.027696+0000 mgr.x (mgr.24542) 5076 : cephadm [ERR] 2021-07-31T03:45:36.105 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:45:36.105 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:45:36.106 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: yield 2021-07-31T03:45:36.106 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:45:36.106 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:45:36.106 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:45:36.107 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:45:36.107 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:45:36.107 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:45:36.107 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:45:36.107 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:45:36.108 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:45:36.108 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: task.result() 2021-07-31T03:45:36.108 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:45:36.108 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:45:35.028133+0000 mgr.x (mgr.24542) 5077 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:45:36.109 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:45:36.109 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:45:36.109 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: yield 2021-07-31T03:45:36.109 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:45:36.109 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:45:36.110 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:45:36.110 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:45:36.110 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:45:36.110 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:45:36.111 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:45:36.111 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:45:36.111 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:45:36.111 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: task.result() 2021-07-31T03:45:36.112 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:45:36.112 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: 2021-07-31T03:45:36.112 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:45:36.112 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: 2021-07-31T03:45:36.112 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:45:36.113 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:45:36.113 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:45:36.113 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:45:36.113 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:45:36.114 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:45:36.114 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:45:36.114 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:45:36.114 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:45:36.115 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:45:36.115 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:45:36.115 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:45:36.115 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:45:36.115 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:45:36.116 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:45:36.116 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:45:36.116 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:45:36.116 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:45:36.116 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: raise 2021-07-31T03:45:36.117 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:45:36.117 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:45:36.117 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:45:36.117 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:45:36.118 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:35 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:45:37.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:36 smithi154 conmon[34553]: cluster 2021-07-31T03:45:35.485333+0000 mgr.x (mgr.24542) 5078 : cluster [DBG] pgmap v5559: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:37.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:36 smithi045 conmon[39718]: cluster 2021-07-31T03:45:35.485333+0000 mgr.x (mgr 2021-07-31T03:45:37.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:36 smithi045 conmon[39718]: .24542) 5078 : cluster [DBG] pgmap v5559: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:37.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:36 smithi045 conmon[33655]: cluster 2021-07-31T03:45:35.485333 2021-07-31T03:45:37.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:36 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 5078 : cluster [DBG] pgmap v5559: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:39.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:38 smithi154 conmon[34553]: cluster 2021-07-31T03:45:37.485743+0000 mgr.x (mgr.24542) 5079 : cluster [DBG] pgmap v5560: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:39.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:38 smithi045 conmon[39718]: cluster 2021-07-31T03:45: 2021-07-31T03:45:39.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:38 smithi045 conmon[39718]: 37.485743+0000 mgr.x (mgr.24542) 5079 : cluster [DBG] pgmap v5560: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:39.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:38 smithi045 conmon[33655]: cluster 2021-07-31T03:45:37.485743+0000 mgr.x (mgr.24542) 5079 : cluster [DBG] pgmap v5560: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:41.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:40 smithi154 conmon[34553]: cluster 2021-07-31T03:45:39.486194+0000 mgr.x (mgr.24542) 5080 : cluster [DBG] pgmap v5561: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:41.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:40 smithi045 conmon[33655]: cluster 2021-07-31T03:45:39.486194+0000 mgr.x (mgr.24542) 5080 : cluster 2021-07-31T03:45:41.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:40 smithi045 conmon[33655]: [DBG] pgmap v5561: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:41.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:40 smithi045 conmon[39718]: cluster 2021-07-31T03:45:39.486194+0000 mgr.x 2021-07-31T03:45:41.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:40 smithi045 conmon[39718]: (mgr.24542) 5080 : cluster [DBG] pgmap v5561: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:43.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:42 smithi154 conmon[34553]: cluster 2021-07-31T03:45:41.486691+0000 mgr.x (mgr.24542) 5081 : cluster [DBG] pgmap v5562: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:43.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:42 smithi045 conmon[39718]: cluster 2021-07-31T03:45:41.486691+0000 mgr.x (mgr. 2021-07-31T03:45:43.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:42 smithi045 conmon[39718]: 24542) 5081 : cluster [DBG] pgmap v5562: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:43.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:42 smithi045 conmon[33655]: cluster 2021-07-31T03:45:41.486691+0000 mgr.x (mgr.24542) 5081 : cluster [DBG] pgmap v5562: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:43.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:42 smithi045 conmon[33655]: 2021-07-31T03:45:45.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:44 smithi154 conmon[34553]: cluster 2021-07-31T03:45:43.487226+0000 mgr.x (mgr.24542) 5082 : cluster [DBG] pgmap v5563: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:45.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:44 smithi045 conmon[33655]: cluster 2021-07-31T03:45:43.487226+0000 mgr.x (mgr.24542) 5082 : cluster 2021-07-31T03:45:45.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:44 smithi045 conmon[33655]: [DBG] pgmap v5563: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:45.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:44 smithi045 conmon[39718]: cluster 2021-07-31T03:45:43.487226+0000 mgr.x ( 2021-07-31T03:45:45.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:44 smithi045 conmon[39718]: mgr.24542) 5082 : cluster [DBG] pgmap v5563: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:47.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:46 smithi154 conmon[34553]: cluster 2021-07-31T03:45:45.487731+0000 mgr.x (mgr.24542) 5083 : cluster 2021-07-31T03:45:47.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:46 smithi154 conmon[34553]: [DBG] pgmap v5564: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:47.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:46 smithi045 conmon[33655]: cluster 2021-07-31T03:45:45.487731+0000 mgr.x (mgr.24542) 5083 : cluster [DBG] pgmap v5564: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:47.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:46 smithi045 conmon[33655]: 2021-07-31T03:45:47.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:46 smithi045 conmon[39718]: cluster 2021-07-31T03:45:45.487731+0000 mgr.x (mgr 2021-07-31T03:45:47.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:46 smithi045 conmon[39718]: .24542) 5083 : cluster [DBG] pgmap v5564: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:49.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:48 smithi154 conmon[34553]: cluster 2021-07-31T03:45:47.488242+0000 mgr.x (mgr.24542) 5084 : cluster [DBG] pgmap v5565: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:49.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:48 smithi045 conmon[33655]: cluster 2021-07-31T03:45:47.488242+0000 mgr.x (mgr.24542) 5084 : cluster [DBG] pgmap v5565: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:49.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:48 smithi045 conmon[39718]: cluster 2021-07-31T03:45:47.488242+0000 mgr.x (mgr.24542) 5084 : cluster 2021-07-31T03:45:49.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:48 smithi045 conmon[39718]: [DBG] pgmap v5565: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:51.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:50 smithi154 conmon[34553]: cluster 2021-07-31T03:45:49.488886+0000 mgr.x (mgr.24542) 5085 : cluster [DBG] pgmap v5566: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:51.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:50 smithi045 conmon[33655]: cluster 2021-07-31T03:45:49.488886+0000 mgr.x (mgr.24542) 5085 : cluster [DBG] 2021-07-31T03:45:51.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:50 smithi045 conmon[33655]: pgmap v5566: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:51.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:50 smithi045 conmon[39718]: cluster 2021-07-31T03:45:49.488886+0000 mgr.x (mgr.24542) 2021-07-31T03:45:51.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:50 smithi045 conmon[39718]: 5085 : cluster [DBG] pgmap v5566: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:53.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:52 smithi154 conmon[34553]: cluster 2021-07-31T03:45:51.489430+0000 mgr.x (mgr.24542) 5086 : cluster [DBG] pgmap v5567: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:53.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:52 smithi154 conmon[34553]: 2021-07-31T03:45:53.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:52 smithi045 conmon[33655]: cluster 2021-07-31T03:45:51.489430+0000 mgr.x (mgr.24542) 5086 : 2021-07-31T03:45:53.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:52 smithi045 conmon[33655]: cluster [DBG] pgmap v5567: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:53.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:52 smithi045 conmon[39718]: cluster 2021-07-31T03:45:51.489430+0000 2021-07-31T03:45:53.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:52 smithi045 conmon[39718]: mgr.x (mgr.24542) 5086 : cluster [DBG] pgmap v5567: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:55.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:54 smithi154 conmon[34553]: cluster 2021-07-31T03:45:53.489972+0000 mgr.x (mgr.24542) 5087 : cluster [DBG] pgmap v5568: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:55.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:54 smithi045 conmon[33655]: cluster 2021-07-31T03:45:53.489972+0000 mgr.x (mgr.24542) 5087 : cluster [DBG] 2021-07-31T03:45:55.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:54 smithi045 conmon[33655]: pgmap v5568: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:55.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:54 smithi045 conmon[39718]: cluster 2021-07-31T03:45:53.489972+0000 mgr.x (mgr.24542) 5087 : cluster [DBG] pgmap v5568: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:55.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:54 smithi045 conmon[39718]: 2021-07-31T03:45:57.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:56 smithi154 conmon[34553]: cluster 2021-07-31T03:45:55.490447+0000 mgr.x (mgr.24542) 5088 : cluster [DBG] pgmap v5569: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:57.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:56 smithi045 conmon[33655]: cluster 2021-07-31T03:45:55.490447+0000 mgr.x (mgr.24542) 5088 : cluster 2021-07-31T03:45:57.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:56 smithi045 conmon[33655]: [DBG] pgmap v5569: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:57.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:56 smithi045 conmon[39718]: cluster 2021-07-31T03:45:55.490447 2021-07-31T03:45:57.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:56 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 5088 : cluster [DBG] pgmap v5569: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:59.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:45:58 smithi154 conmon[34553]: cluster 2021-07-31T03:45:57.490831+0000 mgr.x (mgr.24542) 5089 : cluster [DBG] pgmap v5570: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:59.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:45:58 smithi045 conmon[39718]: cluster 2021-07-31T03:45:57.490831+0000 mgr.x (mgr.24542) 5089 : cluster [DBG] pgmap v5570: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:59.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:58 smithi045 conmon[33655]: cluster 2021-07-31T03:45:57.490831+0000 mgr.x (mgr.24542) 5089 : cluster [DBG] pgmap v5570: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:45:59.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:45:58 smithi045 conmon[33655]: 2021-07-31T03:46:01.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:00 smithi154 conmon[34553]: cluster 2021-07-31T03:45:59.491232+0000 mgr.x (mgr.24542) 5090 : cluster [DBG] pgmap v5571: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:46:01.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:00 smithi045 conmon[39718]: cluster 2021-07-31T03:45:59.491232+0000 mgr.x (mgr 2021-07-31T03:46:01.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:00 smithi045 conmon[39718]: .24542) 5090 : cluster [DBG] pgmap v5571: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:46:01.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:00 smithi045 conmon[33655]: cluster 2021-07-31T03:45:59.491232+0000 mgr.x (mgr.24542) 5090 2021-07-31T03:46:01.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:00 smithi045 conmon[33655]: : cluster [DBG] pgmap v5571: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:46:01.045 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:46:01.323 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:46:01.323 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 2h ago 2h - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:46:01.323 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 118m ago 2h 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:46:01.324 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 118m ago 2h 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:46:01.324 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 2h ago 2h - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:46:01.324 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:46:01.324 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 118m ago 2h 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:46:01.324 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:46:01.325 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 2h ago 2h - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:46:01.325 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 118m ago 2h 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:46:01.325 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:46:01.325 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:46:01.325 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:46:01.326 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:46:01.326 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 118m ago 2h 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:46:01.326 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 118m ago 2h 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:46:01.326 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 118m ago 2h 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:46:01.326 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 118m ago 2h 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:46:01.327 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 118m ago 2h 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:46:01.658 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:46:01.659 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:46:01.659 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:46:01.659 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:46:01.659 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:46:01.659 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:46:01.660 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:46:01.660 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:46:01.660 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:46:01.660 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:46:01.661 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:46:01.661 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:46:01.661 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:46:01.661 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:46:01.661 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:46:01.661 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:46:01.662 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:46:02.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:01 smithi154 conmon[34553]: audit 2021-07-31T03:46:01.033992+0000 mgr.x (mgr.24542) 5091 : audit [DBG] 2021-07-31T03:46:02.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:01 smithi154 conmon[34553]: from='client.20190 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:46:02.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:01 smithi154 conmon[34553]: audit 2021-07-31T03:46:01.659602+0000 mon.c (mon.1) 227 : audit [DBG] from='client.? 172.21.15.45:0/3674466187' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:46:02.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:01 smithi045 conmon[39718]: audit 2021-07-31T03 2021-07-31T03:46:02.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:01 smithi045 conmon[39718]: :46:01.033992+0000 mgr.x (mgr.24542) 5091 : audit [DBG] from='client.20190 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:46:02.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:01 smithi045 conmon[39718]: audit 2021 2021-07-31T03:46:02.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:01 smithi045 conmon[39718]: -07-31T03:46: 2021-07-31T03:46:02.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:01 smithi045 conmon[39718]: 01.659602+0000 mon.c (mon 2021-07-31T03:46:02.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:01 smithi045 conmon[39718]: .1) 227 : audit [DBG] 2021-07-31T03:46:02.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:01 smithi045 conmon[39718]: from='client.? 172.21.15.45:0/3674466187' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:46:02.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:01 smithi045 conmon[33655]: audit 2021-07-31T03:46:01.033992+0000 mgr.x (mgr.24542) 5091 : audit [DBG] from='client.20190 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:46:02.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:01 smithi045 conmon[33655]: 2021-07-31T03:46:02.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:01 smithi045 conmon[33655]: audit 2021-07-31T03:46:01.659602+0000 mon.c (mon.1) 227 : audit [DBG] from='client.? 172.21.15.45:0/3674466187' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:46:03.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:02 smithi154 conmon[34553]: audit 2021-07-31T03:46:01.315562+0000 mgr.x (mgr.24542) 5092 : audit [DBG] from='client.30136 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:46:03.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:02 smithi154 conmon[34553]: cluster 2021-07-31T03:46:01.491720+0000 mgr.x (mgr.24542) 5093 : cluster [DBG] pgmap v5572: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:46:03.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:02 smithi045 conmon[39718]: audit 2021-07-31T03:46:01.315562+0000 mgr.x ( 2021-07-31T03:46:03.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:02 smithi045 conmon[39718]: mgr.24542) 5092 : audit [DBG] from='client.30136 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:46:03.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:02 smithi045 conmon[39718]: cluster 2021-07-31T03:46 2021-07-31T03:46:03.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:02 smithi045 conmon[39718]: :01.491720+0000 mgr.x (mgr.24542) 5093 : 2021-07-31T03:46:03.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:02 smithi045 conmon[39718]: cluster [DBG] pgmap v5572: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:46:03.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:02 smithi045 conmon[33655]: audit 2021-07-31T03:46:01.315562+0000 mgr.x (mgr.24542) 5092 : audit [DBG] from='client.30136 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:46:03.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:02 smithi045 conmon[33655]: cluster 2021-07-31T03:46:01.491720+0000 mgr.x (mgr.24542) 5093 : cluster [DBG] pgmap v5572: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:46:05.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:04 smithi154 conmon[34553]: cluster 2021-07-31T03:46:03.492137+0000 mgr.x (mgr.24542) 5094 : cluster [DBG] 2021-07-31T03:46:05.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:04 smithi154 conmon[34553]: pgmap v5573: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:46:05.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:04 smithi045 conmon[39718]: cluster 2021-07-31T03:46:03.492137+0000 mgr.x (mgr.24542) 5094 : cluster [DBG] pgmap v5573: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:46:05.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:04 smithi045 conmon[33655]: cluster 2021-07-31T03:46:03.492137+0000 mgr.x (mgr.24542) 5094 : cluster [DBG] pgmap v5573: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:46:07.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:06 smithi154 conmon[34553]: cluster 2021-07-31T03:46:05.492595+0000 mgr.x (mgr.24542) 5095 : cluster [DBG] pgmap v5574: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:46:07.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:06 smithi045 conmon[33655]: cluster 2021-07-31T03:46:05.492595+0000 mgr.x (mgr.24542) 5095 : cluster [DBG] pgmap v5574: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:46:07.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:06 smithi045 conmon[33655]: 2021-07-31T03:46:07.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:06 smithi045 conmon[39718]: cluster 2021-07-31T03:46:05.492595+0000 mgr.x (mgr.24542) 5095 : cluster [DBG] pgmap v5574: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:46:09.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:08 smithi154 conmon[34553]: cluster 2021-07-31T03:46:07.493056+0000 mgr.x (mgr.24542) 5096 : cluster [DBG] 2021-07-31T03:46:09.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:08 smithi154 conmon[34553]: pgmap v5575: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:46:09.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:08 smithi045 conmon[33655]: cluster 2021-07-31T03:46:07.493056+0000 mgr.x (mgr.24542) 5096 : cluster [DBG] pgmap v5575: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:46:09.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:08 smithi045 conmon[39718]: cluster 2021-07-31T03:46:07.493056+0000 2021-07-31T03:46:09.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:08 smithi045 conmon[39718]: mgr.x (mgr.24542) 5096 : cluster [DBG] pgmap v5575: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:46:11.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:10 smithi154 conmon[34553]: cluster 2021-07-31T03:46:09.493650+0000 mgr.x (mgr.24542) 5097 : cluster [DBG] pgmap v5576: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:46:11.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:10 smithi045 conmon[39718]: cluster 2021-07-31T03:46:09.493650+0000 mgr.x 2021-07-31T03:46:11.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:10 smithi045 conmon[39718]: (mgr.24542) 5097 : cluster [DBG] pgmap v5576: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:46:11.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:10 smithi045 conmon[33655]: cluster 2021-07-31T03:46:09.493650 2021-07-31T03:46:11.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:10 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 5097 : cluster [DBG] pgmap v5576: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:46:13.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:12 smithi154 conmon[34553]: cluster 2021-07-31T03:46:11.494051+0000 mgr.x (mgr.24542) 5098 : cluster [DBG] pgmap v5577: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:46:13.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:12 smithi045 conmon[39718]: cluster 2021-07-31T03:46:11.494051+0000 mgr.x (mgr 2021-07-31T03:46:13.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:12 smithi045 conmon[39718]: .24542) 5098 : cluster [DBG] pgmap v5577: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:46:13.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:12 smithi045 conmon[33655]: cluster 2021-07-31T03:46:11.494051+0000 mgr.x (mgr.24542) 5098 : cluster [DBG] 2021-07-31T03:46:13.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:12 smithi045 conmon[33655]: pgmap v5577: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:46:15.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:14 smithi154 conmon[34553]: cluster 2021-07-31T03:46:13.494427+0000 mgr.x (mgr.24542) 5099 : cluster [DBG] pgmap v5578: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:46:15.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:14 smithi045 conmon[39718]: cluster 2021-07-31T03:46:13.494427+0000 mgr.x (mgr. 2021-07-31T03:46:15.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:14 smithi045 conmon[39718]: 24542) 5099 : cluster [DBG] pgmap v5578: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:46:15.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:14 smithi045 conmon[33655]: cluster 2021-07-31T03:46:13.494427+0000 mgr.x (mgr.24542) 5099 : cluster [DBG] 2021-07-31T03:46:15.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:14 smithi045 conmon[33655]: pgmap v5578: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:46:17.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:16 smithi154 conmon[34553]: cluster 2021-07-31T03:46:15.495066+0000 mgr.x (mgr.24542) 5100 : cluster [DBG] pgmap v5579: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:46:17.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:16 smithi045 conmon[39718]: cluster 2021-07-31T03:46:15.495066+0000 mgr.x (mgr.24542) 2021-07-31T03:46:17.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:16 smithi045 conmon[39718]: 5100 : cluster [DBG] pgmap v5579: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:46:17.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:16 smithi045 conmon[33655]: cluster 2021-07-31T03:46:15.495066+0000 mgr.x (mgr.24542) 5100 : cluster [DBG] 2021-07-31T03:46:17.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:16 smithi045 conmon[33655]: pgmap v5579: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:46:19.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:18 smithi154 conmon[34553]: cluster 2021-07-31T03:46:17.495462+0000 mgr.x (mgr.24542) 5101 : cluster [DBG] pgmap v5580: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:46:19.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:18 smithi045 conmon[39718]: cluster 2021-07-31T03:46:17.495462+0000 mgr.x (mgr 2021-07-31T03:46:19.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:18 smithi045 conmon[39718]: .24542) 5101 : cluster [DBG] pgmap v5580: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:46:19.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:18 smithi045 conmon[33655]: cluster 2021-07-31T03:46:17.495462+0000 mgr.x (mgr.24542) 5101 : cluster [DBG] 2021-07-31T03:46:19.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:18 smithi045 conmon[33655]: pgmap v5580: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:46:21.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:20 smithi154 conmon[34553]: cluster 2021-07-31T03:46:19.495896+0000 mgr.x (mgr.24542) 5102 : cluster [DBG] pgmap v5581: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:46:21.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:20 smithi045 conmon[39718]: cluster 2021-07-31T03:46:19.495896+0000 mgr.x 2021-07-31T03:46:21.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:20 smithi045 conmon[39718]: (mgr.24542) 5102 : cluster [DBG] pgmap v5581: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:46:21.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:20 smithi045 conmon[33655]: cluster 2021-07-31T03:46:19.495896+0000 mgr.x (mgr. 2021-07-31T03:46:21.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:20 smithi045 conmon[33655]: 24542) 5102 : cluster [DBG] pgmap v5581: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:46:23.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:22 smithi045 conmon[39718]: cluster 2021-07-31T03:46:21.496452+0000 mgr.x (mgr.24542) 5103 : 2021-07-31T03:46:23.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:22 smithi045 conmon[39718]: cluster [DBG] pgmap v5582: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:46:23.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:22 smithi045 conmon[33655]: cluster 2021-07-31T03:46:21.496452+0000 mgr.x (mgr.24542) 5103 : cluster [DBG] 2021-07-31T03:46:23.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:22 smithi045 conmon[33655]: pgmap v5582: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:46:23.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:22 smithi154 conmon[34553]: cluster 2021-07-31T03:46:21.496452+0000 mgr.x (mgr.24542) 5103 : cluster [DBG] pgmap v5582: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:46:25.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:24 smithi045 conmon[33655]: cluster 2021-07-31T03:46:23.496956+0000 mgr.x (mgr.24542) 5104 : 2021-07-31T03:46:25.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:24 smithi045 conmon[33655]: cluster [DBG] pgmap v5583: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:46:25.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:24 smithi045 conmon[39718]: cluster 2021-07-31T03:46:23.496956+0000 mgr.x (mgr 2021-07-31T03:46:25.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:24 smithi045 conmon[39718]: .24542) 5104 : cluster [DBG] pgmap v5583: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:46:25.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:24 smithi154 conmon[34553]: cluster 2021-07-31T03:46:23.496956+0000 mgr.x (mgr.24542) 5104 : cluster [DBG] pgmap v5583: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:46:27.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:26 smithi045 conmon[33655]: cluster 2021-07-31T03:46:25.497390+0000 mgr.x (mgr.24542) 5105 : cluster 2021-07-31T03:46:27.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:26 smithi045 conmon[33655]: [DBG] pgmap v5584: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:46:27.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:26 smithi045 conmon[39718]: cluster 2021-07-31T03:46:25.497390+0000 mgr.x 2021-07-31T03:46:27.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:26 smithi045 conmon[39718]: (mgr.24542) 5105 : cluster [DBG] pgmap v5584: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:46:27.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:26 smithi154 conmon[34553]: cluster 2021-07-31T03:46:25.497390+0000 mgr.x (mgr.24542) 5105 : cluster [DBG] pgmap v5584: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:46:29.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:28 smithi045 conmon[39718]: cluster 2021-07-31T03:46:27.497865+0000 mgr.x (mgr.24542) 5106 : cluster 2021-07-31T03:46:29.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:28 smithi045 conmon[39718]: [DBG] pgmap v5585: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:46:29.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:28 smithi045 conmon[39718]: audit 2021-07-31T03:46: 2021-07-31T03:46:29.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:28 smithi045 conmon[39718]: 28.408753+0000 mon.b (mon.2) 670 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:46:29.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:28 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T03:46:29.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:28 smithi045 conmon[39718]: 03:46:28.409026+0000 mon.b (mon.2) 671 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:46:29.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:28 smithi045 conmon[39718]: 2021-07-31T03:46:29.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:28 smithi045 conmon[39718]: audit 2021-07-31T03: 2021-07-31T03:46:29.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:28 smithi045 conmon[39718]: 46:28.409663+0000 mon.a (mon.0) 3362 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:46:29.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:28 smithi045 conmon[39718]: audit 2021-07-31T03:46:28.410066+0000 2021-07-31T03:46:29.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:28 smithi045 conmon[39718]: mon.a (mon.0) 3363 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:46:29.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:28 smithi045 conmon[33655]: cluster 2021-07-31T03:46:27.497865+0000 mgr.x (mgr.24542) 5106 2021-07-31T03:46:29.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:28 smithi045 conmon[33655]: : cluster [DBG] pgmap v5585: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:46:29.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:28 smithi045 conmon[33655]: audit 2021-07-31T03:46:28.408753+0000 mon.b (mon.2) 670 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:46:29.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:28 smithi045 conmon[33655]: audit 2021-07-31T03 2021-07-31T03:46:29.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:28 smithi045 conmon[33655]: :46:28.409026+0000 mon.b (mon.2) 671 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:46:29.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:28 smithi045 conmon[33655]: audit 2021-07-31T03:46:29.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:28 smithi045 conmon[33655]: 2021-07 2021-07-31T03:46:29.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:28 smithi045 conmon[33655]: -31T03:46:28.409663+0000 2021-07-31T03:46:29.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:28 smithi045 conmon[33655]: mon.a (mon.0) 3362 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:46:29.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:28 smithi045 conmon[33655]: audit 2021-07-31T03:46:28.410066 2021-07-31T03:46:29.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:28 smithi045 conmon[33655]: +0000 mon.a (mon.0) 3363 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:46:29.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:28 smithi154 conmon[34553]: cluster 2021-07-31T03:46:27.497865+0000 mgr.x (mgr.24542) 5106 : cluster [DBG] pgmap v5585: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:46:29.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:28 smithi154 conmon[34553]: audit 2021-07-31T03:46:28.408753+0000 mon.b (mon.2) 670 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:46:29.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:28 smithi154 conmon[34553]: audit 2021-07-31T03:46:28.409026+0000 mon.b (mon.2) 671 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:46:29.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:28 smithi154 conmon[34553]: audit 2021-07-31T03:46:28.409663+0000 mon.a 2021-07-31T03:46:29.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:28 smithi154 conmon[34553]: (mon.0) 3362 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:46:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:28 smithi154 conmon[34553]: audit 2021-07 2021-07-31T03:46:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:28 smithi154 conmon[34553]: -31T03:46:28.410066+0000 mon.a ( 2021-07-31T03:46:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:28 smithi154 conmon[34553]: mon.0) 3363 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:46:31.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:30 smithi045 conmon[39718]: cluster 2021-07-31T03:46:29.498240+0000 mgr.x (mgr. 2021-07-31T03:46:31.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:30 smithi045 conmon[39718]: 24542) 5107 : cluster [DBG] pgmap v5586: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:46:31.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:30 smithi045 conmon[33655]: cluster 2021-07-31T03:46:29.498240+0000 mgr.x (mgr.24542) 5107 : 2021-07-31T03:46:31.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:30 smithi045 conmon[33655]: cluster [DBG] pgmap v5586: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:46:31.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:30 smithi154 conmon[34553]: cluster 2021-07-31T03:46:29.498240+0000 mgr.x (mgr.24542 2021-07-31T03:46:31.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:30 smithi154 conmon[34553]: ) 5107 : cluster [DBG] pgmap v5586: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:46:31.971 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:46:32.248 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:46:32.249 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 2h ago 2h - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:46:32.249 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 119m ago 2h 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:46:32.249 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 119m ago 2h 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:46:32.249 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 2h ago 2h - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:46:32.249 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:46:32.250 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 119m ago 2h 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:46:32.250 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:46:32.250 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 2h ago 2h - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:46:32.250 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 119m ago 2h 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:46:32.251 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:46:32.251 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:46:32.251 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:46:32.251 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:46:32.251 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 119m ago 2h 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:46:32.252 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 119m ago 2h 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:46:32.252 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 119m ago 2h 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:46:32.252 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 119m ago 2h 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:46:32.252 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 119m ago 2h 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:46:32.584 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:46:32.584 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:46:32.584 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:46:32.585 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:46:32.585 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:46:32.585 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:46:32.585 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:46:32.586 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:46:32.586 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:46:32.586 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:46:32.586 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:46:32.587 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:46:32.587 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:46:32.587 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:46:32.587 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:46:32.587 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:46:32.588 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:46:33.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:32 smithi045 conmon[33655]: cluster 2021-07-31T03:46:31.498690+0000 mgr.x (mgr.24542) 2021-07-31T03:46:33.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:32 smithi045 conmon[33655]: 5108 : cluster [DBG] pgmap v5587: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:46:33.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:32 smithi045 conmon[33655]: audit 2021-07-31T03:46:31.959977+0000 mgr.x (mgr.24542) 5109 : audit [DBG] from='client.30148 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:46:33.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:32 smithi045 conmon[33655]: audit 2021-07-31T03:46 2021-07-31T03:46:33.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:32 smithi045 conmon[33655]: :32.584905+0000 mon.a ( 2021-07-31T03:46:33.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:32 smithi045 conmon[33655]: mon.0) 3364 : audit 2021-07-31T03:46:33.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:32 smithi045 conmon[33655]: [DBG] from='client.? 172.21.15.45:0/2144269740' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:46:33.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:32 smithi045 conmon[39718]: cluster 2021-07-31T03:46:31.498690+0000 2021-07-31T03:46:33.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:32 smithi045 conmon[39718]: mgr.x (mgr.24542) 5108 : cluster [DBG] pgmap v5587: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:46:33.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:32 smithi045 conmon[39718]: audit 2021-07-31T03:46:33.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:32 smithi045 conmon[39718]: 2021-07-31T03:46: 2021-07-31T03:46:33.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:32 smithi045 conmon[39718]: 31.959977+0000 mgr.x (mgr.24542) 2021-07-31T03:46:33.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:32 smithi045 conmon[39718]: 5109 : audit [DBG] from='client.30148 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:46:33.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:32 smithi045 conmon[39718]: audit 2021-07-31T03:46:33.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:32 smithi045 conmon[39718]: 2021-07-31T03: 2021-07-31T03:46:33.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:32 smithi045 conmon[39718]: 46:32.584905+0000 mon.a ( 2021-07-31T03:46:33.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:32 smithi045 conmon[39718]: mon.0) 3364 : audit [DBG] from='client.? 172.21.15.45:0/2144269740' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:46:33.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:32 smithi045 conmon[39718]: 2021-07-31T03:46:33.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:32 smithi154 conmon[34553]: cluster 2021-07-31T03:46:31.498690+0000 mgr.x (mgr.24542) 5108 : cluster [DBG] pgmap v5587: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:46:33.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:32 smithi154 conmon[34553]: audit 2021-07-31T03:46:31.959977+0000 mgr.x (mgr.24542) 5109 : audit [DBG] from='client.30148 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:46:33.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:32 smithi154 conmon[34553]: audit 2021-07-31T03:46:32.584905+0000 mon.a (mon.0) 3364 : audit [DBG] from='client.? 172.21.15.45:0/2144269740' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:46:34.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:33 smithi154 conmon[34553]: audit 2021-07-31T03:46:32.241336+0000 mgr.x (mgr.24542) 5110 : audit [DBG] from='client.20214 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:46:34.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:33 smithi045 conmon[33655]: audit 2021-07-31T03:46:32.241336+0000 mgr.x (mgr. 2021-07-31T03:46:34.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:33 smithi045 conmon[33655]: 24542) 5110 : audit [DBG] from='client.20214 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:46:34.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:33 smithi045 conmon[39718]: audit 2021-07-31T03:46:32.241336+0000 mgr.x 2021-07-31T03:46:34.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:33 smithi045 conmon[39718]: (mgr.24542) 5110 : audit [DBG] from='client.20214 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:46:35.055 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:34 smithi154 conmon[34553]: cluster 2021-07-31T03:46:33.499139+0000 mgr.x (mgr.24542) 5111 : cluster [DBG] pgmap v5588: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:46:35.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:34 smithi045 conmon[39718]: cluster 2021-07-31T03:46:33.499139+0000 mgr.x (mgr 2021-07-31T03:46:35.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:34 smithi045 conmon[39718]: .24542) 5111 : cluster [DBG] pgmap v5588: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:46:35.056 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:34 smithi045 conmon[33655]: cluster 2021-07-31T03:46:33.499139+0000 mgr.x (mgr.24542) 5111 : cluster [DBG] pgmap v5588: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:46:35.503 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: debug 2021-07-31T03:46:35.074+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:46:35.503 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:46:35.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:46:35.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: yield 2021-07-31T03:46:35.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:46:35.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:46:35.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:46:35.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:46:35.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:46:35.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:46:35.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:46:35.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:46:35.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:46:35.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: task.result() 2021-07-31T03:46:35.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:46:35.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: debug 2021-07-31T03:46:35.085+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:46:35.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:46:35.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:46:35.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: yield 2021-07-31T03:46:35.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:46:35.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:46:35.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:46:35.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:46:35.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:46:35.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:46:35.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:46:35.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:46:35.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:46:35.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: task.result() 2021-07-31T03:46:35.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:46:35.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: debug 2021-07-31T03:46:35.110+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:46:35.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:46:35.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:46:35.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: yield 2021-07-31T03:46:35.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:46:35.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:46:35.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:46:35.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:46:35.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:46:35.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:46:35.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:46:35.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:46:35.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:46:35.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: task.result() 2021-07-31T03:46:35.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:46:35.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: debug 2021-07-31T03:46:35.111+0000 7fcb97969700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:46:35.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:46:35.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:46:35.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: yield 2021-07-31T03:46:35.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:46:35.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:46:35.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:46:35.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:46:35.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:46:35.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:46:35.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:46:35.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:46:35.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:46:35.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: task.result() 2021-07-31T03:46:35.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:46:35.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: 2021-07-31T03:46:35.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:46:35.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: 2021-07-31T03:46:35.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:46:35.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:46:35.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:46:35.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:46:35.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:46:35.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:46:35.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:46:35.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:46:35.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:46:35.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:46:35.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:46:35.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:46:35.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:46:35.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:46:35.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:46:35.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:46:35.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:46:35.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:46:35.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: raise 2021-07-31T03:46:35.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:46:35.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:46:35.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:46:35.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:46:35.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:46:35.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: debug 2021-07-31T03:46:35.119+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:46:35.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:46:35.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:46:35.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: yield 2021-07-31T03:46:35.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:46:35.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:46:35.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:46:35.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:46:35.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:46:35.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:46:35.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:46:35.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:46:35.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:46:35.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: task.result() 2021-07-31T03:46:35.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:46:35.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: debug 2021-07-31T03:46:35.119+0000 7fcb9816a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:46:35.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:46:35.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:46:35.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: yield 2021-07-31T03:46:35.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:46:35.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:46:35.535 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:46:35.535 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:46:35.535 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:46:35.535 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:46:35.536 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:46:35.536 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:46:35.536 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:46:35.537 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: task.result() 2021-07-31T03:46:35.537 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:46:35.537 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: 2021-07-31T03:46:35.537 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:46:35.538 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: 2021-07-31T03:46:35.538 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:46:35.538 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:46:35.538 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:46:35.539 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:46:35.539 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:46:35.539 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:46:35.539 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:46:35.540 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:46:35.540 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:46:35.540 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:46:35.540 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:46:35.541 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:46:35.541 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:46:35.541 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:46:35.541 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:46:35.542 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:46:35.542 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:46:35.542 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:46:35.543 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: raise 2021-07-31T03:46:35.543 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:46:35.543 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:46:35.543 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:46:35.544 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:46:35.544 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:46:36.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: audit 2021-07-31T03:46:35.028944+0000 mon.b (mon.2) 672 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:46:36.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: audit 2021-07-31T03:46:35.030412+0000 mon.b (mon.2) 673 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:46:36.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: audit 2021-07-31T03:46:35.031528+0000 2021-07-31T03:46:36.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: mon.b (mon.2) 674 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:46:36.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:46:35.032288+0000 mgr.x (mgr.24542) 5112 2021-07-31T03:46:36.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:46:36.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: cephadm 2021 2021-07-31T03:46:36.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: -07-31T03:46:35 2021-07-31T03:46:36.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: .074978+0000 mgr.x (mgr.24542 2021-07-31T03:46:36.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: ) 5113 : cephadm [ERR] 2021-07-31T03:46:36.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:46:36.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:46:36.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: yield 2021-07-31T03:46:36.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:46:36.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:46:36.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:46:36.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:46:36.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:46:36.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:46:36.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:46:36.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:46:36.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:46:36.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: task.result() 2021-07-31T03:46:36.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:46:36.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:46:35.075387+0000 mgr.x (mgr.24542) 5114 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:46:36.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: cephadm 2021-07 2021-07-31T03:46:36.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: -31T03:46:35.086383+0000 mgr.x (mgr. 2021-07-31T03:46:36.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: 24542) 5115 : cephadm [ERR] 2021-07-31T03:46:36.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:46:36.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:46:36.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: yield 2021-07-31T03:46:36.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:46:36.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:46:36.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:46:36.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:46:36.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:46:36.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:46:36.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:46:36.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:46:36.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:46:36.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: task.result() 2021-07-31T03:46:36.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:46:36.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:46:35.087279+0000 mgr.x (mgr.24542) 5116 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:46:36.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: cephadm 2021-07-31 2021-07-31T03:46:36.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: T03:46:35.111516+0000 mgr.x (mgr.24542) 2021-07-31T03:46:36.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: 5117 : cephadm [ERR] 2021-07-31T03:46:36.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:46:36.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:46:36.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: yield 2021-07-31T03:46:36.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:46:36.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:46:36.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:46:36.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:46:36.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:46:36.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:46:36.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:46:36.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:46:36.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:46:36.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: task.result() 2021-07-31T03:46:36.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:46:36.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:46:35.111951+0000 mgr.x (mgr.24542) 5118 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:46:36.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:46:36.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:46:36.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: yield 2021-07-31T03:46:36.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:46:36.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:46:36.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:46:36.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:46:36.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:46:36.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:46:36.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:46:36.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:46:36.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:46:36.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: task.result() 2021-07-31T03:46:36.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:46:36.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: 2021-07-31T03:46:36.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:46:36.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: 2021-07-31T03:46:36.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:46:36.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:46:36.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:46:36.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:46:36.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:46:36.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:46:36.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:46:36.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:46:36.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:46:36.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:46:36.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:46:36.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:46:36.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:46:36.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:46:36.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:46:36.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:46:36.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:46:36.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:46:36.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: raise 2021-07-31T03:46:36.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:46:36.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:46:36.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:46:36.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:46:36.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:46:36.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:46:35.120320+0000 mgr.x (mgr.24542) 5119 : cephadm [ERR] 2021-07-31T03:46:36.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:46:36.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:46:36.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: yield 2021-07-31T03:46:36.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:46:36.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:46:36.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:46:36.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:46:36.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:46:36.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:46:36.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:46:36.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:46:36.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:46:36.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: task.result() 2021-07-31T03:46:36.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:46:36.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:46:35.120692+0000 mgr.x (mgr.24542) 5120 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:46:36.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:46:36.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:46:36.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: yield 2021-07-31T03:46:36.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:46:36.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:46:36.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:46:36.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:46:36.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:46:36.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:46:36.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:46:36.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:46:36.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:46:36.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: task.result() 2021-07-31T03:46:36.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:46:36.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: 2021-07-31T03:46:36.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:46:36.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: 2021-07-31T03:46:36.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:46:36.285 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:46:36.285 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:46:36.285 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:46:36.285 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:46:36.285 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:46:36.286 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:46:36.286 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:46:36.286 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:46:36.286 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:46:36.287 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:46:36.287 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:46:36.287 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:46:36.287 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:46:36.288 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:46:36.288 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:46:36.288 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:46:36.288 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:46:36.288 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: raise 2021-07-31T03:46:36.289 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:46:36.289 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:46:36.289 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:46:36.289 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:46:36.289 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:35 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:46:36.290 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: audit 2021-07-31T03:46:35.028944+0000 mon.b (mon.2) 672 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:46:36.290 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: 2021-07-31T03:46:36.291 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: audit 2021-07-31T03:46:35.030412+0000 mon.b (mon.2) 673 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:46:36.291 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: audit 2021-07-31T03:46:36.291 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: 2021-07-31T03:46:35.031528+0000 mon.b (mon.2) 674 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:46:36.291 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:46:35.032288 2021-07-31T03:46:36.292 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 5112 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:46:36.292 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: cephadm 2021-07-31T03 2021-07-31T03:46:36.292 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: :46:35.074978+0000 mgr.x (mgr.24542) 5113 : cephadm [ERR] 2021-07-31T03:46:36.292 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:46:36.292 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:46:36.293 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: yield 2021-07-31T03:46:36.293 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:46:36.293 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:46:36.293 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:46:36.293 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:46:36.294 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:46:36.294 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:46:36.294 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:46:36.294 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:46:36.295 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:46:36.295 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: task.result() 2021-07-31T03:46:36.295 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:46:36.295 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:46:35.075387+0000 mgr.x (mgr.24542) 5114 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:46:36.296 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:46:35.086383+0000 mgr.x (mgr.24542) 5115 : cephadm [ERR] 2021-07-31T03:46:36.296 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:46:36.296 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:46:36.296 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: yield 2021-07-31T03:46:36.296 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:46:36.297 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:46:36.297 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:46:36.297 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:46:36.297 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:46:36.297 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:46:36.298 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:46:36.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: audit 2021-07-31T03:46:35.028944+0000 mon.b (mon.2) 672 : audit [DBG] 2021-07-31T03:46:36.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:46:36.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: audit 2021-07-31T03:46:35.030412+0000 mon.b (mon.2) 673 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:46:36.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: audit 2021-07-31T03:46:35.031528+0000 mon.b (mon.2) 674 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:46:36.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:46 2021-07-31T03:46:36.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: :35.032288+0000 mgr.x (mgr.24542) 5112 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:46:36.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: cephadm 2021-07-31T03 2021-07-31T03:46:36.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: :46:35.074978+0000 mgr.x (mgr.24542) 5113 : cephadm [ERR] 2021-07-31T03:46:36.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: 2021-07-31T03:46:36.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:46:36.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:46:36.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: yield 2021-07-31T03:46:36.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:46:36.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:46:36.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:46:36.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:46:36.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:46:36.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:46:36.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:46:36.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:46:36.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:46:36.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: task.result() 2021-07-31T03:46:36.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:46:36.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: cephadm 2021 2021-07-31T03:46:36.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: -07-31T03:46:35.075387+0000 mgr.x (mgr.24542) 5114 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:46:36.304 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:46:35.086383+0000 mgr.x (mgr.24542) 5115 : cephadm [ERR] 2021-07-31T03:46:36.304 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:46:36.304 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:46:36.304 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: yield 2021-07-31T03:46:36.305 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:46:36.305 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:46:36.305 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:46:36.305 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:46:36.305 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:46:36.306 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:46:36.306 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:46:36.307 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:46:36.307 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:46:36.307 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: task.result() 2021-07-31T03:46:36.307 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:46:36.307 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:46:35.087279+0000 mgr.x (mgr.24542) 5116 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:46:36.308 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:46:35.111516+0000 mgr.x (mgr.24542) 5117 : cephadm [ERR] 2021-07-31T03:46:36.308 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:46:36.308 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:46:36.308 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: yield 2021-07-31T03:46:36.309 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:46:36.309 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:46:36.309 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:46:36.309 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:46:36.309 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:46:36.310 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:46:36.310 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:46:36.310 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:46:36.310 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:46:36.310 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: task.result() 2021-07-31T03:46:36.311 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:46:36.311 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:46:35.111951+0000 mgr.x (mgr.24542) 2021-07-31T03:46:36.311 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: 5118 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:46:36.311 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:46:36.312 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:46:36.312 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: yield 2021-07-31T03:46:36.312 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:46:36.312 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:46:36.312 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:46:36.313 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:46:36.313 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:46:36.313 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:46:36.313 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:46:36.314 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:46:36.314 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:46:36.314 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: task.result() 2021-07-31T03:46:36.314 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:46:36.314 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: 2021-07-31T03:46:36.315 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:46:36.315 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: 2021-07-31T03:46:36.315 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:46:36.315 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:46:36.315 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:46:36.316 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:46:36.316 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:46:36.316 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:46:36.316 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:46:36.317 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:46:36.317 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:46:36.317 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:46:36.317 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:46:36.317 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:46:36.318 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:46:36.318 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:46:36.318 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:46:36.318 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:46:36.318 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:46:36.319 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:46:36.319 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: raise 2021-07-31T03:46:36.319 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:46:36.319 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:46:36.319 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:46:36.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:46:36.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:46:36.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:46:35.120320+0000 mgr.x (mgr.24542) 5119 : cephadm [ERR] 2021-07-31T03:46:36.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:46:36.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:46:36.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: yield 2021-07-31T03:46:36.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:46:36.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:46:36.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:46:36.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:46:36.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:46:36.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:46:36.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:46:36.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:46:36.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:46:36.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: task.result() 2021-07-31T03:46:36.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:46:36.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:46:35.120692+0000 mgr.x (mgr.24542) 5120 : cephadm [ERR] 2021-07-31T03:46:36.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:46:36.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:46:36.324 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:46:36.324 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:46:36.324 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: task.result() 2021-07-31T03:46:36.325 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:46:36.325 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:46:35.087279+0000 mgr.x (mgr.24542) 5116 2021-07-31T03:46:36.325 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:46:36.325 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:46:35.111516+0000 mgr.x (mgr.24542) 5117 : cephadm [ERR] 2021-07-31T03:46:36.326 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:46:36.326 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:46:36.326 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: yield 2021-07-31T03:46:36.326 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:46:36.326 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:46:36.327 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:46:36.327 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:46:36.327 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:46:36.327 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:46:36.327 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:46:36.328 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:46:36.328 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:46:36.328 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: task.result() 2021-07-31T03:46:36.328 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:46:36.328 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:46:35.111951+0000 mgr.x (mgr.24542) 5118 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:46:36.329 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:46:36.329 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:46:36.329 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: yield 2021-07-31T03:46:36.329 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:46:36.330 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:46:36.330 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:46:36.330 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:46:36.330 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:46:36.330 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:46:36.331 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:46:36.331 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:46:36.331 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:46:36.331 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: task.result() 2021-07-31T03:46:36.331 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:46:36.332 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: 2021-07-31T03:46:36.332 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:46:36.332 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: 2021-07-31T03:46:36.332 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:46:36.332 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:46:36.333 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:46:36.333 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:46:36.333 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: yield 2021-07-31T03:46:36.333 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:46:36.334 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:46:36.334 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:46:36.334 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:46:36.334 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:46:36.335 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:46:36.335 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:46:36.335 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:46:36.335 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:46:36.335 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: task.result() 2021-07-31T03:46:36.336 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:46:36.336 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: 2021-07-31T03:46:36.336 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:46:36.336 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: 2021-07-31T03:46:36.336 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:46:36.337 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:46:36.337 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:46:36.337 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:46:36.337 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:46:36.337 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:46:36.338 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:46:36.338 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:46:36.338 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:46:36.338 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:46:36.339 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:46:36.339 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:46:36.339 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:46:36.339 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:46:36.339 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:46:36.340 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:46:36.340 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:46:36.340 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:46:36.340 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: raise 2021-07-31T03:46:36.340 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:46:36.341 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:46:36.341 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:46:36.341 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:46:36.341 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:46:36.342 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:46:36.342 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:46:36.342 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:46:36.342 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:46:36.343 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:46:36.343 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:46:36.343 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:46:36.343 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:46:36.343 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:46:36.344 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:46:36.344 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:46:36.344 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:46:36.344 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:46:36.344 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:46:36.345 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:46:36.345 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: raise 2021-07-31T03:46:36.345 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:46:36.345 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:46:36.345 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:46:36.346 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:46:36.346 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:46:36.346 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:46:35.120320+0000 mgr.x (mgr.24542) 5119 : cephadm [ERR] 2021-07-31T03:46:36.346 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:46:36.346 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:46:36.347 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: yield 2021-07-31T03:46:36.347 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:46:36.347 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:46:36.347 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:46:36.348 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:46:36.348 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:46:36.348 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:46:36.348 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:46:36.348 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:46:36.349 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:46:36.349 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: task.result() 2021-07-31T03:46:36.349 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:46:36.349 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:46:35.120692+0000 mgr.x (mgr.24542) 5120 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:46:36.349 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:46:36.349 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:46:36.350 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: yield 2021-07-31T03:46:36.350 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:46:36.350 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:46:36.350 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:46:36.351 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:46:36.351 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:46:36.351 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:46:36.351 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:46:36.351 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:46:36.352 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:46:36.352 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: task.result() 2021-07-31T03:46:36.352 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:46:36.352 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: 2021-07-31T03:46:36.352 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:46:36.352 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: 2021-07-31T03:46:36.353 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:46:36.353 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:46:36.353 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:46:36.353 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:46:36.353 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:46:36.354 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:46:36.354 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:46:36.354 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:46:36.354 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:46:36.355 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:46:36.355 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:46:36.355 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:46:36.355 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:46:36.355 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:46:36.356 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:46:36.356 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:46:36.356 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:46:36.356 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:46:36.356 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: raise 2021-07-31T03:46:36.357 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:46:36.357 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:46:36.357 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:46:36.357 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:46:36.357 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:35 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:46:37.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:36 smithi154 conmon[34553]: cluster 2021-07-31T03:46:35.499667+0000 mgr.x (mgr.24542) 5121 : cluster [DBG] pgmap v5589: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:46:37.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:36 smithi045 conmon[39718]: cluster 2021-07-31T03:46:35.499667+0000 mgr.x (mgr.24542) 5121 : cluster [DBG] pgmap v5589: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:46:37.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:36 smithi045 conmon[33655]: cluster 2021-07-31T03:46:35.499667+0000 mgr.x (mgr.24542) 2021-07-31T03:46:37.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:36 smithi045 conmon[33655]: 5121 : cluster [DBG] pgmap v5589: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:46:39.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:38 smithi154 conmon[34553]: cluster 2021-07-31T03:46:37.500146+0000 mgr.x (mgr.24542) 5122 : cluster [DBG] pgmap v5590: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:46:39.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:38 smithi045 conmon[39718]: cluster 2021-07-31T03:46:37.500146+0000 mgr.x (mgr 2021-07-31T03:46:39.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:38 smithi045 conmon[39718]: .24542) 5122 : cluster [DBG] pgmap v5590: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:46:39.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:38 smithi045 conmon[33655]: cluster 2021-07-31T03:46:37.500146+0000 mgr.x (mgr.24542) 5122 : cluster 2021-07-31T03:46:39.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:38 smithi045 conmon[33655]: [DBG] pgmap v5590: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:46:41.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:40 smithi154 conmon[34553]: cluster 2021-07-31T03:46:39.500679+0000 mgr.x (mgr.24542) 5123 : cluster [DBG] pgmap v5591: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:46:41.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:40 smithi045 conmon[39718]: cluster 2021-07-31T03:46:39.500679+0000 2021-07-31T03:46:41.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:40 smithi045 conmon[39718]: mgr.x (mgr.24542) 5123 : cluster [DBG] pgmap v5591: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:46:41.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:40 smithi045 conmon[33655]: cluster 2021-07-31T03:46:39.500679+0000 mgr.x (mgr.24542) 5123 : cluster 2021-07-31T03:46:41.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:40 smithi045 conmon[33655]: [DBG] pgmap v5591: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:46:43.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:42 smithi154 conmon[34553]: cluster 2021-07-31T03:46:41.501203+0000 mgr.x (mgr.24542) 5124 : cluster [DBG] pgmap v5592: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:46:43.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:42 smithi045 conmon[39718]: cluster 2021-07-31T03:46:41.501203+0000 mgr.x ( 2021-07-31T03:46:43.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:42 smithi045 conmon[39718]: mgr.24542) 5124 : cluster [DBG] pgmap v5592: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:46:43.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:42 smithi045 conmon[33655]: cluster 2021-07-31T03:46:41.501203+0000 mgr.x (mgr.24542) 5124 : cluster [DBG] 2021-07-31T03:46:43.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:42 smithi045 conmon[33655]: pgmap v5592: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:46:45.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:44 smithi154 conmon[34553]: cluster 2021-07-31T03:46:43.501636+0000 mgr.x (mgr.24542) 5125 : cluster [DBG] pgmap v5593: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:46:45.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:44 smithi045 conmon[33655]: cluster 2021-07-31T03:46:43.501636+0000 mgr.x (mgr.24542) 5125 : cluster 2021-07-31T03:46:45.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:44 smithi045 conmon[33655]: [DBG] pgmap v5593: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:46:45.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:44 smithi045 conmon[39718]: cluster 2021-07-31T03:46:43.501636+0000 mgr.x (mgr 2021-07-31T03:46:45.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:44 smithi045 conmon[39718]: .24542) 5125 : cluster [DBG] pgmap v5593: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:46:47.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:46 smithi154 conmon[34553]: cluster 2021-07-31T03:46:45.502167+0000 mgr.x (mgr.24542) 5126 : cluster [DBG] pgmap v5594: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:46:47.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:46 smithi045 conmon[33655]: cluster 2021-07-31T03:46:45.502167+0000 mgr.x (mgr.24542 2021-07-31T03:46:47.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:46 smithi045 conmon[33655]: ) 5126 : cluster [DBG] pgmap v5594: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:46:47.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:46 smithi045 conmon[39718]: cluster 2021-07-31T03:46:45.502167+0000 mgr.x (mgr.24542 2021-07-31T03:46:47.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:46 smithi045 conmon[39718]: ) 5126 : cluster [DBG] pgmap v5594: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:46:49.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:48 smithi154 conmon[34553]: cluster 2021-07-31T03:46:47.502655+0000 mgr.x (mgr.24542) 5127 : cluster [DBG] pgmap v5595: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:46:49.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:48 smithi045 conmon[33655]: cluster 2021-07-31T03:46:47.502655+0000 mgr.x (mgr.24542) 5127 2021-07-31T03:46:49.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:48 smithi045 conmon[33655]: : cluster [DBG] pgmap v5595: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:46:49.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:48 smithi045 conmon[39718]: cluster 2021-07-31T03:46:47.502655+0000 mgr.x (mgr.24542) 5127 : cluster [DBG] pgmap v5595: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:46:51.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:50 smithi154 conmon[34553]: cluster 2021-07-31T03:46:49.503251+0000 mgr.x (mgr.24542) 5128 : cluster [DBG] pgmap v5596: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:46:51.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:50 smithi045 conmon[33655]: cluster 2021-07-31T03:46:49.503251+0000 mgr.x (mgr.24542) 5128 : cluster 2021-07-31T03:46:51.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:50 smithi045 conmon[33655]: [DBG] pgmap v5596: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:46:51.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:50 smithi045 conmon[39718]: cluster 2021-07-31T03:46:49.503251+0000 mgr.x 2021-07-31T03:46:51.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:50 smithi045 conmon[39718]: (mgr.24542) 5128 : cluster [DBG] pgmap v5596: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:46:53.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:52 smithi154 conmon[34553]: cluster 2021-07-31T03:46:51.503788+0000 mgr.x (mgr.24542) 5129 : cluster [DBG] pgmap v5597: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:46:53.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:52 smithi045 conmon[33655]: cluster 2021-07-31T03:46:51.503788+0000 mgr.x (mgr. 2021-07-31T03:46:53.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:52 smithi045 conmon[33655]: 24542) 5129 : cluster [DBG] pgmap v5597: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:46:53.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:52 smithi045 conmon[39718]: cluster 2021-07-31T03:46:51.503788+0000 mgr.x (mgr 2021-07-31T03:46:53.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:52 smithi045 conmon[39718]: .24542) 5129 : cluster [DBG] pgmap v5597: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:46:55.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:54 smithi154 conmon[34553]: cluster 2021-07-31T03:46:53.504137+0000 mgr.x (mgr. 2021-07-31T03:46:55.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:54 smithi154 conmon[34553]: 24542) 5130 : cluster [DBG] pgmap v5598: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:46:55.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:54 smithi045 conmon[33655]: cluster 2021-07-31T03:46:53.504137+0000 mgr.x (mgr.24542) 5130 : cluster 2021-07-31T03:46:55.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:54 smithi045 conmon[33655]: [DBG] pgmap v5598: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:46:55.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:54 smithi045 conmon[39718]: cluster 2021-07-31T03:46:53.504137+0000 mgr.x (mgr.24542) 5130 : cluster [DBG] 2021-07-31T03:46:55.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:54 smithi045 conmon[39718]: pgmap v5598: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:46:57.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:56 smithi154 conmon[34553]: cluster 2021-07-31T03:46:55.504657+0000 mgr.x (mgr.24542) 5131 : cluster [DBG] pgmap v5599: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:46:57.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:56 smithi045 conmon[33655]: cluster 2021-07-31T03:46:55.504657+0000 mgr.x (mgr.24542) 5131 2021-07-31T03:46:57.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:56 smithi045 conmon[33655]: : cluster [DBG] pgmap v5599: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:46:57.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:56 smithi045 conmon[39718]: cluster 2021-07-31T03:46:55.504657+0000 mgr.x (mgr 2021-07-31T03:46:57.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:56 smithi045 conmon[39718]: .24542) 5131 : cluster [DBG] pgmap v5599: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:46:59.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:46:58 smithi154 conmon[34553]: cluster 2021-07-31T03:46:57.505327+0000 mgr.x (mgr.24542) 5132 : cluster [DBG] pgmap v5600: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:46:59.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:58 smithi045 conmon[39718]: cluster 2021-07-31T03:46:57. 2021-07-31T03:46:59.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:46:58 smithi045 conmon[39718]: 505327+0000 mgr.x (mgr.24542) 5132 : cluster [DBG] pgmap v5600: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:46:59.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:58 smithi045 conmon[33655]: cluster 2021-07-31T03:46:57.505327+0000 mgr.x (mgr.24542) 5132 2021-07-31T03:46:59.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:46:58 smithi045 conmon[33655]: : cluster [DBG] pgmap v5600: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:01.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:00 smithi154 conmon[34553]: cluster 2021-07-31T03:46:59.505720+0000 mgr.x (mgr.24542) 5133 : cluster [DBG] pgmap v5601: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:01.274 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:00 smithi045 conmon[39718]: cluster 2021-07-31T03:46:59.505720+0000 mgr.x 2021-07-31T03:47:01.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:00 smithi045 conmon[39718]: (mgr.24542) 5133 : cluster [DBG] pgmap v5601: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:01.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:00 smithi045 conmon[33655]: cluster 2021-07-31T03:46:59.505720+0000 mgr.x (mgr.24542) 5133 : cluster 2021-07-31T03:47:01.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:00 smithi045 conmon[33655]: [DBG] pgmap v5601: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:02.899 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:47:03.175 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:47:03.175 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 2h ago 2h - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:47:03.175 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 119m ago 2h 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:47:03.176 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 119m ago 2h 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:47:03.176 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 2h ago 2h - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:47:03.176 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:47:03.176 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 119m ago 2h 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:47:03.177 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:47:03.177 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 2h ago 2h - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:47:03.177 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 119m ago 2h 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:47:03.177 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:47:03.178 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:47:03.178 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:47:03.178 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:47:03.178 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 119m ago 2h 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:47:03.179 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 119m ago 2h 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:47:03.179 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 119m ago 2h 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:47:03.179 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 119m ago 2h 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:47:03.179 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 119m ago 2h 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:47:03.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:02 smithi154 conmon[34553]: cluster 2021-07-31T03:47:01.506153+0000 mgr.x (mgr.24542) 5134 : cluster [DBG] pgmap v5602: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:03.295 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:02 smithi045 conmon[39718]: cluster 2021-07-31T03:47: 2021-07-31T03:47:03.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:02 smithi045 conmon[39718]: 01.506153+0000 mgr.x (mgr.24542) 5134 : cluster [DBG] pgmap v5602: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:03.296 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:02 smithi045 conmon[33655]: cluster 2021-07-31T03:47:01.506153+0000 2021-07-31T03:47:03.296 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:02 smithi045 conmon[33655]: mgr.x (mgr.24542) 5134 : cluster [DBG] pgmap v5602: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:03.512 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:47:03.512 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:47:03.512 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:47:03.513 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:47:03.513 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:47:03.513 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:47:03.513 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:47:03.514 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:47:03.514 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:47:03.514 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:47:03.514 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:47:03.514 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:47:03.515 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:47:03.515 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:47:03.515 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:47:03.515 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:47:03.516 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:47:04.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:03 smithi154 conmon[34553]: audit 2021-07-31T03:47:02.887530+0000 mgr.x (mgr.24542) 5135 : audit 2021-07-31T03:47:04.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:03 smithi154 conmon[34553]: [DBG] from='client.30163 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:47:04.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:03 smithi154 conmon[34553]: audit 2021-07-31T03:47:03.167399+0000 mgr.x (mgr.24542) 5136 : audit [DBG] from='client.30169 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:47:04.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:03 smithi154 conmon[34553]: audit 2021-07-31T03:47:03.512930+0000 mon.a (mon.0) 3365 : audit [DBG] from='client.? 172.21.15.45:0/1495750154' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:47:04.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:03 smithi045 conmon[33655]: audit 2021-07-31T03:47:02.887530+0000 mgr.x (mgr.24542) 2021-07-31T03:47:04.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:03 smithi045 conmon[33655]: 5135 : audit [DBG] from='client.30163 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:47:04.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:03 smithi045 conmon[33655]: audit 2021-07-31T03:47:03.167399+0000 mgr.x (mgr.24542) 5136 : audit [DBG] from='client.30169 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:47:04.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:03 smithi045 conmon[33655]: audit 2021-07-31T03:47:03.512930+0000 mon.a (mon.0) 3365 : audit [DBG] from='client.? 172.21.15.45:0/1495750154' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:47:04.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:03 smithi045 conmon[33655]: 2021-07-31T03:47:04.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:03 smithi045 conmon[39718]: audit 2021-07-31T03:47:02.887530+0000 2021-07-31T03:47:04.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:03 smithi045 conmon[39718]: mgr.x (mgr.24542) 5135 : audit [DBG] from='client.30163 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:47:04.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:03 smithi045 conmon[39718]: audit 2021-07-31T03:47:03.167399+0000 2021-07-31T03:47:04.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:03 smithi045 conmon[39718]: mgr.x (mgr.24542) 5136 : audit [DBG] from='client.30169 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:47:04.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:03 smithi045 conmon[39718]: audit 2021-07 2021-07-31T03:47:04.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:03 smithi045 conmon[39718]: -31T03:47:03.512930+0000 mon.a (mon.0) 3365 : audit [DBG] from='client.? 172.21.15.45:0/1495750154' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:47:05.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:04 smithi154 conmon[34553]: cluster 2021-07-31T03:47:03.506607+0000 mgr.x (mgr.24542) 5137 : cluster 2021-07-31T03:47:05.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:04 smithi154 conmon[34553]: [DBG] pgmap v5603: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:05.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:04 smithi045 conmon[33655]: cluster 2021-07-31T03:47:03.506607+0000 mgr.x (mgr.24542) 5137 : cluster [DBG] pgmap v5603: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:05.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:04 smithi045 conmon[39718]: cluster 2021-07-31T03:47:03.506607+0000 mgr.x ( 2021-07-31T03:47:05.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:04 smithi045 conmon[39718]: mgr.24542) 5137 : cluster [DBG] pgmap v5603: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:07.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:06 smithi154 conmon[34553]: cluster 2021-07-31T03:47:05.507151+0000 mgr.x (mgr.24542) 5138 : 2021-07-31T03:47:07.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:06 smithi154 conmon[34553]: cluster [DBG] pgmap v5604: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:07.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:06 smithi045 conmon[39718]: cluster 2021-07-31T03:47:05.507151 2021-07-31T03:47:07.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:06 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 5138 : cluster [DBG] pgmap v5604: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:07.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:06 smithi045 conmon[33655]: cluster 2021-07-31T03:47:05.507151+0000 2021-07-31T03:47:07.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:06 smithi045 conmon[33655]: mgr.x (mgr.24542) 5138 : cluster [DBG] pgmap v5604: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:09.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:08 smithi154 conmon[34553]: cluster 2021-07-31T03:47:07.507647+0000 mgr.x (mgr.24542) 5139 : cluster 2021-07-31T03:47:09.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:08 smithi154 conmon[34553]: [DBG] pgmap v5605: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:09.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:08 smithi045 conmon[39718]: cluster 2021-07-31T03:47:07.507647+0000 mgr.x 2021-07-31T03:47:09.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:08 smithi045 conmon[39718]: (mgr.24542) 5139 : cluster [DBG] pgmap v5605: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:09.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:08 smithi045 conmon[33655]: cluster 2021-07-31T03:47:07.507647+0000 mgr.x (mgr.24542) 5139 : cluster [DBG] 2021-07-31T03:47:09.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:08 smithi045 conmon[33655]: pgmap v5605: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:11.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:10 smithi154 conmon[34553]: cluster 2021-07-31T03:47:09.508111+0000 mgr.x (mgr.24542) 5140 : cluster [DBG] pgmap v5606: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:11.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:10 smithi045 conmon[39718]: cluster 2021-07-31T03:47:09.508111+0000 mgr.x (mgr.24542) 5140 : cluster [DBG] pgmap v5606: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:11.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:10 smithi045 conmon[33655]: cluster 2021-07-31T03:47:09.508111+0000 mgr.x (mgr.24542) 5140 : cluster [DBG] 2021-07-31T03:47:11.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:10 smithi045 conmon[33655]: pgmap v5606: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:13.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:12 smithi154 conmon[34553]: cluster 2021-07-31T03:47:11.508462+0000 mgr.x (mgr.24542) 5141 : cluster [DBG] pgmap v5607: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:13.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:12 smithi045 conmon[33655]: cluster 2021-07-31T03:47:11.508462+0000 mgr.x (mgr. 2021-07-31T03:47:13.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:12 smithi045 conmon[33655]: 24542) 5141 : cluster [DBG] pgmap v5607: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:13.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:12 smithi045 conmon[39718]: cluster 2021-07-31T03:47:11.508462+0000 mgr.x (mgr.24542) 5141 : cluster [DBG] pgmap v5607: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:15.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:14 smithi154 conmon[34553]: cluster 2021-07-31T03:47:13.509035+0000 mgr.x (mgr.24542) 5142 : cluster [DBG] pgmap v5608: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:15.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:14 smithi045 conmon[33655]: cluster 2021-07-31T03:47:13.509035+0000 mgr.x (mgr.24542) 5142 : cluster 2021-07-31T03:47:15.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:14 smithi045 conmon[33655]: [DBG] pgmap v5608: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:15.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:14 smithi045 conmon[39718]: cluster 2021-07-31T03:47:13.509035+0000 mgr.x (mgr.24542) 5142 : cluster [DBG] pgmap v5608: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:17.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:16 smithi154 conmon[34553]: cluster 2021-07-31T03:47:15.509537+0000 mgr.x (mgr.24542) 5143 : cluster [DBG] pgmap v5609: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:17.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:16 smithi154 conmon[34553]: 2021-07-31T03:47:17.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:16 smithi045 conmon[39718]: cluster 2021-07-31T03:47:15.509537+0000 2021-07-31T03:47:17.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:16 smithi045 conmon[39718]: mgr.x (mgr.24542) 5143 : cluster [DBG] pgmap v5609: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:17.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:16 smithi045 conmon[33655]: cluster 2021-07-31T03:47:15.509537+0000 mgr.x ( 2021-07-31T03:47:17.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:16 smithi045 conmon[33655]: mgr.24542) 5143 : cluster [DBG] pgmap v5609: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:19.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:18 smithi154 conmon[34553]: cluster 2021-07-31T03:47:17.509898+0000 mgr.x (mgr.24542 2021-07-31T03:47:19.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:18 smithi154 conmon[34553]: ) 5144 : cluster [DBG] pgmap v5610: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:19.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:18 smithi045 conmon[39718]: cluster 2021-07-31T03:47:17.509898+0000 mgr.x ( 2021-07-31T03:47:19.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:18 smithi045 conmon[39718]: mgr.24542) 5144 : cluster [DBG] pgmap v5610: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:19.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:18 smithi045 conmon[33655]: cluster 2021-07-31T03:47:17. 2021-07-31T03:47:19.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:18 smithi045 conmon[33655]: 509898+0000 mgr.x (mgr.24542) 5144 : cluster [DBG] pgmap v5610: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:21.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:20 smithi154 conmon[34553]: cluster 2021-07-31T03:47:19.510468+0000 mgr.x (mgr.24542) 5145 : cluster [DBG] pgmap v5611: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:21.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:20 smithi045 conmon[39718]: cluster 2021-07-31T03:47:19.510468+0000 mgr.x (mgr 2021-07-31T03:47:21.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:20 smithi045 conmon[39718]: .24542) 5145 : cluster [DBG] pgmap v5611: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:21.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:20 smithi045 conmon[33655]: cluster 2021-07-31T03:47:19.510468+0000 mgr.x (mgr.24542) 5145 : cluster [DBG] pgmap v5611: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:23.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:22 smithi154 conmon[34553]: cluster 2021-07-31T03:47:21.510904+0000 mgr.x (mgr.24542) 5146 : cluster [DBG] pgmap v5612: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:23.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:22 smithi045 conmon[39718]: cluster 2021-07-31T03:47:21.510904+0000 mgr.x ( 2021-07-31T03:47:23.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:22 smithi045 conmon[39718]: mgr.24542) 5146 : cluster [DBG] pgmap v5612: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:23.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:22 smithi045 conmon[33655]: cluster 2021-07-31T03:47:21.510904+0000 mgr.x (mgr.24542) 5146 : cluster 2021-07-31T03:47:23.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:22 smithi045 conmon[33655]: [DBG] pgmap v5612: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:25.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:24 smithi154 conmon[34553]: cluster 2021-07-31T03:47:23.511337+0000 mgr.x (mgr.24542) 5147 : cluster [DBG] pgmap v5613: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:25.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:24 smithi045 conmon[39718]: cluster 2021-07-31T03:47:23.511337+0000 2021-07-31T03:47:25.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:24 smithi045 conmon[39718]: mgr.x (mgr.24542) 5147 : cluster [DBG] pgmap v5613: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:25.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:24 smithi045 conmon[33655]: cluster 2021-07-31T03:47:23.511337 2021-07-31T03:47:25.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:24 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 5147 : cluster [DBG] pgmap v5613: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:27.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:26 smithi154 conmon[34553]: cluster 2021-07-31T03:47:25.511775+0000 mgr.x (mgr.24542) 5148 : cluster [DBG] pgmap v5614: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:27.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:26 smithi045 conmon[39718]: cluster 2021-07-31T03:47:25.511775+0000 2021-07-31T03:47:27.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:26 smithi045 conmon[39718]: mgr.x (mgr.24542) 5148 : cluster [DBG] pgmap v5614: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:27.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:26 smithi045 conmon[33655]: cluster 2021-07-31T03:47:27.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:26 smithi045 conmon[33655]: 2021-07-31T03:47:25.511775+0000 mgr.x (mgr.24542) 5148 : cluster [DBG] 2021-07-31T03:47:27.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:26 smithi045 conmon[33655]: pgmap v5614: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:29.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:28 smithi154 conmon[34553]: cluster 2021-07-31T03:47:27.512228+0000 mgr.x (mgr.24542) 5149 : cluster [DBG] pgmap v5615: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:29.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:28 smithi154 conmon[34553]: 2021-07-31T03:47:29.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:28 smithi154 conmon[34553]: audit 2021-07-31T03:47:28.412622+0000 mon.b (mon.2) 675 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:47:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:28 smithi154 conmon[34553]: audit 2021-07-31T03:47:28.412941+0000 mon.b ( 2021-07-31T03:47:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:28 smithi154 conmon[34553]: mon.2) 676 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:47:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:28 smithi154 conmon[34553]: audit 2021- 2021-07-31T03:47:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:28 smithi154 conmon[34553]: 07-31T03:47:28.413429+0000 mon.a (mon.0) 3366 2021-07-31T03:47:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:28 smithi154 conmon[34553]: : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:47:29.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:28 smithi154 conmon[34553]: audit 2021- 2021-07-31T03:47:29.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:28 smithi154 conmon[34553]: 07-31T03:47:28.413804+0000 mon.a 2021-07-31T03:47:29.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:28 smithi154 conmon[34553]: (mon.0) 3367 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:47:29.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:28 smithi045 conmon[39718]: cluster 2021-07-31T03:47:27.512228+0000 mgr.x (mgr.24542) 5149 : cluster 2021-07-31T03:47:29.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:28 smithi045 conmon[39718]: [DBG] pgmap v5615: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:29.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:28 smithi045 conmon[39718]: audit 2021-07-31T03:47:28.412622+0000 mon.b (mon.2) 675 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:47:29.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:28 smithi045 conmon[39718]: audit 2021-07-31T03:47:28.412941+0000 mon.b (mon.2) 676 : audit 2021-07-31T03:47:29.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:28 smithi045 conmon[39718]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:47:29.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:28 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T03:47:29.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:28 smithi045 conmon[39718]: T03:47:28.413429+0000 mon.a (mon.0) 3366 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:47:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:28 smithi045 conmon[39718]: audit 2021-07-31T03:47:28.413804 2021-07-31T03:47:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:28 smithi045 conmon[39718]: +0000 mon.a (mon.0) 3367 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:47:29.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:28 smithi045 conmon[33655]: cluster 2021-07-31T03:47:27.512228+0000 mgr.x (mgr.24542) 5149 2021-07-31T03:47:29.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:28 smithi045 conmon[33655]: : cluster [DBG] pgmap v5615: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:29.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:28 smithi045 conmon[33655]: audit 2021-07-31T03:47:28.412622+0000 mon.b (mon.2) 675 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:47:29.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:28 smithi045 conmon[33655]: audit 2021-07-31T03:47:28.412941+0000 2021-07-31T03:47:29.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:28 smithi045 conmon[33655]: mon.b (mon.2) 676 : audit [INF] 2021-07-31T03:47:29.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:28 smithi045 conmon[33655]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:47:29.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:28 smithi045 conmon[33655]: audit 2021-07-31T03: 2021-07-31T03:47:29.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:28 smithi045 conmon[33655]: 47:28.413429+0000 mon.a (mon.0) 2021-07-31T03:47:29.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:28 smithi045 conmon[33655]: 3366 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:47:29.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:28 smithi045 conmon[33655]: audit 2021-07 2021-07-31T03:47:29.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:28 smithi045 conmon[33655]: -31T03:47:28.413804+0000 2021-07-31T03:47:29.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:28 smithi045 conmon[33655]: mon.a (mon.0) 3367 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:47:29.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:28 smithi045 conmon[33655]: 2021-07-31T03:47:31.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:30 smithi154 conmon[34553]: cluster 2021-07-31T03:47:29.512776+0000 mgr.x (mgr.24542) 5150 : cluster [DBG] pgmap v5616: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:31.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:30 smithi045 conmon[39718]: cluster 2021-07-31T03:47:29.512776+0000 mgr.x ( 2021-07-31T03:47:31.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:30 smithi045 conmon[39718]: mgr.24542) 5150 : cluster [DBG] pgmap v5616: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:31.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:30 smithi045 conmon[33655]: cluster 2021-07-31T03:47:29.512776+0000 mgr.x (mgr.24542) 5150 : cluster [DBG] pgmap v5616: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:31.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:30 smithi045 conmon[33655]: 2021-07-31T03:47:33.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:32 smithi154 conmon[34553]: cluster 2021-07-31T03:47:31.513292+0000 mgr.x (mgr.24542) 5151 : cluster [DBG] 2021-07-31T03:47:33.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:32 smithi154 conmon[34553]: pgmap v5617: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:33.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:32 smithi045 conmon[39718]: cluster 2021-07-31T03:47:31.513292+0000 mgr.x ( 2021-07-31T03:47:33.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:32 smithi045 conmon[39718]: mgr.24542) 5151 : cluster [DBG] pgmap v5617: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:33.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:32 smithi045 conmon[33655]: cluster 2021-07-31T03:47:31.513292+0000 mgr.x (mgr.24542) 5151 : cluster [DBG] pgmap v5617: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:33.824 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:47:34.104 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:47:34.104 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 2h ago 2h - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:47:34.104 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 2h ago 2h 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:47:34.105 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 2h ago 2h 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:47:34.105 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 2h ago 2h - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:47:34.105 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:47:34.105 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 2h ago 2h 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:47:34.105 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:47:34.106 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 2h ago 2h - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:47:34.106 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 2h ago 2h 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:47:34.106 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:47:34.106 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:47:34.106 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:47:34.107 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:47:34.107 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 2h ago 2h 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:47:34.107 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 2h ago 2h 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:47:34.107 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 2h ago 2h 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:47:34.107 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 2h ago 2h 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:47:34.108 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 2h ago 2h 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:47:34.439 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:47:34.440 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:47:34.440 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:47:34.440 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:47:34.440 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:47:34.441 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:47:34.441 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:47:34.441 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:47:34.441 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:47:34.441 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:47:34.442 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:47:34.442 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:47:34.442 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:47:34.442 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:47:34.442 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:47:34.443 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:47:34.443 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:47:35.148 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:34 smithi154 conmon[34553]: cluster 2021-07-31T03:47:33.513746+0000 2021-07-31T03:47:35.148 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:34 smithi154 conmon[34553]: mgr.x (mgr.24542) 5152 : cluster [DBG] pgmap v5618: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:35.148 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:34 smithi154 conmon[34553]: audit 2021-07-31T03:47:33.812484+0000 mgr.x (mgr.24542) 5153 : audit [DBG] from='client.30181 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:47:35.148 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:34 smithi154 conmon[34553]: audit 2021-07-31T03:47:34.096639+0000 2021-07-31T03:47:35.149 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:34 smithi154 conmon[34553]: mgr.x (mgr.24542) 5154 : audit [DBG] from='client.30187 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:47:35.149 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:34 smithi154 conmon[34553]: audit 2021-07-31T03 2021-07-31T03:47:35.149 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:34 smithi154 conmon[34553]: :47:34.440719+0000 mon.a (mon.0) 3368 : audit [DBG] from='client.? 172.21.15.45:0/1542926044' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:47:35.149 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:34 smithi154 conmon[34553]: 2021-07-31T03:47:35.150 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:34 smithi045 conmon[33655]: cluster 2021-07-31T03:47:33.513746+0000 mgr.x (mgr 2021-07-31T03:47:35.150 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:34 smithi045 conmon[33655]: .24542) 5152 : cluster [DBG] pgmap v5618: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:35.150 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:34 smithi045 conmon[33655]: audit 2021-07-31T03:47:33.812484+0000 mgr.x (mgr.24542) 5153 : audit [DBG] from='client.30181 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:47:35.151 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:34 smithi045 conmon[33655]: audit 2021-07-31T03: 2021-07-31T03:47:35.151 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:34 smithi045 conmon[33655]: 47:34.096639+0000 mgr.x (mgr.24542) 5154 : audit [DBG] from='client.30187 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:47:35.151 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:34 smithi045 conmon[33655]: audit 2021-07-31T03: 2021-07-31T03:47:35.151 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:34 smithi045 conmon[33655]: 47:34.440719+0000 mon.a (mon.0) 3368 : audit [DBG] from='client.? 172.21.15.45:0/1542926044' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:47:35.152 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:34 smithi045 conmon[39718]: cluster 2021-07-31T03:47:33.513746+0000 2021-07-31T03:47:35.152 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:34 smithi045 conmon[39718]: mgr.x (mgr.24542) 5152 : cluster [DBG] pgmap v5618: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:35.152 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:34 smithi045 conmon[39718]: audit 2021-07-31T03:47:35.152 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:34 smithi045 conmon[39718]: 2021-07-31T03:47:33.812484+0000 mgr.x ( 2021-07-31T03:47:35.152 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:34 smithi045 conmon[39718]: mgr.24542) 5153 : audit [DBG] from='client.30181 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:47:35.153 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:34 smithi045 conmon[39718]: 2021-07-31T03:47:35.153 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:34 smithi045 conmon[39718]: audit 2021-07-31T03:47: 2021-07-31T03:47:35.153 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:34 smithi045 conmon[39718]: 34.096639+0000 mgr.x (mgr.24542) 5154 : audit [DBG] 2021-07-31T03:47:35.153 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:34 smithi045 conmon[39718]: from='client.30187 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:47:35.153 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:34 smithi045 conmon[39718]: audit 2021-07-31T03:47 2021-07-31T03:47:35.154 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:34 smithi045 conmon[39718]: :34.440719+0000 mon.a (mon.0) 3368 : audit [DBG] from='client.? 172.21.15.45:0/1542926044' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:47:35.154 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:34 smithi045 conmon[39718]: 2021-07-31T03:47:35.502 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: debug 2021-07-31T03:47:35.167+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:47:35.502 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:47:35.503 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:47:35.503 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: yield 2021-07-31T03:47:35.503 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:47:35.503 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:47:35.503 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:47:35.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:47:35.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:47:35.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:47:35.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:47:35.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:47:35.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:47:35.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: task.result() 2021-07-31T03:47:35.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:47:35.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: debug 2021-07-31T03:47:35.179+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:47:35.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:47:35.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:47:35.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: yield 2021-07-31T03:47:35.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:47:35.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:47:35.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:47:35.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:47:35.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:47:35.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:47:35.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:47:35.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:47:35.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:47:35.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: task.result() 2021-07-31T03:47:35.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:47:35.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: debug 2021-07-31T03:47:35.205+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:47:35.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:47:35.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:47:35.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: yield 2021-07-31T03:47:35.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:47:35.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:47:35.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:47:35.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:47:35.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:47:35.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:47:35.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:47:35.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:47:35.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:47:35.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: task.result() 2021-07-31T03:47:35.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:47:35.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: debug 2021-07-31T03:47:35.205+0000 7fcb93961700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:47:35.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:47:35.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:47:35.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: yield 2021-07-31T03:47:35.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:47:35.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:47:35.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:47:35.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:47:35.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:47:35.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:47:35.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:47:35.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:47:35.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:47:35.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: task.result() 2021-07-31T03:47:35.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:47:35.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: 2021-07-31T03:47:35.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:47:35.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: 2021-07-31T03:47:35.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:47:35.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:47:35.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:47:35.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:47:35.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:47:35.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:47:35.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:47:35.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:47:35.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:47:35.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:47:35.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:47:35.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:47:35.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:47:35.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:47:35.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:47:35.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:47:35.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:47:35.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:47:35.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: raise 2021-07-31T03:47:35.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:47:35.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:47:35.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:47:35.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:47:35.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:47:35.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: debug 2021-07-31T03:47:35.214+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:47:35.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:47:35.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:47:35.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: yield 2021-07-31T03:47:35.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:47:35.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:47:35.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:47:35.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:47:35.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:47:35.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:47:35.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:47:35.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:47:35.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:47:35.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: task.result() 2021-07-31T03:47:35.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:47:35.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: debug 2021-07-31T03:47:35.214+0000 7fcb94162700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:47:35.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:47:35.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:47:35.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: yield 2021-07-31T03:47:35.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:47:35.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:47:35.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:47:35.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:47:35.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:47:35.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:47:35.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:47:35.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:47:35.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:47:35.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: task.result() 2021-07-31T03:47:35.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:47:35.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: 2021-07-31T03:47:35.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:47:35.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: 2021-07-31T03:47:35.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:47:35.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:47:35.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:47:35.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:47:35.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:47:35.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:47:35.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:47:35.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:47:35.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:47:35.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:47:35.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:47:35.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:47:35.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:47:35.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:47:35.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:47:35.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:47:35.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:47:35.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:47:35.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: raise 2021-07-31T03:47:35.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:47:35.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:47:35.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:47:35.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:47:35.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:47:36.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[34553]: audit 2021-07-31T03:47:35.121452+0000 mon.b (mon.2) 677 : 2021-07-31T03:47:36.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[34553]: audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:47:36.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[34553]: audit 2021-07-31T03:47:35.122912+0000 mon.b (mon.2) 678 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:47:36.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[34553]: audit 2021-07-31T03:47:35.123802+0000 mon.b (mon.2) 679 : audit [INF] 2021-07-31T03:47:36.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[34553]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:47:36.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:47 2021-07-31T03:47:36.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[34553]: :35.124563+0000 mgr.x (mgr.24542) 5155 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:47:36.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[34553]: cephadm 2021-07-31T 2021-07-31T03:47:36.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[34553]: 03:47:35.168612+0000 mgr.x (mgr.24542) 5156 : cephadm [ERR] 2021-07-31T03:47:36.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[34553]: 2021-07-31T03:47:36.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:47:36.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:47:36.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[34553]: yield 2021-07-31T03:47:36.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:47:36.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:47:36.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:47:36.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:47:36.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:47:36.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:47:36.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:47:36.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:47:36.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:47:36.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[34553]: task.result() 2021-07-31T03:47:36.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:47:36.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:35 smithi154 conmon[34553]: cephadm 2021-07-31T03:47:35.169261+0000 mgr.x (mgr.24542) 5157 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:47:36.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:35 smithi045 conmon[33655]: audit 2021-07-31T03:47:35.121452+0000 mon.b 2021-07-31T03:47:36.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:35 smithi045 conmon[33655]: (mon.2) 677 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:47:36.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:35 smithi045 conmon[33655]: audit 2021-07-31T03:47:35.122912+0000 mon.b (mon.2 2021-07-31T03:47:36.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:35 smithi045 conmon[33655]: ) 678 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:47:36.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:35 smithi045 conmon[33655]: audit 2021- 2021-07-31T03:47:36.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:35 smithi045 conmon[33655]: 07-31T03:47:35.123802+0000 mon.b (mon.2) 2021-07-31T03:47:36.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:35 smithi045 conmon[33655]: 679 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:47:36.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:47:36.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:35 smithi045 conmon[33655]: 2021-07-31T03:47:35.124563+0000 mgr.x (mgr.24542) 5155 : cephadm [WRN] 2021-07-31T03:47:36.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:35 smithi045 conmon[33655]: unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:47:36.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:47: 2021-07-31T03:47:36.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:35 smithi045 conmon[33655]: 35.168612+0000 mgr.x (mgr.24542) 5156 2021-07-31T03:47:36.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:35 smithi045 conmon[33655]: : cephadm [ERR] 2021-07-31T03:47:36.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:35 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:47:36.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:47:36.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:35 smithi045 conmon[33655]: yield 2021-07-31T03:47:36.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:35 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:47:36.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:35 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:47:36.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:47:36.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:35 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:47:36.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:47:36.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:35 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:47:36.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:47:36.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:35 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:47:36.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:35 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:47:36.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:35 smithi045 conmon[33655]: task.result() 2021-07-31T03:47:36.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:35 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:47:36.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:35 smithi045 conmon[33655]: cephadm 2021-07-31T03:47:35.169261+0000 mgr.x (mgr.24542) 5157 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:47:36.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:35 smithi045 conmon[39718]: audit 2021-07-31T03:47:35.121452+0000 2021-07-31T03:47:36.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:35 smithi045 conmon[39718]: mon.b (mon.2) 677 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:47:36.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:35 smithi045 conmon[39718]: audit 2021-07-31T03:47:36.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:35 smithi045 conmon[39718]: 2021-07-31 2021-07-31T03:47:36.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:35 smithi045 conmon[39718]: T03:47:35.122912+0000 2021-07-31T03:47:36.282 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:35 smithi045 conmon[39718]: mon.b (mon.2) 678 : audit 2021-07-31T03:47:36.282 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:35 smithi045 conmon[39718]: [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:47:36.282 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:35 smithi045 conmon[39718]: audit 2021-07-31T03:47:36.282 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:35 smithi045 conmon[39718]: 2021-07-31T03: 2021-07-31T03:47:36.282 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:35 smithi045 conmon[39718]: 47:35.123802+0000 mon.b ( 2021-07-31T03:47:36.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:35 smithi045 conmon[39718]: mon.2) 679 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:47:36.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:35 smithi045 conmon[39718]: 2021-07-31T03:47:36.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:35 smithi045 conmon[39718]: cephadm 2021- 2021-07-31T03:47:36.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:35 smithi045 conmon[39718]: 07-31T03:47:35.124563+0000 mgr.x (mgr 2021-07-31T03:47:36.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:35 smithi045 conmon[39718]: .24542) 5155 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:47:36.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:47:36.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:35 smithi045 conmon[39718]: 2021- 2021-07-31T03:47:36.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:35 smithi045 conmon[39718]: 07-31T03:47:35 2021-07-31T03:47:36.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:35 smithi045 conmon[39718]: .168612+0000 mgr.x (mgr.24542) 2021-07-31T03:47:36.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:35 smithi045 conmon[39718]: 5156 : cephadm [ERR] 2021-07-31T03:47:36.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:35 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:47:36.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:47:36.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:35 smithi045 conmon[39718]: yield 2021-07-31T03:47:36.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:35 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:47:36.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:35 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:47:36.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:47:36.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:35 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:47:36.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:47:36.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:35 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:47:36.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:47:36.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:35 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:47:36.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:35 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:47:36.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:35 smithi045 conmon[39718]: task.result() 2021-07-31T03:47:36.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:35 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:47:36.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:35 smithi045 conmon[39718]: cephadm 2021-07-31T03:47:35.169261+0000 mgr.x (mgr.24542) 5157 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:47:37.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: cephadm 2021-07-31T03:47:35.180562+0000 mgr.x (mgr.24542) 5158 : cephadm [ERR] 2021-07-31T03:47:37.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:47:37.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:47:37.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: yield 2021-07-31T03:47:37.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:47:37.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:47:37.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:47:37.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:47:37.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:47:37.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:47:37.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:47:37.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:47:37.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:47:37.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: task.result() 2021-07-31T03:47:37.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:47:37.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: 2021-07-31T03:47:37.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: cephadm 2021-07-31T03:47:35.181416+0000 mgr.x (mgr.24542) 5159 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:47:37.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: cephadm 2021-07-31T03:47:35.206007+0000 mgr.x (mgr.24542) 5160 : cephadm [ERR] 2021-07-31T03:47:37.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:47:37.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:47:37.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: yield 2021-07-31T03:47:37.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:47:37.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:47:37.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:47:37.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:47:37.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:47:37.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:47:37.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:47:37.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:47:37.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:47:37.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: task.result() 2021-07-31T03:47:37.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:47:37.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: cephadm 2021-07-31T03:47:35.206622+0000 mgr.x (mgr.24542) 5161 : 2021-07-31T03:47:37.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:47:37.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:47:37.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:47:37.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: yield 2021-07-31T03:47:37.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:47:37.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:47:37.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:47:37.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:47:37.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:47:37.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:47:37.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:47:37.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:47:37.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:47:37.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: task.result() 2021-07-31T03:47:37.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:47:37.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: 2021-07-31T03:47:37.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:47:37.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: 2021-07-31T03:47:37.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:47:37.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:47:37.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:47:37.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:47:37.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:47:37.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:47:37.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:47:37.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:47:37.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:47:37.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:47:37.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:47:37.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:47:37.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:47:37.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:47:37.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:47:37.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:47:37.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:47:37.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:47:37.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: raise 2021-07-31T03:47:37.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:47:37.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:47:37.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:47:37.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:47:37.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:47:37.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: cephadm 2021-07-31T03:47:35.215166+0000 mgr.x (mgr.24542) 5162 : cephadm [ERR] 2021-07-31T03:47:37.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:47:37.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:47:37.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: yield 2021-07-31T03:47:37.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:47:37.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:47:37.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:47:37.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:47:37.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:47:37.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:47:37.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:47:37.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:47:37.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:47:37.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: task.result() 2021-07-31T03:47:37.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:47:37.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: cephadm 2021-07-31T03:47:35.215560+0000 mgr.x (mgr.24542) 5163 2021-07-31T03:47:37.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:47:37.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:47:37.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:47:37.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: yield 2021-07-31T03:47:37.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:47:37.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:47:37.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:47:37.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:47:37.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:47:37.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:47:37.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:47:37.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:47:37.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:47:37.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: task.result() 2021-07-31T03:47:37.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:47:37.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: 2021-07-31T03:47:37.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:47:37.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: 2021-07-31T03:47:37.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:47:37.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:47:37.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:47:37.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:47:37.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:47:37.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:47:37.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:47:37.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:47:37.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:47:37.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:47:37.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:47:37.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:47:37.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:47:37.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:47:37.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:47:37.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:47:37.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:47:37.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:47:37.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: raise 2021-07-31T03:47:37.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:47:37.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:47:37.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:47:37.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:47:37.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:47:37.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:36 smithi154 conmon[34553]: cluster 2021-07-31T03:47:35.514197+0000 mgr.x (mgr.24542) 5164 : cluster [DBG] pgmap v5619: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:37.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: cephadm 2021-07-31T03:47:35 2021-07-31T03:47:37.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: .180562+0000 mgr.x (mgr.24542) 5158 : cephadm [ERR] 2021-07-31T03:47:37.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:47:37.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:47:37.282 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: yield 2021-07-31T03:47:37.282 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:47:37.282 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:47:37.282 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:47:37.282 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:47:37.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:47:37.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:47:37.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:47:37.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:47:37.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:47:37.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: task.result() 2021-07-31T03:47:37.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:47:37.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: cephadm 2021-07-31T03:47:35.181416+0000 mgr.x (mgr.24542) 5159 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:47:37.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: cephadm 2021-07-31 2021-07-31T03:47:37.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: T03:47:35.206007+0000 mgr.x 2021-07-31T03:47:37.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: (mgr.24542) 5160 : cephadm [ERR] 2021-07-31T03:47:37.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: 2021-07-31T03:47:37.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:47:37.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:47:37.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: yield 2021-07-31T03:47:37.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:47:37.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:47:37.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:47:37.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:47:37.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:47:37.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:47:37.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:47:37.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:47:37.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:47:37.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: task.result() 2021-07-31T03:47:37.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:47:37.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: cephadm 2021-07-31T03:47:35.206622+0000 mgr.x (mgr.24542) 5161 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:47:37.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:47:37.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:47:37.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: yield 2021-07-31T03:47:37.289 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:47:37.289 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:47:37.289 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:47:37.289 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:47:37.289 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:47:37.290 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:47:37.290 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:47:37.290 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:47:37.290 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:47:37.290 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: task.result() 2021-07-31T03:47:37.291 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:47:37.291 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: 2021-07-31T03:47:37.291 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:47:37.291 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: 2021-07-31T03:47:37.291 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:47:37.292 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:47:37.292 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:47:37.292 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:47:37.292 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:47:37.293 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:47:37.293 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:47:37.293 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:47:37.293 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:47:37.293 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:47:37.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:47:37.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:47:37.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:47:37.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:47:37.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:47:37.295 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:47:37.295 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:47:37.295 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:47:37.295 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: raise 2021-07-31T03:47:37.295 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:47:37.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:47:37.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:47:37.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:47:37.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:47:37.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: cephadm 2021-07-31T03:47:35.215166+0000 mgr.x (mgr.24542) 5162 : cephadm [ERR] 2021-07-31T03:47:37.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:47:37.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:47:37.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: yield 2021-07-31T03:47:37.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:47:37.298 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: cephadm 2021-07-31T03:47:35.180562 2021-07-31T03:47:37.298 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 5158 : cephadm [ERR] 2021-07-31T03:47:37.298 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:47:37.298 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:47:37.299 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: yield 2021-07-31T03:47:37.299 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:47:37.299 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:47:37.299 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:47:37.299 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:47:37.300 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:47:37.300 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:47:37.300 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:47:37.300 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:47:37.300 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:47:37.301 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: task.result() 2021-07-31T03:47:37.301 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:47:37.301 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: cephadm 2021- 2021-07-31T03:47:37.301 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: 07-31T03:47:35.181416+0000 mgr.x (mgr.24542) 5159 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:47:37.302 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: cephadm 2021-07-31T03:47:35.206007+0000 mgr.x (mgr.24542) 5160 : cephadm 2021-07-31T03:47:37.302 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: [ERR] 2021-07-31T03:47:37.302 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:47:37.302 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:47:37.302 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: yield 2021-07-31T03:47:37.303 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:47:37.303 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:47:37.303 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:47:37.303 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:47:37.303 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:47:37.304 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:47:37.304 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:47:37.304 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:47:37.304 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:47:37.305 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: task.result() 2021-07-31T03:47:37.305 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:47:37.305 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: cephadm 2021-07-31 2021-07-31T03:47:37.305 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: T03:47:35.206622+0000 mgr.x (mgr.24542) 5161 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:47:37.305 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:47:37.306 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:47:37.306 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: yield 2021-07-31T03:47:37.306 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:47:37.307 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:47:37.307 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:47:37.307 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:47:37.307 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:47:37.308 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:47:37.308 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:47:37.308 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:47:37.308 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:47:37.308 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: task.result() 2021-07-31T03:47:37.309 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:47:37.309 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: cephadm 2021-07-31T03:47:35.215560+0000 mgr.x (mgr.24542) 5163 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:47:37.309 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:47:37.309 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:47:37.309 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: yield 2021-07-31T03:47:37.310 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:47:37.310 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:47:37.310 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:47:37.310 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:47:37.310 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:47:37.311 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:47:37.311 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:47:37.311 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:47:37.311 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:47:37.311 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: task.result() 2021-07-31T03:47:37.312 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:47:37.312 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: 2021-07-31T03:47:37.312 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:47:37.312 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: 2021-07-31T03:47:37.312 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:47:37.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:47:37.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:47:37.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:47:37.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:47:37.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:47:37.314 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:47:37.314 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:47:37.314 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:47:37.314 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:47:37.314 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:47:37.315 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:47:37.315 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:47:37.315 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:47:37.315 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:47:37.315 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:47:37.316 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:47:37.316 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:47:37.316 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: raise 2021-07-31T03:47:37.316 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:47:37.316 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:47:37.317 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:47:37.317 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:47:37.317 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:47:37.317 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: cluster 2021-07-31T03:47:37.317 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[39718]: 2021-07-31T03:47:35.514197+0000 mgr.x (mgr.24542) 5164 : cluster [DBG] pgmap v5619: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:37.318 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:47:37.318 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:47:37.318 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:47:37.319 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:47:37.319 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:47:37.319 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:47:37.319 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:47:37.319 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:47:37.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: task.result() 2021-07-31T03:47:37.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:47:37.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: 2021-07-31T03:47:37.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:47:37.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: 2021-07-31T03:47:37.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:47:37.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:47:37.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:47:37.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:47:37.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:47:37.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:47:37.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:47:37.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:47:37.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:47:37.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:47:37.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:47:37.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:47:37.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:47:37.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:47:37.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:47:37.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:47:37.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:47:37.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:47:37.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: raise 2021-07-31T03:47:37.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:47:37.325 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:47:37.325 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:47:37.325 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:47:37.325 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:47:37.325 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: cephadm 2021-07-31T03:47:35.215166+0000 mgr.x (mgr.24542) 5162 : cephadm 2021-07-31T03:47:37.326 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: [ERR] 2021-07-31T03:47:37.326 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:47:37.326 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:47:37.326 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: yield 2021-07-31T03:47:37.326 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:47:37.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:47:37.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:47:37.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:47:37.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:47:37.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:47:37.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:47:37.328 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:47:37.328 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:47:37.328 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: task.result() 2021-07-31T03:47:37.328 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:47:37.328 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: cephadm 2021-07-31T03:47:35.215560+0000 mgr.x (mgr.24542) 5163 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:47:37.329 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:47:37.329 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:47:37.329 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: yield 2021-07-31T03:47:37.329 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:47:37.329 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:47:37.330 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:47:37.330 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:47:37.330 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:47:37.330 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:47:37.330 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:47:37.331 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:47:37.331 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:47:37.331 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: task.result() 2021-07-31T03:47:37.331 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:47:37.331 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: 2021-07-31T03:47:37.332 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:47:37.332 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: 2021-07-31T03:47:37.332 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:47:37.332 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:47:37.332 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:47:37.333 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:47:37.333 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:47:37.333 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:47:37.333 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:47:37.333 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:47:37.334 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:47:37.334 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:47:37.334 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:47:37.334 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:47:37.334 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:47:37.335 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:47:37.335 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:47:37.335 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:47:37.335 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:47:37.335 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:47:37.336 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: raise 2021-07-31T03:47:37.336 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:47:37.336 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:47:37.336 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:47:37.336 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:47:37.337 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:47:37.337 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:36 smithi045 conmon[33655]: cluster 2021-07-31T03:47:35.514197+0000 mgr.x (mgr.24542) 5164 : cluster [DBG] pgmap v5619: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:39.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:38 smithi154 conmon[34553]: cluster 2021-07-31T03:47:37.514681+0000 mgr.x (mgr.24542) 5165 2021-07-31T03:47:39.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:38 smithi154 conmon[34553]: : cluster [DBG] pgmap v5620: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:39.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:38 smithi045 conmon[33655]: cluster 2021-07-31T03:47:37.514681+0000 mgr.x (mgr.24542) 5165 2021-07-31T03:47:39.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:38 smithi045 conmon[33655]: : cluster [DBG] pgmap v5620: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:39.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:38 smithi045 conmon[39718]: cluster 2021-07-31T03:47:37.514681+0000 mgr.x (mgr.24542) 5165 : cluster [DBG] pgmap v5620: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:41.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:40 smithi154 conmon[34553]: cluster 2021-07-31T03:47:39.515252+0000 mgr.x (mgr.24542) 5166 : cluster 2021-07-31T03:47:41.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:40 smithi154 conmon[34553]: [DBG] pgmap v5621: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:41.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:40 smithi045 conmon[33655]: cluster 2021-07-31T03:47:39.515252+0000 mgr.x (mgr.24542) 5166 2021-07-31T03:47:41.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:40 smithi045 conmon[33655]: : cluster [DBG] pgmap v5621: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:41.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:40 smithi045 conmon[39718]: cluster 2021-07-31T03:47:39.515252+0000 mgr.x (mgr.24542) 5166 : cluster [DBG] pgmap v5621: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:43.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:42 smithi154 conmon[34553]: cluster 2021-07-31T03:47:41.515790+0000 mgr.x (mgr.24542) 5167 : cluster 2021-07-31T03:47:43.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:42 smithi154 conmon[34553]: [DBG] pgmap v5622: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:43.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:42 smithi045 conmon[39718]: cluster 2021-07-31T03:47:41.515790+0000 mgr.x (mgr.24542) 5167 : cluster [DBG] pgmap v5622: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:43.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:42 smithi045 conmon[33655]: cluster 2021-07-31T03:47:41.515790+0000 mgr.x (mgr.24542) 5167 : 2021-07-31T03:47:43.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:42 smithi045 conmon[33655]: cluster [DBG] pgmap v5622: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:45.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:44 smithi154 conmon[34553]: cluster 2021-07-31T03:47:43.516191+0000 mgr.x (mgr.24542) 5168 : cluster [DBG] pgmap v5623: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:45.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:44 smithi045 conmon[33655]: cluster 2021-07-31T03:47:43.516191+0000 mgr.x (mgr.24542) 5168 : 2021-07-31T03:47:45.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:44 smithi045 conmon[33655]: cluster [DBG] pgmap v5623: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:45.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:44 smithi045 conmon[39718]: cluster 2021-07-31T03:47:43.516191+0000 mgr.x (mgr 2021-07-31T03:47:45.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:44 smithi045 conmon[39718]: .24542) 5168 : cluster [DBG] pgmap v5623: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:47.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:46 smithi154 conmon[34553]: cluster 2021-07-31T03:47:45.516648+0000 mgr.x (mgr.24542) 5169 : cluster [DBG] pgmap v5624: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:47.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:46 smithi045 conmon[33655]: cluster 2021-07-31T03:47:45.516648+0000 mgr.x (mgr.24542) 5169 : 2021-07-31T03:47:47.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:46 smithi045 conmon[33655]: cluster [DBG] pgmap v5624: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:47.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:46 smithi045 conmon[39718]: cluster 2021-07-31T03:47:45.516648+0000 mgr.x ( 2021-07-31T03:47:47.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:46 smithi045 conmon[39718]: mgr.24542) 5169 : cluster [DBG] pgmap v5624: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:49.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:48 smithi154 conmon[34553]: cluster 2021-07-31T03:47:47.517193+0000 mgr.x (mgr.24542) 5170 : cluster [DBG] pgmap v5625: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:49.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:48 smithi045 conmon[33655]: cluster 2021-07-31T03:47:47.517193+0000 mgr.x (mgr.24542) 5170 2021-07-31T03:47:49.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:48 smithi045 conmon[33655]: : cluster [DBG] pgmap v5625: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:49.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:48 smithi045 conmon[39718]: cluster 2021-07-31T03:47:47.517193+0000 mgr.x (mgr 2021-07-31T03:47:49.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:48 smithi045 conmon[39718]: .24542) 5170 : cluster [DBG] pgmap v5625: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:51.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:50 smithi154 conmon[34553]: cluster 2021-07-31T03:47:49.517679+0000 mgr.x (mgr.24542) 5171 : cluster [DBG] pgmap v5626: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:51.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:50 smithi045 conmon[33655]: cluster 2021-07-31T03:47:49.517679+0000 2021-07-31T03:47:51.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:50 smithi045 conmon[33655]: mgr.x (mgr.24542) 5171 : cluster [DBG] pgmap v5626: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:51.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:50 smithi045 conmon[39718]: cluster 2021-07-31T03:47:49.517679+0000 mgr.x 2021-07-31T03:47:51.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:50 smithi045 conmon[39718]: (mgr.24542) 5171 : cluster [DBG] pgmap v5626: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:53.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:52 smithi154 conmon[34553]: cluster 2021-07-31T03:47:51.518195+0000 mgr.x (mgr.24542) 5172 : cluster [DBG] pgmap v5627: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:53.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:52 smithi045 conmon[33655]: cluster 2021-07-31T03:47:51.518195+0000 mgr.x (mgr.24542) 5172 : cluster [DBG] pgmap v5627: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:53.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:52 smithi045 conmon[33655]: 2021-07-31T03:47:53.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:52 smithi045 conmon[39718]: cluster 2021-07-31T03:47:51.518195+0000 mgr.x (mgr. 2021-07-31T03:47:53.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:52 smithi045 conmon[39718]: 24542) 5172 : cluster [DBG] pgmap v5627: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:55.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:54 smithi154 conmon[34553]: cluster 2021-07-31T03:47:53.518594+0000 mgr.x (mgr.24542) 5173 : cluster [DBG] pgmap v5628: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:55.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:54 smithi045 conmon[39718]: cluster 2021-07-31T03:47:53.518594+0000 mgr.x ( 2021-07-31T03:47:55.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:54 smithi045 conmon[39718]: mgr.24542) 5173 : cluster [DBG] pgmap v5628: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:55.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:54 smithi045 conmon[33655]: cluster 2021-07-31T03:47:53.518594+0000 mgr.x (mgr.24542 2021-07-31T03:47:55.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:54 smithi045 conmon[33655]: ) 5173 : cluster [DBG] pgmap v5628: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:57.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:56 smithi154 conmon[34553]: cluster 2021-07-31T03:47:55.519011+0000 mgr.x (mgr.24542) 5174 : cluster [DBG] pgmap v5629: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:57.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:56 smithi045 conmon[33655]: cluster 2021-07-31T03:47:55.519011+0000 mgr.x (mgr.24542) 5174 2021-07-31T03:47:57.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:56 smithi045 conmon[33655]: : cluster [DBG] pgmap v5629: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:57.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:56 smithi045 conmon[39718]: cluster 2021-07-31T03:47:55.519011+0000 mgr.x (mgr. 2021-07-31T03:47:57.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:56 smithi045 conmon[39718]: 24542) 5174 : cluster [DBG] pgmap v5629: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:59.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:47:58 smithi154 conmon[34553]: cluster 2021-07-31T03:47:57.519478+0000 mgr.x (mgr.24542) 5175 : cluster [DBG] pgmap v5630: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:59.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:58 smithi045 conmon[39718]: cluster 2021-07-31T03:47:57.519478+0000 mgr.x (mgr.24542) 5175 : 2021-07-31T03:47:59.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:47:58 smithi045 conmon[39718]: cluster [DBG] pgmap v5630: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:47:59.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:58 smithi045 conmon[33655]: cluster 2021-07-31T03:47:57.519478+0000 mgr.x (mgr.24542) 5175 : cluster 2021-07-31T03:47:59.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:47:58 smithi045 conmon[33655]: [DBG] pgmap v5630: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:01.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:00 smithi154 conmon[34553]: cluster 2021-07-31T03:47:59.520025+0000 mgr.x (mgr.24542) 5176 : cluster [DBG] pgmap v5631: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:01.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:00 smithi154 conmon[34553]: 2021-07-31T03:48:01.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:00 smithi045 conmon[39718]: cluster 2021-07-31T03:47:59.520025+0000 mgr.x (mgr.24542) 2021-07-31T03:48:01.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:00 smithi045 conmon[39718]: 5176 : cluster [DBG] pgmap v5631: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:01.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:00 smithi045 conmon[33655]: cluster 2021-07-31T03:47:59.520025+0000 mgr.x (mgr.24542) 5176 : cluster 2021-07-31T03:48:01.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:00 smithi045 conmon[33655]: [DBG] pgmap v5631: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:03.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:02 smithi154 conmon[34553]: cluster 2021-07-31T03:48:01.520408+0000 mgr.x (mgr.24542) 5177 : cluster [DBG] pgmap v5632: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:03.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:02 smithi045 conmon[39718]: cluster 2021-07-31T03:48:01.520408+0000 mgr.x (mgr.24542) 5177 : 2021-07-31T03:48:03.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:02 smithi045 conmon[39718]: cluster [DBG] pgmap v5632: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:03.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:02 smithi045 conmon[33655]: cluster 2021-07-31T03:48:01.520408+0000 mgr.x (mgr.24542) 5177 : cluster [DBG] pgmap v5632: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:04.753 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:48:05.028 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:48:05.029 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 2h ago 2h - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:48:05.029 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 2h ago 2h 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:48:05.029 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 2h ago 2h 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:48:05.030 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 2h ago 2h - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:48:05.030 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:48:05.030 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 2h ago 2h 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:48:05.030 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:48:05.031 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 2h ago 2h - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:48:05.031 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 2h ago 2h 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:48:05.031 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:48:05.031 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:48:05.032 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:48:05.032 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:48:05.032 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 2h ago 2h 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:48:05.032 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 2h ago 2h 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:48:05.033 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 2h ago 2h 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:48:05.033 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 2h ago 2h 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:48:05.033 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 2h ago 2h 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:48:05.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:04 smithi154 conmon[34553]: cluster 2021-07-31T03:48:03.520820+0000 mgr.x (mgr.24542) 5178 : cluster [DBG] pgmap v5633: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:05.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:04 smithi045 conmon[39718]: cluster 2021-07-31T03:48:03.520820 2021-07-31T03:48:05.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:04 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 5178 : cluster [DBG] pgmap v5633: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:05.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:04 smithi045 conmon[33655]: cluster 2021-07-31T03:48:03.520820+0000 mgr.x (mgr. 2021-07-31T03:48:05.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:04 smithi045 conmon[33655]: 24542) 5178 : cluster [DBG] pgmap v5633: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:05.364 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:48:05.364 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:48:05.364 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:48:05.364 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:48:05.365 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:48:05.365 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:48:05.365 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:48:05.365 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:48:05.366 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:48:05.366 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:48:05.366 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:48:05.366 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:48:05.366 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:48:05.367 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:48:05.367 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:48:05.367 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:48:05.367 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:48:06.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:05 smithi154 conmon[34553]: audit 2021-07-31T03:48:04.741468+0000 mgr.x (mgr.24542) 5179 : audit [DBG] from='client.30199 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:48:06.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:05 smithi154 conmon[34553]: audit 2021-07-31T03:48:05.021371+0000 mgr.x (mgr.24542) 5180 : audit [DBG] from='client.30205 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:48:06.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:05 smithi154 conmon[34553]: audit 2021-07-31T03:48:05.364874+0000 mon.a (mon.0) 3369 2021-07-31T03:48:06.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:05 smithi154 conmon[34553]: : audit [DBG] from='client.? 172.21.15.45:0/3635400443' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:48:06.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:05 smithi045 conmon[33655]: audit 2021-07-31T03:48:04.741468+0000 mgr.x (mgr.24542) 2021-07-31T03:48:06.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:05 smithi045 conmon[33655]: 5179 : audit [DBG] from='client.30199 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:48:06.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:05 smithi045 conmon[33655]: audit 2021-07-31T03:48:05.021371+0000 mgr.x (mgr.24542) 5180 : audit [DBG] from='client.30205 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:48:06.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:05 smithi045 conmon[33655]: audit 2021-07-31T03:48:05.364874+0000 mon.a (mon.0) 3369 : audit [DBG] 2021-07-31T03:48:06.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:05 smithi045 conmon[33655]: from='client.? 172.21.15.45:0/3635400443' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:48:06.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:05 smithi045 conmon[39718]: audit 2021-07-31T03:48: 2021-07-31T03:48:06.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:05 smithi045 conmon[39718]: 04.741468+0000 mgr.x (mgr.24542) 5179 : audit [DBG] from='client.30199 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:48:06.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:05 smithi045 conmon[39718]: audit 2021-07-31T03:48:06.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:05 smithi045 conmon[39718]: 2021-07 2021-07-31T03:48:06.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:05 smithi045 conmon[39718]: -31T03:48:05 2021-07-31T03:48:06.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:05 smithi045 conmon[39718]: .021371+0000 mgr.x (mgr.24542 2021-07-31T03:48:06.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:05 smithi045 conmon[39718]: ) 5180 : audit [DBG] from='client.30205 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:48:06.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:05 smithi045 conmon[39718]: 2021-07-31T03:48:06.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:05 smithi045 conmon[39718]: audit 2021-07-31T03 2021-07-31T03:48:06.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:05 smithi045 conmon[39718]: :48:05.364874+0000 mon.a (mon.0) 3369 : audit [DBG] from='client.? 172.21.15.45:0/3635400443' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:48:07.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:06 smithi154 conmon[34553]: cluster 2021-07-31T03:48:05.521333+0000 mgr.x (mgr.24542) 5181 : cluster [DBG] pgmap v5634: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:07.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:06 smithi045 conmon[33655]: cluster 2021-07-31T03:48:05.521333+0000 mgr.x (mgr.24542) 5181 : cluster [DBG] pgmap v5634: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:07.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:06 smithi045 conmon[39718]: cluster 2021-07-31T03:48:05. 2021-07-31T03:48:07.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:06 smithi045 conmon[39718]: 521333+0000 mgr.x (mgr.24542) 5181 : cluster [DBG] pgmap v5634: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:09.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:08 smithi154 conmon[34553]: cluster 2021-07-31T03:48:07.521856+0000 mgr.x (mgr.24542) 5182 : cluster [DBG] pgmap v5635: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:09.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:08 smithi045 conmon[33655]: cluster 2021-07-31T03:48:07.521856+0000 mgr.x (mgr.24542) 5182 : cluster 2021-07-31T03:48:09.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:08 smithi045 conmon[33655]: [DBG] pgmap v5635: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:09.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:08 smithi045 conmon[39718]: cluster 2021-07-31T03:48:07.521856+0000 mgr.x (mgr.24542) 5182 : cluster [DBG] pgmap v5635: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:11.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:10 smithi154 conmon[34553]: cluster 2021-07-31T03:48:09.522469+0000 mgr.x (mgr.24542) 5183 : cluster [DBG] pgmap v5636: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:11.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:10 smithi045 conmon[33655]: cluster 2021-07-31T03:48:09.522469+0000 mgr.x (mgr.24542) 5183 : cluster [DBG] pgmap v5636: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:11.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:10 smithi045 conmon[33655]: 2021-07-31T03:48:11.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:10 smithi045 conmon[39718]: cluster 2021-07-31T03:48:09.522469+0000 mgr.x 2021-07-31T03:48:11.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:10 smithi045 conmon[39718]: (mgr.24542) 5183 : cluster [DBG] pgmap v5636: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:13.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:12 smithi154 conmon[34553]: cluster 2021-07-31T03:48:11.523089+0000 mgr.x (mgr.24542) 5184 : cluster [DBG] pgmap v5637: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:13.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:12 smithi045 conmon[33655]: cluster 2021-07-31T03:48:11.523089+0000 mgr.x (mgr.24542) 5184 : 2021-07-31T03:48:13.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:12 smithi045 conmon[33655]: cluster [DBG] pgmap v5637: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:13.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:12 smithi045 conmon[39718]: cluster 2021-07-31T03:48:11.523089+0000 mgr.x (mgr.24542 2021-07-31T03:48:13.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:12 smithi045 conmon[39718]: ) 5184 : cluster [DBG] pgmap v5637: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:15.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:14 smithi154 conmon[34553]: cluster 2021-07-31T03:48:13.523573+0000 mgr.x (mgr.24542) 5185 : 2021-07-31T03:48:15.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:14 smithi154 conmon[34553]: cluster [DBG] pgmap v5638: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:15.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:14 smithi045 conmon[39718]: cluster 2021-07-31T03:48:13.523573+0000 mgr.x (mgr.24542) 5185 : cluster [DBG] pgmap v5638: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:15.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:14 smithi045 conmon[33655]: cluster 2021-07-31T03:48:13.523573+0000 mgr.x (mgr.24542) 2021-07-31T03:48:15.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:14 smithi045 conmon[33655]: 5185 : cluster [DBG] pgmap v5638: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:17.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:16 smithi154 conmon[34553]: cluster 2021-07-31T03:48:15.524040+0000 mgr.x (mgr.24542) 5186 : cluster [DBG] pgmap v5639: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:17.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:16 smithi045 conmon[39718]: cluster 2021-07-31T03:48:15.524040+0000 mgr.x (mgr.24542) 5186 : cluster 2021-07-31T03:48:17.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:16 smithi045 conmon[39718]: [DBG] pgmap v5639: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:17.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:16 smithi045 conmon[33655]: cluster 2021-07-31T03:48:15.524040+0000 mgr.x (mgr.24542) 5186 2021-07-31T03:48:17.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:16 smithi045 conmon[33655]: : cluster [DBG] pgmap v5639: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:19.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:18 smithi154 conmon[34553]: cluster 2021-07-31T03:48:17.524367+0000 mgr.x (mgr.24542) 5187 : cluster [DBG] pgmap v5640: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:19.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:18 smithi045 conmon[39718]: cluster 2021-07-31T03:48:17.524367+0000 mgr.x (mgr 2021-07-31T03:48:19.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:18 smithi045 conmon[39718]: .24542) 5187 : cluster [DBG] pgmap v5640: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:19.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:18 smithi045 conmon[33655]: cluster 2021-07-31T03:48:17.524367+0000 2021-07-31T03:48:19.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:18 smithi045 conmon[33655]: mgr.x (mgr.24542) 5187 : cluster [DBG] pgmap v5640: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:21.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:20 smithi154 conmon[34553]: cluster 2021-07-31T03:48:19.524836+0000 mgr.x (mgr.24542) 5188 : cluster [DBG] pgmap v5641: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:21.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:20 smithi045 conmon[33655]: cluster 2021-07-31T03:48:19.524836+0000 mgr.x (mgr.24542) 5188 : cluster 2021-07-31T03:48:21.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:20 smithi045 conmon[33655]: [DBG] pgmap v5641: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:21.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:20 smithi045 conmon[39718]: cluster 2021-07-31T03:48:19.524836+0000 mgr.x (mgr. 2021-07-31T03:48:21.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:20 smithi045 conmon[39718]: 24542) 5188 : cluster [DBG] pgmap v5641: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:23.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:22 smithi154 conmon[34553]: cluster 2021-07-31T03:48:21.525247+0000 mgr.x (mgr.24542) 5189 : cluster [DBG] pgmap v5642: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:23.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:22 smithi154 conmon[34553]: 2021-07-31T03:48:23.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:22 smithi045 conmon[33655]: cluster 2021-07-31T03:48:21.525247+0000 mgr.x (mgr.24542) 5189 2021-07-31T03:48:23.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:22 smithi045 conmon[33655]: : cluster [DBG] pgmap v5642: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:23.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:22 smithi045 conmon[39718]: cluster 2021-07-31T03:48:21.525247+0000 mgr.x (mgr 2021-07-31T03:48:23.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:22 smithi045 conmon[39718]: .24542) 5189 : cluster [DBG] pgmap v5642: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:25.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:24 smithi154 conmon[34553]: cluster 2021-07-31T03:48:23.525732+0000 mgr.x (mgr.24542) 5190 : cluster [DBG] pgmap v5643: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:25.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:24 smithi045 conmon[33655]: cluster 2021-07-31T03:48:23.525732+0000 mgr.x ( 2021-07-31T03:48:25.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:24 smithi045 conmon[33655]: mgr.24542) 5190 : cluster [DBG] pgmap v5643: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:25.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:24 smithi045 conmon[39718]: cluster 2021-07-31T03:48:23.525732+0000 mgr.x 2021-07-31T03:48:25.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:24 smithi045 conmon[39718]: (mgr.24542) 5190 : cluster [DBG] pgmap v5643: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:27.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:26 smithi154 conmon[34553]: cluster 2021-07-31T03:48:25.526189+0000 mgr.x (mgr.24542) 2021-07-31T03:48:27.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:26 smithi154 conmon[34553]: 5191 : cluster [DBG] pgmap v5644: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:27.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:26 smithi045 conmon[39718]: cluster 2021-07-31T03:48:25.526189+0000 mgr.x ( 2021-07-31T03:48:27.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:26 smithi045 conmon[39718]: mgr.24542) 5191 : cluster [DBG] pgmap v5644: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:27.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:26 smithi045 conmon[33655]: cluster 2021-07-31T03:48:25.526189 2021-07-31T03:48:27.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:26 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 5191 : cluster [DBG] pgmap v5644: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:29.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:29 smithi154 conmon[34553]: cluster 2021-07-31T03:48:27.526731+0000 mgr.x (mgr.24542) 5192 : cluster [DBG] 2021-07-31T03:48:29.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:29 smithi154 conmon[34553]: pgmap v5645: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:29.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:29 smithi154 conmon[34553]: audit 2021-07-31T03:48:28.416316+0000 mon.b (mon.2) 680 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:48:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:29 smithi154 conmon[34553]: audit 2021-07-31T03:48:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:29 smithi154 conmon[34553]: 2021-07-31T03:48:28.416585+0000 mon.b (mon.2) 681 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:48:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:29 smithi154 conmon[34553]: audit 2021-07- 2021-07-31T03:48:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:29 smithi154 conmon[34553]: 31T03:48:28.417200+0000 mon.a (mon.0) 2021-07-31T03:48:29.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:29 smithi154 conmon[34553]: 3370 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:48:29.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:29 smithi154 conmon[34553]: audit 2021-07-31T03:48:29.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:29 smithi154 conmon[34553]: 2021-07-31T03:48:28.417588+0000 mon.a 2021-07-31T03:48:29.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:29 smithi154 conmon[34553]: (mon.0) 3371 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:48:29.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:29 smithi045 conmon[39718]: cluster 2021-07-31T03:48: 2021-07-31T03:48:29.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:29 smithi045 conmon[39718]: 27.526731+0000 mgr.x (mgr.24542) 5192 : cluster [DBG] pgmap v5645: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:29.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:29 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T03:48:29.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:29 smithi045 conmon[39718]: 03:48:28.416316+0000 mon.b (mon.2) 680 : 2021-07-31T03:48:29.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:29 smithi045 conmon[39718]: audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:48:29.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:29 smithi045 conmon[39718]: audit 2021-07-31T03:48:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:29 smithi045 conmon[39718]: 2021-07-31T03:48:28.416585 2021-07-31T03:48:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:29 smithi045 conmon[39718]: +0000 mon.b (mon.2) 2021-07-31T03:48:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:29 smithi045 conmon[39718]: 681 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:48:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:29 smithi045 conmon[39718]: audit 2021- 2021-07-31T03:48:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:29 smithi045 conmon[39718]: 07-31T03:48:28.417200 2021-07-31T03:48:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:29 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2021-07-31T03:48:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:29 smithi045 conmon[39718]: 3370 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:48:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:29 smithi045 conmon[39718]: audit 2021 2021-07-31T03:48:29.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:29 smithi045 conmon[39718]: -07-31T03:48:28 2021-07-31T03:48:29.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:29 smithi045 conmon[39718]: .417588+0000 mon.a (mon. 2021-07-31T03:48:29.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:29 smithi045 conmon[39718]: 0) 3371 : audit [INF] 2021-07-31T03:48:29.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:29 smithi045 conmon[39718]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:48:29.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:29 smithi045 conmon[33655]: cluster 2021-07-31T03:48:27.526731+0000 mgr.x (mgr.24542) 5192 : cluster [DBG] pgmap v5645: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:29.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:29 smithi045 conmon[33655]: audit 2021-07-31T03:48:28.416316+0000 mon.b (mon.2) 680 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:48:29.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:29 smithi045 conmon[33655]: 2021-07-31T03:48:29.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:29 smithi045 conmon[33655]: audit 2021 2021-07-31T03:48:29.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:29 smithi045 conmon[33655]: -07-31T03:48:28.416585 2021-07-31T03:48:29.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:29 smithi045 conmon[33655]: +0000 mon.b (mon.2) 2021-07-31T03:48:29.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:29 smithi045 conmon[33655]: 681 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:48:29.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:29 smithi045 conmon[33655]: 2021-07-31T03:48:29.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:29 smithi045 conmon[33655]: audit 2021-07-31T03:48:29.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:29 smithi045 conmon[33655]: 2021-07-31T03:48:28.417200 2021-07-31T03:48:29.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:29 smithi045 conmon[33655]: +0000 mon.a (mon.0) 3370 : audit 2021-07-31T03:48:29.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:29 smithi045 conmon[33655]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:48:29.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:29 smithi045 conmon[33655]: audit 2021-07-31T03:48:28.417588+0000 2021-07-31T03:48:29.283 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:29 smithi045 conmon[33655]: mon.a (mon.0) 3371 : audit [INF] 2021-07-31T03:48:29.283 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:29 smithi045 conmon[33655]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:48:31.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:31 smithi045 conmon[39718]: cluster 2021-07-31T03:48:29.527188+0000 mgr.x ( 2021-07-31T03:48:31.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:31 smithi045 conmon[39718]: mgr.24542) 5193 : cluster [DBG] pgmap v5646: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:31.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:31 smithi045 conmon[33655]: cluster 2021-07-31T03:48:29.527188+0000 mgr.x (mgr.24542) 5193 : cluster [DBG] pgmap v5646: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:31.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:31 smithi154 conmon[34553]: cluster 2021-07-31T03:48:29.527188+0000 mgr.x (mgr.24542) 2021-07-31T03:48:31.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:31 smithi154 conmon[34553]: 5193 : cluster [DBG] pgmap v5646: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:33.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:33 smithi045 conmon[39718]: cluster 2021-07-31T03:48:31.527534+0000 mgr.x (mgr.24542) 5194 : 2021-07-31T03:48:33.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:33 smithi045 conmon[39718]: cluster [DBG] pgmap v5647: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:33.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:33 smithi045 conmon[33655]: cluster 2021-07-31T03:48:31.527534+0000 mgr.x (mgr.24542) 5194 : cluster [DBG] pgmap v5647: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:33.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:33 smithi045 conmon[33655]: 2021-07-31T03:48:33.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:33 smithi154 conmon[34553]: cluster 2021-07-31T03:48:31.527534+0000 mgr.x (mgr.24542) 5194 2021-07-31T03:48:33.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:33 smithi154 conmon[34553]: : cluster [DBG] pgmap v5647: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:35.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[34553]: cluster 2021-07-31T03:48:33.527994+0000 mgr.x (mgr.24542) 5195 : cluster 2021-07-31T03:48:35.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[34553]: [DBG] pgmap v5648: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:35.263 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:35 smithi045 conmon[33655]: cluster 2021-07-31T03:48:33.527994+0000 mgr.x (mgr.24542) 2021-07-31T03:48:35.263 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:35 smithi045 conmon[33655]: 5195 : cluster [DBG] pgmap v5648: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:35.263 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:35 smithi045 conmon[39718]: cluster 2021-07-31T03:48:33.527994 2021-07-31T03:48:35.263 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:35 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 5195 : cluster [DBG] pgmap v5648: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:35.676 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:48:35.752 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: debug 2021-07-31T03:48:35.261+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:48:35.753 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:48:35.753 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:48:35.753 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: yield 2021-07-31T03:48:35.753 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:48:35.753 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:48:35.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:48:35.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:48:35.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:48:35.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:48:35.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:48:35.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:48:35.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:48:35.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: task.result() 2021-07-31T03:48:35.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:48:35.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: debug 2021-07-31T03:48:35.273+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:48:35.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:48:35.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:48:35.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: yield 2021-07-31T03:48:35.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:48:35.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:48:35.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:48:35.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:48:35.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:48:35.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:48:35.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:48:35.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:48:35.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:48:35.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: task.result() 2021-07-31T03:48:35.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:48:35.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: debug 2021-07-31T03:48:35.298+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:48:35.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:48:35.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:48:35.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: yield 2021-07-31T03:48:35.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:48:35.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:48:35.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:48:35.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:48:35.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:48:35.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:48:35.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:48:35.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:48:35.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:48:35.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: task.result() 2021-07-31T03:48:35.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:48:35.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: debug 2021-07-31T03:48:35.298+0000 7fcb9295f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:48:35.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:48:35.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:48:35.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: yield 2021-07-31T03:48:35.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:48:35.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:48:35.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:48:35.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:48:35.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:48:35.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:48:35.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:48:35.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:48:35.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:48:35.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: task.result() 2021-07-31T03:48:35.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:48:35.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: 2021-07-31T03:48:35.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:48:35.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: 2021-07-31T03:48:35.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:48:35.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:48:35.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:48:35.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:48:35.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:48:35.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:48:35.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:48:35.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:48:35.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:48:35.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:48:35.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:48:35.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:48:35.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:48:35.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:48:35.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:48:35.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:48:35.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:48:35.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:48:35.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: raise 2021-07-31T03:48:35.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:48:35.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:48:35.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:48:35.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:48:35.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:48:35.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: debug 2021-07-31T03:48:35.307+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:48:35.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:48:35.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:48:35.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: yield 2021-07-31T03:48:35.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:48:35.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:48:35.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:48:35.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:48:35.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:48:35.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:48:35.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:48:35.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:48:35.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:48:35.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: task.result() 2021-07-31T03:48:35.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:48:35.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: debug 2021-07-31T03:48:35.307+0000 7fcb93160700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:48:35.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:48:35.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:48:35.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: yield 2021-07-31T03:48:35.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:48:35.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:48:35.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:48:35.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:48:35.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:48:35.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:48:35.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:48:35.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:48:35.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:48:35.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: task.result() 2021-07-31T03:48:35.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:48:35.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: 2021-07-31T03:48:35.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:48:35.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: 2021-07-31T03:48:35.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:48:35.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:48:35.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:48:35.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:48:35.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:48:35.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:48:35.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:48:35.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:48:35.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:48:35.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:48:35.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:48:35.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:48:35.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:48:35.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:48:35.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:48:35.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:48:35.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:48:35.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:48:35.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: raise 2021-07-31T03:48:35.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:48:35.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:48:35.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:48:35.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:48:35.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:48:35 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:48:35.954 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:48:35.954 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 2h ago 2h - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:48:35.954 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 2h ago 2h 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:48:35.955 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 2h ago 2h 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:48:35.955 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 2h ago 2h - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:48:35.955 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:48:35.955 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 2h ago 2h 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:48:35.955 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:48:35.956 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 2h ago 2h - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:48:35.956 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 2h ago 2h 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:48:35.956 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:48:35.956 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:48:35.956 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:48:35.957 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:48:35.957 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 2h ago 2h 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:48:35.957 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 2h ago 2h 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:48:35.957 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 2h ago 2h 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:48:35.957 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 2h ago 2h 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:48:35.958 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 2h ago 2h 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:48:36.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:36 smithi045 conmon[39718]: audit 2021-07-31T03:48:35 2021-07-31T03:48:36.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:36 smithi045 conmon[39718]: .216444+0000 mon.b (mon.2) 682 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:48:36.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:36 smithi045 conmon[39718]: audit 2021- 2021-07-31T03:48:36.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:36 smithi045 conmon[39718]: 07-31T03:48: 2021-07-31T03:48:36.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:36 smithi045 conmon[39718]: 35.217900+0000 mon.b (mon.2 2021-07-31T03:48:36.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:36 smithi045 conmon[39718]: ) 683 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:48:36.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:36 smithi045 conmon[39718]: audit 2021-07-31T03:48:36.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:36 smithi045 conmon[39718]: 2021-07-31T03: 2021-07-31T03:48:36.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:36 smithi045 conmon[39718]: 48:35.218793+0000 mon.b (mon 2021-07-31T03:48:36.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:36 smithi045 conmon[39718]: .2) 684 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:48:36.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:36 smithi045 conmon[33655]: audit 2021-07- 2021-07-31T03:48:36.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:36 smithi045 conmon[33655]: 31T03:48:35.216444+0000 mon.b (mon.2) 682 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:48:36.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:36 smithi045 conmon[33655]: audit 2021-07-31 2021-07-31T03:48:36.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:36 smithi045 conmon[33655]: T03:48:35.217900+0000 mon.b (mon. 2021-07-31T03:48:36.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:36 smithi045 conmon[33655]: 2) 683 : audit [DBG] 2021-07-31T03:48:36.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:36 smithi045 conmon[33655]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:48:36.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:36 smithi045 conmon[33655]: audit 2021 2021-07-31T03:48:36.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:36 smithi045 conmon[33655]: -07-31T03:48 2021-07-31T03:48:36.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:36 smithi045 conmon[33655]: :35.218793 2021-07-31T03:48:36.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:36 smithi045 conmon[33655]: +0000 mon.b 2021-07-31T03:48:36.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:36 smithi045 conmon[33655]: (mon.2) 684 : audit 2021-07-31T03:48:36.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:36 smithi045 conmon[33655]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:48:36.292 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:48:36.292 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:48:36.293 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:48:36.293 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:48:36.293 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:48:36.293 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:48:36.293 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:48:36.294 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:48:36.294 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:48:36.294 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:48:36.294 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:48:36.295 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:48:36.295 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:48:36.295 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:48:36.295 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:48:36.295 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:48:36.296 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:48:36.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:36 smithi154 conmon[34553]: audit 2021-07-31T03:48:35.216444+0000 mon.b (mon.2) 682 : audit [DBG] 2021-07-31T03:48:36.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:36 smithi154 conmon[34553]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:48:36.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:36 smithi154 conmon[34553]: audit 2021-07-31T03:48:35.217900+0000 mon.b (mon.2) 683 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:48:36.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:36 smithi154 conmon[34553]: audit 2021-07-31T03:48:36.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:36 smithi154 conmon[34553]: 2021-07-31T03:48:35.218793+0000 mon.b (mon.2) 684 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:48:37.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: cephadm 2021-07-31T03:48:35.219633+0000 mgr.x (mgr.24542) 5196 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:48:37.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: 2021-07-31T03:48:37.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: cephadm 2021-07-31T03:48:35.262670+0000 mgr.x (mgr.24542) 5197 : cephadm [ERR] 2021-07-31T03:48:37.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:48:37.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:48:37.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: yield 2021-07-31T03:48:37.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:48:37.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:48:37.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:48:37.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:48:37.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:48:37.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:48:37.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:48:37.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:48:37.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:48:37.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: task.result() 2021-07-31T03:48:37.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:48:37.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: cephadm 2021-07-31T03:48:37.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: 2021-07-31T03:48:35.263159+0000 mgr.x (mgr.24542) 5198 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:48:37.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: cephadm 2021-07-31T03:48 2021-07-31T03:48:37.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: :35.274139+0000 mgr.x (mgr.24542) 2021-07-31T03:48:37.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: 5199 : cephadm [ERR] 2021-07-31T03:48:37.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:48:37.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:48:37.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: yield 2021-07-31T03:48:37.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:48:37.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:48:37.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:48:37.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:48:37.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:48:37.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:48:37.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:48:37.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:48:37.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:48:37.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: task.result() 2021-07-31T03:48:37.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:48:37.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: cephadm 2021-07-31T03:48:35.275004 2021-07-31T03:48:37.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 5200 : 2021-07-31T03:48:37.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:48:37.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: cephadm 2021- 2021-07-31T03:48:37.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: 07-31T03:48:35.299126+0000 2021-07-31T03:48:37.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: mgr.x (mgr.24542) 5201 2021-07-31T03:48:37.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: : cephadm [ERR] 2021-07-31T03:48:37.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:48:37.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:48:37.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: yield 2021-07-31T03:48:37.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:48:37.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:48:37.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:48:37.040 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:48:37.040 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:48:37.040 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:48:37.040 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:48:37.040 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:48:37.041 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:48:37.041 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: task.result() 2021-07-31T03:48:37.041 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:48:37.041 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: cephadm 2021-07-31T03:48:35.299698 2021-07-31T03:48:37.041 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 5202 2021-07-31T03:48:37.042 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:48:37.042 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:48:37.042 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:48:37.042 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: yield 2021-07-31T03:48:37.043 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:48:37.043 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:48:37.043 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:48:37.043 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:48:37.043 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:48:37.044 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:48:37.044 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:48:37.044 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:48:37.044 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:48:37.044 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: task.result() 2021-07-31T03:48:37.045 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:48:37.045 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: 2021-07-31T03:48:37.045 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:48:37.045 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: 2021-07-31T03:48:37.045 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:48:37.045 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:48:37.046 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:48:37.046 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:48:37.046 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:48:37.046 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:48:37.047 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:48:37.047 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:48:37.047 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:48:37.047 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:48:37.047 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:48:37.048 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:48:37.048 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:48:37.048 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:48:37.048 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:48:37.048 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:48:37.049 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:48:37.049 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:48:37.049 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: raise 2021-07-31T03:48:37.049 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:48:37.049 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:48:37.050 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:48:37.050 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:48:37.050 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:48:37.050 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: cephadm 2021-07-31T03:48:35.308186+0000 mgr.x (mgr.24542) 5203 : cephadm [ERR] 2021-07-31T03:48:37.051 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:48:37.051 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:48:37.051 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: yield 2021-07-31T03:48:37.051 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:48:37.051 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:48:37.051 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:48:37.052 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:48:37.052 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:48:37.052 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:48:37.052 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:48:37.052 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:48:37.053 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:48:37.053 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: task.result() 2021-07-31T03:48:37.053 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:48:37.053 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: cephadm 2021-07-31T03:48:35.308571+0000 mgr.x (mgr.24542) 5204 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:48:37.053 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:48:37.054 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:48:37.054 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: yield 2021-07-31T03:48:37.054 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:48:37.054 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:48:37.054 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:48:37.055 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:48:37.055 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:48:37.055 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:48:37.055 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:48:37.055 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:48:37.056 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:48:37.056 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: task.result() 2021-07-31T03:48:37.056 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:48:37.056 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: 2021-07-31T03:48:37.056 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:48:37.057 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: 2021-07-31T03:48:37.057 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:48:37.057 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:48:37.057 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:48:37.057 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:48:37.058 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:48:37.058 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:48:37.058 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:48:37.058 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:48:37.058 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:48:37.059 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:48:37.059 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:48:37.059 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:48:37.059 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:48:37.059 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:48:37.060 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:48:37.060 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:48:37.060 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:48:37.060 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:48:37.060 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: raise 2021-07-31T03:48:37.060 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:48:37.061 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:48:37.061 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:48:37.061 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:48:37.061 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:48:37.061 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: 2021-07-31T03:48:37.062 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: cluster 2021-07-31T03:48:35.528511+0000 mgr.x (mgr.24542) 5205 : cluster [DBG] pgmap v5649: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:37.062 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: audit 2021-07-31T03:48:35.664905+0000 mgr.x (mgr.24542) 5206 : audit [DBG] from='client.30217 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:48:37.062 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: audit 2021-07-31T03:48:35.947049 2021-07-31T03:48:37.062 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 5207 : audit [DBG] from='client.30223 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:48:37.063 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: audit 2021-07-31T 2021-07-31T03:48:37.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: cephadm 2021-07-31T03:48:35.219633+0000 mgr.x (mgr.24542 2021-07-31T03:48:37.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: ) 5196 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:48:37.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: cephadm 2021-07-31T03:48:35.262670+0000 2021-07-31T03:48:37.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: mgr.x (mgr.24542) 5197 : cephadm [ERR] 2021-07-31T03:48:37.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:48:37.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:48:37.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: yield 2021-07-31T03:48:37.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:48:37.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:48:37.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:48:37.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:48:37.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:48:37.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:48:37.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:48:37.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:48:37.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:48:37.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: task.result() 2021-07-31T03:48:37.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:48:37.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: cephadm 2021-07-31T03:48:35.263159+0000 mgr.x (mgr.24542) 5198 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:48:37.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: cephadm 2021-07-31T03:48:35.274139+0000 mgr.x (mgr 2021-07-31T03:48:37.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: .24542) 5199 : cephadm [ERR] 2021-07-31T03:48:37.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:48:37.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:48:37.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: yield 2021-07-31T03:48:37.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:48:37.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:48:37.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:48:37.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:48:37.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:48:37.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:48:37.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:48:37.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:48:37.282 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:48:37.282 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: task.result() 2021-07-31T03:48:37.282 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:48:37.282 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: cephadm 2021-07-31T03:48:35.275004+0000 mgr.x (mgr.24542) 5200 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:48:37.282 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: cephadm 2021-07-31T03:48:35.299126+0000 mgr.x (mgr.24542) 5201 : cephadm [ERR] 2021-07-31T03:48:37.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: 2021-07-31T03:48:37.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:48:37.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:48:37.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: yield 2021-07-31T03:48:37.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:48:37.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:48:37.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:48:37.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:48:37.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:48:37.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:48:37.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:48:37.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:48:37.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:48:37.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: task.result() 2021-07-31T03:48:37.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:48:37.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: cephadm 2021-07-31T03:48:35.299698+0000 mgr.x (mgr.24542) 5202 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:48:37.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:48:37.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:48:37.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: yield 2021-07-31T03:48:37.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:48:37.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:48:37.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:48:37.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:48:37.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:48:37.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:48:37.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:48:37.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:48:37.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:48:37.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: task.result() 2021-07-31T03:48:37.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:48:37.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: 2021-07-31T03:48:37.289 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:48:37.289 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: 2021-07-31T03:48:37.289 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:48:37.289 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:48:37.289 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:48:37.290 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:48:37.290 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:48:37.290 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:48:37.290 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:48:37.290 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:48:37.291 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:48:37.291 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:48:37.291 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: 03:48:36.293380+0000 mon.a (mon 2021-07-31T03:48:37.291 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: .0) 3372 : audit [DBG] from='client.? 172.21.15.45:0/3602354454' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:48:37.292 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[33655]: 2021-07-31T03:48:37.292 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:48:37.292 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:48:37.293 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:48:37.293 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:48:37.293 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:48:37.293 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:48:37.293 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:48:37.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:48:37.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: raise 2021-07-31T03:48:37.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:48:37.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:48:37.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:48:37.295 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:48:37.295 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:48:37.295 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: cephadm 2021-07-31T03:48:35.308186+0000 mgr.x (mgr. 2021-07-31T03:48:37.295 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: 24542) 5203 : cephadm [ERR] 2021-07-31T03:48:37.295 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:48:37.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:48:37.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: yield 2021-07-31T03:48:37.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:48:37.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:48:37.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:48:37.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:48:37.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:48:37.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:48:37.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:48:37.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:48:37.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:48:37.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: task.result() 2021-07-31T03:48:37.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:48:37.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: cephadm 2021-07-31T03:48:35.308571+0000 mgr.x (mgr.24542) 5204 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:48:37.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:48:37.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:48:37.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: yield 2021-07-31T03:48:37.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:48:37.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:48:37.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:48:37.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:48:37.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:48:37.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:48:37.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:48:37.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:48:37.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:48:37.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: task.result() 2021-07-31T03:48:37.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:48:37.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: 2021-07-31T03:48:37.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:48:37.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: 2021-07-31T03:48:37.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:48:37.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:48:37.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:48:37.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:48:37.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:48:37.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:48:37.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:48:37.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:48:37.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:48:37.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:48:37.304 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:48:37.304 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:48:37.304 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:48:37.304 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:48:37.304 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:48:37.305 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:48:37.305 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:48:37.305 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:48:37.305 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: raise 2021-07-31T03:48:37.305 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:48:37.305 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:48:37.306 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:48:37.306 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:48:37.306 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:48:37.306 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: cluster 2021-07-31T03:48:35.528511+0000 mgr.x (mgr.24542) 5205 : cluster [DBG] pgmap v5649: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:37.306 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: audit 2021-07-31T03:48:35.664905+0000 mgr.x (mgr.24542) 5206 : audit [DBG] from='client.30217 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:48:37.307 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: audit 2021-07-31T03:48 2021-07-31T03:48:37.307 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: :35.947049+0000 mgr.x (mgr.24542) 5207 : audit [DBG] from='client.30223 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:48:37.307 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:37 smithi045 conmon[39718]: audit 2021-07-31T03:48:36.293380+0000 mon.a (mon.0) 3372 : audit [DBG] from='client.? 172.21.15.45:0/3602354454' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:48:37.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: cephadm 2021-07-31T03:48:35.219633+0000 mgr.x (mgr.24542) 5196 : cephadm [WRN] 2021-07-31T03:48:37.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:48:37.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: cephadm 2021-07-31T03:48:35.262670+0000 mgr.x (mgr.24542) 5197 : cephadm [ERR] 2021-07-31T03:48:37.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:48:37.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:48:37.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: yield 2021-07-31T03:48:37.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:48:37.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:48:37.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:48:37.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:48:37.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:48:37.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:48:37.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:48:37.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:48:37.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:48:37.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: task.result() 2021-07-31T03:48:37.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:48:37.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: cephadm 2021-07-31T03:48:35.263159+0000 mgr.x (mgr.24542) 5198 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:48:37.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: cephadm 2021-07-31T03:48:35.274139+0000 mgr.x (mgr.24542) 5199 : cephadm [ERR] 2021-07-31T03:48:37.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:48:37.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:48:37.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: yield 2021-07-31T03:48:37.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:48:37.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:48:37.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:48:37.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:48:37.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:48:37.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:48:37.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:48:37.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:48:37.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:48:37.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: task.result() 2021-07-31T03:48:37.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:48:37.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: cephadm 2021-07-31T03:48:35.275004+0000 mgr.x (mgr.24542) 5200 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:48:37.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: cephadm 2021-07-31T03:48:35.299126+0000 mgr.x (mgr.24542) 5201 : cephadm [ERR] 2021-07-31T03:48:37.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:48:37.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:48:37.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: yield 2021-07-31T03:48:37.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:48:37.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:48:37.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:48:37.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:48:37.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:48:37.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:48:37.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:48:37.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:48:37.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:48:37.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: task.result() 2021-07-31T03:48:37.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:48:37.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: cephadm 2021-07-31T03:48:35.299698+0000 mgr.x (mgr.24542) 5202 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:48:37.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:48:37.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:48:37.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: yield 2021-07-31T03:48:37.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:48:37.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:48:37.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:48:37.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:48:37.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:48:37.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:48:37.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:48:37.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:48:37.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:48:37.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: task.result() 2021-07-31T03:48:37.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:48:37.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: 2021-07-31T03:48:37.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:48:37.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: 2021-07-31T03:48:37.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:48:37.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:48:37.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:48:37.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:48:37.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:48:37.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:48:37.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:48:37.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:48:37.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:48:37.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:48:37.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:48:37.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:48:37.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:48:37.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:48:37.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:48:37.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:48:37.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:48:37.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:48:37.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: raise 2021-07-31T03:48:37.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:48:37.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:48:37.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:48:37.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:48:37.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:48:37.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: cephadm 2021-07-31T03:48:35.308186+0000 mgr.x (mgr.24542 2021-07-31T03:48:37.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: ) 5203 : cephadm [ERR] 2021-07-31T03:48:37.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:48:37.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:48:37.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: yield 2021-07-31T03:48:37.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:48:37.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:48:37.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:48:37.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:48:37.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:48:37.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:48:37.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:48:37.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:48:37.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:48:37.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: task.result() 2021-07-31T03:48:37.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:48:37.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: cephadm 2021-07-31T03:48:35.308571+0000 mgr.x (mgr.24542) 5204 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:48:37.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:48:37.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:48:37.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: yield 2021-07-31T03:48:37.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:48:37.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:48:37.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:48:37.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:48:37.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:48:37.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:48:37.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:48:37.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:48:37.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:48:37.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: task.result() 2021-07-31T03:48:37.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:48:37.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: 2021-07-31T03:48:37.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:48:37.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: 2021-07-31T03:48:37.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:48:37.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:48:37.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:48:37.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:48:37.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:48:37.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:48:37.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:48:37.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:48:37.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:48:37.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:48:37.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:48:37.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:48:37.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:48:37.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:48:37.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:48:37.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:48:37.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:48:37.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:48:37.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: raise 2021-07-31T03:48:37.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:48:37.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:48:37.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:48:37.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:48:37.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:48:37.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: cluster 2021-07-31T03:48:35.528511+0000 mgr.x (mgr.24542) 5205 : cluster [DBG] pgmap v5649: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:37.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: audit 2021-07-31T03:48:35.664905+0000 mgr.x (mgr.24542) 5206 : audit [DBG] from='client.30217 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:48:37.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: audit 2021-07-31 2021-07-31T03:48:37.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: T03:48:35.947049+0000 mgr.x (mgr.24542) 5207 : audit [DBG] from='client.30223 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:48:37.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:37 smithi154 conmon[34553]: audit 2021-07-31T03:48:36.293380+0000 mon.a (mon.0) 3372 : audit [DBG] from='client.? 172.21.15.45:0/3602354454' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:48:39.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:39 smithi045 conmon[39718]: cluster 2021-07-31T03:48:37.529028+0000 mgr.x ( 2021-07-31T03:48:39.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:39 smithi045 conmon[39718]: mgr.24542) 5208 : cluster [DBG] pgmap v5650: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:39.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:39 smithi045 conmon[33655]: cluster 2021-07-31T03:48:37. 2021-07-31T03:48:39.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:39 smithi045 conmon[33655]: 529028+0000 mgr.x (mgr.24542) 5208 : cluster [DBG] pgmap v5650: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:39.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:39 smithi154 conmon[34553]: cluster 2021-07-31T03:48:37.529028+0000 mgr.x (mgr.24542) 5208 : cluster [DBG] 2021-07-31T03:48:39.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:39 smithi154 conmon[34553]: pgmap v5650: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:41.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:41 smithi045 conmon[39718]: cluster 2021-07-31T03:48:39.529470+0000 mgr.x ( 2021-07-31T03:48:41.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:41 smithi045 conmon[39718]: mgr.24542) 5209 : cluster [DBG] pgmap v5651: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:41.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:41 smithi045 conmon[33655]: cluster 2021-07-31T03:48:39.529470 2021-07-31T03:48:41.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:41 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 5209 : cluster [DBG] pgmap v5651: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:41.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:41 smithi154 conmon[34553]: cluster 2021-07-31T03:48:39.529470+0000 mgr.x (mgr.24542) 5209 : cluster [DBG] pgmap v5651: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:41.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:41 smithi154 conmon[34553]: 2021-07-31T03:48:43.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:43 smithi154 conmon[34553]: cluster 2021-07-31T03:48:41.529851+0000 mgr.x (mgr.24542) 5210 : cluster [DBG] pgmap v5652: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:43.524 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:43 smithi045 conmon[33655]: cluster 2021-07-31T03:48:41.529851+0000 mgr.x (mgr.24542) 5210 2021-07-31T03:48:43.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:43 smithi045 conmon[33655]: : cluster [DBG] pgmap v5652: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:43.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:43 smithi045 conmon[39718]: cluster 2021-07-31T03:48:41.529851+0000 mgr.x (mgr 2021-07-31T03:48:43.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:43 smithi045 conmon[39718]: .24542) 5210 : cluster [DBG] pgmap v5652: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:45.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:45 smithi154 conmon[34553]: cluster 2021-07-31T03:48:43.530238+0000 mgr.x (mgr.24542) 5211 : cluster [DBG] pgmap v5653: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:45.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:45 smithi045 conmon[33655]: cluster 2021-07-31T03:48:43.530238+0000 mgr.x (mgr.24542) 5211 : cluster [DBG] 2021-07-31T03:48:45.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:45 smithi045 conmon[33655]: pgmap v5653: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:45.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:45 smithi045 conmon[39718]: cluster 2021-07-31T03:48:43.530238+0000 mgr.x (mgr 2021-07-31T03:48:45.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:45 smithi045 conmon[39718]: .24542) 5211 : cluster [DBG] pgmap v5653: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:47.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:47 smithi154 conmon[34553]: cluster 2021-07-31T03:48:45.530771+0000 mgr.x (mgr.24542) 5212 : cluster [DBG] pgmap v5654: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:47.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:47 smithi045 conmon[39718]: cluster 2021-07-31T03:48:45.530771+0000 mgr.x (mgr.24542 2021-07-31T03:48:47.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:47 smithi045 conmon[39718]: ) 5212 : cluster [DBG] pgmap v5654: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:47.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:47 smithi045 conmon[33655]: cluster 2021-07-31T03:48:45.530771+0000 mgr.x (mgr.24542) 5212 : cluster [DBG] pgmap v5654: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:49.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:49 smithi154 conmon[34553]: cluster 2021-07-31T03:48:47.531228+0000 mgr.x (mgr.24542) 5213 : cluster [DBG] pgmap v5655: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:49.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:49 smithi045 conmon[39718]: cluster 2021-07-31T03:48:47.531228+0000 mgr.x (mgr.24542) 5213 : 2021-07-31T03:48:49.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:49 smithi045 conmon[39718]: cluster [DBG] pgmap v5655: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:49.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:49 smithi045 conmon[33655]: cluster 2021-07-31T03:48:47.531228+0000 mgr.x (mgr.24542) 5213 : 2021-07-31T03:48:49.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:49 smithi045 conmon[33655]: cluster [DBG] pgmap v5655: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:51.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:51 smithi154 conmon[34553]: cluster 2021-07-31T03:48:49.531759+0000 mgr.x (mgr.24542) 5214 : cluster [DBG] pgmap v5656: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:51.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:51 smithi045 conmon[33655]: cluster 2021-07-31T03:48:49.531759+0000 mgr.x (mgr.24542 2021-07-31T03:48:51.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:51 smithi045 conmon[33655]: ) 5214 : cluster [DBG] pgmap v5656: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:51.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:51 smithi045 conmon[39718]: cluster 2021-07-31T03:48:49.531759+0000 mgr.x ( 2021-07-31T03:48:51.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:51 smithi045 conmon[39718]: mgr.24542) 5214 : cluster [DBG] pgmap v5656: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:53.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:53 smithi154 conmon[34553]: cluster 2021-07-31T03:48:51.532241+0000 mgr.x (mgr.24542) 5215 : cluster [DBG] pgmap v5657: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:53.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:53 smithi045 conmon[33655]: cluster 2021-07-31T03:48:51.532241+0000 mgr.x (mgr.24542) 5215 : cluster 2021-07-31T03:48:53.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:53 smithi045 conmon[33655]: [DBG] pgmap v5657: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:53.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:53 smithi045 conmon[39718]: cluster 2021-07-31T03:48:51.532241+0000 mgr.x ( 2021-07-31T03:48:53.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:53 smithi045 conmon[39718]: mgr.24542) 5215 : cluster [DBG] pgmap v5657: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:55.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:55 smithi154 conmon[34553]: cluster 2021-07-31T03:48:53.532757+0000 mgr.x (mgr.24542) 5216 : cluster [DBG] pgmap v5658: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:55.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:55 smithi045 conmon[33655]: cluster 2021-07-31T03:48:53.532757+0000 mgr.x (mgr.24542) 5216 : cluster [DBG] pgmap v5658: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:55.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:55 smithi045 conmon[39718]: cluster 2021-07-31T03:48:53.532757+0000 mgr.x 2021-07-31T03:48:55.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:55 smithi045 conmon[39718]: (mgr.24542) 5216 : cluster [DBG] pgmap v5658: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:57.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:57 smithi154 conmon[34553]: cluster 2021-07-31T03:48:55.533229+0000 mgr.x (mgr.24542) 5217 : cluster [DBG] pgmap v5659: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:57.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:57 smithi045 conmon[33655]: cluster 2021-07-31T03:48:55.533229+0000 mgr.x (mgr.24542) 5217 : cluster [DBG] 2021-07-31T03:48:57.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:57 smithi045 conmon[33655]: pgmap v5659: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:57.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:57 smithi045 conmon[39718]: cluster 2021-07-31T03:48:55 2021-07-31T03:48:57.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:57 smithi045 conmon[39718]: .533229+0000 mgr.x (mgr.24542) 5217 : cluster [DBG] pgmap v5659: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:59.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:48:59 smithi154 conmon[34553]: cluster 2021-07-31T03:48:57.533589+0000 mgr.x (mgr.24542) 5218 : cluster [DBG] pgmap v5660: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:59.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:59 smithi045 conmon[33655]: cluster 2021-07-31T03:48:57.533589+0000 mgr.x (mgr.24542) 5218 : 2021-07-31T03:48:59.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:48:59 smithi045 conmon[33655]: cluster [DBG] pgmap v5660: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:48:59.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:59 smithi045 conmon[39718]: cluster 2021-07-31T03:48:57.533589+0000 mgr.x 2021-07-31T03:48:59.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:48:59 smithi045 conmon[39718]: (mgr.24542) 5218 : cluster [DBG] pgmap v5660: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:01.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:01 smithi154 conmon[34553]: cluster 2021-07-31T03:48:59.534114+0000 mgr.x (mgr.24542) 5219 : cluster [DBG] pgmap v5661: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:01.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:01 smithi045 conmon[39718]: cluster 2021-07-31T03:48:59.534114+0000 mgr.x (mgr.24542) 5219 : cluster [DBG] pgmap v5661: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:01.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:01 smithi045 conmon[33655]: cluster 2021-07-31T03:48:59.534114+0000 mgr.x (mgr. 2021-07-31T03:49:01.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:01 smithi045 conmon[33655]: 24542) 5219 : cluster [DBG] pgmap v5661: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:03.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:03 smithi154 conmon[34553]: cluster 2021-07-31T03:49:01.534606+0000 mgr.x (mgr.24542) 5220 : cluster [DBG] pgmap v5662: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:03.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:03 smithi045 conmon[33655]: cluster 2021-07-31T03:49:01.534606+0000 mgr.x (mgr.24542 2021-07-31T03:49:03.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:03 smithi045 conmon[33655]: ) 5220 : cluster [DBG] pgmap v5662: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:03.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:03 smithi045 conmon[39718]: cluster 2021-07-31T03:49:01.534606+0000 mgr.x ( 2021-07-31T03:49:03.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:03 smithi045 conmon[39718]: mgr.24542) 5220 : cluster [DBG] pgmap v5662: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:05.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:05 smithi154 conmon[34553]: cluster 2021-07-31T03:49:03.535125+0000 mgr.x (mgr.24542) 5221 : cluster [DBG] pgmap v5663: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:05.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:05 smithi045 conmon[39718]: cluster 2021-07-31T03:49:03.535125+0000 mgr.x (mgr.24542) 5221 : cluster [DBG] pgmap v5663: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:05.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:05 smithi045 conmon[33655]: cluster 2021-07-31T03:49:03.535125+0000 mgr.x (mgr.24542) 5221 : cluster [DBG] pgmap v5663: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:06.606 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:49:06.883 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:49:06.883 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 2h ago 2h - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:49:06.884 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 2h ago 2h 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:49:06.884 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 2h ago 2h 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:49:06.884 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 2h ago 2h - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:49:06.884 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:49:06.885 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 2h ago 2h 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:49:06.885 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:49:06.885 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 2h ago 2h - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:49:06.886 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 2h ago 2h 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:49:06.886 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:49:06.886 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:49:06.886 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:49:06.887 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:49:06.887 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 2h ago 2h 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:49:06.887 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 2h ago 2h 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:49:06.888 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 2h ago 2h 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:49:06.888 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 2h ago 2h 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:49:06.888 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 2h ago 2h 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:49:07.219 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:49:07.219 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:49:07.219 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:49:07.219 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:49:07.220 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:49:07.220 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:49:07.220 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:49:07.220 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:49:07.220 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:49:07.221 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:49:07.221 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:49:07.221 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:49:07.221 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:49:07.221 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:49:07.222 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:49:07.222 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:49:07.222 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:49:07.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:07 smithi154 conmon[34553]: cluster 2021-07-31T03:49:05.535727+0000 mgr.x (mgr.24542) 5222 : cluster [DBG] pgmap v5664: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:07.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:07 smithi045 conmon[33655]: cluster 2021-07-31T03:49:05.535727+0000 mgr.x 2021-07-31T03:49:07.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:07 smithi045 conmon[33655]: (mgr.24542) 5222 : cluster [DBG] pgmap v5664: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:07.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:07 smithi045 conmon[39718]: cluster 2021-07-31T03:49 2021-07-31T03:49:07.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:07 smithi045 conmon[39718]: :05.535727+0000 mgr.x (mgr.24542) 5222 : cluster [DBG] pgmap v5664: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:08.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:08 smithi154 conmon[34553]: audit 2021-07-31T03:49:06.594715+0000 mgr.x (mgr.24542) 5223 : audit [DBG] from='client.30235 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:49:08.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:08 smithi154 conmon[34553]: audit 2021-07-31T03:49:06.875381+0000 mgr.x (mgr.24542) 5224 : audit [DBG] from='client.20304 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:49:08.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:08 smithi154 conmon[34553]: audit 2021-07-31T03:49:07.219785 2021-07-31T03:49:08.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:08 smithi154 conmon[34553]: +0000 mon.c (mon.1) 228 : audit [DBG] from='client.? 172.21.15.45:0/1793228937' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:49:08.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:08 smithi045 conmon[39718]: audit 2021-07-31T03:49:06.594715+0000 mgr.x (mgr 2021-07-31T03:49:08.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:08 smithi045 conmon[39718]: .24542) 5223 : audit [DBG] from='client.30235 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:49:08.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:08 smithi045 conmon[39718]: audit 2021-07-31T03:49:06 2021-07-31T03:49:08.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:08 smithi045 conmon[39718]: .875381+0000 mgr.x (mgr.24542) 5224 : audit [DBG] from='client.20304 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:49:08.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:08 smithi045 conmon[39718]: audit 2021-07-31T03:49: 2021-07-31T03:49:08.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:08 smithi045 conmon[39718]: 07.219785+0000 mon.c (mon.1) 228 : audit 2021-07-31T03:49:08.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:08 smithi045 conmon[39718]: [DBG] from='client.? 172.21.15.45:0/1793228937' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:49:08.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:08 smithi045 conmon[33655]: audit 2021-07-31T03:49:06.594715+0000 2021-07-31T03:49:08.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:08 smithi045 conmon[33655]: mgr.x (mgr.24542) 5223 : audit [DBG] from='client.30235 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:49:08.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:08 smithi045 conmon[33655]: audit 2021-07-31T03:49:06.875381+0000 mgr.x ( 2021-07-31T03:49:08.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:08 smithi045 conmon[33655]: mgr.24542) 5224 : audit [DBG] from='client.20304 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:49:08.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:08 smithi045 conmon[33655]: audit 2021-07-31T03:49:07.219785 2021-07-31T03:49:08.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:08 smithi045 conmon[33655]: +0000 mon.c (mon.1) 228 : audit [DBG] from='client.? 172.21.15.45:0/1793228937' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:49:09.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:09 smithi154 conmon[34553]: cluster 2021-07-31T03:49:07.536166+0000 mgr.x (mgr.24542) 5225 : cluster [DBG] pgmap v5665: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:09.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:09 smithi154 conmon[34553]: 2021-07-31T03:49:09.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:09 smithi045 conmon[39718]: cluster 2021-07-31T03:49:07.536166+0000 mgr.x (mgr 2021-07-31T03:49:09.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:09 smithi045 conmon[39718]: .24542) 5225 : cluster [DBG] pgmap v5665: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:09.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:09 smithi045 conmon[33655]: cluster 2021-07-31T03:49:07.536166+0000 mgr.x (mgr. 2021-07-31T03:49:09.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:09 smithi045 conmon[33655]: 24542) 5225 : cluster [DBG] pgmap v5665: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:11.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:11 smithi154 conmon[34553]: cluster 2021-07-31T03:49:09.536563+0000 mgr.x (mgr.24542) 5226 : cluster [DBG] pgmap v5666: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:11.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:11 smithi045 conmon[33655]: cluster 2021-07-31T03:49:09.536563+0000 mgr.x (mgr.24542) 5226 : cluster 2021-07-31T03:49:11.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:11 smithi045 conmon[33655]: [DBG] pgmap v5666: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:11.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:11 smithi045 conmon[39718]: cluster 2021-07-31T03:49:09. 2021-07-31T03:49:11.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:11 smithi045 conmon[39718]: 536563+0000 mgr.x (mgr.24542) 5226 : cluster [DBG] pgmap v5666: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:13.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:13 smithi154 conmon[34553]: cluster 2021-07-31T03:49:11.536948+0000 mgr.x (mgr.24542) 5227 : cluster [DBG] pgmap v5667: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:13.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:13 smithi045 conmon[39718]: cluster 2021-07-31T03:49:11.536948+0000 mgr.x (mgr.24542) 5227 : cluster [DBG] pgmap v5667: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:13.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:13 smithi045 conmon[33655]: cluster 2021-07-31T03:49:11.536948+0000 mgr.x (mgr.24542 2021-07-31T03:49:13.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:13 smithi045 conmon[33655]: ) 5227 : cluster [DBG] pgmap v5667: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:15.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:15 smithi154 conmon[34553]: cluster 2021-07-31T03:49:13.537469+0000 mgr.x (mgr.24542) 5228 : cluster [DBG] pgmap v5668: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:15.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:15 smithi045 conmon[39718]: cluster 2021-07-31T03:49:13.537469+0000 mgr.x (mgr. 2021-07-31T03:49:15.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:15 smithi045 conmon[39718]: 24542) 5228 : cluster [DBG] pgmap v5668: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:15.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:15 smithi045 conmon[33655]: cluster 2021-07-31T03:49:13.537469+0000 mgr.x (mgr.24542) 5228 : cluster [DBG] pgmap v5668: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:17.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:17 smithi154 conmon[34553]: cluster 2021-07-31T03:49:15.537986+0000 mgr.x (mgr.24542) 5229 : cluster [DBG] pgmap v5669: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:17.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:17 smithi045 conmon[39718]: cluster 2021-07-31T03:49:15.537986+0000 mgr.x (mgr.24542) 5229 : cluster [DBG] 2021-07-31T03:49:17.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:17 smithi045 conmon[39718]: pgmap v5669: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:17.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:17 smithi045 conmon[33655]: cluster 2021-07-31T03:49:15.537986+0000 mgr.x (mgr.24542) 5229 : cluster [DBG] pgmap v5669: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:19.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:19 smithi154 conmon[34553]: cluster 2021-07-31T03:49:17.538414+0000 mgr.x (mgr.24542) 5230 : cluster [DBG] pgmap v5670: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:19.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:19 smithi045 conmon[39718]: cluster 2021-07-31T03:49:17.538414+0000 mgr.x (mgr.24542) 2021-07-31T03:49:19.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:19 smithi045 conmon[39718]: 5230 : cluster [DBG] pgmap v5670: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:19.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:19 smithi045 conmon[33655]: cluster 2021-07-31T03:49:17. 2021-07-31T03:49:19.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:19 smithi045 conmon[33655]: 538414+0000 mgr.x (mgr.24542) 5230 : cluster [DBG] pgmap v5670: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:21.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:21 smithi154 conmon[34553]: cluster 2021-07-31T03:49:19.538941+0000 mgr.x (mgr.24542) 5231 : cluster [DBG] pgmap v5671: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:21.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:21 smithi045 conmon[33655]: cluster 2021-07-31T03:49:19.538941+0000 mgr.x (mgr.24542) 5231 : cluster [DBG] pgmap v5671: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:21.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:21 smithi045 conmon[39718]: cluster 2021-07-31T03:49:19.538941+0000 mgr.x (mgr 2021-07-31T03:49:21.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:21 smithi045 conmon[39718]: .24542) 5231 : cluster [DBG] pgmap v5671: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:23.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:23 smithi154 conmon[34553]: cluster 2021-07-31T03:49:21.539329+0000 mgr.x (mgr.24542) 5232 : cluster [DBG] pgmap v5672: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:23.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:23 smithi045 conmon[33655]: cluster 2021-07-31T03:49:21.539329+0000 mgr.x (mgr.24542) 2021-07-31T03:49:23.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:23 smithi045 conmon[33655]: 5232 : cluster [DBG] pgmap v5672: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:23.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:23 smithi045 conmon[39718]: cluster 2021-07-31T03:49:21.539329+0000 mgr.x 2021-07-31T03:49:23.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:23 smithi045 conmon[39718]: (mgr.24542) 5232 : cluster [DBG] pgmap v5672: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:25.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:25 smithi154 conmon[34553]: cluster 2021-07-31T03:49:23.539778+0000 mgr.x (mgr.24542) 5233 : cluster [DBG] pgmap v5673: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:25.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:25 smithi045 conmon[33655]: cluster 2021-07-31T03:49:23.539778+0000 mgr.x (mgr 2021-07-31T03:49:25.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:25 smithi045 conmon[33655]: .24542) 5233 : cluster [DBG] pgmap v5673: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:25.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:25 smithi045 conmon[39718]: cluster 2021-07-31T03:49:23.539778+0000 mgr.x (mgr. 2021-07-31T03:49:25.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:25 smithi045 conmon[39718]: 24542) 5233 : cluster [DBG] pgmap v5673: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:27.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:27 smithi154 conmon[34553]: cluster 2021-07-31T03:49:25.540270+0000 mgr.x (mgr.24542) 5234 : cluster [DBG] 2021-07-31T03:49:27.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:27 smithi154 conmon[34553]: pgmap v5674: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:27.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:27 smithi045 conmon[39718]: cluster 2021-07-31T03:49:25.540270+0000 mgr.x 2021-07-31T03:49:27.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:27 smithi045 conmon[39718]: (mgr.24542) 5234 : cluster [DBG] pgmap v5674: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:27.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:27 smithi045 conmon[33655]: cluster 2021-07-31T03:49:25.540270+0000 mgr.x (mgr.24542) 5234 : 2021-07-31T03:49:27.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:27 smithi045 conmon[33655]: cluster [DBG] pgmap v5674: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:29.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:29 smithi154 conmon[34553]: cluster 2021-07-31T03:49:27.540705+0000 mgr.x (mgr.24542) 5235 : cluster [DBG] pgmap v5675: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:29.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:29 smithi154 conmon[34553]: audit 2021-07-31T03:49:28. 2021-07-31T03:49:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:29 smithi154 conmon[34553]: 420184+0000 mon.b (mon.2) 685 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:49:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:29 smithi154 conmon[34553]: audit 2021-07-31T03:49:28.420430+0000 mon.b (mon.2) 686 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:49:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:29 smithi154 conmon[34553]: audit 2021-07-31T03:49:28.420887+0000 mon.a (mon.0 2021-07-31T03:49:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:29 smithi154 conmon[34553]: ) 3373 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:49:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:29 smithi154 conmon[34553]: audit 2021-07-31T03:49: 2021-07-31T03:49:29.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:29 smithi154 conmon[34553]: 28.421320+0000 mon.a (mon.0) 3374 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:49:29.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:29 smithi045 conmon[39718]: cluster 2021-07-31T03:49:27 2021-07-31T03:49:29.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:29 smithi045 conmon[39718]: .540705+0000 mgr.x (mgr.24542) 5235 : cluster [DBG] pgmap v5675: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:29.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:29 smithi045 conmon[39718]: audit 2021-07-31T03:49 2021-07-31T03:49:29.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:29 smithi045 conmon[39718]: :28.420184+0000 mon.b (mon.2) 685 : audit [INF] 2021-07-31T03:49:29.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:29 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:49:29.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:29 smithi045 conmon[39718]: audit 2021- 2021-07-31T03:49:29.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:29 smithi045 conmon[39718]: 07-31T03:49:28.420430+0000 mon.b ( 2021-07-31T03:49:29.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:29 smithi045 conmon[39718]: mon.2) 686 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:49:29.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:29 smithi045 conmon[39718]: 2021-07-31T03:49:29.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:29 smithi045 conmon[39718]: audit 2021-07 2021-07-31T03:49:29.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:29 smithi045 conmon[39718]: -31T03:49:28.420887+0000 mon.a (mon. 2021-07-31T03:49:29.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:29 smithi045 conmon[39718]: 0) 3373 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:49:29.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:29 smithi045 conmon[39718]: audit 2021-07-31T03:49:28 2021-07-31T03:49:29.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:29 smithi045 conmon[39718]: .421320+0000 mon.a (mon.0) 3374 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:49:29.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:29 smithi045 conmon[39718]: 2021-07-31T03:49:29.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:29 smithi045 conmon[33655]: cluster 2021-07-31T03:49:27.540705+0000 mgr.x (mgr.24542) 5235 : cluster [DBG] pgmap v5675: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:29.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:29 smithi045 conmon[33655]: audit 2021-07-31T03:49:28.420184+0000 mon.b (mon.2) 685 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:49:29.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:29 smithi045 conmon[33655]: audit 2021-07-31T03:49:28.420430+0000 mon.b (mon.2) 686 : 2021-07-31T03:49:29.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:29 smithi045 conmon[33655]: audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:49:29.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:29 smithi045 conmon[33655]: audit 2021-07-31T03:49:29.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:29 smithi045 conmon[33655]: 2021-07-31T03:49:28.420887+0000 2021-07-31T03:49:29.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:29 smithi045 conmon[33655]: mon.a (mon.0) 3373 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:49:29.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:29 smithi045 conmon[33655]: audit 2021 2021-07-31T03:49:29.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:29 smithi045 conmon[33655]: -07-31T03:49:28.421320 2021-07-31T03:49:29.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:29 smithi045 conmon[33655]: +0000 mon.a (mon.0) 2021-07-31T03:49:29.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:29 smithi045 conmon[33655]: 3374 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:49:31.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:31 smithi154 conmon[34553]: cluster 2021-07-31T03:49:29.541129+0000 mgr.x (mgr.24542) 5236 : cluster [DBG] pgmap v5676: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:31.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:31 smithi045 conmon[33655]: cluster 2021-07-31T03:49:29.541129+0000 mgr.x (mgr.24542 2021-07-31T03:49:31.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:31 smithi045 conmon[33655]: ) 5236 : cluster [DBG] pgmap v5676: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:31.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:31 smithi045 conmon[39718]: cluster 2021-07-31T03:49:29.541129+0000 mgr.x (mgr.24542 2021-07-31T03:49:31.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:31 smithi045 conmon[39718]: ) 5236 : cluster [DBG] pgmap v5676: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:33.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:33 smithi154 conmon[34553]: cluster 2021-07-31T03:49:31.541595+0000 mgr.x (mgr.24542) 5237 : cluster [DBG] pgmap v5677: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:33.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:33 smithi045 conmon[39718]: cluster 2021-07-31T03:49:31.541595+0000 mgr.x (mgr.24542 2021-07-31T03:49:33.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:33 smithi045 conmon[39718]: ) 5237 : cluster [DBG] pgmap v5677: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:33.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:33 smithi045 conmon[33655]: cluster 2021-07-31T03:49:31.541595+0000 2021-07-31T03:49:33.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:33 smithi045 conmon[33655]: mgr.x (mgr.24542) 5237 : cluster [DBG] pgmap v5677: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:35.382 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[34553]: cluster 2021-07-31T03:49:33.542074+0000 mgr.x (mgr.24542) 5238 : cluster [DBG] pgmap v5678: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:35.382 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: debug 2021-07-31T03:49:35.354+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:49:35.383 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:49:35.383 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:49:35.383 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: yield 2021-07-31T03:49:35.383 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:49:35.383 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:49:35.384 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:49:35.384 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:49:35.384 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:49:35.384 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:49:35.385 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:49:35.385 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:49:35.385 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:49:35.385 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: task.result() 2021-07-31T03:49:35.385 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:49:35.386 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: debug 2021-07-31T03:49:35.366+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:49:35.386 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:49:35.386 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:49:35.386 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: yield 2021-07-31T03:49:35.387 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:49:35.387 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:49:35.387 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:49:35.387 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:49:35.387 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:49:35.388 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:49:35.388 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:49:35.388 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:49:35.388 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:49:35.389 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: task.result() 2021-07-31T03:49:35.389 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:49:35.391 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:35 smithi045 conmon[39718]: cluster 2021-07-31T03:49:33. 2021-07-31T03:49:35.392 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:35 smithi045 conmon[39718]: 542074+0000 mgr.x (mgr.24542) 5238 : cluster [DBG] pgmap v5678: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:35.392 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:35 smithi045 conmon[33655]: cluster 2021-07-31T03:49:33.542074 2021-07-31T03:49:35.392 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:35 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 5238 : cluster [DBG] pgmap v5678: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:35.752 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: debug 2021-07-31T03:49:35.391+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:49:35.753 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:49:35.753 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:49:35.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: yield 2021-07-31T03:49:35.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:49:35.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:49:35.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:49:35.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:49:35.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:49:35.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:49:35.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:49:35.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:49:35.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:49:35.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: task.result() 2021-07-31T03:49:35.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:49:35.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: debug 2021-07-31T03:49:35.392+0000 7fcb9215e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:49:35.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:49:35.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:49:35.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: yield 2021-07-31T03:49:35.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:49:35.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:49:35.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:49:35.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:49:35.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:49:35.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:49:35.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:49:35.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:49:35.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:49:35.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: task.result() 2021-07-31T03:49:35.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:49:35.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: 2021-07-31T03:49:35.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:49:35.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: 2021-07-31T03:49:35.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:49:35.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:49:35.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:49:35.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:49:35.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:49:35.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:49:35.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:49:35.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:49:35.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:49:35.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:49:35.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:49:35.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:49:35.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:49:35.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:49:35.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:49:35.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:49:35.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:49:35.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:49:35.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: raise 2021-07-31T03:49:35.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:49:35.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:49:35.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:49:35.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:49:35.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:49:35.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: debug 2021-07-31T03:49:35.400+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:49:35.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:49:35.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:49:35.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: yield 2021-07-31T03:49:35.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:49:35.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:49:35.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:49:35.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:49:35.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:49:35.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:49:35.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:49:35.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:49:35.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:49:35.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: task.result() 2021-07-31T03:49:35.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:49:35.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: debug 2021-07-31T03:49:35.400+0000 7fcb9195d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:49:35.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:49:35.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:49:35.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: yield 2021-07-31T03:49:35.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:49:35.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:49:35.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:49:35.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:49:35.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:49:35.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:49:35.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:49:35.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:49:35.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:49:35.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: task.result() 2021-07-31T03:49:35.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:49:35.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: 2021-07-31T03:49:35.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:49:35.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: 2021-07-31T03:49:35.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:49:35.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:49:35.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:49:35.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:49:35.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:49:35.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:49:35.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:49:35.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:49:35.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:49:35.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:49:35.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:49:35.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:49:35.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:49:35.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:49:35.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:49:35.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:49:35.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:49:35.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:49:35.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: raise 2021-07-31T03:49:35.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:49:35.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:49:35.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:49:35.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:49:35.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:49:35 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:49:36.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:36 smithi154 conmon[34553]: audit 2021-07-31T03:49:35.309368+0000 mon.b (mon.2) 687 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:49:36.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:36 smithi154 conmon[34553]: audit 2021-07-31T03:49:35.310816+0000 mon.b (mon.2) 688 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:49:36.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:36 smithi154 conmon[34553]: audit 2021-07-31T03:49:35.311725+0000 mon.b (mon.2) 689 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:49:36.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:36 smithi045 conmon[33655]: audit 2021-07-31T03:49:35.309368+0000 mon.b (mon.2) 687 2021-07-31T03:49:36.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:36 smithi045 conmon[33655]: : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:49:36.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:36 smithi045 conmon[33655]: audit 2021-07-31T03:49:35.310816+0000 mon.b (mon.2) 688 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:49:36.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:36 smithi045 conmon[33655]: audit 2021-07-31T03:49:36.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:36 smithi045 conmon[33655]: 2021-07-31T03:49:35.311725+0000 mon.b (mon.2) 689 : audit 2021-07-31T03:49:36.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:36 smithi045 conmon[33655]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:49:36.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:36 smithi045 conmon[39718]: audit 2021-07-31T03:49:35.309368+0000 mon.b 2021-07-31T03:49:36.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:36 smithi045 conmon[39718]: (mon.2) 687 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:49:36.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:36 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T03:49:36.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:36 smithi045 conmon[39718]: 31T03:49:35.310816+0000 mon.b (mon. 2021-07-31T03:49:36.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:36 smithi045 conmon[39718]: 2) 688 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:49:36.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:36 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T03:49:36.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:36 smithi045 conmon[39718]: T03:49:35.311725+0000 mon.b (mon.2) 2021-07-31T03:49:36.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:36 smithi045 conmon[39718]: 689 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:49:37.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: cephadm 2021-07-31T03:49:35.312531+0000 mgr.x (mgr.24542) 5239 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:49:37.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: 2021-07-31T03:49:37.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: cephadm 2021-07-31T03:49:35.355368+0000 mgr.x (mgr.24542) 5240 : cephadm [ERR] 2021-07-31T03:49:37.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:49:37.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:49:37.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: yield 2021-07-31T03:49:37.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:49:37.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:49:37.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:49:37.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:49:37.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:49:37.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:49:37.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:49:37.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:49:37.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:49:37.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: task.result() 2021-07-31T03:49:37.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:49:37.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: cephadm 2021-07-31T03:49:35.355973+0000 mgr.x (mgr.24542) 5241 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:49:37.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: cephadm 2021-07-31T03:49:35.367312+0000 mgr.x (mgr.24542) 5242 : cephadm [ERR] 2021-07-31T03:49:37.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:49:37.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:49:37.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: yield 2021-07-31T03:49:37.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:49:37.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:49:37.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:49:37.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:49:37.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:49:37.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:49:37.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:49:37.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:49:37.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:49:37.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: task.result() 2021-07-31T03:49:37.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:49:37.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: cephadm 2021-07-31T03:49:35.367772+0000 mgr.x (mgr.24542) 5243 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:49:37.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: cephadm 2021-07-31T03:49:35.392837+0000 mgr.x (mgr.24542) 5244 : cephadm [ERR] 2021-07-31T03:49:37.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:49:37.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:49:37.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: yield 2021-07-31T03:49:37.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:49:37.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:49:37.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:49:37.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:49:37.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:49:37.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:49:37.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:49:37.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:49:37.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:49:37.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: task.result() 2021-07-31T03:49:37.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:49:37.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: cephadm 2021-07-31T03:49:37.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: 2021-07-31T03:49:35.393443+0000 mgr.x (mgr.24542) 5245 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:49:37.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:49:37.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:49:37.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: yield 2021-07-31T03:49:37.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:49:37.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:49:37.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:49:37.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:49:37.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:49:37.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:49:37.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:49:37.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:49:37.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:49:37.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: task.result() 2021-07-31T03:49:37.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:49:37.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: 2021-07-31T03:49:37.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:49:37.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: 2021-07-31T03:49:37.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:49:37.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:49:37.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:49:37.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:49:37.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:49:37.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:49:37.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:49:37.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:49:37.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:49:37.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:49:37.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:49:37.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:49:37.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:49:37.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:49:37.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:49:37.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:49:37.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:49:37.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:49:37.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: raise 2021-07-31T03:49:37.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:49:37.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:49:37.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:49:37.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:49:37.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:49:37.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: cephadm 2021-07-31T03:49:35.401480+0000 mgr.x (mgr.24542) 5246 : cephadm [ERR] 2021-07-31T03:49:37.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:49:37.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:49:37.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: yield 2021-07-31T03:49:37.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:49:37.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:49:37.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:49:37.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:49:37.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:49:37.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:49:37.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:49:37.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:49:37.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:49:37.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: task.result() 2021-07-31T03:49:37.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:49:37.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: cephadm 2021-07-31T03:49:35.401872+0000 mgr.x (mgr.24542) 5247 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:49:37.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:49:37.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:49:37.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: yield 2021-07-31T03:49:37.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:49:37.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:49:37.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:49:37.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:49:37.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:49:37.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:49:37.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:49:37.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:49:37.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:49:37.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: task.result() 2021-07-31T03:49:37.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:49:37.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: 2021-07-31T03:49:37.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:49:37.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: 2021-07-31T03:49:37.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:49:37.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:49:37.541 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:49:37.541 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:49:37.541 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:49:37.542 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:49:37.542 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:49:37.543 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:49:37.543 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:49:37.543 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:49:37.544 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:49:37.544 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:49:37.544 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:49:37.544 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:49:37.545 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:49:37.545 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:49:37.545 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:49:37.545 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:49:37.546 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: raise 2021-07-31T03:49:37.546 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:49:37.546 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:49:37.546 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:49:37.547 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:49:37.547 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:49:37.547 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:37 smithi154 conmon[34553]: cluster 2021-07-31T03:49:35.542468+0000 mgr.x (mgr.24542) 5248 : cluster [DBG] pgmap v5679: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:37.548 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: cephadm 2021-07-31T03:49:35.312531+0000 2021-07-31T03:49:37.548 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: mgr.x (mgr.24542) 5239 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:49:37.549 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: cephadm 2021-07-31T03:49:35.355368 2021-07-31T03:49:37.549 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 5240 : cephadm [ERR] 2021-07-31T03:49:37.549 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:49:37.550 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:49:37.550 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: yield 2021-07-31T03:49:37.550 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:49:37.550 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:49:37.551 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:49:37.551 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:49:37.551 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:49:37.552 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:49:37.552 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:49:37.552 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:49:37.552 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:49:37.553 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: task.result() 2021-07-31T03:49:37.553 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:49:37.553 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: cephadm 2021-07-31T03:49:35.355973+0000 mgr.x (mgr.24542) 5241 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:49:37.554 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: cephadm 2021- 2021-07-31T03:49:37.554 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: 07-31T03:49:35.367312+0000 mgr.x (mgr.24542) 2021-07-31T03:49:37.554 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: 5242 : cephadm [ERR] 2021-07-31T03:49:37.554 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:49:37.555 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:49:37.555 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: yield 2021-07-31T03:49:37.555 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:49:37.555 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:49:37.556 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:49:37.556 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:49:37.556 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:49:37.556 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:49:37.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:49:37.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:49:37.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:49:37.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: task.result() 2021-07-31T03:49:37.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:49:37.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: cephadm 2021-07-31T03:49:35.367772+0000 mgr.x (mgr.24542) 5243 : cephadm [INF] 2021-07-31T03:49:37.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:49:37.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: cephadm 2021-07-31T 2021-07-31T03:49:37.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: 03:49:35.392837+0000 mgr.x ( 2021-07-31T03:49:37.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: mgr.24542) 5244 : cephadm 2021-07-31T03:49:37.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: cephadm 2021-07-31T03:49:35.312531+0000 2021-07-31T03:49:37.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: mgr.x (mgr.24542) 5239 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:49:37.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: cephadm 2021-07-31T03:49:37.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: 2021- 2021-07-31T03:49:37.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: 07-31T03:49:35 2021-07-31T03:49:37.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: .355368+0000 mgr.x (mgr.24542) 5240 2021-07-31T03:49:37.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: : cephadm [ERR] 2021-07-31T03:49:37.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:49:37.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:49:37.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: yield 2021-07-31T03:49:37.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:49:37.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:49:37.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:49:37.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:49:37.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:49:37.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:49:37.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:49:37.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:49:37.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:49:37.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: task.result() 2021-07-31T03:49:37.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:49:37.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: cephadm 2021-07-31T03:49:35.355973+0000 mgr.x (mgr.24542) 5241 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:49:37.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: cephadm 2021-07- 2021-07-31T03:49:37.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: 31T03:49:35.367312+0000 2021-07-31T03:49:37.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: mgr.x (mgr.24542) 5242 : cephadm [ERR] 2021-07-31T03:49:37.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: 2021-07-31T03:49:37.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:49:37.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:49:37.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: yield 2021-07-31T03:49:37.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:49:37.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:49:37.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:49:37.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:49:37.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:49:37.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:49:37.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:49:37.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:49:37.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:49:37.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: task.result() 2021-07-31T03:49:37.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:49:37.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: cephadm 2021-07-31T03:49:35.367772+0000 mgr.x (mgr.24542) 5243 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:49:37.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: cephadm 2021-07-31T03:49:37.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: [ERR] 2021-07-31T03:49:37.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:49:37.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:49:37.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: yield 2021-07-31T03:49:37.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:49:37.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:49:37.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:49:37.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:49:37.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:49:37.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:49:37.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:49:37.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:49:37.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:49:37.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: task.result() 2021-07-31T03:49:37.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:49:37.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: cephadm 2021-07-31T03:49:35.393443+0000 mgr.x (mgr.24542) 2021-07-31T03:49:37.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: 5245 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:49:37.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:49:37.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:49:37.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: yield 2021-07-31T03:49:37.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:49:37.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:49:37.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:49:37.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:49:37.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:49:37.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:49:37.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:49:37.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:49:37.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:49:37.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: task.result() 2021-07-31T03:49:37.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:49:37.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: 2021-07-31T03:49:37.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:49:37.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: 2021-07-31T03:49:37.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:49:37.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:49:37.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:49:37.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:49:37.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:49:37.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:49:37.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:49:37.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:49:37.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: 2021-07-31T 2021-07-31T03:49:37.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: 03:49:35.392837+0000 2021-07-31T03:49:37.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: mgr.x (mgr.24542) 5244 : 2021-07-31T03:49:37.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: cephadm [ERR] 2021-07-31T03:49:37.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:49:37.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:49:37.585 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: yield 2021-07-31T03:49:37.585 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:49:37.585 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:49:37.586 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:49:37.586 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:49:37.586 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:49:37.586 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:49:37.587 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:49:37.587 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:49:37.587 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:49:37.587 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: task.result() 2021-07-31T03:49:37.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:49:37.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: cephadm 2021-07-31T03:49:35.393443+0000 mgr.x (mgr.24542) 5245 : 2021-07-31T03:49:37.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:49:37.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:49:37.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:49:37.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: yield 2021-07-31T03:49:37.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:49:37.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:49:37.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:49:37.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:49:37.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:49:37.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:49:37.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:49:37.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:49:37.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:49:37.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: task.result() 2021-07-31T03:49:37.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:49:37.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: 2021-07-31T03:49:37.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:49:37.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: 2021-07-31T03:49:37.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:49:37.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:49:37.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:49:37.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:49:37.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:49:37.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:49:37.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:49:37.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:49:37.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:49:37.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:49:37.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:49:37.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:49:37.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:49:37.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:49:37.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:49:37.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:49:37.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: raise 2021-07-31T03:49:37.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:49:37.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:49:37.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:49:37.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:49:37.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:49:37.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: cephadm 2021-07-31T03:49:35.401480+0000 mgr.x (mgr.24542) 5246 : cephadm [ERR] 2021-07-31T03:49:37.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:49:37.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:49:37.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: yield 2021-07-31T03:49:37.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:49:37.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:49:37.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:49:37.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:49:37.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:49:37.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:49:37.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:49:37.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:49:37.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:49:37.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: task.result() 2021-07-31T03:49:37.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:49:37.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: cephadm 2021-07-31T03:49:35.401872+0000 mgr.x (mgr.24542) 5247 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:49:37.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:49:37.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:49:37.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: yield 2021-07-31T03:49:37.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:49:37.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:49:37.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:49:37.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:49:37.605 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:49:37.605 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:49:37.605 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:49:37.606 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:49:37.606 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:49:37.606 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:49:37.606 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:49:37.606 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:49:37.607 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:49:37.607 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:49:37.607 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:49:37.607 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:49:37.608 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: raise 2021-07-31T03:49:37.608 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:49:37.608 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:49:37.608 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:49:37.609 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:49:37.609 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:49:37.609 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: cephadm 2021-07-31T03:49:35.401480+0000 mgr.x (mgr.24542) 5246 : cephadm [ERR] 2021-07-31T03:49:37.609 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:49:37.610 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:49:37.610 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: yield 2021-07-31T03:49:37.610 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:49:37.610 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:49:37.610 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:49:37.611 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:49:37.611 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:49:37.611 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:49:37.611 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:49:37.612 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:49:37.612 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:49:37.612 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: task.result() 2021-07-31T03:49:37.612 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:49:37.612 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: cephadm 2021-07-31T03:49:35.401872+0000 mgr.x (mgr.24542) 5247 : cephadm 2021-07-31T03:49:37.613 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:49:37.613 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:49:37.613 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:49:37.613 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: yield 2021-07-31T03:49:37.614 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:49:37.614 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:49:37.614 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:49:37.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:49:37.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:49:37.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:49:37.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: task.result() 2021-07-31T03:49:37.616 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:49:37.616 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: 2021-07-31T03:49:37.616 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:49:37.616 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: 2021-07-31T03:49:37.616 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:49:37.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:49:37.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:49:37.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:49:37.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:49:37.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:49:37.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:49:37.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:49:37.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:49:37.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:49:37.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:49:37.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:49:37.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:49:37.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:49:37.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:49:37.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:49:37.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:49:37.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:49:37.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: raise 2021-07-31T03:49:37.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:49:37.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:49:37.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:49:37.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:49:37.622 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:49:37.622 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: 2021-07-31T03:49:37.622 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[33655]: cluster 2021-07-31T03:49:35.542468+0000 mgr.x (mgr.24542) 5248 : cluster [DBG] pgmap v5679: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:37.622 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:49:37.623 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:49:37.623 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:49:37.623 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:49:37.623 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:49:37.624 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:49:37.624 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:49:37.624 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:49:37.624 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: task.result() 2021-07-31T03:49:37.625 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:49:37.625 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: 2021-07-31T03:49:37.625 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:49:37.625 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: 2021-07-31T03:49:37.626 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:49:37.626 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:49:37.626 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:49:37.626 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:49:37.626 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:49:37.627 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:49:37.627 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:49:37.627 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:49:37.627 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:49:37.627 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:49:37.628 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:49:37.628 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:49:37.628 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:49:37.628 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:49:37.629 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:49:37.629 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:49:37.629 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:49:37.629 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:49:37.630 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: raise 2021-07-31T03:49:37.630 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:49:37.630 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:49:37.630 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:49:37.630 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:49:37.631 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:49:37.631 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: cluster 2021-07-31T03:49:35.542468+0000 mgr.x (mgr.24542) 5248 : 2021-07-31T03:49:37.631 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:37 smithi045 conmon[39718]: cluster [DBG] pgmap v5679: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:37.631 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:49:37.808 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:49:37.808 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 2h ago 2h - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:49:37.809 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 2h ago 2h 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:49:37.809 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 2h ago 2h 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:49:37.809 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 2h ago 2h - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:49:37.809 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:49:37.810 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 2h ago 2h 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:49:37.810 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:49:37.810 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 2h ago 2h - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:49:37.810 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 2h ago 2h 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:49:37.811 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:49:37.811 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:49:37.811 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:49:37.811 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:49:37.812 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 2h ago 2h 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:49:37.812 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 2h ago 2h 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:49:37.812 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 2h ago 2h 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:49:37.812 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 2h ago 2h 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:49:37.813 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 2h ago 2h 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:49:38.143 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:49:38.143 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:49:38.144 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:49:38.144 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:49:38.144 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:49:38.144 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:49:38.144 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:49:38.145 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:49:38.145 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:49:38.145 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:49:38.145 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:49:38.145 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:49:38.146 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:49:38.146 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:49:38.146 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:49:38.146 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:49:38.146 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:49:39.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:39 smithi154 conmon[34553]: audit 2021-07-31T03:49:37.520786+0000 mgr.x (mgr.24542) 5249 : audit [DBG] from='client.30253 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:49:39.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:39 smithi154 conmon[34553]: 2021-07-31T03:49:39.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:39 smithi154 conmon[34553]: cluster 2021-07-31T03:49:37.542831+0000 mgr.x (mgr.24542) 5250 : cluster [DBG] pgmap v5680: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:39.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:39 smithi154 conmon[34553]: audit 2021-07-31T03:49:37.800861+0000 mgr.x (mgr.24542) 5251 : audit [DBG] from='client.20322 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:49:39.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:39 smithi154 conmon[34553]: audit 2021-07-31T03:49:38.143905+0000 mon.c (mon.1) 229 : audit [DBG] from='client.? 172.21.15.45:0/3123203790' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:49:39.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:39 smithi045 conmon[33655]: audit 2021-07-31T03:49:37.520786+0000 mgr.x (mgr.24542) 5249 : audit 2021-07-31T03:49:39.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:39 smithi045 conmon[33655]: [DBG] from='client.30253 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:49:39.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:39 smithi045 conmon[33655]: cluster 2021-07-31T03:49:37.542831+0000 mgr.x (mgr.24542) 5250 : cluster [DBG] pgmap v5680: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:39.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:39 smithi045 conmon[33655]: audit 2021-07-31T03:49:37.800861+0000 mgr.x (mgr.24542) 5251 : audit [DBG] from='client.20322 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:49:39.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:39 smithi045 conmon[33655]: audit 2021-07-31T03:49:38.143905+0000 mon.c (mon.1) 229 : 2021-07-31T03:49:39.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:39 smithi045 conmon[33655]: audit [DBG] from='client.? 172.21.15.45:0/3123203790' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:49:39.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:39 smithi045 conmon[39718]: audit 2021-07-31T03:49:37.520786+0000 mgr.x 2021-07-31T03:49:39.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:39 smithi045 conmon[39718]: (mgr.24542) 5249 : audit [DBG] from='client.30253 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:49:39.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:39 smithi045 conmon[39718]: cluster 2021-07-31T03:49 2021-07-31T03:49:39.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:39 smithi045 conmon[39718]: :37.542831+0000 mgr.x (mgr.24542) 5250 : cluster [DBG] pgmap v5680: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:39.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:39 smithi045 conmon[39718]: audit 2021 2021-07-31T03:49:39.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:39 smithi045 conmon[39718]: -07-31T03:49:37.800861 2021-07-31T03:49:39.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:39 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 5251 : audit 2021-07-31T03:49:39.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:39 smithi045 conmon[39718]: [DBG] from='client.20322 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:49:39.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:39 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T03:49:39.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:39 smithi045 conmon[39718]: T03:49:38.143905 2021-07-31T03:49:39.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:39 smithi045 conmon[39718]: +0000 mon.c (mon.1) 2021-07-31T03:49:39.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:39 smithi045 conmon[39718]: 229 : audit [DBG] from='client.? 172.21.15.45:0/3123203790' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:49:41.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:41 smithi154 conmon[34553]: cluster 2021-07-31T03:49:39.543288+0000 mgr.x (mgr.24542) 5252 : cluster [DBG] pgmap v5681: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:41.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:41 smithi045 conmon[39718]: cluster 2021-07-31T03:49:39.543288+0000 mgr.x ( 2021-07-31T03:49:41.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:41 smithi045 conmon[39718]: mgr.24542) 5252 : cluster [DBG] pgmap v5681: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:41.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:41 smithi045 conmon[33655]: cluster 2021-07-31T03:49:39.543288+0000 mgr.x (mgr.24542) 5252 : cluster 2021-07-31T03:49:41.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:41 smithi045 conmon[33655]: [DBG] pgmap v5681: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:43.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:43 smithi154 conmon[34553]: cluster 2021-07-31T03:49:41.543793+0000 mgr.x (mgr.24542) 5253 : cluster 2021-07-31T03:49:43.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:43 smithi154 conmon[34553]: [DBG] pgmap v5682: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:43.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:43 smithi045 conmon[33655]: cluster 2021-07-31T03:49:41.543793+0000 mgr.x (mgr.24542) 5253 : cluster [DBG] pgmap v5682: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:43.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:43 smithi045 conmon[33655]: 2021-07-31T03:49:43.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:43 smithi045 conmon[39718]: cluster 2021-07-31T03:49: 2021-07-31T03:49:43.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:43 smithi045 conmon[39718]: 41.543793+0000 mgr.x (mgr.24542) 5253 : cluster [DBG] pgmap v5682: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:45.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:45 smithi154 conmon[34553]: cluster 2021-07-31T03:49:43.544315+0000 mgr.x (mgr.24542) 5254 : cluster [DBG] pgmap v5683: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:45.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:45 smithi045 conmon[39718]: cluster 2021-07-31T03:49: 2021-07-31T03:49:45.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:45 smithi045 conmon[39718]: 43.544315+0000 mgr.x (mgr.24542) 5254 : cluster [DBG] pgmap v5683: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:45.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:45 smithi045 conmon[33655]: cluster 2021-07-31T03:49:43.544315+0000 mgr.x 2021-07-31T03:49:45.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:45 smithi045 conmon[33655]: (mgr.24542) 5254 : cluster [DBG] pgmap v5683: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:47.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:47 smithi154 conmon[34553]: cluster 2021-07-31T03:49:45.544788+0000 mgr.x (mgr.24542) 5255 : cluster [DBG] pgmap v5684: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:47.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:47 smithi045 conmon[33655]: cluster 2021-07-31T03:49:45.544788+0000 mgr.x 2021-07-31T03:49:47.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:47 smithi045 conmon[33655]: (mgr.24542) 5255 : cluster [DBG] pgmap v5684: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:47.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:47 smithi045 conmon[39718]: cluster 2021-07-31T03:49:45.544788+0000 mgr.x 2021-07-31T03:49:47.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:47 smithi045 conmon[39718]: (mgr.24542) 5255 : cluster [DBG] pgmap v5684: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:49.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:49 smithi154 conmon[34553]: cluster 2021-07-31T03:49:47.545414+0000 mgr.x (mgr.24542) 5256 : cluster [DBG] pgmap v5685: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:49.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:49 smithi045 conmon[33655]: cluster 2021-07-31T03:49:47.545414 2021-07-31T03:49:49.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:49 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 5256 : cluster [DBG] pgmap v5685: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:49.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:49 smithi045 conmon[39718]: cluster 2021-07-31T03:49:47.545414+0000 mgr.x (mgr.24542) 5256 : cluster [DBG] 2021-07-31T03:49:49.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:49 smithi045 conmon[39718]: pgmap v5685: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:51.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:51 smithi154 conmon[34553]: cluster 2021-07-31T03:49:49.546037+0000 mgr.x (mgr.24542) 5257 : cluster [DBG] pgmap v5686: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:51.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:51 smithi045 conmon[39718]: cluster 2021-07-31T03:49:49. 2021-07-31T03:49:51.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:51 smithi045 conmon[39718]: 546037+0000 mgr.x (mgr.24542) 5257 : cluster [DBG] pgmap v5686: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:51.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:51 smithi045 conmon[33655]: cluster 2021-07-31T03:49:49.546037+0000 mgr.x (mgr.24542) 5257 : cluster [DBG] 2021-07-31T03:49:51.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:51 smithi045 conmon[33655]: pgmap v5686: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:53.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:53 smithi154 conmon[34553]: cluster 2021-07-31T03:49:51.546527+0000 mgr.x (mgr.24542) 5258 : cluster [DBG] pgmap v5687: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:53.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:53 smithi045 conmon[33655]: cluster 2021-07-31T03:49:51.546527+0000 mgr.x (mgr.24542) 5258 : cluster 2021-07-31T03:49:53.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:53 smithi045 conmon[33655]: [DBG] pgmap v5687: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:53.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:53 smithi045 conmon[39718]: cluster 2021-07-31T03:49:51.546527+0000 mgr.x (mgr. 2021-07-31T03:49:53.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:53 smithi045 conmon[39718]: 24542) 5258 : cluster [DBG] pgmap v5687: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:55.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:55 smithi154 conmon[34553]: cluster 2021-07-31T03:49:53.546899+0000 mgr.x (mgr.24542) 5259 : cluster [DBG] pgmap v5688: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:55.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:55 smithi045 conmon[33655]: cluster 2021-07-31T03:49:53.546899+0000 mgr.x (mgr.24542) 5259 : cluster 2021-07-31T03:49:55.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:55 smithi045 conmon[33655]: [DBG] pgmap v5688: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:55.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:55 smithi045 conmon[39718]: cluster 2021-07-31T03:49:53.546899+0000 mgr.x ( 2021-07-31T03:49:55.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:55 smithi045 conmon[39718]: mgr.24542) 5259 : cluster [DBG] pgmap v5688: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:57.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:57 smithi154 conmon[34553]: cluster 2021-07-31T03:49:55.547510+0000 mgr.x (mgr.24542) 5260 : cluster [DBG] pgmap v5689: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:57.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:57 smithi045 conmon[33655]: cluster 2021-07-31T03:49:55.547510+0000 mgr.x (mgr.24542) 5260 : cluster 2021-07-31T03:49:57.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:57 smithi045 conmon[33655]: [DBG] pgmap v5689: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:57.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:57 smithi045 conmon[39718]: cluster 2021-07-31T03:49:55.547510+0000 mgr.x ( 2021-07-31T03:49:57.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:57 smithi045 conmon[39718]: mgr.24542) 5260 : cluster [DBG] pgmap v5689: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:59.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:49:59 smithi154 conmon[34553]: cluster 2021-07-31T03:49:57.548068+0000 mgr.x (mgr.24542) 5261 : cluster [DBG] pgmap v5690: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:59.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:59 smithi045 conmon[33655]: cluster 2021-07-31T03:49:57.548068+0000 mgr.x (mgr.24542) 5261 2021-07-31T03:49:59.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:49:59 smithi045 conmon[33655]: : cluster [DBG] pgmap v5690: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:49:59.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:59 smithi045 conmon[39718]: cluster 2021-07-31T03:49:57.548068+0000 mgr.x 2021-07-31T03:49:59.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:49:59 smithi045 conmon[39718]: (mgr.24542) 5261 : cluster [DBG] pgmap v5690: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:00.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:00 smithi154 conmon[34553]: cluster 2021-07-31T03:50:00.000130+0000 mon.a (mon.0) 3375 : cluster [INF] overall HEALTH_OK 2021-07-31T03:50:00.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:00 smithi045 conmon[33655]: cluster 2021-07-31T03:50:00.000130+0000 mon.a (mon.0) 2021-07-31T03:50:00.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:00 smithi045 conmon[33655]: 3375 : cluster [INF] overall HEALTH_OK 2021-07-31T03:50:00.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:00 smithi045 conmon[39718]: cluster 2021-07-31T03:50:00.000130+0000 mon.a (mon 2021-07-31T03:50:00.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:00 smithi045 conmon[39718]: .0) 3375 : cluster [INF] overall HEALTH_OK 2021-07-31T03:50:01.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:01 smithi154 conmon[34553]: cluster 2021-07-31T03:49:59.548489+0000 mgr.x (mgr.24542) 5262 : cluster [DBG] pgmap v5691: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:01.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:01 smithi045 conmon[39718]: cluster 2021-07-31T03:49:59 2021-07-31T03:50:01.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:01 smithi045 conmon[39718]: .548489+0000 mgr.x (mgr.24542) 5262 : cluster [DBG] pgmap v5691: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:01.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:01 smithi045 conmon[33655]: cluster 2021-07-31T03:49:59.548489+0000 mgr.x (mgr.24542) 5262 : 2021-07-31T03:50:01.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:01 smithi045 conmon[33655]: cluster [DBG] pgmap v5691: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:03.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:03 smithi154 conmon[34553]: cluster 2021-07-31T03:50:01.548958+0000 mgr.x (mgr.24542) 5263 : cluster [DBG] pgmap v5692: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:03.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:03 smithi045 conmon[39718]: cluster 2021-07-31T03:50:01. 2021-07-31T03:50:03.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:03 smithi045 conmon[39718]: 548958+0000 mgr.x (mgr.24542) 5263 : cluster [DBG] pgmap v5692: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:03.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:03 smithi045 conmon[33655]: cluster 2021-07-31T03:50:01.548958+0000 mgr.x (mgr.24542) 5263 : cluster [DBG] pgmap v5692: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:05.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:05 smithi154 conmon[34553]: cluster 2021-07-31T03:50:03.549466+0000 mgr.x (mgr.24542) 5264 : cluster [DBG] pgmap v5693: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:05.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:05 smithi045 conmon[33655]: cluster 2021-07-31T03:50:03.549466+0000 mgr.x (mgr.24542 2021-07-31T03:50:05.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:05 smithi045 conmon[33655]: ) 5264 : cluster [DBG] pgmap v5693: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:05.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:05 smithi045 conmon[39718]: cluster 2021-07-31T03:50:03.549466+0000 2021-07-31T03:50:05.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:05 smithi045 conmon[39718]: mgr.x (mgr.24542) 5264 : cluster [DBG] pgmap v5693: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:07.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:07 smithi154 conmon[34553]: cluster 2021-07-31T03:50:05.549979+0000 mgr.x (mgr.24542) 5265 : cluster [DBG] pgmap v5694: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:07.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:07 smithi045 conmon[39718]: cluster 2021-07-31T03:50:05.549979+0000 mgr.x (mgr 2021-07-31T03:50:07.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:07 smithi045 conmon[39718]: .24542) 5265 : cluster [DBG] pgmap v5694: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:07.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:07 smithi045 conmon[33655]: cluster 2021-07-31T03:50:05.549979+0000 mgr.x 2021-07-31T03:50:07.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:07 smithi045 conmon[33655]: (mgr.24542) 5265 : cluster [DBG] pgmap v5694: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:08.456 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:50:08.734 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:50:08.734 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 2h ago 2h - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:50:08.734 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 2h ago 2h 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:50:08.735 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 2h ago 2h 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:50:08.735 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 2h ago 2h - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:50:08.735 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:50:08.735 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 2h ago 2h 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:50:08.735 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:50:08.736 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 2h ago 2h - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:50:08.736 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 2h ago 2h 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:50:08.736 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:50:08.736 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:50:08.737 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:50:08.737 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:50:08.737 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 2h ago 2h 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:50:08.737 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 2h ago 2h 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:50:08.737 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 2h ago 2h 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:50:08.738 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 2h ago 2h 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:50:08.738 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 2h ago 2h 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:50:09.067 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:50:09.068 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:50:09.068 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:50:09.068 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:50:09.069 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:50:09.069 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:50:09.069 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:50:09.069 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:50:09.069 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:50:09.070 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:50:09.070 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:50:09.070 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:50:09.070 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:50:09.071 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:50:09.071 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:50:09.071 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:50:09.071 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:50:09.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:09 smithi154 conmon[34553]: cluster 2021-07-31T03:50:07.550502+0000 mgr.x (mgr.24542) 5266 : cluster [DBG] pgmap v5695: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:09.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:09 smithi154 conmon[34553]: 2021-07-31T03:50:09.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:09 smithi154 conmon[34553]: audit 2021-07-31T03:50:09.068465+0000 mon.a (mon.0) 3376 : audit [DBG] from='client.? 172.21.15.45:0/3027889574' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:50:09.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:09 smithi045 conmon[33655]: cluster 2021-07-31T03:50 2021-07-31T03:50:09.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:09 smithi045 conmon[33655]: :07.550502+0000 mgr.x (mgr.24542) 5266 : cluster [DBG] pgmap v5695: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:09.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:09 smithi045 conmon[33655]: audit 2021-07-31T03:50:09.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:09 smithi045 conmon[33655]: 2021-07-31T03:50:09.068465+0000 mon.a (mon.0 2021-07-31T03:50:09.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:09 smithi045 conmon[33655]: ) 3376 : audit [DBG] from='client.? 172.21.15.45:0/3027889574' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:50:09.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:09 smithi045 conmon[39718]: cluster 2021-07-31T03:50:07.550502+0000 mgr.x (mgr. 2021-07-31T03:50:09.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:09 smithi045 conmon[39718]: 24542) 5266 : cluster [DBG] pgmap v5695: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:09.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:09 smithi045 conmon[39718]: audit 2021-07-31T03:50:09.068465+0000 mon.a (mon. 2021-07-31T03:50:09.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:09 smithi045 conmon[39718]: 0) 3376 : audit [DBG] from='client.? 172.21.15.45:0/3027889574' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:50:10.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:10 smithi154 conmon[34553]: audit 2021-07-31T03:50:08.445260+0000 mgr.x (mgr.24542) 5267 : audit [DBG] 2021-07-31T03:50:10.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:10 smithi154 conmon[34553]: from='client.20334 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:50:10.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:10 smithi154 conmon[34553]: audit 2021-07-31T03:50:08.726258+0000 mgr.x (mgr.24542) 5268 : audit [DBG] from='client.20340 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:50:10.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:10 smithi045 conmon[33655]: audit 2021-07-31T03:50:08.445260 2021-07-31T03:50:10.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:10 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 5267 : audit [DBG] from='client.20334 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:50:10.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:10 smithi045 conmon[33655]: audit 2021-07-31T03:50:08.726258+0000 mgr.x (mgr.24542) 5268 : audit [DBG] from='client.20340 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:50:10.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:10 smithi045 conmon[39718]: audit 2021-07-31T03:50:08.445260+0000 mgr.x (mgr.24542) 2021-07-31T03:50:10.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:10 smithi045 conmon[39718]: 5267 : audit [DBG] from='client.20334 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:50:10.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:10 smithi045 conmon[39718]: audit 2021-07-31T03:50:08.726258+0000 mgr.x (mgr.24542) 5268 : audit [DBG] 2021-07-31T03:50:10.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:10 smithi045 conmon[39718]: from='client.20340 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:50:11.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:11 smithi154 conmon[34553]: cluster 2021-07-31T03:50:09.551025+0000 mgr.x (mgr.24542 2021-07-31T03:50:11.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:11 smithi154 conmon[34553]: ) 5269 : cluster [DBG] pgmap v5696: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:11.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:11 smithi045 conmon[39718]: cluster 2021-07-31T03:50:09.551025+0000 mgr.x (mgr.24542) 5269 : cluster [DBG] pgmap v5696: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:11.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:11 smithi045 conmon[33655]: cluster 2021-07-31T03:50:09.551025+0000 mgr.x (mgr.24542) 2021-07-31T03:50:11.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:11 smithi045 conmon[33655]: 5269 : cluster [DBG] pgmap v5696: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:13.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:13 smithi154 conmon[34553]: cluster 2021-07-31T03:50:11.551458+0000 mgr.x (mgr.24542) 5270 : 2021-07-31T03:50:13.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:13 smithi154 conmon[34553]: cluster [DBG] pgmap v5697: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:13.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:13 smithi045 conmon[39718]: cluster 2021-07-31T03:50:11.551458+0000 mgr.x (mgr 2021-07-31T03:50:13.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:13 smithi045 conmon[39718]: .24542) 5270 : cluster [DBG] pgmap v5697: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:13.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:13 smithi045 conmon[33655]: cluster 2021-07-31T03:50:11.551458+0000 mgr.x (mgr.24542) 5270 : cluster 2021-07-31T03:50:13.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:13 smithi045 conmon[33655]: [DBG] pgmap v5697: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:15.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:15 smithi154 conmon[34553]: cluster 2021-07-31T03:50:13.551889+0000 mgr.x (mgr.24542) 5271 : cluster 2021-07-31T03:50:15.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:15 smithi154 conmon[34553]: [DBG] pgmap v5698: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:15.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:15 smithi045 conmon[33655]: cluster 2021-07-31T03:50:13.551889+0000 mgr.x (mgr.24542) 5271 : cluster [DBG] pgmap v5698: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:15.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:15 smithi045 conmon[39718]: cluster 2021-07-31T03:50:13.551889+0000 2021-07-31T03:50:15.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:15 smithi045 conmon[39718]: mgr.x (mgr.24542) 5271 : cluster [DBG] pgmap v5698: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:16.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:16 smithi154 conmon[34553]: cluster 2021-07-31T03:50:15.552323+0000 mgr.x (mgr.24542) 5272 : cluster [DBG] 2021-07-31T03:50:16.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:16 smithi154 conmon[34553]: pgmap v5699: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:16.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:16 smithi045 conmon[39718]: cluster 2021-07-31T03:50:15. 2021-07-31T03:50:16.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:16 smithi045 conmon[39718]: 552323+0000 mgr.x (mgr.24542) 5272 : cluster [DBG] pgmap v5699: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:16.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:16 smithi045 conmon[33655]: cluster 2021-07-31T03:50 2021-07-31T03:50:16.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:16 smithi045 conmon[33655]: :15.552323+0000 mgr.x (mgr.24542) 5272 : cluster [DBG] pgmap v5699: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:19.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:18 smithi154 conmon[34553]: cluster 2021-07-31T03:50:17.552743+0000 mgr.x (mgr.24542) 5273 : cluster 2021-07-31T03:50:19.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:18 smithi154 conmon[34553]: [DBG] pgmap v5700: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:19.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:18 smithi045 conmon[39718]: cluster 2021-07-31T03:50:17.552743+0000 mgr.x 2021-07-31T03:50:19.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:18 smithi045 conmon[39718]: (mgr.24542) 5273 : cluster [DBG] pgmap v5700: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:19.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:18 smithi045 conmon[33655]: cluster 2021-07-31T03:50:19.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:18 smithi045 conmon[33655]: 2021-07-31T03:50:17.552743+0000 mgr.x (mgr.24542) 5273 : cluster [DBG] pgmap v5700: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:21.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:20 smithi154 conmon[34553]: cluster 2021-07-31T03:50:19.553325+0000 mgr.x (mgr.24542) 5274 : cluster [DBG] pgmap v5701: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:21.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:20 smithi045 conmon[39718]: cluster 2021-07-31T03:50:19.553325+0000 mgr.x ( 2021-07-31T03:50:21.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:20 smithi045 conmon[39718]: mgr.24542) 5274 : cluster [DBG] pgmap v5701: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:21.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:20 smithi045 conmon[33655]: cluster 2021-07-31T03:50:19.553325+0000 mgr.x (mgr.24542) 5274 : cluster 2021-07-31T03:50:21.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:20 smithi045 conmon[33655]: [DBG] pgmap v5701: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:23.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:22 smithi154 conmon[34553]: cluster 2021-07-31T03:50:21.553865+0000 mgr.x (mgr.24542) 5275 : cluster [DBG] pgmap v5702: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:23.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:22 smithi154 conmon[34553]: 2021-07-31T03:50:23.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:22 smithi045 conmon[39718]: cluster 2021-07-31T03:50:21.553865+0000 mgr.x 2021-07-31T03:50:23.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:22 smithi045 conmon[39718]: (mgr.24542) 5275 : cluster [DBG] pgmap v5702: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:23.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:22 smithi045 conmon[33655]: cluster 2021-07-31T03:50:21.553865+0000 mgr.x (mgr.24542) 5275 : 2021-07-31T03:50:23.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:22 smithi045 conmon[33655]: cluster [DBG] pgmap v5702: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:25.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:24 smithi154 conmon[34553]: cluster 2021-07-31T03:50:23.554293+0000 mgr.x (mgr.24542) 5276 : cluster [DBG] pgmap v5703: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:25.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:24 smithi045 conmon[39718]: cluster 2021-07-31T03:50:23.554293+0000 mgr.x (mgr 2021-07-31T03:50:25.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:24 smithi045 conmon[39718]: .24542) 5276 : cluster [DBG] pgmap v5703: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:25.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:24 smithi045 conmon[33655]: cluster 2021-07-31T03:50:23.554293+0000 mgr.x (mgr.24542) 5276 : cluster 2021-07-31T03:50:25.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:24 smithi045 conmon[33655]: [DBG] pgmap v5703: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:27.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:26 smithi154 conmon[34553]: cluster 2021-07-31T03:50:25.554772+0000 mgr.x (mgr.24542) 5277 : cluster [DBG] pgmap v5704: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:27.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:26 smithi045 conmon[33655]: cluster 2021-07-31T03:50:25.554772+0000 mgr.x (mgr.24542) 5277 : cluster [DBG] 2021-07-31T03:50:27.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:26 smithi045 conmon[33655]: pgmap v5704: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:27.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:26 smithi045 conmon[39718]: cluster 2021-07-31T03:50:25.554772+0000 mgr.x ( 2021-07-31T03:50:27.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:26 smithi045 conmon[39718]: mgr.24542) 5277 : cluster [DBG] pgmap v5704: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:29.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:28 smithi154 conmon[34553]: cluster 2021-07-31T03:50:27.555178+0000 mgr.x (mgr.24542) 5278 : cluster [DBG] pgmap v5705: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:29.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:28 smithi154 conmon[34553]: audit 2021-07-31T03:50:28.424566+0000 mon.b (mon.2) 690 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:50:29.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:28 smithi154 conmon[34553]: audit 2021-07-31T03:50:28.424810+0000 mon.b (mon.2) 691 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:50:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:28 smithi154 conmon[34553]: audit 2021 2021-07-31T03:50:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:28 smithi154 conmon[34553]: -07-31T03:50:28.425432+0000 mon.a (mon.0) 3377 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:50:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:28 smithi154 conmon[34553]: audit 2021- 2021-07-31T03:50:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:28 smithi154 conmon[34553]: 07-31T03:50:28.425811+0000 2021-07-31T03:50:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:28 smithi154 conmon[34553]: mon.a (mon.0) 3378 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:50:29.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:28 smithi154 conmon[34553]: 2021-07-31T03:50:29.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:28 smithi045 conmon[33655]: cluster 2021-07-31T03:50:27.555178+0000 mgr.x (mgr.24542) 5278 : cluster [DBG] pgmap v5705: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:29.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:28 smithi045 conmon[33655]: audit 2021-07-31T03:50:28.424566+0000 mon.b (mon.2) 690 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:50:29.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:28 smithi045 conmon[33655]: audit 2021-07-31T03: 2021-07-31T03:50:29.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:28 smithi045 conmon[33655]: 50:28.424810+0000 mon.b (mon.2) 691 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:50:29.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:28 smithi045 conmon[33655]: audit 2021-07- 2021-07-31T03:50:29.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:28 smithi045 conmon[33655]: 31T03:50:28.425432+0000 mon.a (mon.0) 2021-07-31T03:50:29.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:28 smithi045 conmon[33655]: 3377 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:50:29.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:28 smithi045 conmon[33655]: audit 2021-07- 2021-07-31T03:50:29.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:28 smithi045 conmon[33655]: 31T03:50:28.425811+0000 mon.a 2021-07-31T03:50:29.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:28 smithi045 conmon[33655]: (mon.0) 3378 : audit [INF] 2021-07-31T03:50:29.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:28 smithi045 conmon[33655]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:50:29.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:28 smithi045 conmon[39718]: cluster 2021-07-31T03:50:27.555178+0000 2021-07-31T03:50:29.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:28 smithi045 conmon[39718]: mgr.x (mgr.24542) 5278 : cluster [DBG] pgmap v5705: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:29.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:28 smithi045 conmon[39718]: audit 2021- 2021-07-31T03:50:29.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:28 smithi045 conmon[39718]: 07-31T03:50:28.424566+0000 mon.b (mon.2 2021-07-31T03:50:29.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:28 smithi045 conmon[39718]: ) 690 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:50:29.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:28 smithi045 conmon[39718]: audit 2021 2021-07-31T03:50:29.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:28 smithi045 conmon[39718]: -07-31T03:50:28.424810 2021-07-31T03:50:29.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:28 smithi045 conmon[39718]: +0000 mon.b (mon.2) 691 : audit 2021-07-31T03:50:29.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:28 smithi045 conmon[39718]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:50:29.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:28 smithi045 conmon[39718]: audit 2021-07-31T03:50 2021-07-31T03:50:29.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:28 smithi045 conmon[39718]: :28.425432+0000 mon.a (mon.0 2021-07-31T03:50:29.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:28 smithi045 conmon[39718]: ) 3377 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:50:29.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:28 smithi045 conmon[39718]: audit 2021- 2021-07-31T03:50:29.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:28 smithi045 conmon[39718]: 07-31T03:50:28.425811 2021-07-31T03:50:29.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:28 smithi045 conmon[39718]: +0000 mon.a (mon.0) 3378 : audit 2021-07-31T03:50:29.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:28 smithi045 conmon[39718]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:50:31.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:30 smithi154 conmon[34553]: cluster 2021-07-31T03:50:29.555760+0000 mgr.x (mgr.24542) 5279 : cluster [DBG] pgmap v5706: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:31.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:30 smithi045 conmon[33655]: cluster 2021-07-31T03:50:29.555760+0000 mgr.x (mgr.24542 2021-07-31T03:50:31.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:30 smithi045 conmon[33655]: ) 5279 : cluster [DBG] pgmap v5706: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:31.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:30 smithi045 conmon[39718]: cluster 2021-07-31T03:50:29.555760+0000 mgr.x (mgr.24542) 5279 : cluster [DBG] 2021-07-31T03:50:31.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:30 smithi045 conmon[39718]: pgmap v5706: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:33.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:32 smithi154 conmon[34553]: cluster 2021-07-31T03:50:31.556229+0000 mgr.x (mgr.24542) 5280 : cluster [DBG] pgmap v5707: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:33.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:32 smithi045 conmon[33655]: cluster 2021-07-31T03:50:31.556229+0000 mgr.x (mgr.24542) 5280 : cluster [DBG] 2021-07-31T03:50:33.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:32 smithi045 conmon[33655]: pgmap v5707: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:33.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:32 smithi045 conmon[39718]: cluster 2021-07-31T03:50:31.556229+0000 mgr.x (mgr.24542) 5280 2021-07-31T03:50:33.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:32 smithi045 conmon[39718]: : cluster [DBG] pgmap v5707: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:35.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:34 smithi154 conmon[34553]: cluster 2021-07-31T03:50:33.556721+0000 mgr.x (mgr.24542) 5281 : cluster [DBG] pgmap v5708: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:35.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:34 smithi045 conmon[39718]: cluster 2021-07-31T03:50:33.556721+0000 mgr.x 2021-07-31T03:50:35.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:34 smithi045 conmon[39718]: (mgr.24542) 5281 : cluster [DBG] pgmap v5708: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:35.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:34 smithi045 conmon[33655]: cluster 2021-07-31T03:50:33.556721 2021-07-31T03:50:35.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:34 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 5281 : cluster [DBG] pgmap v5708: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:35.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[34553]: audit 2021-07-31T03:50:35.402576+0000 mon.b (mon.2) 692 : 2021-07-31T03:50:35.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[34553]: audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:50:35.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[34553]: audit 2021-07-31T03:50:35.404070+0000 mon.b (mon.2) 693 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:50:35.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[34553]: audit 2021-07-31T03:50:35.404939+0000 mon.b (mon.2) 694 : 2021-07-31T03:50:35.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[34553]: audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:50:35.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: debug 2021-07-31T03:50:35.447+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:50:35.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:50:35.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:50:35.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: yield 2021-07-31T03:50:35.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:50:35.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:50:35.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:50:35.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:50:35.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:50:35.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:50:35.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:50:35.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:50:35.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:50:35.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: task.result() 2021-07-31T03:50:35.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:50:35.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: debug 2021-07-31T03:50:35.459+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:50:35.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:50:35.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:50:35.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: yield 2021-07-31T03:50:35.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:50:35.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:50:35.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:50:35.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:50:35.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:50:35.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:50:35.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:50:35.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:50:35.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:50:35.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: task.result() 2021-07-31T03:50:35.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:50:35.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: debug 2021-07-31T03:50:35.484+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:50:35.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:50:35.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:50:35.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: yield 2021-07-31T03:50:35.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:50:35.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:50:35.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:50:35.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:50:35.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:50:35.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:50:35.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:50:35.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:50:35.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:50:35.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: task.result() 2021-07-31T03:50:35.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:50:35.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: debug 2021-07-31T03:50:35.485+0000 7fcb9115c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:50:35.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:50:35.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:50:35.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: yield 2021-07-31T03:50:35.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:50:35.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:50:35.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:50:35.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:50:35.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:50:35.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:50:35.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:50:35.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:50:35.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:50:35.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: task.result() 2021-07-31T03:50:35.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:50:35.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: 2021-07-31T03:50:35.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:50:35.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: 2021-07-31T03:50:35.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:50:35.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:50:35.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:50:35.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:50:35.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:50:35.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:50:35.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:50:35.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:50:35.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:50:35.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:50:35.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:50:35.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:50:35.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:50:35.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:50:35.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:50:35.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:50:35.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:50:35.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:50:35.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: raise 2021-07-31T03:50:35.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:50:35.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:50:35.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:50:35.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:50:35.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:50:35.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: debug 2021-07-31T03:50:35.494+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:50:35.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:50:35.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:50:35.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: yield 2021-07-31T03:50:35.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:50:35.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:50:35.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:50:35.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:50:35.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:50:35.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:50:35.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:50:35.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:50:35.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:50:35.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: task.result() 2021-07-31T03:50:35.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:50:35.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: debug 2021-07-31T03:50:35.495+0000 7fcb9095b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:50:35.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:50:35.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:50:35.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: yield 2021-07-31T03:50:35.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:50:35.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:50:35.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:50:35.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:50:35.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:50:35.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:50:35.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:50:35.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:50:35.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:50:35.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: task.result() 2021-07-31T03:50:35.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:50:35.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: 2021-07-31T03:50:35.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:50:35.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: 2021-07-31T03:50:35.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:50:35.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:50:35.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:50:35.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:50:35.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:50:35.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:50:35.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:50:35.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:50:35.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:50:35.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:50:35.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:50:35.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:50:35.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:50:35.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:50:35.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:50:35.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:50:35.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:50:35.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:50:35.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: raise 2021-07-31T03:50:35.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:50:35.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:50:35.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:50:35.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:50:35.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:50:35 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:50:35.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:35 smithi045 conmon[39718]: audit 2021-07-31T03:50:35.402576+0000 mon.b (mon.2) 692 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:50:35.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:35 smithi045 conmon[39718]: audit 2021-07-31T03:50:35.404070+0000 2021-07-31T03:50:35.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:35 smithi045 conmon[39718]: mon.b (mon.2) 693 : audit [DBG] 2021-07-31T03:50:35.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:35 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:50:35.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:35 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T03:50:35.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:35 smithi045 conmon[39718]: 31T03:50:35.404939+0000 2021-07-31T03:50:35.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:35 smithi045 conmon[39718]: mon.b (mon.2) 694 : 2021-07-31T03:50:35.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:35 smithi045 conmon[39718]: audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:50:35.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:35 smithi045 conmon[33655]: audit 2021-07-31T03:50:35.402576+0000 mon.b (mon.2) 692 : audit [DBG] 2021-07-31T03:50:35.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:35 smithi045 conmon[33655]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:50:35.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:35 smithi045 conmon[33655]: audit 2021-07-31T03:50:35.404070+0000 mon.b (mon.2) 693 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:50:35.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:35 smithi045 conmon[33655]: audit 2021-07-31T03:50: 2021-07-31T03:50:35.788 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:35 smithi045 conmon[33655]: 35.404939+0000 mon.b (mon.2) 694 2021-07-31T03:50:35.788 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:35 smithi045 conmon[33655]: : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:50:37.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: cephadm 2021-07-31T03:50:35.405683+0000 mgr.x (mgr.24542) 5282 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:50:37.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: cephadm 2021-07-31T03:50:35.448428+0000 mgr.x (mgr.24542) 5283 : cephadm [ERR] 2021-07-31T03:50:37.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:50:37.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:50:37.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: yield 2021-07-31T03:50:37.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:50:37.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:50:37.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:50:37.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:50:37.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:50:37.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:50:37.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:50:37.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:50:37.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:50:37.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: task.result() 2021-07-31T03:50:37.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:50:37.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: cephadm 2021-07-31T03:50:35.448876+0000 mgr.x (mgr.24542) 5284 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:50:37.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: cephadm 2021-07-31T03:50:35.460091+0000 mgr.x (mgr.24542) 5285 : cephadm [ERR] 2021-07-31T03:50:37.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:50:37.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:50:37.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: yield 2021-07-31T03:50:37.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:50:37.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:50:37.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:50:37.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:50:37.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:50:37.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:50:37.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:50:37.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:50:37.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:50:37.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: task.result() 2021-07-31T03:50:37.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:50:37.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: cephadm 2021-07-31T03:50:35.460984+0000 mgr.x (mgr.24542) 5286 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:50:37.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: cephadm 2021-07-31T03:50:35.485714+0000 mgr.x (mgr.24542) 5287 : cephadm [ERR] 2021-07-31T03:50:37.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:50:37.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:50:37.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: yield 2021-07-31T03:50:37.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:50:37.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:50:37.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:50:37.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:50:37.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:50:37.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:50:37.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:50:37.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:50:37.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:50:37.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: task.result() 2021-07-31T03:50:37.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:50:37.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: cephadm 2021-07-31T03:50:35.486380+0000 mgr.x (mgr.24542) 5288 : cephadm [ERR] 2021-07-31T03:50:37.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:50:37.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:50:37.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:50:37.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: yield 2021-07-31T03:50:37.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:50:37.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:50:37.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:50:37.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:50:37.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:50:37.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:50:37.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:50:37.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:50:37.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:50:37.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: task.result() 2021-07-31T03:50:37.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:50:37.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: 2021-07-31T03:50:37.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:50:37.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: 2021-07-31T03:50:37.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:50:37.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:50:37.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:50:37.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:50:37.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:50:37.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:50:37.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:50:37.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:50:37.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:50:37.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:50:37.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:50:37.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:50:37.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:50:37.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:50:37.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:50:37.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:50:37.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:50:37.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:50:37.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: raise 2021-07-31T03:50:37.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:50:37.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:50:37.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:50:37.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:50:37.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:50:37.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: cephadm 2021-07-31T03:50:35.495513+0000 mgr.x (mgr.24542) 5289 : cephadm [ERR] 2021-07-31T03:50:37.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:50:37.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:50:37.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: yield 2021-07-31T03:50:37.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:50:37.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:50:37.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:50:37.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:50:37.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:50:37.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:50:37.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:50:37.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:50:37.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:50:37.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: task.result() 2021-07-31T03:50:37.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:50:37.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: cephadm 2021-07-31T03:50:35 2021-07-31T03:50:37.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: .495934+0000 mgr.x (mgr.24542) 5290 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:50:37.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:50:37.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:50:37.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: yield 2021-07-31T03:50:37.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:50:37.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:50:37.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:50:37.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:50:37.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:50:37.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:50:37.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: cephadm 2021-07-31T03:50:35.405683+0000 mgr.x (mgr. 2021-07-31T03:50:37.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: 24542) 5282 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:50:37.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: cephadm 2021-07-31T03:50:35.448428+0000 mgr.x (mgr. 2021-07-31T03:50:37.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: 24542) 5283 : cephadm [ERR] 2021-07-31T03:50:37.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:50:37.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:50:37.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: yield 2021-07-31T03:50:37.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:50:37.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:50:37.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:50:37.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:50:37.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:50:37.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:50:37.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:50:37.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:50:37.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:50:37.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: task.result() 2021-07-31T03:50:37.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:50:37.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: cephadm 2021-07-31T03:50:35.448876+0000 mgr.x (mgr.24542) 5284 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:50:37.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: cephadm 2021-07-31T03:50 2021-07-31T03:50:37.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: :35.460091+0000 mgr.x (mgr.24542 2021-07-31T03:50:37.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: ) 5285 : cephadm [ERR] 2021-07-31T03:50:37.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:50:37.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:50:37.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: yield 2021-07-31T03:50:37.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:50:37.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:50:37.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:50:37.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:50:37.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:50:37.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:50:37.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:50:37.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:50:37.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:50:37.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: task.result() 2021-07-31T03:50:37.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:50:37.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: 2021-07-31T03:50:37.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: cephadm 2021-07-31T03:50:35.460984+0000 mgr.x (mgr.24542) 5286 2021-07-31T03:50:37.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:50:37.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: cephadm 2021 2021-07-31T03:50:37.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: -07-31T03:50:35. 2021-07-31T03:50:37.037 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:50:37.037 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:50:37.038 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:50:37.038 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: task.result() 2021-07-31T03:50:37.038 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:50:37.038 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: 2021-07-31T03:50:37.038 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:50:37.039 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: 2021-07-31T03:50:37.039 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:50:37.039 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:50:37.039 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:50:37.039 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:50:37.040 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:50:37.040 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:50:37.040 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:50:37.040 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:50:37.040 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:50:37.041 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:50:37.041 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:50:37.041 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:50:37.041 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:50:37.041 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:50:37.042 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:50:37.042 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:50:37.042 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:50:37.042 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:50:37.042 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: raise 2021-07-31T03:50:37.043 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:50:37.043 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:50:37.043 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:50:37.043 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:50:37.043 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:50:37.044 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:36 smithi154 conmon[34553]: cluster 2021-07-31T03:50:35.557161+0000 mgr.x (mgr.24542) 5291 : cluster [DBG] pgmap v5709: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:37.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: cephadm 2021-07 2021-07-31T03:50:37.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: -31T03:50:35.405683+0000 mgr.x (mgr.24542) 5282 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:50:37.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: cephadm 2021- 2021-07-31T03:50:37.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: 07-31T03:50:35.448428 2021-07-31T03:50:37.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 2021-07-31T03:50:37.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: 5283 : cephadm [ERR] 2021-07-31T03:50:37.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:50:37.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:50:37.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: yield 2021-07-31T03:50:37.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:50:37.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:50:37.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:50:37.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:50:37.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:50:37.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:50:37.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:50:37.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:50:37.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:50:37.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: task.result() 2021-07-31T03:50:37.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:50:37.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: cephadm 2021-07-31T03:50:35.448876+0000 mgr.x (mgr.24542) 5284 : cephadm 2021-07-31T03:50:37.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:50:37.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: cephadm 2021- 2021-07-31T03:50:37.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: 07-31T03:50:35. 2021-07-31T03:50:37.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: 460091+0000 mgr.x (mgr.24542 2021-07-31T03:50:37.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: ) 5285 : cephadm [ERR] 2021-07-31T03:50:37.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:50:37.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:50:37.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: yield 2021-07-31T03:50:37.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:50:37.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:50:37.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:50:37.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:50:37.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:50:37.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:50:37.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:50:37.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:50:37.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:50:37.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: task.result() 2021-07-31T03:50:37.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:50:37.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: 2021-07-31T03:50:37.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: cephadm 2021-07-31T03:50:35.460984+0000 mgr.x (mgr.24542) 5286 2021-07-31T03:50:37.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:50:37.053 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: 485714+0000 mgr.x (mgr.24542 2021-07-31T03:50:37.053 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: ) 5287 : cephadm [ERR] 2021-07-31T03:50:37.053 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:50:37.054 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:50:37.054 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: yield 2021-07-31T03:50:37.054 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:50:37.054 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:50:37.055 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:50:37.055 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:50:37.055 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:50:37.055 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:50:37.055 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:50:37.056 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:50:37.056 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:50:37.056 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: task.result() 2021-07-31T03:50:37.056 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:50:37.056 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: 2021-07-31T03:50:37.057 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: cephadm 2021-07-31T03:50:35.486380+0000 2021-07-31T03:50:37.057 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: mgr.x (mgr.24542) 5288 : cephadm [ERR] 2021-07-31T03:50:37.057 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:50:37.057 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:50:37.057 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:50:37.058 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: yield 2021-07-31T03:50:37.058 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:50:37.058 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:50:37.058 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:50:37.058 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:50:37.059 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:50:37.059 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:50:37.059 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:50:37.059 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:50:37.059 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:50:37.060 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: task.result() 2021-07-31T03:50:37.060 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:50:37.060 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: 2021-07-31T03:50:37.060 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:50:37.061 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: 2021-07-31T03:50:37.061 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:50:37.061 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:50:37.061 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:50:37.061 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:50:37.062 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:50:37.062 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:50:37.062 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:50:37.062 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: cephadm 2021 2021-07-31T03:50:37.063 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: -07-31T03:50:35. 2021-07-31T03:50:37.063 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: 485714+0000 mgr.x (mgr.24542 2021-07-31T03:50:37.063 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: ) 5287 : cephadm [ERR] 2021-07-31T03:50:37.063 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:50:37.063 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:50:37.064 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: yield 2021-07-31T03:50:37.064 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:50:37.064 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:50:37.064 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:50:37.064 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:50:37.065 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:50:37.065 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:50:37.065 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:50:37.065 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:50:37.065 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:50:37.066 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: task.result() 2021-07-31T03:50:37.066 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:50:37.066 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: 2021-07-31T03:50:37.066 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: cephadm 2021-07-31T03:50:35.486380+0000 mgr.x 2021-07-31T03:50:37.066 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: (mgr.24542) 5288 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:50:37.067 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:50:37.067 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:50:37.067 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: yield 2021-07-31T03:50:37.067 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:50:37.067 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:50:37.068 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:50:37.068 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:50:37.068 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:50:37.068 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:50:37.068 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:50:37.069 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:50:37.069 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:50:37.069 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: task.result() 2021-07-31T03:50:37.069 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:50:37.069 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: 2021-07-31T03:50:37.070 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:50:37.070 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: 2021-07-31T03:50:37.070 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:50:37.070 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:50:37.070 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:50:37.071 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:50:37.071 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:50:37.071 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:50:37.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:50:37.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:50:37.073 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:50:37.073 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:50:37.073 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:50:37.073 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:50:37.073 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:50:37.074 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:50:37.074 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:50:37.074 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:50:37.074 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:50:37.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: raise 2021-07-31T03:50:37.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:50:37.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:50:37.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:50:37.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:50:37.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:50:37.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: cephadm 2021-07-31T03:50:35.495513+0000 mgr.x (mgr.24542) 5289 : cephadm [ERR] 2021-07-31T03:50:37.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:50:37.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:50:37.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: yield 2021-07-31T03:50:37.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:50:37.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:50:37.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:50:37.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:50:37.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:50:37.078 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:50:37.078 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:50:37.078 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:50:37.078 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:50:37.079 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: task.result() 2021-07-31T03:50:37.079 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:50:37.079 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: cephadm 2021-07-31T03:50:35.495934+0000 mgr.x (mgr.24542) 5290 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:50:37.079 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:50:37.079 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:50:37.080 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: yield 2021-07-31T03:50:37.080 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:50:37.080 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:50:37.080 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:50:37.081 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:50:37.081 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:50:37.081 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:50:37.081 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:50:37.081 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:50:37.082 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:50:37.082 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:50:37.082 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:50:37.082 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:50:37.082 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:50:37.083 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:50:37.083 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: raise 2021-07-31T03:50:37.083 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:50:37.083 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:50:37.083 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:50:37.084 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:50:37.084 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:50:37.084 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: 2021-07-31T03:50:37.084 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: cephadm 2021-07-31T03:50:35.495513+0000 mgr.x (mgr.24542) 5289 : cephadm [ERR] 2021-07-31T03:50:37.084 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:50:37.085 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:50:37.085 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: yield 2021-07-31T03:50:37.085 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:50:37.085 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:50:37.085 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:50:37.086 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:50:37.086 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:50:37.086 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:50:37.086 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:50:37.086 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:50:37.087 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:50:37.087 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: task.result() 2021-07-31T03:50:37.087 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:50:37.087 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: cephadm 2021-07-31T03:50:35.495934+0000 mgr.x (mgr.24542) 5290 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:50:37.087 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:50:37.088 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:50:37.088 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: yield 2021-07-31T03:50:37.088 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:50:37.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:50:37.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:50:37.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:50:37.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:50:37.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:50:37.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:50:37.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:50:37.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: task.result() 2021-07-31T03:50:37.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:50:37.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: 2021-07-31T03:50:37.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:50:37.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: 2021-07-31T03:50:37.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:50:37.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:50:37.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:50:37.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:50:37.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:50:37.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:50:37.093 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:50:37.093 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:50:37.093 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:50:37.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:50:37.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:50:37.095 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:50:37.095 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:50:37.095 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:50:37.095 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:50:37.096 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:50:37.096 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:50:37.096 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:50:37.097 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: raise 2021-07-31T03:50:37.097 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:50:37.097 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:50:37.097 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:50:37.098 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:50:37.098 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:50:37.098 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: cluster 2021-07-31T03:50:35.557161+0000 2021-07-31T03:50:37.099 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[33655]: mgr.x (mgr.24542) 5291 : cluster [DBG] pgmap v5709: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:37.099 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:50:37.099 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:50:37.100 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:50:37.100 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:50:37.100 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:50:37.101 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:50:37.101 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:50:37.101 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:50:37.102 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: task.result() 2021-07-31T03:50:37.102 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:50:37.102 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: 2021-07-31T03:50:37.102 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:50:37.103 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: 2021-07-31T03:50:37.103 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:50:37.103 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:50:37.103 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:50:37.104 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:50:37.104 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:50:37.104 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:50:37.104 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:50:37.104 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:50:37.105 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:50:37.105 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:50:37.105 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:50:37.105 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:50:37.106 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:50:37.106 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:50:37.106 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:50:37.106 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:50:37.107 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:50:37.107 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:50:37.107 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: raise 2021-07-31T03:50:37.107 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:50:37.107 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:50:37.108 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:50:37.108 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:50:37.108 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:50:37.108 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:36 smithi045 conmon[39718]: cluster 2021-07-31T03:50:35.557161+0000 mgr.x (mgr.24542) 5291 : cluster [DBG] pgmap v5709: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:39.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:38 smithi154 conmon[34553]: cluster 2021-07-31T03:50:37.557518+0000 mgr.x (mgr.24542) 5292 : cluster [DBG] pgmap v5710: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:39.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:38 smithi045 conmon[39718]: cluster 2021-07-31T03:50:37.557518 2021-07-31T03:50:39.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:38 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 5292 : cluster [DBG] pgmap v5710: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:39.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:38 smithi045 conmon[33655]: cluster 2021-07-31T 2021-07-31T03:50:39.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:38 smithi045 conmon[33655]: 03:50:37.557518+0000 mgr.x (mgr.24542) 5292 : cluster [DBG] pgmap v5710: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:39.381 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:50:39.661 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:50:39.662 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 2h ago 2h - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:50:39.662 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 2h ago 2h 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:50:39.662 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 2h ago 2h 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:50:39.662 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 2h ago 2h - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:50:39.663 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:50:39.663 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 2h ago 2h 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:50:39.663 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:50:39.663 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 2h ago 2h - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:50:39.663 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 2h ago 2h 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:50:39.664 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:50:39.664 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:50:39.664 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:50:39.664 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:50:39.664 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 2h ago 2h 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:50:39.665 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 2h ago 2h 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:50:39.665 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 2h ago 2h 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:50:39.665 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 2h ago 2h 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:50:39.665 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 2h ago 2h 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:50:39.998 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:50:39.998 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:50:39.999 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:50:39.999 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:50:39.999 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:50:39.999 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:50:39.999 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:50:40.000 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:50:40.000 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:50:40.000 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:50:40.000 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:50:40.000 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:50:40.001 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:50:40.001 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:50:40.001 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:50:40.001 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:50:40.001 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:50:41.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:40 smithi154 conmon[34553]: audit 2021-07-31T03:50:39.370031+0000 mgr.x (mgr.24542) 5293 : audit [DBG] from='client.30289 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:50:41.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:40 smithi154 conmon[34553]: cluster 2021-07-31T03:50:39.558120+0000 2021-07-31T03:50:41.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:40 smithi154 conmon[34553]: mgr.x (mgr.24542) 5294 : cluster [DBG] pgmap v5711: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:41.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:40 smithi154 conmon[34553]: audit 2021-07-31T03:50:39.654618+0000 mgr.x (mgr.24542) 5295 : audit [DBG] from='client.30295 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:50:41.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:40 smithi154 conmon[34553]: audit 2021-07-31T03:50:39.999404+0000 mon.c (mon.1) 230 : audit [DBG] from='client.? 172.21.15.45:0/2156038738' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:50:41.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:40 smithi045 conmon[39718]: audit 2021-07-31T03:50:39.370031+0000 mgr.x ( 2021-07-31T03:50:41.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:40 smithi045 conmon[39718]: mgr.24542) 5293 : audit [DBG] from='client.30289 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:50:41.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:40 smithi045 conmon[39718]: cluster 2021-07-31T03 2021-07-31T03:50:41.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:40 smithi045 conmon[39718]: :50:39.558120+0000 mgr.x (mgr.24542) 5294 : cluster [DBG] pgmap v5711: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:41.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:40 smithi045 conmon[39718]: audit 2021-07-31T03:50:39.654618+0000 2021-07-31T03:50:41.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:40 smithi045 conmon[39718]: mgr.x (mgr.24542) 5295 : audit [DBG] from='client.30295 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:50:41.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:40 smithi045 conmon[39718]: audit 2021-07-31T03:50:41.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:40 smithi045 conmon[39718]: 2021-07-31T03:50:39.999404 2021-07-31T03:50:41.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:40 smithi045 conmon[39718]: +0000 mon.c (mon.1) 230 : 2021-07-31T03:50:41.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:40 smithi045 conmon[39718]: audit [DBG] from='client.? 172.21.15.45:0/2156038738' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:50:41.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:40 smithi045 conmon[33655]: audit 2021-07-31T03:50:39.370031+0000 mgr.x (mgr 2021-07-31T03:50:41.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:40 smithi045 conmon[33655]: .24542) 5293 : audit [DBG] from='client.30289 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:50:41.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:40 smithi045 conmon[33655]: cluster 2021-07-31T03:50:39.558120+0000 mgr.x (mgr.24542) 5294 : cluster [DBG] pgmap v5711: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:41.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:40 smithi045 conmon[33655]: audit 2021-07-31 2021-07-31T03:50:41.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:40 smithi045 conmon[33655]: T03:50:39.654618+0000 mgr.x (mgr.24542) 5295 : audit [DBG] from='client.30295 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:50:41.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:40 smithi045 conmon[33655]: audit 2021-07-31T03:50:39.999404+0000 mon.c (mon 2021-07-31T03:50:41.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:40 smithi045 conmon[33655]: .1) 230 : audit [DBG] from='client.? 172.21.15.45:0/2156038738' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:50:43.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:42 smithi154 conmon[34553]: cluster 2021-07-31T03:50:41.558657+0000 mgr.x (mgr.24542) 5296 : cluster [DBG] pgmap v5712: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:43.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:42 smithi045 conmon[39718]: cluster 2021-07-31T03:50:41.558657+0000 mgr.x ( 2021-07-31T03:50:43.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:42 smithi045 conmon[39718]: mgr.24542) 5296 : cluster [DBG] pgmap v5712: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:43.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:42 smithi045 conmon[33655]: cluster 2021-07-31T03:50:41.558657+0000 mgr.x (mgr.24542) 5296 : 2021-07-31T03:50:43.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:42 smithi045 conmon[33655]: cluster [DBG] pgmap v5712: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:45.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:44 smithi154 conmon[34553]: cluster 2021-07-31T03:50:43.559200+0000 mgr.x (mgr.24542) 5297 : cluster [DBG] pgmap v5713: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:45.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:44 smithi045 conmon[39718]: cluster 2021-07-31T03:50:43.559200+0000 mgr.x (mgr.24542 2021-07-31T03:50:45.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:44 smithi045 conmon[39718]: ) 5297 : cluster [DBG] pgmap v5713: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:45.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:44 smithi045 conmon[33655]: cluster 2021-07-31T03:50:43.559200+0000 mgr.x (mgr.24542) 5297 : cluster 2021-07-31T03:50:45.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:44 smithi045 conmon[33655]: [DBG] pgmap v5713: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:47.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:46 smithi154 conmon[34553]: cluster 2021-07-31T03:50:45.559702+0000 mgr.x (mgr.24542) 5298 : cluster [DBG] pgmap v5714: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:47.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:46 smithi045 conmon[39718]: cluster 2021-07-31T03:50:45.559702+0000 mgr.x ( 2021-07-31T03:50:47.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:46 smithi045 conmon[39718]: mgr.24542) 5298 : cluster [DBG] pgmap v5714: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:47.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:46 smithi045 conmon[33655]: cluster 2021-07-31T03:50:45.559702+0000 mgr.x (mgr.24542 2021-07-31T03:50:47.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:46 smithi045 conmon[33655]: ) 5298 : cluster [DBG] pgmap v5714: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:49.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:48 smithi154 conmon[34553]: cluster 2021-07-31T03:50:47.560174+0000 mgr.x (mgr.24542) 5299 : cluster [DBG] pgmap v5715: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:49.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:48 smithi045 conmon[39718]: cluster 2021-07-31T03:50:47.560174+0000 mgr.x (mgr.24542) 2021-07-31T03:50:49.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:48 smithi045 conmon[39718]: 5299 : cluster [DBG] pgmap v5715: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:49.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:48 smithi045 conmon[33655]: cluster 2021-07-31T03:50:47.560174+0000 mgr.x (mgr.24542) 5299 : 2021-07-31T03:50:49.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:48 smithi045 conmon[33655]: cluster [DBG] pgmap v5715: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:51.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:50 smithi154 conmon[34553]: cluster 2021-07-31T03:50:49.560568+0000 mgr.x (mgr.24542) 5300 : cluster [DBG] pgmap v5716: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:51.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:50 smithi045 conmon[39718]: cluster 2021-07-31T03:50:49.560568+0000 mgr.x (mgr 2021-07-31T03:50:51.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:50 smithi045 conmon[39718]: .24542) 5300 : cluster [DBG] pgmap v5716: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:51.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:50 smithi045 conmon[33655]: cluster 2021-07-31T03:50:49.560568+0000 mgr.x (mgr.24542) 5300 : cluster [DBG] 2021-07-31T03:50:51.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:50 smithi045 conmon[33655]: pgmap v5716: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:53.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:52 smithi154 conmon[34553]: cluster 2021-07-31T03:50:51.561092+0000 mgr.x (mgr.24542) 5301 : cluster [DBG] pgmap v5717: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:53.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:52 smithi045 conmon[33655]: cluster 2021-07-31T03:50:51.561092+0000 mgr.x (mgr.24542) 2021-07-31T03:50:53.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:52 smithi045 conmon[33655]: 5301 : cluster [DBG] pgmap v5717: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:53.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:52 smithi045 conmon[39718]: cluster 2021-07-31T03:50:51.561092+0000 mgr.x (mgr 2021-07-31T03:50:53.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:52 smithi045 conmon[39718]: .24542) 5301 : cluster [DBG] pgmap v5717: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:55.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:54 smithi154 conmon[34553]: cluster 2021-07-31T03:50:53.561484+0000 mgr.x (mgr.24542) 5302 : cluster [DBG] pgmap v5718: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:55.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:54 smithi045 conmon[33655]: cluster 2021-07-31T03:50:53.561484+0000 mgr.x (mgr.24542) 5302 : cluster [DBG] 2021-07-31T03:50:55.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:54 smithi045 conmon[33655]: pgmap v5718: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:55.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:54 smithi045 conmon[39718]: cluster 2021-07-31T03:50:53.561484+0000 2021-07-31T03:50:55.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:54 smithi045 conmon[39718]: mgr.x (mgr.24542) 5302 : cluster [DBG] pgmap v5718: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:57.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:56 smithi154 conmon[34553]: cluster 2021-07-31T03:50:55.561953+0000 mgr.x (mgr.24542) 5303 : cluster [DBG] pgmap v5719: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:57.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:56 smithi045 conmon[39718]: cluster 2021-07-31T03:50:55.561953+0000 mgr.x (mgr 2021-07-31T03:50:57.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:56 smithi045 conmon[39718]: .24542) 5303 : cluster [DBG] pgmap v5719: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:57.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:56 smithi045 conmon[33655]: cluster 2021-07-31T03:50:55.561953+0000 mgr.x (mgr.24542) 5303 : cluster [DBG] pgmap v5719: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:59.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:50:58 smithi154 conmon[34553]: cluster 2021-07-31T03:50:57.562370+0000 mgr.x (mgr.24542) 5304 : cluster [DBG] pgmap v5720: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:59.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:58 smithi045 conmon[33655]: cluster 2021-07-31T03:50:57.562370+0000 mgr.x (mgr.24542) 5304 : cluster 2021-07-31T03:50:59.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:50:58 smithi045 conmon[33655]: [DBG] pgmap v5720: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:59.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:58 smithi045 conmon[39718]: cluster 2021-07-31T03:50:57.562370+0000 mgr.x (mgr.24542) 5304 : cluster [DBG] pgmap v5720: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:50:59.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:50:58 smithi045 conmon[39718]: 2021-07-31T03:51:01.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:00 smithi154 conmon[34553]: cluster 2021-07-31T03:50:59.562937+0000 mgr.x (mgr.24542) 5305 : cluster [DBG] pgmap v5721: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:51:01.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:00 smithi045 conmon[33655]: cluster 2021-07-31T03:50:59.562937+0000 mgr.x (mgr.24542) 5305 : cluster [DBG] pgmap v5721: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:51:01.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:00 smithi045 conmon[33655]: 2021-07-31T03:51:01.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:00 smithi045 conmon[39718]: cluster 2021-07-31T03:50:59.562937+0000 mgr.x (mgr.24542) 5305 : cluster [DBG] pgmap v5721: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:51:01.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:00 smithi045 conmon[39718]: 2021-07-31T03:51:03.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:02 smithi154 conmon[34553]: cluster 2021-07-31T03:51:01.563315+0000 mgr.x (mgr.24542) 5306 : cluster [DBG] pgmap v5722: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:51:03.024 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:02 smithi045 conmon[33655]: cluster 2021-07-31T03:51:01.563315+0000 mgr.x (mgr.24542) 2021-07-31T03:51:03.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:02 smithi045 conmon[33655]: 5306 : cluster [DBG] pgmap v5722: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:51:03.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:02 smithi045 conmon[39718]: cluster 2021-07-31T03:51:01.563315+0000 mgr.x (mgr.24542) 5306 : cluster [DBG] pgmap v5722: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:51:05.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:04 smithi154 conmon[34553]: cluster 2021-07-31T03:51:03.563795+0000 mgr.x (mgr.24542) 5307 : cluster [DBG] pgmap v5723: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:51:05.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:04 smithi045 conmon[39718]: cluster 2021-07-31T03:51:03.563795+0000 mgr.x (mgr.24542) 5307 : cluster [DBG] pgmap v5723: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:51:05.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:04 smithi045 conmon[33655]: cluster 2021-07-31T03:51:03.563795+0000 mgr.x (mgr. 2021-07-31T03:51:05.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:04 smithi045 conmon[33655]: 24542) 5307 : cluster [DBG] pgmap v5723: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:51:07.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:06 smithi154 conmon[34553]: cluster 2021-07-31T03:51:05.564392+0000 mgr.x (mgr.24542) 5308 : cluster [DBG] pgmap v5724: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:51:07.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:06 smithi045 conmon[39718]: cluster 2021-07-31T03:51:05.564392+0000 mgr.x (mgr.24542) 5308 : cluster [DBG] pgmap v5724: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:51:07.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:06 smithi045 conmon[33655]: cluster 2021-07-31T03:51:05.564392+0000 mgr.x (mgr.24542) 5308 : cluster [DBG] 2021-07-31T03:51:07.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:06 smithi045 conmon[33655]: pgmap v5724: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:51:09.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:08 smithi154 conmon[34553]: cluster 2021-07-31T03:51:07.564835+0000 mgr.x (mgr 2021-07-31T03:51:09.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:08 smithi154 conmon[34553]: .24542) 5309 : cluster [DBG] pgmap v5725: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:51:09.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:08 smithi045 conmon[39718]: cluster 2021-07-31T03:51:07.564835+0000 mgr.x (mgr.24542) 5309 : cluster [DBG] pgmap v5725: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:51:09.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:08 smithi045 conmon[33655]: cluster 2021-07-31T03:51:07.564835+0000 mgr.x (mgr.24542) 5309 : cluster [DBG] 2021-07-31T03:51:09.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:08 smithi045 conmon[33655]: pgmap v5725: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:51:10.312 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:51:10.591 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:51:10.591 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 2h ago 2h - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:51:10.592 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 2h ago 2h 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:51:10.592 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 2h ago 2h 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:51:10.592 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 2h ago 2h - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:51:10.592 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:51:10.592 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 2h ago 2h 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:51:10.593 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:51:10.593 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 2h ago 2h - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:51:10.593 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 2h ago 2h 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:51:10.593 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:51:10.593 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:51:10.594 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:51:10.594 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:51:10.594 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 2h ago 2h 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:51:10.594 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 2h ago 2h 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:51:10.594 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 2h ago 2h 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:51:10.595 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 2h ago 2h 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:51:10.595 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 2h ago 2h 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:51:10.930 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:51:10.930 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:51:10.930 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:51:10.931 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:51:10.931 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:51:10.931 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:51:10.931 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:51:10.932 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:51:10.932 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:51:10.932 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:51:10.932 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:51:10.932 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:51:10.933 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:51:10.933 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:51:10.933 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:51:10.933 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:51:10.933 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:51:11.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:10 smithi154 conmon[34553]: cluster 2021-07-31T03:51:09.565415+0000 mgr.x (mgr.24542) 5310 : cluster [DBG] pgmap v5726: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:51:11.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:10 smithi045 conmon[39718]: cluster 2021-07-31T03:51:09.565415+0000 mgr.x (mgr 2021-07-31T03:51:11.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:10 smithi045 conmon[39718]: .24542) 5310 : cluster [DBG] pgmap v5726: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:51:11.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:10 smithi045 conmon[33655]: cluster 2021-07-31T03:51:09.565415+0000 mgr.x ( 2021-07-31T03:51:11.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:10 smithi045 conmon[33655]: mgr.24542) 5310 : cluster [DBG] pgmap v5726: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:51:12.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:11 smithi154 conmon[34553]: audit 2021-07-31T03:51:10.301070+0000 mgr.x (mgr.24542) 5311 : audit [DBG] from='client.20370 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:51:12.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:11 smithi154 conmon[34553]: audit 2021-07-31T03:51:10.583987+0000 mgr.x (mgr.24542) 5312 : audit [DBG] from='client.20376 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:51:12.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:11 smithi154 conmon[34553]: audit 2021-07-31T03:51:10.930682+0000 mon.a (mon.0) 3379 : audit [DBG] from='client.? 172.21.15.45:0/3078445353' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:51:12.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:11 smithi045 conmon[33655]: audit 2021-07-31T03:51:10.301070+0000 mgr.x (mgr.24542 2021-07-31T03:51:12.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:11 smithi045 conmon[33655]: ) 5311 : audit [DBG] from='client.20370 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:51:12.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:11 smithi045 conmon[33655]: audit 2021-07-31T03:51:10.583987+0000 mgr.x (mgr.24542) 5312 : audit [DBG] from='client.20376 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:51:12.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:11 smithi045 conmon[33655]: audit 2021-07-31T03:51:10.930682+0000 mon.a (mon.0) 3379 : audit [DBG] from='client.? 172.21.15.45:0/3078445353' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:51:12.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:11 smithi045 conmon[39718]: audit 2021-07-31T03:51:10.301070+0000 mgr.x 2021-07-31T03:51:12.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:11 smithi045 conmon[39718]: (mgr.24542) 5311 : audit [DBG] from='client.20370 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:51:12.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:11 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T03:51:12.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:11 smithi045 conmon[39718]: 03:51:10.583987+0000 mgr.x (mgr.24542) 5312 : audit 2021-07-31T03:51:12.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:11 smithi045 conmon[39718]: [DBG] from='client.20376 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:51:12.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:11 smithi045 conmon[39718]: audit 2021-07-31T03 2021-07-31T03:51:12.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:11 smithi045 conmon[39718]: :51:10.930682+0000 mon.a (mon. 2021-07-31T03:51:12.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:11 smithi045 conmon[39718]: 0) 3379 : audit [DBG] from='client.? 172.21.15.45:0/3078445353' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:51:13.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:12 smithi154 conmon[34553]: cluster 2021-07-31T03:51:11.565863+0000 mgr.x (mgr.24542) 5313 : cluster [DBG] pgmap v5727: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:51:13.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:12 smithi045 conmon[33655]: cluster 2021-07-31T03:51:11.565863+0000 2021-07-31T03:51:13.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:12 smithi045 conmon[33655]: mgr.x (mgr.24542) 5313 : cluster [DBG] pgmap v5727: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:51:13.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:12 smithi045 conmon[39718]: cluster 2021-07-31T03:51:11.565863+0000 mgr.x (mgr.24542) 5313 : cluster [DBG] pgmap v5727: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:51:15.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:14 smithi154 conmon[34553]: cluster 2021-07-31T03:51:13.566373+0000 mgr.x (mgr.24542) 5314 : cluster [DBG] pgmap v5728: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:51:15.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:14 smithi045 conmon[33655]: cluster 2021-07-31T03:51:13.566373+0000 mgr.x (mgr.24542) 5314 2021-07-31T03:51:15.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:14 smithi045 conmon[33655]: : cluster [DBG] pgmap v5728: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:51:15.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:14 smithi045 conmon[39718]: cluster 2021-07-31T03:51:13.566373+0000 mgr.x (mgr 2021-07-31T03:51:15.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:14 smithi045 conmon[39718]: .24542) 5314 : cluster [DBG] pgmap v5728: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:51:17.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:16 smithi154 conmon[34553]: cluster 2021-07-31T03:51:15.566888+0000 mgr.x (mgr.24542) 5315 : cluster [DBG] pgmap v5729: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:51:17.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:16 smithi045 conmon[33655]: cluster 2021-07-31T03:51:15.566888+0000 2021-07-31T03:51:17.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:16 smithi045 conmon[33655]: mgr.x (mgr.24542) 5315 : cluster [DBG] pgmap v5729: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:51:17.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:16 smithi045 conmon[39718]: cluster 2021-07-31T03:51:15.566888+0000 mgr.x (mgr 2021-07-31T03:51:17.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:16 smithi045 conmon[39718]: .24542) 5315 : cluster [DBG] pgmap v5729: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:51:19.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:18 smithi154 conmon[34553]: cluster 2021-07-31T03:51:17.567341+0000 mgr.x (mgr.24542) 5316 : cluster [DBG] pgmap v5730: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:51:19.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:18 smithi045 conmon[33655]: cluster 2021-07-31T03:51:17.567341+0000 mgr.x 2021-07-31T03:51:19.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:18 smithi045 conmon[33655]: (mgr.24542) 5316 : cluster [DBG] pgmap v5730: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:51:19.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:18 smithi045 conmon[39718]: cluster 2021-07-31T03:51:17.567341+0000 mgr.x (mgr.24542 2021-07-31T03:51:19.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:18 smithi045 conmon[39718]: ) 5316 : cluster [DBG] pgmap v5730: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:51:21.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:20 smithi154 conmon[34553]: cluster 2021-07-31T03:51:19.567782+0000 mgr.x (mgr.24542) 5317 : cluster [DBG] pgmap v5731: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:51:21.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:20 smithi154 conmon[34553]: 2021-07-31T03:51:21.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:20 smithi045 conmon[33655]: cluster 2021-07-31T03:51:19.567782+0000 2021-07-31T03:51:21.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:20 smithi045 conmon[33655]: mgr.x (mgr.24542) 5317 : cluster [DBG] pgmap v5731: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:51:21.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:20 smithi045 conmon[39718]: cluster 2021-07-31T03:51:19.567782+0000 mgr.x 2021-07-31T03:51:21.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:20 smithi045 conmon[39718]: (mgr.24542) 5317 : cluster [DBG] pgmap v5731: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:51:23.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:22 smithi154 conmon[34553]: cluster 2021-07-31T03:51:21.568200+0000 mgr.x (mgr.24542) 5318 : cluster [DBG] pgmap v5732: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:51:23.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:22 smithi045 conmon[33655]: cluster 2021-07-31T03:51:21.568200+0000 mgr.x (mgr.24542) 5318 : cluster [DBG] pgmap v5732: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:51:23.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:22 smithi045 conmon[39718]: cluster 2021-07-31T03:51:21.568200+0000 mgr.x (mgr.24542) 5318 : cluster [DBG] pgmap v5732: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:51:25.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:24 smithi154 conmon[34553]: cluster 2021-07-31T03:51:23.568565+0000 mgr.x (mgr.24542) 5319 : cluster [DBG] pgmap v5733: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:51:25.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:24 smithi045 conmon[33655]: cluster 2021-07-31T03:51:23.568565+0000 mgr.x (mgr.24542) 5319 2021-07-31T03:51:25.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:24 smithi045 conmon[33655]: : cluster [DBG] pgmap v5733: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:51:25.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:24 smithi045 conmon[39718]: cluster 2021-07-31T03:51:23 2021-07-31T03:51:25.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:24 smithi045 conmon[39718]: .568565+0000 mgr.x (mgr.24542) 5319 : cluster [DBG] pgmap v5733: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:51:27.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:26 smithi154 conmon[34553]: cluster 2021-07-31T03:51:25.569131+0000 mgr.x (mgr.24542) 5320 : cluster [DBG] pgmap v5734: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:51:27.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:26 smithi154 conmon[34553]: 2021-07-31T03:51:27.024 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:26 smithi045 conmon[33655]: cluster 2021-07-31T03:51:25.569131+0000 mgr.x (mgr.24542) 5320 2021-07-31T03:51:27.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:26 smithi045 conmon[33655]: : cluster [DBG] pgmap v5734: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:51:27.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:26 smithi045 conmon[39718]: cluster 2021-07-31T03:51:25.569131+0000 mgr.x (mgr 2021-07-31T03:51:27.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:26 smithi045 conmon[39718]: .24542) 5320 : cluster [DBG] pgmap v5734: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:51:29.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:28 smithi154 conmon[34553]: cluster 2021-07-31T03:51:27.569538+0000 mgr.x (mgr.24542) 5321 : cluster [DBG] pgmap v5735: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:51:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:28 smithi154 conmon[34553]: audit 2021-07-31T03:51:28.428368+0000 mon.b (mon.2) 695 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:51:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:28 smithi154 conmon[34553]: audit 2021-07-31T03:51: 2021-07-31T03:51:29.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:28 smithi154 conmon[34553]: 28.428672+0000 mon.b (mon.2) 696 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:51:29.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:28 smithi154 conmon[34553]: audit 2021-07-31T03:51:29.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:28 smithi154 conmon[34553]: 2021-07-31T03:51 2021-07-31T03:51:29.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:28 smithi154 conmon[34553]: :28.429186+0000 mon.a (mon. 2021-07-31T03:51:29.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:28 smithi154 conmon[34553]: 0) 3380 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:51:29.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:28 smithi154 conmon[34553]: 2021-07-31T03:51:29.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:28 smithi154 conmon[34553]: audit 2021 2021-07-31T03:51:29.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:28 smithi154 conmon[34553]: -07-31T03:51: 2021-07-31T03:51:29.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:28 smithi154 conmon[34553]: 28.429584+0000 mon.a (mon.0) 2021-07-31T03:51:29.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:28 smithi154 conmon[34553]: 3381 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:51:29.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:28 smithi045 conmon[33655]: cluster 2021-07-31T03:51:27.569538+0000 mgr.x (mgr.24542) 5321 : cluster [DBG] pgmap v5735: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:51:29.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:28 smithi045 conmon[33655]: 2021-07-31T03:51:29.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:28 smithi045 conmon[33655]: audit 2021-07-31T03:51:28.428368+0000 mon.b (mon.2) 695 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:51:29.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:28 smithi045 conmon[33655]: audit 2021-07-31T03:51:28.428672+0000 mon.b (mon.2) 696 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:51:29.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:28 smithi045 conmon[33655]: audit 2021-07-31T03:51:28.429186+0000 mon.a (mon.0) 3380 : audit [INF] 2021-07-31T03:51:29.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:28 smithi045 conmon[33655]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:51:29.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:28 smithi045 conmon[33655]: audit 2021-07-31T03:51:29.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:28 smithi045 conmon[33655]: 2021-07-31T03:51:28.429584+0000 mon.a (mon.0) 3381 : audit [INF] 2021-07-31T03:51:29.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:28 smithi045 conmon[33655]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:51:29.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:28 smithi045 conmon[39718]: cluster 2021-07-31T03:51:27.569538+0000 mgr.x ( 2021-07-31T03:51:29.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:28 smithi045 conmon[39718]: mgr.24542) 5321 : cluster [DBG] pgmap v5735: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:51:29.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:28 smithi045 conmon[39718]: audit 2021-07-31T03:51:28.428368+0000 mon.b (mon.2) 695 : 2021-07-31T03:51:29.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:28 smithi045 conmon[39718]: audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:51:29.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:28 smithi045 conmon[39718]: audit 2021-07-31T03:51:28.428672 2021-07-31T03:51:29.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:28 smithi045 conmon[39718]: +0000 mon.b (mon.2) 696 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:51:29.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:28 smithi045 conmon[39718]: audit 2021-07-31T03:51:29.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:28 smithi045 conmon[39718]: 2021-07-31T03:51:28.429186+0000 mon.a (mon.0) 3380 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:51:29.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:28 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T03:51:29.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:28 smithi045 conmon[39718]: 03:51:28.429584+0000 mon.a (mon.0) 3381 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:51:29.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:28 smithi045 conmon[39718]: 2021-07-31T03:51:31.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:30 smithi154 conmon[34553]: cluster 2021-07-31T03:51:29.570170+0000 mgr.x (mgr.24542) 5322 : 2021-07-31T03:51:31.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:30 smithi154 conmon[34553]: cluster [DBG] pgmap v5736: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:51:31.024 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:30 smithi045 conmon[33655]: cluster 2021-07-31T03:51:29.570170+0000 mgr.x (mgr 2021-07-31T03:51:31.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:30 smithi045 conmon[33655]: .24542) 5322 : cluster [DBG] pgmap v5736: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:51:31.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:30 smithi045 conmon[39718]: cluster 2021-07-31T03:51:29 2021-07-31T03:51:31.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:30 smithi045 conmon[39718]: .570170+0000 mgr.x (mgr.24542) 5322 : cluster [DBG] pgmap v5736: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:51:33.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:32 smithi154 conmon[34553]: cluster 2021-07-31T03:51:31.570644+0000 mgr.x (mgr.24542) 5323 : cluster [DBG] pgmap v5737: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:51:33.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:32 smithi045 conmon[39718]: cluster 2021-07-31T03:51:31.570644+0000 mgr.x (mgr.24542) 5323 : cluster [DBG] 2021-07-31T03:51:33.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:32 smithi045 conmon[39718]: pgmap v5737: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:51:33.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:32 smithi045 conmon[33655]: cluster 2021-07-31T03:51:31.570644 2021-07-31T03:51:33.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:32 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 5323 : cluster [DBG] pgmap v5737: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:51:35.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:34 smithi154 conmon[34553]: cluster 2021-07-31T03:51:33.571197+0000 mgr.x (mgr.24542) 5324 : cluster [DBG] pgmap v5738: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:51:35.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:34 smithi045 conmon[39718]: cluster 2021-07-31T03:51:33.571197+0000 mgr.x (mgr.24542) 5324 2021-07-31T03:51:35.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:34 smithi045 conmon[39718]: : cluster [DBG] pgmap v5738: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:51:35.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:34 smithi045 conmon[33655]: cluster 2021-07-31T03:51:33.571197+0000 mgr.x (mgr.24542) 2021-07-31T03:51:35.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:34 smithi045 conmon[33655]: 5324 : cluster [DBG] pgmap v5738: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:51:35.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:35 smithi045 conmon[39718]: audit 2021-07-31T03:51:35.496677+0000 2021-07-31T03:51:35.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:35 smithi045 conmon[39718]: mon.b (mon.2) 697 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:51:35.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:35 smithi045 conmon[39718]: audit 2021-07-31T03:51:35.498191+0000 mon.b (mon 2021-07-31T03:51:35.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:35 smithi045 conmon[39718]: .2) 698 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:51:35.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:35 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T03:51:35.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:35 smithi045 conmon[39718]: 31T03:51:35.499030+0000 mon.b (mon. 2021-07-31T03:51:35.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:35 smithi045 conmon[39718]: 2) 699 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:51:35.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:35 smithi045 conmon[33655]: audit 2021-07-31T03:51:35.496677+0000 2021-07-31T03:51:35.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:35 smithi045 conmon[33655]: mon.b (mon.2) 697 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:51:35.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:35 smithi045 conmon[33655]: audit 2021-07-31T03:51:35.498191+0000 mon.b (mon.2) 698 : audit [DBG] 2021-07-31T03:51:35.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:35 smithi045 conmon[33655]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:51:35.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:35 smithi045 conmon[33655]: audit 2021-07-31T03:51:35.499030+0000 2021-07-31T03:51:35.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:35 smithi045 conmon[33655]: mon.b (mon.2) 699 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:51:36.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[34553]: audit 2021-07-31T03:51:35.496677+0000 mon.b (mon. 2021-07-31T03:51:36.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[34553]: 2) 697 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:51:36.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[34553]: audit 2021-07-31T03:51:35.498191+0000 mon.b (mon.2) 698 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:51:36.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[34553]: audit 2021-07-31T03:51:35.499030+0000 mon.b (mon.2) 699 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:51:36.003 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: debug 2021-07-31T03:51:35.541+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:51:36.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:51:36.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:51:36.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: yield 2021-07-31T03:51:36.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:51:36.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:51:36.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:51:36.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:51:36.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:51:36.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:51:36.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:51:36.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:51:36.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:51:36.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: task.result() 2021-07-31T03:51:36.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:51:36.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: debug 2021-07-31T03:51:35.553+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:51:36.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:51:36.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:51:36.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: yield 2021-07-31T03:51:36.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:51:36.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:51:36.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:51:36.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:51:36.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:51:36.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:51:36.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:51:36.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:51:36.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:51:36.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: task.result() 2021-07-31T03:51:36.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:51:36.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: debug 2021-07-31T03:51:35.578+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:51:36.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:51:36.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:51:36.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: yield 2021-07-31T03:51:36.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:51:36.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:51:36.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:51:36.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:51:36.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:51:36.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:51:36.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:51:36.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:51:36.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:51:36.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: task.result() 2021-07-31T03:51:36.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:51:36.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: debug 2021-07-31T03:51:35.579+0000 7fcb97969700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:51:36.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:51:36.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:51:36.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: yield 2021-07-31T03:51:36.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:51:36.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:51:36.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:51:36.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:51:36.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:51:36.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:51:36.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:51:36.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:51:36.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:51:36.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: task.result() 2021-07-31T03:51:36.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:51:36.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: 2021-07-31T03:51:36.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:51:36.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: 2021-07-31T03:51:36.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:51:36.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:51:36.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:51:36.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:51:36.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:51:36.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:51:36.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:51:36.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:51:36.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:51:36.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:51:36.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:51:36.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:51:36.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:51:36.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:51:36.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:51:36.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:51:36.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:51:36.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:51:36.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: raise 2021-07-31T03:51:36.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:51:36.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:51:36.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:51:36.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:51:36.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:51:36.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: debug 2021-07-31T03:51:35.587+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:51:36.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:51:36.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:51:36.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: yield 2021-07-31T03:51:36.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:51:36.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:51:36.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:51:36.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:51:36.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:51:36.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:51:36.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:51:36.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:51:36.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:51:36.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: task.result() 2021-07-31T03:51:36.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:51:36.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: debug 2021-07-31T03:51:35.587+0000 7fcb9816a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:51:36.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:51:36.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:51:36.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: yield 2021-07-31T03:51:36.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:51:36.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:51:36.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:51:36.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:51:36.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:51:36.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:51:36.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:51:36.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:51:36.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:51:36.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: task.result() 2021-07-31T03:51:36.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:51:36.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: 2021-07-31T03:51:36.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:51:36.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: 2021-07-31T03:51:36.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:51:36.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:51:36.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:51:36.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:51:36.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:51:36.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:51:36.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:51:36.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:51:36.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:51:36.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:51:36.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:51:36.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:51:36.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:51:36.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:51:36.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:51:36.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:51:36.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:51:36.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:51:36.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: raise 2021-07-31T03:51:36.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:51:36.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:51:36.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:51:36.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:51:36.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:51:35 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:51:37.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: cephadm 2021-07-31T03:51:35.499807 2021-07-31T03:51:37.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 5325 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:51:37.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: cephadm 2021-07-31T03:51:37.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: 2021-07-31T03: 2021-07-31T03:51:37.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: 51:35.542726+0000 mgr.x 2021-07-31T03:51:37.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: (mgr.24542) 5326 : cephadm 2021-07-31T03:51:37.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: [ERR] 2021-07-31T03:51:37.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:51:37.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:51:37.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: yield 2021-07-31T03:51:37.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:51:37.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:51:37.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:51:37.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:51:37.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:51:37.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:51:37.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:51:37.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:51:37.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:51:37.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: task.result() 2021-07-31T03:51:37.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:51:37.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: cephadm 2021-07-31T03:51:35.543210+0000 mgr.x (mgr.24542) 2021-07-31T03:51:37.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: 5327 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:51:37.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: cephadm 2021 2021-07-31T03:51:37.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: -07-31T03:51:35. 2021-07-31T03:51:37.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: 554208+0000 mgr.x (mgr.24542 2021-07-31T03:51:37.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: ) 5328 : cephadm [ERR] 2021-07-31T03:51:37.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:51:37.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:51:37.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: yield 2021-07-31T03:51:37.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:51:37.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:51:37.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:51:37.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:51:37.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:51:37.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:51:37.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:51:37.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:51:37.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:51:37.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: task.result() 2021-07-31T03:51:37.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:51:37.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: 2021-07-31T03:51:37.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: cephadm 2021-07-31T03: 2021-07-31T03:51:37.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: 51:35.555065+0000 mgr.x (mgr. 2021-07-31T03:51:37.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: cephadm 2021-07-31T03:51:35.499807+0000 mgr.x (mgr.24542) 5325 : cephadm [WRN] 2021-07-31T03:51:37.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:51:37.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: cephadm 2021-07-31T03:51:35.542726+0000 mgr.x (mgr.24542) 5326 : cephadm [ERR] 2021-07-31T03:51:37.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:51:37.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:51:37.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: yield 2021-07-31T03:51:37.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:51:37.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:51:37.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:51:37.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:51:37.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:51:37.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:51:37.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:51:37.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:51:37.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:51:37.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: task.result() 2021-07-31T03:51:37.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:51:37.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: cephadm 2021-07-31T03:51:37.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: 2021-07-31T03:51:35.543210+0000 mgr.x (mgr.24542) 5327 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:51:37.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: cephadm 2021-07-31T03:51:35.554208+0000 mgr.x (mgr.24542) 5328 : cephadm [ERR] 2021-07-31T03:51:37.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:51:37.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:51:37.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: yield 2021-07-31T03:51:37.040 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:51:37.040 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:51:37.040 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:51:37.040 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:51:37.040 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:51:37.041 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:51:37.041 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:51:37.041 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:51:37.041 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:51:37.041 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: task.result() 2021-07-31T03:51:37.042 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:51:37.042 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: cephadm 2021-07-31T03:51:35.555065+0000 mgr.x (mgr.24542) 5329 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:51:37.042 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: cluster 2021-07-31T03:51:35.571639+0000 mgr.x (mgr.24542) 5330 : cluster [DBG] pgmap v5739: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:51:37.042 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: cephadm 2021-07-31T03:51:35 2021-07-31T03:51:37.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: 24542) 5329 : cephadm [INF] 2021-07-31T03:51:37.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:51:37.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: cluster 2021 2021-07-31T03:51:37.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: -07-31T03:51:35 2021-07-31T03:51:37.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: .571639+0000 mgr.x (mgr. 2021-07-31T03:51:37.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: 24542) 5330 : cluster [DBG] 2021-07-31T03:51:37.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: pgmap v5739: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:51:37.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: cephadm 2021 2021-07-31T03:51:37.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: -07-31T03:51:35. 2021-07-31T03:51:37.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: 579572+0000 mgr.x (mgr.24542 2021-07-31T03:51:37.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: ) 5331 : cephadm [ERR] 2021-07-31T03:51:37.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:51:37.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:51:37.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: yield 2021-07-31T03:51:37.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:51:37.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:51:37.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:51:37.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:51:37.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:51:37.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:51:37.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:51:37.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:51:37.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:51:37.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: task.result() 2021-07-31T03:51:37.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:51:37.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: 2021-07-31T03:51:37.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: cephadm 2021-07-31T03:51:35.580009+0000 2021-07-31T03:51:37.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: mgr.x (mgr.24542) 5332 : cephadm 2021-07-31T03:51:37.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:51:37.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:51:37.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:51:37.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: yield 2021-07-31T03:51:37.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:51:37.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:51:37.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:51:37.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:51:37.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:51:37.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:51:37.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:51:37.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:51:37.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:51:37.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: task.result() 2021-07-31T03:51:37.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:51:37.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: 2021-07-31T03:51:37.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:51:37.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: 2021-07-31T03:51:37.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:51:37.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:51:37.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:51:37.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:51:37.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:51:37.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:51:37.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:51:37.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:51:37.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:51:37.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:51:37.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:51:37.054 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:51:37.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:51:37.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:51:37.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:51:37.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:51:37.055 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:51:37.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:51:37.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: raise 2021-07-31T03:51:37.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:51:37.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:51:37.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:51:37.057 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:51:37.057 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:51:37.057 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: cephadm 2021-07-31T03:51:35.588382+0000 mgr.x (mgr.24542) 5333 : cephadm [ERR] 2021-07-31T03:51:37.057 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:51:37.057 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:51:37.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: yield 2021-07-31T03:51:37.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:51:37.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:51:37.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:51:37.058 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:51:37.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:51:37.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:51:37.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:51:37.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:51:37.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:51:37.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: task.result() 2021-07-31T03:51:37.060 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:51:37.060 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: .579572+0000 mgr.x (mgr.24542) 5331 : cephadm [ERR] 2021-07-31T03:51:37.060 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:51:37.061 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:51:37.061 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: yield 2021-07-31T03:51:37.061 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:51:37.061 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:51:37.061 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:51:37.062 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:51:37.062 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:51:37.062 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:51:37.062 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:51:37.062 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:51:37.063 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:51:37.063 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: task.result() 2021-07-31T03:51:37.063 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:51:37.063 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: cephadm 2021-07-31T03:51:35.580009+0000 mgr.x (mgr.24542) 5332 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:51:37.063 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:51:37.063 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:51:37.064 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: yield 2021-07-31T03:51:37.064 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:51:37.064 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:51:37.064 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:51:37.064 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:51:37.065 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:51:37.065 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:51:37.065 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:51:37.065 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:51:37.065 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:51:37.066 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: task.result() 2021-07-31T03:51:37.066 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:51:37.066 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: 2021-07-31T03:51:37.066 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:51:37.066 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: 2021-07-31T03:51:37.067 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:51:37.067 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:51:37.067 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:51:37.067 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:51:37.067 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:51:37.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:51:37.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:51:37.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:51:37.068 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: cephadm 2021-07-31T03:51:35.588798+0000 mgr.x (mgr.24542) 5334 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:51:37.069 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:51:37.069 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:51:37.069 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: yield 2021-07-31T03:51:37.069 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:51:37.069 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:51:37.070 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:51:37.070 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:51:37.070 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:51:37.070 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:51:37.071 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:51:37.071 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:51:37.071 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:51:37.071 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: task.result() 2021-07-31T03:51:37.071 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:51:37.071 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: 2021-07-31T03:51:37.072 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:51:37.072 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: 2021-07-31T03:51:37.072 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:51:37.072 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:51:37.072 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:51:37.073 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:51:37.073 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:51:37.073 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:51:37.073 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:51:37.073 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:51:37.074 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:51:37.074 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:51:37.074 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:51:37.074 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:51:37.074 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:51:37.075 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:51:37.075 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:51:37.075 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:51:37.075 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:51:37.075 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:51:37.076 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: raise 2021-07-31T03:51:37.076 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:51:37.076 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:51:37.076 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:51:37.076 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:51:37.077 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:51:37.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:51:37.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:51:37.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:51:37.078 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:51:37.078 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:51:37.078 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:51:37.078 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:51:37.078 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:51:37.079 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:51:37.079 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:51:37.079 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: raise 2021-07-31T03:51:37.079 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:51:37.080 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:51:37.080 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:51:37.080 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:51:37.080 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:51:37.080 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: cephadm 2021-07-31T03:51:35.588382+0000 mgr.x (mgr.24542) 5333 : cephadm [ERR] 2021-07-31T03:51:37.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:51:37.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:51:37.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: yield 2021-07-31T03:51:37.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:51:37.081 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:51:37.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:51:37.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:51:37.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:51:37.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:51:37.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:51:37.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:51:37.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:51:37.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: task.result() 2021-07-31T03:51:37.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:51:37.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: cephadm 2021-07-31T03:51:37.084 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: 2021-07-31T03:51:35.588798+0000 mgr.x (mgr.24542) 5334 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:51:37.084 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:51:37.084 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:51:37.084 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: yield 2021-07-31T03:51:37.084 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:51:37.085 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:51:37.085 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:51:37.085 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:51:37.085 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:51:37.085 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:51:37.086 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:51:37.086 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:51:37.086 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:51:37.086 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: task.result() 2021-07-31T03:51:37.086 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:51:37.087 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: 2021-07-31T03:51:37.087 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:51:37.087 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: 2021-07-31T03:51:37.087 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:51:37.087 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:51:37.088 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:51:37.088 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:51:37.088 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:51:37.088 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:51:37.088 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:51:37.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:51:37.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:51:37.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:51:37.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:51:37.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:51:37.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:51:37.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:51:37.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:51:37.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:51:37.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:51:37.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:51:37.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: raise 2021-07-31T03:51:37.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:51:37.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:51:37.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:51:37.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:51:37.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:36 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:51:37.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: cephadm 2021-07-31T03:51:35.499807+0000 mgr.x (mgr.24542) 5325 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:51:37.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: 2021-07-31T03:51:37.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: cephadm 2021-07-31T03:51:35.542726+0000 mgr.x (mgr.24542) 5326 : cephadm [ERR] 2021-07-31T03:51:37.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:51:37.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:51:37.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: yield 2021-07-31T03:51:37.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:51:37.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:51:37.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:51:37.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:51:37.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:51:37.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:51:37.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:51:37.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:51:37.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:51:37.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: task.result() 2021-07-31T03:51:37.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:51:37.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: cephadm 2021-07-31T03:51:35.543210+0000 mgr.x (mgr.24542) 5327 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:51:37.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: cephadm 2021-07-31T03:51:35.554208+0000 mgr.x (mgr.24542) 5328 : cephadm [ERR] 2021-07-31T03:51:37.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:51:37.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:51:37.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: yield 2021-07-31T03:51:37.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:51:37.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:51:37.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:51:37.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:51:37.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:51:37.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:51:37.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:51:37.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:51:37.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:51:37.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: task.result() 2021-07-31T03:51:37.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:51:37.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: cephadm 2021-07-31T03:51:35.555065+0000 mgr.x (mgr.24542) 5329 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:51:37.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: cluster 2021-07-31T03:51:35.571639+0000 mgr.x (mgr.24542) 5330 : cluster [DBG] pgmap v5739: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:51:37.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: cephadm 2021-07-31T03:51:35.579572+0000 mgr.x (mgr.24542) 5331 : cephadm [ERR] 2021-07-31T03:51:37.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:51:37.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:51:37.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: yield 2021-07-31T03:51:37.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:51:37.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:51:37.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:51:37.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:51:37.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:51:37.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:51:37.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:51:37.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:51:37.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:51:37.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: task.result() 2021-07-31T03:51:37.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:51:37.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: cephadm 2021-07-31T03:51:35.580009+0000 mgr.x (mgr.24542) 5332 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:51:37.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:51:37.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:51:37.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: yield 2021-07-31T03:51:37.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:51:37.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:51:37.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:51:37.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:51:37.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:51:37.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:51:37.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:51:37.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:51:37.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:51:37.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: task.result() 2021-07-31T03:51:37.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:51:37.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: 2021-07-31T03:51:37.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:51:37.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: 2021-07-31T03:51:37.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:51:37.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:51:37.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:51:37.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:51:37.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:51:37.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:51:37.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:51:37.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:51:37.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:51:37.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:51:37.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:51:37.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:51:37.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:51:37.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:51:37.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:51:37.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:51:37.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:51:37.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:51:37.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: raise 2021-07-31T03:51:37.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:51:37.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:51:37.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:51:37.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:51:37.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:51:37.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: cephadm 2021-07-31T03:51:35.588382+0000 mgr.x (mgr.24542) 5333 : cephadm [ERR] 2021-07-31T03:51:37.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:51:37.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:51:37.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: yield 2021-07-31T03:51:37.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:51:37.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:51:37.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:51:37.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:51:37.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:51:37.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:51:37.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:51:37.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:51:37.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:51:37.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: task.result() 2021-07-31T03:51:37.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:51:37.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: cephadm 2021-07-31T03:51:35.588798+0000 mgr.x (mgr.24542) 5334 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:51:37.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:51:37.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:51:37.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: yield 2021-07-31T03:51:37.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:51:37.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:51:37.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:51:37.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:51:37.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:51:37.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:51:37.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:51:37.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:51:37.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:51:37.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: task.result() 2021-07-31T03:51:37.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:51:37.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: 2021-07-31T03:51:37.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:51:37.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: 2021-07-31T03:51:37.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:51:37.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:51:37.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:51:37.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:51:37.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:51:37.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:51:37.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:51:37.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:51:37.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:51:37.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:51:37.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:51:37.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:51:37.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:51:37.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:51:37.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:51:37.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:51:37.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:51:37.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:51:37.285 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: raise 2021-07-31T03:51:37.285 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:51:37.285 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:51:37.285 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:51:37.285 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:51:37.286 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:36 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:51:39.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:38 smithi045 conmon[33655]: cluster 2021-07-31T03:51:37.572013+0000 mgr.x (mgr.24542) 5335 : cluster 2021-07-31T03:51:39.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:38 smithi045 conmon[33655]: [DBG] pgmap v5740: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:51:39.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:38 smithi045 conmon[39718]: cluster 2021-07-31T03:51:37.572013+0000 mgr.x (mgr.24542 2021-07-31T03:51:39.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:38 smithi045 conmon[39718]: ) 5335 : cluster [DBG] pgmap v5740: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:51:39.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:38 smithi154 conmon[34553]: cluster 2021-07-31T03:51:37.572013+0000 mgr.x (mgr.24542) 5335 : cluster [DBG] pgmap v5740: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:51:41.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:40 smithi045 conmon[39718]: cluster 2021-07-31T03:51:39.572567+0000 mgr.x (mgr 2021-07-31T03:51:41.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:40 smithi045 conmon[39718]: .24542) 5336 : cluster [DBG] pgmap v5741: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:51:41.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:40 smithi045 conmon[33655]: cluster 2021-07-31T03:51:39.572567+0000 mgr.x (mgr.24542) 5336 : cluster 2021-07-31T03:51:41.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:40 smithi045 conmon[33655]: [DBG] pgmap v5741: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:51:41.244 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:51:41.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:40 smithi154 conmon[34553]: cluster 2021-07-31T03:51:39.572567+0000 mgr.x (mgr.24542) 5336 : cluster [DBG] pgmap v5741: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:51:41.522 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:51:41.522 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 2h ago 2h - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:51:41.522 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 2h ago 2h 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:51:41.523 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 2h ago 2h 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:51:41.523 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 2h ago 2h - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:51:41.523 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:51:41.523 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 2h ago 2h 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:51:41.523 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:51:41.524 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 2h ago 2h - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:51:41.524 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 2h ago 2h 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:51:41.524 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:51:41.524 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:51:41.524 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:51:41.525 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:51:41.525 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 2h ago 2h 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:51:41.525 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 2h ago 2h 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:51:41.525 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 2h ago 2h 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:51:41.525 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 2h ago 2h 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:51:41.526 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 2h ago 2h 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:51:41.858 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:51:41.858 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:51:41.859 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:51:41.859 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:51:41.859 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:51:41.859 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:51:41.859 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:51:41.860 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:51:41.860 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:51:41.860 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:51:41.860 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:51:41.860 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:51:41.861 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:51:41.861 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:51:41.861 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:51:41.861 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:51:41.861 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:51:43.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:42 smithi045 conmon[39718]: audit 2021-07-31T03:51:41.232794+0000 mgr.x (mgr.24542) 5337 : audit [DBG] from='client.30325 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:51:43.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:42 smithi045 conmon[39718]: audit 2021-07-31T03:51:41.514855+0000 mgr.x (mgr.24542) 5338 : audit [DBG] from='client.20394 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:51:43.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:42 smithi045 conmon[39718]: cluster 2021-07- 2021-07-31T03:51:43.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:42 smithi045 conmon[39718]: 31T03:51:41.573050 2021-07-31T03:51:43.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:42 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 5339 : 2021-07-31T03:51:43.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:42 smithi045 conmon[39718]: cluster [DBG] pgmap v5742: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:51:43.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:42 smithi045 conmon[39718]: audit 2021-07-31T03:51:43.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:42 smithi045 conmon[39718]: 2021-07-31T03 2021-07-31T03:51:43.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:42 smithi045 conmon[39718]: :51:41.859482+0000 2021-07-31T03:51:43.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:42 smithi045 conmon[39718]: mon.c (mon.1) 231 : audit [DBG] from='client.? 172.21.15.45:0/920039448' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:51:43.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:42 smithi045 conmon[39718]: 2021-07-31T03:51:43.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:42 smithi045 conmon[33655]: audit 2021-07-31T03:51:41.232794 2021-07-31T03:51:43.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:42 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 5337 : audit [DBG] from='client.30325 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:51:43.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:42 smithi045 conmon[33655]: audit 2021-07-31T03:51:43.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:42 smithi045 conmon[33655]: 2021-07-31T03:51:41.514855+0000 mgr.x (mgr.24542) 5338 : audit 2021-07-31T03:51:43.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:42 smithi045 conmon[33655]: [DBG] from='client.20394 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:51:43.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:42 smithi045 conmon[33655]: cluster 2021 2021-07-31T03:51:43.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:42 smithi045 conmon[33655]: -07-31T03:51:41.573050+0000 mgr.x 2021-07-31T03:51:43.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:42 smithi045 conmon[33655]: (mgr.24542) 5339 : cluster [DBG] pgmap v5742: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:51:43.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:42 smithi045 conmon[33655]: audit 2021-07-31T03:51:43.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:42 smithi045 conmon[33655]: 2021-07-31T03:51 2021-07-31T03:51:43.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:42 smithi045 conmon[33655]: :41.859482+0000 mon.c (mon.1) 231 : audit [DBG] 2021-07-31T03:51:43.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:42 smithi045 conmon[33655]: from='client.? 172.21.15.45:0/920039448' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:51:43.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:42 smithi154 conmon[34553]: audit 2021-07-31T03:51:41.232794+0000 mgr.x (mgr.24542) 5337 : audit [DBG] from='client.30325 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:51:43.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:42 smithi154 conmon[34553]: audit 2021-07-31T03:51 2021-07-31T03:51:43.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:42 smithi154 conmon[34553]: :41.514855+0000 mgr.x (mgr.24542) 5338 : audit [DBG] from='client.20394 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:51:43.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:42 smithi154 conmon[34553]: cluster 2021-07-31T03:51:41.573050+0000 mgr.x (mgr.24542) 5339 : cluster [DBG] pgmap v5742: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:51:43.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:42 smithi154 conmon[34553]: audit 2021-07-31T03: 2021-07-31T03:51:43.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:42 smithi154 conmon[34553]: 51:41.859482+0000 mon.c (mon.1) 231 : audit [DBG] from='client.? 172.21.15.45:0/920039448' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:51:45.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:44 smithi045 conmon[39718]: cluster 2021-07-31T03:51:43.573521+0000 mgr.x (mgr.24542 2021-07-31T03:51:45.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:44 smithi045 conmon[39718]: ) 5340 : cluster [DBG] pgmap v5743: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:51:45.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:44 smithi045 conmon[33655]: cluster 2021-07-31T03:51:43.573521+0000 2021-07-31T03:51:45.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:44 smithi045 conmon[33655]: mgr.x (mgr.24542) 5340 : cluster [DBG] pgmap v5743: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:51:45.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:44 smithi154 conmon[34553]: cluster 2021-07-31T03:51:43.573521+0000 mgr.x (mgr.24542) 5340 : cluster [DBG] pgmap v5743: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:51:47.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:46 smithi045 conmon[33655]: cluster 2021-07-31T03:51:45.574110 2021-07-31T03:51:47.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:46 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 5341 : cluster [DBG] pgmap v5744: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:51:47.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:46 smithi045 conmon[39718]: cluster 2021-07-31T03:51:45.574110+0000 mgr.x (mgr 2021-07-31T03:51:47.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:46 smithi045 conmon[39718]: .24542) 5341 : cluster [DBG] pgmap v5744: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:51:47.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:46 smithi154 conmon[34553]: cluster 2021-07-31T03:51:45.574110+0000 mgr.x (mgr.24542) 5341 : cluster 2021-07-31T03:51:47.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:46 smithi154 conmon[34553]: [DBG] pgmap v5744: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:51:49.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:48 smithi154 conmon[34553]: cluster 2021-07-31T03:51:47.574490+0000 mgr.x (mgr.24542) 5342 : cluster [DBG] pgmap v5745: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:51:49.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:48 smithi045 conmon[39718]: cluster 2021-07-31T03:51:47.574490+0000 mgr.x (mgr.24542) 2021-07-31T03:51:49.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:48 smithi045 conmon[39718]: 5342 : cluster [DBG] pgmap v5745: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:51:49.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:48 smithi045 conmon[33655]: cluster 2021-07-31T03:51:47.574490+0000 mgr.x (mgr. 2021-07-31T03:51:49.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:48 smithi045 conmon[33655]: 24542) 5342 : cluster [DBG] pgmap v5745: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:51:51.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:50 smithi154 conmon[34553]: cluster 2021-07-31T03:51:49.574884+0000 mgr.x (mgr.24542) 5343 : cluster [DBG] pgmap v5746: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:51:51.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:50 smithi045 conmon[33655]: cluster 2021-07-31T03:51:49.574884+0000 mgr.x (mgr.24542) 5343 : cluster 2021-07-31T03:51:51.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:50 smithi045 conmon[33655]: [DBG] pgmap v5746: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:51:51.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:50 smithi045 conmon[39718]: cluster 2021-07-31T03: 2021-07-31T03:51:51.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:50 smithi045 conmon[39718]: 51:49.574884+0000 mgr.x (mgr.24542) 5343 : cluster [DBG] pgmap v5746: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:51:53.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:52 smithi154 conmon[34553]: cluster 2021-07-31T03:51:51.575425+0000 mgr.x (mgr.24542) 5344 : cluster [DBG] pgmap v5747: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:51:53.274 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:52 smithi045 conmon[33655]: cluster 2021-07-31T03:51:51.575425+0000 mgr.x (mgr.24542) 5344 2021-07-31T03:51:53.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:52 smithi045 conmon[33655]: : cluster [DBG] pgmap v5747: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:51:53.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:52 smithi045 conmon[39718]: cluster 2021-07-31T03:51:51.575425+0000 mgr.x (mgr.24542 2021-07-31T03:51:53.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:52 smithi045 conmon[39718]: ) 5344 : cluster [DBG] pgmap v5747: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:51:55.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:54 smithi154 conmon[34553]: cluster 2021-07-31T03:51:53.575911+0000 mgr.x (mgr.24542) 5345 : cluster [DBG] pgmap v5748: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:51:55.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:54 smithi045 conmon[39718]: cluster 2021-07-31T03:51:53.575911+0000 mgr.x (mgr. 2021-07-31T03:51:55.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:54 smithi045 conmon[39718]: 24542) 5345 : cluster [DBG] pgmap v5748: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:51:55.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:54 smithi045 conmon[33655]: cluster 2021-07-31T03:51:53.575911+0000 mgr.x (mgr.24542) 5345 : cluster [DBG] pgmap v5748: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:51:57.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:56 smithi154 conmon[34553]: cluster 2021-07-31T03:51:55.576506+0000 mgr.x (mgr.24542) 5346 : cluster [DBG] pgmap v5749: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:51:57.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:56 smithi154 conmon[34553]: 2021-07-31T03:51:57.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:56 smithi045 conmon[33655]: cluster 2021-07-31T03:51:55.576506+0000 mgr.x (mgr.24542) 5346 : cluster 2021-07-31T03:51:57.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:56 smithi045 conmon[33655]: [DBG] pgmap v5749: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:51:57.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:56 smithi045 conmon[39718]: cluster 2021-07-31T03:51:55.576506+0000 mgr.x (mgr.24542) 5346 : cluster [DBG] pgmap v5749: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:51:59.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:51:58 smithi154 conmon[34553]: cluster 2021-07-31T03:51:57.577089+0000 mgr.x (mgr.24542) 5347 : cluster [DBG] pgmap v5750: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:51:59.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:51:58 smithi045 conmon[39718]: cluster 2021-07-31T03:51:57.577089+0000 mgr.x (mgr.24542) 5347 : cluster [DBG] pgmap v5750: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:51:59.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:58 smithi045 conmon[33655]: cluster 2021-07-31T03:51:57.577089+0000 mgr.x (mgr. 2021-07-31T03:51:59.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:51:58 smithi045 conmon[33655]: 24542) 5347 : cluster [DBG] pgmap v5750: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:01.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:00 smithi154 conmon[34553]: cluster 2021-07-31T03:51:59.577507+0000 mgr.x (mgr.24542) 5348 : cluster [DBG] pgmap v5751: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:01.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:00 smithi045 conmon[33655]: cluster 2021-07-31T03:51:59.577507+0000 mgr.x (mgr.24542) 2021-07-31T03:52:01.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:00 smithi045 conmon[33655]: 5348 : cluster [DBG] pgmap v5751: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:01.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:00 smithi045 conmon[39718]: cluster 2021-07-31T03:51:59.577507+0000 mgr.x 2021-07-31T03:52:01.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:00 smithi045 conmon[39718]: (mgr.24542) 5348 : cluster [DBG] pgmap v5751: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:03.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:02 smithi154 conmon[34553]: cluster 2021-07-31T03:52:01.578026+0000 mgr.x (mgr.24542) 5349 : cluster [DBG] pgmap v5752: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:03.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:02 smithi045 conmon[33655]: cluster 2021-07-31T03:52:01.578026+0000 mgr.x (mgr.24542) 5349 : cluster [DBG] 2021-07-31T03:52:03.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:02 smithi045 conmon[33655]: pgmap v5752: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:03.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:02 smithi045 conmon[39718]: cluster 2021-07-31T03:52:01.578026+0000 mgr.x (mgr 2021-07-31T03:52:03.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:02 smithi045 conmon[39718]: .24542) 5349 : cluster [DBG] pgmap v5752: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:05.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:04 smithi154 conmon[34553]: cluster 2021-07-31T03:52:03.578494+0000 mgr.x (mgr.24542) 5350 : cluster [DBG] pgmap v5753: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:05.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:04 smithi045 conmon[33655]: cluster 2021-07-31T03:52:03.578494+0000 mgr.x (mgr.24542) 5350 : cluster [DBG] pgmap v5753: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:05.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:04 smithi045 conmon[39718]: cluster 2021-07-31T03:52:03.578494+0000 mgr.x (mgr. 2021-07-31T03:52:05.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:04 smithi045 conmon[39718]: 24542) 5350 : cluster [DBG] pgmap v5753: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:07.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:06 smithi154 conmon[34553]: cluster 2021-07-31T03:52:05.579102+0000 mgr.x (mgr.24542) 5351 : cluster [DBG] pgmap v5754: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:07.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:06 smithi045 conmon[39718]: cluster 2021-07-31T03:52:05.579102+0000 mgr.x (mgr. 2021-07-31T03:52:07.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:06 smithi045 conmon[39718]: 24542) 5351 : cluster [DBG] pgmap v5754: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:07.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:06 smithi045 conmon[33655]: cluster 2021-07-31T03:52:05.579102+0000 mgr.x (mgr.24542) 5351 : cluster [DBG] pgmap v5754: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:09.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:08 smithi154 conmon[34553]: cluster 2021-07-31T03:52:07.579474+0000 mgr.x (mgr.24542) 5352 : cluster [DBG] pgmap v5755: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:09.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:08 smithi045 conmon[39718]: cluster 2021-07-31T03:52:07.579474+0000 2021-07-31T03:52:09.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:08 smithi045 conmon[39718]: mgr.x (mgr.24542) 5352 : cluster [DBG] pgmap v5755: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:09.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:08 smithi045 conmon[33655]: cluster 2021-07-31T03:52:07.579474+0000 mgr.x (mgr.24542) 5352 : cluster 2021-07-31T03:52:09.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:08 smithi045 conmon[33655]: [DBG] pgmap v5755: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:11.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:10 smithi154 conmon[34553]: cluster 2021-07-31T03:52:09.579903+0000 mgr.x (mgr.24542) 5353 : cluster [DBG] pgmap v5756: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:11.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:10 smithi045 conmon[39718]: cluster 2021-07-31T03:52:09.579903+0000 mgr.x ( 2021-07-31T03:52:11.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:10 smithi045 conmon[39718]: mgr.24542) 5353 : cluster [DBG] pgmap v5756: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:11.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:10 smithi045 conmon[33655]: cluster 2021-07-31T03:52:09.579903+0000 mgr.x (mgr.24542) 5353 : cluster [DBG] pgmap v5756: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:12.172 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:52:12.449 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:52:12.449 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 2h ago 2h - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:52:12.449 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 2h ago 2h 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:52:12.450 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 2h ago 2h 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:52:12.450 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 2h ago 2h - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:52:12.450 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:52:12.451 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 2h ago 2h 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:52:12.451 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:52:12.451 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 2h ago 2h - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:52:12.452 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 2h ago 2h 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:52:12.452 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:52:12.452 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:52:12.453 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:52:12.453 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:52:12.453 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 2h ago 2h 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:52:12.454 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 2h ago 2h 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:52:12.454 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 2h ago 2h 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:52:12.454 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 2h ago 2h 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:52:12.455 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 2h ago 2h 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:52:12.786 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:52:12.786 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:52:12.786 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:52:12.787 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:52:12.787 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:52:12.787 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:52:12.788 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:52:12.788 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:52:12.788 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:52:12.789 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:52:12.789 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:52:12.789 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:52:12.790 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:52:12.790 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:52:12.790 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:52:12.790 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:52:12.791 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:52:13.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:12 smithi154 conmon[34553]: cluster 2021-07-31T03:52:11.580289+0000 mgr.x (mgr.24542) 5354 : 2021-07-31T03:52:13.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:12 smithi154 conmon[34553]: cluster [DBG] pgmap v5757: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:13.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:12 smithi154 conmon[34553]: audit 2021-07-31T03:52:12.160551+0000 mgr.x (mgr.24542) 5355 : audit [DBG] from='client.30343 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:52:13.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:12 smithi154 conmon[34553]: audit 2021-07-31T03:52:12.786245+0000 mon.a (mon.0) 3382 : audit 2021-07-31T03:52:13.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:12 smithi154 conmon[34553]: [DBG] from='client.? 172.21.15.45:0/3331044776' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:52:13.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:12 smithi045 conmon[33655]: cluster 2021-07-31T03:52:11.580289+0000 2021-07-31T03:52:13.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:12 smithi045 conmon[33655]: mgr.x (mgr.24542) 5354 : cluster [DBG] pgmap v5757: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:13.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:12 smithi045 conmon[33655]: audit 2021-07-31T03:52:12.160551+0000 mgr.x (mgr.24542) 5355 : audit [DBG] from='client.30343 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:52:13.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:12 smithi045 conmon[33655]: audit 2021-07-31T03:52:12.786245+0000 mon.a (mon.0) 3382 : 2021-07-31T03:52:13.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:12 smithi045 conmon[33655]: audit [DBG] from='client.? 172.21.15.45:0/3331044776' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:52:13.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:12 smithi045 conmon[39718]: cluster 2021-07-31T03:52:11. 2021-07-31T03:52:13.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:12 smithi045 conmon[39718]: 580289+0000 mgr.x (mgr.24542) 5354 : cluster [DBG] pgmap v5757: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:13.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:12 smithi045 conmon[39718]: audit 2021-07-31T03: 2021-07-31T03:52:13.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:12 smithi045 conmon[39718]: 52:12.160551+0000 mgr.x (mgr. 2021-07-31T03:52:13.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:12 smithi045 conmon[39718]: 24542) 5355 : audit [DBG] from='client.30343 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:52:13.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:12 smithi045 conmon[39718]: audit 2021 2021-07-31T03:52:13.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:12 smithi045 conmon[39718]: -07-31T03:52:12. 2021-07-31T03:52:13.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:12 smithi045 conmon[39718]: 786245+0000 mon.a (mon.0) 3382 : 2021-07-31T03:52:13.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:12 smithi045 conmon[39718]: audit [DBG] from='client.? 172.21.15.45:0/3331044776' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:52:14.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:13 smithi154 conmon[34553]: audit 2021-07-31T03:52:12.441457+0000 mgr.x (mgr.24542) 2021-07-31T03:52:14.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:13 smithi154 conmon[34553]: 5356 : audit [DBG] from='client.30349 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:52:14.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:13 smithi045 conmon[33655]: audit 2021-07-31T03:52:12.441457+0000 mgr.x 2021-07-31T03:52:14.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:13 smithi045 conmon[33655]: (mgr.24542) 5356 : audit [DBG] from='client.30349 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:52:14.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:13 smithi045 conmon[39718]: audit 2021-07-31T03:52:12.441457+0000 2021-07-31T03:52:14.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:13 smithi045 conmon[39718]: mgr.x (mgr.24542) 5356 : audit [DBG] from='client.30349 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:52:15.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:14 smithi154 conmon[34553]: cluster 2021-07-31T03:52:13.580651+0000 mgr.x (mgr.24542) 5357 : cluster [DBG] 2021-07-31T03:52:15.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:14 smithi154 conmon[34553]: pgmap v5758: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:15.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:14 smithi045 conmon[39718]: cluster 2021-07-31T03:52:13.580651+0000 mgr.x ( 2021-07-31T03:52:15.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:14 smithi045 conmon[39718]: mgr.24542) 5357 : cluster [DBG] pgmap v5758: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:15.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:14 smithi045 conmon[33655]: cluster 2021-07-31T03:52:13.580651+0000 mgr.x (mgr.24542) 5357 : cluster [DBG] 2021-07-31T03:52:15.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:14 smithi045 conmon[33655]: pgmap v5758: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:17.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:16 smithi154 conmon[34553]: cluster 2021-07-31T03:52:15.581227+0000 mgr.x (mgr.24542) 5358 : cluster 2021-07-31T03:52:17.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:16 smithi154 conmon[34553]: [DBG] pgmap v5759: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:17.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:16 smithi045 conmon[39718]: cluster 2021-07-31T03:52:15.581227+0000 mgr.x 2021-07-31T03:52:17.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:16 smithi045 conmon[39718]: (mgr.24542) 5358 : cluster [DBG] pgmap v5759: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:17.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:16 smithi045 conmon[33655]: cluster 2021-07-31T03:52:15.581227+0000 mgr.x (mgr.24542) 5358 : 2021-07-31T03:52:17.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:16 smithi045 conmon[33655]: cluster [DBG] pgmap v5759: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:19.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:18 smithi154 conmon[34553]: cluster 2021-07-31T03:52:17.581627+0000 mgr.x (mgr.24542) 5359 : cluster [DBG] 2021-07-31T03:52:19.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:18 smithi154 conmon[34553]: pgmap v5760: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:19.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:18 smithi045 conmon[33655]: cluster 2021-07-31T03:52:17.581627+0000 mgr.x ( 2021-07-31T03:52:19.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:18 smithi045 conmon[33655]: mgr.24542) 5359 : cluster [DBG] pgmap v5760: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:19.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:18 smithi045 conmon[39718]: cluster 2021-07-31T03:52:17.581627+0000 mgr.x (mgr. 2021-07-31T03:52:19.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:18 smithi045 conmon[39718]: 24542) 5359 : cluster [DBG] pgmap v5760: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:21.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:20 smithi154 conmon[34553]: cluster 2021-07-31T03:52:19.582229+0000 mgr.x (mgr.24542) 5360 : cluster [DBG] 2021-07-31T03:52:21.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:20 smithi154 conmon[34553]: pgmap v5761: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:21.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:20 smithi045 conmon[33655]: cluster 2021-07-31T03:52:19.582229+0000 mgr.x (mgr 2021-07-31T03:52:21.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:20 smithi045 conmon[33655]: .24542) 5360 : cluster [DBG] pgmap v5761: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:21.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:20 smithi045 conmon[39718]: cluster 2021-07-31T03:52:19.582229+0000 2021-07-31T03:52:21.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:20 smithi045 conmon[39718]: mgr.x (mgr.24542) 5360 : cluster [DBG] pgmap v5761: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:23.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:22 smithi154 conmon[34553]: cluster 2021-07-31T03:52:21.582629+0000 mgr.x (mgr.24542) 5361 2021-07-31T03:52:23.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:22 smithi154 conmon[34553]: : cluster [DBG] pgmap v5762: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:23.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:22 smithi045 conmon[39718]: cluster 2021-07-31T03:52:21.582629+0000 mgr.x 2021-07-31T03:52:23.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:22 smithi045 conmon[39718]: (mgr.24542) 5361 : cluster [DBG] pgmap v5762: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:23.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:22 smithi045 conmon[33655]: cluster 2021-07-31T03:52:21.582629+0000 mgr.x 2021-07-31T03:52:23.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:22 smithi045 conmon[33655]: (mgr.24542) 5361 : cluster [DBG] pgmap v5762: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:25.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:24 smithi154 conmon[34553]: cluster 2021-07-31T03:52:23.583029+0000 mgr.x (mgr.24542) 5362 : cluster [DBG] 2021-07-31T03:52:25.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:24 smithi154 conmon[34553]: pgmap v5763: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:25.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:24 smithi045 conmon[39718]: cluster 2021-07-31T03:52:23.583029+0000 mgr.x (mgr 2021-07-31T03:52:25.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:24 smithi045 conmon[39718]: .24542) 5362 : cluster [DBG] pgmap v5763: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:25.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:24 smithi045 conmon[33655]: cluster 2021-07-31T03:52:23.583029+0000 mgr.x (mgr.24542 2021-07-31T03:52:25.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:24 smithi045 conmon[33655]: ) 5362 : cluster [DBG] pgmap v5763: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:27.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:26 smithi154 conmon[34553]: cluster 2021-07-31T03:52:25.583612+0000 mgr.x (mgr.24542) 5363 : cluster [DBG] pgmap v5764: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:27.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:26 smithi045 conmon[39718]: cluster 2021-07-31T03:52:25.583612+0000 mgr.x (mgr 2021-07-31T03:52:27.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:26 smithi045 conmon[39718]: .24542) 5363 : cluster [DBG] pgmap v5764: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:27.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:26 smithi045 conmon[33655]: cluster 2021-07-31T03:52:25.583612+0000 mgr.x (mgr.24542) 5363 : cluster 2021-07-31T03:52:27.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:26 smithi045 conmon[33655]: [DBG] pgmap v5764: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:29.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:28 smithi154 conmon[34553]: cluster 2021-07-31T03:52:27.584054+0000 mgr.x (mgr.24542) 5364 : cluster [DBG] pgmap v5765: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:29.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:28 smithi154 conmon[34553]: audit 2021-07-31T03:52:28.431714+0000 mon.b (mon.2) 700 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:52:29.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:28 smithi154 conmon[34553]: audit 2021-07-31T03:52:28.432238+0000 mon.b (mon.2) 701 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:52:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:28 smithi154 conmon[34553]: audit 2021-07-31T03:52:28.432511+0000 mon.a ( 2021-07-31T03:52:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:28 smithi154 conmon[34553]: mon.0) 3383 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:52:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:28 smithi154 conmon[34553]: audit 2021 2021-07-31T03:52:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:28 smithi154 conmon[34553]: -07-31T03:52:28.432883+0000 2021-07-31T03:52:29.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:28 smithi154 conmon[34553]: mon.a (mon.0) 3384 : audit [INF] 2021-07-31T03:52:29.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:28 smithi154 conmon[34553]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:52:29.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:28 smithi045 conmon[39718]: cluster 2021-07-31T03:52: 2021-07-31T03:52:29.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:28 smithi045 conmon[39718]: 27.584054+0000 mgr.x (mgr.24542) 5364 : cluster [DBG] pgmap v5765: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:29.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:28 smithi045 conmon[39718]: audit 2021-07-31T03:52:29.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:28 smithi045 conmon[39718]: 2021-07-31T03:52 2021-07-31T03:52:29.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:28 smithi045 conmon[39718]: :28.431714+0000 mon.b ( 2021-07-31T03:52:29.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:28 smithi045 conmon[39718]: mon.2) 700 : audit 2021-07-31T03:52:29.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:28 smithi045 conmon[39718]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:52:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:28 smithi045 conmon[39718]: audit 2021 2021-07-31T03:52:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:28 smithi045 conmon[39718]: -07-31T03:52:28. 2021-07-31T03:52:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:28 smithi045 conmon[39718]: 432238+0000 mon.b (mon.2 2021-07-31T03:52:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:28 smithi045 conmon[39718]: ) 701 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:52:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:28 smithi045 conmon[39718]: 2021-07-31T03:52:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:28 smithi045 conmon[39718]: audit 2021- 2021-07-31T03:52:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:28 smithi045 conmon[39718]: 07-31T03:52:28. 2021-07-31T03:52:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:28 smithi045 conmon[39718]: 432511+0000 mon.a (mon.0 2021-07-31T03:52:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:28 smithi045 conmon[39718]: ) 3383 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:52:29.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:28 smithi045 conmon[39718]: 2021-07-31T03:52:29.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:28 smithi045 conmon[39718]: audit 2021 2021-07-31T03:52:29.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:28 smithi045 conmon[39718]: -07-31T03:52:28 2021-07-31T03:52:29.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:28 smithi045 conmon[39718]: .432883+0000 mon.a (mon 2021-07-31T03:52:29.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:28 smithi045 conmon[39718]: .0) 3384 : audit [INF] 2021-07-31T03:52:29.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:28 smithi045 conmon[39718]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:52:29.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:28 smithi045 conmon[33655]: cluster 2021-07-31T03:52:27.584054+0000 mgr.x (mgr.24542) 5364 : 2021-07-31T03:52:29.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:28 smithi045 conmon[33655]: cluster [DBG] pgmap v5765: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:29.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:28 smithi045 conmon[33655]: audit 2021-07-31T03:52:28.431714+0000 mon.b (mon.2) 700 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:52:29.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:28 smithi045 conmon[33655]: audit 2021-07-31T03:52: 2021-07-31T03:52:29.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:28 smithi045 conmon[33655]: 28.432238+0000 mon.b (mon.2) 701 : audit [INF] 2021-07-31T03:52:29.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:28 smithi045 conmon[33655]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:52:29.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:28 smithi045 conmon[33655]: audit 2021-07-31T03: 2021-07-31T03:52:29.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:28 smithi045 conmon[33655]: 52:28.432511+0000 mon.a (mon.0) 3383 : 2021-07-31T03:52:29.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:28 smithi045 conmon[33655]: audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:52:29.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:28 smithi045 conmon[33655]: audit 2021- 2021-07-31T03:52:29.283 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:28 smithi045 conmon[33655]: 07-31T03:52:28.432883+0000 mon.a (mon 2021-07-31T03:52:29.283 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:28 smithi045 conmon[33655]: .0) 3384 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:52:31.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:30 smithi154 conmon[34553]: cluster 2021-07-31T03:52:29.584625+0000 mgr.x (mgr.24542) 5365 : cluster [DBG] pgmap v5766: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:31.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:30 smithi045 conmon[39718]: cluster 2021-07-31T03:52:29.584625+0000 mgr.x (mgr.24542) 5365 : cluster [DBG] pgmap v5766: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:31.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:30 smithi045 conmon[33655]: cluster 2021-07-31T03:52:29.584625+0000 mgr.x (mgr.24542) 5365 : cluster 2021-07-31T03:52:31.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:30 smithi045 conmon[33655]: [DBG] pgmap v5766: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:33.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:32 smithi154 conmon[34553]: cluster 2021-07-31T03:52:31.585181+0000 mgr.x (mgr.24542) 5366 : cluster [DBG] pgmap v5767: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:33.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:32 smithi154 conmon[34553]: 2021-07-31T03:52:33.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:32 smithi045 conmon[33655]: cluster 2021-07-31T03:52:31.585181+0000 mgr.x (mgr.24542) 5366 : cluster [DBG] pgmap v5767: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:33.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:32 smithi045 conmon[39718]: cluster 2021-07-31T03:52:31.585181+0000 mgr.x ( 2021-07-31T03:52:33.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:32 smithi045 conmon[39718]: mgr.24542) 5366 : cluster [DBG] pgmap v5767: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:35.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:34 smithi154 conmon[34553]: cluster 2021-07-31T03:52:33.585593+0000 mgr.x (mgr.24542) 5367 : cluster [DBG] pgmap v5768: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:35.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:34 smithi154 conmon[34553]: 2021-07-31T03:52:35.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:34 smithi045 conmon[39718]: cluster 2021-07-31T03:52:33.585593+0000 mgr.x (mgr.24542) 5367 : cluster [DBG] 2021-07-31T03:52:35.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:34 smithi045 conmon[39718]: pgmap v5768: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:35.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:34 smithi045 conmon[33655]: cluster 2021-07-31T03:52:33.585593+0000 mgr.x (mgr 2021-07-31T03:52:35.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:34 smithi045 conmon[33655]: .24542) 5367 : cluster [DBG] pgmap v5768: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:35.869 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: debug 2021-07-31T03:52:35.634+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:52:35.869 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:52:35.869 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:52:35.870 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: yield 2021-07-31T03:52:35.870 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:52:35.870 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:52:35.871 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:52:35.871 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:52:35.871 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:52:35.872 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:52:35.872 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:52:35.872 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:52:35.872 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:52:35.873 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: task.result() 2021-07-31T03:52:35.873 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:52:35.873 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: debug 2021-07-31T03:52:35.646+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:52:35.874 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:52:35.874 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:52:35.874 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: yield 2021-07-31T03:52:35.874 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:52:35.875 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:52:35.875 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:52:35.875 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:52:35.876 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:52:35.876 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:52:35.876 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:52:35.876 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:52:35.877 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:52:35.877 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: task.result() 2021-07-31T03:52:35.877 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:52:35.878 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: debug 2021-07-31T03:52:35.671+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:52:35.878 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:52:35.878 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:52:35.878 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: yield 2021-07-31T03:52:35.879 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:52:35.879 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:52:35.879 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:52:35.880 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:52:35.880 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:52:35.880 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:52:35.880 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:52:35.881 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:52:35.881 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:52:35.881 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: task.result() 2021-07-31T03:52:35.882 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:52:35.882 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: debug 2021-07-31T03:52:35.672+0000 7fcb94162700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:52:35.882 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:52:35.882 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:52:35.883 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: yield 2021-07-31T03:52:35.883 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:52:35.883 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:52:35.883 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:52:35.884 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:52:35.884 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:52:35.884 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:52:35.885 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:52:35.885 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:52:35.885 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:52:35.885 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: task.result() 2021-07-31T03:52:35.886 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:52:35.886 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: 2021-07-31T03:52:35.886 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:52:35.886 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: 2021-07-31T03:52:35.887 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:52:35.887 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:52:35.887 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:52:35.888 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:52:35.888 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:52:35.888 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:52:35.888 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:52:35.889 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:52:35.889 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:52:35.889 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:52:35.889 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:52:35.890 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:52:35.890 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:52:35.890 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:52:35.891 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:52:35.891 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:52:35.891 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:52:35.891 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:52:35.892 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: raise 2021-07-31T03:52:35.892 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:52:35.892 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:52:35.892 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:52:35.893 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:52:35.893 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:52:35.893 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: debug 2021-07-31T03:52:35.680+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:52:35.893 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:52:35.894 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:52:35.894 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: yield 2021-07-31T03:52:35.894 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:52:35.894 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:52:35.895 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:52:35.895 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:52:35.895 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:52:35.895 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:52:35.896 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:52:35.896 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:52:35.896 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:52:35.896 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: task.result() 2021-07-31T03:52:35.897 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:52:35.897 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: debug 2021-07-31T03:52:35.681+0000 7fcb93961700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:52:35.897 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:52:35.898 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:52:35.898 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: yield 2021-07-31T03:52:35.898 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:52:35.898 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:52:35.899 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:52:35.899 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:52:35.899 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:52:35.899 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:52:35.900 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:52:35.900 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:52:35.900 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:52:35.900 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: task.result() 2021-07-31T03:52:35.901 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:52:35.901 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: 2021-07-31T03:52:35.901 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:52:35.901 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: 2021-07-31T03:52:35.902 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:52:35.902 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:52:35.902 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:52:35.902 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:52:35.903 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:52:35.903 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:52:35.903 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:52:35.903 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:52:35.904 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:52:35.904 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:52:35.904 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:52:35.904 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:52:35.905 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:52:35.905 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:52:35.905 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:52:35.905 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:52:35.906 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:52:35.906 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:52:35.906 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: raise 2021-07-31T03:52:35.906 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:52:35.907 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:52:35.907 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:52:35.907 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:52:35.907 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:52:36.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[34553]: audit 2021-07-31T03:52:35.589493+0000 mon.b (mon.2 2021-07-31T03:52:36.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[34553]: ) 702 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:52:36.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[34553]: audit 2021-07-31T03:52:35.590942+0000 mon.b (mon.2) 703 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:52:36.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[34553]: audit 2021-07-31T03:52:36.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:35 smithi154 conmon[34553]: 2021-07-31T03:52:35.591785+0000 mon.b (mon.2) 704 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:52:36.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:35 smithi045 conmon[39718]: audit 2021-07-31T03:52:35.589493+0000 mon.b (mon.2) 702 : audit 2021-07-31T03:52:36.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:35 smithi045 conmon[39718]: [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:52:36.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:35 smithi045 conmon[39718]: audit 2021-07-31T03:52:35.590942+0000 mon.b (mon.2) 703 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:52:36.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:35 smithi045 conmon[39718]: audit 2021-07-31T03:52:35.591785+0000 mon.b (mon.2) 704 : audit 2021-07-31T03:52:36.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:35 smithi045 conmon[39718]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:52:36.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:35 smithi045 conmon[33655]: audit 2021-07-31T03:52:35 2021-07-31T03:52:36.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:35 smithi045 conmon[33655]: .589493+0000 mon.b (mon.2) 702 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:52:36.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:35 smithi045 conmon[33655]: audit 2021-07-31T03:52:36.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:35 smithi045 conmon[33655]: 2021-07-31T03:52:35.590942+0000 mon.b ( 2021-07-31T03:52:36.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:35 smithi045 conmon[33655]: mon.2) 703 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:52:36.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:35 smithi045 conmon[33655]: audit 2021-07-31T 2021-07-31T03:52:36.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:35 smithi045 conmon[33655]: 03:52:35.591785+0000 mon.b (mon.2) 704 2021-07-31T03:52:36.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:35 smithi045 conmon[33655]: : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:52:37.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: cluster 2021-07-31T03:52:35.586132+0000 mgr.x (mgr.24542) 5368 : cluster [DBG] pgmap v5769: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:37.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: cephadm 2021-07-31T03:52:35.592550+0000 mgr.x (mgr.24542) 5369 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:52:37.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: cephadm 2021-07-31T03:52:35.635588+0000 mgr.x (mgr.24542) 5370 : cephadm [ERR] 2021-07-31T03:52:37.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:52:37.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:52:37.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: yield 2021-07-31T03:52:37.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:52:37.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:52:37.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:52:37.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:52:37.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:52:37.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:52:37.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:52:37.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:52:37.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:52:37.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: task.result() 2021-07-31T03:52:37.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:52:37.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: cephadm 2021-07-31T03:52:35.636029+0000 mgr.x (mgr.24542) 5371 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:52:37.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: cephadm 2021-07-31T03:52:35.647058+0000 mgr.x (mgr.24542) 5372 : cephadm [ERR] 2021-07-31T03:52:37.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:52:37.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:52:37.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: yield 2021-07-31T03:52:37.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:52:37.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:52:37.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:52:37.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:52:37.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:52:37.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:52:37.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:52:37.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:52:37.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:52:37.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: task.result() 2021-07-31T03:52:37.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:52:37.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: cephadm 2021-07-31T03:52:35.647870+0000 mgr.x (mgr.24542) 5373 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:52:37.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: cephadm 2021-07-31T03:52:35.672534+0000 mgr.x (mgr.24542) 5374 : cephadm [ERR] 2021-07-31T03:52:37.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:52:37.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:52:37.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: yield 2021-07-31T03:52:37.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:52:37.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:52:37.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:52:37.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:52:37.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:52:37.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:52:37.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:52:37.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:52:37.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:52:37.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: task.result() 2021-07-31T03:52:37.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:52:37.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: cephadm 2021-07-31T03:52:35.672993+0000 mgr.x (mgr.24542) 5375 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:52:37.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:52:37.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:52:37.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: yield 2021-07-31T03:52:37.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:52:37.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:52:37.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:52:37.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:52:37.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:52:37.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:52:37.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:52:37.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:52:37.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:52:37.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: task.result() 2021-07-31T03:52:37.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:52:37.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: 2021-07-31T03:52:37.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:52:37.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: 2021-07-31T03:52:37.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:52:37.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:52:37.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:52:37.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:52:37.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:52:37.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:52:37.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:52:37.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:52:37.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:52:37.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:52:37.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:52:37.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:52:37.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:52:37.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:52:37.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:52:37.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:52:37.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:52:37.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:52:37.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: raise 2021-07-31T03:52:37.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:52:37.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:52:37.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:52:37.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:52:37.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:52:37.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: cephadm 2021-07- 2021-07-31T03:52:37.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: 31T03:52:35.681618+0000 mgr.x (mgr.24542) 5376 : cephadm [ERR] 2021-07-31T03:52:37.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:52:37.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:52:37.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: yield 2021-07-31T03:52:37.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:52:37.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:52:37.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:52:37.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:52:37.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:52:37.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:52:37.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:52:37.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:52:37.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:52:37.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: task.result() 2021-07-31T03:52:37.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:52:37.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: cephadm 2021-07-31T03:52:35.682036+0000 mgr.x (mgr.24542) 5377 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:52:37.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:52:37.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:52:37.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: yield 2021-07-31T03:52:37.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:52:37.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:52:37.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:52:37.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:52:37.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:52:37.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:52:37.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:52:37.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:52:37.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:52:37.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: task.result() 2021-07-31T03:52:37.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:52:37.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: 2021-07-31T03:52:37.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:52:37.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: 2021-07-31T03:52:37.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:52:37.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:52:37.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:52:37.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:52:37.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:52:37.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:52:37.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:52:37.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:52:37.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:52:37.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:52:37.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:52:37.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:52:37.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:52:37.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:52:37.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:52:37.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:52:37.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:52:37.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:52:37.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: raise 2021-07-31T03:52:37.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:52:37.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:52:37.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:52:37.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:52:37.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:36 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:52:37.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: cluster 2021-07-31T03:52:35.586132+0000 mgr.x (mgr 2021-07-31T03:52:37.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: .24542) 5368 : cluster [DBG] pgmap v5769: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:37.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: cephadm 2021-07-31T03:52: 2021-07-31T03:52:37.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: 35.592550+0000 mgr.x (mgr.24542) 5369 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:52:37.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: cephadm 2021-07- 2021-07-31T03:52:37.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: 31T03:52:35.635588+0000 mgr.x ( 2021-07-31T03:52:37.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: mgr.24542) 5370 : cephadm [ERR] 2021-07-31T03:52:37.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:52:37.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:52:37.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: yield 2021-07-31T03:52:37.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:52:37.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:52:37.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:52:37.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:52:37.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:52:37.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:52:37.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:52:37.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:52:37.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:52:37.289 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: task.result() 2021-07-31T03:52:37.289 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:52:37.289 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: cephadm 2021-07-31T03:52:35.636029+0000 mgr.x (mgr.24542) 5371 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:52:37.289 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: cephadm 2021-07-31T03:52:35.647058+0000 2021-07-31T03:52:37.290 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: mgr.x (mgr.24542) 5372 : cephadm [ERR] 2021-07-31T03:52:37.290 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:52:37.290 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:52:37.290 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: yield 2021-07-31T03:52:37.290 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:52:37.291 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:52:37.291 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:52:37.291 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:52:37.291 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:52:37.291 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:52:37.292 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:52:37.292 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:52:37.292 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:52:37.292 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: task.result() 2021-07-31T03:52:37.292 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:52:37.293 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: cephadm 2021-07-31T03:52:35.647870+0000 mgr.x (mgr.24542) 5373 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:52:37.293 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: cephadm 2021-07-31T03:52:35.672534+0000 2021-07-31T03:52:37.293 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: mgr.x (mgr.24542) 5374 : cephadm [ERR] 2021-07-31T03:52:37.293 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:52:37.293 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:52:37.293 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: yield 2021-07-31T03:52:37.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:52:37.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:52:37.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:52:37.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:52:37.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:52:37.295 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:52:37.295 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:52:37.295 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:52:37.295 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:52:37.295 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: task.result() 2021-07-31T03:52:37.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:52:37.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: cephadm 2021-07-31T03:52:35.672993+0000 mgr.x (mgr.24542) 5375 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:52:37.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:52:37.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:52:37.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: yield 2021-07-31T03:52:37.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:52:37.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:52:37.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:52:37.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:52:37.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:52:37.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:52:37.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:52:37.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:52:37.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:52:37.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: task.result() 2021-07-31T03:52:37.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:52:37.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: 2021-07-31T03:52:37.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:52:37.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: 2021-07-31T03:52:37.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:52:37.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:52:37.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:52:37.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:52:37.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:52:37.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:52:37.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:52:37.301 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: cluster 2021-07-31T03:52:35.586132+0000 mgr.x (mgr.24542) 2021-07-31T03:52:37.301 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: 5368 : cluster [DBG] pgmap v5769: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:37.301 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: cephadm 2021-07-31T03:52:35.592550+0000 mgr.x (mgr.24542) 5369 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:52:37.302 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: cephadm 2021-07-31T03:52: 2021-07-31T03:52:37.302 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: 35.635588+0000 mgr.x (mgr.24542) 5370 : cephadm [ERR] 2021-07-31T03:52:37.302 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: 2021-07-31T03:52:37.302 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:52:37.303 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:52:37.303 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: yield 2021-07-31T03:52:37.303 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:52:37.303 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:52:37.304 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:52:37.304 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:52:37.304 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:52:37.304 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:52:37.304 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:52:37.305 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:52:37.305 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:52:37.305 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: task.result() 2021-07-31T03:52:37.305 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:52:37.306 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: cephadm 2021-07-31T03:52:35.636029+0000 mgr.x (mgr.24542) 5371 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:52:37.306 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: cephadm 2021-07-31T03:52:37.306 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: 2021-07-31T03:52:35.647058+0000 mgr.x (mgr. 2021-07-31T03:52:37.306 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: 24542) 5372 : cephadm [ERR] 2021-07-31T03:52:37.306 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:52:37.307 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:52:37.307 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: yield 2021-07-31T03:52:37.307 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:52:37.307 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:52:37.308 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:52:37.308 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:52:37.308 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:52:37.308 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:52:37.308 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:52:37.309 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:52:37.309 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:52:37.309 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: task.result() 2021-07-31T03:52:37.309 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:52:37.309 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: 2021-07-31T03:52:37.310 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: cephadm 2021-07-31T03:52:35.647870+0000 mgr.x (mgr.24542) 2021-07-31T03:52:37.310 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:52:37.311 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:52:37.311 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:52:37.311 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:52:37.311 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:52:37.311 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:52:37.312 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:52:37.312 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:52:37.312 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:52:37.312 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:52:37.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:52:37.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: raise 2021-07-31T03:52:37.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:52:37.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:52:37.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:52:37.314 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:52:37.314 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:52:37.314 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: cephadm 2021-07-31T03: 2021-07-31T03:52:37.314 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: 52:35.681618+0000 mgr.x (mgr.24542) 5376 : cephadm [ERR] 2021-07-31T03:52:37.314 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:52:37.315 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:52:37.315 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: yield 2021-07-31T03:52:37.315 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:52:37.315 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:52:37.315 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:52:37.316 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:52:37.316 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:52:37.316 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:52:37.316 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:52:37.317 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:52:37.317 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:52:37.317 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: task.result() 2021-07-31T03:52:37.317 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:52:37.317 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: cephadm 2021-07-31T03:52:35.682036+0000 mgr.x (mgr.24542) 5377 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:52:37.318 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:52:37.318 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:52:37.318 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: yield 2021-07-31T03:52:37.318 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:52:37.319 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:52:37.319 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: 5373 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:52:37.319 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: cephadm 2021-07 2021-07-31T03:52:37.319 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: -31T03:52:35.672534+0000 mgr.x (mgr. 2021-07-31T03:52:37.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: 24542) 5374 : cephadm [ERR] 2021-07-31T03:52:37.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: 2021-07-31T03:52:37.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:52:37.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:52:37.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: yield 2021-07-31T03:52:37.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:52:37.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:52:37.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:52:37.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:52:37.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:52:37.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:52:37.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:52:37.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:52:37.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:52:37.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: task.result() 2021-07-31T03:52:37.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:52:37.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: cephadm 2021-07-31T03:52:35.672993+0000 mgr.x (mgr.24542) 5375 2021-07-31T03:52:37.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:52:37.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:52:37.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:52:37.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: yield 2021-07-31T03:52:37.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:52:37.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:52:37.325 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:52:37.325 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:52:37.325 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:52:37.325 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:52:37.325 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:52:37.326 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:52:37.326 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:52:37.326 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: task.result() 2021-07-31T03:52:37.326 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:52:37.326 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: 2021-07-31T03:52:37.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:52:37.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: 2021-07-31T03:52:37.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:52:37.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:52:37.328 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:52:37.328 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:52:37.328 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:52:37.328 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:52:37.329 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:52:37.329 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:52:37.329 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:52:37.329 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:52:37.330 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:52:37.330 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:52:37.330 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: task.result() 2021-07-31T03:52:37.330 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:52:37.330 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: 2021-07-31T03:52:37.331 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:52:37.331 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: 2021-07-31T03:52:37.331 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:52:37.331 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:52:37.332 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:52:37.332 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:52:37.332 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:52:37.332 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:52:37.332 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:52:37.333 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:52:37.333 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:52:37.333 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:52:37.333 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:52:37.333 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:52:37.334 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:52:37.334 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:52:37.334 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:52:37.334 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:52:37.335 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:52:37.335 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:52:37.335 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: raise 2021-07-31T03:52:37.335 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:52:37.335 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:52:37.336 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:52:37.336 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:52:37.336 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:52:37.337 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:52:37.337 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:52:37.337 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:52:37.337 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:52:37.338 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:52:37.338 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:52:37.338 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:52:37.338 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:52:37.338 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:52:37.339 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:52:37.339 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:52:37.339 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:52:37.339 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:52:37.340 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: raise 2021-07-31T03:52:37.340 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:52:37.340 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:52:37.340 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:52:37.340 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:52:37.341 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:52:37.341 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: cephadm 2021-07-31T03:52:35.681618+0000 mgr.x (mgr.24542) 5376 : cephadm [ERR] 2021-07-31T03:52:37.341 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:52:37.341 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:52:37.342 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: yield 2021-07-31T03:52:37.342 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:52:37.342 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:52:37.342 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:52:37.342 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:52:37.343 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:52:37.343 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:52:37.343 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:52:37.343 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:52:37.344 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:52:37.344 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: task.result() 2021-07-31T03:52:37.344 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:52:37.344 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: cephadm 2021-07-31T03:52:35.682036+0000 mgr.x (mgr.24542) 5377 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:52:37.344 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:52:37.345 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:52:37.345 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: yield 2021-07-31T03:52:37.345 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:52:37.345 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:52:37.345 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:52:37.346 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:52:37.346 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:52:37.346 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:52:37.346 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:52:37.347 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:52:37.347 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:52:37.347 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: task.result() 2021-07-31T03:52:37.347 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:52:37.347 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: 2021-07-31T03:52:37.348 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:52:37.348 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: 2021-07-31T03:52:37.348 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:52:37.348 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:52:37.349 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:52:37.349 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:52:37.349 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:52:37.349 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:52:37.349 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:52:37.350 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:52:37.350 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:52:37.350 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:52:37.350 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:52:37.350 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:52:37.351 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:52:37.351 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:52:37.351 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:52:37.351 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:52:37.352 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:52:37.352 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:52:37.352 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: raise 2021-07-31T03:52:37.352 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:52:37.352 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:52:37.353 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:52:37.353 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:52:37.353 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:36 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:52:39.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:38 smithi154 conmon[34553]: cluster 2021-07-31T03:52:37.586625+0000 mgr.x (mgr.24542) 5378 : cluster [DBG] pgmap v5770: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:39.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:38 smithi045 conmon[33655]: cluster 2021-07-31T03:52:37.586625+0000 mgr.x (mgr.24542) 5378 : cluster [DBG] 2021-07-31T03:52:39.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:38 smithi045 conmon[33655]: pgmap v5770: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:39.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:38 smithi045 conmon[39718]: cluster 2021-07-31T03:52:37.586625+0000 mgr.x (mgr 2021-07-31T03:52:39.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:38 smithi045 conmon[39718]: .24542) 5378 : cluster [DBG] pgmap v5770: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:41.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:40 smithi154 conmon[34553]: cluster 2021-07-31T03:52:39.587069+0000 mgr.x (mgr.24542) 5379 : cluster [DBG] pgmap v5771: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:41.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:40 smithi045 conmon[33655]: cluster 2021-07-31T03:52:39.587069+0000 mgr.x (mgr.24542 2021-07-31T03:52:41.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:40 smithi045 conmon[33655]: ) 5379 : cluster [DBG] pgmap v5771: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:41.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:40 smithi045 conmon[39718]: cluster 2021-07-31T03:52:39.587069+0000 2021-07-31T03:52:41.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:40 smithi045 conmon[39718]: mgr.x (mgr.24542) 5379 : cluster [DBG] pgmap v5771: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:43.099 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:52:43.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:42 smithi154 conmon[34553]: cluster 2021-07-31T03:52:41.587432+0000 mgr.x (mgr.24542) 5380 : cluster [DBG] pgmap v5772: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:43.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:42 smithi045 conmon[39718]: cluster 2021-07-31T03:52:41.587432+0000 2021-07-31T03:52:43.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:42 smithi045 conmon[39718]: mgr.x (mgr.24542) 5380 : cluster [DBG] pgmap v5772: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:43.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:42 smithi045 conmon[33655]: cluster 2021-07-31T03:52:41.587432+0000 mgr.x (mgr. 2021-07-31T03:52:43.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:42 smithi045 conmon[33655]: 24542) 5380 : cluster [DBG] pgmap v5772: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:43.376 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:52:43.376 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 2h ago 2h - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:52:43.376 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 2h ago 2h 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:52:43.377 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 2h ago 2h 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:52:43.377 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 2h ago 2h - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:52:43.377 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:52:43.378 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 2h ago 2h 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:52:43.378 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:52:43.378 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 2h ago 2h - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:52:43.378 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 2h ago 2h 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:52:43.379 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:52:43.379 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:52:43.379 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:52:43.380 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:52:43.380 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 2h ago 2h 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:52:43.380 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 2h ago 2h 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:52:43.381 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 2h ago 2h 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:52:43.381 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 2h ago 2h 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:52:43.381 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 2h ago 2h 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:52:43.712 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:52:43.713 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:52:43.713 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:52:43.713 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:52:43.713 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:52:43.714 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:52:43.714 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:52:43.714 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:52:43.715 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:52:43.715 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:52:43.715 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:52:43.716 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:52:43.716 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:52:43.716 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:52:43.717 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:52:43.717 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:52:43.717 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:52:44.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:43 smithi154 conmon[34553]: audit 2021-07-31T03:52:43.087094+0000 mgr.x (mgr.24542) 5381 : audit [DBG] from='client.30361 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:52:44.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:43 smithi154 conmon[34553]: 2021-07-31T03:52:44.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:43 smithi154 conmon[34553]: audit 2021-07-31T03:52:43.713308+0000 mon.c (mon.1) 232 : audit [DBG] from='client.? 172.21.15.45:0/1991368551' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:52:44.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:43 smithi045 conmon[33655]: audit 2021-07-31T03:52:43.087094+0000 mgr.x (mgr.24542) 5381 : audit [DBG] from='client.30361 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:52:44.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:43 smithi045 conmon[33655]: audit 2021-07-31T03:52:43.713308+0000 mon.c (mon.1) 232 : audit [DBG] from='client.? 172.21.15.45:0/1991368551' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:52:44.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:43 smithi045 conmon[39718]: audit 2021-07-31T03:52:43.087094+0000 mgr.x (mgr 2021-07-31T03:52:44.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:43 smithi045 conmon[39718]: .24542) 5381 : audit [DBG] from='client.30361 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:52:44.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:43 smithi045 conmon[39718]: audit 2021-07-31T03:52:43.713308+0000 mon.c (mon.1) 232 : audit 2021-07-31T03:52:44.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:43 smithi045 conmon[39718]: [DBG] from='client.? 172.21.15.45:0/1991368551' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:52:45.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:44 smithi154 conmon[34553]: audit 2021-07-31T03:52:43.368751+0000 mgr.x (mgr.24542) 5382 : audit [DBG] from='client.30367 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:52:45.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:44 smithi154 conmon[34553]: cluster 2021-07-31T03:52:43.587973+0000 mgr.x (mgr.24542) 5383 : cluster [DBG] pgmap v5773: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:45.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:44 smithi045 conmon[39718]: audit 2021-07-31T03:52:43.368751+0000 mgr.x (mgr.24542) 5382 : audit [DBG] from='client.30367 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:52:45.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:44 smithi045 conmon[39718]: cluster 2021-07-31T03:52:43.587973+0000 mgr.x (mgr.24542) 5383 : cluster [DBG] pgmap v5773: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:45.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:44 smithi045 conmon[33655]: audit 2021-07-31T03:52:43.368751+0000 mgr.x (mgr.24542) 5382 2021-07-31T03:52:45.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:44 smithi045 conmon[33655]: : audit [DBG] from='client.30367 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:52:45.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:44 smithi045 conmon[33655]: cluster 2021-07-31T03:52:43.587973+0000 mgr.x (mgr.24542) 5383 : cluster [DBG] pgmap v5773: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:47.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:46 smithi154 conmon[34553]: cluster 2021-07-31T03:52:45.588551+0000 mgr.x (mgr.24542) 5384 : cluster [DBG] pgmap v5774: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:47.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:46 smithi045 conmon[39718]: cluster 2021-07-31T03:52:45.588551+0000 mgr.x (mgr.24542) 5384 : cluster [DBG] pgmap v5774: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:47.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:46 smithi045 conmon[33655]: cluster 2021-07-31T03:52:45.588551+0000 mgr.x (mgr.24542 2021-07-31T03:52:47.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:46 smithi045 conmon[33655]: ) 5384 : cluster [DBG] pgmap v5774: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:49.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:48 smithi154 conmon[34553]: cluster 2021-07-31T03:52:47.589089+0000 mgr.x (mgr.24542) 5385 : cluster [DBG] pgmap v5775: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:49.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:48 smithi045 conmon[39718]: cluster 2021-07-31T03:52:47. 2021-07-31T03:52:49.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:48 smithi045 conmon[39718]: 589089+0000 mgr.x (mgr.24542) 5385 : cluster [DBG] pgmap v5775: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:49.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:48 smithi045 conmon[33655]: cluster 2021-07-31T03:52:47.589089+0000 mgr.x (mgr.24542) 5385 : cluster 2021-07-31T03:52:49.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:48 smithi045 conmon[33655]: [DBG] pgmap v5775: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:51.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:50 smithi154 conmon[34553]: cluster 2021-07-31T03:52:49.589442+0000 mgr.x (mgr.24542) 5386 : cluster [DBG] pgmap v5776: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:51.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:50 smithi154 conmon[34553]: 2021-07-31T03:52:51.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:50 smithi045 conmon[39718]: cluster 2021-07-31T03:52:49. 2021-07-31T03:52:51.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:50 smithi045 conmon[39718]: 589442+0000 mgr.x (mgr.24542) 5386 : cluster [DBG] pgmap v5776: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:51.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:50 smithi045 conmon[33655]: cluster 2021-07-31T03:52:49.589442+0000 mgr.x (mgr.24542) 5386 : 2021-07-31T03:52:51.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:50 smithi045 conmon[33655]: cluster [DBG] pgmap v5776: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:53.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:52 smithi154 conmon[34553]: cluster 2021-07-31T03:52:51.589909+0000 mgr.x (mgr.24542) 5387 : cluster [DBG] pgmap v5777: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:53.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:52 smithi154 conmon[34553]: 2021-07-31T03:52:53.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:52 smithi045 conmon[39718]: cluster 2021-07-31T03:52:51.589909+0000 mgr.x (mgr.24542) 5387 : cluster [DBG] pgmap v5777: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:53.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:52 smithi045 conmon[33655]: cluster 2021-07-31T03:52:51.589909+0000 mgr.x (mgr.24542) 5387 : cluster [DBG] pgmap v5777: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:55.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:54 smithi154 conmon[34553]: cluster 2021-07-31T03:52:53.590421+0000 mgr.x (mgr.24542) 5388 : cluster [DBG] pgmap v5778: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:55.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:54 smithi045 conmon[39718]: cluster 2021-07-31T03:52:53.590421+0000 2021-07-31T03:52:55.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:54 smithi045 conmon[39718]: mgr.x (mgr.24542) 5388 : cluster [DBG] pgmap v5778: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:55.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:54 smithi045 conmon[33655]: cluster 2021-07-31T03:52:53.590421+0000 mgr.x (mgr.24542) 5388 : cluster [DBG] pgmap v5778: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:55.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:54 smithi045 conmon[33655]: 2021-07-31T03:52:57.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:56 smithi154 conmon[34553]: cluster 2021-07-31T03:52:55.591069+0000 mgr.x (mgr.24542) 5389 : cluster [DBG] pgmap v5779: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:57.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:56 smithi045 conmon[39718]: cluster 2021-07-31T03:52:55.591069+0000 mgr.x (mgr.24542) 5389 : cluster [DBG] pgmap v5779: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:57.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:56 smithi045 conmon[33655]: cluster 2021-07-31T03:52:55.591069 2021-07-31T03:52:57.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:56 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 5389 : cluster [DBG] pgmap v5779: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:59.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:52:58 smithi154 conmon[34553]: cluster 2021-07-31T03:52:57.591475+0000 mgr.x (mgr.24542) 5390 : cluster [DBG] pgmap v5780: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:59.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:58 smithi045 conmon[33655]: cluster 2021-07-31T03:52:57.591475+0000 2021-07-31T03:52:59.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:52:58 smithi045 conmon[33655]: mgr.x (mgr.24542) 5390 : cluster [DBG] pgmap v5780: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:52:59.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:58 smithi045 conmon[39718]: cluster 2021-07-31T03:52:57.591475+0000 mgr.x ( 2021-07-31T03:52:59.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:52:58 smithi045 conmon[39718]: mgr.24542) 5390 : cluster [DBG] pgmap v5780: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:01.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:00 smithi154 conmon[34553]: cluster 2021-07-31T03:52:59.592098+0000 mgr.x (mgr.24542) 5391 : cluster [DBG] pgmap v5781: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:01.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:00 smithi045 conmon[39718]: cluster 2021-07-31T03:52:59. 2021-07-31T03:53:01.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:00 smithi045 conmon[39718]: 592098+0000 mgr.x (mgr.24542) 5391 : cluster [DBG] pgmap v5781: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:01.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:00 smithi045 conmon[33655]: cluster 2021-07-31T03:52:59.592098+0000 mgr.x (mgr.24542) 5391 : cluster [DBG] pgmap v5781: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:03.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:02 smithi154 conmon[34553]: cluster 2021-07-31T03:53:01.592559+0000 mgr.x (mgr.24542) 5392 : cluster [DBG] pgmap v5782: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:03.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:02 smithi045 conmon[39718]: cluster 2021-07-31T03:53:01.592559+0000 mgr.x (mgr. 2021-07-31T03:53:03.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:02 smithi045 conmon[39718]: 24542) 5392 : cluster [DBG] pgmap v5782: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:03.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:02 smithi045 conmon[33655]: cluster 2021-07-31T03:53:01.592559+0000 mgr.x (mgr.24542 2021-07-31T03:53:03.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:02 smithi045 conmon[33655]: ) 5392 : cluster [DBG] pgmap v5782: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:05.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:04 smithi154 conmon[34553]: cluster 2021-07-31T03:53:03.593009+0000 mgr.x (mgr.24542) 5393 : cluster [DBG] pgmap v5783: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:05.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:04 smithi045 conmon[39718]: cluster 2021-07-31T03:53:03.593009+0000 mgr.x (mgr. 2021-07-31T03:53:05.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:04 smithi045 conmon[39718]: 24542) 5393 : cluster [DBG] pgmap v5783: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:05.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:04 smithi045 conmon[33655]: cluster 2021-07-31T03:53:03.593009+0000 mgr.x (mgr.24542) 5393 : cluster [DBG] pgmap v5783: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:07.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:06 smithi154 conmon[34553]: cluster 2021-07-31T03:53:05.593513+0000 mgr.x (mgr.24542) 5394 : cluster [DBG] pgmap v5784: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:07.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:06 smithi045 conmon[39718]: cluster 2021-07-31T03:53:05.593513+0000 mgr.x (mgr 2021-07-31T03:53:07.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:06 smithi045 conmon[39718]: .24542) 5394 : cluster [DBG] pgmap v5784: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:07.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:06 smithi045 conmon[33655]: cluster 2021-07-31T03:53:05.593513 2021-07-31T03:53:07.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:06 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 5394 : cluster [DBG] pgmap v5784: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:09.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:08 smithi154 conmon[34553]: cluster 2021-07-31T03:53:07.594059+0000 mgr.x (mgr.24542) 5395 : cluster [DBG] pgmap v5785: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:09.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:08 smithi045 conmon[39718]: cluster 2021-07-31T03:53:07.594059+0000 mgr.x (mgr.24542) 5395 : cluster [DBG] pgmap v5785: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:09.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:08 smithi045 conmon[33655]: cluster 2021-07-31T03:53:07.594059+0000 mgr.x (mgr.24542) 5395 : cluster 2021-07-31T03:53:09.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:08 smithi045 conmon[33655]: [DBG] pgmap v5785: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:11.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:10 smithi154 conmon[34553]: cluster 2021-07-31T03:53:09.594491+0000 mgr.x (mgr.24542) 5396 : cluster [DBG] pgmap v5786: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:11.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:10 smithi045 conmon[39718]: cluster 2021-07-31T03:53: 2021-07-31T03:53:11.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:10 smithi045 conmon[39718]: 09.594491+0000 mgr.x (mgr.24542) 5396 : cluster [DBG] pgmap v5786: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:11.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:10 smithi045 conmon[33655]: cluster 2021-07-31T03:53:09.594491+0000 mgr.x (mgr.24542) 2021-07-31T03:53:11.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:10 smithi045 conmon[33655]: 5396 : cluster [DBG] pgmap v5786: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:13.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:12 smithi154 conmon[34553]: cluster 2021-07-31T03:53:11.594887+0000 mgr.x (mgr.24542) 5397 : 2021-07-31T03:53:13.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:12 smithi154 conmon[34553]: cluster [DBG] pgmap v5787: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:13.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:12 smithi045 conmon[39718]: cluster 2021-07-31T03:53:11.594887+0000 mgr.x 2021-07-31T03:53:13.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:12 smithi045 conmon[39718]: (mgr.24542) 5397 : cluster [DBG] pgmap v5787: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:13.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:12 smithi045 conmon[33655]: cluster 2021-07-31T03:53:11.594887+0000 mgr.x (mgr.24542) 2021-07-31T03:53:13.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:12 smithi045 conmon[33655]: 5397 : cluster [DBG] pgmap v5787: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:14.028 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:53:14.303 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:53:14.304 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 2h ago 2h - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:53:14.304 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 2h ago 2h 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:53:14.304 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 2h ago 2h 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:53:14.304 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 2h ago 2h - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:53:14.304 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:53:14.305 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 2h ago 2h 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:53:14.305 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:53:14.305 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 2h ago 2h - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:53:14.305 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 2h ago 2h 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:53:14.306 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:53:14.306 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:53:14.306 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:53:14.306 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:53:14.306 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 2h ago 2h 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:53:14.307 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 2h ago 2h 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:53:14.307 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 2h ago 2h 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:53:14.307 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 2h ago 2h 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:53:14.307 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 2h ago 2h 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:53:14.637 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:53:14.637 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:53:14.637 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:53:14.638 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:53:14.638 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:53:14.638 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:53:14.638 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:53:14.639 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:53:14.639 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:53:14.639 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:53:14.639 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:53:14.640 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:53:14.640 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:53:14.640 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:53:14.640 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:53:14.641 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:53:14.641 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:53:15.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:14 smithi154 conmon[34553]: cluster 2021-07-31T03:53:13.595309+0000 mgr.x (mgr.24542) 5398 : cluster [DBG] pgmap v5788: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:15.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:14 smithi154 conmon[34553]: audit 2021-07-31T03:53:14.017209+0000 mgr.x (mgr.24542) 5399 : audit [DBG] from='client.30379 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:53:15.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:14 smithi154 conmon[34553]: audit 2021-07-31T03:53:14.638092+0000 mon.c (mon.1) 233 : audit [DBG] from='client.? 172.21.15.45:0/1413651698' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:53:15.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:14 smithi045 conmon[39718]: cluster 2021-07-31T03:53:13.595309+0000 2021-07-31T03:53:15.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:14 smithi045 conmon[39718]: mgr.x (mgr.24542) 5398 : cluster [DBG] pgmap v5788: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:15.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:14 smithi045 conmon[39718]: audit 2021- 2021-07-31T03:53:15.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:14 smithi045 conmon[39718]: 07-31T03:53:14.017209+0000 mgr.x (mgr. 2021-07-31T03:53:15.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:14 smithi045 conmon[39718]: 24542) 5399 : audit [DBG] from='client.30379 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:53:15.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:14 smithi045 conmon[39718]: audit 2021-07-31T03: 2021-07-31T03:53:15.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:14 smithi045 conmon[39718]: 53:14.638092+0000 mon.c (mon.1) 233 : audit [DBG] from='client.? 172.21.15.45:0/1413651698' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:53:15.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:14 smithi045 conmon[33655]: cluster 2021-07-31T03:53:13. 2021-07-31T03:53:15.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:14 smithi045 conmon[33655]: 595309+0000 mgr.x (mgr.24542) 5398 : cluster [DBG] pgmap v5788: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:15.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:14 smithi045 conmon[33655]: audit 2021-07-31T03:53:15.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:14 smithi045 conmon[33655]: 2021-07-31T03:53:14.017209+0000 2021-07-31T03:53:15.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:14 smithi045 conmon[33655]: mgr.x (mgr.24542) 5399 : audit 2021-07-31T03:53:15.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:14 smithi045 conmon[33655]: [DBG] from='client.30379 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:53:15.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:14 smithi045 conmon[33655]: audit 2021-07-31T03:53: 2021-07-31T03:53:15.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:14 smithi045 conmon[33655]: 14.638092+0000 mon.c (mon.1) 233 : audit [DBG] from='client.? 172.21.15.45:0/1413651698' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:53:16.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:15 smithi154 conmon[34553]: audit 2021-07-31T03:53:14.296653+0000 mgr.x (mgr.24542) 5400 : audit [DBG] from='client.30385 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:53:16.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:15 smithi154 conmon[34553]: 2021-07-31T03:53:16.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:15 smithi045 conmon[33655]: audit 2021-07-31T03:53:14.296653+0000 mgr.x 2021-07-31T03:53:16.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:15 smithi045 conmon[33655]: (mgr.24542) 5400 : audit [DBG] from='client.30385 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:53:16.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:15 smithi045 conmon[39718]: audit 2021-07-31T03:53:14.296653+0000 mgr.x ( 2021-07-31T03:53:16.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:15 smithi045 conmon[39718]: mgr.24542) 5400 : audit [DBG] from='client.30385 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:53:17.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:16 smithi154 conmon[34553]: cluster 2021-07-31T03:53:15.595836+0000 mgr.x (mgr.24542) 5401 : cluster [DBG] pgmap v5789: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:17.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:16 smithi045 conmon[39718]: cluster 2021-07-31T03:53:15.595836+0000 mgr.x (mgr.24542) 2021-07-31T03:53:17.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:16 smithi045 conmon[39718]: 5401 : cluster [DBG] pgmap v5789: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:17.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:16 smithi045 conmon[33655]: cluster 2021-07-31T03:53:15.595836+0000 mgr.x (mgr.24542) 5401 2021-07-31T03:53:17.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:16 smithi045 conmon[33655]: : cluster [DBG] pgmap v5789: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:19.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:18 smithi154 conmon[34553]: cluster 2021-07-31T03:53:17.596330+0000 mgr.x (mgr.24542) 5402 : cluster [DBG] pgmap v5790: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:19.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:18 smithi045 conmon[33655]: cluster 2021-07-31T03:53:17.596330+0000 mgr.x (mgr.24542) 5402 : cluster 2021-07-31T03:53:19.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:18 smithi045 conmon[33655]: [DBG] pgmap v5790: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:19.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:18 smithi045 conmon[39718]: cluster 2021-07-31T03:53:17.596330+0000 mgr.x (mgr.24542) 5402 : 2021-07-31T03:53:19.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:18 smithi045 conmon[39718]: cluster [DBG] pgmap v5790: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:21.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:20 smithi154 conmon[34553]: cluster 2021-07-31T03:53:19.596887+0000 mgr.x (mgr.24542) 5403 : cluster [DBG] pgmap v5791: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:21.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:20 smithi045 conmon[39718]: cluster 2021-07-31T03:53:19.596887+0000 mgr.x (mgr. 2021-07-31T03:53:21.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:20 smithi045 conmon[39718]: 24542) 5403 : cluster [DBG] pgmap v5791: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:21.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:20 smithi045 conmon[33655]: cluster 2021-07-31T03:53:19.596887+0000 mgr.x (mgr.24542) 5403 : cluster [DBG] pgmap v5791: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:21.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:20 smithi045 conmon[33655]: 2021-07-31T03:53:23.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:22 smithi154 conmon[34553]: cluster 2021-07-31T03:53:21.597302+0000 mgr.x (mgr.24542) 5404 : cluster [DBG] pgmap v5792: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:23.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:22 smithi045 conmon[33655]: cluster 2021-07-31T03:53:21.597302+0000 mgr.x (mgr.24542) 5404 2021-07-31T03:53:23.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:22 smithi045 conmon[33655]: : cluster [DBG] pgmap v5792: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:23.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:22 smithi045 conmon[39718]: cluster 2021-07-31T03:53:21.597302+0000 2021-07-31T03:53:23.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:22 smithi045 conmon[39718]: mgr.x (mgr.24542) 5404 : cluster [DBG] pgmap v5792: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:25.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:24 smithi154 conmon[34553]: cluster 2021-07-31T03:53:23.597809+0000 mgr.x (mgr.24542) 5405 : cluster [DBG] pgmap v5793: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:25.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:24 smithi045 conmon[33655]: cluster 2021-07-31T03:53:23.597809+0000 mgr.x (mgr.24542) 2021-07-31T03:53:25.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:24 smithi045 conmon[33655]: 5405 : cluster [DBG] pgmap v5793: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:25.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:24 smithi045 conmon[39718]: cluster 2021-07-31T03:53:23.597809+0000 mgr.x (mgr 2021-07-31T03:53:25.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:24 smithi045 conmon[39718]: .24542) 5405 : cluster [DBG] pgmap v5793: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:27.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:26 smithi154 conmon[34553]: cluster 2021-07-31T03:53:25.598439+0000 mgr.x (mgr.24542) 5406 : cluster [DBG] pgmap v5794: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:27.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:26 smithi045 conmon[33655]: cluster 2021-07-31T03:53:25.598439+0000 mgr.x (mgr.24542) 2021-07-31T03:53:27.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:26 smithi045 conmon[33655]: 5406 : cluster [DBG] pgmap v5794: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:27.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:26 smithi045 conmon[39718]: cluster 2021-07-31T03:53:25.598439+0000 mgr.x ( 2021-07-31T03:53:27.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:26 smithi045 conmon[39718]: mgr.24542) 5406 : cluster [DBG] pgmap v5794: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:29.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:28 smithi154 conmon[34553]: cluster 2021-07-31T03:53:27.598874+0000 mgr.x (mgr.24542) 5407 : cluster [DBG] pgmap v5795: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:29.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:28 smithi154 conmon[34553]: audit 2021 2021-07-31T03:53:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:28 smithi154 conmon[34553]: -07-31T03:53:28.435167+0000 mon.b (mon.2) 705 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:53:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:28 smithi154 conmon[34553]: audit 2021-07-31T03:53:28.435482+0000 mon.b (mon.2) 706 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:53:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:28 smithi154 conmon[34553]: audit 2021- 2021-07-31T03:53:29.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:28 smithi154 conmon[34553]: 07-31T03:53:28.436194+0000 mon.a (mon.0) 3385 : audit [INF] 2021-07-31T03:53:29.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:28 smithi154 conmon[34553]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:53:29.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:28 smithi154 conmon[34553]: audit 2021-07-31T 2021-07-31T03:53:29.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:28 smithi154 conmon[34553]: 03:53:28.436656+0000 mon.a (mon.0 2021-07-31T03:53:29.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:28 smithi154 conmon[34553]: ) 3386 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:53:29.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:28 smithi045 conmon[33655]: cluster 2021-07-31T03:53:27.598874+0000 mgr.x (mgr.24542) 5407 : cluster [DBG] pgmap v5795: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:29.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:28 smithi045 conmon[33655]: audit 2021-07-31T03:53:28.435167+0000 mon.b (mon.2) 705 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:53:29.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:28 smithi045 conmon[33655]: audit 2021-07-31T03:53:28.435482+0000 mon.b (mon.2) 706 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:53:29.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:28 smithi045 conmon[33655]: audit 2021-07-31T03:53:29.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:28 smithi045 conmon[33655]: 2021-07-31T03:53:28.436194+0000 mon.a (mon.0) 3385 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:53:29.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:28 smithi045 conmon[33655]: audit 2021- 2021-07-31T03:53:29.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:28 smithi045 conmon[33655]: 07-31T03:53:28 2021-07-31T03:53:29.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:28 smithi045 conmon[33655]: .436656+0000 mon.a (mon 2021-07-31T03:53:29.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:28 smithi045 conmon[33655]: .0) 3386 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:53:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:28 smithi045 conmon[39718]: cluster 2021-07-31T03:53:27 2021-07-31T03:53:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:28 smithi045 conmon[39718]: .598874+0000 mgr.x (mgr.24542) 5407 : cluster [DBG] pgmap v5795: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:28 smithi045 conmon[39718]: audit 2021-07 2021-07-31T03:53:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:28 smithi045 conmon[39718]: -31T03:53:28.435167+0000 2021-07-31T03:53:29.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:28 smithi045 conmon[39718]: mon.b (mon.2) 2021-07-31T03:53:29.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:28 smithi045 conmon[39718]: 705 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:53:29.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:28 smithi045 conmon[39718]: audit 2021- 2021-07-31T03:53:29.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:28 smithi045 conmon[39718]: 07-31T03:53:28. 2021-07-31T03:53:29.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:28 smithi045 conmon[39718]: 435482+0000 mon.b (mon.2 2021-07-31T03:53:29.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:28 smithi045 conmon[39718]: ) 706 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:53:29.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:28 smithi045 conmon[39718]: audit 2021 2021-07-31T03:53:29.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:28 smithi045 conmon[39718]: -07-31T03:53:28 2021-07-31T03:53:29.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:28 smithi045 conmon[39718]: .436194+0000 mon.a (mon. 2021-07-31T03:53:29.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:28 smithi045 conmon[39718]: 0) 3385 : audit [INF] 2021-07-31T03:53:29.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:28 smithi045 conmon[39718]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:53:29.282 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:28 smithi045 conmon[39718]: audit 2021- 2021-07-31T03:53:29.282 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:28 smithi045 conmon[39718]: 07-31T03:53:28. 2021-07-31T03:53:29.282 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:28 smithi045 conmon[39718]: 436656+0000 mon.a (mon.0) 2021-07-31T03:53:29.282 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:28 smithi045 conmon[39718]: 3386 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:53:31.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:30 smithi154 conmon[34553]: cluster 2021-07-31T03:53:29.599470+0000 mgr.x (mgr.24542) 5408 : cluster [DBG] pgmap v5796: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:31.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:30 smithi045 conmon[39718]: cluster 2021-07-31T03:53:29.599470+0000 mgr.x (mgr.24542) 5408 : cluster [DBG] pgmap v5796: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:31.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:30 smithi045 conmon[33655]: cluster 2021-07-31T03:53:29.599470+0000 mgr.x (mgr. 2021-07-31T03:53:31.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:30 smithi045 conmon[33655]: 24542) 5408 : cluster [DBG] pgmap v5796: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:33.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:32 smithi154 conmon[34553]: cluster 2021-07-31T03:53:31.600055+0000 mgr.x (mgr.24542) 5409 : cluster [DBG] pgmap v5797: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:33.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:32 smithi045 conmon[39718]: cluster 2021-07-31T03:53:31.600055+0000 mgr.x (mgr.24542) 5409 : cluster [DBG] pgmap v5797: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:33.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:32 smithi045 conmon[33655]: cluster 2021-07-31T03:53:31.600055+0000 mgr.x (mgr.24542) 5409 : 2021-07-31T03:53:33.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:32 smithi045 conmon[33655]: cluster [DBG] pgmap v5797: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:35.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:34 smithi154 conmon[34553]: cluster 2021-07-31T03:53:33.600532+0000 mgr.x (mgr.24542) 5410 : 2021-07-31T03:53:35.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:34 smithi154 conmon[34553]: cluster [DBG] pgmap v5798: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:35.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:34 smithi045 conmon[39718]: cluster 2021-07-31T03:53:33.600532+0000 mgr.x (mgr.24542) 5410 : cluster [DBG] pgmap v5798: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:35.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:34 smithi045 conmon[33655]: cluster 2021-07-31T03:53:33.600532+0000 mgr.x (mgr. 2021-07-31T03:53:35.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:34 smithi045 conmon[33655]: 24542) 5410 : cluster [DBG] pgmap v5798: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:35.987 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: debug 2021-07-31T03:53:35.727+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:53:35.988 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:53:35.988 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:53:35.988 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: yield 2021-07-31T03:53:35.989 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:53:35.989 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:53:35.989 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:53:35.989 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:53:35.990 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:53:35.990 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:53:35.990 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:53:35.991 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:53:35.991 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:53:35.991 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: task.result() 2021-07-31T03:53:35.992 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:53:35.992 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: debug 2021-07-31T03:53:35.739+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:53:35.992 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:53:35.992 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:53:35.993 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: yield 2021-07-31T03:53:35.993 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:53:35.993 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:53:35.993 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:53:35.994 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:53:35.994 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:53:35.994 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:53:35.995 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:53:35.995 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:53:35.995 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:53:35.996 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: task.result() 2021-07-31T03:53:35.996 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:53:35.996 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: debug 2021-07-31T03:53:35.765+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:53:35.996 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:53:35.997 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:53:35.997 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: yield 2021-07-31T03:53:35.997 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:53:35.998 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:53:35.998 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:53:35.998 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:53:35.998 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:53:35.999 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:53:35.999 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:53:35.999 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:53:36.000 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:53:36.000 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: task.result() 2021-07-31T03:53:36.000 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:53:36.000 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: debug 2021-07-31T03:53:35.765+0000 7fcb93160700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:53:36.001 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:53:36.001 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:53:36.001 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: yield 2021-07-31T03:53:36.001 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:53:36.002 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:53:36.002 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:53:36.002 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:53:36.003 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:53:36.003 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:53:36.003 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:53:36.003 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:53:36.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:53:36.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: task.result() 2021-07-31T03:53:36.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:53:36.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: 2021-07-31T03:53:36.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:53:36.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: 2021-07-31T03:53:36.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:53:36.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:53:36.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:53:36.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:53:36.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:53:36.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:53:36.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:53:36.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:53:36.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:53:36.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:53:36.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:53:36.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:53:36.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:53:36.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:53:36.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:53:36.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:53:36.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:53:36.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:53:36.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: raise 2021-07-31T03:53:36.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:53:36.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:53:36.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:53:36.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:53:36.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:53:36.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: debug 2021-07-31T03:53:35.774+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:53:36.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:53:36.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:53:36.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: yield 2021-07-31T03:53:36.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:53:36.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:53:36.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:53:36.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:53:36.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:53:36.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:53:36.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:53:36.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:53:36.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:53:36.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: task.result() 2021-07-31T03:53:36.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:53:36.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: debug 2021-07-31T03:53:35.774+0000 7fcb9295f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:53:36.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:53:36.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:53:36.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: yield 2021-07-31T03:53:36.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:53:36.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:53:36.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:53:36.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:53:36.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:53:36.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:53:36.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:53:36.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:53:36.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:53:36.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: task.result() 2021-07-31T03:53:36.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:53:36.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: 2021-07-31T03:53:36.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:53:36.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: 2021-07-31T03:53:36.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:53:36.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:53:36.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:53:36.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:53:36.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:53:36.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:53:36.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:53:36.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:53:36.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:53:36.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:53:36.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:53:36.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:53:36.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:53:36.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:53:36.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:53:36.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:53:36.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:53:36.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:53:36.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: raise 2021-07-31T03:53:36.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:53:36.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:53:36.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:53:36.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:53:36.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:53:36.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[34553]: audit 2021-07-31T03:53:35.682880+0000 mon.b (mon.2) 707 : audit 2021-07-31T03:53:36.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[34553]: [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:53:36.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[34553]: audit 2021-07-31T03:53:35.684408+0000 mon.b (mon.2) 708 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:53:36.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[34553]: audit 2021-07-31T03 2021-07-31T03:53:36.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:35 smithi154 conmon[34553]: :53:35.685445+0000 mon.b (mon.2) 709 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:53:36.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:35 smithi045 conmon[39718]: audit 2021-07-31T03:53:35.682880+0000 mon.b ( 2021-07-31T03:53:36.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:35 smithi045 conmon[39718]: mon.2) 707 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:53:36.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:35 smithi045 conmon[39718]: audit 2021- 2021-07-31T03:53:36.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:35 smithi045 conmon[39718]: 07-31T03:53:35.684408+0000 mon.b (mon.2) 708 : 2021-07-31T03:53:36.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:35 smithi045 conmon[39718]: audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:53:36.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:35 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T03:53:36.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:35 smithi045 conmon[39718]: T03:53:35.685445+0000 mon.b ( 2021-07-31T03:53:36.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:35 smithi045 conmon[39718]: mon.2) 709 : audit 2021-07-31T03:53:36.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:35 smithi045 conmon[39718]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:53:36.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:35 smithi045 conmon[33655]: audit 2021-07-31T03:53:35.682880+0000 mon.b (mon.2) 2021-07-31T03:53:36.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:35 smithi045 conmon[33655]: 707 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:53:36.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:35 smithi045 conmon[33655]: audit 2021-07-31T03:53:35.684408+0000 mon.b (mon.2) 708 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:53:36.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:35 smithi045 conmon[33655]: audit 2021-07- 2021-07-31T03:53:36.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:35 smithi045 conmon[33655]: 31T03:53:35.685445+0000 2021-07-31T03:53:36.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:35 smithi045 conmon[33655]: mon.b (mon.2) 709 : audit [INF] 2021-07-31T03:53:36.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:35 smithi045 conmon[33655]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:53:37.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: cluster 2021-07-31T03:53:35.600985+0000 mgr.x (mgr.24542) 5411 : cluster [DBG] pgmap v5799: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:37.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: cephadm 2021-07-31T03:53:35.686260+0000 mgr.x (mgr.24542) 5412 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:53:37.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: cephadm 2021-07-31T03:53:35.729711+0000 mgr.x (mgr.24542) 5413 : cephadm [ERR] 2021-07-31T03:53:37.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:53:37.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:53:37.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: yield 2021-07-31T03:53:37.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:53:37.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:53:37.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:53:37.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:53:37.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:53:37.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:53:37.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:53:37.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:53:37.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:53:37.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: task.result() 2021-07-31T03:53:37.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:53:37.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: cephadm 2021-07-31T03:53:35.730148+0000 mgr.x (mgr.24542) 5414 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:53:37.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: cephadm 2021-07-31T03:53:35.741317+0000 mgr.x (mgr.24542) 5415 : cephadm [ERR] 2021-07-31T03:53:37.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:53:37.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:53:37.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: yield 2021-07-31T03:53:37.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:53:37.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:53:37.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:53:37.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:53:37.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:53:37.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:53:37.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:53:37.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:53:37.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:53:37.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: task.result() 2021-07-31T03:53:37.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:53:37.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: cephadm 2021-07-31T03:53:35.741799+0000 mgr.x (mgr.24542) 5416 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:53:37.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: cephadm 2021-07-31T03:53:37.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: 2021-07-31T03:53:35.767107+0000 mgr.x 2021-07-31T03:53:37.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: (mgr.24542) 5417 : cephadm [ERR] 2021-07-31T03:53:37.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: 2021-07-31T03:53:37.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:53:37.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:53:37.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: yield 2021-07-31T03:53:37.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:53:37.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:53:37.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:53:37.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:53:37.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:53:37.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:53:37.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:53:37.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:53:37.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:53:37.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: task.result() 2021-07-31T03:53:37.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:53:37.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: cephadm 2021-07-31T03:53:35.767520+0000 mgr.x (mgr.24542) 5418 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:53:37.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:53:37.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:53:37.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: yield 2021-07-31T03:53:37.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:53:37.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:53:37.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:53:37.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:53:37.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:53:37.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:53:37.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:53:37.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:53:37.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:53:37.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: task.result() 2021-07-31T03:53:37.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:53:37.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: 2021-07-31T03:53:37.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:53:37.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: 2021-07-31T03:53:37.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:53:37.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:53:37.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:53:37.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:53:37.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:53:37.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:53:37.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:53:37.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:53:37.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:53:37.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:53:37.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:53:37.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:53:37.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:53:37.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:53:37.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:53:37.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:53:37.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:53:37.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:53:37.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: raise 2021-07-31T03:53:37.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:53:37.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:53:37.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:53:37.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:53:37.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:53:37.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: cephadm 2021-07-31T03:53:35.776269+0000 mgr.x (mgr.24542) 5419 : cephadm [ERR] 2021-07-31T03:53:37.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:53:37.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:53:37.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: yield 2021-07-31T03:53:37.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:53:37.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:53:37.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:53:37.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:53:37.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:53:37.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:53:37.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:53:37.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:53:37.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:53:37.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: task.result() 2021-07-31T03:53:37.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:53:37.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: cephadm 2021-07-31T03:53:35.776652+0000 mgr.x (mgr.24542) 5420 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:53:37.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:53:37.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:53:37.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: yield 2021-07-31T03:53:37.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:53:37.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:53:37.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:53:37.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:53:37.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:53:37.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:53:37.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:53:37.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:53:37.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:53:37.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: task.result() 2021-07-31T03:53:37.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:53:37.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: 2021-07-31T03:53:37.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:53:37.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: 2021-07-31T03:53:37.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:53:37.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:53:37.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:53:37.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:53:37.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:53:37.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:53:37.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:53:37.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:53:37.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:53:37.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:53:37.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:53:37.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:53:37.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:53:37.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:53:37.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:53:37.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:53:37.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:53:37.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:53:37.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: raise 2021-07-31T03:53:37.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:53:37.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:53:37.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:53:37.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:53:37.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:36 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:53:37.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: cluster 2021-07-31T03:53:35.600985+0000 2021-07-31T03:53:37.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: mgr.x (mgr.24542) 5411 : cluster [DBG] pgmap v5799: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:37.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: cephadm 2021-07-31T03:53:35.686260+0000 mgr.x (mgr.24542) 5412 : cephadm [WRN] 2021-07-31T03:53:37.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:53:37.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: cephadm 2021-07-31T03:53:35.729711+0000 mgr.x 2021-07-31T03:53:37.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: (mgr.24542) 5413 : cephadm [ERR] 2021-07-31T03:53:37.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:53:37.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:53:37.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: yield 2021-07-31T03:53:37.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:53:37.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:53:37.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:53:37.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:53:37.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:53:37.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:53:37.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:53:37.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:53:37.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:53:37.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: task.result() 2021-07-31T03:53:37.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:53:37.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: cephadm 2021 2021-07-31T03:53:37.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: -07-31T03:53:35.730148+0000 mgr.x (mgr.24542) 5414 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:53:37.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: cephadm 2021-07-31T03:53:35.741317+0000 mgr.x (mgr.24542) 5415 : cephadm [ERR] 2021-07-31T03:53:37.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:53:37.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:53:37.290 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: yield 2021-07-31T03:53:37.290 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:53:37.290 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:53:37.290 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:53:37.290 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:53:37.291 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:53:37.291 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:53:37.291 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:53:37.291 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:53:37.291 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:53:37.292 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: task.result() 2021-07-31T03:53:37.292 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:53:37.292 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: cephadm 2021-07-31 2021-07-31T03:53:37.292 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: T03:53:35.741799+0000 mgr.x (mgr.24542) 5416 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:53:37.293 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: cluster 2021-07-31T03:53:35.600985+0000 mgr.x ( 2021-07-31T03:53:37.293 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: mgr.24542) 5411 : cluster [DBG] pgmap v5799: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:37.293 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: cephadm 2021-07-31T 2021-07-31T03:53:37.293 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: 03:53:35.686260+0000 mgr.x (mgr.24542) 5412 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:53:37.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: cephadm 2021-07-31T03:53:37.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: 2021-07-31T 2021-07-31T03:53:37.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: 03:53:35.729711+0000 mgr.x (mgr.24542 2021-07-31T03:53:37.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: ) 5413 : cephadm [ERR] 2021-07-31T03:53:37.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:53:37.295 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:53:37.295 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: yield 2021-07-31T03:53:37.295 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:53:37.295 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:53:37.295 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:53:37.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:53:37.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:53:37.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:53:37.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:53:37.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:53:37.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:53:37.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: task.result() 2021-07-31T03:53:37.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:53:37.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: cephadm 2021-07-31T03:53:35.730148+0000 mgr.x (mgr.24542) 5414 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:53:37.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: cephadm 2021-07-31T03 2021-07-31T03:53:37.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: :53:35.741317+0000 mgr.x (mgr.24542) 5415 : cephadm [ERR] 2021-07-31T03:53:37.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:53:37.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:53:37.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: yield 2021-07-31T03:53:37.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:53:37.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:53:37.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:53:37.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:53:37.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:53:37.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:53:37.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:53:37.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:53:37.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:53:37.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: task.result() 2021-07-31T03:53:37.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:53:37.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: 2021-07-31T03:53:37.301 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: cephadm 2021-07-31T03:53:35.767107+0000 mgr.x (mgr.24542) 5417 : cephadm [ERR] 2021-07-31T03:53:37.301 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:53:37.301 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:53:37.302 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: yield 2021-07-31T03:53:37.302 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:53:37.302 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:53:37.302 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:53:37.302 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:53:37.303 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:53:37.303 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:53:37.303 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:53:37.303 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:53:37.303 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:53:37.304 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: task.result() 2021-07-31T03:53:37.304 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:53:37.304 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: cephadm 2021-07-31T03:53:35.767520+0000 mgr.x (mgr.24542) 5418 : cephadm 2021-07-31T03:53:37.304 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:53:37.304 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:53:37.305 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:53:37.305 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: yield 2021-07-31T03:53:37.305 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:53:37.305 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:53:37.305 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:53:37.306 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:53:37.306 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:53:37.306 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:53:37.306 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:53:37.306 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:53:37.307 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:53:37.307 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: task.result() 2021-07-31T03:53:37.307 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:53:37.307 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: 2021-07-31T03:53:37.307 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:53:37.307 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: 2021-07-31T03:53:37.308 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:53:37.308 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:53:37.308 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:53:37.308 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:53:37.308 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:53:37.309 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:53:37.309 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:53:37.309 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: cephadm 2021-07-31T03:53:35.741799+0000 mgr.x (mgr.24542) 5416 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:53:37.310 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: cephadm 2021-07-31T03:53:35 2021-07-31T03:53:37.310 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: .767107+0000 mgr.x (mgr.24542) 5417 : cephadm [ERR] 2021-07-31T03:53:37.310 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:53:37.310 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:53:37.310 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: yield 2021-07-31T03:53:37.311 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:53:37.311 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:53:37.311 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:53:37.311 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:53:37.311 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:53:37.312 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:53:37.312 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:53:37.312 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:53:37.312 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:53:37.312 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: task.result() 2021-07-31T03:53:37.312 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:53:37.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: cephadm 2021-07-31T03:53:35.767520+0000 mgr.x (mgr.24542) 5418 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:53:37.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:53:37.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:53:37.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: yield 2021-07-31T03:53:37.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:53:37.314 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:53:37.314 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:53:37.314 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:53:37.314 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:53:37.314 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:53:37.315 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:53:37.315 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:53:37.315 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:53:37.315 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: task.result() 2021-07-31T03:53:37.315 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:53:37.316 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: 2021-07-31T03:53:37.316 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:53:37.316 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: 2021-07-31T03:53:37.316 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:53:37.316 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:53:37.317 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:53:37.317 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:53:37.317 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:53:37.317 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:53:37.318 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:53:37.318 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:53:37.318 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:53:37.318 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:53:37.318 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:53:37.319 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:53:37.319 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:53:37.319 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:53:37.319 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:53:37.319 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:53:37.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:53:37.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: raise 2021-07-31T03:53:37.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:53:37.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:53:37.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:53:37.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:53:37.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:53:37.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: cephadm 2021-07-31T03:53:35.776269+0000 mgr.x (mgr.24542) 5419 : cephadm [ERR] 2021-07-31T03:53:37.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:53:37.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:53:37.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: yield 2021-07-31T03:53:37.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:53:37.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:53:37.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:53:37.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:53:37.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:53:37.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:53:37.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:53:37.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:53:37.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:53:37.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: task.result() 2021-07-31T03:53:37.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:53:37.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: cephadm 2021-07-31T03:53:35.776652+0000 mgr.x (mgr.24542) 5420 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:53:37.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:53:37.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:53:37.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: yield 2021-07-31T03:53:37.325 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:53:37.325 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:53:37.325 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:53:37.325 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:53:37.326 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:53:37.326 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:53:37.326 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:53:37.326 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:53:37.326 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:53:37.327 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:53:37.327 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:53:37.327 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:53:37.327 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:53:37.327 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:53:37.328 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: raise 2021-07-31T03:53:37.328 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:53:37.328 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:53:37.328 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:53:37.328 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:53:37.329 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:53:37.329 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: cephadm 2021-07-31T03:53:37.329 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: 2021-07-31T03:53:35.776269+0000 mgr.x (mgr.24542) 5419 : cephadm [ERR] 2021-07-31T03:53:37.329 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:53:37.329 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:53:37.329 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: yield 2021-07-31T03:53:37.330 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:53:37.330 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:53:37.330 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:53:37.330 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:53:37.330 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:53:37.331 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:53:37.331 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:53:37.331 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:53:37.331 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:53:37.331 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: task.result() 2021-07-31T03:53:37.332 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:53:37.332 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: cephadm 2021-07-31T03:53:35.776652+0000 mgr.x (mgr.24542) 5420 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:53:37.332 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:53:37.332 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:53:37.332 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: yield 2021-07-31T03:53:37.333 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:53:37.333 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:53:37.333 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:53:37.333 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:53:37.334 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:53:37.334 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:53:37.334 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:53:37.334 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:53:37.334 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: task.result() 2021-07-31T03:53:37.335 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:53:37.335 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: 2021-07-31T03:53:37.335 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:53:37.335 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: 2021-07-31T03:53:37.335 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:53:37.335 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:53:37.336 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:53:37.336 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:53:37.336 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:53:37.336 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:53:37.336 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:53:37.337 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:53:37.337 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:53:37.337 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:53:37.337 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:53:37.337 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:53:37.338 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:53:37.338 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:53:37.338 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:53:37.338 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:53:37.338 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:53:37.338 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:53:37.339 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: raise 2021-07-31T03:53:37.339 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:53:37.339 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:53:37.339 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:53:37.339 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:53:37.340 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:53:37.340 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:53:37.340 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:53:37.341 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:53:37.341 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:53:37.341 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:53:37.341 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:53:37.341 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:53:37.341 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:53:37.342 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: task.result() 2021-07-31T03:53:37.342 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:53:37.342 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: 2021-07-31T03:53:37.342 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:53:37.342 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: 2021-07-31T03:53:37.343 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:53:37.343 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:53:37.343 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:53:37.343 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:53:37.343 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:53:37.344 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:53:37.344 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:53:37.344 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:53:37.344 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:53:37.344 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:53:37.345 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:53:37.345 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:53:37.345 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:53:37.345 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:53:37.345 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:53:37.346 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:53:37.346 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:53:37.346 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:53:37.346 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: raise 2021-07-31T03:53:37.346 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:53:37.346 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:53:37.347 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:53:37.347 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:53:37.347 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:36 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:53:39.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:38 smithi154 conmon[34553]: cluster 2021-07-31T03:53:37.601496+0000 mgr.x (mgr.24542) 5421 : cluster [DBG] pgmap v5800: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:39.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:38 smithi045 conmon[39718]: cluster 2021-07-31T03:53:37.601496+0000 mgr.x (mgr.24542) 5421 : 2021-07-31T03:53:39.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:38 smithi045 conmon[39718]: cluster [DBG] pgmap v5800: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:39.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:38 smithi045 conmon[33655]: cluster 2021-07-31T03:53:37.601496+0000 mgr.x (mgr.24542) 5421 : cluster [DBG] 2021-07-31T03:53:39.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:38 smithi045 conmon[33655]: pgmap v5800: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:41.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:41 smithi154 conmon[34553]: cluster 2021-07-31T03:53:39.602156+0000 mgr.x (mgr.24542) 5422 : cluster [DBG] pgmap v5801: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:41.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:41 smithi045 conmon[39718]: cluster 2021-07-31T03:53:39.602156+0000 2021-07-31T03:53:41.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:41 smithi045 conmon[39718]: mgr.x (mgr.24542) 5422 : cluster [DBG] pgmap v5801: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:41.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:41 smithi045 conmon[33655]: cluster 2021-07-31T03:53:39.602156+0000 mgr.x (mgr.24542) 5422 : 2021-07-31T03:53:41.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:41 smithi045 conmon[33655]: cluster [DBG] pgmap v5801: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:43.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:43 smithi045 conmon[39718]: cluster 2021-07-31T03:53:41.602605+0000 mgr.x ( 2021-07-31T03:53:43.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:43 smithi045 conmon[39718]: mgr.24542) 5423 : cluster [DBG] pgmap v5802: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:43.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:43 smithi045 conmon[33655]: cluster 2021-07-31T03:53:41.602605+0000 mgr.x (mgr.24542) 5423 : cluster [DBG] 2021-07-31T03:53:43.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:43 smithi045 conmon[33655]: pgmap v5802: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:43.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:43 smithi154 conmon[34553]: cluster 2021-07-31T03:53:41.602605+0000 mgr.x (mgr.24542) 5423 : cluster [DBG] pgmap v5802: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:44.953 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:53:45.230 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:53:45.230 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 2h ago 2h - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:53:45.231 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 2h ago 2h 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:53:45.231 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 2h ago 2h 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:53:45.231 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 2h ago 2h - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:53:45.231 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:53:45.232 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 2h ago 2h 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:53:45.232 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:53:45.232 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 2h ago 2h - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:53:45.232 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 2h ago 2h 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:53:45.233 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:53:45.233 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:53:45.233 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:53:45.234 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:53:45.234 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 2h ago 2h 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:53:45.234 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 2h ago 2h 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:53:45.234 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 2h ago 2h 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:53:45.235 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 2h ago 2h 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:53:45.235 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 2h ago 2h 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:53:45.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:45 smithi045 conmon[33655]: cluster 2021-07-31T03:53:43.603156+0000 mgr.x (mgr.24542) 5424 : cluster [DBG] 2021-07-31T03:53:45.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:45 smithi045 conmon[33655]: pgmap v5803: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:45.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:45 smithi045 conmon[39718]: cluster 2021-07-31T03:53:43. 2021-07-31T03:53:45.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:45 smithi045 conmon[39718]: 603156+0000 mgr.x (mgr.24542) 5424 : cluster [DBG] pgmap v5803: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:45.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:45 smithi154 conmon[34553]: cluster 2021-07-31T03:53:43.603156+0000 mgr.x (mgr.24542) 5424 : cluster [DBG] pgmap v5803: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:45.566 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:53:45.566 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:53:45.566 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:53:45.567 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:53:45.567 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:53:45.567 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:53:45.568 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:53:45.568 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:53:45.568 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:53:45.568 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:53:45.569 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:53:45.569 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:53:45.569 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:53:45.570 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:53:45.570 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:53:45.570 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:53:45.570 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:53:46.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:46 smithi045 conmon[33655]: audit 2021-07-31T03:53:44.941684+0000 mgr.x (mgr.24542) 5425 : audit [DBG] from='client.20460 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:53:46.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:46 smithi045 conmon[33655]: audit 2021-07-31T03:53:45.222453+0000 mgr.x (mgr.24542) 5426 : audit [DBG] from='client.30403 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:53:46.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:46 smithi045 conmon[33655]: audit 2021-07-31T03:53:45.566860+0000 mon.a (mon.0) 3387 : audit [DBG] from='client.? 172.21.15.45:0/2467120704' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:53:46.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:46 smithi045 conmon[39718]: audit 2021-07-31T03:53: 2021-07-31T03:53:46.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:46 smithi045 conmon[39718]: 44.941684+0000 mgr.x (mgr.24542) 5425 : audit [DBG] from='client.20460 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:53:46.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:46 smithi045 conmon[39718]: audit 2021-07-31T03:53:45 2021-07-31T03:53:46.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:46 smithi045 conmon[39718]: .222453+0000 mgr.x (mgr.24542) 5426 : audit 2021-07-31T03:53:46.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:46 smithi045 conmon[39718]: [DBG] from='client.30403 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:53:46.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:46 smithi045 conmon[39718]: audit 2021-07-31T03:53:46.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:46 smithi045 conmon[39718]: 2021-07-31T03 2021-07-31T03:53:46.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:46 smithi045 conmon[39718]: :53:45.566860+0000 mon.a 2021-07-31T03:53:46.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:46 smithi045 conmon[39718]: (mon.0) 3387 : audit 2021-07-31T03:53:46.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:46 smithi045 conmon[39718]: [DBG] from='client.? 172.21.15.45:0/2467120704' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:53:46.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:46 smithi154 conmon[34553]: audit 2021-07-31T03:53:44.941684+0000 mgr.x (mgr.24542) 5425 : audit [DBG] from='client.20460 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:53:46.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:46 smithi154 conmon[34553]: audit 2021-07-31T03:53:45.222453+0000 mgr.x 2021-07-31T03:53:46.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:46 smithi154 conmon[34553]: (mgr.24542) 5426 : audit [DBG] from='client.30403 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:53:46.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:46 smithi154 conmon[34553]: audit 2021-07-31T03:53:45.566860+0000 mon.a (mon.0) 3387 : audit [DBG] from='client.? 172.21.15.45:0/2467120704' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:53:47.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:47 smithi045 conmon[33655]: cluster 2021-07-31T03:53:45.603661+0000 mgr.x (mgr.24542) 5427 : cluster [DBG] pgmap v5804: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:47.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:47 smithi045 conmon[33655]: 2021-07-31T03:53:47.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:47 smithi045 conmon[39718]: cluster 2021-07-31T03:53:45.603661+0000 mgr.x (mgr. 2021-07-31T03:53:47.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:47 smithi045 conmon[39718]: 24542) 5427 : cluster [DBG] pgmap v5804: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:47.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:47 smithi154 conmon[34553]: cluster 2021-07-31T03:53:45.603661+0000 mgr.x (mgr.24542) 5427 : cluster [DBG] pgmap v5804: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:49.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:49 smithi045 conmon[33655]: cluster 2021-07-31T03:53:47.604063+0000 mgr.x (mgr.24542) 5428 2021-07-31T03:53:49.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:49 smithi045 conmon[33655]: : cluster [DBG] pgmap v5805: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:49.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:49 smithi045 conmon[39718]: cluster 2021-07-31T03:53:47.604063+0000 mgr.x (mgr 2021-07-31T03:53:49.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:49 smithi045 conmon[39718]: .24542) 5428 : cluster [DBG] pgmap v5805: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:49.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:49 smithi154 conmon[34553]: cluster 2021-07-31T03:53:47.604063+0000 mgr.x (mgr.24542) 5428 : cluster [DBG] pgmap v5805: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:51.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:51 smithi045 conmon[33655]: cluster 2021-07-31T03:53:49.604667+0000 mgr.x (mgr.24542) 5429 : cluster [DBG] pgmap v5806: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:51.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:51 smithi045 conmon[39718]: cluster 2021-07-31T03:53:49.604667 2021-07-31T03:53:51.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:51 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 5429 : cluster [DBG] pgmap v5806: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:51.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:51 smithi154 conmon[34553]: cluster 2021-07-31T03:53:49.604667+0000 mgr.x (mgr.24542) 5429 : cluster [DBG] pgmap v5806: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:53.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:53 smithi045 conmon[39718]: cluster 2021-07-31T03:53:51.605107 2021-07-31T03:53:53.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:53 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 5430 : cluster [DBG] pgmap v5807: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:53.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:53 smithi045 conmon[33655]: cluster 2021-07-31T03:53:51.605107+0000 mgr.x 2021-07-31T03:53:53.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:53 smithi045 conmon[33655]: (mgr.24542) 5430 : cluster [DBG] pgmap v5807: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:53.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:53 smithi154 conmon[34553]: cluster 2021-07-31T03:53:51.605107+0000 mgr.x (mgr.24542) 5430 : cluster [DBG] pgmap v5807: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:55.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:55 smithi154 conmon[34553]: cluster 2021-07-31T03:53:53.605597+0000 mgr.x (mgr.24542) 5431 2021-07-31T03:53:55.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:55 smithi154 conmon[34553]: : cluster [DBG] pgmap v5808: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:55.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:55 smithi045 conmon[39718]: cluster 2021-07-31T03:53:53.605597+0000 mgr.x (mgr 2021-07-31T03:53:55.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:55 smithi045 conmon[39718]: .24542) 5431 : cluster [DBG] pgmap v5808: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:55.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:55 smithi045 conmon[33655]: cluster 2021-07-31T03:53:53.605597+0000 mgr.x (mgr.24542) 5431 : 2021-07-31T03:53:55.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:55 smithi045 conmon[33655]: cluster [DBG] pgmap v5808: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:57.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:57 smithi154 conmon[34553]: cluster 2021-07-31T03:53:55.606113+0000 mgr.x (mgr.24542) 5432 : cluster [DBG] 2021-07-31T03:53:57.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:57 smithi154 conmon[34553]: pgmap v5809: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:57.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:57 smithi045 conmon[33655]: cluster 2021-07-31T03:53:55.606113+0000 mgr.x (mgr.24542) 5432 : cluster 2021-07-31T03:53:57.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:57 smithi045 conmon[33655]: [DBG] pgmap v5809: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:57.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:57 smithi045 conmon[39718]: cluster 2021-07-31T03:53:55.606113+0000 mgr.x (mgr. 2021-07-31T03:53:57.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:57 smithi045 conmon[39718]: 24542) 5432 : cluster [DBG] pgmap v5809: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:59.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:59 smithi154 conmon[34553]: cluster 2021-07-31T03:53:57.606498+0000 mgr.x (mgr.24542) 5433 : cluster 2021-07-31T03:53:59.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:53:59 smithi154 conmon[34553]: [DBG] pgmap v5810: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:59.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:59 smithi045 conmon[33655]: cluster 2021-07-31T03:53:57.606498+0000 2021-07-31T03:53:59.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:53:59 smithi045 conmon[33655]: mgr.x (mgr.24542) 5433 : cluster [DBG] pgmap v5810: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:53:59.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:59 smithi045 conmon[39718]: cluster 2021-07-31T03:53:57.606498+0000 mgr.x (mgr. 2021-07-31T03:53:59.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:53:59 smithi045 conmon[39718]: 24542) 5433 : cluster [DBG] pgmap v5810: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:01.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:01 smithi154 conmon[34553]: cluster 2021-07-31T03:53:59.607087+0000 mgr.x (mgr.24542) 5434 : cluster [DBG] 2021-07-31T03:54:01.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:01 smithi154 conmon[34553]: pgmap v5811: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:01.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:01 smithi045 conmon[33655]: cluster 2021-07-31T03:53:59.607087+0000 mgr.x (mgr.24542) 5434 : cluster [DBG] pgmap v5811: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:01.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:01 smithi045 conmon[33655]: 2021-07-31T03:54:01.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:01 smithi045 conmon[39718]: cluster 2021-07-31T03:53: 2021-07-31T03:54:01.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:01 smithi045 conmon[39718]: 59.607087+0000 mgr.x (mgr.24542) 5434 : cluster [DBG] pgmap v5811: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:03.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:03 smithi154 conmon[34553]: cluster 2021-07-31T03:54:01.607500+0000 mgr.x (mgr.24542) 5435 : cluster [DBG] pgmap v5812: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:03.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:03 smithi045 conmon[33655]: cluster 2021-07-31T03:54:01.607500+0000 mgr.x (mgr.24542) 5435 : cluster 2021-07-31T03:54:03.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:03 smithi045 conmon[33655]: [DBG] pgmap v5812: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:03.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:03 smithi045 conmon[39718]: cluster 2021-07-31T03:54:01.607500+0000 mgr.x (mgr.24542) 5435 : cluster [DBG] pgmap v5812: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:05.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:05 smithi154 conmon[34553]: cluster 2021-07-31T03:54:03.607875+0000 mgr.x (mgr.24542) 5436 : cluster [DBG] pgmap v5813: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:05.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:05 smithi045 conmon[33655]: cluster 2021-07-31T03:54:03.607875+0000 mgr.x (mgr.24542) 5436 : cluster 2021-07-31T03:54:05.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:05 smithi045 conmon[33655]: [DBG] pgmap v5813: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:05.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:05 smithi045 conmon[39718]: cluster 2021-07-31T03:54:03.607875+0000 mgr.x (mgr.24542) 2021-07-31T03:54:05.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:05 smithi045 conmon[39718]: 5436 : cluster [DBG] pgmap v5813: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:07.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:07 smithi154 conmon[34553]: cluster 2021-07-31T03:54:05.608490+0000 mgr.x (mgr.24542) 5437 : cluster [DBG] pgmap v5814: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:07.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:07 smithi045 conmon[33655]: cluster 2021-07-31T03:54: 2021-07-31T03:54:07.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:07 smithi045 conmon[33655]: 05.608490+0000 mgr.x (mgr.24542) 5437 : cluster [DBG] pgmap v5814: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:07.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:07 smithi045 conmon[39718]: cluster 2021-07-31T03:54:05.608490+0000 mgr.x (mgr. 2021-07-31T03:54:07.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:07 smithi045 conmon[39718]: 24542) 5437 : cluster [DBG] pgmap v5814: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:09.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:09 smithi154 conmon[34553]: cluster 2021-07-31T03:54:07.608956+0000 mgr.x (mgr.24542) 5438 : cluster [DBG] pgmap v5815: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:09.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:09 smithi045 conmon[39718]: cluster 2021-07-31T03:54:07.608956+0000 mgr.x (mgr. 2021-07-31T03:54:09.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:09 smithi045 conmon[39718]: 24542) 5438 : cluster [DBG] pgmap v5815: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:09.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:09 smithi045 conmon[33655]: cluster 2021-07-31T03:54:07.608956+0000 mgr.x (mgr.24542) 5438 : cluster [DBG] 2021-07-31T03:54:09.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:09 smithi045 conmon[33655]: pgmap v5815: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:11.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:11 smithi154 conmon[34553]: cluster 2021-07-31T03:54:09.609425+0000 mgr.x (mgr.24542) 5439 : cluster [DBG] pgmap v5816: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:11.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:11 smithi045 conmon[39718]: cluster 2021-07-31T03:54:09.609425+0000 mgr.x ( 2021-07-31T03:54:11.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:11 smithi045 conmon[39718]: mgr.24542) 5439 : cluster [DBG] pgmap v5816: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:11.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:11 smithi045 conmon[33655]: cluster 2021-07-31T03:54:09.609425 2021-07-31T03:54:11.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:11 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 5439 : cluster [DBG] pgmap v5816: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:13.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:13 smithi154 conmon[34553]: cluster 2021-07-31T03:54:11.609797+0000 mgr.x (mgr.24542) 5440 : cluster [DBG] pgmap v5817: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:13.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:13 smithi045 conmon[39718]: cluster 2021-07-31T03:54:11.609797+0000 mgr.x (mgr.24542) 5440 : cluster [DBG] pgmap v5817: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:13.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:13 smithi045 conmon[33655]: cluster 2021-07-31T03:54:11.609797+0000 mgr.x (mgr 2021-07-31T03:54:13.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:13 smithi045 conmon[33655]: .24542) 5440 : cluster [DBG] pgmap v5817: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:15.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:15 smithi154 conmon[34553]: cluster 2021-07-31T03:54:13.610317+0000 mgr.x (mgr.24542) 5441 : cluster [DBG] pgmap v5818: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:15.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:15 smithi045 conmon[39718]: cluster 2021-07-31T03:54:13.610317+0000 mgr.x (mgr 2021-07-31T03:54:15.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:15 smithi045 conmon[39718]: .24542) 5441 : cluster [DBG] pgmap v5818: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:15.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:15 smithi045 conmon[33655]: cluster 2021-07-31T03:54:13.610317+0000 mgr.x (mgr.24542) 5441 : 2021-07-31T03:54:15.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:15 smithi045 conmon[33655]: cluster [DBG] pgmap v5818: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:15.878 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:54:16.154 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:54:16.154 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 2h ago 2h - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:54:16.155 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 2h ago 2h 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:54:16.155 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 2h ago 2h 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:54:16.155 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 2h ago 2h - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:54:16.155 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:54:16.155 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 2h ago 2h 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:54:16.156 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:54:16.156 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 2h ago 2h - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:54:16.156 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 2h ago 2h 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:54:16.156 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:54:16.157 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:54:16.157 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:54:16.157 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:54:16.157 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 2h ago 2h 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:54:16.157 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 2h ago 2h 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:54:16.158 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 2h ago 2h 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:54:16.158 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 2h ago 2h 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:54:16.158 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 2h ago 2h 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:54:16.493 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:54:16.494 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:54:16.494 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:54:16.494 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:54:16.494 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:54:16.495 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:54:16.495 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:54:16.495 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:54:16.495 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:54:16.495 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:54:16.496 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:54:16.496 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:54:16.496 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:54:16.496 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:54:16.496 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:54:16.497 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:54:16.497 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:54:17.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:17 smithi154 conmon[34553]: cluster 2021-07-31T03:54:15.610795+0000 mgr.x (mgr.24542) 5442 : cluster [DBG] pgmap v5819: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:17.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:17 smithi154 conmon[34553]: audit 2021-07-31T03:54:15.866356+0000 2021-07-31T03:54:17.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:17 smithi154 conmon[34553]: mgr.x (mgr.24542) 5443 : audit [DBG] from='client.20478 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:54:17.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:17 smithi154 conmon[34553]: audit 2021-07-31T03:54:16.146797+0000 mgr.x (mgr.24542) 5444 : audit [DBG] from='client.30421 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:54:17.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:17 smithi154 conmon[34553]: audit 2021-07-31T03:54:16.494563+0000 mon.c (mon.1) 234 : audit 2021-07-31T03:54:17.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:17 smithi154 conmon[34553]: [DBG] from='client.? 172.21.15.45:0/3244546016' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:54:17.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:17 smithi045 conmon[39718]: cluster 2021-07-31T03:54:15.610795+0000 mgr.x (mgr 2021-07-31T03:54:17.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:17 smithi045 conmon[39718]: .24542) 5442 : cluster [DBG] pgmap v5819: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:17.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:17 smithi045 conmon[39718]: audit 2021-07-31T03:54:15.866356+0000 mgr.x (mgr. 2021-07-31T03:54:17.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:17 smithi045 conmon[39718]: 24542) 5443 : audit [DBG] from='client.20478 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:54:17.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:17 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T03:54:17.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:17 smithi045 conmon[39718]: T03:54:16.146797+0000 mgr.x (mgr.24542) 5444 : audit [DBG] from='client.30421 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:54:17.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:17 smithi045 conmon[39718]: audit 2021-07-31T03:54:16.494563 2021-07-31T03:54:17.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:17 smithi045 conmon[39718]: +0000 mon.c (mon.1) 234 : audit [DBG] from='client.? 172.21.15.45:0/3244546016' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:54:17.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:17 smithi045 conmon[33655]: cluster 2021-07-31T03:54:15.610795+0000 mgr.x (mgr.24542) 5442 : cluster [DBG] pgmap v5819: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:17.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:17 smithi045 conmon[33655]: audit 2021-07-31T03:54:15.866356+0000 mgr.x (mgr.24542) 5443 : audit [DBG] from='client.20478 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:54:17.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:17 smithi045 conmon[33655]: audit 2021-07-31T03:54:16.146797+0000 mgr.x (mgr.24542) 5444 : audit [DBG] from='client.30421 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:54:17.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:17 smithi045 conmon[33655]: audit 2021-07-31T03:54:16. 2021-07-31T03:54:17.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:17 smithi045 conmon[33655]: 494563+0000 mon.c (mon.1) 234 : audit [DBG] from='client.? 172.21.15.45:0/3244546016' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:54:19.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:19 smithi154 conmon[34553]: cluster 2021-07-31T03:54:17.611331+0000 mgr.x (mgr.24542) 5445 : cluster [DBG] pgmap v5820: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:19.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:19 smithi045 conmon[39718]: cluster 2021-07-31T03:54:17.611331+0000 mgr.x (mgr.24542) 2021-07-31T03:54:19.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:19 smithi045 conmon[39718]: 5445 : cluster [DBG] pgmap v5820: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:19.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:19 smithi045 conmon[33655]: cluster 2021-07-31T03:54:17.611331+0000 mgr.x (mgr.24542) 5445 : cluster [DBG] pgmap v5820: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:21.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:21 smithi154 conmon[34553]: cluster 2021-07-31T03:54:19.611897+0000 mgr.x (mgr.24542) 5446 : cluster [DBG] pgmap v5821: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:21.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:21 smithi045 conmon[39718]: cluster 2021-07-31T03:54:19.611897+0000 mgr.x (mgr.24542 2021-07-31T03:54:21.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:21 smithi045 conmon[39718]: ) 5446 : cluster [DBG] pgmap v5821: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:21.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:21 smithi045 conmon[33655]: cluster 2021-07-31T03:54:19.611897+0000 mgr.x (mgr.24542) 5446 : 2021-07-31T03:54:21.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:21 smithi045 conmon[33655]: cluster [DBG] pgmap v5821: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:23.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:23 smithi154 conmon[34553]: cluster 2021-07-31T03:54:21.612311+0000 mgr.x (mgr.24542) 5447 : cluster [DBG] pgmap v5822: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:23.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:23 smithi045 conmon[33655]: cluster 2021-07-31T03:54:21.612311+0000 mgr.x (mgr.24542) 5447 2021-07-31T03:54:23.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:23 smithi045 conmon[33655]: : cluster [DBG] pgmap v5822: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:23.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:23 smithi045 conmon[39718]: cluster 2021-07-31T03:54:21.612311+0000 mgr.x ( 2021-07-31T03:54:23.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:23 smithi045 conmon[39718]: mgr.24542) 5447 : cluster [DBG] pgmap v5822: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:25.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:25 smithi154 conmon[34553]: cluster 2021-07-31T03:54:23.612902+0000 mgr.x (mgr.24542) 5448 : cluster [DBG] pgmap v5823: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:25.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:25 smithi154 conmon[34553]: 2021-07-31T03:54:25.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:25 smithi045 conmon[39718]: cluster 2021-07-31T03:54:23.612902+0000 mgr.x (mgr.24542) 5448 : 2021-07-31T03:54:25.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:25 smithi045 conmon[39718]: cluster [DBG] pgmap v5823: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:25.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:25 smithi045 conmon[33655]: cluster 2021-07-31T03:54:23.612902+0000 mgr.x (mgr.24542) 5448 : 2021-07-31T03:54:25.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:25 smithi045 conmon[33655]: cluster [DBG] pgmap v5823: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:27.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:27 smithi154 conmon[34553]: cluster 2021-07-31T03:54:25.613349+0000 mgr.x (mgr.24542) 5449 : cluster [DBG] pgmap v5824: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:27.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:27 smithi045 conmon[33655]: cluster 2021-07-31T03:54:25.613349+0000 mgr.x (mgr.24542) 5449 2021-07-31T03:54:27.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:27 smithi045 conmon[33655]: : cluster [DBG] pgmap v5824: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:27.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:27 smithi045 conmon[39718]: cluster 2021-07-31T03:54:25.613349+0000 2021-07-31T03:54:27.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:27 smithi045 conmon[39718]: mgr.x (mgr.24542) 5449 : cluster [DBG] pgmap v5824: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:29.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:29 smithi154 conmon[34553]: cluster 2021-07-31T03:54:27.613754+0000 mgr.x (mgr.24542) 5450 : cluster [DBG] pgmap v5825: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:29.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:29 smithi154 conmon[34553]: audit 2021-07-31T03:54:28.439265+0000 mon.b (mon.2) 710 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:54:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:29 smithi154 conmon[34553]: audit 2021-07-31T03:54:28.439556+0000 mon.b (mon.2) 711 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:54:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:29 smithi154 conmon[34553]: audit 2021-07-31T03:54:28.440218+0000 mon.a (mon.0) 3388 2021-07-31T03:54:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:29 smithi154 conmon[34553]: : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:54:29.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:29 smithi154 conmon[34553]: audit 2021-07-31 2021-07-31T03:54:29.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:29 smithi154 conmon[34553]: T03:54:28.440629+0000 mon.a (mon. 2021-07-31T03:54:29.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:29 smithi154 conmon[34553]: 0) 3389 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:54:29.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:29 smithi045 conmon[33655]: cluster 2021-07-31T03:54:27.613754+0000 mgr.x (mgr.24542) 5450 : cluster [DBG] pgmap v5825: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:29.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:29 smithi045 conmon[33655]: audit 2021-07-31T03:54:28.439265+0000 mon.b (mon.2) 710 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:54:29.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:29 smithi045 conmon[33655]: audit 2021-07-31T03:54:28.439556+0000 mon.b (mon.2) 711 : audit [INF] 2021-07-31T03:54:29.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:29 smithi045 conmon[33655]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:54:29.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:29 smithi045 conmon[33655]: audit 2021-07- 2021-07-31T03:54:29.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:29 smithi045 conmon[33655]: 31T03:54:28.440218+0000 mon.a ( 2021-07-31T03:54:29.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:29 smithi045 conmon[33655]: mon.0) 3388 : audit [INF] 2021-07-31T03:54:29.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:29 smithi045 conmon[33655]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:54:29.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:29 smithi045 conmon[33655]: audit 2021-07-31T03:54:29.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:29 smithi045 conmon[33655]: 2021-07- 2021-07-31T03:54:29.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:29 smithi045 conmon[33655]: 31T03:54:28.440629 2021-07-31T03:54:29.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:29 smithi045 conmon[33655]: +0000 mon.a (mon.0) 2021-07-31T03:54:29.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:29 smithi045 conmon[33655]: 3389 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:54:29.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:29 smithi045 conmon[39718]: cluster 2021-07-31T03:54:27.613754+0000 mgr.x ( 2021-07-31T03:54:29.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:29 smithi045 conmon[39718]: mgr.24542) 5450 : cluster [DBG] pgmap v5825: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:29.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:29 smithi045 conmon[39718]: audit 2021-07-31T03: 2021-07-31T03:54:29.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:29 smithi045 conmon[39718]: 54:28.439265+0000 mon.b (mon.2) 710 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:54:29.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:29 smithi045 conmon[39718]: 2021-07-31T03:54:29.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:29 smithi045 conmon[39718]: audit 2021-07-31T03: 2021-07-31T03:54:29.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:29 smithi045 conmon[39718]: 54:28.439556+0000 mon.b (mon. 2021-07-31T03:54:29.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:29 smithi045 conmon[39718]: 2) 711 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:54:29.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:29 smithi045 conmon[39718]: audit 2021- 2021-07-31T03:54:29.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:29 smithi045 conmon[39718]: 07-31T03:54:28.440218 2021-07-31T03:54:29.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:29 smithi045 conmon[39718]: +0000 mon.a (mon.0) 3388 : audit 2021-07-31T03:54:29.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:29 smithi045 conmon[39718]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:54:29.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:29 smithi045 conmon[39718]: audit 2021-07-31T03 2021-07-31T03:54:29.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:29 smithi045 conmon[39718]: :54:28.440629+0000 mon.a (mon 2021-07-31T03:54:29.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:29 smithi045 conmon[39718]: .0) 3389 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:54:31.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:31 smithi154 conmon[34553]: cluster 2021-07-31T03:54:29.614211+0000 mgr.x (mgr.24542) 5451 : cluster [DBG] 2021-07-31T03:54:31.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:31 smithi154 conmon[34553]: pgmap v5826: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:31.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:31 smithi045 conmon[33655]: cluster 2021-07-31T03:54:29.614211+0000 mgr.x (mgr.24542) 5451 2021-07-31T03:54:31.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:31 smithi045 conmon[33655]: : cluster [DBG] pgmap v5826: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:31.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:31 smithi045 conmon[39718]: cluster 2021-07-31T03:54:29.614211+0000 mgr.x ( 2021-07-31T03:54:31.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:31 smithi045 conmon[39718]: mgr.24542) 5451 : cluster [DBG] pgmap v5826: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:33.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:33 smithi154 conmon[34553]: cluster 2021-07-31T03:54:31.614673+0000 mgr.x (mgr.24542) 5452 : 2021-07-31T03:54:33.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:33 smithi154 conmon[34553]: cluster [DBG] pgmap v5827: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:33.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:33 smithi045 conmon[33655]: cluster 2021-07-31T03:54:31.614673+0000 mgr.x (mgr.24542) 5452 : cluster [DBG] pgmap v5827: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:33.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:33 smithi045 conmon[39718]: cluster 2021-07-31T03:54:31.614673+0000 mgr.x (mgr.24542) 5452 : cluster [DBG] pgmap v5827: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:35.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[34553]: cluster 2021-07-31T03:54:33.615253+0000 mgr.x (mgr.24542) 5453 : cluster [DBG] pgmap v5828: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:35.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:35 smithi045 conmon[33655]: cluster 2021-07-31T03:54:33.615253+0000 mgr.x (mgr.24542) 5453 : cluster 2021-07-31T03:54:35.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:35 smithi045 conmon[33655]: [DBG] pgmap v5828: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:35.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:35 smithi045 conmon[39718]: cluster 2021-07-31T03:54:33.615253+0000 mgr.x (mgr.24542) 5453 : cluster [DBG] 2021-07-31T03:54:35.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:35 smithi045 conmon[39718]: pgmap v5828: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:36.103 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: debug 2021-07-31T03:54:35.822+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:54:36.103 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:54:36.104 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:54:36.104 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: yield 2021-07-31T03:54:36.104 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:54:36.104 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:54:36.105 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:54:36.105 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:54:36.105 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:54:36.106 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:54:36.106 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:54:36.106 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:54:36.107 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:54:36.107 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: task.result() 2021-07-31T03:54:36.107 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:54:36.108 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: debug 2021-07-31T03:54:35.833+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:54:36.108 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:54:36.108 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:54:36.109 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: yield 2021-07-31T03:54:36.109 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:54:36.109 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:54:36.109 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:54:36.110 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:54:36.110 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:54:36.110 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:54:36.111 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:54:36.111 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:54:36.111 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:54:36.112 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: task.result() 2021-07-31T03:54:36.112 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:54:36.112 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: debug 2021-07-31T03:54:35.858+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:54:36.113 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:54:36.113 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:54:36.113 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: yield 2021-07-31T03:54:36.113 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:54:36.114 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:54:36.114 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:54:36.114 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:54:36.115 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:54:36.115 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:54:36.115 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:54:36.116 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:54:36.116 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:54:36.116 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: task.result() 2021-07-31T03:54:36.116 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:54:36.117 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: debug 2021-07-31T03:54:35.859+0000 7fcb9215e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:54:36.117 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:54:36.117 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:54:36.117 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: yield 2021-07-31T03:54:36.118 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:54:36.118 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:54:36.118 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:54:36.119 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:54:36.119 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:54:36.119 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:54:36.119 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:54:36.120 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:54:36.120 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:54:36.120 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: task.result() 2021-07-31T03:54:36.121 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:54:36.121 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: 2021-07-31T03:54:36.121 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:54:36.121 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: 2021-07-31T03:54:36.122 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:54:36.122 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:54:36.122 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:54:36.123 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:54:36.123 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:54:36.123 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:54:36.123 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:54:36.124 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:54:36.124 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:54:36.124 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:54:36.125 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:54:36.125 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:54:36.125 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:54:36.125 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:54:36.126 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:54:36.126 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:54:36.126 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:54:36.127 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:54:36.127 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: raise 2021-07-31T03:54:36.127 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:54:36.134 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:54:36.134 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:54:36.135 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:54:36.135 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:54:36.135 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: debug 2021-07-31T03:54:35.867+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:54:36.136 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:54:36.136 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:54:36.136 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: yield 2021-07-31T03:54:36.137 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:54:36.137 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:54:36.137 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:54:36.138 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:54:36.138 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:54:36.138 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:54:36.139 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:54:36.139 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:54:36.139 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:54:36.140 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: task.result() 2021-07-31T03:54:36.140 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:54:36.140 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: debug 2021-07-31T03:54:35.868+0000 7fcb9195d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:54:36.140 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:54:36.141 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:54:36.141 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: yield 2021-07-31T03:54:36.141 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:54:36.142 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:54:36.142 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:54:36.142 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:54:36.143 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:54:36.143 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:54:36.143 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:54:36.144 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:54:36.144 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:54:36.144 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: task.result() 2021-07-31T03:54:36.145 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:54:36.145 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: 2021-07-31T03:54:36.145 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:54:36.146 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: 2021-07-31T03:54:36.146 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:54:36.146 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:54:36.146 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:54:36.147 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:54:36.147 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:54:36.147 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:54:36.148 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:54:36.148 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:54:36.148 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:54:36.148 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:54:36.149 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:54:36.149 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:54:36.149 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:54:36.150 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:54:36.150 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:54:36.150 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:54:36.151 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:54:36.151 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:54:36.151 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: raise 2021-07-31T03:54:36.152 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:54:36.152 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:54:36.152 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:54:36.152 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:54:36.153 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:54:35 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:54:36.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:36 smithi154 conmon[34553]: audit 2021-07-31T03:54:35.777388+0000 mon.b (mon.2) 712 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:54:36.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:36 smithi154 conmon[34553]: audit 2021-07-31T03:54:35.778824 2021-07-31T03:54:36.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:36 smithi154 conmon[34553]: +0000 mon.b (mon.2) 713 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:54:36.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:36 smithi154 conmon[34553]: audit 2021-07-31T03:54:35.779650+0000 mon.b (mon.2) 714 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:54:36.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:36 smithi045 conmon[33655]: audit 2021-07-31T03:54:35.777388+0000 mon.b ( 2021-07-31T03:54:36.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:36 smithi045 conmon[33655]: mon.2) 712 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:54:36.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:36 smithi045 conmon[33655]: audit 2021-07-31T03:54:35.778824+0000 mon.b (mon.2) 2021-07-31T03:54:36.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:36 smithi045 conmon[33655]: 713 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:54:36.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:36 smithi045 conmon[33655]: audit 2021 2021-07-31T03:54:36.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:36 smithi045 conmon[33655]: -07-31T03:54: 2021-07-31T03:54:36.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:36 smithi045 conmon[33655]: 35.779650+0000 mon.b (mon 2021-07-31T03:54:36.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:36 smithi045 conmon[33655]: .2) 714 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:54:36.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:36 smithi045 conmon[39718]: audit 2021-07-31T03:54:35.777388+0000 mon.b ( 2021-07-31T03:54:36.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:36 smithi045 conmon[39718]: mon.2) 712 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:54:36.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:36 smithi045 conmon[39718]: audit 2021-07-31T03: 2021-07-31T03:54:36.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:36 smithi045 conmon[39718]: 54:35.778824+0000 mon.b (mon.2) 713 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:54:36.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:36 smithi045 conmon[39718]: 2021-07-31T03:54:36.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:36 smithi045 conmon[39718]: audit 2021-07-31T03:54:36.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:36 smithi045 conmon[39718]: 2021-07-31T03 2021-07-31T03:54:36.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:36 smithi045 conmon[39718]: :54:35.779650+0000 2021-07-31T03:54:36.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:36 smithi045 conmon[39718]: mon.b (mon.2) 714 : audit [INF] 2021-07-31T03:54:36.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:36 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:54:37.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: cluster 2021-07-31T03:54:35.615818+0000 mgr.x (mgr.24542) 5454 : cluster [DBG] pgmap v5829: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:37.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: 2021-07-31T03:54:37.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: cephadm 2021-07-31T03:54:35.780380+0000 mgr.x (mgr.24542) 5455 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:54:37.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: cephadm 2021 2021-07-31T03:54:37.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: -07-31T03:54:35.823102+0000 mgr.x (mgr.24542) 5456 : cephadm [ERR] 2021-07-31T03:54:37.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:54:37.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:54:37.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: yield 2021-07-31T03:54:37.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:54:37.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:54:37.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:54:37.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:54:37.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:54:37.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:54:37.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:54:37.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:54:37.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:54:37.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: task.result() 2021-07-31T03:54:37.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:54:37.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: cephadm 2021-07-31T03:54:35.823528+0000 mgr.x (mgr.24542) 5457 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:54:37.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: cephadm 2021-07-31T03:54:35.834377+0000 mgr.x (mgr.24542) 5458 2021-07-31T03:54:37.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: : cephadm [ERR] 2021-07-31T03:54:37.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:54:37.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:54:37.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: yield 2021-07-31T03:54:37.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:54:37.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:54:37.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:54:37.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:54:37.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:54:37.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:54:37.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:54:37.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:54:37.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:54:37.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: task.result() 2021-07-31T03:54:37.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:54:37.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: cephadm 2021-07-31T03:54:35.835357+0000 mgr.x (mgr.24542) 5459 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:54:37.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: cephadm 2021-07-31T03:54:35.859737+0000 mgr.x (mgr.24542) 5460 : cephadm [ERR] 2021-07-31T03:54:37.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:54:37.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:54:37.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: yield 2021-07-31T03:54:37.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:54:37.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:54:37.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:54:37.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:54:37.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:54:37.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:54:37.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:54:37.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:54:37.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:54:37.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: task.result() 2021-07-31T03:54:37.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:54:37.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: cephadm 2021-07-31T03:54:35.860630+0000 mgr.x (mgr.24542) 5461 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:54:37.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:54:37.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:54:37.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: yield 2021-07-31T03:54:37.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:54:37.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:54:37.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:54:37.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:54:37.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:54:37.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:54:37.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:54:37.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:54:37.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:54:37.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: task.result() 2021-07-31T03:54:37.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:54:37.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: 2021-07-31T03:54:37.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:54:37.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: 2021-07-31T03:54:37.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:54:37.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:54:37.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:54:37.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:54:37.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:54:37.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:54:37.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:54:37.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:54:37.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:54:37.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:54:37.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:54:37.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:54:37.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:54:37.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:54:37.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:54:37.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:54:37.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:54:37.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:54:37.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: raise 2021-07-31T03:54:37.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:54:37.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:54:37.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:54:37.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:54:37.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:54:37.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: cephadm 2021-07-31T03:54:35.868892+0000 mgr.x (mgr.24542) 5462 : cephadm [ERR] 2021-07-31T03:54:37.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:54:37.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:54:37.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: yield 2021-07-31T03:54:37.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:54:37.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:54:37.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:54:37.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:54:37.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:54:37.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:54:37.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:54:37.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:54:37.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:54:37.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: task.result() 2021-07-31T03:54:37.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:54:37.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: cephadm 2021-07-31T03:54:35.869462+0000 mgr.x (mgr.24542) 5463 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:54:37.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:54:37.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:54:37.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: yield 2021-07-31T03:54:37.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:54:37.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:54:37.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:54:37.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:54:37.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:54:37.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:54:37.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:54:37.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:54:37.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:54:37.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: task.result() 2021-07-31T03:54:37.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:54:37.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: 2021-07-31T03:54:37.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:54:37.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: 2021-07-31T03:54:37.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:54:37.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:54:37.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:54:37.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:54:37.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:54:37.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:54:37.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:54:37.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:54:37.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:54:37.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:54:37.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:54:37.541 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:54:37.541 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:54:37.541 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:54:37.541 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:54:37.542 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:54:37.542 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:54:37.542 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:54:37.542 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: raise 2021-07-31T03:54:37.543 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:54:37.543 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:54:37.543 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:54:37.543 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:54:37.544 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:37 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:54:37.544 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: cluster 2021-07-31T03:54:35.615818+0000 mgr.x (mgr.24542) 5454 : cluster 2021-07-31T03:54:37.545 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: [DBG] pgmap v5829: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:37.545 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: cephadm 2021-07-31T03:54:35.780380+0000 mgr.x (mgr.24542) 5455 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:54:37.545 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: cephadm 2021-07-31T03:54:37.546 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: 2021-07-31T03:54:35.823102+0000 mgr.x (mgr.24542) 5456 : cephadm [ERR] 2021-07-31T03:54:37.546 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:54:37.546 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:54:37.546 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: yield 2021-07-31T03:54:37.547 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:54:37.547 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:54:37.547 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:54:37.548 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:54:37.548 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:54:37.548 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:54:37.548 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:54:37.549 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:54:37.549 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:54:37.549 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: task.result() 2021-07-31T03:54:37.549 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:54:37.550 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: cephadm 2021-07-31T03:54:35.823528+0000 mgr.x (mgr.24542) 5457 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:54:37.550 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: cephadm 2021-07-31T03 2021-07-31T03:54:37.550 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: :54:35.834377+0000 mgr.x 2021-07-31T03:54:37.550 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: (mgr.24542) 5458 : cephadm 2021-07-31T03:54:37.551 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: [ERR] 2021-07-31T03:54:37.551 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:54:37.551 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:54:37.551 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: yield 2021-07-31T03:54:37.552 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:54:37.552 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:54:37.552 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:54:37.552 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:54:37.553 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:54:37.553 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:54:37.553 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:54:37.553 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:54:37.554 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:54:37.554 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: task.result() 2021-07-31T03:54:37.554 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:54:37.554 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: cephadm 2021-07-31T03:54:35.835357+0000 mgr.x (mgr.24542) 5459 : cephadm 2021-07-31T03:54:37.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: cluster 2021-07-31T03:54:35.615818+0000 mgr.x ( 2021-07-31T03:54:37.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: mgr.24542) 5454 : cluster [DBG] pgmap v5829: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:37.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: cephadm 2021-07-31T03: 2021-07-31T03:54:37.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: 54:35.780380+0000 mgr.x (mgr.24542) 5455 : cephadm [WRN] 2021-07-31T03:54:37.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:54:37.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: cephadm 2021-07-31T03 2021-07-31T03:54:37.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: :54:35.823102+0000 mgr.x (mgr 2021-07-31T03:54:37.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: .24542) 5456 : cephadm [ERR] 2021-07-31T03:54:37.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:54:37.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:54:37.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: yield 2021-07-31T03:54:37.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:54:37.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:54:37.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:54:37.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:54:37.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:54:37.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:54:37.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:54:37.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:54:37.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:54:37.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: task.result() 2021-07-31T03:54:37.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:54:37.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: cephadm 2021-07-31T03:54:35.823528+0000 mgr.x (mgr.24542) 5457 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:54:37.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: cephadm 2021-07-31T03:54:37.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: 2021-07-31T03:54:35.834377+0000 mgr.x 2021-07-31T03:54:37.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: (mgr.24542) 5458 : cephadm [ERR] 2021-07-31T03:54:37.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:54:37.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:54:37.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: yield 2021-07-31T03:54:37.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:54:37.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:54:37.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:54:37.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:54:37.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:54:37.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:54:37.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:54:37.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:54:37.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:54:37.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: task.result() 2021-07-31T03:54:37.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:54:37.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: 2021-07-31T03:54:37.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: cephadm 2021-07-31T03:54:35.835357+0000 mgr.x (mgr.24542) 5459 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:54:37.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: cephadm 2021-07-31T03:54:37.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: 2021-07-31T03:54:35.859737+0000 mgr.x (mgr 2021-07-31T03:54:37.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: .24542) 5460 : cephadm [ERR] 2021-07-31T03:54:37.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:54:37.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:54:37.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: yield 2021-07-31T03:54:37.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:54:37.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:54:37.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:54:37.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:54:37.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:54:37.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:54:37.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:54:37.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:54:37.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:54:37.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: task.result() 2021-07-31T03:54:37.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:54:37.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: cephadm 2021-07-31T03:54:35.860630+0000 mgr.x (mgr.24542) 5461 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:54:37.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:54:37.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:54:37.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: yield 2021-07-31T03:54:37.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:54:37.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:54:37.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:54:37.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:54:37.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:54:37.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:54:37.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:54:37.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:54:37.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:54:37.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: task.result() 2021-07-31T03:54:37.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:54:37.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: 2021-07-31T03:54:37.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:54:37.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: 2021-07-31T03:54:37.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:54:37.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:54:37.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:54:37.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:54:37.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:54:37.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:54:37.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:54:37.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: cephadm 2021- 2021-07-31T03:54:37.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: 07-31T03:54:35.859737 2021-07-31T03:54:37.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 2021-07-31T03:54:37.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: 5460 : cephadm [ERR] 2021-07-31T03:54:37.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:54:37.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:54:37.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: yield 2021-07-31T03:54:37.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:54:37.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:54:37.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:54:37.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:54:37.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:54:37.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:54:37.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:54:37.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:54:37.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:54:37.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: task.result() 2021-07-31T03:54:37.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:54:37.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: cephadm 2021-07-31T03:54:35.860630+0000 mgr.x (mgr 2021-07-31T03:54:37.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: .24542) 5461 : cephadm [ERR] 2021-07-31T03:54:37.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:54:37.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:54:37.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:54:37.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: yield 2021-07-31T03:54:37.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:54:37.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:54:37.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:54:37.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:54:37.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:54:37.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:54:37.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:54:37.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:54:37.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:54:37.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: task.result() 2021-07-31T03:54:37.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:54:37.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: 2021-07-31T03:54:37.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:54:37.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: 2021-07-31T03:54:37.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:54:37.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:54:37.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:54:37.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:54:37.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:54:37.585 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:54:37.585 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:54:37.585 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:54:37.585 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:54:37.586 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:54:37.586 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:54:37.586 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:54:37.586 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:54:37.586 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:54:37.587 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:54:37.587 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:54:37.587 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:54:37.587 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: raise 2021-07-31T03:54:37.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:54:37.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:54:37.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:54:37.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:54:37.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:54:37.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: cephadm 2021-07-31T03:54:35.868892+0000 mgr.x (mgr.24542) 5462 : cephadm [ERR] 2021-07-31T03:54:37.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:54:37.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:54:37.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: yield 2021-07-31T03:54:37.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:54:37.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:54:37.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:54:37.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:54:37.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:54:37.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:54:37.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:54:37.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:54:37.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:54:37.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: task.result() 2021-07-31T03:54:37.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:54:37.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: cephadm 2021-07-31T03:54:35.869462+0000 mgr.x (mgr.24542) 5463 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:54:37.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:54:37.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:54:37.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: yield 2021-07-31T03:54:37.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:54:37.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:54:37.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:54:37.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:54:37.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:54:37.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:54:37.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:54:37.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:54:37.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:54:37.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:54:37.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:54:37.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:54:37.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:54:37.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:54:37.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: raise 2021-07-31T03:54:37.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:54:37.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:54:37.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:54:37.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:54:37.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:54:37.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: cephadm 2021-07-31T03:54:35.868892+0000 mgr.x (mgr.24542) 5462 : cephadm [ERR] 2021-07-31T03:54:37.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:54:37.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:54:37.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: yield 2021-07-31T03:54:37.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:54:37.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:54:37.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:54:37.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:54:37.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:54:37.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:54:37.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:54:37.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:54:37.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:54:37.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: task.result() 2021-07-31T03:54:37.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:54:37.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: cephadm 2021-07-31T03:54:35.869462+0000 mgr.x (mgr.24542) 5463 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:54:37.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:54:37.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:54:37.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: yield 2021-07-31T03:54:37.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:54:37.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:54:37.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:54:37.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:54:37.603 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:54:37.603 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:54:37.603 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:54:37.603 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:54:37.604 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:54:37.604 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: task.result() 2021-07-31T03:54:37.604 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:54:37.604 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: 2021-07-31T03:54:37.604 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:54:37.605 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: 2021-07-31T03:54:37.605 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:54:37.605 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:54:37.605 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:54:37.605 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:54:37.606 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:54:37.606 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:54:37.606 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:54:37.606 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:54:37.607 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:54:37.607 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:54:37.607 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:54:37.607 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:54:37.607 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:54:37.608 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:54:37.608 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:54:37.608 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:54:37.608 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:54:37.608 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:54:37.609 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: raise 2021-07-31T03:54:37.609 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:54:37.609 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:54:37.609 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:54:37.609 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:54:37.610 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:54:37.610 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:54:37.610 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:54:37.610 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:54:37.611 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:54:37.611 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:54:37.611 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:54:37.611 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:54:37.612 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:54:37.612 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: task.result() 2021-07-31T03:54:37.612 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:54:37.612 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: 2021-07-31T03:54:37.612 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:54:37.613 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: 2021-07-31T03:54:37.613 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:54:37.613 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:54:37.613 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:54:37.613 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:54:37.614 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:54:37.614 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:54:37.614 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:54:37.614 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:54:37.614 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:54:37.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:54:37.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:54:37.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:54:37.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:54:37.616 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:54:37.616 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:54:37.616 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:54:37.616 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:54:37.616 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:54:37.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: raise 2021-07-31T03:54:37.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:54:37.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:54:37.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:54:37.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:54:37.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:37 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:54:39.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:39 smithi154 conmon[34553]: cluster 2021-07-31T03:54:37.616223+0000 mgr.x (mgr.24542) 5464 : cluster [DBG] pgmap v5830: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:39.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:39 smithi045 conmon[39718]: cluster 2021-07-31T03:54:37.616223+0000 mgr.x ( 2021-07-31T03:54:39.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:39 smithi045 conmon[39718]: mgr.24542) 5464 : cluster [DBG] pgmap v5830: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:39.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:39 smithi045 conmon[33655]: cluster 2021-07-31T03:54:37.616223+0000 mgr.x (mgr. 2021-07-31T03:54:39.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:39 smithi045 conmon[33655]: 24542) 5464 : cluster [DBG] pgmap v5830: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:41.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:41 smithi154 conmon[34553]: cluster 2021-07-31T03:54:39.616671+0000 mgr.x (mgr.24542) 5465 : cluster [DBG] pgmap v5831: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:41.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:41 smithi045 conmon[39718]: cluster 2021-07-31T03:54:39.616671+0000 mgr.x ( 2021-07-31T03:54:41.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:41 smithi045 conmon[39718]: mgr.24542) 5465 : cluster [DBG] pgmap v5831: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:41.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:41 smithi045 conmon[33655]: cluster 2021-07-31T03:54:39.616671+0000 mgr.x (mgr.24542) 5465 2021-07-31T03:54:41.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:41 smithi045 conmon[33655]: : cluster [DBG] pgmap v5831: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:43.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:43 smithi154 conmon[34553]: cluster 2021-07-31T03:54:41.617177+0000 mgr.x (mgr.24542) 5466 : cluster [DBG] pgmap v5832: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:43.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:43 smithi045 conmon[39718]: cluster 2021-07-31T03:54:41.617177+0000 mgr.x (mgr 2021-07-31T03:54:43.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:43 smithi045 conmon[39718]: .24542) 5466 : cluster [DBG] pgmap v5832: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:43.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:43 smithi045 conmon[33655]: cluster 2021-07-31T03:54:41.617177+0000 mgr.x (mgr 2021-07-31T03:54:43.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:43 smithi045 conmon[33655]: .24542) 5466 : cluster [DBG] pgmap v5832: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:45.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:45 smithi154 conmon[34553]: cluster 2021-07-31T03:54:43.617681+0000 mgr.x (mgr.24542) 5467 : cluster [DBG] pgmap v5833: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:45.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:45 smithi045 conmon[33655]: cluster 2021-07-31T03:54:43.617681+0000 2021-07-31T03:54:45.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:45 smithi045 conmon[33655]: mgr.x (mgr.24542) 5467 : cluster [DBG] pgmap v5833: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:45.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:45 smithi045 conmon[39718]: cluster 2021-07-31T03:54:43.617681+0000 mgr.x (mgr 2021-07-31T03:54:45.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:45 smithi045 conmon[39718]: .24542) 5467 : cluster [DBG] pgmap v5833: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:46.809 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:54:47.085 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:54:47.085 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 2h ago 2h - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:54:47.085 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 2h ago 2h 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:54:47.086 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 2h ago 2h 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:54:47.086 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 2h ago 2h - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:54:47.086 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:54:47.086 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 2h ago 2h 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:54:47.087 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:54:47.087 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 2h ago 2h - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:54:47.087 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 2h ago 2h 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:54:47.088 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:54:47.088 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:54:47.088 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:54:47.088 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:54:47.089 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 2h ago 2h 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:54:47.089 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 2h ago 2h 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:54:47.089 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 2h ago 2h 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:54:47.090 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 2h ago 2h 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:54:47.090 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 2h ago 2h 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:54:47.422 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:54:47.422 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:54:47.423 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:54:47.423 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:54:47.423 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:54:47.424 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:54:47.424 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:54:47.424 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:54:47.425 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:54:47.425 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:54:47.425 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:54:47.425 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:54:47.426 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:54:47.426 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:54:47.426 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:54:47.427 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:54:47.427 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:54:47.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:47 smithi154 conmon[34553]: cluster 2021-07-31T03:54:45.618228+0000 mgr.x (mgr.24542) 5468 : cluster [DBG] pgmap v5834: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:47.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:47 smithi045 conmon[39718]: cluster 2021-07-31T03:54: 2021-07-31T03:54:47.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:47 smithi045 conmon[39718]: 45.618228+0000 mgr.x (mgr.24542) 5468 : cluster [DBG] pgmap v5834: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:47.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:47 smithi045 conmon[33655]: cluster 2021-07-31T03:54:45.618228+0000 mgr.x (mgr 2021-07-31T03:54:47.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:47 smithi045 conmon[33655]: .24542) 5468 : cluster [DBG] pgmap v5834: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:48.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:48 smithi154 conmon[34553]: audit 2021-07-31T03:54:46.797628+0000 mgr.x (mgr.24542) 5469 : audit [DBG] from='client.30433 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:54:48.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:48 smithi154 conmon[34553]: audit 2021-07-31T03:54:47.078026+0000 mgr.x 2021-07-31T03:54:48.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:48 smithi154 conmon[34553]: (mgr.24542) 5470 : audit [DBG] from='client.30439 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:54:48.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:48 smithi154 conmon[34553]: audit 2021-07-31T03:54:47.422900+0000 mon.a (mon.0) 3390 : audit [DBG] from='client.? 172.21.15.45:0/2925669296' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:54:48.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:48 smithi045 conmon[33655]: audit 2021-07-31T03:54:46.797628+0000 mgr.x ( 2021-07-31T03:54:48.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:48 smithi045 conmon[33655]: mgr.24542) 5469 : audit [DBG] from='client.30433 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:54:48.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:48 smithi045 conmon[33655]: audit 2021-07-31T03:54:47.078026+0000 mgr.x (mgr.24542) 5470 : audit [DBG] from='client.30439 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:54:48.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:48 smithi045 conmon[33655]: audit 2021-07-31T03:54:47.422900+0000 mon.a (mon.0) 3390 : audit [DBG] from='client.? 172.21.15.45:0/2925669296' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:54:48.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:48 smithi045 conmon[39718]: audit 2021-07-31T03:54:46.797628+0000 mgr.x (mgr. 2021-07-31T03:54:48.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:48 smithi045 conmon[39718]: 24542) 5469 : audit [DBG] from='client.30433 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:54:48.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:48 smithi045 conmon[39718]: audit 2021-07-31T03:54:47 2021-07-31T03:54:48.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:48 smithi045 conmon[39718]: .078026+0000 mgr.x (mgr.24542) 5470 : audit [DBG] from='client.30439 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:54:48.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:48 smithi045 conmon[39718]: audit 2021-07-31T03:54:48.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:48 smithi045 conmon[39718]: 2021-07-31T03:54:47.422900+0000 mon.a (mon.0) 3390 : audit 2021-07-31T03:54:48.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:48 smithi045 conmon[39718]: [DBG] from='client.? 172.21.15.45:0/2925669296' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:54:49.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:49 smithi154 conmon[34553]: cluster 2021-07-31T03:54:47.618717+0000 mgr.x (mgr.24542) 5471 : cluster [DBG] pgmap v5835: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:49.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:49 smithi045 conmon[33655]: cluster 2021-07-31T03:54:47.618717+0000 mgr.x (mgr.24542) 5471 : cluster [DBG] pgmap v5835: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:49.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:49 smithi045 conmon[39718]: cluster 2021-07-31T03:54: 2021-07-31T03:54:49.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:49 smithi045 conmon[39718]: 47.618717+0000 mgr.x (mgr.24542) 5471 : cluster [DBG] pgmap v5835: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:51.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:51 smithi154 conmon[34553]: cluster 2021-07-31T03:54:49.619178+0000 mgr.x (mgr.24542) 5472 : cluster [DBG] pgmap v5836: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:51.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:51 smithi045 conmon[33655]: cluster 2021-07-31T03:54:49.619178+0000 mgr.x (mgr.24542) 5472 2021-07-31T03:54:51.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:51 smithi045 conmon[33655]: : cluster [DBG] pgmap v5836: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:51.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:51 smithi045 conmon[39718]: cluster 2021-07-31T03:54:49.619178+0000 mgr.x (mgr 2021-07-31T03:54:51.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:51 smithi045 conmon[39718]: .24542) 5472 : cluster [DBG] pgmap v5836: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:53.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:53 smithi154 conmon[34553]: cluster 2021-07-31T03:54:51.619732+0000 mgr.x (mgr.24542) 5473 : 2021-07-31T03:54:53.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:53 smithi154 conmon[34553]: cluster [DBG] pgmap v5837: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:53.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:53 smithi045 conmon[33655]: cluster 2021-07-31T03:54:51.619732+0000 mgr.x (mgr.24542) 5473 : cluster [DBG] pgmap v5837: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:53.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:53 smithi045 conmon[39718]: cluster 2021-07-31T03:54:51.619732+0000 mgr.x (mgr. 2021-07-31T03:54:53.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:53 smithi045 conmon[39718]: 24542) 5473 : cluster [DBG] pgmap v5837: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:55.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:55 smithi154 conmon[34553]: cluster 2021-07-31T03:54:53.620123+0000 mgr.x (mgr.24542) 5474 : cluster 2021-07-31T03:54:55.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:55 smithi154 conmon[34553]: [DBG] pgmap v5838: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:55.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:55 smithi045 conmon[39718]: cluster 2021-07-31T03:54:53.620123+0000 mgr.x (mgr. 2021-07-31T03:54:55.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:55 smithi045 conmon[39718]: 24542) 5474 : cluster [DBG] pgmap v5838: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:55.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:55 smithi045 conmon[33655]: cluster 2021-07-31T03:54:53.620123+0000 mgr.x (mgr.24542) 5474 : cluster [DBG] pgmap v5838: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:57.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:57 smithi154 conmon[34553]: cluster 2021-07-31T03:54:55.620589+0000 mgr.x (mgr.24542) 5475 : cluster [DBG] 2021-07-31T03:54:57.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:57 smithi154 conmon[34553]: pgmap v5839: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:57.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:57 smithi045 conmon[33655]: cluster 2021-07-31T03:54:55.620589+0000 mgr.x (mgr.24542) 5475 : cluster 2021-07-31T03:54:57.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:57 smithi045 conmon[33655]: [DBG] pgmap v5839: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:57.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:57 smithi045 conmon[39718]: cluster 2021-07-31T03:54:55.620589+0000 mgr.x (mgr.24542 2021-07-31T03:54:57.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:57 smithi045 conmon[39718]: ) 5475 : cluster [DBG] pgmap v5839: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:59.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:59 smithi154 conmon[34553]: cluster 2021-07-31T03:54:57.621006+0000 mgr.x (mgr.24542) 5476 : cluster [DBG] 2021-07-31T03:54:59.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:54:59 smithi154 conmon[34553]: pgmap v5840: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:59.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:54:59 smithi045 conmon[33655]: cluster 2021-07-31T03:54:57.621006+0000 mgr.x (mgr.24542) 5476 : cluster [DBG] pgmap v5840: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:54:59.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:54:59 smithi045 conmon[39718]: cluster 2021-07-31T03:54:57.621006+0000 mgr.x (mgr.24542) 5476 : cluster [DBG] pgmap v5840: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:01.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:01 smithi154 conmon[34553]: cluster 2021-07-31T03:54:59.621618+0000 mgr.x (mgr.24542) 5477 : 2021-07-31T03:55:01.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:01 smithi154 conmon[34553]: cluster [DBG] pgmap v5841: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:01.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:01 smithi045 conmon[39718]: cluster 2021-07-31T03:54:59.621618+0000 mgr.x (mgr.24542) 2021-07-31T03:55:01.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:01 smithi045 conmon[39718]: 5477 : cluster [DBG] pgmap v5841: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:01.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:01 smithi045 conmon[33655]: cluster 2021-07-31T03:54:59.621618+0000 mgr.x (mgr.24542) 5477 : cluster [DBG] pgmap v5841: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:03.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:03 smithi154 conmon[34553]: cluster 2021-07-31T03:55:01.622139+0000 mgr.x (mgr.24542) 5478 2021-07-31T03:55:03.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:03 smithi154 conmon[34553]: : cluster [DBG] pgmap v5842: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:03.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:03 smithi045 conmon[39718]: cluster 2021-07-31T03:55:01 2021-07-31T03:55:03.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:03 smithi045 conmon[39718]: .622139+0000 mgr.x (mgr.24542) 5478 : cluster [DBG] pgmap v5842: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:03.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:03 smithi045 conmon[33655]: cluster 2021-07-31T03:55:01.622139+0000 mgr.x (mgr.24542) 5478 2021-07-31T03:55:03.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:03 smithi045 conmon[33655]: : cluster [DBG] pgmap v5842: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:05.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:05 smithi154 conmon[34553]: cluster 2021-07-31T03:55:03.622621+0000 mgr.x (mgr.24542) 5479 : cluster 2021-07-31T03:55:05.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:05 smithi154 conmon[34553]: [DBG] pgmap v5843: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:05.524 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:05 smithi045 conmon[39718]: cluster 2021-07-31T03:55:03.622621+0000 mgr.x (mgr.24542) 2021-07-31T03:55:05.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:05 smithi045 conmon[39718]: 5479 : cluster [DBG] pgmap v5843: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:05.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:05 smithi045 conmon[33655]: cluster 2021-07-31T03:55:03.622621+0000 mgr.x (mgr.24542) 5479 : cluster 2021-07-31T03:55:05.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:05 smithi045 conmon[33655]: [DBG] pgmap v5843: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:07.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:07 smithi154 conmon[34553]: cluster 2021-07-31T03:55:05.623198+0000 mgr.x (mgr.24542) 5480 : cluster [DBG] pgmap v5844: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:07.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:07 smithi045 conmon[33655]: cluster 2021-07-31T03:55:05.623198+0000 2021-07-31T03:55:07.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:07 smithi045 conmon[33655]: mgr.x (mgr.24542) 5480 : cluster [DBG] pgmap v5844: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:07.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:07 smithi045 conmon[39718]: cluster 2021-07-31T03:55:05.623198+0000 mgr.x (mgr.24542) 5480 2021-07-31T03:55:07.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:07 smithi045 conmon[39718]: : cluster [DBG] pgmap v5844: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:09.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:09 smithi154 conmon[34553]: cluster 2021-07-31T03:55:07.623672+0000 mgr.x (mgr.24542) 5481 : cluster [DBG] 2021-07-31T03:55:09.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:09 smithi154 conmon[34553]: pgmap v5845: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:09.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:09 smithi045 conmon[39718]: cluster 2021-07-31T03:55:07.623672+0000 mgr.x (mgr 2021-07-31T03:55:09.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:09 smithi045 conmon[39718]: .24542) 5481 : cluster [DBG] pgmap v5845: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:09.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:09 smithi045 conmon[33655]: cluster 2021-07-31T03:55: 2021-07-31T03:55:09.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:09 smithi045 conmon[33655]: 07.623672+0000 mgr.x (mgr.24542) 5481 : cluster [DBG] pgmap v5845: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:11.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:11 smithi154 conmon[34553]: cluster 2021-07-31T03:55:09.624104+0000 mgr.x (mgr.24542) 5482 : cluster [DBG] pgmap v5846: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:11.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:11 smithi045 conmon[39718]: cluster 2021-07-31T03:55:09.624104+0000 mgr.x (mgr 2021-07-31T03:55:11.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:11 smithi045 conmon[39718]: .24542) 5482 : cluster [DBG] pgmap v5846: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:11.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:11 smithi045 conmon[33655]: cluster 2021-07-31T03:55:09.624104+0000 mgr.x (mgr.24542) 5482 : cluster [DBG] pgmap v5846: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:13.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:13 smithi154 conmon[34553]: cluster 2021-07-31T03:55:11.624480+0000 mgr.x (mgr.24542) 5483 : cluster [DBG] pgmap v5847: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:13.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:13 smithi045 conmon[33655]: cluster 2021-07 2021-07-31T03:55:13.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:13 smithi045 conmon[33655]: -31T03:55:11.624480+0000 mgr.x (mgr.24542) 5483 : cluster [DBG] pgmap v5847: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:13.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:13 smithi045 conmon[39718]: cluster 2021-07-31T03:55:11.624480+0000 mgr.x (mgr 2021-07-31T03:55:13.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:13 smithi045 conmon[39718]: .24542) 5483 : cluster [DBG] pgmap v5847: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:15.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:15 smithi154 conmon[34553]: cluster 2021-07-31T03:55:13.624828+0000 mgr.x (mgr.24542) 5484 : cluster [DBG] pgmap v5848: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:15.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:15 smithi045 conmon[39718]: cluster 2021-07-31T03:55:13.624828+0000 mgr.x (mgr.24542) 5484 : cluster [DBG] pgmap v5848: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:15.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:15 smithi045 conmon[33655]: cluster 2021-07-31T03:55:13.624828+0000 mgr.x (mgr.24542) 5484 : cluster [DBG] pgmap v5848: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:15.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:15 smithi045 conmon[33655]: 2021-07-31T03:55:17.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:17 smithi154 conmon[34553]: cluster 2021-07-31T03:55:15.625400+0000 mgr.x (mgr.24542) 5485 : cluster [DBG] pgmap v5849: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:17.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:17 smithi045 conmon[39718]: cluster 2021-07-31T03:55:15.625400+0000 mgr.x ( 2021-07-31T03:55:17.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:17 smithi045 conmon[39718]: mgr.24542) 5485 : cluster [DBG] pgmap v5849: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:17.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:17 smithi045 conmon[33655]: cluster 2021-07-31T03:55:15.625400+0000 mgr.x (mgr.24542) 5485 2021-07-31T03:55:17.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:17 smithi045 conmon[33655]: : cluster [DBG] pgmap v5849: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:17.733 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:55:18.009 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:55:18.009 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 2h ago 2h - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:55:18.009 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 2h ago 2h 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:55:18.009 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 2h ago 2h 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:55:18.010 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 2h ago 2h - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:55:18.010 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:55:18.010 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 2h ago 2h 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:55:18.011 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:55:18.011 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 2h ago 2h - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:55:18.011 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 2h ago 2h 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:55:18.011 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:55:18.011 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:55:18.012 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:55:18.012 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:55:18.012 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 2h ago 2h 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:55:18.012 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 2h ago 2h 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:55:18.013 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 2h ago 2h 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:55:18.013 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 2h ago 2h 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:55:18.013 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 2h ago 2h 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:55:18.345 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:55:18.346 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:55:18.346 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:55:18.346 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:55:18.347 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:55:18.347 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:55:18.347 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:55:18.347 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:55:18.348 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:55:18.348 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:55:18.348 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:55:18.349 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:55:18.349 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:55:18.349 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:55:18.350 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:55:18.350 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:55:18.350 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:55:19.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:19 smithi154 conmon[34553]: cluster 2021-07-31T03:55:17.625886+0000 mgr.x (mgr.24542) 5486 : cluster [DBG] pgmap v5850: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:19.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:19 smithi154 conmon[34553]: audit 2021-07-31T03:55:17.722548+0000 mgr.x (mgr.24542) 5487 : audit [DBG] from='client.30451 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:55:19.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:19 smithi154 conmon[34553]: audit 2021-07-31T03:55:18.001499+0000 mgr.x (mgr.24542) 5488 : audit [DBG] from='client.20520 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:55:19.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:19 smithi154 conmon[34553]: audit 2021 2021-07-31T03:55:19.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:19 smithi154 conmon[34553]: -07-31T03:55:18.346363+0000 mon.a (mon.0) 3391 : audit [DBG] from='client.? 172.21.15.45:0/4187622850' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:55:19.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:19 smithi045 conmon[33655]: cluster 2021-07-31T03:55:17.625886+0000 mgr.x (mgr.24542) 5486 : 2021-07-31T03:55:19.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:19 smithi045 conmon[33655]: cluster [DBG] pgmap v5850: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:19.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:19 smithi045 conmon[33655]: audit 2021-07-31T03:55:17.722548+0000 mgr.x (mgr.24542) 5487 : audit [DBG] from='client.30451 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:55:19.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:19 smithi045 conmon[33655]: audit 2021-07-31T03:55:19.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:19 smithi045 conmon[33655]: 2021-07-31T03:55:18.001499+0000 mgr.x (mgr.24542) 2021-07-31T03:55:19.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:19 smithi045 conmon[33655]: 5488 : audit [DBG] from='client.20520 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:55:19.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:19 smithi045 conmon[33655]: audit 2021-07-31T03:55: 2021-07-31T03:55:19.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:19 smithi045 conmon[33655]: 18.346363+0000 mon.a (mon.0) 3391 : audit [DBG] 2021-07-31T03:55:19.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:19 smithi045 conmon[33655]: from='client.? 172.21.15.45:0/4187622850' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:55:19.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:19 smithi045 conmon[39718]: cluster 2021-07-31T03:55:17.625886+0000 mgr.x ( 2021-07-31T03:55:19.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:19 smithi045 conmon[39718]: mgr.24542) 5486 : cluster [DBG] pgmap v5850: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:19.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:19 smithi045 conmon[39718]: audit 2021-07-31T03:55:17.722548+0000 2021-07-31T03:55:19.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:19 smithi045 conmon[39718]: mgr.x (mgr.24542) 5487 : audit [DBG] from='client.30451 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:55:19.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:19 smithi045 conmon[39718]: audit 2021-07-31T03: 2021-07-31T03:55:19.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:19 smithi045 conmon[39718]: 55:18.001499+0000 mgr.x (mgr.24542 2021-07-31T03:55:19.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:19 smithi045 conmon[39718]: ) 5488 : audit [DBG] from='client.20520 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:55:19.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:19 smithi045 conmon[39718]: audit 2021- 2021-07-31T03:55:19.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:19 smithi045 conmon[39718]: 07-31T03:55:18.346363+0000 2021-07-31T03:55:19.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:19 smithi045 conmon[39718]: mon.a (mon.0) 3391 : audit [DBG] 2021-07-31T03:55:19.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:19 smithi045 conmon[39718]: from='client.? 172.21.15.45:0/4187622850' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:55:21.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:21 smithi154 conmon[34553]: cluster 2021-07-31T03:55:19.626342+0000 mgr.x (mgr.24542) 5489 : cluster [DBG] pgmap v5851: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:21.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:21 smithi045 conmon[33655]: cluster 2021-07-31T03:55:19.626342+0000 mgr.x ( 2021-07-31T03:55:21.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:21 smithi045 conmon[33655]: mgr.24542) 5489 : cluster [DBG] pgmap v5851: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:21.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:21 smithi045 conmon[39718]: cluster 2021-07-31T03:55:19.626342+0000 mgr.x (mgr 2021-07-31T03:55:21.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:21 smithi045 conmon[39718]: .24542) 5489 : cluster [DBG] pgmap v5851: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:23.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:23 smithi154 conmon[34553]: cluster 2021-07-31T03:55:21.626720+0000 mgr.x (mgr.24542) 5490 : cluster [DBG] pgmap v5852: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:23.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:23 smithi045 conmon[39718]: cluster 2021-07-31T03:55:21.626720+0000 mgr.x (mgr. 2021-07-31T03:55:23.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:23 smithi045 conmon[39718]: 24542) 5490 : cluster [DBG] pgmap v5852: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:23.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:23 smithi045 conmon[33655]: cluster 2021-07-31T03:55:21.626720+0000 mgr.x (mgr.24542) 5490 : cluster [DBG] pgmap v5852: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:25.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:25 smithi154 conmon[34553]: cluster 2021-07-31T03:55:23.627220+0000 mgr.x (mgr.24542) 5491 : cluster [DBG] pgmap v5853: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:25.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:25 smithi045 conmon[33655]: cluster 2021-07-31T03:55:23.627220+0000 mgr.x (mgr 2021-07-31T03:55:25.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:25 smithi045 conmon[33655]: .24542) 5491 : cluster [DBG] pgmap v5853: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:25.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:25 smithi045 conmon[39718]: cluster 2021-07-31T03:55:23.627220+0000 mgr.x (mgr 2021-07-31T03:55:25.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:25 smithi045 conmon[39718]: .24542) 5491 : cluster [DBG] pgmap v5853: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:27.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:27 smithi154 conmon[34553]: cluster 2021-07-31T03:55:25.627643+0000 mgr.x (mgr.24542) 5492 : cluster [DBG] pgmap v5854: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:27.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:27 smithi154 conmon[34553]: 2021-07-31T03:55:27.524 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:27 smithi045 conmon[33655]: cluster 2021-07-31T03:55:25.627643+0000 mgr.x (mgr 2021-07-31T03:55:27.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:27 smithi045 conmon[33655]: .24542) 5492 : cluster [DBG] pgmap v5854: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:27.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:27 smithi045 conmon[39718]: cluster 2021-07-31T03:55:25.627643+0000 mgr.x (mgr 2021-07-31T03:55:27.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:27 smithi045 conmon[39718]: .24542) 5492 : cluster [DBG] pgmap v5854: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:29.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:29 smithi154 conmon[34553]: cluster 2021-07-31T03:55:27.628097+0000 mgr.x (mgr.24542) 5493 : cluster [DBG] pgmap v5855: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:29 smithi154 conmon[34553]: audit 2021-07-31T03:55:28.442983+0000 mon.b (mon.2) 715 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:55:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:29 smithi154 conmon[34553]: audit 2021-07-31T03:55:28.443355+0000 mon.b (mon.2) 716 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:55:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:29 smithi154 conmon[34553]: audit 2021 2021-07-31T03:55:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:29 smithi154 conmon[34553]: -07-31T03:55:28.443668+0000 mon.a (mon.0) 3392 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:55:29.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:29 smithi154 conmon[34553]: audit 2021-07-31T 2021-07-31T03:55:29.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:29 smithi154 conmon[34553]: 03:55:28.444059+0000 mon.a (mon.0) 3393 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:55:29.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:29 smithi045 conmon[39718]: cluster 2021-07-31T03:55:27.628097+0000 mgr.x (mgr.24542) 5493 : cluster [DBG] pgmap v5855: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:29.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:29 smithi045 conmon[39718]: audit 2021-07-31T03:55:28.442983+0000 mon.b (mon.2) 715 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:55:29.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:29 smithi045 conmon[39718]: audit 2021-07-31T03:55:28.443355+0000 mon.b 2021-07-31T03:55:29.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:29 smithi045 conmon[39718]: (mon.2) 716 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:55:29.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:29 smithi045 conmon[39718]: audit 2021- 2021-07-31T03:55:29.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:29 smithi045 conmon[39718]: 07-31T03:55:28.443668+0000 mon.a (mon.0 2021-07-31T03:55:29.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:29 smithi045 conmon[39718]: ) 3392 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:55:29.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:29 smithi045 conmon[39718]: audit 2021 2021-07-31T03:55:29.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:29 smithi045 conmon[39718]: -07-31T03:55:28.444059 2021-07-31T03:55:29.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:29 smithi045 conmon[39718]: +0000 mon.a (mon.0) 3393 : audit 2021-07-31T03:55:29.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:29 smithi045 conmon[39718]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:55:29.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:29 smithi045 conmon[33655]: cluster 2021-07-31T03:55:27.628097+0000 mgr.x (mgr.24542) 5493 : cluster [DBG] 2021-07-31T03:55:29.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:29 smithi045 conmon[33655]: pgmap v5855: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:29.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:29 smithi045 conmon[33655]: audit 2021-07-31T03:55:28.442983+0000 mon.b (mon.2) 715 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:55:29.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:29 smithi045 conmon[33655]: audit 2021-07-31T03 2021-07-31T03:55:29.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:29 smithi045 conmon[33655]: :55:28.443355+0000 mon.b (mon.2 2021-07-31T03:55:29.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:29 smithi045 conmon[33655]: ) 716 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:55:29.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:29 smithi045 conmon[33655]: audit 2021-07 2021-07-31T03:55:29.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:29 smithi045 conmon[33655]: -31T03:55:28.443668+0000 2021-07-31T03:55:29.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:29 smithi045 conmon[33655]: mon.a (mon.0) 3392 : audit [INF] 2021-07-31T03:55:29.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:29 smithi045 conmon[33655]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:55:29.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:29 smithi045 conmon[33655]: audit 2021-07-31T03:55 2021-07-31T03:55:29.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:29 smithi045 conmon[33655]: :28.444059+0000 mon.a (mon.0) 2021-07-31T03:55:29.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:29 smithi045 conmon[33655]: 3393 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:55:31.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:31 smithi154 conmon[34553]: cluster 2021-07-31T03:55:29.628494+0000 mgr.x (mgr.24542) 5494 : cluster [DBG] pgmap v5856: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:31.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:31 smithi154 conmon[34553]: 2021-07-31T03:55:31.524 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:31 smithi045 conmon[39718]: cluster 2021-07-31T03:55:29.628494+0000 mgr.x (mgr.24542) 5494 : cluster [DBG] pgmap v5856: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:31.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:31 smithi045 conmon[33655]: cluster 2021-07-31T03:55:29.628494+0000 mgr.x (mgr.24542) 5494 : cluster [DBG] 2021-07-31T03:55:31.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:31 smithi045 conmon[33655]: pgmap v5856: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:33.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:33 smithi154 conmon[34553]: cluster 2021-07-31T03:55:31.628984+0000 mgr.x (mgr.24542) 5495 : cluster [DBG] 2021-07-31T03:55:33.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:33 smithi154 conmon[34553]: pgmap v5857: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:33.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:33 smithi045 conmon[39718]: cluster 2021-07-31T03:55:31.628984+0000 mgr.x (mgr.24542) 5495 : cluster [DBG] pgmap v5857: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:33.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:33 smithi045 conmon[33655]: cluster 2021-07-31T03:55:31.628984+0000 mgr.x (mgr.24542) 5495 : cluster [DBG] pgmap v5857: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:33.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:33 smithi045 conmon[33655]: 2021-07-31T03:55:35.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[34553]: cluster 2021-07-31T03:55:33.629514+0000 mgr.x (mgr.24542) 5496 : cluster [DBG] pgmap v5858: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:35.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[34553]: 2021-07-31T03:55:35.524 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:35 smithi045 conmon[33655]: cluster 2021-07-31T03:55:33.629514+0000 mgr.x (mgr.24542 2021-07-31T03:55:35.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:35 smithi045 conmon[33655]: ) 5496 : cluster [DBG] pgmap v5858: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:35.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:35 smithi045 conmon[39718]: cluster 2021-07-31T03:55:33.629514+0000 mgr.x (mgr 2021-07-31T03:55:35.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:35 smithi045 conmon[39718]: .24542) 5496 : cluster [DBG] pgmap v5858: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:36.220 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: debug 2021-07-31T03:55:35.915+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:55:36.220 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:55:36.220 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:55:36.220 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: yield 2021-07-31T03:55:36.221 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:55:36.221 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:55:36.221 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:55:36.221 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:55:36.222 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:55:36.222 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:55:36.222 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:55:36.222 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:55:36.223 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:55:36.223 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: task.result() 2021-07-31T03:55:36.223 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:55:36.223 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: debug 2021-07-31T03:55:35.926+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:55:36.224 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:55:36.224 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:55:36.224 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: yield 2021-07-31T03:55:36.224 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:55:36.224 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:55:36.225 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:55:36.225 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:55:36.225 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:55:36.225 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:55:36.226 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:55:36.226 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:55:36.226 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:55:36.226 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: task.result() 2021-07-31T03:55:36.226 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:55:36.227 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: debug 2021-07-31T03:55:35.952+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:55:36.227 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:55:36.227 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:55:36.227 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: yield 2021-07-31T03:55:36.228 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:55:36.228 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:55:36.228 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:55:36.228 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:55:36.229 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:55:36.229 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:55:36.229 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:55:36.229 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:55:36.230 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:55:36.230 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: task.result() 2021-07-31T03:55:36.230 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:55:36.230 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: debug 2021-07-31T03:55:35.953+0000 7fcb9115c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:55:36.230 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:55:36.231 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:55:36.231 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: yield 2021-07-31T03:55:36.231 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:55:36.231 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:55:36.232 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:55:36.232 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:55:36.232 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:55:36.232 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:55:36.232 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:55:36.233 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:55:36.233 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:55:36.233 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: task.result() 2021-07-31T03:55:36.233 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:55:36.234 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: 2021-07-31T03:55:36.234 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:55:36.234 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: 2021-07-31T03:55:36.234 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:55:36.234 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:55:36.235 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:55:36.235 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:55:36.235 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:55:36.235 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:55:36.236 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:55:36.236 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:55:36.236 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:55:36.236 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:55:36.236 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:55:36.237 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:55:36.237 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:55:36.237 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:55:36.237 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:55:36.238 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:55:36.238 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:55:36.238 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:55:36.238 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: raise 2021-07-31T03:55:36.238 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:55:36.239 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:55:36.239 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:55:36.239 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:55:36.239 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:55:36.240 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: debug 2021-07-31T03:55:35.960+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:55:36.240 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:55:36.240 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:55:36.240 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: yield 2021-07-31T03:55:36.240 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:55:36.241 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:55:36.241 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:55:36.241 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:55:36.241 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:55:36.242 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:55:36.242 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:55:36.242 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:55:36.242 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:55:36.242 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: task.result() 2021-07-31T03:55:36.243 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:55:36.243 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: debug 2021-07-31T03:55:35.961+0000 7fcb9095b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:55:36.243 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:55:36.243 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:55:36.244 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: yield 2021-07-31T03:55:36.244 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:55:36.244 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:55:36.244 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:55:36.244 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:55:36.245 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:55:36.245 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:55:36.245 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:55:36.245 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:55:36.245 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:55:36.246 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: task.result() 2021-07-31T03:55:36.246 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:55:36.246 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: 2021-07-31T03:55:36.246 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:55:36.247 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: 2021-07-31T03:55:36.247 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:55:36.247 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:55:36.247 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:55:36.247 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:55:36.248 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:55:36.248 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:55:36.248 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:55:36.248 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:55:36.249 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:55:36.249 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:55:36.249 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:55:36.250 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:55:36.250 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:55:36.251 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:55:36.251 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:55:36.251 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:55:36.252 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:55:36.252 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:55:36.252 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: raise 2021-07-31T03:55:36.253 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:55:36.253 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:55:36.253 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:55:36.253 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:55:36.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:55:35 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:55:36.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:36 smithi154 conmon[34553]: audit 2021-07-31T03:55:35.870359+0000 mon.b (mon.2) 717 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:55:36.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:36 smithi154 conmon[34553]: audit 2021-07-31T03:55:35.871731+0000 mon.b (mon.2) 718 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:55:36.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:36 smithi154 conmon[34553]: audit 2021-07-31T03:55:35.872606+0000 mon.b (mon.2) 719 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:55:36.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:36 smithi045 conmon[39718]: audit 2021-07-31T03:55:35.870359+0000 2021-07-31T03:55:36.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:36 smithi045 conmon[39718]: mon.b (mon.2) 717 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:55:36.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:36 smithi045 conmon[39718]: audit 2021-07-31T03 2021-07-31T03:55:36.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:36 smithi045 conmon[39718]: :55:35.871731+0000 mon.b (mon.2) 718 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:55:36.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:36 smithi045 conmon[39718]: audit 2021- 2021-07-31T03:55:36.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:36 smithi045 conmon[39718]: 07-31T03:55:35.872606+0000 2021-07-31T03:55:36.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:36 smithi045 conmon[39718]: mon.b (mon.2) 719 : audit [INF] 2021-07-31T03:55:36.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:36 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:55:36.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:36 smithi045 conmon[33655]: audit 2021-07-31T03:55:35.870359+0000 mon.b (mon.2) 717 : 2021-07-31T03:55:36.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:36 smithi045 conmon[33655]: audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:55:36.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:36 smithi045 conmon[33655]: audit 2021-07-31T03:55:35.871731+0000 mon.b (mon.2) 718 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:55:36.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:36 smithi045 conmon[33655]: audit 2021-07-31T03:55:36.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:36 smithi045 conmon[33655]: 2021-07-31T03:55:35.872606+0000 mon.b (mon.2) 719 : audit [INF] 2021-07-31T03:55:36.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:36 smithi045 conmon[33655]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:55:37.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: cluster 2021-07-31T03:55:35.629998+0000 mgr.x (mgr.24542) 5497 : cluster [DBG] pgmap v5859: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:37.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: cephadm 2021-07-31T03:55:35.873377+0000 mgr.x (mgr.24542) 5498 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:55:37.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: cephadm 2021-07-31T03:55:35.916038+0000 mgr.x (mgr.24542) 5499 : cephadm [ERR] 2021-07-31T03:55:37.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:55:37.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:55:37.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: yield 2021-07-31T03:55:37.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:55:37.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:55:37.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:55:37.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:55:37.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:55:37.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:55:37.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:55:37.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:55:37.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:55:37.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: task.result() 2021-07-31T03:55:37.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:55:37.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: cephadm 2021-07-31T03:55:35.916481+0000 mgr.x (mgr.24542) 5500 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:55:37.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: cephadm 2021-07-31T03:55:35.927806+0000 mgr.x (mgr.24542) 5501 : cephadm [ERR] 2021-07-31T03:55:37.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:55:37.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:55:37.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: yield 2021-07-31T03:55:37.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:55:37.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:55:37.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:55:37.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:55:37.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:55:37.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:55:37.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:55:37.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:55:37.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:55:37.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: task.result() 2021-07-31T03:55:37.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:55:37.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: cephadm 2021-07-31T03:55:35.928208+0000 mgr.x (mgr.24542) 5502 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:55:37.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: cephadm 2021-07-31T03:55:35.953317+0000 mgr.x (mgr.24542) 5503 : cephadm [ERR] 2021-07-31T03:55:37.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:55:37.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:55:37.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: yield 2021-07-31T03:55:37.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:55:37.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:55:37.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:55:37.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:55:37.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:55:37.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:55:37.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:55:37.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:55:37.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:55:37.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: task.result() 2021-07-31T03:55:37.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:55:37.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: cephadm 2021-07-31T03:55:35.953931+0000 mgr.x (mgr.24542) 5504 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:55:37.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:55:37.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:55:37.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: yield 2021-07-31T03:55:37.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:55:37.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:55:37.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:55:37.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:55:37.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:55:37.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:55:37.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:55:37.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:55:37.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:55:37.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: task.result() 2021-07-31T03:55:37.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:55:37.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: 2021-07-31T03:55:37.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:55:37.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: 2021-07-31T03:55:37.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:55:37.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:55:37.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:55:37.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:55:37.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:55:37.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:55:37.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:55:37.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:55:37.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:55:37.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:55:37.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:55:37.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:55:37.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:55:37.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:55:37.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:55:37.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:55:37.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:55:37.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:55:37.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: raise 2021-07-31T03:55:37.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:55:37.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:55:37.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:55:37.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:55:37.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:55:37.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: cephadm 2021-07-31T03:55:35.961910+0000 mgr.x (mgr.24542) 5505 : cephadm [ERR] 2021-07-31T03:55:37.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:55:37.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:55:37.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: yield 2021-07-31T03:55:37.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:55:37.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:55:37.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:55:37.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:55:37.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:55:37.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:55:37.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:55:37.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:55:37.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:55:37.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: task.result() 2021-07-31T03:55:37.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:55:37.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: 2021-07-31T03:55:37.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: cephadm 2021-07-31T03:55:35.962329+0000 mgr.x (mgr.24542) 5506 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:55:37.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:55:37.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:55:37.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: yield 2021-07-31T03:55:37.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:55:37.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:55:37.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:55:37.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:55:37.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:55:37.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:55:37.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:55:37.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:55:37.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:55:37.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: task.result() 2021-07-31T03:55:37.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:55:37.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: 2021-07-31T03:55:37.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:55:37.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: 2021-07-31T03:55:37.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:55:37.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:55:37.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:55:37.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:55:37.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:55:37.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:55:37.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:55:37.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:55:37.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:55:37.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:55:37.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:55:37.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:55:37.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:55:37.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:55:37.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:55:37.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:55:37.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:55:37.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:55:37.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: raise 2021-07-31T03:55:37.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:55:37.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:55:37.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:55:37.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:55:37.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:37 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:55:37.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: cluster 2021-07-31T03:55:35.629998+0000 mgr.x (mgr.24542) 2021-07-31T03:55:37.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: 5497 : cluster [DBG] pgmap v5859: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:37.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: cephadm 2021-07-31T03:55:35.873377+0000 mgr.x (mgr.24542) 5498 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:55:37.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: cephadm 2021-07-31T03:55:35.916038+0000 mgr.x (mgr.24542) 5499 : cephadm [ERR] 2021-07-31T03:55:37.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:55:37.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:55:37.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: yield 2021-07-31T03:55:37.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:55:37.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:55:37.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:55:37.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:55:37.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:55:37.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:55:37.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:55:37.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:55:37.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:55:37.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: task.result() 2021-07-31T03:55:37.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:55:37.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: cephadm 2021-07-31T03:55:35.916481+0000 mgr.x (mgr.24542) 5500 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:55:37.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: cephadm 2021-07-31T03:55:35.927806+0000 mgr.x (mgr.24542) 5501 : cephadm [ERR] 2021-07-31T03:55:37.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:55:37.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:55:37.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: yield 2021-07-31T03:55:37.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:55:37.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:55:37.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:55:37.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:55:37.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:55:37.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:55:37.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:55:37.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:55:37.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:55:37.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: task.result() 2021-07-31T03:55:37.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:55:37.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: cephadm 2021- 2021-07-31T03:55:37.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: 07-31T03:55:35.928208+0000 mgr.x (mgr.24542) 5502 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:55:37.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: cephadm 2021-07-31T03:55:35.953317+0000 mgr.x (mgr.24542) 5503 : cephadm [ERR] 2021-07-31T03:55:37.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:55:37.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:55:37.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: yield 2021-07-31T03:55:37.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:55:37.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:55:37.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:55:37.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:55:37.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:55:37.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:55:37.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:55:37.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:55:37.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:55:37.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: task.result() 2021-07-31T03:55:37.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:55:37.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: cephadm 2021-07-31T03:55:35.953931+0000 mgr.x (mgr.24542) 5504 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:55:37.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:55:37.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:55:37.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: yield 2021-07-31T03:55:37.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:55:37.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:55:37.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:55:37.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:55:37.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:55:37.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:55:37.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:55:37.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:55:37.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:55:37.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: task.result() 2021-07-31T03:55:37.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:55:37.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: 2021-07-31T03:55:37.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:55:37.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: 2021-07-31T03:55:37.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:55:37.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:55:37.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:55:37.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:55:37.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:55:37.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:55:37.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:55:37.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:55:37.550 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: cluster 2021-07-31T03:55:35.629998+0000 2021-07-31T03:55:37.550 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: mgr.x (mgr.24542) 5497 : cluster [DBG] pgmap v5859: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:37.550 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: cephadm 2021-07-31T03:55 2021-07-31T03:55:37.550 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: :35.873377+0000 mgr.x (mgr.24542) 5498 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:55:37.550 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: cephadm 2021-07-31T03:55:37.551 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: 2021-07-31T03:55:35.916038 2021-07-31T03:55:37.551 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 5499 : cephadm [ERR] 2021-07-31T03:55:37.551 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:55:37.551 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:55:37.551 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: yield 2021-07-31T03:55:37.552 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:55:37.552 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:55:37.552 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:55:37.552 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:55:37.552 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:55:37.553 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:55:37.553 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:55:37.553 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:55:37.553 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:55:37.553 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: task.result() 2021-07-31T03:55:37.554 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:55:37.554 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: cephadm 2021-07-31T03:55:35.916481+0000 mgr.x (mgr.24542) 5500 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:55:37.554 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: cephadm 2021-07-31T03:55:35.927806+0000 2021-07-31T03:55:37.554 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: mgr.x (mgr.24542) 5501 : cephadm [ERR] 2021-07-31T03:55:37.554 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:55:37.555 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:55:37.555 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: yield 2021-07-31T03:55:37.555 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:55:37.555 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:55:37.555 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:55:37.556 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:55:37.556 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:55:37.556 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:55:37.556 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:55:37.556 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:55:37.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:55:37.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: task.result() 2021-07-31T03:55:37.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:55:37.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: cephadm 2021-07-31T03:55:35.928208+0000 mgr.x (mgr.24542) 5502 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:55:37.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:55:37.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:55:37.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:55:37.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:55:37.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:55:37.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:55:37.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:55:37.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:55:37.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:55:37.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:55:37.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: raise 2021-07-31T03:55:37.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:55:37.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:55:37.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:55:37.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:55:37.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:55:37.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: cephadm 2021-07-31T03:55:35.961910+0000 mgr.x (mgr.24542) 5505 : cephadm [ERR] 2021-07-31T03:55:37.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:55:37.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:55:37.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: yield 2021-07-31T03:55:37.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:55:37.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:55:37.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:55:37.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:55:37.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:55:37.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:55:37.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:55:37.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:55:37.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:55:37.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: task.result() 2021-07-31T03:55:37.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:55:37.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: cephadm 2021-07-31T03:55:35.962329+0000 mgr.x (mgr.24542) 5506 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:55:37.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:55:37.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:55:37.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: yield 2021-07-31T03:55:37.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:55:37.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:55:37.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:55:37.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:55:37.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:55:37.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:55:37.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:55:37.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:55:37.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:55:37.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: task.result() 2021-07-31T03:55:37.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:55:37.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: 2021-07-31T03:55:37.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:55:37.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: 2021-07-31T03:55:37.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:55:37.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:55:37.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:55:37.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:55:37.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:55:37.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:55:37.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:55:37.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:55:37.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:55:37.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:55:37.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:55:37.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:55:37.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:55:37.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:55:37.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:55:37.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:55:37.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:55:37.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:55:37.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: raise 2021-07-31T03:55:37.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:55:37.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:55:37.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:55:37.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:55:37.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:55:37.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: cephadm 2021-07-31T03:55:35.953317+0000 mgr.x (mgr.24542) 5503 : cephadm [ERR] 2021-07-31T03:55:37.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:55:37.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:55:37.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: yield 2021-07-31T03:55:37.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:55:37.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:55:37.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:55:37.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:55:37.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:55:37.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:55:37.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:55:37.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:55:37.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:55:37.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: task.result() 2021-07-31T03:55:37.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:55:37.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: cephadm 2021-07-31T03:55:35.953931+0000 mgr.x (mgr.24542) 5504 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:55:37.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:55:37.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:55:37.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: yield 2021-07-31T03:55:37.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:55:37.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:55:37.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:55:37.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:55:37.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:55:37.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:55:37.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:55:37.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:55:37.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:55:37.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: task.result() 2021-07-31T03:55:37.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:55:37.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: 2021-07-31T03:55:37.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:55:37.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: 2021-07-31T03:55:37.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:55:37.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:55:37.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:55:37.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:55:37.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:55:37.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:55:37.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:55:37.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:55:37.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:55:37.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:55:37.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:55:37.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:55:37.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:55:37.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:55:37.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:55:37.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:55:37.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:55:37.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:55:37.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: raise 2021-07-31T03:55:37.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:55:37.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:55:37.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:55:37.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:55:37.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:55:37.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: cephadm 2021-07-31T03:55:35. 2021-07-31T03:55:37.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: 961910+0000 mgr.x (mgr.24542) 5505 : cephadm [ERR] 2021-07-31T03:55:37.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:55:37.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:55:37.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: yield 2021-07-31T03:55:37.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:55:37.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:55:37.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:55:37.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:55:37.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:55:37.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:55:37.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:55:37.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:55:37.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:55:37.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: task.result() 2021-07-31T03:55:37.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:55:37.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: cephadm 2021-07-31T03:55:35.962329+0000 mgr.x (mgr.24542) 5506 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:55:37.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:55:37.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:55:37.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: yield 2021-07-31T03:55:37.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:55:37.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:55:37.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:55:37.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:55:37.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:55:37.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:55:37.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:55:37.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:55:37.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:55:37.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: task.result() 2021-07-31T03:55:37.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:55:37.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: 2021-07-31T03:55:37.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:55:37.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: 2021-07-31T03:55:37.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:55:37.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:55:37.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:55:37.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:55:37.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:55:37.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:55:37.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:55:37.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:55:37.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:55:37.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:55:37.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:55:37.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:55:37.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:55:37.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:55:37.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:55:37.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:55:37.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:55:37.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:55:37.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: raise 2021-07-31T03:55:37.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:55:37.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:55:37.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:55:37.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:55:37.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:37 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:55:39.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:39 smithi154 conmon[34553]: cluster 2021-07-31T03:55:37.630443+0000 mgr.x (mgr.24542) 5507 : cluster [DBG] pgmap v5860: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:39.524 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:39 smithi045 conmon[39718]: cluster 2021-07-31T03:55:37.630443+0000 mgr.x (mgr. 2021-07-31T03:55:39.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:39 smithi045 conmon[39718]: 24542) 5507 : cluster [DBG] pgmap v5860: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:39.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:39 smithi045 conmon[33655]: cluster 2021-07-31T03:55:37.630443+0000 mgr.x (mgr.24542) 5507 : cluster [DBG] pgmap v5860: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:39.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:39 smithi045 conmon[33655]: 2021-07-31T03:55:41.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:41 smithi154 conmon[34553]: cluster 2021-07-31T03:55:39.630961+0000 mgr.x (mgr.24542) 5508 : cluster [DBG] pgmap v5861: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:41.524 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:41 smithi045 conmon[39718]: cluster 2021-07-31T03:55:39.630961 2021-07-31T03:55:41.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:41 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 5508 : cluster [DBG] pgmap v5861: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:41.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:41 smithi045 conmon[33655]: cluster 2021-07-31T03:55:39.630961+0000 mgr.x 2021-07-31T03:55:41.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:41 smithi045 conmon[33655]: (mgr.24542) 5508 : cluster [DBG] pgmap v5861: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:43.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:43 smithi154 conmon[34553]: cluster 2021-07-31T03:55:41.631415+0000 mgr.x (mgr.24542) 5509 2021-07-31T03:55:43.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:43 smithi154 conmon[34553]: : cluster [DBG] pgmap v5862: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:43.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:43 smithi045 conmon[33655]: cluster 2021-07-31T03:55:41.631415+0000 mgr.x (mgr.24542) 5509 : cluster [DBG] pgmap v5862: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:43.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:43 smithi045 conmon[39718]: cluster 2021-07-31T03:55:41.631415+0000 mgr.x (mgr.24542) 2021-07-31T03:55:43.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:43 smithi045 conmon[39718]: 5509 : cluster [DBG] pgmap v5862: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:45.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:45 smithi154 conmon[34553]: cluster 2021-07-31T03:55:43.631951+0000 mgr.x (mgr.24542) 5510 : cluster 2021-07-31T03:55:45.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:45 smithi154 conmon[34553]: [DBG] pgmap v5863: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:45.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:45 smithi045 conmon[33655]: cluster 2021-07-31T03:55:43.631951+0000 mgr.x 2021-07-31T03:55:45.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:45 smithi045 conmon[33655]: (mgr.24542) 5510 : cluster [DBG] pgmap v5863: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:45.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:45 smithi045 conmon[39718]: cluster 2021-07-31T03:55:43.631951+0000 mgr.x 2021-07-31T03:55:45.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:45 smithi045 conmon[39718]: (mgr.24542) 5510 : cluster [DBG] pgmap v5863: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:47.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:47 smithi154 conmon[34553]: cluster 2021-07-31T03:55:45.632470+0000 mgr.x (mgr.24542) 5511 2021-07-31T03:55:47.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:47 smithi154 conmon[34553]: : cluster [DBG] pgmap v5864: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:47.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:47 smithi045 conmon[33655]: cluster 2021-07-31T03:55:45.632470+0000 2021-07-31T03:55:47.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:47 smithi045 conmon[33655]: mgr.x (mgr.24542) 5511 : cluster [DBG] pgmap v5864: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:47.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:47 smithi045 conmon[39718]: cluster 2021-07-31T03:55:45.632470+0000 mgr.x 2021-07-31T03:55:47.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:47 smithi045 conmon[39718]: (mgr.24542) 5511 : cluster [DBG] pgmap v5864: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:48.657 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:55:48.934 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:55:48.934 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 2h ago 2h - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:55:48.935 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 2h ago 2h 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:55:48.935 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 2h ago 2h 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:55:48.935 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 2h ago 2h - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:55:48.935 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:55:48.936 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 2h ago 2h 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:55:48.936 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:55:48.936 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 2h ago 2h - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:55:48.936 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 2h ago 2h 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:55:48.937 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:55:48.937 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:55:48.937 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:55:48.937 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:55:48.937 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 2h ago 2h 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:55:48.938 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 2h ago 2h 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:55:48.938 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 2h ago 2h 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:55:48.938 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 2h ago 2h 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:55:48.939 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 2h ago 2h 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:55:49.269 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:55:49.269 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:55:49.270 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:55:49.270 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:55:49.270 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:55:49.270 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:55:49.271 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:55:49.271 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:55:49.271 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:55:49.271 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:55:49.271 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:55:49.272 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:55:49.272 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:55:49.272 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:55:49.272 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:55:49.272 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:55:49.273 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:55:49.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:49 smithi154 conmon[34553]: cluster 2021-07-31T03:55:47.632907+0000 mgr.x (mgr.24542) 5512 : cluster [DBG] pgmap v5865: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:49.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:49 smithi045 conmon[33655]: cluster 2021-07-31T03:55:47. 2021-07-31T03:55:49.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:49 smithi045 conmon[33655]: 632907+0000 mgr.x (mgr.24542) 5512 : cluster [DBG] pgmap v5865: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:49.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:49 smithi045 conmon[39718]: cluster 2021-07-31T03:55:47.632907+0000 2021-07-31T03:55:49.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:49 smithi045 conmon[39718]: mgr.x (mgr.24542) 5512 : cluster [DBG] pgmap v5865: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:50.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:50 smithi154 conmon[34553]: audit 2021-07-31T03:55:48.645693+0000 mgr.x (mgr.24542) 5513 : audit [DBG] from='client.20532 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:55:50.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:50 smithi154 conmon[34553]: audit 2021-07-31T03:55:48.926670+0000 mgr.x (mgr.24542) 5514 : audit [DBG] from='client.30475 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:55:50.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:50 smithi154 conmon[34553]: audit 2021-07-31T03:55:49.270448+0000 mon.c (mon.1) 235 : audit [DBG] from='client.? 172.21.15.45:0/2553263497' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:55:50.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:50 smithi045 conmon[39718]: audit 2021-07-31T03:55:48.645693+0000 2021-07-31T03:55:50.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:50 smithi045 conmon[39718]: mgr.x (mgr.24542) 5513 : audit [DBG] from='client.20532 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:55:50.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:50 smithi045 conmon[39718]: audit 2021 2021-07-31T03:55:50.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:50 smithi045 conmon[39718]: -07-31T03:55:48. 2021-07-31T03:55:50.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:50 smithi045 conmon[39718]: 926670+0000 mgr.x (mgr.24542) 5514 : 2021-07-31T03:55:50.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:50 smithi045 conmon[39718]: audit [DBG] from='client.30475 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:55:50.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:50 smithi045 conmon[39718]: audit 2021 2021-07-31T03:55:50.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:50 smithi045 conmon[39718]: -07-31T03:55:49. 2021-07-31T03:55:50.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:50 smithi045 conmon[39718]: 270448+0000 mon.c (mon.1) 235 : 2021-07-31T03:55:50.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:50 smithi045 conmon[39718]: audit [DBG] from='client.? 172.21.15.45:0/2553263497' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:55:50.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:50 smithi045 conmon[33655]: audit 2021-07-31T03:55:48.645693+0000 mgr.x (mgr.24542) 5513 : audit 2021-07-31T03:55:50.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:50 smithi045 conmon[33655]: [DBG] from='client.20532 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:55:50.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:50 smithi045 conmon[33655]: audit 2021-07-31T03:55:48.926670+0000 mgr.x (mgr.24542) 5514 : audit [DBG] from='client.30475 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:55:50.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:50 smithi045 conmon[33655]: audit 2021- 2021-07-31T03:55:50.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:50 smithi045 conmon[33655]: 07-31T03:55:49.270448+0000 mon.c (mon.1) 235 : audit [DBG] 2021-07-31T03:55:50.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:50 smithi045 conmon[33655]: from='client.? 172.21.15.45:0/2553263497' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:55:51.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:51 smithi154 conmon[34553]: cluster 2021-07-31T03:55:49.633445+0000 mgr.x (mgr.24542) 5515 : cluster [DBG] pgmap v5866: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:51.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:51 smithi045 conmon[33655]: cluster 2021-07-31T03:55:49.633445+0000 mgr.x (mgr.24542) 5515 : 2021-07-31T03:55:51.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:51 smithi045 conmon[33655]: cluster [DBG] pgmap v5866: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:51.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:51 smithi045 conmon[39718]: cluster 2021-07-31T03:55:49.633445+0000 mgr.x 2021-07-31T03:55:51.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:51 smithi045 conmon[39718]: (mgr.24542) 5515 : cluster [DBG] pgmap v5866: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:53.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:53 smithi154 conmon[34553]: cluster 2021-07-31T03:55:51.634010+0000 mgr.x (mgr.24542) 5516 : cluster [DBG] pgmap v5867: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:53.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:53 smithi045 conmon[39718]: cluster 2021-07-31T03:55: 2021-07-31T03:55:53.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:53 smithi045 conmon[39718]: 51.634010+0000 mgr.x (mgr.24542) 5516 : cluster [DBG] pgmap v5867: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:53.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:53 smithi045 conmon[33655]: cluster 2021-07-31T03:55:51.634010+0000 mgr.x (mgr. 2021-07-31T03:55:53.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:53 smithi045 conmon[33655]: 24542) 5516 : cluster [DBG] pgmap v5867: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:55.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:55 smithi045 conmon[39718]: cluster 2021-07-31T03:55:53.634493+0000 mgr.x (mgr.24542) 5517 2021-07-31T03:55:55.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:55 smithi045 conmon[39718]: : cluster [DBG] pgmap v5868: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:55.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:55 smithi045 conmon[33655]: cluster 2021-07-31T03:55:53.634493+0000 mgr.x (mgr.24542) 5517 : cluster 2021-07-31T03:55:55.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:55 smithi045 conmon[33655]: [DBG] pgmap v5868: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:55.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:55 smithi154 conmon[34553]: cluster 2021-07-31T03:55:53.634493+0000 mgr.x (mgr.24542) 5517 : cluster [DBG] pgmap v5868: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:57.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:57 smithi045 conmon[39718]: cluster 2021-07-31T03:55:55.635002+0000 mgr.x (mgr.24542) 5518 : cluster [DBG] pgmap v5869: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:57.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:57 smithi045 conmon[33655]: cluster 2021-07-31T03:55:55.635002+0000 mgr.x (mgr.24542) 5518 : cluster [DBG] pgmap v5869: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:57.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:57 smithi154 conmon[34553]: cluster 2021-07-31T03:55:55.635002+0000 mgr.x (mgr.24542) 5518 : cluster [DBG] pgmap v5869: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:59.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:59 smithi045 conmon[33655]: cluster 2021-07-31T03:55:57.635530 2021-07-31T03:55:59.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:55:59 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 5519 : cluster [DBG] pgmap v5870: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:59.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:59 smithi045 conmon[39718]: cluster 2021-07-31T03:55:57 2021-07-31T03:55:59.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:55:59 smithi045 conmon[39718]: .635530+0000 mgr.x (mgr.24542) 5519 : cluster [DBG] pgmap v5870: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:55:59.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:55:59 smithi154 conmon[34553]: cluster 2021-07-31T03:55:57.635530+0000 mgr.x (mgr.24542) 5519 : cluster [DBG] pgmap v5870: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:01.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:01 smithi045 conmon[39718]: cluster 2021-07-31T03:55:59.636031+0000 mgr.x ( 2021-07-31T03:56:01.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:01 smithi045 conmon[39718]: mgr.24542) 5520 : cluster [DBG] pgmap v5871: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:01.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:01 smithi045 conmon[33655]: cluster 2021-07-31T03:55:59.636031+0000 mgr.x (mgr.24542) 5520 : cluster [DBG] pgmap v5871: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:01.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:01 smithi154 conmon[34553]: cluster 2021-07-31T03:55:59.636031+0000 mgr.x (mgr.24542) 5520 : cluster [DBG] pgmap v5871: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:03.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:03 smithi045 conmon[33655]: cluster 2021-07-31T03:56:01.636553+0000 mgr.x (mgr.24542) 5521 : 2021-07-31T03:56:03.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:03 smithi045 conmon[33655]: cluster [DBG] pgmap v5872: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:03.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:03 smithi045 conmon[39718]: cluster 2021-07-31T03:56:01.636553+0000 mgr.x (mgr. 2021-07-31T03:56:03.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:03 smithi045 conmon[39718]: 24542) 5521 : cluster [DBG] pgmap v5872: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:03.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:03 smithi154 conmon[34553]: cluster 2021-07-31T03:56:01.636553+0000 mgr.x (mgr.24542) 5521 : cluster [DBG] pgmap v5872: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:05.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:04 smithi154 conmon[34553]: cluster 2021-07-31T03:56:03.637207+0000 mgr.x (mgr.24542) 5522 : cluster [DBG] pgmap v5873: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:05.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:04 smithi045 conmon[39718]: cluster 2021-07-31T03:56:03.637207+0000 mgr.x (mgr.24542) 5522 : cluster [DBG] pgmap v5873: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:05.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:04 smithi045 conmon[33655]: cluster 2021-07-31T03:56:03.637207+0000 mgr.x (mgr.24542) 5522 : cluster [DBG] 2021-07-31T03:56:05.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:04 smithi045 conmon[33655]: pgmap v5873: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:07.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:06 smithi154 conmon[34553]: cluster 2021-07-31T03:56:05.637777+0000 mgr.x (mgr.24542) 5523 : cluster [DBG] pgmap v5874: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:07.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:06 smithi154 conmon[34553]: 2021-07-31T03:56:07.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:06 smithi045 conmon[33655]: cluster 2021-07-31T03:56:05.637777+0000 mgr.x (mgr.24542) 5523 : cluster [DBG] pgmap v5874: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:07.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:06 smithi045 conmon[39718]: cluster 2021-07-31T03:56:05.637777+0000 mgr.x (mgr 2021-07-31T03:56:07.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:06 smithi045 conmon[39718]: .24542) 5523 : cluster [DBG] pgmap v5874: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:09.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:08 smithi154 conmon[34553]: cluster 2021-07-31T03:56:07.638165+0000 mgr.x (mgr.24542) 5524 : cluster [DBG] pgmap v5875: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:09.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:08 smithi045 conmon[39718]: cluster 2021-07-31T03:56:07.638165+0000 mgr.x (mgr.24542) 5524 : cluster [DBG] pgmap v5875: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:09.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:08 smithi045 conmon[33655]: cluster 2021-07-31T03:56:07.638165+0000 mgr.x (mgr.24542) 2021-07-31T03:56:09.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:08 smithi045 conmon[33655]: 5524 : cluster [DBG] pgmap v5875: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:11.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:10 smithi154 conmon[34553]: cluster 2021-07-31T03:56:09.638713+0000 mgr.x (mgr.24542) 5525 : cluster [DBG] pgmap v5876: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:11.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:10 smithi154 conmon[34553]: 2021-07-31T03:56:11.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:10 smithi045 conmon[39718]: cluster 2021-07-31T03:56:09. 2021-07-31T03:56:11.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:10 smithi045 conmon[39718]: 638713+0000 mgr.x (mgr.24542) 5525 : cluster [DBG] pgmap v5876: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:11.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:10 smithi045 conmon[33655]: cluster 2021-07-31T03:56:09.638713+0000 mgr.x (mgr.24542) 5525 : cluster [DBG] pgmap v5876: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:13.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:12 smithi154 conmon[34553]: cluster 2021-07-31T03:56:11.639265+0000 mgr.x (mgr.24542) 5526 : cluster [DBG] 2021-07-31T03:56:13.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:12 smithi154 conmon[34553]: pgmap v5877: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:13.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:12 smithi045 conmon[39718]: cluster 2021-07-31T03:56:11.639265+0000 mgr.x ( 2021-07-31T03:56:13.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:12 smithi045 conmon[39718]: mgr.24542) 5526 : cluster [DBG] pgmap v5877: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:13.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:12 smithi045 conmon[33655]: cluster 2021-07-31T03:56:11.639265+0000 mgr.x (mgr.24542) 5526 : cluster [DBG] 2021-07-31T03:56:13.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:12 smithi045 conmon[33655]: pgmap v5877: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:15.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:14 smithi154 conmon[34553]: cluster 2021-07-31T03:56:13.639828+0000 mgr.x (mgr.24542) 5527 : cluster [DBG] 2021-07-31T03:56:15.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:14 smithi154 conmon[34553]: pgmap v5878: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:15.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:14 smithi045 conmon[39718]: cluster 2021-07-31T03:56:13.639828+0000 mgr.x (mgr.24542) 5527 : cluster [DBG] pgmap v5878: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:15.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:14 smithi045 conmon[33655]: cluster 2021-07-31T03:56:13.639828+0000 mgr.x (mgr.24542) 5527 : cluster [DBG] pgmap v5878: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:17.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:16 smithi154 conmon[34553]: cluster 2021-07-31T03:56:15.640344+0000 mgr.x (mgr.24542) 5528 : cluster [DBG] pgmap v5879: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:17.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:16 smithi045 conmon[39718]: cluster 2021-07-31T03:56:15. 2021-07-31T03:56:17.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:16 smithi045 conmon[39718]: 640344+0000 mgr.x (mgr.24542) 5528 : cluster [DBG] pgmap v5879: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:17.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:16 smithi045 conmon[33655]: cluster 2021-07-31T03:56:15.640344+0000 mgr.x (mgr.24542) 5528 : cluster [DBG] pgmap v5879: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:19.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:18 smithi154 conmon[34553]: cluster 2021-07-31T03:56:17.640734+0000 mgr.x (mgr.24542) 5529 : cluster [DBG] 2021-07-31T03:56:19.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:18 smithi154 conmon[34553]: pgmap v5880: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:19.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:18 smithi045 conmon[33655]: cluster 2021-07-31T03:56:17.640734+0000 mgr.x (mgr.24542 2021-07-31T03:56:19.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:18 smithi045 conmon[33655]: ) 5529 : cluster [DBG] pgmap v5880: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:19.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:18 smithi045 conmon[39718]: cluster 2021-07-31T03:56:17.640734+0000 mgr.x (mgr.24542) 5529 2021-07-31T03:56:19.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:18 smithi045 conmon[39718]: : cluster [DBG] pgmap v5880: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:19.583 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:56:19.861 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:56:19.862 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 2h ago 2h - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:56:19.862 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 2h ago 2h 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:56:19.862 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 2h ago 2h 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:56:19.862 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 2h ago 2h - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:56:19.863 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:56:19.863 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 2h ago 2h 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:56:19.863 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:56:19.863 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 2h ago 2h - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:56:19.863 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 2h ago 2h 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:56:19.864 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:56:19.864 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:56:19.864 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:56:19.864 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:56:19.864 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 2h ago 2h 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:56:19.865 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 2h ago 2h 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:56:19.865 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 2h ago 2h 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:56:19.865 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 2h ago 2h 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:56:19.865 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 2h ago 2h 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:56:20.198 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:56:20.198 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:56:20.199 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:56:20.199 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:56:20.199 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:56:20.200 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:56:20.200 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:56:20.200 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:56:20.200 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:56:20.201 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:56:20.201 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:56:20.201 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:56:20.202 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:56:20.202 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:56:20.202 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:56:20.202 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:56:20.203 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:56:21.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:20 smithi154 conmon[34553]: audit 2021-07-31T03:56:19.571581+0000 mgr.x (mgr.24542) 5530 : audit [DBG] from='client.30487 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:56:21.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:20 smithi154 conmon[34553]: cluster 2021-07-31T03:56:19.641240+0000 mgr.x (mgr.24542) 5531 : cluster [DBG] pgmap v5881: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:21.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:20 smithi154 conmon[34553]: audit 2021-07-31T03:56:19.854032+0000 mgr.x (mgr.24542) 5532 : audit [DBG] from='client.20556 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:56:21.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:20 smithi154 conmon[34553]: audit 2021-07-31T03:56:21.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:20 smithi154 conmon[34553]: 2021-07-31T03:56:20.198661+0000 mon.c (mon.1) 236 : audit [DBG] from='client.? 172.21.15.45:0/1033510976' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:56:21.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:20 smithi045 conmon[39718]: audit 2021-07-31T03:56:19.571581+0000 mgr.x (mgr.24542 2021-07-31T03:56:21.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:20 smithi045 conmon[39718]: ) 5530 : audit [DBG] from='client.30487 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:56:21.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:20 smithi045 conmon[39718]: cluster 2021-07-31T03:56:19. 2021-07-31T03:56:21.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:20 smithi045 conmon[39718]: 641240+0000 mgr.x (mgr.24542) 5531 : cluster [DBG] pgmap v5881: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:21.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:20 smithi045 conmon[39718]: audit 2021-07-31T03:56:19.854032+0000 2021-07-31T03:56:21.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:20 smithi045 conmon[39718]: mgr.x (mgr.24542) 5532 : audit [DBG] from='client.20556 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:56:21.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:20 smithi045 conmon[39718]: audit 2021- 2021-07-31T03:56:21.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:20 smithi045 conmon[39718]: 07-31T03:56:20.198661+0000 mon.c (mon.1) 236 : 2021-07-31T03:56:21.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:20 smithi045 conmon[39718]: audit [DBG] from='client.? 172.21.15.45:0/1033510976' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:56:21.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:20 smithi045 conmon[33655]: audit 2021-07-31T03:56:19.571581+0000 mgr.x (mgr.24542) 5530 : audit [DBG] from='client.30487 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:56:21.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:20 smithi045 conmon[33655]: cluster 2021-07-31T03:56:19.641240+0000 mgr.x (mgr.24542) 5531 : cluster [DBG] pgmap v5881: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:21.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:20 smithi045 conmon[33655]: audit 2021- 2021-07-31T03:56:21.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:20 smithi045 conmon[33655]: 07-31T03:56:19.854032+0000 mgr.x (mgr.24542) 5532 : audit [DBG] from='client.20556 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:56:21.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:20 smithi045 conmon[33655]: audit 2021-07-31T03:56:21.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:20 smithi045 conmon[33655]: 2021-07- 2021-07-31T03:56:21.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:20 smithi045 conmon[33655]: 31T03:56:20.198661 2021-07-31T03:56:21.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:20 smithi045 conmon[33655]: +0000 mon.c (mon.1) 2021-07-31T03:56:21.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:20 smithi045 conmon[33655]: 236 : audit [DBG] from='client.? 172.21.15.45:0/1033510976' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:56:23.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:22 smithi154 conmon[34553]: cluster 2021-07-31T03:56:21.641673+0000 mgr.x (mgr.24542) 5533 : cluster [DBG] pgmap v5882: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:23.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:22 smithi045 conmon[33655]: cluster 2021-07-31T03:56:21.641673+0000 mgr.x (mgr.24542) 2021-07-31T03:56:23.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:22 smithi045 conmon[33655]: 5533 : cluster [DBG] pgmap v5882: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:23.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:22 smithi045 conmon[39718]: cluster 2021-07-31T03:56:21.641673+0000 mgr.x 2021-07-31T03:56:23.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:22 smithi045 conmon[39718]: (mgr.24542) 5533 : cluster [DBG] pgmap v5882: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:25.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:24 smithi154 conmon[34553]: cluster 2021-07-31T03:56:23.642075+0000 mgr.x (mgr.24542) 5534 : cluster [DBG] pgmap v5883: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:25.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:24 smithi154 conmon[34553]: 2021-07-31T03:56:25.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:24 smithi045 conmon[39718]: cluster 2021-07-31T03:56:23.642075+0000 mgr.x ( 2021-07-31T03:56:25.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:24 smithi045 conmon[39718]: mgr.24542) 5534 : cluster [DBG] pgmap v5883: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:25.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:24 smithi045 conmon[33655]: cluster 2021-07-31T03:56:23.642075+0000 mgr.x (mgr.24542) 5534 : cluster [DBG] pgmap v5883: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:25.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:24 smithi045 conmon[33655]: 2021-07-31T03:56:27.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:26 smithi154 conmon[34553]: cluster 2021-07-31T03:56:25.642637+0000 mgr.x (mgr.24542) 5535 : cluster 2021-07-31T03:56:27.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:26 smithi154 conmon[34553]: [DBG] pgmap v5884: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:27.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:26 smithi045 conmon[33655]: cluster 2021-07-31T03:56:25.642637+0000 mgr.x (mgr.24542) 5535 : cluster [DBG] pgmap v5884: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:27.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:26 smithi045 conmon[33655]: 2021-07-31T03:56:27.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:26 smithi045 conmon[39718]: cluster 2021-07-31T03:56:25.642637+0000 mgr.x ( 2021-07-31T03:56:27.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:26 smithi045 conmon[39718]: mgr.24542) 5535 : cluster [DBG] pgmap v5884: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:29.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:28 smithi154 conmon[34553]: cluster 2021-07-31T03:56:27.643102+0000 mgr.x (mgr.24542) 5536 : cluster [DBG] 2021-07-31T03:56:29.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:28 smithi154 conmon[34553]: pgmap v5885: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:28 smithi154 conmon[34553]: audit 2021-07-31T03:56:28.446438+0000 mon.b (mon.2) 720 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:56:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:28 smithi154 conmon[34553]: audit 2021-07-31T03:56:28.446734+0000 mon.b (mon.2) 721 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:56:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:28 smithi154 conmon[34553]: 2021-07-31T03:56:29.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:28 smithi154 conmon[34553]: audit 2021-07-31T03:56:28.447397+0000 mon.a (mon.0) 3394 : 2021-07-31T03:56:29.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:28 smithi154 conmon[34553]: audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:56:29.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:28 smithi154 conmon[34553]: audit 2021-07-31T03:56: 2021-07-31T03:56:29.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:28 smithi154 conmon[34553]: 28.447772+0000 mon.a (mon.0) 3395 : 2021-07-31T03:56:29.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:28 smithi154 conmon[34553]: audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:56:29.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:28 smithi045 conmon[33655]: cluster 2021-07-31T03:56:27.643102+0000 mgr.x (mgr.24542) 5536 : cluster [DBG] pgmap v5885: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:29.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:28 smithi045 conmon[33655]: audit 2021-07-31T03:56:28.446438+0000 mon.b (mon.2) 720 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:56:29.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:28 smithi045 conmon[33655]: audit 2021-07-31T03:56:28 2021-07-31T03:56:29.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:28 smithi045 conmon[33655]: .446734+0000 mon.b (mon.2) 721 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:56:29.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:28 smithi045 conmon[33655]: audit 2021-07-31T 2021-07-31T03:56:29.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:28 smithi045 conmon[33655]: 03:56:28.447397+0000 mon.a (mon 2021-07-31T03:56:29.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:28 smithi045 conmon[33655]: .0) 3394 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:56:29.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:28 smithi045 conmon[33655]: audit 2021-07-31T03:56:28.447772 2021-07-31T03:56:29.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:28 smithi045 conmon[33655]: +0000 mon.a (mon.0) 3395 : audit 2021-07-31T03:56:29.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:28 smithi045 conmon[33655]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:56:29.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:28 smithi045 conmon[39718]: cluster 2021-07-31T03:56:27 2021-07-31T03:56:29.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:28 smithi045 conmon[39718]: .643102+0000 mgr.x (mgr.24542) 5536 : cluster [DBG] pgmap v5885: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:29.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:28 smithi045 conmon[39718]: audit 2021 2021-07-31T03:56:29.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:28 smithi045 conmon[39718]: -07-31T03:56:28.446438+0000 mon.b ( 2021-07-31T03:56:29.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:28 smithi045 conmon[39718]: mon.2) 720 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:56:29.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:28 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T03:56:29.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:28 smithi045 conmon[39718]: 03:56:28.446734+0000 mon.b (mon.2) 721 : audit 2021-07-31T03:56:29.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:28 smithi045 conmon[39718]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:56:29.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:28 smithi045 conmon[39718]: audit 2021-07-31T03:56:28.447397+0000 mon.a (mon.0) 3394 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:56:29.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:28 smithi045 conmon[39718]: audit 2021-07-31T03:56:28 2021-07-31T03:56:29.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:28 smithi045 conmon[39718]: .447772+0000 mon.a (mon.0) 2021-07-31T03:56:29.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:28 smithi045 conmon[39718]: 3395 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:56:31.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:30 smithi154 conmon[34553]: cluster 2021-07-31T03:56:29.643618+0000 mgr.x (mgr.24542) 5537 : cluster [DBG] pgmap v5886: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:31.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:30 smithi045 conmon[33655]: cluster 2021-07-31T03:56:29.643618+0000 mgr.x (mgr.24542 2021-07-31T03:56:31.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:30 smithi045 conmon[33655]: ) 5537 : cluster [DBG] pgmap v5886: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:31.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:30 smithi045 conmon[39718]: cluster 2021-07-31T03:56:29.643618+0000 2021-07-31T03:56:31.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:30 smithi045 conmon[39718]: mgr.x (mgr.24542) 5537 : cluster [DBG] pgmap v5886: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:33.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:32 smithi154 conmon[34553]: cluster 2021-07-31T03:56:31.644061+0000 mgr.x (mgr.24542) 5538 : cluster [DBG] pgmap v5887: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:33.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:32 smithi045 conmon[33655]: cluster 2021-07-31T03:56:31.644061+0000 mgr.x (mgr 2021-07-31T03:56:33.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:32 smithi045 conmon[33655]: .24542) 5538 : cluster [DBG] pgmap v5887: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:33.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:32 smithi045 conmon[39718]: cluster 2021-07-31T03:56:31.644061+0000 mgr.x ( 2021-07-31T03:56:33.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:32 smithi045 conmon[39718]: mgr.24542) 5538 : cluster [DBG] pgmap v5887: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:35.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:34 smithi154 conmon[34553]: cluster 2021-07-31T03:56:33.644579+0000 mgr.x (mgr.24542) 5539 : cluster [DBG] pgmap v5888: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:35.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:34 smithi045 conmon[33655]: cluster 2021-07-31T03:56:33.644579+0000 mgr.x ( 2021-07-31T03:56:35.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:34 smithi045 conmon[33655]: mgr.24542) 5539 : cluster [DBG] pgmap v5888: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:35.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:34 smithi045 conmon[39718]: cluster 2021-07-31T03:56:33.644579+0000 mgr.x (mgr.24542) 5539 : cluster [DBG] 2021-07-31T03:56:35.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:34 smithi045 conmon[39718]: pgmap v5888: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:36.253 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: debug 2021-07-31T03:56:36.008+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:56:36.253 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:56:36.253 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:56:36.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: yield 2021-07-31T03:56:36.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:56:36.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:56:36.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:56:36.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:56:36.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:56:36.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:56:36.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:56:36.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:56:36.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:56:36.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: task.result() 2021-07-31T03:56:36.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:56:36.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: debug 2021-07-31T03:56:36.020+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:56:36.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:56:36.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:56:36.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: yield 2021-07-31T03:56:36.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:56:36.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:56:36.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:56:36.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:56:36.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:56:36.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:56:36.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:56:36.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:56:36.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:56:36.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: task.result() 2021-07-31T03:56:36.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:56:36.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: debug 2021-07-31T03:56:36.046+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:56:36.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:56:36.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:56:36.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: yield 2021-07-31T03:56:36.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:56:36.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:56:36.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:56:36.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:56:36.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:56:36.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:56:36.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:56:36.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:56:36.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:56:36.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: task.result() 2021-07-31T03:56:36.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:56:36.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: debug 2021-07-31T03:56:36.047+0000 7fcb97969700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:56:36.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:56:36.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:56:36.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: yield 2021-07-31T03:56:36.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:56:36.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:56:36.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:56:36.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:56:36.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:56:36.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:56:36.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:56:36.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:56:36.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:56:36.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: task.result() 2021-07-31T03:56:36.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:56:36.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: 2021-07-31T03:56:36.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:56:36.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: 2021-07-31T03:56:36.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:56:36.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:56:36.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:56:36.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:56:36.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:56:36.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:56:36.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:56:36.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:56:36.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:56:36.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:56:36.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:56:36.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:56:36.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:56:36.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:56:36.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:56:36.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:56:36.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:56:36.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:56:36.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: raise 2021-07-31T03:56:36.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:56:36.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:56:36.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:56:36.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:56:36.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:56:36.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: debug 2021-07-31T03:56:36.055+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:56:36.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:56:36.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:56:36.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: yield 2021-07-31T03:56:36.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:56:36.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:56:36.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:56:36.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:56:36.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:56:36.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:56:36.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:56:36.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:56:36.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:56:36.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: task.result() 2021-07-31T03:56:36.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:56:36.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: debug 2021-07-31T03:56:36.056+0000 7fcb9816a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:56:36.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:56:36.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:56:36.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: yield 2021-07-31T03:56:36.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:56:36.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:56:36.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:56:36.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:56:36.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:56:36.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:56:36.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:56:36.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:56:36.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:56:36.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: task.result() 2021-07-31T03:56:36.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:56:36.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: 2021-07-31T03:56:36.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:56:36.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: 2021-07-31T03:56:36.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:56:36.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:56:36.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:56:36.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:56:36.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:56:36.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:56:36.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:56:36.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:56:36.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:56:36.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:56:36.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:56:36.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:56:36.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:56:36.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:56:36.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:56:36.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:56:36.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:56:36.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:56:36.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: raise 2021-07-31T03:56:36.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:56:36.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:56:36.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:56:36.284 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:56:36.284 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:56:37.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: cluster 2021-07-31T03:56:35.645226+0000 2021-07-31T03:56:37.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: mgr.x (mgr.24542) 5540 : cluster [DBG] pgmap v5889: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:37.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: audit 2021-07-31T03:56:35.963216+0000 mon.b (mon.2) 722 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:56:37.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: audit 2021 2021-07-31T03:56:37.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: -07-31T03:56:35.964838+0000 mon.b (mon.2) 723 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:56:37.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: audit 2021-07- 2021-07-31T03:56:37.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: 31T03:56:35.965678+0000 mon.b (mon.2) 724 : 2021-07-31T03:56:37.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:56:37.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: cephadm 2021-07- 2021-07-31T03:56:37.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: 31T03:56:35.966448+0000 mgr.x (mgr.24542) 5541 : 2021-07-31T03:56:37.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:56:37.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: cephadm 2021-07- 2021-07-31T03:56:37.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: 31T03:56:36.009783+0000 mgr.x (mgr.24542) 2021-07-31T03:56:37.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: 5542 : cephadm [ERR] 2021-07-31T03:56:37.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:56:37.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:56:37.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: yield 2021-07-31T03:56:37.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:56:37.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:56:37.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:56:37.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:56:37.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:56:37.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:56:37.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:56:37.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:56:37.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:56:37.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: task.result() 2021-07-31T03:56:37.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:56:37.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: cephadm 2021-07-31T03:56:36.010306+0000 mgr.x (mgr.24542) 5543 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:56:37.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: cephadm 2021-07-31T03:56:37.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: 2021-07-31T03 2021-07-31T03:56:37.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: :56:36.021372+0000 mgr.x (mgr.24542) 5544 2021-07-31T03:56:37.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: : cephadm [ERR] 2021-07-31T03:56:37.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:56:37.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:56:37.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: yield 2021-07-31T03:56:37.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:56:37.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:56:37.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:56:37.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:56:37.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:56:37.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:56:37.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:56:37.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:56:37.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:56:37.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: task.result() 2021-07-31T03:56:37.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:56:37.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: cephadm 2021-07-31T03:56:36.022482+0000 mgr.x (mgr.24542) 5545 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:56:37.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: cephadm 2021-07- 2021-07-31T03:56:37.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: 31T03:56:36.047671+0000 mgr.x (mgr.24542) 5546 : 2021-07-31T03:56:37.012 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: cephadm [ERR] 2021-07-31T03:56:37.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:56:37.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:56:37.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: yield 2021-07-31T03:56:37.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:56:37.013 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:56:37.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:56:37.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:56:37.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:56:37.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:56:37.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:56:37.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:56:37.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:56:37.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: task.result() 2021-07-31T03:56:37.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:56:37.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: cephadm 2021-07-31T03:56:36.048286+0000 mgr.x (mgr.24542) 5547 : cephadm 2021-07-31T03:56:37.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:56:37.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:56:37.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:56:37.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: yield 2021-07-31T03:56:37.016 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:56:37.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:56:37.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:56:37.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:56:37.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:56:37.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:56:37.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:56:37.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:56:37.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:56:37.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: task.result() 2021-07-31T03:56:37.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:56:37.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: 2021-07-31T03:56:37.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:56:37.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: 2021-07-31T03:56:37.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:56:37.019 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:56:37.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:56:37.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:56:37.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:56:37.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:56:37.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:56:37.020 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:56:37.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:56:37.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:56:37.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:56:37.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:56:37.021 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:56:37.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:56:37.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:56:37.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:56:37.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:56:37.022 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:56:37.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: raise 2021-07-31T03:56:37.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:56:37.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:56:37.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:56:37.023 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:56:37.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:56:37.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: cephadm 2021-07-31T03:56:36.056628+0000 mgr.x (mgr.24542) 5548 : cephadm [ERR] 2021-07-31T03:56:37.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:56:37.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:56:37.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: yield 2021-07-31T03:56:37.024 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:56:37.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:56:37.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:56:37.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:56:37.025 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:56:37.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:56:37.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:56:37.026 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:56:37.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: cluster 2021-07-31T03:56:35.645226+0000 mgr.x (mgr.24542) 5540 : cluster [DBG] pgmap v5889: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:37.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: 2021-07-31T03:56:37.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: audit 2021-07-31T03:56:35.963216+0000 mon.b (mon.2) 722 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:56:37.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: audit 2021-07-31T03:56:37.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: 2021-07-31T03:56:35.964838+0000 mon.b (mon.2) 723 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:56:37.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: audit 2021-07-31 2021-07-31T03:56:37.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: T03:56:35.965678+0000 mon.b 2021-07-31T03:56:37.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: (mon.2) 724 : audit 2021-07-31T03:56:37.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:56:37.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: cephadm 2021 2021-07-31T03:56:37.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: -07-31T03:56:35.966448 2021-07-31T03:56:37.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542 2021-07-31T03:56:37.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: ) 5541 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:56:37.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: 2021-07-31T03:56:37.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: cephadm 2021-07-31T03:56:37.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: 2021-07-31 2021-07-31T03:56:37.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: T03:56:36.009783+0000 2021-07-31T03:56:37.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: mgr.x (mgr.24542) 2021-07-31T03:56:37.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: 5542 : cephadm [ERR] 2021-07-31T03:56:37.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:56:37.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:56:37.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: yield 2021-07-31T03:56:37.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:56:37.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:56:37.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:56:37.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:56:37.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:56:37.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:56:37.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:56:37.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:56:37.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:56:37.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: task.result() 2021-07-31T03:56:37.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:56:37.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: cephadm 2021-07-31T03:56 2021-07-31T03:56:37.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: :36.010306+0000 mgr.x (mgr.24542) 5543 : cephadm 2021-07-31T03:56:37.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:56:37.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: cephadm 2021-07 2021-07-31T03:56:37.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: -31T03:56:36.021372+0000 mgr.x (mgr 2021-07-31T03:56:37.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: .24542) 5544 : cephadm [ERR] 2021-07-31T03:56:37.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:56:37.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:56:37.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: yield 2021-07-31T03:56:37.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:56:37.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: cluster 2021-07-31T03:56:35 2021-07-31T03:56:37.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: .645226+0000 mgr.x (mgr.24542) 5540 : cluster [DBG] pgmap v5889: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:37.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: audit 2021-07-31T03:56: 2021-07-31T03:56:37.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: 35.963216+0000 mon.b (mon.2) 722 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:56:37.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: audit 2021-07-31T03:56:35.964838+0000 2021-07-31T03:56:37.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: mon.b (mon.2) 723 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:56:37.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: audit 2021- 2021-07-31T03:56:37.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: 07-31T03:56:35. 2021-07-31T03:56:37.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: 965678+0000 mon.b (mon.2 2021-07-31T03:56:37.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: ) 724 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:56:37.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: 2021-07-31T03:56:37.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: cephadm 2021-07-31T03:56:37.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: 2021-07-31T03: 2021-07-31T03:56:37.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: 56:35.966448+0000 mgr.x 2021-07-31T03:56:37.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: (mgr.24542) 5541 : cephadm 2021-07-31T03:56:37.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:56:37.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: cephadm 2021 2021-07-31T03:56:37.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: -07-31T03:56:36 2021-07-31T03:56:37.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: .009783+0000 mgr.x (mgr. 2021-07-31T03:56:37.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: 24542) 5542 : cephadm [ERR] 2021-07-31T03:56:37.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:56:37.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:56:37.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: yield 2021-07-31T03:56:37.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:56:37.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:56:37.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:56:37.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:56:37.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:56:37.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:56:37.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:56:37.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:56:37.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:56:37.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: task.result() 2021-07-31T03:56:37.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:56:37.042 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: 2021-07-31T03:56:37.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: cephadm 2021-07-31T03:56:36.010306+0000 mgr.x (mgr.24542) 5543 2021-07-31T03:56:37.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:56:37.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: cephadm 2021- 2021-07-31T03:56:37.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: 07-31T03:56:36. 2021-07-31T03:56:37.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: 021372+0000 mgr.x (mgr.24542 2021-07-31T03:56:37.043 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: ) 5544 : cephadm [ERR] 2021-07-31T03:56:37.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:56:37.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:56:37.044 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: yield 2021-07-31T03:56:37.044 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:56:37.045 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: task.result() 2021-07-31T03:56:37.045 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:56:37.045 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: cephadm 2021-07-31T03:56:36.057179+0000 mgr.x (mgr.24542) 5549 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:56:37.045 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:56:37.045 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:56:37.045 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: yield 2021-07-31T03:56:37.046 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:56:37.046 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:56:37.046 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:56:37.046 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:56:37.046 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:56:37.047 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:56:37.047 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:56:37.047 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:56:37.047 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:56:37.047 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: task.result() 2021-07-31T03:56:37.048 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:56:37.048 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: 2021-07-31T03:56:37.048 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:56:37.048 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: 2021-07-31T03:56:37.048 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:56:37.049 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:56:37.049 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:56:37.049 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:56:37.049 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:56:37.049 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:56:37.049 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:56:37.050 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:56:37.050 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:56:37.050 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:56:37.050 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:56:37.050 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:56:37.051 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:56:37.051 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:56:37.051 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:56:37.051 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:56:37.051 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:56:37.052 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:56:37.052 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: raise 2021-07-31T03:56:37.052 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:56:37.052 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:56:37.052 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:56:37.053 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:56:37.053 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:36 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:56:37.053 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:56:37.053 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:56:37.054 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:56:37.054 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:56:37.054 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:56:37.054 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:56:37.054 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:56:37.055 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:56:37.055 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: task.result() 2021-07-31T03:56:37.055 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:56:37.055 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: cephadm 2021-07-31T03:56:36.022482+0000 mgr.x (mgr.24542) 5545 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:56:37.055 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: cephadm 2021 2021-07-31T03:56:37.056 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: -07-31T03:56:36. 2021-07-31T03:56:37.056 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: 047671+0000 mgr.x (mgr.24542 2021-07-31T03:56:37.056 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: ) 5546 : cephadm [ERR] 2021-07-31T03:56:37.056 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:56:37.056 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:56:37.057 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: yield 2021-07-31T03:56:37.057 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:56:37.057 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:56:37.057 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:56:37.057 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:56:37.057 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:56:37.058 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:56:37.058 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:56:37.058 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:56:37.058 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:56:37.058 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: task.result() 2021-07-31T03:56:37.059 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:56:37.059 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: 2021-07-31T03:56:37.059 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: cephadm 2021-07-31T03:56:36.048286+0000 2021-07-31T03:56:37.059 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: mgr.x (mgr.24542) 5547 : cephadm 2021-07-31T03:56:37.059 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:56:37.060 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:56:37.060 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:56:37.060 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: yield 2021-07-31T03:56:37.060 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:56:37.060 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:56:37.061 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:56:37.061 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:56:37.061 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:56:37.061 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:56:37.062 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:56:37.062 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:56:37.062 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:56:37.062 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:56:37.062 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:56:37.063 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:56:37.063 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:56:37.063 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:56:37.063 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: task.result() 2021-07-31T03:56:37.063 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:56:37.064 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: 2021-07-31T03:56:37.064 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: cephadm 2021-07-31T03:56:36.022482+0000 mgr.x (mgr.24542) 5545 : 2021-07-31T03:56:37.064 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:56:37.064 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: cephadm 2021 2021-07-31T03:56:37.064 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: -07-31T03:56:36.047671 2021-07-31T03:56:37.065 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 5546 2021-07-31T03:56:37.065 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: : cephadm [ERR] 2021-07-31T03:56:37.065 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:56:37.065 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:56:37.065 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: yield 2021-07-31T03:56:37.066 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:56:37.066 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:56:37.066 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:56:37.066 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:56:37.066 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:56:37.067 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:56:37.067 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:56:37.067 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:56:37.067 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:56:37.067 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: task.result() 2021-07-31T03:56:37.068 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:56:37.068 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: cephadm 2021-07-31T03:56:36.048286+0000 mgr.x (mgr.24542) 5547 2021-07-31T03:56:37.068 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:56:37.068 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:56:37.068 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:56:37.069 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: yield 2021-07-31T03:56:37.069 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:56:37.069 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:56:37.069 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:56:37.069 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:56:37.070 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:56:37.070 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:56:37.070 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:56:37.070 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:56:37.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: task.result() 2021-07-31T03:56:37.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:56:37.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: 2021-07-31T03:56:37.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:56:37.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: 2021-07-31T03:56:37.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:56:37.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:56:37.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:56:37.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:56:37.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:56:37.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:56:37.073 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:56:37.073 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:56:37.073 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:56:37.073 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:56:37.073 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:56:37.074 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:56:37.074 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:56:37.074 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:56:37.074 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:56:37.074 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:56:37.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:56:37.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:56:37.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: raise 2021-07-31T03:56:37.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:56:37.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:56:37.075 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:56:37.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:56:37.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:56:37.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: cephadm 2021-07-31T03:56:36.056628+0000 mgr.x (mgr.24542) 5548 : cephadm [ERR] 2021-07-31T03:56:37.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:56:37.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:56:37.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: yield 2021-07-31T03:56:37.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:56:37.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:56:37.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:56:37.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:56:37.078 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:56:37.078 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:56:37.078 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:56:37.078 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:56:37.079 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:56:37.079 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: task.result() 2021-07-31T03:56:37.079 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:56:37.079 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: 2021-07-31T03:56:37.079 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:56:37.080 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: 2021-07-31T03:56:37.080 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:56:37.080 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:56:37.080 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:56:37.080 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:56:37.081 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:56:37.081 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:56:37.081 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:56:37.081 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:56:37.081 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:56:37.081 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:56:37.082 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:56:37.082 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:56:37.082 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:56:37.082 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:56:37.082 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:56:37.083 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:56:37.083 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:56:37.083 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:56:37.083 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: raise 2021-07-31T03:56:37.083 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:56:37.084 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:56:37.084 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:56:37.084 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:56:37.084 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:56:37.084 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: cephadm 2021-07-31T03:56:36.056628+0000 mgr.x (mgr.24542) 5548 : cephadm [ERR] 2021-07-31T03:56:37.085 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:56:37.085 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:56:37.085 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: yield 2021-07-31T03:56:37.085 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:56:37.085 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:56:37.086 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:56:37.086 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:56:37.086 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:56:37.087 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:56:37.087 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:56:37.087 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:56:37.087 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:56:37.087 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: task.result() 2021-07-31T03:56:37.088 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:56:37.088 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: cephadm 2021-07-31T03:56:36.057179+0000 mgr.x (mgr.24542) 5549 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:56:37.088 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:56:37.088 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:56:37.088 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: yield 2021-07-31T03:56:37.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:56:37.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:56:37.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:56:37.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:56:37.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:56:37.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:56:37.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:56:37.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:56:37.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:56:37.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: task.result() 2021-07-31T03:56:37.090 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:56:37.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: 2021-07-31T03:56:37.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:56:37.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: 2021-07-31T03:56:37.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:56:37.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:56:37.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:56:37.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:56:37.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:56:37.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:56:37.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:56:37.093 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:56:37.093 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:56:37.093 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:56:37.093 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:56:37.093 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:56:37.093 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:56:37.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:56:37.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:56:37.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:56:37.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:56:37.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:56:37.095 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: raise 2021-07-31T03:56:37.095 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:56:37.095 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:56:37.095 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:56:37.095 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:56:37.096 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:56:37.096 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[33655]: 2021-07-31T03:56:37.096 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:56:37.096 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:56:37.097 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:56:37.097 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:56:37.097 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:56:37.097 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: task.result() 2021-07-31T03:56:37.097 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:56:37.098 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: cephadm 2021-07-31T03:56:36.057179+0000 mgr.x (mgr.24542) 5549 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:56:37.098 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:56:37.098 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:56:37.098 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: yield 2021-07-31T03:56:37.098 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:56:37.099 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:56:37.099 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:56:37.099 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:56:37.099 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:56:37.099 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:56:37.100 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:56:37.100 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:56:37.100 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:56:37.100 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: task.result() 2021-07-31T03:56:37.100 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:56:37.101 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: 2021-07-31T03:56:37.101 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:56:37.101 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: 2021-07-31T03:56:37.101 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:56:37.101 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:56:37.101 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:56:37.102 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:56:37.102 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:56:37.102 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:56:37.102 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:56:37.102 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:56:37.103 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:56:37.103 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:56:37.103 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:56:37.103 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:56:37.103 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:56:37.104 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:56:37.104 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:56:37.104 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:56:37.104 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:56:37.104 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:56:37.105 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: raise 2021-07-31T03:56:37.105 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:56:37.105 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:56:37.105 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:56:37.105 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:56:37.105 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:36 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:56:39.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:38 smithi154 conmon[34553]: cluster 2021-07-31T03:56:37.645577+0000 mgr.x (mgr.24542) 5550 : cluster [DBG] 2021-07-31T03:56:39.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:38 smithi154 conmon[34553]: pgmap v5890: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:39.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:38 smithi045 conmon[33655]: cluster 2021-07-31T03:56:37.645577+0000 mgr.x (mgr.24542) 5550 : cluster [DBG] pgmap v5890: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:39.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:38 smithi045 conmon[33655]: 2021-07-31T03:56:39.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:38 smithi045 conmon[39718]: cluster 2021-07-31T03:56:37.645577+0000 mgr.x (mgr.24542 2021-07-31T03:56:39.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:38 smithi045 conmon[39718]: ) 5550 : cluster [DBG] pgmap v5890: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:41.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:40 smithi154 conmon[34553]: cluster 2021-07-31T03:56:39.646165+0000 mgr.x (mgr.24542) 5551 : cluster [DBG] pgmap v5891: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:41.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:40 smithi045 conmon[33655]: cluster 2021-07-31T03:56:39.646165+0000 mgr.x (mgr.24542) 5551 : cluster [DBG] 2021-07-31T03:56:41.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:40 smithi045 conmon[33655]: pgmap v5891: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:41.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:40 smithi045 conmon[39718]: cluster 2021-07-31T03:56:39 2021-07-31T03:56:41.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:40 smithi045 conmon[39718]: .646165+0000 mgr.x (mgr.24542) 5551 : cluster [DBG] pgmap v5891: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:43.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:42 smithi154 conmon[34553]: cluster 2021-07-31T03:56:41.646546+0000 mgr.x (mgr.24542) 5552 : cluster [DBG] pgmap v5892: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:43.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:42 smithi045 conmon[33655]: cluster 2021-07-31T03:56:41.646546+0000 mgr.x (mgr.24542) 5552 : cluster 2021-07-31T03:56:43.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:42 smithi045 conmon[33655]: [DBG] pgmap v5892: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:43.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:42 smithi045 conmon[39718]: cluster 2021-07-31T03:56:41.646546+0000 2021-07-31T03:56:43.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:42 smithi045 conmon[39718]: mgr.x (mgr.24542) 5552 : cluster [DBG] pgmap v5892: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:45.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:44 smithi154 conmon[34553]: cluster 2021-07-31T03:56:43.647027+0000 mgr.x (mgr.24542) 5553 : cluster [DBG] pgmap v5893: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:45.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:44 smithi045 conmon[33655]: cluster 2021-07-31T03:56:43.647027+0000 mgr.x (mgr.24542) 2021-07-31T03:56:45.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:44 smithi045 conmon[33655]: 5553 : cluster [DBG] pgmap v5893: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:45.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:44 smithi045 conmon[39718]: cluster 2021-07-31T03:56:43.647027+0000 mgr.x ( 2021-07-31T03:56:45.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:44 smithi045 conmon[39718]: mgr.24542) 5553 : cluster [DBG] pgmap v5893: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:47.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:46 smithi154 conmon[34553]: cluster 2021-07-31T03:56:45.647482+0000 mgr.x (mgr.24542) 5554 : cluster [DBG] pgmap v5894: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:47.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:46 smithi045 conmon[39718]: cluster 2021-07-31T03:56:45.647482+0000 mgr.x (mgr.24542) 5554 : cluster [DBG] pgmap v5894: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:47.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:46 smithi045 conmon[33655]: cluster 2021-07-31T03:56:45.647482+0000 mgr.x (mgr.24542 2021-07-31T03:56:47.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:46 smithi045 conmon[33655]: ) 5554 : cluster [DBG] pgmap v5894: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:49.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:48 smithi154 conmon[34553]: cluster 2021-07-31T03:56:47.647892+0000 mgr.x (mgr.24542) 5555 : cluster [DBG] pgmap v5895: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:49.024 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:48 smithi045 conmon[39718]: cluster 2021-07-31T03:56:47 2021-07-31T03:56:49.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:48 smithi045 conmon[39718]: .647892+0000 mgr.x (mgr.24542) 5555 : cluster [DBG] pgmap v5895: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:49.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:48 smithi045 conmon[33655]: cluster 2021-07-31T03 2021-07-31T03:56:49.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:48 smithi045 conmon[33655]: :56:47.647892+0000 mgr.x (mgr.24542) 5555 : cluster [DBG] pgmap v5895: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:50.514 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:56:50.793 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:56:50.793 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 2h ago 2h - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:56:50.793 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 2h ago 2h 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:56:50.794 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 2h ago 2h 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:56:50.794 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 2h ago 2h - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:56:50.794 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:56:50.794 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 2h ago 2h 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:56:50.794 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:56:50.795 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 2h ago 2h - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:56:50.795 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 2h ago 2h 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:56:50.795 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:56:50.795 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:56:50.795 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:56:50.796 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:56:50.796 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 2h ago 2h 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:56:50.796 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 2h ago 2h 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:56:50.796 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 2h ago 2h 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:56:50.797 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 2h ago 2h 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:56:50.797 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 2h ago 2h 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:56:51.001 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:50 smithi154 conmon[34553]: cluster 2021-07-31T03:56:49.648278+0000 mgr.x (mgr.24542) 5556 : cluster [DBG] pgmap v5896: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:51.024 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:50 smithi045 conmon[39718]: cluster 2021-07-31T03:56:49. 2021-07-31T03:56:51.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:50 smithi045 conmon[39718]: 648278+0000 mgr.x (mgr.24542) 5556 : cluster [DBG] pgmap v5896: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:51.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:50 smithi045 conmon[33655]: cluster 2021-07-31T03:56:49.648278+0000 mgr.x (mgr.24542) 5556 : 2021-07-31T03:56:51.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:50 smithi045 conmon[33655]: cluster [DBG] pgmap v5896: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:51.128 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:56:51.129 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:56:51.129 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:56:51.129 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:56:51.129 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:56:51.130 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:56:51.130 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:56:51.130 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:56:51.130 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:56:51.130 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:56:51.131 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:56:51.131 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:56:51.131 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:56:51.131 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:56:51.131 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:56:51.132 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:56:51.132 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:56:52.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:51 smithi154 conmon[34553]: audit 2021-07-31T03:56:50.502544+0000 mgr.x (mgr.24542) 5557 : audit [DBG] from='client.30505 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:56:52.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:51 smithi154 conmon[34553]: audit 2021-07-31T03:56:50 2021-07-31T03:56:52.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:51 smithi154 conmon[34553]: .785820+0000 mgr.x (mgr.24542) 5558 : audit [DBG] from='client.30511 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:56:52.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:51 smithi154 conmon[34553]: audit 2021-07-31T03:56:51.129925+0000 mon.a (mon.0) 3396 : audit [DBG] from='client.? 172.21.15.45:0/3189925039' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:56:52.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:51 smithi045 conmon[39718]: audit 2021-07-31T03:56:50.502544+0000 mgr.x (mgr.24542) 5557 : audit [DBG] 2021-07-31T03:56:52.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:51 smithi045 conmon[39718]: from='client.30505 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:56:52.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:51 smithi045 conmon[39718]: audit 2021-07-31T03:56 2021-07-31T03:56:52.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:51 smithi045 conmon[39718]: :50.785820+0000 mgr.x (mgr.24542) 5558 : 2021-07-31T03:56:52.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:51 smithi045 conmon[39718]: audit [DBG] from='client.30511 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:56:52.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:51 smithi045 conmon[39718]: audit 2021 2021-07-31T03:56:52.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:51 smithi045 conmon[39718]: -07-31T03:56:51. 2021-07-31T03:56:52.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:51 smithi045 conmon[39718]: 129925+0000 mon.a (mon.0 2021-07-31T03:56:52.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:51 smithi045 conmon[39718]: ) 3396 : audit [DBG] from='client.? 172.21.15.45:0/3189925039' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:56:52.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:51 smithi045 conmon[39718]: 2021-07-31T03:56:52.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:51 smithi045 conmon[33655]: audit 2021-07-31T03:56:50.502544+0000 mgr.x (mgr.24542) 2021-07-31T03:56:52.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:51 smithi045 conmon[33655]: 5557 : audit [DBG] from='client.30505 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:56:52.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:51 smithi045 conmon[33655]: audit 2021-07-31T03:56:50.785820+0000 mgr.x (mgr.24542) 5558 : audit [DBG] from='client.30511 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:56:52.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:51 smithi045 conmon[33655]: audit 2021-07-31T03:56:51.129925+0000 2021-07-31T03:56:52.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:51 smithi045 conmon[33655]: mon.a (mon.0) 3396 : audit [DBG] 2021-07-31T03:56:52.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:51 smithi045 conmon[33655]: from='client.? 172.21.15.45:0/3189925039' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:56:53.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:52 smithi154 conmon[34553]: cluster 2021-07-31T03:56:51.648780+0000 mgr.x (mgr.24542) 5559 : cluster [DBG] pgmap v5897: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:53.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:52 smithi045 conmon[39718]: cluster 2021-07-31T03:56:51. 2021-07-31T03:56:53.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:52 smithi045 conmon[39718]: 648780+0000 mgr.x (mgr.24542) 5559 : cluster [DBG] pgmap v5897: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:53.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:52 smithi045 conmon[33655]: cluster 2021-07-31T03:56:51.648780+0000 mgr.x (mgr.24542) 5559 : 2021-07-31T03:56:53.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:52 smithi045 conmon[33655]: cluster [DBG] pgmap v5897: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:55.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:54 smithi045 conmon[39718]: cluster 2021-07-31T03:56:53.649296+0000 mgr.x (mgr. 2021-07-31T03:56:55.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:54 smithi045 conmon[39718]: 24542) 5560 : cluster [DBG] pgmap v5898: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:55.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:54 smithi045 conmon[33655]: cluster 2021-07-31T03:56:53.649296+0000 mgr.x (mgr.24542) 5560 : cluster [DBG] pgmap v5898: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:55.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:54 smithi154 conmon[34553]: cluster 2021-07-31T03:56:53.649296+0000 mgr.x (mgr.24542) 5560 : cluster [DBG] pgmap v5898: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:55.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:54 smithi154 conmon[34553]: 2021-07-31T03:56:57.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:56 smithi045 conmon[33655]: cluster 2021-07-31T03:56:55.649888+0000 mgr.x (mgr.24542) 5561 : cluster [DBG] pgmap v5899: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:57.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:56 smithi045 conmon[39718]: cluster 2021-07-31T03:56:55.649888+0000 mgr.x (mgr 2021-07-31T03:56:57.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:56 smithi045 conmon[39718]: .24542) 5561 : cluster [DBG] pgmap v5899: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:57.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:56 smithi154 conmon[34553]: cluster 2021-07-31T03:56:55.649888+0000 mgr.x (mgr.24542) 5561 : cluster [DBG] pgmap v5899: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:59.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:56:58 smithi045 conmon[33655]: cluster 2021-07-31T03:56:57.650308+0000 mgr.x (mgr.24542) 5562 : cluster [DBG] pgmap v5900: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:59.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:58 smithi045 conmon[39718]: cluster 2021-07-31T03:56:57.650308+0000 mgr.x ( 2021-07-31T03:56:59.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:56:58 smithi045 conmon[39718]: mgr.24542) 5562 : cluster [DBG] pgmap v5900: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:56:59.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:58 smithi154 conmon[34553]: cluster 2021-07-31T03:56:57.650308+0000 mgr.x (mgr.24542) 5562 : cluster [DBG] 2021-07-31T03:56:59.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:56:58 smithi154 conmon[34553]: pgmap v5900: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:01.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:00 smithi045 conmon[33655]: cluster 2021-07-31T03:56:59.650831+0000 mgr.x (mgr.24542) 5563 : cluster [DBG] 2021-07-31T03:57:01.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:00 smithi045 conmon[33655]: pgmap v5901: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:01.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:00 smithi045 conmon[39718]: cluster 2021-07-31T03:56:59.650831+0000 mgr.x 2021-07-31T03:57:01.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:00 smithi045 conmon[39718]: (mgr.24542) 5563 : cluster [DBG] pgmap v5901: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:01.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:00 smithi154 conmon[34553]: cluster 2021-07-31T03:56:59.650831+0000 mgr.x (mgr.24542) 5563 : cluster [DBG] pgmap v5901: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:03.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:02 smithi045 conmon[33655]: cluster 2021-07-31T03:57:01.651212+0000 mgr.x (mgr.24542) 5564 : 2021-07-31T03:57:03.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:02 smithi045 conmon[33655]: cluster [DBG] pgmap v5902: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:03.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:02 smithi045 conmon[39718]: cluster 2021-07-31T03:57:01.651212+0000 mgr.x (mgr 2021-07-31T03:57:03.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:02 smithi045 conmon[39718]: .24542) 5564 : cluster [DBG] pgmap v5902: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:03.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:02 smithi154 conmon[34553]: cluster 2021-07-31T03:57:01.651212+0000 mgr.x (mgr.24542) 5564 : cluster [DBG] pgmap v5902: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:05.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:04 smithi045 conmon[39718]: cluster 2021-07-31T03:57:03.651731+0000 mgr.x (mgr.24542) 5565 : cluster [DBG] pgmap v5903: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:05.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:04 smithi045 conmon[33655]: cluster 2021-07-31T03:57:03.651731+0000 mgr.x (mgr.24542) 5565 : cluster [DBG] 2021-07-31T03:57:05.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:04 smithi045 conmon[33655]: pgmap v5903: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:05.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:04 smithi154 conmon[34553]: cluster 2021-07-31T03:57:03.651731+0000 mgr.x (mgr.24542) 5565 : cluster [DBG] pgmap v5903: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:07.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:06 smithi154 conmon[34553]: cluster 2021-07-31T03:57:05.652383+0000 mgr.x (mgr.24542) 5566 : cluster [DBG] pgmap v5904: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:07.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:06 smithi154 conmon[34553]: 2021-07-31T03:57:07.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:06 smithi045 conmon[39718]: cluster 2021-07-31T03:57:05.652383+0000 mgr.x 2021-07-31T03:57:07.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:06 smithi045 conmon[39718]: (mgr.24542) 5566 : cluster [DBG] pgmap v5904: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:07.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:06 smithi045 conmon[33655]: cluster 2021-07-31T03:57:05.652383+0000 mgr.x (mgr.24542) 2021-07-31T03:57:07.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:06 smithi045 conmon[33655]: 5566 : cluster [DBG] pgmap v5904: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:09.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:08 smithi154 conmon[34553]: cluster 2021-07-31T03:57:07.652781+0000 mgr.x (mgr.24542) 5567 : cluster [DBG] pgmap v5905: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:09.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:08 smithi045 conmon[39718]: cluster 2021-07-31T03:57:07.652781+0000 mgr.x (mgr.24542) 5567 : cluster [DBG] pgmap v5905: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:09.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:08 smithi045 conmon[33655]: cluster 2021-07-31T03:57:07.652781+0000 mgr.x (mgr.24542) 5567 : cluster [DBG] pgmap v5905: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:11.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:10 smithi154 conmon[34553]: cluster 2021-07-31T03:57:09.653229+0000 mgr.x (mgr.24542) 5568 : cluster [DBG] pgmap v5906: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:11.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:10 smithi045 conmon[39718]: cluster 2021-07-31T03:57:09.653229+0000 mgr.x ( 2021-07-31T03:57:11.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:10 smithi045 conmon[39718]: mgr.24542) 5568 : cluster [DBG] pgmap v5906: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:11.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:10 smithi045 conmon[33655]: cluster 2021-07-31T03:57:09.653229+0000 mgr.x (mgr.24542) 5568 : cluster [DBG] pgmap v5906: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:13.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:12 smithi154 conmon[34553]: cluster 2021-07-31T03:57:11.653763+0000 mgr.x (mgr.24542) 5569 : cluster [DBG] pgmap v5907: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:13.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:12 smithi045 conmon[39718]: cluster 2021-07-31T03:57:11.653763+0000 2021-07-31T03:57:13.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:12 smithi045 conmon[39718]: mgr.x (mgr.24542) 5569 : cluster [DBG] pgmap v5907: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:13.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:12 smithi045 conmon[33655]: cluster 2021-07-31T03:57:11.653763+0000 mgr.x (mgr.24542) 5569 2021-07-31T03:57:13.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:12 smithi045 conmon[33655]: : cluster [DBG] pgmap v5907: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:15.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:14 smithi154 conmon[34553]: cluster 2021-07-31T03:57:13.654218+0000 mgr.x (mgr.24542) 5570 : cluster [DBG] pgmap v5908: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:15.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:14 smithi045 conmon[39718]: cluster 2021-07-31T03:57:13.654218+0000 mgr.x 2021-07-31T03:57:15.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:14 smithi045 conmon[39718]: (mgr.24542) 5570 : cluster [DBG] pgmap v5908: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:15.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:14 smithi045 conmon[33655]: cluster 2021-07-31T03:57:13.654218+0000 mgr.x (mgr.24542) 5570 : cluster [DBG] pgmap v5908: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:15.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:14 smithi045 conmon[33655]: 2021-07-31T03:57:17.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:16 smithi154 conmon[34553]: cluster 2021-07-31T03:57:15.654823+0000 mgr.x (mgr.24542) 5571 : cluster [DBG] pgmap v5909: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:17.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:16 smithi045 conmon[39718]: cluster 2021-07-31T03:57:15.654823+0000 mgr.x ( 2021-07-31T03:57:17.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:16 smithi045 conmon[39718]: mgr.24542) 5571 : cluster [DBG] pgmap v5909: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:17.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:16 smithi045 conmon[33655]: cluster 2021-07-31T03:57:15.654823+0000 mgr.x ( 2021-07-31T03:57:17.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:16 smithi045 conmon[33655]: mgr.24542) 5571 : cluster [DBG] pgmap v5909: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:19.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:18 smithi154 conmon[34553]: cluster 2021-07-31T03:57:17.655410+0000 mgr.x (mgr.24542) 5572 : cluster [DBG] pgmap v5910: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:19.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:18 smithi045 conmon[39718]: cluster 2021-07-31T03:57:17 2021-07-31T03:57:19.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:18 smithi045 conmon[39718]: .655410+0000 mgr.x (mgr.24542) 5572 : cluster [DBG] pgmap v5910: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:19.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:18 smithi045 conmon[33655]: cluster 2021-07-31T03:57:17.655410+0000 mgr.x (mgr.24542) 5572 : cluster 2021-07-31T03:57:19.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:18 smithi045 conmon[33655]: [DBG] pgmap v5910: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:21.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:20 smithi154 conmon[34553]: cluster 2021-07-31T03:57:19.655879+0000 mgr.x (mgr.24542) 5573 : cluster [DBG] pgmap v5911: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:21.274 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:20 smithi045 conmon[39718]: cluster 2021-07-31T03:57:19.655879+0000 mgr.x (mgr.24542 2021-07-31T03:57:21.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:20 smithi045 conmon[39718]: ) 5573 : cluster [DBG] pgmap v5911: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:21.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:20 smithi045 conmon[33655]: cluster 2021-07-31T03:57:19.655879+0000 mgr.x (mgr.24542) 5573 : cluster [DBG] pgmap v5911: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:21.441 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:57:21.717 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:57:21.718 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 2h ago 2h - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:57:21.718 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 2h ago 2h 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:57:21.718 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 2h ago 2h 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:57:21.718 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 2h ago 2h - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:57:21.719 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:57:21.719 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 2h ago 2h 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:57:21.719 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:57:21.719 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 2h ago 2h - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:57:21.720 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 2h ago 2h 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:57:21.720 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:57:21.720 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:57:21.720 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:57:21.721 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:57:21.721 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 2h ago 2h 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:57:21.721 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 2h ago 2h 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:57:21.721 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 2h ago 2h 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:57:21.721 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 2h ago 2h 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:57:21.722 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 2h ago 2h 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:57:22.055 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:57:22.056 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:57:22.056 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:57:22.056 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:57:22.057 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:57:22.057 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:57:22.057 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:57:22.057 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:57:22.057 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:57:22.058 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:57:22.058 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:57:22.058 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:57:22.058 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:57:22.058 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:57:22.059 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:57:22.059 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:57:22.059 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:57:23.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:22 smithi154 conmon[34553]: audit 2021-07-31T03:57:21.430157+0000 mgr.x (mgr.24542) 5574 : audit [DBG] from='client.30523 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:57:23.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:22 smithi154 conmon[34553]: cluster 2021-07-31T03:57:21.656414+0000 mgr.x (mgr.24542) 5575 : cluster [DBG] pgmap v5912: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:23.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:22 smithi154 conmon[34553]: audit 2021-07-31T03:57:21.710346+0000 mgr.x (mgr.24542) 5576 : audit [DBG] from='client.30529 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:57:23.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:22 smithi154 conmon[34553]: audit 2021-07-31T03:57: 2021-07-31T03:57:23.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:22 smithi154 conmon[34553]: 22.056665+0000 mon.c (mon.1) 237 : audit 2021-07-31T03:57:23.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:22 smithi154 conmon[34553]: [DBG] from='client.? 172.21.15.45:0/1365486483' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:57:23.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:22 smithi045 conmon[39718]: audit 2021-07-31T03:57:21. 2021-07-31T03:57:23.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:22 smithi045 conmon[39718]: 430157+0000 mgr.x (mgr.24542) 5574 : audit [DBG] from='client.30523 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:57:23.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:22 smithi045 conmon[39718]: cluster 2021-07-31T03: 2021-07-31T03:57:23.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:22 smithi045 conmon[39718]: 57:21.656414+0000 mgr.x ( 2021-07-31T03:57:23.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:22 smithi045 conmon[39718]: mgr.24542) 5575 : cluster [DBG] 2021-07-31T03:57:23.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:22 smithi045 conmon[39718]: pgmap v5912: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:23.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:22 smithi045 conmon[39718]: audit 2021-07-31T03:57: 2021-07-31T03:57:23.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:22 smithi045 conmon[39718]: 21.710346+0000 mgr.x (mgr.24542) 5576 2021-07-31T03:57:23.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:22 smithi045 conmon[39718]: : audit [DBG] from='client.30529 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:57:23.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:22 smithi045 conmon[39718]: audit 2021 2021-07-31T03:57:23.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:22 smithi045 conmon[39718]: -07-31T03:57:22. 2021-07-31T03:57:23.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:22 smithi045 conmon[39718]: 056665+0000 mon.c (mon.1) 2021-07-31T03:57:23.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:22 smithi045 conmon[39718]: 237 : audit [DBG] from='client.? 172.21.15.45:0/1365486483' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:57:23.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:22 smithi045 conmon[33655]: audit 2021-07-31T03:57:21.430157+0000 mgr.x (mgr.24542) 5574 : audit [DBG] 2021-07-31T03:57:23.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:22 smithi045 conmon[33655]: from='client.30523 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:57:23.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:22 smithi045 conmon[33655]: cluster 2021-07-31T03:57:21.656414+0000 mgr.x (mgr.24542) 5575 : cluster [DBG] pgmap v5912: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:23.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:22 smithi045 conmon[33655]: audit 2021-07-31T03:57 2021-07-31T03:57:23.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:22 smithi045 conmon[33655]: :21.710346+0000 mgr.x (mgr.24542) 5576 2021-07-31T03:57:23.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:22 smithi045 conmon[33655]: : audit [DBG] from='client.30529 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:57:23.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:22 smithi045 conmon[33655]: audit 2021-07 2021-07-31T03:57:23.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:22 smithi045 conmon[33655]: -31T03:57:22.056665+0000 mon.c (mon.1) 2021-07-31T03:57:23.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:22 smithi045 conmon[33655]: 237 : audit [DBG] from='client.? 172.21.15.45:0/1365486483' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:57:25.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:24 smithi154 conmon[34553]: cluster 2021-07-31T03:57:23.656858+0000 mgr.x (mgr.24542) 5577 : cluster [DBG] pgmap v5913: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:25.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:24 smithi045 conmon[39718]: cluster 2021-07-31T03:57:23. 2021-07-31T03:57:25.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:24 smithi045 conmon[39718]: 656858+0000 mgr.x (mgr.24542) 5577 : cluster [DBG] pgmap v5913: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:25.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:24 smithi045 conmon[33655]: cluster 2021-07-31T03:57:23.656858+0000 mgr.x (mgr.24542) 5577 : cluster [DBG] pgmap v5913: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:27.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:26 smithi154 conmon[34553]: cluster 2021-07-31T03:57:25.657373+0000 mgr.x (mgr.24542) 5578 : cluster [DBG] pgmap v5914: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:27.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:26 smithi045 conmon[39718]: cluster 2021-07-31T03:57:25 2021-07-31T03:57:27.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:26 smithi045 conmon[39718]: .657373+0000 mgr.x (mgr.24542) 5578 : cluster [DBG] pgmap v5914: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:27.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:26 smithi045 conmon[33655]: cluster 2021-07-31T03:57:25.657373+0000 mgr.x (mgr. 2021-07-31T03:57:27.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:26 smithi045 conmon[33655]: 24542) 5578 : cluster [DBG] pgmap v5914: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:29.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:28 smithi154 conmon[34553]: cluster 2021-07-31T03:57:27.657843+0000 mgr.x (mgr.24542) 5579 : cluster [DBG] pgmap v5915: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:29.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:28 smithi154 conmon[34553]: audit 2021-07-31T03:57:28.450886+0000 mon.b (mon.2) 725 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:57:29.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:28 smithi154 conmon[34553]: audit 2021-07-31T03:57:28.451151+0000 mon.b (mon.2) 726 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:57:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:28 smithi154 conmon[34553]: audit 2021-07-31T03:57:28.451808+0000 mon.a (mon.0) 3397 : audit 2021-07-31T03:57:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:28 smithi154 conmon[34553]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:57:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:28 smithi154 conmon[34553]: audit 2021-07-31T03: 2021-07-31T03:57:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:28 smithi154 conmon[34553]: 57:28.452206+0000 mon.a (mon.0) 2021-07-31T03:57:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:28 smithi154 conmon[34553]: 3398 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:57:29.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:28 smithi045 conmon[39718]: cluster 2021-07-31T03:57:27.657843+0000 mgr.x (mgr. 2021-07-31T03:57:29.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:28 smithi045 conmon[39718]: 24542) 5579 : cluster [DBG] pgmap v5915: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:29.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:28 smithi045 conmon[39718]: audit 2021-07 2021-07-31T03:57:29.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:28 smithi045 conmon[39718]: -31T03:57:28.450886+0000 mon.b (mon.2) 725 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:57:29.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:28 smithi045 conmon[39718]: audit 2021- 2021-07-31T03:57:29.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:28 smithi045 conmon[39718]: 07-31T03:57:28.451151+0000 mon.b (mon.2) 726 : audit 2021-07-31T03:57:29.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:28 smithi045 conmon[39718]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:57:29.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:28 smithi045 conmon[39718]: audit 2021-07 2021-07-31T03:57:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:28 smithi045 conmon[39718]: -31T03:57:28.451808+0000 mon.a (mon 2021-07-31T03:57:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:28 smithi045 conmon[39718]: .0) 3397 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:57:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:28 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T03:57:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:28 smithi045 conmon[39718]: 03:57:28.452206+0000 mon.a (mon.0) 3398 2021-07-31T03:57:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:28 smithi045 conmon[39718]: : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:57:29.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:28 smithi045 conmon[33655]: cluster 2021-07-31T03:57:27.657843+0000 mgr.x (mgr.24542) 5579 : cluster 2021-07-31T03:57:29.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:28 smithi045 conmon[33655]: [DBG] pgmap v5915: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:29.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:28 smithi045 conmon[33655]: audit 2021-07-31T03:57:28.450886+0000 mon.b (mon.2) 725 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:57:29.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:28 smithi045 conmon[33655]: audit 2021-07-31T03: 2021-07-31T03:57:29.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:28 smithi045 conmon[33655]: 57:28.451151+0000 mon.b (mon.2) 726 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:57:29.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:28 smithi045 conmon[33655]: audit 2021-07-31T03:57:29.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:28 smithi045 conmon[33655]: 2021-07-31T03:57:28.451808+0000 2021-07-31T03:57:29.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:28 smithi045 conmon[33655]: mon.a (mon.0) 3397 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:57:29.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:28 smithi045 conmon[33655]: audit 2021-07- 2021-07-31T03:57:29.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:28 smithi045 conmon[33655]: 31T03:57:28.452206+0000 mon.a (mon.0) 3398 : audit 2021-07-31T03:57:29.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:28 smithi045 conmon[33655]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:57:31.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:30 smithi154 conmon[34553]: cluster 2021-07-31T03:57:29.658510+0000 mgr.x (mgr.24542) 5580 : cluster [DBG] pgmap v5916: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:31.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:30 smithi045 conmon[39718]: cluster 2021-07-31T03:57:29.658510+0000 mgr.x 2021-07-31T03:57:31.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:30 smithi045 conmon[39718]: (mgr.24542) 5580 : cluster [DBG] pgmap v5916: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:31.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:30 smithi045 conmon[33655]: cluster 2021-07-31T03:57:29.658510+0000 mgr.x (mgr.24542) 5580 : cluster [DBG] pgmap v5916: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:33.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:32 smithi154 conmon[34553]: cluster 2021-07-31T03:57:31.658990+0000 mgr.x (mgr.24542) 5581 : cluster [DBG] pgmap v5917: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:33.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:32 smithi045 conmon[33655]: cluster 2021-07-31T03:57:31.658990+0000 mgr.x (mgr.24542) 5581 : cluster 2021-07-31T03:57:33.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:32 smithi045 conmon[33655]: [DBG] pgmap v5917: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:33.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:32 smithi045 conmon[39718]: cluster 2021-07-31T03:57:31.658990+0000 mgr.x ( 2021-07-31T03:57:33.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:32 smithi045 conmon[39718]: mgr.24542) 5581 : cluster [DBG] pgmap v5917: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:35.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:34 smithi154 conmon[34553]: cluster 2021-07-31T03:57:33.659457+0000 mgr.x (mgr.24542) 5582 : cluster [DBG] pgmap v5918: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:35.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:34 smithi045 conmon[39718]: cluster 2021-07-31T03:57:33.659457+0000 mgr.x (mgr.24542) 5582 : cluster [DBG] pgmap v5918: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:35.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:34 smithi045 conmon[33655]: cluster 2021-07-31T03:57:33.659457+0000 mgr.x (mgr.24542) 5582 2021-07-31T03:57:35.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:34 smithi045 conmon[33655]: : cluster [DBG] pgmap v5918: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:36.143 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: debug 2021-07-31T03:57:36.102+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:57:36.143 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:57:36.143 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:57:36.143 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: yield 2021-07-31T03:57:36.144 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:57:36.144 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:57:36.144 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:57:36.144 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:57:36.144 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:57:36.145 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:57:36.145 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:57:36.145 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:57:36.145 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:57:36.145 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: task.result() 2021-07-31T03:57:36.146 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:57:36.146 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: debug 2021-07-31T03:57:36.114+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:57:36.146 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:57:36.146 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:57:36.146 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: yield 2021-07-31T03:57:36.147 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:57:36.147 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:57:36.147 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:57:36.147 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:57:36.147 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:57:36.148 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:57:36.148 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:57:36.148 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:57:36.148 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:57:36.148 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: task.result() 2021-07-31T03:57:36.149 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:57:36.149 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: debug 2021-07-31T03:57:36.140+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:57:36.149 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:57:36.149 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:57:36.149 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: yield 2021-07-31T03:57:36.150 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:57:36.150 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:57:36.150 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:57:36.150 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:57:36.150 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:57:36.151 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:57:36.151 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:57:36.151 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:57:36.151 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:57:36.151 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: task.result() 2021-07-31T03:57:36.152 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:57:36.152 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: debug 2021-07-31T03:57:36.140+0000 7fcb94162700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:57:36.152 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:57:36.152 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:57:36.152 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: yield 2021-07-31T03:57:36.153 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:57:36.153 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:57:36.153 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:57:36.153 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:57:36.153 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:57:36.154 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:57:36.154 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:57:36.154 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:57:36.502 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:57:36.502 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: task.result() 2021-07-31T03:57:36.503 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:57:36.503 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: 2021-07-31T03:57:36.503 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:57:36.503 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: 2021-07-31T03:57:36.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:57:36.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:57:36.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:57:36.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:57:36.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:57:36.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:57:36.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:57:36.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:57:36.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:57:36.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:57:36.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:57:36.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:57:36.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:57:36.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:57:36.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:57:36.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:57:36.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:57:36.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:57:36.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: raise 2021-07-31T03:57:36.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:57:36.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:57:36.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:57:36.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:57:36.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:57:36.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: debug 2021-07-31T03:57:36.150+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:57:36.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:57:36.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:57:36.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: yield 2021-07-31T03:57:36.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:57:36.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:57:36.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:57:36.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:57:36.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:57:36.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:57:36.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:57:36.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:57:36.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:57:36.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: task.result() 2021-07-31T03:57:36.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:57:36.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: debug 2021-07-31T03:57:36.150+0000 7fcb93961700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:57:36.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:57:36.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:57:36.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: yield 2021-07-31T03:57:36.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:57:36.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:57:36.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:57:36.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:57:36.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:57:36.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:57:36.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:57:36.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:57:36.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:57:36.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: task.result() 2021-07-31T03:57:36.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:57:36.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: 2021-07-31T03:57:36.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:57:36.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: 2021-07-31T03:57:36.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:57:36.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:57:36.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:57:36.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:57:36.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:57:36.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:57:36.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:57:36.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:57:36.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:57:36.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:57:36.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:57:36.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:57:36.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:57:36.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:57:36.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:57:36.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:57:36.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:57:36.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:57:36.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: raise 2021-07-31T03:57:36.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:57:36.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:57:36.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:57:36.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:57:36.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:57:37.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: cluster 2021-07-31T03:57:35.659987+0000 mgr.x (mgr.24542) 5583 : cluster [DBG] pgmap v5919: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:37.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: audit 2021-07-31T03:57:36.057877+0000 mon.b (mon.2) 727 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:57:37.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: audit 2021-07-31T03:57:36.059302+0000 mon.b (mon.2) 728 : audit [DBG] 2021-07-31T03:57:37.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:57:37.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: audit 2021-07- 2021-07-31T03:57:37.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: 31T03:57:36.060191+0000 mon.b (mon.2) 729 : audit 2021-07-31T03:57:37.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:57:37.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: cephadm 2021-07-31 2021-07-31T03:57:37.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: T03:57:36.060925+0000 mgr.x (mgr. 2021-07-31T03:57:37.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: 24542) 5584 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:57:37.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: cephadm 2021- 2021-07-31T03:57:37.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: 07-31T03:57:36.103565+0000 2021-07-31T03:57:37.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: mgr.x (mgr.24542) 5585 : cephadm [ERR] 2021-07-31T03:57:37.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:57:37.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:57:37.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: yield 2021-07-31T03:57:37.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:57:37.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:57:37.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:57:37.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:57:37.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:57:37.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:57:37.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:57:37.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:57:37.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:57:37.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: task.result() 2021-07-31T03:57:37.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:57:37.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: 2021-07-31T03:57:37.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: cephadm 2021-07-31T03:57:36.104012+0000 mgr.x (mgr.24542) 5586 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:57:37.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: cephadm 2021-07-31T03:57:36.115027+0000 2021-07-31T03:57:37.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: mgr.x (mgr.24542) 5587 : cephadm [ERR] 2021-07-31T03:57:37.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:57:37.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:57:37.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: yield 2021-07-31T03:57:37.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:57:37.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:57:37.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:57:37.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:57:37.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:57:37.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:57:37.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:57:37.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:57:37.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:57:37.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: task.result() 2021-07-31T03:57:37.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:57:37.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: cephadm 2021-07-31T03:57:36.115552+0000 mgr.x (mgr.24542) 5588 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:57:37.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: cephadm 2021-07-31T03:57:36.141206+0000 mgr.x (mgr.24542) 5589 : cephadm [ERR] 2021-07-31T03:57:37.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:57:37.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:57:37.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: yield 2021-07-31T03:57:37.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:57:37.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:57:37.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:57:37.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:57:37.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:57:37.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:57:37.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:57:37.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:57:37.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:57:37.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: task.result() 2021-07-31T03:57:37.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:57:37.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: cephadm 2021-07-31T03:57:36.141818+0000 mgr.x (mgr.24542) 5590 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:57:37.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:57:37.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:57:37.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: yield 2021-07-31T03:57:37.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:57:37.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:57:37.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:57:37.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:57:37.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:57:37.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:57:37.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:57:37.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:57:37.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:57:37.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: task.result() 2021-07-31T03:57:37.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:57:37.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: 2021-07-31T03:57:37.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:57:37.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: 2021-07-31T03:57:37.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:57:37.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:57:37.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:57:37.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:57:37.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:57:37.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:57:37.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:57:37.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:57:37.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:57:37.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:57:37.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:57:37.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:57:37.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:57:37.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:57:37.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:57:37.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:57:37.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:57:37.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:57:37.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: raise 2021-07-31T03:57:37.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:57:37.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:57:37.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:57:37.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:57:37.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:57:37.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: cephadm 2021-07-31T03:57:36.151110+0000 mgr.x (mgr.24542) 5591 2021-07-31T03:57:37.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: : cephadm [ERR] 2021-07-31T03:57:37.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:57:37.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:57:37.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: yield 2021-07-31T03:57:37.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:57:37.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:57:37.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:57:37.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:57:37.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:57:37.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:57:37.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:57:37.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:57:37.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:57:37.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: task.result() 2021-07-31T03:57:37.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: cluster 2021-07-31T03:57:35.659987+0000 mgr.x (mgr.24542) 5583 : cluster [DBG] pgmap v5919: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:37.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: 2021-07-31T03:57:37.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: audit 2021-07-31T03:57:36.057877+0000 mon.b (mon.2) 727 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:57:37.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: audit 2021-07-31T03:57:36.059302+0000 mon.b (mon.2) 728 : audit [DBG] 2021-07-31T03:57:37.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:57:37.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: audit 2021-07-31T03:57:36.060191+0000 mon.b (mon.2) 729 : audit [INF] 2021-07-31T03:57:37.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:57:37.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: cephadm 2021-07-31T03:57:36. 2021-07-31T03:57:37.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: 060925+0000 mgr.x (mgr.24542) 5584 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:57:37.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: cephadm 2021-07-31T03:57 2021-07-31T03:57:37.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: :36.103565+0000 mgr.x (mgr.24542) 5585 : cephadm [ERR] 2021-07-31T03:57:37.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: 2021-07-31T03:57:37.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:57:37.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:57:37.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: yield 2021-07-31T03:57:37.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:57:37.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:57:37.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:57:37.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:57:37.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:57:37.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:57:37.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:57:37.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:57:37.283 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:57:37.283 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: task.result() 2021-07-31T03:57:37.283 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:57:37.283 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: cephadm 2021-07-31T03:57:36.104012+0000 mgr.x (mgr.24542) 5586 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:57:37.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: cephadm 2021 2021-07-31T03:57:37.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: -07-31T03:57:36. 2021-07-31T03:57:37.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: 115027+0000 mgr.x (mgr. 2021-07-31T03:57:37.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: 24542) 5587 : cephadm [ERR] 2021-07-31T03:57:37.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: 2021-07-31T03:57:37.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:57:37.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:57:37.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: yield 2021-07-31T03:57:37.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:57:37.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:57:37.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:57:37.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: cluster 2021-07-31T03:57:35 2021-07-31T03:57:37.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: .659987+0000 mgr.x (mgr.24542) 5583 : cluster [DBG] pgmap v5919: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:37.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: audit 2021-07-31T03:57:37.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: 2021-07-31T03: 2021-07-31T03:57:37.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: 57:36.057877+0000 mon.b 2021-07-31T03:57:37.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: (mon.2) 727 : audit 2021-07-31T03:57:37.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:57:37.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: audit 2021-07-31T03:57:37.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: 2021-07-31T03:57:36 2021-07-31T03:57:37.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: .059302+0000 mon.b (mon. 2021-07-31T03:57:37.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: 2) 728 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:57:37.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: 2021-07-31T03:57:37.289 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: audit 2021 2021-07-31T03:57:37.289 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: -07-31T03:57:36 2021-07-31T03:57:37.289 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: .060191+0000 mon.b (mon. 2021-07-31T03:57:37.289 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: 2) 729 : audit [INF] 2021-07-31T03:57:37.289 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:57:37.290 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: cephadm 2021-07- 2021-07-31T03:57:37.290 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: 31T03:57:36.060925+0000 2021-07-31T03:57:37.290 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: mgr.x (mgr.24542) 5584 2021-07-31T03:57:37.290 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:57:37.290 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: cephadm 2021 2021-07-31T03:57:37.291 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: -07-31T03:57:36 2021-07-31T03:57:37.291 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: .103565+0000 mgr.x (mgr.24542 2021-07-31T03:57:37.291 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: ) 5585 : cephadm [ERR] 2021-07-31T03:57:37.291 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:57:37.291 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:57:37.292 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: yield 2021-07-31T03:57:37.292 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:57:37.292 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:57:37.292 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:57:37.292 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:57:37.293 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:57:37.293 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:57:37.293 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:57:37.293 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:57:37.293 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:57:37.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: task.result() 2021-07-31T03:57:37.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:57:37.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: 2021-07-31T03:57:37.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: cephadm 2021-07-31T03: 2021-07-31T03:57:37.295 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: 57:36.104012+0000 mgr.x (mgr. 2021-07-31T03:57:37.295 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: 24542) 5586 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:57:37.295 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: 2021-07-31T03:57:37.295 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: cephadm 2021-07-31T03:57:37.295 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: 2021-07-31T03:57:36 2021-07-31T03:57:37.295 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: .115027+0000 mgr.x (mgr. 2021-07-31T03:57:37.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: 24542) 5587 : cephadm [ERR] 2021-07-31T03:57:37.296 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:57:37.296 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: cephadm 2021-07-31T03:57:36.151525+0000 mgr.x (mgr.24542) 5592 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:57:37.296 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:57:37.297 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:57:37.297 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: yield 2021-07-31T03:57:37.297 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:57:37.297 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:57:37.297 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:57:37.298 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:57:37.298 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:57:37.298 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:57:37.298 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:57:37.298 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:57:37.299 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:57:37.299 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: task.result() 2021-07-31T03:57:37.299 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:57:37.299 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: 2021-07-31T03:57:37.299 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:57:37.300 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: 2021-07-31T03:57:37.300 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:57:37.300 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:57:37.300 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:57:37.300 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:57:37.301 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:57:37.301 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:57:37.301 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:57:37.301 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:57:37.301 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:57:37.302 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:57:37.302 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:57:37.302 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:57:37.302 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:57:37.302 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:57:37.303 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:57:37.303 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:57:37.303 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:57:37.303 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:57:37.303 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: raise 2021-07-31T03:57:37.304 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:57:37.304 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:57:37.304 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:57:37.304 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:57:37.304 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:36 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:57:37.305 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:57:37.305 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:57:37.305 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:57:37.306 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:57:37.306 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:57:37.306 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:57:37.306 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: task.result() 2021-07-31T03:57:37.306 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:57:37.307 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: cephadm 2021-07-31T03:57:36.115552+0000 mgr.x (mgr.24542) 5588 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:57:37.307 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: cephadm 2021-07-31T03:57: 2021-07-31T03:57:37.307 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: 36.141206+0000 mgr.x (mgr.24542) 5589 : cephadm [ERR] 2021-07-31T03:57:37.307 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:57:37.307 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:57:37.308 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: yield 2021-07-31T03:57:37.308 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:57:37.308 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:57:37.308 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:57:37.308 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:57:37.309 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:57:37.309 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:57:37.309 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:57:37.309 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:57:37.309 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:57:37.310 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: task.result() 2021-07-31T03:57:37.310 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:57:37.310 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: cephadm 2021-07-31T03:57:36.141818+0000 mgr.x (mgr.24542) 5590 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:57:37.310 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:57:37.310 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:57:37.311 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: yield 2021-07-31T03:57:37.311 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:57:37.311 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:57:37.311 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:57:37.311 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:57:37.312 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:57:37.312 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:57:37.312 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:57:37.312 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:57:37.312 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:57:37.312 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: task.result() 2021-07-31T03:57:37.313 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:57:37.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: 2021-07-31T03:57:37.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:57:37.314 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:57:37.314 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: yield 2021-07-31T03:57:37.314 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:57:37.314 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:57:37.314 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:57:37.315 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:57:37.315 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:57:37.315 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:57:37.315 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:57:37.315 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:57:37.316 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:57:37.316 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: task.result() 2021-07-31T03:57:37.316 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:57:37.316 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: cephadm 2021-07-31T03:57:36.115552+0000 mgr.x (mgr.24542) 5588 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:57:37.316 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: cephadm 2021-07-31T03:57:37.317 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: 2021-07-31T03:57:36 2021-07-31T03:57:37.317 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: .141206+0000 mgr.x (mgr. 2021-07-31T03:57:37.317 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: 24542) 5589 : cephadm [ERR] 2021-07-31T03:57:37.317 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: 2021-07-31T03:57:37.317 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:57:37.318 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:57:37.318 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: yield 2021-07-31T03:57:37.318 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:57:37.318 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:57:37.318 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:57:37.319 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:57:37.319 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:57:37.319 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:57:37.319 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:57:37.319 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:57:37.320 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:57:37.320 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: task.result() 2021-07-31T03:57:37.320 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:57:37.320 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: cephadm 2021-07-31T03:57:36.141818+0000 mgr.x ( 2021-07-31T03:57:37.320 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: mgr.24542) 5590 : cephadm [ERR] 2021-07-31T03:57:37.321 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:57:37.321 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:57:37.321 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:57:37.321 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: yield 2021-07-31T03:57:37.321 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:57:37.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: 2021-07-31T03:57:37.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:57:37.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: 2021-07-31T03:57:37.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:57:37.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:57:37.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:57:37.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:57:37.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:57:37.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:57:37.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:57:37.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:57:37.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:57:37.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:57:37.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:57:37.325 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:57:37.325 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:57:37.325 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:57:37.325 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:57:37.325 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:57:37.326 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:57:37.326 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:57:37.326 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: raise 2021-07-31T03:57:37.326 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:57:37.326 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:57:37.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:57:37.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:57:37.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:57:37.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: cephadm 2021-07-31T03:57:36.151110+0000 mgr.x (mgr.24542) 5591 : cephadm [ERR] 2021-07-31T03:57:37.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:57:37.328 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:57:37.328 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: yield 2021-07-31T03:57:37.328 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:57:37.328 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:57:37.329 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:57:37.329 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:57:37.329 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:57:37.329 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:57:37.329 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:57:37.330 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:57:37.330 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:57:37.330 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: task.result() 2021-07-31T03:57:37.330 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:57:37.331 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:57:37.331 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:57:37.331 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:57:37.331 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:57:37.331 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:57:37.332 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:57:37.332 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:57:37.332 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: task.result() 2021-07-31T03:57:37.332 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:57:37.332 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: 2021-07-31T03:57:37.333 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:57:37.333 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: 2021-07-31T03:57:37.333 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:57:37.333 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:57:37.333 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:57:37.334 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:57:37.334 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:57:37.334 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:57:37.334 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:57:37.334 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:57:37.335 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:57:37.335 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:57:37.335 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:57:37.335 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:57:37.335 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:57:37.336 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:57:37.336 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:57:37.336 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:57:37.336 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:57:37.336 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:57:37.337 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: raise 2021-07-31T03:57:37.337 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:57:37.337 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:57:37.337 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:57:37.337 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:57:37.338 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:57:37.338 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: cephadm 2021-07-31T03:57:36.151110+0000 mgr.x (mgr.24542) 5591 : cephadm [ERR] 2021-07-31T03:57:37.338 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:57:37.338 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:57:37.338 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: yield 2021-07-31T03:57:37.339 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:57:37.339 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:57:37.339 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: cephadm 2021-07-31T03:57:36.151525+0000 mgr.x (mgr.24542) 5592 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:57:37.340 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:57:37.340 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:57:37.340 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: yield 2021-07-31T03:57:37.340 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:57:37.340 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:57:37.341 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:57:37.341 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:57:37.341 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:57:37.341 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:57:37.341 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:57:37.342 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:57:37.342 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:57:37.342 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: task.result() 2021-07-31T03:57:37.342 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:57:37.342 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: 2021-07-31T03:57:37.343 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:57:37.343 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: 2021-07-31T03:57:37.343 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:57:37.343 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:57:37.343 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:57:37.344 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:57:37.344 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:57:37.344 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:57:37.344 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:57:37.344 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:57:37.345 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:57:37.345 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:57:37.345 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:57:37.345 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:57:37.345 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:57:37.346 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:57:37.346 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:57:37.346 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:57:37.346 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:57:37.346 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:57:37.347 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: raise 2021-07-31T03:57:37.347 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:57:37.347 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:57:37.347 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:57:37.347 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:57:37.347 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:57:37.348 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:57:37.348 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:57:37.348 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:57:37.349 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:57:37.349 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:57:37.349 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:57:37.349 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:57:37.349 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:57:37.350 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: task.result() 2021-07-31T03:57:37.350 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:57:37.350 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: cephadm 2021-07- 2021-07-31T03:57:37.350 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: 31T03:57:36.151525+0000 mgr.x (mgr.24542) 5592 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:57:37.350 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:57:37.351 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:57:37.351 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: yield 2021-07-31T03:57:37.351 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:57:37.351 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:57:37.351 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:57:37.352 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:57:37.352 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:57:37.352 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:57:37.352 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:57:37.352 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:57:37.353 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:57:37.353 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: task.result() 2021-07-31T03:57:37.353 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:57:37.353 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: 2021-07-31T03:57:37.353 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:57:37.354 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: 2021-07-31T03:57:37.354 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:57:37.354 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:57:37.354 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:57:37.354 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:57:37.355 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:57:37.355 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:57:37.355 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:57:37.355 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:57:37.355 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:57:37.356 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:57:37.356 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:57:37.356 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:57:37.356 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:57:37.356 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:57:37.357 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:57:37.357 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:57:37.357 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:57:37.357 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:57:37.357 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: raise 2021-07-31T03:57:37.357 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:57:37.358 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:57:37.358 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:57:37.358 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:57:37.358 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:36 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:57:39.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:38 smithi154 conmon[34553]: cluster 2021-07-31T03:57:37.660549+0000 mgr.x (mgr.24542) 5593 : cluster [DBG] 2021-07-31T03:57:39.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:38 smithi154 conmon[34553]: pgmap v5920: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:39.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:38 smithi045 conmon[39718]: cluster 2021-07-31T03:57:37. 2021-07-31T03:57:39.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:38 smithi045 conmon[39718]: 660549+0000 mgr.x (mgr.24542) 5593 : cluster [DBG] pgmap v5920: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:39.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:38 smithi045 conmon[33655]: cluster 2021-07-31T03:57:37.660549+0000 mgr.x (mgr.24542) 5593 2021-07-31T03:57:39.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:38 smithi045 conmon[33655]: : cluster [DBG] pgmap v5920: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:41.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:40 smithi154 conmon[34553]: cluster 2021-07-31T03:57:39.661011+0000 mgr.x (mgr.24542) 5594 : cluster [DBG] pgmap v5921: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:41.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:40 smithi045 conmon[39718]: cluster 2021-07-31T03:57:39.661011+0000 mgr.x 2021-07-31T03:57:41.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:40 smithi045 conmon[39718]: (mgr.24542) 5594 : cluster [DBG] pgmap v5921: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:41.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:40 smithi045 conmon[33655]: cluster 2021-07-31T03:57:39.661011+0000 mgr.x (mgr.24542) 5594 : cluster 2021-07-31T03:57:41.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:40 smithi045 conmon[33655]: [DBG] pgmap v5921: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:43.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:42 smithi154 conmon[34553]: cluster 2021-07-31T03:57:41.661477+0000 mgr.x (mgr.24542) 5595 : cluster 2021-07-31T03:57:43.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:42 smithi154 conmon[34553]: [DBG] pgmap v5922: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:43.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:42 smithi045 conmon[33655]: cluster 2021-07-31T03:57:41.661477+0000 mgr.x (mgr.24542) 5595 : cluster [DBG] pgmap v5922: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:43.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:42 smithi045 conmon[39718]: cluster 2021-07-31T03:57:41.661477+0000 mgr.x (mgr.24542) 5595 : cluster [DBG] pgmap v5922: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:45.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:44 smithi154 conmon[34553]: cluster 2021-07-31T03:57:43.661846+0000 mgr.x (mgr.24542) 5596 : cluster [DBG] pgmap v5923: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:45.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:44 smithi154 conmon[34553]: 2021-07-31T03:57:45.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:44 smithi045 conmon[39718]: cluster 2021-07-31T03:57:43.661846+0000 mgr.x (mgr.24542) 5596 : cluster [DBG] pgmap v5923: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:45.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:44 smithi045 conmon[33655]: cluster 2021-07-31T03:57:43.661846+0000 mgr.x (mgr 2021-07-31T03:57:45.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:44 smithi045 conmon[33655]: .24542) 5596 : cluster [DBG] pgmap v5923: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:47.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:46 smithi154 conmon[34553]: cluster 2021-07-31T03:57:45.662452+0000 mgr.x (mgr.24542) 5597 : 2021-07-31T03:57:47.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:46 smithi154 conmon[34553]: cluster [DBG] pgmap v5924: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:47.274 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:46 smithi045 conmon[33655]: cluster 2021-07-31T03:57:45.662452+0000 mgr.x (mgr.24542) 5597 : cluster 2021-07-31T03:57:47.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:46 smithi045 conmon[33655]: [DBG] pgmap v5924: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:47.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:46 smithi045 conmon[39718]: cluster 2021-07-31T03:57:45.662452+0000 mgr.x 2021-07-31T03:57:47.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:46 smithi045 conmon[39718]: (mgr.24542) 5597 : cluster [DBG] pgmap v5924: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:49.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:48 smithi154 conmon[34553]: cluster 2021-07-31T03:57:47.663016+0000 mgr.x (mgr.24542) 5598 : cluster [DBG] pgmap v5925: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:49.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:48 smithi045 conmon[33655]: cluster 2021-07-31T03:57:47.663016+0000 mgr.x (mgr 2021-07-31T03:57:49.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:48 smithi045 conmon[33655]: .24542) 5598 : cluster [DBG] pgmap v5925: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:49.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:48 smithi045 conmon[39718]: cluster 2021-07-31T03:57:47.663016+0000 mgr.x ( 2021-07-31T03:57:49.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:48 smithi045 conmon[39718]: mgr.24542) 5598 : cluster [DBG] pgmap v5925: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:51.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:50 smithi154 conmon[34553]: cluster 2021-07-31T03:57:49.663486+0000 mgr.x (mgr.24542) 5599 : cluster [DBG] pgmap v5926: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:51.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:50 smithi045 conmon[33655]: cluster 2021-07-31T03:57:49.663486+0000 mgr.x (mgr.24542) 5599 2021-07-31T03:57:51.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:50 smithi045 conmon[33655]: : cluster [DBG] pgmap v5926: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:51.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:50 smithi045 conmon[39718]: cluster 2021-07-31T03:57:49.663486+0000 2021-07-31T03:57:51.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:50 smithi045 conmon[39718]: mgr.x (mgr.24542) 5599 : cluster [DBG] pgmap v5926: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:52.368 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:57:52.641 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:57:52.642 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 2h ago 2h - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:57:52.642 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 2h ago 2h 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:57:52.642 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 2h ago 2h 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:57:52.643 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 2h ago 2h - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:57:52.643 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:57:52.643 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 2h ago 2h 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:57:52.644 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:57:52.644 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 2h ago 2h - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:57:52.644 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 2h ago 2h 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:57:52.645 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:57:52.645 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:57:52.645 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:57:52.645 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:57:52.646 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 2h ago 2h 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:57:52.646 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 2h ago 2h 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:57:52.646 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 2h ago 2h 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:57:52.647 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 2h ago 2h 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:57:52.647 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 2h ago 2h 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:57:52.980 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:57:52.981 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:57:52.981 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:57:52.981 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:57:52.982 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:57:52.982 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:57:52.982 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:57:52.982 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:57:52.983 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:57:52.983 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:57:52.983 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:57:52.984 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:57:52.984 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:57:52.984 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:57:52.985 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:57:52.985 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:57:52.985 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:57:53.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:52 smithi154 conmon[34553]: cluster 2021-07-31T03:57:51.663965+0000 mgr.x (mgr.24542) 5600 : cluster [DBG] pgmap v5927: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:53.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:52 smithi045 conmon[33655]: cluster 2021-07-31T03:57:51.663965+0000 mgr.x (mgr. 2021-07-31T03:57:53.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:52 smithi045 conmon[33655]: 24542) 5600 : cluster [DBG] pgmap v5927: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:53.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:52 smithi045 conmon[39718]: cluster 2021-07-31T03:57:51 2021-07-31T03:57:53.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:52 smithi045 conmon[39718]: .663965+0000 mgr.x (mgr.24542) 5600 : cluster [DBG] pgmap v5927: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:54.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:53 smithi154 conmon[34553]: audit 2021-07-31T03:57:52.356175+0000 mgr.x (mgr.24542) 5601 : audit [DBG] 2021-07-31T03:57:54.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:53 smithi154 conmon[34553]: from='client.30541 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:57:54.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:53 smithi154 conmon[34553]: audit 2021-07-31T03:57:52.634123+0000 mgr.x (mgr.24542) 5602 : audit [DBG] from='client.20610 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:57:54.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:53 smithi154 conmon[34553]: audit 2021-07-31T03:57:52 2021-07-31T03:57:54.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:53 smithi154 conmon[34553]: .981293+0000 mon.a (mon.0) 3399 : audit [DBG] from='client.? 172.21.15.45:0/4057724581' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:57:54.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:53 smithi045 conmon[39718]: audit 2021-07-31T03:57:52.356175+0000 mgr.x (mgr.24542) 2021-07-31T03:57:54.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:53 smithi045 conmon[39718]: 5601 : audit [DBG] from='client.30541 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:57:54.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:53 smithi045 conmon[39718]: audit 2021-07-31T03:57:52.634123+0000 mgr.x (mgr 2021-07-31T03:57:54.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:53 smithi045 conmon[39718]: .24542) 5602 : audit [DBG] from='client.20610 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:57:54.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:53 smithi045 conmon[39718]: audit 2021-07-31T03:57:52.981293 2021-07-31T03:57:54.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:53 smithi045 conmon[39718]: +0000 mon.a (mon.0) 3399 : audit [DBG] 2021-07-31T03:57:54.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:53 smithi045 conmon[39718]: from='client.? 172.21.15.45:0/4057724581' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:57:54.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:53 smithi045 conmon[33655]: audit 2021-07-31T03:57:52.356175+0000 mgr.x 2021-07-31T03:57:54.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:53 smithi045 conmon[33655]: (mgr.24542) 5601 : audit [DBG] from='client.30541 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:57:54.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:53 smithi045 conmon[33655]: audit 2021-07-31T03:57:52.634123+0000 mgr.x (mgr.24542) 5602 : audit [DBG] from='client.20610 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:57:54.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:53 smithi045 conmon[33655]: audit 2021-07-31T03:57:52.981293+0000 mon.a (mon.0) 3399 : audit [DBG] from='client.? 172.21.15.45:0/4057724581' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:57:55.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:54 smithi154 conmon[34553]: cluster 2021-07-31T03:57:53.664301+0000 mgr.x (mgr.24542) 5603 : cluster 2021-07-31T03:57:55.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:54 smithi154 conmon[34553]: [DBG] pgmap v5928: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:55.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:54 smithi045 conmon[39718]: cluster 2021-07-31T03:57:53.664301+0000 mgr.x (mgr 2021-07-31T03:57:55.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:54 smithi045 conmon[39718]: .24542) 5603 : cluster [DBG] pgmap v5928: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:55.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:54 smithi045 conmon[33655]: cluster 2021-07-31T03:57:53.664301+0000 mgr.x (mgr.24542) 5603 2021-07-31T03:57:55.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:54 smithi045 conmon[33655]: : cluster [DBG] pgmap v5928: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:57.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:56 smithi154 conmon[34553]: cluster 2021-07-31T03:57:55.664724+0000 mgr.x (mgr.24542) 5604 : cluster [DBG] 2021-07-31T03:57:57.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:56 smithi154 conmon[34553]: pgmap v5929: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:57.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:56 smithi045 conmon[33655]: cluster 2021-07-31T03:57:55.664724+0000 mgr.x (mgr.24542) 5604 : 2021-07-31T03:57:57.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:56 smithi045 conmon[33655]: cluster [DBG] pgmap v5929: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:57.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:56 smithi045 conmon[39718]: cluster 2021-07-31T03:57:55.664724+0000 2021-07-31T03:57:57.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:56 smithi045 conmon[39718]: mgr.x (mgr.24542) 5604 : cluster [DBG] pgmap v5929: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:59.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:57:58 smithi154 conmon[34553]: cluster 2021-07-31T03:57:57.665218+0000 mgr.x (mgr.24542) 5605 : cluster [DBG] pgmap v5930: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:59.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:57:58 smithi045 conmon[33655]: cluster 2021-07-31T03:57:57.665218+0000 mgr.x (mgr.24542) 5605 : cluster [DBG] pgmap v5930: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:57:59.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:57:58 smithi045 conmon[39718]: cluster 2021-07-31T03:57:57.665218+0000 mgr.x (mgr.24542) 5605 : cluster [DBG] pgmap v5930: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:01.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:00 smithi154 conmon[34553]: cluster 2021-07-31T03:57:59.665765+0000 mgr.x (mgr.24542) 5606 : cluster [DBG] pgmap v5931: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:01.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:00 smithi045 conmon[33655]: cluster 2021-07-31T03:57:59.665765+0000 mgr.x (mgr.24542) 5606 : cluster [DBG] pgmap v5931: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:01.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:00 smithi045 conmon[39718]: cluster 2021-07-31T03:57:59.665765 2021-07-31T03:58:01.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:00 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 5606 : cluster [DBG] pgmap v5931: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:03.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:02 smithi154 conmon[34553]: cluster 2021-07-31T03:58:01.666230+0000 mgr.x (mgr.24542) 5607 : cluster [DBG] pgmap v5932: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:03.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:02 smithi045 conmon[33655]: cluster 2021-07-31T03:58:01.666230+0000 mgr.x (mgr.24542) 5607 : cluster 2021-07-31T03:58:03.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:02 smithi045 conmon[33655]: [DBG] pgmap v5932: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:03.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:02 smithi045 conmon[39718]: cluster 2021-07-31T03:58:01. 2021-07-31T03:58:03.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:02 smithi045 conmon[39718]: 666230+0000 mgr.x (mgr.24542) 5607 : cluster [DBG] pgmap v5932: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:05.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:04 smithi154 conmon[34553]: cluster 2021-07-31T03:58:03.666631+0000 mgr.x (mgr.24542) 5608 : cluster [DBG] pgmap v5933: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:05.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:04 smithi045 conmon[33655]: cluster 2021-07-31T03:58:03.666631+0000 mgr.x (mgr.24542) 5608 : cluster [DBG] pgmap v5933: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:05.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:04 smithi045 conmon[39718]: cluster 2021-07-31T03:58: 2021-07-31T03:58:05.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:04 smithi045 conmon[39718]: 03.666631+0000 mgr.x (mgr.24542) 5608 : cluster [DBG] pgmap v5933: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:07.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:06 smithi154 conmon[34553]: cluster 2021-07-31T03:58:05.667070+0000 mgr.x (mgr.24542) 5609 : cluster [DBG] pgmap v5934: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:07.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:06 smithi045 conmon[33655]: cluster 2021-07-31T03:58:05.667070+0000 mgr.x (mgr.24542) 5609 : cluster [DBG] pgmap v5934: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:07.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:06 smithi045 conmon[39718]: cluster 2021-07-31T03:58:05.667070+0000 mgr.x (mgr.24542) 2021-07-31T03:58:07.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:06 smithi045 conmon[39718]: 5609 : cluster [DBG] pgmap v5934: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:09.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:08 smithi154 conmon[34553]: cluster 2021-07-31T03:58:07.667603+0000 mgr.x (mgr.24542) 5610 : cluster [DBG] pgmap v5935: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:09.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:08 smithi045 conmon[33655]: cluster 2021-07-31T03:58:07.667603+0000 mgr.x (mgr.24542) 5610 : cluster 2021-07-31T03:58:09.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:08 smithi045 conmon[33655]: [DBG] pgmap v5935: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:09.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:08 smithi045 conmon[39718]: cluster 2021-07-31T03:58:07.667603+0000 2021-07-31T03:58:09.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:08 smithi045 conmon[39718]: mgr.x (mgr.24542) 5610 : cluster [DBG] pgmap v5935: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:11.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:10 smithi154 conmon[34553]: cluster 2021-07-31T03:58:09.668151+0000 mgr.x (mgr.24542) 5611 : 2021-07-31T03:58:11.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:10 smithi154 conmon[34553]: cluster [DBG] pgmap v5936: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:11.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:10 smithi045 conmon[33655]: cluster 2021-07-31T03:58:09.668151+0000 mgr.x (mgr.24542) 5611 : cluster [DBG] pgmap v5936: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:11.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:10 smithi045 conmon[39718]: cluster 2021-07-31T03:58:09 2021-07-31T03:58:11.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:10 smithi045 conmon[39718]: .668151+0000 mgr.x (mgr.24542) 5611 : cluster [DBG] pgmap v5936: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:13.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:12 smithi154 conmon[34553]: cluster 2021-07-31T03:58:11.668598+0000 mgr.x (mgr.24542) 5612 : cluster [DBG] 2021-07-31T03:58:13.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:12 smithi154 conmon[34553]: pgmap v5937: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:13.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:12 smithi045 conmon[33655]: cluster 2021-07-31T03:58:11.668598+0000 mgr.x (mgr.24542) 5612 : cluster [DBG] pgmap v5937: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:13.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:12 smithi045 conmon[39718]: cluster 2021-07-31T03:58:11.668598+0000 mgr.x ( 2021-07-31T03:58:13.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:12 smithi045 conmon[39718]: mgr.24542) 5612 : cluster [DBG] pgmap v5937: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:15.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:14 smithi154 conmon[34553]: cluster 2021-07-31T03:58:13.669018+0000 mgr.x (mgr.24542) 5613 : cluster 2021-07-31T03:58:15.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:14 smithi154 conmon[34553]: [DBG] pgmap v5938: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:15.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:14 smithi045 conmon[33655]: cluster 2021-07-31T03:58:13.669018+0000 mgr.x 2021-07-31T03:58:15.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:14 smithi045 conmon[33655]: (mgr.24542) 5613 : cluster [DBG] pgmap v5938: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:15.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:14 smithi045 conmon[39718]: cluster 2021-07-31T03:58:13.669018+0000 mgr.x ( 2021-07-31T03:58:15.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:14 smithi045 conmon[39718]: mgr.24542) 5613 : cluster [DBG] pgmap v5938: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:17.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:16 smithi154 conmon[34553]: cluster 2021-07-31T03:58:15.669501+0000 mgr.x (mgr.24542) 5614 : cluster [DBG] 2021-07-31T03:58:17.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:16 smithi154 conmon[34553]: pgmap v5939: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:17.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:16 smithi045 conmon[33655]: cluster 2021-07-31T03:58:15.669501+0000 mgr.x (mgr.24542) 2021-07-31T03:58:17.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:16 smithi045 conmon[33655]: 5614 : cluster [DBG] pgmap v5939: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:17.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:16 smithi045 conmon[39718]: cluster 2021-07-31T03:58:15.669501+0000 mgr.x (mgr 2021-07-31T03:58:17.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:16 smithi045 conmon[39718]: .24542) 5614 : cluster [DBG] pgmap v5939: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:19.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:18 smithi154 conmon[34553]: cluster 2021-07-31T03:58:17.669900+0000 mgr.x (mgr.24542) 5615 : cluster [DBG] pgmap v5940: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:19.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:18 smithi154 conmon[34553]: 2021-07-31T03:58:19.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:18 smithi045 conmon[33655]: cluster 2021-07-31T03:58:17.669900+0000 mgr.x (mgr.24542) 5615 : cluster 2021-07-31T03:58:19.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:18 smithi045 conmon[33655]: [DBG] pgmap v5940: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:19.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:18 smithi045 conmon[39718]: cluster 2021-07-31T03:58:17 2021-07-31T03:58:19.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:18 smithi045 conmon[39718]: .669900+0000 mgr.x (mgr.24542) 5615 : cluster [DBG] pgmap v5940: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:21.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:20 smithi154 conmon[34553]: cluster 2021-07-31T03:58:19.670548+0000 mgr.x (mgr.24542) 5616 : cluster [DBG] pgmap v5941: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:21.274 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:20 smithi045 conmon[39718]: cluster 2021-07-31T03:58: 2021-07-31T03:58:21.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:20 smithi045 conmon[39718]: 19.670548+0000 mgr.x (mgr.24542) 5616 : cluster [DBG] pgmap v5941: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:21.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:20 smithi045 conmon[33655]: cluster 2021-07-31T03:58:19.670548+0000 mgr.x (mgr.24542) 2021-07-31T03:58:21.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:20 smithi045 conmon[33655]: 5616 : cluster [DBG] pgmap v5941: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:23.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:22 smithi154 conmon[34553]: cluster 2021-07-31T03:58:21.671034+0000 mgr.x (mgr.24542) 5617 : cluster 2021-07-31T03:58:23.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:22 smithi154 conmon[34553]: [DBG] pgmap v5942: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:23.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:22 smithi045 conmon[39718]: cluster 2021-07-31T03:58:21.671034+0000 mgr.x (mgr.24542) 5617 : cluster [DBG] pgmap v5942: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:23.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:22 smithi045 conmon[33655]: cluster 2021-07-31T03:58:21.671034+0000 mgr.x (mgr.24542) 5617 : 2021-07-31T03:58:23.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:22 smithi045 conmon[33655]: cluster [DBG] pgmap v5942: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:23.293 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:58:23.568 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:58:23.568 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 2h ago 2h - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:58:23.568 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 2h ago 2h 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:58:23.569 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 2h ago 2h 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:58:23.569 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 2h ago 2h - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:58:23.569 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:58:23.570 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 2h ago 2h 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:58:23.570 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:58:23.570 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 2h ago 2h - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:58:23.571 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 2h ago 2h 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:58:23.571 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:58:23.571 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:58:23.571 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:58:23.572 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:58:23.572 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 2h ago 2h 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:58:23.572 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 2h ago 2h 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:58:23.573 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 2h ago 2h 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:58:23.573 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 2h ago 2h 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:58:23.573 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 2h ago 2h 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:58:23.908 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:58:23.908 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:58:23.909 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:58:23.909 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:58:23.909 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:58:23.910 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:58:23.910 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:58:23.910 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:58:23.911 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:58:23.911 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:58:23.911 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:58:23.912 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:58:23.912 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:58:23.912 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:58:23.913 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:58:23.913 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:58:23.913 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:58:24.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:23 smithi154 conmon[34553]: audit 2021-07-31T03:58:23.281860+0000 mgr.x (mgr.24542) 5618 : audit [DBG] 2021-07-31T03:58:24.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:23 smithi154 conmon[34553]: from='client.30559 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:58:24.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:23 smithi154 conmon[34553]: audit 2021-07-31T03:58:23.909204+0000 mon.c (mon.1) 238 : audit [DBG] from='client.? 172.21.15.45:0/3792778185' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:58:24.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:23 smithi045 conmon[33655]: audit 2021-07-31T03: 2021-07-31T03:58:24.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:23 smithi045 conmon[33655]: 58:23.281860+0000 mgr.x (mgr.24542) 5618 : audit [DBG] from='client.30559 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:58:24.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:23 smithi045 conmon[33655]: audit 2021- 2021-07-31T03:58:24.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:23 smithi045 conmon[33655]: 07-31T03:58:23.909204+0000 2021-07-31T03:58:24.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:23 smithi045 conmon[33655]: mon.c (mon.1) 238 : audit [DBG] from='client.? 172.21.15.45:0/3792778185' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:58:24.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:23 smithi045 conmon[39718]: audit 2021-07-31T03:58:23.281860 2021-07-31T03:58:24.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:23 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 5618 : audit [DBG] from='client.30559 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:58:24.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:23 smithi045 conmon[39718]: audit 2021-07-31T03:58:24.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:23 smithi045 conmon[39718]: 2021-07-31T03:58:23.909204+0000 mon.c ( 2021-07-31T03:58:24.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:23 smithi045 conmon[39718]: mon.1) 238 : audit [DBG] from='client.? 172.21.15.45:0/3792778185' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:58:24.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:23 smithi045 conmon[39718]: 2021-07-31T03:58:25.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:24 smithi154 conmon[34553]: audit 2021-07-31T03:58:23.560718+0000 mgr.x (mgr.24542) 5619 : audit [DBG] 2021-07-31T03:58:25.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:24 smithi154 conmon[34553]: from='client.20628 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:58:25.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:24 smithi154 conmon[34553]: cluster 2021-07-31T03:58:23.671367+0000 mgr.x (mgr.24542) 5620 : cluster [DBG] pgmap v5943: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:25.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:24 smithi045 conmon[39718]: audit 2021-07-31T03:58:23.560718+0000 2021-07-31T03:58:25.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:24 smithi045 conmon[39718]: mgr.x (mgr.24542) 5619 : audit [DBG] from='client.20628 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:58:25.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:24 smithi045 conmon[39718]: cluster 2021-07-31T03:58:25.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:24 smithi045 conmon[39718]: 2021-07-31T03:58:23.671367+0000 2021-07-31T03:58:25.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:24 smithi045 conmon[39718]: mgr.x (mgr.24542) 5620 : cluster [DBG] 2021-07-31T03:58:25.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:24 smithi045 conmon[39718]: pgmap v5943: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:25.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:24 smithi045 conmon[33655]: audit 2021-07-31T03:58:23.560718+0000 2021-07-31T03:58:25.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:24 smithi045 conmon[33655]: mgr.x (mgr.24542) 5619 : audit [DBG] from='client.20628 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:58:25.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:24 smithi045 conmon[33655]: cluster 2021-07-31T03:58:23.671367+0000 2021-07-31T03:58:25.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:24 smithi045 conmon[33655]: mgr.x (mgr.24542) 5620 : cluster [DBG] pgmap v5943: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:27.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:26 smithi154 conmon[34553]: cluster 2021-07-31T03:58:25.671904+0000 mgr.x (mgr.24542) 5621 : cluster 2021-07-31T03:58:27.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:26 smithi154 conmon[34553]: [DBG] pgmap v5944: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:27.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:26 smithi045 conmon[39718]: cluster 2021-07-31T03:58:25.671904+0000 mgr.x 2021-07-31T03:58:27.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:26 smithi045 conmon[39718]: (mgr.24542) 5621 : cluster [DBG] pgmap v5944: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:27.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:26 smithi045 conmon[33655]: cluster 2021-07-31T03:58:25.671904+0000 mgr.x 2021-07-31T03:58:27.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:26 smithi045 conmon[33655]: (mgr.24542) 5621 : cluster [DBG] pgmap v5944: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:29.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:28 smithi154 conmon[34553]: cluster 2021-07-31T03:58:27.672456+0000 mgr.x (mgr.24542) 5622 : cluster [DBG] pgmap v5945: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:29.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:28 smithi154 conmon[34553]: audit 2021-07-31T03:58:28.454752+0000 mon.b (mon.2) 730 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:58:29.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:28 smithi154 conmon[34553]: audit 2021-07-31T03: 2021-07-31T03:58:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:28 smithi154 conmon[34553]: 58:28.455093+0000 mon.b (mon.2) 731 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:58:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:28 smithi154 conmon[34553]: audit 2021-07 2021-07-31T03:58:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:28 smithi154 conmon[34553]: -31T03:58:28.455498+0000 mon.a ( 2021-07-31T03:58:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:28 smithi154 conmon[34553]: mon.0) 3400 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:58:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:28 smithi154 conmon[34553]: 2021-07-31T03:58:29.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:28 smithi154 conmon[34553]: audit 2021-07 2021-07-31T03:58:29.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:28 smithi154 conmon[34553]: -31T03:58:28.455893+0000 2021-07-31T03:58:29.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:28 smithi154 conmon[34553]: mon.a (mon.0) 3401 : audit 2021-07-31T03:58:29.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:28 smithi154 conmon[34553]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:58:29.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:28 smithi045 conmon[39718]: cluster 2021-07-31T03:58:27.672456+0000 mgr.x (mgr.24542) 5622 : cluster [DBG] pgmap v5945: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:29.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:28 smithi045 conmon[39718]: audit 2021-07-31T03:58:28.454752+0000 mon.b (mon.2) 730 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:58:29.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:28 smithi045 conmon[39718]: audit 2021- 2021-07-31T03:58:29.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:28 smithi045 conmon[39718]: 07-31T03:58:28.455093+0000 2021-07-31T03:58:29.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:28 smithi045 conmon[39718]: mon.b (mon.2) 731 : audit [INF] 2021-07-31T03:58:29.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:28 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:58:29.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:28 smithi045 conmon[39718]: audit 2021-07-31T03:58:28.455498+0000 2021-07-31T03:58:29.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:28 smithi045 conmon[39718]: mon.a (mon.0) 3400 : audit [INF] 2021-07-31T03:58:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:28 smithi045 conmon[39718]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:58:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:28 smithi045 conmon[39718]: audit 2021-07-31T03:58 2021-07-31T03:58:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:28 smithi045 conmon[39718]: :28.455893+0000 mon.a (mon.0 2021-07-31T03:58:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:28 smithi045 conmon[39718]: ) 3401 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:58:29.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:28 smithi045 conmon[33655]: cluster 2021-07-31T03:58:27.672456+0000 mgr.x (mgr.24542) 5622 2021-07-31T03:58:29.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:28 smithi045 conmon[33655]: : cluster [DBG] pgmap v5945: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:29.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:28 smithi045 conmon[33655]: audit 2021-07-31T03:58:28.454752+0000 mon.b (mon.2) 730 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:58:29.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:28 smithi045 conmon[33655]: audit 2021-07-31T03 2021-07-31T03:58:29.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:28 smithi045 conmon[33655]: :58:28.455093+0000 mon.b ( 2021-07-31T03:58:29.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:28 smithi045 conmon[33655]: mon.2) 731 : audit 2021-07-31T03:58:29.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:28 smithi045 conmon[33655]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:58:29.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:28 smithi045 conmon[33655]: audit 2021-07- 2021-07-31T03:58:29.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:28 smithi045 conmon[33655]: 31T03:58:28.455498+0000 2021-07-31T03:58:29.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:28 smithi045 conmon[33655]: mon.a (mon.0) 3400 : audit 2021-07-31T03:58:29.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:28 smithi045 conmon[33655]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:58:29.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:28 smithi045 conmon[33655]: audit 2021- 2021-07-31T03:58:29.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:28 smithi045 conmon[33655]: 07-31T03:58:28. 2021-07-31T03:58:29.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:28 smithi045 conmon[33655]: 455893+0000 mon.a (mon.0) 2021-07-31T03:58:29.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:28 smithi045 conmon[33655]: 3401 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:58:31.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:30 smithi154 conmon[34553]: cluster 2021-07-31T03:58:29.672908+0000 mgr.x (mgr.24542) 5623 : cluster [DBG] pgmap v5946: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:31.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:30 smithi045 conmon[39718]: cluster 2021-07-31T03:58:29.672908+0000 mgr.x ( 2021-07-31T03:58:31.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:30 smithi045 conmon[39718]: mgr.24542) 5623 : cluster [DBG] pgmap v5946: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:31.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:30 smithi045 conmon[33655]: cluster 2021-07-31T03:58:29.672908+0000 mgr.x (mgr.24542) 5623 : cluster 2021-07-31T03:58:31.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:30 smithi045 conmon[33655]: [DBG] pgmap v5946: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:33.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:32 smithi154 conmon[34553]: cluster 2021-07-31T03:58:31.673437+0000 mgr.x (mgr.24542) 5624 : cluster [DBG] pgmap v5947: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:33.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:32 smithi045 conmon[39718]: cluster 2021-07-31T03:58:31.673437+0000 mgr.x ( 2021-07-31T03:58:33.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:32 smithi045 conmon[39718]: mgr.24542) 5624 : cluster [DBG] pgmap v5947: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:33.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:32 smithi045 conmon[33655]: cluster 2021-07-31T03:58:31.673437+0000 mgr.x 2021-07-31T03:58:33.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:32 smithi045 conmon[33655]: (mgr.24542) 5624 : cluster [DBG] pgmap v5947: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:35.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:34 smithi154 conmon[34553]: cluster 2021-07-31T03:58:33.673980+0000 mgr.x (mgr.24542) 5625 : cluster [DBG] pgmap v5948: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:35.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:34 smithi045 conmon[39718]: cluster 2021-07-31T03:58:33.673980+0000 mgr.x (mgr 2021-07-31T03:58:35.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:34 smithi045 conmon[39718]: .24542) 5625 : cluster [DBG] pgmap v5948: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:35.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:34 smithi045 conmon[33655]: cluster 2021- 2021-07-31T03:58:35.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:34 smithi045 conmon[33655]: 07-31T03:58:33.673980+0000 mgr.x (mgr.24542) 5625 : cluster [DBG] pgmap v5948: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:36.502 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: debug 2021-07-31T03:58:36.198+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:58:36.503 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:58:36.503 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:58:36.503 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: yield 2021-07-31T03:58:36.503 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:58:36.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:58:36.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:58:36.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:58:36.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:58:36.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:58:36.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:58:36.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:58:36.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:58:36.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: task.result() 2021-07-31T03:58:36.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:58:36.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: debug 2021-07-31T03:58:36.210+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:58:36.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:58:36.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:58:36.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: yield 2021-07-31T03:58:36.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:58:36.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:58:36.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:58:36.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:58:36.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:58:36.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:58:36.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:58:36.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:58:36.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:58:36.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: task.result() 2021-07-31T03:58:36.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:58:36.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: debug 2021-07-31T03:58:36.236+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:58:36.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:58:36.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:58:36.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: yield 2021-07-31T03:58:36.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:58:36.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:58:36.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:58:36.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:58:36.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:58:36.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:58:36.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:58:36.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:58:36.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:58:36.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: task.result() 2021-07-31T03:58:36.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:58:36.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: debug 2021-07-31T03:58:36.236+0000 7fcb93160700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:58:36.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:58:36.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:58:36.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: yield 2021-07-31T03:58:36.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:58:36.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:58:36.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:58:36.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:58:36.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:58:36.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:58:36.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:58:36.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:58:36.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:58:36.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: task.result() 2021-07-31T03:58:36.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:58:36.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: 2021-07-31T03:58:36.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:58:36.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: 2021-07-31T03:58:36.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:58:36.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:58:36.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:58:36.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:58:36.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:58:36.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:58:36.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:58:36.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:58:36.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:58:36.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:58:36.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:58:36.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:58:36.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:58:36.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:58:36.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:58:36.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:58:36.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:58:36.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:58:36.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: raise 2021-07-31T03:58:36.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:58:36.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:58:36.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:58:36.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:58:36.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:58:36.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: debug 2021-07-31T03:58:36.244+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:58:36.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:58:36.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:58:36.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: yield 2021-07-31T03:58:36.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:58:36.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:58:36.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:58:36.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:58:36.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:58:36.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:58:36.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:58:36.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:58:36.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:58:36.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: task.result() 2021-07-31T03:58:36.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:58:36.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: debug 2021-07-31T03:58:36.245+0000 7fcb9295f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:58:36.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:58:36.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:58:36.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: yield 2021-07-31T03:58:36.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:58:36.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:58:36.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:58:36.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:58:36.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:58:36.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:58:36.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:58:36.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:58:36.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:58:36.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: task.result() 2021-07-31T03:58:36.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:58:36.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: 2021-07-31T03:58:36.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:58:36.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: 2021-07-31T03:58:36.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:58:36.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:58:36.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:58:36.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:58:36.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:58:36.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:58:36.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:58:36.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:58:36.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:58:36.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:58:36.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:58:36.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:58:36.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:58:36.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:58:36.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:58:36.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:58:36.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:58:36.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:58:36.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: raise 2021-07-31T03:58:36.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:58:36.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:58:36.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:58:36.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:58:36.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:58:37.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: cluster 2021-07-31T03:58:35.674453+0000 mgr.x (mgr.24542) 5626 : cluster [DBG] pgmap v5949: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:37.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: 2021-07-31T03:58:37.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: audit 2021-07-31T03:58:36.152386+0000 mon.b (mon.2) 732 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:58:37.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: audit 2021-07-31T03:58 2021-07-31T03:58:37.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: :36.153777+0000 mon.b (mon.2) 733 : audit 2021-07-31T03:58:37.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:58:37.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: audit 2021-07 2021-07-31T03:58:37.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: -31T03:58:36.154935+0000 mon.b ( 2021-07-31T03:58:37.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: mon.2) 734 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:58:37.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: cephadm 2021-07-31T03:58:37.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: 2021-07-31T03:58:36.155677 2021-07-31T03:58:37.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: +0000 mgr.x (mgr.24542) 5627 : cephadm [WRN] 2021-07-31T03:58:37.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:58:37.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: cephadm 2021-07-31T03:58 2021-07-31T03:58:37.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: :36.199629+0000 mgr.x (mgr.24542) 5628 2021-07-31T03:58:37.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: : cephadm [ERR] 2021-07-31T03:58:37.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:58:37.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:58:37.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: yield 2021-07-31T03:58:37.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:58:37.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:58:37.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:58:37.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:58:37.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:58:37.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:58:37.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:58:37.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:58:37.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:58:37.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: task.result() 2021-07-31T03:58:37.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:58:37.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: cephadm 2021-07-31T03:58:36.200220+0000 mgr.x (mgr.24542) 5629 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:58:37.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: cephadm 2021-07-31T03:58:36.211255+0000 mgr.x (mgr.24542) 5630 2021-07-31T03:58:37.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: : cephadm [ERR] 2021-07-31T03:58:37.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:58:37.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:58:37.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: yield 2021-07-31T03:58:37.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:58:37.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:58:37.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:58:37.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:58:37.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:58:37.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:58:37.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:58:37.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:58:37.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:58:37.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: task.result() 2021-07-31T03:58:37.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:58:37.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: cephadm 2021-07-31T03:58:36.212180+0000 mgr.x (mgr.24542) 5631 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:58:37.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: cephadm 2021-07-31T03:58:36.236950+0000 mgr.x (mgr.24542) 5632 : cephadm [ERR] 2021-07-31T03:58:37.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:58:37.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:58:37.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: yield 2021-07-31T03:58:37.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:58:37.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:58:37.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:58:37.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:58:37.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:58:37.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:58:37.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:58:37.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:58:37.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:58:37.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: task.result() 2021-07-31T03:58:37.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:58:37.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: cephadm 2021-07-31T03:58:36.237367+0000 mgr.x (mgr.24542) 5633 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:58:37.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:58:37.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:58:37.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: yield 2021-07-31T03:58:37.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:58:37.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:58:37.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:58:37.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:58:37.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:58:37.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:58:37.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:58:37.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:58:37.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:58:37.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: task.result() 2021-07-31T03:58:37.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:58:37.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: 2021-07-31T03:58:37.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:58:37.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: 2021-07-31T03:58:37.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:58:37.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:58:37.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:58:37.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:58:37.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:58:37.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:58:37.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:58:37.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:58:37.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:58:37.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:58:37.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:58:37.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:58:37.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:58:37.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:58:37.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:58:37.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:58:37.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:58:37.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:58:37.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: raise 2021-07-31T03:58:37.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:58:37.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:58:37.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:58:37.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:58:37.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:58:37.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: cephadm 2021-07-31T03:58:36. 2021-07-31T03:58:37.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: 245808+0000 mgr.x (mgr.24542) 5634 : cephadm [ERR] 2021-07-31T03:58:37.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:58:37.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:58:37.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: yield 2021-07-31T03:58:37.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:58:37.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:58:37.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:58:37.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:58:37.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:58:37.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:58:37.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:58:37.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:58:37.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:58:37.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: task.result() 2021-07-31T03:58:37.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:58:37.285 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: cephadm 2021-07-31T03:58:36.246201+0000 mgr.x (mgr.24542) 5635 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:58:37.285 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:58:37.285 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:58:37.285 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: yield 2021-07-31T03:58:37.286 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:58:37.286 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:58:37.286 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:58:37.286 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:58:37.286 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:58:37.287 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:58:37.287 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:58:37.287 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:58:37.287 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:58:37.288 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: task.result() 2021-07-31T03:58:37.288 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:58:37.288 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: 2021-07-31T03:58:37.288 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:58:37.289 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: 2021-07-31T03:58:37.289 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:58:37.289 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:58:37.289 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:58:37.290 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:58:37.290 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:58:37.290 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:58:37.290 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:58:37.291 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:58:37.291 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:58:37.291 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:58:37.291 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:58:37.291 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:58:37.292 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:58:37.292 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:58:37.292 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:58:37.292 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:58:37.293 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:58:37.293 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:58:37.293 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: raise 2021-07-31T03:58:37.293 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:58:37.294 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:58:37.294 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:58:37.294 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:58:37.294 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:36 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:58:37.295 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: cluster 2021-07-31T03:58:35.674453+0000 mgr.x ( 2021-07-31T03:58:37.295 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: mgr.24542) 5626 : cluster [DBG] pgmap v5949: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:37.295 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: audit 2021-07-31T03:58:36.152386+0000 2021-07-31T03:58:37.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: mon.b (mon.2) 732 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:58:37.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T03:58:37.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: T03:58:36.153777+0000 mon.b ( 2021-07-31T03:58:37.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: mon.2) 733 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:58:37.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: 2021-07-31T03:58:37.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: audit 2021-07-31T03:58 2021-07-31T03:58:37.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: :36.154935+0000 mon.b (mon.2 2021-07-31T03:58:37.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: ) 734 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:58:37.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: cephadm 2021-07 2021-07-31T03:58:37.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: -31T03:58:36.155677+0000 2021-07-31T03:58:37.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: mgr.x (mgr.24542) 5627 : cephadm [WRN] 2021-07-31T03:58:37.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:58:37.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: cephadm 2021-07-31T03 2021-07-31T03:58:37.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: :58:36.199629+0000 mgr.x (mgr 2021-07-31T03:58:37.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: .24542) 5628 : cephadm [ERR] 2021-07-31T03:58:37.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:58:37.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:58:37.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: yield 2021-07-31T03:58:37.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:58:37.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:58:37.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:58:37.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:58:37.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:58:37.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:58:37.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:58:37.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:58:37.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:58:37.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: task.result() 2021-07-31T03:58:37.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:58:37.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: cephadm 2021-07-31T03:58:36.200220+0000 mgr.x (mgr.24542) 5629 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:58:37.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: cephadm 2021- 2021-07-31T03:58:37.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: 07-31T03:58:36.211255+0000 mgr.x (mgr. 2021-07-31T03:58:37.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: 24542) 5630 : cephadm [ERR] 2021-07-31T03:58:37.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:58:37.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:58:37.304 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: yield 2021-07-31T03:58:37.304 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:58:37.304 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: cluster 2021-07-31T03:58:35.674453+0000 mgr.x (mgr.24542) 5626 2021-07-31T03:58:37.305 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: : cluster [DBG] pgmap v5949: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:37.305 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: audit 2021-07-31T03:58:36.152386+0000 mon.b (mon.2) 732 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:58:37.305 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: audit 2021-07-31T03:58:36.153777+0000 mon.b (mon.2) 733 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:58:37.305 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: audit 2021-07-31T03:58:37.306 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: 2021-07-31T 2021-07-31T03:58:37.306 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: 03:58:36.154935+0000 mon.b (mon.2) 734 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:58:37.306 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: cephadm 2021-07-31T03:58:36.155677+0000 2021-07-31T03:58:37.306 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: mgr.x (mgr.24542) 5627 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:58:37.307 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: cephadm 2021-07-31T03:58:37.307 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: 2021-07-31T03:58:36.199629+0000 mgr.x (mgr.24542) 5628 : cephadm [ERR] 2021-07-31T03:58:37.307 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:58:37.307 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:58:37.307 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: yield 2021-07-31T03:58:37.308 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:58:37.308 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:58:37.308 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:58:37.308 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:58:37.309 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:58:37.309 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:58:37.309 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:58:37.309 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:58:37.310 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:58:37.310 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: task.result() 2021-07-31T03:58:37.310 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:58:37.310 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: cephadm 2021-07-31T03:58:36.200220+0000 mgr.x (mgr.24542) 5629 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:58:37.310 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: cephadm 2021-07-31T03:58:36.211255+0000 mgr.x (mgr.24542) 5630 : cephadm [ERR] 2021-07-31T03:58:37.311 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:58:37.311 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:58:37.311 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: yield 2021-07-31T03:58:37.311 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:58:37.312 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:58:37.312 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:58:37.312 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:58:37.312 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:58:37.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:58:37.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:58:37.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:58:37.314 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:58:37.314 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:58:37.314 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:58:37.314 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:58:37.314 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:58:37.315 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: task.result() 2021-07-31T03:58:37.315 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:58:37.315 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: cephadm 2021-07-31T03:58:36.212180+0000 mgr.x (mgr.24542) 5631 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:58:37.315 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: cephadm 2021 2021-07-31T03:58:37.316 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: -07-31T03:58:36.236950+0000 mgr.x 2021-07-31T03:58:37.316 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: (mgr.24542) 5632 : cephadm [ERR] 2021-07-31T03:58:37.316 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:58:37.316 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:58:37.316 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: yield 2021-07-31T03:58:37.317 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:58:37.317 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:58:37.317 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:58:37.317 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:58:37.318 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:58:37.318 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:58:37.318 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:58:37.318 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:58:37.318 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:58:37.319 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: task.result() 2021-07-31T03:58:37.319 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:58:37.319 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: 2021-07-31T03:58:37.319 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: cephadm 2021-07-31T03:58:36.237367+0000 mgr.x (mgr.24542) 5633 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:58:37.320 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:58:37.320 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:58:37.320 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: yield 2021-07-31T03:58:37.320 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:58:37.320 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:58:37.321 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:58:37.321 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:58:37.321 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:58:37.321 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:58:37.322 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:58:37.322 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:58:37.322 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:58:37.322 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: task.result() 2021-07-31T03:58:37.322 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:58:37.323 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: 2021-07-31T03:58:37.323 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:58:37.323 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: 2021-07-31T03:58:37.323 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:58:37.323 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:58:37.324 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:58:37.324 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:58:37.324 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:58:37.324 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:58:37.325 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:58:37.325 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:58:37.325 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:58:37.325 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:58:37.325 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:58:37.326 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:58:37.326 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:58:37.326 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:58:37.326 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:58:37.326 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:58:37.327 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:58:37.327 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:58:37.327 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: raise 2021-07-31T03:58:37.327 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:58:37.328 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:58:37.328 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:58:37.328 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:58:37.328 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:58:37.328 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: cephadm 2021-07-31T03:58:36.245808+0000 mgr.x (mgr.24542) 5634 : cephadm [ERR] 2021-07-31T03:58:37.329 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:58:37.329 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:58:37.329 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: yield 2021-07-31T03:58:37.329 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:58:37.329 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:58:37.330 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:58:37.330 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:58:37.330 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:58:37.330 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:58:37.331 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:58:37.331 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:58:37.331 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:58:37.331 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:58:37.332 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:58:37.332 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: task.result() 2021-07-31T03:58:37.332 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:58:37.332 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: cephadm 2021-07-31T03:58:36.212180+0000 mgr.x (mgr.24542) 5631 : cephadm [INF] 2021-07-31T03:58:37.333 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:58:37.333 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: cephadm 2021-07-31T03:58:36.236950+0000 mgr.x (mgr.24542) 5632 : cephadm [ERR] 2021-07-31T03:58:37.333 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:58:37.333 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:58:37.333 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: yield 2021-07-31T03:58:37.334 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:58:37.334 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:58:37.334 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:58:37.334 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:58:37.335 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:58:37.335 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:58:37.335 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:58:37.335 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:58:37.335 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:58:37.336 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: task.result() 2021-07-31T03:58:37.336 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:58:37.336 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: cephadm 2021-07-31T03:58:36.237367+0000 mgr.x (mgr.24542) 5633 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:58:37.336 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:58:37.336 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:58:37.337 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: yield 2021-07-31T03:58:37.337 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:58:37.337 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:58:37.337 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:58:37.338 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:58:37.338 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:58:37.338 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:58:37.338 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:58:37.338 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:58:37.339 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:58:37.339 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: task.result() 2021-07-31T03:58:37.339 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:58:37.339 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: 2021-07-31T03:58:37.339 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:58:37.340 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: 2021-07-31T03:58:37.340 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:58:37.341 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:58:37.341 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: task.result() 2021-07-31T03:58:37.341 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:58:37.341 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: cephadm 2021-07-31T03:58:36.246201+0000 mgr.x (mgr.24542) 5635 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:58:37.341 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:58:37.342 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:58:37.342 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: yield 2021-07-31T03:58:37.342 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:58:37.342 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:58:37.343 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:58:37.343 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:58:37.343 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:58:37.343 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:58:37.343 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:58:37.344 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:58:37.344 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:58:37.344 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: task.result() 2021-07-31T03:58:37.344 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:58:37.345 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: 2021-07-31T03:58:37.345 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:58:37.345 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: 2021-07-31T03:58:37.345 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:58:37.345 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:58:37.346 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:58:37.346 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:58:37.346 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:58:37.346 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:58:37.347 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:58:37.347 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:58:37.347 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:58:37.347 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:58:37.347 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:58:37.348 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:58:37.348 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:58:37.348 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:58:37.348 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:58:37.348 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:58:37.349 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:58:37.349 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:58:37.349 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: raise 2021-07-31T03:58:37.349 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:58:37.349 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:58:37.350 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:58:37.350 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:58:37.350 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:58:37.351 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:58:37.351 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:58:37.351 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:58:37.352 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:58:37.352 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:58:37.352 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:58:37.352 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:58:37.352 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:58:37.353 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:58:37.353 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:58:37.353 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:58:37.353 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:58:37.353 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:58:37.354 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:58:37.354 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:58:37.354 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:58:37.354 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:58:37.355 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:58:37.355 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: raise 2021-07-31T03:58:37.355 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:58:37.355 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:58:37.355 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:58:37.356 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:58:37.356 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:58:37.356 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: cephadm 2021-07-31T03:58 2021-07-31T03:58:37.356 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: :36.245808+0000 mgr.x (mgr.24542) 5634 : cephadm [ERR] 2021-07-31T03:58:37.356 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:58:37.357 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:58:37.357 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: yield 2021-07-31T03:58:37.357 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:58:37.357 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:58:37.357 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:58:37.358 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:58:37.358 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:58:37.358 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:58:37.358 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:58:37.359 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:58:37.359 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:58:37.359 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: task.result() 2021-07-31T03:58:37.359 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:58:37.359 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: cephadm 2021-07-31T03:58:36.246201+0000 mgr.x (mgr.24542) 5635 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:58:37.360 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:58:37.360 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:58:37.360 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: yield 2021-07-31T03:58:37.360 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:58:37.360 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:58:37.361 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:58:37.361 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:58:37.361 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:58:37.361 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:58:37.361 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:58:37.362 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:58:37.362 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:58:37.362 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: task.result() 2021-07-31T03:58:37.362 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:58:37.362 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: 2021-07-31T03:58:37.363 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:58:37.363 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: 2021-07-31T03:58:37.363 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:58:37.363 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:58:37.363 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:58:37.364 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:58:37.364 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:58:37.364 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:58:37.364 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:58:37.364 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:58:37.365 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:58:37.365 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:58:37.365 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:58:37.365 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:58:37.365 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:58:37.366 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:58:37.366 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:58:37.366 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:58:37.366 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:58:37.366 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:58:37.367 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: raise 2021-07-31T03:58:37.367 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:58:37.367 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:58:37.367 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:58:37.368 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:58:37.368 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:36 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:58:39.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:38 smithi154 conmon[34553]: cluster 2021-07-31T03:58:37.674841+0000 mgr.x (mgr.24542) 5636 : cluster [DBG] pgmap v5950: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:39.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:38 smithi045 conmon[39718]: cluster 2021-07-31T03:58:37.674841+0000 mgr.x (mgr 2021-07-31T03:58:39.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:38 smithi045 conmon[39718]: .24542) 5636 : cluster [DBG] pgmap v5950: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:39.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:38 smithi045 conmon[33655]: cluster 2021-07-31T03:58:37.674841+0000 mgr.x 2021-07-31T03:58:39.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:38 smithi045 conmon[33655]: (mgr.24542) 5636 : cluster [DBG] pgmap v5950: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:41.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:40 smithi154 conmon[34553]: cluster 2021-07-31T03:58:39.675467+0000 mgr.x (mgr.24542) 5637 : cluster [DBG] 2021-07-31T03:58:41.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:40 smithi154 conmon[34553]: pgmap v5951: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:41.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:40 smithi045 conmon[39718]: cluster 2021-07-31T03:58:39.675467+0000 mgr.x (mgr.24542) 5637 : cluster [DBG] pgmap v5951: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:41.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:40 smithi045 conmon[33655]: cluster 2021- 2021-07-31T03:58:41.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:40 smithi045 conmon[33655]: 07-31T03:58:39.675467+0000 mgr.x (mgr.24542) 5637 : cluster [DBG] pgmap v5951: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:43.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:42 smithi154 conmon[34553]: cluster 2021-07-31T03:58:41.676033+0000 mgr.x (mgr.24542) 5638 : cluster [DBG] pgmap v5952: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:43.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:42 smithi045 conmon[39718]: cluster 2021-07-31T03:58:41.676033+0000 mgr.x 2021-07-31T03:58:43.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:42 smithi045 conmon[39718]: (mgr.24542) 5638 : cluster [DBG] pgmap v5952: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:43.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:42 smithi045 conmon[33655]: cluster 2021-07-31T03:58:41.676033+0000 mgr.x (mgr.24542) 2021-07-31T03:58:43.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:42 smithi045 conmon[33655]: 5638 : cluster [DBG] pgmap v5952: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:45.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:44 smithi154 conmon[34553]: cluster 2021-07-31T03:58:43.676546+0000 mgr.x (mgr.24542) 5639 : cluster [DBG] pgmap v5953: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:45.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:44 smithi045 conmon[39718]: cluster 2021-07-31T03:58:43.676546+0000 mgr.x 2021-07-31T03:58:45.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:44 smithi045 conmon[39718]: (mgr.24542) 5639 : cluster [DBG] pgmap v5953: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:45.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:44 smithi045 conmon[33655]: cluster 2021-07-31T03:58:43.676546+0000 mgr.x 2021-07-31T03:58:45.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:44 smithi045 conmon[33655]: (mgr.24542) 5639 : cluster [DBG] pgmap v5953: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:47.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:46 smithi154 conmon[34553]: cluster 2021-07-31T03:58:45.677178+0000 mgr.x (mgr.24542) 5640 : cluster [DBG] pgmap v5954: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:47.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:46 smithi045 conmon[33655]: cluster 2021-07-31T03:58:45.677178+0000 mgr.x (mgr 2021-07-31T03:58:47.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:46 smithi045 conmon[33655]: .24542) 5640 : cluster [DBG] pgmap v5954: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:47.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:46 smithi045 conmon[39718]: cluster 2021-07-31T03:58:45.677178+0000 mgr.x (mgr. 2021-07-31T03:58:47.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:46 smithi045 conmon[39718]: 24542) 5640 : cluster [DBG] pgmap v5954: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:49.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:48 smithi154 conmon[34553]: cluster 2021-07-31T03:58:47.677613+0000 mgr.x (mgr.24542) 5641 : cluster [DBG] pgmap v5955: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:49.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:48 smithi045 conmon[33655]: cluster 2021-07-31T03:58:47.677613+0000 mgr.x (mgr.24542) 5641 : cluster 2021-07-31T03:58:49.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:48 smithi045 conmon[33655]: [DBG] pgmap v5955: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:49.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:48 smithi045 conmon[39718]: cluster 2021-07-31T03:58:47.677613+0000 mgr.x (mgr.24542) 5641 2021-07-31T03:58:49.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:48 smithi045 conmon[39718]: : cluster [DBG] pgmap v5955: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:51.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:50 smithi154 conmon[34553]: cluster 2021-07-31T03:58:49.678069+0000 mgr.x (mgr.24542) 5642 : cluster [DBG] pgmap v5956: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:51.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:50 smithi045 conmon[39718]: cluster 2021-07-31T03:58:49. 2021-07-31T03:58:51.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:50 smithi045 conmon[39718]: 678069+0000 mgr.x (mgr.24542) 5642 : cluster [DBG] pgmap v5956: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:51.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:50 smithi045 conmon[33655]: cluster 2021-07-31T03:58:49.678069+0000 2021-07-31T03:58:51.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:50 smithi045 conmon[33655]: mgr.x (mgr.24542) 5642 : cluster [DBG] pgmap v5956: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:53.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:52 smithi154 conmon[34553]: cluster 2021-07-31T03:58:51.678547+0000 mgr.x (mgr.24542) 5643 : cluster [DBG] pgmap v5957: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:53.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:52 smithi045 conmon[33655]: cluster 2021-07-31T03:58:51.678547+0000 mgr.x (mgr.24542) 5643 : cluster [DBG] pgmap v5957: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:53.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:52 smithi045 conmon[39718]: cluster 2021-07-31T03:58:51.678547+0000 mgr.x (mgr.24542 2021-07-31T03:58:53.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:52 smithi045 conmon[39718]: ) 5643 : cluster [DBG] pgmap v5957: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:54.223 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:58:54.499 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:58:54.499 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 2h ago 2h - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:58:54.500 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 2h ago 2h 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:58:54.500 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 2h ago 2h 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:58:54.500 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 2h ago 2h - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:58:54.501 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:58:54.501 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 2h ago 2h 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:58:54.501 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:58:54.501 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 2h ago 2h - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:58:54.502 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 2h ago 2h 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:58:54.502 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:58:54.502 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:58:54.502 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:58:54.503 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:58:54.503 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 2h ago 2h 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:58:54.503 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 2h ago 2h 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:58:54.504 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 2h ago 2h 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:58:54.504 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 2h ago 2h 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:58:54.504 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 2h ago 2h 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:58:54.835 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:58:54.835 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:58:54.836 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:58:54.836 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:58:54.836 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:58:54.836 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:58:54.837 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:58:54.837 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:58:54.837 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:58:54.837 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:58:54.837 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:58:54.838 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:58:54.838 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:58:54.838 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:58:54.838 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:58:54.838 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:58:54.839 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:58:55.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:54 smithi154 conmon[34553]: cluster 2021-07-31T03:58:53.679079+0000 mgr.x (mgr.24542) 5644 : cluster [DBG] pgmap v5958: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:55.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:54 smithi154 conmon[34553]: audit 2021-07-31T03:58:54.211934+0000 mgr.x (mgr.24542) 5645 : audit [DBG] from='client.20640 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:58:55.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:54 smithi154 conmon[34553]: audit 2021-07-31T03:58:54.836454+0000 mon.c (mon.1) 239 : audit [DBG] from='client.? 172.21.15.45:0/3332552890' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:58:55.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:54 smithi045 conmon[33655]: cluster 2021-07-31T03: 2021-07-31T03:58:55.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:54 smithi045 conmon[33655]: 58:53.679079+0000 mgr.x (mgr.24542) 5644 : cluster [DBG] pgmap v5958: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:55.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:54 smithi045 conmon[33655]: audit 2021 2021-07-31T03:58:55.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:54 smithi045 conmon[33655]: -07-31T 2021-07-31T03:58:55.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:54 smithi045 conmon[33655]: 03:58:54.211934+0000 mgr.x 2021-07-31T03:58:55.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:54 smithi045 conmon[33655]: (mgr.24542) 5645 : audit [DBG] 2021-07-31T03:58:55.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:54 smithi045 conmon[33655]: 2021-07-31T03:58:55.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:54 smithi045 conmon[33655]: from='client.20640 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:58:55.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:54 smithi045 conmon[33655]: audit 2021-07-31T03:58:55.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:54 smithi045 conmon[33655]: 2021-07-31T03:58 2021-07-31T03:58:55.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:54 smithi045 conmon[33655]: :54.836454+0000 mon.c (mon.1) 2021-07-31T03:58:55.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:54 smithi045 conmon[33655]: 239 : audit [DBG] from='client.? 172.21.15.45:0/3332552890' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:58:55.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:54 smithi045 conmon[39718]: cluster 2021-07-31 2021-07-31T03:58:55.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:54 smithi045 conmon[39718]: T03:58:53.679079+0000 mgr.x (mgr.24542) 5644 : cluster 2021-07-31T03:58:55.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:54 smithi045 conmon[39718]: [DBG] pgmap v5958: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:55.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:54 smithi045 conmon[39718]: audit 2021 2021-07-31T03:58:55.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:54 smithi045 conmon[39718]: -07-31T03:58: 2021-07-31T03:58:55.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:54 smithi045 conmon[39718]: 54.211934+0000 mgr.x (mgr 2021-07-31T03:58:55.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:54 smithi045 conmon[39718]: .24542) 5645 : audit [DBG] 2021-07-31T03:58:55.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:54 smithi045 conmon[39718]: from='client.20640 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:58:55.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:54 smithi045 conmon[39718]: audit 2021 2021-07-31T03:58:55.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:54 smithi045 conmon[39718]: -07-31T03:58: 2021-07-31T03:58:55.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:54 smithi045 conmon[39718]: 54.836454+0000 mon.c ( 2021-07-31T03:58:55.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:54 smithi045 conmon[39718]: mon.1) 239 : audit 2021-07-31T03:58:55.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:54 smithi045 conmon[39718]: [DBG] from='client.? 172.21.15.45:0/3332552890' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:58:56.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:55 smithi154 conmon[34553]: audit 2021-07-31T03:58:54.492271+0000 mgr.x (mgr.24542) 5646 : audit 2021-07-31T03:58:56.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:55 smithi154 conmon[34553]: [DBG] from='client.20646 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:58:56.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:55 smithi045 conmon[33655]: audit 2021-07-31T03:58:54.492271+0000 mgr.x (mgr.24542) 5646 : audit 2021-07-31T03:58:56.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:55 smithi045 conmon[33655]: [DBG] from='client.20646 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:58:56.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:55 smithi045 conmon[39718]: audit 2021-07-31T03:58:54.492271+0000 mgr.x ( 2021-07-31T03:58:56.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:55 smithi045 conmon[39718]: mgr.24542) 5646 : audit [DBG] from='client.20646 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:58:57.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:56 smithi154 conmon[34553]: cluster 2021-07-31T03:58:55.679683+0000 mgr.x (mgr.24542) 5647 : cluster [DBG] pgmap v5959: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:57.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:56 smithi045 conmon[39718]: cluster 2021-07-31T03:58:55.679683+0000 mgr.x 2021-07-31T03:58:57.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:56 smithi045 conmon[39718]: (mgr.24542) 5647 : cluster [DBG] pgmap v5959: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:57.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:56 smithi045 conmon[33655]: cluster 2021-07-31T03:58:55.679683+0000 mgr.x (mgr.24542) 5647 2021-07-31T03:58:57.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:56 smithi045 conmon[33655]: : cluster [DBG] pgmap v5959: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:59.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:58:58 smithi154 conmon[34553]: cluster 2021-07-31T03:58:57.680304+0000 mgr.x (mgr.24542) 5648 : cluster [DBG] pgmap v5960: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:59.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:58 smithi045 conmon[39718]: cluster 2021-07-31T03:58:57.680304+0000 mgr.x (mgr.24542) 2021-07-31T03:58:59.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:58:58 smithi045 conmon[39718]: 5648 : cluster [DBG] pgmap v5960: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:58:59.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:58 smithi045 conmon[33655]: cluster 2021-07-31T03:58:57.680304+0000 mgr.x (mgr.24542) 5648 2021-07-31T03:58:59.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:58:58 smithi045 conmon[33655]: : cluster [DBG] pgmap v5960: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:01.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:00 smithi154 conmon[34553]: cluster 2021-07-31T03:58:59.680807+0000 mgr.x (mgr.24542) 5649 : cluster [DBG] pgmap v5961: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:01.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:00 smithi045 conmon[39718]: cluster 2021-07-31T03:58:59.680807+0000 mgr.x 2021-07-31T03:59:01.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:00 smithi045 conmon[39718]: (mgr.24542) 5649 : cluster [DBG] pgmap v5961: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:01.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:00 smithi045 conmon[33655]: cluster 2021-07-31T03:58:59.680807+0000 mgr.x (mgr.24542) 5649 : 2021-07-31T03:59:01.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:00 smithi045 conmon[33655]: cluster [DBG] pgmap v5961: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:03.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:02 smithi154 conmon[34553]: cluster 2021-07-31T03:59:01.681364+0000 mgr.x (mgr.24542) 5650 : cluster [DBG] 2021-07-31T03:59:03.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:02 smithi154 conmon[34553]: pgmap v5962: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:03.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:02 smithi045 conmon[39718]: cluster 2021-07-31T03:59:01.681364+0000 mgr.x (mgr 2021-07-31T03:59:03.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:02 smithi045 conmon[39718]: .24542) 5650 : cluster [DBG] pgmap v5962: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:03.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:02 smithi045 conmon[33655]: cluster 2021-07-31T03:59:01.681364+0000 mgr.x ( 2021-07-31T03:59:03.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:02 smithi045 conmon[33655]: mgr.24542) 5650 : cluster [DBG] pgmap v5962: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:05.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:04 smithi154 conmon[34553]: cluster 2021-07-31T03:59:03.681816+0000 mgr.x (mgr.24542) 5651 : cluster [DBG] pgmap v5963: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:05.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:04 smithi045 conmon[39718]: cluster 2021-07-31T03:59:03.681816+0000 mgr.x (mgr.24542 2021-07-31T03:59:05.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:04 smithi045 conmon[39718]: ) 5651 : cluster [DBG] pgmap v5963: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:05.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:04 smithi045 conmon[33655]: cluster 2021-07-31T03:59 2021-07-31T03:59:05.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:04 smithi045 conmon[33655]: :03.681816+0000 mgr.x (mgr.24542) 5651 : cluster [DBG] pgmap v5963: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:07.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:06 smithi154 conmon[34553]: cluster 2021-07-31T03:59:05.682272+0000 mgr.x (mgr.24542) 5652 : cluster [DBG] pgmap v5964: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:07.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:06 smithi045 conmon[39718]: cluster 2021-07-31T03:59:05.682272+0000 mgr.x (mgr. 2021-07-31T03:59:07.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:06 smithi045 conmon[39718]: 24542) 5652 : cluster [DBG] pgmap v5964: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:07.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:06 smithi045 conmon[33655]: cluster 2021-07-31T03:59:05.682272+0000 mgr.x (mgr.24542) 5652 : 2021-07-31T03:59:07.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:06 smithi045 conmon[33655]: cluster [DBG] pgmap v5964: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:09.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:09 smithi045 conmon[39718]: cluster 2021-07-31T03:59:07.682830+0000 mgr.x (mgr. 2021-07-31T03:59:09.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:09 smithi045 conmon[39718]: 24542) 5653 : cluster [DBG] pgmap v5965: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:09.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:09 smithi045 conmon[33655]: cluster 2021-07-31T03:59:07.682830+0000 mgr.x (mgr.24542) 5653 : cluster [DBG] 2021-07-31T03:59:09.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:09 smithi045 conmon[33655]: pgmap v5965: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:09.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:09 smithi154 conmon[34553]: cluster 2021-07-31T03:59:07.682830+0000 mgr.x (mgr.24542) 5653 : cluster [DBG] pgmap v5965: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:11.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:11 smithi045 conmon[33655]: cluster 2021-07-31T03:59:09.683307+0000 mgr.x (mgr.24542) 5654 : 2021-07-31T03:59:11.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:11 smithi045 conmon[33655]: cluster [DBG] pgmap v5966: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:11.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:11 smithi045 conmon[39718]: cluster 2021-07-31T03:59:09.683307+0000 mgr.x (mgr. 2021-07-31T03:59:11.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:11 smithi045 conmon[39718]: 24542) 5654 : cluster [DBG] pgmap v5966: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:11.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:11 smithi154 conmon[34553]: cluster 2021-07-31T03:59:09.683307+0000 mgr.x (mgr.24542) 5654 : cluster [DBG] pgmap v5966: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:11.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:11 smithi154 conmon[34553]: 2021-07-31T03:59:13.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:13 smithi045 conmon[33655]: cluster 2021-07-31T03:59:11.683755+0000 mgr.x (mgr.24542) 5655 : 2021-07-31T03:59:13.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:13 smithi045 conmon[33655]: cluster [DBG] pgmap v5967: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:13.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:13 smithi045 conmon[39718]: cluster 2021-07-31T03:59:11.683755+0000 mgr.x ( 2021-07-31T03:59:13.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:13 smithi045 conmon[39718]: mgr.24542) 5655 : cluster [DBG] pgmap v5967: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:13.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:13 smithi154 conmon[34553]: cluster 2021-07-31T03:59:11.683755+0000 mgr.x (mgr.24542) 5655 : cluster [DBG] pgmap v5967: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:15.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:15 smithi045 conmon[33655]: cluster 2021-07-31T03:59:13.684116+0000 mgr.x (mgr.24542) 5656 2021-07-31T03:59:15.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:15 smithi045 conmon[33655]: : cluster [DBG] pgmap v5968: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:15.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:15 smithi045 conmon[39718]: cluster 2021-07-31T03: 2021-07-31T03:59:15.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:15 smithi045 conmon[39718]: 59:13.684116+0000 mgr.x (mgr.24542) 5656 : cluster [DBG] pgmap v5968: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:15.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:15 smithi154 conmon[34553]: cluster 2021-07-31T03:59:13.684116+0000 mgr.x (mgr.24542) 5656 : cluster [DBG] pgmap v5968: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:17.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:17 smithi045 conmon[39718]: cluster 2021-07-31T03:59: 2021-07-31T03:59:17.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:17 smithi045 conmon[39718]: 15.684581+0000 mgr.x (mgr.24542) 5657 : cluster [DBG] pgmap v5969: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:17.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:17 smithi045 conmon[33655]: cluster 2021-07-31T03:59:15.684581+0000 mgr.x (mgr.24542) 5657 2021-07-31T03:59:17.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:17 smithi045 conmon[33655]: : cluster [DBG] pgmap v5969: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:17.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:17 smithi154 conmon[34553]: cluster 2021-07-31T03:59:15.684581+0000 mgr.x (mgr.24542) 5657 : cluster [DBG] pgmap v5969: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:17.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:17 smithi154 conmon[34553]: 2021-07-31T03:59:19.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:19 smithi045 conmon[39718]: cluster 2021-07-31T03:59:17.685030+0000 mgr.x (mgr.24542) 5658 : cluster [DBG] pgmap v5970: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:19.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:19 smithi045 conmon[33655]: cluster 2021-07-31T03:59:17.685030+0000 mgr.x (mgr.24542) 5658 : cluster [DBG] pgmap v5970: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:19.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:19 smithi154 conmon[34553]: cluster 2021-07-31T03:59:17.685030+0000 mgr.x (mgr.24542) 5658 : cluster [DBG] pgmap v5970: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:21.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:21 smithi045 conmon[39718]: cluster 2021-07-31T03:59:19.685628+0000 mgr.x (mgr.24542) 5659 : cluster [DBG] pgmap v5971: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:21.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:21 smithi045 conmon[33655]: cluster 2021-07-31T03:59:19.685628+0000 mgr.x (mgr.24542) 5659 : cluster [DBG] pgmap v5971: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:21.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:21 smithi154 conmon[34553]: cluster 2021-07-31T03:59:19.685628+0000 mgr.x (mgr.24542) 5659 : cluster [DBG] pgmap v5971: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:23.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:23 smithi154 conmon[34553]: cluster 2021-07-31T03:59:21.686211+0000 mgr.x (mgr.24542) 5660 : cluster [DBG] pgmap v5972: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:23.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:23 smithi045 conmon[33655]: cluster 2021-07-31T03:59:21.686211+0000 mgr.x (mgr.24542) 5660 : cluster 2021-07-31T03:59:23.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:23 smithi045 conmon[33655]: [DBG] pgmap v5972: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:23.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:23 smithi045 conmon[39718]: cluster 2021-07-31T03:59:21.686211 2021-07-31T03:59:23.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:23 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 5660 : cluster [DBG] pgmap v5972: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:25.149 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:59:25.427 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:59:25.428 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 2h ago 2h - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:59:25.428 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 2h ago 2h 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:59:25.428 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 2h ago 2h 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:59:25.429 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 2h ago 2h - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:59:25.429 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:59:25.429 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 2h ago 2h 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:59:25.430 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:59:25.430 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 2h ago 2h - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:59:25.430 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 2h ago 2h 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:59:25.431 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:59:25.431 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:59:25.431 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:59:25.432 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:59:25.432 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 2h ago 2h 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:59:25.432 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 2h ago 2h 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:59:25.433 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 2h ago 2h 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:59:25.433 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 2h ago 2h 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:59:25.433 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 2h ago 2h 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:59:25.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:25 smithi154 conmon[34553]: cluster 2021-07-31T03:59:23.686743+0000 mgr.x (mgr.24542) 5661 : cluster [DBG] pgmap v5973: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:25.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:25 smithi154 conmon[34553]: 2021-07-31T03:59:25.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:25 smithi045 conmon[33655]: cluster 2021-07-31T03:59:23.686743+0000 mgr.x (mgr. 2021-07-31T03:59:25.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:25 smithi045 conmon[33655]: 24542) 5661 : cluster [DBG] pgmap v5973: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:25.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:25 smithi045 conmon[39718]: cluster 2021-07-31T03: 2021-07-31T03:59:25.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:25 smithi045 conmon[39718]: 59:23.686743+0000 mgr.x (mgr.24542) 5661 : cluster [DBG] pgmap v5973: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:25.760 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:59:25.761 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:59:25.761 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:59:25.761 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:59:25.761 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:59:25.762 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:59:25.762 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:59:25.762 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:59:25.762 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:59:25.763 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:59:25.763 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:59:25.763 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:59:25.763 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:59:25.764 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:59:25.764 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:59:25.764 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:59:25.764 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:59:26.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:26 smithi154 conmon[34553]: audit 2021-07-31T03:59:25.137930+0000 mgr.x (mgr.24542) 5662 : audit [DBG] from='client.30595 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:59:26.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:26 smithi154 conmon[34553]: audit 2021-07-31T03:59:25.761645+0000 mon.a (mon.0) 3402 : audit [DBG] from='client.? 172.21.15.45:0/1896084416' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:59:26.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:26 smithi045 conmon[33655]: audit 2021-07-31T03:59:25.137930+0000 mgr.x (mgr.24542) 5662 : audit 2021-07-31T03:59:26.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:26 smithi045 conmon[33655]: [DBG] from='client.30595 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:59:26.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:26 smithi045 conmon[33655]: audit 2021-07-31T03:59:25.761645+0000 mon.a (mon.0) 3402 : audit [DBG] from='client.? 172.21.15.45:0/1896084416' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:59:26.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:26 smithi045 conmon[39718]: audit 2021-07-31T03:59:25.137930+0000 mgr.x (mgr.24542) 5662 : audit [DBG] from='client.30595 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:59:26.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:26 smithi045 conmon[39718]: audit 2021-07-31T03:59:25.761645+0000 mon.a (mon.0) 3402 : audit [DBG] from='client.? 172.21.15.45:0/1896084416' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:59:27.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:27 smithi154 conmon[34553]: audit 2021-07-31T03:59:25.420104+0000 mgr.x (mgr.24542) 5663 : audit [DBG] from='client.20664 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:59:27.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:27 smithi154 conmon[34553]: cluster 2021-07-31T03:59:25.687227+0000 mgr.x (mgr.24542) 5664 : cluster [DBG] pgmap v5974: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:27.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:27 smithi045 conmon[33655]: audit 2021-07-31T03:59:25.420104+0000 mgr.x (mgr.24542 2021-07-31T03:59:27.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:27 smithi045 conmon[33655]: ) 5663 : audit [DBG] from='client.20664 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:59:27.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:27 smithi045 conmon[33655]: cluster 2021-07-31T03:59:25.687227+0000 mgr.x (mgr.24542) 5664 : cluster [DBG] pgmap v5974: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:27.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:27 smithi045 conmon[39718]: audit 2021-07-31T03:59:25.420104+0000 mgr.x 2021-07-31T03:59:27.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:27 smithi045 conmon[39718]: (mgr.24542) 5663 : audit [DBG] from='client.20664 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:59:27.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:27 smithi045 conmon[39718]: cluster 2021-07-31T03:59:25.687227+0000 2021-07-31T03:59:27.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:27 smithi045 conmon[39718]: mgr.x (mgr.24542) 5664 : cluster [DBG] pgmap v5974: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:29.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:29 smithi154 conmon[34553]: cluster 2021-07-31T03:59:27.687682+0000 mgr.x (mgr.24542) 5665 : cluster [DBG] 2021-07-31T03:59:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:29 smithi154 conmon[34553]: pgmap v5975: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:29 smithi154 conmon[34553]: audit 2021-07-31T03:59:28.458749+0000 mon.b (mon.2) 735 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:59:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:29 smithi154 conmon[34553]: audit 2021- 2021-07-31T03:59:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:29 smithi154 conmon[34553]: 07-31T03:59:28.459023+0000 mon.b (mon.2) 736 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:59:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:29 smithi154 conmon[34553]: audit 2021-07-31T03:59: 2021-07-31T03:59:29.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:29 smithi154 conmon[34553]: 28.459670+0000 mon.a (mon.0) 3403 : audit 2021-07-31T03:59:29.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:29 smithi154 conmon[34553]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:59:29.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:29 smithi154 conmon[34553]: audit 2021-07-31 2021-07-31T03:59:29.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:29 smithi154 conmon[34553]: T03:59:28.459993+0000 mon.a (mon.0 2021-07-31T03:59:29.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:29 smithi154 conmon[34553]: ) 3404 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:59:29.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:29 smithi045 conmon[33655]: cluster 2021-07-31T03:59:27.687682+0000 mgr.x (mgr.24542) 2021-07-31T03:59:29.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:29 smithi045 conmon[33655]: 5665 : cluster [DBG] pgmap v5975: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:29.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:29 smithi045 conmon[33655]: audit 2021-07-31T03:59: 2021-07-31T03:59:29.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:29 smithi045 conmon[33655]: 28.458749+0000 mon.b (mon.2) 735 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:59:29.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:29 smithi045 conmon[33655]: 2021-07-31T03:59:29.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:29 smithi045 conmon[33655]: audit 2021- 2021-07-31T03:59:29.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:29 smithi045 conmon[33655]: 07-31T03:59:28.459023+0000 2021-07-31T03:59:29.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:29 smithi045 conmon[33655]: mon.b (mon.2) 736 : 2021-07-31T03:59:29.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:29 smithi045 conmon[33655]: audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:59:29.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:29 smithi045 conmon[33655]: audit 2021- 2021-07-31T03:59:29.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:29 smithi045 conmon[33655]: 07-31T03:59:28.459670 2021-07-31T03:59:29.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:29 smithi045 conmon[33655]: +0000 mon.a (mon.0 2021-07-31T03:59:29.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:29 smithi045 conmon[33655]: ) 3403 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:59:29.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:29 smithi045 conmon[33655]: 2021-07-31T03:59:29.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:29 smithi045 conmon[33655]: audit 2021-07-31T03:59:29.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:29 smithi045 conmon[33655]: 2021-07-31 2021-07-31T03:59:29.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:29 smithi045 conmon[33655]: T03:59:28.459993+0000 2021-07-31T03:59:29.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:29 smithi045 conmon[33655]: mon.a (mon.0) 3404 : 2021-07-31T03:59:29.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:29 smithi045 conmon[33655]: audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:59:29.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:29 smithi045 conmon[39718]: cluster 2021-07-31T03:59:27.687682 2021-07-31T03:59:29.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:29 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 5665 : cluster [DBG] pgmap v5975: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:29.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:29 smithi045 conmon[39718]: audit 2021-07-31T03:59 2021-07-31T03:59:29.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:29 smithi045 conmon[39718]: :28.458749+0000 mon.b (mon.2) 2021-07-31T03:59:29.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:29 smithi045 conmon[39718]: 735 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:59:29.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:29 smithi045 conmon[39718]: audit 2021 2021-07-31T03:59:29.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:29 smithi045 conmon[39718]: -07-31T03:59:28. 2021-07-31T03:59:29.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:29 smithi045 conmon[39718]: 459023+0000 mon.b (mon.2) 736 : 2021-07-31T03:59:29.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:29 smithi045 conmon[39718]: audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:59:29.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:29 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T03:59:29.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:29 smithi045 conmon[39718]: T03:59:28.459670+0000 mon.a 2021-07-31T03:59:29.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:29 smithi045 conmon[39718]: (mon.0) 3403 : audit [INF] 2021-07-31T03:59:29.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:29 smithi045 conmon[39718]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T03:59:29.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:29 smithi045 conmon[39718]: audit 2021-07-31T03:59 2021-07-31T03:59:29.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:29 smithi045 conmon[39718]: :28.459993+0000 mon.a (mon.0 2021-07-31T03:59:29.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:29 smithi045 conmon[39718]: ) 3404 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T03:59:31.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:31 smithi154 conmon[34553]: cluster 2021-07-31T03:59:29.688293+0000 mgr.x (mgr.24542) 5666 : cluster [DBG] 2021-07-31T03:59:31.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:31 smithi154 conmon[34553]: pgmap v5976: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:31.524 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:31 smithi045 conmon[33655]: cluster 2021-07-31T03:59:29.688293+0000 2021-07-31T03:59:31.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:31 smithi045 conmon[33655]: mgr.x (mgr.24542) 5666 : cluster [DBG] pgmap v5976: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:31.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:31 smithi045 conmon[39718]: cluster 2021-07-31T03:59:29.688293+0000 mgr.x (mgr. 2021-07-31T03:59:31.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:31 smithi045 conmon[39718]: 24542) 5666 : cluster [DBG] pgmap v5976: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:33.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:33 smithi154 conmon[34553]: cluster 2021-07-31T03:59:31.688671+0000 mgr.x (mgr.24542) 5667 : cluster [DBG] 2021-07-31T03:59:33.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:33 smithi154 conmon[34553]: pgmap v5977: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:33.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:33 smithi045 conmon[39718]: cluster 2021-07-31T03:59:31.688671+0000 mgr.x ( 2021-07-31T03:59:33.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:33 smithi045 conmon[39718]: mgr.24542) 5667 : cluster [DBG] pgmap v5977: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:33.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:33 smithi045 conmon[33655]: cluster 2021-07-31T03:59:31.688671+0000 2021-07-31T03:59:33.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:33 smithi045 conmon[33655]: mgr.x (mgr.24542) 5667 : cluster [DBG] pgmap v5977: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:35.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:35 smithi154 conmon[34553]: cluster 2021-07-31T03:59:33.689071+0000 mgr.x (mgr.24542) 5668 : cluster [DBG] 2021-07-31T03:59:35.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:35 smithi154 conmon[34553]: pgmap v5978: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:35.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:35 smithi045 conmon[33655]: cluster 2021-07-31T03:59:33.689071+0000 mgr.x (mgr.24542) 5668 : cluster 2021-07-31T03:59:35.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:35 smithi045 conmon[33655]: [DBG] pgmap v5978: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:35.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:35 smithi045 conmon[39718]: cluster 2021-07-31T03:59:33.689071+0000 mgr.x (mgr.24542) 5668 : cluster 2021-07-31T03:59:35.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:35 smithi045 conmon[39718]: [DBG] pgmap v5978: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:36.753 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: debug 2021-07-31T03:59:36.291+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:59:36.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:59:36.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:59:36.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: yield 2021-07-31T03:59:36.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:59:36.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:59:36.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:59:36.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:59:36.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:59:36.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:59:36.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:59:36.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:59:36.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:59:36.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: task.result() 2021-07-31T03:59:36.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:59:36.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: debug 2021-07-31T03:59:36.303+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:59:36.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:59:36.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:59:36.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: yield 2021-07-31T03:59:36.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:59:36.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:59:36.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:59:36.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:59:36.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:59:36.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:59:36.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:59:36.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:59:36.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:59:36.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: task.result() 2021-07-31T03:59:36.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:59:36.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: debug 2021-07-31T03:59:36.328+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:59:36.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:59:36.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:59:36.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: yield 2021-07-31T03:59:36.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:59:36.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:59:36.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:59:36.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:59:36.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:59:36.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:59:36.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:59:36.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:59:36.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:59:36.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: task.result() 2021-07-31T03:59:36.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:59:36.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: debug 2021-07-31T03:59:36.328+0000 7fcb9215e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:59:36.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:59:36.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:59:36.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: yield 2021-07-31T03:59:36.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:59:36.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:59:36.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:59:36.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:59:36.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:59:36.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:59:36.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:59:36.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:59:36.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:59:36.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: task.result() 2021-07-31T03:59:36.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:59:36.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: 2021-07-31T03:59:36.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:59:36.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: 2021-07-31T03:59:36.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:59:36.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:59:36.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:59:36.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:59:36.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:59:36.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:59:36.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:59:36.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:59:36.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:59:36.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:59:36.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:59:36.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:59:36.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:59:36.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:59:36.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:59:36.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:59:36.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:59:36.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:59:36.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: raise 2021-07-31T03:59:36.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:59:36.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:59:36.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:59:36.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:59:36.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:59:36.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: debug 2021-07-31T03:59:36.337+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T03:59:36.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:59:36.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:59:36.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: yield 2021-07-31T03:59:36.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:59:36.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:59:36.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:59:36.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:59:36.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:59:36.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:59:36.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:59:36.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:59:36.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:59:36.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: task.result() 2021-07-31T03:59:36.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:59:36.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: debug 2021-07-31T03:59:36.337+0000 7fcb9195d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:59:36.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:59:36.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:59:36.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: yield 2021-07-31T03:59:36.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:59:36.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:59:36.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:59:36.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T03:59:36.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:59:36.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T03:59:36.786 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:59:36.786 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: await self._waiter 2021-07-31T03:59:36.786 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:59:36.786 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: task.result() 2021-07-31T03:59:36.787 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T03:59:36.787 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: 2021-07-31T03:59:36.787 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T03:59:36.787 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: 2021-07-31T03:59:36.788 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T03:59:36.788 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:59:36.788 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: return f(*arg) 2021-07-31T03:59:36.788 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:59:36.789 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:59:36.789 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:59:36.789 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:59:36.790 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:59:36.790 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:59:36.790 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:59:36.790 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T03:59:36.791 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:59:36.791 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: raise self._exception 2021-07-31T03:59:36.791 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:59:36.791 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:59:36.792 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:59:36.792 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T03:59:36.792 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:59:36.792 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: raise 2021-07-31T03:59:36.793 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:59:36.793 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T03:59:36.793 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:59:36.793 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:59:36.794 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 03:59:36 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:59:37.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:37 smithi154 conmon[34553]: cluster 2021-07-31T03:59:35.689501+0000 mgr.x (mgr.24542) 5669 : cluster [DBG] pgmap v5979: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:37.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:37 smithi154 conmon[34553]: audit 2021-07-31T03:59:36.246883+0000 mon.b (mon.2) 737 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:59:37.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:37 smithi154 conmon[34553]: audit 2021-07-31T03:59:36.248274+0000 mon.b (mon.2) 738 : audit [DBG] 2021-07-31T03:59:37.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:37 smithi154 conmon[34553]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:59:37.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:37 smithi154 conmon[34553]: audit 2021-07-31T03 2021-07-31T03:59:37.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:37 smithi154 conmon[34553]: :59:36.249159+0000 mon.b ( 2021-07-31T03:59:37.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:37 smithi154 conmon[34553]: mon.2) 739 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:59:37.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:37 smithi154 conmon[34553]: 2021-07-31T03:59:37.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:37 smithi154 conmon[34553]: cephadm 2021-07- 2021-07-31T03:59:37.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:37 smithi154 conmon[34553]: 31T03:59:36.249912+0000 mgr.x ( 2021-07-31T03:59:37.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:37 smithi154 conmon[34553]: mgr.24542) 5670 : cephadm [WRN] 2021-07-31T03:59:37.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:37 smithi154 conmon[34553]: unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:59:37.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:37 smithi154 conmon[34553]: cephadm 2021- 2021-07-31T03:59:37.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:37 smithi154 conmon[34553]: 07-31T03:59:36.292454 2021-07-31T03:59:37.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:37 smithi154 conmon[34553]: +0000 mgr.x (mgr.24542) 5671 2021-07-31T03:59:37.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:37 smithi154 conmon[34553]: : cephadm [ERR] 2021-07-31T03:59:37.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:37 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:59:37.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:59:37.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:37 smithi154 conmon[34553]: yield 2021-07-31T03:59:37.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:59:37.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:37 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:59:37.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:59:37.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:37 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:59:37.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:59:37.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:37 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:59:37.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:59:37.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:37 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:59:37.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:59:37.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:37 smithi154 conmon[34553]: task.result() 2021-07-31T03:59:37.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:37 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:59:37.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:37 smithi154 conmon[34553]: cephadm 2021-07-31T03:59:36.292890+0000 mgr.x (mgr.24542) 5672 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:59:37.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:37 smithi045 conmon[39718]: cluster 2021-07-31T03:59:35.689501+0000 mgr.x (mgr.24542) 5669 : 2021-07-31T03:59:37.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:37 smithi045 conmon[39718]: cluster [DBG] pgmap v5979: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:37.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:37 smithi045 conmon[39718]: audit 2021-07-31T03:59:36.246883+0000 mon.b (mon.2) 737 : 2021-07-31T03:59:37.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:37 smithi045 conmon[39718]: audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:59:37.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:37 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T03:59:37.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:37 smithi045 conmon[39718]: 03:59:36.248274+0000 mon.b (mon.2) 738 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:59:37.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:37 smithi045 conmon[39718]: audit 2021-07-31T03:59 2021-07-31T03:59:37.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:37 smithi045 conmon[39718]: :36.249159+0000 mon.b (mon.2) 739 : audit [INF] 2021-07-31T03:59:37.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:37 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:59:37.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:37 smithi045 conmon[39718]: cephadm 2021- 2021-07-31T03:59:37.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:37 smithi045 conmon[39718]: 07-31T03:59:36.249912+0000 mgr.x (mgr.24542) 5670 : cephadm [WRN] 2021-07-31T03:59:37.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:37 smithi045 conmon[39718]: unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:59:37.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:37 smithi045 conmon[39718]: cephadm 2021 2021-07-31T03:59:37.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:37 smithi045 conmon[39718]: -07-31T03:59:36.292454+0000 mgr.x (mgr.24542) 5671 : cephadm 2021-07-31T03:59:37.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:37 smithi045 conmon[39718]: [ERR] 2021-07-31T03:59:37.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:37 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:59:37.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:59:37.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:37 smithi045 conmon[39718]: yield 2021-07-31T03:59:37.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:59:37.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:37 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:59:37.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:59:37.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:37 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:59:37.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:59:37.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:37 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:59:37.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:59:37.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:37 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:59:37.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:59:37.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:37 smithi045 conmon[39718]: task.result() 2021-07-31T03:59:37.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:37 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:59:37.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:37 smithi045 conmon[39718]: cephadm 2021-07-31T03:59:37.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:37 smithi045 conmon[39718]: 2021-07-31T03:59:36.292890+0000 mgr.x (mgr.24542) 5672 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:59:37.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:37 smithi045 conmon[33655]: cluster 2021-07-31T03:59:35.689501+0000 mgr.x (mgr.24542) 5669 : cluster 2021-07-31T03:59:37.535 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:37 smithi045 conmon[33655]: [DBG] pgmap v5979: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:37.535 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:37 smithi045 conmon[33655]: audit 2021-07-31T03:59:36.246883+0000 mon.b (mon.2) 737 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T03:59:37.535 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:37 smithi045 conmon[33655]: audit 2021-07-31T03:59:36 2021-07-31T03:59:37.535 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:37 smithi045 conmon[33655]: .248274+0000 mon.b (mon.2) 738 : audit [DBG] 2021-07-31T03:59:37.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:37 smithi045 conmon[33655]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T03:59:37.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:37 smithi045 conmon[33655]: audit 2021-07-31 2021-07-31T03:59:37.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:37 smithi045 conmon[33655]: T03:59:36.249159+0000 mon.b (mon.2) 2021-07-31T03:59:37.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:37 smithi045 conmon[33655]: 739 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T03:59:37.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:37 smithi045 conmon[33655]: cephadm 2021-07-31T03:59:37.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:37 smithi045 conmon[33655]: 2021-07-31T03:59:36.249912+0000 mgr.x ( 2021-07-31T03:59:37.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:37 smithi045 conmon[33655]: mgr.24542) 5670 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T03:59:37.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:37 smithi045 conmon[33655]: cephadm 2021-07-31T03:59:36.292454+0000 mgr.x (mgr. 2021-07-31T03:59:37.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:37 smithi045 conmon[33655]: 24542) 5671 : cephadm [ERR] 2021-07-31T03:59:37.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:37 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:59:37.539 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:59:37.539 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:37 smithi045 conmon[33655]: yield 2021-07-31T03:59:37.539 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:59:37.539 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:37 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:59:37.540 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:59:37.540 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:37 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:59:37.540 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:59:37.540 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:37 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:59:37.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:59:37.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:37 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:59:37.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:59:37.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:37 smithi045 conmon[33655]: task.result() 2021-07-31T03:59:37.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:37 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:59:37.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:37 smithi045 conmon[33655]: cephadm 2021-07-31T03:59:36.292890+0000 mgr.x (mgr.24542) 5672 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T03:59:38.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: cephadm 2021-07-31T03:59:36.304132+0000 mgr.x (mgr.24542) 5673 : cephadm [ERR] 2021-07-31T03:59:38.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:59:38.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:59:38.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: yield 2021-07-31T03:59:38.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:59:38.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:59:38.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:59:38.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:59:38.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:59:38.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:59:38.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:59:38.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:59:38.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:59:38.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: task.result() 2021-07-31T03:59:38.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:59:38.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: cephadm 2021-07-31T03:59:36.305019+0000 mgr.x (mgr.24542) 5674 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:59:38.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: cephadm 2021-07-31T03:59:36.329333+0000 mgr.x (mgr.24542) 5675 : cephadm [ERR] 2021-07-31T03:59:38.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:59:38.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:59:38.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: yield 2021-07-31T03:59:38.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:59:38.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:59:38.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:59:38.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:59:38.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:59:38.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:59:38.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:59:38.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:59:38.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:59:38.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: task.result() 2021-07-31T03:59:38.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:59:38.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: cephadm 2021-07-31T03:59:36.329807+0000 2021-07-31T03:59:38.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: mgr.x (mgr.24542) 5676 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:59:38.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:59:38.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:59:38.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: yield 2021-07-31T03:59:38.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:59:38.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:59:38.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:59:38.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:59:38.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:59:38.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:59:38.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:59:38.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:59:38.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:59:38.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: task.result() 2021-07-31T03:59:38.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:59:38.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: 2021-07-31T03:59:38.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:59:38.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: 2021-07-31T03:59:38.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:59:38.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:59:38.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:59:38.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:59:38.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:59:38.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:59:38.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:59:38.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:59:38.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:59:38.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:59:38.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:59:38.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:59:38.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:59:38.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:59:38.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:59:38.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:59:38.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:59:38.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:59:38.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: raise 2021-07-31T03:59:38.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:59:38.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:59:38.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:59:38.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:59:38.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:59:38.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: cephadm 2021-07-31T03:59:36.338333+0000 mgr.x (mgr.24542) 5677 : cephadm [ERR] 2021-07-31T03:59:38.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:59:38.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:59:38.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: yield 2021-07-31T03:59:38.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:59:38.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:59:38.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:59:38.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:59:38.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:59:38.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:59:38.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:59:38.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:59:38.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:59:38.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: task.result() 2021-07-31T03:59:38.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:59:38.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: cephadm 2021- 2021-07-31T03:59:38.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: 07-31T03:59:36.338725+0000 mgr.x (mgr.24542) 5678 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:59:38.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:59:38.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:59:38.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: yield 2021-07-31T03:59:38.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:59:38.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:59:38.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:59:38.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T03:59:38.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:59:38.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T03:59:38.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:59:38.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: await self._waiter 2021-07-31T03:59:38.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:59:38.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: task.result() 2021-07-31T03:59:38.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T03:59:38.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: 2021-07-31T03:59:38.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T03:59:38.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: 2021-07-31T03:59:38.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T03:59:38.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:59:38.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: return f(*arg) 2021-07-31T03:59:38.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:59:38.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:59:38.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:59:38.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:59:38.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:59:38.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:59:38.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:59:38.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T03:59:38.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:59:38.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: raise self._exception 2021-07-31T03:59:38.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:59:38.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:59:38.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:59:38.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T03:59:38.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:59:38.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: raise 2021-07-31T03:59:38.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:59:38.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T03:59:38.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:59:38.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:59:38.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:38 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:59:38.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: cephadm 2021-07-31T03:59:36.304132+0000 mgr.x ( 2021-07-31T03:59:38.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: mgr.24542) 5673 : cephadm [ERR] 2021-07-31T03:59:38.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:59:38.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:59:38.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: yield 2021-07-31T03:59:38.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:59:38.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:59:38.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:59:38.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:59:38.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:59:38.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:59:38.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:59:38.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:59:38.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:59:38.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: task.result() 2021-07-31T03:59:38.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:59:38.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: cephadm 2021-07-31T03:59:36.305019+0000 mgr.x (mgr.24542 2021-07-31T03:59:38.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: ) 5674 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:59:38.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: cephadm 2021-07-31T03:59:36.329333+0000 mgr.x (mgr.24542) 5675 : cephadm [ERR] 2021-07-31T03:59:38.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:59:38.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:59:38.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: yield 2021-07-31T03:59:38.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:59:38.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:59:38.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:59:38.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:59:38.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:59:38.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:59:38.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:59:38.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:59:38.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:59:38.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: task.result() 2021-07-31T03:59:38.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:59:38.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: cephadm 2021-07-31T03:59:36.329807+0000 mgr.x (mgr.24542) 5676 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:59:38.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:59:38.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:59:38.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: yield 2021-07-31T03:59:38.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:59:38.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:59:38.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:59:38.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:59:38.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:59:38.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:59:38.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:59:38.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:59:38.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:59:38.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: task.result() 2021-07-31T03:59:38.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:59:38.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: 2021-07-31T03:59:38.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:59:38.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: 2021-07-31T03:59:38.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:59:38.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:59:38.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:59:38.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:59:38.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:59:38.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:59:38.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:59:38.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:59:38.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:59:38.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:59:38.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:59:38.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:59:38.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:59:38.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:59:38.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:59:38.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:59:38.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:59:38.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:59:38.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: raise 2021-07-31T03:59:38.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:59:38.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:59:38.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:59:38.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:59:38.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:59:38.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: cephadm 2021-07-31T03:59:36.338333+0000 mgr.x (mgr.24542) 5677 : cephadm [ERR] 2021-07-31T03:59:38.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:59:38.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:59:38.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: yield 2021-07-31T03:59:38.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:59:38.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: cephadm 2021-07-31T03:59:36.304132+0000 mgr.x (mgr.24542 2021-07-31T03:59:38.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: ) 5673 : cephadm [ERR] 2021-07-31T03:59:38.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:59:38.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:59:38.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: yield 2021-07-31T03:59:38.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:59:38.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:59:38.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:59:38.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:59:38.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:59:38.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:59:38.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:59:38.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:59:38.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:59:38.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: task.result() 2021-07-31T03:59:38.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:59:38.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: cephadm 2021-07-31T03:59:36.305019+0000 mgr.x (mgr.24542) 5674 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T03:59:38.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: cephadm 2021-07-31T03:59:36.329333+0000 mgr.x (mgr.24542) 5675 : cephadm [ERR] 2021-07-31T03:59:38.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:59:38.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:59:38.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: yield 2021-07-31T03:59:38.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:59:38.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:59:38.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:59:38.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:59:38.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:59:38.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:59:38.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:59:38.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:59:38.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:59:38.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: task.result() 2021-07-31T03:59:38.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:59:38.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: cephadm 2021-07-31T03:59:36.329807+0000 mgr.x (mgr.24542) 5676 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:59:38.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:59:38.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:59:38.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: yield 2021-07-31T03:59:38.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:59:38.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:59:38.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:59:38.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:59:38.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:59:38.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:59:38.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:59:38.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:59:38.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:59:38.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:59:38.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: task.result() 2021-07-31T03:59:38.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:59:38.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: cephadm 2021-07-31T03:59:36.338725+0000 mgr.x (mgr.24542) 5678 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:59:38.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:59:38.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:59:38.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: yield 2021-07-31T03:59:38.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:59:38.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:59:38.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:59:38.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T03:59:38.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:59:38.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T03:59:38.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:59:38.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: await self._waiter 2021-07-31T03:59:38.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:59:38.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: task.result() 2021-07-31T03:59:38.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T03:59:38.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: 2021-07-31T03:59:38.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T03:59:38.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: 2021-07-31T03:59:38.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T03:59:38.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:59:38.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: return f(*arg) 2021-07-31T03:59:38.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:59:38.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:59:38.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:59:38.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:59:38.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:59:38.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:59:38.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:59:38.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T03:59:38.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:59:38.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: raise self._exception 2021-07-31T03:59:38.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:59:38.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:59:38.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:59:38.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T03:59:38.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:59:38.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: raise 2021-07-31T03:59:38.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:59:38.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T03:59:38.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:59:38.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:59:38.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:59:38.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:59:38.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:59:38.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:59:38.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:59:38.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:59:38.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:59:38.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:59:38.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: task.result() 2021-07-31T03:59:38.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:59:38.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: 2021-07-31T03:59:38.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:59:38.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: 2021-07-31T03:59:38.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:59:38.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:59:38.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:59:38.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:59:38.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:59:38.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:59:38.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:59:38.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:59:38.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:59:38.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:59:38.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:59:38.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:59:38.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:59:38.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:59:38.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:59:38.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:59:38.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:59:38.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:59:38.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: raise 2021-07-31T03:59:38.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:59:38.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:59:38.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:59:38.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:59:38.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T03:59:38.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: cephadm 2021-07-31T03:59:36.338333+0000 mgr.x (mgr.24542) 5677 : cephadm [ERR] 2021-07-31T03:59:38.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:59:38.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:59:38.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: yield 2021-07-31T03:59:38.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:59:38.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:59:38.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:59:38.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:59:38.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:59:38.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:59:38.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:59:38.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:59:38.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:59:38.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: task.result() 2021-07-31T03:59:38.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:59:38.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: cephadm 2021-07-31T03:59:36.338725+0000 mgr.x (mgr.24542) 5678 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T03:59:38.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:59:38.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T03:59:38.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: yield 2021-07-31T03:59:38.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T03:59:38.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T03:59:38.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T03:59:38.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T03:59:38.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T03:59:38.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T03:59:38.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T03:59:38.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: await self._waiter 2021-07-31T03:59:38.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T03:59:38.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: task.result() 2021-07-31T03:59:38.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T03:59:38.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: 2021-07-31T03:59:38.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T03:59:38.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: 2021-07-31T03:59:38.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T03:59:38.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T03:59:38.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: return f(*arg) 2021-07-31T03:59:38.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T03:59:38.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T03:59:38.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T03:59:38.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T03:59:38.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T03:59:38.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T03:59:38.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T03:59:38.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T03:59:38.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T03:59:38.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: raise self._exception 2021-07-31T03:59:38.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T03:59:38.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T03:59:38.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T03:59:38.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T03:59:38.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T03:59:38.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: raise 2021-07-31T03:59:38.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T03:59:38.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T03:59:38.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T03:59:38.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T03:59:38.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:38 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T03:59:39.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:39 smithi154 conmon[34553]: cluster 2021-07-31T03:59:37.689961+0000 mgr.x (mgr.24542) 5679 : cluster [DBG] pgmap v5980: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:39.524 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:39 smithi045 conmon[39718]: cluster 2021-07-31T03:59:37.689961+0000 mgr.x (mgr 2021-07-31T03:59:39.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:39 smithi045 conmon[39718]: .24542) 5679 : cluster [DBG] pgmap v5980: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:39.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:39 smithi045 conmon[33655]: cluster 2021-07-31T03:59:37.689961+0000 mgr.x (mgr.24542) 5679 : cluster 2021-07-31T03:59:39.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:39 smithi045 conmon[33655]: [DBG] pgmap v5980: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:41.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:41 smithi154 conmon[34553]: cluster 2021-07-31T03:59:39.690396+0000 mgr.x (mgr.24542) 5680 : cluster [DBG] pgmap v5981: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:41.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:41 smithi045 conmon[33655]: cluster 2021-07-31T03:59:39.690396+0000 mgr.x (mgr.24542 2021-07-31T03:59:41.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:41 smithi045 conmon[33655]: ) 5680 : cluster [DBG] pgmap v5981: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:41.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:41 smithi045 conmon[39718]: cluster 2021-07-31T03:59:39.690396+0000 mgr.x (mgr 2021-07-31T03:59:41.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:41 smithi045 conmon[39718]: .24542) 5680 : cluster [DBG] pgmap v5981: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:43.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:43 smithi154 conmon[34553]: cluster 2021-07-31T03:59:41.690970+0000 mgr.x (mgr.24542) 5681 : cluster [DBG] pgmap v5982: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:43.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:43 smithi045 conmon[39718]: cluster 2021-07-31T03:59:41.690970 2021-07-31T03:59:43.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:43 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 5681 : cluster [DBG] pgmap v5982: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:43.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:43 smithi045 conmon[33655]: cluster 2021-07-31T03:59:41.690970+0000 mgr.x (mgr.24542) 5681 : cluster [DBG] 2021-07-31T03:59:43.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:43 smithi045 conmon[33655]: pgmap v5982: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:45.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:45 smithi154 conmon[34553]: cluster 2021-07-31T03:59:43.691409+0000 mgr.x (mgr.24542) 5682 : cluster [DBG] pgmap v5983: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:45.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:45 smithi045 conmon[33655]: cluster 2021-07-31T03:59:43.691409+0000 mgr.x (mgr.24542) 5682 : cluster 2021-07-31T03:59:45.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:45 smithi045 conmon[33655]: [DBG] pgmap v5983: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:45.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:45 smithi045 conmon[39718]: cluster 2021-07-31T03:59:43.691409+0000 mgr.x (mgr 2021-07-31T03:59:45.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:45 smithi045 conmon[39718]: .24542) 5682 : cluster [DBG] pgmap v5983: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:47.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:47 smithi154 conmon[34553]: cluster 2021-07-31T03:59:45.691967+0000 mgr.x (mgr.24542) 5683 : cluster [DBG] pgmap v5984: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:47.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:47 smithi045 conmon[39718]: cluster 2021-07-31T03:59:45.691967+0000 mgr.x (mgr.24542 2021-07-31T03:59:47.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:47 smithi045 conmon[39718]: ) 5683 : cluster [DBG] pgmap v5984: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:47.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:47 smithi045 conmon[33655]: cluster 2021-07-31T03:59:45.691967+0000 mgr.x (mgr.24542) 5683 : 2021-07-31T03:59:47.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:47 smithi045 conmon[33655]: cluster [DBG] pgmap v5984: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:49.497 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:49 smithi154 conmon[34553]: cluster 2021-07-31T03:59:47.692401+0000 mgr.x (mgr.24542) 5684 : cluster [DBG] pgmap v5985: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:49.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:49 smithi045 conmon[33655]: cluster 2021-07-31T03:59:47. 2021-07-31T03:59:49.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:49 smithi045 conmon[33655]: 692401+0000 mgr.x (mgr.24542) 5684 : cluster [DBG] pgmap v5985: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:49.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:49 smithi045 conmon[39718]: cluster 2021-07-31T03:59:47.692401+0000 2021-07-31T03:59:49.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:49 smithi045 conmon[39718]: mgr.x (mgr.24542) 5684 : cluster [DBG] pgmap v5985: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:51.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:51 smithi154 conmon[34553]: cluster 2021-07-31T03:59:49.692800+0000 mgr.x (mgr.24542) 5685 : cluster 2021-07-31T03:59:51.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:51 smithi154 conmon[34553]: [DBG] pgmap v5986: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:51.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:51 smithi045 conmon[33655]: cluster 2021-07-31T03:59:49.692800+0000 mgr.x (mgr.24542) 5685 : cluster [DBG] pgmap v5986: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:51.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:51 smithi045 conmon[39718]: cluster 2021-07-31T03:59:49.692800+0000 2021-07-31T03:59:51.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:51 smithi045 conmon[39718]: mgr.x (mgr.24542) 5685 : cluster [DBG] pgmap v5986: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:53.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:53 smithi154 conmon[34553]: cluster 2021-07-31T03:59:51.693215+0000 mgr.x (mgr.24542) 5686 : 2021-07-31T03:59:53.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:53 smithi154 conmon[34553]: cluster [DBG] pgmap v5987: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:53.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:53 smithi045 conmon[39718]: cluster 2021-07-31T03:59:51.693215+0000 mgr.x (mgr.24542) 5686 : cluster [DBG] pgmap v5987: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:53.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:53 smithi045 conmon[33655]: cluster 2021-07-31T03:59:51.693215+0000 mgr.x (mgr.24542) 5686 : cluster 2021-07-31T03:59:53.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:53 smithi045 conmon[33655]: [DBG] pgmap v5987: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:55.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:55 smithi154 conmon[34553]: cluster 2021-07-31T03:59:53.693550+0000 mgr.x (mgr.24542) 5687 : cluster 2021-07-31T03:59:55.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:55 smithi154 conmon[34553]: [DBG] pgmap v5988: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:55.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:55 smithi045 conmon[39718]: cluster 2021-07-31T03:59:53.693550+0000 mgr.x (mgr. 2021-07-31T03:59:55.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:55 smithi045 conmon[39718]: 24542) 5687 : cluster [DBG] pgmap v5988: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:55.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:55 smithi045 conmon[33655]: cluster 2021-07-31T03:59:53.693550+0000 mgr.x (mgr.24542) 5687 : 2021-07-31T03:59:55.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:55 smithi045 conmon[33655]: cluster [DBG] pgmap v5988: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:56.073 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T03:59:56.351 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T03:59:56.351 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 2h ago 2h - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T03:59:56.352 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 2h ago 2h 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T03:59:56.352 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 2h ago 2h 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T03:59:56.352 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 2h ago 2h - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T03:59:56.353 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T03:59:56.353 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 2h ago 2h 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T03:59:56.353 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T03:59:56.353 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 2h ago 2h - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T03:59:56.354 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 2h ago 2h 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T03:59:56.354 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T03:59:56.354 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T03:59:56.355 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T03:59:56.355 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T03:59:56.355 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 2h ago 2h 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T03:59:56.356 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 2h ago 2h 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T03:59:56.356 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 2h ago 2h 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T03:59:56.356 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 2h ago 2h 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T03:59:56.357 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 2h ago 2h 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T03:59:56.687 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T03:59:56.687 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T03:59:56.688 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T03:59:56.688 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:59:56.688 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T03:59:56.689 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T03:59:56.689 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:59:56.689 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:59:56.690 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T03:59:56.690 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T03:59:56.690 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T03:59:56.691 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T03:59:56.691 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T03:59:56.691 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T03:59:56.691 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T03:59:56.692 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T03:59:56.692 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T03:59:57.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:57 smithi154 conmon[34553]: cluster 2021-07-31T03:59:55.693917+0000 mgr.x (mgr.24542) 2021-07-31T03:59:57.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:57 smithi154 conmon[34553]: 5688 : cluster [DBG] pgmap v5989: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:57.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:57 smithi154 conmon[34553]: audit 2021-07-31T03:59:56.061957+0000 mgr.x (mgr.24542) 5689 : audit [DBG] from='client.20676 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:59:57.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:57 smithi154 conmon[34553]: audit 2021-07-31T03:59:56.688290+0000 2021-07-31T03:59:57.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:57 smithi154 conmon[34553]: mon.c (mon.1) 240 : audit [DBG] from='client.? 172.21.15.45:0/1509791223' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:59:57.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:57 smithi154 conmon[34553]: 2021-07-31T03:59:57.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:57 smithi045 conmon[33655]: cluster 2021-07-31T03:59:55.693917+0000 mgr.x (mgr.24542) 2021-07-31T03:59:57.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:57 smithi045 conmon[33655]: 5688 : cluster [DBG] pgmap v5989: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:57.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:57 smithi045 conmon[33655]: audit 2021-07-31T03:59:56.061957+0000 mgr.x (mgr.24542) 5689 : audit [DBG] from='client.20676 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:59:57.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:57 smithi045 conmon[33655]: audit 2021-07-31T03:59:56.688290+0000 mon.c ( 2021-07-31T03:59:57.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:57 smithi045 conmon[33655]: mon.1) 240 : audit [DBG] from='client.? 172.21.15.45:0/1509791223' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:59:57.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:57 smithi045 conmon[39718]: cluster 2021-07-31T03:59:55.693917 2021-07-31T03:59:57.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:57 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 5688 : cluster [DBG] pgmap v5989: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:57.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:57 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T03:59:57.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:57 smithi045 conmon[39718]: T03:59:56.061957+0000 mgr.x (mgr.24542) 5689 : audit [DBG] from='client.20676 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:59:57.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:57 smithi045 conmon[39718]: audit 2021-07-31T03:59: 2021-07-31T03:59:57.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:57 smithi045 conmon[39718]: 56.688290+0000 mon.c (mon.1) 240 : audit [DBG] 2021-07-31T03:59:57.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:57 smithi045 conmon[39718]: from='client.? 172.21.15.45:0/1509791223' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T03:59:58.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:58 smithi154 conmon[34553]: audit 2021-07-31T03:59:56.344579+0000 mgr.x (mgr.24542) 5690 : audit [DBG] 2021-07-31T03:59:58.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:58 smithi154 conmon[34553]: from='client.20682 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:59:58.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:58 smithi045 conmon[33655]: audit 2021-07-31T03:59:56.344579+0000 mgr.x (mgr.24542) 2021-07-31T03:59:58.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:58 smithi045 conmon[33655]: 5690 : audit [DBG] from='client.20682 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:59:58.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:58 smithi045 conmon[39718]: audit 2021-07-31T03:59:56.344579+0000 mgr.x (mgr.24542) 2021-07-31T03:59:58.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:58 smithi045 conmon[39718]: 5690 : audit [DBG] from='client.20682 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T03:59:59.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:59 smithi154 conmon[34553]: cluster 2021-07-31T03:59:57.694271+0000 mgr.x (mgr.24542) 5691 2021-07-31T03:59:59.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 03:59:59 smithi154 conmon[34553]: : cluster [DBG] pgmap v5990: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:59.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:59 smithi045 conmon[39718]: cluster 2021-07-31T03:59:57.694271+0000 2021-07-31T03:59:59.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 03:59:59 smithi045 conmon[39718]: mgr.x (mgr.24542) 5691 : cluster [DBG] pgmap v5990: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T03:59:59.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:59 smithi045 conmon[33655]: cluster 2021-07-31T03:59 2021-07-31T03:59:59.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 03:59:59 smithi045 conmon[33655]: :57.694271+0000 mgr.x (mgr.24542) 5691 : cluster [DBG] pgmap v5990: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:00.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:00 smithi154 conmon[34553]: cluster 2021-07-31T04:00:00.000145+0000 mon.a (mon. 2021-07-31T04:00:00.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:00 smithi154 conmon[34553]: 0) 3405 : cluster [INF] overall HEALTH_OK 2021-07-31T04:00:00.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:00 smithi045 conmon[39718]: cluster 2021-07-31T04:00:00.000145+0000 mon.a ( 2021-07-31T04:00:00.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:00 smithi045 conmon[39718]: mon.0) 3405 : cluster [INF] overall HEALTH_OK 2021-07-31T04:00:00.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:00 smithi045 conmon[33655]: cluster 2021-07-31T04:00:00.000145+0000 mon.a (mon.0 2021-07-31T04:00:00.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:00 smithi045 conmon[33655]: ) 3405 : cluster [INF] overall HEALTH_OK 2021-07-31T04:00:01.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:01 smithi154 conmon[34553]: cluster 2021-07-31T03:59:59.694651+0000 mgr.x (mgr.24542) 5692 2021-07-31T04:00:01.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:01 smithi154 conmon[34553]: : cluster [DBG] pgmap v5991: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:01.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:01 smithi045 conmon[39718]: cluster 2021-07-31T03:59:59.694651+0000 mgr.x ( 2021-07-31T04:00:01.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:01 smithi045 conmon[39718]: mgr.24542) 5692 : cluster [DBG] pgmap v5991: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:01.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:01 smithi045 conmon[33655]: cluster 2021-07-31T03:59:59.694651+0000 mgr.x 2021-07-31T04:00:01.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:01 smithi045 conmon[33655]: (mgr.24542) 5692 : cluster [DBG] pgmap v5991: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:03.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:03 smithi154 conmon[34553]: cluster 2021-07-31T04:00:01.695025+0000 mgr.x (mgr. 2021-07-31T04:00:03.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:03 smithi154 conmon[34553]: 24542) 5693 : cluster [DBG] pgmap v5992: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:03.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:03 smithi045 conmon[39718]: cluster 2021-07-31T04:00:01.695025+0000 2021-07-31T04:00:03.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:03 smithi045 conmon[39718]: mgr.x (mgr.24542) 5693 : cluster [DBG] pgmap v5992: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:03.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:03 smithi045 conmon[33655]: cluster 2021-07-31T04:00:01.695025+0000 mgr.x (mgr. 2021-07-31T04:00:03.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:03 smithi045 conmon[33655]: 24542) 5693 : cluster [DBG] pgmap v5992: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:05.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:05 smithi154 conmon[34553]: cluster 2021-07-31T04:00:03.695361+0000 mgr.x (mgr. 2021-07-31T04:00:05.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:05 smithi154 conmon[34553]: 24542) 5694 : cluster [DBG] pgmap v5993: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:05.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:05 smithi045 conmon[33655]: cluster 2021-07-31T04:00:03.695361+0000 mgr.x (mgr.24542) 5694 : cluster [DBG] pgmap v5993: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:05.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:05 smithi045 conmon[39718]: cluster 2021-07-31T04:00:03.695361+0000 mgr.x (mgr.24542 2021-07-31T04:00:05.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:05 smithi045 conmon[39718]: ) 5694 : cluster [DBG] pgmap v5993: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:07.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:07 smithi154 conmon[34553]: cluster 2021-07-31T04:00:05.695712+0000 mgr.x (mgr.24542) 5695 : cluster [DBG] pgmap v5994: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:07.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:07 smithi045 conmon[33655]: cluster 2021-07-31T04:00:05.695712+0000 mgr.x (mgr 2021-07-31T04:00:07.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:07 smithi045 conmon[33655]: .24542) 5695 : cluster [DBG] pgmap v5994: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:07.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:07 smithi045 conmon[39718]: cluster 2021-07-31T04:00:05.695712+0000 mgr.x (mgr. 2021-07-31T04:00:07.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:07 smithi045 conmon[39718]: 24542) 5695 : cluster [DBG] pgmap v5994: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:09.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:09 smithi154 conmon[34553]: cluster 2021-07-31T04:00:07.696067+0000 mgr.x (mgr.24542) 5696 : cluster [DBG] pgmap v5995: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:09.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:09 smithi045 conmon[33655]: cluster 2021-07-31T04:00:07.696067+0000 mgr.x (mgr.24542 2021-07-31T04:00:09.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:09 smithi045 conmon[33655]: ) 5696 : cluster [DBG] pgmap v5995: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:09.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:09 smithi045 conmon[39718]: cluster 2021-07-31T04:00:07.696067+0000 mgr.x ( 2021-07-31T04:00:09.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:09 smithi045 conmon[39718]: mgr.24542) 5696 : cluster [DBG] pgmap v5995: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:11.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:11 smithi154 conmon[34553]: cluster 2021-07-31T04:00:09.696497+0000 mgr.x (mgr.24542) 5697 : cluster 2021-07-31T04:00:11.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:11 smithi154 conmon[34553]: [DBG] pgmap v5996: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:11.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:11 smithi045 conmon[39718]: cluster 2021-07-31T04:00:09.696497+0000 mgr.x (mgr.24542) 5697 : cluster [DBG] pgmap v5996: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:11.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:11 smithi045 conmon[33655]: cluster 2021-07-31T04:00:09.696497+0000 mgr.x (mgr.24542) 5697 : cluster 2021-07-31T04:00:11.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:11 smithi045 conmon[33655]: [DBG] pgmap v5996: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:13.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:13 smithi154 conmon[34553]: cluster 2021-07-31T04:00:11.697019+0000 mgr.x (mgr.24542) 5698 : cluster [DBG] pgmap v5997: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:13.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:13 smithi045 conmon[33655]: cluster 2021-07-31T04:00:11.697019+0000 mgr.x (mgr.24542) 5698 : cluster [DBG] pgmap v5997: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:13.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:13 smithi045 conmon[39718]: cluster 2021-07-31T04:00:11.697019+0000 2021-07-31T04:00:13.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:13 smithi045 conmon[39718]: mgr.x (mgr.24542) 5698 : cluster [DBG] pgmap v5997: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:15.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:15 smithi154 conmon[34553]: cluster 2021-07-31T04:00:13.697446+0000 mgr.x (mgr.24542) 5699 : cluster 2021-07-31T04:00:15.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:15 smithi154 conmon[34553]: [DBG] pgmap v5998: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:15.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:15 smithi045 conmon[33655]: cluster 2021-07-31T04:00:13.697446+0000 mgr.x (mgr.24542) 5699 : cluster 2021-07-31T04:00:15.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:15 smithi045 conmon[33655]: [DBG] pgmap v5998: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:15.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:15 smithi045 conmon[39718]: cluster 2021-07-31T04:00:13.697446+0000 2021-07-31T04:00:15.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:15 smithi045 conmon[39718]: mgr.x (mgr.24542) 5699 : cluster [DBG] pgmap v5998: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:17.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:17 smithi154 conmon[34553]: cluster 2021-07-31T04:00:15.697935+0000 mgr.x (mgr.24542) 5700 : cluster [DBG] pgmap v5999: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:17.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:17 smithi045 conmon[39718]: cluster 2021-07-31T04:00:15 2021-07-31T04:00:17.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:17 smithi045 conmon[39718]: .697935+0000 mgr.x (mgr.24542) 5700 : cluster [DBG] pgmap v5999: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:17.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:17 smithi045 conmon[33655]: cluster 2021-07-31T04:00:15.697935+0000 mgr.x (mgr.24542) 5700 : cluster [DBG] pgmap v5999: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:19.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:19 smithi154 conmon[34553]: cluster 2021-07-31T04:00:17.698475+0000 mgr.x (mgr.24542) 5701 : cluster [DBG] pgmap v6000: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:19.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:19 smithi045 conmon[39718]: cluster 2021-07-31T04:00:17.698475+0000 mgr.x (mgr.24542) 5701 : cluster [DBG] pgmap v6000: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:19.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:19 smithi045 conmon[33655]: cluster 2021-07-31T04:00:17.698475+0000 mgr.x (mgr. 2021-07-31T04:00:19.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:19 smithi045 conmon[33655]: 24542) 5701 : cluster [DBG] pgmap v6000: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:21.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:21 smithi154 conmon[34553]: cluster 2021-07-31T04:00:19.698910+0000 mgr.x (mgr.24542) 5702 : cluster [DBG] pgmap v6001: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:21.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:21 smithi154 conmon[34553]: 2021-07-31T04:00:21.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:21 smithi045 conmon[39718]: cluster 2021-07-31T04:00:19.698910+0000 mgr.x (mgr.24542) 5702 : cluster [DBG] pgmap v6001: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:21.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:21 smithi045 conmon[33655]: cluster 2021-07-31T04:00:19.698910+0000 mgr.x (mgr.24542) 5702 : cluster [DBG] pgmap v6001: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:21.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:21 smithi045 conmon[33655]: 2021-07-31T04:00:23.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:23 smithi154 conmon[34553]: cluster 2021-07-31T04:00:21.699451+0000 mgr.x (mgr.24542) 5703 : cluster [DBG] pgmap v6002: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:23.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:23 smithi045 conmon[39718]: cluster 2021-07-31T04:00:21.699451+0000 mgr.x (mgr.24542) 5703 : cluster [DBG] pgmap v6002: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:23.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:23 smithi045 conmon[33655]: cluster 2021-07-31T04:00:21.699451+0000 2021-07-31T04:00:23.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:23 smithi045 conmon[33655]: mgr.x (mgr.24542) 5703 : cluster [DBG] pgmap v6002: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:25.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:25 smithi154 conmon[34553]: cluster 2021-07-31T04:00:23.699831+0000 mgr.x (mgr.24542) 5704 : cluster [DBG] pgmap v6003: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:25.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:25 smithi045 conmon[33655]: cluster 2021-07-31T04:00:23.699831+0000 mgr.x (mgr 2021-07-31T04:00:25.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:25 smithi045 conmon[33655]: .24542) 5704 : cluster [DBG] pgmap v6003: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:25.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:25 smithi045 conmon[39718]: cluster 2021-07-31T04:00:23.699831+0000 mgr.x ( 2021-07-31T04:00:25.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:25 smithi045 conmon[39718]: mgr.24542) 5704 : cluster [DBG] pgmap v6003: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:27.000 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T04:00:27.278 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T04:00:27.278 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 2h ago 2h - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T04:00:27.279 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 2h ago 2h 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T04:00:27.279 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 2h ago 2h 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T04:00:27.279 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 2h ago 2h - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T04:00:27.279 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T04:00:27.280 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 2h ago 2h 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T04:00:27.280 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T04:00:27.280 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 2h ago 2h - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T04:00:27.280 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 2h ago 2h 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T04:00:27.281 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T04:00:27.281 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T04:00:27.281 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T04:00:27.281 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T04:00:27.282 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 2h ago 2h 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T04:00:27.282 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 2h ago 2h 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T04:00:27.282 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 2h ago 2h 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T04:00:27.282 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 2h ago 2h 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T04:00:27.283 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 2h ago 2h 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T04:00:27.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:27 smithi154 conmon[34553]: cluster 2021-07-31T04:00:25.700308+0000 mgr.x (mgr.24542) 5705 : cluster [DBG] pgmap v6004: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:27.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:27 smithi045 conmon[39718]: cluster 2021-07-31T04:00:25.700308+0000 2021-07-31T04:00:27.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:27 smithi045 conmon[39718]: mgr.x (mgr.24542) 5705 : cluster [DBG] pgmap v6004: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:27.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:27 smithi045 conmon[33655]: cluster 2021-07-31T04:00:25.700308+0000 2021-07-31T04:00:27.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:27 smithi045 conmon[33655]: mgr.x (mgr.24542) 5705 : cluster [DBG] pgmap v6004: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:27.618 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T04:00:27.618 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T04:00:27.618 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T04:00:27.619 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:00:27.619 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T04:00:27.619 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T04:00:27.620 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:00:27.620 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:00:27.620 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T04:00:27.620 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T04:00:27.620 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:00:27.621 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T04:00:27.621 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T04:00:27.621 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T04:00:27.621 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:00:27.622 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T04:00:27.622 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T04:00:28.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:28 smithi154 conmon[34553]: audit 2021-07-31T04:00:26.989034+0000 mgr.x (mgr.24542) 5706 : audit [DBG] from='client.30631 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:00:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:28 smithi154 conmon[34553]: audit 2021-07-31T04:00:27.271221+0000 mgr.x (mgr.24542) 5707 : audit [DBG] from='client.20700 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:00:28.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:28 smithi154 conmon[34553]: audit 2021-07-31T04:00:27.619095+0000 mon.c (mon.1) 241 : audit [DBG] from='client.? 172.21.15.45:0/183462807' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:00:28.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:28 smithi045 conmon[33655]: audit 2021-07-31T04:00:26.989034+0000 mgr.x (mgr.24542) 5706 : audit [DBG] from='client.30631 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:00:28.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:28 smithi045 conmon[33655]: audit 2021-07-31T04:00:27.271221+0000 mgr.x (mgr.24542) 5707 : audit [DBG] from='client.20700 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:00:28.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:28 smithi045 conmon[33655]: audit 2021-07-31T04:00:27.619095+0000 mon.c (mon.1 2021-07-31T04:00:28.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:28 smithi045 conmon[33655]: ) 241 : audit [DBG] from='client.? 172.21.15.45:0/183462807' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:00:28.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:28 smithi045 conmon[39718]: audit 2021-07-31T04:00:26.989034+0000 mgr.x 2021-07-31T04:00:28.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:28 smithi045 conmon[39718]: (mgr.24542) 5706 : audit [DBG] from='client.30631 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:00:28.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:28 smithi045 conmon[39718]: audit 2021-07-31T04:00:27. 2021-07-31T04:00:28.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:28 smithi045 conmon[39718]: 271221+0000 mgr.x (mgr.24542) 5707 : audit [DBG] from='client.20700 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:00:28.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:28 smithi045 conmon[39718]: audit 2021-07-31T04:00:28.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:28 smithi045 conmon[39718]: 2021-07-31T04:00:27.619095+0000 mon.c ( 2021-07-31T04:00:28.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:28 smithi045 conmon[39718]: mon.1) 241 : audit [DBG] from='client.? 172.21.15.45:0/183462807' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:00:28.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:28 smithi045 conmon[39718]: 2021-07-31T04:00:29.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:29 smithi154 conmon[34553]: cluster 2021-07-31T04:00:27.700710+0000 mgr.x (mgr.24542) 5708 : cluster [DBG] pgmap v6005: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:29.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:29 smithi154 conmon[34553]: audit 2021-07-31T04:00:28.462334+0000 2021-07-31T04:00:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:29 smithi154 conmon[34553]: mon.b (mon.2) 740 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:00:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:29 smithi154 conmon[34553]: audit 2021-07-31T04:00:28.462735+0000 mon.b (mon.2) 741 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:00:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:29 smithi154 conmon[34553]: audit 2021-07-31T04:00:28.463417+0000 mon.a (mon.0) 3406 : audit [INF] 2021-07-31T04:00:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:29 smithi154 conmon[34553]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:00:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:29 smithi154 conmon[34553]: audit 2021-07-31T04:00: 2021-07-31T04:00:29.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:29 smithi154 conmon[34553]: 28.463810+0000 mon.a (mon.0) 3407 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:00:29.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:29 smithi045 conmon[33655]: cluster 2021-07-31T04:00:27.700710+0000 mgr.x (mgr 2021-07-31T04:00:29.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:29 smithi045 conmon[33655]: .24542) 5708 : cluster [DBG] pgmap v6005: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:29.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:29 smithi045 conmon[33655]: audit 2021-07-31T04:00:28.462334+0000 mon.b (mon.2) 740 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:00:29.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:29 smithi045 conmon[33655]: audit 2021-07-31T04:00:28 2021-07-31T04:00:29.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:29 smithi045 conmon[33655]: .462735+0000 mon.b (mon.2) 2021-07-31T04:00:29.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:29 smithi045 conmon[33655]: 741 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:00:29.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:29 smithi045 conmon[33655]: audit 2021-07-31T04:00:29.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:29 smithi045 conmon[33655]: 2021-07-31T04 2021-07-31T04:00:29.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:29 smithi045 conmon[33655]: :00:28.463417+0000 mon.a (mon 2021-07-31T04:00:29.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:29 smithi045 conmon[33655]: .0) 3406 : audit [INF] 2021-07-31T04:00:29.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:29 smithi045 conmon[33655]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:00:29.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:29 smithi045 conmon[33655]: audit 2021-07-31T04:00:29.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:29 smithi045 conmon[33655]: 2021-07-31 2021-07-31T04:00:29.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:29 smithi045 conmon[33655]: T04:00:28.463810 2021-07-31T04:00:29.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:29 smithi045 conmon[33655]: +0000 mon.a (mon.0) 2021-07-31T04:00:29.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:29 smithi045 conmon[33655]: 3407 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:00:29.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:29 smithi045 conmon[39718]: cluster 2021-07-31T04:00:27.700710+0000 mgr.x (mgr 2021-07-31T04:00:29.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:29 smithi045 conmon[39718]: .24542) 5708 : cluster [DBG] pgmap v6005: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:29.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:29 smithi045 conmon[39718]: audit 2021-07-31T04:00:28.462334+0000 mon.b (mon 2021-07-31T04:00:29.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:29 smithi045 conmon[39718]: .2) 740 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:00:29.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:29 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T04:00:29.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:29 smithi045 conmon[39718]: 04:00:28.462735+0000 mon.b ( 2021-07-31T04:00:29.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:29 smithi045 conmon[39718]: mon.2) 741 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:00:29.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:29 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T04:00:29.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:29 smithi045 conmon[39718]: T04:00:28.463417+0000 mon.a (mon.0 2021-07-31T04:00:29.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:29 smithi045 conmon[39718]: ) 3406 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:00:29.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:29 smithi045 conmon[39718]: audit 2021- 2021-07-31T04:00:29.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:29 smithi045 conmon[39718]: 07-31T04:00:28.463810+0000 2021-07-31T04:00:29.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:29 smithi045 conmon[39718]: mon.a (mon.0) 3407 : audit 2021-07-31T04:00:29.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:29 smithi045 conmon[39718]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:00:31.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:31 smithi154 conmon[34553]: cluster 2021-07-31T04:00:29.701321+0000 mgr.x (mgr.24542) 5709 : cluster [DBG] 2021-07-31T04:00:31.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:31 smithi154 conmon[34553]: pgmap v6006: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:31.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:31 smithi045 conmon[33655]: cluster 2021-07-31T04:00:29.701321+0000 mgr.x (mgr.24542) 5709 : cluster [DBG] pgmap v6006: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:31.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:31 smithi045 conmon[39718]: cluster 2021-07-31T04:00:29.701321+0000 mgr.x (mgr.24542) 2021-07-31T04:00:31.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:31 smithi045 conmon[39718]: 5709 : cluster [DBG] pgmap v6006: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:33.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:33 smithi154 conmon[34553]: cluster 2021-07-31T04:00:31.701718+0000 mgr.x (mgr.24542) 5710 : cluster [DBG] pgmap v6007: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:33.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:33 smithi045 conmon[33655]: cluster 2021-07-31T04:00:31.701718+0000 mgr.x ( 2021-07-31T04:00:33.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:33 smithi045 conmon[33655]: mgr.24542) 5710 : cluster [DBG] pgmap v6007: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:33.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:33 smithi045 conmon[39718]: cluster 2021-07-31T04:00:31.701718+0000 mgr.x 2021-07-31T04:00:33.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:33 smithi045 conmon[39718]: (mgr.24542) 5710 : cluster [DBG] pgmap v6007: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:35.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:35 smithi154 conmon[34553]: cluster 2021-07-31T04:00:33.702173+0000 mgr.x (mgr.24542) 5711 : cluster [DBG] pgmap v6008: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:35.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:35 smithi045 conmon[33655]: cluster 2021-07-31T04:00:33.702173+0000 mgr.x (mgr.24542) 5711 : cluster [DBG] pgmap v6008: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:35.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:35 smithi045 conmon[39718]: cluster 2021-07-31T04:00:33.702173+0000 mgr.x ( 2021-07-31T04:00:35.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:35 smithi045 conmon[39718]: mgr.24542) 5711 : cluster [DBG] pgmap v6008: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:36.753 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: debug 2021-07-31T04:00:36.385+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:00:36.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:00:36.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:00:36.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: yield 2021-07-31T04:00:36.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:00:36.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:00:36.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:00:36.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:00:36.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:00:36.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:00:36.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:00:36.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:00:36.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:00:36.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: task.result() 2021-07-31T04:00:36.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:00:36.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: debug 2021-07-31T04:00:36.397+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:00:36.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:00:36.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:00:36.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: yield 2021-07-31T04:00:36.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:00:36.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:00:36.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:00:36.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:00:36.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:00:36.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:00:36.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:00:36.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:00:36.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:00:36.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: task.result() 2021-07-31T04:00:36.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:00:36.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: debug 2021-07-31T04:00:36.422+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:00:36.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:00:36.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:00:36.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: yield 2021-07-31T04:00:36.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:00:36.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:00:36.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:00:36.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:00:36.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:00:36.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:00:36.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:00:36.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:00:36.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:00:36.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: task.result() 2021-07-31T04:00:36.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:00:36.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: debug 2021-07-31T04:00:36.423+0000 7fcb9095b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:00:36.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:00:36.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:00:36.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: yield 2021-07-31T04:00:36.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:00:36.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:00:36.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:00:36.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:00:36.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:00:36.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:00:36.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:00:36.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:00:36.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:00:36.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: task.result() 2021-07-31T04:00:36.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:00:36.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: 2021-07-31T04:00:36.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T04:00:36.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: 2021-07-31T04:00:36.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:00:36.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:00:36.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: return f(*arg) 2021-07-31T04:00:36.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:00:36.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:00:36.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:00:36.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:00:36.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:00:36.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:00:36.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:00:36.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T04:00:36.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:00:36.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: raise self._exception 2021-07-31T04:00:36.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:00:36.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:00:36.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:00:36.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T04:00:36.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:00:36.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: raise 2021-07-31T04:00:36.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:00:36.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T04:00:36.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:00:36.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:00:36.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:00:36.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: debug 2021-07-31T04:00:36.431+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:00:36.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:00:36.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:00:36.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: yield 2021-07-31T04:00:36.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:00:36.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:00:36.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:00:36.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:00:36.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:00:36.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:00:36.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:00:36.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:00:36.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:00:36.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: task.result() 2021-07-31T04:00:36.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:00:36.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: debug 2021-07-31T04:00:36.432+0000 7fcb9115c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:00:36.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:00:36.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:00:36.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: yield 2021-07-31T04:00:36.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:00:36.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:00:36.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:00:36.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:00:36.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:00:36.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:00:36.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:00:36.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:00:36.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:00:36.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: task.result() 2021-07-31T04:00:36.786 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:00:36.786 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: 2021-07-31T04:00:36.786 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T04:00:36.786 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: 2021-07-31T04:00:36.787 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:00:36.787 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:00:36.787 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: return f(*arg) 2021-07-31T04:00:36.787 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:00:36.788 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:00:36.788 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:00:36.788 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:00:36.788 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:00:36.789 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:00:36.789 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:00:36.789 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T04:00:36.789 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:00:36.790 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: raise self._exception 2021-07-31T04:00:36.790 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:00:36.790 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:00:36.790 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:00:36.791 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T04:00:36.791 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:00:36.791 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: raise 2021-07-31T04:00:36.791 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:00:36.792 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T04:00:36.792 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:00:36.792 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:00:36.792 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:00:36 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:00:37.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:37 smithi154 conmon[34553]: cluster 2021-07-31T04:00:35.702596+0000 mgr.x (mgr.24542) 5712 : cluster [DBG] pgmap v6009: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:37.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:37 smithi154 conmon[34553]: audit 2021-07-31T04:00:36.339596+0000 mon.b (mon.2) 742 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T04:00:37.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:37 smithi154 conmon[34553]: audit 2021-07-31T04:00:36.341006+0000 mon.b (mon.2) 743 : audit [DBG] 2021-07-31T04:00:37.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:37 smithi154 conmon[34553]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T04:00:37.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:37 smithi154 conmon[34553]: audit 2021-07-31 2021-07-31T04:00:37.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:37 smithi154 conmon[34553]: T04:00:36.341942+0000 mon.b (mon.2) 744 : audit 2021-07-31T04:00:37.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:37 smithi154 conmon[34553]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T04:00:37.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:37 smithi045 conmon[33655]: cluster 2021-07-31T04:00:35.702596+0000 mgr.x (mgr.24542) 5712 : cluster [DBG] pgmap v6009: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:37.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:37 smithi045 conmon[33655]: audit 2021-07-31T04:00:36.339596+0000 mon.b ( 2021-07-31T04:00:37.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:37 smithi045 conmon[33655]: mon.2) 742 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T04:00:37.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:37 smithi045 conmon[33655]: audit 2021-07-31T04:00:36.341006+0000 mon.b (mon.2) 743 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T04:00:37.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:37 smithi045 conmon[33655]: audit 2021-07-31 2021-07-31T04:00:37.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:37 smithi045 conmon[33655]: T04:00:36.341942+0000 mon.b (mon.2) 744 : audit 2021-07-31T04:00:37.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:37 smithi045 conmon[33655]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T04:00:37.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:37 smithi045 conmon[39718]: cluster 2021-07-31T04:00:35.702596+0000 mgr.x (mgr.24542) 5712 : cluster [DBG] pgmap v6009: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:37.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:37 smithi045 conmon[39718]: audit 2021-07-31T04:00:36.339596+0000 mon.b (mon.2) 742 : audit [DBG] 2021-07-31T04:00:37.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:37 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T04:00:37.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:37 smithi045 conmon[39718]: audit 2021-07-31T04:00:37.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:37 smithi045 conmon[39718]: 2021-07-31T04:00:36.341006+0000 mon.b (mon.2) 743 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T04:00:37.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:37 smithi045 conmon[39718]: audit 2021-07-31T04:00: 2021-07-31T04:00:37.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:37 smithi045 conmon[39718]: 36.341942+0000 mon.b (mon.2) 744 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T04:00:38.188 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: cephadm 2021-07-31T04:00:36.342676+0000 mgr.x 2021-07-31T04:00:38.188 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: (mgr.24542) 5713 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T04:00:38.189 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: cephadm 2021-07-31T04:00:36. 2021-07-31T04:00:38.189 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: 386428+0000 mgr.x (mgr.24542) 5714 : cephadm [ERR] 2021-07-31T04:00:38.189 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:00:38.189 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:00:38.189 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: yield 2021-07-31T04:00:38.190 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:00:38.190 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:00:38.190 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:00:38.190 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:00:38.191 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:00:38.191 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:00:38.191 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:00:38.191 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:00:38.191 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:00:38.192 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: task.result() 2021-07-31T04:00:38.192 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:00:38.192 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: cephadm 2021-07-31T04:00:36.386908+0000 mgr.x (mgr.24542) 5715 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T04:00:38.192 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: cephadm 2021-07- 2021-07-31T04:00:38.192 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: 31T04:00:36.398191+0000 mgr.x (mgr.24542) 5716 : cephadm [ERR] 2021-07-31T04:00:38.193 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:00:38.193 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:00:38.193 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: yield 2021-07-31T04:00:38.193 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:00:38.193 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:00:38.194 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:00:38.194 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:00:38.194 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:00:38.194 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:00:38.194 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:00:38.195 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:00:38.195 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:00:38.195 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: task.result() 2021-07-31T04:00:38.195 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:00:38.195 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: 2021-07-31T04:00:38.196 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: cephadm 2021-07-31T04:00:36.399174+0000 mgr.x (mgr.24542) 5717 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T04:00:38.196 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: cephadm 2021-07-31T04:00:36.423648+0000 mgr.x (mgr.24542) 5718 : cephadm [ERR] 2021-07-31T04:00:38.196 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:00:38.197 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:00:38.197 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: yield 2021-07-31T04:00:38.197 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:00:38.197 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:00:38.198 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:00:38.198 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:00:38.198 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:00:38.198 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:00:38.198 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:00:38.199 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:00:38.199 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:00:38.199 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: task.result() 2021-07-31T04:00:38.199 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:00:38.199 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: cephadm 2021-07-31T04:00:38.200 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: 2021-07-31T04:00:36.424240+0000 mgr.x (mgr.24542) 5719 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:00:38.200 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:00:38.200 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:00:38.200 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: yield 2021-07-31T04:00:38.200 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:00:38.201 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:00:38.201 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:00:38.201 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:00:38.201 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:00:38.201 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:00:38.202 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:00:38.202 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:00:38.202 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:00:38.202 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: task.result() 2021-07-31T04:00:38.202 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:00:38.203 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: 2021-07-31T04:00:38.203 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T04:00:38.203 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: 2021-07-31T04:00:38.203 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:00:38.203 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:00:38.204 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: return f(*arg) 2021-07-31T04:00:38.204 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:00:38.204 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:00:38.204 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:00:38.205 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:00:38.205 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:00:38.205 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:00:38.205 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:00:38.205 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T04:00:38.206 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:00:38.206 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: raise self._exception 2021-07-31T04:00:38.206 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:00:38.206 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:00:38.206 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:00:38.206 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T04:00:38.207 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:00:38.207 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: raise 2021-07-31T04:00:38.207 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:00:38.207 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T04:00:38.207 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:00:38.208 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:00:38.208 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:00:38.208 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: cephadm 2021-07-31T04:00:36.432575+0000 mgr.x (mgr.24542) 5720 : cephadm [ERR] 2021-07-31T04:00:38.208 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: 2021-07-31T04:00:38.208 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:00:38.209 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:00:38.209 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: yield 2021-07-31T04:00:38.209 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:00:38.209 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:00:38.209 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:00:38.210 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:00:38.210 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:00:38.210 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:00:38.210 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:00:38.210 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:00:38.211 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:00:38.211 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: task.result() 2021-07-31T04:00:38.211 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:00:38.211 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: cephadm 2021-07-31T04:00:36.433009+0000 mgr.x (mgr.24542) 5721 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:00:38.211 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:00:38.211 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:00:38.212 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: yield 2021-07-31T04:00:38.212 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:00:38.212 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:00:38.212 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:00:38.212 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:00:38.213 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:00:38.213 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:00:38.213 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:00:38.213 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:00:38.213 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:00:38.214 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: task.result() 2021-07-31T04:00:38.214 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:00:38.214 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: 2021-07-31T04:00:38.214 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T04:00:38.214 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: 2021-07-31T04:00:38.215 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:00:38.215 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:00:38.215 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: return f(*arg) 2021-07-31T04:00:38.215 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:00:38.215 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:00:38.216 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:00:38.216 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:00:38.216 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:00:38.216 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:00:38.216 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:00:38.217 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T04:00:38.217 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:00:38.217 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: raise self._exception 2021-07-31T04:00:38.217 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:00:38.217 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:00:38.217 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:00:38.218 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T04:00:38.218 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:00:38.218 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: raise 2021-07-31T04:00:38.218 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:00:38.218 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T04:00:38.219 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:00:38.219 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:00:38.219 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:00:38.220 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: cephadm 2021-07-31T04:00:36.342676+0000 mgr.x (mgr.24542) 5713 : cephadm [WRN] 2021-07-31T04:00:38.220 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T04:00:38.220 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: cephadm 2021-07-31T04:00:36.386428+0000 mgr.x (mgr.24542) 5714 : cephadm [ERR] 2021-07-31T04:00:38.220 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:00:38.220 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:00:38.221 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: yield 2021-07-31T04:00:38.221 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:00:38.221 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:00:38.221 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:00:38.221 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:00:38.222 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:00:38.222 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:00:38.222 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:00:38.222 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:00:38.223 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:00:38.223 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: task.result() 2021-07-31T04:00:38.223 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:00:38.223 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: cephadm 2021-07-31T04:00:36.386908+0000 mgr.x (mgr.24542) 5715 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T04:00:38.223 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: cephadm 2021-07-31T04:00:36.398191+0000 mgr.x (mgr.24542) 5716 : cephadm [ERR] 2021-07-31T04:00:38.224 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:00:38.224 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:00:38.224 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: yield 2021-07-31T04:00:38.224 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:00:38.224 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:00:38.225 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:00:38.225 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:00:38.225 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:00:38.225 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:00:38.225 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:00:38.226 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:00:38.226 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:00:38.226 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: task.result() 2021-07-31T04:00:38.226 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:00:38.226 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: cephadm 2021-07-31T04:00:36.399174+0000 mgr.x (mgr.24542) 5717 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T04:00:38.227 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: cephadm 2021-07-31T04:00:36.423648+0000 mgr.x (mgr.24542) 5718 : cephadm [ERR] 2021-07-31T04:00:38.227 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:00:38.227 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:00:38.227 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: yield 2021-07-31T04:00:38.227 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:00:38.227 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:00:38.228 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:00:38.228 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:00:38.228 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:00:38.228 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:00:38.228 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:00:38.229 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:00:38.229 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:00:38.229 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: task.result() 2021-07-31T04:00:38.229 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:00:38.229 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: cephadm 2021-07-31T04:00:36.424240+0000 mgr.x (mgr.24542) 5719 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:00:38.230 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:00:38.230 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:00:38.230 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: yield 2021-07-31T04:00:38.230 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:00:38.230 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:00:38.231 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:00:38.231 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:00:38.231 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:00:38.231 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:00:38.231 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:00:38.232 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:00:38.232 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:00:38.232 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: task.result() 2021-07-31T04:00:38.232 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:00:38.232 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: 2021-07-31T04:00:38.233 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T04:00:38.233 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: 2021-07-31T04:00:38.233 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:00:38.233 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:00:38.233 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: return f(*arg) 2021-07-31T04:00:38.234 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:00:38.234 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:00:38.234 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:00:38.234 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:00:38.234 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:00:38.235 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:00:38.235 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:00:38.235 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T04:00:38.235 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:00:38.235 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: raise self._exception 2021-07-31T04:00:38.236 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:00:38.236 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:00:38.236 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:00:38.236 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T04:00:38.236 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:00:38.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: cephadm 2021-07-31T04:00:36.342676+0000 mgr.x (mgr.24542) 5713 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T04:00:38.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: cephadm 2021-07-31T04:00:36.386428+0000 mgr.x (mgr.24542) 5714 : cephadm [ERR] 2021-07-31T04:00:38.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:00:38.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:00:38.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: yield 2021-07-31T04:00:38.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:00:38.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:00:38.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:00:38.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:00:38.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:00:38.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:00:38.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:00:38.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:00:38.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:00:38.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: task.result() 2021-07-31T04:00:38.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:00:38.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: cephadm 2021-07-31T04:00:36.386908+0000 mgr.x (mgr.24542) 5715 : 2021-07-31T04:00:38.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T04:00:38.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: cephadm 2021-07-31T04:00:36.398191+0000 mgr.x (mgr.24542) 5716 : cephadm [ERR] 2021-07-31T04:00:38.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:00:38.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:00:38.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: yield 2021-07-31T04:00:38.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:00:38.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:00:38.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:00:38.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:00:38.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:00:38.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:00:38.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:00:38.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:00:38.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:00:38.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: task.result() 2021-07-31T04:00:38.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:00:38.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: cephadm 2021-07-31T04:00:36.399174+0000 mgr.x (mgr.24542) 5717 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T04:00:38.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: cephadm 2021-07-31T04:00:36.423648+0000 mgr.x (mgr.24542) 5718 : cephadm [ERR] 2021-07-31T04:00:38.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:00:38.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:00:38.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: yield 2021-07-31T04:00:38.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:00:38.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:00:38.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:00:38.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:00:38.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:00:38.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:00:38.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:00:38.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:00:38.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:00:38.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: task.result() 2021-07-31T04:00:38.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:00:38.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: cephadm 2021-07-31T04:00:36.424240+0000 mgr.x (mgr.24542) 5719 : cephadm 2021-07-31T04:00:38.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:00:38.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:00:38.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:00:38.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: yield 2021-07-31T04:00:38.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:00:38.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:00:38.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:00:38.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:00:38.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:00:38.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:00:38.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:00:38.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:00:38.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:00:38.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: task.result() 2021-07-31T04:00:38.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:00:38.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: 2021-07-31T04:00:38.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T04:00:38.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: 2021-07-31T04:00:38.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:00:38.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:00:38.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: return f(*arg) 2021-07-31T04:00:38.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:00:38.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:00:38.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:00:38.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:00:38.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:00:38.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:00:38.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:00:38.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T04:00:38.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:00:38.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: raise self._exception 2021-07-31T04:00:38.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:00:38.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:00:38.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:00:38.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T04:00:38.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:00:38.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: raise 2021-07-31T04:00:38.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:00:38.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T04:00:38.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:00:38.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:00:38.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:00:38.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: cephadm 2021-07-31T04:00:36.432575+0000 mgr.x (mgr.24542) 5720 : cephadm [ERR] 2021-07-31T04:00:38.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:00:38.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:00:38.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: yield 2021-07-31T04:00:38.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:00:38.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:00:38.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:00:38.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:00:38.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:00:38.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:00:38.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:00:38.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:00:38.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:00:38.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: task.result() 2021-07-31T04:00:38.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:00:38.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: cephadm 2021-07-31T04:00:36.433009+0000 mgr.x (mgr.24542) 5721 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:00:38.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:00:38.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:00:38.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: yield 2021-07-31T04:00:38.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:00:38.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:00:38.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:00:38.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:00:38.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:00:38.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:00:38.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:00:38.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:00:38.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:00:38.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: task.result() 2021-07-31T04:00:38.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:00:38.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: 2021-07-31T04:00:38.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T04:00:38.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: 2021-07-31T04:00:38.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:00:38.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:00:38.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: return f(*arg) 2021-07-31T04:00:38.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:00:38.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:00:38.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:00:38.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:00:38.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:00:38.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:00:38.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:00:38.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T04:00:38.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:00:38.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: raise self._exception 2021-07-31T04:00:38.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:00:38.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:00:38.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:00:38.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T04:00:38.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:00:38.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: raise 2021-07-31T04:00:38.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:00:38.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T04:00:38.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:00:38.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:00:38.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:38 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:00:38.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: raise 2021-07-31T04:00:38.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:00:38.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T04:00:38.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:00:38.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:00:38.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:00:38.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: cephadm 2021-07-31T04:00:36.432575+0000 mgr.x (mgr.24542) 5720 : cephadm [ERR] 2021-07-31T04:00:38.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:00:38.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:00:38.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: yield 2021-07-31T04:00:38.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:00:38.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:00:38.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:00:38.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:00:38.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:00:38.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:00:38.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:00:38.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:00:38.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:00:38.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: task.result() 2021-07-31T04:00:38.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:00:38.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: cephadm 2021-07-31T04:00:38.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: 2021-07-31T04:00:36.433009+0000 mgr.x (mgr.24542) 5721 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:00:38.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:00:38.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:00:38.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: yield 2021-07-31T04:00:38.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:00:38.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:00:38.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:00:38.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:00:38.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:00:38.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:00:38.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:00:38.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:00:38.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:00:38.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: task.result() 2021-07-31T04:00:38.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:00:38.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: 2021-07-31T04:00:38.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T04:00:38.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: 2021-07-31T04:00:38.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:00:38.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:00:38.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: return f(*arg) 2021-07-31T04:00:38.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:00:38.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:00:38.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:00:38.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:00:38.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:00:38.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:00:38.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:00:38.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T04:00:38.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:00:38.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: raise self._exception 2021-07-31T04:00:38.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:00:38.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:00:38.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:00:38.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T04:00:38.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:00:38.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: raise 2021-07-31T04:00:38.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:00:38.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T04:00:38.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:00:38.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:00:38.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:38 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:00:39.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:39 smithi154 conmon[34553]: cluster 2021-07-31T04:00:37.703089+0000 mgr.x (mgr.24542) 5722 : cluster [DBG] pgmap v6010: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:39.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:39 smithi045 conmon[33655]: cluster 2021-07-31T04:00:37.703089 2021-07-31T04:00:39.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:39 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 5722 : cluster [DBG] pgmap v6010: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:39.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:39 smithi045 conmon[39718]: cluster 2021-07-31T04:00:37.703089+0000 mgr.x (mgr 2021-07-31T04:00:39.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:39 smithi045 conmon[39718]: .24542) 5722 : cluster [DBG] pgmap v6010: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:41.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:41 smithi154 conmon[34553]: cluster 2021-07-31T04:00:39.703540+0000 mgr.x (mgr.24542) 5723 : cluster [DBG] pgmap v6011: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:41.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:41 smithi045 conmon[39718]: cluster 2021-07-31T04:00:39.703540+0000 mgr.x (mgr.24542 2021-07-31T04:00:41.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:41 smithi045 conmon[39718]: ) 5723 : cluster [DBG] pgmap v6011: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:41.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:41 smithi045 conmon[33655]: cluster 2021-07-31T04:00:39. 2021-07-31T04:00:41.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:41 smithi045 conmon[33655]: 703540+0000 mgr.x (mgr.24542) 5723 : cluster [DBG] pgmap v6011: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:43.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:43 smithi154 conmon[34553]: cluster 2021-07-31T04:00:41.704030+0000 mgr.x (mgr.24542) 5724 : cluster [DBG] pgmap v6012: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:43.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:43 smithi045 conmon[33655]: cluster 2021-07-31T04:00:41.704030+0000 mgr.x (mgr.24542) 5724 : cluster 2021-07-31T04:00:43.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:43 smithi045 conmon[33655]: [DBG] pgmap v6012: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:43.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:43 smithi045 conmon[39718]: cluster 2021-07-31T04:00 2021-07-31T04:00:43.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:43 smithi045 conmon[39718]: :41.704030+0000 mgr.x (mgr.24542) 5724 : cluster [DBG] pgmap v6012: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:45.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:45 smithi154 conmon[34553]: cluster 2021-07-31T04:00:43.704422+0000 mgr.x (mgr.24542) 5725 : cluster [DBG] pgmap v6013: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:45.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:45 smithi154 conmon[34553]: 2021-07-31T04:00:45.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:45 smithi045 conmon[39718]: cluster 2021-07-31T04:00:43.704422+0000 mgr.x (mgr.24542) 2021-07-31T04:00:45.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:45 smithi045 conmon[39718]: 5725 : cluster [DBG] pgmap v6013: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:45.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:45 smithi045 conmon[33655]: cluster 2021-07-31T04:00:43.704422+0000 mgr.x (mgr.24542) 5725 2021-07-31T04:00:45.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:45 smithi045 conmon[33655]: : cluster [DBG] pgmap v6013: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:47.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:47 smithi154 conmon[34553]: cluster 2021-07-31T04:00:45.704863+0000 mgr.x (mgr.24542) 5726 : 2021-07-31T04:00:47.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:47 smithi154 conmon[34553]: cluster [DBG] pgmap v6014: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:47.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:47 smithi045 conmon[33655]: cluster 2021-07-31T04:00:45.704863+0000 2021-07-31T04:00:47.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:47 smithi045 conmon[33655]: mgr.x (mgr.24542) 5726 : cluster [DBG] pgmap v6014: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:47.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:47 smithi045 conmon[39718]: cluster 2021-07-31T04:00:45. 2021-07-31T04:00:47.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:47 smithi045 conmon[39718]: 704863+0000 mgr.x (mgr.24542) 5726 : cluster [DBG] pgmap v6014: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:49.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:49 smithi154 conmon[34553]: cluster 2021-07-31T04:00:47.705269+0000 mgr.x (mgr.24542) 5727 : cluster [DBG] pgmap v6015: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:49.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:49 smithi045 conmon[33655]: cluster 2021-07-31T04:00:47.705269+0000 mgr.x (mgr.24542) 5727 : 2021-07-31T04:00:49.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:49 smithi045 conmon[33655]: cluster [DBG] pgmap v6015: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:49.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:49 smithi045 conmon[39718]: cluster 2021-07-31T04:00:47.705269+0000 mgr.x (mgr. 2021-07-31T04:00:49.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:49 smithi045 conmon[39718]: 24542) 5727 : cluster [DBG] pgmap v6015: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:51.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:51 smithi154 conmon[34553]: cluster 2021-07-31T04:00:49.705759+0000 mgr.x (mgr.24542) 5728 : cluster 2021-07-31T04:00:51.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:51 smithi154 conmon[34553]: [DBG] pgmap v6016: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:51.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:51 smithi045 conmon[33655]: cluster 2021-07-31T04:00:49.705759+0000 mgr.x (mgr.24542) 5728 : 2021-07-31T04:00:51.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:51 smithi045 conmon[33655]: cluster [DBG] pgmap v6016: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:51.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:51 smithi045 conmon[39718]: cluster 2021-07-31T04:00:49.705759+0000 mgr.x (mgr 2021-07-31T04:00:51.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:51 smithi045 conmon[39718]: .24542) 5728 : cluster [DBG] pgmap v6016: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:53.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:53 smithi154 conmon[34553]: cluster 2021-07-31T04:00:51.706230+0000 mgr.x (mgr.24542) 5729 : cluster 2021-07-31T04:00:53.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:53 smithi154 conmon[34553]: [DBG] pgmap v6017: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:53.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:53 smithi045 conmon[33655]: cluster 2021-07-31T04:00:51.706230+0000 mgr.x (mgr.24542) 5729 : cluster 2021-07-31T04:00:53.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:53 smithi045 conmon[33655]: [DBG] pgmap v6017: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:53.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:53 smithi045 conmon[39718]: cluster 2021-07-31T04:00:51.706230 2021-07-31T04:00:53.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:53 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 5729 : cluster [DBG] pgmap v6017: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:55.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:55 smithi154 conmon[34553]: cluster 2021-07-31T04:00:53.706631+0000 mgr.x (mgr.24542) 5730 : cluster [DBG] pgmap v6018: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:55.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:55 smithi045 conmon[33655]: cluster 2021-07-31T04:00:53.706631+0000 mgr.x (mgr.24542) 5730 : cluster [DBG] pgmap v6018: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:55.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:55 smithi045 conmon[39718]: cluster 2021-07-31T04:00:53.706631+0000 mgr.x (mgr 2021-07-31T04:00:55.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:55 smithi045 conmon[39718]: .24542) 5730 : cluster [DBG] pgmap v6018: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:57.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:57 smithi154 conmon[34553]: cluster 2021-07-31T04:00:55.707149+0000 mgr.x (mgr.24542) 5731 : cluster 2021-07-31T04:00:57.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:57 smithi154 conmon[34553]: [DBG] pgmap v6019: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:57.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:57 smithi045 conmon[33655]: cluster 2021-07-31T04:00:55.707149+0000 mgr.x (mgr.24542) 5731 : cluster 2021-07-31T04:00:57.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:57 smithi045 conmon[33655]: [DBG] pgmap v6019: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:57.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:57 smithi045 conmon[39718]: cluster 2021-07-31T04:00:55.707149+0000 mgr.x 2021-07-31T04:00:57.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:57 smithi045 conmon[39718]: (mgr.24542) 5731 : cluster [DBG] pgmap v6019: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:57.930 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T04:00:58.207 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T04:00:58.207 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 2h ago 2h - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T04:00:58.207 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 2h ago 2h 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T04:00:58.208 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 2h ago 2h 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T04:00:58.208 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 2h ago 2h - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T04:00:58.208 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T04:00:58.209 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 2h ago 2h 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T04:00:58.209 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T04:00:58.209 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 2h ago 2h - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T04:00:58.210 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 2h ago 2h 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T04:00:58.210 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T04:00:58.210 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T04:00:58.211 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T04:00:58.211 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T04:00:58.211 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 2h ago 2h 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T04:00:58.212 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 2h ago 2h 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T04:00:58.212 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 2h ago 2h 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T04:00:58.212 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 2h ago 2h 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T04:00:58.213 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 2h ago 2h 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T04:00:58.542 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T04:00:58.543 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T04:00:58.543 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T04:00:58.543 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:00:58.544 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T04:00:58.544 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T04:00:58.544 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:00:58.545 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:00:58.545 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T04:00:58.545 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T04:00:58.546 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:00:58.546 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T04:00:58.546 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T04:00:58.547 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T04:00:58.547 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:00:58.547 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T04:00:58.548 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T04:00:59.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:59 smithi154 conmon[34553]: cluster 2021-07-31T04:00:57.707525+0000 mgr.x (mgr.24542) 5732 : cluster [DBG] pgmap v6020: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:59.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:59 smithi154 conmon[34553]: audit 2021-07-31T04:00: 2021-07-31T04:00:59.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:59 smithi154 conmon[34553]: 57.919088+0000 mgr.x (mgr.24542) 5733 : audit [DBG] from='client.30649 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:00:59.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:59 smithi154 conmon[34553]: audit 2021-07-31T04:00:58.199496+0000 mgr.x (mgr.24542) 5734 : audit [DBG] from='client.20718 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:00:59.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:00:59 smithi154 conmon[34553]: audit 2021-07-31T04:00:58.543496+0000 mon.c (mon.1) 242 : audit [DBG] from='client.? 172.21.15.45:0/3733612434' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:00:59.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:59 smithi045 conmon[33655]: cluster 2021-07-31T04:00:57.707525+0000 2021-07-31T04:00:59.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:59 smithi045 conmon[33655]: mgr.x (mgr.24542) 5732 : cluster [DBG] pgmap v6020: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:59.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:59 smithi045 conmon[33655]: audit 2021-07-31T04:00:57.919088+0000 mgr.x (mgr.24542) 5733 : audit [DBG] 2021-07-31T04:00:59.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:59 smithi045 conmon[33655]: from='client.30649 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:00:59.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:59 smithi045 conmon[33655]: audit 2021-07-31T04:00:58.199496+0000 mgr.x ( 2021-07-31T04:00:59.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:59 smithi045 conmon[33655]: mgr.24542) 5734 : audit [DBG] from='client.20718 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:00:59.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:59 smithi045 conmon[33655]: audit 2021-07-31T04:00:58.543496+0000 mon.c (mon 2021-07-31T04:00:59.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:00:59 smithi045 conmon[33655]: .1) 242 : audit [DBG] from='client.? 172.21.15.45:0/3733612434' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:00:59.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:59 smithi045 conmon[39718]: cluster 2021-07-31T04:00:57.707525+0000 mgr.x (mgr.24542) 5732 2021-07-31T04:00:59.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:59 smithi045 conmon[39718]: : cluster [DBG] pgmap v6020: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:00:59.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:59 smithi045 conmon[39718]: audit 2021-07-31T04:00:57.919088+0000 mgr.x (mgr.24542) 5733 : audit [DBG] from='client.30649 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:00:59.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:59 smithi045 conmon[39718]: 2021-07-31T04:00:59.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:59 smithi045 conmon[39718]: audit 2021-07-31T04:00:58.199496 2021-07-31T04:00:59.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:59 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 5734 : audit [DBG] from='client.20718 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:00:59.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:59 smithi045 conmon[39718]: audit 2021-07-31T04:00:58.543496+0000 2021-07-31T04:00:59.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:00:59 smithi045 conmon[39718]: mon.c (mon.1) 242 : audit [DBG] from='client.? 172.21.15.45:0/3733612434' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:01:01.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:01 smithi154 conmon[34553]: cluster 2021-07-31T04:00:59.708052+0000 mgr.x (mgr.24542) 5735 : cluster [DBG] pgmap v6021: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:01.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:01 smithi045 conmon[39718]: cluster 2021-07-31T04:00:59.708052+0000 mgr.x (mgr.24542) 5735 : cluster [DBG] pgmap v6021: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:01.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:01 smithi045 conmon[33655]: cluster 2021-07-31T04:00:59.708052+0000 mgr.x (mgr.24542) 5735 : cluster 2021-07-31T04:01:01.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:01 smithi045 conmon[33655]: [DBG] pgmap v6021: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:03.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:03 smithi154 conmon[34553]: cluster 2021-07-31T04:01:01.708551+0000 mgr.x (mgr.24542) 5736 : cluster [DBG] pgmap v6022: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:03.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:03 smithi045 conmon[39718]: cluster 2021-07-31T04:01:01.708551+0000 mgr.x (mgr.24542) 5736 : 2021-07-31T04:01:03.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:03 smithi045 conmon[39718]: cluster [DBG] pgmap v6022: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:03.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:03 smithi045 conmon[33655]: cluster 2021-07-31T04:01:01.708551+0000 mgr.x (mgr. 2021-07-31T04:01:03.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:03 smithi045 conmon[33655]: 24542) 5736 : cluster [DBG] pgmap v6022: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:05.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:05 smithi154 conmon[34553]: cluster 2021-07-31T04:01:03.709018+0000 mgr.x (mgr.24542) 5737 : cluster [DBG] pgmap v6023: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:05.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:05 smithi045 conmon[39718]: cluster 2021-07-31T04:01:03.709018+0000 mgr.x (mgr 2021-07-31T04:01:05.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:05 smithi045 conmon[39718]: .24542) 5737 : cluster [DBG] pgmap v6023: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:05.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:05 smithi045 conmon[33655]: cluster 2021-07-31T04:01:03.709018+0000 mgr.x (mgr.24542) 5737 : 2021-07-31T04:01:05.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:05 smithi045 conmon[33655]: cluster [DBG] pgmap v6023: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:07.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:07 smithi154 conmon[34553]: cluster 2021-07-31T04:01:05.709550+0000 mgr.x (mgr.24542) 5738 : cluster [DBG] pgmap v6024: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:07.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:07 smithi045 conmon[39718]: cluster 2021-07-31T04:01:05.709550+0000 2021-07-31T04:01:07.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:07 smithi045 conmon[39718]: mgr.x (mgr.24542) 5738 : cluster [DBG] pgmap v6024: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:07.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:07 smithi045 conmon[33655]: cluster 2021-07-31T04:01:05.709550+0000 mgr.x (mgr.24542) 2021-07-31T04:01:07.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:07 smithi045 conmon[33655]: 5738 : cluster [DBG] pgmap v6024: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:09.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:09 smithi154 conmon[34553]: cluster 2021-07-31T04:01:07.710015+0000 mgr.x (mgr.24542) 5739 : cluster [DBG] pgmap v6025: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:09.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:09 smithi045 conmon[39718]: cluster 2021-07-31T04:01:07.710015+0000 mgr.x (mgr.24542) 5739 2021-07-31T04:01:09.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:09 smithi045 conmon[39718]: : cluster [DBG] pgmap v6025: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:09.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:09 smithi045 conmon[33655]: cluster 2021-07-31T04:01:07.710015+0000 mgr.x (mgr.24542) 5739 : cluster [DBG] pgmap v6025: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:11.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:11 smithi154 conmon[34553]: cluster 2021-07-31T04:01:09.710460+0000 mgr.x (mgr.24542) 5740 2021-07-31T04:01:11.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:11 smithi154 conmon[34553]: : cluster [DBG] pgmap v6026: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:11.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:11 smithi045 conmon[33655]: cluster 2021-07-31T04:01:09.710460+0000 mgr.x (mgr.24542) 5740 : cluster [DBG] pgmap v6026: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:11.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:11 smithi045 conmon[33655]: 2021-07-31T04:01:11.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:11 smithi045 conmon[39718]: cluster 2021-07-31T04:01:09.710460+0000 2021-07-31T04:01:11.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:11 smithi045 conmon[39718]: mgr.x (mgr.24542) 5740 : cluster [DBG] pgmap v6026: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:13.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:13 smithi154 conmon[34553]: cluster 2021-07-31T04:01:11.710905+0000 mgr.x (mgr.24542) 5741 : cluster [DBG] pgmap v6027: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:13.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:13 smithi045 conmon[33655]: cluster 2021-07-31T04:01:11.710905+0000 mgr.x (mgr.24542) 5741 : cluster [DBG] 2021-07-31T04:01:13.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:13 smithi045 conmon[33655]: pgmap v6027: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:13.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:13 smithi045 conmon[39718]: cluster 2021-07-31T04:01 2021-07-31T04:01:13.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:13 smithi045 conmon[39718]: :11.710905+0000 mgr.x (mgr.24542) 5741 : cluster [DBG] pgmap v6027: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:15.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:15 smithi154 conmon[34553]: cluster 2021-07-31T04:01:13.711334+0000 mgr.x (mgr.24542) 5742 : cluster [DBG] pgmap v6028: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:15.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:15 smithi045 conmon[33655]: cluster 2021-07-31T04:01:13.711334+0000 mgr.x (mgr.24542) 5742 : cluster [DBG] 2021-07-31T04:01:15.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:15 smithi045 conmon[33655]: pgmap v6028: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:15.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:15 smithi045 conmon[39718]: cluster 2021-07-31T04:01:13.711334+0000 mgr.x (mgr 2021-07-31T04:01:15.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:15 smithi045 conmon[39718]: .24542) 5742 : cluster [DBG] pgmap v6028: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:17.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:17 smithi154 conmon[34553]: cluster 2021-07-31T04:01:15.711810+0000 mgr.x (mgr.24542) 5743 : cluster [DBG] pgmap v6029: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:17.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:17 smithi045 conmon[33655]: cluster 2021-07-31T04:01:15.711810+0000 mgr.x (mgr.24542) 5743 : cluster [DBG] 2021-07-31T04:01:17.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:17 smithi045 conmon[33655]: pgmap v6029: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:17.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:17 smithi045 conmon[39718]: cluster 2021-07-31T04:01:15.711810+0000 mgr.x ( 2021-07-31T04:01:17.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:17 smithi045 conmon[39718]: mgr.24542) 5743 : cluster [DBG] pgmap v6029: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:19.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:19 smithi154 conmon[34553]: cluster 2021-07-31T04:01:17.712242+0000 mgr.x (mgr.24542) 5744 : cluster [DBG] pgmap v6030: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:19.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:19 smithi045 conmon[33655]: cluster 2021-07-31T04:01:17.712242+0000 mgr.x (mgr.24542) 5744 2021-07-31T04:01:19.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:19 smithi045 conmon[33655]: : cluster [DBG] pgmap v6030: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:19.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:19 smithi045 conmon[39718]: cluster 2021-07-31T04:01:17.712242+0000 mgr.x (mgr 2021-07-31T04:01:19.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:19 smithi045 conmon[39718]: .24542) 5744 : cluster [DBG] pgmap v6030: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:21.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:21 smithi045 conmon[39718]: cluster 2021-07-31T04:01:19.712805+0000 mgr.x (mgr.24542 2021-07-31T04:01:21.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:21 smithi045 conmon[39718]: ) 5745 : cluster [DBG] pgmap v6031: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:21.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:21 smithi045 conmon[33655]: cluster 2021-07-31T04:01:19.712805+0000 mgr.x (mgr.24542) 5745 : cluster [DBG] 2021-07-31T04:01:21.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:21 smithi045 conmon[33655]: pgmap v6031: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:21.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:21 smithi154 conmon[34553]: cluster 2021-07-31T04:01:19.712805+0000 mgr.x (mgr.24542) 5745 : cluster [DBG] pgmap v6031: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:23.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:23 smithi045 conmon[39718]: cluster 2021-07-31T04:01:21.713209+0000 mgr.x ( 2021-07-31T04:01:23.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:23 smithi045 conmon[39718]: mgr.24542) 5746 : cluster [DBG] pgmap v6032: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:23.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:23 smithi045 conmon[33655]: cluster 2021-07-31T04:01:21.713209+0000 mgr.x (mgr.24542) 5746 : cluster [DBG] pgmap v6032: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:23.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:23 smithi154 conmon[34553]: cluster 2021-07-31T04:01:21.713209+0000 mgr.x (mgr.24542) 5746 : cluster [DBG] pgmap v6032: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:25.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:25 smithi045 conmon[39718]: cluster 2021-07-31T04:01:23.713620+0000 mgr.x (mgr.24542) 5747 : cluster [DBG] pgmap v6033: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:25.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:25 smithi045 conmon[33655]: cluster 2021-07-31T04:01:23.713620+0000 mgr.x (mgr 2021-07-31T04:01:25.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:25 smithi045 conmon[33655]: .24542) 5747 : cluster [DBG] pgmap v6033: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:25.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:25 smithi154 conmon[34553]: cluster 2021-07-31T04:01:23.713620+0000 mgr.x (mgr.24542) 5747 : cluster 2021-07-31T04:01:25.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:25 smithi154 conmon[34553]: [DBG] pgmap v6033: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:27.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:27 smithi045 conmon[39718]: cluster 2021-07-31T04:01:25.714077+0000 mgr.x (mgr.24542) 5748 : cluster [DBG] pgmap v6034: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:27.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:27 smithi045 conmon[33655]: cluster 2021-07-31T04:01:25.714077+0000 mgr.x 2021-07-31T04:01:27.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:27 smithi045 conmon[33655]: (mgr.24542) 5748 : cluster [DBG] pgmap v6034: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:27.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:27 smithi154 conmon[34553]: cluster 2021-07-31T04:01:25.714077+0000 mgr.x (mgr.24542) 5748 : 2021-07-31T04:01:27.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:27 smithi154 conmon[34553]: cluster [DBG] pgmap v6034: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:28.857 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T04:01:29.136 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T04:01:29.136 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 2h ago 2h - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T04:01:29.136 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 2h ago 2h 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T04:01:29.137 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 2h ago 2h 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T04:01:29.137 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 2h ago 2h - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T04:01:29.137 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T04:01:29.137 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 2h ago 2h 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T04:01:29.138 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T04:01:29.138 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 2h ago 2h - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T04:01:29.138 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 2h ago 2h 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T04:01:29.138 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T04:01:29.138 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T04:01:29.139 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T04:01:29.139 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T04:01:29.139 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 2h ago 2h 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T04:01:29.139 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 2h ago 2h 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T04:01:29.140 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 2h ago 2h 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T04:01:29.140 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 2h ago 2h 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T04:01:29.141 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 2h ago 2h 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T04:01:29.472 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T04:01:29.473 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T04:01:29.473 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T04:01:29.473 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:01:29.474 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T04:01:29.474 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T04:01:29.474 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:01:29.475 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:01:29.475 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T04:01:29.475 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T04:01:29.476 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:01:29.476 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T04:01:29.476 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T04:01:29.476 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T04:01:29.477 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:01:29.477 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T04:01:29.477 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T04:01:29.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:29 smithi045 conmon[33655]: cluster 2021-07-31T04:01:27.714515+0000 mgr.x (mgr 2021-07-31T04:01:29.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:29 smithi045 conmon[33655]: .24542) 5749 : cluster [DBG] pgmap v6035: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:29.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:29 smithi045 conmon[33655]: audit 2021-07-31T04:01:28.466053+0000 mon.b (mon.2) 745 2021-07-31T04:01:29.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:29 smithi045 conmon[33655]: : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:01:29.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:29 smithi045 conmon[33655]: audit 2021-07-31 2021-07-31T04:01:29.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:29 smithi045 conmon[33655]: T04:01:28. 2021-07-31T04:01:29.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:29 smithi045 conmon[33655]: 466472+0000 mon.b (mon 2021-07-31T04:01:29.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:29 smithi045 conmon[33655]: .2) 746 : audit 2021-07-31T04:01:29.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:29 smithi045 conmon[33655]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:01:29.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:29 smithi045 conmon[33655]: audit 2021 2021-07-31T04:01:29.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:29 smithi045 conmon[33655]: -07-31T04:01: 2021-07-31T04:01:29.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:29 smithi045 conmon[33655]: 28.467048+0000 mon.a (mon 2021-07-31T04:01:29.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:29 smithi045 conmon[33655]: .0) 3408 : audit 2021-07-31T04:01:29.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:29 smithi045 conmon[33655]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:01:29.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:29 smithi045 conmon[33655]: audit 2021-07-31T04:01:29.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:29 smithi045 conmon[33655]: 2021-07-31T04: 2021-07-31T04:01:29.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:29 smithi045 conmon[33655]: 01:28.467438+0000 mon.a 2021-07-31T04:01:29.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:29 smithi045 conmon[33655]: (mon.0) 3409 : audit 2021-07-31T04:01:29.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:29 smithi045 conmon[33655]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:01:29.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:29 smithi045 conmon[39718]: cluster 2021-07-31T04:01:27.714515+0000 mgr.x (mgr.24542) 5749 2021-07-31T04:01:29.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:29 smithi045 conmon[39718]: : cluster [DBG] pgmap v6035: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:29.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:29 smithi045 conmon[39718]: audit 2021-07-31T04:01:29.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:29 smithi045 conmon[39718]: 2021-07-31T04:01:28 2021-07-31T04:01:29.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:29 smithi045 conmon[39718]: .466053+0000 mon.b (mon. 2021-07-31T04:01:29.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:29 smithi045 conmon[39718]: 2) 745 : audit [INF] 2021-07-31T04:01:29.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:29 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:01:29.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:29 smithi045 conmon[39718]: audit 2021-07-31T04:01:29.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:29 smithi045 conmon[39718]: 2021-07-31T04 2021-07-31T04:01:29.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:29 smithi045 conmon[39718]: :01:28.466472+0000 2021-07-31T04:01:29.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:29 smithi045 conmon[39718]: mon.b (mon.2) 746 : audit 2021-07-31T04:01:29.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:29 smithi045 conmon[39718]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:01:29.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:29 smithi045 conmon[39718]: audit 2021-07-31T04:01:28 2021-07-31T04:01:29.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:29 smithi045 conmon[39718]: .467048+0000 mon.a (mon.0) 3408 2021-07-31T04:01:29.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:29 smithi045 conmon[39718]: : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:01:29.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:29 smithi045 conmon[39718]: audit 2021- 2021-07-31T04:01:29.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:29 smithi045 conmon[39718]: 07-31T04:01:28. 2021-07-31T04:01:29.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:29 smithi045 conmon[39718]: 467438+0000 mon.a (mon.0) 2021-07-31T04:01:29.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:29 smithi045 conmon[39718]: 3409 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:01:29.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:29 smithi154 conmon[34553]: cluster 2021-07-31T04:01:27.714515+0000 mgr.x (mgr.24542) 5749 : cluster [DBG] 2021-07-31T04:01:29.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:29 smithi154 conmon[34553]: pgmap v6035: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:29.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:29 smithi154 conmon[34553]: audit 2021-07-31T04:01:28.466053+0000 mon.b (mon.2) 745 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:01:29.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:29 smithi154 conmon[34553]: audit 2021-07-31T04:01:28.466472+0000 mon.b (mon.2) 746 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:01:29.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:29 smithi154 conmon[34553]: audit 2021-07-31T 2021-07-31T04:01:29.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:29 smithi154 conmon[34553]: 04:01:28.467048+0000 mon.a (mon.0) 3408 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:01:29.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:29 smithi154 conmon[34553]: 2021-07-31T04:01:29.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:29 smithi154 conmon[34553]: audit 2021-07-31T04: 2021-07-31T04:01:29.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:29 smithi154 conmon[34553]: 01:28.467438+0000 mon.a (mon.0) 3409 : 2021-07-31T04:01:29.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:29 smithi154 conmon[34553]: audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:01:30.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:30 smithi045 conmon[39718]: audit 2021-07-31T04:01:28.845176 2021-07-31T04:01:30.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:30 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 5750 : audit [DBG] from='client.30667 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:01:30.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:30 smithi045 conmon[39718]: audit 2021- 2021-07-31T04:01:30.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:30 smithi045 conmon[39718]: 07-31T04:01:29.128524+0000 mgr.x (mgr.24542) 5751 : audit [DBG] from='client.20736 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:01:30.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:30 smithi045 conmon[39718]: 2021-07-31T04:01:30.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:30 smithi045 conmon[39718]: audit 2021 2021-07-31T04:01:30.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:30 smithi045 conmon[39718]: -07-31T04:01:29.473203+0000 mon.c ( 2021-07-31T04:01:30.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:30 smithi045 conmon[39718]: mon.1) 243 : audit [DBG] from='client.? 172.21.15.45:0/2828243815' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:01:30.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:30 smithi045 conmon[33655]: audit 2021-07-31T04:01:28.845176+0000 mgr.x (mgr.24542) 5750 : audit [DBG] from='client.30667 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:01:30.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:30 smithi045 conmon[33655]: audit 2021-07-31T04:01:29.128524+0000 mgr.x (mgr.24542) 5751 : audit [DBG] from='client.20736 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:01:30.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:30 smithi045 conmon[33655]: audit 2021-07-31T04:01:29.473203+0000 mon.c (mon.1) 243 : audit 2021-07-31T04:01:30.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:30 smithi045 conmon[33655]: [DBG] from='client.? 172.21.15.45:0/2828243815' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:01:30.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:30 smithi154 conmon[34553]: audit 2021-07-31T04:01:28.845176+0000 mgr.x (mgr.24542) 5750 : audit [DBG] 2021-07-31T04:01:30.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:30 smithi154 conmon[34553]: from='client.30667 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:01:30.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:30 smithi154 conmon[34553]: audit 2021-07-31T04:01:29.128524+0000 mgr.x (mgr.24542) 5751 : audit [DBG] from='client.20736 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:01:30.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:30 smithi154 conmon[34553]: audit 2021-07-31T04:01:29 2021-07-31T04:01:30.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:30 smithi154 conmon[34553]: .473203+0000 mon.c (mon.1) 243 : audit [DBG] from='client.? 172.21.15.45:0/2828243815' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:01:31.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:31 smithi154 conmon[34553]: cluster 2021-07-31T04:01:29.715054+0000 mgr.x (mgr.24542) 5752 : 2021-07-31T04:01:31.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:31 smithi154 conmon[34553]: cluster [DBG] pgmap v6036: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:31.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:31 smithi045 conmon[33655]: cluster 2021-07-31T04:01:29.715054+0000 mgr.x (mgr 2021-07-31T04:01:31.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:31 smithi045 conmon[33655]: .24542) 5752 : cluster [DBG] pgmap v6036: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:31.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:31 smithi045 conmon[39718]: cluster 2021-07-31T04:01:29. 2021-07-31T04:01:31.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:31 smithi045 conmon[39718]: 715054+0000 mgr.x (mgr.24542) 5752 : cluster [DBG] pgmap v6036: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:33.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:33 smithi154 conmon[34553]: cluster 2021-07-31T04:01:31.715430+0000 mgr.x (mgr.24542) 5753 : cluster [DBG] pgmap v6037: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:33.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:33 smithi045 conmon[39718]: cluster 2021-07-31T04:01:31.715430+0000 mgr.x (mgr. 2021-07-31T04:01:33.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:33 smithi045 conmon[39718]: 24542) 5753 : cluster [DBG] pgmap v6037: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:33.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:33 smithi045 conmon[33655]: cluster 2021-07-31T04:01:31.715430+0000 mgr.x (mgr.24542) 5753 : cluster [DBG] pgmap v6037: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:33.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:33 smithi045 conmon[33655]: 2021-07-31T04:01:35.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:35 smithi154 conmon[34553]: cluster 2021-07-31T04:01:33.715821+0000 mgr.x (mgr.24542) 5754 : cluster [DBG] pgmap v6038: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:35.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:35 smithi045 conmon[39718]: cluster 2021-07-31T04:01:33.715821+0000 mgr.x (mgr. 2021-07-31T04:01:35.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:35 smithi045 conmon[39718]: 24542) 5754 : cluster [DBG] pgmap v6038: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:35.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:35 smithi045 conmon[33655]: cluster 2021-07-31T04:01:33.715821+0000 mgr.x (mgr.24542) 5754 : cluster [DBG] 2021-07-31T04:01:35.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:35 smithi045 conmon[33655]: pgmap v6038: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:36.753 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: debug 2021-07-31T04:01:36.483+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:01:36.753 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:01:36.753 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:01:36.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: yield 2021-07-31T04:01:36.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:01:36.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:01:36.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:01:36.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:01:36.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:01:36.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:01:36.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:01:36.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:01:36.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:01:36.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: task.result() 2021-07-31T04:01:36.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:01:36.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: debug 2021-07-31T04:01:36.494+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:01:36.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:01:36.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:01:36.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: yield 2021-07-31T04:01:36.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:01:36.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:01:36.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:01:36.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:01:36.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:01:36.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:01:36.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:01:36.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:01:36.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:01:36.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: task.result() 2021-07-31T04:01:36.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:01:36.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: debug 2021-07-31T04:01:36.520+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:01:36.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:01:36.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:01:36.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: yield 2021-07-31T04:01:36.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:01:36.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:01:36.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:01:36.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:01:36.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:01:36.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:01:36.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:01:36.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:01:36.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:01:36.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: task.result() 2021-07-31T04:01:36.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:01:36.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: debug 2021-07-31T04:01:36.520+0000 7fcb97969700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:01:36.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:01:36.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:01:36.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: yield 2021-07-31T04:01:36.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:01:36.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:01:36.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:01:36.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:01:36.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:01:36.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:01:36.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:01:36.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:01:36.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:01:36.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: task.result() 2021-07-31T04:01:36.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:01:36.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: 2021-07-31T04:01:36.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T04:01:36.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: 2021-07-31T04:01:36.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:01:36.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:01:36.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: return f(*arg) 2021-07-31T04:01:36.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:01:36.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:01:36.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:01:36.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:01:36.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:01:36.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:01:36.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:01:36.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T04:01:36.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:01:36.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: raise self._exception 2021-07-31T04:01:36.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:01:36.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:01:36.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:01:36.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T04:01:36.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:01:36.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: raise 2021-07-31T04:01:36.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:01:36.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T04:01:36.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:01:36.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:01:36.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:01:36.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: debug 2021-07-31T04:01:36.529+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:01:36.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:01:36.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:01:36.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: yield 2021-07-31T04:01:36.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:01:36.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:01:36.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:01:36.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:01:36.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:01:36.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:01:36.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:01:36.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:01:36.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:01:36.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: task.result() 2021-07-31T04:01:36.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:01:36.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: debug 2021-07-31T04:01:36.529+0000 7fcb9816a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:01:36.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:01:36.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:01:36.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: yield 2021-07-31T04:01:36.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:01:36.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:01:36.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:01:36.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:01:36.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:01:36.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:01:36.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:01:36.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:01:36.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:01:36.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: task.result() 2021-07-31T04:01:36.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:01:36.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: 2021-07-31T04:01:36.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T04:01:36.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: 2021-07-31T04:01:36.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:01:36.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:01:36.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: return f(*arg) 2021-07-31T04:01:36.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:01:36.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:01:36.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:01:36.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:01:36.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:01:36.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:01:36.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:01:36.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T04:01:36.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:01:36.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: raise self._exception 2021-07-31T04:01:36.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:01:36.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:01:36.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:01:36.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T04:01:36.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:01:36.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: raise 2021-07-31T04:01:36.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:01:36.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T04:01:36.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:01:36.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:01:36.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:01:36 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:01:37.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:37 smithi154 conmon[34553]: cluster 2021-07-31T04:01:35.716321+0000 mgr.x (mgr.24542) 5755 : cluster [DBG] pgmap v6039: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:37.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:37 smithi154 conmon[34553]: audit 2021-07-31T04: 2021-07-31T04:01:37.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:37 smithi154 conmon[34553]: 01:36.433779+0000 mon.b (mon.2) 747 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T04:01:37.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:37 smithi154 conmon[34553]: audit 2021-07-31T04:01:36.435065+0000 mon.b (mon.2) 748 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T04:01:37.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:37 smithi154 conmon[34553]: audit 2021-07-31T04:01:37.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:37 smithi154 conmon[34553]: 2021-07-31T04:01:36.435905+0000 mon.b (mon.2) 749 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T04:01:37.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:37 smithi154 conmon[34553]: 2021-07-31T04:01:37.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:37 smithi045 conmon[39718]: cluster 2021-07-31T04:01:35.716321+0000 mgr.x ( 2021-07-31T04:01:37.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:37 smithi045 conmon[39718]: mgr.24542) 5755 : cluster [DBG] pgmap v6039: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:37.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:37 smithi045 conmon[39718]: audit 2021-07-31T04:01:36.433779+0000 mon.b (mon 2021-07-31T04:01:37.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:37 smithi045 conmon[39718]: .2) 747 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T04:01:37.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:37 smithi045 conmon[39718]: audit 2021-07-31T04:01:37.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:37 smithi045 conmon[39718]: 2021-07-31T04:01:36.435065+0000 mon.b (mon.2) 748 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T04:01:37.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:37 smithi045 conmon[39718]: audit 2021-07-31T04:01: 2021-07-31T04:01:37.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:37 smithi045 conmon[39718]: 36.435905+0000 mon.b (mon.2) 749 : audit [INF] 2021-07-31T04:01:37.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:37 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T04:01:37.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:37 smithi045 conmon[33655]: cluster 2021-07-31T04:01:35.716321+0000 mgr.x (mgr.24542) 5755 2021-07-31T04:01:37.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:37 smithi045 conmon[33655]: : cluster [DBG] pgmap v6039: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:37.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:37 smithi045 conmon[33655]: audit 2021-07-31T04:01:36.433779+0000 mon.b (mon.2) 747 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T04:01:37.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:37 smithi045 conmon[33655]: audit 2021-07-31T04:01:36.435065+0000 mon.b (mon.2) 748 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T04:01:37.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:37 smithi045 conmon[33655]: audit 2021-07-31T04:01:36.435905+0000 mon.b (mon.2) 749 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T04:01:38.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: cephadm 2021-07-31T04:01:36.436676+0000 mgr.x (mgr.24542) 5756 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T04:01:38.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: cephadm 2021-07-31T04:01:38.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: 2021-07-31T04:01:36.484098+0000 mgr.x (mgr.24542) 5757 : cephadm [ERR] 2021-07-31T04:01:38.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:01:38.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:01:38.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: yield 2021-07-31T04:01:38.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:01:38.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:01:38.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:01:38.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:01:38.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:01:38.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:01:38.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:01:38.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:01:38.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:01:38.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: task.result() 2021-07-31T04:01:38.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:01:38.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: cephadm 2021-07-31T04:01:36.484850+0000 mgr.x (mgr.24542) 5758 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T04:01:38.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: cephadm 2021-07 2021-07-31T04:01:38.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: -31T04:01:36.495908+0000 mgr.x (mgr.24542) 5759 : cephadm [ERR] 2021-07-31T04:01:38.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:01:38.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:01:38.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: yield 2021-07-31T04:01:38.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:01:38.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:01:38.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:01:38.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:01:38.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:01:38.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:01:38.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:01:38.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:01:38.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:01:38.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: task.result() 2021-07-31T04:01:38.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:01:38.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: cephadm 2021-07-31T04:01:36.496318+0000 mgr.x (mgr.24542) 5760 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T04:01:38.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: cephadm 2021- 2021-07-31T04:01:38.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: 07-31T04:01:36.520937+0000 mgr.x (mgr.24542) 5761 : cephadm [ERR] 2021-07-31T04:01:38.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:01:38.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:01:38.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: yield 2021-07-31T04:01:38.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:01:38.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:01:38.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:01:38.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:01:38.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:01:38.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:01:38.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:01:38.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:01:38.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:01:38.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: task.result() 2021-07-31T04:01:38.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:01:38.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: cephadm 2021-07-31T04:01:36.521508+0000 mgr.x (mgr.24542) 5762 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:01:38.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:01:38.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:01:38.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: yield 2021-07-31T04:01:38.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:01:38.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:01:38.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:01:38.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:01:38.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:01:38.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:01:38.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:01:38.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:01:38.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:01:38.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: task.result() 2021-07-31T04:01:38.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:01:38.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: 2021-07-31T04:01:38.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T04:01:38.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: 2021-07-31T04:01:38.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:01:38.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:01:38.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: return f(*arg) 2021-07-31T04:01:38.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:01:38.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:01:38.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:01:38.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:01:38.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:01:38.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:01:38.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:01:38.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T04:01:38.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:01:38.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: raise self._exception 2021-07-31T04:01:38.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:01:38.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:01:38.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:01:38.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T04:01:38.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:01:38.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: raise 2021-07-31T04:01:38.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:01:38.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T04:01:38.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:01:38.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:01:38.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:01:38.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: cephadm 2021-07-31T04:01:36.529975+0000 mgr.x (mgr.24542) 5763 : cephadm [ERR] 2021-07-31T04:01:38.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:01:38.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:01:38.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: yield 2021-07-31T04:01:38.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:01:38.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:01:38.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:01:38.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:01:38.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:01:38.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:01:38.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:01:38.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:01:38.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:01:38.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: task.result() 2021-07-31T04:01:38.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:01:38.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: cephadm 2021-07-31T04:01:36.530327+0000 mgr.x (mgr.24542) 5764 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:01:38.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:01:38.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:01:38.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: yield 2021-07-31T04:01:38.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:01:38.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:01:38.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:01:38.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:01:38.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:01:38.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:01:38.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:01:38.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:01:38.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:01:38.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: task.result() 2021-07-31T04:01:38.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:01:38.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: 2021-07-31T04:01:38.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T04:01:38.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: 2021-07-31T04:01:38.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:01:38.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:01:38.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: return f(*arg) 2021-07-31T04:01:38.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:01:38.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:01:38.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:01:38.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:01:38.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:01:38.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:01:38.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:01:38.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T04:01:38.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:01:38.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: raise self._exception 2021-07-31T04:01:38.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:01:38.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:01:38.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:01:38.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T04:01:38.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:01:38.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: raise 2021-07-31T04:01:38.782 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:01:38.782 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T04:01:38.782 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:01:38.782 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:01:38.782 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:38 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:01:38.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: cephadm 2021-07-31T04:01:36.436676+0000 mgr.x (mgr. 2021-07-31T04:01:38.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: 24542) 5756 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T04:01:38.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: cephadm 2021-07-31T04:01:36.484098+0000 mgr.x (mgr.24542) 5757 : cephadm [ERR] 2021-07-31T04:01:38.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: 2021-07-31T04:01:38.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:01:38.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:01:38.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: yield 2021-07-31T04:01:38.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:01:38.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:01:38.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:01:38.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:01:38.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:01:38.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:01:38.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:01:38.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:01:38.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:01:38.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: task.result() 2021-07-31T04:01:38.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:01:38.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: cephadm 2021-07-31T04:01:36.484850+0000 mgr.x 2021-07-31T04:01:38.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: (mgr.24542) 5758 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T04:01:38.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: cephadm 2021-07-31T04:01:36.495908+0000 mgr.x (mgr.24542) 5759 : cephadm [ERR] 2021-07-31T04:01:38.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:01:38.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:01:38.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: yield 2021-07-31T04:01:38.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:01:38.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:01:38.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:01:38.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:01:38.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:01:38.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:01:38.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:01:38.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:01:38.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:01:38.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: task.result() 2021-07-31T04:01:38.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:01:38.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: cephadm 2021-07-31T04:01:36.496318+0000 mgr.x (mgr.24542) 5760 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T04:01:38.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: cephadm 2021-07-31T04:01:36.520937+0000 mgr.x (mgr.24542) 5761 : cephadm [ERR] 2021-07-31T04:01:38.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:01:38.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:01:38.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: yield 2021-07-31T04:01:38.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:01:38.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:01:38.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:01:38.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:01:38.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:01:38.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:01:38.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:01:38.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:01:38.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:01:38.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: task.result() 2021-07-31T04:01:38.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:01:38.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: cephadm 2021-07-31T04:01:36.521508+0000 mgr.x (mgr.24542) 5762 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:01:38.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:01:38.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:01:38.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: yield 2021-07-31T04:01:38.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:01:38.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:01:38.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:01:38.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:01:38.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:01:38.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:01:38.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:01:38.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:01:38.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:01:38.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: task.result() 2021-07-31T04:01:38.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:01:38.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: 2021-07-31T04:01:38.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T04:01:38.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: 2021-07-31T04:01:38.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:01:38.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:01:38.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: return f(*arg) 2021-07-31T04:01:38.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:01:38.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:01:38.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:01:38.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:01:38.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:01:38.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:01:38.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:01:38.799 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: cephadm 2021-07-31T04:01:36.436676+0000 mgr.x ( 2021-07-31T04:01:38.799 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: mgr.24542) 5756 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T04:01:38.799 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: cephadm 2021-07-31T04:01:36.484098+0000 2021-07-31T04:01:38.799 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: mgr.x (mgr.24542) 5757 : cephadm [ERR] 2021-07-31T04:01:38.800 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:01:38.800 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:01:38.800 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: yield 2021-07-31T04:01:38.800 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:01:38.800 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:01:38.801 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:01:38.801 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:01:38.801 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:01:38.801 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:01:38.801 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:01:38.802 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:01:38.802 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:01:38.802 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: task.result() 2021-07-31T04:01:38.802 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:01:38.802 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: cephadm 2021-07-31T04:01:38.803 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: 2021-07-31T04:01:36.484850+0000 mgr.x (mgr.24542) 5758 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T04:01:38.803 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: cephadm 2021-07-31T04:01:36.495908+0000 mgr.x (mgr.24542) 5759 : cephadm [ERR] 2021-07-31T04:01:38.803 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:01:38.803 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:01:38.803 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: yield 2021-07-31T04:01:38.804 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:01:38.804 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:01:38.804 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:01:38.804 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:01:38.804 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:01:38.805 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:01:38.805 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:01:38.805 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:01:38.805 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:01:38.805 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: task.result() 2021-07-31T04:01:38.806 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:01:38.806 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: cephadm 2021-07-31T04:01:36.496318+0000 mgr.x (mgr.24542) 5760 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T04:01:38.806 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: cephadm 2021-07-31T04:01:36.520937+0000 mgr.x (mgr.24542) 5761 : cephadm [ERR] 2021-07-31T04:01:38.806 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:01:38.807 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T04:01:38.807 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:01:38.807 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: raise self._exception 2021-07-31T04:01:38.807 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:01:38.808 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:01:38.808 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:01:38.808 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T04:01:38.808 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:01:38.808 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: raise 2021-07-31T04:01:38.809 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:01:38.809 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T04:01:38.809 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:01:38.809 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:01:38.809 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:01:38.810 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: cephadm 2021-07-31T04:01:36.529975+0000 mgr.x (mgr.24542) 5763 : cephadm [ERR] 2021-07-31T04:01:38.810 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:01:38.810 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:01:38.810 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: yield 2021-07-31T04:01:38.810 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:01:38.811 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:01:38.811 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:01:38.811 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:01:38.811 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:01:38.811 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:01:38.812 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:01:38.812 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:01:38.812 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:01:38.812 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: task.result() 2021-07-31T04:01:38.812 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:01:38.813 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: cephadm 2021-07-31T04:01:36.530327+0000 mgr.x (mgr.24542) 5764 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:01:38.813 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:01:38.813 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:01:38.813 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: yield 2021-07-31T04:01:38.813 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:01:38.813 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:01:38.814 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:01:38.814 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:01:38.814 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:01:38.814 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:01:38.815 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:01:38.815 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: yield 2021-07-31T04:01:38.815 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:01:38.815 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:01:38.816 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:01:38.816 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:01:38.816 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:01:38.816 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:01:38.816 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:01:38.816 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:01:38.817 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:01:38.817 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: task.result() 2021-07-31T04:01:38.817 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:01:38.817 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: cephadm 2021-07-31T04:01:36.521508+0000 mgr.x (mgr.24542) 5762 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:01:38.817 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:01:38.818 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:01:38.818 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: yield 2021-07-31T04:01:38.818 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:01:38.818 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:01:38.818 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:01:38.819 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:01:38.819 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:01:38.819 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:01:38.819 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:01:38.819 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:01:38.820 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:01:38.820 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: task.result() 2021-07-31T04:01:38.820 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:01:38.820 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: 2021-07-31T04:01:38.820 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T04:01:38.821 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: 2021-07-31T04:01:38.821 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:01:38.821 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:01:38.821 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: return f(*arg) 2021-07-31T04:01:38.821 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:01:38.822 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:01:38.822 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:01:38.822 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:01:38.822 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:01:38.822 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:01:38.823 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:01:38.823 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:01:38.823 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:01:38.823 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:01:38.824 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: task.result() 2021-07-31T04:01:38.824 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:01:38.824 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: 2021-07-31T04:01:38.824 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T04:01:38.824 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: 2021-07-31T04:01:38.825 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:01:38.825 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:01:38.825 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: return f(*arg) 2021-07-31T04:01:38.825 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:01:38.825 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:01:38.826 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:01:38.826 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:01:38.826 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:01:38.826 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:01:38.826 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:01:38.827 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T04:01:38.827 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:01:38.827 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: raise self._exception 2021-07-31T04:01:38.827 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:01:38.827 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:01:38.828 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:01:38.828 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T04:01:38.828 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:01:38.828 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: raise 2021-07-31T04:01:38.828 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:01:38.828 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T04:01:38.829 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:01:38.829 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:01:38.829 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:01:38.829 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T04:01:38.830 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:01:38.830 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: raise self._exception 2021-07-31T04:01:38.830 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:01:38.830 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:01:38.831 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:01:38.831 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T04:01:38.831 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:01:38.831 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: raise 2021-07-31T04:01:38.831 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:01:38.832 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T04:01:38.832 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:01:38.832 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:01:38.832 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:01:38.832 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: cephadm 2021-07-31T04:01:36. 2021-07-31T04:01:38.833 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: 529975+0000 mgr.x (mgr.24542) 5763 : cephadm [ERR] 2021-07-31T04:01:38.833 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:01:38.833 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:01:38.833 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: yield 2021-07-31T04:01:38.833 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:01:38.833 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:01:38.834 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:01:38.834 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:01:38.834 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:01:38.834 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:01:38.834 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:01:38.835 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:01:38.835 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:01:38.835 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: task.result() 2021-07-31T04:01:38.835 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:01:38.835 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: cephadm 2021-07-31T04:01:36.530327+0000 mgr.x (mgr.24542) 5764 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:01:38.836 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:01:38.836 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:01:38.836 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: yield 2021-07-31T04:01:38.836 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:01:38.836 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:01:38.837 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:01:38.837 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:01:38.837 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:01:38.837 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:01:38.837 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:01:38.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:01:38.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:01:38.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: task.result() 2021-07-31T04:01:38.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:01:38.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: 2021-07-31T04:01:38.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T04:01:38.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: 2021-07-31T04:01:38.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:01:38.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:01:38.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: return f(*arg) 2021-07-31T04:01:38.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:01:38.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:01:38.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:01:38.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:01:38.841 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:01:38.841 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:01:38.841 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:01:38.841 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T04:01:38.841 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:01:38.842 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: raise self._exception 2021-07-31T04:01:38.842 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:01:38.842 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:01:38.842 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:01:38.843 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T04:01:38.843 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:01:38.843 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: raise 2021-07-31T04:01:38.843 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:01:38.843 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T04:01:38.844 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:01:38.844 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:01:38.844 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:38 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:01:39.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:39 smithi154 conmon[34553]: cluster 2021-07-31T04:01:37.716671+0000 mgr.x (mgr.24542) 5765 : cluster [DBG] pgmap v6040: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:39.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:39 smithi045 conmon[33655]: cluster 2021-07-31T04:01:37.716671+0000 mgr.x (mgr.24542) 5765 : cluster [DBG] pgmap v6040: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:39.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:39 smithi045 conmon[39718]: cluster 2021-07-31T04:01:37.716671+0000 mgr.x (mgr.24542) 5765 : cluster 2021-07-31T04:01:39.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:39 smithi045 conmon[39718]: [DBG] pgmap v6040: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:41.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:41 smithi154 conmon[34553]: cluster 2021-07-31T04:01:39.717104+0000 mgr.x (mgr.24542) 5766 : cluster [DBG] pgmap v6041: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:41.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:41 smithi045 conmon[39718]: cluster 2021-07-31T04:01:39.717104+0000 mgr.x (mgr 2021-07-31T04:01:41.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:41 smithi045 conmon[39718]: .24542) 5766 : cluster [DBG] pgmap v6041: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:41.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:41 smithi045 conmon[33655]: cluster 2021-07-31T04:01:39.717104+0000 mgr.x (mgr. 2021-07-31T04:01:41.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:41 smithi045 conmon[33655]: 24542) 5766 : cluster [DBG] pgmap v6041: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:43.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:43 smithi154 conmon[34553]: cluster 2021-07-31T04:01:41.717580+0000 mgr.x (mgr.24542) 5767 : cluster [DBG] pgmap v6042: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:43.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:43 smithi045 conmon[39718]: cluster 2021-07-31T04:01:41.717580+0000 mgr.x (mgr. 2021-07-31T04:01:43.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:43 smithi045 conmon[39718]: 24542) 5767 : cluster [DBG] pgmap v6042: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:43.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:43 smithi045 conmon[33655]: cluster 2021-07-31T04:01:41.717580 2021-07-31T04:01:43.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:43 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 5767 : cluster [DBG] pgmap v6042: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:45.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:45 smithi154 conmon[34553]: cluster 2021-07-31T04:01:43.718134+0000 mgr.x (mgr.24542) 5768 : cluster [DBG] pgmap v6043: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:45.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:45 smithi045 conmon[39718]: cluster 2021-07-31T04:01:43.718134+0000 mgr.x ( 2021-07-31T04:01:45.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:45 smithi045 conmon[39718]: mgr.24542) 5768 : cluster [DBG] pgmap v6043: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:45.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:45 smithi045 conmon[33655]: cluster 2021-07-31T04:01:43.718134+0000 mgr.x (mgr. 2021-07-31T04:01:45.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:45 smithi045 conmon[33655]: 24542) 5768 : cluster [DBG] pgmap v6043: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:47.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:47 smithi154 conmon[34553]: cluster 2021-07-31T04:01:45.718588+0000 mgr.x (mgr.24542) 5769 : cluster [DBG] pgmap v6044: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:47.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:47 smithi045 conmon[33655]: cluster 2021-07-31T04:01:45.718588+0000 mgr.x (mgr.24542 2021-07-31T04:01:47.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:47 smithi045 conmon[33655]: ) 5769 : cluster [DBG] pgmap v6044: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:47.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:47 smithi045 conmon[39718]: cluster 2021-07-31T04:01:45.718588+0000 mgr.x (mgr.24542) 5769 : cluster 2021-07-31T04:01:47.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:47 smithi045 conmon[39718]: [DBG] pgmap v6044: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:49.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:49 smithi154 conmon[34553]: cluster 2021-07-31T04:01:47.719051+0000 mgr.x (mgr.24542) 5770 : cluster [DBG] 2021-07-31T04:01:49.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:49 smithi154 conmon[34553]: pgmap v6045: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:49.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:49 smithi045 conmon[33655]: cluster 2021-07-31T04:01:47.719051+0000 mgr.x (mgr.24542) 5770 2021-07-31T04:01:49.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:49 smithi045 conmon[33655]: : cluster [DBG] pgmap v6045: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:49.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:49 smithi045 conmon[39718]: cluster 2021-07-31T04:01:47.719051+0000 mgr.x (mgr.24542) 2021-07-31T04:01:49.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:49 smithi045 conmon[39718]: 5770 : cluster [DBG] pgmap v6045: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:51.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:51 smithi154 conmon[34553]: cluster 2021-07-31T04:01:49.719523+0000 mgr.x (mgr.24542) 5771 : 2021-07-31T04:01:51.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:51 smithi154 conmon[34553]: cluster [DBG] pgmap v6046: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:51.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:51 smithi045 conmon[33655]: cluster 2021-07-31T04:01:49.719523+0000 mgr.x (mgr.24542) 5771 : cluster [DBG] pgmap v6046: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:51.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:51 smithi045 conmon[39718]: cluster 2021-07-31T04:01:49.719523+0000 mgr.x (mgr.24542 2021-07-31T04:01:51.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:51 smithi045 conmon[39718]: ) 5771 : cluster [DBG] pgmap v6046: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:53.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:53 smithi154 conmon[34553]: cluster 2021-07-31T04:01:51.719868+0000 mgr.x (mgr.24542) 5772 2021-07-31T04:01:53.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:53 smithi154 conmon[34553]: : cluster [DBG] pgmap v6047: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:53.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:53 smithi045 conmon[39718]: cluster 2021-07-31T04:01:51.719868+0000 mgr.x ( 2021-07-31T04:01:53.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:53 smithi045 conmon[39718]: mgr.24542) 5772 : cluster [DBG] pgmap v6047: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:53.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:53 smithi045 conmon[33655]: cluster 2021-07-31T04:01:51.719868+0000 mgr.x (mgr.24542) 5772 : cluster [DBG] pgmap v6047: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:55.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:55 smithi154 conmon[34553]: cluster 2021-07-31T04:01:53.720246+0000 mgr.x (mgr.24542 2021-07-31T04:01:55.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:55 smithi154 conmon[34553]: ) 5773 : cluster [DBG] pgmap v6048: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:55.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:55 smithi045 conmon[39718]: cluster 2021-07-31T04:01:53.720246+0000 mgr.x ( 2021-07-31T04:01:55.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:55 smithi045 conmon[39718]: mgr.24542) 5773 : cluster [DBG] pgmap v6048: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:55.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:55 smithi045 conmon[33655]: cluster 2021-07-31T04:01:53.720246+0000 mgr.x 2021-07-31T04:01:55.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:55 smithi045 conmon[33655]: (mgr.24542) 5773 : cluster [DBG] pgmap v6048: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:57.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:57 smithi154 conmon[34553]: cluster 2021-07-31T04:01:55.720679+0000 mgr.x (mgr.24542) 5774 : cluster 2021-07-31T04:01:57.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:57 smithi154 conmon[34553]: [DBG] pgmap v6049: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:57.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:57 smithi045 conmon[39718]: cluster 2021-07-31T04:01:55.720679+0000 mgr.x (mgr 2021-07-31T04:01:57.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:57 smithi045 conmon[39718]: .24542) 5774 : cluster [DBG] pgmap v6049: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:57.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:57 smithi045 conmon[33655]: cluster 2021-07-31T04:01:55.720679+0000 mgr.x (mgr.24542) 5774 : cluster [DBG] pgmap v6049: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:58.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:58 smithi154 conmon[34553]: cluster 2021-07-31T04:01:57.721152+0000 mgr.x (mgr.24542) 5775 : 2021-07-31T04:01:58.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:01:58 smithi154 conmon[34553]: cluster [DBG] pgmap v6050: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:58.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:58 smithi045 conmon[33655]: cluster 2021-07-31T04:01:57.721152+0000 mgr.x (mgr.24542) 5775 2021-07-31T04:01:58.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:01:58 smithi045 conmon[33655]: : cluster [DBG] pgmap v6050: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:58.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:58 smithi045 conmon[39718]: cluster 2021-07-31T04:01:57.721152+0000 mgr.x (mgr. 2021-07-31T04:01:58.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:01:58 smithi045 conmon[39718]: 24542) 5775 : cluster [DBG] pgmap v6050: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:01:59.785 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T04:02:00.063 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T04:02:00.063 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 2h ago 2h - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T04:02:00.064 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 2h ago 2h 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T04:02:00.064 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 2h ago 2h 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T04:02:00.064 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 2h ago 2h - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T04:02:00.064 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T04:02:00.064 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 2h ago 2h 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T04:02:00.065 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T04:02:00.065 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 2h ago 2h - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T04:02:00.065 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 2h ago 2h 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T04:02:00.065 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T04:02:00.066 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T04:02:00.066 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T04:02:00.066 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T04:02:00.066 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 2h ago 2h 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T04:02:00.067 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 2h ago 2h 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T04:02:00.067 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 2h ago 2h 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T04:02:00.067 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 2h ago 2h 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T04:02:00.067 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 2h ago 2h 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T04:02:00.398 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T04:02:00.399 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T04:02:00.399 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T04:02:00.399 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:02:00.399 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T04:02:00.400 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T04:02:00.400 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:02:00.400 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:02:00.400 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T04:02:00.400 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T04:02:00.401 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:02:00.401 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T04:02:00.401 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T04:02:00.401 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T04:02:00.401 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:02:00.402 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T04:02:00.402 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T04:02:01.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:00 smithi154 conmon[34553]: cluster 2021-07-31T04:01:59.721653+0000 mgr.x (mgr.24542) 5776 : cluster [DBG] 2021-07-31T04:02:01.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:00 smithi154 conmon[34553]: pgmap v6051: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:02:01.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:00 smithi154 conmon[34553]: audit 2021-07-31T04:01:59.773726+0000 mgr.x (mgr.24542) 5777 : audit [DBG] from='client.30685 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:02:01.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:00 smithi154 conmon[34553]: audit 2021-07-31T04:02:00.055692+0000 mgr.x (mgr.24542) 5778 : audit 2021-07-31T04:02:01.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:00 smithi154 conmon[34553]: [DBG] from='client.20754 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:02:01.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:00 smithi154 conmon[34553]: audit 2021-07-31T04 2021-07-31T04:02:01.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:00 smithi154 conmon[34553]: :02:00.399259+0000 mon.a (mon.0) 3410 2021-07-31T04:02:01.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:00 smithi154 conmon[34553]: : audit [DBG] from='client.? 172.21.15.45:0/3444019117' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:02:01.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:00 smithi045 conmon[39718]: cluster 2021-07-31T04:01:59.721653+0000 mgr.x ( 2021-07-31T04:02:01.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:00 smithi045 conmon[39718]: mgr.24542) 5776 : cluster [DBG] pgmap v6051: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:02:01.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:00 smithi045 conmon[39718]: audit 2021-07-31T04 2021-07-31T04:02:01.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:00 smithi045 conmon[39718]: :01:59.773726+0000 mgr.x (mgr.24542) 5777 : audit [DBG] from='client.30685 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:02:01.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:00 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T04:02:01.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:00 smithi045 conmon[39718]: 04:02:00.055692+0000 mgr.x (mgr.24542) 5778 : 2021-07-31T04:02:01.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:00 smithi045 conmon[39718]: audit [DBG] from='client.20754 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:02:01.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:00 smithi045 conmon[39718]: audit 2021- 2021-07-31T04:02:01.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:00 smithi045 conmon[39718]: 07-31T04:02:00.399259+0000 mon.a 2021-07-31T04:02:01.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:00 smithi045 conmon[39718]: (mon.0) 3410 : audit [DBG] from='client.? 172.21.15.45:0/3444019117' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:02:01.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:00 smithi045 conmon[33655]: cluster 2021-07-31T04:01:59.721653+0000 mgr.x (mgr. 2021-07-31T04:02:01.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:00 smithi045 conmon[33655]: 24542) 5776 : cluster [DBG] pgmap v6051: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:02:01.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:00 smithi045 conmon[33655]: audit 2021-07-31T04:01:59.773726+0000 2021-07-31T04:02:01.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:00 smithi045 conmon[33655]: mgr.x (mgr.24542) 5777 : audit [DBG] from='client.30685 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:02:01.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:00 smithi045 conmon[33655]: audit 2021-07-31 2021-07-31T04:02:01.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:00 smithi045 conmon[33655]: T04:02:00.055692+0000 2021-07-31T04:02:01.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:00 smithi045 conmon[33655]: mgr.x (mgr.24542) 5778 : 2021-07-31T04:02:01.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:00 smithi045 conmon[33655]: audit [DBG] from='client.20754 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:02:01.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:00 smithi045 conmon[33655]: audit 2021- 2021-07-31T04:02:01.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:00 smithi045 conmon[33655]: 07-31T04:02:00.399259 2021-07-31T04:02:01.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:00 smithi045 conmon[33655]: +0000 mon.a (mon.0) 2021-07-31T04:02:01.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:00 smithi045 conmon[33655]: 3410 : audit [DBG] from='client.? 172.21.15.45:0/3444019117' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:02:03.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:02 smithi154 conmon[34553]: cluster 2021-07-31T04:02:01.722071+0000 mgr.x (mgr.24542) 5779 2021-07-31T04:02:03.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:02 smithi154 conmon[34553]: : cluster [DBG] pgmap v6052: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:02:03.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:02 smithi045 conmon[33655]: cluster 2021-07-31T04:02:01.722071+0000 mgr.x (mgr.24542 2021-07-31T04:02:03.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:02 smithi045 conmon[33655]: ) 5779 : cluster [DBG] pgmap v6052: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:02:03.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:02 smithi045 conmon[39718]: cluster 2021-07-31T04:02:01.722071+0000 mgr.x (mgr 2021-07-31T04:02:03.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:02 smithi045 conmon[39718]: .24542) 5779 : cluster [DBG] pgmap v6052: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:02:05.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:04 smithi045 conmon[33655]: cluster 2021-07-31T04:02:03.722449+0000 mgr.x (mgr.24542) 5780 : 2021-07-31T04:02:05.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:04 smithi045 conmon[33655]: cluster [DBG] pgmap v6053: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:02:05.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:04 smithi045 conmon[39718]: cluster 2021-07-31T04:02:03.722449+0000 mgr.x (mgr 2021-07-31T04:02:05.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:04 smithi045 conmon[39718]: .24542) 5780 : cluster [DBG] pgmap v6053: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:02:05.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:04 smithi154 conmon[34553]: cluster 2021-07-31T04:02:03.722449+0000 mgr.x (mgr.24542) 5780 : cluster [DBG] 2021-07-31T04:02:05.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:04 smithi154 conmon[34553]: pgmap v6053: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:02:07.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:06 smithi045 conmon[33655]: cluster 2021-07-31T04:02:05.722982+0000 mgr.x (mgr.24542) 5781 : cluster [DBG] 2021-07-31T04:02:07.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:06 smithi045 conmon[33655]: pgmap v6054: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:02:07.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:06 smithi045 conmon[39718]: cluster 2021-07-31T04:02:05.722982+0000 mgr.x (mgr.24542) 5781 : cluster [DBG] pgmap v6054: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:02:07.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:06 smithi154 conmon[34553]: cluster 2021-07-31T04:02:05.722982+0000 mgr.x (mgr.24542) 5781 : 2021-07-31T04:02:07.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:06 smithi154 conmon[34553]: cluster [DBG] pgmap v6054: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:02:09.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:08 smithi045 conmon[33655]: cluster 2021-07-31T04:02:07.723456+0000 2021-07-31T04:02:09.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:08 smithi045 conmon[33655]: mgr.x (mgr.24542) 5782 : cluster [DBG] pgmap v6055: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:02:09.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:08 smithi045 conmon[39718]: cluster 2021-07-31T04:02:07.723456+0000 mgr.x (mgr.24542) 5782 : cluster [DBG] pgmap v6055: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:02:09.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:08 smithi154 conmon[34553]: cluster 2021-07-31T04:02:07.723456+0000 mgr.x (mgr.24542) 5782 : cluster 2021-07-31T04:02:09.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:08 smithi154 conmon[34553]: [DBG] pgmap v6055: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:02:11.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:10 smithi045 conmon[33655]: cluster 2021-07-31T04:02:09.723969+0000 mgr.x (mgr.24542) 5783 : cluster [DBG] pgmap v6056: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:02:11.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:10 smithi045 conmon[39718]: cluster 2021-07-31T04:02:09.723969+0000 mgr.x ( 2021-07-31T04:02:11.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:10 smithi045 conmon[39718]: mgr.24542) 5783 : cluster [DBG] pgmap v6056: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:02:11.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:10 smithi154 conmon[34553]: cluster 2021-07-31T04:02:09.723969+0000 mgr.x (mgr.24542) 5783 : cluster 2021-07-31T04:02:11.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:10 smithi154 conmon[34553]: [DBG] pgmap v6056: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:02:13.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:12 smithi045 conmon[33655]: cluster 2021-07-31T04:02:11.724368+0000 mgr.x (mgr.24542) 5784 : 2021-07-31T04:02:13.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:12 smithi045 conmon[33655]: cluster [DBG] pgmap v6057: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:02:13.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:12 smithi045 conmon[39718]: cluster 2021-07-31T04:02:11.724368+0000 mgr.x (mgr.24542 2021-07-31T04:02:13.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:12 smithi045 conmon[39718]: ) 5784 : cluster [DBG] pgmap v6057: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:02:13.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:12 smithi154 conmon[34553]: cluster 2021-07-31T04:02:11.724368+0000 mgr.x (mgr.24542) 2021-07-31T04:02:13.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:12 smithi154 conmon[34553]: 5784 : cluster [DBG] pgmap v6057: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:02:15.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:14 smithi045 conmon[39718]: cluster 2021-07-31T04:02:13.724736+0000 mgr.x ( 2021-07-31T04:02:15.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:14 smithi045 conmon[39718]: mgr.24542) 5785 : cluster [DBG] pgmap v6058: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:02:15.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:14 smithi045 conmon[33655]: cluster 2021-07-31T04:02:13.724736+0000 mgr.x (mgr.24542) 5785 : cluster [DBG] pgmap v6058: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:02:15.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:14 smithi154 conmon[34553]: cluster 2021-07-31T04:02:13.724736+0000 mgr.x (mgr.24542) 5785 : cluster 2021-07-31T04:02:15.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:14 smithi154 conmon[34553]: [DBG] pgmap v6058: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:02:17.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:16 smithi045 conmon[33655]: cluster 2021-07-31T04:02:15.725189+0000 mgr.x ( 2021-07-31T04:02:17.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:16 smithi045 conmon[33655]: mgr.24542) 5786 : cluster [DBG] pgmap v6059: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:02:17.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:16 smithi045 conmon[39718]: cluster 2021-07-31T04:02:15.725189+0000 mgr.x (mgr 2021-07-31T04:02:17.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:16 smithi045 conmon[39718]: .24542) 5786 : cluster [DBG] pgmap v6059: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:02:17.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:16 smithi154 conmon[34553]: cluster 2021-07-31T04:02:15.725189+0000 mgr.x (mgr.24542) 5786 : cluster [DBG] 2021-07-31T04:02:17.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:16 smithi154 conmon[34553]: pgmap v6059: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:02:19.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:18 smithi154 conmon[34553]: cluster 2021-07-31T04:02:17.725617+0000 mgr.x (mgr.24542) 5787 : cluster [DBG] pgmap v6060: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:02:19.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:18 smithi045 conmon[33655]: cluster 2021-07-31T04:02:17.725617+0000 mgr.x (mgr.24542) 5787 : cluster [DBG] pgmap v6060: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:02:19.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:18 smithi045 conmon[39718]: cluster 2021-07-31T04:02:17.725617+0000 mgr.x (mgr 2021-07-31T04:02:19.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:18 smithi045 conmon[39718]: .24542) 5787 : cluster [DBG] pgmap v6060: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:02:21.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:20 smithi154 conmon[34553]: cluster 2021-07-31T04:02:19.726039+0000 mgr.x (mgr.24542) 5788 : cluster [DBG] pgmap v6061: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:02:21.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:20 smithi045 conmon[39718]: cluster 2021-07-31T04:02:19.726039+0000 mgr.x (mgr. 2021-07-31T04:02:21.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:20 smithi045 conmon[39718]: 24542) 5788 : cluster [DBG] pgmap v6061: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:02:21.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:20 smithi045 conmon[33655]: cluster 2021-07-31T04:02:19.726039+0000 mgr.x (mgr.24542) 5788 : cluster 2021-07-31T04:02:21.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:20 smithi045 conmon[33655]: [DBG] pgmap v6061: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:02:23.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:22 smithi154 conmon[34553]: cluster 2021-07-31T04:02:21.726550+0000 mgr.x (mgr.24542) 5789 : cluster [DBG] 2021-07-31T04:02:23.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:22 smithi154 conmon[34553]: pgmap v6062: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:02:23.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:22 smithi045 conmon[33655]: cluster 2021-07-31T04:02:21.726550+0000 mgr.x (mgr.24542) 5789 : cluster [DBG] pgmap v6062: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:02:23.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:22 smithi045 conmon[39718]: cluster 2021-07-31T04:02:21.726550+0000 mgr.x (mgr 2021-07-31T04:02:23.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:22 smithi045 conmon[39718]: .24542) 5789 : cluster [DBG] pgmap v6062: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:02:25.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:24 smithi154 conmon[34553]: cluster 2021-07-31T04:02:23.727068+0000 mgr.x (mgr.24542) 5790 : cluster [DBG] pgmap v6063: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:02:25.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:24 smithi045 conmon[39718]: cluster 2021-07-31T04:02:23.727068+0000 mgr.x (mgr 2021-07-31T04:02:25.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:24 smithi045 conmon[39718]: .24542) 5790 : cluster [DBG] pgmap v6063: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:02:25.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:24 smithi045 conmon[33655]: cluster 2021-07-31T04:02:23.727068+0000 mgr.x (mgr.24542) 5790 : cluster [DBG] 2021-07-31T04:02:25.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:24 smithi045 conmon[33655]: pgmap v6063: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:02:27.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:26 smithi154 conmon[34553]: cluster 2021-07-31T04:02:25.727458+0000 mgr.x (mgr.24542) 5791 2021-07-31T04:02:27.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:26 smithi154 conmon[34553]: : cluster [DBG] pgmap v6064: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:02:27.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:26 smithi045 conmon[39718]: cluster 2021-07-31T04:02:25.727458 2021-07-31T04:02:27.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:26 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 5791 : cluster [DBG] pgmap v6064: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:02:27.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:26 smithi045 conmon[33655]: cluster 2021-07-31T04:02:25 2021-07-31T04:02:27.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:26 smithi045 conmon[33655]: .727458+0000 mgr.x (mgr.24542) 5791 : cluster [DBG] pgmap v6064: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:02:29.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:28 smithi154 conmon[34553]: cluster 2021-07-31T04:02:27.727877+0000 mgr.x (mgr.24542) 5792 : cluster [DBG] pgmap v6065: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:02:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:28 smithi154 conmon[34553]: audit 2021-07-31T04:02:28.469889+0000 mon.b (mon.2) 750 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:02:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:28 smithi154 conmon[34553]: audit 2021-07-31T04:02:28.470210+0000 mon.b (mon.2) 751 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:02:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:28 smithi154 conmon[34553]: audit 2021-07-31 2021-07-31T04:02:29.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:28 smithi154 conmon[34553]: T04:02:28.470729+0000 mon.a (mon.0) 3411 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:02:29.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:28 smithi154 conmon[34553]: audit 2021-07-31T04 2021-07-31T04:02:29.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:28 smithi154 conmon[34553]: :02:28.471145+0000 mon.a (mon.0) 3412 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:02:29.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:28 smithi045 conmon[33655]: cluster 2021-07-31T04:02:27.727877+0000 mgr.x 2021-07-31T04:02:29.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:28 smithi045 conmon[33655]: (mgr.24542) 5792 : cluster [DBG] pgmap v6065: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:02:29.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:28 smithi045 conmon[33655]: audit 2021-07- 2021-07-31T04:02:29.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:28 smithi045 conmon[33655]: 31T04:02:28.469889+0000 mon.b (mon.2) 750 : audit 2021-07-31T04:02:29.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:28 smithi045 conmon[33655]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:02:29.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:28 smithi045 conmon[33655]: audit 2021-07-31T04:02:28.470210+0000 mon.b 2021-07-31T04:02:29.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:28 smithi045 conmon[33655]: (mon.2) 751 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:02:29.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:28 smithi045 conmon[33655]: audit 2021-07-31T04:02:28.470729+0000 mon.a (mon.0) 2021-07-31T04:02:29.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:28 smithi045 conmon[33655]: 3411 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:02:29.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:28 smithi045 conmon[33655]: audit 2021-07-31 2021-07-31T04:02:29.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:28 smithi045 conmon[33655]: T04:02:28.471145+0000 mon.a (mon.0) 2021-07-31T04:02:29.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:28 smithi045 conmon[33655]: 3412 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:02:29.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:28 smithi045 conmon[39718]: cluster 2021-07-31T04:02:27.727877+0000 mgr.x 2021-07-31T04:02:29.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:28 smithi045 conmon[39718]: (mgr.24542) 5792 : cluster [DBG] pgmap v6065: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:02:29.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:28 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T04:02:29.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:28 smithi045 conmon[39718]: 04:02:28.469889+0000 mon.b (mon.2) 750 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:02:29.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:28 smithi045 conmon[39718]: audit 2021-07-31T04:02:29.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:28 smithi045 conmon[39718]: 2021-07-31T04:02:28.470210+0000 mon.b (mon.2) 751 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:02:29.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:28 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T04:02:29.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:28 smithi045 conmon[39718]: 04:02:28.470729+0000 mon.a (mon.0) 3411 : audit [INF] 2021-07-31T04:02:29.282 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:28 smithi045 conmon[39718]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:02:29.282 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:28 smithi045 conmon[39718]: audit 2021-07-31T04:02:29.282 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:28 smithi045 conmon[39718]: 2021-07-31T04:02:28.471145+0000 mon.a (mon.0) 3412 : audit [INF] 2021-07-31T04:02:29.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:28 smithi045 conmon[39718]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:02:30.710 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T04:02:30.988 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T04:02:30.989 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 2h ago 2h - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T04:02:30.989 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 2h ago 2h 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T04:02:30.989 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 2h ago 2h 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T04:02:30.989 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 2h ago 2h - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T04:02:30.990 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T04:02:30.990 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 2h ago 2h 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T04:02:30.990 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T04:02:30.990 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 2h ago 2h - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T04:02:30.991 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 2h ago 2h 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T04:02:30.991 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T04:02:30.991 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T04:02:30.991 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T04:02:30.992 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T04:02:30.992 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 2h ago 2h 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T04:02:30.992 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 2h ago 2h 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T04:02:30.992 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 2h ago 2h 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T04:02:30.993 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 2h ago 2h 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T04:02:30.993 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 2h ago 2h 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T04:02:31.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:30 smithi154 conmon[34553]: cluster 2021-07-31T04:02:29.728426+0000 mgr.x (mgr.24542) 5793 : cluster [DBG] pgmap v6066: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:02:31.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:30 smithi154 conmon[34553]: 2021-07-31T04:02:31.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:30 smithi045 conmon[33655]: cluster 2021-07-31T04:02:29.728426+0000 mgr.x (mgr.24542) 5793 2021-07-31T04:02:31.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:30 smithi045 conmon[33655]: : cluster [DBG] pgmap v6066: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:02:31.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:30 smithi045 conmon[39718]: cluster 2021-07-31T04:02:29.728426 2021-07-31T04:02:31.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:30 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 5793 : cluster [DBG] pgmap v6066: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:02:31.323 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T04:02:31.324 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T04:02:31.324 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T04:02:31.324 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:02:31.324 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T04:02:31.325 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T04:02:31.325 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:02:31.325 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:02:31.325 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T04:02:31.326 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T04:02:31.326 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:02:31.326 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T04:02:31.326 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T04:02:31.327 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T04:02:31.327 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:02:31.327 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T04:02:31.327 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T04:02:32.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:31 smithi154 conmon[34553]: audit 2021-07-31T04:02:30.698730+0000 mgr.x (mgr.24542) 5794 : audit [DBG] from='client.30703 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:02:32.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:31 smithi154 conmon[34553]: audit 2021-07-31T04:02:30.981145+0000 mgr.x (mgr.24542) 5795 : audit [DBG] from='client.20772 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:02:32.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:31 smithi154 conmon[34553]: audit 2021-07-31T04:02:31.324656+0000 mon.c (mon.1) 244 : audit [DBG] from='client.? 172.21.15.45:0/3386053200' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:02:32.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:31 smithi045 conmon[33655]: audit 2021-07-31T04:02:30.698730+0000 mgr.x (mgr.24542) 2021-07-31T04:02:32.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:31 smithi045 conmon[33655]: 5794 : audit [DBG] from='client.30703 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:02:32.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:31 smithi045 conmon[33655]: audit 2021-07-31T04:02:30.981145+0000 mgr.x (mgr.24542) 5795 : audit [DBG] from='client.20772 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:02:32.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:31 smithi045 conmon[33655]: audit 2021-07-31 2021-07-31T04:02:32.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:31 smithi045 conmon[33655]: T04:02:31.324656+0000 mon.c (mon.1) 244 : audit [DBG] from='client.? 172.21.15.45:0/3386053200' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:02:32.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:31 smithi045 conmon[39718]: audit 2021-07-31T04:02:30.698730+0000 2021-07-31T04:02:32.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:31 smithi045 conmon[39718]: mgr.x (mgr.24542) 5794 : audit [DBG] from='client.30703 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:02:32.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:31 smithi045 conmon[39718]: audit 2021-07 2021-07-31T04:02:32.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:31 smithi045 conmon[39718]: -31T04:02:30.981145+0000 mgr.x (mgr.24542) 5795 : audit [DBG] from='client.20772 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:02:32.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:31 smithi045 conmon[39718]: audit 2021-07 2021-07-31T04:02:32.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:31 smithi045 conmon[39718]: -31T04:02:31.324656+0000 2021-07-31T04:02:32.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:31 smithi045 conmon[39718]: mon.c (mon.1) 244 : audit 2021-07-31T04:02:32.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:31 smithi045 conmon[39718]: [DBG] from='client.? 172.21.15.45:0/3386053200' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:02:33.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:32 smithi154 conmon[34553]: cluster 2021-07-31T04:02:31.729066+0000 mgr.x (mgr.24542) 5796 : cluster [DBG] pgmap v6067: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:02:33.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:32 smithi045 conmon[39718]: cluster 2021-07-31T04:02:31.729066+0000 mgr.x 2021-07-31T04:02:33.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:32 smithi045 conmon[39718]: (mgr.24542) 5796 : cluster [DBG] pgmap v6067: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:02:33.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:32 smithi045 conmon[33655]: cluster 2021-07-31T04:02:31.729066+0000 mgr.x (mgr.24542) 2021-07-31T04:02:33.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:32 smithi045 conmon[33655]: 5796 : cluster [DBG] pgmap v6067: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:02:35.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:34 smithi154 conmon[34553]: cluster 2021-07-31T04:02:33.729440+0000 mgr.x (mgr.24542) 5797 : cluster [DBG] pgmap v6068: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:02:35.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:34 smithi045 conmon[39718]: cluster 2021-07-31T04:02:33.729440+0000 mgr.x 2021-07-31T04:02:35.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:34 smithi045 conmon[39718]: (mgr.24542) 5797 : cluster [DBG] pgmap v6068: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:02:35.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:34 smithi045 conmon[33655]: cluster 2021-07-31T04:02:33.729440+0000 mgr.x (mgr.24542) 5797 : cluster 2021-07-31T04:02:35.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:34 smithi045 conmon[33655]: [DBG] pgmap v6068: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:02:36.813 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: debug 2021-07-31T04:02:36.577+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:02:36.814 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:02:36.814 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:02:36.814 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: yield 2021-07-31T04:02:36.815 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:02:36.815 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:02:36.815 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:02:36.816 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:02:36.816 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:02:36.816 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:02:36.817 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:02:36.817 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:02:36.817 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:02:36.818 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: task.result() 2021-07-31T04:02:36.818 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:02:36.818 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: debug 2021-07-31T04:02:36.589+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:02:36.819 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:02:36.819 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:02:36.819 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: yield 2021-07-31T04:02:36.820 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:02:36.820 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:02:36.820 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:02:36.821 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:02:36.821 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:02:36.821 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:02:36.821 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:02:36.822 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:02:36.822 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:02:36.822 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: task.result() 2021-07-31T04:02:36.823 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:02:36.823 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: debug 2021-07-31T04:02:36.617+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:02:36.823 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:02:36.824 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:02:36.824 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: yield 2021-07-31T04:02:36.824 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:02:36.824 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:02:36.825 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:02:36.825 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:02:36.825 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:02:36.826 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:02:36.826 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:02:36.826 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:02:36.826 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:02:36.827 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: task.result() 2021-07-31T04:02:36.827 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:02:36.827 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: debug 2021-07-31T04:02:36.617+0000 7fcb94162700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:02:36.828 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:02:36.828 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:02:36.828 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: yield 2021-07-31T04:02:36.828 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:02:36.829 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:02:36.829 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:02:36.829 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:02:36.830 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:02:36.830 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:02:36.830 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:02:36.830 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:02:36.831 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:02:36.831 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: task.result() 2021-07-31T04:02:36.831 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:02:36.832 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: 2021-07-31T04:02:36.832 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T04:02:36.832 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: 2021-07-31T04:02:36.832 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:02:36.833 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:02:36.833 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: return f(*arg) 2021-07-31T04:02:36.833 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:02:36.834 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:02:36.834 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:02:36.834 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:02:36.834 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:02:36.835 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:02:36.835 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:02:36.835 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T04:02:36.836 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:02:36.836 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: raise self._exception 2021-07-31T04:02:36.836 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:02:36.836 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:02:36.837 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:02:36.837 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T04:02:36.837 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:02:36.837 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: raise 2021-07-31T04:02:36.838 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:02:36.838 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T04:02:36.838 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:02:36.839 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:02:36.839 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:02:36.839 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: debug 2021-07-31T04:02:36.626+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:02:36.839 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:02:36.840 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:02:36.840 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: yield 2021-07-31T04:02:36.840 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:02:36.840 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:02:36.841 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:02:36.841 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:02:36.841 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:02:36.842 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:02:36.842 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:02:36.842 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:02:36.842 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:02:36.843 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: task.result() 2021-07-31T04:02:36.843 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:02:36.843 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: debug 2021-07-31T04:02:36.626+0000 7fcb93961700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:02:36.844 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:02:36.844 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:02:36.844 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: yield 2021-07-31T04:02:36.844 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:02:36.845 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:02:36.845 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:02:36.845 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:02:36.846 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:02:36.846 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:02:36.846 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:02:36.846 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:02:36.847 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:02:36.847 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: task.result() 2021-07-31T04:02:36.847 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:02:36.847 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: 2021-07-31T04:02:36.848 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T04:02:36.848 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: 2021-07-31T04:02:36.848 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:02:36.848 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:02:36.849 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: return f(*arg) 2021-07-31T04:02:36.849 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:02:36.849 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:02:36.850 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:02:36.850 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:02:36.850 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:02:36.850 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:02:36.851 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:02:36.851 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T04:02:36.851 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:02:36.851 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: raise self._exception 2021-07-31T04:02:36.852 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:02:36.852 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:02:36.852 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:02:36.852 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T04:02:36.853 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:02:36.853 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: raise 2021-07-31T04:02:36.853 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:02:36.853 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T04:02:36.854 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:02:36.854 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:02:36.854 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:02:37.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[34553]: cluster 2021-07-31T04:02:35.729830+0000 mgr.x (mgr.24542 2021-07-31T04:02:37.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[34553]: ) 5798 : cluster [DBG] pgmap v6069: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:02:37.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[34553]: audit 2021-07-31T04:02:36.531278+0000 mon.b ( 2021-07-31T04:02:37.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[34553]: mon.2) 752 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T04:02:37.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[34553]: audit 2021-07-31 2021-07-31T04:02:37.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[34553]: T04:02:36.532736+0000 mon.b (mon.2) 2021-07-31T04:02:37.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[34553]: 753 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T04:02:37.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[34553]: audit 2021- 2021-07-31T04:02:37.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[34553]: 07-31T04:02:36.533594+0000 mon.b 2021-07-31T04:02:37.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:36 smithi154 conmon[34553]: (mon.2) 754 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T04:02:37.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:36 smithi045 conmon[39718]: cluster 2021-07-31T04:02:35.729830+0000 mgr.x (mgr.24542) 5798 : cluster [DBG] pgmap v6069: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:02:37.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:36 smithi045 conmon[39718]: audit 2021-07-31T04:02:36.531278+0000 mon.b (mon.2) 752 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T04:02:37.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:36 smithi045 conmon[39718]: audit 2021-07-31T04:02:36.532736+0000 mon.b (mon.2) 753 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T04:02:37.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:36 smithi045 conmon[39718]: audit 2021-07-31T04: 2021-07-31T04:02:37.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:36 smithi045 conmon[39718]: 02:36.533594+0000 mon.b (mon.2) 754 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T04:02:37.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:36 smithi045 conmon[33655]: cluster 2021-07-31T04:02:35.729830 2021-07-31T04:02:37.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:36 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 5798 : cluster [DBG] pgmap v6069: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:02:37.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:36 smithi045 conmon[33655]: audit 2021-07- 2021-07-31T04:02:37.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:36 smithi045 conmon[33655]: 31T04:02:36.531278+0000 mon.b 2021-07-31T04:02:37.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:36 smithi045 conmon[33655]: (mon.2) 752 : audit [DBG] 2021-07-31T04:02:37.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:36 smithi045 conmon[33655]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T04:02:37.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:36 smithi045 conmon[33655]: audit 2021-07-31T04:02:37.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:36 smithi045 conmon[33655]: 2021-07-31T04:02:36.532736 2021-07-31T04:02:37.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:36 smithi045 conmon[33655]: +0000 mon.b (mon.2) 753 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T04:02:37.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:36 smithi045 conmon[33655]: audit 2021-07-31T04:02:36.533594+0000 mon.b (mon.2) 754 2021-07-31T04:02:37.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:36 smithi045 conmon[33655]: : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T04:02:38.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: cephadm 2021-07-31T04:02:36.534327+0000 mgr.x (mgr.24542) 5799 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T04:02:38.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: cephadm 2021-07-31T04:02:36. 2021-07-31T04:02:38.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: 578772+0000 mgr.x (mgr.24542) 5800 : cephadm [ERR] 2021-07-31T04:02:38.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:02:38.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:02:38.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: yield 2021-07-31T04:02:38.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:02:38.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:02:38.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:02:38.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:02:38.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:02:38.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:02:38.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:02:38.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:02:38.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:02:38.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: task.result() 2021-07-31T04:02:38.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:02:38.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: cephadm 2021-07-31T04:02:36.579161+0000 mgr.x (mgr.24542) 5801 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T04:02:38.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: cephadm 2021-07-31T04:02:36.590363+0000 mgr.x (mgr.24542 2021-07-31T04:02:38.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: ) 5802 : cephadm [ERR] 2021-07-31T04:02:38.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:02:38.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:02:38.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: yield 2021-07-31T04:02:38.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:02:38.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:02:38.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:02:38.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:02:38.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:02:38.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:02:38.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:02:38.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:02:38.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:02:38.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: task.result() 2021-07-31T04:02:38.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:02:38.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: cephadm 2021-07-31T04:02:36.590822+0000 mgr.x (mgr.24542) 5803 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T04:02:38.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: cephadm 2021-07-31T04:02:36.618067+0000 mgr.x (mgr.24542) 5804 : cephadm [ERR] 2021-07-31T04:02:38.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:02:38.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:02:38.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: yield 2021-07-31T04:02:38.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:02:38.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:02:38.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:02:38.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:02:38.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:02:38.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:02:38.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:02:38.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:02:38.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:02:38.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: task.result() 2021-07-31T04:02:38.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:02:38.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: cephadm 2021-07-31T04:02:36.618475+0000 mgr.x (mgr.24542) 5805 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:02:38.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:02:38.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:02:38.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: yield 2021-07-31T04:02:38.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:02:38.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:02:38.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:02:38.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:02:38.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:02:38.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:02:38.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:02:38.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:02:38.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:02:38.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: task.result() 2021-07-31T04:02:38.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:02:38.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: 2021-07-31T04:02:38.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T04:02:38.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: 2021-07-31T04:02:38.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:02:38.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:02:38.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: return f(*arg) 2021-07-31T04:02:38.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:02:38.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:02:38.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:02:38.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:02:38.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:02:38.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:02:38.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:02:38.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T04:02:38.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:02:38.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: raise self._exception 2021-07-31T04:02:38.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:02:38.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:02:38.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:02:38.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T04:02:38.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:02:38.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: raise 2021-07-31T04:02:38.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:02:38.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T04:02:38.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:02:38.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:02:38.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:02:38.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: cephadm 2021-07-31T04:02:36.627031+0000 mgr.x (mgr.24542) 5806 : cephadm [ERR] 2021-07-31T04:02:38.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:02:38.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:02:38.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: yield 2021-07-31T04:02:38.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:02:38.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:02:38.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:02:38.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:02:38.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:02:38.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:02:38.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:02:38.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:02:38.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:02:38.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: task.result() 2021-07-31T04:02:38.285 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:02:38.285 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: cephadm 2021-07-31T04:02:36.627452+0000 mgr.x (mgr.24542) 5807 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:02:38.285 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:02:38.285 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:02:38.286 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: yield 2021-07-31T04:02:38.286 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:02:38.286 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:02:38.286 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:02:38.287 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:02:38.287 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:02:38.287 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:02:38.288 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:02:38.288 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:02:38.288 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:02:38.288 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: task.result() 2021-07-31T04:02:38.289 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:02:38.289 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: 2021-07-31T04:02:38.289 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T04:02:38.289 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: 2021-07-31T04:02:38.290 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:02:38.290 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:02:38.290 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: return f(*arg) 2021-07-31T04:02:38.290 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:02:38.291 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:02:38.291 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:02:38.291 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:02:38.291 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:02:38.292 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:02:38.292 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:02:38.292 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T04:02:38.293 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:02:38.293 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: raise self._exception 2021-07-31T04:02:38.293 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:02:38.293 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:02:38.294 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:02:38.294 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T04:02:38.294 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:02:38.294 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: raise 2021-07-31T04:02:38.295 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:02:38.295 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T04:02:38.295 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:02:38.295 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:02:38.296 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:37 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:02:38.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: cephadm 2021-07-31T04:02:36.534327+0000 mgr.x (mgr.24542) 5799 2021-07-31T04:02:38.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T04:02:38.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: cephadm 2021-07-31T04:02:36.578772+0000 mgr.x (mgr. 2021-07-31T04:02:38.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: 24542) 5800 : cephadm [ERR] 2021-07-31T04:02:38.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:02:38.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:02:38.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: yield 2021-07-31T04:02:38.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:02:38.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:02:38.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:02:38.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:02:38.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:02:38.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:02:38.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:02:38.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:02:38.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:02:38.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: task.result() 2021-07-31T04:02:38.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:02:38.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: cephadm 2021-07-31T04:02:36.579161+0000 mgr.x (mgr.24542) 5801 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T04:02:38.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: cephadm 2021-07-31T04:02:36.590363+0000 mgr.x (mgr.24542) 5802 : cephadm [ERR] 2021-07-31T04:02:38.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:02:38.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:02:38.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: yield 2021-07-31T04:02:38.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:02:38.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:02:38.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:02:38.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:02:38.304 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:02:38.304 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:02:38.304 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:02:38.304 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:02:38.305 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:02:38.305 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: task.result() 2021-07-31T04:02:38.305 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:02:38.305 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: cephadm 2021-07-31T04:02:36.590822+0000 mgr.x (mgr.24542) 5803 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T04:02:38.306 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: cephadm 2021-07-31T04:02:36. 2021-07-31T04:02:38.306 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: 618067+0000 mgr.x (mgr.24542) 5804 : cephadm [ERR] 2021-07-31T04:02:38.306 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:02:38.307 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:02:38.307 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: yield 2021-07-31T04:02:38.307 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:02:38.307 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:02:38.308 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:02:38.308 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:02:38.308 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:02:38.308 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:02:38.308 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:02:38.309 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:02:38.309 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:02:38.309 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: task.result() 2021-07-31T04:02:38.309 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:02:38.310 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: 2021-07-31T04:02:38.310 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: cephadm 2021-07-31T04:02:36.618475+0000 mgr.x (mgr.24542) 5805 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:02:38.310 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:02:38.310 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:02:38.311 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: yield 2021-07-31T04:02:38.311 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:02:38.311 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:02:38.311 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:02:38.312 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:02:38.312 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:02:38.312 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:02:38.312 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:02:38.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:02:38.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:02:38.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: task.result() 2021-07-31T04:02:38.314 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:02:38.314 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: 2021-07-31T04:02:38.314 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T04:02:38.314 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: 2021-07-31T04:02:38.315 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:02:38.315 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:02:38.315 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: return f(*arg) 2021-07-31T04:02:38.315 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:02:38.316 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:02:38.316 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:02:38.316 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:02:38.316 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:02:38.316 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:02:38.317 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:02:38.317 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: cephadm 2021-07-31T04:02:36.534327+0000 mgr.x (mgr. 2021-07-31T04:02:38.318 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: 24542) 5799 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T04:02:38.318 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: cephadm 2021-07-31T04:02 2021-07-31T04:02:38.318 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: :36.578772+0000 mgr.x (mgr.24542) 5800 : cephadm [ERR] 2021-07-31T04:02:38.318 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:02:38.319 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:02:38.319 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: yield 2021-07-31T04:02:38.319 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:02:38.319 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:02:38.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:02:38.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:02:38.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:02:38.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:02:38.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:02:38.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:02:38.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:02:38.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: task.result() 2021-07-31T04:02:38.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:02:38.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: 2021-07-31T04:02:38.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: cephadm 2021-07-31T04:02:36.579161+0000 mgr.x (mgr.24542) 5801 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T04:02:38.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: cephadm 2021-07-31T04:02 2021-07-31T04:02:38.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: :36.590363+0000 mgr.x (mgr.24542) 2021-07-31T04:02:38.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: 5802 : cephadm [ERR] 2021-07-31T04:02:38.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:02:38.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:02:38.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: yield 2021-07-31T04:02:38.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:02:38.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:02:38.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:02:38.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:02:38.325 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:02:38.325 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:02:38.325 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:02:38.325 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:02:38.326 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:02:38.326 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: task.result() 2021-07-31T04:02:38.326 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:02:38.326 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: cephadm 2021-07-31T04:02:36.590822+0000 mgr.x (mgr.24542) 5803 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T04:02:38.326 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: cephadm 2021-07-31T04:02: 2021-07-31T04:02:38.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: 36.618067+0000 mgr.x (mgr.24542) 5804 : cephadm 2021-07-31T04:02:38.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: [ERR] 2021-07-31T04:02:38.327 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T04:02:38.328 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:02:38.328 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: raise self._exception 2021-07-31T04:02:38.328 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:02:38.328 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:02:38.329 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:02:38.329 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T04:02:38.329 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:02:38.329 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: raise 2021-07-31T04:02:38.330 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:02:38.330 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T04:02:38.330 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:02:38.330 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:02:38.331 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:02:38.331 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: cephadm 2021-07-31T04:02:36.627031+0000 mgr.x ( 2021-07-31T04:02:38.331 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: mgr.24542) 5806 : cephadm [ERR] 2021-07-31T04:02:38.331 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:02:38.332 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:02:38.332 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: yield 2021-07-31T04:02:38.332 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:02:38.332 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:02:38.333 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:02:38.333 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:02:38.333 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:02:38.333 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:02:38.333 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:02:38.334 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:02:38.334 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:02:38.334 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: task.result() 2021-07-31T04:02:38.334 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:02:38.335 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: cephadm 2021-07-31T04:02:36.627452+0000 mgr.x (mgr.24542) 5807 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:02:38.335 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:02:38.335 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:02:38.335 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: yield 2021-07-31T04:02:38.336 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:02:38.336 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:02:38.336 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:02:38.336 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:02:38.336 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:02:38.337 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:02:38.337 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:02:38.337 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: yield 2021-07-31T04:02:38.338 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:02:38.338 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:02:38.338 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:02:38.338 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:02:38.339 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:02:38.339 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:02:38.339 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:02:38.339 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:02:38.339 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:02:38.340 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: task.result() 2021-07-31T04:02:38.340 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:02:38.340 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: cephadm 2021-07-31T04:02:36.618475+0000 mgr.x (mgr.24542) 5805 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:02:38.340 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:02:38.341 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:02:38.341 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: yield 2021-07-31T04:02:38.341 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:02:38.341 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:02:38.342 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:02:38.342 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:02:38.342 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:02:38.342 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:02:38.342 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:02:38.343 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:02:38.343 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:02:38.343 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: task.result() 2021-07-31T04:02:38.343 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:02:38.344 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: 2021-07-31T04:02:38.344 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T04:02:38.344 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: 2021-07-31T04:02:38.344 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:02:38.344 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:02:38.345 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: return f(*arg) 2021-07-31T04:02:38.345 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:02:38.345 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:02:38.345 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:02:38.346 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:02:38.346 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:02:38.346 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:02:38.347 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:02:38.347 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:02:38.347 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:02:38.347 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: task.result() 2021-07-31T04:02:38.348 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:02:38.348 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: 2021-07-31T04:02:38.348 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T04:02:38.348 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: 2021-07-31T04:02:38.348 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:02:38.349 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:02:38.349 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: return f(*arg) 2021-07-31T04:02:38.349 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:02:38.349 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:02:38.350 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:02:38.350 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:02:38.350 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:02:38.350 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:02:38.350 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:02:38.351 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T04:02:38.351 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:02:38.351 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: raise self._exception 2021-07-31T04:02:38.351 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:02:38.352 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:02:38.352 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:02:38.352 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T04:02:38.352 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:02:38.352 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: raise 2021-07-31T04:02:38.353 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:02:38.353 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T04:02:38.353 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:02:38.353 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:02:38.354 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:02:38.354 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:02:38.354 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:02:38.355 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T04:02:38.355 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:02:38.355 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: raise self._exception 2021-07-31T04:02:38.355 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:02:38.356 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:02:38.356 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:02:38.356 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T04:02:38.356 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:02:38.356 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: raise 2021-07-31T04:02:38.357 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:02:38.357 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T04:02:38.357 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:02:38.357 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:02:38.358 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:02:38.358 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: cephadm 2021-07-31T 2021-07-31T04:02:38.358 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: 04:02:36.627031+0000 mgr.x (mgr.24542) 5806 : cephadm [ERR] 2021-07-31T04:02:38.358 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:02:38.358 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:02:38.359 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: yield 2021-07-31T04:02:38.359 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:02:38.359 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:02:38.359 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:02:38.360 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:02:38.360 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:02:38.360 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:02:38.360 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:02:38.360 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:02:38.361 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:02:38.361 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: task.result() 2021-07-31T04:02:38.361 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:02:38.361 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: cephadm 2021-07-31T04:02:36.627452+0000 mgr.x (mgr.24542) 5807 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:02:38.361 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:02:38.362 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:02:38.362 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: yield 2021-07-31T04:02:38.362 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:02:38.362 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:02:38.363 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:02:38.363 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:02:38.363 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:02:38.363 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:02:38.363 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:02:38.364 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:02:38.364 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:02:38.364 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: task.result() 2021-07-31T04:02:38.364 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:02:38.365 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: 2021-07-31T04:02:38.365 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T04:02:38.365 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: 2021-07-31T04:02:38.365 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:02:38.365 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:02:38.366 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: return f(*arg) 2021-07-31T04:02:38.366 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:02:38.366 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:02:38.366 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:02:38.366 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:02:38.367 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:02:38.367 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:02:38.367 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:02:38.367 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T04:02:38.368 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:02:38.368 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: raise self._exception 2021-07-31T04:02:38.368 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:02:38.368 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:02:38.368 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:02:38.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T04:02:38.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:02:38.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: raise 2021-07-31T04:02:38.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:02:38.370 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T04:02:38.370 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:02:38.370 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:02:38.370 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:37 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:02:39.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:38 smithi154 conmon[34553]: cluster 2021-07-31T04:02:37.730246+0000 mgr.x (mgr.24542) 5808 : cluster [DBG] pgmap v6070: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:02:39.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:38 smithi154 conmon[34553]: 2021-07-31T04:02:39.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:38 smithi045 conmon[33655]: cluster 2021-07-31T04:02:37.730246 2021-07-31T04:02:39.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:38 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 5808 : cluster [DBG] pgmap v6070: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:02:39.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:38 smithi045 conmon[39718]: cluster 2021-07-31T04:02:37.730246+0000 mgr.x (mgr 2021-07-31T04:02:39.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:38 smithi045 conmon[39718]: .24542) 5808 : cluster [DBG] pgmap v6070: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:02:41.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:40 smithi154 conmon[34553]: cluster 2021-07-31T04:02:39.730831+0000 mgr.x (mgr.24542) 5809 : cluster [DBG] 2021-07-31T04:02:41.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:40 smithi154 conmon[34553]: pgmap v6071: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:02:41.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:40 smithi045 conmon[33655]: cluster 2021-07-31T04:02:39.730831+0000 mgr.x (mgr.24542) 5809 2021-07-31T04:02:41.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:40 smithi045 conmon[33655]: : cluster [DBG] pgmap v6071: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:02:41.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:40 smithi045 conmon[39718]: cluster 2021-07-31T04:02:39.730831+0000 mgr.x 2021-07-31T04:02:41.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:40 smithi045 conmon[39718]: (mgr.24542) 5809 : cluster [DBG] pgmap v6071: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:02:43.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:42 smithi154 conmon[34553]: cluster 2021-07-31T04:02:41.731361+0000 mgr.x (mgr.24542) 5810 2021-07-31T04:02:43.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:42 smithi154 conmon[34553]: : cluster [DBG] pgmap v6072: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:02:43.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:42 smithi045 conmon[39718]: cluster 2021-07-31T04:02:41.731361+0000 mgr.x (mgr.24542) 5810 : cluster [DBG] pgmap v6072: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:02:43.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:42 smithi045 conmon[39718]: 2021-07-31T04:02:43.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:42 smithi045 conmon[33655]: cluster 2021-07-31T04:02:41.731361+0000 mgr.x (mgr.24542) 5810 : 2021-07-31T04:02:43.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:42 smithi045 conmon[33655]: cluster [DBG] pgmap v6072: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:02:45.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:44 smithi154 conmon[34553]: cluster 2021-07-31T04:02:43.731865+0000 mgr.x (mgr.24542) 5811 : cluster [DBG] 2021-07-31T04:02:45.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:44 smithi154 conmon[34553]: pgmap v6073: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:02:45.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:44 smithi045 conmon[39718]: cluster 2021-07-31T04:02:43.731865+0000 mgr.x (mgr.24542) 5811 : cluster [DBG] pgmap v6073: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:02:45.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:44 smithi045 conmon[33655]: cluster 2021-07-31T04:02:43.731865+0000 mgr.x (mgr.24542) 5811 : cluster 2021-07-31T04:02:45.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:44 smithi045 conmon[33655]: [DBG] pgmap v6073: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:02:47.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:46 smithi154 conmon[34553]: cluster 2021-07-31T04:02:45.732319+0000 mgr.x (mgr.24542) 5812 2021-07-31T04:02:47.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:46 smithi154 conmon[34553]: : cluster [DBG] pgmap v6074: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:02:47.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:46 smithi045 conmon[39718]: cluster 2021-07-31T04:02:45.732319+0000 mgr.x (mgr. 2021-07-31T04:02:47.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:46 smithi045 conmon[39718]: 24542) 5812 : cluster [DBG] pgmap v6074: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:02:47.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:46 smithi045 conmon[33655]: cluster 2021-07-31T04:02:45.732319+0000 mgr.x (mgr.24542) 5812 : cluster [DBG] pgmap v6074: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:02:47.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:46 smithi045 conmon[33655]: 2021-07-31T04:02:49.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:48 smithi154 conmon[34553]: cluster 2021-07-31T04:02:47.732712+0000 mgr.x (mgr.24542) 5813 : cluster [DBG] pgmap v6075: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:02:49.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:48 smithi045 conmon[39718]: cluster 2021-07-31T04:02:47.732712+0000 mgr.x (mgr.24542) 5813 : cluster [DBG] pgmap v6075: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:02:49.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:48 smithi045 conmon[33655]: cluster 2021-07-31T04:02:47.732712+0000 mgr.x (mgr.24542 2021-07-31T04:02:49.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:48 smithi045 conmon[33655]: ) 5813 : cluster [DBG] pgmap v6075: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:02:51.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:50 smithi154 conmon[34553]: cluster 2021-07-31T04:02:49.733236+0000 mgr.x (mgr.24542) 5814 : cluster [DBG] pgmap v6076: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:02:51.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:50 smithi045 conmon[33655]: cluster 2021-07-31T04:02:49.733236+0000 mgr.x (mgr.24542) 2021-07-31T04:02:51.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:50 smithi045 conmon[33655]: 5814 : cluster [DBG] pgmap v6076: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:02:51.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:50 smithi045 conmon[39718]: cluster 2021-07-31T04:02:49.733236+0000 mgr.x (mgr.24542) 5814 : cluster [DBG] pgmap v6076: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:02:53.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:52 smithi154 conmon[34553]: cluster 2021-07-31T04:02:51.733800+0000 mgr.x (mgr.24542) 5815 : cluster [DBG] pgmap v6077: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:02:53.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:52 smithi045 conmon[39718]: cluster 2021-07-31T04:02:51.733800+0000 mgr.x 2021-07-31T04:02:53.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:52 smithi045 conmon[39718]: (mgr.24542) 5815 : cluster [DBG] pgmap v6077: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:02:53.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:52 smithi045 conmon[33655]: cluster 2021-07-31T04:02:51.733800+0000 mgr.x (mgr 2021-07-31T04:02:53.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:52 smithi045 conmon[33655]: .24542) 5815 : cluster [DBG] pgmap v6077: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:02:55.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:54 smithi154 conmon[34553]: cluster 2021-07-31T04:02:53.734166+0000 mgr.x (mgr.24542) 5816 : cluster [DBG] pgmap v6078: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:02:55.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:54 smithi045 conmon[33655]: cluster 2021-07-31T04:02:53.734166+0000 mgr.x (mgr.24542) 5816 : cluster [DBG] pgmap v6078: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:02:55.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:54 smithi045 conmon[39718]: cluster 2021-07-31T04:02:53.734166+0000 mgr.x (mgr. 2021-07-31T04:02:55.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:54 smithi045 conmon[39718]: 24542) 5816 : cluster [DBG] pgmap v6078: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:02:57.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:56 smithi154 conmon[34553]: cluster 2021-07-31T04:02:55.734670+0000 mgr.x (mgr.24542) 5817 : cluster [DBG] pgmap v6079: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:02:57.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:56 smithi045 conmon[39718]: cluster 2021-07-31T04:02:55.734670+0000 mgr.x (mgr.24542 2021-07-31T04:02:57.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:56 smithi045 conmon[39718]: ) 5817 : cluster [DBG] pgmap v6079: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:02:57.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:56 smithi045 conmon[33655]: cluster 2021-07-31T04:02:55.734670+0000 mgr.x (mgr.24542) 5817 : cluster [DBG] 2021-07-31T04:02:57.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:56 smithi045 conmon[33655]: pgmap v6079: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:02:59.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:58 smithi154 conmon[34553]: cluster 2021-07-31T04:02:57.735224+0000 mgr.x (mgr.24542) 5818 : cluster [DBG] pgmap v6080: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:02:59.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:02:58 smithi154 conmon[34553]: 2021-07-31T04:02:59.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:58 smithi045 conmon[33655]: cluster 2021-07-31T04:02:57.735224+0000 mgr.x (mgr.24542 2021-07-31T04:02:59.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:02:58 smithi045 conmon[33655]: ) 5818 : cluster [DBG] pgmap v6080: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:02:59.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:58 smithi045 conmon[39718]: cluster 2021-07-31T04:02:57.735224+0000 mgr.x (mgr.24542 2021-07-31T04:02:59.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:02:58 smithi045 conmon[39718]: ) 5818 : cluster [DBG] pgmap v6080: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:01.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:00 smithi154 conmon[34553]: cluster 2021-07-31T04:02:59.735654+0000 mgr.x (mgr.24542) 5819 : cluster [DBG] pgmap v6081: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:01.274 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:00 smithi045 conmon[39718]: cluster 2021-07-31T04:02:59 2021-07-31T04:03:01.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:00 smithi045 conmon[39718]: .735654+0000 mgr.x (mgr.24542) 5819 : cluster [DBG] pgmap v6081: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:01.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:00 smithi045 conmon[33655]: cluster 2021-07-31T04:02:59.735654+0000 mgr.x (mgr.24542) 2021-07-31T04:03:01.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:00 smithi045 conmon[33655]: 5819 : cluster [DBG] pgmap v6081: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:01.637 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T04:03:01.914 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T04:03:01.914 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 2h ago 2h - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T04:03:01.915 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 2h ago 2h 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T04:03:01.915 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 2h ago 2h 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T04:03:01.915 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 2h ago 2h - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T04:03:01.915 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T04:03:01.915 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 2h ago 2h 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T04:03:01.916 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T04:03:01.916 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 2h ago 2h - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T04:03:01.916 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 2h ago 2h 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T04:03:01.916 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T04:03:01.916 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T04:03:01.917 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T04:03:01.917 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T04:03:01.917 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 2h ago 2h 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T04:03:01.917 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 2h ago 2h 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T04:03:01.917 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 2h ago 2h 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T04:03:01.918 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 2h ago 2h 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T04:03:01.918 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 2h ago 2h 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T04:03:02.249 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T04:03:02.249 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T04:03:02.249 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T04:03:02.250 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:03:02.250 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T04:03:02.250 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T04:03:02.250 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:03:02.250 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:03:02.251 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T04:03:02.251 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T04:03:02.251 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:03:02.251 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T04:03:02.251 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T04:03:02.252 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T04:03:02.252 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:03:02.252 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T04:03:02.252 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T04:03:03.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:02 smithi154 conmon[34553]: audit 2021-07-31T04:03:01.625973+0000 mgr.x (mgr.24542) 5820 : audit [DBG] from='client.30721 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:03:03.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:02 smithi154 conmon[34553]: cluster 2021-07-31T04:03:01.736120 2021-07-31T04:03:03.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:02 smithi154 conmon[34553]: +0000 mgr.x (mgr.24542) 5821 : cluster [DBG] pgmap v6082: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:03.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:02 smithi154 conmon[34553]: audit 2021-07-31T04:03:01.906797+0000 mgr.x (mgr.24542) 5822 : audit [DBG] from='client.30727 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:03:03.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:02 smithi154 conmon[34553]: audit 2021-07-31T04:03:02.250019+0000 mon.c (mon 2021-07-31T04:03:03.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:02 smithi154 conmon[34553]: .1) 245 : audit [DBG] from='client.? 172.21.15.45:0/4287672265' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:03:03.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:02 smithi045 conmon[39718]: audit 2021-07-31T04:03:01.625973+0000 mgr.x (mgr.24542) 2021-07-31T04:03:03.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:02 smithi045 conmon[39718]: 5820 : audit [DBG] from='client.30721 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:03:03.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:02 smithi045 conmon[39718]: cluster 2021-07-31T04:03:01.736120+0000 mgr.x (mgr 2021-07-31T04:03:03.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:02 smithi045 conmon[39718]: .24542) 5821 : cluster [DBG] pgmap v6082: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:03.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:02 smithi045 conmon[39718]: audit 2021-07-31T04: 2021-07-31T04:03:03.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:02 smithi045 conmon[39718]: 03:01.906797+0000 mgr.x (mgr.24542) 5822 : audit [DBG] from='client.30727 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:03:03.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:02 smithi045 conmon[39718]: audit 2021-07-31T04: 2021-07-31T04:03:03.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:02 smithi045 conmon[39718]: 03:02.250019+0000 mon.c (mon.1) 245 : audit [DBG] from='client.? 172.21.15.45:0/4287672265' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:03:03.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:02 smithi045 conmon[33655]: audit 2021-07-31T04:03:01. 2021-07-31T04:03:03.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:02 smithi045 conmon[33655]: 625973+0000 mgr.x (mgr.24542) 5820 : audit [DBG] from='client.30721 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:03:03.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:02 smithi045 conmon[33655]: cluster 2021-07-31T04: 2021-07-31T04:03:03.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:02 smithi045 conmon[33655]: 03:01.736120+0000 mgr.x (mgr.24542) 5821 : cluster [DBG] pgmap v6082: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:03.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:02 smithi045 conmon[33655]: audit 2021- 2021-07-31T04:03:03.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:02 smithi045 conmon[33655]: 07-31T04:03:01.906797+0000 mgr.x (mgr.24542) 2021-07-31T04:03:03.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:02 smithi045 conmon[33655]: 5822 : audit [DBG] from='client.30727 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:03:03.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:02 smithi045 conmon[33655]: audit 2021 2021-07-31T04:03:03.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:02 smithi045 conmon[33655]: -07-31T04:03:02.250019+0000 2021-07-31T04:03:03.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:02 smithi045 conmon[33655]: mon.c (mon.1) 245 : audit [DBG] from='client.? 172.21.15.45:0/4287672265' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:03:05.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:04 smithi154 conmon[34553]: cluster 2021-07-31T04:03:03.736497+0000 mgr.x (mgr.24542) 5823 : cluster [DBG] pgmap v6083: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:05.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:04 smithi045 conmon[39718]: cluster 2021-07-31T04:03:03.736497+0000 mgr.x (mgr 2021-07-31T04:03:05.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:04 smithi045 conmon[39718]: .24542) 5823 : cluster [DBG] pgmap v6083: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:05.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:04 smithi045 conmon[33655]: cluster 2021-07-31T04:03:03.736497+0000 mgr.x (mgr.24542) 5823 2021-07-31T04:03:05.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:04 smithi045 conmon[33655]: : cluster [DBG] pgmap v6083: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:07.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:06 smithi154 conmon[34553]: cluster 2021-07-31T04:03:05.737095+0000 mgr.x (mgr.24542) 5824 : cluster [DBG] pgmap v6084: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:07.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:06 smithi045 conmon[39718]: cluster 2021-07-31T04:03:05.737095+0000 mgr.x (mgr 2021-07-31T04:03:07.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:06 smithi045 conmon[39718]: .24542) 5824 : cluster [DBG] pgmap v6084: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:07.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:06 smithi045 conmon[33655]: cluster 2021-07-31T04:03:05.737095+0000 mgr.x (mgr.24542) 5824 : 2021-07-31T04:03:07.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:06 smithi045 conmon[33655]: cluster [DBG] pgmap v6084: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:09.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:08 smithi154 conmon[34553]: cluster 2021-07-31T04:03:07.737706+0000 mgr.x (mgr.24542) 5825 : cluster [DBG] pgmap v6085: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:09.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:08 smithi045 conmon[39718]: cluster 2021-07-31T04:03:07.737706+0000 mgr.x (mgr. 2021-07-31T04:03:09.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:08 smithi045 conmon[39718]: 24542) 5825 : cluster [DBG] pgmap v6085: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:09.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:08 smithi045 conmon[33655]: cluster 2021-07-31T04:03:07.737706+0000 mgr.x (mgr.24542) 5825 : cluster 2021-07-31T04:03:09.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:08 smithi045 conmon[33655]: [DBG] pgmap v6085: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:11.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:10 smithi154 conmon[34553]: cluster 2021-07-31T04:03:09.738153+0000 mgr.x (mgr.24542) 5826 : cluster [DBG] pgmap v6086: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:11.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:10 smithi045 conmon[39718]: cluster 2021-07-31T04:03:09 2021-07-31T04:03:11.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:10 smithi045 conmon[39718]: .738153+0000 mgr.x (mgr.24542) 5826 : cluster [DBG] pgmap v6086: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:11.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:10 smithi045 conmon[33655]: cluster 2021-07-31T04:03:09.738153+0000 mgr.x (mgr.24542) 5826 : 2021-07-31T04:03:11.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:10 smithi045 conmon[33655]: cluster [DBG] pgmap v6086: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:13.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:12 smithi154 conmon[34553]: cluster 2021-07-31T04:03:11.738701+0000 mgr.x (mgr.24542) 5827 : cluster [DBG] pgmap v6087: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:13.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:12 smithi154 conmon[34553]: 2021-07-31T04:03:13.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:12 smithi045 conmon[39718]: cluster 2021-07-31T04:03:11.738701+0000 mgr.x (mgr 2021-07-31T04:03:13.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:12 smithi045 conmon[39718]: .24542) 5827 : cluster [DBG] pgmap v6087: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:13.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:12 smithi045 conmon[33655]: cluster 2021-07-31T04:03:11 2021-07-31T04:03:13.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:12 smithi045 conmon[33655]: .738701+0000 mgr.x (mgr.24542) 5827 : cluster [DBG] pgmap v6087: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:15.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:14 smithi154 conmon[34553]: cluster 2021-07-31T04:03:13.739237+0000 mgr.x (mgr.24542) 5828 : cluster [DBG] pgmap v6088: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:15.274 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:14 smithi045 conmon[33655]: cluster 2021-07-31T04:03:13.739237+0000 mgr.x (mgr.24542) 5828 : cluster [DBG] pgmap v6088: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:15.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:14 smithi045 conmon[39718]: cluster 2021-07-31T04:03:13.739237+0000 mgr.x (mgr 2021-07-31T04:03:15.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:14 smithi045 conmon[39718]: .24542) 5828 : cluster [DBG] pgmap v6088: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:17.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:16 smithi154 conmon[34553]: cluster 2021-07-31T04:03:15.739717+0000 mgr.x (mgr.24542) 5829 : cluster [DBG] pgmap v6089: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:17.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:16 smithi045 conmon[33655]: cluster 2021-07-31T04:03:15.739717+0000 mgr.x (mgr.24542) 2021-07-31T04:03:17.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:16 smithi045 conmon[33655]: 5829 : cluster [DBG] pgmap v6089: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:17.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:16 smithi045 conmon[39718]: cluster 2021-07-31T04:03:15 2021-07-31T04:03:17.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:16 smithi045 conmon[39718]: .739717+0000 mgr.x (mgr.24542) 5829 : cluster [DBG] pgmap v6089: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:19.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:18 smithi154 conmon[34553]: cluster 2021-07-31T04:03:17.740169+0000 mgr.x (mgr.24542) 5830 : cluster [DBG] pgmap v6090: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:19.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:18 smithi045 conmon[33655]: cluster 2021-07-31T04:03:17.740169+0000 mgr.x 2021-07-31T04:03:19.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:18 smithi045 conmon[33655]: (mgr.24542) 5830 : cluster [DBG] pgmap v6090: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:19.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:18 smithi045 conmon[39718]: cluster 2021-07-31T04:03:17.740169+0000 mgr.x (mgr.24542) 2021-07-31T04:03:19.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:18 smithi045 conmon[39718]: 5830 : cluster [DBG] pgmap v6090: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:21.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:20 smithi154 conmon[34553]: cluster 2021-07-31T04:03:19.740695+0000 mgr.x (mgr.24542) 5831 : cluster [DBG] pgmap v6091: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:21.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:20 smithi154 conmon[34553]: 2021-07-31T04:03:21.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:20 smithi045 conmon[33655]: cluster 2021-07-31T04:03:19.740695+0000 mgr.x (mgr.24542) 5831 : cluster 2021-07-31T04:03:21.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:20 smithi045 conmon[33655]: [DBG] pgmap v6091: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:21.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:20 smithi045 conmon[39718]: cluster 2021-07-31T04:03:19.740695+0000 mgr.x (mgr 2021-07-31T04:03:21.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:20 smithi045 conmon[39718]: .24542) 5831 : cluster [DBG] pgmap v6091: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:23.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:22 smithi154 conmon[34553]: cluster 2021-07-31T04:03:21.741122+0000 mgr.x (mgr.24542) 5832 : cluster [DBG] pgmap v6092: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:23.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:22 smithi045 conmon[39718]: cluster 2021-07-31T04:03:21.741122+0000 mgr.x ( 2021-07-31T04:03:23.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:22 smithi045 conmon[39718]: mgr.24542) 5832 : cluster [DBG] pgmap v6092: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:23.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:22 smithi045 conmon[33655]: cluster 2021-07-31T04:03:21.741122+0000 mgr.x (mgr.24542) 2021-07-31T04:03:23.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:22 smithi045 conmon[33655]: 5832 : cluster [DBG] pgmap v6092: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:25.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:24 smithi154 conmon[34553]: cluster 2021-07-31T04:03:23.741536+0000 mgr.x (mgr.24542) 5833 : cluster [DBG] pgmap v6093: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:25.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:24 smithi045 conmon[39718]: cluster 2021-07-31T04:03:23. 2021-07-31T04:03:25.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:24 smithi045 conmon[39718]: 741536+0000 mgr.x (mgr.24542) 5833 : cluster [DBG] pgmap v6093: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:25.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:24 smithi045 conmon[33655]: cluster 2021-07-31T04:03:23.741536+0000 mgr.x (mgr.24542) 5833 : cluster [DBG] pgmap v6093: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:25.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:24 smithi045 conmon[33655]: 2021-07-31T04:03:27.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:26 smithi154 conmon[34553]: cluster 2021-07-31T04:03:25.742110+0000 mgr.x (mgr.24542) 5834 : cluster [DBG] pgmap v6094: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:27.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:26 smithi045 conmon[33655]: cluster 2021-07-31T04:03:25.742110+0000 mgr.x (mgr.24542) 5834 : 2021-07-31T04:03:27.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:26 smithi045 conmon[33655]: cluster [DBG] pgmap v6094: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:27.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:26 smithi045 conmon[39718]: cluster 2021-07-31T04:03:25.742110+0000 mgr.x (mgr 2021-07-31T04:03:27.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:26 smithi045 conmon[39718]: .24542) 5834 : cluster [DBG] pgmap v6094: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:29.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:28 smithi154 conmon[34553]: cluster 2021-07-31T04:03:27.742524+0000 mgr.x (mgr.24542) 5835 : cluster [DBG] pgmap v6095: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:29.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:28 smithi154 conmon[34553]: audit 2021-07-31T04:03:28.473233+0000 mon.b (mon.2) 755 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:03:29.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:28 smithi154 conmon[34553]: audit 2021-07-31T04:03:28.473626+0000 mon.b (mon.2) 756 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:03:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:28 smithi154 conmon[34553]: audit 2021-07-31T04:03:28.474358 2021-07-31T04:03:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:28 smithi154 conmon[34553]: +0000 mon.a (mon.0) 3413 : audit [INF] 2021-07-31T04:03:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:28 smithi154 conmon[34553]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:03:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:28 smithi154 conmon[34553]: audit 2021-07 2021-07-31T04:03:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:28 smithi154 conmon[34553]: -31T04:03:28.474743 2021-07-31T04:03:29.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:28 smithi154 conmon[34553]: +0000 mon.a (mon.0) 3414 2021-07-31T04:03:29.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:28 smithi154 conmon[34553]: : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:03:29.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:28 smithi045 conmon[33655]: cluster 2021-07-31T04:03:27.742524+0000 mgr.x (mgr.24542) 5835 2021-07-31T04:03:29.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:28 smithi045 conmon[33655]: : cluster [DBG] pgmap v6095: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:29.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:28 smithi045 conmon[33655]: audit 2021-07-31T04:03:28.473233+0000 mon.b (mon.2) 755 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:03:29.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:28 smithi045 conmon[33655]: audit 2021-07-31T04:03:28.473626+0000 mon.b 2021-07-31T04:03:29.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:28 smithi045 conmon[33655]: (mon.2) 756 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:03:29.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:28 smithi045 conmon[33655]: audit 2021-07-31T04: 2021-07-31T04:03:29.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:28 smithi045 conmon[33655]: 03:28.474358+0000 mon.a (mon.0) 3413 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:03:29.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:28 smithi045 conmon[33655]: audit 2021 2021-07-31T04:03:29.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:28 smithi045 conmon[33655]: -07-31T04:03:28.474743 2021-07-31T04:03:29.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:28 smithi045 conmon[33655]: +0000 mon.a (mon.0) 3414 : audit 2021-07-31T04:03:29.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:28 smithi045 conmon[33655]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:03:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:28 smithi045 conmon[39718]: cluster 2021-07-31T04:03:27.742524+0000 mgr.x (mgr.24542 2021-07-31T04:03:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:28 smithi045 conmon[39718]: ) 5835 : cluster [DBG] pgmap v6095: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:28 smithi045 conmon[39718]: audit 2021-07-31T04:03:28 2021-07-31T04:03:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:28 smithi045 conmon[39718]: .473233+0000 mon.b (mon.2) 755 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:03:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:28 smithi045 conmon[39718]: audit 2021 2021-07-31T04:03:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:28 smithi045 conmon[39718]: -07-31T04:03:28.473626+0000 mon.b ( 2021-07-31T04:03:29.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:28 smithi045 conmon[39718]: mon.2) 756 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:03:29.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:28 smithi045 conmon[39718]: audit 2021-07-31T04 2021-07-31T04:03:29.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:28 smithi045 conmon[39718]: :03:28.474358+0000 mon.a (mon.0) 3413 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:03:29.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:28 smithi045 conmon[39718]: audit 2021-07-31T04:03:29.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:28 smithi045 conmon[39718]: 2021-07-31T04:03:28.474743+0000 mon.a (mon.0) 3414 2021-07-31T04:03:29.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:28 smithi045 conmon[39718]: : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:03:31.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:30 smithi154 conmon[34553]: cluster 2021-07-31T04:03:29.743134+0000 mgr.x (mgr.24542) 5836 : cluster [DBG] pgmap v6096: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:31.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:30 smithi045 conmon[39718]: cluster 2021-07-31T04:03:29.743134+0000 mgr.x ( 2021-07-31T04:03:31.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:30 smithi045 conmon[39718]: mgr.24542) 5836 : cluster [DBG] pgmap v6096: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:31.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:30 smithi045 conmon[33655]: cluster 2021-07-31T04:03:29. 2021-07-31T04:03:31.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:30 smithi045 conmon[33655]: 743134+0000 mgr.x (mgr.24542) 5836 : cluster [DBG] pgmap v6096: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:32.566 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T04:03:32.843 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T04:03:32.843 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 2h ago 2h - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T04:03:32.843 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 2h ago 2h 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T04:03:32.844 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 2h ago 2h 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T04:03:32.844 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 2h ago 2h - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T04:03:32.844 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T04:03:32.844 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 2h ago 2h 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T04:03:32.844 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T04:03:32.844 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 2h ago 2h - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T04:03:32.845 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 2h ago 2h 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T04:03:32.845 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T04:03:32.845 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T04:03:32.845 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T04:03:32.845 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T04:03:32.846 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 2h ago 2h 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T04:03:32.846 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 2h ago 2h 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T04:03:32.846 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 2h ago 2h 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T04:03:32.846 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 2h ago 2h 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T04:03:32.846 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 2h ago 2h 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T04:03:33.182 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T04:03:33.182 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T04:03:33.182 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T04:03:33.182 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:03:33.183 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T04:03:33.183 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T04:03:33.183 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:03:33.183 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:03:33.183 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T04:03:33.184 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T04:03:33.184 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:03:33.184 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T04:03:33.184 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T04:03:33.184 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T04:03:33.185 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:03:33.185 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T04:03:33.185 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T04:03:33.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:32 smithi154 conmon[34553]: cluster 2021-07-31T04:03:31.743669+0000 mgr.x (mgr.24542) 5837 : cluster [DBG] pgmap v6097: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:33.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:32 smithi045 conmon[33655]: cluster 2021-07-31T04:03:31. 2021-07-31T04:03:33.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:32 smithi045 conmon[33655]: 743669+0000 mgr.x (mgr.24542) 5837 : cluster [DBG] pgmap v6097: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:33.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:32 smithi045 conmon[39718]: cluster 2021-07-31T 2021-07-31T04:03:33.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:32 smithi045 conmon[39718]: 04:03:31.743669+0000 mgr.x (mgr.24542) 5837 : cluster [DBG] pgmap v6097: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:34.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:33 smithi154 conmon[34553]: audit 2021-07-31T04:03:32.554882+0000 mgr.x (mgr.24542) 5838 : audit [DBG] from='client.30739 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:03:34.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:33 smithi154 conmon[34553]: audit 2021-07-31T04:03:32.835974+0000 mgr.x (mgr.24542) 5839 : audit [DBG] from='client.30745 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:03:34.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:33 smithi154 conmon[34553]: audit 2021-07-31T04:03:33.183048+0000 mon.a (mon.0) 3415 : audit [DBG] from='client.? 172.21.15.45:0/1382223076' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:03:34.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:33 smithi045 conmon[39718]: audit 2021-07-31T04:03:32.554882+0000 mgr.x ( 2021-07-31T04:03:34.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:33 smithi045 conmon[39718]: mgr.24542) 5838 : audit [DBG] from='client.30739 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:03:34.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:33 smithi045 conmon[39718]: audit 2021-07-31T04:03:32.835974+0000 mgr.x (mgr.24542) 5839 : audit [DBG] from='client.30745 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:03:34.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:33 smithi045 conmon[39718]: 2021-07-31T04:03:34.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:33 smithi045 conmon[39718]: audit 2021-07-31T04:03:33.183048+0000 mon.a (mon.0) 3415 2021-07-31T04:03:34.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:33 smithi045 conmon[39718]: : audit [DBG] from='client.? 172.21.15.45:0/1382223076' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:03:34.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:33 smithi045 conmon[33655]: audit 2021-07-31T04:03:32.554882+0000 2021-07-31T04:03:34.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:33 smithi045 conmon[33655]: mgr.x (mgr.24542) 5838 : audit [DBG] from='client.30739 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:03:34.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:33 smithi045 conmon[33655]: audit 2021-07-31T04:03:32.835974+0000 2021-07-31T04:03:34.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:33 smithi045 conmon[33655]: mgr.x (mgr.24542) 5839 : audit [DBG] from='client.30745 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:03:34.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:33 smithi045 conmon[33655]: audit 2021-07-31T04:03:34.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:33 smithi045 conmon[33655]: 2021-07-31T04:03:33. 2021-07-31T04:03:34.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:33 smithi045 conmon[33655]: 183048+0000 mon.a (mon.0) 3415 : audit [DBG] 2021-07-31T04:03:34.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:33 smithi045 conmon[33655]: from='client.? 172.21.15.45:0/1382223076' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:03:35.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:34 smithi154 conmon[34553]: cluster 2021-07-31T04:03:33.744141+0000 mgr.x (mgr.24542) 5840 : cluster [DBG] pgmap v6098: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:35.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:34 smithi045 conmon[39718]: cluster 2021-07-31T04:03:33.744141+0000 mgr.x (mgr.24542) 2021-07-31T04:03:35.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:34 smithi045 conmon[39718]: 5840 : cluster [DBG] pgmap v6098: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:35.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:34 smithi045 conmon[33655]: cluster 2021-07-31T04:03:33.744141+0000 mgr.x (mgr.24542) 5840 : 2021-07-31T04:03:35.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:34 smithi045 conmon[33655]: cluster [DBG] pgmap v6098: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:36.923 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: debug 2021-07-31T04:03:36.673+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:03:36.923 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:03:36.924 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:03:36.924 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: yield 2021-07-31T04:03:36.924 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:03:36.924 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:03:36.924 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:03:36.925 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:03:36.925 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:03:36.925 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:03:36.925 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:03:36.925 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:03:36.926 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:03:36.926 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: task.result() 2021-07-31T04:03:36.926 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:03:36.926 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: debug 2021-07-31T04:03:36.685+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:03:36.927 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:03:36.927 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:03:36.927 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: yield 2021-07-31T04:03:36.927 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:03:36.927 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:03:36.928 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:03:36.928 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:03:36.928 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:03:36.928 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:03:36.928 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:03:36.928 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:03:36.929 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:03:36.929 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: task.result() 2021-07-31T04:03:36.929 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:03:36.929 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: debug 2021-07-31T04:03:36.710+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:03:36.929 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:03:36.930 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:03:36.930 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: yield 2021-07-31T04:03:36.930 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:03:36.930 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:03:36.930 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:03:36.931 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:03:36.931 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:03:36.931 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:03:36.931 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:03:36.931 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:03:36.932 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:03:36.932 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: task.result() 2021-07-31T04:03:36.932 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:03:36.932 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: debug 2021-07-31T04:03:36.710+0000 7fcb93160700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:03:36.933 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:03:36.933 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:03:36.933 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: yield 2021-07-31T04:03:36.933 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:03:36.933 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:03:36.934 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:03:36.934 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:03:36.934 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:03:36.934 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:03:36.934 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:03:36.935 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:03:36.935 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:03:36.935 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: task.result() 2021-07-31T04:03:36.935 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:03:36.935 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: 2021-07-31T04:03:36.936 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T04:03:36.936 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: 2021-07-31T04:03:36.936 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:03:36.936 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:03:36.936 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: return f(*arg) 2021-07-31T04:03:36.937 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:03:36.937 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:03:36.937 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:03:36.937 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:03:36.937 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:03:36.938 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:03:36.938 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:03:36.938 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T04:03:36.938 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:03:36.938 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: raise self._exception 2021-07-31T04:03:36.939 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:03:36.939 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:03:36.939 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:03:36.939 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T04:03:36.940 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:03:36.940 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: raise 2021-07-31T04:03:36.940 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:03:36.940 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T04:03:36.941 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:03:36.941 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:03:36.941 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:03:36.941 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: debug 2021-07-31T04:03:36.719+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:03:36.941 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:03:36.942 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:03:36.942 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: yield 2021-07-31T04:03:36.942 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:03:36.942 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:03:36.943 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:03:36.943 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:03:36.943 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:03:36.943 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:03:36.943 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:03:36.944 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:03:36.944 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:03:36.944 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: task.result() 2021-07-31T04:03:36.944 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:03:36.944 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: debug 2021-07-31T04:03:36.719+0000 7fcb9295f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:03:36.945 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:03:36.945 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:03:36.945 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: yield 2021-07-31T04:03:36.945 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:03:36.945 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:03:36.946 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:03:36.946 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:03:36.946 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:03:36.946 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:03:36.947 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:03:36.947 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:03:36.947 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:03:36.947 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: task.result() 2021-07-31T04:03:36.947 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:03:36.948 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: 2021-07-31T04:03:36.948 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T04:03:36.948 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: 2021-07-31T04:03:36.948 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:03:36.949 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:03:36.949 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: return f(*arg) 2021-07-31T04:03:36.949 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:03:36.949 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:03:36.949 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:03:36.950 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:03:36.950 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:03:36.950 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:03:36.950 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:03:36.950 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T04:03:36.951 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:03:36.951 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: raise self._exception 2021-07-31T04:03:36.951 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:03:36.951 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:03:36.952 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:03:36.952 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T04:03:36.952 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:03:36.952 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: raise 2021-07-31T04:03:36.952 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:03:36.953 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T04:03:36.953 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:03:36.953 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:03:36.953 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:03:37.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[34553]: cluster 2021-07-31T04:03:35.744671+0000 mgr.x (mgr.24542) 5841 : cluster [DBG] pgmap v6099: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:37.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[34553]: audit 2021-07-31T04:03:36.628312+0000 mon.b (mon.2) 757 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T04:03:37.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[34553]: audit 2021-07-31 2021-07-31T04:03:37.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[34553]: T04:03:36.629905+0000 mon.b (mon.2) 758 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T04:03:37.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[34553]: audit 2021 2021-07-31T04:03:37.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[34553]: -07-31T04:03:36.630817+0000 2021-07-31T04:03:37.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[34553]: mon.b (mon.2) 759 : audit [INF] 2021-07-31T04:03:37.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:36 smithi154 conmon[34553]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T04:03:37.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:36 smithi045 conmon[39718]: cluster 2021-07-31T04:03:35.744671+0000 2021-07-31T04:03:37.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:36 smithi045 conmon[39718]: mgr.x (mgr.24542) 5841 : cluster [DBG] pgmap v6099: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:37.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:36 smithi045 conmon[39718]: audit 2021-07-31T04:03:37.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:36 smithi045 conmon[39718]: 2021-07-31T04:03 2021-07-31T04:03:37.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:36 smithi045 conmon[39718]: :36.628312+0000 mon.b (mon.2) 757 2021-07-31T04:03:37.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:36 smithi045 conmon[39718]: : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T04:03:37.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:36 smithi045 conmon[39718]: audit 2021-07 2021-07-31T04:03:37.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:36 smithi045 conmon[39718]: -31T04:03:36.629905+0000 mon.b (mon.2) 2021-07-31T04:03:37.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:36 smithi045 conmon[39718]: 758 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T04:03:37.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:36 smithi045 conmon[39718]: audit 2021 2021-07-31T04:03:37.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:36 smithi045 conmon[39718]: -07-31T04:03:36.630817 2021-07-31T04:03:37.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:36 smithi045 conmon[39718]: +0000 mon.b (mon.2) 759 : audit [INF] 2021-07-31T04:03:37.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:36 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T04:03:37.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:36 smithi045 conmon[33655]: cluster 2021-07-31T04:03:35.744671+0000 mgr.x (mgr.24542) 5841 2021-07-31T04:03:37.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:36 smithi045 conmon[33655]: : cluster [DBG] pgmap v6099: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:37.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:36 smithi045 conmon[33655]: audit 2021-07-31T04 2021-07-31T04:03:37.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:36 smithi045 conmon[33655]: :03:36.628312+0000 mon.b (mon.2) 757 : audit [DBG] 2021-07-31T04:03:37.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:36 smithi045 conmon[33655]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T04:03:37.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:36 smithi045 conmon[33655]: audit 2021-07- 2021-07-31T04:03:37.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:36 smithi045 conmon[33655]: 31T04:03:36.629905+0000 mon.b (mon.2) 758 2021-07-31T04:03:37.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:36 smithi045 conmon[33655]: : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T04:03:37.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:36 smithi045 conmon[33655]: audit 2021-07-31T04:03:36.630817+0000 mon.b (mon.2) 759 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T04:03:38.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: cephadm 2021-07-31T04:03:36.631595+0000 mgr.x (mgr.24542) 5842 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T04:03:38.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: cephadm 2021-07-31T04:03:36.674742+0000 mgr.x (mgr.24542) 5843 : cephadm [ERR] 2021-07-31T04:03:38.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:03:38.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:03:38.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: yield 2021-07-31T04:03:38.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:03:38.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:03:38.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:03:38.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:03:38.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:03:38.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:03:38.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:03:38.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:03:38.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:03:38.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: task.result() 2021-07-31T04:03:38.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:03:38.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: cephadm 2021-07-31T04:03:36.675174+0000 mgr.x (mgr.24542) 5844 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T04:03:38.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: 2021-07-31T04:03:38.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: cephadm 2021-07-31T04:03:36.686136+0000 mgr.x (mgr.24542) 5845 : cephadm [ERR] 2021-07-31T04:03:38.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:03:38.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:03:38.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: yield 2021-07-31T04:03:38.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:03:38.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:03:38.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:03:38.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:03:38.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:03:38.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:03:38.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:03:38.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:03:38.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:03:38.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: task.result() 2021-07-31T04:03:38.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:03:38.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: cephadm 2021-07-31T04:03:36.686915+0000 mgr.x (mgr.24542) 5846 : cephadm 2021-07-31T04:03:38.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T04:03:38.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: cephadm 2021-07-31T04:03:36.711047+0000 mgr.x (mgr.24542) 5847 : cephadm [ERR] 2021-07-31T04:03:38.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:03:38.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:03:38.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: yield 2021-07-31T04:03:38.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:03:38.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:03:38.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:03:38.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:03:38.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:03:38.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:03:38.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:03:38.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:03:38.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:03:38.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: task.result() 2021-07-31T04:03:38.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:03:38.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: cephadm 2021-07-31T04:03:36.711422+0000 mgr.x (mgr.24542) 2021-07-31T04:03:38.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: 5848 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:03:38.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:03:38.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:03:38.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: yield 2021-07-31T04:03:38.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:03:38.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:03:38.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:03:38.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:03:38.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:03:38.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:03:38.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:03:38.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:03:38.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:03:38.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: task.result() 2021-07-31T04:03:38.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:03:38.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: 2021-07-31T04:03:38.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T04:03:38.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: 2021-07-31T04:03:38.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:03:38.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:03:38.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: return f(*arg) 2021-07-31T04:03:38.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:03:38.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:03:38.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:03:38.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:03:38.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:03:38.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:03:38.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:03:38.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T04:03:38.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:03:38.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: raise self._exception 2021-07-31T04:03:38.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:03:38.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:03:38.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:03:38.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T04:03:38.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:03:38.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: raise 2021-07-31T04:03:38.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:03:38.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T04:03:38.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:03:38.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:03:38.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:03:38.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: cephadm 2021-07-31T04:03:36.720011+0000 mgr.x (mgr.24542) 5849 : cephadm [ERR] 2021-07-31T04:03:38.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:03:38.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:03:38.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: yield 2021-07-31T04:03:38.281 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:03:38.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:03:38.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:03:38.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:03:38.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:03:38.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:03:38.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:03:38.283 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:03:38.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:03:38.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: task.result() 2021-07-31T04:03:38.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:03:38.284 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: cephadm 2021-07-31T04:03:36.720355+0000 2021-07-31T04:03:38.285 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: mgr.x (mgr.24542) 5850 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:03:38.285 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:03:38.285 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:03:38.285 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: yield 2021-07-31T04:03:38.286 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:03:38.286 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:03:38.286 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:03:38.286 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:03:38.287 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:03:38.287 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:03:38.287 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:03:38.288 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:03:38.288 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:03:38.288 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: task.result() 2021-07-31T04:03:38.288 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:03:38.289 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: 2021-07-31T04:03:38.289 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T04:03:38.289 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: 2021-07-31T04:03:38.289 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:03:38.290 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:03:38.290 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: return f(*arg) 2021-07-31T04:03:38.290 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:03:38.291 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:03:38.291 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:03:38.291 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:03:38.291 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:03:38.292 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:03:38.292 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:03:38.292 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T04:03:38.292 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:03:38.293 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: raise self._exception 2021-07-31T04:03:38.293 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:03:38.293 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:03:38.293 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:03:38.294 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T04:03:38.294 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:03:38.294 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: raise 2021-07-31T04:03:38.295 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:03:38.295 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T04:03:38.295 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:03:38.295 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:03:38.295 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:37 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:03:38.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: cephadm 2021-07-31T04:03:36.631595+0000 mgr.x (mgr. 2021-07-31T04:03:38.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: 24542) 5842 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T04:03:38.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: cephadm 2021-07-31T04: 2021-07-31T04:03:38.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: 03:36.674742+0000 mgr.x (mgr.24542) 5843 : cephadm [ERR] 2021-07-31T04:03:38.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:03:38.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:03:38.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: yield 2021-07-31T04:03:38.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:03:38.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:03:38.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:03:38.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:03:38.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:03:38.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:03:38.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:03:38.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:03:38.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:03:38.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: task.result() 2021-07-31T04:03:38.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:03:38.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: cephadm 2021-07-31T04:03:36.675174+0000 mgr.x (mgr.24542) 5844 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T04:03:38.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: cephadm 2021-07-31T04:03 2021-07-31T04:03:38.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: :36.686136+0000 mgr.x (mgr.24542) 5845 : cephadm [ERR] 2021-07-31T04:03:38.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:03:38.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:03:38.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: yield 2021-07-31T04:03:38.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:03:38.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:03:38.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:03:38.304 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:03:38.304 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:03:38.304 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:03:38.304 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:03:38.305 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:03:38.305 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:03:38.305 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: task.result() 2021-07-31T04:03:38.305 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:03:38.306 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: cephadm 2021-07-31T04:03:36.686915+0000 mgr.x (mgr.24542) 5846 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T04:03:38.306 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: cephadm 2021-07-31T04:03:36. 2021-07-31T04:03:38.306 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: 711047+0000 mgr.x (mgr.24542) 5847 : cephadm [ERR] 2021-07-31T04:03:38.306 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:03:38.307 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:03:38.307 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: yield 2021-07-31T04:03:38.307 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:03:38.307 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:03:38.308 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:03:38.308 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:03:38.308 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:03:38.308 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:03:38.309 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:03:38.309 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:03:38.309 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:03:38.309 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: task.result() 2021-07-31T04:03:38.310 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:03:38.310 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: cephadm 2021-07-31T04:03:36.711422+0000 mgr.x (mgr.24542) 5848 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:03:38.310 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:03:38.310 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:03:38.311 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: yield 2021-07-31T04:03:38.311 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:03:38.311 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:03:38.312 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:03:38.312 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:03:38.312 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:03:38.312 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:03:38.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:03:38.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:03:38.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:03:38.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: task.result() 2021-07-31T04:03:38.314 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:03:38.314 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: 2021-07-31T04:03:38.314 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T04:03:38.315 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: 2021-07-31T04:03:38.315 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:03:38.315 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:03:38.315 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: return f(*arg) 2021-07-31T04:03:38.316 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:03:38.316 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:03:38.316 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:03:38.316 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:03:38.317 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:03:38.317 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:03:38.317 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:03:38.318 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: cephadm 2021-07-31T04:03:36.631595+0000 mgr.x (mgr.24542) 5842 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T04:03:38.318 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: cephadm 2021-07-31T04:03:36.674742+0000 mgr.x (mgr.24542) 5843 : cephadm [ERR] 2021-07-31T04:03:38.318 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:03:38.319 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:03:38.319 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: yield 2021-07-31T04:03:38.319 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:03:38.319 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:03:38.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:03:38.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:03:38.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:03:38.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:03:38.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:03:38.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:03:38.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:03:38.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: task.result() 2021-07-31T04:03:38.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:03:38.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: cephadm 2021- 2021-07-31T04:03:38.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: 07-31T04:03:36.675174+0000 mgr.x (mgr.24542) 5844 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T04:03:38.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: cephadm 2021 2021-07-31T04:03:38.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: -07-31T04:03:36.686136+0000 2021-07-31T04:03:38.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: mgr.x (mgr.24542) 5845 2021-07-31T04:03:38.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: : cephadm [ERR] 2021-07-31T04:03:38.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:03:38.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:03:38.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: yield 2021-07-31T04:03:38.325 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:03:38.325 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:03:38.325 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:03:38.326 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:03:38.326 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:03:38.326 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:03:38.326 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:03:38.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:03:38.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:03:38.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: task.result() 2021-07-31T04:03:38.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:03:38.328 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: cephadm 2021-07-31T04:03:36.686915+0000 mgr.x (mgr.24542) 5846 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T04:03:38.328 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: 2021-07-31T04:03:38.328 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: cephadm 2021- 2021-07-31T04:03:38.328 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: 07-31T04:03:36 2021-07-31T04:03:38.329 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: .711047+0000 mgr.x (mgr. 2021-07-31T04:03:38.329 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T04:03:38.330 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:03:38.330 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: raise self._exception 2021-07-31T04:03:38.330 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:03:38.330 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:03:38.331 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:03:38.331 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T04:03:38.331 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:03:38.331 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: raise 2021-07-31T04:03:38.332 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:03:38.332 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T04:03:38.332 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:03:38.332 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:03:38.333 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:03:38.333 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: cephadm 2021-07-31T04:03:38.333 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: 2021-07-31T04:03:36.720011+0000 mgr.x (mgr.24542) 5849 : cephadm [ERR] 2021-07-31T04:03:38.333 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:03:38.334 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:03:38.334 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: yield 2021-07-31T04:03:38.334 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:03:38.335 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:03:38.335 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:03:38.335 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:03:38.335 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:03:38.335 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:03:38.336 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:03:38.336 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:03:38.336 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:03:38.337 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: task.result() 2021-07-31T04:03:38.337 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:03:38.337 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: cephadm 2021-07-31T04:03:36.720355+0000 mgr.x (mgr.24542) 5850 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:03:38.337 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:03:38.337 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:03:38.338 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: yield 2021-07-31T04:03:38.338 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:03:38.338 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:03:38.339 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:03:38.339 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:03:38.339 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:03:38.340 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: 24542) 5847 : cephadm [ERR] 2021-07-31T04:03:38.340 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:03:38.340 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:03:38.340 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: yield 2021-07-31T04:03:38.341 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:03:38.341 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:03:38.341 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:03:38.341 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:03:38.342 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:03:38.342 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:03:38.342 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:03:38.342 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:03:38.343 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:03:38.343 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: task.result() 2021-07-31T04:03:38.343 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:03:38.343 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: 2021-07-31T04:03:38.344 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: cephadm 2021-07-31T04:03:36.711422+0000 2021-07-31T04:03:38.344 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: mgr.x (mgr.24542) 5848 : cephadm 2021-07-31T04:03:38.344 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:03:38.344 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:03:38.345 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:03:38.345 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: yield 2021-07-31T04:03:38.345 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:03:38.345 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:03:38.346 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:03:38.346 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:03:38.346 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:03:38.347 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:03:38.347 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:03:38.347 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:03:38.347 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:03:38.348 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: task.result() 2021-07-31T04:03:38.348 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:03:38.348 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: 2021-07-31T04:03:38.348 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T04:03:38.349 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: 2021-07-31T04:03:38.349 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:03:38.349 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:03:38.349 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: return f(*arg) 2021-07-31T04:03:38.350 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:03:38.350 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:03:38.350 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:03:38.350 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:03:38.351 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:03:38.351 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:03:38.351 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:03:38.352 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:03:38.352 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: task.result() 2021-07-31T04:03:38.352 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:03:38.352 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: 2021-07-31T04:03:38.352 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T04:03:38.353 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: 2021-07-31T04:03:38.353 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:03:38.353 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:03:38.353 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: return f(*arg) 2021-07-31T04:03:38.354 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:03:38.354 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:03:38.354 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:03:38.354 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:03:38.355 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:03:38.355 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:03:38.355 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:03:38.355 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T04:03:38.355 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:03:38.356 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: raise self._exception 2021-07-31T04:03:38.356 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:03:38.356 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:03:38.356 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:03:38.356 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T04:03:38.357 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:03:38.357 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: raise 2021-07-31T04:03:38.357 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:03:38.357 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T04:03:38.358 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:03:38.358 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:03:38.358 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:03:38.359 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:03:38.359 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:03:38.359 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:03:38.359 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T04:03:38.360 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:03:38.360 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: raise self._exception 2021-07-31T04:03:38.360 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:03:38.360 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:03:38.361 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:03:38.361 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T04:03:38.361 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:03:38.361 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: raise 2021-07-31T04:03:38.361 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:03:38.362 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T04:03:38.362 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:03:38.362 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:03:38.362 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:03:38.363 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: cephadm 2021-07-31T04:03:36.720011+0000 mgr.x (mgr.24542) 5849 : cephadm [ERR] 2021-07-31T04:03:38.363 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:03:38.363 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:03:38.363 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: yield 2021-07-31T04:03:38.364 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:03:38.364 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:03:38.364 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:03:38.364 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:03:38.364 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:03:38.365 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:03:38.365 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:03:38.365 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:03:38.365 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:03:38.366 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: task.result() 2021-07-31T04:03:38.366 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:03:38.366 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: cephadm 2021-07-31T04:03:36.720355+0000 mgr.x (mgr.24542) 5850 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:03:38.366 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:03:38.366 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:03:38.367 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: yield 2021-07-31T04:03:38.367 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:03:38.367 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:03:38.367 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:03:38.368 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:03:38.368 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:03:38.368 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:03:38.368 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:03:38.368 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:03:38.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:03:38.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: task.result() 2021-07-31T04:03:38.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:03:38.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: 2021-07-31T04:03:38.370 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T04:03:38.370 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: 2021-07-31T04:03:38.370 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:03:38.370 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:03:38.370 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: return f(*arg) 2021-07-31T04:03:38.371 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:03:38.371 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:03:38.371 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:03:38.371 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:03:38.372 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:03:38.372 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:03:38.372 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:03:38.372 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T04:03:38.373 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:03:38.373 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: raise self._exception 2021-07-31T04:03:38.373 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:03:38.373 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:03:38.373 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:03:38.374 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T04:03:38.374 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:03:38.374 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: raise 2021-07-31T04:03:38.374 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:03:38.375 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T04:03:38.375 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:03:38.375 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:03:38.375 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:37 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:03:39.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:38 smithi154 conmon[34553]: cluster 2021-07-31T04:03:37.745145+0000 mgr.x (mgr.24542) 2021-07-31T04:03:39.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:38 smithi154 conmon[34553]: 5851 : cluster [DBG] pgmap v6100: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:39.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:38 smithi045 conmon[39718]: cluster 2021-07-31T04:03:37.745145+0000 mgr.x (mgr 2021-07-31T04:03:39.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:38 smithi045 conmon[39718]: .24542) 5851 : cluster [DBG] pgmap v6100: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:39.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:38 smithi045 conmon[33655]: cluster 2021-07-31T04:03:37.745145+0000 mgr.x (mgr.24542) 5851 : cluster [DBG] 2021-07-31T04:03:39.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:38 smithi045 conmon[33655]: pgmap v6100: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:41.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:40 smithi154 conmon[34553]: cluster 2021-07-31T04:03:39.745644+0000 mgr.x (mgr.24542) 5852 : cluster 2021-07-31T04:03:41.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:40 smithi154 conmon[34553]: [DBG] pgmap v6101: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:41.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:40 smithi045 conmon[33655]: cluster 2021-07-31T04:03:39.745644+0000 mgr.x ( 2021-07-31T04:03:41.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:40 smithi045 conmon[33655]: mgr.24542) 5852 : cluster [DBG] pgmap v6101: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:41.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:40 smithi045 conmon[39718]: cluster 2021-07-31T04:03:39.745644+0000 mgr.x ( 2021-07-31T04:03:41.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:40 smithi045 conmon[39718]: mgr.24542) 5852 : cluster [DBG] pgmap v6101: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:43.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:42 smithi154 conmon[34553]: cluster 2021-07-31T04:03:41.746163+0000 mgr.x (mgr.24542) 5853 2021-07-31T04:03:43.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:42 smithi154 conmon[34553]: : cluster [DBG] pgmap v6102: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:43.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:42 smithi045 conmon[39718]: cluster 2021-07-31T04:03:41.746163+0000 2021-07-31T04:03:43.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:42 smithi045 conmon[39718]: mgr.x (mgr.24542) 5853 : cluster [DBG] pgmap v6102: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:43.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:42 smithi045 conmon[33655]: cluster 2021-07-31T04:03:41 2021-07-31T04:03:43.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:42 smithi045 conmon[33655]: .746163+0000 mgr.x (mgr.24542) 5853 : cluster [DBG] pgmap v6102: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:45.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:44 smithi154 conmon[34553]: cluster 2021-07-31T04:03:43.746566+0000 mgr.x (mgr.24542) 5854 : cluster [DBG] pgmap v6103: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:45.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:44 smithi154 conmon[34553]: 2021-07-31T04:03:45.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:44 smithi045 conmon[33655]: cluster 2021-07-31T04:03:43.746566+0000 mgr.x (mgr.24542) 5854 : cluster 2021-07-31T04:03:45.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:44 smithi045 conmon[33655]: [DBG] pgmap v6103: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:45.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:44 smithi045 conmon[39718]: cluster 2021-07-31T04:03:43.746566+0000 mgr.x (mgr. 2021-07-31T04:03:45.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:44 smithi045 conmon[39718]: 24542) 5854 : cluster [DBG] pgmap v6103: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:47.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:46 smithi154 conmon[34553]: cluster 2021-07-31T04:03:45.747140+0000 mgr.x (mgr.24542) 5855 : cluster 2021-07-31T04:03:47.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:46 smithi154 conmon[34553]: [DBG] pgmap v6104: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:47.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:46 smithi045 conmon[33655]: cluster 2021-07-31T04:03:45.747140+0000 mgr.x (mgr 2021-07-31T04:03:47.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:46 smithi045 conmon[33655]: .24542) 5855 : cluster [DBG] pgmap v6104: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:47.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:46 smithi045 conmon[39718]: cluster 2021-07-31T04:03:45.747140+0000 mgr.x (mgr.24542) 2021-07-31T04:03:47.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:46 smithi045 conmon[39718]: 5855 : cluster [DBG] pgmap v6104: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:49.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:48 smithi154 conmon[34553]: cluster 2021-07-31T04:03:47.747521+0000 mgr.x (mgr.24542) 5856 : cluster [DBG] 2021-07-31T04:03:49.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:48 smithi154 conmon[34553]: pgmap v6105: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:49.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:48 smithi045 conmon[33655]: cluster 2021-07-31T04:03:47.747521+0000 mgr.x (mgr.24542) 5856 : cluster 2021-07-31T04:03:49.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:48 smithi045 conmon[33655]: [DBG] pgmap v6105: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:49.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:48 smithi045 conmon[39718]: cluster 2021-07-31T04:03:47.747521+0000 mgr.x 2021-07-31T04:03:49.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:48 smithi045 conmon[39718]: (mgr.24542) 5856 : cluster [DBG] pgmap v6105: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:51.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:50 smithi154 conmon[34553]: cluster 2021-07-31T04:03:49.747958+0000 mgr.x (mgr.24542) 5857 : cluster [DBG] pgmap v6106: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:51.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:50 smithi154 conmon[34553]: 2021-07-31T04:03:51.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:50 smithi045 conmon[39718]: cluster 2021-07-31T04:03:49.747958+0000 mgr.x ( 2021-07-31T04:03:51.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:50 smithi045 conmon[39718]: mgr.24542) 5857 : cluster [DBG] pgmap v6106: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:51.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:50 smithi045 conmon[33655]: cluster 2021-07-31T04:03:49.747958+0000 mgr.x (mgr.24542) 5857 : cluster [DBG] pgmap v6106: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:53.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:52 smithi154 conmon[34553]: cluster 2021-07-31T04:03:51.748376+0000 mgr.x (mgr.24542) 5858 : cluster [DBG] 2021-07-31T04:03:53.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:52 smithi154 conmon[34553]: pgmap v6107: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:53.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:52 smithi045 conmon[39718]: cluster 2021-07-31T04:03:51.748376+0000 mgr.x ( 2021-07-31T04:03:53.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:52 smithi045 conmon[39718]: mgr.24542) 5858 : cluster [DBG] pgmap v6107: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:53.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:52 smithi045 conmon[33655]: cluster 2021-07-31T04:03:51.748376+0000 mgr.x (mgr.24542) 5858 : cluster 2021-07-31T04:03:53.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:52 smithi045 conmon[33655]: [DBG] pgmap v6107: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:55.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:54 smithi154 conmon[34553]: cluster 2021-07-31T04:03:53.748740+0000 mgr.x (mgr.24542) 5859 : cluster [DBG] pgmap v6108: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:55.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:54 smithi154 conmon[34553]: 2021-07-31T04:03:55.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:54 smithi045 conmon[39718]: cluster 2021-07-31T04:03:53.748740+0000 mgr.x 2021-07-31T04:03:55.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:54 smithi045 conmon[39718]: (mgr.24542) 5859 : cluster [DBG] pgmap v6108: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:55.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:54 smithi045 conmon[33655]: cluster 2021-07-31T04:03:53.748740+0000 mgr.x (mgr.24542) 2021-07-31T04:03:55.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:54 smithi045 conmon[33655]: 5859 : cluster [DBG] pgmap v6108: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:57.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:56 smithi154 conmon[34553]: cluster 2021-07-31T04:03:55.749195+0000 mgr.x (mgr.24542) 5860 : cluster [DBG] pgmap v6109: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:57.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:56 smithi045 conmon[33655]: cluster 2021-07-31T04:03:55.749195+0000 mgr.x (mgr.24542) 5860 : cluster 2021-07-31T04:03:57.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:56 smithi045 conmon[33655]: [DBG] pgmap v6109: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:57.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:56 smithi045 conmon[39718]: cluster 2021-07-31T04:03:55.749195+0000 mgr.x (mgr.24542) 5860 2021-07-31T04:03:57.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:56 smithi045 conmon[39718]: : cluster [DBG] pgmap v6109: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:59.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:03:58 smithi154 conmon[34553]: cluster 2021-07-31T04:03:57.749724+0000 mgr.x (mgr.24542) 5861 : cluster [DBG] pgmap v6110: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:59.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:58 smithi045 conmon[33655]: cluster 2021-07-31T04:03:57.749724+0000 mgr.x (mgr. 2021-07-31T04:03:59.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:03:58 smithi045 conmon[33655]: 24542) 5861 : cluster [DBG] pgmap v6110: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:03:59.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:58 smithi045 conmon[39718]: cluster 2021-07-31T04:03: 2021-07-31T04:03:59.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:03:58 smithi045 conmon[39718]: 57.749724+0000 mgr.x (mgr.24542) 5861 : cluster [DBG] pgmap v6110: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:01.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:00 smithi154 conmon[34553]: cluster 2021-07-31T04:03:59.750363+0000 mgr.x (mgr.24542) 5862 : cluster [DBG] pgmap v6111: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:01.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:00 smithi045 conmon[33655]: cluster 2021-07-31T04:03:59.750363+0000 mgr.x ( 2021-07-31T04:04:01.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:00 smithi045 conmon[33655]: mgr.24542) 5862 : cluster [DBG] pgmap v6111: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:01.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:00 smithi045 conmon[39718]: cluster 2021-07-31T04:03:59.750363+0000 mgr.x (mgr.24542) 5862 : cluster [DBG] 2021-07-31T04:04:01.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:00 smithi045 conmon[39718]: pgmap v6111: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:03.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:02 smithi154 conmon[34553]: cluster 2021-07-31T04:04:01.750837+0000 mgr.x (mgr.24542) 5863 : cluster [DBG] pgmap v6112: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:03.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:02 smithi045 conmon[33655]: cluster 2021-07-31T04:04:01.750837+0000 mgr.x (mgr.24542) 5863 : cluster [DBG] pgmap v6112: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:03.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:02 smithi045 conmon[39718]: cluster 2021-07-31T04:04:01.750837+0000 mgr.x (mgr. 2021-07-31T04:04:03.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:02 smithi045 conmon[39718]: 24542) 5863 : cluster [DBG] pgmap v6112: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:03.493 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T04:04:03.772 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T04:04:03.772 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 2h ago 2h - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T04:04:03.773 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 2h ago 2h 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T04:04:03.773 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 2h ago 2h 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T04:04:03.773 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 2h ago 2h - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T04:04:03.773 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T04:04:03.774 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 2h ago 2h 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T04:04:03.774 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T04:04:03.774 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 2h ago 2h - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T04:04:03.775 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 2h ago 2h 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T04:04:03.775 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T04:04:03.775 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T04:04:03.776 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T04:04:03.776 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T04:04:03.776 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 2h ago 2h 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T04:04:03.777 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 2h ago 2h 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T04:04:03.777 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 2h ago 2h 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T04:04:03.777 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 2h ago 2h 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T04:04:03.777 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 2h ago 2h 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T04:04:04.111 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T04:04:04.111 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T04:04:04.111 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T04:04:04.112 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:04:04.112 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T04:04:04.112 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T04:04:04.113 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:04:04.113 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:04:04.113 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T04:04:04.113 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T04:04:04.114 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:04:04.114 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T04:04:04.114 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T04:04:04.115 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T04:04:04.115 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:04:04.115 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T04:04:04.116 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T04:04:05.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:04 smithi154 conmon[34553]: audit 2021-07-31T04:04:03.481540+0000 mgr.x (mgr.24542) 5864 : audit [DBG] from='client.20820 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:04:05.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:04 smithi154 conmon[34553]: cluster 2021-07-31T04:04:03 2021-07-31T04:04:05.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:04 smithi154 conmon[34553]: .751310+0000 mgr.x (mgr.24542) 5865 : cluster [DBG] pgmap v6113: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:05.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:04 smithi154 conmon[34553]: audit 2021-07-31T04:04:03.765021+0000 mgr.x (mgr.24542) 5866 : audit [DBG] from='client.30763 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:04:05.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:04 smithi154 conmon[34553]: audit 2021-07 2021-07-31T04:04:05.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:04 smithi154 conmon[34553]: -31T04:04:04.111860+0000 mon.a (mon.0) 3416 : audit [DBG] 2021-07-31T04:04:05.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:04 smithi154 conmon[34553]: from='client.? 172.21.15.45:0/4244862216' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:04:05.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:04 smithi045 conmon[33655]: audit 2021-07-31T04: 2021-07-31T04:04:05.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:04 smithi045 conmon[33655]: 04:03.481540+0000 mgr.x (mgr.24542) 5864 : audit [DBG] from='client.20820 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:04:05.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:04 smithi045 conmon[33655]: cluster 2021- 2021-07-31T04:04:05.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:04 smithi045 conmon[33655]: 07-31T04:04:03.751310+0000 mgr.x (mgr.24542 2021-07-31T04:04:05.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:04 smithi045 conmon[33655]: ) 5865 : cluster [DBG] pgmap v6113: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:05.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:04 smithi045 conmon[33655]: audit 2021-07-31T04:04:03. 2021-07-31T04:04:05.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:04 smithi045 conmon[33655]: 765021+0000 mgr.x (mgr.24542) 5866 : audit [DBG] from='client.30763 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:04:05.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:04 smithi045 conmon[33655]: audit 2021-07-31T04:04:05.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:04 smithi045 conmon[33655]: 2021-07-31T04:04:04.111860+0000 mon.a (mon.0) 3416 : audit [DBG] from='client.? 172.21.15.45:0/4244862216' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:04:05.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:04 smithi045 conmon[39718]: audit 2021-07-31T04:04:03. 2021-07-31T04:04:05.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:04 smithi045 conmon[39718]: 481540+0000 mgr.x (mgr.24542) 5864 : audit [DBG] from='client.20820 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:04:05.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:04 smithi045 conmon[39718]: cluster 2021 2021-07-31T04:04:05.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:04 smithi045 conmon[39718]: -07-31T04:04:03.751310+0000 mgr.x (mgr.24542) 2021-07-31T04:04:05.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:04 smithi045 conmon[39718]: 5865 : cluster [DBG] pgmap v6113: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:05.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:04 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T04:04:05.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:04 smithi045 conmon[39718]: 04:04:03.765021+0000 mgr.x ( 2021-07-31T04:04:05.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:04 smithi045 conmon[39718]: mgr.24542) 5866 : audit [DBG] from='client.30763 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:04:05.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:04 smithi045 conmon[39718]: audit 2021- 2021-07-31T04:04:05.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:04 smithi045 conmon[39718]: 07-31T04:04:04.111860 2021-07-31T04:04:05.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:04 smithi045 conmon[39718]: +0000 mon.a (mon. 2021-07-31T04:04:05.282 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:04 smithi045 conmon[39718]: 0) 3416 : audit [DBG] from='client.? 172.21.15.45:0/4244862216' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:04:05.282 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:04 smithi045 conmon[39718]: 2021-07-31T04:04:07.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:06 smithi154 conmon[34553]: cluster 2021-07-31T04:04:05.751727+0000 mgr.x (mgr.24542) 5867 : cluster [DBG] pgmap v6114: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:07.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:06 smithi045 conmon[39718]: cluster 2021-07-31T04:04:05.751727+0000 mgr.x ( 2021-07-31T04:04:07.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:06 smithi045 conmon[39718]: mgr.24542) 5867 : cluster [DBG] pgmap v6114: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:07.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:06 smithi045 conmon[33655]: cluster 2021-07-31T04:04:05.751727+0000 mgr.x (mgr.24542) 5867 : cluster 2021-07-31T04:04:07.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:06 smithi045 conmon[33655]: [DBG] pgmap v6114: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:09.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:08 smithi154 conmon[34553]: cluster 2021-07-31T04:04:07.752192+0000 mgr.x (mgr.24542) 5868 : cluster [DBG] pgmap v6115: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:09.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:08 smithi045 conmon[39718]: cluster 2021-07-31T04:04:07.752192+0000 mgr.x (mgr. 2021-07-31T04:04:09.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:08 smithi045 conmon[39718]: 24542) 5868 : cluster [DBG] pgmap v6115: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:09.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:08 smithi045 conmon[33655]: cluster 2021-07-31T04:04:07.752192+0000 mgr.x ( 2021-07-31T04:04:09.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:08 smithi045 conmon[33655]: mgr.24542) 5868 : cluster [DBG] pgmap v6115: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:11.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:10 smithi154 conmon[34553]: cluster 2021-07-31T04:04:09.752642+0000 mgr.x (mgr.24542) 5869 : cluster [DBG] pgmap v6116: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:11.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:10 smithi045 conmon[39718]: cluster 2021-07-31T04:04:09.752642+0000 mgr.x (mgr. 2021-07-31T04:04:11.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:10 smithi045 conmon[39718]: 24542) 5869 : cluster [DBG] pgmap v6116: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:11.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:10 smithi045 conmon[33655]: cluster 2021-07-31T04:04:09.752642+0000 mgr.x (mgr.24542) 2021-07-31T04:04:11.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:10 smithi045 conmon[33655]: 5869 : cluster [DBG] pgmap v6116: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:13.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:12 smithi154 conmon[34553]: cluster 2021-07-31T04:04:11.753124+0000 mgr.x (mgr.24542) 5870 : cluster [DBG] pgmap v6117: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:13.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:12 smithi045 conmon[39718]: cluster 2021-07-31T04:04:11.753124+0000 mgr.x (mgr.24542) 5870 : cluster [DBG] pgmap v6117: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:13.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:12 smithi045 conmon[33655]: cluster 2021-07-31T04:04:11.753124+0000 2021-07-31T04:04:13.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:12 smithi045 conmon[33655]: mgr.x (mgr.24542) 5870 : cluster [DBG] pgmap v6117: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:15.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:15 smithi154 conmon[34553]: cluster 2021-07-31T04:04:13.753716+0000 mgr.x (mgr.24542) 5871 : 2021-07-31T04:04:15.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:15 smithi154 conmon[34553]: cluster [DBG] pgmap v6118: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:15.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:14 smithi045 conmon[33655]: cluster 2021-07-31T04:04:13.753716+0000 mgr.x (mgr 2021-07-31T04:04:15.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:14 smithi045 conmon[33655]: .24542) 5871 : cluster [DBG] pgmap v6118: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:15.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:15 smithi045 conmon[39718]: cluster 2021-07-31T04:04:13.753716+0000 mgr.x (mgr.24542) 5871 : cluster 2021-07-31T04:04:15.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:15 smithi045 conmon[39718]: [DBG] pgmap v6118: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:17.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:17 smithi045 conmon[33655]: cluster 2021-07-31T04:04:15.754338+0000 mgr.x (mgr.24542) 5872 : cluster [DBG] pgmap v6119: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:17.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:17 smithi045 conmon[39718]: cluster 2021-07-31T04:04:15.754338+0000 mgr.x (mgr.24542) 5872 : cluster [DBG] pgmap v6119: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:17.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:17 smithi154 conmon[34553]: cluster 2021-07-31T04:04:15.754338+0000 mgr.x (mgr.24542) 5872 : cluster [DBG] pgmap v6119: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:19.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:19 smithi045 conmon[39718]: cluster 2021-07-31T04:04:17.754936+0000 mgr.x (mgr.24542) 5873 : cluster [DBG] pgmap v6120: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:19.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:19 smithi045 conmon[33655]: cluster 2021-07-31T04:04:17.754936+0000 mgr.x (mgr.24542) 5873 : cluster [DBG] pgmap v6120: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:19.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:19 smithi154 conmon[34553]: cluster 2021-07-31T04:04:17.754936+0000 mgr.x (mgr.24542) 5873 : cluster [DBG] pgmap v6120: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:21.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:21 smithi045 conmon[33655]: cluster 2021-07-31T04:04:19.755604+0000 mgr.x ( 2021-07-31T04:04:21.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:21 smithi045 conmon[33655]: mgr.24542) 5874 : cluster [DBG] pgmap v6121: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:21.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:21 smithi045 conmon[39718]: cluster 2021-07-31T04:04:19.755604+0000 mgr.x ( 2021-07-31T04:04:21.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:21 smithi045 conmon[39718]: mgr.24542) 5874 : cluster [DBG] pgmap v6121: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:21.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:21 smithi154 conmon[34553]: cluster 2021-07-31T04:04:19.755604+0000 mgr.x (mgr.24542) 5874 : cluster [DBG] pgmap v6121: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:23.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:23 smithi045 conmon[33655]: cluster 2021-07-31T04:04:21.756084+0000 mgr.x (mgr.24542) 5875 2021-07-31T04:04:23.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:23 smithi045 conmon[33655]: : cluster [DBG] pgmap v6122: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:23.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:23 smithi045 conmon[39718]: cluster 2021-07-31T04:04:21.756084+0000 mgr.x 2021-07-31T04:04:23.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:23 smithi045 conmon[39718]: (mgr.24542) 5875 : cluster [DBG] pgmap v6122: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:23.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:23 smithi154 conmon[34553]: cluster 2021-07-31T04:04:21.756084+0000 mgr.x (mgr.24542) 5875 : cluster [DBG] 2021-07-31T04:04:23.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:23 smithi154 conmon[34553]: pgmap v6122: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:25.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:25 smithi045 conmon[39718]: cluster 2021-07-31T04:04:23.756586+0000 mgr.x ( 2021-07-31T04:04:25.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:25 smithi045 conmon[39718]: mgr.24542) 5876 : cluster [DBG] pgmap v6123: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:25.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:25 smithi045 conmon[33655]: cluster 2021-07-31T04:04:23.756586 2021-07-31T04:04:25.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:25 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 5876 : cluster [DBG] pgmap v6123: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:25.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:25 smithi154 conmon[34553]: cluster 2021-07-31T04:04:23.756586+0000 mgr.x (mgr.24542) 5876 : cluster [DBG] pgmap v6123: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:27.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:27 smithi045 conmon[33655]: cluster 2021-07-31T04:04:25.757197+0000 mgr.x (mgr. 2021-07-31T04:04:27.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:27 smithi045 conmon[33655]: 24542) 5877 : cluster [DBG] pgmap v6124: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:27.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:27 smithi045 conmon[39718]: cluster 2021-07-31T04:04:25.757197+0000 mgr.x (mgr.24542) 5877 : 2021-07-31T04:04:27.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:27 smithi045 conmon[39718]: cluster [DBG] pgmap v6124: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:27.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:27 smithi154 conmon[34553]: cluster 2021-07-31T04:04:25.757197+0000 mgr.x (mgr.24542) 5877 : cluster [DBG] pgmap v6124: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:29.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:29 smithi045 conmon[33655]: cluster 2021-07-31T04:04:27.757658+0000 mgr.x (mgr.24542) 5878 : cluster 2021-07-31T04:04:29.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:29 smithi045 conmon[33655]: [DBG] pgmap v6125: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:29.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:29 smithi045 conmon[33655]: audit 2021-07-31T04:04:28.476871+0000 mon.b (mon.2) 760 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:04:29.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:29 smithi045 conmon[33655]: audit 2021-07-31T04:04:28.477174+0000 mon.b (mon.2) 761 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:04:29.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:29 smithi045 conmon[33655]: audit 2021-07-31T04:04:28.477974+0000 mon.a (mon.0) 3417 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:04:29.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:29 smithi045 conmon[33655]: audit 2021-07-31T04:04: 2021-07-31T04:04:29.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:29 smithi045 conmon[33655]: 28.478399+0000 mon.a (mon.0) 3418 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:04:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:29 smithi045 conmon[39718]: cluster 2021-07-31T04:04:27.757658+0000 mgr.x (mgr.24542 2021-07-31T04:04:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:29 smithi045 conmon[39718]: ) 5878 : cluster [DBG] pgmap v6125: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:29 smithi045 conmon[39718]: audit 2021-07-31T04:04:28.476871+0000 mon.b (mon.2) 760 : audit [INF] 2021-07-31T04:04:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:29 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:04:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:29 smithi045 conmon[39718]: audit 2021-07-31T04:04:28.477174+0000 mon.b (mon 2021-07-31T04:04:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:29 smithi045 conmon[39718]: .2) 761 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:04:29.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:29 smithi045 conmon[39718]: audit 2021-07-31T04:04:29.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:29 smithi045 conmon[39718]: 2021-07-31T04:04:28.477974+0000 mon.a (mon.0) 3417 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:04:29.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:29 smithi045 conmon[39718]: 2021-07-31T04:04:29.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:29 smithi045 conmon[39718]: audit 2021- 2021-07-31T04:04:29.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:29 smithi045 conmon[39718]: 07-31T04:04:28.478399+0000 mon.a (mon.0) 3418 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:04:29.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:29 smithi045 conmon[39718]: 2021-07-31T04:04:29.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:29 smithi154 conmon[34553]: cluster 2021-07-31T04:04:27.757658+0000 mgr.x (mgr.24542) 5878 : cluster [DBG] pgmap v6125: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:29.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:29 smithi154 conmon[34553]: audit 2021-07-31T04:04:28.476871+0000 mon.b (mon.2) 760 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:04:29.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:29 smithi154 conmon[34553]: audit 2021-07-31T04:04:28.477174+0000 mon.b (mon.2) 761 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:04:29.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:29 smithi154 conmon[34553]: audit 2021-07-31T04:04:28.477974+0000 mon.a (mon.0) 3417 : audit 2021-07-31T04:04:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:29 smithi154 conmon[34553]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:04:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:29 smithi154 conmon[34553]: audit 2021-07-31T04:04: 2021-07-31T04:04:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:29 smithi154 conmon[34553]: 28.478399+0000 mon.a (mon.0) 3418 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:04:31.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:31 smithi154 conmon[34553]: cluster 2021-07-31T04:04:29.758274+0000 mgr.x (mgr.24542) 5879 : cluster [DBG] pgmap v6126: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:31.524 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:31 smithi045 conmon[39718]: cluster 2021-07-31T04:04:29.758274+0000 mgr.x (mgr.24542) 5879 : cluster [DBG] pgmap v6126: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:31.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:31 smithi045 conmon[33655]: cluster 2021-07-31T04:04:29.758274+0000 mgr.x (mgr.24542) 5879 : cluster [DBG] pgmap v6126: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:33.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:33 smithi154 conmon[34553]: cluster 2021-07-31T04:04:31.758661+0000 mgr.x (mgr.24542) 5880 : cluster [DBG] pgmap v6127: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:33.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:33 smithi045 conmon[33655]: cluster 2021-07-31T04:04:31.758661+0000 mgr.x (mgr.24542) 5880 : cluster [DBG] pgmap v6127: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:33.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:33 smithi045 conmon[39718]: cluster 2021-07-31T04:04:31. 2021-07-31T04:04:33.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:33 smithi045 conmon[39718]: 758661+0000 mgr.x (mgr.24542) 5880 : cluster [DBG] pgmap v6127: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:34.425 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T04:04:34.702 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T04:04:34.703 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 2h ago 2h - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T04:04:34.703 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 2h ago 2h 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T04:04:34.704 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 2h ago 2h 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T04:04:34.704 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 2h ago 2h - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T04:04:34.704 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T04:04:34.705 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 2h ago 2h 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T04:04:34.705 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T04:04:34.705 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 2h ago 2h - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T04:04:34.705 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 2h ago 2h 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T04:04:34.706 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T04:04:34.706 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T04:04:34.706 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T04:04:34.707 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T04:04:34.707 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 2h ago 2h 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T04:04:34.707 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 2h ago 2h 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T04:04:34.708 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 2h ago 2h 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T04:04:34.708 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 2h ago 2h 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T04:04:34.708 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 2h ago 2h 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T04:04:35.037 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T04:04:35.038 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T04:04:35.038 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T04:04:35.038 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:04:35.039 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T04:04:35.039 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T04:04:35.040 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:04:35.040 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:04:35.040 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T04:04:35.040 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T04:04:35.041 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:04:35.041 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T04:04:35.041 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T04:04:35.042 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T04:04:35.042 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:04:35.042 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T04:04:35.043 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T04:04:35.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:35 smithi154 conmon[34553]: cluster 2021-07-31T04:04:33.759008+0000 mgr.x (mgr.24542) 5881 : cluster [DBG] pgmap v6128: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:35.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:35 smithi045 conmon[33655]: cluster 2021-07-31T04:04:33.759008+0000 mgr.x (mgr.24542) 5881 2021-07-31T04:04:35.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:35 smithi045 conmon[33655]: : cluster [DBG] pgmap v6128: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:35.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:35 smithi045 conmon[39718]: cluster 2021-07-31T04:04:33.759008+0000 mgr.x 2021-07-31T04:04:35.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:35 smithi045 conmon[39718]: (mgr.24542) 5881 : cluster [DBG] pgmap v6128: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:36.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[34553]: audit 2021-07-31T04:04:34.413566+0000 mgr.x (mgr.24542) 5882 : audit [DBG] 2021-07-31T04:04:36.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[34553]: from='client.20838 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:04:36.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[34553]: audit 2021-07-31T04:04:34.694971+0000 mgr.x (mgr.24542) 5883 : audit [DBG] from='client.20844 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:04:36.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[34553]: audit 2021-07-31T04:04:35.038589+0000 mon.c 2021-07-31T04:04:36.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[34553]: (mon.1) 246 : audit [DBG] from='client.? 172.21.15.45:0/3066491456' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:04:36.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:36 smithi045 conmon[39718]: audit 2021-07-31T04:04:34.413566+0000 mgr.x (mgr.24542) 5882 : audit [DBG] from='client.20838 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:04:36.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:36 smithi045 conmon[39718]: audit 2021-07-31T04:04:34.694971+0000 mgr.x (mgr.24542) 5883 : audit [DBG] from='client.20844 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:04:36.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:36 smithi045 conmon[39718]: audit 2021-07 2021-07-31T04:04:36.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:36 smithi045 conmon[39718]: -31T04:04:35.038589+0000 mon.c (mon.1) 2021-07-31T04:04:36.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:36 smithi045 conmon[39718]: 246 : audit [DBG] from='client.? 172.21.15.45:0/3066491456' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:04:36.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:36 smithi045 conmon[33655]: audit 2021-07-31T04:04:34.413566+0000 mgr.x (mgr.24542) 5882 : audit [DBG] from='client.20838 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:04:36.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:36 smithi045 conmon[33655]: audit 2021-07-31T04:04:34.694971+0000 mgr.x (mgr.24542) 5883 : audit [DBG] from='client.20844 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:04:36.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:36 smithi045 conmon[33655]: audit 2021-07-31T04:04:36.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:36 smithi045 conmon[33655]: 2021-07-31T04 2021-07-31T04:04:36.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:36 smithi045 conmon[33655]: :04:35.038589+0000 mon.c (mon.1) 246 : audit [DBG] from='client.? 172.21.15.45:0/3066491456' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:04:36.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:36 smithi045 conmon[33655]: 2021-07-31T04:04:37.003 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: debug 2021-07-31T04:04:36.766+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:04:37.003 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:04:37.003 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:04:37.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: yield 2021-07-31T04:04:37.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:04:37.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:04:37.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:04:37.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:04:37.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:04:37.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:04:37.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:04:37.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:04:37.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:04:37.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: task.result() 2021-07-31T04:04:37.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:04:37.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: debug 2021-07-31T04:04:36.778+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:04:37.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:04:37.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:04:37.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: yield 2021-07-31T04:04:37.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:04:37.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:04:37.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:04:37.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:04:37.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:04:37.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:04:37.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:04:37.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:04:37.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:04:37.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: task.result() 2021-07-31T04:04:37.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:04:37.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: debug 2021-07-31T04:04:36.804+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:04:37.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:04:37.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:04:37.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: yield 2021-07-31T04:04:37.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:04:37.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:04:37.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:04:37.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:04:37.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:04:37.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:04:37.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:04:37.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:04:37.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:04:37.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: task.result() 2021-07-31T04:04:37.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:04:37.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: debug 2021-07-31T04:04:36.804+0000 7fcb9215e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:04:37.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:04:37.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:04:37.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: yield 2021-07-31T04:04:37.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:04:37.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:04:37.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:04:37.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:04:37.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:04:37.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:04:37.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:04:37.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:04:37.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:04:37.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: task.result() 2021-07-31T04:04:37.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:04:37.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: 2021-07-31T04:04:37.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T04:04:37.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: 2021-07-31T04:04:37.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:04:37.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:04:37.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: return f(*arg) 2021-07-31T04:04:37.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:04:37.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:04:37.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:04:37.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:04:37.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:04:37.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:04:37.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:04:37.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T04:04:37.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:04:37.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: raise self._exception 2021-07-31T04:04:37.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:04:37.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:04:37.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:04:37.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T04:04:37.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:04:37.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: raise 2021-07-31T04:04:37.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:04:37.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T04:04:37.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:04:37.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:04:37.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:04:37.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: debug 2021-07-31T04:04:36.813+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:04:37.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:04:37.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:04:37.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: yield 2021-07-31T04:04:37.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:04:37.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:04:37.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:04:37.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:04:37.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:04:37.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:04:37.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:04:37.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:04:37.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:04:37.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: task.result() 2021-07-31T04:04:37.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:04:37.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: debug 2021-07-31T04:04:36.813+0000 7fcb9195d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:04:37.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:04:37.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:04:37.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: yield 2021-07-31T04:04:37.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:04:37.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:04:37.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:04:37.035 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:04:37.035 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:04:37.035 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:04:37.035 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:04:37.036 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:04:37.036 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:04:37.036 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: task.result() 2021-07-31T04:04:37.036 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:04:37.037 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: 2021-07-31T04:04:37.037 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T04:04:37.037 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: 2021-07-31T04:04:37.037 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:04:37.038 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:04:37.038 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: return f(*arg) 2021-07-31T04:04:37.038 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:04:37.039 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:04:37.039 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:04:37.039 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:04:37.039 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:04:37.040 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:04:37.040 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:04:37.040 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T04:04:37.040 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:04:37.041 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: raise self._exception 2021-07-31T04:04:37.041 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:04:37.041 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:04:37.041 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:04:37.042 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T04:04:37.042 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:04:37.042 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: raise 2021-07-31T04:04:37.042 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:04:37.043 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T04:04:37.043 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:04:37.043 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:04:37.043 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:04:36 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:04:37.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:37 smithi154 conmon[34553]: cluster 2021-07-31T04:04:35.759515+0000 mgr.x (mgr.24542) 5884 : cluster [DBG] pgmap v6129: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:37.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:37 smithi154 conmon[34553]: audit 2021-07-31T04:04:36.721191+0000 mon.b (mon.2) 762 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T04:04:37.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:37 smithi154 conmon[34553]: audit 2021-07-31T04:04 2021-07-31T04:04:37.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:37 smithi154 conmon[34553]: :36.722735+0000 mon.b (mon.2) 763 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T04:04:37.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:37 smithi154 conmon[34553]: audit 2021-07-31 2021-07-31T04:04:37.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:37 smithi154 conmon[34553]: T04:04:36.723634+0000 mon.b (mon 2021-07-31T04:04:37.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:37 smithi154 conmon[34553]: .2) 764 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T04:04:37.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:37 smithi045 conmon[33655]: cluster 2021-07-31T04:04:35.759515+0000 mgr.x (mgr.24542) 2021-07-31T04:04:37.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:37 smithi045 conmon[33655]: 5884 : cluster [DBG] pgmap v6129: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:37.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:37 smithi045 conmon[33655]: audit 2021-07-31T04:04:36.721191+0000 mon.b (mon.2) 762 : audit [DBG] 2021-07-31T04:04:37.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:37 smithi045 conmon[33655]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T04:04:37.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:37 smithi045 conmon[33655]: audit 2021-07-31T04:04:36.722735+0000 mon.b 2021-07-31T04:04:37.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:37 smithi045 conmon[33655]: (mon.2) 763 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T04:04:37.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:37 smithi045 conmon[33655]: audit 2021 2021-07-31T04:04:37.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:37 smithi045 conmon[33655]: -07-31T04:04:36.723634+0000 mon.b (mon.2) 764 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T04:04:37.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:37 smithi045 conmon[39718]: cluster 2021-07-31T04:04:35.759515+0000 mgr.x (mgr.24542) 5884 : cluster [DBG] pgmap v6129: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:37.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:37 smithi045 conmon[39718]: audit 2021-07-31T04:04: 2021-07-31T04:04:37.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:37 smithi045 conmon[39718]: 36.721191+0000 mon.b (mon.2 2021-07-31T04:04:37.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:37 smithi045 conmon[39718]: ) 762 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T04:04:37.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:37 smithi045 conmon[39718]: 2021-07-31T04:04:37.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:37 smithi045 conmon[39718]: audit 2021- 2021-07-31T04:04:37.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:37 smithi045 conmon[39718]: 07-31T04:04:36.722735 2021-07-31T04:04:37.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:37 smithi045 conmon[39718]: +0000 mon.b (mon.2) 763 2021-07-31T04:04:37.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:37 smithi045 conmon[39718]: : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T04:04:37.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:37 smithi045 conmon[39718]: audit 2021 2021-07-31T04:04:37.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:37 smithi045 conmon[39718]: -07-31T04:04: 2021-07-31T04:04:37.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:37 smithi045 conmon[39718]: 36.723634+0000 mon.b ( 2021-07-31T04:04:37.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:37 smithi045 conmon[39718]: mon.2) 764 : audit 2021-07-31T04:04:37.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:37 smithi045 conmon[39718]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T04:04:38.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: cephadm 2021-07-31T04:04:36.724443+0000 mgr.x (mgr.24542) 5885 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T04:04:38.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: cephadm 2021-07-31T04:04:36.767790+0000 mgr.x (mgr.24542) 5886 : cephadm [ERR] 2021-07-31T04:04:38.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:04:38.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:04:38.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: yield 2021-07-31T04:04:38.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:04:38.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:04:38.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:04:38.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:04:38.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:04:38.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:04:38.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:04:38.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:04:38.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:04:38.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: task.result() 2021-07-31T04:04:38.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:04:38.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: cephadm 2021-07-31T04:04:36.768427+0000 mgr.x 2021-07-31T04:04:38.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: (mgr.24542) 5887 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T04:04:38.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: cephadm 2021-07-31T04:04:36.779443+0000 mgr.x (mgr.24542) 5888 : cephadm [ERR] 2021-07-31T04:04:38.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:04:38.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:04:38.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: yield 2021-07-31T04:04:38.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:04:38.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:04:38.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:04:38.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:04:38.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:04:38.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:04:38.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:04:38.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:04:38.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:04:38.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: task.result() 2021-07-31T04:04:38.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:04:38.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: cephadm 2021-07-31T04:04:36.780573+0000 2021-07-31T04:04:38.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: mgr.x (mgr.24542) 5889 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T04:04:38.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: cephadm 2021-07-31T04:04:36.805194+0000 mgr.x (mgr.24542) 5890 : cephadm [ERR] 2021-07-31T04:04:38.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:04:38.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:04:38.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: yield 2021-07-31T04:04:38.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:04:38.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:04:38.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:04:38.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:04:38.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:04:38.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:04:38.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:04:38.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:04:38.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:04:38.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: task.result() 2021-07-31T04:04:38.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:04:38.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: cephadm 2021-07- 2021-07-31T04:04:38.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: 31T04:04:36.805622+0000 mgr.x (mgr.24542) 5891 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:04:38.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:04:38.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:04:38.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: yield 2021-07-31T04:04:38.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:04:38.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:04:38.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:04:38.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:04:38.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:04:38.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:04:38.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:04:38.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:04:38.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:04:38.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: task.result() 2021-07-31T04:04:38.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:04:38.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: 2021-07-31T04:04:38.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T04:04:38.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: 2021-07-31T04:04:38.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:04:38.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:04:38.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: return f(*arg) 2021-07-31T04:04:38.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:04:38.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:04:38.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:04:38.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:04:38.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:04:38.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:04:38.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:04:38.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T04:04:38.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:04:38.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: raise self._exception 2021-07-31T04:04:38.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:04:38.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:04:38.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:04:38.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T04:04:38.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:04:38.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: raise 2021-07-31T04:04:38.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:04:38.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T04:04:38.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:04:38.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:04:38.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:04:38.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: cephadm 2021-07-31T04:04:36.814320+0000 mgr.x (mgr.24542) 5892 : cephadm [ERR] 2021-07-31T04:04:38.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:04:38.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:04:38.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: yield 2021-07-31T04:04:38.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:04:38.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:04:38.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:04:38.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:04:38.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:04:38.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:04:38.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:04:38.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:04:38.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:04:38.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: task.result() 2021-07-31T04:04:38.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:04:38.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: cephadm 2021-07-31T04:04:36.814852+0000 mgr.x (mgr.24542) 5893 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:04:38.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:04:38.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:04:38.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: yield 2021-07-31T04:04:38.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:04:38.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:04:38.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:04:38.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:04:38.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:04:38.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:04:38.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:04:38.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:04:38.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:04:38.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: task.result() 2021-07-31T04:04:38.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:04:38.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: 2021-07-31T04:04:38.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T04:04:38.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: 2021-07-31T04:04:38.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:04:38.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:04:38.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: return f(*arg) 2021-07-31T04:04:38.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:04:38.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:04:38.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:04:38.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:04:38.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:04:38.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:04:38.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:04:38.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T04:04:38.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:04:38.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: raise self._exception 2021-07-31T04:04:38.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:04:38.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:04:38.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:04:38.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T04:04:38.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:04:38.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: raise 2021-07-31T04:04:38.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:04:38.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T04:04:38.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:04:38.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:04:38.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:38 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:04:38.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: cephadm 2021-07-31T04:04:36.724443+0000 mgr.x (mgr.24542 2021-07-31T04:04:38.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: ) 5885 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T04:04:38.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: cephadm 2021-07-31T04:04:36. 2021-07-31T04:04:38.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: 767790+0000 mgr.x (mgr.24542) 5886 : cephadm [ERR] 2021-07-31T04:04:38.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:04:38.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:04:38.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: yield 2021-07-31T04:04:38.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:04:38.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:04:38.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:04:38.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:04:38.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:04:38.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:04:38.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:04:38.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:04:38.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:04:38.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: task.result() 2021-07-31T04:04:38.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:04:38.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: cephadm 2021-07-31T04:04:36.768427+0000 mgr.x (mgr.24542) 5887 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T04:04:38.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: cephadm 2021-07-31T04:04:36.779443+0000 mgr.x (mgr.24542) 5888 : cephadm [ERR] 2021-07-31T04:04:38.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: 2021-07-31T04:04:38.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:04:38.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:04:38.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: yield 2021-07-31T04:04:38.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:04:38.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:04:38.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:04:38.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:04:38.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:04:38.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:04:38.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:04:38.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:04:38.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:04:38.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: task.result() 2021-07-31T04:04:38.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:04:38.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: cephadm 2021-07-31T04:04:36.780573+0000 mgr.x (mgr.24542) 5889 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T04:04:38.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: cephadm 2021-07-31T04:04:36.805194+0000 mgr.x (mgr.24542) 5890 2021-07-31T04:04:38.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: : cephadm [ERR] 2021-07-31T04:04:38.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:04:38.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:04:38.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: yield 2021-07-31T04:04:38.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:04:38.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:04:38.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:04:38.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:04:38.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:04:38.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:04:38.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:04:38.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:04:38.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:04:38.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: task.result() 2021-07-31T04:04:38.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:04:38.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: cephadm 2021-07-31T04:04:36.805622+0000 mgr.x (mgr.24542) 5891 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:04:38.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:04:38.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:04:38.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: yield 2021-07-31T04:04:38.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:04:38.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:04:38.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:04:38.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:04:38.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:04:38.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:04:38.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:04:38.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:04:38.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:04:38.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: task.result() 2021-07-31T04:04:38.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:04:38.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: 2021-07-31T04:04:38.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T04:04:38.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: 2021-07-31T04:04:38.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:04:38.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:04:38.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: return f(*arg) 2021-07-31T04:04:38.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:04:38.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:04:38.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:04:38.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:04:38.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:04:38.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:04:38.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:04:38.550 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: cephadm 2021-07-31T04:04:36. 2021-07-31T04:04:38.550 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: 724443+0000 mgr.x (mgr.24542) 5885 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T04:04:38.551 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: cephadm 2021-07-31T04:04:36.767790+0000 mgr.x 2021-07-31T04:04:38.551 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: (mgr.24542) 5886 : cephadm [ERR] 2021-07-31T04:04:38.551 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:04:38.551 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:04:38.551 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: yield 2021-07-31T04:04:38.552 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:04:38.552 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:04:38.552 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:04:38.552 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:04:38.552 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:04:38.553 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:04:38.553 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:04:38.553 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:04:38.553 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:04:38.553 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: task.result() 2021-07-31T04:04:38.553 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:04:38.554 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: cephadm 2021-07-31T04:04:38.554 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: 2021-07-31T04:04:36.768427+0000 mgr.x (mgr.24542) 5887 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T04:04:38.554 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: cephadm 2021-07-31T 2021-07-31T04:04:38.554 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: 04:04:36.779443+0000 mgr.x (mgr. 2021-07-31T04:04:38.554 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: 24542) 5888 : cephadm [ERR] 2021-07-31T04:04:38.555 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:04:38.555 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:04:38.555 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: yield 2021-07-31T04:04:38.555 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:04:38.555 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:04:38.556 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:04:38.556 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:04:38.556 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:04:38.556 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:04:38.556 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:04:38.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:04:38.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:04:38.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: task.result() 2021-07-31T04:04:38.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:04:38.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: 2021-07-31T04:04:38.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: cephadm 2021-07-31T04:04:36.780573+0000 mgr.x (mgr.24542) 2021-07-31T04:04:38.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: 5889 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T04:04:38.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: cephadm 2021-07 2021-07-31T04:04:38.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: -31T04:04:36.805194+0000 2021-07-31T04:04:38.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T04:04:38.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:04:38.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: raise self._exception 2021-07-31T04:04:38.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:04:38.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:04:38.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:04:38.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T04:04:38.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:04:38.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: raise 2021-07-31T04:04:38.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:04:38.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T04:04:38.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:04:38.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:04:38.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:04:38.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: cephadm 2021-07-31T04:04:36.814320+0000 2021-07-31T04:04:38.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: mgr.x (mgr.24542) 5892 : cephadm [ERR] 2021-07-31T04:04:38.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:04:38.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:04:38.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: yield 2021-07-31T04:04:38.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:04:38.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:04:38.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:04:38.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:04:38.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:04:38.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:04:38.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:04:38.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:04:38.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:04:38.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: task.result() 2021-07-31T04:04:38.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:04:38.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: cephadm 2021-07-31T04:04:36.814852+0000 mgr.x (mgr.24542) 5893 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:04:38.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:04:38.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:04:38.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: yield 2021-07-31T04:04:38.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:04:38.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:04:38.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:04:38.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:04:38.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:04:38.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:04:38.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:04:38.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:04:38.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:04:38.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: task.result() 2021-07-31T04:04:38.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:04:38.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: 2021-07-31T04:04:38.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T04:04:38.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: 2021-07-31T04:04:38.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:04:38.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:04:38.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: return f(*arg) 2021-07-31T04:04:38.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:04:38.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:04:38.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:04:38.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:04:38.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:04:38.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:04:38.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:04:38.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T04:04:38.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:04:38.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: raise self._exception 2021-07-31T04:04:38.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:04:38.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:04:38.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:04:38.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T04:04:38.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:04:38.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: raise 2021-07-31T04:04:38.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:04:38.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T04:04:38.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:04:38.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:04:38.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:04:38.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: mgr.x (mgr.24542) 5890 : 2021-07-31T04:04:38.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: cephadm [ERR] 2021-07-31T04:04:38.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:04:38.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:04:38.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: yield 2021-07-31T04:04:38.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:04:38.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:04:38.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:04:38.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:04:38.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:04:38.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:04:38.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:04:38.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:04:38.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:04:38.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: task.result() 2021-07-31T04:04:38.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:04:38.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: cephadm 2021-07-31T04:04:36.805622+0000 mgr.x (mgr. 2021-07-31T04:04:38.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: 24542) 5891 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:04:38.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:04:38.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:04:38.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: yield 2021-07-31T04:04:38.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:04:38.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:04:38.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:04:38.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:04:38.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:04:38.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:04:38.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:04:38.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:04:38.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:04:38.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: task.result() 2021-07-31T04:04:38.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:04:38.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: 2021-07-31T04:04:38.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T04:04:38.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: 2021-07-31T04:04:38.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:04:38.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:04:38.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: return f(*arg) 2021-07-31T04:04:38.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:04:38.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:04:38.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:04:38.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:04:38.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:04:38.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:04:38.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:04:38.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T04:04:38.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:04:38.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: raise self._exception 2021-07-31T04:04:38.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:04:38.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:04:38.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:04:38.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T04:04:38.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:04:38.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: raise 2021-07-31T04:04:38.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:04:38.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T04:04:38.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:04:38.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:04:38.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:04:38.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: 2021-07-31T04:04:38.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: cephadm 2021-07-31T04:04:36.814320+0000 mgr.x (mgr.24542) 5892 : cephadm [ERR] 2021-07-31T04:04:38.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:04:38.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:04:38.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: yield 2021-07-31T04:04:38.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:04:38.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:04:38.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:04:38.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:04:38.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:04:38.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:04:38.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:04:38.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:04:38.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:04:38.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: task.result() 2021-07-31T04:04:38.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:04:38.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: cephadm 2021-07-31T04:04:36.814852+0000 mgr.x (mgr.24542) 5893 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:04:38.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:04:38.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:04:38.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: yield 2021-07-31T04:04:38.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:04:38.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:04:38.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:04:38.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:04:38.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:04:38.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:04:38.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:04:38.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:04:38.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:04:38.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: task.result() 2021-07-31T04:04:38.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:04:38.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: 2021-07-31T04:04:38.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T04:04:38.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: 2021-07-31T04:04:38.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:04:38.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:04:38.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: return f(*arg) 2021-07-31T04:04:38.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:04:38.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:04:38.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:04:38.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:04:38.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:04:38.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:04:38.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:04:38.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T04:04:38.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:04:38.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: raise self._exception 2021-07-31T04:04:38.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:04:38.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:04:38.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:04:38.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T04:04:38.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:04:38.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: raise 2021-07-31T04:04:38.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:04:38.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T04:04:38.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:04:38.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:04:38.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:38 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:04:39.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:39 smithi154 conmon[34553]: cluster 2021-07-31T04:04:37.760025+0000 mgr.x (mgr.24542) 5894 : cluster [DBG] pgmap v6130: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:39.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:39 smithi045 conmon[39718]: cluster 2021-07-31T04:04:37 2021-07-31T04:04:39.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:39 smithi045 conmon[39718]: .760025+0000 mgr.x (mgr.24542) 5894 : cluster [DBG] pgmap v6130: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:39.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:39 smithi045 conmon[33655]: cluster 2021-07-31T04:04:37.760025+0000 2021-07-31T04:04:39.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:39 smithi045 conmon[33655]: mgr.x (mgr.24542) 5894 : cluster [DBG] pgmap v6130: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:41.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:41 smithi154 conmon[34553]: cluster 2021-07-31T04:04:39.760469+0000 mgr.x (mgr.24542) 5895 : cluster [DBG] pgmap v6131: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:41.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:41 smithi045 conmon[39718]: cluster 2021-07-31T04:04:39. 2021-07-31T04:04:41.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:41 smithi045 conmon[39718]: 760469+0000 mgr.x (mgr.24542) 5895 : cluster [DBG] pgmap v6131: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:41.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:41 smithi045 conmon[33655]: cluster 2021-07-31T04:04:39.760469+0000 mgr.x (mgr 2021-07-31T04:04:41.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:41 smithi045 conmon[33655]: .24542) 5895 : cluster [DBG] pgmap v6131: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:43.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:43 smithi154 conmon[34553]: cluster 2021-07-31T04:04:41.760841+0000 mgr.x (mgr.24542) 5896 : cluster [DBG] 2021-07-31T04:04:43.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:43 smithi154 conmon[34553]: pgmap v6132: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:43.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:43 smithi045 conmon[33655]: cluster 2021-07-31T04:04:41.760841+0000 mgr.x 2021-07-31T04:04:43.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:43 smithi045 conmon[33655]: (mgr.24542) 5896 : cluster [DBG] pgmap v6132: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:43.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:43 smithi045 conmon[39718]: cluster 2021-07-31T04:04: 2021-07-31T04:04:43.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:43 smithi045 conmon[39718]: 41.760841+0000 mgr.x (mgr.24542) 5896 : cluster [DBG] pgmap v6132: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:45.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:45 smithi154 conmon[34553]: cluster 2021-07-31T04:04:43.761252+0000 mgr.x (mgr.24542) 5897 : cluster [DBG] pgmap v6133: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:45.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:45 smithi045 conmon[39718]: cluster 2021-07-31T04:04:43.761252+0000 mgr.x (mgr.24542) 2021-07-31T04:04:45.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:45 smithi045 conmon[39718]: 5897 : cluster [DBG] pgmap v6133: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:45.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:45 smithi045 conmon[33655]: cluster 2021-07-31T04:04:43.761252+0000 mgr.x ( 2021-07-31T04:04:45.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:45 smithi045 conmon[33655]: mgr.24542) 5897 : cluster [DBG] pgmap v6133: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:47.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:47 smithi154 conmon[34553]: cluster 2021-07-31T04:04:45.761833+0000 mgr.x (mgr.24542) 5898 : cluster [DBG] pgmap v6134: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:47.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:47 smithi045 conmon[33655]: cluster 2021-07-31T04:04:45.761833+0000 mgr.x (mgr.24542) 5898 : cluster 2021-07-31T04:04:47.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:47 smithi045 conmon[33655]: [DBG] pgmap v6134: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:47.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:47 smithi045 conmon[39718]: cluster 2021-07-31T04:04:45.761833+0000 mgr.x (mgr. 2021-07-31T04:04:47.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:47 smithi045 conmon[39718]: 24542) 5898 : cluster [DBG] pgmap v6134: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:49.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:49 smithi154 conmon[34553]: cluster 2021-07-31T04:04:47.762388+0000 mgr.x (mgr.24542) 5899 : cluster [DBG] pgmap v6135: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:49.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:49 smithi045 conmon[39718]: cluster 2021-07-31T04:04:47 2021-07-31T04:04:49.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:49 smithi045 conmon[39718]: .762388+0000 mgr.x (mgr.24542) 5899 : cluster [DBG] pgmap v6135: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:49.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:49 smithi045 conmon[33655]: cluster 2021-07-31T04:04:47.762388+0000 2021-07-31T04:04:49.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:49 smithi045 conmon[33655]: mgr.x (mgr.24542) 5899 : cluster [DBG] pgmap v6135: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:51.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:51 smithi154 conmon[34553]: cluster 2021-07-31T04:04:49.763007+0000 mgr.x (mgr.24542) 5900 : cluster [DBG] pgmap v6136: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:51.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:51 smithi045 conmon[39718]: cluster 2021-07-31T04:04:49 2021-07-31T04:04:51.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:51 smithi045 conmon[39718]: .763007+0000 mgr.x (mgr.24542) 5900 : cluster [DBG] pgmap v6136: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:51.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:51 smithi045 conmon[33655]: cluster 2021-07-31T04:04:49.763007+0000 mgr.x (mgr.24542) 5900 2021-07-31T04:04:51.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:51 smithi045 conmon[33655]: : cluster [DBG] pgmap v6136: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:53.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:53 smithi154 conmon[34553]: cluster 2021-07-31T04:04:51.763465+0000 mgr.x (mgr.24542) 5901 : cluster [DBG] pgmap v6137: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:53.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:53 smithi045 conmon[39718]: cluster 2021-07-31T04:04:51.763465+0000 mgr.x ( 2021-07-31T04:04:53.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:53 smithi045 conmon[39718]: mgr.24542) 5901 : cluster [DBG] pgmap v6137: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:53.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:53 smithi045 conmon[33655]: cluster 2021-07-31T04:04:51.763465+0000 mgr.x (mgr.24542) 5901 : cluster [DBG] pgmap v6137: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:55.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:55 smithi154 conmon[34553]: cluster 2021-07-31T04:04:53.763990+0000 mgr.x (mgr.24542) 5902 : cluster [DBG] pgmap v6138: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:55.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:55 smithi045 conmon[33655]: cluster 2021-07-31T04:04:53.763990+0000 mgr.x (mgr.24542) 5902 2021-07-31T04:04:55.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:55 smithi045 conmon[33655]: : cluster [DBG] pgmap v6138: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:55.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:55 smithi045 conmon[39718]: cluster 2021-07-31T04:04:53.763990+0000 mgr.x (mgr. 2021-07-31T04:04:55.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:55 smithi045 conmon[39718]: 24542) 5902 : cluster [DBG] pgmap v6138: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:57.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:57 smithi154 conmon[34553]: cluster 2021-07-31T04:04:55.764459+0000 mgr.x (mgr.24542) 5903 : cluster [DBG] pgmap v6139: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:57.524 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:57 smithi045 conmon[33655]: cluster 2021-07-31T04:04:55.764459+0000 mgr.x (mgr.24542) 5903 : cluster [DBG] pgmap v6139: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:57.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:57 smithi045 conmon[39718]: cluster 2021-07-31T04:04:55.764459+0000 mgr.x (mgr.24542) 5903 : cluster [DBG] pgmap v6139: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:59.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:04:59 smithi154 conmon[34553]: cluster 2021-07-31T04:04:57.764864+0000 mgr.x (mgr.24542) 5904 : cluster [DBG] pgmap v6140: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:59.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:04:59 smithi045 conmon[39718]: cluster 2021-07-31T04:04:57.764864+0000 mgr.x (mgr.24542) 5904 : cluster [DBG] pgmap v6140: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:04:59.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:59 smithi045 conmon[33655]: cluster 2021-07-31T04:04:57.764864+0000 mgr.x (mgr 2021-07-31T04:04:59.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:04:59 smithi045 conmon[33655]: .24542) 5904 : cluster [DBG] pgmap v6140: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:01.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:01 smithi154 conmon[34553]: cluster 2021-07-31T04:04:59.765334+0000 mgr.x (mgr.24542) 5905 : cluster [DBG] pgmap v6141: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:01.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:01 smithi045 conmon[39718]: cluster 2021-07-31T04:04:59.765334+0000 mgr.x (mgr.24542) 5905 : cluster [DBG] pgmap v6141: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:01.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:01 smithi045 conmon[39718]: 2021-07-31T04:05:01.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:01 smithi045 conmon[33655]: cluster 2021-07-31T04:04:59.765334+0000 mgr.x (mgr.24542) 5905 : cluster 2021-07-31T04:05:01.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:01 smithi045 conmon[33655]: [DBG] pgmap v6141: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:03.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:03 smithi154 conmon[34553]: cluster 2021-07-31T04:05:01.765706+0000 mgr.x (mgr.24542) 2021-07-31T04:05:03.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:03 smithi154 conmon[34553]: 5906 : cluster [DBG] pgmap v6142: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:03.524 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:03 smithi045 conmon[39718]: cluster 2021-07-31T04:05:01.765706+0000 mgr.x (mgr.24542) 5906 : cluster 2021-07-31T04:05:03.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:03 smithi045 conmon[39718]: [DBG] pgmap v6142: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:03.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:03 smithi045 conmon[33655]: cluster 2021-07-31T04:05:01.765706+0000 2021-07-31T04:05:03.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:03 smithi045 conmon[33655]: mgr.x (mgr.24542) 5906 : cluster [DBG] pgmap v6142: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:05.351 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T04:05:05.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:05 smithi154 conmon[34553]: cluster 2021-07-31T04:05:03.766131+0000 mgr.x (mgr.24542) 5907 : cluster [DBG] pgmap v6143: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:05.524 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:05 smithi045 conmon[39718]: cluster 2021-07-31T04:05:03.766131 2021-07-31T04:05:05.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:05 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 5907 : cluster [DBG] pgmap v6143: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:05.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:05 smithi045 conmon[33655]: cluster 2021-07-31T04:05:03.766131 2021-07-31T04:05:05.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:05 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 5907 : cluster [DBG] pgmap v6143: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:05.625 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T04:05:05.626 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 2h ago 2h - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T04:05:05.626 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 2h ago 2h 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T04:05:05.626 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 2h ago 2h 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T04:05:05.626 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 2h ago 2h - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T04:05:05.627 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T04:05:05.627 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 2h ago 2h 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T04:05:05.627 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T04:05:05.627 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 2h ago 2h - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T04:05:05.628 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 2h ago 2h 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T04:05:05.628 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T04:05:05.628 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T04:05:05.628 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T04:05:05.629 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T04:05:05.629 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 2h ago 2h 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T04:05:05.629 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 2h ago 2h 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T04:05:05.629 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 2h ago 2h 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T04:05:05.629 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 2h ago 2h 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T04:05:05.630 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 2h ago 2h 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T04:05:05.964 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T04:05:05.964 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T04:05:05.964 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T04:05:05.965 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:05:05.965 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T04:05:05.965 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T04:05:05.965 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:05:05.965 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:05:05.966 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T04:05:05.966 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T04:05:05.966 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:05:05.966 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T04:05:05.967 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T04:05:05.967 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T04:05:05.967 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:05:05.967 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T04:05:05.968 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T04:05:06.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:06 smithi154 conmon[34553]: audit 2021-07-31T04:05:05.340179+0000 mgr.x (mgr.24542) 5908 : audit [DBG] from='client.30793 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:05:06.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:06 smithi154 conmon[34553]: 2021-07-31T04:05:06.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:06 smithi154 conmon[34553]: audit 2021-07-31T04:05:05.965012+0000 mon.a (mon.0) 3419 : audit [DBG] from='client.? 172.21.15.45:0/3805915498' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:05:06.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:06 smithi045 conmon[33655]: audit 2021-07-31T04:05:05.340179+0000 2021-07-31T04:05:06.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:06 smithi045 conmon[33655]: mgr.x (mgr.24542) 5908 : audit [DBG] from='client.30793 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:05:06.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:06 smithi045 conmon[33655]: audit 2021-07-31T04:05:05.965012+0000 mon.a ( 2021-07-31T04:05:06.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:06 smithi045 conmon[33655]: mon.0) 3419 : audit [DBG] from='client.? 172.21.15.45:0/3805915498' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:05:06.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:06 smithi045 conmon[39718]: audit 2021-07-31T04 2021-07-31T04:05:06.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:06 smithi045 conmon[39718]: :05:05.340179+0000 mgr.x (mgr.24542) 5908 : audit [DBG] from='client.30793 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:05:06.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:06 smithi045 conmon[39718]: audit 2021 2021-07-31T04:05:06.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:06 smithi045 conmon[39718]: -07-31T04:05:05 2021-07-31T04:05:06.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:06 smithi045 conmon[39718]: .965012+0000 mon.a (mon 2021-07-31T04:05:06.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:06 smithi045 conmon[39718]: .0) 3419 : audit [DBG] 2021-07-31T04:05:06.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:06 smithi045 conmon[39718]: from='client.? 172.21.15.45:0/3805915498' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:05:07.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:07 smithi154 conmon[34553]: audit 2021-07-31T04:05:05.618812+0000 mgr.x (mgr.24542) 5909 : audit [DBG] from='client.20862 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:05:07.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:07 smithi154 conmon[34553]: cluster 2021-07-31T04:05:05.766608+0000 mgr.x (mgr.24542) 5910 : cluster [DBG] pgmap v6144: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:07.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:07 smithi045 conmon[33655]: audit 2021-07-31T04:05:05.618812+0000 mgr.x (mgr.24542) 2021-07-31T04:05:07.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:07 smithi045 conmon[33655]: 5909 : audit [DBG] from='client.20862 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:05:07.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:07 smithi045 conmon[33655]: cluster 2021-07-31T04:05:05.766608+0000 mgr.x (mgr.24542) 5910 : cluster [DBG] 2021-07-31T04:05:07.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:07 smithi045 conmon[33655]: pgmap v6144: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:07.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:07 smithi045 conmon[39718]: audit 2021-07-31T04:05:05. 2021-07-31T04:05:07.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:07 smithi045 conmon[39718]: 618812+0000 mgr.x (mgr.24542) 5909 : audit [DBG] from='client.20862 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:05:07.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:07 smithi045 conmon[39718]: cluster 2021-07-31T04:05:07.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:07 smithi045 conmon[39718]: 2021-07-31T04:05:05.766608+0000 2021-07-31T04:05:07.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:07 smithi045 conmon[39718]: mgr.x (mgr.24542) 5910 : 2021-07-31T04:05:07.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:07 smithi045 conmon[39718]: cluster [DBG] pgmap v6144: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:09.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:09 smithi154 conmon[34553]: cluster 2021-07-31T04:05:07.767033+0000 mgr.x (mgr.24542) 5911 : cluster [DBG] pgmap v6145: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:09.524 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:09 smithi045 conmon[33655]: cluster 2021-07-31T04:05:07.767033+0000 mgr.x (mgr.24542) 5911 : cluster [DBG] pgmap v6145: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:09.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:09 smithi045 conmon[39718]: cluster 2021-07-31T04:05:07.767033+0000 mgr.x (mgr.24542) 2021-07-31T04:05:09.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:09 smithi045 conmon[39718]: 5911 : cluster [DBG] pgmap v6145: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:11.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:11 smithi154 conmon[34553]: cluster 2021-07-31T04:05:09.767642+0000 mgr.x (mgr.24542) 5912 : cluster 2021-07-31T04:05:11.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:11 smithi154 conmon[34553]: [DBG] pgmap v6146: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:11.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:11 smithi045 conmon[39718]: cluster 2021-07-31T04:05:09.767642+0000 mgr.x ( 2021-07-31T04:05:11.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:11 smithi045 conmon[39718]: mgr.24542) 5912 : cluster [DBG] pgmap v6146: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:11.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:11 smithi045 conmon[33655]: cluster 2021-07-31T04:05:09.767642+0000 mgr.x (mgr.24542) 5912 : cluster [DBG] 2021-07-31T04:05:11.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:11 smithi045 conmon[33655]: pgmap v6146: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:13.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:13 smithi154 conmon[34553]: cluster 2021-07-31T04:05:11.768085+0000 mgr.x (mgr.24542) 5913 : cluster [DBG] pgmap v6147: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:13.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:13 smithi045 conmon[39718]: cluster 2021-07-31T04:05:11.768085+0000 mgr.x (mgr.24542 2021-07-31T04:05:13.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:13 smithi045 conmon[39718]: ) 5913 : cluster [DBG] pgmap v6147: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:13.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:13 smithi045 conmon[33655]: cluster 2021-07-31T04:05:11.768085+0000 mgr.x (mgr.24542) 5913 : cluster [DBG] 2021-07-31T04:05:13.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:13 smithi045 conmon[33655]: pgmap v6147: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:15.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:15 smithi154 conmon[34553]: cluster 2021-07-31T04:05:13.768463+0000 mgr.x (mgr.24542) 5914 : cluster [DBG] pgmap v6148: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:15.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:15 smithi045 conmon[33655]: cluster 2021-07-31T04:05:13.768463+0000 mgr.x (mgr.24542) 5914 : cluster [DBG] 2021-07-31T04:05:15.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:15 smithi045 conmon[33655]: pgmap v6148: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:15.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:15 smithi045 conmon[39718]: cluster 2021-07-31T04:05:13.768463+0000 mgr.x ( 2021-07-31T04:05:15.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:15 smithi045 conmon[39718]: mgr.24542) 5914 : cluster [DBG] pgmap v6148: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:17.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:17 smithi154 conmon[34553]: cluster 2021-07-31T04:05:15.769024+0000 mgr.x (mgr.24542) 5915 : cluster [DBG] pgmap v6149: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:17.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:17 smithi045 conmon[39718]: cluster 2021-07-31T04:05:15.769024+0000 mgr.x (mgr.24542) 5915 : cluster [DBG] pgmap v6149: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:17.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:17 smithi045 conmon[33655]: cluster 2021-07-31T04:05:15.769024+0000 mgr.x (mgr.24542) 5915 : cluster 2021-07-31T04:05:17.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:17 smithi045 conmon[33655]: [DBG] pgmap v6149: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:19.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:19 smithi154 conmon[34553]: cluster 2021-07-31T04:05:17.769530+0000 mgr.x (mgr.24542) 5916 : cluster [DBG] pgmap v6150: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:19.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:19 smithi154 conmon[34553]: 2021-07-31T04:05:19.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:19 smithi045 conmon[39718]: cluster 2021-07-31T04:05: 2021-07-31T04:05:19.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:19 smithi045 conmon[39718]: 17.769530+0000 mgr.x (mgr.24542) 5916 : cluster [DBG] pgmap v6150: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:19.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:19 smithi045 conmon[33655]: cluster 2021-07-31T04:05:17.769530+0000 mgr.x (mgr.24542) 5916 : cluster 2021-07-31T04:05:19.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:19 smithi045 conmon[33655]: [DBG] pgmap v6150: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:21.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:21 smithi154 conmon[34553]: cluster 2021-07-31T04:05:19.770123+0000 mgr.x (mgr.24542) 5917 : cluster [DBG] pgmap v6151: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:21.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:21 smithi045 conmon[33655]: cluster 2021-07-31T04:05:19.770123+0000 mgr.x 2021-07-31T04:05:21.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:21 smithi045 conmon[33655]: (mgr.24542) 5917 : cluster [DBG] pgmap v6151: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:21.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:21 smithi045 conmon[39718]: cluster 2021-07-31T04:05:19.770123+0000 mgr.x ( 2021-07-31T04:05:21.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:21 smithi045 conmon[39718]: mgr.24542) 5917 : cluster [DBG] pgmap v6151: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:23.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:23 smithi154 conmon[34553]: cluster 2021-07-31T04:05:21.770650+0000 mgr.x (mgr.24542) 5918 : cluster [DBG] pgmap v6152: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:23.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:23 smithi045 conmon[39718]: cluster 2021-07-31T04:05:21.770650+0000 mgr.x (mgr. 2021-07-31T04:05:23.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:23 smithi045 conmon[39718]: 24542) 5918 : cluster [DBG] pgmap v6152: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:23.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:23 smithi045 conmon[33655]: cluster 2021-07-31T04:05:21.770650+0000 mgr.x (mgr.24542) 5918 : cluster [DBG] pgmap v6152: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:25.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:25 smithi154 conmon[34553]: cluster 2021-07-31T04:05:23.771135+0000 mgr.x (mgr.24542) 5919 : cluster [DBG] pgmap v6153: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:25.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:25 smithi154 conmon[34553]: 2021-07-31T04:05:25.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:25 smithi045 conmon[33655]: cluster 2021-07-31T04:05:23.771135+0000 mgr.x (mgr. 2021-07-31T04:05:25.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:25 smithi045 conmon[33655]: 24542) 5919 : cluster [DBG] pgmap v6153: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:25.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:25 smithi045 conmon[39718]: cluster 2021-07-31T04:05:23.771135+0000 mgr.x 2021-07-31T04:05:25.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:25 smithi045 conmon[39718]: (mgr.24542) 5919 : cluster [DBG] pgmap v6153: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:27.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:27 smithi154 conmon[34553]: cluster 2021-07-31T04:05:25.771600+0000 mgr.x (mgr.24542) 5920 : cluster [DBG] pgmap v6154: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:27.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:27 smithi045 conmon[39718]: cluster 2021-07-31T04:05:25.771600+0000 mgr.x 2021-07-31T04:05:27.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:27 smithi045 conmon[39718]: (mgr.24542) 5920 : cluster [DBG] pgmap v6154: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:27.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:27 smithi045 conmon[33655]: cluster 2021-07-31T04:05:25.771600+0000 mgr.x (mgr.24542) 5920 : 2021-07-31T04:05:27.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:27 smithi045 conmon[33655]: cluster [DBG] pgmap v6154: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:29.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:29 smithi154 conmon[34553]: cluster 2021-07-31T04:05:27.772118+0000 mgr.x (mgr.24542) 5921 : cluster [DBG] pgmap v6155: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:29.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:29 smithi154 conmon[34553]: audit 2021-07-31T04:05:28.480838+0000 mon.b (mon.2) 765 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:05:29.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:29 smithi154 conmon[34553]: audit 2021-07-31T04:05:28.481113+0000 mon.b (mon.2) 766 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:05:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:29 smithi154 conmon[34553]: audit 2021-07-31T04:05:28.481910+0000 mon.a ( 2021-07-31T04:05:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:29 smithi154 conmon[34553]: mon.0) 3420 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:05:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:29 smithi154 conmon[34553]: audit 2021-07-31T04:05:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:29 smithi154 conmon[34553]: 2021-07-31T04:05:28.482360 2021-07-31T04:05:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:29 smithi154 conmon[34553]: +0000 mon.a (mon.0) 3421 : audit [INF] 2021-07-31T04:05:29.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:29 smithi154 conmon[34553]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:05:29.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:29 smithi045 conmon[33655]: cluster 2021-07-31T04:05:27.772118+0000 mgr.x (mgr.24542) 5921 : cluster [DBG] pgmap v6155: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:29.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:29 smithi045 conmon[33655]: audit 2021-07-31T04:05:28.480838+0000 mon.b (mon.2) 765 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:05:29.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:29 smithi045 conmon[33655]: audit 2021-07-31T04:05 2021-07-31T04:05:29.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:29 smithi045 conmon[33655]: :28.481113+0000 mon.b (mon.2) 766 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:05:29.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:29 smithi045 conmon[33655]: audit 2021-07-31T04:05 2021-07-31T04:05:29.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:29 smithi045 conmon[33655]: :28.481910+0000 mon.a (mon.0 2021-07-31T04:05:29.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:29 smithi045 conmon[33655]: ) 3420 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:05:29.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:29 smithi045 conmon[33655]: audit 2021- 2021-07-31T04:05:29.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:29 smithi045 conmon[33655]: 07-31T04:05:28.482360+0000 2021-07-31T04:05:29.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:29 smithi045 conmon[33655]: mon.a (mon.0) 3421 : audit [INF] 2021-07-31T04:05:29.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:29 smithi045 conmon[33655]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:05:29.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:29 smithi045 conmon[39718]: cluster 2021-07-31T04:05:27.772118+0000 mgr.x ( 2021-07-31T04:05:29.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:29 smithi045 conmon[39718]: mgr.24542) 5921 : cluster [DBG] pgmap v6155: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:29.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:29 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T04:05:29.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:29 smithi045 conmon[39718]: 31T04:05:28.480838+0000 mon.b (mon.2) 765 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:05:29.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:29 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T04:05:29.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:29 smithi045 conmon[39718]: 04:05:28.481113+0000 mon.b (mon.2) 766 : audit 2021-07-31T04:05:29.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:29 smithi045 conmon[39718]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:05:29.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:29 smithi045 conmon[39718]: audit 2021-07-31T04:05:29.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:29 smithi045 conmon[39718]: 2021-07-31T04:05:28 2021-07-31T04:05:29.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:29 smithi045 conmon[39718]: .481910+0000 mon.a (mon.0) 3420 : 2021-07-31T04:05:29.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:29 smithi045 conmon[39718]: audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:05:29.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:29 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T04:05:29.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:29 smithi045 conmon[39718]: T04:05:28.482360+0000 mon.a (mon. 2021-07-31T04:05:29.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:29 smithi045 conmon[39718]: 0) 3421 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:05:31.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:31 smithi154 conmon[34553]: cluster 2021-07-31T04:05:29.772525+0000 mgr.x (mgr.24542) 5922 : cluster 2021-07-31T04:05:31.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:31 smithi154 conmon[34553]: [DBG] pgmap v6156: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:31.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:31 smithi045 conmon[39718]: cluster 2021-07-31T04:05:29.772525+0000 mgr.x (mgr.24542 2021-07-31T04:05:31.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:31 smithi045 conmon[39718]: ) 5922 : cluster [DBG] pgmap v6156: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:31.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:31 smithi045 conmon[33655]: cluster 2021-07-31T04:05:29.772525+0000 mgr.x (mgr.24542) 5922 : cluster 2021-07-31T04:05:31.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:31 smithi045 conmon[33655]: [DBG] pgmap v6156: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:33.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:33 smithi154 conmon[34553]: cluster 2021-07-31T04:05:31.772907+0000 mgr.x (mgr.24542) 5923 : cluster [DBG] pgmap v6157: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:33.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:33 smithi045 conmon[33655]: cluster 2021-07-31T04:05:31.772907+0000 mgr.x (mgr.24542) 5923 : cluster 2021-07-31T04:05:33.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:33 smithi045 conmon[33655]: [DBG] pgmap v6157: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:33.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:33 smithi045 conmon[39718]: cluster 2021-07-31T04:05:31.772907+0000 mgr.x (mgr 2021-07-31T04:05:33.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:33 smithi045 conmon[39718]: .24542) 5923 : cluster [DBG] pgmap v6157: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:35.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:35 smithi154 conmon[34553]: cluster 2021-07-31T04:05:33.773264+0000 mgr.x (mgr.24542) 5924 : cluster [DBG] pgmap v6158: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:35.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:35 smithi045 conmon[39718]: cluster 2021-07-31T04:05:33.773264+0000 mgr.x (mgr.24542) 5924 : cluster [DBG] pgmap v6158: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:35.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:35 smithi045 conmon[33655]: cluster 2021-07-31T04:05:33.773264+0000 mgr.x (mgr.24542) 5924 : cluster 2021-07-31T04:05:35.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:35 smithi045 conmon[33655]: [DBG] pgmap v6158: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:36.278 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T04:05:36.556 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T04:05:36.556 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 2h ago 2h - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T04:05:36.556 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 2h ago 2h 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T04:05:36.557 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 2h ago 2h 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T04:05:36.557 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 2h ago 2h - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T04:05:36.557 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T04:05:36.557 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 2h ago 2h 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T04:05:36.558 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T04:05:36.558 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 2h ago 2h - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T04:05:36.558 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 2h ago 2h 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T04:05:36.558 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T04:05:36.559 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T04:05:36.559 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T04:05:36.559 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T04:05:36.559 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 2h ago 2h 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T04:05:36.560 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 2h ago 2h 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T04:05:36.560 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 2h ago 2h 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T04:05:36.560 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 2h ago 2h 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T04:05:36.560 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 2h ago 2h 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T04:05:36.892 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T04:05:36.893 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T04:05:36.893 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T04:05:36.893 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:05:36.894 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T04:05:36.894 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T04:05:36.894 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:05:36.895 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:05:36.895 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T04:05:36.895 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T04:05:36.896 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:05:36.896 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T04:05:36.896 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T04:05:36.897 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T04:05:36.897 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:05:36.897 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T04:05:36.897 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T04:05:37.157 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: debug 2021-07-31T04:05:36.859+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:05:37.157 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:05:37.158 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:05:37.158 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: yield 2021-07-31T04:05:37.158 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:05:37.159 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:05:37.159 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:05:37.159 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:05:37.160 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:05:37.160 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:05:37.160 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:05:37.161 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:05:37.161 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:05:37.161 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: task.result() 2021-07-31T04:05:37.161 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:05:37.162 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: debug 2021-07-31T04:05:36.870+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:05:37.162 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:05:37.162 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:05:37.163 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: yield 2021-07-31T04:05:37.163 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:05:37.163 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:05:37.164 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:05:37.164 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:05:37.164 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:05:37.164 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:05:37.165 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:05:37.165 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:05:37.165 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:05:37.166 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: task.result() 2021-07-31T04:05:37.166 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:05:37.166 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: debug 2021-07-31T04:05:36.896+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:05:37.166 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:05:37.167 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:05:37.167 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: yield 2021-07-31T04:05:37.167 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:05:37.168 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:05:37.168 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:05:37.168 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:05:37.168 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:05:37.169 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:05:37.169 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:05:37.169 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:05:37.170 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:05:37.170 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: task.result() 2021-07-31T04:05:37.170 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:05:37.170 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: debug 2021-07-31T04:05:36.896+0000 7fcb9095b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:05:37.171 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:05:37.171 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:05:37.171 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: yield 2021-07-31T04:05:37.172 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:05:37.172 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:05:37.172 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:05:37.172 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:05:37.173 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:05:37.173 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:05:37.173 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:05:37.173 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:05:37.174 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:05:37.174 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: task.result() 2021-07-31T04:05:37.174 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:05:37.175 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: 2021-07-31T04:05:37.175 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T04:05:37.175 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: 2021-07-31T04:05:37.175 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:05:37.176 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:05:37.176 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: return f(*arg) 2021-07-31T04:05:37.176 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:05:37.176 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:05:37.177 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:05:37.177 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:05:37.177 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:05:37.178 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:05:37.178 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:05:37.178 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T04:05:37.178 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:05:37.179 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: raise self._exception 2021-07-31T04:05:37.179 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:05:37.179 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:05:37.179 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:05:37.180 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T04:05:37.180 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:05:37.180 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: raise 2021-07-31T04:05:37.181 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:05:37.181 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T04:05:37.181 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:05:37.182 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:05:37.182 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:05:37.182 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: debug 2021-07-31T04:05:36.905+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:05:37.182 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:05:37.183 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:05:37.183 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: yield 2021-07-31T04:05:37.183 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:05:37.183 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:05:37.184 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:05:37.184 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:05:37.184 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:05:37.184 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:05:37.185 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:05:37.185 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:05:37.185 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:05:37.186 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: task.result() 2021-07-31T04:05:37.186 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:05:37.186 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: debug 2021-07-31T04:05:36.905+0000 7fcb9115c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:05:37.186 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:05:37.187 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:05:37.187 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: yield 2021-07-31T04:05:37.187 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:05:37.187 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:05:37.188 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:05:37.188 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:05:37.188 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:05:37.189 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:05:37.189 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:05:37.189 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:05:37.189 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:05:37.190 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: task.result() 2021-07-31T04:05:37.190 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:05:37.190 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: 2021-07-31T04:05:37.190 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T04:05:37.191 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: 2021-07-31T04:05:37.191 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:05:37.191 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:05:37.191 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: return f(*arg) 2021-07-31T04:05:37.192 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:05:37.192 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:05:37.192 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:05:37.192 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:05:37.193 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:05:37.193 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:05:37.193 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:05:37.193 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T04:05:37.194 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:05:37.194 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: raise self._exception 2021-07-31T04:05:37.194 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:05:37.195 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:05:37.195 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:05:37.195 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T04:05:37.195 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:05:37.196 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: raise 2021-07-31T04:05:37.196 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:05:37.196 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T04:05:37.196 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:05:37.197 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:05:37.197 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:05:36 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:05:37.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:37 smithi154 conmon[34553]: cluster 2021-07-31T04:05:35.773960+0000 mgr.x (mgr.24542) 5925 : cluster [DBG] pgmap v6159: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:37.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:37 smithi154 conmon[34553]: audit 2021-07-31T04:05:36.266720+0000 mgr.x (mgr.24542) 5926 : audit [DBG] from='client.30811 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:05:37.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:37 smithi154 conmon[34553]: audit 2021-07-31T04:05 2021-07-31T04:05:37.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:37 smithi154 conmon[34553]: :36.815718+0000 mon.b (mon.2) 767 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T04:05:37.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:37 smithi154 conmon[34553]: audit 2021 2021-07-31T04:05:37.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:37 smithi154 conmon[34553]: -07-31T04:05:36.817163+0000 2021-07-31T04:05:37.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:37 smithi154 conmon[34553]: mon.b (mon.2) 768 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T04:05:37.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:37 smithi154 conmon[34553]: audit 2021-07-31T04:05:36.818036+0000 2021-07-31T04:05:37.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:37 smithi154 conmon[34553]: mon.b (mon.2) 769 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T04:05:37.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:37 smithi154 conmon[34553]: audit 2021-07-31T04:05:37.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:37 smithi154 conmon[34553]: 2021-07-31T04:05:36.893178+0000 2021-07-31T04:05:37.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:37 smithi154 conmon[34553]: mon.c (mon.1) 247 : audit [DBG] from='client.? 172.21.15.45:0/1460218512' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:05:37.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:37 smithi045 conmon[39718]: cluster 2021-07-31T04:05 2021-07-31T04:05:37.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:37 smithi045 conmon[39718]: :35.773960+0000 mgr.x (mgr.24542) 5925 : cluster [DBG] pgmap v6159: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:37.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:37 smithi045 conmon[39718]: audit 2021 2021-07-31T04:05:37.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:37 smithi045 conmon[39718]: -07-31T04:05:36 2021-07-31T04:05:37.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:37 smithi045 conmon[39718]: .266720+0000 mgr.x (mgr. 2021-07-31T04:05:37.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:37 smithi045 conmon[39718]: 24542) 5926 : audit [DBG] 2021-07-31T04:05:37.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:37 smithi045 conmon[39718]: from='client.30811 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:05:37.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:37 smithi045 conmon[39718]: audit 2021-07 2021-07-31T04:05:37.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:37 smithi045 conmon[39718]: -31T04:05: 2021-07-31T04:05:37.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:37 smithi045 conmon[39718]: 36.815718+0000 mon.b (mon.2 2021-07-31T04:05:37.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:37 smithi045 conmon[39718]: ) 767 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T04:05:37.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:37 smithi045 conmon[39718]: audit 2021 2021-07-31T04:05:37.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:37 smithi045 conmon[39718]: -07-31T04:05:36 2021-07-31T04:05:37.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:37 smithi045 conmon[39718]: .817163+0000 mon.b (mon. 2021-07-31T04:05:37.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:37 smithi045 conmon[39718]: 2) 768 : audit [DBG] 2021-07-31T04:05:37.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:37 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T04:05:37.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:37 smithi045 conmon[39718]: audit 2021 2021-07-31T04:05:37.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:37 smithi045 conmon[39718]: -07-31T04:05: 2021-07-31T04:05:37.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:37 smithi045 conmon[39718]: 36.818036+0000 mon.b (mon 2021-07-31T04:05:37.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:37 smithi045 conmon[39718]: .2) 769 : audit 2021-07-31T04:05:37.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:37 smithi045 conmon[39718]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T04:05:37.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:37 smithi045 conmon[39718]: audit 2021 2021-07-31T04:05:37.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:37 smithi045 conmon[39718]: -07-31T04:05: 2021-07-31T04:05:37.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:37 smithi045 conmon[39718]: 36.893178+0000 mon.c (mon 2021-07-31T04:05:37.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:37 smithi045 conmon[39718]: .1) 247 : audit [DBG] 2021-07-31T04:05:37.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:37 smithi045 conmon[39718]: from='client.? 172.21.15.45:0/1460218512' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:05:37.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:37 smithi045 conmon[33655]: cluster 2021-07-31T04:05:35.773960+0000 mgr.x (mgr.24542) 5925 : cluster [DBG] 2021-07-31T04:05:37.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:37 smithi045 conmon[33655]: pgmap v6159: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:37.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:37 smithi045 conmon[33655]: audit 2021-07-31T04:05:36.266720+0000 mgr.x (mgr.24542) 5926 2021-07-31T04:05:37.535 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:37 smithi045 conmon[33655]: : audit [DBG] from='client.30811 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:05:37.535 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:37 smithi045 conmon[33655]: audit 2021-07-31T 2021-07-31T04:05:37.535 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:37 smithi045 conmon[33655]: 04:05:36.815718+0000 mon.b (mon 2021-07-31T04:05:37.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:37 smithi045 conmon[33655]: .2) 767 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T04:05:37.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:37 smithi045 conmon[33655]: audit 2021-07-31T04:05: 2021-07-31T04:05:37.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:37 smithi045 conmon[33655]: 36.817163+0000 mon.b (mon.2) 768 : audit 2021-07-31T04:05:37.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:37 smithi045 conmon[33655]: [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T04:05:37.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:37 smithi045 conmon[33655]: audit 2021-07-31T04:05: 2021-07-31T04:05:37.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:37 smithi045 conmon[33655]: 36.818036+0000 mon.b (mon.2) 769 : audit [INF] 2021-07-31T04:05:37.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:37 smithi045 conmon[33655]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T04:05:37.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:37 smithi045 conmon[33655]: audit 2021-07-31T04:05:36.893178+0000 mon.c 2021-07-31T04:05:37.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:37 smithi045 conmon[33655]: (mon.1) 247 : audit [DBG] 2021-07-31T04:05:37.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:37 smithi045 conmon[33655]: from='client.? 172.21.15.45:0/1460218512' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:05:38.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: audit 2021-07-31T04:05:36.549097+0000 mgr.x (mgr.24542) 5927 : audit [DBG] 2021-07-31T04:05:38.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: from='client.20880 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:05:38.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: cephadm 2021-07-31T04:05:36.818758+0000 mgr.x (mgr.24542) 5928 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T04:05:38.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: cephadm 2021-07-31T04:05:36.859994 2021-07-31T04:05:38.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: +0000 mgr.x (mgr.24542) 5929 : cephadm [ERR] 2021-07-31T04:05:38.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:05:38.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:05:38.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: yield 2021-07-31T04:05:38.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:05:38.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:05:38.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:05:38.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:05:38.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:05:38.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:05:38.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:05:38.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:05:38.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:05:38.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: task.result() 2021-07-31T04:05:38.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:05:38.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: cephadm 2021-07-31T04:05:36.860404+0000 2021-07-31T04:05:38.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: mgr.x (mgr.24542) 5930 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T04:05:38.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: cephadm 2021-07-31T04:05:36.871807+0000 mgr.x (mgr.24542) 5931 : cephadm [ERR] 2021-07-31T04:05:38.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:05:38.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:05:38.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: yield 2021-07-31T04:05:38.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:05:38.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:05:38.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:05:38.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:05:38.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:05:38.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:05:38.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:05:38.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:05:38.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:05:38.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: task.result() 2021-07-31T04:05:38.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:05:38.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: cephadm 2021-07-31T04:05:36.872517+0000 mgr.x (mgr.24542) 5932 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T04:05:38.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: cephadm 2021-07-31T04:05:38.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: 2021-07-31T04:05:36.897159+0000 mgr.x (mgr.24542) 5933 : cephadm [ERR] 2021-07-31T04:05:38.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:05:38.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:05:38.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: yield 2021-07-31T04:05:38.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:05:38.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:05:38.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:05:38.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:05:38.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:05:38.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:05:38.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:05:38.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:05:38.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:05:38.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: task.result() 2021-07-31T04:05:38.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:05:38.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: cephadm 2021-07-31T04:05:36.897576+0000 mgr.x (mgr.24542) 5934 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:05:38.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:05:38.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:05:38.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: yield 2021-07-31T04:05:38.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:05:38.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:05:38.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:05:38.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:05:38.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:05:38.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:05:38.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:05:38.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:05:38.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:05:38.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: task.result() 2021-07-31T04:05:38.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:05:38.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: 2021-07-31T04:05:38.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T04:05:38.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: 2021-07-31T04:05:38.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:05:38.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:05:38.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: return f(*arg) 2021-07-31T04:05:38.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:05:38.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:05:38.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:05:38.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:05:38.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:05:38.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:05:38.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:05:38.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T04:05:38.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:05:38.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: raise self._exception 2021-07-31T04:05:38.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:05:38.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:05:38.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:05:38.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T04:05:38.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:05:38.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: raise 2021-07-31T04:05:38.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:05:38.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T04:05:38.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:05:38.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:05:38.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:05:38.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: cephadm 2021-07-31T04:05:36.906131+0000 mgr.x (mgr.24542) 5935 : cephadm [ERR] 2021-07-31T04:05:38.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:05:38.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:05:38.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: yield 2021-07-31T04:05:38.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:05:38.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:05:38.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:05:38.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:05:38.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:05:38.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:05:38.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:05:38.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:05:38.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:05:38.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: task.result() 2021-07-31T04:05:38.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:05:38.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: cephadm 2021-07-31T04:05:36.906541+0000 mgr.x (mgr.24542) 5936 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:05:38.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:05:38.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:05:38.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: yield 2021-07-31T04:05:38.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:05:38.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:05:38.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:05:38.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:05:38.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:05:38.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:05:38.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:05:38.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:05:38.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:05:38.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: task.result() 2021-07-31T04:05:38.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:05:38.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: 2021-07-31T04:05:38.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T04:05:38.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: 2021-07-31T04:05:38.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:05:38.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:05:38.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: return f(*arg) 2021-07-31T04:05:38.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:05:38.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:05:38.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:05:38.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:05:38.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:05:38.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:05:38.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:05:38.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T04:05:38.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:05:38.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: raise self._exception 2021-07-31T04:05:38.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:05:38.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:05:38.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:05:38.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T04:05:38.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:05:38.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: raise 2021-07-31T04:05:38.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:05:38.541 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T04:05:38.541 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:05:38.541 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:05:38.541 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:38 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:05:38.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: audit 2021-07-31T04:05:36.549097+0000 mgr.x 2021-07-31T04:05:38.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: (mgr.24542) 5927 : audit [DBG] from='client.20880 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:05:38.543 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: cephadm 2021-07-31T04:05 2021-07-31T04:05:38.543 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: :36.818758+0000 mgr.x (mgr.24542) 5928 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T04:05:38.543 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: cephadm 2021-07-31T04:05:38.543 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: 2021-07-31T04:05 2021-07-31T04:05:38.544 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: :36.859994+0000 mgr.x (mgr 2021-07-31T04:05:38.544 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: .24542) 5929 : cephadm [ERR] 2021-07-31T04:05:38.544 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:05:38.544 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:05:38.545 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: yield 2021-07-31T04:05:38.545 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:05:38.545 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:05:38.545 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:05:38.546 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:05:38.546 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:05:38.546 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:05:38.546 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:05:38.547 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:05:38.547 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:05:38.547 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: task.result() 2021-07-31T04:05:38.547 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:05:38.548 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: cephadm 2021-07-31T04:05:36.860404+0000 mgr.x (mgr.24542) 5930 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T04:05:38.548 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: cephadm 2021-07-31T04:05:36.871807 2021-07-31T04:05:38.548 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 5931 : cephadm [ERR] 2021-07-31T04:05:38.548 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:05:38.549 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:05:38.549 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: yield 2021-07-31T04:05:38.549 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:05:38.549 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:05:38.550 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:05:38.550 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:05:38.550 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:05:38.550 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:05:38.551 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:05:38.551 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:05:38.551 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:05:38.551 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: task.result() 2021-07-31T04:05:38.551 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:05:38.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: audit 2021-07-31T04:05:36.549097+0000 mgr.x ( 2021-07-31T04:05:38.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: mgr.24542) 5927 : audit [DBG] from='client.20880 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:05:38.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: cephadm 2021-07-31T04: 2021-07-31T04:05:38.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: 05:36.818758+0000 mgr.x (mgr.24542) 5928 : cephadm [WRN] 2021-07-31T04:05:38.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T04:05:38.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: cephadm 2021-07-31T04 2021-07-31T04:05:38.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: :05:36.859994+0000 mgr.x (mgr 2021-07-31T04:05:38.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: .24542) 5929 : cephadm [ERR] 2021-07-31T04:05:38.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:05:38.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:05:38.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: yield 2021-07-31T04:05:38.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:05:38.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:05:38.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:05:38.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:05:38.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:05:38.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:05:38.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:05:38.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:05:38.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:05:38.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: task.result() 2021-07-31T04:05:38.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:05:38.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: 2021-07-31T04:05:38.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: cephadm 2021-07-31T04:05:36.860404+0000 mgr.x (mgr.24542) 5930 : cephadm [INF] 2021-07-31T04:05:38.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T04:05:38.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: cephadm 2021-07-31 2021-07-31T04:05:38.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: T04:05:36.871807+0000 mgr.x 2021-07-31T04:05:38.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: (mgr.24542) 5931 : cephadm [ERR] 2021-07-31T04:05:38.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:05:38.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:05:38.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: yield 2021-07-31T04:05:38.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:05:38.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:05:38.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:05:38.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:05:38.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:05:38.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:05:38.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:05:38.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:05:38.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:05:38.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: task.result() 2021-07-31T04:05:38.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: cephadm 2021-07-31T04:05:36.872517+0000 mgr.x (mgr.24542) 5932 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T04:05:38.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: cephadm 2021-07-31T04:05:38.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: 2021-07-31T04:05:36.897159+0000 2021-07-31T04:05:38.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: mgr.x (mgr.24542) 5933 : cephadm [ERR] 2021-07-31T04:05:38.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: 2021-07-31T04:05:38.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:05:38.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:05:38.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: yield 2021-07-31T04:05:38.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:05:38.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:05:38.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:05:38.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:05:38.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:05:38.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:05:38.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:05:38.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:05:38.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:05:38.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: task.result() 2021-07-31T04:05:38.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:05:38.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: cephadm 2021-07-31T04:05:36.897576+0000 mgr.x (mgr.24542) 5934 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:05:38.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:05:38.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:05:38.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: yield 2021-07-31T04:05:38.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:05:38.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:05:38.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:05:38.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:05:38.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:05:38.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:05:38.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:05:38.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:05:38.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:05:38.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: task.result() 2021-07-31T04:05:38.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:05:38.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: 2021-07-31T04:05:38.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T04:05:38.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: 2021-07-31T04:05:38.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:05:38.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:05:38.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: return f(*arg) 2021-07-31T04:05:38.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:05:38.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:05:38.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:05:38.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:05:38.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: 2021-07-31T04:05:38.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: cephadm 2021-07-31T04:05:36.872517+0000 mgr.x (mgr.24542) 5932 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T04:05:38.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: cephadm 2021-07-31T04:05:38.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: 2021-07-31T04:05:36.897159 2021-07-31T04:05:38.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 2021-07-31T04:05:38.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: 5933 : cephadm [ERR] 2021-07-31T04:05:38.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:05:38.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:05:38.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: yield 2021-07-31T04:05:38.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:05:38.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:05:38.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:05:38.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:05:38.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:05:38.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:05:38.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:05:38.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:05:38.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:05:38.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: task.result() 2021-07-31T04:05:38.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:05:38.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: cephadm 2021-07-31T04:05:36.897576+0000 mgr.x (mgr 2021-07-31T04:05:38.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: .24542) 5934 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:05:38.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:05:38.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:05:38.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: yield 2021-07-31T04:05:38.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:05:38.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:05:38.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:05:38.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:05:38.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:05:38.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:05:38.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:05:38.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:05:38.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:05:38.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: task.result() 2021-07-31T04:05:38.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:05:38.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: 2021-07-31T04:05:38.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T04:05:38.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: 2021-07-31T04:05:38.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:05:38.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:05:38.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: return f(*arg) 2021-07-31T04:05:38.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:05:38.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:05:38.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:05:38.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:05:38.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:05:38.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T04:05:38.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:05:38.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: raise self._exception 2021-07-31T04:05:38.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:05:38.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:05:38.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:05:38.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T04:05:38.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:05:38.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: raise 2021-07-31T04:05:38.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:05:38.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T04:05:38.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:05:38.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:05:38.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:05:38.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: cephadm 2021-07-31T04:05:36.906131+0000 mgr.x (mgr.24542) 5935 : cephadm [ERR] 2021-07-31T04:05:38.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:05:38.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:05:38.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: yield 2021-07-31T04:05:38.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:05:38.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:05:38.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:05:38.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:05:38.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:05:38.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:05:38.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:05:38.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:05:38.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:05:38.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: task.result() 2021-07-31T04:05:38.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:05:38.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: cephadm 2021-07-31T04:05:36.906541+0000 mgr.x (mgr.24542) 5936 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:05:38.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:05:38.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:05:38.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: yield 2021-07-31T04:05:38.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:05:38.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:05:38.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:05:38.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:05:38.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:05:38.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:05:38.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:05:38.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T04:05:38.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:05:38.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: raise self._exception 2021-07-31T04:05:38.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:05:38.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:05:38.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:05:38.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T04:05:38.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:05:38.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: raise 2021-07-31T04:05:38.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:05:38.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T04:05:38.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:05:38.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:05:38.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:05:38.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: cephadm 2021-07-31T04:05:36.906131+0000 mgr.x (mgr.24542) 5935 : cephadm [ERR] 2021-07-31T04:05:38.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:05:38.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:05:38.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: yield 2021-07-31T04:05:38.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:05:38.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:05:38.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:05:38.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:05:38.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:05:38.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:05:38.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:05:38.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:05:38.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:05:38.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: task.result() 2021-07-31T04:05:38.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:05:38.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: cephadm 2021-07-31T04:05:36.906541+0000 mgr.x (mgr.24542) 5936 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:05:38.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:05:38.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:05:38.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: yield 2021-07-31T04:05:38.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:05:38.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:05:38.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:05:38.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:05:38.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:05:38.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:05:38.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:05:38.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:05:38.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: task.result() 2021-07-31T04:05:38.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:05:38.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: 2021-07-31T04:05:38.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T04:05:38.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: 2021-07-31T04:05:38.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:05:38.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:05:38.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: return f(*arg) 2021-07-31T04:05:38.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:05:38.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:05:38.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:05:38.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:05:38.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:05:38.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:05:38.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:05:38.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T04:05:38.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:05:38.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: raise self._exception 2021-07-31T04:05:38.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:05:38.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:05:38.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:05:38.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T04:05:38.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:05:38.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: raise 2021-07-31T04:05:38.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:05:38.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T04:05:38.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:05:38.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:05:38.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:05:38.606 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:05:38.606 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:05:38.607 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:05:38.607 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:05:38.607 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:05:38.607 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:05:38.607 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:05:38.608 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:05:38.608 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:05:38.608 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: task.result() 2021-07-31T04:05:38.608 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:05:38.608 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: 2021-07-31T04:05:38.609 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T04:05:38.609 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: 2021-07-31T04:05:38.609 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:05:38.609 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:05:38.609 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: return f(*arg) 2021-07-31T04:05:38.610 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:05:38.610 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:05:38.610 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:05:38.610 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:05:38.610 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:05:38.611 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:05:38.611 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:05:38.611 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T04:05:38.611 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:05:38.611 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: raise self._exception 2021-07-31T04:05:38.612 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:05:38.612 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:05:38.612 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:05:38.612 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T04:05:38.612 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:05:38.613 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: raise 2021-07-31T04:05:38.613 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:05:38.613 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T04:05:38.613 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:05:38.614 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:05:38.614 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:38 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:05:39.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:39 smithi154 conmon[34553]: cluster 2021-07-31T04:05:37.774406+0000 mgr.x (mgr.24542) 5937 : cluster [DBG] pgmap v6160: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:39.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:39 smithi045 conmon[33655]: cluster 2021-07-31T04:05:37.774406+0000 mgr.x (mgr.24542) 2021-07-31T04:05:39.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:39 smithi045 conmon[33655]: 5937 : cluster [DBG] pgmap v6160: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:39.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:39 smithi045 conmon[39718]: cluster 2021-07-31T04:05:37.774406+0000 mgr.x 2021-07-31T04:05:39.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:39 smithi045 conmon[39718]: (mgr.24542) 5937 : cluster [DBG] pgmap v6160: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:41.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:41 smithi154 conmon[34553]: cluster 2021-07-31T04:05:39.774991+0000 mgr.x (mgr. 2021-07-31T04:05:41.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:41 smithi154 conmon[34553]: 24542) 5938 : cluster [DBG] pgmap v6161: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:41.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:41 smithi045 conmon[39718]: cluster 2021-07-31T04:05:39.774991+0000 mgr.x (mgr 2021-07-31T04:05:41.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:41 smithi045 conmon[39718]: .24542) 5938 : cluster [DBG] pgmap v6161: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:41.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:41 smithi045 conmon[33655]: cluster 2021-07-31T04:05:39.774991+0000 mgr.x (mgr.24542) 5938 : 2021-07-31T04:05:41.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:41 smithi045 conmon[33655]: cluster [DBG] pgmap v6161: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:43.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:43 smithi154 conmon[34553]: cluster 2021-07-31T04:05:41.775369+0000 mgr.x (mgr.24542) 5939 : cluster [DBG] pgmap v6162: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:43.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:43 smithi045 conmon[33655]: cluster 2021-07-31T04:05:41.775369+0000 mgr.x (mgr.24542) 5939 2021-07-31T04:05:43.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:43 smithi045 conmon[33655]: : cluster [DBG] pgmap v6162: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:43.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:43 smithi045 conmon[39718]: cluster 2021-07-31T04:05:41.775369+0000 mgr.x (mgr 2021-07-31T04:05:43.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:43 smithi045 conmon[39718]: .24542) 5939 : cluster [DBG] pgmap v6162: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:45.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:45 smithi154 conmon[34553]: cluster 2021-07-31T04:05:43.775808+0000 mgr.x (mgr.24542) 5940 : cluster [DBG] pgmap v6163: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:45.524 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:45 smithi045 conmon[39718]: cluster 2021-07-31T04:05:43.775808+0000 mgr.x (mgr 2021-07-31T04:05:45.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:45 smithi045 conmon[39718]: .24542) 5940 : cluster [DBG] pgmap v6163: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:45.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:45 smithi045 conmon[33655]: cluster 2021-07-31T04:05:43.775808+0000 mgr.x (mgr.24542) 5940 : cluster [DBG] pgmap v6163: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:47.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:47 smithi154 conmon[34553]: cluster 2021-07-31T04:05:45.776395+0000 mgr.x (mgr.24542) 5941 : cluster [DBG] pgmap v6164: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:47.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:47 smithi045 conmon[33655]: cluster 2021-07-31T04:05:45.776395+0000 mgr.x (mgr.24542) 2021-07-31T04:05:47.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:47 smithi045 conmon[33655]: 5941 : cluster [DBG] pgmap v6164: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:47.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:47 smithi045 conmon[39718]: cluster 2021-07-31T04:05:45.776395+0000 mgr.x (mgr.24542) 5941 : cluster [DBG] pgmap v6164: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:49.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:49 smithi154 conmon[34553]: cluster 2021-07-31T04:05:47.776798+0000 mgr.x (mgr.24542) 5942 : cluster [DBG] pgmap v6165: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:49.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:49 smithi045 conmon[33655]: cluster 2021-07-31T04:05:47.776798+0000 mgr.x (mgr.24542) 5942 2021-07-31T04:05:49.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:49 smithi045 conmon[33655]: : cluster [DBG] pgmap v6165: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:49.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:49 smithi045 conmon[39718]: cluster 2021-07-31T04:05:47.776798+0000 mgr.x (mgr. 2021-07-31T04:05:49.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:49 smithi045 conmon[39718]: 24542) 5942 : cluster [DBG] pgmap v6165: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:51.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:51 smithi154 conmon[34553]: cluster 2021-07-31T04:05:49.777230+0000 mgr.x (mgr.24542) 5943 : cluster [DBG] pgmap v6166: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:51.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:51 smithi045 conmon[33655]: cluster 2021-07-31T04:05:49.777230+0000 mgr.x (mgr.24542) 5943 : cluster [DBG] pgmap v6166: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:51.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:51 smithi045 conmon[33655]: 2021-07-31T04:05:51.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:51 smithi045 conmon[39718]: cluster 2021-07-31T04:05:49.777230+0000 mgr.x (mgr. 2021-07-31T04:05:51.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:51 smithi045 conmon[39718]: 24542) 5943 : cluster [DBG] pgmap v6166: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:53.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:53 smithi154 conmon[34553]: cluster 2021-07-31T04:05:51.777723+0000 mgr.x (mgr.24542) 5944 : cluster [DBG] pgmap v6167: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:53.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:53 smithi045 conmon[33655]: cluster 2021-07-31T04:05:51.777723+0000 mgr.x (mgr.24542) 5944 : 2021-07-31T04:05:53.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:53 smithi045 conmon[33655]: cluster [DBG] pgmap v6167: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:53.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:53 smithi045 conmon[39718]: cluster 2021-07-31T04:05:51.777723+0000 mgr.x ( 2021-07-31T04:05:53.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:53 smithi045 conmon[39718]: mgr.24542) 5944 : cluster [DBG] pgmap v6167: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:55.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:55 smithi154 conmon[34553]: cluster 2021-07-31T04:05:53.778266+0000 mgr.x (mgr.24542) 5945 : cluster [DBG] pgmap v6168: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:55.524 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:55 smithi045 conmon[39718]: cluster 2021-07-31T04:05:53.778266+0000 mgr.x 2021-07-31T04:05:55.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:55 smithi045 conmon[39718]: (mgr.24542) 5945 : cluster [DBG] pgmap v6168: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:55.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:55 smithi045 conmon[33655]: cluster 2021-07-31T04:05:53.778266+0000 mgr.x 2021-07-31T04:05:55.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:55 smithi045 conmon[33655]: (mgr.24542) 5945 : cluster [DBG] pgmap v6168: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:57.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:57 smithi154 conmon[34553]: cluster 2021-07-31T04:05:55.778734+0000 mgr.x (mgr.24542) 5946 : cluster [DBG] pgmap v6169: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:57.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:57 smithi045 conmon[33655]: cluster 2021-07-31T04:05:55.778734+0000 mgr.x (mgr.24542) 2021-07-31T04:05:57.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:57 smithi045 conmon[33655]: 5946 : cluster [DBG] pgmap v6169: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:57.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:57 smithi045 conmon[39718]: cluster 2021-07-31T04:05:55.778734+0000 mgr.x (mgr.24542) 5946 : cluster [DBG] pgmap v6169: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:59.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:05:59 smithi154 conmon[34553]: cluster 2021-07-31T04:05:57.779195+0000 mgr.x (mgr.24542) 5947 : cluster [DBG] pgmap v6170: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:59.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:59 smithi045 conmon[39718]: cluster 2021-07-31T04:05:57.779195 2021-07-31T04:05:59.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:05:59 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 5947 : cluster [DBG] pgmap v6170: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:05:59.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:59 smithi045 conmon[33655]: cluster 2021-07-31T04:05:57.779195+0000 mgr.x (mgr.24542) 5947 : cluster [DBG] 2021-07-31T04:05:59.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:05:59 smithi045 conmon[33655]: pgmap v6170: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:01.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:01 smithi154 conmon[34553]: cluster 2021-07-31T04:05:59.779644+0000 mgr.x (mgr.24542) 5948 : cluster [DBG] pgmap v6171: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:01.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:01 smithi045 conmon[33655]: cluster 2021-07-31T04:05:59.779644+0000 mgr.x (mgr.24542) 5948 2021-07-31T04:06:01.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:01 smithi045 conmon[33655]: : cluster [DBG] pgmap v6171: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:01.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:01 smithi045 conmon[39718]: cluster 2021-07-31T04:05:59.779644 2021-07-31T04:06:01.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:01 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 5948 : cluster [DBG] pgmap v6171: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:03.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:03 smithi154 conmon[34553]: cluster 2021-07-31T04:06:01.780066+0000 mgr.x (mgr.24542) 5949 : cluster [DBG] pgmap v6172: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:03.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:03 smithi045 conmon[39718]: cluster 2021-07-31T04:06:01 2021-07-31T04:06:03.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:03 smithi045 conmon[39718]: .780066+0000 mgr.x (mgr.24542) 5949 : cluster [DBG] pgmap v6172: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:03.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:03 smithi045 conmon[33655]: cluster 2021-07-31T04:06:01.780066+0000 mgr.x (mgr.24542) 2021-07-31T04:06:03.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:03 smithi045 conmon[33655]: 5949 : cluster [DBG] pgmap v6172: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:05.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:05 smithi154 conmon[34553]: cluster 2021-07-31T04:06:03.780586+0000 mgr.x (mgr.24542) 5950 : cluster [DBG] pgmap v6173: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:05.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:05 smithi045 conmon[33655]: cluster 2021-07-31T04:06:03.780586+0000 mgr.x (mgr.24542) 5950 : cluster [DBG] pgmap v6173: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:05.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:05 smithi045 conmon[39718]: cluster 2021-07-31T04:06:03. 2021-07-31T04:06:05.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:05 smithi045 conmon[39718]: 780586+0000 mgr.x (mgr.24542) 5950 : cluster [DBG] pgmap v6173: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:07.206 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T04:06:07.484 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T04:06:07.484 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 2h ago 2h - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T04:06:07.484 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 2h ago 2h 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T04:06:07.485 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 2h ago 2h 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T04:06:07.485 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 2h ago 2h - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T04:06:07.485 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T04:06:07.485 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 2h ago 2h 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T04:06:07.485 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T04:06:07.485 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 2h ago 2h - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T04:06:07.486 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 2h ago 2h 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T04:06:07.486 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T04:06:07.486 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T04:06:07.486 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T04:06:07.486 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T04:06:07.487 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 2h ago 2h 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T04:06:07.487 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 2h ago 2h 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T04:06:07.487 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 2h ago 2h 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T04:06:07.487 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 2h ago 2h 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T04:06:07.487 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 2h ago 2h 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T04:06:07.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:07 smithi154 conmon[34553]: cluster 2021-07-31T04:06:05.781126+0000 mgr.x (mgr.24542) 5951 : cluster [DBG] pgmap v6174: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:07.524 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:07 smithi045 conmon[33655]: cluster 2021-07-31T04:06:05.781126 2021-07-31T04:06:07.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:07 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 5951 : cluster [DBG] pgmap v6174: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:07.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:07 smithi045 conmon[39718]: cluster 2021-07-31T04:06:05.781126 2021-07-31T04:06:07.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:07 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 5951 : cluster [DBG] pgmap v6174: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:07.817 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T04:06:07.818 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T04:06:07.818 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T04:06:07.818 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:06:07.818 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T04:06:07.819 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T04:06:07.819 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:06:07.819 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:06:07.819 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T04:06:07.819 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T04:06:07.820 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:06:07.820 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T04:06:07.820 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T04:06:07.820 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T04:06:07.820 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:06:07.821 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T04:06:07.821 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T04:06:08.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:08 smithi154 conmon[34553]: audit 2021-07-31T04:06:07.194998+0000 mgr.x (mgr.24542) 5952 : audit [DBG] from='client.30829 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:06:08.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:08 smithi154 conmon[34553]: audit 2021-07-31T04:06:07.818834 2021-07-31T04:06:08.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:08 smithi154 conmon[34553]: +0000 mon.c (mon.1) 248 : audit [DBG] from='client.? 172.21.15.45:0/2109627063' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:06:08.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:08 smithi045 conmon[33655]: audit 2021-07-31T04:06:07.194998+0000 mgr.x (mgr.24542 2021-07-31T04:06:08.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:08 smithi045 conmon[33655]: ) 5952 : audit [DBG] from='client.30829 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:06:08.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:08 smithi045 conmon[33655]: audit 2021-07-31T04:06:07.818834+0000 mon.c (mon.1) 248 : audit [DBG] from='client.? 172.21.15.45:0/2109627063' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:06:08.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:08 smithi045 conmon[39718]: audit 2021-07-31T04:06:07.194998+0000 mgr.x (mgr.24542 2021-07-31T04:06:08.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:08 smithi045 conmon[39718]: ) 5952 : audit [DBG] from='client.30829 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:06:08.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:08 smithi045 conmon[39718]: audit 2021-07-31T04:06:07.818834+0000 2021-07-31T04:06:08.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:08 smithi045 conmon[39718]: mon.c (mon.1) 248 : audit [DBG] from='client.? 172.21.15.45:0/2109627063' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:06:09.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:09 smithi154 conmon[34553]: audit 2021-07-31T04:06:07.476819+0000 mgr.x (mgr.24542) 5953 : audit [DBG] from='client.30835 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:06:09.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:09 smithi154 conmon[34553]: cluster 2021-07-31T04:06:07.781621+0000 mgr.x (mgr.24542) 5954 : cluster [DBG] pgmap v6175: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:09.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:09 smithi045 conmon[39718]: audit 2021-07-31T04:06:07.476819+0000 mgr.x (mgr 2021-07-31T04:06:09.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:09 smithi045 conmon[39718]: .24542) 5953 : audit [DBG] from='client.30835 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:06:09.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:09 smithi045 conmon[39718]: cluster 2021-07-31T04:06:07.781621+0000 mgr.x (mgr.24542) 2021-07-31T04:06:09.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:09 smithi045 conmon[39718]: 5954 : cluster [DBG] pgmap v6175: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:09.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:09 smithi045 conmon[33655]: audit 2021-07-31T04:06:07.476819+0000 mgr.x (mgr.24542) 5953 : 2021-07-31T04:06:09.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:09 smithi045 conmon[33655]: audit [DBG] from='client.30835 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:06:09.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:09 smithi045 conmon[33655]: cluster 2021-07-31T04:06:07.781621+0000 mgr.x (mgr.24542) 5954 : cluster [DBG] pgmap v6175: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:11.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:11 smithi154 conmon[34553]: cluster 2021-07-31T04:06:09.782186+0000 mgr.x (mgr.24542) 5955 : cluster [DBG] pgmap v6176: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:11.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:11 smithi045 conmon[39718]: cluster 2021-07-31T04:06:09.782186+0000 mgr.x (mgr. 2021-07-31T04:06:11.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:11 smithi045 conmon[39718]: 24542) 5955 : cluster [DBG] pgmap v6176: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:11.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:11 smithi045 conmon[33655]: cluster 2021-07-31T04:06:09.782186+0000 mgr.x (mgr.24542) 5955 : cluster 2021-07-31T04:06:11.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:11 smithi045 conmon[33655]: [DBG] pgmap v6176: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:13.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:13 smithi154 conmon[34553]: cluster 2021-07-31T04:06:11.782731+0000 mgr.x (mgr.24542) 5956 : cluster [DBG] pgmap v6177: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:13.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:13 smithi045 conmon[39718]: cluster 2021-07-31T04:06:11.782731+0000 mgr.x (mgr.24542) 5956 : cluster [DBG] pgmap v6177: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:13.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:13 smithi045 conmon[33655]: cluster 2021-07-31T04:06:11.782731+0000 mgr.x (mgr.24542) 5956 : cluster [DBG] pgmap v6177: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:15.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:15 smithi154 conmon[34553]: cluster 2021-07-31T04:06:13.783145+0000 mgr.x (mgr.24542) 5957 : cluster [DBG] pgmap v6178: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:15.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:15 smithi045 conmon[39718]: cluster 2021-07-31T04:06:13.783145+0000 mgr.x (mgr.24542 2021-07-31T04:06:15.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:15 smithi045 conmon[39718]: ) 5957 : cluster [DBG] pgmap v6178: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:15.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:15 smithi045 conmon[33655]: cluster 2021-07-31T04:06:13.783145+0000 mgr.x (mgr.24542) 5957 : cluster [DBG] 2021-07-31T04:06:15.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:15 smithi045 conmon[33655]: pgmap v6178: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:17.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:17 smithi154 conmon[34553]: cluster 2021-07-31T04:06:15.783721+0000 mgr.x (mgr.24542) 5958 : cluster [DBG] pgmap v6179: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:17.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:17 smithi154 conmon[34553]: 2021-07-31T04:06:17.524 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:17 smithi045 conmon[33655]: cluster 2021-07-31T04:06:15.783721+0000 mgr.x (mgr.24542) 5958 : cluster 2021-07-31T04:06:17.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:17 smithi045 conmon[33655]: [DBG] pgmap v6179: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:17.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:17 smithi045 conmon[39718]: cluster 2021-07-31T04:06:15.783721+0000 mgr.x (mgr. 2021-07-31T04:06:17.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:17 smithi045 conmon[39718]: 24542) 5958 : cluster [DBG] pgmap v6179: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:19.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:19 smithi154 conmon[34553]: cluster 2021-07-31T04:06:17.784129+0000 mgr.x (mgr.24542) 5959 : cluster [DBG] pgmap v6180: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:19.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:19 smithi154 conmon[34553]: 2021-07-31T04:06:19.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:19 smithi045 conmon[33655]: cluster 2021-07-31T04:06:17.784129 2021-07-31T04:06:19.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:19 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 5959 : cluster [DBG] pgmap v6180: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:19.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:19 smithi045 conmon[39718]: cluster 2021-07-31T04:06:17.784129+0000 mgr.x 2021-07-31T04:06:19.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:19 smithi045 conmon[39718]: (mgr.24542) 5959 : cluster [DBG] pgmap v6180: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:21.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:21 smithi154 conmon[34553]: cluster 2021-07-31T04:06:19.784615+0000 mgr.x (mgr.24542) 5960 : cluster [DBG] pgmap v6181: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:21.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:21 smithi045 conmon[33655]: cluster 2021-07-31T04:06:19.784615+0000 2021-07-31T04:06:21.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:21 smithi045 conmon[33655]: mgr.x (mgr.24542) 5960 : cluster [DBG] pgmap v6181: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:21.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:21 smithi045 conmon[39718]: cluster 2021-07-31T04:06:19. 2021-07-31T04:06:21.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:21 smithi045 conmon[39718]: 784615+0000 mgr.x (mgr.24542) 5960 : cluster [DBG] pgmap v6181: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:23.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:23 smithi154 conmon[34553]: cluster 2021-07-31T04:06:21.785052+0000 mgr.x (mgr.24542) 5961 : cluster [DBG] pgmap v6182: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:23.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:23 smithi045 conmon[39718]: cluster 2021-07-31T04:06:21.785052+0000 mgr.x (mgr. 2021-07-31T04:06:23.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:23 smithi045 conmon[39718]: 24542) 5961 : cluster [DBG] pgmap v6182: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:23.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:23 smithi045 conmon[33655]: cluster 2021-07-31T04:06:21.785052 2021-07-31T04:06:23.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:23 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 5961 : cluster [DBG] pgmap v6182: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:25.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:25 smithi154 conmon[34553]: cluster 2021-07-31T04:06:23.785465+0000 mgr.x (mgr.24542) 5962 : cluster [DBG] pgmap v6183: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:25.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:25 smithi045 conmon[39718]: cluster 2021-07-31T04:06:23.785465+0000 mgr.x (mgr.24542) 5962 : 2021-07-31T04:06:25.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:25 smithi045 conmon[39718]: cluster [DBG] pgmap v6183: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:25.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:25 smithi045 conmon[33655]: cluster 2021-07-31T04:06:23.785465+0000 mgr.x (mgr.24542) 2021-07-31T04:06:25.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:25 smithi045 conmon[33655]: 5962 : cluster [DBG] pgmap v6183: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:27.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:27 smithi045 conmon[33655]: cluster 2021-07-31T04:06:25.786052+0000 mgr.x (mgr 2021-07-31T04:06:27.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:27 smithi045 conmon[33655]: .24542) 5963 : cluster [DBG] pgmap v6184: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:27.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:27 smithi045 conmon[39718]: cluster 2021-07-31T04:06:25.786052+0000 mgr.x 2021-07-31T04:06:27.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:27 smithi045 conmon[39718]: (mgr.24542) 5963 : cluster [DBG] pgmap v6184: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:27.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:27 smithi154 conmon[34553]: cluster 2021-07-31T04:06:25.786052+0000 mgr.x (mgr.24542) 5963 : cluster [DBG] pgmap v6184: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:29.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:29 smithi045 conmon[39718]: cluster 2021-07-31T04:06: 2021-07-31T04:06:29.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:29 smithi045 conmon[39718]: 27.786489+0000 mgr.x (mgr.24542) 5964 : cluster [DBG] pgmap v6185: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:29.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:29 smithi045 conmon[39718]: audit 2021 2021-07-31T04:06:29.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:29 smithi045 conmon[39718]: -07-31T04:06: 2021-07-31T04:06:29.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:29 smithi045 conmon[39718]: 28.484302+0000 mon.b (mon 2021-07-31T04:06:29.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:29 smithi045 conmon[39718]: .2) 770 : audit [INF] 2021-07-31T04:06:29.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:29 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:06:29.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:29 smithi045 conmon[39718]: audit 2021- 2021-07-31T04:06:29.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:29 smithi045 conmon[39718]: 07-31T04:06:28.484756 2021-07-31T04:06:29.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:29 smithi045 conmon[39718]: +0000 mon.b (mon.2) 771 2021-07-31T04:06:29.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:29 smithi045 conmon[39718]: : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:06:29.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:29 smithi045 conmon[39718]: audit 2021- 2021-07-31T04:06:29.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:29 smithi045 conmon[39718]: 07-31T04:06:28. 2021-07-31T04:06:29.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:29 smithi045 conmon[39718]: 485357+0000 mon.a (mon.0 2021-07-31T04:06:29.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:29 smithi045 conmon[39718]: ) 3422 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:06:29.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:29 smithi045 conmon[39718]: 2021-07-31T04:06:29.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:29 smithi045 conmon[39718]: audit 2021- 2021-07-31T04:06:29.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:29 smithi045 conmon[39718]: 07-31T04:06:28 2021-07-31T04:06:29.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:29 smithi045 conmon[39718]: .485679+0000 mon.a (mon. 2021-07-31T04:06:29.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:29 smithi045 conmon[39718]: 0) 3423 : audit [INF] 2021-07-31T04:06:29.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:29 smithi045 conmon[39718]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:06:29.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:29 smithi045 conmon[33655]: cluster 2021-07-31T04:06:27.786489+0000 mgr.x (mgr 2021-07-31T04:06:29.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:29 smithi045 conmon[33655]: .24542) 5964 : cluster [DBG] pgmap v6185: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:29.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:29 smithi045 conmon[33655]: audit 2021-07-31T04:06:28.484302+0000 mon.b (mon.2) 770 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:06:29.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:29 smithi045 conmon[33655]: 2021-07-31T04:06:29.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:29 smithi045 conmon[33655]: audit 2021-07-31 2021-07-31T04:06:29.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:29 smithi045 conmon[33655]: T04:06:28.484756+0000 mon.b (mon.2) 771 : audit [INF] 2021-07-31T04:06:29.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:29 smithi045 conmon[33655]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:06:29.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:29 smithi045 conmon[33655]: audit 2021-07-31T04:06:29.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:29 smithi045 conmon[33655]: 2021-07-31T04:06:28.485357+0000 2021-07-31T04:06:29.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:29 smithi045 conmon[33655]: mon.a (mon.0) 3422 2021-07-31T04:06:29.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:29 smithi045 conmon[33655]: : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:06:29.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:29 smithi045 conmon[33655]: audit 2021-07-31T04:06 2021-07-31T04:06:29.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:29 smithi045 conmon[33655]: :28.485679+0000 mon.a (mon.0) 3423 : audit [INF] 2021-07-31T04:06:29.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:29 smithi045 conmon[33655]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:06:29.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:29 smithi154 conmon[34553]: cluster 2021-07-31T04:06:27.786489+0000 mgr.x (mgr.24542) 5964 : cluster [DBG] pgmap v6185: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:29.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:29 smithi154 conmon[34553]: audit 2021-07-31T04:06:28.484302+0000 mon.b (mon.2) 770 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:06:29.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:29 smithi154 conmon[34553]: audit 2021-07-31T04:06:28.484756+0000 mon.b (mon.2) 771 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:06:29.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:29 smithi154 conmon[34553]: audit 2021-07-31T04:06:29.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:29 smithi154 conmon[34553]: 2021-07-31T04:06:28.485357+0000 mon.a (mon.0) 3422 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:06:29.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:29 smithi154 conmon[34553]: audit 2021 2021-07-31T04:06:29.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:29 smithi154 conmon[34553]: -07-31T04:06:28.485679+0000 2021-07-31T04:06:29.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:29 smithi154 conmon[34553]: mon.a (mon.0) 3423 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:06:31.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:31 smithi045 conmon[39718]: cluster 2021-07-31T04:06: 2021-07-31T04:06:31.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:31 smithi045 conmon[39718]: 29.786912+0000 mgr.x (mgr.24542) 5965 : cluster [DBG] pgmap v6186: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:31.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:31 smithi045 conmon[33655]: cluster 2021-07-31T04:06:29.786912+0000 mgr.x (mgr.24542) 5965 : 2021-07-31T04:06:31.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:31 smithi045 conmon[33655]: cluster [DBG] pgmap v6186: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:31.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:31 smithi154 conmon[34553]: cluster 2021-07-31T04:06:29.786912+0000 mgr.x (mgr.24542) 5965 : cluster [DBG] pgmap v6186: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:31.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:31 smithi154 conmon[34553]: 2021-07-31T04:06:33.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:33 smithi045 conmon[39718]: cluster 2021-07-31T04:06:31.787446+0000 mgr.x ( 2021-07-31T04:06:33.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:33 smithi045 conmon[39718]: mgr.24542) 5966 : cluster [DBG] pgmap v6187: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:33.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:33 smithi045 conmon[33655]: cluster 2021-07-31T04:06:31.787446+0000 mgr.x (mgr. 2021-07-31T04:06:33.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:33 smithi045 conmon[33655]: 24542) 5966 : cluster [DBG] pgmap v6187: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:33.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:33 smithi154 conmon[34553]: cluster 2021-07-31T04:06:31.787446+0000 mgr.x (mgr.24542) 5966 : cluster [DBG] pgmap v6187: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:35.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:35 smithi045 conmon[39718]: cluster 2021-07-31T04:06:33.787930+0000 mgr.x ( 2021-07-31T04:06:35.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:35 smithi045 conmon[39718]: mgr.24542) 5967 : cluster [DBG] pgmap v6188: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:35.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:35 smithi045 conmon[33655]: cluster 2021-07-31T04:06:33.787930+0000 mgr.x (mgr.24542) 5967 : cluster [DBG] 2021-07-31T04:06:35.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:35 smithi045 conmon[33655]: pgmap v6188: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:35.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:35 smithi154 conmon[34553]: cluster 2021-07-31T04:06:33.787930+0000 mgr.x (mgr.24542) 5967 : cluster [DBG] pgmap v6188: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:37.253 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: debug 2021-07-31T04:06:36.952+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:06:37.253 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:06:37.253 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:06:37.253 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: yield 2021-07-31T04:06:37.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:06:37.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:06:37.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:06:37.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:06:37.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:06:37.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:06:37.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:06:37.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:06:37.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:06:37.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: task.result() 2021-07-31T04:06:37.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:06:37.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: debug 2021-07-31T04:06:36.964+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:06:37.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:06:37.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:06:37.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: yield 2021-07-31T04:06:37.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:06:37.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:06:37.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:06:37.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:06:37.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:06:37.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:06:37.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:06:37.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:06:37.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:06:37.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: task.result() 2021-07-31T04:06:37.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:06:37.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: debug 2021-07-31T04:06:36.989+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:06:37.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:06:37.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:06:37.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: yield 2021-07-31T04:06:37.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:06:37.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:06:37.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:06:37.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:06:37.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:06:37.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:06:37.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:06:37.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:06:37.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:06:37.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: task.result() 2021-07-31T04:06:37.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:06:37.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: debug 2021-07-31T04:06:36.990+0000 7fcb97969700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:06:37.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:06:37.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:06:37.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: yield 2021-07-31T04:06:37.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:06:37.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:06:37.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:06:37.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:06:37.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:06:37.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:06:37.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:06:37.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:06:37.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:06:37.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: task.result() 2021-07-31T04:06:37.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:06:37.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: 2021-07-31T04:06:37.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T04:06:37.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: 2021-07-31T04:06:37.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:06:37.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:06:37.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: return f(*arg) 2021-07-31T04:06:37.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:06:37.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:06:37.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:06:37.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:06:37.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:06:37.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:06:37.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:06:37.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T04:06:37.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:06:37.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: raise self._exception 2021-07-31T04:06:37.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:06:37.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:06:37.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:06:37.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T04:06:37.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:06:37.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: raise 2021-07-31T04:06:37.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:06:37.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T04:06:37.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:06:37.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:06:37.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:06:37.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: debug 2021-07-31T04:06:36.998+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:06:37.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:06:37.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:06:37.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: yield 2021-07-31T04:06:37.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:06:37.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:06:37.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:06:37.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:06:37.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:06:37.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:06:37.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:06:37.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:06:37.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:06:37.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: task.result() 2021-07-31T04:06:37.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:36 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:06:37.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:37 smithi154 conmon[57492]: debug 2021-07-31T04:06:36.999+0000 7fcb9816a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:06:37.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:37 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:06:37.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:06:37.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:37 smithi154 conmon[57492]: yield 2021-07-31T04:06:37.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:06:37.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:37 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:06:37.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:06:37.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:37 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:06:37.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:06:37.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:37 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:06:37.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:06:37.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:37 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:06:37.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:06:37.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:37 smithi154 conmon[57492]: task.result() 2021-07-31T04:06:37.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:37 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:06:37.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:37 smithi154 conmon[57492]: 2021-07-31T04:06:37.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:37 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T04:06:37.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:37 smithi154 conmon[57492]: 2021-07-31T04:06:37.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:37 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:06:37.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:06:37.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:37 smithi154 conmon[57492]: return f(*arg) 2021-07-31T04:06:37.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:06:37.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:37 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:06:37.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:06:37.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:37 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:06:37.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:06:37.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:37 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:06:37.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:37 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:06:37.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:37 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T04:06:37.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:37 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:06:37.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:37 smithi154 conmon[57492]: raise self._exception 2021-07-31T04:06:37.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:06:37.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:37 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:06:37.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:06:37.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:37 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T04:06:37.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:06:37.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:37 smithi154 conmon[57492]: raise 2021-07-31T04:06:37.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:37 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:06:37.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:37 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T04:06:37.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:06:37.284 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:37 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:06:37.284 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:06:37 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:06:37.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:37 smithi045 conmon[39718]: cluster 2021-07-31T04:06:35 2021-07-31T04:06:37.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:37 smithi045 conmon[39718]: .788404+0000 mgr.x (mgr.24542) 5968 : cluster [DBG] pgmap v6189: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:37.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:37 smithi045 conmon[39718]: audit 2021- 2021-07-31T04:06:37.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:37 smithi045 conmon[39718]: 07-31T04:06:36.907321 2021-07-31T04:06:37.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:37 smithi045 conmon[39718]: +0000 mon.b (mon.2) 2021-07-31T04:06:37.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:37 smithi045 conmon[39718]: 772 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T04:06:37.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:37 smithi045 conmon[39718]: audit 2021-07-31T04:06:37.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:37 smithi045 conmon[39718]: 2021-07-31T04:06:36.908717+0000 2021-07-31T04:06:37.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:37 smithi045 conmon[39718]: mon.b (mon.2) 773 2021-07-31T04:06:37.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:37 smithi045 conmon[39718]: : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T04:06:37.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:37 smithi045 conmon[39718]: audit 2021- 2021-07-31T04:06:37.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:37 smithi045 conmon[39718]: 07-31T04:06:36. 2021-07-31T04:06:37.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:37 smithi045 conmon[39718]: 909769+0000 mon.b (mon.2 2021-07-31T04:06:37.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:37 smithi045 conmon[39718]: ) 774 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T04:06:37.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:37 smithi045 conmon[39718]: 2021-07-31T04:06:37.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:37 smithi045 conmon[33655]: cluster 2021-07-31T04:06:35.788404+0000 mgr.x (mgr.24542) 5968 : 2021-07-31T04:06:37.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:37 smithi045 conmon[33655]: cluster [DBG] pgmap v6189: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:37.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:37 smithi045 conmon[33655]: audit 2021-07-31T04:06:36.907321+0000 mon.b (mon.2) 772 : audit [DBG] 2021-07-31T04:06:37.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:37 smithi045 conmon[33655]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T04:06:37.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:37 smithi045 conmon[33655]: audit 2021-07-31 2021-07-31T04:06:37.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:37 smithi045 conmon[33655]: T04:06:36.908717 2021-07-31T04:06:37.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:37 smithi045 conmon[33655]: +0000 mon.b (mon.2) 773 : 2021-07-31T04:06:37.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:37 smithi045 conmon[33655]: audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T04:06:37.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:37 smithi045 conmon[33655]: audit 2021- 2021-07-31T04:06:37.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:37 smithi045 conmon[33655]: 07-31T04:06:36.909769+0000 mon.b 2021-07-31T04:06:37.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:37 smithi045 conmon[33655]: (mon.2) 774 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T04:06:37.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:37 smithi154 conmon[34553]: cluster 2021-07-31T04:06:35.788404+0000 mgr.x (mgr.24542) 5968 : cluster [DBG] pgmap v6189: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:37.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:37 smithi154 conmon[34553]: audit 2021-07-31T04:06:36.907321+0000 mon.b (mon.2) 772 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T04:06:37.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:37 smithi154 conmon[34553]: audit 2021-07-31T04:06:36 2021-07-31T04:06:37.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:37 smithi154 conmon[34553]: .908717+0000 mon.b (mon.2) 773 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T04:06:37.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:37 smithi154 conmon[34553]: audit 2021-07 2021-07-31T04:06:37.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:37 smithi154 conmon[34553]: -31T04:06:36.909769+0000 mon.b 2021-07-31T04:06:37.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:37 smithi154 conmon[34553]: (mon.2) 774 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T04:06:37.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:37 smithi154 conmon[34553]: 2021-07-31T04:06:38.139 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T04:06:38.420 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T04:06:38.420 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 2h ago 2h - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T04:06:38.420 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 2h ago 2h 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T04:06:38.420 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 2h ago 2h 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T04:06:38.421 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 2h ago 2h - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T04:06:38.421 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T04:06:38.421 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 2h ago 2h 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T04:06:38.421 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T04:06:38.421 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 2h ago 2h - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T04:06:38.422 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 2h ago 2h 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T04:06:38.422 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T04:06:38.422 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T04:06:38.422 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T04:06:38.422 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T04:06:38.423 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 2h ago 2h 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T04:06:38.423 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 2h ago 2h 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T04:06:38.423 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 2h ago 2h 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T04:06:38.423 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 2h ago 2h 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T04:06:38.423 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 2h ago 2h 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T04:06:38.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: cephadm 2021-07-31T04:06:36.910532+0000 2021-07-31T04:06:38.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: mgr.x (mgr.24542) 5969 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T04:06:38.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: cephadm 2021-07-31T04:06:36.953585+0000 mgr.x (mgr.24542) 2021-07-31T04:06:38.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: 5970 : cephadm [ERR] 2021-07-31T04:06:38.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:06:38.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:06:38.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: yield 2021-07-31T04:06:38.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:06:38.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:06:38.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:06:38.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:06:38.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:06:38.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:06:38.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:06:38.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:06:38.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:06:38.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: task.result() 2021-07-31T04:06:38.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:06:38.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: cephadm 2021-07- 2021-07-31T04:06:38.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: 31T04:06:36.953992+0000 mgr.x (mgr.24542) 5971 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T04:06:38.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: cephadm 2021-07-31T04:06:36.965021+0000 mgr.x (mgr.24542) 5972 : cephadm [ERR] 2021-07-31T04:06:38.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:06:38.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:06:38.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: yield 2021-07-31T04:06:38.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:06:38.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:06:38.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:06:38.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:06:38.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:06:38.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:06:38.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:06:38.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:06:38.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:06:38.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: task.result() 2021-07-31T04:06:38.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:06:38.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: cephadm 2021-07-31T04:06:36.965959+0000 mgr.x (mgr.24542) 5973 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T04:06:38.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: cephadm 2021-07-31 2021-07-31T04:06:38.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: T04:06:36.990673+0000 mgr.x ( 2021-07-31T04:06:38.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: mgr.24542) 5974 : cephadm 2021-07-31T04:06:38.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: [ERR] 2021-07-31T04:06:38.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: cephadm 2021-07-31T04: 2021-07-31T04:06:38.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: 06:36.910532+0000 mgr.x (mgr.24542) 5969 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T04:06:38.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: cephadm 2021-07-31T04 2021-07-31T04:06:38.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: :06:36.953585+0000 mgr.x ( 2021-07-31T04:06:38.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: mgr.24542) 5970 : cephadm 2021-07-31T04:06:38.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: [ERR] 2021-07-31T04:06:38.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:06:38.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:06:38.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: yield 2021-07-31T04:06:38.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:06:38.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:06:38.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:06:38.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:06:38.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:06:38.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:06:38.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:06:38.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:06:38.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:06:38.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: task.result() 2021-07-31T04:06:38.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:06:38.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: cephadm 2021-07-31T04:06:36.953992+0000 mgr.x (mgr.24542) 5971 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T04:06:38.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: cephadm 2021-07- 2021-07-31T04:06:38.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: 31T04:06:36.965021+0000 mgr.x 2021-07-31T04:06:38.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: (mgr.24542) 5972 : cephadm [ERR] 2021-07-31T04:06:38.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: 2021-07-31T04:06:38.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:06:38.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:06:38.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: yield 2021-07-31T04:06:38.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:06:38.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:06:38.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:06:38.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:06:38.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:06:38.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:06:38.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:06:38.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:06:38.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:06:38.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: task.result() 2021-07-31T04:06:38.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:06:38.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: cephadm 2021-07-31T04:06:36.965959+0000 mgr.x (mgr.24542) 5973 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T04:06:38.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: cephadm 2021-07-31T04:06:36.990673+0000 2021-07-31T04:06:38.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: mgr.x (mgr.24542) 5974 : cephadm [ERR] 2021-07-31T04:06:38.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:06:38.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:06:38.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: yield 2021-07-31T04:06:38.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:06:38.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:06:38.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:06:38.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:06:38.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:06:38.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:06:38.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:06:38.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:06:38.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:06:38.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: task.result() 2021-07-31T04:06:38.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:06:38.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: 2021-07-31T04:06:38.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: cephadm 2021-07-31T04:06:36.991138+0000 mgr.x (mgr.24542) 5975 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:06:38.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:06:38.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:06:38.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: yield 2021-07-31T04:06:38.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:06:38.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:06:38.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:06:38.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:06:38.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:06:38.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:06:38.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:06:38.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:06:38.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:06:38.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: task.result() 2021-07-31T04:06:38.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:06:38.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: 2021-07-31T04:06:38.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T04:06:38.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: 2021-07-31T04:06:38.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:06:38.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:06:38.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: return f(*arg) 2021-07-31T04:06:38.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:06:38.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:06:38.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:06:38.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:06:38.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:06:38.550 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:06:38.551 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:06:38.551 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: yield 2021-07-31T04:06:38.551 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:06:38.551 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:06:38.552 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:06:38.552 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:06:38.552 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:06:38.552 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:06:38.552 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:06:38.553 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:06:38.553 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:06:38.553 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: task.result() 2021-07-31T04:06:38.553 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:06:38.553 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: cephadm 2021-07-31T04:06:36.991138+0000 mgr.x (mgr.24542) 5975 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:06:38.554 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:06:38.554 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:06:38.554 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: yield 2021-07-31T04:06:38.554 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:06:38.554 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:06:38.555 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:06:38.555 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:06:38.555 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:06:38.555 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:06:38.556 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:06:38.556 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:06:38.556 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:06:38.556 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: task.result() 2021-07-31T04:06:38.556 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:06:38.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: 2021-07-31T04:06:38.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T04:06:38.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: 2021-07-31T04:06:38.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:06:38.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:06:38.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: return f(*arg) 2021-07-31T04:06:38.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:06:38.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:06:38.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:06:38.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:06:38.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:06:38.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:06:38.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:06:38.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:06:38.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T04:06:38.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:06:38.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: raise self._exception 2021-07-31T04:06:38.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:06:38.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:06:38.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:06:38.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T04:06:38.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:06:38.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: raise 2021-07-31T04:06:38.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:06:38.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T04:06:38.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:06:38.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:06:38.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:06:38.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: cephadm 2021-07-31T04:06:38.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: 2021-07-31T04:06:36.999599+0000 mgr.x (mgr.24542) 5976 : cephadm [ERR] 2021-07-31T04:06:38.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:06:38.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:06:38.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: yield 2021-07-31T04:06:38.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:06:38.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:06:38.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:06:38.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:06:38.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:06:38.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:06:38.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:06:38.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:06:38.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:06:38.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: task.result() 2021-07-31T04:06:38.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:06:38.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: cephadm 2021-07-31T04:06:37.000027+0000 mgr.x (mgr.24542) 5977 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:06:38.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:06:38.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:06:38.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: yield 2021-07-31T04:06:38.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:06:38.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:06:38.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:06:38.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:06:38.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T04:06:38.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:06:38.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: raise self._exception 2021-07-31T04:06:38.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:06:38.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:06:38.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:06:38.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T04:06:38.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:06:38.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: raise 2021-07-31T04:06:38.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:06:38.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T04:06:38.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:06:38.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:06:38.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:06:38.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: cephadm 2021-07-31T04:06:36.999599+0000 mgr.x (mgr.24542) 5976 : cephadm [ERR] 2021-07-31T04:06:38.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:06:38.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:06:38.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: yield 2021-07-31T04:06:38.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:06:38.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:06:38.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:06:38.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:06:38.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:06:38.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:06:38.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:06:38.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:06:38.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:06:38.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: task.result() 2021-07-31T04:06:38.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:06:38.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: cephadm 2021-07-31T04:06:37.000027+0000 mgr.x (mgr.24542) 5977 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:06:38.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:06:38.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:06:38.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: yield 2021-07-31T04:06:38.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:06:38.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:06:38.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:06:38.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:06:38.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:06:38.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:06:38.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:06:38.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:06:38.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:06:38.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:06:38.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:06:38.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: task.result() 2021-07-31T04:06:38.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:06:38.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: 2021-07-31T04:06:38.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T04:06:38.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: 2021-07-31T04:06:38.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:06:38.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:06:38.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: return f(*arg) 2021-07-31T04:06:38.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:06:38.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:06:38.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:06:38.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:06:38.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:06:38.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:06:38.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:06:38.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T04:06:38.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:06:38.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: raise self._exception 2021-07-31T04:06:38.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:06:38.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:06:38.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:06:38.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T04:06:38.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:06:38.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: raise 2021-07-31T04:06:38.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:06:38.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T04:06:38.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:06:38.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:06:38.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:06:38.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:06:38.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:06:38.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:06:38.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:06:38.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: task.result() 2021-07-31T04:06:38.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:06:38.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: 2021-07-31T04:06:38.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T04:06:38.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: 2021-07-31T04:06:38.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:06:38.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:06:38.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: return f(*arg) 2021-07-31T04:06:38.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:06:38.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:06:38.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:06:38.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:06:38.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:06:38.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:06:38.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:06:38.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T04:06:38.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:06:38.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: raise self._exception 2021-07-31T04:06:38.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:06:38.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:06:38.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:06:38.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T04:06:38.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:06:38.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: raise 2021-07-31T04:06:38.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:06:38.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T04:06:38.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:06:38.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:06:38.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:38 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:06:38.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: cephadm 2021-07-31T04:06:36.910532+0000 mgr.x (mgr.24542) 5969 : cephadm [WRN] 2021-07-31T04:06:38.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T04:06:38.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: cephadm 2021-07-31T04:06:36.953585+0000 mgr.x (mgr.24542) 5970 : cephadm [ERR] 2021-07-31T04:06:38.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:06:38.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:06:38.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: yield 2021-07-31T04:06:38.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:06:38.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:06:38.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:06:38.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:06:38.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:06:38.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:06:38.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:06:38.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:06:38.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:06:38.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: task.result() 2021-07-31T04:06:38.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:06:38.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: cephadm 2021-07-31T04:06:36.953992+0000 2021-07-31T04:06:38.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: mgr.x (mgr.24542) 5971 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T04:06:38.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: cephadm 2021-07-31T04:06:36.965021+0000 mgr.x (mgr.24542) 5972 : cephadm [ERR] 2021-07-31T04:06:38.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:06:38.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:06:38.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: yield 2021-07-31T04:06:38.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:06:38.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:06:38.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:06:38.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:06:38.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:06:38.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:06:38.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:06:38.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:06:38.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:06:38.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: task.result() 2021-07-31T04:06:38.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:06:38.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: cephadm 2021-07-31T04:06:36.965959+0000 mgr.x (mgr.24542) 2021-07-31T04:06:38.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: 5973 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T04:06:38.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: cephadm 2021-07-31T04:06:36.990673+0000 mgr.x (mgr.24542) 5974 : cephadm [ERR] 2021-07-31T04:06:38.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:06:38.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:06:38.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: yield 2021-07-31T04:06:38.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:06:38.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:06:38.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:06:38.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:06:38.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:06:38.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:06:38.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:06:38.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:06:38.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:06:38.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: task.result() 2021-07-31T04:06:38.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:06:38.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: cephadm 2021-07-31T04:06:36.991138+0000 mgr.x 2021-07-31T04:06:38.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: (mgr.24542) 5975 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:06:38.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:06:38.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:06:38.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: yield 2021-07-31T04:06:38.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:06:38.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:06:38.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:06:38.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:06:38.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:06:38.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:06:38.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:06:38.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:06:38.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:06:38.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: task.result() 2021-07-31T04:06:38.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:06:38.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: 2021-07-31T04:06:38.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T04:06:38.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: 2021-07-31T04:06:38.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:06:38.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:06:38.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: return f(*arg) 2021-07-31T04:06:38.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:06:38.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:06:38.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:06:38.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:06:38.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:06:38.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:06:38.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:06:38.769 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T04:06:38.770 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T04:06:38.770 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T04:06:38.770 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:06:38.770 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T04:06:38.770 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T04:06:38.771 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:06:38.771 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:06:38.771 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T04:06:38.771 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T04:06:38.771 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:06:38.772 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T04:06:38.772 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T04:06:38.772 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T04:06:38.772 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:06:38.772 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T04:06:38.773 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T04:06:38.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T04:06:38.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:06:38.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: raise self._exception 2021-07-31T04:06:38.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:06:38.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:06:38.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:06:38.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T04:06:38.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:06:38.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: raise 2021-07-31T04:06:38.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:06:38.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T04:06:38.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:06:38.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:06:38.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:06:38.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: cephadm 2021-07-31T04:06:36.999599+0000 mgr.x (mgr.24542) 5976 : cephadm [ERR] 2021-07-31T04:06:38.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:06:38.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:06:38.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: yield 2021-07-31T04:06:38.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:06:38.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:06:38.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:06:38.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:06:38.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:06:38.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:06:38.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:06:38.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:06:38.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:06:38.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: task.result() 2021-07-31T04:06:38.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:06:38.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: cephadm 2021-07-31T04:06: 2021-07-31T04:06:38.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: 37.000027+0000 mgr.x (mgr.24542) 5977 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:06:38.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:06:38.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:06:38.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: yield 2021-07-31T04:06:38.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:06:38.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:06:38.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:06:38.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:06:38.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:06:38.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:06:38.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:06:38.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:06:38.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:06:38.782 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: task.result() 2021-07-31T04:06:38.782 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:06:38.782 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: 2021-07-31T04:06:38.782 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T04:06:38.782 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: 2021-07-31T04:06:38.783 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:06:38.783 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:06:38.783 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: return f(*arg) 2021-07-31T04:06:38.783 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:06:38.783 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:06:38.784 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:06:38.784 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:06:38.784 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:06:38.784 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:06:38.784 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:06:38.785 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T04:06:38.785 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:06:38.785 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: raise self._exception 2021-07-31T04:06:38.785 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:06:38.785 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:06:38.786 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:06:38.786 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T04:06:38.786 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:06:38.786 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: raise 2021-07-31T04:06:38.786 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:06:38.787 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T04:06:38.787 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:06:38.787 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:06:38.787 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:38 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:06:39.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:39 smithi045 conmon[33655]: cluster 2021-07-31T04:06:37.788968+0000 mgr.x (mgr.24542) 5978 : cluster 2021-07-31T04:06:39.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:39 smithi045 conmon[33655]: [DBG] pgmap v6190: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:39.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:39 smithi045 conmon[33655]: audit 2021-07-31T04:06:38.128218+0000 mgr.x (mgr.24542) 5979 : audit [DBG] from='client.20910 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:06:39.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:39 smithi045 conmon[33655]: audit 2021-07-31T04:06:38 2021-07-31T04:06:39.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:39 smithi045 conmon[33655]: .757840+0000 mon.c (mon.1) 249 : audit [DBG] 2021-07-31T04:06:39.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:39 smithi045 conmon[33655]: from='client.? 172.21.15.45:0/670589914' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:06:39.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:39 smithi045 conmon[39718]: cluster 2021-07-31T04:06 2021-07-31T04:06:39.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:39 smithi045 conmon[39718]: :37.788968+0000 mgr.x (mgr.24542) 5978 : cluster [DBG] pgmap v6190: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:39.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:39 smithi045 conmon[39718]: audit 2021-07-31T04:06:38. 2021-07-31T04:06:39.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:39 smithi045 conmon[39718]: 128218+0000 mgr.x (mgr.24542) 2021-07-31T04:06:39.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:39 smithi045 conmon[39718]: 5979 : audit [DBG] from='client.20910 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:06:39.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:39 smithi045 conmon[39718]: audit 2021- 2021-07-31T04:06:39.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:39 smithi045 conmon[39718]: 07-31T04:06:38. 2021-07-31T04:06:39.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:39 smithi045 conmon[39718]: 757840+0000 mon.c (mon.1 2021-07-31T04:06:39.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:39 smithi045 conmon[39718]: ) 249 : audit [DBG] from='client.? 172.21.15.45:0/670589914' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:06:39.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:39 smithi045 conmon[39718]: 2021-07-31T04:06:39.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:39 smithi154 conmon[34553]: cluster 2021-07-31T04:06:37.788968+0000 mgr.x (mgr.24542) 5978 : cluster [DBG] pgmap v6190: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:39.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:39 smithi154 conmon[34553]: 2021-07-31T04:06:39.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:39 smithi154 conmon[34553]: audit 2021-07-31T04:06:38.128218+0000 mgr.x (mgr.24542) 5979 : audit [DBG] from='client.20910 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:06:39.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:39 smithi154 conmon[34553]: audit 2021-07-31T04:06:38.757840+0000 mon.c (mon.1) 249 : audit 2021-07-31T04:06:39.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:39 smithi154 conmon[34553]: [DBG] from='client.? 172.21.15.45:0/670589914' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:06:40.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:40 smithi154 conmon[34553]: audit 2021-07-31T04:06:38.412311+0000 mgr.x (mgr.24542) 5980 : audit [DBG] from='client.20916 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:06:40.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:40 smithi154 conmon[34553]: 2021-07-31T04:06:40.774 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:40 smithi045 conmon[33655]: audit 2021-07-31T04:06:38.412311+0000 mgr.x ( 2021-07-31T04:06:40.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:40 smithi045 conmon[33655]: mgr.24542) 5980 : audit [DBG] from='client.20916 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:06:40.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:40 smithi045 conmon[39718]: audit 2021-07-31T04:06:38.412311+0000 mgr.x 2021-07-31T04:06:40.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:40 smithi045 conmon[39718]: (mgr.24542) 5980 : audit [DBG] from='client.20916 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:06:41.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:41 smithi154 conmon[34553]: cluster 2021-07-31T04:06:39.789546+0000 mgr.x (mgr.24542) 5981 : cluster 2021-07-31T04:06:41.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:41 smithi154 conmon[34553]: [DBG] pgmap v6191: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:41.774 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:41 smithi045 conmon[39718]: cluster 2021-07-31T04:06:39.789546+0000 mgr.x (mgr.24542) 5981 : cluster [DBG] pgmap v6191: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:41.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:41 smithi045 conmon[33655]: cluster 2021-07-31T04:06:39.789546+0000 mgr.x (mgr.24542) 5981 : cluster [DBG] pgmap v6191: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:41.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:41 smithi045 conmon[33655]: 2021-07-31T04:06:43.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:43 smithi154 conmon[34553]: cluster 2021-07-31T04:06:41.790055+0000 mgr.x (mgr.24542) 5982 : cluster [DBG] 2021-07-31T04:06:43.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:43 smithi154 conmon[34553]: pgmap v6192: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:43.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:43 smithi045 conmon[39718]: cluster 2021-07-31T04:06:41 2021-07-31T04:06:43.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:43 smithi045 conmon[39718]: .790055+0000 mgr.x (mgr.24542) 5982 : cluster [DBG] pgmap v6192: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:43.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:43 smithi045 conmon[33655]: cluster 2021-07-31T04:06:41.790055+0000 mgr.x (mgr.24542) 5982 : cluster [DBG] pgmap v6192: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:45.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:45 smithi154 conmon[34553]: cluster 2021-07-31T04:06:43.790419+0000 mgr.x (mgr.24542) 5983 : cluster [DBG] 2021-07-31T04:06:45.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:45 smithi154 conmon[34553]: pgmap v6193: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:45.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:45 smithi045 conmon[39718]: cluster 2021-07-31T04:06:43. 2021-07-31T04:06:45.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:45 smithi045 conmon[39718]: 790419+0000 mgr.x (mgr.24542) 5983 : cluster [DBG] pgmap v6193: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:45.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:45 smithi045 conmon[33655]: cluster 2021-07-31T04:06:43.790419+0000 mgr.x (mgr.24542 2021-07-31T04:06:45.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:45 smithi045 conmon[33655]: ) 5983 : cluster [DBG] pgmap v6193: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:47.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:47 smithi154 conmon[34553]: cluster 2021-07-31T04:06:45.790964+0000 mgr.x (mgr.24542) 5984 : cluster [DBG] pgmap v6194: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:47.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:47 smithi045 conmon[39718]: cluster 2021-07-31T04:06:45.790964+0000 mgr.x (mgr.24542) 5984 : cluster [DBG] 2021-07-31T04:06:47.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:47 smithi045 conmon[39718]: pgmap v6194: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:47.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:47 smithi045 conmon[33655]: cluster 2021-07-31T04:06:45.790964+0000 2021-07-31T04:06:47.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:47 smithi045 conmon[33655]: mgr.x (mgr.24542) 5984 : cluster [DBG] pgmap v6194: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:49.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:49 smithi154 conmon[34553]: cluster 2021-07-31T04:06:47.791423+0000 mgr.x (mgr.24542) 5985 : cluster [DBG] pgmap v6195: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:49.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:49 smithi045 conmon[39718]: cluster 2021-07-31T04:06:47.791423+0000 mgr.x ( 2021-07-31T04:06:49.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:49 smithi045 conmon[39718]: mgr.24542) 5985 : cluster [DBG] pgmap v6195: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:49.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:49 smithi045 conmon[33655]: cluster 2021-07-31T04:06:47.791423+0000 mgr.x (mgr.24542) 5985 : 2021-07-31T04:06:49.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:49 smithi045 conmon[33655]: cluster [DBG] pgmap v6195: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:51.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:51 smithi154 conmon[34553]: cluster 2021-07-31T04:06:49.792032+0000 mgr.x (mgr.24542) 5986 : cluster [DBG] pgmap v6196: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:51.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:51 smithi045 conmon[39718]: cluster 2021-07-31T04:06:49.792032+0000 2021-07-31T04:06:51.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:51 smithi045 conmon[39718]: mgr.x (mgr.24542) 5986 : cluster [DBG] pgmap v6196: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:51.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:51 smithi045 conmon[33655]: cluster 2021-07-31T04:06:49.792032 2021-07-31T04:06:51.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:51 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 5986 : cluster [DBG] pgmap v6196: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:53.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:53 smithi154 conmon[34553]: cluster 2021-07-31T04:06:51.792569+0000 mgr.x (mgr.24542) 5987 : cluster [DBG] pgmap v6197: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:53.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:53 smithi045 conmon[39718]: cluster 2021-07-31T04:06:51.792569 2021-07-31T04:06:53.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:53 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 5987 : cluster [DBG] pgmap v6197: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:53.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:53 smithi045 conmon[33655]: cluster 2021-07-31T04:06:51.792569+0000 2021-07-31T04:06:53.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:53 smithi045 conmon[33655]: mgr.x (mgr.24542) 5987 : cluster [DBG] pgmap v6197: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:55.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:55 smithi154 conmon[34553]: cluster 2021-07-31T04:06:53.793150+0000 mgr.x (mgr.24542) 5988 : cluster [DBG] pgmap v6198: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:55.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:55 smithi045 conmon[39718]: cluster 2021-07-31T04:06:53.793150+0000 mgr.x (mgr 2021-07-31T04:06:55.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:55 smithi045 conmon[39718]: .24542) 5988 : cluster [DBG] pgmap v6198: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:55.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:55 smithi045 conmon[33655]: cluster 2021-07-31T04:06:53.793150+0000 mgr.x (mgr 2021-07-31T04:06:55.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:55 smithi045 conmon[33655]: .24542) 5988 : cluster [DBG] pgmap v6198: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:57.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:57 smithi154 conmon[34553]: cluster 2021-07-31T04:06:55.793694+0000 mgr.x (mgr.24542) 5989 : cluster [DBG] pgmap v6199: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:57.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:57 smithi045 conmon[39718]: cluster 2021-07-31T04:06:55.793694+0000 mgr.x (mgr.24542) 5989 : cluster [DBG] pgmap v6199: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:57.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:57 smithi045 conmon[33655]: cluster 2021-07-31T04:06:55.793694+0000 mgr.x 2021-07-31T04:06:57.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:57 smithi045 conmon[33655]: (mgr.24542) 5989 : cluster [DBG] pgmap v6199: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:59.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:06:59 smithi154 conmon[34553]: cluster 2021-07-31T04:06:57.794080+0000 mgr.x (mgr.24542) 5990 : cluster [DBG] pgmap v6200: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:59.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:06:59 smithi045 conmon[39718]: cluster 2021-07-31T04:06:57.794080+0000 mgr.x (mgr.24542) 5990 : cluster [DBG] pgmap v6200: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:06:59.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:59 smithi045 conmon[33655]: cluster 2021-07-31T04:06:57.794080+0000 mgr.x (mgr.24542) 5990 2021-07-31T04:06:59.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:06:59 smithi045 conmon[33655]: : cluster [DBG] pgmap v6200: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:01.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:01 smithi154 conmon[34553]: cluster 2021-07-31T04:06:59.794711+0000 mgr.x (mgr.24542) 5991 : cluster [DBG] pgmap v6201: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:01.774 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:01 smithi045 conmon[39718]: cluster 2021-07-31T04:06:59.794711+0000 2021-07-31T04:07:01.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:01 smithi045 conmon[39718]: mgr.x (mgr.24542) 5991 : cluster [DBG] pgmap v6201: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:01.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:01 smithi045 conmon[33655]: cluster 2021-07-31T04:06:59.794711+0000 mgr.x (mgr.24542) 2021-07-31T04:07:01.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:01 smithi045 conmon[33655]: 5991 : cluster [DBG] pgmap v6201: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:03.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:03 smithi154 conmon[34553]: cluster 2021-07-31T04:07:01.795116+0000 mgr.x (mgr.24542) 5992 : 2021-07-31T04:07:03.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:03 smithi154 conmon[34553]: cluster [DBG] pgmap v6202: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:03.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:03 smithi045 conmon[39718]: cluster 2021-07-31T04:07:01.795116+0000 mgr.x (mgr.24542) 5992 : cluster [DBG] pgmap v6202: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:03.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:03 smithi045 conmon[33655]: cluster 2021-07-31T04:07:01.795116+0000 mgr.x (mgr.24542) 5992 2021-07-31T04:07:03.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:03 smithi045 conmon[33655]: : cluster [DBG] pgmap v6202: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:05.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:05 smithi154 conmon[34553]: cluster 2021-07-31T04:07:03.795668+0000 mgr.x (mgr.24542) 5993 : cluster [DBG] 2021-07-31T04:07:05.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:05 smithi154 conmon[34553]: pgmap v6203: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:05.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:05 smithi045 conmon[33655]: cluster 2021-07-31T04:07:03.795668+0000 mgr.x (mgr.24542) 5993 : cluster 2021-07-31T04:07:05.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:05 smithi045 conmon[33655]: [DBG] pgmap v6203: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:05.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:05 smithi045 conmon[39718]: cluster 2021-07-31T04:07:03.795668+0000 2021-07-31T04:07:05.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:05 smithi045 conmon[39718]: mgr.x (mgr.24542) 5993 : cluster [DBG] pgmap v6203: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:07.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:07 smithi154 conmon[34553]: cluster 2021-07-31T04:07:05.796148+0000 mgr.x (mgr.24542) 5994 : cluster 2021-07-31T04:07:07.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:07 smithi154 conmon[34553]: [DBG] pgmap v6204: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:07.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:07 smithi045 conmon[39718]: cluster 2021-07-31T04:07:05.796148+0000 mgr.x ( 2021-07-31T04:07:07.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:07 smithi045 conmon[39718]: mgr.24542) 5994 : cluster [DBG] pgmap v6204: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:07.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:07 smithi045 conmon[33655]: cluster 2021-07-31T04:07:05.796148+0000 2021-07-31T04:07:07.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:07 smithi045 conmon[33655]: mgr.x (mgr.24542) 5994 : cluster [DBG] pgmap v6204: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:09.064 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T04:07:09.343 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T04:07:09.343 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 2h ago 2h - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T04:07:09.343 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 2h ago 2h 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T04:07:09.343 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 2h ago 2h 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T04:07:09.343 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 2h ago 2h - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T04:07:09.344 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T04:07:09.344 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 2h ago 2h 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T04:07:09.344 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T04:07:09.344 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 2h ago 2h - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T04:07:09.344 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 2h ago 2h 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T04:07:09.345 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T04:07:09.345 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T04:07:09.345 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T04:07:09.345 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T04:07:09.345 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 2h ago 2h 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T04:07:09.346 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 2h ago 2h 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T04:07:09.346 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 2h ago 2h 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T04:07:09.346 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 2h ago 2h 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T04:07:09.346 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 2h ago 2h 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T04:07:09.680 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T04:07:09.681 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T04:07:09.681 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T04:07:09.681 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:07:09.681 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T04:07:09.681 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T04:07:09.682 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:07:09.682 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:07:09.682 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T04:07:09.682 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T04:07:09.683 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:07:09.683 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T04:07:09.683 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T04:07:09.683 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T04:07:09.684 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:07:09.684 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T04:07:09.684 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T04:07:09.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:09 smithi154 conmon[34553]: cluster 2021-07-31T04:07:07.796640+0000 mgr.x (mgr.24542) 5995 : cluster [DBG] pgmap v6205: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:09.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:09 smithi045 conmon[39718]: cluster 2021-07-31T04:07: 2021-07-31T04:07:09.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:09 smithi045 conmon[39718]: 07.796640+0000 mgr.x (mgr.24542) 5995 : cluster [DBG] pgmap v6205: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:09.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:09 smithi045 conmon[33655]: cluster 2021-07-31T04:07:07.796640+0000 mgr.x ( 2021-07-31T04:07:09.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:09 smithi045 conmon[33655]: mgr.24542) 5995 : cluster [DBG] pgmap v6205: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:10.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:10 smithi154 conmon[34553]: audit 2021-07-31T04:07:09.053271+0000 mgr.x (mgr. 2021-07-31T04:07:10.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:10 smithi154 conmon[34553]: 24542) 5996 : audit [DBG] from='client.30865 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:07:10.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:10 smithi154 conmon[34553]: audit 2021-07-31T04:07:09.335044+0000 mgr.x (mgr.24542) 5997 : audit [DBG] from='client.30871 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:07:10.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:10 smithi154 conmon[34553]: audit 2021-07-31T04:07:09.681507+0000 mon.a (mon.0) 3424 : 2021-07-31T04:07:10.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:10 smithi154 conmon[34553]: audit [DBG] from='client.? 172.21.15.45:0/3085190438' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:07:10.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:10 smithi045 conmon[33655]: audit 2021-07-31T04:07:09.053271+0000 mgr.x (mgr.24542 2021-07-31T04:07:10.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:10 smithi045 conmon[33655]: ) 5996 : audit [DBG] from='client.30865 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:07:10.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:10 smithi045 conmon[33655]: audit 2021-07-31T04:07:09.335044+0000 mgr.x (mgr.24542) 5997 : audit [DBG] from='client.30871 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:07:10.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:10 smithi045 conmon[33655]: audit 2021-07-31T04:07:09.681507+0000 mon.a (mon.0) 3424 : audit [DBG] from='client.? 172.21.15.45:0/3085190438' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:07:10.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:10 smithi045 conmon[33655]: 2021-07-31T04:07:10.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:10 smithi045 conmon[39718]: audit 2021-07-31T04:07:09.053271 2021-07-31T04:07:10.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:10 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 5996 : audit [DBG] from='client.30865 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:07:10.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:10 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T04:07:10.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:10 smithi045 conmon[39718]: 31T04:07:09.335044 2021-07-31T04:07:10.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:10 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 5997 2021-07-31T04:07:10.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:10 smithi045 conmon[39718]: : audit [DBG] from='client.30871 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:07:10.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:10 smithi045 conmon[39718]: audit 2021 2021-07-31T04:07:10.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:10 smithi045 conmon[39718]: -07-31T04:07:09. 2021-07-31T04:07:10.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:10 smithi045 conmon[39718]: 681507+0000 mon.a (mon. 2021-07-31T04:07:10.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:10 smithi045 conmon[39718]: 0) 3424 : audit [DBG] from='client.? 172.21.15.45:0/3085190438' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:07:10.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:10 smithi045 conmon[39718]: 2021-07-31T04:07:11.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:11 smithi154 conmon[34553]: cluster 2021-07-31T04:07:09.797098+0000 mgr.x (mgr.24542) 5998 : cluster [DBG] pgmap v6206: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:11.774 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:11 smithi045 conmon[33655]: cluster 2021-07-31T04:07:09.797098+0000 mgr.x (mgr.24542) 5998 : cluster 2021-07-31T04:07:11.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:11 smithi045 conmon[33655]: [DBG] pgmap v6206: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:11.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:11 smithi045 conmon[39718]: cluster 2021-07-31T04:07:09.797098+0000 mgr.x (mgr.24542) 5998 : cluster [DBG] pgmap v6206: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:11.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:11 smithi045 conmon[39718]: 2021-07-31T04:07:13.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:13 smithi154 conmon[34553]: cluster 2021-07-31T04:07:11.797565+0000 mgr.x (mgr.24542) 5999 : cluster [DBG] 2021-07-31T04:07:13.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:13 smithi154 conmon[34553]: pgmap v6207: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:13.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:13 smithi045 conmon[39718]: cluster 2021-07-31T04:07:11.797565+0000 mgr.x (mgr.24542) 2021-07-31T04:07:13.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:13 smithi045 conmon[39718]: 5999 : cluster [DBG] pgmap v6207: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:13.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:13 smithi045 conmon[33655]: cluster 2021-07-31T04:07:11.797565+0000 mgr.x (mgr.24542) 5999 : cluster 2021-07-31T04:07:13.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:13 smithi045 conmon[33655]: [DBG] pgmap v6207: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:15.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:15 smithi154 conmon[34553]: cluster 2021-07-31T04:07:13.798031+0000 mgr.x (mgr.24542) 6000 : cluster [DBG] 2021-07-31T04:07:15.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:15 smithi154 conmon[34553]: pgmap v6208: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:15.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:15 smithi045 conmon[39718]: cluster 2021-07-31T04:07:13.798031+0000 mgr.x (mgr.24542) 6000 : cluster [DBG] 2021-07-31T04:07:15.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:15 smithi045 conmon[39718]: pgmap v6208: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:15.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:15 smithi045 conmon[33655]: cluster 2021-07-31T04:07:13.798031+0000 mgr.x (mgr.24542) 6000 : cluster [DBG] pgmap v6208: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:17.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:17 smithi154 conmon[34553]: cluster 2021-07-31T04:07:15.798570+0000 mgr.x (mgr.24542) 6001 : cluster [DBG] pgmap v6209: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:17.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:17 smithi045 conmon[33655]: cluster 2021-07-31T04:07:15.798570+0000 mgr.x (mgr.24542) 6001 2021-07-31T04:07:17.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:17 smithi045 conmon[33655]: : cluster [DBG] pgmap v6209: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:17.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:17 smithi045 conmon[39718]: cluster 2021-07-31T04:07:15.798570+0000 mgr.x (mgr 2021-07-31T04:07:17.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:17 smithi045 conmon[39718]: .24542) 6001 : cluster [DBG] pgmap v6209: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:19.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:19 smithi154 conmon[34553]: cluster 2021-07-31T04:07:17.799126+0000 mgr.x (mgr.24542) 6002 : cluster [DBG] pgmap v6210: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:19.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:19 smithi045 conmon[33655]: cluster 2021-07-31T04:07:17.799126+0000 mgr.x (mgr.24542) 6002 2021-07-31T04:07:19.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:19 smithi045 conmon[33655]: : cluster [DBG] pgmap v6210: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:19.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:19 smithi045 conmon[39718]: cluster 2021-07-31T04:07:17.799126+0000 mgr.x 2021-07-31T04:07:19.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:19 smithi045 conmon[39718]: (mgr.24542) 6002 : cluster [DBG] pgmap v6210: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:21.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:21 smithi154 conmon[34553]: cluster 2021-07-31T04:07:19.799755+0000 mgr.x (mgr.24542) 6003 : cluster [DBG] pgmap v6211: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:21.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:21 smithi045 conmon[39718]: cluster 2021-07-31T04:07 2021-07-31T04:07:21.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:21 smithi045 conmon[39718]: :19.799755+0000 mgr.x (mgr.24542) 6003 : cluster [DBG] pgmap v6211: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:21.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:21 smithi045 conmon[33655]: cluster 2021-07-31T04:07:19.799755+0000 mgr.x ( 2021-07-31T04:07:21.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:21 smithi045 conmon[33655]: mgr.24542) 6003 : cluster [DBG] pgmap v6211: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:23.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:23 smithi154 conmon[34553]: cluster 2021-07-31T04:07:21.800291+0000 mgr.x (mgr.24542) 6004 : cluster [DBG] 2021-07-31T04:07:23.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:23 smithi154 conmon[34553]: pgmap v6212: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:23.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:23 smithi045 conmon[39718]: cluster 2021-07-31T04:07:21.800291+0000 mgr.x (mgr.24542) 6004 : cluster 2021-07-31T04:07:23.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:23 smithi045 conmon[39718]: [DBG] pgmap v6212: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:23.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:23 smithi045 conmon[33655]: cluster 2021-07-31T04:07:21.800291+0000 mgr.x (mgr.24542) 6004 : cluster [DBG] pgmap v6212: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:25.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:25 smithi154 conmon[34553]: cluster 2021-07-31T04:07:23.800681+0000 mgr.x (mgr.24542) 6005 : cluster [DBG] pgmap v6213: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:25.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:25 smithi045 conmon[39718]: cluster 2021-07-31T04:07:23.800681+0000 mgr.x (mgr.24542) 6005 : cluster [DBG] pgmap v6213: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:25.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:25 smithi045 conmon[33655]: cluster 2021-07-31T04:07:23.800681+0000 mgr.x (mgr.24542) 6005 : cluster [DBG] pgmap v6213: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:27.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:27 smithi154 conmon[34553]: cluster 2021-07-31T04:07:25.801146+0000 mgr.x (mgr.24542) 6006 : cluster [DBG] pgmap v6214: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:27.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:27 smithi154 conmon[34553]: 2021-07-31T04:07:27.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:27 smithi045 conmon[39718]: cluster 2021-07-31T04:07:25.801146+0000 mgr.x (mgr.24542) 2021-07-31T04:07:27.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:27 smithi045 conmon[39718]: 6006 : cluster [DBG] pgmap v6214: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:27.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:27 smithi045 conmon[33655]: cluster 2021-07-31T04:07:25.801146+0000 mgr.x (mgr 2021-07-31T04:07:27.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:27 smithi045 conmon[33655]: .24542) 6006 : cluster [DBG] pgmap v6214: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:29.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:29 smithi154 conmon[34553]: cluster 2021-07-31T04:07:27.801547+0000 mgr.x (mgr.24542) 6007 : cluster [DBG] pgmap v6215: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:29.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:29 smithi154 conmon[34553]: audit 2021-07-31T04:07:28.487932+0000 mon.b (mon.2) 775 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:07:29.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:29 smithi154 conmon[34553]: audit 2021-07-31T04:07:28.488196+0000 mon.b (mon.2) 776 : audit 2021-07-31T04:07:29.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:29 smithi154 conmon[34553]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:07:29.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:29 smithi154 conmon[34553]: audit 2021-07- 2021-07-31T04:07:29.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:29 smithi154 conmon[34553]: 31T04:07:28.489021+0000 mon.a (mon.0) 3425 : audit 2021-07-31T04:07:29.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:29 smithi154 conmon[34553]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:07:29.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:29 smithi154 conmon[34553]: audit 2021-07-31T 2021-07-31T04:07:29.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:29 smithi154 conmon[34553]: 04:07:28.489445+0000 mon.a (mon.0 2021-07-31T04:07:29.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:29 smithi154 conmon[34553]: ) 3426 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:07:29.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:29 smithi045 conmon[33655]: cluster 2021-07-31T04:07:27.801547+0000 mgr.x (mgr.24542) 6007 : 2021-07-31T04:07:29.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:29 smithi045 conmon[33655]: cluster [DBG] pgmap v6215: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:29.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:29 smithi045 conmon[33655]: audit 2021-07-31T04:07:28.487932+0000 mon.b (mon.2) 775 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:07:29.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:29 smithi045 conmon[33655]: audit 2021-07-31T04:07:29.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:29 smithi045 conmon[33655]: 2021-07-31T04:07:28.488196 2021-07-31T04:07:29.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:29 smithi045 conmon[33655]: +0000 mon.b (mon.2) 776 : audit [INF] 2021-07-31T04:07:29.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:29 smithi045 conmon[33655]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:07:29.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:29 smithi045 conmon[33655]: audit 2021-07 2021-07-31T04:07:29.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:29 smithi045 conmon[33655]: -31T04:07:28.489021+0000 mon.a ( 2021-07-31T04:07:29.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:29 smithi045 conmon[33655]: mon.0) 3425 : audit 2021-07-31T04:07:29.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:29 smithi045 conmon[33655]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:07:29.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:29 smithi045 conmon[33655]: audit 2021 2021-07-31T04:07:29.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:29 smithi045 conmon[33655]: -07-31T04:07:28.489445+0000 2021-07-31T04:07:29.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:29 smithi045 conmon[33655]: mon.a (mon.0) 3426 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:07:29.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:29 smithi045 conmon[39718]: cluster 2021-07-31T04:07: 2021-07-31T04:07:29.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:29 smithi045 conmon[39718]: 27.801547+0000 mgr.x (mgr.24542) 6007 : cluster [DBG] pgmap v6215: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:29.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:29 smithi045 conmon[39718]: audit 2021- 2021-07-31T04:07:29.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:29 smithi045 conmon[39718]: 07-31T04:07:28.487932+0000 mon.b (mon 2021-07-31T04:07:29.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:29 smithi045 conmon[39718]: .2) 775 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:07:29.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:29 smithi045 conmon[39718]: audit 2021-07 2021-07-31T04:07:29.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:29 smithi045 conmon[39718]: -31T04:07:28.488196+0000 mon.b (mon.2) 2021-07-31T04:07:29.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:29 smithi045 conmon[39718]: 776 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:07:29.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:29 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T04:07:29.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:29 smithi045 conmon[39718]: 31T04:07:28.489021+0000 mon.a (mon.0) 3425 2021-07-31T04:07:29.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:29 smithi045 conmon[39718]: : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:07:29.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:29 smithi045 conmon[39718]: audit 2021-07 2021-07-31T04:07:29.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:29 smithi045 conmon[39718]: -31T04:07:28.489445+0000 mon.a (mon.0) 3426 : audit 2021-07-31T04:07:29.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:29 smithi045 conmon[39718]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:07:31.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:31 smithi154 conmon[34553]: cluster 2021-07-31T04:07:29.801946+0000 mgr.x (mgr.24542) 6008 : cluster 2021-07-31T04:07:31.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:31 smithi154 conmon[34553]: [DBG] pgmap v6216: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:31.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:31 smithi045 conmon[33655]: cluster 2021-07-31T04:07:29.801946+0000 mgr.x ( 2021-07-31T04:07:31.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:31 smithi045 conmon[33655]: mgr.24542) 6008 : cluster [DBG] pgmap v6216: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:31.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:31 smithi045 conmon[39718]: cluster 2021-07-31T04:07:29.801946+0000 mgr.x (mgr.24542) 6008 : cluster 2021-07-31T04:07:31.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:31 smithi045 conmon[39718]: [DBG] pgmap v6216: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:33.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:33 smithi154 conmon[34553]: cluster 2021-07-31T04:07:31.802464+0000 mgr.x (mgr.24542) 6009 : cluster [DBG] pgmap v6217: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:33.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:33 smithi045 conmon[33655]: cluster 2021-07-31T04:07:31.802464+0000 mgr.x (mgr.24542) 6009 : cluster 2021-07-31T04:07:33.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:33 smithi045 conmon[33655]: [DBG] pgmap v6217: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:33.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:33 smithi045 conmon[39718]: cluster 2021-07-31T04:07:31.802464+0000 mgr.x (mgr 2021-07-31T04:07:33.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:33 smithi045 conmon[39718]: .24542) 6009 : cluster [DBG] pgmap v6217: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:35.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:35 smithi154 conmon[34553]: cluster 2021-07-31T04:07:33.803012+0000 mgr.x (mgr.24542) 6010 : cluster [DBG] pgmap v6218: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:35.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:35 smithi154 conmon[34553]: 2021-07-31T04:07:35.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:35 smithi045 conmon[39718]: cluster 2021-07-31T04:07:33.803012+0000 mgr.x (mgr.24542) 6010 : cluster [DBG] pgmap v6218: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:35.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:35 smithi045 conmon[33655]: cluster 2021-07-31T04:07:33.803012+0000 mgr.x (mgr.24542) 6010 : cluster 2021-07-31T04:07:35.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:35 smithi045 conmon[33655]: [DBG] pgmap v6218: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:37.387 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: debug 2021-07-31T04:07:37.045+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:07:37.388 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:07:37.388 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:07:37.388 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: yield 2021-07-31T04:07:37.389 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:07:37.389 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:07:37.389 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:07:37.389 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:07:37.390 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:07:37.390 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:07:37.390 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:07:37.391 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:07:37.391 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:07:37.391 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: task.result() 2021-07-31T04:07:37.391 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:07:37.392 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: debug 2021-07-31T04:07:37.056+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:07:37.392 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:07:37.392 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:07:37.392 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: yield 2021-07-31T04:07:37.393 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:07:37.393 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:07:37.393 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:07:37.393 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:07:37.394 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:07:37.394 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:07:37.394 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:07:37.395 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:07:37.395 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:07:37.395 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: task.result() 2021-07-31T04:07:37.395 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:07:37.396 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: debug 2021-07-31T04:07:37.082+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:07:37.396 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:07:37.396 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:07:37.396 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: yield 2021-07-31T04:07:37.397 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:07:37.397 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:07:37.397 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:07:37.397 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:07:37.398 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:07:37.398 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:07:37.398 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:07:37.399 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:07:37.399 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:07:37.399 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: task.result() 2021-07-31T04:07:37.399 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:07:37.400 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: debug 2021-07-31T04:07:37.082+0000 7fcb94162700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:07:37.400 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:07:37.400 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:07:37.400 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: yield 2021-07-31T04:07:37.401 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:07:37.401 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:07:37.401 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:07:37.401 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:07:37.402 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:07:37.402 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:07:37.402 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:07:37.402 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:07:37.403 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:07:37.403 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: task.result() 2021-07-31T04:07:37.403 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:07:37.403 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: 2021-07-31T04:07:37.404 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T04:07:37.404 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: 2021-07-31T04:07:37.404 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:07:37.404 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:07:37.405 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: return f(*arg) 2021-07-31T04:07:37.405 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:07:37.405 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:07:37.405 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:07:37.406 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:07:37.406 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:07:37.406 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:07:37.407 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:07:37.407 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T04:07:37.407 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:07:37.407 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: raise self._exception 2021-07-31T04:07:37.408 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:07:37.408 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:07:37.408 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:07:37.408 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T04:07:37.409 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:07:37.409 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: raise 2021-07-31T04:07:37.409 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:07:37.409 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T04:07:37.410 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:07:37.410 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:07:37.410 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:07:37.410 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: debug 2021-07-31T04:07:37.091+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:07:37.411 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:07:37.411 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:07:37.411 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: yield 2021-07-31T04:07:37.411 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:07:37.412 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:07:37.412 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:07:37.412 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:07:37.412 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:07:37.413 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:07:37.413 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:07:37.413 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:07:37.413 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:07:37.414 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: task.result() 2021-07-31T04:07:37.414 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:07:37.414 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: debug 2021-07-31T04:07:37.091+0000 7fcb93961700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:07:37.414 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:07:37.415 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:07:37.415 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: yield 2021-07-31T04:07:37.415 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:07:37.415 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:07:37.415 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:07:37.416 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:07:37.416 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:07:37.416 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:07:37.416 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:07:37.417 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:07:37.417 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:07:37.417 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: task.result() 2021-07-31T04:07:37.417 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:07:37.418 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: 2021-07-31T04:07:37.418 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T04:07:37.418 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: 2021-07-31T04:07:37.418 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:07:37.419 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:07:37.419 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: return f(*arg) 2021-07-31T04:07:37.419 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:07:37.419 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:07:37.420 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:07:37.420 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:07:37.420 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:07:37.420 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:07:37.421 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:07:37.421 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T04:07:37.421 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:07:37.421 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: raise self._exception 2021-07-31T04:07:37.422 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:07:37.422 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:07:37.422 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:07:37.422 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T04:07:37.422 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:07:37.423 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: raise 2021-07-31T04:07:37.423 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:07:37.423 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T04:07:37.423 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:07:37.424 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:07:37.424 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:07:37.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[34553]: cluster 2021-07-31T04:07:35.803623+0000 mgr.x (mgr.24542) 6011 : cluster [DBG] pgmap v6219: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:37.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[34553]: audit 2021-07- 2021-07-31T04:07:37.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[34553]: 31T04:07:37.000845+0000 mon.b (mon.2) 777 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T04:07:37.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[34553]: audit 2021-07-31T04:07:37.002349+0000 mon.b (mon.2) 778 : audit [DBG] 2021-07-31T04:07:37.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[34553]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T04:07:37.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[34553]: audit 2021-07-31T04:07:37.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[34553]: 2021-07-31T04:07:37.003400+0000 2021-07-31T04:07:37.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[34553]: mon.b (mon.2) 779 : audit [INF] 2021-07-31T04:07:37.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:37 smithi154 conmon[34553]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T04:07:37.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:37 smithi045 conmon[33655]: cluster 2021-07-31T04:07:35.803623+0000 mgr.x (mgr.24542) 2021-07-31T04:07:37.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:37 smithi045 conmon[33655]: 6011 : cluster [DBG] pgmap v6219: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:37.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:37 smithi045 conmon[33655]: audit 2021-07-31T04:07:37.000845+0000 mon.b ( 2021-07-31T04:07:37.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:37 smithi045 conmon[33655]: mon.2) 777 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T04:07:37.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:37 smithi045 conmon[33655]: audit 2021- 2021-07-31T04:07:37.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:37 smithi045 conmon[33655]: 07-31T04:07:37.002349 2021-07-31T04:07:37.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:37 smithi045 conmon[33655]: +0000 mon.b (mon.2) 778 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T04:07:37.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:37 smithi045 conmon[33655]: 2021-07-31T04:07:37.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:37 smithi045 conmon[33655]: audit 2021-07-31T04:07:37.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:37 smithi045 conmon[33655]: 2021-07-31T04:07:37.003400 2021-07-31T04:07:37.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:37 smithi045 conmon[33655]: +0000 mon.b (mon.2) 779 : audit [INF] 2021-07-31T04:07:37.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:37 smithi045 conmon[33655]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T04:07:37.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:37 smithi045 conmon[39718]: cluster 2021-07-31T04:07:35. 2021-07-31T04:07:37.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:37 smithi045 conmon[39718]: 803623+0000 mgr.x (mgr.24542) 6011 : cluster [DBG] pgmap v6219: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:37.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:37 smithi045 conmon[39718]: audit 2021-07-31T04:07: 2021-07-31T04:07:37.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:37 smithi045 conmon[39718]: 37.000845+0000 mon.b (mon.2) 777 : audit [DBG] 2021-07-31T04:07:37.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:37 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T04:07:37.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:37 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T04:07:37.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:37 smithi045 conmon[39718]: 31T04:07:37.002349+0000 mon.b (mon.2) 778 2021-07-31T04:07:37.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:37 smithi045 conmon[39718]: : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T04:07:37.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:37 smithi045 conmon[39718]: audit 2021-07 2021-07-31T04:07:37.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:37 smithi045 conmon[39718]: -31T04:07:37.003400+0000 mon.b (mon.2 2021-07-31T04:07:37.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:37 smithi045 conmon[39718]: ) 779 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T04:07:38.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: cephadm 2021-07-31T04:07:37.004206+0000 mgr.x (mgr.24542) 6012 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T04:07:38.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: 2021-07-31T04:07:38.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: cephadm 2021-07-31T04:07:37.047462+0000 mgr.x (mgr.24542) 6013 : cephadm [ERR] 2021-07-31T04:07:38.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:07:38.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:07:38.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: yield 2021-07-31T04:07:38.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:07:38.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:07:38.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:07:38.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:07:38.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:07:38.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:07:38.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:07:38.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:07:38.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:07:38.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: task.result() 2021-07-31T04:07:38.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:07:38.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: cephadm 2021-07-31T04:07:37.047946+0000 mgr.x (mgr.24542) 2021-07-31T04:07:38.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: 6014 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T04:07:38.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: cephadm 2021-07-31T04:07:37.058824+0000 mgr.x (mgr.24542) 6015 : cephadm [ERR] 2021-07-31T04:07:38.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:07:38.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:07:38.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: yield 2021-07-31T04:07:38.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:07:38.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:07:38.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:07:38.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:07:38.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:07:38.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:07:38.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:07:38.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:07:38.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:07:38.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: task.result() 2021-07-31T04:07:38.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:07:38.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: cephadm 2021-07-31T04:07:37.059697+0000 mgr.x (mgr.24542) 6016 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T04:07:38.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: cephadm 2021-07-31T04:07:37.084196+0000 mgr.x (mgr.24542) 6017 : cephadm [ERR] 2021-07-31T04:07:38.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:07:38.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:07:38.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: yield 2021-07-31T04:07:38.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:07:38.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:07:38.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:07:38.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:07:38.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:07:38.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:07:38.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:07:38.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:07:38.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:07:38.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: task.result() 2021-07-31T04:07:38.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:07:38.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: cephadm 2021-07-31T04:07:37.084683+0000 mgr.x (mgr.24542) 6018 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:07:38.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:07:38.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:07:38.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: yield 2021-07-31T04:07:38.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:07:38.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:07:38.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:07:38.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:07:38.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:07:38.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:07:38.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:07:38.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:07:38.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:07:38.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: task.result() 2021-07-31T04:07:38.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:07:38.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: 2021-07-31T04:07:38.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T04:07:38.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: 2021-07-31T04:07:38.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:07:38.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:07:38.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: return f(*arg) 2021-07-31T04:07:38.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:07:38.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:07:38.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:07:38.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:07:38.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:07:38.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:07:38.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:07:38.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T04:07:38.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:07:38.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: raise self._exception 2021-07-31T04:07:38.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:07:38.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:07:38.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:07:38.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T04:07:38.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:07:38.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: raise 2021-07-31T04:07:38.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:07:38.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T04:07:38.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:07:38.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:07:38.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:07:38.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: cephadm 2021-07-31T04:07:37.093354+0000 mgr.x (mgr.24542) 6019 : cephadm [ERR] 2021-07-31T04:07:38.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:07:38.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:07:38.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: yield 2021-07-31T04:07:38.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:07:38.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:07:38.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:07:38.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:07:38.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:07:38.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:07:38.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:07:38.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:07:38.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:07:38.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: task.result() 2021-07-31T04:07:38.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:07:38.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: cephadm 2021-07-31T04:07:37.093776+0000 mgr.x (mgr.24542) 6020 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:07:38.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:07:38.782 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:07:38.782 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: yield 2021-07-31T04:07:38.782 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:07:38.782 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:07:38.783 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:07:38.783 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:07:38.783 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:07:38.784 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:07:38.784 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:07:38.784 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:07:38.784 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:07:38.785 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: task.result() 2021-07-31T04:07:38.785 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:07:38.785 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: 2021-07-31T04:07:38.785 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T04:07:38.786 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: 2021-07-31T04:07:38.786 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:07:38.786 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:07:38.786 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: return f(*arg) 2021-07-31T04:07:38.787 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:07:38.787 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:07:38.787 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:07:38.787 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:07:38.788 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:07:38.788 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:07:38.788 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:07:38.788 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T04:07:38.789 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:07:38.789 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: raise self._exception 2021-07-31T04:07:38.789 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:07:38.789 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:07:38.789 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:07:38.790 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T04:07:38.790 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:07:38.790 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: raise 2021-07-31T04:07:38.791 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:07:38.791 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T04:07:38.791 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:07:38.791 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:07:38.791 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:38 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:07:38.792 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: cephadm 2021-07-31T04:07:37.004206+0000 mgr.x (mgr.24542) 2021-07-31T04:07:38.792 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: 6012 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T04:07:38.793 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: cephadm 2021-07-31T04:07:37.047462+0000 mgr.x (mgr.24542) 6013 : cephadm [ERR] 2021-07-31T04:07:38.793 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:07:38.793 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:07:38.793 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: yield 2021-07-31T04:07:38.794 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:07:38.794 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:07:38.794 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:07:38.794 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:07:38.795 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:07:38.795 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:07:38.795 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:07:38.795 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:07:38.795 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:07:38.796 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: task.result() 2021-07-31T04:07:38.796 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:07:38.796 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: cephadm 2021-07-31T04:07:38.796 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: 2021-07-31T04:07:37.047946+0000 mgr.x (mgr.24542) 6014 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T04:07:38.797 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: cephadm 2021-07-31T04:07:37.058824+0000 mgr.x (mgr.24542) 6015 : cephadm 2021-07-31T04:07:38.797 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: [ERR] 2021-07-31T04:07:38.797 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:07:38.797 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:07:38.798 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: yield 2021-07-31T04:07:38.798 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:07:38.798 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:07:38.798 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:07:38.798 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:07:38.799 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:07:38.799 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:07:38.799 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:07:38.799 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:07:38.800 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:07:38.800 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: task.result() 2021-07-31T04:07:38.800 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:07:38.800 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: cephadm 2021-07-31T04:07:37.059697+0000 mgr.x ( 2021-07-31T04:07:38.800 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: mgr.24542) 6016 : cephadm [INF] 2021-07-31T04:07:38.801 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T04:07:38.801 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: cephadm 2021-07-31T04:07:38.801 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: 2021-07 2021-07-31T04:07:38.801 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: -31T04:07:37 2021-07-31T04:07:38.801 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: .084196+0000 mgr.x (mgr. 2021-07-31T04:07:38.802 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: cephadm 2021-07-31T04:07:37 2021-07-31T04:07:38.802 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: .004206+0000 mgr.x (mgr.24542) 6012 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T04:07:38.802 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: cephadm 2021-07-31T04:07:38.803 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: 2021-07-31T04: 2021-07-31T04:07:38.803 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: 07:37.047462+0000 mgr.x (mgr. 2021-07-31T04:07:38.803 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: 24542) 6013 : cephadm [ERR] 2021-07-31T04:07:38.803 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:07:38.804 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:07:38.804 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: yield 2021-07-31T04:07:38.804 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:07:38.804 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:07:38.804 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:07:38.805 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:07:38.805 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:07:38.805 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:07:38.805 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:07:38.806 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:07:38.806 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:07:38.806 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: task.result() 2021-07-31T04:07:38.806 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:07:38.806 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: 2021-07-31T04:07:38.807 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: cephadm 2021-07-31T04:07:37.047946+0000 mgr.x (mgr.24542 2021-07-31T04:07:38.807 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: ) 6014 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T04:07:38.807 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: cephadm 2021- 2021-07-31T04:07:38.807 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: 07-31T04:07:37. 2021-07-31T04:07:38.808 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: 058824+0000 mgr.x (mgr.24542) 2021-07-31T04:07:38.808 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: 6015 : cephadm [ERR] 2021-07-31T04:07:38.808 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:07:38.808 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:07:38.808 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: yield 2021-07-31T04:07:38.809 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:07:38.809 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:07:38.809 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:07:38.809 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:07:38.809 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:07:38.810 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:07:38.810 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:07:38.810 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:07:38.810 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:07:38.811 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: task.result() 2021-07-31T04:07:38.811 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:07:38.811 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: 2021-07-31T04:07:38.811 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: cephadm 2021-07-31T04:07:37.059697+0000 mgr.x (mgr.24542) 6016 : 2021-07-31T04:07:38.812 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: 24542) 6017 : cephadm [ERR] 2021-07-31T04:07:38.812 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:07:38.812 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:07:38.812 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: yield 2021-07-31T04:07:38.813 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:07:38.813 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:07:38.813 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:07:38.813 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:07:38.814 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:07:38.814 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:07:38.814 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:07:38.814 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:07:38.815 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:07:38.815 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: task.result() 2021-07-31T04:07:38.815 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:07:38.815 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: cephadm 2021-07-31T04:07:37.084683+0000 mgr.x (mgr.24542) 6018 : cephadm [ERR] 2021-07-31T04:07:38.815 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:07:38.816 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:07:38.816 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:07:38.816 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: yield 2021-07-31T04:07:38.816 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:07:38.816 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:07:38.817 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:07:38.817 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:07:38.817 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:07:38.817 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:07:38.818 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:07:38.818 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:07:38.818 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:07:38.818 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: task.result() 2021-07-31T04:07:38.818 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:07:38.819 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: 2021-07-31T04:07:38.819 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T04:07:38.819 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: 2021-07-31T04:07:38.819 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:07:38.820 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:07:38.820 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: return f(*arg) 2021-07-31T04:07:38.820 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:07:38.820 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:07:38.820 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:07:38.821 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:07:38.821 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T04:07:38.821 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: cephadm 2021- 2021-07-31T04:07:38.822 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: 07-31T04:07:37. 2021-07-31T04:07:38.822 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: 084196+0000 mgr.x (mgr.24542 2021-07-31T04:07:38.822 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: ) 6017 : cephadm [ERR] 2021-07-31T04:07:38.822 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:07:38.822 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:07:38.823 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: yield 2021-07-31T04:07:38.823 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:07:38.823 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:07:38.823 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:07:38.823 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:07:38.824 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:07:38.824 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:07:38.824 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:07:38.824 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:07:38.825 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:07:38.825 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: task.result() 2021-07-31T04:07:38.825 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:07:38.825 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: 2021-07-31T04:07:38.825 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: cephadm 2021-07-31T04:07:37.084683+0000 mgr.x (mgr.24542) 2021-07-31T04:07:38.826 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: 6018 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:07:38.826 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:07:38.826 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:07:38.826 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: yield 2021-07-31T04:07:38.826 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:07:38.827 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:07:38.827 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:07:38.827 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:07:38.827 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:07:38.828 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:07:38.828 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:07:38.828 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:07:38.828 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:07:38.828 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: task.result() 2021-07-31T04:07:38.829 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:07:38.829 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: 2021-07-31T04:07:38.829 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T04:07:38.829 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: 2021-07-31T04:07:38.829 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:07:38.830 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:07:38.830 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: return f(*arg) 2021-07-31T04:07:38.830 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:07:38.830 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:07:38.831 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:07:38.831 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:07:38.831 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:07:38.832 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T04:07:38.832 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:07:38.832 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: raise self._exception 2021-07-31T04:07:38.832 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:07:38.832 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:07:38.833 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:07:38.833 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T04:07:38.833 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:07:38.833 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: raise 2021-07-31T04:07:38.834 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:07:38.834 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T04:07:38.834 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:07:38.834 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:07:38.834 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:07:38.835 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: cephadm 2021-07-31T04:07:37.093354+0000 mgr.x (mgr.24542) 6019 : cephadm [ERR] 2021-07-31T04:07:38.835 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:07:38.835 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:07:38.835 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: yield 2021-07-31T04:07:38.835 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:07:38.836 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:07:38.836 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:07:38.836 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:07:38.836 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:07:38.837 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:07:38.837 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:07:38.837 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:07:38.837 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:07:38.837 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: task.result() 2021-07-31T04:07:38.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:07:38.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: cephadm 2021-07-31T04:07:37.093776 2021-07-31T04:07:38.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 6020 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:07:38.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:07:38.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:07:38.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: yield 2021-07-31T04:07:38.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:07:38.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:07:38.839 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:07:38.840 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:07:38.840 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:07:38.840 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:07:38.840 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:07:38.841 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T04:07:38.841 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:07:38.841 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: raise self._exception 2021-07-31T04:07:38.841 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:07:38.841 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:07:38.842 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:07:38.842 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T04:07:38.842 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:07:38.842 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: raise 2021-07-31T04:07:38.843 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:07:38.843 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T04:07:38.843 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:07:38.843 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:07:38.843 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:07:38.844 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: cephadm 2021-07-31T04:07:37.093354+0000 mgr.x (mgr.24542) 6019 : cephadm [ERR] 2021-07-31T04:07:38.844 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:07:38.844 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:07:38.844 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: yield 2021-07-31T04:07:38.844 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:07:38.845 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:07:38.845 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:07:38.845 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:07:38.845 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:07:38.846 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:07:38.846 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:07:38.846 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:07:38.846 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:07:38.846 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: task.result() 2021-07-31T04:07:38.847 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:07:38.847 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: cephadm 2021-07-31T04:07:37.093776+0000 mgr.x (mgr.24542) 6020 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:07:38.847 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:07:38.847 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:07:38.847 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: yield 2021-07-31T04:07:38.848 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:07:38.848 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:07:38.848 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:07:38.849 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:07:38.849 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:07:38.849 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:07:38.849 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:07:38.849 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:07:38.850 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: task.result() 2021-07-31T04:07:38.850 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:07:38.850 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: 2021-07-31T04:07:38.850 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T04:07:38.851 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: 2021-07-31T04:07:38.851 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:07:38.851 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:07:38.851 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: return f(*arg) 2021-07-31T04:07:38.851 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:07:38.852 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:07:38.852 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:07:38.852 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:07:38.852 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:07:38.852 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:07:38.853 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:07:38.853 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T04:07:38.853 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:07:38.853 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: raise self._exception 2021-07-31T04:07:38.853 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:07:38.854 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:07:38.854 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:07:38.854 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T04:07:38.854 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:07:38.854 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: raise 2021-07-31T04:07:38.855 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:07:38.855 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T04:07:38.855 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:07:38.855 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:07:38.855 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:07:38.856 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:07:38.856 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:07:38.856 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:07:38.856 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:07:38.857 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:07:38.857 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:07:38.857 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:07:38.857 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:07:38.857 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: task.result() 2021-07-31T04:07:38.858 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:07:38.858 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: 2021-07-31T04:07:38.858 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T04:07:38.858 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: 2021-07-31T04:07:38.859 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:07:38.859 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:07:38.859 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: return f(*arg) 2021-07-31T04:07:38.859 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:07:38.859 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:07:38.860 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:07:38.860 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:07:38.860 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:07:38.860 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:07:38.860 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:07:38.861 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T04:07:38.861 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:07:38.861 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: raise self._exception 2021-07-31T04:07:38.861 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:07:38.861 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:07:38.862 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:07:38.862 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T04:07:38.862 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:07:38.862 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: raise 2021-07-31T04:07:38.863 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:07:38.863 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T04:07:38.863 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:07:38.863 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:07:38.863 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:38 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:07:39.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:39 smithi154 conmon[34553]: cluster 2021-07-31T04:07:37.804221+0000 mgr.x (mgr.24542 2021-07-31T04:07:39.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:39 smithi154 conmon[34553]: ) 6021 : cluster [DBG] pgmap v6220: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:39.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:39 smithi045 conmon[39718]: cluster 2021-07-31T04:07: 2021-07-31T04:07:39.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:39 smithi045 conmon[39718]: 37.804221+0000 mgr.x (mgr.24542) 6021 : cluster [DBG] pgmap v6220: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:39.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:39 smithi045 conmon[33655]: cluster 2021-07-31T04:07:37.804221+0000 mgr.x 2021-07-31T04:07:39.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:39 smithi045 conmon[33655]: (mgr.24542) 6021 : cluster [DBG] pgmap v6220: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:39.992 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T04:07:40.272 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T04:07:40.272 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 2h ago 2h - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T04:07:40.273 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 2h ago 2h 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T04:07:40.273 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 2h ago 2h 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T04:07:40.273 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 2h ago 2h - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T04:07:40.273 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T04:07:40.274 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 2h ago 2h 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T04:07:40.274 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T04:07:40.274 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 2h ago 2h - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T04:07:40.274 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 2h ago 2h 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T04:07:40.275 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T04:07:40.275 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T04:07:40.275 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T04:07:40.275 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T04:07:40.275 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 2h ago 2h 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T04:07:40.276 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 2h ago 2h 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T04:07:40.276 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 2h ago 2h 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T04:07:40.276 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 2h ago 2h 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T04:07:40.276 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 2h ago 2h 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T04:07:40.611 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T04:07:40.612 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T04:07:40.612 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T04:07:40.612 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:07:40.613 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T04:07:40.613 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T04:07:40.613 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:07:40.614 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:07:40.614 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T04:07:40.614 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T04:07:40.615 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:07:40.615 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T04:07:40.615 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T04:07:40.616 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T04:07:40.616 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:07:40.616 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T04:07:40.616 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T04:07:40.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:40 smithi154 conmon[34553]: cluster 2021-07-31T04:07:39.804711+0000 mgr.x (mgr.24542) 6022 : cluster [DBG] 2021-07-31T04:07:40.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:40 smithi154 conmon[34553]: pgmap v6221: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:40.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:40 smithi154 conmon[34553]: audit 2021-07-31T04:07:39.980889+0000 mgr.x (mgr.24542) 6023 : audit [DBG] from='client.30883 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:07:40.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:40 smithi154 conmon[34553]: audit 2021-07-31T04:07:40.264541+0000 mgr.x (mgr 2021-07-31T04:07:40.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:40 smithi154 conmon[34553]: .24542) 6024 : audit [DBG] from='client.30889 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:07:40.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:40 smithi045 conmon[33655]: cluster 2021-07-31T04:07:39.804711+0000 mgr.x (mgr.24542 2021-07-31T04:07:40.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:40 smithi045 conmon[33655]: ) 6022 : cluster [DBG] pgmap v6221: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:40.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:40 smithi045 conmon[33655]: audit 2021-07-31T04:07:39.980889+0000 mgr.x (mgr.24542) 6023 : audit [DBG] from='client.30883 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:07:40.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:40 smithi045 conmon[33655]: audit 2021-07-31 2021-07-31T04:07:40.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:40 smithi045 conmon[33655]: T04:07:40.264541+0000 mgr.x 2021-07-31T04:07:40.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:40 smithi045 conmon[33655]: (mgr.24542) 6024 : audit 2021-07-31T04:07:40.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:40 smithi045 conmon[33655]: [DBG] from='client.30889 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:07:40.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:40 smithi045 conmon[39718]: cluster 2021-07-31T04:07 2021-07-31T04:07:40.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:40 smithi045 conmon[39718]: :39.804711+0000 mgr.x (mgr.24542) 6022 : cluster [DBG] pgmap v6221: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:40.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:40 smithi045 conmon[39718]: audit 2021-07-31T04:07:40.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:40 smithi045 conmon[39718]: 2021-07-31T04:07:39. 2021-07-31T04:07:40.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:40 smithi045 conmon[39718]: 980889+0000 mgr.x (mgr.24542 2021-07-31T04:07:40.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:40 smithi045 conmon[39718]: ) 6023 : audit [DBG] from='client.30883 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:07:40.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:40 smithi045 conmon[39718]: 2021-07-31T04:07:40.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:40 smithi045 conmon[39718]: audit 2021-07-31T04:07:40.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:40 smithi045 conmon[39718]: 2021-07-31T04: 2021-07-31T04:07:40.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:40 smithi045 conmon[39718]: 07:40.264541+0000 mgr.x ( 2021-07-31T04:07:40.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:40 smithi045 conmon[39718]: mgr.24542) 6024 : audit 2021-07-31T04:07:40.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:40 smithi045 conmon[39718]: [DBG] from='client.30889 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:07:41.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:41 smithi154 conmon[34553]: audit 2021-07-31T04:07:40.612250+0000 mon.c (mon.1) 250 : audit [DBG] from='client.? 172.21.15.45:0/479156952' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:07:41.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:41 smithi045 conmon[39718]: audit 2021-07-31T04:07:40. 2021-07-31T04:07:41.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:41 smithi045 conmon[39718]: 612250+0000 mon.c (mon.1) 250 : audit [DBG] from='client.? 172.21.15.45:0/479156952' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:07:41.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:41 smithi045 conmon[33655]: audit 2021-07-31T04:07:40.612250+0000 mon.c (mon.1) 2021-07-31T04:07:41.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:41 smithi045 conmon[33655]: 250 : audit [DBG] from='client.? 172.21.15.45:0/479156952' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:07:42.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:42 smithi154 conmon[34553]: cluster 2021-07-31T04:07:41.805231+0000 mgr.x (mgr.24542) 6025 : cluster [DBG] 2021-07-31T04:07:42.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:42 smithi154 conmon[34553]: pgmap v6222: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:42.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:42 smithi045 conmon[33655]: cluster 2021-07-31T04:07:41.805231+0000 mgr.x (mgr.24542) 6025 : cluster 2021-07-31T04:07:42.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:42 smithi045 conmon[33655]: [DBG] pgmap v6222: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:42.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:42 smithi045 conmon[39718]: cluster 2021-07-31T04:07:41.805231+0000 mgr.x (mgr 2021-07-31T04:07:42.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:42 smithi045 conmon[39718]: .24542) 6025 : cluster [DBG] pgmap v6222: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:45.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:44 smithi154 conmon[34553]: cluster 2021-07-31T04:07:43.805716+0000 mgr.x (mgr.24542) 6026 : cluster [DBG] pgmap v6223: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:45.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:44 smithi045 conmon[33655]: cluster 2021-07-31T04:07:43.805716+0000 mgr.x (mgr.24542) 2021-07-31T04:07:45.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:44 smithi045 conmon[33655]: 6026 : cluster [DBG] pgmap v6223: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:45.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:44 smithi045 conmon[39718]: cluster 2021-07-31T04:07:43.805716+0000 mgr.x (mgr 2021-07-31T04:07:45.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:44 smithi045 conmon[39718]: .24542) 6026 : cluster [DBG] pgmap v6223: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:47.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:46 smithi154 conmon[34553]: cluster 2021-07-31T04:07:45.806364+0000 mgr.x (mgr.24542) 6027 : cluster 2021-07-31T04:07:47.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:46 smithi154 conmon[34553]: [DBG] pgmap v6224: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:47.274 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:46 smithi045 conmon[39718]: cluster 2021-07-31T04:07:45.806364+0000 mgr.x (mgr.24542) 2021-07-31T04:07:47.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:46 smithi045 conmon[39718]: 6027 : cluster [DBG] pgmap v6224: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:47.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:46 smithi045 conmon[33655]: cluster 2021-07-31T04:07:45.806364+0000 mgr.x ( 2021-07-31T04:07:47.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:46 smithi045 conmon[33655]: mgr.24542) 6027 : cluster [DBG] pgmap v6224: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:49.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:48 smithi154 conmon[34553]: cluster 2021-07-31T04:07:47.806844+0000 mgr.x (mgr.24542) 6028 : cluster [DBG] pgmap v6225: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:49.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:48 smithi045 conmon[39718]: cluster 2021-07-31T04:07:47.806844+0000 mgr.x (mgr 2021-07-31T04:07:49.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:48 smithi045 conmon[39718]: .24542) 6028 : cluster [DBG] pgmap v6225: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:49.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:48 smithi045 conmon[33655]: cluster 2021-07-31T04:07:47.806844 2021-07-31T04:07:49.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:48 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 6028 : cluster [DBG] pgmap v6225: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:51.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:50 smithi154 conmon[34553]: cluster 2021-07-31T04:07:49.807339+0000 mgr.x (mgr.24542) 6029 : cluster [DBG] pgmap v6226: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:51.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:50 smithi045 conmon[33655]: cluster 2021-07-31T04:07:49.807339+0000 mgr.x (mgr.24542) 6029 : cluster 2021-07-31T04:07:51.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:50 smithi045 conmon[33655]: [DBG] pgmap v6226: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:51.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:50 smithi045 conmon[39718]: cluster 2021-07-31T04:07:49.807339+0000 mgr.x (mgr. 2021-07-31T04:07:51.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:50 smithi045 conmon[39718]: 24542) 6029 : cluster [DBG] pgmap v6226: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:53.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:52 smithi154 conmon[34553]: cluster 2021-07-31T04:07:51.807700+0000 mgr.x (mgr.24542 2021-07-31T04:07:53.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:52 smithi154 conmon[34553]: ) 6030 : cluster [DBG] pgmap v6227: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:53.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:52 smithi045 conmon[33655]: cluster 2021-07-31T04:07:51.807700+0000 mgr.x (mgr.24542) 6030 : 2021-07-31T04:07:53.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:52 smithi045 conmon[33655]: cluster [DBG] pgmap v6227: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:53.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:52 smithi045 conmon[39718]: cluster 2021-07-31T04:07:51.807700+0000 mgr.x ( 2021-07-31T04:07:53.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:52 smithi045 conmon[39718]: mgr.24542) 6030 : cluster [DBG] pgmap v6227: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:55.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:54 smithi154 conmon[34553]: cluster 2021-07-31T04:07:53.808099+0000 mgr.x (mgr.24542) 6031 : cluster [DBG] pgmap v6228: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:55.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:54 smithi045 conmon[33655]: cluster 2021-07-31T04:07:53.808099+0000 mgr.x (mgr.24542) 6031 2021-07-31T04:07:55.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:54 smithi045 conmon[33655]: : cluster [DBG] pgmap v6228: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:55.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:54 smithi045 conmon[39718]: cluster 2021-07-31T04:07:53.808099+0000 mgr.x (mgr 2021-07-31T04:07:55.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:54 smithi045 conmon[39718]: .24542) 6031 : cluster [DBG] pgmap v6228: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:57.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:56 smithi154 conmon[34553]: cluster 2021-07-31T04:07:55.808596+0000 mgr.x (mgr.24542) 6032 : cluster [DBG] pgmap v6229: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:57.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:56 smithi045 conmon[39718]: cluster 2021-07-31T04:07:55.808596+0000 mgr.x ( 2021-07-31T04:07:57.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:56 smithi045 conmon[39718]: mgr.24542) 6032 : cluster [DBG] pgmap v6229: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:57.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:56 smithi045 conmon[33655]: cluster 2021-07-31T04:07:55.808596+0000 mgr.x (mgr.24542) 6032 : cluster [DBG] pgmap v6229: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:59.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:07:58 smithi154 conmon[34553]: cluster 2021-07-31T04:07:57.808965+0000 mgr.x (mgr.24542) 6033 : cluster [DBG] pgmap v6230: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:59.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:58 smithi045 conmon[39718]: cluster 2021-07-31T04:07:57.808965+0000 mgr.x ( 2021-07-31T04:07:59.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:07:58 smithi045 conmon[39718]: mgr.24542) 6033 : cluster [DBG] pgmap v6230: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:07:59.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:07:58 smithi045 conmon[33655]: cluster 2021-07-31T04:07:57.808965+0000 mgr.x (mgr.24542) 6033 : cluster [DBG] pgmap v6230: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:01.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:00 smithi154 conmon[34553]: cluster 2021-07-31T04:07:59.809395+0000 mgr.x (mgr.24542) 6034 : cluster [DBG] pgmap v6231: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:01.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:00 smithi045 conmon[33655]: cluster 2021-07-31T04:07:59.809395+0000 mgr.x (mgr.24542) 6034 : cluster [DBG] pgmap v6231: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:01.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:00 smithi045 conmon[39718]: cluster 2021-07-31T04:07:59.809395+0000 mgr.x (mgr.24542 2021-07-31T04:08:01.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:00 smithi045 conmon[39718]: ) 6034 : cluster [DBG] pgmap v6231: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:03.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:02 smithi154 conmon[34553]: cluster 2021-07-31T04:08:01.809839+0000 mgr.x (mgr.24542) 6035 : cluster [DBG] pgmap v6232: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:03.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:02 smithi154 conmon[34553]: 2021-07-31T04:08:03.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:02 smithi045 conmon[33655]: cluster 2021-07-31T04:08:01.809839+0000 2021-07-31T04:08:03.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:02 smithi045 conmon[33655]: mgr.x (mgr.24542) 6035 : cluster [DBG] pgmap v6232: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:03.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:02 smithi045 conmon[39718]: cluster 2021-07-31T04:08:01.809839+0000 mgr.x (mgr.24542 2021-07-31T04:08:03.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:02 smithi045 conmon[39718]: ) 6035 : cluster [DBG] pgmap v6232: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:05.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:04 smithi154 conmon[34553]: cluster 2021-07-31T04:08:03.810191+0000 mgr.x (mgr.24542) 6036 : cluster [DBG] pgmap v6233: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:05.274 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:04 smithi045 conmon[33655]: cluster 2021-07-31T04:08:03.810191+0000 mgr.x (mgr.24542) 2021-07-31T04:08:05.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:04 smithi045 conmon[33655]: 6036 : cluster [DBG] pgmap v6233: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:05.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:04 smithi045 conmon[39718]: cluster 2021-07-31T04:08:03.810191+0000 mgr.x (mgr.24542) 6036 2021-07-31T04:08:05.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:04 smithi045 conmon[39718]: : cluster [DBG] pgmap v6233: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:07.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:06 smithi154 conmon[34553]: cluster 2021-07-31T04:08:05.810650+0000 mgr.x (mgr.24542) 6037 : cluster [DBG] pgmap v6234: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:07.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:06 smithi154 conmon[34553]: 2021-07-31T04:08:07.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:06 smithi045 conmon[33655]: cluster 2021-07-31T04:08:05.810650+0000 mgr.x (mgr.24542) 2021-07-31T04:08:07.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:06 smithi045 conmon[33655]: 6037 : cluster [DBG] pgmap v6234: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:07.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:06 smithi045 conmon[39718]: cluster 2021-07-31T04:08:05.810650+0000 mgr.x ( 2021-07-31T04:08:07.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:06 smithi045 conmon[39718]: mgr.24542) 6037 : cluster [DBG] pgmap v6234: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:09.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:08 smithi154 conmon[34553]: cluster 2021-07-31T04:08:07.811099+0000 mgr.x (mgr.24542) 6038 : cluster [DBG] pgmap v6235: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:09.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:08 smithi045 conmon[39718]: cluster 2021-07-31T04:08:07.811099+0000 mgr.x (mgr. 2021-07-31T04:08:09.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:08 smithi045 conmon[39718]: 24542) 6038 : cluster [DBG] pgmap v6235: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:09.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:08 smithi045 conmon[33655]: cluster 2021-07-31T04:08:07.811099+0000 mgr.x (mgr.24542) 6038 : cluster [DBG] 2021-07-31T04:08:09.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:08 smithi045 conmon[33655]: pgmap v6235: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:10.926 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T04:08:11.203 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T04:08:11.203 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 2h ago 2h - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T04:08:11.203 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 2h ago 2h 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T04:08:11.204 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 2h ago 2h 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T04:08:11.204 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 2h ago 2h - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T04:08:11.204 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T04:08:11.205 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 2h ago 2h 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T04:08:11.205 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T04:08:11.205 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 2h ago 2h - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T04:08:11.205 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 2h ago 2h 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T04:08:11.206 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T04:08:11.206 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T04:08:11.206 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T04:08:11.207 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T04:08:11.207 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 2h ago 2h 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T04:08:11.207 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 2h ago 2h 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T04:08:11.208 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 2h ago 2h 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T04:08:11.208 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 2h ago 2h 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T04:08:11.208 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 2h ago 2h 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T04:08:11.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:10 smithi154 conmon[34553]: cluster 2021-07-31T04:08:09.811555+0000 mgr.x (mgr.24542) 6039 : cluster [DBG] pgmap v6236: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:11.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:10 smithi154 conmon[34553]: 2021-07-31T04:08:11.274 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:10 smithi045 conmon[39718]: cluster 2021-07-31T04:08:09.811555 2021-07-31T04:08:11.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:10 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 6039 : cluster [DBG] pgmap v6236: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:11.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:10 smithi045 conmon[33655]: cluster 2021-07-31T04:08:09. 2021-07-31T04:08:11.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:10 smithi045 conmon[33655]: 811555+0000 mgr.x (mgr.24542) 6039 : cluster [DBG] pgmap v6236: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:11.539 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T04:08:11.540 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T04:08:11.540 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T04:08:11.540 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:08:11.540 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T04:08:11.541 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T04:08:11.541 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:08:11.541 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:08:11.541 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T04:08:11.541 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T04:08:11.542 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:08:11.542 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T04:08:11.542 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T04:08:11.542 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T04:08:11.542 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:08:11.543 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T04:08:11.543 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T04:08:12.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:11 smithi154 conmon[34553]: audit 2021-07-31T04:08:10.914426+0000 mgr.x (mgr.24542) 6040 : audit [DBG] from='client.20964 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:08:12.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:11 smithi154 conmon[34553]: audit 2021-07-31T04:08:11.195339+0000 mgr.x (mgr.24542) 6041 : audit [DBG] from='client.30907 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:08:12.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:11 smithi154 conmon[34553]: audit 2021-07-31T04:08:11.540598+0000 mon.c (mon 2021-07-31T04:08:12.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:11 smithi154 conmon[34553]: .1) 251 : audit [DBG] from='client.? 172.21.15.45:0/4247738665' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:08:12.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:11 smithi045 conmon[33655]: audit 2021-07-31T04:08:10.914426+0000 mgr.x (mgr 2021-07-31T04:08:12.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:11 smithi045 conmon[33655]: .24542) 6040 : audit [DBG] from='client.20964 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:08:12.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:11 smithi045 conmon[33655]: audit 2021-07-31T04:08:11.195339+0000 mgr.x (mgr.24542) 6041 : audit [DBG] from='client.30907 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:08:12.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:11 smithi045 conmon[33655]: audit 2021- 2021-07-31T04:08:12.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:11 smithi045 conmon[33655]: 07-31T04:08:11.540598+0000 mon.c (mon.1) 251 : audit [DBG] from='client.? 172.21.15.45:0/4247738665' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:08:12.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:11 smithi045 conmon[39718]: audit 2021-07-31T04:08:10.914426 2021-07-31T04:08:12.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:11 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 6040 : audit [DBG] from='client.20964 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:08:12.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:11 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T04:08:12.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:11 smithi045 conmon[39718]: 04:08:11.195339+0000 2021-07-31T04:08:12.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:11 smithi045 conmon[39718]: mgr.x (mgr.24542) 6041 : audit [DBG] 2021-07-31T04:08:12.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:11 smithi045 conmon[39718]: from='client.30907 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:08:12.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:11 smithi045 conmon[39718]: audit 2021- 2021-07-31T04:08:12.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:11 smithi045 conmon[39718]: 07-31T04:08:11 2021-07-31T04:08:12.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:11 smithi045 conmon[39718]: .540598+0000 mon.c (mon 2021-07-31T04:08:12.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:11 smithi045 conmon[39718]: .1) 251 : audit [DBG] from='client.? 172.21.15.45:0/4247738665' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:08:13.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:12 smithi154 conmon[34553]: cluster 2021-07-31T04:08:11.811902+0000 mgr.x (mgr.24542) 6042 : cluster [DBG] pgmap v6237: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:13.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:12 smithi045 conmon[33655]: cluster 2021-07-31T04:08:11.811902+0000 mgr.x (mgr. 2021-07-31T04:08:13.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:12 smithi045 conmon[33655]: 24542) 6042 : cluster [DBG] pgmap v6237: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:13.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:12 smithi045 conmon[39718]: cluster 2021-07-31T04:08:11.811902+0000 mgr.x ( 2021-07-31T04:08:13.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:12 smithi045 conmon[39718]: mgr.24542) 6042 : cluster [DBG] pgmap v6237: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:15.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:14 smithi154 conmon[34553]: cluster 2021-07-31T04:08:13.812345+0000 mgr.x (mgr.24542) 6043 : cluster [DBG] pgmap v6238: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:15.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:14 smithi045 conmon[33655]: cluster 2021-07-31T04:08:13.812345+0000 2021-07-31T04:08:15.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:14 smithi045 conmon[33655]: mgr.x (mgr.24542) 6043 : cluster [DBG] pgmap v6238: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:15.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:14 smithi045 conmon[39718]: cluster 2021-07-31T04:08:13.812345+0000 mgr.x 2021-07-31T04:08:15.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:14 smithi045 conmon[39718]: (mgr.24542) 6043 : cluster [DBG] pgmap v6238: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:17.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:16 smithi154 conmon[34553]: cluster 2021-07-31T04:08:15.812888+0000 mgr.x (mgr.24542) 6044 : cluster [DBG] pgmap v6239: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:17.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:16 smithi045 conmon[33655]: cluster 2021-07-31T04:08:15.812888 2021-07-31T04:08:17.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:16 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 6044 : cluster [DBG] pgmap v6239: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:17.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:16 smithi045 conmon[39718]: cluster 2021-07-31T04:08:15.812888+0000 2021-07-31T04:08:17.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:16 smithi045 conmon[39718]: mgr.x (mgr.24542) 6044 : cluster [DBG] pgmap v6239: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:19.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:18 smithi154 conmon[34553]: cluster 2021-07-31T04:08:17.813407+0000 mgr.x (mgr.24542) 6045 : cluster [DBG] pgmap v6240: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:19.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:18 smithi045 conmon[39718]: cluster 2021-07-31T04:08:17.813407+0000 mgr.x (mgr 2021-07-31T04:08:19.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:18 smithi045 conmon[39718]: .24542) 6045 : cluster [DBG] pgmap v6240: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:19.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:18 smithi045 conmon[33655]: cluster 2021-07-31T04:08:17.813407+0000 mgr.x (mgr.24542) 6045 2021-07-31T04:08:19.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:18 smithi045 conmon[33655]: : cluster [DBG] pgmap v6240: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:21.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:20 smithi154 conmon[34553]: cluster 2021-07-31T04:08:19.813883+0000 mgr.x (mgr.24542) 6046 : cluster [DBG] pgmap v6241: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:21.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:20 smithi045 conmon[39718]: cluster 2021-07-31T04:08:19.813883+0000 mgr.x (mgr.24542) 6046 : cluster 2021-07-31T04:08:21.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:20 smithi045 conmon[39718]: [DBG] pgmap v6241: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:21.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:20 smithi045 conmon[33655]: cluster 2021-07-31T04:08:19.813883+0000 mgr.x (mgr.24542) 6046 : cluster [DBG] 2021-07-31T04:08:21.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:20 smithi045 conmon[33655]: pgmap v6241: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:23.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:22 smithi154 conmon[34553]: cluster 2021-07-31T04:08:21.814433+0000 mgr.x (mgr.24542) 6047 : cluster [DBG] pgmap v6242: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:23.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:22 smithi045 conmon[39718]: cluster 2021-07-31T04:08:21.814433+0000 mgr.x (mgr.24542) 6047 : cluster [DBG] 2021-07-31T04:08:23.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:22 smithi045 conmon[39718]: pgmap v6242: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:23.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:22 smithi045 conmon[33655]: cluster 2021-07-31T04:08:21.814433+0000 mgr.x (mgr.24542) 6047 : cluster [DBG] pgmap v6242: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:25.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:24 smithi154 conmon[34553]: cluster 2021-07-31T04:08:23.814879+0000 mgr.x (mgr.24542) 6048 : cluster [DBG] pgmap v6243: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:25.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:24 smithi045 conmon[33655]: cluster 2021-07-31T04:08:23.814879+0000 mgr.x (mgr.24542) 6048 : cluster [DBG] pgmap v6243: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:25.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:24 smithi045 conmon[39718]: cluster 2021-07-31T04:08:23.814879+0000 mgr.x (mgr.24542) 6048 : cluster [DBG] 2021-07-31T04:08:25.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:24 smithi045 conmon[39718]: pgmap v6243: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:27.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:26 smithi154 conmon[34553]: cluster 2021-07-31T04:08:25.815362+0000 mgr.x (mgr.24542) 6049 : cluster [DBG] 2021-07-31T04:08:27.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:26 smithi154 conmon[34553]: pgmap v6244: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:27.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:26 smithi045 conmon[39718]: cluster 2021-07-31T04:08:25.815362+0000 mgr.x (mgr.24542) 6049 : cluster 2021-07-31T04:08:27.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:26 smithi045 conmon[39718]: [DBG] pgmap v6244: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:27.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:26 smithi045 conmon[33655]: cluster 2021-07-31T04:08:25.815362+0000 2021-07-31T04:08:27.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:26 smithi045 conmon[33655]: mgr.x (mgr.24542) 6049 : cluster [DBG] pgmap v6244: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:29.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:28 smithi154 conmon[34553]: cluster 2021-07-31T04:08:27.815807+0000 mgr.x (mgr.24542) 6050 : cluster [DBG] pgmap v6245: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:29.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:28 smithi154 conmon[34553]: audit 2021-07-31T04:08:28.491562+0000 mon.b (mon.2) 780 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:08:29.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:28 smithi154 conmon[34553]: audit 2021-07-31T04:08:28.492154+0000 mon.b (mon.2) 781 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:08:29.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:28 smithi154 conmon[34553]: audit 2021-07-31T04:08:28.492478+0000 mon.a (mon.0) 3427 2021-07-31T04:08:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:28 smithi154 conmon[34553]: : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:08:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:28 smithi154 conmon[34553]: audit 2021-07 2021-07-31T04:08:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:28 smithi154 conmon[34553]: -31T04:08:28.492867+0000 mon.a ( 2021-07-31T04:08:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:28 smithi154 conmon[34553]: mon.0) 3428 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:08:29.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:28 smithi154 conmon[34553]: 2021-07-31T04:08:29.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:28 smithi045 conmon[39718]: cluster 2021-07-31T04:08:27.815807+0000 mgr.x ( 2021-07-31T04:08:29.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:28 smithi045 conmon[39718]: mgr.24542) 6050 : cluster [DBG] pgmap v6245: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:29.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:28 smithi045 conmon[39718]: audit 2021-07-31T04:08:28.491562+0000 mon.b ( 2021-07-31T04:08:29.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:28 smithi045 conmon[39718]: mon.2) 780 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:08:29.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:28 smithi045 conmon[39718]: audit 2021-07-31T04 2021-07-31T04:08:29.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:28 smithi045 conmon[39718]: :08:28.492154+0000 mon.b (mon. 2021-07-31T04:08:29.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:28 smithi045 conmon[39718]: 2) 781 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:08:29.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:28 smithi045 conmon[39718]: audit 2021-07-31T04:08:28.492478+0000 mon.a (mon 2021-07-31T04:08:29.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:28 smithi045 conmon[39718]: .0) 3427 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:08:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:28 smithi045 conmon[39718]: audit 2021-07 2021-07-31T04:08:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:28 smithi045 conmon[39718]: -31T04:08:28.492867 2021-07-31T04:08:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:28 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2021-07-31T04:08:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:28 smithi045 conmon[39718]: 3428 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:08:29.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:28 smithi045 conmon[33655]: cluster 2021-07-31T04:08:27.815807+0000 mgr.x (mgr.24542) 6050 2021-07-31T04:08:29.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:28 smithi045 conmon[33655]: : cluster [DBG] pgmap v6245: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:29.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:28 smithi045 conmon[33655]: audit 2021-07-31T04:08:28.491562+0000 mon.b (mon.2) 780 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:08:29.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:28 smithi045 conmon[33655]: audit 2021-07-31T04:08:28.492154+0000 mon.b ( 2021-07-31T04:08:29.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:28 smithi045 conmon[33655]: mon.2) 781 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:08:29.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:28 smithi045 conmon[33655]: audit 2021-07-31T04:08:28.492478+0000 mon.a (mon. 2021-07-31T04:08:29.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:28 smithi045 conmon[33655]: 0) 3427 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:08:29.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:28 smithi045 conmon[33655]: audit 2021-07-31T04:08:29.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:28 smithi045 conmon[33655]: 2021-07-31T04:08:28.492867+0000 mon.a (mon.0) 3428 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:08:29.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:28 smithi045 conmon[33655]: 2021-07-31T04:08:31.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:30 smithi154 conmon[34553]: cluster 2021-07-31T04:08:29.816318+0000 mgr.x (mgr.24542) 6051 : cluster [DBG] pgmap v6246: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:31.274 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:30 smithi045 conmon[39718]: cluster 2021-07-31T04:08:29.816318 2021-07-31T04:08:31.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:30 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 6051 : cluster [DBG] pgmap v6246: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:31.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:30 smithi045 conmon[33655]: cluster 2021-07-31T04:08:29.816318+0000 mgr.x (mgr.24542) 6051 : cluster [DBG] pgmap v6246: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:33.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:32 smithi154 conmon[34553]: cluster 2021-07-31T04:08:31.816835+0000 mgr.x (mgr.24542) 6052 : cluster [DBG] pgmap v6247: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:33.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:32 smithi045 conmon[39718]: cluster 2021-07-31T04:08:31. 2021-07-31T04:08:33.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:32 smithi045 conmon[39718]: 816835+0000 mgr.x (mgr.24542) 6052 : cluster [DBG] pgmap v6247: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:33.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:32 smithi045 conmon[33655]: cluster 2021-07-31T04:08:31.816835+0000 mgr.x (mgr.24542) 6052 2021-07-31T04:08:33.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:32 smithi045 conmon[33655]: : cluster [DBG] pgmap v6247: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:35.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:34 smithi154 conmon[34553]: cluster 2021-07-31T04:08:33.817240+0000 mgr.x (mgr.24542) 6053 : cluster [DBG] pgmap v6248: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:35.274 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:34 smithi045 conmon[39718]: cluster 2021-07-31T04:08:33.817240+0000 mgr.x (mgr.24542) 6053 : cluster [DBG] pgmap v6248: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:35.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:34 smithi045 conmon[33655]: cluster 2021-07-31T04:08:33.817240+0000 mgr.x (mgr.24542) 6053 : 2021-07-31T04:08:35.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:34 smithi045 conmon[33655]: cluster [DBG] pgmap v6248: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:37.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:36 smithi154 conmon[34553]: cluster 2021-07-31T04:08:35.817754+0000 mgr.x (mgr.24542) 6054 : cluster [DBG] pgmap v6249: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:37.253 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: debug 2021-07-31T04:08:37.140+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:08:37.253 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:08:37.253 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:08:37.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: yield 2021-07-31T04:08:37.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:08:37.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:08:37.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:08:37.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:08:37.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:08:37.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:08:37.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:08:37.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:08:37.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:08:37.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: task.result() 2021-07-31T04:08:37.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:08:37.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: debug 2021-07-31T04:08:37.151+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:08:37.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:08:37.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:08:37.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: yield 2021-07-31T04:08:37.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:08:37.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:08:37.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:08:37.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:08:37.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:08:37.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:08:37.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:08:37.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:08:37.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:08:37.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: task.result() 2021-07-31T04:08:37.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:08:37.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: debug 2021-07-31T04:08:37.177+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:08:37.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:08:37.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:08:37.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: yield 2021-07-31T04:08:37.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:08:37.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:08:37.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:08:37.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:08:37.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:08:37.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:08:37.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:08:37.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:08:37.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:08:37.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: task.result() 2021-07-31T04:08:37.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:08:37.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: debug 2021-07-31T04:08:37.177+0000 7fcb9295f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:08:37.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:08:37.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:08:37.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: yield 2021-07-31T04:08:37.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:08:37.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:08:37.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:08:37.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:08:37.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:08:37.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:08:37.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:08:37.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:08:37.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:08:37.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: task.result() 2021-07-31T04:08:37.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:08:37.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: 2021-07-31T04:08:37.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T04:08:37.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: 2021-07-31T04:08:37.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:08:37.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:08:37.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: return f(*arg) 2021-07-31T04:08:37.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:08:37.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:08:37.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:08:37.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:08:37.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:08:37.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:08:37.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:08:37.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T04:08:37.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:08:37.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: raise self._exception 2021-07-31T04:08:37.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:08:37.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:08:37.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:08:37.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T04:08:37.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:08:37.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: raise 2021-07-31T04:08:37.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:08:37.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T04:08:37.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:08:37.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:08:37.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:08:37.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: debug 2021-07-31T04:08:37.186+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:08:37.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:08:37.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:08:37.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: yield 2021-07-31T04:08:37.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:08:37.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:08:37.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:08:37.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:08:37.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:08:37.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:08:37.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:08:37.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:08:37.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:08:37.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: task.result() 2021-07-31T04:08:37.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:08:37.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: debug 2021-07-31T04:08:37.187+0000 7fcb93160700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:08:37.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:08:37.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:08:37.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: yield 2021-07-31T04:08:37.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:08:37.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:08:37.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:08:37.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:08:37.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:08:37.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:08:37.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:08:37.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:08:37.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:08:37.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: task.result() 2021-07-31T04:08:37.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:08:37.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: 2021-07-31T04:08:37.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T04:08:37.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: 2021-07-31T04:08:37.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:08:37.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:08:37.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: return f(*arg) 2021-07-31T04:08:37.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:08:37.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:08:37.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:08:37.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:08:37.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:08:37.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:08:37.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:08:37.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T04:08:37.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:08:37.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: raise self._exception 2021-07-31T04:08:37.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:08:37.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:08:37.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:08:37.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T04:08:37.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:08:37.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: raise 2021-07-31T04:08:37.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:08:37.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T04:08:37.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:08:37.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:08:37.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:08:37.283 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:36 smithi045 conmon[33655]: cluster 2021-07-31T04:08 2021-07-31T04:08:37.283 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:36 smithi045 conmon[33655]: :35.817754+0000 mgr.x (mgr.24542) 6054 : cluster [DBG] pgmap v6249: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:37.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:36 smithi045 conmon[39718]: cluster 2021-07- 2021-07-31T04:08:37.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:36 smithi045 conmon[39718]: 31T04:08:35.817754+0000 mgr.x (mgr.24542) 6054 : cluster [DBG] pgmap v6249: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:38.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: audit 2021-07-31T04:08:37.094589+0000 mon.b (mon.2) 782 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T04:08:38.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: 2021-07-31T04:08:38.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: audit 2021-07-31T04:08:37.095997+0000 mon.b (mon.2) 783 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T04:08:38.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: audit 2021-07-31T04:08:37.097072+0000 mon.b (mon.2) 784 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T04:08:38.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: cephadm 2021-07-31T04:08:37.097844+0000 mgr.x (mgr.24542) 6055 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T04:08:38.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: cephadm 2021-07-31T04:08:37 2021-07-31T04:08:38.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: .141088+0000 mgr.x (mgr.24542) 6056 : cephadm [ERR] 2021-07-31T04:08:38.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:08:38.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:08:38.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: yield 2021-07-31T04:08:38.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:08:38.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:08:38.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:08:38.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:08:38.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:08:38.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:08:38.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:08:38.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:08:38.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:08:38.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: task.result() 2021-07-31T04:08:38.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:08:38.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: cephadm 2021-07-31T04:08:37.141701+0000 mgr.x (mgr.24542) 6057 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T04:08:38.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: cephadm 2021-07-31T04:08:37.152716+0000 mgr.x (mgr.24542) 6058 : cephadm [ERR] 2021-07-31T04:08:38.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:08:38.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:08:38.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: yield 2021-07-31T04:08:38.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:08:38.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:08:38.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:08:38.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:08:38.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:08:38.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:08:38.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:08:38.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:08:38.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:08:38.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: task.result() 2021-07-31T04:08:38.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:08:38.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: cephadm 2021-07-31T04:08:38.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: 2021-07-31T04:08:37.153596+0000 mgr.x (mgr.24542) 6059 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T04:08:38.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: cephadm 2021-07-31T04:08:37.177929+0000 mgr.x (mgr.24542) 6060 : cephadm [ERR] 2021-07-31T04:08:38.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:08:38.262 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:08:38.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: yield 2021-07-31T04:08:38.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:08:38.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:08:38.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:08:38.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:08:38.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:08:38.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:08:38.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:08:38.264 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:08:38.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:08:38.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: task.result() 2021-07-31T04:08:38.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:08:38.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: cephadm 2021-07-31T04:08:37. 2021-07-31T04:08:38.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: 178382+0000 mgr.x (mgr.24542) 6061 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:08:38.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:08:38.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:08:38.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: yield 2021-07-31T04:08:38.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:08:38.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:08:38.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:08:38.267 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:08:38.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:08:38.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:08:38.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:08:38.268 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:08:38.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:08:38.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: task.result() 2021-07-31T04:08:38.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:08:38.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: 2021-07-31T04:08:38.269 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T04:08:38.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: 2021-07-31T04:08:38.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:08:38.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:08:38.270 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: return f(*arg) 2021-07-31T04:08:38.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:08:38.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:08:38.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:08:38.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:08:38.271 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:08:38.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:08:38.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:08:38.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T04:08:38.272 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:08:38.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: raise self._exception 2021-07-31T04:08:38.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:08:38.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:08:38.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:08:38.273 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T04:08:38.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:08:38.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: raise 2021-07-31T04:08:38.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:08:38.274 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T04:08:38.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:08:38.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:08:38.275 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:08:38.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: cephadm 2021-07-31T04:08:37.187137+0000 mgr.x (mgr.24542) 6062 : cephadm [ERR] 2021-07-31T04:08:38.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:08:38.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:08:38.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: yield 2021-07-31T04:08:38.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:08:38.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:08:38.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:08:38.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:08:38.277 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:08:38.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:08:38.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:08:38.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:08:38.278 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:08:38.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: task.result() 2021-07-31T04:08:38.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:08:38.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: cephadm 2021-07- 2021-07-31T04:08:38.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: 31T04:08:37.188241+0000 mgr.x (mgr.24542) 6063 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:08:38.279 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:08:38.280 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:08:38.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: audit 2021-07-31T04:08:37.094589+0000 mon.b ( 2021-07-31T04:08:38.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: mon.2) 782 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T04:08:38.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: audit 2021-07-31T04:08 2021-07-31T04:08:38.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: :37. 2021-07-31T04:08:38.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: 095997+0000 mon.b (mon. 2021-07-31T04:08:38.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: 2) 783 : audit [DBG] 2021-07-31T04:08:38.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T04:08:38.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: audit 2021-07-31T04:08:38.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: 2021-07-31T04:08:38.283 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: 2021-07-31T04:08 2021-07-31T04:08:38.283 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: :37.097072+0000 mon.b 2021-07-31T04:08:38.283 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: (mon.2) 784 : 2021-07-31T04:08:38.283 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T04:08:38.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: cephadm 2021 2021-07-31T04:08:38.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: -07-31T04:08:37.097844+0000 mgr.x (mgr 2021-07-31T04:08:38.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: .24542) 6055 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T04:08:38.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: cephadm 2021 2021-07-31T04:08:38.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: -07-31T04:08:37. 2021-07-31T04:08:38.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: 141088+0000 mgr.x (mgr.24542 2021-07-31T04:08:38.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: ) 6056 : cephadm [ERR] 2021-07-31T04:08:38.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:08:38.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:08:38.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: yield 2021-07-31T04:08:38.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:08:38.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:08:38.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:08:38.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:08:38.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:08:38.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:08:38.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:08:38.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:08:38.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:08:38.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: task.result() 2021-07-31T04:08:38.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:08:38.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: 2021-07-31T04:08:38.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: cephadm 2021-07-31T04:08:37.141701+0000 mgr.x (mgr.24542) 6057 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T04:08:38.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: 2021-07-31T04:08:38.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: cephadm 2021-07-31T04:08:38.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: 2021-07-31T04:08:37.152716+0000 2021-07-31T04:08:38.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: mgr.x (mgr.24542) 6058 : cephadm 2021-07-31T04:08:38.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: [ERR] 2021-07-31T04:08:38.290 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:08:38.290 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:08:38.290 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: yield 2021-07-31T04:08:38.290 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:08:38.291 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: yield 2021-07-31T04:08:38.291 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:08:38.291 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:08:38.291 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:08:38.292 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:08:38.292 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:08:38.292 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:08:38.292 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:08:38.292 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:08:38.293 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:08:38.293 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: task.result() 2021-07-31T04:08:38.293 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:08:38.293 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: 2021-07-31T04:08:38.293 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T04:08:38.294 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: 2021-07-31T04:08:38.294 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:08:38.294 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:08:38.294 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: return f(*arg) 2021-07-31T04:08:38.294 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:08:38.295 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:08:38.295 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:08:38.295 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:08:38.295 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:08:38.296 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:08:38.296 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:08:38.296 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T04:08:38.296 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:08:38.296 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: raise self._exception 2021-07-31T04:08:38.297 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:08:38.297 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:08:38.297 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:08:38.297 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T04:08:38.297 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:08:38.298 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: raise 2021-07-31T04:08:38.298 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:08:38.298 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T04:08:38.298 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:08:38.298 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:08:38.299 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:37 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:08:38.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: audit 2021-07-31T04:08: 2021-07-31T04:08:38.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: 37.094589+0000 mon.b (mon.2) 782 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T04:08:38.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: audit 2021 2021-07-31T04:08:38.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: -07-31T04:08: 2021-07-31T04:08:38.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: 37.095997+0000 mon.b (mon 2021-07-31T04:08:38.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: .2) 783 : audit [DBG] 2021-07-31T04:08:38.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T04:08:38.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: audit 2021- 2021-07-31T04:08:38.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: 07-31T04:08:37. 2021-07-31T04:08:38.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: 097072+0000 mon.b (mon.2 2021-07-31T04:08:38.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: ) 784 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T04:08:38.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: 2021-07-31T04:08:38.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: cephadm 2021-07-31T04:08:38.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: 2021-07-31T04: 2021-07-31T04:08:38.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: 08:37.097844+0000 mgr.x 2021-07-31T04:08:38.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: (mgr.24542) 6055 : cephadm 2021-07-31T04:08:38.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T04:08:38.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: cephadm 2021- 2021-07-31T04:08:38.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: 07-31T04:08:37. 2021-07-31T04:08:38.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: 141088+0000 mgr.x (mgr.24542) 2021-07-31T04:08:38.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: 6056 : cephadm [ERR] 2021-07-31T04:08:38.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:08:38.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:08:38.304 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: yield 2021-07-31T04:08:38.304 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:08:38.304 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:08:38.304 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:08:38.304 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:08:38.305 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:08:38.305 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:08:38.305 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:08:38.305 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:08:38.305 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:08:38.306 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: task.result() 2021-07-31T04:08:38.306 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:08:38.306 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: cephadm 2021-07-31T04:08:37.141701+0000 mgr.x (mgr.24542) 6057 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T04:08:38.306 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: cephadm 2021-07-31T04 2021-07-31T04:08:38.306 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: :08:37.152716+0000 mgr.x (mgr.24542 2021-07-31T04:08:38.307 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: ) 6058 : cephadm 2021-07-31T04:08:38.307 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: [ERR] 2021-07-31T04:08:38.307 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:08:38.307 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:08:38.308 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: yield 2021-07-31T04:08:38.308 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:08:38.308 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:08:38.308 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:08:38.309 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:08:38.309 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:08:38.309 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:08:38.309 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:08:38.310 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:08:38.310 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:08:38.310 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: task.result() 2021-07-31T04:08:38.310 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:08:38.310 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: cephadm 2021-07-31T04:08:37.153596+0000 mgr.x (mgr.24542) 6059 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T04:08:38.311 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: cephadm 2021-07-31T04:08:38.311 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: 2021-07-31T04: 2021-07-31T04:08:38.311 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: 08:37.177929+0000 mgr.x 2021-07-31T04:08:38.311 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: (mgr.24542) 6060 : cephadm 2021-07-31T04:08:38.311 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: [ERR] 2021-07-31T04:08:38.312 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:08:38.312 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:08:38.312 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: yield 2021-07-31T04:08:38.312 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:08:38.312 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:08:38.313 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:08:38.313 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:08:38.313 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:08:38.313 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:08:38.313 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:08:38.314 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:08:38.314 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:08:38.314 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: task.result() 2021-07-31T04:08:38.314 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:08:38.315 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: cephadm 2021-07-31T04:08:37.178382+0000 mgr.x (mgr.24542) 6061 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:08:38.315 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:08:38.315 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:08:38.315 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: yield 2021-07-31T04:08:38.315 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:08:38.316 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:08:38.316 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:08:38.316 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:08:38.316 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:08:38.316 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:08:38.316 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:08:38.317 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:08:38.317 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:08:38.317 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:08:38.318 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:08:38.318 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:08:38.318 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:08:38.318 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:08:38.319 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:08:38.319 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: task.result() 2021-07-31T04:08:38.319 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:08:38.319 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: cephadm 2021-07-31T04:08:37.153596+0000 mgr.x (mgr.24542) 6059 2021-07-31T04:08:38.319 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T04:08:38.320 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: cephadm 2021- 2021-07-31T04:08:38.320 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: 07-31T04:08:37.177929+0000 2021-07-31T04:08:38.320 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: mgr.x (mgr.24542) 6060 : cephadm 2021-07-31T04:08:38.320 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: [ERR] 2021-07-31T04:08:38.320 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:08:38.321 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:08:38.321 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: yield 2021-07-31T04:08:38.321 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:08:38.321 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:08:38.321 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:08:38.322 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:08:38.322 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:08:38.322 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:08:38.322 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:08:38.323 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:08:38.323 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:08:38.323 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: task.result() 2021-07-31T04:08:38.323 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:08:38.323 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: cephadm 2021-07-31T04:08:37.178382+0000 mgr.x (mgr.24542) 6061 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:08:38.324 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:08:38.324 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:08:38.324 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: yield 2021-07-31T04:08:38.324 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:08:38.324 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:08:38.325 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:08:38.325 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:08:38.325 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:08:38.325 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:08:38.325 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:08:38.326 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:08:38.326 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:08:38.326 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:08:38.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: task.result() 2021-07-31T04:08:38.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:08:38.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: 2021-07-31T04:08:38.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T04:08:38.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: 2021-07-31T04:08:38.328 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:08:38.328 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:08:38.328 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: return f(*arg) 2021-07-31T04:08:38.328 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:08:38.328 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:08:38.329 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:08:38.329 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:08:38.329 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:08:38.329 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:08:38.329 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:08:38.330 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T04:08:38.330 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:08:38.330 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: raise self._exception 2021-07-31T04:08:38.330 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:08:38.330 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:08:38.331 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:08:38.331 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T04:08:38.331 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:08:38.331 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: raise 2021-07-31T04:08:38.331 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:08:38.332 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T04:08:38.332 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:08:38.332 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:08:38.332 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:08:38.332 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: cephadm 2021-07-31T04:08:37.187137+0000 mgr.x (mgr.24542) 6062 : cephadm [ERR] 2021-07-31T04:08:38.333 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:08:38.333 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:08:38.333 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: yield 2021-07-31T04:08:38.333 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:08:38.333 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:08:38.334 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:08:38.334 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:08:38.334 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:08:38.334 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:08:38.335 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:08:38.335 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: task.result() 2021-07-31T04:08:38.335 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:08:38.335 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: 2021-07-31T04:08:38.336 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T04:08:38.336 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: 2021-07-31T04:08:38.336 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:08:38.336 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:08:38.337 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: return f(*arg) 2021-07-31T04:08:38.337 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:08:38.337 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:08:38.337 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:08:38.337 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:08:38.338 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:08:38.338 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:08:38.338 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:08:38.338 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T04:08:38.339 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:08:38.339 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: raise self._exception 2021-07-31T04:08:38.339 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:08:38.339 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:08:38.339 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:08:38.340 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T04:08:38.340 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:08:38.340 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: raise 2021-07-31T04:08:38.340 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:08:38.340 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T04:08:38.341 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:08:38.341 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:08:38.341 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:08:38.341 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: 2021-07-31T04:08:38.341 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: cephadm 2021-07-31T04:08:37.187137+0000 mgr.x (mgr.24542) 6062 : cephadm [ERR] 2021-07-31T04:08:38.342 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:08:38.342 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:08:38.342 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: yield 2021-07-31T04:08:38.342 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:08:38.342 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:08:38.343 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:08:38.343 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:08:38.343 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:08:38.343 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:08:38.344 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:08:38.344 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:08:38.344 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:08:38.344 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: task.result() 2021-07-31T04:08:38.345 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:08:38.345 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: cephadm 2021- 2021-07-31T04:08:38.345 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: 07-31T04:08:37.188241+0000 mgr.x (mgr.24542) 6063 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:08:38.345 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:08:38.345 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:08:38.346 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: yield 2021-07-31T04:08:38.346 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:08:38.346 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:08:38.346 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:08:38.347 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:08:38.347 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:08:38.347 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:08:38.347 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:08:38.347 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:08:38.348 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:08:38.348 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: task.result() 2021-07-31T04:08:38.348 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:08:38.348 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: 2021-07-31T04:08:38.348 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T04:08:38.349 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: 2021-07-31T04:08:38.349 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:08:38.349 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:08:38.349 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: return f(*arg) 2021-07-31T04:08:38.349 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:08:38.350 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:08:38.350 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:08:38.350 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:08:38.350 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:08:38.350 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:08:38.351 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:08:38.351 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T04:08:38.351 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:08:38.351 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: raise self._exception 2021-07-31T04:08:38.352 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:08:38.352 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:08:38.352 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:08:38.352 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T04:08:38.352 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:08:38.352 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: raise 2021-07-31T04:08:38.353 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:08:38.353 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T04:08:38.353 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:08:38.353 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:08:38.354 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:08:38.354 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:08:38.354 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:08:38.354 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:08:38.355 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: task.result() 2021-07-31T04:08:38.355 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:08:38.355 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: cephadm 2021-07-31T04:08:37.188241+0000 mgr.x (mgr.24542) 6063 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:08:38.355 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:08:38.356 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:08:38.356 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: yield 2021-07-31T04:08:38.356 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:08:38.356 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:08:38.356 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:08:38.357 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:08:38.357 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:08:38.357 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:08:38.357 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:08:38.357 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:08:38.358 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:08:38.358 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: task.result() 2021-07-31T04:08:38.358 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:08:38.358 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: 2021-07-31T04:08:38.359 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T04:08:38.359 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: 2021-07-31T04:08:38.359 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:08:38.359 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:08:38.359 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: return f(*arg) 2021-07-31T04:08:38.360 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:08:38.360 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:08:38.360 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:08:38.360 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:08:38.360 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:08:38.361 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:08:38.361 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:08:38.361 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T04:08:38.361 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:08:38.361 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: raise self._exception 2021-07-31T04:08:38.362 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:08:38.362 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:08:38.362 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:08:38.362 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T04:08:38.363 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:08:38.363 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: raise 2021-07-31T04:08:38.363 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:08:38.363 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T04:08:38.363 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:08:38.364 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:08:38.364 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:37 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:08:39.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:38 smithi154 conmon[34553]: cluster 2021-07-31T04:08:37.818330+0000 mgr.x (mgr.24542) 6064 : cluster [DBG] pgmap v6250: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:39.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:38 smithi045 conmon[39718]: cluster 2021-07-31T04:08:37 2021-07-31T04:08:39.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:38 smithi045 conmon[39718]: .818330+0000 mgr.x (mgr.24542) 6064 : cluster [DBG] pgmap v6250: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:39.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:38 smithi045 conmon[33655]: cluster 2021-07-31T04:08:37.818330+0000 mgr.x (mgr.24542) 6064 : 2021-07-31T04:08:39.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:38 smithi045 conmon[33655]: cluster [DBG] pgmap v6250: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:41.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:40 smithi154 conmon[34553]: cluster 2021-07-31T04:08:39.818960+0000 mgr.x (mgr.24542) 6065 : cluster [DBG] pgmap v6251: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:41.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:40 smithi045 conmon[39718]: cluster 2021-07-31T04:08:39 2021-07-31T04:08:41.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:40 smithi045 conmon[39718]: .818960+0000 mgr.x (mgr.24542) 6065 : cluster [DBG] pgmap v6251: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:41.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:40 smithi045 conmon[33655]: cluster 2021-07-31T04:08:39.818960+0000 mgr.x (mgr.24542) 2021-07-31T04:08:41.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:40 smithi045 conmon[33655]: 6065 : cluster [DBG] pgmap v6251: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:41.853 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T04:08:42.130 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T04:08:42.131 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 2h ago 2h - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T04:08:42.131 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 2h ago 2h 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T04:08:42.131 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 2h ago 2h 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T04:08:42.131 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 2h ago 2h - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T04:08:42.131 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T04:08:42.132 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 2h ago 2h 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T04:08:42.132 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T04:08:42.132 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 2h ago 2h - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T04:08:42.132 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 2h ago 2h 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T04:08:42.132 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T04:08:42.133 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T04:08:42.133 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T04:08:42.133 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T04:08:42.133 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 2h ago 2h 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T04:08:42.133 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 2h ago 2h 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T04:08:42.134 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 2h ago 2h 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T04:08:42.134 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 2h ago 2h 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T04:08:42.134 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 2h ago 2h 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T04:08:42.464 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T04:08:42.465 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T04:08:42.465 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T04:08:42.465 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:08:42.465 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T04:08:42.466 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T04:08:42.466 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:08:42.466 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:08:42.466 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T04:08:42.467 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T04:08:42.467 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:08:42.467 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T04:08:42.467 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T04:08:42.467 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T04:08:42.468 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:08:42.468 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T04:08:42.468 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T04:08:43.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:42 smithi154 conmon[34553]: cluster 2021-07-31T04:08:41.819339+0000 mgr.x (mgr.24542) 6066 : cluster [DBG] pgmap v6252: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:43.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:42 smithi154 conmon[34553]: audit 2021-07-31T04:08:41.842002+0000 mgr.x (mgr.24542) 6067 : audit [DBG] from='client.20982 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:08:43.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:42 smithi154 conmon[34553]: audit 2021-07-31T04:08:42.123257+0000 mgr.x (mgr.24542) 6068 : audit 2021-07-31T04:08:43.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:42 smithi154 conmon[34553]: [DBG] from='client.20988 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:08:43.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:42 smithi154 conmon[34553]: audit 2021- 2021-07-31T04:08:43.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:42 smithi154 conmon[34553]: 07-31T04:08:42.465782+0000 mon.a (mon.0) 2021-07-31T04:08:43.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:42 smithi154 conmon[34553]: 3429 : audit [DBG] from='client.? 172.21.15.45:0/327629995' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:08:43.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:42 smithi045 conmon[39718]: cluster 2021-07-31T04:08:41.819339+0000 mgr.x ( 2021-07-31T04:08:43.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:42 smithi045 conmon[39718]: mgr.24542) 6066 : cluster [DBG] pgmap v6252: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:43.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:42 smithi045 conmon[39718]: audit 2021- 2021-07-31T04:08:43.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:42 smithi045 conmon[39718]: 07-31T04:08:41.842002+0000 mgr.x (mgr.24542) 6067 : audit [DBG] from='client.20982 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:08:43.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:42 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T04:08:43.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:42 smithi045 conmon[39718]: 04:08:42.123257+0000 mgr.x (mgr.24542) 6068 : audit [DBG] 2021-07-31T04:08:43.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:42 smithi045 conmon[39718]: from='client.20988 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:08:43.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:42 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T04:08:43.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:42 smithi045 conmon[39718]: 31T04:08:42.465782+0000 mon.a (mon. 2021-07-31T04:08:43.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:42 smithi045 conmon[39718]: 0) 3429 : audit [DBG] from='client.? 172.21.15.45:0/327629995' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:08:43.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:42 smithi045 conmon[33655]: cluster 2021-07-31T04:08:41.819339+0000 mgr.x (mgr.24542) 6066 : cluster [DBG] pgmap v6252: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:43.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:42 smithi045 conmon[33655]: audit 2021-07-31T04:08:41.842002+0000 mgr.x (mgr.24542) 6067 : audit [DBG] from='client.20982 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:08:43.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:42 smithi045 conmon[33655]: audit 2021-07-31T04:08:42.123257+0000 mgr.x (mgr.24542) 6068 : audit [DBG] from='client.20988 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:08:43.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:42 smithi045 conmon[33655]: audit 2021-07-31T04:08:42.465782+0000 mon.a (mon.0) 3429 : audit [DBG] from='client.? 172.21.15.45:0/327629995' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:08:45.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:44 smithi154 conmon[34553]: cluster 2021-07-31T04:08:43.819853+0000 mgr.x (mgr.24542) 6069 : cluster [DBG] pgmap v6253: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:45.274 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:44 smithi045 conmon[39718]: cluster 2021-07-31T04:08:43.819853+0000 mgr.x (mgr. 2021-07-31T04:08:45.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:44 smithi045 conmon[39718]: 24542) 6069 : cluster [DBG] pgmap v6253: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:45.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:44 smithi045 conmon[33655]: cluster 2021-07-31T04:08:43.819853+0000 mgr.x (mgr.24542) 6069 : cluster [DBG] pgmap v6253: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:47.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:46 smithi154 conmon[34553]: cluster 2021-07-31T04:08:45.820458+0000 mgr.x (mgr.24542) 6070 : cluster [DBG] pgmap v6254: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:47.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:46 smithi045 conmon[33655]: cluster 2021-07-31T04:08:45.820458+0000 mgr.x ( 2021-07-31T04:08:47.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:46 smithi045 conmon[33655]: mgr.24542) 6070 : cluster [DBG] pgmap v6254: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:47.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:46 smithi045 conmon[39718]: cluster 2021-07-31T04:08:45 2021-07-31T04:08:47.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:46 smithi045 conmon[39718]: .820458+0000 mgr.x (mgr.24542) 6070 : cluster [DBG] pgmap v6254: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:49.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:48 smithi154 conmon[34553]: cluster 2021-07-31T04:08:47.820855+0000 mgr.x (mgr.24542) 6071 : cluster [DBG] pgmap v6255: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:49.274 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:48 smithi045 conmon[39718]: cluster 2021-07-31T04:08:47.820855+0000 mgr.x (mgr.24542) 6071 : cluster [DBG] pgmap v6255: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:49.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:48 smithi045 conmon[33655]: cluster 2021-07-31T04:08:47.820855+0000 mgr.x (mgr.24542) 6071 : cluster 2021-07-31T04:08:49.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:48 smithi045 conmon[33655]: [DBG] pgmap v6255: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:51.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:50 smithi154 conmon[34553]: cluster 2021-07-31T04:08:49.821440+0000 mgr.x (mgr.24542) 6072 : cluster [DBG] 2021-07-31T04:08:51.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:50 smithi154 conmon[34553]: pgmap v6256: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:51.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:50 smithi045 conmon[39718]: cluster 2021-07-31T04:08:49.821440+0000 2021-07-31T04:08:51.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:50 smithi045 conmon[39718]: mgr.x (mgr.24542) 6072 : cluster [DBG] pgmap v6256: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:51.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:50 smithi045 conmon[33655]: cluster 2021-07-31T04:08:49.821440+0000 mgr.x (mgr.24542) 6072 : cluster [DBG] pgmap v6256: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:53.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:52 smithi154 conmon[34553]: cluster 2021-07-31T04:08:51.821995+0000 mgr.x (mgr.24542) 6073 : cluster [DBG] 2021-07-31T04:08:53.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:52 smithi154 conmon[34553]: pgmap v6257: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:53.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:52 smithi045 conmon[33655]: cluster 2021-07-31T04:08:51.821995+0000 mgr.x (mgr.24542) 6073 2021-07-31T04:08:53.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:52 smithi045 conmon[33655]: : cluster [DBG] pgmap v6257: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:53.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:52 smithi045 conmon[39718]: cluster 2021-07-31T04:08:51.821995+0000 mgr.x (mgr.24542) 6073 : cluster 2021-07-31T04:08:53.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:52 smithi045 conmon[39718]: [DBG] pgmap v6257: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:55.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:54 smithi154 conmon[34553]: cluster 2021-07-31T04:08:53.822367+0000 mgr.x (mgr.24542) 6074 : cluster 2021-07-31T04:08:55.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:54 smithi154 conmon[34553]: [DBG] pgmap v6258: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:55.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:54 smithi045 conmon[33655]: cluster 2021-07-31T04:08:53.822367+0000 mgr.x (mgr.24542) 6074 : 2021-07-31T04:08:55.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:54 smithi045 conmon[33655]: cluster [DBG] pgmap v6258: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:55.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:54 smithi045 conmon[39718]: cluster 2021-07-31T04:08:53. 2021-07-31T04:08:55.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:54 smithi045 conmon[39718]: 822367+0000 mgr.x (mgr.24542) 6074 : cluster [DBG] pgmap v6258: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:57.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:56 smithi154 conmon[34553]: cluster 2021-07-31T04:08:55.822860+0000 mgr.x (mgr.24542) 6075 : cluster [DBG] pgmap v6259: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:57.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:56 smithi045 conmon[39718]: cluster 2021-07-31T04:08:55.822860+0000 mgr.x (mgr. 2021-07-31T04:08:57.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:56 smithi045 conmon[39718]: 24542) 6075 : cluster [DBG] pgmap v6259: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:57.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:56 smithi045 conmon[33655]: cluster 2021-07-31T04:08:55.822860+0000 mgr.x (mgr.24542) 6075 : cluster 2021-07-31T04:08:57.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:56 smithi045 conmon[33655]: [DBG] pgmap v6259: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:59.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:08:59 smithi154 conmon[34553]: cluster 2021-07-31T04:08:57.823311+0000 mgr.x (mgr.24542) 6076 : cluster [DBG] pgmap v6260: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:59.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:59 smithi045 conmon[39718]: cluster 2021-07-31T04:08:57.823311+0000 mgr.x 2021-07-31T04:08:59.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:08:59 smithi045 conmon[39718]: (mgr.24542) 6076 : cluster [DBG] pgmap v6260: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:08:59.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:59 smithi045 conmon[33655]: cluster 2021-07-31T04:08:59.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:08:59 smithi045 conmon[33655]: 2021-07-31T04:08:57.823311+0000 mgr.x (mgr.24542) 6076 : cluster [DBG] pgmap v6260: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:01.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:01 smithi045 conmon[39718]: cluster 2021-07-31T04:08:59.823779+0000 mgr.x 2021-07-31T04:09:01.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:01 smithi045 conmon[39718]: (mgr.24542) 6077 : cluster [DBG] pgmap v6261: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:01.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:01 smithi045 conmon[33655]: cluster 2021-07-31T04:08:59.823779+0000 mgr.x (mgr.24542) 6077 : cluster 2021-07-31T04:09:01.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:01 smithi045 conmon[33655]: [DBG] pgmap v6261: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:01.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:01 smithi154 conmon[34553]: cluster 2021-07-31T04:08:59.823779+0000 mgr.x (mgr.24542) 6077 : cluster [DBG] pgmap v6261: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:03.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:03 smithi045 conmon[33655]: cluster 2021-07-31T04:09:01.824182+0000 mgr.x (mgr.24542) 6078 : cluster [DBG] pgmap v6262: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:03.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:03 smithi045 conmon[39718]: cluster 2021-07-31T04:09:01.824182+0000 mgr.x (mgr 2021-07-31T04:09:03.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:03 smithi045 conmon[39718]: .24542) 6078 : cluster [DBG] pgmap v6262: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:03.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:03 smithi154 conmon[34553]: cluster 2021-07-31T04:09:01.824182+0000 mgr.x (mgr.24542) 6078 : cluster [DBG] pgmap v6262: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:05.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:05 smithi045 conmon[33655]: cluster 2021-07-31T04:09:03.824686+0000 mgr.x (mgr.24542) 2021-07-31T04:09:05.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:05 smithi045 conmon[33655]: 6079 : cluster [DBG] pgmap v6263: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:05.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:05 smithi045 conmon[39718]: cluster 2021-07-31T04:09:03.824686+0000 mgr.x (mgr.24542) 6079 : cluster [DBG] pgmap v6263: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:05.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:05 smithi154 conmon[34553]: cluster 2021-07-31T04:09:03.824686+0000 mgr.x (mgr.24542) 6079 : cluster [DBG] pgmap v6263: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:05.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:05 smithi154 conmon[34553]: 2021-07-31T04:09:07.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:07 smithi045 conmon[39718]: cluster 2021-07-31T04:09:05.825298+0000 mgr.x (mgr. 2021-07-31T04:09:07.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:07 smithi045 conmon[39718]: 24542) 6080 : cluster [DBG] pgmap v6264: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:07.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:07 smithi045 conmon[33655]: cluster 2021-07-31T04:09:05.825298+0000 mgr.x (mgr.24542) 6080 : cluster [DBG] pgmap v6264: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:07.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:07 smithi154 conmon[34553]: cluster 2021-07-31T04:09: 2021-07-31T04:09:07.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:07 smithi154 conmon[34553]: 05.825298+0000 mgr.x (mgr.24542) 6080 : cluster [DBG] pgmap v6264: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:09.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:09 smithi045 conmon[33655]: cluster 2021-07-31T04:09:07.825878+0000 mgr.x (mgr.24542) 6081 2021-07-31T04:09:09.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:09 smithi045 conmon[33655]: : cluster [DBG] pgmap v6265: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:09.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:09 smithi045 conmon[39718]: cluster 2021-07-31T04:09:07. 2021-07-31T04:09:09.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:09 smithi045 conmon[39718]: 825878+0000 mgr.x (mgr.24542) 6081 : cluster [DBG] pgmap v6265: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:09.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:09 smithi154 conmon[34553]: cluster 2021-07-31T04:09:07.825878+0000 mgr.x (mgr.24542) 6081 : cluster 2021-07-31T04:09:09.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:09 smithi154 conmon[34553]: [DBG] pgmap v6265: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:11.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:11 smithi045 conmon[33655]: cluster 2021-07-31T04:09:09.826373+0000 mgr.x (mgr.24542) 6082 : cluster 2021-07-31T04:09:11.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:11 smithi045 conmon[33655]: [DBG] pgmap v6266: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:11.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:11 smithi045 conmon[39718]: cluster 2021-07-31T04:09 2021-07-31T04:09:11.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:11 smithi045 conmon[39718]: :09.826373+0000 mgr.x (mgr.24542) 6082 : cluster [DBG] pgmap v6266: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:11.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:11 smithi154 conmon[34553]: cluster 2021-07-31T04:09:09.826373+0000 mgr.x (mgr.24542) 6082 : cluster [DBG] pgmap v6266: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:12.777 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T04:09:13.055 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T04:09:13.055 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 2h ago 2h - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T04:09:13.055 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 2h ago 2h 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T04:09:13.056 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 2h ago 2h 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T04:09:13.056 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 2h ago 2h - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T04:09:13.056 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T04:09:13.056 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 2h ago 2h 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T04:09:13.057 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T04:09:13.057 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 2h ago 2h - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T04:09:13.057 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 2h ago 2h 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T04:09:13.057 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T04:09:13.057 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T04:09:13.058 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T04:09:13.058 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T04:09:13.058 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 2h ago 2h 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T04:09:13.058 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 2h ago 2h 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T04:09:13.058 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 2h ago 2h 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T04:09:13.059 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 2h ago 2h 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T04:09:13.059 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 2h ago 2h 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T04:09:13.274 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:13 smithi045 conmon[33655]: cluster 2021-07-31T04:09:11.826890+0000 2021-07-31T04:09:13.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:13 smithi045 conmon[33655]: mgr.x (mgr.24542) 6083 : cluster [DBG] pgmap v6267: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:13.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:13 smithi045 conmon[39718]: cluster 2021-07-31T04:09:11.826890+0000 mgr.x (mgr.24542) 6083 : 2021-07-31T04:09:13.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:13 smithi045 conmon[39718]: cluster [DBG] pgmap v6267: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:13.389 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T04:09:13.390 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T04:09:13.390 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T04:09:13.390 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:09:13.391 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T04:09:13.391 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T04:09:13.391 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:09:13.391 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:09:13.391 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T04:09:13.392 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T04:09:13.392 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:09:13.392 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T04:09:13.392 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T04:09:13.392 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T04:09:13.393 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:09:13.393 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T04:09:13.393 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T04:09:13.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:13 smithi154 conmon[34553]: cluster 2021-07-31T04:09:11.826890+0000 mgr.x (mgr.24542) 6083 : cluster [DBG] pgmap v6267: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:14.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:14 smithi154 conmon[34553]: audit 2021-07-31T04:09:12.765914+0000 mgr.x (mgr.24542) 6084 : audit [DBG] from='client.30937 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:09:14.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:14 smithi154 conmon[34553]: audit 2021-07-31T04:09:13.048069+0000 mgr.x (mgr.24542) 6085 : audit [DBG] from='client.21006 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:09:14.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:14 smithi154 conmon[34553]: audit 2021-07-31T04:09:13.390890+0000 mon.c (mon.1) 252 : audit [DBG] from='client.? 172.21.15.45:0/2647346210' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:09:14.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:14 smithi045 conmon[33655]: audit 2021-07-31T04:09:12.765914+0000 mgr.x (mgr. 2021-07-31T04:09:14.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:14 smithi045 conmon[33655]: 24542) 6084 : audit [DBG] from='client.30937 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:09:14.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:14 smithi045 conmon[33655]: audit 2021-07-31T04:09:13.048069+0000 mgr.x (mgr.24542) 6085 : audit [DBG] from='client.21006 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:09:14.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:14 smithi045 conmon[33655]: audit 2021-07-31T04:09:14.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:14 smithi045 conmon[33655]: 2021-07-31T04:09:13.390890+0000 mon.c (mon.1) 252 : audit [DBG] from='client.? 172.21.15.45:0/2647346210' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:09:14.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:14 smithi045 conmon[39718]: audit 2021-07-31T04:09:12.765914+0000 mgr.x (mgr. 2021-07-31T04:09:14.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:14 smithi045 conmon[39718]: 24542) 6084 : audit [DBG] from='client.30937 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:09:14.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:14 smithi045 conmon[39718]: audit 2021-07-31T04: 2021-07-31T04:09:14.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:14 smithi045 conmon[39718]: 09:13.048069+0000 mgr.x (mgr.24542) 6085 : audit [DBG] from='client.21006 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:09:14.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:14 smithi045 conmon[39718]: audit 2021 2021-07-31T04:09:14.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:14 smithi045 conmon[39718]: -07-31T04:09:13.390890+0000 mon.c (mon. 2021-07-31T04:09:14.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:14 smithi045 conmon[39718]: 1) 252 : audit [DBG] from='client.? 172.21.15.45:0/2647346210' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:09:15.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:15 smithi154 conmon[34553]: cluster 2021-07-31T04:09:13.827285+0000 mgr.x (mgr.24542) 6086 : 2021-07-31T04:09:15.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:15 smithi154 conmon[34553]: cluster [DBG] pgmap v6268: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:15.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:15 smithi045 conmon[33655]: cluster 2021-07-31T04:09:13.827285+0000 mgr.x (mgr 2021-07-31T04:09:15.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:15 smithi045 conmon[33655]: .24542) 6086 : cluster [DBG] pgmap v6268: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:15.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:15 smithi045 conmon[39718]: cluster 2021-07-31T04:09:13.827285+0000 2021-07-31T04:09:15.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:15 smithi045 conmon[39718]: mgr.x (mgr.24542) 6086 : cluster [DBG] pgmap v6268: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:17.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:17 smithi154 conmon[34553]: cluster 2021-07-31T04:09:15.827899+0000 mgr.x (mgr.24542) 6087 : cluster [DBG] pgmap v6269: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:17.524 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:17 smithi045 conmon[33655]: cluster 2021-07-31T04:09:15.827899+0000 mgr.x (mgr.24542) 6087 : cluster 2021-07-31T04:09:17.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:17 smithi045 conmon[33655]: [DBG] pgmap v6269: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:17.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:17 smithi045 conmon[39718]: cluster 2021-07-31T04:09:15.827899+0000 2021-07-31T04:09:17.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:17 smithi045 conmon[39718]: mgr.x (mgr.24542) 6087 : cluster [DBG] pgmap v6269: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:19.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:19 smithi154 conmon[34553]: cluster 2021-07-31T04:09:17.828318+0000 mgr.x (mgr.24542) 6088 : cluster [DBG] 2021-07-31T04:09:19.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:19 smithi154 conmon[34553]: pgmap v6270: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:19.524 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:19 smithi045 conmon[33655]: cluster 2021-07-31T04:09:17.828318+0000 mgr.x (mgr.24542) 6088 : cluster [DBG] pgmap v6270: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:19.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:19 smithi045 conmon[39718]: cluster 2021-07-31T04:09:17.828318+0000 2021-07-31T04:09:19.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:19 smithi045 conmon[39718]: mgr.x (mgr.24542) 6088 : cluster [DBG] pgmap v6270: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:21.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:21 smithi154 conmon[34553]: cluster 2021-07-31T04:09:19.828749+0000 mgr.x (mgr.24542) 6089 : cluster [DBG] pgmap v6271: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:21.524 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:21 smithi045 conmon[39718]: cluster 2021-07-31T04:09:19.828749+0000 mgr.x ( 2021-07-31T04:09:21.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:21 smithi045 conmon[39718]: mgr.24542) 6089 : cluster [DBG] pgmap v6271: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:21.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:21 smithi045 conmon[33655]: cluster 2021-07-31T04:09:19.828749+0000 mgr.x (mgr.24542) 6089 : cluster 2021-07-31T04:09:21.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:21 smithi045 conmon[33655]: [DBG] pgmap v6271: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:23.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:23 smithi154 conmon[34553]: cluster 2021-07-31T04:09:21.829177+0000 mgr.x (mgr.24542) 6090 : cluster [DBG] pgmap v6272: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:23.524 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:23 smithi045 conmon[33655]: cluster 2021-07-31T04:09:21.829177+0000 mgr.x (mgr.24542) 6090 : cluster [DBG] pgmap v6272: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:23.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:23 smithi045 conmon[39718]: cluster 2021-07-31T04:09:21.829177+0000 2021-07-31T04:09:23.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:23 smithi045 conmon[39718]: mgr.x (mgr.24542) 6090 : cluster [DBG] pgmap v6272: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:25.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:25 smithi154 conmon[34553]: cluster 2021-07-31T04:09:23.829688+0000 mgr.x (mgr.24542) 6091 : cluster [DBG] pgmap v6273: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:25.524 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:25 smithi045 conmon[33655]: cluster 2021-07-31T04: 2021-07-31T04:09:25.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:25 smithi045 conmon[33655]: 09:23.829688+0000 mgr.x (mgr.24542) 6091 : cluster [DBG] pgmap v6273: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:25.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:25 smithi045 conmon[39718]: cluster 2021-07-31T04:09:23.829688+0000 mgr.x (mgr. 2021-07-31T04:09:25.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:25 smithi045 conmon[39718]: 24542) 6091 : cluster [DBG] pgmap v6273: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:27.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:27 smithi154 conmon[34553]: cluster 2021-07-31T04:09:25.830123+0000 mgr.x (mgr.24542) 6092 : cluster [DBG] pgmap v6274: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:27.524 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:27 smithi045 conmon[33655]: cluster 2021-07-31T04:09:25.830123+0000 mgr.x (mgr.24542) 6092 : cluster [DBG] pgmap v6274: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:27.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:27 smithi045 conmon[39718]: cluster 2021-07-31T04:09:25.830123+0000 mgr.x (mgr. 2021-07-31T04:09:27.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:27 smithi045 conmon[39718]: 24542) 6092 : cluster [DBG] pgmap v6274: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:29.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:29 smithi154 conmon[34553]: cluster 2021-07-31T04:09:27.830635+0000 mgr.x (mgr.24542) 6093 : cluster [DBG] pgmap v6275: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:29.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:29 smithi154 conmon[34553]: audit 2021-07-31T04:09:28.495950+0000 mon.b (mon.2) 785 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:09:29.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:29 smithi154 conmon[34553]: audit 2021-07-31T04:09:28.496180+0000 mon.b (mon.2) 786 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:09:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:29 smithi154 conmon[34553]: audit 2021-07-31T04:09:28.496945+0000 mon.a (mon.0 2021-07-31T04:09:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:29 smithi154 conmon[34553]: ) 3430 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:09:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:29 smithi154 conmon[34553]: audit 2021-07-31T 2021-07-31T04:09:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:29 smithi154 conmon[34553]: 04:09:28.497360+0000 mon.a (mon.0 2021-07-31T04:09:29.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:29 smithi154 conmon[34553]: ) 3431 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:09:29.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:29 smithi045 conmon[39718]: cluster 2021-07-31T04:09:27.830635+0000 mgr.x (mgr 2021-07-31T04:09:29.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:29 smithi045 conmon[39718]: .24542) 6093 : cluster [DBG] pgmap v6275: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:29.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:29 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T04:09:29.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:29 smithi045 conmon[39718]: 04:09:28.495950+0000 mon.b (mon.2) 785 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:09:29.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:29 smithi045 conmon[39718]: audit 2021-07-31T04:09:28.496180+0000 mon.b (mon.2) 786 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:09:29.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:29 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T04:09:29.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:29 smithi045 conmon[39718]: 04:09:28.496945+0000 mon.a (mon.0) 3430 : 2021-07-31T04:09:29.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:29 smithi045 conmon[39718]: audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:09:29.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:29 smithi045 conmon[39718]: audit 2021-07-31T04:09:28.497360+0000 mon.a ( 2021-07-31T04:09:29.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:29 smithi045 conmon[39718]: mon.0) 3431 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:09:29.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:29 smithi045 conmon[33655]: cluster 2021-07-31T04:09:27.830635+0000 2021-07-31T04:09:29.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:29 smithi045 conmon[33655]: mgr.x (mgr.24542) 6093 : cluster [DBG] pgmap v6275: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:29.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:29 smithi045 conmon[33655]: audit 2021-07-31T04:09:28.495950+0000 mon.b (mon.2) 785 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:09:29.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:29 smithi045 conmon[33655]: audit 2021-07-31T04:09:28.496180+0000 mon.b (mon 2021-07-31T04:09:29.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:29 smithi045 conmon[33655]: .2) 786 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:09:29.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:29 smithi045 conmon[33655]: audit 2021-07 2021-07-31T04:09:29.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:29 smithi045 conmon[33655]: -31T04:09:28.496945+0000 mon.a (mon.0) 3430 : audit 2021-07-31T04:09:29.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:29 smithi045 conmon[33655]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:09:29.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:29 smithi045 conmon[33655]: audit 2021-07-31T04:09:28.497360+0000 2021-07-31T04:09:29.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:29 smithi045 conmon[33655]: mon.a (mon.0) 3431 : audit 2021-07-31T04:09:29.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:29 smithi045 conmon[33655]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:09:31.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:31 smithi154 conmon[34553]: cluster 2021-07-31T04:09:29.831127+0000 mgr.x (mgr.24542) 6094 : cluster [DBG] pgmap v6276: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:31.524 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:31 smithi045 conmon[39718]: cluster 2021-07-31T04:09:29.831127+0000 mgr.x (mgr.24542) 6094 : cluster [DBG] pgmap v6276: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:31.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:31 smithi045 conmon[33655]: cluster 2021-07-31T04:09:29.831127+0000 mgr.x (mgr.24542) 6094 : cluster [DBG] pgmap v6276: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:33.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:33 smithi154 conmon[34553]: cluster 2021-07-31T04:09:31.831654+0000 mgr.x (mgr.24542) 6095 : cluster [DBG] pgmap v6277: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:33.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:33 smithi045 conmon[33655]: cluster 2021-07-31T04:09:31.831654+0000 mgr.x (mgr.24542) 2021-07-31T04:09:33.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:33 smithi045 conmon[33655]: 6095 : cluster [DBG] pgmap v6277: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:33.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:33 smithi045 conmon[39718]: cluster 2021-07-31T04:09:31.831654+0000 2021-07-31T04:09:33.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:33 smithi045 conmon[39718]: mgr.x (mgr.24542) 6095 : cluster [DBG] pgmap v6277: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:35.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:35 smithi154 conmon[34553]: cluster 2021-07-31T04:09:33.832217+0000 mgr.x (mgr.24542) 2021-07-31T04:09:35.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:35 smithi154 conmon[34553]: 6096 : cluster [DBG] pgmap v6278: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:35.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:35 smithi045 conmon[33655]: cluster 2021-07-31T04:09:33.832217+0000 mgr.x (mgr.24542) 6096 : cluster [DBG] 2021-07-31T04:09:35.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:35 smithi045 conmon[33655]: pgmap v6278: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:35.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:35 smithi045 conmon[39718]: cluster 2021-07-31T04:09:33.832217+0000 mgr.x 2021-07-31T04:09:35.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:35 smithi045 conmon[39718]: (mgr.24542) 6096 : cluster [DBG] pgmap v6278: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:37.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[34553]: cluster 2021-07-31T04:09:35.832843+0000 mgr.x (mgr.24542) 6097 : cluster [DBG] pgmap v6279: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:37.503 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: debug 2021-07-31T04:09:37.234+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:09:37.503 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:09:37.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:09:37.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: yield 2021-07-31T04:09:37.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:09:37.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:09:37.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:09:37.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:09:37.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:09:37.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:09:37.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:09:37.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:09:37.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:09:37.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: task.result() 2021-07-31T04:09:37.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:09:37.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: debug 2021-07-31T04:09:37.246+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:09:37.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:09:37.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:09:37.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: yield 2021-07-31T04:09:37.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:09:37.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:09:37.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:09:37.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:09:37.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:09:37.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:09:37.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:09:37.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:09:37.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:09:37.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: task.result() 2021-07-31T04:09:37.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:09:37.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: debug 2021-07-31T04:09:37.272+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:09:37.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:09:37.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:09:37.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: yield 2021-07-31T04:09:37.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:09:37.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:09:37.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:09:37.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:09:37.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:09:37.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:09:37.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:09:37.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:09:37.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:09:37.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: task.result() 2021-07-31T04:09:37.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:09:37.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: debug 2021-07-31T04:09:37.272+0000 7fcb9215e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:09:37.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:09:37.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:09:37.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: yield 2021-07-31T04:09:37.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:09:37.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:09:37.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:09:37.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:09:37.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:09:37.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:09:37.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:09:37.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:09:37.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:09:37.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: task.result() 2021-07-31T04:09:37.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:09:37.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: 2021-07-31T04:09:37.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T04:09:37.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: 2021-07-31T04:09:37.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:09:37.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:09:37.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: return f(*arg) 2021-07-31T04:09:37.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:09:37.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:09:37.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:09:37.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:09:37.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:09:37.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:09:37.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:09:37.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T04:09:37.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:09:37.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: raise self._exception 2021-07-31T04:09:37.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:09:37.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:09:37.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:09:37.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T04:09:37.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:09:37.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: raise 2021-07-31T04:09:37.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:09:37.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T04:09:37.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:09:37.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:09:37.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:09:37.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: debug 2021-07-31T04:09:37.281+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:09:37.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:09:37.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:09:37.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: yield 2021-07-31T04:09:37.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:09:37.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:09:37.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:09:37.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:09:37.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:09:37.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:09:37.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:09:37.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:09:37.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:09:37.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: task.result() 2021-07-31T04:09:37.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:09:37.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: debug 2021-07-31T04:09:37.281+0000 7fcb9195d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:09:37.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:09:37.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:09:37.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: yield 2021-07-31T04:09:37.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:09:37.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:09:37.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:09:37.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:09:37.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:09:37.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:09:37.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:09:37.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:09:37.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:09:37.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: task.result() 2021-07-31T04:09:37.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:09:37.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: 2021-07-31T04:09:37.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T04:09:37.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: 2021-07-31T04:09:37.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:09:37.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:09:37.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: return f(*arg) 2021-07-31T04:09:37.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:09:37.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:09:37.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:09:37.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:09:37.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:09:37.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:09:37.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:09:37.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T04:09:37.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:09:37.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: raise self._exception 2021-07-31T04:09:37.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:09:37.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:09:37.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:09:37.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T04:09:37.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:09:37.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: raise 2021-07-31T04:09:37.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:09:37.535 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T04:09:37.535 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:09:37.535 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:09:37.535 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:09:37 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:09:37.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:37 smithi045 conmon[33655]: cluster 2021-07-31T04:09:35.832843+0000 mgr.x (mgr.24542) 6097 : cluster [DBG] pgmap v6279: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:37.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:37 smithi045 conmon[39718]: cluster 2021-07-31T04:09:35.832843+0000 mgr.x (mgr. 2021-07-31T04:09:37.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:37 smithi045 conmon[39718]: 24542) 6097 : cluster [DBG] pgmap v6279: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:38.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: audit 2021-07-31T04:09:37.189082+0000 mon.b (mon.2) 787 : audit 2021-07-31T04:09:38.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T04:09:38.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: audit 2021-07-31T04:09:37.190484+0000 mon.b (mon.2) 788 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T04:09:38.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: audit 2021-07-31T04:09:37.191457+0000 mon.b (mon.2) 789 : audit 2021-07-31T04:09:38.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T04:09:38.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: cephadm 2021-07-31T04 2021-07-31T04:09:38.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: :09:37.192206+0000 mgr.x (mgr.24542) 6098 2021-07-31T04:09:38.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T04:09:38.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: cephadm 2021- 2021-07-31T04:09:38.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: 07-31T04:09:37.235690+0000 mgr.x (mgr 2021-07-31T04:09:38.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: .24542) 6099 : cephadm [ERR] 2021-07-31T04:09:38.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:09:38.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:09:38.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: yield 2021-07-31T04:09:38.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:09:38.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:09:38.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:09:38.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:09:38.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:09:38.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:09:38.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:09:38.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:09:38.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:09:38.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: task.result() 2021-07-31T04:09:38.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:09:38.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: cephadm 2021-07-31T04:09:37.236150+0000 mgr.x (mgr.24542) 6100 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T04:09:38.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: cephadm 2021-07-31T04:09:37.247547+0000 2021-07-31T04:09:38.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: mgr.x (mgr.24542) 6101 : cephadm [ERR] 2021-07-31T04:09:38.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:09:38.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:09:38.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: yield 2021-07-31T04:09:38.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:09:38.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:09:38.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:09:38.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:09:38.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:09:38.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:09:38.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:09:38.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:09:38.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:09:38.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: task.result() 2021-07-31T04:09:38.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:09:38.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: cephadm 2021-07-31T04:09:37.248591+0000 mgr.x (mgr.24542) 6102 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T04:09:38.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: cephadm 2021-07-31T04:09:37.273360+0000 mgr.x 2021-07-31T04:09:38.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: (mgr.24542) 6103 : cephadm [ERR] 2021-07-31T04:09:38.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:09:38.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:09:38.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: yield 2021-07-31T04:09:38.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:09:38.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:09:38.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:09:38.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:09:38.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:09:38.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:09:38.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:09:38.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:09:38.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:09:38.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: task.result() 2021-07-31T04:09:38.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:09:38.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: cephadm 2021-07-31T04:09:37.273796+0000 mgr.x (mgr.24542) 6104 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:09:38.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:09:38.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:09:38.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: yield 2021-07-31T04:09:38.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:09:38.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:09:38.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:09:38.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:09:38.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:09:38.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:09:38.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:09:38.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:09:38.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:09:38.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: task.result() 2021-07-31T04:09:38.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:09:38.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: 2021-07-31T04:09:38.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T04:09:38.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: 2021-07-31T04:09:38.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:09:38.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:09:38.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: return f(*arg) 2021-07-31T04:09:38.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:09:38.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:09:38.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:09:38.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:09:38.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:09:38.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:09:38.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:09:38.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T04:09:38.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:09:38.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: raise self._exception 2021-07-31T04:09:38.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:09:38.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:09:38.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:09:38.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T04:09:38.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:09:38.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: raise 2021-07-31T04:09:38.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:09:38.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T04:09:38.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:09:38.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:09:38.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:09:38.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: cephadm 2021-07-31T04:09:37.282003+0000 2021-07-31T04:09:38.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: mgr.x (mgr.24542) 6105 : cephadm [ERR] 2021-07-31T04:09:38.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:09:38.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:09:38.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: yield 2021-07-31T04:09:38.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:09:38.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:09:38.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:09:38.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:09:38.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:09:38.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:09:38.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:09:38.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:09:38.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:09:38.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: task.result() 2021-07-31T04:09:38.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:09:38.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: cephadm 2021-07-31T04:09:37.282413+0000 mgr.x (mgr.24542) 6106 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:09:38.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: audit 2021-07-31T04:09:37.189082+0000 mon.b (mon.2) 787 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T04:09:38.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: audit 2021-07-31T04:09:37.190484+0000 mon.b (mon.2) 788 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T04:09:38.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: audit 2021-07-31T04:09:37.191457+0000 mon.b (mon.2) 789 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T04:09:38.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: cephadm 2021-07-31T04:09:37.192206+0000 2021-07-31T04:09:38.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: mgr.x (mgr.24542) 6098 : 2021-07-31T04:09:38.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T04:09:38.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: cephadm 2021-07-31T04:09:37. 2021-07-31T04:09:38.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: 235690+0000 mgr.x (mgr.24542) 6099 : cephadm [ERR] 2021-07-31T04:09:38.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:09:38.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:09:38.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: yield 2021-07-31T04:09:38.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:09:38.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:09:38.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:09:38.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:09:38.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:09:38.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:09:38.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:09:38.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:09:38.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:09:38.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: task.result() 2021-07-31T04:09:38.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:09:38.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: cephadm 2021-07-31T04:09:37.236150+0000 mgr.x (mgr.24542) 6100 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T04:09:38.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: cephadm 2021-07-31T04:09:38.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: 2021-07-31T04 2021-07-31T04:09:38.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: :09:37.247547+0000 mgr.x 2021-07-31T04:09:38.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: (mgr.24542) 6101 : cephadm 2021-07-31T04:09:38.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: [ERR] 2021-07-31T04:09:38.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:09:38.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:09:38.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: yield 2021-07-31T04:09:38.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:09:38.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:09:38.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:09:38.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:09:38.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:09:38.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:09:38.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:09:38.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:09:38.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: yield 2021-07-31T04:09:38.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:09:38.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:09:38.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:09:38.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:09:38.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:09:38.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:09:38.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:09:38.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:09:38.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:09:38.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: task.result() 2021-07-31T04:09:38.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:09:38.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: 2021-07-31T04:09:38.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T04:09:38.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: 2021-07-31T04:09:38.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:09:38.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:09:38.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: return f(*arg) 2021-07-31T04:09:38.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:09:38.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:09:38.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:09:38.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:09:38.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:09:38.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:09:38.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:09:38.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T04:09:38.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:09:38.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: raise self._exception 2021-07-31T04:09:38.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:09:38.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:09:38.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:09:38.541 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T04:09:38.541 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:09:38.541 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: raise 2021-07-31T04:09:38.541 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:09:38.541 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T04:09:38.541 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:09:38.542 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:09:38.542 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:38 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:09:38.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: audit 2021-07-31T04:09:37.189082+0000 mon.b (mon.2) 787 : audit 2021-07-31T04:09:38.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T04:09:38.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: audit 2021-07-31T04:09:37.190484+0000 mon.b (mon.2) 788 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T04:09:38.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: audit 2021-07-31T04:09:37.191457+0000 mon.b (mon.2) 789 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T04:09:38.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: cephadm 2021-07-31T04:09:37.192206 2021-07-31T04:09:38.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 6098 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T04:09:38.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: cephadm 2021-07-31T04:09:37 2021-07-31T04:09:38.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: .235690+0000 mgr.x (mgr.24542) 6099 : cephadm [ERR] 2021-07-31T04:09:38.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:09:38.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:09:38.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: yield 2021-07-31T04:09:38.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:09:38.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:09:38.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:09:38.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:09:38.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:09:38.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:09:38.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:09:38.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:09:38.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:09:38.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: task.result() 2021-07-31T04:09:38.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:09:38.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: 2021-07-31T04:09:38.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: cephadm 2021-07-31T04:09:37.236150+0000 mgr.x (mgr.24542) 6100 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T04:09:38.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: cephadm 2021-07-31T04:09:37.247547+0000 mgr.x (mgr.24542) 6101 : cephadm [ERR] 2021-07-31T04:09:38.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:09:38.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:09:38.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: yield 2021-07-31T04:09:38.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:09:38.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:09:38.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:09:38.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:09:38.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:09:38.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:09:38.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:09:38.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:09:38.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:09:38.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: task.result() 2021-07-31T04:09:38.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:09:38.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: cephadm 2021-07-31T04:09:38.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: 2021-07-31T04:09:37.248591+0000 mgr.x (mgr.24542) 6102 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T04:09:38.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: cephadm 2021-07-31T04:09:37.273360+0000 mgr.x (mgr.24542) 6103 : cephadm [ERR] 2021-07-31T04:09:38.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:09:38.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:09:38.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: yield 2021-07-31T04:09:38.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:09:38.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:09:38.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:09:38.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:09:38.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:09:38.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:09:38.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:09:38.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:09:38.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:09:38.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: task.result() 2021-07-31T04:09:38.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:09:38.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: cephadm 2021-07-31T04:09:37.273796+0000 mgr.x (mgr.24542) 6104 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:09:38.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:09:38.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:09:38.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: yield 2021-07-31T04:09:38.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:09:38.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:09:38.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:09:38.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:09:38.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:09:38.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:09:38.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:09:38.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:09:38.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:09:38.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: task.result() 2021-07-31T04:09:38.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:09:38.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: 2021-07-31T04:09:38.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T04:09:38.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: 2021-07-31T04:09:38.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:09:38.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:09:38.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: return f(*arg) 2021-07-31T04:09:38.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:09:38.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:09:38.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:09:38.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: task.result() 2021-07-31T04:09:38.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:09:38.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: cephadm 2021-07-31T04:09:37.248591+0000 mgr.x (mgr.24542) 6102 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T04:09:38.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: cephadm 2021-07-31T04:09:38.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: 2021-07-31T04: 2021-07-31T04:09:38.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: 09:37.273360+0000 mgr.x (mgr 2021-07-31T04:09:38.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: .24542) 6103 : cephadm [ERR] 2021-07-31T04:09:38.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:09:38.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:09:38.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: yield 2021-07-31T04:09:38.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:09:38.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:09:38.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:09:38.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:09:38.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:09:38.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:09:38.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:09:38.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:09:38.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:09:38.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: task.result() 2021-07-31T04:09:38.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:09:38.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: cephadm 2021-07-31T04:09:37.273796+0000 mgr.x (mgr.24542) 6104 : cephadm 2021-07-31T04:09:38.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:09:38.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:09:38.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:09:38.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: yield 2021-07-31T04:09:38.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:09:38.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:09:38.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:09:38.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:09:38.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:09:38.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:09:38.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:09:38.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:09:38.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:09:38.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: task.result() 2021-07-31T04:09:38.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:09:38.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: 2021-07-31T04:09:38.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T04:09:38.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: 2021-07-31T04:09:38.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:09:38.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:09:38.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: return f(*arg) 2021-07-31T04:09:38.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:09:38.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:09:38.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:09:38.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:09:38.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:09:38.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:09:38.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:09:38.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T04:09:38.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:09:38.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: raise self._exception 2021-07-31T04:09:38.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:09:38.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:09:38.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:09:38.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T04:09:38.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:09:38.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: raise 2021-07-31T04:09:38.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:09:38.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T04:09:38.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:09:38.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:09:38.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:09:38.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: cephadm 2021-07-31T04:09:37.282003+0000 mgr.x (mgr.24542) 6105 : cephadm [ERR] 2021-07-31T04:09:38.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:09:38.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:09:38.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: yield 2021-07-31T04:09:38.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:09:38.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:09:38.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:09:38.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:09:38.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:09:38.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:09:38.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:09:38.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:09:38.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:09:38.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: task.result() 2021-07-31T04:09:38.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:09:38.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:09:38.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:09:38.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:09:38.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:09:38.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:09:38.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:09:38.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:09:38.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T04:09:38.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:09:38.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: raise self._exception 2021-07-31T04:09:38.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:09:38.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:09:38.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:09:38.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T04:09:38.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:09:38.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: raise 2021-07-31T04:09:38.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:09:38.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T04:09:38.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:09:38.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:09:38.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:09:38.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: cephadm 2021-07-31T04:09:37.282003+0000 mgr.x (mgr.24542) 6105 : cephadm [ERR] 2021-07-31T04:09:38.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:09:38.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:09:38.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: yield 2021-07-31T04:09:38.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:09:38.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:09:38.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:09:38.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:09:38.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:09:38.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:09:38.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:09:38.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:09:38.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:09:38.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: task.result() 2021-07-31T04:09:38.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:09:38.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: cephadm 2021-07-31T04:09:37.282413+0000 mgr.x (mgr.24542) 6106 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:09:38.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:09:38.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:09:38.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: cephadm 2021-07-31T04:09:37.282413+0000 mgr.x (mgr.24542) 6106 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:09:38.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:09:38.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:09:38.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: yield 2021-07-31T04:09:38.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:09:38.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:09:38.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:09:38.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:09:38.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:09:38.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:09:38.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:09:38.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:09:38.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:09:38.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: task.result() 2021-07-31T04:09:38.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:09:38.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: 2021-07-31T04:09:38.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T04:09:38.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: 2021-07-31T04:09:38.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:09:38.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:09:38.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: return f(*arg) 2021-07-31T04:09:38.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:09:38.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:09:38.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:09:38.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:09:38.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:09:38.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:09:38.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:09:38.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T04:09:38.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:09:38.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: raise self._exception 2021-07-31T04:09:38.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:09:38.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:09:38.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:09:38.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T04:09:38.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:09:38.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: raise 2021-07-31T04:09:38.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:09:38.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T04:09:38.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:09:38.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:09:38.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:09:38.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: yield 2021-07-31T04:09:38.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:09:38.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:09:38.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:09:38.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:09:38.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:09:38.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:09:38.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:09:38.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:09:38.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:09:38.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: task.result() 2021-07-31T04:09:38.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:09:38.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: 2021-07-31T04:09:38.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T04:09:38.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: 2021-07-31T04:09:38.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:09:38.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:09:38.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: return f(*arg) 2021-07-31T04:09:38.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:09:38.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:09:38.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:09:38.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:09:38.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:09:38.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:09:38.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:09:38.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T04:09:38.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:09:38.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: raise self._exception 2021-07-31T04:09:38.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:09:38.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:09:38.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:09:38.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T04:09:38.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:09:38.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: raise 2021-07-31T04:09:38.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:09:38.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T04:09:38.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:09:38.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:09:38.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:38 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:09:39.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:39 smithi154 conmon[34553]: cluster 2021-07-31T04:09:37.833269+0000 mgr.x (mgr.24542) 6107 : cluster [DBG] pgmap v6280: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:39.524 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:39 smithi045 conmon[33655]: cluster 2021-07-31 2021-07-31T04:09:39.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:39 smithi045 conmon[33655]: T04:09:37.833269+0000 mgr.x (mgr.24542) 6107 : cluster [DBG] pgmap v6280: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:39.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:39 smithi045 conmon[39718]: cluster 2021-07-31T04:09:37.833269+0000 mgr.x (mgr 2021-07-31T04:09:39.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:39 smithi045 conmon[39718]: .24542) 6107 : cluster [DBG] pgmap v6280: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:41.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:41 smithi154 conmon[34553]: cluster 2021-07-31T04:09:39.833792+0000 mgr.x (mgr.24542) 6108 : cluster [DBG] pgmap v6281: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:41.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:41 smithi045 conmon[39718]: cluster 2021-07-31T04:09:39.833792+0000 mgr.x 2021-07-31T04:09:41.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:41 smithi045 conmon[39718]: (mgr.24542) 6108 : cluster [DBG] pgmap v6281: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:41.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:41 smithi045 conmon[33655]: cluster 2021-07-31T04:09:39.833792+0000 mgr.x (mgr. 2021-07-31T04:09:41.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:41 smithi045 conmon[33655]: 24542) 6108 : cluster [DBG] pgmap v6281: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:43.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:43 smithi154 conmon[34553]: cluster 2021-07-31T04:09:41.834289+0000 mgr.x (mgr.24542) 6109 : cluster [DBG] pgmap v6282: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:43.524 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:43 smithi045 conmon[39718]: cluster 2021-07-31T04:09:41.834289+0000 mgr.x (mgr 2021-07-31T04:09:43.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:43 smithi045 conmon[39718]: .24542) 6109 : cluster [DBG] pgmap v6282: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:43.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:43 smithi045 conmon[33655]: cluster 2021-07-31T04:09:41.834289+0000 mgr.x 2021-07-31T04:09:43.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:43 smithi045 conmon[33655]: (mgr.24542) 6109 : cluster [DBG] pgmap v6282: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:43.701 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T04:09:43.979 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T04:09:43.980 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 2h ago 2h - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T04:09:43.980 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 2h ago 2h 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T04:09:43.980 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 2h ago 2h 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T04:09:43.980 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 2h ago 2h - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T04:09:43.980 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T04:09:43.981 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 2h ago 2h 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T04:09:43.981 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T04:09:43.981 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 2h ago 2h - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T04:09:43.981 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 2h ago 2h 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T04:09:43.981 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T04:09:43.982 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T04:09:43.982 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T04:09:43.982 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T04:09:43.982 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 2h ago 2h 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T04:09:43.982 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 2h ago 2h 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T04:09:43.983 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 2h ago 2h 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T04:09:43.983 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 2h ago 2h 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T04:09:43.983 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 2h ago 2h 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T04:09:44.314 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T04:09:44.314 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T04:09:44.315 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T04:09:44.315 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:09:44.315 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T04:09:44.315 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T04:09:44.315 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:09:44.316 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:09:44.316 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T04:09:44.316 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T04:09:44.316 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:09:44.317 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T04:09:44.317 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T04:09:44.317 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T04:09:44.317 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:09:44.317 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T04:09:44.318 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T04:09:45.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:45 smithi154 conmon[34553]: audit 2021-07-31T04:09:43.689673+0000 mgr.x (mgr.24542) 6110 : audit [DBG] from='client.21018 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:09:45.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:45 smithi154 conmon[34553]: cluster 2021-07-31T04:09:43.834616+0000 mgr.x (mgr.24542) 6111 : cluster [DBG] pgmap v6283: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:45.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:45 smithi154 conmon[34553]: audit 2021-07-31T04:09:43.972063+0000 mgr.x (mgr.24542) 6112 : audit [DBG] from='client.30961 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:09:45.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:45 smithi154 conmon[34553]: audit 2021 2021-07-31T04:09:45.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:45 smithi154 conmon[34553]: -07-31T04:09:44.315312+0000 mon.a (mon.0) 3432 : audit [DBG] from='client.? 172.21.15.45:0/3910001819' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:09:45.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:45 smithi045 conmon[39718]: audit 2021-07-31T04:09:43.689673+0000 2021-07-31T04:09:45.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:45 smithi045 conmon[39718]: mgr.x (mgr.24542) 6110 : audit [DBG] from='client.21018 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:09:45.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:45 smithi045 conmon[39718]: cluster 2021-07-31T04:09:45.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:45 smithi045 conmon[39718]: 2021-07-31T04:09:43.834616+0000 mgr.x (mgr.24542) 2021-07-31T04:09:45.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:45 smithi045 conmon[39718]: 6111 : cluster [DBG] pgmap v6283: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:45.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:45 smithi045 conmon[39718]: audit 2021 2021-07-31T04:09:45.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:45 smithi045 conmon[39718]: -07-31T04:09:43.972063 2021-07-31T04:09:45.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:45 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 6112 : audit 2021-07-31T04:09:45.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:45 smithi045 conmon[39718]: [DBG] from='client.30961 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:09:45.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:45 smithi045 conmon[39718]: audit 2021 2021-07-31T04:09:45.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:45 smithi045 conmon[39718]: -07-31T04:09:44.315312 2021-07-31T04:09:45.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:45 smithi045 conmon[39718]: +0000 mon.a (mon.0) 3432 : audit [DBG] from='client.? 172.21.15.45:0/3910001819' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:09:45.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:45 smithi045 conmon[39718]: 2021-07-31T04:09:45.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:45 smithi045 conmon[33655]: audit 2021-07-31T04:09:43.689673+0000 mgr.x (mgr.24542) 6110 : audit [DBG] from='client.21018 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:09:45.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:45 smithi045 conmon[33655]: cluster 2021-07-31T04:09:43.834616+0000 mgr.x (mgr.24542) 6111 : cluster [DBG] pgmap v6283: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:45.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:45 smithi045 conmon[33655]: audit 2021-07-31T04:09 2021-07-31T04:09:45.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:45 smithi045 conmon[33655]: :43.972063+0000 mgr.x (mgr.24542) 6112 : audit [DBG] from='client.30961 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:09:45.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:45 smithi045 conmon[33655]: audit 2021-07 2021-07-31T04:09:45.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:45 smithi045 conmon[33655]: -31T04:09:44.315312+0000 mon.a (mon.0) 3432 : audit [DBG] from='client.? 172.21.15.45:0/3910001819' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:09:45.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:45 smithi045 conmon[33655]: 2021-07-31T04:09:47.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:47 smithi154 conmon[34553]: cluster 2021-07-31T04:09:45.835252+0000 mgr.x (mgr.24542) 6113 : cluster [DBG] pgmap v6284: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:47.524 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:47 smithi045 conmon[33655]: cluster 2021-07-31T04:09:45.835252+0000 mgr.x (mgr.24542) 6113 2021-07-31T04:09:47.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:47 smithi045 conmon[33655]: : cluster [DBG] pgmap v6284: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:47.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:47 smithi045 conmon[39718]: cluster 2021-07-31T04:09:45.835252+0000 mgr.x ( 2021-07-31T04:09:47.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:47 smithi045 conmon[39718]: mgr.24542) 6113 : cluster [DBG] pgmap v6284: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:49.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:49 smithi154 conmon[34553]: cluster 2021-07-31T04:09:47.835780+0000 mgr.x (mgr.24542) 6114 : cluster [DBG] pgmap v6285: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:49.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:49 smithi045 conmon[33655]: cluster 2021-07-31T04:09:47.835780+0000 mgr.x (mgr.24542) 2021-07-31T04:09:49.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:49 smithi045 conmon[33655]: 6114 : cluster [DBG] pgmap v6285: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:49.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:49 smithi045 conmon[39718]: cluster 2021-07-31T04:09:47.835780+0000 mgr.x 2021-07-31T04:09:49.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:49 smithi045 conmon[39718]: (mgr.24542) 6114 : cluster [DBG] pgmap v6285: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:51.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:51 smithi154 conmon[34553]: cluster 2021-07-31T04:09:49.836460+0000 mgr.x (mgr.24542) 6115 : cluster [DBG] pgmap v6286: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:51.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:51 smithi045 conmon[33655]: cluster 2021-07-31T04:09:49.836460+0000 mgr.x (mgr.24542) 6115 : cluster 2021-07-31T04:09:51.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:51 smithi045 conmon[33655]: [DBG] pgmap v6286: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:51.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:51 smithi045 conmon[39718]: cluster 2021-07-31T04:09:49.836460+0000 mgr.x (mgr.24542) 6115 : cluster [DBG] pgmap v6286: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:53.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:53 smithi154 conmon[34553]: cluster 2021-07-31T04:09:51.836900+0000 mgr.x (mgr.24542) 6116 : cluster [DBG] pgmap v6287: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:53.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:53 smithi045 conmon[33655]: cluster 2021-07-31T04:09:51.836900+0000 mgr.x (mgr.24542) 2021-07-31T04:09:53.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:53 smithi045 conmon[33655]: 6116 : cluster [DBG] pgmap v6287: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:53.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:53 smithi045 conmon[39718]: cluster 2021-07-31T04:09:51.836900+0000 mgr.x (mgr.24542 2021-07-31T04:09:53.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:53 smithi045 conmon[39718]: ) 6116 : cluster [DBG] pgmap v6287: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:55.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:55 smithi154 conmon[34553]: cluster 2021-07-31T04:09:53.837413+0000 mgr.x (mgr.24542) 6117 : cluster [DBG] pgmap v6288: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:55.524 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:55 smithi045 conmon[39718]: cluster 2021-07-31T04:09:53.837413+0000 mgr.x 2021-07-31T04:09:55.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:55 smithi045 conmon[39718]: (mgr.24542) 6117 : cluster [DBG] pgmap v6288: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:55.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:55 smithi045 conmon[33655]: cluster 2021-07-31T04:09:53.837413+0000 mgr.x (mgr.24542) 6117 : 2021-07-31T04:09:55.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:55 smithi045 conmon[33655]: cluster [DBG] pgmap v6288: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:57.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:57 smithi154 conmon[34553]: cluster 2021-07-31T04:09:55.837890+0000 mgr.x (mgr.24542) 6118 : cluster [DBG] 2021-07-31T04:09:57.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:57 smithi154 conmon[34553]: pgmap v6289: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:57.524 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:57 smithi045 conmon[39718]: cluster 2021-07-31T04:09:55.837890+0000 mgr.x (mgr.24542) 6118 : cluster [DBG] pgmap v6289: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:57.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:57 smithi045 conmon[39718]: 2021-07-31T04:09:57.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:57 smithi045 conmon[33655]: cluster 2021-07-31T04:09:55.837890+0000 mgr.x (mgr.24542) 6118 : cluster [DBG] pgmap v6289: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:57.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:57 smithi045 conmon[33655]: 2021-07-31T04:09:59.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:09:59 smithi154 conmon[34553]: cluster 2021-07-31T04:09:57.838345+0000 mgr.x (mgr.24542) 6119 : cluster [DBG] pgmap v6290: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:59.524 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:59 smithi045 conmon[33655]: cluster 2021-07-31T04:09:57.838345+0000 2021-07-31T04:09:59.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:09:59 smithi045 conmon[33655]: mgr.x (mgr.24542) 6119 : cluster [DBG] pgmap v6290: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:09:59.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:59 smithi045 conmon[39718]: cluster 2021-07-31T04:09:57.838345+0000 2021-07-31T04:09:59.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:09:59 smithi045 conmon[39718]: mgr.x (mgr.24542) 6119 : cluster [DBG] pgmap v6290: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:00.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:00 smithi154 conmon[34553]: cluster 2021-07-31T04:10:00.000128+0000 mon.a (mon.0) 3433 : cluster [INF] overall HEALTH_OK 2021-07-31T04:10:00.524 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:00 smithi045 conmon[33655]: cluster 2021-07-31T04:10:00. 2021-07-31T04:10:00.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:00 smithi045 conmon[33655]: 000128+0000 mon.a (mon.0) 3433 : cluster [INF] overall HEALTH_OK 2021-07-31T04:10:00.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:00 smithi045 conmon[39718]: cluster 2021-07-31T04:10:00.000128+0000 mon.a 2021-07-31T04:10:00.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:00 smithi045 conmon[39718]: (mon.0) 3433 : cluster [INF] overall HEALTH_OK 2021-07-31T04:10:01.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:01 smithi154 conmon[34553]: cluster 2021-07-31T04:09:59.838783+0000 mgr.x (mgr.24542) 6120 : cluster [DBG] pgmap v6291: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:01.524 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:01 smithi045 conmon[33655]: cluster 2021-07-31T04:09:59.838783 2021-07-31T04:10:01.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:01 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 6120 : cluster [DBG] pgmap v6291: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:01.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:01 smithi045 conmon[39718]: cluster 2021-07-31T 2021-07-31T04:10:01.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:01 smithi045 conmon[39718]: 04:09:59.838783+0000 mgr.x (mgr.24542) 6120 : cluster [DBG] pgmap v6291: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:03.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:03 smithi154 conmon[34553]: cluster 2021-07-31T04:10:01.839205+0000 mgr.x (mgr.24542) 6121 : cluster [DBG] pgmap v6292: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:03.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:03 smithi045 conmon[39718]: cluster 2021-07-31T04:10:01.839205+0000 mgr.x (mgr 2021-07-31T04:10:03.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:03 smithi045 conmon[39718]: .24542) 6121 : cluster [DBG] pgmap v6292: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:03.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:03 smithi045 conmon[33655]: cluster 2021-07-31T04:10:01.839205+0000 mgr.x (mgr.24542) 2021-07-31T04:10:03.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:03 smithi045 conmon[33655]: 6121 : cluster [DBG] pgmap v6292: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:05.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:05 smithi154 conmon[34553]: cluster 2021-07-31T04:10:03.839802+0000 mgr.x (mgr.24542) 6122 : cluster [DBG] pgmap v6293: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:05.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:05 smithi045 conmon[33655]: cluster 2021-07-31T04:10: 2021-07-31T04:10:05.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:05 smithi045 conmon[33655]: 03.839802+0000 mgr.x (mgr.24542) 6122 : cluster [DBG] pgmap v6293: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:05.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:05 smithi045 conmon[39718]: cluster 2021-07-31T04:10:03.839802+0000 mgr.x (mgr. 2021-07-31T04:10:05.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:05 smithi045 conmon[39718]: 24542) 6122 : cluster [DBG] pgmap v6293: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:07.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:07 smithi154 conmon[34553]: cluster 2021-07-31T04:10:05.840415+0000 mgr.x (mgr.24542) 6123 : cluster [DBG] 2021-07-31T04:10:07.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:07 smithi154 conmon[34553]: pgmap v6294: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:07.524 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:07 smithi045 conmon[33655]: cluster 2021-07-31T04:10:05.840415+0000 mgr.x (mgr.24542) 6123 : cluster [DBG] pgmap v6294: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:07.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:07 smithi045 conmon[39718]: cluster 2021-07-31T04:10:05.840415+0000 mgr.x (mgr 2021-07-31T04:10:07.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:07 smithi045 conmon[39718]: .24542) 6123 : cluster [DBG] pgmap v6294: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:09.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:09 smithi154 conmon[34553]: cluster 2021-07-31T04:10:07.840804+0000 mgr.x (mgr.24542) 2021-07-31T04:10:09.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:09 smithi154 conmon[34553]: 6124 : cluster [DBG] pgmap v6295: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:09.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:09 smithi045 conmon[33655]: cluster 2021-07-31T04:10:07.840804+0000 mgr.x (mgr.24542) 6124 2021-07-31T04:10:09.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:09 smithi045 conmon[33655]: : cluster [DBG] pgmap v6295: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:09.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:09 smithi045 conmon[39718]: cluster 2021-07-31T04:10:07.840804+0000 2021-07-31T04:10:09.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:09 smithi045 conmon[39718]: mgr.x (mgr.24542) 6124 : cluster [DBG] pgmap v6295: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:11.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:11 smithi154 conmon[34553]: cluster 2021-07-31T04:10:09.841294+0000 mgr.x (mgr.24542) 6125 : cluster [DBG] pgmap v6296: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:11.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:11 smithi045 conmon[33655]: cluster 2021-07-31T04:10:09.841294+0000 mgr.x (mgr.24542) 6125 : cluster [DBG] pgmap v6296: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:11.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:11 smithi045 conmon[39718]: cluster 2021-07-31T04:10:09.841294+0000 mgr.x (mgr.24542) 6125 : cluster [DBG] pgmap v6296: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:13.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:13 smithi154 conmon[34553]: cluster 2021-07-31T04:10:11.841846+0000 mgr.x (mgr.24542) 6126 : cluster [DBG] pgmap v6297: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:13.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:13 smithi045 conmon[33655]: cluster 2021-07-31T04:10:11.841846 2021-07-31T04:10:13.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:13 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 6126 : cluster [DBG] pgmap v6297: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:13.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:13 smithi045 conmon[39718]: cluster 2021-07-31T04:10:11.841846+0000 mgr.x (mgr 2021-07-31T04:10:13.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:13 smithi045 conmon[39718]: .24542) 6126 : cluster [DBG] pgmap v6297: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:14.626 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T04:10:14.899 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T04:10:14.899 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 2h ago 2h - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T04:10:14.900 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 2h ago 2h 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T04:10:14.900 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 2h ago 2h 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T04:10:14.900 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 2h ago 2h - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T04:10:14.900 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T04:10:14.900 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 2h ago 2h 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T04:10:14.901 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T04:10:14.901 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 2h ago 2h - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T04:10:14.901 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 2h ago 2h 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T04:10:14.901 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T04:10:14.901 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T04:10:14.902 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T04:10:14.902 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T04:10:14.902 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 2h ago 2h 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T04:10:14.902 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 2h ago 2h 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T04:10:14.902 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 2h ago 2h 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T04:10:14.903 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 2h ago 2h 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T04:10:14.903 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 2h ago 2h 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T04:10:15.233 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T04:10:15.233 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T04:10:15.233 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T04:10:15.234 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:10:15.234 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T04:10:15.234 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T04:10:15.234 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:10:15.235 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:10:15.235 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T04:10:15.235 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T04:10:15.235 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:10:15.235 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T04:10:15.236 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T04:10:15.236 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T04:10:15.236 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:10:15.236 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T04:10:15.236 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T04:10:15.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:15 smithi154 conmon[34553]: cluster 2021-07-31T04:10:13.842294+0000 mgr.x (mgr.24542) 6127 : cluster [DBG] pgmap v6298: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:15.524 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:15 smithi045 conmon[33655]: cluster 2021-07-31T04:10:13.842294+0000 mgr.x ( 2021-07-31T04:10:15.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:15 smithi045 conmon[33655]: mgr.24542) 6127 : cluster [DBG] pgmap v6298: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:15.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:15 smithi045 conmon[39718]: cluster 2021-07-31T04:10:13.842294+0000 mgr.x ( 2021-07-31T04:10:15.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:15 smithi045 conmon[39718]: mgr.24542) 6127 : cluster [DBG] pgmap v6298: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:16.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:16 smithi154 conmon[34553]: audit 2021-07-31T04:10:14.614641+0000 mgr.x (mgr.24542) 6128 : audit [DBG] from='client.30973 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:10:16.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:16 smithi154 conmon[34553]: audit 2021-07-31T04:10:14.892301+0000 mgr.x (mgr.24542) 6129 : audit [DBG] from='client.30979 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:10:16.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:16 smithi154 conmon[34553]: audit 2021-07-31T04 2021-07-31T04:10:16.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:16 smithi154 conmon[34553]: :10:15.234501+0000 mon.a (mon.0) 3434 : audit [DBG] from='client.? 172.21.15.45:0/2291215363' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:10:16.524 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:16 smithi045 conmon[33655]: audit 2021-07-31T04:10:14.614641+0000 mgr.x (mgr.24542) 6128 : audit [DBG] from='client.30973 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:10:16.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:16 smithi045 conmon[33655]: 2021-07-31T04:10:16.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:16 smithi045 conmon[33655]: audit 2021-07-31T04:10:14.892301+0000 mgr.x (mgr.24542) 6129 : audit [DBG] from='client.30979 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:10:16.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:16 smithi045 conmon[33655]: audit 2021-07-31T04:10:15.234501+0000 mon.a (mon 2021-07-31T04:10:16.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:16 smithi045 conmon[33655]: .0) 3434 : audit [DBG] from='client.? 172.21.15.45:0/2291215363' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:10:16.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:16 smithi045 conmon[39718]: audit 2021-07-31T04:10:14.614641+0000 mgr.x ( 2021-07-31T04:10:16.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:16 smithi045 conmon[39718]: mgr.24542) 6128 : audit [DBG] from='client.30973 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:10:16.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:16 smithi045 conmon[39718]: audit 2021-07-31T04:10:14.892301+0000 mgr.x (mgr.24542) 2021-07-31T04:10:16.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:16 smithi045 conmon[39718]: 6129 : audit [DBG] from='client.30979 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:10:16.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:16 smithi045 conmon[39718]: audit 2021- 2021-07-31T04:10:16.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:16 smithi045 conmon[39718]: 07-31T04:10:15.234501+0000 mon.a (mon.0 2021-07-31T04:10:16.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:16 smithi045 conmon[39718]: ) 3434 : audit [DBG] from='client.? 172.21.15.45:0/2291215363' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:10:17.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:17 smithi154 conmon[34553]: cluster 2021-07-31T04:10:15.842779+0000 mgr.x (mgr.24542) 6130 : cluster [DBG] pgmap v6299: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:17.524 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:17 smithi045 conmon[39718]: cluster 2021-07-31T04:10:15.842779+0000 mgr.x ( 2021-07-31T04:10:17.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:17 smithi045 conmon[39718]: mgr.24542) 6130 : cluster [DBG] pgmap v6299: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:17.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:17 smithi045 conmon[33655]: cluster 2021-07-31T04:10:15.842779+0000 mgr.x (mgr.24542) 6130 : 2021-07-31T04:10:17.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:17 smithi045 conmon[33655]: cluster [DBG] pgmap v6299: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:19.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:19 smithi154 conmon[34553]: cluster 2021-07-31T04:10:17.843183+0000 mgr.x (mgr.24542) 6131 : cluster [DBG] pgmap v6300: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:19.524 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:19 smithi045 conmon[39718]: cluster 2021-07-31T04:10:17.843183+0000 mgr.x (mgr. 2021-07-31T04:10:19.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:19 smithi045 conmon[39718]: 24542) 6131 : cluster [DBG] pgmap v6300: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:19.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:19 smithi045 conmon[33655]: cluster 2021-07-31T04:10:17.843183+0000 mgr.x (mgr. 2021-07-31T04:10:19.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:19 smithi045 conmon[33655]: 24542) 6131 : cluster [DBG] pgmap v6300: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:21.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:21 smithi154 conmon[34553]: cluster 2021-07-31T04:10:19.843777+0000 mgr.x (mgr.24542) 6132 : cluster [DBG] pgmap v6301: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:21.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:21 smithi045 conmon[39718]: cluster 2021-07-31T04:10:19.843777+0000 mgr.x (mgr.24542 2021-07-31T04:10:21.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:21 smithi045 conmon[39718]: ) 6132 : cluster [DBG] pgmap v6301: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:21.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:21 smithi045 conmon[33655]: cluster 2021-07-31T04:10:19.843777+0000 mgr.x (mgr.24542) 6132 : cluster [DBG] 2021-07-31T04:10:21.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:21 smithi045 conmon[33655]: pgmap v6301: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:23.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:23 smithi154 conmon[34553]: cluster 2021-07-31T04:10:21.844201+0000 mgr.x (mgr.24542) 6133 : 2021-07-31T04:10:23.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:23 smithi154 conmon[34553]: cluster [DBG] pgmap v6302: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:23.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:23 smithi045 conmon[39718]: cluster 2021-07-31T04:10:21.844201+0000 mgr.x ( 2021-07-31T04:10:23.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:23 smithi045 conmon[39718]: mgr.24542) 6133 : cluster [DBG] pgmap v6302: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:23.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:23 smithi045 conmon[33655]: cluster 2021-07-31T04:10:21.844201+0000 mgr.x (mgr.24542) 6133 : cluster [DBG] 2021-07-31T04:10:23.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:23 smithi045 conmon[33655]: pgmap v6302: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:25.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:25 smithi154 conmon[34553]: cluster 2021-07-31T04:10:23.844722+0000 mgr.x (mgr.24542) 6134 : cluster [DBG] pgmap v6303: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:25.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:25 smithi154 conmon[34553]: 2021-07-31T04:10:25.524 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:25 smithi045 conmon[33655]: cluster 2021-07-31T04:10:23.844722+0000 mgr.x (mgr.24542) 6134 : cluster [DBG] pgmap v6303: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:25.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:25 smithi045 conmon[39718]: cluster 2021-07-31T04:10:23.844722+0000 mgr.x ( 2021-07-31T04:10:25.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:25 smithi045 conmon[39718]: mgr.24542) 6134 : cluster [DBG] pgmap v6303: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:27.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:27 smithi154 conmon[34553]: cluster 2021-07-31T04:10:25.845288+0000 mgr.x (mgr.24542) 6135 : cluster [DBG] 2021-07-31T04:10:27.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:27 smithi154 conmon[34553]: pgmap v6304: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:27.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:27 smithi045 conmon[33655]: cluster 2021-07-31T04:10:25.845288 2021-07-31T04:10:27.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:27 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 6135 : cluster [DBG] pgmap v6304: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:27.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:27 smithi045 conmon[39718]: cluster 2021-07-31T04:10:25.845288+0000 mgr.x (mgr 2021-07-31T04:10:27.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:27 smithi045 conmon[39718]: .24542) 6135 : cluster [DBG] pgmap v6304: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:29.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:29 smithi154 conmon[34553]: cluster 2021-07-31T04:10:27.845845+0000 mgr.x (mgr.24542) 6136 : cluster [DBG] pgmap v6305: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:29.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:29 smithi154 conmon[34553]: audit 2021-07-31T04:10:28.500300+0000 mon.b (mon.2) 790 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:10:29.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:29 smithi154 conmon[34553]: audit 2021-07-31T04:10:28.500552+0000 mon.b (mon.2) 791 : audit 2021-07-31T04:10:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:29 smithi154 conmon[34553]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:10:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:29 smithi154 conmon[34553]: audit 2021-07-31 2021-07-31T04:10:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:29 smithi154 conmon[34553]: T04:10:28.501414+0000 2021-07-31T04:10:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:29 smithi154 conmon[34553]: mon.a (mon.0) 3435 : audit 2021-07-31T04:10:29.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:29 smithi154 conmon[34553]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:10:29.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:29 smithi154 conmon[34553]: audit 2021-07- 2021-07-31T04:10:29.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:29 smithi154 conmon[34553]: 31T04:10:28.501799+0000 mon.a (mon 2021-07-31T04:10:29.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:29 smithi154 conmon[34553]: .0) 3436 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:10:29.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:29 smithi045 conmon[39718]: cluster 2021-07-31T04:10:27.845845+0000 2021-07-31T04:10:29.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:29 smithi045 conmon[39718]: mgr.x (mgr.24542) 6136 : cluster [DBG] pgmap v6305: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:29.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:29 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T04:10:29.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:29 smithi045 conmon[39718]: 04:10:28.500300+0000 mon.b (mon.2) 790 : audit [INF] 2021-07-31T04:10:29.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:29 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:10:29.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:29 smithi045 conmon[39718]: audit 2021-07-31T04:10:29.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:29 smithi045 conmon[39718]: 2021-07-31T04:10:28.500552+0000 2021-07-31T04:10:29.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:29 smithi045 conmon[39718]: mon.b (mon.2) 791 : audit [INF] 2021-07-31T04:10:29.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:29 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:10:29.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:29 smithi045 conmon[39718]: audit 2021-07-31T04:10:29.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:29 smithi045 conmon[39718]: 2021-07 2021-07-31T04:10:29.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:29 smithi045 conmon[39718]: -31T04:10:28.501414+0000 2021-07-31T04:10:29.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:29 smithi045 conmon[39718]: mon.a (mon.0) 3435 : audit 2021-07-31T04:10:29.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:29 smithi045 conmon[39718]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:10:29.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:29 smithi045 conmon[39718]: audit 2021 2021-07-31T04:10:29.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:29 smithi045 conmon[39718]: -07-31T04:10: 2021-07-31T04:10:29.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:29 smithi045 conmon[39718]: 28.501799+0000 mon.a (mon. 2021-07-31T04:10:29.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:29 smithi045 conmon[39718]: 0) 3436 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:10:29.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:29 smithi045 conmon[33655]: cluster 2021-07-31T04:10:27.845845+0000 mgr.x (mgr.24542) 6136 : cluster [DBG] pgmap v6305: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:29.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:29 smithi045 conmon[33655]: audit 2021-07-31T04:10:28.500300+0000 mon.b (mon.2) 790 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:10:29.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:29 smithi045 conmon[33655]: audit 2021-07-31T 2021-07-31T04:10:29.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:29 smithi045 conmon[33655]: 04:10:28.500552+0000 mon.b (mon.2) 791 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:10:29.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:29 smithi045 conmon[33655]: audit 2021-07-31 2021-07-31T04:10:29.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:29 smithi045 conmon[33655]: T04:10:28.501414+0000 mon.a (mon.0 2021-07-31T04:10:29.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:29 smithi045 conmon[33655]: ) 3435 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:10:29.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:29 smithi045 conmon[33655]: audit 2021 2021-07-31T04:10:29.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:29 smithi045 conmon[33655]: -07-31T04:10:28.501799+0000 mon.a (mon 2021-07-31T04:10:29.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:29 smithi045 conmon[33655]: .0) 3436 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:10:31.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:31 smithi154 conmon[34553]: cluster 2021-07-31T04:10:29.846466+0000 mgr.x (mgr.24542) 6137 : 2021-07-31T04:10:31.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:31 smithi154 conmon[34553]: cluster [DBG] pgmap v6306: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:31.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:31 smithi045 conmon[39718]: cluster 2021-07-31T04:10:29.846466+0000 mgr.x ( 2021-07-31T04:10:31.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:31 smithi045 conmon[39718]: mgr.24542) 6137 : cluster [DBG] pgmap v6306: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:31.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:31 smithi045 conmon[33655]: cluster 2021-07-31T04:10:29.846466+0000 mgr.x (mgr.24542) 6137 : cluster [DBG] 2021-07-31T04:10:31.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:31 smithi045 conmon[33655]: pgmap v6306: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:33.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:33 smithi154 conmon[34553]: cluster 2021-07-31T04:10:31.846867+0000 mgr.x (mgr.24542) 6138 : cluster [DBG] pgmap v6307: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:33.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:33 smithi045 conmon[33655]: cluster 2021-07-31T04:10:31.846867+0000 mgr.x (mgr.24542) 6138 : cluster [DBG] pgmap v6307: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:33.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:33 smithi045 conmon[39718]: cluster 2021-07-31T04:10:31.846867+0000 mgr.x (mgr 2021-07-31T04:10:33.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:33 smithi045 conmon[39718]: .24542) 6138 : cluster [DBG] pgmap v6307: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:35.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:35 smithi154 conmon[34553]: cluster 2021-07-31T04:10:33.847302+0000 mgr.x (mgr.24542) 6139 : cluster [DBG] 2021-07-31T04:10:35.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:35 smithi154 conmon[34553]: pgmap v6308: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:35.524 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:35 smithi045 conmon[33655]: cluster 2021-07-31T04:10:33.847302+0000 mgr.x (mgr.24542) 6139 : cluster 2021-07-31T04:10:35.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:35 smithi045 conmon[33655]: [DBG] pgmap v6308: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:35.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:35 smithi045 conmon[39718]: cluster 2021-07-31T04:10:33.847302+0000 mgr.x (mgr.24542 2021-07-31T04:10:35.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:35 smithi045 conmon[39718]: ) 6139 : cluster [DBG] pgmap v6308: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:37.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[34553]: cluster 2021-07-31T04:10:35.847734+0000 mgr.x (mgr.24542) 6140 : cluster [DBG] pgmap v6309: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:37.503 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: debug 2021-07-31T04:10:37.328+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:10:37.503 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:10:37.503 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:10:37.503 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: yield 2021-07-31T04:10:37.503 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:10:37.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:10:37.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:10:37.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:10:37.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:10:37.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:10:37.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:10:37.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:10:37.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:10:37.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: task.result() 2021-07-31T04:10:37.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:10:37.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: debug 2021-07-31T04:10:37.340+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:10:37.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:10:37.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:10:37.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: yield 2021-07-31T04:10:37.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:10:37.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:10:37.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:10:37.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:10:37.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:10:37.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:10:37.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:10:37.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:10:37.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:10:37.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: task.result() 2021-07-31T04:10:37.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:10:37.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: debug 2021-07-31T04:10:37.366+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:10:37.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:10:37.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:10:37.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: yield 2021-07-31T04:10:37.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:10:37.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:10:37.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:10:37.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:10:37.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:10:37.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:10:37.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:10:37.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:10:37.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:10:37.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: task.result() 2021-07-31T04:10:37.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:10:37.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: debug 2021-07-31T04:10:37.366+0000 7fcb9115c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:10:37.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:10:37.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:10:37.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: yield 2021-07-31T04:10:37.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:10:37.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:10:37.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:10:37.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:10:37.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:10:37.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:10:37.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:10:37.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:10:37.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:10:37.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: task.result() 2021-07-31T04:10:37.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:10:37.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: 2021-07-31T04:10:37.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T04:10:37.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: 2021-07-31T04:10:37.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:10:37.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:10:37.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: return f(*arg) 2021-07-31T04:10:37.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:10:37.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:10:37.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:10:37.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:10:37.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:10:37.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:10:37.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:10:37.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T04:10:37.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:10:37.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: raise self._exception 2021-07-31T04:10:37.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:10:37.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:10:37.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:10:37.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T04:10:37.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:10:37.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: raise 2021-07-31T04:10:37.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:10:37.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T04:10:37.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:10:37.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:10:37.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:10:37.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: debug 2021-07-31T04:10:37.375+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:10:37.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:10:37.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:10:37.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: yield 2021-07-31T04:10:37.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:10:37.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:10:37.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:10:37.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:10:37.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:10:37.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:10:37.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:10:37.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:10:37.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:10:37.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: task.result() 2021-07-31T04:10:37.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:10:37.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: debug 2021-07-31T04:10:37.375+0000 7fcb9095b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:10:37.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:10:37.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:10:37.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: yield 2021-07-31T04:10:37.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:10:37.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:10:37.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:10:37.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:10:37.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:10:37.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:10:37.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:10:37.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:10:37.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:10:37.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: task.result() 2021-07-31T04:10:37.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:10:37.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: 2021-07-31T04:10:37.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T04:10:37.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: 2021-07-31T04:10:37.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:10:37.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:10:37.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: return f(*arg) 2021-07-31T04:10:37.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:10:37.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:10:37.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:10:37.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:10:37.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:10:37.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:10:37.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:10:37.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T04:10:37.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:10:37.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: raise self._exception 2021-07-31T04:10:37.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:10:37.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:10:37.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:10:37.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T04:10:37.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:10:37.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: raise 2021-07-31T04:10:37.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:10:37.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T04:10:37.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:10:37.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:10:37.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:10:37 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:10:37.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:37 smithi045 conmon[39718]: cluster 2021-07-31T04:10:35.847734+0000 mgr.x ( 2021-07-31T04:10:37.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:37 smithi045 conmon[39718]: mgr.24542) 6140 : cluster [DBG] pgmap v6309: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:37.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:37 smithi045 conmon[33655]: cluster 2021-07-31T04:10:35.847734+0000 mgr.x (mgr.24542) 6140 : cluster [DBG] 2021-07-31T04:10:37.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:37 smithi045 conmon[33655]: pgmap v6309: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:38.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: audit 2021-07-31T04:10:37.283155+0000 mon.b (mon.2) 792 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T04:10:38.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: 2021-07-31T04:10:38.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: audit 2021-07-31T04:10:37.284580+0000 mon.b (mon.2) 793 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T04:10:38.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: audit 2021-07-31T04:10:37 2021-07-31T04:10:38.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: .285501+0000 mon.b (mon.2) 794 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T04:10:38.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: cephadm 2021- 2021-07-31T04:10:38.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: 07-31T04:10:37.286275+0000 mgr.x 2021-07-31T04:10:38.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: (mgr.24542) 6141 : cephadm [WRN] 2021-07-31T04:10:38.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T04:10:38.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: cephadm 2021- 2021-07-31T04:10:38.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: 07-31T04:10:37.329570+0000 2021-07-31T04:10:38.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: mgr.x (mgr.24542) 6142 : cephadm 2021-07-31T04:10:38.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: [ERR] 2021-07-31T04:10:38.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:10:38.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:10:38.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: yield 2021-07-31T04:10:38.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:10:38.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:10:38.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:10:38.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:10:38.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:10:38.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:10:38.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:10:38.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:10:38.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:10:38.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: task.result() 2021-07-31T04:10:38.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:10:38.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: cephadm 2021-07-31T04:10:37.330011+0000 mgr.x (mgr.24542) 6143 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T04:10:38.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: cephadm 2021- 2021-07-31T04:10:38.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: 07-31T04:10:37.341208 2021-07-31T04:10:38.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: +0000 mgr.x (mgr.24542) 6144 2021-07-31T04:10:38.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: : cephadm [ERR] 2021-07-31T04:10:38.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:10:38.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:10:38.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: yield 2021-07-31T04:10:38.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:10:38.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:10:38.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:10:38.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:10:38.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:10:38.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:10:38.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:10:38.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:10:38.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:10:38.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: task.result() 2021-07-31T04:10:38.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:10:38.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: cephadm 2021-07-31T04:10:37.341657+0000 mgr.x (mgr.24542) 6145 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T04:10:38.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: cephadm 2021- 2021-07-31T04:10:38.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: 07-31T04:10:37.367189 2021-07-31T04:10:38.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: +0000 mgr.x (mgr.24542) 6146 : 2021-07-31T04:10:38.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: cephadm [ERR] 2021-07-31T04:10:38.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:10:38.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:10:38.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: yield 2021-07-31T04:10:38.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:10:38.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:10:38.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:10:38.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:10:38.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:10:38.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:10:38.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:10:38.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:10:38.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:10:38.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: task.result() 2021-07-31T04:10:38.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:10:38.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: cephadm 2021-07-31T04:10:37.367626+0000 mgr.x (mgr.24542) 6147 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:10:38.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:10:38.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:10:38.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: yield 2021-07-31T04:10:38.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:10:38.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:10:38.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:10:38.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:10:38.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:10:38.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:10:38.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:10:38.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:10:38.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:10:38.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: task.result() 2021-07-31T04:10:38.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:10:38.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: 2021-07-31T04:10:38.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T04:10:38.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: 2021-07-31T04:10:38.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:10:38.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:10:38.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: return f(*arg) 2021-07-31T04:10:38.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:10:38.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:10:38.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:10:38.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:10:38.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:10:38.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:10:38.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:10:38.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T04:10:38.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:10:38.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: raise self._exception 2021-07-31T04:10:38.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:10:38.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:10:38.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:10:38.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T04:10:38.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:10:38.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: raise 2021-07-31T04:10:38.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:10:38.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T04:10:38.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:10:38.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:10:38.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:10:38.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: cephadm 2021-07-31T04:10:37.376251+0000 mgr.x (mgr.24542) 6148 : cephadm [ERR] 2021-07-31T04:10:38.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:10:38.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:10:38.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: yield 2021-07-31T04:10:38.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:10:38.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:10:38.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:10:38.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:10:38.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:10:38.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:10:38.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:10:38.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:10:38.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:10:38.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: task.result() 2021-07-31T04:10:38.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:10:38.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: cephadm 2021-07-31T04:10:37.376639+0000 mgr.x (mgr.24542) 6149 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:10:38.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:10:38.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:10:38.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: yield 2021-07-31T04:10:38.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:10:38.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:10:38.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:10:38.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:10:38.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:10:38.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:10:38.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:10:38.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:10:38.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:10:38.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: task.result() 2021-07-31T04:10:38.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:10:38.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: 2021-07-31T04:10:38.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T04:10:38.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: 2021-07-31T04:10:38.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:10:38.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:10:38.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: return f(*arg) 2021-07-31T04:10:38.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:10:38.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:10:38.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:10:38.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:10:38.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:10:38.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:10:38.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:10:38.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T04:10:38.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:10:38.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: raise self._exception 2021-07-31T04:10:38.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:10:38.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:10:38.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:10:38.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T04:10:38.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:10:38.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: raise 2021-07-31T04:10:38.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:10:38.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T04:10:38.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:10:38.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:10:38.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:38 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:10:38.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: audit 2021-07-31T04:10:37.283155+0000 mon.b (mon.2) 792 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T04:10:38.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: audit 2021-07 2021-07-31T04:10:38.543 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: -31T04:10:37.284580+0000 mon.b (mon.2) 793 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T04:10:38.543 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: audit 2021-07-31T04:10:37.285501+0000 mon.b (mon.2) 794 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T04:10:38.543 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: cephadm 2021-07-31T04:10: 2021-07-31T04:10:38.543 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: 37.286275+0000 mgr.x (mgr.24542) 6141 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T04:10:38.544 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: 2021-07-31T04:10:38.544 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: cephadm 2021-07-31T04:10:38.544 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: 2021-07-31T04:10:37.329570+0000 mgr.x (mgr.24542) 6142 : 2021-07-31T04:10:38.544 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: cephadm [ERR] 2021-07-31T04:10:38.544 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:10:38.545 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:10:38.545 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: yield 2021-07-31T04:10:38.545 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:10:38.545 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:10:38.546 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:10:38.546 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:10:38.546 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:10:38.546 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:10:38.546 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:10:38.547 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:10:38.547 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:10:38.547 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: task.result() 2021-07-31T04:10:38.547 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:10:38.548 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: cephadm 2021-07-31T04:10:38.548 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: 2021-07-31T04:10:37.330011+0000 mgr.x (mgr.24542) 6143 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T04:10:38.548 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: cephadm 2021-07-31T04:10:37.341208+0000 mgr.x (mgr.24542) 6144 : cephadm [ERR] 2021-07-31T04:10:38.548 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:10:38.548 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:10:38.549 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: yield 2021-07-31T04:10:38.549 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:10:38.549 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:10:38.549 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:10:38.549 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:10:38.550 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:10:38.550 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:10:38.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: audit 2021-07-31T04:10:37.283155+0000 mon.b ( 2021-07-31T04:10:38.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: mon.2) 792 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T04:10:38.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: audit 2021-07-31T04:10:37 2021-07-31T04:10:38.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: .284580+0000 mon.b (mon.2) 793 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T04:10:38.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: 2021-07-31T04:10:38.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: audit 2021-07-31T04:10:38.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: 2021-07-31T04:10 2021-07-31T04:10:38.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: :37.285501+0000 mon.b (mon 2021-07-31T04:10:38.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: .2) 794 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T04:10:38.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: cephadm 2021 2021-07-31T04:10:38.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: -07-31T04:10:37.286275+0000 2021-07-31T04:10:38.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: mgr.x (mgr.24542) 6141 : cephadm [WRN] 2021-07-31T04:10:38.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T04:10:38.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: cephadm 2021-07-31T04:10:38.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: 2021-07 2021-07-31T04:10:38.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: -31T04:10:37.329570 2021-07-31T04:10:38.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 6142 : cephadm [ERR] 2021-07-31T04:10:38.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: 2021-07-31T04:10:38.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:10:38.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:10:38.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: yield 2021-07-31T04:10:38.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:10:38.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:10:38.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:10:38.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:10:38.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:10:38.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:10:38.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:10:38.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:10:38.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:10:38.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: task.result() 2021-07-31T04:10:38.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:10:38.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: cephadm 2021-07-31T04:10:37.330011+0000 mgr.x (mgr.24542) 6143 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T04:10:38.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: cephadm 2021- 2021-07-31T04:10:38.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: 07-31T04:10:37.341208+0000 mgr.x (mgr. 2021-07-31T04:10:38.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: 24542) 6144 : cephadm [ERR] 2021-07-31T04:10:38.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:10:38.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:10:38.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: yield 2021-07-31T04:10:38.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:10:38.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:10:38.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:10:38.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:10:38.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:10:38.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:10:38.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: task.result() 2021-07-31T04:10:38.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:10:38.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: cephadm 2021-07-31T04:10 2021-07-31T04:10:38.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: :37.341657+0000 mgr.x (mgr.24542) 6145 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T04:10:38.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: cephadm 2021-07-31T04:10:37.367189+0000 mgr.x (mgr.24542) 6146 : cephadm [ERR] 2021-07-31T04:10:38.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:10:38.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:10:38.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: yield 2021-07-31T04:10:38.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:10:38.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:10:38.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:10:38.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:10:38.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:10:38.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:10:38.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:10:38.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:10:38.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:10:38.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: task.result() 2021-07-31T04:10:38.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:10:38.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: cephadm 2021-07-31T04:10:37.367626+0000 mgr.x (mgr.24542) 6147 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:10:38.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:10:38.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:10:38.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: yield 2021-07-31T04:10:38.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:10:38.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:10:38.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:10:38.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:10:38.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:10:38.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:10:38.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:10:38.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:10:38.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:10:38.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: task.result() 2021-07-31T04:10:38.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:10:38.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: 2021-07-31T04:10:38.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T04:10:38.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: 2021-07-31T04:10:38.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:10:38.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:10:38.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:10:38.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:10:38.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:10:38.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:10:38.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:10:38.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:10:38.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: task.result() 2021-07-31T04:10:38.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:10:38.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: cephadm 2021-07-31T04:10:37.341657+0000 mgr.x (mgr.24542) 6145 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T04:10:38.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: cephadm 2021-07-31T 2021-07-31T04:10:38.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: 04:10:37.367189+0000 mgr.x (mgr.24542) 2021-07-31T04:10:38.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: 6146 : cephadm [ERR] 2021-07-31T04:10:38.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:10:38.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:10:38.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: yield 2021-07-31T04:10:38.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:10:38.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:10:38.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:10:38.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:10:38.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:10:38.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:10:38.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:10:38.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:10:38.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:10:38.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: task.result() 2021-07-31T04:10:38.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:10:38.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: cephadm 2021-07-31T04:10:37.367626+0000 mgr.x (mgr.24542) 6147 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:10:38.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:10:38.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:10:38.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: yield 2021-07-31T04:10:38.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:10:38.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:10:38.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:10:38.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:10:38.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:10:38.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:10:38.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:10:38.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:10:38.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:10:38.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: task.result() 2021-07-31T04:10:38.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: return f(*arg) 2021-07-31T04:10:38.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:10:38.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:10:38.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:10:38.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:10:38.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:10:38.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:10:38.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:10:38.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T04:10:38.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:10:38.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: raise self._exception 2021-07-31T04:10:38.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:10:38.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:10:38.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:10:38.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T04:10:38.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:10:38.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: raise 2021-07-31T04:10:38.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:10:38.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T04:10:38.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:10:38.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:10:38.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:10:38.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: cephadm 2021-07-31T04:10:37.376251+0000 mgr.x (mgr.24542) 6148 : cephadm [ERR] 2021-07-31T04:10:38.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:10:38.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:10:38.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: yield 2021-07-31T04:10:38.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:10:38.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:10:38.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:10:38.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:10:38.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:10:38.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:10:38.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:10:38.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:10:38.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:10:38.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: task.result() 2021-07-31T04:10:38.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:10:38.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: cephadm 2021-07-31T04:10:37.376639+0000 mgr.x (mgr.24542) 6149 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:10:38.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:10:38.585 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:10:38.585 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: 2021-07-31T04:10:38.585 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T04:10:38.585 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: 2021-07-31T04:10:38.586 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:10:38.586 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:10:38.586 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: return f(*arg) 2021-07-31T04:10:38.586 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:10:38.586 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:10:38.587 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:10:38.587 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:10:38.587 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:10:38.587 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:10:38.587 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:10:38.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T04:10:38.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:10:38.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: raise self._exception 2021-07-31T04:10:38.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:10:38.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:10:38.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:10:38.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T04:10:38.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:10:38.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: raise 2021-07-31T04:10:38.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:10:38.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T04:10:38.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:10:38.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:10:38.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:10:38.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: cephadm 2021-07-31T04:10:38.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: 2021-07-31T04:10:37.376251+0000 mgr.x (mgr.24542) 6148 : cephadm [ERR] 2021-07-31T04:10:38.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:10:38.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:10:38.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: yield 2021-07-31T04:10:38.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:10:38.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:10:38.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:10:38.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:10:38.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:10:38.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:10:38.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:10:38.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:10:38.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:10:38.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: yield 2021-07-31T04:10:38.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:10:38.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:10:38.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:10:38.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:10:38.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:10:38.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:10:38.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:10:38.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:10:38.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:10:38.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: task.result() 2021-07-31T04:10:38.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:10:38.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: 2021-07-31T04:10:38.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T04:10:38.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: 2021-07-31T04:10:38.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:10:38.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:10:38.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: return f(*arg) 2021-07-31T04:10:38.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:10:38.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:10:38.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:10:38.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:10:38.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:10:38.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:10:38.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:10:38.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T04:10:38.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:10:38.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: raise self._exception 2021-07-31T04:10:38.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:10:38.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:10:38.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:10:38.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T04:10:38.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:10:38.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: raise 2021-07-31T04:10:38.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:10:38.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T04:10:38.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:10:38.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:10:38.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:10:38.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:10:38.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: task.result() 2021-07-31T04:10:38.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:10:38.603 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: cephadm 2021-07-31T04:10:37.376639+0000 mgr.x (mgr.24542) 6149 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:10:38.603 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:10:38.603 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:10:38.603 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: yield 2021-07-31T04:10:38.604 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:10:38.604 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:10:38.604 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:10:38.604 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:10:38.604 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:10:38.604 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:10:38.605 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:10:38.605 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:10:38.605 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:10:38.605 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: task.result() 2021-07-31T04:10:38.605 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:10:38.606 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: 2021-07-31T04:10:38.606 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T04:10:38.606 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: 2021-07-31T04:10:38.606 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:10:38.606 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:10:38.607 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: return f(*arg) 2021-07-31T04:10:38.607 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:10:38.607 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:10:38.607 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:10:38.607 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:10:38.608 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:10:38.608 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:10:38.608 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:10:38.608 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T04:10:38.608 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:10:38.609 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: raise self._exception 2021-07-31T04:10:38.609 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:10:38.609 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:10:38.609 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:10:38.609 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T04:10:38.610 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:10:38.610 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: raise 2021-07-31T04:10:38.610 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:10:38.610 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T04:10:38.610 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:10:38.611 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:10:38.611 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:38 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:10:39.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:39 smithi154 conmon[34553]: cluster 2021-07-31T04:10:37.848293+0000 mgr.x (mgr.24542) 6150 : cluster [DBG] pgmap v6310: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:39.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:39 smithi045 conmon[33655]: cluster 2021-07-31T04:10:37.848293+0000 mgr.x (mgr.24542) 6150 : 2021-07-31T04:10:39.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:39 smithi045 conmon[33655]: cluster [DBG] pgmap v6310: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:39.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:39 smithi045 conmon[39718]: cluster 2021-07-31T04:10:37.848293+0000 mgr.x (mgr.24542 2021-07-31T04:10:39.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:39 smithi045 conmon[39718]: ) 6150 : cluster [DBG] pgmap v6310: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:41.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:41 smithi154 conmon[34553]: cluster 2021-07-31T04:10:39.848837+0000 mgr.x (mgr.24542) 6151 : cluster [DBG] pgmap v6311: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:41.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:41 smithi045 conmon[39718]: cluster 2021-07-31T04:10:39.848837+0000 mgr.x (mgr.24542) 6151 : 2021-07-31T04:10:41.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:41 smithi045 conmon[39718]: cluster [DBG] pgmap v6311: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:41.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:41 smithi045 conmon[33655]: cluster 2021-07-31T04:10:39.848837+0000 mgr.x (mgr.24542) 6151 2021-07-31T04:10:41.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:41 smithi045 conmon[33655]: : cluster [DBG] pgmap v6311: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:43.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:43 smithi154 conmon[34553]: cluster 2021-07-31T04:10:41.849279+0000 mgr.x (mgr.24542) 6152 : cluster [DBG] pgmap v6312: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:43.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:43 smithi154 conmon[34553]: 2021-07-31T04:10:43.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:43 smithi045 conmon[33655]: cluster 2021-07-31T04:10:41.849279+0000 mgr.x (mgr.24542) 6152 : cluster [DBG] pgmap v6312: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:43.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:43 smithi045 conmon[39718]: cluster 2021-07-31T04:10:41.849279+0000 mgr.x (mgr. 2021-07-31T04:10:43.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:43 smithi045 conmon[39718]: 24542) 6152 : cluster [DBG] pgmap v6312: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:45.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:45 smithi154 conmon[34553]: cluster 2021-07-31T04:10:43.849768+0000 mgr.x (mgr.24542) 6153 : cluster [DBG] pgmap v6313: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:45.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:45 smithi154 conmon[34553]: 2021-07-31T04:10:45.524 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:45 smithi045 conmon[39718]: cluster 2021-07-31T04:10:43.849768+0000 mgr.x 2021-07-31T04:10:45.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:45 smithi045 conmon[39718]: (mgr.24542) 6153 : cluster [DBG] pgmap v6313: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:45.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:45 smithi045 conmon[33655]: cluster 2021-07-31T04:10:43.849768+0000 mgr.x (mgr.24542) 2021-07-31T04:10:45.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:45 smithi045 conmon[33655]: 6153 : cluster [DBG] pgmap v6313: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:45.545 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T04:10:45.823 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T04:10:45.824 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 2h ago 2h - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T04:10:45.824 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 2h ago 2h 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T04:10:45.824 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 2h ago 2h 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T04:10:45.824 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 2h ago 2h - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T04:10:45.825 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T04:10:45.825 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 2h ago 2h 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T04:10:45.825 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T04:10:45.825 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 2h ago 2h - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T04:10:45.825 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 2h ago 2h 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T04:10:45.826 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T04:10:45.826 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T04:10:45.826 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T04:10:45.826 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T04:10:45.827 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 2h ago 2h 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T04:10:45.827 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 2h ago 2h 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T04:10:45.827 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 2h ago 2h 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T04:10:45.827 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 2h ago 2h 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T04:10:45.827 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 2h ago 2h 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T04:10:46.158 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T04:10:46.159 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T04:10:46.159 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T04:10:46.159 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:10:46.160 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T04:10:46.160 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T04:10:46.160 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:10:46.160 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:10:46.160 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T04:10:46.161 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T04:10:46.161 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:10:46.161 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T04:10:46.161 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T04:10:46.161 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T04:10:46.162 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:10:46.162 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T04:10:46.162 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T04:10:46.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:46 smithi154 conmon[34553]: audit 2021-07-31T04:10:46.159626+0000 mon.c (mon.1) 253 : audit [DBG] 2021-07-31T04:10:46.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:46 smithi154 conmon[34553]: from='client.? 172.21.15.45:0/4292742276' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:10:46.524 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:46 smithi045 conmon[33655]: audit 2021-07-31T04:10:46.159626+0000 mon.c (mon.1) 253 : audit [DBG] 2021-07-31T04:10:46.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:46 smithi045 conmon[33655]: from='client.? 172.21.15.45:0/4292742276' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:10:46.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:46 smithi045 conmon[39718]: audit 2021-07-31T04:10 2021-07-31T04:10:46.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:46 smithi045 conmon[39718]: :46.159626+0000 mon.c (mon.1) 253 : audit [DBG] from='client.? 172.21.15.45:0/4292742276' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:10:47.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:47 smithi154 conmon[34553]: audit 2021-07-31T04:10:45.534211+0000 mgr.x (mgr.24542) 6154 : audit [DBG] from='client.21054 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:10:47.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:47 smithi154 conmon[34553]: audit 2021-07-31T04:10:45.816212+0000 mgr.x (mgr.24542) 6155 : audit [DBG] from='client.30997 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:10:47.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:47 smithi154 conmon[34553]: cluster 2021-07-31T04:10:45.850311+0000 mgr.x (mgr.24542) 6156 : cluster 2021-07-31T04:10:47.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:47 smithi154 conmon[34553]: [DBG] pgmap v6314: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:47.524 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:47 smithi045 conmon[33655]: audit 2021-07-31T04:10:45.534211 2021-07-31T04:10:47.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:47 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 6154 : audit [DBG] from='client.21054 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:10:47.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:47 smithi045 conmon[33655]: audit 2021-07-31T04:10:45.816212+0000 mgr.x (mgr. 2021-07-31T04:10:47.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:47 smithi045 conmon[33655]: 24542) 6155 : audit [DBG] from='client.30997 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:10:47.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:47 smithi045 conmon[33655]: cluster 2021-07-31T04:10: 2021-07-31T04:10:47.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:47 smithi045 conmon[33655]: 45.850311+0000 mgr.x (mgr.24542) 6156 : cluster [DBG] pgmap v6314: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:47.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:47 smithi045 conmon[39718]: audit 2021-07-31T04:10:45.534211+0000 mgr.x (mgr.24542) 6154 : audit 2021-07-31T04:10:47.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:47 smithi045 conmon[39718]: [DBG] from='client.21054 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:10:47.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:47 smithi045 conmon[39718]: audit 2021-07-31T04:10:45.816212+0000 mgr.x (mgr.24542) 6155 : audit [DBG] 2021-07-31T04:10:47.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:47 smithi045 conmon[39718]: from='client.30997 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:10:47.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:47 smithi045 conmon[39718]: cluster 2021-07-31T04:10:45.850311+0000 mgr.x (mgr 2021-07-31T04:10:47.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:47 smithi045 conmon[39718]: .24542) 6156 : cluster [DBG] pgmap v6314: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:49.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:49 smithi154 conmon[34553]: cluster 2021-07-31T04:10:47.850846+0000 mgr.x (mgr.24542) 6157 : cluster [DBG] pgmap v6315: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:49.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:49 smithi045 conmon[33655]: cluster 2021-07-31T04:10:47.850846+0000 mgr.x 2021-07-31T04:10:49.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:49 smithi045 conmon[33655]: (mgr.24542) 6157 : cluster [DBG] pgmap v6315: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:49.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:49 smithi045 conmon[39718]: cluster 2021-07-31T04:10:47.850846+0000 mgr.x (mgr 2021-07-31T04:10:49.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:49 smithi045 conmon[39718]: .24542) 6157 : cluster [DBG] pgmap v6315: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:51.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:51 smithi154 conmon[34553]: cluster 2021-07-31T04:10:49.851348+0000 mgr.x (mgr.24542) 6158 : cluster [DBG] pgmap v6316: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:51.524 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:51 smithi045 conmon[39718]: cluster 2021-07-31T04:10:49.851348+0000 mgr.x (mgr.24542) 6158 : cluster [DBG] pgmap v6316: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:51.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:51 smithi045 conmon[33655]: cluster 2021-07-31T04:10:49.851348+0000 mgr.x (mgr.24542) 6158 : cluster [DBG] pgmap v6316: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:53.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:53 smithi154 conmon[34553]: cluster 2021-07-31T04:10:51.851799+0000 mgr.x (mgr.24542) 6159 : cluster [DBG] pgmap v6317: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:53.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:53 smithi045 conmon[33655]: cluster 2021-07-31T04:10:51.851799+0000 mgr.x (mgr.24542) 6159 : 2021-07-31T04:10:53.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:53 smithi045 conmon[33655]: cluster [DBG] pgmap v6317: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:53.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:53 smithi045 conmon[39718]: cluster 2021-07-31T04:10:51.851799+0000 mgr.x 2021-07-31T04:10:53.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:53 smithi045 conmon[39718]: (mgr.24542) 6159 : cluster [DBG] pgmap v6317: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:55.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:55 smithi154 conmon[34553]: cluster 2021-07-31T04:10:53.852311+0000 mgr.x (mgr.24542) 6160 : cluster [DBG] pgmap v6318: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:55.524 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:55 smithi045 conmon[39718]: cluster 2021-07-31T04:10:53.852311+0000 mgr.x ( 2021-07-31T04:10:55.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:55 smithi045 conmon[39718]: mgr.24542) 6160 : cluster [DBG] pgmap v6318: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:55.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:55 smithi045 conmon[33655]: cluster 2021-07-31T04:10:53.852311+0000 mgr.x (mgr.24542) 2021-07-31T04:10:55.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:55 smithi045 conmon[33655]: 6160 : cluster [DBG] pgmap v6318: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:57.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:57 smithi154 conmon[34553]: cluster 2021-07-31T04:10:55.852796+0000 mgr.x (mgr.24542) 6161 : cluster [DBG] pgmap v6319: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:57.524 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:57 smithi045 conmon[33655]: cluster 2021-07-31T04:10:55.852796 2021-07-31T04:10:57.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:57 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 6161 : cluster [DBG] pgmap v6319: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:57.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:57 smithi045 conmon[39718]: cluster 2021-07-31T04:10:55.852796+0000 2021-07-31T04:10:57.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:57 smithi045 conmon[39718]: mgr.x (mgr.24542) 6161 : cluster [DBG] pgmap v6319: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:59.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:59 smithi154 conmon[34553]: cluster 2021-07-31T04:10:57.853350+0000 mgr.x (mgr.24542) 6162 : cluster [DBG] pgmap v6320: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:59.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:10:59 smithi154 conmon[34553]: 2021-07-31T04:10:59.524 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:10:59 smithi045 conmon[33655]: cluster 2021-07-31T04:10:57.853350+0000 mgr.x (mgr.24542) 6162 : cluster [DBG] pgmap v6320: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:10:59.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:59 smithi045 conmon[39718]: cluster 2021-07-31T04:10:57.853350+0000 mgr.x (mgr. 2021-07-31T04:10:59.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:10:59 smithi045 conmon[39718]: 24542) 6162 : cluster [DBG] pgmap v6320: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:01.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:01 smithi154 conmon[34553]: cluster 2021-07-31T04:10:59.854003+0000 mgr.x (mgr.24542) 6163 : cluster [DBG] pgmap v6321: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:01.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:01 smithi045 conmon[33655]: cluster 2021-07-31T04:10:59.854003+0000 mgr.x (mgr.24542) 6163 : cluster [DBG] pgmap v6321: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:01.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:01 smithi045 conmon[39718]: cluster 2021-07-31T04:10:59.854003+0000 mgr.x (mgr 2021-07-31T04:11:01.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:01 smithi045 conmon[39718]: .24542) 6163 : cluster [DBG] pgmap v6321: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:03.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:03 smithi154 conmon[34553]: cluster 2021-07-31T04:11:01.854491+0000 mgr.x (mgr.24542) 6164 : cluster 2021-07-31T04:11:03.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:03 smithi154 conmon[34553]: [DBG] pgmap v6322: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:03.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:03 smithi045 conmon[33655]: cluster 2021-07-31T04:11:01.854491+0000 mgr.x (mgr.24542) 6164 : cluster [DBG] pgmap v6322: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:03.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:03 smithi045 conmon[39718]: cluster 2021-07-31T04:11:01.854491+0000 mgr.x (mgr.24542 2021-07-31T04:11:03.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:03 smithi045 conmon[39718]: ) 6164 : cluster [DBG] pgmap v6322: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:05.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:05 smithi045 conmon[33655]: cluster 2021-07-31T04:11:03.854832+0000 mgr.x (mgr.24542) 6165 : cluster 2021-07-31T04:11:05.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:05 smithi045 conmon[33655]: [DBG] pgmap v6323: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:05.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:05 smithi045 conmon[39718]: cluster 2021-07-31T04:11:03.854832+0000 mgr.x (mgr.24542 2021-07-31T04:11:05.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:05 smithi045 conmon[39718]: ) 6165 : cluster [DBG] pgmap v6323: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:05.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:05 smithi154 conmon[34553]: cluster 2021-07-31T04:11:03.854832+0000 mgr.x (mgr.24542) 6165 : cluster [DBG] 2021-07-31T04:11:05.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:05 smithi154 conmon[34553]: pgmap v6323: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:07.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:07 smithi045 conmon[39718]: cluster 2021-07-31T04:11:05.855246+0000 mgr.x (mgr.24542 2021-07-31T04:11:07.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:07 smithi045 conmon[39718]: ) 6166 : cluster [DBG] pgmap v6324: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:07.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:07 smithi045 conmon[33655]: cluster 2021-07-31T04:11:05.855246+0000 mgr.x (mgr.24542) 6166 : cluster [DBG] pgmap v6324: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:07.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:07 smithi154 conmon[34553]: cluster 2021-07-31T04:11:05.855246+0000 mgr.x (mgr.24542) 6166 : cluster [DBG] 2021-07-31T04:11:07.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:07 smithi154 conmon[34553]: pgmap v6324: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:09.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:09 smithi045 conmon[33655]: cluster 2021-07-31T04:11:07.855709+0000 mgr.x (mgr.24542) 6167 : cluster [DBG] pgmap v6325: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:09.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:09 smithi045 conmon[39718]: cluster 2021-07-31T04:11:07.855709+0000 mgr.x (mgr.24542 2021-07-31T04:11:09.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:09 smithi045 conmon[39718]: ) 6167 : cluster [DBG] pgmap v6325: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:09.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:09 smithi154 conmon[34553]: cluster 2021-07-31T04:11:07.855709+0000 mgr.x (mgr.24542) 6167 : cluster [DBG] 2021-07-31T04:11:09.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:09 smithi154 conmon[34553]: pgmap v6325: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:11.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:11 smithi045 conmon[39718]: cluster 2021-07-31T04:11:09.856176+0000 mgr.x (mgr.24542 2021-07-31T04:11:11.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:11 smithi045 conmon[39718]: ) 6168 : cluster [DBG] pgmap v6326: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:11.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:11 smithi045 conmon[33655]: cluster 2021-07-31T04:11:09.856176+0000 mgr.x 2021-07-31T04:11:11.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:11 smithi045 conmon[33655]: (mgr.24542) 6168 : cluster [DBG] pgmap v6326: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:11.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:11 smithi154 conmon[34553]: cluster 2021-07-31T04:11:09.856176+0000 mgr.x (mgr.24542) 6168 : cluster [DBG] pgmap v6326: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:11.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:11 smithi154 conmon[34553]: 2021-07-31T04:11:13.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:13 smithi045 conmon[33655]: cluster 2021-07-31T04:11:11. 2021-07-31T04:11:13.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:13 smithi045 conmon[33655]: 856690+0000 mgr.x (mgr.24542) 6169 : cluster [DBG] pgmap v6327: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:13.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:13 smithi045 conmon[39718]: cluster 2021-07-31T04:11:11.856690+0000 mgr.x (mgr 2021-07-31T04:11:13.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:13 smithi045 conmon[39718]: .24542) 6169 : cluster [DBG] pgmap v6327: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:13.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:13 smithi154 conmon[34553]: cluster 2021-07-31T04:11:11.856690+0000 mgr.x (mgr.24542) 6169 : cluster [DBG] pgmap v6327: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:13.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:13 smithi154 conmon[34553]: 2021-07-31T04:11:15.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:15 smithi045 conmon[39718]: cluster 2021-07-31T04:11:13.857242+0000 mgr.x (mgr. 2021-07-31T04:11:15.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:15 smithi045 conmon[39718]: 24542) 6170 : cluster [DBG] pgmap v6328: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:15.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:15 smithi045 conmon[33655]: cluster 2021-07-31T04:11:13.857242+0000 mgr.x (mgr.24542) 6170 : cluster [DBG] pgmap v6328: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:15.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:15 smithi154 conmon[34553]: cluster 2021-07-31T04:11:13.857242+0000 mgr.x (mgr.24542) 6170 : cluster [DBG] pgmap v6328: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:16.473 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T04:11:16.748 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T04:11:16.749 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 2h ago 2h - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T04:11:16.749 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 2h ago 2h 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T04:11:16.749 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 2h ago 2h 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T04:11:16.749 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 2h ago 2h - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T04:11:16.750 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T04:11:16.750 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 2h ago 2h 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T04:11:16.750 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T04:11:16.750 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 2h ago 2h - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T04:11:16.751 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 2h ago 2h 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T04:11:16.751 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T04:11:16.751 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T04:11:16.751 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T04:11:16.751 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T04:11:16.752 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 2h ago 2h 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T04:11:16.752 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 2h ago 2h 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T04:11:16.752 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 2h ago 2h 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T04:11:16.752 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 2h ago 2h 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T04:11:16.752 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 2h ago 2h 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T04:11:17.084 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T04:11:17.118 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T04:11:17.118 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T04:11:17.118 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:11:17.119 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T04:11:17.119 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T04:11:17.119 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:11:17.119 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:11:17.119 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T04:11:17.120 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T04:11:17.120 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:11:17.120 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T04:11:17.120 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T04:11:17.120 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T04:11:17.121 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:11:17.121 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T04:11:17.121 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T04:11:17.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:17 smithi154 conmon[34553]: cluster 2021-07-31T04:11:15.857715+0000 mgr.x (mgr.24542) 6171 : cluster [DBG] pgmap v6329: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:17.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:17 smithi154 conmon[34553]: audit 2021-07-31T04:11:17.085669+0000 mon.c (mon.1) 254 : audit [DBG] from='client.? 172.21.15.45:0/1099791212' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:11:17.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:17 smithi045 conmon[39718]: cluster 2021-07-31T04:11 2021-07-31T04:11:17.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:17 smithi045 conmon[39718]: :15.857715+0000 mgr.x (mgr.24542) 6171 : cluster [DBG] pgmap v6329: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:17.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:17 smithi045 conmon[39718]: audit 2021-07-31T04:11:17.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:17 smithi045 conmon[39718]: 2021- 2021-07-31T04:11:17.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:17 smithi045 conmon[39718]: 07-31T04:11:17 2021-07-31T04:11:17.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:17 smithi045 conmon[39718]: .085669+0000 mon.c (mon 2021-07-31T04:11:17.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:17 smithi045 conmon[39718]: .1) 254 : audit [DBG] 2021-07-31T04:11:17.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:17 smithi045 conmon[39718]: from='client.? 172.21.15.45:0/1099791212' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:11:17.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:17 smithi045 conmon[33655]: cluster 2021-07-31T04:11:15.857715+0000 mgr.x ( 2021-07-31T04:11:17.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:17 smithi045 conmon[33655]: mgr.24542) 6171 : cluster [DBG] pgmap v6329: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:17.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:17 smithi045 conmon[33655]: audit 2021-07-31T04:11:17.085669+0000 2021-07-31T04:11:17.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:17 smithi045 conmon[33655]: mon.c (mon.1) 254 : audit [DBG] from='client.? 172.21.15.45:0/1099791212' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:11:18.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:18 smithi154 conmon[34553]: audit 2021-07-31T04:11:16.462207+0000 mgr.x (mgr.24542) 6172 : audit [DBG] from='client.21072 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:11:18.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:18 smithi154 conmon[34553]: audit 2021-07-31T04:11:16.741379+0000 mgr.x (mgr.24542) 6173 : audit [DBG] from='client.21078 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:11:18.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:18 smithi045 conmon[39718]: audit 2021-07-31T04:11:16.462207+0000 mgr.x (mgr 2021-07-31T04:11:18.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:18 smithi045 conmon[39718]: .24542) 6172 : audit [DBG] from='client.21072 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:11:18.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:18 smithi045 conmon[39718]: audit 2021-07-31T04:11:16.741379+0000 mgr.x (mgr.24542 2021-07-31T04:11:18.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:18 smithi045 conmon[39718]: ) 6173 : audit [DBG] from='client.21078 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:11:18.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:18 smithi045 conmon[33655]: audit 2021-07-31T04:11:16.462207+0000 mgr.x 2021-07-31T04:11:18.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:18 smithi045 conmon[33655]: (mgr.24542) 6172 : audit [DBG] from='client.21072 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:11:18.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:18 smithi045 conmon[33655]: audit 2021-07-31T04:11:16.741379+0000 mgr.x (mgr.24542) 6173 : audit [DBG] from='client.21078 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:11:19.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:19 smithi154 conmon[34553]: cluster 2021-07-31T04:11:17.858097+0000 mgr.x (mgr.24542) 6174 : cluster [DBG] pgmap v6330: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:19.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:19 smithi045 conmon[39718]: cluster 2021-07-31T04:11:17.858097+0000 mgr.x 2021-07-31T04:11:19.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:19 smithi045 conmon[39718]: (mgr.24542) 6174 : cluster [DBG] pgmap v6330: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:19.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:19 smithi045 conmon[33655]: cluster 2021-07-31T04:11:17.858097+0000 mgr.x (mgr.24542) 6174 : 2021-07-31T04:11:19.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:19 smithi045 conmon[33655]: cluster [DBG] pgmap v6330: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:21.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:21 smithi154 conmon[34553]: cluster 2021-07-31T04:11:19.858637+0000 mgr.x (mgr.24542) 6175 : cluster [DBG] pgmap v6331: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:21.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:21 smithi045 conmon[39718]: cluster 2021-07-31T04:11:19.858637+0000 mgr.x (mgr.24542) 6175 : cluster [DBG] pgmap v6331: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:21.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:21 smithi045 conmon[33655]: cluster 2021-07-31T04:11:19.858637+0000 mgr.x (mgr.24542) 6175 : cluster 2021-07-31T04:11:21.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:21 smithi045 conmon[33655]: [DBG] pgmap v6331: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:23.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:23 smithi154 conmon[34553]: cluster 2021-07-31T04:11:21.859175+0000 mgr.x (mgr.24542) 6176 : cluster [DBG] pgmap v6332: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:23.774 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:23 smithi045 conmon[33655]: cluster 2021-07-31T04:11:21.859175+0000 mgr.x (mgr.24542) 2021-07-31T04:11:23.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:23 smithi045 conmon[33655]: 6176 : cluster [DBG] pgmap v6332: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:23.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:23 smithi045 conmon[39718]: cluster 2021-07-31T04:11:21.859175+0000 mgr.x ( 2021-07-31T04:11:23.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:23 smithi045 conmon[39718]: mgr.24542) 6176 : cluster [DBG] pgmap v6332: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:25.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:25 smithi154 conmon[34553]: cluster 2021-07-31T04:11:23.859545+0000 mgr.x (mgr.24542) 6177 : cluster [DBG] pgmap v6333: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:25.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:25 smithi045 conmon[39718]: cluster 2021-07-31T04:11:23.859545+0000 mgr.x (mgr 2021-07-31T04:11:25.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:25 smithi045 conmon[39718]: .24542) 6177 : cluster [DBG] pgmap v6333: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:25.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:25 smithi045 conmon[33655]: cluster 2021-07-31T04:11:23.859545+0000 mgr.x (mgr.24542) 6177 : cluster 2021-07-31T04:11:25.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:25 smithi045 conmon[33655]: [DBG] pgmap v6333: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:27.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:27 smithi154 conmon[34553]: cluster 2021-07-31T04:11:25.860072+0000 mgr.x (mgr.24542) 6178 : cluster [DBG] pgmap v6334: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:27.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:27 smithi045 conmon[33655]: cluster 2021-07-31T04:11:25.860072+0000 mgr.x (mgr.24542) 6178 : cluster [DBG] pgmap v6334: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:27.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:27 smithi045 conmon[39718]: cluster 2021-07-31T04:11:25.860072+0000 mgr.x ( 2021-07-31T04:11:27.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:27 smithi045 conmon[39718]: mgr.24542) 6178 : cluster [DBG] pgmap v6334: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:29.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:29 smithi154 conmon[34553]: cluster 2021-07-31T04:11:27.860578+0000 mgr.x (mgr.24542) 6179 : cluster [DBG] pgmap v6335: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:29.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:29 smithi154 conmon[34553]: audit 2021-07-31T04:11:28.504690+0000 mon.b (mon.2) 795 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:11:29.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:29 smithi154 conmon[34553]: audit 2021-07-31T04:11:28.504991+0000 mon.b (mon.2 2021-07-31T04:11:29.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:29 smithi154 conmon[34553]: ) 796 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:11:29.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:29 smithi154 conmon[34553]: audit 2021 2021-07-31T04:11:29.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:29 smithi154 conmon[34553]: -07-31T04:11:28. 2021-07-31T04:11:29.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:29 smithi154 conmon[34553]: 505708+0000 mon.a (mon.0) 3437 : 2021-07-31T04:11:29.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:29 smithi154 conmon[34553]: audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:11:29.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:29 smithi154 conmon[34553]: audit 2021-07 2021-07-31T04:11:29.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:29 smithi154 conmon[34553]: -31T04:11:28.506018+0000 mon.a 2021-07-31T04:11:29.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:29 smithi154 conmon[34553]: (mon.0) 3438 : audit [INF] 2021-07-31T04:11:29.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:29 smithi154 conmon[34553]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:11:29.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:29 smithi045 conmon[33655]: cluster 2021-07-31T04:11:27.860578+0000 mgr.x (mgr.24542) 6179 2021-07-31T04:11:29.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:29 smithi045 conmon[33655]: : cluster [DBG] pgmap v6335: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:29.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:29 smithi045 conmon[33655]: audit 2021-07-31T04:11:28.504690+0000 mon.b (mon.2) 795 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:11:29.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:29 smithi045 conmon[33655]: audit 2021-07-31T04:11:28.504991+0000 mon.b (mon.2) 2021-07-31T04:11:29.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:29 smithi045 conmon[33655]: 796 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:11:29.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:29 smithi045 conmon[33655]: audit 2021-07-31T04:11:28.505708+0000 mon.a (mon.0 2021-07-31T04:11:29.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:29 smithi045 conmon[33655]: ) 3437 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:11:29.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:29 smithi045 conmon[33655]: audit 2021-07-31T04:11:29.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:29 smithi045 conmon[33655]: 2021-07-31T 2021-07-31T04:11:29.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:29 smithi045 conmon[33655]: 04:11:28.506018+0000 mon.a 2021-07-31T04:11:29.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:29 smithi045 conmon[33655]: (mon.0) 3438 : audit 2021-07-31T04:11:29.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:29 smithi045 conmon[33655]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:11:29.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:29 smithi045 conmon[39718]: cluster 2021-07-31T04:11:27.860578+0000 mgr.x (mgr.24542) 6179 : cluster [DBG] pgmap v6335: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:29.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:29 smithi045 conmon[39718]: audit 2021-07-31T04:11:29.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:29 smithi045 conmon[39718]: 2021-07-31T04:11 2021-07-31T04:11:29.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:29 smithi045 conmon[39718]: :28.504690+0000 mon.b ( 2021-07-31T04:11:29.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:29 smithi045 conmon[39718]: mon.2) 795 : audit 2021-07-31T04:11:29.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:29 smithi045 conmon[39718]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:11:29.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:29 smithi045 conmon[39718]: audit 2021- 2021-07-31T04:11:29.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:29 smithi045 conmon[39718]: 07-31T04:11:28.504991+0000 2021-07-31T04:11:29.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:29 smithi045 conmon[39718]: mon.b (mon.2) 796 2021-07-31T04:11:29.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:29 smithi045 conmon[39718]: : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:11:29.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:29 smithi045 conmon[39718]: audit 2021 2021-07-31T04:11:29.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:29 smithi045 conmon[39718]: -07-31T04:11:28. 2021-07-31T04:11:29.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:29 smithi045 conmon[39718]: 505708+0000 mon.a (mon. 2021-07-31T04:11:29.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:29 smithi045 conmon[39718]: 0) 3437 : audit [INF] 2021-07-31T04:11:29.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:29 smithi045 conmon[39718]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:11:29.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:29 smithi045 conmon[39718]: audit 2021- 2021-07-31T04:11:29.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:29 smithi045 conmon[39718]: 07-31T04:11:28. 2021-07-31T04:11:29.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:29 smithi045 conmon[39718]: 506018+0000 mon.a (mon.0 2021-07-31T04:11:29.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:29 smithi045 conmon[39718]: ) 3438 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:11:29.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:29 smithi045 conmon[39718]: 2021-07-31T04:11:31.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:31 smithi154 conmon[34553]: cluster 2021-07-31T04:11:29.861169+0000 mgr.x (mgr.24542) 6180 : cluster [DBG] pgmap v6336: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:31.774 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:31 smithi045 conmon[39718]: cluster 2021-07-31T04:11:29.861169+0000 2021-07-31T04:11:31.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:31 smithi045 conmon[39718]: mgr.x (mgr.24542) 6180 : cluster [DBG] pgmap v6336: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:31.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:31 smithi045 conmon[33655]: cluster 2021-07-31T04:11:29.861169+0000 mgr.x (mgr. 2021-07-31T04:11:31.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:31 smithi045 conmon[33655]: 24542) 6180 : cluster [DBG] pgmap v6336: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:33.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:33 smithi154 conmon[34553]: cluster 2021-07-31T04:11:31.861698+0000 mgr.x (mgr.24542) 6181 : cluster [DBG] pgmap v6337: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:33.774 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:33 smithi045 conmon[39718]: cluster 2021-07-31T04:11:31.861698+0000 mgr.x 2021-07-31T04:11:33.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:33 smithi045 conmon[39718]: (mgr.24542) 6181 : cluster [DBG] pgmap v6337: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:33.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:33 smithi045 conmon[33655]: cluster 2021-07-31T04:11:31.861698 2021-07-31T04:11:33.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:33 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 6181 : cluster [DBG] pgmap v6337: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:35.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:35 smithi154 conmon[34553]: cluster 2021-07-31T04:11:33.862144+0000 mgr.x (mgr.24542) 6182 : cluster [DBG] pgmap v6338: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:35.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:35 smithi045 conmon[39718]: cluster 2021-07-31T04:11:33.862144+0000 2021-07-31T04:11:35.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:35 smithi045 conmon[39718]: mgr.x (mgr.24542) 6182 : cluster [DBG] pgmap v6338: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:35.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:35 smithi045 conmon[33655]: cluster 2021-07-31T04:11:33.862144+0000 mgr.x (mgr.24542) 6182 : cluster 2021-07-31T04:11:35.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:35 smithi045 conmon[33655]: [DBG] pgmap v6338: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:37.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[34553]: cluster 2021-07-31T04:11:35.862529+0000 mgr.x (mgr.24542) 6183 : cluster [DBG] pgmap v6339: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:37.753 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: debug 2021-07-31T04:11:37.422+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:11:37.753 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:11:37.753 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:11:37.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: yield 2021-07-31T04:11:37.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:11:37.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:11:37.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:11:37.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:11:37.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:11:37.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:11:37.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:11:37.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:11:37.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:11:37.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: task.result() 2021-07-31T04:11:37.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:11:37.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: debug 2021-07-31T04:11:37.434+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:11:37.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:11:37.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:11:37.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: yield 2021-07-31T04:11:37.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:11:37.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:11:37.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:11:37.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:11:37.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:11:37.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:11:37.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:11:37.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:11:37.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:11:37.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: task.result() 2021-07-31T04:11:37.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:11:37.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: debug 2021-07-31T04:11:37.460+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:11:37.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:11:37.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:11:37.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: yield 2021-07-31T04:11:37.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:11:37.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:11:37.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:11:37.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:11:37.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:11:37.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:11:37.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:11:37.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:11:37.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:11:37.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: task.result() 2021-07-31T04:11:37.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:11:37.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: debug 2021-07-31T04:11:37.460+0000 7fcb97969700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:11:37.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:11:37.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:11:37.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: yield 2021-07-31T04:11:37.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:11:37.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:11:37.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:11:37.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:11:37.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:11:37.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:11:37.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:11:37.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:11:37.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:11:37.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: task.result() 2021-07-31T04:11:37.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:11:37.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: 2021-07-31T04:11:37.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T04:11:37.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: 2021-07-31T04:11:37.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:11:37.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:11:37.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: return f(*arg) 2021-07-31T04:11:37.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:11:37.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:11:37.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:11:37.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:11:37.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:11:37.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:11:37.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:11:37.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T04:11:37.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:11:37.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: raise self._exception 2021-07-31T04:11:37.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:11:37.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:11:37.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:11:37.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T04:11:37.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:11:37.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: raise 2021-07-31T04:11:37.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:11:37.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T04:11:37.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:11:37.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:11:37.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:11:37.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: debug 2021-07-31T04:11:37.469+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:11:37.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:11:37.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:11:37.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: yield 2021-07-31T04:11:37.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:11:37.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:11:37.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:11:37.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:11:37.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:11:37.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:11:37.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:11:37.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:11:37.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:11:37.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: task.result() 2021-07-31T04:11:37.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:11:37.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: debug 2021-07-31T04:11:37.469+0000 7fcb9816a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:11:37.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:11:37.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:11:37.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: yield 2021-07-31T04:11:37.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:11:37.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:11:37.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:11:37.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:11:37.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:11:37.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:11:37.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:11:37.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:11:37.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:11:37.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: task.result() 2021-07-31T04:11:37.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:11:37.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: 2021-07-31T04:11:37.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T04:11:37.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: 2021-07-31T04:11:37.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:11:37.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:11:37.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: return f(*arg) 2021-07-31T04:11:37.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:11:37.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:11:37.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:11:37.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:11:37.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:11:37.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:11:37.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:11:37.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T04:11:37.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:11:37.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: raise self._exception 2021-07-31T04:11:37.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:11:37.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:11:37.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:11:37.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T04:11:37.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:11:37.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: raise 2021-07-31T04:11:37.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:11:37.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T04:11:37.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:11:37.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:11:37.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:11:37 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:11:37.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:37 smithi045 conmon[39718]: cluster 2021-07-31T04:11:35.862529+0000 2021-07-31T04:11:37.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:37 smithi045 conmon[39718]: mgr.x (mgr.24542) 6183 : cluster [DBG] pgmap v6339: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:37.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:37 smithi045 conmon[33655]: cluster 2021-07-31T04:11:35.862529 2021-07-31T04:11:37.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:37 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 6183 : cluster [DBG] pgmap v6339: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:38.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:38 smithi154 conmon[34553]: audit 2021-07-31T04:11:37.377446+0000 mon.b (mon.2) 797 : audit 2021-07-31T04:11:38.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:38 smithi154 conmon[34553]: [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T04:11:38.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:38 smithi154 conmon[34553]: audit 2021-07-31T04:11:37.378799+0000 mon.b (mon.2) 798 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T04:11:38.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:38 smithi154 conmon[34553]: audit 2021-07-31T04:11:37.379629+0000 mon.b (mon. 2021-07-31T04:11:38.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:38 smithi154 conmon[34553]: 2) 799 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T04:11:38.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:38 smithi154 conmon[34553]: cephadm 2021 2021-07-31T04:11:38.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:38 smithi154 conmon[34553]: -07-31T04:11:37.380373+0000 mgr.x (mgr.24542) 6184 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T04:11:38.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:38 smithi154 conmon[34553]: cephadm 2021-07-31T04:11:38.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:38 smithi154 conmon[34553]: 2021-07-31T04:11:37.423352+0000 mgr.x 2021-07-31T04:11:38.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:38 smithi154 conmon[34553]: (mgr.24542) 6185 : cephadm [ERR] 2021-07-31T04:11:38.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:38 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:11:38.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:11:38.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:38 smithi154 conmon[34553]: yield 2021-07-31T04:11:38.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:38 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:11:38.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:38 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:11:38.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:11:38.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:38 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:11:38.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:11:38.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:38 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:11:38.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:11:38.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:38 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:11:38.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:38 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:11:38.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:38 smithi154 conmon[34553]: task.result() 2021-07-31T04:11:38.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:38 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:11:38.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:38 smithi154 conmon[34553]: cephadm 2021-07-31T04:11:37.423797+0000 mgr.x (mgr.24542) 6186 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T04:11:38.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:38 smithi045 conmon[33655]: audit 2021-07-31T04:11:37.377446+0000 mon.b (mon.2) 797 2021-07-31T04:11:38.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:38 smithi045 conmon[33655]: : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T04:11:38.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:38 smithi045 conmon[33655]: audit 2021-07-31T04:11:37.378799+0000 mon.b (mon.2) 798 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T04:11:38.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:38 smithi045 conmon[33655]: audit 2021 2021-07-31T04:11:38.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:38 smithi045 conmon[33655]: -07-31T04:11:37.379629 2021-07-31T04:11:38.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:38 smithi045 conmon[33655]: +0000 mon.b (mon.2) 799 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T04:11:38.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:38 smithi045 conmon[33655]: cephadm 2021-07 2021-07-31T04:11:38.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:38 smithi045 conmon[33655]: -31T04:11:37.380373+0000 mgr.x (mgr 2021-07-31T04:11:38.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:38 smithi045 conmon[33655]: .24542) 6184 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T04:11:38.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:38 smithi045 conmon[33655]: cephadm 2021-07-31 2021-07-31T04:11:38.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:38 smithi045 conmon[33655]: T04:11:37.423352+0000 mgr.x ( 2021-07-31T04:11:38.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:38 smithi045 conmon[33655]: mgr.24542) 6185 : cephadm [ERR] 2021-07-31T04:11:38.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:38 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:11:38.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:11:38.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:38 smithi045 conmon[33655]: yield 2021-07-31T04:11:38.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:38 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:11:38.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:38 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:11:38.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:11:38.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:38 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:11:38.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:11:38.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:38 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:11:38.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:11:38.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:38 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:11:38.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:38 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:11:38.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:38 smithi045 conmon[33655]: task.result() 2021-07-31T04:11:38.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:38 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:11:38.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:38 smithi045 conmon[33655]: 2021-07-31T04:11:38.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:38 smithi045 conmon[33655]: cephadm 2021-07-31T04:11:37.423797+0000 mgr.x (mgr.24542) 6186 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T04:11:38.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:38 smithi045 conmon[39718]: audit 2021-07-31T04:11:37.377446+0000 2021-07-31T04:11:38.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:38 smithi045 conmon[39718]: mon.b (mon.2) 797 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T04:11:38.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:38 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T04:11:38.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:38 smithi045 conmon[39718]: 04:11:37.378799+0000 mon.b (mon.2) 798 : audit 2021-07-31T04:11:38.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:38 smithi045 conmon[39718]: [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T04:11:38.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:38 smithi045 conmon[39718]: audit 2021-07-31T04: 2021-07-31T04:11:38.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:38 smithi045 conmon[39718]: 11:37.379629+0000 mon.b (mon.2 2021-07-31T04:11:38.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:38 smithi045 conmon[39718]: ) 799 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T04:11:38.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:38 smithi045 conmon[39718]: cephadm 2021- 2021-07-31T04:11:38.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:38 smithi045 conmon[39718]: 07-31T04:11:37.380373 2021-07-31T04:11:38.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:38 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 6184 : cephadm 2021-07-31T04:11:38.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:38 smithi045 conmon[39718]: [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T04:11:38.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:38 smithi045 conmon[39718]: cephadm 2021-07-31T 2021-07-31T04:11:38.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:38 smithi045 conmon[39718]: 04:11:37.423352+0000 mgr.x ( 2021-07-31T04:11:38.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:38 smithi045 conmon[39718]: mgr.24542) 6185 : cephadm [ERR] 2021-07-31T04:11:38.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:38 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:11:38.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:11:38.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:38 smithi045 conmon[39718]: yield 2021-07-31T04:11:38.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:38 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:11:38.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:38 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:11:38.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:11:38.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:38 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:11:38.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:11:38.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:38 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:11:38.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:11:38.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:38 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:11:38.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:38 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:11:38.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:38 smithi045 conmon[39718]: task.result() 2021-07-31T04:11:38.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:38 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:11:38.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:38 smithi045 conmon[39718]: 2021-07-31T04:11:38.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:38 smithi045 conmon[39718]: cephadm 2021-07-31T04:11:37.423797+0000 mgr.x (mgr.24542) 6186 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T04:11:39.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:11:37. 2021-07-31T04:11:39.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: 435345+0000 mgr.x (mgr.24542) 6187 : cephadm [ERR] 2021-07-31T04:11:39.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:11:39.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:11:39.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: yield 2021-07-31T04:11:39.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:11:39.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:11:39.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:11:39.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:11:39.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:11:39.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:11:39.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:11:39.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:11:39.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:11:39.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: task.result() 2021-07-31T04:11:39.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:11:39.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: cephadm 2021-07-31T04: 2021-07-31T04:11:39.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: 11:37.435781+0000 mgr.x (mgr.24542) 6188 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T04:11:39.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:11:37.461177+0000 mgr.x (mgr.24542) 6189 : cephadm [ERR] 2021-07-31T04:11:39.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:11:39.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:11:39.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: yield 2021-07-31T04:11:39.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:11:39.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:11:39.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:11:39.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:11:39.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:11:39.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:11:39.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:11:39.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:11:39.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:11:39.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: task.result() 2021-07-31T04:11:39.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:11:39.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: cephadm 2021-07-31 2021-07-31T04:11:39.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: T04:11:37.461611+0000 mgr.x (mgr.24542) 6190 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:11:39.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:11:39.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:11:39.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: yield 2021-07-31T04:11:39.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:11:39.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:11:39.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:11:39.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:11:39.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:11:39.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:11:39.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:11:39.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:11:39.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:11:39.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: task.result() 2021-07-31T04:11:39.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:11:39.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: 2021-07-31T04:11:39.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T04:11:39.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: 2021-07-31T04:11:39.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:11:39.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:11:39.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: return f(*arg) 2021-07-31T04:11:39.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:11:39.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:11:39.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:11:39.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:11:39.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:11:39.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:11:39.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:11:39.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T04:11:39.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:11:39.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: raise self._exception 2021-07-31T04:11:39.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:11:39.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:11:39.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:11:39.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T04:11:39.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:11:39.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: raise 2021-07-31T04:11:39.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:11:39.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T04:11:39.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:11:39.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:11:39.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:11:39.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:11:37.470422+0000 mgr.x (mgr.24542) 6191 : cephadm [ERR] 2021-07-31T04:11:39.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:11:39.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:11:39.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: yield 2021-07-31T04:11:39.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:11:39.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:11:39.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:11:39.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:11:39.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:11:39.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:11:39.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:11:39.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:11:39.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:11:39.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: task.result() 2021-07-31T04:11:39.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:11:39.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:11:37.470822+0000 mgr.x (mgr.24542) 6192 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:11:39.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:11:39.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:11:39.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: yield 2021-07-31T04:11:39.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:11:39.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:11:39.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:11:39.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:11:39.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:11:39.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:11:39.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:11:39.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:11:39.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:11:39.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: task.result() 2021-07-31T04:11:39.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:11:39.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: 2021-07-31T04:11:39.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T04:11:39.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: 2021-07-31T04:11:39.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:11:39.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:11:39.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: return f(*arg) 2021-07-31T04:11:39.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:11:39.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:11:39.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:11:39.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:11:39.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:11:39.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:11:39.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:11:39.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T04:11:39.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:11:39.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: raise self._exception 2021-07-31T04:11:39.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:11:39.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:11:39.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:11:39.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T04:11:39.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:11:39.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: raise 2021-07-31T04:11:39.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:11:39.782 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T04:11:39.782 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:11:39.782 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:11:39.782 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:11:39.783 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: 2021-07-31T04:11:39.783 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:39 smithi154 conmon[34553]: cluster 2021-07-31T04:11:37.862990+0000 mgr.x (mgr.24542) 6193 : cluster [DBG] pgmap v6340: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:39.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: cephadm 2021-07-31T04:11:37.435345+0000 2021-07-31T04:11:39.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: mgr.x (mgr.24542) 6187 : cephadm [ERR] 2021-07-31T04:11:39.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:11:39.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:11:39.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: yield 2021-07-31T04:11:39.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:11:39.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:11:39.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:11:39.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:11:39.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:11:39.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:11:39.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:11:39.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:11:39.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:11:39.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: task.result() 2021-07-31T04:11:39.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:11:39.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: cephadm 2021-07-31T 2021-07-31T04:11:39.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: 04:11:37.435781+0000 mgr.x (mgr.24542) 6188 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T04:11:39.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: cephadm 2021-07-31T04:11:37.461177+0000 mgr.x (mgr. 2021-07-31T04:11:39.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: 24542) 6189 : cephadm [ERR] 2021-07-31T04:11:39.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:11:39.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:11:39.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: yield 2021-07-31T04:11:39.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:11:39.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:11:39.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:11:39.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:11:39.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:11:39.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:11:39.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:11:39.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:11:39.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:11:39.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: task.result() 2021-07-31T04:11:39.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:11:39.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: cephadm 2021-07-31T04:11:37.461611+0000 mgr.x (mgr.24542) 6190 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:11:39.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:11:39.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:11:39.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: yield 2021-07-31T04:11:39.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:11:39.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:11:39.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:11:39.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:11:39.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:11:39.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:11:39.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:11:39.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:11:39.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:11:39.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: task.result() 2021-07-31T04:11:39.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:11:39.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: 2021-07-31T04:11:39.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T04:11:39.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: 2021-07-31T04:11:39.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:11:39.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:11:39.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: return f(*arg) 2021-07-31T04:11:39.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:11:39.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:11:39.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:11:39.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:11:39.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:11:39.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:11:39.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:11:39.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T04:11:39.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:11:39.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: raise self._exception 2021-07-31T04:11:39.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:11:39.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:11:39.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:11:39.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T04:11:39.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:11:39.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: raise 2021-07-31T04:11:39.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:11:39.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T04:11:39.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:11:39.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:11:39.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:11:39.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: cephadm 2021-07-31T04:11:37.470422+0000 mgr.x (mgr.24542) 6191 : cephadm [ERR] 2021-07-31T04:11:39.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:11:39.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:11:39.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: yield 2021-07-31T04:11:39.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:11:39.801 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: cephadm 2021-07-31T04:11:37.435345+0000 mgr.x (mgr.24542) 6187 : cephadm [ERR] 2021-07-31T04:11:39.802 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:11:39.802 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:11:39.802 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: yield 2021-07-31T04:11:39.802 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:11:39.803 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:11:39.803 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:11:39.803 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:11:39.803 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:11:39.803 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:11:39.804 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:11:39.804 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:11:39.804 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:11:39.804 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: task.result() 2021-07-31T04:11:39.805 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:11:39.805 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: cephadm 2021-07-31T04:11:39.805 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: 2021-07-31T04:11:37.435781+0000 mgr.x (mgr.24542) 6188 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T04:11:39.805 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: cephadm 2021-07-31T04:11:37.461177+0000 mgr.x (mgr.24542) 6189 : cephadm [ERR] 2021-07-31T04:11:39.805 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:11:39.806 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:11:39.806 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: yield 2021-07-31T04:11:39.806 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:11:39.806 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:11:39.807 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:11:39.807 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:11:39.807 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:11:39.807 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:11:39.807 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:11:39.808 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:11:39.808 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:11:39.808 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: task.result() 2021-07-31T04:11:39.808 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:11:39.809 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: cephadm 2021-07-31T04:11:37.461611+0000 mgr.x (mgr.24542) 6190 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:11:39.809 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:11:39.809 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:11:39.809 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: yield 2021-07-31T04:11:39.809 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:11:39.810 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:11:39.810 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:11:39.810 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:11:39.811 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:11:39.811 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:11:39.811 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:11:39.811 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:11:39.812 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:11:39.812 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:11:39.812 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: task.result() 2021-07-31T04:11:39.812 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:11:39.812 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: cephadm 2021-07-31T04:11:37.470822+0000 mgr.x (mgr.24542) 6192 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:11:39.813 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:11:39.813 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:11:39.813 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: yield 2021-07-31T04:11:39.813 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:11:39.814 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:11:39.814 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:11:39.814 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:11:39.814 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:11:39.814 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:11:39.815 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:11:39.815 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:11:39.815 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:11:39.815 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: task.result() 2021-07-31T04:11:39.816 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:11:39.816 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: 2021-07-31T04:11:39.816 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T04:11:39.816 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: 2021-07-31T04:11:39.816 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:11:39.817 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:11:39.817 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: return f(*arg) 2021-07-31T04:11:39.817 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:11:39.817 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:11:39.817 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:11:39.818 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:11:39.818 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:11:39.818 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:11:39.818 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:11:39.819 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T04:11:39.819 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:11:39.819 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: raise self._exception 2021-07-31T04:11:39.819 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:11:39.819 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:11:39.820 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:11:39.820 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T04:11:39.820 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:11:39.820 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: raise 2021-07-31T04:11:39.820 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:11:39.821 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T04:11:39.821 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:11:39.821 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:11:39.821 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:11:39.822 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: cluster 2021-07-31T04:11:39.822 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[39718]: 2021-07-31T04:11:37.862990+0000 mgr.x (mgr.24542) 6193 : cluster [DBG] pgmap v6340: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:39.822 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:11:39.823 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:11:39.823 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:11:39.823 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:11:39.823 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:11:39.824 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:11:39.824 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:11:39.824 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: task.result() 2021-07-31T04:11:39.824 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:11:39.825 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: 2021-07-31T04:11:39.825 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T04:11:39.825 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: 2021-07-31T04:11:39.825 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:11:39.825 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:11:39.826 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: return f(*arg) 2021-07-31T04:11:39.826 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:11:39.826 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:11:39.826 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:11:39.827 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:11:39.827 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:11:39.827 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:11:39.827 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:11:39.827 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T04:11:39.828 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:11:39.828 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: raise self._exception 2021-07-31T04:11:39.828 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:11:39.828 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:11:39.828 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:11:39.829 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T04:11:39.829 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:11:39.829 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: raise 2021-07-31T04:11:39.829 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:11:39.830 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T04:11:39.830 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:11:39.830 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:11:39.830 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:11:39.830 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: cephadm 2021-07-31T04:11:37.470422+0000 mgr.x (mgr.24542) 6191 : cephadm [ERR] 2021-07-31T04:11:39.831 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: 2021-07-31T04:11:39.831 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:11:39.831 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:11:39.831 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: yield 2021-07-31T04:11:39.831 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:11:39.832 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:11:39.832 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:11:39.832 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:11:39.832 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:11:39.833 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:11:39.833 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:11:39.833 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:11:39.833 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:11:39.833 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: task.result() 2021-07-31T04:11:39.834 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:11:39.834 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: cephadm 2021-07-31T04:11:37.470822+0000 mgr.x (mgr.24542) 6192 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:11:39.834 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:11:39.834 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:11:39.834 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: yield 2021-07-31T04:11:39.835 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:11:39.835 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:11:39.835 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:11:39.835 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:11:39.836 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:11:39.836 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:11:39.836 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:11:39.836 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:11:39.836 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:11:39.837 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: task.result() 2021-07-31T04:11:39.837 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:11:39.837 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: 2021-07-31T04:11:39.837 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T04:11:39.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: 2021-07-31T04:11:39.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:11:39.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:11:39.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: return f(*arg) 2021-07-31T04:11:39.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:11:39.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:11:39.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:11:39.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:11:39.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:11:39.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:11:39.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:11:39.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T04:11:39.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:11:39.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: raise self._exception 2021-07-31T04:11:39.841 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:11:39.841 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:11:39.841 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:11:39.841 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T04:11:39.841 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:11:39.842 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: raise 2021-07-31T04:11:39.842 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:11:39.842 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T04:11:39.842 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:11:39.843 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:11:39.843 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:11:39.843 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:39 smithi045 conmon[33655]: cluster 2021-07-31T04:11:37.862990+0000 mgr.x (mgr.24542) 6193 : cluster [DBG] pgmap v6340: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:41.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:41 smithi154 conmon[34553]: cluster 2021-07-31T04:11:39.863575+0000 mgr.x (mgr.24542) 6194 : cluster 2021-07-31T04:11:41.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:41 smithi154 conmon[34553]: [DBG] pgmap v6341: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:41.774 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:41 smithi045 conmon[33655]: cluster 2021-07-31T04:11:39.863575+0000 2021-07-31T04:11:41.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:41 smithi045 conmon[33655]: mgr.x (mgr.24542) 6194 : cluster [DBG] pgmap v6341: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:41.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:41 smithi045 conmon[39718]: cluster 2021-07-31T04:11:39.863575+0000 mgr.x ( 2021-07-31T04:11:41.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:41 smithi045 conmon[39718]: mgr.24542) 6194 : cluster [DBG] pgmap v6341: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:43.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:43 smithi154 conmon[34553]: cluster 2021-07-31T04:11:41.864063+0000 mgr.x (mgr.24542) 2021-07-31T04:11:43.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:43 smithi154 conmon[34553]: 6195 : cluster [DBG] pgmap v6342: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:43.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:43 smithi045 conmon[39718]: cluster 2021-07-31T04:11:41.864063+0000 mgr.x 2021-07-31T04:11:43.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:43 smithi045 conmon[39718]: (mgr.24542) 6195 : cluster [DBG] pgmap v6342: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:43.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:43 smithi045 conmon[33655]: cluster 2021-07-31T04:11:41.864063+0000 mgr.x (mgr.24542) 6195 : cluster [DBG] pgmap v6342: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:45.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:45 smithi154 conmon[34553]: cluster 2021-07-31T04:11:43.864553+0000 mgr.x (mgr.24542) 6196 : cluster [DBG] 2021-07-31T04:11:45.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:45 smithi154 conmon[34553]: pgmap v6343: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:45.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:45 smithi045 conmon[39718]: cluster 2021-07-31T04:11: 2021-07-31T04:11:45.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:45 smithi045 conmon[39718]: 43.864553+0000 mgr.x (mgr.24542) 6196 : cluster [DBG] pgmap v6343: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:45.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:45 smithi045 conmon[33655]: cluster 2021-07-31T04:11:43.864553+0000 mgr.x (mgr.24542) 6196 2021-07-31T04:11:45.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:45 smithi045 conmon[33655]: : cluster [DBG] pgmap v6343: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:47.400 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T04:11:47.679 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T04:11:47.679 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 2h ago 2h - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T04:11:47.679 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 2h ago 2h 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T04:11:47.679 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 2h ago 2h 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T04:11:47.680 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 2h ago 2h - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T04:11:47.680 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T04:11:47.680 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 2h ago 2h 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T04:11:47.680 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T04:11:47.681 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 2h ago 2h - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T04:11:47.681 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 2h ago 2h 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T04:11:47.681 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T04:11:47.681 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T04:11:47.681 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T04:11:47.682 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T04:11:47.682 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 2h ago 2h 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T04:11:47.682 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 2h ago 2h 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T04:11:47.682 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 2h ago 2h 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T04:11:47.682 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 2h ago 2h 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T04:11:47.683 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 2h ago 2h 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T04:11:47.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:47 smithi154 conmon[34553]: cluster 2021-07-31T04:11:45.864992+0000 mgr.x (mgr.24542) 6197 : cluster 2021-07-31T04:11:47.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:47 smithi154 conmon[34553]: [DBG] pgmap v6344: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:47.774 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:47 smithi045 conmon[39718]: cluster 2021-07-31T04:11:45.864992 2021-07-31T04:11:47.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:47 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 6197 : cluster [DBG] pgmap v6344: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:47.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:47 smithi045 conmon[33655]: cluster 2021-07-31T04:11:45 2021-07-31T04:11:47.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:47 smithi045 conmon[33655]: .864992+0000 mgr.x (mgr.24542) 6197 : cluster [DBG] pgmap v6344: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:48.014 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T04:11:48.015 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T04:11:48.015 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T04:11:48.015 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:11:48.015 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T04:11:48.015 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T04:11:48.016 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:11:48.016 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:11:48.016 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T04:11:48.016 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T04:11:48.017 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:11:48.017 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T04:11:48.017 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T04:11:48.017 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T04:11:48.017 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:11:48.018 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T04:11:48.018 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T04:11:48.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:48 smithi154 conmon[34553]: audit 2021-07-31T04:11:47.388742+0000 mgr.x (mgr.24542) 6198 : audit 2021-07-31T04:11:48.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:48 smithi154 conmon[34553]: [DBG] from='client.21090 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:11:48.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:48 smithi154 conmon[34553]: audit 2021-07-31T04:11:48.015500+0000 mon.c (mon.1) 255 : audit [DBG] from='client.? 172.21.15.45:0/192783595' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:11:48.774 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:48 smithi045 conmon[33655]: audit 2021-07-31T04:11:47.388742+0000 mgr.x (mgr.24542) 2021-07-31T04:11:48.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:48 smithi045 conmon[33655]: 6198 : audit [DBG] from='client.21090 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:11:48.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:48 smithi045 conmon[33655]: audit 2021-07-31T04:11:48.015500+0000 mon.c (mon.1) 255 : audit [DBG] from='client.? 172.21.15.45:0/192783595' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:11:48.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:48 smithi045 conmon[39718]: audit 2021-07-31T04:11:47 2021-07-31T04:11:48.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:48 smithi045 conmon[39718]: .388742+0000 mgr.x (mgr.24542) 6198 : audit [DBG] from='client.21090 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:11:48.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:48 smithi045 conmon[39718]: audit 2021 2021-07-31T04:11:48.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:48 smithi045 conmon[39718]: -07-31T04:11: 2021-07-31T04:11:48.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:48 smithi045 conmon[39718]: 48.015500+0000 mon.c (mon.1 2021-07-31T04:11:48.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:48 smithi045 conmon[39718]: ) 255 : audit [DBG] from='client.? 172.21.15.45:0/192783595' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:11:49.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:49 smithi154 conmon[34553]: audit 2021-07-31T04:11:47.671492+0000 mgr.x (mgr.24542) 6199 : audit [DBG] 2021-07-31T04:11:49.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:49 smithi154 conmon[34553]: from='client.31033 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:11:49.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:49 smithi154 conmon[34553]: cluster 2021-07-31T04:11:47.865382+0000 mgr.x (mgr.24542) 6200 : cluster [DBG] pgmap v6345: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:49.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:49 smithi045 conmon[33655]: audit 2021-07-31T04:11:47.671492+0000 mgr.x (mgr.24542) 6199 : audit [DBG] from='client.31033 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:11:49.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:49 smithi045 conmon[33655]: cluster 2021-07-31T04:11:47.865382+0000 mgr.x (mgr.24542) 6200 : cluster [DBG] pgmap v6345: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:49.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:49 smithi045 conmon[39718]: audit 2021-07-31T04:11:47.671492+0000 mgr.x ( 2021-07-31T04:11:49.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:49 smithi045 conmon[39718]: mgr.24542) 6199 : audit [DBG] from='client.31033 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:11:49.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:49 smithi045 conmon[39718]: cluster 2021-07-31T04:11:47.865382+0000 mgr.x (mgr.24542) 6200 2021-07-31T04:11:49.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:49 smithi045 conmon[39718]: : cluster [DBG] pgmap v6345: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:51.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:51 smithi154 conmon[34553]: cluster 2021-07-31T04:11:49.865845+0000 mgr.x (mgr.24542) 6201 : cluster 2021-07-31T04:11:51.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:51 smithi154 conmon[34553]: [DBG] pgmap v6346: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:51.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:51 smithi045 conmon[33655]: cluster 2021-07-31T04:11 2021-07-31T04:11:51.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:51 smithi045 conmon[33655]: :49.865845+0000 mgr.x (mgr.24542) 6201 : cluster [DBG] pgmap v6346: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:51.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:51 smithi045 conmon[39718]: cluster 2021-07-31T04:11:49.865845+0000 mgr.x (mgr 2021-07-31T04:11:51.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:51 smithi045 conmon[39718]: .24542) 6201 : cluster [DBG] pgmap v6346: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:53.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:53 smithi154 conmon[34553]: cluster 2021-07-31T04:11:51.866318+0000 mgr.x (mgr.24542) 6202 : 2021-07-31T04:11:53.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:53 smithi154 conmon[34553]: cluster [DBG] pgmap v6347: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:53.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:53 smithi045 conmon[39718]: cluster 2021-07-31T04:11:51.866318+0000 mgr.x (mgr 2021-07-31T04:11:53.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:53 smithi045 conmon[39718]: .24542) 6202 : cluster [DBG] pgmap v6347: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:53.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:53 smithi045 conmon[33655]: cluster 2021-07-31T04:11:51.866318+0000 mgr.x (mgr.24542) 6202 2021-07-31T04:11:53.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:53 smithi045 conmon[33655]: : cluster [DBG] pgmap v6347: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:55.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:55 smithi154 conmon[34553]: cluster 2021-07-31T04:11:53.866856+0000 mgr.x (mgr.24542) 6203 : cluster [DBG] 2021-07-31T04:11:55.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:55 smithi154 conmon[34553]: pgmap v6348: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:55.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:55 smithi045 conmon[39718]: cluster 2021-07-31T04:11:53.866856+0000 mgr.x ( 2021-07-31T04:11:55.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:55 smithi045 conmon[39718]: mgr.24542) 6203 : cluster [DBG] pgmap v6348: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:55.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:55 smithi045 conmon[33655]: cluster 2021-07-31T04:11:53.866856+0000 mgr.x (mgr.24542) 6203 : cluster [DBG] pgmap v6348: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:57.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:57 smithi154 conmon[34553]: cluster 2021-07-31T04:11:55.867384+0000 mgr.x (mgr.24542) 6204 : cluster 2021-07-31T04:11:57.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:57 smithi154 conmon[34553]: [DBG] pgmap v6349: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:57.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:57 smithi045 conmon[33655]: cluster 2021-07-31T04:11:55.867384+0000 mgr.x (mgr.24542) 6204 : 2021-07-31T04:11:57.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:57 smithi045 conmon[33655]: cluster [DBG] pgmap v6349: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:57.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:57 smithi045 conmon[39718]: cluster 2021-07-31T04:11:55.867384+0000 mgr.x (mgr 2021-07-31T04:11:57.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:57 smithi045 conmon[39718]: .24542) 6204 : cluster [DBG] pgmap v6349: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:59.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:59 smithi154 conmon[34553]: cluster 2021-07-31T04:11:57.867881+0000 mgr.x (mgr.24542) 6205 : cluster [DBG] 2021-07-31T04:11:59.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:11:59 smithi154 conmon[34553]: pgmap v6350: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:59.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:59 smithi045 conmon[33655]: cluster 2021-07-31T04:11:57.867881+0000 mgr.x (mgr.24542) 6205 : 2021-07-31T04:11:59.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:11:59 smithi045 conmon[33655]: cluster [DBG] pgmap v6350: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:11:59.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:59 smithi045 conmon[39718]: cluster 2021-07-31T04:11:57 2021-07-31T04:11:59.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:11:59 smithi045 conmon[39718]: .867881+0000 mgr.x (mgr.24542) 6205 : cluster [DBG] pgmap v6350: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:01.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:01 smithi154 conmon[34553]: cluster 2021-07-31T04:11:59.868521+0000 mgr.x (mgr.24542) 6206 : cluster [DBG] 2021-07-31T04:12:01.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:01 smithi154 conmon[34553]: pgmap v6351: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:01.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:01 smithi045 conmon[33655]: cluster 2021-07-31T04:11:59.868521+0000 mgr.x (mgr.24542) 6206 : cluster [DBG] pgmap v6351: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:01.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:01 smithi045 conmon[39718]: cluster 2021-07-31T04:11:59.868521+0000 mgr.x 2021-07-31T04:12:01.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:01 smithi045 conmon[39718]: (mgr.24542) 6206 : cluster [DBG] pgmap v6351: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:03.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:03 smithi154 conmon[34553]: cluster 2021-07-31T04:12:01.869079+0000 mgr.x (mgr.24542) 6207 : 2021-07-31T04:12:03.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:03 smithi154 conmon[34553]: cluster [DBG] pgmap v6352: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:03.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:03 smithi045 conmon[39718]: cluster 2021-07-31T04:12:01.869079+0000 mgr.x (mgr.24542) 6207 : cluster [DBG] pgmap v6352: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:03.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:03 smithi045 conmon[33655]: cluster 2021-07-31T04:12:01.869079+0000 mgr.x (mgr.24542) 6207 2021-07-31T04:12:03.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:03 smithi045 conmon[33655]: : cluster [DBG] pgmap v6352: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:05.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:05 smithi154 conmon[34553]: cluster 2021-07-31T04:12:03.869638+0000 mgr.x (mgr.24542) 6208 : cluster [DBG] pgmap v6353: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:05.774 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:05 smithi045 conmon[33655]: cluster 2021-07-31T04:12:03.869638+0000 mgr.x (mgr.24542) 6208 : cluster 2021-07-31T04:12:05.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:05 smithi045 conmon[33655]: [DBG] pgmap v6353: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:05.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:05 smithi045 conmon[39718]: cluster 2021-07-31T04:12:03.869638+0000 2021-07-31T04:12:05.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:05 smithi045 conmon[39718]: mgr.x (mgr.24542) 6208 : cluster [DBG] pgmap v6353: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:07.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:07 smithi154 conmon[34553]: cluster 2021-07-31T04:12:05.870085+0000 mgr.x (mgr.24542) 6209 : cluster [DBG] pgmap v6354: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:07.774 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:07 smithi045 conmon[33655]: cluster 2021-07-31T04:12:05.870085+0000 2021-07-31T04:12:07.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:07 smithi045 conmon[33655]: mgr.x (mgr.24542) 6209 : cluster [DBG] pgmap v6354: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:07.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:07 smithi045 conmon[39718]: cluster 2021-07-31T04:12:05.870085+0000 mgr.x 2021-07-31T04:12:07.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:07 smithi045 conmon[39718]: (mgr.24542) 6209 : cluster [DBG] pgmap v6354: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:09.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:09 smithi154 conmon[34553]: cluster 2021-07-31T04:12:07.870604+0000 mgr.x (mgr.24542) 6210 : cluster [DBG] pgmap v6355: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:09.774 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:09 smithi045 conmon[39718]: cluster 2021-07-31T04:12:07 2021-07-31T04:12:09.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:09 smithi045 conmon[39718]: .870604+0000 mgr.x (mgr.24542) 6210 : cluster [DBG] pgmap v6355: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:09.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:09 smithi045 conmon[33655]: cluster 2021-07-31T04:12:07.870604+0000 mgr.x (mgr.24542) 6210 2021-07-31T04:12:09.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:09 smithi045 conmon[33655]: : cluster [DBG] pgmap v6355: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:11.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:11 smithi154 conmon[34553]: cluster 2021-07-31T04:12:09.871046+0000 mgr.x (mgr.24542) 6211 : cluster [DBG] pgmap v6356: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:11.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:11 smithi045 conmon[39718]: cluster 2021-07-31T04:12 2021-07-31T04:12:11.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:11 smithi045 conmon[39718]: :09.871046+0000 mgr.x (mgr.24542) 6211 : cluster [DBG] pgmap v6356: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:11.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:11 smithi045 conmon[33655]: cluster 2021-07-31T04:12:09.871046+0000 mgr.x (mgr.24542) 6211 : cluster [DBG] pgmap v6356: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:13.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:13 smithi154 conmon[34553]: cluster 2021-07-31T04:12:11.871418+0000 mgr.x (mgr.24542) 6212 : cluster [DBG] pgmap v6357: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:13.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:13 smithi045 conmon[39718]: cluster 2021-07-31T04:12:11.871418+0000 mgr.x 2021-07-31T04:12:13.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:13 smithi045 conmon[39718]: (mgr.24542) 6212 : cluster [DBG] pgmap v6357: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:13.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:13 smithi045 conmon[33655]: cluster 2021-07-31T04:12:11.871418+0000 mgr.x (mgr.24542) 6212 : cluster [DBG] pgmap v6357: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:15.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:15 smithi154 conmon[34553]: cluster 2021-07-31T04:12:13.872002+0000 mgr.x (mgr.24542) 6213 : cluster [DBG] pgmap v6358: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:15.774 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:15 smithi045 conmon[39718]: cluster 2021-07-31T04:12:13.872002+0000 mgr.x (mgr.24542 2021-07-31T04:12:15.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:15 smithi045 conmon[39718]: ) 6213 : cluster [DBG] pgmap v6358: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:15.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:15 smithi045 conmon[33655]: cluster 2021- 2021-07-31T04:12:15.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:15 smithi045 conmon[33655]: 07-31T04:12:13.872002+0000 mgr.x (mgr.24542) 6213 : cluster [DBG] pgmap v6358: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:17.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:17 smithi154 conmon[34553]: cluster 2021-07-31T04:12:15.872443+0000 mgr.x (mgr.24542) 6214 : cluster [DBG] pgmap v6359: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:17.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:17 smithi045 conmon[39718]: cluster 2021-07-31T04:12:15.872443+0000 mgr.x (mgr.24542 2021-07-31T04:12:17.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:17 smithi045 conmon[39718]: ) 6214 : cluster [DBG] pgmap v6359: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:17.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:17 smithi045 conmon[33655]: cluster 2021-07-31T04:12:15.872443+0000 mgr.x (mgr.24542) 6214 : cluster [DBG] pgmap v6359: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:18.328 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T04:12:18.607 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T04:12:18.607 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 2h ago 2h - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T04:12:18.607 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 2h ago 2h 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T04:12:18.607 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 2h ago 2h 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T04:12:18.608 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 2h ago 2h - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T04:12:18.608 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T04:12:18.608 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 2h ago 2h 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T04:12:18.608 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T04:12:18.608 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 2h ago 2h - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T04:12:18.609 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 2h ago 2h 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T04:12:18.609 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T04:12:18.609 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T04:12:18.609 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T04:12:18.609 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T04:12:18.610 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 2h ago 2h 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T04:12:18.610 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 2h ago 2h 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T04:12:18.610 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 2h ago 2h 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T04:12:18.610 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 2h ago 2h 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T04:12:18.610 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 2h ago 2h 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T04:12:18.945 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T04:12:18.945 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T04:12:18.945 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T04:12:18.946 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:12:18.946 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T04:12:18.946 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T04:12:18.946 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:12:18.946 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:12:18.947 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T04:12:18.947 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T04:12:18.947 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:12:18.947 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T04:12:18.947 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T04:12:18.948 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T04:12:18.948 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:12:18.948 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T04:12:18.948 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T04:12:19.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:19 smithi154 conmon[34553]: cluster 2021-07-31T04:12:17.872965+0000 mgr.x (mgr.24542) 6215 : cluster [DBG] pgmap v6360: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:19.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:19 smithi154 conmon[34553]: audit 2021-07-31T04:12:18.316404 2021-07-31T04:12:19.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:19 smithi154 conmon[34553]: +0000 mgr.x (mgr.24542) 6216 : audit [DBG] from='client.31045 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:12:19.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:19 smithi154 conmon[34553]: audit 2021-07-31T04:12:18.946333+0000 mon.c (mon.1) 256 : audit [DBG] from='client.? 172.21.15.45:0/2659177217' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:12:19.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:19 smithi045 conmon[33655]: cluster 2021-07-31T04:12:17.872965+0000 mgr.x (mgr.24542 2021-07-31T04:12:19.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:19 smithi045 conmon[33655]: ) 6215 : cluster [DBG] pgmap v6360: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:19.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:19 smithi045 conmon[33655]: audit 2021-07-31T04:12:18.316404+0000 mgr.x 2021-07-31T04:12:19.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:19 smithi045 conmon[33655]: (mgr.24542) 6216 : audit [DBG] from='client.31045 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:12:19.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:19 smithi045 conmon[33655]: audit 2021-07-31T04:12:19.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:19 smithi045 conmon[33655]: 2021-07- 2021-07-31T04:12:19.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:19 smithi045 conmon[33655]: 31T04:12:18.946333+0000 mon.c (mon.1) 2021-07-31T04:12:19.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:19 smithi045 conmon[33655]: 256 : audit [DBG] from='client.? 172.21.15.45:0/2659177217' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:12:19.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:19 smithi045 conmon[39718]: cluster 2021-07-31T04:12:17.872965+0000 2021-07-31T04:12:19.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:19 smithi045 conmon[39718]: mgr.x (mgr.24542) 6215 : cluster [DBG] pgmap v6360: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:19.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:19 smithi045 conmon[39718]: audit 2021- 2021-07-31T04:12:19.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:19 smithi045 conmon[39718]: 07-31T04:12:18.316404+0000 mgr.x ( 2021-07-31T04:12:19.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:19 smithi045 conmon[39718]: mgr.24542) 6216 : audit 2021-07-31T04:12:19.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:19 smithi045 conmon[39718]: [DBG] from='client.31045 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:12:19.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:19 smithi045 conmon[39718]: audit 2021-07 2021-07-31T04:12:19.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:19 smithi045 conmon[39718]: -31T04:12:18.946333+0000 mon.c ( 2021-07-31T04:12:19.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:19 smithi045 conmon[39718]: mon.1) 256 : audit [DBG] from='client.? 172.21.15.45:0/2659177217' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:12:19.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:19 smithi045 conmon[39718]: 2021-07-31T04:12:20.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:20 smithi154 conmon[34553]: audit 2021-07-31T04:12:18.599513+0000 mgr.x (mgr.24542) 6217 : audit [DBG] from='client.21114 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:12:20.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:20 smithi154 conmon[34553]: 2021-07-31T04:12:20.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:20 smithi045 conmon[33655]: audit 2021-07-31T04:12:18.599513+0000 mgr.x (mgr.24542) 6217 : audit [DBG] from='client.21114 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:12:20.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:20 smithi045 conmon[39718]: audit 2021-07-31T04:12:18.599513+0000 mgr.x (mgr.24542) 6217 2021-07-31T04:12:20.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:20 smithi045 conmon[39718]: : audit [DBG] from='client.21114 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:12:21.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:21 smithi154 conmon[34553]: cluster 2021-07-31T04:12:19.873574+0000 mgr.x (mgr.24542) 6218 : cluster [DBG] pgmap v6361: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:21.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:21 smithi045 conmon[33655]: cluster 2021-07-31T04:12:19.873574+0000 mgr.x (mgr.24542 2021-07-31T04:12:21.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:21 smithi045 conmon[33655]: ) 6218 : cluster [DBG] pgmap v6361: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:21.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:21 smithi045 conmon[39718]: cluster 2021-07-31T04:12:19.873574+0000 mgr.x 2021-07-31T04:12:21.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:21 smithi045 conmon[39718]: (mgr.24542) 6218 : cluster [DBG] pgmap v6361: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:23.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:23 smithi154 conmon[34553]: cluster 2021-07-31T04:12:21.874170+0000 mgr.x (mgr.24542) 6219 : cluster [DBG] pgmap v6362: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:23.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:23 smithi045 conmon[33655]: cluster 2021-07-31T04:12:21.874170+0000 mgr.x (mgr.24542) 6219 : cluster [DBG] 2021-07-31T04:12:23.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:23 smithi045 conmon[33655]: pgmap v6362: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:23.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:23 smithi045 conmon[39718]: cluster 2021-07-31T04:12:21.874170+0000 mgr.x 2021-07-31T04:12:23.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:23 smithi045 conmon[39718]: (mgr.24542) 6219 : cluster [DBG] pgmap v6362: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:25.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:25 smithi154 conmon[34553]: cluster 2021-07-31T04:12:23.874530+0000 mgr.x (mgr.24542) 6220 : cluster [DBG] pgmap v6363: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:25.774 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:25 smithi045 conmon[39718]: cluster 2021-07-31T04:12:23.874530+0000 mgr.x 2021-07-31T04:12:25.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:25 smithi045 conmon[39718]: (mgr.24542) 6220 : cluster [DBG] pgmap v6363: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:25.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:25 smithi045 conmon[33655]: cluster 2021-07-31T04:12:23.874530+0000 mgr.x (mgr.24542 2021-07-31T04:12:25.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:25 smithi045 conmon[33655]: ) 6220 : cluster [DBG] pgmap v6363: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:27.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:27 smithi154 conmon[34553]: cluster 2021-07-31T04:12:25.874956+0000 mgr.x (mgr.24542) 6221 : cluster [DBG] pgmap v6364: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:27.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:27 smithi154 conmon[34553]: 2021-07-31T04:12:27.774 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:27 smithi045 conmon[39718]: cluster 2021-07-31T04:12:25.874956+0000 mgr.x (mgr.24542) 6221 : cluster [DBG] pgmap v6364: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:27.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:27 smithi045 conmon[33655]: cluster 2021-07-31T04:12:25.874956+0000 mgr.x (mgr.24542) 6221 : cluster [DBG] 2021-07-31T04:12:27.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:27 smithi045 conmon[33655]: pgmap v6364: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:29.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:29 smithi154 conmon[34553]: cluster 2021-07-31T04:12:27.875371+0000 mgr.x (mgr.24542) 6222 : cluster [DBG] pgmap v6365: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:29.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:29 smithi154 conmon[34553]: 2021-07-31T04:12:29.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:29 smithi154 conmon[34553]: audit 2021-07-31T04:12:28.508192+0000 mon.b (mon.2) 800 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:12:29.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:29 smithi154 conmon[34553]: audit 2021-07-31T04:12:28.508570+0000 mon.b (mon.2) 801 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:12:29.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:29 smithi154 conmon[34553]: audit 2021-07-31T04:12:28.509221+0000 mon.a (mon.0) 3439 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:12:29.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:29 smithi154 conmon[34553]: audit 2021-07-31T04:12:28.509606+0000 mon.a (mon.0) 3440 2021-07-31T04:12:29.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:29 smithi154 conmon[34553]: : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:12:29.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:29 smithi045 conmon[39718]: cluster 2021-07-31T04:12:27.875371+0000 mgr.x (mgr.24542) 6222 : cluster [DBG] 2021-07-31T04:12:29.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:29 smithi045 conmon[39718]: pgmap v6365: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:29.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:29 smithi045 conmon[39718]: audit 2021-07-31T04:12:28.508192+0000 mon.b (mon.2) 800 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:12:29.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:29 smithi045 conmon[39718]: audit 2021-07-31T04:12:28.508570+0000 mon.b (mon.2) 801 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:12:29.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:29 smithi045 conmon[39718]: 2021-07-31T04:12:29.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:29 smithi045 conmon[39718]: audit 2021-07-31T04: 2021-07-31T04:12:29.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:29 smithi045 conmon[39718]: 12:28.509221+0000 mon.a (mon.0) 3439 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:12:29.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:29 smithi045 conmon[39718]: audit 2021-07-31T04:12: 2021-07-31T04:12:29.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:29 smithi045 conmon[39718]: 28.509606+0000 mon.a (mon.0) 3440 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:12:29.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:29 smithi045 conmon[33655]: cluster 2021-07-31T04:12:27.875371+0000 mgr.x ( 2021-07-31T04:12:29.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:29 smithi045 conmon[33655]: mgr.24542) 6222 : cluster [DBG] pgmap v6365: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:29.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:29 smithi045 conmon[33655]: audit 2021-07-31T04:12:28.508192+0000 mon.b (mon.2) 800 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:12:29.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:29 smithi045 conmon[33655]: audit 2021-07-31T04 2021-07-31T04:12:29.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:29 smithi045 conmon[33655]: :12:28.508570+0000 mon.b (mon.2) 801 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:12:29.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:29 smithi045 conmon[33655]: audit 2021-07-31T04:12:28.509221+0000 mon.a (mon.0) 3439 : audit 2021-07-31T04:12:29.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:29 smithi045 conmon[33655]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:12:29.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:29 smithi045 conmon[33655]: audit 2021-07-31T04:12:28.509606+0000 mon.a (mon.0) 2021-07-31T04:12:29.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:29 smithi045 conmon[33655]: 3440 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:12:31.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:31 smithi154 conmon[34553]: cluster 2021-07-31T04:12:29.875898+0000 mgr.x (mgr.24542) 6223 : cluster [DBG] pgmap v6366: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:31.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:31 smithi045 conmon[39718]: cluster 2021-07-31T04:12:29.875898+0000 2021-07-31T04:12:31.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:31 smithi045 conmon[39718]: mgr.x (mgr.24542) 6223 : cluster [DBG] pgmap v6366: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:31.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:31 smithi045 conmon[33655]: cluster 2021-07-31T04:12:29.875898+0000 2021-07-31T04:12:31.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:31 smithi045 conmon[33655]: mgr.x (mgr.24542) 6223 : cluster [DBG] pgmap v6366: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:33.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:33 smithi154 conmon[34553]: cluster 2021-07-31T04:12:31.876387+0000 mgr.x (mgr.24542) 6224 : 2021-07-31T04:12:33.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:33 smithi154 conmon[34553]: cluster [DBG] pgmap v6367: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:33.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:33 smithi045 conmon[39718]: cluster 2021-07-31T04:12:31.876387+0000 2021-07-31T04:12:33.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:33 smithi045 conmon[39718]: mgr.x (mgr.24542) 6224 : cluster [DBG] pgmap v6367: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:33.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:33 smithi045 conmon[33655]: cluster 2021-07-31T04:12:31 2021-07-31T04:12:33.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:33 smithi045 conmon[33655]: .876387+0000 mgr.x (mgr.24542) 6224 : cluster [DBG] pgmap v6367: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:35.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:35 smithi154 conmon[34553]: cluster 2021-07-31T04:12:33.877006+0000 mgr.x (mgr.24542) 6225 : cluster [DBG] pgmap v6368: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:35.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:35 smithi045 conmon[39718]: cluster 2021-07-31T04:12:33.877006 2021-07-31T04:12:35.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:35 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 6225 : cluster [DBG] pgmap v6368: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:35.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:35 smithi045 conmon[33655]: cluster 2021-07-31T04:12:33. 2021-07-31T04:12:35.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:35 smithi045 conmon[33655]: 877006+0000 mgr.x (mgr.24542) 6225 : cluster [DBG] pgmap v6368: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:37.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[34553]: cluster 2021-07-31T04:12:35.877379+0000 mgr.x (mgr.24542) 6226 : cluster [DBG] pgmap v6369: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:37.752 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: debug 2021-07-31T04:12:37.517+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:12:37.753 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:12:37.753 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:12:37.753 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: yield 2021-07-31T04:12:37.753 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:12:37.753 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:12:37.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:12:37.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:12:37.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:12:37.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:12:37.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:12:37.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:12:37.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:12:37.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: task.result() 2021-07-31T04:12:37.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:12:37.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: debug 2021-07-31T04:12:37.528+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:12:37.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:12:37.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:12:37.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: yield 2021-07-31T04:12:37.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:12:37.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:12:37.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:12:37.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:12:37.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:12:37.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:12:37.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:12:37.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:12:37.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:12:37.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: task.result() 2021-07-31T04:12:37.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:12:37.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: debug 2021-07-31T04:12:37.553+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:12:37.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:12:37.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:12:37.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: yield 2021-07-31T04:12:37.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:12:37.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:12:37.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:12:37.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:12:37.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:12:37.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:12:37.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:12:37.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:12:37.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:12:37.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: task.result() 2021-07-31T04:12:37.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:12:37.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: debug 2021-07-31T04:12:37.554+0000 7fcb94162700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:12:37.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:12:37.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:12:37.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: yield 2021-07-31T04:12:37.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:12:37.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:12:37.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:12:37.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:12:37.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:12:37.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:12:37.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:12:37.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:12:37.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:12:37.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: task.result() 2021-07-31T04:12:37.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:12:37.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: 2021-07-31T04:12:37.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T04:12:37.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: 2021-07-31T04:12:37.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:12:37.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:12:37.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: return f(*arg) 2021-07-31T04:12:37.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:12:37.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:12:37.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:12:37.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:12:37.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:12:37.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:12:37.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:12:37.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T04:12:37.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:12:37.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: raise self._exception 2021-07-31T04:12:37.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:12:37.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:12:37.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:12:37.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T04:12:37.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:12:37.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: raise 2021-07-31T04:12:37.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:12:37.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T04:12:37.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:12:37.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:12:37.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:12:37.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: debug 2021-07-31T04:12:37.562+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:12:37.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:12:37.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:12:37.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: yield 2021-07-31T04:12:37.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:12:37.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:12:37.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:12:37.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:12:37.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:12:37.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:12:37.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:12:37.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:12:37.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:12:37.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: task.result() 2021-07-31T04:12:37.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:12:37.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: debug 2021-07-31T04:12:37.562+0000 7fcb93961700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:12:37.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:12:37.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:12:37.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: yield 2021-07-31T04:12:37.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:12:37.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:12:37.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:12:37.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:12:37.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:12:37.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:12:37.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:12:37.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:12:37.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:12:37.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: task.result() 2021-07-31T04:12:37.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:12:37.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: 2021-07-31T04:12:37.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T04:12:37.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: 2021-07-31T04:12:37.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:12:37.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:12:37.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: return f(*arg) 2021-07-31T04:12:37.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:12:37.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:12:37.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:12:37.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:12:37.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:12:37.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:12:37.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:12:37.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T04:12:37.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:12:37.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: raise self._exception 2021-07-31T04:12:37.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:12:37.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:12:37.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:12:37.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T04:12:37.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:12:37.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: raise 2021-07-31T04:12:37.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:12:37.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T04:12:37.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:12:37.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:12:37.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:12:37 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:12:37.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:37 smithi045 conmon[33655]: cluster 2021-07-31T04:12:35.877379+0000 mgr.x 2021-07-31T04:12:37.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:37 smithi045 conmon[33655]: (mgr.24542) 6226 : cluster [DBG] pgmap v6369: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:37.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:37 smithi045 conmon[39718]: cluster 2021-07-31T04:12:35.877379+0000 mgr.x (mgr.24542) 6226 : cluster 2021-07-31T04:12:37.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:37 smithi045 conmon[39718]: [DBG] pgmap v6369: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:38.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:38 smithi154 conmon[34553]: audit 2021-07-31T04:12:37.471645+0000 mon.b (mon.2) 802 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T04:12:38.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:38 smithi154 conmon[34553]: audit 2021-07-31T04:12:37.473133+0000 mon.b (mon.2) 803 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T04:12:38.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:38 smithi154 conmon[34553]: audit 2021-07-31T04:12:37.473986+0000 mon.b (mon.2 2021-07-31T04:12:38.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:38 smithi154 conmon[34553]: ) 804 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T04:12:38.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:38 smithi045 conmon[33655]: audit 2021-07-31T04:12 2021-07-31T04:12:38.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:38 smithi045 conmon[33655]: :37.471645+0000 mon.b (mon.2) 802 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T04:12:38.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:38 smithi045 conmon[33655]: audit 2021 2021-07-31T04:12:38.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:38 smithi045 conmon[33655]: -07-31T04:12:37.473133+0000 mon.b (mon.2 2021-07-31T04:12:38.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:38 smithi045 conmon[33655]: ) 803 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T04:12:38.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:38 smithi045 conmon[33655]: audit 2021-07-31T04:12 2021-07-31T04:12:38.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:38 smithi045 conmon[33655]: :37.473986+0000 mon.b (mon. 2021-07-31T04:12:38.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:38 smithi045 conmon[33655]: 2) 804 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T04:12:38.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:38 smithi045 conmon[39718]: audit 2021-07-31T04:12:37.471645+0000 2021-07-31T04:12:38.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:38 smithi045 conmon[39718]: mon.b (mon.2) 802 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T04:12:38.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:38 smithi045 conmon[39718]: audit 2021- 2021-07-31T04:12:38.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:38 smithi045 conmon[39718]: 07-31T04:12:37.473133+0000 mon.b (mon.2) 803 : 2021-07-31T04:12:38.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:38 smithi045 conmon[39718]: audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T04:12:38.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:38 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T04:12:38.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:38 smithi045 conmon[39718]: 04:12:37.473986+0000 mon.b (mon.2) 2021-07-31T04:12:38.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:38 smithi045 conmon[39718]: 804 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T04:12:39.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:12:37.474739+0000 mgr.x (mgr.24542) 6227 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T04:12:39.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:12:37.517967+0000 mgr.x (mgr.24542) 6228 : cephadm [ERR] 2021-07-31T04:12:39.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:12:39.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:12:39.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: yield 2021-07-31T04:12:39.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:12:39.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:12:39.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:12:39.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:12:39.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:12:39.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:12:39.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:12:39.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:12:39.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:12:39.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: task.result() 2021-07-31T04:12:39.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:12:39.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:12:37.518433+0000 mgr.x (mgr.24542) 6229 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T04:12:39.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:12:37.529403+0000 mgr.x (mgr.24542) 6230 : cephadm [ERR] 2021-07-31T04:12:39.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:12:39.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:12:39.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: yield 2021-07-31T04:12:39.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:12:39.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:12:39.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:12:39.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:12:39.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:12:39.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:12:39.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:12:39.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:12:39.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:12:39.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: task.result() 2021-07-31T04:12:39.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:12:39.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:12:37.530197+0000 mgr.x (mgr.24542) 6231 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T04:12:39.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:12:37.554510+0000 mgr.x (mgr.24542) 6232 : cephadm [ERR] 2021-07-31T04:12:39.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:12:39.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:12:39.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: yield 2021-07-31T04:12:39.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:12:39.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:12:39.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:12:39.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:12:39.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:12:39.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:12:39.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:12:39.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:12:39.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:12:39.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: task.result() 2021-07-31T04:12:39.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:12:39.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:12:37.554946+0000 mgr.x 2021-07-31T04:12:39.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: (mgr.24542) 6233 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:12:39.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:12:39.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:12:39.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: yield 2021-07-31T04:12:39.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:12:39.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:12:39.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:12:39.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:12:39.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:12:39.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:12:39.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:12:39.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:12:39.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:12:39.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: task.result() 2021-07-31T04:12:39.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:12:39.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: 2021-07-31T04:12:39.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T04:12:39.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: 2021-07-31T04:12:39.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:12:39.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:12:39.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: return f(*arg) 2021-07-31T04:12:39.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:12:39.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:12:39.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:12:39.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:12:39.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:12:39.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:12:39.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:12:39.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T04:12:39.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:12:39.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: raise self._exception 2021-07-31T04:12:39.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:12:39.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:12:39.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:12:39.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T04:12:39.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:12:39.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: raise 2021-07-31T04:12:39.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:12:39.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T04:12:39.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:12:39.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:12:39.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:12:39.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:12:37.563318+0000 mgr.x (mgr.24542) 6234 : cephadm [ERR] 2021-07-31T04:12:39.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:12:39.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:12:39.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: yield 2021-07-31T04:12:39.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:12:39.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:12:39.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:12:39.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:12:39.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:12:39.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:12:39.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:12:39.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:12:39.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:12:39.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: task.result() 2021-07-31T04:12:39.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:12:39.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:12:37.563717+0000 mgr.x (mgr.24542) 6235 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:12:39.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:12:39.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:12:39.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: yield 2021-07-31T04:12:39.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:12:39.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:12:39.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:12:39.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:12:39.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:12:39.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:12:39.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:12:39.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:12:39.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:12:39.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: task.result() 2021-07-31T04:12:39.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:12:39.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: 2021-07-31T04:12:39.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T04:12:39.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: 2021-07-31T04:12:39.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:12:39.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:12:39.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: return f(*arg) 2021-07-31T04:12:39.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:12:39.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:12:39.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:12:39.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:12:39.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:12:39.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:12:39.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:12:39.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T04:12:39.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:12:39.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: raise self._exception 2021-07-31T04:12:39.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:12:39.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:12:39.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:12:39.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T04:12:39.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:12:39.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: raise 2021-07-31T04:12:39.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:12:39.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T04:12:39.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:12:39.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:12:39.782 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:12:39.782 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:39 smithi154 conmon[34553]: cluster 2021-07-31T04:12:37.877900+0000 mgr.x (mgr.24542) 6236 : cluster [DBG] pgmap v6370: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:39.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: cephadm 2021-07-31T04:12:37.474739+0000 mgr.x (mgr.24542) 6227 : cephadm 2021-07-31T04:12:39.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T04:12:39.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: cephadm 2021-07-31T04:12:37.517967+0000 mgr.x (mgr.24542) 6228 : cephadm [ERR] 2021-07-31T04:12:39.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:12:39.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:12:39.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: yield 2021-07-31T04:12:39.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:12:39.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:12:39.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:12:39.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:12:39.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:12:39.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:12:39.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:12:39.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:12:39.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:12:39.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: task.result() 2021-07-31T04:12:39.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:12:39.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: cephadm 2021-07-31T04:12:37.518433+0000 mgr.x (mgr.24542) 6229 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T04:12:39.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: cephadm 2021-07-31T04:12:37.529403+0000 mgr.x (mgr.24542) 6230 : cephadm [ERR] 2021-07-31T04:12:39.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:12:39.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:12:39.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: yield 2021-07-31T04:12:39.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:12:39.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:12:39.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:12:39.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:12:39.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:12:39.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:12:39.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:12:39.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:12:39.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:12:39.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: task.result() 2021-07-31T04:12:39.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:12:39.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: cephadm 2021-07-31T04 2021-07-31T04:12:39.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: :12:37.530197+0000 mgr.x (mgr.24542) 6231 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T04:12:39.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: cephadm 2021-07-31T04:12:37.554510+0000 mgr.x (mgr.24542) 6232 : cephadm [ERR] 2021-07-31T04:12:39.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:12:39.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:12:39.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: yield 2021-07-31T04:12:39.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:12:39.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:12:39.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:12:39.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:12:39.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:12:39.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:12:39.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:12:39.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:12:39.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:12:39.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: task.result() 2021-07-31T04:12:39.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:12:39.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: cephadm 2021-07-31T04:12:37.554946+0000 mgr.x (mgr.24542) 6233 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:12:39.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:12:39.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:12:39.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: yield 2021-07-31T04:12:39.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:12:39.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:12:39.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:12:39.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:12:39.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:12:39.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:12:39.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:12:39.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:12:39.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:12:39.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: task.result() 2021-07-31T04:12:39.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:12:39.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: 2021-07-31T04:12:39.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T04:12:39.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: 2021-07-31T04:12:39.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:12:39.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:12:39.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: return f(*arg) 2021-07-31T04:12:39.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:12:39.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:12:39.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:12:39.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:12:39.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:12:39.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:12:39.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:12:39.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T04:12:39.798 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: cephadm 2021-07-31T04:12:37.474739+0000 mgr.x (mgr.24542 2021-07-31T04:12:39.798 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: ) 6227 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T04:12:39.799 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: cephadm 2021-07-31T04:12:37.517967+0000 mgr.x (mgr.24542) 6228 : cephadm [ERR] 2021-07-31T04:12:39.799 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:12:39.799 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:12:39.799 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: yield 2021-07-31T04:12:39.799 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:12:39.800 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:12:39.800 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:12:39.800 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:12:39.800 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:12:39.801 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:12:39.801 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:12:39.801 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:12:39.801 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:12:39.801 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: task.result() 2021-07-31T04:12:39.802 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:12:39.802 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: cephadm 2021-07-31T04:12:37.518433+0000 mgr.x (mgr.24542) 6229 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T04:12:39.802 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: cephadm 2021-07-31T04:12:37.529403+0000 mgr.x (mgr.24542) 6230 : cephadm [ERR] 2021-07-31T04:12:39.802 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:12:39.802 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:12:39.803 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: yield 2021-07-31T04:12:39.803 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:12:39.803 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:12:39.803 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:12:39.803 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:12:39.804 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:12:39.804 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:12:39.804 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:12:39.804 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:12:39.804 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:12:39.804 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: task.result() 2021-07-31T04:12:39.805 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:12:39.805 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: cephadm 2021 2021-07-31T04:12:39.805 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: -07-31T04:12:37.530197+0000 mgr.x (mgr.24542) 6231 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T04:12:39.805 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: cephadm 2021-07-31T04:12:37.554510+0000 mgr.x (mgr.24542) 6232 : cephadm [ERR] 2021-07-31T04:12:39.805 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:12:39.806 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:12:39.806 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:12:39.806 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: raise self._exception 2021-07-31T04:12:39.807 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:12:39.807 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:12:39.807 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:12:39.807 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T04:12:39.807 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:12:39.807 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: raise 2021-07-31T04:12:39.808 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:12:39.808 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T04:12:39.808 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:12:39.808 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:12:39.808 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:12:39.809 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: cephadm 2021-07- 2021-07-31T04:12:39.809 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: 31T04:12:37.563318+0000 mgr.x (mgr.24542) 6234 : cephadm [ERR] 2021-07-31T04:12:39.809 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:12:39.809 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:12:39.809 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: yield 2021-07-31T04:12:39.810 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:12:39.810 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:12:39.810 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:12:39.810 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:12:39.810 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:12:39.811 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:12:39.811 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:12:39.811 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:12:39.811 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:12:39.811 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: task.result() 2021-07-31T04:12:39.812 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:12:39.812 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: cephadm 2021-07-31T04:12:37.563717+0000 mgr.x (mgr.24542) 6235 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:12:39.812 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:12:39.812 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:12:39.812 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: yield 2021-07-31T04:12:39.813 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:12:39.813 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:12:39.813 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:12:39.813 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:12:39.813 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:12:39.814 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:12:39.814 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: yield 2021-07-31T04:12:39.814 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:12:39.814 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:12:39.815 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:12:39.815 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:12:39.815 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:12:39.815 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:12:39.815 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:12:39.816 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:12:39.816 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:12:39.816 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: task.result() 2021-07-31T04:12:39.816 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:12:39.816 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: cephadm 2021-07-31T04: 2021-07-31T04:12:39.817 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: 12:37.554946+0000 mgr.x (mgr.24542) 6233 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:12:39.817 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:12:39.817 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:12:39.817 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: yield 2021-07-31T04:12:39.817 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:12:39.818 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:12:39.818 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:12:39.818 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:12:39.818 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:12:39.818 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:12:39.819 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:12:39.819 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:12:39.819 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:12:39.819 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: task.result() 2021-07-31T04:12:39.819 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:12:39.820 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: 2021-07-31T04:12:39.820 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T04:12:39.820 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: 2021-07-31T04:12:39.820 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:12:39.820 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:12:39.820 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: return f(*arg) 2021-07-31T04:12:39.821 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:12:39.821 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:12:39.821 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:12:39.821 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:12:39.821 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:12:39.822 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:12:39.822 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:12:39.822 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:12:39.822 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:12:39.823 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:12:39.823 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: task.result() 2021-07-31T04:12:39.823 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:12:39.823 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: 2021-07-31T04:12:39.823 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T04:12:39.824 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: 2021-07-31T04:12:39.824 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:12:39.824 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:12:39.824 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: return f(*arg) 2021-07-31T04:12:39.824 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:12:39.825 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:12:39.825 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:12:39.825 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:12:39.825 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:12:39.825 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:12:39.826 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:12:39.826 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T04:12:39.826 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:12:39.826 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: raise self._exception 2021-07-31T04:12:39.826 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:12:39.827 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:12:39.827 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:12:39.827 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T04:12:39.827 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:12:39.827 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: raise 2021-07-31T04:12:39.827 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:12:39.828 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T04:12:39.828 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:12:39.828 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:12:39.828 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:12:39.828 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[39718]: cluster 2021-07-31T04:12:37.877900+0000 mgr.x (mgr.24542) 6236 : cluster [DBG] pgmap v6370: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:39.829 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T04:12:39.829 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:12:39.829 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: raise self._exception 2021-07-31T04:12:39.830 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:12:39.830 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:12:39.830 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:12:39.830 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T04:12:39.830 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:12:39.831 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: raise 2021-07-31T04:12:39.831 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:12:39.831 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T04:12:39.831 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:12:39.831 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:12:39.832 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:12:39.832 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: cephadm 2021-07-31T04:12:37.563318+0000 mgr.x (mgr.24542) 6234 : cephadm [ERR] 2021-07-31T04:12:39.832 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:12:39.832 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:12:39.832 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: yield 2021-07-31T04:12:39.833 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:12:39.833 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:12:39.833 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:12:39.833 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:12:39.833 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:12:39.833 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:12:39.834 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:12:39.834 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:12:39.834 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:12:39.834 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: task.result() 2021-07-31T04:12:39.834 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:12:39.835 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: cephadm 2021-07-31T04:12:37.563717+0000 mgr.x (mgr.24542) 6235 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:12:39.835 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:12:39.835 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:12:39.835 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: yield 2021-07-31T04:12:39.835 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:12:39.836 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:12:39.836 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:12:39.836 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:12:39.836 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:12:39.836 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:12:39.837 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:12:39.837 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:12:39.837 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:12:39.837 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: task.result() 2021-07-31T04:12:39.837 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:12:39.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: 2021-07-31T04:12:39.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T04:12:39.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: 2021-07-31T04:12:39.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:12:39.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:12:39.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: return f(*arg) 2021-07-31T04:12:39.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:12:39.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:12:39.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:12:39.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:12:39.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:12:39.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:12:39.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:12:39.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T04:12:39.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:12:39.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: raise self._exception 2021-07-31T04:12:39.841 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:12:39.841 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:12:39.841 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:12:39.841 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T04:12:39.841 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:12:39.842 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: raise 2021-07-31T04:12:39.842 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:12:39.842 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T04:12:39.842 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:12:39.842 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:12:39.843 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:12:39.843 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:39 smithi045 conmon[33655]: cluster 2021-07-31T04:12:37.877900+0000 mgr.x (mgr.24542) 6236 : cluster [DBG] pgmap v6370: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:41.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:41 smithi154 conmon[34553]: cluster 2021-07-31T04:12:39.878378+0000 mgr.x (mgr.24542) 6237 : cluster [DBG] pgmap v6371: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:41.774 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:41 smithi045 conmon[33655]: cluster 2021-07-31T04:12:39 2021-07-31T04:12:41.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:41 smithi045 conmon[33655]: .878378+0000 mgr.x (mgr.24542) 6237 : cluster [DBG] pgmap v6371: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:41.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:41 smithi045 conmon[39718]: cluster 2021-07-31T04:12:39.878378+0000 mgr.x ( 2021-07-31T04:12:41.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:41 smithi045 conmon[39718]: mgr.24542) 6237 : cluster [DBG] pgmap v6371: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:42.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:42 smithi154 conmon[34553]: cluster 2021-07-31T04:12:41.878915+0000 mgr.x (mgr.24542) 6238 : cluster [DBG] pgmap v6372: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:42.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:42 smithi045 conmon[33655]: cluster 2021-07-31T04:12:41.878915+0000 mgr.x (mgr.24542) 6238 : cluster [DBG] pgmap v6372: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:42.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:42 smithi045 conmon[39718]: cluster 2021-07-31T04:12:41 2021-07-31T04:12:42.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:42 smithi045 conmon[39718]: .878915+0000 mgr.x (mgr.24542) 6238 : cluster [DBG] pgmap v6372: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:45.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:44 smithi154 conmon[34553]: cluster 2021-07-31T04:12:43.879389+0000 mgr.x (mgr.24542) 6239 : cluster [DBG] pgmap v6373: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:45.274 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:44 smithi045 conmon[33655]: cluster 2021-07-31T04:12:43.879389+0000 mgr.x (mgr.24542) 6239 : cluster [DBG] pgmap v6373: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:45.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:44 smithi045 conmon[39718]: cluster 2021-07-31T04:12:43 2021-07-31T04:12:45.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:44 smithi045 conmon[39718]: .879389+0000 mgr.x (mgr.24542) 6239 : cluster [DBG] pgmap v6373: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:47.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:46 smithi154 conmon[34553]: cluster 2021-07-31T04:12:45.879807+0000 mgr.x (mgr.24542) 6240 : cluster [DBG] pgmap v6374: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:47.274 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:46 smithi045 conmon[39718]: cluster 2021-07-31T04:12:45.879807+0000 mgr.x (mgr.24542 2021-07-31T04:12:47.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:46 smithi045 conmon[39718]: ) 6240 : cluster [DBG] pgmap v6374: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:47.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:46 smithi045 conmon[33655]: cluster 2021-07-31T04:12:45.879807+0000 mgr.x (mgr.24542) 6240 : cluster [DBG] 2021-07-31T04:12:47.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:46 smithi045 conmon[33655]: pgmap v6374: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:49.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:48 smithi154 conmon[34553]: cluster 2021-07-31T04:12:47.880305+0000 mgr.x (mgr.24542) 6241 : cluster [DBG] pgmap v6375: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:49.257 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T04:12:49.274 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:48 smithi045 conmon[39718]: cluster 2021-07-31T04:12:47.880305+0000 mgr.x 2021-07-31T04:12:49.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:48 smithi045 conmon[39718]: (mgr.24542) 6241 : cluster [DBG] pgmap v6375: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:49.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:48 smithi045 conmon[33655]: cluster 2021-07-31T04:12:47.880305+0000 mgr.x (mgr.24542) 6241 : 2021-07-31T04:12:49.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:48 smithi045 conmon[33655]: cluster [DBG] pgmap v6375: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:49.534 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T04:12:49.535 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 2h ago 2h - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T04:12:49.535 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 2h ago 2h 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T04:12:49.535 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 2h ago 2h 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T04:12:49.535 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 2h ago 2h - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T04:12:49.536 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T04:12:49.536 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 2h ago 2h 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T04:12:49.536 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T04:12:49.536 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 2h ago 2h - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T04:12:49.536 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 2h ago 2h 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T04:12:49.537 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T04:12:49.537 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T04:12:49.537 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T04:12:49.537 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T04:12:49.538 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 2h ago 2h 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T04:12:49.538 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 2h ago 2h 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T04:12:49.538 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 2h ago 2h 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T04:12:49.538 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 2h ago 2h 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T04:12:49.538 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 2h ago 2h 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T04:12:49.869 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T04:12:49.870 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T04:12:49.870 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T04:12:49.870 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:12:49.871 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T04:12:49.871 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T04:12:49.871 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:12:49.871 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:12:49.872 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T04:12:49.872 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T04:12:49.872 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:12:49.872 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T04:12:49.872 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T04:12:49.873 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T04:12:49.873 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:12:49.873 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T04:12:49.873 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T04:12:50.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:49 smithi154 conmon[34553]: audit 2021-07-31T04:12:49.246017+0000 mgr.x (mgr.24542) 6242 : audit [DBG] from='client.21126 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:12:50.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:49 smithi154 conmon[34553]: audit 2021-07-31T04:12: 2021-07-31T04:12:50.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:49 smithi154 conmon[34553]: 49.870692+0000 mon.c (mon.1) 257 : audit [DBG] from='client.? 172.21.15.45:0/4213058754' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:12:50.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:49 smithi045 conmon[33655]: audit 2021-07-31T04:12:49.246017+0000 mgr.x (mgr. 2021-07-31T04:12:50.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:49 smithi045 conmon[33655]: 24542) 6242 : audit [DBG] from='client.21126 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:12:50.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:49 smithi045 conmon[33655]: audit 2021-07-31T04:12:49.870692+0000 mon.c (mon.1) 257 : audit [DBG] from='client.? 172.21.15.45:0/4213058754' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:12:50.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:49 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T04:12:50.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:49 smithi045 conmon[39718]: 04:12:49.246017+0000 mgr.x (mgr.24542) 6242 : audit [DBG] 2021-07-31T04:12:50.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:49 smithi045 conmon[39718]: from='client.21126 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:12:50.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:49 smithi045 conmon[39718]: audit 2021-07 2021-07-31T04:12:50.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:49 smithi045 conmon[39718]: -31T04:12:49. 2021-07-31T04:12:50.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:49 smithi045 conmon[39718]: 870692+0000 mon.c (mon.1 2021-07-31T04:12:50.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:49 smithi045 conmon[39718]: ) 257 : audit [DBG] from='client.? 172.21.15.45:0/4213058754' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:12:50.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:49 smithi045 conmon[39718]: 2021-07-31T04:12:51.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:50 smithi154 conmon[34553]: audit 2021-07-31T04:12:49.527012+0000 mgr.x (mgr.24542) 6243 : audit [DBG] from='client.31069 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:12:51.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:50 smithi154 conmon[34553]: cluster 2021-07-31T04:12:51.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:50 smithi154 conmon[34553]: 2021-07-31T04:12:49.880901+0000 mgr.x (mgr.24542) 6244 : cluster [DBG] pgmap v6376: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:51.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:50 smithi045 conmon[33655]: audit 2021-07-31T04:12:49.527012+0000 mgr.x (mgr.24542) 6243 : 2021-07-31T04:12:51.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:50 smithi045 conmon[33655]: audit [DBG] from='client.31069 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:12:51.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:50 smithi045 conmon[33655]: cluster 2021-07-31T04:12:49.880901+0000 mgr.x (mgr.24542) 6244 : cluster [DBG] pgmap v6376: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:51.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:50 smithi045 conmon[39718]: audit 2021-07-31T04:12:49.527012+0000 mgr.x (mgr.24542) 6243 : audit [DBG] from='client.31069 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:12:51.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:50 smithi045 conmon[39718]: cluster 2021-07-31T04:12: 2021-07-31T04:12:51.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:50 smithi045 conmon[39718]: 49.880901+0000 mgr.x (mgr.24542) 2021-07-31T04:12:51.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:50 smithi045 conmon[39718]: 6244 : cluster [DBG] pgmap v6376: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:53.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:52 smithi154 conmon[34553]: cluster 2021-07-31T04:12:51.881379+0000 mgr.x (mgr.24542) 6245 2021-07-31T04:12:53.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:52 smithi154 conmon[34553]: : cluster [DBG] pgmap v6377: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:53.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:52 smithi045 conmon[33655]: cluster 2021-07-31T04:12:51.881379+0000 mgr.x (mgr.24542) 2021-07-31T04:12:53.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:52 smithi045 conmon[33655]: 6245 : cluster [DBG] pgmap v6377: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:53.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:52 smithi045 conmon[39718]: cluster 2021-07-31T04:12:51.881379+0000 mgr.x ( 2021-07-31T04:12:53.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:52 smithi045 conmon[39718]: mgr.24542) 6245 : cluster [DBG] pgmap v6377: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:55.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:54 smithi154 conmon[34553]: cluster 2021-07-31T04:12:53.881962+0000 mgr.x (mgr.24542) 6246 : cluster [DBG] pgmap v6378: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:55.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:54 smithi045 conmon[39718]: cluster 2021-07-31T04:12:53.881962+0000 mgr.x 2021-07-31T04:12:55.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:54 smithi045 conmon[39718]: (mgr.24542) 6246 : cluster [DBG] pgmap v6378: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:55.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:54 smithi045 conmon[33655]: cluster 2021-07-31T04:12:53.881962+0000 2021-07-31T04:12:55.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:54 smithi045 conmon[33655]: mgr.x (mgr.24542) 6246 : cluster [DBG] pgmap v6378: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:57.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:56 smithi154 conmon[34553]: cluster 2021-07-31T04:12:55.882524+0000 mgr.x (mgr.24542) 6247 : cluster [DBG] pgmap v6379: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:57.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:56 smithi045 conmon[33655]: cluster 2021-07-31T04:12:55.882524+0000 mgr.x (mgr.24542) 6247 : cluster [DBG] pgmap v6379: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:57.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:56 smithi045 conmon[33655]: 2021-07-31T04:12:57.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:56 smithi045 conmon[39718]: cluster 2021-07-31T04:12:55.882524+0000 mgr.x ( 2021-07-31T04:12:57.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:56 smithi045 conmon[39718]: mgr.24542) 6247 : cluster [DBG] pgmap v6379: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:59.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:12:58 smithi154 conmon[34553]: cluster 2021-07-31T04:12:57.882936+0000 mgr.x (mgr.24542) 6248 : cluster [DBG] pgmap v6380: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:59.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:58 smithi045 conmon[39718]: cluster 2021-07-31T04:12:57.882936+0000 mgr.x ( 2021-07-31T04:12:59.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:12:58 smithi045 conmon[39718]: mgr.24542) 6248 : cluster [DBG] pgmap v6380: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:12:59.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:58 smithi045 conmon[33655]: cluster 2021-07-31T04:12:57.882936+0000 mgr.x (mgr.24542) 2021-07-31T04:12:59.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:12:58 smithi045 conmon[33655]: 6248 : cluster [DBG] pgmap v6380: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:01.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:00 smithi154 conmon[34553]: cluster 2021-07-31T04:12:59.883338+0000 mgr.x (mgr.24542) 6249 : cluster [DBG] pgmap v6381: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:01.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:00 smithi045 conmon[39718]: cluster 2021-07-31T04:12:59.883338+0000 mgr.x (mgr. 2021-07-31T04:13:01.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:00 smithi045 conmon[39718]: 24542) 6249 : cluster [DBG] pgmap v6381: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:01.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:00 smithi045 conmon[33655]: cluster 2021-07-31T04:12: 2021-07-31T04:13:01.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:00 smithi045 conmon[33655]: 59.883338+0000 mgr.x (mgr.24542) 6249 : cluster [DBG] pgmap v6381: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:03.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:02 smithi154 conmon[34553]: cluster 2021-07-31T04:13:01.883728+0000 mgr.x (mgr.24542) 6250 : cluster [DBG] pgmap v6382: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:03.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:02 smithi154 conmon[34553]: 2021-07-31T04:13:03.274 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:02 smithi045 conmon[33655]: cluster 2021-07-31T04:13:01.883728+0000 mgr.x 2021-07-31T04:13:03.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:02 smithi045 conmon[33655]: (mgr.24542) 6250 : cluster [DBG] pgmap v6382: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:03.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:02 smithi045 conmon[39718]: cluster 2021-07-31T04:13:01.883728+0000 mgr.x (mgr.24542) 6250 : cluster [DBG] pgmap v6382: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:05.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:04 smithi154 conmon[34553]: cluster 2021-07-31T04:13:03.884329+0000 mgr.x (mgr.24542) 6251 : cluster [DBG] 2021-07-31T04:13:05.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:04 smithi154 conmon[34553]: pgmap v6383: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:05.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:04 smithi045 conmon[33655]: cluster 2021-07-31T04:13:03.884329+0000 mgr.x (mgr.24542) 6251 : cluster [DBG] 2021-07-31T04:13:05.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:04 smithi045 conmon[33655]: pgmap v6383: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:05.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:04 smithi045 conmon[39718]: cluster 2021-07-31T04:13:03.884329+0000 mgr.x (mgr.24542) 6251 : cluster [DBG] 2021-07-31T04:13:05.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:04 smithi045 conmon[39718]: pgmap v6383: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:07.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:06 smithi154 conmon[34553]: cluster 2021-07-31T04:13:05.884688+0000 mgr.x (mgr.24542) 6252 : cluster [DBG] pgmap v6384: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:07.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:06 smithi045 conmon[33655]: cluster 2021-07-31T04:13:05.884688+0000 mgr.x 2021-07-31T04:13:07.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:06 smithi045 conmon[33655]: (mgr.24542) 6252 : cluster [DBG] pgmap v6384: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:07.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:06 smithi045 conmon[39718]: cluster 2021-07-31T04:13:05.884688+0000 mgr.x (mgr.24542) 6252 : cluster [DBG] 2021-07-31T04:13:07.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:06 smithi045 conmon[39718]: pgmap v6384: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:09.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:08 smithi154 conmon[34553]: cluster 2021-07-31T04:13:07.885221+0000 mgr.x (mgr.24542) 6253 : cluster [DBG] pgmap v6385: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:09.274 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:08 smithi045 conmon[39718]: cluster 2021-07-31T04:13:07.885221+0000 mgr.x ( 2021-07-31T04:13:09.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:08 smithi045 conmon[39718]: mgr.24542) 6253 : cluster [DBG] pgmap v6385: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:09.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:08 smithi045 conmon[33655]: cluster 2021-07-31T 2021-07-31T04:13:09.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:08 smithi045 conmon[33655]: 04:13:07.885221+0000 mgr.x (mgr.24542) 6253 : cluster [DBG] pgmap v6385: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:11.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:10 smithi154 conmon[34553]: cluster 2021-07-31T04:13:09.885783+0000 mgr.x (mgr.24542) 6254 : cluster 2021-07-31T04:13:11.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:10 smithi154 conmon[34553]: [DBG] pgmap v6386: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:11.274 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:10 smithi045 conmon[39718]: cluster 2021-07-31T04:13: 2021-07-31T04:13:11.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:10 smithi045 conmon[39718]: 09.885783+0000 mgr.x (mgr.24542) 6254 : cluster [DBG] pgmap v6386: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:11.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:10 smithi045 conmon[33655]: cluster 2021-07-31T04:13:09.885783+0000 mgr.x (mgr.24542) 6254 : cluster [DBG] pgmap v6386: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:13.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:12 smithi154 conmon[34553]: cluster 2021-07-31T04:13:11.886327+0000 mgr.x (mgr.24542) 6255 : cluster 2021-07-31T04:13:13.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:12 smithi154 conmon[34553]: [DBG] pgmap v6387: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:13.274 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:12 smithi045 conmon[39718]: cluster 2021-07-31T04:13: 2021-07-31T04:13:13.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:12 smithi045 conmon[39718]: 11.886327+0000 mgr.x (mgr.24542) 6255 : cluster [DBG] pgmap v6387: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:13.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:12 smithi045 conmon[33655]: cluster 2021-07-31T04:13:11.886327+0000 mgr.x (mgr.24542) 6255 : cluster [DBG] pgmap v6387: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:15.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:14 smithi154 conmon[34553]: cluster 2021-07-31T04:13:13.886781+0000 mgr.x (mgr.24542) 6256 : cluster [DBG] pgmap v6388: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:15.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:14 smithi154 conmon[34553]: 2021-07-31T04:13:15.274 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:14 smithi045 conmon[33655]: cluster 2021-07-31T04:13:13.886781+0000 mgr.x (mgr.24542) 6256 : cluster [DBG] pgmap v6388: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:15.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:14 smithi045 conmon[39718]: cluster 2021-07-31T04:13:13.886781+0000 mgr.x (mgr 2021-07-31T04:13:15.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:14 smithi045 conmon[39718]: .24542) 6256 : cluster [DBG] pgmap v6388: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:17.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:16 smithi154 conmon[34553]: cluster 2021-07-31T04:13:15.887300+0000 mgr.x (mgr.24542) 6257 : cluster [DBG] pgmap v6389: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:17.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:16 smithi154 conmon[34553]: 2021-07-31T04:13:17.274 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:16 smithi045 conmon[33655]: cluster 2021-07-31T04:13:15.887300+0000 mgr.x (mgr.24542) 6257 2021-07-31T04:13:17.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:16 smithi045 conmon[33655]: : cluster [DBG] pgmap v6389: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:17.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:16 smithi045 conmon[39718]: cluster 2021-07-31T04:13:15.887300+0000 mgr.x (mgr. 2021-07-31T04:13:17.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:16 smithi045 conmon[39718]: 24542) 6257 : cluster [DBG] pgmap v6389: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:19.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:18 smithi154 conmon[34553]: cluster 2021-07-31T04:13:17.887757+0000 mgr.x (mgr.24542) 6258 : cluster [DBG] pgmap v6390: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:19.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:18 smithi154 conmon[34553]: 2021-07-31T04:13:19.274 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:18 smithi045 conmon[33655]: cluster 2021-07-31T04:13:17.887757+0000 mgr.x (mgr.24542) 6258 : cluster [DBG] pgmap v6390: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:19.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:18 smithi045 conmon[39718]: cluster 2021-07-31T04:13:17.887757+0000 mgr.x (mgr.24542 2021-07-31T04:13:19.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:18 smithi045 conmon[39718]: ) 6258 : cluster [DBG] pgmap v6390: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:20.184 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T04:13:20.465 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T04:13:20.465 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 2h ago 2h - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T04:13:20.466 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 2h ago 2h 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T04:13:20.466 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 2h ago 2h 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T04:13:20.466 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 2h ago 2h - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T04:13:20.466 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T04:13:20.467 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 2h ago 2h 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T04:13:20.467 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T04:13:20.467 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 2h ago 2h - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T04:13:20.467 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 2h ago 2h 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T04:13:20.467 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T04:13:20.467 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T04:13:20.468 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T04:13:20.468 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T04:13:20.468 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 2h ago 2h 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T04:13:20.468 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 2h ago 2h 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T04:13:20.468 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 2h ago 2h 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T04:13:20.469 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 2h ago 2h 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T04:13:20.469 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 2h ago 2h 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T04:13:20.800 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T04:13:20.800 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T04:13:20.801 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T04:13:20.801 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:13:20.801 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T04:13:20.801 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T04:13:20.802 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:13:20.802 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:13:20.802 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T04:13:20.802 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T04:13:20.802 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:13:20.803 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T04:13:20.803 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T04:13:20.803 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T04:13:20.803 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:13:20.803 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T04:13:20.804 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T04:13:21.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:21 smithi154 conmon[34553]: cluster 2021-07-31T04:13:19.888213+0000 mgr.x (mgr.24542) 6259 : cluster [DBG] pgmap v6391: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:21.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:21 smithi154 conmon[34553]: audit 2021-07-31T04:13:20.173129+0000 mgr.x (mgr.24542) 6260 : audit [DBG] from='client.31081 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:13:21.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:21 smithi154 conmon[34553]: audit 2021-07-31T04:13:20.801293+0000 mon.a (mon.0) 3441 : audit [DBG] from='client.? 172.21.15.45:0/3375840295' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:13:21.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:21 smithi045 conmon[33655]: cluster 2021-07-31T04:13:19.888213+0000 mgr.x ( 2021-07-31T04:13:21.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:21 smithi045 conmon[33655]: mgr.24542) 6259 : cluster [DBG] pgmap v6391: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:21.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:21 smithi045 conmon[33655]: audit 2021-07-31T04:13:20.173129+0000 mgr.x (mgr.24542) 6260 : audit [DBG] from='client.31081 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:13:21.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:21 smithi045 conmon[33655]: audit 2021-07-31T04:13:20.801293+0000 mon.a ( 2021-07-31T04:13:21.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:21 smithi045 conmon[33655]: mon.0) 3441 : audit [DBG] from='client.? 172.21.15.45:0/3375840295' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:13:21.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:21 smithi045 conmon[33655]: 2021-07-31T04:13:21.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:21 smithi045 conmon[39718]: cluster 2021-07-31T04:13:19.888213 2021-07-31T04:13:21.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:21 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 6259 : cluster [DBG] pgmap v6391: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:21.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:21 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T04:13:21.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:21 smithi045 conmon[39718]: T04:13:20.173129+0000 mgr.x (mgr.24542) 6260 : audit [DBG] from='client.31081 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:13:21.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:21 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T04:13:21.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:21 smithi045 conmon[39718]: 31T04:13:20.801293+0000 mon.a (mon.0) 3441 : audit 2021-07-31T04:13:21.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:21 smithi045 conmon[39718]: [DBG] from='client.? 172.21.15.45:0/3375840295' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:13:22.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:22 smithi045 conmon[33655]: audit 2021-07-31T04:13:20.458159+0000 2021-07-31T04:13:22.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:22 smithi045 conmon[33655]: mgr.x (mgr.24542) 6261 : audit [DBG] from='client.21150 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:13:22.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:22 smithi045 conmon[39718]: audit 2021-07-31T04:13:20.458159+0000 mgr.x 2021-07-31T04:13:22.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:22 smithi045 conmon[39718]: (mgr.24542) 6261 : audit [DBG] from='client.21150 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:13:22.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:22 smithi154 conmon[34553]: audit 2021-07-31T04:13:20.458159+0000 mgr.x (mgr.24542) 6261 : audit [DBG] from='client.21150 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:13:23.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:23 smithi045 conmon[33655]: cluster 2021-07-31T04:13:21.888674+0000 mgr.x (mgr.24542) 2021-07-31T04:13:23.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:23 smithi045 conmon[33655]: 6262 : cluster [DBG] pgmap v6392: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:23.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:23 smithi045 conmon[39718]: cluster 2021-07-31T04:13:21.888674+0000 mgr.x 2021-07-31T04:13:23.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:23 smithi045 conmon[39718]: (mgr.24542) 6262 : cluster [DBG] pgmap v6392: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:23.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:23 smithi154 conmon[34553]: cluster 2021-07-31T04:13:21.888674+0000 mgr.x (mgr.24542) 6262 : 2021-07-31T04:13:23.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:23 smithi154 conmon[34553]: cluster [DBG] pgmap v6392: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:25.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:25 smithi045 conmon[33655]: cluster 2021-07-31T04:13:23.889326+0000 mgr.x (mgr.24542) 2021-07-31T04:13:25.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:25 smithi045 conmon[33655]: 6263 : cluster [DBG] pgmap v6393: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:25.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:25 smithi045 conmon[39718]: cluster 2021-07-31T04:13:23.889326+0000 mgr.x 2021-07-31T04:13:25.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:25 smithi045 conmon[39718]: (mgr.24542) 6263 : cluster [DBG] pgmap v6393: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:25.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:25 smithi154 conmon[34553]: cluster 2021-07-31T04:13:23.889326+0000 mgr.x (mgr.24542) 6263 : cluster [DBG] pgmap v6393: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:27.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:27 smithi045 conmon[33655]: cluster 2021-07-31T04:13:25.889735+0000 mgr.x (mgr.24542) 6264 : cluster [DBG] pgmap v6394: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:27.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:27 smithi045 conmon[33655]: 2021-07-31T04:13:27.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:27 smithi045 conmon[39718]: cluster 2021-07-31T04:13:25.889735+0000 mgr.x (mgr 2021-07-31T04:13:27.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:27 smithi045 conmon[39718]: .24542) 6264 : cluster [DBG] pgmap v6394: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:27.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:27 smithi154 conmon[34553]: cluster 2021-07-31T04:13:25.889735+0000 mgr.x (mgr.24542) 6264 : cluster [DBG] pgmap v6394: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:27.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:27 smithi154 conmon[34553]: 2021-07-31T04:13:29.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:29 smithi045 conmon[33655]: cluster 2021-07-31T04:13:27. 2021-07-31T04:13:29.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:29 smithi045 conmon[33655]: 890136+0000 mgr.x (mgr.24542) 6265 : cluster [DBG] pgmap v6395: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:29.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:29 smithi045 conmon[33655]: audit 2021-07-31T04:13 2021-07-31T04:13:29.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:29 smithi045 conmon[33655]: :28.511807+0000 mon.b (mon.2) 805 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:13:29.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:29 smithi045 conmon[33655]: audit 2021-07-31T 2021-07-31T04:13:29.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:29 smithi045 conmon[33655]: 04:13:28.512181+0000 mon.b (mon.2) 806 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:13:29.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:29 smithi045 conmon[33655]: audit 2021-07-31T04: 2021-07-31T04:13:29.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:29 smithi045 conmon[33655]: 13:28.512723+0000 mon.a (mon.0) 3442 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:13:29.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:29 smithi045 conmon[33655]: 2021-07-31T04:13:29.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:29 smithi045 conmon[33655]: audit 2021-07-31T04:13 2021-07-31T04:13:29.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:29 smithi045 conmon[33655]: :28.513383+0000 mon.a (mon.0) 3443 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:13:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:29 smithi045 conmon[39718]: cluster 2021-07-31T04:13:27.890136+0000 mgr.x (mgr.24542) 6265 : cluster 2021-07-31T04:13:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:29 smithi045 conmon[39718]: [DBG] pgmap v6395: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:29 smithi045 conmon[39718]: audit 2021-07-31T04:13:28.511807+0000 mon.b (mon.2) 805 : audit [INF] 2021-07-31T04:13:29.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:29 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:13:29.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:29 smithi045 conmon[39718]: audit 2021-07-31T04 2021-07-31T04:13:29.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:29 smithi045 conmon[39718]: :13:28.512181+0000 mon.b (mon.2) 806 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:13:29.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:29 smithi045 conmon[39718]: audit 2021-07-31T04:13:29.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:29 smithi045 conmon[39718]: 2021-07-31T04:13:28. 2021-07-31T04:13:29.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:29 smithi045 conmon[39718]: 512723+0000 mon.a (mon.0) 3442 : audit [INF] 2021-07-31T04:13:29.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:29 smithi045 conmon[39718]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:13:29.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:29 smithi045 conmon[39718]: audit 2021-07-31T04: 2021-07-31T04:13:29.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:29 smithi045 conmon[39718]: 13:28.513383+0000 mon.a (mon.0) 3443 2021-07-31T04:13:29.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:29 smithi045 conmon[39718]: : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:13:29.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:29 smithi154 conmon[34553]: cluster 2021-07-31T04:13:27.890136+0000 mgr.x (mgr.24542) 6265 : cluster [DBG] pgmap v6395: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:29.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:29 smithi154 conmon[34553]: 2021-07-31T04:13:29.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:29 smithi154 conmon[34553]: audit 2021-07-31T04:13:28.511807+0000 mon.b (mon.2) 805 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:13:29.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:29 smithi154 conmon[34553]: audit 2021-07-31T04:13:28.512181+0000 mon.b (mon.2) 806 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:13:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:29 smithi154 conmon[34553]: audit 2021-07-31T04:13:28.512723+0000 mon.a (mon.0) 3442 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:13:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:29 smithi154 conmon[34553]: audit 2021-07-31T 2021-07-31T04:13:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:29 smithi154 conmon[34553]: 04:13:28.513383+0000 mon.a (mon.0) 3443 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:13:31.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:31 smithi045 conmon[39718]: cluster 2021-07-31T04:13:29.890709+0000 mgr.x (mgr.24542) 6266 2021-07-31T04:13:31.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:31 smithi045 conmon[39718]: : cluster [DBG] pgmap v6396: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:31.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:31 smithi045 conmon[33655]: cluster 2021-07-31T04:13:29.890709+0000 mgr.x (mgr.24542) 6266 2021-07-31T04:13:31.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:31 smithi045 conmon[33655]: : cluster [DBG] pgmap v6396: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:31.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:31 smithi154 conmon[34553]: cluster 2021-07-31T04:13:29.890709+0000 mgr.x (mgr.24542) 6266 : cluster [DBG] pgmap v6396: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:33.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:33 smithi154 conmon[34553]: cluster 2021-07-31T04:13:31.891193+0000 mgr.x (mgr.24542) 6267 : cluster [DBG] pgmap v6397: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:33.524 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:33 smithi045 conmon[39718]: cluster 2021-07-31T04:13:31.891193+0000 mgr.x (mgr.24542) 6267 : 2021-07-31T04:13:33.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:33 smithi045 conmon[39718]: cluster [DBG] pgmap v6397: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:33.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:33 smithi045 conmon[33655]: cluster 2021-07-31T04:13:31.891193+0000 mgr.x (mgr.24542) 6267 : cluster 2021-07-31T04:13:33.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:33 smithi045 conmon[33655]: [DBG] pgmap v6397: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:35.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:35 smithi154 conmon[34553]: cluster 2021-07-31T04:13:33.891636+0000 mgr.x (mgr.24542) 6268 : cluster [DBG] pgmap v6398: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:35.524 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:35 smithi045 conmon[39718]: cluster 2021-07-31T04:13:33.891636+0000 mgr.x (mgr.24542) 2021-07-31T04:13:35.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:35 smithi045 conmon[39718]: 6268 : cluster [DBG] pgmap v6398: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:35.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:35 smithi045 conmon[33655]: cluster 2021-07-31T04:13:33.891636+0000 mgr.x 2021-07-31T04:13:35.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:35 smithi045 conmon[33655]: (mgr.24542) 6268 : cluster [DBG] pgmap v6398: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:37.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[34553]: cluster 2021-07-31T04:13:35.892019+0000 mgr.x (mgr.24542) 6269 : cluster [DBG] pgmap v6399: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:37.524 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:37 smithi045 conmon[33655]: cluster 2021-07-31T04:13:35.892019+0000 2021-07-31T04:13:37.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:37 smithi045 conmon[33655]: mgr.x (mgr.24542) 6269 : cluster [DBG] pgmap v6399: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:37.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:37 smithi045 conmon[39718]: cluster 2021-07-31T04:13:35.892019+0000 2021-07-31T04:13:37.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:37 smithi045 conmon[39718]: mgr.x (mgr.24542) 6269 : cluster [DBG] pgmap v6399: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:38.002 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: debug 2021-07-31T04:13:37.610+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:13:38.003 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:13:38.003 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:13:38.003 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: yield 2021-07-31T04:13:38.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:13:38.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:13:38.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:13:38.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:13:38.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:13:38.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:13:38.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:13:38.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:13:38.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:13:38.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: task.result() 2021-07-31T04:13:38.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:13:38.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: debug 2021-07-31T04:13:37.622+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:13:38.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:13:38.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:13:38.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: yield 2021-07-31T04:13:38.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:13:38.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:13:38.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:13:38.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:13:38.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:13:38.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:13:38.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:13:38.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:13:38.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:13:38.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: task.result() 2021-07-31T04:13:38.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:13:38.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: debug 2021-07-31T04:13:37.646+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:13:38.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:13:38.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:13:38.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: yield 2021-07-31T04:13:38.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:13:38.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:13:38.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:13:38.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:13:38.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:13:38.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:13:38.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:13:38.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:13:38.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:13:38.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: task.result() 2021-07-31T04:13:38.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:13:38.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: debug 2021-07-31T04:13:37.647+0000 7fcb9295f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:13:38.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:13:38.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:13:38.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: yield 2021-07-31T04:13:38.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:13:38.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:13:38.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:13:38.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:13:38.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:13:38.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:13:38.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:13:38.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:13:38.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:13:38.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: task.result() 2021-07-31T04:13:38.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:13:38.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: 2021-07-31T04:13:38.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T04:13:38.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: 2021-07-31T04:13:38.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:13:38.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:13:38.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: return f(*arg) 2021-07-31T04:13:38.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:13:38.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:13:38.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:13:38.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:13:38.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:13:38.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:13:38.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:13:38.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T04:13:38.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:13:38.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: raise self._exception 2021-07-31T04:13:38.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:13:38.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:13:38.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:13:38.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T04:13:38.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:13:38.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: raise 2021-07-31T04:13:38.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:13:38.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T04:13:38.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:13:38.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:13:38.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:13:38.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: debug 2021-07-31T04:13:37.656+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:13:38.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:13:38.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:13:38.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: yield 2021-07-31T04:13:38.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:13:38.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:13:38.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:13:38.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:13:38.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:13:38.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:13:38.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:13:38.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:13:38.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:13:38.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: task.result() 2021-07-31T04:13:38.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:13:38.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: debug 2021-07-31T04:13:37.656+0000 7fcb93160700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:13:38.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:13:38.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:13:38.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: yield 2021-07-31T04:13:38.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:13:38.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:13:38.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:13:38.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:13:38.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:13:38.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:13:38.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:13:38.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:13:38.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:13:38.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: task.result() 2021-07-31T04:13:38.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:13:38.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: 2021-07-31T04:13:38.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T04:13:38.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: 2021-07-31T04:13:38.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:13:38.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:13:38.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: return f(*arg) 2021-07-31T04:13:38.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:13:38.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:13:38.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:13:38.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:13:38.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:13:38.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:13:38.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:13:38.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T04:13:38.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:13:38.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: raise self._exception 2021-07-31T04:13:38.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:13:38.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:13:38.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:13:38.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T04:13:38.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:13:38.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: raise 2021-07-31T04:13:38.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:13:38.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T04:13:38.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:13:38.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:13:38.034 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:13:37 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:13:38.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:38 smithi154 conmon[34553]: audit 2021-07-31T04:13:37.564643+0000 mon.b (mon.2) 807 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T04:13:38.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:38 smithi154 conmon[34553]: audit 2021-07-31T04:13:37.566160+0000 mon.b (mon 2021-07-31T04:13:38.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:38 smithi154 conmon[34553]: .2) 808 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T04:13:38.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:38 smithi154 conmon[34553]: audit 2021-07-31T04:13:37.567255+0000 mon.b (mon.2) 809 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T04:13:38.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:38 smithi045 conmon[33655]: audit 2021-07-31T04:13:37.564643+0000 mon.b (mon. 2021-07-31T04:13:38.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:38 smithi045 conmon[33655]: 2) 807 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T04:13:38.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:38 smithi045 conmon[33655]: audit 2021-07-31T04:13:37.566160+0000 mon.b (mon.2) 808 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T04:13:38.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:38 smithi045 conmon[33655]: audit 2021-07-31T04:13:38.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:38 smithi045 conmon[33655]: 2021-07-31T04:13 2021-07-31T04:13:38.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:38 smithi045 conmon[33655]: :37.567255+0000 mon.b ( 2021-07-31T04:13:38.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:38 smithi045 conmon[33655]: mon.2) 809 : audit 2021-07-31T04:13:38.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:38 smithi045 conmon[33655]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T04:13:38.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:38 smithi045 conmon[39718]: audit 2021-07-31T04:13:37 2021-07-31T04:13:38.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:38 smithi045 conmon[39718]: .564643+0000 mon.b (mon.2) 807 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T04:13:38.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:38 smithi045 conmon[39718]: audit 2021- 2021-07-31T04:13:38.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:38 smithi045 conmon[39718]: 07-31T04:13:37.566160 2021-07-31T04:13:38.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:38 smithi045 conmon[39718]: +0000 mon.b (mon.2) 808 2021-07-31T04:13:38.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:38 smithi045 conmon[39718]: : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T04:13:38.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:38 smithi045 conmon[39718]: audit 2021 2021-07-31T04:13:38.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:38 smithi045 conmon[39718]: -07-31T04:13:37. 2021-07-31T04:13:38.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:38 smithi045 conmon[39718]: 567255+0000 mon.b (mon.2 2021-07-31T04:13:38.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:38 smithi045 conmon[39718]: ) 809 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T04:13:38.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:38 smithi045 conmon[39718]: 2021-07-31T04:13:39.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:13:37.567997+0000 mgr.x (mgr.24542) 6270 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T04:13:39.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:13:37.611692+0000 mgr.x (mgr.24542) 6271 : cephadm [ERR] 2021-07-31T04:13:39.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:13:39.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:13:39.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: yield 2021-07-31T04:13:39.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:13:39.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:13:39.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:13:39.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:13:39.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:13:39.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:13:39.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:13:39.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:13:39.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:13:39.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: task.result() 2021-07-31T04:13:39.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:13:39.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:13:37.612137+0000 mgr.x (mgr.24542) 6272 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T04:13:39.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: cephadm 2021-07 2021-07-31T04:13:39.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: -31T04:13:37.623064+0000 mgr.x (mgr.24542) 6273 : cephadm [ERR] 2021-07-31T04:13:39.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:13:39.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:13:39.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: yield 2021-07-31T04:13:39.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:13:39.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:13:39.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:13:39.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:13:39.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:13:39.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:13:39.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:13:39.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:13:39.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:13:39.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: task.result() 2021-07-31T04:13:39.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:13:39.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:13:37.623873+0000 mgr.x (mgr.24542) 6274 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T04:13:39.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:13:37.647814+0000 mgr.x (mgr.24542) 6275 : cephadm [ERR] 2021-07-31T04:13:39.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:13:39.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:13:39.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: yield 2021-07-31T04:13:39.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:13:39.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:13:39.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:13:39.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:13:39.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:13:39.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:13:39.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:13:39.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:13:39.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:13:39.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: task.result() 2021-07-31T04:13:39.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:13:39.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:13:37.648405+0000 mgr.x (mgr.24542) 6276 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:13:39.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:13:39.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:13:39.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: yield 2021-07-31T04:13:39.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:13:39.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:13:39.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:13:39.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:13:39.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:13:39.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:13:39.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:13:39.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:13:39.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:13:39.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: task.result() 2021-07-31T04:13:39.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:13:39.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: 2021-07-31T04:13:39.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T04:13:39.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: 2021-07-31T04:13:39.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:13:39.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:13:39.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: return f(*arg) 2021-07-31T04:13:39.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:13:39.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:13:39.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:13:39.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:13:39.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:13:39.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:13:39.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:13:39.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T04:13:39.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:13:39.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: raise self._exception 2021-07-31T04:13:39.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:13:39.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:13:39.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:13:39.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T04:13:39.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:13:39.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: raise 2021-07-31T04:13:39.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:13:39.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T04:13:39.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:13:39.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:13:39.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:13:39.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:13:37.656933+0000 mgr.x (mgr.24542) 6277 : cephadm [ERR] 2021-07-31T04:13:39.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:13:39.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:13:39.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: yield 2021-07-31T04:13:39.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:13:39.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:13:39.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:13:39.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:13:39.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:13:39.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:13:39.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:13:39.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:13:39.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:13:39.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: task.result() 2021-07-31T04:13:39.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:13:39.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:13:37.657477+0000 mgr.x (mgr.24542) 6278 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:13:39.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:13:39.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:13:39.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: yield 2021-07-31T04:13:39.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:13:39.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:13:39.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:13:39.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:13:39.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:13:39.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:13:39.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:13:39.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:13:39.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:13:39.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: task.result() 2021-07-31T04:13:39.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:13:39.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: 2021-07-31T04:13:39.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T04:13:39.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: 2021-07-31T04:13:39.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:13:39.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:13:39.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: return f(*arg) 2021-07-31T04:13:39.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:13:39.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:13:39.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:13:39.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:13:39.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:13:39.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:13:39.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:13:39.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T04:13:39.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:13:39.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: raise self._exception 2021-07-31T04:13:39.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:13:39.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:13:39.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:13:39.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T04:13:39.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:13:39.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: raise 2021-07-31T04:13:39.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:13:39.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T04:13:39.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:13:39.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:13:39.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:13:39.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:39 smithi154 conmon[34553]: cluster 2021-07-31T04:13:37.892388+0000 mgr.x (mgr.24542) 6279 : cluster [DBG] pgmap v6400: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:39.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: cephadm 2021-07-31T04:13:37.567997+0000 mgr.x (mgr.24542) 6270 : cephadm [WRN] 2021-07-31T04:13:39.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T04:13:39.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: cephadm 2021-07-31T04:13:37.611692+0000 mgr.x (mgr.24542) 6271 : cephadm [ERR] 2021-07-31T04:13:39.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:13:39.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:13:39.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: yield 2021-07-31T04:13:39.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:13:39.539 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:13:39.539 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:13:39.539 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:13:39.539 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:13:39.540 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:13:39.540 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:13:39.540 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:13:39.540 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:13:39.540 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: task.result() 2021-07-31T04:13:39.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:13:39.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: cephadm 2021-07-31T04:13:37.612137+0000 mgr.x (mgr.24542) 6272 : cephadm 2021-07-31T04:13:39.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T04:13:39.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: cephadm 2021-07-31T04:13:37.623064+0000 mgr.x (mgr.24542) 6273 : cephadm [ERR] 2021-07-31T04:13:39.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:13:39.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:13:39.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: yield 2021-07-31T04:13:39.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:13:39.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:13:39.543 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:13:39.543 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:13:39.543 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:13:39.543 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:13:39.544 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:13:39.544 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:13:39.544 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:13:39.544 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: task.result() 2021-07-31T04:13:39.545 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:13:39.545 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: cephadm 2021-07-31T04:13:37.623873+0000 mgr.x (mgr.24542) 6274 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T04:13:39.545 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: cephadm 2021-07-31T04:13:39.545 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: 2021-07-31T04:13:37.647814+0000 mgr.x (mgr.24542 2021-07-31T04:13:39.545 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: ) 6275 : cephadm [ERR] 2021-07-31T04:13:39.546 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:13:39.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: cephadm 2021-07-31T04:13:37.567997+0000 mgr.x (mgr.24542) 6270 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T04:13:39.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: cephadm 2021-07-31T04:13:37.611692+0000 mgr.x (mgr 2021-07-31T04:13:39.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: .24542) 6271 : cephadm [ERR] 2021-07-31T04:13:39.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:13:39.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:13:39.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: yield 2021-07-31T04:13:39.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:13:39.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:13:39.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:13:39.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:13:39.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:13:39.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:13:39.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:13:39.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:13:39.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:13:39.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: task.result() 2021-07-31T04:13:39.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:13:39.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: cephadm 2021-07-31T04:13:37.612137+0000 mgr.x (mgr.24542) 6272 : cephadm 2021-07-31T04:13:39.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T04:13:39.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: cephadm 2021-07-31T 2021-07-31T04:13:39.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: 04:13:37.623064+0000 mgr.x 2021-07-31T04:13:39.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: (mgr.24542) 6273 : cephadm 2021-07-31T04:13:39.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: [ERR] 2021-07-31T04:13:39.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:13:39.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:13:39.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: yield 2021-07-31T04:13:39.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:13:39.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:13:39.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:13:39.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:13:39.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:13:39.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:13:39.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:13:39.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:13:39.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:13:39.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: task.result() 2021-07-31T04:13:39.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:13:39.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: cephadm 2021-07-31T04:13:37.623873+0000 mgr.x (mgr.24542) 6274 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T04:13:39.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: cephadm 2021- 2021-07-31T04:13:39.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: 07-31T04:13:37.647814 2021-07-31T04:13:39.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542 2021-07-31T04:13:39.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: ) 6275 : cephadm [ERR] 2021-07-31T04:13:39.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:13:39.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:13:39.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: yield 2021-07-31T04:13:39.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:13:39.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:13:39.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:13:39.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:13:39.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:13:39.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:13:39.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:13:39.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:13:39.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:13:39.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: task.result() 2021-07-31T04:13:39.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:13:39.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: cephadm 2021-07-31T04:13 2021-07-31T04:13:39.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: :37.648405+0000 mgr.x (mgr 2021-07-31T04:13:39.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: .24542) 6276 : cephadm [ERR] 2021-07-31T04:13:39.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:13:39.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:13:39.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:13:39.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: yield 2021-07-31T04:13:39.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:13:39.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:13:39.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:13:39.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:13:39.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:13:39.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:13:39.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:13:39.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:13:39.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:13:39.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: task.result() 2021-07-31T04:13:39.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:13:39.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: 2021-07-31T04:13:39.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T04:13:39.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: 2021-07-31T04:13:39.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:13:39.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:13:39.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: return f(*arg) 2021-07-31T04:13:39.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:13:39.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:13:39.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:13:39.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:13:39.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:13:39.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: yield 2021-07-31T04:13:39.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:13:39.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:13:39.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:13:39.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:13:39.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:13:39.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:13:39.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:13:39.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:13:39.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:13:39.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: task.result() 2021-07-31T04:13:39.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:13:39.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: cephadm 2021-07-31T04:13:37.648405+0000 mgr.x (mgr.24542) 2021-07-31T04:13:39.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: 6276 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:13:39.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:13:39.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:13:39.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: yield 2021-07-31T04:13:39.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:13:39.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:13:39.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:13:39.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:13:39.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:13:39.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:13:39.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:13:39.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:13:39.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:13:39.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: task.result() 2021-07-31T04:13:39.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:13:39.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: 2021-07-31T04:13:39.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T04:13:39.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: 2021-07-31T04:13:39.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:13:39.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:13:39.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: return f(*arg) 2021-07-31T04:13:39.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:13:39.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:13:39.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:13:39.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:13:39.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:13:39.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:13:39.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:13:39.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:13:39.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:13:39.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T04:13:39.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:13:39.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: raise self._exception 2021-07-31T04:13:39.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:13:39.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:13:39.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:13:39.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T04:13:39.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:13:39.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: raise 2021-07-31T04:13:39.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:13:39.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T04:13:39.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:13:39.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:13:39.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:13:39.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: cephadm 2021-07-31T04:13:37.656933+0000 mgr.x (mgr.24542) 6277 : cephadm [ERR] 2021-07-31T04:13:39.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:13:39.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:13:39.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: yield 2021-07-31T04:13:39.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:13:39.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:13:39.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:13:39.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:13:39.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:13:39.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:13:39.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:13:39.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:13:39.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:13:39.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: task.result() 2021-07-31T04:13:39.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:13:39.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: cephadm 2021-07-31T04:13:37.657477+0000 mgr.x (mgr.24542) 6278 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:13:39.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:13:39.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:13:39.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: yield 2021-07-31T04:13:39.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:13:39.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:13:39.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:13:39.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T04:13:39.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:13:39.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: raise self._exception 2021-07-31T04:13:39.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:13:39.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:13:39.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:13:39.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T04:13:39.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:13:39.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: raise 2021-07-31T04:13:39.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:13:39.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T04:13:39.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:13:39.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:13:39.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:13:39.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: cephadm 2021-07-31T04:13:37.656933+0000 mgr.x (mgr.24542) 6277 : cephadm [ERR] 2021-07-31T04:13:39.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:13:39.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:13:39.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: yield 2021-07-31T04:13:39.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:13:39.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:13:39.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:13:39.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:13:39.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:13:39.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:13:39.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:13:39.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:13:39.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:13:39.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: task.result() 2021-07-31T04:13:39.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:13:39.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: cephadm 2021-07-31T04:13:37.657477+0000 mgr.x (mgr.24542) 6278 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:13:39.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:13:39.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:13:39.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: yield 2021-07-31T04:13:39.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:13:39.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:13:39.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:13:39.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:13:39.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:13:39.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:13:39.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:13:39.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:13:39.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:13:39.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:13:39.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:13:39.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:13:39.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: task.result() 2021-07-31T04:13:39.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:13:39.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: 2021-07-31T04:13:39.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T04:13:39.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: 2021-07-31T04:13:39.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:13:39.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:13:39.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: return f(*arg) 2021-07-31T04:13:39.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:13:39.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:13:39.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:13:39.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:13:39.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:13:39.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:13:39.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:13:39.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T04:13:39.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:13:39.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: raise self._exception 2021-07-31T04:13:39.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:13:39.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:13:39.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:13:39.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T04:13:39.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:13:39.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: raise 2021-07-31T04:13:39.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:13:39.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T04:13:39.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:13:39.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:13:39.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:13:39.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: 2021-07-31T04:13:39.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[39718]: cluster 2021-07-31T04:13:37.892388+0000 mgr.x (mgr.24542) 6279 : cluster [DBG] pgmap v6400: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:39.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:13:39.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:13:39.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:13:39.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:13:39.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: task.result() 2021-07-31T04:13:39.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:13:39.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: 2021-07-31T04:13:39.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T04:13:39.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: 2021-07-31T04:13:39.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:13:39.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:13:39.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: return f(*arg) 2021-07-31T04:13:39.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:13:39.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:13:39.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:13:39.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:13:39.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:13:39.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:13:39.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:13:39.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T04:13:39.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:13:39.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: raise self._exception 2021-07-31T04:13:39.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:13:39.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:13:39.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:13:39.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T04:13:39.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:13:39.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: raise 2021-07-31T04:13:39.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:13:39.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T04:13:39.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:13:39.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:13:39.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:13:39.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: cluster 2021-07-31T04:13:37.892388 2021-07-31T04:13:39.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:39 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 6279 : cluster [DBG] pgmap v6400: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:41.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:41 smithi154 conmon[34553]: cluster 2021-07-31T04:13:39.892828+0000 mgr.x (mgr.24542) 6280 : cluster [DBG] pgmap v6401: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:41.524 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:41 smithi045 conmon[39718]: cluster 2021-07-31T04:13:39.892828+0000 mgr.x (mgr.24542) 6280 : cluster [DBG] pgmap v6401: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:41.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:41 smithi045 conmon[33655]: cluster 2021-07-31 2021-07-31T04:13:41.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:41 smithi045 conmon[33655]: T04:13:39.892828+0000 mgr.x (mgr.24542) 6280 : cluster [DBG] pgmap v6401: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:43.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:43 smithi154 conmon[34553]: cluster 2021-07-31T04:13:41.893310+0000 mgr.x (mgr.24542) 6281 : cluster [DBG] pgmap v6402: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:43.524 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:43 smithi045 conmon[39718]: cluster 2021-07-31T04:13:41.893310+0000 mgr.x (mgr 2021-07-31T04:13:43.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:43 smithi045 conmon[39718]: .24542) 6281 : cluster [DBG] pgmap v6402: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:43.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:43 smithi045 conmon[33655]: cluster 2021-07-31T04:13:41.893310+0000 mgr.x (mgr.24542) 6281 : cluster 2021-07-31T04:13:43.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:43 smithi045 conmon[33655]: [DBG] pgmap v6402: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:45.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:45 smithi154 conmon[34553]: cluster 2021-07-31T04:13:43.893791+0000 mgr.x (mgr.24542) 6282 : cluster [DBG] pgmap v6403: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:45.524 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:45 smithi045 conmon[33655]: cluster 2021-07-31T04:13:43.893791+0000 mgr.x (mgr.24542) 6282 2021-07-31T04:13:45.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:45 smithi045 conmon[33655]: : cluster [DBG] pgmap v6403: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:45.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:45 smithi045 conmon[39718]: cluster 2021-07-31T04:13:43.893791+0000 mgr.x ( 2021-07-31T04:13:45.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:45 smithi045 conmon[39718]: mgr.24542) 6282 : cluster [DBG] pgmap v6403: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:47.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:47 smithi154 conmon[34553]: cluster 2021-07-31T04:13:45.894404+0000 mgr.x (mgr.24542) 6283 : cluster [DBG] pgmap v6404: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:47.524 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:47 smithi045 conmon[33655]: cluster 2021-07-31T04:13:45.894404+0000 mgr.x (mgr.24542 2021-07-31T04:13:47.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:47 smithi045 conmon[33655]: ) 6283 : cluster [DBG] pgmap v6404: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:47.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:47 smithi045 conmon[39718]: cluster 2021-07-31T04:13:45.894404+0000 mgr.x (mgr.24542) 6283 : cluster [DBG] 2021-07-31T04:13:47.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:47 smithi045 conmon[39718]: pgmap v6404: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:49.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:49 smithi154 conmon[34553]: cluster 2021-07-31T04:13:47.894782+0000 mgr.x (mgr.24542) 6284 : cluster [DBG] pgmap v6405: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:49.524 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:49 smithi045 conmon[33655]: cluster 2021-07-31T04:13:47.894782+0000 mgr.x (mgr.24542) 6284 2021-07-31T04:13:49.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:49 smithi045 conmon[33655]: : cluster [DBG] pgmap v6405: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:49.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:49 smithi045 conmon[39718]: cluster 2021-07-31T04:13:47.894782+0000 mgr.x (mgr.24542) 6284 : cluster [DBG] pgmap v6405: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:51.114 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T04:13:51.394 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T04:13:51.394 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 2h ago 2h - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T04:13:51.394 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 2h ago 2h 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T04:13:51.395 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 2h ago 2h 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T04:13:51.395 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 2h ago 2h - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T04:13:51.395 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T04:13:51.395 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 2h ago 2h 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T04:13:51.395 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T04:13:51.396 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 2h ago 2h - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T04:13:51.396 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 2h ago 2h 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T04:13:51.396 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T04:13:51.396 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T04:13:51.396 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T04:13:51.397 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T04:13:51.397 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 2h ago 2h 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T04:13:51.397 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 2h ago 2h 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T04:13:51.397 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 2h ago 2h 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T04:13:51.397 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 2h ago 2h 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T04:13:51.398 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 2h ago 2h 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T04:13:51.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:51 smithi154 conmon[34553]: cluster 2021-07-31T04:13:49.895200+0000 mgr.x (mgr.24542) 6285 : cluster [DBG] pgmap v6406: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:51.524 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:51 smithi045 conmon[39718]: cluster 2021-07-31T04:13:49.895200+0000 2021-07-31T04:13:51.524 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:51 smithi045 conmon[39718]: mgr.x (mgr.24542) 6285 : cluster [DBG] pgmap v6406: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:51.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:51 smithi045 conmon[33655]: cluster 2021-07-31T04:13:49.895200+0000 mgr.x (mgr.24542) 6285 : cluster 2021-07-31T04:13:51.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:51 smithi045 conmon[33655]: [DBG] pgmap v6406: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:51.730 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T04:13:51.730 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T04:13:51.731 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T04:13:51.731 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:13:51.731 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T04:13:51.731 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T04:13:51.732 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:13:51.732 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:13:51.732 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T04:13:51.732 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T04:13:51.732 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:13:51.733 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T04:13:51.733 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T04:13:51.733 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T04:13:51.733 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:13:51.733 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T04:13:51.734 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T04:13:52.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:52 smithi154 conmon[34553]: audit 2021-07-31T04:13:51.102974+0000 mgr.x (mgr.24542) 6286 : audit [DBG] from='client.21162 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:13:52.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:52 smithi154 conmon[34553]: audit 2021-07-31T04:13:51.387306+0000 2021-07-31T04:13:52.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:52 smithi154 conmon[34553]: mgr.x (mgr.24542) 6287 : audit [DBG] from='client.21168 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:13:52.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:52 smithi154 conmon[34553]: audit 2021-07-31T04:13:51.731368+0000 mon.a (mon.0) 3444 : audit [DBG] from='client.? 172.21.15.45:0/2364229788' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:13:52.524 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:52 smithi045 conmon[33655]: audit 2021-07-31T04:13:51.102974+0000 mgr.x (mgr.24542 2021-07-31T04:13:52.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:52 smithi045 conmon[33655]: ) 6286 : audit [DBG] from='client.21162 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:13:52.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:52 smithi045 conmon[33655]: audit 2021-07-31T04:13:51.387306+0000 mgr.x (mgr.24542) 6287 : audit [DBG] from='client.21168 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:13:52.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:52 smithi045 conmon[33655]: audit 2021-07-31T04:13: 2021-07-31T04:13:52.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:52 smithi045 conmon[33655]: 51.731368+0000 mon.a (mon. 2021-07-31T04:13:52.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:52 smithi045 conmon[33655]: 0) 3444 : audit [DBG] from='client.? 172.21.15.45:0/2364229788' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:13:52.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:52 smithi045 conmon[39718]: audit 2021-07-31T04:13:51.102974+0000 mgr.x ( 2021-07-31T04:13:52.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:52 smithi045 conmon[39718]: mgr.24542) 6286 : audit [DBG] from='client.21162 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:13:52.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:52 smithi045 conmon[39718]: audit 2021-07-31T04: 2021-07-31T04:13:52.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:52 smithi045 conmon[39718]: 13:51.387306+0000 mgr.x (mgr.24542) 6287 : audit [DBG] from='client.21168 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:13:52.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:52 smithi045 conmon[39718]: audit 2021-07-31T04:13:51.731368+0000 mon.a (mon.0 2021-07-31T04:13:52.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:52 smithi045 conmon[39718]: ) 3444 : audit [DBG] from='client.? 172.21.15.45:0/2364229788' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:13:53.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:53 smithi154 conmon[34553]: cluster 2021-07-31T04:13:51.895565+0000 mgr.x (mgr.24542) 6288 : cluster [DBG] pgmap v6407: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:53.524 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:53 smithi045 conmon[33655]: cluster 2021-07-31T04:13:51.895565+0000 mgr.x (mgr.24542) 6288 2021-07-31T04:13:53.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:53 smithi045 conmon[33655]: : cluster [DBG] pgmap v6407: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:53.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:53 smithi045 conmon[39718]: cluster 2021-07-31T04:13:51.895565+0000 mgr.x (mgr. 2021-07-31T04:13:53.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:53 smithi045 conmon[39718]: 24542) 6288 : cluster [DBG] pgmap v6407: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:55.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:55 smithi154 conmon[34553]: cluster 2021-07-31T04:13:53.896122+0000 mgr.x (mgr.24542) 6289 : cluster [DBG] pgmap v6408: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:55.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:55 smithi045 conmon[33655]: cluster 2021-07-31T04:13:53.896122+0000 mgr.x (mgr.24542) 6289 : cluster [DBG] pgmap v6408: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:55.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:55 smithi045 conmon[39718]: cluster 2021-07-31T04:13:53.896122+0000 mgr.x ( 2021-07-31T04:13:55.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:55 smithi045 conmon[39718]: mgr.24542) 6289 : cluster [DBG] pgmap v6408: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:57.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:57 smithi154 conmon[34553]: cluster 2021-07-31T04:13:55.896625+0000 mgr.x (mgr.24542) 6290 : cluster [DBG] pgmap v6409: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:57.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:57 smithi045 conmon[33655]: cluster 2021-07-31T04:13:55.896625+0000 mgr.x (mgr.24542 2021-07-31T04:13:57.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:57 smithi045 conmon[33655]: ) 6290 : cluster [DBG] pgmap v6409: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:57.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:57 smithi045 conmon[39718]: cluster 2021-07-31T04:13:55.896625+0000 mgr.x 2021-07-31T04:13:57.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:57 smithi045 conmon[39718]: (mgr.24542) 6290 : cluster [DBG] pgmap v6409: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:59.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:13:59 smithi154 conmon[34553]: cluster 2021-07-31T04:13:57.897005+0000 mgr.x (mgr.24542) 6291 : cluster [DBG] pgmap v6410: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:59.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:59 smithi045 conmon[39718]: cluster 2021-07-31T04:13:57.897005+0000 mgr.x (mgr 2021-07-31T04:13:59.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:13:59 smithi045 conmon[39718]: .24542) 6291 : cluster [DBG] pgmap v6410: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:13:59.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:59 smithi045 conmon[33655]: cluster 2021-07-31T04:13:57.897005+0000 mgr.x (mgr.24542) 6291 : cluster 2021-07-31T04:13:59.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:13:59 smithi045 conmon[33655]: [DBG] pgmap v6410: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:01.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:01 smithi154 conmon[34553]: cluster 2021-07-31T04:13:59.897437+0000 mgr.x (mgr.24542) 6292 : cluster [DBG] pgmap v6411: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:01.524 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:01 smithi045 conmon[33655]: cluster 2021-07-31T04:13:59.897437+0000 2021-07-31T04:14:01.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:01 smithi045 conmon[33655]: mgr.x (mgr.24542) 6292 : cluster [DBG] pgmap v6411: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:01.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:01 smithi045 conmon[39718]: cluster 2021-07-31T04:13:59.897437+0000 2021-07-31T04:14:01.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:01 smithi045 conmon[39718]: mgr.x (mgr.24542) 6292 : cluster [DBG] pgmap v6411: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:03.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:03 smithi154 conmon[34553]: cluster 2021-07-31T04:14:01.897977+0000 mgr.x (mgr.24542) 6293 : cluster [DBG] pgmap v6412: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:03.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:03 smithi045 conmon[39718]: cluster 2021-07-31T04:14:01.897977+0000 mgr.x 2021-07-31T04:14:03.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:03 smithi045 conmon[39718]: (mgr.24542) 6293 : cluster [DBG] pgmap v6412: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:03.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:03 smithi045 conmon[33655]: cluster 2021-07-31T04:14:01.897977+0000 mgr.x (mgr 2021-07-31T04:14:03.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:03 smithi045 conmon[33655]: .24542) 6293 : cluster [DBG] pgmap v6412: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:05.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:05 smithi154 conmon[34553]: cluster 2021-07-31T04:14:03.898411+0000 mgr.x (mgr.24542) 6294 : cluster [DBG] pgmap v6413: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:05.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:05 smithi045 conmon[33655]: cluster 2021-07-31T04:14:03.898411+0000 mgr.x (mgr.24542) 6294 2021-07-31T04:14:05.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:05 smithi045 conmon[33655]: : cluster [DBG] pgmap v6413: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:05.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:05 smithi045 conmon[39718]: cluster 2021-07-31T04:14:03.898411+0000 2021-07-31T04:14:05.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:05 smithi045 conmon[39718]: mgr.x (mgr.24542) 6294 : cluster [DBG] pgmap v6413: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:07.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:07 smithi154 conmon[34553]: cluster 2021-07-31T04:14:05.898942+0000 mgr.x (mgr.24542) 6295 : cluster [DBG] pgmap v6414: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:07.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:07 smithi045 conmon[39718]: cluster 2021-07-31T04:14:05.898942+0000 mgr.x (mgr.24542) 2021-07-31T04:14:07.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:07 smithi045 conmon[39718]: 6295 : cluster [DBG] pgmap v6414: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:07.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:07 smithi045 conmon[33655]: cluster 2021-07-31T04:14:05.898942+0000 mgr.x (mgr.24542) 6295 : cluster [DBG] 2021-07-31T04:14:07.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:07 smithi045 conmon[33655]: pgmap v6414: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:09.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:09 smithi154 conmon[34553]: cluster 2021-07-31T04:14:07.899325+0000 mgr.x (mgr.24542) 6296 : cluster [DBG] pgmap v6415: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:09.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:09 smithi045 conmon[33655]: cluster 2021-07-31T04:14:07.899325+0000 mgr.x (mgr.24542 2021-07-31T04:14:09.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:09 smithi045 conmon[33655]: ) 6296 : cluster [DBG] pgmap v6415: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:09.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:09 smithi045 conmon[39718]: cluster 2021-07-31T04:14: 2021-07-31T04:14:09.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:09 smithi045 conmon[39718]: 07.899325+0000 mgr.x (mgr.24542) 6296 : cluster [DBG] pgmap v6415: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:11.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:11 smithi154 conmon[34553]: cluster 2021-07-31T04:14:09.899668+0000 mgr.x (mgr.24542) 6297 : cluster [DBG] pgmap v6416: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:11.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:11 smithi045 conmon[39718]: cluster 2021-07-31T04:14: 2021-07-31T04:14:11.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:11 smithi045 conmon[39718]: 09.899668+0000 mgr.x (mgr.24542) 6297 : cluster [DBG] pgmap v6416: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:11.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:11 smithi045 conmon[33655]: cluster 2021-07-31T04:14:09.899668+0000 mgr.x ( 2021-07-31T04:14:11.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:11 smithi045 conmon[33655]: mgr.24542) 6297 : cluster [DBG] pgmap v6416: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:13.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:13 smithi154 conmon[34553]: cluster 2021-07-31T04:14:11.900087+0000 mgr.x (mgr.24542) 6298 : cluster [DBG] pgmap v6417: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:13.524 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:13 smithi045 conmon[39718]: cluster 2021-07-31T04:14:11.900087+0000 mgr.x (mgr.24542) 6298 : cluster [DBG] pgmap v6417: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:13.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:13 smithi045 conmon[39718]: 2021-07-31T04:14:13.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:13 smithi045 conmon[33655]: cluster 2021-07-31T04:14:11.900087+0000 mgr.x (mgr.24542) 6298 2021-07-31T04:14:13.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:13 smithi045 conmon[33655]: : cluster [DBG] pgmap v6417: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:15.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:15 smithi154 conmon[34553]: cluster 2021-07-31T04:14:13.900734+0000 mgr.x (mgr.24542) 6299 : cluster [DBG] pgmap v6418: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:15.524 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:15 smithi045 conmon[39718]: cluster 2021-07-31T04:14:13 2021-07-31T04:14:15.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:15 smithi045 conmon[39718]: .900734+0000 mgr.x (mgr.24542) 6299 : cluster [DBG] pgmap v6418: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:15.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:15 smithi045 conmon[33655]: cluster 2021-07-31T04:14:13.900734+0000 mgr.x (mgr.24542 2021-07-31T04:14:15.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:15 smithi045 conmon[33655]: ) 6299 : cluster [DBG] pgmap v6418: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:17.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:17 smithi154 conmon[34553]: cluster 2021-07-31T04:14:15.901322+0000 mgr.x (mgr.24542) 6300 : cluster [DBG] pgmap v6419: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:17.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:17 smithi045 conmon[39718]: cluster 2021-07-31T04:14:15.901322+0000 mgr.x (mgr.24542) 6300 : cluster [DBG] pgmap v6419: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:17.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:17 smithi045 conmon[33655]: cluster 2021-07-31T04:14:15.901322+0000 mgr.x (mgr.24542) 6300 : 2021-07-31T04:14:17.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:17 smithi045 conmon[33655]: cluster [DBG] pgmap v6419: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:19.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:19 smithi154 conmon[34553]: cluster 2021-07-31T04:14:17.901774+0000 mgr.x (mgr.24542) 6301 : cluster [DBG] pgmap v6420: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:19.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:19 smithi154 conmon[34553]: 2021-07-31T04:14:19.524 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:19 smithi045 conmon[33655]: cluster 2021-07-31T04:14:17.901774+0000 mgr.x (mgr.24542) 6301 : cluster [DBG] 2021-07-31T04:14:19.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:19 smithi045 conmon[33655]: pgmap v6420: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:19.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:19 smithi045 conmon[39718]: cluster 2021-07-31T04:14:17.901774+0000 mgr.x ( 2021-07-31T04:14:19.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:19 smithi045 conmon[39718]: mgr.24542) 6301 : cluster [DBG] pgmap v6420: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:21.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:21 smithi154 conmon[34553]: cluster 2021-07-31T04:14:19.902378+0000 mgr.x (mgr.24542) 6302 : cluster [DBG] pgmap v6421: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:21.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:21 smithi154 conmon[34553]: 2021-07-31T04:14:21.524 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:21 smithi045 conmon[33655]: cluster 2021-07-31T04:14:19.902378+0000 mgr.x (mgr 2021-07-31T04:14:21.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:21 smithi045 conmon[33655]: .24542) 6302 : cluster [DBG] pgmap v6421: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:21.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:21 smithi045 conmon[39718]: cluster 2021-07-31T04:14:19.902378+0000 mgr.x (mgr.24542) 6302 : cluster [DBG] pgmap v6421: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:22.045 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T04:14:22.322 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T04:14:22.323 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 2h ago 2h - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T04:14:22.323 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 2h ago 2h 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T04:14:22.323 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 2h ago 2h 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T04:14:22.323 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 2h ago 2h - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T04:14:22.323 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T04:14:22.324 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 2h ago 2h 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T04:14:22.324 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T04:14:22.324 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 2h ago 2h - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T04:14:22.324 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 2h ago 2h 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T04:14:22.324 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T04:14:22.325 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T04:14:22.325 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T04:14:22.325 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T04:14:22.325 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 2h ago 2h 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T04:14:22.326 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 2h ago 2h 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T04:14:22.326 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 2h ago 2h 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T04:14:22.326 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 2h ago 2h 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T04:14:22.326 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 2h ago 2h 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T04:14:22.658 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T04:14:22.658 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T04:14:22.658 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T04:14:22.659 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:14:22.659 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T04:14:22.659 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T04:14:22.659 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:14:22.659 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:14:22.660 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T04:14:22.660 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T04:14:22.660 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:14:22.660 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T04:14:22.660 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T04:14:22.661 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T04:14:22.661 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:14:22.661 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T04:14:22.661 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T04:14:23.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:23 smithi154 conmon[34553]: cluster 2021-07-31T04:14:21.902955+0000 mgr.x (mgr.24542) 6303 : cluster [DBG] pgmap v6422: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:23.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:23 smithi154 conmon[34553]: audit 2021-07-31T04:14: 2021-07-31T04:14:23.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:23 smithi154 conmon[34553]: 22.033961+0000 mgr.x (mgr.24542) 6304 : audit [DBG] from='client.21180 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:14:23.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:23 smithi154 conmon[34553]: audit 2021-07-31T04:14:22.315407+0000 mgr.x (mgr.24542) 6305 : audit [DBG] from='client.21186 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:14:23.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:23 smithi154 conmon[34553]: audit 2021-07-31T04:14:22.659287 2021-07-31T04:14:23.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:23 smithi154 conmon[34553]: +0000 mon.c (mon.1) 258 : audit [DBG] from='client.? 172.21.15.45:0/2777549591' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:14:23.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:23 smithi045 conmon[39718]: cluster 2021-07-31T04:14:21.902955+0000 mgr.x 2021-07-31T04:14:23.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:23 smithi045 conmon[39718]: (mgr.24542) 6303 : cluster [DBG] pgmap v6422: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:23.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:23 smithi045 conmon[39718]: audit 2021 2021-07-31T04:14:23.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:23 smithi045 conmon[39718]: -07-31T04:14:22.033961+0000 2021-07-31T04:14:23.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:23 smithi045 conmon[39718]: mgr.x (mgr.24542) 6304 : audit 2021-07-31T04:14:23.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:23 smithi045 conmon[39718]: [DBG] from='client.21180 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:14:23.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:23 smithi045 conmon[39718]: audit 2021 2021-07-31T04:14:23.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:23 smithi045 conmon[39718]: -07-31T04:14 2021-07-31T04:14:23.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:23 smithi045 conmon[39718]: :22.315407+0000 mgr.x (mgr. 2021-07-31T04:14:23.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:23 smithi045 conmon[39718]: 24542) 6305 : audit [DBG] from='client.21186 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:14:23.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:23 smithi045 conmon[39718]: audit 2021-07-31T04:14:22.659287+0000 mon.c (mon.1 2021-07-31T04:14:23.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:23 smithi045 conmon[39718]: ) 258 : audit [DBG] from='client.? 172.21.15.45:0/2777549591' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:14:23.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:23 smithi045 conmon[33655]: cluster 2021-07-31T04:14:21.902955+0000 mgr.x (mgr 2021-07-31T04:14:23.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:23 smithi045 conmon[33655]: .24542) 6303 : cluster [DBG] pgmap v6422: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:23.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:23 smithi045 conmon[33655]: audit 2021-07-31T04:14:22 2021-07-31T04:14:23.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:23 smithi045 conmon[33655]: .033961+0000 mgr.x (mgr.24542) 6304 : audit [DBG] from='client.21180 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:14:23.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:23 smithi045 conmon[33655]: 2021-07-31T04:14:23.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:23 smithi045 conmon[33655]: audit 2021-07- 2021-07-31T04:14:23.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:23 smithi045 conmon[33655]: 31T04:14:22.315407+0000 2021-07-31T04:14:23.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:23 smithi045 conmon[33655]: mgr.x (mgr.24542) 6305 : audit 2021-07-31T04:14:23.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:23 smithi045 conmon[33655]: [DBG] from='client.21186 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:14:23.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:23 smithi045 conmon[33655]: audit 2021 2021-07-31T04:14:23.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:23 smithi045 conmon[33655]: -07-31T04:14:22. 2021-07-31T04:14:23.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:23 smithi045 conmon[33655]: 659287+0000 mon.c (mon.1) 2021-07-31T04:14:23.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:23 smithi045 conmon[33655]: 258 : audit [DBG] from='client.? 172.21.15.45:0/2777549591' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:14:25.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:25 smithi154 conmon[34553]: cluster 2021-07-31T04:14:23.903486+0000 mgr.x (mgr.24542) 6306 : cluster [DBG] pgmap v6423: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:25.524 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:25 smithi045 conmon[33655]: cluster 2021-07-31T04:14:23.903486+0000 mgr.x (mgr.24542 2021-07-31T04:14:25.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:25 smithi045 conmon[33655]: ) 6306 : cluster [DBG] pgmap v6423: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:25.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:25 smithi045 conmon[39718]: cluster 2021-07-31T04:14: 2021-07-31T04:14:25.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:25 smithi045 conmon[39718]: 23.903486+0000 mgr.x (mgr.24542) 6306 : cluster [DBG] pgmap v6423: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:27.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:27 smithi154 conmon[34553]: cluster 2021-07-31T04:14:25.904022+0000 mgr.x (mgr.24542) 6307 : cluster [DBG] pgmap v6424: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:27.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:27 smithi045 conmon[33655]: cluster 2021-07-31T04:14:25.904022+0000 mgr.x (mgr. 2021-07-31T04:14:27.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:27 smithi045 conmon[33655]: 24542) 6307 : cluster [DBG] pgmap v6424: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:27.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:27 smithi045 conmon[39718]: cluster 2021-07-31T04:14: 2021-07-31T04:14:27.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:27 smithi045 conmon[39718]: 25.904022+0000 mgr.x (mgr.24542) 6307 : cluster [DBG] pgmap v6424: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:29.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:29 smithi154 conmon[34553]: cluster 2021-07-31T04:14:27.904439+0000 mgr.x (mgr.24542) 6308 : cluster [DBG] pgmap v6425: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:29.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:29 smithi154 conmon[34553]: audit 2021-07-31T04:14:28.515623+0000 mon.b (mon.2) 810 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:14:29.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:29 smithi154 conmon[34553]: audit 2021-07-31T04:14:28.515853+0000 mon.b (mon.2) 811 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:14:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:29 smithi154 conmon[34553]: audit 2021-07-31T04:14:28.516599+0000 mon.a 2021-07-31T04:14:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:29 smithi154 conmon[34553]: (mon.0) 3445 : audit [INF] 2021-07-31T04:14:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:29 smithi154 conmon[34553]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:14:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:29 smithi154 conmon[34553]: audit 2021-07-31T 2021-07-31T04:14:29.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:29 smithi154 conmon[34553]: 04:14:28.516930+0000 mon.a (mon.0 2021-07-31T04:14:29.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:29 smithi154 conmon[34553]: ) 3446 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:14:29.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:29 smithi045 conmon[33655]: cluster 2021-07-31T04:14:27.904439+0000 mgr.x (mgr.24542) 6308 : cluster 2021-07-31T04:14:29.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:29 smithi045 conmon[33655]: [DBG] pgmap v6425: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:29.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:29 smithi045 conmon[33655]: audit 2021-07-31T04:14:28.515623+0000 mon.b (mon.2) 810 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:14:29.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:29 smithi045 conmon[33655]: audit 2021-07-31T04:14: 2021-07-31T04:14:29.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:29 smithi045 conmon[33655]: 28.515853+0000 mon.b (mon. 2021-07-31T04:14:29.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:29 smithi045 conmon[33655]: 2) 811 : audit [INF] 2021-07-31T04:14:29.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:29 smithi045 conmon[33655]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:14:29.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:29 smithi045 conmon[33655]: audit 2021-07-31T04:14:29.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:29 smithi045 conmon[33655]: 2021-07-31T04 2021-07-31T04:14:29.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:29 smithi045 conmon[33655]: :14:28.516599 2021-07-31T04:14:29.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:29 smithi045 conmon[33655]: +0000 mon.a (mon.0 2021-07-31T04:14:29.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:29 smithi045 conmon[33655]: ) 3445 : audit [INF] 2021-07-31T04:14:29.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:29 smithi045 conmon[33655]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:14:29.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:29 smithi045 conmon[33655]: audit 2021-07-31T04:14:29.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:29 smithi045 conmon[33655]: 2021-07-31T04:14: 2021-07-31T04:14:29.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:29 smithi045 conmon[33655]: 28.516930+0000 mon.a (mon 2021-07-31T04:14:29.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:29 smithi045 conmon[33655]: .0) 3446 : audit [INF] 2021-07-31T04:14:29.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:29 smithi045 conmon[33655]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:14:29.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:29 smithi045 conmon[39718]: cluster 2021-07-31T04:14:27.904439 2021-07-31T04:14:29.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:29 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 6308 : cluster [DBG] pgmap v6425: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:29.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:29 smithi045 conmon[39718]: audit 2021-07-31T04:14:29.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:29 smithi045 conmon[39718]: 2021-07-31T04:14 2021-07-31T04:14:29.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:29 smithi045 conmon[39718]: :28.515623+0000 mon.b (mon 2021-07-31T04:14:29.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:29 smithi045 conmon[39718]: .2) 810 : audit [INF] 2021-07-31T04:14:29.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:29 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:14:29.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:29 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T04:14:29.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:29 smithi045 conmon[39718]: 31T04:14:28.515853+0000 2021-07-31T04:14:29.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:29 smithi045 conmon[39718]: mon.b (mon.2) 811 : audit 2021-07-31T04:14:29.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:29 smithi045 conmon[39718]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:14:29.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:29 smithi045 conmon[39718]: audit 2021 2021-07-31T04:14:29.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:29 smithi045 conmon[39718]: -07-31T04:14:28 2021-07-31T04:14:29.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:29 smithi045 conmon[39718]: .516599+0000 mon.a (mon. 2021-07-31T04:14:29.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:29 smithi045 conmon[39718]: 0) 3445 : audit [INF] 2021-07-31T04:14:29.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:29 smithi045 conmon[39718]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:14:29.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:29 smithi045 conmon[39718]: audit 2021 2021-07-31T04:14:29.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:29 smithi045 conmon[39718]: -07-31T04:14:28. 2021-07-31T04:14:29.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:29 smithi045 conmon[39718]: 516930+0000 mon.a (mon. 2021-07-31T04:14:29.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:29 smithi045 conmon[39718]: 0) 3446 : audit [INF] 2021-07-31T04:14:29.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:29 smithi045 conmon[39718]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:14:31.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:31 smithi154 conmon[34553]: cluster 2021-07-31T04:14:29.904892+0000 mgr.x (mgr.24542) 6309 : cluster [DBG] pgmap v6426: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:31.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:31 smithi045 conmon[39718]: cluster 2021-07-31T04:14:29.904892+0000 mgr.x (mgr.24542) 6309 : cluster [DBG] pgmap v6426: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:31.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:31 smithi045 conmon[33655]: cluster 2021-07-31T04:14:29.904892 2021-07-31T04:14:31.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:31 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 6309 : cluster [DBG] pgmap v6426: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:33.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:33 smithi154 conmon[34553]: cluster 2021-07-31T04:14:31.905439+0000 mgr.x (mgr.24542) 6310 : cluster [DBG] pgmap v6427: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:33.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:33 smithi045 conmon[39718]: cluster 2021-07-31T04:14:31.905439+0000 mgr.x (mgr.24542) 6310 : cluster [DBG] pgmap v6427: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:33.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:33 smithi045 conmon[33655]: cluster 2021-07-31T04:14:31. 2021-07-31T04:14:33.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:33 smithi045 conmon[33655]: 905439+0000 mgr.x (mgr.24542) 6310 : cluster [DBG] pgmap v6427: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:35.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:35 smithi154 conmon[34553]: cluster 2021-07-31T04:14:33.905963+0000 mgr.x (mgr.24542) 6311 : cluster [DBG] pgmap v6428: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:35.524 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:35 smithi045 conmon[39718]: cluster 2021-07-31T04:14:33.905963+0000 mgr.x (mgr.24542) 6311 : cluster [DBG] pgmap v6428: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:35.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:35 smithi045 conmon[33655]: cluster 2021-07-31T04:14:33.905963+0000 mgr.x (mgr.24542) 6311 2021-07-31T04:14:35.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:35 smithi045 conmon[33655]: : cluster [DBG] pgmap v6428: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:37.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[34553]: cluster 2021-07-31T04:14:35.906308+0000 mgr.x (mgr.24542) 6312 : cluster [DBG] pgmap v6429: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:37.524 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:37 smithi045 conmon[39718]: cluster 2021-07-31T04:14:35.906308+0000 2021-07-31T04:14:37.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:37 smithi045 conmon[39718]: mgr.x (mgr.24542) 6312 : cluster [DBG] pgmap v6429: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:37.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:37 smithi045 conmon[33655]: cluster 2021-07-31T04:14:35.906308+0000 mgr.x (mgr.24542) 6312 : 2021-07-31T04:14:37.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:37 smithi045 conmon[33655]: cluster [DBG] pgmap v6429: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:38.002 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: debug 2021-07-31T04:14:37.703+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:14:38.003 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:14:38.003 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:14:38.003 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: yield 2021-07-31T04:14:38.003 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:14:38.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:14:38.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:14:38.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:14:38.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:14:38.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:14:38.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:14:38.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:14:38.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:14:38.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: task.result() 2021-07-31T04:14:38.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:14:38.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: debug 2021-07-31T04:14:37.715+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:14:38.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:14:38.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:14:38.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: yield 2021-07-31T04:14:38.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:14:38.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:14:38.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:14:38.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:14:38.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:14:38.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:14:38.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:14:38.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:14:38.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:14:38.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: task.result() 2021-07-31T04:14:38.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:14:38.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: debug 2021-07-31T04:14:37.742+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:14:38.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:14:38.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:14:38.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: yield 2021-07-31T04:14:38.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:14:38.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:14:38.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:14:38.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:14:38.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:14:38.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:14:38.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:14:38.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:14:38.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:14:38.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: task.result() 2021-07-31T04:14:38.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:14:38.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: debug 2021-07-31T04:14:37.742+0000 7fcb9215e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:14:38.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:14:38.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:14:38.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: yield 2021-07-31T04:14:38.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:14:38.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:14:38.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:14:38.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:14:38.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:14:38.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:14:38.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:14:38.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:14:38.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:14:38.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: task.result() 2021-07-31T04:14:38.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:14:38.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: 2021-07-31T04:14:38.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T04:14:38.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: 2021-07-31T04:14:38.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:14:38.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:14:38.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: return f(*arg) 2021-07-31T04:14:38.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:14:38.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:14:38.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:14:38.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:14:38.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:14:38.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:14:38.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:14:38.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T04:14:38.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:14:38.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: raise self._exception 2021-07-31T04:14:38.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:14:38.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:14:38.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:14:38.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T04:14:38.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:14:38.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: raise 2021-07-31T04:14:38.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:14:38.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T04:14:38.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:14:38.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:14:38.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:14:38.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: debug 2021-07-31T04:14:37.749+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:14:38.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:14:38.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:14:38.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: yield 2021-07-31T04:14:38.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:14:38.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:14:38.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:14:38.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:14:38.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:14:38.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:14:38.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:14:38.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:14:38.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:14:38.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: task.result() 2021-07-31T04:14:38.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:14:38.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: debug 2021-07-31T04:14:37.750+0000 7fcb9195d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:14:38.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:14:38.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:14:38.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: yield 2021-07-31T04:14:38.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:14:38.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:14:38.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:14:38.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:14:38.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:14:38.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:14:38.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:14:38.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:14:38.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:14:38.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: task.result() 2021-07-31T04:14:38.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:14:38.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: 2021-07-31T04:14:38.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T04:14:38.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: 2021-07-31T04:14:38.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:14:38.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:14:38.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: return f(*arg) 2021-07-31T04:14:38.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:14:38.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:14:38.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:14:38.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:14:38.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:14:38.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:14:38.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:14:38.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T04:14:38.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:14:38.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: raise self._exception 2021-07-31T04:14:38.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:14:38.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:14:38.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:14:38.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T04:14:38.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:14:38.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: raise 2021-07-31T04:14:38.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:14:38.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T04:14:38.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:14:38.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:14:38.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:14:37 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:14:38.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:38 smithi154 conmon[34553]: audit 2021-07-31T04:14:37.658387+0000 mon.b (mon.2) 812 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T04:14:38.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:38 smithi154 conmon[34553]: audit 2021-07-31T04:14:37.659894+0000 mon.b (mon.2) 813 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T04:14:38.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:38 smithi154 conmon[34553]: audit 2021-07-31T 2021-07-31T04:14:38.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:38 smithi154 conmon[34553]: 04:14:37.660942+0000 mon.b (mon.2) 814 : audit 2021-07-31T04:14:38.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:38 smithi154 conmon[34553]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T04:14:38.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:38 smithi045 conmon[39718]: audit 2021-07-31T04:14:37.658387+0000 mon.b ( 2021-07-31T04:14:38.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:38 smithi045 conmon[39718]: mon.2) 812 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T04:14:38.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:38 smithi045 conmon[39718]: audit 2021-07-31T04:14:37.659894+0000 mon.b (mon.2) 813 2021-07-31T04:14:38.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:38 smithi045 conmon[39718]: : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T04:14:38.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:38 smithi045 conmon[39718]: audit 2021-07-31T04:14:37.660942 2021-07-31T04:14:38.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:38 smithi045 conmon[39718]: +0000 mon.b (mon.2) 814 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T04:14:38.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:38 smithi045 conmon[33655]: audit 2021-07-31T04:14:37.658387+0000 mon.b (mon. 2021-07-31T04:14:38.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:38 smithi045 conmon[33655]: 2) 812 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T04:14:38.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:38 smithi045 conmon[33655]: audit 2021-07-31T04:14:37.659894+0000 mon.b (mon.2) 813 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T04:14:38.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:38 smithi045 conmon[33655]: audit 2021-07-31T04:14:37.660942+0000 mon.b (mon.2) 814 : audit [INF] 2021-07-31T04:14:38.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:38 smithi045 conmon[33655]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T04:14:39.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:14:37.661726+0000 mgr.x (mgr.24542) 6313 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T04:14:39.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:14:37.704673+0000 mgr.x (mgr.24542) 6314 : cephadm [ERR] 2021-07-31T04:14:39.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:14:39.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:14:39.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: yield 2021-07-31T04:14:39.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:14:39.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:14:39.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:14:39.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:14:39.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:14:39.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:14:39.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:14:39.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:14:39.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:14:39.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: task.result() 2021-07-31T04:14:39.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:14:39.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:14:37.705165+0000 mgr.x (mgr.24542) 6315 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T04:14:39.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: cephadm 2021-07-31T04 2021-07-31T04:14:39.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: :14:37.716135+0000 mgr.x (mgr.24542) 6316 : cephadm [ERR] 2021-07-31T04:14:39.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:14:39.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:14:39.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: yield 2021-07-31T04:14:39.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:14:39.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:14:39.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:14:39.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:14:39.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:14:39.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:14:39.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:14:39.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:14:39.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:14:39.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: task.result() 2021-07-31T04:14:39.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:14:39.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:14:37.716635+0000 mgr.x (mgr.24542) 6317 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T04:14:39.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:14:37.743080+0000 mgr.x (mgr.24542) 6318 : cephadm [ERR] 2021-07-31T04:14:39.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:14:39.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:14:39.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: yield 2021-07-31T04:14:39.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:14:39.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:14:39.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:14:39.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:14:39.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:14:39.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:14:39.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:14:39.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:14:39.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:14:39.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: task.result() 2021-07-31T04:14:39.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:14:39.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:14:37.743513+0000 mgr.x (mgr.24542) 6319 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:14:39.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:14:39.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:14:39.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: yield 2021-07-31T04:14:39.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:14:39.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:14:39.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:14:39.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:14:39.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:14:39.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:14:39.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:14:39.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:14:39.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:14:39.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: task.result() 2021-07-31T04:14:39.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:14:39.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: 2021-07-31T04:14:39.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T04:14:39.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: 2021-07-31T04:14:39.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:14:39.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:14:39.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: return f(*arg) 2021-07-31T04:14:39.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:14:39.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:14:39.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:14:39.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:14:39.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:14:39.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:14:39.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:14:39.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T04:14:39.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:14:39.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: raise self._exception 2021-07-31T04:14:39.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:14:39.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:14:39.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:14:39.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T04:14:39.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:14:39.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: raise 2021-07-31T04:14:39.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:14:39.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T04:14:39.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:14:39.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:14:39.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:14:39.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:14:37.750860+0000 mgr.x (mgr.24542) 6320 : cephadm [ERR] 2021-07-31T04:14:39.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:14:39.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:14:39.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: yield 2021-07-31T04:14:39.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:14:39.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:14:39.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:14:39.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:14:39.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:14:39.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:14:39.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:14:39.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:14:39.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:14:39.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: task.result() 2021-07-31T04:14:39.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:14:39.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:14:37.751303+0000 mgr.x (mgr.24542) 6321 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:14:39.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:14:39.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:14:39.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: yield 2021-07-31T04:14:39.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:14:39.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:14:39.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:14:39.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:14:39.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:14:39.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:14:39.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:14:39.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:14:39.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:14:39.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: task.result() 2021-07-31T04:14:39.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:14:39.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: 2021-07-31T04:14:39.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T04:14:39.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: 2021-07-31T04:14:39.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:14:39.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:14:39.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: return f(*arg) 2021-07-31T04:14:39.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:14:39.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:14:39.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:14:39.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:14:39.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:14:39.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:14:39.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:14:39.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T04:14:39.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:14:39.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: raise self._exception 2021-07-31T04:14:39.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:14:39.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:14:39.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:14:39.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T04:14:39.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:14:39.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: raise 2021-07-31T04:14:39.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:14:39.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T04:14:39.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:14:39.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:14:39.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:14:39.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:39 smithi154 conmon[34553]: cluster 2021-07-31T04:14:37.906801+0000 mgr.x (mgr.24542) 6322 : cluster [DBG] pgmap v6430: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:39.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: cephadm 2021-07-31T04:14:37.661726+0000 mgr.x 2021-07-31T04:14:39.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: (mgr.24542) 6313 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T04:14:39.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: cephadm 2021-07-31T04:14:37.704673 2021-07-31T04:14:39.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 6314 : cephadm [ERR] 2021-07-31T04:14:39.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:14:39.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:14:39.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: yield 2021-07-31T04:14:39.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:14:39.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:14:39.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:14:39.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:14:39.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:14:39.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:14:39.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:14:39.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:14:39.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:14:39.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: task.result() 2021-07-31T04:14:39.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:14:39.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: cephadm 2021-07-31T04:14:37.705165+0000 mgr.x (mgr.24542) 6315 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T04:14:39.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: cephadm 2021-07-31T04:14:37.716135+0000 mgr.x (mgr.24542) 6316 : cephadm [ERR] 2021-07-31T04:14:39.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:14:39.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:14:39.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: yield 2021-07-31T04:14:39.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:14:39.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:14:39.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:14:39.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:14:39.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:14:39.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:14:39.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:14:39.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:14:39.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:14:39.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: task.result() 2021-07-31T04:14:39.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:14:39.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: cephadm 2021-07-31T04:14:37.716635+0000 mgr.x (mgr.24542) 6317 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T04:14:39.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: cephadm 2021-07-31T04:14:37.743080+0000 mgr.x (mgr.24542) 6318 : cephadm 2021-07-31T04:14:39.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: [ERR] 2021-07-31T04:14:39.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:14:39.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:14:39.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: yield 2021-07-31T04:14:39.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:14:39.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:14:39.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:14:39.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:14:39.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:14:39.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:14:39.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:14:39.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:14:39.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:14:39.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: task.result() 2021-07-31T04:14:39.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:14:39.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: cephadm 2021-07-31T04:14:37.743513+0000 mgr.x (mgr.24542) 6319 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:14:39.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:14:39.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:14:39.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: yield 2021-07-31T04:14:39.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:14:39.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:14:39.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:14:39.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:14:39.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:14:39.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:14:39.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:14:39.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:14:39.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:14:39.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: task.result() 2021-07-31T04:14:39.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:14:39.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: 2021-07-31T04:14:39.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T04:14:39.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: 2021-07-31T04:14:39.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:14:39.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:14:39.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: return f(*arg) 2021-07-31T04:14:39.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:14:39.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:14:39.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:14:39.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:14:39.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:14:39.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:14:39.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:14:39.550 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: cephadm 2021-07-31T04:14:37.661726+0000 mgr.x (mgr.24542) 6313 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T04:14:39.550 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: cephadm 2021-07-31T04:14:37.704673+0000 mgr.x (mgr.24542) 6314 : cephadm [ERR] 2021-07-31T04:14:39.550 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:14:39.551 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:14:39.551 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: yield 2021-07-31T04:14:39.551 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:14:39.551 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:14:39.552 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:14:39.552 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:14:39.552 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:14:39.552 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:14:39.552 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:14:39.553 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:14:39.553 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:14:39.553 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: task.result() 2021-07-31T04:14:39.553 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:14:39.553 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: cephadm 2021-07-31T04:14:37 2021-07-31T04:14:39.554 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: .705165+0000 mgr.x (mgr.24542) 6315 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T04:14:39.554 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: cephadm 2021-07-31T04:14:37.716135+0000 mgr.x (mgr.24542) 6316 : cephadm [ERR] 2021-07-31T04:14:39.554 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:14:39.554 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:14:39.554 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: yield 2021-07-31T04:14:39.555 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:14:39.555 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:14:39.555 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:14:39.555 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:14:39.555 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:14:39.556 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:14:39.556 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:14:39.556 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:14:39.556 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:14:39.556 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: task.result() 2021-07-31T04:14:39.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:14:39.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: cephadm 2021-07-31T04:14:37.716635+0000 mgr.x (mgr.24542) 6317 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T04:14:39.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: cephadm 2021-07-31T04:14: 2021-07-31T04:14:39.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: 37.743080+0000 mgr.x (mgr.24542) 6318 : cephadm [ERR] 2021-07-31T04:14:39.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:14:39.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:14:39.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T04:14:39.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:14:39.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: raise self._exception 2021-07-31T04:14:39.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:14:39.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:14:39.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:14:39.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T04:14:39.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:14:39.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: raise 2021-07-31T04:14:39.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:14:39.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T04:14:39.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:14:39.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:14:39.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:14:39.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: cephadm 2021-07-31T04:14 2021-07-31T04:14:39.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: :37.750860+0000 mgr.x (mgr.24542) 6320 : cephadm [ERR] 2021-07-31T04:14:39.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:14:39.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:14:39.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: yield 2021-07-31T04:14:39.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:14:39.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:14:39.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:14:39.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:14:39.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:14:39.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:14:39.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:14:39.563 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:14:39.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:14:39.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: task.result() 2021-07-31T04:14:39.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:14:39.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: cephadm 2021-07-31T04:14:37.751303+0000 mgr.x (mgr.24542) 6321 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:14:39.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:14:39.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:14:39.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: yield 2021-07-31T04:14:39.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:14:39.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:14:39.565 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:14:39.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:14:39.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:14:39.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:14:39.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: yield 2021-07-31T04:14:39.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:14:39.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:14:39.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:14:39.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:14:39.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:14:39.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:14:39.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:14:39.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:14:39.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:14:39.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: task.result() 2021-07-31T04:14:39.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:14:39.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: cephadm 2021-07-31T04:14:37.743513+0000 mgr.x (mgr.24542) 6319 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:14:39.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:14:39.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:14:39.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: yield 2021-07-31T04:14:39.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:14:39.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:14:39.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:14:39.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:14:39.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:14:39.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:14:39.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:14:39.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:14:39.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:14:39.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: task.result() 2021-07-31T04:14:39.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:14:39.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: 2021-07-31T04:14:39.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T04:14:39.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: 2021-07-31T04:14:39.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:14:39.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:14:39.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: return f(*arg) 2021-07-31T04:14:39.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:14:39.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:14:39.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:14:39.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:14:39.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:14:39.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:14:39.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:14:39.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T04:14:39.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:14:39.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:14:39.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:14:39.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: task.result() 2021-07-31T04:14:39.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:14:39.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: 2021-07-31T04:14:39.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T04:14:39.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: 2021-07-31T04:14:39.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:14:39.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:14:39.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: return f(*arg) 2021-07-31T04:14:39.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:14:39.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:14:39.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:14:39.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:14:39.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:14:39.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:14:39.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:14:39.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T04:14:39.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:14:39.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: raise self._exception 2021-07-31T04:14:39.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:14:39.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:14:39.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:14:39.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T04:14:39.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:14:39.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: raise 2021-07-31T04:14:39.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:14:39.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T04:14:39.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:14:39.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:14:39.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:14:39.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[39718]: cluster 2021-07-31T04:14:37.906801+0000 mgr.x (mgr.24542) 6322 : cluster [DBG] pgmap v6430: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:39.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:14:39.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: raise self._exception 2021-07-31T04:14:39.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:14:39.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:14:39.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:14:39.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T04:14:39.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:14:39.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: raise 2021-07-31T04:14:39.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:14:39.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T04:14:39.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:14:39.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:14:39.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:14:39.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: cephadm 2021-07-31T04:14:37.750860+0000 mgr.x (mgr.24542) 6320 : cephadm [ERR] 2021-07-31T04:14:39.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:14:39.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:14:39.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: yield 2021-07-31T04:14:39.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:14:39.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:14:39.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:14:39.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:14:39.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:14:39.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:14:39.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:14:39.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:14:39.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:14:39.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: task.result() 2021-07-31T04:14:39.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:14:39.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: cephadm 2021-07-31T04: 2021-07-31T04:14:39.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: 14:37.751303+0000 mgr.x (mgr.24542) 6321 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:14:39.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:14:39.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:14:39.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: yield 2021-07-31T04:14:39.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:14:39.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:14:39.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:14:39.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:14:39.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:14:39.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:14:39.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:14:39.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:14:39.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:14:39.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: task.result() 2021-07-31T04:14:39.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:14:39.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: 2021-07-31T04:14:39.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T04:14:39.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: 2021-07-31T04:14:39.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:14:39.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:14:39.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: return f(*arg) 2021-07-31T04:14:39.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:14:39.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:14:39.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:14:39.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:14:39.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:14:39.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:14:39.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:14:39.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T04:14:39.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:14:39.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: raise self._exception 2021-07-31T04:14:39.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:14:39.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:14:39.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:14:39.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T04:14:39.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:14:39.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: raise 2021-07-31T04:14:39.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:14:39.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T04:14:39.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:14:39.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:14:39.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:14:39.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:39 smithi045 conmon[33655]: cluster 2021-07-31T04:14:37.906801+0000 mgr.x (mgr.24542) 6322 : cluster [DBG] pgmap v6430: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:41.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:41 smithi154 conmon[34553]: cluster 2021-07-31T04:14:39.907324+0000 mgr.x (mgr.24542) 6323 : cluster [DBG] pgmap v6431: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:41.524 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:41 smithi045 conmon[39718]: cluster 2021-07-31T04:14:39.907324+0000 mgr.x (mgr. 2021-07-31T04:14:41.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:41 smithi045 conmon[39718]: 24542) 6323 : cluster [DBG] pgmap v6431: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:41.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:41 smithi045 conmon[33655]: cluster 2021-07-31T04:14:39.907324+0000 mgr.x (mgr.24542) 6323 : 2021-07-31T04:14:41.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:41 smithi045 conmon[33655]: cluster [DBG] pgmap v6431: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:43.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:43 smithi154 conmon[34553]: cluster 2021-07-31T04:14:41.907676+0000 mgr.x (mgr.24542) 6324 2021-07-31T04:14:43.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:43 smithi154 conmon[34553]: : cluster [DBG] pgmap v6432: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:43.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:43 smithi045 conmon[39718]: cluster 2021-07-31T04:14:41.907676+0000 mgr.x ( 2021-07-31T04:14:43.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:43 smithi045 conmon[39718]: mgr.24542) 6324 : cluster [DBG] pgmap v6432: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:43.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:43 smithi045 conmon[33655]: cluster 2021-07-31T04:14:41.907676+0000 2021-07-31T04:14:43.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:43 smithi045 conmon[33655]: mgr.x (mgr.24542) 6324 : cluster [DBG] pgmap v6432: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:45.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:45 smithi154 conmon[34553]: cluster 2021-07-31T04:14:43.908116+0000 mgr.x (mgr.24542) 6325 : cluster [DBG] pgmap v6433: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:45.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:45 smithi154 conmon[34553]: 2021-07-31T04:14:45.524 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:45 smithi045 conmon[39718]: cluster 2021-07-31T04:14:43.908116+0000 mgr.x (mgr.24542 2021-07-31T04:14:45.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:45 smithi045 conmon[39718]: ) 6325 : cluster [DBG] pgmap v6433: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:45.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:45 smithi045 conmon[33655]: cluster 2021-07-31T04:14:43.908116+0000 mgr.x (mgr.24542) 2021-07-31T04:14:45.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:45 smithi045 conmon[33655]: 6325 : cluster [DBG] pgmap v6433: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:47.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:47 smithi154 conmon[34553]: cluster 2021-07-31T04:14:45.908587+0000 mgr.x (mgr.24542) 6326 : cluster [DBG] 2021-07-31T04:14:47.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:47 smithi154 conmon[34553]: pgmap v6434: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:47.524 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:47 smithi045 conmon[33655]: cluster 2021-07-31T04:14:45.908587+0000 mgr.x (mgr.24542) 6326 : cluster [DBG] 2021-07-31T04:14:47.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:47 smithi045 conmon[33655]: pgmap v6434: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:47.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:47 smithi045 conmon[39718]: cluster 2021-07-31T04:14:45.908587+0000 mgr.x (mgr.24542) 6326 : cluster [DBG] 2021-07-31T04:14:47.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:47 smithi045 conmon[39718]: pgmap v6434: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:49.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:49 smithi154 conmon[34553]: cluster 2021-07-31T04:14:47.909054+0000 mgr.x (mgr.24542) 6327 : cluster [DBG] 2021-07-31T04:14:49.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:49 smithi154 conmon[34553]: pgmap v6435: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:49.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:49 smithi045 conmon[39718]: cluster 2021-07-31T04:14:47.909054+0000 mgr.x (mgr.24542) 6327 : cluster 2021-07-31T04:14:49.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:49 smithi045 conmon[39718]: [DBG] pgmap v6435: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:49.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:49 smithi045 conmon[33655]: cluster 2021-07-31T04:14:47.909054+0000 mgr.x (mgr.24542) 6327 : cluster [DBG] pgmap v6435: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:51.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:51 smithi154 conmon[34553]: cluster 2021-07-31T04:14:49.909490+0000 mgr.x (mgr.24542) 6328 : cluster 2021-07-31T04:14:51.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:51 smithi154 conmon[34553]: [DBG] pgmap v6436: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:51.524 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:51 smithi045 conmon[39718]: cluster 2021-07-31T04:14:49.909490+0000 mgr.x (mgr 2021-07-31T04:14:51.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:51 smithi045 conmon[39718]: .24542) 6328 : cluster [DBG] pgmap v6436: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:51.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:51 smithi045 conmon[33655]: cluster 2021-07-31T04:14:49.909490+0000 mgr.x (mgr.24542) 2021-07-31T04:14:51.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:51 smithi045 conmon[33655]: 6328 : cluster [DBG] pgmap v6436: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:52.972 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T04:14:53.250 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T04:14:53.250 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 2h ago 2h - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T04:14:53.250 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 2h ago 2h 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T04:14:53.250 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 2h ago 2h 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T04:14:53.251 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 2h ago 2h - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T04:14:53.251 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T04:14:53.251 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 2h ago 2h 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T04:14:53.251 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T04:14:53.252 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 2h ago 2h - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T04:14:53.252 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 2h ago 2h 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T04:14:53.252 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T04:14:53.252 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T04:14:53.252 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T04:14:53.252 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T04:14:53.253 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 2h ago 2h 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T04:14:53.253 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 2h ago 2h 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T04:14:53.253 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 2h ago 2h 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T04:14:53.253 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 2h ago 2h 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T04:14:53.253 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 2h ago 2h 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T04:14:53.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:53 smithi154 conmon[34553]: cluster 2021-07-31T04:14:51.910052+0000 mgr.x (mgr.24542) 6329 : cluster [DBG] pgmap v6437: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:53.524 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:53 smithi045 conmon[39718]: cluster 2021-07-31T04:14:51.910052+0000 2021-07-31T04:14:53.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:53 smithi045 conmon[39718]: mgr.x (mgr.24542) 6329 : cluster [DBG] pgmap v6437: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:53.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:53 smithi045 conmon[33655]: cluster 2021-07-31T04:14:51.910052+0000 mgr.x 2021-07-31T04:14:53.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:53 smithi045 conmon[33655]: (mgr.24542) 6329 : cluster [DBG] pgmap v6437: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:53.583 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T04:14:53.583 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T04:14:53.583 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T04:14:53.584 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:14:53.584 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T04:14:53.584 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T04:14:53.584 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:14:53.584 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:14:53.585 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T04:14:53.585 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T04:14:53.585 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:14:53.585 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T04:14:53.585 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T04:14:53.586 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T04:14:53.586 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:14:53.586 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T04:14:53.586 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T04:14:54.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:54 smithi154 conmon[34553]: audit 2021-07-31T04:14:52.961253+0000 mgr.x (mgr.24542) 6330 : audit [DBG] 2021-07-31T04:14:54.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:54 smithi154 conmon[34553]: from='client.21198 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:14:54.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:54 smithi154 conmon[34553]: audit 2021-07-31T04:14:53.242949+0000 mgr.x (mgr.24542) 6331 : audit [DBG] from='client.31141 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:14:54.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:54 smithi154 conmon[34553]: audit 2021-07-31T04:14:53.584396+0000 mon.c ( 2021-07-31T04:14:54.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:54 smithi154 conmon[34553]: mon.1) 259 : audit [DBG] 2021-07-31T04:14:54.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:54 smithi154 conmon[34553]: from='client.? 172.21.15.45:0/376860366' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:14:54.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:54 smithi045 conmon[33655]: audit 2021-07-31T04:14:52.961253+0000 2021-07-31T04:14:54.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:54 smithi045 conmon[33655]: mgr.x (mgr.24542) 6330 : audit [DBG] from='client.21198 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:14:54.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:54 smithi045 conmon[33655]: audit 2021-07-31T04:14:53.242949+0000 mgr.x (mgr 2021-07-31T04:14:54.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:54 smithi045 conmon[33655]: .24542) 6331 : audit [DBG] from='client.31141 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:14:54.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:54 smithi045 conmon[33655]: audit 2021-07-31 2021-07-31T04:14:54.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:54 smithi045 conmon[33655]: T04:14:53.584396+0000 mon.c (mon.1) 259 2021-07-31T04:14:54.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:54 smithi045 conmon[33655]: : audit [DBG] from='client.? 172.21.15.45:0/376860366' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:14:54.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:54 smithi045 conmon[39718]: audit 2021-07-31T04:14:52.961253+0000 mgr.x (mgr.24542) 6330 2021-07-31T04:14:54.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:54 smithi045 conmon[39718]: : audit [DBG] from='client.21198 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:14:54.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:54 smithi045 conmon[39718]: audit 2021-07-31T04:14:53.242949+0000 mgr.x (mgr. 2021-07-31T04:14:54.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:54 smithi045 conmon[39718]: 24542) 6331 : audit [DBG] from='client.31141 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:14:54.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:54 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T04:14:54.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:54 smithi045 conmon[39718]: 31T04:14:53.584396+0000 mon.c (mon.1) 259 : audit [DBG] from='client.? 172.21.15.45:0/376860366' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:14:54.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:54 smithi045 conmon[39718]: 2021-07-31T04:14:55.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:55 smithi154 conmon[34553]: cluster 2021-07-31T04:14:53.910532+0000 mgr.x (mgr.24542) 6332 : cluster [DBG] pgmap v6438: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:55.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:55 smithi045 conmon[33655]: cluster 2021-07-31T04:14:53.910532+0000 mgr.x (mgr.24542) 6332 : cluster 2021-07-31T04:14:55.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:55 smithi045 conmon[33655]: [DBG] pgmap v6438: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:55.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:55 smithi045 conmon[39718]: cluster 2021-07-31T04:14:53.910532+0000 mgr.x ( 2021-07-31T04:14:55.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:55 smithi045 conmon[39718]: mgr.24542) 6332 : cluster [DBG] pgmap v6438: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:57.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:57 smithi154 conmon[34553]: cluster 2021-07-31T04:14:55.910894+0000 mgr.x (mgr.24542) 6333 : cluster [DBG] pgmap v6439: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:57.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:57 smithi045 conmon[33655]: cluster 2021-07-31T04:14:55.910894+0000 mgr.x (mgr.24542 2021-07-31T04:14:57.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:57 smithi045 conmon[33655]: ) 6333 : cluster [DBG] pgmap v6439: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:57.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:57 smithi045 conmon[39718]: cluster 2021-07-31T04:14:55.910894+0000 2021-07-31T04:14:57.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:57 smithi045 conmon[39718]: mgr.x (mgr.24542) 6333 : cluster [DBG] pgmap v6439: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:59.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:14:59 smithi154 conmon[34553]: cluster 2021-07-31T04:14:57.911376+0000 mgr.x (mgr.24542) 6334 : cluster [DBG] pgmap v6440: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:59.524 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:59 smithi045 conmon[33655]: cluster 2021-07-31T04:14:57.911376+0000 mgr.x 2021-07-31T04:14:59.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:14:59 smithi045 conmon[33655]: (mgr.24542) 6334 : cluster [DBG] pgmap v6440: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:14:59.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:59 smithi045 conmon[39718]: cluster 2021-07-31T04:14:57.911376+0000 mgr.x ( 2021-07-31T04:14:59.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:14:59 smithi045 conmon[39718]: mgr.24542) 6334 : cluster [DBG] pgmap v6440: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:01.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:01 smithi154 conmon[34553]: cluster 2021-07-31T04:14:59.911964+0000 mgr.x (mgr.24542) 6335 : cluster [DBG] pgmap v6441: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:01.524 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:01 smithi045 conmon[33655]: cluster 2021-07-31T04:14:59.911964+0000 2021-07-31T04:15:01.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:01 smithi045 conmon[33655]: mgr.x (mgr.24542) 6335 : cluster [DBG] pgmap v6441: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:01.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:01 smithi045 conmon[39718]: cluster 2021-07-31T04:14:59.911964+0000 mgr.x (mgr. 2021-07-31T04:15:01.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:01 smithi045 conmon[39718]: 24542) 6335 : cluster [DBG] pgmap v6441: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:03.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:03 smithi154 conmon[34553]: cluster 2021-07-31T04:15:01.912407+0000 mgr.x (mgr.24542) 6336 : cluster [DBG] pgmap v6442: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:03.524 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:03 smithi045 conmon[39718]: cluster 2021-07-31T04:15:01.912407+0000 mgr.x (mgr.24542 2021-07-31T04:15:03.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:03 smithi045 conmon[39718]: ) 6336 : cluster [DBG] pgmap v6442: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:03.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:03 smithi045 conmon[33655]: cluster 2021-07-31T04:15:01.912407+0000 mgr.x (mgr.24542) 6336 : cluster [DBG] 2021-07-31T04:15:03.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:03 smithi045 conmon[33655]: pgmap v6442: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:05.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:05 smithi154 conmon[34553]: cluster 2021-07-31T04:15:03.912980+0000 mgr.x (mgr.24542) 6337 : cluster [DBG] pgmap v6443: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:05.524 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:05 smithi045 conmon[39718]: cluster 2021-07-31T04:15:03.912980+0000 mgr.x (mgr.24542) 6337 : cluster [DBG] pgmap v6443: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:05.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:05 smithi045 conmon[33655]: cluster 2021-07-31T04:15:03.912980+0000 mgr.x (mgr. 2021-07-31T04:15:05.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:05 smithi045 conmon[33655]: 24542) 6337 : cluster [DBG] pgmap v6443: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:07.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:07 smithi154 conmon[34553]: cluster 2021-07-31T04:15:05.913408+0000 mgr.x (mgr.24542) 6338 : cluster [DBG] pgmap v6444: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:07.524 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:07 smithi045 conmon[39718]: cluster 2021-07-31T04:15:05.913408+0000 mgr.x (mgr.24542) 6338 : cluster [DBG] pgmap v6444: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:07.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:07 smithi045 conmon[33655]: cluster 2021-07-31T04:15:05.913408+0000 mgr.x (mgr.24542) 6338 : cluster 2021-07-31T04:15:07.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:07 smithi045 conmon[33655]: [DBG] pgmap v6444: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:09.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:09 smithi154 conmon[34553]: cluster 2021-07-31T04:15:07.913839+0000 mgr.x (mgr.24542) 6339 : cluster [DBG] pgmap v6445: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:09.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:09 smithi045 conmon[33655]: cluster 2021-07-31T04:15:07.913839+0000 mgr.x (mgr.24542) 6339 : cluster 2021-07-31T04:15:09.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:09 smithi045 conmon[33655]: [DBG] pgmap v6445: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:09.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:09 smithi045 conmon[39718]: cluster 2021-07-31T04:15:07.913839+0000 mgr.x (mgr.24542) 6339 : cluster [DBG] 2021-07-31T04:15:09.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:09 smithi045 conmon[39718]: pgmap v6445: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:11.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:11 smithi154 conmon[34553]: cluster 2021-07-31T04:15:09.914287+0000 mgr.x (mgr.24542) 6340 : cluster [DBG] pgmap v6446: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:11.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:11 smithi045 conmon[39718]: cluster 2021-07-31T04:15:09.914287+0000 mgr.x (mgr.24542) 6340 : cluster [DBG] 2021-07-31T04:15:11.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:11 smithi045 conmon[39718]: pgmap v6446: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:11.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:11 smithi045 conmon[33655]: cluster 2021-07-31T04:15:09.914287+0000 mgr.x (mgr.24542) 6340 : cluster [DBG] pgmap v6446: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:13.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:13 smithi154 conmon[34553]: cluster 2021-07-31T04:15:11.914678+0000 mgr.x (mgr.24542) 6341 : cluster [DBG] pgmap v6447: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:13.524 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:13 smithi045 conmon[39718]: cluster 2021-07-31T04:15:11 2021-07-31T04:15:13.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:13 smithi045 conmon[39718]: .914678+0000 mgr.x (mgr.24542) 6341 : cluster [DBG] pgmap v6447: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:13.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:13 smithi045 conmon[33655]: cluster 2021-07-31T04:15:11.914678+0000 mgr.x (mgr.24542) 2021-07-31T04:15:13.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:13 smithi045 conmon[33655]: 6341 : cluster [DBG] pgmap v6447: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:15.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:15 smithi154 conmon[34553]: cluster 2021-07-31T04:15:13.915160+0000 mgr.x (mgr.24542) 6342 : cluster [DBG] pgmap v6448: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:15.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:15 smithi045 conmon[39718]: cluster 2021-07-31T04:15:13. 2021-07-31T04:15:15.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:15 smithi045 conmon[39718]: 915160+0000 mgr.x (mgr.24542) 6342 : cluster [DBG] pgmap v6448: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:15.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:15 smithi045 conmon[33655]: cluster 2021-07-31T04:15:13.915160+0000 mgr.x (mgr.24542 2021-07-31T04:15:15.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:15 smithi045 conmon[33655]: ) 6342 : cluster [DBG] pgmap v6448: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:17.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:17 smithi154 conmon[34553]: cluster 2021-07-31T04:15:15.915665+0000 mgr.x (mgr.24542) 6343 : cluster [DBG] pgmap v6449: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:17.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:17 smithi154 conmon[34553]: 2021-07-31T04:15:17.524 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:17 smithi045 conmon[39718]: cluster 2021-07-31T04:15:15.915665+0000 mgr.x 2021-07-31T04:15:17.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:17 smithi045 conmon[39718]: (mgr.24542) 6343 : cluster [DBG] pgmap v6449: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:17.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:17 smithi045 conmon[33655]: cluster 2021-07-31T04:15:15.915665+0000 mgr.x ( 2021-07-31T04:15:17.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:17 smithi045 conmon[33655]: mgr.24542) 6343 : cluster [DBG] pgmap v6449: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:19.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:19 smithi154 conmon[34553]: cluster 2021-07-31T04:15:17.916215+0000 mgr.x (mgr.24542) 6344 : cluster [DBG] pgmap v6450: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:19.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:19 smithi045 conmon[33655]: cluster 2021-07-31T04:15:17.916215+0000 mgr.x (mgr.24542) 6344 2021-07-31T04:15:19.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:19 smithi045 conmon[33655]: : cluster [DBG] pgmap v6450: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:19.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:19 smithi045 conmon[39718]: cluster 2021-07-31T04:15:17.916215+0000 mgr.x (mgr.24542) 6344 : cluster 2021-07-31T04:15:19.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:19 smithi045 conmon[39718]: [DBG] pgmap v6450: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:21.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:21 smithi154 conmon[34553]: cluster 2021-07-31T04:15:19.916752+0000 mgr.x (mgr.24542) 6345 : cluster [DBG] pgmap v6451: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:21.524 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:21 smithi045 conmon[33655]: cluster 2021-07-31T04:15:19.916752+0000 mgr.x (mgr.24542) 6345 2021-07-31T04:15:21.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:21 smithi045 conmon[33655]: : cluster [DBG] pgmap v6451: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:21.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:21 smithi045 conmon[39718]: cluster 2021-07-31T04:15:19.916752+0000 mgr.x (mgr.24542) 6345 : cluster 2021-07-31T04:15:21.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:21 smithi045 conmon[39718]: [DBG] pgmap v6451: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:23.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:23 smithi154 conmon[34553]: cluster 2021-07-31T04:15:21.917228+0000 mgr.x (mgr.24542) 6346 : cluster [DBG] pgmap v6452: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:23.524 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:23 smithi045 conmon[39718]: cluster 2021-07-31T04:15:21.917228+0000 2021-07-31T04:15:23.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:23 smithi045 conmon[39718]: mgr.x (mgr.24542) 6346 : cluster [DBG] pgmap v6452: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:23.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:23 smithi045 conmon[33655]: cluster 2021-07-31T04:15:21.917228+0000 mgr.x (mgr.24542) 6346 : 2021-07-31T04:15:23.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:23 smithi045 conmon[33655]: cluster [DBG] pgmap v6452: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:23.901 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T04:15:24.178 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T04:15:24.179 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 2h ago 2h - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T04:15:24.179 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 2h ago 2h 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T04:15:24.179 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 2h ago 2h 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T04:15:24.179 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 2h ago 2h - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T04:15:24.180 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T04:15:24.180 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 2h ago 2h 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T04:15:24.180 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T04:15:24.180 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 2h ago 2h - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T04:15:24.180 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 2h ago 2h 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T04:15:24.181 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T04:15:24.181 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T04:15:24.181 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T04:15:24.181 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T04:15:24.181 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 2h ago 2h 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T04:15:24.182 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 2h ago 2h 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T04:15:24.182 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 2h ago 2h 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T04:15:24.182 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 2h ago 2h 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T04:15:24.182 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 2h ago 2h 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T04:15:24.515 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T04:15:24.516 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T04:15:24.516 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T04:15:24.516 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:15:24.516 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T04:15:24.516 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T04:15:24.517 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:15:24.517 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:15:24.517 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T04:15:24.517 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T04:15:24.517 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:15:24.518 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T04:15:24.518 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T04:15:24.518 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T04:15:24.518 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:15:24.519 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T04:15:24.519 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T04:15:25.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:25 smithi154 conmon[34553]: audit 2021-07-31T04:15:23.890210+0000 mgr.x (mgr.24542) 6347 : audit [DBG] 2021-07-31T04:15:25.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:25 smithi154 conmon[34553]: from='client.21216 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:15:25.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:25 smithi154 conmon[34553]: cluster 2021-07-31T04:15:23.917701+0000 mgr.x (mgr.24542) 6348 : cluster [DBG] pgmap v6453: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:25.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:25 smithi154 conmon[34553]: audit 2021-07-31T04:15:24.171534+0000 mgr.x 2021-07-31T04:15:25.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:25 smithi154 conmon[34553]: (mgr.24542) 6349 : audit [DBG] from='client.21222 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:15:25.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:25 smithi154 conmon[34553]: audit 2021-07-31 2021-07-31T04:15:25.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:25 smithi154 conmon[34553]: T04:15:24.516571+0000 mon.c (mon.1) 260 : audit [DBG] 2021-07-31T04:15:25.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:25 smithi154 conmon[34553]: from='client.? 172.21.15.45:0/3617245288' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:15:25.524 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:25 smithi045 conmon[39718]: audit 2021-07-31T04:15:23.890210+0000 mgr.x ( 2021-07-31T04:15:25.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:25 smithi045 conmon[39718]: mgr.24542) 6347 : audit [DBG] from='client.21216 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:15:25.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:25 smithi045 conmon[39718]: cluster 2021-07-31T04:15:23.917701 2021-07-31T04:15:25.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:25 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 6348 : cluster [DBG] pgmap v6453: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:25.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:25 smithi045 conmon[39718]: audit 2021-07-31T04:15:25.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:25 smithi045 conmon[39718]: 2021-07-31T04: 2021-07-31T04:15:25.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:25 smithi045 conmon[39718]: 15:24.171534+0000 mgr.x (mgr.24542) 6349 : audit [DBG] from='client.21222 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:15:25.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:25 smithi045 conmon[39718]: audit 2021- 2021-07-31T04:15:25.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:25 smithi045 conmon[39718]: 07-31T04:15:24.516571+0000 2021-07-31T04:15:25.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:25 smithi045 conmon[39718]: mon.c (mon.1) 260 : audit 2021-07-31T04:15:25.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:25 smithi045 conmon[39718]: [DBG] from='client.? 172.21.15.45:0/3617245288' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:15:25.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:25 smithi045 conmon[33655]: audit 2021-07-31T04:15:23.890210+0000 mgr.x (mgr.24542) 6347 : audit 2021-07-31T04:15:25.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:25 smithi045 conmon[33655]: [DBG] from='client.21216 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:15:25.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:25 smithi045 conmon[33655]: cluster 2021-07-31T04:15:23.917701+0000 mgr.x (mgr.24542) 6348 : cluster [DBG] pgmap v6453: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:25.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:25 smithi045 conmon[33655]: audit 2021-07-31T04: 2021-07-31T04:15:25.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:25 smithi045 conmon[33655]: 15:24.171534+0000 mgr.x (mgr.24542) 6349 : audit [DBG] from='client.21222 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:15:25.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:25 smithi045 conmon[33655]: audit 2021- 2021-07-31T04:15:25.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:25 smithi045 conmon[33655]: 07-31T04:15:24.516571+0000 mon.c (mon.1) 260 : audit [DBG] from='client.? 172.21.15.45:0/3617245288' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:15:27.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:27 smithi154 conmon[34553]: cluster 2021-07-31T04:15:25.918083+0000 mgr.x (mgr.24542) 6350 : cluster 2021-07-31T04:15:27.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:27 smithi154 conmon[34553]: [DBG] pgmap v6454: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:27.524 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:27 smithi045 conmon[33655]: cluster 2021-07-31T04:15:25.918083+0000 mgr.x (mgr.24542) 6350 : cluster [DBG] pgmap v6454: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:27.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:27 smithi045 conmon[39718]: cluster 2021-07-31T04:15:25.918083+0000 mgr.x 2021-07-31T04:15:27.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:27 smithi045 conmon[39718]: (mgr.24542) 6350 : cluster [DBG] pgmap v6454: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:29.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:29 smithi154 conmon[34553]: cluster 2021-07-31T04:15:27.918443+0000 mgr.x (mgr.24542) 6351 : cluster [DBG] 2021-07-31T04:15:29.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:29 smithi154 conmon[34553]: pgmap v6455: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:29 smithi154 conmon[34553]: audit 2021-07-31T04:15:28.519377+0000 mon.b (mon.2) 815 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:15:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:29 smithi154 conmon[34553]: audit 2021-07-31T04:15:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:29 smithi154 conmon[34553]: 2021-07-31T04:15:28.519612+0000 mon.b (mon.2) 816 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:15:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:29 smithi154 conmon[34553]: audit 2021-07- 2021-07-31T04:15:29.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:29 smithi154 conmon[34553]: 31T04:15:28.520370+0000 mon.a (mon.0 2021-07-31T04:15:29.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:29 smithi154 conmon[34553]: ) 3447 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:15:29.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:29 smithi154 conmon[34553]: audit 2021 2021-07-31T04:15:29.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:29 smithi154 conmon[34553]: -07-31T04:15:28.520760+0000 mon.a (mon 2021-07-31T04:15:29.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:29 smithi154 conmon[34553]: .0) 3448 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:15:29.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:29 smithi045 conmon[39718]: cluster 2021-07-31T04:15:27.918443+0000 2021-07-31T04:15:29.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:29 smithi045 conmon[39718]: mgr.x (mgr.24542) 6351 : cluster [DBG] pgmap v6455: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:29.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:29 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T04:15:29.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:29 smithi045 conmon[39718]: T04:15:28.519377+0000 mon.b (mon.2) 815 : 2021-07-31T04:15:29.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:29 smithi045 conmon[39718]: audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:15:29.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:29 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T04:15:29.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:29 smithi045 conmon[39718]: T04:15:28.519612+0000 mon.b 2021-07-31T04:15:29.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:29 smithi045 conmon[39718]: (mon.2) 816 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:15:29.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:29 smithi045 conmon[39718]: 2021-07-31T04:15:29.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:29 smithi045 conmon[39718]: audit 2021-07-31T04:15:29.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:29 smithi045 conmon[39718]: 2021-07-31T04:15:28.520370+0000 2021-07-31T04:15:29.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:29 smithi045 conmon[39718]: mon.a (mon.0) 3447 : audit [INF] 2021-07-31T04:15:29.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:29 smithi045 conmon[39718]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:15:29.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:29 smithi045 conmon[39718]: audit 2021-07-31T04: 2021-07-31T04:15:29.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:29 smithi045 conmon[39718]: 15:28.520760+0000 mon.a (mon. 2021-07-31T04:15:29.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:29 smithi045 conmon[39718]: 0) 3448 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:15:29.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:29 smithi045 conmon[33655]: cluster 2021-07-31T04:15:27.918443+0000 mgr.x (mgr.24542) 6351 : cluster [DBG] pgmap v6455: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:29.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:29 smithi045 conmon[33655]: 2021-07-31T04:15:29.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:29 smithi045 conmon[33655]: audit 2021-07-31T04:15:28.519377+0000 mon.b (mon.2) 815 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:15:29.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:29 smithi045 conmon[33655]: audit 2021-07-31T04:15:28.519612+0000 mon.b (mon.2) 816 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:15:29.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:29 smithi045 conmon[33655]: audit 2021-07-31T04:15 2021-07-31T04:15:29.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:29 smithi045 conmon[33655]: :28.520370+0000 mon.a (mon.0) 3447 : audit [INF] 2021-07-31T04:15:29.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:29 smithi045 conmon[33655]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:15:29.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:29 smithi045 conmon[33655]: audit 2021-07-31T04:15:29.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:29 smithi045 conmon[33655]: 2021-07-31T04:15:28.520760+0000 mon.a ( 2021-07-31T04:15:29.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:29 smithi045 conmon[33655]: mon.0) 3448 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:15:31.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:31 smithi045 conmon[33655]: cluster 2021-07-31T04:15:29.918993+0000 mgr.x (mgr.24542 2021-07-31T04:15:31.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:31 smithi045 conmon[33655]: ) 6352 : cluster [DBG] pgmap v6456: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:31.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:31 smithi045 conmon[39718]: cluster 2021-07-31T04:15:29 2021-07-31T04:15:31.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:31 smithi045 conmon[39718]: .918993+0000 mgr.x (mgr.24542) 6352 : cluster [DBG] pgmap v6456: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:31.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:31 smithi154 conmon[34553]: cluster 2021-07-31T04:15:29.918993+0000 mgr.x (mgr.24542) 6352 2021-07-31T04:15:31.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:31 smithi154 conmon[34553]: : cluster [DBG] pgmap v6456: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:33.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:33 smithi045 conmon[33655]: cluster 2021-07-31T04:15:31.919394+0000 mgr.x (mgr.24542) 6353 : cluster [DBG] pgmap v6457: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:33.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:33 smithi045 conmon[39718]: cluster 2021-07-31T04:15:31.919394+0000 mgr.x (mgr 2021-07-31T04:15:33.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:33 smithi045 conmon[39718]: .24542) 6353 : cluster [DBG] pgmap v6457: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:33.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:33 smithi154 conmon[34553]: cluster 2021-07-31T04:15:31.919394+0000 mgr.x (mgr.24542) 2021-07-31T04:15:33.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:33 smithi154 conmon[34553]: 6353 : cluster [DBG] pgmap v6457: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:35.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:35 smithi045 conmon[39718]: cluster 2021-07-31T04:15:33.919996+0000 mgr.x 2021-07-31T04:15:35.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:35 smithi045 conmon[39718]: (mgr.24542) 6354 : cluster [DBG] pgmap v6458: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:35.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:35 smithi045 conmon[33655]: cluster 2021-07-31T04:15:33.919996+0000 2021-07-31T04:15:35.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:35 smithi045 conmon[33655]: mgr.x (mgr.24542) 6354 : cluster [DBG] pgmap v6458: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:35.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:35 smithi154 conmon[34553]: cluster 2021-07-31T04:15:33.919996+0000 mgr.x (mgr.24542) 6354 2021-07-31T04:15:35.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:35 smithi154 conmon[34553]: : cluster [DBG] pgmap v6458: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:37.524 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:37 smithi045 conmon[33655]: cluster 2021-07-31T04:15:35.920389+0000 mgr.x (mgr 2021-07-31T04:15:37.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:37 smithi045 conmon[33655]: .24542) 6355 : cluster [DBG] pgmap v6459: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:37.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:37 smithi045 conmon[39718]: cluster 2021-07-31T04:15:35.920389+0000 mgr.x (mgr.24542 2021-07-31T04:15:37.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:37 smithi045 conmon[39718]: ) 6355 : cluster [DBG] pgmap v6459: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:37.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[34553]: cluster 2021-07-31T04:15:35.920389+0000 mgr.x (mgr.24542) 6355 : cluster 2021-07-31T04:15:37.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[34553]: [DBG] pgmap v6459: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:38.252 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: debug 2021-07-31T04:15:37.798+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:15:38.252 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:15:38.252 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:15:38.253 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: yield 2021-07-31T04:15:38.253 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:15:38.253 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:15:38.253 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:15:38.253 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:15:38.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:15:38.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:15:38.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:15:38.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:15:38.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:15:38.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: task.result() 2021-07-31T04:15:38.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:15:38.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: debug 2021-07-31T04:15:37.810+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:15:38.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:15:38.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:15:38.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: yield 2021-07-31T04:15:38.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:15:38.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:15:38.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:15:38.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:15:38.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:15:38.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:15:38.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:15:38.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:15:38.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:15:38.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: task.result() 2021-07-31T04:15:38.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:15:38.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: debug 2021-07-31T04:15:37.836+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:15:38.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:15:38.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:15:38.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: yield 2021-07-31T04:15:38.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:15:38.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:15:38.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:15:38.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:15:38.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:15:38.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:15:38.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:15:38.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:15:38.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:15:38.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: task.result() 2021-07-31T04:15:38.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:15:38.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: debug 2021-07-31T04:15:37.836+0000 7fcb9115c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:15:38.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:15:38.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:15:38.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: yield 2021-07-31T04:15:38.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:15:38.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:15:38.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:15:38.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:15:38.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:15:38.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:15:38.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:15:38.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:15:38.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:15:38.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: task.result() 2021-07-31T04:15:38.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:15:38.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: 2021-07-31T04:15:38.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T04:15:38.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: 2021-07-31T04:15:38.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:15:38.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:15:38.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: return f(*arg) 2021-07-31T04:15:38.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:15:38.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:15:38.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:15:38.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:15:38.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:15:38.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:15:38.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:15:38.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T04:15:38.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:15:38.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: raise self._exception 2021-07-31T04:15:38.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:15:38.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:15:38.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:15:38.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T04:15:38.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:15:38.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: raise 2021-07-31T04:15:38.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:15:38.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T04:15:38.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:15:38.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:15:38.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:15:38.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: debug 2021-07-31T04:15:37.845+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:15:38.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:15:38.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:15:38.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: yield 2021-07-31T04:15:38.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:15:38.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:15:38.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:15:38.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:15:38.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:15:38.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:15:38.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:15:38.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:15:38.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:15:38.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: task.result() 2021-07-31T04:15:38.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:15:38.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: debug 2021-07-31T04:15:37.845+0000 7fcb9095b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:15:38.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:15:38.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:15:38.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: yield 2021-07-31T04:15:38.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:15:38.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:15:38.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:15:38.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:15:38.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:15:38.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:15:38.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:15:38.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:15:38.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:15:38.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: task.result() 2021-07-31T04:15:38.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:15:38.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: 2021-07-31T04:15:38.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T04:15:38.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: 2021-07-31T04:15:38.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:15:38.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:15:38.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: return f(*arg) 2021-07-31T04:15:38.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:15:38.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:15:38.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:15:38.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:15:38.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:15:38.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:15:38.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:15:38.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T04:15:38.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:15:38.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: raise self._exception 2021-07-31T04:15:38.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:15:38.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:15:38.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:15:38.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T04:15:38.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:15:38.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: raise 2021-07-31T04:15:38.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:15:38.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T04:15:38.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:15:38.284 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:15:38.284 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:15:37 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:15:38.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:38 smithi045 conmon[33655]: audit 2021-07-31T04:15:37.752018+0000 mon.b ( 2021-07-31T04:15:38.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:38 smithi045 conmon[33655]: mon.2) 817 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T04:15:38.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:38 smithi045 conmon[33655]: audit 2021-07-31T04:15:37.753584+0000 mon.b (mon.2) 818 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T04:15:38.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:38 smithi045 conmon[33655]: audit 2021-07-31T04:15:37 2021-07-31T04:15:38.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:38 smithi045 conmon[33655]: .754621+0000 mon.b (mon.2) 2021-07-31T04:15:38.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:38 smithi045 conmon[33655]: 819 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T04:15:38.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:38 smithi045 conmon[39718]: audit 2021-07-31T04:15:37.752018+0000 mon.b ( 2021-07-31T04:15:38.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:38 smithi045 conmon[39718]: mon.2) 817 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T04:15:38.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:38 smithi045 conmon[39718]: audit 2021-07-31T04:15:37.753584+0000 mon.b (mon.2) 2021-07-31T04:15:38.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:38 smithi045 conmon[39718]: 818 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T04:15:38.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:38 smithi045 conmon[39718]: audit 2021-07-31T04:15:37 2021-07-31T04:15:38.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:38 smithi045 conmon[39718]: .754621+0000 mon.b (mon.2) 819 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T04:15:38.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:38 smithi154 conmon[34553]: audit 2021-07-31T04:15:37.752018+0000 mon.b (mon.2) 817 : audit 2021-07-31T04:15:38.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:38 smithi154 conmon[34553]: [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T04:15:38.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:38 smithi154 conmon[34553]: audit 2021-07-31T04:15:37.753584+0000 mon.b (mon.2) 818 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T04:15:38.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:38 smithi154 conmon[34553]: audit 2021- 2021-07-31T04:15:38.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:38 smithi154 conmon[34553]: 07-31T04:15:37.754621+0000 2021-07-31T04:15:38.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:38 smithi154 conmon[34553]: mon.b (mon.2) 819 : audit 2021-07-31T04:15:38.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:38 smithi154 conmon[34553]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T04:15:39.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: cephadm 2021-07-31T04:15:37.755372+0000 mgr.x (mgr.24542) 6356 2021-07-31T04:15:39.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T04:15:39.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: cephadm 2021-07-31T04:15:37.799605+0000 mgr.x (mgr.24542) 6357 : cephadm [ERR] 2021-07-31T04:15:39.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:15:39.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:15:39.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: yield 2021-07-31T04:15:39.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:15:39.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:15:39.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:15:39.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:15:39.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:15:39.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:15:39.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:15:39.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:15:39.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:15:39.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: task.result() 2021-07-31T04:15:39.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:15:39.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: cephadm 2021-07-31T04:15:37.800055+0000 mgr.x (mgr.24542) 6358 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T04:15:39.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: cephadm 2021-07-31T04 2021-07-31T04:15:39.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: :15:37.811264+0000 mgr.x (mgr. 2021-07-31T04:15:39.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: 24542) 6359 : cephadm [ERR] 2021-07-31T04:15:39.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:15:39.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:15:39.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: yield 2021-07-31T04:15:39.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:15:39.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:15:39.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:15:39.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:15:39.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:15:39.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:15:39.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:15:39.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:15:39.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:15:39.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: task.result() 2021-07-31T04:15:39.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:15:39.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: cephadm 2021-07-31T04:15:37.812363+0000 mgr.x (mgr.24542) 6360 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T04:15:39.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: cephadm 2021- 2021-07-31T04:15:39.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: 07-31T04:15:37.837055 2021-07-31T04:15:39.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 6361 2021-07-31T04:15:39.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: : cephadm [ERR] 2021-07-31T04:15:39.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: cephadm 2021-07-31T04:15:37. 2021-07-31T04:15:39.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: 755372+0000 mgr.x (mgr.24542) 6356 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T04:15:39.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: cephadm 2021-07-31T04:15:39.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: 2021-07-31T04 2021-07-31T04:15:39.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: :15:37.799605+0000 2021-07-31T04:15:39.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: mgr.x (mgr.24542) 6357 : 2021-07-31T04:15:39.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: cephadm [ERR] 2021-07-31T04:15:39.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:15:39.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:15:39.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: yield 2021-07-31T04:15:39.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:15:39.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:15:39.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:15:39.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:15:39.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:15:39.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:15:39.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:15:39.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:15:39.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:15:39.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: task.result() 2021-07-31T04:15:39.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:15:39.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: cephadm 2021-07-31T04:15:37.800055+0000 mgr.x (mgr. 2021-07-31T04:15:39.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: 24542) 6358 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T04:15:39.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: 2021-07-31T04:15:39.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: cephadm 2021- 2021-07-31T04:15:39.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: 07-31T04:15:37.811264 2021-07-31T04:15:39.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 6359 2021-07-31T04:15:39.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: : cephadm [ERR] 2021-07-31T04:15:39.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:15:39.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:15:39.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: yield 2021-07-31T04:15:39.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:15:39.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:15:39.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:15:39.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:15:39.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:15:39.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:15:39.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:15:39.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:15:39.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:15:39.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: task.result() 2021-07-31T04:15:39.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:15:39.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: cephadm 2021-07-31T04:15:37.812363+0000 mgr.x (mgr.24542) 2021-07-31T04:15:39.544 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:15:39.544 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:15:39.544 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: yield 2021-07-31T04:15:39.544 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:15:39.545 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:15:39.545 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:15:39.545 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:15:39.545 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:15:39.545 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:15:39.546 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:15:39.546 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:15:39.546 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:15:39.546 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: task.result() 2021-07-31T04:15:39.546 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:15:39.547 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: cephadm 2021-07-31T04:15:37.837478+0000 mgr.x (mgr.24542) 6362 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:15:39.547 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:15:39.547 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:15:39.547 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: yield 2021-07-31T04:15:39.547 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:15:39.548 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:15:39.548 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:15:39.548 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:15:39.548 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:15:39.549 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:15:39.549 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:15:39.549 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:15:39.549 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:15:39.549 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: task.result() 2021-07-31T04:15:39.550 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:15:39.550 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: 2021-07-31T04:15:39.550 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T04:15:39.550 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: 2021-07-31T04:15:39.550 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:15:39.551 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:15:39.551 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: return f(*arg) 2021-07-31T04:15:39.551 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:15:39.551 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:15:39.551 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:15:39.552 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:15:39.552 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:15:39.552 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:15:39.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: 6360 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T04:15:39.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: cephadm 2021- 2021-07-31T04:15:39.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: 07-31T04:15:37.837055 2021-07-31T04:15:39.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 6361 2021-07-31T04:15:39.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: : cephadm [ERR] 2021-07-31T04:15:39.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:15:39.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:15:39.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: yield 2021-07-31T04:15:39.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:15:39.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:15:39.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:15:39.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:15:39.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:15:39.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:15:39.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:15:39.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:15:39.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:15:39.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: task.result() 2021-07-31T04:15:39.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:15:39.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: cephadm 2021-07-31T04:15:37.837478+0000 mgr.x 2021-07-31T04:15:39.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: (mgr.24542) 6362 : cephadm 2021-07-31T04:15:39.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:15:39.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:15:39.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:15:39.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: yield 2021-07-31T04:15:39.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:15:39.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:15:39.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:15:39.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:15:39.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:15:39.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:15:39.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:15:39.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:15:39.559 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:15:39.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: task.result() 2021-07-31T04:15:39.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:15:39.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: 2021-07-31T04:15:39.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T04:15:39.560 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: 2021-07-31T04:15:39.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:15:39.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:15:39.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: return f(*arg) 2021-07-31T04:15:39.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:15:39.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:15:39.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:15:39.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T04:15:39.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:15:39.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: raise self._exception 2021-07-31T04:15:39.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:15:39.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:15:39.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:15:39.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T04:15:39.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:15:39.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: raise 2021-07-31T04:15:39.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:15:39.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T04:15:39.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:15:39.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:15:39.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:15:39.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: cephadm 2021-07-31T04:15:37.846090+0000 mgr.x (mgr.24542) 6363 : cephadm [ERR] 2021-07-31T04:15:39.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:15:39.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:15:39.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: yield 2021-07-31T04:15:39.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:15:39.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:15:39.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:15:39.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:15:39.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:15:39.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:15:39.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:15:39.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:15:39.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:15:39.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: task.result() 2021-07-31T04:15:39.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:15:39.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: cephadm 2021-07-31T04:15:37.846493+0000 mgr.x (mgr.24542) 6364 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:15:39.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:15:39.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:15:39.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: yield 2021-07-31T04:15:39.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:15:39.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:15:39.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:15:39.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:15:39.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:15:39.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:15:39.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:15:39.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:15:39.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:15:39.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:15:39.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T04:15:39.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:15:39.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: raise self._exception 2021-07-31T04:15:39.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:15:39.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:15:39.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:15:39.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T04:15:39.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:15:39.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: raise 2021-07-31T04:15:39.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:15:39.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T04:15:39.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:15:39.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:15:39.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:15:39.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: cephadm 2021-07-31T04:15:37.846090+0000 mgr.x (mgr.24542) 6363 : cephadm [ERR] 2021-07-31T04:15:39.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:15:39.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:15:39.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: yield 2021-07-31T04:15:39.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:15:39.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:15:39.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:15:39.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:15:39.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:15:39.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:15:39.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:15:39.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:15:39.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:15:39.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: task.result() 2021-07-31T04:15:39.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:15:39.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: cephadm 2021-07-31T04:15:37.846493+0000 mgr.x (mgr.24542) 6364 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:15:39.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:15:39.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:15:39.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: yield 2021-07-31T04:15:39.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:15:39.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:15:39.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:15:39.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:15:39.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:15:39.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: task.result() 2021-07-31T04:15:39.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:15:39.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: 2021-07-31T04:15:39.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T04:15:39.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: 2021-07-31T04:15:39.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:15:39.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:15:39.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: return f(*arg) 2021-07-31T04:15:39.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:15:39.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:15:39.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:15:39.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:15:39.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:15:39.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:15:39.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:15:39.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T04:15:39.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:15:39.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: raise self._exception 2021-07-31T04:15:39.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:15:39.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:15:39.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:15:39.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T04:15:39.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:15:39.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: raise 2021-07-31T04:15:39.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:15:39.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T04:15:39.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:15:39.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:15:39.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:15:39.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[33655]: cluster 2021-07-31T04:15:37.920764+0000 mgr.x (mgr.24542) 6365 : cluster [DBG] pgmap v6460: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:39.587 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:15:39.587 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:15:39.587 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:15:39.587 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:15:39.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:15:39.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:15:39.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:15:39.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:15:39.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: task.result() 2021-07-31T04:15:39.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:15:39.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: 2021-07-31T04:15:39.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T04:15:39.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: 2021-07-31T04:15:39.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:15:39.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:15:39.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: return f(*arg) 2021-07-31T04:15:39.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:15:39.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:15:39.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:15:39.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:15:39.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:15:39.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:15:39.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:15:39.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T04:15:39.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:15:39.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: raise self._exception 2021-07-31T04:15:39.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:15:39.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:15:39.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:15:39.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T04:15:39.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:15:39.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: raise 2021-07-31T04:15:39.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:15:39.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T04:15:39.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:15:39.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:15:39.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:15:39.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:39 smithi045 conmon[39718]: cluster 2021-07-31T04:15:37.920764+0000 mgr.x (mgr.24542) 6365 : cluster [DBG] pgmap v6460: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:39.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:15:37.755372+0000 mgr.x (mgr.24542) 6356 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T04:15:39.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:15:37.799605+0000 mgr.x (mgr.24542) 6357 : cephadm 2021-07-31T04:15:39.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: [ERR] 2021-07-31T04:15:39.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:15:39.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:15:39.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: yield 2021-07-31T04:15:39.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:15:39.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:15:39.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:15:39.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:15:39.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:15:39.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:15:39.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:15:39.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:15:39.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:15:39.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: task.result() 2021-07-31T04:15:39.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:15:39.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:15:37.800055+0000 mgr.x (mgr.24542) 6358 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T04:15:39.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:15:37 2021-07-31T04:15:39.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: .811264+0000 mgr.x (mgr.24542) 6359 : cephadm [ERR] 2021-07-31T04:15:39.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:15:39.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:15:39.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: yield 2021-07-31T04:15:39.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:15:39.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:15:39.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:15:39.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:15:39.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:15:39.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:15:39.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:15:39.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:15:39.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:15:39.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: task.result() 2021-07-31T04:15:39.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:15:39.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:15:37.812363+0000 mgr.x (mgr.24542) 6360 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T04:15:39.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:15:37.837055+0000 mgr.x (mgr.24542) 6361 : cephadm [ERR] 2021-07-31T04:15:39.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:15:39.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:15:39.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: yield 2021-07-31T04:15:39.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:15:39.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:15:39.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:15:39.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:15:39.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:15:39.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:15:39.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:15:39.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:15:39.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:15:39.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: task.result() 2021-07-31T04:15:39.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:15:39.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:15:37. 2021-07-31T04:15:39.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: 837478+0000 mgr.x (mgr.24542) 6362 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:15:39.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:15:39.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:15:39.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: yield 2021-07-31T04:15:39.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:15:39.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:15:39.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:15:39.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:15:39.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:15:39.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:15:39.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:15:39.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:15:39.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:15:39.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: task.result() 2021-07-31T04:15:39.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:15:39.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: 2021-07-31T04:15:39.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T04:15:39.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: 2021-07-31T04:15:39.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:15:39.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:15:39.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: return f(*arg) 2021-07-31T04:15:39.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:15:39.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:15:39.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:15:39.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:15:39.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:15:39.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:15:39.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:15:39.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T04:15:39.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:15:39.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: raise self._exception 2021-07-31T04:15:39.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:15:39.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:15:39.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:15:39.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T04:15:39.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:15:39.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: raise 2021-07-31T04:15:39.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:15:39.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T04:15:39.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:15:39.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:15:39.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:15:39.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:15:39.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: 2021-07-31T04:15:37.846090+0000 mgr.x (mgr.24542) 6363 : cephadm [ERR] 2021-07-31T04:15:39.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:15:39.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:15:39.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: yield 2021-07-31T04:15:39.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:15:39.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:15:39.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:15:39.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:15:39.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:15:39.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:15:39.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:15:39.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:15:39.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:15:39.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: task.result() 2021-07-31T04:15:39.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:15:39.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:15:37.846493+0000 mgr.x (mgr.24542) 6364 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:15:39.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:15:39.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:15:39.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: yield 2021-07-31T04:15:39.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:15:39.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:15:39.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:15:39.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:15:39.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:15:39.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:15:39.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:15:39.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:15:39.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:15:39.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: task.result() 2021-07-31T04:15:39.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:15:39.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: 2021-07-31T04:15:39.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T04:15:39.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: 2021-07-31T04:15:39.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:15:39.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:15:39.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: return f(*arg) 2021-07-31T04:15:39.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:15:39.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:15:39.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:15:39.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:15:39.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:15:39.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:15:39.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:15:39.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T04:15:39.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:15:39.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: raise self._exception 2021-07-31T04:15:39.782 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:15:39.782 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:15:39.782 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:15:39.782 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T04:15:39.782 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:15:39.783 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: raise 2021-07-31T04:15:39.783 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:15:39.783 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T04:15:39.783 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:15:39.783 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:15:39.784 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:15:39.784 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:39 smithi154 conmon[34553]: cluster 2021-07-31T04:15:37.920764+0000 mgr.x (mgr.24542) 6365 : cluster [DBG] pgmap v6460: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:41.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:41 smithi045 conmon[33655]: cluster 2021-07-31T04:15:39.921238+0000 2021-07-31T04:15:41.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:41 smithi045 conmon[33655]: mgr.x (mgr.24542) 6366 : cluster [DBG] pgmap v6461: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:41.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:41 smithi045 conmon[39718]: cluster 2021-07-31T04:15:39 2021-07-31T04:15:41.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:41 smithi045 conmon[39718]: .921238+0000 mgr.x (mgr.24542) 6366 : cluster [DBG] pgmap v6461: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:41.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:41 smithi154 conmon[34553]: cluster 2021-07-31T04:15:39.921238+0000 mgr.x (mgr.24542) 6366 : cluster [DBG] pgmap v6461: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:41.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:41 smithi154 conmon[34553]: 2021-07-31T04:15:43.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:43 smithi154 conmon[34553]: cluster 2021-07-31T04:15:41.921880+0000 mgr.x (mgr. 2021-07-31T04:15:43.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:43 smithi154 conmon[34553]: 24542) 6367 : cluster [DBG] pgmap v6462: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:43.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:43 smithi045 conmon[39718]: cluster 2021-07-31T04:15:41.921880 2021-07-31T04:15:43.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:43 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 6367 : cluster [DBG] pgmap v6462: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:43.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:43 smithi045 conmon[33655]: cluster 2021-07-31T04:15:41.921880+0000 mgr.x (mgr.24542) 6367 : cluster 2021-07-31T04:15:43.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:43 smithi045 conmon[33655]: [DBG] pgmap v6462: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:45.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:45 smithi154 conmon[34553]: cluster 2021-07-31T04:15:43.922465+0000 mgr.x (mgr.24542) 6368 : cluster [DBG] pgmap v6463: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:45.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:45 smithi045 conmon[39718]: cluster 2021-07-31T04:15:43 2021-07-31T04:15:45.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:45 smithi045 conmon[39718]: .922465+0000 mgr.x (mgr.24542) 6368 : cluster [DBG] pgmap v6463: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:45.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:45 smithi045 conmon[33655]: cluster 2021-07-31T04:15:43.922465+0000 2021-07-31T04:15:45.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:45 smithi045 conmon[33655]: mgr.x (mgr.24542) 6368 : cluster [DBG] pgmap v6463: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:47.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:47 smithi154 conmon[34553]: cluster 2021-07-31T04:15:45.922929+0000 mgr.x (mgr.24542) 6369 : cluster [DBG] pgmap v6464: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:47.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:47 smithi045 conmon[33655]: cluster 2021-07-31T04:15:45.922929+0000 mgr.x (mgr.24542) 6369 : cluster 2021-07-31T04:15:47.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:47 smithi045 conmon[33655]: [DBG] pgmap v6464: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:47.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:47 smithi045 conmon[39718]: cluster 2021-07-31T04:15:45. 2021-07-31T04:15:47.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:47 smithi045 conmon[39718]: 922929+0000 mgr.x (mgr.24542) 6369 : cluster [DBG] pgmap v6464: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:49.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:49 smithi154 conmon[34553]: cluster 2021-07-31T04:15:47.923425+0000 mgr.x (mgr.24542) 6370 : cluster [DBG] pgmap v6465: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:49.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:49 smithi045 conmon[39718]: cluster 2021-07-31T04:15:47.923425+0000 mgr.x (mgr.24542) 6370 : cluster [DBG] pgmap v6465: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:49.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:49 smithi045 conmon[33655]: cluster 2021-07-31T04:15:47.923425+0000 mgr.x (mgr.24542 2021-07-31T04:15:49.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:49 smithi045 conmon[33655]: ) 6370 : cluster [DBG] pgmap v6465: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:51.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:51 smithi154 conmon[34553]: cluster 2021-07-31T04:15:49.923834+0000 mgr.x (mgr.24542) 6371 : cluster [DBG] pgmap v6466: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:51.774 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:51 smithi045 conmon[39718]: cluster 2021-07-31T04:15:49.923834+0000 mgr.x (mgr.24542) 6371 : cluster [DBG] pgmap v6466: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:51.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:51 smithi045 conmon[33655]: cluster 2021-07-31T04:15:49.923834 2021-07-31T04:15:51.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:51 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 6371 : cluster [DBG] pgmap v6466: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:53.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:53 smithi154 conmon[34553]: cluster 2021-07-31T04:15:51.924364+0000 mgr.x (mgr.24542) 6372 : cluster [DBG] pgmap v6467: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:53.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:53 smithi045 conmon[33655]: cluster 2021-07-31T04:15:51.924364+0000 mgr.x (mgr.24542 2021-07-31T04:15:53.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:53 smithi045 conmon[33655]: ) 6372 : cluster [DBG] pgmap v6467: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:53.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:53 smithi045 conmon[39718]: cluster 2021-07-31T04:15:51.924364+0000 2021-07-31T04:15:53.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:53 smithi045 conmon[39718]: mgr.x (mgr.24542) 6372 : cluster [DBG] pgmap v6467: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:54.830 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T04:15:55.109 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T04:15:55.109 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 2h ago 2h - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T04:15:55.110 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 2h ago 2h 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T04:15:55.110 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 2h ago 2h 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T04:15:55.110 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 2h ago 2h - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T04:15:55.110 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T04:15:55.110 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 2h ago 2h 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T04:15:55.111 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T04:15:55.111 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 2h ago 2h - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T04:15:55.111 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 2h ago 2h 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T04:15:55.111 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T04:15:55.111 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T04:15:55.112 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T04:15:55.112 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T04:15:55.112 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 2h ago 2h 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T04:15:55.112 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 2h ago 2h 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T04:15:55.112 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 2h ago 2h 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T04:15:55.113 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 2h ago 2h 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T04:15:55.113 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 2h ago 2h 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T04:15:55.447 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T04:15:55.447 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T04:15:55.447 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T04:15:55.447 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:15:55.448 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T04:15:55.448 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T04:15:55.448 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:15:55.448 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:15:55.448 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T04:15:55.449 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T04:15:55.449 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:15:55.449 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T04:15:55.449 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T04:15:55.449 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T04:15:55.450 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:15:55.450 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T04:15:55.450 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T04:15:55.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:55 smithi154 conmon[34553]: cluster 2021-07-31T04:15:53.924879+0000 mgr.x (mgr.24542) 6373 : cluster [DBG] pgmap v6468: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:55.774 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:55 smithi045 conmon[33655]: cluster 2021-07-31T04:15:53.924879+0000 mgr.x (mgr.24542 2021-07-31T04:15:55.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:55 smithi045 conmon[33655]: ) 6373 : cluster [DBG] pgmap v6468: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:55.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:55 smithi045 conmon[39718]: cluster 2021-07-31T04: 2021-07-31T04:15:55.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:55 smithi045 conmon[39718]: 15:53.924879+0000 mgr.x (mgr.24542) 6373 : cluster [DBG] pgmap v6468: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:56.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:56 smithi154 conmon[34553]: audit 2021-07-31T04:15:54.819591+0000 mgr.x (mgr.24542) 6374 : audit [DBG] from='client.21234 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:15:56.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:56 smithi154 conmon[34553]: audit 2021-07-31T04:15:55.101951+0000 mgr.x (mgr.24542) 6375 : audit [DBG] from='client.21240 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:15:56.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:56 smithi154 conmon[34553]: audit 2021-07-31T04:15:55.448132+0000 mon.a (mon.0) 2021-07-31T04:15:56.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:56 smithi154 conmon[34553]: 3449 : audit [DBG] from='client.? 172.21.15.45:0/1762156071' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:15:56.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:56 smithi045 conmon[39718]: audit 2021-07-31T04:15:54.819591+0000 2021-07-31T04:15:56.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:56 smithi045 conmon[39718]: mgr.x (mgr.24542) 6374 : audit [DBG] from='client.21234 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:15:56.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:56 smithi045 conmon[39718]: audit 2021- 2021-07-31T04:15:56.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:56 smithi045 conmon[39718]: 07-31T04:15:55.101951+0000 mgr.x (mgr.24542) 2021-07-31T04:15:56.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:56 smithi045 conmon[39718]: 6375 : audit [DBG] from='client.21240 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:15:56.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:56 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T04:15:56.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:56 smithi045 conmon[39718]: 31T04:15:55.448132+0000 2021-07-31T04:15:56.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:56 smithi045 conmon[39718]: mon.a (mon.0) 3449 : audit [DBG] 2021-07-31T04:15:56.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:56 smithi045 conmon[39718]: from='client.? 172.21.15.45:0/1762156071' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:15:56.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:56 smithi045 conmon[33655]: audit 2021-07-31T04:15:54.819591+0000 mgr.x (mgr.24542) 6374 : audit 2021-07-31T04:15:56.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:56 smithi045 conmon[33655]: [DBG] from='client.21234 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:15:56.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:56 smithi045 conmon[33655]: audit 2021-07-31T04:15:55.101951+0000 mgr.x (mgr.24542) 6375 : audit [DBG] from='client.21240 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:15:56.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:56 smithi045 conmon[33655]: audit 2021-07-31T04:15:55.448132+0000 2021-07-31T04:15:56.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:56 smithi045 conmon[33655]: mon.a (mon.0) 3449 : audit [DBG] 2021-07-31T04:15:56.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:56 smithi045 conmon[33655]: from='client.? 172.21.15.45:0/1762156071' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:15:57.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:57 smithi154 conmon[34553]: cluster 2021-07-31T04:15:55.925332+0000 mgr.x (mgr.24542) 6376 : cluster [DBG] pgmap v6469: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:57.774 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:57 smithi045 conmon[39718]: cluster 2021-07-31T04:15:55.925332+0000 mgr.x (mgr.24542) 6376 : cluster [DBG] pgmap v6469: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:57.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:57 smithi045 conmon[33655]: cluster 2021-07-31T04:15:55.925332+0000 mgr.x (mgr.24542) 6376 : 2021-07-31T04:15:57.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:57 smithi045 conmon[33655]: cluster [DBG] pgmap v6469: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:59.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:15:59 smithi154 conmon[34553]: cluster 2021-07-31T04:15:57.925752+0000 mgr.x (mgr.24542) 6377 : cluster [DBG] pgmap v6470: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:59.774 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:59 smithi045 conmon[33655]: cluster 2021-07-31T04:15:57.925752+0000 mgr.x (mgr.24542) 6377 2021-07-31T04:15:59.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:15:59 smithi045 conmon[33655]: : cluster [DBG] pgmap v6470: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:15:59.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:59 smithi045 conmon[39718]: cluster 2021-07-31T04:15: 2021-07-31T04:15:59.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:15:59 smithi045 conmon[39718]: 57.925752+0000 mgr.x (mgr.24542) 6377 : cluster [DBG] pgmap v6470: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:01.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:01 smithi154 conmon[34553]: cluster 2021-07-31T04:15:59.926198+0000 mgr.x (mgr.24542) 6378 : cluster 2021-07-31T04:16:01.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:01 smithi154 conmon[34553]: [DBG] pgmap v6471: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:01.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:01 smithi045 conmon[33655]: cluster 2021-07-31T04:15:59.926198+0000 mgr.x (mgr.24542) 2021-07-31T04:16:01.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:01 smithi045 conmon[33655]: 6378 : cluster [DBG] pgmap v6471: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:01.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:01 smithi045 conmon[39718]: cluster 2021-07-31T04:15:59.926198+0000 mgr.x (mgr 2021-07-31T04:16:01.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:01 smithi045 conmon[39718]: .24542) 6378 : cluster [DBG] pgmap v6471: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:03.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:03 smithi154 conmon[34553]: cluster 2021-07-31T04:16:01.926672+0000 mgr.x (mgr.24542) 6379 : cluster [DBG] 2021-07-31T04:16:03.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:03 smithi154 conmon[34553]: pgmap v6472: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:03.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:03 smithi045 conmon[39718]: cluster 2021-07-31T04:16:01.926672+0000 mgr.x (mgr 2021-07-31T04:16:03.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:03 smithi045 conmon[39718]: .24542) 6379 : cluster [DBG] pgmap v6472: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:03.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:03 smithi045 conmon[33655]: cluster 2021-07-31T04:16:01.926672+0000 mgr.x (mgr.24542) 6379 : cluster 2021-07-31T04:16:03.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:03 smithi045 conmon[33655]: [DBG] pgmap v6472: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:05.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:05 smithi154 conmon[34553]: cluster 2021-07-31T04:16:03.927234+0000 mgr.x (mgr.24542) 6380 : cluster [DBG] pgmap v6473: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:05.774 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:05 smithi045 conmon[39718]: cluster 2021-07-31T04:16:03.927234+0000 mgr.x (mgr.24542) 6380 : cluster [DBG] pgmap v6473: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:05.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:05 smithi045 conmon[33655]: cluster 2021-07-31T04:16:03.927234+0000 mgr.x (mgr.24542) 6380 : cluster [DBG] 2021-07-31T04:16:05.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:05 smithi045 conmon[33655]: pgmap v6473: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:07.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:07 smithi154 conmon[34553]: cluster 2021-07-31T04:16:05.927716+0000 mgr.x (mgr.24542) 6381 : cluster [DBG] pgmap v6474: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:07.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:07 smithi045 conmon[39718]: cluster 2021-07-31T04:16:05.927716+0000 mgr.x ( 2021-07-31T04:16:07.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:07 smithi045 conmon[39718]: mgr.24542) 6381 : cluster [DBG] pgmap v6474: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:07.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:07 smithi045 conmon[33655]: cluster 2021-07-31T04:16:05.927716+0000 mgr.x (mgr.24542) 2021-07-31T04:16:07.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:07 smithi045 conmon[33655]: 6381 : cluster [DBG] pgmap v6474: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:09.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:09 smithi154 conmon[34553]: cluster 2021-07-31T04:16:07.928119+0000 mgr.x (mgr.24542) 6382 : cluster [DBG] pgmap v6475: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:09.774 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:09 smithi045 conmon[39718]: cluster 2021-07-31T04:16:07.928119+0000 mgr.x 2021-07-31T04:16:09.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:09 smithi045 conmon[39718]: (mgr.24542) 6382 : cluster [DBG] pgmap v6475: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:09.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:09 smithi045 conmon[33655]: cluster 2021-07-31T04:16:07.928119+0000 mgr.x (mgr 2021-07-31T04:16:09.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:09 smithi045 conmon[33655]: .24542) 6382 : cluster [DBG] pgmap v6475: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:11.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:11 smithi154 conmon[34553]: cluster 2021-07-31T04:16:09.928571+0000 mgr.x (mgr.24542) 6383 : cluster [DBG] pgmap v6476: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:11.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:11 smithi045 conmon[39718]: cluster 2021-07-31T04:16:09.928571+0000 mgr.x (mgr. 2021-07-31T04:16:11.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:11 smithi045 conmon[39718]: 24542) 6383 : cluster [DBG] pgmap v6476: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:11.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:11 smithi045 conmon[33655]: cluster 2021-07-31T04:16:09.928571+0000 mgr.x (mgr.24542) 6383 : cluster [DBG] pgmap v6476: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:13.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:13 smithi154 conmon[34553]: cluster 2021-07-31T04:16:11.929034+0000 mgr.x (mgr.24542) 6384 : cluster [DBG] pgmap v6477: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:13.774 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:13 smithi045 conmon[33655]: cluster 2021-07-31T04:16:11.929034+0000 mgr.x (mgr.24542) 2021-07-31T04:16:13.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:13 smithi045 conmon[33655]: 6384 : cluster [DBG] pgmap v6477: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:13.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:13 smithi045 conmon[39718]: cluster 2021-07-31T04:16:11.929034+0000 mgr.x (mgr 2021-07-31T04:16:13.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:13 smithi045 conmon[39718]: .24542) 6384 : cluster [DBG] pgmap v6477: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:15.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:15 smithi154 conmon[34553]: cluster 2021-07-31T04:16:13.929431+0000 mgr.x (mgr.24542) 6385 : 2021-07-31T04:16:15.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:15 smithi154 conmon[34553]: cluster [DBG] pgmap v6478: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:15.774 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:15 smithi045 conmon[39718]: cluster 2021-07-31T04:16:13.929431+0000 mgr.x (mgr.24542) 6385 : cluster [DBG] pgmap v6478: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:15.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:15 smithi045 conmon[33655]: cluster 2021-07-31T04:16:13.929431+0000 mgr.x (mgr. 2021-07-31T04:16:15.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:15 smithi045 conmon[33655]: 24542) 6385 : cluster [DBG] pgmap v6478: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:17.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:17 smithi154 conmon[34553]: cluster 2021-07-31T04:16:15.929916+0000 mgr.x (mgr.24542) 6386 : cluster [DBG] pgmap v6479: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:17.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:17 smithi154 conmon[34553]: 2021-07-31T04:16:17.774 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:17 smithi045 conmon[39718]: cluster 2021-07-31T04:16:15.929916+0000 mgr.x ( 2021-07-31T04:16:17.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:17 smithi045 conmon[39718]: mgr.24542) 6386 : cluster [DBG] pgmap v6479: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:17.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:17 smithi045 conmon[33655]: cluster 2021-07-31T04:16:15.929916+0000 mgr.x (mgr.24542) 6386 : cluster [DBG] 2021-07-31T04:16:17.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:17 smithi045 conmon[33655]: pgmap v6479: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:19.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:19 smithi154 conmon[34553]: cluster 2021-07-31T04:16:17.930378+0000 mgr.x (mgr.24542) 6387 : cluster [DBG] pgmap v6480: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:19.774 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:19 smithi045 conmon[33655]: cluster 2021-07-31T04:16:17.930378+0000 mgr.x (mgr.24542) 2021-07-31T04:16:19.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:19 smithi045 conmon[33655]: 6387 : cluster [DBG] pgmap v6480: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:19.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:19 smithi045 conmon[39718]: cluster 2021-07-31T04:16:17.930378+0000 mgr.x (mgr.24542) 6387 2021-07-31T04:16:19.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:19 smithi045 conmon[39718]: : cluster [DBG] pgmap v6480: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:21.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:21 smithi154 conmon[34553]: cluster 2021-07-31T04:16:19.930861+0000 mgr.x (mgr.24542) 6388 : cluster [DBG] pgmap v6481: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:21.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:21 smithi045 conmon[33655]: cluster 2021-07-31T04:16:19.930861+0000 mgr.x (mgr.24542) 6388 : cluster 2021-07-31T04:16:21.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:21 smithi045 conmon[33655]: [DBG] pgmap v6481: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:21.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:21 smithi045 conmon[39718]: cluster 2021-07-31T04:16:19.930861+0000 mgr.x (mgr 2021-07-31T04:16:21.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:21 smithi045 conmon[39718]: .24542) 6388 : cluster [DBG] pgmap v6481: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:23.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:23 smithi154 conmon[34553]: cluster 2021-07-31T04:16:21.931284+0000 mgr.x (mgr.24542) 6389 : cluster [DBG] pgmap v6482: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:23.774 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:23 smithi045 conmon[39718]: cluster 2021-07-31T04:16:21.931284+0000 mgr.x (mgr.24542 2021-07-31T04:16:23.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:23 smithi045 conmon[39718]: ) 6389 : cluster [DBG] pgmap v6482: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:23.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:23 smithi045 conmon[33655]: cluster 2021-07-31T04:16:21.931284+0000 mgr.x (mgr.24542) 6389 2021-07-31T04:16:23.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:23 smithi045 conmon[33655]: : cluster [DBG] pgmap v6482: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:25.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:25 smithi154 conmon[34553]: cluster 2021-07-31T04:16:23.931832+0000 mgr.x (mgr.24542) 6390 : cluster [DBG] pgmap v6483: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:25.760 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T04:16:25.774 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:25 smithi045 conmon[33655]: cluster 2021-07-31T04:16:23.931832 2021-07-31T04:16:25.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:25 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 6390 : cluster [DBG] pgmap v6483: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:25.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:25 smithi045 conmon[39718]: cluster 2021-07-31T04:16:23.931832+0000 mgr.x (mgr. 2021-07-31T04:16:25.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:25 smithi045 conmon[39718]: 24542) 6390 : cluster [DBG] pgmap v6483: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:26.036 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T04:16:26.037 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 2h ago 2h - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T04:16:26.037 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 2h ago 2h 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T04:16:26.037 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 2h ago 2h 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T04:16:26.037 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 2h ago 2h - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T04:16:26.037 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T04:16:26.038 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 2h ago 2h 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T04:16:26.038 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T04:16:26.038 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 2h ago 2h - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T04:16:26.038 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 2h ago 2h 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T04:16:26.038 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T04:16:26.039 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T04:16:26.039 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T04:16:26.039 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T04:16:26.039 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 2h ago 2h 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T04:16:26.039 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 2h ago 2h 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T04:16:26.040 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 2h ago 2h 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T04:16:26.040 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 2h ago 2h 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T04:16:26.040 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 2h ago 2h 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T04:16:26.372 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T04:16:26.372 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T04:16:26.372 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T04:16:26.372 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:16:26.372 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T04:16:26.373 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T04:16:26.373 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:16:26.373 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:16:26.373 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T04:16:26.373 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T04:16:26.374 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:16:26.374 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T04:16:26.374 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T04:16:26.374 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T04:16:26.374 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:16:26.375 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T04:16:26.375 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T04:16:27.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:27 smithi154 conmon[34553]: audit 2021-07-31T04:16:25.749365+0000 mgr.x (mgr.24542) 6391 : audit [DBG] 2021-07-31T04:16:27.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:27 smithi154 conmon[34553]: from='client.21252 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:16:27.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:27 smithi154 conmon[34553]: cluster 2021-07-31T04:16:25.932201+0000 mgr.x (mgr.24542) 6392 : cluster [DBG] pgmap v6484: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:27.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:27 smithi154 conmon[34553]: audit 2021-07-31T04:16:26.029559+0000 mgr.x (mgr.24542) 6393 : audit [DBG] 2021-07-31T04:16:27.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:27 smithi154 conmon[34553]: from='client.31195 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:16:27.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:27 smithi154 conmon[34553]: audit 2021-07-31T04:16:26.372988+0000 mon.c (mon.1) 261 : audit [DBG] from='client.? 172.21.15.45:0/851131533' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:16:27.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:27 smithi045 conmon[33655]: audit 2021-07-31T04:16:25.749365+0000 mgr.x (mgr.24542) 6391 : 2021-07-31T04:16:27.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:27 smithi045 conmon[33655]: audit [DBG] from='client.21252 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:16:27.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:27 smithi045 conmon[33655]: cluster 2021-07-31T04:16:25.932201+0000 mgr.x (mgr.24542) 6392 : cluster [DBG] pgmap v6484: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:27.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:27 smithi045 conmon[33655]: audit 2021-07-31T04:16:26.029559+0000 mgr.x (mgr.24542) 6393 : audit [DBG] from='client.31195 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:16:27.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:27 smithi045 conmon[33655]: audit 2021 2021-07-31T04:16:27.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:27 smithi045 conmon[33655]: -07-31T04:16:26.372988+0000 mon.c (mon.1) 2021-07-31T04:16:27.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:27 smithi045 conmon[33655]: 261 : audit [DBG] from='client.? 172.21.15.45:0/851131533' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:16:27.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:27 smithi045 conmon[39718]: audit 2021-07-31T04:16:25 2021-07-31T04:16:27.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:27 smithi045 conmon[39718]: .749365+0000 mgr.x (mgr.24542) 6391 : audit [DBG] from='client.21252 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:16:27.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:27 smithi045 conmon[39718]: cluster 2021- 2021-07-31T04:16:27.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:27 smithi045 conmon[39718]: 07-31T04:16:25 2021-07-31T04:16:27.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:27 smithi045 conmon[39718]: .932201+0000 mgr.x (mgr. 2021-07-31T04:16:27.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:27 smithi045 conmon[39718]: 24542) 6392 : cluster [DBG] 2021-07-31T04:16:27.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:27 smithi045 conmon[39718]: pgmap v6484: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:27.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:27 smithi045 conmon[39718]: audit 2021- 2021-07-31T04:16:27.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:27 smithi045 conmon[39718]: 07-31T04:16:26. 2021-07-31T04:16:27.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:27 smithi045 conmon[39718]: 029559+0000 mgr.x (mgr.24542) 2021-07-31T04:16:27.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:27 smithi045 conmon[39718]: 6393 : audit [DBG] from='client.31195 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:16:27.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:27 smithi045 conmon[39718]: audit 2021 2021-07-31T04:16:27.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:27 smithi045 conmon[39718]: -07-31T04:16:26 2021-07-31T04:16:27.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:27 smithi045 conmon[39718]: .372988+0000 mon.c (mon. 2021-07-31T04:16:27.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:27 smithi045 conmon[39718]: 1) 261 : audit [DBG] 2021-07-31T04:16:27.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:27 smithi045 conmon[39718]: from='client.? 172.21.15.45:0/851131533' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:16:29.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:29 smithi154 conmon[34553]: cluster 2021-07-31T04:16:27.932731+0000 mgr.x (mgr.24542) 6394 : cluster [DBG] pgmap v6485: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:29.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:29 smithi154 conmon[34553]: audit 2021-07-31T04:16:28.523058+0000 mon.b (mon.2) 820 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:16:29.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:29 smithi154 conmon[34553]: audit 2021-07-31T04:16:28.523294+0000 mon.b (mon.2) 821 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:16:29.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:29 smithi154 conmon[34553]: audit 2021-07-31T04:16:29.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:29 smithi154 conmon[34553]: 2021-07-31T04:16:28.524076+0000 mon.a (mon.0) 3450 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:16:29.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:29 smithi154 conmon[34553]: audit 2021 2021-07-31T04:16:29.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:29 smithi154 conmon[34553]: -07-31T04:16:28.524463+0000 2021-07-31T04:16:29.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:29 smithi154 conmon[34553]: mon.a (mon.0) 3451 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:16:29.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:29 smithi045 conmon[33655]: cluster 2021-07-31T04:16:27.932731+0000 mgr.x ( 2021-07-31T04:16:29.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:29 smithi045 conmon[33655]: mgr.24542) 6394 : cluster [DBG] pgmap v6485: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:29.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:29 smithi045 conmon[33655]: audit 2021-07-31T04:16:28.523058+0000 mon.b (mon.2) 820 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:16:29.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:29 smithi045 conmon[33655]: audit 2021-07-31T04:16:28.523294+0000 mon.b (mon.2) 821 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:16:29.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:29 smithi045 conmon[33655]: audit 2021-07-31T04:16:28.524076+0000 mon.a (mon.0) 3450 : audit [INF] 2021-07-31T04:16:29.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:29 smithi045 conmon[33655]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:16:29.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:29 smithi045 conmon[33655]: audit 2021-07-31T04:16:28.524463+0000 mon.a (mon.0) 3451 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:16:29.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:29 smithi045 conmon[39718]: cluster 2021-07-31T04:16:27.932731+0000 2021-07-31T04:16:29.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:29 smithi045 conmon[39718]: mgr.x (mgr.24542) 6394 : cluster [DBG] pgmap v6485: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:29.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:29 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T04:16:29.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:29 smithi045 conmon[39718]: 31T04:16:28.523058+0000 mon.b (mon.2) 2021-07-31T04:16:29.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:29 smithi045 conmon[39718]: 820 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:16:29.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:29 smithi045 conmon[39718]: audit 2021- 2021-07-31T04:16:29.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:29 smithi045 conmon[39718]: 07-31T04:16:28.523294 2021-07-31T04:16:29.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:29 smithi045 conmon[39718]: +0000 mon.b (mon.2) 821 : audit 2021-07-31T04:16:29.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:29 smithi045 conmon[39718]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:16:29.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:29 smithi045 conmon[39718]: audit 2021- 2021-07-31T04:16:29.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:29 smithi045 conmon[39718]: 07-31T04:16:28.524076 2021-07-31T04:16:29.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:29 smithi045 conmon[39718]: +0000 mon.a (mon.0) 3450 : 2021-07-31T04:16:29.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:29 smithi045 conmon[39718]: audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:16:29.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:29 smithi045 conmon[39718]: audit 2021-07-31T04:16:28. 2021-07-31T04:16:29.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:29 smithi045 conmon[39718]: 524463+0000 mon.a (mon.0) 3451 : audit 2021-07-31T04:16:29.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:29 smithi045 conmon[39718]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:16:31.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:31 smithi154 conmon[34553]: cluster 2021-07-31T04:16:29.933317+0000 mgr.x (mgr.24542) 6395 : cluster 2021-07-31T04:16:31.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:31 smithi154 conmon[34553]: [DBG] pgmap v6486: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:31.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:31 smithi045 conmon[33655]: cluster 2021-07-31 2021-07-31T04:16:31.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:31 smithi045 conmon[33655]: T04:16:29.933317+0000 mgr.x (mgr.24542) 6395 : cluster [DBG] pgmap v6486: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:31.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:31 smithi045 conmon[39718]: cluster 2021-07-31T04:16:29.933317+0000 mgr.x (mgr.24542 2021-07-31T04:16:31.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:31 smithi045 conmon[39718]: ) 6395 : cluster [DBG] pgmap v6486: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:33.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:33 smithi154 conmon[34553]: cluster 2021-07-31T04:16:31.933828+0000 mgr.x (mgr.24542) 6396 2021-07-31T04:16:33.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:33 smithi154 conmon[34553]: : cluster [DBG] pgmap v6487: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:33.774 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:33 smithi045 conmon[33655]: cluster 2021-07-31T04:16:31.933828+0000 mgr.x (mgr.24542) 6396 : cluster [DBG] pgmap v6487: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:33.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:33 smithi045 conmon[39718]: cluster 2021-07-31T04:16:31.933828 2021-07-31T04:16:33.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:33 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 6396 : cluster [DBG] pgmap v6487: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:35.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:35 smithi154 conmon[34553]: cluster 2021-07-31T04:16:33.934210+0000 mgr.x (mgr.24542) 6397 : cluster [DBG] pgmap v6488: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:35.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:35 smithi045 conmon[33655]: cluster 2021-07-31T04:16:33.934210 2021-07-31T04:16:35.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:35 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 6397 : cluster [DBG] pgmap v6488: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:35.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:35 smithi045 conmon[39718]: cluster 2021-07-31T04:16:33.934210+0000 mgr.x (mgr. 2021-07-31T04:16:35.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:35 smithi045 conmon[39718]: 24542) 6397 : cluster [DBG] pgmap v6488: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:37.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[34553]: cluster 2021-07-31T04:16:35.934587+0000 mgr.x (mgr.24542) 6398 : cluster [DBG] pgmap v6489: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:37.774 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:37 smithi045 conmon[33655]: cluster 2021-07-31T04:16:35.934587+0000 mgr.x (mgr.24542) 6398 : cluster [DBG] 2021-07-31T04:16:37.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:37 smithi045 conmon[33655]: pgmap v6489: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:37.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:37 smithi045 conmon[39718]: cluster 2021-07-31T04:16:35.934587+0000 mgr.x (mgr.24542) 2021-07-31T04:16:37.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:37 smithi045 conmon[39718]: 6398 : cluster [DBG] pgmap v6489: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:38.253 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: debug 2021-07-31T04:16:37.892+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:16:38.253 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:16:38.253 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:16:38.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: yield 2021-07-31T04:16:38.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:16:38.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:16:38.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:16:38.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:16:38.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:16:38.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:16:38.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:16:38.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:16:38.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:16:38.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: task.result() 2021-07-31T04:16:38.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:16:38.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: debug 2021-07-31T04:16:37.904+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:16:38.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:16:38.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:16:38.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: yield 2021-07-31T04:16:38.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:16:38.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:16:38.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:16:38.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:16:38.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:16:38.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:16:38.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:16:38.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:16:38.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:16:38.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: task.result() 2021-07-31T04:16:38.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:16:38.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: debug 2021-07-31T04:16:37.929+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:16:38.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:16:38.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:16:38.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: yield 2021-07-31T04:16:38.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:16:38.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:16:38.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:16:38.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:16:38.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:16:38.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:16:38.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:16:38.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:16:38.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:16:38.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: task.result() 2021-07-31T04:16:38.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:16:38.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: debug 2021-07-31T04:16:37.930+0000 7fcb9816a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:16:38.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:16:38.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:16:38.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: yield 2021-07-31T04:16:38.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:16:38.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:16:38.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:16:38.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:16:38.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:16:38.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:16:38.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:16:38.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:16:38.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:16:38.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: task.result() 2021-07-31T04:16:38.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:16:38.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: 2021-07-31T04:16:38.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T04:16:38.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: 2021-07-31T04:16:38.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:16:38.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:16:38.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: return f(*arg) 2021-07-31T04:16:38.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:16:38.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:16:38.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:16:38.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:16:38.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:16:38.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:16:38.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:16:38.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T04:16:38.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:16:38.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: raise self._exception 2021-07-31T04:16:38.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:16:38.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:16:38.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:16:38.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T04:16:38.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:16:38.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: raise 2021-07-31T04:16:38.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:16:38.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T04:16:38.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:16:38.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:16:38.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:16:38.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: debug 2021-07-31T04:16:37.938+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:16:38.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:16:38.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:16:38.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: yield 2021-07-31T04:16:38.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:16:38.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:16:38.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:16:38.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:16:38.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:16:38.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:16:38.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:16:38.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:16:38.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:16:38.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: task.result() 2021-07-31T04:16:38.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:16:38.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: debug 2021-07-31T04:16:37.938+0000 7fcb97969700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:16:38.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:16:38.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:16:38.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: yield 2021-07-31T04:16:38.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:16:38.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:16:38.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:16:38.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:16:38.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:16:38.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:16:38.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:16:38.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:16:38.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:16:38.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: task.result() 2021-07-31T04:16:38.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:16:38.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: 2021-07-31T04:16:38.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T04:16:38.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: 2021-07-31T04:16:38.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:16:38.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:16:38.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: return f(*arg) 2021-07-31T04:16:38.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:16:38.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:16:38.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:16:38.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:16:38.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:16:38.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:16:38.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:16:38.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T04:16:38.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:16:38.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: raise self._exception 2021-07-31T04:16:38.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:16:38.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:16:38.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:16:38.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T04:16:38.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:16:38.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: raise 2021-07-31T04:16:38.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:16:38.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T04:16:38.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:16:38.284 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:16:38.284 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:16:37 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:16:38.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:38 smithi154 conmon[34553]: audit 2021-07-31T04:16:37.847255+0000 mon.b (mon.2) 822 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T04:16:38.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:38 smithi154 conmon[34553]: audit 2021-07-31T04:16:37.848706+0000 mon.b (mon.2) 823 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T04:16:38.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:38 smithi154 conmon[34553]: audit 2021-07-31T 2021-07-31T04:16:38.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:38 smithi154 conmon[34553]: 04:16:37.849631+0000 mon.b (mon. 2021-07-31T04:16:38.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:38 smithi154 conmon[34553]: 2) 824 : audit 2021-07-31T04:16:38.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:38 smithi154 conmon[34553]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T04:16:38.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:38 smithi045 conmon[39718]: audit 2021-07-31T04:16:37.847255+0000 mon.b (mon.2) 822 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T04:16:38.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:38 smithi045 conmon[39718]: audit 2021-07-31T04:16:37.848706+0000 mon.b (mon.2) 2021-07-31T04:16:38.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:38 smithi045 conmon[39718]: 823 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T04:16:38.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:38 smithi045 conmon[39718]: audit 2021-07-31T04:16:38.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:38 smithi045 conmon[39718]: 2021-07-31T04:16 2021-07-31T04:16:38.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:38 smithi045 conmon[39718]: :37.849631+0000 mon.b (mon 2021-07-31T04:16:38.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:38 smithi045 conmon[39718]: .2) 824 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T04:16:38.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:38 smithi045 conmon[33655]: audit 2021-07-31T04:16:37.847255+0000 mon.b (mon.2) 822 : 2021-07-31T04:16:38.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:38 smithi045 conmon[33655]: audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T04:16:38.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:38 smithi045 conmon[33655]: audit 2021-07-31T04:16:37.848706 2021-07-31T04:16:38.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:38 smithi045 conmon[33655]: +0000 mon.b (mon.2) 823 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T04:16:38.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:38 smithi045 conmon[33655]: audit 2021-07-31T04:16:38.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:38 smithi045 conmon[33655]: 2021-07-31T04:16:37.849631+0000 mon.b ( 2021-07-31T04:16:38.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:38 smithi045 conmon[33655]: mon.2) 824 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T04:16:39.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:16:37.850413+0000 mgr.x (mgr.24542) 6399 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T04:16:39.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: cephadm 2021-07 2021-07-31T04:16:39.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: -31T04:16:37.893554+0000 mgr.x (mgr.24542) 6400 : cephadm [ERR] 2021-07-31T04:16:39.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:16:39.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:16:39.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: yield 2021-07-31T04:16:39.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:16:39.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:16:39.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:16:39.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:16:39.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:16:39.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:16:39.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:16:39.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:16:39.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:16:39.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: task.result() 2021-07-31T04:16:39.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:16:39.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:16:37.894059+0000 mgr.x (mgr.24542) 6401 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T04:16:39.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: 2021-07-31T04:16:39.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:16:37.905111+0000 mgr.x (mgr.24542) 6402 : cephadm [ERR] 2021-07-31T04:16:39.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:16:39.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:16:39.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: yield 2021-07-31T04:16:39.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:16:39.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:16:39.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:16:39.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:16:39.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:16:39.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:16:39.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:16:39.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:16:39.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:16:39.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: task.result() 2021-07-31T04:16:39.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:16:39.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: cephadm 2021- 2021-07-31T04:16:39.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: 07-31T04:16:37.905552+0000 mgr.x (mgr.24542) 6403 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T04:16:39.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:16:37.930728+0000 mgr.x (mgr.24542) 6404 : cephadm [ERR] 2021-07-31T04:16:39.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:16:39.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:16:39.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: yield 2021-07-31T04:16:39.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:16:39.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:16:39.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:16:39.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:16:39.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:16:39.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:16:39.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:16:39.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:16:39.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:16:39.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: task.result() 2021-07-31T04:16:39.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:16:39.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:16:37.931172+0000 mgr.x (mgr.24542) 6405 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:16:39.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:16:39.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:16:39.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: yield 2021-07-31T04:16:39.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:16:39.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:16:39.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:16:39.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:16:39.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:16:39.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:16:39.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:16:39.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:16:39.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:16:39.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: task.result() 2021-07-31T04:16:39.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:16:39.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: 2021-07-31T04:16:39.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T04:16:39.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: 2021-07-31T04:16:39.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:16:39.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:16:39.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: return f(*arg) 2021-07-31T04:16:39.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:16:39.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:16:39.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:16:39.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:16:39.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:16:39.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:16:39.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:16:39.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T04:16:39.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:16:39.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: raise self._exception 2021-07-31T04:16:39.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:16:39.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:16:39.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:16:39.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T04:16:39.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:16:39.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: raise 2021-07-31T04:16:39.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:16:39.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T04:16:39.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:16:39.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:16:39.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:16:39.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: cluster 2021-07-31T04:16:37.935120+0000 mgr.x (mgr.24542) 6406 : cluster [DBG] pgmap v6490: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:39.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:16:37.939218+0000 mgr.x (mgr.24542) 6407 : cephadm [ERR] 2021-07-31T04:16:39.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:16:39.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:16:39.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: yield 2021-07-31T04:16:39.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:16:39.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:16:39.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:16:39.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:16:39.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:16:39.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:16:39.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:16:39.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:16:39.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:16:39.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: task.result() 2021-07-31T04:16:39.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:16:39.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:16:37.939613+0000 mgr.x (mgr.24542) 6408 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:16:39.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:16:39.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:16:39.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: yield 2021-07-31T04:16:39.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:16:39.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:16:39.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:16:39.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:16:39.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:16:39.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:16:39.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:16:39.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:16:39.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:16:39.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: task.result() 2021-07-31T04:16:39.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:16:39.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: 2021-07-31T04:16:39.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T04:16:39.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: 2021-07-31T04:16:39.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:16:39.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:16:39.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: return f(*arg) 2021-07-31T04:16:39.782 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:16:39.782 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:16:39.782 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:16:39.782 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:16:39.783 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:16:39.783 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:16:39.783 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:16:39.783 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T04:16:39.783 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:16:39.784 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: raise self._exception 2021-07-31T04:16:39.784 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:16:39.784 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:16:39.784 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:16:39.785 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T04:16:39.785 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:16:39.785 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: raise 2021-07-31T04:16:39.785 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:16:39.785 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T04:16:39.786 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:16:39.786 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:16:39.786 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:39 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:16:39.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: cephadm 2021-07-31T04:16:37.850413+0000 mgr.x ( 2021-07-31T04:16:39.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: mgr.24542) 6399 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T04:16:39.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: cephadm 2021-07-31T04:16:39.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: 2021-07-31T04:16:37.893554+0000 2021-07-31T04:16:39.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: mgr.x (mgr.24542) 6400 2021-07-31T04:16:39.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: : cephadm [ERR] 2021-07-31T04:16:39.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:16:39.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:16:39.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: yield 2021-07-31T04:16:39.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:16:39.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:16:39.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:16:39.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:16:39.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:16:39.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:16:39.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:16:39.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:16:39.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:16:39.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: task.result() 2021-07-31T04:16:39.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:16:39.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: cephadm 2021-07-31T04:16:37.894059+0000 mgr.x (mgr.24542) 6401 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T04:16:39.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: cephadm 2021-07-31T04:16:39.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: 2021-07-31T04: 2021-07-31T04:16:39.792 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: 16:37.905111+0000 mgr.x ( 2021-07-31T04:16:39.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: mgr.24542) 6402 : cephadm [ERR] 2021-07-31T04:16:39.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:16:39.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:16:39.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: yield 2021-07-31T04:16:39.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:16:39.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:16:39.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:16:39.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:16:39.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:16:39.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:16:39.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:16:39.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:16:39.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:16:39.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: task.result() 2021-07-31T04:16:39.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:16:39.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: 2021-07-31T04:16:39.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: cephadm 2021-07-31T04:16:37.905552+0000 mgr.x (mgr.24542) 6403 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T04:16:39.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: cephadm 2021-07-31T04:16:39.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: 2021-07-31T04: 2021-07-31T04:16:39.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: 16:37.930728+0000 mgr.x ( 2021-07-31T04:16:39.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: mgr.24542) 6404 : cephadm [ERR] 2021-07-31T04:16:39.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:16:39.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:16:39.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: yield 2021-07-31T04:16:39.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:16:39.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:16:39.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:16:39.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:16:39.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:16:39.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:16:39.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:16:39.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:16:39.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:16:39.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: task.result() 2021-07-31T04:16:39.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:16:39.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: 2021-07-31T04:16:39.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: cephadm 2021-07-31T04:16:37.931172+0000 mgr.x (mgr.24542) 6405 : cephadm 2021-07-31T04:16:39.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:16:39.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:16:39.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:16:39.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: yield 2021-07-31T04:16:39.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:16:39.802 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:16:39.802 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:16:39.802 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:16:39.802 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:16:39.803 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:16:39.803 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:16:39.803 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:16:39.803 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:16:39.803 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: task.result() 2021-07-31T04:16:39.804 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:16:39.804 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: 2021-07-31T04:16:39.804 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T04:16:39.804 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: 2021-07-31T04:16:39.804 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:16:39.805 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:16:39.805 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: return f(*arg) 2021-07-31T04:16:39.805 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:16:39.805 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:16:39.806 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:16:39.806 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: cephadm 2021-07-31T04:16:37.850413+0000 mgr.x (mgr.24542) 2021-07-31T04:16:39.806 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: 6399 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T04:16:39.807 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: cephadm 2021-07-31T04:16:37.893554+0000 mgr.x (mgr.24542) 6400 : cephadm [ERR] 2021-07-31T04:16:39.807 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:16:39.807 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:16:39.807 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: yield 2021-07-31T04:16:39.807 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:16:39.808 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:16:39.808 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:16:39.808 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:16:39.808 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:16:39.809 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:16:39.809 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:16:39.809 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:16:39.809 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:16:39.809 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: task.result() 2021-07-31T04:16:39.810 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:16:39.810 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: cephadm 2021-07-31T04:16:37.894059+0000 mgr.x (mgr.24542) 6401 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T04:16:39.810 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: cephadm 2021-07-31T04 2021-07-31T04:16:39.810 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: :16:37.905111+0000 mgr.x (mgr.24542) 6402 : cephadm [ERR] 2021-07-31T04:16:39.811 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: 2021-07-31T04:16:39.811 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:16:39.811 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:16:39.811 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: yield 2021-07-31T04:16:39.811 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:16:39.812 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:16:39.812 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:16:39.812 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:16:39.812 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:16:39.812 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:16:39.813 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:16:39.813 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:16:39.813 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:16:39.813 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: task.result() 2021-07-31T04:16:39.814 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:16:39.814 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: cephadm 2021-07-31T04:16:37.905552+0000 mgr.x (mgr.24542) 6403 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T04:16:39.814 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: 2021-07-31T04:16:39.814 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: cephadm 2021- 2021-07-31T04:16:39.814 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: 07-31T04:16:37.930728 2021-07-31T04:16:39.815 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 6404 2021-07-31T04:16:39.815 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: : cephadm [ERR] 2021-07-31T04:16:39.815 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:16:39.816 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:16:39.816 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:16:39.816 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:16:39.816 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T04:16:39.816 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:16:39.817 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: raise self._exception 2021-07-31T04:16:39.817 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:16:39.817 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:16:39.817 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:16:39.818 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T04:16:39.818 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:16:39.818 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: raise 2021-07-31T04:16:39.818 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:16:39.818 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T04:16:39.819 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:16:39.819 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:16:39.819 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:16:39.819 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: cluster 2021-07-31T04:16:37.935120+0000 mgr.x (mgr.24542) 6406 : cluster [DBG] pgmap v6490: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:39.820 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: cephadm 2021-07-31T04:16:37.939218+0000 mgr.x (mgr.24542) 6407 : cephadm [ERR] 2021-07-31T04:16:39.820 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:16:39.820 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:16:39.820 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: yield 2021-07-31T04:16:39.820 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:16:39.821 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:16:39.821 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:16:39.821 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:16:39.821 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:16:39.821 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:16:39.822 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:16:39.822 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:16:39.822 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:16:39.822 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: task.result() 2021-07-31T04:16:39.823 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:16:39.823 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: cephadm 2021-07-31T04: 2021-07-31T04:16:39.823 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: 16:37.939613+0000 mgr.x (mgr.24542) 6408 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:16:39.823 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:16:39.823 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:16:39.824 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: yield 2021-07-31T04:16:39.824 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:16:39.824 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:16:39.825 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: yield 2021-07-31T04:16:39.825 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:16:39.825 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:16:39.825 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:16:39.826 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:16:39.826 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:16:39.826 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:16:39.826 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:16:39.827 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:16:39.827 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:16:39.827 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: task.result() 2021-07-31T04:16:39.827 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:16:39.827 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: cephadm 2021-07-31T04:16:37.931172+0000 mgr.x (mgr.24542 2021-07-31T04:16:39.828 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: ) 6405 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:16:39.828 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:16:39.828 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:16:39.828 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: yield 2021-07-31T04:16:39.828 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:16:39.829 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:16:39.829 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:16:39.829 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:16:39.829 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:16:39.830 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:16:39.830 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:16:39.830 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:16:39.830 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:16:39.830 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: task.result() 2021-07-31T04:16:39.831 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:16:39.831 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: 2021-07-31T04:16:39.831 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T04:16:39.831 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: 2021-07-31T04:16:39.831 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:16:39.832 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:16:39.832 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: return f(*arg) 2021-07-31T04:16:39.832 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:16:39.832 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:16:39.833 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:16:39.833 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:16:39.833 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:16:39.833 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:16:39.834 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:16:39.834 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T04:16:39.834 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:16:39.834 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: raise self._exception 2021-07-31T04:16:39.835 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:16:39.835 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:16:39.835 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:16:39.835 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T04:16:39.836 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:16:39.836 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: raise 2021-07-31T04:16:39.836 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:16:39.836 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T04:16:39.837 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:16:39.837 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:16:39.837 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:16:39.837 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: cluster 2021-07-31T04:16:37.935120+0000 mgr.x (mgr.24542) 6406 : cluster [DBG] pgmap v6490: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:39.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: cephadm 2021-07-31T04:16:37.939218+0000 mgr.x (mgr.24542) 6407 : cephadm [ERR] 2021-07-31T04:16:39.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:16:39.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:16:39.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: yield 2021-07-31T04:16:39.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:16:39.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:16:39.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:16:39.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:16:39.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:16:39.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:16:39.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:16:39.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:16:39.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:16:39.841 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: task.result() 2021-07-31T04:16:39.841 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:16:39.841 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: cephadm 2021-07-31T04:16:37.939613+0000 mgr.x (mgr.24542) 6408 : cephadm [ERR] 2021-07-31T04:16:39.841 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:16:39.842 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:16:39.842 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:16:39.842 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: yield 2021-07-31T04:16:39.842 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:16:39.843 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:16:39.843 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:16:39.843 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:16:39.844 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:16:39.844 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:16:39.844 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:16:39.844 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:16:39.845 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:16:39.845 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:16:39.845 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: task.result() 2021-07-31T04:16:39.845 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:16:39.845 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: 2021-07-31T04:16:39.846 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T04:16:39.846 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: 2021-07-31T04:16:39.846 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:16:39.846 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:16:39.847 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: return f(*arg) 2021-07-31T04:16:39.847 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:16:39.847 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:16:39.847 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:16:39.847 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:16:39.848 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:16:39.848 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:16:39.848 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:16:39.848 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T04:16:39.849 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:16:39.849 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: raise self._exception 2021-07-31T04:16:39.849 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:16:39.849 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:16:39.849 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:16:39.850 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T04:16:39.850 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:16:39.850 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: raise 2021-07-31T04:16:39.850 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:16:39.850 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T04:16:39.851 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:16:39.851 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:16:39.851 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:16:39.851 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:16:39.852 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:16:39.852 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:16:39.852 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:16:39.852 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:16:39.853 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:16:39.853 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:16:39.853 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:16:39.853 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: task.result() 2021-07-31T04:16:39.854 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:16:39.854 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: 2021-07-31T04:16:39.854 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T04:16:39.854 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: 2021-07-31T04:16:39.854 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:16:39.855 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:16:39.855 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: return f(*arg) 2021-07-31T04:16:39.855 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:16:39.855 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:16:39.856 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:16:39.856 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:16:39.856 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:16:39.856 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:16:39.856 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:16:39.857 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T04:16:39.857 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:16:39.857 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: raise self._exception 2021-07-31T04:16:39.857 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:16:39.857 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:16:39.858 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:16:39.858 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T04:16:39.858 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:16:39.858 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: raise 2021-07-31T04:16:39.859 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:16:39.859 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T04:16:39.859 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:16:39.859 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:16:39.859 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:39 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:16:41.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:41 smithi154 conmon[34553]: cluster 2021-07-31T04:16:39.935616+0000 mgr.x (mgr.24542) 6409 : cluster [DBG] pgmap v6491: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:41.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:41 smithi045 conmon[33655]: cluster 2021-07-31T04:16:39.935616+0000 mgr.x (mgr.24542) 6409 : cluster [DBG] pgmap v6491: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:41.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:41 smithi045 conmon[39718]: cluster 2021-07-31T04:16:39 2021-07-31T04:16:41.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:41 smithi045 conmon[39718]: .935616+0000 mgr.x (mgr.24542) 6409 : cluster [DBG] pgmap v6491: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:43.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:43 smithi154 conmon[34553]: cluster 2021-07-31T04:16:41.936102+0000 mgr.x (mgr.24542) 6410 : cluster [DBG] pgmap v6492: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:43.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:43 smithi045 conmon[39718]: cluster 2021-07-31T04:16:41.936102+0000 mgr.x (mgr. 2021-07-31T04:16:43.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:43 smithi045 conmon[39718]: 24542) 6410 : cluster [DBG] pgmap v6492: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:43.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:43 smithi045 conmon[33655]: cluster 2021-07-31T04:16:41.936102+0000 mgr.x (mgr.24542 2021-07-31T04:16:43.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:43 smithi045 conmon[33655]: ) 6410 : cluster [DBG] pgmap v6492: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:45.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:45 smithi154 conmon[34553]: cluster 2021-07-31T04:16:43.936676+0000 mgr.x (mgr.24542) 6411 : cluster [DBG] pgmap v6493: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:45.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:45 smithi045 conmon[39718]: cluster 2021-07-31T04:16:43.936676+0000 mgr.x (mgr.24542) 6411 : cluster [DBG] pgmap v6493: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:45.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:45 smithi045 conmon[39718]: 2021-07-31T04:16:45.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:45 smithi045 conmon[33655]: cluster 2021-07-31T04:16:43.936676+0000 mgr.x (mgr.24542) 6411 : 2021-07-31T04:16:45.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:45 smithi045 conmon[33655]: cluster [DBG] pgmap v6493: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:47.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:47 smithi154 conmon[34553]: cluster 2021-07-31T04:16:45.937282+0000 mgr.x (mgr.24542) 6412 : cluster [DBG] pgmap v6494: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:47.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:47 smithi045 conmon[33655]: cluster 2021-07-31T04:16:45.937282+0000 mgr.x (mgr.24542) 6412 2021-07-31T04:16:47.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:47 smithi045 conmon[33655]: : cluster [DBG] pgmap v6494: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:47.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:47 smithi045 conmon[39718]: cluster 2021-07-31T04:16:45.937282+0000 mgr.x (mgr.24542 2021-07-31T04:16:47.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:47 smithi045 conmon[39718]: ) 6412 : cluster [DBG] pgmap v6494: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:49.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:49 smithi154 conmon[34553]: cluster 2021-07-31T04:16:47.937788+0000 mgr.x (mgr.24542) 6413 : cluster [DBG] pgmap v6495: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:49.774 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:49 smithi045 conmon[33655]: cluster 2021- 2021-07-31T04:16:49.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:49 smithi045 conmon[33655]: 07-31T04:16:47.937788+0000 mgr.x (mgr.24542) 6413 : cluster [DBG] pgmap v6495: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:49.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:49 smithi045 conmon[39718]: cluster 2021-07-31T04:16:47.937788+0000 mgr.x (mgr.24542) 6413 : cluster [DBG] pgmap v6495: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:51.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:51 smithi154 conmon[34553]: cluster 2021-07-31T04:16:49.938287+0000 mgr.x (mgr.24542) 6414 : cluster [DBG] pgmap v6496: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:51.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:51 smithi045 conmon[33655]: cluster 2021-07-31T04:16:49.938287+0000 mgr.x (mgr. 2021-07-31T04:16:51.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:51 smithi045 conmon[33655]: 24542) 6414 : cluster [DBG] pgmap v6496: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:51.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:51 smithi045 conmon[39718]: cluster 2021-07-31T04:16:49.938287+0000 mgr.x (mgr.24542) 2021-07-31T04:16:51.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:51 smithi045 conmon[39718]: 6414 : cluster [DBG] pgmap v6496: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:53.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:53 smithi154 conmon[34553]: cluster 2021-07-31T04:16:51.938832+0000 mgr.x (mgr.24542) 6415 : cluster [DBG] 2021-07-31T04:16:53.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:53 smithi154 conmon[34553]: pgmap v6497: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:53.774 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:53 smithi045 conmon[33655]: cluster 2021-07-31T04:16:51.938832+0000 mgr.x (mgr.24542) 6415 : cluster [DBG] pgmap v6497: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:53.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:53 smithi045 conmon[39718]: cluster 2021-07-31T04:16:51.938832+0000 mgr.x (mgr.24542) 6415 2021-07-31T04:16:53.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:53 smithi045 conmon[39718]: : cluster [DBG] pgmap v6497: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:55.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:55 smithi154 conmon[34553]: cluster 2021-07-31T04:16:53.939435+0000 mgr.x (mgr.24542) 6416 : cluster [DBG] pgmap v6498: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:55.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:55 smithi045 conmon[33655]: cluster 2021-07-31T04:16:53.939435+0000 mgr.x (mgr.24542 2021-07-31T04:16:55.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:55 smithi045 conmon[33655]: ) 6416 : cluster [DBG] pgmap v6498: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:55.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:55 smithi045 conmon[39718]: cluster 2021-07-31T04:16:53.939435+0000 mgr.x (mgr.24542) 6416 2021-07-31T04:16:55.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:55 smithi045 conmon[39718]: : cluster [DBG] pgmap v6498: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:56.686 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T04:16:56.962 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T04:16:56.963 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 2h ago 2h - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T04:16:56.963 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 2h ago 2h 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T04:16:56.963 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 2h ago 2h 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T04:16:56.963 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 2h ago 2h - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T04:16:56.964 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T04:16:56.964 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 2h ago 2h 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T04:16:56.964 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T04:16:56.964 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 2h ago 2h - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T04:16:56.964 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 2h ago 2h 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T04:16:56.965 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T04:16:56.965 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T04:16:56.965 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T04:16:56.965 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T04:16:56.965 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 2h ago 2h 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T04:16:56.966 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 2h ago 2h 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T04:16:56.966 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 2h ago 2h 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T04:16:56.966 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 2h ago 2h 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T04:16:56.966 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 2h ago 2h 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T04:16:57.298 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T04:16:57.298 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T04:16:57.298 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T04:16:57.299 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:16:57.299 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T04:16:57.299 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T04:16:57.299 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:16:57.299 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:16:57.300 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T04:16:57.300 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T04:16:57.300 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:16:57.300 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T04:16:57.300 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T04:16:57.301 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T04:16:57.301 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:16:57.301 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T04:16:57.301 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T04:16:57.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:57 smithi154 conmon[34553]: cluster 2021-07-31T04:16:55.939893+0000 mgr.x (mgr.24542) 6417 : cluster [DBG] pgmap v6499: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:57.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:57 smithi154 conmon[34553]: audit 2021-07-31T04:16:57.299098+0000 mon.c (mon.1) 262 : audit [DBG] from='client.? 172.21.15.45:0/1386749845' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:16:57.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:57 smithi045 conmon[33655]: cluster 2021-07-31T04:16:55.939893+0000 2021-07-31T04:16:57.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:57 smithi045 conmon[33655]: mgr.x (mgr.24542) 6417 : cluster [DBG] pgmap v6499: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:57.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:57 smithi045 conmon[33655]: audit 2021-07-31T04:16:57.299098+0000 mon.c (mon.1) 262 2021-07-31T04:16:57.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:57 smithi045 conmon[33655]: : audit [DBG] from='client.? 172.21.15.45:0/1386749845' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:16:57.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:57 smithi045 conmon[39718]: cluster 2021-07-31T04:16:55. 2021-07-31T04:16:57.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:57 smithi045 conmon[39718]: 939893+0000 mgr.x (mgr.24542) 6417 : cluster [DBG] pgmap v6499: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:57.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:57 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T04:16:57.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:57 smithi045 conmon[39718]: T04:16:57.299098+0000 mon.c ( 2021-07-31T04:16:57.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:57 smithi045 conmon[39718]: mon.1) 262 : audit [DBG] from='client.? 172.21.15.45:0/1386749845' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:16:57.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:57 smithi045 conmon[39718]: 2021-07-31T04:16:58.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:58 smithi154 conmon[34553]: audit 2021-07-31T04:16:56.674808+0000 mgr.x (mgr.24542) 6418 : audit [DBG] from='client.31207 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:16:58.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:58 smithi154 conmon[34553]: audit 2021-07-31T04:16:56.955378+0000 mgr.x (mgr.24542) 6419 : audit [DBG] from='client.21276 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:16:58.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:58 smithi045 conmon[33655]: audit 2021-07-31T04:16:56.674808+0000 mgr.x (mgr.24542) 6418 : 2021-07-31T04:16:58.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:58 smithi045 conmon[33655]: audit [DBG] from='client.31207 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:16:58.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:58 smithi045 conmon[33655]: audit 2021-07-31T04:16:56.955378+0000 mgr.x (mgr.24542) 6419 : audit [DBG] from='client.21276 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:16:58.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:58 smithi045 conmon[39718]: audit 2021-07-31T04:16:56.674808+0000 mgr.x (mgr.24542) 2021-07-31T04:16:58.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:58 smithi045 conmon[39718]: 6418 : audit [DBG] from='client.31207 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:16:58.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:58 smithi045 conmon[39718]: audit 2021-07-31T04:16:56.955378+0000 mgr.x (mgr.24542 2021-07-31T04:16:58.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:58 smithi045 conmon[39718]: ) 6419 : audit [DBG] from='client.21276 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:16:59.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:16:59 smithi154 conmon[34553]: cluster 2021-07-31T04:16:57.940427+0000 mgr.x (mgr.24542) 6420 : cluster [DBG] pgmap v6500: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:59.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:16:59 smithi045 conmon[33655]: cluster 2021-07-31T04:16:57.940427+0000 mgr.x (mgr.24542) 6420 : cluster [DBG] pgmap v6500: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:16:59.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:59 smithi045 conmon[39718]: cluster 2021-07-31T04:16:57.940427+0000 mgr.x (mgr.24542) 6420 : cluster 2021-07-31T04:16:59.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:16:59 smithi045 conmon[39718]: [DBG] pgmap v6500: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:01.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:01 smithi154 conmon[34553]: cluster 2021-07-31T04:16:59.941027+0000 mgr.x (mgr.24542) 6421 : cluster [DBG] pgmap v6501: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:01.774 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:01 smithi045 conmon[39718]: cluster 2021-07-31T04:16:59. 2021-07-31T04:17:01.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:01 smithi045 conmon[39718]: 941027+0000 mgr.x (mgr.24542) 6421 : cluster [DBG] pgmap v6501: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:01.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:01 smithi045 conmon[33655]: cluster 2021-07-31T04:16:59.941027+0000 mgr.x (mgr.24542) 6421 : cluster 2021-07-31T04:17:01.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:01 smithi045 conmon[33655]: [DBG] pgmap v6501: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:03.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:03 smithi154 conmon[34553]: cluster 2021-07-31T04:17:01.941527+0000 mgr.x (mgr.24542) 6422 : cluster [DBG] pgmap v6502: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:03.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:03 smithi045 conmon[39718]: cluster 2021-07-31T04:17:01.941527+0000 mgr.x (mgr 2021-07-31T04:17:03.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:03 smithi045 conmon[39718]: .24542) 6422 : cluster [DBG] pgmap v6502: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:03.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:03 smithi045 conmon[33655]: cluster 2021-07-31T04:17:01.941527+0000 mgr.x (mgr.24542) 6422 : cluster [DBG] 2021-07-31T04:17:03.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:03 smithi045 conmon[33655]: pgmap v6502: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:05.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:05 smithi154 conmon[34553]: cluster 2021-07-31T04:17:03.942216+0000 mgr.x (mgr.24542) 6423 : cluster [DBG] pgmap v6503: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:05.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:05 smithi154 conmon[34553]: 2021-07-31T04:17:05.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:05 smithi045 conmon[33655]: cluster 2021-07-31T04:17:03.942216+0000 mgr.x (mgr.24542) 6423 : cluster [DBG] pgmap v6503: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:05.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:05 smithi045 conmon[39718]: cluster 2021-07-31T04:17:03.942216+0000 mgr.x (mgr. 2021-07-31T04:17:05.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:05 smithi045 conmon[39718]: 24542) 6423 : cluster [DBG] pgmap v6503: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:07.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:07 smithi154 conmon[34553]: cluster 2021-07-31T04:17:05.942667+0000 mgr.x (mgr.24542) 6424 : cluster [DBG] pgmap v6504: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:07.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:07 smithi045 conmon[39718]: cluster 2021-07-31T04:17:05.942667+0000 mgr.x (mgr 2021-07-31T04:17:07.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:07 smithi045 conmon[39718]: .24542) 6424 : cluster [DBG] pgmap v6504: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:07.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:07 smithi045 conmon[33655]: cluster 2021-07-31T04:17:05.942667+0000 mgr.x (mgr.24542) 6424 : cluster [DBG] pgmap v6504: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:07.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:07 smithi045 conmon[33655]: 2021-07-31T04:17:09.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:09 smithi154 conmon[34553]: cluster 2021-07-31T04:17:07.943077+0000 mgr.x (mgr.24542) 6425 : cluster [DBG] pgmap v6505: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:09.774 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:09 smithi045 conmon[39718]: cluster 2021-07-31T04:17:07.943077+0000 mgr.x (mgr.24542) 6425 : cluster [DBG] pgmap v6505: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:09.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:09 smithi045 conmon[33655]: cluster 2021-07-31T04:17:07.943077+0000 mgr.x (mgr.24542) 6425 : 2021-07-31T04:17:09.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:09 smithi045 conmon[33655]: cluster [DBG] pgmap v6505: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:11.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:11 smithi154 conmon[34553]: cluster 2021-07-31T04:17:09.943671+0000 mgr.x (mgr.24542) 6426 : 2021-07-31T04:17:11.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:11 smithi154 conmon[34553]: cluster [DBG] pgmap v6506: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:11.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:11 smithi045 conmon[39718]: cluster 2021-07-31T04:17:09.943671+0000 2021-07-31T04:17:11.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:11 smithi045 conmon[39718]: mgr.x (mgr.24542) 6426 : cluster [DBG] pgmap v6506: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:11.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:11 smithi045 conmon[33655]: cluster 2021-07-31T04:17:11.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:11 smithi045 conmon[33655]: 2021-07-31T04:17:09.943671+0000 mgr.x (mgr.24542) 6426 : cluster [DBG] pgmap v6506: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:13.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:13 smithi154 conmon[34553]: cluster 2021-07-31T04:17:11.944132+0000 mgr.x (mgr.24542) 6427 : cluster [DBG] 2021-07-31T04:17:13.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:13 smithi154 conmon[34553]: pgmap v6507: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:13.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:13 smithi045 conmon[39718]: cluster 2021-07-31T04:17:11.944132+0000 mgr.x (mgr.24542) 6427 : cluster 2021-07-31T04:17:13.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:13 smithi045 conmon[39718]: [DBG] pgmap v6507: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:13.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:13 smithi045 conmon[33655]: cluster 2021-07-31T04:17:11.944132+0000 2021-07-31T04:17:13.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:13 smithi045 conmon[33655]: mgr.x (mgr.24542) 6427 : cluster [DBG] pgmap v6507: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:15.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:15 smithi154 conmon[34553]: cluster 2021-07-31T04:17:13.944530+0000 mgr.x (mgr.24542) 6428 : cluster [DBG] pgmap v6508: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:15.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:15 smithi045 conmon[39718]: cluster 2021-07-31T04:17:13.944530+0000 mgr.x ( 2021-07-31T04:17:15.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:15 smithi045 conmon[39718]: mgr.24542) 6428 : cluster [DBG] pgmap v6508: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:15.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:15 smithi045 conmon[33655]: cluster 2021-07-31T04: 2021-07-31T04:17:15.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:15 smithi045 conmon[33655]: 17:13.944530+0000 mgr.x (mgr.24542) 6428 : cluster [DBG] pgmap v6508: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:17.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:17 smithi154 conmon[34553]: cluster 2021-07-31T04:17:15.945128+0000 mgr.x (mgr.24542) 6429 : cluster [DBG] pgmap v6509: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:17.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:17 smithi045 conmon[33655]: cluster 2021-07-31T04:17:15.945128+0000 mgr.x (mgr.24542) 6429 : cluster [DBG] pgmap v6509: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:17.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:17 smithi045 conmon[39718]: cluster 2021-07-31T04:17:15 2021-07-31T04:17:17.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:17 smithi045 conmon[39718]: .945128+0000 mgr.x (mgr.24542) 6429 : cluster [DBG] pgmap v6509: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:19.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:19 smithi154 conmon[34553]: cluster 2021-07-31T04:17:17.945557+0000 mgr.x (mgr.24542) 6430 : cluster [DBG] pgmap v6510: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:19.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:19 smithi045 conmon[33655]: cluster 2021-07-31T04:17:17.945557 2021-07-31T04:17:19.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:19 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 6430 : cluster [DBG] pgmap v6510: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:19.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:19 smithi045 conmon[39718]: cluster 2021-07-31T04:17: 2021-07-31T04:17:19.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:19 smithi045 conmon[39718]: 17.945557+0000 mgr.x (mgr.24542) 6430 : cluster [DBG] pgmap v6510: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:21.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:21 smithi154 conmon[34553]: cluster 2021-07-31T04:17:19.946023+0000 mgr.x (mgr.24542) 6431 : cluster [DBG] pgmap v6511: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:21.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:21 smithi045 conmon[33655]: cluster 2021-07-31T04:17:19.946023+0000 mgr.x (mgr.24542) 6431 2021-07-31T04:17:21.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:21 smithi045 conmon[33655]: : cluster [DBG] pgmap v6511: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:21.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:21 smithi045 conmon[39718]: cluster 2021-07-31T04:17: 2021-07-31T04:17:21.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:21 smithi045 conmon[39718]: 19.946023+0000 mgr.x (mgr.24542) 6431 : cluster [DBG] pgmap v6511: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:23.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:23 smithi154 conmon[34553]: cluster 2021-07-31T04:17:21.946542+0000 mgr.x (mgr.24542) 6432 : cluster [DBG] pgmap v6512: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:23.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:23 smithi045 conmon[39718]: cluster 2021-07-31T04:17:21.946542+0000 mgr.x (mgr 2021-07-31T04:17:23.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:23 smithi045 conmon[39718]: .24542) 6432 : cluster [DBG] pgmap v6512: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:23.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:23 smithi045 conmon[33655]: cluster 2021-07-31T04:17:21.946542+0000 mgr.x (mgr.24542) 6432 : cluster [DBG] pgmap v6512: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:25.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:25 smithi154 conmon[34553]: cluster 2021-07-31T04:17:23.947139+0000 mgr.x (mgr.24542) 6433 : cluster [DBG] pgmap v6513: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:25.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:25 smithi045 conmon[39718]: cluster 2021-07-31T04:17:23.947139+0000 mgr.x (mgr.24542) 6433 : cluster [DBG] pgmap v6513: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:25.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:25 smithi045 conmon[33655]: cluster 2021-07-31T04:17:23.947139+0000 2021-07-31T04:17:25.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:25 smithi045 conmon[33655]: mgr.x (mgr.24542) 6433 : cluster [DBG] pgmap v6513: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:27.613 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T04:17:27.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:27 smithi154 conmon[34553]: cluster 2021-07-31T04:17:25.947668+0000 mgr.x (mgr.24542) 6434 : cluster [DBG] pgmap v6514: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:27.774 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:27 smithi045 conmon[33655]: cluster 2021-07-31T04:17:25.947668+0000 mgr.x (mgr.24542) 6434 : 2021-07-31T04:17:27.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:27 smithi045 conmon[33655]: cluster [DBG] pgmap v6514: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:27.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:27 smithi045 conmon[39718]: cluster 2021-07-31T04:17:25.947668+0000 mgr.x ( 2021-07-31T04:17:27.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:27 smithi045 conmon[39718]: mgr.24542) 6434 : cluster [DBG] pgmap v6514: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:27.890 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T04:17:27.891 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 2h ago 2h - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T04:17:27.891 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 2h ago 2h 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T04:17:27.891 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 2h ago 2h 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T04:17:27.891 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 2h ago 2h - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T04:17:27.891 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T04:17:27.892 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 2h ago 2h 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T04:17:27.892 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T04:17:27.892 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 2h ago 2h - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T04:17:27.892 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 2h ago 2h 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T04:17:27.892 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T04:17:27.893 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T04:17:27.893 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T04:17:27.893 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T04:17:27.893 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 2h ago 2h 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T04:17:27.893 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 2h ago 2h 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T04:17:27.894 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 2h ago 2h 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T04:17:27.894 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 2h ago 2h 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T04:17:27.894 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 2h ago 2h 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T04:17:28.230 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T04:17:28.230 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T04:17:28.230 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T04:17:28.231 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:17:28.231 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T04:17:28.231 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T04:17:28.231 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:17:28.231 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:17:28.232 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T04:17:28.232 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T04:17:28.232 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:17:28.232 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T04:17:28.232 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T04:17:28.233 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T04:17:28.233 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:17:28.233 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T04:17:28.233 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T04:17:28.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:28 smithi154 conmon[34553]: audit 2021-07-31T04:17:28.231278+0000 mon.c (mon.1) 263 : audit [DBG] from='client.? 172.21.15.45:0/4253023554' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:17:28.774 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:28 smithi045 conmon[33655]: audit 2021-07-31T04:17:28.231278+0000 mon.c (mon.1) 2021-07-31T04:17:28.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:28 smithi045 conmon[33655]: 263 : audit [DBG] from='client.? 172.21.15.45:0/4253023554' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:17:28.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:28 smithi045 conmon[39718]: audit 2021-07-31T04:17:28.231278+0000 mon.c 2021-07-31T04:17:28.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:28 smithi045 conmon[39718]: (mon.1) 263 : audit [DBG] from='client.? 172.21.15.45:0/4253023554' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:17:29.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:29 smithi154 conmon[34553]: audit 2021-07-31T04:17:27.601248+0000 mgr.x (mgr.24542) 6435 : audit [DBG] from='client.21288 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:17:29.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:29 smithi154 conmon[34553]: audit 2021-07-31T04:17:27.883239+0000 mgr.x (mgr.24542) 6436 : audit [DBG] from='client.31231 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:17:29.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:29 smithi154 conmon[34553]: cluster 2021-07-31T04:17:27.948151+0000 mgr.x (mgr.24542) 6437 : cluster [DBG] pgmap v6515: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:29.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:29 smithi154 conmon[34553]: audit 2021-07-31T04:17:28.527518+0000 mon.b (mon.2) 825 : audit 2021-07-31T04:17:29.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:29 smithi154 conmon[34553]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:17:29.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:29 smithi154 conmon[34553]: audit 2021-07-31T04:17:29.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:29 smithi154 conmon[34553]: 2021-07-31T04:17:28.527750+0000 mon.b (mon. 2021-07-31T04:17:29.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:29 smithi154 conmon[34553]: 2) 826 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:17:29.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:29 smithi154 conmon[34553]: audit 2021-07-31 2021-07-31T04:17:29.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:29 smithi154 conmon[34553]: T04:17:28.528574+0000 mon.a (mon.0) 3452 : audit 2021-07-31T04:17:29.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:29 smithi154 conmon[34553]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:17:29.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:29 smithi154 conmon[34553]: audit 2021-07- 2021-07-31T04:17:29.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:29 smithi154 conmon[34553]: 31T04:17:28.528919+0000 mon.a (mon.0 2021-07-31T04:17:29.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:29 smithi154 conmon[34553]: ) 3453 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:17:29.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:29 smithi045 conmon[33655]: audit 2021-07-31T04:17:27.601248+0000 mgr.x (mgr.24542) 6435 : audit 2021-07-31T04:17:29.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:29 smithi045 conmon[33655]: [DBG] from='client.21288 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:17:29.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:29 smithi045 conmon[33655]: audit 2021-07-31T04:17:27.883239+0000 mgr.x (mgr.24542) 6436 : audit [DBG] from='client.31231 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:17:29.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:29 smithi045 conmon[33655]: cluster 2021-07-31T 2021-07-31T04:17:29.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:29 smithi045 conmon[33655]: 04:17:27.948151+0000 mgr.x (mgr.24542) 6437 : cluster [DBG] pgmap v6515: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:29.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:29 smithi045 conmon[33655]: audit 2021-07-31T04:17:28.527518+0000 mon.b ( 2021-07-31T04:17:29.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:29 smithi045 conmon[33655]: mon. 2021-07-31T04:17:29.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:29 smithi045 conmon[33655]: 2) 825 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:17:29.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:29 smithi045 conmon[33655]: audit 2021-07-31T 2021-07-31T04:17:29.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:29 smithi045 conmon[33655]: 04:17:28.527750+0000 mon.b (mon.2) 826 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:17:29.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:29 smithi045 conmon[33655]: audit 2021-07-31 2021-07-31T04:17:29.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:29 smithi045 conmon[33655]: T04:17:28.528574+0000 mon.a (mon.0) 3452 : audit 2021-07-31T04:17:29.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:29 smithi045 conmon[33655]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:17:29.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:29 smithi045 conmon[33655]: audit 2021-07-31T04:17:29.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:29 smithi045 conmon[33655]: 2021-07-31T04:17:28.528919+0000 mon.a (mon.0) 3453 : audit 2021-07-31T04:17:29.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:29 smithi045 conmon[33655]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:17:29.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:29 smithi045 conmon[39718]: audit 2021-07-31T04:17:27.601248+0000 mgr.x (mgr 2021-07-31T04:17:29.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:29 smithi045 conmon[39718]: .24542) 6435 : audit [DBG] from='client.21288 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:17:29.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:29 smithi045 conmon[39718]: audit 2021-07-31T04:17:27.883239 2021-07-31T04:17:29.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:29 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 6436 : audit [DBG] from='client.31231 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:17:29.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:29 smithi045 conmon[39718]: cluster 2021-07-31T04:17:29.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:29 smithi045 conmon[39718]: 2021-07-31T04:17: 2021-07-31T04:17:29.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:29 smithi045 conmon[39718]: 27.948151+0000 mgr.x (mgr.24542) 6437 : 2021-07-31T04:17:29.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:29 smithi045 conmon[39718]: cluster [DBG] pgmap v6515: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:29.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:29 smithi045 conmon[39718]: audit 2021 2021-07-31T04:17:29.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:29 smithi045 conmon[39718]: -07-31T04:17:28.527518+0000 mon.b (mon.2) 2021-07-31T04:17:29.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:29 smithi045 conmon[39718]: 825 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:17:29.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:29 smithi045 conmon[39718]: audit 2021-07-31T04:17:29.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:29 smithi045 conmon[39718]: 2021-07-31T04:17:28.527750+0000 mon.b (mon. 2021-07-31T04:17:29.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:29 smithi045 conmon[39718]: 2) 826 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:17:29.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:29 smithi045 conmon[39718]: audit 2021 2021-07-31T04:17:29.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:29 smithi045 conmon[39718]: -07-31T04:17:28.528574+0000 mon.a (mon.0) 3452 2021-07-31T04:17:29.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:29 smithi045 conmon[39718]: : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:17:29.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:29 smithi045 conmon[39718]: audit 2021-07-31T04:17:29.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:29 smithi045 conmon[39718]: 2021-07-31T04:17:28.528919+0000 2021-07-31T04:17:29.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:29 smithi045 conmon[39718]: mon.a (mon.0) 3453 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:17:29.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:29 smithi045 conmon[39718]: 2021-07-31T04:17:31.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:31 smithi154 conmon[34553]: cluster 2021-07-31T04:17:29.948572+0000 mgr.x (mgr.24542) 6438 : cluster [DBG] pgmap v6516: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:31.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:31 smithi045 conmon[39718]: cluster 2021-07-31T04:17:29.948572+0000 mgr.x (mgr.24542) 6438 : cluster 2021-07-31T04:17:31.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:31 smithi045 conmon[39718]: [DBG] pgmap v6516: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:31.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:31 smithi045 conmon[33655]: cluster 2021-07-31T04:17:29.948572+0000 mgr.x (mgr.24542) 6438 : cluster [DBG] pgmap v6516: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:33.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:33 smithi154 conmon[34553]: cluster 2021-07-31T04:17:31.949076+0000 mgr.x (mgr.24542) 6439 : cluster [DBG] pgmap v6517: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:33.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:33 smithi154 conmon[34553]: 2021-07-31T04:17:33.774 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:33 smithi045 conmon[39718]: cluster 2021-07-31T04:17:31.949076+0000 mgr.x (mgr.24542) 6439 : cluster [DBG] 2021-07-31T04:17:33.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:33 smithi045 conmon[39718]: pgmap v6517: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:33.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:33 smithi045 conmon[33655]: cluster 2021-07-31T04:17:31.949076+0000 mgr.x (mgr.24542) 6439 : cluster [DBG] pgmap v6517: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:35.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:35 smithi154 conmon[34553]: cluster 2021-07-31T04:17:33.949583+0000 mgr.x (mgr.24542) 6440 : cluster [DBG] pgmap v6518: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:35.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:35 smithi045 conmon[39718]: cluster 2021-07-31T04:17:33.949583+0000 mgr.x (mgr.24542) 6440 : 2021-07-31T04:17:35.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:35 smithi045 conmon[39718]: cluster [DBG] pgmap v6518: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:35.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:35 smithi045 conmon[33655]: cluster 2021-07-31T04:17:33.949583+0000 mgr.x (mgr.24542) 6440 : cluster [DBG] pgmap v6518: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:36.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:36 smithi154 conmon[34553]: cluster 2021-07-31T04:17:35.949967+0000 mgr.x (mgr.24542) 6441 : cluster [DBG] pgmap v6519: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:36.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:36 smithi045 conmon[39718]: cluster 2021-07-31T04:17:35.949967+0000 mgr.x (mgr.24542) 6441 : cluster [DBG] 2021-07-31T04:17:36.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:36 smithi045 conmon[39718]: pgmap v6519: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:36.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:36 smithi045 conmon[33655]: cluster 2021-07-31T04:17:35.949967+0000 mgr.x (mgr.24542) 6441 : cluster [DBG] 2021-07-31T04:17:36.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:36 smithi045 conmon[33655]: pgmap v6519: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:38.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:37 smithi154 conmon[34553]: audit 2021-07-31T04:17:37.940495+0000 mon.b (mon.2) 827 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T04:17:38.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:37 smithi154 conmon[34553]: audit 2021-07-31T04:17:37.941955+0000 mon.b (mon.2) 828 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T04:17:38.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:37 smithi154 conmon[34553]: audit 2021-07-31T04 2021-07-31T04:17:38.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:37 smithi154 conmon[34553]: :17:37.942836+0000 mon.b (mon. 2021-07-31T04:17:38.253 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:37 smithi154 conmon[34553]: 2) 829 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T04:17:38.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:37 smithi154 conmon[57492]: debug 2021-07-31T04:17:37.986+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:17:38.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:37 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:17:38.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:17:38.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:37 smithi154 conmon[57492]: yield 2021-07-31T04:17:38.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:17:38.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:37 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:17:38.255 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:17:38.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:37 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:17:38.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:17:38.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:37 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:17:38.256 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:17:38.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:37 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:17:38.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:17:38.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:37 smithi154 conmon[57492]: task.result() 2021-07-31T04:17:38.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:37 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:17:38.257 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:37 smithi154 conmon[57492]: debug 2021-07-31T04:17:37.998+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:17:38.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:37 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:17:38.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:17:38.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:37 smithi154 conmon[57492]: yield 2021-07-31T04:17:38.258 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:37 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:17:38.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:37 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:17:38.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:17:38.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:37 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:17:38.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:17:38.259 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:37 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:17:38.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:17:38.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:37 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:17:38.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:37 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:17:38.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:37 smithi154 conmon[57492]: task.result() 2021-07-31T04:17:38.260 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:37 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:17:38.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: debug 2021-07-31T04:17:38.024+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:17:38.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:17:38.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:17:38.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: yield 2021-07-31T04:17:38.261 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:17:38.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:17:38.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:17:38.262 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:17:38.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:17:38.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:17:38.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:17:38.263 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:17:38.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:17:38.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: task.result() 2021-07-31T04:17:38.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:17:38.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: debug 2021-07-31T04:17:38.025+0000 7fcb93961700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:17:38.264 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:17:38.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:17:38.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: yield 2021-07-31T04:17:38.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:17:38.265 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:17:38.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:17:38.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:17:38.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:17:38.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:17:38.266 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:17:38.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:17:38.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:17:38.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: task.result() 2021-07-31T04:17:38.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:17:38.267 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: 2021-07-31T04:17:38.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T04:17:38.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: 2021-07-31T04:17:38.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:17:38.268 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:17:38.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: return f(*arg) 2021-07-31T04:17:38.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:17:38.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:17:38.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:17:38.269 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:17:38.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:17:38.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:17:38.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:17:38.270 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T04:17:38.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:17:38.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: raise self._exception 2021-07-31T04:17:38.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:17:38.271 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:17:38.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:17:38.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T04:17:38.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:17:38.272 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: raise 2021-07-31T04:17:38.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:17:38.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T04:17:38.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:17:38.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:17:38.273 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:17:38.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: debug 2021-07-31T04:17:38.033+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:17:38.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:17:38.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:17:38.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: yield 2021-07-31T04:17:38.274 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:17:38.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:17:38.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:17:38.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:17:38.275 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:17:38.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:17:38.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:17:38.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:17:38.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:17:38.276 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: task.result() 2021-07-31T04:17:38.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:17:38.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: debug 2021-07-31T04:17:38.033+0000 7fcb94162700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:17:38.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:17:38.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:17:38.277 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: yield 2021-07-31T04:17:38.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:17:38.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:17:38.278 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:17:38.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:17:38.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:17:38.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:17:38.279 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:17:38.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:17:38.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:17:38.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: task.result() 2021-07-31T04:17:38.280 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:17:38.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: 2021-07-31T04:17:38.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T04:17:38.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: 2021-07-31T04:17:38.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:17:38.281 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:17:38.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: return f(*arg) 2021-07-31T04:17:38.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:17:38.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:17:38.282 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:17:38.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:17:38.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:17:38.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:17:38.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:17:38.283 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T04:17:38.284 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:17:38.284 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: raise self._exception 2021-07-31T04:17:38.284 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:17:38.284 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:17:38.284 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:17:38.285 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T04:17:38.285 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:17:38.285 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: raise 2021-07-31T04:17:38.285 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:17:38.285 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T04:17:38.286 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:17:38.286 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:17:38.286 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:17:38 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:17:38.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:37 smithi045 conmon[33655]: audit 2021-07-31T04:17:37.940495+0000 mon.b (mon.2) 827 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T04:17:38.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:37 smithi045 conmon[33655]: audit 2021-07-31T04:17:37.941955+0000 mon.b (mon.2) 828 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T04:17:38.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:37 smithi045 conmon[33655]: audit 2021-07-31T04:17:37. 2021-07-31T04:17:38.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:37 smithi045 conmon[33655]: 942836+0000 mon.b (mon.2) 829 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T04:17:38.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:37 smithi045 conmon[39718]: audit 2021-07-31T04:17:37.940495+0000 mon.b 2021-07-31T04:17:38.289 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:37 smithi045 conmon[39718]: (mon.2) 827 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T04:17:38.289 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:37 smithi045 conmon[39718]: audit 2021-07-31T04:17:37.941955+0000 mon.b ( 2021-07-31T04:17:38.289 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:37 smithi045 conmon[39718]: mon.2) 828 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T04:17:38.289 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:37 smithi045 conmon[39718]: audit 2021-07-31T04:17:37.942836+0000 2021-07-31T04:17:38.290 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:37 smithi045 conmon[39718]: mon.b (mon.2) 829 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T04:17:39.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: cephadm 2021-07-31T04:17:37.943612+0000 mgr.x (mgr.24542) 2021-07-31T04:17:39.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: 6442 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T04:17:39.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: cluster 2021-07-31T04:17:37.950321+0000 mgr.x (mgr.24542) 6443 : cluster [DBG] pgmap v6520: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:39.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: 2021-07-31T04:17:39.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: cephadm 2021-07-31T04:17:37.987114+0000 mgr.x (mgr.24542 2021-07-31T04:17:39.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: ) 6444 : cephadm [ERR] 2021-07-31T04:17:39.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:17:39.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:17:39.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: yield 2021-07-31T04:17:39.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:17:39.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:17:39.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:17:39.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:17:39.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:17:39.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:17:39.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:17:39.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:17:39.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:17:39.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: task.result() 2021-07-31T04:17:39.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:17:39.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: cephadm 2021-07 2021-07-31T04:17:39.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: -31T04:17:37.987649+0000 mgr.x (mgr.24542) 6445 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T04:17:39.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: cephadm 2021-07-31T04:17:37.999406+0000 mgr.x (mgr.24542) 6446 : cephadm [ERR] 2021-07-31T04:17:39.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:17:39.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:17:39.282 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: yield 2021-07-31T04:17:39.282 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:17:39.282 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:17:39.282 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:17:39.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:17:39.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:17:39.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:17:39.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:17:39.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:17:39.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:17:39.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: task.result() 2021-07-31T04:17:39.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:17:39.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: cephadm 2021-07-31 2021-07-31T04:17:39.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: T04:17:38.000335+0000 mgr.x (mgr.24542) 6447 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T04:17:39.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: cephadm 2021-07-31T04:17:38.025571+0000 mgr.x (mgr.24542) 6448 : cephadm [ERR] 2021-07-31T04:17:39.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:17:39.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:17:39.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: yield 2021-07-31T04:17:39.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:17:39.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:17:39.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:17:39.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:17:39.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:17:39.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:17:39.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:17:39.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:17:39.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:17:39.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: task.result() 2021-07-31T04:17:39.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:17:39.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: cephadm 2021-07-31T04:17:38.026792+0000 mgr.x (mgr.24542) 6449 : cephadm [ERR] 2021-07-31T04:17:39.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:17:39.289 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:17:39.289 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:17:39.289 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: yield 2021-07-31T04:17:39.289 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:17:39.290 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:17:39.290 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:17:39.290 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:17:39.290 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:17:39.291 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:17:39.291 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:17:39.291 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:17:39.291 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:17:39.291 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: task.result() 2021-07-31T04:17:39.292 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:17:39.292 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: 2021-07-31T04:17:39.292 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T04:17:39.292 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: 2021-07-31T04:17:39.293 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:17:39.293 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:17:39.293 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: return f(*arg) 2021-07-31T04:17:39.293 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:17:39.293 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:17:39.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:17:39.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:17:39.294 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: cephadm 2021-07-31T04:17:37.943612+0000 mgr.x (mgr.24542) 6442 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T04:17:39.295 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: cluster 2021-07-31T04:17:37.950321+0000 mgr.x (mgr.24542) 6443 : cluster [DBG] 2021-07-31T04:17:39.295 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: pgmap v6520: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:39.295 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: cephadm 2021 2021-07-31T04:17:39.295 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: -07-31T04:17:37. 2021-07-31T04:17:39.295 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: 987114+0000 mgr.x (mgr. 2021-07-31T04:17:39.296 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: 24542) 6444 : cephadm [ERR] 2021-07-31T04:17:39.296 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: 2021-07-31T04:17:39.296 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:17:39.296 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:17:39.297 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: yield 2021-07-31T04:17:39.297 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:17:39.297 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:17:39.297 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:17:39.297 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:17:39.298 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:17:39.298 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:17:39.298 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:17:39.298 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:17:39.299 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:17:39.299 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: task.result() 2021-07-31T04:17:39.299 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:17:39.299 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: cephadm 2021-07-31T04:17:39.300 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: 2021-07-31T04:17:37.987649+0000 mgr.x (mgr.24542) 6445 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T04:17:39.300 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: cephadm 2021-07-31T04:17:37.999406+0000 mgr.x (mgr.24542) 6446 : cephadm [ERR] 2021-07-31T04:17:39.300 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:17:39.300 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:17:39.300 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: yield 2021-07-31T04:17:39.301 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:17:39.301 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:17:39.301 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:17:39.301 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:17:39.302 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:17:39.302 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:17:39.302 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:17:39.302 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:17:39.302 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:17:39.303 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: task.result() 2021-07-31T04:17:39.303 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:17:39.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:17:39.304 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:17:39.304 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:17:39.304 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T04:17:39.304 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:17:39.305 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: raise self._exception 2021-07-31T04:17:39.305 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:17:39.305 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:17:39.305 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:17:39.305 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T04:17:39.306 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:17:39.306 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: raise 2021-07-31T04:17:39.306 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:17:39.306 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T04:17:39.307 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:17:39.307 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:17:39.307 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:17:39.307 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: cephadm 2021-07-31T04:17:38.034337+0000 mgr.x (mgr.24542) 6450 : cephadm [ERR] 2021-07-31T04:17:39.307 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:17:39.308 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:17:39.308 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: yield 2021-07-31T04:17:39.308 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:17:39.308 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:17:39.309 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:17:39.309 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:17:39.309 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:17:39.309 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:17:39.309 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:17:39.310 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:17:39.310 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:17:39.310 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: task.result() 2021-07-31T04:17:39.310 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:17:39.311 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: cephadm 2021-07-31T04:17:38. 2021-07-31T04:17:39.311 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: 034871+0000 mgr.x (mgr.24542) 6451 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:17:39.311 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:17:39.311 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:17:39.312 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: yield 2021-07-31T04:17:39.312 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:17:39.312 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:17:39.313 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: cephadm 2021-07-31T04:17:38.000335+0000 mgr.x (mgr.24542) 6447 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T04:17:39.313 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: cephadm 2021-07-31T04:17:38.025571+0000 mgr.x (mgr.24542) 6448 : cephadm [ERR] 2021-07-31T04:17:39.313 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:17:39.313 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:17:39.314 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: yield 2021-07-31T04:17:39.314 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:17:39.314 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:17:39.314 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:17:39.315 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:17:39.315 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:17:39.315 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:17:39.315 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:17:39.316 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:17:39.316 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:17:39.316 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: task.result() 2021-07-31T04:17:39.316 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:17:39.316 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: cephadm 2021-07-31T04:17:38.026792+0000 mgr.x (mgr.24542) 6449 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:17:39.317 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:17:39.317 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:17:39.317 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: yield 2021-07-31T04:17:39.317 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:17:39.318 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:17:39.318 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:17:39.318 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:17:39.318 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:17:39.319 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:17:39.319 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:17:39.319 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:17:39.319 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:17:39.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: task.result() 2021-07-31T04:17:39.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:17:39.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: 2021-07-31T04:17:39.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T04:17:39.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: 2021-07-31T04:17:39.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:17:39.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:17:39.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: return f(*arg) 2021-07-31T04:17:39.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:17:39.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:17:39.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:17:39.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:17:39.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:17:39.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:17:39.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:17:39.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T04:17:39.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:17:39.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: raise self._exception 2021-07-31T04:17:39.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:17:39.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:17:39.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:17:39.324 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T04:17:39.325 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:17:39.325 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: raise 2021-07-31T04:17:39.325 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:17:39.325 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T04:17:39.325 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:17:39.326 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:17:39.326 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:17:39.326 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: cephadm 2021-07-31 2021-07-31T04:17:39.326 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: T04:17:38.034337+0000 mgr.x (mgr.24542) 6450 : cephadm [ERR] 2021-07-31T04:17:39.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:17:39.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:17:39.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: yield 2021-07-31T04:17:39.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:17:39.328 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:17:39.328 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:17:39.328 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:17:39.328 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:17:39.328 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:17:39.329 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:17:39.329 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:17:39.329 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:17:39.329 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: task.result() 2021-07-31T04:17:39.329 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:17:39.330 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: cephadm 2021-07-31T04:17:38.034871+0000 mgr.x (mgr.24542) 6451 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:17:39.330 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:17:39.330 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:17:39.330 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: yield 2021-07-31T04:17:39.331 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:17:39.331 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:17:39.331 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:17:39.332 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:17:39.332 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:17:39.332 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:17:39.332 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:17:39.332 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:17:39.333 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:17:39.333 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: task.result() 2021-07-31T04:17:39.333 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:17:39.333 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: 2021-07-31T04:17:39.334 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T04:17:39.334 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: 2021-07-31T04:17:39.334 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:17:39.334 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:17:39.334 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: return f(*arg) 2021-07-31T04:17:39.335 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:17:39.335 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:17:39.335 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:17:39.335 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:17:39.336 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:17:39.336 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:17:39.336 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:17:39.336 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T04:17:39.336 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:17:39.337 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: raise self._exception 2021-07-31T04:17:39.337 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:17:39.337 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:17:39.337 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:17:39.337 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T04:17:39.338 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:17:39.338 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: raise 2021-07-31T04:17:39.338 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:17:39.338 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T04:17:39.339 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:17:39.339 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:17:39.339 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:17:39.339 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:17:39.340 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:17:39.340 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:17:39.340 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:17:39.340 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:17:39.341 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:17:39.341 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:17:39.341 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: task.result() 2021-07-31T04:17:39.341 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:17:39.341 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: 2021-07-31T04:17:39.342 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T04:17:39.342 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: 2021-07-31T04:17:39.342 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:17:39.342 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:17:39.343 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: return f(*arg) 2021-07-31T04:17:39.343 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:17:39.343 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:17:39.343 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:17:39.344 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:17:39.344 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:17:39.344 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:17:39.344 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:17:39.344 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T04:17:39.345 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:17:39.345 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: raise self._exception 2021-07-31T04:17:39.345 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:17:39.345 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:17:39.346 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:17:39.346 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T04:17:39.346 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:17:39.346 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: raise 2021-07-31T04:17:39.347 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:17:39.347 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T04:17:39.347 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:17:39.347 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:17:39.347 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:39 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:17:39.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:17:37.943612+0000 mgr.x (mgr.24542) 6442 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T04:17:39.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: cluster 2021-07-31T04:17:37.950321+0000 mgr.x (mgr.24542) 6443 : cluster [DBG] pgmap v6520: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:39.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:17:37.987114+0000 mgr.x (mgr.24542) 6444 : cephadm [ERR] 2021-07-31T04:17:39.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:17:39.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:17:39.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: yield 2021-07-31T04:17:39.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:17:39.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:17:39.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:17:39.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:17:39.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:17:39.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:17:39.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:17:39.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:17:39.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:17:39.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: task.result() 2021-07-31T04:17:39.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:17:39.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:17:37.987649+0000 mgr.x (mgr.24542) 6445 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T04:17:39.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:17:37.999406+0000 mgr.x (mgr.24542) 6446 : cephadm [ERR] 2021-07-31T04:17:39.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:17:39.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:17:39.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: yield 2021-07-31T04:17:39.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:17:39.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:17:39.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:17:39.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:17:39.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:17:39.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:17:39.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:17:39.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:17:39.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:17:39.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: task.result() 2021-07-31T04:17:39.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:17:39.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:17:38.000335+0000 mgr.x (mgr.24542) 6447 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T04:17:39.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:17:38.025571+0000 mgr.x (mgr.24542) 2021-07-31T04:17:39.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: 6448 : cephadm [ERR] 2021-07-31T04:17:39.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:17:39.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:17:39.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: yield 2021-07-31T04:17:39.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:17:39.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:17:39.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:17:39.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:17:39.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:17:39.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:17:39.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:17:39.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:17:39.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:17:39.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: task.result() 2021-07-31T04:17:39.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:17:39.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:17:38.026792+0000 mgr.x (mgr.24542) 6449 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:17:39.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:17:39.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:17:39.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: yield 2021-07-31T04:17:39.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:17:39.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:17:39.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:17:39.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:17:39.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:17:39.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:17:39.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:17:39.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:17:39.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:17:39.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: task.result() 2021-07-31T04:17:39.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:17:39.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: 2021-07-31T04:17:39.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T04:17:39.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: 2021-07-31T04:17:39.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:17:39.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:17:39.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: return f(*arg) 2021-07-31T04:17:39.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:17:39.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:17:39.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:17:39.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:17:39.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:17:39.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:17:39.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:17:39.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T04:17:39.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:17:39.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: raise self._exception 2021-07-31T04:17:39.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:17:39.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:17:39.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:17:39.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T04:17:39.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:17:39.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: raise 2021-07-31T04:17:39.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:17:39.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T04:17:39.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:17:39.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:17:39.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:17:39.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:17:38.034337+0000 mgr.x (mgr.24542) 6450 : cephadm [ERR] 2021-07-31T04:17:39.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: 2021-07-31T04:17:39.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:17:39.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:17:39.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: yield 2021-07-31T04:17:39.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:17:39.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:17:39.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:17:39.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:17:39.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:17:39.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:17:39.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:17:39.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:17:39.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:17:39.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: task.result() 2021-07-31T04:17:39.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:17:39.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:17:38.034871+0000 mgr.x (mgr.24542) 6451 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:17:39.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:17:39.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:17:39.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: yield 2021-07-31T04:17:39.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:17:39.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:17:39.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:17:39.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:17:39.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:17:39.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:17:39.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:17:39.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:17:39.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:17:39.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: task.result() 2021-07-31T04:17:39.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:17:39.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: 2021-07-31T04:17:39.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T04:17:39.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: 2021-07-31T04:17:39.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:17:39.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:17:39.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: return f(*arg) 2021-07-31T04:17:39.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:17:39.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:17:39.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:17:39.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:17:39.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:17:39.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:17:39.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:17:39.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T04:17:39.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:17:39.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: raise self._exception 2021-07-31T04:17:39.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:17:39.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:17:39.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:17:39.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T04:17:39.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:17:39.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: raise 2021-07-31T04:17:39.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:17:39.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T04:17:39.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:17:39.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:17:39.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:39 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:17:41.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:41 smithi045 conmon[39718]: cluster 2021-07-31T04:17:39.950866 2021-07-31T04:17:41.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:41 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 6452 : cluster [DBG] pgmap v6521: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:41.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:41 smithi045 conmon[33655]: cluster 2021-07-31T04:17:39.950866+0000 mgr.x (mgr.24542) 6452 : cluster [DBG] pgmap v6521: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:41.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:41 smithi154 conmon[34553]: cluster 2021-07-31T04:17:39.950866+0000 mgr.x (mgr.24542) 6452 : cluster [DBG] pgmap v6521: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:43.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:43 smithi045 conmon[39718]: cluster 2021-07-31T04:17:41.951393+0000 mgr.x (mgr.24542) 6453 : cluster [DBG] pgmap v6522: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:43.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:43 smithi045 conmon[33655]: cluster 2021-07-31T04:17:41.951393 2021-07-31T04:17:43.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:43 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 6453 : cluster [DBG] pgmap v6522: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:43.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:43 smithi154 conmon[34553]: cluster 2021-07-31T04:17:41.951393+0000 mgr.x (mgr.24542) 6453 : cluster [DBG] pgmap v6522: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:43.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:43 smithi154 conmon[34553]: 2021-07-31T04:17:45.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:45 smithi045 conmon[33655]: cluster 2021-07-31T04:17:43.951902+0000 mgr.x (mgr.24542 2021-07-31T04:17:45.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:45 smithi045 conmon[33655]: ) 6454 : cluster [DBG] pgmap v6523: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:45.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:45 smithi045 conmon[39718]: cluster 2021-07-31T04:17:43.951902+0000 mgr.x ( 2021-07-31T04:17:45.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:45 smithi045 conmon[39718]: mgr.24542) 6454 : cluster [DBG] pgmap v6523: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:45.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:45 smithi154 conmon[34553]: cluster 2021-07-31T04:17:43.951902+0000 mgr.x (mgr.24542) 6454 : cluster [DBG] pgmap v6523: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:47.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:47 smithi045 conmon[33655]: cluster 2021-07-31T04:17:45.952268+0000 mgr.x (mgr.24542) 6455 : cluster 2021-07-31T04:17:47.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:47 smithi045 conmon[33655]: [DBG] pgmap v6524: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:47.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:47 smithi045 conmon[39718]: cluster 2021-07-31T04:17:45.952268+0000 mgr.x (mgr.24542 2021-07-31T04:17:47.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:47 smithi045 conmon[39718]: ) 6455 : cluster [DBG] pgmap v6524: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:47.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:47 smithi154 conmon[34553]: cluster 2021-07-31T04:17:45.952268+0000 mgr.x (mgr.24542) 6455 : cluster [DBG] pgmap v6524: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:49.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:49 smithi045 conmon[39718]: cluster 2021-07-31T04:17:47.952676+0000 mgr.x (mgr 2021-07-31T04:17:49.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:49 smithi045 conmon[39718]: .24542) 6456 : cluster [DBG] pgmap v6525: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:49.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:49 smithi045 conmon[33655]: cluster 2021-07-31T04:17:47.952676+0000 mgr.x (mgr.24542) 6456 : cluster [DBG] pgmap v6525: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:49.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:49 smithi154 conmon[34553]: cluster 2021-07-31T04:17:47.952676+0000 mgr.x (mgr.24542) 6456 : cluster [DBG] pgmap v6525: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:51.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:51 smithi045 conmon[33655]: cluster 2021-07-31T04:17:49.953248+0000 mgr.x (mgr. 2021-07-31T04:17:51.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:51 smithi045 conmon[33655]: 24542) 6457 : cluster [DBG] pgmap v6526: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:51.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:51 smithi045 conmon[39718]: cluster 2021-07-31T04:17:49.953248+0000 2021-07-31T04:17:51.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:51 smithi045 conmon[39718]: mgr.x (mgr.24542) 6457 : cluster [DBG] pgmap v6526: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:51.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:51 smithi154 conmon[34553]: cluster 2021-07-31T04:17:49.953248+0000 mgr.x (mgr.24542) 6457 : cluster [DBG] pgmap v6526: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:53.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:53 smithi154 conmon[34553]: cluster 2021-07-31T04:17:51.953697+0000 mgr.x (mgr.24542) 6458 : cluster [DBG] pgmap v6527: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:53.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:53 smithi045 conmon[33655]: cluster 2021-07-31T04:17:51.953697+0000 2021-07-31T04:17:53.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:53 smithi045 conmon[33655]: mgr.x (mgr.24542) 6458 : cluster [DBG] pgmap v6527: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:53.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:53 smithi045 conmon[39718]: cluster 2021-07-31T04:17:51.953697+0000 mgr.x (mgr.24542) 2021-07-31T04:17:53.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:53 smithi045 conmon[39718]: 6458 : cluster [DBG] pgmap v6527: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:55.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:55 smithi154 conmon[34553]: cluster 2021-07-31T04:17:53.954257+0000 mgr.x (mgr.24542) 6459 : cluster [DBG] pgmap v6528: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:55.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:55 smithi045 conmon[33655]: cluster 2021-07-31T04:17:53. 2021-07-31T04:17:55.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:55 smithi045 conmon[33655]: 954257+0000 mgr.x (mgr.24542) 6459 : cluster [DBG] pgmap v6528: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:55.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:55 smithi045 conmon[39718]: cluster 2021-07-31T04:17:53.954257+0000 mgr.x (mgr. 2021-07-31T04:17:55.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:55 smithi045 conmon[39718]: 24542) 6459 : cluster [DBG] pgmap v6528: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:57.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:57 smithi154 conmon[34553]: cluster 2021-07-31T04:17:55.954846+0000 mgr.x (mgr.24542) 6460 : cluster [DBG] pgmap v6529: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:57.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:57 smithi045 conmon[33655]: cluster 2021-07-31T04:17:55.954846+0000 mgr.x (mgr.24542) 6460 : 2021-07-31T04:17:57.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:57 smithi045 conmon[33655]: cluster [DBG] pgmap v6529: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:57.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:57 smithi045 conmon[39718]: cluster 2021-07-31T04:17:55.954846+0000 mgr.x (mgr. 2021-07-31T04:17:57.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:57 smithi045 conmon[39718]: 24542) 6460 : cluster [DBG] pgmap v6529: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:58.546 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T04:17:58.821 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T04:17:58.821 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 2h ago 2h - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T04:17:58.821 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 2h ago 2h 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T04:17:58.822 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 2h ago 2h 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T04:17:58.822 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 2h ago 2h - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T04:17:58.822 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T04:17:58.822 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 2h ago 2h 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T04:17:58.823 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T04:17:58.823 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 2h ago 2h - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T04:17:58.823 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 2h ago 2h 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T04:17:58.823 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T04:17:58.823 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T04:17:58.824 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T04:17:58.824 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T04:17:58.824 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 2h ago 2h 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T04:17:58.824 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 2h ago 2h 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T04:17:58.824 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 2h ago 2h 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T04:17:58.825 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 2h ago 2h 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T04:17:58.825 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 2h ago 2h 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T04:17:59.159 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T04:17:59.159 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T04:17:59.159 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T04:17:59.160 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:17:59.160 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T04:17:59.160 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T04:17:59.160 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:17:59.161 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:17:59.161 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T04:17:59.161 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T04:17:59.161 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:17:59.161 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T04:17:59.162 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T04:17:59.162 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T04:17:59.162 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:17:59.162 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T04:17:59.162 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T04:17:59.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:17:59 smithi154 conmon[34553]: cluster 2021-07-31T04:17:57.955364+0000 mgr.x (mgr.24542) 6461 : cluster [DBG] pgmap v6530: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:59.524 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:59 smithi045 conmon[33655]: cluster 2021-07-31T04:17:57 2021-07-31T04:17:59.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:17:59 smithi045 conmon[33655]: .955364+0000 mgr.x (mgr.24542) 6461 : cluster [DBG] pgmap v6530: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:17:59.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:59 smithi045 conmon[39718]: cluster 2021-07-31T04:17:57. 2021-07-31T04:17:59.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:17:59 smithi045 conmon[39718]: 955364+0000 mgr.x (mgr.24542) 6461 : cluster [DBG] pgmap v6530: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:00.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:00 smithi154 conmon[34553]: audit 2021-07-31T04:17:58.534183+0000 mgr.x (mgr.24542) 6462 : audit [DBG] from='client.31243 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:18:00.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:00 smithi154 conmon[34553]: 2021-07-31T04:18:00.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:00 smithi154 conmon[34553]: audit 2021-07-31T04:17:58.813679+0000 mgr.x (mgr.24542) 6463 : audit [DBG] from='client.21312 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:18:00.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:00 smithi154 conmon[34553]: audit 2021-07-31T04:17:59.160566 2021-07-31T04:18:00.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:00 smithi154 conmon[34553]: +0000 mon.a (mon.0) 3454 : audit [DBG] from='client.? 172.21.15.45:0/2219473457' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:18:00.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:00 smithi045 conmon[33655]: audit 2021-07-31T04:17:58.534183+0000 mgr.x (mgr.24542) 6462 : audit [DBG] from='client.31243 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:18:00.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:00 smithi045 conmon[33655]: audit 2021-07-31T04:17:58.813679+0000 mgr.x (mgr.24542) 6463 : audit [DBG] from='client.21312 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:18:00.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:00 smithi045 conmon[33655]: audit 2021-07-31T04 2021-07-31T04:18:00.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:00 smithi045 conmon[33655]: :17:59.160566+0000 mon.a (mon.0) 3454 : audit [DBG] 2021-07-31T04:18:00.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:00 smithi045 conmon[33655]: from='client.? 172.21.15.45:0/2219473457' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:18:00.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:00 smithi045 conmon[39718]: audit 2021-07-31T04:17:58.534183+0000 mgr.x (mgr.24542) 2021-07-31T04:18:00.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:00 smithi045 conmon[39718]: 6462 : audit [DBG] from='client.31243 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:18:00.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:00 smithi045 conmon[39718]: audit 2021-07-31T04:17:58.813679+0000 mgr.x (mgr.24542) 6463 : audit [DBG] 2021-07-31T04:18:00.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:00 smithi045 conmon[39718]: from='client.21312 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:18:00.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:00 smithi045 conmon[39718]: audit 2021-07-31T04:17:59.160566+0000 mon.a (mon. 2021-07-31T04:18:00.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:00 smithi045 conmon[39718]: 0) 3454 : audit [DBG] from='client.? 172.21.15.45:0/2219473457' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:18:01.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:01 smithi154 conmon[34553]: cluster 2021-07-31T04:17:59.955875+0000 mgr.x (mgr.24542) 6464 : cluster [DBG] pgmap v6531: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:01.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:01 smithi045 conmon[39718]: cluster 2021-07-31T04:17:59.955875+0000 2021-07-31T04:18:01.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:01 smithi045 conmon[39718]: mgr.x (mgr.24542) 6464 : cluster [DBG] pgmap v6531: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:01.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:01 smithi045 conmon[33655]: cluster 2021-07-31T04:17:59.955875+0000 mgr.x (mgr.24542) 6464 2021-07-31T04:18:01.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:01 smithi045 conmon[33655]: : cluster [DBG] pgmap v6531: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:03.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:03 smithi154 conmon[34553]: cluster 2021-07-31T04:18:01.956362+0000 mgr.x (mgr.24542) 6465 : cluster [DBG] pgmap v6532: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:03.524 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:03 smithi045 conmon[33655]: cluster 2021-07-31T04:18:01.956362+0000 mgr.x (mgr.24542) 6465 : cluster [DBG] 2021-07-31T04:18:03.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:03 smithi045 conmon[33655]: pgmap v6532: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:03.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:03 smithi045 conmon[39718]: cluster 2021-07-31T04:18:01.956362+0000 mgr.x 2021-07-31T04:18:03.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:03 smithi045 conmon[39718]: (mgr.24542) 6465 : cluster [DBG] pgmap v6532: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:05.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:05 smithi154 conmon[34553]: cluster 2021-07-31T04:18:03.956915+0000 mgr.x (mgr.24542) 6466 : cluster [DBG] pgmap v6533: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:05.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:05 smithi045 conmon[33655]: cluster 2021-07-31T04:18:03.956915+0000 mgr.x (mgr.24542) 6466 : cluster [DBG] pgmap v6533: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:05.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:05 smithi045 conmon[33655]: 2021-07-31T04:18:05.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:05 smithi045 conmon[39718]: cluster 2021-07-31T04:18:03.956915+0000 mgr.x ( 2021-07-31T04:18:05.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:05 smithi045 conmon[39718]: mgr.24542) 6466 : cluster [DBG] pgmap v6533: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:07.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:07 smithi154 conmon[34553]: cluster 2021-07-31T04:18:05.957293+0000 mgr.x (mgr.24542) 6467 : cluster [DBG] pgmap v6534: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:07.524 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:07 smithi045 conmon[39718]: cluster 2021-07-31T04:18:05.957293+0000 mgr.x (mgr 2021-07-31T04:18:07.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:07 smithi045 conmon[39718]: .24542) 6467 : cluster [DBG] pgmap v6534: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:07.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:07 smithi045 conmon[33655]: cluster 2021-07-31T04:18:05.957293+0000 mgr.x (mgr.24542) 6467 : cluster [DBG] 2021-07-31T04:18:07.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:07 smithi045 conmon[33655]: pgmap v6534: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:09.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:09 smithi154 conmon[34553]: cluster 2021-07-31T04:18:07.957744+0000 mgr.x (mgr.24542) 6468 : cluster [DBG] pgmap v6535: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:09.524 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:09 smithi045 conmon[39718]: cluster 2021-07-31T04:18:07.957744+0000 mgr.x 2021-07-31T04:18:09.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:09 smithi045 conmon[39718]: (mgr.24542) 6468 : cluster [DBG] pgmap v6535: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:09.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:09 smithi045 conmon[33655]: cluster 2021-07-31T04:18:07.957744+0000 mgr.x (mgr.24542) 6468 : cluster 2021-07-31T04:18:09.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:09 smithi045 conmon[33655]: [DBG] pgmap v6535: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:11.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:11 smithi154 conmon[34553]: cluster 2021-07-31T04:18:09.958241+0000 mgr.x (mgr.24542) 6469 : cluster [DBG] pgmap v6536: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:11.524 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:11 smithi045 conmon[39718]: cluster 2021-07-31T04:18:09.958241+0000 mgr.x 2021-07-31T04:18:11.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:11 smithi045 conmon[39718]: (mgr.24542) 6469 : cluster [DBG] pgmap v6536: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:11.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:11 smithi045 conmon[33655]: cluster 2021-07-31T04:18:09.958241+0000 mgr.x (mgr.24542) 6469 : cluster 2021-07-31T04:18:11.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:11 smithi045 conmon[33655]: [DBG] pgmap v6536: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:13.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:13 smithi154 conmon[34553]: cluster 2021-07-31T04:18:11.958623+0000 mgr.x (mgr.24542) 6470 : cluster [DBG] pgmap v6537: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:13.524 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:13 smithi045 conmon[39718]: cluster 2021-07-31T04:18:11.958623+0000 mgr.x ( 2021-07-31T04:18:13.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:13 smithi045 conmon[39718]: mgr.24542) 6470 : cluster [DBG] pgmap v6537: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:13.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:13 smithi045 conmon[33655]: cluster 2021-07-31T04:18:11.958623+0000 mgr.x (mgr.24542) 6470 : cluster 2021-07-31T04:18:13.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:13 smithi045 conmon[33655]: [DBG] pgmap v6537: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:15.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:15 smithi154 conmon[34553]: cluster 2021-07-31T04:18:13.959252+0000 mgr.x (mgr.24542) 6471 : cluster [DBG] pgmap v6538: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:15.524 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:15 smithi045 conmon[39718]: cluster 2021-07-31T04:18:13.959252+0000 mgr.x (mgr 2021-07-31T04:18:15.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:15 smithi045 conmon[39718]: .24542) 6471 : cluster [DBG] pgmap v6538: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:15.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:15 smithi045 conmon[33655]: cluster 2021-07-31T04:18:13.959252+0000 mgr.x (mgr.24542) 6471 : cluster 2021-07-31T04:18:15.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:15 smithi045 conmon[33655]: [DBG] pgmap v6538: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:17.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:17 smithi154 conmon[34553]: cluster 2021-07-31T04:18:15.959746+0000 mgr.x (mgr.24542) 6472 : cluster [DBG] pgmap v6539: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:17.524 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:17 smithi045 conmon[39718]: cluster 2021-07-31T04:18:15.959746+0000 mgr.x (mgr.24542) 6472 : cluster [DBG] pgmap v6539: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:17.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:17 smithi045 conmon[33655]: cluster 2021-07-31T04:18:15.959746+0000 mgr.x (mgr.24542) 6472 : cluster [DBG] 2021-07-31T04:18:17.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:17 smithi045 conmon[33655]: pgmap v6539: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:19.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:19 smithi154 conmon[34553]: cluster 2021-07-31T04:18:17.960175+0000 mgr.x (mgr.24542) 6473 : cluster [DBG] pgmap v6540: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:19.524 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:19 smithi045 conmon[39718]: cluster 2021-07-31T04:18:17.960175+0000 2021-07-31T04:18:19.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:19 smithi045 conmon[39718]: mgr.x (mgr.24542) 6473 : cluster [DBG] pgmap v6540: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:19.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:19 smithi045 conmon[33655]: cluster 2021-07-31T04 2021-07-31T04:18:19.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:19 smithi045 conmon[33655]: :18:17.960175+0000 mgr.x (mgr.24542) 6473 : cluster [DBG] pgmap v6540: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:21.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:21 smithi154 conmon[34553]: cluster 2021-07-31T04:18:19.960649+0000 mgr.x (mgr.24542) 6474 : cluster [DBG] pgmap v6541: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:21.524 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:21 smithi045 conmon[33655]: cluster 2021-07-31T04:18:19.960649+0000 mgr.x ( 2021-07-31T04:18:21.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:21 smithi045 conmon[33655]: mgr.24542) 6474 : cluster [DBG] pgmap v6541: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:21.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:21 smithi045 conmon[39718]: cluster 2021-07-31T04:18:19.960649+0000 mgr.x (mgr.24542) 6474 : cluster [DBG] pgmap v6541: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:23.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:23 smithi154 conmon[34553]: cluster 2021-07-31T04:18:21.961206+0000 mgr.x (mgr.24542) 6475 : cluster [DBG] pgmap v6542: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:23.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:23 smithi045 conmon[33655]: cluster 2021-07-31T04:18:21.961206+0000 mgr.x (mgr 2021-07-31T04:18:23.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:23 smithi045 conmon[33655]: .24542) 6475 : cluster [DBG] pgmap v6542: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:23.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:23 smithi045 conmon[39718]: cluster 2021-07-31T04:18: 2021-07-31T04:18:23.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:23 smithi045 conmon[39718]: 21.961206+0000 mgr.x (mgr.24542) 6475 : cluster [DBG] pgmap v6542: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:25.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:25 smithi154 conmon[34553]: cluster 2021-07-31T04:18:23.961652+0000 mgr.x (mgr.24542) 6476 : cluster [DBG] pgmap v6543: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:25.524 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:25 smithi045 conmon[39718]: cluster 2021-07-31T04:18:23.961652+0000 mgr.x (mgr.24542 2021-07-31T04:18:25.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:25 smithi045 conmon[39718]: ) 6476 : cluster [DBG] pgmap v6543: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:25.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:25 smithi045 conmon[33655]: cluster 2021-07-31T04:18:23.961652+0000 mgr.x (mgr.24542) 2021-07-31T04:18:25.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:25 smithi045 conmon[33655]: 6476 : cluster [DBG] pgmap v6543: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:27.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:27 smithi154 conmon[34553]: cluster 2021-07-31T04:18:25.962182+0000 mgr.x (mgr.24542) 6477 : cluster [DBG] pgmap v6544: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:27.524 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:27 smithi045 conmon[33655]: cluster 2021-07-31T04:18:25. 2021-07-31T04:18:27.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:27 smithi045 conmon[33655]: 962182+0000 mgr.x (mgr.24542) 6477 : cluster [DBG] pgmap v6544: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:27.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:27 smithi045 conmon[39718]: cluster 2021-07-31T04:18:25.962182 2021-07-31T04:18:27.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:27 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 6477 : cluster [DBG] pgmap v6544: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:29.473 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T04:18:29.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:29 smithi154 conmon[34553]: cluster 2021-07-31T04:18:27.962698+0000 mgr.x (mgr.24542) 6478 : cluster [DBG] pgmap v6545: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:29.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:29 smithi154 conmon[34553]: audit 2021-07-31T04:18:28.531216+0000 mon.b (mon.2) 830 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:18:29.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:29 smithi154 conmon[34553]: audit 2021-07-31T04:18:28.531520+0000 mon.b (mon.2) 831 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:18:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:29 smithi154 conmon[34553]: audit 2021-07-31T04:18:28.532221+0000 mon.a (mon.0) 3455 : audit [INF] 2021-07-31T04:18:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:29 smithi154 conmon[34553]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:18:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:29 smithi154 conmon[34553]: audit 2021- 2021-07-31T04:18:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:29 smithi154 conmon[34553]: 07-31T04:18:28.532670+0000 mon.a (mon.0) 3456 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:18:29.524 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:29 smithi045 conmon[39718]: cluster 2021-07-31T04:18:27.962698+0000 mgr.x (mgr.24542) 6478 : 2021-07-31T04:18:29.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:29 smithi045 conmon[39718]: cluster [DBG] pgmap v6545: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:29.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:29 smithi045 conmon[39718]: audit 2021-07-31T04:18:28.531216+0000 mon.b (mon.2) 830 : 2021-07-31T04:18:29.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:29 smithi045 conmon[39718]: audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:18:29.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:29 smithi045 conmon[39718]: audit 2021-07-31T04:18:28. 2021-07-31T04:18:29.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:29 smithi045 conmon[39718]: 531520+0000 mon.b (mon.2) 831 : audit 2021-07-31T04:18:29.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:29 smithi045 conmon[39718]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:18:29.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:29 smithi045 conmon[39718]: audit 2021-07-31T04:18:28.532221+0000 mon.a (mon.0 2021-07-31T04:18:29.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:29 smithi045 conmon[39718]: ) 3455 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:18:29.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:29 smithi045 conmon[39718]: audit 2021-07-31T04:18:29.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:29 smithi045 conmon[39718]: 2021-07-31T04:18:28.532670+0000 mon.a (mon.0) 3456 : audit [INF] 2021-07-31T04:18:29.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:29 smithi045 conmon[39718]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:18:29.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:29 smithi045 conmon[33655]: cluster 2021-07-31T04:18:27.962698 2021-07-31T04:18:29.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:29 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 6478 : cluster [DBG] pgmap v6545: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:29.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:29 smithi045 conmon[33655]: audit 2021-07 2021-07-31T04:18:29.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:29 smithi045 conmon[33655]: -31T04:18:28.531216+0000 mon.b (mon. 2021-07-31T04:18:29.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:29 smithi045 conmon[33655]: 2) 830 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:18:29.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:29 smithi045 conmon[33655]: audit 2021-07-31T 2021-07-31T04:18:29.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:29 smithi045 conmon[33655]: 04:18:28.531520+0000 mon.b (mon.2) 831 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:18:29.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:29 smithi045 conmon[33655]: audit 2021-07-31T04 2021-07-31T04:18:29.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:29 smithi045 conmon[33655]: :18:28.532221+0000 mon.a (mon.0) 3455 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:18:29.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:29 smithi045 conmon[33655]: audit 2021-07-31T04:18:29.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:29 smithi045 conmon[33655]: 2021-07-31T04:18:28.532670+0000 mon.a (mon.0) 3456 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:18:29.749 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T04:18:29.750 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 2h ago 2h - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T04:18:29.750 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 2h ago 2h 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T04:18:29.750 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 2h ago 2h 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T04:18:29.750 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 2h ago 2h - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T04:18:29.751 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T04:18:29.751 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 2h ago 2h 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T04:18:29.751 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T04:18:29.751 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 2h ago 2h - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T04:18:29.751 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 2h ago 2h 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T04:18:29.752 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T04:18:29.752 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T04:18:29.752 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T04:18:29.752 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T04:18:29.752 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 2h ago 2h 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T04:18:29.753 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 2h ago 2h 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T04:18:29.753 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 2h ago 2h 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T04:18:29.753 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 2h ago 2h 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T04:18:29.753 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 2h ago 2h 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T04:18:30.082 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T04:18:30.083 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T04:18:30.083 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T04:18:30.083 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:18:30.083 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T04:18:30.084 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T04:18:30.084 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:18:30.084 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:18:30.084 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T04:18:30.084 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T04:18:30.085 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:18:30.085 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T04:18:30.085 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T04:18:30.085 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T04:18:30.085 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:18:30.086 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T04:18:30.086 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T04:18:30.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:30 smithi154 conmon[34553]: audit 2021-07-31T04:18:29.462388+0000 mgr.x (mgr.24542) 6479 : audit [DBG] from='client.21324 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:18:30.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:30 smithi154 conmon[34553]: audit 2021-07-31T04:18:30.083436+0000 mon.a (mon.0) 3457 : audit [DBG] from='client.? 172.21.15.45:0/2545821922' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:18:30.524 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:30 smithi045 conmon[33655]: audit 2021-07-31T04:18:29.462388 2021-07-31T04:18:30.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:30 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 6479 : audit [DBG] from='client.21324 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:18:30.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:30 smithi045 conmon[33655]: audit 2021- 2021-07-31T04:18:30.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:30 smithi045 conmon[33655]: 07-31T04:18:30.083436+0000 mon.a (mon.0 2021-07-31T04:18:30.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:30 smithi045 conmon[33655]: ) 3457 : audit [DBG] 2021-07-31T04:18:30.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:30 smithi045 conmon[33655]: from='client.? 172.21.15.45:0/2545821922' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:18:30.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:30 smithi045 conmon[39718]: audit 2021-07-31T04:18:29.462388+0000 mgr.x (mgr. 2021-07-31T04:18:30.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:30 smithi045 conmon[39718]: 24542) 6479 : audit [DBG] from='client.21324 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:18:30.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:30 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T04:18:30.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:30 smithi045 conmon[39718]: T04:18:30.083436 2021-07-31T04:18:30.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:30 smithi045 conmon[39718]: +0000 mon.a (mon.0) 2021-07-31T04:18:30.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:30 smithi045 conmon[39718]: 3457 : audit [DBG] from='client.? 172.21.15.45:0/2545821922' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:18:30.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:30 smithi045 conmon[39718]: 2021-07-31T04:18:31.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:31 smithi154 conmon[34553]: audit 2021-07-31T04:18:29.742579+0000 mgr.x (mgr.24542) 6480 : audit [DBG] from='client.31267 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:18:31.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:31 smithi154 conmon[34553]: cluster 2021-07-31T04:18:29.963127+0000 mgr.x (mgr.24542) 6481 : cluster [DBG] pgmap v6546: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:31.524 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:31 smithi045 conmon[39718]: audit 2021-07-31T04:18:29.742579 2021-07-31T04:18:31.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:31 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 6480 : audit [DBG] from='client.31267 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:18:31.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:31 smithi045 conmon[39718]: cluster 2021-07-31T04:18:31.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:31 smithi045 conmon[39718]: 2021-07-31T04:18:29.963127+0000 mgr.x (mgr.24542) 6481 2021-07-31T04:18:31.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:31 smithi045 conmon[39718]: : cluster [DBG] pgmap v6546: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:31.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:31 smithi045 conmon[33655]: audit 2021-07-31T04:18:29.742579+0000 mgr.x 2021-07-31T04:18:31.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:31 smithi045 conmon[33655]: (mgr.24542) 6480 : audit [DBG] from='client.31267 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:18:31.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:31 smithi045 conmon[33655]: cluster 2021-07-31T04:18:29.963127+0000 mgr.x (mgr. 2021-07-31T04:18:31.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:31 smithi045 conmon[33655]: 24542) 6481 : cluster [DBG] pgmap v6546: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:33.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:33 smithi154 conmon[34553]: cluster 2021-07-31T04:18:31.963478+0000 mgr.x (mgr.24542) 6482 : cluster [DBG] pgmap v6547: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:33.524 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:33 smithi045 conmon[39718]: cluster 2021-07-31T04:18:31.963478+0000 mgr.x (mgr.24542 2021-07-31T04:18:33.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:33 smithi045 conmon[39718]: ) 6482 : cluster [DBG] pgmap v6547: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:33.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:33 smithi045 conmon[33655]: cluster 2021-07-31T04:18:31.963478+0000 mgr.x (mgr. 2021-07-31T04:18:33.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:33 smithi045 conmon[33655]: 24542) 6482 : cluster [DBG] pgmap v6547: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:35.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:35 smithi154 conmon[34553]: cluster 2021-07-31T04:18:33.964058+0000 mgr.x (mgr.24542) 6483 : cluster [DBG] pgmap v6548: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:35.524 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:35 smithi045 conmon[39718]: cluster 2021-07-31T04:18:33.964058+0000 mgr.x (mgr.24542) 6483 : cluster 2021-07-31T04:18:35.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:35 smithi045 conmon[39718]: [DBG] pgmap v6548: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:35.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:35 smithi045 conmon[33655]: cluster 2021-07-31T04:18:33.964058+0000 mgr.x (mgr.24542) 6483 : cluster [DBG] pgmap v6548: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:37.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:37 smithi154 conmon[34553]: cluster 2021-07-31T04:18:35.964609+0000 mgr.x (mgr.24542) 6484 : cluster [DBG] pgmap v6549: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:37.524 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:37 smithi045 conmon[39718]: cluster 2021-07-31T04:18:35.964609+0000 mgr.x (mgr.24542) 6484 : cluster [DBG] pgmap v6549: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:37.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:37 smithi045 conmon[33655]: cluster 2021-07-31T04:18:35.964609+0000 mgr.x (mgr.24542) 6484 : cluster [DBG] pgmap v6549: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:37.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:37 smithi045 conmon[33655]: 2021-07-31T04:18:38.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[34553]: audit 2021-07-31T04:18:38.035656+0000 mon.b (mon 2021-07-31T04:18:38.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[34553]: .2) 832 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T04:18:38.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[34553]: audit 2021-07-31T04:18:38 2021-07-31T04:18:38.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[34553]: .037047+0000 mon.b (mon.2) 833 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T04:18:38.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[34553]: audit 2021-07-31T 2021-07-31T04:18:38.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[34553]: 04:18:38.037880+0000 mon.b (mon 2021-07-31T04:18:38.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[34553]: .2) 834 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T04:18:38.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: debug 2021-07-31T04:18:38.080+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:18:38.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:18:38.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:18:38.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: yield 2021-07-31T04:18:38.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:18:38.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:18:38.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:18:38.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:18:38.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:18:38.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:18:38.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:18:38.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:18:38.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:18:38.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: task.result() 2021-07-31T04:18:38.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:18:38.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: debug 2021-07-31T04:18:38.091+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:18:38.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:18:38.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:18:38.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: yield 2021-07-31T04:18:38.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:18:38.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:18:38.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:18:38.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:18:38.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:18:38.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:18:38.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:18:38.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:18:38.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:18:38.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: task.result() 2021-07-31T04:18:38.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:18:38.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: debug 2021-07-31T04:18:38.115+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:18:38.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:18:38.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:18:38.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: yield 2021-07-31T04:18:38.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:18:38.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:18:38.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:18:38.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:18:38.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:18:38.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:18:38.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:18:38.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:18:38.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:18:38.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: task.result() 2021-07-31T04:18:38.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:18:38.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: debug 2021-07-31T04:18:38.119+0000 7fcb9295f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:18:38.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:18:38.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:18:38.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: yield 2021-07-31T04:18:38.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:18:38.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:18:38.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:18:38.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:18:38.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:18:38.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:18:38.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:18:38.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:18:38.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:18:38.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: task.result() 2021-07-31T04:18:38.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:18:38.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: 2021-07-31T04:18:38.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T04:18:38.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: 2021-07-31T04:18:38.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:18:38.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:18:38.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: return f(*arg) 2021-07-31T04:18:38.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:18:38.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:18:38.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:18:38.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:18:38.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:18:38.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:18:38.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:18:38.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T04:18:38.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:18:38.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: raise self._exception 2021-07-31T04:18:38.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:18:38.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:18:38.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:18:38.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T04:18:38.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:18:38.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: raise 2021-07-31T04:18:38.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:18:38.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T04:18:38.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:18:38.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:18:38.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:18:38.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: debug 2021-07-31T04:18:38.131+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:18:38.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:18:38.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:18:38.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: yield 2021-07-31T04:18:38.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:18:38.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:18:38.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:18:38.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:18:38.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:18:38.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:18:38.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:18:38.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:18:38.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:18:38.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: task.result() 2021-07-31T04:18:38.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:18:38.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: debug 2021-07-31T04:18:38.131+0000 7fcb93160700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:18:38.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:18:38.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:18:38.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: yield 2021-07-31T04:18:38.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:18:38.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:18:38.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:18:38.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:18:38.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:18:38.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:18:38.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:18:38.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:18:38.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:18:38.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: task.result() 2021-07-31T04:18:38.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:18:38.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: 2021-07-31T04:18:38.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T04:18:38.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: 2021-07-31T04:18:38.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:18:38.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:18:38.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: return f(*arg) 2021-07-31T04:18:38.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:18:38.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:18:38.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:18:38.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:18:38.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:18:38.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:18:38.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:18:38.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T04:18:38.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:18:38.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: raise self._exception 2021-07-31T04:18:38.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:18:38.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:18:38.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:18:38.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T04:18:38.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:18:38.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: raise 2021-07-31T04:18:38.535 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:18:38.535 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T04:18:38.535 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:18:38.535 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:18:38.536 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:18:38 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:18:38.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:38 smithi045 conmon[33655]: audit 2021-07-31T04:18:38.035656+0000 mon.b (mon.2) 832 : audit 2021-07-31T04:18:38.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:38 smithi045 conmon[33655]: [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T04:18:38.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:38 smithi045 conmon[33655]: audit 2021-07-31T04:18:38.037047+0000 mon.b (mon.2) 833 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T04:18:38.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:38 smithi045 conmon[33655]: audit 2021-07-31T04:18:38.037880+0000 mon.b (mon 2021-07-31T04:18:38.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:38 smithi045 conmon[33655]: .2) 834 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T04:18:38.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:38 smithi045 conmon[39718]: audit 2021-07-31T04:18 2021-07-31T04:18:38.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:38 smithi045 conmon[39718]: :38.035656+0000 mon.b (mon.2) 832 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T04:18:38.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:38 smithi045 conmon[39718]: audit 2021-07 2021-07-31T04:18:38.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:38 smithi045 conmon[39718]: -31T04:18:38.037047 2021-07-31T04:18:38.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:38 smithi045 conmon[39718]: +0000 mon.b (mon.2) 833 2021-07-31T04:18:38.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:38 smithi045 conmon[39718]: : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T04:18:38.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:38 smithi045 conmon[39718]: audit 2021 2021-07-31T04:18:38.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:38 smithi045 conmon[39718]: -07-31T04:18:38 2021-07-31T04:18:38.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:38 smithi045 conmon[39718]: .037880+0000 mon.b (mon. 2021-07-31T04:18:38.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:38 smithi045 conmon[39718]: 2) 834 : audit [INF] 2021-07-31T04:18:38.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:38 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T04:18:39.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: cluster 2021-07-31T04:18:37.965097+0000 mgr.x (mgr.24542) 6485 : cluster [DBG] pgmap v6550: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:39.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:18:38.038683+0000 mgr.x (mgr.24542) 6486 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T04:18:39.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:18:38.081572+0000 mgr.x (mgr.24542) 6487 : cephadm [ERR] 2021-07-31T04:18:39.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:18:39.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:18:39.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: yield 2021-07-31T04:18:39.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:18:39.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:18:39.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:18:39.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:18:39.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:18:39.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:18:39.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:18:39.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:18:39.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:18:39.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: task.result() 2021-07-31T04:18:39.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:18:39.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:18:38.082026+0000 mgr.x (mgr.24542) 6488 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T04:18:39.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:18:38.093219+0000 mgr.x (mgr.24542) 6489 : cephadm [ERR] 2021-07-31T04:18:39.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:18:39.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:18:39.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: yield 2021-07-31T04:18:39.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:18:39.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:18:39.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:18:39.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:18:39.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:18:39.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:18:39.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:18:39.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:18:39.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:18:39.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: task.result() 2021-07-31T04:18:39.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:18:39.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:18:38.093750+0000 mgr.x (mgr.24542) 6490 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T04:18:39.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:18:38.117899+0000 mgr.x (mgr.24542) 6491 : cephadm [ERR] 2021-07-31T04:18:39.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:18:39.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:18:39.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: yield 2021-07-31T04:18:39.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:18:39.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:18:39.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:18:39.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:18:39.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:18:39.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:18:39.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:18:39.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:18:39.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:18:39.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: task.result() 2021-07-31T04:18:39.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:18:39.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:18:38.121651+0000 mgr.x (mgr.24542) 6492 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:18:39.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:18:39.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:18:39.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: yield 2021-07-31T04:18:39.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:18:39.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:18:39.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:18:39.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:18:39.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:18:39.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:18:39.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:18:39.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:18:39.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:18:39.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: task.result() 2021-07-31T04:18:39.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:18:39.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: 2021-07-31T04:18:39.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T04:18:39.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: 2021-07-31T04:18:39.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:18:39.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:18:39.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: return f(*arg) 2021-07-31T04:18:39.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:18:39.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:18:39.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:18:39.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:18:39.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:18:39.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:18:39.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:18:39.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T04:18:39.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:18:39.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: raise self._exception 2021-07-31T04:18:39.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:18:39.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:18:39.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:18:39.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T04:18:39.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:18:39.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: raise 2021-07-31T04:18:39.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:18:39.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T04:18:39.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:18:39.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:18:39.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:18:39.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:18:38. 2021-07-31T04:18:39.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: 132461+0000 mgr.x (mgr.24542) 6493 : cephadm [ERR] 2021-07-31T04:18:39.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:18:39.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:18:39.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: yield 2021-07-31T04:18:39.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:18:39.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:18:39.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:18:39.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:18:39.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:18:39.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:18:39.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:18:39.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:18:39.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:18:39.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: task.result() 2021-07-31T04:18:39.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:18:39.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:18:38.133040+0000 mgr.x (mgr.24542) 6494 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:18:39.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:18:39.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:18:39.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: yield 2021-07-31T04:18:39.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:18:39.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:18:39.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:18:39.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:18:39.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:18:39.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:18:39.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:18:39.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:18:39.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:18:39.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: task.result() 2021-07-31T04:18:39.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:18:39.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: 2021-07-31T04:18:39.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T04:18:39.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: 2021-07-31T04:18:39.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:18:39.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:18:39.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: return f(*arg) 2021-07-31T04:18:39.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:18:39.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:18:39.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:18:39.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:18:39.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:18:39.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:18:39.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:18:39.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T04:18:39.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:18:39.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: raise self._exception 2021-07-31T04:18:39.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:18:39.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:18:39.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:18:39.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T04:18:39.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:18:39.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: raise 2021-07-31T04:18:39.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:18:39.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T04:18:39.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:18:39.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:18:39.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:39 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:18:39.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: cluster 2021-07-31T04:18:37.965097+0000 2021-07-31T04:18:39.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: mgr.x (mgr.24542) 6485 : cluster [DBG] pgmap v6550: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:39.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: cephadm 2021-07-31T 2021-07-31T04:18:39.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: 04:18:38.038683+0000 mgr.x (mgr.24542) 2021-07-31T04:18:39.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: 6486 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T04:18:39.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: cephadm 2021- 2021-07-31T04:18:39.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: 07-31T04:18:38.081572 2021-07-31T04:18:39.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 6487 : cephadm 2021-07-31T04:18:39.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: [ERR] 2021-07-31T04:18:39.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:18:39.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:18:39.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: yield 2021-07-31T04:18:39.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:18:39.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:18:39.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:18:39.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:18:39.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:18:39.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:18:39.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:18:39.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:18:39.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:18:39.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: task.result() 2021-07-31T04:18:39.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:18:39.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: cephadm 2021-07-31T04:18:38.082026+0000 mgr.x (mgr.24542) 6488 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T04:18:39.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: cephadm 2021-07- 2021-07-31T04:18:39.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: 31T04:18:38.093219 2021-07-31T04:18:39.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 6489 2021-07-31T04:18:39.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: : cephadm [ERR] 2021-07-31T04:18:39.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:18:39.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:18:39.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: yield 2021-07-31T04:18:39.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:18:39.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:18:39.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:18:39.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:18:39.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:18:39.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:18:39.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:18:39.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:18:39.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:18:39.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: task.result() 2021-07-31T04:18:39.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:18:39.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: cephadm 2021-07-31T04:18:38.093750+0000 mgr.x (mgr.24542) 6490 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T04:18:39.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: cephadm 2021- 2021-07-31T04:18:39.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: 07-31T04:18:38.117899 2021-07-31T04:18:39.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 6491 : cephadm 2021-07-31T04:18:39.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: [ERR] 2021-07-31T04:18:39.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:18:39.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:18:39.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: yield 2021-07-31T04:18:39.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:18:39.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:18:39.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:18:39.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:18:39.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:18:39.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:18:39.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:18:39.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:18:39.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:18:39.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: task.result() 2021-07-31T04:18:39.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:18:39.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: cephadm 2021-07-31T04:18:38.121651+0000 mgr.x (mgr.24542) 6492 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:18:39.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:18:39.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:18:39.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: yield 2021-07-31T04:18:39.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:18:39.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:18:39.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:18:39.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:18:39.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:18:39.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:18:39.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:18:39.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:18:39.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:18:39.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: task.result() 2021-07-31T04:18:39.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:18:39.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: 2021-07-31T04:18:39.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T04:18:39.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: 2021-07-31T04:18:39.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:18:39.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:18:39.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: return f(*arg) 2021-07-31T04:18:39.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:18:39.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:18:39.554 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: cluster 2021-07-31T04:18:37.965097 2021-07-31T04:18:39.554 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 6485 : cluster [DBG] pgmap v6550: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:39.554 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: cephadm 2021-07-31T04:18:38 2021-07-31T04:18:39.555 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: .038683+0000 mgr.x (mgr.24542) 6486 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T04:18:39.555 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: cephadm 2021-07-31T04:18:38.081572+0000 mgr.x (mgr.24542) 6487 : 2021-07-31T04:18:39.555 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: cephadm [ERR] 2021-07-31T04:18:39.555 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:18:39.555 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:18:39.556 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: yield 2021-07-31T04:18:39.556 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:18:39.556 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:18:39.556 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:18:39.556 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:18:39.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:18:39.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:18:39.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:18:39.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:18:39.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:18:39.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: task.result() 2021-07-31T04:18:39.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:18:39.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: cephadm 2021-07-31T04:18:38 2021-07-31T04:18:39.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: .082026+0000 mgr.x (mgr.24542) 6488 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T04:18:39.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: cephadm 2021-07-31T04:18:38.093219+0000 mgr.x (mgr.24542) 6489 : cephadm [ERR] 2021-07-31T04:18:39.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:18:39.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:18:39.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: yield 2021-07-31T04:18:39.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:18:39.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:18:39.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:18:39.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:18:39.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:18:39.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:18:39.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:18:39.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:18:39.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:18:39.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: task.result() 2021-07-31T04:18:39.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:18:39.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: cephadm 2021-07-31T04:18:38.093750+0000 mgr.x (mgr.24542) 6490 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T04:18:39.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: cephadm 2021-07-31T04:18:38.117899 2021-07-31T04:18:39.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 6491 : cephadm [ERR] 2021-07-31T04:18:39.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:18:39.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:18:39.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: yield 2021-07-31T04:18:39.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:18:39.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:18:39.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:18:39.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:18:39.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:18:39.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:18:39.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:18:39.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:18:39.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:18:39.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: task.result() 2021-07-31T04:18:39.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:18:39.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: cephadm 2021-07-31T04:18:38.121651+0000 mgr.x (mgr.24542) 6492 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:18:39.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:18:39.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:18:39.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: yield 2021-07-31T04:18:39.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:18:39.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:18:39.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:18:39.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:18:39.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:18:39.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:18:39.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:18:39.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:18:39.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:18:39.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: task.result() 2021-07-31T04:18:39.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:18:39.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: 2021-07-31T04:18:39.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T04:18:39.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: 2021-07-31T04:18:39.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:18:39.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:18:39.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: return f(*arg) 2021-07-31T04:18:39.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:18:39.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:18:39.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:18:39.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:18:39.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:18:39.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:18:39.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:18:39.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T04:18:39.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:18:39.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: raise self._exception 2021-07-31T04:18:39.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:18:39.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:18:39.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:18:39.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T04:18:39.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:18:39.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: raise 2021-07-31T04:18:39.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:18:39.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T04:18:39.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:18:39.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:18:39.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:18:39.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: cephadm 2021-07-31T04:18:38.132461+0000 mgr.x (mgr.24542) 6493 : cephadm [ERR] 2021-07-31T04:18:39.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:18:39.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:18:39.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: yield 2021-07-31T04:18:39.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:18:39.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:18:39.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:18:39.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:18:39.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:18:39.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:18:39.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:18:39.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:18:39.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:18:39.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: task.result() 2021-07-31T04:18:39.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:18:39.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: cephadm 2021- 2021-07-31T04:18:39.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: 07-31T04:18:38.133040+0000 mgr.x (mgr.24542) 6494 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:18:39.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:18:39.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:18:39.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: yield 2021-07-31T04:18:39.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:18:39.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:18:39.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:18:39.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:18:39.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:18:39.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:18:39.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:18:39.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T04:18:39.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:18:39.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: raise self._exception 2021-07-31T04:18:39.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:18:39.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:18:39.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:18:39.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T04:18:39.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:18:39.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: raise 2021-07-31T04:18:39.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:18:39.580 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T04:18:39.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:18:39.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:18:39.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:18:39.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: cephadm 2021-07-31T04:18:38.132461+0000 mgr.x (mgr.24542) 6493 : cephadm [ERR] 2021-07-31T04:18:39.581 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:18:39.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:18:39.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: yield 2021-07-31T04:18:39.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:18:39.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:18:39.582 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:18:39.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:18:39.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:18:39.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:18:39.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:18:39.583 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:18:39.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:18:39.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: task.result() 2021-07-31T04:18:39.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:18:39.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: cephadm 2021-07-31T04:18:38.133040+0000 mgr.x (mgr.24542) 6494 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:18:39.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:18:39.584 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:18:39.585 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: yield 2021-07-31T04:18:39.585 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:18:39.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:18:39.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:18:39.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:18:39.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:18:39.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:18:39.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:18:39.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:18:39.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: task.result() 2021-07-31T04:18:39.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:18:39.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: 2021-07-31T04:18:39.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T04:18:39.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: 2021-07-31T04:18:39.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:18:39.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:18:39.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: return f(*arg) 2021-07-31T04:18:39.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:18:39.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:18:39.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:18:39.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:18:39.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:18:39.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:18:39.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:18:39.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T04:18:39.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:18:39.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: raise self._exception 2021-07-31T04:18:39.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:18:39.590 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:18:39.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:18:39.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T04:18:39.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:18:39.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: raise 2021-07-31T04:18:39.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:18:39.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T04:18:39.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:18:39.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:18:39.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:18:39.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:18:39.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:18:39.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:18:39.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:18:39.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:18:39.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:18:39.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:18:39.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:18:39.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: task.result() 2021-07-31T04:18:39.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:18:39.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: 2021-07-31T04:18:39.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T04:18:39.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: 2021-07-31T04:18:39.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:18:39.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:18:39.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: return f(*arg) 2021-07-31T04:18:39.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:18:39.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:18:39.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:18:39.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:18:39.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:18:39.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:18:39.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:18:39.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T04:18:39.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:18:39.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: raise self._exception 2021-07-31T04:18:39.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:18:39.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:18:39.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:18:39.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T04:18:39.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:18:39.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: raise 2021-07-31T04:18:39.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:18:39.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T04:18:39.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:18:39.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:18:39.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:39 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:18:41.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:41 smithi154 conmon[34553]: cluster 2021-07-31T04:18:39.965637+0000 mgr.x (mgr.24542) 6495 : cluster [DBG] 2021-07-31T04:18:41.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:41 smithi154 conmon[34553]: pgmap v6551: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:41.524 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:41 smithi045 conmon[33655]: cluster 2021-07-31T04:18:39.965637+0000 mgr.x (mgr.24542) 2021-07-31T04:18:41.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:41 smithi045 conmon[33655]: 6495 : cluster [DBG] pgmap v6551: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:41.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:41 smithi045 conmon[39718]: cluster 2021-07-31T04:18:39.965637 2021-07-31T04:18:41.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:41 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 6495 : cluster [DBG] pgmap v6551: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:43.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:43 smithi154 conmon[34553]: cluster 2021-07-31T04:18:41.966163+0000 mgr.x (mgr.24542) 6496 : cluster 2021-07-31T04:18:43.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:43 smithi154 conmon[34553]: [DBG] pgmap v6552: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:43.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:43 smithi045 conmon[39718]: cluster 2021-07-31T04:18:41.966163+0000 mgr.x (mgr.24542) 6496 : cluster [DBG] pgmap v6552: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:43.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:43 smithi045 conmon[33655]: cluster 2021-07-31T04:18:41.966163+0000 mgr.x (mgr.24542 2021-07-31T04:18:43.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:43 smithi045 conmon[33655]: ) 6496 : cluster [DBG] pgmap v6552: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:45.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:45 smithi154 conmon[34553]: cluster 2021-07-31T04:18:43.966671+0000 mgr.x (mgr.24542) 6497 : cluster [DBG] pgmap v6553: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:45.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:45 smithi045 conmon[39718]: cluster 2021-07-31T04:18:43.966671+0000 mgr.x (mgr.24542) 6497 : cluster [DBG] 2021-07-31T04:18:45.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:45 smithi045 conmon[39718]: pgmap v6553: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:45.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:45 smithi045 conmon[33655]: cluster 2021-07-31T 2021-07-31T04:18:45.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:45 smithi045 conmon[33655]: 04:18:43.966671+0000 mgr.x (mgr.24542) 6497 : cluster [DBG] pgmap v6553: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:47.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:47 smithi154 conmon[34553]: cluster 2021-07-31T04:18:45.967067+0000 mgr.x (mgr.24542) 6498 : cluster 2021-07-31T04:18:47.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:47 smithi154 conmon[34553]: [DBG] pgmap v6554: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:47.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:47 smithi045 conmon[39718]: cluster 2021-07-31T04:18:45.967067+0000 mgr.x (mgr.24542) 6498 : cluster 2021-07-31T04:18:47.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:47 smithi045 conmon[39718]: [DBG] pgmap v6554: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:47.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:47 smithi045 conmon[33655]: cluster 2021-07-31T04:18:45.967067+0000 mgr.x (mgr.24542) 6498 : 2021-07-31T04:18:47.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:47 smithi045 conmon[33655]: cluster [DBG] pgmap v6554: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:49.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:49 smithi154 conmon[34553]: cluster 2021-07-31T04:18:47.967491+0000 mgr.x (mgr.24542) 6499 : cluster [DBG] pgmap v6555: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:49.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:49 smithi045 conmon[33655]: cluster 2021-07-31T04:18:47.967491+0000 mgr.x (mgr.24542) 6499 : 2021-07-31T04:18:49.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:49 smithi045 conmon[33655]: cluster [DBG] pgmap v6555: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:49.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:49 smithi045 conmon[39718]: cluster 2021-07-31T04:18:47.967491+0000 mgr.x (mgr.24542) 6499 : cluster [DBG] 2021-07-31T04:18:49.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:49 smithi045 conmon[39718]: pgmap v6555: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:51.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:51 smithi154 conmon[34553]: cluster 2021-07-31T04:18:49.967996+0000 mgr.x (mgr.24542) 6500 : cluster [DBG] pgmap v6556: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:51.524 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:51 smithi045 conmon[39718]: cluster 2021-07-31T04:18:49.967996+0000 mgr.x (mgr.24542 2021-07-31T04:18:51.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:51 smithi045 conmon[39718]: ) 6500 : cluster [DBG] pgmap v6556: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:51.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:51 smithi045 conmon[33655]: cluster 2021-07-31T04:18 2021-07-31T04:18:51.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:51 smithi045 conmon[33655]: :49.967996+0000 mgr.x (mgr.24542) 6500 : cluster [DBG] pgmap v6556: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:53.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:53 smithi154 conmon[34553]: cluster 2021-07-31T04:18:51.968522+0000 mgr.x (mgr.24542) 6501 : cluster [DBG] pgmap v6557: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:53.524 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:53 smithi045 conmon[33655]: cluster 2021-07-31T04:18:51.968522+0000 mgr.x (mgr.24542) 6501 : 2021-07-31T04:18:53.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:53 smithi045 conmon[33655]: cluster [DBG] pgmap v6557: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:53.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:53 smithi045 conmon[39718]: cluster 2021-07-31T04:18:51.968522+0000 mgr.x (mgr.24542 2021-07-31T04:18:53.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:53 smithi045 conmon[39718]: ) 6501 : cluster [DBG] pgmap v6557: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:55.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:55 smithi154 conmon[34553]: cluster 2021-07-31T04:18:53.969135+0000 mgr.x (mgr.24542) 6502 : cluster [DBG] pgmap v6558: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:55.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:55 smithi045 conmon[33655]: cluster 2021-07-31T04:18:53.969135+0000 mgr.x (mgr.24542) 6502 2021-07-31T04:18:55.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:55 smithi045 conmon[33655]: : cluster [DBG] pgmap v6558: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:55.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:55 smithi045 conmon[39718]: cluster 2021-07-31T04:18:53.969135+0000 mgr.x (mgr 2021-07-31T04:18:55.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:55 smithi045 conmon[39718]: .24542) 6502 : cluster [DBG] pgmap v6558: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:57.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:57 smithi154 conmon[34553]: cluster 2021-07-31T04:18:55.969485+0000 mgr.x (mgr.24542) 6503 : cluster [DBG] pgmap v6559: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:57.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:57 smithi045 conmon[33655]: cluster 2021-07-31T04:18:55.969485+0000 mgr.x (mgr.24542) 6503 : cluster [DBG] pgmap v6559: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:57.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:57 smithi045 conmon[39718]: cluster 2021-07-31T04:18:55.969485+0000 mgr.x (mgr.24542) 6503 : cluster [DBG] pgmap v6559: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:59.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:18:59 smithi154 conmon[34553]: cluster 2021-07-31T04:18:57.970029+0000 mgr.x (mgr.24542) 6504 : cluster [DBG] pgmap v6560: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:59.524 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:18:59 smithi045 conmon[39718]: cluster 2021-07-31T04:18:57.970029+0000 mgr.x (mgr.24542) 6504 : cluster [DBG] pgmap v6560: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:18:59.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:59 smithi045 conmon[33655]: cluster 2021-07-31T04:18:57.970029+0000 mgr.x (mgr 2021-07-31T04:18:59.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:18:59 smithi045 conmon[33655]: .24542) 6504 : cluster [DBG] pgmap v6560: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:00.399 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T04:19:00.680 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T04:19:00.680 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 2h ago 2h - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T04:19:00.680 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 2h ago 2h 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T04:19:00.680 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 2h ago 2h 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T04:19:00.681 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 2h ago 2h - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T04:19:00.681 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T04:19:00.681 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 2h ago 2h 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T04:19:00.681 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T04:19:00.681 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 2h ago 2h - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T04:19:00.682 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 2h ago 2h 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T04:19:00.682 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T04:19:00.682 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T04:19:00.682 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T04:19:00.683 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T04:19:00.683 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 2h ago 2h 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T04:19:00.683 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 2h ago 2h 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T04:19:00.683 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 2h ago 2h 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T04:19:00.683 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 2h ago 2h 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T04:19:00.684 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 2h ago 2h 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T04:19:01.021 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T04:19:01.022 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T04:19:01.022 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T04:19:01.022 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:19:01.022 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T04:19:01.023 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T04:19:01.023 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:19:01.023 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:19:01.023 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T04:19:01.023 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T04:19:01.024 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:19:01.024 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T04:19:01.024 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T04:19:01.024 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T04:19:01.024 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:19:01.025 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T04:19:01.025 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T04:19:01.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:01 smithi154 conmon[34553]: cluster 2021-07-31T04:18:59.970548+0000 mgr.x (mgr.24542) 6505 : cluster [DBG] pgmap v6561: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:01.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:01 smithi154 conmon[34553]: audit 2021-07-31T04:19:00.388049+0000 mgr.x (mgr.24542) 6506 : audit [DBG] 2021-07-31T04:19:01.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:01 smithi154 conmon[34553]: from='client.31279 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:19:01.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:01 smithi154 conmon[34553]: audit 2021-07-31T04:19:01.022717+0000 mon.c (mon.1) 264 : audit [DBG] from='client.? 172.21.15.45:0/3495093343' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:19:01.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:01 smithi045 conmon[39718]: cluster 2021-07-31T04:18:59.970548+0000 2021-07-31T04:19:01.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:01 smithi045 conmon[39718]: mgr.x (mgr.24542) 6505 : cluster [DBG] pgmap v6561: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:01.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:01 smithi045 conmon[39718]: audit 2021-07-31T04:19:01.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:01 smithi045 conmon[39718]: 2021-07-31T04:19:00.388049+0000 mgr.x ( 2021-07-31T04:19:01.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:01 smithi045 conmon[39718]: mgr.24542) 6506 : audit [DBG] from='client.31279 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:19:01.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:01 smithi045 conmon[39718]: 2021-07-31T04:19:01.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:01 smithi045 conmon[39718]: audit 2021-07-31T04:19:01 2021-07-31T04:19:01.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:01 smithi045 conmon[39718]: .022717+0000 mon.c (mon.1) 264 : audit 2021-07-31T04:19:01.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:01 smithi045 conmon[39718]: [DBG] from='client.? 172.21.15.45:0/3495093343' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:19:01.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:01 smithi045 conmon[33655]: cluster 2021-07-31T04:18:59.970548+0000 mgr.x (mgr.24542 2021-07-31T04:19:01.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:01 smithi045 conmon[33655]: ) 6505 : cluster [DBG] pgmap v6561: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:01.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:01 smithi045 conmon[33655]: audit 2021-07-31T04:19:00.388049+0000 mgr.x (mgr.24542) 6506 : audit [DBG] from='client.31279 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:19:01.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:01 smithi045 conmon[33655]: audit 2021-07-31T 2021-07-31T04:19:01.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:01 smithi045 conmon[33655]: 04:19:01.022717+0000 mon.c (mon.1) 264 : audit [DBG] from='client.? 172.21.15.45:0/3495093343' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:19:02.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:02 smithi154 conmon[34553]: audit 2021-07-31T04:19:00.672633+0000 mgr.x (mgr.24542) 6507 : audit [DBG] from='client.31285 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:19:02.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:02 smithi045 conmon[39718]: audit 2021-07-31T04:19:00. 2021-07-31T04:19:02.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:02 smithi045 conmon[39718]: 672633+0000 mgr.x (mgr.24542) 6507 : audit [DBG] from='client.31285 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:19:02.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:02 smithi045 conmon[33655]: audit 2021-07-31T04:19:00.672633+0000 mgr.x (mgr.24542) 6507 : audit 2021-07-31T04:19:02.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:02 smithi045 conmon[33655]: [DBG] from='client.31285 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:19:03.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:03 smithi154 conmon[34553]: cluster 2021-07-31T04:19:01.970973+0000 mgr.x (mgr.24542) 6508 : 2021-07-31T04:19:03.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:03 smithi154 conmon[34553]: cluster [DBG] pgmap v6562: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:03.524 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:03 smithi045 conmon[33655]: cluster 2021-07-31T04:19:01.970973+0000 mgr.x (mgr.24542) 6508 2021-07-31T04:19:03.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:03 smithi045 conmon[33655]: : cluster [DBG] pgmap v6562: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:03.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:03 smithi045 conmon[39718]: cluster 2021-07-31T04:19:01.970973+0000 2021-07-31T04:19:03.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:03 smithi045 conmon[39718]: mgr.x (mgr.24542) 6508 : cluster [DBG] pgmap v6562: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:05.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:05 smithi154 conmon[34553]: cluster 2021-07-31T04:19:03.971497+0000 mgr.x (mgr.24542) 6509 : cluster [DBG] pgmap v6563: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:05.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:05 smithi045 conmon[39718]: cluster 2021-07-31T04:19:03.971497+0000 mgr.x (mgr.24542) 6509 : cluster [DBG] pgmap v6563: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:05.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:05 smithi045 conmon[39718]: 2021-07-31T04:19:05.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:05 smithi045 conmon[33655]: cluster 2021-07-31T04:19:03.971497+0000 mgr.x (mgr.24542) 6509 : cluster [DBG] pgmap v6563: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:07.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:07 smithi154 conmon[34553]: cluster 2021-07-31T04:19:05.971857+0000 mgr.x (mgr.24542) 6510 : cluster [DBG] pgmap v6564: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:07.524 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:07 smithi045 conmon[39718]: cluster 2021-07-31T04:19:05.971857+0000 mgr.x (mgr.24542 2021-07-31T04:19:07.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:07 smithi045 conmon[39718]: ) 6510 : cluster [DBG] pgmap v6564: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:07.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:07 smithi045 conmon[33655]: cluster 2021-07-31T04:19:05.971857+0000 mgr.x (mgr.24542) 6510 : cluster 2021-07-31T04:19:07.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:07 smithi045 conmon[33655]: [DBG] pgmap v6564: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:09.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:09 smithi154 conmon[34553]: cluster 2021-07-31T04:19:07.972321+0000 mgr.x (mgr.24542) 6511 : cluster [DBG] pgmap v6565: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:09.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:09 smithi045 conmon[33655]: cluster 2021-07-31T04:19:07.972321+0000 mgr.x (mgr. 2021-07-31T04:19:09.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:09 smithi045 conmon[33655]: 24542) 6511 : cluster [DBG] pgmap v6565: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:09.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:09 smithi045 conmon[39718]: cluster 2021-07-31T04:19:07.972321+0000 mgr.x (mgr.24542 2021-07-31T04:19:09.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:09 smithi045 conmon[39718]: ) 6511 : cluster [DBG] pgmap v6565: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:11.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:11 smithi154 conmon[34553]: cluster 2021-07-31T04:19:09.972933+0000 mgr.x (mgr.24542) 6512 : cluster [DBG] 2021-07-31T04:19:11.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:11 smithi154 conmon[34553]: pgmap v6566: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:11.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:11 smithi045 conmon[33655]: cluster 2021-07-31T04:19:09.972933 2021-07-31T04:19:11.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:11 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 6512 : cluster [DBG] pgmap v6566: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:11.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:11 smithi045 conmon[39718]: cluster 2021-07-31T04:19:09.972933+0000 mgr.x (mgr 2021-07-31T04:19:11.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:11 smithi045 conmon[39718]: .24542) 6512 : cluster [DBG] pgmap v6566: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:13.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:13 smithi154 conmon[34553]: cluster 2021-07-31T04:19:11.973357+0000 mgr.x (mgr.24542) 6513 : cluster [DBG] pgmap v6567: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:13.524 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:13 smithi045 conmon[33655]: cluster 2021-07-31T04:19:11.973357+0000 mgr.x (mgr.24542 2021-07-31T04:19:13.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:13 smithi045 conmon[33655]: ) 6513 : cluster [DBG] pgmap v6567: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:13.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:13 smithi045 conmon[39718]: cluster 2021-07-31T04:19:11.973357+0000 mgr.x (mgr.24542) 6513 : cluster [DBG] 2021-07-31T04:19:13.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:13 smithi045 conmon[39718]: pgmap v6567: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:15.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:15 smithi154 conmon[34553]: cluster 2021-07-31T04:19:13.973906+0000 mgr.x (mgr.24542) 6514 : cluster [DBG] pgmap v6568: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:15.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:15 smithi045 conmon[39718]: cluster 2021-07-31T04:19:13.973906+0000 mgr.x ( 2021-07-31T04:19:15.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:15 smithi045 conmon[39718]: mgr.24542) 6514 : cluster [DBG] pgmap v6568: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:15.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:15 smithi045 conmon[33655]: cluster 2021-07-31T04:19:13.973906+0000 mgr.x (mgr.24542) 6514 : cluster [DBG] 2021-07-31T04:19:15.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:15 smithi045 conmon[33655]: pgmap v6568: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:17.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:17 smithi154 conmon[34553]: cluster 2021-07-31T04:19:15.974307+0000 mgr.x (mgr.24542) 6515 : cluster [DBG] pgmap v6569: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:17.524 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:17 smithi045 conmon[33655]: cluster 2021-07-31T04:19: 2021-07-31T04:19:17.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:17 smithi045 conmon[33655]: 15.974307+0000 mgr.x (mgr.24542) 6515 : cluster [DBG] pgmap v6569: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:17.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:17 smithi045 conmon[39718]: cluster 2021-07-31T04:19:15.974307+0000 mgr.x 2021-07-31T04:19:17.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:17 smithi045 conmon[39718]: (mgr.24542) 6515 : cluster [DBG] pgmap v6569: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:19.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:19 smithi154 conmon[34553]: cluster 2021-07-31T04:19:17.974744+0000 mgr.x (mgr.24542) 6516 : cluster [DBG] pgmap v6570: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:19.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:19 smithi045 conmon[39718]: cluster 2021-07-31T04:19:17.974744+0000 mgr.x (mgr 2021-07-31T04:19:19.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:19 smithi045 conmon[39718]: .24542) 6516 : cluster [DBG] pgmap v6570: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:19.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:19 smithi045 conmon[33655]: cluster 2021-07-31T04:19:17.974744+0000 mgr.x (mgr.24542) 6516 : 2021-07-31T04:19:19.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:19 smithi045 conmon[33655]: cluster [DBG] pgmap v6570: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:21.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:21 smithi154 conmon[34553]: cluster 2021-07-31T04:19:19.975215+0000 mgr.x (mgr.24542) 6517 : cluster [DBG] 2021-07-31T04:19:21.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:21 smithi154 conmon[34553]: pgmap v6571: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:21.524 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:21 smithi045 conmon[39718]: cluster 2021-07-31T04:19:19.975215 2021-07-31T04:19:21.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:21 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 6517 : cluster [DBG] pgmap v6571: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:21.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:21 smithi045 conmon[33655]: cluster 2021-07-31T04:19:19.975215+0000 mgr.x (mgr.24542) 6517 : cluster [DBG] pgmap v6571: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:23.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:23 smithi154 conmon[34553]: cluster 2021-07-31T04:19:21.975532+0000 mgr.x (mgr.24542) 6518 : 2021-07-31T04:19:23.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:23 smithi154 conmon[34553]: cluster [DBG] pgmap v6572: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:23.524 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:23 smithi045 conmon[39718]: cluster 2021-07-31T04:19:21.975532+0000 mgr.x ( 2021-07-31T04:19:23.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:23 smithi045 conmon[39718]: mgr.24542) 6518 : cluster [DBG] pgmap v6572: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:23.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:23 smithi045 conmon[33655]: cluster 2021-07-31T04:19:21.975532+0000 mgr.x (mgr.24542) 6518 : cluster [DBG] pgmap v6572: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:25.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:25 smithi154 conmon[34553]: cluster 2021-07-31T04:19:23.976077+0000 mgr.x (mgr.24542) 6519 : cluster [DBG] pgmap v6573: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:25.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:25 smithi045 conmon[33655]: cluster 2021-07-31T04:19:23.976077 2021-07-31T04:19:25.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:25 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 6519 : cluster [DBG] pgmap v6573: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:25.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:25 smithi045 conmon[39718]: cluster 2021-07-31T04:19:23.976077+0000 mgr.x (mgr 2021-07-31T04:19:25.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:25 smithi045 conmon[39718]: .24542) 6519 : cluster [DBG] pgmap v6573: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:27.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:27 smithi154 conmon[34553]: cluster 2021-07-31T04:19:25.976588+0000 mgr.x (mgr.24542) 6520 : cluster [DBG] pgmap v6574: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:27.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:27 smithi045 conmon[33655]: cluster 2021-07-31T04:19:25 2021-07-31T04:19:27.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:27 smithi045 conmon[33655]: .976588+0000 mgr.x (mgr.24542) 6520 : cluster [DBG] pgmap v6574: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:27.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:27 smithi045 conmon[39718]: cluster 2021-07-31T04:19:25.976588+0000 2021-07-31T04:19:27.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:27 smithi045 conmon[39718]: mgr.x (mgr.24542) 6520 : cluster [DBG] pgmap v6574: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:29.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:29 smithi154 conmon[34553]: cluster 2021-07-31T04:19:27.977083+0000 mgr.x (mgr.24542) 6521 : cluster [DBG] 2021-07-31T04:19:29.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:29 smithi154 conmon[34553]: pgmap v6575: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:29 smithi154 conmon[34553]: audit 2021-07-31T04:19:28.534888+0000 mon.b (mon.2) 835 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:19:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:29 smithi154 conmon[34553]: audit 2021-07-31T04:19:28.535215+0000 mon.b (mon.2) 836 : 2021-07-31T04:19:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:29 smithi154 conmon[34553]: audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:19:29.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:29 smithi154 conmon[34553]: audit 2021-07-31T04:19:28.535951+0000 mon.a ( 2021-07-31T04:19:29.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:29 smithi154 conmon[34553]: mon.0) 3458 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:19:29.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:29 smithi154 conmon[34553]: audit 2021-07-31T 2021-07-31T04:19:29.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:29 smithi154 conmon[34553]: 04:19:28.536394+0000 mon.a (mon. 2021-07-31T04:19:29.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:29 smithi154 conmon[34553]: 0) 3459 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:19:29.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:29 smithi045 conmon[33655]: cluster 2021-07-31T04:19:27.977083+0000 mgr.x (mgr.24542) 6521 : 2021-07-31T04:19:29.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:29 smithi045 conmon[33655]: cluster [DBG] pgmap v6575: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:29.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:29 smithi045 conmon[33655]: audit 2021-07-31T04:19:28.534888+0000 mon.b (mon.2) 835 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:19:29.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:29 smithi045 conmon[33655]: audit 2021-07-31T04:19:29.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:29 smithi045 conmon[33655]: 2021-07-31T04:19:28.535215+0000 mon.b (mon.2) 836 : audit 2021-07-31T04:19:29.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:29 smithi045 conmon[33655]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:19:29.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:29 smithi045 conmon[33655]: audit 2021-07-31T04:19:29.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:29 smithi045 conmon[33655]: 2021-07-31T04:19 2021-07-31T04:19:29.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:29 smithi045 conmon[33655]: :28.535951+0000 mon.a (mon.0 2021-07-31T04:19:29.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:29 smithi045 conmon[33655]: ) 3458 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:19:29.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:29 smithi045 conmon[33655]: 2021-07-31T04:19:29.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:29 smithi045 conmon[33655]: audit 2021-07- 2021-07-31T04:19:29.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:29 smithi045 conmon[33655]: 31T04:19:28.536394+0000 mon.a (mon.0) 3459 2021-07-31T04:19:29.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:29 smithi045 conmon[33655]: : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:19:29.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:29 smithi045 conmon[39718]: cluster 2021-07-31T04:19:27.977083+0000 2021-07-31T04:19:29.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:29 smithi045 conmon[39718]: mgr.x (mgr.24542) 6521 : cluster [DBG] pgmap v6575: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:29.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:29 smithi045 conmon[39718]: audit 2021-07-31T04:19:29.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:29 smithi045 conmon[39718]: 2021-07-31T04:19:28. 2021-07-31T04:19:29.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:29 smithi045 conmon[39718]: 534888+0000 mon.b (mon.2 2021-07-31T04:19:29.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:29 smithi045 conmon[39718]: ) 835 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:19:29.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:29 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T04:19:29.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:29 smithi045 conmon[39718]: 31T04:19:28.535215+0000 2021-07-31T04:19:29.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:29 smithi045 conmon[39718]: mon.b (mon.2) 836 : audit [INF] 2021-07-31T04:19:29.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:29 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:19:29.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:29 smithi045 conmon[39718]: audit 2021-07-31T04:19:29.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:29 smithi045 conmon[39718]: 2021-07 2021-07-31T04:19:29.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:29 smithi045 conmon[39718]: -31T04:19:28. 2021-07-31T04:19:29.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:29 smithi045 conmon[39718]: 535951+0000 mon.a (mon.0) 3458 2021-07-31T04:19:29.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:29 smithi045 conmon[39718]: : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:19:29.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:29 smithi045 conmon[39718]: audit 2021-07 2021-07-31T04:19:29.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:29 smithi045 conmon[39718]: -31T04:19:28.536394 2021-07-31T04:19:29.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:29 smithi045 conmon[39718]: +0000 mon.a (mon.0) 3459 : 2021-07-31T04:19:29.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:29 smithi045 conmon[39718]: audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:19:31.339 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T04:19:31.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:31 smithi154 conmon[34553]: cluster 2021-07-31T04:19:29.977476+0000 mgr.x (mgr.24542) 6522 : cluster [DBG] pgmap v6576: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:31.524 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:31 smithi045 conmon[39718]: cluster 2021-07-31T04:19:29.977476 2021-07-31T04:19:31.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:31 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 6522 : cluster [DBG] pgmap v6576: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:31.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:31 smithi045 conmon[33655]: cluster 2021-07-31T04:19: 2021-07-31T04:19:31.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:31 smithi045 conmon[33655]: 29.977476+0000 mgr.x (mgr.24542) 6522 : cluster [DBG] pgmap v6576: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:31.616 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T04:19:31.617 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 2h ago 2h - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T04:19:31.617 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 2h ago 2h 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T04:19:31.617 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 2h ago 2h 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T04:19:31.617 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 2h ago 2h - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T04:19:31.618 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T04:19:31.618 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 2h ago 2h 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T04:19:31.618 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T04:19:31.618 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 2h ago 2h - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T04:19:31.619 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 2h ago 2h 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T04:19:31.619 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T04:19:31.619 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T04:19:31.619 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T04:19:31.619 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T04:19:31.620 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 2h ago 2h 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T04:19:31.620 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 2h ago 2h 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T04:19:31.620 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 2h ago 2h 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T04:19:31.620 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 2h ago 2h 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T04:19:31.620 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 2h ago 2h 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T04:19:31.952 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T04:19:31.953 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T04:19:31.953 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T04:19:31.953 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:19:31.954 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T04:19:31.954 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T04:19:31.954 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:19:31.954 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:19:31.954 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T04:19:31.955 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T04:19:31.955 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:19:31.955 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T04:19:31.955 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T04:19:31.955 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T04:19:31.956 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:19:31.956 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T04:19:31.956 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T04:19:32.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:32 smithi154 conmon[34553]: audit 2021-07-31T04:19:31.327451+0000 mgr.x (mgr.24542) 6523 : audit [DBG] from='client.21360 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:19:32.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:32 smithi154 conmon[34553]: audit 2021-07-31T04:19:31 2021-07-31T04:19:32.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:32 smithi154 conmon[34553]: .953620+0000 mon.c (mon.1) 265 : audit [DBG] from='client.? 172.21.15.45:0/1956403773' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:19:32.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:32 smithi045 conmon[39718]: audit 2021-07-31T04:19:31.327451+0000 mgr.x (mgr. 2021-07-31T04:19:32.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:32 smithi045 conmon[39718]: 24542) 6523 : audit [DBG] from='client.21360 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:19:32.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:32 smithi045 conmon[39718]: audit 2021-07-31T04:19:31.953620+0000 mon.c (mon 2021-07-31T04:19:32.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:32 smithi045 conmon[39718]: .1) 265 : audit [DBG] from='client.? 172.21.15.45:0/1956403773' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:19:32.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:32 smithi045 conmon[33655]: audit 2021-07-31T04: 2021-07-31T04:19:32.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:32 smithi045 conmon[33655]: 19:31.327451+0000 mgr.x (mgr.24542) 6523 : audit [DBG] from='client.21360 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:19:32.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:32 smithi045 conmon[33655]: audit 2021-07-31T04:19:31.953620+0000 mon.c (mon.1) 265 : audit [DBG] from='client.? 172.21.15.45:0/1956403773' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:19:33.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:33 smithi154 conmon[34553]: audit 2021-07-31T04:19:31.609021+0000 mgr.x (mgr.24542) 6524 : audit [DBG] from='client.31303 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:19:33.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:33 smithi154 conmon[34553]: cluster 2021-07-31T04:19:31.977999+0000 mgr.x (mgr.24542) 6525 : cluster [DBG] pgmap v6577: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:33.524 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:33 smithi045 conmon[33655]: audit 2021-07-31T04:19:31.609021+0000 mgr.x (mgr.24542 2021-07-31T04:19:33.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:33 smithi045 conmon[33655]: ) 6524 : audit [DBG] from='client.31303 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:19:33.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:33 smithi045 conmon[33655]: cluster 2021-07-31T04:19:31.977999+0000 mgr.x (mgr.24542) 6525 : cluster [DBG] pgmap v6577: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:33.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:33 smithi045 conmon[39718]: audit 2021-07-31T04:19:31 2021-07-31T04:19:33.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:33 smithi045 conmon[39718]: .609021+0000 mgr.x (mgr.24542) 6524 : audit [DBG] from='client.31303 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:19:33.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:33 smithi045 conmon[39718]: cluster 2021-07 2021-07-31T04:19:33.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:33 smithi045 conmon[39718]: -31T04:19:31.977999 2021-07-31T04:19:33.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:33 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 2021-07-31T04:19:33.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:33 smithi045 conmon[39718]: 6525 : cluster [DBG] pgmap v6577: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:35.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:35 smithi154 conmon[34553]: cluster 2021-07-31T04:19:33.978404+0000 mgr.x (mgr.24542) 6526 : cluster [DBG] pgmap v6578: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:35.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:35 smithi045 conmon[39718]: cluster 2021-07-31T04:19:33.978404+0000 mgr.x ( 2021-07-31T04:19:35.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:35 smithi045 conmon[39718]: mgr.24542) 6526 : cluster [DBG] pgmap v6578: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:35.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:35 smithi045 conmon[33655]: cluster 2021-07-31T04:19:33.978404+0000 mgr.x (mgr.24542) 6526 : cluster [DBG] 2021-07-31T04:19:35.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:35 smithi045 conmon[33655]: pgmap v6578: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:37.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:37 smithi154 conmon[34553]: cluster 2021-07-31T04:19:35.978775+0000 mgr.x (mgr.24542) 6527 : cluster [DBG] pgmap v6579: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:37.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:37 smithi045 conmon[39718]: cluster 2021-07-31T04:19:35.978775+0000 mgr.x (mgr.24542) 6527 : cluster [DBG] 2021-07-31T04:19:37.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:37 smithi045 conmon[39718]: pgmap v6579: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:37.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:37 smithi045 conmon[33655]: cluster 2021-07-31T04:19:35.978775+0000 mgr.x (mgr.24542) 6527 : cluster [DBG] pgmap v6579: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:37.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:37 smithi045 conmon[33655]: 2021-07-31T04:19:38.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[34553]: audit 2021-07-31T04:19: 2021-07-31T04:19:38.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[34553]: 38.133802+0000 mon.b (mon.2) 837 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T04:19:38.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[34553]: audit 2021-07-31T04: 2021-07-31T04:19:38.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[34553]: 19:38.135316+0000 mon.b (mon 2021-07-31T04:19:38.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[34553]: .2) 838 : audit [DBG] 2021-07-31T04:19:38.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[34553]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T04:19:38.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[34553]: audit 2021- 2021-07-31T04:19:38.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[34553]: 07-31T04:19:38.136230 2021-07-31T04:19:38.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[34553]: +0000 mon.b (mon.2) 2021-07-31T04:19:38.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[34553]: 839 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T04:19:38.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: debug 2021-07-31T04:19:38.179+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:19:38.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:19:38.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:19:38.505 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: yield 2021-07-31T04:19:38.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:19:38.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:19:38.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:19:38.506 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:19:38.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:19:38.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:19:38.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:19:38.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:19:38.507 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:19:38.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: task.result() 2021-07-31T04:19:38.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:19:38.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: debug 2021-07-31T04:19:38.191+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:19:38.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:19:38.508 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:19:38.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: yield 2021-07-31T04:19:38.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:19:38.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:19:38.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:19:38.509 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:19:38.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:19:38.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:19:38.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:19:38.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:19:38.510 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:19:38.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: task.result() 2021-07-31T04:19:38.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:19:38.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: debug 2021-07-31T04:19:38.216+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:19:38.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:19:38.511 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:19:38.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: yield 2021-07-31T04:19:38.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:19:38.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:19:38.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:19:38.512 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:19:38.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:19:38.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:19:38.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:19:38.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:19:38.513 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:19:38.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: task.result() 2021-07-31T04:19:38.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:19:38.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: debug 2021-07-31T04:19:38.216+0000 7fcb9215e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:19:38.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:19:38.514 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:19:38.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: yield 2021-07-31T04:19:38.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:19:38.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:19:38.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:19:38.515 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:19:38.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:19:38.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:19:38.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:19:38.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:19:38.516 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:19:38.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: task.result() 2021-07-31T04:19:38.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:19:38.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: 2021-07-31T04:19:38.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T04:19:38.517 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: 2021-07-31T04:19:38.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:19:38.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:19:38.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: return f(*arg) 2021-07-31T04:19:38.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:19:38.518 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:19:38.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:19:38.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:19:38.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:19:38.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:19:38.519 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:19:38.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T04:19:38.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:19:38.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: raise self._exception 2021-07-31T04:19:38.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:19:38.520 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:19:38.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:19:38.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T04:19:38.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:19:38.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: raise 2021-07-31T04:19:38.521 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:19:38.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T04:19:38.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:19:38.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:19:38.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:19:38.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: debug 2021-07-31T04:19:38.225+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:19:38.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:19:38.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:19:38.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: yield 2021-07-31T04:19:38.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:19:38.523 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:19:38.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:19:38.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:19:38.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:19:38.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:19:38.524 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:19:38.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:19:38.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:19:38.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: task.result() 2021-07-31T04:19:38.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:19:38.525 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: debug 2021-07-31T04:19:38.225+0000 7fcb9195d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:19:38.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:19:38.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:19:38.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: yield 2021-07-31T04:19:38.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:19:38.526 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:19:38.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:19:38.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:19:38.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:19:38.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:19:38.527 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:19:38.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:19:38.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:19:38.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: task.result() 2021-07-31T04:19:38.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:19:38.528 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: 2021-07-31T04:19:38.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T04:19:38.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: 2021-07-31T04:19:38.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:19:38.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:19:38.529 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: return f(*arg) 2021-07-31T04:19:38.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:19:38.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:19:38.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:19:38.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:19:38.530 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:19:38.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:19:38.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:19:38.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T04:19:38.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:19:38.531 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: raise self._exception 2021-07-31T04:19:38.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:19:38.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:19:38.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:19:38.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T04:19:38.532 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:19:38.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: raise 2021-07-31T04:19:38.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:19:38.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T04:19:38.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:19:38.533 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:19:38.534 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:19:38 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:19:38.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:38 smithi045 conmon[33655]: audit 2021-07-31T04:19:38.133802+0000 mon.b (mon.2) 837 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T04:19:38.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:38 smithi045 conmon[33655]: audit 2021-07-31T04:19:38.135316+0000 mon.b (mon.2) 838 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T04:19:38.535 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:38 smithi045 conmon[33655]: audit 2021-07- 2021-07-31T04:19:38.535 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:38 smithi045 conmon[33655]: 31T04:19:38.136230+0000 mon.b (mon.2) 839 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T04:19:38.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:38 smithi045 conmon[39718]: audit 2021-07-31T04:19:38. 2021-07-31T04:19:38.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:38 smithi045 conmon[39718]: 133802+0000 mon.b (mon.2) 837 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T04:19:38.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:38 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T04:19:38.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:38 smithi045 conmon[39718]: T04:19:38.135316+0000 2021-07-31T04:19:38.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:38 smithi045 conmon[39718]: mon.b (mon.2) 838 2021-07-31T04:19:38.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:38 smithi045 conmon[39718]: : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T04:19:38.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:38 smithi045 conmon[39718]: audit 2021 2021-07-31T04:19:38.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:38 smithi045 conmon[39718]: -07-31T04:19:38 2021-07-31T04:19:38.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:38 smithi045 conmon[39718]: .136230+0000 mon.b (mon. 2021-07-31T04:19:38.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:38 smithi045 conmon[39718]: 2) 839 : audit [INF] 2021-07-31T04:19:38.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:38 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T04:19:39.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: cluster 2021-07-31T04:19:37.979175+0000 mgr.x (mgr.24542) 6528 : cluster [DBG] pgmap v6580: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:39.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: 2021-07-31T04:19:39.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:19:38.136985+0000 mgr.x (mgr.24542) 6529 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T04:19:39.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:19:38.180472+0000 mgr.x (mgr.24542) 6530 : cephadm 2021-07-31T04:19:39.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: [ERR] 2021-07-31T04:19:39.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:19:39.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:19:39.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: yield 2021-07-31T04:19:39.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:19:39.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:19:39.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:19:39.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:19:39.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:19:39.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:19:39.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:19:39.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:19:39.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:19:39.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: task.result() 2021-07-31T04:19:39.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:19:39.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: cephadm 2021-07-31T04: 2021-07-31T04:19:39.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: 19:38.180899+0000 mgr.x (mgr.24542) 6531 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T04:19:39.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:19:38.191930+0000 mgr.x (mgr.24542) 6532 : cephadm [ERR] 2021-07-31T04:19:39.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:19:39.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:19:39.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: yield 2021-07-31T04:19:39.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:19:39.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:19:39.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:19:39.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:19:39.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:19:39.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:19:39.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:19:39.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:19:39.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:19:39.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: task.result() 2021-07-31T04:19:39.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:19:39.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:19:39.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: 2021-07-31T04:19:38.192351+0000 mgr.x (mgr.24542) 6533 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T04:19:39.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:19:38.217459+0000 mgr.x (mgr.24542) 6534 : cephadm [ERR] 2021-07-31T04:19:39.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:19:39.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:19:39.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: yield 2021-07-31T04:19:39.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:19:39.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:19:39.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:19:39.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:19:39.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:19:39.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:19:39.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:19:39.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:19:39.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:19:39.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: task.result() 2021-07-31T04:19:39.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:19:39.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:19:38.217886+0000 mgr.x (mgr.24542) 6535 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:19:39.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:19:39.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:19:39.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: yield 2021-07-31T04:19:39.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:19:39.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:19:39.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:19:39.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:19:39.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:19:39.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:19:39.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:19:39.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:19:39.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:19:39.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: task.result() 2021-07-31T04:19:39.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:19:39.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: 2021-07-31T04:19:39.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T04:19:39.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: 2021-07-31T04:19:39.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:19:39.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:19:39.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: return f(*arg) 2021-07-31T04:19:39.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:19:39.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:19:39.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:19:39.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:19:39.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:19:39.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:19:39.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:19:39.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T04:19:39.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:19:39.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: raise self._exception 2021-07-31T04:19:39.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:19:39.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:19:39.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:19:39.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T04:19:39.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:19:39.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: raise 2021-07-31T04:19:39.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:19:39.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T04:19:39.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:19:39.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:19:39.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:19:39.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:19:38.226148+0000 mgr.x (mgr.24542) 6536 : cephadm 2021-07-31T04:19:39.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: [ERR] 2021-07-31T04:19:39.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:19:39.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:19:39.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: yield 2021-07-31T04:19:39.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:19:39.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:19:39.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:19:39.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:19:39.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:19:39.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:19:39.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:19:39.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:19:39.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:19:39.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: task.result() 2021-07-31T04:19:39.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:19:39.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:19:38.226542+0000 mgr.x (mgr.24542) 6537 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:19:39.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:19:39.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:19:39.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: yield 2021-07-31T04:19:39.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:19:39.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:19:39.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:19:39.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:19:39.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:19:39.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:19:39.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:19:39.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:19:39.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:19:39.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: task.result() 2021-07-31T04:19:39.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:19:39.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: 2021-07-31T04:19:39.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T04:19:39.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: 2021-07-31T04:19:39.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:19:39.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:19:39.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: return f(*arg) 2021-07-31T04:19:39.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:19:39.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:19:39.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:19:39.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:19:39.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:19:39.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:19:39.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:19:39.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T04:19:39.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:19:39.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: raise self._exception 2021-07-31T04:19:39.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:19:39.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:19:39.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:19:39.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T04:19:39.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:19:39.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: raise 2021-07-31T04:19:39.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:19:39.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T04:19:39.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:19:39.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:19:39.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:39 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:19:39.539 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: cluster 2021-07-31T04:19:37.979175+0000 mgr.x (mgr.24542) 6528 2021-07-31T04:19:39.539 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: : cluster [DBG] pgmap v6580: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:39.539 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: cephadm 2021-07-31T04:19:38.136985+0000 mgr.x (mgr.24542) 6529 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T04:19:39.540 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: cephadm 2021-07-31T04:19:38.180472+0000 mgr.x (mgr.24542) 6530 : cephadm [ERR] 2021-07-31T04:19:39.540 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: 2021-07-31T04:19:39.540 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:19:39.540 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:19:39.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: yield 2021-07-31T04:19:39.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:19:39.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:19:39.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:19:39.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:19:39.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:19:39.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:19:39.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:19:39.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:19:39.543 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:19:39.543 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: task.result() 2021-07-31T04:19:39.543 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:19:39.543 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: cephadm 2021-07-31T04:19:38.180899+0000 mgr.x (mgr.24542) 6531 : cephadm [INF] 2021-07-31T04:19:39.543 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T04:19:39.544 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: cephadm 2021-07-31T04:19:39.544 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: 2021- 2021-07-31T04:19:39.544 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: 07-31T04:19: 2021-07-31T04:19:39.544 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: 38.191930+0000 mgr.x ( 2021-07-31T04:19:39.545 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: mgr.24542) 6532 : cephadm 2021-07-31T04:19:39.545 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: [ERR] 2021-07-31T04:19:39.545 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:19:39.545 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:19:39.545 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: yield 2021-07-31T04:19:39.546 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:19:39.546 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:19:39.546 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:19:39.547 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:19:39.547 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:19:39.547 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:19:39.547 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:19:39.547 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:19:39.548 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:19:39.548 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: task.result() 2021-07-31T04:19:39.548 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:19:39.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: cluster 2021-07-31T04:19:37 2021-07-31T04:19:39.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: .979175+0000 mgr.x (mgr.24542) 6528 : cluster [DBG] pgmap v6580: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:39.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: cephadm 2021-07- 2021-07-31T04:19:39.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: 31T04:19:38.136985 2021-07-31T04:19:39.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 2021-07-31T04:19:39.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: 6529 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T04:19:39.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: cephadm 2021- 2021-07-31T04:19:39.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: 07-31T04:19:38.180472 2021-07-31T04:19:39.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 2021-07-31T04:19:39.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: 6530 : cephadm [ERR] 2021-07-31T04:19:39.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:19:39.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:19:39.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: yield 2021-07-31T04:19:39.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:19:39.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:19:39.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:19:39.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:19:39.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:19:39.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:19:39.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:19:39.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:19:39.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:19:39.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: task.result() 2021-07-31T04:19:39.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:19:39.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: cephadm 2021-07-31T04:19:38.180899+0000 mgr.x (mgr.24542) 6531 : cephadm 2021-07-31T04:19:39.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T04:19:39.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: cephadm 2021-07-31 2021-07-31T04:19:39.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: T04:19:38.191930+0000 2021-07-31T04:19:39.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: mgr.x (mgr.24542) 6532 : cephadm 2021-07-31T04:19:39.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: [ERR] 2021-07-31T04:19:39.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:19:39.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:19:39.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: yield 2021-07-31T04:19:39.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:19:39.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:19:39.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:19:39.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:19:39.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:19:39.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:19:39.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:19:39.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:19:39.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:19:39.558 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: task.result() 2021-07-31T04:19:39.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: cephadm 2021-07-31T04:19:38.192351+0000 mgr.x (mgr.24542 2021-07-31T04:19:39.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: ) 6533 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T04:19:39.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: cephadm 2021- 2021-07-31T04:19:39.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: 07-31T04:19:38.217459 2021-07-31T04:19:39.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 2021-07-31T04:19:39.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: 6534 : cephadm [ERR] 2021-07-31T04:19:39.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:19:39.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:19:39.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: yield 2021-07-31T04:19:39.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:19:39.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:19:39.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:19:39.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:19:39.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:19:39.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:19:39.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:19:39.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:19:39.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:19:39.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: task.result() 2021-07-31T04:19:39.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:19:39.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: 2021-07-31T04:19:39.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: cephadm 2021-07-31T04:19:38.217886+0000 mgr.x (mgr.24542) 6535 : cephadm 2021-07-31T04:19:39.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:19:39.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:19:39.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:19:39.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: yield 2021-07-31T04:19:39.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:19:39.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:19:39.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:19:39.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:19:39.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:19:39.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:19:39.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:19:39.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:19:39.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:19:39.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: task.result() 2021-07-31T04:19:39.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:19:39.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: 2021-07-31T04:19:39.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T04:19:39.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: 2021-07-31T04:19:39.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:19:39.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:19:39.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: return f(*arg) 2021-07-31T04:19:39.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:19:39.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:19:39.568 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: cephadm 2021-07-31T04:19:38.192351+0000 mgr.x (mgr.24542) 6533 : cephadm [INF] 2021-07-31T04:19:39.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T04:19:39.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: cephadm 2021-07-31T04:19:39.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: 2021-07-31T04 2021-07-31T04:19:39.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: :19:38.217459+0000 2021-07-31T04:19:39.569 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: mgr.x (mgr.24542) 6534 : 2021-07-31T04:19:39.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: cephadm [ERR] 2021-07-31T04:19:39.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:19:39.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:19:39.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: yield 2021-07-31T04:19:39.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:19:39.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:19:39.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:19:39.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:19:39.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:19:39.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:19:39.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:19:39.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:19:39.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:19:39.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: task.result() 2021-07-31T04:19:39.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:19:39.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: cephadm 2021-07-31T04:19:38.217886+0000 mgr.x (mgr.24542 2021-07-31T04:19:39.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: ) 6535 : cephadm 2021-07-31T04:19:39.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:19:39.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:19:39.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:19:39.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: yield 2021-07-31T04:19:39.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:19:39.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:19:39.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:19:39.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:19:39.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:19:39.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:19:39.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:19:39.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:19:39.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:19:39.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: task.result() 2021-07-31T04:19:39.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:19:39.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: 2021-07-31T04:19:39.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T04:19:39.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: 2021-07-31T04:19:39.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:19:39.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:19:39.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:19:39.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:19:39.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:19:39.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:19:39.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:19:39.579 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:19:39.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T04:19:39.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:19:39.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: raise self._exception 2021-07-31T04:19:39.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:19:39.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:19:39.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:19:39.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T04:19:39.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:19:39.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: raise 2021-07-31T04:19:39.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:19:39.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T04:19:39.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:19:39.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:19:39.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:19:39.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: cephadm 2021-07-31T04:19:38.226148+0000 mgr.x (mgr.24542) 6536 : cephadm [ERR] 2021-07-31T04:19:39.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:19:39.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:19:39.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: yield 2021-07-31T04:19:39.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:19:39.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:19:39.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:19:39.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:19:39.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:19:39.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:19:39.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:19:39.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:19:39.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:19:39.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: task.result() 2021-07-31T04:19:39.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:19:39.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: cephadm 2021-07-31T04:19:38.226542+0000 mgr.x (mgr.24542) 6537 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:19:39.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:19:39.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:19:39.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: yield 2021-07-31T04:19:39.587 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: return f(*arg) 2021-07-31T04:19:39.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:19:39.588 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:19:39.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:19:39.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:19:39.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:19:39.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:19:39.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:19:39.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T04:19:39.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:19:39.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: raise self._exception 2021-07-31T04:19:39.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:19:39.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:19:39.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:19:39.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T04:19:39.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:19:39.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: raise 2021-07-31T04:19:39.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:19:39.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T04:19:39.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:19:39.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:19:39.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:19:39.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: cephadm 2021-07-31T04:19:38.226148+0000 mgr.x (mgr.24542) 6536 : cephadm [ERR] 2021-07-31T04:19:39.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:19:39.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:19:39.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: yield 2021-07-31T04:19:39.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:19:39.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:19:39.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:19:39.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:19:39.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:19:39.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:19:39.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:19:39.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:19:39.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:19:39.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: task.result() 2021-07-31T04:19:39.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:19:39.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: cephadm 2021-07-31T04:19:38.226542+0000 mgr.x (mgr.24542) 6537 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:19:39.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:19:39.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:19:39.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:19:39.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:19:39.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:19:39.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:19:39.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:19:39.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:19:39.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:19:39.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:19:39.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: task.result() 2021-07-31T04:19:39.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:19:39.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: 2021-07-31T04:19:39.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T04:19:39.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: 2021-07-31T04:19:39.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:19:39.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:19:39.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: return f(*arg) 2021-07-31T04:19:39.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:19:39.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:19:39.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:19:39.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:19:39.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:19:39.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:19:39.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:19:39.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T04:19:39.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:19:39.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: raise self._exception 2021-07-31T04:19:39.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:19:39.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:19:39.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:19:39.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T04:19:39.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:19:39.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: raise 2021-07-31T04:19:39.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:19:39.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T04:19:39.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:19:39.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:19:39.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:19:39.605 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:19:39.605 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: yield 2021-07-31T04:19:39.606 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:19:39.606 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:19:39.606 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:19:39.606 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:19:39.607 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:19:39.607 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:19:39.607 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:19:39.607 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:19:39.607 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:19:39.608 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: task.result() 2021-07-31T04:19:39.608 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:19:39.608 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: 2021-07-31T04:19:39.608 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T04:19:39.608 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: 2021-07-31T04:19:39.609 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:19:39.609 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:19:39.609 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: return f(*arg) 2021-07-31T04:19:39.609 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:19:39.609 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:19:39.610 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:19:39.610 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:19:39.610 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:19:39.610 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:19:39.611 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:19:39.611 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T04:19:39.611 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:19:39.611 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: raise self._exception 2021-07-31T04:19:39.611 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:19:39.612 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:19:39.612 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:19:39.612 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T04:19:39.612 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:19:39.612 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: raise 2021-07-31T04:19:39.613 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:19:39.613 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T04:19:39.613 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:19:39.614 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:19:39.614 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:39 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:19:41.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:41 smithi154 conmon[34553]: cluster 2021-07-31T04:19:39.979681+0000 mgr.x (mgr.24542) 6538 : cluster [DBG] pgmap v6581: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:41.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:41 smithi154 conmon[34553]: 2021-07-31T04:19:41.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:41 smithi045 conmon[33655]: cluster 2021-07-31T04:19:39.979681+0000 mgr.x (mgr.24542) 6538 : cluster [DBG] pgmap v6581: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:41.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:41 smithi045 conmon[33655]: 2021-07-31T04:19:41.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:41 smithi045 conmon[39718]: cluster 2021-07-31T04:19:39.979681 2021-07-31T04:19:41.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:41 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 6538 : cluster [DBG] pgmap v6581: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:43.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:43 smithi154 conmon[34553]: cluster 2021-07-31T04:19:41.980081+0000 mgr.x (mgr.24542) 6539 : cluster 2021-07-31T04:19:43.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:43 smithi154 conmon[34553]: [DBG] pgmap v6582: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:43.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:43 smithi045 conmon[39718]: cluster 2021-07-31T04:19:41.980081 2021-07-31T04:19:43.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:43 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 6539 : cluster [DBG] pgmap v6582: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:43.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:43 smithi045 conmon[33655]: cluster 2021-07-31T04:19:41.980081+0000 mgr.x (mgr.24542 2021-07-31T04:19:43.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:43 smithi045 conmon[33655]: ) 6539 : cluster [DBG] pgmap v6582: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:45.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:45 smithi154 conmon[34553]: cluster 2021-07-31T04:19:43.980653+0000 mgr.x (mgr.24542) 6540 : cluster [DBG] pgmap v6583: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:45.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:45 smithi045 conmon[39718]: cluster 2021-07-31T04:19:43.980653 2021-07-31T04:19:45.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:45 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 6540 : cluster [DBG] pgmap v6583: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:45.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:45 smithi045 conmon[33655]: cluster 2021-07-31T04:19:43.980653+0000 mgr.x (mgr.24542) 6540 : 2021-07-31T04:19:45.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:45 smithi045 conmon[33655]: cluster [DBG] pgmap v6583: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:47.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:47 smithi154 conmon[34553]: cluster 2021-07-31T04:19:45.981025+0000 mgr.x (mgr.24542) 6541 : cluster [DBG] 2021-07-31T04:19:47.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:47 smithi154 conmon[34553]: pgmap v6584: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:47.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:47 smithi045 conmon[33655]: cluster 2021-07-31T04:19:45.981025+0000 mgr.x (mgr.24542) 2021-07-31T04:19:47.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:47 smithi045 conmon[33655]: 6541 : cluster [DBG] pgmap v6584: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:47.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:47 smithi045 conmon[39718]: cluster 2021-07-31T04:19:45.981025+0000 mgr.x (mgr 2021-07-31T04:19:47.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:47 smithi045 conmon[39718]: .24542) 6541 : cluster [DBG] pgmap v6584: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:49.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:49 smithi045 conmon[39718]: cluster 2021-07-31T04:19:47.981409+0000 mgr.x (mgr.24542 2021-07-31T04:19:49.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:49 smithi045 conmon[39718]: ) 6542 : cluster [DBG] pgmap v6585: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:49.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:49 smithi045 conmon[33655]: cluster 2021-07-31T04:19:47.981409+0000 mgr.x (mgr.24542) 6542 : 2021-07-31T04:19:49.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:49 smithi045 conmon[33655]: cluster [DBG] pgmap v6585: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:49.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:49 smithi154 conmon[34553]: cluster 2021-07-31T04:19:47.981409+0000 mgr.x (mgr.24542) 6542 : cluster [DBG] pgmap v6585: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:49.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:49 smithi154 conmon[34553]: 2021-07-31T04:19:51.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:51 smithi045 conmon[39718]: cluster 2021-07-31T04:19:49.981819+0000 mgr.x (mgr.24542) 6543 2021-07-31T04:19:51.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:51 smithi045 conmon[39718]: : cluster [DBG] pgmap v6586: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:51.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:51 smithi045 conmon[33655]: cluster 2021-07-31T04:19:49.981819+0000 2021-07-31T04:19:51.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:51 smithi045 conmon[33655]: mgr.x (mgr.24542) 6543 : cluster [DBG] pgmap v6586: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:51.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:51 smithi154 conmon[34553]: cluster 2021-07-31T04:19:49.981819+0000 mgr.x (mgr.24542) 6543 : cluster [DBG] pgmap v6586: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:53.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:53 smithi045 conmon[39718]: cluster 2021-07-31T04:19:51.982210+0000 mgr.x (mgr.24542 2021-07-31T04:19:53.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:53 smithi045 conmon[39718]: ) 6544 : cluster [DBG] pgmap v6587: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:53.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:53 smithi045 conmon[33655]: cluster 2021-07-31T04:19:51.982210+0000 mgr.x (mgr.24542) 6544 2021-07-31T04:19:53.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:53 smithi045 conmon[33655]: : cluster [DBG] pgmap v6587: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:53.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:53 smithi154 conmon[34553]: cluster 2021-07-31T04:19:51.982210+0000 mgr.x (mgr.24542) 6544 : cluster 2021-07-31T04:19:53.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:53 smithi154 conmon[34553]: [DBG] pgmap v6587: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:55.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:55 smithi045 conmon[39718]: cluster 2021-07-31T04:19:53. 2021-07-31T04:19:55.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:55 smithi045 conmon[39718]: 982776+0000 mgr.x (mgr.24542) 6545 : cluster [DBG] pgmap v6588: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:55.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:55 smithi045 conmon[33655]: cluster 2021-07-31T04:19:53.982776+0000 mgr.x (mgr.24542) 6545 2021-07-31T04:19:55.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:55 smithi045 conmon[33655]: : cluster [DBG] pgmap v6588: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:55.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:55 smithi154 conmon[34553]: cluster 2021-07-31T04:19:53.982776+0000 mgr.x (mgr.24542) 6545 : cluster [DBG] 2021-07-31T04:19:55.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:55 smithi154 conmon[34553]: pgmap v6588: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:57.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:57 smithi045 conmon[33655]: cluster 2021-07-31T04:19:55.983125+0000 mgr.x (mgr.24542) 6546 : 2021-07-31T04:19:57.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:57 smithi045 conmon[33655]: cluster [DBG] pgmap v6589: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:57.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:57 smithi045 conmon[39718]: cluster 2021-07-31T04:19:55.983125+0000 mgr.x (mgr 2021-07-31T04:19:57.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:57 smithi045 conmon[39718]: .24542) 6546 : cluster [DBG] pgmap v6589: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:57.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:57 smithi154 conmon[34553]: cluster 2021-07-31T04:19:55.983125+0000 mgr.x (mgr.24542) 6546 : cluster [DBG] pgmap v6589: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:59.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:19:59 smithi045 conmon[33655]: cluster 2021-07-31T04:19:57.983489+0000 mgr.x (mgr.24542) 6547 : cluster [DBG] pgmap v6590: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:59.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:59 smithi045 conmon[39718]: cluster 2021-07-31T04:19:57.983489+0000 mgr.x (mgr 2021-07-31T04:19:59.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:19:59 smithi045 conmon[39718]: .24542) 6547 : cluster [DBG] pgmap v6590: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:19:59.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:19:59 smithi154 conmon[34553]: cluster 2021-07-31T04:19:57.983489+0000 mgr.x (mgr.24542) 6547 : cluster [DBG] pgmap v6590: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:00.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:00 smithi045 conmon[39718]: cluster 2021-07-31T04:20:00.000146+0000 mon.a (mon 2021-07-31T04:20:00.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:00 smithi045 conmon[39718]: .0) 3460 : cluster [INF] overall HEALTH_OK 2021-07-31T04:20:00.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:00 smithi045 conmon[33655]: cluster 2021-07-31T04:20:00.000146+0000 2021-07-31T04:20:00.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:00 smithi045 conmon[33655]: mon.a (mon.0) 3460 : cluster [INF] overall HEALTH_OK 2021-07-31T04:20:00.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:00 smithi154 conmon[34553]: cluster 2021-07-31T04:20:00.000146+0000 mon.a (mon.0) 3460 : cluster [INF] overall HEALTH_OK 2021-07-31T04:20:01.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:01 smithi154 conmon[34553]: cluster 2021-07-31T04:19:59.983997+0000 mgr.x (mgr.24542) 6548 : cluster [DBG] pgmap v6591: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:01.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:01 smithi045 conmon[33655]: cluster 2021-07-31T04:19:59.983997+0000 2021-07-31T04:20:01.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:01 smithi045 conmon[33655]: mgr.x (mgr.24542) 6548 : cluster [DBG] pgmap v6591: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:01.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:01 smithi045 conmon[39718]: cluster 2021-07-31T04:19:59. 2021-07-31T04:20:01.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:01 smithi045 conmon[39718]: 983997+0000 mgr.x (mgr.24542) 6548 : cluster [DBG] pgmap v6591: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:02.266 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T04:20:02.543 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T04:20:02.543 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 2h ago 2h - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T04:20:02.544 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 2h ago 2h 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T04:20:02.544 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 2h ago 2h 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T04:20:02.544 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 2h ago 2h - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T04:20:02.544 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T04:20:02.544 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 2h ago 2h 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T04:20:02.545 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T04:20:02.545 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 2h ago 2h - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T04:20:02.545 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 2h ago 2h 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T04:20:02.545 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T04:20:02.546 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T04:20:02.546 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T04:20:02.546 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T04:20:02.546 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 2h ago 2h 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T04:20:02.546 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 2h ago 2h 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T04:20:02.547 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 2h ago 2h 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T04:20:02.547 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 2h ago 2h 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T04:20:02.547 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 2h ago 2h 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T04:20:02.876 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T04:20:02.876 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T04:20:02.876 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T04:20:02.877 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:20:02.877 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T04:20:02.877 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T04:20:02.877 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:20:02.877 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:20:02.878 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T04:20:02.878 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T04:20:02.878 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:20:02.878 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T04:20:02.879 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T04:20:02.879 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T04:20:02.879 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:20:02.879 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T04:20:02.879 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T04:20:03.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:03 smithi154 conmon[34553]: cluster 2021-07-31T04:20:01.984493+0000 mgr.x (mgr.24542) 6549 : cluster [DBG] pgmap v6592: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:03.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:03 smithi154 conmon[34553]: 2021-07-31T04:20:03.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:03 smithi154 conmon[34553]: audit 2021-07-31T04:20:02.254705+0000 mgr.x (mgr.24542) 6550 : audit [DBG] from='client.31315 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:20:03.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:03 smithi154 conmon[34553]: audit 2021-07-31T04:20:02.876904+0000 2021-07-31T04:20:03.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:03 smithi154 conmon[34553]: mon.c (mon.1) 266 : audit 2021-07-31T04:20:03.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:03 smithi154 conmon[34553]: [DBG] from='client.? 172.21.15.45:0/1580960653' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:20:03.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:03 smithi045 conmon[33655]: cluster 2021-07-31T04:20:01.984493+0000 mgr.x ( 2021-07-31T04:20:03.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:03 smithi045 conmon[33655]: mgr.24542) 6549 : cluster [DBG] pgmap v6592: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:03.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:03 smithi045 conmon[33655]: audit 2021-07-31T04:20:02.254705+0000 mgr.x (mgr.24542) 6550 : audit [DBG] from='client.31315 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:20:03.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:03 smithi045 conmon[33655]: audit 2021-07-31T04:20:02.876904+0000 mon.c (mon.1) 266 2021-07-31T04:20:03.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:03 smithi045 conmon[33655]: : audit [DBG] from='client.? 172.21.15.45:0/1580960653' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:20:03.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:03 smithi045 conmon[39718]: cluster 2021-07-31T04:20:01.984493+0000 2021-07-31T04:20:03.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:03 smithi045 conmon[39718]: mgr.x (mgr.24542) 6549 : cluster [DBG] pgmap v6592: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:03.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:03 smithi045 conmon[39718]: audit 2021-07-31T04:20:02.254705+0000 2021-07-31T04:20:03.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:03 smithi045 conmon[39718]: mgr.x (mgr.24542) 6550 : audit [DBG] from='client.31315 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:20:03.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:03 smithi045 conmon[39718]: audit 2021 2021-07-31T04:20:03.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:03 smithi045 conmon[39718]: -07-31T04:20:02.876904+0000 mon.c (mon.1) 266 : audit [DBG] 2021-07-31T04:20:03.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:03 smithi045 conmon[39718]: from='client.? 172.21.15.45:0/1580960653' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:20:04.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:04 smithi154 conmon[34553]: audit 2021-07-31T04:20:02.535765+0000 mgr.x (mgr.24542) 6551 : audit [DBG] from='client.31321 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:20:04.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:04 smithi045 conmon[39718]: audit 2021-07-31T04:20:02.535765+0000 2021-07-31T04:20:04.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:04 smithi045 conmon[39718]: mgr.x (mgr.24542) 6551 : audit [DBG] from='client.31321 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:20:04.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:04 smithi045 conmon[33655]: audit 2021-07-31T04:20:02.535765+0000 mgr.x (mgr.24542) 6551 : audit [DBG] 2021-07-31T04:20:04.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:04 smithi045 conmon[33655]: from='client.31321 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:20:05.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:05 smithi154 conmon[34553]: cluster 2021-07-31T04:20:03.984872+0000 mgr.x (mgr.24542) 6552 : cluster [DBG] pgmap v6593: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:05.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:05 smithi045 conmon[39718]: cluster 2021-07-31T04:20:03.984872+0000 2021-07-31T04:20:05.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:05 smithi045 conmon[39718]: mgr.x (mgr.24542) 6552 : cluster [DBG] pgmap v6593: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:05.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:05 smithi045 conmon[33655]: cluster 2021-07-31T04:20:03 2021-07-31T04:20:05.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:05 smithi045 conmon[33655]: .984872+0000 mgr.x (mgr.24542) 6552 : cluster [DBG] pgmap v6593: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:07.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:07 smithi154 conmon[34553]: cluster 2021-07-31T04:20:05.985309+0000 mgr.x (mgr.24542) 6553 : cluster [DBG] pgmap v6594: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:07.774 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:07 smithi045 conmon[33655]: cluster 2021-07-31T04:20:05.985309+0000 mgr.x (mgr.24542) 6553 : cluster [DBG] pgmap v6594: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:07.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:07 smithi045 conmon[39718]: cluster 2021-07-31T04:20:05.985309+0000 mgr.x ( 2021-07-31T04:20:07.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:07 smithi045 conmon[39718]: mgr.24542) 6553 : cluster [DBG] pgmap v6594: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:09.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:09 smithi154 conmon[34553]: cluster 2021-07-31T04:20:07.985795+0000 mgr.x (mgr.24542) 6554 : cluster [DBG] pgmap v6595: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:09.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:09 smithi045 conmon[39718]: cluster 2021-07-31T04:20:07.985795+0000 mgr.x (mgr. 2021-07-31T04:20:09.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:09 smithi045 conmon[39718]: 24542) 6554 : cluster [DBG] pgmap v6595: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:09.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:09 smithi045 conmon[33655]: cluster 2021-07-31T04:20:07.985795+0000 mgr.x (mgr.24542) 6554 : cluster [DBG] pgmap v6595: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:11.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:11 smithi154 conmon[34553]: cluster 2021-07-31T04:20:09.986362+0000 mgr.x (mgr.24542) 6555 : cluster [DBG] pgmap v6596: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:11.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:11 smithi045 conmon[39718]: cluster 2021-07-31T04:20:09.986362+0000 mgr.x (mgr.24542 2021-07-31T04:20:11.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:11 smithi045 conmon[39718]: ) 6555 : cluster [DBG] pgmap v6596: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:11.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:11 smithi045 conmon[33655]: cluster 2021-07-31T04:20:09.986362+0000 mgr.x (mgr.24542 2021-07-31T04:20:11.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:11 smithi045 conmon[33655]: ) 6555 : cluster [DBG] pgmap v6596: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:13.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:13 smithi154 conmon[34553]: cluster 2021-07-31T04:20:11.986741+0000 mgr.x (mgr.24542) 6556 : cluster [DBG] pgmap v6597: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:13.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:13 smithi045 conmon[39718]: cluster 2021-07-31T04:20:11.986741+0000 mgr.x (mgr 2021-07-31T04:20:13.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:13 smithi045 conmon[39718]: .24542) 6556 : cluster [DBG] pgmap v6597: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:13.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:13 smithi045 conmon[33655]: cluster 2021-07-31T 2021-07-31T04:20:13.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:13 smithi045 conmon[33655]: 04:20:11.986741+0000 mgr.x (mgr.24542) 6556 : cluster [DBG] pgmap v6597: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:15.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:15 smithi154 conmon[34553]: cluster 2021-07-31T04:20:13.987275+0000 mgr.x (mgr.24542) 6557 : cluster [DBG] pgmap v6598: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:15.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:15 smithi045 conmon[39718]: cluster 2021-07-31T04:20:13.987275+0000 mgr.x 2021-07-31T04:20:15.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:15 smithi045 conmon[39718]: (mgr.24542) 6557 : cluster [DBG] pgmap v6598: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:15.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:15 smithi045 conmon[33655]: cluster 2021-07-31T04:20:13.987275+0000 mgr.x (mgr.24542) 6557 : cluster 2021-07-31T04:20:15.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:15 smithi045 conmon[33655]: [DBG] pgmap v6598: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:17.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:17 smithi154 conmon[34553]: cluster 2021-07-31T04:20:15.987763+0000 mgr.x (mgr.24542) 6558 : cluster [DBG] pgmap v6599: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:17.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:17 smithi045 conmon[33655]: cluster 2021-07-31T04:20:15.987763+0000 mgr.x (mgr.24542) 6558 : cluster [DBG] pgmap v6599: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:17.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:17 smithi045 conmon[39718]: cluster 2021-07-31T04:20:15.987763+0000 mgr.x (mgr.24542) 6558 : 2021-07-31T04:20:17.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:17 smithi045 conmon[39718]: cluster [DBG] pgmap v6599: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:19.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:19 smithi154 conmon[34553]: cluster 2021-07-31T04:20:17.988236+0000 mgr.x (mgr.24542) 6559 : cluster [DBG] pgmap v6600: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:19.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:19 smithi045 conmon[33655]: cluster 2021-07-31T04:20:17.988236+0000 mgr.x (mgr.24542) 6559 : cluster [DBG] pgmap v6600: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:19.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:19 smithi045 conmon[39718]: cluster 2021-07-31T04:20:17.988236+0000 mgr.x (mgr.24542) 6559 2021-07-31T04:20:19.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:19 smithi045 conmon[39718]: : cluster [DBG] pgmap v6600: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:21.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:21 smithi154 conmon[34553]: cluster 2021-07-31T04:20:19.988757+0000 mgr.x (mgr.24542) 6560 : cluster [DBG] pgmap v6601: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:21.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:21 smithi045 conmon[33655]: cluster 2021-07-31T04:20:19.988757+0000 mgr.x (mgr.24542) 6560 : cluster [DBG] pgmap v6601: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:21.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:21 smithi045 conmon[39718]: cluster 2021-07-31T04:20:19.988757+0000 mgr.x (mgr 2021-07-31T04:20:21.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:21 smithi045 conmon[39718]: .24542) 6560 : cluster [DBG] pgmap v6601: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:23.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:23 smithi154 conmon[34553]: cluster 2021-07-31T04:20:21.989340+0000 mgr.x (mgr.24542) 6561 : cluster [DBG] pgmap v6602: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:23.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:23 smithi154 conmon[34553]: 2021-07-31T04:20:23.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:23 smithi045 conmon[39718]: cluster 2021-07-31T04:20:21. 2021-07-31T04:20:23.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:23 smithi045 conmon[39718]: 989340+0000 mgr.x (mgr.24542) 6561 : cluster [DBG] pgmap v6602: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:23.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:23 smithi045 conmon[33655]: cluster 2021-07-31T04:20:21.989340+0000 2021-07-31T04:20:23.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:23 smithi045 conmon[33655]: mgr.x (mgr.24542) 6561 : cluster [DBG] pgmap v6602: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:25.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:25 smithi154 conmon[34553]: cluster 2021-07-31T04:20:23.989865+0000 mgr.x (mgr.24542) 6562 : cluster [DBG] pgmap v6603: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:25.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:25 smithi045 conmon[33655]: cluster 2021-07-31T04:20:23.989865+0000 2021-07-31T04:20:25.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:25 smithi045 conmon[33655]: mgr.x (mgr.24542) 6562 : cluster [DBG] pgmap v6603: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:25.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:25 smithi045 conmon[39718]: cluster 2021-07-31T04:20:23.989865+0000 mgr.x ( 2021-07-31T04:20:25.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:25 smithi045 conmon[39718]: mgr.24542) 6562 : cluster [DBG] pgmap v6603: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:27.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:27 smithi154 conmon[34553]: cluster 2021-07-31T04:20:25.990340+0000 mgr.x (mgr.24542) 6563 : cluster 2021-07-31T04:20:27.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:27 smithi154 conmon[34553]: [DBG] pgmap v6604: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:27.774 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:27 smithi045 conmon[33655]: cluster 2021-07-31T04:20:25.990340+0000 mgr.x (mgr. 2021-07-31T04:20:27.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:27 smithi045 conmon[33655]: 24542) 6563 : cluster [DBG] pgmap v6604: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:27.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:27 smithi045 conmon[39718]: cluster 2021-07-31T04:20:25.990340+0000 mgr.x ( 2021-07-31T04:20:27.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:27 smithi045 conmon[39718]: mgr.24542) 6563 : cluster [DBG] pgmap v6604: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:29.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:29 smithi154 conmon[34553]: cluster 2021-07-31T04:20:27.990707+0000 mgr.x (mgr.24542) 6564 : cluster [DBG] pgmap v6605: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:29.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:29 smithi154 conmon[34553]: audit 2021-07-31T04:20:28.538550+0000 mon.b (mon.2) 840 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:20:29.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:29 smithi154 conmon[34553]: audit 2021- 2021-07-31T04:20:29.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:29 smithi154 conmon[34553]: 07-31T04:20:28.538965+0000 mon.b (mon 2021-07-31T04:20:29.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:29 smithi154 conmon[34553]: .2) 841 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:20:29.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:29 smithi154 conmon[34553]: audit 2021-07-31 2021-07-31T04:20:29.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:29 smithi154 conmon[34553]: T04:20:28.539458+0000 mon.a (mon.0) 3461 2021-07-31T04:20:29.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:29 smithi154 conmon[34553]: : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:20:29.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:29 smithi154 conmon[34553]: audit 2021-07 2021-07-31T04:20:29.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:29 smithi154 conmon[34553]: -31T04:20:28.539894+0000 mon.a 2021-07-31T04:20:29.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:29 smithi154 conmon[34553]: (mon.0) 3462 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:20:29.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:29 smithi045 conmon[33655]: cluster 2021-07-31T04:20:27.990707+0000 mgr.x (mgr.24542) 2021-07-31T04:20:29.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:29 smithi045 conmon[33655]: 6564 : cluster [DBG] pgmap v6605: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:29.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:29 smithi045 conmon[33655]: audit 2021-07-31T04:20:28.538550+0000 mon.b (mon.2) 840 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:20:29.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:29 smithi045 conmon[33655]: audit 2021-07-31T04:20:28.538965+0000 mon.b (mon.2) 841 : 2021-07-31T04:20:29.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:29 smithi045 conmon[33655]: audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:20:29.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:29 smithi045 conmon[33655]: audit 2021-07-31T04:20:28.539458+0000 mon.a (mon.0) 3461 : 2021-07-31T04:20:29.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:29 smithi045 conmon[33655]: audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:20:29.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:29 smithi045 conmon[33655]: audit 2021-07-31T04:20:29.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:29 smithi045 conmon[33655]: 2021-07-31T04:20:28.539894+0000 mon.a (mon.0) 3462 2021-07-31T04:20:29.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:29 smithi045 conmon[33655]: : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:20:29.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:29 smithi045 conmon[39718]: cluster 2021-07-31T04:20:27.990707+0000 mgr.x (mgr.24542) 6564 : cluster [DBG] pgmap v6605: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:29.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:29 smithi045 conmon[39718]: audit 2021-07-31T04:20: 2021-07-31T04:20:29.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:29 smithi045 conmon[39718]: 28.538550+0000 mon.b (mon.2 2021-07-31T04:20:29.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:29 smithi045 conmon[39718]: ) 840 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:20:29.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:29 smithi045 conmon[39718]: 2021-07-31T04:20:29.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:29 smithi045 conmon[39718]: audit 2021-07-31 2021-07-31T04:20:29.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:29 smithi045 conmon[39718]: T04:20:28.538965+0000 2021-07-31T04:20:29.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:29 smithi045 conmon[39718]: mon.b (mon.2) 841 : audit 2021-07-31T04:20:29.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:29 smithi045 conmon[39718]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:20:29.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:29 smithi045 conmon[39718]: audit 2021 2021-07-31T04:20:29.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:29 smithi045 conmon[39718]: -07-31T04:20:28 2021-07-31T04:20:29.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:29 smithi045 conmon[39718]: .539458+0000 mon.a (mon. 2021-07-31T04:20:29.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:29 smithi045 conmon[39718]: 0) 3461 : audit [INF] 2021-07-31T04:20:29.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:29 smithi045 conmon[39718]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:20:29.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:29 smithi045 conmon[39718]: audit 2021 2021-07-31T04:20:29.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:29 smithi045 conmon[39718]: -07-31T04:20:28 2021-07-31T04:20:29.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:29 smithi045 conmon[39718]: .539894+0000 mon.a (mon. 2021-07-31T04:20:29.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:29 smithi045 conmon[39718]: 0) 3462 : audit [INF] 2021-07-31T04:20:29.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:29 smithi045 conmon[39718]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:20:31.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:31 smithi154 conmon[34553]: cluster 2021-07-31T04:20:29.991294+0000 mgr.x (mgr.24542) 6565 : cluster 2021-07-31T04:20:31.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:31 smithi154 conmon[34553]: [DBG] pgmap v6606: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:31.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:31 smithi045 conmon[39718]: cluster 2021-07-31T04:20:29.991294+0000 mgr.x (mgr.24542) 2021-07-31T04:20:31.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:31 smithi045 conmon[39718]: 6565 : cluster [DBG] pgmap v6606: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:31.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:31 smithi045 conmon[33655]: cluster 2021-07-31T04:20:29.991294+0000 mgr.x (mgr.24542) 6565 : cluster [DBG] pgmap v6606: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:33.189 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T04:20:33.467 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T04:20:33.467 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 2h ago 2h - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T04:20:33.467 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 2h ago 2h 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T04:20:33.467 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 2h ago 2h 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T04:20:33.468 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 2h ago 2h - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T04:20:33.468 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T04:20:33.468 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 2h ago 2h 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T04:20:33.468 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T04:20:33.469 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 2h ago 2h - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T04:20:33.469 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 2h ago 2h 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T04:20:33.469 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T04:20:33.469 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T04:20:33.469 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T04:20:33.470 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T04:20:33.470 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 2h ago 2h 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T04:20:33.470 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 2h ago 2h 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T04:20:33.470 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 2h ago 2h 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T04:20:33.471 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 2h ago 2h 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T04:20:33.471 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 2h ago 2h 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T04:20:33.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:33 smithi154 conmon[34553]: cluster 2021-07-31T04:20:31.991760+0000 mgr.x (mgr.24542) 6566 : cluster [DBG] pgmap v6607: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:33.774 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:33 smithi045 conmon[39718]: cluster 2021-07-31T04:20:31.991760+0000 2021-07-31T04:20:33.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:33 smithi045 conmon[39718]: mgr.x (mgr.24542) 6566 : cluster [DBG] pgmap v6607: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:33.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:33 smithi045 conmon[33655]: cluster 2021-07-31T04:20:31.991760+0000 mgr.x 2021-07-31T04:20:33.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:33 smithi045 conmon[33655]: (mgr.24542) 6566 : cluster [DBG] pgmap v6607: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:33.803 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T04:20:33.803 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T04:20:33.804 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T04:20:33.804 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:20:33.804 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T04:20:33.804 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T04:20:33.804 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:20:33.805 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:20:33.805 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T04:20:33.805 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T04:20:33.805 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:20:33.805 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T04:20:33.806 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T04:20:33.806 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T04:20:33.806 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:20:33.806 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T04:20:33.807 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T04:20:34.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:34 smithi154 conmon[34553]: audit 2021-07-31T04:20:33.177444+0000 mgr.x (mgr.24542) 6567 : audit [DBG] from='client.21396 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:20:34.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:34 smithi154 conmon[34553]: audit 2021-07-31T04:20:33.459393+0000 mgr.x (mgr.24542) 6568 : audit [DBG] from='client.21402 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:20:34.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:34 smithi154 conmon[34553]: audit 2021-07-31T04:20:33.804534+0000 mon.c (mon.1) 267 : audit [DBG] from='client.? 172.21.15.45:0/3975030227' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:20:34.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:34 smithi045 conmon[33655]: audit 2021-07-31T04:20:33.177444+0000 mgr.x 2021-07-31T04:20:35.260 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:34 smithi045 conmon[33655]: (mgr.24542) 6567 : audit [DBG] from='client.21396 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:20:35.260 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:34 smithi045 conmon[33655]: audit 2021-07-31T04:20:33.459393+0000 mgr.x (mgr.24542) 6568 : audit [DBG] from='client.21402 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:20:35.260 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:34 smithi045 conmon[33655]: audit 2021-07-31T04:20: 2021-07-31T04:20:35.261 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:34 smithi045 conmon[33655]: 33.804534+0000 mon.c (mon. 2021-07-31T04:20:35.261 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:34 smithi045 conmon[33655]: 1) 267 : audit [DBG] from='client.? 172.21.15.45:0/3975030227' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:20:35.261 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:34 smithi045 conmon[39718]: audit 2021-07-31T04:20 2021-07-31T04:20:35.261 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:34 smithi045 conmon[39718]: :33.177444+0000 mgr.x (mgr.24542) 6567 : audit [DBG] from='client.21396 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:20:35.262 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:34 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T04:20:35.262 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:34 smithi045 conmon[39718]: 31T04:20:33.459393 2021-07-31T04:20:35.262 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:34 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 2021-07-31T04:20:35.262 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:34 smithi045 conmon[39718]: 6568 : audit [DBG] from='client.21402 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:20:35.263 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:34 smithi045 conmon[39718]: audit 2021- 2021-07-31T04:20:35.263 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:34 smithi045 conmon[39718]: 07-31T04:20:33.804534 2021-07-31T04:20:35.263 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:34 smithi045 conmon[39718]: +0000 mon.c (mon.1) 2021-07-31T04:20:35.263 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:34 smithi045 conmon[39718]: 267 : audit [DBG] from='client.? 172.21.15.45:0/3975030227' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:20:35.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:35 smithi154 conmon[34553]: cluster 2021-07-31T04:20:33.992324+0000 mgr.x (mgr.24542) 6569 : cluster 2021-07-31T04:20:35.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:35 smithi154 conmon[34553]: [DBG] pgmap v6608: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:35.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:35 smithi045 conmon[33655]: cluster 2021-07-31T04: 2021-07-31T04:20:35.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:35 smithi045 conmon[33655]: 20:33.992324+0000 mgr.x (mgr.24542) 6569 : cluster [DBG] pgmap v6608: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:35.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:35 smithi045 conmon[39718]: cluster 2021-07-31T04:20:33.992324+0000 mgr.x (mgr 2021-07-31T04:20:35.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:35 smithi045 conmon[39718]: .24542) 6569 : cluster [DBG] pgmap v6608: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:37.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:37 smithi154 conmon[34553]: cluster 2021-07-31T04:20:35.992652+0000 mgr.x (mgr.24542) 6570 : cluster [DBG] pgmap v6609: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:37.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:37 smithi045 conmon[33655]: cluster 2021-07-31T04:20:35.992652+0000 2021-07-31T04:20:37.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:37 smithi045 conmon[33655]: mgr.x (mgr.24542) 6570 : cluster [DBG] pgmap v6609: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:37.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:37 smithi045 conmon[39718]: cluster 2021-07-31T04:20:35.992652+0000 2021-07-31T04:20:37.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:37 smithi045 conmon[39718]: mgr.x (mgr.24542) 6570 : cluster [DBG] pgmap v6609: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:38.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:38 smithi045 conmon[33655]: audit 2021-07-31T04:20:38.227456+0000 mon.b 2021-07-31T04:20:38.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:38 smithi045 conmon[33655]: (mon.2) 842 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T04:20:38.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:38 smithi045 conmon[33655]: audit 2021-07-31T04:20:38 2021-07-31T04:20:38.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:38 smithi045 conmon[33655]: .228954+0000 mon.b (mon.2) 843 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T04:20:38.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:38 smithi045 conmon[33655]: 2021-07-31T04:20:38.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:38 smithi045 conmon[33655]: audit 2021- 2021-07-31T04:20:38.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:38 smithi045 conmon[33655]: 07-31T04:20:38.229807+0000 2021-07-31T04:20:38.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:38 smithi045 conmon[33655]: mon.b (mon.2) 844 : audit [INF] 2021-07-31T04:20:38.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:38 smithi045 conmon[33655]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T04:20:38.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:38 smithi045 conmon[39718]: audit 2021-07-31T04:20:38.227456 2021-07-31T04:20:38.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:38 smithi045 conmon[39718]: +0000 mon.b (mon.2) 842 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T04:20:38.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:38 smithi045 conmon[39718]: audit 2021-07-31T04:20:38.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:38 smithi045 conmon[39718]: 2021-07-31T04:20:38.228954+0000 mon.b (mon.2 2021-07-31T04:20:38.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:38 smithi045 conmon[39718]: ) 843 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T04:20:38.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:38 smithi045 conmon[39718]: audit 2021- 2021-07-31T04:20:38.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:38 smithi045 conmon[39718]: 07-31T04:20:38.229807 2021-07-31T04:20:38.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:38 smithi045 conmon[39718]: +0000 mon.b (mon.2) 844 2021-07-31T04:20:38.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:38 smithi045 conmon[39718]: : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T04:20:38.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[34553]: audit 2021-07-31T04:20:38.227456+0000 mon.b (mon.2) 842 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T04:20:38.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[34553]: audit 2021-07-31 2021-07-31T04:20:38.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[34553]: T04:20:38.228954+0000 mon.b (mon.2) 843 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T04:20:38.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[34553]: audit 2021-07-31T04:20:38.229807+0000 mon.b (mon.2) 844 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T04:20:38.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: debug 2021-07-31T04:20:38.273+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:20:38.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:20:38.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:20:38.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: yield 2021-07-31T04:20:38.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:20:38.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:20:38.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:20:38.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:20:38.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:20:38.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:20:38.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:20:38.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:20:38.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:20:38.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: task.result() 2021-07-31T04:20:38.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:20:38.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: debug 2021-07-31T04:20:38.285+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:20:38.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:20:38.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:20:38.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: yield 2021-07-31T04:20:38.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:20:38.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:20:38.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:20:38.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:20:38.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:20:38.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:20:38.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:20:38.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:20:38.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:20:38.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: task.result() 2021-07-31T04:20:38.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:20:38.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: debug 2021-07-31T04:20:38.311+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:20:38.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:20:38.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:20:38.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: yield 2021-07-31T04:20:38.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:20:38.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:20:38.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:20:38.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:20:38.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:20:38.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:20:38.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:20:38.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:20:38.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:20:38.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: task.result() 2021-07-31T04:20:38.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:20:38.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: debug 2021-07-31T04:20:38.311+0000 7fcb9115c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:20:38.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:20:38.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:20:38.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: yield 2021-07-31T04:20:38.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:20:38.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:20:38.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:20:38.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:20:38.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:20:38.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:20:38.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:20:38.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:20:38.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:20:38.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: task.result() 2021-07-31T04:20:38.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:20:38.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: 2021-07-31T04:20:38.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T04:20:38.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: 2021-07-31T04:20:38.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:20:38.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:20:38.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: return f(*arg) 2021-07-31T04:20:38.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:20:38.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:20:38.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:20:38.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:20:38.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:20:38.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:20:38.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:20:38.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T04:20:38.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:20:38.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: raise self._exception 2021-07-31T04:20:38.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:20:38.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:20:38.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:20:38.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T04:20:38.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:20:38.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: raise 2021-07-31T04:20:38.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:20:38.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T04:20:38.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:20:38.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:20:38.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:20:38.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: debug 2021-07-31T04:20:38.320+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:20:38.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:20:38.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:20:38.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: yield 2021-07-31T04:20:38.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:20:38.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:20:38.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:20:38.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:20:38.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:20:38.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:20:38.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:20:38.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:20:38.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:20:38.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: task.result() 2021-07-31T04:20:38.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:20:38.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: debug 2021-07-31T04:20:38.320+0000 7fcb9095b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:20:38.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:20:38.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:20:38.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: yield 2021-07-31T04:20:38.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:20:38.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:20:38.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:20:38.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:20:38.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:20:38.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:20:38.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:20:38.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:20:38.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:20:38.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: task.result() 2021-07-31T04:20:38.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:20:38.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: 2021-07-31T04:20:38.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T04:20:38.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: 2021-07-31T04:20:38.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:20:38.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:20:38.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: return f(*arg) 2021-07-31T04:20:38.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:20:38.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:20:38.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:20:38.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:20:38.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:20:38.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:20:38.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:20:38.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T04:20:38.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:20:38.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: raise self._exception 2021-07-31T04:20:38.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:20:38.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:20:38.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:20:38.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T04:20:38.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:20:38.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: raise 2021-07-31T04:20:38.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:20:38.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T04:20:38.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:20:38.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:20:38.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:20:38 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:20:39.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: cluster 2021-07-31T04:20:37.993161+0000 mgr.x (mgr.24542) 6571 : cluster [DBG] pgmap v6610: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:39.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:20:38.230607+0000 mgr.x (mgr.24542) 6572 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T04:20:39.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:20:38.274634+0000 mgr.x (mgr.24542) 6573 : cephadm [ERR] 2021-07-31T04:20:39.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:20:39.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:20:39.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: yield 2021-07-31T04:20:39.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:20:39.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:20:39.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:20:39.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:20:39.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:20:39.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:20:39.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:20:39.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:20:39.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:20:39.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: task.result() 2021-07-31T04:20:39.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:20:39.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:20:38.275105+0000 mgr.x (mgr.24542) 6574 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T04:20:39.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:20:38.286326+0000 mgr.x (mgr.24542) 6575 : cephadm [ERR] 2021-07-31T04:20:39.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:20:39.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:20:39.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: yield 2021-07-31T04:20:39.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:20:39.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:20:39.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:20:39.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:20:39.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:20:39.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:20:39.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:20:39.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:20:39.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:20:39.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: task.result() 2021-07-31T04:20:39.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:20:39.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:20:38.287208+0000 mgr.x (mgr.24542) 6576 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T04:20:39.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:20:38.312289+0000 mgr.x (mgr.24542) 6577 2021-07-31T04:20:39.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: : cephadm [ERR] 2021-07-31T04:20:39.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:20:39.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:20:39.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: yield 2021-07-31T04:20:39.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:20:39.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:20:39.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:20:39.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:20:39.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:20:39.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:20:39.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:20:39.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:20:39.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:20:39.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: task.result() 2021-07-31T04:20:39.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:20:39.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:20:38.312671+0000 mgr.x (mgr.24542) 6578 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:20:39.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:20:39.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:20:39.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: yield 2021-07-31T04:20:39.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:20:39.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:20:39.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:20:39.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:20:39.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:20:39.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:20:39.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:20:39.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:20:39.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:20:39.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: task.result() 2021-07-31T04:20:39.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:20:39.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: 2021-07-31T04:20:39.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T04:20:39.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: 2021-07-31T04:20:39.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:20:39.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:20:39.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: return f(*arg) 2021-07-31T04:20:39.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:20:39.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:20:39.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:20:39.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:20:39.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:20:39.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:20:39.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:20:39.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T04:20:39.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:20:39.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: raise self._exception 2021-07-31T04:20:39.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:20:39.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:20:39.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:20:39.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T04:20:39.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:20:39.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: raise 2021-07-31T04:20:39.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:20:39.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T04:20:39.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:20:39.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:20:39.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:20:39.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:20:39.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: 2021-07-31T04:20:38.321275+0000 mgr.x (mgr.24542) 6579 : cephadm [ERR] 2021-07-31T04:20:39.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:20:39.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:20:39.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: yield 2021-07-31T04:20:39.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:20:39.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:20:39.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:20:39.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:20:39.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:20:39.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:20:39.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:20:39.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:20:39.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:20:39.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: task.result() 2021-07-31T04:20:39.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:20:39.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:20:38.321692+0000 mgr.x (mgr.24542) 6580 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:20:39.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:20:39.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:20:39.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: yield 2021-07-31T04:20:39.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:20:39.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:20:39.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:20:39.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:20:39.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: cluster 2021-07-31T04:20:37.993161+0000 mgr.x (mgr.24542) 6571 : cluster 2021-07-31T04:20:39.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: [DBG] pgmap v6610: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:39.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: cephadm 2021-07-31T04:20:38.230607+0000 mgr.x (mgr.24542) 6572 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T04:20:39.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: cephadm 2021-07-31 2021-07-31T04:20:39.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: T04:20:38.274634+0000 mgr.x (mgr.24542) 6573 : cephadm [ERR] 2021-07-31T04:20:39.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:20:39.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:20:39.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: yield 2021-07-31T04:20:39.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:20:39.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:20:39.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:20:39.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:20:39.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:20:39.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:20:39.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:20:39.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:20:39.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:20:39.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: task.result() 2021-07-31T04:20:39.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:20:39.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: cephadm 2021-07-31T04:20:38.275105+0000 mgr.x (mgr.24542) 6574 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T04:20:39.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: cephadm 2021-07-31T04:20:38.286326+0000 mgr.x (mgr. 2021-07-31T04:20:39.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: 24542) 6575 : cephadm [ERR] 2021-07-31T04:20:39.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:20:39.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:20:39.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: yield 2021-07-31T04:20:39.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:20:39.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:20:39.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:20:39.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:20:39.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:20:39.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:20:39.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:20:39.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:20:39.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:20:39.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: task.result() 2021-07-31T04:20:39.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:20:39.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: cephadm 2021-07-31T04:20:38.287208+0000 mgr.x (mgr.24542) 6576 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T04:20:39.787 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:20:39.788 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:20:39.788 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:20:39.788 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:20:39.788 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:20:39.788 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: task.result() 2021-07-31T04:20:39.789 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:20:39.789 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: 2021-07-31T04:20:39.789 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T04:20:39.789 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: 2021-07-31T04:20:39.790 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:20:39.790 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:20:39.790 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: return f(*arg) 2021-07-31T04:20:39.790 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:20:39.790 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:20:39.791 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:20:39.791 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:20:39.791 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:20:39.791 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:20:39.792 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:20:39.792 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T04:20:39.792 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:20:39.792 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: raise self._exception 2021-07-31T04:20:39.792 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:20:39.793 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:20:39.793 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:20:39.793 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T04:20:39.793 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:20:39.793 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: raise 2021-07-31T04:20:39.794 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:20:39.794 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T04:20:39.794 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:20:39.794 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:20:39.795 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:39 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:20:39.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: cluster 2021-07-31T04:20:37.993161+0000 mgr.x (mgr 2021-07-31T04:20:39.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: .24542) 6571 : cluster [DBG] pgmap v6610: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:39.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: cephadm 2021-07-31T04:20:38.230607+0000 mgr.x (mgr. 2021-07-31T04:20:39.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: 24542) 6572 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T04:20:39.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: cephadm 2021-07-31T04:20:38. 2021-07-31T04:20:39.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: 274634+0000 mgr.x (mgr.24542) 6573 : cephadm 2021-07-31T04:20:39.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: [ERR] 2021-07-31T04:20:39.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:20:39.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:20:39.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: yield 2021-07-31T04:20:39.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:20:39.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:20:39.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:20:39.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:20:39.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:20:39.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:20:39.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:20:39.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:20:39.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:20:39.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: task.result() 2021-07-31T04:20:39.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:20:39.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: cephadm 2021-07-31T04:20:38.275105+0000 mgr.x (mgr.24542) 6574 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T04:20:39.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: cephadm 2021-07-31T04:20 2021-07-31T04:20:39.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: :38.286326+0000 mgr.x (mgr.24542) 6575 : cephadm 2021-07-31T04:20:39.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: [ERR] 2021-07-31T04:20:39.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:20:39.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:20:39.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: yield 2021-07-31T04:20:39.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:20:39.802 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:20:39.802 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:20:39.802 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:20:39.802 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:20:39.802 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:20:39.803 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:20:39.803 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:20:39.803 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:20:39.803 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: task.result() 2021-07-31T04:20:39.804 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:20:39.804 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: cephadm 2021-07-31T04:20:38.312289+0000 mgr.x (mgr.24542) 6577 : cephadm [ERR] 2021-07-31T04:20:39.804 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:20:39.805 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:20:39.805 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: yield 2021-07-31T04:20:39.805 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:20:39.805 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:20:39.805 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:20:39.806 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:20:39.806 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:20:39.806 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:20:39.806 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:20:39.807 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:20:39.807 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:20:39.807 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: task.result() 2021-07-31T04:20:39.808 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:20:39.808 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: cephadm 2021-07-31T04:20:38.312671+0000 mgr.x (mgr.24542) 6578 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:20:39.808 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:20:39.808 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:20:39.808 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: yield 2021-07-31T04:20:39.809 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:20:39.809 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:20:39.809 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:20:39.809 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:20:39.809 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:20:39.810 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:20:39.810 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:20:39.810 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:20:39.810 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:20:39.810 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: task.result() 2021-07-31T04:20:39.811 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:20:39.811 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: 2021-07-31T04:20:39.811 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T04:20:39.811 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: 2021-07-31T04:20:39.811 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:20:39.812 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:20:39.812 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: return f(*arg) 2021-07-31T04:20:39.812 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:20:39.812 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:20:39.812 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:20:39.813 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:20:39.813 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:20:39.813 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: cephadm 2021-07-31T04:20:38.287208+0000 mgr.x (mgr.24542) 6576 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T04:20:39.814 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: cephadm 2021-07-31T04:20:38.312289+0000 mgr.x (mgr.24542) 6577 : cephadm [ERR] 2021-07-31T04:20:39.814 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:20:39.814 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:20:39.814 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: yield 2021-07-31T04:20:39.814 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:20:39.815 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:20:39.815 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:20:39.815 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:20:39.815 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:20:39.815 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:20:39.816 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:20:39.816 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:20:39.816 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:20:39.816 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: task.result() 2021-07-31T04:20:39.816 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:20:39.817 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: cephadm 2021-07-31T04:20:38.312671+0000 mgr.x (mgr.24542) 6578 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:20:39.817 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:20:39.817 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:20:39.817 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: yield 2021-07-31T04:20:39.817 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:20:39.818 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:20:39.818 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:20:39.818 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:20:39.818 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:20:39.818 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:20:39.819 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:20:39.819 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:20:39.819 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:20:39.819 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: task.result() 2021-07-31T04:20:39.819 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:20:39.820 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: 2021-07-31T04:20:39.820 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T04:20:39.820 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: 2021-07-31T04:20:39.820 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:20:39.820 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:20:39.821 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: return f(*arg) 2021-07-31T04:20:39.821 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:20:39.821 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:20:39.821 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:20:39.821 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:20:39.822 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:20:39.822 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:20:39.822 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T04:20:39.823 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:20:39.823 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: raise self._exception 2021-07-31T04:20:39.823 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:20:39.823 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:20:39.823 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:20:39.824 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T04:20:39.824 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:20:39.824 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: raise 2021-07-31T04:20:39.824 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:20:39.824 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T04:20:39.825 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:20:39.825 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:20:39.825 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:20:39.825 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: cephadm 2021-07-31T04:20:38.321275+0000 mgr.x (mgr.24542) 6579 : cephadm [ERR] 2021-07-31T04:20:39.825 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:20:39.826 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:20:39.826 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: yield 2021-07-31T04:20:39.826 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:20:39.826 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:20:39.826 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:20:39.827 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:20:39.827 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:20:39.827 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:20:39.827 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:20:39.827 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:20:39.828 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:20:39.828 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: task.result() 2021-07-31T04:20:39.828 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:20:39.828 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: cephadm 2021-07-31T04 2021-07-31T04:20:39.828 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: :20:38.321692+0000 mgr.x (mgr.24542) 6580 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:20:39.828 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:20:39.829 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:20:39.829 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: yield 2021-07-31T04:20:39.829 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:20:39.829 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:20:39.829 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:20:39.830 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:20:39.830 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:20:39.830 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:20:39.830 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:20:39.831 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T04:20:39.831 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:20:39.831 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: raise self._exception 2021-07-31T04:20:39.831 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:20:39.831 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:20:39.832 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:20:39.832 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T04:20:39.832 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:20:39.832 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: raise 2021-07-31T04:20:39.832 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:20:39.833 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T04:20:39.833 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:20:39.833 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:20:39.833 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:20:39.833 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: cephadm 2021-07 2021-07-31T04:20:39.834 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: -31T04:20:38.321275+0000 mgr.x (mgr.24542) 6579 : cephadm [ERR] 2021-07-31T04:20:39.834 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:20:39.834 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:20:39.834 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: yield 2021-07-31T04:20:39.834 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:20:39.835 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:20:39.835 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:20:39.835 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:20:39.835 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:20:39.836 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:20:39.836 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:20:39.836 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:20:39.836 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:20:39.836 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: task.result() 2021-07-31T04:20:39.837 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:20:39.837 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: cephadm 2021-07-31T04:20:38.321692+0000 mgr.x (mgr.24542) 6580 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:20:39.837 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:20:39.837 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:20:39.837 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: yield 2021-07-31T04:20:39.838 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:20:39.838 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:20:39.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:20:39.838 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:20:39.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:20:39.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:20:39.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:20:39.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: task.result() 2021-07-31T04:20:39.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:20:39.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: 2021-07-31T04:20:39.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T04:20:39.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: 2021-07-31T04:20:39.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:20:39.841 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:20:39.841 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: return f(*arg) 2021-07-31T04:20:39.841 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:20:39.841 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:20:39.841 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:20:39.842 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:20:39.842 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:20:39.842 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:20:39.842 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:20:39.842 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T04:20:39.843 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:20:39.843 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: raise self._exception 2021-07-31T04:20:39.843 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:20:39.843 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:20:39.843 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:20:39.844 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T04:20:39.844 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:20:39.844 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: raise 2021-07-31T04:20:39.844 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:20:39.844 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T04:20:39.845 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:20:39.845 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:20:39.845 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:20:39.845 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:20:39.846 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:20:39.846 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:20:39.846 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:20:39.846 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:20:39.846 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:20:39.847 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:20:39.847 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: task.result() 2021-07-31T04:20:39.847 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:20:39.847 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: 2021-07-31T04:20:39.847 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T04:20:39.848 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: 2021-07-31T04:20:39.848 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:20:39.848 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:20:39.848 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: return f(*arg) 2021-07-31T04:20:39.848 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:20:39.849 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:20:39.849 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:20:39.849 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:20:39.849 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:20:39.849 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:20:39.850 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:20:39.850 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T04:20:39.850 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:20:39.850 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: raise self._exception 2021-07-31T04:20:39.850 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:20:39.851 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:20:39.851 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:20:39.851 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T04:20:39.851 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:20:39.851 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: raise 2021-07-31T04:20:39.852 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:20:39.852 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T04:20:39.852 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:20:39.852 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:20:39.852 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:39 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:20:41.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:41 smithi154 conmon[34553]: cluster 2021-07-31T04:20:39.993651+0000 mgr.x (mgr.24542) 6581 : cluster [DBG] pgmap v6611: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:41.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:41 smithi045 conmon[39718]: cluster 2021-07-31T04:20:39.993651+0000 mgr.x (mgr.24542 2021-07-31T04:20:41.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:41 smithi045 conmon[39718]: ) 6581 : cluster [DBG] pgmap v6611: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:41.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:41 smithi045 conmon[33655]: cluster 2021-07-31T04:20:39.993651+0000 mgr.x (mgr.24542) 6581 : cluster [DBG] pgmap v6611: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:41.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:41 smithi045 conmon[33655]: 2021-07-31T04:20:43.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:43 smithi154 conmon[34553]: cluster 2021-07-31T04:20:41.994145+0000 mgr.x (mgr.24542) 6582 : cluster 2021-07-31T04:20:43.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:43 smithi154 conmon[34553]: [DBG] pgmap v6612: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:43.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:43 smithi045 conmon[39718]: cluster 2021-07-31T04:20:41. 2021-07-31T04:20:43.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:43 smithi045 conmon[39718]: 994145+0000 mgr.x (mgr.24542) 6582 : cluster [DBG] pgmap v6612: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:43.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:43 smithi045 conmon[33655]: cluster 2021-07-31T04:20:41.994145+0000 mgr.x (mgr.24542) 6582 : cluster 2021-07-31T04:20:43.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:43 smithi045 conmon[33655]: [DBG] pgmap v6612: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:45.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:45 smithi154 conmon[34553]: cluster 2021-07-31T04:20:43.994546+0000 mgr.x (mgr.24542) 6583 : cluster [DBG] 2021-07-31T04:20:45.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:45 smithi154 conmon[34553]: pgmap v6613: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:45.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:45 smithi045 conmon[39718]: cluster 2021-07-31T04:20:43.994546+0000 mgr.x (mgr.24542) 6583 2021-07-31T04:20:45.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:45 smithi045 conmon[39718]: : cluster [DBG] pgmap v6613: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:45.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:45 smithi045 conmon[33655]: cluster 2021-07-31T04:20:43.994546 2021-07-31T04:20:45.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:45 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 6583 : cluster [DBG] pgmap v6613: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:47.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:47 smithi154 conmon[34553]: cluster 2021-07-31T04:20:45.995058+0000 mgr.x (mgr.24542) 6584 : cluster [DBG] 2021-07-31T04:20:47.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:47 smithi154 conmon[34553]: pgmap v6614: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:47.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:47 smithi045 conmon[39718]: cluster 2021-07-31T04:20:45.995058+0000 mgr.x ( 2021-07-31T04:20:47.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:47 smithi045 conmon[39718]: mgr.24542) 6584 : cluster [DBG] pgmap v6614: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:47.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:47 smithi045 conmon[33655]: cluster 2021-07-31T04:20:45.995058+0000 mgr.x (mgr.24542) 6584 : cluster [DBG] pgmap v6614: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:47.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:47 smithi045 conmon[33655]: 2021-07-31T04:20:49.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:49 smithi154 conmon[34553]: cluster 2021-07-31T04:20:47.995567+0000 mgr.x (mgr.24542) 6585 : cluster [DBG] pgmap v6615: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:49.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:49 smithi045 conmon[33655]: cluster 2021-07-31T04:20:47.995567+0000 mgr.x (mgr.24542) 6585 : cluster [DBG] pgmap v6615: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:49.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:49 smithi045 conmon[39718]: cluster 2021-07-31T04:20:47.995567+0000 mgr.x ( 2021-07-31T04:20:49.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:49 smithi045 conmon[39718]: mgr.24542) 6585 : cluster [DBG] pgmap v6615: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:51.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:51 smithi154 conmon[34553]: cluster 2021-07-31T04:20:49.996062+0000 mgr.x (mgr.24542) 6586 : cluster 2021-07-31T04:20:51.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:51 smithi154 conmon[34553]: [DBG] pgmap v6616: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:51.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:51 smithi045 conmon[33655]: cluster 2021-07-31T04:20:49.996062+0000 2021-07-31T04:20:51.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:51 smithi045 conmon[33655]: mgr.x (mgr.24542) 6586 : cluster [DBG] pgmap v6616: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:51.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:51 smithi045 conmon[39718]: cluster 2021-07-31T04:20:49. 2021-07-31T04:20:51.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:51 smithi045 conmon[39718]: 996062+0000 mgr.x (mgr.24542) 6586 : cluster [DBG] pgmap v6616: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:53.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:53 smithi154 conmon[34553]: cluster 2021-07-31T04:20:51.996468+0000 mgr.x (mgr.24542) 6587 : cluster [DBG] pgmap v6617: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:53.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:53 smithi045 conmon[33655]: cluster 2021-07-31T04:20:51.996468+0000 mgr.x 2021-07-31T04:20:53.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:53 smithi045 conmon[33655]: (mgr.24542) 6587 : cluster [DBG] pgmap v6617: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:53.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:53 smithi045 conmon[39718]: cluster 2021-07-31T04:20:51.996468+0000 2021-07-31T04:20:53.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:53 smithi045 conmon[39718]: mgr.x (mgr.24542) 6587 : cluster [DBG] pgmap v6617: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:55.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:55 smithi154 conmon[34553]: cluster 2021-07-31T04:20:53.997086+0000 mgr.x (mgr.24542) 6588 : cluster [DBG] pgmap v6618: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:55.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:55 smithi045 conmon[39718]: cluster 2021-07-31T04:20:53.997086+0000 mgr.x (mgr.24542) 6588 : 2021-07-31T04:20:55.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:55 smithi045 conmon[39718]: cluster [DBG] pgmap v6618: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:55.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:55 smithi045 conmon[33655]: cluster 2021-07-31T04:20:53.997086+0000 mgr.x (mgr.24542) 6588 2021-07-31T04:20:55.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:55 smithi045 conmon[33655]: : cluster [DBG] pgmap v6618: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:57.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:57 smithi154 conmon[34553]: cluster 2021-07-31T04:20:55.997618+0000 mgr.x (mgr.24542) 6589 : cluster [DBG] pgmap v6619: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:57.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:57 smithi045 conmon[33655]: cluster 2021-07-31T04:20:55.997618+0000 mgr.x (mgr.24542) 2021-07-31T04:20:57.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:57 smithi045 conmon[33655]: 6589 : cluster [DBG] pgmap v6619: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:57.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:57 smithi045 conmon[39718]: cluster 2021-07-31T04:20:55.997618+0000 mgr.x (mgr 2021-07-31T04:20:57.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:57 smithi045 conmon[39718]: .24542) 6589 : cluster [DBG] pgmap v6619: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:59.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:20:59 smithi154 conmon[34553]: cluster 2021-07-31T04:20:57.998175+0000 mgr.x (mgr.24542) 6590 : cluster [DBG] pgmap v6620: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:59.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:59 smithi045 conmon[39718]: cluster 2021-07-31T04:20: 2021-07-31T04:20:59.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:20:59 smithi045 conmon[39718]: 57.998175+0000 mgr.x (mgr.24542) 6590 : cluster [DBG] pgmap v6620: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:20:59.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:59 smithi045 conmon[33655]: cluster 2021-07-31T04:20:57.998175+0000 mgr.x (mgr.24542) 2021-07-31T04:20:59.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:20:59 smithi045 conmon[33655]: 6590 : cluster [DBG] pgmap v6620: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:21:01.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:01 smithi154 conmon[34553]: cluster 2021-07-31T04:20:59.998600+0000 mgr.x (mgr.24542) 6591 : cluster [DBG] pgmap v6621: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:21:01.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:01 smithi045 conmon[39718]: cluster 2021-07-31T04:20:59.998600+0000 mgr.x (mgr. 2021-07-31T04:21:01.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:01 smithi045 conmon[39718]: 24542) 6591 : cluster [DBG] pgmap v6621: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:21:01.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:01 smithi045 conmon[33655]: cluster 2021-07-31T04:20:59.998600+0000 mgr.x (mgr.24542) 6591 : cluster 2021-07-31T04:21:01.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:01 smithi045 conmon[33655]: [DBG] pgmap v6621: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:21:03.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:03 smithi154 conmon[34553]: cluster 2021-07-31T04:21:01.999069+0000 mgr.x (mgr.24542) 6592 : cluster [DBG] pgmap v6622: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:21:03.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:03 smithi045 conmon[39718]: cluster 2021-07-31T04:21:01.999069+0000 mgr.x (mgr.24542) 6592 : cluster [DBG] pgmap v6622: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:21:03.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:03 smithi045 conmon[33655]: cluster 2021-07-31T04:21:01.999069+0000 2021-07-31T04:21:03.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:03 smithi045 conmon[33655]: mgr.x (mgr.24542) 6592 : cluster [DBG] pgmap v6622: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:21:04.117 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T04:21:04.392 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T04:21:04.392 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 2h ago 2h - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T04:21:04.392 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 2h ago 2h 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T04:21:04.393 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 2h ago 2h 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T04:21:04.393 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 2h ago 2h - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T04:21:04.393 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T04:21:04.393 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 2h ago 2h 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T04:21:04.394 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T04:21:04.394 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 2h ago 2h - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T04:21:04.394 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 2h ago 2h 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T04:21:04.394 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T04:21:04.394 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T04:21:04.395 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T04:21:04.395 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T04:21:04.395 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 2h ago 2h 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T04:21:04.395 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 2h ago 2h 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T04:21:04.395 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 2h ago 2h 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T04:21:04.396 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 2h ago 2h 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T04:21:04.396 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 2h ago 2h 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T04:21:04.730 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T04:21:04.730 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T04:21:04.731 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T04:21:04.731 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:21:04.731 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T04:21:04.731 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T04:21:04.731 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:21:04.732 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:21:04.732 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T04:21:04.732 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T04:21:04.732 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:21:04.732 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T04:21:04.733 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T04:21:04.733 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T04:21:04.733 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:21:04.733 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T04:21:04.733 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T04:21:05.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:05 smithi154 conmon[34553]: cluster 2021-07-31T04:21:03.999580+0000 mgr.x (mgr.24542) 6593 : cluster [DBG] pgmap v6623: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:21:05.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:05 smithi154 conmon[34553]: 2021-07-31T04:21:05.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:05 smithi154 conmon[34553]: audit 2021-07-31T04:21:04.105833+0000 mgr.x (mgr.24542) 6594 : audit [DBG] from='client.31351 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:21:05.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:05 smithi154 conmon[34553]: audit 2021-07-31T04:21:04.384966+0000 mgr.x (mgr 2021-07-31T04:21:05.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:05 smithi154 conmon[34553]: .24542) 6595 : audit [DBG] from='client.31357 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:21:05.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:05 smithi154 conmon[34553]: audit 2021 2021-07-31T04:21:05.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:05 smithi154 conmon[34553]: -07-31T04:21:04.731101+0000 mon.a (mon.0) 3463 2021-07-31T04:21:05.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:05 smithi154 conmon[34553]: : audit [DBG] from='client.? 172.21.15.45:0/1381924565' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:21:05.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:05 smithi045 conmon[39718]: cluster 2021-07-31T04:21:03.999580+0000 2021-07-31T04:21:05.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:05 smithi045 conmon[39718]: mgr.x (mgr.24542) 6593 : cluster [DBG] pgmap v6623: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:21:05.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:05 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T04:21:05.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:05 smithi045 conmon[39718]: 04:21:04.105833+0000 mgr.x (mgr.24542) 6594 : audit 2021-07-31T04:21:05.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:05 smithi045 conmon[39718]: [DBG] from='client.31351 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:21:05.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:05 smithi045 conmon[39718]: audit 2021-07-31T04 2021-07-31T04:21:05.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:05 smithi045 conmon[39718]: :21:04.384966+0000 mgr.x (mgr 2021-07-31T04:21:05.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:05 smithi045 conmon[39718]: .24542) 6595 : audit [DBG] from='client.31357 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:21:05.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:05 smithi045 conmon[39718]: audit 2021-07-31T04 2021-07-31T04:21:05.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:05 smithi045 conmon[39718]: :21:04.731101+0000 mon.a (mon.0) 2021-07-31T04:21:05.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:05 smithi045 conmon[39718]: 3463 : audit [DBG] from='client.? 172.21.15.45:0/1381924565' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:21:05.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:05 smithi045 conmon[33655]: cluster 2021-07-31T04:21:03.999580+0000 mgr.x (mgr.24542) 6593 : 2021-07-31T04:21:05.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:05 smithi045 conmon[33655]: cluster [DBG] pgmap v6623: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:21:05.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:05 smithi045 conmon[33655]: audit 2021-07-31T04:21:04.105833+0000 mgr.x (mgr.24542) 6594 : audit [DBG] from='client.31351 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:21:05.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:05 smithi045 conmon[33655]: audit 2021-07-31T04:21:04.384966+0000 mgr.x (mgr. 2021-07-31T04:21:05.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:05 smithi045 conmon[33655]: 24542) 6595 : audit [DBG] from='client.31357 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:21:05.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:05 smithi045 conmon[33655]: audit 2021-07-31T 2021-07-31T04:21:05.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:05 smithi045 conmon[33655]: 04:21:04.731101+0000 mon.a (mon.0) 3463 : audit 2021-07-31T04:21:05.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:05 smithi045 conmon[33655]: [DBG] from='client.? 172.21.15.45:0/1381924565' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:21:07.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:07 smithi154 conmon[34553]: cluster 2021-07-31T04:21:05.999966+0000 mgr.x (mgr.24542) 6596 : cluster [DBG] pgmap v6624: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:21:07.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:07 smithi045 conmon[39718]: cluster 2021-07-31T04:21:05.999966+0000 mgr.x ( 2021-07-31T04:21:07.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:07 smithi045 conmon[39718]: mgr.24542) 6596 : cluster [DBG] pgmap v6624: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:21:07.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:07 smithi045 conmon[33655]: cluster 2021-07-31T04:21:05.999966+0000 mgr.x (mgr.24542 2021-07-31T04:21:07.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:07 smithi045 conmon[33655]: ) 6596 : cluster [DBG] pgmap v6624: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:21:09.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:09 smithi154 conmon[34553]: cluster 2021-07-31T04:21:08.000368+0000 mgr.x (mgr.24542) 6597 : cluster [DBG] pgmap v6625: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:21:09.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:09 smithi045 conmon[33655]: cluster 2021-07-31T04:21:08.000368+0000 mgr.x ( 2021-07-31T04:21:09.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:09 smithi045 conmon[33655]: mgr.24542) 6597 : cluster [DBG] pgmap v6625: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:21:09.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:09 smithi045 conmon[39718]: cluster 2021-07-31T04:21: 2021-07-31T04:21:09.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:09 smithi045 conmon[39718]: 08.000368+0000 mgr.x (mgr.24542) 6597 : cluster [DBG] pgmap v6625: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:21:11.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:11 smithi154 conmon[34553]: cluster 2021-07-31T04:21:10.000803+0000 mgr.x (mgr.24542) 6598 : cluster [DBG] pgmap v6626: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:21:11.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:11 smithi045 conmon[33655]: cluster 2021-07-31T04:21:10.000803+0000 mgr.x (mgr 2021-07-31T04:21:11.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:11 smithi045 conmon[33655]: .24542) 6598 : cluster [DBG] pgmap v6626: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:21:11.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:11 smithi045 conmon[39718]: cluster 2021-07-31T04:21: 2021-07-31T04:21:11.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:11 smithi045 conmon[39718]: 10.000803+0000 mgr.x (mgr.24542) 6598 : cluster [DBG] pgmap v6626: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:21:13.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:13 smithi154 conmon[34553]: cluster 2021-07-31T04:21:12.001256+0000 mgr.x (mgr.24542) 6599 : cluster [DBG] 2021-07-31T04:21:13.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:13 smithi154 conmon[34553]: pgmap v6627: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:21:13.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:13 smithi045 conmon[33655]: cluster 2021-07-31T04:21:12.001256+0000 mgr.x (mgr.24542) 6599 : cluster [DBG] pgmap v6627: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:21:13.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:13 smithi045 conmon[39718]: cluster 2021-07-31T04:21:12. 2021-07-31T04:21:13.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:13 smithi045 conmon[39718]: 001256+0000 mgr.x (mgr.24542) 6599 : cluster [DBG] pgmap v6627: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:21:15.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:15 smithi154 conmon[34553]: cluster 2021-07-31T04:21:14.001771+0000 mgr.x (mgr.24542) 6600 : cluster [DBG] 2021-07-31T04:21:15.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:15 smithi154 conmon[34553]: pgmap v6628: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:21:15.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:15 smithi045 conmon[33655]: cluster 2021-07-31T04:21:14.001771+0000 mgr.x (mgr.24542) 2021-07-31T04:21:15.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:15 smithi045 conmon[33655]: 6600 : cluster [DBG] pgmap v6628: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:21:15.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:15 smithi045 conmon[39718]: cluster 2021-07-31T04:21:14.001771+0000 mgr.x 2021-07-31T04:21:15.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:15 smithi045 conmon[39718]: (mgr.24542) 6600 : cluster [DBG] pgmap v6628: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:21:17.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:17 smithi154 conmon[34553]: cluster 2021-07-31T04:21:16.002368+0000 mgr.x (mgr.24542) 6601 : cluster [DBG] pgmap v6629: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:21:17.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:17 smithi045 conmon[39718]: cluster 2021-07-31T04:21:16.002368+0000 2021-07-31T04:21:17.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:17 smithi045 conmon[39718]: mgr.x (mgr.24542) 6601 : cluster [DBG] pgmap v6629: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:21:17.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:17 smithi045 conmon[33655]: cluster 2021-07-31T04:21:16.002368+0000 mgr.x (mgr.24542) 6601 : cluster [DBG] pgmap v6629: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:21:19.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:19 smithi154 conmon[34553]: cluster 2021-07-31T04:21:18.002771+0000 mgr.x (mgr.24542) 6602 : cluster [DBG] pgmap v6630: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:21:19.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:19 smithi154 conmon[34553]: 2021-07-31T04:21:19.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:19 smithi045 conmon[33655]: cluster 2021-07-31T04:21:18.002771+0000 mgr.x (mgr.24542) 6602 : cluster [DBG] pgmap v6630: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:21:19.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:19 smithi045 conmon[39718]: cluster 2021-07-31T04:21:18.002771+0000 mgr.x ( 2021-07-31T04:21:19.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:19 smithi045 conmon[39718]: mgr.24542) 6602 : cluster [DBG] pgmap v6630: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:21:21.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:21 smithi154 conmon[34553]: cluster 2021-07-31T04:21:20.003260+0000 mgr.x (mgr.24542) 6603 : cluster [DBG] pgmap v6631: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:21:21.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:21 smithi045 conmon[33655]: cluster 2021-07-31T04:21:20.003260+0000 mgr.x (mgr.24542) 6603 : cluster [DBG] pgmap v6631: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:21:21.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:21 smithi045 conmon[33655]: 2021-07-31T04:21:21.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:21 smithi045 conmon[39718]: cluster 2021-07-31T04:21:20.003260+0000 mgr.x (mgr.24542) 6603 : cluster [DBG] pgmap v6631: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:21:23.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:23 smithi154 conmon[34553]: cluster 2021-07-31T04:21:22.003650+0000 mgr.x (mgr.24542) 6604 : cluster [DBG] 2021-07-31T04:21:23.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:23 smithi154 conmon[34553]: pgmap v6632: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:21:23.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:23 smithi045 conmon[33655]: cluster 2021-07-31T04:21:22.003650+0000 mgr.x (mgr.24542) 6604 : cluster [DBG] pgmap v6632: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:21:23.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:23 smithi045 conmon[39718]: cluster 2021-07-31T04:21:22.003650+0000 mgr.x (mgr 2021-07-31T04:21:23.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:23 smithi045 conmon[39718]: .24542) 6604 : cluster [DBG] pgmap v6632: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:21:25.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:25 smithi154 conmon[34553]: cluster 2021-07-31T04:21:24.004115+0000 mgr.x (mgr.24542) 6605 : cluster [DBG] pgmap v6633: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:21:25.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:25 smithi045 conmon[39718]: cluster 2021-07-31T04:21:24.004115+0000 mgr.x ( 2021-07-31T04:21:25.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:25 smithi045 conmon[39718]: mgr.24542) 6605 : cluster [DBG] pgmap v6633: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:21:25.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:25 smithi045 conmon[33655]: cluster 2021-07-31T04:21:24.004115+0000 mgr.x (mgr.24542) 6605 : cluster [DBG] pgmap v6633: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:21:27.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:27 smithi154 conmon[34553]: cluster 2021-07-31T04:21:26.004478+0000 mgr.x (mgr.24542) 6606 : cluster [DBG] pgmap v6634: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:21:27.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:27 smithi154 conmon[34553]: 2021-07-31T04:21:27.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:27 smithi045 conmon[39718]: cluster 2021-07-31T04:21:26.004478+0000 mgr.x (mgr.24542 2021-07-31T04:21:27.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:27 smithi045 conmon[39718]: ) 6606 : cluster [DBG] pgmap v6634: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:21:27.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:27 smithi045 conmon[33655]: cluster 2021-07-31T04:21:26.004478+0000 mgr.x (mgr.24542) 6606 : cluster [DBG] pgmap v6634: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:21:29.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:29 smithi154 conmon[34553]: cluster 2021-07-31T04:21:28.005011+0000 mgr.x (mgr.24542) 6607 : cluster [DBG] pgmap v6635: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:21:29.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:29 smithi154 conmon[34553]: audit 2021-07-31T04:21:28.542456+0000 2021-07-31T04:21:29.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:29 smithi154 conmon[34553]: mon.b (mon.2) 845 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:21:29.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:29 smithi154 conmon[34553]: audit 2021-07-31T04:21:28.542694+0000 mon.b (mon.2) 846 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:21:29.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:29 smithi154 conmon[34553]: audit 2021-07-31T04:21:28.543461+0000 mon.a (mon.0) 3464 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:21:29.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:29 smithi154 conmon[34553]: 2021-07-31T04:21:29.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:29 smithi154 conmon[34553]: audit 2021-07-31T04:21:28.543881 2021-07-31T04:21:29.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:29 smithi154 conmon[34553]: +0000 mon.a (mon.0) 3465 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:21:29.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:29 smithi045 conmon[39718]: cluster 2021-07-31T04:21:28.005011+0000 mgr.x (mgr.24542) 6607 : cluster [DBG] pgmap v6635: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:21:29.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:29 smithi045 conmon[39718]: audit 2021-07-31T04:21:28.542456+0000 mon.b (mon.2) 845 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:21:29.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:29 smithi045 conmon[39718]: audit 2021-07-31T04:21:28.542694+0000 mon.b (mon.2) 2021-07-31T04:21:29.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:29 smithi045 conmon[39718]: 846 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:21:29.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:29 smithi045 conmon[39718]: audit 2021-07-31T04:21:29.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:29 smithi045 conmon[39718]: 2021-07-31T04:21:28.543461+0000 mon.a (mon.0) 3464 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:21:29.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:29 smithi045 conmon[39718]: audit 2021-07-31T04:21:28 2021-07-31T04:21:29.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:29 smithi045 conmon[39718]: .543881+0000 mon.a (mon.0) 3465 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:21:29.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:29 smithi045 conmon[33655]: cluster 2021-07-31T04:21:28.005011+0000 mgr.x ( 2021-07-31T04:21:29.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:29 smithi045 conmon[33655]: mgr.24542) 6607 : cluster [DBG] pgmap v6635: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:21:29.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:29 smithi045 conmon[33655]: audit 2021-07-31T04:21:28.542456+0000 mon.b (mon.2) 845 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:21:29.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:29 smithi045 conmon[33655]: audit 2021-07-31T04:21:28.542694+0000 mon.b (mon.2) 846 2021-07-31T04:21:29.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:29 smithi045 conmon[33655]: : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:21:29.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:29 smithi045 conmon[33655]: audit 2021-07-31T04:21:29.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:29 smithi045 conmon[33655]: 2021-07-31T04:21:28.543461+0000 mon.a (mon.0) 3464 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:21:29.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:29 smithi045 conmon[33655]: audit 2021-07-31T04:21:28.543881+0000 mon.a (mon.0) 3465 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:21:31.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:31 smithi154 conmon[34553]: cluster 2021-07-31T04:21:30.005423+0000 mgr.x (mgr.24542) 6608 : cluster [DBG] pgmap v6636: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:21:31.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:31 smithi045 conmon[39718]: cluster 2021-07-31T04:21:30.005423+0000 mgr.x ( 2021-07-31T04:21:31.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:31 smithi045 conmon[39718]: mgr.24542) 6608 : cluster [DBG] pgmap v6636: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:21:31.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:31 smithi045 conmon[33655]: cluster 2021-07-31T04:21:30.005423+0000 mgr.x (mgr.24542 2021-07-31T04:21:31.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:31 smithi045 conmon[33655]: ) 6608 : cluster [DBG] pgmap v6636: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:21:33.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:33 smithi154 conmon[34553]: cluster 2021-07-31T04:21:32.005818+0000 mgr.x (mgr.24542) 6609 : cluster [DBG] pgmap v6637: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:21:33.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:33 smithi045 conmon[39718]: cluster 2021-07-31T04:21:32.005818+0000 mgr.x ( 2021-07-31T04:21:33.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:33 smithi045 conmon[39718]: mgr.24542) 6609 : cluster [DBG] pgmap v6637: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:21:33.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:33 smithi045 conmon[33655]: cluster 2021-07-31T04:21:32.005818+0000 mgr.x (mgr.24542) 6609 : cluster [DBG] pgmap v6637: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:21:35.048 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T04:21:35.326 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T04:21:35.327 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 2h ago 2h - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T04:21:35.327 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 2h ago 2h 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T04:21:35.327 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 2h ago 2h 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T04:21:35.327 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 2h ago 2h - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T04:21:35.327 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T04:21:35.328 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 2h ago 2h 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T04:21:35.328 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T04:21:35.328 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 2h ago 2h - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T04:21:35.328 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 2h ago 2h 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T04:21:35.328 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T04:21:35.329 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T04:21:35.329 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T04:21:35.329 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T04:21:35.329 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 2h ago 2h 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T04:21:35.329 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 2h ago 2h 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T04:21:35.330 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 2h ago 2h 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T04:21:35.330 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 2h ago 2h 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T04:21:35.330 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 2h ago 2h 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T04:21:35.662 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T04:21:35.662 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T04:21:35.662 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T04:21:35.662 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:21:35.663 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T04:21:35.663 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T04:21:35.663 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:21:35.663 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:21:35.664 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T04:21:35.664 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T04:21:35.664 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:21:35.664 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T04:21:35.664 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T04:21:35.665 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T04:21:35.665 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:21:35.665 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T04:21:35.665 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T04:21:35.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:35 smithi154 conmon[34553]: cluster 2021-07-31T04:21:34.006345+0000 mgr.x (mgr.24542) 6610 : cluster [DBG] pgmap v6638: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:21:35.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:35 smithi045 conmon[39718]: cluster 2021-07-31T04:21:34.006345 2021-07-31T04:21:35.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:35 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 6610 : cluster [DBG] pgmap v6638: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:21:35.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:35 smithi045 conmon[33655]: cluster 2021-07-31T04:21:34.006345 2021-07-31T04:21:35.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:35 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 6610 : cluster [DBG] pgmap v6638: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:21:36.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:36 smithi154 conmon[34553]: audit 2021-07-31T04:21:35.036554+0000 mgr.x (mgr.24542) 6611 : audit [DBG] 2021-07-31T04:21:36.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:36 smithi154 conmon[34553]: from='client.31369 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:21:36.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:36 smithi154 conmon[34553]: audit 2021-07-31T04:21:35.319336+0000 mgr.x (mgr.24542) 6612 : audit [DBG] from='client.31375 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:21:36.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:36 smithi154 conmon[34553]: audit 2021-07-31T04:21:35.662790+0000 2021-07-31T04:21:36.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:36 smithi154 conmon[34553]: mon.a (mon.0) 3466 : audit [DBG] from='client.? 172.21.15.45:0/3020466275' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:21:36.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:36 smithi045 conmon[33655]: audit 2021-07-31T04:21:35.036554+0000 mgr.x (mgr.24542) 6611 2021-07-31T04:21:36.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:36 smithi045 conmon[33655]: : audit [DBG] from='client.31369 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:21:36.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:36 smithi045 conmon[33655]: audit 2021-07-31T04:21:35.319336+0000 mgr.x (mgr.24542) 6612 : audit [DBG] from='client.31375 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:21:36.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:36 smithi045 conmon[33655]: audit 2021 2021-07-31T04:21:36.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:36 smithi045 conmon[33655]: -07-31T04:21:35 2021-07-31T04:21:36.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:36 smithi045 conmon[33655]: .662790+0000 mon.a (mon. 2021-07-31T04:21:36.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:36 smithi045 conmon[33655]: 0) 3466 : audit [DBG] from='client.? 172.21.15.45:0/3020466275' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:21:36.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:36 smithi045 conmon[39718]: audit 2021-07-31T04:21:35.036554+0000 mgr.x (mgr.24542) 6611 : audit [DBG] from='client.31369 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:21:36.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:36 smithi045 conmon[39718]: audit 2021-07-31T04:21:35.319336+0000 mgr.x (mgr.24542) 6612 : audit [DBG] from='client.31375 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:21:36.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:36 smithi045 conmon[39718]: audit 2021-07-31T04:21 2021-07-31T04:21:36.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:36 smithi045 conmon[39718]: :35.662790+0000 mon.a (mon 2021-07-31T04:21:36.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:36 smithi045 conmon[39718]: .0) 3466 : audit [DBG] 2021-07-31T04:21:36.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:36 smithi045 conmon[39718]: from='client.? 172.21.15.45:0/3020466275' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:21:37.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:37 smithi154 conmon[34553]: cluster 2021-07-31T04:21:36.006742+0000 mgr.x (mgr.24542) 6613 2021-07-31T04:21:37.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:37 smithi154 conmon[34553]: : cluster [DBG] pgmap v6639: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:21:37.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:37 smithi045 conmon[39718]: cluster 2021-07-31T04:21:36.006742+0000 2021-07-31T04:21:37.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:37 smithi045 conmon[39718]: mgr.x (mgr.24542) 6613 : cluster [DBG] pgmap v6639: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:21:37.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:37 smithi045 conmon[33655]: cluster 2021-07-31T04:21:36.006742+0000 mgr.x (mgr.24542) 6613 : 2021-07-31T04:21:37.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:37 smithi045 conmon[33655]: cluster [DBG] pgmap v6639: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:21:38.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[34553]: audit 2021-07-31T04:21:38.322525+0000 mon.b (mon.2) 847 2021-07-31T04:21:38.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[34553]: : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T04:21:38.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[34553]: audit 2021-07-31T04:21:38.324094+0000 mon.b (mon.2) 848 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T04:21:38.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[34553]: audit 2021-07-31T04:21: 2021-07-31T04:21:38.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[34553]: 38.325248+0000 mon.b (mon. 2021-07-31T04:21:38.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[34553]: 2) 849 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T04:21:38.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[34553]: 2021-07-31T04:21:38.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: debug 2021-07-31T04:21:38.367+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:21:38.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:21:38.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:21:38.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: yield 2021-07-31T04:21:38.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:21:38.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:21:38.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:21:38.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:21:38.756 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:21:38.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:21:38.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:21:38.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:21:38.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:21:38.757 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: task.result() 2021-07-31T04:21:38.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:21:38.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: debug 2021-07-31T04:21:38.379+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:21:38.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:21:38.758 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:21:38.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: yield 2021-07-31T04:21:38.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:21:38.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:21:38.759 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:21:38.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:21:38.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:21:38.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:21:38.760 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:21:38.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:21:38.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:21:38.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: task.result() 2021-07-31T04:21:38.761 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:21:38.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: debug 2021-07-31T04:21:38.403+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:21:38.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:21:38.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:21:38.762 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: yield 2021-07-31T04:21:38.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:21:38.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:21:38.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:21:38.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:21:38.763 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:21:38.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:21:38.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:21:38.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:21:38.764 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:21:38.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: task.result() 2021-07-31T04:21:38.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:21:38.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: debug 2021-07-31T04:21:38.404+0000 7fcb97969700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:21:38.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:21:38.765 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:21:38.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: yield 2021-07-31T04:21:38.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:21:38.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:21:38.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:21:38.766 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:21:38.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:21:38.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:21:38.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:21:38.767 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:21:38.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:21:38.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: task.result() 2021-07-31T04:21:38.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:21:38.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: 2021-07-31T04:21:38.768 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T04:21:38.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: 2021-07-31T04:21:38.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:21:38.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:21:38.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: return f(*arg) 2021-07-31T04:21:38.769 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:21:38.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:21:38.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:21:38.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:21:38.770 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:21:38.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:21:38.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:21:38.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T04:21:38.771 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:21:38.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: raise self._exception 2021-07-31T04:21:38.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:21:38.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:21:38.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:21:38.772 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T04:21:38.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:21:38.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: raise 2021-07-31T04:21:38.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:21:38.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T04:21:38.773 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:21:38.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:21:38.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:21:38.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: debug 2021-07-31T04:21:38.413+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:21:38.774 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:21:38.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:21:38.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: yield 2021-07-31T04:21:38.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:21:38.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:21:38.775 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:21:38.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:21:38.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:21:38.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:21:38.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:21:38.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:21:38.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:21:38.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: task.result() 2021-07-31T04:21:38.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:21:38.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: debug 2021-07-31T04:21:38.413+0000 7fcb9816a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:21:38.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:21:38.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:21:38.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: yield 2021-07-31T04:21:38.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:21:38.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:21:38.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:21:38.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:21:38.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:21:38.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:21:38.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:21:38.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:21:38.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:21:38.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: task.result() 2021-07-31T04:21:38.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:21:38.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: 2021-07-31T04:21:38.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T04:21:38.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: 2021-07-31T04:21:38.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:21:38.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:21:38.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: return f(*arg) 2021-07-31T04:21:38.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:21:38.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:21:38.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:21:38.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:21:38.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:21:38.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:21:38.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:21:38.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T04:21:38.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:21:38.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: raise self._exception 2021-07-31T04:21:38.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:21:38.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:21:38.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:21:38.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T04:21:38.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:21:38.786 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: raise 2021-07-31T04:21:38.786 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:21:38.786 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T04:21:38.786 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:21:38.787 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:21:38.787 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:21:38 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:21:38.788 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:38 smithi045 conmon[33655]: audit 2021-07-31T04:21:38.322525+0000 mon.b (mon.2) 847 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T04:21:38.788 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:38 smithi045 conmon[33655]: audit 2021-07-31T04:21:38.324094+0000 mon.b (mon.2) 848 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T04:21:38.788 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:38 smithi045 conmon[33655]: audit 2021-07-31T04:21:38.325248+0000 mon.b (mon.2) 849 : audit [INF] 2021-07-31T04:21:38.788 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:38 smithi045 conmon[33655]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T04:21:38.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:38 smithi045 conmon[39718]: audit 2021-07-31T04:21:38.322525+0000 mon.b (mon.2) 847 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T04:21:38.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:38 smithi045 conmon[39718]: audit 2021-07-31T04:21:38.324094+0000 mon.b (mon.2) 848 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T04:21:38.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:38 smithi045 conmon[39718]: audit 2021-07- 2021-07-31T04:21:38.789 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:38 smithi045 conmon[39718]: 31T04:21:38.325248+0000 mon.b (mon.2) 849 : audit 2021-07-31T04:21:38.790 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:38 smithi045 conmon[39718]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T04:21:39.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: cluster 2021-07-31T04:21:38.007143+0000 mgr.x (mgr.24542) 6614 : cluster [DBG] 2021-07-31T04:21:39.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: pgmap v6640: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:21:39.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:21:38.326033+0000 mgr.x (mgr.24542) 6615 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T04:21:39.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:21:38.368845+0000 mgr.x (mgr.24542) 6616 : cephadm [ERR] 2021-07-31T04:21:39.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:21:39.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:21:39.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: yield 2021-07-31T04:21:39.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:21:39.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:21:39.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:21:39.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:21:39.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:21:39.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:21:39.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:21:39.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:21:39.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:21:39.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: task.result() 2021-07-31T04:21:39.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:21:39.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:21:38.369268+0000 mgr.x (mgr.24542) 6617 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T04:21:39.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:21:38.380466+0000 mgr.x (mgr.24542) 6618 : cephadm [ERR] 2021-07-31T04:21:39.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:21:39.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:21:39.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: yield 2021-07-31T04:21:39.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:21:39.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:21:39.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:21:39.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:21:39.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:21:39.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:21:39.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:21:39.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:21:39.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:21:39.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: task.result() 2021-07-31T04:21:39.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:21:39.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:21:38.381359+0000 mgr.x (mgr.24542) 6619 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T04:21:39.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:21:38.405834+0000 mgr.x (mgr.24542) 6620 : cephadm [ERR] 2021-07-31T04:21:39.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:21:39.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:21:39.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: yield 2021-07-31T04:21:39.761 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:21:39.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:21:39.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:21:39.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:21:39.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:21:39.762 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:21:39.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:21:39.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:21:39.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:21:39.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: task.result() 2021-07-31T04:21:39.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:21:39.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:21:38.406270+0000 mgr.x (mgr.24542) 6621 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:21:39.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:21:39.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:21:39.764 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: yield 2021-07-31T04:21:39.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:21:39.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:21:39.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:21:39.765 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:21:39.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:21:39.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:21:39.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:21:39.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:21:39.766 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:21:39.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: task.result() 2021-07-31T04:21:39.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:21:39.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: 2021-07-31T04:21:39.767 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T04:21:39.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: 2021-07-31T04:21:39.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:21:39.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:21:39.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: return f(*arg) 2021-07-31T04:21:39.768 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:21:39.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:21:39.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:21:39.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:21:39.769 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:21:39.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:21:39.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:21:39.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T04:21:39.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:21:39.770 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: raise self._exception 2021-07-31T04:21:39.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:21:39.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:21:39.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:21:39.771 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T04:21:39.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:21:39.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: raise 2021-07-31T04:21:39.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:21:39.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T04:21:39.772 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:21:39.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:21:39.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:21:39.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:21:38.414946+0000 mgr.x 2021-07-31T04:21:39.773 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: (mgr.24542) 6622 : cephadm [ERR] 2021-07-31T04:21:39.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:21:39.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:21:39.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: yield 2021-07-31T04:21:39.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:21:39.774 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:21:39.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:21:39.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:21:39.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:21:39.775 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:21:39.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:21:39.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:21:39.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:21:39.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: task.result() 2021-07-31T04:21:39.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:21:39.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:21:38.415290+0000 mgr.x (mgr.24542) 6623 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:21:39.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:21:39.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:21:39.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: yield 2021-07-31T04:21:39.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:21:39.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:21:39.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:21:39.778 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:21:39.779 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:21:39.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:21:39.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:21:39.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:21:39.780 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:21:39.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: task.result() 2021-07-31T04:21:39.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:21:39.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: 2021-07-31T04:21:39.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T04:21:39.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: 2021-07-31T04:21:39.782 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:21:39.782 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:21:39.782 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: return f(*arg) 2021-07-31T04:21:39.782 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:21:39.783 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:21:39.783 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:21:39.783 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:21:39.783 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:21:39.784 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:21:39.784 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:21:39.784 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T04:21:39.784 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:21:39.784 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: raise self._exception 2021-07-31T04:21:39.785 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:21:39.785 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:21:39.785 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:21:39.785 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T04:21:39.786 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:21:39.786 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: raise 2021-07-31T04:21:39.786 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:21:39.786 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T04:21:39.787 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:21:39.787 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:21:39.792 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:39 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:21:39.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: cluster 2021-07-31T04:21:38.007143+0000 mgr.x (mgr.24542) 6614 : cluster [DBG] 2021-07-31T04:21:39.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: pgmap v6640: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:21:39.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: cephadm 2021-07-31T04:21:38.326033+0000 mgr.x (mgr.24542) 6615 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T04:21:39.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: cephadm 2021-07-31T04:21:38.368845+0000 mgr.x (mgr.24542) 6616 : cephadm [ERR] 2021-07-31T04:21:39.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:21:39.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:21:39.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: yield 2021-07-31T04:21:39.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:21:39.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:21:39.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:21:39.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:21:39.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:21:39.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:21:39.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:21:39.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:21:39.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:21:39.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: task.result() 2021-07-31T04:21:39.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:21:39.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: cephadm 2021-07-31T04:21:38.369268+0000 mgr.x (mgr.24542) 6617 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T04:21:39.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: cephadm 2021-07-31T04:21:38.380466+0000 mgr.x (mgr.24542) 6618 : cephadm [ERR] 2021-07-31T04:21:39.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:21:39.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:21:39.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: yield 2021-07-31T04:21:39.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:21:39.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:21:39.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:21:39.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:21:39.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:21:39.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:21:39.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:21:39.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:21:39.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:21:39.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: task.result() 2021-07-31T04:21:39.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:21:39.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: cephadm 2021-07-31T04:21:38.381359+0000 mgr.x (mgr.24542) 6619 : cephadm [INF] 2021-07-31T04:21:39.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T04:21:39.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: cephadm 2021-07-31T04:21:38.405834+0000 mgr.x (mgr.24542) 6620 : cephadm [ERR] 2021-07-31T04:21:39.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:21:39.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:21:39.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: yield 2021-07-31T04:21:39.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:21:39.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:21:39.802 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:21:39.802 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:21:39.802 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:21:39.802 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:21:39.803 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:21:39.803 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:21:39.803 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:21:39.803 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: task.result() 2021-07-31T04:21:39.803 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:21:39.804 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: cephadm 2021-07-31T04:21:38.406270+0000 mgr.x (mgr.24542) 6621 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:21:39.804 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:21:39.804 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:21:39.804 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: yield 2021-07-31T04:21:39.804 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:21:39.805 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:21:39.805 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:21:39.805 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:21:39.805 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:21:39.805 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:21:39.806 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:21:39.806 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:21:39.806 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:21:39.806 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: task.result() 2021-07-31T04:21:39.806 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:21:39.807 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: 2021-07-31T04:21:39.807 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T04:21:39.807 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: 2021-07-31T04:21:39.807 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:21:39.807 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:21:39.808 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: return f(*arg) 2021-07-31T04:21:39.808 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:21:39.808 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:21:39.808 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:21:39.809 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:21:39.809 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:21:39.809 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: cluster 2021-07-31T04:21:38.007143+0000 mgr.x (mgr. 2021-07-31T04:21:39.810 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: 24542) 6614 : cluster [DBG] pgmap v6640: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:21:39.810 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: cephadm 2021-07-31T04:21:38.326033+0000 mgr.x (mgr.24542) 6615 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T04:21:39.810 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: cephadm 2021-07-31T04:21:38.368845+0000 mgr.x (mgr.24542) 6616 : cephadm [ERR] 2021-07-31T04:21:39.810 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:21:39.810 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:21:39.811 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: yield 2021-07-31T04:21:39.811 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:21:39.811 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:21:39.811 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:21:39.811 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:21:39.812 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:21:39.812 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:21:39.812 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:21:39.812 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:21:39.812 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:21:39.813 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: task.result() 2021-07-31T04:21:39.813 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:21:39.813 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: 2021-07-31T04:21:39.813 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: cephadm 2021-07-31T04:21:38.369268+0000 mgr.x (mgr.24542) 6617 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T04:21:39.813 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: cephadm 2021-07-31T04:21:38.380466+0000 mgr.x (mgr.24542) 6618 : cephadm [ERR] 2021-07-31T04:21:39.814 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:21:39.814 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:21:39.814 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: yield 2021-07-31T04:21:39.814 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:21:39.814 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:21:39.815 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:21:39.815 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:21:39.815 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:21:39.815 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:21:39.815 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:21:39.816 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:21:39.816 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:21:39.816 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: task.result() 2021-07-31T04:21:39.816 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:21:39.816 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: cephadm 2021-07-31T04:21:38.381359+0000 mgr.x (mgr.24542) 6619 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T04:21:39.817 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: cephadm 2021-07-31T04:21:38.405834+0000 mgr.x (mgr.24542) 6620 : cephadm 2021-07-31T04:21:39.817 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:21:39.817 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:21:39.818 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T04:21:39.818 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:21:39.818 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: raise self._exception 2021-07-31T04:21:39.818 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:21:39.818 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:21:39.819 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:21:39.819 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T04:21:39.819 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:21:39.819 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: raise 2021-07-31T04:21:39.819 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:21:39.820 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T04:21:39.820 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:21:39.820 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:21:39.820 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:21:39.820 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: cephadm 2021-07-31T04:21:38 2021-07-31T04:21:39.821 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: .414946+0000 mgr.x (mgr.24542) 6622 : cephadm [ERR] 2021-07-31T04:21:39.821 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:21:39.821 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:21:39.821 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: yield 2021-07-31T04:21:39.821 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:21:39.822 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:21:39.822 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:21:39.822 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:21:39.822 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:21:39.822 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:21:39.823 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:21:39.823 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:21:39.823 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:21:39.823 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: task.result() 2021-07-31T04:21:39.823 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:21:39.824 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: cephadm 2021-07-31T04:21:38.415290+0000 mgr.x (mgr.24542) 6623 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:21:39.824 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:21:39.824 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:21:39.824 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: yield 2021-07-31T04:21:39.824 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:21:39.825 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:21:39.825 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:21:39.825 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:21:39.825 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: [ERR] 2021-07-31T04:21:39.826 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:21:39.826 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:21:39.826 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: yield 2021-07-31T04:21:39.826 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:21:39.826 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:21:39.827 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:21:39.827 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:21:39.827 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:21:39.827 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:21:39.827 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:21:39.828 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:21:39.828 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:21:39.828 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: task.result() 2021-07-31T04:21:39.828 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:21:39.828 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: cephadm 2021-07-31T04 2021-07-31T04:21:39.829 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: :21:38.406270+0000 mgr.x (mgr.24542) 6621 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:21:39.829 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:21:39.829 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:21:39.829 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: yield 2021-07-31T04:21:39.829 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:21:39.830 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:21:39.830 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:21:39.830 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:21:39.830 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:21:39.830 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:21:39.831 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:21:39.831 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:21:39.831 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:21:39.831 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: task.result() 2021-07-31T04:21:39.831 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:21:39.832 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: 2021-07-31T04:21:39.832 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T04:21:39.832 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: 2021-07-31T04:21:39.832 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:21:39.832 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:21:39.833 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: return f(*arg) 2021-07-31T04:21:39.833 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:21:39.833 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:21:39.833 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:21:39.833 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:21:39.833 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:21:39.834 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:21:39.834 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:21:39.834 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:21:39.835 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:21:39.835 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:21:39.835 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: task.result() 2021-07-31T04:21:39.835 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:21:39.835 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: 2021-07-31T04:21:39.836 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T04:21:39.836 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: 2021-07-31T04:21:39.836 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:21:39.836 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:21:39.836 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: return f(*arg) 2021-07-31T04:21:39.837 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:21:39.837 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:21:39.837 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:21:39.837 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:21:39.837 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:21:39.838 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:21:39.838 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:21:39.838 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T04:21:39.838 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:21:39.838 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: raise self._exception 2021-07-31T04:21:39.839 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:21:39.839 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:21:39.839 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:21:39.839 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T04:21:39.839 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:21:39.840 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: raise 2021-07-31T04:21:39.840 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:21:39.840 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T04:21:39.840 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:21:39.840 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:21:39.841 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:21:39.841 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:21:39.841 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:21:39.842 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T04:21:39.842 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:21:39.842 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: raise self._exception 2021-07-31T04:21:39.842 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:21:39.842 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:21:39.843 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:21:39.843 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T04:21:39.843 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:21:39.843 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: raise 2021-07-31T04:21:39.843 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:21:39.844 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T04:21:39.844 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:21:39.844 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:21:39.844 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:21:39.844 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: cephadm 2021-07-31T04:21:38.414946+0000 mgr.x (mgr.24542) 6622 : cephadm [ERR] 2021-07-31T04:21:39.845 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:21:39.845 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:21:39.845 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: yield 2021-07-31T04:21:39.845 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:21:39.845 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:21:39.846 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:21:39.846 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:21:39.846 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:21:39.846 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:21:39.846 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:21:39.847 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:21:39.847 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:21:39.847 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: task.result() 2021-07-31T04:21:39.847 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:21:39.847 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: cephadm 2021-07-31T04:21:38.415290+0000 mgr.x (mgr.24542) 6623 2021-07-31T04:21:39.848 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:21:39.848 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:21:39.848 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:21:39.848 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: yield 2021-07-31T04:21:39.848 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:21:39.849 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:21:39.849 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:21:39.849 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:21:39.849 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:21:39.849 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:21:39.850 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:21:39.850 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:21:39.850 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:21:39.850 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: task.result() 2021-07-31T04:21:39.850 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:21:39.851 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: 2021-07-31T04:21:39.851 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T04:21:39.851 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: 2021-07-31T04:21:39.851 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:21:39.851 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:21:39.852 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: return f(*arg) 2021-07-31T04:21:39.852 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:21:39.852 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:21:39.852 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:21:39.852 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:21:39.853 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:21:39.853 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:21:39.853 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:21:39.853 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T04:21:39.853 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:21:39.854 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: raise self._exception 2021-07-31T04:21:39.854 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:21:39.854 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:21:39.854 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:21:39.854 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T04:21:39.855 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:21:39.855 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: raise 2021-07-31T04:21:39.855 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:21:39.855 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T04:21:39.855 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:21:39.856 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:21:39.856 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:39 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:21:41.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:41 smithi154 conmon[34553]: cluster 2021-07-31T04:21:40.007712+0000 mgr.x (mgr.24542) 6624 : cluster [DBG] 2021-07-31T04:21:41.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:41 smithi154 conmon[34553]: pgmap v6641: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:21:41.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:41 smithi045 conmon[39718]: cluster 2021-07-31T04:21:40.007712+0000 mgr.x (mgr. 2021-07-31T04:21:41.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:41 smithi045 conmon[39718]: 24542) 6624 : cluster [DBG] pgmap v6641: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:21:41.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:41 smithi045 conmon[33655]: cluster 2021-07-31T04:21:40.007712 2021-07-31T04:21:41.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:41 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 6624 : cluster [DBG] pgmap v6641: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:21:43.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:43 smithi154 conmon[34553]: cluster 2021-07-31T04:21:42.008186+0000 mgr.x (mgr.24542) 6625 : cluster [DBG] 2021-07-31T04:21:43.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:43 smithi154 conmon[34553]: pgmap v6642: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:21:43.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:43 smithi045 conmon[39718]: cluster 2021-07-31T04:21:42.008186+0000 2021-07-31T04:21:43.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:43 smithi045 conmon[39718]: mgr.x (mgr.24542) 6625 : cluster [DBG] pgmap v6642: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:21:43.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:43 smithi045 conmon[33655]: cluster 2021-07-31T04:21:42.008186+0000 mgr.x (mgr.24542) 2021-07-31T04:21:43.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:43 smithi045 conmon[33655]: 6625 : cluster [DBG] pgmap v6642: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:21:45.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:45 smithi154 conmon[34553]: cluster 2021-07-31T04:21:44.008695+0000 mgr.x (mgr.24542) 6626 : cluster [DBG] pgmap v6643: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:21:45.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:45 smithi045 conmon[33655]: cluster 2021-07-31T04:21:44.008695+0000 mgr.x (mgr.24542 2021-07-31T04:21:45.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:45 smithi045 conmon[33655]: ) 6626 : cluster [DBG] pgmap v6643: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:21:45.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:45 smithi045 conmon[39718]: cluster 2021-07-31T04:21:44.008695+0000 mgr.x 2021-07-31T04:21:45.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:45 smithi045 conmon[39718]: (mgr.24542) 6626 : cluster [DBG] pgmap v6643: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:21:47.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:47 smithi154 conmon[34553]: cluster 2021-07-31T04:21:46.009183+0000 mgr.x (mgr.24542) 6627 : cluster [DBG] 2021-07-31T04:21:47.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:47 smithi154 conmon[34553]: pgmap v6644: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:21:47.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:47 smithi045 conmon[33655]: cluster 2021-07-31T04:21:46.009183+0000 mgr.x (mgr.24542) 6627 : 2021-07-31T04:21:47.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:47 smithi045 conmon[33655]: cluster [DBG] pgmap v6644: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:21:47.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:47 smithi045 conmon[39718]: cluster 2021-07-31T04:21:46.009183+0000 mgr.x (mgr 2021-07-31T04:21:47.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:47 smithi045 conmon[39718]: .24542) 6627 : cluster [DBG] pgmap v6644: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:21:49.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:49 smithi154 conmon[34553]: cluster 2021-07-31T04:21:48.009642+0000 mgr.x (mgr.24542) 6628 : cluster [DBG] pgmap v6645: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:21:49.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:49 smithi045 conmon[39718]: cluster 2021-07-31T04:21:48.009642+0000 mgr.x (mgr 2021-07-31T04:21:49.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:49 smithi045 conmon[39718]: .24542) 6628 : cluster [DBG] pgmap v6645: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:21:49.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:49 smithi045 conmon[33655]: cluster 2021-07-31T04:21:48.009642+0000 mgr.x (mgr.24542) 6628 : 2021-07-31T04:21:49.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:49 smithi045 conmon[33655]: cluster [DBG] pgmap v6645: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:21:51.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:51 smithi154 conmon[34553]: cluster 2021-07-31T04:21:50.010190+0000 mgr.x (mgr.24542) 6629 : cluster [DBG] pgmap v6646: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:21:51.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:51 smithi045 conmon[39718]: cluster 2021-07-31T04:21:50.010190+0000 mgr.x ( 2021-07-31T04:21:51.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:51 smithi045 conmon[39718]: mgr.24542) 6629 : cluster [DBG] pgmap v6646: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:21:51.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:51 smithi045 conmon[33655]: cluster 2021-07-31T04:21:50.010190 2021-07-31T04:21:51.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:51 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 6629 : cluster [DBG] pgmap v6646: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:21:53.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:53 smithi154 conmon[34553]: cluster 2021-07-31T04:21:52.010699+0000 mgr.x (mgr.24542) 6630 : cluster [DBG] pgmap v6647: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:21:53.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:53 smithi045 conmon[39718]: cluster 2021-07-31T04:21:52.010699+0000 mgr.x (mgr.24542) 6630 : cluster [DBG] pgmap v6647: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:21:53.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:53 smithi045 conmon[33655]: cluster 2021-07-31T04:21:52.010699+0000 mgr.x 2021-07-31T04:21:53.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:53 smithi045 conmon[33655]: (mgr.24542) 6630 : cluster [DBG] pgmap v6647: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:21:55.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:55 smithi154 conmon[34553]: cluster 2021-07-31T04: 2021-07-31T04:21:55.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:55 smithi154 conmon[34553]: 21:54.011220+0000 mgr.x (mgr.24542) 6631 : cluster [DBG] pgmap v6648: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:21:55.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:55 smithi045 conmon[33655]: cluster 2021-07-31T04:21:54.011220+0000 mgr.x (mgr.24542) 6631 2021-07-31T04:21:55.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:55 smithi045 conmon[33655]: : cluster [DBG] pgmap v6648: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:21:55.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:55 smithi045 conmon[39718]: cluster 2021-07-31T04:21:54.011220 2021-07-31T04:21:55.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:55 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 6631 : cluster [DBG] pgmap v6648: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:21:57.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:57 smithi045 conmon[39718]: cluster 2021-07-31T04:21:56. 2021-07-31T04:21:57.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:57 smithi045 conmon[39718]: 011646+0000 mgr.x (mgr.24542) 6632 : cluster [DBG] pgmap v6649: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:21:57.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:57 smithi045 conmon[33655]: cluster 2021-07-31T04:21:56.011646+0000 mgr.x (mgr.24542) 6632 : 2021-07-31T04:21:57.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:57 smithi045 conmon[33655]: cluster [DBG] pgmap v6649: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:21:58.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:57 smithi154 conmon[34553]: cluster 2021-07-31T04:21:56.011646+0000 mgr.x (mgr.24542) 6632 : cluster [DBG] pgmap v6649: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:21:59.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:59 smithi045 conmon[39718]: cluster 2021-07-31T04:21:58.012135+0000 mgr.x (mgr 2021-07-31T04:21:59.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:21:59 smithi045 conmon[39718]: .24542) 6633 : cluster [DBG] pgmap v6650: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:21:59.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:59 smithi045 conmon[33655]: cluster 2021-07-31T04:21:58.012135+0000 mgr.x (mgr.24542) 6633 : 2021-07-31T04:21:59.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:21:59 smithi045 conmon[33655]: cluster [DBG] pgmap v6650: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:00.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:59 smithi154 conmon[34553]: cluster 2021-07-31T04:21:58.012135+0000 mgr.x (mgr.24542) 6633 : cluster [DBG] pgmap v6650: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:00.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:21:59 smithi154 conmon[34553]: 2021-07-31T04:22:01.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:01 smithi045 conmon[39718]: cluster 2021-07-31T04:22:00.012547+0000 mgr.x ( 2021-07-31T04:22:01.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:01 smithi045 conmon[39718]: mgr.24542) 6634 : cluster [DBG] pgmap v6651: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:01.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:01 smithi045 conmon[33655]: cluster 2021-07-31T04:22:00.012547+0000 mgr.x (mgr.24542) 6634 : cluster 2021-07-31T04:22:01.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:01 smithi045 conmon[33655]: [DBG] pgmap v6651: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:02.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:01 smithi154 conmon[34553]: cluster 2021-07-31T04:22:00.012547+0000 mgr.x (mgr.24542) 6634 : cluster [DBG] pgmap v6651: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:03.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:03 smithi045 conmon[33655]: cluster 2021-07-31T04:22:02.013114+0000 2021-07-31T04:22:03.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:03 smithi045 conmon[33655]: mgr.x (mgr.24542) 6635 : cluster [DBG] pgmap v6652: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:03.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:03 smithi045 conmon[39718]: cluster 2021-07-31T04:22:02.013114+0000 mgr.x (mgr.24542) 2021-07-31T04:22:03.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:03 smithi045 conmon[39718]: 6635 : cluster [DBG] pgmap v6652: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:04.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:03 smithi154 conmon[34553]: cluster 2021-07-31T04:22:02.013114+0000 mgr.x (mgr.24542) 6635 : cluster [DBG] pgmap v6652: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:05.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:05 smithi045 conmon[39718]: cluster 2021-07-31T04:22:04.013684+0000 mgr.x (mgr. 2021-07-31T04:22:05.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:05 smithi045 conmon[39718]: 24542) 6636 : cluster [DBG] pgmap v6653: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:05.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:05 smithi045 conmon[33655]: cluster 2021-07-31T04:22:04.013684+0000 mgr.x (mgr.24542 2021-07-31T04:22:05.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:05 smithi045 conmon[33655]: ) 6636 : cluster [DBG] pgmap v6653: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:05.976 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T04:22:06.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:05 smithi154 conmon[34553]: cluster 2021-07-31T04:22:04.013684+0000 mgr.x (mgr.24542) 6636 : cluster [DBG] pgmap v6653: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:06.255 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T04:22:06.255 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 2h ago 2h - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T04:22:06.255 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 2h ago 2h 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T04:22:06.255 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 2h ago 2h 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T04:22:06.255 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 2h ago 2h - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T04:22:06.256 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T04:22:06.256 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 2h ago 2h 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T04:22:06.256 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T04:22:06.256 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 2h ago 2h - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T04:22:06.256 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 2h ago 2h 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T04:22:06.257 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T04:22:06.257 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T04:22:06.257 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T04:22:06.257 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T04:22:06.257 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 2h ago 2h 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T04:22:06.258 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 2h ago 2h 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T04:22:06.258 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 2h ago 2h 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T04:22:06.258 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 2h ago 2h 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T04:22:06.258 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 2h ago 2h 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T04:22:06.592 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T04:22:06.592 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T04:22:06.592 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T04:22:06.593 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:22:06.593 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T04:22:06.593 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T04:22:06.593 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:22:06.593 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:22:06.594 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T04:22:06.594 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T04:22:06.594 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:22:06.594 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T04:22:06.594 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T04:22:06.595 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T04:22:06.595 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:22:06.595 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T04:22:06.595 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T04:22:07.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:07 smithi045 conmon[39718]: audit 2021-07-31T04:22:05 2021-07-31T04:22:07.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:07 smithi045 conmon[39718]: .964899+0000 mgr.x (mgr.24542) 6637 : audit [DBG] from='client.31387 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:22:07.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:07 smithi045 conmon[39718]: cluster 2021-07 2021-07-31T04:22:07.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:07 smithi045 conmon[39718]: -31T04:22:06.014202 2021-07-31T04:22:07.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:07 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 6638 2021-07-31T04:22:07.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:07 smithi045 conmon[39718]: : cluster [DBG] pgmap v6654: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:07.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:07 smithi045 conmon[39718]: audit 2021- 2021-07-31T04:22:07.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:07 smithi045 conmon[39718]: 07-31T04:22:06.247514+0000 2021-07-31T04:22:07.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:07 smithi045 conmon[39718]: mgr.x (mgr.24542) 6639 : audit 2021-07-31T04:22:07.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:07 smithi045 conmon[39718]: [DBG] from='client.31393 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:22:07.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:07 smithi045 conmon[39718]: audit 2021-07-31T04:22:06.593317 2021-07-31T04:22:07.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:07 smithi045 conmon[39718]: +0000 mon.a (mon.0) 3467 : audit [DBG] from='client.? 172.21.15.45:0/3443297726' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:22:07.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:07 smithi045 conmon[33655]: audit 2021-07-31T04:22:05.964899+0000 mgr.x (mgr.24542) 6637 2021-07-31T04:22:07.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:07 smithi045 conmon[33655]: : audit [DBG] from='client.31387 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:22:07.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:07 smithi045 conmon[33655]: cluster 2021-07-31T04:22:06.014202+0000 mgr.x (mgr.24542) 6638 : cluster [DBG] pgmap v6654: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:07.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:07 smithi045 conmon[33655]: audit 2021-07-31T04:22:07.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:07 smithi045 conmon[33655]: 2021-07-31T04:22:06.247514+0000 mgr.x (mgr.24542) 6639 : audit [DBG] from='client.31393 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:22:07.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:07 smithi045 conmon[33655]: audit 2021-07-31T04:22:07.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:07 smithi045 conmon[33655]: 2021-07-31T04:22:06 2021-07-31T04:22:07.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:07 smithi045 conmon[33655]: .593317+0000 mon.a (mon.0) 3467 2021-07-31T04:22:07.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:07 smithi045 conmon[33655]: : audit [DBG] from='client.? 172.21.15.45:0/3443297726' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:22:08.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:07 smithi154 conmon[34553]: audit 2021-07-31T04:22:05.964899+0000 mgr.x (mgr.24542) 6637 : audit [DBG] from='client.31387 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:22:08.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:07 smithi154 conmon[34553]: cluster 2021-07-31T04:22:06.014202+0000 mgr.x (mgr.24542) 6638 : cluster [DBG] pgmap v6654: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:08.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:07 smithi154 conmon[34553]: audit 2021-07-31T04:22:06.247514+0000 mgr.x (mgr.24542) 6639 : audit [DBG] from='client.31393 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:22:08.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:07 smithi154 conmon[34553]: audit 2021-07-31T04:22:06.593317+0000 mon.a (mon.0) 3467 : audit [DBG] from='client.? 172.21.15.45:0/3443297726' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:22:10.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:09 smithi154 conmon[34553]: cluster 2021-07-31T04:22:08.014569+0000 mgr.x (mgr.24542) 6640 : cluster [DBG] pgmap v6655: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:10.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:09 smithi154 conmon[34553]: 2021-07-31T04:22:10.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:09 smithi045 conmon[33655]: cluster 2021-07-31T04:22:08.014569+0000 2021-07-31T04:22:10.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:09 smithi045 conmon[33655]: mgr.x (mgr.24542) 6640 : cluster [DBG] pgmap v6655: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:10.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:09 smithi045 conmon[39718]: cluster 2021-07-31T04:22:08.014569+0000 mgr.x (mgr.24542 2021-07-31T04:22:10.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:09 smithi045 conmon[39718]: ) 6640 : cluster [DBG] pgmap v6655: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:12.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:11 smithi154 conmon[34553]: cluster 2021-07-31T04:22:10.015118+0000 mgr.x (mgr.24542) 6641 : cluster [DBG] pgmap v6656: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:12.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:11 smithi045 conmon[39718]: cluster 2021-07-31T04:22:10.015118+0000 mgr.x ( 2021-07-31T04:22:12.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:11 smithi045 conmon[39718]: mgr.24542) 6641 : cluster [DBG] pgmap v6656: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:12.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:11 smithi045 conmon[33655]: cluster 2021-07-31T04:22:10.015118+0000 mgr.x (mgr.24542) 6641 : cluster [DBG] 2021-07-31T04:22:12.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:11 smithi045 conmon[33655]: pgmap v6656: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:14.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:13 smithi154 conmon[34553]: cluster 2021-07-31T04:22:12.015475+0000 mgr.x (mgr.24542) 6642 : cluster [DBG] pgmap v6657: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:14.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:13 smithi045 conmon[39718]: cluster 2021-07-31T04:22:12.015475+0000 mgr.x ( 2021-07-31T04:22:14.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:13 smithi045 conmon[39718]: mgr.24542) 6642 : cluster [DBG] pgmap v6657: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:14.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:13 smithi045 conmon[33655]: cluster 2021-07-31T04:22:12.015475+0000 mgr.x (mgr.24542) 6642 : cluster [DBG] pgmap v6657: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:16.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:15 smithi154 conmon[34553]: cluster 2021-07-31T04:22:14.015959+0000 mgr.x (mgr.24542) 6643 : cluster [DBG] pgmap v6658: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:16.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:15 smithi045 conmon[39718]: cluster 2021-07-31T04:22:14.015959+0000 mgr.x ( 2021-07-31T04:22:16.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:15 smithi045 conmon[39718]: mgr.24542) 6643 : cluster [DBG] pgmap v6658: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:16.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:15 smithi045 conmon[33655]: cluster 2021-07-31T04:22:14.015959+0000 mgr.x ( 2021-07-31T04:22:16.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:15 smithi045 conmon[33655]: mgr.24542) 6643 : cluster [DBG] pgmap v6658: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:18.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:17 smithi154 conmon[34553]: cluster 2021-07-31T04:22:16.016374+0000 mgr.x (mgr.24542) 6644 : cluster [DBG] pgmap v6659: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:18.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:17 smithi045 conmon[33655]: cluster 2021-07-31T04:22:16.016374 2021-07-31T04:22:18.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:17 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 6644 : cluster [DBG] pgmap v6659: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:18.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:17 smithi045 conmon[39718]: cluster 2021-07-31T04:22:16.016374+0000 mgr.x (mgr.24542) 6644 : cluster [DBG] 2021-07-31T04:22:18.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:17 smithi045 conmon[39718]: pgmap v6659: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:20.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:19 smithi154 conmon[34553]: cluster 2021-07-31T04:22:18.016828+0000 mgr.x (mgr.24542) 6645 : cluster [DBG] pgmap v6660: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:20.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:19 smithi045 conmon[39718]: cluster 2021-07-31T04:22:18.016828+0000 mgr.x (mgr 2021-07-31T04:22:20.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:19 smithi045 conmon[39718]: .24542) 6645 : cluster [DBG] pgmap v6660: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:20.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:19 smithi045 conmon[33655]: cluster 2021-07-31T04:22:18.016828+0000 mgr.x (mgr.24542) 6645 : cluster [DBG] pgmap v6660: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:22.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:21 smithi154 conmon[34553]: cluster 2021-07-31T04:22:20.017239+0000 mgr.x (mgr.24542) 6646 : cluster [DBG] pgmap v6661: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:22.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:21 smithi045 conmon[39718]: cluster 2021-07-31T04:22:20.017239+0000 mgr.x 2021-07-31T04:22:22.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:21 smithi045 conmon[39718]: (mgr.24542) 6646 : cluster [DBG] pgmap v6661: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:22.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:21 smithi045 conmon[33655]: cluster 2021-07-31T04:22:20.017239+0000 mgr.x (mgr.24542) 6646 : cluster 2021-07-31T04:22:22.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:21 smithi045 conmon[33655]: [DBG] pgmap v6661: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:23.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:22 smithi154 conmon[34553]: cluster 2021-07-31T04:22:22.017684+0000 mgr.x (mgr.24542) 2021-07-31T04:22:23.002 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:22 smithi154 conmon[34553]: 6647 : cluster [DBG] pgmap v6662: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:23.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:22 smithi045 conmon[33655]: cluster 2021-07-31T04:22:22.017684+0000 mgr.x (mgr.24542) 6647 : cluster [DBG] pgmap v6662: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:23.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:22 smithi045 conmon[33655]: 2021-07-31T04:22:23.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:22 smithi045 conmon[39718]: cluster 2021-07-31T04:22:22.017684+0000 mgr.x (mgr.24542) 2021-07-31T04:22:23.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:22 smithi045 conmon[39718]: 6647 : cluster [DBG] pgmap v6662: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:25.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:25 smithi154 conmon[34553]: cluster 2021-07-31T04:22:24.018142+0000 mgr.x (mgr.24542) 6648 : cluster [DBG] 2021-07-31T04:22:25.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:25 smithi154 conmon[34553]: pgmap v6663: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:25.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:25 smithi045 conmon[39718]: cluster 2021-07-31T04:22:24.018142+0000 mgr.x 2021-07-31T04:22:25.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:25 smithi045 conmon[39718]: (mgr.24542) 6648 : cluster [DBG] pgmap v6663: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:25.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:25 smithi045 conmon[33655]: cluster 2021-07-31T04:22:24.018142+0000 mgr.x (mgr.24542) 6648 : cluster [DBG] pgmap v6663: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:27.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:27 smithi154 conmon[34553]: cluster 2021-07-31T04:22:26.018509+0000 mgr.x (mgr.24542) 6649 : cluster 2021-07-31T04:22:27.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:27 smithi154 conmon[34553]: [DBG] pgmap v6664: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:27.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:27 smithi045 conmon[33655]: cluster 2021-07-31T04:22:26.018509+0000 mgr.x (mgr.24542) 6649 : cluster 2021-07-31T04:22:27.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:27 smithi045 conmon[33655]: [DBG] pgmap v6664: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:27.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:27 smithi045 conmon[39718]: cluster 2021-07-31T04:22:26. 2021-07-31T04:22:27.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:27 smithi045 conmon[39718]: 018509+0000 mgr.x (mgr.24542) 6649 : cluster [DBG] pgmap v6664: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:29.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:29 smithi154 conmon[34553]: cluster 2021-07-31T04:22:28.018970+0000 mgr.x (mgr.24542) 6650 : cluster [DBG] 2021-07-31T04:22:29.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:29 smithi154 conmon[34553]: pgmap v6665: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:29.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:29 smithi154 conmon[34553]: audit 2021-07-31T04:22:28.546087+0000 mon.b (mon.2) 850 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:22:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:29 smithi154 conmon[34553]: audit 2021-07- 2021-07-31T04:22:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:29 smithi154 conmon[34553]: 31T04:22:28.546472+0000 mon.b (mon.2) 851 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:22:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:29 smithi154 conmon[34553]: audit 2021-07-31T04:22:28. 2021-07-31T04:22:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:29 smithi154 conmon[34553]: 547046+0000 mon.a (mon.0) 3468 : audit [INF] 2021-07-31T04:22:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:29 smithi154 conmon[34553]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:22:29.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:29 smithi154 conmon[34553]: audit 2021-07-31T04 2021-07-31T04:22:29.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:29 smithi154 conmon[34553]: :22:28.547474+0000 mon.a (mon.0) 3469 2021-07-31T04:22:29.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:29 smithi154 conmon[34553]: : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:22:29.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:29 smithi045 conmon[33655]: cluster 2021-07-31T04:22:28. 2021-07-31T04:22:29.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:29 smithi045 conmon[33655]: 018970+0000 mgr.x (mgr.24542) 6650 : cluster [DBG] pgmap v6665: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:29.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:29 smithi045 conmon[33655]: audit 2021-07-31T04: 2021-07-31T04:22:29.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:29 smithi045 conmon[33655]: 22:28.546087+0000 mon.b (mon.2) 850 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:22:29.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:29 smithi045 conmon[33655]: audit 2021-07-31T04:22:29.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:29 smithi045 conmon[33655]: 2021-07-31T04:22:28.546472+0000 mon.b (mon.2) 851 : audit [INF] 2021-07-31T04:22:29.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:29 smithi045 conmon[33655]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:22:29.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:29 smithi045 conmon[33655]: audit 2021-07-31T04:22:29.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:29 smithi045 conmon[33655]: 2021-07-31T04:22:28.547046+0000 mon.a (mon.0) 3468 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:22:29.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:29 smithi045 conmon[33655]: audit 2021-07-31T04 2021-07-31T04:22:29.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:29 smithi045 conmon[33655]: :22:28.547474+0000 mon.a (mon.0) 3469 : audit [INF] 2021-07-31T04:22:29.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:29 smithi045 conmon[33655]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:22:29.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:29 smithi045 conmon[39718]: cluster 2021-07-31T04:22:28.018970+0000 mgr.x (mgr.24542 2021-07-31T04:22:29.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:29 smithi045 conmon[39718]: ) 6650 : cluster [DBG] pgmap v6665: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:29.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:29 smithi045 conmon[39718]: audit 2021-07-31T04: 2021-07-31T04:22:29.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:29 smithi045 conmon[39718]: 22:28.546087+0000 mon.b (mon.2) 850 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:22:29.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:29 smithi045 conmon[39718]: audit 2021-07-31T04: 2021-07-31T04:22:29.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:29 smithi045 conmon[39718]: 22:28.546472+0000 mon.b (mon.2) 851 2021-07-31T04:22:29.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:29 smithi045 conmon[39718]: : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:22:29.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:29 smithi045 conmon[39718]: audit 2021-07-31T 2021-07-31T04:22:29.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:29 smithi045 conmon[39718]: 04:22:28.547046+0000 mon.a (mon.0) 3468 : audit 2021-07-31T04:22:29.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:29 smithi045 conmon[39718]: [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:22:29.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:29 smithi045 conmon[39718]: audit 2021-07-31T04:22:29.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:29 smithi045 conmon[39718]: 2021-07-31T04:22:28.547474+0000 mon.a (mon 2021-07-31T04:22:29.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:29 smithi045 conmon[39718]: .0) 3469 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:22:31.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:31 smithi154 conmon[34553]: cluster 2021-07-31T04:22:30.019496+0000 mgr.x (mgr.24542) 6651 2021-07-31T04:22:31.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:31 smithi154 conmon[34553]: : cluster [DBG] pgmap v6666: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:31.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:31 smithi045 conmon[39718]: cluster 2021-07-31T04:22:30.019496 2021-07-31T04:22:31.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:31 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 6651 : cluster [DBG] pgmap v6666: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:31.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:31 smithi045 conmon[33655]: cluster 2021-07-31T04:22:30.019496+0000 mgr.x (mgr.24542) 6651 : cluster [DBG] pgmap v6666: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:33.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:33 smithi154 conmon[34553]: cluster 2021-07-31T04:22:32.019883+0000 mgr.x (mgr.24542) 2021-07-31T04:22:33.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:33 smithi154 conmon[34553]: 6652 : cluster [DBG] pgmap v6667: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:33.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:33 smithi045 conmon[39718]: cluster 2021-07-31T04:22:32.019883 2021-07-31T04:22:33.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:33 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 6652 : cluster [DBG] pgmap v6667: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:33.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:33 smithi045 conmon[33655]: cluster 2021-07-31T04:22:32.019883 2021-07-31T04:22:33.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:33 smithi045 conmon[33655]: +0000 mgr.x (mgr.24542) 6652 : cluster [DBG] pgmap v6667: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:35.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:35 smithi154 conmon[34553]: cluster 2021-07-31T04:22:34.020422+0000 mgr.x (mgr.24542) 6653 : cluster [DBG] 2021-07-31T04:22:35.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:35 smithi154 conmon[34553]: pgmap v6668: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:35.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:35 smithi045 conmon[33655]: cluster 2021-07-31T04:22:34.020422+0000 mgr.x (mgr.24542) 6653 : cluster [DBG] pgmap v6668: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:35.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:35 smithi045 conmon[39718]: cluster 2021-07-31T04:22:34.020422+0000 mgr.x (mgr 2021-07-31T04:22:35.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:35 smithi045 conmon[39718]: .24542) 6653 : cluster [DBG] pgmap v6668: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:36.904 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T04:22:37.184 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T04:22:37.184 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 2h ago 2h - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T04:22:37.184 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 2h ago 2h 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T04:22:37.185 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 2h ago 2h 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T04:22:37.185 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 2h ago 2h - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T04:22:37.185 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T04:22:37.185 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 2h ago 2h 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T04:22:37.185 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T04:22:37.186 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 2h ago 2h - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T04:22:37.186 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 2h ago 2h 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T04:22:37.186 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T04:22:37.186 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T04:22:37.186 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T04:22:37.187 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T04:22:37.187 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 2h ago 2h 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T04:22:37.187 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 2h ago 2h 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T04:22:37.187 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 2h ago 2h 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T04:22:37.187 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 2h ago 2h 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T04:22:37.188 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 2h ago 2h 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T04:22:37.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:37 smithi154 conmon[34553]: cluster 2021-07-31T04:22:36.020867+0000 mgr.x (mgr.24542) 6654 : cluster [DBG] 2021-07-31T04:22:37.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:37 smithi154 conmon[34553]: pgmap v6669: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:37.518 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T04:22:37.519 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T04:22:37.519 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T04:22:37.519 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:22:37.519 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T04:22:37.519 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T04:22:37.520 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:22:37.520 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:22:37.520 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T04:22:37.520 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T04:22:37.520 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:22:37.521 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T04:22:37.521 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T04:22:37.521 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T04:22:37.521 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:22:37.521 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T04:22:37.522 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T04:22:37.524 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:37 smithi045 conmon[33655]: cluster 2021-07-31T04:22:36 2021-07-31T04:22:37.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:37 smithi045 conmon[33655]: .020867+0000 mgr.x (mgr.24542) 6654 : cluster [DBG] pgmap v6669: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:37.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:37 smithi045 conmon[39718]: cluster 2021-07-31T04:22:36.020867+0000 2021-07-31T04:22:37.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:37 smithi045 conmon[39718]: mgr.x (mgr.24542) 6654 : cluster [DBG] pgmap v6669: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:38.442 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[34553]: audit 2021-07-31T04:22:36.892953+0000 mgr.x (mgr.24542) 6655 : 2021-07-31T04:22:38.442 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[34553]: audit [DBG] from='client.21468 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:22:38.443 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[34553]: audit 2021-07-31T04:22:37.176324+0000 mgr.x (mgr.24542) 6656 : audit [DBG] from='client.31411 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:22:38.443 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[34553]: audit 2021-07-31T04:22:37.519537+0000 mon.a (mon.0) 3470 : audit 2021-07-31T04:22:38.443 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[34553]: [DBG] from='client.? 172.21.15.45:0/2012734963' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:22:38.444 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:38 smithi045 conmon[33655]: audit 2021-07-31T04:22:36.892953+0000 mgr.x (mgr.24542) 6655 2021-07-31T04:22:38.444 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:38 smithi045 conmon[33655]: : audit [DBG] from='client.21468 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:22:38.444 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:38 smithi045 conmon[33655]: audit 2021-07-31T04:22:37.176324+0000 mgr.x (mgr.24542) 6656 : audit [DBG] from='client.31411 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:22:38.444 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:38 smithi045 conmon[33655]: audit 2021-07-31T04:22:37. 2021-07-31T04:22:38.445 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:38 smithi045 conmon[33655]: 519537+0000 mon.a (mon.0) 3470 : audit [DBG] from='client.? 172.21.15.45:0/2012734963' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:22:38.445 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:38 smithi045 conmon[39718]: audit 2021-07-31T04:22:36 2021-07-31T04:22:38.445 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:38 smithi045 conmon[39718]: .892953+0000 mgr.x (mgr.24542) 6655 : audit [DBG] from='client.21468 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:22:38.445 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:38 smithi045 conmon[39718]: audit 2021-07-31T04:22: 2021-07-31T04:22:38.446 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:38 smithi045 conmon[39718]: 37.176324+0000 mgr.x (mgr.24542) 2021-07-31T04:22:38.446 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:38 smithi045 conmon[39718]: 6656 : audit [DBG] from='client.31411 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:22:38.446 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:38 smithi045 conmon[39718]: audit 2021- 2021-07-31T04:22:38.446 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:38 smithi045 conmon[39718]: 07-31T04:22:37.519537 2021-07-31T04:22:38.447 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:38 smithi045 conmon[39718]: +0000 mon.a (mon.0) 3470 : audit 2021-07-31T04:22:38.447 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:38 smithi045 conmon[39718]: [DBG] from='client.? 172.21.15.45:0/2012734963' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:22:38.752 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: debug 2021-07-31T04:22:38.461+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:22:38.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:22:38.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:22:38.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: yield 2021-07-31T04:22:38.779 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:22:38.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:22:38.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:22:38.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:22:38.780 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:22:38.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:22:38.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:22:38.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:22:38.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:22:38.781 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: task.result() 2021-07-31T04:22:38.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:22:38.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: debug 2021-07-31T04:22:38.472+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:22:38.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:22:38.782 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:22:38.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: yield 2021-07-31T04:22:38.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:22:38.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:22:38.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:22:38.783 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:22:38.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:22:38.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:22:38.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:22:38.784 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:22:38.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:22:38.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: task.result() 2021-07-31T04:22:38.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:22:38.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: debug 2021-07-31T04:22:38.497+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:22:38.785 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:22:38.786 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:22:38.786 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: yield 2021-07-31T04:22:38.786 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:22:38.786 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:22:38.786 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:22:38.787 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:22:38.787 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:22:38.787 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:22:38.787 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:22:38.788 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:22:38.788 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:22:38.788 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: task.result() 2021-07-31T04:22:38.788 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:22:38.788 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: debug 2021-07-31T04:22:38.498+0000 7fcb93961700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:22:38.789 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:22:38.789 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:22:38.789 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: yield 2021-07-31T04:22:38.789 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:22:38.789 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:22:38.790 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:22:38.790 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:22:38.790 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:22:38.790 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:22:38.791 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:22:38.791 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:22:38.791 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:22:38.791 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: task.result() 2021-07-31T04:22:38.791 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:22:38.792 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: 2021-07-31T04:22:38.792 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T04:22:38.792 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: 2021-07-31T04:22:38.792 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:22:38.792 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:22:38.793 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: return f(*arg) 2021-07-31T04:22:38.793 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:22:38.793 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:22:38.793 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:22:38.793 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:22:38.794 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:22:38.794 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:22:38.794 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:22:38.794 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T04:22:38.794 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:22:38.795 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: raise self._exception 2021-07-31T04:22:38.795 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:22:38.795 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:22:38.795 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:22:38.795 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T04:22:38.796 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:22:38.796 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: raise 2021-07-31T04:22:38.797 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:22:38.797 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T04:22:38.797 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:22:38.797 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:22:38.798 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:22:38.798 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: debug 2021-07-31T04:22:38.506+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:22:38.798 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:22:38.798 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:22:38.798 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: yield 2021-07-31T04:22:38.799 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:22:38.799 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:22:38.799 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:22:38.799 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:22:38.800 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:22:38.800 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:22:38.800 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:22:38.800 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:22:38.800 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:22:38.801 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: task.result() 2021-07-31T04:22:38.801 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:22:38.801 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: debug 2021-07-31T04:22:38.507+0000 7fcb94162700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:22:38.801 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:22:38.801 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:22:38.802 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: yield 2021-07-31T04:22:38.802 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:22:38.802 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:22:38.802 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:22:38.802 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:22:38.803 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:22:38.803 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:22:38.803 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:22:38.803 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:22:38.804 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:22:38.804 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: task.result() 2021-07-31T04:22:38.804 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:22:38.804 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: 2021-07-31T04:22:38.804 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T04:22:38.805 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: 2021-07-31T04:22:38.805 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:22:38.805 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:22:38.805 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: return f(*arg) 2021-07-31T04:22:38.805 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:22:38.806 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:22:38.806 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:22:38.806 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:22:38.806 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:22:38.807 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:22:38.807 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:22:38.807 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T04:22:38.807 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:22:38.807 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: raise self._exception 2021-07-31T04:22:38.808 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:22:38.808 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:22:38.808 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:22:38.808 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T04:22:38.809 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:22:38.809 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: raise 2021-07-31T04:22:38.809 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:22:38.809 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T04:22:38.809 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:22:38.810 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:22:38.810 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:22:38 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:22:39.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: cluster 2021-07-31T04:22:38.021319+0000 mgr.x (mgr.24542) 6657 : cluster [DBG] 2021-07-31T04:22:39.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: pgmap v6670: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:39.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: audit 2021-07-31T04:22:38.415990+0000 mon.b (mon.2) 852 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T04:22:39.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: audit 2021-07-31T04:22:38.417416+0000 mon.b (mon.2) 853 : audit [DBG] 2021-07-31T04:22:39.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T04:22:39.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: audit 2021-07-31T04:22: 2021-07-31T04:22:39.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: 38.418332+0000 mon.b (mon.2) 854 : audit 2021-07-31T04:22:39.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T04:22:39.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: cephadm 2021-07-31 2021-07-31T04:22:39.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: T04:22:38.419120+0000 mgr.x (mgr.24542 2021-07-31T04:22:39.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: ) 6658 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T04:22:39.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:22:39.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: 2021-07-31T04:22:38.462312+0000 mgr.x ( 2021-07-31T04:22:39.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: mgr.24542) 6659 : cephadm [ERR] 2021-07-31T04:22:39.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:22:39.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:22:39.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: yield 2021-07-31T04:22:39.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:22:39.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:22:39.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:22:39.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:22:39.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:22:39.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:22:39.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:22:39.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:22:39.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:22:39.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: task.result() 2021-07-31T04:22:39.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:22:39.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:22:38.462723+0000 mgr.x (mgr.24542) 6660 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T04:22:39.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:22:38.473654+0000 2021-07-31T04:22:39.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: mgr.x (mgr.24542) 6661 : cephadm [ERR] 2021-07-31T04:22:39.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:22:39.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:22:39.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: yield 2021-07-31T04:22:39.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:22:39.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:22:39.512 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:22:39.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:22:39.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:22:39.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:22:39.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:22:39.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:22:39.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:22:39.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: task.result() 2021-07-31T04:22:39.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:22:39.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:22:38.474059+0000 mgr.x (mgr.24542) 6662 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T04:22:39.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:22:38.498741+0000 mgr.x (mgr. 2021-07-31T04:22:39.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: 24542) 6663 : cephadm [ERR] 2021-07-31T04:22:39.515 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:22:39.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:22:39.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: yield 2021-07-31T04:22:39.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:22:39.516 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:22:39.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:22:39.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:22:39.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:22:39.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:22:39.517 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:22:39.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:22:39.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:22:39.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: task.result() 2021-07-31T04:22:39.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:22:39.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:22:38.499171+0000 mgr.x (mgr.24542) 6664 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:22:39.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:22:39.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:22:39.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: yield 2021-07-31T04:22:39.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:22:39.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:22:39.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:22:39.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:22:39.520 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:22:39.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:22:39.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:22:39.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:22:39.521 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:22:39.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: task.result() 2021-07-31T04:22:39.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:22:39.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: 2021-07-31T04:22:39.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T04:22:39.522 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: 2021-07-31T04:22:39.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:22:39.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:22:39.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: return f(*arg) 2021-07-31T04:22:39.523 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:22:39.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:22:39.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:22:39.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:22:39.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:22:39.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:22:39.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:22:39.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T04:22:39.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:22:39.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: raise self._exception 2021-07-31T04:22:39.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:22:39.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:22:39.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:22:39.526 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T04:22:39.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:22:39.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: raise 2021-07-31T04:22:39.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:22:39.527 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T04:22:39.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:22:39.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:22:39.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:22:39.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:22:38.507640+0000 mgr.x (mgr.24542) 6665 : cephadm [ERR] 2021-07-31T04:22:39.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:22:39.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:22:39.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: yield 2021-07-31T04:22:39.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:22:39.529 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:22:39.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:22:39.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:22:39.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:22:39.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:22:39.530 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:22:39.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:22:39.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:22:39.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: task.result() 2021-07-31T04:22:39.531 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:22:39.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:22:38.508051+0000 mgr.x (mgr.24542) 6666 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:22:39.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:22:39.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:22:39.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: yield 2021-07-31T04:22:39.532 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:22:39.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:22:39.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:22:39.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:22:39.533 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:22:39.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:22:39.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:22:39.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:22:39.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:22:39.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: task.result() 2021-07-31T04:22:39.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:22:39.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: 2021-07-31T04:22:39.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T04:22:39.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: 2021-07-31T04:22:39.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:22:39.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:22:39.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: return f(*arg) 2021-07-31T04:22:39.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:22:39.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:22:39.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:22:39.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:22:39.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:22:39.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:22:39.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:22:39.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T04:22:39.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:22:39.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: raise self._exception 2021-07-31T04:22:39.538 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:22:39.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:22:39.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:22:39.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T04:22:39.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:22:39.539 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: raise 2021-07-31T04:22:39.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:22:39.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T04:22:39.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:22:39.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:22:39.541 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:39 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:22:39.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: cluster 2021-07-31T04: 2021-07-31T04:22:39.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: 22:38.021319+0000 mgr.x (mgr.24542) 6657 : cluster [DBG] pgmap v6670: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:39.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: audit 2021- 2021-07-31T04:22:39.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: 07-31T04:22:38.415990 2021-07-31T04:22:39.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: +0000 mon.b (mon.2) 2021-07-31T04:22:39.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: 852 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T04:22:39.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: audit 2021 2021-07-31T04:22:39.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: -07-31T04:22:38 2021-07-31T04:22:39.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: .417416+0000 mon.b (mon. 2021-07-31T04:22:39.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: 2) 853 : audit [DBG] 2021-07-31T04:22:39.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T04:22:39.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: audit 2021 2021-07-31T04:22:39.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: -07-31T04:22:38. 2021-07-31T04:22:39.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: 418332+0000 mon.b (mon.2 2021-07-31T04:22:39.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: ) 854 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T04:22:39.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: 2021-07-31T04:22:39.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: cephadm 2021 2021-07-31T04:22:39.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: -07-31T04:22:38. 2021-07-31T04:22:39.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: 419120+0000 mgr.x (mgr.24542 2021-07-31T04:22:39.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: ) 6658 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T04:22:39.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: 2021-07-31T04:22:39.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: cephadm 2021-07-31T04:22:39.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: 2021-07-31T04:22: 2021-07-31T04:22:39.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: 38.462312+0000 mgr.x ( 2021-07-31T04:22:39.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: mgr.24542) 6659 : cephadm 2021-07-31T04:22:39.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: [ERR] 2021-07-31T04:22:39.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:22:39.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:22:39.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: yield 2021-07-31T04:22:39.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:22:39.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:22:39.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:22:39.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:22:39.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:22:39.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:22:39.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:22:39.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:22:39.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:22:39.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: task.result() 2021-07-31T04:22:39.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:22:39.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: cephadm 2021-07-31T04:22:38.462723+0000 mgr.x (mgr.24542) 6660 : cephadm 2021-07-31T04:22:39.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T04:22:39.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: cephadm 2021 2021-07-31T04:22:39.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: -07-31T04:22: 2021-07-31T04:22:39.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: 38.473654+0000 mgr.x (mgr 2021-07-31T04:22:39.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: .24542) 6661 : cephadm [ERR] 2021-07-31T04:22:39.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: 2021-07-31T04:22:39.552 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: cluster 2021-07-31T04:22:38.021319+0000 mgr.x (mgr.24542) 6657 : 2021-07-31T04:22:39.552 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: cluster [DBG] pgmap v6670: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:39.553 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: audit 2021-07-31T04:22:38.415990+0000 mon.b (mon.2) 852 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T04:22:39.553 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: audit 2021-07-31T 2021-07-31T04:22:39.553 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: 04:22:38.417416+0000 mon.b (mon.2) 853 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T04:22:39.553 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: audit 2021-07-31T04:22:38.418332 2021-07-31T04:22:39.554 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: +0000 mon.b (mon.2) 854 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T04:22:39.554 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: cephadm 2021 2021-07-31T04:22:39.554 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: -07-31T04:22:38.419120+0000 mgr.x (mgr.24542) 2021-07-31T04:22:39.554 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: 6658 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T04:22:39.554 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: cephadm 2021- 2021-07-31T04:22:39.555 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: 07-31T04:22:38.462312+0000 2021-07-31T04:22:39.555 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: mgr.x (mgr.24542) 6659 : cephadm [ERR] 2021-07-31T04:22:39.555 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: 2021-07-31T04:22:39.555 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:22:39.556 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:22:39.556 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: yield 2021-07-31T04:22:39.556 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:22:39.556 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:22:39.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:22:39.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:22:39.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:22:39.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:22:39.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:22:39.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:22:39.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:22:39.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: task.result() 2021-07-31T04:22:39.558 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:22:39.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: cephadm 2021-07-31T04:22:39.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: 2021-07-31T04:22:38.462723+0000 mgr.x (mgr.24542) 6660 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T04:22:39.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: cephadm 2021-07-31T04:22:38.473654+0000 mgr.x (mgr 2021-07-31T04:22:39.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: .24542) 6661 : cephadm [ERR] 2021-07-31T04:22:39.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:22:39.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:22:39.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: yield 2021-07-31T04:22:39.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:22:39.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:22:39.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:22:39.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:22:39.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:22:39.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:22:39.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:22:39.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:22:39.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:22:39.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: task.result() 2021-07-31T04:22:39.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:22:39.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: cephadm 2021-07-31T04:22:38.474059+0000 mgr.x (mgr.24542) 6662 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T04:22:39.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: cephadm 2021-07-31T04:22 2021-07-31T04:22:39.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: :38.498741+0000 mgr.x (mgr.24542) 6663 : cephadm 2021-07-31T04:22:39.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: [ERR] 2021-07-31T04:22:39.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:22:39.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:22:39.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: yield 2021-07-31T04:22:39.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:22:39.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:22:39.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:22:39.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:22:39.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:22:39.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:22:39.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:22:39.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:22:39.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:22:39.566 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: task.result() 2021-07-31T04:22:39.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:22:39.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: cephadm 2021-07-31T04:22:38.499171+0000 mgr.x (mgr.24542) 6664 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:22:39.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:22:39.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:22:39.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: yield 2021-07-31T04:22:39.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:22:39.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:22:39.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:22:39.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:22:39.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:22:39.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:22:39.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:22:39.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:22:39.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:22:39.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: task.result() 2021-07-31T04:22:39.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:22:39.570 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:22:39.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: yield 2021-07-31T04:22:39.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:22:39.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:22:39.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:22:39.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:22:39.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:22:39.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:22:39.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:22:39.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:22:39.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:22:39.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: task.result() 2021-07-31T04:22:39.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:22:39.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: cephadm 2021-07-31T04:22:38.474059+0000 mgr.x (mgr.24542) 6662 : cephadm [INF] 2021-07-31T04:22:39.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T04:22:39.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: cephadm 2021-07-31T04:22:39.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: 2021-07-31T04:22: 2021-07-31T04:22:39.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: 38.498741+0000 mgr.x (mgr 2021-07-31T04:22:39.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: .24542) 6663 : cephadm [ERR] 2021-07-31T04:22:39.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: 2021-07-31T04:22:39.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:22:39.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:22:39.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: yield 2021-07-31T04:22:39.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:22:39.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:22:39.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:22:39.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:22:39.576 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:22:39.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:22:39.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:22:39.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:22:39.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:22:39.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: task.result() 2021-07-31T04:22:39.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:22:39.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: cephadm 2021-07-31T04:22:38.499171+0000 mgr.x (mgr.24542) 6664 : 2021-07-31T04:22:39.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:22:39.578 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:22:39.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:22:39.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: yield 2021-07-31T04:22:39.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:22:39.579 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:22:39.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:22:39.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: 2021-07-31T04:22:39.580 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T04:22:39.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: 2021-07-31T04:22:39.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:22:39.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:22:39.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: return f(*arg) 2021-07-31T04:22:39.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:22:39.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:22:39.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:22:39.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:22:39.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:22:39.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:22:39.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:22:39.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T04:22:39.583 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:22:39.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: raise self._exception 2021-07-31T04:22:39.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:22:39.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:22:39.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:22:39.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T04:22:39.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:22:39.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: raise 2021-07-31T04:22:39.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:22:39.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T04:22:39.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:22:39.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:22:39.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:22:39.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: cephadm 2021-07-31T04 2021-07-31T04:22:39.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: :22:38.507640+0000 mgr.x (mgr.24542) 6665 : cephadm [ERR] 2021-07-31T04:22:39.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:22:39.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:22:39.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: yield 2021-07-31T04:22:39.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:22:39.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:22:39.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:22:39.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:22:39.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:22:39.588 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:22:39.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:22:39.589 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:22:39.589 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:22:39.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:22:39.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:22:39.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:22:39.590 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:22:39.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:22:39.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:22:39.591 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: task.result() 2021-07-31T04:22:39.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:22:39.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: 2021-07-31T04:22:39.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T04:22:39.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: 2021-07-31T04:22:39.592 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:22:39.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:22:39.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: return f(*arg) 2021-07-31T04:22:39.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:22:39.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:22:39.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:22:39.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:22:39.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:22:39.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:22:39.594 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:22:39.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T04:22:39.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:22:39.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: raise self._exception 2021-07-31T04:22:39.595 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:22:39.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:22:39.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:22:39.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T04:22:39.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:22:39.596 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: raise 2021-07-31T04:22:39.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:22:39.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T04:22:39.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:22:39.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:22:39.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:22:39.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: cephadm 2021-07-31T04:22:38.507640+0000 mgr.x (mgr.24542) 6665 : cephadm [ERR] 2021-07-31T04:22:39.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:22:39.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:22:39.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: yield 2021-07-31T04:22:39.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:22:39.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:22:39.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: task.result() 2021-07-31T04:22:39.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:22:39.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: cephadm 2021-07-31T04:22:38.508051+0000 mgr.x (mgr.24542) 6666 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:22:39.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:22:39.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:22:39.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: yield 2021-07-31T04:22:39.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:22:39.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:22:39.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:22:39.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:22:39.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:22:39.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:22:39.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:22:39.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:22:39.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:22:39.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: task.result() 2021-07-31T04:22:39.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:22:39.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: 2021-07-31T04:22:39.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T04:22:39.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: 2021-07-31T04:22:39.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:22:39.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:22:39.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: return f(*arg) 2021-07-31T04:22:39.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:22:39.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:22:39.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:22:39.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:22:39.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:22:39.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:22:39.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:22:39.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T04:22:39.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:22:39.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: raise self._exception 2021-07-31T04:22:39.608 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:22:39.608 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:22:39.608 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:22:39.608 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T04:22:39.608 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:22:39.609 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: raise 2021-07-31T04:22:39.609 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:22:39.609 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T04:22:39.609 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:22:39.610 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:22:39.610 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:22:39.611 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:22:39.611 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:22:39.611 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:22:39.611 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:22:39.612 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:22:39.612 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:22:39.612 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:22:39.612 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:22:39.612 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: task.result() 2021-07-31T04:22:39.613 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:22:39.613 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: cephadm 2021-07-31T04:22:38.508051+0000 mgr.x (mgr.24542) 6666 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:22:39.613 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:22:39.613 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:22:39.614 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: yield 2021-07-31T04:22:39.614 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:22:39.614 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:22:39.614 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:22:39.614 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:22:39.615 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:22:39.615 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:22:39.615 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:22:39.615 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:22:39.616 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:22:39.616 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: task.result() 2021-07-31T04:22:39.616 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:22:39.616 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: 2021-07-31T04:22:39.616 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T04:22:39.617 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: 2021-07-31T04:22:39.617 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:22:39.617 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:22:39.617 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: return f(*arg) 2021-07-31T04:22:39.617 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:22:39.618 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:22:39.618 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:22:39.618 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:22:39.618 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:22:39.619 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:22:39.619 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:22:39.619 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T04:22:39.619 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:22:39.619 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: raise self._exception 2021-07-31T04:22:39.620 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:22:39.620 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:22:39.620 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:22:39.620 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T04:22:39.621 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:22:39.621 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: raise 2021-07-31T04:22:39.621 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:22:39.621 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T04:22:39.622 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:22:39.622 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:22:39.622 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:39 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:22:41.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:41 smithi154 conmon[34553]: cluster 2021-07-31T04:22:40.021973+0000 mgr.x (mgr.24542) 6667 2021-07-31T04:22:41.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:41 smithi154 conmon[34553]: : cluster [DBG] pgmap v6671: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:41.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:41 smithi045 conmon[39718]: cluster 2021-07-31T04:22:40.021973+0000 mgr.x ( 2021-07-31T04:22:41.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:41 smithi045 conmon[39718]: mgr.24542) 6667 : cluster [DBG] pgmap v6671: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:41.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:41 smithi045 conmon[33655]: cluster 2021-07-31T04:22:40.021973+0000 mgr.x (mgr.24542) 6667 : cluster [DBG] 2021-07-31T04:22:41.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:41 smithi045 conmon[33655]: pgmap v6671: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:43.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:43 smithi154 conmon[34553]: cluster 2021-07-31T04:22:42.022476+0000 mgr.x (mgr.24542) 6668 : cluster 2021-07-31T04:22:43.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:43 smithi154 conmon[34553]: [DBG] pgmap v6672: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:43.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:43 smithi045 conmon[39718]: cluster 2021-07-31T04:22:42. 2021-07-31T04:22:43.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:43 smithi045 conmon[39718]: 022476+0000 mgr.x (mgr.24542) 6668 : cluster [DBG] pgmap v6672: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:43.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:43 smithi045 conmon[33655]: cluster 2021-07-31T04:22:42.022476+0000 mgr.x (mgr.24542) 6668 : cluster [DBG] pgmap v6672: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:45.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:45 smithi154 conmon[34553]: cluster 2021-07-31T04:22:44.023114+0000 mgr.x (mgr.24542) 6669 : cluster [DBG] 2021-07-31T04:22:45.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:45 smithi154 conmon[34553]: pgmap v6673: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:45.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:45 smithi045 conmon[39718]: cluster 2021-07-31T04:22:44.023114+0000 mgr.x (mgr.24542) 6669 : cluster [DBG] pgmap v6673: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:45.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:45 smithi045 conmon[33655]: cluster 2021-07-31T04:22:44.023114+0000 mgr.x (mgr.24542) 6669 : cluster [DBG] 2021-07-31T04:22:45.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:45 smithi045 conmon[33655]: pgmap v6673: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:47.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:47 smithi154 conmon[34553]: cluster 2021-07-31T04:22:46.023633+0000 mgr.x (mgr.24542) 6670 : cluster 2021-07-31T04:22:47.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:47 smithi154 conmon[34553]: [DBG] pgmap v6674: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:47.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:47 smithi045 conmon[39718]: cluster 2021-07-31T04:22:46.023633+0000 mgr.x 2021-07-31T04:22:47.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:47 smithi045 conmon[39718]: (mgr.24542) 6670 : cluster [DBG] pgmap v6674: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:47.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:47 smithi045 conmon[33655]: cluster 2021-07-31T04:22:46.023633+0000 mgr.x (mgr.24542) 6670 : cluster [DBG] pgmap v6674: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:49.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:49 smithi154 conmon[34553]: cluster 2021-07-31T04:22:48.024099+0000 mgr.x (mgr.24542) 6671 : cluster 2021-07-31T04:22:49.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:49 smithi154 conmon[34553]: [DBG] pgmap v6675: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:49.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:49 smithi045 conmon[39718]: cluster 2021-07-31T04:22:48.024099+0000 mgr.x (mgr.24542) 6671 : cluster [DBG] pgmap v6675: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:49.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:49 smithi045 conmon[33655]: cluster 2021-07-31T04:22:48.024099+0000 mgr.x (mgr.24542) 2021-07-31T04:22:49.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:49 smithi045 conmon[33655]: 6671 : cluster [DBG] pgmap v6675: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:51.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:51 smithi154 conmon[34553]: cluster 2021-07-31T04:22:50.024591+0000 mgr.x (mgr.24542) 6672 : cluster 2021-07-31T04:22:51.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:51 smithi154 conmon[34553]: [DBG] pgmap v6676: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:51.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:51 smithi045 conmon[39718]: cluster 2021-07-31T04:22:50.024591+0000 mgr.x (mgr.24542) 6672 : cluster [DBG] pgmap v6676: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:51.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:51 smithi045 conmon[33655]: cluster 2021-07-31T04:22:50.024591+0000 mgr.x (mgr. 2021-07-31T04:22:51.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:51 smithi045 conmon[33655]: 24542) 6672 : cluster [DBG] pgmap v6676: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:53.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:53 smithi154 conmon[34553]: cluster 2021-07-31T04:22:52.025014+0000 mgr.x (mgr.24542) 6673 : cluster [DBG] pgmap v6677: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:53.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:53 smithi045 conmon[33655]: cluster 2021-07-31T04:22:52.025014+0000 mgr.x (mgr.24542) 6673 : 2021-07-31T04:22:53.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:53 smithi045 conmon[33655]: cluster [DBG] pgmap v6677: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:53.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:53 smithi045 conmon[39718]: cluster 2021-07-31T04:22:52.025014+0000 mgr.x (mgr.24542) 2021-07-31T04:22:53.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:53 smithi045 conmon[39718]: 6673 : cluster [DBG] pgmap v6677: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:55.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:55 smithi154 conmon[34553]: cluster 2021-07-31T04:22:54.025446+0000 mgr.x (mgr.24542) 6674 : cluster [DBG] pgmap v6678: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:55.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:55 smithi045 conmon[39718]: cluster 2021-07-31T04:22:54.025446+0000 mgr.x (mgr.24542) 6674 : cluster [DBG] 2021-07-31T04:22:55.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:55 smithi045 conmon[39718]: pgmap v6678: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:55.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:55 smithi045 conmon[33655]: cluster 2021-07-31T04:22:54.025446+0000 mgr.x (mgr.24542) 6674 : 2021-07-31T04:22:55.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:55 smithi045 conmon[33655]: cluster [DBG] pgmap v6678: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:57.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:57 smithi154 conmon[34553]: cluster 2021-07-31T04:22:56.025962+0000 mgr.x (mgr.24542) 6675 : cluster [DBG] pgmap v6679: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:57.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:57 smithi045 conmon[33655]: cluster 2021-07-31T04:22:56.025962+0000 mgr.x (mgr.24542) 6675 : cluster 2021-07-31T04:22:57.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:57 smithi045 conmon[33655]: [DBG] pgmap v6679: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:57.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:57 smithi045 conmon[39718]: cluster 2021-07-31T04:22:56.025962+0000 2021-07-31T04:22:57.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:57 smithi045 conmon[39718]: mgr.x (mgr.24542) 6675 : cluster [DBG] pgmap v6679: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:59.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:22:59 smithi154 conmon[34553]: cluster 2021-07-31T04:22:58.026517+0000 mgr.x (mgr.24542) 6676 : cluster [DBG] pgmap v6680: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:59.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:59 smithi045 conmon[39718]: cluster 2021-07-31T04:22:58.026517+0000 mgr.x 2021-07-31T04:22:59.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:22:59 smithi045 conmon[39718]: (mgr.24542) 6676 : cluster [DBG] pgmap v6680: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:22:59.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:59 smithi045 conmon[33655]: cluster 2021-07-31T04:22:58.026517+0000 mgr.x (mgr.24542) 6676 2021-07-31T04:22:59.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:22:59 smithi045 conmon[33655]: : cluster [DBG] pgmap v6680: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:01.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:01 smithi154 conmon[34553]: cluster 2021-07-31T04:23:00.027104+0000 mgr.x (mgr.24542) 6677 : cluster [DBG] pgmap v6681: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:01.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:01 smithi045 conmon[33655]: cluster 2021-07-31T04:23:00.027104+0000 2021-07-31T04:23:01.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:01 smithi045 conmon[33655]: mgr.x (mgr.24542) 6677 : cluster [DBG] pgmap v6681: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:01.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:01 smithi045 conmon[39718]: cluster 2021-07-31T04:23:00.027104+0000 mgr.x 2021-07-31T04:23:01.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:01 smithi045 conmon[39718]: (mgr.24542) 6677 : cluster [DBG] pgmap v6681: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:03.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:03 smithi154 conmon[34553]: cluster 2021-07-31T04:23:02.027450+0000 mgr.x (mgr.24542) 6678 : cluster [DBG] pgmap v6682: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:03.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:03 smithi045 conmon[33655]: cluster 2021-07-31T 2021-07-31T04:23:03.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:03 smithi045 conmon[33655]: 04:23:02.027450+0000 mgr.x (mgr.24542) 6678 : cluster [DBG] pgmap v6682: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:03.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:03 smithi045 conmon[39718]: cluster 2021-07-31T04:23:02.027450+0000 2021-07-31T04:23:03.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:03 smithi045 conmon[39718]: mgr.x (mgr.24542) 6678 : cluster [DBG] pgmap v6682: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:05.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:05 smithi154 conmon[34553]: cluster 2021-07-31T04:23:04.027944+0000 mgr.x (mgr.24542) 6679 : cluster [DBG] pgmap v6683: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:05.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:05 smithi045 conmon[39718]: cluster 2021-07-31T04:23:04.027944+0000 2021-07-31T04:23:05.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:05 smithi045 conmon[39718]: mgr.x (mgr.24542) 6679 : cluster [DBG] pgmap v6683: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:05.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:05 smithi045 conmon[33655]: cluster 2021-07-31T04:23:04.027944+0000 mgr.x (mgr.24542) 6679 : cluster 2021-07-31T04:23:05.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:05 smithi045 conmon[33655]: [DBG] pgmap v6683: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:07.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:07 smithi154 conmon[34553]: cluster 2021-07-31T04:23:06.028363+0000 mgr.x (mgr.24542) 6680 : cluster [DBG] pgmap v6684: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:07.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:07 smithi045 conmon[33655]: cluster 2021-07-31T04:23:06.028363+0000 mgr.x ( 2021-07-31T04:23:07.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:07 smithi045 conmon[33655]: mgr.24542) 6680 : cluster [DBG] pgmap v6684: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:07.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:07 smithi045 conmon[39718]: cluster 2021-07-31T04:23:06.028363+0000 mgr.x (mgr.24542) 2021-07-31T04:23:07.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:07 smithi045 conmon[39718]: 6680 : cluster [DBG] pgmap v6684: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:07.833 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T04:23:08.110 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T04:23:08.110 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 2h ago 2h - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T04:23:08.110 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 2h ago 2h 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T04:23:08.110 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 2h ago 2h 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T04:23:08.111 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 2h ago 2h - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T04:23:08.111 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T04:23:08.111 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 2h ago 2h 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T04:23:08.111 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T04:23:08.111 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 2h ago 2h - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T04:23:08.112 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 2h ago 2h 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T04:23:08.112 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T04:23:08.112 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T04:23:08.112 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T04:23:08.113 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T04:23:08.113 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 2h ago 2h 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T04:23:08.113 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 2h ago 2h 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T04:23:08.113 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 2h ago 2h 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T04:23:08.113 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 2h ago 2h 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T04:23:08.113 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 2h ago 2h 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T04:23:08.446 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T04:23:08.446 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T04:23:08.446 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T04:23:08.447 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:23:08.447 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T04:23:08.447 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T04:23:08.447 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:23:08.447 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:23:08.448 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T04:23:08.448 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T04:23:08.448 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:23:08.448 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T04:23:08.449 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T04:23:08.449 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T04:23:08.449 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:23:08.449 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T04:23:08.449 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T04:23:09.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:09 smithi154 conmon[34553]: audit 2021-07-31T04:23:07.822194+0000 mgr.x (mgr.24542) 6681 : audit [DBG] from='client.21486 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:23:09.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:09 smithi154 conmon[34553]: cluster 2021-07-31T04:23:08.028711 2021-07-31T04:23:09.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:09 smithi154 conmon[34553]: +0000 mgr.x (mgr.24542) 6682 : cluster [DBG] pgmap v6685: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:09.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:09 smithi154 conmon[34553]: audit 2021-07-31T04:23:08.102512+0000 mgr.x (mgr.24542) 6683 : audit [DBG] from='client.21492 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:23:09.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:09 smithi154 conmon[34553]: audit 2021-07-31T04:23:08.446781 2021-07-31T04:23:09.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:09 smithi154 conmon[34553]: +0000 mon.a (mon.0) 3471 : audit [DBG] from='client.? 172.21.15.45:0/2970650885' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:23:09.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:09 smithi045 conmon[33655]: audit 2021-07-31T04:23:07.822194+0000 mgr.x (mgr.24542) 6681 : audit [DBG] from='client.21486 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:23:09.553 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:09 smithi045 conmon[33655]: cluster 2021-07-31T04:23:09.554 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:09 smithi045 conmon[33655]: 2021-07-31T04:23:08.028711+0000 mgr.x (mgr.24542) 6682 : cluster [DBG] pgmap v6685: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:09.554 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:09 smithi045 conmon[33655]: audit 2021-07-31T04:23:08.102512+0000 mgr.x (mgr.24542) 6683 : audit [DBG] from='client.21492 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:23:09.554 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:09 smithi045 conmon[33655]: audit 2021-07-31T04:23:08 2021-07-31T04:23:09.555 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:09 smithi045 conmon[33655]: .446781+0000 mon.a (mon.0) 3471 : audit [DBG] from='client.? 172.21.15.45:0/2970650885' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:23:09.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:09 smithi045 conmon[39718]: audit 2021-07-31T04:23:07.822194+0000 mgr.x (mgr 2021-07-31T04:23:09.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:09 smithi045 conmon[39718]: .24542) 6681 : audit [DBG] from='client.21486 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:23:09.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:09 smithi045 conmon[39718]: cluster 2021-07-31T04:23:08.028711+0000 mgr.x (mgr. 2021-07-31T04:23:09.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:09 smithi045 conmon[39718]: 24542) 6682 : cluster [DBG] pgmap v6685: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:09.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:09 smithi045 conmon[39718]: audit 2021-07-31T04:23 2021-07-31T04:23:09.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:09 smithi045 conmon[39718]: :08.102512+0000 mgr.x (mgr.24542) 6683 : audit [DBG] from='client.21492 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:23:09.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:09 smithi045 conmon[39718]: audit 2021-07-31T04:23 2021-07-31T04:23:09.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:09 smithi045 conmon[39718]: :08.446781+0000 mon.a (mon.0) 3471 : audit [DBG] 2021-07-31T04:23:09.557 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:09 smithi045 conmon[39718]: from='client.? 172.21.15.45:0/2970650885' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:23:11.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:11 smithi154 conmon[34553]: cluster 2021-07-31T04:23:10.029369+0000 mgr.x (mgr.24542) 6684 : cluster [DBG] pgmap v6686: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:11.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:11 smithi045 conmon[33655]: cluster 2021-07-31T04:23:10.029369+0000 mgr.x (mgr.24542) 6684 : 2021-07-31T04:23:11.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:11 smithi045 conmon[33655]: cluster [DBG] pgmap v6686: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:11.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:11 smithi045 conmon[39718]: cluster 2021-07-31T04:23:10.029369+0000 mgr.x ( 2021-07-31T04:23:11.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:11 smithi045 conmon[39718]: mgr.24542) 6684 : cluster [DBG] pgmap v6686: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:13.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:13 smithi154 conmon[34553]: cluster 2021-07-31T04:23:12.029760+0000 mgr.x (mgr.24542) 6685 : cluster [DBG] pgmap v6687: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:13.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:13 smithi045 conmon[39718]: cluster 2021-07-31T04:23:12.029760+0000 mgr.x (mgr. 2021-07-31T04:23:13.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:13 smithi045 conmon[39718]: 24542) 6685 : cluster [DBG] pgmap v6687: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:13.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:13 smithi045 conmon[33655]: cluster 2021-07-31T04:23:12.029760+0000 mgr.x (mgr.24542) 6685 : cluster [DBG] pgmap v6687: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:15.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:15 smithi154 conmon[34553]: cluster 2021-07-31T04:23:14.030323+0000 mgr.x (mgr.24542) 6686 : cluster [DBG] pgmap v6688: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:15.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:15 smithi045 conmon[39718]: cluster 2021-07-31T04:23:14.030323+0000 mgr.x (mgr 2021-07-31T04:23:15.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:15 smithi045 conmon[39718]: .24542) 6686 : cluster [DBG] pgmap v6688: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:15.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:15 smithi045 conmon[33655]: cluster 2021-07-31T04:23:14.030323+0000 mgr.x (mgr.24542) 6686 : cluster [DBG] pgmap v6688: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:17.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:17 smithi154 conmon[34553]: cluster 2021-07-31T04:23:16.030810+0000 mgr.x (mgr.24542) 6687 : cluster [DBG] pgmap v6689: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:17.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:17 smithi045 conmon[33655]: cluster 2021-07-31T04:23:16.030810+0000 mgr.x (mgr.24542) 6687 : cluster [DBG] pgmap v6689: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:17.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:17 smithi045 conmon[39718]: cluster 2021-07-31T04:23:16.030810+0000 mgr.x ( 2021-07-31T04:23:17.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:17 smithi045 conmon[39718]: mgr.24542) 6687 : cluster [DBG] pgmap v6689: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:19.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:19 smithi154 conmon[34553]: cluster 2021-07-31T04:23:18.031226+0000 mgr.x (mgr.24542) 6688 : cluster [DBG] pgmap v6690: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:19.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:19 smithi045 conmon[39718]: cluster 2021-07-31T04:23:18.031226+0000 mgr.x ( 2021-07-31T04:23:19.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:19 smithi045 conmon[39718]: mgr.24542) 6688 : cluster [DBG] pgmap v6690: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:19.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:19 smithi045 conmon[33655]: cluster 2021-07-31T04:23:18.031226+0000 2021-07-31T04:23:19.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:19 smithi045 conmon[33655]: mgr.x (mgr.24542) 6688 : cluster [DBG] pgmap v6690: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:21.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:21 smithi154 conmon[34553]: cluster 2021-07-31T04:23:20.031759+0000 mgr.x (mgr.24542) 6689 : cluster [DBG] pgmap v6691: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:21.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:21 smithi045 conmon[39718]: cluster 2021-07-31T04:23:20.031759+0000 mgr.x (mgr. 2021-07-31T04:23:21.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:21 smithi045 conmon[39718]: 24542) 6689 : cluster [DBG] pgmap v6691: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:21.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:21 smithi045 conmon[33655]: cluster 2021-07-31T04:23:20.031759+0000 mgr.x (mgr.24542) 6689 : 2021-07-31T04:23:21.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:21 smithi045 conmon[33655]: cluster [DBG] pgmap v6691: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:23.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:23 smithi154 conmon[34553]: cluster 2021-07-31T04:23:22.032151+0000 mgr.x (mgr.24542) 6690 : cluster [DBG] pgmap v6692: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:23.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:23 smithi045 conmon[33655]: cluster 2021-07-31T04:23:22.032151+0000 mgr.x (mgr.24542) 6690 : cluster [DBG] pgmap v6692: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:23.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:23 smithi045 conmon[39718]: cluster 2021-07-31T04:23:22.032151+0000 mgr.x (mgr. 2021-07-31T04:23:23.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:23 smithi045 conmon[39718]: 24542) 6690 : cluster [DBG] pgmap v6692: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:25.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:25 smithi154 conmon[34553]: cluster 2021-07-31T04:23:24.032555+0000 mgr.x (mgr.24542) 6691 : cluster [DBG] pgmap v6693: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:25.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:25 smithi045 conmon[39718]: cluster 2021-07-31T04:23:24.032555+0000 mgr.x 2021-07-31T04:23:25.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:25 smithi045 conmon[39718]: (mgr.24542) 6691 : cluster [DBG] pgmap v6693: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:25.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:25 smithi045 conmon[33655]: cluster 2021-07-31T04:23:24.032555+0000 mgr.x ( 2021-07-31T04:23:25.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:25 smithi045 conmon[33655]: mgr.24542) 6691 : cluster [DBG] pgmap v6693: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:27.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:27 smithi154 conmon[34553]: cluster 2021-07-31T04:23:26.032968+0000 mgr.x (mgr.24542) 6692 : cluster [DBG] pgmap v6694: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:27.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:27 smithi045 conmon[39718]: cluster 2021-07-31T04:23: 2021-07-31T04:23:27.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:27 smithi045 conmon[39718]: 26.032968+0000 mgr.x (mgr.24542) 6692 : cluster [DBG] pgmap v6694: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:27.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:27 smithi045 conmon[33655]: cluster 2021-07-31T04:23:26.032968+0000 mgr.x (mgr.24542) 6692 : cluster [DBG] pgmap v6694: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:27.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:27 smithi045 conmon[33655]: 2021-07-31T04:23:29.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:29 smithi154 conmon[34553]: cluster 2021-07-31T04:23:28.033511+0000 mgr.x (mgr.24542) 6693 : cluster [DBG] pgmap v6695: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:29.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:29 smithi154 conmon[34553]: audit 2021-07-31T04:23:28.549741+0000 mon.b (mon.2) 855 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:23:29.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:29 smithi154 conmon[34553]: audit 2021-07-31T04:23:28.550362+0000 mon.b (mon.2) 856 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:23:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:29 smithi154 conmon[34553]: audit 2021-07-31T04:23:28.550679+0000 mon.a 2021-07-31T04:23:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:29 smithi154 conmon[34553]: (mon.0) 3472 : audit [INF] 2021-07-31T04:23:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:29 smithi154 conmon[34553]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:23:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:29 smithi154 conmon[34553]: audit 2021-07-31T04 2021-07-31T04:23:29.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:29 smithi154 conmon[34553]: :23:28.551088+0000 mon.a (mon.0) 3473 : 2021-07-31T04:23:29.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:29 smithi154 conmon[34553]: audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:23:29.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:29 smithi045 conmon[33655]: cluster 2021-07-31T04:23:28.033511+0000 mgr.x (mgr.24542) 6693 : cluster [DBG] pgmap v6695: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:29.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:29 smithi045 conmon[33655]: audit 2021-07-31T04:23:28.549741+0000 mon.b (mon.2) 855 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:23:29.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:29 smithi045 conmon[33655]: audit 2021-07-31T04:23:28.550362+0000 mon.b (mon.2 2021-07-31T04:23:29.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:29 smithi045 conmon[33655]: ) 856 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:23:29.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:29 smithi045 conmon[33655]: audit 2021-07-31T04:23:28.550679+0000 2021-07-31T04:23:29.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:29 smithi045 conmon[33655]: mon.a (mon.0) 3472 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:23:29.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:29 smithi045 conmon[33655]: audit 2021-07- 2021-07-31T04:23:29.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:29 smithi045 conmon[33655]: 31T04:23:28.551088+0000 mon.a (mon.0) 3473 : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:23:29.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:29 smithi045 conmon[39718]: cluster 2021-07-31T04:23:28.033511+0000 mgr.x 2021-07-31T04:23:29.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:29 smithi045 conmon[39718]: (mgr.24542) 6693 : cluster [DBG] pgmap v6695: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:29.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:29 smithi045 conmon[39718]: audit 2021-07-31T04:23:28. 2021-07-31T04:23:29.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:29 smithi045 conmon[39718]: 549741+0000 mon.b (mon.2) 855 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:23:29.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:29 smithi045 conmon[39718]: audit 2021-07 2021-07-31T04:23:29.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:29 smithi045 conmon[39718]: -31T04:23:28.550362+0000 2021-07-31T04:23:29.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:29 smithi045 conmon[39718]: mon.b (mon.2) 856 : audit [INF] 2021-07-31T04:23:29.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:29 smithi045 conmon[39718]: from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:23:29.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:29 smithi045 conmon[39718]: audit 2021-07-31T04:23: 2021-07-31T04:23:29.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:29 smithi045 conmon[39718]: 28.550679+0000 mon.a (mon.0) 3472 2021-07-31T04:23:29.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:29 smithi045 conmon[39718]: : audit [INF] from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-07-31T04:23:29.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:29 smithi045 conmon[39718]: audit 2021-07 2021-07-31T04:23:29.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:29 smithi045 conmon[39718]: -31T04:23:28.551088+0000 2021-07-31T04:23:29.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:29 smithi045 conmon[39718]: mon.a (mon.0) 3473 : audit [INF] 2021-07-31T04:23:29.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:29 smithi045 conmon[39718]: from='mgr.24542 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-07-31T04:23:31.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:31 smithi154 conmon[34553]: cluster 2021-07-31T04:23:30.034088+0000 mgr.x (mgr.24542) 6694 : cluster [DBG] pgmap v6696: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:31.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:31 smithi045 conmon[39718]: cluster 2021-07-31T04:23:30.034088+0000 mgr.x (mgr. 2021-07-31T04:23:31.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:31 smithi045 conmon[39718]: 24542) 6694 : cluster [DBG] pgmap v6696: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:31.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:31 smithi045 conmon[33655]: cluster 2021-07-31T04:23:30.034088+0000 mgr.x (mgr.24542) 6694 : cluster [DBG] pgmap v6696: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:33.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:33 smithi154 conmon[34553]: cluster 2021-07-31T04:23:32.034613+0000 mgr.x (mgr.24542) 6695 : cluster [DBG] pgmap v6697: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:33.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:33 smithi045 conmon[33655]: cluster 2021-07-31T04:23:32.034613+0000 mgr.x (mgr. 2021-07-31T04:23:33.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:33 smithi045 conmon[33655]: 24542) 6695 : cluster [DBG] pgmap v6697: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:33.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:33 smithi045 conmon[39718]: cluster 2021-07-31T04:23:32.034613+0000 mgr.x (mgr 2021-07-31T04:23:33.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:33 smithi045 conmon[39718]: .24542) 6695 : cluster [DBG] pgmap v6697: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:35.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:35 smithi154 conmon[34553]: cluster 2021-07-31T04:23:34.035268+0000 mgr.x (mgr.24542) 6696 : cluster [DBG] pgmap v6698: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:35.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:35 smithi045 conmon[33655]: cluster 2021-07-31T04:23:34.035268+0000 mgr.x (mgr.24542) 6696 : cluster [DBG] 2021-07-31T04:23:35.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:35 smithi045 conmon[33655]: pgmap v6698: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:35.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:35 smithi045 conmon[39718]: cluster 2021-07-31T04:23:34.035268+0000 mgr.x (mgr.24542) 6696 : cluster [DBG] pgmap v6698: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:37.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:37 smithi154 conmon[34553]: cluster 2021-07-31T04:23:36.035864+0000 mgr.x (mgr.24542) 6697 : cluster [DBG] pgmap v6699: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:37.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:37 smithi045 conmon[33655]: cluster 2021-07-31T04:23:36.035864+0000 mgr.x (mgr.24542) 6697 : cluster [DBG] pgmap v6699: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:37.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:37 smithi045 conmon[39718]: cluster 2021-07-31T04:23:36. 2021-07-31T04:23:37.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:37 smithi045 conmon[39718]: 035864+0000 mgr.x (mgr.24542) 6697 : cluster [DBG] pgmap v6699: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:38.759 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T04:23:39.003 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: debug 2021-07-31T04:23:38.554+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:23:39.003 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:23:39.003 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:23:39.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: yield 2021-07-31T04:23:39.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:23:39.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:23:39.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:23:39.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:23:39.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:23:39.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:23:39.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:23:39.005 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:23:39.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:23:39.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: task.result() 2021-07-31T04:23:39.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:23:39.006 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: debug 2021-07-31T04:23:38.566+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:23:39.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:23:39.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:23:39.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: yield 2021-07-31T04:23:39.007 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:23:39.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:23:39.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:23:39.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:23:39.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:23:39.008 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:23:39.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:23:39.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:23:39.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:23:39.009 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: task.result() 2021-07-31T04:23:39.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:23:39.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: debug 2021-07-31T04:23:38.592+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:23:39.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:23:39.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:23:39.010 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: yield 2021-07-31T04:23:39.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:23:39.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:23:39.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:23:39.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:23:39.011 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:23:39.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:23:39.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:23:39.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:23:39.012 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:23:39.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: task.result() 2021-07-31T04:23:39.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:23:39.013 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: debug 2021-07-31T04:23:38.592+0000 7fcb9295f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:23:39.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:23:39.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:23:39.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: yield 2021-07-31T04:23:39.014 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:23:39.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:23:39.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:23:39.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:23:39.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:23:39.015 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:23:39.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:23:39.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:23:39.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:23:39.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: task.result() 2021-07-31T04:23:39.016 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:23:39.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: 2021-07-31T04:23:39.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T04:23:39.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: 2021-07-31T04:23:39.017 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:23:39.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:23:39.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: return f(*arg) 2021-07-31T04:23:39.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:23:39.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:23:39.018 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:23:39.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:23:39.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:23:39.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:23:39.019 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:23:39.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T04:23:39.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:23:39.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: raise self._exception 2021-07-31T04:23:39.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:23:39.020 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:23:39.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:23:39.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T04:23:39.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:23:39.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: raise 2021-07-31T04:23:39.021 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:23:39.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T04:23:39.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:23:39.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:23:39.022 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:23:39.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: debug 2021-07-31T04:23:38.602+0000 7fcb8d955700 -1 log_channel(cephadm) log [ERR] : 2021-07-31T04:23:39.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:23:39.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:23:39.023 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: yield 2021-07-31T04:23:39.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:23:39.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:23:39.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:23:39.024 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:23:39.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:23:39.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:23:39.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:23:39.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:23:39.025 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:23:39.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: task.result() 2021-07-31T04:23:39.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:23:39.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: debug 2021-07-31T04:23:38.603+0000 7fcb93160700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:23:39.026 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:23:39.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:23:39.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: yield 2021-07-31T04:23:39.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:23:39.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:23:39.027 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:23:39.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: 'Opening SSH connection to') 2021-07-31T04:23:39.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:23:39.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: await conn.wait_established() 2021-07-31T04:23:39.028 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:23:39.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: await self._waiter 2021-07-31T04:23:39.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:23:39.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: task.result() 2021-07-31T04:23:39.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: concurrent.futures._base.CancelledError 2021-07-31T04:23:39.029 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: 2021-07-31T04:23:39.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: During handling of the above exception, another exception occurred: 2021-07-31T04:23:39.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: 2021-07-31T04:23:39.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: Traceback (most recent call last): 2021-07-31T04:23:39.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:23:39.030 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: return f(*arg) 2021-07-31T04:23:39.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:23:39.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:23:39.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:23:39.031 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:23:39.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:23:39.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:23:39.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:23:39.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: return self.__get_result() 2021-07-31T04:23:39.032 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:23:39.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: raise self._exception 2021-07-31T04:23:39.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:23:39.033 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:23:39.035 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:23:39.035 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: conn = await self._remote_connection(host, addr) 2021-07-31T04:23:39.035 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:23:39.035 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: raise 2021-07-31T04:23:39.036 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:23:39.036 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: self.gen.throw(type, value, traceback) 2021-07-31T04:23:39.036 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:23:39.036 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:23:39.037 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Jul 31 04:23:38 smithi154 conmon[57492]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:23:39.038 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T04:23:39.038 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 2h ago 2h - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T04:23:39.038 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 2h ago 2h 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T04:23:39.038 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 2h ago 2h 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T04:23:39.039 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 2h ago 2h - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T04:23:39.039 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T04:23:39.039 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 2h ago 2h 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T04:23:39.039 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T04:23:39.040 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 2h ago 2h - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T04:23:39.040 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 2h ago 2h 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T04:23:39.040 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T04:23:39.040 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T04:23:39.040 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T04:23:39.040 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T04:23:39.041 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 2h ago 2h 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T04:23:39.041 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 2h ago 2h 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T04:23:39.041 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 2h ago 2h 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T04:23:39.041 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 2h ago 2h 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T04:23:39.041 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 2h ago 2h 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T04:23:39.389 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T04:23:39.389 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T04:23:39.390 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T04:23:39.390 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:23:39.390 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T04:23:39.390 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T04:23:39.390 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:23:39.391 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:23:39.391 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T04:23:39.391 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T04:23:39.391 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:23:39.391 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T04:23:39.392 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T04:23:39.392 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T04:23:39.392 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:23:39.392 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T04:23:39.392 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T04:23:39.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:39 smithi154 conmon[34553]: cluster 2021-07-31T04:23:38.036290+0000 mgr.x (mgr.24542) 6698 2021-07-31T04:23:39.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:39 smithi154 conmon[34553]: : cluster [DBG] pgmap v6700: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:39.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:39 smithi154 conmon[34553]: audit 2021-07-31T04:23:38.508938+0000 mon.b (mon.2) 857 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T04:23:39.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:39 smithi154 conmon[34553]: audit 2021- 2021-07-31T04:23:39.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:39 smithi154 conmon[34553]: 07-31T04:23:38.510506+0000 mon.b (mon.2) 858 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T04:23:39.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:39 smithi154 conmon[34553]: audit 2021-07- 2021-07-31T04:23:39.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:39 smithi154 conmon[34553]: 31T04:23:38.511526+0000 mon.b ( 2021-07-31T04:23:39.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:39 smithi154 conmon[34553]: mon.2) 859 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T04:23:39.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:39 smithi154 conmon[34553]: cephadm 2021- 2021-07-31T04:23:39.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:39 smithi154 conmon[34553]: 07-31T04:23:38.512241+0000 2021-07-31T04:23:39.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:39 smithi154 conmon[34553]: mgr.x (mgr.24542) 6699 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T04:23:39.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:39 smithi154 conmon[34553]: 2021-07-31T04:23:39.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:23 2021-07-31T04:23:39.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:39 smithi154 conmon[34553]: :38.555424+0000 mgr.x (mgr.24542) 6700 : 2021-07-31T04:23:39.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:39 smithi154 conmon[34553]: cephadm [ERR] 2021-07-31T04:23:39.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:39 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:23:39.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:23:39.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:39 smithi154 conmon[34553]: yield 2021-07-31T04:23:39.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:39 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:23:39.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:39 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:23:39.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:23:39.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:39 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:23:39.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:23:39.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:39 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:23:39.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:23:39.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:39 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:23:39.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:39 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:23:39.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:39 smithi154 conmon[34553]: task.result() 2021-07-31T04:23:39.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:39 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:23:39.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:39 smithi154 conmon[34553]: cephadm 2021-07-31T04:23:38.555851+0000 mgr.x (mgr.24542) 6701 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T04:23:39.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:39 smithi045 conmon[39718]: cluster 2021-07-31T04:23:38.036290+0000 mgr.x (mgr 2021-07-31T04:23:39.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:39 smithi045 conmon[39718]: .24542) 6698 : cluster [DBG] pgmap v6700: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:39.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:39 smithi045 conmon[39718]: audit 2021-07-31T04:23:38 2021-07-31T04:23:39.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:39 smithi045 conmon[39718]: .508938+0000 mon.b (mon.2) 857 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T04:23:39.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:39 smithi045 conmon[39718]: audit 2021-07-31T04:23:38.510506+0000 mon.b ( 2021-07-31T04:23:39.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:39 smithi045 conmon[39718]: mon.2) 858 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T04:23:39.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:39 smithi045 conmon[39718]: audit 2021-07-31T04:23:38.511526+0000 2021-07-31T04:23:39.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:39 smithi045 conmon[39718]: mon.b (mon.2) 859 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T04:23:39.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:39 smithi045 conmon[39718]: cephadm 2021-07-31T04:23 2021-07-31T04:23:39.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:39 smithi045 conmon[39718]: :38.512241+0000 mgr.x (mgr.24542) 6699 : cephadm [WRN] 2021-07-31T04:23:39.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:39 smithi045 conmon[39718]: unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T04:23:39.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:39 smithi045 conmon[39718]: cephadm 2021-07-31T 2021-07-31T04:23:39.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:39 smithi045 conmon[39718]: 04:23:38.555424+0000 mgr.x (mgr.24542) 6700 : cephadm [ERR] 2021-07-31T04:23:39.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:39 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:23:39.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:23:39.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:39 smithi045 conmon[39718]: yield 2021-07-31T04:23:39.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:39 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:23:39.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:39 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:23:39.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:23:39.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:39 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:23:39.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:23:39.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:39 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:23:39.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:23:39.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:39 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:23:39.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:39 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:23:39.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:39 smithi045 conmon[39718]: task.result() 2021-07-31T04:23:39.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:39 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:23:39.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:39 smithi045 conmon[39718]: cephadm 2021-07-31T04:23:38.555851+0000 mgr.x (mgr.24542) 6701 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T04:23:39.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:39 smithi045 conmon[33655]: cluster 2021-07-31T04:23:38.036290+0000 mgr.x (mgr 2021-07-31T04:23:39.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:39 smithi045 conmon[33655]: .24542) 6698 : cluster [DBG] pgmap v6700: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:39.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:39 smithi045 conmon[33655]: audit 2021-07-31T04:23:38.508938 2021-07-31T04:23:39.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:39 smithi045 conmon[33655]: +0000 mon.b (mon.2) 857 : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-07-31T04:23:39.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:39 smithi045 conmon[33655]: audit 2021-07-31T04:23:38.510506+0000 mon.b (mon.2) 858 2021-07-31T04:23:39.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:39 smithi045 conmon[33655]: : audit [DBG] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-07-31T04:23:39.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:39 smithi045 conmon[33655]: audit 2021- 2021-07-31T04:23:39.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:39 smithi045 conmon[33655]: 07-31T04:23:38.511526+0000 mon.b (mon 2021-07-31T04:23:39.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:39 smithi045 conmon[33655]: .2) 859 : audit [INF] from='mgr.24542 172.21.15.154:0/3673032120' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-07-31T04:23:39.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:39 smithi045 conmon[33655]: cephadm 2021-07-31 2021-07-31T04:23:39.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:39 smithi045 conmon[33655]: T04:23:38.512241+0000 mgr.x (mgr. 2021-07-31T04:23:39.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:39 smithi045 conmon[33655]: 24542) 6699 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(host_pattern='*'): Cannot place : No matching hosts 2021-07-31T04:23:39.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:39 smithi045 conmon[33655]: cephadm 2021-07-31 2021-07-31T04:23:39.535 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:39 smithi045 conmon[33655]: T04:23:38.555424+0000 mgr.x ( 2021-07-31T04:23:39.535 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:39 smithi045 conmon[33655]: mgr.24542) 6700 : cephadm [ERR] 2021-07-31T04:23:39.535 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:39 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:23:39.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:23:39.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:39 smithi045 conmon[33655]: yield 2021-07-31T04:23:39.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:39 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:23:39.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:39 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:23:39.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:23:39.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:39 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:23:39.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:23:39.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:39 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:23:39.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:23:39.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:39 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:23:39.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:39 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:23:39.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:39 smithi045 conmon[33655]: task.result() 2021-07-31T04:23:39.539 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:39 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:23:39.539 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:39 smithi045 conmon[33655]: cephadm 2021-07-31T04:23:38.555851+0000 mgr.x (mgr.24542) 6701 : cephadm [INF] Removing smithi045:/etc/ceph/ceph.conf 2021-07-31T04:23:40.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: cephadm 2021-07-31T04:23:38.567444+0000 mgr.x (mgr.24542) 6702 : cephadm [ERR] 2021-07-31T04:23:40.617 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:23:40.618 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:23:40.618 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: yield 2021-07-31T04:23:40.618 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:23:40.619 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:23:40.619 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:23:40.619 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:23:40.619 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:23:40.619 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:23:40.620 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:23:40.620 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:23:40.620 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:23:40.620 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: task.result() 2021-07-31T04:23:40.620 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:23:40.621 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: cephadm 2021-07-31T04:23:38.568118+0000 mgr.x (mgr.24542) 6703 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T04:23:40.621 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: cephadm 2021-07-31T04:23:38.593256+0000 mgr.x (mgr.24542) 6704 : cephadm [ERR] 2021-07-31T04:23:40.621 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:23:40.621 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:23:40.621 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: yield 2021-07-31T04:23:40.622 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:23:40.622 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:23:40.622 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:23:40.622 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:23:40.622 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:23:40.623 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:23:40.623 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:23:40.623 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:23:40.623 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:23:40.623 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: task.result() 2021-07-31T04:23:40.624 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:23:40.624 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: cephadm 2021-07-31T04:23:38.593699+0000 mgr.x (mgr.24542) 6705 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:23:40.624 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:23:40.624 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:23:40.624 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: yield 2021-07-31T04:23:40.625 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:23:40.625 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:23:40.625 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:23:40.625 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:23:40.626 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:23:40.626 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:23:40.626 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:23:40.626 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:23:40.626 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:23:40.627 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: task.result() 2021-07-31T04:23:40.627 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:23:40.627 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: 2021-07-31T04:23:40.627 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T04:23:40.627 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: 2021-07-31T04:23:40.628 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:23:40.628 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:23:40.628 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: return f(*arg) 2021-07-31T04:23:40.628 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:23:40.629 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:23:40.629 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:23:40.629 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:23:40.629 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:23:40.629 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:23:40.630 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:23:40.630 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T04:23:40.630 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:23:40.630 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: raise self._exception 2021-07-31T04:23:40.630 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:23:40.631 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:23:40.631 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:23:40.631 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T04:23:40.631 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:23:40.631 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: raise 2021-07-31T04:23:40.632 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:23:40.632 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T04:23:40.632 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:23:40.632 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:23:40.633 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:23:40.633 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: cephadm 2021-07-31T04:23:38.603372+0000 mgr.x (mgr.24542) 6706 : cephadm [ERR] 2021-07-31T04:23:40.633 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:23:40.633 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:23:40.633 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: yield 2021-07-31T04:23:40.634 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:23:40.634 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:23:40.634 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:23:40.634 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:23:40.634 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:23:40.635 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:23:40.635 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:23:40.635 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:23:40.635 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:23:40.635 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: task.result() 2021-07-31T04:23:40.636 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:23:40.636 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: cephadm 2021-07-31T04:23:38.603959+0000 mgr.x (mgr.24542) 6707 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:23:40.636 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:23:40.636 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:23:40.636 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: yield 2021-07-31T04:23:40.637 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:23:40.637 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:23:40.637 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:23:40.637 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: 'Opening SSH connection to') 2021-07-31T04:23:40.637 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:23:40.638 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: await conn.wait_established() 2021-07-31T04:23:40.638 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:23:40.638 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: await self._waiter 2021-07-31T04:23:40.638 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:23:40.638 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: task.result() 2021-07-31T04:23:40.639 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: concurrent.futures._base.CancelledError 2021-07-31T04:23:40.639 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: 2021-07-31T04:23:40.639 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: During handling of the above exception, another exception occurred: 2021-07-31T04:23:40.639 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: 2021-07-31T04:23:40.639 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: Traceback (most recent call last): 2021-07-31T04:23:40.640 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:23:40.640 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: return f(*arg) 2021-07-31T04:23:40.640 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:23:40.640 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:23:40.640 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:23:40.641 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:23:40.641 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:23:40.641 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:23:40.641 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:23:40.641 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: return self.__get_result() 2021-07-31T04:23:40.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:23:40.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: raise self._exception 2021-07-31T04:23:40.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:23:40.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:23:40.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:23:40.643 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: conn = await self._remote_connection(host, addr) 2021-07-31T04:23:40.643 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:23:40.643 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: raise 2021-07-31T04:23:40.643 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:23:40.643 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: self.gen.throw(type, value, traceback) 2021-07-31T04:23:40.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:23:40.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:23:40.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:23:40.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: audit 2021-07-31T04:23:38.748329+0000 mgr.x (mgr.24542) 6708 : audit [DBG] from='client.21504 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:23:40.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: audit 2021-07-31T04:23:39.030536+0000 mgr.x (mgr.24542) 6709 : audit [DBG] from='client.31447 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:23:40.645 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:40 smithi154 conmon[34553]: audit 2021-07-31T04:23:39.390243+0000 mon.a (mon.0) 3474 : audit [DBG] from='client.? 172.21.15.45:0/4152242066' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:23:40.645 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: cephadm 2021-07-31T04:23:38.567444+0000 mgr.x (mgr.24542) 6702 : cephadm [ERR] 2021-07-31T04:23:40.646 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:23:40.646 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:23:40.646 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: yield 2021-07-31T04:23:40.646 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:23:40.646 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:23:40.647 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:23:40.647 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:23:40.647 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:23:40.647 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:23:40.647 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:23:40.648 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:23:40.648 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:23:40.648 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: task.result() 2021-07-31T04:23:40.648 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:23:40.648 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: cephadm 2021-07-31T04:23:38.568118+0000 mgr.x (mgr.24542) 6703 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T04:23:40.649 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: cephadm 2021-07-31T04:23:38.593256+0000 mgr.x (mgr.24542) 6704 : cephadm [ERR] 2021-07-31T04:23:40.649 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:23:40.649 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:23:40.649 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: yield 2021-07-31T04:23:40.649 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:23:40.650 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:23:40.650 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:23:40.650 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:23:40.650 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:23:40.650 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:23:40.651 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:23:40.651 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:23:40.651 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:23:40.651 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: task.result() 2021-07-31T04:23:40.651 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:23:40.652 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: cephadm 2021-07-31T04:23:38.593699+0000 mgr.x (mgr 2021-07-31T04:23:40.652 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: .24542) 6705 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:23:40.652 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:23:40.652 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:23:40.652 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: yield 2021-07-31T04:23:40.653 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:23:40.653 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:23:40.653 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: cephadm 2021-07-31T04:23:38.567444+0000 2021-07-31T04:23:40.653 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: mgr.x (mgr.24542) 6702 : cephadm [ERR] 2021-07-31T04:23:40.654 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:23:40.654 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:23:40.654 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: yield 2021-07-31T04:23:40.654 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:23:40.654 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:23:40.655 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:23:40.655 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:23:40.655 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:23:40.655 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:23:40.655 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:23:40.656 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:23:40.656 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:23:40.656 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: task.result() 2021-07-31T04:23:40.656 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:23:40.656 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: cephadm 2021 2021-07-31T04:23:40.657 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: -07-31T04:23:38.568118+0000 mgr.x (mgr.24542) 6703 : cephadm [INF] Removing smithi154:/etc/ceph/ceph.conf 2021-07-31T04:23:40.657 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: cephadm 2021-07-31T04:23:38.593256+0000 mgr.x (mgr.24542) 6704 : 2021-07-31T04:23:40.657 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: cephadm [ERR] 2021-07-31T04:23:40.657 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:23:40.657 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:23:40.658 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: yield 2021-07-31T04:23:40.658 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:23:40.658 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:23:40.658 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:23:40.658 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:23:40.659 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:23:40.659 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:23:40.659 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:23:40.659 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:23:40.659 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:23:40.660 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: task.result() 2021-07-31T04:23:40.660 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:23:40.660 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: cephadm 2021-07-31T04:23:38.593699+0000 mgr.x (mgr.24542) 6705 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:23:40.660 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:23:40.660 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:23:40.661 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: yield 2021-07-31T04:23:40.661 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:23:40.661 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:23:40.661 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:23:40.662 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:23:40.662 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:23:40.662 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:23:40.662 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:23:40.662 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:23:40.663 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:23:40.663 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: task.result() 2021-07-31T04:23:40.663 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:23:40.663 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: 2021-07-31T04:23:40.663 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T04:23:40.664 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: 2021-07-31T04:23:40.664 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:23:40.664 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:23:40.664 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: return f(*arg) 2021-07-31T04:23:40.664 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:23:40.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:23:40.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:23:40.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:23:40.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:23:40.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:23:40.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:23:40.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T04:23:40.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:23:40.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: raise self._exception 2021-07-31T04:23:40.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:23:40.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:23:40.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:23:40.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T04:23:40.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:23:40.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: raise 2021-07-31T04:23:40.668 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:23:40.668 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T04:23:40.668 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:23:40.668 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:23:40.668 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:23:40.668 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: cephadm 2021-07-31T04:23:38.603372+0000 mgr.x (mgr.24542) 6706 : cephadm [ERR] 2021-07-31T04:23:40.669 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:23:40.669 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:23:40.669 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: yield 2021-07-31T04:23:40.669 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:23:40.669 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:23:40.670 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:23:40.670 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:23:40.670 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:23:40.670 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:23:40.671 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:23:40.671 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:23:40.671 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:23:40.671 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: task.result() 2021-07-31T04:23:40.671 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:23:40.672 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: 2021-07-31T04:23:40.672 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T04:23:40.672 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: 2021-07-31T04:23:40.672 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:23:40.672 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:23:40.673 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: return f(*arg) 2021-07-31T04:23:40.673 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:23:40.673 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:23:40.673 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:23:40.673 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:23:40.674 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:23:40.674 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:23:40.674 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:23:40.674 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T04:23:40.674 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:23:40.675 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: raise self._exception 2021-07-31T04:23:40.675 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:23:40.675 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:23:40.675 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:23:40.675 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T04:23:40.676 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:23:40.676 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: raise 2021-07-31T04:23:40.676 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:23:40.676 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T04:23:40.676 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:23:40.677 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:23:40.677 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi045 (172.21.15.45). 2021-07-31T04:23:40.677 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: cephadm 2021-07-31T04 2021-07-31T04:23:40.677 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: :23:38.603372+0000 mgr.x (mgr.24542) 6706 : cephadm [ERR] 2021-07-31T04:23:40.677 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:23:40.677 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:23:40.678 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: yield 2021-07-31T04:23:40.678 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:23:40.678 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:23:40.679 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:23:40.679 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:23:40.679 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:23:40.679 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:23:40.679 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:23:40.680 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:23:40.680 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: task.result() 2021-07-31T04:23:40.680 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:23:40.680 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: 2021-07-31T04:23:40.680 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: cephadm 2021-07-31T04:23:38.603959+0000 mgr.x (mgr.24542) 6707 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:23:40.681 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:23:40.681 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:23:40.681 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: yield 2021-07-31T04:23:40.681 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:23:40.681 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:23:40.682 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:23:40.682 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: 'Opening SSH connection to') 2021-07-31T04:23:40.682 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:23:40.682 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: await conn.wait_established() 2021-07-31T04:23:40.682 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:23:40.683 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: await self._waiter 2021-07-31T04:23:40.683 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:23:40.683 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: task.result() 2021-07-31T04:23:40.683 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: concurrent.futures._base.CancelledError 2021-07-31T04:23:40.683 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: 2021-07-31T04:23:40.684 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: During handling of the above exception, another exception occurred: 2021-07-31T04:23:40.684 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: 2021-07-31T04:23:40.684 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: Traceback (most recent call last): 2021-07-31T04:23:40.684 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:23:40.684 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: return f(*arg) 2021-07-31T04:23:40.684 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:23:40.685 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:23:40.685 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:23:40.685 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:23:40.685 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:23:40.685 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:23:40.686 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:23:40.686 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: return self.__get_result() 2021-07-31T04:23:40.686 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:23:40.687 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: raise self._exception 2021-07-31T04:23:40.687 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:23:40.687 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:23:40.687 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:23:40.688 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:23:40.688 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:23:40.688 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:23:40.688 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:23:40.688 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:23:40.689 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: task.result() 2021-07-31T04:23:40.689 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:23:40.689 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: cephadm 2021-07-31T04:23:38.603959+0000 mgr.x (mgr.24542) 6707 : cephadm [ERR] executing refresh((['smithi045', 'smithi154'],)) failed. 2021-07-31T04:23:40.689 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:23:40.689 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in redirect_log 2021-07-31T04:23:40.690 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: yield 2021-07-31T04:23:40.690 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 75, in _remote_connection 2021-07-31T04:23:40.690 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname]) 2021-07-31T04:23:40.690 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2021-07-31T04:23:40.690 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: 'Opening SSH connection to') 2021-07-31T04:23:40.691 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2021-07-31T04:23:40.691 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: await conn.wait_established() 2021-07-31T04:23:40.691 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2021-07-31T04:23:40.691 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: await self._waiter 2021-07-31T04:23:40.692 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 829, in _reap_task 2021-07-31T04:23:40.692 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: task.result() 2021-07-31T04:23:40.692 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: concurrent.futures._base.CancelledError 2021-07-31T04:23:40.692 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: 2021-07-31T04:23:40.692 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: During handling of the above exception, another exception occurred: 2021-07-31T04:23:40.692 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: 2021-07-31T04:23:40.693 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: Traceback (most recent call last): 2021-07-31T04:23:40.693 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2021-07-31T04:23:40.693 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: return f(*arg) 2021-07-31T04:23:40.693 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 301, in refresh 2021-07-31T04:23:40.693 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: self.mgr.ssh.check_execute_command(host, cmd) 2021-07-31T04:23:40.694 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 161, in check_execute_command 2021-07-31T04:23:40.694 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: return self.mgr.event_loop.get_result(self._check_execute_command(*args, **kwargs)) 2021-07-31T04:23:40.694 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 46, in get_result 2021-07-31T04:23:40.694 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2021-07-31T04:23:40.694 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2021-07-31T04:23:40.695 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: return self.__get_result() 2021-07-31T04:23:40.695 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-07-31T04:23:40.695 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: raise self._exception 2021-07-31T04:23:40.695 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:23:40.696 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:23:40.696 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:23:40.696 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: conn = await self._remote_connection(host, addr) 2021-07-31T04:23:40.696 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:23:40.696 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: raise 2021-07-31T04:23:40.697 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:23:40.697 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: self.gen.throw(type, value, traceback) 2021-07-31T04:23:40.697 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:23:40.697 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:23:40.697 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:23:40.698 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: audit 2021-07-31T04:23:38.748329+0000 mgr.x (mgr.24542) 6708 : audit [DBG] from='client.21504 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:23:40.698 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: audit 2021-07-31T04:23:39.030536+0000 mgr.x (mgr.24542) 6709 : audit [DBG] from='client.31447 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:23:40.698 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: audit 2021-07-31T04:23:40.698 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[33655]: 2021-07-31T04:23:39.390243+0000 mon.a (mon.0) 3474 : audit [DBG] from='client.? 172.21.15.45:0/4152242066' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:23:40.699 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 153, in _check_execute_command 2021-07-31T04:23:40.699 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2021-07-31T04:23:40.699 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 127, in _execute_command 2021-07-31T04:23:40.699 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: conn = await self._remote_connection(host, addr) 2021-07-31T04:23:40.699 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 81, in _remote_connection 2021-07-31T04:23:40.700 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: raise 2021-07-31T04:23:40.700 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2021-07-31T04:23:40.700 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: self.gen.throw(type, value, traceback) 2021-07-31T04:23:40.700 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 112, in redirect_log 2021-07-31T04:23:40.700 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: raise OrchestratorError(f'Failed to connect to {host} ({addr}). {str(e)}') 2021-07-31T04:23:40.701 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: orchestrator._interface.OrchestratorError: Failed to connect to smithi154 (172.21.15.154). 2021-07-31T04:23:40.701 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: audit 2021-07-31T04:23:38.748329+0000 mgr.x (mgr.24542) 6708 : audit [DBG] from='client.21504 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:23:40.701 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: audit 2021-07-31T04:23:39.030536+0000 mgr.x (mgr.24542) 6709 : audit 2021-07-31T04:23:40.701 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: [DBG] from='client.31447 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:23:40.701 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:40 smithi045 conmon[39718]: audit 2021-07-31T04:23:39.390243+0000 mon.a (mon.0) 3474 : audit [DBG] from='client.? 172.21.15.45:0/4152242066' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:23:41.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:41 smithi154 conmon[34553]: cluster 2021-07-31T04:23:40.036847+0000 mgr.x (mgr.24542) 6710 : cluster [DBG] pgmap v6701: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:41.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:41 smithi045 conmon[39718]: cluster 2021-07-31T04:23:40.036847+0000 mgr.x ( 2021-07-31T04:23:41.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:41 smithi045 conmon[39718]: mgr.24542) 6710 : cluster [DBG] pgmap v6701: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:41.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:41 smithi045 conmon[33655]: cluster 2021-07-31T04:23:40.036847+0000 mgr.x (mgr.24542) 6710 : 2021-07-31T04:23:41.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:41 smithi045 conmon[33655]: cluster [DBG] pgmap v6701: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:43.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:43 smithi154 conmon[34553]: cluster 2021-07-31T04:23:42.037250+0000 mgr.x (mgr.24542) 6711 : cluster [DBG] pgmap v6702: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:43.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:43 smithi045 conmon[39718]: cluster 2021-07-31T04:23:42.037250+0000 2021-07-31T04:23:43.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:43 smithi045 conmon[39718]: mgr.x (mgr.24542) 6711 : cluster [DBG] pgmap v6702: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:43.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:43 smithi045 conmon[33655]: cluster 2021-07-31T04:23:42.037250+0000 mgr.x (mgr.24542) 6711 2021-07-31T04:23:43.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:43 smithi045 conmon[33655]: : cluster [DBG] pgmap v6702: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:45.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:45 smithi154 conmon[34553]: cluster 2021-07-31T04:23:44.037745+0000 mgr.x (mgr.24542) 6712 : cluster [DBG] pgmap v6703: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:45.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:45 smithi045 conmon[33655]: cluster 2021-07-31T04:23:44.037745+0000 mgr.x (mgr.24542) 6712 : cluster [DBG] pgmap v6703: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:45.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:45 smithi045 conmon[33655]: 2021-07-31T04:23:45.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:45 smithi045 conmon[39718]: cluster 2021-07-31T04:23:44.037745+0000 2021-07-31T04:23:45.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:45 smithi045 conmon[39718]: mgr.x (mgr.24542) 6712 : cluster [DBG] pgmap v6703: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:47.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:47 smithi154 conmon[34553]: cluster 2021-07-31T04:23:46.038296+0000 mgr.x (mgr.24542) 6713 : cluster [DBG] pgmap v6704: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:47.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:47 smithi045 conmon[39718]: cluster 2021-07-31T04:23:46.038296+0000 mgr.x (mgr.24542) 6713 : cluster [DBG] pgmap v6704: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:47.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:47 smithi045 conmon[33655]: cluster 2021-07-31T04:23:46.038296+0000 mgr.x (mgr.24542) 6713 : cluster [DBG] pgmap v6704: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:47.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:47 smithi045 conmon[33655]: 2021-07-31T04:23:49.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:49 smithi154 conmon[34553]: cluster 2021-07-31T04:23:48.038675+0000 mgr.x (mgr.24542) 6714 : cluster [DBG] pgmap v6705: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:49.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:49 smithi154 conmon[34553]: 2021-07-31T04:23:49.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:49 smithi045 conmon[33655]: cluster 2021-07-31T04:23:48.038675+0000 mgr.x (mgr.24542) 6714 2021-07-31T04:23:49.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:49 smithi045 conmon[33655]: : cluster [DBG] pgmap v6705: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:49.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:49 smithi045 conmon[39718]: cluster 2021-07-31T04:23: 2021-07-31T04:23:49.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:49 smithi045 conmon[39718]: 48.038675+0000 mgr.x (mgr.24542) 6714 : cluster [DBG] pgmap v6705: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:51.235 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:51 smithi045 conmon[33655]: cluster 2021-07-31T04:23:50.039173+0000 mgr.x (mgr.24542) 2021-07-31T04:23:51.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:51 smithi154 conmon[34553]: cluster 2021-07-31T04:23:50.039173+0000 mgr.x (mgr.24542) 6715 : cluster [DBG] 2021-07-31T04:23:51.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:51 smithi154 conmon[34553]: pgmap v6706: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:51.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:51 smithi045 conmon[33655]: 6715 : cluster [DBG] pgmap v6706: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:51.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:51 smithi045 conmon[39718]: cluster 2021-07-31T04:23:50.039173+0000 mgr.x ( 2021-07-31T04:23:51.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:51 smithi045 conmon[39718]: mgr.24542) 6715 : cluster [DBG] pgmap v6706: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:53.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:53 smithi154 conmon[34553]: cluster 2021-07-31T04:23:52.039548+0000 mgr.x (mgr.24542) 6716 : cluster [DBG] pgmap v6707: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:53.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:53 smithi045 conmon[39718]: cluster 2021-07-31T04:23 2021-07-31T04:23:53.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:53 smithi045 conmon[39718]: :52.039548+0000 mgr.x (mgr.24542) 6716 : cluster [DBG] pgmap v6707: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:53.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:53 smithi045 conmon[33655]: cluster 2021-07-31T04:23:52.039548+0000 mgr.x (mgr.24542) 6716 : cluster [DBG] pgmap v6707: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:53.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:53 smithi045 conmon[33655]: 2021-07-31T04:23:55.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:55 smithi154 conmon[34553]: cluster 2021-07-31T04:23:54.040111+0000 mgr.x (mgr.24542) 6717 : cluster [DBG] pgmap v6708: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:55.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:55 smithi045 conmon[39718]: cluster 2021-07-31T04:23:54.040111+0000 mgr.x ( 2021-07-31T04:23:55.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:55 smithi045 conmon[39718]: mgr.24542) 6717 : cluster [DBG] pgmap v6708: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:55.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:55 smithi045 conmon[33655]: cluster 2021-07-31T04:23:54.040111+0000 mgr.x (mgr.24542) 6717 : cluster 2021-07-31T04:23:55.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:55 smithi045 conmon[33655]: [DBG] pgmap v6708: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:57.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:57 smithi154 conmon[34553]: cluster 2021-07-31T04:23:56.040567+0000 mgr.x (mgr.24542) 6718 : cluster [DBG] pgmap v6709: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:57.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:57 smithi154 conmon[34553]: 2021-07-31T04:23:57.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:57 smithi045 conmon[39718]: cluster 2021-07-31T04:23:56.040567+0000 mgr.x 2021-07-31T04:23:57.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:57 smithi045 conmon[39718]: (mgr.24542) 6718 : cluster [DBG] pgmap v6709: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:57.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:57 smithi045 conmon[33655]: cluster 2021-07-31T04:23:56.040567+0000 mgr.x (mgr.24542) 2021-07-31T04:23:57.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:57 smithi045 conmon[33655]: 6718 : cluster [DBG] pgmap v6709: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:59.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:59 smithi154 conmon[34553]: cluster 2021-07-31T04:23:58.041108+0000 mgr.x (mgr.24542) 6719 : cluster 2021-07-31T04:23:59.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:23:59 smithi154 conmon[34553]: [DBG] pgmap v6710: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:59.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:59 smithi045 conmon[33655]: cluster 2021-07-31T04:23:58.041108+0000 mgr.x (mgr.24542) 6719 : 2021-07-31T04:23:59.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:23:59 smithi045 conmon[33655]: cluster [DBG] pgmap v6710: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:23:59.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:59 smithi045 conmon[39718]: cluster 2021-07-31T04:23:58.041108+0000 mgr.x (mgr.24542) 6719 : 2021-07-31T04:23:59.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:23:59 smithi045 conmon[39718]: cluster [DBG] pgmap v6710: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:24:01.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:24:01 smithi154 conmon[34553]: cluster 2021-07-31T04:24:00.041674+0000 mgr.x (mgr.24542) 6720 : cluster [DBG] 2021-07-31T04:24:01.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:24:01 smithi154 conmon[34553]: pgmap v6711: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:24:01.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:24:01 smithi045 conmon[33655]: cluster 2021-07-31T04:24:00.041674+0000 mgr.x (mgr.24542) 6720 : cluster [DBG] pgmap v6711: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:24:01.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:24:01 smithi045 conmon[39718]: cluster 2021-07-31T04:24:00.041674+0000 mgr.x (mgr.24542 2021-07-31T04:24:01.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:24:01 smithi045 conmon[39718]: ) 6720 : cluster [DBG] pgmap v6711: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:24:03.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:24:03 smithi045 conmon[33655]: cluster 2021-07-31T04:24:02.042070+0000 mgr.x (mgr.24542) 6721 : cluster [DBG] 2021-07-31T04:24:03.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:24:03 smithi045 conmon[33655]: pgmap v6712: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:24:03.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:24:03 smithi045 conmon[39718]: cluster 2021-07-31T04:24: 2021-07-31T04:24:03.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:24:03 smithi045 conmon[39718]: 02.042070+0000 mgr.x (mgr.24542) 6721 : cluster [DBG] pgmap v6712: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:24:03.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:24:03 smithi154 conmon[34553]: cluster 2021-07-31T04:24:02.042070+0000 mgr.x (mgr.24542) 6721 : cluster [DBG] pgmap v6712: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:24:03.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:24:03 smithi154 conmon[34553]: 2021-07-31T04:24:05.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:24:05 smithi045 conmon[39718]: cluster 2021-07-31T04:24:04.042503+0000 mgr.x (mgr.24542) 6722 : cluster [DBG] pgmap v6713: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:24:05.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:24:05 smithi045 conmon[33655]: cluster 2021-07-31T04:24:04.042503+0000 mgr.x (mgr.24542) 6722 : cluster [DBG] pgmap v6713: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:24:05.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:24:05 smithi154 conmon[34553]: cluster 2021-07-31T04:24:04.042503+0000 mgr.x (mgr.24542) 6722 : cluster [DBG] pgmap v6713: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:24:05.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:24:05 smithi154 conmon[34553]: 2021-07-31T04:24:07.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:24:07 smithi045 conmon[39718]: cluster 2021-07-31T04:24:06 2021-07-31T04:24:07.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:24:07 smithi045 conmon[39718]: .043015+0000 mgr.x (mgr.24542) 6723 : cluster [DBG] pgmap v6714: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:24:07.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:24:07 smithi045 conmon[33655]: cluster 2021-07-31T04:24:06.043015+0000 mgr.x (mgr.24542) 2021-07-31T04:24:07.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:24:07 smithi045 conmon[33655]: 6723 : cluster [DBG] pgmap v6714: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:24:07.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:24:07 smithi154 conmon[34553]: cluster 2021-07-31T04:24:06.043015+0000 mgr.x (mgr.24542) 6723 : cluster [DBG] pgmap v6714: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:24:09.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:24:09 smithi045 conmon[39718]: cluster 2021-07-31T04:24:08.043408+0000 mgr.x (mgr.24542 2021-07-31T04:24:09.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:24:09 smithi045 conmon[39718]: ) 6724 : cluster [DBG] pgmap v6715: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:24:09.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:24:09 smithi045 conmon[33655]: cluster 2021-07-31T04:24:08.043408+0000 mgr.x (mgr.24542) 6724 : cluster [DBG] pgmap v6715: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:24:09.705 INFO:teuthology.orchestra.run.smithi045.stdout:true 2021-07-31T04:24:09.751 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:24:09 smithi154 conmon[34553]: cluster 2021-07-31T04:24:08.043408+0000 mgr.x (mgr.24542) 6724 : cluster [DBG] pgmap v6715: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:24:09.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:24:09 smithi154 conmon[34553]: 2021-07-31T04:24:09.982 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2021-07-31T04:24:09.982 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.a smithi045 *:9093,9094 host is offline 2h ago 2h - - 0.20.0 0881eb8f169f 3c43b4d8c4d3 2021-07-31T04:24:09.982 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.a smithi154 *:3000 host is offline 2h ago 2h 38.0M - 6.7.4 3b1077a63311 60b9d954f747 2021-07-31T04:24:09.982 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi154 *:8443 host is offline 2h ago 2h 54.4M - 17.0.0-6562-gf25860a2 974f23a2aee6 1260dfd1d879 2021-07-31T04:24:09.983 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi045 *:9283 host is offline 2h ago 2h - - 16.2.5 a29040f3000e 4472872c68fa 2021-07-31T04:24:09.983 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e 98c0a6dfe709 2021-07-31T04:24:09.983 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi154 host is offline 2h ago 2h 88.4M 2048M 16.2.5 a29040f3000e ea0311fb6425 2021-07-31T04:24:09.983 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi045 host is offline 2h ago 2h - 2048M 16.2.5 a29040f3000e a763c867885c 2021-07-31T04:24:09.983 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.a smithi045 *:9100 host is offline 2h ago 2h - - 0.18.1 e5a616e4b9cf 71bdbcbf130b 2021-07-31T04:24:09.984 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.b smithi154 *:9100 host is offline 2h ago 2h 13.9M - 0.18.1 e5a616e4b9cf f373a51f817a 2021-07-31T04:24:09.984 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e d3d6f9968caf 2021-07-31T04:24:09.984 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 02449b7173ab 2021-07-31T04:24:09.984 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 9a41da3863db 2021-07-31T04:24:09.984 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 host is offline 2h ago 2h - 4038M 16.2.5 a29040f3000e 471a87375a6f 2021-07-31T04:24:09.985 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi154 host is offline 2h ago 2h 143M 4292M 16.2.5 a29040f3000e 84d3cab3c41d 2021-07-31T04:24:09.985 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi154 host is offline 2h ago 2h 112M 4292M 16.2.5 a29040f3000e 10255adb7c24 2021-07-31T04:24:09.985 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi154 host is offline 2h ago 2h 106M 4292M 16.2.5 a29040f3000e 5d84820ff005 2021-07-31T04:24:09.985 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi154 host is offline 2h ago 2h 135M 4292M 16.2.5 a29040f3000e 36a93b1e5353 2021-07-31T04:24:09.985 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.a smithi154 *:9095 host is offline 2h ago 2h 38.4M - 2.18.1 de242295e225 535e16fd8eca 2021-07-31T04:24:10.316 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-07-31T04:24:10.316 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2021-07-31T04:24:10.316 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2021-07-31T04:24:10.316 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:24:10.316 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2021-07-31T04:24:10.317 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2021-07-31T04:24:10.317 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:24:10.317 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:24:10.317 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2021-07-31T04:24:10.317 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2021-07-31T04:24:10.318 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-07-31T04:24:10.318 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": {}, 2021-07-31T04:24:10.318 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2021-07-31T04:24:10.318 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2021-07-31T04:24:10.318 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.0.0-6562-gf25860a2 (f25860a2cae30438a8a8d15f1f74f6043cc73d75) quincy (dev)": 1 2021-07-31T04:24:10.319 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-07-31T04:24:10.319 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-07-31T04:24:11.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:24:11 smithi045 conmon[33655]: audit 2021-07-31T04:24:09.693037+0000 mgr.x (mgr.24542) 6725 : audit [DBG] 2021-07-31T04:24:11.689 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:24:11 smithi045 conmon[33655]: from='client.31459 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:24:11.690 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:24:11 smithi045 conmon[33655]: audit 2021-07-31T04:24:09.974300+0000 mgr.x (mgr.24542) 6726 : audit [DBG] from='client.31465 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:24:11.690 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:24:11 smithi045 conmon[33655]: cluster 2021-07-31T04:24:10.043880+0000 mgr.x (mgr.24542) 6727 : cluster [DBG] pgmap v6716: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:24:11.690 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:24:11 smithi045 conmon[33655]: audit 2021-07-31T04:24:10.316598+0000 mon.a (mon.0) 3475 : audit [DBG] 2021-07-31T04:24:11.690 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:24:11 smithi045 conmon[33655]: from='client.? 172.21.15.45:0/2327840932' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:24:11.691 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:24:11 smithi045 conmon[39718]: audit 2021-07-31T04:24:09.693037+0000 mgr.x (mgr.24542) 6725 : audit [DBG] 2021-07-31T04:24:11.691 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:24:11 smithi045 conmon[39718]: from='client.31459 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:24:11.691 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:24:11 smithi045 conmon[39718]: audit 2021-07-31T04:24:09.974300+0000 mgr.x 2021-07-31T04:24:11.691 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:24:11 smithi045 conmon[39718]: (mgr.24542) 6726 : audit [DBG] from='client.31465 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:24:11.691 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:24:11 smithi045 conmon[39718]: cluster 2021-07-31T04:24:11.692 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:24:11 smithi045 conmon[39718]: 2021-07-31T04:24:10.043880+0000 mgr.x (mgr.24542) 6727 : cluster 2021-07-31T04:24:11.692 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:24:11 smithi045 conmon[39718]: [DBG] pgmap v6716: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:24:11.692 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:24:11 smithi045 conmon[39718]: audit 2021-07-31T04:24:11.692 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:24:11 smithi045 conmon[39718]: 2021-07-31T04:24:10. 2021-07-31T04:24:11.692 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:24:11 smithi045 conmon[39718]: 316598+0000 mon.a (mon.0) 3475 : audit [DBG] from='client.? 172.21.15.45:0/2327840932' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:24:11.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:24:11 smithi154 conmon[34553]: audit 2021-07-31T04:24:09.693037+0000 mgr.x (mgr.24542) 6725 : audit [DBG] from='client.31459 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:24:11.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:24:11 smithi154 conmon[34553]: audit 2021-07-31T04:24:09.974300+0000 2021-07-31T04:24:11.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:24:11 smithi154 conmon[34553]: mgr.x (mgr.24542) 6726 : audit [DBG] from='client.31465 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-07-31T04:24:11.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:24:11 smithi154 conmon[34553]: cluster 2021-07-31T04:24:10.043880+0000 mgr.x (mgr.24542) 6727 : cluster [DBG] pgmap v6716: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:24:11.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:24:11 smithi154 conmon[34553]: audit 2021-07-31T04:24:10.316598+0000 mon.a (mon.0) 3475 : audit 2021-07-31T04:24:11.753 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:24:11 smithi154 conmon[34553]: [DBG] from='client.? 172.21.15.45:0/2327840932' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2021-07-31T04:24:13.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:24:13 smithi045 conmon[33655]: cluster 2021-07-31T04:24:12.044333+0000 mgr.x (mgr.24542) 6728 2021-07-31T04:24:13.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:24:13 smithi045 conmon[33655]: : cluster [DBG] pgmap v6717: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:24:13.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:24:13 smithi045 conmon[39718]: cluster 2021-07-31T04:24:12.044333 2021-07-31T04:24:13.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:24:13 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 6728 : cluster [DBG] pgmap v6717: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:24:13.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:24:13 smithi154 conmon[34553]: cluster 2021-07-31T04:24:12.044333+0000 mgr.x (mgr.24542) 6728 : cluster [DBG] pgmap v6717: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:24:15.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:24:15 smithi045 conmon[39718]: cluster 2021-07-31T04:24:14.044797+0000 mgr.x (mgr.24542) 6729 : cluster [DBG] pgmap v6718: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:24:15.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:24:15 smithi045 conmon[33655]: cluster 2021-07-31T04:24:14.044797+0000 mgr.x (mgr.24542) 6729 : 2021-07-31T04:24:15.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:24:15 smithi045 conmon[33655]: cluster [DBG] pgmap v6718: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:24:15.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:24:15 smithi154 conmon[34553]: cluster 2021-07-31T04:24:14.044797+0000 mgr.x (mgr.24542) 6729 : cluster [DBG] pgmap v6718: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:24:17.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:24:17 smithi154 conmon[34553]: cluster 2021-07-31T04:24:16.045156+0000 mgr.x (mgr.24542) 6730 : cluster [DBG] pgmap v6719: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:24:17.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:24:17 smithi045 conmon[33655]: cluster 2021-07-31T04:24:16.045156+0000 mgr.x (mgr.24542) 6730 : 2021-07-31T04:24:17.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:24:17 smithi045 conmon[33655]: cluster [DBG] pgmap v6719: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:24:17.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:24:17 smithi045 conmon[39718]: cluster 2021-07-31T04:24:16.045156+0000 mgr.x (mgr 2021-07-31T04:24:17.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:24:17 smithi045 conmon[39718]: .24542) 6730 : cluster [DBG] pgmap v6719: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:24:19.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:24:19 smithi154 conmon[34553]: cluster 2021-07-31T04:24:18.045664+0000 mgr.x (mgr.24542) 6731 : cluster [DBG] pgmap v6720: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:24:19.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:24:19 smithi045 conmon[33655]: cluster 2021-07-31T04:24:18.045664+0000 mgr.x (mgr.24542) 6731 : 2021-07-31T04:24:19.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:24:19 smithi045 conmon[33655]: cluster [DBG] pgmap v6720: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:24:19.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:24:19 smithi045 conmon[39718]: cluster 2021-07-31T04:24:18.045664+0000 mgr.x ( 2021-07-31T04:24:19.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:24:19 smithi045 conmon[39718]: mgr.24542) 6731 : cluster [DBG] pgmap v6720: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:24:21.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Jul 31 04:24:21 smithi154 conmon[34553]: cluster 2021-07-31T04:24:20.046315+0000 mgr.x (mgr.24542) 6732 : cluster [DBG] pgmap v6721: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:24:21.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:24:21 smithi045 conmon[33655]: cluster 2021-07-31T04:24:20.046315+0000 mgr.x (mgr.24542) 6732 : 2021-07-31T04:24:21.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Jul 31 04:24:21 smithi045 conmon[33655]: cluster [DBG] pgmap v6721: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:24:21.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:24:21 smithi045 conmon[39718]: cluster 2021-07-31T04:24:20.046315 2021-07-31T04:24:21.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Jul 31 04:24:21 smithi045 conmon[39718]: +0000 mgr.x (mgr.24542) 6732 : cluster [DBG] pgmap v6721: 9 pgs: 9 active+clean; 4.6 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2021-07-31T04:24:22.072 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2021-07-31T04:24:22.073 DEBUG:teuthology.task.console_log:Killing console logger for smithi045 2021-07-31T04:24:22.074 DEBUG:teuthology.task.console_log:Killing console logger for smithi154 2021-07-31T04:24:22.075 DEBUG:teuthology.exit:Finished running handlers